mirror of
https://github.com/Elysola/WebHooker.git
synced 2026-11-21 01:23:40 +00:00
feat(storage): Add delivery state store switch (D1 by default or KV) (#12)
This commit is contained in:
21 files changed
+756
-184
No files matched your search
+17
-16
@@ -45,22 +45,23 @@ WebHooker requires several secrets to function. For local development, store the
|
||||
|
||||
### Optional Secrets
|
||||
|
||||
| Variable | Description | Default |
|
||||
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
|
||||
| `DISCORD_PUBLIC_KEY` | Discord application public key (Developer Portal) — required for interactions | Unset → interactions return `401` |
|
||||
| `DISCORD_APPLICATION_ID` | Discord application id; auto-resolved when omitted | Auto-resolved |
|
||||
| `TELEGRAM_WEBHOOK_SECRET` | Secret token for `POST /telegram/webhook` verification (X-Telegram-Bot-Api-Secret-Token) | Derived from `TELEGRAM_TOKEN` |
|
||||
| `FEISHU_ENCRYPT_KEY` | Feishu event Encrypt Key: verifies `X-Lark-Signature` and decrypts encrypted callbacks | Unset |
|
||||
| `FEISHU_VERIFICATION_TOKEN` | Feishu event Verification Token, compared with the token in every callback | Unset |
|
||||
| `TELEGRAM_RICH_HEADER_HOST` | Base URL of an external rich-header service; when unset, the built-in `GET /api/richheader` serves the Telegram avatar card | Built-in `/api/richheader` |
|
||||
| `BASE_URL` | Public URL of the worker, for OAuth callbacks and the Telegram webhook sync | Unset — required in production |
|
||||
| `ADMIN_USER_IDS` | Comma-separated GitHub user IDs (or logins) allowed to access the Web UI; prefer numeric ids — a login is bound to the first account that signs in with it | Disabled |
|
||||
| `DEBUG_MODE` | Local development only: exact `true` bypasses all `/admin` authentication and grants full access | Disabled |
|
||||
| `ALLOW_SELF_SIGNUP` | When enabled (`1`/`true`), GitHub users without any group access get an isolated personal group on first login instead of `403` | Disabled |
|
||||
| `AUDIT_RETENTION_DAYS` | Audit-log retention in days for the scheduled cleanup | `90` |
|
||||
| `NUXT_PUBLIC_DOCS_URL` | Docs site URL used by the landing page (client-side runtime config) | Landing page defaults |
|
||||
| `NUXT_PUBLIC_REPO_URL` | GitHub repo URL used by the landing page | Landing page defaults |
|
||||
| `NUXT_PUBLIC_LEGAL_CONTACT` | Contact shown on `/terms` and `/privacy` | Unset → placeholder text |
|
||||
| Variable | Description | Default |
|
||||
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
|
||||
| `DISCORD_PUBLIC_KEY` | Discord application public key (Developer Portal) — required for interactions | Unset → interactions return `401` |
|
||||
| `DISCORD_APPLICATION_ID` | Discord application id; auto-resolved when omitted | Auto-resolved |
|
||||
| `TELEGRAM_WEBHOOK_SECRET` | Secret token for `POST /telegram/webhook` verification (X-Telegram-Bot-Api-Secret-Token) | Derived from `TELEGRAM_TOKEN` |
|
||||
| `FEISHU_ENCRYPT_KEY` | Feishu event Encrypt Key: verifies `X-Lark-Signature` and decrypts encrypted callbacks | Unset |
|
||||
| `FEISHU_VERIFICATION_TOKEN` | Feishu event Verification Token, compared with the token in every callback | Unset |
|
||||
| `TELEGRAM_RICH_HEADER_HOST` | Base URL of an external rich-header service; when unset, the built-in `GET /api/richheader` serves the Telegram avatar card | Built-in `/api/richheader` |
|
||||
| `BASE_URL` | Public URL of the worker, for OAuth callbacks and the Telegram webhook sync | Unset — required in production |
|
||||
| `ADMIN_USER_IDS` | Comma-separated GitHub user IDs (or logins) allowed to access the Web UI; prefer numeric ids — a login is bound to the first account that signs in with it | Disabled |
|
||||
| `DEBUG_MODE` | Local development only: exact `true` bypasses all `/admin` authentication and grants full access | Disabled |
|
||||
| `ALLOW_SELF_SIGNUP` | When enabled (`1`/`true`), GitHub users without any group access get an isolated personal group on first login instead of `403` | Disabled |
|
||||
| `AUDIT_RETENTION_DAYS` | Audit-log retention in days for the scheduled cleanup | `90` |
|
||||
| `DELIVERY_STATE_STORE` | Where webhook delivery state lives: `d1` (default) or `kv`. KV answers at the edge but the free plan allows only ~1,000 writes/day — see [Storage decisions](./storage#storage-decisions) | `d1` |
|
||||
| `NUXT_PUBLIC_DOCS_URL` | Docs site URL used by the landing page (client-side runtime config) | Landing page defaults |
|
||||
| `NUXT_PUBLIC_REPO_URL` | GitHub repo URL used by the landing page | Landing page defaults |
|
||||
| `NUXT_PUBLIC_LEGAL_CONTACT` | Contact shown on `/terms` and `/privacy` | Unset → placeholder text |
|
||||
|
||||
## Web UI
|
||||
|
||||
|
||||
+1
-1
@@ -38,4 +38,4 @@ No — the worker requires the KV and D1 bindings declared in `wrangler.jsonc` a
|
||||
|
||||
## Where is data stored?
|
||||
|
||||
Configuration lives in D1 (`d1_routes`/`d1_groups`, cached in memory for at most 60 seconds); webhook dedup, delivery state and message-update tracking also live in D1 with KV fallback; send/audit logs and platform↔GitHub links live in D1. R2 optionally parks oversized payloads. See [Storage Layout](./storage#storage-decisions).
|
||||
Configuration lives in D1 (`d1_routes`/`d1_groups`, cached in memory for at most 60 seconds); webhook dedup, delivery state and message-update tracking live in D1 too, with KV fallback and an opt-in `DELIVERY_STATE_STORE=kv`; send/audit logs and platform↔GitHub links live in D1. R2 optionally parks oversized payloads. See [Storage Layout](./storage#storage-decisions).
|
||||
@@ -22,7 +22,7 @@ GitHub / Gitea Webhook → Cloudflare Worker (Nuxt 4 / Nitro)
|
||||
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Cloudflare Worker** | HTTP ingress, signature verification, delivery dedup, event parsing, route matching, platform dispatch |
|
||||
| **Interactions Endpoint** | Verifies Ed25519 signatures and handles `/gh` interactions (slash commands, context-menu commands, buttons, modals) |
|
||||
| **KV** | Token storage (`token:{userId}`), OAuth state (`state:{hex}`), admin sessions, invites, delivery dedup/state/message-tracking fallback when D1 is unavailable (`delivery:*`, `delivery-state:*`, `msg:*`), message-update locks (`msg:lock:*`) |
|
||||
| **KV** | Token storage (`token:{userId}`), OAuth state (`state:{hex}`), admin sessions, invites, delivery dedup/state/message-tracking when D1 is unavailable or `DELIVERY_STATE_STORE=kv` (`delivery:*`, `delivery-state:*`, `msg:*`), message-update locks (`msg:lock:*`) |
|
||||
| **D1** | Routes/groups (`d1_routes`/`d1_groups`), send logs (`send_logs`), audit logs (`audit_logs`), dedup (`dedup_keys`), delivery state (`delivery_state`), message tracking (`message_tracking`), Discord↔GitHub links (`discord_links`), Telegram↔GitHub links (`telegram_links`), with R2 optionally parking oversized payloads |
|
||||
|
||||
### Data Flow
|
||||
|
||||
+33
-32
@@ -2,37 +2,37 @@
|
||||
|
||||
## KV Storage Layout
|
||||
|
||||
KV keeps only cache data and short-lived/ephemeral state. High-frequency writes (webhook dedup, delivery state, message tracking) live in D1 and only fall back to KV when D1 is unavailable or not yet migrated (see [Storage decisions](./storage#storage-decisions)).
|
||||
KV keeps only cache data and short-lived/ephemeral state. High-frequency writes (webhook dedup, delivery state, message tracking) live in D1 and only fall back to KV when D1 is unavailable or not yet migrated (see [Storage decisions](./storage#storage-decisions)). Setting `DELIVERY_STATE_STORE=kv` keeps that delivery state in KV instead.
|
||||
|
||||
| Key Pattern | Value | TTL |
|
||||
| ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
|
||||
| `config:routes` / `config:groups` | Legacy (pre-D1) configuration, read once to seed empty D1 tables | Permanent |
|
||||
| `config:d1-seeded` | Marks that D1 was seeded (an emptied D1 is never re-seeded) | Permanent |
|
||||
| `session:{id}` | Admin session `{ userId, login, loginTrusted }` | 1 day |
|
||||
| `token:{userId}` | `{ userId, accessToken, expiresAt, live? }` (a token is used for 10 hours) | 9 hours |
|
||||
| `token-reverse:{sha256}` | User id for reverse lookup by token | 9 hours |
|
||||
| `state:{hex}` | `{ redirectTo, expiresAt, link? }` (`link`: the chat account a bot `/gh login` binds) | 600 seconds |
|
||||
| `link:{nonce}` | A bot account link waiting for the GitHub user's confirmation | 600 seconds |
|
||||
| `invite:{token}` | `{ groupId, role, expiresAt, createdBy, note?, groupInstance, issuer }` | 7 days |
|
||||
| `invite:group:{id}` | Token index per group (keeps invite listing consistent) | 7 days |
|
||||
| `invite:by-group:{id}:{token}` / `invite:by-instance:{instance}:{token}` | Per-invite listing markers (by group id and by the issuing group's instance), so concurrent creates or a half-finished rename never hide an invite | Invite lifetime |
|
||||
| `login-owner:{login}` | GitHub user id of the first account seen with this login; another account using the login later is matched by user id only | Permanent |
|
||||
| `delivery:{provider}:{scope}:{id}` | Webhook delivery dedup fallback (D1 `dedup_keys` is primary); `scope` is `@global` for the global endpoint or `@i:{instance}` (the receiving group's immutable instance) for a group endpoint | 2–10 minutes |
|
||||
| `delivery-state:{provider}:{scope}:{id}` | Queue delivery state fallback (D1 `delivery_state` is primary) | 7 days |
|
||||
| `queue:payload:{provider}:{scope}:{id}:{uuid}` | Oversized webhook payload parked for the queue consumer when there is no R2 binding (one key per enqueue) | 4 days |
|
||||
| `tenant:{groupId}` / `tenant:i:{instance}` | Legacy per-group webhook secrets, read only for groups whose secret was never moved into their D1 row; removed on rotate/delete | Permanent |
|
||||
| `group:renamed:{oldId}` | KV-only deployments: where a renamed group went (with D1 this alias is a `dedup_keys` row written in the rename transaction) | 4 days |
|
||||
| `msg:{chain}:{platform}:{target}` | Message tracking fallback (D1 `message_tracking` is primary) | 30 days |
|
||||
| `msg:lock:…` | Lock serializing updates of one message chain when D1 is unavailable (with D1 it is a `dedup_keys` row) | 240 seconds |
|
||||
| `storage:kv-fallback` | Marks that state was written to KV while a D1 table was missing; while it lives, D1 misses are also looked up in KV | 35 days |
|
||||
| `cmd:guilds:{appId}` | Per Discord application: each guild's registered command hash and the sync cursor | Permanent |
|
||||
| `cmd:registered:global:{appId}` | Hash of the commands last registered globally (re-registered when it changes, and daily) | 1 day |
|
||||
| `config:discord-app-id:{tokenHash}` | Discord application id resolved for a bot token | Permanent |
|
||||
| `discord-modal:{id}` | Comment-modal context, only when it does not fit in the 100-character `custom_id` | 1 day |
|
||||
| `feishu:token:{appId}` | Feishu `tenant_access_token` per app (also held in memory) | Token expiry − 6 minutes |
|
||||
| `telegram:webhook:{botId}` | Hash of the webhook registration last set (`setWebhook` is skipped while it matches) | 1 day |
|
||||
| `i18n:{lang}` | Translation overrides merged on top of English | Permanent |
|
||||
| `config:identity-migrated` | KV-only deployments: group identities are managed by this server (the one-time identity migration ran) | Permanent |
|
||||
| Key Pattern | Value | TTL |
|
||||
| ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
|
||||
| `config:routes` / `config:groups` | Legacy (pre-D1) configuration, read once to seed empty D1 tables | Permanent |
|
||||
| `config:d1-seeded` | Marks that D1 was seeded (an emptied D1 is never re-seeded) | Permanent |
|
||||
| `session:{id}` | Admin session `{ userId, login, loginTrusted }` | 1 day |
|
||||
| `token:{userId}` | `{ userId, accessToken, expiresAt, live? }` (a token is used for 10 hours) | 9 hours |
|
||||
| `token-reverse:{sha256}` | User id for reverse lookup by token | 9 hours |
|
||||
| `state:{hex}` | `{ redirectTo, expiresAt, link? }` (`link`: the chat account a bot `/gh login` binds) | 600 seconds |
|
||||
| `link:{nonce}` | A bot account link waiting for the GitHub user's confirmation | 600 seconds |
|
||||
| `invite:{token}` | `{ groupId, role, expiresAt, createdBy, note?, groupInstance, issuer }` | 7 days |
|
||||
| `invite:group:{id}` | Token index per group (keeps invite listing consistent) | 7 days |
|
||||
| `invite:by-group:{id}:{token}` / `invite:by-instance:{instance}:{token}` | Per-invite listing markers (by group id and by the issuing group's instance), so concurrent creates or a half-finished rename never hide an invite | Invite lifetime |
|
||||
| `login-owner:{login}` | GitHub user id of the first account seen with this login; another account using the login later is matched by user id only | Permanent |
|
||||
| `delivery:{provider}:{scope}:{id}` | Webhook delivery dedup (D1 `dedup_keys` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable); `scope` is `@global` for the global endpoint or `@i:{instance}` (the receiving group's immutable instance) for a group endpoint | 2–10 minutes |
|
||||
| `delivery-state:{provider}:{scope}:{id}` | Queue delivery state (D1 `delivery_state` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable) | 7 days |
|
||||
| `queue:payload:{provider}:{scope}:{id}:{uuid}` | Oversized webhook payload parked for the queue consumer when there is no R2 binding (one key per enqueue) | 4 days |
|
||||
| `tenant:{groupId}` / `tenant:i:{instance}` | Legacy per-group webhook secrets, read only for groups whose secret was never moved into their D1 row; removed on rotate/delete | Permanent |
|
||||
| `group:renamed:{oldId}` | KV-only deployments: where a renamed group went (with D1 this alias is a `dedup_keys` row written in the rename transaction) | 4 days |
|
||||
| `msg:{chain}:{platform}:{target}` | Message tracking (D1 `message_tracking` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable) | 30 days |
|
||||
| `msg:lock:…` | Lock serializing updates of one message chain (D1 `dedup_keys` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable) | 240 seconds |
|
||||
| `storage:kv-fallback` | Marks that state was written to KV instead of D1 (missing table, or `DELIVERY_STATE_STORE=kv`); while it lives, D1 misses are also looked up in KV | 35 days |
|
||||
| `cmd:guilds:{appId}` | Per Discord application: each guild's registered command hash and the sync cursor | Permanent |
|
||||
| `cmd:registered:global:{appId}` | Hash of the commands last registered globally (re-registered when it changes, and daily) | 1 day |
|
||||
| `config:discord-app-id:{tokenHash}` | Discord application id resolved for a bot token | Permanent |
|
||||
| `discord-modal:{id}` | Comment-modal context, only when it does not fit in the 100-character `custom_id` | 1 day |
|
||||
| `feishu:token:{appId}` | Feishu `tenant_access_token` per app (also held in memory) | Token expiry − 6 minutes |
|
||||
| `telegram:webhook:{botId}` | Hash of the webhook registration last set (`setWebhook` is skipped while it matches) | 1 day |
|
||||
| `i18n:{lang}` | Translation overrides merged on top of English | Permanent |
|
||||
| `config:identity-migrated` | KV-only deployments: group identities are managed by this server (the one-time identity migration ran) | Permanent |
|
||||
|
||||
## D1 Storage Layout
|
||||
|
||||
@@ -45,7 +45,7 @@ The D1 database (`DB` binding, database `webhooker`) holds the source of truth f
|
||||
| `send_logs` | One row per dispatch attempt (route id, event, target, ok/error, duration, error code, detail) |
|
||||
| `audit_logs` | One row per admin operation (login/logout, group/route/member/invite changes) |
|
||||
| `dedup_keys` | Atomic one-time claims and records (`INSERT ... ON CONFLICT` UPSERT, key + expiry): webhook delivery dedup and its confirmations (`accepted:*`), custom-webhook nonces (`nonce:{secret-scope}:{nonce}`), single-use and revoked invites (`invite-used:*`, `invite-revoked:*`), bot command/event dedup (`bot:{platform}:…`), revoked sessions and tokens and each user's live token (`session-revoked:*`, `token-revoked:*`, `token-live:*`), used account-link confirmations (`link-used:*`), retired group ids (`group-retired:*`) and rename aliases (`group-alias:*`), installation provisioning records (`installation-provisioned:*`, `installation-bound:*`), route × targets already finished or failed for good by a queued delivery (`delivery-target:*`, `delivery-failed:*`), group log summaries already sent, message-chain locks (`msg:lock:*`) and one-time setup markers |
|
||||
| `delivery_state` | Queue delivery state (`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`); taking a delivery is a conditional UPSERT (a lease), so two consumers never dispatch it at once |
|
||||
| `delivery_state` | Queue delivery state (`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`); taking a delivery is a conditional UPSERT (a lease), so two consumers never dispatch it at once in D1 mode (with `DELIVERY_STATE_STORE=kv` the state lives in KV, where a fresh lease stamp plays the same role without fencing) |
|
||||
| `message_tracking` | Message tracking for in-place updates (`event_id` + `target_id` → message id and the version it shows); chains are keyed by the group's instance, so a group created later under the same id never edits its messages |
|
||||
| `d1_fragments` | Named filter fragments per group |
|
||||
| `feishu_links` | Maps `feishu_user_id` → `github_user_id` for `/gh` Feishu commands |
|
||||
@@ -69,9 +69,10 @@ A payload is parked when the queue message would exceed 120,000 bytes (UTF-8). W
|
||||
## Storage Decisions
|
||||
|
||||
- **D1 is authoritative** for config and delivery metadata; KV holds only caches and short-lived state.
|
||||
- The `canUseD1` probe (`server/lib/storage/d1.ts`, checks for `prepare` + `batch`) gates every D1 store. The high-frequency stores (dedup and other one-time claims, delivery state, message tracking) also fall back to KV while their D1 table does not exist yet (an unmigrated database). Any other D1 failure propagates — the request answers with an error and the sender or queue retries — because switching stores per call would split a claim between D1 and KV.
|
||||
- The `canUseD1` probe (`server/lib/storage/d1.ts`, checks for `prepare` + `batch`) gates every D1 store except the delivery ones, which follow `DELIVERY_STATE_STORE` (`server/lib/storage/state-store.ts`). The high-frequency stores (dedup and other one-time claims, delivery state, message tracking) also fall back to KV while their D1 table does not exist yet (an unmigrated database). Any other D1 failure propagates — the request answers with an error and the sender or queue retries — because switching stores per call would split a claim between D1 and KV.
|
||||
- When D1 configuration reads fail, the request answers with an error and the sender or queue retries; there is no KV copy of the configuration, and dispatch never treats unreadable configuration as empty. A stored route or group row that cannot be read only makes its own group's deliveries fail visibly (and retry) — see [Routes → Unreadable routes](./routes#unreadable-routes).
|
||||
- Configuration writes never fall back: with a D1 binding, a failed write is reported as an error instead of being parked in KV. Writes are per group (one row, or one group's routes/fragments in one transaction) with compare-and-swap on the group row, so concurrent edits of different groups never overwrite each other. Reads for access checks, read-modify-write paths, webhook ingress and the queue consumer bypass every cache (a revoked route or target stops at once); other reads are cached in memory for at most 60 seconds per isolate.
|
||||
- Legacy KV configuration is copied into D1 once, only while both D1 tables are empty, and never over existing rows; a marker prevents an emptied configuration from being resurrected.
|
||||
- This keeps per-event KV writes near zero on the Workers Free plan (1,000 writes/day): dedup, delivery state and message tracking now write D1 rows instead (D1 Free allows 100,000 rows written/day).
|
||||
- `DELIVERY_STATE_STORE=kv` moves the delivery hot path back to KV: ingress dedup and replay nonces, delivery state and leases, the finished-target ledger and permanent-failure markers, message tracking and chain locks, and group-log summary claims. It trades D1's generous write quota for edge latency; on the free plan KV allows about 1,000 writes/day, so low-volume deployments fit and busy ones do not. A fresh lease stamp still keeps a second consumer off a delivery that is being sent, but KV has no compare-and-set: a lease taken over after its holder vanished, or a raced claim, can still be duplicated. Claims and states written in KV mode are marked (`storage:kv-fallback`) and still honored after switching back to D1; the other direction does not read D1, so the D1 rows of the last days (claims and their confirmations, delivery states, the finished-target ledger, message tracking for 30 days) are invisible — an already-finished event redelivered is sent, already-sent targets are sent, and an in-progress edit chain posts a new message. Draining the queue only keeps in-flight deliveries from overlapping; those rows stay invisible either way, so carry the state over (migrate it, or accept the repeats). While the marker lives, the D1 claims that stay in D1 (invites, sessions, tokens, bot interactions, group ids) also read KV once per claim to honor copies written in KV mode; KV reads are far cheaper than writes. Everything else — configuration, send logs, audit, metrics, and the claims tied to a D1 transaction of their own (invites, sessions, tokens, bot interactions, group ids, installation provisioning) — stays in D1 in either mode. An unrecognized value is reported and treated as `d1`.
|
||||
- R2's free tier (10 GB-month storage, 1M Class A ops/month) comfortably absorbs payload parking without touching the KV write quota.
|
||||
Reference in new issue
Block a user