Files
WebHooker/docs/guide/storage.md
T

79 lines
27 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Storage Layout
## KV Storage Layout
KV keeps only cache data and short-lived/ephemeral state. High-frequency writes (webhook dedup, delivery state, message tracking) live in D1 and only fall back to KV when D1 is unavailable or not yet migrated (see [Storage decisions](./storage#storage-decisions)). Setting `DELIVERY_STATE_STORE=kv` keeps that delivery state in KV instead.
| Key Pattern | Value | TTL |
| ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
| `config:routes` / `config:groups` | Legacy (pre-D1) configuration, read once to seed empty D1 tables | Permanent |
| `config:d1-seeded` | Marks that D1 was seeded (an emptied D1 is never re-seeded) | Permanent |
| `session:{id}` | Admin session `{ userId, login, loginTrusted }` | 1 day |
| `token:{userId}` | `{ userId, accessToken, expiresAt, live? }` (a token is used for 10 hours) | 9 hours |
| `token-reverse:{sha256}` | User id for reverse lookup by token | 9 hours |
| `state:{hex}` | `{ redirectTo, expiresAt, link? }` (`link`: the chat account a bot `/gh login` binds) | 600 seconds |
| `link:{nonce}` | A bot account link waiting for the GitHub user's confirmation | 600 seconds |
| `invite:{token}` | `{ groupId, role, expiresAt, createdBy, note?, groupInstance, issuer }` | 7 days |
| `invite:group:{id}` | Token index per group (keeps invite listing consistent) | 7 days |
| `invite:by-group:{id}:{token}` / `invite:by-instance:{instance}:{token}` | Per-invite listing markers (by group id and by the issuing group's instance), so concurrent creates or a half-finished rename never hide an invite | Invite lifetime |
| `login-owner:{login}` | GitHub user id of the first account seen with this login; another account using the login later is matched by user id only | Permanent |
| `delivery:{provider}:{scope}:{id}` | Webhook delivery dedup (D1 `dedup_keys` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable); `scope` is `@global` for the global endpoint or `@i:{instance}` (the receiving group's immutable instance) for a group endpoint | 2–10 minutes |
| `delivery-state:{provider}:{scope}:{id}` | Queue delivery state (D1 `delivery_state` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable) | 7 days |
| `queue:payload:{provider}:{scope}:{id}:{uuid}` | Oversized webhook payload parked for the queue consumer when there is no R2 binding (one key per enqueue) | 4 days |
| `tenant:{groupId}` / `tenant:i:{instance}` | Legacy per-group webhook secrets, read only for groups whose secret was never moved into their D1 row; removed on rotate/delete | Permanent |
| `group:renamed:{oldId}` | KV-only deployments: where a renamed group went (with D1 this alias is a `dedup_keys` row written in the rename transaction) | 4 days |
| `msg:{chain}:{platform}:{target}` | Message tracking (D1 `message_tracking` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable) | 30 days |
| `msg:lock:…` | Lock serializing updates of one message chain (D1 `dedup_keys` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable) | 240 seconds |
| `storage:kv-fallback` | Marks that state was written to KV instead of D1 (missing table, or `DELIVERY_STATE_STORE=kv`); while it lives, D1 misses are also looked up in KV | 35 days |
| `cmd:guilds:{appId}` | Per Discord application: each guild's registered command hash and the sync cursor | Permanent |
| `cmd:registered:global:{appId}` | Hash of the commands last registered globally (re-registered when it changes, and daily) | 1 day |
| `config:discord-app-id:{tokenHash}` | Discord application id resolved for a bot token | Permanent |
| `discord-modal:{id}` | Comment-modal context, only when it does not fit in the 100-character `custom_id` | 1 day |
| `feishu:token:{appId}` | Feishu `tenant_access_token` per app (also held in memory) | Token expiry − 6 minutes |
| `telegram:webhook:{botId}` | Hash of the webhook registration last set (`setWebhook` is skipped while it matches) | 1 day |
| `i18n:{lang}` | Translation overrides merged on top of English | Permanent |
| `config:identity-migrated` | KV-only deployments: group identities are managed by this server (the one-time identity migration ran) | Permanent |
## D1 Storage Layout
The D1 database (`DB` binding, database `webhooker`) holds the source of truth for configuration, delivery logs and high-frequency ephemeral state:
| Table | Purpose |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `d1_groups` | Groups (authoritative config, seeded from legacy KV `config:groups`). Each row also holds server-only fields that are never sent to the console: the immutable `instance` id, `legacyId`, and the group's webhook secret |
| `d1_routes` | Routes per group (authoritative config, seeded from legacy KV `config:routes`) |
| `send_logs` | One row per dispatch attempt (route id, event, target, ok/error, duration, error code, detail) |
| `audit_logs` | One row per admin operation (login/logout, group/route/member/invite changes) |
| `dedup_keys` | Atomic one-time claims and records (`INSERT ... ON CONFLICT` UPSERT, key + expiry): webhook delivery dedup and its confirmations (`accepted:*`), custom-webhook nonces (`nonce:{secret-scope}:{nonce}`), single-use and revoked invites (`invite-used:*`, `invite-revoked:*`), bot command/event dedup (`bot:{platform}:…`), revoked sessions and tokens and each user's live token (`session-revoked:*`, `token-revoked:*`, `token-live:*`), used account-link confirmations (`link-used:*`), retired group ids (`group-retired:*`) and rename aliases (`group-alias:*`), installation provisioning records (`installation-provisioned:*`, `installation-bound:*`), route × targets already finished or failed for good by a queued delivery (`delivery-target:*`, `delivery-failed:*`), group log summaries already sent, message-chain locks (`msg:lock:*`) and one-time setup markers |
| `delivery_state` | Queue delivery state (`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`); taking a delivery is a conditional UPSERT (a lease), so two consumers never dispatch it at once in D1 mode (with `DELIVERY_STATE_STORE=kv` the state lives in KV, where a fresh lease stamp plays the same role without fencing) |
| `message_tracking` | Message tracking for in-place updates (`event_id` + `target_id` → message id and the version it shows); chains are keyed by the group's instance, so a group created later under the same id never edits its messages |
| `d1_fragments` | Named filter fragments per group |
| `feishu_links` | Maps `feishu_user_id` → `github_user_id` for `/gh` Feishu commands |
| `discord_links` | Maps `discord_user_id` → `github_user_id` for `/gh` Discord commands |
| `telegram_links` | Maps `telegram_user_id` → `github_user_id` for `/gh` Telegram commands |
`audit_logs` is pruned by the scheduled trigger after `AUDIT_RETENTION_DAYS` (default 90). The `storage-prune` task removes expired `dedup_keys`, `delivery_state` rows older than 7 days, `message_tracking` rows not updated for 30 days (every edit refreshes a row) and orphaned R2 payloads. `send_logs` is not pruned automatically. See [Logs](./logs) for the log row fields.
Migration `0013_prune_indexes.sql` adds the indexes these reads rely on: `delivery_state(updated_at)` for the prune, two partial indexes on failed `send_logs` rows for the metrics page, and `audit_logs(group_id, ts)` in place of the single-column group index. Every index also costs a D1 row write when its columns are written, so only indexes whose reads would otherwise cost far more are added.
## R2 Storage Layout
R2 (`PAYLOAD` binding, bucket `webhooker-payloads`) stores oversized webhook payloads that are too large for a queue message or KV:
| Object Pattern | Purpose | Retention |
| --------------------------------- | ----------------------------------------------- | --------------------------------------------------- |
| `webhooks/YYYY/MM/DD/<uuid>.json` | Oversized payload parked for the queue consumer | deleted after dispatch; orphans pruned after 5 days |
A payload is parked when the queue message would exceed 120,000 bytes (UTF-8). When the `PAYLOAD` binding is absent, oversized payloads fall back to the KV key `queue:payload:{provider}:{scope}:{id}:{uuid}` (4 day TTL, matching the queue's message retention). If sending the queue message fails, the parked copy is kept (the queue may have accepted the message even though the call failed); unused copies expire or are pruned. A parked payload that is gone when the consumer needs it is never dispatched as an empty event: the delivery stays `retrying` and a `PAYLOAD_MISSING` send-log entry asks for the webhook to be redelivered from its source.
## Storage Decisions
- **D1 is authoritative** for config and delivery metadata; KV holds only caches and short-lived state.
- The `canUseD1` probe (`server/lib/storage/d1.ts`, checks for `prepare` + `batch`) gates every D1 store except the delivery ones, which follow `DELIVERY_STATE_STORE` (`server/lib/storage/state-store.ts`). The high-frequency stores (dedup and other one-time claims, delivery state, message tracking) also fall back to KV while their D1 table does not exist yet (an unmigrated database). Any other D1 failure propagates — the request answers with an error and the sender or queue retries — because switching stores per call would split a claim between D1 and KV.
- When D1 configuration reads fail, the request answers with an error and the sender or queue retries; there is no KV copy of the configuration, and dispatch never treats unreadable configuration as empty. A stored route or group row that cannot be read only makes its own group's deliveries fail visibly (and retry) — see [Routes → Unreadable routes](./routes#unreadable-routes).
- Configuration writes never fall back: with a D1 binding, a failed write is reported as an error instead of being parked in KV. Writes are per group (one row, or one group's routes/fragments in one transaction) with compare-and-swap on the group row, so concurrent edits of different groups never overwrite each other. Reads for access checks, read-modify-write paths, webhook ingress and the queue consumer bypass every cache (a revoked route or target stops at once); other reads are cached in memory for at most 60 seconds per isolate.
- Legacy KV configuration is copied into D1 once, only while both D1 tables are empty, and never over existing rows; a marker prevents an emptied configuration from being resurrected.
- This keeps per-event KV writes near zero on the Workers Free plan (1,000 writes/day): dedup, delivery state and message tracking now write D1 rows instead (D1 Free allows 100,000 rows written/day).
- `DELIVERY_STATE_STORE=kv` moves the delivery hot path back to KV: ingress dedup and replay nonces, delivery state and leases, the finished-target ledger and permanent-failure markers, message tracking and chain locks, and group-log summary claims. It trades D1's generous write quota for edge latency; on the free plan KV allows about 1,000 writes/day, so low-volume deployments fit and busy ones do not. A fresh lease stamp still keeps a second consumer off a delivery that is being sent, but KV has no compare-and-set: a lease taken over after its holder vanished, or a raced claim, can still be duplicated. Claims and states written in KV mode are marked (`storage:kv-fallback`) and still honored after switching back to D1; the other direction does not read D1, so the D1 rows of the last days (claims and their confirmations, delivery states, the finished-target ledger, message tracking for 30 days) are invisible — an already-finished event redelivered is sent, already-sent targets are sent, and an in-progress edit chain posts a new message. Draining the queue only keeps in-flight deliveries from overlapping; those rows stay invisible either way, so carry the state over (migrate it, or accept the repeats). While the marker lives, the D1 claims that stay in D1 (invites, sessions, tokens, bot interactions, group ids) also read KV once per claim to honor copies written in KV mode; KV reads are far cheaper than writes. Everything else — configuration, send logs, audit, metrics, and the claims tied to a D1 transaction of their own (invites, sessions, tokens, bot interactions, group ids, installation provisioning) — stays in D1 in either mode. An unrecognized value is reported and treated as `d1`.
- R2's free tier (10 GB-month storage, 1M Class A ops/month) comfortably absorbs payload parking without touching the KV write quota.