feat(storage): Add delivery state store switch (D1 by default or KV) (#12)

This commit is contained in:
w9auto committed 2026-11-17 22:46:50 +08:00
1 parent fc4495dfb4
commit 0c126d4fee
21 files changed
+756 -184

No files matched your search

+17 -16
View File
@@ -45,22 +45,23 @@ WebHooker requires several secrets to function. For local development, store the
### Optional Secrets
| Variable | Description | Default |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `DISCORD_PUBLIC_KEY` | Discord application public key (Developer Portal) — required for interactions | Unset → interactions return `401` |
| `DISCORD_APPLICATION_ID` | Discord application id; auto-resolved when omitted | Auto-resolved |
| `TELEGRAM_WEBHOOK_SECRET` | Secret token for `POST /telegram/webhook` verification (X-Telegram-Bot-Api-Secret-Token) | Derived from `TELEGRAM_TOKEN` |
| `FEISHU_ENCRYPT_KEY` | Feishu event Encrypt Key: verifies `X-Lark-Signature` and decrypts encrypted callbacks | Unset |
| `FEISHU_VERIFICATION_TOKEN` | Feishu event Verification Token, compared with the token in every callback | Unset |
| `TELEGRAM_RICH_HEADER_HOST` | Base URL of an external rich-header service; when unset, the built-in `GET /api/richheader` serves the Telegram avatar card | Built-in `/api/richheader` |
| `BASE_URL` | Public URL of the worker, for OAuth callbacks and the Telegram webhook sync | Unset — required in production |
| `ADMIN_USER_IDS` | Comma-separated GitHub user IDs (or logins) allowed to access the Web UI; prefer numeric ids — a login is bound to the first account that signs in with it | Disabled |
| `DEBUG_MODE` | Local development only: exact `true` bypasses all `/admin` authentication and grants full access | Disabled |
| `ALLOW_SELF_SIGNUP` | When enabled (`1`/`true`), GitHub users without any group access get an isolated personal group on first login instead of `403` | Disabled |
| `AUDIT_RETENTION_DAYS` | Audit-log retention in days for the scheduled cleanup | `90` |
| `NUXT_PUBLIC_DOCS_URL` | Docs site URL used by the landing page (client-side runtime config) | Landing page defaults |
| `NUXT_PUBLIC_REPO_URL` | GitHub repo URL used by the landing page | Landing page defaults |
| `NUXT_PUBLIC_LEGAL_CONTACT` | Contact shown on `/terms` and `/privacy` | Unset → placeholder text |
| Variable | Description | Default |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `DISCORD_PUBLIC_KEY` | Discord application public key (Developer Portal) — required for interactions | Unset → interactions return `401` |
| `DISCORD_APPLICATION_ID` | Discord application id; auto-resolved when omitted | Auto-resolved |
| `TELEGRAM_WEBHOOK_SECRET` | Secret token for `POST /telegram/webhook` verification (X-Telegram-Bot-Api-Secret-Token) | Derived from `TELEGRAM_TOKEN` |
| `FEISHU_ENCRYPT_KEY` | Feishu event Encrypt Key: verifies `X-Lark-Signature` and decrypts encrypted callbacks | Unset |
| `FEISHU_VERIFICATION_TOKEN` | Feishu event Verification Token, compared with the token in every callback | Unset |
| `TELEGRAM_RICH_HEADER_HOST` | Base URL of an external rich-header service; when unset, the built-in `GET /api/richheader` serves the Telegram avatar card | Built-in `/api/richheader` |
| `BASE_URL` | Public URL of the worker, for OAuth callbacks and the Telegram webhook sync | Unset — required in production |
| `ADMIN_USER_IDS` | Comma-separated GitHub user IDs (or logins) allowed to access the Web UI; prefer numeric ids — a login is bound to the first account that signs in with it | Disabled |
| `DEBUG_MODE` | Local development only: exact `true` bypasses all `/admin` authentication and grants full access | Disabled |
| `ALLOW_SELF_SIGNUP` | When enabled (`1`/`true`), GitHub users without any group access get an isolated personal group on first login instead of `403` | Disabled |
| `AUDIT_RETENTION_DAYS` | Audit-log retention in days for the scheduled cleanup | `90` |
| `DELIVERY_STATE_STORE` | Where webhook delivery state lives: `d1` (default) or `kv`. KV answers at the edge but the free plan allows only ~1,000 writes/day — see [Storage decisions](./storage#storage-decisions) | `d1` |
| `NUXT_PUBLIC_DOCS_URL` | Docs site URL used by the landing page (client-side runtime config) | Landing page defaults |
| `NUXT_PUBLIC_REPO_URL` | GitHub repo URL used by the landing page | Landing page defaults |
| `NUXT_PUBLIC_LEGAL_CONTACT` | Contact shown on `/terms` and `/privacy` | Unset → placeholder text |
## Web UI
+1 -1
View File
@@ -38,4 +38,4 @@ No — the worker requires the KV and D1 bindings declared in `wrangler.jsonc` a
## Where is data stored?
Configuration lives in D1 (`d1_routes`/`d1_groups`, cached in memory for at most 60 seconds); webhook dedup, delivery state and message-update tracking also live in D1 with KV fallback; send/audit logs and platform↔GitHub links live in D1. R2 optionally parks oversized payloads. See [Storage Layout](./storage#storage-decisions).
Configuration lives in D1 (`d1_routes`/`d1_groups`, cached in memory for at most 60 seconds); webhook dedup, delivery state and message-update tracking live in D1 too, with KV fallback and an opt-in `DELIVERY_STATE_STORE=kv`; send/audit logs and platform↔GitHub links live in D1. R2 optionally parks oversized payloads. See [Storage Layout](./storage#storage-decisions).
+1 -1
View File
@@ -22,7 +22,7 @@ GitHub / Gitea Webhook → Cloudflare Worker (Nuxt 4 / Nitro)
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Cloudflare Worker** | HTTP ingress, signature verification, delivery dedup, event parsing, route matching, platform dispatch |
| **Interactions Endpoint** | Verifies Ed25519 signatures and handles `/gh` interactions (slash commands, context-menu commands, buttons, modals) |
| **KV** | Token storage (`token:{userId}`), OAuth state (`state:{hex}`), admin sessions, invites, delivery dedup/state/message-tracking fallback when D1 is unavailable (`delivery:*`, `delivery-state:*`, `msg:*`), message-update locks (`msg:lock:*`) |
| **KV** | Token storage (`token:{userId}`), OAuth state (`state:{hex}`), admin sessions, invites, delivery dedup/state/message-tracking when D1 is unavailable or `DELIVERY_STATE_STORE=kv` (`delivery:*`, `delivery-state:*`, `msg:*`), message-update locks (`msg:lock:*`) |
| **D1** | Routes/groups (`d1_routes`/`d1_groups`), send logs (`send_logs`), audit logs (`audit_logs`), dedup (`dedup_keys`), delivery state (`delivery_state`), message tracking (`message_tracking`), Discord↔GitHub links (`discord_links`), Telegram↔GitHub links (`telegram_links`), with R2 optionally parking oversized payloads |
### Data Flow
+33 -32
View File
@@ -2,37 +2,37 @@
## KV Storage Layout
KV keeps only cache data and short-lived/ephemeral state. High-frequency writes (webhook dedup, delivery state, message tracking) live in D1 and only fall back to KV when D1 is unavailable or not yet migrated (see [Storage decisions](./storage#storage-decisions)).
KV keeps only cache data and short-lived/ephemeral state. High-frequency writes (webhook dedup, delivery state, message tracking) live in D1 and only fall back to KV when D1 is unavailable or not yet migrated (see [Storage decisions](./storage#storage-decisions)). Setting `DELIVERY_STATE_STORE=kv` keeps that delivery state in KV instead.
| Key Pattern | Value | TTL |
| ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
| `config:routes` / `config:groups` | Legacy (pre-D1) configuration, read once to seed empty D1 tables | Permanent |
| `config:d1-seeded` | Marks that D1 was seeded (an emptied D1 is never re-seeded) | Permanent |
| `session:{id}` | Admin session `{ userId, login, loginTrusted }` | 1 day |
| `token:{userId}` | `{ userId, accessToken, expiresAt, live? }` (a token is used for 10 hours) | 9 hours |
| `token-reverse:{sha256}` | User id for reverse lookup by token | 9 hours |
| `state:{hex}` | `{ redirectTo, expiresAt, link? }` (`link`: the chat account a bot `/gh login` binds) | 600 seconds |
| `link:{nonce}` | A bot account link waiting for the GitHub user's confirmation | 600 seconds |
| `invite:{token}` | `{ groupId, role, expiresAt, createdBy, note?, groupInstance, issuer }` | 7 days |
| `invite:group:{id}` | Token index per group (keeps invite listing consistent) | 7 days |
| `invite:by-group:{id}:{token}` / `invite:by-instance:{instance}:{token}` | Per-invite listing markers (by group id and by the issuing group's instance), so concurrent creates or a half-finished rename never hide an invite | Invite lifetime |
| `login-owner:{login}` | GitHub user id of the first account seen with this login; another account using the login later is matched by user id only | Permanent |
| `delivery:{provider}:{scope}:{id}` | Webhook delivery dedup fallback (D1 `dedup_keys` is primary); `scope` is `@global` for the global endpoint or `@i:{instance}` (the receiving group's immutable instance) for a group endpoint | 2–10 minutes |
| `delivery-state:{provider}:{scope}:{id}` | Queue delivery state fallback (D1 `delivery_state` is primary) | 7 days |
| `queue:payload:{provider}:{scope}:{id}:{uuid}` | Oversized webhook payload parked for the queue consumer when there is no R2 binding (one key per enqueue) | 4 days |
| `tenant:{groupId}` / `tenant:i:{instance}` | Legacy per-group webhook secrets, read only for groups whose secret was never moved into their D1 row; removed on rotate/delete | Permanent |
| `group:renamed:{oldId}` | KV-only deployments: where a renamed group went (with D1 this alias is a `dedup_keys` row written in the rename transaction) | 4 days |
| `msg:{chain}:{platform}:{target}` | Message tracking fallback (D1 `message_tracking` is primary) | 30 days |
| `msg:lock:…` | Lock serializing updates of one message chain when D1 is unavailable (with D1 it is a `dedup_keys` row) | 240 seconds |
| `storage:kv-fallback` | Marks that state was written to KV while a D1 table was missing; while it lives, D1 misses are also looked up in KV | 35 days |
| `cmd:guilds:{appId}` | Per Discord application: each guild's registered command hash and the sync cursor | Permanent |
| `cmd:registered:global:{appId}` | Hash of the commands last registered globally (re-registered when it changes, and daily) | 1 day |
| `config:discord-app-id:{tokenHash}` | Discord application id resolved for a bot token | Permanent |
| `discord-modal:{id}` | Comment-modal context, only when it does not fit in the 100-character `custom_id` | 1 day |
| `feishu:token:{appId}` | Feishu `tenant_access_token` per app (also held in memory) | Token expiry − 6 minutes |
| `telegram:webhook:{botId}` | Hash of the webhook registration last set (`setWebhook` is skipped while it matches) | 1 day |
| `i18n:{lang}` | Translation overrides merged on top of English | Permanent |
| `config:identity-migrated` | KV-only deployments: group identities are managed by this server (the one-time identity migration ran) | Permanent |
| Key Pattern | Value | TTL |
| ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
| `config:routes` / `config:groups` | Legacy (pre-D1) configuration, read once to seed empty D1 tables | Permanent |
| `config:d1-seeded` | Marks that D1 was seeded (an emptied D1 is never re-seeded) | Permanent |
| `session:{id}` | Admin session `{ userId, login, loginTrusted }` | 1 day |
| `token:{userId}` | `{ userId, accessToken, expiresAt, live? }` (a token is used for 10 hours) | 9 hours |
| `token-reverse:{sha256}` | User id for reverse lookup by token | 9 hours |
| `state:{hex}` | `{ redirectTo, expiresAt, link? }` (`link`: the chat account a bot `/gh login` binds) | 600 seconds |
| `link:{nonce}` | A bot account link waiting for the GitHub user's confirmation | 600 seconds |
| `invite:{token}` | `{ groupId, role, expiresAt, createdBy, note?, groupInstance, issuer }` | 7 days |
| `invite:group:{id}` | Token index per group (keeps invite listing consistent) | 7 days |
| `invite:by-group:{id}:{token}` / `invite:by-instance:{instance}:{token}` | Per-invite listing markers (by group id and by the issuing group's instance), so concurrent creates or a half-finished rename never hide an invite | Invite lifetime |
| `login-owner:{login}` | GitHub user id of the first account seen with this login; another account using the login later is matched by user id only | Permanent |
| `delivery:{provider}:{scope}:{id}` | Webhook delivery dedup (D1 `dedup_keys` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable); `scope` is `@global` for the global endpoint or `@i:{instance}` (the receiving group's immutable instance) for a group endpoint | 2–10 minutes |
| `delivery-state:{provider}:{scope}:{id}` | Queue delivery state (D1 `delivery_state` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable) | 7 days |
| `queue:payload:{provider}:{scope}:{id}:{uuid}` | Oversized webhook payload parked for the queue consumer when there is no R2 binding (one key per enqueue) | 4 days |
| `tenant:{groupId}` / `tenant:i:{instance}` | Legacy per-group webhook secrets, read only for groups whose secret was never moved into their D1 row; removed on rotate/delete | Permanent |
| `group:renamed:{oldId}` | KV-only deployments: where a renamed group went (with D1 this alias is a `dedup_keys` row written in the rename transaction) | 4 days |
| `msg:{chain}:{platform}:{target}` | Message tracking (D1 `message_tracking` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable) | 30 days |
| `msg:lock:…` | Lock serializing updates of one message chain (D1 `dedup_keys` by default; used while `DELIVERY_STATE_STORE=kv` or when D1 is unavailable) | 240 seconds |
| `storage:kv-fallback` | Marks that state was written to KV instead of D1 (missing table, or `DELIVERY_STATE_STORE=kv`); while it lives, D1 misses are also looked up in KV | 35 days |
| `cmd:guilds:{appId}` | Per Discord application: each guild's registered command hash and the sync cursor | Permanent |
| `cmd:registered:global:{appId}` | Hash of the commands last registered globally (re-registered when it changes, and daily) | 1 day |
| `config:discord-app-id:{tokenHash}` | Discord application id resolved for a bot token | Permanent |
| `discord-modal:{id}` | Comment-modal context, only when it does not fit in the 100-character `custom_id` | 1 day |
| `feishu:token:{appId}` | Feishu `tenant_access_token` per app (also held in memory) | Token expiry − 6 minutes |
| `telegram:webhook:{botId}` | Hash of the webhook registration last set (`setWebhook` is skipped while it matches) | 1 day |
| `i18n:{lang}` | Translation overrides merged on top of English | Permanent |
| `config:identity-migrated` | KV-only deployments: group identities are managed by this server (the one-time identity migration ran) | Permanent |
## D1 Storage Layout
@@ -45,7 +45,7 @@ The D1 database (`DB` binding, database `webhooker`) holds the source of truth f
| `send_logs` | One row per dispatch attempt (route id, event, target, ok/error, duration, error code, detail) |
| `audit_logs` | One row per admin operation (login/logout, group/route/member/invite changes) |
| `dedup_keys` | Atomic one-time claims and records (`INSERT ... ON CONFLICT` UPSERT, key + expiry): webhook delivery dedup and its confirmations (`accepted:*`), custom-webhook nonces (`nonce:{secret-scope}:{nonce}`), single-use and revoked invites (`invite-used:*`, `invite-revoked:*`), bot command/event dedup (`bot:{platform}:…`), revoked sessions and tokens and each user's live token (`session-revoked:*`, `token-revoked:*`, `token-live:*`), used account-link confirmations (`link-used:*`), retired group ids (`group-retired:*`) and rename aliases (`group-alias:*`), installation provisioning records (`installation-provisioned:*`, `installation-bound:*`), route × targets already finished or failed for good by a queued delivery (`delivery-target:*`, `delivery-failed:*`), group log summaries already sent, message-chain locks (`msg:lock:*`) and one-time setup markers |
| `delivery_state` | Queue delivery state (`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`); taking a delivery is a conditional UPSERT (a lease), so two consumers never dispatch it at once |
| `delivery_state` | Queue delivery state (`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`); taking a delivery is a conditional UPSERT (a lease), so two consumers never dispatch it at once in D1 mode (with `DELIVERY_STATE_STORE=kv` the state lives in KV, where a fresh lease stamp plays the same role without fencing) |
| `message_tracking` | Message tracking for in-place updates (`event_id` + `target_id` → message id and the version it shows); chains are keyed by the group's instance, so a group created later under the same id never edits its messages |
| `d1_fragments` | Named filter fragments per group |
| `feishu_links` | Maps `feishu_user_id` → `github_user_id` for `/gh` Feishu commands |
@@ -69,9 +69,10 @@ A payload is parked when the queue message would exceed 120,000 bytes (UTF-8). W
## Storage Decisions
- **D1 is authoritative** for config and delivery metadata; KV holds only caches and short-lived state.
- The `canUseD1` probe (`server/lib/storage/d1.ts`, checks for `prepare` + `batch`) gates every D1 store. The high-frequency stores (dedup and other one-time claims, delivery state, message tracking) also fall back to KV while their D1 table does not exist yet (an unmigrated database). Any other D1 failure propagates — the request answers with an error and the sender or queue retries — because switching stores per call would split a claim between D1 and KV.
- The `canUseD1` probe (`server/lib/storage/d1.ts`, checks for `prepare` + `batch`) gates every D1 store except the delivery ones, which follow `DELIVERY_STATE_STORE` (`server/lib/storage/state-store.ts`). The high-frequency stores (dedup and other one-time claims, delivery state, message tracking) also fall back to KV while their D1 table does not exist yet (an unmigrated database). Any other D1 failure propagates — the request answers with an error and the sender or queue retries — because switching stores per call would split a claim between D1 and KV.
- When D1 configuration reads fail, the request answers with an error and the sender or queue retries; there is no KV copy of the configuration, and dispatch never treats unreadable configuration as empty. A stored route or group row that cannot be read only makes its own group's deliveries fail visibly (and retry) — see [Routes → Unreadable routes](./routes#unreadable-routes).
- Configuration writes never fall back: with a D1 binding, a failed write is reported as an error instead of being parked in KV. Writes are per group (one row, or one group's routes/fragments in one transaction) with compare-and-swap on the group row, so concurrent edits of different groups never overwrite each other. Reads for access checks, read-modify-write paths, webhook ingress and the queue consumer bypass every cache (a revoked route or target stops at once); other reads are cached in memory for at most 60 seconds per isolate.
- Legacy KV configuration is copied into D1 once, only while both D1 tables are empty, and never over existing rows; a marker prevents an emptied configuration from being resurrected.
- This keeps per-event KV writes near zero on the Workers Free plan (1,000 writes/day): dedup, delivery state and message tracking now write D1 rows instead (D1 Free allows 100,000 rows written/day).
- `DELIVERY_STATE_STORE=kv` moves the delivery hot path back to KV: ingress dedup and replay nonces, delivery state and leases, the finished-target ledger and permanent-failure markers, message tracking and chain locks, and group-log summary claims. It trades D1's generous write quota for edge latency; on the free plan KV allows about 1,000 writes/day, so low-volume deployments fit and busy ones do not. A fresh lease stamp still keeps a second consumer off a delivery that is being sent, but KV has no compare-and-set: a lease taken over after its holder vanished, or a raced claim, can still be duplicated. Claims and states written in KV mode are marked (`storage:kv-fallback`) and still honored after switching back to D1; the other direction does not read D1, so the D1 rows of the last days (claims and their confirmations, delivery states, the finished-target ledger, message tracking for 30 days) are invisible — an already-finished event redelivered is sent, already-sent targets are sent, and an in-progress edit chain posts a new message. Draining the queue only keeps in-flight deliveries from overlapping; those rows stay invisible either way, so carry the state over (migrate it, or accept the repeats). While the marker lives, the D1 claims that stay in D1 (invites, sessions, tokens, bot interactions, group ids) also read KV once per claim to honor copies written in KV mode; KV reads are far cheaper than writes. Everything else — configuration, send logs, audit, metrics, and the claims tied to a D1 transaction of their own (invites, sessions, tokens, bot interactions, group ids, installation provisioning) — stays in D1 in either mode. An unrecognized value is reported and treated as `d1`.
- R2's free tier (10 GB-month storage, 1M Class A ops/month) comfortably absorbs payload parking without touching the KV write quota.
+17 -16
View File
@@ -43,22 +43,23 @@ WebHooker 的运行需要若干密钥。本地开发时放入 `.dev.vars`,生
### 可选密钥
| 变量 | 说明 | 默认值 |
| --------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------ |
| `DISCORD_PUBLIC_KEY` | Discord 应用的公钥(开发者门户获取),交互功能必需 | 未设置时交互返回 401 |
| `DISCORD_APPLICATION_ID` | Discord 应用 ID;省略时自动获取 | 自动获取 |
| `TELEGRAM_WEBHOOK_SECRET` | `POST /telegram/webhook` 验签密钥(X-Telegram-Bot-Api-Secret-Token) | 由 `TELEGRAM_TOKEN` 派生 |
| `FEISHU_ENCRYPT_KEY` | 飞书事件 Encrypt Key:校验 `X-Lark-Signature` 并解密加密回调 | 未设置 |
| `FEISHU_VERIFICATION_TOKEN` | 飞书事件 Verification Token:与每个回调中的 token 比对 | 未设置 |
| `TELEGRAM_RICH_HEADER_HOST` | 外部 rich-header 服务的基础 URL;未设置时使用内置 `GET /api/richheader` 提供 Telegram 头像卡片 | 内置 `/api/richheader` |
| `BASE_URL` | worker 的公网地址,用于 OAuth 回调与 Telegram webhook 同步 | 未设置——生产环境必填 |
| `ADMIN_USER_IDS` | 允许访问 WebUI 的 GitHub 用户 ID(或登录名),逗号分隔;建议写数字 ID——登录名会绑定到第一个用它登录的账号 | 未设置时 WebUI 关闭 |
| `DEBUG_MODE` | 仅本地开发:精确设为 `true` 时跳过全部 `/admin` 身份验证并授予完整权限 | 关闭 |
| `ALLOW_SELF_SIGNUP` | 开启(`1`/`true`)后,没有任何分组权限的 GitHub 用户首次登录会自动获得隔离的个人分组而非 403 | 关闭 |
| `AUDIT_RETENTION_DAYS` | 定时清理时审计日志的保留天数 | `90` |
| `NUXT_PUBLIC_DOCS_URL` | 落地页使用的文档站 URL(客户端运行时配置) | 落地页默认值 |
| `NUXT_PUBLIC_REPO_URL` | 落地页使用的 GitHub 仓库 URL | 落地页默认值 |
| `NUXT_PUBLIC_LEGAL_CONTACT` | `/terms` 与 `/privacy` 页面展示的联系方式 | 未设置时显示占位文本 |
| 变量 | 说明 | 默认值 |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
| `DISCORD_PUBLIC_KEY` | Discord 应用的公钥(开发者门户获取),交互功能必需 | 未设置时交互返回 401 |
| `DISCORD_APPLICATION_ID` | Discord 应用 ID;省略时自动获取 | 自动获取 |
| `TELEGRAM_WEBHOOK_SECRET` | `POST /telegram/webhook` 验签密钥(X-Telegram-Bot-Api-Secret-Token) | 由 `TELEGRAM_TOKEN` 派生 |
| `FEISHU_ENCRYPT_KEY` | 飞书事件 Encrypt Key:校验 `X-Lark-Signature` 并解密加密回调 | 未设置 |
| `FEISHU_VERIFICATION_TOKEN` | 飞书事件 Verification Token:与每个回调中的 token 比对 | 未设置 |
| `TELEGRAM_RICH_HEADER_HOST` | 外部 rich-header 服务的基础 URL;未设置时使用内置 `GET /api/richheader` 提供 Telegram 头像卡片 | 内置 `/api/richheader` |
| `BASE_URL` | worker 的公网地址,用于 OAuth 回调与 Telegram webhook 同步 | 未设置——生产环境必填 |
| `ADMIN_USER_IDS` | 允许访问 WebUI 的 GitHub 用户 ID(或登录名),逗号分隔;建议写数字 ID——登录名会绑定到第一个用它登录的账号 | 未设置时 WebUI 关闭 |
| `DEBUG_MODE` | 仅本地开发:精确设为 `true` 时跳过全部 `/admin` 身份验证并授予完整权限 | 关闭 |
| `ALLOW_SELF_SIGNUP` | 开启(`1`/`true`)后,没有任何分组权限的 GitHub 用户首次登录会自动获得隔离的个人分组而非 403 | 关闭 |
| `AUDIT_RETENTION_DAYS` | 定时清理时审计日志的保留天数 | `90` |
| `DELIVERY_STATE_STORE` | webhook 投递状态存放位置:`d1`(默认)或 `kv`。KV 在边缘响应,但免费版每日仅约 1000 次写——见[存储决策](./storage#存储决策) | `d1` |
| `NUXT_PUBLIC_DOCS_URL` | 落地页使用的文档站 URL(客户端运行时配置) | 落地页默认值 |
| `NUXT_PUBLIC_REPO_URL` | 落地页使用的 GitHub 仓库 URL | 落地页默认值 |
| `NUXT_PUBLIC_LEGAL_CONTACT` | `/terms` 与 `/privacy` 页面展示的联系方式 | 未设置时显示占位文本 |
## Web 控制台
+1 -1
View File
@@ -38,4 +38,4 @@
## 数据存储在哪里?
配置存于 D1(`d1_routes`/`d1_groups`,在内存中最多缓存 60 秒);webhook 去重、投递状态与消息更新追踪同样存于 D1(KV 回退);发送/审计日志与平台↔GitHub 绑定存于 D1;超大负载可选存于 R2。见[存储布局](./storage#存储决策)。
配置存于 D1(`d1_routes`/`d1_groups`,在内存中最多缓存 60 秒);webhook 去重、投递状态与消息更新追踪同样存于 D1(KV 回退,可选 `DELIVERY_STATE_STORE=kv` 切换到 KV);发送/审计日志与平台↔GitHub 绑定存于 D1;超大负载可选存于 R2。见[存储布局](./storage#存储决策)。
+1 -1
View File
@@ -22,7 +22,7 @@ GitHub / Gitea Webhook → Cloudflare Worker (Nuxt 4 / Nitro)
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Cloudflare Worker** | HTTP 入口、签名验证、投递去重、事件解析、路由匹配、平台分发 |
| **Interactions Endpoint** | 验证 Ed25519 签名并处理 `/gh` 交互(斜杠命令、右键菜单、按钮、modal) |
| **KV** | Token 存储 (`token:{userId}`)、OAuth 状态 (`state:{hex}`)、管理员会话、邀请、投递去重/投递状态/消息更新追踪的回退(`delivery:*`、`delivery-state:*`、`msg:*`,仅在 D1 不可用时使用)、消息更新锁 (`msg:lock:*`) |
| **KV** | Token 存储 (`token:{userId}`)、OAuth 状态 (`state:{hex}`)、管理员会话、邀请、D1 不可用或 `DELIVERY_STATE_STORE=kv` 时的投递去重/投递状态/消息更新追踪(`delivery:*`、`delivery-state:*`、`msg:*`)、消息更新锁 (`msg:lock:*`) |
| **D1** | 路由/分组 (`d1_routes`/`d1_groups`)、发送日志 (`send_logs`)、审计日志 (`audit_logs`)、去重 (`dedup_keys`)、投递状态 (`delivery_state`)、消息追踪 (`message_tracking`)、Discord↔GitHub 绑定 (`discord_links`)、Telegram↔GitHub 绑定 (`telegram_links`),超大负载可选的 R2 存储 |
### 数据流
+33 -32
View File
@@ -2,37 +2,37 @@
## KV 存储布局
KV 只保留缓存数据和短期/临时状态。高频写入(webhook 去重、投递状态、消息追踪)存放在 D1,仅在 D1 不可用或未迁移时才回退到 KV(见[存储决策](#存储决策))。
KV 只保留缓存数据和短期/临时状态。高频写入(webhook 去重、投递状态、消息追踪)存放在 D1,仅在 D1 不可用或未迁移时才回退到 KV(见[存储决策](#存储决策))。设置 `DELIVERY_STATE_STORE=kv` 时这些投递状态改为存放在 KV。
| 键模式 | 值 | TTL |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| `config:routes` / `config:groups` | 旧版(D1 之前)配置,仅在 D1 表为空时读取一次用于播种 | 永久 |
| `config:d1-seeded` | 标记 D1 已播种(已清空的 D1 不会被再次播种) | 永久 |
| `session:{id}` | 管理员会话 `{ userId, login, loginTrusted }` | 1 天 |
| `token:{userId}` | `{ userId, accessToken, expiresAt, live? }`(Token 可使用 10 小时) | 9 小时 |
| `token-reverse:{sha256}` | 用于按 Token 反查的用户 id | 9 小时 |
| `state:{hex}` | `{ redirectTo, expiresAt, link? }`(`link`:机器人 `/gh login` 要绑定的聊天账号) | 600 秒 |
| `link:{nonce}` | 等待 GitHub 用户确认的机器人账号绑定 | 600 秒 |
| `invite:{token}` | `{ groupId, role, expiresAt, createdBy, note?, groupInstance, issuer }` | 7 天 |
| `invite:group:{id}` | 每组的 Token 索引(保证邀请列表一致性) | 7 天 |
| `invite:by-group:{id}:{token}` / `invite:by-instance:{instance}:{token}` | 每个邀请的列表标记(按分组 id 与签发分组的 instance),并发创建或改名中断时邀请也不会从列表中消失 | 邀请有效期 |
| `login-owner:{login}` | 第一个使用该登录名的 GitHub 账号 id;之后用同名登录的其他账号只按 user id 匹配 | 永久 |
| `delivery:{provider}:{scope}:{id}` | Webhook 投递去重回退(D1 `dedup_keys` 为主);全局入口的 `scope` 为 `@global`,分组入口为 `@i:{instance}`(接收分组不可变的 instance) | 2~10 分钟 |
| `delivery-state:{provider}:{scope}:{id}` | 队列投递状态回退(D1 `delivery_state` 为主) | 7 天 |
| `queue:payload:{provider}:{scope}:{id}:{uuid}` | 未绑定 R2 时暂存供队列消费者读取的超大 webhook 负载(每次入队一个键) | 4 天 |
| `tenant:{groupId}` / `tenant:i:{instance}` | 旧版分组 webhook secret,仅在密钥尚未存入 D1 分组行时作为回退读取;轮换或删除时清除 | 永久 |
| `group:renamed:{oldId}` | 仅 KV 部署:分组改名后的新 id(有 D1 时该别名是改名事务中写入的 `dedup_keys` 行) | 4 天 |
| `msg:{chain}:{platform}:{target}` | 消息追踪回退(D1 `message_tracking` 为主) | 30 天 |
| `msg:lock:…` | D1 不可用时串行化同一消息链更新的锁(有 D1 时是 `dedup_keys` 行) | 240 秒 |
| `storage:kv-fallback` | 标记曾在 D1 表缺失时把状态写入 KV;有效期内 D1 未命中时也会查 KV | 35 天 |
| `cmd:guilds:{appId}` | 每个 Discord 应用一条:各服务器已注册命令的哈希与同步游标 | 永久 |
| `cmd:registered:global:{appId}` | 最近一次全局注册的命令哈希(变化时重新注册,并每天一次) | 1 天 |
| `config:discord-app-id:{tokenHash}` | 按 bot token 解析出的 Discord 应用 id | 永久 |
| `discord-modal:{id}` | 评论模态框上下文,仅在放不进 100 字符的 `custom_id` 时写入 | 1 天 |
| `feishu:token:{appId}` | 每个飞书应用的 `tenant_access_token`(内存中也会保留) | Token 有效期 − 6 分钟 |
| `telegram:webhook:{botId}` | 最近一次 webhook 注册内容的哈希(相同时跳过 `setWebhook`) | 1 天 |
| `i18n:{lang}` | 叠加在英文之上的翻译覆盖 | 永久 |
| `config:identity-migrated` | 仅 KV 部署:分组身份已由本服务器管理(一次性身份迁移已执行) | 永久 |
| 键模式 | 值 | TTL |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| `config:routes` / `config:groups` | 旧版(D1 之前)配置,仅在 D1 表为空时读取一次用于播种 | 永久 |
| `config:d1-seeded` | 标记 D1 已播种(已清空的 D1 不会被再次播种) | 永久 |
| `session:{id}` | 管理员会话 `{ userId, login, loginTrusted }` | 1 天 |
| `token:{userId}` | `{ userId, accessToken, expiresAt, live? }`(Token 可使用 10 小时) | 9 小时 |
| `token-reverse:{sha256}` | 用于按 Token 反查的用户 id | 9 小时 |
| `state:{hex}` | `{ redirectTo, expiresAt, link? }`(`link`:机器人 `/gh login` 要绑定的聊天账号) | 600 秒 |
| `link:{nonce}` | 等待 GitHub 用户确认的机器人账号绑定 | 600 秒 |
| `invite:{token}` | `{ groupId, role, expiresAt, createdBy, note?, groupInstance, issuer }` | 7 天 |
| `invite:group:{id}` | 每组的 Token 索引(保证邀请列表一致性) | 7 天 |
| `invite:by-group:{id}:{token}` / `invite:by-instance:{instance}:{token}` | 每个邀请的列表标记(按分组 id 与签发分组的 instance),并发创建或改名中断时邀请也不会从列表中消失 | 邀请有效期 |
| `login-owner:{login}` | 第一个使用该登录名的 GitHub 账号 id;之后用同名登录的其他账号只按 user id 匹配 | 永久 |
| `delivery:{provider}:{scope}:{id}` | Webhook 投递去重(默认 D1 `dedup_keys`;`DELIVERY_STATE_STORE=kv` 或 D1 不可用时使用);全局入口的 `scope` 为 `@global`,分组入口为 `@i:{instance}`(接收分组不可变的 instance) | 2~10 分钟 |
| `delivery-state:{provider}:{scope}:{id}` | 队列投递状态(默认 D1 `delivery_state`;`DELIVERY_STATE_STORE=kv` 或 D1 不可用时使用) | 7 天 |
| `queue:payload:{provider}:{scope}:{id}:{uuid}` | 未绑定 R2 时暂存供队列消费者读取的超大 webhook 负载(每次入队一个键) | 4 天 |
| `tenant:{groupId}` / `tenant:i:{instance}` | 旧版分组 webhook secret,仅在密钥尚未存入 D1 分组行时作为回退读取;轮换或删除时清除 | 永久 |
| `group:renamed:{oldId}` | 仅 KV 部署:分组改名后的新 id(有 D1 时该别名是改名事务中写入的 `dedup_keys` 行) | 4 天 |
| `msg:{chain}:{platform}:{target}` | 消息追踪(默认 D1 `message_tracking`;`DELIVERY_STATE_STORE=kv` 或 D1 不可用时使用) | 30 天 |
| `msg:lock:…` | 串行化同一消息链更新的锁(默认 D1 `dedup_keys`;`DELIVERY_STATE_STORE=kv` 或 D1 不可用时使用) | 240 秒 |
| `storage:kv-fallback` | 标记状态写在了 KV 而非 D1(表缺失,或 `DELIVERY_STATE_STORE=kv`);有效期内 D1 未命中时也会查 KV | 35 天 |
| `cmd:guilds:{appId}` | 每个 Discord 应用一条:各服务器已注册命令的哈希与同步游标 | 永久 |
| `cmd:registered:global:{appId}` | 最近一次全局注册的命令哈希(变化时重新注册,并每天一次) | 1 天 |
| `config:discord-app-id:{tokenHash}` | 按 bot token 解析出的 Discord 应用 id | 永久 |
| `discord-modal:{id}` | 评论模态框上下文,仅在放不进 100 字符的 `custom_id` 时写入 | 1 天 |
| `feishu:token:{appId}` | 每个飞书应用的 `tenant_access_token`(内存中也会保留) | Token 有效期 − 6 分钟 |
| `telegram:webhook:{botId}` | 最近一次 webhook 注册内容的哈希(相同时跳过 `setWebhook`) | 1 天 |
| `i18n:{lang}` | 叠加在英文之上的翻译覆盖 | 永久 |
| `config:identity-migrated` | 仅 KV 部署:分组身份已由本服务器管理(一次性身份迁移已执行) | 永久 |
## D1 存储布局
@@ -45,7 +45,7 @@ D1 数据库(`DB` 绑定,数据库 `webhooker`)保存配置、投递日志
| `send_logs` | 每次分发尝试一行(路由 id、事件、目标、ok/error、耗时、错误码、详情) |
| `audit_logs` | 每次管理员操作一行(登录/登出、分组/路由/成员/邀请变更) |
| `dedup_keys` | 原子的一次性占用与记录(`INSERT ... ON CONFLICT` UPSERT,键 + 过期时间):webhook 投递去重及其确认(`accepted:*`)、自定义 webhook nonce(`nonce:{secret 作用域}:{nonce}`)、已使用与已撤销的邀请(`invite-used:*`、`invite-revoked:*`)、机器人命令/事件去重(`bot:{platform}:…`)、已撤销的会话与 token 及每个用户当前有效的 token(`session-revoked:*`、`token-revoked:*`、`token-live:*`)、已使用的账号绑定确认(`link-used:*`)、已退役的分组 id(`group-retired:*`)与改名别名(`group-alias:*`)、安装自动配置记录(`installation-provisioned:*`、`installation-bound:*`)、排队投递中已完成或已永久失败的路由 × 目标(`delivery-target:*`、`delivery-failed:*`)、已发送的分组日志摘要、消息链锁(`msg:lock:*`)以及一次性初始化标记 |
| `delivery_state` | 队列投递状态(`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`);领取投递是一次条件 UPSERT(租约),两个消费者不会同时分发同一投递 |
| `delivery_state` | 队列投递状态(`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`);领取投递是一次条件 UPSERT(租约),D1 模式下两个消费者不会同时分发同一投递(`DELIVERY_STATE_STORE=kv` 时状态存于 KV,由新鲜的租约时间戳承担同样职责,但没有 fencing) |
| `message_tracking` | 原地更新用消息追踪(`event_id` + `target_id` → 消息 id 及其所显示的版本);消息链按分组 instance 建键,之后以同一 id 新建的分组不会编辑这些消息 |
| `d1_fragments` | 每个分组的命名过滤器片段 |
| `feishu_links` | 映射 `feishu_user_id` → `github_user_id`,供 `/gh` 飞书命令使用 |
@@ -69,9 +69,10 @@ R2(`PAYLOAD` 绑定,bucket `webhooker-payloads`)存储对队列消息或 K
## 存储决策
- **D1 是配置与投递元数据的权威数据源**;KV 只保存缓存和短期状态。
- `canUseD1` 探测(`server/lib/storage/d1.ts`,检查 `prepare` + `batch`)为每个 D1 存储做门槛判定。高频存储(去重及其他一次性占用、投递状态、消息追踪)在其 D1 表尚不存在时(数据库未迁移)回退到 KV。其他 D1 故障会直接抛出——请求返回错误,由发送方或队列重试——因为按调用切换存储会让同一占用分散在 D1 与 KV 两处。
- `canUseD1` 探测(`server/lib/storage/d1.ts`,检查 `prepare` + `batch`)为除投递相关之外的每个 D1 存储做门槛判定;投递存储跟随 `DELIVERY_STATE_STORE`(`server/lib/storage/state-store.ts`)。高频存储(去重及其他一次性占用、投递状态、消息追踪)在其 D1 表尚不存在时(数据库未迁移)回退到 KV。其他 D1 故障会直接抛出——请求返回错误,由发送方或队列重试——因为按调用切换存储会让同一占用分散在 D1 与 KV 两处。
- D1 配置读取失败时,请求返回错误,由发送方或队列重试;配置没有 KV 副本,分发也绝不会把不可读的配置当作空配置。存储中无法读取的路由或分组行只会让所属分组的投递可见地失败并重试——见[路由 → 不可读的路由](./routes#不可读的路由)。
- 配置写入从不回退:存在 D1 绑定时,写入失败会作为错误返回,而不是暂存到 KV。写入按分组进行(单行,或单个分组的路由/片段在一个事务中),分组行使用比较并交换,不同分组的并发编辑互不覆盖。权限检查、读改写路径、webhook 入口与队列消费者的读取绕过所有缓存(撤销的路由或目标立即停止投递);其他读取在每个 isolate 的内存中最多缓存 60 秒。
- 旧版 KV 配置只在两张 D1 表都为空时复制一次,且从不覆盖已有行;标记位保证被清空的配置不会复活。
- 这让 Workers 免费版的每事件 KV 写入趋近于零(每日 1000 次写):去重、投递状态和消息追踪改为写 D1 行(D1 免费版每日可写 10 万行)。
- `DELIVERY_STATE_STORE=kv` 把投递热路径移回 KV:入口去重与重放 nonce、投递状态与租约、已完成目标台账与永久失败标记、消息追踪与消息链锁、分组日志摘要占用。它用 D1 宽松的写入额度换取边缘延迟;免费版 KV 每日约 1000 次写,低流量部署够用,繁忙部署不够。新鲜的租约时间戳仍能挡住第二个消费者接手正在发送的投递,但 KV 没有比较并交换:持有者消失后租约被接手、或并发抢占,仍可能重复投递。KV 模式下写入的占用与状态会打上 `storage:kv-fallback` 标记,切回 D1 后仍被认可;反方向不读取 D1,最近几天写入的 D1 行(占用及其确认、投递状态、已完成目标台账、30 天的消息追踪)都不可见——已完成的投递重投会再次发送、已发送的目标会再次发送、进行中的更新链会新发一条消息——排空队列只能减少在途任务重叠,这些 D1 行仍然不可见;要保留它们需要迁移状态,或接受重复发送。标记有效期内,仍留在 D1 的占用(邀请、会话、Token、机器人交互、分组 id)每次 claim 也会多读一次 KV,以认可 KV 模式下写入的副本;KV 读取远比写入便宜。其余部分——配置、发送日志、审计、指标,以及依附于各自 D1 事务的占用(邀请、会话、Token、机器人交互、分组 id、安装绑定)——在两种模式下都留在 D1。无法识别的取值会被记录并当作 `d1` 处理。
- R2 免费额度(10 GB-月存储、每月 100 万次 A 类操作)可以轻松承载负载暂存,不占用 KV 写配额。