mirror of
https://github.com/ReCloudStudio/WebHooker.git
synced 2026-09-22 16:11:29 +00:00
feat(storage): migrate config, dedup and delivery state to D1
- Move oversized queue payloads from KV to R2 (PAYLOAD binding, webhooks/YYYY/MM/DD/*.json, KV queue:payload:* fallback)
- Persist routes/groups to D1 (d1_routes/d1_groups) with memory -> KV -> D1 three-tier cache, seeded from legacy KV config keys
- Move webhook dedup (dedup_keys), delivery state (delivery_state) and message tracking (message_tracking) to D1 via canUseD1 probe with automatic KV fallback
- Batch send_logs inserts (recordSendBatch) and add group_id/ts index
- Add storage-prune scheduled task for expired dedup/state/tracking rows
- Add TTL to invite:group:{id} index and audit all ephemeral KV keys
- Add D1 indexes for the new tables
- Sync AGENTS.md, README.md/zh and docs/ (en/zh) with the new storage layout
This commit is contained in:
parent
2e1b0f022e
commit
25ebae4ae5
46 changed files with 1450 additions and 117 deletions
|
|
@ -34,7 +34,7 @@ The console itself is served at `/admin`; its tabs are deep-linkable via the URL
|
|||
|
||||
## Validation
|
||||
|
||||
- `PUT /admin/api/routes` — Body `{ "routes": Route[] }`; validates each route (id pattern, unique id within its group, name, enabled, `groupId`, filters — empty only allowed for `fallback` routes — optional `discordRoleIds` (list of role id strings), and platform-aware targets: `target.channelId` for Discord, `target.chatId` for Telegram) and persists to KV `config:routes`. Returns `200 { ok, count }` or `400 { error }` / `401 { error }` / `403 { error }`. Unchanged routes skip the full validation.
|
||||
- `PUT /admin/api/routes` — Body `{ "routes": Route[] }`; validates each route (id pattern, unique id within its group, name, enabled, `groupId`, filters — empty only allowed for `fallback` routes — optional `discordRoleIds` (list of role id strings), and platform-aware targets: `target.channelId` for Discord, `target.chatId` for Telegram) and persists to D1 `d1_routes`. Returns `200 { ok, count }` or `400 { error }` / `401 { error }` / `403 { error }`. Unchanged routes skip the full validation.
|
||||
- `PUT /admin/api/groups` — Validates group ids, member roles (at least one `owner`), `providers` (`github` / `gitea`), and `installationId`.
|
||||
- Limits: at most 200 routes and 100 groups per instance.
|
||||
|
||||
|
|
|
|||
|
|
@ -60,7 +60,7 @@ WebHooker ships with a built-in config console at `/admin` for managing routes,
|
|||
|
||||
The console is served as an SPA at `/admin`; its tabs are deep-linkable via the URL path (`/admin/groups`, `/admin/logs`, `/admin/audit`). URLs outside `/admin` that do not match an endpoint return a plain `404` instead of the console.
|
||||
|
||||
All management endpoints (`/admin/api/*`) are documented in the [Admin API](../api/admin). Saved routes are written to KV `config:routes` immediately and the config cache is invalidated so the webhook pipeline picks them up on the next run.
|
||||
All management endpoints (`/admin/api/*`) are documented in the [Admin API](../api/admin). Saved routes and groups are persisted to D1 (`d1_routes` / `d1_groups`) immediately, the KV cache is invalidated and the config cache is refreshed so the webhook pipeline picks them up on the next run.
|
||||
|
||||
## Filter Types
|
||||
|
||||
|
|
|
|||
|
|
@ -81,11 +81,24 @@ bunx wrangler d1 execute webhooker --remote --file ./migrations/0002_log_detail.
|
|||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0003_telegram_links.sql
|
||||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0004_add_group_id.sql
|
||||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0005_audit_logs.sql
|
||||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0006_config_d1.sql
|
||||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0007_send_logs_index.sql
|
||||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0008_storage_d1.sql
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
### 4. Create Queues (Optional)
|
||||
### 4. Create R2 Bucket (Optional)
|
||||
|
||||
The `PAYLOAD` binding parks oversized webhook payloads in R2 (`webhooker-payloads`) instead of KV. Without it, oversized payloads fall back to the KV key `queue:payload:*`. See [Storage](/guide/storage) for the layout.
|
||||
|
||||
```bash
|
||||
bunx wrangler r2 bucket create webhooker-payloads
|
||||
```
|
||||
|
||||
The bucket is already declared in `wrangler.jsonc` (`r2_buckets`), so no binding change is needed.
|
||||
|
||||
### 5. Create Queues (Optional)
|
||||
|
||||
The `QUEUE` binding routes webhook delivery through Cloudflare Queues (async dispatch with retry backoff and a dead-letter queue). Skip this step to keep dispatch inline (synchronous).
|
||||
|
||||
|
|
@ -96,7 +109,7 @@ bunx wrangler queues create webhooker-delivery-dlq
|
|||
|
||||
The queues are already declared in `wrangler.jsonc` (`queues.producers` / `queues.consumers`), so no binding change is needed. The `webhooker-delivery` consumer retries retryable failures with exponential backoff (5s/30s/2m/10m) up to `max_retries`, after which the message is moved to `webhooker-delivery-dlq` and marked dead.
|
||||
|
||||
### 5. Deploy
|
||||
### 6. Deploy
|
||||
|
||||
```bash
|
||||
bunx wrangler deploy
|
||||
|
|
@ -104,13 +117,13 @@ bunx wrangler deploy
|
|||
|
||||
Your worker is now live at `https://webhooker.<your-subdomain>.workers.dev`.
|
||||
|
||||
### 6. Configure GitHub Webhook
|
||||
### 7. Configure GitHub Webhook
|
||||
|
||||
1. Go to your GitHub App settings
|
||||
2. Set **Webhook URL** to `https://webhooker.<your-subdomain>.workers.dev/webhook`
|
||||
3. Set **Webhook secret** to match `GITHUB_WEBHOOK_SECRET`
|
||||
|
||||
### 7. (Optional) Configure Gitea Webhook
|
||||
### 8. (Optional) Configure Gitea Webhook
|
||||
|
||||
1. In your Gitea repo, go to **Settings → Webhooks → Add Webhook → Gitea**
|
||||
2. Set **Target URL** to `https://webhooker.<your-subdomain>.workers.dev/webhook`
|
||||
|
|
|
|||
|
|
@ -34,4 +34,4 @@ No — the worker requires the KV and D1 bindings declared in `wrangler.jsonc` a
|
|||
|
||||
## Where is data stored?
|
||||
|
||||
Configuration lives in Cloudflare KV (`config:routes`, `config:groups`); send/audit logs and platform↔GitHub links live in D1. See [Storage Layout](./storage#kv-storage-layout).
|
||||
Configuration lives in D1 (`d1_routes`/`d1_groups`, with KV as a cache); webhook dedup, delivery state and message-update tracking also live in D1 with KV fallback; send/audit logs and platform↔GitHub links live in D1. R2 optionally parks oversized payloads. See [Storage Layout](./storage#storage-decisions).
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Groups & Access Control
|
||||
|
||||
Routes belong to groups. Groups scope admin access and can restrict which events flow into them. They are stored in Cloudflare KV under the key `config:groups` as a JSON array, managed via the [Web UI](./configuration#web-ui) or the [Admin API](../api/admin). At most **100 groups** can be saved per instance.
|
||||
Routes belong to groups. Groups scope admin access and can restrict which events flow into them. They are stored in D1 (`d1_groups`, seeded from the legacy KV `config:groups` key on first load), managed via the [Web UI](./configuration#web-ui) or the [Admin API](../api/admin). At most **100 groups** can be saved per instance.
|
||||
|
||||
## Group Schema
|
||||
|
||||
|
|
|
|||
|
|
@ -22,14 +22,14 @@ GitHub / Gitea Webhook → Cloudflare Worker (Nuxt 4 / Nitro)
|
|||
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| **Cloudflare Worker** | HTTP ingress, signature verification, delivery dedup, event parsing, route matching, platform dispatch |
|
||||
| **Interactions Endpoint** | Verifies Ed25519 signatures and handles `/gh` interactions (slash commands, context-menu commands, buttons, modals) |
|
||||
| **KV** | Token storage (`token:{userId}`), OAuth state (`state:{hex}`), route config (`config:routes`), group config (`config:groups`), admin sessions, delivery dedup, message-update tracking (`msg:*`) |
|
||||
| **D1** | Send logs (`send_logs`), Discord↔GitHub links (`discord_links`), Telegram↔GitHub links (`telegram_links`) |
|
||||
| **KV** | Token storage (`token:{userId}`), OAuth state (`state:{hex}`), admin sessions, per-group secrets, config cache, delivery dedup/state/message-tracking fallback when D1 is unavailable (`delivery:*`, `delivery-state:*`, `msg:*`), message-update locks (`msg:lock:*`) |
|
||||
| **D1** | Routes/groups (`d1_routes`/`d1_groups`), send logs (`send_logs`), audit logs (`audit_logs`), dedup (`dedup_keys`), delivery state (`delivery_state`), message tracking (`message_tracking`), Discord↔GitHub links (`discord_links`), Telegram↔GitHub links (`telegram_links`), with R2 optionally parking oversized payloads |
|
||||
|
||||
### Data Flow
|
||||
|
||||
1. A forge (GitHub or Gitea) sends a webhook to `POST /webhook`
|
||||
2. Worker detects the provider from its headers (`X-GitHub-Event` / `X-Gitea-Event`) and verifies the provider-specific HMAC-SHA256 signature
|
||||
3. Worker deduplicates by the delivery id (KV, short TTL) to drop repeat deliveries
|
||||
3. Worker deduplicates by the delivery id (D1, short TTL, KV fallback) to drop repeat deliveries
|
||||
4. Worker parses the event type and normalizes the payload to a GitHub-shaped event
|
||||
5. Routes are evaluated against filters (event, repo, actor, action, branch, keyword) and group owner restrictions
|
||||
6. Matching routes trigger formatter functions that produce platform-neutral messages
|
||||
|
|
@ -42,7 +42,7 @@ GitHub / Gitea Webhook → Cloudflare Worker (Nuxt 4 / Nitro)
|
|||
- **Discord delivery**: Discord REST API (interactions via an Ed25519-verified HTTPS Interactions Endpoint)
|
||||
- **Telegram delivery**: Telegram Bot API (webhook with optional secret-token verification)
|
||||
- **Web UI**: Nuxt 4 (Vue 3 + Tailwind CSS v3) — server-rendered home/legal pages, client-side `/admin` console
|
||||
- **Storage**: Cloudflare KV + D1
|
||||
- **Storage**: Cloudflare D1 (authoritative for config & logs) + KV (cache/transient state) + optional R2 (oversized payloads)
|
||||
- **Auth**: Web Crypto API (HMAC-SHA256, Ed25519), octokit (GitHub API)
|
||||
- **Language**: TypeScript
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Routes & Targets
|
||||
|
||||
Routes define which events get forwarded to which channel (Discord or Telegram). They are stored in Cloudflare KV under the key `config:routes` as a JSON array, managed via the [Web UI](./configuration#web-ui), the [Admin API](../api/admin), or `config.example.yaml`.
|
||||
Routes define which events get forwarded to which channel (Discord or Telegram). They are stored in D1 (`d1_routes`, seeded from the legacy KV `config:routes` key on first load), managed via the [Web UI](./configuration#web-ui), the [Admin API](../api/admin), or `config.example.yaml`.
|
||||
|
||||
There are **no default routes** — each route must define its own target. If no routes are configured, no events are forwarded. At most **200 routes** can be saved per instance.
|
||||
|
||||
|
|
|
|||
|
|
@ -2,22 +2,24 @@
|
|||
|
||||
## KV Storage Layout
|
||||
|
||||
KV keeps only cache data and short-lived/ephemeral state. High-frequency writes (webhook dedup, delivery state, message tracking) live in D1 and only fall back to KV when D1 is unavailable or not yet migrated (see [Storage decisions](./storage#storage-decisions)).
|
||||
|
||||
| Key Pattern | Value | TTL |
|
||||
| ------------------------------------------ | ------------------------------------------------------------------------------------ | ------------------ |
|
||||
| `config:routes` | JSON array of routes | Permanent |
|
||||
| `config:groups` | JSON array of groups | Permanent |
|
||||
| `config:routes` | Route config cache (D1 `d1_routes` is authoritative) | 1 hour |
|
||||
| `config:groups` | Group config cache (D1 `d1_groups` is authoritative) | 1 hour |
|
||||
| `session:{id}` | Admin session `{ userId, login }` | 7 days |
|
||||
| `token:{userId}` | `{ userId, accessToken, expiresAt, refreshToken? }` | 0.9 × token expiry |
|
||||
| `token-reverse:{sha256}` | User id for reverse lookup by token | 0.9 × token expiry |
|
||||
| `state:{hex}` | `{ redirectTo, expiresAt, discordUserId?, telegramUserId?, telegramChatId? }` | 600 seconds |
|
||||
| `invite:{token}` | `{ groupId, role, expiresAt, createdBy, note? }` | 7 days |
|
||||
| `invite:group:{id}` | Token index per group (keeps invite listing consistent) | Permanent |
|
||||
| `delivery:{provider}:{groupId}:{id}` | Webhook delivery dedup (provider- and tenant-scoped) | 300 seconds |
|
||||
| `delivery-state:{provider}:{groupId}:{id}` | Queue delivery state (`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`) | 1 day |
|
||||
| `queue:payload:{provider}:{groupId}:{id}` | Oversized webhook payload parked for the queue consumer | 1 day |
|
||||
| `invite:group:{id}` | Token index per group (keeps invite listing consistent) | 7 days |
|
||||
| `delivery:{provider}:{groupId}:{id}` | Webhook delivery dedup fallback (D1 `dedup_keys` is primary) | 7 days |
|
||||
| `delivery-state:{provider}:{groupId}:{id}` | Queue delivery state fallback (D1 `delivery_state` is primary) | 1 hour |
|
||||
| `queue:payload:{provider}:{groupId}:{id}` | Oversized webhook payload parked for the queue consumer (R2 is primary) | 1 hour |
|
||||
| `nonce:{nonce}` | Custom-webhook replay protection nonce (single use) | 600 seconds |
|
||||
| `tenant:{groupId}` | Per-group webhook secret (64-char hex, generated from the console) | Permanent |
|
||||
| `msg:{routeId}:{key}:{target}` | Message id tracking for in-place updates (e.g. `workflow_run` / `check_run`) | 7 days |
|
||||
| `msg:{routeId}:{key}:{target}` | Message id tracking fallback (D1 `message_tracking` is primary) | 1 day |
|
||||
| `cmd:guild:{id}` | Guild id whose commands were registered (dedup) | Permanent |
|
||||
| `cmd:registered:global` | Global command registration marker (dedup) | 1 day |
|
||||
| `config:discord-app-id` | Cached Discord application id | Permanent |
|
||||
|
|
@ -25,13 +27,35 @@
|
|||
|
||||
## D1 Storage Layout
|
||||
|
||||
The D1 database (`DB` binding, database `webhooker`) holds four tables:
|
||||
The D1 database (`DB` binding, database `webhooker`) holds the source of truth for configuration, delivery logs and high-frequency ephemeral state:
|
||||
|
||||
| Table | Purpose |
|
||||
| ---------------- | ---------------------------------------------------------------------------------------------- |
|
||||
| `send_logs` | One row per dispatch attempt (route id, event, target, ok/error, duration, error code, detail) |
|
||||
| `audit_logs` | One row per admin operation (login/logout, group/route/member/invite changes) |
|
||||
| `discord_links` | Maps `discord_user_id` → `github_user_id` for `/gh` Discord commands |
|
||||
| `telegram_links` | Maps `telegram_user_id` → `github_user_id` for `/gh` Telegram commands |
|
||||
| Table | Purpose |
|
||||
| ------------------ | ---------------------------------------------------------------------------------------------------- |
|
||||
| `d1_groups` | Groups (authoritative config, seeded from legacy KV `config:groups`) |
|
||||
| `d1_routes` | Routes per group (authoritative config, seeded from legacy KV `config:routes`) |
|
||||
| `send_logs` | One row per dispatch attempt (route id, event, target, ok/error, duration, error code, detail) |
|
||||
| `audit_logs` | One row per admin operation (login/logout, group/route/member/invite changes) |
|
||||
| `dedup_keys` | Webhook delivery dedup (atomic `INSERT ... ON CONFLICT` UPSERT, key + expiry) |
|
||||
| `delivery_state` | Queue delivery state (`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`) |
|
||||
| `message_tracking` | Message id tracking for in-place updates (`event_id` + `target_id` → `message_id`) |
|
||||
| `discord_links` | Maps `discord_user_id` → `github_user_id` for `/gh` Discord commands |
|
||||
| `telegram_links` | Maps `telegram_user_id` → `github_user_id` for `/gh` Telegram commands |
|
||||
|
||||
`audit_logs` is pruned automatically by the scheduled trigger after `AUDIT_RETENTION_DAYS` (default 90). See [Logs](./logs) for the row fields.
|
||||
`audit_logs` is pruned by the scheduled trigger after `AUDIT_RETENTION_DAYS` (default 90). The `storage-prune` task removes expired `dedup_keys`, `delivery_state` rows older than 7 days and `message_tracking` rows older than 30 days. See [Logs](./logs) for the log row fields.
|
||||
|
||||
## R2 Storage Layout
|
||||
|
||||
R2 (`PAYLOAD` binding, bucket `webhooker-payloads`) stores oversized webhook payloads that are too large for a queue message or KV:
|
||||
|
||||
| Object Pattern | Purpose | Retention |
|
||||
| ----------------------------- | ---------------------------------------------- | --------- |
|
||||
| `webhooks/YYYY/MM/DD/<uuid>.json` | Oversized payload parked for the queue consumer | deleted after dispatch |
|
||||
|
||||
When the `PAYLOAD` binding is absent, oversized payloads fall back to the KV key `queue:payload:{provider}:{groupId}:{id}` (1 hour TTL).
|
||||
|
||||
## Storage Decisions
|
||||
|
||||
- **D1 is authoritative** for config and delivery metadata; KV holds only caches and short-lived state.
|
||||
- The `canUseD1` probe (`server/lib/storage/d1.ts`, checks for `prepare` + `batch`) gates every D1 store: when D1 is unavailable or not yet migrated, all three high-frequency stores (dedup, delivery state, message tracking) transparently fall back to KV so behavior is unchanged during migration.
|
||||
- This keeps per-event KV writes near zero on the Workers Free plan (1,000 writes/day): dedup, delivery state and message tracking now write D1 rows instead (D1 Free allows 100,000 rows written/day).
|
||||
- R2's free tier (10 GB-month storage, 1M Class A ops/month) comfortably absorbs payload parking without touching the KV write quota.
|
||||
|
|
|
|||
|
|
@ -1,11 +1,12 @@
|
|||
# Scheduled Tasks
|
||||
|
||||
WebHooker runs three maintenance tasks on the scheduled trigger (`*/5 * * * *`, every 5 minutes). They only run on the deployed worker (Cloudflare cron); local `wrangler dev` runs them when triggered via `wrangler dev --test-scheduled`.
|
||||
WebHooker runs four maintenance tasks on the scheduled trigger (`*/5 * * * *`, every 5 minutes). They only run on the deployed worker (Cloudflare cron); local `wrangler dev` runs them when triggered via `wrangler dev --test-scheduled`.
|
||||
|
||||
| Task | Purpose |
|
||||
| --------------- | ---------------------------------------------------------------------------------------------------------------- |
|
||||
| `discord-sync` | Registers the Discord slash/context-menu commands: per-guild (instant) and globally (24h dedup, ~1h propagation) |
|
||||
| `telegram-sync` | Calls `setWebhook` to `{BASE_URL}/telegram/webhook` (with `TELEGRAM_WEBHOOK_SECRET` as `secret_token` when set) |
|
||||
| `audit-prune` | Deletes `audit_logs` entries older than `AUDIT_RETENTION_DAYS` (default 90) |
|
||||
| `storage-prune` | Deletes expired `dedup_keys` rows, `delivery_state` rows older than 7 days and `message_tracking` rows older than 30 days |
|
||||
|
||||
There is nothing to configure beyond the secrets the tasks use (`DISCORD_TOKEN`, `DISCORD_APPLICATION_ID`, `TELEGRAM_TOKEN`, `BASE_URL`, `AUDIT_RETENTION_DAYS`).
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@
|
|||
|
||||
## 校验
|
||||
|
||||
- `PUT /admin/api/routes` — 请求体为 `{ "routes": Route[] }`；校验每条路由（id 格式、组内唯一 id、name、enabled、groupId、过滤器——**仅 `fallback` 路由允许空过滤器**——可选的 `discordRoleIds`（身份组 id 字符串列表）、平台感知的 targets：Discord 需 `target.channelId`，Telegram 需 `target.chatId`）并持久化到 KV `config:routes`。返回 `200 { ok, count }` 或 `400 { error }` / `401 { error }` / `403 { error }`。未变更的路由跳过完整校验。
|
||||
- `PUT /admin/api/routes` — 请求体为 `{ "routes": Route[] }`；校验每条路由（id 格式、组内唯一 id、name、enabled、groupId、过滤器——**仅 `fallback` 路由允许空过滤器**——可选的 `discordRoleIds`（身份组 id 字符串列表）、平台感知的 targets：Discord 需 `target.channelId`，Telegram 需 `target.chatId`）并持久化到 D1 `d1_routes`。返回 `200 { ok, count }` 或 `400 { error }` / `401 { error }` / `403 { error }`。未变更的路由跳过完整校验。
|
||||
- `PUT /admin/api/groups` — 校验分组 id、成员角色（至少一个 `owner`）、`providers`（`github` / `gitea`）与 `installationId`。
|
||||
- 上限：每个实例最多 200 条路由与 100 个分组。
|
||||
|
||||
|
|
|
|||
|
|
@ -59,7 +59,7 @@ WebHooker 在 `/admin` 提供内置配置控制台，可在浏览器中管理路
|
|||
|
||||
控制台以 SPA 形式在 `/admin` 提供；其标签页可通过 URL 路径直达（`/admin/groups`、`/admin/logs`、`/admin/audit`）。`/admin` 之外未匹配到端点的 URL 直接返回 `404`，而不会展示控制台。
|
||||
|
||||
所有管理端点（`/admin/api/*`）见 [Admin API](../api/admin)。保存的路由会立即写入 KV `config:routes` 并使配置缓存失效，下一次 webhook 处理即会生效。
|
||||
所有管理端点（`/admin/api/*`）见 [Admin API](../api/admin)。保存的路由和分组会立即持久化到 D1（`d1_routes` / `d1_groups`），并使 KV 缓存失效、刷新配置缓存，下一次 webhook 处理即会生效。
|
||||
|
||||
## 过滤器类型
|
||||
|
||||
|
|
|
|||
|
|
@ -81,11 +81,24 @@ bunx wrangler d1 execute webhooker --remote --file ./migrations/0002_log_detail.
|
|||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0003_telegram_links.sql
|
||||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0004_add_group_id.sql
|
||||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0005_audit_logs.sql
|
||||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0006_config_d1.sql
|
||||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0007_send_logs_index.sql
|
||||
bunx wrangler d1 execute webhooker --remote --file ./migrations/0008_storage_d1.sql
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
### 4. 创建队列（可选）
|
||||
### 4. 创建 R2 Bucket（可选）
|
||||
|
||||
`PAYLOAD` 绑定将超大 webhook 负载暂存在 R2（`webhooker-payloads`）而非 KV。未设置时，超大负载回退到 KV 键 `queue:payload:*`。布局见[存储](/zh/guide/storage)。
|
||||
|
||||
```bash
|
||||
bunx wrangler r2 bucket create webhooker-payloads
|
||||
```
|
||||
|
||||
bucket 已在 `wrangler.jsonc` 中声明（`r2_buckets`），无需修改绑定。
|
||||
|
||||
### 5. 创建队列（可选）
|
||||
|
||||
`QUEUE` 绑定会通过 Cloudflare Queues 投递 webhook（异步分发，带重试退避与死信队列）。跳过此步则保持同步内联分发。
|
||||
|
||||
|
|
@ -96,7 +109,7 @@ bunx wrangler queues create webhooker-delivery-dlq
|
|||
|
||||
队列已在 `wrangler.jsonc` 中声明（`queues.producers` / `queues.consumers`），无需修改绑定。`webhooker-delivery` 消费者对可重试失败做指数退避重试（5s/30s/2m/10m），达到 `max_retries` 后消息进入 `webhooker-delivery-dlq` 并标记为 dead。
|
||||
|
||||
### 5. 部署
|
||||
### 6. 部署
|
||||
|
||||
```bash
|
||||
bunx wrangler deploy
|
||||
|
|
@ -104,13 +117,13 @@ bunx wrangler deploy
|
|||
|
||||
Worker 现在可通过 `https://webhooker.<your-subdomain>.workers.dev` 访问。
|
||||
|
||||
### 6. 配置 GitHub Webhook
|
||||
### 7. 配置 GitHub Webhook
|
||||
|
||||
1. 进入 GitHub App 设置页面
|
||||
2. 设置 **Webhook URL** 为 `https://webhooker.<your-subdomain>.workers.dev/webhook`
|
||||
3. 设置 **Webhook secret** 与 `GITHUB_WEBHOOK_SECRET` 一致
|
||||
|
||||
### 7.（可选）配置 Gitea Webhook
|
||||
### 8.（可选）配置 Gitea Webhook
|
||||
|
||||
1. 在 Gitea 仓库中进入 **设置 → Web 钩子 → 添加 Web 钩子 → Gitea**
|
||||
2. 设置 **目标 URL** 为 `https://webhooker.<your-subdomain>.workers.dev/webhook`
|
||||
|
|
|
|||
|
|
@ -34,4 +34,4 @@
|
|||
|
||||
## 数据存储在哪里？
|
||||
|
||||
配置存于 Cloudflare KV（`config:routes`、`config:groups`）；发送/审计日志与平台↔GitHub 绑定存于 D1。见[存储布局](./storage#kv-存储布局)。
|
||||
配置存于 D1（`d1_routes`/`d1_groups`，KV 仅作缓存）；webhook 去重、投递状态与消息更新追踪同样存于 D1（KV 回退）；发送/审计日志与平台↔GitHub 绑定存于 D1；超大负载可选存于 R2。见[存储布局](./storage#存储决策)。
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# 分组与访问控制
|
||||
|
||||
路由归属于分组。分组用于划分管理权限，并可限制进入其中的事件。它们以 JSON 数组形式存储在 Cloudflare KV 的 `config:groups` 键下，可通过 [Web 控制台](./configuration#web-控制台)或 [Admin API](../api/admin) 管理。每个实例最多可保存 **100 个分组**。
|
||||
路由归属于分组。分组用于划分管理权限，并可限制进入其中的事件。它们存储在 D1（`d1_groups`，首次加载时从旧版 KV `config:groups` 键同步），可通过 [Web 控制台](./configuration#web-控制台)或 [Admin API](../api/admin) 管理。每个实例最多可保存 **100 个分组**。
|
||||
|
||||
## 分组模式
|
||||
|
||||
|
|
|
|||
|
|
@ -22,14 +22,14 @@ GitHub / Gitea Webhook → Cloudflare Worker (Nuxt 4 / Nitro)
|
|||
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Cloudflare Worker** | HTTP 入口、签名验证、投递去重、事件解析、路由匹配、平台分发 |
|
||||
| **Interactions Endpoint** | 验证 Ed25519 签名并处理 `/gh` 交互（斜杠命令、右键菜单、按钮、modal） |
|
||||
| **KV** | Token 存储 (`token:{userId}`)、OAuth 状态 (`state:{hex}`)、路由配置 (`config:routes`)、分组配置 (`config:groups`)、管理员会话、投递去重、消息更新追踪 (`msg:*`) |
|
||||
| **D1** | 发送日志 (`send_logs`)、Discord↔GitHub 绑定 (`discord_links`)、Telegram↔GitHub 绑定 (`telegram_links`) |
|
||||
| **KV** | Token 存储 (`token:{userId}`)、OAuth 状态 (`state:{hex}`)、管理员会话、分组级 secret、配置缓存、投递去重/投递状态/消息更新追踪的回退（`delivery:*`、`delivery-state:*`、`msg:*`，仅在 D1 不可用时使用）、消息更新锁 (`msg:lock:*`) |
|
||||
| **D1** | 路由/分组 (`d1_routes`/`d1_groups`)、发送日志 (`send_logs`)、审计日志 (`audit_logs`)、去重 (`dedup_keys`)、投递状态 (`delivery_state`)、消息追踪 (`message_tracking`)、Discord↔GitHub 绑定 (`discord_links`)、Telegram↔GitHub 绑定 (`telegram_links`)，超大负载可选的 R2 存储 |
|
||||
|
||||
### 数据流
|
||||
|
||||
1. 某个 forge（GitHub 或 Gitea）发送 webhook 到 `POST /webhook`
|
||||
2. Worker 根据请求头识别提供方（`X-GitHub-Event` / `X-Gitea-Event`）并验证对应提供的 HMAC-SHA256 签名
|
||||
3. Worker 按投递 ID 去重（KV，短 TTL），丢弃重复投递
|
||||
3. Worker 按投递 ID 去重（D1，短 TTL，KV 回退），丢弃重复投递
|
||||
4. Worker 解析事件类型并将载荷归一化为 GitHub 形状的事件
|
||||
5. 根据过滤器（event、repo、actor、action、branch、keyword）与分组所有者限制评估路由
|
||||
6. 匹配的路由触发格式化器函数生成平台中立消息
|
||||
|
|
@ -42,7 +42,7 @@ GitHub / Gitea Webhook → Cloudflare Worker (Nuxt 4 / Nitro)
|
|||
- **Discord 投递**: Discord REST API（交互通过 Ed25519 验签的 HTTPS Interactions Endpoint）
|
||||
- **Telegram 投递**: Telegram Bot API（webhook 带可选 secret-token 校验）
|
||||
- **Web UI**: Nuxt 4（Vue 3 + Tailwind CSS v3）——首页/法律页面服务端渲染，`/admin` 控制台客户端渲染
|
||||
- **存储**: Cloudflare KV + D1
|
||||
- **存储**: Cloudflare D1（配置与日志权威）+ KV（缓存/临时状态）+ 可选 R2（超大负载）
|
||||
- **鉴权**: Web Crypto API (HMAC-SHA256、Ed25519)、octokit (GitHub API)
|
||||
- **语言**: TypeScript
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# 路由与目标
|
||||
|
||||
路由决定哪些事件被转发到哪个频道（Discord 或 Telegram）。它们以 JSON 数组形式存储在 Cloudflare KV 的 `config:routes` 键下，可通过 [Web 控制台](./configuration#web-控制台)、[Admin API](../api/admin) 或 `config.example.yaml` 管理。
|
||||
路由决定哪些事件被转发到哪个频道（Discord 或 Telegram）。它们存储在 D1（`d1_routes`，首次加载时从旧版 KV `config:routes` 键同步），可通过 [Web 控制台](./configuration#web-控制台)、[Admin API](../api/admin) 或 `config.example.yaml` 管理。
|
||||
|
||||
**没有默认路由**——每条路由都必须定义自己的目标。未配置任何路由时不会转发任何事件。每个实例最多可保存 **200 条路由**。
|
||||
|
||||
|
|
|
|||
|
|
@ -2,22 +2,24 @@
|
|||
|
||||
## KV 存储布局
|
||||
|
||||
KV 只保留缓存数据和短期/临时状态。高频写入（webhook 去重、投递状态、消息追踪）存放在 D1，仅在 D1 不可用或未迁移时才回退到 KV（见[存储决策](#存储决策)）。
|
||||
|
||||
| 键模式 | 值 | TTL |
|
||||
| ------------------------------------------ | ----------------------------------------------------------------------------- | ------------------ |
|
||||
| `config:routes` | 路由 JSON 数组 | 永久 |
|
||||
| `config:groups` | 分组 JSON 数组 | 永久 |
|
||||
| `config:routes` | 路由配置缓存（D1 `d1_routes` 为权威数据源） | 1 小时 |
|
||||
| `config:groups` | 分组配置缓存（D1 `d1_groups` 为权威数据源） | 1 小时 |
|
||||
| `session:{id}` | 管理员会话 `{ userId, login }` | 7 天 |
|
||||
| `token:{userId}` | `{ userId, accessToken, expiresAt, refreshToken? }` | 0.9 × Token 有效期 |
|
||||
| `token-reverse:{sha256}` | 用于按 Token 反查的用户 id | 0.9 × Token 有效期 |
|
||||
| `state:{hex}` | `{ redirectTo, expiresAt, discordUserId?, telegramUserId?, telegramChatId? }` | 600 秒 |
|
||||
| `invite:{token}` | `{ groupId, role, expiresAt, createdBy, note? }` | 7 天 |
|
||||
| `invite:group:{id}` | 每组的 Token 索引（保证邀请列表一致性） | 永久 |
|
||||
| `delivery:{provider}:{groupId}:{id}` | Webhook 投递去重（按 provider 与租户隔离） | 300 秒 |
|
||||
| `delivery-state:{provider}:{groupId}:{id}` | 队列投递状态（`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`） | 1 天 |
|
||||
| `queue:payload:{provider}:{groupId}:{id}` | 暂存供队列消费者读取的超大 webhook 负载 | 1 天 |
|
||||
| `invite:group:{id}` | 每组的 Token 索引（保证邀请列表一致性） | 7 天 |
|
||||
| `delivery:{provider}:{groupId}:{id}` | Webhook 投递去重回退（D1 `dedup_keys` 为主） | 7 天 |
|
||||
| `delivery-state:{provider}:{groupId}:{id}` | 队列投递状态回退（D1 `delivery_state` 为主） | 1 小时 |
|
||||
| `queue:payload:{provider}:{groupId}:{id}` | 暂存供队列消费者读取的超大 webhook 负载（R2 为主） | 1 小时 |
|
||||
| `nonce:{nonce}` | 自定义 webhook 重放防护 nonce（一次性） | 600 秒 |
|
||||
| `tenant:{groupId}` | 分组 webhook secret（64 位 hex，控制台生成） | 永久 |
|
||||
| `msg:{routeId}:{key}:{target}` | 原地更新用消息 id 追踪（如 `workflow_run` / `check_run`） | 7 天 |
|
||||
| `msg:{routeId}:{key}:{target}` | 消息 id 追踪回退（D1 `message_tracking` 为主） | 1 天 |
|
||||
| `cmd:guild:{id}` | 已注册命令的服务器 id（去重） | 永久 |
|
||||
| `cmd:registered:global` | 全局命令注册标记（去重） | 1 天 |
|
||||
| `config:discord-app-id` | 缓存的 Discord 应用 id | 永久 |
|
||||
|
|
@ -25,13 +27,35 @@
|
|||
|
||||
## D1 存储布局
|
||||
|
||||
D1 数据库（`DB` 绑定，数据库 `webhooker`）包含四张表：
|
||||
D1 数据库（`DB` 绑定，数据库 `webhooker`）保存配置、投递日志和高频临时状态的权威数据源：
|
||||
|
||||
| 表 | 用途 |
|
||||
| ---------------- | ---------------------------------------------------------------------- |
|
||||
| `send_logs` | 每次分发尝试一行（路由 id、事件、目标、ok/error、耗时、错误码、详情） |
|
||||
| `audit_logs` | 每次管理员操作一行（登录/登出、分组/路由/成员/邀请变更） |
|
||||
| `discord_links` | 映射 `discord_user_id` → `github_user_id`，供 `/gh` Discord 命令使用 |
|
||||
| `telegram_links` | 映射 `telegram_user_id` → `github_user_id`，供 `/gh` Telegram 命令使用 |
|
||||
| 表 | 用途 |
|
||||
| ------------------ | ---------------------------------------------------------------------- |
|
||||
| `d1_groups` | 分组（权威配置，从旧版 KV `config:groups` 播种） |
|
||||
| `d1_routes` | 每组分组的路由（权威配置，从旧版 KV `config:routes` 播种） |
|
||||
| `send_logs` | 每次分发尝试一行（路由 id、事件、目标、ok/error、耗时、错误码、详情） |
|
||||
| `audit_logs` | 每次管理员操作一行（登录/登出、分组/路由/成员/邀请变更） |
|
||||
| `dedup_keys` | Webhook 投递去重（原子 `INSERT ... ON CONFLICT` UPSERT，键 + 过期时间） |
|
||||
| `delivery_state` | 队列投递状态（`pending`/`processing`/`delivered`/`retrying`/`failed`/`dead`） |
|
||||
| `message_tracking` | 原地更新用消息 id 追踪（`event_id` + `target_id` → `message_id`） |
|
||||
| `discord_links` | 映射 `discord_user_id` → `github_user_id`，供 `/gh` Discord 命令使用 |
|
||||
| `telegram_links` | 映射 `telegram_user_id` → `github_user_id`，供 `/gh` Telegram 命令使用 |
|
||||
|
||||
`audit_logs` 由定时任务在 `AUDIT_RETENTION_DAYS`（默认 90）后自动清理。行字段说明见[日志](./logs)。
|
||||
`audit_logs` 由定时任务在 `AUDIT_RETENTION_DAYS`（默认 90）后自动清理。`storage-prune` 任务会清理过期的 `dedup_keys`、超过 7 天的 `delivery_state` 行以及超过 30 天的 `message_tracking` 行。日志行字段说明见[日志](./logs)。
|
||||
|
||||
## R2 存储布局
|
||||
|
||||
R2（`PAYLOAD` 绑定，bucket `webhooker-payloads`）存储对队列消息或 KV 来说过大的 webhook 负载：
|
||||
|
||||
| 对象模式 | 用途 | 保留期 |
|
||||
| ------------------------------- | ----------------------------------------------- | ------------ |
|
||||
| `webhooks/YYYY/MM/DD/<uuid>.json` | 暂存供队列消费者读取的超大负载 | 分发后删除 |
|
||||
|
||||
当缺少 `PAYLOAD` 绑定时，超大负载回退到 KV 键 `queue:payload:{provider}:{groupId}:{id}`（1 小时 TTL）。
|
||||
|
||||
## 存储决策
|
||||
|
||||
- **D1 是配置与投递元数据的权威数据源**；KV 只保存缓存和短期状态。
|
||||
- `canUseD1` 探测（`server/lib/storage/d1.ts`，检查 `prepare` + `batch`）为每个 D1 存储做门槛判定：当 D1 不可用或尚未迁移时，三种高频存储（去重、投递状态、消息追踪）都会透明回退到 KV，迁移期间行为不变。
|
||||
- 这让 Workers 免费版的每事件 KV 写入趋近于零（每日 1000 次写）：去重、投递状态和消息追踪改为写 D1 行（D1 免费版每日可写 10 万行）。
|
||||
- R2 免费额度（10 GB-月存储、每月 100 万次 A 类操作）可以轻松承载负载暂存，不占用 KV 写配额。
|
||||
|
|
|
|||
|
|
@ -1,11 +1,12 @@
|
|||
# 定时任务
|
||||
|
||||
WebHooker 通过定时触发器（`*/5 * * * *`，每 5 分钟）运行三个维护任务。它们只在部署后的 Worker 上运行（Cloudflare cron）；本地 `wrangler dev` 可用 `wrangler dev --test-scheduled` 触发。
|
||||
WebHooker 通过定时触发器（`*/5 * * * *`，每 5 分钟）运行四个维护任务。它们只在部署后的 Worker 上运行（Cloudflare cron）；本地 `wrangler dev` 可用 `wrangler dev --test-scheduled` 触发。
|
||||
|
||||
| 任务 | 用途 |
|
||||
| --------------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| `discord-sync` | 注册 Discord 斜杠/右键菜单命令：按服务器即时注册，并全局注册（24h 去重，约 1 小时传播） |
|
||||
| `telegram-sync` | 调用 `setWebhook` 指向 `{BASE_URL}/telegram/webhook`（设置了 `TELEGRAM_WEBHOOK_SECRET` 时作为 `secret_token` 传入） |
|
||||
| `audit-prune` | 删除早于 `AUDIT_RETENTION_DAYS`（默认 90）天的 `audit_logs` 记录 |
|
||||
| `storage-prune` | 删除已过期的 `dedup_keys` 记录、超过 7 天的 `delivery_state` 记录和超过 30 天的 `message_tracking` 记录 |
|
||||
|
||||
除任务用到的密钥（`DISCORD_TOKEN`、`DISCORD_APPLICATION_ID`、`TELEGRAM_TOKEN`、`BASE_URL`、`AUDIT_RETENTION_DAYS`）外无需其他配置。
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue