mirror of
https://github.com/Elysola/WebHooker.git
synced 2026-11-20 17:13:49 +00:00
feat(storage): Add delivery state store switch (D1 by default or KV) (#12)
This commit is contained in:
21 files changed
+756
-184
No files matched your search
@@ -25,8 +25,9 @@ Core pipeline: GitHub Webhook → Worker (verify + filter + format) → Discord
|
||||
- Audit log: every admin operation (logins, group/route/member/invite changes) recorded in D1 `audit_logs`; pruned by the scheduled trigger after `AUDIT_RETENTION_DAYS` (default 90)
|
||||
- Group webhook log channel: optional `Group.logTarget` (Discord channel/thread or Telegram chat/topic) receives one summary message per webhook the group's routes dispatched (event, repo, delivery id, per route×target ✅/❌ outcome, green/red color); best-effort, not recorded in `send_logs`
|
||||
- Per-group forge branding: optional `Group.forgeSources` (a list of `{ host, type: "github" | "gitea", name? }` the group defines itself) labels each message's footer with the first entry whose type matches the event's provider and whose host matches the repository URL's hostname (GitHub matches `github.com`, so two Gitea instances can be `git1.example.com`/`git2.example.com`); the label is the entry's optional `name` (fallback: host); links are derived from the repo URL and footer icons use raster PNGs Discord renders (GitHub's `fluidicon.png`, Gitea's `/assets/img/favicon.png` — `.ico` favicons are silently ignored); Discord shows the footer icon + name, Telegram a linked name
|
||||
- Delivery queue: when the `QUEUE` binding is present, webhook ingress enqueues one message per event (not per target) to a Cloudflare Queue (`webhooker-delivery`) and answers `200` only after the send succeeded (`503` + dedup claim released otherwise); a Nitro `cloudflare:queue` plugin consumes batches, takes each delivery with a 5-minute lease (`acquireDelivery`, conditional UPSERT on `delivery_state`), dispatches, and retries when any target failed transiently (5xx/429/network/rate-limit/`FEISHU_TOKEN`, a crashed route, an unreadable ledger, a busy update lock) with exponential backoff (5s/30s/2m/10m) up to the queue `max_retries`, after which the DLQ (`webhooker-delivery-dlq`) marks the delivery dead. Targets that succeeded or failed permanently are recorded (`delivery-target:*` in the idempotency store, keyed by the group's instance, written right after each target and read with one range query) and skipped on retries, so retries never duplicate and one bad target never cancels the others; a permanent failure is remembered (`delivery-failed:*`) so the delivery ends `failed`, not `delivered`. A message whose storage calls fail is retried on its own (the batch is never failed whole); pre-upgrade `delivery-state:{provider}:{groupId}:*` rows are honored only for the group that owned the id (`legacyId`), never for the global endpoint. The consumer runs `max_batch_size: 3` with one per-batch budget (12 minutes, 20 sends — the D1 query / subrequest limits are per invocation): a message left unstarted is retried (copied instead on its last retry if it was never started), and one that made progress continues in a new message. Oversized payloads (queue message over 120,000 UTF-8 bytes) are parked in R2 (`webhooks/YYYY/MM/DD/*.json`, falling back to KV `queue:payload:*` without the R2 binding, 4-day TTL; kept when the queue send fails, since the queue may have accepted it; orphans pruned by `storage-prune`) and resolved by the consumer — a missing/corrupt parked payload never dispatches `{}`: the delivery stays `retrying` with a `PAYLOAD_MISSING` send-log entry asking for a redelivery; delivery state is tracked as the D1 `delivery_state` table (KV `delivery-state:*` fallback). Without the `QUEUE` binding, dispatch stays inline (existing behavior)
|
||||
- Delivery queue: when the `QUEUE` binding is present, webhook ingress enqueues one message per event (not per target) to a Cloudflare Queue (`webhooker-delivery`) and answers `200` only after the send succeeded (`503` + dedup claim released otherwise); a Nitro `cloudflare:queue` plugin consumes batches, takes each delivery with a 5-minute lease (`acquireDelivery`, conditional UPSERT on `delivery_state`), dispatches, and retries when any target failed transiently (5xx/429/network/rate-limit/`FEISHU_TOKEN`, a crashed route, an unreadable ledger, a busy update lock) with exponential backoff (5s/30s/2m/10m) up to the queue `max_retries`, after which the DLQ (`webhooker-delivery-dlq`) marks the delivery dead. Targets that succeeded or failed permanently are recorded (`delivery-target:*` in the idempotency store, keyed by the group's instance, written right after each target and read with one range query) and skipped on retries, so retries never duplicate and one bad target never cancels the others; a permanent failure is remembered (`delivery-failed:*`) so the delivery ends `failed`, not `delivered`. A message whose storage calls fail is retried on its own (the batch is never failed whole); pre-upgrade `delivery-state:{provider}:{groupId}:*` rows are honored only for the group that owned the id (`legacyId`), never for the global endpoint. The consumer runs `max_batch_size: 3` with one per-batch budget (12 minutes, 20 sends — the D1 query / subrequest limits are per invocation): a message left unstarted is retried (copied instead on its last retry if it was never started), and one that made progress continues in a new message. Oversized payloads (queue message over 120,000 UTF-8 bytes) are parked in R2 (`webhooks/YYYY/MM/DD/*.json`, falling back to KV `queue:payload:*` without the R2 binding, 4-day TTL; kept when the queue send fails, since the queue may have accepted it; orphans pruned by `storage-prune`) and resolved by the consumer — a missing/corrupt parked payload never dispatches `{}`: the delivery stays `retrying` with a `PAYLOAD_MISSING` send-log entry asking for a redelivery; delivery state is tracked as the D1 `delivery_state` table (KV `delivery-state:*` fallback, and the store itself when `DELIVERY_STATE_STORE=kv`). Without the `QUEUE` binding, dispatch stays inline (existing behavior)
|
||||
- Storage quotas: the design avoids KV write pressure (Workers Free KV allows ~1,000 writes/day) by keeping high-frequency ephemeral writes in D1 instead — webhook dedup and every other one-time claim (`dedup_keys`, atomic `ON CONFLICT` UPSERT via `idempotencyStore` with `has`/`claim`/`release`), delivery state and message tracking all use D1 rows via `canUseD1` (prepare+batch probe) with automatic KV fallback when D1 is unavailable or a table is missing (`isMissingTable`, unmigrated); other D1 errors propagate so one claim never splits across D1 and KV. D1 Free allows 100,000 rows written/day, so the per-event KV write cost drops to ~0; config is cached in memory only, with D1 authoritative. In-flight work shared across concurrent requests (setup, fallback marker, Feishu / App tokens, translations) goes through `sharedFlight`, which waits on another request's run only for a bounded time (a canceled request's promise may never settle)
|
||||
- Delivery state store (`DELIVERY_STATE_STORE`, `d1` default, `kv` opt-in via `server/lib/storage/state-store.ts`): the delivery hot path — ingress dedup claims and replay nonces (`deliveryClaimStore`), `delivery_state` leases, the `delivery-target:*`/`delivery-failed:*` ledger, `message_tracking` and the chain lock, and group-log summary claims — follows the switch; KV answers at the edge and keeps a live lease off a second consumer with its stamp, but allows ~1,000 writes/day on Workers Free and has no compare-and-set: a lease taken over after its holder vanished, or a claim raced, can still be duplicated. State written in KV mode is marked (`storage:kv-fallback`) and still honored after switching back to D1; the other direction does not read D1, so the D1 rows of the last days (claims and their confirmations, delivery states, the target ledger, message tracking for 30 days) are invisible — an already-finished event redelivered is sent again, already-sent targets are sent again, an in-progress edit chain posts a new message. Draining the queue only keeps in-flight deliveries from overlapping; those rows stay invisible either way, so carry the state over or accept the repeats. Configuration, logs, audit, metrics and the claims tied to a D1 transaction of their own (invites, sessions, tokens, bot interactions, group ids, installation provisioning) stay in D1 in both modes; an unrecognized value is logged and treated as `d1`
|
||||
- Filter DSL: a route's flat `filters` can be replaced by a nested `ast` (`all`/`any`/`not` nodes, `server/lib/events/filter-ast.ts`) that takes precedence when present; the `field` filter type reads any payload value by JSONPath (`path`, arrays expand so any element matches) and 12 comparison operators (`op`: `eq`/`ne`/`contains`/`startsWith`/`endsWith`/`regex`/`gt`/`gte`/`lt`/`lte`/`in`/`exists`) extend the legacy glob/regex/case-insensitive-exact semantics; named filter fragments (D1 `d1_fragments` via `server/lib/fragments.ts`, per-group) are editor-side templates inlined into the route's `ast` on insert — the matcher never resolves fragment references; the route editor (`RouteEditor.vue`) exposes a recursive AST builder (`FilterNodeEditor.vue`: plain selects for filter type/operator, an all/any segmented toggle, NOT toggle + delete pinned per condition), searchable multi-value / path inputs (`FilterCombobox.vue`) fed by a bilingual suggestion catalog (`app/utils/filter-catalog.ts`: GitHub events/actions from `@octokit/webhooks` plus Gitea extras, common payload paths and their known values, English keys searchable in either language) where custom values are allowed but rejected when the matcher could never match them (`validateFilterValue`), a live explanation and a stateless `POST /admin/api/test-match` dry-run
|
||||
- Local dev: wrangler + Miniflare
|
||||
|
||||
@@ -143,17 +144,21 @@ server/ # Nitro server
|
||||
│ └── richheader.ts # GET /api/richheader: Open Graph page for Telegram avatar link-preview card
|
||||
├── observability/ # delivery metrics aggregation
|
||||
│ └── metrics.ts # DeliveryMetrics + getDeliveryMetrics (SQL GROUP BY over send_logs in one D1 batch: totals, per platform/event/status, duration, attempts, recent failures)
|
||||
├── storage/ # storage primitives: canUseD1 probe, D1 config store (memory cache ≤60 s → D1), R2 payload store
|
||||
├── storage/ # storage primitives: canUseD1 probe, D1 config store (memory cache ≤60 s → D1), R2 payload store,
|
||||
│ # state-store.ts (DELIVERY_STATE_STORE: deliveryStateStore/deliveryStateUsesD1/deliveryClaimStore)
|
||||
│ ├── d1.ts # canUseD1(db) — prepare+batch probe for D1 availability (falls back to KV)
|
||||
│ ├── config-store.ts # ConfigStore: d1ConfigStore (per-group writes, CAS updateGroup, atomic rename/delete, one-time KV seed, one-time identity migration, unreadable-row isolation)
|
||||
│ │ # and kvConfigStore (no D1); `position` order when migrated; ConfigConflictError
|
||||
│ ├── state-store.ts # DELIVERY_STATE_STORE: deliveryStateStore (d1|kv), deliveryStateUsesD1, deliveryClaimStore (delivery-path idempotency store)
|
||||
│ └── payload.ts # PayloadStore backed by R2 (webhooks/YYYY/MM/DD/*.json), no-binding stub throws
|
||||
└── lib/ # shared infra
|
||||
├── i18n.ts # loadTranslations (KV i18n:{lang} overrides), t() with param interpolation
|
||||
├── idempotency.ts # IdempotencyStore (has/claim/release) + kv/d1 stores (D1 atomic UPSERT, KV fallback only for a missing table) + tenantScope (`@global`) + deliveryKey
|
||||
├── idempotency.ts # IdempotencyStore (has/claim/release) + kv/d1 stores (D1 atomic UPSERT, KV fallback only for a missing table) + tenantScope (`@global`) + deliveryKey;
|
||||
│ # the D1 argument is optional (undefined = KV only, used by the delivery store switch)
|
||||
├── correlation.ts # newCorrelationId() — per-request/delivery correlation id for logs + responses
|
||||
├── shared-flight.ts # sharedFlight: in-flight work shared by concurrent requests, waited on for a bounded time
|
||||
├── message-tracker.ts # MessageTracker (versioned { messageId, version } entries, legacy bare ids read as version 0) + kv/d1 trackers with KV fallback
|
||||
├── message-tracker.ts # MessageTracker (versioned { messageId, version } entries, legacy bare ids read as version 0) + kv/d1 trackers with KV fallback;
|
||||
│ # messageTracker/chainLock take an explicit useD1 (the delivery store switch)
|
||||
├── send-log.ts # SendRecord, recordSend/getSendLog/getSendLogById/getSendLogByDelivery/getFailedSendLog (D1 send_logs)
|
||||
├── send-log-batch.ts # recordSendBatch (D1 batch INSERT for send_logs)
|
||||
├── audit.ts # recordAudit/getAuditLog/pruneAuditLogs (D1 audit_logs, best-effort writes)
|
||||
@@ -161,7 +166,7 @@ server/ # Nitro server
|
||||
└── locales/ # en.ts, zh.ts translation dictionaries
|
||||
|
||||
tests/ # bun test unit tests (webhook, formatter, discord, telegram, admin, groups, invites, audit, send-log, token-store,
|
||||
# data-integrity, ingress-dispatch, discord-interactions, telegram-webhook, ...)
|
||||
# data-integrity, ingress-dispatch, discord-interactions, telegram-webhook, delivery-state-store, ...)
|
||||
tests/d1-sqlite.ts # in-memory SQLite behind the D1 API (all migrations applied, FKs on) + a Map-backed KV, for real-SQL tests
|
||||
tests/fixtures/ # provider payload fixtures (github/gitea/custom) feeding provider + formatter tests
|
||||
tests/__snapshots__/ # formatter snapshot golden files (toMatchSnapshot)
|
||||
@@ -284,6 +289,7 @@ Rule: no functional change ships without its documentation; docs and code must n
|
||||
- **Queue**: optional `QUEUE` producer binding plus consumers `webhooker-delivery` and its DLQ `webhooker-delivery-dlq` (declared in `wrangler.jsonc`); when absent, webhook dispatch stays inline
|
||||
- **R2**: optional `PAYLOAD` binding (bucket `webhooker-payloads`) for oversized queue payloads; without it, oversized payloads fall back to KV `queue:payload:*`
|
||||
- **Access control**: `ADMIN_USER_IDS` (super admins), `DEBUG_MODE=true` (local-only authentication bypass), `ALLOW_SELF_SIGNUP` (optional personal group on first login), `AUDIT_RETENTION_DAYS` (default 90) — all plain env vars, not secrets
|
||||
- **Delivery state store**: `DELIVERY_STATE_STORE` (`d1` default; `kv` moves the delivery hot path — ingress dedup and replay nonces, delivery leases and state, the target ledger, message tracking and chain locks, group-log summary claims — to KV) — a plain env var, not a secret
|
||||
- **Discord**: `DISCORD_PUBLIC_KEY` (Interactions Endpoint signature verification, from Discord Developer Portal) and `DISCORD_APPLICATION_ID` (optional, auto-resolved via `GET /oauth2/applications/@me` when omitted) are required for interactions
|
||||
- **Telegram**: `TELEGRAM_TOKEN` (Bot API token from BotFather) required for Telegram routes; `TELEGRAM_WEBHOOK_SECRET` (optional secret token for `POST /telegram/webhook` verification — derived from `TELEGRAM_TOKEN` when unset, never disabled); avatars are sent as a link-preview card via the built-in `GET /api/richheader` (overridable with `TELEGRAM_RICH_HEADER_HOST`)
|
||||
- **Feishu**: `FEISHU_APP_ID` / `FEISHU_APP_SECRET` for sending; inbound commands/buttons need `FEISHU_ENCRYPT_KEY` and/or `FEISHU_VERIFICATION_TOKEN`
|
||||
|
||||
Reference in new issue
Block a user