From 978c53eeb3dee6ade9a0d1fbd0e5a12b40547fef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=BF=85=E8=86=80=E5=96=B5?= Date: Sat, 24 Oct 2026 16:19:06 +0800 Subject: [PATCH] fix(security): harden webhook delivery pipeline and group storage (#11) * fix(log): preserve structured error details Error objects lose their useful details when serialized as plain JSON. Keep the message, stack and cause in structured log output. * fix(runtime): bound shared in-flight waits A canceled request can leave its shared promise pending forever. Bound joins and coordinate a single replacement so later requests make progress without multiplying upstream work. * fix(bot): reject oversized callback bodies Reject declared oversized bot callbacks before reading their bodies. This keeps unauthenticated requests from wasting worker resources. * test(d1): exercise real sqlite transactions Exercise real foreign keys, transactions and conditional statements instead of mocks that cannot reproduce lost updates or cascades. * fix(storage): preserve claims across kv fallback Only missing tables justify KV fallback; other failures must surface. Remember fallback writes so migration to D1 cannot forget claims and replay completed work. Share marker checks without extra writes. * fix(filter): use isolated linear-time patterns Backtracking patterns can exhaust worker CPU on untrusted payloads. Use linear-time matching and separate bounded cache namespaces so glob and regex entries cannot corrupt each other. * fix(filter): read each event branch correctly Deployment and check branches live in event-specific nested fields. Read those fields and remove only the actual leading ref prefix. * fix(filter): read only own payload properties Inherited object properties are not webhook payload data. Read own properties only so JSONPath cannot traverse prototypes. * fix(filter): validate executable condition trees Malformed trees and unsupported patterns must not silently turn an exclude rule into a pass. Validate the shape and only the patterns the evaluator actually executes. * fix(i18n): bound and cache translation loads Repeated misses and simultaneous override loads spend KV reads for the same language. Cache confirmed misses and share bounded loads without caching read failures as missing translations. * i18n(bot): explain command and account failures Explain account linking, command failures and skipped delivery outcomes consistently in both supported languages. * fix(github): bound and reuse app token fetches Repeated token exchanges waste requests and JWT work for every event. Reuse valid installation tokens, share refreshes and back off briefly after failures while keeping every shared wait bounded. * fix(github): restrict enrichment request urls Webhook-provided URLs must never receive an installation token on another host. Restrict enrichment to GitHub repository API paths, reject redirects and bound optional requests. * fix(formatter): preserve invalid code points Out-of-range numeric entities can throw while formatting a delivery. Keep invalid code points verbatim instead of aborting the message. * fix(formatter): preserve bounded markdown text Code spans, fences, indentation and autolinks are Markdown content, not HTML tags to strip. Preserve them while bounding cleanup input and reusing results across routes for the same event. * fix(formatter): strip only leading ref prefixes A short branch or tag may itself contain refs/heads or refs/tags. Strip one leading full-ref prefix at the caller and keep short names unchanged when generating labels and links. * fix(gitea): link push authors to their forge A Gitea author is not the same-named GitHub account. Build author profile links from the originating forge. * fix(formatter): render deployment state values The deployment status object stringifies as [object Object]. Render its actual state in the status field. * fix(formatter): read security severity levels The scanner severity is not the security severity of an alert. Prefer the security level so critical and high alerts get the right label and color. * fix(formatter): hide actions on other forges Action buttons call the GitHub API as the linked user. Hide them for other forges to avoid operating on a same-named GitHub repository. * fix(check): isolate updates by check suite Same-named checks from different suites must not edit one message. Include the suite or app identity and retain the former key for controlled adoption of existing tracked messages. * fix(logs): scope queries before limiting results Filter authorized groups in SQL before LIMIT and aggregate only visible rows. Preserve current group attribution for late writes, surface read failures and combine related totals in one scan. * perf(d1): index pruning and failed send queries Pruning and failure views should not scan all historical rows. Index the high-value access paths without adding a write-heavy index for infrequent delivery-detail lookups. * fix(discord): bound individual api requests An unbounded API request can outlive an update-chain lock. Bound each request so another holder cannot race a late edit. * fix(discord): defer long rate-limit waits Long rate-limit sleeps block delivery progress inside a worker. Return a retryable 429 for the queue when the wait exceeds the bound. * fix(discord): encode channel path segments Stored channel and message ids are path segments, not URL paths. Encode them so aliases cannot address another Discord resource. * fix(discord): reuse nonces after lost responses A successful create may lose its response. Reuse a stable nonce with enforce_nonce so retries recover that message instead of posting duplicates or accepting an unreadable success. * fix(discord): constrain embeds and role mentions Respect combined embed and component limits as well as field caps. Restrict allowed mentions to validated route roles so payload text cannot notify everyone. * fix(discord): resume bounded command syncs Application changes, command rollbacks and large guild lists can leave commands stale indefinitely. Track the current hash per app and resume bounded paginated work without needless KV writes. * fix(github): verify installation management An installation id is user-controlled, and repository access is not installation management. Verify the owning account or an active organization admin using the user token and read:org scope. * fix(github): resolve immutable member ids A login can be renamed and reclaimed before its first sign-in. Resolve the current immutable user id when membership is granted so a future holder of the login cannot inherit access. * fix(auth): revoke stale sessions and tokens Revoked or rotated credentials must stop working despite stale KV replicas. Track the live token and revocations in D1 and preserve immutable identity bindings rather than trusting a recycled login. * fix(auth): bind sessions and tokens to identity Automatic retries can repeat non-idempotent user actions. Disable those retries and expose a write hook so bot handlers can record completion immediately before the GitHub write. * fix(discord): defer and deduplicate interactions Acknowledge slow actions before Discord times out and deduplicate interaction writes. Preserve comment text and reject edits whose source changed after the modal opened. * fix(telegram): always verify webhook secrets Webhook verification must not silently disable itself when the optional secret is absent. Derive a stable secret from the bot token and reject callbacks when verification is unavailable. * fix(telegram): bound individual api requests Bound individual platform calls so they cannot outlive the lock protecting a message update or stall a whole delivery invocation. * fix(telegram): honor nested rate-limit delays Telegram places retry_after inside parameters, not at the top level. Read that value and defer long waits to the delivery queue. * fix(telegram): report unreadable send results An unreadable successful response may already have sent the message. Report an explicit uncertain result instead of retrying a create without an idempotency key or pretending it succeeded. * fix(telegram): keep editable messages as text Guessing photo versus text from message content makes later edits use the wrong API and a smaller caption budget. Always send text and use the avatar as a link preview. * fix(telegram): render hard breaks as newlines Telegram HTML does not support br tags. Render Markdown hard breaks as plain newlines so otherwise valid messages are accepted. * fix(telegram): clamp visible html characters Telegram counts visible characters, not HTML source bytes. Truncate without splitting tags, entities or surrogate pairs. * fix(telegram): honor the author emoji toggle The author marker must obey the group emoji preference just like formatter-added decoration. * fix(telegram): keep completed commands claimed Redelivery after a completed write must never post another comment. Keep completion claims despite reply failures, scope updates per bot and only issue account-login links in private chats. * fix(feishu): bound individual api requests Bound individual platform calls so they cannot outlive the lock protecting a message update or stall a whole delivery invocation. * fix(feishu): classify retryable api failures Business error codes can be transient even in HTTP 200 responses. Classify token and rate-limit errors for retry and defer long waits. * fix(feishu): retry sends with stable uuids An unreadable response does not prove a send failed. Reuse the same uuid for retries so recovering the result cannot create a second Feishu message. * fix(feishu): refresh and evict tokens safely Separate app token caches, stop reusing refused tokens and share refreshes. Cache failures briefly and keep valid tokens in memory when a best-effort KV cache write fails. * fix(feishu): use the card update endpoint Delayed card updates use the card update endpoint; the config endpoint cannot update an interaction response. * fix(feishu): neutralize untrusted mentions Forge text is untrusted and can contain at-all tags. Neutralize mentions in card bodies and footers before rendering. * fix(feishu): authenticate and dedupe callbacks Authenticate signed or token-verified callbacks before dispatching commands. Decode encrypted envelopes and handle card actions using the correct fields without claiming irrelevant chat events. * fix(target): normalize actual destinations Different target spellings can name the same actual destination. Normalize channel, thread and topic identities for ownership checks and per-route duplicate suppression. * fix(config): transact per-group configuration Whole-table writes and cached snapshots can overwrite other groups or resurrect removed configuration. Use per-group guarded writes, one-time identity migration and explicit unreadable-row failures. * fix(group): preserve identity through lifecycle Group names are mutable and reusable, so they cannot own durable state. Preserve immutable identities across lifecycle operations and commit invite use with membership under transaction-time guards. * fix(oauth): confirm account links and installs A login link or installation id is not proof of account ownership. Verify installation management, bind browser state and require an explicit account-link confirmation before granting chat access. * fix(queue): fence leases and record target outcomes Retries must remember completed targets and distinguish active lease holders. Fence state updates, preserve permanent outcomes and reject missing parked payloads instead of dispatching empty data. * fix(dispatch): isolate and resume tenant sends Check tenant sources before filters and never dispatch orphan routes. Record each target outcome, isolate update chains by forge and group identity, and keep stale events from overwriting newer messages. * fix(queue): bound batches and resume deliveries A queue invocation has shared query and subrequest limits. Use a small batch and shared send budget, renew fenced leases and resume only unfinished work without exhausting unstarted messages. * fix(admin): guard revisions and tenant targets Read-time authorization alone cannot protect concurrent edits. Guard revisions, current roles and destination ownership in the write transaction, and use per-group endpoints for isolated changes. * fix(webhook): confirm only accepted deliveries Acknowledging before queue acceptance permanently loses deliveries. Confirm only accepted work, release failed claims, scope replay identities and strip tenant-supplied installation authority. * fix(storage): prune abandoned parked payloads Oversized payloads can remain after interrupted enqueue attempts. Prune objects beyond their retry lifetime without deleting payloads that a current delivery may still need. * fix(console): expose api conflict statuses Callers need the HTTP status to distinguish stale edits from other failures and reload current data after a conflict. * refactor(console): model group revisions Carry server revisions explicitly so console writes can reject changes based on an outdated group snapshot. * fix(console): send loaded route revisions Send the revision actually loaded with the route list and adopt the server response after a save. Reload on conflict instead of silently overwriting another editor. * fix(console): guard fragment saves by revision Fragments share a group version with other configuration writes. Submit the loaded revision and reload on conflict to preserve concurrent edits. * fix(console): save groups with guarded writes Whole-list saves can overwrite unrelated groups. Use per-group revision-aware writes, retain rename results and confirm deletion on the exact group, routes and version shown to the user. * fix(console): reject edits of stale routes A conflict reload can leave an older route editor open. Compare its original value before saving so it cannot overwrite the newly loaded route. * fix(console): show overview loading failures Failed overview requests must not look like an empty dashboard or a successful refresh. Show the reason while retaining prior data. * fix(console): explain rejected install bindings Explain why an installation binding was refused instead of silently returning to the console with no visible outcome. * fix(console): allow selecting a group named all The all sentinel collides with a valid group id. Use a value that cannot be a group id so that group stays selectable. * fix(console): discard stale log detail responses An older detail request can finish after a newer selection or close. Discard it so the dialog never shows the previously clicked record. * fix(console): retain legacy group owners Groups predating member roles store owners in adminIds. Load those as owners so adding a member never drops all old owners. * fix(console): lock member edits during saves Edits made during a save are overwritten by its result. Disable member controls and guard handlers until the write settles. * fix(console): show invite loading failures Failed invite loads must not look like no pending invites. Show the error and discard responses from a previously shown group. * fix(console): retain legacy discord log targets A stored log target without platform means Discord. Keep that meaning in the editor instead of clearing the target. * fix(console): retain undisplayed forge sources An editor that only displays some providers must preserve the others. Keep stored custom sources when saving unrelated group settings. * fix(console): lock the group editor while saving Prevent edits while a group save is in flight and make super-only installation settings readonly instead of silently ignoring edits. * fix(console): lock route edits and show failures Lock the route editor while saving and show fragment load failures with a retry action rather than presenting a false empty list. * fix(config): expose the legal contact override Declare the public runtime key so the documented environment override reaches legal pages in deployed Nuxt builds. * build(deploy): migrate d1 before deployment Apply pending D1 migrations before deploying code that requires them. Stop deployment when migration fails instead of running against an incomplete schema. * docs(worker): align guides with runtime behavior Synchronize both language guides, API references and examples with the implemented authorization, storage and delivery behavior. Document migration requirements and accepted operational limits. * fix(storage): repair orphaned KV routes Repair orphaned KV routes before serving KV-only configuration. * fix(security): fully escape forge HTML --------- Co-authored-by: RhenCloud --- .env.example | 8 +- AGENTS.md | 125 +- README.md | 64 +- README.zh.md | 65 +- app/components/AdminHome.vue | 3 + app/components/AuditLog.vue | 6 +- app/components/ConsolePage.vue | 83 +- app/components/GroupEditor.vue | 424 +-- app/components/MembersPanel.vue | 40 +- app/components/MetricsPanel.vue | 6 +- app/components/RouteEditor.vue | 472 +-- app/components/SendLogs.vue | 21 +- app/composables/useApi.ts | 22 +- app/composables/useFragments.ts | 31 +- app/composables/useGroupRoutes.ts | 35 +- app/composables/useGroups.ts | 89 +- app/composables/useI18n.ts | 17 + app/composables/useInvites.ts | 3 +- app/types.ts | 6 + bun.lock | 4 +- config.example.yaml | 31 +- docs/api/actions.md | 22 +- docs/api/admin.md | 74 +- docs/api/oauth.md | 19 +- docs/api/overview.md | 26 +- docs/contributing.md | 14 +- docs/events/supported.md | 37 +- docs/guide/commands.md | 26 +- docs/guide/configuration.md | 57 +- docs/guide/deployment.md | 48 +- docs/guide/faq.md | 4 +- docs/guide/filters.md | 47 +- docs/guide/getting-started.md | 11 +- docs/guide/groups.md | 51 +- docs/guide/i18n.md | 12 +- docs/guide/ingress.md | 45 +- docs/guide/introduction.md | 4 +- docs/guide/logs.md | 46 +- docs/guide/message-format.md | 4 +- docs/guide/routes.md | 22 +- docs/guide/storage.md | 90 +- docs/guide/tasks.md | 12 +- docs/zh/api/actions.md | 22 +- docs/zh/api/admin.md | 76 +- docs/zh/api/oauth.md | 19 +- docs/zh/api/overview.md | 28 +- docs/zh/contributing.md | 16 +- docs/zh/events/supported.md | 21 +- docs/zh/guide/commands.md | 26 +- docs/zh/guide/configuration.md | 54 +- docs/zh/guide/deployment.md | 46 +- docs/zh/guide/faq.md | 4 +- docs/zh/guide/filters.md | 47 +- docs/zh/guide/getting-started.md | 11 +- docs/zh/guide/groups.md | 51 +- docs/zh/guide/i18n.md | 12 +- docs/zh/guide/ingress.md | 23 +- docs/zh/guide/introduction.md | 4 +- docs/zh/guide/logs.md | 46 +- docs/zh/guide/message-format.md | 4 +- docs/zh/guide/routes.md | 22 +- docs/zh/guide/storage.md | 90 +- docs/zh/guide/tasks.md | 12 +- migrations/0013_prune_indexes.sql | 19 + nuxt.config.ts | 4 +- package.json | 3 +- server/lib/cf.ts | 13 + server/lib/config.ts | 123 +- server/lib/config/schema.ts | 6 + server/lib/core/dispatch.ts | 1017 +++++-- server/lib/drivers/discord/commands.ts | 173 +- server/lib/drivers/discord/index.ts | 8 +- server/lib/drivers/discord/interactions.ts | 293 +- server/lib/drivers/discord/render.ts | 76 +- server/lib/drivers/discord/rest.ts | 63 +- server/lib/drivers/feishu/index.ts | 16 +- server/lib/drivers/feishu/render.ts | 17 +- server/lib/drivers/feishu/rest.ts | 175 +- server/lib/drivers/feishu/updates.ts | 420 ++- server/lib/drivers/index.ts | 17 + server/lib/drivers/telegram/commands.ts | 268 +- server/lib/drivers/telegram/index.ts | 46 +- server/lib/drivers/telegram/render.ts | 64 +- server/lib/drivers/telegram/rest.ts | 44 +- server/lib/drivers/telegram/secret.ts | 22 + server/lib/drivers/telegram/updates.ts | 18 +- server/lib/events/filter-ast.ts | 163 +- server/lib/formatters/check.ts | 13 +- server/lib/formatters/deployment.ts | 8 +- server/lib/formatters/helpers.ts | 119 +- server/lib/formatters/index.ts | 19 +- server/lib/formatters/push.ts | 19 +- server/lib/formatters/security.ts | 11 +- server/lib/fragments.ts | 128 +- server/lib/github/app-token.ts | 74 +- server/lib/github/check-run.ts | 16 +- server/lib/github/oauth.ts | 200 +- server/lib/github/store.ts | 216 +- server/lib/github/workflow-jobs.ts | 19 +- server/lib/lib/audit.ts | 67 +- server/lib/lib/i18n.ts | 20 + server/lib/lib/idempotency.ts | 208 +- server/lib/lib/locales/en.ts | 1 + server/lib/lib/locales/zh.ts | 1 + server/lib/lib/log.ts | 43 +- server/lib/lib/message-tracker.ts | 269 +- server/lib/lib/send-log-batch.ts | 95 +- server/lib/lib/send-log.ts | 69 +- server/lib/lib/shared-flight.ts | 81 + server/lib/observability/metrics.ts | 65 +- server/lib/providers/custom/index.ts | 57 +- server/lib/queue/consumer.ts | 429 ++- server/lib/queue/delivery.ts | 474 ++- server/lib/storage/config-store.ts | 1494 ++++++++-- server/lib/storage/d1.ts | 89 + server/lib/types.ts | 82 +- server/lib/web/actions.ts | 2 +- server/lib/web/admin.ts | 1521 ++++++++-- server/lib/web/auth.ts | 20 +- server/lib/web/group-lifecycle.ts | 148 + server/lib/web/groups.ts | 350 ++- server/lib/web/invites.ts | 486 +++- server/lib/web/oauth.ts | 427 ++- server/lib/web/session.ts | 67 +- server/lib/web/tenants.ts | 99 +- server/lib/webhook.ts | 405 ++- server/middleware/config-store.ts | 13 + server/plugins/queue-consumer.ts | 2 + server/routes/admin/api/[...slug].ts | 8 + server/routes/auth/github/link.post.ts | 3 + server/routes/discord/interactions.post.ts | 3 +- server/routes/feishu/webhook.post.ts | 3 +- server/routes/telegram/webhook.post.ts | 3 +- server/tasks/storage-prune.ts | 25 +- tests/admin-api.test.ts | 11 +- tests/admin.test.ts | 30 +- tests/audit.test.ts | 104 +- tests/config-store.test.ts | 698 +++-- tests/custom-replay.test.ts | 23 +- tests/d1-sqlite.ts | 126 + tests/d1-stores.test.ts | 168 +- tests/data-integrity.test.ts | 931 ++++++ tests/delivery-metrics.test.ts | 132 +- tests/delivery.test.ts | 113 +- tests/discord-interactions.test.ts | 313 ++ tests/discord.test.ts | 49 +- tests/feishu-inbound.test.ts | 249 +- tests/filter-ast.test.ts | 33 + tests/formatter.test.ts | 21 +- tests/github-workflow-jobs.test.ts | 2 +- tests/idempotency.test.ts | 5 +- tests/identity-and-claims.test.ts | 229 ++ tests/ingress-dispatch.test.ts | 560 ++++ tests/install.test.ts | 176 +- tests/invites.test.ts | 169 +- tests/lifecycle-safety.test.ts | 3047 ++++++++++++++++++++ tests/message-tracker.test.ts | 15 +- tests/platform-contract.test.ts | 23 + tests/queue-consumer.test.ts | 447 ++- tests/telegram-commands.test.ts | 436 ++- tests/telegram-webhook.test.ts | 38 + tests/telegram.test.ts | 15 +- wrangler.jsonc | 4 +- 163 files changed, 18459 insertions(+), 3607 deletions(-) create mode 100644 migrations/0013_prune_indexes.sql create mode 100644 server/lib/drivers/telegram/secret.ts create mode 100644 server/lib/lib/shared-flight.ts create mode 100644 server/lib/web/group-lifecycle.ts create mode 100644 server/middleware/config-store.ts create mode 100644 server/routes/auth/github/link.post.ts create mode 100644 tests/d1-sqlite.ts create mode 100644 tests/data-integrity.test.ts create mode 100644 tests/discord-interactions.test.ts create mode 100644 tests/identity-and-claims.test.ts create mode 100644 tests/ingress-dispatch.test.ts create mode 100644 tests/lifecycle-safety.test.ts create mode 100644 tests/telegram-webhook.test.ts diff --git a/.env.example b/.env.example index 3dfb0f1..2d83589 100644 --- a/.env.example +++ b/.env.example @@ -15,15 +15,19 @@ DISCORD_APPLICATION_ID=your-application-id # Telegram TELEGRAM_TOKEN=your-bot-token -TELEGRAM_WEBHOOK_SECRET=your-webhook-secret +TELEGRAM_WEBHOOK_SECRET=your-webhook-secret # optional: derived from TELEGRAM_TOKEN when unset # TELEGRAM_RICH_HEADER_HOST=https://your-domain # Feishu (Lark / 飞书) FEISHU_APP_ID=your-app-id FEISHU_APP_SECRET=your-app-secret +# Inbound commands/buttons need at least one of these (Events & Callbacks → Encryption Strategy): +FEISHU_ENCRYPT_KEY=your-encrypt-key +FEISHU_VERIFICATION_TOKEN=your-verification-token # Admin -ADMIN_USER_IDS=your-github-id,your-github-login +# Prefer numeric GitHub user ids: a login is bound to the first account that signs in with it +ADMIN_USER_IDS=your-github-id # DEBUG_MODE=true # local development only: bypasses all admin authentication # ALLOW_SELF_SIGNUP=1 # non-admin GitHub users get a personal group on first login (default off) # AUDIT_RETENTION_DAYS=90 # audit log retention for the scheduled cleanup (default 90) diff --git a/AGENTS.md b/AGENTS.md index 832eea8..f77e2d8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,24 +8,25 @@ Core pipeline: GitHub Webhook → Worker (verify + filter + format) → Discord ## Key Decisions -- Runtime: Cloudflare Workers via the Nitro `cloudflare_module` preset (`_worker.js`), H3 event handlers in `server/routes/` +- Runtime: Cloudflare Workers via the Nitro `cloudflare_module` preset (entry `.output/server/index.mjs`), H3 event handlers in `server/routes/` - UI: Vue 3 + Tailwind CSS v4 (`@tailwindcss/vite`) + `@elysola/ui` (Nuxt module, `Rc` prefix); admin console is a client-side SPA (`routeRules: "/admin/**": { ssr: false }`), home/legal pages render server-side; app-level composites (`FilterCombobox`, `ConfirmDialog`) are built on the reka-ui primitives that `@elysola/ui` ships (same copy, pre-bundled via `vite.optimizeDeps.include` so dev keeps a single instance and dialogs/popovers stack correctly) and styled like their Rc counterparts - Styling: Elysola UI supplies semantic components and blue theme tokens; `app/assets/css/main.css` retains `--wh-*` RGB-triplet aliases for app-specific Tailwind utilities and bespoke layouts (avoiding collision with Elysola's `--surface`/`--text`/`--border` hex tokens); theme switching supports auto/light/dark via `useTheme` / `ThemeToggle`, toggling `.dark` with inline anti-flash head script - Discord interactions: HTTPS Interactions Endpoint (`POST /discord/interactions`, Ed25519-signed) — no Discord Gateway / Durable Object; bot stays offline, messages always sent via REST -- Storage: D1 is the source of truth for config (routes/groups via `d1_routes`/`d1_groups`, see `server/lib/storage/config-store.ts`; each route's per-group order lives in `d1_routes.position` — migration 0012 — and routes load ordered by `group_id, position, id`, falling back to id order on an unmigrated database), send/audit logs, dedup (`dedup_keys`), delivery state (`delivery_state`) and message tracking (`message_tracking` via `server/lib/storage/d1.ts` `canUseD1` gate). KV keeps only cache + short-lived/ephemeral state (tokens, OAuth state, admin sessions, `msg:*`-adjacent locks, per-group secrets `tenant:*`, invites, i18n overrides) with explicit TTLs. R2 parks oversized queue payloads (`webhooks/YYYY/MM/DD/*.json`, `server/lib/storage/payload.ts`) instead of KV. A `storage-prune` scheduled task cleans up expired dedup/delivery/message-tracking rows +- Storage: D1 is the source of truth for config (routes/groups via `d1_routes`/`d1_groups`, see `server/lib/storage/config-store.ts`; each route's per-group order lives in `d1_routes.position` — migration 0012 — and routes load ordered by `group_id, position, id`, falling back to id order on an unmigrated database). Config writes are per group — one group row (`createGroup` insert-if-absent, `updateGroup` compare-and-swap on `d1_groups.version`), one group's routes or fragments in one transaction (`saveGroupRoutes`, `saveGroupFragments`), `renameGroup`/`deleteGroup` as single batches — and never fall back to KV: a failed D1 write is an error. Access checks, read-modify-write paths, webhook ingress and the queue consumer read with `{ fresh: true }` (no cache, so revoked routes/targets stop at once); other reads are cached in memory ≤60 s per isolate and `loadConfig` returns a fresh object per call. A stored route or group row that cannot be read (corrupt JSON, wrong shape, a pattern RE2 cannot compile) is reported as `unreadable` for its group only: that group's deliveries fail visibly and retry, and its routes cannot be replaced without `discardUnreadable`. Legacy KV `config:*` is seeded into D1 once (marker `config:d1-seeded`, only while both tables are empty); `server/middleware/config-store.ts` binds the D1 store on every request. The admin API returns a content `revision` for every group and per-group route/fragment list and answers `409` when a write's revision is stale; the check runs inside the write's transaction (`groupVersionGuard` aborts the D1 batch unless `d1_groups.version` is unchanged, and every route/fragment write bumps it), so two writes from the same revision can never both succeed. Rename/delete commit history moves/detachment and the retired-id marker in the same batch. Every group carries an immutable server-assigned `instance` (never sent to clients; existing groups got one in a one-time identity migration, which also sets `legacyId`): secrets, invites, delivery/dedup scopes (`@i:{instance}`), target ledgers, message-tracking chains and late send/audit rows follow the instance, so a group re-created under a reused id inherits nothing of the former one. Dispatch never treats unreadable configuration as empty (no KV copy → error → retry). Also in D1: send/audit logs, dedup (`dedup_keys`), delivery state (`delivery_state`) and message tracking (`message_tracking` via `server/lib/storage/d1.ts` `canUseD1` gate). KV keeps only short-lived/ephemeral state (tokens, OAuth state, admin sessions, invites, Feishu token / Discord command-sync / Telegram webhook records, i18n overrides; legacy `tenant:*` secrets are only read as a fallback) with explicit TTLs. R2 parks oversized queue payloads (`webhooks/YYYY/MM/DD/*.json`, `server/lib/storage/payload.ts`) instead of KV. A `storage-prune` scheduled task cleans up expired dedup/delivery/message-tracking rows - Signature verification: Web Crypto API (HMAC-SHA256 for GitHub/Gitea, Ed25519 for Discord, timing-safe secret-token compare for Telegram) - Webhook providers: pluggable forge adapters under `server/lib/providers/` (github, gitea) — each verifies its own signature format and normalizes its payload to a GitHub-shaped `WebhookEvent`; a `custom` provider accepts arbitrary signed JSON posts (`X-WebHooker-Signature`) as `custom` events; GitLab etc. can be added later -- Per-group webhook ingress: optional `POST /webhook/{groupId}` with a per-group secret in KV (`tenant:{groupId}`) — Gitea/classic-GitHub/custom webhooks are verified against the group's secret instead of the operator's global ones; only that group's routes fire. The legacy `POST /webhook` (global secrets, all routes) stays untouched -- GitHub App tenant isolation: `Group.installationId` binds a group to one GitHub App installation; events whose `payload.installation.id` differs are rejected at dispatch (hard isolation on top of the optional `owners` list). The App's Setup URL points at `GET /auth/github/install`, which renders a choice page (create `inst-{installationId}` or bind to a group the signed-in user owns, verified by role); `POST /auth/github/install/bind` performs the provisioning. `installation.created` webhook events auto-provision as a fallback (create `inst-{installationId}` or bind existing groups whose `owners` match the installing account) +- Per-group webhook ingress: optional `POST /webhook/{groupId}` with a per-group secret stored in the group's D1 row (rotated/removed by CAS with an owner re-check; legacy KV `tenant:*` read as a fallback) — Gitea/classic-GitHub/custom webhooks are verified against the group's secret instead of the operator's global ones; only that group's routes fire. Tenant-signed payloads lose their `installationId` (no App-token enrichment, no installation provisioning; the installation check is skipped for the group's own ingress). The legacy `POST /webhook` (global secrets, all routes except isolated groups without a bound source) stays as before +- GitHub App tenant isolation: `Group.installationId` binds a group to one GitHub App installation; events whose `payload.installation.id` differs are rejected at dispatch (hard isolation on top of the optional `owners` list). The App's Setup URL points at `GET /auth/github/install`, which renders a choice page (create `inst-{installationId}` or bind to a group the signed-in user owns, verified by role); `POST /auth/github/install/bind` performs the provisioning. Both first verify with the user's own OAuth token (scopes `repo read:org`) that they manage the installation (`userManagesInstallation`: the installation's User account, or an active admin of its organization — mere access is not enough; needs `GITHUB_APP_ID`/`GITHUB_PRIVATE_KEY`; super admins bypass); self-signup only makes the installer owner of the dedicated `inst-{id}` group while it is bound to that installation and has no members. Ids `inst-*` / `u-*` are reserved for super admins, and an unbound existing `inst-*` group is never auto-bound. `installation.created` webhook events (global endpoint only) auto-provision as a fallback (create `inst-{installationId}` or bind existing groups whose `owners` match the installing account) - GitHub OAuth: octokit (token is stored hashed for reverse lookup) - Admin WebUI: `/admin` config console, OAuth-session protected via `ADMIN_USER_IDS` whitelist (local-only `DEBUG_MODE=true` bypasses authentication); left sidebar navigation (Overview / Groups / Send Logs / Audit / Stats) + topbar, deep-linkable routes (`/admin` overview · `/admin/groups` · `/admin/groups/{groupId}` a group's routes, members and webhook · `/admin/logs` · `/admin/audit` · `/admin/metrics`, parsed by `app/utils/admin-path.ts`; the admin page's `validate` 404s any other path) — the page has a fixed `key`, so one console instance (and its in-flight saves) persists across these URLs while each view refreshes its data when shown; every page is a history entry, so the browser's back button returns from a group to the list (no in-page back button) and leaving with an open editor or an unsaved route order asks first; route switches/actions lock while a route write is in flight; sign-out asks for confirmation (sidebar, or a top-right button on narrow screens); editor dialogs open without focusing a field (`useDialogFocus`); the overview dashboard (`AdminHome.vue`) aggregates delivery KPIs, a routes table and recent send logs; the stats page (`MetricsPanel.vue`) renders KPI cards plus per-platform/per-event ok/failed bar legends and a per-status breakdown, and filters by group (`?groupId=`); the route/group editors (`RouteEditor.vue` / `GroupEditor.vue`) are centered, scrollable `RcDialog` forms with an eyebrow header and sectioned bodies -- Access control: every group has role-based members (`owner` / `admin` / `viewer`); super admins bypass; legacy `adminIds` are read as owners (backward compatible); owners manage members + invites; `owners` field stays super-only -- Invites: single-use 7-day links (`invite:{token}`) for joining a group as admin/viewer; `ALLOW_SELF_SIGNUP=1` gives access-less users a personal group on first login (self-service SaaS entry) +- Access control: every group has role-based members (`owner` / `admin` / `viewer`); super admins bypass; legacy `adminIds` are read as owners (backward compatible); owners manage members + invites; the source-isolation fields `owners`, `installationId` and `isolated` stay super-only (an owner's write keeps the stored values). Self-signup personal groups are `isolated: true` — the global endpoint never dispatches to them until a super admin binds a source (`groupAcceptsGlobalIngress`); routes whose group is missing never fire +- Invites: single-use 7-day links (`invite:{token}`, KV TTL derived from `expiresAt`) for joining a group as admin/viewer; an invite belongs to the issuing group's instance; acceptance holds a short claim and commits the `invite-used` record in the same D1 transaction as the CAS member update, guarded so a used or revoked invite (revocations are D1 records) or one whose issuer is no longer an owner never admits anyone; `ALLOW_SELF_SIGNUP=1` gives access-less users an isolated personal group on first login (self-service SaaS entry) +- Group lifecycle (`server/lib/web/group-lifecycle.ts`): delete detaches send/audit history (`group_id = NULL`), drops message tracking, revokes invites + webhook secret and retires the id (`group-retired:{id}` in the idempotency store, 4 days); rename moves routes + fragments and send/audit history atomically and records a `group-alias:*` row (D1, 4 days; KV `group:renamed:*` without D1) in the same transaction so queued tenant deliveries of pre-instance messages follow; secrets, invites and message tracking stay with the instance; the old id is retired - Audit log: every admin operation (logins, group/route/member/invite changes) recorded in D1 `audit_logs`; pruned by the scheduled trigger after `AUDIT_RETENTION_DAYS` (default 90) - Group webhook log channel: optional `Group.logTarget` (Discord channel/thread or Telegram chat/topic) receives one summary message per webhook the group's routes dispatched (event, repo, delivery id, per route×target ✅/❌ outcome, green/red color); best-effort, not recorded in `send_logs` - Per-group forge branding: optional `Group.forgeSources` (a list of `{ host, type: "github" | "gitea", name? }` the group defines itself) labels each message's footer with the first entry whose type matches the event's provider and whose host matches the repository URL's hostname (GitHub matches `github.com`, so two Gitea instances can be `git1.example.com`/`git2.example.com`); the label is the entry's optional `name` (fallback: host); links are derived from the repo URL and footer icons use raster PNGs Discord renders (GitHub's `fluidicon.png`, Gitea's `/assets/img/favicon.png` — `.ico` favicons are silently ignored); Discord shows the footer icon + name, Telegram a linked name -- Delivery queue: when the `QUEUE` binding is present, webhook ingress enqueues one message per event (not per target) to a Cloudflare Queue (`webhooker-delivery`); a Nitro `cloudflare:queue` plugin consumes batches, dispatches, and retries retryable failures (5xx/network/429-exhaustion) with exponential backoff (5s/30s/2m/10m) up to the queue `max_retries`, after which the DLQ (`webhooker-delivery-dlq`) marks the delivery dead. Oversized payloads (>~100 KB) are parked in R2 (`webhooks/YYYY/MM/DD/*.json`, falling back to KV `queue:payload:*` without the R2 binding) and resolved by the consumer; delivery state is tracked as the D1 `delivery_state` table (KV `delivery-state:*` fallback). Without the `QUEUE` binding, dispatch stays inline (existing behavior) -- Storage quotas: the design avoids KV write pressure (Workers Free KV allows ~1,000 writes/day) by keeping high-frequency ephemeral writes in D1 instead — webhook dedup (`dedup_keys`, atomic `ON CONFLICT` UPSERT), delivery state and message tracking all use D1 rows via `canUseD1` (prepare+batch probe) with automatic KV fallback when D1 is unavailable or unmigrated. D1 Free allows 100,000 rows written/day, so the per-event KV write cost drops to ~0; config stays cached in memory + KV with D1 authoritative +- Delivery queue: when the `QUEUE` binding is present, webhook ingress enqueues one message per event (not per target) to a Cloudflare Queue (`webhooker-delivery`) and answers `200` only after the send succeeded (`503` + dedup claim released otherwise); a Nitro `cloudflare:queue` plugin consumes batches, takes each delivery with a 5-minute lease (`acquireDelivery`, conditional UPSERT on `delivery_state`), dispatches, and retries when any target failed transiently (5xx/429/network/rate-limit/`FEISHU_TOKEN`, a crashed route, an unreadable ledger, a busy update lock) with exponential backoff (5s/30s/2m/10m) up to the queue `max_retries`, after which the DLQ (`webhooker-delivery-dlq`) marks the delivery dead. Targets that succeeded or failed permanently are recorded (`delivery-target:*` in the idempotency store, keyed by the group's instance, written right after each target and read with one range query) and skipped on retries, so retries never duplicate and one bad target never cancels the others; a permanent failure is remembered (`delivery-failed:*`) so the delivery ends `failed`, not `delivered`. A message whose storage calls fail is retried on its own (the batch is never failed whole); pre-upgrade `delivery-state:{provider}:{groupId}:*` rows are honored only for the group that owned the id (`legacyId`), never for the global endpoint. The consumer runs `max_batch_size: 3` with one per-batch budget (12 minutes, 20 sends — the D1 query / subrequest limits are per invocation): a message left unstarted is retried (copied instead on its last retry if it was never started), and one that made progress continues in a new message. Oversized payloads (queue message over 120,000 UTF-8 bytes) are parked in R2 (`webhooks/YYYY/MM/DD/*.json`, falling back to KV `queue:payload:*` without the R2 binding, 4-day TTL; kept when the queue send fails, since the queue may have accepted it; orphans pruned by `storage-prune`) and resolved by the consumer — a missing/corrupt parked payload never dispatches `{}`: the delivery stays `retrying` with a `PAYLOAD_MISSING` send-log entry asking for a redelivery; delivery state is tracked as the D1 `delivery_state` table (KV `delivery-state:*` fallback). Without the `QUEUE` binding, dispatch stays inline (existing behavior) +- Storage quotas: the design avoids KV write pressure (Workers Free KV allows ~1,000 writes/day) by keeping high-frequency ephemeral writes in D1 instead — webhook dedup and every other one-time claim (`dedup_keys`, atomic `ON CONFLICT` UPSERT via `idempotencyStore` with `has`/`claim`/`release`), delivery state and message tracking all use D1 rows via `canUseD1` (prepare+batch probe) with automatic KV fallback when D1 is unavailable or a table is missing (`isMissingTable`, unmigrated); other D1 errors propagate so one claim never splits across D1 and KV. D1 Free allows 100,000 rows written/day, so the per-event KV write cost drops to ~0; config is cached in memory only, with D1 authoritative. In-flight work shared across concurrent requests (setup, fallback marker, Feishu / App tokens, translations) goes through `sharedFlight`, which waits on another request's run only for a bounded time (a canceled request's promise may never settle) - Filter DSL: a route's flat `filters` can be replaced by a nested `ast` (`all`/`any`/`not` nodes, `server/lib/events/filter-ast.ts`) that takes precedence when present; the `field` filter type reads any payload value by JSONPath (`path`, arrays expand so any element matches) and 12 comparison operators (`op`: `eq`/`ne`/`contains`/`startsWith`/`endsWith`/`regex`/`gt`/`gte`/`lt`/`lte`/`in`/`exists`) extend the legacy glob/regex/case-insensitive-exact semantics; named filter fragments (D1 `d1_fragments` via `server/lib/fragments.ts`, per-group) are editor-side templates inlined into the route's `ast` on insert — the matcher never resolves fragment references; the route editor (`RouteEditor.vue`) exposes a recursive AST builder (`FilterNodeEditor.vue`: plain selects for filter type/operator, an all/any segmented toggle, NOT toggle + delete pinned per condition), searchable multi-value / path inputs (`FilterCombobox.vue`) fed by a bilingual suggestion catalog (`app/utils/filter-catalog.ts`: GitHub events/actions from `@octokit/webhooks` plus Gitea extras, common payload paths and their known values, English keys searchable in either language) where custom values are allowed but rejected when the matcher could never match them (`validateFilterValue`), a live explanation and a stateless `POST /admin/api/test-match` dry-run - Local dev: wrangler + Miniflare @@ -41,35 +42,42 @@ app/ # Vue 3 UI (Nuxt app dir) │ # AST builder + FilterCombobox searchable inputs), GroupEditor, MembersPanel, WebhookPanel, │ # ConfirmDialog (app-wide destructive-action confirm), SendLogs, AuditLog, MetricsPanel, │ # AppToasts, LegalLayout -├── composables/ # useI18n, useToasts, useConfirm (promise-based confirm), useBackdropGuard (editor dialogs ignore -│ # backdrop clicks), useDialogFocus (editor dialogs open without focusing a field), useGroups, useGroupRoutes, useFilterNode (AST helpers), useFragments, useLogs, +├── composables/ # useApi (ApiError + isConflict for 409s), useI18n, useToasts, useConfirm (promise-based confirm), useBackdropGuard (editor dialogs ignore +│ # backdrop clicks), useDialogFocus (editor dialogs open without focusing a field), useGroups (per-group create/update/remove), +│ # useGroupRoutes + useFragments (send the loaded revision, reload on 409), useFilterNode (AST helpers), useLogs, │ # useAudit, useInvites, useWebhook ├── types.ts # shared client types (Route, Group, Filter, ...) └── utils/ # legal.ts (terms/privacy HTML bodies, zh/en), format.ts, admin-path.ts (console URL parsing), # filter-catalog.ts (bilingual filter suggestions + value/path validators) server/ # Nitro server ├── routes/ # H3 handlers: /health, /webhook[/:groupId], /discord/interactions, /telegram/webhook, -│ # /auth/github*, /admin/{login,logout,invite,api/**}, /api/{comment,merge,close,react,richheader} +│ # /auth/github* (incl. POST /auth/github/link — bot account-link confirmation), +│ # /admin/{login,logout,invite,api/**}, /api/{comment,merge,close,react,richheader} +├── middleware/ # config-store (binds the D1 config store on every request) ├── tasks/ # scheduled (cron */5): discord-sync, telegram-sync, audit-prune, storage-prune ├── plugins/ # Nitro plugins: queue-consumer (hooks cloudflare:queue → handleQueueBatch) ├── error-handler.ts # JSON error handler └── lib/ ├── types.ts # Env, Config, Route, Filter, Group, WebhookEvent, NeutralMessage - ├── config.ts # loadRoutes/saveRoutes (delegates to D1 ConfigStore when available, else KV config:routes), loadConfig from env - ├── fragments.ts # named filter fragments (loadFragments/saveFragments — D1 d1_fragments, no KV) + ├── config.ts # getConfigStore/initConfigStore (D1 store, else KV-only), loadRoutes/saveRoutes/saveGroupRoutes, loadConfig (fresh object per call) + ├── fragments.ts # named filter fragments (loadFragments(groupId?)/saveGroupFragments — D1 d1_fragments, no KV; read errors propagate) ├── config/ # config schema + migration + validation │ └── schema.ts # CONFIG_SCHEMA_VERSION, valibot route/group/filter schemas, migrateRoutes/Groups, validateRoutes/Groups (non-destructive), explainRoute ├── cf.ts # cfEnv(event) — env bindings from event.context.cloudflare ├── http.ts # shared HTTP helpers - ├── webhook.ts # processWebhook/handleWebhook: tenant lookup, provider detect/verify/parse, dedup (D1/KV), enqueue (or inline dispatch) + ├── webhook.ts # processWebhook/handleWebhook: tenant lookup, provider detect/verify/parse, tenant installation strip, dedup (D1/KV), awaited enqueue (503 + claim release on failure) or inline dispatch ├── queue/ # Cloudflare Queue delivery pipeline - │ ├── delivery.ts # DeliveryMessage, enqueueWebhook, retry backoff, delivery-state (D1 w/ KV fallback), payload-overflow parking (R2 → KV) - │ └── consumer.ts # handleQueueBatch: resolve payload → dispatch → classify → ack/retry/DLQ + │ ├── delivery.ts # DeliveryMessage, enqueueWebhook (byte-size check, parked-payload cleanup), classifyDelivery/isRetryableFailure, retry backoff, + │ │ # acquireDelivery lease, delivery-state (D1 w/ KV fallback), per-target done ledger, payload parking (R2 → KV), PayloadUnavailableError + │ └── consumer.ts # handleQueueBatch: lease → resolve payload → dispatch (tenant alias, skipTarget on retries) → classify → ack/retry/DLQ ├── core/ - │ └── dispatch.ts # Platform-neutral dispatch: match routes → formatEvent → driver.send/edit (batched recordSend + group filter + per-group webhook log) + │ └── dispatch.ts # Platform-neutral dispatch: source checks (tenant/isolation/installation/owners/providers) → match routes → formatEvent (FORMAT + │ # failures recorded) → driver.send or deliverUpdate (locked, versioned, forge-origin-keyed edit chains) → batched recordSend + group log ├── events/ │ ├── match.ts # matchRoute, eventOwners; unified pattern syntax (*/? globs + //-wrapped regex) - │ └── filter-ast.ts # FilterNode evaluator (all/any/not + field JSONPath + 12 ops), containsKeyword, explainFilter/explainFilterNode; pattern helpers (regex/glob compile) + │ └── filter-ast.ts # FilterNode evaluator (all/any/not + field JSONPath + 12 ops), containsKeyword, explainFilter/explainFilterNode; pattern helpers + │ # (globs and user regexes compiled with RE2 via `re2js` — linear-time matching, no ReDoS; bounded cache with separate + │ # key spaces; regexSafetyIssue rejects patterns RE2 cannot compile, e.g. backreferences/lookaround) ├── providers/ # Forge webhook providers (verify + parse/normalize to GitHub-shaped events) │ ├── types.ts # Provider interface (matches/verify/parse) │ ├── hmac.ts # HMAC-SHA256 + timing-safe compare helpers @@ -100,50 +108,61 @@ server/ # Nitro server │ │ ├── interactions.ts # Ed25519 verify + interaction handlers (/gh, buttons, modals) │ │ └── commands.ts # APP_COMMANDS + registerGlobalCommands/syncGuildCommands/syncCommands │ └── telegram/ - │ ├── index.ts # TelegramDriver: send/edit → renderNeutralMessage + rest.sendMessage (avatar rich-header card) + │ ├── index.ts # TelegramDriver: send/edit → renderNeutralMessage + rest.sendMessage/editMessageText (always text messages; + │ │ # avatar as link preview: rich-header card, or the avatar image without a host) │ ├── render.ts # renderNeutralMessage: NeutralMessage → Telegram HTML (parse_mode HTML) │ ├── rest.ts # Telegram Bot API sendMessage/sendPhoto/editMessage* (chat_id + message_thread_id), retry - │ ├── updates.ts # POST /telegram/webhook: secret-token verify + handleTelegramUpdate - │ └── commands.ts # Telegram /gh login|logout|comment|merge|close + reply-message parsing + syncTelegramWebhook + │ ├── updates.ts # POST /telegram/webhook: secret-token verify (always) + handleTelegramUpdate + │ ├── secret.ts # telegramWebhookSecret: TELEGRAM_WEBHOOK_SECRET or a secret derived from TELEGRAM_TOKEN + │ └── commands.ts # Telegram /gh login (private chat only)|logout|comment|merge|close + reply-message parsing + update_id dedup + syncTelegramWebhook │ └── feishu/ │ ├── index.ts # FeishuDriver: send/edit → renderNeutralMessage + rest.sendMessage/updateMessage │ ├── render.ts # renderNeutralMessage: NeutralMessage → Feishu interactive card -│ ├── rest.ts # getTenantAccessToken (KV cache) + sendMessage/updateMessage/sendText/updateCard, retry -│ └── updates.ts # X-Lark-Signature verify + url_verification + /gh commands + card.action.trigger buttons +│ ├── rest.ts # getTenantAccessToken (memory + KV cache per app, refused tokens dropped) + sendMessage/updateMessage/sendText/updateCard, retry +│ └── updates.ts # FEISHU_ENCRYPT_KEY signature verify + decrypt / FEISHU_VERIFICATION_TOKEN check (refuses all without either) + url_verification +│ # + event_id dedup + /gh commands (login p2p only) + card.action.trigger buttons ├── github/ - │ ├── oauth.ts # OAuth URL, callback token exchange, getUserOctokit, comment/getComment/editComment/deleteComment/merge/close actions - │ ├── app-token.ts # createAppJwt-backed getInstallationToken + GITHUB_API_VERSION (shared App auth) + │ ├── oauth.ts # OAuth URL, callback token exchange, getUserOctokit, getInstallationDetails/userManagesInstallation, comment/getComment/editComment/deleteComment/merge/close actions + │ ├── app-token.ts # createAppJwt-backed getInstallationToken + GITHUB_API_VERSION (shared App auth) + githubApiUrl (only https://api.github.com/repos/* is fetched, no redirects) │ ├── check-run.ts # getCheckSuiteBuildLogUrl: Check-suite build-log enrichment (installation token) │ ├── workflow-jobs.ts # getWorkflowRunJobs: fetch a workflow_run's jobs via jobs_url (App token or unauthenticated) - │ └── store.ts # KV token CRUD + D1 discord-link/telegram-link mapping (was token-store.ts) + │ └── store.ts # KV token CRUD (rotation drops the old reverse key; findUserIdByToken checks the current stored token) + D1 discord/telegram/feishu links ├── web/ # HTTP UI/API logic (called from server/routes) - │ ├── oauth.ts # handleOAuthStart/Callback, install page + bind, personal-group self-signup + │ ├── oauth.ts # handleOAuthStart/Callback (state cookie for browser logins), bot-link confirmation page + handleLinkConfirm, install page + bind + │ │ # (installation access verified), isolated personal-group self-signup, token delete (own token or super admin) │ ├── actions.ts # POST /api/comment|merge|close|react (Bearer token auth via shared middleware) - │ ├── admin.ts # adminLogin/Logout, adminApi* (routes|groups|me|logs|invites|audit|webhook|metrics|delivery) + │ ├── admin.ts # adminLogin/Logout, adminApi* (routes|groups|me|logs|invites|audit|webhook|metrics|delivery), per-group create/update/delete, + │ │ # revisions + 409 conflicts, super-only isolation fields, SQL-scoped logs/audit/metrics │ ├── auth.ts # Shared auth middleware + guards: requireAnyAccess, requireGroup(Role), bearerUserId, clientIp │ ├── invites.ts # Invite CRUD (KV invite:{token}, 7d TTL) + acceptInvite (join group as admin/viewer) │ ├── session.ts # Session CRUD (KV session:{id}), isAdminUser, cookie helpers - │ ├── groups.ts # Group CRUD (config:groups), member roles (normalizeGroupMembers/memberRole), resolveScope + role helpers (roleAt/canEditRoutes/canEditGroup) - │ ├── tenants.ts # Per-group webhook secret CRUD (KV tenant:{groupId}, 32-byte random hex) + │ ├── groups.ts # Group CRUD wrappers (create/update CAS/delete/rename via the config store), member roles (normalizeGroupMembers/memberRole), + │ │ # source checks (groupAccepts*), ensureInstallationGroup (lock-free), resolveScope + role helpers (roleAt/canEditRoutes/canEditGroup) + │ ├── group-lifecycle.ts # retire/alias group ids, move/detach group history on rename/delete + │ ├── tenants.ts # Per-group webhook secret (32-byte random hex in the D1 group row, CAS + owner re-check; legacy KV tenant:* read fallback) │ └── richheader.ts # GET /api/richheader: Open Graph page for Telegram avatar link-preview card ├── observability/ # delivery metrics aggregation │ └── metrics.ts # DeliveryMetrics + getDeliveryMetrics (SQL GROUP BY over send_logs in one D1 batch: totals, per platform/event/status, duration, attempts, recent failures) - ├── storage/ # storage primitives: canUseD1 probe, D1 config store (L1 memory → L2 KV cache → L3 D1), R2 payload store + ├── storage/ # storage primitives: canUseD1 probe, D1 config store (memory cache ≤60 s → D1), R2 payload store │ ├── d1.ts # canUseD1(db) — prepare+batch probe for D1 availability (falls back to KV) - │ ├── config-store.ts # ConfigStore (loadRoutes/saveRoutes/loadGroups/saveGroups/invalidateCache) backed by d1_groups/d1_routes (per-group `position` order when migrated), seeds from KV + syncs cache + │ ├── config-store.ts # ConfigStore: d1ConfigStore (per-group writes, CAS updateGroup, atomic rename/delete, one-time KV seed, one-time identity migration, unreadable-row isolation) + │ │ # and kvConfigStore (no D1); `position` order when migrated; ConfigConflictError │ └── payload.ts # PayloadStore backed by R2 (webhooks/YYYY/MM/DD/*.json), no-binding stub throws └── lib/ # shared infra ├── i18n.ts # loadTranslations (KV i18n:{lang} overrides), t() with param interpolation - ├── idempotency.ts # IdempotencyStore interface + kvIdempotencyStore/d1IdempotencyStore (delivery dedup via claim/has, D1 atomic UPSERT w/ KV fallback) + deliveryKey + ├── idempotency.ts # IdempotencyStore (has/claim/release) + kv/d1 stores (D1 atomic UPSERT, KV fallback only for a missing table) + tenantScope (`@global`) + deliveryKey ├── correlation.ts # newCorrelationId() — per-request/delivery correlation id for logs + responses - ├── message-tracker.ts # MessageTracker interface + kvMessageTracker/d1MessageTracker (msg:{eventId}:{targetId} for workflow_run/check_run edits) + ├── shared-flight.ts # sharedFlight: in-flight work shared by concurrent requests, waited on for a bounded time + ├── message-tracker.ts # MessageTracker (versioned { messageId, version } entries, legacy bare ids read as version 0) + kv/d1 trackers with KV fallback ├── send-log.ts # SendRecord, recordSend/getSendLog/getSendLogById/getSendLogByDelivery/getFailedSendLog (D1 send_logs) ├── send-log-batch.ts # recordSendBatch (D1 batch INSERT for send_logs) ├── audit.ts # recordAudit/getAuditLog/pruneAuditLogs (D1 audit_logs, best-effort writes) ├── log.ts # JSON console logger (info/warn/error/fatal) └── locales/ # en.ts, zh.ts translation dictionaries -tests/ # bun test unit tests (webhook, formatter, discord, telegram, admin, groups, invites, audit, send-log, token-store, ...) +tests/ # bun test unit tests (webhook, formatter, discord, telegram, admin, groups, invites, audit, send-log, token-store, + # data-integrity, ingress-dispatch, discord-interactions, telegram-webhook, ...) +tests/d1-sqlite.ts # in-memory SQLite behind the D1 API (all migrations applied, FKs on) + a Map-backed KV, for real-SQL tests tests/fixtures/ # provider payload fixtures (github/gitea/custom) feeding provider + formatter tests tests/__snapshots__/ # formatter snapshot golden files (toMatchSnapshot) ``` @@ -152,16 +171,17 @@ tests/__snapshots__/ # formatter snapshot golden files (toMatchSnapshot) - Verify GitHub webhook signatures (Web Crypto HMAC-SHA256, `X-Hub-Signature-256`) - Verify Gitea webhook signatures (Web Crypto HMAC-SHA256, plain hex `X-Gitea-Signature`) -- Verify custom webhook signatures (Web Crypto HMAC-SHA256, GitHub-style `sha256=` via `X-WebHooker-Signature`; optional replay protection via `X-WebHooker-Timestamp` + `X-WebHooker-Nonce` — signature over `{timestamp}.{nonce}.{body}`, ±5 min window, nonce dedup in KV) +- Verify custom webhook signatures (Web Crypto HMAC-SHA256, GitHub-style `sha256=` via `X-WebHooker-Signature`; optional replay protection via `X-WebHooker-Timestamp` + `X-WebHooker-Nonce` — signature over `{timestamp}.{nonce}.{body}`, ±5 min window, nonces claimed in the idempotency store; only JSON objects are accepted) - Normalize Gitea webhook payloads to a GitHub-shaped `WebhookEvent` (push `compare_url` → `compare`, `pull_request_comment` → `pull_request_review_comment`, ...) - Verify Discord interactions (Web Crypto Ed25519, X-Signature-Ed25519 over timestamp + body) -- Verify Telegram webhook calls (X-Telegram-Bot-Api-Secret-Token when configured) +- Verify Telegram webhook calls (X-Telegram-Bot-Api-Secret-Token always: `TELEGRAM_WEBHOOK_SECRET` or a secret derived from `TELEGRAM_TOKEN`; closed without a token) +- Verify Feishu callbacks (`X-Lark-Signature` = SHA-256 of timestamp + nonce + `FEISHU_ENCRYPT_KEY` + body, AES-256-CBC decryption of encrypted bodies, and/or the `FEISHU_VERIFICATION_TOKEN` in the payload; all callbacks are refused when neither is configured) - Filter events by: event type, repo name, actor, action, branch, keyword, or any payload field (`field` with a JSONPath `path` — arrays expand so any element matches) — every filter type supports `*`/`?` glob matching and `//`-wrapped regular expressions (case-insensitive) plus 12 comparison operators (`op`: `eq`/`ne`/`contains`/`startsWith`/`endsWith`/`regex`/`gt`/`gte`/`lt`/`lte`/`in`/`exists`) - Combine filters into an AST (`Route.ast`: `all`/`any`/`not` nodes) that overrides the flat `filters` AND-list; `explainRoute`/`explainFilterNode` render a human-readable description of the tree - Store named filter fragments (D1 `d1_fragments`, per group, `GET`/`PUT /admin/api/groups/:groupId/fragments`) as editor-side templates; inserting a fragment inlines its node into the route's `ast` — dispatch never resolves fragment references - Offer a stateless filter dry-run (`POST /admin/api/test-match`, body `{ node | filters, event?, payload }`) that evaluates in memory and returns `{ matched, explanation }` — no event payload is stored -- Validate route/group config against valibot schemas on load (non-destructive: invalid entries log a warning but still load); `CONFIG_SCHEMA_VERSION` marks the schema version and `migrateRoutes`/`migrateGroups` migrate legacy shapes (e.g. `target` → `targets`) -- Filter routes by group owner restriction (`Group.owners`), group source-platform restriction (`Group.providers`: github/gitea), GitHub App installation restriction (`Group.installationId`), and skip a group's fallback routes whenever a regular route of that group matched; stop evaluating the group's further routes when a matched route has `stop: true` (both scoped per group, so one group's routes never silence another's) +- Validate route/group config against valibot schemas (`validateRoutes`/`validateGroups` only log warnings; a D1 row that fails the shape checks on load is not loaded but reported as unreadable for its group); `CONFIG_SCHEMA_VERSION` marks the schema version and `migrateRoutes`/`migrateGroups` migrate legacy shapes (e.g. `target` → `targets`) +- Filter routes by group owner restriction (`Group.owners`), group source-platform restriction (`Group.providers`: github/gitea), GitHub App installation restriction (`Group.installationId`), isolation (`Group.isolated` groups take no global-endpoint events without a bound source) and group existence (routes of a missing group never fire) — all before any filter is evaluated — and skip a group's fallback routes whenever a regular route of that group matched; stop evaluating the group's further routes when a matched route has `stop: true` (both scoped per group, so one group's routes never silence another's) - Auto-provision GitHub App installs: the App's Setup URL flow (`/auth/github/install` choice page + `POST /auth/github/install/bind`, owner-role verified for existing groups) and the `installation.created` webhook fallback both create `inst-{installationId}` groups or bind existing groups - Enforce role-based access on every admin API: super admins bypass, `owner` manages the group (routes/members/invites/settings), `admin` edits routes, `viewer` is read-only; legacy `adminIds` groups resolve to `owner` members - Issue single-use 7-day group invite links (`invite:{token}`); accepting joins as admin/viewer (never owner); `ALLOW_SELF_SIGNUP=1` creates a deterministic personal group (`u-{userId}`) on first login @@ -170,17 +190,18 @@ tests/__snapshots__/ # formatter snapshot golden files (toMatchSnapshot) - Format 28 GitHub/Gitea event types plus `custom` webhooks as platform-neutral messages (Discord embeds + Telegram HTML) - Show the forge source (named per `Group.forgeSources` host entries) in the message footer when the group defines a host matching the event's repository - Route messages to Discord channels/threads and Telegram chats/topics via REST -- Edit already-sent messages in place for `workflow_run` / `check_run` progress (stable `updateKey`, `message_tracking` via D1 with KV `msg:*` fallback); `workflow_run` keys on the workflow id while `check_run` keys on check name + commit SHA (falling back to the run id) so providers that emit a fresh run id per phase — e.g. Cloudflare Pages — still edit the same message +- Edit already-sent messages in place for `workflow_run` / `check_run` progress (stable `updateKey`, chain keyed by group + route + forge origin (provider@host) + update key and target platform, `message_tracking` via D1 with KV `msg:*` fallback; older events never overwrite newer state, a transient edit failure is reported instead of posting a duplicate, a new message is sent only when the old one is gone); `workflow_run` keys on the run id while `check_run` keys on check suite (or app) + check name + commit SHA (falling back to the run id) so providers that emit a fresh run id per phase — e.g. Cloudflare Pages — still edit the same message - Record every dispatch attempt to D1 `send_logs` (route id, event, target, ok/error, duration, error code) -- Aggregate delivery metrics (`server/lib/observability/metrics.ts`) from `send_logs` — totals, ok/failed counts + failure rate, per-platform/per-event/per-status breakdowns, average duration and attempts, recent failures; `getDeliveryMetrics(db, groupId?)` sends all seven queries as one D1 `batch` (a single round trip and a consistent snapshot) and scopes every query by `group_id` when a group is passed -- Expose admin observability endpoints — `GET /admin/api/metrics?groupId=` (delivery metrics, optional group scope; recent failures group-scoped for non-super) and `GET /admin/api/delivery/:deliveryId` (all send-log attempts for one delivery, group-scoped) — through the `/admin/api/[...slug]` catch-all route (`server/routes/admin/api/[...slug].ts`) that wires every admin API handler to its method+path -- Serve a per-group webhook ingress (`POST /webhook/{groupId}`, per-group secret in KV `tenant:{groupId}`) for Gitea/classic-GitHub/custom senders; only that group's routes fire; dedup keys are provider- and tenant-scoped (`delivery:{provider}:{groupId}:{id}` via `idempotencyStore`, D1 `dedup_keys` with KV fallback) +- Aggregate delivery metrics (`server/lib/observability/metrics.ts`) from `send_logs` — totals, ok/failed counts + failure rate, per-platform/per-event/per-status breakdowns, average duration and attempts, recent failures; `getDeliveryMetrics(db, groupId?)` sends all five queries as one D1 `batch` (totals, average duration and attempts share one scan) (a single round trip and a consistent snapshot) and scopes every query by `group_id` when a group is passed +- Expose admin observability endpoints — `GET /admin/api/metrics?groupId=` (delivery metrics; every aggregate is limited in SQL to the reader's groups, `getDeliveryMetrics(db, groupId | groupIds)`) and `GET /admin/api/delivery/:deliveryId` (the delivery's attempts that belong to the reader's groups); logs and audit are scoped in SQL before the limit — through the `/admin/api/[...slug]` catch-all route (`server/routes/admin/api/[...slug].ts`) that wires every admin API handler to its method+path +- Serve a per-group webhook ingress (`POST /webhook/{groupId}`, per-group secret in the group's D1 row) for Gitea/classic-GitHub/custom senders; only that group's routes fire; dedup keys are provider- and tenant-scoped (`delivery:{provider}:@i:{instance}:{id}` via `idempotencyStore`, D1 `dedup_keys` with KV fallback) - Issue a per-request correlation id (`requestId`) in webhook responses and dispatch logs - When the `QUEUE` binding is present, enqueue each verified webhook as a single Queue message (`webhooker-delivery`) instead of dispatching inline; the consumer resolves the payload, re-scopes routes to the tenant group, and dispatches; retryable failures (5xx/network/429-exhaustion) are retried with exponential backoff (5s/30s/2m/10m) up to the queue `max_retries`, then the DLQ marks the delivery dead - Track delivery state in the D1 `delivery_state` table (`delivery-state:*` KV fallback: pending/processing/delivered/retrying/failed/dead) so redelivered messages are skipped idempotently; oversized payloads are parked in R2 (`webhooks/YYYY/MM/DD/*.json`, KV `queue:payload:*` fallback) and deleted after dispatch - Prune expired D1 rows via the scheduled `storage-prune` task (dedup keys past expiry, delivery state older than 7 days, message tracking older than 30 days; audits via `audit-prune`) - Send a per-event summary (event, repo, delivery id, per route×target ✅/❌ outcome) to the group's `logTarget` when configured -- Serve `/gh` slash commands + message context-menu commands + PR merge/close buttons + comment modals +- Serve `/gh` slash commands + message context-menu commands + PR merge/close buttons (github.com repositories only; deferred replies edited via `/webhooks/{application_id}/{token}/messages/@original`) + comment modals (comments over 4000 characters are edited on GitHub instead); every interaction id runs once +- Link chat accounts to GitHub only after the GitHub user confirms on a page naming both accounts (`POST /auth/github/link`); Telegram/Feishu `/gh login` answers only in private chats - Serve Telegram `/gh` commands (login/logout/comment/merge/close) via reply-message parsing - Serve Feishu `/gh` commands (login/logout/comment/merge/close) and card Merge/Close buttons via the `/feishu/webhook` endpoint (X-Lark-Signature verify + `url_verification` challenge) - Sync application commands from the scheduled trigger (global ~1h propagation + per-guild instant) @@ -202,7 +223,7 @@ tests/__snapshots__/ # formatter snapshot golden files (toMatchSnapshot) and inline comments use the same `{repo}{#number}: {title}` title as their parent object. - All event-specific emoji live in `server/lib/formatters/` (via the `emojiPrefix` helper), never in the locale files. Emoji is controlled per group through the `Group.emoji` toggle (default true); - `showEmoji=false` must strip every emoji from titles, descriptions, fields and links. + `showEmoji=false` must leave out every emoji the formatters add to titles, descriptions, fields and links (text taken from the payload keeps its own emoji). - Milestone progress bars (🟢🟡🟠⬜) are data visualization and are exempt from the emoji toggle. - `workflow_run` messages render a **Jobs** field as a Discord `diff` fenced code block (`server/lib/formatters/workflow.ts`): one line per job, `${marker} ${name} ${emoji}${status}` plus @@ -259,12 +280,13 @@ Rule: no functional change ships without its documentation; docs and code must n - **Production**: `wrangler secret put ` for each secret - **Routes**: config in D1 `d1_routes`/`d1_groups` (authoritative), seeded from legacy KV `config:routes`/`config:groups` on first load; cache-only KV keys - **KV namespace**: Required binding for token/state/session/cache storage (dedup/delivery-state/message-tracking fall back to KV when D1 is unavailable) -- **D1 database**: Binding `DB` (database `webhooker`, id `214a0104-3235-47c0-b7bf-ddda95f3c8ac`) for `send_logs` + `audit_logs` + `discord_links` + `telegram_links` + `d1_groups` + `d1_routes` + `d1_fragments` + `dedup_keys` + `delivery_state` + `message_tracking` tables +- **D1 database**: Binding `DB` (database `webhooker`; its id is set in `wrangler.jsonc`) for `send_logs` + `audit_logs` + `discord_links` + `telegram_links` + `d1_groups` + `d1_routes` + `d1_fragments` + `dedup_keys` + `delivery_state` + `message_tracking` tables - **Queue**: optional `QUEUE` producer binding plus consumers `webhooker-delivery` and its DLQ `webhooker-delivery-dlq` (declared in `wrangler.jsonc`); when absent, webhook dispatch stays inline - **R2**: optional `PAYLOAD` binding (bucket `webhooker-payloads`) for oversized queue payloads; without it, oversized payloads fall back to KV `queue:payload:*` - **Access control**: `ADMIN_USER_IDS` (super admins), `DEBUG_MODE=true` (local-only authentication bypass), `ALLOW_SELF_SIGNUP` (optional personal group on first login), `AUDIT_RETENTION_DAYS` (default 90) — all plain env vars, not secrets - **Discord**: `DISCORD_PUBLIC_KEY` (Interactions Endpoint signature verification, from Discord Developer Portal) and `DISCORD_APPLICATION_ID` (optional, auto-resolved via `GET /oauth2/applications/@me` when omitted) are required for interactions -- **Telegram**: `TELEGRAM_TOKEN` (Bot API token from BotFather) required for Telegram routes; `TELEGRAM_WEBHOOK_SECRET` (optional secret token for `POST /telegram/webhook` verification); avatars are sent as a link-preview card via the built-in `GET /api/richheader` (overridable with `TELEGRAM_RICH_HEADER_HOST`) +- **Telegram**: `TELEGRAM_TOKEN` (Bot API token from BotFather) required for Telegram routes; `TELEGRAM_WEBHOOK_SECRET` (optional secret token for `POST /telegram/webhook` verification — derived from `TELEGRAM_TOKEN` when unset, never disabled); avatars are sent as a link-preview card via the built-in `GET /api/richheader` (overridable with `TELEGRAM_RICH_HEADER_HOST`) +- **Feishu**: `FEISHU_APP_ID` / `FEISHU_APP_SECRET` for sending; inbound commands/buttons need `FEISHU_ENCRYPT_KEY` and/or `FEISHU_VERIFICATION_TOKEN` - **Webhook providers**: `GITEA_WEBHOOK_SECRET` (required to receive Gitea webhooks; Gitea signs `X-Gitea-Signature` with the hex HMAC-SHA256 of the body) ## Deployment @@ -280,19 +302,20 @@ bunx wrangler d1 create webhooker bunx wrangler queues create webhooker-delivery bunx wrangler queues create webhooker-delivery-dlq # Queues are declared in wrangler.jsonc (QUEUE binding); no env var needed -bun run db:migrate:prod # wrangler d1 migrations apply webhooker --remote (migrations/0001..0012) +bun run db:migrate:prod # wrangler d1 migrations apply webhooker --remote (migrations/0001..0013) # R2 bucket for oversized payloads (P0): bunx wrangler r2 bucket create webhooker-payloads -bunx wrangler deploy +bun run deploy # db:migrate:prod (pending migrations only) && wrangler deploy ``` Full list of secrets used: `GITHUB_WEBHOOK_SECRET`, `GITEA_WEBHOOK_SECRET`, `GITHUB_APP_ID`, `GITHUB_PRIVATE_KEY` (PKCS#8 PEM), `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`, `DISCORD_TOKEN`, -`DISCORD_PUBLIC_KEY`, `TELEGRAM_TOKEN`, `TELEGRAM_WEBHOOK_SECRET`, `ADMIN_USER_IDS`, `DEBUG_MODE`, +`DISCORD_PUBLIC_KEY`, `TELEGRAM_TOKEN`, `TELEGRAM_WEBHOOK_SECRET`, `FEISHU_APP_ID`, `FEISHU_APP_SECRET`, +`FEISHU_ENCRYPT_KEY`, `FEISHU_VERIFICATION_TOKEN`, `ADMIN_USER_IDS`, `DEBUG_MODE`, plus optional `BASE_URL`, `DISCORD_APPLICATION_ID`, `TELEGRAM_RICH_HEADER_HOST`, `NUXT_PUBLIC_DOCS_URL`, `NUXT_PUBLIC_REPO_URL`, `NUXT_PUBLIC_LEGAL_CONTACT`. See `.env.example` and `docs/guide/configuration.md`. ## Notes -- Commands sync from the scheduled trigger (`*/5 * * * *`): registered per-guild for instant availability and globally (24h dedup, ~1h propagation). +- Commands sync from the scheduled trigger (`*/5 * * * *`): registered per-guild for instant availability (at most 30 API calls per run, resuming from a saved cursor) and globally (~1h propagation); both re-register when the command hash changes, the global set at least daily. - The bot is always offline (no Discord Gateway); interactions arrive via the HTTP endpoint. diff --git a/README.md b/README.md index 86a182a..854b977 100644 --- a/README.md +++ b/README.md @@ -39,8 +39,8 @@ GitHub Webhook → Cloudflare Worker (Nuxt 4 / Nitro) - **Cloudflare Worker** — HTTP ingress, signature verification, routing, platform dispatch - **Interactions Endpoint** — HTTPS callback (no Discord Gateway connection, no Durable Object); the bot stays offline and commands are registered via the API -- **KV** — cache + ephemeral state: token storage (`token:{userId}`), OAuth state (`state:{hex}`), admin sessions (`session:{id}`), per-group webhook secrets (`tenant:{groupId}`), invites, config cache, delivery dedup/state/message-tracking **fallback** (`delivery:*`, `delivery-state:*`, `msg:*` used only when D1 is unavailable) and message-update locks (`msg:lock:*`) -- **D1** — source of truth for routes/groups (`d1_routes`/`d1_groups`), named filter fragments (`d1_fragments`), send logs (`send_logs`), audit logs (`audit_logs`), dedup (`dedup_keys`), delivery state (`delivery_state`), message-update tracking (`message_tracking`), Discord↔GitHub links (`discord_links`), Telegram↔GitHub links (`telegram_links`) +- **KV** — cache + ephemeral state: token storage (`token:{userId}`), OAuth state (`state:{hex}`), admin sessions (`session:{id}`), invites, delivery dedup/state/message-tracking **fallback** (`delivery:*`, `delivery-state:*`, `msg:*` used only when D1 is unavailable) and message-update locks (`msg:lock:*`) +- **D1** — source of truth for routes/groups (`d1_routes`/`d1_groups`, including each group's webhook secret), named filter fragments (`d1_fragments`), send logs (`send_logs`), audit logs (`audit_logs`), dedup (`dedup_keys`), delivery state (`delivery_state`), message-update tracking (`message_tracking`), Discord↔GitHub links (`discord_links`), Telegram↔GitHub links (`telegram_links`) - **Queue** — async delivery when `QUEUE` is bound: `webhooker-delivery` (exponential retry) + DLQ `webhooker-delivery-dlq`; oversized payloads parked in R2 (`PAYLOAD` binding, `webhooks/YYYY/MM/DD/*.json`, falling back to KV `queue:payload:*`) ## Quick Start @@ -48,42 +48,44 @@ GitHub Webhook → Cloudflare Worker (Nuxt 4 / Nitro) ```bash bun install # package manager is bun (lockfile: bun.lock) cp .env.example .dev.vars # Fill in secrets for local dev -bun run build # Production build first (wrangler dev serves the built worker) -bunx wrangler dev # Start local dev server +bun run db:migrate # Apply the D1 migrations to the local database (first run) +bun run cf:dev # Build, then serve the worker locally with wrangler dev ``` ## Configuration ### Secrets (`.dev.vars` for local, Worker Secrets for production) -| Variable | Description | -| --------------------------- | ---------------------------------------------------------------------------------------------- | -| `GITHUB_WEBHOOK_SECRET` | Webhook secret from GitHub | -| `GITEA_WEBHOOK_SECRET` | Webhook secret from Gitea (required only to receive Gitea webhooks) | -| `GITHUB_APP_ID` | GitHub App ID (used by the App install flow to resolve the installing account) | -| `GITHUB_PRIVATE_KEY` | App private key (PKCS#8 PEM; used by the App install flow; optional) | -| `GITHUB_CLIENT_ID` | OAuth client ID | -| `GITHUB_CLIENT_SECRET` | OAuth client secret | -| `DISCORD_TOKEN` | Bot token | -| `DISCORD_PUBLIC_KEY` | Discord application public key (from the Developer Portal) — required for interactions | -| `DISCORD_APPLICATION_ID` | Discord application id (optional; auto-resolved via `GET /oauth2/applications/@me` if omitted) | -| `TELEGRAM_TOKEN` | Telegram bot token (from BotFather) — required for Telegram routes | -| `TELEGRAM_WEBHOOK_SECRET` | Optional secret token for `POST /telegram/webhook` verification | -| `TELEGRAM_RICH_HEADER_HOST` | Optional base URL overriding the built-in `GET /api/richheader` for Telegram avatar cards | -| `FEISHU_APP_ID` | Feishu app ID (from the app Credentials page) — required for Feishu routes | -| `FEISHU_APP_SECRET` | Feishu app secret — required for Feishu routes | -| `BASE_URL` | Public URL for OAuth callbacks and the Telegram webhook sync | -| `ADMIN_USER_IDS` | Comma-separated GitHub user IDs (or logins) allowed to access `/admin` | -| `DEBUG_MODE` | Local development only: set to exact `true` to bypass all `/admin` authentication | -| `ALLOW_SELF_SIGNUP` | `1` to give access-less GitHub users a personal group on first login (default off) | -| `AUDIT_RETENTION_DAYS` | Audit-log retention in days for the scheduled cleanup (default 90) | -| `NUXT_PUBLIC_DOCS_URL` | Optional docs site URL used by the landing page | -| `NUXT_PUBLIC_REPO_URL` | Optional GitHub repo URL used by the landing page | -| `NUXT_PUBLIC_LEGAL_CONTACT` | Optional contact shown on `/terms` and `/privacy` | +| Variable | Description | +| --------------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| `GITHUB_WEBHOOK_SECRET` | Webhook secret from GitHub | +| `GITEA_WEBHOOK_SECRET` | Webhook secret from Gitea (required only to receive Gitea webhooks) | +| `GITHUB_APP_ID` | GitHub App ID — the App install flow needs it to verify the installer (otherwise only super admins can bind installations) | +| `GITHUB_PRIVATE_KEY` | App private key (PKCS#8 PEM) — needed together with `GITHUB_APP_ID`; also authenticates API enrichment | +| `GITHUB_CLIENT_ID` | OAuth client ID | +| `GITHUB_CLIENT_SECRET` | OAuth client secret | +| `DISCORD_TOKEN` | Bot token | +| `DISCORD_PUBLIC_KEY` | Discord application public key (from the Developer Portal) — required for interactions | +| `DISCORD_APPLICATION_ID` | Discord application id (optional; auto-resolved via `GET /oauth2/applications/@me` if omitted) | +| `TELEGRAM_TOKEN` | Telegram bot token (from BotFather) — required for Telegram routes | +| `TELEGRAM_WEBHOOK_SECRET` | Optional secret token for `POST /telegram/webhook` verification (derived from the token if unset) | +| `TELEGRAM_RICH_HEADER_HOST` | Optional base URL overriding the built-in `GET /api/richheader` for Telegram avatar cards | +| `FEISHU_APP_ID` | Feishu app ID (from the app Credentials page) — required for Feishu routes | +| `FEISHU_APP_SECRET` | Feishu app secret — required for Feishu routes | +| `FEISHU_ENCRYPT_KEY` | Feishu event Encrypt Key (signature check + decryption) — this or the token below is required for inbound Feishu commands | +| `FEISHU_VERIFICATION_TOKEN` | Feishu event Verification Token, checked on every inbound callback | +| `BASE_URL` | Public URL for OAuth callbacks and the Telegram webhook sync | +| `ADMIN_USER_IDS` | Comma-separated GitHub user IDs (or logins; numeric ids recommended) of super admins | +| `DEBUG_MODE` | Local development only: set to exact `true` to bypass all `/admin` authentication | +| `ALLOW_SELF_SIGNUP` | `1` to give access-less GitHub users a personal group on first login (default off) | +| `AUDIT_RETENTION_DAYS` | Audit-log retention in days for the scheduled cleanup (default 90) | +| `NUXT_PUBLIC_DOCS_URL` | Optional docs site URL used by the landing page | +| `NUXT_PUBLIC_REPO_URL` | Optional GitHub repo URL used by the landing page | +| `NUXT_PUBLIC_LEGAL_CONTACT` | Optional contact shown on `/terms` and `/privacy` | ### Routes -Routes are stored in D1 (`d1_routes`, seeded from legacy KV `config:routes` on first load). There are **no default routes** — every route (including its target) must be defined explicitly, either via the Web UI (`/admin`) or by storing a JSON array in D1. A route may carry multiple `targets`, so one rule can forward to several channels at once: +Routes are stored in D1 (`d1_routes`, seeded from legacy KV `config:routes` on first load). There are **no default routes** — every route (including its target) must be defined explicitly, via the Web UI (`/admin`) or the Admin API. A route may carry multiple `targets`, so one rule can forward to several channels at once: ```json [ @@ -103,7 +105,7 @@ Routes are stored in D1 (`d1_routes`, seeded from legacy KV `config:routes` on f ] ``` -`target.platform` selects the push target: `discord` (default), `telegram`, or `feishu`. Discord targets require `target.channelId` (optional `threadId` for a thread); Telegram targets require `target.chatId` (optional `topicId` for a topic); Feishu targets require `target.chatId` (optional `topicId` for a topic). A route may also set `stop: true` (skip later routes of its group). Routes belong to **groups** (D1 `d1_groups`, seeded from legacy KV `config:groups`) that scope admin access and can restrict which org/user events flow in. See the [Routes & Targets](https://webhooker.docs.elysola.org/guide/routes) and [Groups & Access Control](https://webhooker.docs.elysola.org/guide/groups) guides for the full schema. +`target.platform` selects the push target: `discord` (default), `telegram`, or `feishu`. Discord targets require `target.channelId` (optional `threadId` for a thread); Telegram targets require a numeric `target.chatId` (optional `topicId` for a topic); Feishu targets require `target.chatId` (`oc_…`; no topics). A route may also set `stop: true` (skip later routes of its group). Routes belong to **groups** (D1 `d1_groups`, seeded from legacy KV `config:groups`) that scope admin access and can restrict which org/user events flow in. See the [Routes & Targets](https://webhooker.docs.elysola.org/guide/routes) and [Groups & Access Control](https://webhooker.docs.elysola.org/guide/groups) guides for the full schema. ### Web UI (`/admin`) @@ -172,7 +174,7 @@ Every filter supports plain text, `*`/`?` globs, and `/regex/` patterns (case-in ### Bot Commands (comment on GitHub as yourself) -The bot registers native **slash** and **message context-menu** commands, synced by the scheduled trigger (every 5 minutes): per-guild for instant availability, and globally (24h dedup, ~1h propagation). After `/gh login` you can comment on issues/PRs as yourself, edit/delete your comments, and merge/close PRs via buttons — all replies are ephemeral and GitHub enforces permission. +The bot registers native **slash** and **message context-menu** commands, synced by the scheduled trigger (every 5 minutes): per-guild for instant availability, and globally (~1h propagation); both are re-registered when the command definitions change. After `/gh login` you can comment on issues/PRs as yourself, edit/delete your comments, and merge/close PRs via buttons — all replies are ephemeral and GitHub enforces permission. ``` /gh login /gh logout diff --git a/README.zh.md b/README.zh.md index 3c5b025..684c6d0 100644 --- a/README.zh.md +++ b/README.zh.md @@ -39,8 +39,8 @@ GitHub Webhook → Cloudflare Worker (Nuxt 4 / Nitro) - **Cloudflare Worker** — HTTP 入口、签名验证、路由分发 - **Interactions Endpoint** — HTTPS 回调(无 Discord Gateway 连接、无 Durable Object);bot 保持离线,命令通过 API 注册 -- **KV** — 缓存 + 临时状态:Token 存储(`token:{userId}`)、OAuth state(`state:{hex}`)、管理员会话(`session:{id}`)、分组级 webhook secret(`tenant:{groupId}`)、邀请、配置缓存、投递去重/投递状态/消息更新追踪的回退(`delivery:*`、`delivery-state:*`、`msg:*` 仅在 D1 不可用时使用)与消息更新锁(`msg:lock:*`) -- **D1** — 路由/分组(`d1_routes`/`d1_groups`)、命名过滤器片段(`d1_fragments`)、发送日志(`send_logs`)、审计日志(`audit_logs`)、去重(`dedup_keys`)、投递状态(`delivery_state`)、消息更新追踪(`message_tracking`)、Discord↔GitHub 绑定(`discord_links`)、Telegram↔GitHub 绑定(`telegram_links`) +- **KV** — 缓存 + 临时状态:Token 存储(`token:{userId}`)、OAuth state(`state:{hex}`)、管理员会话(`session:{id}`)、邀请、投递去重/投递状态/消息更新追踪的回退(`delivery:*`、`delivery-state:*`、`msg:*` 仅在 D1 不可用时使用)与消息更新锁(`msg:lock:*`) +- **D1** — 路由/分组(`d1_routes`/`d1_groups`,含各分组的 webhook 密钥)、命名过滤器片段(`d1_fragments`)、发送日志(`send_logs`)、审计日志(`audit_logs`)、去重(`dedup_keys`)、投递状态(`delivery_state`)、消息更新追踪(`message_tracking`)、Discord↔GitHub 绑定(`discord_links`)、Telegram↔GitHub 绑定(`telegram_links`) - **Queue** — 绑定 `QUEUE` 时异步投递:`webhooker-delivery`(指数退避重试)+ 死信队列 `webhooker-delivery-dlq`;超大负载暂存于 R2(`PAYLOAD` 绑定,`webhooks/YYYY/MM/DD/*.json`,回退 KV `queue:payload:*`) ## 快速开始 @@ -48,42 +48,44 @@ GitHub Webhook → Cloudflare Worker (Nuxt 4 / Nitro) ```bash bun install # 包管理器为 bun(锁文件:bun.lock) cp .env.example .dev.vars # 填写本地开发密钥 -bun run build # 先构建生产产物(wrangler dev 运行的是构建产物) -bunx wrangler dev # 启动本地开发服务器 +bun run db:migrate # 将 D1 迁移应用到本地数据库(首次运行) +bun run cf:dev # 先构建,再用 wrangler dev 在本地运行 worker ``` ## 配置 ### 密钥(本地用 `.dev.vars`,生产用 Worker Secrets) -| 变量 | 说明 | -| --------------------------- | --------------------------------------------------------------------------- | -| `GITHUB_WEBHOOK_SECRET` | GitHub webhook 密钥 | -| `GITEA_WEBHOOK_SECRET` | Gitea webhook 密钥(仅接收 Gitea webhook 时需要) | -| `GITHUB_APP_ID` | GitHub App ID(用于 App 安装流程解析安装所属账号) | -| `GITHUB_PRIVATE_KEY` | App 私钥(PKCS#8 PEM;用于 App 安装流程,可选) | -| `GITHUB_CLIENT_ID` | OAuth Client ID | -| `GITHUB_CLIENT_SECRET` | OAuth Client Secret | -| `DISCORD_TOKEN` | 机器人 token | -| `DISCORD_PUBLIC_KEY` | Discord 应用的公钥(开发者门户获取)—— 交互功能必需 | -| `DISCORD_APPLICATION_ID` | Discord 应用 ID(可选;省略时通过 `GET /oauth2/applications/@me` 自动获取) | -| `TELEGRAM_TOKEN` | Telegram Bot Token(BotFather 获取)—— Telegram 路由必需 | -| `TELEGRAM_WEBHOOK_SECRET` | 可选;`POST /telegram/webhook` 的验签密钥 | -| `TELEGRAM_RICH_HEADER_HOST` | 可选;覆盖内置 `GET /api/richheader` 的 Telegram 头像卡片地址 | -| `FEISHU_APP_ID` | 飞书应用 ID(应用凭证页获取)—— 飞书路由必需 | -| `FEISHU_APP_SECRET` | 飞书应用密钥 —— 飞书路由必需 | -| `BASE_URL` | 公网地址(用于 OAuth 回调与 Telegram webhook 同步) | -| `ADMIN_USER_IDS` | 允许访问 `/admin` 的 GitHub 用户 ID(或登录名),逗号分隔 | -| `DEBUG_MODE` | 仅本地开发:精确设为 `true` 时跳过全部 `/admin` 身份验证 | -| `ALLOW_SELF_SIGNUP` | 设为 `1` 时,无权限的 GitHub 用户首次登录自动获得个人分组(默认关闭) | -| `AUDIT_RETENTION_DAYS` | 定时清理时审计日志的保留天数(默认 90) | -| `NUXT_PUBLIC_DOCS_URL` | 可选;落地页使用的文档站点 URL | -| `NUXT_PUBLIC_REPO_URL` | 可选;落地页使用的 GitHub 仓库 URL | -| `NUXT_PUBLIC_LEGAL_CONTACT` | 可选;`/terms` 与 `/privacy` 页面展示的联系方式 | +| 变量 | 说明 | +| --------------------------- | ----------------------------------------------------------------------------- | +| `GITHUB_WEBHOOK_SECRET` | GitHub webhook 密钥 | +| `GITEA_WEBHOOK_SECRET` | Gitea webhook 密钥(仅接收 Gitea webhook 时需要) | +| `GITHUB_APP_ID` | GitHub App ID——App 安装流程需要它核实安装者(没有时只有超级管理员能绑定安装) | +| `GITHUB_PRIVATE_KEY` | App 私钥(PKCS#8 PEM)——需与 `GITHUB_APP_ID` 一起配置,也用于 API 增强鉴权 | +| `GITHUB_CLIENT_ID` | OAuth Client ID | +| `GITHUB_CLIENT_SECRET` | OAuth Client Secret | +| `DISCORD_TOKEN` | 机器人 token | +| `DISCORD_PUBLIC_KEY` | Discord 应用的公钥(开发者门户获取)—— 交互功能必需 | +| `DISCORD_APPLICATION_ID` | Discord 应用 ID(可选;省略时通过 `GET /oauth2/applications/@me` 自动获取) | +| `TELEGRAM_TOKEN` | Telegram Bot Token(BotFather 获取)—— Telegram 路由必需 | +| `TELEGRAM_WEBHOOK_SECRET` | 可选;`POST /telegram/webhook` 的验签密钥(未设置时由 token 派生) | +| `TELEGRAM_RICH_HEADER_HOST` | 可选;覆盖内置 `GET /api/richheader` 的 Telegram 头像卡片地址 | +| `FEISHU_APP_ID` | 飞书应用 ID(应用凭证页获取)—— 飞书路由必需 | +| `FEISHU_APP_SECRET` | 飞书应用密钥 —— 飞书路由必需 | +| `FEISHU_ENCRYPT_KEY` | 飞书事件 Encrypt Key(验签 + 解密)—— 飞书入站命令需要它或下面的 token | +| `FEISHU_VERIFICATION_TOKEN` | 飞书事件 Verification Token,每个入站回调都会校验 | +| `BASE_URL` | 公网地址(用于 OAuth 回调与 Telegram webhook 同步) | +| `ADMIN_USER_IDS` | 超级管理员的 GitHub 用户 ID(或登录名,建议写数字 ID),逗号分隔 | +| `DEBUG_MODE` | 仅本地开发:精确设为 `true` 时跳过全部 `/admin` 身份验证 | +| `ALLOW_SELF_SIGNUP` | 设为 `1` 时,无权限的 GitHub 用户首次登录自动获得个人分组(默认关闭) | +| `AUDIT_RETENTION_DAYS` | 定时清理时审计日志的保留天数(默认 90) | +| `NUXT_PUBLIC_DOCS_URL` | 可选;落地页使用的文档站点 URL | +| `NUXT_PUBLIC_REPO_URL` | 可选;落地页使用的 GitHub 仓库 URL | +| `NUXT_PUBLIC_LEGAL_CONTACT` | 可选;`/terms` 与 `/privacy` 页面展示的联系方式 | ### 路由配置 -路由存储在 D1(`d1_routes`,首次加载时从旧版 KV `config:routes` 同步)。**没有默认路由**——每条路由(包括目标)都必须显式定义,可通过 Web 控制台(`/admin`)或直接向 D1 存储 JSON 数组。一条路由可携带多个 `targets`,因此一个规则可以同时转发到多个频道: +路由存储在 D1(`d1_routes`,首次加载时从旧版 KV `config:routes` 同步)。**没有默认路由**——每条路由(包括目标)都必须显式定义,可通过 Web 控制台(`/admin`)或 Admin API 管理。一条路由可携带多个 `targets`,因此一个规则可以同时转发到多个频道: ```json [ @@ -103,7 +105,7 @@ bunx wrangler dev # 启动本地开发服务器 ] ``` -`target.platform` 选择推送目标:`discord`(默认)、`telegram` 或 `feishu`。Discord 目标需 `target.channelId`(可选 `threadId` 指向子区);Telegram 目标需 `target.chatId`(可选 `topicId` 指向话题);飞书目标需 `target.chatId`(可选 `topicId` 指向子话题)。路由还可设置 `stop: true`(跳过本分组的后续路由)。路由隶属于**分组**(D1 `d1_groups`,首次加载时从旧版 KV `config:groups` 同步),分组用于限定管理权限,并可限制哪些组织/用户的事件流入。完整模式见[路由与目标](https://webhooker.docs.elysola.org/zh/guide/routes)与[分组与访问控制](https://webhooker.docs.elysola.org/zh/guide/groups)指南。 +`target.platform` 选择推送目标:`discord`(默认)、`telegram` 或 `feishu`。Discord 目标需 `target.channelId`(可选 `threadId` 指向子区);Telegram 目标需数字形式的 `target.chatId`(可选 `topicId` 指向话题);飞书目标需 `target.chatId`(`oc_…`;没有话题)。路由还可设置 `stop: true`(跳过本分组的后续路由)。路由隶属于**分组**(D1 `d1_groups`,首次加载时从旧版 KV `config:groups` 同步),分组用于限定管理权限,并可限制哪些组织/用户的事件流入。完整模式见[路由与目标](https://webhooker.docs.elysola.org/zh/guide/routes)与[分组与访问控制](https://webhooker.docs.elysola.org/zh/guide/groups)指南。 ### Web 控制台(`/admin`) @@ -167,11 +169,12 @@ bunx wrangler dev # 启动本地开发服务器 - **GitHub App** — 创建应用、订阅事件、配置 OAuth 与 _Setup URL_(租户隔离):见 [GitHub App 配置](https://webhooker.docs.elysola.org/zh/guide/deployment#github-app-设置) - **Discord 机器人** — 创建机器人、以 `applications.commands` scope 邀请(组合权限整数 `274877910016`)、配置 Interactions Endpoint:见 [Discord Bot 配置](https://webhooker.docs.elysola.org/zh/guide/deployment#discord-bot-设置)。bot 从不连接 Discord Gateway,因此显示为**离线**——消息推送不受影响(始终走 REST)。 - **Telegram 机器人** — 用 [@BotFather](https://t.me/BotFather) 创建机器人,设置 `TELEGRAM_TOKEN`(可选 `TELEGRAM_WEBHOOK_SECRET`);webhook 由定时任务自动同步:见 [Telegram 机器人配置](https://webhooker.docs.elysola.org/zh/guide/deployment#telegram-机器人配置) +- **飞书机器人** — 创建自建应用、开启机器人能力、将机器人加入目标群聊,并设置 `FEISHU_APP_ID` + `FEISHU_APP_SECRET`:见[飞书机器人配置](https://webhooker.docs.elysola.org/zh/guide/deployment#飞书机器人配置) - **部署** — KV 命名空间、D1 数据库与迁移(含 0008 存储表)、可选 R2 Bucket 与 Queues、密钥、部署:见[部署指南](https://webhooker.docs.elysola.org/zh/guide/deployment) ### Bot 指令(以本人身份评论 GitHub) -bot 通过定时任务(每 5 分钟)同步注册原生的**斜杠命令**与**消息右键菜单命令**:按服务器注册以获得即时可用性,并全局注册(24h 去重,约 1 小时传播)。执行 `/gh login` 后即可评论 issue/PR、编辑/删除自己的评论,并通过按钮合并/关闭 PR——所有回复均为临时消息(仅你可见),权限交由 GitHub 判定。 +bot 通过定时任务(每 5 分钟)同步注册原生的**斜杠命令**与**消息右键菜单命令**:按服务器注册以获得即时可用性,并全局注册(约 1 小时传播);命令定义变化时两者都会重新注册。执行 `/gh login` 后即可评论 issue/PR、编辑/删除自己的评论,并通过按钮合并/关闭 PR——所有回复均为临时消息(仅你可见),权限交由 GitHub 判定。 ``` /gh login /gh logout diff --git a/app/components/AdminHome.vue b/app/components/AdminHome.vue index f1d9d85..15ff13c 100644 --- a/app/components/AdminHome.vue +++ b/app/components/AdminHome.vue @@ -8,6 +8,8 @@ const props = defineProps<{ groupsLoading: boolean; logsLoading: boolean; metricsLoading: boolean; + /** Why the latest load failed (what is shown may be empty or older). */ + error?: string; }>(); const { t } = useI18n(); @@ -72,6 +74,7 @@ function fmtTime(ts: number): string {