Build trigger engine and webhook source

Epic 5336 — Event-driven Task triggers. Upstream: PRD = Task 5337 `prd.md`; Architecture = Task 5338 `architecture.md` (artifact 3566). READ architecture §0, §2, §3.1–§3.10 and §11 IN FULL: they are the pinned contracts every later child builds against, and they are summarized here, not repeated. **SECURITY-SURFACE GATE**: stored secrets plus a public unauthenticated-until-verified route. ASK for approval before merge, per the charter. ## Original requirement (epic body, verbatim) > A `task_triggers` row beside the cron row. Sources: webhook (HMAC with a secret the agent never sees, templated prompt with `{{body.…}}`, `{{headers.…}}`, `{{query.…}}`) … Each trigger has: a preflight: a repo script or a cheap model call that returns run/skip or a list of items with ids, a dedup store keyed by item id, and a small persisted state document; limits: concurrency cap, queue depth, per-run budget, rate limit; health: Active / Degraded / Failed / Paused with reason, auto-pause after five consecutive failures or ten missed days, a deliveries log (dispatched, filtered, deduped, rate-capped, skipped) … Every trigger files a Task shape in its own project under the licence of the human who created it (R30) … a webhook can file an epic that is analysed, planned and verified by personas, with a human gate where policy says so. ## Requirements (PRD, verbatim) **R1** A `task_triggers` row sits beside `crons`. It follows the standard entity shape (id, name, slug, description, created_by/at, updated_by/at). It belongs to one org and one project. It carries a source kind, a source config, a target Task shape, preflight, limits, health and state. **R2** A trigger files **one Task shape** in **its own project**. The shape is any persona a cron may schedule, plus `epic` (never `advisor`). The trigger carries a prompt template, a Full Auto toggle and a hidden flag. The filed Task's `created_by_id` is the trigger's owner (its creator). The owner's `tasks:create` on the project is re-checked at **every** filing, as crons do today. **R3** Each webhook trigger has its own public URL, `/api/public/triggers/<opaque id>` (D2). It authenticates by **HMAC-SHA256** over the raw body with a per-trigger secret, or by an optional **bearer token** for senders that cannot sign. Jaah generates the secret and shows it **once** to the creator, who can rotate it. It is stored encrypted at rest. **No runner ever sees it**: it appears in no prompt, task body, environment variable, artifact, log line, deliveries log or digest. **R4** Every source renders a template from its event … Preflight items expose `{{item.id}}` and `{{item.…}}`. **R8 (seam only)** Preflight is optional. When set, it runs before any filing and returns either `{"run": true|false, "reason"?}` or `{"items": [{"id": "...", ...}]}`, plus an optional new state document. … `run:false` → `filtered`. `items` → one Task per item not already in the dedup store (R9). Each of those is subject to limits (R11). An item already seen → `deduped`. A preflight error, a timeout (A9: default 120 s), or output that is not valid JSON → a **failure** (counts toward R13). Its error text shows in the log, readable. **R9** Per trigger, keyed by item id (or by the event's natural id when there is no preflight …). An entry records when it was seen and the Task it filed. Entries are kept 90 days (A5). The owner can view entries and clear one. **R10** A small JSON document per trigger, capped at 64 KiB (A4). The preflight reads and replaces it. The owner can view and edit it in the console. Output over the cap is a preflight failure, and the old state is kept. **R11** Concurrency cap (default 1) … Queue depth (default 10) … dispatches automatically when a filed Task closes. Past the depth, it is logged `skipped` with reason "queue full". Rate limit (default 10 filings per rolling hour): above it, the delivery is logged `rate-capped` and dropped, not queued. **R12** Active / Degraded / Failed (stopped by the system) / Paused (stopped by a human, recording who and when). Every non-Active state shows a human-readable **reason**. While Failed or Paused, deliveries are acknowledged (2xx to the sender) and logged `skipped` "trigger paused". Resume is a human action, and it resets the failure counter. A trigger's failures are: preflight failure, template or filing error, and a filed Task ending `system_failed`. **R13 (consecutive-failure half)** Consecutive failures: default **5**; can be set to off (D9). … sends the trigger to **Failed** with the reason. **R14** One row per inbound delivery, or per preflight item. It records time, outcome, reason, item id and the filed Task link. … plus `failed` and `queued` (A1). The owner can filter the log by outcome. Bodies are stored redacted of auth headers, and truncated to a readable preview. **R18** Each trigger has a **Full Auto** toggle, default ON (D12). ON → the filed Task is `autonomous` and `unattended`, exactly as a cron filing is. OFF → it files with `autonomous=false`, `unattended=false`, so its analysis personas interview a human. ## Approach (architecture, settled) - **Schema (§3.1), one migration chain**: `task_triggers` (all §3.1 columns, PLUS `created_by_task_id` uuid/bigint FK tasks SET NULL for the advisor-verbs child), `trigger_deliveries` (incl. the partial unique index `uq_trigger_deliveries_one_preflighting`), `trigger_dedup`, `inbound_emails.trigger_routed_at` (for the Gmail child), `tasks.created_by_trigger_id` + `created_by_trigger_name`. Packaging decision (planner): the advisor and Gmail columns ship here so the later parallel wave carries only ONE migration (`trigger_notices`, owned by the health child). Do NOT create `trigger_notices`. - Modules: `trigger_config.py` (pydantic union of the §3.3 `source_config` shapes for all four kinds — the console child needs every kind savable), `trigger_template.py` (§3.8 grammar, refusals, caps), `trigger_secrets.py` (`open_secret` — the single plaintext seam, AES-256-GCM tier-1 as `credential_crypto`), `trigger_health.py` (§3.10: pure `trigger_health`, `apply_outcome` per §3.9 rows 1-5, and `on_failed`/`fold_filed_outcomes`/`fold_budget` as no-op stubs), `trigger_intake.py` (§3.6 processor, `PreflightInput/Result/Error/Unavailable`, `PREFLIGHT_RUNNERS` with `none` registered, shared `parse_preflight_output`, `PREFLIGHT_POOL`), `task_autonomy.py::apply_full_auto` (§3.7 — move the cron runner's `autonomous`/`unattended` writes here, BEHAVIOUR-PRESERVING with a regression test for Full Auto ON and OFF; reword `models.py` ~:2490 and `dispatch_compose` docstrings) and `_effective_unattended` beside the epic autonomy reader (`routes/tasks.py` ~:2850). - `trigger-intake` system activity (interval 60 s, `timeout_seconds=60`, `min_gap_seconds=2`), fired early after ingest commit via `cron_runtime.fire_now_activity`; `task_wake.wake_activities_for` adds it for `done` and `system_failed`. - Ingress `POST /api/public/triggers/{ingress_token}` per §3.4, in that exact check order (413 → 404 → 429 → 401 + one `skipped` row per bad request, never counted → 202 skipped when Paused/Failed → 202 `{delivery_id}`), stored event shape and natural dedup keys per §3.4. `webhook` source only here; GitHub scheme is a later child. - Console API `routes/triggers.py` per §3.5 (org admin, org-scoped; operator sees all). `HOOKS_PUBLIC_BASE_URL` read once in `app/config.py`; when empty, `webhook`/`github` creation → 409 "This control plane has no public hooks address". - Task read gains `created_by_trigger_id`, `created_by_trigger_name`, `trigger_budget_exceeded_usd`. Filing sets `source = task_source.trigger_source(kind)` (from the Foundation child's module). - Retention: deliveries 90 d (`event` purged at 30 d), dedup 90 d, via `cron_retention`. ## Acceptance (PRD, verbatim) - **AC-1.3** A user who is not an org admin cannot reach the create/edit controls or the API. → verify: … A direct `POST` to the create route returns 403. (API half here.) - **AC-2.1** A delivery that passes preflight files a Task in the trigger's project, with `created_by_id` = owner, the configured persona, and the rendered prompt as its body. → verify: post a signed test delivery to a dev webhook trigger. Open the filed Task and observe its project, persona, creator and body. - **AC-2.2** If the owner has lost `tasks:create` on the project, nothing is filed. The delivery is logged `skipped` with reason "owner lacks tasks:create", and the trigger goes **Failed** with that reason. → verify: revoke the owner's role on dev, post a delivery, and observe the log row and the health badge. - **AC-2.3** A trigger configured with persona `epic` files an epic that starts its pipeline: `business_analysis` → `architecture` → `plan` children appear and dispatch, the same as an epic started from the console. → verify: on dev, configure a webhook trigger with persona `epic`, post a delivery, and observe the epic's BA child become `active`. - **AC-2.4** A trigger can never file into another project or another org. → verify: an API call that names a foreign project on create or edit gets 4xx. A test asserts the filed `project_id` always equals the trigger's. - **AC-3.1** The URL is reachable from the **public internet on dev and prod** (D2). → verify: from a machine outside the tailnet, `curl` a signed request to the dev trigger URL and observe 2xx and a `dispatched` (or `filtered`) log row. - **AC-3.2** A missing or wrong signature or token gets 401. It is logged `skipped` with reason "invalid signature" and files nothing. It does **not** count as a trigger failure, so an attacker cannot pause the trigger (A6). → verify: `curl` with a wrong signature 6 times. Observe 401 each time, 6 skipped rows, and health still Active. - **AC-3.3** The secret never leaves the console's one-time reveal. → verify: grep a filed Task's body, its dispatch prompt and its runner transcript for the secret, and observe no match. A test asserts the trigger read API never returns it after creation. - **AC-3.4** Templates render `{{body.a.b}}` (JSON path, array index allowed), `{{headers.x}}` (case-insensitive) and `{{query.y}}`. A missing path renders as empty, and the delivery's log row names it (A7). The `authorization` header and the signature header can never be templated. Rendered values sit inside the untrusted-data fence of the filed Task (ADR-0027). → verify: post `{"issue":{"title":"Login broken"}}` with a template `Fix: {{body.issue.title}}`. Observe the filed body contains "Fix: Login broken" inside the fence. A template naming `{{headers.authorization}}` is refused at save. - **AC-3.5** Payload limits: a body over 1 MiB gets 413 and is logged skipped (A8). A per-trigger inbound request rate above the abuse cap gets 429. → verify: a test posts an oversize body and observes 413. - **AC-9.1** Posting the same item id twice files once. The second delivery logs `deduped`, naming the first Task. → verify: on dev. - **AC-9.2** Clearing an entry lets the item file again. → verify: on dev. - **AC-11.1** With cap 1 and depth 1, three rapid deliveries produce: one dispatched, one queued then dispatched after the first Task closes, and one `skipped` "queue full". → verify: on dev, then close the first Task and observe the queued one file. - **AC-11.2** With rate 2/hour, a third delivery in the hour logs `rate-capped`. → verify: on dev. - **AC-12.1** Each of the four states can be produced on dev and shows its badge and reason. (Here: Active, Degraded via a filing failure, Paused, Failed via lost authority — through the API; budget-Degraded is the health child's.) - **AC-12.2** Resume returns a Failed trigger to Active and a fresh delivery files. → verify: on dev. - **AC-18.1** On dev, a trigger with Full Auto OFF filing an epic produces a BA child that raises an AskUserQuestion round, visible in the console. → verify: observe `awaiting_response`. Out of scope: Do NOT build the console UI (Triggers console child), the GitHub/Slack/Gmail ingest (their children), the model or script preflight runners (their children; register only `none`), `trigger_notices` / notices / digest / budget folding / idle auto-pause (health child — ship the three stubs only), or advisor verbs. Do NOT change cron behaviour beyond the behaviour-preserving `apply_full_auto` move. Do NOT build R16 memory/last report (dropped). PRD §5 (verbatim, binds): No deferred source in this epic. No skill attachment. No budget enforcement — never stop, kill or fail a runner for budget. No GitHub reads that decide filing (ADR-0112). No public exposure beyond `/api/public/triggers/*` and `/api/public/email/gmail-push` on dev. Cron behaviour is unchanged. No new licence for any persona other than advisor. Slack: no Events API. Secrets: no webhook secret, bearer token or credential in any prompt, task body, log, digest, deliveries log or artifact. Done means: - Every §3.5 route exists and has a route test: list (with `health:{state,reason}`), create (secret returned once; foreign project → 404; Gmail outside operator org → 403; no hooks address → 409 for webhook/github), get (never the secret), patch (project only within org), delete, reveal-secret (200 once then 409), rotate-secret, pause/resume (resume clears paused_*, failed_*, consecutive_failures), test-render (files nothing), deliveries (outcome filter, paging), dedup list + delete, PUT state (>64 KiB → 422), unmatched (operator only). Non-admin POST → 403 (AC-1.3). - Unit tests: §3.4 check order; HMAC + bearer with `compare_digest`; `trigger_template` grammar + refusals + caps; §3.9 counter table row by row; `parse_preflight_output` accepts/rejects; the claim CAS (a stale worker writes nothing); dedup reservation rolled back for `rate_capped`/`queue full`; filed `project_id` always equals the trigger's (AC-2.4); the secret absent from the task read, preflight input and filed body (AC-3.3); cron filing `autonomous`/`unattended` unchanged for Full Auto ON and OFF. - On dev (after the hooks edge is applied), cite the observed row/state for each: AC-3.1 (curl from outside the tailnet to `https://hooks.dev.jaah.ai/api/public/triggers/<token>` → 202 + `dispatched`), AC-2.1, AC-2.2, AC-2.3 (BA child `active`), AC-3.2 (6× 401, 6 skipped rows, health Active), AC-3.4 ("Fix: Login broken" inside the fence), AC-9.1, AC-9.2, AC-11.1 (incl. the queued row filing after the first Task closes), AC-11.2, AC-12.1 (four states via API), AC-12.2, AC-18.1 (`awaiting_response`). Every log `reason` is readable human text, never a code. - `./dc check full` green.

Built the trigger engine, webhook ingress and /api/triggers console API (rev 0387). Full gate green; PR #4680 merged after security approval. Remainder: dev AC checks — the dev console isn't reachable from this node.