Verify dev and cut prod over

## Original request (verbatim) > Migrate the emails system from GMail API to AWS SES. Use all the best practices. The quality of communication is of highest priority. Spare no time or expense. > > [jaah.ai](http://jaah.ai) MX is currently at GMail and will remain so, but we can carve out whatever we need Epic context (BA PRD Task 9352, Architecture Task 9353) — settled: all system mail (send + receive) moves to SES on `mail.jaah.ai` (dev: `mail.dev.jaah.ai`, dev account, sends/receives real mail) with suppression, operator-only delivery log, RFC 8058 unsubscribe, DMARC monitoring, operator-approved template redesign; tenant Gmail (agent send, automation action, receive) is built in this epic behind one plan feature. All system AWS calls use the hand-signed SigV4 seam (`apps/api/app/aws/common.py` `signed_call`), no boto3. SES events and inbound mail are PULLED from SQS by system crons (precedent `apps/api/app/alarm_relay.py`, `apps/api/app/aws/sqs.py`); the only new public path is unsubscribe on the hooks edge. ## Scope of this child C11 Prod cutover (verify): after dev passes, confirm jaah-main production access, prod `terraform plan` clean, mail-tester 10/10 on prod, a cutover checklist (operator re-points triggers whose `to` names the old mailbox — AS5), hand promotion develop→master to the operator, then run AC3/AC6/AC7 on prod. Delivers AC19 and the prod halves of AC1/AC6. ## Requirements (verbatim from the PRD) - **AC19 Availability.** Every capability above is ON by default in the dev deployment (no flag left off), wired by Terraform + console settings, and reachable by the flows above on `https://dev.jaah.ai`; prod is cut over after dev passes AC1–AC17. AS5. Existing operator email triggers are **not auto-rewritten**; a trigger whose `to` names the old Gmail mailbox address must be re-pointed by the operator, and the runbook says so. AS10. Until SES production access is approved, the epic is not done. Operator decision I5: **Runner files; operator promotes** — C1 files production access; this child prepares/verifies the cutover; the operator promotes develop→master. ## Approach (from the settled architecture) - Risk 3: the release that removes the Gmail system path carries the prod Terraform; if prod SES access is not yet granted, prod mail stops. The checklist blocks promotion until `ProductionAccessEnabled` is true in jaah-main and the prod identity is `SUCCESS`. - Run, on dev, every live oracle the earlier children deferred on sandbox exit (AC2, AC3, AC10 full, AC11 substitute, AC13) once jaah-dev production access is granted; record results. - Cutover checklist in `docs/integrations/ses.md` (current-state, not a dated log). - Final AC5 grep (§9.9). Credentials & reach: | Oracle | Credential / identity / environment | Status | |---|---|---| | AC6, terraform plan, SES state, production-access filing (C1, C11) | `me/jaah/system/bootstrap-admin` (management 155534476372) assuming `OrganizationAccountAccessRole` in jaah-dev 905905585302 and jaah-main 366008913850 | **verified** — `credential_get` ok; read-only probe from this node: `sts get-caller-identity`, `sesv2 get-account`, `route53 list-hosted-zones` in both accounts answered | | SES production access (sandbox exit) — gates AC2, AC3, AC11, AC13 and C11 | AWS approval of `PutAccountDetails` in jaah-dev and jaah-main (filed by C1) | **deferred** — reason: AWS reviews it; both accounts probed `ProductionAccessEnabled=false`. Owner C1 files on day one; dependent oracles run after approval | | AC4 (local), AC7, AC9, AC10, AC13 (after sandbox exit), AC17(b) in the dev console | `me/jaah/debug/main/operator-login` (operator, `https://app.dev.jaah.ai`) | **verified** — `credential_get` ok; `https://app.dev.jaah.ai` answered 200 from this node | | C11 prod oracles: AC3/AC7 on prod (Unmatched emails, delivery log), prod verifier user, prod console knobs | `me/jaah/debug/main/operator-login` on `https://app.jaah.ai` (its description: works on dev and main, switches between operator and several tenants) | **listed, not probed by the planner** — `credential_list` shows it; C11 verifies sign-in on `https://app.jaah.ai` first; if it fails, those prod oracles are deferred and handed to the operator in the cutover checklist (ask once, header `Credential`) | | AC10 (non-operator 403), AC17 tenant side | `org/console/dev-orgb-login` (org B admin) | **verified** — `credential_get` ok (its recorded base_url is legacy; use `https://app.dev.jaah.ai`) | | AC2 deliverability | mail-tester.com, no account (free tier) | **verified (reach)** — `https://www.mail-tester.com` answered 301→site; the check runs after the SES production-access row | | AC1/AC11 Gmail UI checks (inbox placement, native Unsubscribe, Show original) | a Google login to admin@jaah.ai | **deferred** — operator: "Skip it" (I2) | | AC14/AC15/AC16/AC17(a) live (AC16 needs a connected Gmail too) | Jaah Google OAuth client (gmail.send) + a test Workspace SA with DWD | **deferred** — operator: "Build now, verify later" (I4) | | AC5, AC18, unit/integration suites | worktree only | **verified** — nothing beyond the worktree | Out of scope: - Do NOT promote develop→master yourself — human gate (operator promotes, I5). - Do NOT rewrite operator triggers (AS5); list them for the operator. - Do NOT build features; file a follow-up for any gap found. Epic out-of-scope (PRD §7, verbatim): - MUST NOT change the apex `jaah.ai` MX, SPF or `_dmarc.jaah.ai` record, nor any Workspace mailbox, alias or routing. - MUST NOT add SES Virtual Deliverability Manager, BIMI/VMC, or a dedicated IP / dedicated IP pool. - MUST NOT request Google's restricted `gmail.readonly` (or any restricted Gmail scope) via OAuth; no CASA assessment. - MUST NOT add open/click tracking pixels or link rewriting to system mail. - MUST NOT let tenant Gmail replace system mail for an org's members (Q9 option not chosen). - MUST NOT expose the delivery log or suppression list to tenant admins or users (operator only, Q7). - MUST NOT relax the verified-address rule or invitation budgets for system mail (AS6). - MUST NOT add boto3/botocore to `apps/api/requirements.txt` unless the Architect records why the hand-signed SigV4 seam (`apps/api/app/aws/`) cannot serve SES (the repo guards its absence). - MUST NOT put a tenant Gmail credential, OAuth token or service-account key on any node or in a runner's environment. - MUST NOT change Slack/Telegram delivery, the support chat, or site enquiries (no email today). Done means: Verifier-recipient mechanism (Planner decision — the operator test send goes only to the caller's own verified login address, `routes/email_templates.py:318-331`, plus the four §9.1 simulator addresses; the verified-address rule AS6 is NOT relaxed, and no privileged account ever gets a shared or public inbox): - Loopback `probe@<mail_domain>`: a **non-privileged verifier user** — an ordinary member invited into org B by its admin (`org/console/dev-orgb-login`, a non-operator org), never superadmin, never in the operator org — whose login email is `probe@mail.dev.jaah.ai` (prod, C11 only: `probe@mail.jaah.ai`, invited only into an operator-owned TEST tenant on `https://app.jaah.ai` (reached with `me/jaah/debug/main/operator-login`), named in the cutover checklist — never a customer org; created and deleted within the cutover run — Planner decision under Full Auto. If no operator-owned test tenant exists, skip the prod verifier user, defer the prod loopback of user-triggered mail, and use only the raw send to `probe@mail.jaah.ai`, which needs no member). Its mail lands as a raw object under `inbound/` in the inbound bucket (readable with `me/jaah/system/bootstrap-admin`; also under the operator's Unmatched emails); the verification link is taken from there. Use it for user-triggered mail (mention, password reset, invitation, billing notice). - Every template type / mail-tester.com: render each type's raw MIME with the api's own code path in the **local `./dc` api container, built from the commit deployed to that environment, using built-in template content** (a script that calls `email_templates` + `mail_message.build` exactly as the outbox does, with the S2 tags) — never by exec into a deployed task or reading a deployed DB (Jaah is operated only through its UI) — and send it with `aws sesv2 send-email --content Raw` from the environment's identity and configuration set via `me/jaah/system/bootstrap-admin`. mail-tester needs SES production access (it is an unverified recipient). No account's login email ever points at a mail-tester address. Operator-edited templates are covered separately through the console's own preview / operator test send, not the raw path. The free tier caps tests per day — spread runs across days; if a paid credit pack is needed, that is a vendor purchase: ask once with header `Human act`. - The verifier user is deleted at the end of the run that created it (Done-means item: verify it no longer exists in the org's members list). - **AC19 Availability.** Every capability above is ON by default in the dev deployment (no flag left off), wired by Terraform + console settings, and reachable by the flows above on `https://dev.jaah.ai`; prod is cut over after dev passes AC1–AC17. → verify: walk each child's live oracle on dev (with the documented deferrals I2/I3/I4) and confirm SES knobs, `email_unsubscribe_secret`, hooks URL and the plan switch are set/ON on dev; then on prod after promotion set the same knobs through the console (Jaah UI only). - **AC6 (prod half)** → verify after promotion: `dig +short TXT _dmarc.mail.jaah.ai` → `v=DMARC1; p=reject; …; rua=mailto:…`; `dig +short MX mail.jaah.ai` → SES inbound host; `dig +short TXT bounce.mail.jaah.ai` → SPF including amazonses; `dig +short MX jaah.ai` still `1 smtp.google.com.` and `_dmarc.jaah.ai` unchanged. `terraform plan` shows no drift. - **AC1 (prod half)** → mail-tester.com report on prod shows DKIM pass `d=mail.jaah.ai`, SPF pass `bounce.mail.jaah.ai`, DMARC pass; loopback to the prod verifier user `probe@mail.jaah.ai` verdicts all PASS. (Inbox-not-Spam deferred — I2.) - **AC2 on prod** → every template type scores 10/10 on mail-tester.com. - **AC3 and AC7 on prod** → each send path arrives from `mail.jaah.ai` and shows `delivered`; inbound to `anything@mail.jaah.ai` appears under Unmatched emails within 2 minutes. - Promotion gate: `aws sesv2 get-account` in jaah-main shows `ProductionAccessEnabled=true` and the prod identity `SUCCESS` before the operator is handed the promotion. - **AC5 final** → `grep -rn "gmail.googleapis.com" apps/api/app` returns only the tenant-Gmail module(s). - **AC18 (R13) Docs.** → verify: `python3 tools/check_context.py` green; the runbook, reader doc and ADR entry exist and no living doc still says "Why Gmail, not SES". - Quick gate `./dc check quick` green and changed-area tests green; `python3 tools/check_context.py` green (any doc this child adds/edits is routed and single-homed).