Send all system mail through SES

## Original request (verbatim) > Migrate the emails system from GMail API to AWS SES. Use all the best practices. The quality of communication is of highest priority. Spare no time or expense. > > [jaah.ai](http://jaah.ai) MX is currently at GMail and will remain so, but we can carve out whatever we need Epic context (BA PRD Task 9352, Architecture Task 9353) — settled: all system mail (send + receive) moves to SES on `mail.jaah.ai` (dev: `mail.dev.jaah.ai`, dev account, sends/receives real mail) with suppression, operator-only delivery log, RFC 8058 unsubscribe, DMARC monitoring, operator-approved template redesign; tenant Gmail (agent send, automation action, receive) is built in this epic behind one plan feature. All system AWS calls use the hand-signed SigV4 seam (`apps/api/app/aws/common.py` `signed_call`), no boto3. SES events and inbound mail are PULLED from SQS by system crons (precedent `apps/api/app/alarm_relay.py`, `apps/api/app/aws/sqs.py`); the only new public path is unsubscribe on the hooks edge. ## Scope of this child C2a SES send: `aws/ses.py` (send), `system_mail.py`, `mail_message.py` extended for S4 (C7 created it), the outbox rewrite, the `email_deliveries` table written at enqueue, the operator SES knobs, the operator test send's Simulator choice, and every send path in PRD §3 moved off `gmail_mailbox.send` (retire the Gmail system **send**). Delivers R1 and the send half of AC5. Event processing, suppression and the delivery-log/suppression UI are C2b. ## Requirements (verbatim from the PRD) ### R1 — System mail is sent through SES (A1, A5, Q2, Q5) Every send path listed in §3 sends through Amazon SES (sesv2) from `notifications@mail.jaah.ai` (prod) / `notifications@mail.dev.jaah.ai` (dev) with the operator's sender name, as multipart text + HTML, with a stable `Message-ID` recorded against the delivery-log row. The caller-facing error contract (not configured → today's 409/"not configured" behaviour; provider refusal → today's 502/retry behaviour) is preserved. Send paths (PRD §3, each must work on SES): invitations + org-invitation outcome (`user_invite.py:89,142`); sign-up code/link (`onboarding_mail.py:46`, `routes/signup.py:306`); password reset (`routes/password_reset.py:99`); new-application admin alert (`routes/signup.py:743`); approval/rejection/billing-start (`org_approval.py:156,248,328,347`); welcome (`org_welcome.py:70`); trust-request decision (`trust_requests.py:330`); event notifications — mentions, governance, merge lane, alerts (`email_notify.py:130-186`); billing notices, budget/plan messages, payment links (`email_notify.py:242-332`, `billing/notices.py:656`, `budgets.py:330`, `plan_rollover.py:128`, `routes/billing_orgs.py:760`); trigger-owner notices (`email_notify.py:363`, `trigger_notices.py`); operator test send. AS6. Assumed the outbox's at-least-3-tries durability and the "only to a verified address" rule stay exactly as they are. AS7. Assumed the delivery log retention is **90 days** (then deleted). AS8. Assumed "Reply-To" is not set by default: replies to `notifications@mail.jaah.ai` land in SES inbound and appear under Unmatched emails (operator), unless an operator trigger takes them. ## Approach (from the settled architecture) - `aws/ses.py` — `send_raw(raw: bytes, *, configuration_set: str, tags: dict[str,str]) -> str` (SES MessageId); REST-JSON via `common.signed_call` (no `target`), service `ses`, region from `AWS_REGION`; errors → `SesError(json11.AwsError)` carrying the HTTP status. SES v2 `SendEmail` with `Content.Raw`. (C2b adds the suppression calls to this module.) - `mail_message.py` (created by C7): extend for S4. No caller `Message-ID`/`Date` (SES overrides both); the stable id is the SES `MessageId`, stored as `email_deliveries.provider_message_id`; S2 tags `delivery_id`, `mail_type` on every send. - `system_mail.py` — provider-neutral contract replacing `gmail_mailbox` on the system path: `NotConfigured`, `SendError(.status)`, `configured()`, `send(OutboundEmail, *, delivery_id) -> str`. Reads the S1 env (unset ⇒ `NotConfigured` ⇒ today's 409, AC4) plus two NEW operator knobs under *Settings > Integrations > Email*: sender name and From local part (default `notifications`). - Knobs: ADD the SES knobs; do NOT delete the Gmail knobs (`operator_settings/catalog_integrations.py:163-191`) — `email_credential_id`/`email_address` are still read by the live Gmail receive path (`gmail_mailbox.load_config/watch/sync/maintain`, the `gmail-mailbox` cron) until C3 retires it; C3 deletes every Gmail knob and owns the UI-field half of AC5. - Non-send importers of `gmail_mailbox` that ask "is email configured / which address": move to `system_mail.configured()` / the SES From address — `notify_targets.py:652-657,765`, `operator_settings/writes.py:73` (`single_address`), `email_notify.py:110-116,201,304,347`, `routes/users.py:531-535`. - `email_deliveries` table (org-zero): id, created_at, mail_type, essential, recipient, recipient_user_id (nullable), subject, status (`queued|sent|delivered|delayed|bounced|complained|suppressed|failed|not_configured`; SES `Reject`/`Rendering Failure` → `failed` with reason in status_detail), status_detail, provider_message_id (unique, nullable), attempts, sent_at, last_event_at, events JSONB (capped list). Rows created at enqueue so `failed` is never silent. Retention 90 d registered in `app/cron_retention.py`. - `durable/notify_outbox.py` — `email()` gains keyword-only `mail_type`, `essential`, `unsubscribe: UnsubscribeTarget | None = None` (ship the dataclass stub in `email_unsubscribe.py`; C5 fills it); inserts the queued delivery row, seals only delivery id + body. `_email_attempt`: `NotConfigured` ⇒ `not_configured` (final); `SendError` 4xx other than throttling ⇒ `failed` (final); 5xx/throttle ⇒ retried (≥3 tries, AS6); success ⇒ `sent` + `provider_message_id`. (C2b adds the suppressed pre-check.) - Operator test send (`routes/email_templates.py:328`) goes through `system_mail.send` with a delivery row, and gains an operator-only *Simulator* recipient choice admitting only the four `*@simulator.amazonses.com` addresses (bounce, complaint, success, suppressionlist) — a fixed enum, not a typed field; update the route's "never a typed one" docstring (`routes/email_templates.py:318`) to say so (§9.1). - Move every send caller to the new outbox signature with the right `mail_type`/`essential` (essential = PRD R6's list: invitations, sign-up codes, password reset, welcome/approval/rejection, trust decisions, billing notices and payment links) and delete `gmail_mailbox.send` (§9.9 — module deletion is C3's). - Set the SES knobs on dev through the console (Jaah UI only) so AC19's "no flag left off" holds. Pinned seams (build against these exactly; other children build against them too): - S1 env (C1 → C2/C3): `SES_MAIL_DOMAIN`, `SES_CONFIGURATION_SET`, `SES_EVENTS_QUEUE_URL`, `SES_INBOUND_QUEUE_URL`, `SES_INBOUND_BUCKET`; inbound objects under prefix `inbound/`. - S2 send tags (C2 ↔ C1 event destination): `EmailTags` `delivery_id=<uuid hex>`, `mail_type=<template key>`. - S4 system send (C2 → C5/C6 and every caller). System mail is always one recipient (verified-address rule): `OutboundEmail(to: str, subject: str, text: str, html: str|None, mail_type: str, essential: bool, headers: Mapping[str,str] = {}, reply_to: str|None = None)`; `system_mail.send(msg, *, delivery_id: UUID) -> str`; `notify_outbox.email(address, subject, text, html, *, what, mail_type, essential, unsubscribe=None) -> bool`; `UnsubscribeTarget(user_id: UUID, event: str)` defined in `email_unsubscribe.py` (C5) — until C5 lands, C2 ships the dataclass stub there and C5 fills it. Credentials & reach: | Oracle | Credential / identity / environment | Status | |---|---|---| | AC6, terraform plan, SES state, production-access filing (C1, C11) | `me/jaah/system/bootstrap-admin` (management 155534476372) assuming `OrganizationAccountAccessRole` in jaah-dev 905905585302 and jaah-main 366008913850 | **verified** — `credential_get` ok; read-only probe from this node: `sts get-caller-identity`, `sesv2 get-account`, `route53 list-hosted-zones` in both accounts answered | | AC4 (local), AC7, AC9, AC10, AC13 (after sandbox exit), AC17(b) in the dev console | `me/jaah/debug/main/operator-login` (operator, `https://app.dev.jaah.ai`) | **verified** — `credential_get` ok; `https://app.dev.jaah.ai` answered 200 from this node | | SES production access (sandbox exit) — gates AC2, AC3, AC11, AC13 and C11 | AWS approval of `PutAccountDetails` in jaah-dev and jaah-main (filed by C1) | **deferred** — reason: AWS reviews it; both accounts probed `ProductionAccessEnabled=false`. Owner C1 files on day one; dependent oracles run after approval | | AC2 deliverability | mail-tester.com, no account (free tier) | **verified (reach)** — `https://www.mail-tester.com` answered 301→site; the check runs after the SES production-access row | | AC1/AC11 Gmail UI checks (inbox placement, native Unsubscribe, Show original) | a Google login to admin@jaah.ai | **deferred** — operator: "Skip it" (I2) | | AC5, AC18, unit/integration suites | worktree only | **verified** — nothing beyond the worktree | Out of scope: - Do NOT build `ses-events`, suppression, the delivery-log/suppression routes or pages (C2b). - Do NOT build inbound/receive, retire gmail-push, the `gmail-mailbox` cron or the Gmail knobs, or delete `gmail_mailbox.py` (C3). - Do NOT add unsubscribe headers/footer/public route (C5) or redesign templates (C6); ship only the `UnsubscribeTarget` stub. - Do NOT add Reply-To by default (AS8). Epic out-of-scope (PRD §7, verbatim): - MUST NOT change the apex `jaah.ai` MX, SPF or `_dmarc.jaah.ai` record, nor any Workspace mailbox, alias or routing. - MUST NOT add SES Virtual Deliverability Manager, BIMI/VMC, or a dedicated IP / dedicated IP pool. - MUST NOT request Google's restricted `gmail.readonly` (or any restricted Gmail scope) via OAuth; no CASA assessment. - MUST NOT add open/click tracking pixels or link rewriting to system mail. - MUST NOT let tenant Gmail replace system mail for an org's members (Q9 option not chosen). - MUST NOT expose the delivery log or suppression list to tenant admins or users (operator only, Q7). - MUST NOT relax the verified-address rule or invitation budgets for system mail (AS6). - MUST NOT add boto3/botocore to `apps/api/requirements.txt` unless the Architect records why the hand-signed SigV4 seam (`apps/api/app/aws/`) cannot serve SES (the repo guards its absence). - MUST NOT put a tenant Gmail credential, OAuth token or service-account key on any node or in a runner's environment. - MUST NOT change Slack/Telegram delivery, the support chat, or site enquiries (no email today). Done means: Verifier-recipient mechanism (Planner decision — the operator test send goes only to the caller's own verified login address, `routes/email_templates.py:318-331`, plus the four §9.1 simulator addresses; the verified-address rule AS6 is NOT relaxed, and no privileged account ever gets a shared or public inbox): - Loopback `probe@<mail_domain>`: a **non-privileged verifier user** — an ordinary member invited into org B by its admin (`org/console/dev-orgb-login`, a non-operator org), never superadmin, never in the operator org — whose login email is `probe@mail.dev.jaah.ai` (prod, C11 only: `probe@mail.jaah.ai`, invited only into an operator-owned TEST tenant on `https://app.jaah.ai` (reached with `me/jaah/debug/main/operator-login`), named in the cutover checklist — never a customer org; created and deleted within the cutover run — Planner decision under Full Auto. If no operator-owned test tenant exists, skip the prod verifier user, defer the prod loopback of user-triggered mail, and use only the raw send to `probe@mail.jaah.ai`, which needs no member). Its mail lands as a raw object under `inbound/` in the inbound bucket (readable with `me/jaah/system/bootstrap-admin`; after C3, also under the operator's Unmatched emails); the verification link is taken from there. Sending to an address on the verified SES domain works inside the SES sandbox. Use it for user-triggered mail (mention, password reset, invitation, billing notice). - Every template type / mail-tester.com: render each type's raw MIME with the api's own code path in the **local `./dc` api container, built from the commit deployed to that environment, using built-in template content** (a script that calls `email_templates` + `mail_message.build` exactly as the outbox does, with the S2 tags) — never by exec into a deployed task or reading a deployed DB (Jaah is operated only through its UI) — and send it with `aws sesv2 send-email --content Raw` from the environment's identity and configuration set via `me/jaah/system/bootstrap-admin`. mail-tester needs SES production access (it is an unverified recipient). No account's login email ever points at a mail-tester address. Operator-edited templates are covered separately through the console's own preview / operator test send, not the raw path. The free tier caps tests per day — spread runs across days; if a paid credit pack is needed, that is a vendor purchase: ask once with header `Human act`. - The verifier user is deleted at the end of the run that created it (Done-means item: verify it no longer exists in the org's members list). - **AC1 (R1, R2) Authenticated, aligned send.** → verify: as operator on dev, Profile > Alerts → Email on, then trigger a mention; open the received message in a Gmail inbox → *Show original*: `From: … <notifications@mail.dev.jaah.ai>`, `DKIM: PASS` with `header.d=mail.dev.jaah.ai`, `SPF: PASS` with `smtp.mailfrom=bounce.mail.dev.jaah.ai`, `DMARC: PASS`; lands in **Inbox, not Spam**. Repeat on prod after cutover with `mail.jaah.ai`. (Gmail UI part deferred — operator "Skip it" (I2). Substitute: trigger a mention and a password reset to the loopback verifier user (`probe@mail.dev.jaah.ai`), and a raw render-and-send of the operator test template to it; the raw object in the inbound bucket shows `From: … <notifications@mail.dev.jaah.ai>`, a `DKIM-Signature` with `d=mail.dev.jaah.ai`, Return-Path at `bounce.mail.dev.jaah.ai`, and the SES receipt notification's SPF/DKIM/DMARC verdicts all PASS. The mail-tester half runs with AC2 after sandbox exit.) - **AC3 (R1) Every send path works.** → verify on dev: invite a user; resend invitation; sign-up code; password reset; org approve/reject; welcome; trust decision; mention; billing payment link; budget warning; trigger notice; operator test send — each arrives from `mail.dev.jaah.ai` and appears in the delivery log as `delivered`. (Live run deferred until SES production access in jaah-dev — C11 runs it; `delivered` status needs C2b. Now: an automated test per send path asserts it enqueues through `notify_outbox.email` with the right `mail_type`/`essential`, and a test asserts no module imports `gmail_mailbox.send`; plus the loopback verifier user receives at least the password-reset and mention mails, each with an `email_deliveries` row `sent` carrying the SES MessageId.) - **AC4 (R1) Unconfigured state.** → verify: with SES settings blank (local `./dc` stack), `POST` resend invitation returns the same 409 "not configured" as today; `./dc test tests/` email suites green. - **AC5 (R1) No Gmail on the system path (send half).** → verify: no system send path calls the Gmail API — `grep -rn "gmail_mailbox" apps/api/app` hits only the receive path C3 retires (`gmail_mailbox.py` itself, `routes/public.py` gmail-push, `cron_system_activities.py`, `trigger_intake.py`, the Gmail knobs) and docstrings; *Settings > Integrations > Email* shows the SES sender name and From local part fields. (The no-Gmail-fields UI half is C3's.) - Outbox semantics: tests prove 4xx ⇒ final `failed` with reason; 5xx/throttle ⇒ retried ≥3 tries; `NotConfigured` ⇒ `not_configured`; every enqueue creates a `queued` row; the 90 d retention purge is registered. - Quick gate `./dc check quick` green and changed-area tests green; `python3 tools/check_context.py` green (any doc this child adds/edits is routed and single-homed).