API keys backend: model, auth, routes

Part of epic 9326 (Jaah API keys). Child C1 of the Architect's decomposition: the whole backend. It is the epic's ONLY schema migration. ## Original requirement (verbatim) > Implement Jaah API keys workflow. > - Permissions: same as we have internally: subject:action. Access to object is limited to the owner of the key. Should be very easy to configure on the UI. By default all is max, but user can remove some > - API Keys can only be created by at least Org Admin > - Keys have expiration > Do a web research and suggest the most modern approach to this for this type of systems Research outcome (BA, accepted by operator D9): opaque, prefixed, checksummed bearer secrets, stored only as a hash, shown once (GitHub fine-grained PAT / Stripe restricted key / GitLab PAT pattern). Server-side lookup per request, so revocation is immediate. OAuth client-credentials is rejected for now. ## Requirements this child delivers (PRD, verbatim) - **R1 Key = personal credential of its creator ("owner").** A request bearing a valid key is authenticated as the owner, with rights narrowed to the key's permissions. - **R2 Creation gate.** Only a signed-in Org Owner or Org Admin can create a key. Nobody else sees the create control or can call the create endpoint (403). A key itself can never create, edit, rotate or revoke keys. - **R3 Owner cap.** Effective rights of a request = key's ticked permissions ∩ owner's rights *at request time*. Org Owner/Admin, active member → key works (capped). Demoted below Org Admin → key refused (401) while demoted; works again if re-promoted (unless expired/revoked). Removed from org / user deleted / disabled → all their keys permanently revoked. Superadmin/operator rights are never conveyed by a key. Act-as/impersonation is never available to a key. - **R4 Permission vocabulary = the internal `feature:action` catalog**, read from its single home (`authz/catalog.py`), never a parallel list; plus one extra toggle **"Org administration"** that lets the key pass routes gated by Org Admin (members, role bindings, SSO, org settings) except key management. - **R6 Expiration** (API side): an expired key is refused (401). - **R7 Reach = whole org** — no per-project narrowing. - **R8 Secret format & handling**: prefixed + checksummed, stored hashed only, shown once; sent as `Authorization: Bearer <key>`. Never logged, never in audit payloads, never returned by any read endpoint (list shows a non-secret identifier, prefix + last 4). - **R9 Lifecycle actions.** Edit in place (rename, description, permissions, IP allowlist; secret unchanged; takes effect on the next request). Rotate: new secret, same settings; overlap None / 1h / 24h / 7d. Revoke: immediate, irreversible. Expiry cannot be extended. - **R10 Visibility & control.** Owner creates/edits/rotates/revokes own keys. Every Org Owner and Org Admin sees all org keys (never the secret) and can revoke any. Edit/rotate is owner-only. - **R12 Last used + audit.** Each key shows last-used time and source IP. Audit events for create, edit, rotate, revoke, expire, and refusal (bad scope / IP / expired). Requests made with a key are attributed in the audit log as "<owner> via API key <name>". - **R14 Per-key IP allowlist.** Optional list of IPs/CIDRs; empty = any IP. A request from outside is refused (403) and audited. The org IP allowlist (`app/ip_allowlist.py`) applies to key requests too. ## Approach (Architecture, pinned contract — build exactly this) Components K1–K7: - **K1 models** `apps/api/app/models/api_keys.py` + migration `apps/api/migrations/versions/<YYYYMMDDHHMM>_api_keys.py`. - `api_keys`: `EntityMixin` + `OrgScoped`, no slug. Columns: `owner_id` uuid FK `admin_users.id` ON DELETE RESTRICT NOT NULL indexed; `grants_all` bool default true (true ⇒ key set = honoured catalog keys at request time, follows the catalog, D-A1); `permissions text[]` default `{}` (used only when grants_all false; validated ⊆ `PERMISSION_KEYS`; unknown keys dropped on read); `org_admin` bool default true; `ip_allowlist cidr[]` default `{}` (bare IP stored /32 or /128); `expires_at` timestamptz NULL (NULL = never; immutable); `revoked_at`, `revoked_by_id` (FK admin_users SET NULL), `revoke_reason` (`revoked`/`owner_disabled`); `last_used_at`, `last_used_ip` inet (written at most once per 60 s, the `auth.py:57` throttle pattern); `reminded_7d_at`, `reminded_1d_at`, `reminder_seen_at`, `expiry_recorded_at` timestamptz NULL (consumed by the sweep child — create them HERE so the sweep adds no migration). - `api_key_secrets` (child, not an entity: add to the exempt list in `tests/test_entity_shape.py`; `OrgScoped`, classified in the tenancy manifest): `id` uuid v7, `api_key_id` FK CASCADE, `token_hash` char(64) UNIQUE (sha256 hex), `last4` char(4), `created_at`, `valid_until` NULL (NULL = current). - Secret: `jaah_` + 32 base62 chars (`secrets.choice`) + 6 base62 chars of CRC32 over the random part (43 chars). Checksum verified before any DB hit. Display id `jaah_…<last4>`. Plain SHA-256 (the `scim.py:108` precedent). - **K2 rules module** `apps/api/app/api_keys.py`: single home of mint, parse/checksum, verify, narrow, status, revoke, revoke_for_owner, forget_owner, record_refusal, refuse_key. - **K3 auth integration** (Architecture §4): - Middleware `auth.py:510-550`: only when the request has NO `COOKIE_NAME` cookie and `Authorization: Bearer jaah_…` (cookie always wins). (1) `parse` — bad shape/checksum → 401 `{"detail":"invalid_api_key"}`. (2) `verify(token, host)` in a threadpool on the identity lane (`IdentitySessionLocal` + `seam.identity_scope()`): one join secret→key→owner; 401 on unknown hash, secret past `valid_until`, revoked, expired, disabled owner, owner not currently Org Owner/Admin of a tenant org, owner with `is_superadmin`. Key `ip_allowlist` excludes host → 403 `{"code":"ip_not_allowed"}`. (3) org allowlist `ip_allowlist.session_refusal(owner_id, host)` (the comment at `auth.py:537-538`). (4) refusals of a resolved key write one `api_key_refused` audit row via `record_refusal`, throttled once per 60 s per (key, reason); unknown hash writes nothing. (6) success sets `request.state.admin_id = owner_id` and `request.state.api_key = ApiKeyContext(key_id, name, owner_id, keys: frozenset[str], org_admin: bool)` (frozen), touches last_used (throttled); `console_session`/`assist` unset. - `_session_user` (`auth.py:600`): with no cookie and `request.state.api_key`, load owner on the identity lane, re-check `disabled_at`, `set_request_actor`, skip `_touch_last_seen`. `_acting_seat` ignores `X-Jaah-Act-As` for a key. - `_resolve_principal` (`authz/principal.py:192`): return `api_keys.narrow(principal_for_user(session, owner), ctx)` — `permissions ∩ K`, `org_keys ∩ K`, `scope_permissions` filtered to K, `is_superadmin=False`, `acting=False`, `org_role` kept only if `ctx.org_admin`, `bindings=()`, `system_project_ids` via `_system_project_ids(org_id)` when org_admin off, `api_key_id=ctx.key_id` (new `Principal` field, default None). K = `PERMISSION_KEYS − UNHONOURED_PERMISSION_KEYS` when grants_all else stored set. Narrow ALL THREE sets and clear the flag (risk: partial narrowing). - `_enforce` (`enforce.py:286-301`): skip the `own_credential` and `author_draft` fallbacks when `principal.api_key_id` is set. A FEATURE/FEATURE_ANY/ORG_ADMIN denial with `api_key_id` calls `record_refusal(reason="scope", detail=<key or "org_admin">)` before 403. - `GET /api/auth/me` with a key returns narrowed `effective_permissions` plus `api_key: {id, name, display, expires_at}`. - **K4 management routes** `apps/api/app/routes/api_keys.py` (registered beside `routes/scim.py`), all `@session_only`, `Access.ORG_ADMIN`, tenant tenancy. Create requires a non-acting tenant principal (operator acting-as can list/revoke, 403 on create). Edit/rotate require `key.owner_id == caller.id` else 403. Rotate on revoked/expired → 409; keeps `expires_at`; overlap `none` deletes every non-current secret row; overlap X sets current secret `valid_until = now + X`. Admin revoke stamps `revoked_at`/`revoked_by_id` only, never `updated_by_id`. | Method + path | Body | Returns | |---|---|---| | `GET /api/api-keys` | — | `ApiKeyOut[]` newest first | | `POST /api/api-keys` | `{name, description="", grants_all=true, permissions=[], org_admin=true, ip_allowlist=[], expires_at: datetime|null}` (future or null) | `201 {key, secret}` | | `GET /api/api-keys/{id}` | — | `ApiKeyOut` | | `PATCH /api/api-keys/{id}` | `{name?, description?, grants_all?, permissions?, org_admin?, ip_allowlist?}` (no expires_at) | `ApiKeyOut` | | `POST /api/api-keys/{id}/rotate` | `{overlap: "none"|"1h"|"24h"|"7d"}` | `{key, secret}` | | `POST /api/api-keys/{id}/revoke` | — | `ApiKeyOut` (idempotent) | `ApiKeyOut = {id, name, description, owner:{id,label}, display, grants_all, permissions[], org_admin, ip_allowlist[], expires_at|null, created_at, created_by:{id,label}, updated_at, last_used_at|null, last_used_ip|null, status: "active"|"expiring"|"expired"|"revoked"|"suspended", revoked_at|null, revoked_by:{id,label}|null, revoke_reason|null, previous_secret_valid_until|null, can_edit, can_revoke}`. Status precedence revoked > expired > suspended > expiring (≤7 days) > active. No response carries a hash or secret except the one-time `secret`. Writes record `api_key_created`/`_edited` (changed fields)/`_rotated` (overlap)/`_revoked` via `user_activity.record_write`. Also: `GET /api/permissions` (`routes/auth.py:1580`) gains `feature_label` (from `catalog._FEATURE_LABELS`) and `honoured: bool` per row — reads the catalog, does not change it. `GET /api/audit` (`routes/audit.py:91`) gains an `api_key=<id>` filter. - **K5 attribution**: `request_caller.py` binds `api_key_id` and `api_key_name`, added to `CALLER_KEYS`. (The web Audit rendering is the UI child's.) - **K6 owner lifecycle**: disable (`routes/users.py:1119`, also SCIM deprovision) → `api_keys.revoke_for_owner(..., reason="owner_disabled")` + audit per key. Delete (`delete_member`, `routes/users.py:793`, beside `scim.forget_creator`) → `forget_owner` writes `api_key_revoked` audit (reason `owner_removed`, key name, owner email, display id) then deletes key rows (secrets cascade); `record_write` data gains `api_keys_deleted`. Demotion writes nothing (401 while demoted). - **K7 `@session_only` marker** in `authz/declarations.py` (beside `own_credential`, `:80`), checked in `authz/guard.py:97` BEFORE the `_decides_without_io` early return → 403 `{"code":"api_key_not_allowed"}`. Mark: every `/api/auth/` route except `GET /api/auth/me` (password, MFA incl. recovery codes/trusted devices, password reset, sign-out, sessions, org switch, invitation accept, `PUT /api/auth/me`); every route in `routes/api_keys.py`; `POST /api/users/{id}/mfa/reset`, `POST /api/users/{id}/sso/reset`, `POST /api/orgs/{org_id}/scim/tokens`, `POST /api/nodes`, `POST /api/nodes/{node_id}/token`, `POST /api/nodes/{node_id}/tailnet-reenrol`; `GET /api/console/{project_id}/{number}/direct`; member creation `POST /api/users`, `POST /api/users/{id}/invitation` and every route creating an `OrgInvitation` or org membership (grep). NOT marked: `GET /api/permissions`, `POST /api/users/{id}/memberships`, `routes/access.py` binding writes. `PATCH /api/users/{id}` password/email arms call `api_keys.refuse_key(request)` (403 same code). Census test fails on any unmarked route of those classes; helper-mediated mints (`vnodes.issue_enrolment`, `durable/hosting_change.py:542`) recorded by name. Streams/WebSockets (`routes/stream.py:297`, `routes/console.py:224,234`) are not taught the key → 401; a test pins "REST only". `@always` routes: key acts as owner (D-A7); census prints every `@always` write a key reaches. - Confirm `stale_tab_guard.py:79` and `login_funnel.py:500/594/656` pass a cookie-less bearer request untouched. Tests: `apps/api/tests/test_api_keys.py` — mint/checksum, narrowing (incl. object-level reach via `scope_key_clause` with an unticked key), demote/re-promote, disable → revoked, delete → rows gone + audit, `@session_only` refusals + census, fallback skip, streams 401, operator route 403, rotation overlap, per-key and org IP allowlist, a sample of FEATURE/ORG_ADMIN/`@always` routes; a log-capture test that the secret never appears in logs or audit `data`. Credentials & reach: - AC5, AC6, AC8, AC11, AC13, AC15 on dev | `me/jaah/debug/main/operator-login` on `https://app.dev.jaah.ai` (control plane `dev.jaah.ai`), signed in to a tenant as Owner or acting as that tenant's seat | **verified**: `credential_get` succeeded; a read-only `POST /api/auth/login` probe answered 200 (role owner). - AC7, AC12 (non-admin member, second admin who demotes or revokes) | Throwaway members created by the verifier with operator-login (D-A4): create the member on the tenant's Users page, set the password through the member edit (`PATCH /api/users/{id}`, driven from the console as the operator), sign in as that member in a second Playwright context and enrol TOTP when prompted; remove the members afterwards | **verified** for the creating identity. - AC10 (hash only, `./dc psql`), the API test suites | Local `./dc` stack in the worktree | **verified**. - AC10 "secret absent from API logs" on dev | dev API logs | **deferred**: proven by this child's log-capture test on the local stack instead (reading dev logs outside the console breaks the hub rule "operate Jaah only through the Jaah UI"). Out of scope: Do NOT build the Settings UI, the grid, the Audit actor rendering or reader docs (UI child); do NOT build `user_notify` (choke-point child) or the sweep/cron/inbox kind (sweep child) — only create their columns. No OAuth client-credentials / JWT issuance / token endpoint. No org-owned service accounts; no keys issued on behalf of another member. No per-project narrowing of a key. No custom-role editor and no change to built-in roles or the `feature:action` catalog (`authz/catalog.py`, `access_catalog.py`) beyond reading it. No change to node tokens (`NODE_TOKEN`, `node_credentials.py`), SCIM tokens, or provider credential API keys (`models/credentials.py` `auth_kind='api_key'`). No re-enabling of public OpenAPI (`main.py` `openapi_url=None` stays). No org-level max-lifetime policy, no key approval workflow, no rate limiting per key, no GitHub secret-scanning partner registration. No change to cookie-session auth behaviour for existing console users. Keys on streams/WebSockets stay refused. Done means: - AC5 Key authenticates as owner — `curl -H "Authorization: Bearer <key>" https://app.dev.jaah.ai/api/projects` → 200 with the same project list the owner sees in the console. No key / malformed key (bad checksum) → 401. - AC6 Unticked permission is refused — Edit the key: untick `tasks:create` → save. `POST` a task with the key → 403; `GET` tasks → 200. Re-tick → `POST` succeeds within 60 s. - AC7 Owner cap (amended oracle, Architecture §9 / D-A2) — Owner demoted to project role → key call → 401; re-promote → 200. Owner **disabled** → key shows Revoked; call → 401. Owner **removed from org** → the key is no longer listed; `/audit` shows `api_key_revoked` with reason `owner_removed` and the key's name; the old secret answers 401. - AC8 "Org administration" toggle — With toggle on, `GET` an Org-Admin-gated route (e.g. org members) → 200; untick → 403. Key calling any API-key management endpoint → 403 regardless. Key calling an operator/superadmin route → 403. - AC9 (API half) — a key past `expires_at` → 401; `status` = `expired`. - AC10 Secret never re-shown or stored plain (API side) — no API response other than the create/rotate reply returns the secret (list/detail show prefix+last 4 only). `./dc psql` on a local stack: key table holds only a hash. Log-capture test: secret absent from logs and audit `data`. - AC11 Rotate with overlap (API) — Rotate with `1h` → new secret returned once; old secret works (200) until overlap ends, then 401; `none` → old secret 401 immediately. - AC12 Revoke (API) — Owner revokes → next call 401, status revoked. A second Org Admin (not owner) can revoke (200) but PATCH/rotate → 403; `can_edit=false`, `can_revoke=true` for them. - AC13 Last used + audit (API side) — after AC5, `GET /api/api-keys/{id}` shows `last_used_at` and `last_used_ip`; `/api/audit` holds created, edited, rotated, revoked, refused (with reason) rows, and key requests carry `api_key_id`/`api_key_name`; no secret in any audit payload. - AC15 Per-key IP allowlist — Set allowlist to an IP that is not the caller → call → 403 + audit refusal; add caller IP → 200. Org IP allowlist excluding the caller → key call refused too. - `GET /api/permissions` rows carry `feature_label` and `honoured`; `GET /api/audit?api_key=<id>` filters. - `apps/api/tests/test_api_keys.py` green incl. census, fallback-skip, streams-401, partial-narrowing and log-capture tests; `./dc check quick` green.

API keys backend merged (#6736 → f3c1e69): model+migration, auth/narrowing, @session_only, routes, audit. Quick gate + 25 key tests green; review lanes pass after fixes. Deploy superseded by f48b197. Follow-up draft #9395 (key-started dispatch narrowing). Memory: proposed 1 gotcha.