Build app panel, metering and hand-over

Part of epic "Build sponsored Easy Mode app hosting" (PRD task 13412, architecture task 13413). Architecture child **C9 Console + metering**. Builds on C6 (`hosted_apps`, `hosted_app_usage` tables — which already carry `last_deploy_*`, `pause_*`, `handed_over_to`, so no new migration is expected here — the pause/unpause primitive, operator routes) and C3 (the deploy run it reports). Uses C4's `ledger.expire_sponsored` and C2's `easy_mode_terms`. ## Original requirement (verbatim) > Caps (proposed defaults, plan terms with per-org overrides): 2 apps per org, 100k back-end requests/month, 10 GB egress/month, 500 MB database per app, Lambda reserved concurrency 2, optional sponsorship end date. Over cap: app paused to an "over limit" page, org notified. > Deploy: … Repo hand-over to the client later (transfer + trust removal). > 8. Console + metering: the app's URL, last deploy and usage against caps on the project; daily usage cron, over-cap pause and notification, repo hand-over. PRD decisions (binding): D11 over a monthly cap (requests, egress) the paused app reopens automatically on the 1st of the next month (UTC); the operator can unpause early or raise the cap. D12 sponsorship end: admins warned 7 days ahead; on the date apps show the paused page, unused sponsored credit expires, the org is asked to add its own Anthropic API key (not a subscription login) or buy Jaah AI credit, admins notified; shared node left alone. D13 hand-over: operator-only, repo transferred, OIDC trust removed, app keeps serving its last version. D14 delete with typed confirmation; repo kept. A1 "org notified" = every Owner/Admin via `user_notify.deliver`. A2 DB-size over cap stays paused until a daily check finds it under cap or the operator acts. A3 operator pause/unpause/delete on the customer page; members see the panel. A6 one paused page naming the reason. ## Approach (settled by the Architect) - Project page "App" panel (Easy Mode projects only): address (link), state with reason ("Paused — over limit: requests"), last deploy (time, short sha, result), usage vs each cap with units, sponsorship end; Retry on `failed` (`POST /api/hosted-apps/{id}/retry`). Operator controls Pause / Unpause / Delete / Hand over on the customer page's Easy Mode tab (apps list there). Easy Mode project delete dialog requires typing the address. Jaah AI page's sponsorship-ended state with the two choices. - Deploy status: the CP reads the last run with the admin credential — `GET /repos/{repo}/actions/runs?branch=main&per_page=1` in the usage cron and on panel open (no webhook); writes `last_deploy_at/sha/result`. - Cron `hosted-app-usage` daily 02:00 UTC (`cron_system_activities._system`, pattern `:1258-1287`): per env invoke `ea-<env>-ops usage`, upsert `hosted_app_usage`, compare with `easy_mode_terms`; trip → `pause(reason='over_limit'|'db_size', cap=…)` + `automation_alerts.stage` (`automation_alerts.py:71-86`: one bell notice per Owner/Admin and, after commit, `user_notify.deliver` per person), once per (app, month, cap). Operator "Run now" uses the existing cron trigger. - Cron `hosted-app-month-open` 00:10 UTC on day 1: unpause `over_limit` apps; `db_size` unpauses when the daily run sees it under cap. - Cron `easy-mode-sponsorship` daily: `sponsor_end − 7d` → notice once; on/after end → pause all org apps (`sponsor_ended`) and `ledger.expire_sponsored`. Jaah AI mode left as is (the floor hold, `jaah_routing/routing.py:95-96`, stops work unless the org bought credit); the notice names the two choices (add an Anthropic API key at `POST /api/accounts` `auth_kind=api_key` — link to the accounts page; buy Jaah AI credit). Shared node untouched. - Hand-over (operator): `POST /api/operator/hosted-apps/{id}/hand-over {account}` → `POST /repos/{repo}/transfer {new_owner}` with the admin credential; replace the deploy role's trust with deny-all at request time; state `handed_over`, `handed_over_to`; app keeps serving. Transfer to a personal account waits for acceptance: panel shows "Transfer pending <account>" until the repo owner changes (checked by the daily cron). ## Acceptance criteria (verbatim, BA) - **AC-15 Usage vs caps.** The project's app panel shows, for each cap, used / limit with units (requests this month, egress GB this month, DB MB, concurrency limit) and the sponsorship end date if set, refreshed at least daily by a usage cron. → verify: dev, generate traffic, wait for (or trigger as operator) the usage run, observe numbers move. - **AC-16 Over cap.** When usage passes a monthly cap the app serves the paused "over limit" page on both `/` and `/api/*`, the panel shows "Paused — over limit" with the cap that tripped, and each org Owner/Admin is notified once; on the 1st of the next month (UTC) it reopens automatically; the operator can unpause early or raise the cap. DB-size over cap per A2. → verify: dev, set the org's request cap override to a small number, generate requests, run the usage cron, `curl` the URL → paused page; observe one notification per admin; operator Unpause → live again. - **AC-17 Operator controls** (UI). The operator can pause, unpause and delete any app of the org from the customer page; a paused-by-operator app shows its reason. → verify: dev, pause then unpause the test app; `curl` reflects each. - **AC-18 Sponsorship end.** With an end date set: org admins are notified 7 days before; on the date, apps serve the paused page ("sponsorship ended"), the unspent sponsored credit is expired by a ledger entry (bought credit untouched), sponsored AI work stops, and admins are notified with two choices — add an Anthropic API key (own account; no subscription login offered) or buy Jaah AI credit; the shared node keeps running. → verify: dev, set the test org's end date to today+7 then to the past, run the crons, observe each notice, the ledger entry, the paused page and the Jaah AI page's two choices. - **AC-19 Hand-over.** The operator hands the repo over by naming a GitHub account/org; the repo is transferred, its OIDC trust is removed (a later push to the transferred repo's `main` no longer deploys), the app keeps serving its last version and the panel shows "Handed over to <account>". → verify: dev, hand over a test app to a test account; push; observe the deploy step fails to assume the role and the URL still serves the last version. - **AC-23 Delete** (UI). Deleting an Easy Mode project (typed confirmation naming the address) removes its app, database, address and AWS resources; the repo remains in Jaah's GitHub org; the slug becomes free. → verify: dev, delete; `curl https://<slug>.dev.apps.jaah.ai` no longer serves the app; repo still exists. - **AC-11** (panel part). "The project panel shows the URL (clickable) and the last deploy (time, commit, result)." - **AC-9** (Retry). "Partial failure leaves the app in a readable `failed` state with a Retry, never a silent half-app." - **AC-1** (this child's part): with Easy Mode off and no hosted apps, no App panel or hosting text shows on project, org or billing pages. All UI oracles on the dev console in a browser; copy per `docs/standards/ui-copy-standards.md`; phone width passes; fails if text truncated, overlapping, or a raw slug/id/enum shows instead of a readable label (e.g. `over_limit` must read "over limit"). Credentials & reach: - AC-1…AC-9, AC-14…AC-20, AC-22, AC-23, AC-3b (dev UI) — `me/jaah/debug/main/operator-login` (operator; switches into the test tenant, covers member role) on `app.dev.jaah.ai`; member view also via `org/console/dev-tpl-member-login` — verified — both on roster, `credential_get` ok; operator login `POST /api/auth/login` 200 from this node. - AC-9, AC-11, AC-19, C3 template creation, C6 GitHub writes — `me/users/roman@gmind.tech/github_admin` (classic token, `repo, workflow, admin:org`; admin of `jaah-ai`); hand-over target org `ruzhukov-genai` (admin, so transfer needs no acceptance) — verified — `GET /user/memberships/orgs/jaah-ai` = admin/active. - AC-16, AC-18 notices to Owners/Admins — operator login acting in the test org (`POST /api/auth/switch`), test org Owner = that login — verified (same login, mem-30597 recipe). - Lambda concurrency 1,000 — AWS approval — deferred; live traffic oracles need a deployed app from C6. - Live DB reads — `tools/cp_psql.sh dev` — deferred — broker refused for this task; use the console/API. Out of scope: a separate GitHub org; a separate app apex domain; custom domains, registrar purchase and paid hosting after sponsorship (draft task 10233 stays as that later work); the always-on shared-box hosting tier. Do NOT give tenants pause/unpause/hand-over controls; do NOT offer a subscription login in the sponsorship-end choices; do NOT delete repos; do NOT touch the shared node at sponsorship end; do NOT expire bought or signup credit; do NOT change the wizard (C7) or provisioning steps (C6) beyond calling their primitives. Done means: - AC-15 — dev: panel shows used / limit with units for requests, egress GB, DB MB, concurrency and the sponsorship end; after traffic and an operator "Run now" of `hosted-app-usage` the numbers move (before/after screenshots, desktop + phone). - AC-16 — dev: requests cap override set small, traffic, usage run → `curl` `/` and `/api/health` serve the paused page naming "over limit"; panel "Paused — over limit: requests"; exactly one notice per Owner/Admin (a second run sends none); operator Unpause → live; api test: `hosted-app-month-open` reopens `over_limit` apps and leaves `db_size` until under cap. - AC-17 — dev: operator pause/unpause from the customer page; `curl` reflects each; panel shows "Paused by operator". - AC-18 — dev: end date today+7 → one warning notice; end date in the past → paused page "sponsorship ended", a `sponsored_expiry` ledger entry equal to the unspent sponsored credit (bought credit untouched), next AI task held, notice and Jaah AI page show the two choices (API key, buy credit), shared node still live. - AC-19 — dev: hand over to `ruzhukov-genai`; repo transferred; a push to its `main` fails at assume-role; the URL still serves the last version; panel "Handed over to ruzhukov-genai". - AC-23 — dev: delete dialog requires typing the address; after delete the URL no longer serves and the repo still exists. - AC-11 (panel) / AC-9 (Retry) — panel shows clickable address and last deploy time, short sha and result after a push; a `failed` app shows its readable failure and a working Retry. - AC-1 (part) — an org with Easy Mode off and no apps shows no App panel or hosting text. - `./dc check quick` green.