Build hosted-app provisioning API and workflows

Part of epic "Build sponsored Easy Mode app hosting" (PRD task 13412, architecture task 13413). Architecture child **C6 Provisioning API**. Builds on C1 (apps-account infra + SSM config), C2 (`easy_mode` switch, `easy_mode_terms`), C3 (template repos, for the deploy oracle) and C4 (`projects.easy_mode` column, which C4 alone migrates). Calls C5's `easy_mode.assign_project` (same wave; if it has not merged yet, rebase onto it before merging — C7 runs the joined flow). ## Original requirement (verbatim) > Hosting: serverless in the existing apps AWS account — static front-end on S3 + one shared multi-tenant CloudFront distribution, back-end container on Lambda (Lambda Web Adapter), one database per app on one shared small RDS Postgres. Address `<app-slug>.apps.jaah.ai` (existing apps zone). Any small app with a back-end. > Deploy: auto on push to main, from a repo Jaah creates and owns (temporarily) in Jaah's existing GitHub org, from a template carrying the deploy workflow (per-app OIDC role scoped to that app's resources only). Repo hand-over to the client later (transfer + trust removal). > 6. Provisioning API: `hosted_apps` table (standard entity shape); create, pause, delete; creates the CloudFront tenant, S3 prefix, Lambda, database, role and secret, the repo from template with its Actions variables. PRD: D9 dev console apps at `<slug>.dev.apps.jaah.ai`, live at `<slug>.apps.jaah.ai`. D14 deleting an Easy Mode project deletes its hosted app, database and address (typed confirmation); the repo is kept. A4 slug derived from the project name, DNS-safe, unique per environment; taken or reserved names get a numeric suffix shown before create. A7 the apps cap counts live (not deleted) apps; a 3rd is refused readably. A9 every door returns 403 with the plan sentence when off. ## Approach (settled by the Architect) **Names** (`env ∈ {dev, main}` from `APP_ENV`/deployment; slug DNS-safe ≤ 40 chars; reserved: `www, api, admin, app, apps, dev, staging, test, mail, status, docs, static, assets, cdn` plus any delegated child zone of `apps.jaah.ai`, read from Route 53 at slug check): address `<slug>.apps.jaah.ai` / `<slug>.dev.apps.jaah.ai`; repo `jaah-ai/app-<slug>` / `jaah-ai/app-dev-<slug>`; CloudFront tenant `ea-<env>-<slug>`; S3 prefix `s3://jaah-ea-sites-<acct>/<env>/<slug>/`; ECR repo `ea/<env>/<slug>`; Lambda `ea-<env>-<slug>` (arm64, image, 512 MB, 30 s, VPC private subnets, reserved concurrency = `EasyModeTerms.concurrency`, function URL auth `NONE` + `lambda:InvokeFunctionUrl` permission for `*` with `FunctionUrlAuthType=NONE`, env `DATABASE_URL`, `JAAH_ORIGIN_KEY`, `APP_URL`); database/login role `ea_<env>_<slug_>` (`CONNECTION LIMIT 5`, `REVOKE CONNECT FROM PUBLIC`); secret `ea/<env>/<slug>/db`; deploy role `ea-<env>-<slug>-deploy`, boundary `ea-app-boundary`, trust OIDC `aud=sts.amazonaws.com` + `sub` (below); deploy role policy: `s3:PutObject/DeleteObject/ListBucket` on its prefix only; ECR push/pull on its repo only (+ `GetAuthorizationToken`); `lambda:UpdateFunctionCode/GetFunction/GetFunctionConfiguration/InvokeFunction` on its function only; `cloudfront:CreateInvalidationForDistributionTenant` on its tenant only. Nothing else. **Config (no hidden flags)**: operator Settings knobs (catalog pattern `operator_settings/catalog_billing.py:92-95`): `easy_mode_aws_provisioner_role_arn`, `easy_mode_github_credential_id` (the github-admin credential, Q2 — every CP GitHub write for Easy Mode uses it; it never leaves the CP), `easy_mode_github_org` default `jaah-ai`, `easy_mode_templates` default `{"node":"jaah-ai/easy-template-node","python":"jaah-ai/easy-template-python"}` (C8 adds `easy_mode_runner_app_credential_id`). Infra facts are read at run time from SSM `/jaah/ea/<env>/config` (keys: region, account_id, bucket, distribution_id, distribution_arn, connection_group_domain, zone_suffix, subnet_ids, lambda_sg_id, runtime_role_arn, boundary_arn, oidc_provider_arn, ecr_registry, bootstrap_image, ops_function) after assuming the provisioner role. Create refuses readably ("App hosting isn't set up.") when any knob is blank. **Tables (this wave's one migration)**: `hosted_apps (id uuid pk, org_id, project_id unique fk, name, slug, description, env, template ('node'|'python'), repo_full_name, repo_id bigint, state ('provisioning'|'live'|'paused'|'handed_over'|'failed'|'deleting'), pause_reason ('over_limit'|'sponsor_ended'|'operator'|'db_size'|null), pause_cap text null, failure_step text null, failure_detail text null, last_deploy_at, last_deploy_sha, last_deploy_result, handed_over_to text null, deleted_at, created_by, created_at, updated_by, updated_at)`; unique `(env, slug) where deleted_at is null`. `hosted_app_usage (app_id, month date, requests bigint, egress_bytes bigint, db_bytes bigint, measured_at)` pk `(app_id, month)` (written by C9). These carry every column C9 needs (deploy status, hand-over), so C9 adds no migration for them. **Doors**: create, retry and slug preview require `require_switch(easy_mode)`; reads of an existing app and Easy Mode project delete stay open for an org that has hosted apps. **Routes**: `GET /api/hosted-apps/slug?name=` → `{slug, address}`; `POST /api/hosted-apps {name, template}` → 201 `{project_id, app}` — checks apps cap ("Your plan allows 2 apps."), creates the project (`easy_mode=true`, `repo_url=https://github.com/jaah-ai/<repo>`, the seven required repo-access fields (`models/projects.py:62-63`) from a code-side template catalogue next to the template ids: node — `integration_branch=main`, `branch_prefix=fleet/`, `push_enabled=true`, `gate_quick_cmd='npm ci && npm test'`, `gate_full_cmd='npm ci && npm test && npm run build'`, `stack_up_cmd=''`, `stack_down_cmd=''`; python — same branch fields, `gate_quick_cmd='pip install -r api/requirements.txt && pytest -q'`, `gate_full_cmd='pip install -r api/requirements.txt && pytest -q && (cd web && npm ci && npm run build)'`, stack `''`), assigns it via `easy_mode.assign_project`, row in `provisioning`, starts workflow `hosted-app-provision:{app_id}`; `GET /api/hosted-apps?project_id=` / `GET /api/hosted-apps/{id}` → row + usage + terms; `POST /api/hosted-apps/{id}/retry` (failed only); operator `POST /api/operator/hosted-apps/{id}/pause|unpause`, `DELETE /api/operator/hosted-apps/{id}`; (hand-over route is C9). Project delete (existing route) for an Easy Mode project requires typed confirmation of the address and runs workflow `hosted-app-delete` (repo kept). **DBOS workflow** `apps/api/app/durable/hosted_app.py` (pattern `durable/hosting_change.py:792-799`), each step idempotent (create-if-absent by name): `github_repo` (generate from template, private) → `github_repo_settings` (`allow_auto_merge=true`, `delete_branch_on_merge=true`, ruleset on `main`: PR required, required check = the template's `test` job) → `github_env` (environment `production`, deployment branch policy `main` only; `PUT /repos/jaah-ai/<repo>/actions/oidc/customization/sub {"use_default":false,"include_claim_keys":["repository_id","environment"]}` ⇒ expected `sub = repository_id:<repoId>:environment:production`; the first deploy of each template logs the decoded `sub` and the trust is pinned to the observed form before closing; trust on `sub` + `aud` only) → `aws_ecr` → `aws_role` → `ops_create_db` (ops function `create_db`; copy `database_url` into the Lambda env, never persist it) → `aws_lambda` (from `bootstrap_image`; generates `origin_key` = 32 random bytes base64url — its only stored copies are the Lambda env and the tenant parameter; Retry reads it back with `GetFunctionConfiguration`; nothing in the CP DB holds it) → `cf_tenant` (parameters `site`=`<env>/<slug>`, `fn_url_host`, `state=live`, `origin_key`; domain = address) → `github_vars` (the ten Actions variables: `AWS_REGION, AWS_ROLE_ARN, APP_SLUG, SITE_BUCKET, SITE_PREFIX, ECR_REPOSITORY, LAMBDA_FUNCTION, CF_DISTRIBUTION_ID, CF_TENANT_ID, APP_URL`) → state `live`. On a step error: `failed`, `failure_step`, readable `failure_detail`. **Pause/unpause primitive** (pinned, used by C9): `hosted_apps_service.pause(session, app, reason, cap=None)` sets tenant `state` + Lambda reserved concurrency 0; `unpause` restores `state=live` + the concurrency term. **Delete**: tenant, Lambda, role, ECR repo, S3 prefix, secret, ops `drop_db`; row `deleted_at`. **Isolation proof** (AC-12): in app A's repo, a `workflow_dispatch` job `isolation-probe` in environment `production` (OIDC as A's role) runs against B's names — added through an ordinary PR merged with the admin token; DB denial via ops `probe_connect`. Keep the probe out of the templates. ## Acceptance criteria (verbatim, BA) - **AC-9 What create makes.** Creating it yields: a project; a repo in Jaah's GitHub org created from the chosen template with its Actions variables set; one `hosted_apps` row (standard entity shape: id, name, slug, description, created_by/at, updated_by/at) linked to the project; its CloudFront tenant, S3 prefix, Lambda, database + role, secret and per-app IAM role. Partial failure leaves the app in a readable `failed` state with a Retry, never a silent half-app. A 3rd app beyond the cap is refused with a readable message (A7). → verify: dev, after create, open the project's app panel (status Live/Not deployed yet); `jaah-forge api get repos/jaah-ai/<repo>` shows `template_repository`; the provisioning API's read route lists the row. (Panel + Retry button are C9; this child verifies via the API.) - **AC-11 Push publishes.** A push to `main` of the new repo (both templates) deploys automatically; within the workflow run, `https://<slug>.dev.apps.jaah.ai/` serves the front-end (HTTP 200, HTML) and `https://<slug>.dev.apps.jaah.ai/api/<health>` answers 200 from the back-end, which reads/writes its own database (the migration step ran). The live console's apps use `<slug>.apps.jaah.ai`. The project panel shows the URL (clickable) and the last deploy (time, commit, result). → verify: dev, push a commit to each template's repo; watch the Actions run green; `curl -i` both URLs; observe the panel update. (Panel half is C9.) - **AC-24 Templates** (end-to-end half). "… each deploys green from a fresh copy without edits." → verify: AC-11 for each template. - **AC-12 Isolation (denied call).** Using app A's deploy role, each of: writing to app B's S3 prefix, updating app B's Lambda, pushing/pulling app B's ECR tags, and connecting to app B's database with A's credentials is DENIED; the broad `jaah-apps-github-actions-deploy` role is not trusted by any client repo. → verify: from app A's workflow (or an assumed session of A's role), run each call against B and observe AccessDenied / auth failure; record the four outputs. - **AC-17 Operator controls** (API). The operator can pause, unpause and delete any app of the org …; a paused-by-operator app shows its reason. → verify: dev, pause then unpause the test app; `curl` reflects each. - **AC-23 Delete** (API/workflow). Deleting an Easy Mode project (typed confirmation naming the address) removes its app, database, address and AWS resources; the repo remains in Jaah's GitHub org; the slug becomes free. → verify: dev, delete; `curl https://<slug>.dev.apps.jaah.ai` no longer serves the app; repo still exists. - **AC-1** (create route). "`POST` to any Easy Mode/hosted-app route returns 403 with the plan-switch sentence." → verify: `curl` the create-app route with a session of an org without the override, observe 403. Credentials & reach: - AC-9, AC-11, AC-19, C3 template creation, C6 GitHub writes — `me/users/roman@gmind.tech/github_admin` (classic token, `repo, workflow, admin:org`; admin of `jaah-ai`); hand-over target org `ruzhukov-genai` (admin, so transfer needs no acceptance) — verified — `GET /user/memberships/orgs/jaah-ai` = admin/active. - AC-12 isolation denied calls — S3/Lambda/ECR denials: from app A's own repo, a `workflow_dispatch` job `isolation-probe` in environment `production` (OIDC as A's role) against B's names, added to A's repo through an ordinary PR merged by the admin token; DB denial: ops `probe_connect` invoked with the management credential → apps admin role — verified — the GitHub admin token is admin of `jaah-ai` (can add and dispatch the probe job); the management credential reaches the apps account. - AC-1…AC-9, AC-14…AC-20, AC-22, AC-23, AC-3b (dev UI) — `me/jaah/debug/main/operator-login` (operator; switches into the test tenant, covers member role) on `app.dev.jaah.ai`; member view also via `org/console/dev-tpl-member-login` — verified — both on roster, `credential_get` ok; operator login `POST /api/auth/login` 200 from this node. - AC-10, AC-21 terraform apply + budgets — `me/jaah/system/bootstrap-admin` (management account) → `OrganizationAccountAccessRole` in apps account 863741479793 — verified (used here only for `probe_connect` and read-only checks). - Lambda concurrency 1,000 — AWS Service Quotas approval (filed by C1) — deferred — AWS approval; C6's deploy oracle waits for it. (Build, merge and prove everything else; if approval is still pending, settle with AC-11/AC-12 live runs named as blocked on it.) - C1 env-root grant — main deferred — promotion to `master` is a human gate; all ACs run on dev. - Live DB reads — `tools/cp_psql.sh dev` — deferred — broker refused for this task; use the console/API. Out of scope: a separate GitHub org; a separate app apex domain; custom domains, registrar purchase and paid hosting after sponsorship (draft task 10233 stays as that later work); the always-on shared-box hosting tier. Do NOT use or widen `jaah-apps-github-actions-deploy`; do NOT run client apps in the control-plane account; do NOT give tenants pause/unpause/hand-over; do NOT delete repos automatically; do NOT build the wizard UI (C7), the app panel, crons, metering, notifications or hand-over (C9); do NOT edit C1's terraform except to fix a defect it blocks on (say so in the PR). Done means: - AC-1 (create route) — dev: `curl` `POST /api/hosted-apps` as a member of an org with Easy Mode off → 403 with the plan sentence. - AC-9 — dev, test org with Easy Mode on: `POST /api/hosted-apps` for each template → 201; `GET /api/hosted-apps/{id}` lists the row in standard entity shape; `jaah-forge api get repos/jaah-ai/app-dev-<slug>` shows `template_repository`; Actions variables, environment, ruleset and OIDC subject set; tenant, prefix, Lambda, db, secret and deploy role exist (cite describe calls). A forced step failure lands `failed` with a readable `failure_step`/`failure_detail` and `POST …/retry` completes it. A 3rd app → readable refusal "Your plan allows 2 apps.". - AC-11 / AC-24 — for both templates: a push to `main` runs deploy green; `curl -i https://<slug>.dev.apps.jaah.ai/` → 200 HTML; `/api/health` → 200 with a DB round-trip; a keyless call straight to the function URL → 403; `last_deploy_*` recorded by the read route or noted for C9's poll. - AC-12 — the four denied outputs (S3 put to B's prefix, `lambda update-function-code` on B, ECR push/pull on B's repo, `probe_connect` A→B) recorded verbatim; no client repo trusts `jaah-apps-github-actions-deploy` (cite its trust policy). - AC-17 (API) — operator pause → `curl` both `/` and `/api/health` serve the readable paused page / JSON 503 naming "paused by operator"; unpause → live again. - AC-23 (API) — Easy Mode project delete with the typed address → app URL no longer serves, AWS resources and database gone (cite), repo still exists, the slug is offered again by `GET /api/hosted-apps/slug`. - `./dc check quick` green; workflow steps idempotent (test re-running each step).