Build apps-account hosting infrastructure (Terraform)

Part of epic "Build sponsored Easy Mode app hosting" (PRD task 13412, architecture task 13413). Architecture child **C1 Infra**. ## Original requirement (verbatim) > Give starting clients a very cheap, Jaah-sponsored way to build and host small web apps (like Lovable), hosted in our existing apps AWS account. > Hosting: serverless in the existing apps AWS account — static front-end on S3 + one shared multi-tenant CloudFront distribution, back-end container on Lambda (Lambda Web Adapter), one database per app on one shared small RDS Postgres. Address `<app-slug>.apps.jaah.ai` (existing apps zone). Any small app with a back-end. > 1. Infra (Terraform, apps account): multi-tenant CloudFront + wildcard cert for `*.apps.jaah.ai`, shared ECR, shared RDS Postgres in private subnets, NAT instance, control-plane provisioner role, per-app role module, budget alarm. PRD D9 (binding): apps made from the dev console publish at `<slug>.dev.apps.jaah.ai`; apps from the live console at `<slug>.apps.jaah.ai`. Same apps AWS account; the two never clash. The certificate must also serve `*.dev.apps.jaah.ai`. ## Approach (settled by the Architect — realize, do not reopen) **Operator decisions:** Q1 lift VPC Block Public Access for the apps account: in `infra/jaah/terraform/org` set `block_vpc_internet_gateways = false` on `module.baseline_apps` (`org/baseline.tf:114-129`) and add the apps account to the EC2.172 accepted-risk rule R-SOC-37 (`org/security-services.tf:985-1040`). The apps SCP is NOT edited. Q4: in the org root add `aws_servicequotas_service_quota` Lambda concurrent executions `L-B99A9384` = 1000 — file it FIRST (AWS approval gates the provisioning child's deploy oracle). **New root `infra/jaah/terraform/apps-hosting/`** (provider assumes `OrganizationAccountAccessRole` in the apps account from management credentials, same as `org/versions.tf:193-200`; state in the org state bucket, own key; README for the root, routed per the context-architecture policy). Contents: - VPC 10.80.0.0/16: 2 private subnets (Lambdas, RDS, ops), 1 public subnet with a t4g.nano NAT instance (source/dest check off) + IGW. - RDS Postgres 16 `db.t4g.small`, single-AZ, 20 GB gp3, 7-day backups, private, SG from the Lambda SG only; master secret RDS-managed in Secrets Manager. - S3 `jaah-ea-sites-<acct>` private, prefixes `<env>/<slug>/`; bucket policy for the distribution (OAC to S3, explicit since origin path is parameterised). - ACM cert `*.apps.jaah.ai` + SAN `*.dev.apps.jaah.ai` (DNS-validated in the `apps.jaah.ai` zone); Route 53 wildcard aliases for both names → the connection group. Existing delegated child zones (e.g. `yumapos.apps.jaah.ai`) keep winning over the wildcard. - Multi-tenant CloudFront distribution: default behavior → S3 origin, origin path `/{{site}}`; behavior `/api/*` → custom origin `{{fn_url_host}}`, all methods, caching disabled. No static origin header. - CloudFront Function (viewer-request, JS 2.0) associated with BOTH the default and `/api/*` behaviors: sets `x-jaah-origin = origin_key`; if `state != live` return the paused page (HTML for `/`, JSON 503 for `/api/*`) naming the reason (over limit / sponsorship ended / paused by operator / database too large); else SPA fallback (`/` and extension-less paths → `/index.html`). - Standard logging v2 → S3 (parquet) with `distribution-tenant-id`; Glue table + Athena workgroup for metering. - Permissions boundary `ea-app-boundary` and the per-app role module = the boundary + a JSON policy template stored in config (per-app roles are created by the control plane at runtime — documented deviation of "per-app role module"). - Shared Lambda execution role `ea-<env>-runtime` (logs + VPC ENI only). - ECR: one repository PER APP `ea/<env>/<slug>` (deviation from "shared ECR": IAM cannot fence image tags) via a repository creation template for prefix `ea/` (keep last 5 images, scan on push, repo policy granting `lambda.amazonaws.com` `ecr:BatchGetImage`/`GetDownloadUrlForLayer` with `aws:SourceAccount` = apps account); repository `ea/bootstrap` holding placeholder image `ea/bootstrap:latest` (200 on `/api/health`) that new Lambdas start from. - Ops function per env `ea-dev-ops`, `ea-main-ops` (Python, in VPC, code under `infra/jaah/terraform/apps-hosting/ops/`), each touching only its own env prefix. Contract (pinned, JSON in/out, errors `{"ok":false,"error":"<readable>"}`): `create_db {slug, conn_limit:5}` → `{ok, database:"ea_<env>_s", user, secret_arn (ea/<env>/<slug>/db), database_url}` (idempotent; login role `CONNECTION LIMIT 5`, `REVOKE CONNECT … FROM PUBLIC`); `drop_db {slug}` → `{ok}` (terminates sessions, drops db+role+secret); `usage {month:"YYYY-MM"}` → `{ok, apps:{<slug>:{requests, egress_bytes, db_bytes}}}` (requests = Σ CloudWatch `AWS/Lambda Invocations` for `ea-<env>-<slug>`; egress = Athena Σ `sc-bytes` by `distribution-tenant-id`; db = `pg_database_size`); `probe_connect` (dev only) `{as_slug, target_slug}` → `{ok, denied:true, detail}`. - Provisioner roles `ea-dev-provisioner` (trusted by the dev CP task role) and `ea-main-provisioner` (prod CP task role), actions exactly (`<P>` = `ea-<env>`): config `ssm:GetParameter`+`kms:Decrypt` on `parameter/jaah/ea/<env>/config`; ECR `DescribeRepositories, CreateRepository, DeleteRepository, TagResource, SetRepositoryPolicy, GetRepositoryPolicy` on `repository/ea/<env>/*`; IAM `CreateRole` (cond. `iam:PermissionsBoundary` = boundary), `PutRolePolicy, GetRolePolicy, DeleteRolePolicy, UpdateAssumeRolePolicy, DeleteRole, GetRole, TagRole` on `role/<P>-*-deploy`; Lambda `CreateFunction, GetFunction, GetFunctionConfiguration, UpdateFunctionConfiguration, UpdateFunctionCode, PutFunctionConcurrency, DeleteFunctionConcurrency, CreateFunctionUrlConfig, GetFunctionUrlConfig, DeleteFunctionUrlConfig, AddPermission, RemovePermission, DeleteFunction, TagResource` on `<P>-*`, `iam:PassRole` only `runtime_role_arn` to `lambda.amazonaws.com`, `ec2:Describe{Subnets,SecurityGroups,Vpcs}` on `*`; `lambda:InvokeFunction` on `function:<P>-ops`; CloudFront `ListDistributionTenants, CreateDistributionTenant, GetDistributionTenant, UpdateDistributionTenant, DeleteDistributionTenant, CreateInvalidationForDistributionTenant` on the one distribution and its tenants; S3 `ListBucket` (prefix `<env>/*`), `DeleteObject` on `bucket/<env>/*`; `route53:ListResourceRecordSets` on the `apps.jaah.ai` zone. - SSM `/jaah/ea/<env>/config` (SecureString JSON, one per env, the single home of infra facts): `{region, account_id, bucket, distribution_id, distribution_arn, connection_group_domain, zone_suffix ("dev.apps.jaah.ai"|"apps.jaah.ai"), subnet_ids, lambda_sg_id, runtime_role_arn, boundary_arn, oidc_provider_arn, ecr_registry, bootstrap_image, ops_function}`. - Tenant parameters (the only list): `site` = `<env>/<slug>`, `fn_url_host`, `state` ∈ `live|over_limit|sponsor_ended|operator|db_size`, `origin_key`. - AWS Budgets monthly on the apps account, 80%/100% actual + forecast, email to the operator address. - **Environment roots** (`infra/jaah/terraform/environments/{dev,main}` ECS task role): `sts:AssumeRole` on that env's provisioner ARN only. - Dev and live are fenced by name prefix + tag (`ea-dev-*` / `ea-main-*`, tag `JaahEnv`) and separate provisioner roles and ops functions. **Risk R1 — do this first:** a one-tenant spike proving S3 via `/{{site}}`, `/api/*` via `{{fn_url_host}}`, the function's `x-jaah-origin` arriving at the Lambda, and the paused page on both behaviors. Fallback if multi-tenant cannot do it (Lambda@Edge not used): one classic distribution per app behind the same pinned names and the same function — record it in the README and the PR. Plan before apply; apply is part of Done (management credential below). ## Acceptance criteria (verbatim, BA) - **AC-10 Infra applied.** The apps account carries (applied, not just planned) the shared multi-tenant CloudFront, a certificate serving `*.apps.jaah.ai` and `*.dev.apps.jaah.ai`, shared ECR, shared RDS Postgres in private subnets, a NAT instance, the control-plane provisioner role, the per-app role module and a budget alarm; `terraform plan` on that root is clean afterwards. → verify: `terraform plan` shows no changes; AC-11 succeeds end to end. - **AC-21 Budget alarm.** A budget alarm on the apps account notifies the operator. → verify: AWS Budgets lists it with an operator recipient. Credentials & reach: - AC-10, AC-21 terraform apply + budgets — `me/jaah/system/bootstrap-admin` (management account) → `OrganizationAccountAccessRole` in apps account 863741479793 — verified — `sts get-caller-identity` ok; assume-role ok; read-only `describe-vpcs`, `lambda get-account-settings`, `acm list-certificates` from this node. - C1 env-root grant (CP task role → provisioner) — dev: `jaah-deploy.yml` applies `environments/dev` on merge to develop (Environment-pinned OIDC role); main: applied by the same workflow on promotion to `master` — dev verified (workflow path, `jaah-deploy.yml:440-470`); main deferred — promotion to `master` is a human gate, so live-console Easy Mode waits for it; all ACs run on dev. - Lambda concurrency 1,000 — AWS Service Quotas approval (filed by C1) — deferred — AWS approval; C6's deploy oracle waits for it. Out of scope: a separate GitHub org; a separate app apex domain; custom domains, registrar purchase and paid hosting after sponsorship (draft task 10233 stays as that later work); the always-on shared-box hosting tier. Do NOT edit the apps account's SCPs or DNS query logging; do NOT use or widen `jaah-apps-github-actions-deploy`; do NOT run client apps in the control-plane AWS account. Do NOT create any per-app resources, control-plane code, routes or Settings knobs (provisioning child C6), nor the template repos (C3). Done means: - AC-10 — `terraform plan` on `org/` and `apps-hosting/` both show no changes after apply (cite the outputs); every listed component exists in the apps account (cite describe calls). The "AC-11 succeeds end to end" half is proven by the provisioning child C6. - AC-21 — `aws budgets describe-budgets` (apps account) lists the budget with the operator email at 80%/100% actual + forecast. - R1 spike — a throwaway tenant answers `/` from S3 and `/api/*` from a Lambda function URL that logs the received `x-jaah-origin`; setting the tenant's `state` to `over_limit` serves the readable paused page on `/` (HTML) and `/api/x` (JSON 503); spike tenant removed afterwards. Cite the curl outputs. - SSM `/jaah/ea/dev/config` and `/jaah/ea/main/config` exist with every pinned key (cite key names, not values). - Ops function `ea-dev-ops`: `create_db` then `drop_db` on a throwaway slug both return `ok:true`; `usage` for the current month returns `ok:true`. - Lambda quota request `L-B99A9384` = 1000 filed (cite request id/status). - `environments/dev` grant deployed (Jaah Deploy green); `ea-dev-provisioner` assumable from the dev CP task role (cite a successful `ssm get-parameter` under it, value not printed).