Make gmind a tenant: narrow the operator plane

Task 5198 · Done · Epic · created · updated

Follow-up to Task 5190 (step 1 merged: #4569; gmind tenant exceptions pre-staged in revision 0370). Goal: `apps/api/app/tenancy/ids.py::OPERATOR_PLANE_ORG_IDS` = org zero only, so gmind (`DEFAULT_ORG_ID`, slug `gmind`) is an ordinary tenant and `operator` is the one operator org. Known so far: 1. Product change is small: `OPERATOR_PLANE_ORG_IDS` → `{ORG_ZERO_ID}`; `app/seed.py` bootstrap admin + superadmin fixture logins seated in org zero (they keep an org-zero seat, like revision 0369 gave every default superadmin); comments naming "the default org" as operator. 2. Test fallout is large: flipping it fails 1,718 API tests (~200 files — credentials, access_explain, virtual_nodes, notify_targets, crons, billing authz, commands…). Root cause: fixtures use one org as both "the operator's" and "the tenant under test" (`apps/api/tests/conftest.py` `_SEAT_SHAPES["operator"]`, `FIXTURE_ORG_ID`, the seeded admin). Needs a fixture redesign, not per-test patches. 3. Behaviour changes that columns cannot neutralise (decide, don't discover): - gmind token usage becomes billable under Jaah mode (`token_billing/metering.py::_billable`). - New gmind cloud nodes signed by an operator AWS credential are billable/refused (`billing/hosted.py`), get `tag:org-gmind`+`tag:tenant` and no fleet EFS share (`cloud_node_launch.py`). On dev the fleet AWS credential `aws-gmind-skynet` is now in org zero and gmind's node group still names it. Needs a gmind-owned AWS credential or an operator call. - Tenant credentials with an unpinned OAuth token endpoint stop refreshing (`oauth2_tokens.py`). - Existing nodes keep their tags and share snapshot (no auto retag). 4. Prod: revisions 0369/0370 reach prod on the next master promotion; verify the same way as dev (org renamed, system creds + Multi-Node Hosts in operator, operator seats). Done means: flag flipped, `./dc check full` green, dev verified (gmind nodes heartbeat, dispatch works, OAuth accounts dispatch, crons fire), operator still administers the fleet from `operator`.

Subtasks

  • [business_analysis] Make gmind a tenant: narrow the operator plane Done
    Settled 13 requirements (flag→org zero, seed seats, 2-org fixture redesign, gmind billed, AWS cred copied to gmind, fleet EFS→gmind, gmind nodes retagged, no gmind cred breaks, knob creds→org zero, dev verified, prod checklist). 11 operator answers in PRD.
  • [architecture] Make gmind a tenant: narrow the operator plane Done
    Data-first cutover: C2 OAuth2 destination rule (all orgs), C3 gmind boot step (AWS copy, EFS adopt), C4 knob-cred migration, C5 retag+ACL (gated) → C1 flip+fixtures in org zero, merged last → C6 dev verify + prod checklist.
  • [plan] Make gmind a tenant: narrow the operator plane Done
    Split along the architecture's cutover seams into 6 coders in 3 phases. Groundwork: C2 OAuth2, C3 boot step, C4 knob migration (after C3, to order revisions), C5 retag+ACL. Then C1 flip+fixtures, then C6 dev verify + prod checklist. Critic fixes applied.
  • OAuth2 token-endpoint destination rule, every org Done
    Row-supplied OAuth2 Token URLs now judged by destination (https + public) in every org; refresh + connect share one helper. Full gate green, dev census 0 rows, security gate approved. PR #4601 merged. Deferred: /test + wizard refusal.
  • gmind tenancy boot step, adopted share Done
    gmind tenancy boot step built (copy+reseal, defaults, gated org-zero clear, repoint, adopted store + 409 guard, rev 0372, read-only UI). Full gate + CI green; PR #4606 merged. 409 wording tenant-safe (no "fleet").
  • Move operator-knob credentials to org zero Done
    Rev 0373 moves knob-named gmind credentials (+children) to org zero; still-shared ones stay with the pinned log line. Tests cover move/stay/untouched/decrypt; single head 0373. Full gate + CI green; PR #4609 merged.
  • Tailnet retag action and gmind ACL Done
    Retag converge + POST /api/orgs/{id}/tailnet-retag + tag:org-gmind 2049 seed merged (#4603), full gate green. Live dev policy edit applied on operator approval, read back; no device retagged. Follow-up for C6: gmind SSH rule.
  • Flip operator plane, redesign fixtures Done
    Operator plane is now org zero only; gmind is a tenant. Seeds give superadmins an org-zero seat; fixtures redesigned (FIXTURE_ORG_ID=org zero). Full gate green, dev checks 1–3 pass, security gate approved. PR #4624 merged. Deferred: system actor still in gmind.
  • Dev cutover verification, prod checklist Done
    Dev retag done (2 gmind devices → org-gmind+tenant); R5/6/7/8a/8e/9/10a/d/e/11/13 pass on dev. Prod checklist merged (#4628), gate green. Blocked: gmind dispatch (nodes stopped reporting worktrees Oct 7, pre-flip); R8b/d mount check (no aws-gmind-skynet).
  • [finish] Make gmind a tenant: narrow the operator plane Done
    All 6 coders merged; flip live on dev, operator plane = org zero. Ask unmet: gmind dispatch dead on dev (enumeration frozen since Oct 7) → filed 5313; retag dropped operator SSH to gmind nodes → filed 5314.
  • Restore gmind dispatch on dev: nodes stopped Done
    Restored gmind dispatch on dev with a node-side fix, no code change. A tailnet route covering the overlapping VPC CIDR hung the EFS mount, which wedged jaahd's reconcile. Pinned the mount-target IPs to the VPC on dev1/pool-b-host. All 3 nodes fresh; task 4792 active on pool-b via shared OAuth cod1. Follow-ups named in report.
  • Restore operator SSH into gmind nodes after the tenant Done
    Added ssh rule tag:operator-host→tag:org-gmind (rz only) to seed + prod checklist 1a; operator approved, live on dev headscale (read back). Full gate green; PR #4632 merged. Dev has no operator-host device yet, so no e2e SSH test.

Artifacts

Activity

  • Created
  • Updated
  • Runner started
  • Ready
  • Ready
  • Ready
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Runner ended (done)
  • Runner started
  • Runner started
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner ended (done)
  • Runner started
  • Runner ended (done)
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Ready
  • Ready
  • Runner ended (done)
  • Runner started
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner ended (done)
  • Runner ended (done)