Cloud nodes: auto-create + attach a tagged SSM

Task 5725 · Done · Coder · created · updated

Operator ask (Task 5722): "we must create (if does not exist yet - automatically) and attach instance profile with SSM roles and whatever else is needed. Everything Jaah creates must be properly tagged so that we know what is ours." Why: cloud node host `skynet-node-aws1-host` (i-034524217781d116e, jaah-hosting account 664831647587) filled its 100 GB root disk, and nobody could get a shell on it. It was launched with NO instance profile, so SSM was unavailable, and tailnet SSH is closed. During Task 5722 a profile was created and attached by hand: role + instance profile `jaah-node-ssm` (trust ec2.amazonaws.com, managed policy AmazonSSMManagedInstanceCore, tags ManagedBy=jaah, jaah:purpose=node-ssm, jaah:created-by-task=5722). Adopt it or replace it. Don't leave two. Current state (recon, file:line): - `apps/api/app/cloud_node_launch.py:864` passes `connection.instance_profile or None`. That is an operator-typed free-text column, `apps/api/app/models.py:4541`, default "". Nothing creates a role or profile (code or terraform). The only `aws_iam_instance_profile` is the headscale host, `infra/jaah/terraform/headscale.tf:513`. - The launch credential policy `infra/jaah/terraform/node-launcher.tf:33-74` has no `iam:PassRole`. The hosting role in `infra/jaah/terraform/hosting/iam.tf:28-36` also launches. Check which one launches into which account. - Tags: the instance gets `Name=skynet-node-<name>`, `ManagedBy=skynet-cp`, `SkynetNode=<name>` (`apps/api/app/aws/ec2.py:86,260,308`; `cloud_node_launch.py:863,878`). The root EBS volume and the ENI are UNTAGGED (`ec2.py:252-260`). The node SG `jaah-node` uses ManagedBy=skynet-cp (`apps/api/app/cloud_connection_network.py:27`). The EFS FS/SG/access point use `skynet:org_id` plus ManagedBy=skynet-cp (`apps/api/app/org_shared_store.py:37,183`). Requirements: 1. On cloud node launch (cloud nodes, node groups, hostings/co-located hosts), the CP ensures an instance profile exists in the target account and region. If missing, it creates a role and instance profile automatically: trust ec2.amazonaws.com, AmazonSSMManagedInstanceCore, plus whatever else a node genuinely needs. The CP passes the profile in RunInstances. An operator-set `instance_profile` still overrides it. Creation is idempotent and race-safe. 2. Existing running cloud nodes without a profile get one associated (a reconcile pass or a one-shot backfill), so every Jaah host is reachable over SSM. 3. Grant the launch credential(s) the minimal IAM actions needed (scoped `iam:PassRole` on the Jaah role ARN, plus `iam:Get/Create*Role|InstanceProfile`, `AddRoleToInstanceProfile`, `AttachRolePolicy`, `TagRole/TagInstanceProfile`, `ec2:AssociateIamInstanceProfile`/`Describe*`), restricted to a Jaah name/path prefix (e.g. path `/jaah/`). Update the terraform. 4. Tagging: EVERY AWS resource Jaah creates carries one consistent Jaah ownership tag set (e.g. `ManagedBy=jaah` plus a node/org identifier), applied at creation through TagSpecifications. That covers instances, root EBS volumes, ENIs, security groups, IAM roles and instance profiles, and EFS FS/SG/access points. Migrate the legacy `skynet-cp`/`SkynetNode`/`skynet:org_id` naming per the rename policy (`apps/jaahd/README.md` § legacy names; gate `tools/name_residue.toml`). Anything that finds resources by tag must keep finding old ones during the transition. 5. Docs: single-home the "what Jaah creates in a customer account and how it's tagged" list in the existing cloud-node doc home. Don't copy it. Out of scope: diagnosing or cleaning the aws1-host disk (Task 5722). Changing the root volume default size. Any non-AWS forge or cloud. Done means: - A fresh cloud node launched in an account with no Jaah profile comes up with `jaah-node-ssm` (or the chosen name) attached and shows Online in SSM (`aws ssm describe-instance-information`). - An existing profile-less running node gets one associated by the reconcile/backfill. - `aws ec2 describe-volumes`/`describe-network-interfaces`/`describe-security-groups` and `aws iam list-role-tags` on the created resources all show the Jaah ownership tags. - Unit tests cover the ensure-profile path (exists / missing / race) and the TagSpecifications for every resource type. - `./dc check full` green.

Outcome

Cloud launches ensure+pass tagged SSM profile jaah-node-ssm (agent-only policy); node-ssm cron backfills; one ManagedBy=jaah tag set. PR #5003 merged, dev deploy green. Gate green bar 3 pre-existing flakes. Pending: human TF apply, live SSM check.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)