Fix CP autoscaling and memory distill retry
From the lag analysis in Task 13902 (report attached, items 1 and 3). Production console reads wait ~2 s in the app because the single-process API sits at ~0.9 core, while the autoscaler waits for 60% of a 2-vCPU task (~1.2 cores), so it never scales out.
1. Make prod CP autoscaling see the real limit. Either lower the jaah-main-cp service's CPU target to about 35%, or move the API to 1-vCPU tasks with a higher minimum count. Apply the same rule to dev. The change goes in infra/jaah/terraform/.
2. Fix the `memory distill: task=N transcript unreadable (AttributeError)` failure in apps/api/app/project_memory/distill.py. Prod logs it 143,868 times in 24 h, about 100 per minute. It is a code bug, not a missing transcript. After a task's first failure, back off instead of retrying it every tick for TRANSCRIPT_WAIT (24 h).
Out of scope: node polling intervals, relay behaviour, client changes. Those are separate tasks.
Done means:
- The terraform plan shows the new scaling target or task size. After the deploy, the service runs more than its minimum of 2 tasks whenever the API process is above ~0.7 core.
- A regression test reproduces the AttributeError on a real transcript shape and passes after the fix.
- A failing transcript is retried with backoff, not on every tick.
- Prod logs show the distill warning rate drop by more than 95%.
Outcome
PR #7973 merged (cf2280f5). CP CPU target is now 35% on main (70% on dev, still off). Fixed the distill AttributeError (missing manifest key) and added a 5 min→4 h retry backoff. Quick gate and tests green; dev apply succeeded. Still to check after master promotion: prod plan, scale-out, warning-rate drop. Memory: proposed 2.
Artifacts
Attachments
Activity
- Created
- Updated
- Run
- Runner started
- Awaiting response
- Awaiting response
- Runner resumed
- Runner ended (done)