Fix CP autoscaling and memory distill retry

Task 13920 · Done · Coder · created · updated

From the lag analysis in Task 13902 (report attached, items 1 and 3). Production console reads wait ~2 s in the app because the single-process API sits at ~0.9 core, while the autoscaler waits for 60% of a 2-vCPU task (~1.2 cores), so it never scales out. 1. Make prod CP autoscaling see the real limit. Either lower the jaah-main-cp service's CPU target to about 35%, or move the API to 1-vCPU tasks with a higher minimum count. Apply the same rule to dev. The change goes in infra/jaah/terraform/. 2. Fix the `memory distill: task=N transcript unreadable (AttributeError)` failure in apps/api/app/project_memory/distill.py. Prod logs it 143,868 times in 24 h, about 100 per minute. It is a code bug, not a missing transcript. After a task's first failure, back off instead of retrying it every tick for TRANSCRIPT_WAIT (24 h). Out of scope: node polling intervals, relay behaviour, client changes. Those are separate tasks. Done means: - The terraform plan shows the new scaling target or task size. After the deploy, the service runs more than its minimum of 2 tasks whenever the API process is above ~0.7 core. - A regression test reproduces the AttributeError on a real transcript shape and passes after the fix. - A failing transcript is retried with backoff, not on every tick. - Prod logs show the distill warning rate drop by more than 95%.

Outcome

PR #7973 merged (cf2280f5). CP CPU target is now 35% on main (70% on dev, still off). Fixed the distill AttributeError (missing manifest key) and added a 5 min→4 h retry backoff. Quick gate and tests green; dev apply succeeded. Still to check after master promotion: prod plan, scale-out, warning-rate drop. Memory: proposed 2.

Pull request

Artifacts

Attachments

Activity

  • Created
  • Updated
  • Run
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)