Stop doubling reads on slow servers

Task 13922 · Running · Coder · created · updated

From the lag analysis in Task 13902 (report attached, items 4 and 5). 1. Stop the read relay from doubling reads on a slow server. 1,085 of one user's 1,270 slow reads in 24 h ran in "suspect + dropping" relay mode. In that mode every GET races at once over the WebSocket relay, and the relay replays it through the full app (apps/web/src/net/relay.ts around lines 130-148, apps/api/app/routes/read_relay.py around lines 271-329). A slow server therefore gets twice the load from that user. Race only after a failed probe. Don't treat a slow 2xx whose `Server-Timing` shows server time as a network fault. Cancel or drop the losing copy. 2. Keep `Server-Timing` on relayed answers (read_relay.py `_ANSWER_HEADERS`). 3. Add an EMF event-loop lag gauge, and a request-admission wait metric, to the API. Add a CloudWatch alarm on loop lag, and offer it as a scale-out signal. Out of scope: autoscaling targets, node polling, chunk loading. Done means: - A unit test shows a slow-but-answering direct read does not set suspect mode or race every later read. - Relayed answers carry `Server-Timing`. - Loop-lag and admission-wait metrics appear in CloudWatch for dev, and the alarm exists in terraform.

Outcome

Pull request

Attachments

Activity

  • Created
  • Updated
  • Run
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed