Implement ECS Backend CPU Profiling Telemetry

Task 5389 · Done · Coder · created · updated

The backend (ECS) seems to have abnormally high CPU usage. To analyze that implement internal profiling or telemetry logging to understand which exact parts and little pieces consume CPU cycles. Perhaps we could use third party tools such as AWS XRay or something similar from open source. Propose the most robust solution that also minimizes the number of additional lines of code. We don't necessarily need a graphical representation, but we need to be able to intelligently measure down to smallest parts by queries logs or db.

Outcome

Built py-spy CPU profiling for the CP api: a Settings switch (off by default) logs the hottest stacks as JSON to CloudWatch every minute, queryable in Logs Insights. Adds SYS_PTRACE on ECS. Full gate green; PR #4694 merged. Needs a deploy.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)