Build outcome-driven routing and agent marketplace

Task 12251 · Running · Epic · created · updated

Agent choice is a fixed precedence chain (agent_resolution.py resolve_agent), the model tier is a static per-persona setting (persona_registry.py, model_catalog.py), and account placement is fewest-active-runners behind quota gates (jaah_scheduler.py) with no history, cost or complexity input. Review, security and test lanes are in-session subagents, neither attributable nor routable. Build the router over the outcome ledger. Depends on the outcome ledger epic (Task 12195 epic 1) reaching its profile phase. PRD: the attached proposal report (Task 12195), decisions table and "Epic 3". 1. Routing decision record: `routing_decisions`, one per dispatch — candidates with features and scores, the chosen configuration, mode, the human's final pick when it differed, a plain-language reason. The ledger row links to it so every decision is graded by its outcome. 2. Features: persona; complexity bucket (body length, files named, repo language, clarity score, prior similar tasks); org policy (allowed providers, model floor/ceiling, budget headroom from budgets.py); live availability (quota gates and headroom from jaah_scheduler.py, node capacity, SDK-capable nodes); per-configuration ledger statistics for the stratum, org first, operator-plane prior where the org has too little data. 3. Policy: Thompson sampling over configurations in the allowed set; objective expected survival probability per dollar with a latency cap; exploration capped at a per-project budget share, default 10%. Falls back to today's precedence chain whenever the allowed set is one, data is below the minimum, or the mode is off; with the feature OFF resolve_agent and placement stay byte-identical. 4. Modes via plan_terms like merge_lane_enabled: Shadow writes decisions and grades them against the actual dispatch, with a "router vs default" scoreboard; Recommend shows the choice and reason on the task form and the dispatch plan decision, one click to accept; Auto dispatches on it inside org policy, every decision visible and overridable on the task. The plan feature `intelligent_routing` is OFF initially; the org Admin sets the org's maximum mode; each project opts in to a mode up to that maximum (default inherits off). Shadow decision records are measurement and are always written when the ledger exists; Recommend and Auto are behaviour and need the feature plus project opt-in. 5. Routable review units: new CP-dispatched personas `reviewer`, `security_reviewer`, `tester` as blind children of a coder task for Major and security-sensitive review tiers (diff + spec only, never the author's self-assessment; may run on a different vendor than the author); Minor/Standard keep in-session lanes whose model/agent the dispatch prompt names from the routing decision and telemetry attributes. Their findings write the ledger's security-findings facts. 6. Marketplace: the agent catalog (built-in providers, org SDK agents, operator-published catalog connectors from the ACP registry) rendered with its ledger profile, eligibility per persona, cost band and availability; an org enables a catalog agent and the router may pick it. No vendor self-reporting: a listing's numbers are only the ledger's. Out of scope: changing account quota semantics or spillover; training a model from scratch; routing analysis personas before coder tasks have data; public marketplace billing; any ledger metric definition (owned by the outcome ledger epic). Done means: - With the ledger present and routing OFF, every dispatch still has a routing_decisions row in Shadow and the scoreboard shows router vs default survival and cost over a period; resolve_agent and placement are byte-identical to today (regression test). - In Recommend, the task form and dispatch plan decision show the choice with its reason; the accepted/overridden rate is recorded. - In Auto on a project with two allowed providers, both receive dispatches over time with the exploration share bounded by the project knob; an org Admin lowering the org maximum demotes every project above it. - A Major-tier coder task spawns a blind `reviewer` child on a different provider when policy asks; its findings land on the ledger row. - The marketplace page shows each catalog agent with ledger numbers only and no self-reported figure. - Required checks: ./dc check quick green; changed-area tests green; merge lane full gate green.

Subtasks

Attachments

Activity

  • Created
  • Updated
  • Runner started
  • Ready
  • Ready
  • Ready
  • Runner started
  • Awaiting response