Project memory store, switch and gates

Epic: Centralized project memory for runners, behind a per-task switch. This child is C1. It is the foundation every other memory child builds on: one migration, the `apps/api/app/project_memory/` package core, the switch resolver, the capture gates, events and bundle compile. Design sources: the BA PRD (Task 5403, artifact `prd`) and the settled architecture (Task 5404, artifact `architecture`). Fetch them with `task_read_artifact` if a detail below is not enough. The R-*/AC-*/N-*/D-*/AD-* ids refer to them. The contracts pinned below are binding, because sibling children code against them in parallel. ## Epic requirement (verbatim) > Runners re-derive what other runners already learned: the only cross-task memory today is recon_map (direct dependents inside one epic), OMC notepad/wiki dies with the worktree, and nothing is pooled or searchable across tasks or nodes. Build a centralized project-memory layer per the attached architecture ("Jaah Project Memory") and its companion comparison, which lists seven amendments to fold in (exclusion policy at capture, decay by default, generated repo wiki feed, graduation to guards before docs, read-rate as the kill metric, no `decision` kind, re-inject after compaction). The attachments are research input: the BA and Architect own the design and may depart from them with reasons. > 1. Learned memory is a control-plane home: project-scoped, provenance-stamped, trust-scored units every runner can contribute to and search; canon (repo docs, directives, rules) stays in git and is never copied. > 5. The switch: a "Project memory" setting in the task/epic Advanced section, inherited by an epic's children; its default is an operator System Settings entry. OFF means dispatch, hooks, tools and settle behave exactly as today, byte-for-byte on the prompt and with no new hook output. > 7. Robust at scale: concurrent writes from many runners on many nodes, bounded prompt budget shared with the router's caps, kill switch via runner flags, no secret ever persisted, injected units rendered as untrusted data. ## Settled requirements this child implements (BA, verbatim) - R-UNIT-1 Learned memory lives in ONE control-plane home (DB), project-scoped, every row carrying: project, scope (`project` only — `org` reserved, N-3), kind, title (≤80 chars), body (bounded, injectable whole), related repo paths, personas, provenance (feed, task id, session id, persona, model, observed commit sha), status, trust score, validity interval, standard entity shape (id/name-or-title/created_by/created_at/updated_by/updated_at per `docs/standards/coding-standards.md`). - R-UNIT-2 Kinds are exactly `codemap`, `gotcha`, `recipe`, `env`, `qa`. There is no `decision` kind (A6). - R-UNIT-3 Canon (repo docs, directive blocks, `.claude/rules`, ADRs, skills) is never copied into the store; a unit may point at a canon path, never restate it. - R-UNIT-4 Every runner of the project, on any node, can contribute and search. Reads obey the same visibility rule as `task_read`/`task_search`; writes are stamped with the calling task/session derived by the daemon. - R-SW-2 The default is an operator System Settings entry "Project memory" (On/Off) in the existing operator-settings catalogue (`operator_settings.py`) … resolved SERVER-SIDE at dispatch, not as a create-form prefill (D-1). - R-SW-3 Resolution order at dispatch: task's own explicit On/Off → else its parent epic's explicit On/Off → else System Settings default. Inheritance reaches ALL of an epic's children (every persona) (D-2). - R-SW-4 The effective value is resolved once per dispatch attempt and recorded on the task (the A/B arm). - R-SW-7 The System Settings entry ships ON (D-4). - R-CAP-5 Exclusion policy at capture (A1): before any other curation, every candidate from every feed is rejected if it is a session summary, task state (PR numbers, "what was done"), a fact one grep away, a duplicate of canon, or a secret. A `gotcha` must carry a reproduction. Rejected candidate text is not persisted (counts only). - R-CUR-1 Secret gate: every candidate from every feed passes a server-side secret check (transcript redaction patterns + model check) before persistence; a hit discards the candidate (no quarantine store of its text). (This child: the pattern pass. The model check is C6a.) - R-CUR-2 Dedupe: exact (content hash) … candidates merge into … the existing unit; no duplicate accepted units. (This child: exact dedupe. Near-dupe is C6a.) - R-SC-1 Concurrent writes from many runners on many nodes never lose a write, never create duplicate accepted units for the same content, never double-count a vote. - R-SC-3 Kill switch: a new runner flag in the existing runner-flags mechanism disables all memory delivery and capture fleet-wide (this child adds the flag; consumers honour it). - R-SC-4 No secret ever persisted — in units, rejected candidates, bundles, node cache, logs, recon_map additions, or console. - R-DEL-6 Native CLI memory files are never a home (state it in the package docstring; nothing to build). ## Approach (architecture, binding) - **Package** `apps/api/app/project_memory/` with these modules: `vocabulary.py` (the ONE home of every constant, status set and predicate), `models.py`, `switch.py`, `ingest.py`, `gates.py`, `events.py`, `bundle.py`. The feature is documented in its own home, the `__init__.py` docstring (AD-19). It makes no hub, router-row or canon edits. That docstring also records the rollout-only runner-flags `_usable` caveat. - **Migration** `0388_project_memory.py`. Use the next free number if 0388 is taken. It is the ONLY migration in this epic, and it must be additive. Schema: ``` project_memories EntityMixin(id uuid7, name varchar(80)=title, description NULL, created_by/at, updated_by/at) org_id, project_id FK projects ON DELETE CASCADE, number int, UNIQUE(project_id, number) scope text DEFAULT 'project' CHECK in ('project','org') kind text CHECK in KINDS = ('codemap','gotcha','recipe','env','qa') body text CHECK char_length(body) <= 1200 paths, personas, keywords, commands text[] DEFAULT '{}' feed text CHECK in ('propose','settle','stop_delta','distill','codemap') status text CHECK in ('proposed','accepted','flagged','stale','retired','graduated') injectable bool DEFAULT false, status_reason text NULL trust real DEFAULT 0.5, confirms int DEFAULT 0, flags int DEFAULT 0 anchor_sha text NULL, anchor_blobs jsonb NULL, verified_at timestamptz NULL, verified_sha text NULL valid_from timestamptz, valid_to timestamptz NULL, superseded_by uuid NULL FK self slice bool DEFAULT false graduation_task_id int NULL FK tasks ON DELETE SET NULL, graduated_to text NULL content_hash char(64), embedding vector(768) NULL, embedding_model text NULL tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', name||' '||body||' '||array_to_string(keywords,' '))) STORED accepted_at, last_injected_at, last_used_at timestamptz NULL UNIQUE INDEX (project_id, kind, content_hash) WHERE status IN LIVE_STATUSES -- ('proposed','accepted','flagged','stale') HNSW(embedding vector_cosine_ops); GIN(tsv); GIN(paths); (project_id, status, injectable) memory_sources (unit_id FK CASCADE, task_id FK SET NULL, session_id text, persona, model, feed, observed_sha NULL, created_at) UNIQUE(unit_id, task_id) memory_votes (unit_id FK CASCADE, task_id FK CASCADE, verdict ('confirm','flag'), source ('tool','citation','stale'), created_at) UNIQUE(unit_id, task_id) memory_events (id bigserial, project_id, task_id FK CASCADE, unit_id FK CASCADE, kind ('injected','read','cited','followed'), channel ('prompt','hook','tool','summary','pr'), provisional bool DEFAULT false, session_id text NULL, at) UNIQUE(task_id, unit_id, kind) memory_capture_log (task_id FK CASCADE, feed, done_at, produced int, rejected int, PK(task_id, feed)) memory_rejections (project_id, feed, reason, day date, count int, PK(project_id, feed, reason, day)) memory_claims (project_id, sha, job ('verify','codemap'), task_id, claimed_at, done_at NULL, PK(project_id, sha, job)) -- expiry 30 min project_memory_state (project_id PK FK CASCADE, next_number int DEFAULT 1, bundle_revision bigint DEFAULT 0, codemap_sha NULL, verified_head NULL, canon_sha NULL) project_memory_canon (project_id FK CASCADE, sha, route_id, doc_path, triggers jsonb, directive_text, PK(project_id, route_id)) tasks.project_memory bool NULL; tasks.memory_arm bool NULL; tasks.memory_arm_source text NULL ('task','epic','default','holdout','killed') sessions.memory_arm bool NULL ``` - **Python API**, pinned. Siblings import exactly these names: ```python # vocabulary.py KINDS; FEEDS; STATUSES; LIVE_STATUSES; RELATION_TERMS TITLE_MAX = 80; BODY_MAX = 1200; PROMPT_CAP_CHARS = 4000; PROMPT_MAX_UNITS = 8 TRUST_FLOOR = 0.35; DECAY_DAYS = 30; DECAY_DAYS_CONFIRMED = 90; VERIFY_MAX_AGE_HOURS = 24 BUNDLE_MAX_UNITS = 500; BUNDLE_MAX_BYTES = 256 * 1024; CODEMAP_BUNDLE_SHARE = 0.2 HOOK_SESSION_BUDGET = 12; HOOK_EVENT_MAX = 4; COMPARISON_MIN_N = 10 @dataclass(frozen=True) class Candidate: kind; title; body; paths: tuple = (); personas: tuple = (); source_text: str|None = None # verbatim check only; never stored @dataclass(frozen=True) class IngestResult: unit_id; number; status; rejected def effective_memory(session, task) -> tuple[bool, str] def memory_active(task) -> bool # task.memory_arm IS TRUE AND get_bool("runner_project_memory"), read per request def ingest_candidate(session, *, project_id, candidate, feed, task_id, session_id, persona, model, injectable_on_accept: bool) -> IngestResult # never commits def record_events(session, *, task_id, events) -> int # (unit, kind, channel, session_id, provisional); ON CONFLICT DO NOTHING; updates last_injected_at / last_used_at (read, followed, non-provisional cited) def memory_counts(session, task_ids) -> dict[int, dict[str, int]] def compile_bundle(session, project_id) -> dict def bump_bundle_revision(session, project_id) -> int def unit_by_token(session, project_id, token: str) -> ProjectMemory | None ``` - **Switch** (`switch.py`): ``` effective_memory(session, task): if not get_bool("runner_project_memory"): return False, "killed" if task.project_memory is not None: return task.project_memory, "task" if epic and epic.project_memory is not None: return epic.project_memory, "epic" pct = get_int("project_memory_holdout_percent") # default 0 if pct and stable_hash(epic.id if epic else task.id) % 100 < pct: return False, "holdout" return get_bool("project_memory_default"), "default" ``` This child provides the function only. C3 calls it from both compose callers and stamps `tasks.memory_arm`/`memory_arm_source`. This child also copies `sessions.memory_arm` from `tasks.memory_arm` when a runner binds (where the `sessions` row is written). - **Settings knobs** in `operator_settings.py` CATALOG: - `project_memory_default`: operator, `tenant_visible`, bool, default **True**. This is the System Settings "Project memory" entry. - `project_memory_holdout_percent`: operator, int 0–50, default 0. - `runner_project_memory`: operator, bool, default True. This is the kill switch: False makes every task OFF. Add it to `RUNNER_FLAG_KEYS` and to `tools/runner_flags.py` `FLAG_KEYS`, with a test that `_usable` tolerates an extra key. - `memory_llm_credential_id`: operator, str, default "". It names an anthropic or deepseek credential. Blank means the LLM checks are inactive, with no env fallback. - Generalise the `support_llm` resolver in `llm_client.py` to take a knob key, without changing support's behaviour. - Embeddings reuse `support_kb_credential_id`. - **Identity:** a per-project `number`, giving the citable token `mem-<number>` (regex `\bmem-(\d{1,9})\b`), resolved within the citing task's project. Allocate it like this: `INSERT INTO project_memory_state(project_id) … ON CONFLICT DO NOTHING`, then `SELECT … FOR UPDATE`, then `next_number += 1`. That ordering leaves no first-ingest race. - **Ingest gate order** (synchronous, in `ingest_candidate`): 1. **Exclusion** `gates.exclusion_reason()` returns one of these reasons: - `session_summary` - `task_state`: PR numbers or URLs, "I implemented / this PR / merged", task ids as content. - `locator_only`: a `qa`/`codemap`-from-recon answer, with paths, line refs and identifiers removed, must keep ≥6 words AND a term from `RELATION_TERMS` ("because", "must", "before", "after", "only", "never", "unless", "breaks", "requires", "instead", …). - `verbatim_file_text`: the body is ≥80% a substring of `source_text`. - `canon_duplicate`: token-set Jaccard ≥0.6 against any directive text in `project_memory_canon` for the project. A unit MAY name a canon path. - `gotcha_without_repro`: no `Repro:` line or fenced command. - `decision_kind`. 2. **Secret** `gates.secret_hit()`: the `app/secret_redact` pattern pass over title + body + paths + note. Any substitution discards the candidate. Load that module; never fork it. 3. **Exact dedupe:** `content_hash = sha256(normalize(kind|title|body))`. Then `INSERT … ON CONFLICT (project_id, kind, content_hash) WHERE status IN LIVE_STATUSES DO UPDATE`, which only adds a `memory_sources` row. Rejections persist only `memory_rejections(...) += 1`. The ingest writes status `proposed` and stores `injectable_on_accept` for C6a's promotion. Scope is always `'project'`. **Logging rule for the whole package:** log ids, feed and reason only — never title, body or note. - **Bundle** `compile_bundle`: - It returns eligible units only: `status='accepted' AND injectable AND trust ≥ 0.35 AND (no paths OR verified_at ≥ now() − 24 h)`. - AD-8: it carries only units visible to EVERY project member, i.e. every source task is neither hidden, unsaved nor otherwise restricted, or the unit has no task source. - CP-ranked, ≤500 units, ≤256 KiB, codemap ≤20%. - Titles are sanitised: control and ANSI characters stripped, leading `[`, `-`, `#` and `•` stripped, single line. - Shape: `{"v":1,"project_id":12,"revision":4711,"integration_branch":"develop","codemap_sha":null,"canon_sha":null,"units":[{"id":"mem-42","kind":"gotcha","title":"…","paths":[…],"keywords":[…],"commands":[…],"personas":[],"trust":0.71,"slice":false,"anchor":{"sha":"<sha>|null","paths":[…]}}]}`. ## Acceptance criteria (BA, verbatim — this child's legs) - AC-3 Default + override resolution. The resolution matrix (task On/Off/Default × epic On/Off/Default/none × system On/Off) yields the R-SW-3 value and is recorded on the task. *Verify:* a parametrized API test over all 24 combos. (This child: the test over `effective_memory` plus the kill-switch → `killed` and holdout cases. The dev-CP leg is C9.) - AC-14 Exclusion. Candidates that are session summaries, task state, a file's verbatim text, canon duplicates, or gotchas without a reproduction are rejected and their text not stored. *Verify:* table-driven test, DB inspected after. - AC-16 Dedupe & concurrency. 50 concurrent proposes of the same content from different tasks/nodes → exactly one accepted unit, all provenance preserved, one vote per task. *Verify:* concurrency test. (This child: 50 concurrent `ingest_candidate` calls in separate DB sessions → one live row, 50 `memory_sources`; concurrent `memory_votes` inserts for the same (unit, task) → one row.) - AC-19b No decision kind. `kind=decision` is refused by schema, tools and mining. *Verify:* test. (This child: DB CHECK, `Candidate` validation and `ingest_candidate` refuse it. The tools leg is C4; the mining leg is C6a.) - Quality bar: the secret gate drops a candidate holding a fixture token, key or connection string. A grep of every table this child writes, and of captured logs, finds no fragment of it. Out of scope: Epic (verbatim): "moving OMC state to shared storage (ADR-0099); changing canon docs or the router's existing deterministic routes; org-wide (cross-project) memory beyond a scope field reserved for it; adopting an external memory SaaS." BA fences (binding): - N-1: no OMC state moves to shared storage. - N-2: no edits to canon docs' content, `.claude/context-routes.json` routes, or the router's deterministic matching, caps and output. - N-3: only the `project` scope is written or served. - N-4: no external memory SaaS. - N-5: OFF-task prompts and hooks are unchanged. - N-6: the `task_settle` contract and the recon_map rendering are unchanged. - N-7: no caller-supplied task id on any memory verb. - N-8: no unit text in any repo or tracked worktree file. - N-9: no `decision` kind anywhere. - N-10: distillation never blocks settle, dispatch or hooks. Sibling fences: do NOT build compose or prompt rendering (C3), the UI (C2), the agent memory routes or verbs (C4), the router (C5), the crons, near-dupe, canon-LLM or model secret check (C6a/C6c), the node helper (C6b), or the console (C8). Do not edit `operator_settings` UI pages beyond what a new catalog knob auto-renders. Deferred, do not build: native CLI projections, console confirm/flag/edit, org serving. Done means: - Migration applies and downgrades cleanly on a fresh `./dc` stack. All tables, CHECKs and the partial unique index exist — `./dc psql` `\d project_memories` shows them. - AC-3: `pytest apps/api/tests -k memory_switch` passes over the 24 combos plus the killed/holdout cases. - AC-14: a table-driven `pytest -k memory_exclusion` rejects each class. The DB holds only `memory_rejections` counts afterwards. - AC-16: `pytest -k memory_ingest_concurrency` gives 50 concurrent ingests → one live unit + 50 sources, and duplicate votes → one row. - AC-19b: `pytest -k memory_no_decision_kind` passes for the DB CHECK and the ingest refusal. - The secret fixture never persists: the test greps every table and the caplog. - `runner_project_memory` is served by `GET /api/agent/settings/runner-flags`. A `_usable` extra-key test passes. - `project_memory_default` defaults True and is visible to tenants. - `./dc check full` green. Required checks: gate

Project memory C1 (store, switch, gates, ingest, events, bundle; revision now 0391) merged via #4709; full gate green. Dev deploy failed on a 0390 collision with 5415; swap fix landed as #4711 and Jaah Deploy cd8714f succeeded.