Sensitive-content publish check with ratchet
Original work item (epic Task 10454, verbatim):
- Remove Draft state from anywhere of the jaah-ce, agent-sdk, agent-connectors. It needs to be removed at the source and then propagated using standard mechanism
- remove any references on the source of documents (especially do not mention any brands)
- agent-sdk. must borrow and share the process from jaah-ce, as well as terms. We want them mirrors from the main repo. Only main branch for now
- do we want the same with agent-connector? make sure it is up to date with agent-sdk
- Look for any sensitive or internal information or any brands
- scrub all old versions from all three repos
Full settled inputs: the epic's `prd` and `architecture` artifacts (Task 10454; read with `task_read`/`artifact_read`). Decisions quoted below are verbatim from them.
Child B — Sensitive check + ratchet (architecture §1.4). Builds on Child A's `tools/mirror/check.py` `CHECKS` seam.
Requirement (PRD verbatim): R7 "The publish check of every mirror (jaah-ce's `check_tree.py`, and the same check for the two others) fails the publish on any §5 category, and every current hit is fixed at the source."
PRD §5 Forbidden-content categories (verbatim): "1. Personal identifiers — any person's name, handle or email (e.g. `@ruzhukov`), except a generic org contact. 2. Internal infra — internal hostnames (dev control plane, tailnet `*.ts.net`, headscale), cloud account IDs, private repository names/URLs (`jaah-ai/jaah`, `jaah-ce-staging`), internal-only URLs. 3. Internal tracking refs — `Task NNNN`, `epic NNNN`, `ADR-NNNN`, work-unit labels (`WU-…`), and paths to docs that do not ship. 4. Secrets — keys, tokens, connection strings (gitleaks stays). 5. Brands (per S1) — non-required third-party names, n8n."
Build `tools/mirror/sensitive.py` registered in A's `tools/mirror/check.py` `CHECKS` (so the CE `check_tree.py` and the SDK/connectors checks all run it). Each finding names file, line and category, never the matched value for categories 1, 2 and 4.
- personal: the `instance_data` denylist (`.context-arch.json` → `portfolio/projects/jaah.md` `instance-terms` block, harvested emails) plus a new fenced block `mirror-people` in `portfolio/projects/jaah.md` (names, handles).
- infra: regexes for `*.ts.net`, `dev.<jaah domain>`, any `<sub>.<jaah domain>` other than public `app`/`docs`/`www`/`api`, `jaah-ai/jaah` not followed by `-`, `jaah-ce-staging`, 12-digit AWS account ids, and the `gmind` word. (The `skynet` word joins this category in K6, not here.)
- tracking: `\b(Task|task|epic|Epic) #?\d{3,}\b`, `ADR-\d{4}`, `WU-[A-Z]\d`, and every repo-path-shaped token in every shipped text file that names a path missing from the built tree; runtime-created paths on a scoped manifest allowlist.
- secrets: gitleaks as today.
- brands: denylist `tools/mirror/brands.toml` (seeded with `n8n`; Child D fills it from its sweep); scoped allowlist via manifest `[sensitive] allow`; `n8n` allowed only in `apps/api/app/credentials/registry.py` and its console and test fixtures (A7).
- Ratchet: generate `tools/mirror/sensitive_baseline.toml` (per file and category, allowed hit count) over each built tree; a new full-gate stage in `tools/check.sh` builds every mirror tree (each manifest found, so the connectors tree is covered when C lands) and runs the check with the baseline — a file over its baseline fails. `publish.py` and the workflow check step run with NO baseline: any hit refuses the publish (belt on top of the workflow's check step).
- Also add the one-off codemod `tools/mirror/strip_refs.py` (removes only the bare parenthetical forms `(Task N)`, `(ADR-NNNN)`, `, Task N)` and the trailing `— Task N`) for E1–E3 to use; H deletes it.
Credentials & reach:
- AC3/AC5/AC7 read oracles (branches, commits, tags, releases, file contents) | the runner's `jaah-forge` (task forge auth), GitHub API | **verified**: `jaah-forge api get repos/jaah-ai/agent-connectors/git/trees/develop` and `repos/jaah-ai/jaah/environments` answered
- `./dc check full`, unit and gate oracles | nothing beyond the worktree | n/a
Out of scope:
- Do NOT scrub any hit at source (Children D, E1–E3, K6) — only generate the baseline. Do NOT add the `skynet` word to the infra category (K6).
- Do NOT rename or change the product's `draft` task state or the SDK spec's "draft task" wording (D1).
- Do NOT change any repo's visibility (D8); do NOT publish packages to PyPI/npm.
- Do NOT relicense agent-sdk or agent-connectors (Apache-2.0 stays); do NOT change the CE license's terms beyond removing the draft banner and origin line.
- Do NOT remove functionally required third-party names (S1) — provider CLIs, protocols, dependencies, forge names.
- Do NOT edit monorepo files that ship to no mirror for §5 hygiene (S2) (the `skynet` rename alone is exempt, A2); do NOT rewrite the monorepo's own history.
- Do NOT change jaah-ce's `ce-staging` environment or the edition seam / API cut.
- Do NOT edit the mirrors directly except the one-time history reset and branch/tag/release/package deletion in R5; every content change lands in the monorepo and reaches the mirror by its publish workflow.
- Do NOT change the SDK v2 API or the `jaah.ai` extension spec semantics.
- Do NOT write to any mirror repo before the final publish child (A5).
Done means:
- AC7 (PRD verbatim): "each mirror's publish check run over its built tree passes; a planted line in a shipped file for each §5 category (one at a time, e.g. `# Task 1234`, `@ruzhukov`, `dev.jaah.ai`, `ADR-0001`, `n8n`) makes the check fail naming the file and category." — oracle: planted-line tests in `tools/tests/test_mirror.py` (or a sibling) for each of the five categories, each asserting failure names the file, line and category and does NOT echo the value for categories 1, 2, 4; the gate stage with the baseline passes on today's trees.
- Strict mode: `check.py <tree> --manifest <m>` without `--baseline` fails on today's CE tree (proving publish would refuse) — cite the count per category.
- `tools/mirror/publish.py` itself refuses (exits non-zero, pushes nothing) a tree with a single sensitive hit, with no baseline option available to it (test against a bare repo).
- Ratchet proof: adding one new hit to a baselined file fails the gate stage (test).
- `n8n` in `apps/api/app/credentials/registry.py` is allowed; `n8n` in any other shipped file fails (test).
- `./dc check full` green in the merge lane.
Activity
- Draft
- Ready