Sensitive-content publish check with ratchet

Task 10481 · Queued · Coder · created · updated

Original work item (epic Task 10454, verbatim): - Remove Draft state from anywhere of the jaah-ce, agent-sdk, agent-connectors. It needs to be removed at the source and then propagated using standard mechanism - remove any references on the source of documents (especially do not mention any brands) - agent-sdk. must borrow and share the process from jaah-ce, as well as terms. We want them mirrors from the main repo. Only main branch for now - do we want the same with agent-connector? make sure it is up to date with agent-sdk - Look for any sensitive or internal information or any brands - scrub all old versions from all three repos Full settled inputs: the epic's `prd` and `architecture` artifacts (Task 10454; read with `task_read`/`artifact_read`). Decisions quoted below are verbatim from them. Child B — Sensitive check + ratchet (architecture §1.4). Builds on Child A's `tools/mirror/check.py` `CHECKS` seam. Requirement (PRD verbatim): R7 "The publish check of every mirror (jaah-ce's `check_tree.py`, and the same check for the two others) fails the publish on any §5 category, and every current hit is fixed at the source." PRD §5 Forbidden-content categories (verbatim): "1. Personal identifiers — any person's name, handle or email (e.g. `@ruzhukov`), except a generic org contact. 2. Internal infra — internal hostnames (dev control plane, tailnet `*.ts.net`, headscale), cloud account IDs, private repository names/URLs (`jaah-ai/jaah`, `jaah-ce-staging`), internal-only URLs. 3. Internal tracking refs — `Task NNNN`, `epic NNNN`, `ADR-NNNN`, work-unit labels (`WU-…`), and paths to docs that do not ship. 4. Secrets — keys, tokens, connection strings (gitleaks stays). 5. Brands (per S1) — non-required third-party names, n8n." Build `tools/mirror/sensitive.py` registered in A's `tools/mirror/check.py` `CHECKS` (so the CE `check_tree.py` and the SDK/connectors checks all run it). Each finding names file, line and category, never the matched value for categories 1, 2 and 4. - personal: the `instance_data` denylist (`.context-arch.json` → `portfolio/projects/jaah.md` `instance-terms` block, harvested emails) plus a new fenced block `mirror-people` in `portfolio/projects/jaah.md` (names, handles). - infra: regexes for `*.ts.net`, `dev.<jaah domain>`, any `<sub>.<jaah domain>` other than public `app`/`docs`/`www`/`api`, `jaah-ai/jaah` not followed by `-`, `jaah-ce-staging`, 12-digit AWS account ids, and the `gmind` word. (The `skynet` word joins this category in K6, not here.) - tracking: `\b(Task|task|epic|Epic) #?\d{3,}\b`, `ADR-\d{4}`, `WU-[A-Z]\d`, and every repo-path-shaped token in every shipped text file that names a path missing from the built tree; runtime-created paths on a scoped manifest allowlist. - secrets: gitleaks as today. - brands: denylist `tools/mirror/brands.toml` (seeded with `n8n`; Child D fills it from its sweep); scoped allowlist via manifest `[sensitive] allow`; `n8n` allowed only in `apps/api/app/credentials/registry.py` and its console and test fixtures (A7). - Ratchet: generate `tools/mirror/sensitive_baseline.toml` (per file and category, allowed hit count) over each built tree; a new full-gate stage in `tools/check.sh` builds every mirror tree (each manifest found, so the connectors tree is covered when C lands) and runs the check with the baseline — a file over its baseline fails. `publish.py` and the workflow check step run with NO baseline: any hit refuses the publish (belt on top of the workflow's check step). - Also add the one-off codemod `tools/mirror/strip_refs.py` (removes only the bare parenthetical forms `(Task N)`, `(ADR-NNNN)`, `, Task N)` and the trailing `— Task N`) for E1–E3 to use; H deletes it. Credentials & reach: - AC3/AC5/AC7 read oracles (branches, commits, tags, releases, file contents) | the runner's `jaah-forge` (task forge auth), GitHub API | **verified**: `jaah-forge api get repos/jaah-ai/agent-connectors/git/trees/develop` and `repos/jaah-ai/jaah/environments` answered - `./dc check full`, unit and gate oracles | nothing beyond the worktree | n/a Out of scope: - Do NOT scrub any hit at source (Children D, E1–E3, K6) — only generate the baseline. Do NOT add the `skynet` word to the infra category (K6). - Do NOT rename or change the product's `draft` task state or the SDK spec's "draft task" wording (D1). - Do NOT change any repo's visibility (D8); do NOT publish packages to PyPI/npm. - Do NOT relicense agent-sdk or agent-connectors (Apache-2.0 stays); do NOT change the CE license's terms beyond removing the draft banner and origin line. - Do NOT remove functionally required third-party names (S1) — provider CLIs, protocols, dependencies, forge names. - Do NOT edit monorepo files that ship to no mirror for §5 hygiene (S2) (the `skynet` rename alone is exempt, A2); do NOT rewrite the monorepo's own history. - Do NOT change jaah-ce's `ce-staging` environment or the edition seam / API cut. - Do NOT edit the mirrors directly except the one-time history reset and branch/tag/release/package deletion in R5; every content change lands in the monorepo and reaches the mirror by its publish workflow. - Do NOT change the SDK v2 API or the `jaah.ai` extension spec semantics. - Do NOT write to any mirror repo before the final publish child (A5). Done means: - AC7 (PRD verbatim): "each mirror's publish check run over its built tree passes; a planted line in a shipped file for each §5 category (one at a time, e.g. `# Task 1234`, `@ruzhukov`, `dev.jaah.ai`, `ADR-0001`, `n8n`) makes the check fail naming the file and category." — oracle: planted-line tests in `tools/tests/test_mirror.py` (or a sibling) for each of the five categories, each asserting failure names the file, line and category and does NOT echo the value for categories 1, 2, 4; the gate stage with the baseline passes on today's trees. - Strict mode: `check.py <tree> --manifest <m>` without `--baseline` fails on today's CE tree (proving publish would refuse) — cite the count per category. - `tools/mirror/publish.py` itself refuses (exits non-zero, pushes nothing) a tree with a single sensitive hit, with no baseline option available to it (test against a bare repo). - Ratchet proof: adding one new hit to a baselined file fails the gate stage (test). - `n8n` in `apps/api/app/credentials/registry.py` is allowed; `n8n` in any other shipped file fails (test). - `./dc check full` green in the merge lane.

Activity

  • Draft
  • Ready