Build integration catalog core API
## Original requirement (verbatim, epic 13348)
Deep research all third party integrations in Lovable (Connect) and implement all of them.
Ensure the highest level of code reuse, abstract out any generic code across all integrations. Users need an easy way to connect. Documentation must be as detailed as in Lovable or better.
If something needs to be done on the partner website (e.g. register an app) and it can only be done by human create a follow up task per third party with exact step by step of what user needs to do). Mark it as user only task to set up but the runner will validate it was done correctly.
You can refactor anything to make working with creds more robust.
## Settled inputs
Full PRD and architecture: the epic's `prd` and `architecture` artifacts (Task 13348 BA, Task 13349 Architect). Section numbers below (§3.2 item 6, AC2.3, …) refer to them. They are settled: realize them, do not reopen them. Operator decisions O1–O6 and D1–D8 bind.
## This child's scope
C1, architecture §3.1 and §3.2 items 1–9.
- §3.1: `apps/api/app/integrations/catalog.py` loader + the pinned TOML schema (`apps/api/app/integrations/vendors/<slug>.toml`, `logos/<slug>.svg`), closed category enum with labels single-homed in the loader, validation at import (malformed vendor or missing `vendor_docs_url` fails the gate), derivation into CREDENTIAL_TYPES / PROVIDERS / declared tester, conflict guard against hand-written slugs (it explicitly allows a catalog spec layered over an existing or unlisted provider when the fields match, e.g. a later `slack` spec over the provider C7 registers), `EXPECTED.txt` oracle list + test that loaded slugs equal it.
- §3.2: (1) HttpTestSpec (method, body, header/basic/query/SigV4 auth, `{field}` tenant base URL dialled through `http_bounded.vetted_addresses`), `tester = "module:function"` hook, `test.kind = "format"` fallback; (2) `credentials.last_test_message`; (3) `needs_reconnect_at` / `needs_reconnect_reason`, set by `oauth2_tokens._refreshed` on invalid_grant, cleared on Connect, `credential-token-refresh` cron in `cron_system_activities.py` (15-minute look-ahead), runner `credential.get` on a dead row returns 410 Gone "needs reconnect"; (4) generic `oauth_app` type + `oauth_app.<slug>` knobs derived in one loop in `operator_settings/catalog_integrations.py` (existing *_app types untouched); (5) `OAuthProvider.client_source` platform|row|platform_or_row, BYO row client; `app_not_configured` state; (6) `POST /api/integrations/{slug}/connect/start` with the row created on the EXISTING `GET /api/credentials/connect/callback`, FlowCookie gains `integration` + `pending` (exactly one of credential_id/pending; BYO secret sealed like `verifier`), callback runs the Test and redirects to `/credentials/<id>?connected=1`; also the post-exchange hook contract `OAuthProvider.on_grant` (`"module:function"`, signature `(session, provider, grant, pending) -> Credential`) used later by C6/C7/C8; (7) `GET /api/integrations` (`IntegrationOut`, worst_state over the caller's visible rows) + `GET /api/integrations/logos/{slug}.svg` (immutable cache, image/svg+xml, SVGs sanitised: no script, no external href); (8) no plan gate; `edition_feature` only where the base type has one; confirm with the `tools/community_edition` check; (9) `credentials.system_owned` excluded from `task_secrets._granted_credentials` and the roster (404).
- Fixtures: a fake key vendor and a fake OAuth vendor in tests only. `EXPECTED.txt` starts empty; real spec files (including those adopting existing types such as stripe, jira, github) are written by the vendor waves with full docs prose, so no thin spec ever meets the docs check (planner decision).
- ALL additive DDL of this epic lands in C1's ONE alembic revision (so later concurrent children carry no migration): the credentials columns above, `slack_installations.credential_id` and `gmail_connections.credential_id` (nullable FKs), table `project_mcp_links(project_id, credential_id, created_by, created_at)` PK on both ids, table `custom_connectors` (columns per architecture §4, standard entity shape), and the data back-fill of `credentials.needs_reconnect_at` from forge `config.grant_dead_at`. Models may be added with the tables; their routes/logic belong to the later children.
- Connect start must also serve an OAuthProvider that has no gallery card (an unlisted provider registered in PROVIDERS with an `on_grant` hook, e.g. bitbucket, azure_devops, gmail), so C6/C7/C8 can move onto the engine before any vendor wave lands; unlisted providers are excluded from `GET /api/integrations`, `EXPECTED.txt` and the docs check.
Security surface (credentials/OAuth/authz): run the autonomous security review lane; SSRF unit test against 169.254.169.254 and RFC 1918 addresses for tenant URLs.
Credentials & reach:
- AC1.4 dev (revoked connection shows Needs reconnect) — deferred: unit test only; revoking a live roster grant is refused as destructive
- Unit tests, `./dc check quick`, `tools/check_context.py` — verified (no credential)
Out of scope (PRD §6, verbatim, plus what NOT to touch):
- No app-user connectors (per-end-user OAuth, connector gateway for published apps).
- No "Managed by Jaah" vendor keys or credit billing; no change to `apps/api/app/billing/`.
- No per-vendor API client libraries or MCP tool wrappers built by Jaah (except where the vendor's own hosted MCP server is the connector).
- No Lovable action-approval UI (allow once/always).
- No change to runner auth to model providers (`account_auth.py`, `Account` table) or to login/sign-in OAuth (`routes/login_oauth.py`).
- No change to NodeVault crypto semantics (`credential_crypto.py` encryption tiers) beyond what a refactor needs to keep behaviour identical.
- Do not break existing rows: every existing credential type and every existing row keeps working; migrations additive (two-release overlap rule).
- Supabase/project-backend and git-sync are not new work; Figma plugin/.fig upload out.
- Never write a credential value into a task, doc, log or PR.
- Do NOT file Tasks (no licence): the human-only OAuth registration tasks are filed outside this child.
Done means:
- AC1.1 Adding a key-based vendor requires only declarative data (one catalog entry: slug, label, category, logo, fields, tester spec, doc link) — no new route, no new React component. — a unit-test fixture vendor defined with data only is served by `GET /api/integrations`, Connect and the Test route (`pytest` on the new catalog tests passes).
- AC1.2 Adding an OAuth vendor requires only an `OAuthProvider`-style entry (endpoints, scopes, refresh/wire quirks, Jaah-app knob name) — the shared Connect route serves it. — unit test drives authorize→callback→token-store→refresh for a fake provider through `/api/integrations/{slug}/connect/start` and the shared callback; passes; abandoned flow leaves no row.
- AC1.4 OAuth tokens refresh automatically before expiry and a revoked/expired grant marks the connection Needs reconnect (never a silent failure at runner time). — unit test: expired token + failing refresh sets `needs_reconnect_at`; the cron refreshes a row expiring in <15 min; runner `credential.get` on the dead row returns 410. Dev revoke check deferred (see Credentials & reach).
- AC2.5 Every connection has a scope chooser (org / project / personal) and is fetchable by a runner via `credential_get` with its documented field names. — runner-side test calls `credential_get` for a project-scoped catalog connection and receives the fields.
- AC2.6 Validation errors are specific, never a raw stack or a generic "failed". — tester unit test with a 401 shows "<Vendor> rejected the key (401)" and `last_test_message` stores it.
- AC3.2 A vendor with no safe read-only endpoint is listed with the chosen alternative (format check) — no vendor silently lacks a test. — registry test asserting every catalog entry resolves to a credential type and a Test (HTTP or format); format result reads "Format checked — <Vendor> was not called".
- AC7.2 (API) With the knob unset and no BYO client, the vendor is returned as `app_not_configured` (oauth.app_configured=false, byo_allowed=true) — unit test.
- System-owned rows never reach a runner — `credential_get` on a `system_owned` row returns 404; unit test.
- SSRF: tenant base URL pointing at 169.254.169.254 or 10.0.0.1 is refused by the Test — unit test.
- Every existing credential row and type still works — existing credential/OAuth test suites pass unchanged.
- The single revision upgrades and downgrades cleanly on a DB with existing forge rows; the back-fill sets `needs_reconnect_at` for a row with `config.grant_dead_at` — migration unit test.
- AC2.5 dev half (a dev task run reads a catalog connection) — (deferred — verified in the W1 child with a roster connection once catalog specs exist).
- AC1.5 `./dc check quick` green and `tools/check_context.py` green on this PR — the gate output cited in the PR body.
Outcome
Built the integration catalog core API (TOML vendor specs, tester, one-click Connect, needs-reconnect + refresh cron, one additive revision); quick gate green, 3 review lanes pass. PR #7877 queued to the merge lane. Memory: used mem-32955.
Artifacts
Activity
- Draft
- Ready
- Runner started
- Runner ended (done)