KNUCK-A036
KNUCK-A036: evidence-artifact gate for empirical claims in PR bodies + commits + migrations
Suggested
Justin Cooke
Body
## Context
On [REDACTED-DOB], migrate-prod for KNUCK-A007b1a (PR #278) failed on its own pre-flight guard because the agent had baked a false empirical claim into the migration file header, the PR body, and PDT memo D-23:
> TaskDependencyEdge: 4/4 populated with canonical actor prefix (0 NULL), verified via direct Neon query 2026-09-07T13:08Z.
The claim was false. All 4 edge rows held a bare cuid without actor prefix. The agent asserted a regex match it never ran. Prod schema was NOT altered because the pre-flight guard added by the same PR fired -- the substrate caught what the narrative failed to catch. See `audit/evidence/knuck-a007b1a/` (PR #280) for the correction narrative and pre-flight-guard-catch.json.
Root cause: agent conflated "I said I ran the query" with "I ran the query." No mechanical control required the claim to be backed by an independent artifact.
Named the failure pattern "Self-Certified Empirical Claim" in memo D-23 postmortem. Sibling of the Shared-Substrate Detector Fallacy. Both root in the same problem: certification surfaces are not independent of their subjects.
## Acceptance Criteria
- New CI check `evidence-artifact-gate` runs on every PR
- Greps PR body, commit messages in the diff, and any new/modified migration file for empirical-claim patterns:
- `\d+/\d+ populated`
- `\d+/\d+ non-null` and variants (case-insensitive)
- `verified (via )?direct .* query`
- `verified \d{4}-\d{2}-\d{2}`
- `queried (prod|production|neon|the database)` with a numeric result nearby
- `all rows match` / `all N rows` and variants
- For each matched claim, the diff MUST add a file under `audit/evidence/<ticket-or-branch-slug>/` that contains the referenced numbers or query output
- If the diff modifies a migration file's header but doesn't add matching evidence, fail (as with A007b1a; would have caught the false 4/4 claim before merge)
- Escape hatch: PR label `evidence-narrative-only` skips the check with a required PR-body justification section
- False-positive rate target: < 5% on 30-PR historical sample
- Ships as required check for both `Operant-Health/operant-studio` and `Operant-Health/portico`
## Non-goals
- Not enforcing that evidence artifacts are correct (mechanical claim-to-artifact linkage only)
- Not deleting existing false claims retroactively (preserved as forensic history; correction narratives in `audit/evidence/`)
## Effort
Single Ledger ticket. Parallelizable with A037-A041. Blockers: none.
## Filed as follow-on to
PR #278 (KNUCK-A007b1a). Direct control that would have caught the false-verification failure. Ships first among A036-A041.
---
<!-- admission-rejected -->
**Admission rejected (missing_feature_id):** ticket body is missing `feature_id: FEAT-...` frontmatter — declare which feature this ticket implements
Attachments
Loading attachments…
Comments
Loading comments…