OPER-183
fix(ci): drift-check job conflates pending migrations with real drift
Band
SMALL
Suggested
Justin Cooke
Body
feature_id: FEAT-studio-ci-cd ## Context `.github/workflows/migrate-prod.yml` has two jobs that run `prisma migrate status` against prod: 1. **`migrate` job, `Prisma migrate status (pre-flight, drift check)` step** — has correct logic added in OPER-93 ([REDACTED-DOB]). Greps the combined output for `not found locally in prisma/migrations` (real drift) and treats rc=1 with `have not yet been applied` (pending migrations) as pass, since pending migrations are the whole point of the next `migrate deploy` step. 2. **`drift-check` job, scheduled every 6 hours** (`cron: 17 */6 * * *`) — added in OPER-83 ([REDACTED-DOB]) as a safety net for the silent-skip class of bug. But this job runs `prisma migrate status` and exits 1 on any non-zero return code from the status command. It has no equivalent of the pre-flight step's grep logic. **Bug:** `prisma migrate status` returns rc=1 for both real drift AND pending migrations. Since [REDACTED-DOB], the `drift-check` job has been failing every 6 hours with `conclusion: failure`, alerting `#dev-alerts` with a "drift" signal when in fact the cause was pending migrations (bot-merged migration PRs that silently skipped `migrate-prod`, tracked separately in OPER-182). **Signal degradation:** Every failed `drift-check` fires a Slack alert. Justin's team has received these alerts every 6 hours for 2+ days. The signal is now noise — the team can't distinguish "real drift, someone changed prod DB out of band" from "pending migrations, migrate-prod didn't fire." When OPER-182 lands and fixes the silent-skip, this drift-check job will keep firing false positives for any migration PR merged in the window between merge and next scheduled `drift-check` run (up to 6 hours). **Observed:** Every scheduled run since [REDACTED-DOB] 21:26 UTC has `conclusion: failure`. Latest confirmed at [REDACTED-DOB] 16:21 UTC. Log output shows exactly the pending-migrations path, no `not found locally` line. ## Acceptance Criteria - [ ] `drift-check` job uses the same grep logic as the `migrate` job's pre-flight step: pass on rc=0, pass on rc=1 with `have not yet been applied` (no drift marker), fail with drift signal only on rc=1 with `not found locally in prisma/migrations`, fail-safe on any other non-zero. - [ ] The Slack notification body distinguishes "DRIFT DETECTED" (real drift, someone changed prod DB out of band) from "PENDING MIGRATIONS" (migrations queued waiting for migrate-prod to fire). Pending-migrations state should either be silent or use a distinct low-severity signal. - [ ] Existing pre-flight-step grep logic in the `migrate` job is unchanged. - [ ] A comment header on the `drift-check` job cross-references OPER-83 (why the job exists) and this ticket (why the grep matters). - [ ] Regression check: a manual `workflow_dispatch` on a state with only pending migrations (no real drift) does NOT fail the `drift-check` job. ## Risk classification Sandbox (Risk 3 / Reach 3). The `drift-check` job Slack-alerts only — it does not gate deploys, does not auto-apply, does not modify DB state. A bad fix at worst produces a wrong Slack message; it cannot break prod. writeSets are a single `.yml` file and a Slack message template. ## Depends on None. Independent of OPER-182 — this ticket fixes the alert signal quality, OPER-182 fixes the underlying silent-skip. ## Blocks Nothing. ## Related tickets - OPER-182: fix(ci): migrate-prod silently skips bot-merged migration PRs. This ticket is the companion — OPER-182 fixes the root cause, this ticket fixes the noisy alert that has been firing while the root cause went unfixed. ## Out of scope - Auto-applying pending migrations from the `drift-check` job (out — that's a policy change, needs PDT input) - Rewriting the Slack notification format broadly (out — this ticket only differentiates the two states in the body) - Changing the 6-hour cadence (out — separate concern) - Alerting on Vercel/DB schema mismatch (out — different signal, would need Vercel deploy hook) --- <!-- admission-rejected --> **Admission rejected (unknown_feature_id):** feature_id=FEAT-studio-ci-cd is not declared in the feature-map.md of any admission-enabled product owned by this team. Valid feature_ids for product studio (products/studio/feature-map.md): - FEAT-studio-agent-workers - FEAT-studio-audit - FEAT-studio-audit-log - FEAT-studio-branch-protection-rulesets - FEAT-studio-coverage-policy - FEAT-studio-dependency-map-policy - FEAT-studio-dispatcher - FEAT-studio-env-protection - FEAT-studio-fabricated-evidence-tripwire - FEAT-studio-guard-inventory - FEAT-studio-incident-runbook - FEAT-studio-mutation-testing - FEAT-studio-phi-guard - FEAT-studio-pipeline-observability - FEAT-studio-planning-validation - FEAT-studio-platform - FEAT-studio-product-manifest - FEAT-studio-qa - FEAT-studio-qa-projects - FEAT-studio-review-rounds ... and 2 more; see products/studio/feature-map.md
Attachments
Loading attachments…
Comments
Loading comments…