Operant Studio
OPER-50

Post-incident: deadman_pipeline_health_stale triggered 2026-07-25T20:26:05.954Z

Body

## Context

Post-incident tombstone for `deadman_pipeline_health_stale` (SEV-2) fired 2026-07-25T20:26:05Z. `lastMetricAt=2026-07-24T17:55:29Z` — pipeline-health cron had not emitted a metric row for ~2h35m when the deadman switch tripped. Slack ping to `#dev-alerts-testing` failed to send (`ts=failed-[REDACTED-MRN]`).

## Acceptance Criteria

- [x] Confirm root cause is the same LEDGER_BEARER rotation window that produced OPE-1..OPE-4.
- [x] Confirm invariant has not fired again since [REDACTED-DOB] (6+ days silent).
- [x] Confirm systemic fix is in-flight or shipped.
- [x] Cancel with resolution note pointing at systemic fix.

## Original alert payload

```
{
  "invariant": "deadman_pipeline_health_stale",
  "severity": "SEV-2",
  "evidence": {
    "lastMetricAt": "2026-07-24T17:55:29.797Z",
    "staleThresholdMinutes": 15,
    "detectedBy": "worker-reaper deadman switch"
  },
  "slack": {
    "channelId": "#dev-alerts-testing",
    "ts": "failed-[REDACTED-MRN]"
  }
}
```

## Resolution — cancel ([REDACTED-DOB])

**Root cause:** LEDGER_BEARER token rotation window ([REDACTED-DOB] → [REDACTED-DOB]). The pipeline-health cron path called auto-planner over HTTP with the rotated bearer; auto-planner returned 401, cron exited without emitting a metric row, deadman switch fired ~2h35m after the last emit.

**Not recurred:** invariant has been silent for 6 days ([REDACTED-DOB] → [REDACTED-DOB]), consistent with the LEDGER_BEARER outage ending naturally. Sibling post-incident tickets from the same window (OPE-1..OPE-4) were all canceled by the same rationale.

**Systemic fix in-flight:** PR #167 (`refactor(auto-planner): eliminate LEDGER_BEARER round-trip via in-process runPlanValidation`, OPER-59) removes the HTTP hop entirely — the cron can no longer be broken by a token rotation. Nothing to do on this ticket.

**Guardrail already shipped:** PR #166 (OPER-58) adds the `auto_planner_silent` SEV-1 invariant so a future silent-stall trips sooner than the 15-min deadman window.

Canceled by Justin (via Sweetness) after review [REDACTED-DOB].

Attachments

Loading attachments…

Comments

Loading comments…