Operant Studio
OPER-7A

Pipeline self-heal Phase A: detection + metrics + alerts (DOC-approved)

Body

DOC Council-of-9 unanimous approval. Detection-only, no auto-heal.

Deliverables:
1. Extend pipeline-health cron (*/5) with 3 invariants:
   - #1 IN_PROGRESS tasks with no matching workflow_dispatch run within 60s (SEV-3)
   - #2 IN_PROGRESS tasks with no branch in target repo (SEV-3)
   - #3 Queue liveness: PLANNING_VALIDATED>0 AND inFlight<cap AND no new dispatches in 15min (SEV-2)
2. PipelineHealthMetric table + PipelineHealthMonthlyRollup (30d TTL + monthly aggregate)
3. DispatchAudit table per workflow_dispatch attempt
4. Emit via unified audit-event path + structured JSON logs
5. Deadman switch on pipeline-health
6. GitHub-API-health circuit-breaker on checkBranchExists (distinguish missing vs check-failed)
7. Slack alerts to team-scoped channels via TeamAlertRouting map:
   - Portico team -> #alerts-portico-pipeline
   - Operant team -> #alerts-operant-pipeline
   - Coalesced, thread-replied on recurrence, dev-mode flag
8. Auto-file post-incident ticket on SEV-2
9. Runbook docs/operations/pipeline-runbook.md
10. SLO doc docs/operations/pipeline-slos.md (99% dispatched within 5min; 4x burn alert)

Follow-up tickets: OPER-7B (auto-heal, tomorrow), OPER-8 (dashboard UI), OPER-9 (PAT rotation alerts).

Attachments

Loading attachments…

Comments

Loading comments…