Stall Detection & Auto-Heal Loop for IN_PROGRESS tasks

# OPER-19: Stall Detection & Auto-Heal Loop

## Update (2026-07-18)

Since this spec was drafted (2026-07-17 early AM), several of its components have shipped:

- **OPER-11A (branch prefix fix):** shipped (operant-studio #54)
- **OPER-11B (auto-merge filter):** shipped (portico #465)
- **OPER-20 (task timeline instrumentation):** shipped (operant-studio #55, migration applied). `dispatchedAt`, `prOpenedAt`, `firstReadyAt`, `checksGreenAt` now populate on new work.
- **OPER-21 (auto-rebase-dirty loop):** shipped (portico #519). Handles `dirty` stall class. 3-strike cap → `awaiting-human` label.
- **OPER-22 (sweeper 403 fix):** shipped (portico #527). Restored the existing `agent-pr-sweeper.yml` which already handles `draft-green` and `no-auto-merge-queued` stalls.

The remaining stall classes that are NOT yet handled autonomously:

- `no-pr` (agent-dispatch succeeded but no PR opened)
- `ci-fail` (needs triage + escalation to AWAITING_HUMAN)
- `ci-stuck` (CI in queue > 30 min)
- `already-merged` (PR merged but task still IN_PROGRESS — worker-reaper covers some of this)
- `phantom-branch` (task branch missing from GH but content on main)
- `stacked-blocked` (PR baseRef is not main)
- `duplicate` (two PRs for same identifier)

OPER-19's remaining scope is a **unified stall-detector cron** that catches all of the above in one place, plus emits size-normalized SLO alerts.

## Problem Statement

On 2026-07-16 evening, 27 Portico tasks sat IN_PROGRESS with no automated intervention. Manual audit uncovered 10 distinct failure modes, most silent — no alert, no metric, no auto-heal.

## Data — What Normal Looks Like

Portico last 30 days:

| Size | n | p50 mins | Recommended SLO (3× p50) |
|---|---|---|---|
| S | 16 | 105 | 180 min alert / 300 min AWAITING_HUMAN |
| M | 40 | 65 | 195 min alert / 300 min AWAITING_HUMAN |
| L | 11 | 30 | 150 min alert / 240 min AWAITING_HUMAN |
| unsized | 18 | 48 | fallback to M thresholds |

Tail (p90) currently ~4,500 min at every size = broken flow, not big work.

## The Loop

**Cron:** `/api/cron/stall-detector` — every 10 minutes.

For each IN_PROGRESS task on Portico + Operant teams:

1. Compute `time_in_progress = NOW() - startedAt`
2. Compute size-normalized SLO from table above
3. If `time_in_progress > SLO_alert`:
   a. Fetch corresponding PR via `gitBranchName`
   b. Classify stall into one of the buckets below

### Stall Classification Matrix (residual scope after OPER-11A/11B/20/21/22)

| Stall Type | Signal | Auto-Heal Action | Terminal |
|---|---|---|---|
| **no-pr** | Branch exists, no PR opened | Re-dispatch OR reconcile-to-DONE if content already on main | maybe |
| **ci-fail** | PR open + any required check FAILURE | Post triage comment, flip task → AWAITING_HUMAN | yes |
| **ci-stuck** | PR open + CI IN_PROGRESS/QUEUED > 30 min | Cancel + restart CI, alert if repeats | no |
| **already-merged** | PR merged but task still IN_PROGRESS | Flip task → DONE + audit | yes |
| **phantom-branch** | Task branch does not exist on GH but title matches merged commit on main | Flip task → DONE + audit ("content-match reconciliation") | yes |
| **stacked-blocked** | PR baseRef is not `main` and base PR is open | Log; heal will cascade when base merges | no |
| **duplicate** | Two PRs for same task identifier | Post comment listing both, flip task → AWAITING_HUMAN | yes |

(Already handled by shipped systems: `no-branch`, `draft-green`, `dirty`.)

4. Emit `DispatchAudit` row per action with outcome
5. Emit metric `stallDetector.autoHealed{stallType}` to existing metrics infra
6. Alert to `#alerts-portico-pipeline` on:
   - Any `ci-fail` (needs human)
   - Any `duplicate` (needs human)
   - Any stall type recurring 3× in 30 min (auto-heal not working)

### SLO Thresholds

- `SLO_alert = 3 × size_p50` → auto-heal attempt
- `SLO_hard = 5 × size_p50` → flip to AWAITING_HUMAN, notify

### Idempotency

- All flips are conditional `updateMany` guarded on current status (same pattern as workflow-reaper.ts)
- All actions are logged; the same task on the next tick sees "already healed <N min ago" and skips

## Deliverables

1. **Migration:** none — reuses `Task`, `DispatchAudit`, existing metrics. *(Must go in `packages/database/prisma/schema/migrations/` if any are added — POR-450 regression pattern.)*
2. **Code:** `apps/web/src/app/api/cron/stall-detector/route.ts` + `packages/dispatcher/src/stall-detector.ts`
3. **Config:** `vercel.json` cron entry every 10 min
4. **Tests:** unit tests for each stall classification, integration test spawning mock task + asserting heal
5. **Slack:** `TeamAlertRouting.stallDetector` channel routing
6. **Metrics:** counters per stall type, per outcome
7. **Runbook:** `docs/operations/stall-detector-runbook.md`

## Success Criteria

- 24h with zero task IN_PROGRESS > `SLO_hard` for its size
- p90 time-to-done drops from 4,500 min to under 300 min across all sizes
- Auto-heal action rate: >90% of stall detections resolve without human intervention

## Risks

- **Race with workflow-reaper:** both write to `Task.status`. Guard: stall-detector must run *after* reaper on the same tick, OR only touch stalls that reaper does not classify.
- **False-positive AWAITING_HUMAN flips:** mitigation is only escalate at `SLO_hard`, not `SLO_alert`.
- **Content-match phantom-branch reconciliation** is heuristic. First implementation should require manual confirmation via comment before auto-flip.

## DOC review required before build

This is a cross-cutting policy change (writes to `Task.status`) and needs DOC (DevOps Council) sign-off. Post to `#alerts-operant-pipeline` for review alongside OPER-23.