OPER-14A
worker-reaper failing to release IN_PROGRESS tasks whose workflow completed
Body
## Problem worker-reaper runs every 2 minutes but is not releasing Portico tasks whose agent-dispatch workflow already completed (success OR failure). 17 tasks sat IN_PROGRESS for 12+ hours consuming all maxInFlight slots. ## Evidence All 17 stuck tasks had matching workflow runs with `status=completed`: - 15 with `conclusion=failure` (never released → should reap to PLANNING_VALIDATED) - 2 with `conclusion=success` (POR-11 → PR #389 merged, POR-21 → PR #401 merged; should reap to DONE) Dispatcher slot occupancy: 17/20 for ~12h despite queue of 205 PLANNING_VALIDATED and cron ticking every minute (`lastTickAt` current). ## Suspected causes 1. worker-reaper query does not match the workflow run to the Task (branch-name mismatch, task-id lookup, or workflow-name filter) 2. worker-reaper only reaps on timeout, not on workflow-completion; timeout threshold >12h 3. Cron may not be firing (verify with a diagnostic log line) ## Acceptance - worker-reaper detects `completed/failure` workflow runs and reverts task to PLANNING_VALIDATED with gitBranchName=NULL within one tick (~2 min) - worker-reaper detects `completed/success` runs and promotes task to DONE (or AWAITING_HUMAN if PR-review gate exists) within one tick - No task remains IN_PROGRESS more than 30 min after its workflow finishes - Emits DispatchAudit rows with outcome=`reaped_failure` / `reaped_success` ## Related - OPER-11A (shipped) — branch-ensure unblocks fresh claims but reaper failure still stalls throughput - Feeds into OPER-8A (dashboard) — reap counts should surface there
Attachments
Loading attachments…
Comments
Loading comments…