Operant Studio
OPER-9B

Dual-PAT failover in dispatcher + auto-requeue on PAT recovery

Band
MEDIUM
Suggested
Justin Cooke

Body

feature_id: FEAT-studio-agent-dispatch

## Context

Every GitHub call in the agent pipeline (dispatch, workflow-status polling, branch checks, PR reads) Bearer-auths with one shared `GH_DISPATCH_PAT`. When it expires or its repo-access list rotates, the whole fleet silently stops dispatching — as observed on [REDACTED-DOB], when 21 OPER-QA / OPER-10x tickets bounced to `AWAITING_HUMAN` with `dispatcher:branch-create-failed` HTTP 404 within seconds of PLANNING_VALIDATED.

OPER-9A's pat-expiry-check cron already alerts at T-60/-30/-7/-1/expired milestones, but recovery is fully manual: a human must (a) notice the alert, (b) mint a new PAT with the right scopes on both Operant-Health/operant-studio and Operant-Health/portico, (c) update Vercel env, (d) redeploy, and (e) manually PATCH every bounced ticket back to PLANNING_VALIDATED. During today's incident the recovery window was ~4 hours.

This ticket adds two mechanisms:

1. **Dual-PAT failover in the dispatcher.** Read `GH_DISPATCH_PAT_A` first, fall back to `GH_DISPATCH_PAT_B` on any 401/403/404 from GitHub. Both PATs must independently authorize against Operant-Health/{operant-studio, portico}. Rotation ritual becomes: rotate B, restart dispatcher, promote B->A, mint new B — the fleet never drops below one working PAT.

2. **Automated re-queue on PAT recovery.** When the dispatcher observes a PAT-recovery signal (a 200 from GitHub on a call that previously 4xx'd for the same PAT), it bulk-flips every task with `Comment.authorLabel = dispatcher:branch-create-failed` created in the last 24h AND `Task.status = AWAITING_HUMAN` back to PLANNING_VALIDATED. Dedupe key: `(taskId, unbouncedAt)` in-memory over 24h; hard ceiling of 3 auto-requeues per (task, day). Never fires on a timer — only on an observed recovery.

Backwards compatible: if only `GH_DISPATCH_PAT` (no _A/_B) is set, the dispatcher reads that as PAT A and skips the failover path. Existing behavior preserved.

## Acceptance Criteria

- `packages/dispatcher/src/github-dispatch.ts`: `createBranch`, `dispatchWorkflow`, `getBranch`, and every other GH_API caller wrap their fetch in a `withPatFailover(fn)` helper that tries A, catches 401/403/404, retries with B, and records which PAT succeeded on a `DispatcherState.lastSuccessfulPat` field.
- `packages/dispatcher/src/pat-recovery.ts` (new file): exports `detectPatRecovery(db)` — returns `{recovered: true, patLabel: 'A'|'B'}` when a PAT transitioned from failing to succeeding within the current tick. Exports `requeueBouncedTasks(db, teamId, since)` — bulk-flips AWAITING_HUMAN tasks with a recent branch-create-failed comment back to PLANNING_VALIDATED, capped at 3 auto-requeues per (task, day).
- `packages/dispatcher/src/backend.ts`: after each tick, call `detectPatRecovery`, and on `{recovered: true}` call `requeueBouncedTasks` for every team the dispatcher just processed.
- `packages/dispatcher/src/github-pat-expiry.ts`: extend `fetchPatExpiry()` to `fetchPatExpiries()` returning both PATs' status. `noExpiration` and `degraded` semantics preserved per PAT.
- `apps/web/src/app/api/cron/pat-expiry-check/route.ts`: probe both PATs; alert summary names which PAT (A or B) is expiring so the rotation ritual has clear steps.
- Existing behavior preserved when only `GH_DISPATCH_PAT` is set (no _A/_B): read as PAT A, no failover attempted, no code path changes for single-PAT deployments.
- Unit tests: dual-PAT rotation (A fails -> B succeeds -> A recovers -> re-queue fires exactly once per bounced task); ceiling enforcement (4th auto-requeue in 24h is skipped); backwards-compat for GH_DISPATCH_PAT only.
- Integration test: seed 3 AWAITING_HUMAN tasks with `dispatcher:branch-create-failed` comments in the last 24h, flip a mock PAT from 404 to 200, tick the dispatcher, assert all 3 flip to PLANNING_VALIDATED and exactly one `dispatcher:auto-requeued` comment lands on each.
- No PHI in comments, alerts, or logs: log only `{patLabel, httpStatus, taskCount}` — never the token, never the ref/branch name.
- `pnpm check:feature-map` passes locally.
- `.claude/rules/multi-tenant-rls.md` and `.cursor/rules/operant-boilerplate.md` unchanged (dispatcher work, not schema).

## Rollout

Land the code change with both PATs unset by default (single-PAT mode is the fallback). Deploy. Then set `GH_DISPATCH_PAT_A` = current working PAT, `GH_DISPATCH_PAT_B` = new fine-grained PAT with matching scope. Restart dispatcher. Retire `GH_DISPATCH_PAT` on the next rotation cycle.


---
<!-- admission-rejected -->
**Admission rejected (unknown_feature_id):** feature_id=FEAT-studio-agent-dispatch is not declared in the feature-map.md of any admission-enabled product owned by this team

Attachments

Loading attachments…

Comments

Loading comments…