Operant Studio
OPER-49

Right-size Prisma connection pool for the reliability loop

Body

# OPER-49 · Right-size Prisma connection pool for the reliability loop

## Problem

Reliability-loop ticks are failing with 4/5 sensors returning `outcome: "error"` on every run since 2026-07-23. Error text across sensors:

```
Timed out fetching a new connection from the connection pool.
(Current connection pool timeout: 10, connection limit: 5)
```

and (from `$queryRawUnsafe` code paths where pool exhaustion surfaces as a different message):

```
Can't reach database server at ep-wild-resonance-atf8ez9o-pooler...
```

Neon itself is healthy — verified from the sandbox:
- `pg_stat_activity` shows 14/901 conns in use, 1 active, 6 idle
- TCP connect + `SELECT 1` returns in <30ms
- Neon status page: All Systems Operational

**Root cause:** The `@operant/database` `PrismaClient` singleton uses Prisma's default `connection_limit` (2× CPU count = **5** on Vercel Fluid), and the reliability loop fans out **5 sensors in parallel** via `Promise.allSettled` — every sensor grabs ≥1 connection and the pool is exhausted before the loop finishes.

## Fix

File: `packages/database/src/index.ts`

Change the `PrismaClient` construction to explicitly set a pooled `datasourceUrl` derived from `DATABASE_URL` (or an existing `DATABASE_URL_POOLED`), appending `?connection_limit=25&pool_timeout=20`.

**Before**
```ts
export const prisma =
  globalThis.__prisma ??
  new PrismaClient({
    log:
      process.env.NODE_ENV === "development"
        ? ["query", "error", "warn"]
        : ["error"],
  });
```

**After**
```ts
function buildPooledUrl(): string | undefined {
  const raw = process.env.DATABASE_URL_POOLED ?? process.env.DATABASE_URL;
  if (!raw) return undefined;
  try {
    const u = new URL(raw);
    // Prisma reads these from the query string when connecting via PgBouncer / Neon pooler.
    if (!u.searchParams.has("connection_limit")) u.searchParams.set("connection_limit", "25");
    if (!u.searchParams.has("pool_timeout")) u.searchParams.set("pool_timeout", "20");
    if (!u.searchParams.has("pgbouncer")) u.searchParams.set("pgbouncer", "true");
    return u.toString();
  } catch {
    return raw; // if URL parsing fails, fall back to raw — Prisma will error clearly.
  }
}

export const prisma =
  globalThis.__prisma ??
  new PrismaClient({
    datasourceUrl: buildPooledUrl(),
    log:
      process.env.NODE_ENV === "development"
        ? ["query", "error", "warn"]
        : ["error"],
  });
```

Rationale for 25 / 20s:
- Neon pooler supports thousands of client conns; 25 per Vercel instance is well below any limit.
- Reliability loop has 5 sensors, some using `$queryRawUnsafe` which takes a dedicated conn — 25 gives 5× headroom for concurrent ticks + normal request traffic.
- `pool_timeout=20` doubles the current 10s ceiling to survive cold-start bursts.
- `pgbouncer=true` disables prepared statement caching that PgBouncer transaction pooling doesn't support (Neon pooler is transaction-mode).

## Tests

File: `packages/database/src/__tests__/index.test.ts` (create if missing)

- Test `buildPooledUrl` with a URL that has no query params → returns URL with all three params set.
- Test with a URL that already has `connection_limit=10` → preserves the existing value.
- Test with an invalid URL → returns the raw string unchanged.
- Test with `DATABASE_URL_POOLED` set → prefers it over `DATABASE_URL`.
- Test with neither set → returns `undefined`.

## Verification

```bash
npx vitest run packages/database/src/__tests__/index.test.ts
```

Post-merge, verify on next reliability-loop tick:

```sql
SELECT "sensorName", "outcome", COUNT(*)
FROM "ReliabilityEvent"
WHERE "createdAt" > NOW() - INTERVAL '30 minutes'
GROUP BY 1,2;
```

Expected: zero rows with `outcome='error'` and error text about "connection pool" or "Can't reach database".

## Out of scope

- Any Prisma version bump.
- Splitting the pool into read/write.
- Changing the reliability-loop concurrency model (keep `Promise.allSettled` fan-out).
- Fixing the `dispatch_retry_storm` body bug — tracked separately in OPER-48.

## Non-negotiables

- No new tables, no migration.
- `DATABASE_URL` must remain a valid fallback (dev + local tests).
- Do not remove the `globalThis.__prisma` HMR-cache pattern.
- PR title: `fix(database): explicit Prisma pool config for reliability loop (OPER-49)`
- PR body must include `Ticket: OPER-49` trailer.

## Post-merge follow-through

- Add a runbook entry `docs/operations/pipeline-runbook.md#prisma-pool-exhaustion` describing:
  - How to spot pool exhaustion in ReliabilityEvent (`error` outcome + "connection pool" errorText)
  - Which env var to bump and to what value
  - How to verify recovery (query above)

If the runbook file exists, append; do not create a duplicate.

Attachments

Loading attachments…

Comments

Loading comments…