Dashboard/Algolia-Central2/wiki/syntheses/llm-harness-reliability.md
Algolia-Central2

wiki/syntheses/llm-harness-reliability.md

LLM Harness Reliability — SOP & Runbook

Canonical, always-current copy lives in the repo: Algolia-Central2/docs/sop/llm-harness-reliability.md (read that for the full Failure Catalog table + Pre-Run Checklist + file references). This vault page is the discoverable summary; sync from the repo when it changes.

Why this exists

The 2026-06-14 autocorrect-loop session crashed twice and ran ~4h before fixes. All failures were a single class: unreliable batch-LLM / agent-orchestration code. This SOP encodes the standards + a fix-runbook so the error class doesn't recur as we productize the lab / eval-loop harness.

Standing standards (one line each)

  1. Bounded parallelism for every multi-item LLM batch — never sequential await loops (mapWithConcurrency).
  2. Resilient provider clients — retry {429,500,502,503,504} + thrown network errors; injectable backoff.
  3. Guard every orchestration seam — one failed call degrades (record error / return incumbent), never crashes the run.
  4. Thinking-model token budgets — output ≈ 4–8× the answer (reasoning eats the budget → empty MAX_TOKENS).
  5. Cache invariant baselines in eval loops — re-measure only what changes.
  6. Extract pure logic for testability — no orchestration glue with zero tests.
  7. No fabricated data in the UI — empty-state CTAs, not mock scores.
  8. Observability in long runs — timestamps + heartbeat + final summary.
  9. Debugging discipline — judge progress from log mtime/tail (not a summary grep); reproduce against the real component before blaming; 0% CPU ≠ hung for I/O; keep a non-MCP REST fallback.

Failure catalog (summary — full table in the repo doc)

F1 slow/stuck loop → parallelize. F2 2× cost → cache floor. F3 fatal crash on MAX_TOKENS empty → bigger token budget + guard the seam. F4 transient 502/fetch failed poisoned scores → widen retry set + network-throw retry. F5 zero test coverage → extract pure fns + tests. F6 misjudged state / wrong root cause → evidence-first debugging. F7 no phase timing → observability. F8 mock data in UI → empty-state CTA.

  • finding-corrected-verdict · repo docs/experiment/finding-typo-tolerance-false-refusal.md
  • Memory: feedback-llm-harness-reliability