Algolia-Central2

eval-loop-product.md

Eval-Loop — AI Judge + Autocorrect

A provider-agnostic toolkit for measuring LLM answer quality and automatically improving it, extracted from the Algolia Answer-Quality Lab as a standalone, reusable sub-product. Packaged as a Claude Code plugin (eval-loop/ in the repo; installed globally at ~/.claude/skills/{ai-judge,autocorrect-loop} + ~/.claude/eval-loop/packages/).

Purpose

Beating keyword links is trivial; beating an LLM's own generated answers (e.g. Algolia Ask AI) is the real bar — and proving it requires a trustworthy, reproducible judge plus a way to systematically improve the system under test. These two pieces are valuable far beyond this one project, so they are built as injectable, dependency-light modules.

Component 1 — AI Judge (`lab/judge/`)

What: scores one answer with a blind 3-persona panel (Skeptic / Referee / Advocate) on a weighted rubric, reconciled to one consensus quality score, plus a grounding hard-gate (an answer that states facts its sources don't support is capped, regardless of prose).
Portability: the only external seam is one injected LlmComplete function (prompt, opts) => Promise<string>. No vendor SDK is imported. Bring any model with a ~30-line adapter.
Entry: judgeArtifactMultiRound(artifact, DEFAULT_JUDGE_CONFIG, llm, rounds) → { aggregate: { finalScore, meanPreGateScore, gateTripped, dimensionMeans, ... } }.

Component 2 — Autocorrect loop (`lab/autocorrect/`)

What: Karpathy-"AutoResearch"-style loop: evaluate → diagnose weakest dimension → propose a config change → re-evaluate → keep only if measurably better, else roll back. Grounding is a hard constraint; an overfit guard validates wins on a held-out split.
Portability: the loop core knows nothing about the system under test. Inject 3 seams: deploy(config), evaluate(split), propose(...). Algolia wiring (deploy via Agent Studio, evaluate via the harness, propose via an LLM) lives in lab/server/src/autocorrectAdapters.ts.
Entry: runAutocorrect({ seams, baseline, cfg }) → { best, stopReason, history }.

Decision: zero-flicker is non-negotiable (2026-06-13)

Rationale: the loop decides keep/rollback from the judge score. If the same answer scores differently run-to-run, the loop chases noise — keeps bad changes, discards good ones. A self-correcting system on an unreliable metric is worse than no system.

Root cause of the flicker (proven, not assumed): (1) the multi-round gate counted heterogeneous per-round trips — a different imagined claim each round could gate; (2) the gating Skeptic ran at temperature 0.2, so its violation confidence wobbled across a sharp 0.7 cutoff → the same answer scored 3 or 8 (±5 swing).

Fix: (1) claim-recurrence gate — cluster violations across rounds by stemmed token-Jaccard similarity; trip only when the same claim recurs in a supermajority of rounds (one-off flags = noise). (2) All judges at temperature 0 — diversity comes from the personas, not random sampling.

Proof: judged identical answers twice end-to-end → 0 / 21 grounding-gate flips. Honest residual: the continuous quality score still varies ≤ ~0.3 (the provider is not bit-deterministic even at temp 0) — but it never flips a gate decision and is far below the win target.

Consequence / how to apply: the loop is noise-safe by construction — minImprovement (set to the measured noise floor, ~0.3 for Gemini) means a gain smaller than the noise is treated as "no change." The loop can never act on a wrong measurement; at worst it does nothing. Two standing invariants when reusing: keep judges at temperature 0; always measure the noise floor and set minImprovement above it before trusting verdicts.

Open questions / next

Full optimization run (more rounds / larger budget) — only a smoke + a first real run done so far.
Loop efficiency: evaluate currently re-runs the fixed floor (② Ask AI) every round (~2× cost) — cache the floor before long runs.
The global plugin copy lives outside git; the canonical/committed copy is eval-loop/ in the repo (refresh the global copy after edits).