Dashboard/Algolia-Search-Audit/Known-Issues.md
Algolia-Search-Audit

Known-Issues.md

Algolia Search Audit — Known Issues

Comprehensive list of failure modes, bugs, and design debt. As of 2026-04-08.

Critical — Pipeline-Breaking

1. Context Window Exhaustion (Root Cause of Incomplete Audits)

Symptom: Audit skills 3+ in the pipeline get skipped, truncated, or produce garbage. Cause: The Skill tool runs inline in the current context window. "Isolated agents" are actually sharing one 200K context. By skill #3, context is 70-80% full. The LLM enters "context anxiety" — wraps up prematurely. Impact: Out of 20 modules, typically only 4-6 execute properly. Requires 4-6 hours of manual shepherding. Fix: Python harness calling claude -p as subprocess — one fresh context per skill. See Refactor-Architecture.

2. Report Reads .md Files, Not .json

Symptom: algolia-audit-report ingests 11+ prose research documents into context. Cause: Skill reads research/*.md to assemble audit-data.json. Impact: Report skill alone can fill 60-80% of context. Produces incomplete or hallucinated data. Fix: Switch to reading .json files exclusively. Use generate-audit-data.py to merge JSONs.

3. Partner Intel Has No JSON Output

Symptom: 07-partner-intel.md exists but no .json. Everything downstream parses prose. Cause: algolia-intel-partner never had a JSON output defined. Impact: Report skill must LLM-extract partner data from Markdown. Inconsistent, lossy. Fix: Add 07-partner-intel.json output with Pydantic schema.

4. No Pydantic Schemas — Schema Drift Across Runs

Symptom: Same module produces different JSON field names on different runs. Report fails to find expected fields. Cause: No schema validation on any intermediate output. Each LLM invocation invents its own structure. Impact: Downstream skills break when fields are missing, misspelled, or nested differently. Fix: Pydantic model per module, validated by harness before marking complete.

High — Data Quality

5. calculate-roi.py Exists But Isn't Used

Symptom: Business case ROI numbers vary wildly between runs. Cause: algolia-synth-business-case does all 6 ROI formulas via LLM. The Python script calculate-roi.py exists in scripts/ but isn't wired into the skill. Impact: ROI numbers are inconsistent. Same inputs → different outputs. Fix: Wire calculate-roi.py into the pipeline. LLM adds narrative only.

6. Financial-Public Revenue Fields Nested Instead of Top-Level

Symptom: revenue_fy2025 ends up inside financials.* instead of at the JSON root. Cause: LLM ignores "CRITICAL: top-level keys" instruction. The skill mentions this twice, which signals it's a recurring problem. Impact: Report skill can't find revenue data. ROI calculation breaks. Fix: Pydantic schema enforces field position. Validation rejects nested placements.

7. margin_zone Calculated by LLM

Symptom: GREEN/YELLOW/RED classification is sometimes wrong. Cause: 3-line math (>40% = GREEN) done by LLM inference instead of Python. Impact: Incorrect risk classification in reports. Fix: Python function in calculate-roi.py or separate utility.

8. SEC/Earnings Data Fetched Twice (1E + 1G Overlap)

Symptom: Same SEC 10-K and earnings transcripts fetched by both financial-public and investor skills. Cause: No shared utility. Both skills independently WebFetch the same documents. Impact: Wasted API calls, context budget, and potential for different extractions from same source. Fix: Shared collect-sec-data.py — fetch once, both skills read from output.

9. ABX Campaign Mutates audit-data.json

Symptom: audit-data.json changes after factcheck has run. Cause: algolia-campaign-abx appends email bodies into audit-data.json. Impact: Factchecked file is modified post-factcheck. Report may be stale. Fix: ABX writes to abx-data.json. Report template reads both files.

10. meta vs _meta Key Confusion

Symptom: Some modules write _meta, others write meta. Downstream reads fail. Cause: Inconsistent field naming across skill definitions. Competitor skill has a literal typo: "use meta key, NOT meta". Impact: Gate checks for meta.skill_enrichment_completed fail silently when key is _meta. Fix: Pydantic schema uses meta everywhere. _meta rejected by validation.

Medium — Consistency

11. Algolia Customer List Scraped Every Run

Symptom: Each audit scrapes algolia.com/customers to find case studies. Cause: No static dataset. LLM browses the customer page. Impact: Slow, inconsistent results. Same customer list changes interpretation between runs. Fix: Curated algolia_customers.json — domain/vertical/metric/URL — refreshed monthly.

12. No Observability

Symptom: No way to know which modules ran, which failed, timing, cost. Cause: audit-progress.jsonl is defined in the orchestrator but skills don't consistently write to it. Impact: When audit fails, no way to diagnose where without reading all files manually. Fix: Harness manages progress log externally. Per-module cost/token tracking.

13. Factcheck Does 15 Mechanical Checks via LLM

Symptom: Factcheck takes 30-40 minutes and costs $15+. Cause: "Does file X exist?" and "Is field Y not null?" are done by LLM inference. Impact: Expensive, slow, inconsistent (LLM might miss a null field). Fix: factcheck-mechanical.py for dims 1-12 and 18-20. LLM only for dims 13-17.

14. Hiring Role Classification is LLM Judgment

Symptom: Same title ("VP Digital Commerce") classified differently across runs. Cause: No deterministic title → tier mapping. Impact: ICP mapping inconsistency. Fix: Keyword dictionary: "VP"/"Director"/"SVP" → Economic Buyer; "Engineer"/"Developer" → Technical Buyer; "Product Manager"/"Search" → Champion. LLM fallback for ambiguous.

15. Query Generation Has No Structured Output

Symptom: 05-test-queries.md has different format every run (numbered list vs table vs sections). Cause: No output schema. LLM chooses format each time. Impact: Browser audit skill must parse unpredictable Markdown to extract queries. Fix: Output as JSON with typed arrays per query type.

Low — Nice to Have

16. Industry Benchmarks Re-Scraped Every Run

Could be a curated vertical-benchmarks.json with Baymard/Forrester stats, refreshed quarterly.

17. Social/News Relevance Scoring is Expensive

LLM scores every scraped post for relevance. Keyword list would handle 90% of cases.

18. No Shared Source Labeling Utility

Every skill re-implements [FACT — source, date] formatting. Should be a Python utility.

19. Browser Audit Screenshot Naming Inconsistent

Sometimes step-2a-query.png, sometimes autocomplete-test.png. Needs convention.

20. Capability Matrix Hardcoded Competitor Names

Report template uses nike_has, asics_has — must be dynamically mapped to actual competitors.