Data health — current production data

App: Visibility (Algolia's live incumbent search) — App ID 1QDAWL72TQ, index ALGOLIA_WWW_PROD_V2. This is the live Algolia application serving search on algolia.com today. Full audit, 17,138 of 17,138 records read (verified against Algolia's own browse-reported count, not sampled). Fetched 2026-07-10/11.

Full detail with real record examples: docs/algolia-data/ in the Algolia-Central2 repo, and the interactive version at bible.chowmes.com/ac2/chapter-2-1-current-production-data.html.

Big picture

A content-engagement agent needs three things before it can proactively engage a visitor: to know what a piece of content is about (category, tags), whether it's safe to reference (not a dead link, not disabled), and whether it has something distinct to say (a real, non-duplicated description). Today this corpus is reliable for none of those three across the board, and the gaps concentrate in the two sections closest to the POC's target journeys.

Documentation is 49.6% of the entire corpus (8,500 of 17,138 records) and has zero category structure. Resources (ebooks, white papers — closest to case-study/pricing journeys) has 43.6% of records with no description at all. Half the index has no link-health signal. 28.4% of all records are duplicates of a URL already in the index, including one confirmed indexing bug (one ebook written into 38 separate records).

None of this blocks starting the POC — the two highest-impact fixes (link-health sweep, category backfill) are also the cheapest, since both derive automatically from data that already exists (the URL itself).

Action items, ranked by impact vs effort

1. Link-health sweep (High impact, Low effort) — 50.3% of corpus (~8,600 records) untracked for is404. Automated HTTP status check across all 12,278 distinct URLs.
2. Backfill category structure (High impact, Low effort) — Documentation + Developers + Customer Stories = 56.2% of corpus with zero hierarchicalCategories. Parse from the existing URL path, deterministic.
3. Filter job postings (Med impact, Low effort) — 120 records tagged category: Careers sitting inside the content corpus.
4. Dedupe by URL (High impact, Low effort) — 28.4% of records (4,860) are 2nd+ copies.
5. Backfill Resources descriptions (High impact, Med effort) — 43.6% of Resources has no description/abstract; needs extraction from underlying PDF/ebook assets.
6. Auto-generate tags/keywords (Med impact, Med effort) — 70-78% of records have an empty tags/keywords/authors array.
7. Differentiate description vs abstract (Med impact, High effort) — 23.8% of records with both fields filled have them byte-identical; needs editorial rewriting.
8. Resolve algoliaDisabled ambiguity (Unknown impact, Low effort) — present on only 11.9% of records, always false; needs a person to answer whether disabled content is filtered pre-index or the flag is vestigial.

Field-tier summary

Search behavior on this application

Analytics pulled from the same app, 2026-04-11 to 2026-07-09 (90 days, fully paginated — 58,567 distinct queries, not a top-N sample):

• Conversion rate is 0.00% on every single day of the 90-day window, verified at the row level across all 58,567 distinct queries, not just in aggregate.
• Only 1,845 of 58,567 distinct queries (3.15%) ever received a click.
• A large, real population of high-volume, near-zero-satisfaction queries sits exactly in the AI/agentic-search topic space (e.g. "building agentic ai": 10,434 searches, 6 clicks) — directly relevant to what Content Engagement is trying to build for.
• A real, quantified non-English content gap: top zero-result queries by volume are dominated by German and French text.
• 54% of raw "search volume" is the empty query (browse behavior, not typed intent), plus further widget/title-echo noise — don't quote raw search-volume numbers without this caveat.

See data-health-enriched-copy for the same audit run against the enriched AC2 copy, and data-health-final-verdict for whether that enrichment actually improved on this baseline.