Current Production Data
The baseline. Production data exactly as it stands today, on the live Algolia application serving search for algolia.com, before any enrichment work.
App name: Visibility (Algolia's live incumbent search)
App ID: 1QDAWL72TQ Index: ALGOLIA_WWW_PROD_V2
This is the corpus any Content Engagement build would ground itself on, so its real condition matters before any UI work starts. 2.2 documents the fresh copy of this same data after enrichment; this section is the "before" against which that work is measured.
Big picture
A content-engagement agent needs three things before it can proactively engage a visitor: it needs to know what a piece of content is about (category, tags), whether it's safe to reference (not a dead link, not disabled), and whether it has something distinct to say (a real, non-duplicated description). Today the corpus is reliable for exactly none of those three across the board, and the gaps are concentrated in the two sections closest to the POC's named journeys.
Documentation is 49.6% of the entire corpus (8,500 of 17,138 records) and has zero category structure. Resources (ebooks, white papers) sits closest to case-study/pricing-type journeys, and 43.6% of it has no description at all. Half the index has no link-health signal, so roughly 1 in 2 records is an unknown risk of pointing a visitor at a dead page. Just over a quarter of all records (28.4%) are duplicates of a URL already in the index, including one confirmed indexing bug where the same ebook was written into 38 separate records.
None of this blocks starting the POC. Two of the highest-impact fixes below (link-health sweep, category backfill) are also the cheapest, because both can be derived automatically from data that already exists (the URL itself) rather than requiring new content to be written.
Action items, sorted by bang for the buck
Best return on effort first. Impact = how much it unblocks Content Engagement. Effort = how much work the fix actually takes. Tracked live in the Tracker.
| Fix | Impact | Effort | Action | |
|---|---|---|---|---|
| 1 | Link-health sweep 50.3% of corpus, ~8,600 records untracked |
High | Low | Automated HTTP status check across all 12,278 distinct URLs. No content team needed, fully scriptable, re-runnable on a schedule. |
| 2 | Backfill category structure Documentation (8,500) + Developers (870) + Customer Stories (265) = 56.2% of corpus |
High | Low | Parse hierarchicalCategories from the existing URL path. The structure already exists in the URL, it just hasn't been lifted into the field. Deterministic, scriptable. |
| 3 | Filter out job postings 120 records tagged category: Careers |
Med | Low | One category exclusion filter on whatever retrieval scope an engagement agent uses. |
| 4 | Dedupe by URL 28.4% of all records, 4,860 of them |
High | Low | Dedupe at query/retrieval time immediately. Separately, flag the chunk-ID collision bug (one ebook indexed 38 times) to whoever owns the crawl/ingestion pipeline. |
| 5 | Backfill Resources descriptions 43.6% of Resources, 962 total records |
High | Med | Extract real summaries from the underlying PDF/ebook assets, not CMS metadata that was never filled in. |
| 6 | Auto-generate tags/keywords 70-78% of corpus has an empty array, worst in Documentation |
Med | Med | LLM extraction from title + description + body, prioritizing Documentation. Needs a quality check pass before trusting it for live topic-routing. |
| 7 | Differentiate description vs abstract 23.8% of records with both fields filled, ~3,729 records |
Med | High | Actual editorial rewriting, not scriptable. Someone has to decide what each field is for, then write two different things. |
| 8 | Resolve algoliaDisabled ambiguity 11.9% of corpus, 2,033 records, always false |
Unknown | Low | Not a data fix, a question: ask whoever owns the ingestion pipeline whether disabled content is filtered before indexing or the flag is vestigial. |
Findings in detail, worst data quality first
Click any finding to expand it.
Worst Half the corpus's biggest section has zero category structure
hierarchicalCategories data. Same for Developers (870) and Customer Stories (265).objectID: https://www.algolia.com/doc/value-engineering/radartitle: "Radar"category: "Doc"hierarchicalCategories: null
Backfill from the URL path (e.g. /doc/value-engineering/radar → lvl0 "Documentation", lvl1 "Value Engineering"). Deterministic, scriptable.
An engagement agent can't reason about topic clusters for 56.2% of the entire corpus, the single biggest content section.
Worst Link health is a coin flip for half the index
is404 value at all. The rest is not "confirmed live", it's simply untracked.title: "Workplace App Search | Algolia" · url: /fr/developers/code-exchange/workplace-app-search · is404: trueRun an HTTP status sweep across all 12,278 distinct URLs. Re-run on a schedule.
An engagement agent risks sending a visitor to a dead page for half the index, with zero warning. Directly damages the "human companion" trust the whole pitch depends on.
Bad Resources, the section closest to your target POC journeys, is the thinnest
title: "7 ways to get more out of algolia search" · category: "Ebooks" · description: "" · abstract: ""
Extract real summaries from the underlying PDF/ebook body, not CMS metadata that was never filled in.
For roughly 4 in 10 Resources assets, an agent has nothing to ground a conversation on beyond a title. Directly blocks the stated POC scope.
Bad Duplicate content, including one confirmed indexing bug
/fr/resources/asset/ebook-technical-buyers-guide-to-site-search appears 38 times, byte-identical content except objectID, sharing one base UUID with a chunk-index suffix.
Dedupe by URL at retrieval time. Flag the chunk-ID collision to the crawl/ingestion pipeline owner.
A related-content feature could recommend the same ebook two or three times in a row. Obvious, embarrassing tell in a demo meant to prove the concept works.
Mid Topic metadata is structurally sparse, not just occasionally missing
tags, keywords, or authors array.LLM extraction from title + description + body, starting with Documentation.
No cheap way to know what a page is about without reading the full body on every contextual decision, at the exact scale (8,500 records) where it matters most.
Mid 1 in 4 records pays for two fields to say the same thing once
description and abstract filled have them byte-identical.Decide what each field is actually for, then write two different things instead of copy-pasting.
An engagement opener built on abstract reads like SEO meta-copy for 1 in 4 pieces of content, not a human companion.
Minor Job postings are sitting inside a content-engagement corpus
department/location fields, actual job postings.Exclude category: "Careers" from product/marketing retrieval scope.
Cheap to fix now. Left alone, a context-aware agent could surface a job posting to someone reading a case study.
Where things stand, by field tier
| Tier | Fields | Status |
|---|---|---|
| Clean | title, url, language_code, source, published_at, lastUpdated, environment | 100% present & meaningful |
| Present, thin in places | description, abstract, category, thumbnail | 93-99% present, real gaps concentrated in Resources |
| Structurally sparse | tags, keywords, authors, facets | 22-32% meaningfully filled |
| Half-tracked | is404, hierarchicalCategories | Genuinely unknown for large parts of the corpus |
| Vestigial | algoliaDisabled, department/location | Minor or off-topic for content engagement |
Search behavior on this same application
Analytics were pulled from the same app (1QDAWL72TQ) covering 2026-04-11 to 2026-07-09 (90 days, fully paginated: 58,567 distinct queries, not a top-N sample). Headline findings:
- Conversion rate is 0.00% on every single day of the 90-day window, verified at the row level across all 58,567 distinct queries, not just in aggregate.
- Only 1,845 of 58,567 distinct queries (3.15%) ever received a click.
- There is a large, real population of high-volume, near-zero-satisfaction queries in exactly the AI/agentic-search topic space (e.g. "building agentic ai": 10,434 searches, 6 clicks), which is directly relevant to what Content Engagement is trying to build for.
- A real, quantified non-English content gap exists: the top zero-result queries by volume are dominated by German and French text.
- A large share of raw "search volume" (54% empty-query, plus widget/title-echo noise) is not organic human query behavior and should not be quoted as a headline number without that caveat.
Full detail: docs/algolia-data/analytics.md in this repo.