Production Data Enriched
What we did: took a fresh copy of the production index from 2.1, and enriched it. This is that copy, audited the same way.
App name: AC2 (Algolia-Central2's enriched copy of production)
App ID: 0EXRPAXB56 Index: AC2_WWW_MULTI_NEURAL
Lineage: this index started life as a copy of the same algolia.com content documented in 2.1. Session 5-6 (SESSION.md) ran a crawler-enrichment pass against it, adding body, h1, and crawled_* fields that 2.1 does not have at all. This section documents where that enrichment landed, in full, not just the parts that worked. See Chapter 0 for the decision record on why this is framed as lineage, not a competing index.
Big picture
The enrichment pass added 6 fields 2.1 never had: body, h1, crawled_metaDescription, crawled_bodyLen, enriched, added_by_crawl. Where it ran, it's real: one sampled record carries a 26,805-character crawled body and a distinct h1. But it only ran against 40.6% of this copy's 8,353 records. The other 59.4% still look exactly like unenriched 2.1 records.
The clearest win: this copy has zero duplicate records, versus 2.1's 28.4%. Whatever process built this copy deduped cleanly, or never introduced the chunk-collision bug documented in 2.1.
Everything else is not simply better. This copy's topic metadata (tags, keywords, authors, facets) is thinner than 2.1's, not richer, and its description/abstract duplication rate (30.9%) is worse than 2.1's (23.8%). It also carries the exact same structural gap as 2.1: zero category hierarchy for Documentation, its own single biggest section. The enrichment work was real but partial, and it didn't touch every dimension of data health, only the one it targeted (body text).
Action items, sorted by bang for the buck
| Fix | Impact | Effort | Action | |
|---|---|---|---|---|
| 1 | Complete the enrichment pass Only 40.6% of records have body/h1/crawled_* fields at all |
High | Med | The enrichment pipeline exists and works (see example below), it just hasn't been run against the other 59.4% of records. Highest priority here because it's this copy's actual reason to exist. |
| 2 | Link-health sweep 54.1% untracked, ~4,519 records |
High | Low | Same script as 2.1's recommendation. Worse coverage here than in 2.1 (45.9% vs 49.7% tracked). |
| 3 | Backfill category structure Documentation (4,330) + Developers (290) + Customer Stories (83) = 56.3% of this copy |
High | Low | Identical gap to 2.1, same fix. The enrichment pass didn't touch this, so it carried straight through from the original copy. |
| 4 | Backfill topic metadata tags 20.6%, keywords 18.6%, authors 13.9%, facets 18.2%, all worse than 2.1 |
Med | Med | This copy needs this more urgently than 2.1 does, since it's currently behind on every one of these fields, not just even with the baseline. |
| 5 | Differentiate description vs abstract 30.9% identical, worse than 2.1's 23.8% |
Med | High | Same editorial problem as 2.1, slightly worse here. |
| 6 | Filter out job postings 90 records, category: Careers |
Med | Low | Same issue as 2.1 (120 records there), same fix. |
Findings in detail, worst data quality first
Click any finding to expand it.
Worst Documentation has zero category structure here too
hierarchicalCategories. Same for Developers (290) and Customer Stories (83).title: "Analytics API" · url: /doc/rest-api/analytics · category: "Doc" · hierarchicalCategories: null
Being identical in shape to 2.1's Documentation gap (both exactly 0%, both the largest section) confirms this came from the source content pipeline, before the copy was even taken, not something the enrichment work introduced or could have fixed on its own.
Worst Link health is untracked for more than half this copy
is404 value at all, worse than 2.1's 50.3% untracked.title: "devcon" · url: /devcon · source: Other · is404: trueSame HTTP sweep as 2.1, run against this copy's URLs.
Same trust risk as 2.1, slightly larger blind spot.
Bad Topic metadata is thinner here than on the unenriched copy
| Field | 2.2 (this copy) | 2.1 (baseline) |
|---|---|---|
| tags | 20.6% | 31.5% |
| keywords | 18.6% | 29.4% |
| authors | 13.9% | 22.2% |
| facets | 18.2% | 28.8% |
The enrichment pass added body/h1 fields but did not preserve or improve the topic-metadata fields that already existed at copy time. Whatever process took the fresh copy either dropped some metadata in transit, or the source has been separately updated since the copy was taken and this copy hasn't re-synced.
Mid The enrichment pass only covers 4 in 10 records
body, h1, and the crawled_* enrichment fields at all. The other 59.4% are unenriched, indistinguishable from 2.1's plain records.title: "Conversational AI in ecommerce..."body: "Listen to the brief: Your browser does not support the audio element. Most ecommerce teams have already tried some form of chatbot..." (26,805 chars)h1: "Conversational AI in ecommerce: use cases, implementation, and real-world ROI"
Re-run the existing crawler-enrichment pipeline (already built, see scripts/setup/enrich/ in this repo) against the remaining 59.4%.
An inconsistent enrichment layer is arguably worse than none: 6 in 10 records look thin, 4 in 10 look rich, purely due to which batch they were in, not their real content.
Mid 1 in 3 records duplicates description into abstract
description & abstract, byte-identical: "Explore how conversational AI transforms ecommerce through product discovery, personalized recommendations, and support automation, including ROI benchmarks."
Good Zero duplicate records
Whatever process built this copy either deduped on ingest or never introduced the chunk-collision bug documented in 2.1. Worth understanding why, so the same discipline can be applied back to the source pipeline for 2.1.
Where things stand, by field tier
| Tier | Fields | Status |
|---|---|---|
| Clean | title, url, published_at, lastUpdated (97-100%) | Near-universal |
| Present, thin in places | description (92.6%), abstract (94.0%), category (93.0%), thumbnail (96.1%) | Similar to 2.1 |
| Structurally sparse, worse than 2.1 | tags, keywords, authors, facets (13.9-20.6%) | Behind the baseline on this dimension |
| Half-tracked | is404 (45.9%), hierarchicalCategories (41.5%) | Same shape of gap as 2.1 |
| New from enrichment, partial | body, h1, crawled_bodyLen, crawled_metaDescription, enriched (all ~40.6%) | A real capability 2.1 doesn't have, run on less than half this copy |
Search behavior on this copy — read with a hard caveat
This copy's analytics show only 59 distinct queries and 522 total searches over the same 90-day window 2.1 logged 5,514,444 searches for, with an 84.7% click rate and a 37.8% average daily conversion rate. It is not a real comparison. This copy is not visitor-facing, it's a build/eval index. This traffic is almost certainly the honing-gym harness, bait-query tests, and A/B eval scripts already documented in SESSION.md, not real people. Never quote these two traffic sets side by side as if they answer the same question.