03 — Record Extraction: recordExtractor & Helpers

What recordExtractor does

recordExtractor is a JavaScript function you write that receives the page content and returns an array of Algolia records. It runs once per matched URL.

recordExtractor: ({ url, $, contentLength, fileType, dataSources, helpers }) => {
  // url     — Location object: url.href, url.pathname, url.hostname, url.search
  // $       — Cheerio instance (jQuery-like DOM traversal on the raw HTML)
  // contentLength — page size in bytes
  // fileType — "html", "pdf", "doc", etc.
  // dataSources — external data mapped by source ID
  // helpers — built-in extraction utilities

  return [
    {
      objectID: "...",   // if autoGenerateObjectIDs: false, MUST be present
      // ... your fields
    }
  ]
  // Return [] to skip this page
}

---

Built-in helpers

helpers.page() — generic page

const records = helpers.page()

Returns: { objectID, url, hostname, path, depth, fileType, contentLength, title, description, keywords, image, headers, content }

Best for: generic informational pages where you want all text content as a single record.

helpers.article() — structured article

const records = helpers.article()

Returns: { objectID, url, lang, headline, description, keywords, tags, image, authors, datePublished, dateModified, category, content }

Requires: og:type="article" or JSON-LD schema (Article, NewsArticle, BlogPosting, Report).

helpers.product() — product pages

const records = helpers.product()

Returns: { objectID, url, lang, name, sku, description, image, price, priceCurrency, category }

Requires: JSON-LD Product schema on the page.

helpers.docsearch() — documentation

const records = helpers.docsearch({
  recordProps: {
    lvl0: { selectors: "header h1" },
    lvl1: { selectors: "article h2" },
    lvl2: { selectors: "article h3" },
    content: { selectors: ["article p", "article li"] }
  },
  aggregateContent: true,
  indexHeadings: true
})

Returns hierarchical records with lvl0–lvl6 fields. Designed for documentation navigation.

helpers.splitContentIntoRecords() — long pages

const records = helpers.splitContentIntoRecords({
  $elements: $("article p, article li"),
  baseRecord: { url: url.href, title: $("h1").text() },
  maxRecordBytes: 10000,
  textAttributeName: "content",
  orderingAttributeName: "position"
})

Splits a large page into multiple records, each under maxRecordBytes. Use when a single page would produce a record too large for Algolia (10KB attribute limit) or when you want finer-grained search.

helpers.codeSnippets() — code extraction

const snippets = helpers.codeSnippets()

Returns: [{ content, languageClassPrefix, codeUrl }] — extracted from <pre> elements.

---

Actual RC2 Algolia index schema (live — verified 2026-04-30)

Queried from algolia-central_enterprise_ledger (app 0EXRPAXB56). 10,615 records, 9,615 with status: indexed.

Parent doc record — full field schema (crawler sets ALL of these in one pass):

objectID         string   deterministic hash of normalized URL
record_type      string   "enterprise_ledger"
url              string   normalized, no fragment, no tracking params
title            string
description      string   meta description
content          string   clean page text, max 9000 chars
source_type      string   doc/support/blog/other/developer/customer_story/academy/resource/changelog
language_code    string   "en", "fr", etc. from html[lang]
is_chunk         boolean  always false (crawler never creates chunks)
status           string   "indexed" (one-pass — no queue)
created_at       number   unix timestamp
updated_at       number   unix timestamp

── Classification fields (all set in recordExtractor, all proper-case) ──
product_tag      string?  "AI Search"|"InstantSearch"|"Analytics"|"Recommend"|"Autocomplete"|"DocSearch"|
                          "Query Suggestions"|"Merchandising"|"Personalization"|"DocSearch"|null
feature_tag      string?  "Rules"|"Faceting"|"A/B Testing"|"Synonyms"|"Query Suggestions"|"Ingestion"|
                          "Monitoring"|"Geo Search"|"Highlighting"|"Pagination"|"Ranking"|"Typo Tolerance"|
                          "Crawler"|"Indexing"|"Event Tracking"|"Personalization"|null
solution_tag     string?  "Site Search"|"App Search"|"Headless Commerce"|"Recommendations"|"Merchandising"|
                          "Enterprise Search"|"Analytics"|"Personalization"|"Voice Search"|"Mobile Search"|null
industry_tag     string?  "Ecommerce"|"Retail"|"SaaS"|"B2B Commerce"|"Media"|"Marketplace"|"Healthcare"|
                          "Finance"|"Gaming"|"Education"|"Fashion"|"Grocery"|"Government"|"Non-profit"|null
customer_tag     string?  Customer name title-cased from /customers/{slug} URL. null if not a customer page.
                          Examples: "Lacoste", "Gymshark Headless", "Walgreens"
category         string?  "case_study"|"product"|"doc_section"|"vision"|null

── NOT set by crawler (belong to product_map records only) ──
industry_tags    —        plural field — product_map records only, NOT enterprise_ledger
features_used    —        product_map records only
integrations     —        product_map records only

Chunk record (is_chunk: true) — created by L2 Enrichment (NOT the crawler):

objectID         "{parent_uuid}_chunk_{index}"
parent_id        string   UUID of the parent doc
chunk_index      number
chunk_total      number
content          string   "Document Context: {summary}\n\nFragment: {chunk_text}"
title            string   "{title} (Part {n}/{total})"
is_chunk         true
[all other fields inherited from parent]

Key differences from the RC1 n8n/Supabase schema:

Supabase (RC1)	Algolia RC2	Note
markdown	content	renamed
doc_summary	summary	renamed
content_type	record_type + category	restructured
product_tag lowercase	product_tag proper case	"instantsearch" → "InstantSearch"
feature_tag snake_case	feature_tag proper case	"query_rules" → "Rules"
content_hash	not present	dedup via objectID instead
canonical_url	not present	handled pre-crawl
clean_char_count	not present	dropped
is_reject	ingestion_rejections table	separate table in RC1, not in Algolia
not present	solution_tag	new in RC2
not present	customer_tag	new in RC2
not present	industry_tag	new in RC2
not present	is_chunk, parent_id, chunk_index, chunk_total	chunking now in Algolia

source_type values (from live facet count):

doc: 5,475 | support: 1,732 | blog: 1,161 | other: 638 | developer: 256 | customer_story: 149 | academy: 139 | resource: 54 | changelog: 11

product_tag values (from live facet count, proper case):

"AI Search" (5,496) | "InstantSearch" (1,393) | "Analytics" (1,236) | "Recommend" (501) | "Autocomplete" (380) | "DocSearch" (37)

---

Custom extraction pattern (our use case)

For the L1 enterprise knowledge ledger crawl, we write a custom extractor that:

1. Extracts fields matching the current RC2 Algolia schema exactly
2. Sets source_type from URL routing (same logic as n8n Firehose Normalize fields node)
3. Sets status: "pending" — L2 flips to "indexed" after enrichment
4. Sets is_chunk: false — chunking is L2's job
5. Sets explicit objectID for deduplication
6. Returns [] for quality rejects (no separate rejection table in Algolia — just skip)

recordExtractor: ({ url, $}) => {
  const rawUrl = url.href
  const urlLower = rawUrl.toLowerCase()
  const pathLower = url.pathname.toLowerCase()
  const host = url.hostname.toLowerCase()

  const title = $("title").text().trim() || $("h1").first().text().trim()
  if (!title) return []

  // source_type — same routing logic as n8n Firehose "Normalize fields"
  let source_type = "other"
  if (host.includes("support.algolia.com") && pathLower.includes("/hc")) source_type = "support"
  else if (host.includes("academy.algolia.com")) source_type = "academy"
  else if (host.includes("changelog.algolia.com")) source_type = "changelog"
  else if (host.includes("stories.algolia.com")) source_type = "customer_story"
  else if (/\/doc(?:\/|$)/.test(pathLower) || /\/docs(?:\/|$)/.test(pathLower)) source_type = "doc"
  else if (/\/developers(?:\/|$)/.test(pathLower)) source_type = "developer"
  else if (/\/customers(?:\/|$)/.test(pathLower)) source_type = "customer_story"
  else if (/\/blog(?:\/|$)/.test(pathLower)) source_type = "blog"
  else if (/\/resources(?:\/|$)/.test(pathLower)) source_type = "resource"

  // Quality gate: skip canonical duplicates
  const canonical = $('link[rel="canonical"]').attr("href")
  if (canonical && canonical !== rawUrl) return []

  // Content extraction — prefer article/main over full body
  let content = $("article, main, [role='main'], .content").text().replace(/\s+/g, " ").trim()
  if (!content) content = $("body").text().replace(/\s+/g, " ").trim()
  if (content.length < 300) return []  // quality gate

  // Deterministic objectID — dedup anchor
  const objectID = require("crypto")
    .createHash("sha256").update(rawUrl).digest("hex").substring(0, 36)

  const now = Math.floor(Date.now() / 1000)

  return [{
    objectID,
    record_type: "enterprise_ledger",
    url: rawUrl,
    title,
    description: $('meta[name="description"]').attr("content") || "",
    content: content.substring(0, 9000),
    summary: null,          // L2 fills: GPT 3-sentence summary
    source_type,
    language_code: $("html").attr("lang")?.split("-")[0] || "en",
    is_chunk: false,
    status: "pending",      // L2 flips to "indexed"
    created_at: now,
    updated_at: now,
    product_tag: null,      // L2 fills
    feature_tag: null,      // L2 fills
    solution_tag: null,     // L2 fills
    customer_tag: null,     // L2 fills
    industry_tag: null,     // L2 fills
  }]
}

---

Tagging strategy: crawl vs enrich

The crawler (recordExtractor) should NOT attempt ML-based classification. The correct separation is:

Layer	What it does
L1 Crawler	Extract clean text, set structural fields (url, title, description, content, domain, path, crawled_at)
L2 Enrichment	Apply ML classification: category, entity_type, industry, vendor, sentiment, etc.

The crawler sets category: null as a placeholder. L2 reads the index, classifies, and updates the record.

---

ObjectID strategy — deduplication anchor

autoGenerateObjectIDs: false — we own the objectID.

Design:

objectID = base64(url.href).slice(0, 64)

Why URL-based: - Same URL re-crawled → same objectID → saveObjects overwrites (no duplicate) - URL change → new objectID → new record (correct — it's a different page) - Normalisation: strip UTM/session params via ignoreQueryParams before this

Alternative: use Algolia's distinct: true + attributeForDistinct: "url" for query-time dedup. We use indexing-time dedup (objectID) as primary, query-time distinct as secondary safety.

---

Handling the 750-record-per-page limit

If a page generates > 750 records (e.g., a very long article split into paragraphs), the crawl fails. Use:

// Check estimated record count and fall back to page-level record if too many
const estimated = $("p").length
if (estimated > 500) {
  return [{
    objectID: ...,
    url: url.href,
    content: $("body").text().substring(0, 8000)
  }]
}

Or use helpers.splitContentIntoRecords() with a byte budget that keeps you well under limit.

---

Filtering pages to index

Return [] to skip a page. Use this to exclude: - Navigation-only pages (no real content) - Error pages (check $("title").text().includes("404")) - Pages behind login (check for login form presence) - Pages with too-short content (< 200 chars) - Duplicate content detected via canonical tag mismatch

// Skip if canonical points to a different URL
const canonical = $('link[rel="canonical"]').attr("href")
if (canonical && canonical !== url.href) return []