01 — Algolia Crawler: Overview & Architecture

What it is

The Algolia Crawler is Algolia's managed web crawling service. It:

1. Visits a set of start URLs (or discovers them from sitemaps)
2. Follows links to discover additional pages
3. Runs a recordExtractor function on each matching page to produce Algolia records
4. Writes those records directly into a target Algolia index
5. Handles scheduling, deduplication, and safe re-indexing out of the box

You do not write or host any scraping infrastructure. You write config (JavaScript object) and the managed service runs it.

Core data flow

Seeder list / startUrls / sitemaps
        ↓
  URL Discovery Queue
  (discoveryPatterns, following links)
        ↓
  pathsToMatch filter
  (only matching URLs get extracted)
        ↓
  recordExtractor(url, $, helpers, dataSources)
  → returns JSON array of records
        ↓
  Algolia Index (target indexName)
  (with safety checks, safetyChecks.maxLostRecordsPercentage)

Three entry points

We use the standard crawler.

Key advantages over custom Apify pipeline

1. No middleware — records go straight from extractor → Algolia index
2. URL-based deduplication built-in — crawler manages objectIDs, won't double-index
3. Safety net — safetyChecks.maxLostRecordsPercentage (default 10%) stops a bad crawl from wiping your index
4. Cache-aware re-crawls — by default, caching is on; only modified pages are re-processed on incremental runs
5. Monitoring dashboard — Monitoring, Inspector, URL Tester, Path Explorer, Data Analysis — no custom logging needed
6. JavaScript rendering — built-in headless browser via renderJavaScript config
7. Scheduling — cron-like schedule in config, no external scheduler

What the crawler does NOT do

• It does not enrich records with external data automatically (you do that in recordExtractor via dataSources)
• It does not run ML classification or entity extraction (that's our L2 job)
• It does not guarantee crawl order (BFS-ish traversal)
• It does not support authenticated pages out of the box (custom headers / cookie injection needed)

Record model

Every record returned by recordExtractor must be a JSON object. If the array is empty, the page is skipped. If autoGenerateObjectIDs: true (default), Algolia assigns a UUID objectID. For our use case we will set objectID explicitly (URL-based hash) to control deduplication.

The crawler replaces an existing record with the same objectID on re-crawl. This is the deduplication mechanism.

Limits to be aware of

• Records per page: max 750 records per recordExtractor call. Use helpers.splitContentIntoRecords() for large pages.
• Crawl URLs API: rate limited to 500 requests per 24 hours (POST to /1/crawlers/{id}/urls/crawl)
• rateLimit: controls concurrent tasks/second; start conservative, raise incrementally
• renderJavaScript: headless browser timeout defaults to 20s; enable only for dynamic pages (slower + inconsistency risk)
• URL dedup: crawler normalises URLs based on ignoreQueryParams config — UTM params etc. should be stripped

Monitoring capabilities

The Crawler dashboard provides four operational views: