Algolia-Central
Knowledge/AlgoliaCrawler/README.md
Algolia Crawler — Knowledge Wiki
Created: 2026-04-30 | Author: Developer context for L1 Data Fetch (RC3 Phoenix) Purpose: Replace Apify scraper in the L1 Data Fetch pipeline with Algolia Crawler
What this wiki covers
The Algolia Crawler is Algolia's managed web crawling service. It visits URLs, extracts structured records, and populates Algolia indices — without you maintaining custom scraping code.
This wiki was built specifically to support the L1 Data Fetch workstream in RC3 Phoenix, where we are migrating from Apify → Algolia Crawler for the seeder-list-based crawl that feeds the enterprise knowledge ledger.
Routing table — task type → file
| Task | File |
|---|---|
| "How does the crawler work?" | 01-overview-and-architecture.md |
| "What config params do I need?" | 02-configuration-reference.md |
| "How do I extract and shape records?" | 03-record-extraction.md |
| "How do I control which URLs get crawled?" | 04-url-management.md |
| "How do I run, schedule, and monitor it?" | 05-operations-and-api.md |
| "How do we implement L1 Data Fetch?" | 06-l1-implementation-plan.md |
| "What are the Pydantic models and contract tests?" | 07-schema-contracts.md |
| "Complete production-ready config" | 08-crawler-config-complete.md |
| "Seeder list management + multi-domain + sitemaps" | 09-seeder-list-management.md |
Files in this wiki
01-overview-and-architecture.md— Core concepts, data flow, when to use02-configuration-reference.md— Full top-level config parameter reference03-record-extraction.md— recordExtractor, helpers, data shaping patterns04-url-management.md— startUrls, sitemaps, patterns, deduplication, objectID strategy05-operations-and-api.md— REST API, CLI, monitoring dashboard, scheduling06-l1-implementation-plan.md— L1 Data Fetch specific design: seeder list, tagging, dedup, verification07-schema-contracts.md— Pydantic models (CrawlerRecord + EnrichmentInput), contract tests, pyright gates
Key decision: Algolia Crawler vs Apify
| Dimension | Apify | Algolia Crawler |
|---|---|---|
| Maintenance | Custom actor code | Config-driven |
| Algolia integration | Manual push via API | Native |
| Deduplication | Manual | URL-as-objectID + index safety checks |
| Scheduling | Apify cron | Built-in schedule param |
| Monitoring | Apify dashboard | Crawler dashboard (Inspector, Monitoring, Path Explorer) |
| JavaScript rendering | Puppeteer/Playwright | renderJavaScript option |
| Record count limit | Actor memory | 750 records/page (splitContentIntoRecords for large pages) |
| Cost | Apify credits | Included in Algolia plan |
Decision: Use Algolia Crawler. The Apify module is being replaced because the native crawler gives us built-in dedup, monitoring, and direct index integration with zero middleware.