Dashboard/Algolia-Central/Knowledge/AlgoliaCrawler/06-l1-implementation-plan.md
Algolia-Central

Knowledge/AlgoliaCrawler/06-l1-implementation-plan.md

06 — L1 Data Fetch: Algolia Crawler Implementation Plan

Status: BUILT + DEPLOYED — crawler running as of 2026-04-30 08:10 UTC Layer: L1 Data Fetch (RC3 Phoenix engineering spec)

Mandatory standards (CodingSOPs §7 + TestingSOPs Layer 3): - Pydantic v2 model for every data structure crossing the L1→L2 boundary — no raw dicts - pyright --strict on all L1 modules - Contract test validates CrawlerRecord output satisfies EnrichmentInput schema - See 07-schema-contracts.md for the full contract design

Inputs received and analysed

Input Status Key findings
n8n Firehose v2 ✓ Read URL normalization logic, source_type routing, classification taxonomy
n8n Refinery ✓ Read Chunking strategy (technical 600/150, narrative 1500/300), GPT summary, status flag
Seeder list PDF ✓ Read 18 sections, ~67 unique URLs, single-vendor (all algolia.com properties)
Live Algolia index ✓ Queried 10,615 records, exact field schema, source_type distribution, tag taxonomies

Seeder list analysis

Critical insight: this is NOT a broad web crawl. All 67 seeds are algolia.com properties.

The 6 domains

Domain Seeds Live records Notes
www.algolia.com 47 seeds ~3,200 Main site — blog, customers, resources, about, policies
algolia.com (no www) 7 seeds ~5,475 Docs — the deepest section
support.algolia.com 4 seeds 1,732 Zendesk HC — standard HTML
academy.algolia.com 4 seeds 139 LMS — likely SPA, needs renderJavaScript
stories.algolia.com 1 seed 149 Medium-like — SSR
changelog.algolia.com 1 seed 11 Static page

Section priorities

Section Priority Why
§1 Core product + §2 Docs sub-areas MUST-HAVE Bread-and-butter knowledge — 5,475 doc records in live index
§6 Support KB MUST-HAVE 1,732 support records — agent's primary Q&A source
§3 Customer stories MUST-HAVE 149 customer_story records — proof/social
§4 Blog MUST-HAVE 1,161 blog records — thought leadership
§8 Academy MUST-HAVE Learning content
§10 Changelog MUST-HAVE Release notes
§11–17 GOOD-TO-HAVE GOOD Pricing, use-cases, industries, partners, policies
§18 Fragment quick-links SKIP Hash fragments collapse to customer-hub — 4 duplicates

Fragment URL issue (§18)

These 4 seed URLs contain # fragments:

https://www.algolia.com/customer-hub#documentation
https://www.algolia.com/customer-hub#knowledge-base
https://www.algolia.com/customer-hub#algolia-academy
https://www.algolia.com/customer-hub#changelog

The Algolia Crawler strips # fragments — all 4 normalize to https://www.algolia.com/customer-hub. Only one record produced. Remove these 4 from startUrls and keep just https://www.algolia.com/customer-hub.

Resource center ?type= params

These MUST be kept — they represent distinct page content:

?type=webinar, ?type=report, ?type=ebook, ?type=video, ?type=customer-stories

Do NOT include type in ignoreQueryParams.

All decisions locked

D1: Crawl scope — FULL DOMAIN per seed domain

Since all seeds are algolia.com properties, use full domain crawl for each of the 6 domains. The seeds act as entry points + content anchors; the crawler discovers the rest via links.

This is correct because: - We already have ~9,600 records from RC1/RC2 — full domain worked before - The seeds cover all meaningful subsections — discovery within those domains is safe - discoveryPatterns + exclusionPatterns control depth, not seed count

D2: objectID scheme — SHA256 of normalized URL

objectID = require("crypto").createHash("sha256").update(normalizedUrl).digest("hex").substring(0, 36)

SHA256 over base64: cleaner, collision-proof, stable. Normalized = fragment stripped, tracking params removed.

D3: Field schema — LOCKED to live RC2 index

See 03-record-extraction.md — full schema verified against live index 2026-04-30. L1 sets: objectID, record_type, url, title, description, content, source_type, language_code, is_chunk=false, status="pending", created_at, updated_at L2 fills: summary, product_tag, feature_tag, solution_tag, customer_tag, industry_tag, status→"indexed"

D4: Index name — algolia-central_enterprise_ledger

App ID 0EXRPAXB56. Use indexPrefix: "dev_" for local testing.

D5: renderJavaScript — academy only

Only academy.algolia.com/** likely needs it (SPA/LMS). All other domains are SSR/static.

D6: rateLimit — 10

Single vendor, well-resourced CDN. Start at 10; raise if needed.

D7: maxUrls — 30,000

Current index has ~9,600 ledger records. 30,000 gives room for growth with a safety cap.

D8: schedule — manual initially, then weekly

Run manually during RC3 build phase. Add schedule: "every 7 days" once config is validated.

D9: source_type routing — FROM n8n Firehose (ported to JS)

Exact same logic as the Normalize fields node in Firehose v2. Do not deviate.

D10: content field — markdown preferred

The Refinery chunked on markdown field. The Algolia Crawler can extract markdown-like text. Use the $ Cheerio instance to extract clean text from main content areas, not the full page HTML.

Complete crawler config

See 08-crawler-config-complete.md for the production-ready JavaScript config.

Type safety and testing gates

Per CodingSOPs §7 and TestingSOPs, the following are non-negotiable before L1 ships:

Gate Tool When
CrawlerRecord Pydantic model defined Pydantic v2 Before writing any extractor
EnrichmentInput Pydantic model defined Pydantic v2 In sync with L2 team
pyright --strict clean on L1 module pyright Pre-commit
Contract test: L1 output satisfies L2 input pytest Layer 3 On every commit
Unit tests on extractor logic pytest Layer 1 Before any crawl
Integration test against live crawler API pytest Layer 2 Before full crawl run

See 07-schema-contracts.md for the full schema + contract test design.

Build → test → execute sequence

Phase 0: Config authoring (NOW — unblocked)

  • Write full crawler config (08-crawler-config-complete.md → scripts/crawler/config.js)
  • Write Pydantic models (CrawlerRecord, EnrichmentInput — 07-schema-contracts.md)
  • Write contract tests

Phase 1: Smoke test (URL Tester, no full crawl)

  • Use Crawler dashboard URL Tester on 10 sample URLs across all 6 domains
  • Verify: title extracted, content clean, source_type correct, objectID stable
  • Specific URLs to test (one from each domain):
  • https://algolia.com/doc/guides/sending-and-managing-data/prepare-your-data/
  • https://support.algolia.com/hc/en-us/articles/4406981910289
  • https://www.algolia.com/blog/product/algolia-ai-search-2024/
  • https://www.algolia.com/customers/lacoste/
  • https://academy.algolia.com/courses/
  • https://changelog.algolia.com/

Phase 2: Limited crawl (maxUrls: 500)

  • Run with hard cap — validate success rate, inspect 20 records
  • Compare source_type distribution vs live index
  • Check Data Analysis tab for missing fields

Phase 3: Schema audit

  • Export 100 sample records from Algolia
  • Validate against CrawlerRecord Pydantic model (model_validate() on all 100)
  • All must parse clean — any failure = extractor bug

Phase 4: Full crawl

  • Remove maxUrls cap
  • Monitor: Monitoring tab, success rate > 85%
  • Watch safetyChecks — set maxLostRecordsPercentage: 15 for first run

Phase 5: Deduplication audit

  • Query index: facet on url, check for any URL with count > 1
  • Verify total record count is in expected range (within 20% of ~9,600)

Phase 6: L2 handoff validation

  • Query index for status: pending — should be ~all new records
  • Run L2 Enrichment on 50 pending records manually
  • Verify tags populated, status flips to indexed

Open questions — RESOLVED

Question Answer
How many domains? 6 (all algolia.com properties)
Seeder URL count? ~67 unique (after dedupping §18 fragments)
Any SPA/JS-heavy? academy.algolia.com only
Any auth required? No — all public pages
What ?type= params to keep? All — they represent distinct content
Estimated record count? ~9,600 parent docs + chunks (matches RC2 live)
maxUrls safe cap? 30,000