02 — Algolia Crawler: Configuration Reference

Top-level configuration object

{
  // --- URL Discovery ---
  startUrls: string[],               // Required: entry point URLs
  sitemaps: string[],                // Optional: XML sitemap URLs
  discoveryPatterns: string[],       // Optional: URL patterns to follow but NOT extract from
  extraUrls: string[],               // Optional: additional starting URLs (API-managed)
  exclusionPatterns: string[],       // Optional: URL patterns to skip entirely
  ignoreQueryParams: string[],       // Optional: query param names to strip before dedup

  // --- Crawl Behaviour ---
  rateLimit: number,                 // Required: max concurrent tasks/second
  maxUrls: number,                   // Optional: cap on pages checked
  schedule: string,                  // Optional: e.g. "every 1 day"
  renderJavaScript: boolean | string[] | object,  // Optional: headless browser (default false)
  cache: boolean,                    // Optional: incremental crawl (default true)
  ignoreCanonicalTo: boolean,        // Optional: don't follow canonical redirects
  ignoreNoFollowTo: boolean,         // Optional: follow nofollow links anyway

  // --- Safety ---
  safetyChecks: {
    maxLostRecordsPercentage: number, // Default 10 — abort if records drop > N%
    maxFailedUrls: number            // Abort if failed URL count exceeds this
  },

  // --- Index Routing ---
  indexPrefix: string,               // Optional: prepended to all action.indexName values

  // --- Extraction ---
  actions: Action[],                 // Required: one or more action definitions
}

Parameter details

startUrls (required)

startUrls: ["https://example.com/", "https://example.com/blog/"]

Entry points. The crawler visits these first, then follows links found on them. For our seeder-list use case, every URL in the seeder list becomes a startUrl.

sitemaps

sitemaps: ["https://example.com/sitemap.xml"]

XML sitemaps to discover pages. The crawler parses <loc> elements and queues those URLs. Use alongside startUrls for comprehensive coverage.

discoveryPatterns

discoveryPatterns: ["https://example.com/**"]

Patterns matched using micromatch. The crawler follows links to these URLs to discover content pages, but does not extract records from them (they're navigation/hub pages). Use when you need to traverse listing/category pages to reach actual content.

exclusionPatterns

exclusionPatterns: [
  "**/login/**",
  "**/cart/**",
  "**?sessionid=**"
]

URLs matching these patterns are never crawled. Always exclude auth flows, session-param URLs, and admin paths.

ignoreQueryParams

ignoreQueryParams: ["ref", "utm_*", "fbclid", "gclid"]

Strips these query params before URL deduplication. Without this, page.html?utm_source=A and page.html?utm_source=B are indexed as two separate pages. Use * suffix for wildcard families.

rateLimit

rateLimit: 8

Max concurrent URL tasks. Formula: MAX(urls_added_last_second, urls_currently_processing) <= rateLimit. Start at 4–8 for new crawlers. High values increase pressure on target servers.

renderJavaScript

// Simple: render all pages
renderJavaScript: true

// Targeted: only dynamic paths
renderJavaScript: ["https://example.com/dynamic/**"]

// Advanced
renderJavaScript: {
  enabled: true,
  patterns: ["https://example.com/spa/**"],
  waitTime: { min: 500, max: 10000 },
  adblock: false
}

Uses headless Chromium. Default timeout 20s. Enable only for pages that require JS rendering — it's slower and can produce inconsistent records. For our seeder list, most pages will be HTML — enable only for SPA/dynamic pages.

schedule

schedule: "every 1 day"    // daily
schedule: "every 7 days"   // weekly

Sets automatic recurring crawls. Set on the crawler-level config or per-action.

cache (default: true)

When true, only re-crawls pages that have changed since the last crawl (based on HTTP modified headers). Keeps index fresh with minimal re-processing on incremental runs.

safetyChecks

safetyChecks: {
  maxLostRecordsPercentage: 15,  // abort if 15%+ records disappear
  maxFailedUrls: 100             // abort if 100+ URLs fail
}

Safety net against bad crawls that would wipe the index. Default maxLostRecordsPercentage is 10. For initial crawls where the index is empty, the crawler skips this check automatically.

indexPrefix

indexPrefix: "algolia_central_"

Prepended to all action.indexName values. Useful for environment namespacing (dev/staging/prod).

Action object

actions: [
  {
    indexName: "enterprise_ledger",    // Required
    pathsToMatch: ["https://example.com/**"],  // Required
    recordExtractor: ({ url, $, helpers, dataSources }) => {
      // ... return records array
    },
    name: "main-content",              // Required if schedule set
    schedule: "every 1 day",           // Optional per-action schedule
    selectorsToMatch: [".article-body"], // Optional: only index pages with this selector
    fileTypesToMatch: ["html"],         // Default ["html"]; also pdf, doc, xls, etc.
    autoGenerateObjectIDs: false,       // Set false when WE control objectID
  }
]

Multiple actions can target different index names or apply different extractors to different URL patterns. Process by specificity — most specific pathsToMatch first.