Dashboard/Algolia-Central/Knowledge/AlgoliaCrawler/09-seeder-list-management.md
Algolia-Central

Knowledge/AlgoliaCrawler/09-seeder-list-management.md

09 — Seeder List Management

The problem: hardcoding startUrls in the crawler config doesn't scale. Tomorrow it's adobe.com. The week after it's salesforce.com. Those sites are 50x heavier and change every single day. This file defines the architecture that makes the crawl list live, not frozen.

The core problem

The crawler config we have today hardcodes 67 Algolia URLs as startUrls. This breaks the moment:

  • We add a second domain (adobe.com, salesforce.com)
  • The seeder list grows (new content areas discovered)
  • A target site restructures its URL patterns
  • We want to re-crawl just one domain without touching the rest
  • We want to add a new URL without deploying a config change

Hardcoded startUrls is config-as-code for what should be data.

The right model: config vs. data separation

What BELONGS in crawler config (stable, per-domain rules):
  - discoveryPatterns (which links to follow)
  - exclusionPatterns (what to skip)
  - rateLimit, renderJavaScript, cache (per-domain behaviour)
  - recordExtractor (classification logic)
  - safetyChecks (protection thresholds)

What BELONGS in the seeder list (dynamic, changes often):
  - Specific entry-point URLs to start crawling from
  - Per-URL crawl priority / depth hint
  - Domain-level metadata (owner, crawl frequency, last crawled)

Architecture: Seeder List Manager

seeder-list.json  ←→  Seeder List Manager script
                            ↓
                   Algolia Crawler API
                   (PATCH /config for startUrls
                    OR POST /urls/crawl for incremental)
                            ↓
                   Crawler runs + indexes
                            ↓
                   Inspector API tracks what was crawled

The seeder list as a JSON file

// scripts/crawler/seeder-list.json
{
  "version": "1.0.0",
  "updated_at": "2026-04-30",
  "domains": [
    {
      "id": "algolia-main",
      "domain": "algolia.com",
      "crawler_id": "YOUR_CRAWLER_ID",
      "priority": "must-have",
      "renderJavaScript": false,
      "rateLimit": 10,
      "seeds": [
        { "url": "https://algolia.com/doc", "section": "docs", "depth": "deep" },
        { "url": "https://algolia.com/doc/guides", "section": "docs", "depth": "deep" },
        { "url": "https://www.algolia.com/blog", "section": "blog", "depth": "full" },
        { "url": "https://www.algolia.com/customers", "section": "customer_stories", "depth": "full" },
        { "url": "https://support.algolia.com/hc/en-us", "section": "support", "depth": "full" }
      ]
    },
    {
      "id": "adobe-main",
      "domain": "adobe.com",
      "crawler_id": null,
      "priority": "planned",
      "renderJavaScript": ["https://www.adobe.com/products/**"],
      "rateLimit": 4,
      "seeds": [
        { "url": "https://www.adobe.com/products/experience-manager.html", "section": "product", "depth": "shallow" }
      ]
    }
  ]
}

This file is the single source of truth for what gets crawled. It lives in the repo. Every change is a commit. Git history = crawl history.

Seeder List Manager: the script

# scripts/crawler/manage_seeder.py
"""
Syncs seeder-list.json to the Algolia Crawler API.

Modes:
  sync      — full sync: PATCH crawler config with all seeds as startUrls
  add       — add one URL to a domain's seeds + call POST /urls/crawl immediately
  remove    — remove a URL from seeds (next full sync removes it from config)
  status    — show crawl status for a domain from the Inspector API
  crawl     — trigger an immediate crawl of specific URLs without changing config

Usage:
  uv run scripts/crawler/manage_seeder.py sync --domain algolia-main
  uv run scripts/crawler/manage_seeder.py add --domain algolia-main --url https://algolia.com/doc/new-section
  uv run scripts/crawler/manage_seeder.py status --domain algolia-main
  uv run scripts/crawler/manage_seeder.py crawl --domain algolia-main --urls-file batch.txt
"""
from __future__ import annotations
import argparse
import base64
import json
import logging
import os
import sys
from pathlib import Path
from typing import Any

import httpx
from pydantic import BaseModel, Field

logger = logging.getLogger(__name__)

SEEDER_FILE = Path(__file__).parent / "seeder-list.json"
CRAWLER_BASE = "https://crawler.algolia.com/api/1"


# ── Pydantic models ────────────────────────────────────────────────────────────

class SeedEntry(BaseModel):
    url: str
    section: str
    depth: str = "full"


class DomainConfig(BaseModel):
    id: str
    domain: str
    crawler_id: str | None
    priority: str
    render_java_script: bool | list[str] = Field(False, alias="renderJavaScript")
    rate_limit: int = Field(10, alias="rateLimit")
    seeds: list[SeedEntry]

    model_config = {"populate_by_name": True}


class SeederList(BaseModel):
    version: str
    updated_at: str
    domains: list[DomainConfig]

    def get_domain(self, domain_id: str) -> DomainConfig:
        """Fail loudly if domain not found — no silent noop."""
        for d in self.domains:
            if d.id == domain_id:
                return d
        raise ValueError(f"Domain '{domain_id}' not found in seeder list")


# ── API client ─────────────────────────────────────────────────────────────────

class CrawlerClient:
    """Thin wrapper around Algolia Crawler REST API."""

    def __init__(self, user_id: str, api_key: str) -> None:
        credentials = base64.b64encode(f"{user_id}:{api_key}".encode()).decode()
        self._client = httpx.Client(
            headers={"Authorization": f"Basic {credentials}",
                     "Content-Type": "application/json"},
            timeout=30
        )

    def get_config(self, crawler_id: str) -> dict[str, Any]:
        """Fetch current crawler config."""
        resp = self._client.get(f"{CRAWLER_BASE}/crawlers/{crawler_id}?withConfig=true")
        resp.raise_for_status()
        return resp.json()

    def patch_config(self, crawler_id: str, patch: dict[str, Any]) -> None:
        """Partial-update crawler config — only the keys in patch are changed."""
        resp = self._client.patch(
            f"{CRAWLER_BASE}/crawlers/{crawler_id}/config",
            json=patch
        )
        resp.raise_for_status()
        logger.info("patch_config | crawler=%s | keys=%s", crawler_id, list(patch.keys()))

    def crawl_urls(self, crawler_id: str, urls: list[str], save: bool = False) -> str:
        """Trigger immediate crawl of specific URLs. Returns taskId."""
        resp = self._client.post(
            f"{CRAWLER_BASE}/crawlers/{crawler_id}/urls/crawl",
            json={"urls": urls, "save": save}
        )
        resp.raise_for_status()
        task_id: str = resp.json()["taskId"]
        logger.info("crawl_urls | crawler=%s | count=%d | taskId=%s", crawler_id, len(urls), task_id)
        return task_id

    def start_reindex(self, crawler_id: str) -> str:
        """Start a full crawl. Returns taskId."""
        resp = self._client.post(f"{CRAWLER_BASE}/crawlers/{crawler_id}/reindex")
        resp.raise_for_status()
        return resp.json()["taskId"]


# ── Commands ───────────────────────────────────────────────────────────────────

def cmd_sync(client: CrawlerClient, domain: DomainConfig) -> None:
    """Replace startUrls in crawler config with current seeder list for this domain."""
    if not domain.crawler_id:
        raise ValueError(f"Domain '{domain.id}' has no crawler_id — create the crawler first")

    start_urls = [s.url for s in domain.seeds]
    client.patch_config(domain.crawler_id, {"startUrls": start_urls})
    logger.info("sync | domain=%s | seeds=%d urls synced to startUrls", domain.id, len(start_urls))


def cmd_add(client: CrawlerClient, domain: DomainConfig,
            seeder_list: SeederList, url: str, section: str = "other") -> None:
    """Add a URL to the seeder list + trigger immediate crawl."""
    if not domain.crawler_id:
        raise ValueError(f"Domain '{domain.id}' has no crawler_id")

    # Add to in-memory list
    domain.seeds.append(SeedEntry(url=url, section=section))

    # Persist to file
    seeder_list_data = json.loads(SEEDER_FILE.read_text())
    for d in seeder_list_data["domains"]:
        if d["id"] == domain.id:
            d["seeds"].append({"url": url, "section": section, "depth": "full"})
    SEEDER_FILE.write_text(json.dumps(seeder_list_data, indent=2))
    logger.info("add | domain=%s | url=%s | written to seeder-list.json", domain.id, url)

    # Sync config
    cmd_sync(client, domain)

    # Also trigger immediate crawl so it doesn't wait for next scheduled run
    # Rate limit: 500 POST /urls/crawl requests per 24 hours
    task_id = client.crawl_urls(domain.crawler_id, [url], save=False)
    logger.info("add | immediate crawl triggered | taskId=%s", task_id)


def cmd_status(client: CrawlerClient, domain: DomainConfig) -> None:
    """Print current crawler status for a domain."""
    if not domain.crawler_id:
        logger.info("status | domain=%s | no crawler_id — not yet provisioned", domain.id)
        return
    info = client.get_config(domain.crawler_id)
    print(json.dumps({
        "domain": domain.id,
        "running": info.get("running"),
        "reindexing": info.get("reindexing"),
        "blocked": info.get("blocked"),
        "blockingError": info.get("blockingError"),
        "lastReindexStartAt": info.get("lastReindexStartAt"),
        "lastReindexEndedAt": info.get("lastReindexEndedAt"),
    }, indent=2))


def main() -> None:
    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")

    parser = argparse.ArgumentParser(description="Manage Algolia Crawler seeder list")
    parser.add_argument("command", choices=["sync", "add", "status", "crawl"])
    parser.add_argument("--domain", required=True)
    parser.add_argument("--url", help="URL to add (for 'add' command)")
    parser.add_argument("--section", default="other", help="Section tag for 'add' command")
    args = parser.parse_args()

    user_id = os.environ.get("ALGOLIA_CRAWLER_USER_ID")
    api_key = os.environ.get("ALGOLIA_CRAWLER_API_KEY")
    if not user_id or not api_key:
        sys.exit("ALGOLIA_CRAWLER_USER_ID and ALGOLIA_CRAWLER_API_KEY must be set")

    seeder_list = SeederList.model_validate_json(SEEDER_FILE.read_text())
    domain = seeder_list.get_domain(args.domain)
    client = CrawlerClient(user_id, api_key)

    if args.command == "sync":
        cmd_sync(client, domain)
    elif args.command == "add":
        if not args.url:
            sys.exit("--url required for 'add' command")
        cmd_add(client, domain, seeder_list, args.url, args.section)
    elif args.command == "status":
        cmd_status(client, domain)
    else:
        sys.exit(f"Unknown command: {args.command}")


if __name__ == "__main__":
    main()

Per-domain crawlers, not one monolithic config

For adobe.com or salesforce.com, the approach is one Algolia Crawler instance per domain — not one crawler trying to handle everything. Reasons:

  1. Different rate limits — algolia.com can handle rateLimit 10; adobe.com needs rateLimit 2-4 or they'll block you
  2. Different renderJavaScript rules — adobe.com product pages are SPA-heavy; salesforce.com docs are SSR
  3. Independent scheduling — algolia.com weekly refresh; breaking news sections might need daily
  4. Independent safety thresholds — a 10% record loss on a 500-page site is different from a 10% loss on a 50,000-page site
  5. Independent monitoring — one blocked crawler doesn't stop the others
// seeder-list.json — multi-domain setup
{
  "domains": [
    {
      "id": "algolia-main",
      "crawler_id": "algolia-crawler-uuid",
      "rateLimit": 10,
      "seeds": [...]
    },
    {
      "id": "adobe-experience-cloud",
      "crawler_id": null,  // to be provisioned
      "rateLimit": 3,
      "renderJavaScript": ["https://www.adobe.com/products/**"],
      "seeds": [
        { "url": "https://www.adobe.com/products/experience-manager.html", "section": "product" },
        { "url": "https://business.adobe.com/products/", "section": "product" }
      ]
    },
    {
      "id": "salesforce-help",
      "crawler_id": null,
      "rateLimit": 4,
      "seeds": [
        { "url": "https://help.salesforce.com/s/", "section": "support" },
        { "url": "https://trailhead.salesforce.com/", "section": "academy" }
      ]
    }
  ]
}

Handling sites that change every day (adobe, salesforce scale)

For large sites that change constantly, startUrls + link-following is not enough. Use sitemaps.

{
  "id": "adobe-experience-cloud",
  "crawler_id": "...",
  "sitemaps": [
    "https://www.adobe.com/sitemap.xml",
    "https://business.adobe.com/sitemap.xml"
  ],
  "seeds": [
    // Seeds as fallback/anchors only — sitemaps drive discovery
    { "url": "https://www.adobe.com/", "section": "home", "depth": "shallow" }
  ]
}

With sitemaps: - The site's own team maintains what's discoverable - New pages added to the sitemap get picked up on next scheduled crawl - Removed pages get flagged by safetyChecks - You don't maintain a URL list at all — the sitemap IS the list

When sitemaps aren't available (many enterprise marketing sites don't expose them), fall back to: 1. Discovery via discoveryPatterns — follow all links within the domain 2. RSS feeds as startUrls — blog/news sections often have RSS that lists recent pages 3. The Algolia Crawler extraUrls API for programmatic batch additions

Upkeep workflow: adding new content areas

Adding a new Algolia section:

uv run scripts/crawler/manage_seeder.py add \
  --domain algolia-main \
  --url https://www.algolia.com/about/news \
  --section newsroom

This writes to seeder-list.json, patches the crawler config, and triggers immediate crawl. One command, commit the JSON change.

Adding a new domain: 1. Add domain entry to seeder-list.json with crawler_id: null 2. Provision a new Algolia Crawler instance via the dashboard 3. Write a domain-specific config for it (separate file, or extend 08-crawler-config-complete.md) 4. Update seeder-list.json with the new crawler_id 5. Run sync: uv run scripts/crawler/manage_seeder.py sync --domain adobe-main

Removing a stale URL: 1. Remove from seeder-list.json 2. Run sync to update startUrls 3. The record stays in Algolia until you delete it explicitly (use Algolia Admin API or the delete-by-filter endpoint)

What to build (in order)

  1. seeder-list.json — authoritative file for all domains and seeds (RC3 phase: algolia-main only)
  2. scripts/crawler/manage_seeder.py — Pydantic-validated, pyright-clean manager script
  3. Unit tests for SeederList model validation, cmd_sync, cmd_add
  4. Integration test: cmd_status against real crawler API
  5. Per-domain crawler configs (separate from the main config in 08-crawler-config-complete.md)
  6. Sitemaps field populated as we add each new domain

What the current config in 08-crawler-config-complete.md becomes

The hardcoded startUrls: [67 URLs] gets replaced with:

startUrls: [],  // populated by manage_seeder.py sync — not hardcoded

The config becomes domain logic only (patterns, extraction, classification). The URLs live in seeder-list.json and are pushed to the crawler by the management script.