05 — Operations: REST API, CLI, Monitoring, Scheduling

Crawler REST API

Base URL: https://crawler.algolia.com/api Auth: Authorization: Basic base64(<userId>:<apiKey>) — crawler-specific credentials, NOT standard Algolia API key.

Endpoints

Method	Path	Purpose
GET	/1/crawlers/{id}	Get crawler details (add ?withConfig=true for full config)
POST	/1/crawlers/{id}/reindex	Start or resume a full crawl
POST	/1/crawlers/{id}/urls/crawl	Crawl specific URLs immediately
PATCH	/1/crawlers/{id}/config	Update crawler configuration
GET	/1/crawlers/{id}/tasks/{taskId}	Check task status

Start a crawl

curl -X POST "https://crawler.algolia.com/api/1/crawlers/{crawlerId}/reindex" \
  -H "Authorization: Basic $(echo -n 'userId:apiKey' | base64)"

Response: { "taskId": "uuid" }

Crawl specific URLs (incremental)

curl -X POST "https://crawler.algolia.com/api/1/crawlers/{id}/urls/crawl" \
  -H "Authorization: Basic ..." \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://example.com/new-page"], "save": true }'

• save: true → adds URL to extraUrls config for future crawls
• Rate limit: 500 requests / 24 hours

Get crawler status

curl "https://crawler.algolia.com/api/1/crawlers/{id}?withConfig=true" \
  -H "Authorization: Basic ..."

Response includes:

{
  "running": true,
  "reindexing": true,
  "blocked": false,
  "lastReindexStartAt": "2026-04-30T10:00:00Z",
  "lastReindexEndedAt": null,
  "config": { ... }
}

Update configuration

curl -X PATCH "https://crawler.algolia.com/api/1/crawlers/{id}/config" \
  -H "Authorization: Basic ..." \
  -H "Content-Type: application/json" \
  -d '{ "startUrls": ["https://example.com/new-section"] }'

---

Algolia CLI (crawler commands)

# List crawlers
algolia crawler list

# Start a reindex
algolia crawler reindex --crawler-id {id}

# Get crawler status
algolia crawler get --crawler-id {id}

CLI credential setup: requires ALGOLIA_CRAWLER_USER_ID and ALGOLIA_CRAWLER_API_KEY env vars (or config file).

---

Scheduling

Config-level schedule (entire crawler)

{
  schedule: "every 1 day"     // daily
  schedule: "every 7 days"    // weekly  
  schedule: "every 30 days"   // monthly
}

Action-level schedule (per action)

actions: [{
  name: "blog-content",    // required if schedule is set on action
  schedule: "every 1 day",
  // ...
}]

Schedules are relative to the last successful crawl completion, not a fixed clock time.

For our L1 use case: initially manual/API-triggered crawls during buildout. Add schedule once the config is validated.

---

Monitoring dashboard — operational use

Access: https://crawler.algolia.com/admin/crawlers/{id}

Monitoring tab

• Current crawl status (running / completed / failed)
• URL counts: total queued, processed, succeeded, ignored, failed
• Bandwidth used
• Record count before/after crawl

Inspector tab

• Search for any URL → see its crawl result
• Per-URL: processing time, links in/out, extracted records preview
• Filter by status: success / ignored / failed
• Use for: debugging why a specific page didn't get indexed

URL Tester

• Test your recordExtractor against a single URL without running a full crawl
• Output tabs: All, HTTP, Logs, Errors, Records, Links, External Data, HTML
• Use for: iterating on extractor code before committing

Path Explorer

• Tree view of crawled paths
• Per path: URL count, bandwidth, record count, error count
• Use for: identifying path prefixes generating errors or low record yield

Data Analysis

• Cross-record consistency report
• Flags: missing attributes, type mismatches, empty arrays, suspicious structures
• Use for: validating schema consistency after a crawl

---

Operational runbook: first crawl

1. Write crawler config (startUrls from seeder list, recordExtractor, exclusionPatterns)
2. Use URL Tester in dashboard to validate extractor on 5–10 sample URLs
3. Start a limited crawl: maxUrls: 100 for smoke test
4. Check Monitoring tab: review success/fail counts
5. Check Inspector tab: inspect 5–10 crawled records, verify fields
6. Check Data Analysis: any missing attributes?
7. Fix extractor issues, re-test
8. Remove maxUrls cap, run full crawl
9. Verify record count in Algolia index matches expected
10. Run a test query against the index to validate search quality
11. Enable schedule for recurring crawls

---

Operational runbook: incremental additions

When new URLs added to seeder list:

1. Add URLs to crawler via API: POST /urls/crawl with save: true
2. Monitor task completion via GET /tasks/{taskId}
3. Inspect new records in Inspector
4. Verify no duplicates: query Algolia for url field

---

Safety checks — what triggers an abort

safetyChecks: {
  maxLostRecordsPercentage: 10,  // default
  maxFailedUrls: 50
}

If a crawl would delete > 10% of existing records, the crawler aborts before committing the new index. This protects against: - Config mistakes that exclude previously indexed content - Network issues causing widespread fetch failures - JavaScript rendering timeouts causing mass empty extractions

First crawl exception: safety check is skipped when the index is empty (nothing to lose).

When a crawl aborts due to safety check: 1. Check blockingError field via GET crawler API 2. Inspect pathsToMatch — did the exclusion patterns widen? 3. Check maxFailedUrls — is the target site down? 4. Raise maxLostRecordsPercentage temporarily if the record drop is intentional (site restructure)