Algolia-Central

Knowledge/AlgoliaCrawler/04-url-management.md

04 — URL Management: Seeder Lists, Patterns, Deduplication

The three URL entry points

startUrls       → Direct entry: crawler visits these first
sitemaps        → Parsed for <loc> elements, each queued
extraUrls       → API-managed: added programmatically via Crawl URLs endpoint

For our seeder-list architecture, every URL from the seeder list becomes a startUrl or extraUrl.

Seeder list integration strategy

We have a custom GPT-generated seeder list used previously with Apify. Migration path:

Option A: Bulk startUrls (recommended for initial crawl)

startUrls: [
  "https://company1.com",
  "https://company2.com",
  // ... all seeder list entries
]

Embed the full seeder list in the config. Simple and transparent. Rerun = update config.

Option B: extraUrls via API (recommended for ongoing additions)

Use POST /1/crawlers/{id}/urls/crawl to add new URLs discovered after initial setup:

{
  "urls": ["https://newcompany.com"],
  "save": true
}

save: true persists the URL to the crawler's extraUrls config so it's included in future crawls.

Rate limit: 500 crawl-url API requests per 24 hours.

Recommended hybrid approach

Initial seeder list → startUrls in config
New additions from discovery → extraUrls via API
Maintain a canonical seeder list file in the repo, synced to config

URL pattern matching: micromatch syntax

The pathsToMatch, discoveryPatterns, and exclusionPatterns all use micromatch patterns.

// Match all pages on a domain
"https://example.com/**"

// Match specific paths
"https://example.com/blog/**"
"https://example.com/products/**"

// Negate (exclude within a match context)
"!https://example.com/blog/drafts/**"

// Multiple domains
["https://company1.com/**", "https://company2.com/**"]

pathsToMatch vs discoveryPatterns

Config	Crawler visits	Crawler extracts records
`startUrls`	Yes	No (just enqueues)
`discoveryPatterns`	Yes (follows links)	No
`pathsToMatch` (in action)	Yes	Yes

A URL only produces records if it matches pathsToMatch in at least one action.

URL deduplication: two layers

Layer 1 — objectID-based (indexing time)

saveObjects with a stable objectID → re-crawl of same URL = overwrite, not duplicate.

objectID = deterministic hash of normalized URL

URL normalization happens before hashing: 1. Remove query params listed in ignoreQueryParams 2. Lowercase hostname 3. Trailing slash normalisation

Layer 2 — Distinct (query time)

Index settings:

{
  "distinct": true,
  "attributeForDistinct": "url"
}

Even if somehow two records with different objectIDs share the same URL, distinct collapses them at search time.

Layer 3 — Canonical tag filtering

In recordExtractor, skip pages where canonical URL ≠ current URL:

const canonical = $('link[rel="canonical"]').attr("href")
if (canonical && canonical !== url.href && !url.href.startsWith(canonical)) return []

This prevents indexing paginated duplicates, syndicated copies, and parameter variants.

ignoreQueryParams — critical for clean dedup

Without this, the crawler treats these as different pages: - page.html?utm_source=email - page.html?utm_source=twitter - page.html?fbclid=ABC - page.html?ref=sidebar

Configuration:

ignoreQueryParams: [
  "utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content",
  "fbclid", "gclid", "ref", "referrer",
  "sessionid", "PHPSESSID", "_ga"
]

Keep meaningful params: page, category, id, q — these represent distinct content.

Exclusion patterns: what to always exclude

exclusionPatterns: [
  // Auth and admin
  "**/login/**", "**/logout/**", "**/admin/**", "**/wp-admin/**",
  // Session/token URLs
  "**?token=**", "**?auth=**", "**?session=**",
  // CDN and assets
  "**/*.css", "**/*.js", "**/*.png", "**/*.jpg", "**/*.gif",
  "**/*.pdf",   // unless you want PDF indexing
  // Pagination beyond a limit
  "**?page=[2-9][0-9]**", "**?p=[5-9][0-9]**",
  // Search result pages (don't index a search engine's search results)
  "**/search?**", "**/results?**",
  // Feed and API endpoints
  "**/feed/**", "**/api/**", "**/rss/**"
]

Crawl depth management

The crawler follows links found on visited pages. If discoveryPatterns is broad, crawl scope can explode. Control strategies:

maxUrls: hard cap — maxUrls: 5000 stops at 5000 URLs checked
Narrow discoveryPatterns: only follow links within specific path prefixes
Restrictive pathsToMatch: even if the crawler visits a URL, it only indexes if pathsToMatch matches
exclusionPatterns: aggressively exclude irrelevant subtrees

For seeder-list crawls where we want shallow crawl (just the seeder page + 1 level down), use:

discoveryPatterns: seederListUrls.map(u => `${u}/**`),
// Only follow links within the seeder domain, not the whole internet
exclusionPatterns: ["**/page/[2-9]**", "**/page/[1-9][0-9]**"]

Record tracking and roster management

To prevent re-crawling already-complete URLs in a new crawl session:

The crawler's cache (cache: true) handles this by default — only changed pages are re-processed
For explicit URL roster management, use the Inspector tab — export crawled URL list, diff against seeder list
For programmatic dedup check: query Algolia for url field before crawling a batch