08 — Complete Crawler Config

Derived from: seeder list PDF, n8n Firehose v2 logic, live RC2 index schema (verified 2026-04-30) Target: algolia-central_enterprise_ledger (app 0EXRPAXB56) Ready to paste into Algolia Crawler dashboard editor

Architecture note — one pass, not two

The n8n Firehose + Refinery two-step is gone. That split existed because Supabase needed a queue (status: pending) between raw doc storage and vector generation. Algolia Crawler replaces both:

• Firehose (clean + classify + upsert) → recordExtractor does all of this in one function
• Refinery (GPT summary + chunk + OpenAI embed) → not needed: Algolia NeuralSearch handles vectors internally. The index already runs in "mode": "neuralSearch" — Algolia's ML creates its own vectors from your text records. No chunking, no OpenAI embeddings.

The summary field that exists in the RC2 live index came from a one-time migration enrichment. For RC3 we skip it initially; if AEs need summaries surfaced in the UI, add an optional post-crawl LLM enrichment script separately — not as part of the crawler.

Deduplication is automatic: same URL → same SHA256 objectID → saveObjects overwrites existing record. cache: true skips unchanged pages on re-crawls entirely. No hash-check-then-upsert logic needed.

Source_type routing function

Ported verbatim from n8n Firehose v2 Normalize fields node. Must stay in sync with that node if ever updated.

function getSourceType(rawUrl) {
  const lower = (rawUrl || '').toLowerCase()
  if (lower.includes('support.algolia.com/hc')) return 'support'
  if (lower.includes('academy.algolia.com'))     return 'academy'
  if (lower.includes('changelog.algolia.com'))   return 'changelog'
  if (lower.includes('stories.algolia.com'))     return 'customer_story'
  if (/\/doc(?:\/|$)/.test(lower) || /\/docs(?:\/|$)/.test(lower)) return 'doc'
  if (/\/developers(?:\/|$)/.test(lower))        return 'developer'
  if (/\/customers(?:\/|$)/.test(lower) || lower.includes('resources?type=customer-stories')) return 'customer_story'
  if (/\/blog(?:\/|$)/.test(lower))              return 'blog'
  if (/\/resources(?:\/|$)/.test(lower))         return 'resource'
  return 'other'
}

Complete config

// Algolia Crawler config — RC3 Phoenix L1 Data Fetch
// Source: seeder list PDF + n8n Firehose v2 + live index schema
// App: 0EXRPAXB56 | Index: algolia-central_enterprise_ledger

new Crawler({
  appId: "0EXRPAXB56",
  apiKey: "YOUR_WRITE_API_KEY",  // use Admin or Crawler key — NOT the search key from .env

  // ─── URL DISCOVERY ─────────────────────────────────────────────────────────

  startUrls: [
    // §1 Core product & docs
    "https://www.algolia.com/",
    "https://www.algolia.com/products",
    "https://www.algolia.com/use-cases",
    "https://www.algolia.com/industries",
    "https://algolia.com/doc",
    "https://www.algolia.com/developers",
    // §2 Doc sub-areas (deep crawl entry points)
    "https://algolia.com/doc/guides",
    "https://algolia.com/doc/api-client",
    "https://algolia.com/doc/rest-api",
    "https://algolia.com/doc/integration",
    "https://algolia.com/doc/tools",
    "https://algolia.com/doc/resources",
    // §3 Customer stories
    "https://www.algolia.com/customers",
    "https://www.algolia.com/resources?type=customer-stories",
    "https://stories.algolia.com/tagged/customer-stories",
    // §4 Blog
    "https://www.algolia.com/blog",
    "https://www.algolia.com/blog/product",
    "https://www.algolia.com/blog/ai",
    "https://www.algolia.com/blog/ecommerce",
    "https://www.algolia.com/blog/engineering",
    "https://www.algolia.com/blog/ux",
    // §5 Newsroom
    "https://www.algolia.com/about/news",
    // §6 Support KB
    "https://support.algolia.com/hc/en-us",
    "https://support.algolia.com/hc/en-us/categories",
    "https://support.algolia.com/hc/en-us/sections",
    "https://support.algolia.com/hc/en-us/articles",
    // §7 Customer Hub (NOTE: §18 fragment variants are duplicates of this URL — omitted)
    "https://www.algolia.com/customer-hub",
    "https://www.algolia.com/customers/whats-new",
    "https://www.algolia.com/customers/knowledge-base",
    // §8 Academy
    "https://academy.algolia.com/",
    "https://academy.algolia.com/collections",
    "https://academy.algolia.com/path",
    "https://academy.algolia.com/courses",
    // §9 Resources
    "https://www.algolia.com/resources",
    "https://www.algolia.com/resources?type=webinar",
    "https://www.algolia.com/resources?type=report",
    "https://www.algolia.com/resources?type=ebook",
    "https://www.algolia.com/resources?type=video",
    // §10 Changelog
    "https://changelog.algolia.com/",
    // §11 Pricing
    "https://www.algolia.com/pricing",
    // §12 Use-case solution pages
    "https://www.algolia.com/use-cases/enterprise",
    "https://www.algolia.com/use-cases/headless-commerce",
    "https://www.algolia.com/use-cases/site-search",
    "https://www.algolia.com/use-cases/mobile-app-search",
    "https://www.algolia.com/use-cases/image-search",
    "https://www.algolia.com/use-cases/retail-media-network",
    "https://www.algolia.com/use-cases/visual-search",
    "https://www.algolia.com/use-cases/voice-search",
    // §13 Industry pages
    "https://www.algolia.com/industries/ecommerce",
    "https://www.algolia.com/industries/b2b-commerce",
    "https://www.algolia.com/industries/marketplaces",
    "https://www.algolia.com/industries/saas",
    "https://www.algolia.com/industries/media",
    "https://www.algolia.com/industries/fashion",
    "https://www.algolia.com/industries/higher-education",
    "https://www.algolia.com/industries/grocery",
    // §14 Developer Hub
    "https://www.algolia.com/developers/search-api",
    "https://www.algolia.com/developers/code-exchange",
    "https://www.algolia.com/developers/web-instantsearch-vue",
    "https://www.algolia.com/developers/integrations",
    "https://www.algolia.com/developers/ui-components",
    "https://www.algolia.com/developers/autocomplete",
    // §15 Policies
    "https://www.algolia.com/policies/support",
    "https://www.algolia.com/policies",
    "https://www.algolia.com/policies/privacy",
    "https://www.algolia.com/policies/terms",
    // §16 Company & partners
    "https://www.algolia.com/about",
    "https://www.algolia.com/about/leadership",
    "https://www.algolia.com/about/events",
    "https://www.algolia.com/about/awards",
    "https://www.algolia.com/partners",
    "https://www.algolia.com/partners/technology",
    "https://www.algolia.com/partners/solution",
    // §17 OSS
    "https://www.algolia.com/for-open-source",
  ],

  sitemaps: [
    // Add algolia.com sitemap if available — supplements link discovery
    // "https://www.algolia.com/sitemap.xml",
  ],

  // Follow links within all 6 algolia.com domains but don't extract from navigation-only pages
  discoveryPatterns: [
    "https://www.algolia.com/**",
    "https://algolia.com/**",
    "https://support.algolia.com/**",
    "https://academy.algolia.com/**",
    "https://stories.algolia.com/**",
    "https://changelog.algolia.com/**",
  ],

  // ─── URL NORMALISATION ────────────────────────────────────────────────────

  // Strip tracking params but KEEP ?type= (resources filter), ?page= (pagination we control)
  ignoreQueryParams: [
    "utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content",
    "fbclid", "gclid", "ref", "referrer", "_ga", "PHPSESSID",
    "mc_cid", "mc_eid",   // Mailchimp
    "hsCtaTracking",      // HubSpot
    "yclid",              // Yandex
  ],

  // ─── EXCLUSIONS ───────────────────────────────────────────────────────────

  exclusionPatterns: [
    // Auth / account flows
    "**/login**", "**/logout**", "**/signup**", "**/register**",
    "**/account/**", "**/dashboard/**",
    // Admin
    "**/admin/**", "**/wp-admin/**",
    // Zendesk auth and internal
    "**/hc/*/signin**", "**/hc/*/requests**", "**/hc/*/activity**",
    "**/hc/*/contributions**", "**/hc/*/community/**",
    // Asset files
    "**/*.css", "**/*.js", "**/*.png", "**/*.jpg", "**/*.jpeg",
    "**/*.gif", "**/*.svg", "**/*.ico", "**/*.woff**", "**/*.ttf",
    // API / feed endpoints
    "**/api/**", "**/feed/**", "**/rss/**", "**/sitemap**",
    // Algolia-specific non-content paths
    "**/search?**",
    "https://www.algolia.com/developers/documentation/**",  // redirects to /doc
    // Pagination beyond p.10 (support KB articles can have deep pagination)
    "**?page=1[0-9]**",
    "**?page=[2-9][0-9]**",
    // Fragment-only variants (normalized to parent URL anyway, but explicit is clean)
    // Note: crawler strips # by default; this is belt-and-suspenders
  ],

  // ─── CRAWL BEHAVIOUR ──────────────────────────────────────────────────────

  rateLimit: 10,          // Single vendor (algolia.com CDN) — can handle this
  maxUrls: 30000,         // Safety cap; RC2 had ~9,600 records — 30k gives room
  cache: true,            // Incremental: only re-crawl changed pages on re-runs
  schedule: "every 7 days",  // Comment out during Phase 1 smoke test

  // JavaScript rendering — only academy (SPA/LMS)
  renderJavaScript: {
    enabled: true,
    patterns: ["https://academy.algolia.com/**"],
    waitTime: { min: 500, max: 15000 },
    adblock: false,
  },

  safetyChecks: {
    maxLostRecordsPercentage: 15,  // Slightly higher for initial RC3 run
    maxFailedUrls: 300,
  },

  // ─── EXTRACTION ───────────────────────────────────────────────────────────

  actions: [
    {
      indexName: "algolia-central_enterprise_ledger",
      pathsToMatch: [
        "https://www.algolia.com/**",
        "https://algolia.com/**",
        "https://support.algolia.com/**",
        "https://academy.algolia.com/**",
        "https://stories.algolia.com/**",
        "https://changelog.algolia.com/**",
      ],
      autoGenerateObjectIDs: false,  // We own objectID — dedup guarantee

      recordExtractor: ({ url, $ }) => {
        const rawUrl = url.href
        const urlLower = rawUrl.toLowerCase()
        const pathLower = url.pathname.toLowerCase()
        const hostLower = url.hostname.toLowerCase()

        // ────────────────────────────────────────────────────────────────────
        // STEP 1: CLEAN — strip chrome, normalize whitespace
        // (replicates n8n Clean Data node)
        // ────────────────────────────────────────────────────────────────────
        $("nav, header, footer, aside, .sidebar, .nav, .navigation, " +
          ".cookie-banner, .cookie-consent, .announcement-bar, " +
          "script, style, noscript, iframe").remove()

        let content = $("article, main, [role='main'], .content, " +
                        ".article-content, .page-content, .prose").text()
        if (!content || content.trim().length < 200) {
          content = $("body").text()
        }
        // Normalize line endings, collapse 3+ blank lines → 2 (same as n8n Clean Data)
        content = content
          .replace(/\r\n/g, "\n")
          .replace(/\n{3,}/g, "\n\n")
          .trim()

        // ────────────────────────────────────────────────────────────────────
        // STEP 2: QUALITY GATES — reject before any indexing
        // (replicates n8n Classify Content + Keep/Reject nodes)
        // ────────────────────────────────────────────────────────────────────
        const title = $("title").text().trim() || $("h1").first().text().trim()
        if (!title) return []                      // no title → skip
        if (content.length < 300) return []        // near-empty → skip

        const titleLower = title.toLowerCase()
        const contentLower = content.toLowerCase()
        if (titleLower.includes("page not found")) return []
        if (contentLower.includes("## page not found")) return []
        if (contentLower.includes("404") && contentLower.includes("not found")) return []

        // Skip canonical duplicates (e.g. paginated variants, param variants)
        const canonical = $('link[rel="canonical"]').attr("href")
        if (canonical && canonical !== rawUrl && !rawUrl.startsWith(canonical + "?")) return []

        // ────────────────────────────────────────────────────────────────────
        // STEP 3: CLASSIFY — source_type, product_tag, feature_tag
        // (replicates n8n Normalize fields + Classify Content nodes)
        // Values must be proper-case strings matching the live RC2 index taxonomy.
        // ────────────────────────────────────────────────────────────────────

        // source_type — URL-host routing (from n8n Normalize fields, verbatim)
        let source_type = "other"
        if (hostLower.includes("support.algolia.com") && pathLower.includes("/hc")) {
          source_type = "support"
        } else if (hostLower.includes("academy.algolia.com")) {
          source_type = "academy"
        } else if (hostLower.includes("changelog.algolia.com")) {
          source_type = "changelog"
        } else if (hostLower.includes("stories.algolia.com")) {
          source_type = "customer_story"
        } else if (/\/doc(?:\/|$)/.test(pathLower) || /\/docs(?:\/|$)/.test(pathLower)) {
          source_type = "doc"
        } else if (/\/developers(?:\/|$)/.test(pathLower)) {
          source_type = "developer"
        } else if (/\/customers(?:\/|$)/.test(pathLower) ||
                   urlLower.includes("resources?type=customer-stories")) {
          source_type = "customer_story"
        } else if (/\/blog(?:\/|$)/.test(pathLower)) {
          source_type = "blog"
        } else if (/\/resources(?:\/|$)/.test(pathLower)) {
          source_type = "resource"
        }

        // product_tag — URL keyword routing (proper case, matching live RC2 index values)
        // Live values: "AI Search", "InstantSearch", "Analytics", "Recommend", "Autocomplete", "DocSearch"
        let product_tag = null
        if (urlLower.includes("neural") || urlLower.includes("ai-search") ||
            urlLower.includes("neuralsearch")) {
          product_tag = "AI Search"
        } else if (urlLower.includes("instantsearch") || urlLower.includes("widget") ||
                   urlLower.includes("component") || urlLower.includes("api-reference/widgets")) {
          product_tag = "InstantSearch"
        } else if (urlLower.includes("analytics") || urlLower.includes("insights")) {
          product_tag = "Analytics"
        } else if (urlLower.includes("recommend")) {
          product_tag = "Recommend"
        } else if (urlLower.includes("autocomplete")) {
          product_tag = "Autocomplete"
        } else if (urlLower.includes("docsearch") || urlLower.includes("doc-search")) {
          product_tag = "DocSearch"
        } else if (urlLower.includes("query-suggestions")) {
          product_tag = "Query Suggestions"
        } else if (urlLower.includes("merchandis")) {
          product_tag = "Merchandising"
        } else if (urlLower.includes("personalization")) {
          product_tag = "Personalization"
        } else if (source_type === "doc" || source_type === "developer" ||
                   source_type === "support") {
          product_tag = "AI Search"  // default for doc/support content
        }

        // feature_tag — URL keyword routing (proper case, matching live RC2 index values)
        // Live values: "Rules", "Faceting", "A/B Testing", "Synonyms", "Query Suggestions", etc.
        let feature_tag = null
        if (urlLower.includes("synonym")) {
          feature_tag = "Synonyms"
        } else if (urlLower.includes("/rules") || urlLower.includes("query-rules")) {
          feature_tag = "Rules"
        } else if (urlLower.includes("facet") || urlLower.includes("/filter")) {
          feature_tag = "Faceting"
        } else if (urlLower.includes("abtest") || urlLower.includes("ab-testing")) {
          feature_tag = "A/B Testing"
        } else if (urlLower.includes("personalization")) {
          feature_tag = "Personalization"
        } else if (urlLower.includes("geo-search") || urlLower.includes("geosearch")) {
          feature_tag = "Geo Search"
        } else if (urlLower.includes("analytics")) {
          feature_tag = "Event Tracking"
        } else if (urlLower.includes("highlight")) {
          feature_tag = "Highlighting"
        } else if (urlLower.includes("pagination") || urlLower.includes("infinite")) {
          feature_tag = "Pagination"
        } else if (urlLower.includes("ranking") || urlLower.includes("relevance")) {
          feature_tag = "Ranking"
        } else if (urlLower.includes("typo")) {
          feature_tag = "Typo Tolerance"
        } else if (urlLower.includes("crawler") || urlLower.includes("crawling")) {
          feature_tag = "Crawler"
        } else if (urlLower.includes("ingestion") || urlLower.includes("indexing")) {
          feature_tag = "Indexing"
        }

        // ── solution_tag ──
        // Live values (by count): Site Search, App Search, Personalization,
        // Headless Commerce, Recommendations, Merchandising, Enterprise Search,
        // Analytics, Voice Search, Mobile Search
        let solution_tag = null
        if (urlLower.includes("headless") || urlLower.includes("headless-commerce")) {
          solution_tag = "Headless Commerce"
        } else if (urlLower.includes("personalization")) {
          solution_tag = "Personalization"
        } else if (urlLower.includes("recommend") && !hostLower.includes("academy")) {
          solution_tag = "Recommendations"
        } else if (urlLower.includes("merchandis")) {
          solution_tag = "Merchandising"
        } else if (urlLower.includes("enterprise") || urlLower.includes("/use-cases/enterprise")) {
          solution_tag = "Enterprise Search"
        } else if (urlLower.includes("analytics")) {
          solution_tag = "Analytics"
        } else if (urlLower.includes("voice-search") || urlLower.includes("voice_search")) {
          solution_tag = "Voice Search"
        } else if (urlLower.includes("mobile-app") || urlLower.includes("mobile_app")) {
          solution_tag = "Mobile Search"
        } else if (urlLower.includes("app-search") || urlLower.includes("/developers/")) {
          solution_tag = "App Search"
        } else if (source_type === "doc" || source_type === "support" ||
                   pathLower.includes("/site-search")) {
          solution_tag = "Site Search"   // default for doc/support content
        }

        // ── industry_tag ──
        // Live values: Ecommerce, Retail, SaaS, B2B Commerce, Media, Marketplace,
        // Healthcare, Finance, Gaming, Education, Fashion, Government, Grocery, Non-profit
        // Primary signal: /industries/{slug} path. Secondary: customer story page content.
        let industry_tag = null
        if (pathLower.includes("/industries/ecommerce")) {
          industry_tag = "Ecommerce"
        } else if (pathLower.includes("/industries/b2b-commerce")) {
          industry_tag = "B2B Commerce"
        } else if (pathLower.includes("/industries/marketplaces")) {
          industry_tag = "Marketplace"
        } else if (pathLower.includes("/industries/saas")) {
          industry_tag = "SaaS"
        } else if (pathLower.includes("/industries/media")) {
          industry_tag = "Media"
        } else if (pathLower.includes("/industries/fashion")) {
          industry_tag = "Fashion"
        } else if (pathLower.includes("/industries/higher-education")) {
          industry_tag = "Education"
        } else if (pathLower.includes("/industries/grocery")) {
          industry_tag = "Grocery"
        } else if (pathLower.includes("/industries/healthcare")) {
          industry_tag = "Healthcare"
        } else if (pathLower.includes("/industries/financial") || pathLower.includes("/industries/finance")) {
          industry_tag = "Finance"
        } else if (source_type === "customer_story") {
          // For customer stories without explicit industry path:
          // attempt content-based detection from page body keywords
          const bodyLower = contentLower
          if (bodyLower.includes("fashion") || bodyLower.includes("apparel") ||
              bodyLower.includes("clothing")) {
            industry_tag = "Fashion"
          } else if (bodyLower.includes("grocery") || bodyLower.includes("food delivery")) {
            industry_tag = "Grocery"
          } else if (bodyLower.includes("media") || bodyLower.includes("publishing") ||
                     bodyLower.includes("news")) {
            industry_tag = "Media"
          } else if (bodyLower.includes("healthcare") || bodyLower.includes("medical")) {
            industry_tag = "Healthcare"
          } else if (bodyLower.includes("marketplace") || bodyLower.includes("classifieds")) {
            industry_tag = "Marketplace"
          } else if (bodyLower.includes("b2b") || bodyLower.includes("wholesale")) {
            industry_tag = "B2B Commerce"
          } else {
            industry_tag = "Ecommerce"   // safe fallback for customer stories
          }
        }

        // ── customer_tag ──
        // Live values: 57 specific customer names (Lacoste, Gymshark, Walgreens, etc.)
        // Signal: /customers/{slug} URL. Slug → title-case name.
        // Note: live index has variants like "Gymshark Headless" vs "Gymshark Recommend"
        // — we extract the slug here; variant detail can be added in optional enrichment.
        let customer_tag = null
        // Handle locale prefix: /customers/slug, /de/customers/slug, /fr/customers/slug
        const customerMatch = url.pathname.match(/^(?:\/[a-z]{2})?\/customers\/([^\/]+)\/?$/)
        if (customerMatch && customerMatch[1] &&
            !["whats-new", "knowledge-base"].includes(customerMatch[1])) {
          // Convert slug to title-case: "gymshark-headless" → "Gymshark Headless"
          customer_tag = customerMatch[1]
            .split("-")
            .map(w => w.charAt(0).toUpperCase() + w.slice(1))
            .join(" ")
        }

        // ── category ──
        // Live values: case_study (952), product (16), doc_section (14), vision (13), capability (5)
        let category = null
        if (source_type === "customer_story") {
          category = "case_study"
        } else if (source_type === "doc" || source_type === "support") {
          category = "doc_section"
        } else if (pathLower.includes("/products") || pathLower.includes("/use-cases") ||
                   pathLower.includes("/industries")) {
          category = "product"
        } else if (pathLower.includes("/about") || pathLower.includes("/about/news") ||
                   pathLower.includes("/about/awards")) {
          category = "vision"
        }

        // ── language_code ──
        const language_code = ($("html").attr("lang") || "en").split("-")[0].toLowerCase()

        // ── description ──
        const description = $('meta[name="description"]').attr("content")
          || $('meta[property="og:description"]').attr("content")
          || ""

        // ────────────────────────────────────────────────────────────────────
        // STEP 4: DEDUPLICATION — deterministic objectID from normalized URL
        // Same URL on every crawl = same objectID = overwrite (never duplicate).
        // URL already normalized by ignoreQueryParams before extractor runs.
        // ────────────────────────────────────────────────────────────────────
        const objectID = rawUrl.split("").reduce((hash, char) => {
          return (((hash << 5) - hash) + char.charCodeAt(0)) | 0
        }, 0).toString(36) + "-" + rawUrl.length.toString(36) + "-" +
          rawUrl.slice(8, 28).replace(/[^a-z0-9]/g, "").substring(0, 12)

        const now = Math.floor(Date.now() / 1000)

        // ────────────────────────────────────────────────────────────────────
        // STEP 5: RECORD — matches live RC2 Algolia schema exactly
        // Fields verified against algolia-central_enterprise_ledger 2026-04-30
        // ────────────────────────────────────────────────────────────────────
        return [{
          objectID,
          record_type: "enterprise_ledger",
          url: rawUrl,
          title,
          description,
          content: content.substring(0, 9000),  // stay under 10KB attribute limit
          source_type,
          language_code,
          // Classification fields — all derived in-extractor, no post-processing needed
          product_tag,
          feature_tag,
          solution_tag,
          industry_tag,
          customer_tag,
          category,
          // NOTE: industry_tags (plural), features_used, integrations are product_map
          // record fields only — NOT set by crawler
          is_chunk: false,
          status: "indexed",    // one-pass: crawler IS the full pipeline
          created_at: now,
          updated_at: now,
        }]
      },
    },
  ],
})

objectID note

The inline objectID function above is a fallback for environments where crypto is unavailable in the Crawler sandbox. If require("crypto") works in the extractor context, replace with:

const objectID = require("crypto").createHash("sha256").update(rawUrl).digest("hex").substring(0, 36)

Test which works via the URL Tester — if the extractor throws on require("crypto"), use the inline version.

Per-phase config variants

Phase 1 smoke test config changes

// Add these overrides during smoke testing:
maxUrls: 100,
// schedule: "every 7 days",  // comment out
safetyChecks: { maxLostRecordsPercentage: 99 },  // disable safety for initial empty run

Phase 2 limited crawl

maxUrls: 500,

Phase 3 full crawl

// Remove maxUrls override (or set to 30000)
// Re-enable schedule
// Reset safetyChecks to production values

What to verify in URL Tester before running (Phase 1 checklist)

For each of these test URLs, verify in URL Tester → Records tab: - objectID is present and non-empty - status = "pending" - is_chunk = false - source_type matches expected value - content length > 300 chars and contains page text (not nav/footer noise) - title matches actual page title - summary, product_tag, feature_tag all null

n8n parity — what maps, what changed, what's gone

Audit trail for skipped URLs: Use Crawler dashboard → Inspector → filter by "ignored/failed". This replaces the ingestion_rejections Supabase table. You can also call GET /1/crawlers/{id}/urls via the REST API to export the full crawled URL list with statuses.