Dashboard/Crawler-Factory/Reference/Algolia-Central-Context.md
Crawler-Factory

Reference/Algolia-Central-Context.md

Algolia Central — Context for Crawler Factory

Why this file exists. Crawler Factory is one module of a multi-module platform called Algolia Central / Content Engagement. Without that context, anyone touching Crawler Factory will optimize the wrong axis. Read this first when joining the project, when re-routing a feature, or when scoping new work.

Master brief: [Brief](/p/crawler-factory/../Algolia-Central/Brief.md) is the durable source of truth for the parent product. This file lifts the parts of that brief that bear directly on the Crawler.

1. What Algolia Central is

External name: Algolia Content Experience.

Three layers of existing Algolia capability, packaged together for public and semi-public knowledge — documentation, support content, learning content, blogs, product/service guidance.

       ┌──────────────────────────────────────────────────────────┐
       │  Layer 3 — Agent Studio                                 │
       │  Context-aware, citation-backed agents. Personas. A2A.  │
       └──────────────────────────────────────────────────────────┘
                                  ▲
       ┌──────────────────────────────────────────────────────────┐
       │  Layer 2 — Algolia AI Retrieval                         │
       │  Hybrid keyword + semantic search. Grounding layer.     │
       │  Eliminates hallucination.                              │
       └──────────────────────────────────────────────────────────┘
                                  ▲
       ┌──────────────────────────────────────────────────────────┐
       │  Layer 1 — Data Layer  (Crawler Factory lives here)     │
       │  Connector → Transformer → Enrichment → Normalization → │
       │  Indexing.                                              │
       └──────────────────────────────────────────────────────────┘

Crawler Factory is the FIRST stage of Layer 1. Everything downstream depends on it.

2. The pipeline (bottom-up)

Stage Module name What it does Crawler Factory's role
1 Connectors / Crawlers Fan out to content silos (web, CMS, KB, video transcripts, code repos) and pull raw content THIS IS US. First-class crawler is web. Future connectors: CMS, KB, video, code.
2 Transformer Homogenize heterogeneous content into a common record structure Reads Crawler Factory's per-domain blueprints + records
3 Enrichment Algolia Fetch enriches each record before indexing Operates on the records the crawler emits
4 Normalization Flatten / denormalize into Algolia-native shape
5 Algolia NeuralSearch Index everything; hybrid keyword + semantic search on top — "the heart and soul" Consumes the per-domain indices Crawler Factory provisions
6 Agentic layer (Agent Studio) Service / sales / advisory / recommendations agents grounded on the index Each specialist agent maps to a Crawler-Factory-produced index

This pipeline shape comes from Algolia-Central/Brief.md Part 2. Verify there before quoting elsewhere.

3. Two deployment modes (Crawler Factory must support both)

Mode Who owns the agent surface Crawler Factory implication
Algolia-as-platform Algolia owns retrieval, orchestration, and UI. End users interact directly with our agents. Crawler Factory provisions indices Algolia-side. Standard.
Headless / Agent-to-Agent (A2A) Customer's orchestrator (e.g. Adobe Brand Concierge, Microsoft Foundry) routes content queries to our agents via A2A protocols. They own the UI. Crawler Factory must produce indices and metadata that other agent frameworks can consume. Multi-tenant + per-customer namespacing matters here.

Both pilots (Sherwin Williams, MasterCard) are confirmed. Adobe Brand Concierge expressed pilot interest.

4. The "Central" pattern — Crawler Factory is the template engine

Algolia Central, Sherwin Central, MasterCard Central, Adobe Central — the "Central" pattern is the product. Same architecture, different index, different content, different agents.

This means Crawler Factory is not a one-off internal tool. It is the template engine that stamps out every customer's "Central." Every customer Central needs: - A web crawler set (Crawler Factory) - Per-content-domain indices - Agent personas grounded on those indices

Every UX decision in Crawler Factory should be evaluated through this lens: will this work when we duplicate it 10 times for 10 customers?

5. What Algolia Central / Content Engagement IS and IS NOT

IS

  • Public website content (Marketing, Blogs, News)
  • Technical Documentation & Service Manuals
  • Public Support Knowledge Bases (guides, troubleshooting articles)
  • Video transcripts (YouTube tutorials, Academy courses)
  • Code Repositories (Code Exchange)

Capabilities

  • Public content ingestion → Crawler Factory
  • Algolia AI Retrieval
  • Agent Studio orchestration
  • Headless A2A integration
  • Source-cited answers

IS NOT

  • ❌ File search (PDF, PPT, etc.)
  • ❌ Secure / internal employee content search
  • ❌ Generation of new content
  • ❌ Custom-services-heavy AI build

6. Strategic positioning — why this matters now

Low risk, capital-efficient. We're not inventing a new product category. We're packaging existing strengths (ingestion, indexing, hybrid retrieval, relevance, analytics, AI agents) to solve a high-frequency customer problem.

Initial focus = adjacent to ecommerce. B2B SaaS docs, support sites, developer portals, product education content. These are easier to operationalize than secure internal enterprise search.

Strategic outcome (3 things at once): 1. Incremental growth from an adjacent problem set. 2. Broader market visibility for what Algolia does beyond commerce. 3. A low-risk proving ground for "trusted retrieval infrastructure for AI workflows."

For Sales/Marketing this is a sharper story for accounts that don't see themselves as ecommerce buyers, plus an expansion path inside existing customers who use Algolia for one search type but have broader discovery problems elsewhere.

Source: Google Doc "Content Experience as a Low-Risk Growth Wedge Beyond Ecommerce" (executive summary). Captured verbatim in Reference/Content-Engagement-Memo.md.

7. How Crawler Factory's design decisions trace back to Central

Crawler Factory decision Reason rooted in Algolia Central
Per-domain indices (one crawler → N indices) Each index becomes the grounding for a specialist agent at Layer 3. The factory is the agent-readiness scaffolder.
Streaming sitemap walker, no caps Content Engagement promises broad public-content coverage; can't cap at 50K when MasterCard has millions of docs URLs.
Per-domain blueprint registry Transformer + Enrichment stages will consume blueprints. The blueprint IS the contract between Layer 1 and Layers 2-3.
Multi-tenant index naming (algoliacentral_*) The "Central" pattern means many customer instances. Naming convention prevents collision.
Dashboard design matches Algolia's existing dashboard (Session 2 decision) Crawler Factory will eventually merge into the Algolia dashboard alongside other Layer 1 modules. Designing in the Algolia visual language from day one avoids a re-skin later.
Pause / Resume / Re-crawl as our value-add (Session 2 decision) Algolia's base crawler is fire-and-forget. Long-running multi-domain ingestion (the Central use case) needs operator control. This is what makes Crawler Factory a layer, not a replacement.

8. Open questions that span both projects

These need resolution at the Algolia Central level. Crawler Factory will follow the answer.

ID Question Status
Q-AS1 One agent or two? Partially answered. See Algolia-Central/Brief.md Part 5 §Q-AS1.
Q-AS4 How is the content sliced into N indices — by content type? by vertical? by source? Open. Drives Crawler Factory's classification taxonomy.
Q-AS5 Does the 2-panel UI ship with or after agent re-architecture? Decided UI is parallel workstream. Crawler Factory dashboard work proceeds independently.
NEW (Session 2) When does Crawler Factory's UI merge into the main Algolia dashboard? Not answered. Affects how we package the design language.
NEW (Session 2) What is the navigation path inside the Algolia dashboard from Data Sources to Crawler Factory? Not answered. Likely "Data sources → Crawlers → Crawler Factory" but needs Algolia internal product alignment.

9. Cross-references (bidirectional)

Inbound (Algolia Central → Crawler Factory): - **Brief** Part 2 ("the stack, bottom to top") — Crawlers fan-out is Layer 1. - **RC3-Phoenix-Product-Family-Vision** mentions Crawler Factory as a member of the product family. - **Active-Context** should reference Crawler Factory whenever Layer 1 work is active.

Outbound (Crawler Factory → Algolia Central): - **00-Plan.md** references Content Engagement. - **05-Architecture-Hub-Spoke.md** references the agent architecture. - **02-DSS.md** references the content-domain taxonomy. - This file (Algolia-Central-Context.md) is the canonical bidirectional anchor.

10. When this file is wrong

If Algolia Central's vision shifts (the brief changes, a pilot pivots, the pipeline is restructured), update this file and the corresponding row in Algolia-Central/Brief.md. Do not let them drift.