Crawler-Factory

Core/05-Architecture-Hub-Spoke.md

Future Agent Architecture — Hub-and-Spoke Vision

Extracted from 00-Plan.md §15.

15. Future agent architecture (hub-and-spoke vision)

The Crawler Factory is the data-foundation layer for a multi-agent future. This section captures the vision so the factory's architecture (especially the algoliacentral_factory_blueprints index and the per-domain index naming) is forward-compatible.

15a. The vision

Today: Algolia Central has Maverick (sales), Elena (architect), Bruno (solutions engineer) — three agents sharing one index (algolia-central_enterprise_ledger). Routing between them is hand-coded in scripts/agent-studio/maverick-agent-config.ts.

Future: N specialist agents, each owning one content domain. They share a common orchestrator that fans questions out and synthesizes answers.

                          ┌──────────────────────┐
                          │   ORCHESTRATOR HUB   │
                          │  (routes, dissects,  │
                          │   fans out, merges)  │
                          └──────────┬───────────┘
                                     │
        ┌──────────┬──────────┬──────┼──────┬──────────┬──────────┐
        ▼          ▼          ▼      ▼      ▼          ▼          ▼
   ┌────────┐ ┌────────┐ ┌────────┐ ...  ┌────────┐ ┌────────┐ ┌────────┐
   │ market │ │supprt  │ │educ.   │      │tech.   │ │custmr  │ │social  │
   │ agent  │ │agent   │ │agent   │      │agent   │ │stories │ │agent   │
   │   ↓    │ │   ↓    │ │   ↓    │      │   ↓    │ │agent ↓ │ │   ↓    │
   │mkting  │ │support │ │edu     │      │tech    │ │cust    │ │social  │
   │index   │ │index   │ │index   │      │index   │ │index   │ │index   │
   └────────┘ └────────┘ └────────┘      └────────┘ └────────┘ └────────┘

15b. How the factory enables this

Every successful crawler creation writes a blueprint record to algoliacentral_factory_blueprints (§4e). The blueprint contains: - The content domain (e.g., support) - The index name (e.g., algoliacentral_support) - The schema.org types in scope (so the agent knows what shapes of records to expect) - The URL paths that feed it (so the agent knows what the user's question MIGHT be about) - An agent_slot field, currently null

When we build the specialist-agent factory (a future spike, not v1), it reads algoliacentral_factory_blueprints, generates an Agent Studio config per blueprint (system prompt, tool list pointing at the right index, voice tuned to the domain), and writes the new agent's UUID back to agent_slot. The orchestrator then queries blueprints to know which specialists exist and what they cover.

15c. Orchestrator query flow (example)

User: "What's our latest product launch and what are people saying about it?"

  1. Orchestrator parses → identifies sub-questions: - "What's our latest product launch?" → marketing index - "What are people saying about it?" → social index + customer-stories index
  2. Orchestrator fans out: 3 parallel Agent Studio calls
  3. Each specialist runs its own retrieval against its tuned index
  4. Orchestrator collects responses, deduplicates evidence, synthesizes a unified answer with citations from all 3 specialists.

15d. What v1 of the factory delivers toward this future

  • Per-domain indices created and populated → ✅ data foundation ready
  • Blueprints index with agent_slot field → ✅ scaffolding hook ready
  • DSS-driven config per domain (different searchableAttributes, customRanking, facets) → ✅ each index pre-tuned for its agent's needs
  • Naming convention algoliacentral_<domain> → ✅ predictable for the future agent factory to discover

What v1 does NOT do (intentionally, future work): - Specialist agent scaffolding - Orchestrator routing logic - A2A communication protocol - Specialist agent prompt templates per domain

These are all future spikes that consume the v1 outputs.