Crawler-Factory

Core/06-Multi-Tenant.md

Multi-Tenant Orchestration

Extracted from 00-Plan.md §17 + §18.

17. Multi-tenant + multi-domain orchestration (v1.5)

The user's vision includes scenarios where one logical entity (LVMH) has many brands (Louis Vuitton, Dior, Céline, ...), each on its own domain, each with its own crawl. The factory needs a "tenant" abstraction so these are managed together but crawled independently.

17a. Tenant model

Tenant (e.g., "LVMH")
  ├─ FactorySession 1 → corporate (lvmh.com)
  ├─ FactorySession 2 → louisvuitton.com
  ├─ FactorySession 3 → dior.com
  ├─ FactorySession 4 → celine.com
  └─ ... (one per brand domain)

Each FactorySession produces N crawlers. Each crawler writes to a domain-specific Algolia index. Indices CAN be shared across tenant brands (e.g., all LVMH brand-marketing → algoliacentral_marketing with tenant_id as a facet).

17b. Algolia meta-index for tenants (NEW)

Index: algoliacentral_factory_tenants
  objectID: <tenantId>
  name: "LVMH"
  description: "Luxury conglomerate; 75+ brands across fashion, wines, spirits, cosmetics, watches"
  parent_domain: "lvmh.com"
  brand_domains: ["louisvuitton.com", "dior.com", "celine.com", ...]
  session_ids: ["sess_001", "sess_002", ...]
  blueprint_ids: [...]
  created_at: ts

Tenant creation is OPTIONAL in v1 — single-domain workflows skip it. The "Discover brands" CTA on a parent corporate session creates a tenant and links sub-sessions.

17c. v1 vs v2 split for multi-tenant

v1 (this spike): - Single-session, single-domain workflow works end-to-end - "Detect brand domains" sub-flow on multi-brand parent: scrapes brand-index page, lists external domains, lets user start ONE brand session at a time (manually triggered) - Tenant record created automatically when 2+ sessions share a parent

v2 (deferred): - Bulk brand crawl: "crawl all 12 brands in parallel" - Cross-tenant search routing for the orchestrator - Tenant-level dashboards (combined progress, combined record counts)


18. Data the factory exposes for downstream automation

Every artifact the factory produces is queryable via Algolia indices. This makes it consumable by future automation (the SOP-driven engineering spec generator, the agent factory, evaluation pipelines, etc.):

Index Purpose Consumed by
algoliacentral_factory_sessions Session state (sharded: session, url_shard, sample, log records) Factory wizard UI, audit trails
algoliacentral_factory_blueprints Crawler blueprints (agent-ready metadata) Future specialist-agent factory
algoliacentral_factory_tenants Multi-tenant grouping Future cross-brand orchestration
algoliacentral_<domain> (×N) Actual content indices, one per content domain Specialist agents, current chat agents

The factory writes; everything else reads. No tight coupling. Each downstream automation reads from a stable, queryable contract.