RC3-Phoenix-Product-Family-Vision.md
RC3 Phoenix — Algolia Product Family Vision
Captured 2026-04-30 from operator strategic direction. The Crawler Factory we just spec'd is ONE piece of a larger commercial product family that Algolia takes to market as an enterprise data-onboarding tool set.
Status: Vision capture. Not yet committed scope. Not yet validated commercially.
Source: Operator (Arijit Chowdhury), Crawler Factory planning session 2026-04-30.
Adjacency: This document sits alongside Brief.md, Active-Context.md, Messaging-Positioning-Pivot.md. It is forward-looking — RC3 Phoenix as Algolia's product, not just a POC pivot.
The pitch in one paragraph
"Hey, you want to bring your data into Algolia — content, product, commerce, whatever — here is a tool set."
Algolia goes to enterprise customers with a complete onboarding kit. The kit handles every step from "I have a data source" to "I have a search-and-AI experience." No bespoke integration projects. No multi-quarter Solutions Engineering engagements. The customer drops sources in; the kit produces purpose-built search indices and the agents to inquire them.
The product family — components
| # | Component | Purpose | RC3 Phoenix status |
|---|---|---|---|
| 1 | Crawler Factory | Web data onboarding. Operator gives a website URL; the factory auto-builds N Algolia crawlers (one per content domain), each writing to a purpose-tuned index. | ✅ Plan + 13 specs frozen 2026-04-30. Build pending Sonnet 4.6. |
| 2 | Connector Factory | Non-web source onboarding. Same pattern as Crawler Factory but for: SaaS apps (Salesforce, Hubspot, Zendesk), file stores (S3, GDrive, SharePoint), databases (Postgres, Snowflake, BigQuery), event streams (Kafka). | Pre-design. Build follows the Crawler Factory pattern. |
| 3 | Transformer | Normalize disparate source schemas into per-content-domain Algolia record schemas (the DSS). | Pre-design. Reuses the DSS from Crawler Factory. |
| 4 | Enricher | Add LLM-derived signals to every record: summary, tags, embeddings, intent classification, entity extraction. Per-content-domain enrichment policy. | Pre-design. |
| 5 | Indexer | Idempotent multi-index writer. Per-content-domain index settings (searchable attrs, customRanking, faceting, neuralSearch) derived from DSS. Reindex orchestration. | Partially designed — IndexManager in Crawler Factory Spec 06 is the foundation. |
| 6 | Agent Set | Specialist agent per content-domain index + orchestrator agent (hub-and-spoke). Customer asks one question; the orchestrator routes to specialists, fans out, synthesizes. | Architecture documented in Crawler Factory Plan §15. Build pending. |
The Crawler Factory's algoliacentral_factory_blueprints index is the integration point — every crawler writes a blueprint with an agent_slot field that the Agent Set consumes.
Customer narrative — what Algolia actually sells
Promise to enterprise:
"Bring your data. We bring everything around it."
What the customer experiences: 1. Customer logs into Algolia Console → "Onboard a source" wizard 2. Customer selects source type: - Website → Crawler Factory wizard - SaaS App → Connector Factory + OAuth flow - File store → Connector Factory + bucket connection - Database → Connector Factory + read replica connection 3. Wizard discovers content, classifies into domains, samples, configures, tests, commits 4. Indices populate. Per-domain agents become available. 5. Customer chats with the orchestrator agent OR uses the Algolia search API directly with neural search enabled 6. Audit trail of every crawler/connector/transformer/enricher decision lives in Algolia's own meta-indices
What this displaces in the customer's stack: - Bespoke ETL projects (Fivetran, Airbyte custom connectors, in-house pipelines) - Headless CMS migration teams - Search-relevance tuning consultants - Custom RAG agent implementations
What the customer keeps doing: - Owning the data sources - Owning the front-end - Owning user-facing disclosure / consent (per the dual-deployment shared-responsibility model)
Architectural posture
Every component in the family follows the same pattern Crawler Factory established:
- Per-content-domain indices. Marketing index, support index, education index, product-catalog index, customer-stories index, etc. Each tuned for that domain's user-search behavior.
- DSS-driven config. Schema, index settings, recordExtractor, agent prompt — all derive from one Domain Schema Standard registry.
- Blueprint persistence. Every onboarding action writes a record to
algoliacentral_factory_blueprintsso the agent layer knows what specialists exist and what they cover. - Hub-and-spoke agents. Orchestrator routes; specialists answer.
- Operator-in-the-loop config. Wizards, not black-box auto-onboarding. Customer sees what's being created and approves before commit.
Why this matters for Algolia commercially
Today: Algolia sells search. Customer brings data via custom integration; Algolia gives a fast index. Solutions Engineering owns the integration burden.
RC3 Phoenix: Algolia sells time-to-value on data. Customer brings a connection string; Algolia gives a working multi-domain search-and-AI experience in days, not quarters. SE owns the wizard UX, not the integration code.
Pricing implication: the family enables a packaged "enterprise onboarding tier" or per-source pricing. Each connector/crawler is a billable unit. The agent set is a value-add on top of indexing.
Competitive positioning: - vs. Glean / Coveo: Algolia owns the index AND the agents AND the onboarding tooling. Single throat to choke. - vs. ElasticSearch: Algolia ships the wizards and the agents. Elastic ships the engine; everything else is on the customer. - vs. building in-house: customer offloads schema design, recordExtractor authoring, index tuning, and agent prompt engineering.
What's already validated (from Crawler Factory session)
- Web Almanac 2024 + 60-site empirical audit confirms: most enterprise sites lack reliable schema.org → must use CMS-fingerprint cascade. The DSS approach handles this.
- ~40% of enterprise sites WAF-block crawlers. Playwright stealth in v1 is the right call.
- Multi-brand corporates (LVMH, Diageo) need federated discovery. Multi-tenant abstraction works.
- Massive-scale sites (Microsoft Learn 1M-10M URLs) need streaming + sharded persistence. The Crawler Factory architecture handles this.
These findings transfer to the rest of the product family. The Connector Factory will face similar API-rate-limiting and source-schema-drift patterns. The Enricher faces the per-domain-LLM-prompt problem. The Agent Set faces the orchestrator-routing problem.
Scope discipline
This document is vision capture, not committed scope. Each component (Connector Factory, Transformer, Enricher, Agent Set) requires its own pass through the Idea-to-Build pipeline (Projects/Vibe-Coded-Product-Methodology/):
- Dossier intake
- Empirical research
- Plan freeze
- Contract lock
- Spec generation
- Multi-axis verify
- Build
No component skips the methodology. Every component is its own commercial bet that needs validation.
Open commercial questions
| Question | Status | Owner |
|---|---|---|
| Pricing model — per-source, per-index, per-query, tiered? | Open | Product / SE |
| GTM motion — sold as "tool set" or "platform"? | Open | Marketing |
| Customer trial path — self-serve wizard or SE-led POC? | Open | Sales |
| Data residency / compliance posture | Open | Legal / Security |
| Connector breadth at GA — minimum viable list of source types? | Open | Product |
| Agent Studio dependency — required or pluggable? | Open | Eng |
| Existing Maverick/Elena/Bruno relationship — same agents under orchestrator, or sunset? | Open | Eng / Product |
| Pricing of the Agent Set — separate SKU or bundled? | Open | Product |
Connection back to the methodology
The methodology being built (Projects/Vibe-Coded-Product-Methodology/) is the engine that PRODUCES this product family. Each component goes through it. The methodology is generic; this is one Algolia-specific application.
If the methodology works as designed, building Components 2-6 should be cheaper per component than the Crawler Factory was — because each component reuses the same: - DSS pattern - Blueprint pattern - Multi-axis verification skill (once it exists) - Contract Lock pattern
That's the test of the methodology: does the next component cost less than the last one?