Crawler-Factory

Wiki/DemoScript.md

Demo Script — Crawler Factory

Last updated: 2026-05-03 Status: Demo-ready (UI verified S4)

Purpose

Crawler Factory eliminates the manual work of setting up Algolia crawlers: paste a URL, watch the system discover every content type on the site, and walk out with one production-ready crawler and index per domain — in under three minutes.

Demo angle

Story arc: The problem is invisible until you've done it manually. Start by surfacing the pain (configuring one crawler per content area by hand), then show the wizard running against algolia.com in real time. The streaming discovery animation is the product's strongest proof point — let it breathe. End on the Done screen with four real crawlers created and ready to run.

Prerequisites

Dev server running: npm run dev (UI on http://localhost:5200, API on port 3006)
Navigate to http://localhost:5200/admin/factory before the audience arrives
The algolia_com factory entity and old_crawler row should be visible in the list
Screen resolution: at least 1440px wide — the workspace shell needs the left nav visible

Runtime: ~3 minutes

Scene 1 — The problem (30 seconds)

What to show: Crawler list view — two rows in the table

What to say:

"Here's how most Algolia customers set up crawlers today. You have algolia.com. Documentation lives at /docs. Blog at /blog. Community at /community. Products at /products. Each one needs its own crawler, its own index, its own extractor config. That's four separate setup flows, four index naming decisions, four times you're writing a recordExtractor function.

And if you miss a content area, or the URL structure changes, you find out when search results go stale.

The row at the bottom — old_crawler — that's the manual path. Failed status. The row at the top is what we built instead."

Pause for: Nods of recognition from anyone who has set up Algolia crawlers before. If the audience is less technical, skip the extractor mention and just say "four separate setup flows."

Point at: The old_crawler row (Failed, Manual tag) and then the algolia_com row (Running, Factory tag, "3 crawlers" sub-badge).

Scene 2 — The entry point (15 seconds)

What to show: The blue promo banner and the "+ Add new crawler" button

What to say:

"From the standard Crawler list, there's one new entry point. This banner. Click it — or hit '+ Add new crawler' — and instead of dropping into a blank crawler form, you land in the Factory wizard."

Pause for: None. This is a transition beat — keep moving.

Action: Click "Try it →" on the blue banner (or "+ Add new crawler"). The wizard opens at Step 1.

Scene 3 — Step 1: Discover (30 seconds)

What to show: Wizard Step 1 — "Tell us your website"

What to say:

"Step 1. One field. Paste your domain.

We're not asking for a sitemap URL, a crawl budget, a URL pattern, or a record extractor. Just the domain.

Watch what happens when I type algolia.com."

Action: Click the algolia.com quick-fill chip (or type it manually into the domain field). Point out the chips — "We pre-load common sites so you can demo without typing."

What to say (while pointing at the 'WHAT HAPPENS NEXT' card):

"The system tells you upfront what it's going to do. Walk the sitemap, group URLs by content type, build one crawler per type. No surprises."

Pause for: Question about whether it works on private/authenticated sites. Answer: "We only read public pages — no login, no scraping behind paywalls. There's a note right there." Point at the privacy note.

Action: Click "Continue."

Scene 4 — Step 2 mid-run: The streaming animation (45 seconds — this is the wow moment)

What to show: Wizard Step 2 Review, mid-discovery state — progress bar animating, stats ticking up, first category appearing

What to say:

"This is the moment. The system is walking the sitemap right now — live.

Watch the stat tiles. Three sitemaps found. Four thousand six hundred URLs, and counting. One category already detected — Documentation, high confidence, the /docs/** path, 2,840 URLs.

The confidence score matters. High confidence means the URL pattern is clean and unambiguous. Medium means there's some overlap we'll let you resolve. Low confidence lands unchecked by default — the system flags it but doesn't assume you want it."

Pause for: The audience watching the animation. Let the "Classifying content types..." status text and the animating progress bar do the work. This is 10-15 seconds of purposeful silence. If someone asks what's happening under the hood:

"It's reading sitemaps, clustering URLs by path prefix and content signals, running a confidence classifier. The whole thing takes about 15 to 30 seconds for a site this size."

What to watch for: The first category card appearing while the rest still show "Classifying..." — that progressive reveal is the visual payoff. Point at it explicitly.

Scene 5 — Step 2 complete: Review categories (30 seconds)

What to show: Wizard Step 2 Review, complete state — "Discovery complete — 5 categories found."

What to say:

"Done. Five categories. Fifteen seconds for a site with four thousand URLs across three sitemaps.

Documentation, Blog, Community, Products — all checked. Support is there too, but it came back low confidence, so it's unchecked by default. We're not hiding it — you can include it if you want — but the system isn't going to create a crawler for a category it's not sure about without your explicit sign-off.

Four of five selected. I'll leave Support off for now."

Action: Leave the default selection as-is (4 checked, Support unchecked). Click "Confirm categories."

Pause for: Questions about the confidence levels. If asked:

"Confidence comes from URL pattern clarity and record density. /docs/** with 2,840 URLs is unambiguous. /support/** with 440 URLs and some overlap with Community paths — that's where the model hedges."

Scene 6 — Step 3: Configure (30 seconds)

What to show: Wizard Step 3 Configure — "Configure your crawlers"

What to say:

"Step 3. The system has pre-filled an index name for each category. documentation_index, blog_index, community_index, products_index.

You can rename any of them before we build. The name is what appears in your Algolia app.

The preview on the right shows you exactly what you're committing to: index name, URL pattern, estimated record count. No surprises at creation time."

Action: Click through the category list on the left — show how selecting each one updates the right panel. You don't need to change any names for the demo — the defaults are clean.

Point at: The "4 of 4 ready" footer badge. "Everything is configured. One click to build."

What to say:

"Four crawlers. Four indices. One click."

Action: Click "Build crawlers."

Scene 7 — Step 4: Done (30 seconds)

What to show: Wizard Step 4 Done — "4 crawlers created."

What to say:

"Done. Four crawlers — Documentation, Blog, Community, Products. Each one is idle, waiting for its first crawl. They're real Algolia crawlers. You can open any of them right now and start a crawl, or schedule them for later.

The whole flow: URL in, categories reviewed, names confirmed, four crawlers created. Under three minutes. That's what a site like algolia.com used to take hours to set up by hand."

Action: Click "Open →" on documentation_index to show it transitions to the standard Algolia crawler workspace — demonstrating these are real crawlers, not stubs.

Pause for: The applause moment. Then:

"And when you come back to the Factory list, you'll see this factory — algolia_com — as one entry with a '4 crawlers' sub-badge. One entity to manage, four crawlers underneath."

Scene 8 — The workspace (optional, if time allows — 30 seconds)

What to show: Factory workspace Overview tab for an existing factory (the algolia_com row from the list)

What to say:

"For factories that are already running, the workspace gives you the full picture in one place. Eight thousand records indexed. Last crawled two hours ago. Ninety-nine percent success rate.

The metadata card at the bottom shows you the provenance: Factory version, which blueprint was used to generate the extractors, the discovery session ID. Full audit trail."

Point at: The "Crawler Factory metadata" card — Source: Factory v1.1, Blueprint: algolia.com/docs, Discovery session: sess_01HZ.

Pause for: Questions about re-running discovery if the site structure changes. Answer: "The Setup wizard in the left nav lets you re-run discovery at any time. The factory entity persists; you're just refreshing the blueprint."

Key talking points

One URL in, N crawlers out. The operator doesn't need to know what content types exist on the site before starting. Discovery does that work.
Confidence-gated creation. The system won't create crawlers for content it's uncertain about without explicit operator approval. Low-confidence categories are surfaced but unchecked. This is the anti-footgun design.
Real crawlers, not stubs. Every crawler created by the Factory is a first-class Algolia crawler. It lives in your app, it's manageable from the standard Crawler dashboard, and it respects all the same crawl controls (pause, resume, schedule, URL inspector).
The streaming animation is the demo. The 15-second discovery run is the product's clearest proof of value. Everything before it is setup; everything after it is confirmation. Build your timing around that moment.
Factory entity = one management surface. Instead of four independent crawlers scattered across the dashboard, you get one factory entity with a sub-crawler count badge. That's the operational story — you manage the factory, not the crawlers.

Objection handling

"We already have crawlers set up manually. Why would we migrate?"

"You don't have to migrate. New sites or new content areas are where this pays off immediately. And if your existing crawler setup is fragile — one big crawler covering everything — Factory gives you a clean path to a domain-per-crawler architecture without rebuilding everything from scratch."

"What happens when my site structure changes?"

"Re-run discovery from the Setup wizard. The factory entity persists; discovery produces a new set of categories. You confirm what changed, and the factory reconciles — adding new crawlers, flagging deprecated ones. You're not starting from zero."

"How does it know what a 'documentation page' is vs a 'blog post'?"

"URL path clustering plus content signals from the sitemap — last-modified dates, changefreq, page density patterns. It's not doing full page renders at this stage; it's working from the sitemap graph. The confidence score tells you how clean the signal was. High confidence means the URL pattern was unambiguous. Low confidence means there was noise — you review it, you decide."

Demo reset

To return to start state for another run:

Navigate to http://localhost:5200/admin/factory — the list view is the entry point
The algolia_com factory entity and old_crawler row should be present
If you clicked through a new wizard run and created additional crawlers, refresh the page — the list resets to its seeded state in the mock data layer
If the dev server has drifted, run npm run dev from /Users/arijitchowdhury/AI-Development/Crawler_Factory to restart both servers

The quick-fill chips on Step 1 (algolia.com, shopify.com, stripe.com, vercel.com) all trigger the same mocked discovery flow — any of them work for a repeat demo run.