Dashboard/Crawler-Factory/Engineering-Specs/13-bundle-smoke-docs.md
Crawler-Factory

Engineering-Specs/13-bundle-smoke-docs.md

Crawler Factory: Spec 13: Bundle + Smoke Test + Docs + Vault Sync Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Final verification cluster: bundle every new factory API endpoint into api/factory/*.mjs, provision the two factory meta-indices, run an end-to-end live smoke against https://www.algolia.com that proves discover → categorize → configure → REAL test crawl → create-crawler → live monitor for 3 distinct content domains, then write the operator README and sync vault Status.md so the v1 spike is closed.

Architecture: This is the LAST spec to execute: it does NOT add new product surface area. It bundles, provisions, exercises, documents, and closes the loop. The smoke test is the ultimate truth: if https://www.algolia.com produces 3 working crawlers each writing into a distinct content-domain index, with a resumable session and persisted blueprints, the spike has shipped. Bundling and provisioning happen first so the smoke runs against the same artifact Vercel will serve. Documentation and vault sync happen last so they reflect any deviations the smoke surfaces.

Tech Stack: TypeScript (strict), zod 3.25, algoliasearch v5, vitest, esbuild via scripts/bundle-api.mjs, tsx for the bootstrap script, pino logger via lib/utils/logger.ts.

Depends on: Specs 01–12 all merged on rc3-phoenix. Specifically: - Spec 01: lib/factory/dss.ts, types.ts, session-store.ts, blueprint-store.ts - Spec 06: lib/factory/index-manager.ts (must export ensureFactoryMetaIndices) - Specs 02–10: all api-src/factory/*.ts source endpoints exist - Specs 11–12: frontend /admin/factory route + LiveCrawlMonitor component shipped

Consumed by: Nothing. This is the terminal cluster.

Plan source: Projects/Crawler-Factory/00-Plan.md §7 Tasks 30–31, §9 (Verification end-to-end), §16p (Validation challenge protocol: applied to algolia.com as the smoke target).

File structure

Path Responsibility
api/factory/discover.mjs Bundled SSE discover endpoint (generated, committed)
api/factory/sample.mjs Bundled sample endpoint (generated, committed)
api/factory/generate-extractor.mjs Bundled extractor-generator endpoint (generated, committed)
api/factory/test-sandbox.mjs Bundled sandbox-test endpoint (generated, committed)
api/factory/test-real.mjs Bundled real-test endpoint (generated, committed)
api/factory/create-crawler.mjs Bundled create-crawler endpoint (generated, committed)
api/factory/crawl-progress.mjs Bundled crawl-progress SSE endpoint (generated, committed)
api/factory/session.mjs Bundled session CRUD endpoint (generated, committed)
api/factory/blueprints.mjs Bundled blueprints CRUD endpoint (generated, committed)
scripts/factory-bootstrap.ts One-off provisioning script that calls IndexManager.ensureFactoryMetaIndices()
docs/factory/README.md Operator README: concept diagram, env vars, resume mechanics, troubleshooting
SESSION.md Repo session log: Crawler Factory v1 marked complete + new active task
Projects/Crawler-Factory/Status.md (vault) Cluster status table: all clusters ✅ + Lessons learned section

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every artifact in this spec MUST follow these rules.

  1. Verification before completion. No step is "done" until its expected output has been observed. The smoke task in particular is a real network operation: every bullet is an observation gate, not a hope.
  2. Logger usage. The bootstrap script imports createLogger from lib/utils/logger.ts. No console.log in the bootstrap script.
  3. Try/catch every external call. The bootstrap script wraps the ensureFactoryMetaIndices() call. Errors are logged with full context, then rethrown.
  4. Functions ≤20 lines, ≤3 params. Bootstrap script is short by design.
  5. Conventional commits. chore(factory):, docs(factory):, chore(session):: one logical change per commit.
  6. Evidence on every claim. The smoke task records observations (URL counter values, dashboard screenshots, blueprint object IDs): not "looked fine".
  7. WritingSOPs for the README. Operator-readable. No marketing voice. Concrete commands, expected outputs, env-var names. Troubleshooting entries describe the symptom first, then the fix.
  8. Vault sync discipline. Status.md is updated AFTER smoke passes, not before. Lessons learned section captures any deviation from the plan as data, not as apology.
  9. Branch hygiene. All work on rc3-phoenix. main and rc2-algolia are sealed (per project CLAUDE.md). No commits to either.
  10. Bundle reproducibility. scripts/bundle-api.mjs is the single source of truth: no hand-edits to api/factory/*.mjs. If a bundle is wrong, fix the source api-src/factory/*.ts and re-run the bundler.

Task 1: Bundle the API

Files: - Modify: api/factory/discover.mjs, api/factory/sample.mjs, api/factory/generate-extractor.mjs, api/factory/test-sandbox.mjs, api/factory/test-real.mjs, api/factory/create-crawler.mjs, api/factory/crawl-progress.mjs, api/factory/session.mjs, api/factory/blueprints.mjs (all generated by the bundler: committed but not hand-edited)

  • [ ] Step 1.1: Confirm api-src/factory/ has all 9 endpoint sources
ls api-src/factory/

Expected output (file order may vary):

blueprints.ts
crawl-progress.ts
create-crawler.ts
discover.ts
generate-extractor.ts
sample.ts
session.ts
test-real.ts
test-sandbox.ts

If any file is missing, STOP: the corresponding earlier spec did not land. Do not proceed.

  • [ ] Step 1.2: Confirm scripts/bundle-api.mjs knows how to bundle the factory endpoints
grep -n "factory" scripts/bundle-api.mjs

Expected: at least one match referencing the api-src/factory/ glob. If no match, the bundler entrypoint list does not include factory endpoints: fix the bundler before continuing (this should have been done in an earlier spec; flag it as a gap).

  • [ ] Step 1.3: Run the bundler
node scripts/bundle-api.mjs

Expected: completes without error. Stdout lists each bundled entry including api/factory/discover.mjs, api/factory/sample.mjs, etc.

  • [ ] Step 1.4: Verify each api/factory/*.mjs exists
ls -1 api/factory/

Expected output:

blueprints.mjs
crawl-progress.mjs
create-crawler.mjs
discover.mjs
generate-extractor.mjs
sample.mjs
session.mjs
test-real.mjs
test-sandbox.mjs

If any are missing: re-check the bundler entrypoint list, fix, re-run Step 1.3. Do NOT hand-edit api/factory/*.mjs.

  • [ ] Step 1.5: Spot-check one bundled file is valid JS
node --check api/factory/discover.mjs

Expected: no output (success).

Repeat for at least 2 more bundles:

node --check api/factory/create-crawler.mjs && node --check api/factory/crawl-progress.mjs

Expected: no output (success).

  • [ ] Step 1.6: Commit the bundles
git add api/factory/
git commit -m "chore(factory): bundle factory API endpoints"

Expected: commit succeeds; git log -1 --stat shows 9 new .mjs files under api/factory/.

Task 2: Run full test suite + tsc

Files: - (Read-only verification)

  • [ ] Step 2.1: Run the full vitest suite
npx vitest run

Expected: ALL tests pass. Baseline 407 + every test added across Specs 01–12. There must be ZERO failures and ZERO skipped tests that were not already skipped pre-Spec-01.

If any test fails: 1. STOP. Do not proceed to Step 2.2. 2. Diagnose root cause (use superpowers:systematic-debugging if the cause is non-obvious). 3. Fix the underlying issue. Re-run from Step 2.1.

Do NOT skip a test to make this step pass. Do NOT mark this step done with failures present.

  • [ ] Step 2.2: Run TypeScript type-checking
npx tsc --noEmit

Expected: no output (clean). Zero errors.

If errors surface: 1. STOP. 2. Fix every error. The factory cluster's invariant is tsc --noEmit is clean. 3. Re-run from Step 2.1 (test suite first, then tsc: fixing types can break logic).

  • [ ] Step 2.3: Verify no uncommitted changes from the previous steps
git status

Expected output (working tree clean, or only Task 3+ files in progress):

On branch rc3-phoenix
nothing to commit, working tree clean

If Task 1 left untracked files, commit or clean them now before proceeding: Task 3 needs a clean baseline.

Task 3: Provision factory meta-indices

Files: - Create: scripts/factory-bootstrap.ts

The two factory meta-indices (algoliacentral_factory_sessions, algoliacentral_factory_blueprints) must exist on the Algolia app with the correct attributesForFaceting BEFORE the smoke runs. Spec 06's IndexManager.ensureFactoryMetaIndices() is the canonical provisioner: this task simply runs it.

  • [ ] Step 3.1: Confirm IndexManager.ensureFactoryMetaIndices exists
grep -n "ensureFactoryMetaIndices" lib/factory/index-manager.ts

Expected: at least one match (the method definition). If no match, Spec 06 is incomplete: STOP and address the gap there before proceeding.

  • [ ] Step 3.2: Confirm Algolia credentials are present
node -e "console.log('appId:', !!process.env.ALGOLIA_APP_ID, 'adminKey:', !!process.env.ALGOLIA_ADMIN_KEY)"

Expected output:

appId: true adminKey: true

If either is false, source .env.local (or your shell's env file) and retry. Per project CLAUDE.md (memory: feedback_algolia_central_app_id), the Algolia Central app id is 0EXRPAXB56: confirm it matches:

echo "$ALGOLIA_APP_ID"

Expected: 0EXRPAXB56. If it shows 1QDAWL72TQ (the old RC2 app), STOP: this is the wrong app and would create indices in the wrong place.

  • [ ] Step 3.3: Write the bootstrap script

Create scripts/factory-bootstrap.ts:

/**
 * One-off bootstrap: provision the two factory meta-indices on Algolia.
 *
 * Why a separate script: IndexManager.ensureFactoryMetaIndices() is normally
 * called lazily on first endpoint use. The smoke test needs the indices to
 * exist BEFORE the first request, so we run it once explicitly.
 *
 * Run: npx tsx scripts/factory-bootstrap.ts
 * Idempotent: safe to re-run; ensure* methods skip when settings already match.
 */

import { createLogger } from '../lib/utils/logger';
import { IndexManager } from '../lib/factory/index-manager';

const log = createLogger('factory:bootstrap');

async function main(): Promise<void> {
  const appId = process.env.ALGOLIA_APP_ID;
  const adminKey = process.env.ALGOLIA_ADMIN_KEY;
  if (!appId || !adminKey) {
    log.error({ event: 'bootstrap_missing_creds' }, 'ALGOLIA_APP_ID or ALGOLIA_ADMIN_KEY not set');
    process.exit(1);
  }
  const manager = new IndexManager(appId, adminKey);
  try {
    const result = await manager.ensureFactoryMetaIndices();
    log.info({ event: 'bootstrap_done', result }, 'factory meta-indices ensured');
  } catch (err) {
    log.error({ event: 'bootstrap_failed', err: String(err) }, 'failed to ensure meta-indices');
    process.exit(1);
  }
}

main();
  • [ ] Step 3.4: Run the bootstrap
npx tsx scripts/factory-bootstrap.ts

Expected: structured pino log lines, ending with an event: 'bootstrap_done' entry. Exit code 0.

If exit code is nonzero, read the error log line. Common causes: - Forbidden: admin key lacks editSettings ACL → fix the key, retry. - Network error → retry. - Wrong app id → confirm Step 3.2 again.

  • [ ] Step 3.5: Verify both indices exist with correct facets in the Algolia dashboard

Open the Algolia dashboard for app 0EXRPAXB56 → Indices.

Expected: - algoliacentral_factory_sessions exists. Click → Configuration → Facets → confirm attributesForFaceting contains filterOnly(record_type) and filterOnly(parent_id). - algoliacentral_factory_blueprints exists. Click → Configuration → Facets → confirm attributesForFaceting contains filterOnly(content_domain) and filterOnly(status).

If either index is missing or has wrong facets, the ensureFactoryMetaIndices() implementation in Spec 06 has a bug: STOP, fix it there, re-run Step 3.4, re-verify.

  • [ ] Step 3.6: Commit
git add scripts/factory-bootstrap.ts
git commit -m "chore(factory): bootstrap factory meta-indices"

Task 4: End-to-end manual smoke against https://www.algolia.com

Files: - (Read-only verification: runs the live UI)

This task is the truth. It exercises every code path Specs 01–12 added. If any sub-step fails, the spike has not shipped: fix the underlying spec, re-bundle, re-run.

The site under test is https://www.algolia.com because per §14 / §16p of 00-Plan.md it is the canonical reference site for this validation: schema.org coverage, sitemap structure, JSON-LD presence, and cross-domain content (blog + docs + customers) are all known.

  • [ ] Step 4.1: Start dev:api on port 3005

In one terminal:

npm run dev:api

Expected: tsx dev-server.mjs boots; logs listening on :3005 (or equivalent). Leave running.

  • [ ] Step 4.2: Start the Vite dev server on port 5173

In a second terminal:

npm run dev

Expected: Vite logs Local: http://localhost:5173/. Leave running.

  • [ ] Step 4.3: Open the factory UI

Browse to http://localhost:5173/admin/factory.

Expected: the factory page renders (DiscoveryPanel visible: empty input, Submit disabled until a URL is typed).

If the page 404s or errors, check the React Router config from Spec 11. Do not proceed.

  • [ ] Step 4.4: Submit https://www.algolia.com

In the URL input, type https://www.algolia.com. Click Submit.

Expected: - Page transitions out of "empty" state. - RollingLog appears and starts populating. - The browser URL updates to include ?session=<sessionId> (Spec 11 hooks set this).

  • [ ] Step 4.5: Watch the discover SSE stream populate

Watch RollingLog for ~1–3 minutes. Expected event sequence (order matters):

  1. robots.txt OK
  2. sitemap-index found at <url>
  3. URL count climbs continuously (every 1000 URLs persisted as a url_shard: log entry per batch). Counter does NOT plateau at any artificial cap.
  4. pathGroups events emit ~10–20 path groups (e.g. /blog/*, /docs/*, /customers/*, /integrations/*).
  5. classified events emit per-pathGroup with content domain + confidence + method (json-ld, cms, heuristic, or llm).
  6. Final done event with sessionId.

Capture in your notes: - Total URL count seen. - Number of pathGroups. - Schema.org coverage percentage (should be ≥40% for algolia.com per §16p).

If URL count plateaus suspiciously low (< 1000 for algolia.com), the streaming walker has a cap bug: stop and fix Spec 02.

  • [ ] Step 4.6: Confirm CategoryReview tree renders correctly

Expected: - Tree shows the 9 content domains as parent nodes. - Under marketing: at least one path group from /blog/* or similar. - Under technical: at least one path group from /docs/* or similar. - Under customer-stories: at least one path group from /customers/* or similar. - Each path group shows urlCount + detected confidence + method.

If any expected domain has zero pathGroups, classification has misrouted: note which domain and which path groups were misrouted.

  • [ ] Step 4.7: Select 3 path groups (one marketing, one technical, one customer-stories)

Tick the checkbox next to ONE pathGroup under marketing, ONE under technical, ONE under customer-stories. The selected count badge should show (3).

  • [ ] Step 4.8: Click "Configure Selected (3)"

Expected: drawer opens for the FIRST selected pathGroup. Inside the drawer: 1. Within 5–15s: samples appear (5 sample URLs with HTML preview). 2. structureAnalysis renders (DSS-aware analysis of detected fields). 3. Generated extractor source code visible (this is recordExtractor.js).

If samples never load, check the sample API endpoint logs in the dev:api terminal.

  • [ ] Step 4.9: Run sandbox test (drawer 1)

Click "Run sandbox test". Expected: - Sandbox returns within ~5s. - Records list rendered. - Each record passes DSS validation (visible PASS badge or zero validation errors). - Field-coverage bars show ≥80% on required fields per the DSS row.

If validation fails, the extractor generator (Spec 09) produced wrong fields: note specifics.

  • [ ] Step 4.10: Run REAL test crawl (drawer 1)

Click "Run REAL test crawl".

WARNING: this consumes 1 of 500/24h Algolia Crawler API quota for crawl_urls. Do not click multiple times.

Expected: - Returns within ~10s. - Records returned and rendered. - Records visible in the Algolia dashboard at index algoliacentral_factory_test (filter by current parent_id if you want only this run's records).

  • [ ] Step 4.11: Commit + Create Crawler (drawer 1)

Click "Commit + Create Crawler".

Expected (each is a separate observation): 1. New domain index created (e.g. algoliacentral_marketing) if not pre-existing: verify in the Algolia dashboard. 2. Crawler appears in the Algolia Crawler dashboard. Name format: algoliacentral-www.algolia.com-<pathGroupSlug> (e.g. algoliacentral-www.algolia.com-marketing-blog). 3. New record in algoliacentral_factory_blueprints: query the index with filters: source_domain:algolia.com to confirm. 4. New file at crawler-configs/www.algolia.com/<pathGroupSlug>.js exists in the repo working tree (use ls crawler-configs/www.algolia.com/ to verify).

Capture the crawlerId, blueprint objectID, and crawler-config file path.

  • [ ] Step 4.12: Watch LiveCrawlMonitor (drawer 1)

Drawer transitions to LiveCrawlMonitor. Expected: - "Reindex started" status visible. - URL counter ticks up over ~30–60s (real crawl in progress). - No error states.

You don't need to wait for the full crawl to finish: confirm reindex is running.

  • [ ] Step 4.13: Repeat Steps 4.8–4.12 for the technical pathGroup (drawer 2)

Same observation gates. Note that the generated extractor for technical should use fields from the DSS technical row (name, articleBody, programmingLanguage).

  • [ ] Step 4.14: Repeat Steps 4.8–4.12 for the customer-stories pathGroup (drawer 3)

Same observation gates. Generated extractor for customer-stories should use headline, articleBody, about.name, quotes.

  • [ ] Step 4.15: Hard reload the page: verify session resume

Note the current ?session=<id> URL param. In the browser, hit Cmd+Shift+R (hard reload).

Expected: - Page reloads. - Session resumes from the same state: CategoryReview shows the same path groups, the 3 selected ones still selected, the 3 created crawlers visible in LiveCrawlMonitor. - No "no session found" error.

If the reload returns to empty state, the session-load hook (Spec 11) is broken: note and fix.

  • [ ] Step 4.16: Session resume observation gate (Plan §9 step 14)

Hard-reload the browser (Cmd+Shift+R). Confirm the URL still has ?session=<id> and the page renders the same LiveCrawlMonitor state with all 3 crawler blocks intact (per Steps 4.12 / 4.13 / 4.14). The session must reload from algoliacentral_factory_sessions without any data loss: DevTools Network tab should show a fresh GET against the sessions index returning the same parent_id:<sessionId> shard set verified in Step 4.17 (1 session + N url_shards + N samples + N logs). This step covers Plan §9 step 14 explicitly.

  • [ ] Step 4.17: Final dashboard inspection: sessions index

Algolia dashboard → algoliacentral_factory_sessions → Browse → filter by parent_id:<sessionId>.

Expected record types (each filterable via the record_type facet): - 1× record_type:session: main session metadata. - N× record_type:url_shard: N = ceil(totalUrls / 1000). - N× record_type:sample: at least 15 (5 samples × 3 selected pathGroups). - N× record_type:log: many narrative log entries.

Confirm each shard count is non-zero. Note actual counts.

  • [ ] Step 4.18: Final dashboard inspection: blueprints index

Algolia dashboard → algoliacentral_factory_blueprints → Browse → filter by source_domain:algolia.com AND created_at desc.

Expected: 3 newest records, one per crawler created in this smoke. Each has: - content_domain ∈ {marketing, technical, customer-stories} - index_name matches the corresponding domain (algoliacentral_marketing, etc.) - agent_slot: null (per Spec 01 schema: the future agent factory populates this; v1 does not). - status: 'live' - recordExtractor_path matches the file at crawler-configs/www.algolia.com/....

  • [ ] Step 4.19: Final dashboard inspection: content-domain indices

Algolia dashboard → confirm the 3 new content-domain indices exist and are receiving records as crawlers reindex: - algoliacentral_marketing - algoliacentral_technical - algoliacentral_customers

Each should have ≥1 record (per the real test crawl in Step 4.10 and the live reindex in Step 4.12).

  • [ ] Step 4.20: Smoke verdict

If every observation gate above passed: smoke is GREEN. Proceed to Task 5.

If ANY observation failed: 1. STOP. Do not proceed to Task 5. 2. Note which step failed and what was actually observed. 3. The failure points to a specific earlier spec: fix it there, re-bundle (Task 1), re-run smoke from the failure point. 4. Do not document or sync vault until smoke is GREEN.

Task 5: Write docs/factory/README.md

Files: - Create: docs/factory/README.md

The README is for an operator running the factory for a new site. It is NOT marketing copy. Operator-readable means: concrete commands, expected outputs, troubleshooting symptoms first.

  • [ ] Step 5.1: Create the README directory
mkdir -p docs/factory
  • [ ] Step 5.2: Write docs/factory/README.md

Create with the following content (verbatim: you may add specifics from the smoke run, but do not delete sections):

# Crawler Factory — Operator README

The Crawler Factory turns any web domain into a set of Algolia indices, one per content domain (marketing, technical, customer-stories, education, …), with one Algolia Crawler per content-domain × source-domain pair.

This README is for operators running the factory. For architecture and design rationale, see `Projects/Crawler-Factory/00-Plan.md` in the vault.

## Concept (FSM)

```
[idle]
   │  user submits URL
   ▼
[discovering]   robots.txt → sitemap-index → streaming sitemap walk
   │            URLs persisted as url_shards (1000 per shard)
   │            pathGroups emitted as found
   ▼
[categorized]   each pathGroup classified into a DSS content domain
   │            (json-ld → cms → microdata → opengraph → heuristic → llm)
   ▼
[configuring]   user selects N pathGroups, drawer per pathGroup:
   │            sample → structure-analyze → generate-extractor →
   │            sandbox-test → real-test → commit
   ▼
[crawling]      one Algolia Crawler created per committed pathGroup;
   │            reindex runs against the source domain;
   │            LiveCrawlMonitor shows progress.
   ▼
[done]          blueprint persisted; crawler-config copy in repo;
                content-domain index populated.
```

## Required environment variables

| Variable | Purpose |
|---|---|
| `ALGOLIA_APP_ID` | Algolia Central app — must be `0EXRPAXB56` in production. |
| `ALGOLIA_ADMIN_KEY` | Admin key with `editSettings`, `addObject`, `deleteIndex` ACL. |
| `ALGOLIA_CRAWLER_USER_ID` | Algolia Crawler API user id. |
| `ALGOLIA_CRAWLER_API_KEY` | Algolia Crawler API key. |
| `LLM_PROVIDER` | `minimax` (current default). See `lib/llm/` for adapter list. |
| `LLM_API_KEY` | Bearer token for the chosen provider. For MiniMax, expires periodically — refresh from Algolia inference console. |

Set in `.env.local` for dev. For Vercel, add to project env vars.

## Run locally

```bash
# Provision the two factory meta-indices (one-off; idempotent)
npx tsx scripts/factory-bootstrap.ts

# Two-process dev environment
npm run dev:api    # port 3005
npm run dev        # port 5173
```

Open `http://localhost:5173/admin/factory`.

## Resume a session

Every session has a stable id; the URL bar carries `?session=<id>`. To resume:
- After a hard reload — the page rehydrates from the same URL.
- From a different machine — copy the URL (`http://.../admin/factory?session=<id>`) and open it.
- After a Vercel 504 mid-discovery — re-submit the same URL with the same `?session=<id>` param. The streaming discoverer continues from the last persisted url_shard.

To list past sessions: query `algoliacentral_factory_sessions` with `filters: record_type:session` in the Algolia dashboard.

## Troubleshooting

### Symptom: "Submit" does nothing; no SSE events arrive
- Check the dev:api terminal for `listening on :3005`. If absent, re-run `npm run dev:api`.
- In the browser DevTools Network tab, confirm the request to `/api/factory/discover` started and stays open (SSE).
- If discover returns 401: `ALGOLIA_ADMIN_KEY` is missing or wrong.

### Symptom: URL count plateaus far below site size
- Likely cause: WAF (Cloudflare/Akamai) is blocking the sitemap fetch or returning 403.
- Check the dev:api logs for `event: walker_fetch_failed`. If 403 — the site needs a cookied / authenticated fetch path; v1 does not support this.

### Symptom: Schema.org coverage is 0%
- Either the site has no JSON-LD/microdata, or the parser failed on malformed markup.
- Cascade falls through to LLM classification (slower, costs tokens). Confirm `LLM_API_KEY` is valid.
- For sites where schema.org coverage is < 25%, expect more LLM cascade hits and higher latency on classification.

### Symptom: "Run REAL test crawl" returns 429
- Algolia Crawler API rate limit (500/24h) is exhausted.
- Wait until the quota resets (rolling 24h window), or use a different Crawler API key.

### Symptom: Crawler created but reindex never starts
- First-time crawling a new source domain may require domain verification in the Algolia Crawler dashboard. Look for a "Verify domain" CTA there.
- Once verified, re-trigger the reindex from the dashboard or by re-clicking "Commit + Create Crawler" (the index manager is idempotent).

### Symptom: Generated extractor produces zero records in sandbox
- The path group's HTML pattern doesn't match the DSS content domain it was classified into. Re-classify manually in CategoryReview, or override the generator's domain selection in the drawer.

### Symptom: `tsc --noEmit` fails after a code change
- The factory cluster invariant: tsc must stay clean. Do not commit until it passes.

## Schema.org coverage tips

| Coverage % on samples | What to expect |
|---|---|
| ≥ 60% | Cascade resolves at json-ld layer. High confidence (0.85+). LLM rarely fires. |
| 25–60% | Mixed cascade — some pathGroups via json-ld, others via heuristic/cms. Confidence 0.6–0.85. |
| < 25% | Heavy LLM fallback. Latency higher; may need manual classification overrides via CategoryReview. |

To measure coverage upfront for a candidate site (before running the factory): fetch 5 random sample pages, grep for `application/ld+json` in the source, count how many have a content-classifying `@type` (BlogPosting, TechArticle, Product, …).

## Where things land

| Artifact | Location |
|---|---|
| Session state | `algoliacentral_factory_sessions` (sharded across 4 record types) |
| Blueprints | `algoliacentral_factory_blueprints` (one per crawler created) |
| Content records | `algoliacentral_<domain>` (one index per content domain) |
| recordExtractor source | `crawler-configs/<source-domain>/<pathGroup>.js` (committed to repo) |
| Operator README | This file |
  • [ ] Step 5.3: Run the standards-writing skill against the README
# In Claude Code, invoke skill: standards-writing
# Target: docs/factory/README.md

Expected: PASS or WARN on all WritingSOPs criteria. If FAIL, address the flagged issues, regenerate, re-check.

  • [ ] Step 5.4: Commit
git add docs/factory/README.md
git commit -m "docs(factory): add operator README"

Task 6: Update vault Projects/Crawler-Factory/Status.md

Files: - Modify: ~/Library/CloudStorage/GoogleDrive-arijit.chowdhury@algolia.com/My Drive/AI-Docs/Obsidian/ArijitOS-Brain/Projects/Crawler-Factory/Status.md

The vault is iCloud-synced; no git operation is required.

  • [ ] Step 6.1: Open Status.md

Path:

~/Library/CloudStorage/GoogleDrive-arijit.chowdhury@algolia.com/My Drive/AI-Docs/Obsidian/ArijitOS-Brain/Projects/Crawler-Factory/Status.md
  • [ ] Step 6.2: Mark every cluster ✅ in the cluster status table

In the cluster status table at the top of Status.md, change every ⏸ or 🟡 (or any other in-progress/blocked marker) to ✅ for clusters 1–13.

The table after this edit must look like (cluster names may differ slightly: preserve the existing names; only update the status column):

| Cluster 1 (Foundations)        | ✅ | DSS, types, session/blueprint stores |
| Cluster 2 (Sitemap walker)     | ✅ | streaming walker, no caps |
| Cluster 3 (Path grouper)       | ✅ | URL → path-group clustering |
| Cluster 4 (Detection cascade)  | ✅ | json-ld → cms → … → llm |
| Cluster 5 (Sampler + structure)| ✅ | sample HTML, DSS-aware structure analysis |
| Cluster 6 (Index manager + crawler client) | ✅ | per-domain indices + TS crawler client |
| Cluster 7 (Extractor generator)| ✅ | per-domain recordExtractor generation |
| Cluster 8 (Sandbox + real test)| ✅ | new Function sandbox + crawl_urls real test |
| Cluster 9 (API endpoints)      | ✅ | discover, sample, …, blueprints |
| Cluster 10 (Frontend hooks)    | ✅ | useFactorySession, SSE consumers |
| Cluster 11 (UI shell + components) | ✅ | /admin/factory, RollingLog, CategoryReview |
| Cluster 12 (Configurator + monitor) | ✅ | drawer flow + LiveCrawlMonitor |
| Cluster 13 (Bundle + smoke + docs) | ✅ | this spec — final verification |
  • [ ] Step 6.3: Update the phase line at the top of the document

Find the Phase: (or equivalent) line near the top. Change to:

Phase: v1 spike shipped

If no such line exists, add it directly under the H1.

  • [ ] Step 6.4: Add a "Lessons learned" section at the bottom of Status.md

Append (or fill in if the section already exists):

## Lessons learned (v1 spike)

Capture deviations from the plan as data, not apology. Each entry: what the plan said, what actually happened, what we'd change next time.

- **Schema.org coverage on algolia.com:** plan assumed ≥60%; actual was <fill from smoke run>%. <impact on cascade fallback rate>.
- **URL count for algolia.com:** plan assumed ~10k–20k; actual was <fill>. <impact on shard count>.
- **PathGroup count:** plan assumed ~10–20; actual was <fill>. <impact on UX scrolling>.
- **Extractor generation latency:** plan assumed 5–15s per drawer; actual was <fill>. <if outside band, why>.
- **Real-test crawl quota burn:** consumed <fill> of 500/24h during smoke. <if higher than expected, why>.
- **Any spec that needed a hotfix during smoke:** <list spec number + what was wrong + how it was fixed>. If none — say "none".
- **Anything we'd cut from v2:** <list>.
- **Anything v2 must add immediately:** <list>.

Fill in the placeholders from your smoke-run notes (Steps 4.5, 4.16, 4.17). Do NOT leave <fill> markers in the final file. If a value is genuinely unknown, write "not measured" rather than a placeholder.

  • [ ] Step 6.5: Save the file

iCloud sync handles the rest. No git commit needed in the vault.

  • [ ] Step 6.6: Verify the file rendered correctly
ls -la "$HOME/Library/CloudStorage/GoogleDrive-arijit.chowdhury@algolia.com/My Drive/AI-Docs/Obsidian/ArijitOS-Brain/Projects/Crawler-Factory/Status.md"

Expected: file exists, modified time is within the last few minutes.

Task 7: Update repo SESSION.md

Files: - Modify: SESSION.md (in the repo root)

  • [ ] Step 7.1: Open SESSION.md

Path: /Users/arijitchowdhury/AI-Development/RAG/AlgoliaRAG-Google/rc3-phoenix/SESSION.md

  • [ ] Step 7.2: Update the "Current task" section

Replace whatever is in the active-task block with:

## Current task

**Crawler Factory v1 — COMPLETE.** Spike shipped on <YYYY-MM-DD per smoke run>.

- All 13 clusters merged on `rc3-phoenix`.
- Smoke against `https://www.algolia.com` produced 3 working crawlers, 3 distinct content-domain indices populating, 3 blueprints persisted, session resumable.
- Operator README at `docs/factory/README.md`.
- Vault Status.md marked v1 spike shipped.

## Resume action

The next active task is TBD by the user. Common candidates:
- Crawler Factory v1.5 (multi-tenant orchestration — see `00-Plan.md` §17).
- Specialist-agent factory (consume blueprints — see `00-Plan.md` §15).
- Extending DSS to vertical sub-schemas (see `00-Plan.md` §16o re: furniture).
- Wire the live POST to Agent Studio for the Maverick lift (per memory `session_pointer`).

User picks. Until then, no active task.

Fill in the actual smoke-run date.

  • [ ] Step 7.3: Update the "Last action" / footer block

Whatever the existing footer convention is in the file, append a line of the form:

**Last action (Spec 13):** Bundled api/factory/*.mjs (9 endpoints), provisioned factory meta-indices, smoke against algolia.com GREEN (3 crawlers, 3 blueprints, session resumable), wrote docs/factory/README.md, synced vault Status.md.
  • [ ] Step 7.4: Commit
git add SESSION.md
git commit -m "chore(session): mark Crawler Factory v1 complete"
  • [ ] Step 7.5: Final repo state check
git status && git log --oneline -10

Expected: - Working tree clean. - Recent commits include (in order): - chore(factory): bundle factory API endpoints (Task 1) - chore(factory): bootstrap factory meta-indices (Task 3) - docs(factory): add operator README (Task 5) - chore(session): mark Crawler Factory v1 complete (Task 7) - All on branch rc3-phoenix.

Acceptance criteria: Spec 13 done means:

  1. ✅ Task 1: node scripts/bundle-api.mjs produced all 9 api/factory/*.mjs bundles; each node --checks clean; bundles committed.
  2. ✅ Task 2: full npx vitest run is GREEN (407 baseline + every factory test added by Specs 01–12); npx tsc --noEmit is clean.
  3. ✅ Task 3: scripts/factory-bootstrap.ts exists, ran successfully, both meta-indices visible in the Algolia dashboard with correct attributesForFaceting (filterOnly(record_type), filterOnly(parent_id) on sessions; filterOnly(content_domain), filterOnly(status) on blueprints).
  4. ✅ Task 4: live smoke against https://www.algolia.com GREEN: 3 distinct content-domain crawlers created (one each for marketing, technical, customer-stories), 3 blueprints persisted with agent_slot: null, session resumes after hard reload, 3 content-domain indices receiving records.
  5. ✅ Task 5: docs/factory/README.md written with FSM diagram, env vars, resume mechanics, troubleshooting, schema.org coverage tips; passes standards-writing skill at WARN-or-better; committed.
  6. ✅ Task 6: vault Status.md shows every cluster ✅, phase line reads v1 spike shipped, "Lessons learned" section populated with actual smoke-run data (no placeholders left).
  7. ✅ Task 7: repo SESSION.md marks Crawler Factory v1 complete, names the new active task as TBD by user, "Last action" footer updated; committed.
  8. ✅ Branch is rc3-phoenix throughout. No commits to main or rc2-algolia.

Out of scope (explicitly NOT done by this spec)

  • Pushing rc3-phoenix to remote / Vercel deployment promotion. The user authorizes pushes; this spec stops at local commits.
  • v1.5 multi-tenant orchestration (LVMH-style brand federation): 00-Plan.md §17.
  • Specialist-agent factory that consumes blueprints: 00-Plan.md §15.
  • v2 vertical sub-schemas (furniture, apparel, electronics): 00-Plan.md §16o.
  • Bulk multi-domain crawl orchestration: 00-Plan.md §17c (v2 deferred).
  • Cross-tenant search routing for an orchestrator: 00-Plan.md §17c.
  • Performance / load testing of the factory itself under multi-session concurrency.
  • Hardening the LLM cascade against prompt injection in scraped HTML (research item; not in v1 scope).