Core/02-DSS.md
Domain Schema Standard (DSS) — Reference
Extracted from
00-Plan.md§4. The DSS is our internal type system mapping content domains to schema.org types, record schemas, indices, and Algolia config tuning.
Source of truth: 00-Plan.md §4.
4. Data model — Domain Schema Standard (DSS)
The fundamental shift: instead of a single enterprise_ledger record type writing to a single index, we have content domains. Each content domain:
- Maps to one or more schema.org types (e.g., marketing → BlogPosting, NewsArticle, OpinionArticle)
- Has its own record schema (different fields per domain — a recipe has cookTime, a course has learningResourceType, a support article has lastReviewed)
- Writes to its own index with config tuned for that domain (different searchableAttributes, customRanking, attributesForFaceting)
- Is the data foundation for a future specialist agent (support-agent on support index, etc.)
4a. The Domain Schema Standard (DSS) — the core taxonomy
DSS is our internal type system. Each row maps a content domain to: schema.org parent types, record schema, index name, and Algolia config tuning. DSS is data, not code — it lives in lib/factory/dss.ts as a typed registry, easy to extend.
| Content domain | Index name | schema.org types in scope | Distinctive record fields | Algolia config tuning |
|---|---|---|---|---|
| marketing | algoliacentral_marketing |
BlogPosting, NewsArticle, OpinionArticle, Article, AnnouncementPage, SpecialAnnouncement |
headline, articleBody, datePublished, author, articleSection, keywords |
searchable: [headline, articleBody, keywords]; customRanking: [desc(datePublished), desc(view_count)]; facets: [articleSection, author, year] |
| support | algoliacentral_support |
TechArticle (troubleshooting), FAQPage, Question, Answer, HowTo (fix-it) |
name, articleBody, lastReviewed, acceptedAnswer, severity, productAffected, supportLevel* |
searchable: [name, articleBody, acceptedAnswer]; customRanking: [desc(lastReviewed), desc(helpful_count)]; facets: [productAffected, severity, status] |
| education | algoliacentral_education |
Course, LearningResource, HowTo, Guide, VideoObject (educational) |
name, description, learningResourceType, educationalLevel, timeRequired, hasCourseInstance, transcript (video) |
searchable: [name, description, articleBody, transcript]; customRanking: [desc(datePublished), desc(enrollment_count)]; facets: [educationalLevel, learningResourceType, language, duration] |
| technical | algoliacentral_technical |
TechArticle, APIReference*, SoftwareSourceCode, SoftwareApplication, WebAPI |
name, articleBody, programmingLanguage, version, applicationCategory, codeSampleType, apiEndpoint* |
searchable: [name, articleBody, programmingLanguage, codeSampleType]; customRanking: [desc(version), desc(view_count)]; facets: [programmingLanguage, framework, version, apiEndpoint] |
| customer-stories | algoliacentral_customers |
Article (case study), Review, Person, Organization |
headline, articleBody, about (Organization name), industry, companySize, region, metrics, quotes |
searchable: [headline, articleBody, about.name, quotes]; customRanking: [desc(datePublished), desc(prominence)]; facets: [industry, companySize, region, useCase] |
| product-catalog | algoliacentral_products |
Product, Offer, Service, Brand |
name, description, sku, offers, brand, category, aggregateRating, image |
searchable: [name, description, brand]; customRanking: [desc(aggregateRating), desc(popularity)]; facets: [brand, category, priceRange, rating] |
| events | algoliacentral_events |
Event, EducationEvent, BusinessEvent, Conference |
name, startDate, endDate, location, organizer, eventType, recordingUrl |
searchable: [name, description]; customRanking: [asc(startDate)] (upcoming first); facets: [eventType, location, year] |
| legal | algoliacentral_legal |
DigitalDocument, Legislation, TermsOfService, PrivacyPolicy |
name, text, version, datePublished, jurisdiction, documentType* |
searchable: [name, text]; customRanking: [desc(version), desc(datePublished)]; facets: [jurisdiction, documentType] |
| social | algoliacentral_social |
VideoObject (YouTube), Comment, Conversation, SocialMediaPosting* |
text, transcript, uploadDate, channel, viewCount, commentCount, platform, sentiment |
searchable: [text, transcript]; customRanking: [desc(uploadDate), desc(engagement)]; facets: [platform, channel, sentiment] |
Asterisked fields are not in schema.org — they're our extensions, defined in DSS as dss: namespace fields.
The DSS is extensible at runtime: a user (or future automation) can add a new content domain by adding a new entry to the registry. No code changes needed if the new domain reuses an existing schema.org type set.
4b. Detection cascade — REVISED based on Web Almanac 2024 + 14-site empirical audit
Cascade order is CMS-first, not schema.org-first. Schema.org is a confirmer/refiner, not the foundation. See §3b for the empirical justification and §14 for the source data.
| # | Method | What it inspects | Expected hit rate | Confidence when hit |
|---|---|---|---|---|
| 1 | CMS fingerprint | URL paths (/wp-content/, /content/dam/, /sites/default/files/), HTML class signatures (wp-block-*, region-*, data-cmp-*, data-aura-*), robots.txt patterns (Drupal's Crawl-delay: 10), known JSON endpoints (/wp-json/wp/v2/posts, Shopify product API) |
51% of all sites | 0.80–0.90 |
| 2 | URL pattern | /blog/, /customers/, /docs/, /help/, /support/, /courses/, /products/, /{YYYY}/{MM}/, etc. (full table in §3b) |
100% (always available) | 0.50–0.75 |
| 3 | JSON-LD parser | <script type="application/ld+json"> blocks. Walks @type chain. Matches against DSS schema.org types. |
41% of pages globally; ~7% with content-classifying types | 0.85–0.95 |
| 4 | OpenGraph + Twitter Card | og:type (article/website/video/product), og:article:section, twitter:card |
64% of pages | 0.55–0.70 |
| 5 | Microdata + RDFa | itemtype="https://schema.org/X"; RDFa typeof |
26% / 66% — declining | 0.40–0.65 |
| 6 | Date + author regex | "Published: YYYY-MM-DD", "by Author Name", <time datetime=...> |
99% of articles | 0.40–0.60 (article-family only) |
| 7 | Semantic HTML heuristics | <article>, <main>, <header>, breadcrumbs, code-block density, table-of-contents shape |
~5–8% (low) | 0.45–0.60 |
| 8 | LLM classification | Cheerio-extracted text (first 4KB) + DSS table → "which content domain best fits?" Cheap; ~$0.001 per call. | 100% (always available) | 0.50–0.85 |
Aggregation rules:
- Each layer outputs {contentDomain, confidence} or null.
- Final confidence = max-confidence layer + boost when multiple layers AGREE on the same domain (each agreement adds 0.05, capped at 0.95).
- LLM (layer 8) only fires when combined confidence from layers 1–7 is < 0.55.
- UI shows the contributing layers per pathGroup so the user can see WHY a classification was made and override if needed.
Confidence thresholds: - ≥ 0.85: green — auto-accept, recommend "Configure" - 0.65–0.84: yellow — show confidence, recommend manual review - < 0.65: amber — flag for manual override; default action is LLM tiebreak + manual confirmation
4c. Record schemas — one zod schema per content domain
Each content domain has its own zod schema. Common base + domain-specific extension:
┌──────────────────────────────────────────────────────────────────────────┐
│ Outer dashboard chrome │
│ ┌─Algolia mark + product rail (left) │
│ ├─Section sub-nav (e.g. Data sources → Crawler) │
│ └─Top bar (app picker, search, Upgrade, AI Assist, avatar) │
│ ── Main content: Lists, summaries, dashboards across assets ── │
└──────────────────────────────────────────────────────────────────────────┘
User clicks an asset (e.g. a crawler row in the list)
↓
┌──────────────────────────────────────────────────────────────────────────┐
│ Focused workspace chrome (replaces outer chrome) │
│ ┌─Module wordmark (top-left) ┌─Persistent CTA + close X (top-right)│
│ ├─Scope indicator (top-center) │ │
│ └─Vertical sectioned nav (left) └─Tabs / page content (right) │
└──────────────────────────────────────────────────────────────────────────┘
The legacy CrawlerRecord (packages/crawl/src/crawl/models.py) is kept as-is for the existing algolia-central_enterprise_ledger index — it's not deleted. New crawlers from the factory write to per-domain indices using the new schemas. Migration is opt-in.
4d. Sharded session storage (no record-size cap)
A single Algolia record has a 100KB limit. For sites with 100K+ URLs, one giant session record won't fit. Shard:
SETUP ▼ ← create / configure the asset
Overview
Configuration
Editor (advanced)
Suggestions
STATUS ▼ ← what is happening / what happened
URL inspector
Monitoring
Data Analysis
Path Explorer
Logs Explorer
CONFIGURATION ▼ ← admin / metadata / danger
External Data
Settings
attributesForFaceting: [filterOnly(record_type), filterOnly(parent_id)]. Reads filter on parent_id:<sessionId> then split by record_type.
The factory streams URLs into shards as discovery proceeds — no waiting for full discovery, no memory pressure.
4e. Factory blueprints — agent-ready metadata
Every successfully created crawler also writes a blueprint to a separate index. This is the metadata a future orchestrator/specialist-agent scaffolder will read.
<H1 page title> [optional secondary control]
<optional 1-line description>
[optional tabs (bottom-border underline, blue active)] [optional badge counts]
────────────────────────────────────────────────────────────────────────
<page body — usually a vertical stack of cards or one big table>
This index is the link between the crawl world and the agent world. When we build the orchestrator + specialist agents (future spike), it reads from algoliacentral_factory_blueprints to know what specialists exist and what domains they cover.
4f. Session main record (zod, slimmed for 100KB compliance)
┌──────────────────────────────────────────────────────────────────────┐
│ [icon-tile] Setting name <value> [Edit] │
│ One-line description, muted. │
└──────────────────────────────────────────────────────────────────────┘
4g. API contract (revised)
┌─────────────── Add Markdown Index ──────────────[×]┐
│ │
│ Index Name * │
│ [ Enter index name (e.g., markdown-index) ] │
│ │
│ Content Tag * │
│ [ main ] │
│ │
│ Template * │
│ [ Non-DocSearch (Generic) ▾ ] │
│ Choose the template that matches… │
│ │
│ [ Cancel ] [ Add Index ] │
└────────────────────────────────────────────────────┘