Dashboard/Crawler-Factory/Engineering-Specs/07-brand-discovery.md
Crawler-Factory

Engineering-Specs/07-brand-discovery.md

Crawler Factory: Spec 07: Brand Domain Discovery Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: When the factory detects a multi-brand parent corporate site (LVMH, Diageo, Unilever, P&G), scrape its brand-index page (/our-brands, /portfolio, /companies, /family-of-brands) to discover external brand domains so the user can launch a separate factory session for each. v1 delivers detection + enumeration only: multi-tenant index management and parallel brand crawls are deferred.

Architecture: Single pure module: lib/factory/brand-domain-discoverer.ts: composed of three exported functions: findBrandIndexPage (link heuristic on the parent homepage), extractBrandDomains (cheerio-based outbound-link extraction with social-media filtering), and discoverBrands (orchestrator that consumes the Spec 03 sampler and Spec 04 classifier, only proceeding when site-type-rubric returns multi-brand-corporate). The module is read-only: it surfaces brand candidates; it does NOT create sessions, write blueprints, or manage tenants. Those flows live in Spec 08 (API endpoints) and a future Spec extension covering §17 multi-tenant orchestration.

Tech Stack: TypeScript (strict), zod 3.25, cheerio (already in deps from Spec 01), vitest, pino logger via lib/utils/logger.ts.

Depends on: - Spec 01: types (ContentDomainEnum, logger conventions). - Spec 03: lib/factory/sampler.ts exports sampleUrls(urls, opts) returning Array<{ url, html, status, contentLength, isSpa, headers }>. We consume the html field of a single-URL sample. - Spec 04: lib/factory/classifier.ts and lib/factory/site-type-rubric.ts. Spec 04 exposes detectSiteType(input): SiteType: a synchronous predicate ladder that returns the bare site-type token (no { siteType, confidence } envelope). The value 'multi-brand-corporate' is the §19d row that gates Spec 07.

Consumed by: - Spec 08: backend /api/factory/discover endpoint calls discoverBrands(rootUrl, sampler, classifier) after the parent-site discovery completes; returns the BrandRef[] to the frontend so the UI can render the "Detected N external brand domains" CTA.

Plan source: Projects/Crawler-Factory/00-Plan.md §16k (Multi-brand federated), §17 (Multi-tenant orchestration v1.5: context only; v1 only enumerates), §14f (LVMH/Diageo/Unilever/P&G empirical findings), §19d (site-type rubric: multi-brand-corporate row).

File structure

Path Responsibility
lib/factory/brand-domain-discoverer.ts Brand-index page locator + outbound-domain extractor + discoverBrands orchestrator. Three exported functions, one zod-validated BrandRef schema, social-media domain blocklist constant.
lib/factory/__tests__/brand-domain-discoverer.test.ts Unit tests with mocked sampler + classifier. Realistic Diageo HTML fixture for the happy path; non-multi-brand site fixture for the negative path; social-media-link fixture for the filter; missing-brand-index fixture for the empty path.

No other files are touched. The module is pure-data-in, pure-data-out. No I/O outside of what the injected sampler does.

SOP rules applied (non-negotiable; from CodingSOPs.md, TestingSOPs.md, WritingSOPs.md)

Every file in this spec MUST follow these rules. They are not optional.

  1. Two-layer type safety. zod for runtime boundary validation (BrandRefSchema validates every brand returned to callers); tsc --noEmit strict for write-time. No any. Use T | null, not T | undefined, when absence is meaningful (e.g. findBrandIndexPage returns string | null).
  2. Logger in every module. Top of brand-domain-discoverer.ts: import { createLogger } from '../utils/logger'; and const log = createLogger('factory:brand-discoverer');. No console.log. Format: log.info({ event: 'name', ...kvs }, 'short_msg').
  3. Try/catch every fallible call. Cheerio parsing is wrapped (cheerio throws on malformed HTML in some edge paths). The injected sampler call is wrapped. Catch specific error first, generic last. Log with full context before rethrowing.
  4. Functions ≤20 lines, ≤3 params. discoverBrands takes (rootUrl, sampler, classifier): exactly 3, all typed. Helpers stay short via early-return / pure-list-comprehension patterns.
  5. Docstring on every exported symbol. Single line explaining purpose. Comment WHY, not WHAT.
  6. No raw dicts crossing module boundaries. BrandRef and DiscoverBrandsResult are zod schemas; every public return is validated through them.
  7. TDD cadence. Test first → run, expect FAIL → implement → run, expect PASS → commit.
  8. Mock external services in unit tests. The sampler and classifier are passed as function parameters precisely so tests can inject vi.fn() mocks. No real fetch in unit tests.
  9. Naming. TypeScript: camelCase for vars/functions, PascalCase for types/classes, UPPER_SNAKE_CASE for module constants. Files: kebab-case.ts. Tests: same name with .test.ts.
  10. Conventional commits. feat(factory):, test(factory):, chore(factory):. One logical change per commit.
  11. Render-path-first / data-truthful logging. Every log entry references the rootUrl so multi-tenant grep stays clean.

Task 1: BrandRef zod schema + social-media blocklist + module skeleton

Files: - Create: lib/factory/brand-domain-discoverer.ts - Create: lib/factory/__tests__/brand-domain-discoverer.test.ts

This task lays down the zod-validated boundary types and the module's logger so subsequent tasks can import them.

  • [ ] Step 1.1: Write the failing test for BrandRefSchema

Create lib/factory/__tests__/brand-domain-discoverer.test.ts:

import { describe, it, expect } from 'vitest';
import { BrandRefSchema, SOCIAL_MEDIA_DOMAINS } from '../brand-domain-discoverer';

describe('BrandRefSchema', () => {
  it('accepts a minimal valid brand reference', () => {
    const valid = {
      name: 'Louis Vuitton',
      domain: 'louisvuitton.com',
      indexedFromUrl: 'https://www.lvmh.com/houses/fashion-leather-goods/louis-vuitton/',
    };
    expect(() => BrandRefSchema.parse(valid)).not.toThrow();
  });

  it('rejects a brand reference with an empty name', () => {
    const invalid = {
      name: '',
      domain: 'louisvuitton.com',
      indexedFromUrl: 'https://www.lvmh.com/houses/fashion-leather-goods/louis-vuitton/',
    };
    expect(() => BrandRefSchema.parse(invalid)).toThrow();
  });

  it('rejects a brand reference with an empty domain', () => {
    const invalid = {
      name: 'Louis Vuitton',
      domain: '',
      indexedFromUrl: 'https://www.lvmh.com/houses/fashion-leather-goods/louis-vuitton/',
    };
    expect(() => BrandRefSchema.parse(invalid)).toThrow();
  });

  it('rejects a brand reference with a non-URL indexedFromUrl', () => {
    const invalid = {
      name: 'Louis Vuitton',
      domain: 'louisvuitton.com',
      indexedFromUrl: 'not-a-url',
    };
    expect(() => BrandRefSchema.parse(invalid)).toThrow();
  });
});

describe('SOCIAL_MEDIA_DOMAINS constant', () => {
  it('includes the canonical social platforms we filter out', () => {
    expect(SOCIAL_MEDIA_DOMAINS).toContain('twitter.com');
    expect(SOCIAL_MEDIA_DOMAINS).toContain('linkedin.com');
    expect(SOCIAL_MEDIA_DOMAINS).toContain('instagram.com');
    expect(SOCIAL_MEDIA_DOMAINS).toContain('facebook.com');
    expect(SOCIAL_MEDIA_DOMAINS).toContain('youtube.com');
  });
});
  • [ ] Step 1.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/brand-domain-discoverer.test.ts

Expected: FAIL: Cannot find module '../brand-domain-discoverer'.

  • [ ] Step 1.3: Implement the module skeleton with BrandRefSchema and the blocklist

Create lib/factory/brand-domain-discoverer.ts:

/**
 * Brand-domain discoverer — for multi-brand corporate parent sites
 * (LVMH, Diageo, Unilever, P&G). Locates the parent's brand-index page,
 * extracts outbound brand domains, and surfaces them to the caller so the
 * user can launch a separate factory session per brand.
 *
 * Why: per 00-Plan.md §16k, multi-brand corporates are FEDERATED — the
 * parent sitemap does NOT recursively cover child brand URLs. Without this
 * sub-flow, the factory would index the parent's marketing/legal pages
 * and silently miss every child brand domain. v1 enumerates only; the user
 * triggers each brand's discovery manually. Multi-tenant index management
 * is deferred to a later spec (§17).
 */

import { z } from 'zod';
import { createLogger } from '../utils/logger';

const log = createLogger('factory:brand-discoverer');

// ─── Constants ──────────────────────────────────────────────────────────────

/**
 * Canonical brand-index path candidates. Ordered by empirical frequency
 * across the §14f corporate sample (Diageo uses /our-brands; LVMH uses
 * /houses; Unilever uses /brands; P&G uses /brands; "portfolio" and
 * "companies" and "family-of-brands" appear in long-tail corporates).
 *
 * Stored as suffix substrings so href matching tolerates trailing slashes,
 * locale prefixes (`/en/our-brands`), and query strings.
 */
export const BRAND_INDEX_PATH_CANDIDATES: readonly string[] = Object.freeze([
  '/our-brands',
  '/brands',
  '/portfolio',
  '/companies',
  '/family-of-brands',
  '/houses',
]);

/**
 * Social-media domains we never surface as "brand domains" even when the
 * parent's brand-index page links to them (corporate footers always do).
 * Stored without scheme; matched via endsWith on the parsed hostname.
 */
export const SOCIAL_MEDIA_DOMAINS: readonly string[] = Object.freeze([
  'twitter.com',
  'x.com',
  'linkedin.com',
  'instagram.com',
  'facebook.com',
  'youtube.com',
  'tiktok.com',
  'pinterest.com',
  'snapchat.com',
  'threads.net',
  'reddit.com',
]);

// ─── Schemas ────────────────────────────────────────────────────────────────

/**
 * One detected brand. `name` is the link text (best-effort). `domain` is the
 * apex domain of the outbound link. `indexedFromUrl` is the URL we found it on.
 */
export const BrandRefSchema = z.object({
  name: z.string().min(1),
  domain: z.string().min(1),
  indexedFromUrl: z.string().url(),
});
export type BrandRef = z.infer<typeof BrandRefSchema>;

/**
 * Result of `discoverBrands`. `detected=false` means the site is NOT a
 * multi-brand corporate (or no brand-index page was found). When `detected`
 * is false, `brandIndexUrl` is null and `brands` is empty.
 */
export const DiscoverBrandsResultSchema = z.object({
  detected: z.boolean(),
  brandIndexUrl: z.string().url().nullable(),
  brands: z.array(BrandRefSchema),
});
export type DiscoverBrandsResult = z.infer<typeof DiscoverBrandsResultSchema>;
  • [ ] Step 1.4: Run, expect PASS for schema + blocklist tests
npx vitest run lib/factory/__tests__/brand-domain-discoverer.test.ts

Expected: 5 tests pass.

  • [ ] Step 1.5: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 1.6: Commit
git add lib/factory/brand-domain-discoverer.ts lib/factory/__tests__/brand-domain-discoverer.test.ts
git commit -m "feat(factory): brand-discoverer schema + social-media blocklist"

Task 2: findBrandIndexPage(rootUrl, sample)

Files: - Modify: lib/factory/brand-domain-discoverer.ts - Modify: lib/factory/__tests__/brand-domain-discoverer.test.ts

findBrandIndexPage consumes a homepage HTML sample, scans <a href> links, and returns the first absolute URL whose path ends with one of BRAND_INDEX_PATH_CANDIDATES. Returns null if no candidate matches. The function does NOT fetch the brand-index page itself: it only locates the URL.

The sample parameter is the shape returned by Spec 03's sampleUrls (a single element of that array): { url: string; html: string; status: number; ... }. We accept the smallest viable subset to keep the dependency loose.

  • [ ] Step 2.1: Write failing tests for findBrandIndexPage

Append to lib/factory/__tests__/brand-domain-discoverer.test.ts:

import { findBrandIndexPage } from '../brand-domain-discoverer';

describe('findBrandIndexPage', () => {
  it('finds /our-brands when the homepage links to it', () => {
    const sample = {
      url: 'https://www.diageo.com/',
      html: `
        <html><body>
          <nav>
            <a href="/about">About</a>
            <a href="/our-brands">Our Brands</a>
            <a href="/news">News</a>
          </nav>
        </body></html>
      `,
      status: 200,
    };
    const result = findBrandIndexPage('https://www.diageo.com', sample);
    expect(result).toBe('https://www.diageo.com/our-brands');
  });

  it('finds /portfolio (alternate canonical path)', () => {
    const sample = {
      url: 'https://www.example-corp.com/',
      html: `<a href="https://www.example-corp.com/portfolio">Portfolio</a>`,
      status: 200,
    };
    const result = findBrandIndexPage('https://www.example-corp.com', sample);
    expect(result).toBe('https://www.example-corp.com/portfolio');
  });

  it('finds /houses (LVMH-style canonical path)', () => {
    const sample = {
      url: 'https://www.lvmh.com/',
      html: `<a href="/houses/">Maisons</a>`,
      status: 200,
    };
    const result = findBrandIndexPage('https://www.lvmh.com', sample);
    expect(result).toBe('https://www.lvmh.com/houses/');
  });

  it('returns null when no candidate path is linked from the homepage', () => {
    const sample = {
      url: 'https://www.example.com/',
      html: `<a href="/about">About</a><a href="/contact">Contact</a>`,
      status: 200,
    };
    const result = findBrandIndexPage('https://www.example.com', sample);
    expect(result).toBeNull();
  });

  it('returns null on empty HTML', () => {
    const sample = { url: 'https://www.example.com/', html: '', status: 200 };
    const result = findBrandIndexPage('https://www.example.com', sample);
    expect(result).toBeNull();
  });

  it('tolerates relative hrefs and resolves against rootUrl', () => {
    const sample = {
      url: 'https://www.diageo.com/en/',
      html: `<a href="our-brands/">Our Brands</a>`,
      status: 200,
    };
    const result = findBrandIndexPage('https://www.diageo.com', sample);
    // Resolved against rootUrl + relative -> absolute
    expect(result).toMatch(/^https:\/\/www\.diageo\.com.*our-brands/);
  });
});
  • [ ] Step 2.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/brand-domain-discoverer.test.ts -t findBrandIndexPage

Expected: FAIL: findBrandIndexPage is not exported.

  • [ ] Step 2.3: Implement findBrandIndexPage

Append to lib/factory/brand-domain-discoverer.ts:

import * as cheerio from 'cheerio';

/**
 * Minimal sampler-output shape we consume. Compatible with the full
 * Spec 03 sampler return type (we ignore extra fields).
 */
export interface BrandSample {
  url: string;
  html: string;
  status: number;
}

/**
 * Locate the brand-index page on a multi-brand parent site by scanning
 * the parent's homepage HTML for outbound links matching one of
 * BRAND_INDEX_PATH_CANDIDATES. Returns an absolute URL or null.
 *
 * Why a heuristic and not a fixed path: §14f shows the canonical path
 * differs across LVMH (/houses), Diageo (/our-brands), Unilever (/brands),
 * P&G (/brands), and long-tail corporates (/portfolio, /companies).
 */
export function findBrandIndexPage(rootUrl: string, sample: BrandSample): string | null {
  if (!sample.html) {
    log.info({ event: 'findBrandIndex_empty_html', rootUrl }, 'empty html — skipping');
    return null;
  }
  let $: cheerio.CheerioAPI;
  try {
    $ = cheerio.load(sample.html);
  } catch (err) {
    log.error({ event: 'findBrandIndex_cheerio_failed', rootUrl, err: String(err) }, 'cheerio parse failed');
    return null;
  }
  const hrefs = $('a[href]')
    .map((_, el) => $(el).attr('href'))
    .get()
    .filter((h): h is string => typeof h === 'string' && h.length > 0);
  for (const href of hrefs) {
    const absolute = resolveHref(href, rootUrl);
    if (!absolute) continue;
    if (matchesBrandIndexPath(absolute)) {
      log.info({ event: 'findBrandIndex_hit', rootUrl, brandIndexUrl: absolute });
      return absolute;
    }
  }
  log.info({ event: 'findBrandIndex_miss', rootUrl, scanned: hrefs.length }, 'no brand-index candidate found');
  return null;
}

/**
 * Resolve an href (which may be relative, absolute, or scheme-relative)
 * against rootUrl. Returns null if the URL constructor rejects it.
 */
function resolveHref(href: string, rootUrl: string): string | null {
  try {
    return new URL(href, rootUrl).toString();
  } catch {
    return null;
  }
}

/**
 * True if the absolute URL's pathname ends with (or matches with optional
 * trailing slash) one of the canonical brand-index path candidates.
 */
function matchesBrandIndexPath(absoluteUrl: string): boolean {
  let parsed: URL;
  try {
    parsed = new URL(absoluteUrl);
  } catch {
    return false;
  }
  const path = parsed.pathname.replace(/\/+$/, ''); // strip trailing slashes
  return BRAND_INDEX_PATH_CANDIDATES.some((candidate) => path.endsWith(candidate));
}
  • [ ] Step 2.4: Run findBrandIndexPage tests, expect PASS
npx vitest run lib/factory/__tests__/brand-domain-discoverer.test.ts -t findBrandIndexPage

Expected: 6 tests pass.

  • [ ] Step 2.5: Run tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 2.6: Commit
git add lib/factory/brand-domain-discoverer.ts lib/factory/__tests__/brand-domain-discoverer.test.ts
git commit -m "feat(factory): findBrandIndexPage heuristic over canonical brand-index paths"

Task 3: extractBrandDomains(brandIndexHtml, rootDomain)

Files: - Modify: lib/factory/brand-domain-discoverer.ts - Modify: lib/factory/__tests__/brand-domain-discoverer.test.ts

extractBrandDomains parses brand-index page HTML via cheerio, walks every <a href>, keeps only outbound absolute links whose registrable domain differs from rootDomain, filters out social-media domains, deduplicates by domain, and returns BrandRef[].

The name field is best-effort: it uses the trimmed link text; if empty (icon-only links), we fall back to the host's first path segment as a placeholder so downstream UI has something to render. zod validation enforces name.min(1) so we never return blank-named brands.

  • [ ] Step 3.1: Write failing tests for extractBrandDomains

Append to lib/factory/__tests__/brand-domain-discoverer.test.ts:

import { extractBrandDomains } from '../brand-domain-discoverer';

describe('extractBrandDomains', () => {
  const indexedFromUrl = 'https://www.diageo.com/our-brands';

  it('extracts outbound brand domains that differ from rootDomain', () => {
    const html = `
      <html><body>
        <a href="https://www.johnniewalker.com/">Johnnie Walker</a>
        <a href="https://www.guinness.com/">Guinness</a>
        <a href="https://www.diageo.com/our-brands">Internal</a>
        <a href="https://www.diageo.com/news">News</a>
      </body></html>
    `;
    const result = extractBrandDomains(html, 'diageo.com', indexedFromUrl);
    expect(result).toHaveLength(2);
    expect(result.map((b) => b.domain).sort()).toEqual(['guinness.com', 'johnniewalker.com']);
    const johnnie = result.find((b) => b.domain === 'johnniewalker.com')!;
    expect(johnnie.name).toBe('Johnnie Walker');
    expect(johnnie.indexedFromUrl).toBe(indexedFromUrl);
  });

  it('filters out social-media domains', () => {
    const html = `
      <html><body>
        <a href="https://www.johnniewalker.com/">Johnnie Walker</a>
        <a href="https://twitter.com/diageo">Twitter</a>
        <a href="https://www.linkedin.com/company/diageo">LinkedIn</a>
        <a href="https://www.instagram.com/diageo/">Instagram</a>
        <a href="https://www.facebook.com/diageo">Facebook</a>
        <a href="https://www.youtube.com/diageo">YouTube</a>
      </body></html>
    `;
    const result = extractBrandDomains(html, 'diageo.com', indexedFromUrl);
    expect(result).toHaveLength(1);
    expect(result[0].domain).toBe('johnniewalker.com');
  });

  it('deduplicates brand domains (first occurrence wins for name)', () => {
    const html = `
      <html><body>
        <a href="https://www.guinness.com/en">Guinness</a>
        <a href="https://www.guinness.com/products">Guinness Products</a>
        <a href="https://www.guinness.com/about">About Guinness</a>
      </body></html>
    `;
    const result = extractBrandDomains(html, 'diageo.com', indexedFromUrl);
    expect(result).toHaveLength(1);
    expect(result[0].domain).toBe('guinness.com');
    expect(result[0].name).toBe('Guinness');
  });

  it('treats www and apex as the same domain (apex wins)', () => {
    const html = `
      <html><body>
        <a href="https://www.guinness.com/">Guinness</a>
        <a href="https://guinness.com/about">Guinness Apex</a>
      </body></html>
    `;
    const result = extractBrandDomains(html, 'diageo.com', indexedFromUrl);
    expect(result).toHaveLength(1);
    expect(result[0].domain).toBe('guinness.com');
  });

  it('skips links with no http(s) scheme (mailto, tel, javascript)', () => {
    const html = `
      <html><body>
        <a href="mailto:hi@diageo.com">Email</a>
        <a href="tel:+1234">Call</a>
        <a href="javascript:void(0)">JS</a>
        <a href="https://www.guinness.com/">Guinness</a>
      </body></html>
    `;
    const result = extractBrandDomains(html, 'diageo.com', indexedFromUrl);
    expect(result).toHaveLength(1);
    expect(result[0].domain).toBe('guinness.com');
  });

  it('falls back to host-based name when link text is empty', () => {
    const html = `
      <html><body>
        <a href="https://www.guinness.com/"><img src="/g.png" alt=""/></a>
      </body></html>
    `;
    const result = extractBrandDomains(html, 'diageo.com', indexedFromUrl);
    expect(result).toHaveLength(1);
    expect(result[0].domain).toBe('guinness.com');
    expect(result[0].name).toBe('guinness');
  });

  it('returns empty array on malformed HTML without throwing', () => {
    const result = extractBrandDomains('<<<not html', 'diageo.com', indexedFromUrl);
    expect(result).toEqual([]);
  });
});
  • [ ] Step 3.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/brand-domain-discoverer.test.ts -t extractBrandDomains

Expected: FAIL: extractBrandDomains is not exported.

  • [ ] Step 3.3: Implement extractBrandDomains and helpers

Append to lib/factory/brand-domain-discoverer.ts:

/**
 * Extract outbound brand-domain references from a brand-index page's HTML.
 * Filters out: same-domain links, social-media domains, non-http(s) schemes.
 * Deduplicates by apex domain. First link-text per domain wins.
 *
 * Returns an array of BrandRef. Always validates each element through
 * BrandRefSchema before returning — invalid entries are dropped, not thrown.
 */
export function extractBrandDomains(
  brandIndexHtml: string,
  rootDomain: string,
  indexedFromUrl: string
): BrandRef[] {
  let $: cheerio.CheerioAPI;
  try {
    $ = cheerio.load(brandIndexHtml);
  } catch (err) {
    log.error({ event: 'extractBrands_cheerio_failed', err: String(err) }, 'cheerio parse failed');
    return [];
  }
  const seen = new Map<string, BrandRef>();
  $('a[href]').each((_, el) => {
    const $el = $(el);
    const href = $el.attr('href');
    if (!href) return;
    const candidate = parseAnchor(href, $el.text(), rootDomain, indexedFromUrl);
    if (!candidate) return;
    if (!seen.has(candidate.domain)) {
      seen.set(candidate.domain, candidate);
    }
  });
  const validated: BrandRef[] = [];
  for (const ref of seen.values()) {
    const parsed = BrandRefSchema.safeParse(ref);
    if (parsed.success) {
      validated.push(parsed.data);
    } else {
      log.warn({ event: 'extractBrands_invalid_ref', ref, issues: parsed.error.issues }, 'dropped invalid brand ref');
    }
  }
  log.info(
    { event: 'extractBrands_done', rootDomain, indexedFromUrl, count: validated.length },
    'extracted brand domains'
  );
  return validated;
}

/**
 * Parse a single anchor into a BrandRef candidate, or null if it should
 * be skipped. Handles scheme filtering, same-domain check, social-media
 * blocklist, name fallback.
 */
function parseAnchor(
  href: string,
  rawText: string,
  rootDomain: string,
  indexedFromUrl: string
): BrandRef | null {
  let parsed: URL;
  try {
    parsed = new URL(href);
  } catch {
    return null; // relative or malformed — skip (brand-index is expected to use absolute outbound links)
  }
  if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') return null;
  const apex = apexOf(parsed.hostname);
  if (!apex) return null;
  if (apex === apexOf(rootDomain)) return null;
  if (SOCIAL_MEDIA_DOMAINS.some((s) => apex === s || apex.endsWith(`.${s}`))) return null;
  const text = rawText.trim();
  const name = text.length > 0 ? text : apex.split('.')[0];
  return { name, domain: apex, indexedFromUrl };
}

/**
 * Strip a leading "www." from a hostname so apex comparisons collapse
 * www and root forms. We don't try to do full PSL parsing in v1 — that's
 * deferred to a later iteration if multi-subdomain brand sites surface.
 */
function apexOf(hostname: string): string {
  if (!hostname) return '';
  return hostname.toLowerCase().replace(/^www\./, '');
}
  • [ ] Step 3.4: Run extractBrandDomains tests, expect PASS
npx vitest run lib/factory/__tests__/brand-domain-discoverer.test.ts -t extractBrandDomains

Expected: 7 tests pass.

  • [ ] Step 3.5: Run all module tests + tsc --noEmit
npx vitest run lib/factory/__tests__/brand-domain-discoverer.test.ts && npx tsc --noEmit

Expected: 18 tests pass; tsc clean.

  • [ ] Step 3.6: Commit
git add lib/factory/brand-domain-discoverer.ts lib/factory/__tests__/brand-domain-discoverer.test.ts
git commit -m "feat(factory): extractBrandDomains with social-media filter and dedupe"

Task 4: discoverBrands(rootUrl, sampler, classifier) orchestrator

Files: - Modify: lib/factory/brand-domain-discoverer.ts - Modify: lib/factory/__tests__/brand-domain-discoverer.test.ts

discoverBrands is the public entry point. It accepts injected sampler and classifier functions (so tests can mock them and the consumer in Spec 08 wires the real Spec 03 + Spec 04 implementations). The flow:

  1. Sample the homepage via sampler([rootUrl]).
  2. Run classifier(sample): if siteType !== 'multi-brand-corporate', return { detected: false, brandIndexUrl: null, brands: [] }.
  3. Call findBrandIndexPage(rootUrl, sample). If null, return detected-but-empty: { detected: true, brandIndexUrl: null, brands: [] } (rationale: classifier confirmed multi-brand site, but the brand-index page heuristic missed: the user still benefits from knowing the type).
  4. Sample the brand-index URL via sampler([brandIndexUrl]).
  5. Call extractBrandDomains(html, rootDomain, brandIndexUrl).
  6. Return { detected: true, brandIndexUrl, brands }, validated through DiscoverBrandsResultSchema.

Type contracts (kept loose to decouple from upstream specs): - Sampler = (urls: string[]) => Promise<BrandSample[]> - Classifier = (sample: BrandSample) => Promise<string>: returns the bare siteType token (matches Spec 04 detectSiteType plus a Promise<> wrapper so async classifier flavours can plug in).

These match the smallest viable subset of Spec 03's sampleUrls and Spec 04's detectSiteType. Spec 08 wires the real implementations: anything tighter would couple Spec 07 to module shapes that are still being finalized in Spec 04.

  • [ ] Step 4.1: Write failing tests for discoverBrands: happy path (Diageo)

Append to lib/factory/__tests__/brand-domain-discoverer.test.ts:

import { describe, it, expect, vi } from 'vitest';
import { discoverBrands } from '../brand-domain-discoverer';

describe('discoverBrands — happy path (Diageo)', () => {
  it('returns detected brands when classifier says multi-brand-corporate', async () => {
    const homepageHtml = `
      <html><body>
        <nav>
          <a href="/our-brands">Our Brands</a>
          <a href="/news">News</a>
        </nav>
      </body></html>
    `;
    const brandIndexHtml = `
      <html><body>
        <a href="https://www.johnniewalker.com/">Johnnie Walker</a>
        <a href="https://www.guinness.com/">Guinness</a>
        <a href="https://twitter.com/diageo">Twitter</a>
      </body></html>
    `;
    const sampler = vi
      .fn()
      .mockResolvedValueOnce([{ url: 'https://www.diageo.com/', html: homepageHtml, status: 200 }])
      .mockResolvedValueOnce([{ url: 'https://www.diageo.com/our-brands', html: brandIndexHtml, status: 200 }]);
    const classifier = vi.fn().mockResolvedValue('multi-brand-corporate');

    const result = await discoverBrands('https://www.diageo.com', sampler, classifier);

    expect(result.detected).toBe(true);
    expect(result.brandIndexUrl).toBe('https://www.diageo.com/our-brands');
    expect(result.brands).toHaveLength(2);
    expect(result.brands.map((b) => b.domain).sort()).toEqual(['guinness.com', 'johnniewalker.com']);
    expect(sampler).toHaveBeenCalledTimes(2);
    expect(classifier).toHaveBeenCalledTimes(1);
  });
});

describe('discoverBrands — non-multi-brand site', () => {
  it('returns detected:false when classifier rejects multi-brand-corporate', async () => {
    const homepageHtml = `<html><body><a href="/blog">Blog</a></body></html>`;
    const sampler = vi
      .fn()
      .mockResolvedValueOnce([{ url: 'https://www.example.com/', html: homepageHtml, status: 200 }]);
    const classifier = vi.fn().mockResolvedValue('wordpress');

    const result = await discoverBrands('https://www.example.com', sampler, classifier);

    expect(result.detected).toBe(false);
    expect(result.brandIndexUrl).toBeNull();
    expect(result.brands).toEqual([]);
    // Sampler called only once (homepage); brand-index sampling skipped
    expect(sampler).toHaveBeenCalledTimes(1);
  });
});

describe('discoverBrands — multi-brand confirmed but brand-index page missing', () => {
  it('returns detected:true with empty brands when brand-index page is not found', async () => {
    const homepageHtml = `
      <html><body>
        <a href="/about">About</a>
        <a href="/contact">Contact</a>
      </body></html>
    `;
    const sampler = vi
      .fn()
      .mockResolvedValueOnce([{ url: 'https://www.example-corp.com/', html: homepageHtml, status: 200 }]);
    const classifier = vi.fn().mockResolvedValue('multi-brand-corporate');

    const result = await discoverBrands('https://www.example-corp.com', sampler, classifier);

    expect(result.detected).toBe(true);
    expect(result.brandIndexUrl).toBeNull();
    expect(result.brands).toEqual([]);
    expect(sampler).toHaveBeenCalledTimes(1);
  });
});

describe('discoverBrands — input guards', () => {
  it('returns detected:false when sampler returns empty', async () => {
    const sampler = vi.fn().mockResolvedValue([]);
    const classifier = vi.fn();

    const result = await discoverBrands('https://www.example.com', sampler, classifier);

    expect(result.detected).toBe(false);
    expect(result.brandIndexUrl).toBeNull();
    expect(result.brands).toEqual([]);
    expect(classifier).not.toHaveBeenCalled();
  });

  it('returns detected:false when sampler throws', async () => {
    const sampler = vi.fn().mockRejectedValue(new Error('network down'));
    const classifier = vi.fn();

    const result = await discoverBrands('https://www.example.com', sampler, classifier);

    expect(result.detected).toBe(false);
    expect(result.brands).toEqual([]);
    expect(classifier).not.toHaveBeenCalled();
  });
});
  • [ ] Step 4.2: Run, expect FAIL
npx vitest run lib/factory/__tests__/brand-domain-discoverer.test.ts -t discoverBrands

Expected: FAIL: discoverBrands is not exported.

  • [ ] Step 4.3: Implement discoverBrands

Append to lib/factory/brand-domain-discoverer.ts:

/**
 * Sampler contract — minimal subset of Spec 03's sampleUrls. Decoupled
 * type so Spec 07 doesn't import Spec 03's full module surface.
 */
export type Sampler = (urls: string[]) => Promise<BrandSample[]>;

/**
 * Classifier contract — minimal subset of Spec 04's site-type rubric output.
 */
export type Classifier = (sample: BrandSample) => Promise<string>;

/**
 * The site-type token (per §19d rubric) that gates Spec 07. If the
 * classifier returns any other value, we don't probe for brand domains.
 */
const MULTI_BRAND_SITE_TYPE = 'multi-brand-corporate';

/**
 * Orchestrator. Probes a parent corporate site for external brand domains.
 * Read-only — does not create sessions or write any persistent state.
 *
 * Steps:
 *   1. Sample the parent homepage.
 *   2. Classify it via the §19d site-type rubric.
 *   3. If not multi-brand-corporate → return detected:false.
 *   4. Find the brand-index page on the homepage's <a> links.
 *   5. If not found → return detected:true with empty brands.
 *   6. Sample the brand-index page and extract outbound brand domains.
 *   7. Return detected:true with the extracted brands.
 */
export async function discoverBrands(
  rootUrl: string,
  sampler: Sampler,
  classifier: Classifier
): Promise<DiscoverBrandsResult> {
  const empty = (detected: boolean): DiscoverBrandsResult => ({
    detected,
    brandIndexUrl: null,
    brands: [],
  });

  // 1. Sample homepage
  let homepageSample: BrandSample | null = null;
  try {
    const homepageSamples = await sampler([rootUrl]);
    homepageSample = homepageSamples[0] ?? null;
  } catch (err) {
    log.error(
      { event: 'discoverBrands_sampler_failed_homepage', rootUrl, err: String(err) },
      'homepage sampler failed'
    );
    return DiscoverBrandsResultSchema.parse(empty(false));
  }
  if (!homepageSample) {
    log.info({ event: 'discoverBrands_no_homepage_sample', rootUrl }, 'sampler returned no samples');
    return DiscoverBrandsResultSchema.parse(empty(false));
  }

  // 2. Classify
  let siteType: string;
  try {
    siteType = await classifier(homepageSample);
  } catch (err) {
    log.error({ event: 'discoverBrands_classifier_failed', rootUrl, err: String(err) }, 'classifier failed');
    return DiscoverBrandsResultSchema.parse(empty(false));
  }
  if (siteType !== MULTI_BRAND_SITE_TYPE) {
    log.info(
      { event: 'discoverBrands_not_multi_brand', rootUrl, siteType },
      'site is not multi-brand-corporate; skipping'
    );
    return DiscoverBrandsResultSchema.parse(empty(false));
  }

  // 3-5. Find brand-index page + sample + extract
  return discoverBrandsForFederatedSite(rootUrl, homepageSample, sampler);
}

/**
 * Helper invoked once the site is confirmed multi-brand-corporate.
 * Split out to keep `discoverBrands` ≤20 lines per CodingSOPs §4.
 */
async function discoverBrandsForFederatedSite(
  rootUrl: string,
  homepageSample: BrandSample,
  sampler: Sampler
): Promise<DiscoverBrandsResult> {
  const brandIndexUrl = findBrandIndexPage(rootUrl, homepageSample);
  if (!brandIndexUrl) {
    log.info({ event: 'discoverBrands_no_brand_index', rootUrl }, 'multi-brand site but no /our-brands-style page found');
    return DiscoverBrandsResultSchema.parse({ detected: true, brandIndexUrl: null, brands: [] });
  }
  let brandIndexSample: BrandSample | null = null;
  try {
    const samples = await sampler([brandIndexUrl]);
    brandIndexSample = samples[0] ?? null;
  } catch (err) {
    log.error(
      { event: 'discoverBrands_sampler_failed_index', rootUrl, brandIndexUrl, err: String(err) },
      'brand-index sampler failed'
    );
    return DiscoverBrandsResultSchema.parse({ detected: true, brandIndexUrl, brands: [] });
  }
  if (!brandIndexSample) {
    return DiscoverBrandsResultSchema.parse({ detected: true, brandIndexUrl, brands: [] });
  }
  const rootDomain = hostnameOf(rootUrl);
  const brands = extractBrandDomains(brandIndexSample.html, rootDomain, brandIndexUrl);
  log.info(
    { event: 'discoverBrands_done', rootUrl, brandIndexUrl, count: brands.length },
    'discoverBrands complete'
  );
  return DiscoverBrandsResultSchema.parse({ detected: true, brandIndexUrl, brands });
}

/**
 * Extract the apex hostname from a URL. Returns '' on parse failure so
 * downstream comparisons quietly miss instead of throwing.
 */
function hostnameOf(url: string): string {
  try {
    return apexOf(new URL(url).hostname);
  } catch {
    return '';
  }
}
  • [ ] Step 4.4: Run discoverBrands tests, expect PASS
npx vitest run lib/factory/__tests__/brand-domain-discoverer.test.ts -t discoverBrands

Expected: 5 tests pass.

  • [ ] Step 4.5: Run all module tests + tsc --noEmit
npx vitest run lib/factory/__tests__/brand-domain-discoverer.test.ts && npx tsc --noEmit

Expected: 23 tests pass; tsc clean.

  • [ ] Step 4.6: Commit
git add lib/factory/brand-domain-discoverer.ts lib/factory/__tests__/brand-domain-discoverer.test.ts
git commit -m "feat(factory): discoverBrands orchestrator gated by multi-brand-corporate rubric"

Task 5: Cluster validation: full regression + Status.md update

Files: - (Read-only verification + vault note)

  • [ ] Step 5.1: Run the full project test suite
npx vitest run

Expected: existing baseline tests + 23 new brand-discoverer tests, all GREEN.

  • [ ] Step 5.2: Run full tsc --noEmit
npx tsc --noEmit

Expected: clean.

  • [ ] Step 5.3: Confirm cheerio is already in the bundler externals

Spec 01 added cheerio to scripts/bundle-api.mjs externals. Spec 07 reuses that: no bundler change needed. Verify by inspecting the externals array:

grep -n "'cheerio'" scripts/bundle-api.mjs

Expected: a single line showing 'cheerio', in the externals array. If absent (i.e., Spec 01 wasn't merged), STOP and resolve the dependency before continuing.

  • [ ] Step 5.4: Mark Spec 07 done in vault Status.md

In the vault Projects/Crawler-Factory/Status.md, change:

- | Cluster 7 (Brand domain discovery) | ⏸ | brand-domain-discoverer.ts |
+ | Cluster 7 (Brand domain discovery) | ✅ | brand-domain-discoverer.ts |
  • [ ] Step 5.5: Push branch (only if part of a clean checkpoint)
# Optional — only if you're at a clean checkpoint
git push origin rc3-phoenix

Acceptance criteria: Spec 07 done means:

  1. ✅ lib/factory/brand-domain-discoverer.ts exists and exports: - BrandRefSchema, BrandRef - DiscoverBrandsResultSchema, DiscoverBrandsResult - BrandSample interface - Sampler, Classifier type aliases - BRAND_INDEX_PATH_CANDIDATES, SOCIAL_MEDIA_DOMAINS constants - findBrandIndexPage(rootUrl, sample): string | null - extractBrandDomains(html, rootDomain, indexedFromUrl): BrandRef[] - discoverBrands(rootUrl, sampler, classifier): Promise<DiscoverBrandsResult>
  2. ✅ All public functions have docstrings explaining WHY.
  3. ✅ Logger initialized via createLogger('factory:brand-discoverer'); every public path emits at least one structured log event with rootUrl in the meta.
  4. ✅ Every fallible call (cheerio parse, sampler invocation, classifier invocation) is wrapped in try/catch; errors are logged before falling through to a safe default.
  5. ✅ BrandRefSchema and DiscoverBrandsResultSchema validate every public boundary.
  6. ✅ 23 unit tests in lib/factory/__tests__/brand-domain-discoverer.test.ts cover: - BrandRefSchema accept/reject (4) - SOCIAL_MEDIA_DOMAINS blocklist content (1) - findBrandIndexPage happy path (3 canonical paths) + miss + empty html + relative href resolution (6) - extractBrandDomains happy path + social-media filter + dedupe + apex/www collapse + scheme filter + name fallback + malformed html (7) - discoverBrands happy path (Diageo) + non-multi-brand + brand-index missing + sampler empty + sampler throws (5)
  7. ✅ All tests GREEN; full project test suite still GREEN.
  8. ✅ tsc --noEmit clean.
  9. ✅ Per CodingSOPs: every public function ≤20 lines, ≤3 params; no any; named exports only; conventional-commit messages used per task.
  10. ✅ Vault Status.md Cluster 7 row flipped to ✅.

Out of scope (handled by other specs / deferred to v2)

  • Multi-tenant index management: algoliacentral_factory_tenants writes, tenant-id facet propagation, cross-brand session linking. Deferred to a Spec-08-extension that builds the §17 tenant model on top of the existing session-store. v1 lets the user launch separate single-domain sessions and link them mentally; the factory does not auto-link them.
  • Bulk parallel brand crawl: "crawl all 12 brands at once" is explicitly v2 (§17c). v1 surfaces brand candidates; the user triggers each one manually.
  • Playwright-stealth fallback for WAF-blocked brand-index pages: handled in Spec 03's playwright-fetcher.ts. If the injected sampler already escalates to Playwright on 403, Spec 07 inherits that capability for free; if not, a brand-index page behind WAF returns empty html and Spec 07 quietly returns detected:true, brands:[]. No duplication of WAF handling here.
  • JS-rendered brand-index pages where the link list is hydrated client-side: same answer: the injected sampler is responsible for delivering rendered HTML. If Spec 03's sampler is cheerio-only (static), Playwright-rendered brand-index pages will return empty brands; the user falls back to the manual "supply seeds" flow. v2 can add a Playwright-required heuristic that re-samples through the stealth fetcher when the static sample yields zero outbound links from a confirmed brand-index page.
  • Public Suffix List (PSL) parsing for compound TLDs: apexOf does naive www. stripping only. Brands on co.uk, com.au, etc. still resolve to a comparable apex string for dedupe purposes (e.g., johnniewalker.co.uk and www.johnniewalker.co.uk collapse), but johnniewalker.co.uk and johnniewalker.com are correctly treated as distinct domains. Full PSL is deferred until a real-world site demands it.
  • API endpoint wiring: /api/factory/discover-brands (or extension of /api/factory/discover) is Spec 08's job. Spec 07 only ships the pure module.
  • Frontend "Detected N brand domains" UI: Spec 10/11. Spec 07 ships the data; the frontend renders the CTA.
  • Tenant record creation when 2+ sessions share a parent: §17c v1 says "tenant record created automatically when 2+ sessions share a parent." That auto-creation is Spec 08 / session-store extension territory, not Spec 07.