Dashboard/PRISM (Second-Brain)/wiki/decisions/2026-06-27-search-vendor-packet-detection.md
PRISM (Second-Brain)

wiki/decisions/2026-06-27-search-vendor-packet-detection.md

ADR: Search-Vendor Detection via Live Network-Packet Inspection

Date: 2026-06-27 Status: Accepted Module: prism_platform/v2/detection/search_vendor.py → intel_competitors Commit: a72aea6

Context

PRISM's intel-competitors module needs to know what search technology a prospect and its competitors run, the "Golden Angle" being a competitor already on Algolia (a citable case study). BuiltWith was removed (no budget), so detection must be in-app and free.

The prior in-app approach was an HTML substring source-scan: fetch the rendered homepage, lowercase the HTML, and substring-match against ~12 hardcoded vendor signatures. It was faulty and non-deterministic: - False positive: algolia.com's homepage matched magento → reported "Adobe Commerce". - False negative: it failed to detect Algolia on algolia.com, the algolia.net string only appears in the live API call (fired on query), never the static homepage.

A challenge from Arijit drove the rethink: substring matching is a guess, not proof; a hardcoded vendor list is brittle (new vendors, domain changes); determination must be based on the actual API call, not suppositions. No LLM.

Decision

Detect the search vendor by inspecting the live network packet, not the page source.

A real (stealth) browser loads the site, dismisses consent, and types a query with real keystrokes; all network traffic is captured. A request is a CONFIRMED search call only if it hits a search-endpoint PATH (/search, /queries, /indexes/, /autocomplete…) AND carries the exact query typed. This is zero false positives by construction, an ad/analytics host cannot carry the typed query to a search endpoint by accident.

The vendor identity is the registrable host of the confirmed call, read live off the wire. A small packet-fingerprint layer (vendor-specific header keys / host / param tokens, e.g. x-algolia-application-id) unmasks vendors proxied behind a first-party host. Per-vendor extractors pull deep data from the packet (Algolia app_id / api_key / index).

Multi-vendor sites report ALL confirmed vendors and headline the search engine by precedence: fingerprinted vendor > known third-party host > unknown third-party > first-party.

Rationale

  • Real data, not inference. The browser physically connected to {appid}-dsn.algolia.net and the app_id in the host matched the header, that is proof, not a pattern guess.
  • Vendor-list is no longer a gatekeeper. Detection works for any vendor (incl. unknown / renamed / re-hosted) because identity = the live endpoint domain. The known-names map is cosmetic (pretty labels) and the fingerprints only unmask proxies, absence degrades gracefully to "first-party / proprietary", never a false call.
  • Deterministic. Confirmed-call rule + fingerprint precedence + retry-on-weak removes the run-to-run flips seen when forcing a single "primary" vendor.

Alternatives Considered

Option Why rejected
HTML substring source-scan (prior) Faulty: false positives (decathlon→Magento) + misses live-only vendors (Algolia on algolia.com). Not deterministic.
Shell out to existing detect-search.js (Node) Adds a Node runtime to the Python deploy + two copies of the logic that drift. PRISM already has Playwright-Python.
Hardcoded N-vendor registry as the detector Brittle, new vendors invisible, domain changes break silently. Kept only as cosmetic labels + proxy-unmask fingerprints.
Literal "100% positive detection on every site" Physically impossible (bot-walls, no on-site search, backend engines). Adopted zero-FP + classified exceptions instead.

Consequences

  • New dependency: playwright-stealth>=2.0 (Stealth.use_async wrapper + anti-automation launch args; helps but does not beat headless Akamai → BOT_BLOCKED exception).
  • New API: detect_search_vendor(domain, browser=None) + batch scan_search_vendors(domains) (one shared browser). SearchVendorResult schema gains app_id / api_key / index_name / endpoint_host / all_vendors / proxied. Status taxonomy: DETECTED / NO_ONSITE_SEARCH / BOT_BLOCKED / ERROR / TIMEOUT.
  • Robustness: real-keystroke typing (press_sequentially, not el.fill()), consent dismissal, ⌘K/DocSearch triggers, per-site 55s asyncio.wait_for ceiling so one hung page can't stall a batch, retry-on-weak.
  • Wired into intel_competitors/collector.py: one shared browser scans prospect + competitors → Golden Angle. Fixes the prior competitors_scanned=0 bug (now reads context.competitors).
  • Documented exception classes (undetectable client-side, by design, never guessed): backend/self-hosted engines (Elasticsearch, Solr, self-hosted Typesense); visual search (Syte); proxied vendors that don't forward headers (dell→Coveo); bot-walled (Akamai/CF).

Validation

17 vendors, ~230 real customer sites (auto-sourced lists). 59 confirmed detections, ZERO false positives. The detector repeatedly caught customer-list errors by naming the real vendor: moonmagic/thule = Klevu (not Searchspring), decathlon US = Shopify, twilio/alcltd/ vuori = Algolia, docusign = Inbenta, castorama = Bloomreach, xxl.no = Apptus.

Per-vendor confirmed: Algolia 15, Constructor 12, Cludo 7, Searchspring 5, Coveo 3, Unbxd 3, Klevu 2, Yext 2, Typesense 2, Nosto 2, HawkSearch 2, Bloomreach 2, Doofinder 1, AddSearch 1, Elastic 0, Syte 0, Loop54 0 (exception classes above).

Evidence: docs/workspace/search-detector-validation/REPORT.md + per-vendor results/*.jsonl. Verified: ruff clean; 433 non-browser v2 tests pass; 4 live browser tests pass; collector end-to-end proven (decathlon prospect first-party; gymshark/nuts → Algolia; Golden Angle = [gymshark.com, nuts.com]).