Module: intel-traffic
WAVE 1 SPOKE. Collects comprehensive web traffic intelligence via SimilarWeb
(10 API endpoints) plus Google Trends momentum via Perplexity enrichment.
Produces monthly visits, traffic sources, engagement metrics, keyword analysis,
geo breakdown, and side-by-side competitor traffic comparison. Consumed by
synthesis modules for ROI modeling and business case generation.
Identity
| Field |
Value |
| Name |
intel-traffic |
| Version |
0.1.0 |
| Layer |
Intelligence (Wave 1) |
| Wave |
1 (runs after intel-company; concurrent with other spoke modules) |
| Description |
Web traffic and engagement intelligence via SimilarWeb API (10 endpoints) plus Google Trends momentum via Perplexity sonar-pro. Produces monthly visits, traffic source breakdown, engagement metrics, device split, geo breakdown, organic/paid keywords, seasonal pattern, competitor traffic comparison, and brand search trend direction. |
| LLM Tier |
Minimal (1 Perplexity call for Google Trends momentum; no LLM for structuring) |
| Timeout |
300 seconds (5 minutes -- SimilarWeb can be slow for large domains) |
| Max Retries |
2 (module level) |
| Gate Module |
NO. Failure returns partial result; does not abort audit. |
Dependencies
| Dependency |
Field Read |
Why |
| intel-company |
Account.competitors (DB query by domain or account_id) |
Loads competitor domains for comparative traffic collection |
If intel-company has not run, _read_competitor_info returns an empty list and the module collects prospect-only data. This is non-fatal; comparative_summary will be empty.
Hub-and-Spoke: What This Spoke Provides
intel-traffic writes to module_executions. Downstream consumers:
| Consumer |
Field Read |
Why |
| synth-business-case |
monthly_visits, engagement.bounce_rate, organic_keywords |
Traffic volume baseline for ROI model; bounce rate for improvement opportunity sizing |
| intel-competitors |
competitor_traffic, comparative_summary |
Competitor traffic benchmarking |
| audit-report |
engagement, traffic_sources, seasonal_pattern, google_trends |
Narrative traffic profile sections |
| algolia-audit-skill |
tech_stack_summary, all fields |
Output file 03-traffic-data.md |
| Field |
Type |
Required |
Description |
| domain |
str |
yes |
Website domain to analyze, e.g. "dell.com" |
Competitor domains are read from the accounts table at runtime, not passed in the input schema.
Output Schema: TrafficOutput
Core Traffic Metrics
| Field |
Type |
Required |
Description |
| domain |
str |
yes |
Domain analyzed, e.g. "dell.com" |
| monthly_visits |
list[MonthlyVisit] |
no |
Monthly visit counts for the last 6 months (SimilarWeb data lag: ~2 months) |
| traffic_sources |
list[TrafficSource] |
no |
Traffic source breakdown by channel type |
| engagement |
Engagement or None |
no |
Bounce rate, pages/visit, avg visit duration, total visits |
| device_split |
DeviceSplit or None |
no |
Desktop vs mobile traffic split |
| demographics |
Demographics or None |
no |
Age and gender distribution. Currently None -- SimilarWeb demographics requires premium tier. |
Geo and Keywords
| Field |
Type |
Required |
Description |
| top_countries |
list[GeoBreakdown] |
no |
Top 10 countries by traffic share |
| organic_keywords |
list[Keyword] |
no |
Top 20 organic search keywords. Includes is_branded flag (set post-fetch). |
| paid_keywords |
list[Keyword] |
no |
Top 20 paid search keywords |
Seasonal and Trends
| Field |
Type |
Required |
Description |
| seasonal_pattern |
str |
no |
Detected seasonal pattern, e.g. "Peak in Nov, Dec; Dip in Jan, Feb". Empty if insufficient data (<3 months). |
| google_trends |
GoogleTrendsMomentum or None |
no |
Brand search interest trend from Perplexity. Direction: rising, stable, declining, insufficient_data. |
Competitive
| Field |
Type |
Required |
Description |
| competitor_traffic |
list[CompetitorTraffic] |
no |
Traffic profiles for each competitor domain |
| comparative_summary |
str |
no |
Narrative comparing prospect traffic to competitors, e.g. "dell.com has 3.2x more traffic than HP" |
Sub-Schema: TrafficSource
| Field |
Type |
Required |
Description |
| source_type |
Literal["direct", "organic_search", "paid_search", "social", "referral", "email", "display"] |
yes |
Channel type |
| share_pct |
float |
yes |
Percentage share of total traffic, 0-100 |
| visits |
int or None |
no |
Absolute visit count for this source, if available |
Sub-Schema: MonthlyVisit
| Field |
Type |
Required |
Description |
| year |
int |
yes |
Year of data point (e.g. 2026) |
| month |
int |
yes |
Month of data point, 1-12 |
| visits |
int |
yes |
Total visits for this month |
Sub-Schema: DeviceSplit
| Field |
Type |
Required |
Description |
| desktop_pct |
float |
yes |
Desktop traffic percentage, 0-100 |
| mobile_pct |
float |
yes |
Mobile traffic percentage, 0-100 |
| tablet_pct |
float or None |
no |
Tablet traffic percentage (often not available) |
| desktop_visits |
int or None |
no |
Absolute desktop visit count |
| mobile_visits |
int or None |
no |
Absolute mobile visit count |
Sub-Schema: GeoBreakdown
| Field |
Type |
Required |
Description |
| country |
str |
yes |
Country name, e.g. "United States" |
| country_code |
str |
yes |
ISO 3166-1 alpha-2 code, e.g. "US" |
| share_pct |
float |
yes |
Percentage share of total traffic, 0-100 |
| visits |
int or None |
no |
Absolute visit count from this country |
SimilarWeb returns ISO 3166-1 numeric country codes. The collector maps these to (name, alpha-2) via a hardcoded lookup table covering 47 countries.
Sub-Schema: Keyword
| Field |
Type |
Required |
Description |
| keyword |
str |
yes |
The search keyword text |
| share_pct |
float |
yes |
Percentage of search traffic from this keyword, 0-100 |
| change_pct |
float or None |
no |
Period-over-period change in share percentage |
| search_volume |
int or None |
no |
Estimated monthly search volume |
| is_branded |
bool |
yes |
True if keyword contains the company name or domain base (set post-fetch via brand term matching) |
Sub-Schema: Engagement
| Field |
Type |
Required |
Description |
| bounce_rate |
float or None |
no |
Bounce rate as a decimal 0-1, e.g. 0.45 = 45% |
| pages_per_visit |
float or None |
no |
Average pages viewed per visit |
| avg_visit_duration_seconds |
float or None |
no |
Average visit duration in seconds |
| total_visits |
int or None |
no |
Sum of visits across the reporting period |
Sub-Schema: GoogleTrendsMomentum
| Field |
Type |
Required |
Description |
| company_name |
str |
yes |
Company or brand name assessed |
| direction |
Literal["rising", "stable", "declining", "insufficient_data"] |
yes |
Trend direction: rising = >5% YoY increase; stable = within +/-5%; declining = >5% YoY decrease |
| yoy_change_description |
str |
no |
Human-readable YoY change description extracted from Perplexity response |
| evidence |
str |
no |
First 500 characters of Perplexity response as evidence (capped) |
Sub-Schema: CompetitorTraffic
| Field |
Type |
Required |
Description |
| company_name |
str |
yes |
Competitor company name |
| domain |
str |
yes |
Competitor domain, e.g. "hp.com" |
| total_visits |
int or None |
no |
Total visits in reporting period |
| bounce_rate |
float or None |
no |
Bounce rate as decimal 0-1 |
| pages_per_visit |
float or None |
no |
Average pages per visit |
| traffic_sources |
list[TrafficSource] |
no |
Traffic source breakdown |
| top_keywords |
list[Keyword] |
no |
Top 5 organic keywords |
| device_split |
DeviceSplit or None |
no |
Desktop vs mobile split |
Collection Strategy
Source 1: SimilarWeb API -- Prospect Domain (10 Endpoints)
| Aspect |
Detail |
| Adapter |
Direct httpx (TrafficCollector) |
| Base URL |
https://api.similarweb.com/v1/website/{domain}/ |
| Timeout |
30 seconds per endpoint |
| Date Range |
6 months ending 2 months ago (SimilarWeb data lag) |
| Country |
"world" (global aggregate) |
| Auth |
api_key query param (SIMILARWEB_API_KEY env var) |
| Evidence Tier |
VERIFIED |
10 endpoints called sequentially (within a single httpx.AsyncClient session):
| # |
Endpoint |
Output Field |
Path |
| 1 |
total-traffic-and-engagement/visits |
monthly_visits |
/v1/website/{domain}/total-traffic-and-engagement/visits |
| 2 |
traffic-sources/overview |
traffic_sources |
/v1/website/{domain}/traffic-sources/overview |
| 3 |
geo/traffic-by-country |
top_countries |
/v1/website/{domain}/geo/traffic-by-country |
| 4 |
search-keywords/organic |
organic_keywords |
/v1/website/{domain}/search-keywords/organic |
| 5 |
search-keywords/paid |
paid_keywords |
/v1/website/{domain}/search-keywords/paid |
| 6 |
total-traffic-and-engagement/visits (aggregate) |
engagement |
Same as #1 but reads bounce_rate, pages_per_visit, avg_visit_duration at top level |
| 7 |
total-traffic-and-engagement/visits?device=Desktop |
device_split.desktop |
Device-filtered variant |
| 8 |
total-traffic-and-engagement/visits?device=MobileWeb |
device_split.mobile |
Device-filtered variant |
| 9 |
audience-interests/also-visited |
audience_interests (not in output schema, raw) |
/v1/website/{domain}/audience-interests/also-visited |
| 10 |
popular-pages |
popular_pages (not in output schema, raw) |
/v1/website/{domain}/popular-pages |
Note: Endpoints 9 and 10 (audience interests and popular pages) are collected but not currently surfaced in TrafficOutput. They are in the raw prospect_data dict available for future use.
Every endpoint call is wrapped in individual try/catch. A 403 or 404 from SimilarWeb on any endpoint logs a warning and returns an empty/None result; collection continues with remaining endpoints.
Source 2: SimilarWeb API -- Competitor Domains (Subset)
For each competitor, the collector calls 5 of the 10 endpoints:
- Engagement (total visits, bounce rate, pages/visit)
- Traffic sources
- Organic keywords (top 5 only)
- Device split (Desktop + MobileWeb)
This reduced set limits API usage while still producing the comparative_summary narrative.
Competitors are collected sequentially (not in parallel) in the current v1 implementation. Each competitor is wrapped in its own try/except; a failure for one competitor does not stop others.
Source 3: Perplexity API -- Google Trends Enrichment
| Aspect |
Detail |
| Adapter |
Direct httpx (TrafficEnricher) |
| Endpoint |
POST https://api.perplexity.ai/chat/completions |
| Model |
sonar-pro ($3/1M input tokens, $15/1M output tokens) |
| Timeout |
60 seconds |
| Calls per audit |
1 (prospect only; competitor trends are not assessed in v1) |
| Evidence Tier |
WEBSEARCH |
| Temperature |
0.1 (near-deterministic) |
| Max Tokens |
2048 |
Prompt strategy: Asks Perplexity to assess brand search interest for the company over 12 months, classify as exactly one of rising/stable/declining/insufficient_data, provide YoY change percentage, and cite sources. Response is parsed by looking for a DIRECTION: prefix line to extract the classification deterministically.
Enrichment Strategy
Brand Keyword Tagging (Post-Fetch)
After organic keywords are fetched, each keyword is checked against brand terms derived from _extract_brand_terms(company_name, domain):
- Domain base (e.g. "dell" from "dell.com")
- Company name words excluding common suffixes (Inc, Corp, Ltd, LLC, Technologies, Group -- words under 3 chars also excluded)
If any brand term appears in the keyword text (case-insensitive substring), is_branded = True.
Seasonal Pattern Detection (Deterministic)
_detect_seasonal_pattern() runs on monthly_visits:
- Builds a month-to-visits dict
- Calculates average across all months
- Flags months with > +15% deviation as peaks, < -15% as dips
- Returns human-readable description, e.g. "Peak in Nov, Dec; Dip in Jan"
- Returns "Relatively stable traffic across months" if no month exceeds the threshold
- Returns "" if < 3 months of data
Comparative Summary (Deterministic)
_build_comparative_summary() compares engagement.total_visits between prospect and each competitor:
- If both have visit counts: calculates X.Yx multiplier and states who has more
- If only competitor has count: reports competitor absolute visit count
- Returns "" if no competitor traffic data exists
Validation Rules (9 Checks)
| # |
Rule |
Severity |
Threshold |
| 1 |
monthly_visits has at least 3 months |
required |
>= 3 MonthlyVisit records |
| 2 |
traffic_sources has at least 2 source types |
required |
>= 2 TrafficSource records |
| 3 |
engagement is not None |
warning |
Engagement model populated |
| 4 |
top_countries has at least 1 entry |
required |
>= 1 GeoBreakdown record |
| 5 |
organic_keywords has at least 3 entries |
warning |
>= 3 Keyword records |
| 6 |
domain in output matches input |
required |
output.domain == expected_domain (case-insensitive) |
| 7 |
All traffic_source share_pct values are 0-100 |
required |
No out-of-range values |
| 8 |
comparative_summary present when competitors exist |
warning |
Not empty if competitor_traffic populated |
| 9 |
At least 1 source provenance recorded |
required |
sources list not empty |
Required failures -> status "partial". Warning failures -> logged but passed.
Note: Checks 3, 5, 8 are warnings. SimilarWeb does not have data for all domains, especially smaller or newer sites. Missing engagement or keyword data is expected for some prospects.
Evidence Requirements
| Field |
Minimum Tier |
Rationale |
| monthly_visits |
VERIFIED |
Direct SimilarWeb API with API key authentication |
| traffic_sources |
VERIFIED |
Direct SimilarWeb API |
| engagement |
VERIFIED |
Direct SimilarWeb API |
| top_countries |
VERIFIED |
Direct SimilarWeb API |
| organic_keywords |
VERIFIED |
Direct SimilarWeb API |
| google_trends.direction |
WEBSEARCH |
Perplexity synthesis of web sources |
| seasonal_pattern |
WEBSEARCH |
Derived from verified visit data (WEBSEARCH by convention for derived fields) |
| comparative_summary |
WEBSEARCH |
Derived narrative from verified competitor data |
Persistence
intel-traffic writes to module_executions only. It does NOT write directly to the accounts table.
module_executions Fields Written
module_name="intel-traffic",
module_version="0.1.0",
status="success" | "partial" | "failed",
output={...TrafficOutput fields...},
sources=[...Source records...],
duration_ms=...,
llm_calls=1, # 1 Perplexity call for Google Trends
llm_cost_usd=... # Estimated: ~$0.002-0.005 per run
Status logic:
- success: monthly_visits list is not empty (SimilarWeb returned visit data)
- partial: monthly_visits is empty but module did not throw
- failed: unhandled exception in execute()
Cost Profile
| Metric |
Expected Value |
| LLM Calls |
1 (Perplexity sonar-pro for Google Trends) |
| Estimated LLM Cost |
~$0.002-0.005 per run (sonar-pro at ~3K input + 2K output tokens) |
| Expected Duration |
30-120 seconds (SimilarWeb can be slow for large domains) |
| External API Calls |
10 (SimilarWeb prospect) + 5N (SimilarWeb competitor, N = competitor count) + 1 (Perplexity) |
Retry Architecture
Module Level
| Trigger |
Behavior |
| Exception in execute() |
Caught with logger.exception, returns ModuleExecutorResult(status="failed") |
| Max Retries |
2 (via ModuleConfig.max_retries, handled by ModuleExecutor) |
Collector Level (per endpoint)
| Trigger |
Behavior |
| httpx.TimeoutException |
logger.error, returns [] or None for that endpoint |
| httpx.HTTPStatusError |
logger.warning with status_code, returns [] or None |
| Any other exception |
logger.exception, returns [] or None |
Each of the 10 prospect endpoints has independent error handling. A 403 on the keywords endpoint does not prevent engagement or geo data from being returned.
Enricher Level (Perplexity)
| Trigger |
Behavior |
| httpx.TimeoutException |
Returns GoogleTrendsMomentum(direction="insufficient_data", yoy="API timeout") |
| httpx.HTTPStatusError |
Returns GoogleTrendsMomentum(direction="insufficient_data", yoy="API error: HTTP {code}") |
| Empty response |
Returns GoogleTrendsMomentum(direction="insufficient_data", yoy="Perplexity returned no data") |
| Any other exception |
logger.exception, returns (None, 0, 0.0) -- google_trends field will be None in output |
Runtime Notes
SaaS Runtime (Temporal/Python)
- Module:
prism_platform/v2/modules/intel_traffic/module.py
- Collector:
prism_platform/v2/modules/intel_traffic/collector.py
- Enricher:
prism_platform/v2/modules/intel_traffic/enricher.py
- Validator:
prism_platform/v2/modules/intel_traffic/validator.py
- Schemas:
prism_platform/v2/modules/intel_traffic/schemas.py
- Config keys:
settings.similarweb_api_key (SIMILARWEB_API_KEY), settings.perplexity_api_key (PERPLEXITY_API_KEY)
- health_check(): Returns True only if BOTH API keys are set
Skill Runtime (Claude Code)
- Skill file:
~/.claude/skills/algolia-intel-traffic/SKILL.md
- Reads from:
01-company-context.json (competitor domains and company name)
- Writes to:
03-traffic-data.md, 03-traffic-data.json
- MCP tools used: SimilarWeb MCP (all 11 endpoints for prospect), SimilarWeb MCP per competitor, WebSearch (Google Trends assessment)
What Makes This Production-Grade (vs a Simple Prompt)
| Aspect |
Simple Prompt |
PRISM Module |
| Data source |
"Estimate traffic for X" |
SimilarWeb API: 10 endpoints, authenticated, date-range scoped |
| Competitor coverage |
Ask about each competitor separately |
Automated fan-out to N competitors using stored Account.competitors |
| Search vendor correlation |
Not connected |
traffic_sources + organic_keywords together reveal paid search investment alongside tech stack from intel-techstack |
| Seasonality detection |
LLM guess |
Deterministic: month-by-month deviation from average, >15% threshold |
| Google Trends |
Not researched |
Perplexity sonar-pro with structured DIRECTION: parsing, not freeform |
| Brand keyword classification |
Not tagged |
Post-fetch brand term extraction from company name + domain |
| Partial data handling |
Fail entirely |
Per-endpoint try/catch; SimilarWeb 403s are warnings, not failures |
| Source provenance |
None |
Source record per SimilarWeb endpoint + 1 for Perplexity |
| Validation |
Trust the LLM |
9-point validation checklist (6 required, 3 warning) |
| Cost tracking |
None |
llm_calls and llm_cost_usd recorded in ModuleExecutorResult |
Changelog
| Date |
Change |
Reason |
| 2026-04-14 |
Module spec written to vault |
Knowledge extraction for v2 planning |
Evolution: v1 -> v2
What Changes in v2
| Dimension |
v1 (current) |
v2 (planned) |
| Module structure |
module.py + collector.py + enricher.py + validator.py |
config.py + playbook.md + schemas.py + generic executor |
| Competitor input |
DB query in _read_competitor_info (domain or account_id) |
Read from typed Account.competitors via FindingQuery contract -- no raw SQL |
| Competitor collection |
Sequential per-competitor loop in execute() |
Declared execution_strategy: per_company in config.py -- executor handles fan-out generically |
| Demographics |
Not collected (SimilarWeb premium required, returns None) |
v2 adds a premium-tier flag in config.py; executor skips endpoint if flag not set |
| Enricher coupling |
TrafficEnricher called directly from module.py |
Enricher defined as an enrichment_step in playbook.md; executor orchestrates it |
| Finding model |
Raw dict in module_executions.output |
Typed Finding with FindingCategory (TRAFFIC_PROFILE, BRAND_MOMENTUM, COMPETITIVE_TRAFFIC) |
| Claim registry |
Not generated |
ClaimRegistryEntry auto-generated per Finding for citation chain |
| Audience interests |
Collected but discarded |
v2 decides: either surface in output schema or remove the endpoint call |
| Popular pages |
Collected but discarded |
Same decision needed in v2 |
| Google Trends |
Prospect-only |
v2 config option to enable competitor trends enrichment (currently stubbed as assess_competitor_trends) |
v2 FindingCategory Taxonomy
TRAFFIC_PROFILE -- prospect monthly visits, bounce rate, engagement summary
BRAND_MOMENTUM -- Google Trends direction (rising/stable/declining)
TRAFFIC_SOURCE_MIX -- channel breakdown (direct, organic, paid, etc.)
COMPETITIVE_TRAFFIC -- competitor traffic benchmarks and comparative summary
SEASONAL_PATTERN -- detected peak and dip months
TOP_KEYWORDS -- organic keyword landscape with branded/non-branded split
Key v2 Design Decisions
-
execution_strategy: per_company for competitor fan-out replaces the manual loop in execute(). This makes intel-traffic consistent with intel-techstack and other modules that operate on competitor lists.
-
Surface or drop audience_interests and popular_pages. Currently these 2 endpoints are called and their data is dropped. v2 must decide: add them to the output schema (adds value for audit reports) or remove the API calls (reduces cost). Current lean: add audience_interests to output and surface as AUDIENCE_INTERESTS Finding.
-
Competitor trends enrichment. assess_competitor_trends() is already implemented in TrafficEnricher but not called in v1 execute(). v2 can enable it via a config flag. Default: off (adds N Perplexity calls to cost profile).
-
Demographics behind a premium flag. SimilarWeb demographics requires a higher-tier license. v2 config.py will have demographics_enabled: bool = False. When True, executor calls the demographics endpoint; when False, skips it cleanly rather than silently returning None.
- Module-Catalog -- all 20 modules overview
- Intel-Company -- provides competitor list and company name for brand term extraction
- Evidence-Tier-Spec -- evidence tier system
- Adapter-Interfaces -- SimilarWeb and Perplexity adapter interfaces