Dashboard/Competitive Intelligence/logs/2026-06-18-competitive-research-v2-worklog.md
Competitive Intelligence

logs/2026-06-18-competitive-research-v2-worklog.md

Competitive research skill v2 worklog

Related workspace note: competitive-intelligence-os-notes

Executive summary

The competitive-research skill was rebuilt from a weak "search buckets -> brief" flow into a public-source competitive intelligence engine with a SQLite signal ledger, source coverage audit, daily Telegram pulse, weekly HTML report, and Hermes cron delivery.

Current status: code is operational for the Algolia pack's direct-source baseline workflow, Hermes is updated, Telegram delivery has been smoke-tested, and data is being stored. Content quality is not yet fully proven because the live system has mostly baseline/no-change data. It needs 1-2 real daily cycles before evaluating whether the intelligence output is useful.

The most important lesson from the rebuild: the report must follow the data. If there are zero changed/material signals, the output must say that directly and avoid forcing an AI, positioning, or battlecard narrative.

Why this rebuild happened

The prior competitive research skill was not good enough. It over-relied on broad search dumps, did not distinguish baseline captures from real deltas, produced weak or forced strategic narratives, and did not make it clear whether "nothing happened" meant market quiet or broken collection.

The desired methodology came from B2B SaaS competitive intelligence best practices:

  • Monitor competitor changelogs, docs, blogs, press releases, case studies, product pages, pricing/packaging, partnerships, analyst/review surfaces, communities, and AI visibility surfaces.
  • Use win/loss and field data later, but keep v1 public-source only.
  • Operationalize the intelligence into daily pulses and weekly synthesis, not static dashboards.
  • Separate source coverage from signal generation.
  • Give every recommendation a clear owner and source-backed evidence.

Architecture implemented

The skill now follows this flow:

collect -> normalize -> score -> store -> synthesize -> deliver

Core runtime paths:

Local Chowmes project:
/Volumes/Data/Dropbox/AI-Development/Personal/ChowMes

Hermes skill:
/opt/data/knowledge/obsidian/MyOS/Projects/Competitive Intelligence/skills/competitive-research

Hermes cron wrappers:
/opt/data/scripts/competitive-research-daily.sh
/opt/data/scripts/competitive-research-weekly.sh

Hermes ledger and artifacts:
/opt/data/knowledge/obsidian/MyOS/Projects/Competitive Intelligence/artifacts/competitive-research/ci.sqlite
/opt/data/knowledge/obsidian/MyOS/Projects/Competitive Intelligence/artifacts/competitive-research/briefs/
/opt/data/knowledge/obsidian/MyOS/Projects/Competitive Intelligence/artifacts/competitive-research/reports/
/opt/data/knowledge/obsidian/MyOS/Projects/Competitive Intelligence/artifacts/competitive-research/raw/

Local review artifacts:
/Volumes/Data/Dropbox/AI-Development/Personal/Obsidian-Vault/MyOS/Projects/Competitive Intelligence/artifacts/competitive-research/

Core files:

skills/competitive-research/scripts/ci_core.py
skills/competitive-research/scripts/daily-research-run.py
skills/competitive-research/scripts/weekly-review.py
skills/competitive-research/scripts/source-coverage-audit.py
skills/competitive-research/scripts/competitive-research-daily.sh
skills/competitive-research/scripts/competitive-research-weekly.sh
skills/competitive-research/references/sources.yaml

SQLite tables:

  • sources: configured source registry.
  • snapshots: fetched page snapshots, hashes, fetch status, errors.
  • signals: normalized competitive signals.
  • synthesis_runs: generated markdown and HTML report records.

Algolia pack source coverage

The Algolia pack currently has 43 configured sources.

Configured source types:

  • blog: 15
  • changelog: 4
  • docs: 3
  • news: 2
  • press: 1
  • product: 7
  • case_study: 3
  • forum: 4
  • analyst: 1
  • protocol: 2
  • roadmap: 1

Competitor and watchlist coverage includes:

  • Coveo
  • Bloomreach
  • Constructor
  • Google Vertex AI Search
  • Elastic
  • Lucidworks
  • Yext
  • Algonomy
  • Meilisearch
  • Typesense
  • AWS OpenSearch / CloudSearch
  • Perplexity AI
  • OpenAI / ChatGPT
  • AI Agent Ecosystem
  • Glean
  • Gartner MQ Search and Product Discovery
  • CMSWire Digital Experience
  • Martech Edge
  • Contra Collective
  • Community sources
  • Algolia MCP baseline docs

Live coverage audit result

After forcing a full baseline pass, Hermes checked all configured sources.

Latest live coverage:

  • Enabled sources: 43
  • Checked sources: 43
  • Successful fetches: 30
  • Failed fetches: 13
  • Missing sources: 0
  • Changed/material competitive signals after noise filtering: 0

Important failing sources:

  • Gartner MQ Search and Product Discovery: 403 Forbidden
  • Lucidworks blog: 403 Forbidden
  • Lucidworks product page: 403 Forbidden
  • Yext product page: 503 Service Unavailable
  • Model Context Protocol blog: 404 Not Found
  • CMSWire Digital Experience: 500 Internal Server Error
  • OpenAI blog: 403 Forbidden
  • Perplexity blog: 403 Forbidden
  • Algonomy blog: 404 Not Found
  • Elastic release notes URL: 404 Not Found
  • Reddit ecommerce/SEO/MachineLearning communities: 403 Blocked

Interpretation:

  • 403 does not mean the market is quiet. It means the current direct fetch method cannot observe the source.
  • 404 means the configured URL should be replaced.
  • 500/503 may be transient, but recurring failures need replacement or alternate collection.
  • Gartner is likely gated and should be marked manual or routed through credentialed access.
  • Reddit/community should use an API, RSS/search fallback, or a different community monitoring method.

Key decisions made

V1 stays inside the skill, not a separate app

Decision: keep v1 inside the Hermes/Codex skill rather than building a separate backend app.

Reasoning:

  • Hermes is strong enough as the reasoning and delivery layer.
  • The missing layer was not an app UI. It was a structured signal ledger.
  • The right proving ground is daily/weekly operational use.
  • A separate app can come later if the ledger proves useful and repeatable.

Public-source only for v1

Decision: v1 uses only public sources.

Deferred:

  • Gong
  • Salesforce
  • Slack
  • G2 paid exports
  • Semrush API
  • internal win/loss
  • field intel

Reasoning: prove the public-source ledger and synthesis quality first.

Daily pulse equals deltas, not broad discovery

Decision: production daily runs now use direct-source diffs only.

Current wrapper:

daily-research-run.py --skip-search --skip-monitors

Reasoning:

  • Broad parallel_search polluted the daily ledger with old public search results.
  • The daily report should answer "what changed today?"
  • Broad discovery belongs in weekly expansion, explicit deep-dive mode, or a separately validated collector.

Weekly synthesis carries heavier analysis

Decision: weekly reports can include source coverage, baselines, quiet sources, and collection failure analysis.

Reasoning:

  • Weekly is the right cadence for patterns, coverage gaps, and source health.
  • Daily should stay compact and action-oriented.

Quiet days must be deterministic

Decision: if there are zero material daily signals, daily synthesis uses a deterministic quiet-day pulse instead of asking the model to generate strategy.

Reasoning:

  • Hermes/LLM synthesis was tempted to invent a Product Marketing recommendation from baseline signals.
  • That violated the principle that the data should dictate the text.
  • Quiet days should say: no material public competitive signal, source coverage health, and what failed.

Baselines are not competitive moves

Decision: baseline captures are setup evidence, not intelligence.

Rules:

  • Do not update battlecards from baselines.
  • Do not claim a strategic pattern from baselines.
  • Do not headline a topic with a zero count.
  • If all signals are baselines, say "Baseline established" and "No diffs yet."

Search and monitor ingestion are deferred

Decision: broad search and monitor ingestion are disabled in production daily delivery until validated separately.

Reasoning:

  • Search ingestion created 75 noisy rows in one test day.
  • Monitor path appeared to contribute to wrapper latency and needs a separate bounded smoke test.
  • These should return later as separate, clearly labeled collectors.

Telegram gets compact pulse plus HTML artifact

Decision: Telegram should receive a compact text pulse and the HTML report as an attachment via MEDIA:.

Reasoning:

  • Telegram should not receive raw dumps, long tables, stack traces, or full reports in chat.
  • HTML is better for structured review.
  • Telegram is the delivery surface, not the debugging surface.

UX and report design decisions

The weekly HTML report was redesigned around decision usefulness.

Final weekly information architecture:

  • Hero with clickable stat cards.
  • First screen: decision state, owner, urgency, evidence count, top competitor, confidence.
  • Readout cards:
  • What changed
  • Strategic pattern
  • Recommendation
  • Signal count audit with accordions.
  • Source coverage audit.
  • Recommended actions by owner.
  • Battlecard updates.
  • Top signal cards.
  • Coverage gaps.
  • Source ledger.

Important UI decisions:

  • Source ledger stays because it provides trust and auditability, but it should be compact.
  • Signal cards capped at 3 for weekly.
  • Hero/stat cards are clickable and visually button-like.
  • Source ledger moved to full-width to reduce scrolling and improve readability.
  • Weekly pulse shows the date range.
  • The report now uses Algolia-style restrained UI: Sora, Algolia blue, white surfaces, compact cards.

Bugs and false starts fixed

Forced AI narrative

Problem: report said "AI visibility concentrated" even when 0 of 5 signals were AI visibility.

Fix:

  • AI can only headline if ai_count > 0 and it is actually the dominant category.
  • Baseline-only weeks force Baseline established and No diffs yet.

Local fixture polluted review

Problem: local ledger had old 82-signal fixture data while live Hermes ledger had 5 real signals.

Fix:

  • Copied live Hermes artifacts back into local review folder before UI review.
  • Treat live Hermes ledger as the decision source.

Incorrect coverage-gap claims

Problem: synthesis initially claimed competitors or source types were not configured when they were configured but had no stored signal.

Fix:

  • Added source_inventory and source_coverage to synthesis packet.
  • Prompt now requires distinguishing:
  • checked and quiet
  • missing coverage
  • collection failure

Problem: Bloomreach diffs were mostly nav/header changes such as Careers 84 to Careers 85, but synthesis treated them as material.

Fix:

  • Added is_boilerplate_diff().
  • Marked existing Bloomreach nav-noise changed signals as ignored.
  • Ignored signals are excluded from counts, packets, and coverage.

Hacker News front-page churn treated as community signal

Problem: HN front-page changes created "Community changed forum" signals that were not related to Algolia, competitors, search, ecommerce, product discovery, MCP, or AI search.

Fix:

  • Added community relevance filter.
  • Forum/front-page diffs become signals only if they mention Algolia, named competitors, or relevant search/discovery terms.
  • Existing HN churn signals were marked ignored.

Broad search polluted daily ledger

Problem: parallel_search inserted old search results as "daily signals," leading to inflated counts and bad recommendations.

Fix:

  • Production daily wrapper now runs --skip-search --skip-monitors.
  • Daily material filter excludes parallel_search.
  • Broad search is deferred to weekly/deep-dive mode after validation.

Daily quiet day model overreach

Problem: even with zero material diffs, model-generated daily pulse recommended a Product Marketing task based on baseline signals.

Fix:

  • Daily runner now uses deterministic local synthesis when packet["signals"] is empty.
  • Quiet-day output focuses on source coverage and failed sources.

Cron wrapper path and permission issues

Problem: earlier deployment copied wrapper files to /opt/data/... on the host, but /opt/data is the container path.

Fix:

  • Deploy into the Hermes container via docker cp and docker exec.
  • Ensure files are owned by hermes:hermes.
  • Health check verifies no root-owned runtime drift.

Current production daily behavior

Daily cron wrapper:

/opt/data/scripts/competitive-research-daily.sh

Runs:

/opt/hermes/.venv/bin/python /opt/data/knowledge/obsidian/MyOS/Projects/Competitive Intelligence/skills/competitive-research/scripts/daily-research-run.py --skip-search --skip-monitors

Behavior:

  • Fetch direct public URLs.
  • Store snapshots and content hashes.
  • Compare against prior snapshot.
  • Suppress boilerplate and irrelevant forum churn.
  • Store material changed signals only.
  • If no material signals exist, generate deterministic quiet-day pulse.
  • Save Markdown and HTML.
  • Emit compact Telegram text with MEDIA: HTML attachment.

Latest daily Telegram output summary:

  • No material public competitive signal detected.
  • 43 of 43 sources checked.
  • 30 succeeded.
  • 13 failed.
  • 0 missing.
  • Recommendation: Competitive Intelligence should repair or replace failed sources before treating quiet days as confirmed market quiet.

Current weekly behavior

Weekly cron wrapper:

/opt/data/scripts/competitive-research-weekly.sh

Weekly report:

/opt/data/knowledge/obsidian/MyOS/Projects/Competitive Intelligence/artifacts/competitive-research/reports/YYYY-MM-DD-weekly.html

Latest report status:

  • Baseline established.
  • No diffs yet after noise filtering.
  • Source coverage audit visible in HTML.
  • No forced AI narrative.
  • No battlecard updates from baselines.

Tests and verification completed

Static checks:

  • Python compile passed for core scripts:
  • ci_core.py
  • daily-research-run.py
  • weekly-review.py
  • source-coverage-audit.py
  • setup-monitors.py

Source validation:

  • sources.yaml loads via runtime loader.
  • 43 source records flatten correctly.
  • Required fields present.

Live Hermes verification:

  • Live skill contains:
  • get_source_coverage
  • is_boilerplate_diff
  • clean_synthesis_markdown
  • source-coverage-audit.py
  • Daily and weekly competitive cron entries exist.
  • ci.sqlite exists on Hermes and is being written.
  • Markdown and HTML artifacts are saved.

Telegram verification:

  • Daily wrapper output was sent to Telegram home channel.
  • Health-check Telegram send succeeded.
  • HTML artifact attachment line exists via MEDIA:.

Health check:

  • scripts/chowmes-health-check --repair --send-test passed.
  • No root-owned runtime file drift.
  • Gateway running.
  • Parallel web backend healthy.
  • Telegram delivery works.

What is code-complete

Code-complete for v2 direct-source baseline workflow:

  • Source registry.
  • Direct URL collection.
  • Snapshot storage.
  • Diffing.
  • Signal normalization.
  • SQLite ledger.
  • Source coverage audit.
  • Daily quiet-day pulse.
  • Weekly HTML report.
  • Telegram delivery.
  • Hermes cron deployment.

Not yet code-complete:

  • Reliable broad search ingestion.
  • Reliable monitor ingestion.
  • Browser/headless fallback collector.
  • Reddit/API collector.
  • Gartner authenticated/manual workflow.
  • Generic multi-company profile system.
  • Dashboard app.

Content quality status

Content quality is not proven yet.

Reason:

  • The system has mostly baseline/no-change data.
  • No real material competitor delta has been observed after filtering.
  • A useful report can only be judged after 1-2 real daily cycles create actual diffs.

Current confidence:

  • Engineering/plumbing: medium-high.
  • Source coverage: medium with known failures.
  • Daily content quality: acceptable for quiet-day operational pulse.
  • Weekly content quality: acceptable for setup/baseline report.
  • Strategic recommendations: not validated until real material changes appear.

Next moves

Run 1-2 real daily cycles

Let the Algolia pack run naturally for at least 1-2 days.

Evaluate:

  • Did snapshots continue to store?
  • Did failed source count change?
  • Did any real changed signal appear?
  • Did quiet-day Telegram output remain restrained?
  • Did HTML report stay readable?

Repair high-value failing sources

Priority:

  1. Gartner MQ Search and Product Discovery.
  2. Lucidworks blog and product pages.
  3. OpenAI / ChatGPT.
  4. Perplexity AI.
  5. Reddit/community sources.
  6. CMSWire.
  7. Algonomy.
  8. Elastic release notes URL.
  9. Yext product page.
  10. MCP blog URL.

Recommended approach:

  • Replace broken URLs first.
  • Mark gated sources as manual or authenticated.
  • Use API/RSS/search fallback for community.
  • Use browser/headless fallback only where source value justifies it.

Validate monitors separately

Do not re-enable monitors in daily cron until tested.

Test requirements:

  • Bounded runtime.
  • No session hang.
  • Events must include source URLs.
  • No broad dumping into daily signal ledger.
  • Monitor alerts should be material changed signal candidates, not search-style discovery.

Validate search separately

Do not re-enable broad parallel_search in daily.

Possible future role:

  • Weekly source discovery.
  • Manual deep dive.
  • New competitor detection.
  • Enrichment for a known signal.

Search results should not become daily signals without freshness, date, source authority, and deduping checks.

Strategy for generic CI OS

Do not generalize yet. First make the Algolia pack trustworthy.

Eventual architecture:

Core CI engine
  sources
  snapshots
  diffs
  signals
  scoring
  source coverage
  synthesis
  reports
  delivery

Company profile pack
  target company
  competitors
  taxonomy
  action owners
  brand voice
  source registry
  report theme
  delivery config

Collector adapters
  direct URL
  search
  monitor
  browser/headless
  RSS/API
  paid/internal integrations later

Output adapters
  Telegram
  HTML
  Slack
  email
  dashboard later

The Algolia pack should become one profile pack, not the whole product.

Profile examples:

  • algolia-search
  • salesforce-crm
  • shopify-commerce
  • hubspot-marketing
  • snowflake-data-cloud

Generalization criteria:

  • Algolia pack runs cleanly for multiple cycles.
  • Quiet days are trustworthy.
  • Material diffs are correctly detected and summarized.
  • Source coverage failures are manageable.
  • Output produces useful action without model overreach.

Durable operating lessons

  • A signal ledger is the foundation. Raw search dumps are not intelligence.
  • Coverage health must be visible before any strategic conclusion.
  • Baselines are setup, not movement.
  • Zero counts must never become headlines.
  • Daily reports should be delta-first.
  • Weekly reports should be pattern and coverage-first.
  • Telegram should be compact and artifact-led.
  • HTML is the decision artifact.
  • Broad search is discovery, not daily evidence.
  • Community/front-page feeds need relevance filters.
  • Direct fetch failures need explicit classification.
  • Do not call a market quiet when important sources are blocked.