ResearchModule — Specification

Core engine for Intelligence Research OS: interactive scoping -> autonomous researcher (AAR) -> multi-tool deep-research fan-out -> synthesized plan with preserved provenance.

Layer: 1 (shared primitive — reusable across PRISM, LENS, CoE) Parent: BaseModule Status: Implementation started. Milestone 1A, 1B, 1C, UX gate, Milestone 2 AAR Planner, Milestone 3A Fan-out, and Milestone 4A Basic Synthesis foundations exist in /Users/arijitchowdhury/AI-Development/Research/src/research_module/ (see 2026-04-19-initial-design).

Product concept: Intelligence Research OS is the product/use-case layer. ResearchModule remains the core engine and Python package.

Purpose

Turns a vague user question into a fully-sourced synthesized plan, via four phases:

1. Scoper — interactive conversation that dissects the question, challenges assumptions, and produces a locked research brief
2. AAR Planner — always on; generates angles, critiques the brief, runs pre-research, writes research_plan.json, outline_seed.json, and research prompts
3. Research Supervisor / Fan-out — bounded research tasks across provider adapters. 3A ships Perplexity Sonar Pro first.
4. Synthesis Studio — extract atomic claims, build a final outline, write a basic cited report, and verify citations. Heavy multi-POV Synthesis remains 4B.

Automates the manual journey documented in sample.txt → Discovery-OS-v1.md.

The Intelligence Research OS layer uses this engine for repeatable intelligence missions: competitive intelligence, market pulse, opportunity discovery, content research, account research, customer voice, tool radar, and workflow-callable outputs.

Scheduling is explicitly external orchestration. Cron, Codex automations, GitHub Actions, Airflow, Temporal, n8n, Zapier, or application schedulers call the engine. The product moat is the intelligence layer: source packs, signal scoring, delta detection, opportunity interpretation, and evidence-backed output packs.

Sub-modules

Every sub-module: - Runs in its own isolated process (fresh context, direct vendor SDK calls) - Reads inputs from disk (Pydantic-validated) - Writes outputs to disk atomically (.tmp → validate → rename) - Advances state.json only on schema-validated success - Reports cost and duration to trace.log

Inputs

from pydantic import BaseModel, Field
from datetime import datetime

class ResearchModuleInput(BaseModel):
    """User-facing input. Passed from /research CLI to orchestrator."""
    version: str = "1.0.0"
    topic: str = Field(min_length=10, description="User's initial question — Scoper will refine")
    budget_cap_usd: float = 50.0
    time_cap_minutes: int = 120
    workspace_parent: str = "~/AI-Development/Research/runs"
    vault_slug: str | None = None  # optional: if topic matches a known project
    resume_run_id: str | None = None  # optional: resume interrupted run

Scoper's output — ResearchBriefV1 — becomes input to Phase 2. See Design doc §4.6.

Outputs

from pydantic import BaseModel

class ResearchModuleOutput(ModuleOutput):
    """Per-run deliverables. All artifacts on disk; ModuleOutput holds pointers."""
    version: str = "1.0.0"
    run_id: str
    status: str  # "success" | "partial" | "failed"
    failure_category: str | None  # F1..F5 per Manifesto
    plan_path: str  # 03-synthesis/plan.md — headline deliverable
    dossier_paths: list[str]  # 02-dossier/*.md — raw deep-research outputs
    scoper_brief_path: str  # 00-scoper-brief.md
    aar_artifacts: dict[str, str]  # 01-aar/* paths
    synthesis_artifacts: dict[str, str]  # 03-synthesis/*
    trace_log_path: str
    cost_report_path: str
    total_cost_usd: float
    duration_seconds: float
    evidence_tiers: dict[str, int]  # counts of 🟢/🟡/🔵 claims in plan.md

Config schema

class ResearchModuleConfig(BaseModel):
    # LLM model selection per phase
    scoper_model: str = "claude-opus-4-7"
    aar_angle_gen_model: str = "claude-sonnet-4-6"
    aar_critic_model: str = "claude-sonnet-4-6"
    aar_pre_researcher_model: str = "claude-opus-4-7"
    aar_prompt_composer_model: str = "claude-opus-4-7"
    synthesis_model: str = "claude-opus-4-7"
    compaction_model: str = "claude-haiku-4-5"
    canary_model: str = "claude-haiku-4-5"

    # Deep research adapter enablement
    enable_perplexity: bool = True
    enable_claude_dr: bool = True
    enable_openai_dr: bool = True
    enable_gemini_dr: bool = True

    # Circuit breakers
    budget_cap_usd: float = 50.0
    time_cap_minutes: int = 120
    phase_3_canary_first: bool = True

    # Scoper interview
    min_scoper_turns: int = 3
    first_compaction_turn: int = 20
    recurring_compaction_turns: int = 8
    compaction_ratio: float = 0.6

    # Retry
    max_retries_per_call: int = 3

Techniques / Patterns

1. Pure-code orchestrator. Deterministic control flow. LLMs are function calls, not decision-makers about what runs next.

2. Vendor-native SDKs. anthropic, openai, google-genai, httpx for Perplexity REST. No LLM-wrapping abstractions (no LangChain, no agent frameworks).

3. File-gated state machine. Every step has a precondition (previous step's output file exists + validates). No step advances until prior artifact hash is recorded in state.json.

4. Atomic writes. .tmp → Pydantic validation → rename. No partially-written files ever visible.

5. Circuit breakers. Schema-gate between phases, canary-first fan-out (1 cheap tool before firing all 4), budget + time caps.

6. Explicit context compaction. Scoper compresses conversation every 5 turns via a cheap Haiku call. Code-decided, not LLM-decided.

7. Strict citation gate. No source means the claim does not exist. Every accepted claim must link to evidence; every evidence item must link to a recorded source. Passing citation audits cannot contain invalid citations or unsupported claims.

8. Full traceability. trace.log records every LLM call, tool call, cost, duration, and decision. Not summarized.

9. Resumability. state.json is authoritative. Any crash/kill recoverable by re-invoking with --resume <run_id>. Completed steps never re-run.

10. 3-layer tests per TestingSOPs. Unit (mocked vendors), integration (real adapters with cheap models), e2e (full pipeline on a small real run, e.g., 3-sentence topic).

Definition of done

• [x] Gate 1A: Milestone 1 contracts — ResearchBriefV1, AAR Planner schemas, source/evidence registry schemas, synthesis schemas, CitationAudit, artifact envelope, workspace, and minimal state are implemented as Pydantic v2 models with strict validation.
• [x] Gate 1B: Runtime input + CLI/state foundation — ResearchModuleInput, RuntimeConfig, uv run research, persisted state.json, resume behavior, and EnvSecretsProvider skeleton.
• [x] Gate 1C: Live Scoper foundation — live-by-default Challenger Scoper CLI, Anthropic adapter, persisted conversation state, brief draft, source registry, Scoper tools, deterministic lock gates, compaction, and --offline-scoper fallback.
• [x] Gate 1D: UX architecture gate — CLI + skill wrapper first, power-researcher UX, command surface for status, runs, resume, show brief, show sources, and doctor; app/dashboard deferred.
• [x] Gate 2A: AAR Planner foundation — aggregate AAR artifact schemas, live-first injectable AAR runner, research continue, research show aar, AAR status phases, dossier source registry updates, and structured AAR failure artifacts.
• [x] Gate 2B: Intelligence Research OS roadmap — product-layer rename, use-case catalog, Intelligence Layer roadmap, source packs, signal/delta/output concepts, and UX screen concepts are documented.
• [x] Gate 2C: Intelligence OS mission foundation — Competitive Intel proof gate, GPT Image 2 UX prompt contract, mission-layer schemas, and research mission create/run/status/show output CLI foundation are implemented.
• [ ] Gate 2: Correctness — e2e test with a 3-sentence topic produces a valid plan.md + full dossier + complete state.json
• [ ] Gate 3: Operational — health check validates all 4 vendor SDKs; F1-F5 failures return ModuleOutput not exception; circuit breakers measured and verified
• [ ] Gate 4: Performance — e2e runs documented for 3 topic sizes (small/medium/large) with cost + duration budgets respected
• [ ] Gate 5: Documentation — README in repo, spec complete in vault, Design doc linked, example run checked into runs/example/

Delivery plan (approved 2026-04-19)

Approach 3 (paper-scale), incremental in 3 weekly milestones:

1. Milestone 1A: Foundation shipped 2026-05-20: Python scaffold, Q6 schemas, strict citation gate, atomic artifact store, workspace/state helpers, offline Scoper brief lock path.
2. Milestone 1B: Shipped 2026-05-20: CLI + current-working-directory confirmation prompt + state.json persistence + resume behavior + config/secrets scaffolding with .env.local.
3. Milestone 1C: Shipped 2026-05-20: live-by-default interactive Challenger Scoper foundation with persisted conversation/draft/source artifacts, tool calls, lock gates, and compaction.
4. UX Gate: Shipped 2026-05-21: CLI + skill wrapper UX contract before AAR, including status, runs, resume, show, and doctor commands.
5. Milestone 2: AAR Planner foundation shipped 2026-05-21: AAR schemas, live-first runner, pre-research source recording, CLI continuation/status/show behavior, and failure artifacts.
6. Milestone 3A: Shipped 2026-05-21: Fan-out Supervisor with Perplexity Sonar Pro, canary-first execution, source capture, raw outputs, evidence items, compressed notes, coverage gate, CLI continuation, and show dossier.
7. Milestone 4A: Shipped 2026-05-21: Basic Synthesis with deterministic claims, final outline, cited markdown report, strict citation audit, CLI continuation, and show report.
8. Milestone 3B: Multi-provider Fan-out triangulation — OpenAI, Anthropic, Gemini, richer retries, and provider-specific cost accounting.
9. Milestone 4B: Shipped 2026-05-21: Heavy multi-POV Synthesis Studio with isolated lenses, reconciliation, default CLI synthesis routing, report draft, citation audit, and POV/conflict counts.
10. Milestone 4C: Report polish — premium editorial report structure, richer contradiction handling, and stronger final narrative shaping.
11. Milestone 5A/5B: Shipped 2026-05-21: Competitive Intel proof gate, GPT Image 2 UX prompt contract, mission schemas, mission service, and mission CLI commands.
12. Milestone 5C: Competitive Intelligence vertical — source-backed signal classification, battlecard/digest generation, positioning shifts, sales talk tracks, and product implications.
13. Milestone 5D: Delta memory — first-run baseline and repeated-run delta reports.
14. Milestone 5E: Expand use cases in waves across Market Pulse, Content Research, GitHub/Tool Radar, Customer Voice, Sales/Account Intel, Product Strategy, Founder Opportunity Mining, Analyst/Narrative Intel, Hiring/Talent Signal, Pricing/Packaging Watch, and Regulatory/Policy Watch.
15. Milestone 6: Intelligence UX — Mission Control, Source Radar, Signal Inbox, Evidence Board, Opportunity Map, Battlecard, Content Workbench.
16. Milestone 7: Workflow / Agent Integration — CLI/API/skill callable missions, cron-compatible commands, JSON/Markdown outputs.

Technology stack (hard requirements)

• Language: Python 3.12+
• Mandatory modules: pydantic v2, ruff, pytest + pytest-asyncio, httpx, structlog, tenacity, uv, pyright --strict
• Vendor SDKs: anthropic, openai, google-genai (Perplexity via httpx + REST)
• No substitutes — see project memory project_python_stack.md for rationale

Related

• 2026-04-19-initial-design — full design decisions + rationale + brainstorm status
• Intelligence-Research-OS — product-layer concept and intelligence feature set
• Use-Cases — use-case catalog
• Competitive-Intel-Proof — first vertical proof gate
• Roadmap — roadmap phases
• UX-Research-Plan — UX research plan
• Screen-Concepts — screen concept inventory
• UX-Mockups-GPT-Image-2 — GPT Image 2 prompt contract
• Manifesto — system-wide rules this module inherits
• Module-Contract — base contract (execute, validate, health_check, ModuleResult)
• Evidence-Tier-Spec — the 🟢/🟡/🔵 tagging system used in plan output
• Adapter-Interfaces — adapter contracts for vendor SDKs
• CodingSOPs — coding discipline
• TestingSOPs — TDD, 3-layer tests
• FolderStructure — monorepo layout