Curriculum Research Agent — Repeatable Monthly Methodology (All US States)

This supersedes the prior single-source GA approach. The agent must DISCOVER authoritative sources per state (not hardcode one), scrape two tiers (SEA standards + district pacing), reconcile with frequency, export, and emit monthly deltas. Founder-provided; treat as the contract.

Inputs (runtime)

TARGET_STATE, TARGET_GRADES (default K/G1/G2), RUN_DATE, PREVIOUS_RUN_FILE (or null), OUTPUT_DIR.

Phases (run in order; mark each complete)

• Phase 0 — Bootstrap state config (run ONCE per state; skip if config exists — DECIDED: discover-once + cache, refresh periodically): 0.1 Identify SEA (name + domain; validate .gov/.org). 0.2 Identify standards framework name (GSE/TEKS/NGSS/NGSSS…). 0.3 Find official standards portal URL (must be SEA-owned). 0.4 Get district list (NCES nces.ed.gov; validate .k12.[st].us / verified district .org). 0.5 Save config_{STATE}.json {state, sea_name, sea_domain, standards_name, standards_portal_url, districts[], bootstrap_date}. Engine: live web search per the discovery step, validating official domains; cached to disk; refreshed periodically.
• Phase 1 — Tier-1 SEA standards (every run): scrape portal filtered to Science + grades. Per standard: standard_code, domain, grade_level, topic_short, full_description (verbatim), last_updated_date?, source_url. 1.2 SEA instructional resources. 1.3 revision detection vs previous run (ADDED/DESCRIPTION_UPDATED/DEPRECATED). Output: tier1_standards[], tier1_resources[], tier1_changes[].
• Phase 2 — Tier-2 district pacing (every run): for each district, find its science curriculum/pacing page (validate domain; reject 3rd-party/login). Extract per grade/unit: district_name, grade_level, unit_name, topic_title, pacing_period, standards_covered[], source_url, access_date, file_type. Log INACCESSIBLE (not_found/login_required/broken_link). 2.3 detect pacing/topic/url changes. Output: tier2_pacing_guides[], tier2_inaccessible[], tier2_changes[].
• Phase 3 — Dedup + reconcile: anchor each Tier-2 topic to a Tier-1 standard (exact code → keyword → domain+grade; else DISTRICT_ONLY flag). Frequency per {grade, standard_code}: district_count/total_districts → frequency_pct; label CONSENSUS ≥75% / COMMON 40–74% / DISTRICT-SPECIFIC <40%. Consensus pacing = mode of district pacing_periods (tie → "Varies by district"). Remove exact dups; one canonical record per {grade, standard_code}.
• Phase 4 — Export: canonical record per standard (see schema in source prompt) → science_curricula_{STATE}_{YYYY-MM}.json (with metadata header) + .csv (flat) + inaccessible_districts_{STATE}_{YYYY-MM}.csv.
• Phase 5 — Delta report: diff current vs PREVIOUS_RUN_FILE by {grade, standard_code} → delta_report_{STATE}_{YYYY-MM}.json (STANDARD_ADDED/DEPRECATED/DESCRIPTION_UPDATED/PACING_CHANGED/FREQUENCY_CHANGED/URL_CHANGED/TOPIC_ADDED/TOPIC_REMOVED). Console summary.

(Full verbatim phase spec: see the founder prompt pasted in the 2026-06-03 session log / dev-log.)

Persistence & upsert (corruption-safe) [DECIDED 2026-06-03]

• After voting/reconcile, the finalized list is stored to the DB with dates. Monthly re-runs must NOT create duplicates.
• Natural key = topic_id (state+grade+standard_code) → idempotent upsert (one row per standard; no dup rows).
• Preserve human approvals: re-research MERGES freshly-researched fields but keeps lifecycle status + validated_by UNLESS content materially changed; if changed → keep row, stamp last_updated/run_date, set a "needs re-review" flag (drives Phase-5 delta) rather than silently reverting PUBLISHED→DRAFT or duplicating.
• Add columns/fields: first_seen, last_updated, run_date, catalog_version. The DB is the durable record; JSON/CSV are per-run exports.

Scheduling / CLI [DECIDED 2026-06-03]

• A CLI entrypoint (e.g. python -m curriculum_service.cli research --state GA --grades K,G1,G2 --out <dir>), callable on demand and on a monthly schedule (cron). Runs Phase 0–5 for a state. Phase 0 cached (skip if config exists, --refresh to force).

Honest build risks

• Discovery + district scraping is the hard, uncertain part — real district sites vary, gate content, or publish PDFs; bot-detection is common. Build pluggable (web-search + fetch + LLM extraction; fakes for tests). GA verifies cleanly; other states will be structurally complete but need iteration.
• Tier-1 portals differ by state (GA=CASE API/JSON; others may be HTML/PDF). The portal reader must be generic with per-framework adapters; GA CASE adapter already exists.