Provider Credit Incident And CI Fail-Closed Repair

Date: 2026-06-28 Status: Implemented and deployed to Chowmes

What Happened

Athena appeared to fail in Telegram with a generic model-provider message. The live provider error was OpenRouter HTTP 402 caused by exhausted credits, not an Athena identity or Telegram routing failure.

The OpenRouter credit check reported:

• Total credits: $35.0000
• Total usage: $35.2587
• Remaining: approximately -$0.2587

The same provider powers Athena/default Telegram, Vulcan, and Competitive Intelligence synthesis, so the failure could affect multiple agents and scheduled intelligence jobs.

Token Burn Findings

The large token usage was not primarily a hidden idle process. It came from long Telegram/operator sessions with large context and tool loops.

Top token sessions found in Hermes logs:

Model-level usage found in logs:

Interpretation: the spend problem is context/tool-loop control and provider budgeting, not simply the CI dashboard or a single scheduled job.

CI Silent Failure Root Cause

The daily and weekly CI runners caught synthesize_with_hermes failures, printed a warning, fell back to local synthesis, saved artifacts, and returned success.

That behavior is acceptable for local review. It is not acceptable for production CI, because it can make a provider failure look like a real successful intelligence run.

Repairs Implemented

Gateway Provider Error Clarity

The live Hermes gateway now classifies OpenRouter credit failures separately from generic provider failures. HTTP 402 / insufficient-credit errors produce a Telegram message saying that model-provider credits are exhausted or below required balance and that raw diagnostics are in gateway logs.

Live backup before patch:

/opt/hermes/gateway/run.py.bak.credit-message-20260628231108

Follow-up correction:

The first credit-specific message was still too robotic and could appear twice in Telegram. Root cause: the gateway sanitized provider errors in both the status-callback path and final-response path. Under provider failure, Telegram could receive both.

The live gateway was patched again so Telegram provider-failure status callbacks return None; only the final sanitized response is delivered. The credit failure line was also rewritten to match Athena's operational voice:

Arijit, I’m blocked before I can think: OpenRouter credits are exhausted. Add credits or switch my provider/model, then message me again. I’ve kept the raw provider details in the gateway logs.

Second live backup:

/opt/hermes/gateway/run.py.bak.credit-voice-20260628234617

CI Production Fail-Closed

Added:

skills/competitive-research/scripts/ci-provider-preflight.py

Daily and weekly production wrappers now:

• Load both /opt/data/.env and /root/.hermes/.env.
• Run OpenRouter credit preflight before CI synthesis.
• Use --fail-on-synthesis-error.
• Exit 75 instead of publishing a fallback report when model synthesis fails.

Chowmes Provider Credit Watch

Added:

scripts/chowmes-provider-credit-watch

Deployed to:

/opt/data/scripts/chowmes-provider-credit-watch

Hermes cron:

• ID: e19777169ce2
• Name: chowmes-provider-credit-watch
• Schedule: 45 8 * * *
• Delivery: Telegram
• Mode: no-agent

Purpose: alert before the 09:00 CI jobs if OpenRouter credits are below the configured floor.

Verification

Local CI tests in the Competitive Intelligence skill passed:

28 passed in 0.57s

Dashboard static build passed:

Dashboard static seed verified.

VPS wrapper checks intentionally failed closed because credits are still exhausted:

• Daily wrapper exit: 75
• Weekly wrapper exit: 75

Forced Hermes cron tick set the daily CI job to error with a clear credit-block message instead of success.

Chowmes health check passed after gateway/session repair, and Telegram delivery smoke test succeeded.

Remaining Blocker

OpenRouter credits are still exhausted. Until credits are added or the provider/model is changed, the next production CI run should fail closed and notify instead of publishing a trusted report.

Decision

Production CI must not silently fall back to local synthesis after provider failure. Fallback artifacts are allowed for developer review only. Scheduled intelligence must either publish a trusted model-backed report or fail closed with a reason.