Dashboard/Algolia-Central/Knowledge/AgentStudio/11-decision-rubrics.md
Algolia-Central

Knowledge/AgentStudio/11-decision-rubrics.md

11 — Decision Rubrics

When-to-do-X trees and tradeoff tables. For when you're in the middle of a refactor and need to make a fast call.

"Should I add a tool to this agent?"

1. Does the output have STRUCTURE the frontend would render specially? (cards, tables, diagrams, code blocks)
   YES → consider a tool
   NO  → keep in narrative text. STOP.

2. Is the structure renderable as plain markdown well? (headings, bullets, code fences)
   YES → keep in narrative markdown. STOP.
   NO  → tool is the right fit. CONTINUE.

3. Will any frontend reuse the structured data? (e.g., copy a code block, save a comparison)
   YES → tool. CONTINUE.
   NO  → consider markdown. (But if step 2 said no, tool wins anyway.)

4. Does the tool result need to feed back into the agent's reasoning?
   YES → client_side tool with addToolResult. CONTINUE.
   NO  → still client_side, but result can be `{ ok: true }` ack. CONTINUE.

5. Are we already at 5 tools on this agent?
   YES → consider consolidating into a polymorphic tool with a `kind` enum. ALSO consider whether the agent is doing too much.
   NO  → add the tool.

"Should I add an index to this agent?"

1. Does the new content have a different ranking model than existing indices?
   (different popularity signals, freshness rules, custom ranking)
   YES → split index. CONTINUE.
   NO  → use a filter on the existing index. STOP.

2. Does the new content have meaningfully different facets / searchable attributes?
   YES → split index. CONTINUE.
   NO  → reconsider — maybe filter is enough.

3. Will the volume cause the index to grow >2x? (rule of thumb: split big content types out)
   YES → split.
   NO  → either is fine.

4. Are we already at 8 indices on this agent's tool?
   YES → consolidate something OR split this content into a separate agent.
   NO  → add the index.

5. Is the routing rule for this content easily expressible?
   ("If user asks about X, prefer this index.")
   YES → add the index + update Multi-Intent Handling section.
   NO  → reconsider — overlapping content is harder to route.

"Should I split one agent into multiple?"

1. Do the agents share most tools and indices?
   YES → don't split — add intent recognition for the new role inside the existing agent. STOP.
   NO  → CONTINUE.

2. Are the prompts so different that one prompt couldn't hold both?
   YES → split. CONTINUE.
   NO  → don't split. STOP.

3. Will users SEE the role distinction? (different name, avatar, tone)
   YES → split is reasonable.
   NO  → don't split — internal complexity not exposed to user is wasted work.

4. Do the agents need different platform features? (one needs memory, other doesn't; one needs filterSuggestions widget)
   YES → split.
   NO  → consider keeping merged.

5. Will splitting introduce inter-agent latency that the user will notice?
   YES → don't split unless step 3 is a hard yes.
   NO  → split is fine.

DEFAULT: don't split. The deprecated multi-agent pattern (§09) means the platform's evolution is toward fewer agents.

"Should I use platform feature X or build custom?"

Platform feature Build custom when
memory.enabled (conversation persistence) You need rich state beyond chat history (8-signal discovery, qualification, tracked metadata across turns) — keep custom for that, but use platform memory for chat history regardless
suggestions.enabled (follow-up prompts) Almost never. Only build custom if you need suggestions to drive frontend state (clickable chips that pre-fill UI, not just chat input)
Caching Almost never. Just leave useCache: true. If you need different cache behavior per request, use ?cache=false query param
Secured user tokens Almost always use platform — generating JWTs with filter restrictions is much safer than building per-row authorization in your code
Personalization (enablePersonalization) Build custom if your personalization model is fundamentally non-Algolia (your own ML model). Otherwise, use platform
Conversation IDs (alg_cnv_, alg_msg_) Always use platform IDs. Custom IDs gain you nothing

Rule of thumb: Use platform features unless you have a SPECIFIC reason custom is needed. Default to platform.

"Single agent with N indices vs multiple specialist agents?"

See 06-multi-index-routing for full analysis. Short version:

1. Is N <= 10?
   YES → single agent with N indices.
   NO → reconsider — too much for one agent. Maybe regroup content first.

2. Do the indices share a common audience and synthesis style?
   YES → single agent. STOP.
   NO  → CONTINUE.

3. Do they have wildly different output formats? (one needs Mermaid, one needs JSON, one needs code)
   YES → single agent with multiple render TOOLS, not multiple agents.
   NO  → single agent.

4. Do they have different regulatory boundaries? (HIPAA vs SOX, different audit/retention)
   YES → split agents.
   NO  → don't split.

DEFAULT: single agent with N indices. The platform supports it cleanly. Multi-agent is rarely the answer.

"How many tools should this agent have?"

- Pure text agent (classifier, query builder, suggestion generator): 0 tools
- AskAI / DocSearch: 1 search tool with N indices, 0-3 client-side render tools (target: 2-4 total)
- PDP product expert: 2-3 tools (search + suggest_searches + optional outfit-builder)
- Shopping assistant: 11-13 tools (10-tool core + vertical-specific)
- Filter Suggestions agent: 0 tools

If you're between archetypes and unsure, default to "fewer tools, richer prompts." Tools are easy to add later. Removing tools after the prompt depends on them is harder.

"Which model should I use?"

Per LLM Leaderboard data (06-blogs-and-use-cases.md §4):

1. What's the latency budget?
   <5s   → gemini-3-pro-preview, gemini-2.5-flash-lite
   5-10s → gpt-4.1-mini, gemini-2.5-flash, claude-haiku-4-5
   10-30s → gpt-5, gpt-5-mini, claude-sonnet-4-5
   No budget → gpt-5.1, claude-opus-4-5

2. What's the cost budget?
   Sub-penny → gemini-2.5-flash-lite (best quality/cost ratio per leaderboard)
   1-10 cents → gpt-4.1-mini, gpt-5-mini, claude-haiku-4-5
   10+ cents → gpt-5, claude-sonnet-4-5
   $1+ per query → claude-opus-4-5 (luxury option)

3. Does the task need extended thinking / reasoning?
   The leaderboard shows extended thinking can HURT quality on agent tasks. Default OFF.
   Only enable for complex multi-step reasoning where the agent has many tools and the orchestration matters.

4. Quality threshold?
   Compare candidates on YOUR data, not leaderboard. The leaderboard says "the #1 model is often the wrong choice for your agent." Validate.

For our agents: - Elena, Bruno: gemini-2.5-flash today; gemini-2.5-flash-lite is cheaper/faster and likely as good. Test before switching. - Maverick: gemini-2.5-flash-lite already (changed during Summit incident). Stay there. - Suggestions feature: gemini-2.5-flash-lite — fast, cheap, good enough for follow-up prompts.

"Should I keep the Maverick → Elena/Bruno handoff or merge?"

1. Is Maverick's role (discovery, value-selling) genuinely different from Elena/Bruno's (technical depth)?
   YES → keep separate roles. CONTINUE.
   NO  → merge into one agent. STOP.

2. Does the user benefit from seeing the handoff explicitly?
   ("Now Bruno will take over with the architecture deep-dive")
   YES → keep handoff visible.
   NO  → make handoff invisible (no UI signal) — agent switches under the hood.

3. Does the handshake step (scope confirmation before execute) earn its 2-call latency cost?
   YES → keep handshake.
   NO  → drop handshake. Specialist starts executing immediately on receipt of scope.

DEFAULT: keep roles separate, drop handshake step (cost > value).

"When is breaking change in beta widget acceptable?"

The native <Chat> widget is BETA. Breaking changes are possible in minor versions. Decision:

- Upgrading to a new version → read the changelog. Pin the version in package.json. Test in dev before deploying.
- Building new features ON the widget → do it on a pinned version. Don't blindly auto-upgrade.
- Encountering a breaking change → if the platform's new behavior is better, adopt; if not, stay pinned and revisit at the next major release.
- Critical incident caused by widget upgrade → revert to pinned version. Don't try to debug a beta widget under fire.

What this section does NOT cover