Haygen
Wiki/Constraints.md
Constraints — heygen-avatar
Explicit Non-Goals (MVP)
| Non-goal | Notes |
|---|---|
| Screen annotation / training mode | Phase 2. HeyGen has no native screen interaction. Would require annotation overlay layer, structured LLM output, and element targeting API. Fully designed in PRD Phase 2. |
| Built-in RAG / vector database | MVP injects full knowledge string into LLM context. No retrieval step. Sufficient for knowledge bases up to ~6,000 words. Larger corpora can use HeyGen Knowledge Base API (knowledgeId). |
knowledgeId support |
MVP delivers knowledge as knowledgeBase string. Pre-creating a reusable KB via HeyGen's API is a V2 enhancement. |
| Multi-avatar sessions | Single avatar per component instance. Concurrent avatars would require multiple hooks/components wired to separate token endpoints. |
| Built-in UI design / CSS | Component ships with minimal un-opinionated styling. Developer applies their own via className / style props. Intentional. |
| Built-in text input UI | useHeygenAvatar exposes speak() — developer builds their own text input. Component has no bundled text field. |
| Real-time LLM streaming into avatar | Avatar speaks complete sentences. Streaming LLM tokens → avatar is architecturally possible in V2 but not in MVP. |
| WebSocket / SSE support | Token proxy uses standard HTTP POST. No persistent connection. |
| Browser support below modern baseline | Requires WebRTC support. No IE, no legacy Safari. |
Known Limits
| Area | Limit | Notes |
|---|---|---|
| Knowledge base size | ~6,000 words recommended for Full mode | HeyGen injects full string into GPT-4o-mini context. Larger = degraded quality, not a hard error. Package warns on truncation. |
| Lite mode history | In-memory only | Conversation history lives in the attachLiteLoop closure. Clears on session stop. No persistence. |
| Lite mode concurrency | Single conversation thread | One pending transcript at a time. Rapid overlapping speech not handled. |
| Python module | No async streaming | create_session_token is async httpx — awaitable but not streaming. |
Dependencies we cannot control
| Dependency | What we rely on | What breaks if it changes |
|---|---|---|
@heygen/streaming-avatar v2.1.0 |
StreamingAvatar class API, StreamingEvents enum, createStartAvatar, speak, startVoiceChat, on/off |
Any breaking change to these methods requires a package update. Pinned to ^2.1.0. |
HeyGen /v1/streaming/create_token endpoint |
POST returns { data: { token } } |
Token extraction path data.data.token in session.py would break. Guard added in session.py. |
USER_END_MESSAGE event |
Has no message field — we accumulate from USER_TALKING_MESSAGE |
If HeyGen ever adds a message field to END, the accumulation pattern still works correctly. |
pdf-parse v2.4.5 |
PDFParse class-based API: new PDFParse({ data }) → .getText() → .text |
Major version changes may alter the API. |
cheerio |
{ load } + $('body').text() for HTML extraction |
Stable; unlikely to change. |