Independent system · Desktop + live interaction
Snowmie
A realtime desktop system for operating a virtual on-stream character: live voice, interruption, local memory, broadcast context, avatar motion, and operator telemetry coordinated in one runtime.
- Format
- Electron desktop application
- Interface
- React Mission Control + Three.js state visualiser
- Runtime
- TypeScript · Node.js · WebSockets · Zustand
- Storage
- SQLite · sqlite-vec · local embeddings
- Connected systems
- Twitch · OBS Studio · VTube Studio · realtime voice
- Scope
- System design, implementation, interface, and verification by Briar
01 · Mission Control
Conversation, character state, and service health.
The desktop interface keeps the conversation, character state, and live-service health adjacent. It is an operating console rather than a chat window: each column exposes a different part of the running system.
Conversation
Activity, transcript, and Twitch chat remain separate so source and timing are legible during a stream.
- Input + output transcript
- Tool and memory activity
- Live chat context
Voice stage
The WebGL orb changes form and colour with idle, listening, processing, speaking, memory, and error state.
- Audio-reactive GLSL surface
- State + interruption control
- Input and typed-message controls
System telemetry
Mood, service condition, subjects, memory distribution, and chat activity remain visible without opening settings.
- Per-service health + reconnect
- Memory and subject signals
- Conversation activity
- 01Electron keeps service clients and persistent storage in the main process; React uses a typed preload command and event surface.
- 02IPC handlers route renderer commands to memory, audio, configuration, and live-service adapters.
- 03Connection failure degrades one adapter without collapsing the whole interface.
02 · Realtime orchestration
Turn-taking is a state machine.
A single synchronous transition table owns conversation state. Voice events, generated output, tool rounds, playback, interruption, and recovery all move through the same six phases.
One source of truth
Generation, playback, response origin, and pending calls live with the phase, not in parallel UI flags.
Synchronous cancellation
Interrupt and output-safety aborts are observable on the next line of the orchestrator call stack.
Lost events are bounded
Per-phase watchdogs clear a stuck turn instead of silencing every later event for the session.
03 · Audio + interruption
Audio ends the turn, not token generation.
The model can finish generating before the listener has heard the sentence. Snowmie waits for renderer playback to finish before starting cooldown or releasing queued speech, preventing one response from talking over another.
User speech interrupts output
Incoming speech aborts the active response, stops local playback, clears pending calls, and returns control to the user-speaking phase.
Grace and watchdog recovery
A response with no playback completes after a short grace window. A played turn with no finish signal recovers at 60 seconds rather than remaining wedged.
Three defensive layers
Prompt framing, transcript sanitisation, and immediate audio abort prevent internal tool tokens or private cues from reaching a live output.
04 · Local memory
Search, reinforcement, and maintenance remain inspectable.
Memories and subjects are stored locally, embedded on-device, ranked for recall, and exposed through a dedicated browser. Operators can search, inspect relationships, export data, and run maintenance directly.
- 01RecordFact · event · preference · relationship · episode
- 02Embed384-value local transformer representation
- 03PersistSQLite record + sqlite-vec cosine index
- 04RecallSimilarity, recency, and emotional weighting
- 05MaintainReinforce duplicates · decay · consolidate · clean up
If the packaged vector extension cannot load, writes continue and recall falls back to the complete in-memory embedding index.
05 · Live systems
Each service has its own boundary.
Twitch, OBS Studio, VTube Studio, and realtime voice connect through separate adapters. Health is reported as up, degraded, or down. Unaffected services continue running when one adapter fails.
Twitch
Chat reading, sentiment, follows, cheers, subscriptions, raids, redemptions, and stream-state events. EventSub reconnects overlap sockets until handoff completes, then deduplicates by message ID.
OBS Studio
The WebSocket adapter tracks scenes and captures frames for optional visual context. Authentication failures stop automatic retries; unexpected disconnects use bounded backoff.
VTube Studio
Speech and emotional state drive mouth shape, expression, blinking, head movement, body motion, and small idle behaviours through the plugin API.
Mission Control
Connection state, transcript, chat, mood, memory activity, service health, and controls stay visible in one desktop interface. Services can reconnect without restarting the application.
Only enumerated, leak-safe reasons enter model-readable status context; raw service errors remain in operator diagnostics.
06 · Verification
Offline tests exercise the realtime client and orchestrator together.
The test harness drives the real realtime client and orchestrator with scripted events. It checks the failures that matter in a live session: doubled responses, interrupted playback, unsafe output, dropped completion, and reconnect churn.
tests passing across 11 files
- TypeScript
- type-check passes
- Production
- Electron build passes
- Lint
- zero errors
- CI definition
- Windows · Node 22
Verification above was run against the private source snapshot used for this case study. External voice, Twitch, OBS, and VTube Studio services were not contacted during the offline suite or interface capture.