Sub-15ms Acoustic Synthesis: Voicing an On-Device Anime Companion
How we engineered a streaming phonetic synthesis pipeline that runs on Apple Silicon unified memory with under 15ms time-to-first-token audio feedback.
When an engineer writes code at 2 AM, latency is the difference between a natural presence and an annoying notification. If an alter-ego takes two seconds to respond to a failing unit test, the flow state has already shattered.
Kanna's acoustic voice pipeline was designed from day one to operate under a strict budget: 15 milliseconds from token emission to DAC audio output.
Eliminating Cloud Round-Trips
Conventional voice synthesis models require compressing text, transmitting HTTP payloads across continents, awaiting GPU cluster scheduling, and streaming raw PCM buffers back to the client. This introduces a baseline latency of 600ms to 1200ms.
Client (IDE) ---> Cloud Gateway (80ms) ---> Model Queue (350ms) ---> Vocoder (200ms) ---> Audio Buffer (150ms)
Total Latency: ~780ms (Unacceptable for cognitive mirroring)
Kanna bypasses the network entirely by deploying a quantized 4-bit acoustic vocoder directly onto Apple Silicon MPS unified memory or NVIDIA TensorRT cores:
Local Terminal Hook ---> Local Enclave (4.2ms) ---> Quantized Vocoder (6.8ms) ---> DAC Buffer (1.2ms)
Total Latency: 12.2ms (Zero network overhead)
Phonetic Intonation and Restraint
A companion that talks too much quickly becomes intolerable. Kanna’s speech synthesis is decoupled into three levels of vocal expression:
- Level 0 (Silent): Visual state transitions only (earring shimmer, cat tail twitch).
- Level 1 (Soft Phonetic Murmur): Non-verbal acknowledgments (
mm,hmm) during successful compilation passes. - Level 2 (Spoken Synthesis): Direct, concise sentence audio when a critical build deadlock or high-severity syntax regression is detected.