Architecting Sub-300ms Voice AI Agents: From SIP Codecs to Gemini Live

1. The Latency Bottleneck in Legacy Voice Bots

In conversational human interaction, **latency is everything**. Human turn-taking physics dictate that pauses exceeding 500 milliseconds feel awkward, forced, and disorienting. To achieve natural human parity, Voice AI platforms must maintain an end-to-end response latency under 300 milliseconds.

Legacy telephony voicebot pipelines suffer from cumulative sequential latency. A traditional system routes PSTN G.711 audio through a cloud ASR engine, waits for complete sentence transcription, feeds text to an LLM, waits for complete text generation, and finally pushes text to a TTS engine before playing back synthesized audio.

2. Zero-Buffer PCM & Opus Streaming

To break through the 300ms barrier, Dialiqo abandons intermediate text transcriptions for continuous audio streaming. Audio packets are sliced into tight 20ms PCM frames, compressed using the Opus codec, and streamed via bi-directional WebSockets directly to neural speech models like Gemini Live and ElevenLabs S2S.

By processing audio as a continuous liquid stream rather than discrete text blocks, the neural model begins generating audio tokens while the human speaker is finishing their final word.

// Dialiqo High-Throughput Audio Frame Socket Connector
import { WebSocket } from 'ws';
import { OpusEncoder } from '@discordjs/opus';

export class VoiceAISocketPipeline {
  private encoder = new OpusEncoder(16000, 1);
  private socket: WebSocket;

  constructor(serverUrl: string, apiKey: string) {
    this.socket = new WebSocket(serverUrl, {
      headers: { 'Authorization': `Bearer ${apiKey}` }
    });
  }

  public pushPCMFrame(pcmBuffer: Buffer): void {
    // Slice raw 16kHz PCM into 20ms (640-byte) frames
    const encodedFrame = this.encoder.encode(pcmBuffer);
    if (this.socket.readyState === WebSocket.OPEN) {
      this.socket.send(encodedFrame, { binary: true });
    }
  }
}

3. FreeSWITCH C-Module WebSocket Engine

Standard SIP softswitches are built for static RTP relaying, not high-frequency WebSocket duplexing. We wrote a lightweight, native C-module (`mod_dialiqo_s2s`) directly inside the FreeSWITCH C-core.

This module taps raw RTP audio streams directly from the memory bus, bypassing user-space copy overheads and piping raw 16kHz audio directly into edge cloud connectors.

4. Voice Activity Detection & Instant Muting

A key hallmark of natural human conversation is the ability to interrupt. If a caller says "Wait, hold on" mid-sentence, the AI agent must immediately stop talking.

We implement an ultra-low latency Spectral Power Voice Activity Detector (VAD) running directly on edge Kamailio proxies. The moment speech power exceeds threshold for 2 consecutive frames (40ms), an out-of-band `CLEAR_AUDIO_BUFFER` SIP message flushes the outbound speaker queue.

"Interruption handling is not an aesthetic luxury — it is a foundational safety requirement for healthcare and financial telephone systems."

— Alexei Petrov, Chief VoIP & AI Architect

5. Production Benchmarks & Architectural Rules

Across 4.2 million benchmarked calls in production enterprise environments, the Dialiqo sub-300ms architecture achieved an average round-trip audio latency of **242 milliseconds**.

Key takeaways for telecom engineers: co-locate SIP media servers in the same cloud availability zones as AI inference GPUs, use Opus 16kHz audio, and enforce hardware-backed VAD flush triggers.