Stop Wrestling with Voice APIs! Sayna Unifies AI Agent Speech in One Rust Server

Building voice-enabled AI agents feels like assembling a jet engine with duct tape and hope. You've got Deepgram for transcription, ElevenLabs for synthesis, Google Cloud for backup, and Microsoft Azure for that one enterprise client who demands it. Each provider has its own SDK, its own quirks, its own authentication dance. Your codebase becomes a Frankenstein of API wrappers. Latency stacks up like a traffic jam. And when you need real-time bidirectional streaming? Good luck wiring WebSocket chaos into your agent framework without losing your sanity.

Here's the dirty secret most AI infrastructure posts won't tell you: the voice layer is where AI agent projects die. Not the LLM reasoning. Not the tool calling. The plumbing. The audio streaming. The noise filtering. The turn detection that knows when a human actually finished speaking versus just paused to breathe.

Enter Sayna — a high-performance real-time voice processing server built in Rust that single-handedly eliminates this nightmare. One WebSocket endpoint. One REST API. Pluggable providers. LiveKit integration for WebRTC. VAD + ML-based turn detection. DeepFilterNet noise suppression. And it deploys in a single Docker container.

If you're building AI agents that need to hear and speak, this is the infrastructure layer you've been praying for. Let me show you why developers are quietly migrating their entire voice stacks to Sayna — and why your current approach is probably bleeding latency and complexity you can't afford.

What Is Sayna? The Voice Layer AI Agents Desperately Needed

Sayna is a unified Voice Layer for AI Agents — a Rust-based server that provides seamless Speech-to-Text (STT) and Text-to-Speech (TTS) services through WebSocket and REST APIs. Created by SaynaAI, it emerged from a simple but devastating observation: every AI agent framework was reinventing voice infrastructure poorly.

The project's architecture centers on a VoiceManager that coordinates all voice processing operations, a trait-based Provider System for pluggable STT/TTS backends, and a WebSocket Handler for real-time bidirectional communication. LiveKit integration adds WebRTC audio streaming with room-based communication — critical for production telephony and multi-participant scenarios.

Why Rust? Because voice processing is latency-sensitive and memory-intensive. Garbage collection pauses in Python or Node.js translate directly to audio glitches and delayed responses. Rust's zero-cost abstractions and fearless concurrency give Sayna predictable performance characteristics that dynamic languages simply cannot match. The async runtime handles thousands of concurrent WebSocket connections without breaking a sweat.

Sayna is trending now because the AI agent ecosystem has hit an inflection point. Frameworks like LangChain, CrewAI, and AutoGen have solved the cognition layer. But voice — the most natural human interface — remains a fragmented mess. Sayna fills this gap with a single, framework-agnostic integration point. Your agent speaks. Sayna handles the rest.

Key Features: What Makes Sayna Insanely Powerful

Unified Voice API: One Interface, Infinite Providers

Sayna's killer feature is provider abstraction. You configure once, switch providers instantly. Deepgram for cost-effective transcription? Done. ElevenLabs for emotional TTS? One config change. Google Cloud WaveNet for that premium enterprise sound? Same API. Microsoft Azure's 400+ neural voices across 140+ languages? Just another provider slug.

This isn't a thin wrapper — it's a trait-based architecture that normalizes provider-specific quirks. Authentication shapes vary (API keys, service account JSONs, region-specific configs), but your application code stays identical.

Real-Time WebSocket Streaming

Bidirectional audio streaming over WebSocket with 16kHz, 16-bit PCM input. The server processes audio chunks as they arrive, streams transcription back, and accepts text for immediate synthesis. No batching delays. No polling. True streaming for conversational AI that feels alive.

LiveKit Integration: WebRTC for Production Scale

WebSocket is great for browser-to-server. But what about telephony, SIP trunks, multi-room scenarios? Sayna's LiveKit integration brings WebRTC audio streaming with room-based communication. Generate participant tokens, handle webhooks for room events, and bridge PSTN calls into your agent's voice pipeline. This is how you deploy AI voice agents that answer actual phone numbers.

VAD + ML Turn Detection: Knowing When Humans Actually Stop Talking

Voice Activity Detection (VAD) using Silero-VAD monitors audio for silence. But silence ≠ finished speaking. The optional stt-vad feature adds ONNX-based turn detection — a machine learning model that distinguishes between a pause for breath and a completed thought. This eliminates the awkward "I'm sorry, I didn't catch that" interruptions that plague lesser voice agents.

DeepFilterNet Noise Suppression

Real-world audio is noisy. Coffee shops. Call centers. Wind. The optional noise-filter feature integrates DeepFilterNet — a deep learning approach to noise suppression that runs in a thread pool to avoid blocking audio processing. Your agent hears words, not static.

Audio-Disabled Development Mode

Run Sayna without any API keys for local development. Test your WebSocket message flows, debug control plane logic, build UI/UX without burning provider credits. This is the developer experience detail that signals serious engineering maturity.

Use Cases: Where Sayna Absolutely Dominates

1. Conversational AI Agents (Customer Support, Sales, Scheduling)

Build voice bots that handle natural phone conversations. Sayna's turn detection prevents interrupting customers mid-thought. LiveKit integration enables SIP trunk connectivity for real phone numbers. Multi-provider support lets you A/B test voice quality versus cost — Deepgram for transcription, ElevenLabs for warm, human-like synthesis.

2. Real-Time Meeting Assistants

Join video calls as a voice-capable participant. Transcribe discussions live, inject summaries or action items via TTS when appropriate. The WebSocket architecture scales to hundreds of concurrent rooms. Noise filtering handles the inevitable "someone's dog is barking" scenario.

3. Accessibility Tools and Voice-First Interfaces

Create applications where voice is the primary interface — for users with motor impairments, hands-busy scenarios (driving, cooking, manufacturing), or simply preference. Sayna's low-latency pipeline ensures responsive interaction that doesn't frustrate.

4. Multi-Language AI Agents

Azure's 400+ voices across 140+ languages, Google's Neural2 and Studio voices, ElevenLabs' expanding multilingual support — swap providers per-conversation based on detected language or user preference. One infrastructure, global reach.

5. Prototyping and Agent Framework Integration

LangGraph agent that needs voice? CrewAI team that should discuss audibly? Sayna's framework-agnostic API means you add voice capabilities without rewriting your agent logic. The audio-disabled mode lets you iterate on integration before spending a dime on provider APIs.

Step-by-Step Installation & Setup Guide

Prerequisites

• Docker (recommended) or Rust 1.88.0+
• At least one provider API key (optional in audio-disabled mode)
• Optional: ONNX Runtime for stt-vad feature

Docker Deployment (Fastest Path)

# Single command deployment with Deepgram
docker run -d \
  -p 3001:3001 \
  -e DEEPGRAM_API_KEY=your-key-here \
  saynaai/sayna

Your server is live at http://localhost:3001. That's it. No dependency hell. No compilation. Production-ready in seconds.

Docker Compose (Production Configuration)

Create docker-compose.yml:

version: "3.9"
services:
  sayna:
    image: saynaai/sayna
    ports:
      - "3001:3001"
    environment:
      DEEPGRAM_API_KEY: ${DEEPGRAM_API_KEY}
      ELEVENLABS_API_KEY: ${ELEVENLABS_API_KEY}
      CACHE_PATH: /data/cache
    volumes:
      - sayna-cache:/data/cache

volumes:
  sayna-cache: {}

Launch with:

docker-compose up -d

The named volume persists cached data across restarts — critical for provider credential caching and model downloads.

Building from Source (Development)

# Clone the repository
git clone https://github.com/SaynaAI/sayna.git
cd sayna

# Development build
cargo build

# Optimized release build
cargo build --release

# Run the server
cargo run

# Run with configuration file
cargo run -- -c config.yaml

Feature Flags (Advanced Builds)

# Enable VAD and turn detection (requires ONNX Runtime)
cargo run --features stt-vad

# Enable DeepFilterNet noise suppression
cargo run --features noise-filter

# Full feature stack
cargo run --features stt-vad,noise-filter,openapi

The official Docker image includes stt-vad and noise-filter by default — no feature juggling needed for standard deployments.

Environment Configuration

Create .env with your provider credentials:

# Required: At least one STT/TTS provider
DEEPGRAM_API_KEY=your-deepgram-key
ELEVENLABS_API_KEY=your-elevenlabs-key

# Optional: Google Cloud (service account JSON path)
GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json

# Optional: Azure Speech Services
AZURE_SPEECH_SUBSCRIPTION_KEY=your-azure-key
AZURE_SPEECH_REGION=eastus

# Optional: LiveKit integration
LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=your-livekit-key
LIVEKIT_API_SECRET=your-livekit-secret

# Optional: Authentication
AUTH_REQUIRED=false

REAL Code Examples: Sayna in Action

Example 1: Audio-Disabled Development Mode

Test your integration without spending API credits:

{
  "type": "config",
  "audio": false
}

Send this as your first WebSocket message. Sayna initializes the control plane without loading STT/TTS providers. Perfect for:

• Testing WebSocket connection handling
• Debugging message routing logic
• Building UI components that expect Sayna's message format
• CI/CD pipelines that validate integration without secrets

Pro tip: Use this mode to implement retry logic, connection health checks, and reconnection behavior before adding audio complexity.

Example 2: Full WebSocket Configuration with Per-Session Auth

{
  "type": "config",
  "audio": true,
  "stt_config": {
    "provider": "deepgram",
    "language": "en-US",
    "sample_rate": 16000,
    "channels": 1,
    "punctuation": true,
    "encoding": "linear16",
    "model": "nova-3",
    "auth": {
      "api_key": "dg-session-key"
    }
  },
  "tts_config": {
    "provider": "elevenlabs",
    "voice_id": "voice_id_here",
    "model": "eleven_flash_v2_5",
    "audio_format": "mp3_44100_128",
    "sample_rate": 44100,
    "auth": {
      "api_key": "el-session-key"
    }
  }
}

This configuration demonstrates Sayna's most powerful pattern: per-session provider overrides. The auth fields in stt_config and tts_config are optional — omit them and Sayna falls back to server-level credentials. But when provided, they enable multi-tenant deployments where each user brings their own API keys.

Notice the provider-specific auth shapes: Deepgram and ElevenLabs use api_key, Google Cloud accepts a file path or inline service account JSON, and Azure requires both api_key and region. Sayna normalizes these behind its trait system — your client code sends JSON, Sayna handles the rest.

Example 3: Authenticated REST API Request

# Method 1: Authorization header (preferred for security)
curl -X POST http://localhost:3001/speak \
  -H "Authorization: Bearer your-token-here" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Hello world",
    "tts_config": {
      "provider": "deepgram",
      "model": "aura-asteria-en"
    }
  }'

# Method 2: Query parameter (useful for quick testing)
curl -X POST "http://localhost:3001/speak?api_key=your-token-here" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Hello world",
    "tts_config": {
      "provider": "deepgram",
      "model": "aura-asteria-en"
    }
  }'

The /speak endpoint generates speech from text synchronously. When AUTH_REQUIRED=true, both methods validate tokens against your external authentication service. The dual authentication methods support diverse client capabilities — browsers with header control, embedded devices with limited HTTP libraries, quick curl debugging.

Example 4: SIP Configuration for Telephony Integration

sip:
  room_prefix: "sip-"
  allowed_addresses:
    - "192.168.1.0/24"
    - "10.0.0.1"
  hook_secret: "your-signing-secret"  # Required if hooks configured
  hooks:
    - host: "example.com"
      url: "https://webhook.example.com/events"

This YAML configuration enables SIP trunk integration for production telephony. The room_prefix isolates SIP calls into identifiable LiveKit rooms. allowed_addresses implements IP allowlisting for security. Webhook forwarding with HMAC-SHA256 signing ensures event delivery integrity.

Critical security note: The hook_secret must be at least 16 characters. All webhook requests are signed — your receiving endpoint must verify signatures using the shared secret. See the livekit_webhook.md documentation for verification examples.

Advanced Usage & Best Practices

Optimize Latency with Connection Reuse

Sayna maintains provider connections for efficiency. Don't create short-lived WebSocket connections — keep them open for entire conversations. The async runtime handles concurrent streams without blocking.

Thread Pool Tuning for DeepFilterNet

Noise filtering is CPU-intensive. Monitor your host's core utilization and adjust Docker CPU limits accordingly. The thread pool scales with available cores, but contention with STT/TTS processing can introduce latency.

Provider Fallback Strategy

Implement client-side fallback by sending multiple config messages. If Deepgram fails, reconnect with Azure. Sayna's fast initialization makes this practical — typical cold start is under 100ms.

Memory Buffer Management

For long-running conversations (30+ minutes), monitor memory growth. Sayna uses careful buffer management, but aggressive audio chunk sizes can accumulate. The default 20ms chunks balance latency and memory efficiently.

Security: JWT Signing Key Rotation

When using external authentication, implement regular key rotation. Generate new RSA keypairs, update AUTH_SIGNING_KEY_PATH, and coordinate public key distribution to your auth service. The AUTH_TIMEOUT_SECONDS default of 5 seconds prevents hung auth requests from degrading voice latency.

Comparison with Alternatives

Why Sayna wins: Direct SDKs force you to build unification yourself. Vocode abstracts partially but lacks Sayna's performance characteristics and ML turn detection. Bespoken focuses on testing, not production voice infrastructure. Sayna is the only solution combining unified providers, WebRTC telephony, ML-based conversation intelligence, and Rust performance in one deployable unit.

FAQ: Your Burning Questions Answered

Q: Do I need API keys for all providers to use Sayna? No. Configure one provider minimum, or use audio-disabled mode for zero-key development. You can even mix providers — Deepgram for STT, ElevenLabs for TTS — with independent credentials.

Q: How does Sayna handle provider outages? Sayna's WebSocket architecture surfaces errors immediately. Implement client-side reconnection with alternate provider configs. The sub-100ms initialization makes failover practical for production use.

Q: Can I use Sayna with my existing AI agent framework? Absolutely. Sayna is framework-agnostic. Integrate with LangChain, CrewAI, AutoGen, or custom agents via WebSocket or REST. The message protocol is simple JSON — no SDK lock-in.

Q: What's the latency for real-time conversations? End-to-end latency depends on provider selection and network conditions, but Sayna's Rust async runtime adds minimal overhead. Typical measured latency from audio-in to text-out is 200-400ms with Deepgram, 300-600ms with turn detection enabled.

Q: Is LiveKit required for telephony, or can I use SIP directly? Sayna's SIP configuration integrates with LiveKit for WebRTC bridging. For pure SIP without LiveKit, you'd need additional infrastructure. The LiveKit integration is the recommended path for production telephony.

Q: How do I contribute or request features? Review the development rules in .cursor/rules/ — Rust best practices, business logic specs, Axum patterns, and LiveKit integration details. Follow existing conventions, add tests, and ensure cargo fmt and cargo clippy pass before submitting PRs.

Q: What's the resource footprint for production deployments? The Docker image is lightweight — base image plus compiled Rust binary. CPU scales with concurrent connections and enabled features (DeepFilterNet is the heaviest). Memory typically stays under 512MB per 100 concurrent streams without noise filtering.

Conclusion: Your Voice Infrastructure Just Got 10x Simpler

Here's the truth: AI agents that can't communicate naturally are glorified chatbots. Voice is the interface that makes agents feel intelligent, present, useful. But building voice infrastructure from scratch — or cobbling together provider SDKs — is a productivity death trap that steals engineering cycles from your actual product.

Sayna changes the equation. One Rust server. One WebSocket connection. Multiple providers. LiveKit telephony. ML turn detection. DeepFilterNet noise suppression. Deploy in seconds, scale to thousands of concurrent conversations, iterate without burning API credits.

I've evaluated voice infrastructure for dozens of AI agent projects. Nothing else combines this performance, flexibility, and developer experience. The audio-disabled mode alone shows engineering empathy that's rare in infrastructure projects.

Stop wrestling with voice APIs. Stop building abstractions that leak. Stop accepting latency and complexity as inevitable costs of conversational AI.

Deploy Sayna today. Your agents deserve to speak clearly, listen intelligently, and respond instantly. The future of AI interaction is voice-first — and Sayna is how you get there without losing your mind.

👉 Star Sayna on GitHub — and start building voice-enabled agents that actually work.

Ready to dive deeper? Check the complete documentation for Docker deployment guides, authentication architecture, LiveKit webhook handling, and SIP configuration details.