Voice sessions

Agentium offers three voice paths. Choose according to who should own the conversation and tool loop.

API	Conversation and execution	Use when
`VoiceAgent`	Native OpenAI Realtime or Gemini Live session	You want the provider’s realtime speech model
`StreamingVoicePipeline` with `AgentVoiceBrain`	Coordinator-owned history and an ordinary Agent tool loop	You want independent STT, model, TTS, and transport choices
`VoicePipeline`	Buffered transcription → one model call → buffered speech	You process completed audio files and do not need Agent tools

Import voice adapters from @agentium/core/voice. Upgrading from 3.x? Review the v4 voice migration.

Open a native session

Install ws and the optional SDK required by your provider. Set OPENAI_API_KEY for this example. Your application supplies microphone capture and playback.

import { OpenAIRealtimeProvider, VoiceAgent } from "@agentium/core/voice";

const voice = new VoiceAgent({
  name: "support",
  provider: new OpenAIRealtimeProvider(),
  instructions: "Help the caller with concise spoken answers.",
  turnDetection: { type: "semantic_vad", eagerness: "low" },
});
const session = await voice.connect({
  tenantId: "local-demo",
  userId: "developer",
  sessionId: "call-1",
});

// Connect your media callbacks before sending input.
// session.sendAudio(pcmBuffer);
// session.commitAudio(); // For manually committed input.
// Keep the session open for the call, then:
await session.close();

The connection example does not capture a microphone. Send audio in the negotiated codec/rate and route audio events to your playback device. Use Socket.IO voice gateway for a browser-facing transport or implement a media adapter in your host.

Provider contracts

Provider	Implementation default	Media contract
`OpenAIRealtimeProvider`	`gpt-realtime-2.1`	PCM at 24 kHz or supported G.711 formats; nested GA audio configuration
`GoogleLiveProvider`	`gemini-3.8-live`	PCM at 16 kHz input and 24 kHz output

These are defaults in the checked-out adapter, not guarantees of model access in every account. Pin a model when your deployment requires it. Unsupported provider-specific options reject rather than silently changing semantics. OpenAI temperature and remote reusable prompt objects are rejected. Input transcription defaults to gpt-transcribe for native OpenAI WebSocket sessions. Supply application-owned transcriptionContext for language/keyword hints. See streaming voice for standalone recognizers and synthesizers.

Tools and ownership

Local tools use the normal validation, approval, execution policy, and cancellation path. Provider-executed remote MCP tools cannot be intercepted locally; native remote MCP cannot be combined with local policy/approval configuration. Register MCP functions as ordinary local tools when you need shared enforcement. A memory-enabled VoiceAgent binds to the first connection’s tenant and user. Later owner changes fail, including anonymous-to-authenticated changes. Use separate instances and scoped backing storage per owner. This guard does not partition a shared storage client automatically.

Playback-confirmed history

Generated text is not proof that a user heard it. A session keeps generated speech and playback-confirmed text separate. Route trusted playback offsets through acknowledgePlayback({ generationId, playedCharacters }); confirm completion only after the final transcript has actually played. Interrupted or unacknowledged text remains delivery-unknown in getTranscript(). Native provider-side conversation state is still provider-owned; local playback acknowledgements do not rewrite it. Use AgentVoiceBrain when your application needs coordinator-owned, playback-confirmed history.

Recovery

Recovery is opt-in through VoiceAgentConfig.recovery. Configure total attempts, delays, connection/elapsed deadlines, and fallback: "stop" or an explicit "fresh" fallback.

Gemini: session resumption requires a safe idle checkpoint. Input, generation, tool dispatch, or uncertain effects invalidate it.
OpenAI: fresh recovery creates a new configured session and loses provider conversation history. It does not replay audio or tools.
Either: uncertain effects can prevent recovery. The host must reconcile them before restarting.

Listen for recovery states recovering, recovered, and failed. Pause capture while recovering and request new input after reconnection. See the gateway client contract.

WebRTC and outbound calls

createRealtimeClientSecret and createRealtimeCall are exported from @agentium/core. The latter accepts WebRTC sdp and returns { id, sdp, raw }; the returned SDP is the answer. It rejects sipUri because it does not place outbound SIP calls. Authenticate and authorize your application endpoint before minting a client secret. Outbound carrier call control is a separate telephony API. It does not itself transport audio.

Validation limits

Local contract tests cover tool concurrency, interruption, transcript reconciliation, buffer limits, and cleanup. They do not establish speech accuracy, live carrier interoperability, or latency for your deployment. Measure actual client playback when reporting first-audible latency.

Build

Agent applications

Harness runtime

Teams and workflows

Data and capabilities

Interfaces and jobs

Voice and telephony

Verify and manage

Open a native session

Provider contracts

Tools and ownership

Playback-confirmed history

Recovery

WebRTC and outbound calls

Validation limits

​Open a native session

​Provider contracts

​Tools and ownership

​Playback-confirmed history

​Recovery

​WebRTC and outbound calls

​Validation limits

Open a native session

Provider contracts

Tools and ownership

Playback-confirmed history

Recovery

WebRTC and outbound calls

Validation limits