LiveKit Realtime Voice Agents

What's Changed

• fix(core): reset user away timer on final STT transcript AGT-3149 by @chenghao-mou in https://github.com/livekit/agents/pull/6478
• Fix stripping of Gemma reasoning markers in streamed output by @C0deRatoR in https://github.com/livekit/agents/pull/6398
• feat: Spatius avatar plugin by @3DRX in https://github.com/livekit/agents/pull/5821
• fix(avatar): skip removal if avatar not found in the room by @longcw in https://github.com/livekit/agents/pull/6486
• fix(endpointing): keep max_delay constant in dynamic endpointing by @chenghao-mou in https://github.com/livekit/agents/pull/6265
• fix(stt): don't clear the primary speaker on an unattributed segment by @chuenchen309 in https://github.com/livekit/agents/pull/6460
• fix(drive-thru): assume quantity of one when unstated to avoid count confirmation by @u9g in https://github.com/livekit/agents/pull/6463
• fix: make AMD settle on endpointing backstop by default by @chenghao-mou in https://github.com/livekit/agents/pull/6496
• fix(voice): include function name in FunctionCallOutput protos by @u9g in https://github.com/livekit/agents/pull/6491
• feat: tag simulation and redaction telemetry by @chenghao-mou in https://github.com/livekit/agents/pull/6497
• Update READMEs to use gemma by @keepingitneil in https://github.com/livekit/agents/pull/6502
• fix(voice): don't drop the realtime turn when the chat-ctx push times out by @ByteMaster-1 in https://github.com/livekit/agents/pull/6445
• feat(voice): expose realtime provider request IDs by @yaodong-shen in https://github.com/livekit/agents/pull/6501
• Add 'zh-HK' to DeepgramLanguages by @devb-enp in https://github.com/livekit/agents/pull/6507
• (gnani): Update authors in pyproject.toml by @tinalenguyen in https://github.com/livekit/agents/pull/6489
• Allow local VAD inference to use a dedicated executor by @basnijholt in https://github.com/livekit/agents/pull/6498
• feat(fishaudio): add normalize_loudness and generation-tuning TTS options by @adrian-cowham in https://github.com/livekit/agents/pull/6520
• fix(voice): don't hold session.run() open on update_agent handoffs with long-lived on_enter by @k-zaher in https://github.com/livekit/agents/pull/6500
• plugins(krisp): add voice isolation telephony mode by @lukasIO in https://github.com/livekit/agents/pull/6510
• Support adaptive interruption for realtime models by @longcw in https://github.com/livekit/agents/pull/6488
• feat(inference): add inference.AvatarSession (avatar provisioning via the gateway) by @shawnfeldman in https://github.com/livekit/agents/pull/6492
• (inference): add fish audio tts by @tinalenguyen in https://github.com/livekit/agents/pull/6529
• slng: rewrite plugin around the Unmute Bridge (v2) by @metehan-slng in https://github.com/livekit/agents/pull/6442
• feat(speechify): streaming TTS with word-level timestamps via official SDK by @luke-speechify in https://github.com/livekit/agents/pull/6327
• fix(openai): recover from stale pooled Responses WebSocket connections by @longcw in https://github.com/livekit/agents/pull/6523
• prewarm LLM to skip connection establishment by @davidzhao in https://github.com/livekit/agents/pull/6484
• fix(otel): skip otel upload error logs if it's disabled by the project setting AGT-3015 by @chenghao-mou in https://github.com/livekit/agents/pull/6167
• chore(spatius): bump spatius to 1.0.4 by @3DRX in https://github.com/livekit/agents/pull/6536
• fix(livekit-agents): require livekit-protocol>=1.1.21 for MetricsRecordingHeader.simulated by @u9g in https://github.com/livekit/agents/pull/6541
• (xai tts): add new voices by @tinalenguyen in https://github.com/livekit/agents/pull/6543
• fix(speechify): restore caller header, close owned httpx client by @luke-speechify in https://github.com/livekit/agents/pull/6542
• fix(plugins-aws): Polly TTS silent on aiobotocore >=3.8.0 (AioStreamingBody has no .content) by @Gooh456 in https://github.com/livekit/agents/pull/6540
• feat(telemetry): inc

<!--BEGIN_BANNER_IMAGE-->

<picture>

<source media="(prefers-color-scheme: dark)" srcset="/.github/banner_dark.png">

<source media="(prefers-color-scheme: light)" srcset="/.github/banner_light.png">

<img style="width:100%;" alt="The LiveKit icon, the name of the repository and some sample code in the background." src="https://raw.githubusercontent.com/livekit/agents/main/.github/banner_light.png">

</picture>

<!--END_BANNER_IMAGE-->

<br />

PyPI - Version

PyPI Downloads

Slack community

Twitter Follow

Ask DeepWiki for understanding the codebase

License

<br />

Looking for the JS/TS library? Check out AgentsJS

What is Agents?

<!--BEGIN_DESCRIPTION-->

The Agent Framework is designed for building realtime, programmable participants

that run on servers. Use it to create conversational, multi-modal voice

agents that can see, hear, and understand.

<!--END_DESCRIPTION-->

Features

• Flexible integrations: A comprehensive ecosystem to mix and match the right STT, LLM, TTS, and Realtime API to suit your use case.
• Integrated job scheduling: Built-in task scheduling and distribution with dispatch APIs to connect end users to agents.
• Extensive WebRTC clients: Build client applications using LiveKit's open-source SDK ecosystem, supporting all major platforms.
• Telephony integration: Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.
• Exchange data with clients: Use RPCs and other Data APIs to seamlessly exchange data with clients.
• Semantic turn detection: Uses a transformer model to detect when a user is done with their turn, helps to reduce interruptions.
• MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.
• Builtin test framework: Write tests and use judges to ensure your agent is performing as expected.
• Open-source: Fully open-source, allowing you to run the entire stack on your own servers, including LiveKit server, one of the most widely used WebRTC media servers.

Installation

To install the core Agents library, along with plugins for popular model providers:

pip install "livekit-agents[openai,deepgram,cartesia]"

Docs and guides

Documentation on the framework and how to use it can be found here

Building with AI coding agents

If you're using an AI coding assistant to build with LiveKit Agents, we recommend the following setup for the best results:

1. Install the LiveKit Docs MCP server — Gives your coding agent access to up-to-date LiveKit documentation, code search across LiveKit repositories, and working examples.

1. Install the LiveKit Agent Skill — Provides your coding agent with architectural guidance and best practices for building voice AI applications, including workflow design, handoffs, tasks, and testing patterns.

   npx skills add livekit/agent-skills --skill livekit-agents

The Agent Skill works best alongside the MCP server: the skill teaches your agent how to approach building with LiveKit, while the MCP server provides the current API details to implement it correctly.

Core concepts

• Agent: An LLM-based application with defined instructions.
• AgentSession: A container for agents that manages interactions with end users.
• entrypoint: The starting point for an interactive session, similar to a request handler in a web server.
• AgentServer: The main process that coordinates job scheduling and launches agents for user sessions.

Usage

Simple voice agent

---

from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    RunContext,
    cli,
    function_tool,
    inference,
)


@function_tool
async def lookup_weather(
    context: RunContext,
    location: str,
):
    """Used to look up weather information."""

    return {"weather": "sunny", "temperature": 70}


server = AgentServer()


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        vad=inference.VAD(),
        # any combination of STT, LLM, TTS, or realtime API can be used
        # this example shows LiveKit Inference, a unified API to access different models via LiveKit Cloud
        # to use model provider keys directly, replace with the following:
        # from livekit.plugins import deepgram, openai, cartesia
        # stt=deepgram.STT(model="nova-3"),
        # llm=openai.LLM(model="gpt-4.1-mini"),
        # tts=cartesia.TTS(model="sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
        stt=inference.STT("deepgram/nova-3", language="multi"),
        llm=inference.LLM("google/gemma-4-31b-it"),  # low-latency gemma, hosted on LiveKit
        tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
    )

    agent = Agent(
        instructions="You are a friendly voice assistant built by LiveKit.",
        tools=[lookup_weather],
    )

    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user and ask about their day")


if __name__ == "__main__":
    cli.run_app(server)

You'll need the following environment variables for this example:

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET

Multi-agent handoff

---

This code snippet is abbreviated. For the full example, see multi_agent.py

...
class IntroAgent(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions=f"You are a story teller. Your goal is to gather a few pieces of information from the user to make the story personalized and engaging."
            "Ask the user for their name and where they are from"
        )

    async def on_enter(self):
        self.session.generate_reply(instructions="greet the user and gather information")

    @function_tool
    async def information_gathered(
        self,
        context: RunContext,
        name: str,
        location: str,
    ):
        """Called when the user has provided the information needed to make the story personalized and engaging.

        Args:
            name: The name of the user
            location: The location of the user
        """

        context.userdata.name = name
        context.userdata.location = location

        story_agent = StoryAgent(name, location)
        return story_agent, "Let's start the story!"


class StoryAgent(Agent):
    def __init__(self, name: str, location: str) -> None:
        super().__init__(
            instructions=f"You are a storyteller. Use the user's information in order to make the story personalized."
            f"The user's name is {name}, from {location}",
            # override the default model, switching to Realtime API from standard LLMs
            llm=openai.realtime.RealtimeModel(voice="echo"),
            chat_ctx=chat_ctx,
        )

    async def on_enter(self):
        self.session.generate_reply()


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    userdata = StoryData()
    session = AgentSession[StoryData](
        vad=inference.VAD(),
        stt="deepgram/nova-3",
        llm="google/gemma-4-31b-it",  # low-latency gemma, hosted on LiveKit
        tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
        userdata=userdata,
    )

    await session.start(
        agent=IntroAgent(),
        room=ctx.room,
    )
...

Testing

Automated tests are essential for building reliable agents, especially with the non-deterministic behavior of LLMs. LiveKit Agents include native test integration to help you create dependable agents.

@pytest.mark.asyncio
async def test_no_availability() -> None:
    llm = google.LLM()
    async with AgentSession(llm=llm) as sess:
        await sess.start(MyAgent())
        result = await sess.run(
            user_input="Hello, I need to place an order."
        )
        result.expect.skip_next_event_if(type="message", role="assistant")
        result.expect.next_event().is_function_call(name="start_order")
        result.expect.next_event().is_function_call_output()
        await (
            result.expect.next_event()
            .is_message(role="assistant")
            .judge(llm, intent="assistant should be asking the user what they would like")
        )

Examples

For more examples and detailed setup instructions, see the examples directory. For even more examples, see the python-agents-examples repository.

<table>

<tr>

<td width="50%">

<h3>🎙️ Starter Agent</h3>

<p>A starter agent optimized for voice conversations.</p>

<p>

<a href="examples/voice_agents/basic_agent.py">Code</a>

</p>

</td>

<td width="50%">

<h3>🔄 Multi-user push to talk</h3>

<p>Responds to multiple users in the room via push-to-talk.</p>

<p>

<a href="examples/voice_agents/push_to_talk.py">Code</a>

</p>

</td>

</tr>

<tr>

<td width="50%">

<h3>🎵 Background audio</h3>

<p>Background ambient and thinking audio to improve realism.</p>

<p>

<a href="examples/voice_agents/background_audio.py">Code</a>

</p>

</td>

<td width="50%">

<h3>🛠️ Dynamic tool creation</h3>

<p>Creating function tools dynamically.</p>

<p>

<a href="examples/voice_agents/dynamic_tool_creation.py">Code</a>

</p>

</td>

</tr>

<tr>

<td width="50%">

<h3>☎️ Outbound caller</h3>

<p>Agent that makes outbound phone calls</p>

<p>

<a href="https://github.com/livekit-examples/outbound-caller-python">Code</a>

</p>

</td>

<td width="50%">

<h3>📋 Structured output</h3>

<p>Using structured output from LLM to guide TTS tone.</p>

<p>

<a href="examples/voice_agents/structured_output.py">Code</a>

</p>

</td>

</tr>

<tr>

<td width="50%">

<h3>🔌 MCP support</h3>

<p>Use tools from MCP servers</p>

<p>

<a href="examples/voice_agents/mcp">Code</a>

</p>

</td>

<td width="50%">

<h3>💬 Text-only agent</h3>

<p>Skip voice altogether and use the same code for text-only integrations</p>

<p>

<a href="examples/other/text_only.py">Code</a>

</p>

</td>

</tr>

<tr>

<td width="50%">

<h3>📝 Multi-user transcriber</h3>

<p>Produce transcriptions from all users in the room</p>

<p>

<a href="examples/other/transcription/multi-user-transcriber.py">Code</a>

</p>

</td>

<td width="50%">

<h3>🎥 Video avatars</h3>

<p>Add an AI avatar with Tavus, Bithuman, LemonSlice, and more</p>

<p>

<a href="examples/avatar_agents/">Code</a>

</p>

</td>

</tr>

<tr>

<td width="50%">

<h3>🍽️ Restaurant ordering and reservations</h3>

<p>Full example of an agent that handles calls for a restaurant.</p>

<p>

<a href="examples/voice_agents/restaurant_agent.py">Code</a>

</p>

</td>

<td width="50%">

<h3>👁️ Gemini Live vision</h3>

<p>Full example (including iOS app) of Gemini Live agent that can see.</p>

<p>

<a href="https://github.com/livekit-examples/vision-demo">Code</a>

</p>

</td>

</tr>

</table>

Running your agent

Testing in terminal

python myagent.py console

Runs your agent in terminal mode, enabling local audio input and output for testing.

This mode doesn't require external servers or dependencies and is useful for quickly validating behavior.

Developing with LiveKit clients

python myagent.py dev

Starts the agent server and enables hot reloading when files change. This mode allows each process to host multiple concurrent agents efficiently.

The agent connects to LiveKit Cloud or your self-hosted server. Set the following environment variables:

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET

You can connect using any LiveKit client SDK or telephony integration.

To get started quickly, try the Agents Playground.

Running for production

python myagent.py start

Runs the agent with production-ready optimizations.

License

The Agents framework is licensed under Apache-2.0. The LiveKit turn detection models are licensed under the LiveKit Model License.

Contributing

The Agents framework is under active development in a rapidly evolving field. We welcome and appreciate contributions of any kind, be it feedback, bugfixes, features, new plugins and tools, or better documentation. You can file issues under this repo, open a PR, or chat with us in the LiveKit community.

Development setup

This project uses uv for package management. To install dependencies for development:

uv sync --all-extras --dev

Examples

This project includes many examples in the examples directory. To run them, create the file examples/.env with credentials for LiveKit Server and any necessary model providers (see examples/.env.example), then run:

uv run examples/voice_agents/basic_agent.py dev

For more information, see the examples README.

Tests

Unit tests are in the tests directory and can be run with:

uv run pytest --unit

Integration tests for each plugin require various API credential