LiteLLM

Verify Docker Image Signature

All LiteLLM Docker images are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.

Verify using the pinned commit hash (recommended):

A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:v1.94.0

Verify using the release tag (convenience):

Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/v1.94.0/cosign.pub \
  ghcr.io/berriai/litellm:v1.94.0

Expected output:

The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

---

What's Changed

• feat(ui): working Test Connection for the complexity auto router by @akapur99 in https://github.com/BerriAI/litellm/pull/32950
• fix(xecguard): use StandardLoggingGuardrailInformation in logging hook by @yucheng-berri in https://github.com/BerriAI/litellm/pull/32911
• feat(ui): adopt openapi-react-query ($api) and convert useCustomers by @ryan-crabbe-berri in https://github.com/BerriAI/litellm/pull/32949
• refactor(ui): colocate the mcp-servers view, keeping the shared mcp_tools surface by @ryan-crabbe-berri in https://github.com/BerriAI/litellm/pull/32968
• refactor(ui): convert endpoint usage charts to shadcn/recharts by @ryan-crabbe-berri in https://github.com/BerriAI/litellm/pull/32723
• fix(proxy-auth): stop unrecognized model namespaces slipping through provider wildcard keys by @mateo-berri in https://github.com/BerriAI/litellm/pull/32979
• feat(router): random-pick multi-model complexity tiers by @krrish-berri-2 in https://github.com/BerriAI/litellm/pull/32967
• fix(xecguard): sanitize scan result before recording it for logging by @yucheng-berri in https://github.com/BerriAI/litellm/pull/32935
• fix(auto_router): filter embedding models in complexity tab dropdowns, require all tiers, inline validation by @akapur99 in https://github.com/BerriAI/litellm/pull/32978
• fix(anthropic): translate raw adaptive thinking for pre-4.6 models on chat completions and Bedrock Converse by @akapur99 in https://github.com/BerriAI/litellm/pull/32944
• feat(router): add Router(plugins=[...]) routing-plugin pipeline by @krrish-berri-2 in https://github.com/BerriAI/litellm/pull/32972
• feat(router): soft-floor adaptive mode for complexity router by @krrish-berri-2 in https://github.com/BerriAI/litellm/pull/32947
• docs(github): add QA runbook section to the PR template by @mateo-berri in https://github.com/BerriAI/litellm/pull/32965
• fix(model_cost): add supports_reasoning: false to Gemini image generation models by @mateo-berri in https://github.com/BerriAI/litellm/pull/32836
• build(dev-env): add make bootstrap and unprovisioned-checkout preflight to pre-commit by @mateo-berri in https://github.com/BerriAI/litellm/pull/32981
• ci(ui): report only error-level knip findings in CI by @ryan-crabbe-berri in https://github.com/BerriAI/litellm/pull/32971
• feat(batches): track cost for unmanaged Bedrock batches, generalize the flag by @Sameerlite in https://github.com/BerriAI/litellm/pull/32315
• fix(guardrails): walk custom_tool_call_output items in _content_utils by @yucheng-berri in https://github.com/BerriAI/litellm/pull/32969
• fix: show and allow editing team model aliases after team creation by @ryan-crabbe-berri in https://github.com/BerriAI/litellm/pull/33047
• chore(deps): bump pillow to 12.3.0 to resolve osv-scan CVEs by @yucheng-berri in https://github.com/BerriA

<h1 align="center">

🚅 LiteLLM

</h1>

<p align="center">

<p align="center">LiteLLM AI Gateway

</p>

<p align="center">Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.</p>

<p align="center">

<a href="https://render.com/deploy?repo=https://github.com/BerriAI/litellm" target="_blank" rel="nofollow"><img src="https://render.com/images/deploy-to-render-button.svg" alt="Deploy to Render" height="40"></a>

<a href="https://railway.com/deploy/RhvhdC?referralCode=7mRv9K&utm_medium=integration&utm_source=template&utm_campaign=generic"><img src="https://railway.com/button.svg" alt="Deploy on Railway" height="40"></a>

<a href="https://console.aws.amazon.com/cloudshell/home" target="_blank" rel="nofollow"><img src="./.github/deploy-on-aws.png" alt="Deploy on AWS" height="40"></a>

<a href="https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FBerriAI%2Flitellm&cloudshell_workspace=terraform%2Flitellm%2Fgcp%2Fexamples%2Fdefault&cloudshell_tutorial=TUTORIAL.md&cloudshell_image=gcr.io/ds-artifacts-cloudshell/deploystack_custom_image&shellonly=true" target="_blank" rel="nofollow"><img src="./.github/deploy-on-gcp.png" alt="Deploy on GCP" height="40"></a>

</p>

</p>

<h4 align="center"><a href="https://docs.litellm.ai/docs/simple_proxy" target="_blank">LiteLLM Proxy Server (AI Gateway)</a> | <a href="https://docs.litellm.ai/docs/enterprise#hosted-litellm-proxy" target="_blank"> Hosted Proxy</a> | <a href="https://litellm.ai/enterprise"target="_blank">Enterprise Tier</a> | <a href="https://www.litellm.ai/ai-gateway" target="_blank">Website</a></h4>

<h4 align="center">

<a href="https://pypi.org/project/litellm/" target="_blank">

<img src="https://img.shields.io/pypi/v/litellm.svg" alt="PyPI Version">

</a>

<a href="https://github.com/BerriAI/litellm" target="_blank">

<img src="https://img.shields.io/github/stars/BerriAI/litellm.svg?style=social" alt="GitHub Stars">

</a>

<a href="https://www.ycombinator.com/companies/berriai">

<img src="https://img.shields.io/badge/Y%20Combinator-W23-orange?style=flat-square" alt="Y Combinator W23">

</a>

<a href="https://wa.link/huol9n">

<img src="https://img.shields.io/static/v1?label=Chat%20on&message=WhatsApp&color=success&logo=WhatsApp&style=flat-square" alt="Whatsapp">

</a>

<a href="https://discord.gg/wuPM9dRgDw">

<img src="https://img.shields.io/static/v1?label=Chat%20on&message=Discord&color=blue&logo=Discord&style=flat-square" alt="Discord">

</a>

<a href="https://www.litellm.ai/support">

<img src="https://img.shields.io/static/v1?label=Chat%20on&message=Slack&color=black&logo=Slack&style=flat-square" alt="Slack">

</a>

<a href="https://codspeed.io/BerriAI/litellm?utm_source=badge">

<img src="https://img.shields.io/endpoint?url=https://codspeed.io/badge.json" alt="CodSpeed"/>

</a>

</h4>

<img alt="LiteLLM AI Gateway" src="https://github.com/user-attachments/assets/c5ee0412-6fb5-4fb6-ab5b-bafae4209ca6" />

---

What is LiteLLM

LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.

Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.

Jump to LiteLLM Proxy (LLM Gateway) Docs <br>

Jump to Supported LLM Providers

---

Why LiteLLM

Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:

• Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
• Drop-in OpenAI compatibility — swap providers without rewriting your code
• Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
• 8ms P95 latency at 1k RPS (benchmarks)

OSS Adopters

<table>

<tr>

<td><img height="60" alt="Stripe" src="https://github.com/user-attachments/assets/f7296d4f-9fbd-460d-9d05-e4df31697c4b" /></td>

<td><img height="60" alt="image" src="https://github.com/user-attachments/assets/436fca71-988b-40bb-b5fe-8450c80fdbd0" /></td>

<td><img height="60" alt="Google ADK" src="https://github.com/user-attachments/assets/caf270a2-5aee-45c4-8222-41a2070c4f19" /></td>

<td><img height="60" alt="Greptile" src="https://github.com/user-attachments/assets/3db0ae72-0843-4005-a56d-bba1dde2193d" /></td>

<td><img height="60" alt="OpenHands" src="https://github.com/user-attachments/assets/a6150c4c-149e-4cae-888b-8b92be6e003f" /></td>

<td><h2>Netflix</h2></td>

<td><img height="60" alt="OpenAI Agents SDK" src="https://github.com/user-attachments/assets/c02f7be0-8c2e-4d27-aea7-7c024bfaebc0" /></td>

</tr>

</table>

---

Features

<details open>

<summary><b>LLMs</b> - Call 100+ LLMs (Python SDK + AI Gateway)</summary>

All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.

Python SDK

uv add litellm

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

# Anthropic
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

AI Gateway (Proxy Server)

Getting Started - E2E Tutorial - Setup virtual keys, make your first request

uv tool install 'litellm[proxy]'
litellm --model gpt-4o

import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

Docs: LLM Providers

</details>

<details>

<summary><b>Agents</b> - Invoke A2A Agents (Python SDK + AI Gateway)</summary>

Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI

Python SDK - A2A Protocol

from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4

client = A2AClient(base_url="http://localhost:10001")

request = SendMessageRequest(
    id=str(uuid4()),
    params=MessageSendParams(
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": "Hello!"}],
            "messageId": uuid4().hex,
        }
    )
)
response = await client.send_message(request)

AI Gateway (Proxy Server)

Step 1. Add your Agent to the AI Gateway — set protocolVersion to 1.0 or 0.3 per agent

Step 2. Call Agent via A2A SDK (requires a2a-sdk>=1.1.0)

import httpx
from a2a.client import A2ACardResolver, ClientConfig, ClientFactory
from a2a.types import Message, Part, Role, SendMessageRequest
from a2a.utils.constants import TransportProtocol
from uuid import uuid4

base_url = "http://localhost:4000/a2a/my-agent"  # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"}    # LiteLLM Virtual Key

async with httpx.AsyncClient(headers=headers, timeout=60.0) as http_client:
    resolver = A2ACardResolver(httpx_client=http_client, base_url=base_url)
    agent_card = await resolver.get_agent_card()
    config = ClientConfig(
        httpx_client=http_client,
        streaming=False,
        supported_protocol_bindings=[TransportProtocol.JSONRPC, TransportProtocol.HTTP_JSON],
    )
    client = ClientFactory(config).create(agent_card)

    request = SendMessageRequest(
        message=Message(
            message_id=uuid4().hex,
            role=Role.ROLE_USER,
            parts=[Part(text="Hello!")],
        )
    )
    async for event in client.send_message(request):
        populated = event.ListFields()
        if populated and populated[0][0].name in ("message", "msg"):
            print("".join(getattr(p, "text", "") or "" for p in populated[0][1].parts))

Docs: A2A Agent Gateway

</details>

<details>

<summary><b>MCP Tools</b> - Connect MCP servers to any LLM (Python SDK + AI Gateway)</summary>

Python SDK - MCP Bridge

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm

server_params = StdioServerParameters(command="python", args=["mcp_server.py"])

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # Load MCP tools in OpenAI format
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")

        # Use with any LiteLLM model
        response = await litellm.acompletion(
            model="gpt-4o",
            messages=[{"role": "user", "content": "What's 3 + 5?"}],
            tools=tools
        )

AI Gateway - MCP Gateway

Step 1. Add your MCP Server to the AI Gateway

Step 2. Call MCP tools via /chat/completions

curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'

Use with Cursor IDE

{
  "mcpServers": {
    "LiteLLM": {
      "url": "http://localhost:4000/mcp/",
      "headers": {
        "x-litellm-api-key": "Bearer sk-1234"
      }
    }
  }
}

Docs: MCP Gateway

</details>

Supported Providers (Website Supported Models | Docs)

| Provider | /chat/completions | /messages | /responses | /embeddings | /image/generations | /audio/transcriptions | /audio/speech | /moderations | /batches | /rerank |

|-------------------------------------------------------------------------------------|---------------------|-------------|--------------|---------------|----------------------|-------------------------|-----------------|----------------|-----------|-----------|

| Abliteration (abliteration) | ✅ | | | | | | | | | |

| AI/ML API (aiml) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |

| AI21 (ai21) | ✅ | ✅ | ✅ | | | | | | | |

| AI21 Chat (ai21_chat) | ✅ | ✅ | ✅ | | | | | | | |

| Aleph Alpha | ✅ | ✅ | ✅ | | | | | | | |

| Amazon Nova | ✅ | ✅ | ✅ | | | | | | | |

| Anthropic (anthropic) | ✅ | ✅ | ✅ | | | | | | ✅ | |

| Anthropic Text (anthropic_text) | ✅ | ✅ | ✅ | | | | | | ✅ | |

| Anyscale | ✅ | ✅ | ✅ | | | | | | | |

| AssemblyAI (assemblyai) | ✅ | ✅ | ✅ | | | ✅ | | | | |

| Auto Router (auto_router) | ✅ | ✅ | ✅ | | | | | | | |

| AWS - Bedrock (bedrock) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ |

| AWS - Sagemaker (sagemaker) | ✅ | ✅ | ✅ | ✅ | | | | | | |

| Azure (azure) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |

| Azure AI (azure_ai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |

| Azure Text (azure_text) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | |

| Baseten (baseten) | ✅ | ✅ | ✅ | | | | | | | |

| Bytez (bytez) | ✅ | ✅ | ✅ | | | | | | | |

| Cerebras (cerebras) | ✅ | ✅ | ✅ | | | | | | | |

| Clarifai (clarifai) | ✅ | ✅ | ✅ | | | | | | | |

| Cloudflare AI Workers (cloudflare) | ✅ | ✅ | ✅ | | | | | | | |

| Codestral (codestral) | ✅ | ✅ | ✅ | | | | | | | |

| Cohere (cohere) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ |

| Cohere Chat (cohere_chat) | ✅ | ✅ | ✅ | | | | | | | |

| [CometAPI (cometapi)](https://docs.litellm.ai/docs/provide