Mistral.rs

What's Changed

• Fix CUDA version extraction in install.sh by @zaqxsw-dev in https://github.com/EricLBuehler/mistral.rs/pull/2287
• feat(metal): improve runtime and aot compilation system by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2288
• feat(release): split cuda workers for docker release by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2289
• feat(install): cleanup install system for updating by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2290
• feat(distributed): more sophisticated TP fallback and resolution system by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2291
• chore(mistralrs-core): cap tojson indent and remove chat template panic paths by @eharris128 in https://github.com/EricLBuehler/mistral.rs/pull/2286
• feat(quant): improve uqff resolution per model ids by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2293
• Fix agentic loop not starting for registered tool callbacks by @aeiklorvy in https://github.com/EricLBuehler/mistral.rs/pull/2279
• feat(models): Support LFM 2.5 (dense + MoE) and LFM 2.5 VL! by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2295
• feat(cuda): preload cuda ptx modules by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2296
• refactor(core): move code exec shims from lib.rs to proper location by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2297
• docs(models): improve supported models readability by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2298
• feat(multi-model): support planning for loading multi-model by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2299
• feat(cuda): update cutlass git revision by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2300
• feat(cpu): aarch64 cpu repacking kernels and performance improvements by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2304
• feat(cpu): optimize decode and prefill performance on x86 and aarch64 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2311
• fix(tune): detect CUDA backend on Windows by @matt-minev in https://github.com/EricLBuehler/mistral.rs/pull/2312
• fix(server): keep Responses output item IDs stable by @matt-minev in https://github.com/EricLBuehler/mistral.rs/pull/2309

New Contributors

• @zaqxsw-dev made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/2287
• @eharris128 made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/2286
• @aeiklorvy made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/2279
• @matt-minev made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/2312

Full Changelog: https://github.com/EricLBuehler/mistral.rs/compare/v0.8.23...v0.9.0

<a name="top"></a>

<!--

<h1 align="center">

mistral.rs

</h1>

-->

<div align="center">

<picture>

<source media="(prefers-color-scheme: dark)" srcset="res/banner-dark.gif">

<source media="(prefers-color-scheme: light)" srcset="res/banner-light.gif">

<img src="res/banner-dark.png" alt="mistral.rs - Fast, flexible LLM inference." width="100%" style="max-width: 800px;">

</picture>

</div>

<p align="center">

| <a href="https://ericlbuehler.github.io/mistral.rs/"><b>Documentation</b></a> | <a href="https://ericlbuehler.github.io/mistral.rs/quickstart/"><b>Quickstart</b></a> | <a href="https://ericlbuehler.github.io/mistral.rs/reference/supported-models/"><b>Supported models</b></a> | <a href="https://crates.io/crates/mistralrs"><b>Rust SDK</b></a> | <a href="https://ericlbuehler.github.io/mistral.rs/guides/python/getting-started/"><b>Python SDK</b></a> | <a href="https://discord.gg/SZrecqK8qw"><b>Discord</b></a> |

</p>

<p align="center">

<a href="https://github.com/EricLBuehler/mistral.rs/stargazers">

<img src="https://img.shields.io/github/stars/EricLBuehler/mistral.rs?style=social&label=Star" alt="GitHub stars">

</a>

</p>

Latest

• OpenAI-compatible Skills: upload /v1/skills bundles and reference them from Responses requests for reusable procedures, helper scripts, and local data. Guide
• OpenAI-compatible file inputs: upload /v1/files, attach Responses input_file or Chat file parts, and mount request files into shell/code sessions. Guide
• DiffusionGemma: block-diffusion text generation. Fully integrated: paged attention, prefix caching, ISQ, multimodal, and tool calling. Guide
• Anthropic Messages API: mistralrs serve now exposes Anthropic-compatible /v1/messages and /v1/messages/count_tokens endpoints alongside the OpenAI-compatible /v1 API. Guide
• v0.8.2 CUDA performance: paged-attention and MoE optimizations deliver strong results on GB10, B200, and H100 SXM. Benchmarks
• Agentic runtime: web search, local Python code execution, shell execution, OpenAI-compatible Skills, session management, and custom tool hooks. Guide
• Gemma 4: full multimodal: text, image, video, and audio input. Supported models | Video setup

Benchmarks

<details>

<summary><b>v0.8.2 CUDA benchmarks</b></summary>

Mean tokens per second across prompt lengths and decode depths from 128 to 16384 tokens. Decode uses 256 generated tokens. See the full v0.8.2 report for commands, model revisions, host metadata, and appendix tables.

Q8 prefill TPS: mistral.rs UQFF q8 vs llama.cpp GGUF Q8_0

| Model | Hardware | mistral.rs | llama.cpp |

|---|---|---:|---:|

| Gemma 4 E4B | GB10 | 7395.7 | 3973.7 |

| Gemma 4 E4B | B200 | 27705.6 | 11992.4 |

| Gemma 4 E4B | H100 SXM | 26220.6 | 11702.1 |

| Gemma 4 26B-A4B | GB10 | 2947.0 | 2178.5 |

| Gemma 4 26B-A4B | B200 | 12725.3 | 8503.4 |

| Gemma 4 26B-A4B | H100 SXM | 12362.3 | 8055.1 |

Q8 decode TPS: mistral.rs UQFF q8 vs llama.cpp GGUF Q8_0

| Model | Hardware | mistral.rs | llama.cpp |

|---|---|---:|---:|

| Gemma 4 E4B | GB10 | 44.1 | 40.5 |

| Gemma 4 E4B | B200 | 241.4 | 194.4 |

| Gemma 4 E4B | H100 SXM | 223.1 | 183.0 |

| Gemma 4 26B-A4B | GB10 | 46.8 | 46.4 |

| Gemma 4 26B-A4B | B200 | 210.9 | 192.2 |

| Gemma 4 26B-A4B | H100 SXM | 199.8 | 183.9 |

BF16 prefill TPS: mistral.rs BF16 vs vLLM BF16

| Model | Hardware | mistral.rs | vLLM |

|---|---|---:|---:|

| Gemma 4 E4B | GB10 | 5838.9 | 5812.9 |

| Gemma 4 E4B | B200 | 43547.8 | 39431.2 |

| Gemma 4 E4B | H100 SXM | 35852.2 | 39293.7 |

| Gemma 4 26B-A4B | GB10 | 592.2 | 3878.6 |

| Gemma 4 26B-A4B | B200 | 3467.3 | 28532.8 |

| Gemma 4 26B-A4B | H100 SXM | 2766.0 | 26295.9 |

BF16 decode TPS: mistral.rs BF16 vs vLLM BF16

| Model | Hardware | mistral.rs | vLLM |

|---|---|---:|---:|

| Gemma 4 E4B | GB10 | 25.1 | 18.8 |

| Gemma 4 E4B | B200 | 202.6 | 196.2 |

| Gemma 4 E4B | H100 SXM | 174.4 | 153.0 |

| Gemma 4 26B-A4B | GB10 | 26.9 | 23.2 |

| Gemma 4 26B-A4B | B200 | 159.6 | 220.2 |

| Gemma 4 26B-A4B | H100 SXM | 138.7 | 148.0 |

</details>

Why mistral.rs?

• Any Hugging Face model, zero config: Just mistralrs run -m user/model. Architecture, quantization format, and chat template are auto-detected.
• True multimodality: Text, vision, video, and audio, speech generation, image generation, and embeddings in one engine.
• Smart quantization: --quant automatically selects the best quantization format at that level: using a prebuilt UQFF if one is published, otherwise applying ISQ. Docs
• OpenAI + Anthropic compatible serving: The same mistralrs serve process exposes OpenAI-compatible /v1 endpoints and Anthropic-compatible Messages endpoints.
• Prometheus metrics: mistralrs serve exposes a /metrics endpoint in Prometheus format, recording per-request counts and latency labeled by method, route, and status. Docs
• Built-in web UI: Served at /ui by default. Shows reasoning, code execution, plots, and files inline. Edit any message and the new branch runs with its own Python state. Pass --no-ui to disable.
• Hardware-aware: mistralrs tune recommends quantization and device mapping from the model config and your detected hardware.
• Flexible SDKs: Python package and Rust crate to build your projects.
• Native agentic support: built-in agentic loop with web search, local Python code execution, shell execution, OpenAI-compatible Skills, session management, and custom tool hooks.

Quick Start

Install

Linux/macOS:

curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.ps1 | iex

Downloads a self-contained prebuilt binary for your platform (Metal on Apple Silicon; per-GPU CUDA or CPU on Linux; CPU on Windows), falling back to a source build if none matches. Standard acceleration needs no Rust or CUDA toolkit. Optional cuTile acceleration requires NVIDIA's separately installed tileiras tool.

Manual installation, accelerator details & other platforms

Run Your First Model

# Interactive chat
mistralrs run -m Qwen/Qwen3-4B

# One-shot prompt (no interactive session)
mistralrs run -m Qwen/Qwen3-4B -i "What is the capital of France?"

# One-shot with an image
mistralrs run -m google/gemma-4-E4B-it --image photo.jpg -i "Describe this image"

# Agentic REPL: search + code execution + shell from the terminal
mistralrs run --agent -m Qwen/Qwen3-4B

# Start an API server with the built-in web UI
mistralrs serve -m google/gemma-4-E4B-it

For the server command, visit http://localhost:1234/ui for the web chat interface. OpenAI-compatible clients use http://localhost:1234/v1; Anthropic-compatible clients use http://localhost:1234.

The mistralrs CLI

The CLI is designed to be zero-config: just point it at a model and go.

• Auto-detection: Automatically detects model architecture, quantization format, and chat template
• All-in-one: Single binary for chat, server, benchmarks, and web UI (run, serve, bench)
• Hardware-aware tuning: mistralrs tune recommends quantization and device mapping for your model and hardware
• Format-agnostic: Works with Hugging Face models, GGUF files, and UQFF quantizations seamlessly

# Recommend settings for your hardware and emit a config file
mistralrs tune -m Qwen/Qwen3-4B --emit-config config.toml

# Run using the generated config
mistralrs from-config -f config.toml

# Diagnose system issues (CUDA, Metal, HuggingFace connectivity)
mistralrs doctor

Full CLI documentation

<details open>

<summary><b>UI Demo</b></summary>

<br>

<img src="https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/res/ui.gif" alt="UI Demo" />

</details>

What Makes It Fast

Performance

• Continuous batching support by default on all devices.
• CUDA with FlashAttention V2/V3, Metal, and multi-GPU/distributed inference
• PagedAttention for high throughput continuous batching on CUDA or Apple Silicon, prefix caching (including multimodal)

Quantization (full docs)

• In-situ quantization (ISQ) of any Hugging Face model
• GGUF (2-8 bit), GPTQ, AWQ, HQQ, FP8, BNB support
• ⭐ Per-layer topology: Fine-tune quantization per layer for optimal quality/speed
• ⭐ Auto-select fastest quant method for your hardware

Flexibility

• LoRA & X-LoRA with per-request LoRA selection and X-LoRA adapter mixing
• AnyMoE: Create mixture-of-experts on any base model
• Multiple models: Load/unload at runtime

Agentic Features

• Integrated tool calling with grammar enforcement and strict schema mode
• ⭐ Server-side agentic loop: auto-execute tools and feed results back
• ⭐ Python code execution: persistent Jupyter-like sessions with matplotlib capture and multimodal feedback
• ⭐ Shell execution: persistent command-line sessions with sandboxing and approval controls
• ⭐ OpenAI-compatible Skills: uploaded skill bundles for Responses API agents
• ⭐ OpenAI-compatible file inputs: /v1/files, Responses input_file, Chat file, and workdir mounts
• ⭐ Web search integration with embedding-based ranking
• ⭐ Tool dispatch URL: POST tool calls to your own endpoint
• ⭐ MCP client: Connect to external tools via Process, HTTP, or WebSocket
• Python/Rust tool callbacks for custom execution

Full feature documentation

Supported Models

Text, multimodal, speech, image generation, and embedding models across 45+ architectures. The supported models reference is the single source of truth: it explains how to check whether your model's config.json is supported, lists every architecture with copy-paste run commands, and is generated directly from the engine's loader registry so it never drifts.

Supported models reference | Request a new model

Python SDK

pip install mistralrs

In-process inference from Python: load a model with Runner and send OpenAI-shaped requests, no server required. Accelerator-specific wheels (CUDA, Metal, MKL, Accelerate) are listed in the getting-started guide.

Get started | API reference | Examples

Rust SDK

cargo add mistralrs

Embed the engine in a Rust application with the high-level mistralrs crate.

Get started | docs.rs | Crate | Examples

Docker

Prebuilt CPU and CUDA images are published to GHCR. Pull commands, tags, and Kubernetes notes are in the Docker guide.

Documentation

For complete documentation, see the Documentation.

Quick Links:

• Quickstart - Install, first run, first serve
• CLI Reference - All commands and options
• Anthropic Messages API - Anthropic-compatible Messages, streaming, tool use, and token counting
• HTTP API - OpenAI-compatible and Anthr