LMDeploy

<!-- Release notes generated using configuration in .github/release.yml at main -->

What's Changed

🚀 Features

• Support long-context and MTP prefix-cache hits by @grimoire in https://github.com/InternLM/lmdeploy/pull/4688
• [Feature] Add guided decoding support for speculative decoding by @windreamer in https://github.com/InternLM/lmdeploy/pull/4559
• feat(turbomind): memory allocator, object cache, and scheduler integration by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4717
• feat: add AgRs all2all backend by @irexyc in https://github.com/InternLM/lmdeploy/pull/4739
• DeepSeek V4 support by @grimoire in https://github.com/InternLM/lmdeploy/pull/4554
• Support memdecode by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4767

💥 Improvements

• Force blksize=128 for linear attention on ascend by @jinminxi104 in https://github.com/InternLM/lmdeploy/pull/4705
• refactor: unify interleaved MRoPE rotary embedding by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4644
• Refine multi-node support on ascend-A3 by @jinminxi104 in https://github.com/InternLM/lmdeploy/pull/4711
• [Improve]: Remove dlblas from lmdeploy by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4682
• replace sync with wait event in h2d by @grimoire in https://github.com/InternLM/lmdeploy/pull/4709
• Respect --server-port in DP mode when proxy-url is set by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4712
• add --language-model-only for text-only VLM inference and remove --disable-vision-encoder by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4716
• Optimize TTFT by @grimoire in https://github.com/InternLM/lmdeploy/pull/4695
• Optimize BaseResponseParser streaming and add parser benchmark by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4697
• Support fp8 moe only for qwen3.5 by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4740
• clear runtime state in sleep by @grimoire in https://github.com/InternLM/lmdeploy/pull/4729
• feat(serve): add --generation-config CLI for server sampling defaults by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4708
• fix: gzip torch profiler traces by default by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4747
• Optimize tp fp8 moe for small average router per expert by @grimoire in https://github.com/InternLM/lmdeploy/pull/4751
• refactor(pytorch): clarify scheduler and input-maker control flow by @grimoire in https://github.com/InternLM/lmdeploy/pull/4727
• Guard DP dummy inputs around pending work by @grimoire in https://github.com/InternLM/lmdeploy/pull/4738
• Remove interactive chat and make inference stateless by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4730
• feat(turbomind): Derive composable TurboMind parallel configurations by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4769
• Add generic tensor copy and architecture-aware Gated Delta Rule support by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4757
• Add GDR CP controls and legacy kernel override by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4779
• fix(turbomind): fix zero-centered RMSNorm for Qwen3.5 by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4790

🐞 Bug fixes

• fix prefix caching by @grimoire in https://github.com/InternLM/lmdeploy/pull/4700
• fix _reduce_split_kernel for triton 3.5.1 by @irexyc in https://github.com/InternLM/lmdeploy/pull/4696
• fix triton fp8 all_reduce group by @grimoire in https://github.com/InternLM/lmdeploy/pull/4702
• fix(serve): use unique chatcmpl id for chat completions responses by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4707
• [Bugfix] Fix ImportError in get_chat_template for builtin chat-template names by @waynehacking8 in https://github.com/InternLM/lmdeploy/pull/4690
• [Bugfix] Fix InternVL/InternVL3 LoRA loading TypeError in adapter fallback by @waynehacking8 in https://github.com/InternLM/lmdeploy/pull/4684

<div align="center">

<img src="docs/en/_static/image/lmdeploy-logo.svg" width="450"/>

PyPI

PyPI - Downloads

license

issue resolution

open issues

📘Documentation |

🛠️Quick Start |

🤔Reporting Issues

English | 简体中文 | 日本語

👋 join us on Static Badge

Static Badge

Static Badge

</div>

---

Latest News 🎉

<details open>

<summary><b>2026</b></summary>

• \[2026/04\] PyPI has expanded the storage quota for LMDeploy and wheel uploads have resumed. v0.12.3 is now available on PyPI, so you can install it directly via pip install lmdeploy.
• \[2026/02\] Support Qwen3.5
• \[2026/02\] Support vllm-project/llm-compressor 4bit symmetric/asymmetric quantization. Refer here for detailed guide

</details>

<details close>

<summary><b>2025</b></summary>

• \[2025/09\] TurboMind supports MXFP4 on NVIDIA GPUs starting from V100, achieving 1.5x the performmance of vLLM on H800 for openai gpt-oss models!
• \[2025/06\] Comprehensive inference optimization for FP8 MoE Models
• \[2025/06\] DeepSeek PD Disaggregation deployment is now supported through integration with DLSlime and Mooncake. Huge thanks to both teams!
• \[2025/04\] Enhance DeepSeek inference performance by integration deepseek-ai techniques: FlashMLA, DeepGemm, DeepEP, MicroBatch and eplb
• \[2025/01\] Support DeepSeek V3 and R1

</details>

<details close>

<summary><b>2024</b></summary>

• \[2024/11\] Support Mono-InternVL with PyTorch engine
• \[2024/10\] PyTorchEngine supports graph mode on ascend platform, doubling the inference speed
• \[2024/09\] LMDeploy PyTorchEngine adds support for Huawei Ascend. See supported models here
• \[2024/09\] LMDeploy PyTorchEngine achieves 1.3x faster on Llama3-8B inference by introducing CUDA graph
• \[2024/08\] LMDeploy is integrated into modelscope/swift as the default accelerator for VLMs inference
• \[2024/07\] Support Llama3.1 8B, 70B and its TOOLS CALLING
• \[2024/07\] Support InternVL2 full-series models, InternLM-XComposer2.5 and function call of InternLM2.5
• \[2024/06\] PyTorch engine support DeepSeek-V2 and several VLMs, such as CogVLM2, Mini-InternVL, LlaVA-Next
• \[2024/05\] Balance vision model when deploying VLMs with multiple GPUs
• \[2024/05\] Support 4-bits weight-only quantization and inference on VLMs, such as InternVL v1.5, LLaVa, InternLMXComposer2
• \[2024/04\] Support Llama3 and more VLMs, such as InternVL v1.1, v1.2, MiniGemini, InternLMXComposer2.
• \[2024/04\] TurboMind adds online int8/int4 KV cache quantization and inference for all supported devices. Refer here for detailed guide
• \[2024/04\] TurboMind latest upgrade boosts GQA, rocketing the internlm2-20b model inference to 16+ RPS, about 1.8x faster than vLLM.
• \[2024/04\] Support Qwen1.5-MOE and dbrx.
• \[2024/03\] Support DeepSeek-VL offline inference pipeline and serving.
• \[2024/03\] Support VLM offline inference pipeline and serving.
• \[2024/02\] Support Qwen 1.5, Gemma, Mistral, Mixtral, Deepseek-MOE and so on.
• \[2024/01\] OpenAOE seamless integration with LMDeploy Serving Service.
• \[2024/01\] Support for multi-model, multi-machine, multi-card inference services. For usage instructions, please refer to here
• \[2024/01\] Support PyTorch inference engine, developed entirely in Python, helping to lower the barriers for developers and enable rapid experimentation with new features and technologies.

</details>

<details close>

<summary><b>2023</b></summary>

• \[2023/12\] Turbomind supports multimodal input.
• \[2023/11\] Turbomind supports loading hf model directly. Click here for details.
• \[2023/11\] TurboMind major upgrades, including: Paged Attention, faster attention kernels without sequence length limitation, 2x faster KV8 kernels, Split-K decoding (Flash Decoding), and W4A16 inference for sm_75
• \[2023/09\] TurboMind supports Qwen-14B
• \[2023/09\] TurboMind supports InternLM-20B
• \[2023/09\] TurboMind supports all features of Code Llama: code completion, infilling, chat / instruct, and python specialist. Click here for deployment guide
• \[2023/09\] TurboMind supports Baichuan2-7B
• \[2023/08\] TurboMind supports flash-attention2.
• \[2023/08\] TurboMind supports Qwen-7B, dynamic NTK-RoPE scaling and dynamic logN scaling
• \[2023/08\] TurboMind supports Windows (tp=1)
• \[2023/08\] TurboMind supports 4-bit inference, 2.4x faster than FP16, the fastest open-source implementation. Check this guide for detailed info
• \[2023/08\] LMDeploy has launched on the HuggingFace Hub, providing ready-to-use 4-bit models.
• \[2023/08\] LMDeploy supports 4-bit quantization using the AWQ algorithm.
• \[2023/07\] TurboMind supports Llama-2 70B with GQA.
• \[2023/07\] TurboMind supports Llama-2 7B/13B.
• \[2023/07\] TurboMind supports tensor-parallel inference of InternLM.

</details>

---

Introduction

LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the MMRazor and MMDeploy teams. It has the following core features:

• Efficient Inference: LMDeploy delivers up to 1.8x higher request throughput than vLLM, by introducing key features like persistent batch(a.k.a. continuous batching), blocked KV cache, dynamic split&fuse, tensor parallelism, high-performance CUDA kernels and so on.

• Effective Quantization: LMDeploy supports weight-only and k/v quantization, and the 4-bit inference performance is 2.4x higher than FP16. The quantization quality has been confirmed via OpenCompass evaluation.

• Effortless Distribution Server: Leveraging the request distribution service, LMDeploy facilitates an easy and efficient deployment of multi-model services across multiple machines and cards.

• Excellent Compatibility: LMDeploy supports KV Cache Quant, AWQ and Automatic Prefix Caching to be used simultaneously.

Performance

v0 1 0-benchmark

Supported Models

<table>

<tbody>

<tr align="center" valign="middle">

<td>

<b>LLMs</b>

</td>

<td>

<b>VLMs</b>

</td>

<tr valign="top">

<td align="left" valign="top">

<ul>

<li>Llama (7B - 65B)</li>

<li>Llama2 (7B - 70B)</li>

<li>Llama3 (8B, 70B)</li>

<li>Llama3.1 (8B, 70B)</li>

<li>Llama3.2 (1B, 3B)</li>

<li>InternLM2 (7B - 20B)</li>

<li>InternLM3 (8B)</li>

<li>InternLM2.5 (7B)</li>

<li>Qwen1.5 (0.5B - 110B)</li>

<li>Qwen1.5 - MoE (0.5B - 72B)</li>

<li>Qwen2 (0.5B - 72B)</li>

<li>Qwen2-MoE (57BA14B)</li>

<li>Qwen2.5 (0.5B - 32B)</li>

<li>Qwen3, Qwen3-MoE</li>

<li>Qwen3-Next(80B)</li>

<li>Code Llama (7B - 34B)</li>

<li>ChatGLM2 (6B)</li>

<li>GLM-4 (9B)</li>

<li>GLM-4-0414 (9B, 32B)</li>

<li>CodeGeeX4 (9B)</li>

<li>YI (6B-34B)</li>

<li>Mistral (7B)</li>

<li>DeepSeek-MoE (16B)</li>

<li>DeepSeek-V2 (16B, 236B)</li>

<li>DeepSeek-V2.5 (236B)</li>

<li>DeepSeek-V3 (685B)</li>

<li>DeepSeek-V3.2 (685B)</li>

<li>Mixtral (8x7B, 8x22B)</li>

<li>Gemma (2B - 7B)</li>

<li>Phi-3-mini (3.8B)</li>

<li>Phi-3.5-mini (3.8B)</li>

<li>Phi-3.5-MoE (16x3.8B)</li>

<li>Phi-4-mini (3.8B)</li>

<li>MiniCPM3 (4B)</li>

<li>SDAR (1.7B-30B)</li>

<li>gpt-oss (20B, 120B)</li>

<li>GLM-4.7-Flash (30B)</li>

<li>GLM-5 (754B)</li>

</ul>

</td>

<td>

<ul>

<li>LLaVA(1.5,1.6) (7B-34B)</li>

<li>Qwen2-VL (2B, 7B, 72B)</li>

<li>Qwen2.5-VL (3B, 7B, 72B)</li>

<li>Qwen3-VL (2B - 235B)</li>

<li>Qwen3.5 (0.8B - 397B)</li>

<li>Qwen3-Omni (30B-A3B)</li>

<li>DeepSeek-VL (7B)</li>

<li>DeepSeek-VL2 (3B, 16B, 27B)</li>

<li>InternVL-Chat (v1.1-v1.5)</li>

<li>InternVL2 (1B-76B)</li>

<li>InternVL2.5(MPO) (1B-78B)</li>

<li>InternVL3 (1B-78B)</li>

<li>InternVL3.5 (1B-241BA28B)</li>

<li>Intern-S1 (241B)</li>

<li>Intern-S1-mini (8.3B)</li>

<li>Intern-S1-Pro (1TB)</li>

<li>Intern-S2-Preview (35B-A3B)</li>

<li>ChemVLM (8B-26B)</li>

<li>CogVLM-Chat (17B)</li>

<li>CogVLM2-Chat (19B)</li>

<li>MiniCPM-Llama3-V-2_5</li>

<li>MiniCPM-V-2_6</li>

<li>Phi-3-vision (4.2B)</li>

<li>Phi-3.5-vision (4.2B)</li>

<li>GLM-4V (9B)</li>

<li>GLM-4.1V-Thinking (9B)</li>

<li>Molmo (7B-D,72B)</li>

<li>Gemma3 (1B - 27B)</li>

<li>Llama4 (Scout, Maverick)</li>

</ul>

</td>

</tr>

</tbody>

</table>

LMDeploy has developed two inference engines - TurboMind and PyTorch, each with a different focus. The former strives for ultimate optimization of inference performance, while the latter, developed purely in Python, aims to decrease the barriers for developers.

They differ in the types of supported models and the inference data type. Please refer to this table for each engine's capability and choose the proper one that best fits your actual needs.

Quick Start Open In Colab

Installation

It is recommended installing lmdeploy using pip in a conda environment (python 3.10 - 3.13):

conda create -n lmdeploy python=3.12 -y
conda activate lmdeploy
pip install lmdeploy

Starting from v0.13.0, the default prebuilt wheels published on PyPI are built against CUDA 12.8, so pip install lmdeploy is sufficient for typical setups including GeForce RTX 50 series.

Offline Batch Inference

import lmdeploy
with lmdeploy.pipeline("internlm/internlm3-8b-instruct") as pipe:
    response = pipe(["Hi, pls intro yourself", "Shanghai is"])
    print(response)

\[!NOTE\]

By default, LMDeploy downloads model from HuggingFace. If you would like to use models from ModelScope, please install ModelScope by pip install modelscope and set the environment variable:

export LMDEPLOY_USE_MODELSCOPE=True

If you would like to use models from openMind Hub, please install openMind Hub by pip install openmind_hub and set the environment variable:

export LMDEPLOY_USE_OPENMIND_HUB=True

For more information about inference pipeline, please refer to here.

Tutorials

Please review getting_started section for the basic usage of LMDeploy.

For detailed user guides and advanced guides, please refer to our tutorials:

• User Guide
• LLM Inference pipeline Open In Colab
• VLM Inference pipeline Open In Colab
• LLM Serving
• VLM Serving
• Quantization
• Advance Guide
• Inference Engine - TurboMind
• Inference Engine - PyTorch
• Customize chat templates
• Add a new model
• gemm tuning
• Long context inference
• Multi-model inference service

Third-party projects

• Deploying LLMs offline on the NVIDIA Jetson platform by LMDeploy: LMDeploy-Jetson

• Example project for deploying LLMs using LMDeploy and BentoML: BentoLMDeploy

Contributing

We appreciate all contributions to LMDeploy. Please refer to CONTRIBUTING.md for the contributing guideline.

Acknowledgement

• [FasterTransformer](https://