llama.cpp

<details open>

server: support inp embd to generate next token (#26313)

• server: support embd for sampled token

• fix ~server_batch()

</details>

Website:

• <https://llama.app>

macOS/iOS:

• macOS Apple Silicon (arm64)
• macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
• macOS Intel (x64)
• iOS XCFramework

Linux:

• Ubuntu x64 (CPU)
• Ubuntu arm64 (CPU)
• Ubuntu s390x (CPU)
• Ubuntu x64 (Vulkan)
• Ubuntu arm64 (Vulkan)
• Ubuntu x64 (ROCm 7.2)
• Ubuntu x64 (OpenVINO)
• Ubuntu x64 (SYCL FP32)
• Ubuntu x64 (SYCL FP16)

Android:

• Android arm64 (CPU)

Windows:

• Windows x64 (CPU)
• Windows arm64 (CPU)
• Windows arm64 (OpenCL Adreno)
• Windows x64 (CUDA 12) - CUDA 12.4 DLLs
• Windows x64 (CUDA 13) - CUDA 13.3 DLLs
• Windows x64 (Vulkan)
• Windows x64 (OpenVINO)
• Windows x64 (SYCL)
• Windows x64 (HIP)

openEuler:

• DISABLED
• openEuler x86 (310p)
• openEuler x86 (910b, ACL Graph)
• openEuler aarch64 (310p)
• openEuler aarch64 (910b, ACL Graph)

UI:

• UI

llama.cpp

llama

<div align="center">

<b>LLM inference in C/C++</b>

License: MIT

Release

Server

Docker

Winget

manifesto / ggml / ops / maintainer PRs%20sort%3Aupdated-desc) / dev branches / compile times / lib llama API / llama-server REST API

</div>

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

<table align="center">

<tr>

<td align="center" width=50%>

<img width="1310" height="888" alt="VLM session with llama cli" src="https://github.com/user-attachments/assets/88726b48-1713-48aa-a525-95a02e78afc4" />

<i>VLM session with <b>llama cli</b></i>

</td>

<td align="center">

<img width="1392" height="958" alt="Built-in web UI against llama serve running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" />

<i>Built-in web UI against <b>llama serve</b></i>

</td>

</tr>

<table>

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on

a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

| Backend | Target devices |

| --- | --- |

| BLAS | All |

| BLIS | All |

| CANN | Ascend NPU |

| CUDA | Nvidia GPU |

| HIP | AMD GPU |

| [Hexagon [In Progress]](docs/backend/snapdragon/README.md) | Snapdragon |

| IBM zDNN | IBM Z & LinuxONE |

| MUSA | Moore Threads GPU |

| Metal | Apple Silicon |

| OpenCL | Adreno GPU |

| [OpenVINO [In Progress]](docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs |

| RPC | All |

| SYCL | Intel GPU |

| VirtGPU | VirtGPU APIR |

| Vulkan | GPU |

| WebGPU | All |

| ZenDNN | AMD CPU |

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
• subprocess.h - Single-header process launching solution for C and C++ - Public domain