llama-cpp-python

<p align="center">

<img src="https://raw.githubusercontent.com/abetlen/llama-cpp-python/main/docs/icon.svg" style="height: 5rem; width: 5rem">

</p>

Python Bindings for llama.cpp

Documentation Status

Tests

PyPI

PyPI - Python Version

PyPI - License

PyPI - Downloads

[Github All Releases]()

Simple Python bindings for @ggerganov's llama.cpp library.

This package provides:

• Low-level access to C API via ctypes interface.
• High-level Python API for text completion
• OpenAI-like API
• LangChain compatibility
• LlamaIndex compatibility
• OpenAI compatible web server
• Local Copilot replacement
• Function Calling support
• Vision API support
• Multiple Models

Documentation is available at https://llama-cpp-python.readthedocs.io/en/latest.

Installation

Requirements:

• Python 3.8+
• C compiler
• Linux: gcc or clang
• Windows: Visual Studio or MinGW
• MacOS: Xcode

To install the package, run:

pip install llama-cpp-python

This will also build llama.cpp from source and install it alongside this python package.

If this fails, add --verbose to the pip install see the full cmake build log.

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with basic CPU support.

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

Installation Configuration

llama.cpp supports a number of hardware acceleration backends to speed up inference as well as backend specific options. See the llama.cpp README for a full list.

All llama.cpp cmake build options can be set via the CMAKE_ARGS environment variable or via the --config-settings / -C cli flag during installation.

<details open>

<summary>Environment Variables</summary>

# Linux and Mac
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" \
  pip install llama-cpp-python

# Windows
$env:CMAKE_ARGS = "-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS"
pip install llama-cpp-python

</details>

<details>

<summary>CLI / requirements.txt</summary>

They can also be set via pip install -C / --config-settings command and saved to a requirements.txt file:

pip install --upgrade pip # ensure pip is up to date
pip install llama-cpp-python \
  -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

# requirements.txt

llama-cpp-python -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

</details>

Supported Backends

Below are some common backends, their build commands and any additional environment variables required.

<details open>

<summary>OpenBLAS (CPU)</summary>

To install with OpenBLAS, set the GGML_BLAS and GGML_BLAS_VENDOR environment variables before installing:

CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" pip install llama-cpp-python

</details>

<details>

<summary>CUDA</summary>

To install with CUDA support, set the GGML_CUDA=on environment variable before installing:

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with CUDA support. As long as your system meets some requirements:

• CUDA Version is 11.8, 12.1, 12.2, 12.3, 12.4, 12.5, 13.0 or 13.2
• NVIDIA GPU compute capability is 6.0 through 8.9 for CUDA 11.8 wheels, 6.0 or newer for CUDA 12 wheels, or 7.5 or newer for CUDA 13 wheels
• Python Version is 3.10, 3.11 or 3.12

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/<cuda-version>

Where <cuda-version> is one of the following:

• cu118: CUDA 11.8
• cu121: CUDA 12.1
• cu122: CUDA 12.2
• cu123: CUDA 12.3
• cu124: CUDA 12.4
• cu125: CUDA 12.5
• cu130: CUDA 13.0
• cu132: CUDA 13.2

For example, to install the CUDA 12.1 wheel:

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121

</details>

<details>

<summary>Metal</summary>

To install with Metal (MPS), set the GGML_METAL=on environment variable before installing:

CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with Metal support. As long as your system meets some requirements:

• MacOS Version is 11.0 or later
• Python Version is 3.10, 3.11 or 3.12

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal

</details>

<details>

<summary>HIP (ROCm)</summary>

To install with HIP / ROCm support for AMD cards, set the GGML_HIP=on environment variable before installing:

CMAKE_ARGS="-DGGML_HIP=on" pip install llama-cpp-python

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with ROCm support for Linux:

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/rocm72

Or a pre-built wheel with HIP Radeon support for Windows:

pip install llama-cpp-python `
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/hip-radeon

</details>

<details>

<summary>Vulkan</summary>

To install with Vulkan support, set the GGML_VULKAN=on environment variable before installing:

CMAKE_ARGS="-DGGML_VULKAN=on" pip install llama-cpp-python

Pre-built Wheel (New)

It is also possible to install a pre-built wheel with Vulkan support for Linux or Windows:

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/vulkan

</details>

<details>

<summary>SYCL</summary>

To install with SYCL support, set the GGML_SYCL=on environment variable before installing:

source /opt/intel/oneapi/setvars.sh
CMAKE_ARGS="-DGGML_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx" pip install llama-cpp-python

</details>

<details>

<summary>RPC</summary>

To install with RPC support, set the GGML_RPC=on environment variable before installing:

source /opt/intel/oneapi/setvars.sh
CMAKE_ARGS="-DGGML_RPC=on" pip install llama-cpp-python

</details>

Windows Notes

<details>

<summary>Error: Can't find 'nmake' or 'CMAKE_C_COMPILER'</summary>

If you run into issues where it complains it can't find 'nmake' '?' or CMAKE_C_COMPILER, you can extract w64devkit as mentioned in llama.cpp repo and add those manually to CMAKE_ARGS before running pip install:

$env:CMAKE_GENERATOR = "MinGW Makefiles"
$env:CMAKE_ARGS = "-DGGML_OPENBLAS=on -DCMAKE_C_COMPILER=C:/w64devkit/bin/gcc.exe -DCMAKE_CXX_COMPILER=C:/w64devkit/bin/g++.exe"

See the above instructions and set CMAKE_ARGS to the BLAS backend you want to use.

</details>

MacOS Notes

Detailed MacOS Metal GPU install documentation is available at docs/install/macos.md

<details>

<summary>M1 Mac Performance Issue</summary>

Note: If you are using Apple Silicon (M1) Mac, make sure you have installed a version of Python that supports arm64 architecture. For example:

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.sh

Otherwise, while installing it will build the llama.cpp x86 version which will be 10x slower on Apple Silicon (M1) Mac.

</details>

<details>

<summary>M Series Mac Error: (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))</summary>

Try installing with

CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64 -DGGML_METAL=on" pip install --upgrade --verbose --force-reinstall --no-cache-dir llama-cpp-python

</details>

Upgrading and Reinstalling

To upgrade and rebuild llama-cpp-python add --upgrade --force-reinstall --no-cache-dir flags to the pip install command to ensure the package is rebuilt from source.

High-level API

API Reference

The high-level API provides a simple managed interface through the Llama class.

Below is a short example demonstrating how to use the high-level API for basic text completion:

from llama_cpp import Llama

llm = Llama(
      model_path="./models/7B/llama-model.gguf",
      # n_gpu_layers=-1, # Uncomment to use GPU acceleration
      # seed=1337, # Uncomment to set a specific seed
      # n_ctx=2048, # Uncomment to increase the context window
)
output = llm(
      "Q: Name the planets in the solar system? A: ", # Prompt
      max_tokens=32, # Generate up to 32 tokens, set to None to generate up to the end of the context window
      stop=["Q:", "\n"], # Stop generating just before the model would generate a new question
      echo=True # Echo the prompt back in the output
) # Generate a completion, can also call create_completion
print(output)

By default llama-cpp-python generates completions in an OpenAI compatible format:

{
  "id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "object": "text_completion",
  "created": 1679561337,
  "model": "./models/7B/llama-model.gguf",
  "choices": [
    {
      "text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
      "index": 0,
      "logprobs": None,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 28,
    "total_tokens": 42
  }
}

Text completion is available through the __call__ and create_completion methods of the Llama class.

Pulling models from Hugging Face Hub

You can download Llama models in gguf format directly from Hugging Face using the from_pretrained method.

You'll need to install the huggingface-hub package to use this feature (pip install huggingface-hub).

llm = Llama.from_pretrained(
    repo_id="lmstudio-community/Qwen3.5-0.8B-GGUF",
    filename="*Q8_0.gguf",
    verbose=False
)

By default from_pretrained will download the model to the huggingface cache directory, you can then manage installed model files with the hf tool.

Chat Completion

The high-level API also provides a simple interface for chat completion.

Chat completion requires that the model knows how to format the messages into a single prompt.

The Llama class does this using pre-registered chat formats (ie. chatml, llama-2, gemma, etc) or by providing a custom chat handler object.

The model will format the messages into a single prompt using the following order of precedence:

• Use the chat_handler if provided
• Use the chat_format if provided
• Use the tokenizer.chat_template from the gguf model's metadata (should work for most new models, older models may not have this)
• else, fallback to the llama-2 chat format

Set verbose=True to see the selected chat format.

from llama_cpp import Llama
llm = Llama(
      model_path="path/to/llama-2/llama-model.gguf",
      chat_format="llama-2"
)
llm.create_chat_completion(
      messages = [
          {"role": "system", "content": "You are an assistant who perfectly describes images."},
          {
              "role": "user",
              "content": "Describe this image in detail please."
          }
      ]
)

Chat completion is available through the create_chat_completion method of the Llama class.

For OpenAI API v1 compatibility, you use the [create_chat_completion_openai_v1](https://llama-cpp-python.readthedocs