FastChat

Highlights

• Added SGLang worker for vision language models, lower latency and higher throughput https://github.com/lm-sys/FastChat/pull/2928
• Vision langauge WebUI https://github.com/lm-sys/FastChat/pull/2960
• OpenAI-compatible API server now supports image input https://github.com/lm-sys/FastChat/pull/2928
• Added LightLLM worker for higher throughput https://github.com/lm-sys/FastChat/blob/main/docs/lightllm_integration.md
• Added Apple MLX worker https://github.com/lm-sys/FastChat/pull/2940

What's Changed

• fix specify local path issue use model from www.modelscope.cn by @liuyhwangyh in https://github.com/lm-sys/FastChat/pull/2934
• support openai embedding for topic clustering by @CodingWithTim in https://github.com/lm-sys/FastChat/pull/2729
• Remove duplicate API endpoint by @surak in https://github.com/lm-sys/FastChat/pull/2949
• Update Hermes Mixtral by @teknium1 in https://github.com/lm-sys/FastChat/pull/2938
• Enablement of REST API Usage within Google Colab Free Tier by @ggcr in https://github.com/lm-sys/FastChat/pull/2940
• Create a new worker implementation for Apple MLX by @aliasaria in https://github.com/lm-sys/FastChat/pull/2937
• feat: support Model Yuan2.0, a new generation Fundamental Large Language Model developed by IEIT System by @cauwulixuan in https://github.com/lm-sys/FastChat/pull/2936
• Fix the pooling method of BGE embedding model by @staoxiao in https://github.com/lm-sys/FastChat/pull/2926
• SGLang Worker by @BabyChouSr in https://github.com/lm-sys/FastChat/pull/2928
• Update mlx_worker to be async by @aliasaria in https://github.com/lm-sys/FastChat/pull/2958
• Integrate LightLLM into serve worker by @zeyugao in https://github.com/lm-sys/FastChat/pull/2888
• Copy button by @surak in https://github.com/lm-sys/FastChat/pull/2963
• feat: train with template by @congchan in https://github.com/lm-sys/FastChat/pull/2951
• fix content maybe a str by @zhouzaida in https://github.com/lm-sys/FastChat/pull/2968
• Adding download folder information in README by @dheeraj-326 in https://github.com/lm-sys/FastChat/pull/2972
• use cl100k_base as the default tiktoken encoding by @bjwswang in https://github.com/lm-sys/FastChat/pull/2974
• Update README.md by @merrymercy in https://github.com/lm-sys/FastChat/pull/2975
• Fix tokenizer for vllm worker by @Michaelvll in https://github.com/lm-sys/FastChat/pull/2984
• update yuan2.0 generation by @wangpengfei1013 in https://github.com/lm-sys/FastChat/pull/2989
• fix: tokenization mismatch when training with different templates by @congchan in https://github.com/lm-sys/FastChat/pull/2996
• fix: inconsistent tokenization by llama tokenizer by @congchan in https://github.com/lm-sys/FastChat/pull/3006
• Fix type hint for play_a_match_single by @MonkeyLeeT in https://github.com/lm-sys/FastChat/pull/3008
• code update by @infwinston in https://github.com/lm-sys/FastChat/pull/2997
• Update model_support.md by @infwinston in https://github.com/lm-sys/FastChat/pull/3016
• Update lightllm_integration.md by @eltociear in https://github.com/lm-sys/FastChat/pull/3014
• Upgrade gradio to 4.17 by @infwinston in https://github.com/lm-sys/FastChat/pull/3027
• Update MLX integration to use new generate_step function signature by @aliasaria in https://github.com/lm-sys/FastChat/pull/3021
• Update readme by @merrymercy in https://github.com/lm-sys/FastChat/pull/3028
• Update gradio version in pyproject.toml and fix a bug by @merrymercy in https://github.com/lm-sys/FastChat/pull/3029
• Update gradio demo and API model providers by @merrymercy in https://github.com/lm-sys/FastChat/pull/3030
• Gradio Web Server for Multimodal Models by @BabyChouSr in https://github.com/lm-sys/FastChat/pull/2960
• Migrate the gradio server to openai v1 by @merrymercy in https://github.com/lm-sys/FastChat/pull/3032
• Update version to 0.2.36 by @merrymercy in https://github.com/lm-sys/FastChat/pull/3033

New Contributors

• @teknium1 made their first contrib

FastChat

| Demo | Discord | X |

FastChat is an open platform for training, serving, and evaluating large language model based chatbots.

• FastChat powers Chatbot Arena (lmarena.ai), serving over 10 million chat requests for 70+ LLMs.
• Chatbot Arena has collected over 1.5M human votes from side-by-side LLM battles to compile an online LLM Elo leaderboard.

FastChat's core features include:

• The training and evaluation code for state-of-the-art models (e.g., Vicuna, MT-Bench).
• A distributed multi-model serving system with web UI and OpenAI-compatible RESTful APIs.

News

• [2024/03] 🔥 We released Chatbot Arena technical report.
• [2023/09] We released LMSYS-Chat-1M, a large-scale real-world LLM conversation dataset. Read the report.
• [2023/08] We released Vicuna v1.5 based on Llama 2 with 4K and 16K context lengths. Download weights.
• [2023/07] We released Chatbot Arena Conversations, a dataset containing 33k conversations with human preferences. Download it here.

<details>

<summary>More</summary>

• [2023/08] We released LongChat v1.5 based on Llama 2 with 32K context lengths. Download weights.
• [2023/06] We introduced MT-bench, a challenging multi-turn question set for evaluating chatbots. Check out the blog post.
• [2023/06] We introduced LongChat, our long-context chatbots and evaluation tools. Check out the blog post.
• [2023/05] We introduced Chatbot Arena for battles among LLMs. Check out the blog post.
• [2023/03] We released Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality. Check out the blog post.

</details>

<a href="https://lmarena.ai"><img src="assets/demo_narrow.gif" width="70%"></a>

Contents

• Install
• Model Weights
• Inference with Command Line Interface
• Serving with Web GUI
• API
• Evaluation
• Fine-tuning
• Citation

Install

Method 1: With pip

pip3 install "fschat[model_worker,webui]"

Method 2: From source

1. Clone this repository and navigate to the FastChat folder

git clone https://github.com/lm-sys/FastChat.git
cd FastChat

If you are running on Mac:

brew install rust cmake

1. Install Package

pip3 install --upgrade pip  # enable PEP 660 support
pip3 install -e ".[model_worker,webui]"

Model Weights

Vicuna Weights

Vicuna is based on Llama 2 and should be used under Llama's model license.

You can use the commands below to start chatting. It will automatically download the weights from Hugging Face repos.

Downloaded weights are stored in a .cache folder in the user's home folder (e.g., ~/.cache/huggingface/hub/<model_name>).

See more command options and how to handle out-of-memory in the "Inference with Command Line Interface" section below.

NOTE: transformers>=4.31 is required for 16K versions.

| Size | Chat Command | Hugging Face Repo |

| --- | --- | --- |

| 7B | python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 | lmsys/vicuna-7b-v1.5 |

| 7B-16k | python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5-16k | lmsys/vicuna-7b-v1.5-16k |

| 13B | python3 -m fastchat.serve.cli --model-path lmsys/vicuna-13b-v1.5 | lmsys/vicuna-13b-v1.5 |

| 13B-16k | python3 -m fastchat.serve.cli --model-path lmsys/vicuna-13b-v1.5-16k | lmsys/vicuna-13b-v1.5-16k |

| 33B | python3 -m fastchat.serve.cli --model-path lmsys/vicuna-33b-v1.3 | lmsys/vicuna-33b-v1.3 |

Old weights: see docs/vicuna_weights_version.md for all versions of weights and their differences.

Other Models

Besides Vicuna, we also released two additional models: LongChat and FastChat-T5.

You can use the commands below to chat with them. They will automatically download the weights from Hugging Face repos.

| Model | Chat Command | Hugging Face Repo |

| --- | --- | --- |

| LongChat-7B | python3 -m fastchat.serve.cli --model-path lmsys/longchat-7b-32k-v1.5 | lmsys/longchat-7b-32k |

| FastChat-T5-3B | python3 -m fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0 | lmsys/fastchat-t5-3b-v1.0 |

Inference with Command Line Interface

<a href="https://lmarena.ai"><img src="assets/screenshot_cli.png" width="70%"></a>

(Experimental Feature: You can specify --style rich to enable rich text output and better text streaming quality for some non-ASCII content. This may not work properly on certain terminals.)

Supported Models

FastChat supports a wide range of models, including

LLama 2, Vicuna, Alpaca, Baize, ChatGLM, Dolly, Falcon, FastChat-T5, GPT4ALL, Guanaco, MTP, OpenAssistant, OpenChat, RedPajama, StableLM, WizardLM, xDAN-AI and more.

See a complete list of supported models and instructions to add a new model here.

Single GPU

The command below requires around 14GB of GPU memory for Vicuna-7B and 28GB of GPU memory for Vicuna-13B.

See the "Not Enough Memory" section below if you do not have enough memory.

--model-path can be a local folder or a Hugging Face repo name.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5

Multiple GPUs

You can use model parallelism to aggregate GPU memory from multiple GPUs on the same machine.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2

Tips:

Sometimes the "auto" device mapping strategy in huggingface/transformers does not perfectly balance the memory allocation across multiple GPUs.

You can use --max-gpu-memory to specify the maximum memory per GPU for storing model weights.

This allows it to allocate more memory for activations, so you can use longer context lengths or larger batch sizes. For example,

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2 --max-gpu-memory 8GiB

CPU Only

This runs on the CPU only and does not require GPU. It requires around 30GB of CPU memory for Vicuna-7B and around 60GB of CPU memory for Vicuna-13B.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpu

Use Intel AI Accelerator AVX512_BF16/AMX to accelerate CPU inference.

CPU_ISA=amx python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpu

Metal Backend (Mac Computers with Apple Silicon or AMD GPUs)

Use --device mps to enable GPU acceleration on Mac computers (requires torch >= 2.0).

Use --load-8bit to turn on 8-bit compression.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device mps --load-8bit

Vicuna-7B can run on a 32GB M1 Macbook with 1 - 2 words / second.

Intel XPU (Intel Data Center and Arc A-Series GPUs)

Install the Intel Extension for PyTorch. Set the OneAPI environment variables:

source /opt/intel/oneapi/setvars.sh

Use --device xpu to enable XPU/GPU acceleration.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device xpu

Vicuna-7B can run on an Intel Arc A770 16GB.

Ascend NPU

Install the Ascend PyTorch Adapter. Set the CANN environment variables:

source /usr/local/Ascend/ascend-toolkit/set_env.sh

Use --device npu to enable NPU acceleration.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device npu

Vicuna-7B/13B can run on an Ascend NPU.

Not Enough Memory

If you do not have enough memory, you can enable 8-bit compression by adding --load-8bit to commands above.

This can reduce memory usage by around half with slightly degraded model quality.

It is compatible with the CPU, GPU, and Metal backend.

Vicuna-13B with 8-bit compression can run on a single GPU with 16 GB of VRAM, like an Nvidia RTX 3090, RTX 4080, T4, V100 (16GB), or an AMD RX 6800 XT.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --load-8bit

In addition to that, you can add --cpu-offloading to commands above to offload weights that don't fit on your GPU onto the CPU memory.

This requires 8-bit compression to be enabled and the bitsandbytes package to be installed, which is only available on linux operating systems.

More Platforms and Quantization

• For AMD GPU users, please install ROCm and the ROCm version of PyTorch before you install FastChat. See also this post.
• FastChat supports ExLlama V2. See docs/exllama_v2.md.
• FastChat supports GPTQ 4bit inference with GPTQ-for-LLaMa. See docs/gptq.md.
• FastChat supports AWQ 4bit inference with mit-han-lab/llm-awq. See docs/awq.md.
• MLC LLM, backed by TVM Unity compiler, deploys Vicuna natively on phones, consumer-class GPUs and web browsers via Vulkan, Metal, CUDA and WebGPU.

Use models from modelscope

For Chinese users, you can use models from www.modelscope.cn via specify the following environment variables.

export FASTCHAT_USE_MODELSCOPE=True

Serving with Web GUI

<a href="https://lmarena.ai"><img src="assets/screenshot_gui.png" width="70%"></a>

To serve using the web UI, you need three main components: web servers that interface with users, model workers that host one or more models, and a controller to coordinate the webserver and model workers. You can learn more about the architecture here.

Here are the commands to follow in your terminal:

Launch the controller

python3 -m fastchat.serve.controller

This controller manages the distributed workers.

Launch the model worker(s)

python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5

Wait until the process finishes loading the model and you see "Uvicorn running on ...". The model worker will register itself to the controller .

To ensure that your model worker is connected to your controller properly, send a test message using the following command:

python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5

You will see a short output.

Launch the Gradio web server

python3 -m fastchat.serve.gradio_web_server

This is the user interface that users will interact with.

By following these steps, you will be able to serve your models using the web UI. You can open your browser and chat with a model now.

If the models do not show up, try to reboot the gradio web server.

Launch Chatbot Arena (side-by-side battle UI)

Currently, Chatbot Arena is powered by FastChat. Here is how you can launch an instance of Chatbot Arena locally.

FastChat supports popular API-based models such as OpenAI, Anthropic, Gemini, Mistral and more. To add a custom API, please refer to the model support doc. Below we take OpenAI models as an example.

Create a JSON configuration file api_endpoint.json with the api endpoints of the models you want to serve, for example:

{
    "gpt-4o-2024-05-13": {
        "model_name": "gpt-4o-2024-05-13",
        "api_base": "https://api.openai.com/v1",
        "api_type": "openai",
        "api_key": [Insert API Key],
        "anony_only": false
    }
}

For Anthropic models, specify "api_type": "anthropic_message" with your Anthropic key. Similarly, for gemini model, specify "api_type": "gemini". More details can be found in api_provider.py.

To serve your own model using local gpus, follow the instructions in Serving with Web GUI.

Now you're ready to launch the server:

python3 -m fastchat.serve.gradio_web_server_multi --register-api-endpoint-file api_endpoint.json

(Optional): Advanced Features, Scalability, Third Party UI

• You can register multiple model workers to a single controller, which can be used for serving a single model with higher throughput or serving multiple models at the same time. When doing so, please allocate different GPUs and ports for different model workers.

# worker 0
CUDA_VISIBLE_DEVICES=0 python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5 --controller http://localhost:21001 --port 31000 --worker http://localhost:31000
# worker 1
CUDA_VISIBLE_DEVICES=1 python3 -m fastchat.serve.model_worker --model-path lmsys/fastchat-t5-3b-v1.0 --controller http://localhost:21001 --port 31001 --worker http://localhost:31001

• You can also launch a multi-tab gradio server, which includes the Chatbot Arena tabs.

python3 -m fastchat.serve.gradio_web_server_multi

• The default model worker based on huggi