Text Embeddings Inference

What's Changed

• Use rust-toolchain.toml before rustup on Dockerfile-{cuda,cuda-all} by @alvarobartt in https://github.com/huggingface/text-embeddings-inference/pull/842
• fix(backend): replace bare except with Exception in device check by @llukito in https://github.com/huggingface/text-embeddings-inference/pull/821
• Set version 1.9.3 by @alvarobartt in https://github.com/huggingface/text-embeddings-inference/pull/849

New Contributors

• @llukito made their first contribution in https://github.com/huggingface/text-embeddings-inference/pull/821

Full Changelog: https://github.com/huggingface/text-embeddings-inference/compare/v1.9.2...v1.9.3

<div align="center">

Text Embeddings Inference

<a href="https://github.com/huggingface/text-embeddings-inference">

<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/huggingface/text-embeddings-inference?style=social">

</a>

<a href="https://huggingface.github.io/text-embeddings-inference">

<img alt="Swagger API documentation" src="https://img.shields.io/badge/API-Swagger-informational">

</a>

A blazing fast inference solution for text embeddings models.

Benchmark for BAAI/bge-base-en-v1.5 on an NVIDIA A10 with a sequence

length of 512 tokens:

<p>

<img src="assets/bs1-lat.png" width="400" />

<img src="assets/bs1-tp.png" width="400" />

</p>

<p>

<img src="assets/bs32-lat.png" width="400" />

<img src="assets/bs32-tp.png" width="400" />

</p>

</div>

Table of contents

• Get Started
• Supported Models
• Docker
• Docker Images
• API Documentation
• Using a private or gated model
• Air gapped deployment
• Using Re-rankers models
• Using Sequence Classification models
• Using SPLADE pooling
• Distributed Tracing
• gRPC
• Local Install
• Apple Silicon (Homebrew)
• Docker Build
• ARM64 / aarch64
• AMD Instinct GPUs (ROCm)
• Examples

Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence

classification models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding,

Ember, GTE and E5. TEI implements many features such as:

• No model graph compilation step
• Metal support for local execution on Macs
• Small docker images and fast boot times. Get ready for true serverless!
• Token based dynamic batching
• Optimized transformers code for inference using Flash Attention,

Candle

and cuBLASLt

• Safetensors weight loading
• ONNX weight loading
• Production ready (distributed tracing with Open Telemetry, Prometheus metrics)

Get Started

Supported Models

Text Embeddings

Text Embeddings Inference currently supports Nomic, BERT, CamemBERT, XLM-RoBERTa models with absolute positions, JinaBERT

model with Alibi positions and Mistral, Alibaba GTE, Qwen2 models with Rope positions, MPNet, ModernBERT, Qwen3, and Gemma3.

Below are some examples of the currently supported models:

| MTEB Rank | Model Size | Model Type | Model ID |

|-----------|------------------------|----------------|--------------------------------------------------------------------------------------------------|

| 2 | 7.57B (Very Expensive) | Qwen3 | Qwen/Qwen3-Embedding-8B |

| 3 | 4.02B (Very Expensive) | Qwen3 | Qwen/Qwen3-Embedding-4B |

| 4 | 509M | Qwen3 | Qwen/Qwen3-Embedding-0.6B |

| 6 | 7.61B (Very Expensive) | Qwen2 | Alibaba-NLP/gte-Qwen2-7B-instruct |

| 7 | 560M | XLM-RoBERTa | intfloat/multilingual-e5-large-instruct |

| 8 | 308M | Gemma3 | google/embeddinggemma-300m (gated) |

| 15 | 1.78B (Expensive) | Qwen2 | Alibaba-NLP/gte-Qwen2-1.5B-instruct |

| 18 | 7.11B (Very Expensive) | Mistral | Salesforce/SFR-Embedding-2_R |

| 35 | 568M | XLM-RoBERTa | Snowflake/snowflake-arctic-embed-l-v2.0 |

| 41 | 305M | Alibaba GTE | Snowflake/snowflake-arctic-embed-m-v2.0 |

| 52 | 335M | BERT | WhereIsAI/UAE-Large-V1 |

| 58 | 137M | NomicBERT | nomic-ai/nomic-embed-text-v1 |

| 79 | 137M | NomicBERT | nomic-ai/nomic-embed-text-v1.5 |

| 103 | 109M | MPNet | sentence-transformers/all-mpnet-base-v2 |

| N/A | 475M-A305M | NomicBERT | nomic-ai/nomic-embed-text-v2-moe |

| N/A | 434M | Alibaba GTE | Alibaba-NLP/gte-large-en-v1.5 |

| N/A | 396M | ModernBERT | answerdotai/ModernBERT-large |

| N/A | 340M | Qwen3 | voyageai/voyage-4-nano |

| N/A | 137M | JinaBERT | jinaai/jina-embeddings-v2-base-en |

| N/A | 137M | JinaBERT | jinaai/jina-embeddings-v2-base-code |

To explore the list of best performing text embeddings models, visit the

Massive Text Embedding Benchmark (MTEB) Leaderboard.

Sequence Classification and Re-Ranking

Text Embeddings Inference currently supports CamemBERT, and XLM-RoBERTa Sequence Classification models with absolute positions.

Below are some examples of the currently supported models:

| Task | Model Type | Model ID |

|--------------------|-------------|-----------------------------------------------------------------------------------------------------------------|

| Re-Ranking | XLM-RoBERTa | BAAI/bge-reranker-large |

| Re-Ranking | XLM-RoBERTa | BAAI/bge-reranker-base |

| Re-Ranking | GTE | Alibaba-NLP/gte-multilingual-reranker-base |

| Re-Ranking | ModernBert | Alibaba-NLP/gte-reranker-modernbert-base |

| Sentiment Analysis | RoBERTa | SamLowe/roberta-base-go_emotions |

Docker

model=Qwen/Qwen3-Embedding-0.6B
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

And then you can make requests like

curl 127.0.0.1:8080/embed \
    -X POST \
    -d '{"inputs":"What is Deep Learning?"}' \
    -H 'Content-Type: application/json'

Note: To use GPUs, you need to install

the NVIDIA Container Toolkit.

NVIDIA drivers on your machine need to be compatible with CUDA version 12.2 or higher.

To see all options to serve your models:

$ text-embeddings-router --help
Text Embedding Webserver

Usage: text-embeddings-router [OPTIONS] --model-id <MODEL_ID>

Options:
      --model-id <MODEL_ID>
          The Hugging Face model ID, can be any model listed on <https://huggingface.co/models> with the `text-embeddings-inference` tag (meaning it's compatible with Text Embeddings Inference).

          Alternatively, the specified ID can also be a path to a local directory containing the necessary model files saved by the `save_pretrained(...)` methods of either Transformers or Sentence Transformers.

          [env: MODEL_ID=]

      --revision <REVISION>
          The actual revision of the model if you're referring to a model on the hub. You can use a specific commit id or a branch like `refs/pr/2`

          [env: REVISION=]

      --tokenization-workers <TOKENIZATION_WORKERS>
          Optionally control the number of tokenizer workers used for payload tokenization, validation and truncation. Default to the number of CPU cores on the machine

          [env: TOKENIZATION_WORKERS=]

      --dtype <DTYPE>
          The dtype to be forced upon the model

          [env: DTYPE=]
          [possible values: float16, float32]

      --served-model-name <SERVED_MODEL_NAME>
          The name of the model that is being served. If not specified, defaults to `--model-id`. It is only used for the OpenAI-compatible endpoints via HTTP

          [env: SERVED_MODEL_NAME=]

      --pooling <POOLING>
          Optionally control the pooling method for embedding models.

          If `pooling` is not set, the pooling configuration will be parsed from the model `1_Pooling/config.json` configuration.

          If `pooling` is set, it will override the model pooling configuration

          [env: POOLING=]

          Possible values:
          - cls:        Select the CLS token as embedding
          - mean:       Apply Mean pooling to the model embeddings
          - splade:     Apply SPLADE (Sparse Lexical and Expansion) to the model embeddings. This option is only available if the loaded model is a `ForMaskedLM` Transformer model
          - last-token: Select the last token as embedding

      --max-concurrent-requests <MAX_CONCURRENT_REQUESTS>
          The maximum amount of concurrent requests for this particular deployment. Having a low limit will refuse clients requests instead of having them wait for too long and is usually good to handle backpressure correctly

          [env: MAX_CONCURRENT_REQUESTS=]
          [default: 512]

      --max-batch-tokens <MAX_BATCH_TOKENS>
          **IMPORTANT** This is one critical control to allow maximum usage of the available hardware.

          This represents the total amount of potential tokens within a batch.

          For `max_batch_tokens=1000`, you could fit `10` queries of `total_tokens=100` or a single query of `1000` tokens.

          Overall this number should be the largest possible until the model is compute bound. Since the actual memory overhead depends on the model implementation, text-embeddings-inference cannot infer this number automatically.

          [env: MAX_BATCH_TOKENS=]
          [default: 16384]

      --max-batch-requests <MAX_BATCH_REQUESTS>
          Optionally control the maximum number of individual requests in a batch

          [env: MAX_BATCH_REQUESTS=]

      --max-client-batch-size <MAX_CLIENT_BATCH_SIZE>
          Control the maximum number of inputs that a client can send in a single request

          [env: MAX_CLIENT_BATCH_SIZE=]
          [default: 32]

      --auto-truncate
          Control automatic truncation of inputs that exceed the model's maximum supported size. Defaults to `true` (truncation enabled). Set to `false` to disable truncation; when disabled and the model's maximum input length exceeds `--max-batch-tokens`, the server will refuse to start with an error instead of silently truncating sequences.

          Unused for gRPC servers

          [env: AUTO_TRUNCATE=]

      --default-prompt-name <DEFAULT_PROMPT_NAME>
          The name of the prompt that should be used by default for encoding. If not set, no prompt will be applied.

          Must be a key in the `sentence-transformers` configuration `prompts` dictionary.

          For example if ``default_prompt_name`` is "query" and the ``prompts`` is {"query": "query: ", ...}, then the sentence "What is the capital of France?" will be encoded as "query: What is the capital of France?" because the prompt text will be prepended before any text to encode.

          The argument '--default-prompt-name <DEFAULT_PROMPT_NAME>' cannot be used with '--default-prompt <DEFAULT_PROMPT>`

          [env: DEFAULT_PROMPT_NAME=]

      --default-prompt <DEFAULT_PROMPT>
          The prompt that should be used by default for encoding. If not set, no prompt will be applied.

          For example if ``default_prompt`` is "query: " then the sentence "What is the capital of France?" will be encoded as "query: What is the capital of France?" because the prompt text will be prepended before any text to encode.

          The argument '--default-prompt <DEFAULT_PROMPT>' cannot be used with '--default-prompt-name <DEFAULT_PROMPT_NAME>`

          [env: DEFAULT_PROMPT=]

      --dense-path <DENSE_PATH>
          Optionally, define the path to the Dense module required for some embe