Sentence Transformers

v5.6.1 - Flash Attention Fix for XLM-R and RoBERTa Models

This patch release fixes silently degraded embeddings for RoBERTa-family models when flash attention is requested with transformers v5, notably every XLM-R based multilingual embedding model (BAAI/bge-m3, intfloat/multilingual-e5-large, etc.). The bug affected v5.5.0, v5.5.1, and v5.6.0.

Install this version with

# Training + Inference
pip install sentence-transformers[train]==5.6.1

# Inference only, use one of:
pip install sentence-transformers==5.6.1
pip install sentence-transformers[onnx-gpu]==5.6.1
pip install sentence-transformers[onnx]==5.6.1
pip install sentence-transformers[openvino]==5.6.1

# Multimodal dependencies (optional):
pip install sentence-transformers[image]==5.6.1
pip install sentence-transformers[audio]==5.6.1
pip install sentence-transformers[video]==5.6.1

# Or combine as needed:
pip install sentence-transformers[train,onnx,image]==5.6.1

Fix position_ids offsetting for RoBERTa-family models when flattening inputs (#3879)

Since v5.5.0, the Transformer module flattens text-only batches into one packed sequence when flash attention is requested, skipping all padding overhead for a notable performance improvement. The position_ids of that packed sequence restart at 0 for every text, which is correct for the vast majority of models. RoBERTa-family architectures however compute positions as padding_idx + 1 + n for the n-th token, so every token read a position embedding shifted by padding_idx + 1 (usually 2). Nothing crashes, the embeddings are just silently worse.

from sentence_transformers import SentenceTransformer

# An affected configuration: flash attention with an XLM-R based model
model = SentenceTransformer(
    "BAAI/bge-m3",
    model_kwargs={"attn_implementation": "flash_attention_2"},
)

Measured on BAAI/bge-m3:

| Evaluation | padded | packed, 0-based positions | packed, with this fix |

|---|---|---|---|

| stsb test Spearman | 0.8485 | 0.7239 | 0.8485 |

| NanoBEIR mean nDCG@10 | 0.6041 | 0.5414 | 0.6050 |

The quality loss recovers exactly once the offset is applied. The fix scans the loaded model's modules once for an int padding_idx stored next to a learned position_embeddings table, and offsets the packed position_ids when that pair is found. An audit of transformers finds 16 architectures with that pair (roberta, xlm_roberta, xlm_roberta_xl, camembert, roberta_prelayernorm, xmod, data2vec_text, longformer, luke, ibert, mpnet, markuplm, lilt, layoutlmv3, esm, and pp_doclayout_v2), all offset by exactly padding_idx + 1, and no 0-based or rotary architecture matches.

You are only affected if you encoded text with flash attention requested on transformers v5 with a RoBERTa-family checkpoint. The default padded path (e.g. sdpa) was never affected, and neither were MPNet models like all-mpnet-base-v2 despite mpnet appearing in the audit: transformers does not support flash attention for MPNet at all. If you did index a corpus with such a configuration, re-encode it after upgrading: pre-fix embeddings score notably worse and do not mix with post-fix embeddings.

What's Changed

• [ci] Exclude librosa/numba/llvmlite on Python 3.13 by @tomaarsen in https://github.com/huggingface/sentence-transformers/pull/3835
• [tests] Skip bf16 + Windows + CPU forwards, as they can WindowsError on torch 2.13 by @tomaarsen in https://github.com/huggingface/sentence-transformers/pull/3863
• 🚨 Fix position_ids offsetting for RoBERTa-family models when flattening inputs by @tomaarsen in https://github.com/huggingface/sentence-transformers/pull/3879

Full Changelog: https://github.com/huggingface/sentence-transformers/compare/v5.6.0...v5.6.1

<!--- BADGES: START --->

HF Models

[GitHub - License][#github-license]

[PyPI - Python Version][#pypi-package]

[PyPI - Package Version][#pypi-package]

[Docs - GitHub.io][#docs-package]

<!-- [PyPI - Downloads][#pypi-package] -->

<!--- BADGES: END --->

Sentence Transformers: Embeddings, Retrieval, and Reranking

This framework provides an easy method to compute embeddings for accessing, using, and training state-of-the-art embedding and reranker models. It can be used to compute embeddings using Sentence Transformer models (quickstart), to calculate similarity scores using Cross-Encoder (a.k.a. reranker) models (quickstart) or to generate sparse embeddings using Sparse Encoder models (quickstart). This unlocks a wide range of applications, including semantic search, semantic textual similarity, and paraphrase mining.

A wide selection of over 15,000 pre-trained Sentence Transformers models are available for immediate use on 🤗 Hugging Face, including many of the state-of-the-art models from the Massive Text Embeddings Benchmark (MTEB) leaderboard. Additionally, it is easy to train or finetune your own embedding models, reranker models or sparse encoder models using Sentence Transformers, enabling you to create custom models for your specific use cases.

For the full documentation, see www.SBERT.net.

Installation

We recommend Python 3.10+, PyTorch 1.11.0+, and transformers v4.41.0+.

pip install -U sentence-transformers

See Installation in the docs for uv, conda, source, and editable installs, CUDA setup, and extras ([image], [audio], [video], [train], [onnx], [openvino], [dev]).

Getting Started

See Quickstart in our documentation.

Embedding Models

First download a pretrained embedding a.k.a. Sentence Transformer model.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

Then provide some texts to the model.

sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# => (3, 384)

And that's already it. We now have numpy arrays with the embeddings, one for each text. We can use these to compute similarities.

similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6660, 0.1046],
#         [0.6660, 1.0000, 0.1411],
#         [0.1046, 0.1411, 1.0000]])

Reranker Models

First download a pretrained reranker a.k.a. Cross Encoder model.

from sentence_transformers import CrossEncoder

# 1. Load a pretrained CrossEncoder model
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

Then provide some texts to the model.

# The texts for which to predict similarity scores
query = "How many people live in Berlin?"
passages = [
    "Berlin had a population of 3,520,031 registered inhabitants in an area of 891.82 square kilometers.",
    "Berlin has a yearly total of about 135 million day visitors, making it one of the most-visited cities in the European Union.",
    "In 2013 around 600,000 Berliners were registered in one of the more than 2,300 sport and fitness clubs.",
]

# 2a. predict scores for pairs of texts
scores = model.predict([(query, passage) for passage in passages])
print(scores)
# => [8.607139 5.506266 6.352977]

And we're good to go. You can also use model.rank to avoid having to perform the reranking manually:

# 2b. Rank a list of passages for a query
ranks = model.rank(query, passages, return_documents=True)

print("Query:", query)
for rank in ranks:
    print(f"- #{rank['corpus_id']} ({rank['score']:.2f}): {rank['text']}")
"""
Query: How many people live in Berlin?
- #0 (8.61): Berlin had a population of 3,520,031 registered inhabitants in an area of 891.82 square kilometers.
- #2 (6.35): In 2013 around 600,000 Berliners were registered in one of the more than 2,300 sport and fitness clubs.
- #1 (5.51): Berlin has a yearly total of about 135 million day visitors, making it one of the most-visited cities in the European Union.
"""

Sparse Encoder Models

First download a pretrained sparse embedding a.k.a. Sparse Encoder model.

from sentence_transformers import SparseEncoder

# 1. Load a pretrained SparseEncoder model
model = SparseEncoder("naver/splade-cocondenser-ensembledistil")

# The sentences to encode
sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium.",
]

# 2. Calculate sparse embeddings by calling model.encode()
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 30522] - sparse representation with vocabulary size dimensions

# 3. Calculate the embedding similarities
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[   35.629,     9.154,     0.098],
#         [    9.154,    27.478,     0.019],
#         [    0.098,     0.019,    29.553]])

# 4. Check sparsity stats
stats = SparseEncoder.sparsity(embeddings)
print(f"Sparsity: {stats['sparsity_ratio']:.2%}")
# Sparsity: 99.84%

Pre-Trained Models

We provide a large list of pretrained models for more than 100 languages. Some models are general purpose models, while others produce embeddings for specific use cases.

• Pretrained Sentence Transformer (Embedding) Models
• Pretrained Cross Encoder (Reranker) Models
• Pretrained Sparse Encoder (Sparse Embeddings) Models

Training

Tip: Using an AI coding agent (Claude Code, Codex, Cursor, Gemini CLI, ...)? Install the train-sentence-transformers Hugging Face Agent Skill via hf skills add train-sentence-transformers [--claude] [--global] and ask your agent to fine-tune a model on your data.

This framework allows you to fine-tune your own sentence embedding methods, so that you get task-specific sentence embeddings. You have various options to choose from in order to get perfect sentence embeddings for your specific task.

• Embedding Models
• Sentence Transformer > Training Overview
• Sentence Transformer > Training Examples or training examples on GitHub.
• Reranker Models
• Cross Encoder > Training Overview
• Cross Encoder > Training Examples or training examples on GitHub.
• Sparse Embedding Models
• Sparse Encoder > Training Overview
• Sparse Encoder > Training Examples or training examples on GitHub.

Some highlights across the different types of training are:

• Support of various transformer networks including BERT, RoBERTa, XLM-R, DistilBERT, Electra, BART, ...
• Multilingual and multi-task learning
• Evaluation during training to find optimal model
• 20+ loss functions for embedding models, 10+ loss functions for reranker models and 10+ loss functions for sparse embedding models, allowing you to tune models specifically for semantic search, paraphrase mining, semantic similarity comparison, clustering, triplet loss, contrastive loss, etc.

Companion Blog Posts

The following Hugging Face blog posts complement this documentation with narrative walkthroughs and full training examples:

Training guides:

• Training and Finetuning Embedding Models: end-to-end training of bi-encoder embedding models.
• Training and Finetuning Reranker Models: training Cross Encoder models for the second stage of retrieve-and-rerank pipelines.
• Training and Finetuning Sparse Embedding Models: training SPLADE and other sparse encoders.

Multimodal:

• Multimodal Embedding & Reranker Models: using text, image, audio, and video models through a single API.
• Training and Finetuning Multimodal Embedding & Reranker Models: training multimodal models, with a Visual Document Retrieval walkthrough.

Efficiency techniques:

• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Train 400x faster Static Embedding Models: CPU-friendly embedding models without attention.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.

Application Examples

You can use this framework for:

• Computing Sentence Embeddings

• Dense Embeddings
• Sparse Embeddings

• Semantic Textual Similarity

• Dense STS
• Sparse STS

• Semantic Search

• Dense Search
• Sparse Search

• Retrieve & Re-Rank

• Dense only Retrieval
• Sparse/Dense/Hybrid Retrieval

• Clustering

• Paraphrase Mining

• Translated Sentence Mining

• Multilingual Image Search, Clustering & Duplicate Detection

and many more use-cases.

For all examples, see examples/sentence_transformer/applications.

Development setup

After cloning the repo (or a fork) to your machine, in a virtual environment, run:

python -m pip install -e ".[dev]"

pre-commit install

To test your changes, run:

pytest

Citing & Authors

If you find this repository helpful, feel free to cite our publication Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks:

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",