Evidently

What's Changed

• feat: enable logo display in API documentation generation by @DimaAmega in https://github.com/evidentlyai/evidently/pull/1793
• fix: Add pagination to reports table to fix UI performance by @jasperbruin in https://github.com/evidentlyai/evidently/pull/1797
• UI: snapshots pagination bit changes by @DimaAmega in https://github.com/evidentlyai/evidently/pull/1804
• Pandas 3 support by @Liraim in https://github.com/evidentlyai/evidently/pull/1807
• CI: deploy artifacts better by @DimaAmega in https://github.com/evidentlyai/evidently/pull/1812
• fix: correct output prefix syntax in GitHub Actions workflow by @DimaAmega in https://github.com/evidentlyai/evidently/pull/1814
• ci: prevent deployment on cancelled workflow runs in GitHub Actions by @DimaAmega in https://github.com/evidentlyai/evidently/pull/1815
• fix: correct HTML report path in GitHub Actions workflow by @DimaAmega in https://github.com/evidentlyai/evidently/pull/1817
• CI: explicit download artifacts by @DimaAmega in https://github.com/evidentlyai/evidently/pull/1822
• Drop python 3.9 support by @mike0sv in https://github.com/evidentlyai/evidently/pull/1811
• CI: remove redundant test by @DimaAmega in https://github.com/evidentlyai/evidently/pull/1821
• Enhance README with API Reference and documentation update by @DimaAmega in https://github.com/evidentlyai/evidently/pull/1819
• Update API reference documentation in README.md for clarity and structure by @DimaAmega in https://github.com/evidentlyai/evidently/pull/1823
• Implement Embeddings drift metric in new API by @Liraim in https://github.com/evidentlyai/evidently/pull/1828
• Support nbinsx for ValueDrift metric. by @Liraim in https://github.com/evidentlyai/evidently/pull/1826
• Fail early on empty input data in Report.run by @Si-ra-kri in https://github.com/evidentlyai/evidently/pull/1820
• Add methods to handle near-zero values for MAPE metric. by @Liraim in https://github.com/evidentlyai/evidently/pull/1835
• Fix MAPE warnings by @Liraim in https://github.com/evidentlyai/evidently/pull/1837
• Examples Cleanup by @emeli-dral in https://github.com/evidentlyai/evidently/pull/1836
• Fix CategoryCount with multiple labels if some labels are missing by @Liraim in https://github.com/evidentlyai/evidently/pull/1838

New Contributors

• @Si-ra-kri made their first contribution in https://github.com/evidentlyai/evidently/pull/1820

Full Changelog: https://github.com/evidentlyai/evidently/compare/v0.7.20...v0.7.21

<h1 align="center">Evidently</h1>

<p align="center"><b>An open-source framework to evaluate, test and monitor ML and LLM-powered systems.</b></p>

<p align="center">

<a href="https://pepy.tech/project/evidently" target="_blank"><img src="https://pepy.tech/badge/evidently" alt="PyPi Downloads"></a>

<a href="https://github.com/evidentlyai/evidently/blob/main/LICENSE" target="_blank"><img src="https://img.shields.io/github/license/evidentlyai/evidently" alt="License"></a>

<a href="https://pypi.org/project/evidently/" target="_blank"><img src="https://img.shields.io/pypi/v/evidently" alt="PyPi"></a>

Evidently

</p>

<p align="center">

<a href="https://docs.evidentlyai.com">Documentation</a>

|

<a href="https://evidentlyai.github.io/evidently/api-reference">API Reference</a>

|

<a href="https://discord.gg/xZjKRaNp8b">Discord Community</a>

|

<a href="https://evidentlyai.com/blog">Blog</a>

|

<a href="https://twitter.com/EvidentlyAI">Twitter</a>

|

<a href="https://www.evidentlyai.com/register">Evidently Cloud</a>

</p>

:bar_chart: What is Evidently?

Evidently is an open-source Python library to evaluate, test, and monitor ML and LLM systems—from experiments to production.

• 🔡 Works with tabular and text data.
• ✨ Supports evals for predictive and generative tasks, from classification to RAG.
• 📚 100+ built-in metrics from data drift detection to LLM judges.
• 🛠️ Python interface for custom metrics.
• 🚦 Both offline evals and live monitoring.
• 💻 Open architecture: easily export data and integrate with existing tools.

Evidently is very modular. You can start with one-off evaluations or host a full monitoring service.

1. Reports and Test Suites

Reports compute and summarize various data, ML and LLM quality evals.

• Start with Presets and built-in metrics or customize.
• Best for experiments, exploratory analysis and debugging.
• View interactive Reports in Python or export as JSON, Python dictionary, HTML, or view in monitoring UI.

Turn any Report into a Test Suite by adding pass/fail conditions.

• Best for regression testing, CI/CD checks, or data validation.
• Zero setup option: auto-generate test conditions from the reference dataset.
• Simple syntax to set test conditions as gt (greater than), lt (less than), etc.

| Reports |

|--|

|Report example|

2. Monitoring Dashboard

Monitoring UI service helps visualize metrics and test results over time.

You can choose:

• Self-host the open-source version. Live demo.
• Sign up for Evidently Cloud (Recommended).

Evidently Cloud offers a generous free tier and extra features like dataset and user management, alerting, and no-code evals. Compare OSS vs Cloud.

| Dashboard |

|--|

|Dashboard example|

:woman_technologist: Install Evidently

To install from PyPI:

pip install evidently

To install Evidently using the Conda installer, run:

conda install -c conda-forge evidently

:arrow_forward: Getting started

Reports

LLM evals

This is a simple Hello World. Check the Tutorials for more: LLM evaluation.

Import the necessary components:

import pandas as pd
from evidently import Report
from evidently import Dataset, DataDefinition
from evidently.descriptors import Sentiment, TextLength, Contains
from evidently.presets import TextEvals

Create a toy dataset with questions and answers.

eval_df = pd.DataFrame([
    ["What is the capital of Japan?", "The capital of Japan is Tokyo."],
    ["Who painted the Mona Lisa?", "Leonardo da Vinci."],
    ["Can you write an essay?", "I'm sorry, but I can't assist with homework."]],
                       columns=["question", "answer"])

Create an Evidently Dataset object and add descriptors: row-level evaluators. We'll check for sentiment of each response, its length and whether it contains words indicative of denial.

eval_dataset = Dataset.from_pandas(pd.DataFrame(eval_df),
data_definition=DataDefinition(),
descriptors=[
    Sentiment("answer", alias="Sentiment"),
    TextLength("answer", alias="Length"),
    Contains("answer", items=['sorry', 'apologize'], mode="any", alias="Denials")
])

You can view the dataframe with added scores:

eval_dataset.as_dataframe()

To get a summary Report to see the distribution of scores:

report = Report([
    TextEvals()
])

my_eval = report.run(eval_dataset)
my_eval
# my_eval.json()
# my_eval.dict()

You can also choose other evaluators, including LLM-as-a-judge and configure pass/fail conditions.

Data and ML evals

This is a simple Hello World. Check the Tutorials for more: Tabular data.

Import the Report, evaluation Preset and toy tabular dataset.

import pandas as pd
from sklearn import datasets

from evidently import Report
from evidently.presets import DataDriftPreset

iris_data = datasets.load_iris(as_frame=True)
iris_frame = iris_data.frame

Run the Data Drift evaluation preset that will test for shift in column distributions. Take the first 60 rows of the dataframe as "current" data and the following as reference. Get the output in Jupyter notebook:

report = Report([
    DataDriftPreset(method="psi")
],
include_tests="True")
my_eval = report.run(iris_frame.iloc[:60], iris_frame.iloc[60:])
my_eval

You can also save an HTML file. You'll need to open it from the destination folder.

my_eval.save_html("file.html")

To get the output as JSON or Python dictionary:

my_eval.json()
# my_eval.dict()

You can choose other Presets, create Reports from individual Metrics and configure pass/fail conditions.

Monitoring dashboard

This launches a demo project in the locally hosted Evidently UI. Sign up for Evidently Cloud to instantly get a managed version with additional features.

if you have uv you can run Evidently UI with a single command.

uv run --with evidently evidently ui --demo-projects all

If you haven’t installed uv, create a virtual environment using the standard approach.

pip install virtualenv
virtualenv venv
source venv/bin/activate

After installing Evidently (pip install evidently), run the Evidently UI with the demo projects:

evidently ui --demo-projects all

Visit localhost:8000 to access the UI.

🚦 What can you evaluate?

Evidently has 100+ built-in evals. You can also add custom ones.

Here are examples of things you can check:

| | |

|:-------------------------:|:------------------------:|

| 🔡 Text descriptors | 📝 LLM outputs |

| Length, sentiment, toxicity, language, special symbols, regular expression matches, etc. | Semantic similarity, retrieval relevance, summarization quality, etc. with model- and LLM-based evals. |

| 🛢 Data quality | 📊 Data distribution drift |

| Missing values, duplicates, min-max ranges, new categorical values, correlations, etc. | 20+ statistical tests and distance metrics to compare shifts in data distribution. |

| 🎯 Classification | 📈 Regression |

| Accuracy, precision, recall, ROC AUC, confusion matrix, bias, etc. | MAE, ME, RMSE, error distribution, error normality, error bias, etc. |

| 🗂 Ranking (inc. RAG) | 🛒 Recommendations |

| NDCG, MAP, MRR, Hit Rate, etc. | Serendipity, novelty, diversity, popularity bias, etc. |

:computer: Contributions

We welcome contributions! Read the Guide to learn more.

:books: Documentation

For more examples, refer to the complete <a href="https://docs.evidentlyai.com">Documentation</a>.

Browse the <a href="https://evidentlyai.github.io/evidently/api-reference">API Reference</a> for detailed API documentation.

:white_check_mark: Discord Community

If you want to chat and connect, join our Discord community!