DeepEval

New to deepeval? Get started here.

✨ New Feature

• add flaky flag to metrics and test cases (#2951 by @penguine-ip)
• Read the docs here: https://deepeval.com/docs/evaluation-test-cases#mark-test-cases-as-flaky

🐛 Bug Fix

• system instruction parsing (#2941 by @tanayvaswani)
• set additionalProperties recursively for gateway schemas (#2930 by @eno42)

New contributors: @eno42

A huge thank you to everyone who contributed to this release ❤️

<p align="center">

<picture>

<source media="(prefers-color-scheme: dark)" srcset="assets/hero/wordmark-dark.svg">

<img alt="DeepEval." src="assets/hero/wordmark-light.svg" width="520">

</picture>

</p>

<p align="center">

<h1 align="center">The LLM Evaluation Framework</h1>

</p>

<p align="center">

<a href="https://trendshift.io/repositories/5917" target="_blank"><img src="https://trendshift.io/api/badge/repositories/5917" alt="confident-ai%2Fdeepeval | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>

</p>

<p align="center">

<a href="https://discord.gg/3SEyvpgu2f">

<img alt="discord-invite" src="https://dcbadge.limes.pink/api/server/3SEyvpgu2f?style=flat">

</a>

<a href="https://www.reddit.com/r/deepeval/">

<img alt="reddit-community" src="https://img.shields.io/badge/Reddit-r%2Fdeepeval-FF4500?logo=reddit&logoColor=white">

</a>

</p>

<h4 align="center">

<p>

<a href="https://deepeval.com/docs/getting-started?utm_source=GitHub">Documentation</a> |

<a href="#-metrics-and-features">Metrics and Features</a> |

<a href="#-quickstart">Getting Started</a> |

<a href="#-integrations">Integrations</a> |

<a href="https://www.confident-ai.com?utm_source=deepeval&utm_medium=github&utm_content=header_nav">Confident AI</a>

<p>

</h4>

<p align="center">

<a href="https://github.com/confident-ai/deepeval/releases">

<img alt="GitHub release" src="https://img.shields.io/github/release/confident-ai/deepeval.svg?color=violet">

</a>

<a href="https://colab.research.google.com/drive/1PPxYEBa6eu__LquGoFFJZkhYgWVYE6kh?usp=sharing">

<img alt="Try Quickstart in Colab" src="https://colab.research.google.com/assets/colab-badge.svg">

</a>

<a href="https://github.com/confident-ai/deepeval/blob/master/LICENSE.md">

<img alt="License" src="https://img.shields.io/github/license/confident-ai/deepeval.svg?color=yellow">

</a>

<a href="https://x.com/deepeval">

<img alt="Twitter Follow" src="https://img.shields.io/twitter/follow/deepeval?style=social&logo=x">

</a>

</p>

<p align="center">

<!-- Keep these links. Translations will automatically update with the README. -->

<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=de">Deutsch</a> |

<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=es">Español</a> |

<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=fr">français</a> |

<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=ja">日本語</a> |

<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=ko">한국어</a> |

<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=pt">Português</a> |

<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=ru">Русский</a> |

<a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=zh">中文</a>

</p>

DeepEval is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps. DeepEval incorporates the latest research to run evals via metrics such as G-Eval, task completion, answer relevancy, hallucination, etc., which uses LLM-as-a-judge and other NLP models that run locally on your machine.

Whether you're building AI agents, RAG pipelines, or chatbots, implemented via LangChain or OpenAI, DeepEval has you covered. With it, you can easily determine the optimal models, prompts, and architecture to improve your AI quality, prevent prompt drifting, or even transition from OpenAI to Claude with confidence.

[!IMPORTANT]

Need a place for your DeepEval testing data to live 🏡❤️? Sign up to Confident AI to compare iterations of your LLM app, generate & share testing reports, and more.

Demo GIF

Want to talk LLM evaluation, need help picking metrics, or just to say hi? Come join our discord.

<br />

🔥 Metrics and Features

• 📐 Large variety of ready-to-use LLM eval metrics (all with explanations) powered by ANY LLM of your choice, statistical methods, or NLP models that run locally on your machine covering all use cases:

• Custom, All-Purpose Metrics:

• G-Eval — a research-backed LLM-as-a-judge metric for evaluating on any custom criteria with human-like accuracy
• DAG — DeepEval's graph-based deterministic LLM-as-a-judge metric builder

• <details>

<summary><b>Agentic Metrics</b></summary>

• Task Completion — evaluate whether an agent accomplished its goal
• Tool Correctness — check if the right tools were called with the right arguments
• Goal Accuracy — measure how accurately the agent achieved the intended goal
• Step Efficiency — evaluate whether the agent took unnecessary steps
• Plan Adherence — check if the agent followed the expected plan
• Plan Quality — evaluate the quality of the agent's plan
• Tool Use — measure quality of tool usage
• Argument Correctness — validate tool call arguments

</details>

• <details>

<summary><b>RAG Metrics</b></summary>

• Answer Relevancy — measure how relevant the RAG pipeline's output is to the input
• Faithfulness — evaluate whether the RAG pipeline's output factually aligns with the retrieval context
• Contextual Recall — measure how well the RAG pipeline's retrieval context aligns with the expected output
• Contextual Precision — evaluate whether relevant nodes in the RAG pipeline's retrieval context are ranked higher
• Contextual Relevancy — measure the overall relevance of the RAG pipeline's retrieval context to the input
• RAGAS — average of answer relevancy, faithfulness, contextual precision, and contextual recall

</details>

• <details>

<summary><b>Multi-Turn Metrics</b></summary>

• Knowledge Retention — evaluate whether the chatbot retains factual information throughout a conversation
• Conversation Completeness — measure whether the chatbot satisfies user needs throughout a conversation
• Turn Relevancy — evaluate whether the chatbot generates consistently relevant responses throughout a conversation
• Turn Faithfulness — check if the chatbot's responses are factually grounded in retrieval context across turns
• Role Adherence — evaluate whether the chatbot adheres to its assigned role throughout a conversation

</details>

• <details>

<summary><b>MCP Metrics</b></summary>

• MCP Task Completion — evaluate how effectively an MCP-based agent accomplishes a task
• MCP Use — measure how effectively an agent uses its available MCP servers
• Multi-Turn MCP Use — evaluate MCP server usage across conversation turns

</details>

• <details>

<summary><b>Multimodal Metrics</b></summary>

• Text to Image — evaluate image generation quality based on semantic consistency and perceptual quality
• Image Editing — evaluate image editing quality based on semantic consistency and perceptual quality
• Image Coherence — measure how well images align with their accompanying text
• Image Helpfulness — evaluate how effectively images contribute to user comprehension of the text
• Image Reference — evaluate how accurately images are referred to or explained by accompanying text

</details>

• <details>

<summary><b>Other Metrics</b></summary>

• Hallucination — check whether the LLM generates factually correct information against provided context
• Summarization — evaluate whether summaries are factually correct and include necessary details
• Bias — detect gender, racial, or political bias in LLM outputs
• Toxicity — evaluate toxicity in LLM outputs
• JSON Correctness — check whether the output matches an expected JSON schema
• Prompt Alignment — measure whether the output aligns with instructions in the prompt template

</details>

• 🎯 Supports both end-to-end and component-level LLM evaluation.
• 🧩 Build your own custom metrics that are automatically integrated with DeepEval's ecosystem.
• 🔮 Generate both single and multi-turn synthetic datasets for evaluation.
• 🔗 Integrates seamlessly with ANY CI/CD environment.
• 🧬 Optimize prompts automatically based on evaluation results.
• 🏆 Easily benchmark ANY LLM on popular LLM benchmarks in under 10 lines of code., including MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, GSM8K.

<br />

🔌 Integrations

DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and more. To scale evals across your team — or let anyone run them without writing code — Confident AI gives you a native platform integration.

Frameworks

• OpenAI — evaluate and trace OpenAI applications via a client wrapper
• OpenAI Agents — evaluate OpenAI Agents end-to-end in under a minute
• LangChain — evaluate LangChain applications with a callback handler
• LangGraph — evaluate LangGraph agents with a callback handler
• Pydantic AI — evaluate Pydantic AI agents with type-safe validation
• CrewAI — evaluate CrewAI multi-agent systems
• Anthropic — evaluate and trace Claude applications via a client wrapper
• AWS AgentCore — evaluate agents deployed on Amazon AgentCore
• LlamaIndex — evaluate RAG applications built with LlamaIndex

☁️ Platform + Ecosystem

Confident AI is an all-in-one platform that integrates natively with DeepEval.

• Manage datasets, trace LLM applications, run evaluations, and monitor responses in production — all from one platform.
• Don't need a UI? Confident AI can also be your data persistent layer - run evals, pull datasets, and inspect traces straight from claude code, cursor, via Confident AI's MCP server.

<p align="center">

<img src="assets/confident-mcp-architecture.png" alt="Confident AI MCP Architecture" width="500">

</p>

<br />

🤖 Vibe-Coder QuickStart

Want your coding agent to add evals and fix failures for you? Install the DeepEval skill, point it at your agent, RAG pipeline, or chatbot, and ask it to generate a dataset, write the eval suite, run deepeval test run, and iterate on the failing metrics.

Start with the 5-minute vibe-coder guide.

<br />

🚀 Human QuickStart

Let's pretend your LLM application is a RAG based customer support chatbot; here's how DeepEval can help test what you've built.

Installation

Deepeval works with Python>=3.9+.

pip install -U deepeval

Create an account (highly recommended)

Using the deepeval platform will allow you to generate sharable testing reports on the cloud. It is free, takes no additional code to setup, and we highly recommend giving it a try.

To login, run:

deepeval login

Follow the instructions in the CLI to create an account, copy your API key, and paste it into the CLI. All test cases will automatically be logged (find more information on data privacy [here](https://deepeval.com/docs/d