Add skill6

Discover / LLM Ops & Observability

Promptfoo

by promptfooTypeScript

Test, evaluate, and red-team LLM prompts and apps from the CLI.

Toolexperimental

Maturity: experimental because latest release 0.121.20 is pre 1.0. Derived from release and commit history, not a rating.

Stars
24k
Forks
2.1k
Downloads / mo
—
Last commit
2026-08-03
License
MIT
Open issues
474

Market and trust evidence

Edition not yet matched

No exact skills.sh identity match is available for this repository. Repository adoption and freshness remain visible above; install momentum is not inferred.

Trust analysis is a screening signal, not a security warranty. Read the ranking and trust methodology.

In practice

Written by AI from this repository’s README · high confidence

Prompt and model changes are judged by trial and error with no repeatable metric or security check.

Use it when

Use it when you want repeatable prompt evals, side by side model comparison, or vulnerability scanning in CI.

Not the right pick when

It needs Node.js 20.20+ or 22.22+, and Node.js 20 support ends July 30, 2026 per the README.

Capabilities

  • automated evaluations of prompts and models
  • red teaming and vulnerability scanning with report generation
  • side by side model comparison across OpenAI, Anthropic, Bedrock and Ollama
  • automated checks in CI/CD pipelines
  • code scanning of pull requests for LLM security issues
  • local runs with live reload and caching

Requirements

  • Node.js ^20.20.0 or >=22.22.0 for npm and npx usage
  • an API key for most LLM providers, for example OPENAI_API_KEY

Cost: Free and open source

Install

Derived from the published package name in the repository, not from a model.

CompareRepository ↗Homepage ↗npm ↗

Video walkthroughs

Third-party YouTube uploads matched to this tool by title, channel and repository name on 2026-08-03. Not made, reviewed or endorsed by SkillPilot. View counts and publish months are as of the match date and the month is approximate. Nothing loads from YouTube until you press play.

What the repository ships

Ships CLAUDE.mdCursor rulesHas testsHas docsHas examplesDocker imageSecurity policyCI configured

Detected from the actual files in the repository root.

Latest release 0.121.20

Published 2026-07-31

0.121.20 (2026-07-30)

Features

Bug Fixes

  • assertions: default conversation-relevance threshold to 0.5 (#10142) (2812d76)
  • azure: avoid crash on empty completion choices (#10124) (17d8d03)
  • deps: align Renovate npm registry (#10115) (688a905)
  • deps: update anthropic packages (#10146) (cfa9bee)
  • deps: update anthropic packages (#10199) (2d3f3f1)
  • deps: update anthropic packages (#10229) (2834215)
  • deps: update dependency js-yaml to v5.2.2 [security] (#10218) (18ea879)
  • eval: avoid prompt index collisions for duplicate providers (#10208) (5aa0958)
  • eval: isolate conversation history by result column (#10212) (a377bb7)
  • integrations: migrate Langfuse prompts to SDK v5 (#10131) (08b3f50)
  • providers: align OpenAI models and capabilities (#10125) (e17ce87)
  • providers: preserve WebSocket retries for certificate-like URLs (#10197) (85e0710)
  • providers: reject Vertex Omni passthrough follow-ups (#10135) (8482570)
  • providers: restore secret-safe WebSocket retries (#10176) ([6d89cbb](https://github.com/promptfoo/promptfoo/commit/6d89cbbd609

Tags

Evalsevalstestingred-team

README

Promptfoo: LLM evals & red teaming

<p align="center">

<a href="https://npmjs.com/package/promptfoo"><img src="https://img.shields.io/npm/v/promptfoo" alt="npm"></a>

<a href="https://npmjs.com/package/promptfoo"><img src="https://img.shields.io/npm/dm/promptfoo" alt="npm"></a>

<a href="https://github.com/promptfoo/promptfoo/actions/workflows/main.yml"><img src="https://img.shields.io/github/actions/workflow/status/promptfoo/promptfoo/main.yml" alt="GitHub Workflow Status"></a>

<a href="https://github.com/promptfoo/promptfoo/blob/main/LICENSE"><img src="https://img.shields.io/github/license/promptfoo/promptfoo" alt="MIT license"></a>

<a href="https://discord.gg/promptfoo"><img src="https://img.shields.io/discord/1146610656779440188?logo=discord&label=promptfoo" alt="Discord"></a>

</p>

<p align="center">

<code>promptfoo</code> is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps.

</p>

<p align="center">

<a href="https://www.promptfoo.dev">Website</a> ·

<a href="https://www.promptfoo.dev/docs/getting-started/">Getting Started</a> ·

<a href="https://www.promptfoo.dev/docs/red-team/">Red Teaming</a> ·

<a href="https://www.promptfoo.dev/docs/">Documentation</a> ·

<a href="https://discord.gg/promptfoo">Discord</a>

</p>

Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed. Read the company update.

Quick Start

Requires Node.js ^20.20.0 or >=22.22.0 for npm and npx usage.

Node.js 20 support ends July 30, 2026 at 00:00 UTC; upgrade to Node.js 24 LTS before updating promptfoo at or after the cutoff.


npm install -g promptfoo
promptfoo init --example getting-started

Also available via brew install promptfoo and pip install promptfoo. You can also use npx promptfoo@latest to run any command without installing.

Most LLM providers require an API key. Set yours as an environment variable:


export OPENAI_API_KEY=sk-abc123

Once you're in the example directory, run an eval and view results:


cd getting-started
promptfoo eval
promptfoo view

See Getting Started (evals) or Red Teaming (vulnerability scanning) for more.

What can you do with Promptfoo?

  • Test your prompts and models with automated evaluations
  • Secure your LLM apps with red teaming and vulnerability scanning
  • Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
  • Automate checks in CI/CD
  • Review pull requests for LLM-related security and compliance issues with code scanning
  • Share results with your team

Here's what it looks like in action:

<img src="site/static/img/claude-vs-gpt-example@2x.png" alt="prompt evaluation matrix - web viewer" width="700">

It works on the command line too:

<img src="https://www.promptfoo.dev/img/docs/self-grading.gif" alt="promptfoo command line" width="700">

It also can generate security vulnerability reports:

<img src="https://www.promptfoo.dev/img/redteam-dashboard@2x.jpg" alt="gen ai red team" width="700">

Why Promptfoo?

  • Developer-first: Fast, with features like live reload and caching
  • Private: LLM evals run 100% locally - your prompts never leave your machine
  • Flexible: Works with any LLM API or programming language
  • Battle-tested: Powers LLM apps serving 10M+ users in production
  • Data-driven: Make decisions based on metrics, not gut feel
  • Open source: MIT licensed, with an active community

Learn More

Contributing

We welcome contributions! Check out our contributing guide to get started.

Join our Discord community for help and discussion.

<a href="https://github.com/promptfoo/promptfoo/graphs/contributors">

<img src="https://contrib.rocks/image?repo=promptfoo/promptfoo" />

</a>

Related tools

DeepEvalconfident-aiOpen source evaluation framework for unit testing large language model outputs.Opikcomet-mlOpen-source LLM evaluation, tracing, and monitoring by Comet.RagasexplodinggradientsFramework for evaluating retrieval augmented generation pipelines with reference free metrics.LM Evaluation HarnessEleutherAIUnified framework for evaluating language models across hundreds of benchmark tasks.TruLenstrueraLibrary for evaluating and tracking the performance of LLM applications with feedback functions.Prompt FlowmicrosoftToolkit for building, evaluating and deploying prompt based LLM application workflows.