Firecrawl

Firecrawl v2.11.0

Improvements

• Firecrawl Research Index — Added a specialized index for agentic AI/ML research: search across 3M+ arXiv papers and the GitHub code behind them (issues, merged PRs, and READMEs, refreshed daily), fetch a paper's details or related work, and check claims against full text. It has state-of-the-art recall on arXivQA, outperforming the next best provider by 18% at comparable cost. Available via the API, SDKs, MCP, and CLI.
• Keyless access for core endpoints — Use /scrape, /search, /interact, and /parse without an API key from official MCP, CLI, and SDK clients.
• Automatic PII redaction — Added a redactPII option that strips personal and sensitive data like names, emails, phone numbers, addresses, and secrets out of scraped content before it's returned.
• deterministicJson format — Added a format that returns structured JSON without running an LLM on every request. Firecrawl generates a reusable extractor for your schema and caches it per site, so repeat scrapes are cheaper and return consistent results.
• Video discovery on any page — Expanded the video format to find videos on any page, not just supported providers like YouTube, returning each video's URL, title, thumbnail, duration, and more.
• Attach your own browser automation — Added a CDP WebSocket URL (cdpUrl) to browser session responses, so you can drive a live Firecrawl browser session directly with Playwright, Puppeteer, or any other CDP client.
• Smarter monitor alerts — Added a goal to monitors so an LLM judges each detected change as meaningful or noise against what you actually care about, cutting alert spam and surfacing the changes that matter first in summary emails.
• Field-level JSON diffs for monitors — Monitors that scrape in JSON mode now compare the actual field values between runs instead of the rendered page, so you see exactly which fields changed rather than noise from layout shifts.
• Monitor email confirmation — Added an opt-in confirmation flow with one-click unsubscribe for external monitor recipients; team members are auto-confirmed, and Monitor responses now report each recipient's subscription status.
• AM/PM monitor schedules — Added support for 12-hour schedule inputs like daily at 9am and daily at 5:30pm, converted to the correct UTC cron expression.
• Monitor webhook delivery status — Added delivery status to each monitor check, so you can see whether its webhook was attempted, delivered, or failed — and why.
• Steadier monitor checks — Monitors now wait for pages to finish rendering before diffing, cutting false alerts caused by partially-loaded pages.
• PDF size cap — Raised the PDF download and scrape size cap from 30 MB to 50 MB.
• Python SDK crawl() scrape kwargs — Added direct scrape kwargs (formats, headers, include_tags, exclude_tags, etc.) to crawl() and start_crawl(), removing the need to wrap them in ScrapeOptions(...).
• Clearer .data errors — Improved the error raised when accessing .data on a search result to point at .web, .news, and .images with their counts, instead of returning a silent None.
• Python format defaults — Removed the required type= argument on JsonFormat and ChangeTrackingFormat, defaulting it like ScreenshotFormat.
• ChangeTrackingFormat casing — Added acceptance of both change_tracking and changeTracking for the format type so payloads round-trip between snake_case and camelCase clients.

Fixes

• Resolved security advisories across the API and SDKs by upgrading axios, esbuild, ws, openssl, and other dependencies.
• Fixed scrape workers stalling for tens of seconds on very large LLM-extractor inputs, which previously caused dropped jobs and worker restarts.
• Fixed Wikipedia scrapes missing metadata.ogImage on roughly half of Wikimedia URLs.
• Fixed crawl and batch cancellation not draining the

<h3 align="center">

<a name="readme-top"></a>

<img

src="https://raw.githubusercontent.com/firecrawl/firecrawl/main/img/firecrawl_logo.png"

height="200"

</h3>

<div align="center">

<a href="https://github.com/firecrawl/firecrawl/blob/main/LICENSE">

<img src="https://img.shields.io/github/license/firecrawl/firecrawl" alt="License">

</a>

<a href="https://pepy.tech/project/firecrawl-py">

<img src="https://static.pepy.tech/badge/firecrawl-py" alt="Downloads">

</a>

<a href="https://GitHub.com/firecrawl/firecrawl/graphs/contributors">

<img src="https://img.shields.io/github/contributors/firecrawl/firecrawl.svg" alt="GitHub Contributors">

</a>

<a href="https://firecrawl.dev">

<img src="https://img.shields.io/badge/Visit-firecrawl.dev-orange" alt="Visit firecrawl.dev">

</a>

</div>

<div>

<p align="center">

<a href="https://twitter.com/firecrawl">

<img src="https://img.shields.io/badge/Follow%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white" alt="Follow on X" />

</a>

<a href="https://www.linkedin.com/company/104100957">

<img src="https://img.shields.io/badge/Follow%20on%20LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white" alt="Follow on LinkedIn" />

</a>

<a href="https://discord.gg/firecrawl">

<img src="https://img.shields.io/badge/Join%20our%20Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join our Discord" />

</a>

</p>

</div>

---

🔥 Firecrawl

The API to search, scrape, and interact with the web at scale. 🔥 The web context API to find sources, extract content, and turn it into clean Markdown or structured data your agents can ship with. Open source and available as a hosted service.

_Pst. Hey, you, join our stargazers :)_

<a href="https://github.com/firecrawl/firecrawl">

<img src="https://img.shields.io/github/stars/firecrawl/firecrawl.svg?style=social&label=Star&maxAge=2592000" alt="GitHub stars">

</a>

---

Why Firecrawl?

• Industry-leading reliability: Covers 96% of the web, including JS-heavy pages — no proxy headaches, just clean data (see benchmarks)
• Blazingly fast: P95 latency of 3.4s across millions of pages, built for real-time agents and dynamic apps
• LLM-ready output: Clean markdown, structured JSON, screenshots, and more — spend fewer tokens, build better AI apps
• We handle the hard stuff: Rotating proxies, orchestration, rate limits, JS-blocked content, and more — zero configuration
• Agent ready: Connect Firecrawl to any AI agent or MCP client with a single command
• Media parsing: Parse and extract content from web-hosted PDFs, DOCX, and more
• Actions: Click, scroll, write, wait, and press before extracting content
• Open source: Developed transparently and collaboratively — join our community

---

Feature Overview

Core Endpoints

| Feature | Description |

|---------|-------------|

| Search | Search the web and get full page content from results |

| Scrape | Convert any URL to markdown, HTML, screenshots, or structured JSON |

| Interact | Scrape a page, then interact with it using AI prompts or code |

More

| Feature | Description |

|---------|-------------|

| Agent | Automated data gathering, just describe what you need |

| Crawl | Scrape all URLs of a website with a single request |

| Map | Discover all URLs on a website instantly |

| Batch Scrape | Scrape thousands of URLs asynchronously |

---

Quick Start

Sign up at firecrawl.dev to get your API key. Try the playground to test it out.

Search

Search the web and get full content from results.

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

search_result = app.search("firecrawl", limit=5)

<details>

<summary><b>Node.js / cURL / CLI</b></summary>

Node.js

import { Firecrawl } from 'firecrawl';

const app = new Firecrawl({apiKey: "fc-YOUR_API_KEY"});

app.search("firecrawl", { limit: 5 })

cURL

curl -X POST 'https://api.firecrawl.dev/v2/search' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
  "query": "firecrawl",
  "limit": 5
}'

CLI

firecrawl search "firecrawl" --limit 5

</details>

Output:

[
  {
    "url": "https://firecrawl.dev",
    "title": "Firecrawl",
    "markdown": "Turn websites into..."
  },
  {
    "url": "https://docs.firecrawl.dev",
    "title": "Firecrawl Docs",
    "markdown": "# Getting Started..."
  }
]

Scrape

Get LLM-ready data from any website — markdown, JSON, screenshots, and more.

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

result = app.scrape('firecrawl.dev')

<details>

<summary><b>Node.js / cURL / CLI</b></summary>

Node.js

import { Firecrawl } from 'firecrawl';

const app = new Firecrawl({ apiKey: "fc-YOUR_API_KEY" });

app.scrape('firecrawl.dev')

cURL

curl -X POST 'https://api.firecrawl.dev/v2/scrape' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
  "url": "firecrawl.dev"
}'

CLI

firecrawl scrape https://firecrawl.dev
firecrawl https://firecrawl.dev --only-main-content

</details>

Output:

# Firecrawl

Firecrawl helps AI agents search, scrape, and interact with the web.

## Features
- Search: Find information across the web
- Scrape: Clean data from any page
- Interact: Click, navigate, and operate pages
- Agent: Autonomous data gathering

Interact

Scrape a page, then interact with it using AI prompts or code.

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

result = app.scrape("https://amazon.com")
scrape_id = result.metadata.scrape_id

app.interact(scrape_id, prompt="Search for 'mechanical keyboard'")
app.interact(scrape_id, prompt="Click the first result")

<details>

<summary><b>Node.js / cURL / CLI</b></summary>

Node.js

import { Firecrawl } from 'firecrawl';

const app = new Firecrawl({apiKey: "fc-YOUR_API_KEY"});

const result = await app.scrape("https://amazon.com");

await app.interact(result.metadata.scrapeId, {
  prompt: "Search for 'mechanical keyboard'"
});
await app.interact(result.metadata.scrapeId, {
  prompt: "Click the first result"
});

cURL

# 1. Scrape the page
curl -X POST 'https://api.firecrawl.dev/v2/scrape' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"url": "https://amazon.com"}'

# 2. Interact with the page (use scrapeId from step 1)
curl -X POST 'https://api.firecrawl.dev/v2/scrape/SCRAPE_ID/interact' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"prompt": "Search for mechanical keyboard"}'

CLI

firecrawl scrape https://amazon.com
firecrawl interact exec --prompt "Search for 'mechanical keyboard'"
firecrawl interact exec --prompt "Click the first result"

</details>

Output:

{
  "success": true,
  "output": "Keyboard available at $100",
  "liveViewUrl": "https://liveview.firecrawl.dev/..."
}

---

Power Your Agent

Connect Firecrawl to any AI agent or MCP client in minutes.

Skill

Give your agent easy access to real-time web data with one command.

npx -y firecrawl-cli@latest init --all --browser

Restart your agent after installing. Works with Claude Code, Antigravity, OpenCode, and more.

MCP

Connect any MCP-compatible client to the web in seconds.

{
  "mcpServers": {
    "firecrawl-mcp": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "fc-YOUR_API_KEY"
      }
    }
  }
}

Agent Onboarding

Are you an AI agent? Fetch this skill to sign up your user, get an API key, and start building with Firecrawl.

curl -s https://firecrawl.dev/agent-onboarding/SKILL.md

See the Skill + CLI documentation for all available commands. For MCP, see firecrawl-mcp-server.

---

More Endpoints

Agent

The easiest way to get data from the web. Describe what you need, and our AI agent searches, navigates, and retrieves it. No URLs required.

Agent is the evolution of our /extract endpoint: faster, more reliable, and doesn't require you to know the URLs upfront.

curl -X POST 'https://api.firecrawl.dev/v2/agent' \
  -H 'Authorization: Bearer fc-YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "prompt": "Find the pricing plans for Notion"
  }'

Response:

{
  "success": true,
  "data": {
    "result": "Notion offers the following pricing plans:\n\n1. Free - $0/month...\n2. Plus - $10/seat/month...\n3. Business - $18/seat/month...",
    "sources": ["https://www.notion.so/pricing"]
  }
}

Agent with Structured Output

Use a schema to get structured data:

from firecrawl import Firecrawl
from pydantic import BaseModel, Field
from typing import List, Optional

app = Firecrawl(api_key="fc-YOUR_API_KEY")

class Founder(BaseModel):
    name: str = Field(description="Full name of the founder")
    role: Optional[str] = Field(None, description="Role or position")

class FoundersSchema(BaseModel):
    founders: List[Founder] = Field(description="List of founders")

result = app.agent(
    prompt="Find the founders of Firecrawl",
    schema=FoundersSchema
)

print(result.data)

{
  "founders": [
    {"name": "Eric Ciarla", "role": "Co-founder"},
    {"name": "Nicolas Camara", "role": "Co-founder"},
    {"name": "Caleb Peffer", "role": "Co-founder"}
  ]
}

Agent with URLs (Optional)

Focus the agent on specific pages:

result = app.agent(
    urls=["https://docs.firecrawl.dev", "https://firecrawl.dev/pricing"],
    prompt="Compare the features and pricing information"
)

Model Selection

Choose between two models based on your needs:

| Model | Cost | Best For |

|-------|------|----------|

| spark-1-mini (default) | 60% cheaper | Most tasks |

| spark-1-pro | Standard | Complex research, critical data gathering |

result = app.agent(
    prompt="Compare enterprise features across Firecrawl, Apify, and ScrapingBee",
    model="spark-1-pro"
)

When to use Pro:

• Comparing data across multiple websites
• Extracting from sites with complex navigation or auth
• Research tasks where the agent needs to explore multiple paths
• Critical data where accuracy is paramount

Learn more about Spark models in our Agent documentation.

Crawl

Crawl an entire website and get content from all pages.

curl -X POST 'https://api.firecrawl.dev/v2/crawl' \
  -H 'Authorization: Bearer fc-YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://docs.firecrawl.dev",
    "limit": 100,
    "scrapeOptions": {
      "formats": ["markdown"]
    }
  }'

Returns a job ID:

{
  "success": true,
  "id": "123-456-789",
  "url": "https://api.firecrawl.dev/v2/crawl/123-456-789"
}

Check Crawl Status

curl -X GET 'https://api.firecrawl.dev/v2/crawl/123-456-789' \
  -H 'Authorization: Bearer fc-YOUR_API_KEY'

{
  "status": "completed",
  "total": 50,
  "completed": 50,
  "creditsUsed": 50,
  "data": [
    {
      "markdown": "# Page Title\n\nContent...",
      "metadata": {"title": "Page Title", "sourceURL": "https://..."}
    }
  ]
}

Note: The SDKs handle polling automatically for a better developer experience.

Map

Discover all URLs on a website instantly.

curl -X POST 'https://api.firecrawl.dev/v2/map' \
  -H 'Authorization: Bearer fc-YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://firecrawl.dev"}'

Response:

{
  "success": true,
  "links": [
    {"url": "https://firecrawl.dev", "title": "Firecrawl", "description": "Turn websites into LLM-ready data"},
    {"url": "https://firecrawl.dev/pricing", "title": "Pricing", "description": "Firecrawl pricing plans"},
    {"url": "https://firecrawl.dev/blog", "title": "Blog", "description": "Firecrawl blog"}
  ]
}

Map with Search

Find specific URLs within a site:

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

result = app.map("https://firecrawl.dev", search="pricing")
# Returns URLs ordered by relevance to "pricing"

Batch Scrape

Scrape multiple URLs at once:

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

job = app.batch_scrape([
    "https://firecrawl.dev",
    "https://docs.firecrawl.dev",
    "https://firecrawl.dev/pricing"
], formats=["markdown"])

for doc in job.data:
    print(doc.metadata.source_url)

---

SDKs

Our SDKs provide a convenient way to use all Firecrawl features and automatically handle polling for async operations.

Python

Install the SDK:

pip install firecrawl-py

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

# Scrape a single URL
doc = app.scrape("https://firecrawl.dev", formats=["markdown"])
print(doc.markdown)

# Use the Agent for autonomous data gathering
result = app.agent(prompt="Find the founders of Stripe")
print(result.data)

# Crawl a website (automatically waits for completion)
docs = app.crawl("https://docs.firecrawl.dev", limit=50)
for doc in docs.data:
    print(doc.metadata.source_url, doc.markdown[:100])

# Search the web
results =