Firecrawl Debug Bundle

Current State

!node --version 2>/dev/null || echo 'N/A'

!npm list @mendable/firecrawl-js 2>/dev/null | grep firecrawl || echo 'SDK not installed'

Overview

Collect all diagnostic information needed for Firecrawl support tickets. Tests API connectivity, checks SDK version, verifies credentials, captures error context, and packages it all into a redacted bundle.

Prerequisites

• Firecrawl SDK installed
• FIRECRAWLAPIKEY environment variable set
• Access to application logs

Instructions

Step 1: Create Debug Bundle Script

#!/bin/bash
set -euo pipefail
# firecrawl-debug-bundle.sh

BUNDLE_DIR="firecrawl-debug-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BUNDLE_DIR"

echo "=== Firecrawl Debug Bundle ===" > "$BUNDLE_DIR/summary.txt"
echo "Generated: $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$BUNDLE_DIR/summary.txt"
echo "" >> "$BUNDLE_DIR/summary.txt"

# Environment
echo "--- Runtime ---" >> "$BUNDLE_DIR/summary.txt"
node --version >> "$BUNDLE_DIR/summary.txt" 2>&1 || echo "Node: N/A" >> "$BUNDLE_DIR/summary.txt"
echo "OS: $(uname -a)" >> "$BUNDLE_DIR/summary.txt"
echo "FIRECRAWL_API_KEY: ${FIRECRAWL_API_KEY:+SET (${#FIRECRAWL_API_KEY} chars)}" >> "$BUNDLE_DIR/summary.txt"
echo "FIRECRAWL_API_URL: ${FIRECRAWL_API_URL:-https://api.firecrawl.dev (default)}" >> "$BUNDLE_DIR/summary.txt"

Step 2: Collect SDK and API Status

set -euo pipefail
# SDK version
echo "" >> "$BUNDLE_DIR/summary.txt"
echo "--- SDK ---" >> "$BUNDLE_DIR/summary.txt"
npm list @mendable/firecrawl-js 2>/dev/null >> "$BUNDLE_DIR/summary.txt" || echo "Not found in npm" >> "$BUNDLE_DIR/summary.txt"
pip show firecrawl-py 2>/dev/null >> "$BUNDLE_DIR/summary.txt" || true

# API connectivity test
echo "" >> "$BUNDLE_DIR/summary.txt"
echo "--- API Connectivity ---" >> "$BUNDLE_DIR/summary.txt"
API_RESPONSE=$(curl -s -w "\n%{http_code}" https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer ${FIRECRAWL_API_KEY:-missing}" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}' 2>&1)
HTTP_CODE=$(echo "$API_RESPONSE" | tail -1)
echo "API Status: HTTP $HTTP_CODE" >> "$BUNDLE_DIR/summary.txt"

# Credit balance
echo "" >> "$BUNDLE_DIR/summary.txt"
echo "--

Firecrawl Deploy Integration

Overview

Deploy applications using Firecrawl's web scraping API to production. Covers Vercel serverless, Cloud Run containers, self-hosted Firecrawl via Docker, and webhook endpoint deployment for async crawl results.

Prerequisites

• Firecrawl API key (FIRECRAWLAPIKEY)
• Application using @mendable/firecrawl-js
• Platform CLI (vercel, docker, or gcloud)

Instructions

Step 1: Configure Platform Secrets

set -euo pipefail
# Vercel
vercel env add FIRECRAWL_API_KEY production

# Cloud Run
echo -n "$FIRECRAWL_API_KEY" | gcloud secrets create firecrawl-api-key --data-file=-

# Docker
# Use --env-file or docker secrets

Step 2: Vercel Serverless API Route

// app/api/scrape/route.ts (Next.js App Router)
import FirecrawlApp from "@mendable/firecrawl-js";
import { NextRequest, NextResponse } from "next/server";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

export async function POST(req: NextRequest) {
  const { url, formats = ["markdown"] } = await req.json();

  if (!url) {
    return NextResponse.json({ error: "URL required" }, { status: 400 });
  }

  try {
    const result = await firecrawl.scrapeUrl(url, {
      formats,
      onlyMainContent: true,
      waitFor: 3000,
    });

    return NextResponse.json({
      success: result.success,
      markdown: result.markdown,
      title: result.metadata?.title,
      sourceURL: result.metadata?.sourceURL,
    });
  } catch (error: any) {
    return NextResponse.json(
      { error: error.message, status: error.statusCode },
      { status: error.statusCode || 500 }
    );
  }
}

Step 3: Self-Hosted Firecrawl (Docker Compose)

# docker-compose.yml
services:
  firecrawl:
    image: mendableai/firecrawl:latest
    ports:
      - "3002:3002"
    environment:
      - PORT=3002
      - USE_DB_AUTHENTICATION=false
      - REDIS_URL=redis://redis:6379
      - REDIS_RATE_LIMIT_URL=redis://redis:6379
      - NUM_WORKERS_PER_QUEUE=2
      - BULL_AUTH_KEY=${BULL_AUTH_KEY:-changeme}
    depends_on:
      redis:
        condition: service_healthy

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      - FIRECRAWL_API_KEY=fc-self-hosted
      - FIRECRAWL_API_URL=http://firecrawl:3002
    depends_on:
      - firecrawl

// Point app to self-hosted Firecrawl
const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
  apiUrl: process.env

Firecrawl Enterprise RBAC

Overview

Control access to Firecrawl scraping resources through API key management, domain allowlists, and credit budgets per team. Firecrawl's credit-based pricing means access control is primarily about limiting credit consumption and restricting scrape targets per consumer.

Prerequisites

• Firecrawl Team or Scale plan
• Dashboard access at firecrawl.dev/app
• Understanding of credit-per-page billing

Instructions

Step 1: Separate API Keys per Consumer

set -euo pipefail
# Create dedicated keys at firecrawl.dev/app for each team/service

# Content indexing pipeline — high volume
# Key: fc-content-indexer-prod (monthly credit limit: 50,000)

# Sales team prospect research — scrape only
# Key: fc-sales-research (monthly credit limit: 5,000)

# Dev/testing — minimal
# Key: fc-dev-testing (monthly credit limit: 500)

Step 2: Gateway Proxy with Domain Allowlists

import FirecrawlApp from "@mendable/firecrawl-js";

const TEAM_POLICIES: Record<string, {
  apiKey: string;
  allowedDomains: string[];
  maxPagesPerCrawl: number;
  dailyCreditLimit: number;
}> = {
  "content-team": {
    apiKey: process.env.FIRECRAWL_KEY_CONTENT!,
    allowedDomains: ["docs.*", "*.readthedocs.io", "medium.com"],
    maxPagesPerCrawl: 200,
    dailyCreditLimit: 2000,
  },
  "sales-team": {
    apiKey: process.env.FIRECRAWL_KEY_SALES!,
    allowedDomains: ["linkedin.com", "crunchbase.com", "g2.com"],
    maxPagesPerCrawl: 20,
    dailyCreditLimit: 500,
  },
  "engineering": {
    apiKey: process.env.FIRECRAWL_KEY_ENGINEERING!,
    allowedDomains: ["*"],  // unrestricted
    maxPagesPerCrawl: 100,
    dailyCreditLimit: 1000,
  },
};

function isDomainAllowed(team: string, url: string): boolean {
  const policy = TEAM_POLICIES[team];
  if (!policy) return false;
  const domain = new URL(url).hostname;
  return policy.allowedDomains.some(pattern =>
    pattern === "*" || domain.endsWith(pattern.replace("*.", "").replace("*", ""))
  );
}

function getTeamClient(team: string): FirecrawlApp {
  const policy = TEAM_POLICIES[team];
  if (!policy) throw new Error(`Unknown team: ${team}`);
  return new FirecrawlApp({ apiKey: policy.apiKey });
}

Step 3: Credit Budget Enforcement

class TeamBudget {
  private usage = new Map<string, Map<string, number>>(); // team -> date -> credits

  record(team: string, credits: number) {
    const today = new Date().toISOString().split("T")[0];
    if (!this.usage.has(team)) this.usage.set(team, new Map());
    const teamUsage = this.usage.get(team)!;
    teamUsage.set(t

Firecrawl Hello World

Overview

Four minimal examples covering Firecrawl's core endpoints: scrape (single page), crawl (multi-page), map (URL discovery), and extract (LLM structured data). Each is a standalone snippet you can run immediately.

Prerequisites

• @mendable/firecrawl-js installed (npm install @mendable/firecrawl-js)
• FIRECRAWLAPIKEY environment variable set

Instructions

Step 1: Single-Page Scrape

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// Scrape one page — returns markdown, HTML, metadata, links
const result = await firecrawl.scrapeUrl("https://docs.firecrawl.dev", {
  formats: ["markdown"],
});

console.log("Title:", result.metadata?.title);
console.log("Markdown:", result.markdown?.substring(0, 500));

Step 2: Multi-Page Crawl

// Crawl a site recursively — follows links, respects robots.txt
const crawlResult = await firecrawl.crawlUrl("https://docs.firecrawl.dev", {
  limit: 10,  // max 10 pages (saves credits)
  scrapeOptions: {
    formats: ["markdown"],
  },
});

console.log(`Crawled ${crawlResult.data?.length} pages`);
for (const page of crawlResult.data || []) {
  console.log(`  ${page.metadata?.title} — ${page.metadata?.sourceURL}`);
}

Step 3: Map a Site (URL Discovery)

// Discover all URLs on a site in ~2-3 seconds (uses sitemap + SERP)
const mapResult = await firecrawl.mapUrl("https://docs.firecrawl.dev");

console.log(`Found ${mapResult.links?.length} URLs`);
mapResult.links?.slice(0, 10).forEach(url => console.log(`  ${url}`));

Step 4: LLM Extract (Structured Data)

// Extract structured data from a page using an LLM + JSON schema
const extracted = await firecrawl.scrapeUrl("https://firecrawl.dev/pricing", {
  formats: ["extract"],
  extract: {
    schema: {
      type: "object",
      properties: {
        plans: {
          type: "array",
          items: {
            type: "object",
            properties: {
              name: { type: "string" },
              price: { type: "string" },
              credits: { type: "number" },
            },
          },
        },
      },
    },
  },
});

console.log("Pricing plans:", JSON.stringify(extracted.extract, null, 2));

Output

• Single-page markdown scraped from a live URL
• Multi-page crawl results with titles and source URLs
• Site map with all discovered URLs
• St

Firecrawl Load & Scale

Overview

Load test and scale Firecrawl scraping pipelines. Firecrawl's rate limits are per-plan (RPM and concurrent connections), so scaling means maximizing throughput within those limits using batch scraping, async crawls, and queue-based request management.

Rate Limits by Plan

Instructions

Step 1: Measure Baseline Throughput

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

async function measureThroughput(urls: string[], concurrency: number) {
  const start = Date.now();
  const results: Array<{ url: string; durationMs: number; success: boolean; chars: number }> = [];

  // Process in batches of `concurrency`
  for (let i = 0; i < urls.length; i += concurrency) {
    const batch = urls.slice(i, i + concurrency);
    const batchResults = await Promise.all(
      batch.map(async url => {
        const t0 = Date.now();
        try {
          const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
          return { url, durationMs: Date.now() - t0, success: true, chars: result.markdown?.length || 0 };
        } catch {
          return { url, durationMs: Date.now() - t0, success: false, chars: 0 };
        }
      })
    );
    results.push(...batchResults);
  }

  const totalMs = Date.now() - start;
  const succeeded = results.filter(r => r.success).length;

  console.log(`=== Throughput Report ===`);
  console.log(`URLs: ${urls.length}, Concurrency: ${concurrency}`);
  console.log(`Total time: ${totalMs}ms`);
  console.log(`Success: ${succeeded}/${urls.length}`);
  console.log(`Throughput: ${(urls.length / (totalMs / 1000)).toFixed(1)} pages/sec`);
  console.log(`Avg latency: ${(results.reduce((s, r) => s + r.durationMs, 0) / results.length).toFixed(0)}ms`);

  return results;
}

Step 2: Use Batch Scrape for Maximum Efficiency

// batchScrapeUrls is the most efficient way to scrape multiple known URLs
async function scaledBatchScrape(urls: string[], batchSize = 50) {
  const allResults: any[] = [];

  for (let i = 0; i < urls.length; i += batchSize) {
    const batch = urls.slice(i, i + batchSize);
    console.log(`Batch ${i / batchSize + 1}: scraping ${batch.length} URLs...`);

    const result = await firecrawl.batchScrapeUrls(batch, {
      formats: ["markdown"],
      onlyM

Firecrawl Local Dev Loop

Overview

Set up a fast development workflow for Firecrawl integrations. Use self-hosted Firecrawl via Docker to avoid burning API credits during development, mock the SDK for unit tests, and run integration tests against the local instance.

Prerequisites

• Node.js 18+ with npm/pnpm
• Docker + Docker Compose (for self-hosted Firecrawl)
• @mendable/firecrawl-js installed

Instructions

Step 1: Project Structure

my-firecrawl-project/
├── src/
│   ├── scraper.ts          # Firecrawl business logic
│   └── config.ts           # Environment-aware config
├── tests/
│   ├── scraper.test.ts     # Unit tests (mocked SDK)
│   └── integration.test.ts # Integration tests (real API)
├── docker-compose.yml      # Self-hosted Firecrawl
├── .env.local              # Dev secrets (git-ignored)
├── .env.example            # Template for team
└── package.json

Step 2: Self-Hosted Firecrawl for Zero-Credit Dev

# docker-compose.yml
services:
  firecrawl:
    image: mendableai/firecrawl:latest
    ports:
      - "3002:3002"
    environment:
      - PORT=3002
      - USE_DB_AUTHENTICATION=false
      - REDIS_URL=redis://redis:6379
      - NUM_WORKERS_PER_QUEUE=1
      - BULL_AUTH_KEY=devonly
    depends_on:
      redis:
        condition: service_healthy

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

set -euo pipefail
# Start local Firecrawl
docker compose up -d

# Verify it's running
curl -s http://localhost:3002/health | jq .

Step 3: Environment-Aware Configuration

// src/config.ts
import FirecrawlApp from "@mendable/firecrawl-js";

export function getFirecrawl(): FirecrawlApp {
  const isDev = process.env.NODE_ENV !== "production";

  return new FirecrawlApp({
    apiKey: process.env.FIRECRAWL_API_KEY || "fc-dev",
    // Point to local Docker instance in dev
    ...(isDev && process.env.FIRECRAWL_API_URL
      ? { apiUrl: process.env.FIRECRAWL_API_URL }
      : {}),
  });
}

# .env.local (for development — zero API credits used)
FIRECRAWL_API_KEY=fc-localdev
FIRECRAWL_API_URL=http://localhost:3002
NODE_ENV=development

Step 4: Unit Tests with Mocked SDK

// tests/scraper.test.ts
import { describe, it, expect, vi, beforeEach } from "vitest";

// Mock the SDK
vi.mock("@mendable/firecrawl-js", () => ({
  default: vi.fn().mockImplementation(() => ({
    scrapeUrl: vi.fn().mockResolvedValue({
      success: true,
      markdown: "# Hello World\n\nS

Firecrawl Migration Deep Dive

Current State

!npm list puppeteer playwright cheerio 2>/dev/null | grep -E "puppeteer|playwright|cheerio" || echo 'No scraping libs found'

Overview

Migrate from custom scraping (Puppeteer, Playwright, Cheerio) or competing APIs to Firecrawl. Firecrawl eliminates browser management, anti-bot handling, and JS rendering infrastructure. This skill shows equivalent code for common scraping patterns.

Migration Comparison

Instructions

Step 1: Replace Puppeteer Single-Page Scrape

// BEFORE: Puppeteer (20+ lines, browser management)
import puppeteer from "puppeteer";

async function scrapePuppeteer(url: string) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: "networkidle2" });
  const html = await page.content();
  const title = await page.title();
  await browser.close();
  return { html, title };
}

// AFTER: Firecrawl (5 lines, no browser needed)
import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });

async function scrapeFirecrawl(url: string) {
  const result = await firecrawl.scrapeUrl(url, {
    formats: ["markdown"],
    onlyMainContent: true,
    waitFor: 2000,
  });
  return { markdown: result.markdown, title: result.metadata?.title };
}

Step 2: Replace Cheerio HTML Parsing

// BEFORE: fetch + cheerio (manual parsing)
import * as cheerio from "cheerio";

async function scrapeCheerio(url: string) {
  const html = await fetch(url).then(r => r.text());
  const $ = cheerio.load(html);
  return {
    title: $("h1").first().text(),
    content: $("main").text(),
    links: $("a").map((_, el) => $(el).attr("href")).get(),
  };
}

// AFTER: Firecrawl with extract (LLM-powered, no CSS selectors)
async function extractFirecrawl(url: string) {
  const result = await firecrawl.scrapeUrl(url, {
    formats: ["extract", "links"],
    extract: {
      schema: {
        type: "

Firecrawl Multi-Environment Setup

Overview

Firecrawl's credit-based pricing makes environment separation critical. Development should use self-hosted Firecrawl or strict limits to avoid burning production credits during testing. This skill covers per-environment config, self-hosted Docker for dev, and credit budget enforcement.

Environment Strategy

Instructions

Step 1: Environment-Aware Configuration

// config/firecrawl.ts
import FirecrawlApp from "@mendable/firecrawl-js";

type Env = "development" | "staging" | "production";

interface FirecrawlConfig {
  apiKey: string;
  apiUrl?: string;
  maxPagesPerCrawl: number;
  maxDepth: number;
  concurrency: number;
  waitFor: number;
}

const configs: Record<Env, FirecrawlConfig> = {
  development: {
    apiKey: process.env.FIRECRAWL_API_KEY_DEV || "fc-localdev",
    apiUrl: process.env.FIRECRAWL_API_URL_DEV || "http://localhost:3002",
    maxPagesPerCrawl: 10,
    maxDepth: 2,
    concurrency: 1,
    waitFor: 2000,
  },
  staging: {
    apiKey: process.env.FIRECRAWL_API_KEY_STAGING!,
    maxPagesPerCrawl: 50,
    maxDepth: 3,
    concurrency: 2,
    waitFor: 3000,
  },
  production: {
    apiKey: process.env.FIRECRAWL_API_KEY_PROD!,
    maxPagesPerCrawl: 500,
    maxDepth: 5,
    concurrency: 5,
    waitFor: 3000,
  },
};

export function getConfig(): FirecrawlConfig {
  const env = (process.env.NODE_ENV || "development") as Env;
  return configs[env] || configs.development;
}

export function getFirecrawl(): FirecrawlApp {
  const cfg = getConfig();
  return new FirecrawlApp({
    apiKey: cfg.apiKey,
    ...(cfg.apiUrl ? { apiUrl: cfg.apiUrl } : {}),
  });
}

Step 2: Self-Hosted Firecrawl for Development

# docker-compose.dev.yml
services:
  firecrawl:
    image: mendableai/firecrawl:latest
    ports:
      - "3002:3002"
    environment:
      - PORT=3002
      - USE_DB_AUTHENTICATION=false
      - REDIS_URL=redis://redis:6379
      - NUM_WORKERS_PER_QUEUE=1
      - BULL_AUTH_KEY=devonly
    depends_on:
      redis:
        condition: service_healthy

  redis:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

set -euo pipefail
docker compose -

Firecrawl Observability

Overview

Monitor Firecrawl web scraping pipelines for success rates, credit consumption, content quality, and latency. Key signals: scrape success rate, crawl job completion, credit burn velocity, extraction quality (did markdown actually contain useful content vs error pages), and webhook delivery health.

Key Metrics

Instructions

Step 1: Instrumented Firecrawl Wrapper

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// Counters (use your metrics library: prom-client, statsd, datadog, etc.)
function emit(metric: string, value: number, tags?: Record<string, string>) {
  console.log(JSON.stringify({ metric, value, tags, timestamp: Date.now() }));
}

export async function instrumentedScrape(url: string) {
  const start = Date.now();
  try {
    const result = await firecrawl.scrapeUrl(url, {
      formats: ["markdown"],
      onlyMainContent: true,
    });

    const duration = Date.now() - start;
    const quality = evaluateQuality(result);

    emit("firecrawl_scrapes_total", 1, { status: "success" });
    emit("firecrawl_scrape_duration_ms", duration);
    emit("firecrawl_credits_used", 1);
    emit("firecrawl_content_quality", 1, { quality });

    return result;
  } catch (error: any) {
    emit("firecrawl_scrapes_total", 1, {
      status: "error",
      error_code: String(error.statusCode || "unknown"),
    });
    emit("firecrawl_scrape_duration_ms", Date.now() - start);
    throw error;
  }
}

function evaluateQuality(result: any): string {
  const md = result.markdown || "";
  if (md.length < 100) return "empty";
  if (/404|not found|access denied|captcha/i.test(md)) return "error_page";
  if (!/^#{1,3}\s/m.test(md)) return "no_structure";
  return "good";
}

Step 2: Credit Consumption Monitor

async function checkCreditHealth() {
  const response = await fetch("https://api.firecrawl.dev/v1/team/credits", {
    headers: {

Firecrawl Performance Tuning

Overview

Optimize Firecrawl API performance by choosing efficient scraping modes, caching results, using batch endpoints, and minimizing unnecessary rendering. Key levers: format selection (markdown vs HTML vs screenshot), waitFor tuning, onlyMainContent, and batch vs individual scraping.

Latency Benchmarks

Instructions

Step 1: Minimize Formats (Biggest Win)

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// SLOW: requesting everything
const slow = await firecrawl.scrapeUrl(url, {
  formats: ["markdown", "html", "links", "screenshot"],
  // screenshot + full HTML = 3-5x slower
});

// FAST: request only what you need
const fast = await firecrawl.scrapeUrl(url, {
  formats: ["markdown"],   // markdown only = fastest
  onlyMainContent: true,   // skip nav/footer/sidebar
});

Step 2: Tune waitFor for JS-Heavy Pages

// Default: no JS wait (fastest, works for static sites)
const staticResult = await firecrawl.scrapeUrl("https://docs.example.com", {
  formats: ["markdown"],
  // No waitFor needed — content is in initial HTML
});

// SPA/dynamic pages: add minimal wait
const spaResult = await firecrawl.scrapeUrl("https://app.example.com", {
  formats: ["markdown"],
  waitFor: 3000,  // 3s — enough for most SPAs
  onlyMainContent: true,
});

// Heavy interactive page: use actions instead of long wait
const heavyResult = await firecrawl.scrapeUrl("https://dashboard.example.com", {
  formats: ["markdown"],
  actions: [
    { type: "wait", selector: ".data-table" },  // wait for specific element
    { type: "scroll", direction: "down" },        // trigger lazy loading
  ],
});

Step 3: Cache Scraped Content

import { LRUCache } from "lru-cache";
import { createHash } from "crypto";

const scrapeCache = new LRUCache<string, any>({
  max: 500,          // max 500 cached pages
  ttl: 3600000,      // 1 hour TTL
});

async function cachedScrape(url: string) {
  const key = createHash("md5").update(

Firecrawl Policy Guardrails

Overview

Automated guardrails for Firecrawl scraping pipelines. Web scraping carries legal (robots.txt, ToS), ethical (rate limiting, attribution), and cost (credit burn) risks. This skill implements domain blocklists, credit budgets, content quality gates, and per-domain rate limits as enforceable policies.

Instructions

Step 1: Domain Policy Enforcement

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

class ScrapePolicy {
  // Domains that explicitly prohibit scraping in their ToS
  static BLOCKED_DOMAINS = [
    "facebook.com", "instagram.com",  // Meta ToS
    "linkedin.com",                    // LinkedIn ToS
    "twitter.com", "x.com",           // X/Twitter ToS
  ];

  // Domains with sensitive/regulated content
  static SENSITIVE_DOMAINS = [
    "*.gov", "*.mil",                 // Government
    "*.edu",                          // Educational (FERPA)
  ];

  static validateUrl(url: string): void {
    const hostname = new URL(url).hostname;

    for (const blocked of this.BLOCKED_DOMAINS) {
      if (hostname === blocked || hostname.endsWith(`.${blocked}`)) {
        throw new PolicyViolation(`Domain "${hostname}" is blocked: ToS prohibits scraping`);
      }
    }

    for (const pattern of this.SENSITIVE_DOMAINS) {
      const regex = new RegExp("^" + pattern.replace("*.", ".*\\.") + "$");
      if (regex.test(hostname)) {
        console.warn(`CAUTION: "${hostname}" matches sensitive domain pattern "${pattern}"`);
      }
    }
  }
}

class PolicyViolation extends Error {
  constructor(message: string) {
    super(message);
    this.name = "PolicyViolation";
  }
}

Step 2: Credit Budget Enforcement

class CrawlBudget {
  private usage = new Map<string, number>();
  private dailyLimit: number;

  constructor(dailyLimit = 5000) {
    this.dailyLimit = dailyLimit;
  }

  authorize(estimatedPages: number): void {
    const today = new Date().toISOString().split("T")[0];
    const used = this.usage.get(today) || 0;

    if (used + estimatedPages > this.dailyLimit) {
      throw new PolicyViolation(
        `Daily credit limit would be exceeded: ${used} used + ${estimatedPages} requested > ${this.dailyLimit} limit`
      );
    }
  }

  record(pagesScraped: number) {
    const today = new Date().toISOString().split("T")[0];
    this.usage.set(today, (this.usage.get(today) || 0) + pagesScraped);
  }
}

const budget = new CrawlBudget(5000);

Step 3: Content Quality Gate

function validateScrapedContent(result: any): {
  accepted: boolean;
  reas

Firecrawl Production Checklist

Overview

Pre-deployment validation checklist for applications using Firecrawl's scrape, crawl, map, and extract APIs. Covers credential management, crawl safety limits, error handling, monitoring, and rollback.

Prerequisites

• Staging environment tested and passing
• Production API key from firecrawl.dev/app
• Monitoring infrastructure ready

Pre-Deployment Checklist

Credentials & Security

• [ ] Production FIRECRAWLAPIKEY in secure vault (not in code or .env)
• [ ] Key starts with fc- and is scoped to production
• [ ] Different API keys for dev/staging/production
• [ ] .env files in .gitignore
• [ ] Webhook secrets stored securely
• [ ] Git history scanned for leaked keys

Crawl Safety

• [ ] All crawlUrl calls have limit parameter set
• [ ] maxDepth configured to prevent unbounded crawling
• [ ] includePaths / excludePaths filters applied where appropriate
• [ ] Credit budget tracking implemented (daily limit alerts)
• [ ] No hardcoded URLs in production code

Error Handling

• [ ] 429 rate limit handling with exponential backoff
• [ ] 402 credit exhaustion handled gracefully (no crash)
• [ ] 401 auth failure logged and alerted
• [ ] Async crawl jobs have timeout with deadline
• [ ] Fallback from crawl to individual scrape on failure
• [ ] Empty markdown detection (JS rendering issues)

Monitoring & Alerting

• [ ] Scrape success/failure rate tracked
• [ ] Credit consumption monitored
• [ ] Crawl job completion rate tracked
• [ ] Alert on credit balance below threshold
• [ ] Alert on error rate > 5%
• [ ] Webhook delivery failures logged

Instructions

Step 1: Verify API Connectivity

set -euo pipefail
# Test production key
curl -s https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY_PROD" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}' | jq '.success'

# Check credit balance
curl -s https://api.firecrawl.dev/v1/team/credits \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY_PROD" | jq .

Step 2: Health Check Endpoint

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

export async function healthCheck() {
  const start = Date.now();
  try {
    const result = await firecrawl.scrapeUrl("

Firecrawl Rate Limits

Overview

Firecrawl enforces rate limits per API key measured in requests per minute and concurrent connections. When exceeded, the API returns 429 Too Many Requests with a Retry-After header. This skill covers backoff strategies, request queuing, and proactive throttling.

Rate Limit Tiers

Concurrent crawl jobs count against concurrency limits. If the queue is full, new jobs are rejected with 429.

Instructions

Step 1: Exponential Backoff with Jitter

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

async function withBackoff<T>(
  operation: () => Promise<T>,
  config = { maxRetries: 5, baseDelayMs: 1000, maxDelayMs: 32000 }
): Promise<T> {
  for (let attempt = 0; attempt <= config.maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error: any) {
      if (attempt === config.maxRetries) throw error;

      const status = error.statusCode || error.status;
      // Only retry on 429 (rate limit) and 5xx (server error)
      if (status && status !== 429 && status < 500) throw error;

      // Exponential delay with random jitter to prevent thundering herd
      const exponentialDelay = config.baseDelayMs * Math.pow(2, attempt);
      const jitter = Math.random() * 500;
      const delay = Math.min(exponentialDelay + jitter, config.maxDelayMs);

      console.warn(`Rate limited (${status}). Retry ${attempt + 1}/${config.maxRetries} in ${delay.toFixed(0)}ms`);
      await new Promise(r => setTimeout(r, delay));
    }
  }
  throw new Error("Unreachable");
}

// Usage
const result = await withBackoff(() =>
  firecrawl.scrapeUrl("https://example.com", { formats: ["markdown"] })
);

Step 2: Queue-Based Rate Limiting with p-queue

import PQueue from "p-queue";

// Limit to 5 concurrent requests, max 10 per second
const scrapeQueue = new PQueue({
  concurrency: 5,
  interval: 1000,
  intervalCap: 10,
});

async function queuedScrape(url: string) {
  return scrapeQueue.add(() =>
    withBackoff(() =>
      firecrawl.scrapeUrl(url, { formats: ["markdown"] })
    )
  );
}

// Scrape many URLs respecting rate limits
const urls = ["https://a.com", "https://b.com"

Firecrawl Reference Architecture

Overview

Production architecture for web scraping and content ingestion with Firecrawl. Covers three tiers: on-demand scraping, scheduled crawl pipelines, and real-time RAG ingestion. Uses all four Firecrawl endpoints: scrape, crawl, map, and extract.

Architecture Diagram

┌─────────────────────────────────────────────────────────┐
│                   Firecrawl Pipeline                     │
│                                                          │
│  ┌──────────┐  ┌──────────┐  ┌──────┐  ┌───────────┐   │
│  │ scrapeUrl│  │ crawlUrl │  │mapUrl│  │ extract   │   │
│  │ (1 page) │  │ (N pages)│  │(URLs)│  │ (LLM+JSON)│   │
│  └────┬─────┘  └────┬─────┘  └──┬───┘  └─────┬─────┘   │
│       │              │            │            │          │
│       ▼              ▼            ▼            ▼          │
│  ┌───────────────────────────────────────────────────┐   │
│  │            Content Processing Layer                │   │
│  │  Clean MD │ Validate │ Deduplicate │ Chunk        │   │
│  └─────────────────────┬─────────────────────────────┘   │
│                         │                                 │
│  ┌─────────────────────┴─────────────────────────────┐   │
│  │              Storage & Output                      │   │
│  │  Files │ Database │ Vector Store │ Search Index    │   │
│  └───────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Instructions

Step 1: Firecrawl Service Layer

// src/firecrawl/service.ts
import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// Single page scrape
export async function scrapePage(url: string) {
  return firecrawl.scrapeUrl(url, {
    formats: ["markdown"],
    onlyMainContent: true,
    waitFor: 2000,
  });
}

// Site-wide crawl with safety limits
export async function crawlSite(baseUrl: string, opts?: {
  maxPages?: number;
  paths?: string[];
  excludePaths?: string[];
}) {
  return firecrawl.crawlUrl(baseUrl, {
    limit: opts?.maxPages || 50,
    maxDepth: 3,
    includePaths: opts?.paths,
    excludePaths: opts?.excludePaths || ["/blog/*", "/news/*"],
    scrapeOptions: { formats: ["markdown"], onlyMainContent: true },
  });
}

// Fast URL discovery
export async function discoverUrls(baseUrl: string) {
  const map = await firecrawl.mapUrl(baseUrl);
  return map.links || [];
}

// Structured data extraction
export async function extractData(url: string, schema: object) {
  return firecrawl.scrapeUrl(url, {
    formats: ["extract"],
    extract: { schema },
  });
}

Step 2: Content Processing Pipeline

// src/pipeline/processor.ts
import { createHash } from "crypto";

interface ProcessedPa

Firecrawl Reliability Patterns

Overview

Production reliability patterns for Firecrawl scraping pipelines. Firecrawl's async crawl model, JS rendering, and credit-based pricing create specific reliability challenges: crawl jobs may time out, scraped content may be empty (bot detection, JS failures), and credits can be burned by runaway crawls. This skill covers battle-tested patterns for each.

Instructions

Step 1: Robust Crawl with Timeout and Backoff

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

async function reliableCrawl(
  url: string,
  opts: { limit: number; paths?: string[] },
  timeoutMs = 600000
) {
  const job = await firecrawl.asyncCrawlUrl(url, {
    limit: opts.limit,
    includePaths: opts.paths,
    scrapeOptions: { formats: ["markdown"], onlyMainContent: true },
  });

  const deadline = Date.now() + timeoutMs;
  let pollInterval = 2000;

  while (Date.now() < deadline) {
    const status = await firecrawl.checkCrawlStatus(job.id);

    if (status.status === "completed") return status;
    if (status.status === "failed") {
      throw new Error(`Crawl failed: ${status.error}`);
    }

    await new Promise(r => setTimeout(r, pollInterval));
    pollInterval = Math.min(pollInterval * 1.5, 30000); // back off to 30s max
  }

  throw new Error(`Crawl timed out after ${timeoutMs}ms (job: ${job.id})`);
}

Step 2: Content Quality Validation

interface ScrapedPage {
  url: string;
  markdown: string;
  metadata: { title?: string; statusCode?: number };
}

function validateContent(page: ScrapedPage): {
  valid: boolean;
  reason?: string;
} {
  if (!page.markdown || page.markdown.length < 100) {
    return { valid: false, reason: "Content too short" };
  }

  if (page.metadata.statusCode && page.metadata.statusCode >= 400) {
    return { valid: false, reason: `HTTP ${page.metadata.statusCode}` };
  }

  const errorPatterns = [
    "access denied", "403 forbidden", "page not found",
    "captcha", "please verify", "enable javascript",
  ];
  const lower = page.markdown.toLowerCase();
  for (const pattern of errorPatterns) {
    if (lower.includes(pattern)) {
      return { valid: false, reason: `Error page detected: "${pattern}"` };
    }
  }

  return { valid: true };
}

Step 3: Crawl-to-Scrape Fallback

// If a full crawl fails, fall back to scraping critical pages individually
async function resilientFetch(urls: string[]): Promise<any[]> {
  // Try batch scrape first (most efficient)
  try {
    const batch = await firecrawl.batchScrapeUrls(urls, {
      formats: ["markdown"],
      onlyMainContent: true,
   

Firecrawl SDK Patterns

Overview

Production-ready patterns for Firecrawl SDK (@mendable/firecrawl-js / firecrawl-py). Covers singleton client, typed wrappers, retry with backoff, response validation, and reusable scraping service patterns.

Prerequisites

• @mendable/firecrawl-js installed
• Understanding of async/await patterns
• TypeScript strict mode recommended

Instructions

Step 1: Singleton Client with Configuration

// src/firecrawl/client.ts
import FirecrawlApp from "@mendable/firecrawl-js";

let instance: FirecrawlApp | null = null;

export function getFirecrawl(): FirecrawlApp {
  if (!instance) {
    if (!process.env.FIRECRAWL_API_KEY) {
      throw new Error("FIRECRAWL_API_KEY environment variable is required");
    }
    instance = new FirecrawlApp({
      apiKey: process.env.FIRECRAWL_API_KEY,
      ...(process.env.FIRECRAWL_API_URL
        ? { apiUrl: process.env.FIRECRAWL_API_URL }
        : {}),
    });
  }
  return instance;
}

Step 2: Typed Scrape Wrapper

// src/firecrawl/scrape.ts
import { getFirecrawl } from "./client";

interface ScrapeResult {
  url: string;
  title: string;
  markdown: string;
  links: string[];
  scrapedAt: string;
}

export async function scrapePage(
  url: string,
  options?: { waitFor?: number; includeLinks?: boolean }
): Promise<ScrapeResult> {
  const firecrawl = getFirecrawl();
  const formats: string[] = ["markdown"];
  if (options?.includeLinks) formats.push("links");

  const result = await firecrawl.scrapeUrl(url, {
    formats,
    onlyMainContent: true,
    ...(options?.waitFor ? { waitFor: options.waitFor } : {}),
  });

  if (!result.success) {
    throw new Error(`Scrape failed for ${url}: ${result.error}`);
  }

  return {
    url: result.metadata?.sourceURL || url,
    title: result.metadata?.title || "",
    markdown: result.markdown || "",
    links: result.links || [],
    scrapedAt: new Date().toISOString(),
  };
}

Step 3: Retry with Exponential Backoff

// src/firecrawl/retry.ts
export async function withRetry<T>(
  operation: () => Promise<T>,
  config = { maxRetries: 3, baseDelayMs: 1000, maxDelayMs: 30000 }
): Promise<T> {
  for (let attempt = 0; attempt <= config.maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error: any) {
      if (attempt === config.maxRetries) throw error;

      const status = error.statusCode || error.status;
      // Only retry on rate limits (429) and server errors (5xx)
      if (status && status !== 429 && status < 500) throw error;

      const delay = Math.min(
        config.baseDelayMs * Math.pow(2, attempt) + Math.random() * 500,
        

Firecrawl Security Basics

Overview

Security best practices for Firecrawl API keys, webhook signature verification, and scraped content handling. Firecrawl API keys start with fc- and grant full access to scrape, crawl, map, and extract endpoints — protecting them is critical.

Prerequisites

• Firecrawl API key
• Understanding of environment variables
• Webhook endpoint (if using async crawl callbacks)

Instructions

Step 1: Secure API Key Storage

# .env (NEVER commit to git)
FIRECRAWL_API_KEY=fc-your-api-key-here

# .gitignore — add these patterns
echo -e "\n.env\n.env.local\n.env.*.local" >> .gitignore

// Validate key exists before creating client
import FirecrawlApp from "@mendable/firecrawl-js";

if (!process.env.FIRECRAWL_API_KEY?.startsWith("fc-")) {
  throw new Error("FIRECRAWL_API_KEY must be set and start with 'fc-'");
}

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY,
});

Step 2: Verify Webhook Signatures

Firecrawl signs webhook payloads with HMAC-SHA256 via the X-Firecrawl-Signature header.

import crypto from "crypto";

function verifyWebhookSignature(
  payload: string,
  signature: string,
  secret: string
): boolean {
  const expected = crypto
    .createHmac("sha256", secret)
    .update(payload)
    .digest("hex");

  // Timing-safe comparison prevents timing attacks
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expected)
  );
}

// Express webhook handler with verification
app.post("/webhooks/firecrawl", (req, res) => {
  const signature = req.headers["x-firecrawl-signature"] as string;
  const rawBody = JSON.stringify(req.body);

  if (!verifyWebhookSignature(rawBody, signature, process.env.FIRECRAWL_WEBHOOK_SECRET!)) {
    console.error("Invalid webhook signature — rejecting");
    return res.status(401).json({ error: "Invalid signature" });
  }

  // Process verified webhook
  const { type, data } = req.body;
  console.log(`Verified webhook: ${type}`);
  res.status(200).json({ received: true });
});

Step 3: Separate Keys per Environment

# GitHub Actions secrets
gh secret set FIRECRAWL_API_KEY_DEV --body "fc-dev-..."
gh secret set FIRECRAWL_API_KEY_STAGING --body "fc-staging-..."
gh secret set FIRECRAWL_API_KEY_PROD --body "fc-prod-..."

// Load correct key based on environment
const KEY_MAP: Record<string, string> = {
  development: "FIRECRAWL_API_KEY_DEV",
  staging: "FIRECRAWL_API_KEY_STAGING",
  production: "FIRECRAWL_API_KEY_PROD",

Firecrawl Upgrade & Migration

Current State

!npm list @mendable/firecrawl-js 2>/dev/null | grep firecrawl || echo 'Not installed'

Overview

Guide for upgrading @mendable/firecrawl-js SDK versions and migrating from Firecrawl API v0/v1 to v2. Covers breaking changes in import paths, method signatures, response formats, and the new extract v2 schema format.

Version History

Instructions

Step 1: Check Current Version

set -euo pipefail
# Check installed version
npm list @mendable/firecrawl-js

# Check latest available
npm view @mendable/firecrawl-js version

Step 2: Create Upgrade Branch

set -euo pipefail
git checkout -b upgrade/firecrawl-sdk
npm install @mendable/firecrawl-js@latest
npm test

Step 3: Migration — v0 to v1/v2

Import Changes

// No change needed — import has been stable
import FirecrawlApp from "@mendable/firecrawl-js";

Crawl Method Changes (v0 -> v1)

// BEFORE (v0): crawlUrl with waitUntilDone
const result = await firecrawl.crawlUrl("https://example.com", {
  crawlerOptions: { limit: 50 },
  pageOptions: { onlyMainContent: true },
  waitUntilDone: true,
});

// AFTER (v1+): crawlUrl returns synchronously, or use asyncCrawlUrl
const result = await firecrawl.crawlUrl("https://example.com", {
  limit: 50,
  scrapeOptions: {
    formats: ["markdown"],
    onlyMainContent: true,
  },
});

// For large crawls, use async with polling
const job = await firecrawl.asyncCrawlUrl("https://example.com", {
  limit: 500,
  scrapeOptions: { formats: ["markdown"] },
});
const status = await firecrawl.checkCrawlStatus(job.id);

Scrape Options Changes (v0 -> v1)

// BEFORE (v0)
await firecrawl.scrapeUrl("https://example.com", {
  pageOptions: { onlyMainContent: true },
  extractorOptions: { mode: "llm-extraction", schema: mySchema },
});

// AFTER (v1+)
await firecrawl.scrapeUrl("https://example.com", {
  formats: ["markdown", "extract"],
  onlyMainContent: true,
  extract: { schema: mySchema },
});

Extract v2 Format (v1 -> v2)

// BEFORE (v1): extract as top-level option
await firecrawl.scrapeUrl(url, {
  formats: ["extract"],
  extract: { schema: { type: "object", ... 

Firecrawl Webhooks & Events

Overview

Handle Firecrawl webhooks for real-time notifications on async crawl and batch scrape jobs. Instead of polling checkCrawlStatus, configure a webhook URL and Firecrawl will POST events as pages are scraped and jobs complete. Signed with HMAC-SHA256 via X-Firecrawl-Signature.

Webhook Event Types

Instructions

Step 1: Start Crawl with Webhook

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// Webhook as string (simple)
const job = await firecrawl.asyncCrawlUrl("https://docs.example.com", {
  limit: 100,
  scrapeOptions: { formats: ["markdown"] },
  webhook: "https://api.yourapp.com/webhooks/firecrawl",
});

console.log(`Crawl started: ${job.id}`);

// Webhook as object (with metadata and event filtering)
const job2 = await firecrawl.asyncCrawlUrl("https://docs.example.com", {
  limit: 100,
  scrapeOptions: { formats: ["markdown"] },
  webhook: {
    url: "https://api.yourapp.com/webhooks/firecrawl",
    events: ["completed", "page"],  // only these events
    metadata: {
      projectId: "my-project",
      triggeredBy: "cron",
    },
  },
});

Step 2: Webhook Handler with Signature Verification

import express from "express";
import crypto from "crypto";

const app = express();
app.use(express.json());

function verifySignature(body: string, signature: string): boolean {
  if (!process.env.FIRECRAWL_WEBHOOK_SECRET) return true; // skip if not configured
  const expected = crypto
    .createHmac("sha256", process.env.FIRECRAWL_WEBHOOK_SECRET)
    .update(body)
    .digest("hex");
  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}

app.post("/webhooks/firecrawl", express.raw({ type: "application/json" }), async (req, res) => {
  const rawBody = req.body.toString();
  const signature = req.headers["x-firecrawl-signature"] as string;

  if (!verifySignature(rawBody, signature)) {
    return res.status(401).json({ error: "Invalid signatur