Firecrawl Migration Deep Dive

Current State

!npm list puppeteer playwright cheerio 2>/dev/null | grep -E "puppeteer|playwright|cheerio" || echo 'No scraping libs found'

Overview

Migrate from custom scraping (Puppeteer, Playwright, Cheerio) or competing APIs to Firecrawl. Firecrawl eliminates browser management, anti-bot handling, and JS rendering infrastructure. This skill shows equivalent code for common scraping patterns.

Prerequisites

• An inventory of existing targets, permissions, extraction schemas, rate controls, and data-retention requirements.
• A staging environment with approved target pages or synthetic fixtures; do not use production credentials in migration tests.
• A rollback owner and a period in which the old path remains available for reconciliation.

Migration Comparison

Instructions

Step 1: Replace Puppeteer Single-Page Scrape

// BEFORE: Puppeteer (20+ lines, browser management)
import puppeteer from "puppeteer";

async function scrapePuppeteer(url: string) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: "networkidle2" });
  const html = await page.content();
  const title = await page.title();
  await browser.close();
  return { html, title };
}

// AFTER: Firecrawl (5 lines, no browser needed)
import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });

async function scrapeFirecrawl(url: string) {
  const result = await firecrawl.scrapeUrl(url, {
    formats: ["markdown"],
    onlyMainContent: true,
    waitFor: 2000,
  });
  return { markdown: result.markdown, title: result.metadata?.title };
}

Output

Keep a migration receipt listing each target, old and proposed behavior, validation result, permitted data destination, canary outcome, and rollback decision. Redact page content, credentials, and any personal data collected during comparison.

Examples

Choose one approved staging page and compare only metadata,

Firecrawl Observability

Overview

Monitor Firecrawl web scraping pipelines for success rates, credit consumption, content quality, and latency. Key signals: scrape success rate, crawl job completion, credit burn velocity, extraction quality (did markdown actually contain useful content vs error pages), and webhook delivery health.

Prerequisites

• An approved telemetry schema that uses aggregate measurements and opaque job identifiers.
• Named alert owners, escalation thresholds, data-retention rules, and dashboards with restricted access.
• Synthetic fixtures for alert testing and a documented way to suppress noisy signals without hiding incidents.

Output

Produce an observability receipt with metric definitions, dashboard reference, alert thresholds, test result, owner, and review date. Do not send page content, API keys, or personal data to metrics, traces, or alert payloads.

Examples

Use a synthetic target to create one successful crawl and one policy rejection. Confirm dashboards count both outcomes by category, an alert fires only on the configured threshold, and the alert contains an opaque job ID rather than captured content.

Key Metrics

Instructions

Step 1: Instrumented Firecrawl Wrapper

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// Counters (use your metrics library: prom-client, statsd, datadog, etc.)
function emit(metric: string, value: number, tags?: Record<string, string>) {
  console.log(JSON.stringify({ metric, value, tags, timestamp: Date.now() }));
}

export async function instrumentedScrape(url: string) {
  const start = Date.now();
  try {
    const result = await firecrawl.scrapeUrl(url, {
      formats: ["markdown"],
      onlyMainContent: true,
    });

    const duration = Date.now() - start;
    const quality = evaluateQuality(result);

    emit("firecrawl_scrapes_total", 1, { status: "success" });
    emit("firecrawl_scrape_duration_ms", duration);
    emit("firecrawl_credits_used", 1);
    emit("firecrawl_content_quality", 1, { quality });

Firecrawl Performance Tuning

Overview

Optimize Firecrawl API performance by choosing efficient scraping modes, caching results, using batch endpoints, and minimizing unnecessary rendering. Key levers: format selection (markdown vs HTML vs screenshot), waitFor tuning, onlyMainContent, and batch vs individual scraping.

Prerequisites

• A redacted baseline for latency, success rate, queue age, and budget consumption.
• Explicit freshness, retention, target-policy, and capacity requirements for each workflow.
• Synthetic or approved staging targets and a rollback switch for concurrency or cache changes.

Output

Produce a performance receipt with baseline and post-change aggregate metrics, selected format, concurrency/cache settings, target policy version, owner, and rollback result. Exclude scraped content and credentials.

Latency Benchmarks

Instructions

Step 1: Minimize Formats (Biggest Win)

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// SLOW: requesting everything
const slow = await firecrawl.scrapeUrl(url, {
  formats: ["markdown", "html", "links", "screenshot"],
  // screenshot + full HTML = 3-5x slower
});

// FAST: request only what you need
const fast = await firecrawl.scrapeUrl(url, {
  formats: ["markdown"],   // markdown only = fastest
  onlyMainContent: true,   // skip nav/footer/sidebar
});

Step 2: Tune waitFor for JS-Heavy Pages

// Default: no JS wait (fastest, works for static sites)
const staticResult = await firecrawl.scrapeUrl("https://docs.example.com", {
  formats: ["markdown"],
  // No waitFor needed — content is in initial HTML
});

// SPA/dynamic pages: add minimal wait
const spaResult = await firecrawl.scrapeUrl("https://app.example.com", {
  formats: ["markdown"],
  waitFor: 3000,  // 3s — enough for most SPAs
  onlyMainContent: true,
});

// Heavy interactive page: use actions instead of long wait
const heavyResult = await firecrawl.scrapeUrl("https://dashboard.example.com", {
  formats: ["markdown"],
  actions: [
    { type: "wait", selector: ".d

Firecrawl Policy Guardrails

Overview

Automated guardrails for Firecrawl scraping pipelines. Web scraping carries legal (robots.txt, ToS), ethical (rate limiting, attribution), and cost (credit burn) risks. This skill implements domain blocklists, credit budgets, content quality gates, and per-domain rate limits as enforceable policies.

Prerequisites

• A policy owner, written domain allowlist/blocklist, and review process for exceptions.
• Current source terms and robots guidance assessed for each target where required; this guide does not replace legal review.
• Enforced budget, rate-control, retention, and incident-response settings with synthetic test targets.

Output

Keep a policy decision receipt naming the target, policy version, allow/deny result, exception owner, aggregate budget use, retention decision, and review date. Do not store captured content or credentials in the receipt.

Instructions

Step 1: Domain Policy Enforcement

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

class ScrapePolicy {
  // Domains that explicitly prohibit scraping in their ToS
  static BLOCKED_DOMAINS = [
    "facebook.com", "instagram.com",  // Meta ToS
    "linkedin.com",                    // LinkedIn ToS
    "twitter.com", "x.com",           // X/Twitter ToS
  ];

  // Domains with sensitive/regulated content
  static SENSITIVE_DOMAINS = [
    "*.gov", "*.mil",                 // Government
    "*.edu",                          // Educational (FERPA)
  ];

  static validateUrl(url: string): void {
    const hostname = new URL(url).hostname;

    for (const blocked of this.BLOCKED_DOMAINS) {
      if (hostname === blocked || hostname.endsWith(`.${blocked}`)) {
        throw new PolicyViolation(`Domain "${hostname}" is blocked: ToS prohibits scraping`);
      }
    }

    for (const pattern of this.SENSITIVE_DOMAINS) {
      const regex = new RegExp("^" + pattern.replace("*.", ".*\\.") + "$");
      if (regex.test(hostname)) {
        console.warn(`CAUTION: "${hostname}" matches sensitive domain pattern "${pattern}"`);
      }
    }
  }
}

class PolicyViolation extends Error {
  constructor(message: string) {
    super(message);
    this.name = "PolicyViolation";
  }
}

Step 2: Credit Budget Enforcement

class CrawlBudget {
  private usage = new Map<string, number>();
  private dailyLimit: number;

  constructor(dailyLimit = 5000) {
    this.dailyLimit = dailyLimit;
  }

  authorize(estimatedPages: number): void {
    const today = new Date().toISOString().split("T")[0];
    const used = this.usage.get(tod

Firecrawl Production Checklist

Overview

Pre-deployment validation checklist for applications using Firecrawl's scrape, crawl, map, and extract APIs. Covers credential management, crawl safety limits, error handling, monitoring, and rollback.

Output

Produce a launch receipt listing the approved deployment version, allowlist, crawl budget, evidence for completed controls, canary metrics, rollback owner, and unresolved exceptions. Exclude API keys and captured page content.

Error Handling

• Stop promotion on credential exposure, allowlist bypass, unbounded crawl behavior, or a threshold breach.
• Move exhausted jobs to a reviewed queue rather than retrying indefinitely.
• Revoke affected credentials and preserve only redacted incident evidence when data or access boundaries may have been violated.

Examples

Deploy a staging canary that crawls one approved synthetic target under a small credit budget. Simulate a throttle response and verify the job pauses safely, reports aggregate state, and can be rolled back without replaying pages.

Prerequisites

• Staging environment tested and passing
• Production API key from firecrawl.dev/app
• Monitoring infrastructure ready

Pre-Deployment Checklist

Credentials & Security

• [ ] Production FIRECRAWL_API_KEY in secure vault (not in code or .env)
• [ ] Key starts with fc- and is scoped to production
• [ ] Different API keys for dev/staging/production
• [ ] .env files in .gitignore
• [ ] Webhook secrets stored securely
• [ ] Git history scanned for leaked keys

Crawl Safety

• [ ] All crawlUrl calls have limit parameter set
• [ ] maxDepth configured to prevent unbounded crawling
• [ ] includePaths / excludePaths filters applied where appropriate
• [ ] Credit budget tracking implemented (daily limit alerts)
• [ ] No hardcoded URLs in production code

Error Handling

• [ ] 429 rate limit handling with exponential backoff
• [ ] 402 credit exhaustion handled gracefully (no crash)
• [ ] 401 auth failure logged and alerted
• [ ] Async crawl jobs have timeout with deadline
• [ ] Fallback from crawl to individual scrape on failure
• [ ] Empty markdown detection (JS rendering issues)

Monitoring & Alerting

• [ ] Scrape success/failure rate tracked
• [ ] Credit consumption monitored
• [ ] Crawl job completion rate tracked
• [ ] Alert on credit balance below threshold
• [ ] Alert on error rate > 5%
• [ ] Webhook delivery failures logged

Instructions

Step 1: Verify API

Firecrawl Reference Architecture

Overview

Production architecture for web scraping and content ingestion with Firecrawl. Covers three tiers: on-demand scraping, scheduled crawl pipelines, and real-time RAG ingestion. Uses all four Firecrawl endpoints: scrape, crawl, map, and extract.

Prerequisites

• A decision owner, approved target policy, retention classification, budget, and documented consumer access boundaries.
• Separate environments and scoped identities for requester, worker, storage, and downstream index.
• Synthetic fixtures, redacted telemetry, and a tested disable/rollback path for ingestion.

Output

Maintain an architecture record naming the selected pipeline, trust boundaries, data destinations, target and budget controls, retention rule, owner, and rollback switch. Captured pages and credentials are not architecture evidence.

Architecture Diagram

┌─────────────────────────────────────────────────────────┐
│                   Firecrawl Pipeline                     │
│                                                          │
│  ┌──────────┐  ┌──────────┐  ┌──────┐  ┌───────────┐   │
│  │ scrapeUrl│  │ crawlUrl │  │mapUrl│  │ extract   │   │
│  │ (1 page) │  │ (N pages)│  │(URLs)│  │ (LLM+JSON)│   │
│  └────┬─────┘  └────┬─────┘  └──┬───┘  └─────┬─────┘   │
│       │              │            │            │          │
│       ▼              ▼            ▼            ▼          │
│  ┌───────────────────────────────────────────────────┐   │
│  │            Content Processing Layer                │   │
│  │  Clean MD │ Validate │ Deduplicate │ Chunk        │   │
│  └─────────────────────┬─────────────────────────────┘   │
│                         │                                 │
│  ┌─────────────────────┴─────────────────────────────┐   │
│  │              Storage & Output                      │   │
│  │  Files │ Database │ Vector Store │ Search Index    │   │
│  └───────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Instructions

Step 1: Firecrawl Service Layer

// src/firecrawl/service.ts
import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// Single page scrape
export async function scrapePage(url: string) {
  return firecrawl.scrapeUrl(url, {
    formats: ["markdown"],
    onlyMainContent: true,
    waitFor: 2000,
  });
}

// Site-wide crawl with safety limits
export async function crawlSite(baseUrl: string, opts?: {
  maxPages?: number;
  paths?: string[];
  excludePaths?: string[];
}) {
  return firecrawl.crawlUrl(baseUrl, {
    limit: opts?.maxPages || 50,
    maxDepth: 3,
    includePaths: opts?.paths,
    excludePaths: opts?.excludePaths || ["/blog/*", "/news/*"],
    scra

Firecrawl Reliability Patterns

Overview

Production reliability patterns for Firecrawl scraping pipelines. Firecrawl's async crawl model, JS rendering, and credit-based pricing create specific reliability challenges: crawl jobs may time out, scraped content may be empty (bot detection, JS failures), and credits can be burned by runaway crawls. This skill covers battle-tested patterns for each.

Prerequisites

• A documented crawl allowlist, owner, credit budget, and retention policy for captured content.
• Aggregate telemetry for job state, queue age, error category, and budget use.
• A staging target or synthetic fixture for exercising timeouts, retries, and fallback logic.

Output

Emit a reliability receipt with crawl ID, approved target, attempt count, content-quality result, aggregate budget use, and terminal disposition. Redact page contents and credentials; move exhausted or policy-rejected jobs to a reviewed queue.

Instructions

Step 1: Robust Crawl with Timeout and Backoff

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

async function reliableCrawl(
  url: string,
  opts: { limit: number; paths?: string[] },
  timeoutMs = 600000
) {
  const job = await firecrawl.asyncCrawlUrl(url, {
    limit: opts.limit,
    includePaths: opts.paths,
    scrapeOptions: { formats: ["markdown"], onlyMainContent: true },
  });

  const deadline = Date.now() + timeoutMs;
  let pollInterval = 2000;

  while (Date.now() < deadline) {
    const status = await firecrawl.checkCrawlStatus(job.id);

    if (status.status === "completed") return status;
    if (status.status === "failed") {
      throw new Error(`Crawl failed: ${status.error}`);
    }

    await new Promise(r => setTimeout(r, pollInterval));
    pollInterval = Math.min(pollInterval * 1.5, 30000); // back off to 30s max
  }

  throw new Error(`Crawl timed out after ${timeoutMs}ms (job: ${job.id})`);
}

Step 2: Content Quality Validation

interface ScrapedPage {
  url: string;
  markdown: string;
  metadata: { title?: string; statusCode?: number };
}

function validateContent(page: ScrapedPage): {
  valid: boolean;
  reason?: string;
} {
  if (!page.markdown || page.markdown.length < 100) {
    return { valid: false, reason: "Content too short" };
  }

  if (page.metadata.statusCode && page.metadata.statusCode >= 400) {
    return { valid: false, reason: `HTTP ${page.metadata.statusCode}` };
  }

  const errorPatterns = [
    "access denied", "403 forbidden", "page not found",
    "captcha", "please verify", "enable javascript",
  ];
  const lower = page.markdown.toLowerCase();
  for (const pattern of

Firecrawl SDK Patterns

Overview

Production-ready patterns for Firecrawl SDK (@mendable/firecrawl-js / firecrawl-py). Covers singleton client, typed wrappers, retry with backoff, response validation, and reusable scraping service patterns.

Prerequisites

• @mendable/firecrawl-js installed
• Understanding of async/await patterns
• TypeScript strict mode recommended

Instructions

Step 1: Singleton Client with Configuration

// src/firecrawl/client.ts
import FirecrawlApp from "@mendable/firecrawl-js";

let instance: FirecrawlApp | null = null;

export function getFirecrawl(): FirecrawlApp {
  if (!instance) {
    if (!process.env.FIRECRAWL_API_KEY) {
      throw new Error("FIRECRAWL_API_KEY environment variable is required");
    }
    instance = new FirecrawlApp({
      apiKey: process.env.FIRECRAWL_API_KEY,
      ...(process.env.FIRECRAWL_API_URL
        ? { apiUrl: process.env.FIRECRAWL_API_URL }
        : {}),
    });
  }
  return instance;
}

Step 2: Typed Scrape Wrapper

// src/firecrawl/scrape.ts
import { getFirecrawl } from "./client";

interface ScrapeResult {
  url: string;
  title: string;
  markdown: string;
  links: string[];
  scrapedAt: string;
}

export async function scrapePage(
  url: string,
  options?: { waitFor?: number; includeLinks?: boolean }
): Promise<ScrapeResult> {
  const firecrawl = getFirecrawl();
  const formats: string[] = ["markdown"];
  if (options?.includeLinks) formats.push("links");

  const result = await firecrawl.scrapeUrl(url, {
    formats,
    onlyMainContent: true,
    ...(options?.waitFor ? { waitFor: options.waitFor } : {}),
  });

  if (!result.success) {
    throw new Error(`Scrape failed for ${url}: ${result.error}`);
  }

  return {
    url: result.metadata?.sourceURL || url,
    title: result.metadata?.title || "",
    markdown: result.markdown || "",
    links: result.links || [],
    scrapedAt: new Date().toISOString(),
  };
}

Step 3: Retry with Exponential Backoff

// src/firecrawl/retry.ts
export async function withRetry<T>(
  operation: () => Promise<T>,
  config = { maxRetries: 3, baseDelayMs: 1000, maxDelayMs: 30000 }
): Promise<T> {
  for (let attempt = 0; attempt <= config.maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error: any) {
      if (attempt === config.maxRetries) throw error;

      const status = error.statusCode || error.status;
      // Only retry on rate limits (429) and server errors (5xx)
      if (status && status !== 429 && status < 500) throw error;

      const delay = Math.min(
        config.baseDelayMs * Math.pow(2, attempt) + Math.random() * 500,
        

Firecrawl Security Basics

Overview

Security best practices for Firecrawl API keys, webhook signature verification, and scraped content handling. Firecrawl API keys start with fc- and grant full access to scrape, crawl, map, and extract endpoints — protecting them is critical.

Output

Keep a security review receipt with secret-manager reference, identity scope, approved target policy, webhook verification outcome, access-review date, and revocation/rotation owner. Never include the key, captured pages, or sensitive request headers.

Examples

Use a synthetic target with a least-privilege staging key. Confirm that the handler rejects an invalid signature without revealing comparison data, then revoke the test key and verify the request is denied before granting any production access.

Prerequisites

• Firecrawl API key
• Understanding of environment variables
• Webhook endpoint (if using async crawl callbacks)

Instructions

Step 1: Secure API Key Storage

# .env (NEVER commit to git)
FIRECRAWL_API_KEY=fc-your-api-key-here

# .gitignore — add these patterns
echo -e "\n.env\n.env.local\n.env.*.local" >> .gitignore

// Validate key exists before creating client
import FirecrawlApp from "@mendable/firecrawl-js";

if (!process.env.FIRECRAWL_API_KEY?.startsWith("fc-")) {
  throw new Error("FIRECRAWL_API_KEY must be set and start with 'fc-'");
}

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY,
});

Step 2: Verify Webhook Signatures

Firecrawl signs webhook payloads with HMAC-SHA256 via the X-Firecrawl-Signature header.

import crypto from "crypto";

function verifyWebhookSignature(
  payload: string,
  signature: string,
  secret: string
): boolean {
  const expected = crypto
    .createHmac("sha256", secret)
    .update(payload)
    .digest("hex");

  // Timing-safe comparison prevents timing attacks
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expected)
  );
}

// Express webhook handler with verification
app.post("/webhooks/firecrawl", (req, res) => {
  const signature = req.headers["x-firecrawl-signature"] as string;
  const rawBody = JSON.stringify(req.body);

  if (!verifyWebhookSignature(rawBody, signature, process.env.FIRECRAWL_WEBHOOK_SECRET!)) {
    console.error("Invalid webhook signature — rejecting");
    return res.status(401).json({ error: "Invalid signature" });
  }

  // Process verified webhook
  const { type, data } = req.body;
  console.log(`Verified webhook: ${type}`);
  res.status(200).json({ received: true });
});

Step 3: Separate Keys per Environment