Firecrawl Layered Troubleshooting

Overview

Diagnose a failing integration without leaking API keys or scraped content. Change one variable at a time and distinguish Firecrawl transport failures from target-site behavior and downstream parsing defects.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Treat the v2 endpoint reference and the installed SDK types as the contract. SDK calls return the data object directly, REST errors normally return success false plus an error string, and completed crawl or batch results may require pagination. Use metadata.statusCode to distinguish a captured origin error page from a Firecrawl request failure.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Record the client package, resolved version, runtime, endpoint family, request class, opaque request or job ID, and first failing timestamp. Redact URLs when their paths or queries are sensitive.
2. Reproduce with one approved public or synthetic URL and the smallest output format. If that succeeds, the credential and base route are probably sound; do not infer that the target is healthy.
3. Compare the failing request with the current v2 schema. Remove optional actions, headers, proxy choices, JSON extraction, and cache overrides one at a time.
4. For async work, inspect queue status, job state, pagination cursor, crawl errors, and terminal status. Do not call a running job failed only because the first status page has no documents.
5. Classify the result as request validation, authentication, credits, rate/concurrency, rendering, origin response, extraction/schema, retention-policy conflict, or downstream processing.
6. Retry only statuses marked retryable by the official error catalog. Honor Retry-After when present, use bounded jittered backoff, and retain the original idempotency decision.
7. Reintroduce options individually, verify the smallest fix on a canary, and produce a redacted evidence receipt before rollout.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this s…

Firecrawl Architecture Selection

Overview

Turn workload, freshness, compliance, throughput, and recovery requirements into an explicit architecture decision. Prefer the smallest Firecrawl surface that meets the outcome.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Scrape is a synchronous single-resource primitive; crawl and batch scrape support asynchronous jobs, pagination, webhooks, and status retrieval; map discovers URLs without retrieving page content; search discovers external sources; parse handles local file bytes. The default self-hosted Compose stack does not provide every Cloud capability and is not a production security design.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Inventory the source domains, ownership or authorization basis, expected page volume, freshness target, output formats, data classification, and recovery objective.
2. Choose scrape for bounded single pages, map plus selective scrape for curated URL sets, batch scrape for known URL collections, and crawl for recursive site discovery.
3. Choose polling, WebSocket streaming, or signed webhooks for asynchronous delivery based on network topology and recovery needs.
4. Decide Cloud versus self-hosting using required capabilities, data flows, operator staffing, availability target, and upgrade ownership. Record unsupported self-hosted features explicitly.
5. Place a policy gateway before Firecrawl for domain authorization, request shaping, budgets, key selection, and audit receipts. Keep content storage and indexing downstream.
6. Define queue limits, explicit crawl limits, retention/cache choices, idempotency keys, pagination ownership, and degraded modes.
7. Validate the chosen variant with one approved canary and document failure, retry, cancellation, and rollback paths.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before selecting self-hosting, introducing a new external provider, permitting au…

Firecrawl CI Contract

Overview

Prove request construction, response handling, pagination, redaction, retries, and policy enforcement without spending credits or exposing production targets in routine pull-request CI.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Pin a reviewed Firecrawl v2 OpenAPI or SDK commit for contract tests. Cover SDK direct-data responses separately from REST success/data envelopes. Treat retryability, webhook event names, and pagination cursors as contract assertions rather than loose snapshots.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Inspect the repository runtime, package manager, existing test framework, generated-client policy, CI trust model, and secret-scanning controls.
2. Create synthetic fixtures for scrape documents, captured origin error pages, paginated crawl and batch results, validation failures, 402, 403, 429, and retryable server errors.
3. Add unit tests for request shaping, explicit crawl limits, URL policy, redaction, cache and retention choices, Retry-After parsing, bounded retries, cancellation, and idempotency.
4. Pin the v2 contract source and fail on incompatible endpoint, auth, required-field, event, or response-shape drift. Review intentional changes before updating the pin.
5. Run static analysis, unit tests, contract tests, secret scanning, and artifact inspection on untrusted pull requests with fake credentials only.
6. If live validation is justified, place one read-only synthetic canary behind a protected environment, disable it for forks, cap its credits and duration, and discard content bodies.
7. Make required failures fail closed and emit a receipt with the contract pin and exact test result.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before adding dependencies, updating the contract pin, enabling networked CI, granting a CI secret, changing required checks, or retaining a live response.

Output

Firecrawl Error Triage

Overview

Translate an error into a safe next action using Firecrawl's published error catalog. Do not retry every non-success response and do not confuse a returned document with an origin error status for an API failure.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Non-2xx REST responses normally contain success false and an error string, sometimes details or code. The current retryable status set is 408, 429, 500, 502, 503, and 504; 422 is conditional. Authentication, insufficient-credit, restriction, and request-shape failures require correction rather than blind retry.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Capture the HTTP status, error string, optional code/details, request or job ID, operation, timestamp, SDK version, and a redacted request shape.
2. Check metadata.statusCode on returned documents. A captured 403 or 404 page consumes a result path and should not be handled like an API transport error.
3. For 400 or 422, validate the request against the current endpoint schema and simplify JSON extraction requirements before retrying.
4. For 401, verify secret injection and key selection without printing the key. For 402, stop work and route the credit or billing decision to the owner.
5. For 403, inspect plan entitlement, team ownership, endpoint/format key restrictions, IP restrictions, or policy controls; never bypass them with another uncontrolled key.
6. For 429, distinguish rate from concurrency pressure, honor Retry-After when available, reduce producers, and use queue status evidence.
7. For retryable timeouts or server failures, use bounded exponential backoff with jitter, then escalate with redacted IDs and timestamps.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before rotating credentials, enabling pay-as-you-go, upgrading a plan, relaxing a key/IP restriction, increasing concurrency, or sharing target details with support…

Firecrawl Scrape and Crawl Workflow

Overview

Use scrape for one URL and crawl for recursive discovery. Keep the authorization basis, scope, cost ceiling, result completeness, and storage decision explicit.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

The current Node client is Firecrawl from the firecrawl package; Python uses Firecrawl from firecrawl-py. Use scrape for a single URL, crawl as the waiter, startCrawl for asynchronous submission, getCrawlStatus for status and pagination, and cancelCrawl for cancellation. REST uses the /v2 routes and Bearer authentication.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Confirm the target is authorized, normalize its origin, and document allowed paths, excluded paths, subdomain/external-link policy, desired formats, and freshness.
2. Create the client from FIRECRAWL_API_KEY for authenticated work. Keep the key in a secret manager and never place it in source, arguments, logs, or generated examples.
3. For one page, call scrape with only the required formats and options. Validate the returned document, metadata.sourceURL, metadata.statusCode, and required content fields.
4. For a site, set an explicit crawl limit and path/depth policy. Use crawl when blocking is acceptable or startCrawl when another worker owns status and cancellation.
5. Retrieve every required result page. Preserve the next cursor or URL until pagination is complete, and record partial completion separately from terminal success.
6. Deduplicate by canonical source URL and content hash, validate output quality, and write only approved fields to the downstream store.
7. Emit counts, creditsUsed when returned, cache state, failures, policy version, and rollback/cancellation outcome without logging page bodies.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before authenticated-page scraping, external-link traversal, increasing scope or limit, enabling sensitive header…

Firecrawl Discovery and Structured Extraction

Overview

Discover first, select deliberately, and retrieve only the sources required for the outcome. Separate URL discovery from content acquisition and validate all model-produced JSON.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Use map for site URLs, search for web discovery, batchScrape or startBatchScrape for known URL sets, and parse for local/non-public file bytes. In v2, structured extraction is a json format object with prompt and optional JSON Schema; the legacy extract format name is not the current scrape contract.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Define the research question, approved domains, recency, maximum results/pages, required fields, JSON Schema, and evidence-retention policy.
2. Use map for an owned site or search for broader discovery. Store normalized URLs and source metadata, not content, until selection policy passes.
3. Filter duplicates, unsupported schemes, disallowed domains, and unnecessary query variants. Require review for sensitive or authenticated sources.
4. Choose batchScrape for known URLs and startBatchScrape when work must be asynchronous. Retrieve all required pages through getBatchScrapeStatus pagination.
5. For local PDF, DOCX, XLSX, HTML, or other supported bytes, use parse rather than inventing a public URL. Apply file-size and classification gates first.
6. For typed extraction, request the v2 json format with the narrowest schema and prompt. Treat output as untrusted model data and validate types, constraints, provenance, and completeness.
7. Return discovery, selection, retrieval, validation, cost, and failure receipts as separate stages so partial results are auditable.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before broad web search, uploading non-public files, using LLM-backed extraction on sensitive content, increasing batch size, or retaining raw source cont…

Firecrawl Credit and Spend Control

Overview

Optimize against measured business value rather than assumed one-credit requests. Endpoint and option costs differ, modifiers can stack, and crawl or batch charges arrive as pages complete.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Use the current billing page and Credit Usage APIs as the pricing authority. Base scrape/crawl page costs, search/result charges, JSON or other format modifiers, ZDR, parse behavior, Interact minutes, and lockdown outcomes can differ. Polling status does not itself consume credits, while asynchronous page processing can make usage appear later.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Capture the current billing policy, plan, pay-as-you-go state, per-key spend limits, monthly cap, and a baseline by operation and workload class.
2. Attribute usage to an owner, environment, source policy, endpoint, requested formats, page counts, cache behavior, and success category. Never estimate solely from request count.
3. Set explicit crawl and batch bounds. Use map plus selective retrieval when discovery is cheaper than collecting every page.
4. Request only required formats and expensive options. Validate whether JSON extraction, prompt-injection checks, PDF parsing, ZDR, audio/video, or browser interaction earns its added cost.
5. Use maxAge and cache policy only when the freshness SLA permits it; use storeInCache false or ZDR when retention requirements outweigh savings.
6. Add preflight budget checks, alert thresholds, per-key controls where available, and a fail-closed response when the approved ceiling is reached.
7. Run a bounded canary, compare quality-adjusted cost with the baseline, and roll back if savings reduce completeness or violate policy.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before enabling or raising pay-as-you-go, changing plan, increasing a per-key spend limit, trading retention …

Firecrawl Content Data Handling

Overview

Treat scraped pages, metadata, screenshots, file parses, and model-extracted JSON as untrusted external data. Preserve provenance while storing only what the approved use case requires.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

SDKs return document data directly; REST returns it under data. metadata.sourceURL and metadata.statusCode are essential provenance and quality fields. storeInCache, zeroDataRetention, lockdown, screenshots, raw HTML, file upload, and persistent browser profiles create materially different data-handling obligations.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Document source authorization, data classification, permitted fields, purpose, storage region, retention period, deletion path, and downstream consumers.
2. Validate the document envelope and origin status before processing. Reject unsupported content types, captured error pages, oversized fields, and missing provenance.
3. Normalize canonical URLs, strip fragments and disallowed query material, and compute content hashes for deduplication without treating the hash as authorization.
4. Sanitize HTML/Markdown for the destination, neutralize active content, and keep scraped instructions outside trusted agent/system context.
5. Validate JSON extraction against the declared schema and business constraints. Preserve source links and confidence/review state with every record.
6. Separate raw quarantine, approved normalized content, embeddings/indexes, and audit receipts. Encrypt sensitive data and enforce least-privilege access.
7. Implement expiry, source deletion, legal hold, reprocessing, and downstream tombstone tests; verify disposal with counts and hashes.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before storing raw HTML, screenshots, authenticated content, personal data, uploaded files, persistent profiles, or extending retention and downstream u…

Firecrawl Privacy-Safe Support Evidence

Overview

Produce structured evidence that can reproduce or classify a failure without dumping the repository, environment, request headers, URLs, or response bodies.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Useful evidence includes SDK/runtime version, v2 operation, timestamps, opaque request/job/webhook IDs, HTTP status, documented error/code, queue/job state, pagination state, option names, and content-free metrics. Firecrawl support does not need a broad tarball or the API key.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Confirm the incident owner, destination, approved evidence fields, retention window, and whether target URLs themselves are sensitive.
2. Collect package/runtime versions, operating context, v2 endpoint or SDK method, UTC timestamps, and a normalized option-name list.
3. Record status, error/code/details after redaction, response headers on an allowlist, request/job/webhook IDs, queue state, pagination cursor presence, counts, sizes, timings, and metadata.statusCode distribution.
4. Represent URLs with approved host labels or salted hashes; never include credentials, cookies, Authorization, custom headers, page content, screenshots, uploaded bytes, prompts, or extracted values.
5. Run secret, token-pattern, URL-query, email, and content-leak scans over the proposed bundle. Manually inspect the exact files.
6. Create a manifest with file hashes, collector version, redaction policy, known omissions, recipient, expiry, and deletion owner.
7. Require recipient approval, transmit through the approved channel, and record deletion or ticket linkage.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before collecting from production, including a hostname or request option value, sharing outside the organization, or extending bundle retention.

Output

Return a small manifest and redacted structured evidence …

Firecrawl Production Deployment

Overview

Deploy the application client and the Firecrawl service as separate responsibilities. Cloud integrations need a protected key and egress policy; self-hosting adds databases, queues, browsers, storage, authentication, upgrades, and recovery.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Cloud clients use the v2 API and secret-managed FIRECRAWL_API_KEY. The official self-host guide's Compose example is an evaluation baseline with authentication disabled and without durable storage, TLS, high availability, or every Cloud capability. Pin the reviewed release and its own Compose contract; do not copy a floating main configuration into production.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Inventory the application release, Firecrawl Cloud or self-host decision, required features, environments, domains, data flows, SLOs, and rollback owner.
2. For Cloud, inject the key from the platform secret manager, restrict outbound destinations, choose retention/cache policy, and prevent request or response bodies from application logs.
3. For self-hosting, pin a verified release and review its Compose, SELF_HOST, and feature-support documentation. Map every optional provider and outbound flow.
4. Before exposure, add supported authentication, network controls, TLS, durable PostgreSQL/Redis/RabbitMQ storage where required, backups, restore tests, monitoring, capacity limits, and upgrade rollback.
5. Deploy a no-traffic revision, verify process readiness separately from one approved functional v2 scrape, then run a bounded content-free canary.
6. Observe errors, queue pressure, latency, credit or capacity use, and output-quality metrics. Promote gradually with explicit stop thresholds.
7. Record the release/image digest, configuration hashes, canary evidence, approvals, and rollback result.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before expo…

Firecrawl Enterprise Access Governance

Overview

Translate Firecrawl's actual team and key controls into a least-privilege operating model. Do not invent fine-grained roles that the dashboard does not provide.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

The dashboard documents two team roles, Admin and Member, with administrative actions reserved for Admins. Enterprise key restrictions can enforce per-key endpoint and output-format allowlists; team-scoped IP restrictions can restrict authenticated origins; SSO is an enterprise capability. Empty restriction lists mean unrestricted, and changes can take time to propagate.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Inventory teams, Admins, Members, API keys, service owners, environments, permitted endpoints/formats, source networks, spend limits, SSO state, and break-glass access.
2. Remove shared human keys from services. Assign each workload an owned key and secret-manager path with a documented purpose and rotation/revocation procedure.
3. Use the smallest team role. Keep invitations, member removal, role changes, key administration, billing, and enterprise-control changes behind Admin review.
4. For eligible enterprise keys, configure non-empty endpoint and format allowlists; remember that an empty list does not restrict that dimension.
5. Apply team-scoped IP restrictions only after inventorying every legitimate egress address and testing a break-glass path from an approved network.
6. Align SSO, offboarding, key rotation, SIEM evidence, spend limits, and quarterly access review with the organization's identity controls.
7. Test allow, deny, propagation, revoked-key, removed-user, unexpected-IP, and disaster-recovery cases and retain only redacted evidence.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require two-person approval for Admin grants, SSO or IP changes, break-glass use, restriction removal, new unrestricted keys, an…

Firecrawl v2 First Verified Scrape

Overview

Prove one controlled page can be retrieved through the current v2 surface. A successful transport is not enough: verify the document came from the expected URL and did not capture an origin error page.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

For Node, install firecrawl, import the named Firecrawl client, and call scrape. For Python, install firecrawl-py, import Firecrawl, and call scrape. REST uses POST https://api.firecrawl.dev/v2/scrape with Bearer authentication. The SDK returns the document data directly; REST wraps it in success and data.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Choose an approved public or synthetic URL with stable, non-sensitive content and record the expected title or marker.
2. Install the official SDK with the repository's locked package manager, or use REST. Record the resolved package version without changing unrelated dependencies.
3. Inject FIRECRAWL_API_KEY from the secret manager for authenticated use. Keyless evaluation is allowed only for the operations and limits currently documented.
4. Request markdown only and avoid actions, custom headers, screenshots, raw HTML, location, proxy overrides, and retention changes in the first test.
5. Assert the expected metadata.sourceURL, a successful metadata.statusCode, non-empty markdown, and the expected marker. Do not print the full body.
6. Record latency, cacheState and creditsUsed when returned, SDK/API surface, and redacted result size.
7. Remove temporary output, keep the minimal test as a synthetic smoke if useful, and document the next approved workflow.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before using a private URL, supplying site credentials or headers, retaining the response, or turning the example into a recurring job.

Output

Return the exact install/runtime surface, approved URL classification, assertion…

Firecrawl Incident Response

Overview

Stabilize the affected workflow, preserve privacy-safe evidence, and restore service through a tested degraded mode or rollback. Separate Firecrawl service health, target-origin behavior, and internal pipeline failures.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Firecrawl's error catalog defines retryability; async operations expose job/status/cancellation surfaces; queue status helps distinguish capacity pressure; status polling remains the recovery path when webhooks are delayed or exhausted. A crawl webhook set does not include a crawl.failed event in the current documented event list.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Declare incident owner, severity, affected operation/environment, start time, customer impact, data risk, approved communication channel, and next update time.
2. Pause or bound producers before investigating if retries, crawl scope, or pay-as-you-go could amplify cost or target load.
3. Check internal deployments and dependencies, Firecrawl service evidence, credentials, credits, team restrictions, queue/concurrency, job state, webhook delivery, and target-origin status.
4. Classify the failure with the official error catalog. Retry only documented retryable classes, honor Retry-After, and cap attempts.
5. Choose a reversible mitigation: reduce concurrency, narrow limits, switch to polling, serve last known approved content, disable an expensive option, or roll back the application release.
6. Verify recovery with a synthetic or approved canary and confirm queue drain, error rate, output quality, data integrity, and spend stabilization.
7. Communicate resolution, retain redacted evidence, and create owned corrective actions for detection, prevention, runbook, and rollback gaps.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require incident-command approval before key rotation, plan or pay-as-you-go changes, traffic fai…

Firecrawl SDK Installation and Authentication

Overview

Make the client dependency, v2 surface, identity owner, and secret path explicit. Do not preserve the legacy method surface merely because an old package still resolves.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Current first-party docs install the Node package as firecrawl and construct the named Firecrawl client; the underlying published SDK is maintained in firecrawl/firecrawl. Python installs firecrawl-py and imports Firecrawl. Top-level clients default to v2, while feature-frozen v1 compatibility is separate. REST Cloud requests use Bearer keys.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Inspect the repository language, runtime, lockfile, package manager, current Firecrawl dependency/imports, endpoint family, and secret conventions.
2. Confirm the official package from Firecrawl's docs and source, then add it through the existing package manager with a reviewed version range and lockfile change.
3. Replace legacy FirecrawlApp/scrapeUrl-style usage only as part of an explicit v2 migration; do not mix old and new methods under one untyped wrapper.
4. Create one client per process boundary using FIRECRAWL_API_KEY from the approved secret manager. Permit a custom API URL only for an explicitly selected self-hosted environment.
5. Validate configuration at startup using presence and shape checks without logging the key. Associate the secret with an owning team, workload, environment, and rotation record.
6. Run offline construction/type tests, then one approved minimal scrape if network validation is authorized. Assert provenance and metadata.statusCode without logging content.
7. Record the resolved package version, source, lockfile diff, auth mode, test evidence, and rollback.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before adding the dependency, changing its major version, creating or rotating a key, en…

Firecrawl Integration Pitfall Review

Overview

Find high-impact failure modes before production. Tie every finding to current Firecrawl behavior and repository evidence instead of applying a generic checklist.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Common current hazards include legacy scrapeUrl/crawlUrl method names, implicit crawl scope, confusing API success with origin status, dropping pagination, retrying 4xx errors, assuming one credit per request, trusting scraped instructions, enabling cache on sensitive data, skipping webhook HMAC verification, and exposing the self-host quickstart.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Locate Firecrawl dependencies, imports, wrapper clients, REST paths, config, queues, webhooks, stores, tests, and deployment files.
2. Flag legacy v0/v1 routes or FirecrawlApp-style methods unless they are intentionally isolated behind the documented feature-frozen compatibility surface.
3. Find every crawl, batch, search, agent, and browser call. Require explicit scope, credit/time limits, cancellation ownership, and environment policy.
4. Verify SDK versus REST response handling, metadata.statusCode checks, complete pagination, terminal-state handling, and idempotent webhook processing.
5. Compare retry logic with the official error catalog. Reject retries for auth, credits, restrictions, validation, and other non-retryable failures.
6. Trace scraped or extracted content into logs, prompts, stores, and actions. Require sanitization, provenance, prompt-injection boundaries, retention, and deletion.
7. Rank findings by exploitability and impact, propose minimal fixes and regression tests, and verify the corrected paths.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before changing dependencies, expanding crawl scope, enabling raw formats/actions, weakening cache or retention safeguards, or auto-fixing production code.

Output

Firecrawl Capacity and Load Validation

Overview

Find the safe operating envelope with synthetic or explicitly authorized targets. A load test must not become an uncontrolled scrape campaign or consume unapproved credits.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Firecrawl separates per-team requests-per-minute limits from concurrent browser capacity. Work beyond browser capacity can queue, queue time counts against request timeout, and queue status exposes availability. Crawl and batch calls also accept maxConcurrency, while a crawl delay forces concurrency to one.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Define the approved target set, environment, maximum requests/pages/credits, concurrency steps, duration, stop thresholds, and owner. Prefer a controlled synthetic origin.
2. Measure a single-worker baseline for submission latency, completion latency, throughput, queue time, error classes, origin status, output size, quality, and credit usage.
3. Increase producer concurrency in small steps below the current plan/team limits. Observe Firecrawl queue status and the application's own backlog separately.
4. For crawl or batch, test maxConcurrency and explicit limits; do not assume client request concurrency equals page-processing concurrency.
5. Exercise 429 rate pressure, concurrency queuing, Retry-After handling, timeout, cancellation, partial pagination, and backpressure using synthetic responses before live load.
6. Stop on error, spend, target-load, latency, queue-age, or quality thresholds. Drain or cancel work according to the test plan.
7. Report the sustainable envelope with headroom, bottleneck evidence, configuration, cost, and rollback; never publish captured bodies.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before live load, increasing credits or plan capacity, using third-party targets, changing target delay/concurrency, or extending the test wind…

Firecrawl Local Development Loop

Overview

Keep routine development offline and deterministic. Use self-hosting only when behavior must be exercised end to end, and treat the official Compose quickstart as a local evaluation stack.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

The current self-host guide pins a specific verified release, exposes the API on localhost port 3002, and disables database authentication for its first-run baseline. It explicitly lacks production authentication, durable storage, TLS, high availability, and several Cloud capabilities. Its release and Compose contract must be re-read before use.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Inspect the application runtime, official SDK version, wrapper boundary, test framework, and operations the integration actually uses.
2. Define typed adapter interfaces for scrape, crawl submission/status, batch, map/search, and any other required methods so tests do not mock internals.
3. Create synthetic fixtures for direct SDK documents, REST envelopes, origin error pages, paginated jobs, throttling, credits, policy denial, invalid extraction, and webhook signatures.
4. Run unit and contract tests against fakes by default. Prohibit production keys, real customer URLs, captured bodies, and external network access.
5. When end-to-end behavior is necessary, follow the current official self-host guide at its pinned release on a trusted local network. Do not improvise a one-container substitute.
6. Verify readiness separately from one functional /v2/scrape, record unsupported features, and tear down or isolate the evaluation stack after use.
7. Keep lockfiles, fixtures, contract pins, and local setup instructions current; add a regression before fixing each discovered defect.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before starting Docker services, downloading a new release, using Cloud from local tests, capturing…

Firecrawl Workload Migration

Overview

Replace a scraper only after defining observable equivalence. Preserve the old adapter until the new path meets content, policy, performance, and recovery gates.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Firecrawl can replace rendering, crawl discovery, mapping, search, parsing, and typed extraction, but one endpoint is not equivalent to every legacy workflow. Scrape, crawl, map, batch, parse, JSON extraction, actions, Browser, and Cloud/self-hosted capabilities have different contracts and costs.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Inventory legacy sources, login/session behavior, selectors/actions, discovery rules, formats, parsers, retries, proxies, schedules, storage, compliance controls, metrics, and operating cost.
2. Create a source-to-Firecrawl capability map. Mark unsupported, Cloud-only, self-host-dependent, behavior-changing, and policy-sensitive requirements.
3. Define an adapter contract and golden synthetic corpus with provenance, normalized content hashes, required-field coverage, latency, cost, and accepted variance.
4. Implement the smallest v2 path behind a feature flag. Keep target authorization, explicit scope/limits, cache/retention, cancellation, and error typing visible.
5. Shadow both paths on approved canaries, compare content and metadata without duplicating downstream writes, and investigate meaningful differences.
6. Shift traffic in stages only after parity, security, spend, load, and rollback gates pass. Keep the old path available through the observation window.
7. Remove legacy code and secrets only after rollback expiry, archive the decision and evidence, and verify dependency and data cleanup.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before replaying authenticated sessions, changing target behavior or proxies, enabling Cloud-only processing, increasing spend, shifting traffic, or deleti…

Firecrawl Environment Separation

Overview

Prevent a test from consuming production credits, crawling production targets, or writing captured content into the wrong store. Environment identity must be explicit and fail closed.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Cloud keys are team-scoped credentials and team limits are shared across keys, so separate keys alone do not guarantee capacity isolation. Key restrictions, spend limits, IP restrictions, distinct teams/accounts where appropriate, and an application policy gateway provide different isolation layers. Self-hosted evaluation has a separate capability and security contract.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Inventory each environment's owner, team/account, key, secret path, egress, allowed domains, endpoints/formats, spend ceiling, queue capacity, retention, store, and alert destination.
2. Create distinct secret-manager entries and workload identities. Validate that configuration names the environment and refuses missing or cross-environment values.
3. Apply environment-specific domain policy, explicit crawl/batch limits, endpoint/format restrictions where available, and non-production credit ceilings.
4. Keep production targets and captured fixtures out of developer machines and pull-request CI. Use synthetic fakes by default and a protected staging tenant for live tests.
5. Route telemetry and audit receipts with an environment label while redacting keys, URLs, custom headers, prompts, and bodies.
6. Test cross-environment secret, target, store, webhook, and queue mistakes; assert they fail before a Firecrawl request or downstream write.
7. Document promotion as configuration and evidence review, not copying a .env file; test independent rotation and rollback.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before creating a team/key, adding a production target to staging, sharing capacity, chang…

Firecrawl Operational Observability

Overview

Measure the service as a pipeline: submission, Firecrawl processing, origin result, pagination, validation, storage, and freshness. Transport success alone does not prove usable content.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Use Credit Usage endpoints and dashboard activity for spend, Queue Status for capacity, job/status APIs for async progress, webhookId for delivery identity, metadata.statusCode for origin response, and creditsUsed where returned. Metric labels must not contain raw URLs, page content, prompts, keys, or unbounded IDs.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Define SLIs and SLOs for accepted-result success, end-to-end latency, freshness, queue age, pagination completeness, origin-status distribution, output quality, webhook recovery, and spend.
2. Instrument each stage with low-cardinality operation, environment, source-class, terminal-state, retry-class, and policy-version labels.
3. Record opaque request/job/webhook IDs only in access-controlled traces. Hash or classify URLs and exclude custom headers, bodies, extracted values, screenshots, and secrets.
4. Poll or stream async state within bounds, count every result page, and reconcile submitted, processed, accepted, rejected, stored, and deleted totals.
5. Collect queue and credit evidence at a cadence that respects API limits. Account for delayed asynchronous billing before alerting on unexplained deltas.
6. Alert on user-impacting SLO burn, sustained queue pressure, 402/403/429 shifts, origin errors, webhook retry exhaustion, schema rejection, and freshness lag.
7. Run synthetic failure and recovery tests, verify dashboards against raw aggregate counts, and document alert ownership and runbook links.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before adding high-cardinality labels, retaining target identifiers, exporting telemetry to a new pro…

Firecrawl Performance Tuning

Overview

Tune the whole path from submission to accepted downstream record. Preserve freshness, completeness, target policy, and cost while changing one control at a time.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Firecrawl v2 cache behavior is controlled by maxAge/minAge and related options; current defaults and effects belong to the scrape documentation. Crawl and batch provide waiter and asynchronous paths, pagination can dominate retrieval time, and queue wait consumes request timeout. maxConcurrency affects page processing but remains bounded by team capacity.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Define an SLO and baseline for submission, queue, processing, pagination, validation, storage, freshness, accepted-result rate, and credits.
2. Segment by operation, target class, format, cache state, origin status, content size, and async job size without putting raw URLs or content in metrics.
3. Remove unnecessary formats, actions, waits, screenshots, raw HTML, JSON extraction, and over-broad crawl scope before adding concurrency.
4. Set maxAge or cache-only behavior only when the freshness SLA permits it. Verify cacheState and quality rather than assuming a cache hit is acceptable.
5. Use batch for known URL sets, async submission for long work, and complete pagination efficiently. Bound maxConcurrency below observed team capacity.
6. Tune one factor per canary, compare tail latency and accepted-output quality, and watch rate, concurrency, queue, origin, and credit effects.
7. Promote only improvements that meet all guardrails; retain baseline, candidate, and rollback evidence.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before reducing freshness, enabling storage/cache on sensitive sources, increasing concurrency or scope, changing formats, or accepting lower completeness.

Output

Return the baseline, bottleneck att…

Firecrawl Collection Policy Guardrails

Overview

Put enforceable policy before every Firecrawl request and downstream action. Provider capability does not establish permission to collect, retain, or reuse a source.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Firecrawl crawl supports scope controls, explicit limits, delay, maxConcurrency, and an enterprise robotsUserAgent option. Enterprise controls include endpoint/format key restrictions, IP restrictions, threat protection, and SIEM integration. Request options such as headers, actions, proxy, cache, ZDR, lockdown, and raw formats have separate risk and availability implications.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Create a versioned source registry with owner, authorization basis, allowed purpose, domains/paths, robots and terms review, data classes, retention, and expiry.
2. Canonicalize URLs before policy evaluation. Deny unsupported schemes, embedded credentials, private/link-local addresses, disallowed ports, redirects outside scope, and lookalike hosts.
3. Allow only required operations, formats, actions, headers, locations, proxy modes, and crawl settings. Set explicit page, time, credit, and concurrency ceilings.
4. Apply provider-side key, IP, threat-protection, and audit controls where entitled, but keep application policy authoritative and fail closed if it is unavailable.
5. Treat retrieved content as untrusted data. Separate it from system instructions, validate extracted JSON, sanitize active content, and gate downstream actions.
6. Log policy version, decision, source class, request class, limits, and opaque IDs without keys, URLs with secrets, headers, prompts, or bodies.
7. Test redirect, DNS rebinding, wildcard, internationalized-domain, restriction bypass, budget, retention, and prompt-injection cases before rollout.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require policy-owner and legal/security review bef…

Firecrawl Production Readiness Gate

Overview

Produce evidence for a go/no-go decision. A checked box without an artifact, test, owner, or exact release reference is not evidence.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

The release must use current v2 endpoints or an explicitly isolated legacy path, the official SDK surface, complete async pagination, documented retryability, signed webhooks, explicit crawl/batch limits, current billing controls, and approved cache/retention behavior. Self-hosted releases require their own production controls beyond Compose evaluation.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Pin the application release, dependency/lockfile, Firecrawl contract source, configuration hashes, environment, owners, target policy, and rollback candidate.
2. Verify secret-managed identity, team ownership, key/format/endpoint/IP restrictions where available, rotation, fork-CI isolation, and no secret in source or artifacts.
3. Verify domain authorization, URL canonicalization, explicit scope/limits, allowed formats/actions/headers/proxies, retention/cache/ZDR choices, and deletion.
4. Test SDK/REST response handling, origin status, pagination, cancellation, 402/403/429, Retry-After, bounded server retries, partial results, webhook HMAC/idempotency, and downstream deduplication.
5. Verify cost ceilings, pay-as-you-go policy, queue/concurrency headroom, dashboards, SLO alerts, incident ownership, vendor escalation, and degraded modes.
6. Run one bounded staging canary with approved content-free assertions; then test rollback without replaying or duplicating accepted pages.
7. Issue go only when required gates pass and approvals are attached; record exceptions with owner and expiry.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require named security/data/product/operations approval for their controls and release-owner approval for go-live, plan/spend changes, exceptions, and …

Firecrawl Rate, Concurrency, and Queue Control

Overview

Control request production and page-processing concurrency as different resources. Derive limits from the current team plan and live queue evidence, not copied constants.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Firecrawl documents per-team request-per-minute limits and concurrent browser limits. All keys on one team share rate counters. Work beyond browser capacity can queue, queue wait counts toward timeout, and excessive queued jobs can return 429. The error catalog says to honor Retry-After when present; batch shares crawl limits.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Read the current rate-limit documentation and team configuration for the exact operation; record the evidence timestamp without embedding plan numbers in code.
2. Model submission RPM, in-flight browser work, Firecrawl queue, application backlog, target-origin limits, and credit ceiling separately.
3. Use a bounded token bucket or equivalent for submissions and a bounded worker pool for jobs. Keep crawl/batch maxConcurrency within the approved envelope.
4. On 429, inspect error classification and Retry-After. Pause the relevant producer, add jitter, cap attempts and total wait, and preserve idempotency.
5. Poll Queue Status at a modest cadence, apply backpressure before timeout risk rises, and shed or defer low-priority work using an owned policy.
6. Test rate 429, concurrency 429, long queue, missing/invalid Retry-After, cancellation, and recovery with synthetic responses.
7. Tune from observed throughput with safety headroom; emit queue, throttle, retry, drop/defer, latency, and cost receipts.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before raising producer or page concurrency, extending total retry time, increasing plan capacity, bypassing backpressure, or prioritizing one workload over another.

Output

Return current…

Firecrawl Governed Ingestion Architecture

Overview

Create explicit trust boundaries from request intake through deletion. Keep source discovery, content retrieval, validation, and downstream publication independently retryable and auditable.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Firecrawl v2 provides scrape, crawl, map, search, batch scrape, parse, JSON extraction, agentic, and browser surfaces with different async, credit, retention, and availability contracts. The SDK can auto-wait and paginate, while explicit start/status methods support durable orchestration. Provider webhooks supplement but do not replace reconciliation.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Define tenants, source authorization, freshness, scale, formats, data classification, SLO/RPO/RTO, retention, deletion, and cost constraints.
2. Place authentication, tenant isolation, URL canonicalization, policy versioning, endpoint/format allowlists, budgets, and idempotency at the intake gateway.
3. Separate discovery into map/search, retrieval into scrape/batch/crawl/parse, and model extraction into a validated stage. Use durable job records for asynchronous work.
4. Persist normalized provenance and job/page state before downstream writes. Complete pagination and deduplicate by tenant, canonical source, version, and content hash.
5. Run schema, origin-status, content-quality, malware/active-content, and prompt-injection gates before storage or agent context.
6. Use an outbox or equivalent for indexing and publication; make retries idempotent and support tombstones across raw, normalized, embedding, cache, and search stores.
7. Observe queue, job, origin, validation, freshness, spend, webhook, and deletion SLIs; test restore, reconciliation, cancellation, and regional/provider failure.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require architecture and security/data approval before adding a new endpoint class,…

Firecrawl Reliability Controls

Overview

Make every unit of work recoverable without duplicate collection or downstream writes. Provider retry, client retry, queue replay, webhook redelivery, and operator replay must share one idempotency model.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Only errors classified retryable by Firecrawl's current error catalog should enter automatic backoff. Async crawl, batch, agent, and related jobs require durable IDs and terminal-state handling; crawl and batch results may paginate. Webhooks retry delivery on their schedule, while status reconciliation remains necessary after missed or exhausted delivery.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Define a stable work identity from tenant, approved source, operation, policy version, and requested version; store attempts and downstream state durably.
2. Separate submission from completion. Persist the provider job ID before acknowledging work and resume status/pagination from durable state.
3. Classify validation, authentication, credits, restrictions, origin responses, rate/concurrency, timeouts, server failures, and downstream errors independently.
4. Retry only documented retryable classes with jitter, Retry-After, maximum attempts, total deadline, and a circuit breaker. Never rotate keys to evade limits.
5. Make page acceptance, storage, indexing, webhook handling, and tombstones idempotent. Deduplicate webhook deliveries by webhookId plus event context.
6. Reconcile provider job state, all result pages, accepted/rejected totals, downstream receipts, and spend on a schedule; cancel abandoned work where safe.
7. Exercise crash-after-submit, crash-after-write, duplicate webhook, partial pagination, provider outage, stale cache, and rollback in tests.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before expanding retry budgets, replaying production jobs, cancelling shared work, using…

Firecrawl v2 SDK Boundary

Overview

Keep Firecrawl behind a narrow application-owned interface so provider changes, policy, retries, and tests do not leak across the codebase.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Node uses the named Firecrawl client from the firecrawl install surface; Python uses Firecrawl from firecrawl-py. Current top-level v2 methods include scrape, crawl, startCrawl, getCrawlStatus, map, batchScrape, startBatchScrape, getBatchScrapeStatus, search, parse, and agent surfaces. Feature-frozen v1 lives behind a separate compatibility boundary.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Inspect the runtime, official package and resolved version, compiler settings, existing HTTP/client layer, and exact operations required.
2. Define application request/result types containing only approved fields, explicit scope/limits, provenance, origin status, terminal state, and pagination metadata.
3. Construct one client per process boundary from validated configuration; inject it into services and tests rather than creating clients inside business logic.
4. Map provider documents and errors into a discriminated result model. Keep API failures, captured origin errors, validation rejection, partial results, and cancellation distinct.
5. Expose waiter methods only where blocking fits the deadline; otherwise expose submit/status/cancel and durable pagination through an owned job service.
6. Centralize bounded retry, policy, redaction, metrics, and version reporting. Do not erase SDK types with any or dictionary-shaped pass-throughs.
7. Add synthetic contract tests, pin the dependency through the lockfile, and document the reviewed upgrade and rollback process.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval before adding/upgrading the SDK, exposing a new Firecrawl operation, relaxing types, changing retry policy, or accepting a provider response directly in domain co…

Firecrawl Security Baseline

Overview

Protect both directions of the integration: credentials and requests sent to Firecrawl, and hostile or sensitive content returned from target sites.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Cloud uses Bearer API keys; enterprise controls can restrict key endpoints/formats and source IPs and can add threat protection and SIEM evidence. Webhooks use X-Firecrawl-Signature with sha256=hex over the raw body. Cache, ZDR, lockdown, headers/actions, profiles, screenshots, and self-hosting have distinct data and threat boundaries.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Inventory secrets, teams/roles, key owners, environments, source networks, allowed endpoints/formats, domains, custom headers, retained data, webhooks, stores, and downstream actions.
2. Move every key and webhook secret to an approved secret manager; use workload identity, least privilege, rotation, revocation, and separate environments.
3. Canonicalize and authorize targets before requests. Block private/link-local networks, embedded credentials, disallowed redirects/ports, lookalike hosts, and unauthorized authenticated pages.
4. Minimize formats, actions, headers, profiles, proxies, and retention. Apply key/IP/threat controls where available and keep application policy fail closed.
5. Verify every webhook against raw bytes, require the sha256 prefix, decode equal-length hex, compare timing-safely, deduplicate webhookId, and acknowledge only accepted work.
6. Treat scraped content and extracted JSON as untrusted. Sanitize active content, isolate it from privileged prompts/actions, validate schemas, and scan stored artifacts.
7. For self-hosting, add supported authentication, network controls, TLS, durable stores, patching, backups, monitoring, and provider data-flow review before exposure.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require security/data approval before authenticated…

Firecrawl v1-to-v2 Upgrade

Overview

Treat the version change as a behavior migration, not a search-and-replace. Inventory method, option, response, cache, extraction, pagination, and billing differences before switching traffic.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

The current top-level SDK defaults to v2. Key mappings include scrapeUrl to scrape, crawlUrl to crawl, asyncCrawlUrl to startCrawl, checkCrawlStatus to getCrawlStatus, mapUrl to map, batchScrapeUrls to batchScrape, asyncBatchScrapeUrls to startBatchScrape, and checkBatchScrapeStatus to getBatchScrapeStatus. The legacy extract format became a json format object; v2 cache defaults and option shapes also changed.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Pin the current dependency, routes, methods, options, response assumptions, fixtures, usage, and rollback release. Identify intentional feature-frozen v1 callers.
2. Read current migration, SDK, endpoint, billing, and error references; build a per-call mapping including renamed, removed, defaulted, and behavior-changing fields.
3. Add golden synthetic tests for requests, direct SDK versus REST responses, metadata fields, origin status, JSON extraction, crawl/batch jobs, pagination, errors, and cost-sensitive options.
4. Introduce a typed v2 adapter behind a feature flag. Keep v1 compatibility isolated and do not silently fall back from v2 on failure.
5. Run offline contract tests, then shadow approved canaries with content hashes, schema coverage, latency, cache state, and credits rather than raw body diffs.
6. Shift traffic gradually, monitor errors, partial results, quality, freshness, queue, and spend, and exercise rollback before removing the flag.
7. After the observation window, remove legacy packages/routes/secrets, update docs/fixtures, and verify no v0/v1 strings remain except migration evidence.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked.

Approval Boundaries

Require approval befo…

Firecrawl Signed Webhook Processing

Overview

Treat a webhook as an authenticated delivery hint into a durable state machine. Verify before parsing or side effects, acknowledge promptly, and reconcile through status APIs after missed delivery.

Prerequisites

• The target repository or integration path and the requested operator outcome.
• The source authorization, data classification, and environment policy.
• Current Firecrawl documentation, credentials only when needed, and an owner for approvals.

Current Contract

Firecrawl signs the raw request body with HMAC-SHA256 in X-Firecrawl-Signature formatted sha256=hex. The documented events are crawl.started/page/completed; batch_scrape.started/page/completed; extract.started/completed/failed; agent.started/action/completed/failed/cancelled; monitor.page and monitor.check.completed. Endpoints must return 2xx within 10 seconds; failed delivery retries after 1, 5, and 15 minutes, then stops.

Authentication

For authenticated Cloud operations, inject FIRECRAWL_API_KEY from an approved secret manager. REST requests use Authorization: Bearer with the key. Never print, commit, transmit, or place a key in a URL. Keyless access is suitable only where the current documentation explicitly allows it and the workload accepts its limits; production workflows should make identity and team ownership explicit.

Instructions

1. Register an HTTPS endpoint and choose only the events required for the workflow. Store the account webhook secret in an approved secret manager.
2. Capture the raw body before any JSON parser. Require one X-Firecrawl-Signature value, split and validate the sha256 prefix, hex-decode safely, and compare equal-length digests timing-safely.
3. Reject missing, malformed, or mismatched signatures before parsing, logging, queuing, or mutating state. Never treat a missing secret as verification success.
4. Parse the verified envelope, allowlist exact event types and schema, bind it to the expected team/job, and deduplicate by webhookId plus event/job context.
5. Persist an inbox record and respond with 2xx inside the deadline; process content and downstream writes asynchronously with tenant isolation and idempotency.
6. Handle page and terminal events according to their documented family. Do not wait for nonexistent crawl.failed or batch_scrape.failed events; reconcile job status and pagination.
7. Test signatures, raw-body mutation, duplicates, reordering, retry timing, unknown events, partial pages, crash recovery, and status reconciliation.

Tool Discipline

Use Read, Glob, and Grep to inspect code, configuration, tests, and evidence. Use Write/Edit only for approved implementation or documentation changes. Do not call Firecrawl, rotate keys, change account settings, scrape a target, or deploy merely because this skill was invoked…