Apify Rate Limits

Overview

The Apify API enforces rate limits per resource. The apify-client library

auto-retries 429s (up to 8 times with exponential backoff), so most workloads never

notice a limit. You reach for this skill when bulk operations, custom API calls, or

large fan-outs push past what the built-in retry can absorb — you then batch, queue,

stagger, and monitor to stay under the ceiling.

Full runnable code for every step is in

implementation.md; combined scenarios are in

examples.md.

Apify rate limit rules

"Per resource" means: calls to dataset A and dataset B each get 60 req/sec

independently. Every response carries X-RateLimit-Limit,

X-RateLimit-Remaining, and X-RateLimit-Reset (epoch seconds) headers.

Prerequisites

• An Apify account with API access and APIFY_TOKEN set in the environment.
• The apify-client package installed (npm install apify-client).
• For custom queuing: p-queue (npm install p-queue); crawlee for sleep and

crawler-level concurrency.

Instructions

The workflow is five steps. Each is summarized here with its core lever; the full

runnable code for every step is in

implementation.md.

1. Understand built-in retries — apify-client already retries 429/500+ with

exponential backoff. Tune maxRetries / minDelayBetweenRetriesMillis only when

the defaults are wrong for your endpoint:

   import { ApifyClient } from 'apify-client';
   const client = new ApifyClient({
     token: process.env.APIFY_TOKEN,
     maxRetries: 5,                      // Default: 8
     minDelayBetweenRetriesMillis: 500,  // Default: 500
   });

1. Batch operations (biggest lever) — collapse per-item loops into one batched

call (up to 9 MB), chunking only for very large datasets:

   await client.dataset(dsId).pushItems(items);   // 1 call, not N

1. Queue custom calls — gate raw API calls through p-queue

(concurrency + intervalCap) so fan-out reads never exceed 60 req/sec. See

implementation.md § Step 3.

1. Stagger Actor starts — insert a ~200 ms delay between start() calls so the

runs endpoint never 429s, then waitForFinish() in parallel. See

implementation.md § Step 4.

1. Monitor headers — feed X-RateLimit-* into a small monitor that warns before

the wall and pauses exactly until reset. See

implementation.md § Step 5.

Target-website throttling is a separate ceiling from the platform API — cap it with

Crawlee's maxConcurrency / maxRequestsPerMinute

(implementation.md § Crawlee-level concurrency).

Output

Applying this skill produces a rate-aware Apify integration:

• A configured ApifyClient with an explicit retry envelope.
• Batched/chunked dataset writes that cut API-call count by orders of magnitude.
• A p-queue-gated call path that holds requests under 60 req/sec per resource.
• Staggered Actor starts and, optionally, a header-driven monitor that pauses before

exhaustion — the net effect being zero (or transparently retried) 429s under load.

Error Handling

Examples

Worked end-to-end scenarios live in examples.md:

• Bulk dataset push without 429s — 50,000 rows in ~50 calls via chunked batching.
• Fan-out reads through a queue — 500 Actor reads held under 50 req/sec.
• Launch 100 runs safely — staggered starts, then parallel wait-for-finish.
• Pause on header-driven exhaustion — sleep exactly until the limit resets.

Resources

• Apify API Rate Limits
• p-queue Documentation
• Crawlee Auto-scaling

For security configuration, see apify-security-basics.