Apify Rate Limits

Overview

The Apify API enforces rate limits per resource. The apify-client library auto-retries 429s (up to 8 times with exponential backoff), but you need to understand the limits for bulk operations and custom API calls.

Apify Rate Limit Rules

"Per resource" means: calls to dataset A and dataset B each get 60 req/sec independently.

Rate limit headers returned:

• X-RateLimit-Limit — max requests per interval
• X-RateLimit-Remaining — remaining requests
• X-RateLimit-Reset — epoch seconds when limit resets

Instructions

Step 1: Understand Built-in Retries

The apify-client package handles rate limits automatically:

import { ApifyClient } from 'apify-client';

// Default: retries up to 8 times on 429 and 500+ errors
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

// Customize retry behavior
const client = new ApifyClient({
  token: process.env.APIFY_TOKEN,
  maxRetries: 5,         // Default: 8
  minDelayBetweenRetriesMillis: 500,  // Default: 500
});

Step 2: Batch Operations to Reduce API Calls

// BAD: 1000 API calls (easily rate limited)
for (const item of items) {
  await client.dataset(dsId).pushItems([item]);
}

// GOOD: 1 API call (up to 9MB payload)
await client.dataset(dsId).pushItems(items);

// GOOD: Chunked for very large datasets
function chunkArray<T>(arr: T[], size: number): T[][] {
  const chunks: T[][] = [];
  for (let i = 0; i < arr.length; i += size) {
    chunks.push(arr.slice(i, i + size));
  }
  return chunks;
}

for (const chunk of chunkArray(items, 1000)) {
  await client.dataset(dsId).pushItems(chunk);
}

Step 3: Queue-Based Rate Limiting for Custom Calls

import PQueue from 'p-queue';

// 50 requests per second with max 10 concurrent
const apiQueue = new PQueue({
  concurrency: 10,
  interval: 1000,
  intervalCap: 50,
});

// All API calls go through the queue
async function rateLimitedCall<T>(fn: () => Promise<T>): Promise<T> {
  return apiQueue.add(fn) as Promise<T>;
}

// Usage
const results = await Promise.all(
  actorIds.map(id =>
    rateLimitedCall(() => client.ac

Apify Reference Architecture

Overview

Production-ready architecture patterns for applications built on Apify. Covers standalone Actor projects, multi-Actor pipelines, and full-stack applications that integrate Apify as a data source.

Architecture Pattern 1: Standalone Actor

For a single scraper deployed to Apify platform.

my-scraper/
├── .actor/
│   ├── actor.json           # Actor metadata
│   ├── INPUT_SCHEMA.json    # Input definition (generates UI)
│   └── Dockerfile           # Build configuration
├── src/
│   ├── main.ts              # Entry point (Actor.main)
│   ├── routes/
│   │   ├── listing.ts       # Router handler: listing pages
│   │   └── detail.ts        # Router handler: detail pages
│   ├── types.ts             # Input/output TypeScript types
│   └── utils/
│       ├── extractors.ts    # Data extraction functions
│       └── validators.ts    # Input/output validation
├── tests/
│   ├── extractors.test.ts   # Unit tests for extraction logic
│   └── integration.test.ts  # Integration tests (live API)
├── storage/                  # Local storage (git-ignored)
├── package.json
├── tsconfig.json
└── .gitignore

Key Files

// src/main.ts — Actor entry point
import { Actor } from 'apify';
import { CheerioCrawler } from 'crawlee';
import { router } from './routes/listing';
import { validateInput, ScraperInput } from './types';

await Actor.main(async () => {
  const rawInput = await Actor.getInput<ScraperInput>();
  const input = validateInput(rawInput);

  const proxyConfiguration = input.proxyConfig?.useApifyProxy
    ? await Actor.createProxyConfiguration({ groups: input.proxyConfig.groups })
    : undefined;

  const crawler = new CheerioCrawler({
    requestHandler: router,
    proxyConfiguration,
    maxRequestsPerCrawl: input.maxItems ?? 100,
    maxConcurrency: input.concurrency ?? 10,
  });

  await crawler.run(input.startUrls.map(s => s.url));
});

// src/types.ts — Shared types and validation
import { z } from 'zod';

export const InputSchema = z.object({
  startUrls: z.array(z.object({ url: z.string().url() })).min(1),
  maxItems: z.number().int().positive().optional().default(100),
  concurrency: z.number().int().min(1).max(50).optional().default(10),
  proxyConfig: z.object({
    useApifyProxy: z.boolean(),
    groups: z.array(z.string()).optional(),
  }).optional(),
});

export type ScraperInput = z.infer<typeof InputSchema>;

export function validateInput(raw: unknown): ScraperInput {
  return InputSchema.parse(raw);
}

export interface ProductOutput {
  url: string;
  name: string;
  price: number | null;
  currency: string;
  inStock: boolean;
  scrapedAt: string;
}

Architecture Pattern 2: Multi-Actor Pipeline

For complex scraping workflows with multiple stages.

┌──

Apify SDK Patterns

Overview

Production patterns for both the apify SDK (building Actors) and apify-client (calling Actors remotely). Covers Crawlee crawler selection, data storage, proxy configuration, and typed client wrappers.

Prerequisites

• apify-client and/or apify + crawlee installed
• APIFY_TOKEN configured
• TypeScript recommended

Pattern 1: Typed Client Singleton

// src/apify/client.ts
import { ApifyClient } from 'apify-client';

let instance: ApifyClient | null = null;

export function getApifyClient(): ApifyClient {
  if (!instance) {
    const token = process.env.APIFY_TOKEN;
    if (!token) throw new Error('APIFY_TOKEN is required');
    instance = new ApifyClient({ token });
  }
  return instance;
}

// Reset for testing
export function resetClient(): void {
  instance = null;
}

Pattern 2: Crawlee Crawler Selection

Choose the right crawler for the job:

import { CheerioCrawler, PlaywrightCrawler, PuppeteerCrawler } from 'crawlee';

// CHEERIO — Fast, lightweight, no JavaScript rendering
// Use for: static HTML, server-rendered pages, APIs
const cheerioCrawler = new CheerioCrawler({
  async requestHandler({ request, $, enqueueLinks }) {
    const title = $('title').text();
    await Actor.pushData({ url: request.url, title });
    await enqueueLinks({ strategy: 'same-domain' });
  },
});

// PLAYWRIGHT — Full browser, all engines, modern API
// Use for: SPAs, JavaScript-heavy pages, complex interactions
const playwrightCrawler = new PlaywrightCrawler({
  launchContext: { launchOptions: { headless: true } },
  async requestHandler({ page, request, enqueueLinks }) {
    await page.waitForSelector('h1');
    const title = await page.title();
    const content = await page.$eval('main', el => el.textContent);
    await Actor.pushData({ url: request.url, title, content });
    await enqueueLinks({ strategy: 'same-domain' });
  },
});

// PUPPETEER — Chromium-only browser automation
// Use for: when you need Chromium specifically or legacy Puppeteer code
const puppeteerCrawler = new PuppeteerCrawler({
  async requestHandler({ page, request }) {
    const title = await page.title();
    await Actor.pushData({ url: request.url, title });
  },
});

Pattern 3: Actor Lifecycle with Error Handling

import { Actor } from 'apify';
import { CheerioCrawler, log } from 'crawlee';

// Actor.main() wraps init + exit + error handling
await Actor.main(async () => {
  const input = await Actor.getInput<{
    startUrls: { url: string }[];
    maxPages?: number;
    proxyConfig?: { useApifyProxy: boolean; groups?: string[] };
  }>();

  if (!input?.startUrls?.lengt

Apify Security Basics

Overview

Security best practices for Apify API tokens, Actor data, proxy credentials, and webhook verification. Apify uses personal API tokens (prefixed apifyapi) for all authentication.

Prerequisites

• Apify account with Console access
• Understanding of environment variables
• Access to your deployment platform's secrets management

Token Architecture

Apify uses a single API token per user account for full API access. There is no scope-based permission system per token, so token security is critical.

Instructions

Step 1: Secure Token Storage

# .env (NEVER commit — must be in .gitignore)
APIFY_TOKEN=apify_api_YOUR_TOKEN_HERE

# .gitignore — mandatory entries
.env
.env.local
.env.*.local
storage/   # Local Apify storage may contain scraped data

// Validate token exists at startup
function requireToken(): string {
  const token = process.env.APIFY_TOKEN;
  if (!token) {
    throw new Error(
      'APIFY_TOKEN is required. Get yours at ' +
      'https://console.apify.com/account/integrations'
    );
  }
  if (!token.startsWith('apify_api_')) {
    console.warn('Warning: APIFY_TOKEN does not have expected prefix');
  }
  return token;
}

Step 2: Per-Environment Token Isolation

Use separate Apify accounts (or at minimum separate tokens) per environment:

# Development — your personal account
APIFY_TOKEN=apify_api_dev_token

# Staging — shared team account (limited usage)
APIFY_TOKEN=apify_api_staging_token

# Production — production account (separate billing)
APIFY_TOKEN=apify_api_prod_token

Platform secrets management:

# GitHub Actions
gh secret set APIFY_TOKEN --body "apify_api_prod_token"

# Vercel
vercel env add APIFY_TOKEN production

# Google Cloud Secret Manager
echo -n "apify_api_prod_token" | \
  gcloud secrets create apify-token --data-file=-

Step 3: Token Rotation Procedure

# 1. Generate new token in Console > Settings > Integrations
#    (old token remains valid until explicitly revoked)

# 2. Update in all environments
gh secret set APIFY_TOKEN --body "apify_api_NEW_TOKEN"

# 3. Verify new token works
curl -sf -H "Authorization: Bearer $NEW_TOKEN" \
  https://api.apify.com/v2/users/me | jq '.data.username'

# 4. Revoke old token in Console
#

Apify Upgrade & Migration

Overview

Guide for upgrading apify, apify-client, and crawlee packages. The biggest migration in Apify's history was SDK v2 to v3, which split crawling functionality into the crawlee package. This skill covers that migration plus general upgrade procedures.

Prerequisites

• Git branch for the upgrade
• Test suite available
• Current versions documented

Instructions

Step 1: Check Current Versions

# Check installed versions
npm list apify apify-client crawlee 2>/dev/null

# Check latest available versions
npm view apify version
npm view apify-client version
npm view crawlee version

# Check for outdated packages
npm outdated apify apify-client crawlee

Step 2: Create Upgrade Branch

git checkout -b upgrade/apify-packages

Step 3: Upgrade Packages

# Upgrade to latest
npm install apify@latest crawlee@latest apify-client@latest

# Or upgrade to specific version
npm install apify@3.2.0 crawlee@3.11.0

# Check for peer dependency issues
npm ls 2>&1 | grep "ERESOLVE\|peer dep"

Step 4: Run Tests and Fix Issues

npm test
npm run build  # Catch TypeScript errors

Major Migration: Apify SDK v2 to v3 (Crawlee Split)

This is the most common migration. In v3, crawling code moved to crawlee.

Import Changes

// ---- BEFORE (SDK v2) ----
import Apify from 'apify';
const { CheerioCrawler, PlaywrightCrawler, log } = Apify;

// ---- AFTER (SDK v3 + Crawlee) ----
import { Actor } from 'apify';
import { CheerioCrawler, PlaywrightCrawler, log } from 'crawlee';

Initialization Changes

// ---- BEFORE (v2) ----
Apify.main(async () => {
  const input = await Apify.getInput();
  const dataset = await Apify.openDataset();
  await Apify.pushData({ url: 'https://example.com' });
  await Apify.setValue('OUTPUT', { done: true });
});

// ---- AFTER (v3) ----
await Actor.main(async () => {
  const input = await Actor.getInput();
  const dataset = await Actor.openDataset();
  await Actor.pushData({ url: 'https://example.com' });
  await Actor.setValue('OUTPUT', { done: true });
});

Crawler Configuration Changes

// ---- BEFORE (v2) ----
const crawler = new Apify.CheerioCrawler({
  handlePageFunction: async ({ request, $ }) => {
    // ...
  },
  handleFailedRequestFunction: async ({ request }) => {
    // ...
  },
});

// ---- AFTER (v3 / Crawlee) ----
const crawler = new CheerioCrawler({
  requestHandler: async ({ request, $ }) => {
    // renamed from handlePa

Apify Webhooks & Events

Overview

Configure webhooks to receive notifications when Actor runs complete, fail, or time out. Apify supports both persistent webhooks (for all runs of an Actor) and ad-hoc webhooks (for a single run). Event-driven architecture is the recommended pattern for production Apify integrations.

Event Types

Instructions

Step 1: Create a Persistent Webhook

Persistent webhooks fire for every run of an Actor:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const webhook = await client.webhooks().create({
  eventTypes: [
    'ACTOR.RUN.SUCCEEDED',
    'ACTOR.RUN.FAILED',
    'ACTOR.RUN.TIMED_OUT',
  ],
  condition: {
    actorId: 'YOUR_ACTOR_ID',
  },
  requestUrl: 'https://your-app.com/api/webhooks/apify',
  payloadTemplate: JSON.stringify({
    eventType: '{{eventType}}',
    createdAt: '{{createdAt}}',
    actorId: '{{actorId}}',
    actorRunId: '{{actorRunId}}',
    defaultDatasetId: '{{resource.defaultDatasetId}}',
    defaultKeyValueStoreId: '{{resource.defaultKeyValueStoreId}}',
    status: '{{resource.status}}',
    statusMessage: '{{resource.statusMessage}}',
    startedAt: '{{resource.startedAt}}',
    finishedAt: '{{resource.finishedAt}}',
  }),
  isAdHoc: false,
});

console.log(`Webhook created: ${webhook.id}`);

Step 2: Use Ad-Hoc Webhooks for Single Runs

Ad-hoc webhooks are created at run time and fire only for that specific run:

// Ad-hoc webhook via API (pass webhooks array when starting a run)
const run = await client.actor('username/my-actor').start(
  { startUrls: [{ url: 'https://example.com' }] },
  {
    webhooks: [
      {
        eventTypes: ['ACTOR.RUN.SUCCEEDED', 'ACTOR.RUN.FAILED'],
        requestUrl: 'https://your-app.com/api/webhooks/apify',
        payloadTemplate: JSON.stringify({
          runId: '{{actorRunId}}',
          status: '{{resource.status}}',
          datasetId: '{{resource.defaultDatasetId}}',
        }),
      },