Apify CI Integration

Overview

Automate Apify Actor builds, tests, and deployments using GitHub Actions — test-on-PR, deploy-on-merge, live-API integration testing, and Docker build verification. Three workflow files do the work (test, deploy, verify-build); full copy-paste-ready definitions live in references/workflows.md.

Prerequisites

• GitHub repository with Actions enabled
• Apify API token stored as a GitHub secret
• Actor code in the repository

Authentication

CI authenticates to Apify with a personal API token, never a hard-coded credential:

• Store the token as a GitHub Actions secret (gh secret set APIFY_TOKEN), and expose it to a job only via env: APIFY_TOKEN: ${{ secrets.APIFY_TOKEN }}.
• The Apify CLI reads it as apify login --token $APIFY_TOKEN; the REST API and apify-client read it from the APIFY_TOKEN environment variable.
• Use separate APIFY_TOKEN_TEST / APIFY_TOKEN_PROD secrets so integration runs never touch production data. Get tokens from Apify Console → Settings → Integrations.

Instructions

Step 1: Configure GitHub Secrets

# Store Apify token for CI
gh secret set APIFY_TOKEN --body "apify_api_YOUR_CI_TOKEN"

# Optional: separate tokens for test vs production
gh secret set APIFY_TOKEN_TEST --body "apify_api_test_token"
gh secret set APIFY_TOKEN_PROD --body "apify_api_prod_token"

Step 2: Create the test workflow

Add .github/workflows/apify-test.yml with two jobs — unit-tests on every PR and integration-tests gated to push (merge) events. The integration job proves connectivity before running tests:

      - name: Verify Apify connection
        run: |
          curl -sf -H "Authorization: Bearer $APIFY_TOKEN" \
            https://api.apify.com/v2/users/me | jq '.data.username'

Full two-job workflow: references/workflows.md (Test Workflow).

Step 3: Create the deploy workflow

Add .github/workflows/apify-deploy.yml — triggered on merges that touch src/**, package.json, or .actor/** (plus workflow_dispatch). It builds, tests, installs the Apify CLI, apify pushes, then runs a minimal smoke call to confirm the new build actually starts. Full definition: references/workflows.md (Deploy Workflow).

Step 4: Write integration tests

Gate live-API tests on the token so they skip cleanly when it is absent:

const SKIP_INTEGRATION = !p…

Apify Common Errors

Overview

Quick diagnostic reference for the most common Apify errors. Covers Actor run failures, API errors, proxy problems, anti-bot blocks, and platform-specific issues. The full per-error catalog — raw signature, cause, diagnosis, and copy-paste fix for all ten errors — lives in references/error-reference.md; this file is the lean triage path that points you there.

Prerequisites

• An Apify API token exported as APIFY_TOKEN (Console > Settings > Integrations).
• Access to the Apify Console for reading run and build logs.
• curl and jq on PATH for the diagnostic commands below; apify CLI optional for build inspection.

Instructions

Work the failure from signal to fix:

1. Capture the exact error. Grab the Status / StatusMessage from the run, or the ApifyApiError message and HTTP code from the API response. The raw string is the key you match on.
2. Match the signature. Find the matching entry in the table below, then open references/error-reference.md at that number for the cause, diagnosis, and fix.
3. Diagnose before changing code. For run failures, pull the log via the API or Console (see the FAILED entry). For auth/rate/payload errors, the HTTP code already tells you the class — see Error Handling.
4. Apply the fix from the reference, re-run, and confirm with the diagnostic commands.

Error signature → reference

Full…

Apify Core Workflow A — Build & Deploy a Scraper

Overview

End-to-end workflow: define input schema, build a Crawlee-based Actor, extract structured data, store results in datasets, test locally, and deploy to Apify platform. This is the primary money-path workflow for Apify.

Prerequisites

• npm install apify crawlee in your project
• npm install -g apify-cli and apify login completed
• For programmatic retrieval (Step 6), an API token in APIFY_TOKEN — read it from the environment (process.env.APIFY_TOKEN), never hard-code it
• Familiarity with apify-sdk-patterns

Instructions

Step 1: Define Input Schema

Create .actor/INPUT_SCHEMA.json:

{
  "title": "E-Commerce Scraper",
  "type": "object",
  "schemaVersion": 1,
  "properties": {
    "startUrls": {
      "title": "Start URLs",
      "type": "array",
      "description": "Product listing page URLs to scrape",
      "editor": "requestListSources",
      "prefill": [{ "url": "https://example-store.com/products" }]
    },
    "maxItems": {
      "title": "Max items",
      "type": "integer",
      "description": "Maximum number of products to scrape",
      "default": 100,
      "minimum": 1,
      "maximum": 10000
    },
    "proxyConfig": {
      "title": "Proxy configuration",
      "type": "object",
      "description": "Select proxy to use",
      "editor": "proxy",
      "default": { "useApifyProxy": true }
    }
  },
  "required": ["startUrls"]
}

Step 2: Build the Actor with Router Pattern

Use a Crawlee router that splits handling by page type: the default handler enqueues product links + pagination from listing pages, and a PRODUCT-labeled handler extracts structured fields from detail pages. The entry point wires proxy config, concurrency, a failed-request handler, and a run summary into the key-value store. Skeleton:

// src/main.ts
import { Actor } from 'apify';
import { CheerioCrawler, createCheerioRouter, Dataset, log } from 'crawlee';

const router = createCheerioRouter();
router.addDefaultHandler(async ({ enqueueLinks }) => {
  await enqueueLinks({ selector: 'a.product-card', label: 'PRODUCT' });
  await enqueueLinks({ selector: 'a.next-page', label: 'LISTING' });
});
router.addHandler('PRODUCT', async ({ request, $ }) =…

Apify Core Workflow B — Storage & Pipelines

Overview

Manage Apify's three storage types (datasets, key-value stores, request queues) and orchestrate multi-Actor pipelines using the apify-client JS SDK. Covers CRUD operations, data export, automatic pagination, and chaining Actors together (scrape → transform → export).

This SKILL.md gives you the high-level workflow plus the essential first example for each storage type. Drill into the reference files for the complete, copy-ready code:

• Storage operations — full reference — every dataset, key-value store, and request queue operation with pagination, format export, and binary records.
• Pipelines & run monitoring — full reference — the multi-Actor pipeline function and Actor-run status/cost/abort monitoring.

Prerequisites

• Node.js with apify-client installed (npm install apify-client).
• An Apify account token exported as APIFY_TOKEN (see Authentication below).
• Familiarity with apify-core-workflow-a (Actor invocation and run lifecycle), since pipelines chain Actor runs and read their default storages.

Authentication

All operations authenticate with an Apify API token. Never hard-code it — read it from the environment and construct the client once:

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

Generate a token at Apify Console → Settings → Integrations, then export it (export APIFY_TOKEN=apify_api_...) or load it from your secrets manager.

Storage Types at a Glance

Named storages persist indefinitely. Unnamed (default run) storages expire after 7 days.

Instructions

Pick the storage type you need, use the skeleton below to get started, then open the linked reference for the full operation set.

Datasets — append-only item lists

getOrCreate a named dataset, push items, and list them (pagination is manual):

const dataset = await client.datasets().getOrCreate('product-catalog');
const dsClient = client.dataset(dataset.id);
await dsClien…

Apify Cost Tuning

Overview

Apify charges on three axes: compute units (CU), proxy traffic (GB), and storage. One CU = 1 GB of memory running for 1 hour, so cost scales with both memory allocation and run duration. This skill walks the investigate → tune → guard loop that finds where spend is going, cuts it at the biggest lever (memory), and installs guardrails so it stays down.

Full pricing tables (plan CU prices, proxy rates, storage rules) live in pricing-model.md.

Prerequisites

• An Apify account with API access and APIFY_TOKEN set in the environment.
• The apify-client package installed (npm install apify-client).
• At least one Actor with run history to analyze.

Instructions

The workflow is six steps. Each is summarized here with its core lever; the full, runnable code for every step is in implementation.md.

1. Analyze current costs — roll up the last N days of runs into total CU, USD, and duration, and surface the single most expensive run:

   import { ApifyClient } from 'apify-client';
   const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

   const { items: runs } = await client.actor(actorId).runs().list({ limit: 1000, desc: true });
   const totalUsd = runs.reduce((s, r) => s + (r.usageTotalUsd ?? 0), 0);

1. Reduce memory allocation (biggest lever) — sweep memory from 4096 MB down to 256 MB and stop at the first failure to find the sweet spot. Most CheerioCrawler Actors are over-provisioned. Sweet spots: simple Cheerio 256-512 MB, complex 512-1024 MB, Playwright 2048-4096 MB.

1. Optimize crawl duration — higher maxConcurrency, tighter requestHandlerTimeoutSecs, a maxRequestsPerCrawl cap, fewer retries, and selective enqueueLinks. Faster crawls consume fewer CUs.

1. Minimize proxy costs — prefer datacenter (free with plan), only reach for residential when a site blocks it, block images/fonts/CSS to save residential GB, and reuse proxy sessions with useSessionPool.

1. Cost guard for runaway Actors — start the run, poll usageTotalUsd every 30s, and .abort() once spend crosses a hard cap.

1. Monitor monthly usage — iterate every Actor's runs since the 1st of the month and print a cost-descending report so the top spenders are obvious.

See full walkthrough for the complete code of each step, including the memory sweep, proxy hooks, budget guard, and monthly report.

Output

Running this skill produces…

Apify Debug Bundle

Overview

Collect all diagnostic information needed to troubleshoot failed Actor runs and prepare Apify support tickets. Pulls run metadata, logs, dataset samples, and environment info into a single bundle so a support engineer (or you) can diagnose the failure without live access to your account.

Prerequisites

• apify-client installed
• APIFY_TOKEN configured
• A failed or problematic run ID to investigate

Authentication

All API calls authenticate with the APIFY_TOKEN as a Bearer header (Authorization: Bearer $APIFY_TOKEN), and the SDK reads the same token from process.env.APIFY_TOKEN. Get the token from the Apify Console under Settings → Integrations → Personal API tokens. Never commit it — the bundle script redacts any local .env before packaging, and the platform auto-redacts secrets inside run logs.

Instructions

The workflow has four steps. The skeleton below is enough to run it; each step's full implementation lives in implementation.md.

1. Investigate the failed run — pull run summary, dataset stats, and the log tail via the SDK. The core call:

   const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
   const run = await client.run(runId).get();
   const log = await client.run(runId).log().get();

1. Create the debug bundle — run apify-debug-bundle.sh <RUN_ID>. It collects environment info, run details, log, a 5-item dataset sample, key-value store keys, a redacted .env, and platform health, then packages everything into a timestamped .tar.gz. Full script in implementation.md.

1. Compare against a good run (optional) — diff a successful and failed run field-by-field to spot the delta (compareRuns(successId, failId)).

1. Live-tail a running Actor (optional) — stream logs when the final log is not yet available.

For copy-pasteable code for every step, see implementation.md.

Output

A single timestamped tarball, apify-debug-YYYYMMDD-HHMMSS.tar.gz, containing:

Apify Deploy Integration

Overview

Deploy Actors to the Apify platform and integrate their results into external applications. Covers apify push deployment, API-triggered runs from web apps (synchronous and async patterns), webhook receivers, scheduled scraping pipelines, and container deployment.

SKILL.md gives you the workflow and the core skeleton. Complete, copy-paste code for every pattern lives in references/implementation.md; end-to-end worked scenarios are in references/examples.md.

Prerequisites

• Actor tested locally (apify run)
• apify login completed (stores CLI credentials)
• Target application ready for integration

Authentication

Apps authenticate with an Apify API token. Generate one in Apify Console → Settings → Integrations and expose it as the APIFY_TOKEN environment variable — never hard-code it. The apify CLI uses its own credentials from apify login, separate from APIFY_TOKEN. Full auth notes: references/implementation.md.

Instructions

Step 1: Deploy the Actor to the platform

# Push Actor code to Apify
apify push

# Push to a specific Actor (creates if it doesn't exist)
apify push username/my-scraper

# Pull an existing Actor to modify
apify pull username/existing-actor

Step 2: Trigger the Actor from your app

Instantiate ApifyClient with your token, then either call() (blocks until the run finishes) or start() (returns immediately for polling). Here is the core synchronous skeleton:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('username/product-scraper').call({
  startUrls: [{ url: 'https://store.example.com' }],
  maxItems: 500,
});
if (run.status !== 'SUCCEEDED') throw new Error(run.statusMessage);

const { items } = await client.dataset(run.defaultDatasetId).listItems();

The full typed service — scrapeProducts (blocking), startScrape + getScrapeResults (async poll) — is in references/implementation.md.

Step 3: Choose an integration pattern

Pick the pattern that matches your app, then copy the full handler from the reference:

• Next.js API route — start a run in a POST, poll by run ID in a GET. Avoids serverless timeouts. See implementation.md.
• Express webhook receiver — register an Ap…

Apify Hello World

Overview

Run a public Actor from the Apify Store, wait for it to finish, and retrieve the scraped data. This demonstrates the fundamental call-wait-collect pattern used in every Apify integration.

Prerequisites

• npm install apify-client completed
• APIFY_TOKEN environment variable set
• See apify-install-auth if not ready

Authentication

Every call authenticates with a personal API token passed to the client constructor: new ApifyClient({ token: process.env.APIFY_TOKEN }). Keep the token in the APIFY_TOKEN environment variable — never hard-code it in the script. Full setup (where to generate the token, how to export it) lives in the apify-install-auth skill.

Core Pattern: Call Actor, Get Data

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

// 1. Run an Actor and wait for it to finish
const run = await client.actor('apify/website-content-crawler').call({
  startUrls: [{ url: 'https://docs.apify.com/academy' }],
  maxCrawlPages: 5,
});

// 2. Retrieve results from the default dataset
const { items } = await client.dataset(run.defaultDatasetId).listItems();

console.log(`Crawled ${items.length} pages:`);
items.forEach(item => {
  console.log(`  - ${item.url}: ${item.text?.substring(0, 80)}...`);
});

This is the whole workflow at a high level: authenticate, .call() an Actor, then read its default dataset. For sync-vs-async execution, pagination, downloads, key-value store retrieval, run-configuration options, and a table of popular starter Actors, see run & retrieval patterns.

Instructions

Step 1: Create the Script

Use Write (or Edit an existing file) to create hello-apify.ts (or .js) with the Core Pattern code above. Use Read to confirm the file contents before running.

Step 2: Run It

# With tsx (recommended)
npx tsx hello-apify.ts

# Or with Node.js (plain JS)
node hello-apify.js

Step 3: Understand the Output

The Actor runs on Apify's cloud infrastructure. See the Output section below for the run-object fields returned when it finishes.

Output

A successful run returns a run object plus a populated dataset. The fields you read most:

Apify Install & Auth

Overview

Set up the Apify ecosystem: the apify-client JS library (for calling Actors remotely), the apify SDK (for building Actors), the Apify CLI (for deploying), and Crawlee (for crawling). Each package serves a different purpose — install only what the task needs, then wire up a single API token.

Package Map

Prerequisites

• Node.js 18+ (required by SDK v3+)
• Apify account at https://console.apify.com
• API token from Settings > Integrations in Apify Console

Instructions

Step 1: Install Packages

# For CALLING existing Actors from your app:
npm install apify-client

# For BUILDING your own Actors:
npm install apify crawlee

# For CLI deployment:
npm install -g apify-cli

Step 2: Configure Authentication

Pick one. Read any existing .env first so you do not clobber it, then Write the token in.

# Option A: Environment variable (recommended for apps)
export APIFY_TOKEN="apify_api_YOUR_TOKEN_HERE"

# Option B: .env file (add .env to .gitignore)
echo 'APIFY_TOKEN=apify_api_YOUR_TOKEN_HERE' >> .env

# Option C: CLI login (for interactive development)
apify login
# Paste your token when prompted

The APIFY_TOKEN env var is auto-detected by both apify-client and the apify SDK. For every place a token can be supplied (constructor option, header, precedence) plus the full platform env-var list, see authentication reference.

Step 3: Verify Connection

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

// List your Actors to confirm auth works
const { items } = await client.actors().list();
console.log(`Authenticated. You have ${items.length} Actors.`);

Step 4: Verify CLI (if installed)

apify login --token YOUR_TOKEN
apify info  # Shows your account info

Output

A working…

Apify Local Dev Loop

Overview

Build and test Apify Actors on your local machine before deploying to the platform. The Apify CLI (apify run) emulates the platform environment locally — creating storage directories for datasets, key-value stores, and request queues — giving you a tight edit → run → inspect loop with no cloud round-trip.

Prerequisites

• npm install -g apify-cli (global CLI)
• apify login completed with valid token
• Node.js 18+

Authentication

The CLI authenticates with your Apify API token. Run apify login once (it stores the token under ~/.apify/), or export APIFY_TOKEN in the shell for non-interactive use. Local runs (apify run) do not require auth — only apify push / apify call reach the platform. Never commit the token or a plaintext .env containing it.

Actor Project Structure

my-actor/
├── .actor/
│   ├── actor.json          # Actor metadata and config
│   └── INPUT_SCHEMA.json   # Input schema (auto-generates UI on platform)
├── src/
│   └── main.ts             # Entry point
├── storage/                # Created by apify run (git-ignored)
│   ├── datasets/default/
│   ├── key_value_stores/default/
│   └── request_queues/default/
├── package.json
└── tsconfig.json

Instructions

Full config files and Actor source live in implementation.md; the high-level loop is:

Step 1: Create a New Actor Project

# Create from template (interactive)
apify create my-actor

# Or create from specific template
apify create my-actor --template project_cheerio_crawler_ts
# Templates: project_empty, project_cheerio_crawler_ts,
#   project_playwright_crawler_ts, project_puppeteer_crawler_ts

Step 2: Configure and code

Read and Edit the scaffolded .actor/actor.json (metadata + optional dataset view), define .actor/INPUT_SCHEMA.json (validates input and auto-generates the platform UI), and write your crawler in src/main.ts. See implementation.md for the complete actor.json, input schema, and a Cheerio-based main.ts that reads validated input and pushes structured rows via Actor.pushData().

Step 3: Run Locally

# Run with default input from storage/key_value_stores/default/INPUT.json
apify run

# Run with input from command line
apify run --input='{"startUrls":[{"url":"https://example.com"}],"maxPages":5}'

# View results
cat storage/datasets/default/*.json | jq '.'

Step 4: Provide Local Input

Create storage/key_value_stores/default/INPU…

Apify Performance Tuning

Overview

Optimize Apify Actors for speed, cost, and reliability. Covers Crawlee concurrency settings, memory profiling, proxy rotation strategies, request batching, and crawler selection for different workloads.

The workflow is a repeatable loop: measure a baseline, apply one lever, re-measure. The single highest-impact lever is usually crawler choice — swapping a browser crawler for CheerioCrawler on non-JS pages is a 5-10x speedup on its own. The full six-step walkthrough, with every code block, lives in references/implementation.md.

Prerequisites

• Existing Actor with measurable baseline performance
• Understanding of apify-sdk-patterns
• Access to Actor run stats in Apify Console
• APIFY_TOKEN in the environment for reading run stats via ApifyClient

Instructions

Work the levers in order. Each step is expanded — with copy-paste code — in the reference file linked below.

1. Measure a baseline. Pull runTimeSecs, requestsFinished, memAvgBytes, and usageTotalUsd from the run stats before changing anything. You cannot judge an optimization without a before number.
2. Choose the right crawler. HttpCrawler/CheerioCrawler for static HTML or JSON (low memory, fast); PlaywrightCrawler/PuppeteerCrawler only when the page genuinely needs JavaScript rendering.
3. Tune concurrency. Raise maxConcurrency for Cheerio (up to ~50); keep it low (~3-5) for browser crawlers because each page costs ~200MB. Let autoscaledPoolOptions adjust within the band.
4. Optimize memory. Push data immediately instead of accumulating arrays; for browser crawlers, block images/CSS/fonts in preNavigationHooks and cap concurrent browsers.
5. Right-size the memory allocation. Compute units bill on memory x duration — start low (512 MB for Cheerio) and only raise it if the Actor is memory-starved.
6. Rotate proxies and tune requests. Start on datacenter proxies, fall back to residential on 403/blocked; use a session pool for IP rotation and ban detection.

Full step-by-step walkthrough with all code: references/implementation.md.

The minimal starting skeleton — swap a browser crawler for Cheerio and push immediately:

import { CheerioCrawler } from 'crawlee';
import { Actor } from 'apify';

const crawler = new CheerioCrawler({
  maxConcurrency: 50,             // Cheerio is cheap — parallelize hard
  maxRequestsPerMinute: 300,      // But cap the rate to p…

Apify Production Checklist

Overview

Complete checklist for deploying Actors to the Apify platform and integrating them into production applications. Covers Actor configuration, scheduling, monitoring, alerting, and rollback. Work top to bottom: clear the pre-deployment gates, then run the six deploy steps, then wire the alert conditions.

Prerequisites

• Actor tested locally with apify run
• apify login configured with production token
• Familiarity with apify-core-workflow-a and apify-deploy-integration

Pre-Deployment Checklist

Actor Configuration

• [ ] .actor/actor.json has correct name, title, description
• [ ] INPUT_SCHEMA.json validates all required inputs
• [ ] Dockerfile uses pinned base image version (apify/actor-node:20, not latest)
• [ ] package-lock.json committed (deterministic installs)
• [ ] Memory set appropriately (start at 1024MB, tune after profiling)
• [ ] Timeout set with buffer (2x expected runtime)

Code Quality

• [ ] Actor.main() wraps entry point (handles init/exit/errors)
• [ ] failedRequestHandler logs failures without crashing Actor
• [ ] Input validation at Actor start (if (!input?.startUrls) throw ...)
• [ ] No hardcoded URLs, credentials, or magic numbers
• [ ] Proxy configured for target sites that block datacenter IPs
• [ ] maxRequestsPerCrawl set to prevent runaway costs

Data Output

• [ ] Dataset schema documented (consistent field names)
• [ ] SUMMARY key-value store record saved with run stats
• [ ] Large payloads chunked (9MB dataset push limit)
• [ ] PII sanitized before storage

Instructions

Read the Actor's .actor/actor.json, INPUT_SCHEMA.json, and Dockerfile first to confirm the pre-deployment gates above, then run the six deploy steps. Each step's full command and code block lives in references/implementation.md; the skeleton is below.

1. Deploy Actor — apify push, then apify builds ls to confirm the build, then apify actors call with a small production-like input to smoke-test on-platform.
2. Configure Scheduling — create a cron schedule with client.schedules().create({...}) (or Apify Console: Actors > Your Actor > Schedules). Set cronExpression, runInput, and runOptions (memory/timeout).
3. Set Up Webhooks — client.webhooks().create({...}) on …

Apify Rate Limits

Overview

The Apify API enforces rate limits per resource. The apify-client library auto-retries 429s (up to 8 times with exponential backoff), so most workloads never notice a limit. You reach for this skill when bulk operations, custom API calls, or large fan-outs push past what the built-in retry can absorb — you then batch, queue, stagger, and monitor to stay under the ceiling.

Full runnable code for every step is in implementation.md; combined scenarios are in examples.md.

Apify rate limit rules

"Per resource" means: calls to dataset A and dataset B each get 60 req/sec independently. Every response carries X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset (epoch seconds) headers.

Prerequisites

• An Apify account with API access and APIFY_TOKEN set in the environment.
• The apify-client package installed (npm install apify-client).
• For custom queuing: p-queue (npm install p-queue); crawlee for sleep and crawler-level concurrency.

Instructions

The workflow is five steps. Each is summarized here with its core lever; the full runnable code for every step is in implementation.md.

1. Understand built-in retries — apify-client already retries 429/500+ with exponential backoff. Tune maxRetries / minDelayBetweenRetriesMillis only when the defaults are wrong for your endpoint:

   import { ApifyClient } from 'apify-client';
   const client = new ApifyClient({
     token: process.env.APIFY_TOKEN,
     maxRetries: 5,                      // Default: 8
     minDelayBetweenRetriesMillis: 500,  // Default: 500
   });

1. Batch operations (biggest lever) — collapse per-item loops into one batched call (up to 9 MB), chunking only for very large datasets:

   await client.dataset(dsId).pushItems(items);   // 1 call, not N

1. Queue custom calls — gate raw API calls thr…

Apify Reference Architecture

Overview

Production-ready architecture patterns for applications built on Apify. Three patterns scale from a single scraper to a full-stack integration:

1. Standalone Actor — one scraper deployed to the Apify platform.
2. Multi-Actor Pipeline — a discover → scrape → transform chain of Actors.
3. Full-Stack Integration — an application using Apify as a data source behind a service layer.

This skill helps you choose the right pattern, lay out the directory structure, and wire the skeleton code. Full directory trees, diagrams, and code for every pattern live in references/architecture-patterns.md; the service layer, configuration loader, and health check live in references/implementation.md.

Prerequisites

• Runtime: Node.js >=18, TypeScript, and the Apify CLI (npm i -g apify-cli).
• Packages: apify + crawlee (inside an Actor), apify-client (calling Actors from an app), zod (input validation).
• Auth: an Apify API token. Set APIFY_TOKEN in the environment; the Apify SDK and apify-client read it automatically (or pass it explicitly to new ApifyClient({ token })). Never hardcode the token — inject it via env var and validate at startup.
• Access: Read and Grep the target repository so you can match the recommended layout against the code already on disk before proposing changes.

Instructions

1. Pick the pattern. One scraper → Pattern 1. A staged workflow that discovers, scrapes, then cleans → Pattern 2. An app that consumes scraped data → Pattern 3.
2. Grep the existing repo for apify, apify-client, and Actor.main to see what is already wired, so you extend rather than duplicate structure.
3. Lay out the directory from the pattern's tree in references/architecture-patterns.md. Keep routing, extraction, and validation in separate modules.
4. Add typed input validation with zod (see src/types.ts in the reference) so bad input fails fast at the Actor boundary instead of mid-crawl.
5. Isolate every Apify call behind a service layer (Pattern 3) using the ApifyService class in references/implementation.md — the rest of the app never imports apify-client directly.
6. Load configuration once at startup via lo…

Apify SDK Patterns

Overview

Production patterns for both the apify SDK (building Actors) and apify-client (calling Actors remotely). Covers Crawlee crawler selection, data storage, proxy configuration, and typed client wrappers. This skill gives you the essential skeletons inline; the full eight-pattern catalog and two worked scenarios live in references/ for progressive drill-down.

Prerequisites

• Install what you need: apify-client for calling Actors remotely, or apify + crawlee for building Actors. Both can coexist in one project.
• Set APIFY_TOKEN in the environment — read it via process.env.APIFY_TOKEN, never hard-code it. This is the only credential these patterns require (Apify uses a personal API token, not OAuth).
• TypeScript is recommended; every snippet here is typed and runs under ts-node or a compiled build.

Instructions

Use the two skeletons below to start, then reach into the reference catalog for the pattern that matches your task.

Pattern 1: Typed Client Singleton

Create one lazily-initialized, token-validated ApifyClient and reuse it everywhere. A resetClient() hook keeps it testable.

// src/apify/client.ts
import { ApifyClient } from 'apify-client';

let instance: ApifyClient | null = null;

export function getApifyClient(): ApifyClient {
  if (!instance) {
    const token = process.env.APIFY_TOKEN;
    if (!token) throw new Error('APIFY_TOKEN is required');
    instance = new ApifyClient({ token });
  }
  return instance;
}

// Reset for testing
export function resetClient(): void {
  instance = null;
}

Pattern 2: Crawlee Crawler Selection

Choose the crawler that matches the page, not the other way around:

import { CheerioCrawler, PlaywrightCrawler, PuppeteerCrawler } from 'crawlee';

// CHEERIO — Fast, lightweight, no JavaScript rendering
// Use for: static HTML, server-rendered pages, APIs
// PLAYWRIGHT — Full browser, all engines, modern API
// Use for: SPAs, JavaScript-heavy pages, complex interactions
// PUPPETEER — Chromium-only browser automation
// Use for: when you need Chromium specifically or legacy Puppeteer code

Patterns 3–8: Full catalog

The remaining six patterns are moved verbatim into patterns.md so this file stays scannable. Pick the one you need:

• Pattern 3 — Actor lifecycle with error handling: Actor.main() wrapping input validation, conditional proxy, and a failedRequestHandler.
• Pattern 4 — Dataset operations: push from inside an Actor; list/create/download from an external app.
• …

Apify Security Basics

Overview

Security best practices for Apify API tokens, Actor data, proxy credentials, and webhook verification. Apify uses personal API tokens (prefixed apify_api_) for all authentication. Because a single token grants full account access with no per-token scoping, token hygiene is the whole game.

Prerequisites

• Apify account with Console access
• Understanding of environment variables
• Access to your deployment platform's secrets management

Token Architecture

Apify uses a single API token per user account for full API access. There is no scope-based permission system per token, so token security is critical.

Instructions

Follow the six hardening steps in order. Each has a lean summary below; the full copy-paste code for every step is in references/implementation.md.

1. Secure token storage — keep the token in .env (never hardcoded) and add .env, .env.*.local, and storage/ to .gitignore. Validate presence at startup so the app fails fast:

   function requireToken(): string {
     const token = process.env.APIFY_TOKEN;
     if (!token) throw new Error('APIFY_TOKEN is required');
     if (!token.startsWith('apify_api_')) console.warn('unexpected token prefix');
     return token;
   }

1. Per-environment token isolation — separate tokens (ideally separate accounts) for dev / staging / prod, injected via each platform's secret store (gh secret set, vercel env add, GCP Secret Manager).
2. Token rotation — generate the new token first (old stays valid), push to every environment, verify it authenticates, then revoke the old one.
3. Webhook payload verification — Apify does not sign webhooks; confirm the run ID in the payload actually exists, or gate on a shared URL secret compared with crypto.timingSafeEqual.
4. Actor data security — redact sensitive fields before pushData; keep datasets named and private (no public sharing).
5. Proxy security — never log proxyConfig.newUrl() (it embeds the proxy password); log the proxy group only.

See references/implementation.md for the complete code of every step, and …

Apify Upgrade & Migration

Overview

Guide for upgrading apify, apify-client, and crawlee packages. The biggest migration in Apify's history was SDK v2 to v3, which split crawling functionality into the crawlee package. This skill covers that migration plus general upgrade procedures. Read and edit source files with Grep/Read/Edit to apply the rename-heavy changes, then verify with the packaged script.

Prerequisites

Before starting, confirm the working tree is in a recoverable state:

• A dedicated git branch for the upgrade, so a bad bump can be reverted cleanly.
• A runnable test suite (npm test) plus a build step (npm run build) to catch TypeScript interface changes.
• The current installed versions recorded (npm list apify apify-client crawlee) so rollback targets are known.
• APIFY_TOKEN in the environment if the verification script's API-connection check will run.

Instructions

Step 1: Check Current Versions

# Check installed versions
npm list apify apify-client crawlee 2>/dev/null

# Check latest available versions
npm view apify version
npm view apify-client version
npm view crawlee version

# Check for outdated packages
npm outdated apify apify-client crawlee

Step 2: Create Upgrade Branch

git checkout -b upgrade/apify-packages

Step 3: Upgrade Packages

# Upgrade to latest
npm install apify@latest crawlee@latest apify-client@latest

# Or upgrade to specific version
npm install apify@3.2.0 crawlee@3.11.0

# Check for peer dependency issues
npm ls 2>&1 | grep "ERESOLVE\|peer dep"

Step 4: Apply Code Changes

If crossing the v2→v3 boundary, use Grep to find every Apify. reference and Edit each call site. The full before/after set — imports, Actor.main, crawler option renames (handlePageFunction → requestHandler), proxy config, request queues, and the new router pattern — is in the v2-to-v3 migration guide. Minimal shape:

// v2
import Apify from 'apify';
const { CheerioCrawler } = Apify;
// v3
import { Actor } from 'apify';
import { CheerioCrawler } from 'crawlee';

Step 5: Verify and Test

npm test
npm run build  # Catch TypeScript errors

Then run the packaged verification script from verify-and-rollback.md, which checks imports resolve, the client connects, and a crawler instantiates.

Output

A successful u…

Apify Webhooks & Events

Overview

Configure webhooks to receive notifications when Actor runs complete, fail, or time out. Apify supports both persistent webhooks (for all runs of an Actor) and ad-hoc webhooks (for a single run). Event-driven architecture is the recommended pattern for production Apify integrations.

Prerequisites

• npm install apify-client in your project (and express if you build an HTTP handler)
• An API token in APIFY_TOKEN — read it from the environment (process.env.APIFY_TOKEN) or pass Authorization: Bearer $APIFY_TOKEN on REST calls, never hard-code it
• A public HTTPS endpoint for Apify to POST to (use ngrok while developing)
• Familiarity with apify-sdk-patterns

Event Types

Instructions

Step 1: Create a Persistent Webhook

Persistent webhooks fire for every run of an Actor. Set condition.actorId, list the eventTypes you care about, and shape the delivered body with payloadTemplate:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const webhook = await client.webhooks().create({
  eventTypes: ['ACTOR.RUN.SUCCEEDED', 'ACTOR.RUN.FAILED', 'ACTOR.RUN.TIMED_OUT'],
  condition: { actorId: 'YOUR_ACTOR_ID' },
  requestUrl: 'https://your-app.com/api/webhooks/apify',
  payloadTemplate: JSON.stringify({
    eventType: '{{eventType}}',
    actorRunId: '{{actorRunId}}',
    defaultDatasetId: '{{resource.defaultDatasetId}}',
    status: '{{resource.status}}',
    statusMessage: '{{resource.statusMessage}}',
  }),
  isAdHoc: false,
});

console.log(`Webhook created: ${webhook.id}`);

The full payload template (all run fields) and the complete variable table are in payload templates & local testing.

Step 2: Use Ad-Hoc Webhooks for Single Runs

Ad-hoc webhooks are created at run time and fire only for that specific run — pass a webhooks array when starting the Actor: