Apify Install & Auth

Overview

Set up the Apify ecosystem: the apify-client JS library (for calling Actors remotely), the apify SDK (for building Actors), the Apify CLI (for deploying), and Crawlee (for crawling). Each package serves a different purpose — install only what the task needs, then wire up a single API token.

Package Map

Prerequisites

• Node.js 18+ (required by SDK v3+)
• Apify account at https://console.apify.com
• API token from Settings > Integrations in Apify Console

Instructions

Step 1: Install Packages

# For CALLING existing Actors from your app:
npm install apify-client

# For BUILDING your own Actors:
npm install apify crawlee

# For CLI deployment:
npm install -g apify-cli

Step 2: Configure Authentication

Pick one. Read any existing .env first so you do not clobber it, then Write the token in.

# Option A: Environment variable (recommended for apps)
export APIFY_TOKEN="apify_api_YOUR_TOKEN_HERE"

# Option B: .env file (add .env to .gitignore)
echo 'APIFY_TOKEN=apify_api_YOUR_TOKEN_HERE' >> .env

# Option C: CLI login (for interactive development)
apify login
# Paste your token when prompted

The APIFY_TOKEN env var is auto-detected by both apify-client and the apify SDK. For every place a token can be supplied (constructor option, header, precedence) plus the full platform env-var list, see authentication reference.

Step 3: Verify Connection

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

// List your Actors to confirm auth works
const { items } = await client.actors().list();
console.log(`Authenticated. You have ${items.length} Actors.`);

Step 4: Verify CLI (if installed)

apify login --token YOUR_TOKEN
apify info  # Shows your account info

Output

A working, authentica

Apify Local Dev Loop

Overview

Build and test Apify Actors on your local machine before deploying to the platform. The Apify CLI (apify run) emulates the platform environment locally — creating storage directories for datasets, key-value stores, and request queues — giving you a tight edit → run → inspect loop with no cloud round-trip.

Prerequisites

• npm install -g apify-cli (global CLI)
• apify login completed with valid token
• Node.js 18+

Authentication

The CLI authenticates with your Apify API token. Run apify login once (it stores

the token under ~/.apify/), or export APIFY_TOKEN in the shell for

non-interactive use. Local runs (apify run) do not require auth — only

apify push / apify call reach the platform. Never commit the token or a

plaintext .env containing it.

Actor Project Structure

my-actor/
├── .actor/
│   ├── actor.json          # Actor metadata and config
│   └── INPUT_SCHEMA.json   # Input schema (auto-generates UI on platform)
├── src/
│   └── main.ts             # Entry point
├── storage/                # Created by apify run (git-ignored)
│   ├── datasets/default/
│   ├── key_value_stores/default/
│   └── request_queues/default/
├── package.json
└── tsconfig.json

Instructions

Full config files and Actor source live in

implementation.md; the high-level loop is:

Step 1: Create a New Actor Project

# Create from template (interactive)
apify create my-actor

# Or create from specific template
apify create my-actor --template project_cheerio_crawler_ts
# Templates: project_empty, project_cheerio_crawler_ts,
#   project_playwright_crawler_ts, project_puppeteer_crawler_ts

Step 2: Configure and code

Read and Edit the scaffolded .actor/actor.json (metadata + optional dataset

view), define .actor/INPUT_SCHEMA.json (validates input and auto-generates the

platform UI), and write your crawler in src/main.ts. See

implementation.md for the complete actor.json,

input schema, and a Cheerio-based main.ts that reads validated input and pushes

structured rows via Actor.pushData().

Step 3: Run Locally

# Run with default input from storage/key_value_stores/default/INPUT.json
apify run

# Run with input from command line
apify run --input='{"startUrls":[{"url":"https://example.com"}],"maxPages":5}'

# View results
cat storage/datasets/default/*.json | jq '.'

Step 4: Provide Local Input<

Apify Performance Tuning

Overview

Optimize Apify Actors for speed, cost, and reliability. Covers Crawlee concurrency settings, memory profiling, proxy rotation strategies, request batching, and crawler selection for different workloads.

The workflow is a repeatable loop: measure a baseline, apply one lever, re-measure. The single highest-impact lever is usually crawler choice — swapping a browser crawler for CheerioCrawler on non-JS pages is a 5-10x speedup on its own. The full six-step walkthrough, with every code block, lives in references/implementation.md.

Prerequisites

• Existing Actor with measurable baseline performance
• Understanding of apify-sdk-patterns
• Access to Actor run stats in Apify Console
• APIFY_TOKEN in the environment for reading run stats via ApifyClient

Instructions

Work the levers in order. Each step is expanded — with copy-paste code — in the reference file linked below.

1. Measure a baseline. Pull runTimeSecs, requestsFinished, memAvgBytes, and usageTotalUsd from the run stats before changing anything. You cannot judge an optimization without a before number.
2. Choose the right crawler. HttpCrawler/CheerioCrawler for static HTML or JSON (low memory, fast); PlaywrightCrawler/PuppeteerCrawler only when the page genuinely needs JavaScript rendering.
3. Tune concurrency. Raise maxConcurrency for Cheerio (up to ~50); keep it low (~3-5) for browser crawlers because each page costs ~200MB. Let autoscaledPoolOptions adjust within the band.
4. Optimize memory. Push data immediately instead of accumulating arrays; for browser crawlers, block images/CSS/fonts in preNavigationHooks and cap concurrent browsers.
5. Right-size the memory allocation. Compute units bill on memory x duration — start low (512 MB for Cheerio) and only raise it if the Actor is memory-starved.
6. Rotate proxies and tune requests. Start on datacenter proxies, fall back to residential on 403/blocked; use a session pool for IP rotation and ban detection.

Full step-by-step walkthrough with all code: references/implementation.md.

The minimal starting skeleton — swap a browser crawler for Cheerio and push immediately:

import { CheerioCrawler } from 'crawlee';
import { Actor } from 'apify';

const crawler = new CheerioCrawler({
  maxConcurrency: 50,             // Cheerio is cheap — parallelize hard
  maxRequestsPerMinute: 300,      // But cap the rate to protect the tar