CAST AI CI Integration

Overview

Add CAST AI cost validation to CI/CD pipelines: verify savings thresholds, validate Terraform plans before apply, and gate deployments on autoscaler health.

Prerequisites

• GitHub Actions enabled
• CAST AI API key stored as repository secret
• Terraform state accessible from CI (if using Terraform)

Instructions

Step 1: GitHub Actions -- Savings Gate

# .github/workflows/castai-check.yml
name: CAST AI Cost Check

on:
  pull_request:
    paths: ["terraform/**", "k8s/**"]
  schedule:
    - cron: "0 8 * * 1"  # Weekly Monday report

jobs:
  cost-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Check CAST AI Savings
        env:
          CASTAI_API_KEY: ${{ secrets.CASTAI_API_KEY }}
          CASTAI_CLUSTER_ID: ${{ secrets.CASTAI_CLUSTER_ID }}
        run: |
          SAVINGS=$(curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
            "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/savings")

          PERCENT=$(echo "$SAVINGS" | jq -r '.savingsPercentage')
          MONTHLY=$(echo "$SAVINGS" | jq -r '.monthlySavings')

          echo "### CAST AI Savings Report" >> $GITHUB_STEP_SUMMARY
          echo "- Monthly savings: \$${MONTHLY}" >> $GITHUB_STEP_SUMMARY
          echo "- Savings percentage: ${PERCENT}%" >> $GITHUB_STEP_SUMMARY

          # Fail if savings drop below threshold
          if (( $(echo "$PERCENT < 10" | bc -l) )); then
            echo "WARNING: Savings below 10% threshold"
            exit 1
          fi

      - name: Verify Agent Health
        env:
          CASTAI_API_KEY: ${{ secrets.CASTAI_API_KEY }}
          CASTAI_CLUSTER_ID: ${{ secrets.CASTAI_CLUSTER_ID }}
        run: |
          STATUS=$(curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
            "https://api.cast.ai/v1/kubernetes/external-clusters/${CASTAI_CLUSTER_ID}" \
            | jq -r '.agentStatus')

          if [ "$STATUS" != "online" ]; then
            echo "CAST AI agent is ${STATUS}, expected online"
            exit 1
          fi

Step 2: Terraform Plan Validation

  terraform-plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Init & Plan
        working-directory: terraform/
        env:
          CASTAI_API_TOKEN: ${{ secrets.CASTAI_API_KEY }}
        run: |
          terraform init
          terraform plan -var-file=environments/prod.tfvars \
            -out=plan.tfplan -no-color | tee plan-output.txt

      - name: Check for Destructive Changes
        run: |
          if grep -q …

CAST AI Common Errors

Overview

Diagnostic guide for the 10 most common CAST AI issues, covering agent connectivity, API errors, autoscaler failures, and node provisioning problems.

Prerequisites

• kubectl access to the cluster
• CASTAI_API_KEY configured
• Access to CAST AI console for log correlation

Instructions

Identify the affected cluster and environment first, collect the minimum redacted evidence needed to classify the symptom, then use the matching runbook below. Change one control at a time, observe health and workload impact over the defined window, and keep production changes behind the approved change process. Escalate rather than guessing when the issue involves credentials, unexpected node termination, data loss risk, or an unclear provider-side failure.

Error Handling

Error Reference

1. Agent Pod CrashLoopBackOff

kubectl get pods -n castai-agent
kubectl logs -n castai-agent deployment/castai-agent --tail=50

Causes and fixes:

• Invalid API key: Regenerate at console.cast.ai > API
• Wrong provider: Set --set provider=eks|gke|aks correctly in Helm
• RBAC missing: Apply the required ClusterRole and ClusterRoleBinding
• Network blocked: Ensure outbound HTTPS to api.cast.ai is allowed

2. Agent Shows "Disconnected" in Console

# Check agent heartbeat
kubectl logs -n castai-agent deployment/castai-agent | grep -i "heartbeat\|connect\|error"

# Verify network connectivity from inside the cluster
kubectl run castai-debug --image=curlimages/curl --rm -it --restart=Never -- \
  curl -s -o /dev/null -w "%{http_code}" https://api.cast.ai/v1/kubernetes/external-clusters

Fix: Restart the agent pod: kubectl rollout restart deployment/castai-agent -n castai-agent

3. API Returns 401 Unauthorized

# Test API key
curl -s -o /dev/null -w "%{http_code}" \
  -H …

CAST AI Core Workflow: Autoscaler & Policies

Overview

Primary workflow for CAST AI: configure autoscaler policies to optimize cluster costs. Covers enabling spot instances, configuring the node downscaler and evictor, setting cluster CPU/memory limits, and creating node templates for workload-specific requirements.

Prerequisites

• Completed castai-install-auth with Phase 2 (cluster controller + evictor)
• CASTAI_API_KEY and CASTAI_CLUSTER_ID set
• Cluster in "ready" status

Instructions

Step 1: Read Current Policies

curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
  | jq .

Step 2: Enable Cost-Optimized Autoscaling

curl -X PUT -H "X-API-Key: ${CASTAI_API_KEY}" \
  -H "Content-Type: application/json" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
  -d '{
    "enabled": true,
    "unschedulablePods": {
      "enabled": true,
      "headroom": {
        "cpuPercentage": 10,
        "memoryPercentage": 10,
        "enabled": true
      }
    },
    "nodeDownscaler": {
      "enabled": true,
      "emptyNodes": {
        "enabled": true,
        "delaySeconds": 180
      }
    },
    "spotInstances": {
      "enabled": true,
      "clouds": ["aws"],
      "spotDiversityEnabled": true,
      "spotDiversityPriceIncreaseLimitPercent": 20
    },
    "clusterLimits": {
      "enabled": true,
      "cpu": {
        "minCores": 4,
        "maxCores": 100
      }
    }
  }'

Step 3: Configure Node Templates via Terraform

resource "castai_node_template" "spot_workers" {
  cluster_id = castai_eks_cluster.this.id
  name       = "spot-workers"
  is_default = false
  is_enabled = true

  constraints {
    min_cpu               = 2
    max_cpu               = 16
    min_memory            = 4096
    max_memory            = 65536
    spot                  = true
    use_spot_fallbacks    = true
    fallback_restore_rate_seconds = 600

    instance_families {
      include = ["m5", "m6i", "c5", "c6i", "r5", "r6i"]
    }

    architectures = ["amd64"]
  }

  custom_labels = {
    "workload-type" = "batch"
  }
}

resource "castai_node_template" "gpu_ondemand" {
  cluster_id = castai_eks_cluster.this.id
  name       = "gpu-ondemand"
  is_default = false
  is_en…

CAST AI Core Workflow: Workload Autoscaler

Overview

CAST AI Workload Autoscaler right-sizes pod resource requests based on actual usage, reducing over-provisioning without manual VPA tuning. This skill covers enabling the workload autoscaler, configuring scaling policies per workload, and using annotations for fine-grained control.

Prerequisites

• Completed castai-core-workflow-a (cluster-level policies)
• CAST AI agent v1.60+ installed
• Workload Autoscaler enabled in CAST AI console

Instructions

Step 1: Install Workload Autoscaler Components

helm upgrade --install castai-workload-autoscaler \
  castai-helm/castai-workload-autoscaler \
  -n castai-agent \
  --set castai.apiKey="${CASTAI_API_KEY}" \
  --set castai.clusterID="${CASTAI_CLUSTER_ID}"

Step 2: Query Workload Recommendations

# Get resource recommendations for a specific workload
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/workload-autoscaling/clusters/${CASTAI_CLUSTER_ID}/workloads" \
  | jq '.items[] | {
    name: .workloadName,
    namespace: .namespace,
    currentCpu: .currentCpuRequest,
    recommendedCpu: .recommendedCpuRequest,
    currentMemory: .currentMemoryRequest,
    recommendedMemory: .recommendedMemoryRequest,
    savingsPercent: .estimatedSavingsPercent
  }'

Step 3: Configure Per-Workload Policies via Annotations

# Add annotations to deployments for CAST AI workload autoscaler
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api
  annotations:
    # Enable workload autoscaling
    autoscaling.cast.ai/enabled: "true"
    # CPU configuration
    autoscaling.cast.ai/cpu-min: "100m"
    autoscaling.cast.ai/cpu-max: "4000m"
    autoscaling.cast.ai/cpu-headroom: "15"
    # Memory configuration
    autoscaling.cast.ai/memory-min: "128Mi"
    autoscaling.cast.ai/memory-max: "8Gi"
    autoscaling.cast.ai/memory-headroom: "20"
    # Apply changes automatically vs recommendation-only
    autoscaling.cast.ai/apply-type: "immediate"
spec:
  template:
    spec:
      containers:
        - name: api
          resources:
            requests:
              cpu: "500m"      # Will be auto-adjusted by CAST AI
              memory: "512Mi"  # Will be auto-adjusted by CAST AI

Step 4: Create a Scaling Policy via API

curl -X POST -H "X-API-Key: ${CASTAI_API_KEY}" \
  -H "Content-Type: application/json" \
  "https://api.cast.ai/v1/workload-autoscaling/clusters/${CASTAI_CLUSTER_ID}/policies" \
  -d '{
    "name": "cost-optimized",
    "applyType": "IMMEDIAT…

CAST AI Cost Tuning

Overview

Maximize Kubernetes cost savings through CAST AI: spot instance strategies, workload right-sizing, cluster hibernation, and savings tracking. Typical savings: 50-70% on cloud compute costs.

Prerequisites

• CAST AI Phase 2 enabled with full automation
• Savings report available (requires 24h+ of data)
• Understanding of workload criticality tiers

Instructions

Step 1: Analyze Current Savings

# Get savings breakdown
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/savings" \
  | jq '{
    currentMonthlyCost: .currentMonthlyCost,
    optimizedMonthlyCost: .optimizedMonthlyCost,
    monthlySavings: .monthlySavings,
    savingsPercentage: .savingsPercentage,
    spotSavings: .spotSavings,
    rightSizingSavings: .rightSizingSavings
  }'

Step 2: Maximize Spot Usage

# Enable aggressive spot with diversity and fallbacks
curl -X PUT -H "X-API-Key: ${CASTAI_API_KEY}" \
  -H "Content-Type: application/json" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
  -d '{
    "enabled": true,
    "spotInstances": {
      "enabled": true,
      "clouds": ["aws"],
      "spotDiversityEnabled": true,
      "spotDiversityPriceIncreaseLimitPercent": 20,
      "spotBackups": {
        "enabled": true,
        "spotBackupRestoreRateSeconds": 600
      }
    }
  }'

Spot allocation strategy by workload tier:

Step 3: Workload Right-Sizing

# Get resource waste analysis
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/workload-autoscaling/clusters/${CASTAI_CLUSTER_ID}/workloads" \
  | jq '[.items[] | select(.estimatedSavingsPercent > 20) | {
    name: .workloadName,
    namespace: .namespace,
    wastedCpu: (.currentCpuRequest - .recommendedCpuRequest),
    wastedMemory: (.currentMemoryRequest - .recommendedMemoryRequest),
    savingsPercent: .estimatedSavingsPercent
  }] | sort_by(-.savingsPercent) | .[0:10]'

Step 4: Cluster Hibernation (Dev/Staging)

…

CAST AI Debug Bundle

Overview

Collect all CAST AI component logs, cluster state, and configuration into a single archive for troubleshooting or support tickets. The bundle captures agent status, Helm releases, autoscaler policies, node inventory, and recent events.

Prerequisites

• kubectl access to the cluster running CAST AI
• CASTAI_API_KEY and CASTAI_CLUSTER_ID configured
• helm installed

Instructions

Step 1: Run the Debug Bundle Script

#!/bin/bash
# castai-debug-bundle.sh
set -euo pipefail

BUNDLE_DIR="castai-debug-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BUNDLE_DIR"

echo "=== CAST AI Debug Bundle ===" | tee "$BUNDLE_DIR/summary.txt"
echo "Generated: $(date -u)" | tee -a "$BUNDLE_DIR/summary.txt"
echo "Cluster ID: ${CASTAI_CLUSTER_ID:-unknown}" | tee -a "$BUNDLE_DIR/summary.txt"

# 1. Helm releases
echo "--- Helm Releases ---" >> "$BUNDLE_DIR/summary.txt"
helm list -n castai-agent -o yaml > "$BUNDLE_DIR/helm-releases.yaml" 2>&1

# 2. Pod status
echo "--- Pod Status ---" >> "$BUNDLE_DIR/summary.txt"
kubectl get pods -n castai-agent -o wide > "$BUNDLE_DIR/pod-status.txt" 2>&1

# 3. Agent logs (last 200 lines each)
for deploy in castai-agent cluster-controller castai-evictor castai-spot-handler castai-workload-autoscaler; do
  kubectl logs -n castai-agent "deployment/$deploy" --tail=200 \
    > "$BUNDLE_DIR/${deploy}-logs.txt" 2>&1 || echo "Not found: $deploy" >> "$BUNDLE_DIR/summary.txt"
done

# 4. Cluster events (CAST AI related)
kubectl get events -n castai-agent --sort-by='.lastTimestamp' \
  > "$BUNDLE_DIR/events.txt" 2>&1

# 5. Node inventory
kubectl get nodes -o wide > "$BUNDLE_DIR/nodes.txt" 2>&1

# 6. API cluster status (redact API key from output)
if [ -n "${CASTAI_API_KEY:-}" ] && [ -n "${CASTAI_CLUSTER_ID:-}" ]; then
  curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
    "https://api.cast.ai/v1/kubernetes/external-clusters/${CASTAI_CLUSTER_ID}" \
    | jq '{name, status, agentStatus, providerType, createdAt}' \
    > "$BUNDLE_DIR/cluster-status.json" 2>&1

  curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
    "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
    > "$BUNDLE_DIR/policies.json" 2>&1
fi

# 7. RBAC check
kubectl get clusterrole -l app.kubernetes.io/managed-by=castai \
  > "$BUNDLE_DIR/rbac.txt" 2>&1

# 8. Package bundle
tar -czf "$BUNDLE_DIR.tar.gz" "$BUNDLE_DIR"
rm -rf "$BUNDLE_DIR"
echo "Bundl…

CAST AI Deploy Integration

Overview

Deploy CAST AI to EKS, GKE, and AKS clusters using official Terraform modules. Each cloud provider has a dedicated CAST AI module that handles IAM roles, node configuration, and autoscaler setup.

Prerequisites

• Terraform 1.0+
• CAST AI Full Access API key
• Cloud provider credentials configured
• Existing Kubernetes cluster

Instructions

EKS Deployment

# main.tf -- EKS cluster onboarding
module "castai_eks" {
  source  = "castai/eks-cluster/castai"
  version = "~> 3.0"

  api_token           = var.castai_api_token
  aws_account_id      = data.aws_caller_identity.current.account_id
  aws_cluster_region  = var.region
  aws_cluster_name    = var.cluster_name

  # IAM role for CAST AI to manage nodes
  aws_instance_profile_arn = aws_iam_instance_profile.castai.arn

  # Autoscaler configuration
  autoscaler_policies_json = jsonencode({
    enabled = true
    unschedulablePods = { enabled = true }
    nodeDownscaler = {
      enabled = true
      emptyNodes = { enabled = true, delaySeconds = 300 }
    }
    spotInstances = {
      enabled = true
      spotDiversityEnabled = true
    }
    clusterLimits = {
      enabled = true
      cpu = { minCores = 4, maxCores = 200 }
    }
  })

  # Node templates
  default_node_configuration = module.castai_eks.castai_node_configurations["default"]
}

GKE Deployment

module "castai_gke" {
  source  = "castai/gke-cluster/castai"
  version = "~> 2.0"

  api_token            = var.castai_api_token
  project_id           = var.gcp_project_id
  gke_cluster_name     = var.cluster_name
  gke_cluster_location = var.region

  gke_credentials = base64decode(
    google_container_cluster.this.master_auth[0].cluster_ca_certificate
  )

  autoscaler_policies_json = jsonencode({
    enabled = true
    unschedulablePods = { enabled = true }
    nodeDownscaler = {
      enabled = true
      emptyNodes = { enabled = true, delaySeconds = 300 }
    }
  })
}

AKS Deployment

module "castai_aks" {
  source  = "castai/aks/castai"
  version = "~> 1.0"

  api_token              = var.castai_api_token
  aks_cluster_name       = var.cluster_name
  aks_cluster_region     = var.region
  node_resource_group    = azurerm_kubernetes_cluster.this.node_resource_group
  azure_subscription_id  = data.azurerm_subscription.current.subscription_id
  azure_tenant_id        = data.azurerm_client_config.current.tenant_id

  autoscaler_policies_json = jsonencode({
    enabled = true
    unschedulablePods = { enabled = true }
    spotInstances = { enabled = true }
  })
}

Multi-Cluster Deployment Pattern

# Deploy CAST AI across all cluste…

CAST AI Hello World

Overview

First API calls against the CAST AI REST API: list connected clusters, retrieve the savings report, and inspect node inventory. All examples use curl with the X-API-Key header -- no SDK required.

Prerequisites

• Completed castai-install-auth setup
• CASTAI_API_KEY environment variable set
• At least one cluster connected to CAST AI

Instructions

Step 1: List Connected Clusters

curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  https://api.cast.ai/v1/kubernetes/external-clusters \
  | jq '.items[] | {id, name, status, providerType}'

Expected output:

{
  "id": "abc123-def456",
  "name": "production-eks",
  "status": "ready",
  "providerType": "eks"
}

Step 2: Get Cluster Savings Report

export CASTAI_CLUSTER_ID="your-cluster-id"

# Current month savings
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/savings" \
  | jq '{
    monthlySavings: .monthlySavings,
    savingsPercentage: .savingsPercentage,
    currentCost: .currentMonthlyCost,
    optimizedCost: .optimizedMonthlyCost
  }'

Step 3: List Cluster Nodes

curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/external-clusters/${CASTAI_CLUSTER_ID}/nodes" \
  | jq '.items[] | {
    name: .name,
    instanceType: .instanceType,
    lifecycle: .lifecycle,
    cpu: .allocatableCpu,
    memory: .allocatableMemory,
    zone: .zone
  }'

Step 4: Check Autoscaler Policies

curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
  | jq '{
    enabled: .enabled,
    unschedulablePods: .unschedulablePods.enabled,
    nodeDownscaler: .nodeDownscaler.enabled,
    spotInstances: .spotInstances.enabled
  }'

Output

The first read-only run returns a list of connected clusters, a monthly-savings snapshot, node inventory, and autoscaler-policy state. Save only the fields needed for the onboarding record; cluster names, account metadata, and spend data should be treated as environment-sensitive operational information.

Examples

Run the cluster-list command against a non-production key, select one expected cluster ID, then request its node inventory and policy state. A successful onboarding result shows the expected cluster name and an online agent; a 401 or offline agent is a stop condition, not a reason to retry with a broader key.

Err…

CAST AI Install & Auth

Overview

Connect a Kubernetes cluster (EKS, GKE, AKS, or KOPS) to CAST AI for cost optimization, autoscaling, and security scanning. Covers API key generation, Helm chart installation of the CAST AI agent, and Terraform provider setup.

Prerequisites

• A running Kubernetes cluster (EKS, GKE, AKS, or KOPS)
• kubectl configured with cluster admin access
• helm v3 installed
• A CAST AI account at https://console.cast.ai

Instructions

Step 1: Generate an API Key

Log in to https://console.cast.ai and navigate to API > API Access Keys. Create a Full Access key for Terraform-managed clusters, or Read-Only for monitoring-only.

export CASTAI_API_KEY="your-api-key-here"

# Verify the key works
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  https://api.cast.ai/v1/kubernetes/external-clusters | jq '.items | length'

Step 2: Install the CAST AI Agent via Helm

# Add the CAST AI Helm repository
helm repo add castai-helm https://castai.github.io/helm-charts
helm repo update

# Install the read-only monitoring agent (Phase 1)
helm upgrade --install castai-agent castai-helm/castai-agent \
  -n castai-agent --create-namespace \
  --set apiKey="${CASTAI_API_KEY}" \
  --set provider="eks"  # eks | gke | aks

kubectl get pods -n castai-agent

Step 3: Enable Full Automation (Phase 2)

export CASTAI_CLUSTER_ID="your-cluster-id"

# Cluster controller -- manages node lifecycle
helm upgrade --install cluster-controller castai-helm/castai-cluster-controller \
  -n castai-agent \
  --set castai.apiKey="${CASTAI_API_KEY}" \
  --set castai.clusterID="${CASTAI_CLUSTER_ID}"

# Evictor -- consolidates underutilized nodes
helm upgrade --install castai-evictor castai-helm/castai-evictor \
  -n castai-agent \
  --set castai.apiKey="${CASTAI_API_KEY}" \
  --set castai.clusterID="${CASTAI_CLUSTER_ID}"

# Spot handler -- graceful spot instance interruption
helm upgrade --install castai-spot-handler castai-helm/castai-spot-handler \
  -n castai-agent \
  --set castai.provider="eks" \
  --set castai.clusterID="${CASTAI_CLUSTER_ID}"

Step 4: Terraform Provider (Alternative)

terraform {
  required_providers {
    castai = {
      source  = "castai/castai"
      version = "~> 7.0"
    }
  }
}

provider "castai" {
  api_token = var.castai_api_token
}

variable "castai_api_token" {
  type      = string
  sensitive = true
}

resource "castai_eks_cluster" "this" {
  account_id = data.aws_caller_identity.current.account_id
  region     = var.aws_…

CAST AI Local Dev Loop

Overview

Fast iteration workflow for CAST AI integrations: test autoscaler policies in a dev cluster, validate Terraform modules before applying to production, and use the CAST AI API to measure savings impact during development.

Prerequisites

• Completed castai-install-auth setup
• A development Kubernetes cluster (kind, minikube, or cloud dev cluster)
• kubectl, helm, and optionally terraform installed

Instructions

Step 1: Project Structure

my-castai-infra/
├── terraform/
│   ├── environments/
│   │   ├── dev.tfvars
│   │   ├── staging.tfvars
│   │   └── prod.tfvars
│   ├── modules/
│   │   └── castai-cluster/
│   │       ├── main.tf
│   │       ├── variables.tf
│   │       └── outputs.tf
│   └── main.tf
├── policies/
│   ├── dev-policy.json
│   └── prod-policy.json
├── scripts/
│   ├── check-savings.sh
│   └── validate-policies.sh
├── .env.dev            # Dev API key (git-ignored)
└── .env.example

Step 2: Dev Cluster with Relaxed Policies

# Connect your dev cluster (read-only first)
helm upgrade --install castai-agent castai-helm/castai-agent \
  -n castai-agent --create-namespace \
  --set apiKey="${CASTAI_API_KEY_DEV}" \
  --set provider="eks"

# Apply development-safe autoscaler policy
curl -X PUT -H "X-API-Key: ${CASTAI_API_KEY_DEV}" \
  -H "Content-Type: application/json" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
  -d '{
    "enabled": true,
    "unschedulablePods": { "enabled": true },
    "nodeDownscaler": {
      "enabled": true,
      "emptyNodes": { "enabled": true, "delaySeconds": 300 }
    },
    "clusterLimits": {
      "enabled": true,
      "cpu": { "minCores": 2, "maxCores": 16 }
    }
  }'

Step 3: Quick Savings Check Script

#!/bin/bash
# scripts/check-savings.sh
set -euo pipefail

API_KEY="${CASTAI_API_KEY_DEV}"
CLUSTER_ID="${CASTAI_CLUSTER_ID}"

echo "=== CAST AI Dev Cluster Savings ==="
curl -s -H "X-API-Key: ${API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CLUSTER_ID}/savings" \
  | jq '{
    monthlySavings: .monthlySavings,
    percentage: .savingsPercentage,
    spotNodes: [.nodes[] | select(.lifecycle == "spot")] | length,
    totalNodes: [.nodes[]] | length
  }'

Step 4: Terraform Plan-Apply Loop

# Plan with dev variables
cd terraform/
terraform plan -var-file=environments/dev.tfvars -out=plan.tfplan

# Apply and check CAST AI result
terraform apply plan.tfplan

# Verify policies too…

CAST AI Performance Tuning

Overview

Tune CAST AI for faster node provisioning, more responsive autoscaling, and efficient API usage. Covers headroom configuration, instance family selection, and API caching for multi-cluster dashboards.

Prerequisites

• CAST AI Phase 2 (full automation) enabled
• Understanding of workload scheduling patterns
• Access to autoscaler policy configuration

Instructions

Step 1: Optimize Node Provisioning Speed

# Configure headroom for proactive scaling (avoids waiting for pending pods)
curl -X PUT -H "X-API-Key: ${CASTAI_API_KEY}" \
  -H "Content-Type: application/json" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
  -d '{
    "enabled": true,
    "unschedulablePods": {
      "enabled": true,
      "headroom": {
        "enabled": true,
        "cpuPercentage": 15,
        "memoryPercentage": 15
      }
    }
  }'

Headroom pre-provisions spare capacity so pods schedule immediately instead of waiting 2-5 minutes for new nodes.

Step 2: Instance Family Optimization

# Terraform: Prefer instance families with fast launch times
resource "castai_node_template" "fast_launch" {
  cluster_id = castai_eks_cluster.this.id
  name       = "fast-launch-workers"

  constraints {
    spot                  = true
    use_spot_fallbacks    = true
    fallback_restore_rate_seconds = 300

    # Newer instance types launch faster and have better availability
    instance_families {
      include = ["m6i", "m7i", "c6i", "c7i", "r6i", "r7i"]
    }

    # Enable spot diversity for faster provisioning
    spot_diversity_price_increase_limit_percent = 25

    architectures = ["amd64"]
  }
}

Step 3: Evictor Tuning for Faster Consolidation

# Reduce empty node delay for dev/staging (faster downscale)
helm upgrade castai-evictor castai-helm/castai-evictor \
  -n castai-agent \
  --reuse-values \
  --set evictor.aggressiveMode=true \
  --set evictor.cycleInterval=120

# For production, use non-aggressive with longer intervals
# --set evictor.aggressiveMode=false
# --set evictor.cycleInterval=600

Step 4: API Performance for Multi-Cluster Dashboards

import { LRUCache } from "lru-cache";

const cache = new LRUCache<string, unknown>({ max: 100, ttl: 60_000 });

interface ClusterSummary {
  id: string;
  name: string;
  savings: number;
  savingsPercent: number;
  nodeCount: number;
  spotPercent: number;
}

async function getClusterSummary(clusterId: string): Promise<ClusterSummary> {
  const cacheKey = `s…

CAST AI Production Checklist

Overview

Complete checklist for enabling CAST AI cost optimization on a production Kubernetes cluster. Covers Phase 1 (monitoring) through Phase 2 (full automation) with validation steps at each stage.

Prerequisites

• CAST AI tested on a staging cluster first
• Production API key (Full Access)
• Change management approval for node lifecycle changes

Instructions

Complete the phases in order and preserve evidence for every checked item. Start in monitoring-only mode, verify the production cluster identity and baseline metrics, then request the approved change window before enabling automation. Use a two-person review for capacity limits, disruption budgets, and the emergency-disable procedure; do not copy staging keys, policies, or test evidence into the production record without revalidation.

Phase 1: Monitoring Only

• [ ] Agent installed with read-only key
• [ ] Agent pod healthy: kubectl get pods -n castai-agent
• [ ] Console shows cluster as "Connected"
• [ ] Savings report generating (wait 24h for full data)
• [ ] Review savings estimate before enabling automation

Phase 2: Autoscaling Enabled

• [ ] Full Access API key provisioned and stored in secrets manager
• [ ] Cluster controller installed
• [ ] Evictor installed with conservative settings (non-aggressive)
• [ ] Spot handler installed for graceful interruption handling
• [ ] Autoscaler policies configured with appropriate limits:
• [ ] clusterLimits.cpu.maxCores set to safe ceiling
• [ ] unschedulablePods.headroom configured (10-15%)
• [ ] nodeDownscaler.emptyNodes.delaySeconds >= 300 for production
• [ ] spotInstances.spotDiversityEnabled = true
• [ ] Node templates created for workload-specific needs (GPU, high-memory)
• [ ] PodDisruptionBudgets set on all critical workloads

Workload Autoscaler

• [ ] Workload autoscaler installed
• [ ] Critical deployments annotated with min/max resource bounds
• [ ] Anti-shrink cooldown set (300s minimum)
• [ ] Memory headroom >= 20% for production workloads

Security

• [ ] API key in secrets manager (not Helm values files)
• [ ] Kvisor security agent installed
• [ ] Network policies applied to castai-agent namespace
• [ ] RBAC reviewed and minimized
• [ ] Key rotation scheduled (90-day interval)

Monitoring and Alerting

• [ ] Alert on agent pod restarts: kube_pod_container_status_restarts_total{namespace="castai-agent"}
• [ ] Alert on API errors in agent logs
• [ ] CAST AI console email notifications enabled
• [ ] Savings report reviewed weekly
…

CAST AI Rate Limits

Overview

The CAST AI REST API enforces rate limits per API key. The autoscaler agent communicates cluster state at 15-second intervals. For custom API integrations, implement exponential backoff and request queuing to avoid hitting limits.

Prerequisites

• CAST AI API key configured
• Understanding of the API endpoints you call

Rate Limit Behavior

Instructions

Step 1: Detect Rate Limits from Response Headers

async function castaiRequest(path: string): Promise<Response> {
  const response = await fetch(`https://api.cast.ai${path}`, {
    headers: { "X-API-Key": process.env.CASTAI_API_KEY! },
  });

  // Log rate limit headers for monitoring
  const remaining = response.headers.get("X-RateLimit-Remaining");
  const reset = response.headers.get("X-RateLimit-Reset");
  if (remaining) {
    console.log(`Rate limit remaining: ${remaining}, resets: ${reset}`);
  }

  if (response.status === 429) {
    const retryAfter = parseInt(response.headers.get("Retry-After") ?? "5");
    throw new RateLimitError(retryAfter);
  }

  return response;
}

class RateLimitError extends Error {
  constructor(public retryAfterSeconds: number) {
    super(`Rate limited. Retry after ${retryAfterSeconds}s`);
  }
}

Step 2: Exponential Backoff with Jitter

async function withBackoff<T>(
  fn: () => Promise<T>,
  maxRetries = 5
): Promise<T> {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (attempt === maxRetries) throw err;

      let delayMs: number;
      if (err instanceof RateLimitError) {
        delayMs = err.retryAfterSeconds * 1000;
      } else {
        delayMs = Math.min(1000 * Math.pow(2, attempt), 30000);
      }
      // Add jitter to prevent thundering herd
      delayMs += Math.random() * 1000;

      console.log(`Retry ${attempt + 1}/${maxRetries} in ${delayMs}ms`);
      await new Promise((r) => setTimeout(r, delayMs));
    }
  }
  throw new Error("Unreachable");
}

Step 3: Request Queue for Batch Operations

import PQueue from "p-queue";

// Limit concurrent requests and enforce interval
const castaiQueue = new PQueue({
  concurrency: 3,
  interval: 1000,
  intervalCap: 5,   // Max 5 requests per second
});

async function queuedCastAIReque…

CAST AI Reference Architecture

Overview

Production-grade architecture for managing CAST AI across multiple Kubernetes clusters. Covers Terraform module layout, per-environment policies, API key management, and observability integration.

Prerequisites

• Multiple Kubernetes clusters (dev, staging, production)
• Terraform for infrastructure management
• Centralized secrets management
• Monitoring stack (Prometheus, Grafana, or Datadog)

Instructions

Define every environment as an independent security and state boundary, then pin the module and provider versions, declare capacity and interruption guardrails, and review the Terraform plan before applying. Roll out one non-production environment at a time, verify agent health and monitoring, and promote only the configuration that has matching approvals and evidence. Keep all secret references environment-scoped and use the documented rollback path when the observed behavior differs from the approved design.

Terraform Module Structure

infrastructure/
├── modules/
│   └── castai-cluster/
│       ├── main.tf              # CAST AI provider resources
│       ├── variables.tf         # Cluster-specific inputs
│       ├── outputs.tf           # Cluster ID, savings metrics
│       ├── policies.tf          # Autoscaler policy configuration
│       ├── node-templates.tf    # Node template definitions
│       └── security.tf          # Kvisor, RBAC
├── environments/
│   ├── dev/
│   │   ├── main.tf              # Dev cluster onboarding
│   │   ├── terraform.tfvars     # Dev-specific values
│   │   └── backend.tf           # State storage
│   ├── staging/
│   │   ├── main.tf
│   │   ├── terraform.tfvars
│   │   └── backend.tf
│   └── prod/
│       ├── main.tf
│       ├── terraform.tfvars
│       └── backend.tf
└── shared/
    ├── api-keys.tf              # Key management
    └── monitoring.tf            # Alerting rules

Reusable Module

# modules/castai-cluster/main.tf
variable "cluster_name" { type = string }
variable "cluster_id" { type = string }
variable "environment" { type = string }
variable "api_token" { type = string; sensitive = true }
variable "provider_type" { type = string }  # eks, gke, aks
variable "max_cpu_cores" { type = number; default = 100 }
variable "spot_enabled" { type = bool; default = true }
variable "hibernation_enabled" { type = bool; default = false }
variable "evictor_aggressive" { type = bool; default = false }

resource "castai_autoscaler" "this" {
  cluster_id = var.cluster_id

  autoscaler_policies_json = jsonencode({
    enabled = true
    unschedulablePods = {
      enabled = true
      headroom = {
        enabled          = true
        cpuPercentage    = var.environment == "prod" ? 15 : …

CAST AI SDK Patterns

Overview

CAST AI uses a REST API with X-API-Key header authentication. There is no official SDK -- build typed wrappers around fetch or requests. These patterns cover singleton clients, typed responses, retry with backoff, and multi-cluster management.

Prerequisites

• Completed castai-install-auth setup
• TypeScript 5+ or Python 3.10+
• Familiarity with async/await patterns

Instructions

Step 1: TypeScript API Client

// src/castai/client.ts
interface CastAIConfig {
  apiKey: string;
  baseUrl?: string;
  timeoutMs?: number;
}

interface CastAICluster {
  id: string;
  name: string;
  status: string;
  providerType: "eks" | "gke" | "aks";
  agentStatus: string;
  createdAt: string;
}

interface CastAISavings {
  monthlySavings: number;
  savingsPercentage: number;
  currentMonthlyCost: number;
  optimizedMonthlyCost: number;
}

interface CastAINode {
  name: string;
  instanceType: string;
  lifecycle: "on-demand" | "spot";
  allocatableCpu: string;
  allocatableMemory: string;
  zone: string;
}

class CastAIClient {
  private apiKey: string;
  private baseUrl: string;
  private timeoutMs: number;

  constructor(config: CastAIConfig) {
    this.apiKey = config.apiKey;
    this.baseUrl = config.baseUrl ?? "https://api.cast.ai";
    this.timeoutMs = config.timeoutMs ?? 30000;
  }

  private async request<T>(path: string, options?: RequestInit): Promise<T> {
    const controller = new AbortController();
    const timeout = setTimeout(() => controller.abort(), this.timeoutMs);

    try {
      const response = await fetch(`${this.baseUrl}${path}`, {
        ...options,
        headers: {
          "X-API-Key": this.apiKey,
          "Content-Type": "application/json",
          ...options?.headers,
        },
        signal: controller.signal,
      });

      if (!response.ok) {
        const body = await response.text();
        throw new CastAIError(response.status, body, path);
      }

      return response.json();
    } finally {
      clearTimeout(timeout);
    }
  }

  async listClusters(): Promise<CastAICluster[]> {
    const data = await this.request<{ items: CastAICluster[] }>(
      "/v1/kubernetes/external-clusters"
    );
    return data.items;
  }

  async getSavings(clusterId: string): Promise<CastAISavings> {
    return this.request(`/v1/kubernetes/clusters/${clusterId}/savings`);
  }

  async listNodes(clusterId: string): Promise<CastAINode[]> {
    const data = await this.request<{ items: CastAINode[] }>(
      `/v1/kubernetes/external-clusters/${clusterId}/nodes`
    );
    return data.items;
  }

  async updatePolicies(clusterId: string, policies: Record<string, unknown>): Promise…

CAST AI Security Basics

Overview

Secure your CAST AI integration: API key management, RBAC least-privilege, Kvisor runtime security agent, and network policy configuration.

Prerequisites

• CAST AI agent installed on cluster
• Cluster admin access for RBAC configuration
• Secrets manager (AWS Secrets Manager, Vault, etc.)

Instructions

Step 1: API Key Management

# Use separate keys per environment
# console.cast.ai > API > API Access Keys

# Development: Read-Only key (monitoring only)
# Staging: Full Access key with limited cluster scope
# Production: Full Access key, rotated every 90 days

# Store in secrets manager, never in code
aws secretsmanager create-secret \
  --name "castai/prod/api-key" \
  --secret-string "${CASTAI_API_KEY}"

# Rotate key procedure:
# 1. Generate new key in console
# 2. Update secrets manager
# 3. Restart CAST AI agent pods to pick up new key
# 4. Verify agent reconnects
# 5. Revoke old key in console

Step 2: RBAC Least-Privilege Review

# Audit CAST AI ClusterRoles
kubectl get clusterroles -l app.kubernetes.io/managed-by=castai -o yaml

# The CAST AI agent needs these minimum permissions:
# - get/list/watch: pods, nodes, events, namespaces, replicasets
# - get: persistentvolumes, storageclasses
# The cluster controller additionally needs:
# - create/delete: nodes (for autoscaling)
# - patch: pods/eviction (for evictor)

# Check for overly broad permissions
kubectl auth can-i --list --as=system:serviceaccount:castai-agent:castai-agent

Step 3: Enable Kvisor Security Agent

# Kvisor scans for CVEs, misconfigurations, and runtime threats
helm upgrade --install castai-kvisor castai-helm/castai-kvisor \
  -n castai-agent \
  --set castai.apiKey="${CASTAI_API_KEY}" \
  --set castai.clusterID="${CASTAI_CLUSTER_ID}" \
  --set controller.extraArgs.image-scan-enabled=true \
  --set controller.extraArgs.kube-bench-enabled=true

# Verify Kvisor is running
kubectl get pods -n castai-agent -l app.kubernetes.io/name=castai-kvisor

Step 4: Network Policies

# Restrict CAST AI agent egress to only api.cast.ai
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: castai-agent-egress
  namespace: castai-agent
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: castai-agent
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0  # api.cast.ai resolves dynamically
      ports:
        - protocol: TCP
          port: 443
    - to:  # Allow DNS
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53

Step 5: Security Checklist

• [ ] API keys stored in secrets manager, not Helm values files
…

CAST AI Upgrade & Migration

Overview

Upgrade CAST AI components: Helm charts for the agent/controller/evictor, Terraform provider version, and workload autoscaler. Includes rollback procedures for each component.

Prerequisites

• Current CAST AI components installed
• Staging cluster for testing upgrades first
• Helm and kubectl access
• Change management approval for production

Instructions

Step 1: Check Current Versions

# Helm chart versions
helm list -n castai-agent -o json | jq '.[] | {name: .name, chart: .chart, appVersion: .app_version}'

# Available versions
helm repo update
helm search repo castai-helm --versions | head -20

# Terraform provider version
grep "castai/castai" .terraform.lock.hcl
terraform providers

Step 2: Review Changelog

# Check CAST AI changelog for breaking changes
# https://docs.cast.ai/changelog/

# Check Terraform provider releases
# https://github.com/castai/terraform-provider-castai/releases

Step 3: Upgrade on Staging First

# Upgrade agent
helm upgrade castai-agent castai-helm/castai-agent \
  -n castai-agent --reuse-values

# Upgrade cluster controller
helm upgrade cluster-controller castai-helm/castai-cluster-controller \
  -n castai-agent --reuse-values

# Upgrade evictor
helm upgrade castai-evictor castai-helm/castai-evictor \
  -n castai-agent --reuse-values

# Upgrade workload autoscaler
helm upgrade castai-workload-autoscaler castai-helm/castai-workload-autoscaler \
  -n castai-agent --reuse-values

# Verify all pods restarted cleanly
kubectl get pods -n castai-agent -w

Step 4: Upgrade Terraform Provider

# Update version constraint in versions.tf
terraform {
  required_providers {
    castai = {
      source  = "castai/castai"
      version = "~> 7.5"  # Update to target version
    }
  }
}

# Upgrade provider
terraform init -upgrade
terraform plan -var-file=environments/staging.tfvars

# Review plan carefully for resource recreation
# Apply if plan looks safe
terraform apply -var-file=environments/staging.tfvars

Step 5: Validate After Upgrade

# Verify agent is online
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/external-clusters/${CASTAI_CLUSTER_ID}" \
  | jq '{agentStatus, name}'

# Verify autoscaler policies still applied
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
  | jq '.enabled'

# Check for errors in new agent version
kubectl logs -n castai-agent deployment/castai-agent --tail=50 | grep -i error

Rollback…

CAST AI Webhooks & Events

Overview

CAST AI emits events for node lifecycle changes, autoscaler decisions, and security findings. Configure webhook endpoints or use the audit log API to track all cluster operations. Integrates with Slack, PagerDuty, and custom HTTP endpoints.

Prerequisites

• CAST AI cluster connected and active
• HTTPS endpoint for receiving webhooks (or Slack webhook URL)
• API key with Full Access

Instructions

Step 1: Configure Notification Channels in Console

Navigate to console.cast.ai > your cluster > Notifications. Available channels:

• Slack: Webhook URL integration
• Email: Per-user notifications
• PagerDuty: Incident escalation
• Custom webhook: Any HTTPS endpoint

Step 2: Query Audit Log via API

# Get recent cluster operations
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
  "https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/audit-log?limit=20" \
  | jq '.items[] | {
    time: .createdAt,
    action: .action,
    initiator: .initiatedBy,
    details: .details
  }'

Step 3: Build a Custom Notification Handler

// castai-webhook-handler.ts
import express from "express";

const app = express();
app.use(express.json());

interface CastAIEvent {
  eventType: string;
  clusterId: string;
  clusterName: string;
  timestamp: string;
  data: {
    nodeName?: string;
    instanceType?: string;
    lifecycle?: string;
    action?: string;
    savingsImpact?: number;
  };
}

app.post("/castai/events", async (req, res) => {
  const event: CastAIEvent = req.body;

  switch (event.eventType) {
    case "node.added":
      console.log(
        `Node added: ${event.data.nodeName} (${event.data.instanceType}, ${event.data.lifecycle})`
      );
      await notifySlack(
        `New ${event.data.lifecycle} node: ${event.data.instanceType}`
      );
      break;

    case "node.removed":
      console.log(`Node removed: ${event.data.nodeName}`);
      break;

    case "node.spot_interrupted":
      console.log(`Spot interruption: ${event.data.nodeName}`);
      await notifyPagerDuty("Spot instance interrupted", event);
      break;

    case "savings.threshold":
      console.log(`Savings threshold crossed: ${event.data.savingsImpact}%`);
      break;

    default:
      console.log(`Unhandled event: ${event.eventType}`);
  }

  res.status(200).json({ received: true });
});

async function notifySlack(message: string): Promise<void> {
  await fetch(process.env.SLACK_WEBHOOK_URL!, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      t…