Groq Cost Tuning

Overview

Optimize Groq inference costs through smart model routing, token minimization, and caching. Groq pricing is already extremely competitive, but at high volume the savings from routing classification to 8B vs 70B are 12x per request.

Groq Pricing (per million tokens)

Check current pricing at groq.com/pricing.

Instructions

Step 1: Smart Model Routing

import Groq from "groq-sdk";

const groq = new Groq();

// Route to cheapest model that meets quality requirements
interface ModelConfig {
  model: string;
  inputCostPer1M: number;
  outputCostPer1M: number;
}

const ROUTING: Record<string, ModelConfig> = {
  classification: { model: "llama-3.1-8b-instant", inputCostPer1M: 0.05, outputCostPer1M: 0.08 },
  extraction:     { model: "llama-3.1-8b-instant", inputCostPer1M: 0.05, outputCostPer1M: 0.08 },
  summarization:  { model: "llama-3.1-8b-instant", inputCostPer1M: 0.05, outputCostPer1M: 0.08 },
  reasoning:      { model: "llama-3.3-70b-versatile", inputCostPer1M: 0.59, outputCostPer1M: 0.79 },
  codeReview:     { model: "llama-3.3-70b-versatile", inputCostPer1M: 0.59, outputCostPer1M: 0.79 },
  chat:           { model: "llama-3.3-70b-versatile", inputCostPer1M: 0.59, outputCostPer1M: 0.79 },
  vision:         { model: "meta-llama/llama-4-scout-17b-16e-instruct", inputCostPer1M: 0.11, outputCostPer1M: 0.34 },
};

function getModel(useCase: string): string {
  return ROUTING[useCase]?.model || "llama-3.1-8b-instant";
}

// Classification on 8B: $0.05/M  vs  70B: $0.59/M  =  12x savings

Step 2: Minimize Tokens Per Request

// COST SAVINGS: Reduce system prompt tokens
// Groq charges for BOTH input and output tokens

// Verbose system prompt: ~200 tokens ($0.012 per 1000 calls on 70B)
const expensive = "You are a highly skilled AI assistant specializing in text classification. When given a piece of text, carefully analyze the sentiment, considering tone, word choice, connotation...";

// Concise system prompt: ~15 tokens ($0.001 per 1000 calls on 70B)
const cheap = "Classify sentiment: positive/negative/neutral. One word.";

// COST SAVINGS: Limit output tokens
async function cheapC

Groq Data Handling

Overview

Manage data flowing through Groq's inference API. Covers prompt sanitization before sending to Groq, response filtering after receiving, PII redaction, conversation audit logging, and token usage tracking. Key fact: Groq does not use API data for model training (Groq Privacy Policy).

Groq Data Policy

• Groq does not train on API request/response data
• Prompts and completions are processed and discarded
• Groq may temporarily log requests for abuse prevention
• For enterprise: contact Groq for DPA and SOC 2 compliance details

Instructions

Step 1: Prompt Sanitization Layer

import Groq from "groq-sdk";

const groq = new Groq();

interface RedactionRule {
  name: string;
  pattern: RegExp;
  replacement: string;
}

const PII_RULES: RedactionRule[] = [
  { name: "email", pattern: /\b[\w.+-]+@[\w-]+\.[\w.]+\b/g, replacement: "[EMAIL]" },
  { name: "phone", pattern: /\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/g, replacement: "[PHONE]" },
  { name: "ssn", pattern: /\b\d{3}-\d{2}-\d{4}\b/g, replacement: "[SSN]" },
  { name: "credit_card", pattern: /\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b/g, replacement: "[CARD]" },
  { name: "ip_address", pattern: /\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/g, replacement: "[IP]" },
];

function sanitizeText(text: string): { sanitized: string; redactedTypes: string[] } {
  let sanitized = text;
  const redactedTypes: string[] = [];

  for (const rule of PII_RULES) {
    if (rule.pattern.test(sanitized)) {
      redactedTypes.push(rule.name);
      sanitized = sanitized.replace(rule.pattern, rule.replacement);
    }
  }

  return { sanitized, redactedTypes };
}

function sanitizeMessages(messages: any[]): { messages: any[]; hadPII: boolean } {
  let hadPII = false;
  const sanitized = messages.map((m) => {
    if (typeof m.content !== "string") return m;
    const { sanitized: text, redactedTypes } = sanitizeText(m.content);
    if (redactedTypes.length > 0) hadPII = true;
    return { ...m, content: text };
  });

  return { messages: sanitized, hadPII };
}

Step 2: Safe Completion Wrapper

async function safeCompletion(
  messages: any[],
  model = "llama-3.3-70b-versatile",
  options?: { maxTokens?: number }
) {
  // Sanitize input
  const { messages: sanitized, hadPII } = sanitizeMessages(messages);
  if (hadPII) {
    console.warn("[groq-data] PII detected and redacted before sending to Groq API");
  }

  // Call Groq
  const completion = await groq.chat.completions.create({
    model,
    messages: sanitized,
    max_tokens: options?.maxTokens ?? 1024,
  });

  // Filter response
  const responseContent = completion.choi

Groq Debug Bundle

Current State

!node --version 2>/dev/null || echo 'N/A'

!python3 --version 2>/dev/null || echo 'N/A'

!npm list groq-sdk 2>/dev/null | grep groq-sdk || echo 'groq-sdk not installed'

Overview

Collect all diagnostic information needed to resolve Groq API issues. Produces a redacted support bundle with environment info, SDK version, connectivity test results, and rate limit status.

Prerequisites

• GROQAPIKEY set in environment
• curl and jq available
• Access to application logs

Instructions

Step 1: Create Debug Bundle Script

#!/bin/bash
set -euo pipefail

BUNDLE_DIR="groq-debug-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BUNDLE_DIR"
echo "Collecting Groq debug bundle..."

# === Environment ===
cat > "$BUNDLE_DIR/environment.txt" <<ENVEOF
=== Groq Debug Bundle ===
Generated: $(date -u +"%Y-%m-%dT%H:%M:%SZ")
Hostname: $(hostname)
OS: $(uname -sr)
Node.js: $(node --version 2>/dev/null || echo 'not installed')
Python: $(python3 --version 2>/dev/null || echo 'not installed')
npm groq-sdk: $(npm list groq-sdk 2>/dev/null | grep groq-sdk || echo 'not installed')
pip groq: $(pip show groq 2>/dev/null | grep Version || echo 'not installed')
GROQ_API_KEY: ${GROQ_API_KEY:+SET (${#GROQ_API_KEY} chars, prefix: ${GROQ_API_KEY:0:4}...)}${GROQ_API_KEY:-NOT SET}
ENVEOF

Step 2: API Connectivity Test

# Test API endpoint and capture headers
echo "--- API Connectivity ---" >> "$BUNDLE_DIR/connectivity.txt"

# Models endpoint (lightweight, confirms auth)
curl -s -w "\nHTTP Status: %{http_code}\nTime: %{time_total}s\n" \
  https://api.groq.com/openai/v1/models \
  -H "Authorization: Bearer $GROQ_API_KEY" \
  | jq '.data | length' >> "$BUNDLE_DIR/connectivity.txt" 2>&1

echo "Models available: $(curl -s https://api.groq.com/openai/v1/models \
  -H "Authorization: Bearer $GROQ_API_KEY" | jq -r '.data[].id' | wc -l)" \
  >> "$BUNDLE_DIR/connectivity.txt"

Step 3: Rate Limit Status

# Make a minimal request and capture rate limit headers
echo "--- Rate Limit Status ---" >> "$BUNDLE_DIR/rate-limits.txt"

curl -si https://api.groq.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $GROQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-3.1-8b-instant","messages":[{"role":"user","content":"ping"}],"max_tokens":1}' \
  2>/dev/null | grep -iE "^(x-ratelim

Groq Deploy Integration

Overview

Deploy applications using Groq's inference API to Vercel Edge, Cloud Run, Docker, and other platforms. Groq's sub-200ms latency makes it ideal for edge deployments and real-time applications.

Prerequisites

• Groq API key stored in GROQAPIKEY
• Application using groq-sdk package
• Platform CLI installed (vercel, docker, or gcloud)

Instructions

Step 1: Vercel Edge Function

// app/api/chat/route.ts (Next.js App Router)
import Groq from "groq-sdk";

export const runtime = "edge";

export async function POST(req: Request) {
  const groq = new Groq({ apiKey: process.env.GROQ_API_KEY! });
  const { messages, stream: useStream } = await req.json();

  if (useStream) {
    const stream = await groq.chat.completions.create({
      model: "llama-3.3-70b-versatile",
      messages,
      stream: true,
      max_tokens: 2048,
    });

    const encoder = new TextEncoder();
    const readable = new ReadableStream({
      async start(controller) {
        for await (const chunk of stream) {
          const content = chunk.choices[0]?.delta?.content;
          if (content) {
            controller.enqueue(
              encoder.encode(`data: ${JSON.stringify({ content })}\n\n`)
            );
          }
        }
        controller.enqueue(encoder.encode("data: [DONE]\n\n"));
        controller.close();
      },
    });

    return new Response(readable, {
      headers: {
        "Content-Type": "text/event-stream",
        "Cache-Control": "no-cache",
        Connection: "keep-alive",
      },
    });
  }

  const completion = await groq.chat.completions.create({
    model: "llama-3.3-70b-versatile",
    messages,
    max_tokens: 2048,
  });

  return Response.json(completion);
}

Step 2: Vercel Deployment

set -euo pipefail
# Set secret
vercel env add GROQ_API_KEY production

# Deploy
vercel --prod

Step 3: Docker Container

FROM node:20-slim AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-slim
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json .
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=5s CMD curl -sf http://localhost:3000/health || exit 1
CMD ["node", "dist/index.js"]

Step 4: Cloud Run Deployment

set -euo pipefail
# Store API key in Secret Manager
echo -n "$GROQ_API_KEY" | gcloud secrets create groq-api-key --data-file=-

# Deploy with streaming support
gcloud run deploy groq-api \
  --source . \
  --region us-central1 \
  --set-secrets=GROQ_API

Groq Enterprise Access Management

Overview

Manage team access to Groq's inference API through API key strategy, model-level routing controls, spending limits, and usage monitoring. Groq uses flat API keys (gsk_ prefix) with no built-in scoping -- access control is implemented at the application layer.

Groq Access Model

• API keys are per-organization, not per-user
• No built-in scopes -- every key has full API access
• Rate limits are per-organization, shared across all keys
• Spending limits are configurable in the Groq Console
• Projects allow creating isolated API keys with separate limits

Instructions

Step 1: API Key Strategy

// Create separate keys per team/service via Groq Console Projects
// Each project gets its own API key and can have independent rate limits

// Key naming convention: {team}-{environment}-{purpose}
const KEY_REGISTRY = {
  // Each team gets a separate Groq Project
  "chatbot-prod":         "gsk_...",  // Project: chatbot-production
  "chatbot-staging":      "gsk_...",  // Project: chatbot-staging
  "analytics-prod":       "gsk_...",  // Project: analytics-production
  "batch-processor":      "gsk_...",  // Project: batch-processing
} as const;

Step 2: Application-Level Model Access Control

// Since Groq keys don't have model scoping, implement it in your gateway
interface TeamConfig {
  allowedModels: string[];
  maxTokensPerRequest: number;
  monthlyBudgetUsd: number;
  rateLimitRPM: number;
}

const TEAM_CONFIGS: Record<string, TeamConfig> = {
  chatbot: {
    allowedModels: ["llama-3.3-70b-versatile", "llama-3.1-8b-instant"],
    maxTokensPerRequest: 2048,
    monthlyBudgetUsd: 200,
    rateLimitRPM: 60,
  },
  analytics: {
    allowedModels: ["llama-3.1-8b-instant"],  // Only cheapest model
    maxTokensPerRequest: 512,
    monthlyBudgetUsd: 50,
    rateLimitRPM: 30,
  },
  research: {
    allowedModels: [
      "llama-3.3-70b-versatile",
      "llama-3.1-8b-instant",
      "meta-llama/llama-4-scout-17b-16e-instruct",
    ],
    maxTokensPerRequest: 4096,
    monthlyBudgetUsd: 500,
    rateLimitRPM: 120,
  },
};

function validateRequest(team: string, model: string, maxTokens: number): void {
  const config = TEAM_CONFIGS[team];
  if (!config) throw new Error(`Unknown team: ${team}`);
  if (!config.allowedModels.includes(model)) {
    throw new Error(`Team ${team} not authorized for model ${model}`);
  }
  if (maxTokens > config.maxTokensPerRequest) {
    throw new Error(`max_tokens ${maxTokens} exceeds limit ${config.maxTokensPerRequest} for team ${team}`);
  }
}

Groq Incident Runbook

Overview

Rapid incident response procedures for Groq API failures. Groq is a third-party inference provider -- when it goes down, your mitigation options are: wait, fall back to a different model, or fall back to a different provider.

Severity Levels

Quick Triage (Run First)

set -euo pipefail
echo "=== 1. Groq API Status ==="
curl -sf https://status.groq.com > /dev/null && echo "status.groq.com: REACHABLE" || echo "status.groq.com: UNREACHABLE"

echo ""
echo "=== 2. API Authentication ==="
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
  https://api.groq.com/openai/v1/models \
  -H "Authorization: Bearer $GROQ_API_KEY")
echo "GET /models: HTTP $HTTP_CODE"

echo ""
echo "=== 3. Model Availability ==="
for model in "llama-3.1-8b-instant" "llama-3.3-70b-versatile"; do
  CODE=$(curl -s -o /dev/null -w "%{http_code}" \
    https://api.groq.com/openai/v1/chat/completions \
    -H "Authorization: Bearer $GROQ_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}],\"max_tokens\":1}")
  echo "$model: HTTP $CODE"
done

echo ""
echo "=== 4. Rate Limit Status ==="
curl -si https://api.groq.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $GROQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-3.1-8b-instant","messages":[{"role":"user","content":"ping"}],"max_tokens":1}' \
  2>/dev/null | grep -iE "^(x-ratelimit|retry-after)" || echo "No rate limit headers"

Decision Tree

Is the Groq API responding?
├─ NO (timeout/connection refused):
│   ├─ Check status.groq.com
│   │   ├─ Incident reported → Wait, enable fallback provider
│   │   └─ No incident → Network issue on our side (check DNS, firewall, proxy)
│   └─ Check if api.groq.com resolves: dig api.groq.com
│
├─ YES, but 401/403:
│   ├─ API key revoked or expire

Groq Install & Auth

Overview

Install the official Groq SDK and configure API key authentication. Groq provides ultra-fast LLM inference on custom LPU hardware through an OpenAI-compatible REST API at api.groq.com/openai/v1/.

Prerequisites

• Node.js 18+ or Python 3.8+
• Package manager (npm, pnpm, or pip)
• Groq account at console.groq.com
• API key from GroqCloud console (Settings > API Keys)

Instructions

Step 1: Install the SDK

set -euo pipefail
# TypeScript / JavaScript
npm install groq-sdk

# Python
pip install groq

Step 2: Get Your API Key

1. Go to console.groq.com/keys
2. Click "Create API Key"
3. Copy the key (starts with gsk_)
4. Store it securely -- you cannot view it again

Step 3: Configure Environment

# Set environment variable (recommended)
export GROQ_API_KEY="gsk_your_key_here"

# Or create .env file (add .env to .gitignore first)
echo 'GROQ_API_KEY=gsk_your_key_here' >> .env

Step 4: Verify Connection (TypeScript)

import Groq from "groq-sdk";

const groq = new Groq({
  apiKey: process.env.GROQ_API_KEY,
});

async function verify() {
  const models = await groq.models.list();
  console.log("Connected! Available models:");
  for (const model of models.data) {
    console.log(`  ${model.id} (owned by ${model.owned_by})`);
  }
}

verify().catch(console.error);

Step 5: Verify Connection (Python)

import os
from groq import Groq

client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

models = client.models.list()
print("Connected! Available models:")
for model in models.data:
    print(f"  {model.id} (owned by {model.owned_by})")

SDK Defaults

The Groq SDK auto-reads GROQAPIKEY from environment if no apiKey is passed to the constructor. Additional constructor options:

const groq = new Groq({
  apiKey: process.env.GROQ_API_KEY,  // Optional if env var set
  baseURL: "https://api.groq.com/openai/v1",  // Default
  maxRetries: 2,      // Default retry count
  timeout: 60_000,    // 60 second timeout (ms)
});

API Key Formats

Groq uses a single key type. There are no separate read/write scopes -- all keys have full API access. Restrict access through organizational controls in the console.

Error Handling