Groq Multi-Environment Setup

Overview

Configure Groq API access across development, staging, and production with the right model, rate limit strategy, and secret management per environment. Key insight: use llama-3.1-8b-instant in development (cheapest, fastest), match production model in staging, and harden production with retries and fallbacks.

Prerequisites

• A Groq account with API keys from console.groq.com/keys — ideally a separate key (or organization) per environment.
• Node project with the groq-sdk package installed (npm install groq-sdk).
• NODE_ENV set per environment (development / staging / production).
• A secret store for staging/production keys: GitHub Actions secrets, AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault.

Environment Strategy

Instructions

The full, copy-paste implementation lives in the reference files — this section

is the map. Read the implementation walkthrough

for the code module, service wrapper, and verify script, and

secrets & deployment for per-platform

key management, Docker Compose profiles, and rate-limit inspection.

1. Build the config module (config/groq.ts). One configs record keyed by

environment resolves model, token budget, retries, timeout, and logging, then

validates that a key is present with an environment-specific error message.

The essential skeleton:

   const configs: Record<string, GroqEnvConfig> = {
     development: { model: "llama-3.1-8b-instant", maxRetries: 1, logRequests: true, /* ... */ },
     staging:     { model: "llama-3.3-70b-versatile", maxRetries: 3, logRequests: false, /* ... */ },
     production:  { model: "llama-3.3-70b-versatile", maxRetries: 5, logRequests: false, /* ... */ },
   };
   export function getGroqConfig(): GroqEnvConfig {
     return configs[process.env.NODE_ENV || "development"] || configs.development;
   }

See implementation.md § Step 1 for t

Groq Observability

Overview

Monitor Groq LPU inference for latency, token throughput, rate limit utilization, and cost. Groq's defining advantage is speed (280-560 tok/s), so latency degradation is the highest-priority signal. The API returns rich timing metadata (queuetime, prompttime, completion_time) and rate limit headers on every response.

Prerequisites

• A Groq account with an API key exported as the GROQAPIKEY environment variable — the groq-sdk client reads it automatically (new Groq()).
• Node.js with groq-sdk and prom-client installed (npm install groq-sdk prom-client).
• A Prometheus scrape target and (optionally) Grafana for the dashboard panels.

Key Metrics to Track

Instructions

Apply these six steps in order. Steps 1-2 are the core instrumentation loop —

wrap the client, then feed a Prometheus instrument set from each call. Steps 3-6

add rate-limit tracking, alerting, structured logs, and dashboards on top. The

lean client skeleton is below; the full code for every step lives in

references/implementation.md.

1. Instrumented client — wrap groq.chat.completions.create so latency, tokens, queue time, and estimated cost are captured on the same path as the request (trackedCompletion).
2. Prometheus metrics — register a histogram (latency), counters (tokens, cost, errors), and gauges (throughput, rate-limit remaining), then feed them from emitMetrics.
3. Rate limit header tracking — parse x-ratelimit-remaining-* off every response into a gauge so you alert before a 429, not after.
4. Prometheus alert rules — ship latency/rate-limit/throughput/error/cost alerts tuned to Groq's sub-200ms,

Groq Performance Tuning

Overview

Maximize Groq's LPU inference speed advantage. Groq already delivers extreme throughput (280-560 tok/s) and low latency (<200ms TTFT), but client-side optimization -- model selection, prompt size, streaming, caching, and parallelism -- determines whether your application fully exploits that speed.

This skill walks through six tuning levers at a high level; the complete, copy-pasteable code for each lives in references/implementation.md, and end-to-end worked scenarios live in references/examples.md.

Prerequisites

• Groq API key — set GROQAPIKEY in the environment. The groq-sdk client (new Groq()) reads it automatically; never hardcode the key.
• Node.js 18+ with the groq-sdk package installed (npm install groq-sdk).
• Optional packages for the caching and parallelism steps: lru-cache and p-queue (npm install lru-cache p-queue).
• A baseline latency measurement of your current integration so you can confirm the tuning actually helps.

Groq Speed Benchmarks

TTFT = Time to First Token. Actual values depend on prompt size and server load.

Instructions

Apply these six levers in order. Each is a small, independent change — start with the ones that match your bottleneck (model choice and caching give the biggest wins on most workloads). The full code for every step is in references/implementation.md.

1. Choose the right model for speed. Map each call site to a speed tier: llama-3.1-8b-instant for latency-critical paths, llama-3.3-70b-versatile for quality-sensitive paths, llama-3.3-70b-specdec for 70b quality at higher throughput. Set temperature: 0 so responses are deterministic (and cacheable).
2. Minimize token count. Trim verbose system prompts to their essence and set max_tokens to the expected output size, not a safe-looking ceiling. Fewer tokens means faster responses and less TPM-quota pressure.
3. Stream for perceived performance. For any output the user w

Groq Production Checklist

Overview

Complete pre-launch checklist for deploying Groq-powered applications to production. Covers API key security, model selection, rate limit planning, fallback strategies, and monitoring setup. Work top-to-bottom: each section is a gate that must be green before the go-live verification runs.

Deep code (fallback function, health-check endpoint, go-live script) lives in references/ so this file stays scannable — drill in when you reach that step.

Prerequisites

• Staging environment tested with Groq API
• Groq Developer or Enterprise plan (free tier is not suitable for production)
• Production API key created in console.groq.com
• Monitoring and alerting infrastructure ready

Instructions

Read the target app's Groq integration and config, then walk each gate below. Tick every box; an unchecked item is a launch blocker.

1. API Key & Auth

• [ ] Production API key stored in secret manager (not .env files)
• [ ] Key is NOT shared with development or staging environments
• [ ] Key rotation procedure documented and tested
• [ ] Pre-commit hook blocks gsk_ pattern in code

2. Model Selection

• [ ] Production model chosen and tested (recommend llama-3.3-70b-versatile)
• [ ] Fallback model configured (llama-3.1-8b-instant)
• [ ] Deprecated model IDs removed (check deprecations)
• [ ] max_tokens set to actual expected output size (not context max)

3. Rate Limit Planning

• [ ] Production rate limits known (check console.groq.com/settings/limits)
• [ ] Estimated peak RPM < 80% of limit
• [ ] Estimated peak TPM < 80% of limit
• [ ] Exponential backoff with retry-after header implemented
• [ ] Request queue for burst protection (p-queue or similar)

4. Error Handling & Fallback

• [ ] All Groq error types caught (Groq.APIError, Groq.APIConnectionError)
• [ ] 429 errors retried with backoff
• [ ] 5xx errors retried with backoff
• [ ] 401 errors trigger alert (key may be revoked)
• [ ] Network timeouts configured (default 60s may be too long)
• [ ] Circuit breaker pattern for sustained failures
• [ ] Fallback-to-degradation wrapper in place — see the completionWithFallback pattern in references/implementation.md

5. Health Check

• [ ] /api/health (or /healthz) probes Groq with a 1-token request and returns 503 when degraded — full route in references/implementation.md

6. Monitor

Groq Rate Limits

Overview

Handle Groq rate limits using the retry-after header, exponential backoff, and request queuing. Groq enforces limits at the organization level with both RPM (requests/minute) and TPM (tokens/minute) constraints -- hitting either one triggers a 429.

The workflow builds up in five composable layers: parse the rate-limit headers, wrap calls in retry-with-backoff, gate concurrency through a queue, monitor remaining capacity proactively, and fall back across models when one pool is exhausted. Read SKILL.md for the high-level flow, then drill into the full implementation for every code block and the reference tables + worked examples for header definitions and composed clients.

Prerequisites

• A Groq API key (GROQAPIKEY) — get one at console.groq.com.
• groq-sdk installed: npm install groq-sdk.
• For queuing (Step 3): p-queue installed: npm install p-queue.
• Node.js 18+ (for native fetch and the SDK).
• Know your plan's limits — check console.groq.com/settings/limits.

Rate Limits at a Glance

Groq applies RPM, RPD, TPM, and TPD limits simultaneously — you must stay under every one, and either RPM or TPM can trip a 429. Every response (even a success) carries x-ratelimit-* headers describing remaining capacity and reset timing; 429 responses add a retry-after header. Full header and constraint tables: reference.md.

Instructions

Compose these five steps into one client wrapper (queue → monitor → retry). Each step's complete, copy-pasteable code is in implementation.md.

Step 1: Parse Rate Limit Headers

Read the x-ratelimit-* headers off every response into a typed RateLimitInfo so downstream logic can reason about remaining capacity. Groq reports reset times as strings like "1.2s" or "120ms" — normalize them to milliseconds.

Step 2: Exponential Backoff with Retry-After

Wrap each API call in a retry loop. Prefer Groq's retry-after header when present; otherwise back off exponentially with jitter, capped at maxDelayMs. Retry only 429 and 5xx — other 4xx errors are not retryable.

Step 3: Request Queue with Concurrency Control

Gate all requests through a p-queue sized to your plan's RPM (intervalCap over a 60s interval) so bursts never exceed the limit in the first place.