Groq Reference Architecture

Overview

Production architecture for applications built on Groq's LPU inference API. It

covers four concerns that every serious Groq integration needs: routing requests

to the right model by latency/capability/cost, a middleware band (cache, metrics,

retry), a multi-provider fallback chain, and a streaming pipeline. The service

layer built here is reusable across a chat UI, an API backend, a batch processor,

or an agent.

The full layer diagram and how the pieces interact lives in

references/architecture.md; the complete,

copy-ready TypeScript for every layer is in

references/implementation.md.

Prerequisites

• Groq API key — create one at console.groq.com

and export it as GROQAPIKEY. The Groq SDK reads it from the environment;

the client is constructed as new Groq({ apiKey: process.env.GROQAPIKEY }).

Never hardcode the key.

• Runtime: Node.js 18+ (for performance.now() and native fetch).
• Packages: groq-sdk and lru-cache (npm install groq-sdk lru-cache).
• Optional backup provider: an OpenAI-compatible key if you extend the

fallback chain beyond Groq's own models.

Instructions

Build the service layer in five ordered steps. Each step is one file under

src/groq/. The router depends on the registry; the middleware and fallback

depend on the client; the streaming pipeline stands alone. Full source for every

step (verbatim) is in references/implementation.md.

1. Model Registry (models.ts) — declare a ModelSpec for each model with

its tier, context window, speed, cost, and capabilities. Skeleton:

   export const MODELS: Record<string, ModelSpec> = {
     "llama-3.1-8b-instant":     { tier: "speed",   /* fast, cheap */ },
     "llama-3.3-70b-versatile":  { tier: "quality", /* tools + JSON */ },
     "meta-llama/llama-4-scout-17b-16e-instruct": { tier: "vision" },
     "whisper-large-v3-turbo":   { tier: "audio" },
   };

1. Model Router (router.ts) — selectModel(req) maps requirements

(maxLatencyMs, needsVision, needsTools, costSensitive) to the cheapest

model that satisfies them. Callers pass requirements, never hardcoded ids.

1. Middleware (middleware.ts) — completionWithMiddleware() wraps each call

with an LRU cache (deterministic requests only, temperature === 0), latency +

token metrics, and a pluggable metrics sink.

1. Fallback Chain (fallback.ts) — completionWithFallback() tries the

primary model, drops to a model in a different rate-limit pool on 429/5xx, then

returns a graceful-degradation payload instead of throwing.

1. Streaming Pipeline (streaming.ts) — streamCompletion() is an async

generator yielding { type: "token" | "done" | "error" } for real-time SSE UIs.

When applying this to an existing repo, Read the current src/ layout and

Grep for direct groq.chat.completions.create calls to find code that should

route through the middleware and fallback wrappers instead.

Integration Patterns

Output

Applying this skill produces a src/groq/ service layer with six files

(client.ts, models.ts, router.ts, middleware.ts, fallback.ts,

streaming.ts) plus the service and API layers that consume it. At runtime you get:

• Routed completions — selectModel() returns a ModelSpec; callers never

hardcode a model id, so cost/latency policy lives in one place.

• Cached deterministic responses — repeated temperature: 0 calls return from

the LRU cache instead of re-billing the API.

• Resilient calls — completionWithFallback() returns a valid completion shape

even when Groq is rate-limited, never surfacing a raw 429 to the user.

• Streamed tokens — streamCompletion() yields { type, content } events for

SSE, with a terminal done or error event.

• Metrics — every call emits { model, latencyMs, tokens, cached } to your

metrics sink (Prometheus, Datadog, or console.log by default).

Error Handling

Examples

A latency-critical chat turn routes to the speed tier and returns one completion:

const model = selectModel({ maxLatencyMs: 80, costSensitive: true });
// → llama-3.1-8b-instant
const res = await completionWithMiddleware(groq, model.id, messages);

Streaming a UI consumes the async generator token-by-token:

for await (const event of streamCompletion(groq, messages)) {
  if (event.type === "token") process.stdout.write(event.content!);
}

Four fully worked examples — latency-critical, quality-with-fallback, streaming,

and vision routing — are in references/examples.md.

Resources

• Groq API Documentation
• Groq Models
• Groq Rate Limits
• Groq Pricing

Next Steps

For multi-environment deployment, see the groq-multi-env-setup skill, which

extends this service layer with per-environment configuration and secrets handling.