Together AI Dynamic Rate Limits

Overview

This skill replaces fixed quota tables with adaptive control driven by Together's current response headers and observed workload demand.

Prerequisites

• The exact model IDs and traffic profile by request and token volume
• Access to response headers from the HTTP or SDK transport
• Queue-latency, retry, spend, and failure budgets
• A decision owner for batch, provisioned, or dedicated capacity

Tool Discipline

Use Read, Glob, and Grep to locate concurrency, queue, retry, and telemetry code. Use WebFetch for the current dynamic-limit header contract. Use Write or Edit only after the target limiter and tests are identified.

Current Contract

• Serverless limits are dynamic per organization and model; historical account tiers are retired.
• Read request and token limit/remaining/reset headers on every response and maintain state per model.
• Treat displayed RPM as a per-second enforcement budget when Together documents that conversion.
• Use batch for delay-tolerant bursts and dedicated capacity for predictable reserved throughput.

Authentication

Limit headers arrive on authenticated serverless inference responses using the project-scoped Bearer key. Record numeric headers and model alias only; never retain the credential or sensitive body.

Instructions

1. Inventory callers, models, concurrency, token sizes, queue depth, and existing retries.
2. Capture the current x-ratelimit-* and x-tokenlimit-* fields from redacted responses.
3. Build independent request and token budgets keyed by model and organization context.
4. Admit work through a bounded queue; reserve headroom for retries and interactive traffic.
5. On 429, honor reset evidence, apply jitter, and stop at the request deadline.
6. Load-test below approval bounds, then choose serverless, batch, or dedicated capacity from measured demand.

Approval Boundaries

Do not manufacture fixed limits, increase retry concurrency after throttling, or move to paid reserved hardware without explicit cost and capacity approval.

Output

Return observed headers, per-model limiter settings, queue/retry budgets, load-test evidence, dropped/deferred work, and capacity recommendation.

Error Handling

Examples

The example below shows the minimum redacted evidence expected from a successful invocation of this operator workflow.

model=catalog-id; limits=header-derived; queue=bounded; retry=jittered; overflow=batch-candidate

Resources

• Skill-specific official documentation
• Serverless rate limits
• Usage limits and analytics
• Batch overview