Limits
The ceilings that apply to a request, a key and an account, and how to live within them.
Per request
| Limit | Value |
|---|---|
| Context window | 262,144 tokens for cog-fast and cog-pro |
| Maximum output | 32,768 tokens |
cog-pro input | 64K to 256K depending on plan, see Models and prices |
| Embedding context | 32,768 input tokens |
A request over a ceiling is rejected with 413 rather than truncated. That is deliberate: a silently
truncated prompt produces an answer that looks fine and is wrong.
Per key
| Limit | Where it comes from |
|---|---|
| Models | The allowlist you set on the key |
| Spend | The monthly budget you set on the key |
| Concurrency | The ceiling that comes with your plan |
These are the three controls worth setting on any key that runs unattended. A budget alone does not stop a retry storm from being slow, and concurrency alone does not stop a long job from being expensive.
Rate limits
Concurrency is the limit that matters for how the API behaves under load: it is what keeps one
account's burst from becoming another account's latency. When you exceed it, the API answers 429
with a retry hint rather than queueing indefinitely.
Handle 429 the way you would handle any retryable error: back off with jitter, and cap the number
of attempts. A client that retries in a tight loop turns a momentary limit into a sustained one.
Abandoned streams
If a client disconnects mid-stream, generation stops and you are not billed for tokens that were never produced. Holding a stream open on a client that is no longer reading ties up concurrency without producing anything, which is why the client-side timeout should be shorter than your patience.
Asking for more
If a ceiling is the thing standing between you and a workload, say so. Concurrency is the dimension we can usually move, and reserved capacity exists precisely for the case where shared limits are not acceptable.