Guides

Long context

Sending hundreds of thousands of tokens without the connection dying mid-read.

Both chat models take 262,144 tokens of context. A prompt that size is a different kind of request from a chat message: the model reads for a long time before it writes anything, and the things that break during that read are not the model's fault.

Enable streaming on the same API

curl -N https://api.cognitivers.com/v1/chat/completions \
  -H "Authorization: Bearer $COGNITIVERS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "cog-fast",
    "stream": true,
    "messages": [{ "role": "user", "content": "<200K tokens of logs>" }]
  }'

Use the same base URL and API key as any other request. Streaming is selected by stream: true in the request body, not by a separate hostname.

Keep-alives are not noise

While the model reads, nothing is generated yet, so a connection can look idle to everything between you and us: a load balancer, a proxy, a corporate network appliance, a client library with a read timeout. SSE clients should ignore comment lines when present. Do not assume a fixed time to first token or rely on a guaranteed heartbeat interval.

Two practical consequences:

  • If you wrote your own SSE parser, skip lines that begin with a colon. Treating one as data is the most common self-inflicted failure here.
  • If you are behind a proxy of your own, do not buffer the response. A proxy that waits for a complete body before relaying it will time out on a long read.

Choose the model by what it reads

cog-fast is the long-context model by design. For a very large input it is the one that reads cheapest and fastest, and it serves the whole 262,144-token window on every plan.

cog-pro reads less: 64K on pay-as-you-go, 128K on Builder, the full window on Pro and above. Its input ceiling is a commercial term rather than a hardware limit, and it is in Models and prices.

What to do when a request is too big

  1. Enable streaming with stream: true on the same base URL. Check the input limit for your model and plan.
  2. Reuse the prefix. If you resend the same long system prompt or document on every turn, keep it byte-identical at the start of the message list. A cached prefix reads at a tenth of the input rate.
  3. Split by meaning, not by count. Two requests over two coherent halves of a document usually beat one request over the whole thing, and they fail independently.
  4. Summarise once, then reuse. If a corpus will be queried repeatedly, embed it with cog-embed and retrieve passages per request instead of resending the corpus.

Verifying that a long request works

Before building a pipeline on the assumption, send the real shape once, from the environment that will run it, through whatever proxy sits in between. A 200K-token request that works from a laptop and fails from a container behind an egress proxy is a proxy configuration problem, and it is much cheaper to find that out now than during a batch run.