Guides

Streaming

Server-sent events, retries, and what not to do with a partial answer.

Set "stream": true and the response arrives as server-sent events instead of one body.

curl -N https://api.cognitivers.com/v1/chat/completions \
  -H "Authorization: Bearer $COGNITIVERS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "cog-fast",
    "stream": true,
    "messages": [{ "role": "user", "content": "Write a short brief." }]
  }'
data: {"choices":[{"delta":{"role":"assistant"}}]}

data: {"choices":[{"delta":{"content":"The"}}]}

data: {"choices":[{"delta":{"content":" brief"}}]}

data: {"choices":[{"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Use curl -N when testing by hand. Without it curl buffers the output and shows you nothing until the request ends, which looks exactly like a stall.

Handling the stream

Every current OpenAI client handles this already, so prefer the client over your own parser. If you are writing one, four rules cover it:

  1. Ignore comment lines. A line starting with : is a keep-alive, not data.
  2. Split on blank lines. One event is one or more data: lines followed by an empty line.
  3. Stop at [DONE]. It is a sentinel, not JSON.
  4. Concatenate deltas. Each chunk carries a fragment, and finish_reason marks the last one.

Retries

A stream can end early for reasons that have nothing to do with the model: a dropped connection, a client-side read timeout, a proxy restart. A retry is safe because requests are stateless, but only the last complete answer counts: do not append a retry to a partial answer you have already shown or stored, because the two generations are different continuations of the same prompt and the seam shows.

The pattern that behaves well: hold the answer in memory, mark it committed when finish_reason arrives, and retry the whole request on failure rather than resuming it.

Long reads

Large inputs can take longer to produce the first token. Keep the same base URL and enable streaming. Configure a suitable client timeout and verify the complete request path with a representative input. See Long context.

Usage in a stream

Ask for usage in the final chunk rather than counting tokens yourself. Counting deltas on your side drifts from what is billed, and the difference shows up as an argument at invoice time.