Streaming

How the gateway forwards and measures a streamed response.

const stream = await client.chat.completions.create({
  model: 'gpt-4o-mini',
  messages,
  stream: true,
})

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? '')
}

What Meted does

The provider's SSE stream is piped through as it arrives. The gateway does not buffer the response and does not wait for it to finish before forwarding the first byte. Events are parsed as they pass so that token counts, finish reason and time to first token can be recorded.

One event is filtered: a usage-only chunk that Meted requested on the caller's behalf. Every other event is forwarded, including data: [DONE]. See below.

The usage event

A streamed response carries no token counts unless the request sets stream_options: { include_usage: true }, which adds a final event containing usage and an empty choices array.

On a streaming request the gateway:

1. Sets stream_options.include_usage on the upstream request when the caller did not set it. This happens in every mode, including observe.
2. Reads usage from the event when it arrives.
3. Drops that event from the stream, if the caller did not ask for usage.

A chunk is dropped only when it carries usage and no choices. If you set include_usage yourself, the event is forwarded like any other.

Which values arrive when

Response headers are sent before the body, so they carry only what is known before the upstream call: the request id, the mode, the engine decision and the allocation. See Response headers.

Output tokens, finish_reason and time to first token are known only once the stream ends. HTTP response headers cannot be changed after the body has started, so these values are not available there. They reach the embedding process through the onRecord summary, which the gateway builds when the stream closes. meted dev prints its log line from that summary, which is why the log line appears after the response completes.

Time to first token

ttftMs on the summary is measured from the start of gateway handling to the first content delta in the stream.

Applying a ceiling changes max_tokens on the upstream request. Whether that affects time to first token depends on the provider and the model. Compare the two with meted eval, which reports median latency for a run with and without a ceiling.

When usage is unavailable

If a provider returns no usage event even when asked, the gateway estimates output tokens from the streamed text at roughly four characters per token and sets usageMeasured: false on the summary. meted eval labels the output line mixed when a run contains estimated counts.