Observe Mode

The default mode: compute an allocation without applying it.

In observe mode the gateway runs the engine on every request, reports the allocation it would have applied, and sends the request upstream without a ceiling.

What observe mode preserves in the request body: no max_tokens or max_completion_tokens is set, and no length hint is added to the messages. The allocation is reported with x-meted-applied: false and x-meted-skip-reason: observe_mode.

The gateway still processes the request in every other respect, in observe mode as in the others:

The inbound Authorization header is removed and replaced with the credential from the gateway's own environment.
Hop-by-hop headers (host, connection, content-length, accept-encoding) and any inbound x-meted-* headers are dropped, and content-type: application/json is set.
On a streaming request, stream_options.include_usage is added if the caller did not set it, and the resulting usage event is removed from the stream before it reaches the caller. See Streaming.
x-meted-* headers are added to the response.

Why it is the default

Meted shows you what it would do against your own workload, and you decide.

Observe Mode is also the only state in which the un-optimised output of a request is directly observable: nothing was capped, so the length the model chose is the length it wanted. Every other mode has to model that number instead.

What you get from it

$ meted dev

ID    TASK          ACTUAL  TARGET    DIFF    TOTAL  ALLOCATION
a19f  definition       184      38    -79%    902ms  observe_mode
b91c  coding           722     680     -6%     3.1s  observe_mode
7daa  fact              91      18    -80%    410ms  observe_mode

ACTUAL is the output tokens the provider generated. TARGET is the engine's estimate. DIFF is the difference between them as a percentage of actual output.

DIFF is not a measured saving. It compares an observed output against an estimate of what the answer needed, and observe mode never tested whether an answer of the target length would have been acceptable. To measure that, run meted eval, which sends each prompt with and without a ceiling and compares the two answers.

For the coding row the estimate and the actual output are close, so a ceiling would have changed little.

Every request also carries its decision back on the response, whether you are watching the log or not:

x-meted-task-type       definition
x-meted-target-tokens   38
x-meted-max-tokens      64
x-meted-confidence      0.96
x-meted-applied         false
x-meted-skip-reason     observe_mode

Deciding whether to apply ceilings

Observe mode shows the size of the gap between estimate and output. It does not show what applying a ceiling would do to the answers, because no ceiling was applied.

To get that, assemble a set of requests representative of your traffic and run meted eval. It sends each prompt twice, once without a ceiling and once with one, and reports output tokens, truncation, latency and whether each answer still passed the suite's checks.

The log is useful for two things in the meantime:

Rows where TARGET exceeds ACTUAL. A positive DIFF means the engine estimated more than the model produced. A ceiling would not reduce those requests.
low_confidence in the ALLOCATION column. Active modes skip those requests, so they are unaffected whichever mode you choose.

Changing mode

meted mode conservative   # ceiling × 1.5, floor 256
meted mode balanced       # ceiling × 1.0, floor 96
meted mode observe        # back to reporting only

A running meted dev keeps its current mode until restarted. See Policies for what each mode applies.