meted eval

Compare natural and metered generation on real prompts.

meted eval

Runs every prompt twice, once straight at your provider and once through Meted, and compares them.

Meted Eval

Prompts                     250

                         NATURAL       METED
Output tokens             91,482      60,817
Median output                287         183
Median latency              1.4s        1.1s

Estimated net reduction                32.7%
Sufficiency                            97.6%
Under-allocation                        2.4%

This calls your provider twice per prompt and costs money. The built-in suite is 76 prompts, so a full run is 152 requests.

Options

Flag
`--model <id>`	Model to evaluate. Default `gpt-4o-mini`
`--mode <mode>`	Mode to evaluate. Default `balanced`
`--limit <n>`	Only the first n prompts
`--concurrency <n>`	Prompts in flight. Default 4
`--judge`	Also grade with an LLM judge where a rubric exists
`--judge-model <id>`	Model for the judge
`--evaluator <path>`	Module exporting a custom evaluator
`--out <path>`	Write the full JSON report to a file
`--verbose`	Per-category breakdown and under-allocation detail
`--json`	Machine-readable output

Your own suite

meted eval ./evals.json

See Eval format.

Trying it cheaply

meted eval --limit 12

Twelve prompts, twenty-four requests.

Reading the result

Estimated net reduction is 1 − (metered output + optimizer overhead) ÷ natural output.

Sufficiency is the share of metered answers that did the job, judged by the suite's own assertions. A truncated answer always counts as a failure, whatever else it satisfied.

Under-allocation is the share where Meted came up short and an unconstrained run did not. Watch this one when deciding whether to adopt a mode.

Where it went wrong

meted eval --verbose

Under-allocations

code-rate-limiter (coding) target=1500 applied=1920: stopped at the token ceiling
res-llm-caching (research) target=800 applied=1024: failed: at least 300 words

Each line names the prompt, its category, what was allocated, and what the evaluator objected to.

Isolation

Eval traffic runs through a gateway built for the run and thrown away with it. It does not read or modify meted.config.json, and writes no files unless you pass --out <path>.