Running evals

Measuring Meted against your own prompts.

meted eval

Every prompt is run twice, once straight at the provider and once through Meted, back to back, so provider load and model version are as close to identical as they can be.

Output below is illustrative, showing the shape of the report rather than a recorded run:

Meted Eval

Prompts                      76

                         NATURAL       METED
Output tokens             27,940      18,610
Median output                287         183
Median latency              1.4s        1.1s
Optimizer overhead             -           0

Estimated net reduction                33.4%
Sufficiency                            97.4%
Under-allocation                        2.6%

This costs money. The built-in suite is 76 prompts and each one is sent twice, so a full run is 152 provider requests.

The built-in suite

76 prompts across facts, definitions, explanations, how-to, coding, debugging, summarisation, translation, rewriting, classification, extraction, comparison, recommendations, research, analysis, creative work, explicit long-form requests and complex multi-part requests.

It is weighted towards the cases where Meted could do damage: long-form, multi-part asks, code with tests.

The suite is at evals/builtin.json in the repository. Copy it and edit it.

Your own suite

meted eval ./evals.json

Prompts sampled from your own product measure your own workload. See Eval format.

Starting small

meted eval --limit 12

Comparing modes

meted eval --mode conservative --limit 30 --out conservative.json
meted eval --mode balanced     --limit 30 --out balanced.json
meted eval --mode aggressive   --limit 30 --out aggressive.json

Each report gives output tokens, sufficiency, under-allocation and median latency for that mode. Compare them against the ceiling each mode applies, which is listed under Policies.

Where it went wrong

meted eval --verbose

Under-allocations

code-rate-limiter (coding) target=1500 applied=1920: stopped at the token ceiling
res-llm-caching (research) target=800 applied=1024: failed: at least 300 words

In CI

meted eval --json --limit 40 > report.json
node -e '
  const r = require("./report.json");
  if (r.underAllocation > 0.05) {
    console.error("under-allocation regressed:", r.underAllocation);
    process.exit(1);
  }
'

Run it on a schedule rather than per commit. Provider models change underneath you, and a regression there shows up the same way.

Isolation

Eval traffic runs through a gateway constructed for the run and discarded when it ends. It does not read or modify meted.config.json, and --mode applies to the run only.

The run writes no files unless you pass --out <path>, which saves the full JSON report there.