Sufficiency scoring

How an answer is judged, and how under-allocation is counted.

An answer is sufficient when it does the job the request asked for.

The evaluator chain

Evaluators compose. An answer counts as sufficient when no evaluator in the chain returns sufficient: false.

interface Evaluator {
  readonly name: string
  evaluate(context: EvaluationContext): Promise<EvaluationResult>
}

interface EvaluationResult {
  sufficient: boolean
  score: number // 0..1
  reason: string
  evaluator: string
}

Deterministic checks

The default, and always present. An answer fails immediately if it is empty or was cut off at the ceiling; otherwise it is scored against the suite's assertions, and score is the share that passed.

Truncation is an automatic failure regardless of the other checks. An answer that contains the right keyword and then stops mid-sentence still counts as a failure.

LLM judge

Optional, opt-in with --judge, and only for prompts that carry a judge rubric.

{
  "id": "explain-index-writes",
  "prompt": "Why can adding an index make writes slower?",
  "judge": "Does it explain that indexes must be maintained on every write?",
}

A judge that cannot be reached returns sufficient: true with a reason naming the failure, so an outage on the judge does not report a regression that did not happen.

This means an unreachable judge does not veto an answer, and the prompt is scored on its deterministic checks alone. Those prompts still count towards Sufficiency, so a run where the judge failed repeatedly reports a higher rate than a run where it answered. Check the reason field for judge unavailable before comparing two runs.

Your own

// evaluators/checkout.mjs
export function createEvaluator() {
  return {
    name: 'checkout-policy',
    async evaluate({ answer, prompt }) {
      const mentionsPolicy = /refund|return|30 days/i.test(answer)
      return {
        sufficient: mentionsPolicy,
        score: mentionsPolicy ? 1 : 0,
        reason: mentionsPolicy ? 'policy referenced' : 'no policy reference',
        evaluator: 'checkout-policy',
      }
    },
  }
}

meted eval ./evals.json --evaluator ./evaluators/checkout.mjs

Export createEvaluator, a default export of either shape, or an object with an evaluate method.

The two headline rates

Sufficiency is the share of metered answers judged sufficient.

Under-allocation is the share of prompts matching either of two conditions:

underAllocated =
  metedFinishReason === "length"
  || (!metedSufficient && naturalSufficient)

The first condition counts every metered answer that stopped at the ceiling, whatever the unconstrained run did.

The second condition counts a metered answer that failed the checks only when the unconstrained answer passed them. Where both failed, the prompt or the model is at fault rather than the ceiling.

The two rates do not have to sum to 100%. An answer can be insufficient for a reason that has nothing to do with allocation.

In production

meted eval compares two runs of a fixed suite. In production there is no second run to compare against, so truncation is the available signal:

ID    TASK          ACTUAL  TARGET    DIFF    TOTAL  ALLOCATION
7daa  fact              64      18   -256%    410ms  applied 64  truncated

A response that stopped at the ceiling carries finish_reason: "length", and the meted dev log flags it truncated.

x-meted-applied-max-tokens reports the limit that was sent upstream. It does not indicate whether the response reached that limit. To detect truncation from a response, read finish_reason from the body, or choices[].finish_reason from the final streamed chunk.

To track the rate over time, count finish_reason: "length" where your application handles responses, or from the onRecord summary.