Eval format

Writing a suite of your own.

A suite is JSON.

{
  "name": "Checkout assistant",
  "description": "Prompts from the live product, sampled over a week.",
  "model": "gpt-4o-mini",
  "prompts": [
    {
      "id": "refund-policy",
      "category": "fact",
      "prompt": "What is our refund window?",
      "expect": { "contains": ["30 days"], "maxWords": 80 },
    },
  ],
}

Suite fields

Field
`name`	Shown in the report
`description`	For your own benefit
`model`	Default model. `--model` overrides it
`evaluator`	Path to a module exporting a custom evaluator
`prompts`	At least one prompt

Prompt fields

Field
`id`	Unique. Shown in under-allocation output
`category`	Groups the per-category breakdown. Default `general`
`prompt`	Shorthand for a single user message
`messages`	Full message list, when a system prompt or prior turns matter
`expect`	Deterministic assertions
`judge`	A rubric for the optional LLM judge
`skip`	Skip without deleting

Use messages when conversation state is part of what you are testing:

{
  "id": "followup-expand",
  "category": "conversation",
  "messages": [
    { "role": "user", "content": "What is a monad?" },
    {
      "role": "assistant",
      "content": "A way of sequencing computations that carry extra context.",
    },
    { "role": "user", "content": "tell me more" },
  ],
  "expect": { "minWords": 60 },
}

Assertions

Key
`contains`	All of these substrings must appear (case-insensitive)
`containsAny`	At least one must appear
`excludes`	None may appear
`regex`	The answer must match
`minWords`, `maxWords`	Word-count bounds
`json`	Must parse as JSON, with or without a code fence
`code`	Must contain a fenced code block
`minItems`	Must contain at least this many list items

A prompt with no assertions is still evaluated for emptiness and truncation.

Writing good assertions

Assert the floor, not the shape. minWords: 300 on a research prompt catches under-allocation. maxWords: 320 on the same prompt fails on a good answer.

contains and containsAny differ in how many entries must match. Every string in contains must appear. At least one string in containsAny must appear. Use containsAny to accept alternatives:

"expect": { "containsAny": ["Canberra", "canberra"] }

Check the fact, not the phrasing. ["Canberra"] passes whatever sentence the model builds around it. ["The capital of Australia is Canberra"] fails on any rewording, which reports a truncation problem that is not there.

Use code: true for anything that must ship code.

Keep rubrics for what deterministic checks cannot see. Every prompt with a judge rubric costs an extra model call on every run.

Regenerating the built-in suite

pnpm --filter @meted/eval build
node packages/eval/scripts/emit-builtin.mjs

packages/eval/src/builtin.ts is the source of truth; evals/builtin.json is generated from it.