Architecture

How the pieces fit, and why the request path is small.

Application
    ↓
Meted Gateway
    ↓
Sufficiency Engine
    ↓
LLM Provider

The request path

request → engine decision → apply allocation → provider → stream → summarise

Allocation is skipped when the mode is observe, when the decision falls below the mode's minConfidence, or when the caller's own max_tokens is already lower than the ceiling Meted would apply. In each case the request is forwarded without a ceiling and the reason is reported as skipReason.

When the engine times out, throws or returns an invalid decision, the allocation step is skipped entirely and the request is forwarded unchanged. See Errors.

Anything that can be resolved at start-up is resolved at start-up.

Measured for every request:

The engine decision and the allocation are reported on the response headers. Token counts, finish_reason and the latency figures are known only after the response completes and reach the embedding process through onRecord.

The packages

Dependencies flow one way:

engine-interface → config → gateway → eval → cli
providers ──────────────────↗

Nothing depends on cli. config splits its Node-specific parts into a /node export so the rest stays importable from a Worker.

Why the engine is a separate contract

The Sufficiency Engine is the one part with a proprietary implementation, so it is the one part behind an interface strict enough to swap.

interface SufficiencyEngine {
  readonly name: string
  readonly version: string
  readonly proprietary?: boolean
  evaluate(request: SufficiencyRequest, options?): Promise<EngineDecision>
}

The loader resolves, in order: an explicit WASM module, an explicit JS module, the private @meted/engine package if installed, then the Apache-2.0 baseline. All of them run in-process.

The open-source repository ships a working engine, and the proprietary engine is a drop-in replacement for it.

onRecord

The gateway holds no request history. To collect per-request data, pass an onRecord callback to createGateway. It is called once per finished request with a RequestSummary of counts, durations and labels:

const gateway = createGateway({
  config,
  engine,
  apiKey,
  onRecord: (summary) => log.info(summary),
})

meted dev uses this callback to print its log line. Send the summary to your metrics system to aggregate it:

onRecord: (summary) => {
  metrics.histogram("meted.output_tokens", summary.outputTokens ?? 0, {
    task: summary.taskType ?? "unknown",
    applied: String(summary.allocationApplied),
  });
  if (summary.finishReason === "length") metrics.increment("meted.truncated");
},

The callback runs after the response has been returned to the caller, so it does not add latency. On Cloudflare it runs inside ctx.waitUntil.

Why the provider is an adapter

"Which field is the token ceiling" is a per-provider question with per-model exceptions. Confining it to an adapter keeps it out of the gateway. See Provider adapters.

Where state lives