An inference-efficiency gateway for LLM applications

Meted sits between your application and your provider, estimates how many output tokens a request needs, and applies a generation budget before the tokens are generated.

Most LLM applications ask for an answer and take whatever length arrives. A one-line factual question and a request to design a distributed system are sent with the same ceiling, usually none at all, and the model decides how much to write.

Sufficiency, not brevity

Meted estimates sufficiency: the number of output tokens a task needs. That estimate is small for some requests and large for others.

$ meted inspect "What is a semaphore?"

definition · simple

Target       38
Maximum      64
Confidence   96%

$ meted inspect "Implement a sliding-window rate limiter in TypeScript with tests"

coding · moderate

Target       980
Maximum     1536
Confidence   90%

Target is the engine's estimate of the tokens the answer needs. Maximum is the ceiling it would enforce, which carries headroom above the target. See Target vs maximum.

Observe mode

Projects start in observe mode. The gateway runs the engine, reports the allocation, and does not apply a ceiling.

$ meted dev

ID    TASK          ACTUAL  TARGET    DIFF    TOTAL  ALLOCATION
a19f  definition       184      38    -79%    902ms  observe_mode
b91c  coding           722     680     -6%     3.1s  observe_mode
7daa  fact              91      18    -80%    410ms  observe_mode

ACTUAL is the output the provider generated. TARGET is the engine's estimate. DIFF is the difference between them as a percentage of actual output.

To decide whether to apply ceilings, run a sample of representative requests through meted eval, which sends each prompt with and without a ceiling and compares output tokens, truncation, answer quality and latency.

meted mode balanced

One line in your application

import OpenAI from 'openai'

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: 'http://localhost:8787/v1',
})

Streaming, tools and response formats work as they did before, and provider errors are forwarded unchanged. If the Sufficiency Engine fails, the request is forwarded without a ceiling and the failure is reported on the response. See Errors.

Local execution

The gateway and the Sufficiency Engine both run in your process, or in your own Cloudflare Worker. Prompts are sent to the provider you configure and to no other host.

Meted reads the provider credential from an environment variable at request time. meted.config.json records the name of that variable rather than its value. See Secrets.

The gateway holds no request history. Per-request details are available on the x-meted-* response headers and through the onRecord callback. To retain them, forward either into your own metrics system.