First request
Sending traffic through Meted and reading what it decided.
With meted dev running, change one line in your application.
import OpenAI from 'openai'
const client = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
baseURL: 'http://localhost:8787/v1',
})
const completion = await client.chat.completions.create({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'What is a semaphore?' }],
})
Any OpenAI-compatible client works the same way:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8787/v1")
curl http://localhost:8787/v1/chat/completions \
-H "content-type: application/json" \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"What is a semaphore?"}]}'
What comes back
The response body is what the provider returned. The decision travels in headers:
x-meted-request-id a19f0c4e91b3d27a
x-meted-mode observe
x-meted-task-type definition
x-meted-complexity simple
x-meted-target-tokens 38
x-meted-max-tokens 64
x-meted-confidence 0.96
x-meted-engine-ms 0.41
x-meted-overhead-tokens 0
x-meted-applied false
x-meted-skip-reason observe_mode
applied: false with skip_reason: observe_mode: Meted worked out that a
sufficient answer costs about 38 tokens, reported that, and sent your request
upstream untouched.
In the JS SDK, reach the headers with .withResponse():
const { data, response } = await client.chat.completions
.create({ model: 'gpt-4o-mini', messages })
.withResponse()
const requestId = response.headers.get('x-meted-request-id')
const target = response.headers.get('x-meted-target-tokens')
Read the request log
The terminal running meted dev prints a line per request:
ID TASK ACTUAL TARGET DIFF TOTAL ALLOCATION
a19f definition 184 38 -79% 902ms observe_mode
The provider generated 184 tokens; the engine estimated a target of 38. Observe
mode does not test whether a 38-token answer would have satisfied the request.
meted eval does, by sending the same prompt with and without a
ceiling.
Keeping the numbers
Record the response metadata in your application's metrics system to track it over time:
const { response } = await client.chat.completions
.create({ model, messages })
.withResponse()
metrics.record({
taskType: response.headers.get('x-meted-task-type'),
target: Number(response.headers.get('x-meted-target-tokens')),
applied: response.headers.get('x-meted-applied') === 'true',
})
The gateway keeps no request history, so anything you want to query later has
to be recorded here or through
onRecord.
Next
Run a set of requests representative of your traffic, then compare answer
quality, truncation, token use and latency with and without a ceiling using
meted eval. Modes covers how to apply one.