Measure the right question.
Use a metric for the question it can answer. Interpret latency distributions, reported usage, modeled costs and task-specific quality without collapsing them into a single unexplained score.
METRIC REFERENCE
Read the definition first.
| Signal | Useful interpretation | Important boundary |
|---|---|---|
| Median latency | The middle observation in an ordered sample set. | Not a guarantee for every request. |
| Tail latency | A high percentile that describes slower observations. | Small samples do not establish a production SLA. |
| Time-to-first-token | Time until the first output token is observed in a streaming measurement. | Not implemented by the current non-streaming benchmark CLI. |
| Output tokens/sec | Token output divided by the explicitly defined measurement interval. | Full-response throughput is not streaming decode speed. |
| Per-request cost | Usage multiplied by a clearly identified pricing model. | Requires actual usage, pricing and retry/cache accounting. |
| Quality signals | Task-specific evaluation outcomes and operational feedback. | Latency or cost alone does not measure answer quality. |
WORKING WITH SAMPLES
Keep conditions comparable.
Record the served model, prompt category, output limit, environment, concurrency, warmup and cache state when known. Label unknown infrastructure explicitly.
The separate record workspace currently calculates descriptive nearest-rank latency percentiles, mean latency and sequential reported-token throughput. This broader analytics architecture is a concept, not a claim that all those measurements are available in the current collector.
Read the documentation ↗