Methodology
A benchmark is only useful if you can run it again.
Hardware decisions are expensive and hard to reverse. Our benchmarking is designed so that every number we give you can be traced, questioned and reproduced by your own team.
Process
Seven steps from workload to evidence.
- 01
Define the workload
We write down exactly what is being measured: model and weights revision, precision, the distribution of prompt and output lengths, concurrency, and the latency target the service must hold.
- 02
Establish the baseline
The same workload runs on your current hardware and serving stack. Every alternative is compared against this reference, not against a published figure.
- 03
Pin the environment
Drivers, firmware, compute platform versions, framework versions and container digests are recorded alongside every result, so a run can be reproduced later.
- 04
Measure under load
Runs include warm-up, repeat several times and hold load long enough to expose instability. We report distributions and percentiles rather than the best run.
- 05
Check quality parity
Outputs are compared with the baseline using evaluations agreed up front. Numerical divergence and behaviour changes are reported, not hidden.
- 06
Price the result
Throughput at the latency target becomes cost per million tokens, using hardware prices you provide or public pricing, with every assumption stated.
- 07
Publish the evidence
You receive the report together with the scripts, configurations and raw measurements, so your team can re-run any number in it.
Metrics
What we measure, and how we report it.
| Metric | What it tells you | How we report it |
|---|---|---|
| Throughput | How much work the hardware completes | Output tokens per second at the target concurrency |
| Time to first token | How quickly a user sees a response begin | p50, p95 and p99 across the run |
| Inter-token latency | How smoothly a response streams | p50, p95 and p99 across the run |
| Cost per million tokens | What serving the workload actually costs | At the agreed latency target, with pricing assumptions stated |
| Quality parity | Whether the model still behaves the same | Differences from the baseline on the agreed evaluations |
| Stability | Whether it holds up in production conditions | Error rate and memory headroom over a sustained run |
The report
What you receive.
Every benchmark ends in the same structure, so results from different workloads and different hardware can be compared side by side.
- R / 01 Workload definition Exactly what was measured and why.
- R / 02 Environment record Hardware, drivers, software versions and container digests.
- R / 03 Results against baseline Every metric, as distributions, for each hardware option.
- R / 04 Quality parity Evaluation results and any divergence from the baseline.
- R / 05 Cost model Cost per million tokens with each pricing assumption stated.
- R / 06 Limitations What was not tested, and what could change the conclusion.
- R / 07 Reproduction kit Scripts, configurations and raw data to re-run everything.
Ground rules
What we will not report.
- Best-run figures presented as typical performance
- Comparisons against an untuned or outdated baseline
- Throughput quoted without the latency it was measured at
- Results without the configuration needed to reproduce them