Methodology

A benchmark is only useful if you can run it again.

Hardware decisions are expensive and hard to reverse. Our benchmarking is designed so that every number we give you can be traced, questioned and reproduced by your own team.

Process

Seven steps from workload to evidence.

  1. 01

    Define the workload

    We write down exactly what is being measured: model and weights revision, precision, the distribution of prompt and output lengths, concurrency, and the latency target the service must hold.

  2. 02

    Establish the baseline

    The same workload runs on your current hardware and serving stack. Every alternative is compared against this reference, not against a published figure.

  3. 03

    Pin the environment

    Drivers, firmware, compute platform versions, framework versions and container digests are recorded alongside every result, so a run can be reproduced later.

  4. 04

    Measure under load

    Runs include warm-up, repeat several times and hold load long enough to expose instability. We report distributions and percentiles rather than the best run.

  5. 05

    Check quality parity

    Outputs are compared with the baseline using evaluations agreed up front. Numerical divergence and behaviour changes are reported, not hidden.

  6. 06

    Price the result

    Throughput at the latency target becomes cost per million tokens, using hardware prices you provide or public pricing, with every assumption stated.

  7. 07

    Publish the evidence

    You receive the report together with the scripts, configurations and raw measurements, so your team can re-run any number in it.

Metrics

What we measure, and how we report it.

MetricWhat it tells youHow we report it
ThroughputHow much work the hardware completesOutput tokens per second at the target concurrency
Time to first tokenHow quickly a user sees a response beginp50, p95 and p99 across the run
Inter-token latencyHow smoothly a response streamsp50, p95 and p99 across the run
Cost per million tokensWhat serving the workload actually costsAt the agreed latency target, with pricing assumptions stated
Quality parityWhether the model still behaves the sameDifferences from the baseline on the agreed evaluations
StabilityWhether it holds up in production conditionsError rate and memory headroom over a sustained run

The report

What you receive.

Every benchmark ends in the same structure, so results from different workloads and different hardware can be compared side by side.

  1. R / 01 Workload definition Exactly what was measured and why.
  2. R / 02 Environment record Hardware, drivers, software versions and container digests.
  3. R / 03 Results against baseline Every metric, as distributions, for each hardware option.
  4. R / 04 Quality parity Evaluation results and any divergence from the baseline.
  5. R / 05 Cost model Cost per million tokens with each pricing assumption stated.
  6. R / 06 Limitations What was not tested, and what could change the conclusion.
  7. R / 07 Reproduction kit Scripts, configurations and raw data to re-run everything.

Ground rules

What we will not report.

  • Best-run figures presented as typical performance
  • Comparisons against an untuned or outdated baseline
  • Throughput quoted without the latency it was measured at
  • Results without the configuration needed to reproduce them