Telling a real benchmark difference from noise

A benchmark table has one question per row: is Prismio faster, slower, or level? Answering it with a fixed percentage fails in both directions on short workloads. A workload that runs for 0.6 ms is dominated by things no code change moves: process start, which core the scheduler picks, the clock ramping up.

The problem: two timing modes

Those effects show up as two timing modes, a fast one and one roughly a quarter slower, and the share of runs landing in each differs from one binary to the next. Consider five recorded runs of two workloads, in microseconds:

large_buffer_copy is a smooth cloud. Every Prismio run is slower than every C++ run, so the difference is real. edit_distance is two clusters, near 570 µs and near 720 µs, in both languages. The medians differ because three of Prismio's five runs fell in the fast cluster and four of C++'s five in the slow one, while the fastest runs, 566 and 587, are within 4% of each other. The code takes the same time in each mode.

Under a flat 4% rule the median ratio, 0.81, would be reported as Prismio being 19% faster. That is as wrong as the opposite reading of a different set of five runs.

The rule

A comparison is a win or a loss only when both of these leave a tolerance, and is parity otherwise:

1. the ratio of medians, against the tolerance max(parity, spread, floor / median);
2. the ratio of best runs (minimums), against max(parity, floor / minimum).

There are two ratios because noise only adds time. The best run is the steadiest estimate of what the code costs; the median says what a run usually costs. A real difference moves both. Equal best runs with different medians is the signature of timing modes, and is parity.

The spread is the interquartile range, not the median absolute deviation. When more than half the runs sit in one mode the deviation is close to zero, so the older estimate reported edit_distance's noise as nearly 0% and a 700 µs against 560 µs split as a loss. The interquartile range sees both modes.

Result

Applied to the rows above, and to a third much shorter workload:

In a live run the same distinction appears in the summary. edit_distance at 1.22 times C++ is printed as parity, and large_buffer_copy at 1.16 times is printed as a loss, because the colour follows the verdict and not the ratio.

Reproducing it

python3 benchmarks/run.py --only edit_distance --only large_buffer_copy --runs 7

Since results schema version 3 the verdict is written into the result file, so no reader has to recompute it:

"verdict": {
  "cpp": {
    "outcome": "parity",
    "ratio": 0.809,
    "best_ratio": 0.9631,
    "tolerance": 0.2126,
    "noise": 0.2126
  }
}

outcome is win, parity or loss. ratio and best_ratio are Prismio's time over the other arm's, so below 1 means Prismio is ahead. tolerance is what the medians were judged against, and noise is the spread it contains.

Reading a disagreement

When a row is called a loss and the verdict is in doubt, check the three numbers before the code.

• Samples in two clusters with equal minimums are timing modes, and parity is the right answer. Rerun with more runs if a number is needed.
• A smooth cloud whose ratios agree, with a tolerance near 0.04, is a real difference. Time the phases of the workload separately in both languages before changing the compiler.
• A large tolerance because the spread is large means the verdict cannot be trusted either way. Increase the number of runs, close other work, and check the power state recorded with the results.

Limits

• The rule is not a significance test. Five samples a side cannot support a p-value, and the samples are interleaved rather than randomised, so they are not independent draws.
• It assumes noise only adds time. A workload whose fastest run is an outlier in the other direction would be judged on a minimum that is not representative, so the raw samples should be read.
• It does not call timing modes a defect in either language. If one binary lands in the slow mode more often for a systematic reason, that is a finding for a separate investigation.
• The 25 µs floor was chosen from one host. On a machine with faster process start it is generous.
• An arm compiled with less information is slower for reasons unrelated to the language. The Prismio arm is built as one whole program; the C++ and Rust arms are built with link-time optimisation so that a constant passed across a file boundary reaches the code that uses it in all three. Without that, one graph workload divided by a run-time value in C++ and Rust and by a folded constant in Prismio, and read as an 8% loss that C++ at the same constant did not show.

@misc{jaiswal2026tellingadifferencefromnoise,
  title        = {Telling a real benchmark difference from noise},
  author       = {Saksham Jaiswal},
  year         = {2026},
  month        = {oct},
  howpublished = {Prismio Research},
  note         = {Prismio 0.1.0},
  url          = {https://research.prismio.org/papers/telling-a-difference-from-noise}
}