blas-microbenchmark1.5.0-dev

bmb_report(1)

one HTML page of charts from blas-microbenchmark results

bmb_report result.json... > report.html

bmb_report turns the JSON results written by the benchmarks of blas-microbenchmark(1) (bmb_routine -o result.json) into one HTML page of charts, written to standard output.

The page is self-contained: data, styles and script are all inside it and nothing is fetched, so it opens from disk on a machine with no network, and can be sent as a single file. Open it with a browser.

Each part appears only when the results can support it:

  • a summary: the best figure for each routine, and where each series came from (library, CPU, caches, OS, date, and the CPU frequency governor and turbo where the machine exposed them) — and whether each series was checked against a reference before it was timed, once any was (the benchmarks do it unless run with -C);
  • performance against size, and for levels 1 and 2 bandwidth against size, with the points where a sweep leaves L1, L2 and L3 marked. For a series with repeated runs (see Repeated runs and noise), a band shows the spread from the fastest run to the slowest; otherwise, for results measured with -s, from the fastest sample to the mean;
  • each library compared with a reference, on a log scale centred on ×1, so that twice as slow and twice as fast sit at the same distance from it. The reference is OpenBLAS when there is one, and can be changed on the page. With repeated runs, a hollow point is a difference within run-to-run noise;
  • the history of a library measured in three versions or more (see History of a library);
  • thread scaling, at the largest size measured at every thread count, against ideal linear scaling;
  • a heatmap of every shape, for two-dimension routines measured with both -m and -M;
  • the raw data, every row of every file, sortable, with whether each point was checked when any was.

The charts come in a section per routine, level 1 first; within a level, operations come in alphabetical order, each one's precisions together in BLAS order: sgemm, dgemm, cgemm, zgemm, then zhemm, zherk, dsymm. Hovering a chart gives the exact values; the arrow keys do the same from the keyboard.

Results are grouped into series by backend, library version, CPU, label (-l) and storage order (-L; a series measured column-major is named so, and results without one, from vector routines or from before 1.4, count as row-major). Files that share all five make one curve: a sweep extended the next day, or thread counts measured in separate runs, merge on their own. A point measured in more than one file is drawn at the fastest measurement, and the page says so with the gap, so that a run which did not reproduce does not go unnoticed; the others are kept as its runs. To compare two conditions on one machine, give each run its own label.

Series that differ only by the library's version — the same backend, CPU, label and storage order — make a history once there are three of them; two are the comparison chart's. For each routine, at the thread count chosen on the page, it shows the largest size every version measured, version after version, ordered by the version number in the library's version string, oldest first. With repeated runs, each version is tested against the one before (see below), and one slower beyond the noise is marked slower. Measure every version on the same machine: a history across machines measures the machines.

Running the same command several times, each into its own file, gives each point several runs. The figure drawn stays the fastest run, as within a run it is the fastest sample. The band around a curve then shows how far the runs spread, the tooltip gives their number, median and slowest, and the comparison with the reference is tested against that spread: a Mann-Whitney U test on the times of both series' runs, two-sided, at p < 0.05. A point within the noise is drawn hollow, and its tooltip gives the p value.

It takes runs to tell a difference from noise: 4 of each series, or 3 against 5. With 3 against 3, no outcome can reach p < 0.05, so the page says there are too few runs rather than calling the difference noise.

Results of the same major version as bmb_report itself. Anything else is refused before a page is written, with the file name and the reason: results from before 0.6.0 measured a mean rather than the fastest sample, and those from 0.6.x do not record the machine. One refused file means no page at all, since a page missing a library would be read as complete.

Show a summary of the usage.
Show the version.

0 when the page was written; 1 when a file was refused, and then nothing is written; 2 for a usage error, including standard output being a terminal — the page is HTML, to be redirected to a file.

OpenBLAS against BLIS, each build run with the same commands:

bmb_dgemm -m 128:2048 -t 1:8 -o results/dgemm-openblas.json
bmb_dgemm -m 128:2048 -t 1:8 -o results/dgemm-blis.json  # BLIS build
bmb_report results/*.json > report.html
firefox report.html

With one build against FlexiBLAS, the library is chosen when it runs, and each comes out as its own series:

FLEXIBLAS=OPENBLAS-OPENMP bmb_dgemm -m 128:2048 -o results/dgemm-openblas.json
FLEXIBLAS=BLIS-OPENMP bmb_dgemm -m 128:2048 -o results/dgemm-blis.json
bmb_report results/*.json > report.html

Three versions of OpenBLAS, each installed in turn, five runs each, make a history (the version comes from the library itself):

for i in 1 2 3 4 5; do bmb_dgemm -m 1024 -o results/openblas-0.3.28-$i.json; done
bmb_report results/*.json > report.html

The same, five runs of each, so that the comparison says which differences are beyond noise:

for i in 1 2 3 4 5; do bmb_dgemm -m 1024 -o results/openblas-$i.json; done
bmb_report results/*.json > report.html

blas-microbenchmark(1)

https://github.com/MathieuCAISSA/blas-microbenchmark