The question blas-microbenchmark answers is a comparison: this library against that one, this machine against another, this setting against the default. The benchmarks measure; bmb_report lays the results side by side.

Measure each library

There are two ways to run the same benchmarks against several libraries:

• A build per library, each in its own directory (see Install). The same commands, run with each build, into separate files:

  build-openblas/src/c/level3/double/bmb_dgemm -m 128:2048 -t 1:8 -o results/dgemm-openblas.json
  build-blis/src/c/level3/double/bmb_dgemm -m 128:2048 -t 1:8 -o results/dgemm-blis.json

• One build against FlexiBLAS, which loads the library FLEXIBLAS names when the benchmark starts (flexiblas list gives the names):

  FLEXIBLAS=OPENBLAS-OPENMP bmb_dgemm -m 128:2048 -t 1:8 -o results/dgemm-openblas.json
  FLEXIBLAS=BLIS-OPENMP bmb_dgemm -m 128:2048 -t 1:8 -o results/dgemm-blis.json

Every file records the library it ran against, its version, the CPU and the date: the report keeps them apart without you naming them.

Make the report

bmb_report results/*.json > report.html
firefox report.html

Give it the machine's peak as you know it, a core's FLOP rate in double precision and the memory bandwidth, and it judges the rates by it: bmb_report --peak-gflops 76.8 --peak-gbs 120 results/*.json > report.html. It is never estimated.

The page is a single file, with nothing to fetch: it opens offline, and can be sent as is. For each routine, it shows what the results can support:

• performance, and bandwidth for levels 1 and 2, against size, with the points where the data leaves the L1, L2 and L3 caches marked;
• each library against a reference (OpenBLAS, or one you choose on the page), on a log scale where twice as slow and twice as fast sit at the same distance;
• thread scaling against ideal;
• a heatmap of every shape, for sweeps over two dimensions (-m and -M);
• given the peak, a roofline for levels 1 and 2, each size under the memory and compute roofs, and a line at the peak on level 3's performance chart;
• a summary, and every row of every file, sortable.

The Outputs page shows each part.

The whole suite at once

bmb_suite runs every benchmark, or those of some levels, precisions or routines, at the sizes of a profile, into one directory, which bmb_report takes whole. quick measures a few sizes a level, in about a minute; full sweeps from the caches to memory, in several. The options after -- go to every benchmark:

bmb_suite --profile quick -o results -- -t 1,8
bmb_report results > report.html

With --blas <name>=<suite>, once a library, it runs the blas-microbenchmark of each build above, an install prefix or a build tree, into a directory a name, each benchmark labelled with it; each result is checked to record the library its build was made for:

bmb_suite --blas openblas=build-openblas --blas blis=build-blis --profile quick -o results
bmb_report results > report.html

With one build against FlexiBLAS, --blas flexiblas:OPENBLAS-OPENMP,BLIS-OPENMP runs it once a library, FLEXIBLAS set to each name.

Its manual, bmb_suite, has the profiles, how to write your own, and the same with Spack.

Is the difference real?

Two runs of the same command never give quite the same numbers. Run each command several times, each into its own file:

for run in 1 2 3 4; do
    bmb_ddot -v 1024:16777216 -o results/ddot-openblas-$run.json
done

The report then draws a band showing how far the runs spread, and tests each difference between libraries against that spread (a Mann-Whitney U test): a point within the noise is drawn hollow. A comparison from single runs shows the differences, but not which ones to believe.

One library, several versions

Measure three versions or more of the same library on the same machine, and the report adds a history: each routine's performance, version after version, with a version slower than the one before beyond the noise marked slower. The version comes from the library itself: two versions are two series, whatever they are called.

The benchmarks measure the library they were built against, so each version, or each build of one version (another compiler, other options), gets a blas-microbenchmark of its own: one Spack install each, or one ./configure --with-blas-incpath=… --with-blas-libpath=… each, under a prefix of its own. bmb_suite then runs them all, each under a name of its own:

bmb_suite --blas openblas-0.3.26=/opt/bmb-openblas-0.3.26 --blas openblas-0.3.30=/opt/bmb-openblas-0.3.30 --blas openblas-0.3.34=/opt/bmb-openblas-0.3.34 -o results
bmb_report results > report.html

The names matter in two cases:

• Two builds of one version may give the same version string, so only their names tell them apart: say what differs, as in openblas-0.3.34-gcc13 and openblas-0.3.34-gcc14.
• A history takes three versions or more whose names differ only by the version, written as the library prints it: the report takes the version out of the name to group them. openblas-0.3.26-gcc14, openblas-0.3.30-gcc14 and openblas-0.3.34-gcc14 are one history, their -gcc13 builds another.

How to build them: bmb_suite's manual.

Two conditions on one machine

To compare a setting rather than a library (turbo, a thread placement, a compiler flag of the library), give each run a label; the report makes a series of each:

bmb_dgemm -m 1024:8192 -l "turbo on" -o results/turbo-on.json
bmb_dgemm -m 1024:8192 -l "turbo off" -o results/turbo-off.json

A fair comparison

• The same operands. Every library gets the same values, scalars, sizes, alignment and layout: the benchmarks fill the operands themselves, the same way whatever the library, with no subnormals, NaN or infinities, which some processors and libraries handle more slowly. Only the library differs (The operands).
• One after the other. bmb_suite runs the libraries in turn, so a machine heating up or a job started meanwhile can favour one: run it four times or more, each time into a directory of its own, and give bmb_report every directory (Is the difference real?).
• Same machine, same conditions. A comparison across machines measures the machines.
• A steady frequency. The performance governor and turbo off, if you can; the benchmark warns when they are not, and the report says under what conditions each series ran.
• The same threads, pinned. See pinning threads, and Threads.
• The library's own choices. MKL and AOCL-BLAS may use fewer threads than asked on small problems; see The BLAS libraries.

Everything the report draws, and how: bmb_report's manual.