The results

Each routine writes dir/routine.json, and what the benchmark printed goes to dir/routine.log. dir is created, and refused when it already holds files: the results of two runs would mix in the report. A line on standard error says which routine runs, and how many are left.

A benchmark that fails, a result wrong under --verify or an option it refuses, is said with the end of its log, and the others still run; one that skips (exit 77: an extension of the BLAS the library lacks, axpby, gemmt or sbgemm, a batch mode it does not have, or a batch it would crash on) is said too, with its log and no result.

Sizes and profiles

Without --profile, each benchmark runs at its level's default size. A profile gives the sizes instead: a line a level (level1, level2, level3) or a routine, then the options it is run with; a routine's line wins over its level's. A # starts a comment. Two are shipped. quick measures a few sizes a level, about a minute on one core of a laptop:

level1 -v 1000,100000,10000000
level2 -m 200,2000
level3 -m 200,1000
sgbmv -m 200,2000 -M 32
dgbmv -m 200,2000 -M 32
stbsv -m 200,2000 -M 32
dtbsv -m 200,2000 -M 32

full doubles the size from the first-level cache to memory at levels 1 and 2, and up to 2048 at level 3, several minutes on the same core:

level1 -v 256:16777216
level2 -m 32:4096
level3 -m 32:2048
sgbmv -m 32:4096 -M 32
dgbmv -m 32:4096 -M 32
stbsv -m 32:4096 -M 32
dtbsv -m 32:4096 -M 32

The band routines get K = 32 diagonals: by default, K is N, a band as wide as the matrix.

Several libraries

The benchmarks measure the library they were built against: a blas-microbenchmark a library. --blas name=suite runs the one in suite, an install prefix (where libexec/blas-microbenchmark is), a build tree (where src/c is) or the directory of its levels, into dir/name. It can be given once a library; then the suite bmb_suite is installed with is not run, unless given too. The selection, the profile and the options apply to each, and bmb_report(1) takes dir whole, a series a library.

Each benchmark is run with --label name, which tells apart two builds of one library (two versions, two compilers) in the report; a name with the library's version in it (openblas-0.3.26) lets the report draw the versions' history, so -l is refused among the options passed on. The backend each suite was built for is read from its benchmarks' --version before anything runs, and each result must record it: one recording another fails, rather than becoming a series under the wrong name. A name is letters, digits, ., _ and -, each given once.

Every suite runs the same benchmarks (but an extension of the BLAS a library lacks, which it skips), with the same options, on the same operands: their values, sizes, alignment and layout come from the benchmarks, never from the library (see The operands in blas-microbenchmark(1)). The suites run one after the other, so a machine that heats up, or a job started meanwhile, can favour one: run bmb_suite four times or more, each time into a directory of its own, and give bmb_report(1) every directory, which tests each difference against the spread of the runs.

With Spack, a blas-microbenchmark a library:

spack install blas-microbenchmark ^openblas
spack install blas-microbenchmark ^blis
spack install blas-microbenchmark ^intel-oneapi-mkl
O=$(spack location -i blas-microbenchmark ^openblas)
B=$(spack location -i blas-microbenchmark ^blis)
M=$(spack location -i blas-microbenchmark ^intel-oneapi-mkl)
bmb_suite --blas openblas=$O --blas blis=$B --blas mkl=$M --profile quick -o results
bmb_report results > report.html

or the three in a Spack environment, a spack.yaml of its own directory:

spack:


  specs:


  - blas-microbenchmark ^openblas


  - blas-microbenchmark ^blis


  - blas-microbenchmark ^intel-oneapi-mkl


  concretizer:


    unify: false

which spack -e . install builds, and spack -e . location -i then finds, as above.

A build against FlexiBLAS loads the library FLEXIBLAS names when a benchmark starts, and records it (flexiblas/openblas-openmp). --blas flexiblas:name,... runs the suite bmb_suite is installed with, which must be built against FlexiBLAS, once a library, with FLEXIBLAS set to its name, into dir/name:

bmb_suite --blas flexiblas:OPENBLAS-OPENMP,BLIS-OPENMP --profile quick -o results

flexiblas list gives the names. For a name it does not know, FlexiBLAS loads its default library, so each name is checked before anything runs: the benchmarks' --version must record it. Both forms of --blas can be given together. What it costs: FlexiBLAS must be installed with its libraries (Fedora packages it, and Spack builds it; Debian and Ubuntu do not), and its indirection, a call through a pointer, shows on the smallest level 1 calls.

One library, several builds

One library can be given more than once: several versions of it, or one version built with two compilers or two sets of options. Each build gets a blas-microbenchmark of its own, built against it, and a name of its own:

spack install blas-microbenchmark ^openblas@0.3.26
spack install blas-microbenchmark ^openblas@0.3.30
spack install blas-microbenchmark ^openblas@0.3.34
A=$(spack location -i blas-microbenchmark ^openblas@0.3.26)
B=$(spack location -i blas-microbenchmark ^openblas@0.3.30)
C=$(spack location -i blas-microbenchmark ^openblas@0.3.34)
bmb_suite --blas openblas-0.3.26=$A --blas openblas-0.3.30=$B \


    --blas openblas-0.3.34=$C -o results

Without Spack, each one is configured against its build of the library, which the benchmarks then load wherever it is (--with-blas-libpath adds its directory to their run path), and installed under a prefix of its own:

./configure --prefix=/opt/bmb-openblas-0.3.34-gcc14 \


    --with-blas-backend=openblas \


    --with-blas-incpath=/opt/openblas-0.3.34-gcc14/include \


    --with-blas-libpath=/opt/openblas-0.3.34-gcc14/lib

The report tells two versions apart by the version the library gives, whatever their names; two builds of one version, which may give the same string, only by their names: give them names that say what differs (openblas-0.3.34-gcc13, openblas-0.3.34-gcc14). Three versions or more whose names differ only by the version, written as the library prints it, make a history: the report takes the version out of the name to group them, so openblas-0.3.26-gcc14, openblas-0.3.30-gcc14 and openblas-0.3.34-gcc14 are one history, and their -gcc13 builds another.

Options for every benchmark

The words after -- go to every benchmark, after the profile's: -t 1,8 for the thread counts, --cache cold, -i 50. bmb_suite gives each -f json -o itself, last. It refuses -o and -f there, and --peak-gflops, whose value is in the routine's precision and would be wrong for half of them: give the peak to bmb_report(1) instead. An option a routine has no use for is ignored with a warning in its log; one it refuses (--trans on ddot) makes it fail.