blas-microbenchmark1.5.0-dev

blas-microbenchmark(1)

microbenchmarks for BLAS routines

bmb_routine [options]

Each bmb_routine times one BLAS routine, called through CBLAS, over a sweep of sizes and thread counts, and prints one row per point: the time of a call and the rate it achieved. All of them take the same options. They run against the BLAS library chosen when the package was built (OpenBLAS, BLIS, Netlib reference BLAS, NVPL, ArmPL or Intel MKL), or, built against FlexiBLAS, the one FlexiBLAS loads when they start (FLEXIBLAS, see ENVIRONMENT); --version says which.

To compare libraries, build the package once against each and run the same command with each build, or build it once against FlexiBLAS and run the command under each library; then turn the results into charts with bmb_report(1).

Level 1 routines work on vectors, level 2 on a matrix and a vector, level 3 on matrices:

Level 1 dasum daxpy dcopy ddot dnrm2 dscal dswap
saxpy sdot zdotc
Level 2 dgemv dger dsymv dsyr dsyr2 dtrmv dtrsv
sgemv
Level 3 dgemm dsymm dsyrk dsyr2k dtrmm dtrsm
sgemm cgemm zgemm strsm ztrsm ssyrk zherk zhemm

The first letter is the precision, as in BLAS: s single, d double, c single complex, z double complex. Every operation is measured in double precision, and the most used ones in the others too: single precision moves half the bytes and fits twice as many numbers in a vector register, and a complex multiply-add does four times the arithmetic of a real one on twice the data. Three exist for complex numbers only: zdotc, the dot product with the first vector conjugated, and zherk and zhemm, the Hermitian counterparts of dsyrk and dsymm.

Level 1 routines are sized with -v. Level 2 and 3 routines are sized with -m and, for those with two dimensions, -M. Which dimension is which:

dgemv sgemv dger dsymm zhemm -m is M, -M is N
dtrmm dtrsm strsm ztrsm -m is M, -M is N
dgemm sgemm cgemm zgemm -m is M and K, -M is N
dsyrk ssyrk zherk dsyr2k -m is N, -M is K
dsymv dsyr dsyr2 dtrmv dtrsv -m is N (square; no -M)

Untimed calls before measuring each point. Default: 1.
Timed samples taken at each point. Default: 10.
Calls averaged per sample; 0 chooses automatically, per point (see HOW A NUMBER IS MEASURED). --batch 1 times every call on its own. dtrmv, dtrsv, dtrmm, dtrsm, strsm and ztrsm are always timed that way: their operand is restored before every call, and a larger -b is ignored with a warning. Default: 0.
Vector sizes, for level 1 routines. Default: 4096.
First matrix dimension, for level 2 and 3 routines. Default: 4096.
Second matrix dimension. Without it, matrices are square: the second dimension follows the first at every point. With it, every combination of the two is measured.
Number of BLAS threads. Default: 1. See THREADS.
Also report, per point, the mean of the samples, their standard deviation (over the samples as measured, divided by n rather than n-1), the slowest sample, the batch size and the median of the samples, as columns in that order after the rates.
Before timing each point, check that the library computes the right result there; stop with exit status 2 if it does not. See CHECKING RESULTS. Default: on.
Do not check the results.
How the matrices are stored and passed to CBLAS: row (row-major, as C stores arrays) or col (column-major, as Fortran and LAPACK do). A library may handle the two differently: CBLAS row-major calls are often turned into column-major ones on the transposed problem, so a Fortran program can see other numbers than the default does. The problem is the same either way (the same sizes, the upper triangle, no transpose); only its storage changes. Ignored, with a warning, by the vector routines. Default: row.
Tag the results, for what no automatic field records: a BIOS setting, a NUMA binding, SMT off. 1 to 200 characters, no control characters. Default: none, and no label is written.
Also save the results to file.
csv or json for -o. Default: guessed from the extension of the -o file, otherwise csv.
Show a summary of the options.
Show the version, the BLAS library and the machine, without running anything.

The four size options each take a sweep:

max that one size 4096
min:max doubling 256:4096 → 256 512 1024 2048 4096
min:max:step linear 1000:4000:1000 → 1000 2000 3000 4000
v1,v2,... exactly those 64,1000,4096

max is always measured, even when the step would overshoot it: 100:1000:300 gives 100, 400, 700 and 1000. A sweep has at most 256 points. A size whose memory would not fit in size_t is refused before anything is allocated.

Reading the clock costs tens of nanoseconds, which is the whole duration of a small level 1 call. So each sample times a batch of calls and divides by the batch size. The batch size is chosen per point, so that a batch lasts long enough for the clock not to matter; calls that already take longer get a batch of 1, so nothing changes for level 3 or for large sizes. -s shows the batch size chosen.

The reported time is the fastest sample, not the mean. A sample is already an average over a whole batch, so the fastest is not a lucky call: it is the batch that ran with the least interference. One scheduling hiccup inflates a sample a great deal, and would drag a mean with it; -s shows the mean and the spread for that reason.

Two rates are derived from that time:

the conventional operation count of the routine (2×M×N×K for dgemm, K×N×(N+1) for dsyrk, ...) divided by the time. Not shown for dcopy and dswap, which do no arithmetic. Complex routines count real operations: a complex multiply-add is four real multiplications and four additions, so each counts four times its real counterpart's operations (cgemm and zgemm 8×M×N×K where dgemm counts 2×M×N×K; zherk four times dsyrk's), and their GFLOP/s compare with the real routine's.
the bytes the routine must move — each operand read once, each result written once, counted the way STREAM counts them, 4 bytes an element in single precision and 8 in double — divided by the time. Levels 1 and 2 only: level 3 routines reuse their operands from cache, so a rate built from that traffic would not measure bandwidth.

A benchmark times a call whatever it computes: a library that gets the result wrong — a broken build, a bug at one size or thread count — can look fast. So each point is first checked, unless -C is given: one call on the operands about to be timed, at that size and thread count, compared with a reference computed independently.

Level 1 and 2 results are recomputed directly, which costs about one more call. Level 3 results are checked by random projection: for a random vector r, C*r is compared with A*(B*r), and for dtrsm, strsm and ztrsm the residual U*(X*r) with B*r. That takes as long as a level 2 call, where recomputing C would take as long as the call itself, and a wrong C gives a wrong C*r for almost every r. Routines that update one triangle are also checked to leave the other as it was.

The tolerance is a bound on rounding error, proportional to the number of terms in each element and to their magnitudes: a correct library is well inside it, a wrong one far outside. In single precision the reference is still computed in double, and the bound half a billion times wider: it catches a library that computes the wrong thing, but not a small error in a long sum (at a million elements, sdot is only checked to within the size of its terms). A wrong result is reported with the routine, the size, the thread count and the element that differs; nothing more is measured, and no -o file is written. The check needs memory for a few vectors the size of the result, and for daxpy, saxpy, dswap, dgemv, sgemv and dsymv a copy of the vector they write.

The check is on by default, as it is in STREAM and HPL: a number from a library that computes the wrong thing is worth less than none. -C turns it off, for a library already checked, or where that memory matters.

-t sets the number of threads through the library's own interface. If an environment variable that the library reads says otherwise, -t wins, and a warning says so:

$ OMP_NUM_THREADS=4 bmb_dgemm -t 1
OMP_NUM_THREADS is ignored! Set to 4 but option -t is set to 1.

Without -t, those variables apply as usual: the first one set gives the thread count. A value that is not a whole number of threads, from 1 to 2147483647 (0, -2, 2,1, or one with a blank in it), is ignored with a warning, and the next variable is read. Netlib reference BLAS is single-threaded, and -t has no effect with it. MKL may run a small problem on fewer threads than it is given (MKL_DYNAMIC, on by default); MKL_DYNAMIC=false makes it use them all.

Nothing here sets thread affinity, and operands are allocated and filled by the main thread, so on a machine with several NUMA nodes every page lands on that thread's node. Both matter as soon as more than one thread runs; see EXAMPLES.

GFLOP/s and GB/s say how fast, not why. Hardware counters — instructions, cycles, cache misses, the floating-point operations the CPU actually executed — say why, and the benchmarks leave reading them to the tools made for it, perf(1) and likwid-perfctr, run around them. Both count the whole process: allocating and filling the operands, the --verify check, the batch calibration and the warm-up, not only the timed calls.

Two runs that differ only in -i take that away. With -b 1 each sample is one call, preceded by the operand restore of the routines that need one, so the difference between the two counts, divided by the difference in -i, is what one call costs:

perf stat -e instructions,cycles -- bmb_dgemm -x 0 -b 1 -i 10 -m 1024
perf stat -e instructions,cycles -- bmb_dgemm -x 0 -b 1 -i 110 -m 1024

Here, one call is a hundredth of the difference. Keep to one size and one thread count per pair of runs, and to sizes where a call takes long enough that the clock read around it does not count.

The floating-point events have CPU-specific names, which perf list gives: for instance fp_arith_inst_retired.256b_packed_double (_single for the s and c routines) on recent Intel CPUs, or fp_ret_sse_avx_ops.all on AMD Zen. likwid-perfctr groups them per CPU, under names most CPUs share (FLOPS_DP, FLOPS_SP for the s and c routines, L3; likwid-perfctr -a lists the groups of the one at hand), and pins the threads to the cores it measures:

likwid-perfctr -C 0-3 -g FLOPS_DP bmb_dgemm -x 0 -b 1 -i 10 -t 4 -m 4096

Most virtual machines, containers and WSL expose no hardware counters, and perf then reports them as not supported. On a real machine, reading them may need /proc/sys/kernel/perf_event_paranoid at 2 or lower for perf, or the access daemon or root for likwid-perfctr.

Results go to standard output, one row per point, as they are measured; warnings and errors go to standard error. Every result starts with where it came from:

# blas-microbenchmark 1.5.0-dev
# backend: openblas (OpenBLAS 0.3.26 DYNAMIC_ARCH Haswell MAX_THREADS=64)
# verified: every point, against a reference, before it was timed
# layout: row-major
# cpu: Intel(R) Core(TM) Ultra 7 155U (14 logical CPUs, 1 NUMA node)
# caches: L1d 48K, L1i 64K, L2 2M, L3 12M
# frequency: governor performance, turbo off
# os: Linux 6.6.87 x86_64
# date: 2026-10-02T09:05:04Z
# routine: dgemm

The library's version string, where it exposes one, tells apart two builds of the same library. The caches show where a sweep leaves each level. The frequency line gives the CPU frequency governor (several, joined by /, when CPUs differ) and whether turbo is on, as Linux exposes them in /sys; virtual machines and containers often expose neither. When either lets the frequency change during the run, a governor other than performance or turbo on, a warning on standard error says so and gives the commands that fix it; the run goes on. A field that cannot be read on a machine is left out, never guessed. The hostname is deliberately not recorded. With -l, a # label: line follows the backend, and unless -C was given, a # verified: line comes next, then, for the routines with a matrix, the # layout: (-L).

With -o, the results are also saved to a file:

the same comment lines, then a header and one row per point. Readers must skip lines starting with #, e.g. pd.read_csv(file, comment="#") in pandas.
one object holding the same information as fields: version, backend, blas, label, verified, layout (row or col, for the routines with a matrix), date; machine with cpu, logical_cpus, numa_nodes, caches, governor, turbo (true or false) and os; routine and, naming its dimensions, dim1_label and dim2_label. Then a results array, one object per point: thread_count, dim1, dim2, time_s, gflops, gbytes_per_s, and with -s, mean_s, stddev_s, max_s, calls_per_sample and median_s. A field that does not apply is left out. This is what bmb_report(1) reads.

0 when every point was measured and every result written; 1 for an invalid option or size, or when standard output or the -o file could not be written — a table cut short by a full disk must not pass for a complete one; 2 when a result was wrong (see CHECKING RESULTS), in which case nothing is written to the -o file.

The thread-count variables the library reads, in the order it reads them:

OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS
BLIS_NUM_THREADS, OMP_NUM_THREADS
OMP_NUM_THREADS
MKL_NUM_THREADS, OMP_NUM_THREADS. The count is set for MKL's BLAS domain, which overrides MKL_DOMAIN_NUM_THREADS; a warning says so when it is set.
those of the library it loaded, as above; none for Netlib.

FLEXIBLAS picks the library a build against FlexiBLAS loads, by the name flexiblas list gives it:

FLEXIBLAS=OPENBLAS-OPENMP bmb_dgemm -m 512:2048 -o openblas.json
FLEXIBLAS=BLIS-OPENMP bmb_dgemm -m 512:2048 -o blis.json

The results name the library loaded: backend flexiblas/openblas-openmp, and FlexiBLAS's version with the library's name.

OMP_PROC_BIND and OMP_PLACES pin the threads of the OpenMP-threaded libraries.

NO_COLOR, when set and not empty, turns off the colours of warnings and errors, which are only used on a terminal anyway.

/usr/local/libexec/blas-microbenchmark/level1
/usr/local/libexec/blas-microbenchmark/level2
/usr/local/libexec/blas-microbenchmark/level3
The benchmarks, grouped by BLAS level rather than put on the PATH. To run them by name:

BMB=/usr/local/libexec/blas-microbenchmark
export PATH="$BMB/level1:$BMB/level2:$BMB/level3:$PATH"

Sweep ddot from 1K to 16M elements, with the statistics, saved as CSV:

bmb_ddot -v 1024:16777216 -s -o ddot.csv

Every point is checked before it is timed. For a library already checked, skip that:

bmb_dgemm -m 512:4096 -t 1:8 -C -o dgemm.json

The sizes that matter to you, and nothing else:

bmb_dgemm -m 1024,2048,4096

Square matrices, then every shape (the grid bmb_report(1) draws as a heatmap):

bmb_dgemm -m 512:2048                 # 512², 1024², 2048²
bmb_dgemm -m 512:2048 -M 512:2048     # all 9 combinations

Thread scaling, threads pinned — OMP_* for the OpenMP-threaded libraries, taskset for a pthread-built OpenBLAS:

OMP_PROC_BIND=close OMP_PLACES=cores bmb_dgemm -t 1:8 -o dgemm.json
taskset -c 0-7 bmb_dgemm -t 1:8 -o dgemm.json

On a machine with several NUMA nodes, spread the pages over all of them, or measure one node at a time — usually what you want to compare:

numactl --interleave=all bmb_dgemm -t 1:32
numactl --cpunodebind=0 --membind=0 bmb_dgemm -t 1:16

Two conditions on the same machine, told apart in the report by their labels (turbo is recorded, but a series is not split by it):

bmb_dgemm -m 1024:8192 -l "turbo on"  -o turbo-on.json
bmb_dgemm -m 1024:8192 -l "turbo off" -o turbo-off.json
bmb_report turbo-*.json > report.html

bmb_report(1), numactl(8), taskset(1)

https://github.com/MathieuCAISSA/blas-microbenchmark