Sites that check their systems with ReFrame get every benchmark as a ReFrame test. They are installed with the benchmarks, in <prefix>/share/blas-microbenchmark/reframe, and written for ReFrame 4.

• One test per routine, at one size per level, on all of a node's cores.
• Sanity is the benchmark's own check: it exits 0, and every point was verified against a reference before it was timed.
• The performance variables are the rates measured: gflops, and gbytes_per_s at levels 1 and 2.
• Tags: blas, and level1, level2 or level3.

Running them

Run from where they were installed, the tests run that install's benchmarks, with nothing to set:

R=/usr/local/share/blas-microbenchmark/reframe
reframe -C site-config.py -c $R/blas_microbenchmark.py -n 'bmb_run' -r

With Spack, R=$(spack location -i blas-microbenchmark)/share/blas-microbenchmark/reframe. -n 'bmb_run%routine=dgemm$' runs one routine (without $, dgemmt too), -t level3 one level.

Sizes, threads and the rest

-S sets the tests' variables:

Variable	Default
level1_size	10000000	the vector length at level 1
level2_size, level3_size	5000, 2000	M = N = K at levels 2 and 3
num_threads	the partition's cores	-t
bmb_options	none	more options, such as -S bmb_options=-i,50
install_prefix	this file's install, or PATH	where the benchmarks were installed
reference_file	none	your references, below

Your references

Every site's numbers are its own, so none ship with the tests. Give them in a JSON file, by system (or system:partition, or *), routine and performance variable: the value, then the lower and upper thresholds as fractions of it, null for none.

{
  "mycluster:cpu": {
    "dgemm": {"gflops": [2900, -0.05, null]},
    "daxpy": {"gflops": [10, -0.1, null], "gbytes_per_s": [120, -0.1, null]}
  }
}

reframe -c $R/blas_microbenchmark.py -n 'bmb_run' -S reference_file=references.json -r

A test below its reference fails in ReFrame's performance stage; one without a reference reports its rates, and passes.

Building a release in each environment

bmb_build_run builds a release first, once per programming environment, then runs the same tests on it: the release is downloaded and checked against its SHA256SUMS, and -S bmb_tarball.blas_backend=… picks the library.

A report of the runs

Each test keeps the benchmark's result.json in ReFrame's output directory:

bmb_report output/*/*/*/*/result.json > report.html

Everything else, including bmb_build_run's variables: the ReFrame README.