On a cluster, the questions are which library each partition should use, and whether a node does what it should. The benchmarks run as any program does: from a module or Spack, in a batch job, in a container if that is how the site works.

Installing it

• From a module's library. configure finds a BLAS loaded with module load through the variables the module sets (OPENBLAS_ROOT, MKLROOT, AOCL_ROOT, ARMPL_LIBDIR, …): load it, then build as from a release (Install).
• With Spack, which builds the library too, or uses the site's:

  spack install blas-microbenchmark ^intel-oneapi-mkl
  spack load blas-microbenchmark

• With Apptainer, from the published image, without building anything (Install). It is built for x86_64: on aarch64 nodes, build from a release.

In a Slurm job

Ask for one task with as many CPUs as threads, and give the same count to -t:

#!/bin/sh
#SBATCH --nodes=1 --ntasks=1 --cpus-per-task=32 --exclusive
export OMP_PROC_BIND=close OMP_PLACES=cores
bmb_dgemm -m 1024:16384 -t 1,2,4,8,16,32 -l "$SLURM_JOB_PARTITION" -o "dgemm-$SLURM_JOB_PARTITION.json"

Slurm restricts the job to its CPUs (its affinity, its cgroup). If -t asks for more threads than those, the benchmark warns: the threads would take turns, and the timings would measure that. --exclusive keeps other jobs off the node, which would otherwise be measured too.

Pinning threads and memory

Where threads run, and which memory they use, change the numbers as soon as there is more than one:

• Threads. OMP_PROC_BIND=close OMP_PLACES=cores pins the threads of the OpenMP-threaded libraries (BLIS, AOCL-BLAS, MKL, NVPL, ArmPL, OpenBLAS built with OpenMP); taskset -c 0-7 a pthreads-built OpenBLAS.
• Memory. The operands are allocated and filled by one thread, so on a machine with several NUMA nodes every page lands on that thread's node. Measure one node at a time, which is usually what you want to compare, or spread the pages over all of them:

  numactl --cpunodebind=0 --membind=0 bmb_dgemm -t 1:16
  numactl --interleave=all bmb_dgemm -t 1:32

The result records the CPU, its NUMA nodes and caches, the CPUs the process could run on, and the placement variables set (OMP_PROC_BIND, OMP_PLACES, GOMP_CPU_AFFINITY, KMP_AFFINITY); not numactl's memory policy, so put that in a label (-l). More in Threads.

One job a core

Many HPC codes run one process a core, MPI ranks or the jobs of a task farm, rather than one call split across threads. --star measures that: each count of -t runs as that many single-thread instances, each pinned to a core of its own, all measured at once. The rates are their sum, against as many cores' peak; a level 1 routine shows how the memory bandwidth is shared, a level 3 one what the cores reach together.

bmb_ddot --star -t 1,2,4,8 -v 10000000
bmb_dgemm --star -m 2048

The CPU frequency

Each result records the frequency governor and turbo, where the node exposes them, and the benchmark warns when they let the frequency move during the run. On a node you can configure, sudo cpupower frequency-set -g performance and turbo off give steadier numbers; otherwise the report says under what conditions each series ran.

Collecting the results

A file per partition, per library or per node, with a label where the file name is not enough, and one report of all of them:

bmb_report results/*.json > report.html

The report keeps apart what differs (library, version, CPU, label), and merges what does not: a sweep extended in a second job joins the first. See Compare libraries.

In the site's test suite

To check each node after maintenance, or each library after an update, against the site's own references: ReFrame tests.