Threads

From the manual of the benchmarks, man blas-microbenchmark

-t sets the number of threads through the library's own interface. If an environment variable that the library reads says otherwise, -t wins, and a warning says so:

$ OMP_NUM_THREADS=4 bmb_dgemm -t 1
OMP_NUM_THREADS is ignored! Set to 4 but option -t is set to 1.

Without -t, those variables apply as usual: the first one set gives the thread count. A value that is not a whole number of threads, from 1 to 2147483647 (0, -2, 2,1, or one with a blank in it), is ignored with a warning, and the next variable is read. Netlib reference BLAS is single-threaded, and -t has no effect with it. MKL may run a small problem on fewer threads than it is given (MKL_DYNAMIC, on by default); MKL_DYNAMIC=false makes it use them all. AOCL-BLAS does the same (AOCL Dynamic), unless the library was built with --disable-aocl-dynamic.

Nothing here sets thread affinity, and operands are allocated and filled by the main thread, so on a machine with several NUMA nodes every page lands on that thread's node. Both matter as soon as more than one thread runs; see EXAMPLES.

Threads beyond the CPUs the process may run on take turns, and the timings measure that. A warning says so when the largest thread count is more than those CPUs: the ones its affinity allows (taskset, a Slurm job, a cpuset), capped by the CPU quota of its cgroup (cgroup v2 cpu.max, which docker run --cpus sets).

Star mode

A run at one thread measures a core with the machine's caches and memory to itself; a run at several, how the library splits one call. Neither is what a machine running an independent job on each core gets, as MPI ranks or the jobs of a task farm run: HPC Challenge's "star" mode. --star measures that: each count N of -t runs N copies of the benchmark, each a process of its own, its library given one thread whatever the environment says, pinned to a core of its own: the first CPU of each core this process may run on (its affinity), in order, so that two never share a core's threads. Each instance makes its own operands, on its own core, and checks them under --verify.

The instances start each sample together and a sample lasts until the slowest is done: its time is the slowest instance's, and the GFLOP/s and GB/s are the N instances' work over that time. They agree on everything else: the batch, the largest any chose; the samples, as -T, -e and -E judge the slowest's times; a point one could not set up is skipped by all; a wrong result in one ends the run, told by the first instance that found it, which names its CPU. On a machine whose cores differ (performance and efficiency cores), the slowest core sets the time.

$ bmb_ddot --star -t 1,2,4,8 -v 100000:10000000

shows how the memory bandwidth is shared as instances are added; a level 3 routine, the rate the cores reach together, against N cores' peak (--peak-gflops). More instances than cores is refused. The results say how they were measured: a # star: line with the CPUs, and star in the JSON's run block; bmb_report draws them as a series of their own.