HARDWARE COUNTERS

GFLOP/s and GB/s say how fast, not why. Hardware counters — instructions, cycles, cache misses, the floating-point operations the CPU actually executed — say why, and the benchmarks leave reading them to the tools made for it, perf(1) and likwid-perfctr, run around them. Both count the whole process: allocating and filling the operands, the --verify check, the batch calibration and the warm-up, not only the timed calls.

Two runs that differ only in -i take that away, as long as the warm-up does not depend on time: -w 0 turns off the warm-up by time, whose number of calls varies from run to run. With -b 1 each sample is one call, preceded by the operand restore of the routines that need one, so the difference between the two counts, divided by the difference in -i, is what one call costs:

perf stat -e instructions,cycles -- bmb_dgemm -x 0 -w 0 -b 1 -i 10 -m 1024
perf stat -e instructions,cycles -- bmb_dgemm -x 0 -w 0 -b 1 -i 110 -m 1024

Here, one call is a hundredth of the difference. Keep to one size and one thread count per pair of runs, and to sizes where a call takes long enough that the clock read around it does not count.

The floating-point events have CPU-specific names, which perf list gives: for instance fp_arith_inst_retired.256b_packed_double (_single for the s and c routines) on recent Intel CPUs, or fp_ret_sse_avx_ops.all on AMD Zen. likwid-perfctr groups them per CPU, under names most CPUs share (FLOPS_DP, FLOPS_SP for the s and c routines, L3; likwid-perfctr -a lists the groups of the one at hand), and pins the threads to the cores it measures:

likwid-perfctr -C 0-3 -g FLOPS_DP bmb_dgemm -x 0 -w 0 -b 1 -i 10 -t 4 -m 4096

Most virtual machines, containers and WSL expose no hardware counters, and perf then reports them as not supported. On a real machine, reading them may need /proc/sys/kernel/perf_event_paranoid at 2 or lower for perf, or the access daemon or root for likwid-perfctr.