HOW A NUMBER IS MEASURED

Reading the clock costs tens of nanoseconds, which is the whole duration of a small level 1 call. So each sample times a batch of calls and divides by the batch size. The batch size is chosen per point, so that a batch lasts long enough for the clock not to matter; calls that already take longer get a batch of 1, so nothing changes for level 3 or for large sizes. calls_per_sample gives the batch size chosen.

The reported time is the fastest sample, not the mean. A sample is already an average over a whole batch, so the fastest is not a lucky call: it is the batch that ran with the least interference. One scheduling hiccup inflates a sample a great deal, and would drag a mean with it. How steady the samples were goes with it, in every output: their median, and their quartiles, interpolated between the samples either side as R and NumPy do; the table gives the interquartile range as a percentage of the median, IQR [%]. -s adds the mean, the standard deviation and the slowest sample.

Ten samples say little about noise, and a small point measured in a millisecond may run before the CPU has left its idle clock. -w warms up for a time rather than a number of calls, once a thread count. -T takes as many samples as fill a time, and -e as many as it takes for them to agree:

bmb_ddot -v 1024:1048576 -T 0.1 -e 2

The spread is the interquartile range, not the coefficient of variation: one preempted sample in thousands keeps a standard deviation from ever settling, and a quartile does not see it. The samples taken are in the results, and in the table when -T or -e is given. The fastest of more samples is faster still: compare runs taken with the same settings, which the results record.

Two rates are derived from that time:

GFLOP/s

the conventional operation count of the routine (2×M×N×K for dgemm, K×N×(N+1) for dsyrk, ...) divided by the time. Not shown for the copy and swap routines, which do no arithmetic, nor for isamax, idamax, icamax and izamax, which search. Complex routines count real operations: a complex multiply-add is four real multiplications and four additions, so each counts four times its real counterpart's operations (cgemm and zgemm 8×M×N×K where dgemm counts 2×M×N×K; zherk four times dsyrk's), and their GFLOP/s compare with the real routine's. Those that are not multiply-adds count each real part: scasum and dzasum twice dasum's N, scnrm2 and dznrm2 twice dnrm2's 2×N, and cscal and zscal 6×N, a complex product.

GB/s

the bytes the routine must move — each operand read once, each result written once, counted the way STREAM counts them, 4 bytes an element in single precision, 8 in double and in single complex, 16 in double complex — divided by the time. Levels 1 and 2 only: level 3 routines reuse their operands from cache, so a rate built from that traffic would not measure bandwidth. The comparison with STREAM holds for operands out of the caches: past the last-level cache, or at any size with -k cold; a smaller vector measured hot is read from a cache, at that cache's bandwidth.

-k cold sets up as many copies of the operands as it takes for the others to reach twice the last-level cache, from what the operands of one call take, and gives each call the next one; a copy whose operand the routine overwrites is restored right after its call, never before the next, which would bring it back into the cache. Once the operands alone reach that far, there is one copy, each call pushing its own operands out. The copies take about twice the last-level cache in memory, on top of one set of operands. The copies are at most 16384: below the size where that is not enough, the points are partly warm, and a warning says from which size. The size of the last-level cache comes from /sys; where it cannot be read, -k cold is refused. A cold call on a small vector waits for memory more than it streams from it, so its GB/s are below STREAM's.