CHECKING RESULTS

A benchmark times a call whatever it computes: a library that gets the result wrong — a broken build, a bug at one size or thread count — can look fast. So each point is first checked, unless -C is given: one call on the operands about to be timed, at that size and thread count, compared with a reference computed independently.

Level 1 and 2 results are recomputed directly, which costs about one more call. Level 3 results are checked by random projection: for a random vector r, C*r is compared with A*(B*r), and for trsm, in every precision, the residual op(A)*(X*r) (X*(op(A)*r) with --side R) with B*r. That takes as long as a level 2 call, where recomputing C would take as long as the call itself, and a wrong C gives a wrong C*r for almost every r. Routines that update one triangle are also checked to leave the other as it was; gemmt, to leave A and B as they were, which OpenBLAS before 0.3.30 did not, conjugating one in place.

The tolerance is a bound on rounding error, proportional to the number of terms in each element and to their magnitudes: a correct library is well inside it, a wrong one far outside. In single precision the reference is still computed in double, and the bound half a billion times wider: it catches a library that computes the wrong thing, but not a small error in a long sum (at a million elements, sdot is only checked to within the size of its terms). A wrong result is reported with the routine, the size, the thread count and the element that differs; nothing more is measured, and no -o file is written. The check needs memory for a few vectors the size of the result, and for the routines that write a vector they also read (axpy, swap, rot, gemv, gbmv, symv, spmv, hemv) a copy of it.

The check is on by default, as it is in STREAM and HPL: a number from a library that computes the wrong thing is worth less than none. -C turns it off, for a library already checked, or where that memory matters.