The BLAS libraries
The eight libraries it builds against: how each is chosen, found and threaded, and what its results record.
The benchmarks call BLAS through its C interface, CBLAS, so the same code runs against any library. The library is chosen when building, with ./configure --with-blas-backend=…; each is built and checked in CI on every change.
| Library | --with-blas-backend | Spack | Thread variables, in the order read |
|---|---|---|---|
| OpenBLAS (default) | openblas | ^openblas | OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS |
| BLIS | blis | ^blis | BLIS_NUM_THREADS, OMP_NUM_THREADS |
| AMD AOCL-BLAS | aocl | ^amdblis | BLIS_NUM_THREADS, OMP_NUM_THREADS |
| AMD AOCL-BLAS | yes | yes (as gemmtr) | no |
| Intel MKL | mkl | ^intel-oneapi-mkl | MKL_NUM_THREADS, OMP_NUM_THREADS |
| NVIDIA NVPL (aarch64) | nvpl | ^nvpl-blas | OMP_NUM_THREADS |
| Arm ArmPL (aarch64) | armpl | ^armpl-gcc | OMP_NUM_THREADS |
| Netlib reference BLAS | netlib | ^netlib-lapack | none: single-threaded |
| FlexiBLAS | flexiblas | ^flexiblas | those of the library it loaded |
Whatever the library, -t sets the thread count through its own interface, and wins over these variables. Each result records the library, and its own version string where it has one (openblas (OpenBLAS 0.3.26 …), aocl (AOCL-BLAS 5.3.2 Build …)).
Where configure looks
For each library, in this order, the last found winning:
- where the distribution installs it (Debian and Ubuntu's
/usr/include/<triplet>/blis-openmp, Intel's/opt/intel/oneapi/mkl/latest, …); - the variables a module or Spack sets:
<NAME>_ROOT,_DIRor_HOMEfor the prefix,<NAME>_INCDIRand<NAME>_LIBDIRfor the directories, with<NAME>the library's (OPENBLAS,BLIS,AOCL,MKL,NVPL,ARMPL, …), andMKLROOT; --with-blas-incpath=DIRand--with-blas-libpath=DIR, which always win.
OpenBLAS
The default, and what most distributions ship. Its build options (threading, the CPU it was tuned for) are part of its version string, and so of each result.
BLIS
From libblis-openmp-dev on Debian and Ubuntu, which keep its cblas.h in a directory of its own that configure finds. Only its OpenMP or pthreads builds use more than one thread.
AMD AOCL-BLAS
AMD's fork of BLIS, for AMD EPYC. It comes from AMD's installer, a module (AOCL_ROOT) or Spack (^amdblis); no distribution packages it. The benchmarks link its threaded libblis-mt: its libblis is single-threaded, and would ignore -t. AOCL-BLAS may run a small problem on fewer threads than it is given (AOCL Dynamic); only a library built with --disable-aocl-dynamic does not.
Intel MKL
oneAPI MKL, from Intel's repository, a module (MKLROOT) or Spack. It is linked with its GNU OpenMP layer and 32-bit integers (-lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core), or 64-bit ones with --enable-ilp64 (below). -t sets the thread count of its BLAS domain, which overrides MKL_DOMAIN_NUM_THREADS (a warning says so). MKL may also use fewer threads on a small problem; MKL_DYNAMIC=false makes it use them all.
NVPL and ArmPL
NVIDIA's and Arm's libraries, for aarch64 servers (NVIDIA Grace, Neoverse). Both come from their vendor's apt repository or a module, and take their threads from OpenMP.
Netlib
The reference BLAS: unoptimised, single-threaded, the yardstick the others are measured against. It has no C interface of its own, so the benchmarks bring theirs, over its Fortran routines.
FlexiBLAS
Not a library but a switch: one build, and the library chosen when the benchmark starts, with FLEXIBLAS. Each result names the library FlexiBLAS loaded (flexiblas/openblas-openmp), so the report keeps them apart. Fedora packages it, with OpenBLAS, BLIS and Netlib to load; Debian and Ubuntu do not.
FLEXIBLAS=OPENBLAS-OPENMP bmb_dgemm -m 512:2048 -o openblas.json FLEXIBLAS=BLIS-OPENMP bmb_dgemm -m 512:2048 -o blis.json
GPU libraries (cuBLAS, rocBLAS, MAGMA, …) are planned for 2.0: see the project.
What each library adds
Three benchmarks measure calls libraries add to the BLAS: axpby, gemmt (one triangle of a GEMM) and sbgemm (a GEMM of bfloat16 inputs into single precision). configure looks for each in the library it builds against; a benchmark whose call it did not find is still installed, measures nothing, and says which call is missing (exit 77). What the CI finds:
| Library | axpby | gemmt | sbgemm |
|---|---|---|---|
| OpenBLAS | yes | from 0.3.28's header | when built with BUILD_BFLOAT16 |
| BLIS | yes | yes | no |
| AMD AOCL-BLAS | yes | yes (as gemmtr) | no |
| Intel MKL | yes | yes (as gemmtr) | yes (cblas_gemm_bf16bf16f32) |
| NVIDIA NVPL | yes | yes | no |
| ArmPL | yes | yes | yes |
| Netlib | no | no | no |
Ubuntu's OpenBLAS package is built without bfloat16, and its 0.3.26 does not declare gemmt. Some OpenBLAS versions compute some of these wrong; --verify stops on them, and the manual says which.
64-bit integers
A BLAS takes its dimensions as 32-bit integers, unless it was built for 64-bit ones (ILP64). configure --enable-ilp64 builds against OpenBLAS's ILP64 build (Debian's and Ubuntu's libopenblas64-*-dev, or one built with INTERFACE64=1) or MKL's ILP64 interface; a vector may then hold more than 2³¹ elements. Any other library is refused. --version and every result say which integers the library takes.