blas-microbenchmark1.5.0-dev

Development

How it is tested, and what CI runs. To work on it, start with AGENTS.md.

make check runs all of them. Each is a program or a script that exits 0 to pass, 77 to report SKIP when something it needs is missing (a browser, groff, mandoc), and anything else to fail. Each says what it checks in the words of its own source.

test_bmb_machine.c src/c/common
Checks the machine probe.
test_bmb_options.c src/c/common
Unit tests for the command-line layer.
test_bmb_verify.c src/c/common
Unit tests for the building blocks of --verify: the reference products on small matrices worked out by hand, real and complex, in double and single precision, the comparison and its message, the tolerances, the probe vector, and the corruption hooks the benchmark tests use to prove each check can fail.
test_netlib_cblas.c src/c/netlib
Checks the netlib CBLAS shim against naive reference implementations.
test_bmb_batch.sh src/c
Checks that -b cannot batch the routines whose operand is restored before every call (dtrmv, dtrsv, dtrmm, dtrsm, strsm, ztrsm): N calls in a row would run on an operand drifting towards infinity or denormals, and the timing would stop meaning anything (#48). They stay at one call per sample, which -s shows as calls_per_sample, with a warning; another routine keeps the batch it was given, without one. Tiny sizes only.
test_bmb_ignored_options.sh src/c
An option a benchmark has no use for is ignored with a warning naming it, never silently: --layout on a vector routine (#7), --vector-size on a matrix one, --matrix-dim1 on a vector one, --matrix-dim2 on a routine with a single dimension. The run still succeeds. Where the option does apply, there is no such warning. Tiny sizes only.
test_bmb_threads_env.sh src/c
Checks how a benchmark takes its thread count from the environment, for the variables of the backend it was built with, in that backend's order: without -t the first one set to a number of threads is used; with -t, -t wins and a variable that disagrees is reported; a value that is not a number of threads (0, -2, "3 ", 2,1, ...) is ignored with a warning, where it used to label a run with 4294967294 threads; another backend's variables are not read. The thread count is read back from the table and the JSON, where a wrong one would end up in the report. Under MKL, a count per domain (MKL_DOMAIN_NUM_THREADS) is overridden with a warning.
test_bmb_verify.sh src/c
Runs every benchmark with --verify (on by default), and proves each check can fail: with BMB_VERIFY_CORRUPT set, every benchmark's check sees its result altered (by a relative 1e-6 in double precision; doubled in single, see bmb_verify.h), and has to stop with exit status 2, say what is wrong, and write no results file.
test_bmb_write_errors.sh src/c
A run whose results cannot be written must fail, not pass with a short or missing file (doc/dev/benchmarks.md, "Write failures"): a -o file that cannot be opened, a -o file on a full disk, in JSON and in CSV, and stdout on a full disk. Each exits non-zero and says what was lost. /dev/full, a device on which every write fails, stands for the full disk; where there is none, those checks are left out with a note.
test_helper.sh src/c
Shared assertions for the per-routine smoke tests, sourced as: . "${srcdir:-.}/../test_helper.sh"
test_bmb_size_limits.sh src/c/level1
A size whose square wraps size_t must be refused by the parser, not turned into a small allocation the setup loop then overruns. 1518500250 is the smallest such value: 1518500250^2 * 8 comes back down to 291 MB.
test_bmb_output_formats.sh src/c/level3
The --output files are not covered by the per-routine smoke tests, which only look at stdout. dgemm is the routine used here because its "Matrix dim1 (M=K)" label is the one that exercises the CSV column-name rewriting hardest.
test_bmb_square_default.sh src/c/level3
Without -M, a two-dimension routine measures square matrices: dim2 follows dim1 point by point. It used to copy -m's range instead and measure the whole cross product — nine runs for -m 8:32, where --help promised three square ones.
test_bmb_<routine>.sh src/c/level1–3, 32 of them
One per benchmark: each runs its benchmark on tiny sizes, and the assertions of src/c/test_helper.sh check what it printed.

test_assemble.sh src/report
Checks assemble.awk, which builds bmb_report by inserting report.html into bmb_report.sh: that it copies the template byte for byte, and that it refuses to build in each case where the result would be broken.
test_bmb_report.sh src/report
Checks bmb_report: what it refuses, and that what it writes is one self-contained page carrying every result.
test_bmb_report_js.sh src/report
Runs the unit tests of the page's logic (test_report.js) in a headless browser.
test_bmb_report_render.sh src/report
Renders a report in a headless browser and checks that every chart drew.
test_report.js src/report
Unit tests for the page's logic: the functions report.html exposes as window.bmbReport, fed small hand-built results so that each decision of #1 is pinned by a check that names it.

test_actions_pinned.sh tests
Checks that every action the workflows use is pinned to a full commit SHA, with its version in a comment (actions/checkout@<40 hex> # v7.0.1). A tag can be moved to other code, by its owner or by whoever takes over the repository; a commit cannot. Dependabot updates the SHA and the comment together.
test_configure_paths.sh tests
--with-blas-incpath and --with-blas-libpath are the most specific way to name a BLAS, so they come first in the search: ahead of a module's variables (BLIS_LIBDIR, ...) and of the directories configure finds by itself in a distribution's install (/usr/lib/*/blis-openmp, ...). They used to be applied before both and come last, and a library named on the command line lost to the system's (#58).
test_doc_links.sh tests
Checks every relative link in the Markdown files in git: the file or directory it points to is in git too, and so is the heading its #anchor names, spelt the way GitHub spells anchors. The developer guide is split across AGENTS.md and doc/dev/, and a section moved or renamed in one file breaks links in the others without a word.
test_editorconfig.sh tests
Checks every file in git against .editorconfig, so that what it promises an editor is also true of the files already there: UTF-8, LF line ends, a final newline, no trailing whitespace, and indentation with spaces — a tab first in Makefile.am, where make needs it (spaces may follow, to align a continuation line).
test_release_files.sh tests
Checks that CHANGELOG.md and CITATION.cff agree with the version being built: a -dev version has an [Unreleased] section; a release has its own dated section, and is the version CITATION.cff cites, on the same date. Between releases, CITATION.cff cites the latest released version.

test_man_content.sh man
Checks what the man pages say against what the programs do: the routines and their dimensions, the sweeps and their limit, the JSON fields, the environment variables, the exit statuses, and the commands in the examples. A page can render perfectly and still describe last year's program; each check here runs the program and compares.
test_man_html.sh man
Checks the documentation site html.sh builds from the man pages: every page and the index are there, every link on the site leads somewhere — to a file and an anchor that exist, or to https — nothing is loaded from elsewhere, the pages link to each other, and each page holds what its man page does (every option of --help, the version).
test_man_install.sh man
Installs the man pages into a staging root and opens them the way a user would: man blas-microbenchmark, man bmb_report, and man bmb_<routine> for every benchmark, through its alias. Then uninstalls, and checks nothing is left behind.
test_man_options.sh man
Checks that the man pages document exactly the options --help lists, and give the same defaults. --help is the summary and the man page the reference; this is what notices when one moves without the other.
test_man_render.sh man
Checks that the man pages render cleanly: every placeholder substituted, a date, no groff warning, nothing mandoc's lint objects to, and no command, option or path that would come out as a typographic hyphen.

GitHub Actions runs four workflows: CI and CodeQL on every pull request and push to main (CodeQL weekly as well), the site on every push to main, the release on a version tag and, publishing nothing, on a pull request that changes it. Each job is described by its workflow file: the comment above it, where it runs, how it configures the build, and its steps in order.

.github/workflows/ci.yml

The jobs only read the repository; nothing they run gets a token that could write to it.

openblas ubuntu-latest

../configure --enable-werror

Install build dependencies → autoreconf → configure → make → Provenance → make check → Verified at real sizes → The page was checked in a browser → The files and their links were checked → Install layout → Upload test logs → make distcheck

blis ubuntu-latest

../configure --with-blas-backend=blis --enable-werror

Install build dependencies → autoreconf → configure → make → Provenance → make check → Verified at real sizes → Upload test logs

netlib ubuntu-latest

../configure --with-blas-backend=netlib --enable-werror

Install build dependencies → autoreconf → configure → make → Provenance → make check → Verified at real sizes → Upload test logs

browsers ubuntu-latest, once for each of firefox, microsoft-edge

The report page in the browsers the Chrome-based jobs above do not cover: Firefox, which the README tells users to open it with, and Edge. Firefox has no --dump-dom, so browser.sh drives it through geckodriver; Edge is driven like Chrome. The runner image ships Firefox, geckodriver and, normally, Edge.

../configure --enable-werror

Install build dependencies → Firefox and geckodriver → Edge → autoreconf → configure → make → Report tests in browser → The page was checked in browser → Upload test logs

man ubuntu-latest, in fedora:latest

The man pages, on another distribution than the other jobs: Fedora, with its own groff and man-db builds, and mandoc, whose lint is stricter than groff. BMB_MAN_STRICT turns a missing tool into a failure rather than a SKIP, so this job cannot pass without checking everything. No --enable-werror: Fedora's newer GCC is not what this job is testing.

OPENBLAS_INCDIR=/usr/include/openblas ../configure

Install build dependencies → autoreconf → configure → make → Tools → Man page tests, nothing skipped → Upload test logs

nvpl ubuntu-24.04-arm

NVPL and ArmPL both target aarch64 (NVIDIA Grace / Arm servers), so these two jobs run on GitHub's arm64-hosted runners (free for public repos) instead of ubuntu-latest.

../configure --with-blas-backend=nvpl --enable-werror

Install build dependencies → Install NVPL BLAS → autoreconf → configure → make → Provenance → make check → Verified at real sizes → Upload test logs

armpl ubuntu-24.04-arm

../configure --with-blas-backend=armpl --enable-werror

Install build dependencies → Install Arm Performance Libraries → autoreconf → configure → make → Provenance → make check → Verified at real sizes → Upload test logs

sanitizers ubuntu-latest

Built at -O1 with AddressSanitizer and UndefinedBehaviorSanitizer. This is not a duplicate of the openblas job: it catches what a plain build cannot see at all (out-of-bounds accesses, signed overflow), and the lower optimisation level makes GCC's diagnostics land differently -- the first -Werror failure this job ever produced was a format-truncation warning invisible at -O2.

../configure --enable-werror CFLAGS="-O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer" LDFLAGS="-fsanitize=address,undefined"

Install build dependencies → autoreconf → configure → make → Provenance → make check → Verified at real sizes → Upload test logs

coverage ubuntu-latest

Which lines of the C code make check runs, and which it never reaches: each test was seen to fail, but that says nothing of the paths no test takes. No threshold -- the report is there to be read. The per-file summary is on the run's page; the line-by-line HTML report and an lcov file are in the coverage artifact.

../configure --enable-werror CFLAGS="-O0 -g --coverage" LDFLAGS="--coverage"

Install build dependencies → autoreconf → configure → make → make check → Coverage report → Upload the coverage report → Upload test logs

cppcheck ubuntu-latest

cppcheck over the C code: what -Wall -Wextra does not see, on paths no test takes -- its first run found pointer arithmetic on a failed malloc in every verify(). Any finding fails the job; a wrong one is suppressed where it is, with the reason (.github/cppcheck.sh). CodeQL runs in its own workflow, codeql.yml.

../configure

Install build dependencies → autoreconf → configure → cppcheck

flexiblas ubuntu-latest, in fedora:latest

FlexiBLAS loads the BLAS library at run time (FLEXIBLAS=...), so one build measures several. Debian and Ubuntu do not package it; Fedora does, as its default BLAS, with OpenBLAS, BLIS and Netlib as backends. Everything runs once per backend from the same build, and the results of two of them must name the library each loaded: two series.

../configure --with-blas-backend=flexiblas --enable-werror

Install build dependencies → Backends → autoreconf → configure → make → Provenance → make check, once per backend → Verified at real sizes, once per backend → One build, two libraries → Upload test logs

mkl ubuntu-latest

Intel oneAPI MKL, the BLAS most x86 clusters measure against, from Intel's apt repository: its current release, as sites install it. The benchmarks link its GNU OpenMP threading layer and set its BLAS thread count (mkl_domain_set_num_threads); the runner's CPU, Intel or AMD, is whatever GitHub gives.

../configure --with-blas-backend=mkl --enable-werror

Install build dependencies → Install oneAPI MKL → autoreconf → configure → make → Provenance → make check → Verified at real sizes → Upload test logs

spack ubuntu-latest, once for each of openblas, blis, netlib-lapack, flexiblas, intel-oneapi-mkl

The Spack package (packaging/spack), as an HPC user installs it: built by Spack against each BLAS it can build on x86_64, from this commit's tarball and from the latest release's, then checked by its own tests (spack test run). The distribution's BLIS or Netlib is installed as well, so that linking it instead of Spack's would fail test_linked_library (#58). MKL is Intel's apt package, declared to Spack as an external: Spack's own would download Intel's installer, gigabytes, on every cache miss. Spack and its package recipes are pinned to commits, as the actions are; what Spack builds is cached, keyed on both and on the recipe.

(cd build && ../configure && make dist)

Install build dependencies → This commit's tarball → Spack → spack install and spack test

.github/workflows/codeql.yml

GitHub's static analysis of the C code, with its security and quality queries: data flow across functions, which cppcheck (in ci.yml) does not follow. Findings go to the repository's Security tab (code scanning), where each one is fixed or dismissed with a reason. Weekly as well, since the queries are updated without any change here.

analyze ubuntu-latest

../configure

Install build dependencies → Build → No alert left open

.github/workflows/pages.yml

The documentation site: the man pages as HTML (make html, in man/), published on GitHub Pages from main. It is published only once the man page tests pass on the same commit, so the site never shows a page they would have refused. One deployment at a time; a newer push replaces one still waiting.

build ubuntu-latest

../configure

Install build dependencies → Build → Man page tests → Build the site

deploy ubuntu-latest

.github/workflows/release.yml

Builds the distribution tarball and attaches it to the GitHub release for a pushed tag. GitHub's own auto-generated "Source code" archives are plain git exports with no pre-generated `configure`, so end users need this one. The tarball carries a signed attestation of where it was built: gh attestation verify checks it came from this workflow, in this repository. A pull request that changes this file runs it too, publishing nothing, so that a broken release is found before the tag rather than at it.

tarball ubuntu-latest

../configure

Install build dependencies → Build the distribution tarball → Verify the tarball builds with no Autotools → Checksums → Attest where the tarball was built → Verify the attestation → Upload the tarball to the release