blas-microbenchmark1.11.0-dev

With an AI agent

Let an AI agent run the benchmarks: bmb_mcp, an MCP server over the suite, and skills that say how to measure as this project does.

An AI agent can measure, compare and read BLAS results with blas-microbenchmark, through two pieces installed with it: bmb_mcp, a server that gives the agent tools to run the benchmarks within limits, and skills, which tell it how to measure as this project does.

bmb_mcp speaks the Model Context Protocol on its standard input and output: the agent's client starts it. Its tools list the routines, describe the machine, run a benchmark or the suite, read a result, and make the report. Their arguments are fields (a routine, its sizes, its threads, its arguments), never a command line.

Every run is checked before it starts: its sizes against the memory, its threads against the CPUs, every path inside the results directory; it is stopped at a time cap, and every result is verified against a reference, as the benchmarks do by default. The limits are options of the server:

bmb_mcp --results-dir bmb-results --time-cap 600

A client that reads its servers from JSON starts it with an entry such as:

{"mcpServers": {"blas-microbenchmark": {"command": "bmb_mcp", "args": ["--results-dir", "bmb-results"]}}}

In Claude Code, the repository is a plugin marketplace: its plugin brings the server and the skills below, once blas-microbenchmark is installed and bmb_mcp on PATH:

/plugin install blas-microbenchmark --marketplace MathieuCAISSA/blas-microbenchmark

From the Docker image, a client runs docker run --rm -i -v "$PWD:/results" ghcr.io/mathieucaissa/blas-microbenchmark bmb_mcp as its command.

run_suite with several libraries runs the benchmarks of the installs it is given, by path: have the client ask before it calls it. Everything else is in its manual.

Three skills, in the Agent Skills format (a SKILL.md each), installed in share/blas-microbenchmark/skills/ under the install prefix:

  • blas-benchmark: measure one routine right: the machine and its frequency first, sizes across the caches, threads within the CPUs, the checking kept on, every result in a file, and the IQR read before concluding.
  • blas-compare: compare libraries or versions: one build a library, the same suite for each with bmb_suite --blas, repeated runs, and a difference counted only beyond their noise.
  • blas-results: read what comes out: time, median and IQR, GFLOP/s against GB/s by level, what a result records, the peak and the roofline, and what makes two results comparable.

Each uses the server's tools when the agent has them, and the commands otherwise, with the same steps. Point a client that reads skills at that directory, or copy them where it looks.