# Kernel Multi-Bench

**Repository Path**: OpenCloudOS/kernel-multi-bench

## Basic Information

- **Project Name**: Kernel Multi-Bench
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: GPL-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-05-21
- **Last Updated**: 2026-06-18

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Kernel Multi-Bench (KMB)

Benchmark multiple Linux kernels across reboots on real hardware or a dedicated
VM. Uses **systemd** + **grubby** to switch kernels, reboot, and run tests
automatically. No nested virtualization, no sidecar controllers, and no need
for remote management hardware to control the machine.

Runs every script on every kernel, rebooting between each test. Every reboot
gives a clean, fresh start and acts as a hard separator between tests,
producing stable averages free of cross-test interference and thermal drift.
Can also be easily configured to run in round-robin fashion for repeated
sampling.

## Quick start

```bash
# With an explicit kernel inventory:
kmultibench run inventories/example testsuites/example-suite my-run

# Auto-discover kernels from /boot/ (opens $EDITOR to pick):
kmultibench run testsuites/example-suite my-run

# Build kernels from git commits, install, then benchmark:
kmultibench build-run [options] <kernel-src> <commit>... -- <test-suite> <run-name>
```

> **WARNING:** these commands reboot your machine repeatedly until every script
> has run on every kernel. Do not run on production machines unless you fully
> understand what each test case does.

Results land under `/var/lib/kmultibench/results/<run-name>/`.

## Installation

```bash
make install      # copy systemd unit (optional, bench-start does it at runtime)
make uninstall    # remove it
```

Add `kmultibench` and `bin/` to your `$PATH`, or invoke by full path.

## Prerequisites

- Linux with **systemd**, **grubby**, and root access
- Multiple kernels installed under `/boot/`
- Some workloads need: `redis-server`, `fio`, `sysbench`, `mariadb`, `cpupower`, `numactl`, etc.

## Repository layout

```text
kernel-multi-bench/
├── kmultibench          # Main CLI: run / build-run / next (`kmb` symlinks here)
├── libexec/             # Core framework: lib.sh, lib-bench.sh, bench-* stages
├── bin/                 # Utility scripts (kmb-*): ZRAM/BRD/PMEM setup, THP
│                        #   control, log parsing, stat, adhoc tests, etc.
├── inventories/         # Kernel inventories: which kernels to test
├── testsuites/          # Test suites: what to benchmark (one dir per suite)
├── assets/              # Pre-built assets (e.g. linux.tar.xz for kbuild)
├── furniture/           # Shared build recipes and dependencies
├── misc/                # Miscellaneous files (systemd unit, etc.)
```

## How it works

KMB runs a fully automated, reboot-driven pipeline over *kernels × scripts*:
every script runs on kernel 1, rebooting between each, then switches to kernel 2,
and so on until all combinations are done.

```text
kmultibench run [inventory] <suite-dir> <run-name>
   │
   ├─ bench-run        Parse args, auto-discover kernels if no inventory
   ├─ bench-preflight  Validate inventory, run .prepare.sh hook
   ├─ bench-start      Save state, enable systemd service, reboot to kernel 1
   │
   └─ [after each boot, systemd calls kmultibench next]
        bench-next     → restore state, run next script, tee to log, advance, reboot
                       → when no scripts remain, switch to next kernel & reboot
                       → when no kernels remain, exit (bench-end cleans up)
```

### Design

- **One reboot per script**:  each reboot gives a clean, fresh start, acting as a
  hard separator that eliminates cross-test interference. Running each script
  multiple times in round-robin fashion yields stable, trustworthy averages
  while letting the machine cool down between runs, avoiding thermal drift and
  other uncontrollable performance noise.
- **Logging**: all output is `tee`-ed to per-script logs with diagnostics.
- **Reboot-driven**: since every test reboot cycles the machine, scripts can
  freely update the kernel cmdline, toggle kernel features via sysfs, or load
  modules that require a fresh boot, no special orchestration needed.
- **Directly runnable**: each script is a self-contained executable that can be
  run individually. No framework harness required, making development and
  debugging straightforward.

### State persistence

Before every reboot, all runtime variables are serialized to
`$KMB_ENV` (`/var/lib/kmultibench/kmultibench.running.env`) and re-sourced on
the next boot. This makes the pipeline fully reboot-resilient.

### systemd integration

`bench-start` writes a kickstarter script at `/kmultibench.runner` and enables
the systemd unit `kernel-multibench-runner.service`. The unit has
`ConditionPathExists=/kmultibench.runner`, so it only runs while a benchmark is
in progress. `Type=idle` ensures it waits until all other boot jobs finish.
When the run completes (or fails), `bench-end` / `bench-cleanup` remove the
kickstarter and disable the service, ending the reboot loop.

A failing script (non-zero exit) is logged as a warning but the run continues.

> **Limitation:** kernel panics are not handled. If a kernel panics the machine
> may hang and require manual intervention. This could be improved with kdump
> to automatically capture the crash and reboot into the next kernel.

## Writing benchmark scripts

Scripts are plain executables run top-to-bottom. Source `lib-bench.sh` for
helpers like `stabilize_performance`, cgroup isolation, THP control, reboot
orchestration, and post-run diagnostics (`dump_sys_info`). The `bin/` directory
ships several standalone tools, most notably `kmb-util-thp-ctl.sh` for
fine-grained mTHP control, `kmb-util-setup-zram.sh` for ZRAM swap devices, and
`kmb-grep-time.sh` / `kmb-shell-avgs.sh` for parsing results.

```sh
#!/bin/sh
# Suites source the framework via the testsuites/lib-bench.sh symlink.
. "$(dirname "$(realpath "$0")")/../lib-bench.sh"

stabilize_performance                     # governor=performance, disable turbo/ASLR/...
disable_zswap
kmb-util-thp-ctl.sh --all never --anon 64
switch_into_benchmark_cgroup
set_benchmark_cgroup_attr memory.max 4G

/usr/bin/time redis-benchmark -r 3000000 -n 3000000 -d 1024 -c 6 -P 32 -t get

dump_sys_info                            # post-run diagnostics
```

### Conventions

- `chmod +x` every script. Preflight aborts if any aren't executable.
- Use numeric prefixes (`10-`, `20-`, ..., `99-`) to control execution order.
- Recommended: start with `stabilize_performance`, end with `dump_sys_info`.
- Script's stdout is redirected to log file, results should be extractable
  from the logs.
- The `furniture/` directory holds shared prepare infrastructure:
  Docker images, benchmark binaries, dataset populator scripts. Each
  furniture `make` target added to `PREPARE_TARGETS` must be re-entry safe
  (idempotent) and fast on repeat runs: cached builds, already-pulled
  images, and already-installed packages should all be detected as no-ops.
- Optionally add a `.prepare.sh` hook in the suite directory. It runs once
  during preflight after validation, before any benchmark starts. Use it to
  call `make` to build suite-specific dependencies.

## Kernel inventories

Inventories live in `inventories/<name>/`. A directory may hold
multiple inventory files, processed in sorted order; all kernels from all
files are tested sequentially.

### Static list (`.list`), one kernel path per line, `#` for comments:

```text
/boot/vmlinuz-6.6.6-orig+
/boot/vmlinuz-6.6.6-patched+
```

### Dynamic generator (`.sh`), implements `next()`, which sets
`$KMB_RUN_KERNEL_NEXT`. Return `0` to continue, `255` when done. Useful for
kernel bisection:

```bash
next() {
    # $1 is the previously tested kernel path
    return 255
}
```

### Auto-discovery

Omit the inventory argument and the framework lists all
`/boot/vmlinuz-*`, opens `$EDITOR` to trim, then runs.

## Results & analysis

```text
/var/lib/kmultibench/results/<run-name>/
├── 1-6.6.6-orig+/
│   ├── 00-begin.sh.log
│   ├── 10-blk-zram-4k-j12.sh.log
│   └── ...
├── 2-6.6.6-patched+/
│   └── ...
```

Each `.log` holds the full script output with `set -x` tracing. Parse them
with the `bin/` tools. All grep tools accept multiple files (tagged like `grep`):

```bash
# Single file
kmb-grep-time.sh --real results/my-run/1-kernelA/10-blk-zram.sh.log

# Multiple files — auto-tagged with filename
kmb-grep-time.sh --real results/my-run/*/10-blk-zram.sh.log

kmb-grep-usemem.sh --ts results/my-run/*/20-vm-scala.sh.log
echo "1.2 3.4 2.1" | kmb-shell-avgs.sh
```