On-device model leaderboard

tok/s, tok/J, and heat for models that fit from 4 GB to 8 GB to 16 GB and more, on real hardware.
Jetson today. Pi, phones, and Mac minis are still in the oven.

0 families 0 configs 0 live

Families

Detailed reports

Full analysis and reports are available on the blogs:

#	Model	Quant	Backend	Mode	tok/s	tok/J	TTFT	ITL	E2E	prefill	W	TJ°C	ctx	gen

Latency / throughput cells show aiperf mean ± std over ~20 requests.
Sort and chart still use aiperf p50 (median).
Click i on a column for the statistic used there (and for tok/J, power, thermals that are not aiperf).

How it’s done

Same pipeline on every board: load the model, lock the power mode, sweep fixed prompt × gen cells with aiperf, then align power/thermal logs. Details live in each device README; this page is the short map.

Load generator: aiperf 0.11.0 (44addf0), concurrency 1, synthetic prompts at fixed token counts (usually 20 reqs per cell). From the clone root: uv sync (Mac Mini also uv sync --extra mac). Do not pip install aiperf from PyPI.
Table latency/throughput cells show aiperf mean ± std over the ~20 requests; sort and chart still use aiperf p50. Power / thermals are tegrastats; tok/J is decode-phase energy — see column i tips.
Headline metric: output tok/J - formula differs by hardware (decode-phase energy on Jetson/Mac posts; throughput ÷ average power on Pi). Do not cross-compare boards on tok/J alone.
Artifacts under artifacts/<backend>/<model>/gen<G>/ctx<P>/.

General flow

01Prepare Build or install the backend(s). Pull GGUFs (and MLX packs on Mac) via hf. Activate the device’s aiperf venv.
02Stabilize Lock power mode / clocks where the board supports it. Start power sampling. Relaunch inside tmux for multi-hour runs.
03Serve One server process per model (llama.cpp, Ollama, or MLX-LM). Smoke a short request before the sweep.
04Sweep aiperf drives prompt × gen cells (typically 20 reqs each, concurrency 1). Failed cells are logged; --resume skips done work.
05Record Align power/thermal logs to each cell. Write profile_export_aiperf.json under artifacts/<backend>/<model>/gen<G>/ctx<P>/, then charts/report.

By machine

Open a board for the concrete stack, sweep, and how to run it from that folder’s README / .sh.

Jetson Orin Nano Super

Folders: benchmark-jetson-nano-orin-super/single-node/non-reasoning-models/ (bench-non-reasoning.sh), bonsai-models/. Hardware: Orin Nano Super 8 GB, JetPack R36.x / CUDA 12.x, Ampere SM_87.

Stack

llama-server with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=87, and/or Ollama on the same local GGUFs.
aiperf 0.11.0 (44addf0) from the clone .venv via uv sync (fallback ~/venv / ~/aiperf-env).
sudo nvpmodel / jetson_clocks for modes 7W · 15W · 25W · MAXN; tegrastats for VDD_CPU_GPU_CV and junction temps.

Sweep

Non-reasoning: prompt ∈ {128, 512, 1024, 2048} × gen ∈ {64, 128, 256}, 20 requests per cell. Bonsai: prompt ∈ {256..2048} × gen ∈ {128, 256, 512}. Tiny instruct models (~135M-1.2B) plus Bonsai / Ternary-Bonsai sizes.

Tok/J

Published Jetson posts use decode-phase energy: OSL ÷ (decode_power_W × p50_decode_s). Same definition in the Bonsai H.3 appendix. Scripts also log whole-window rail averages from tegrastats.

Run

Main command (25W sweet spot):

bash bench-non-reasoning.sh --power-mode 1

Useful flags:

--backend llamacpp|ollama|both: which stack to sweep
--only <substring>: filter models by name
--reqs N: requests per cell (default 20)
--resume <dir>: skip cells that already finished
--dry-run: print the plan without starting servers

Auto-relaunches into tmux session non-reasoning-bench.

Full command and setup details: Jetson non-reasoning README.

Mac Mini M4

Folder: benchmark-mac-mini-m4/non-reasoning-models/ (benchmark_non_reasoning.sh). Hardware: Mac Mini M4, 16 GB unified memory, macOS Sequoia+.

Stack

llama.cpp Metal (-DGGML_METAL=ON) serving Q4_K_M GGUFs, or MLX-LM 4-bit via mlx_lm.server (--backend mlxlm).
Clone-root .venv via uv sync --extra mac (aiperf 0.11.0 + mlx-lm). PyPI aiperf is a yanked stub. Host setup tested on MacBook Air M1 (2020) and Mac Mini M4 (2025) 16 GB.
Power via sudo powermetrics (CPU + GPU + ANE); skip with --no-power. Peak RSS sampled per combo.

Sweep

prompt ∈ {256, 512, 1024, 2048, 4096} × gen ∈ {256, 512, 1024} (15 cells), context 6144, ~10 dense instruct models (Granite / Nemotron / Qwen / Gemma 3 class). Same aiperf methodology for both backends so rows stay comparable.

Tok/J

Decode-only: OSL_p50 / (decode_power_W × p50_decode_s), using per-request phase timestamps from the aiperf export to classify powermetrics samples.

Run

Main command (bash 4+):

bash benchmark_non_reasoning.sh

Useful flags:

--backend llamacpp|mlxlm: llama.cpp Metal GGUF or MLX-LM 4-bit
--stream: download, bench, then delete one model (saves disk)
--only <substring>: filter models by name
--resume <dir>: skip cells that already finished
--no-power: skip powermetrics (no sudo)

Auto-relaunches into tmux. Unlike Jetson/Pi, a failed aiperf combo aborts the run; fix and --resume.

Gated HF models (e.g. Gemma 3) need hf auth login and license acceptance first.

Full command and setup details: Mac Mini README.

Raspberry Pi 5

Folder: benchmark-raspberrypi5/ (benchmark_all_cpu.sh). CPU-only tiny LLM sweep with llama.cpp and/or Ollama.

Stack

llama.cpp built with BLIS (-DGGML_BLAS_VENDOR=FLAME -DGGML_NATIVE=ON). Thread env at server launch: BLIS/OpenBLAS/OMP = 1 thread, LLAMA_ARG_THREADS=4.
Ollama pinned to v0.24.0 (later 0.30.x rejects several benchmark GGUFs). Script sets OLLAMA_NUM_PARALLEL=1, flash attention, single loaded model.
aiperf 0.11.0 (44addf0) from the clone .venv via uv sync (fallback ~/venv / ~/aiperf-env).

Stabilize clocks

Manual (not done by the script): lock 2400 MHz via force_turbo=1 / arm_freq=2400 in config.txt and performance governor. Check vcgencmd get_throttled / measure_temp before and after; soft throttle ~80 °C. Details: clock locking.

Sweep

Same shape as Jetson non-reasoning: prompt ∈ {128, 512, 1024, 2048} × gen ∈ {64, 128, 256}, 20 requests, concurrency 1. Backends: llamacpp|ollama|both.

Tok/J / power

PMIC via vcgencmd pmic_read_adc every 500 ms (sum of measurable rails; undercounts wall power ~10-15%). Per-combo power.log. Report tok/J is throughput ÷ average power for the cell window. See power measurement.

Run

Main command:

bash benchmark_all_cpu.sh

Useful flags:

--backend llamacpp|ollama|both: which stack to sweep
--only <substring>: filter models by name
--reqs N: requests per cell (default 20)
--resume <dir>: skip cells that already finished
--dry-run: print the plan without starting servers

Auto-relaunches into tmux. Artifacts under artifacts/blog-all-YYYYMMDD-HHMM/.

Full command and setup details: Raspberry Pi 5 README.

Citation

If you use smolperfleaderboard (the site, the harness, or the results), please credit this work and cite it as:

@misc{singh2026smolperfleaderboard,
      title={smolperfleaderboard: On-Device LLM Leaderboard},
      author={Yuvraj Singh},
      year={2026},
      howpublished={\url{https://github.com/YuvrajSingh-mist/smolperfleaderboard}},
      doi={10.5281/zenodo.23196838},
}

License: Apache-2.0 for the harness, CC BY 4.0 for results and artifacts · attribution required when you use or republish these results

On-device model leaderboard

Coming soon

Detailed reports

Problem

Vision

What we do

How it’s done

General flow

By machine

Stack

Sweep

Tok/J

Run

Stack

Sweep

Tok/J

Run

Stack

Stabilize clocks

Sweep

Tok/J / power

Run

Citation