Run a benchmark, step by step
The runner is a local command-line tool — your audio and models never leave your machine. Producing a result never needs network access; only the last step, submitting it, does.
Prefer not to memorize flags? After installing (step 1), just type goesb with no arguments — an arrow-key menu walks you through picking a profile, a pack, and running or submitting a benchmark, covering steps 2–4 below.
1. Install the runner
Four ways to get it — pick whichever fits your platform. Not sure? Use the quick install below; it covers most machines people actually run this on.
Quick install (recommended) — no Python needed
One command, no dependencies. Covers 64-bit Intel/AMD and ARM64 — including Raspberry Pi (64-bit OS) and rk3588-based boards, which show up as ARM64 like any other Linux machine.
macOS / Linux
curl -fsSL https://raw.githubusercontent.com/taktx-io/GOESB/main/scripts/install.sh | shWindows (PowerShell)
irm https://raw.githubusercontent.com/taktx-io/GOESB/main/scripts/install.ps1 | iexInstalls faster-whisper by default. Pick a different engine with an environment variable set before the command above:
GOESB_ENGINE=vosk # macOS/Linux, prepend to the curl command
$env:GOESB_ENGINE = "vosk" # Windows, run before the irm commandfaster-whisper— best accuracy/speed with a few GB of RAM to spare; runs on CPU by default and needs an explicitgoesb run --backend cudato use an NVIDIA GPU — GOESB never auto-selects a backend, since that would make the same profile/hardware silently mean different things run to run.vosk— smallest footprint, CPU only — the right default on the most constrained boards.whisper-cpp— also CPU by default;--backend cudaasks it to use its compiled-in GPU support (CUDA or Metal), where available.
Two more engines exist but have no standalone binary — they bundle PyTorch, which would push a single download past GitHub's release-asset limit. Install those with pipx/pip below instead: parakeet (NVIDIA Parakeet-TDT, one multilingual checkpoint, runs on CPU, CUDA or Metal) and nemotron (NVIDIA Nemotron 3.5 ASR, genuinely cache-aware streaming). nemotron is GPU-only — it declares no CPU backend at all, so --backend cpu against a nemotron profile is a hard error rather than a slow run. Use --backend cuda (NVIDIA) or --backend metal (Apple Silicon); goesb doctor tells you which your machine has.
This binary bundles that one engine only — no on-demand install like pipx/pip has, since there's no Python environment underneath it to add anything to. Picking a profile that needs a different engine (on the command line or in the wizard) fails clearly instead of silently: the wizard drops it from the batch with skipping ('vosk' not installed), a direct goesb run tells you which binary to download instead. To add or switch engines, just re-run the install command above with a different GOESB_ENGINE — it keeps each engine under its own name (goesb-vosk, goesb-faster-whisper, ...) and only re-points the plain goesb to whichever you installed most recently, so installing another never removes the one you already have; run a specific one directly by its full name if you want more than one available side by side.
Not covered by this installer: 32-bit ARM (e.g. older Raspberry Pi OS images still on armv7) and Intel Mac — use pipx/pip below on those. The script tells you clearly if your machine isn't covered rather than failing silently.
Have an NVIDIA GPU and want to use it? Run goesb doctor first — it reports what it detects (GPU, driver, whether cuBLAS/cuDNN actually loads) and what to fix, without running a benchmark. This is what catches a Windows install with a GPU present but no CUDA libraries installed, before --backend cuda fails partway through a real run instead.
pipx / pip — needs Python 3.11+, works on every architecture
The one to reach for on 32-bit ARM, Intel Mac, or anywhere else the quick installer doesn't cover. Check your version first — python3 --version (Windows: py --version). Then install via pipx — gives it its own isolated environment automatically, so it works cleanly even on Debian/Ubuntu (which blocks a bare pip install outside a venv by default). No engine to pick here — see below.
# if you don't have pipx yet:
sudo apt install pipx && pipx ensurepath # Debian/Ubuntu
brew install pipx # macOS
py -m pip install --user pipx && py -m pipx ensurepath # Windows
pipx install goesb-runnerPrefer a plain venv instead? python3 -m venv .venv && source .venv/bin/activate && pip install goesb-runner (Windows: .venv\Scripts\Activate.ps1) works the same way — inside an activated venv, plain pip/python already point at the right one. Outside a venv on a system where python/pip still resolve to Python 2, use python3/pip3 instead.
No engine (faster-whisper/vosk/whisper-cpp/parakeet/nemotron) is installed up front — the first goesb run that needs one prompts to install it on the spot, pinned to the exact runner version you already have. Running non-interactively (a script, CI, no TTY)? It prints the exact command instead of prompting: pip install "goesb-runner[faster-whisper]" (swap in vosk, whisper-cpp, parakeet or nemotron) — or install one ahead of time the same way if you'd rather not be prompted at all. parakeet and nemotron pull in PyTorch, so those two are a multi-hundred-MB download; nemotron additionally needs an NVIDIA GPU or Apple Silicon, and refuses to run on CPU rather than being slow.
Manual binary download — same binaries the quick installer uses
Useful on an air-gapped machine, or if you'd rather inspect the file before running it. One binary per engine, each one locked to that engine only — same limitation as the quick installer above — download from the latest release:
goesb-faster-whisper-macos-arm64 # or -linux-x64, -linux-arm64, -windows-x64.exe
goesb-vosk-macos-arm64
goesb-whisper-cpp-macos-arm64parakeet and nemotron aren't here on purpose: both bundle PyTorch and a single-file build would exceed GitHub's per-asset size limit. Install those with pipx/pip.
macOS/Linux: chmod +x the file after downloading. Every command below is the same, just run ./goesb-faster-whisper-macos-arm64 (or whichever you downloaded) instead of goesb.
From source — for contributing, or authoring a new pack
git clone https://github.com/taktx-io/GOESB.git
cd GOESB
python3 -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\Activate.ps1
pip install -e "./runner[dev]"dev is test/lint tooling, not an engine — same on-demand install as pipx/pip above applies here too. Want one ready before your first run instead? pip install -e "./runner[dev,faster-whisper]" (or vosk/whisper-cpp/parakeet/nemotron). whisper-cpp builds a native extension and needs a C++ toolchain (Xcode Command Line Tools + CMake on macOS, build-essential cmake on Debian/Ubuntu, Visual Studio Build Tools + CMake on Windows) — the other two install prebuilt wheels and need nothing extra.
Upgrading
Check what you have with goesb version, then upgrade with whichever method you installed with:
pipx upgrade goesb-runner # pipx (add --force if it reports no change but the version is still stale)
pip install --upgrade goesb-runner # plain venv (already have an engine installed? it upgrades too)
git pull # from source — editable install picks it up automaticallyQuick installer or manual binary — there's no separate upgrade command: the install script and every binary download always fetch the latest release and overwrite what's there, so re-running the exact command from step 1 (or downloading the file again) is the upgrade.
2. Pick a profile and pack
List what's available instead of guessing an id:
goesb list-profiles
goesb list-packsGOESB never hosts audio (privacy-first), so where the audio for a pack comes from depends on the pack:
- Official open packs (built from FLEURS or LibriSpeech, both plain ungated downloads) — nothing to do here.
goesb runin step 3 fetches the audio automatically, no clone or separate script needed. - Already have your own audio? Bring your own
pack.yaml(the in-browser builder makes one from your files) and point--audio-dirat it in step 3. - Common Voice packs — also auto-fetched, but gated behind a personal API key rather than a plain URL (ADR-0010): set
MDC_API_KEYto a key from the Mozilla Data Collective and fetching happens the same as any other pack. This needs thedatacollectivepackage, which isn't installed by default — the first run that needs it offers to install it on the spot; see the troubleshooting note below if that offer fails or you're running non-interactively. - Anything else (a private/custom pack, or a corpus with no scriptable source at all) —
goesb runcan't auto-fetch it and prints that pack's own manual fetch instructions instead.
Troubleshooting: “datacollective is not installed” — if you declined the install offer, or you're running goesb non-interactively (a script, CI, no TTY), the error tells you the exact command to run yourself, matched to how you installed the runner. If you installed with pipx, that's pipx inject goesb-runner datacollective — not pipx install datacollective, which installs it into its own separate, throwaway environment that goesb-runner can't see (the datacollective package is a dependency goesb-runner imports, not a standalone command-line tool of its own). Inside a plain venv, pip install datacollective is correct as-is — activating the venv already makes pip/goesb point at the same place.
3. Validate and run a benchmark
validate checks a local file you have in hand — useful while authoring a new profile/pack, not needed for an existing official one. run is the one that fetches: give it an official profile/pack id and it pulls whatever isn't already local — profile, pack metadata, and (for a known-source open pack) the audio itself:
# --model-override tiny keeps this fast for a first run
goesb run whisper-medium-en-batch librispeech-en --model-override tinyThis writes a signed, hashed result document to runs/results/, plus a sibling *.utterances.jsonl — one line per utterance per repeat, with the reference transcript next to exactly what the engine recognized, so you can spot-check quality yourself instead of trusting the aggregate WER blind. It's local-only and never submitted alongside the result. Fetched profiles and packs are cached under ~/.goesb/cache; auto-fetched audio lands right next to the pack (packs/<pack_id>/audio locally, or alongside the cached pack when it was fetched too) — either way, a second run of the same one is fully offline. Add --offline to refuse any network access and fail clearly if something isn't already local. --api-url overrides where it fetches from (defaults to https://www.goesb.com/api). Bringing your own audio? Add --audio-dir pointing at it.
4. Submit your result
A fresh, single-use signing token is requested automatically — your private key never leaves your machine.
goesb submit "$(ls -t runs/results/*.json | head -1)"$(ls -t runs/results/*.json | head -1) picks the most recently produced result automatically (macOS/Linux) — or substitute the exact filename the previous step printed. submit checks its target is actually compatible with this runner's version before sending anything, and tells you plainly to upgrade if not — see "Upgrading" in step 1 above.