outsourc-e / bench-loop

What is bench-loop?

Bench-loop is a local-first CLI tool for benchmarking LLMs on your own hardware, testing quality across suites like tool calling, data extraction, instruction following, reasoning math, coding, and multi-turn agent loops, plus raw speed and reliability. Developers run `benchloop run --endpoint http://localhost:11434` to evaluate local setups like Ollama or OpenAI-compatible servers (LM Studio, vLLM), getting hardware-aware scores for tokens/sec, pass rates, and overall value. It includes a bundled web dashboard for results visualization and exports JSON for public leaderboards.

Why is it gaining traction?

Unlike cloud benchmarks, bench-loop runs fully local-first on real hardware, capturing GPU temps, VRAM limits, and multi-turn agent behavior that remote evals miss. The CLI auto-detects providers by port, handles harnesses for Hermes/Qwen prompting, and spits out reproducible reports with median latencies—no setup hell. Early users share bench loop notes and survey results on forums, hooking hardware tinkerers comparing quants like Qwen3 on RTX 4090s.

Who should use this?

Local LLM runners benchmarking Ollama models before deployment, like AI engineers tuning agent pipelines or inference speed on consumer GPUs. Hardware enthusiasts doing bench loop trails on M-series Macs vs. NVIDIA rigs, or teams evaluating tool-use reliability in coding/dataextract tasks. Skip if you're only on cloud APIs—it's for CLI-driven, local-first github workflows.

Verdict

Solid beta for local LLM benchmarking (Python, MIT, FastAPI dashboard), but 14 stars and 1.0% credibility signal early days—docs are README-focused, no deep tests visible. Try it for agent/bench loop baselines; contribute suites to mature it.

(198 words)