ytang928 / BrainBench

What is BrainBench?

BrainBench runs a 100-question benchmark across 20 commonsense reasoning categories to expose failure modes in LLMs, like implicit physical constraints or wrong vantage points—traps humans dodge but models hit hard. It provides English and Chinese datasets in JSON, plus Python evaluation code that queries frontier models via OpenAI, Anthropic, Google, or OpenRouter APIs, judges responses automatically, and aggregates accuracy plus reliability scores. Unlike generic evals, it targets brainteasers where LLMs flop despite scale.

Why is it gaining traction?

It stands out with bilingual support and "thinking" mode tests for models like Claude Opus or GPT-5.4, generating plots on category breakdowns and consistency—ideal for spotting real reasoning gaps beyond leaderboards. The CLI lets you benchmark any model in minutes with resume support, multiple runs, and analysis scripts for publication-ready visuals, hooking devs tired of vague MMLU scores.

Who should use this?

AI researchers benchmarking LLMs for papers, like brainbench technology evals on commonsense datasets. LLM teams at startups testing deployment candidates across English/Chinese. Model trainers diagnosing category-specific flaws before fine-tuning.

Verdict

Grab it if you're evaluating LLMs—solid docs, MIT license, and ready-to-run CLI make it useful now, despite 41 stars and 1.0% credibility signaling early maturity. Run your models; the results will reveal more than hype. (187 words)