BennettSchwartz / theobench

What is theobench?

TheoBench is a TypeScript benchmark measuring how well large language models know Theo Browne (t3.gg), using 70 fixed questions on his personal history, tech opinions, T3 stack, company, and content. Run the CLI to query models via OpenRouter, Claude, or Codex, auto-judge scores from 0.0 to 1.0 against references, and export JSON for a React leaderboard site deployed to Cloudflare Workers. It tracks official full runs versus unofficial partial progress, like a benchmark measuring tool for LLM built-in knowledge.

Why is it gaining traction?

Unlike generic benchmarks, this niches in on Theo Browne's world, revealing quirks in model training data around dev influencers—perfect for spotting blind spots in semantic knowledge. The CLI handles concurrency across 100+ models, balanced question ordering, and one-command exports to a polished Tailwind leaderboard with category filters and pagination. Devs dig the GitHub Action-ready workflow and easy local dev with pnpm.

Who should use this?

AI eval engineers benchmarking LLMs on person-specific recall, like testing GitHub Copilot or custom fine-tunes against influencer lore. Theo fans or T3 stack devs validating model opinions on Next.js vs. others. Researchers extending it for similar benchmarks, such as measuring head knowledge in niche domains.

Verdict

Fun niche benchmark with strong docs and deploy scripts, but 16 stars and 1.0% credibility signal early maturity—fork it for custom evals rather than production. Solid starter for LLM leaderboard experiments.

(198 words)