qlabs-eng / slowrun

What is slowrun?

Slowrun is a Python benchmark for training transformer language models on a fixed 100M tokens from FineWeb, with no time limit—lowest validation loss wins. It flips the script on speedruns like modded-nanogpt by prioritizing compute-heavy techniques over wall-clock speed, letting you hammer 100M tokens for maximum learning. Run python prepare_data.py to tokenize the data, then torchrun train.py on H100s for the 2.7B baseline hitting 3.402 val loss; submit PRs to leaderboards for unlimited or 1-hour limited tracks.

Why is it gaining traction?

Unlike data-rich speedruns, slowrun unlocks heavy regularization, exotic optimizers like Muon, and massive overparameterization on github 100m datasets, potentially yielding GPT-3-style leaps. Karpathy's endorsement adds cred, and the simple PR-based leaderboard hooks competitive devs chasing best val loss without infinite data worries. Early traction from its contrast to prithvi 100m github or tahoe 100m github experiments.

Who should use this?

ML engineers with H100 clusters testing optimizers or scaling laws in data-limited regimes. Researchers benchmarking regularization on 100M token caps, or teams iterating slowrun speedrun hybrids for production LM fine-tunes. Avoid if you're on consumer GPUs—needs Hopper for Flash Attention.

Verdict

Grab it if you have the hardware to beat the baseline; 84 stars and 1.0% credibility score signal an early, barebones benchmark with solid docs but no tests or broad validation yet. Fork and PR your improvements—low barrier for real contributions.

(198 words)