LongHorizonReasoning / longcot

What is longcot?

LongCoT is a Python benchmark for testing LLMs on long-horizon chain-of-thought reasoning, with 2,500 expert-designed problems across logic, computer science, chemistry, chess, and math domains. It pairs short prompts with massive reasoning traces—tens to hundreds of thousands of tokens—where difficulty comes from sustaining coherence over many steps, like tracking state or recovering from errors. Users get CLI tools to run inference on providers like OpenAI or Anthropic (`uv run python run_inference.py --config oai_gpt52`), save JSONL responses, and evaluate accuracy via deterministic verifiers (`uv run python run_eval.py`), plus a Python API for custom harnesses.

Why is it gaining traction?

Unlike generic LLM benchmarking GitHub repos, LongCoT focuses on long-horizon tasks that expose real weaknesses in frontier models, with programmatic verification in every domain—no fuzzy human judging. Built-in configs for models like Claude Sonnet or DeepSeek, parallel inference with retries, and fallback LLM judges for edge cases make evals fast and reliable. The arXiv paper and leaderboard submissions hook researchers chasing deepplanning benchmarks for chain-of-thought limits.

Who should use this?

LLM eval engineers benchmarking reasoning depth on production models. AI researchers comparing long CoT across providers before fine-tuning. Devs at labs like Oxford or LLNL validating long-horizon capabilities in logic puzzles or chemistry synthesis without building verifiers from scratch.

Verdict

Grab it for specialized LLM benchmarking on GitHub if long-horizon CoT is your focus—docs are thorough, setup via uv is smooth, and the API integrates easily. At 20 stars and 1.0% credibility, it's early but backed by a strong arXiv paper; test on LongCoT-Mini first to confirm fit.

(198 words)