hkust-nlp / LOCA-bench

What is LOCA-bench?

LOCA-bench is a Python benchmarking suite for testing language agents—LLMs paired with scaffolds like ReAct or programmatic tool calling—under controllable context growth up to extreme lengths like 128K+ tokens, while keeping task semantics fixed. Developers run evals via a simple CLI (`loca run`) on presets from 8K to 256K tokens across 15+ tasks including games, math problems, code generation, QA, and real-world scenarios like A/B testing analysis or academic warnings via mock services for Google Cloud, Canvas, and email. It outputs detailed trajectories, token stats, and web visualizations for dissecting agent behavior in long-context settings.

Why is it gaining traction?

Unlike standard LLM benchmarks focused on fixed inputs, LOCA-bench lets you scale context dynamically to probe limits in controllable generation under diversified instructions, retrieval-augmented setups, or code syntax understanding—key pain points as models push 1M+ tokens. Built on a solid agent framework with Claude/Anthropic support and context reset strategies, it delivers aggregated results.json plus per-task breakdowns, making it dead simple to compare models like DeepSeek or Sonnet on agentic long-context performance.

Who should use this?

NLP researchers benchmarking large language models on extreme context growth or agents in retrieval-augmented generation. Teams evaluating vision-language models for tasks like handwritten text recognition or cultural understanding via scalable envs. Devs tuning scaffolds for code generation creativity or news summarization under growing inputs.

Verdict

Grab it if you're into llm benchmarking github for agents—solid CLI and outputs make it practical despite 17 stars and 1.0% credibility signaling early days. Docs are clear with arXiv paper, but expect beta rough edges; fork and contribute to mature this long-context bench.

(198 words)