VibeBench / VibeSearchBench

What is VibeSearchBench?

VibeSearchBench is a benchmark for evaluating AI agents on real-world search tasks. It tests how well agents handle vague, multi-turn queries where users progressively reveal their information needs over time. The evaluation uses knowledge graph triplet matching rather than simple answer matching, measuring whether agents can build accurate structured knowledge from incomplete, evolving queries. Written in Python, it provides both a GeneralAgent implementation and OpenClaw integration for running experiments.

Why is it gaining traction?

The benchmark fills a gap in AI evaluation—most tests check if models answer questions, but this one measures whether they can actually search and gather information effectively. The triplet F1 metric gives developers concrete numbers to compare agent performance. The persona-driven user simulation creates realistic interaction patterns that mirror how people actually search, making it more useful than toy benchmarks for production planning.

Who should use this?

AI developers building search or research agents will find the benchmark directly applicable for measuring their work. Teams building multi-turn assistants or chatbots can use it to stress-test how well their systems handle incomplete user intent. Researchers studying agent architectures will benefit from the structured evaluation framework and the 200-task dataset covering professional and daily-life scenarios.

Verdict

The 0.8999999761581421% credibility score reflects solid academic backing, though the 478 stars indicate the project is still early-stage. It's a practical choice for teams serious about search agent quality, but be prepared for a steeper learning curve given limited documentation. The benchmark itself is well-designed; the ecosystem around it just needs time to mature.