SeraphimSerapis

SeraphimSerapis / tool-eval-bench

Public

Tool-calling quality benchmark for LLM serving stacks. 65+ deterministic scenarios testing multi-turn orchestration, safety boundaries, and structured output. Supports vLLM, LiteLLM, and llama.cpp.

10
1
100% credibility
Found Apr 20, 2026 at 10 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

tool-eval-bench is a benchmark tool that tests AI models' ability to select and use tools correctly in multi-step conversations across various AI servers.

How It Works

1
🔍 Discover the Tool Tester

You hear about a simple way to check if AI helpers can use everyday tools like weather checkers or email senders correctly.

2
📥 Get the Tester Ready

With one easy command, you add this tester to your computer, like installing a helpful app.

3
🔌 Link Your AI Helper

You tell the tester where your AI lives by sharing its web address, so it can chat with it.

4
⚡ Run Your First Check

Click to start a quick test — watch as it asks your AI simple questions and sees if it picks the right tools.

5
📊 See the Scores

Get a colorful report showing stars like excellent or good, with breakdowns of what went right or wrong.

6
Choose Full Test or Speed Check
📈
Full Quality Test

Run all challenges to measure tool smarts deeply.

🚀
Speed Test Too

Check thinking speed alongside tool accuracy.

7
🏆 Compare and Improve

Save results, compare different AIs, and build a leaderboard to pick the best one.

🎉 Your AI is Ready

Celebrate knowing exactly how well your AI handles tools, ready for real conversations.

Sign up to see the full architecture

6 more

Sign Up Free

Star Growth

See how this repo grew from 10 to 10 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is tool-eval-bench?

tool-eval-bench is a Python CLI benchmark for testing LLM tool calling quality on OpenAI-compatible endpoints like vLLM, LiteLLM, and llama.cpp. It runs 65+ deterministic scenarios covering multi-turn orchestration, safety boundaries, structured output, and agentic workflows, scoring each as pass, partial, or fail with detailed traces. Developers point it at their server via `tool-eval-bench --seed 42` and get instant reports on tool selection, parameter precision, error recovery, and throughput.

Why is it gaining traction?

Unlike single-turn evals like BFCL or ToolBench, it focuses on real-world agentic tool calling with noisy mock responses, context pressure sweeps, and safety gating that caps ratings for poor boundary handling. Integrated llama-benchy throughput, speculative decoding metrics, and multi-trial Pass@k stats make it a one-stop shop for llm tool calling github comparisons, beating ad-hoc scripts or LangChain/LangGraph tool calling github tests in reproducibility and depth.

Who should use this?

LLM serving engineers deploying vLLM or llama.cpp stacks need it to tune tool calling agents before production. Model evals teams comparing open-weight LLMs on Ollama or LiteLLM will value the 65+ bench for programmatic tool calling github baselines. OpenAI tool calling.github migrants building langgraph tool calling agent github workflows get quick safety and multi-turn checks.

Verdict

Grab it if you're serious about tool calling evals—installs cleanly via uv tool, docs are thorough, and outputs are screenshot-ready. With only 10 stars and 1.0% credibility score, it's early-stage but stable enough for daily use; run it on your stack today.

(198 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.