vishwanathakuthota

vishwanathakuthota / openvals

Public

Open-source AI model evaluation and benchmarking framework for LLMs (OpenAI, Ollama, Claude, Gemini)

openvalidations.com ai-agents ai-evaluation ai-evaluation-framework ai-quality ai-reliability
17
3
89% credibility
Found May 05, 2026 at 17 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

OpenVals is an open-source Python tool for evaluating and benchmarking AI language models on metrics like accuracy, semantic similarity, latency, reliability, and safety to recommend the best model for specific tasks.

How It Works

1
📖 Discover OpenVals

You hear about a friendly tool that helps test and compare different AI assistants to find the best one for your needs.

2
💻 Set it up simply

You add the tool to your computer with an easy download, and everything is ready to go.

3
📝 Prepare your tests

You make a simple list of questions and expected answers to see how well AIs perform.

4
🧠 Link your AI helpers

You connect the AI assistants you want to test, like local ones running on your machine.

5
⚡ Run the evaluations

You press go, and the tool asks each AI your questions, measuring speed, accuracy, safety, and more.

6
📊 Get smart rankings

You see a clear leaderboard with scores, showing which AI excels in reliability, quickness, and trustworthiness.

✅ Choose with confidence

Now you have a full report to pick the perfect AI for your work, knowing it's safe and effective.

Sign up to see the full architecture

5 more

Sign Up Free

Star Growth

See how this repo grew from 17 to 17 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is openvals?

Openvals is a Python framework for evaluating and benchmarking LLMs from OpenAI, Ollama, Claude, and Gemini, acting as an open source model benchmark tool for local and cloud AI. Load your JSON datasets, run single-model evals or multi-model comparisons via CLI commands like `openvals benchmark`, and get normalized scores across accuracy, semantic similarity, latency, reliability, safety, and more. It ranks models with custom weights, generates HTML reports, and offers recommendations, helping teams quantify trust before deploying open-source models like those from Hugging Face.

Why is it gaining traction?

As a lightweight, self-hosted open source GitHub tool via Ollama, it skips proprietary dashboards for pip-install simplicity and local runs—no vendor lock-in. Developers hook it for weighted scoring tied to use cases like low-latency or high-safety, plus tradeoff analysis in minutes, standing out from basic metric scripts or paid services. Its extensible model adapters and CLI make benchmarking open-source LLMs feel like a GitHub Actions alternative for AI validation.

Who should use this?

AI engineers at SaaS firms comparing Ollama-hosted open-source models for production. ML teams in regulated industries needing safety and reliability scores on custom datasets. Devs evaluating open-source models ai like GPT-Neo alternatives before integrating into apps.

Verdict

Grab it for quick LLM benchmarks if you're on Ollama—solid alpha with MIT license and clear docs, despite 17 stars and 0.9% credibility score signaling early days. Test on toy datasets first; extend for real wins as it matures.

(198 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.