bahadiraraz

bahadiraraz / LLMTest

Public

The pytest for LLMs. Fast, deterministic assertions for AI applications.

assertllm.dev llms pytest python testing
12
0
100% credibility
Found Mar 10, 2026 at 12 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

assertllm is a Python testing tool that checks AI language model responses for accuracy, speed, cost, and structure, much like unit tests for code.

How It Works

1
📰 Discover assertllm

You hear about a friendly tool that lets you test your AI helper's answers, just like checking if a recipe turns out right.

2
📦 Add it to your toolbox

You bring this testing helper into your project with a simple download, ready to watch your AI closely.

3
✍️ Write a simple check

You jot down easy rules like 'answer must mention Paris, arrive in seconds, and cost pennies' for your AI questions.

4
▶️ Run your checks

You hit go, it asks your AI real questions and automatically verifies every answer against your rules.

5
📊 See the friendly report

A clear summary pops up with green ticks for wins, red flags for issues, plus speed and cost details to celebrate or tweak.

✅ AI answers shine reliably

Your AI now delivers spot-on, speedy responses every time, giving you total confidence without surprises.

Sign up to see the full architecture

4 more

Sign Up Free

Star Growth

See how this repo grew from 12 to 12 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is LLMTest?

LLMTest (assertllm) brings pytest-style testing to LLM applications, letting you write fast, deterministic assertions on AI outputs like text content, latency, cost, and tool calls. Install via pip with provider extras (OpenAI, Anthropic, Ollama), decorate tests with @llm_test and expect.contains("Paris") or expect.latency_under(2000), then run pytest as usual for detailed reports. It solves flaky LLM evals by skipping extra model calls for most checks, using Pydantic for structured validation in RAG or agent apps.

Why is it gaining traction?

Unlike deepeval or gkamradt/llmtest_needleinahaystack, it plugs straight into pytest for llm test github workflows, with JUnit XML and JSON reporters for GitHub Actions CI/CD—annotate failures, track costs, and gate deploys on critical assertions. CLI tools like llmtest init and llmtest run tests/ --junit handle datasets from YAML/JSON for llmtestcase documentation, while agent checks catch loops and tool order without custom scripts. Zero-LLM overhead makes it instant for pytest llms txt regressions.

Who should use this?

Python backend devs building LLM-powered APIs, RAG pipelines, or agents who already use pytest for unit tests. AI/ML engineers needing reproducible evals in GitHub CI, or teams evaluating llm test needleinahaystack on OpenAI/Anthropic before prod.

Verdict

Solid alpha for pytest github action report fans—grab it if you want deterministic llm test case assertions in CI, but watch maturity (10 stars, 1.0% credibility). Docs are crisp; test locally with Ollama before scaling.

(198 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.