TaimoorKhan10 / replayd

What is replayd?

replayd is a Python library that turns failed AI agent runs into replayable regression tests. When your AI agent screws up in production, you capture that failure, save it as a test, and replay it before every deployment to make sure the same bug never comes back. The tool grades on deterministic facts (which tools got called, with what arguments) rather than fuzzy output matching, so tests stay reliable even when LLMs generate different text for the same correct behavior. Optional semantic grading using an LLM-as-judge handles failures that require reading context, like "did the agent approve a refund that exceeds policy?"

Why is it gaining traction?

This solves a real pain point: AI agents regress silently. You fix a bug, update a prompt, or switch models, and the same mistake creeps back without anyone noticing until users complain. replayd makes that impossible by blocking releases when known failures resurface. Unlike observability tools that tell you what happened after the fact, this is an active release gate. The structural grading approach (check tool calls first, LLM second) is smart design -- it avoids expensive LLM calls for straightforward failures and keeps most tests fast and deterministic.

Who should use this?

Teams deploying AI agents in production who need confidence that shipped changes won't reintroduce known bugs. If you're running customer-facing agents with tool-calling capabilities (approvals, refunds, searches, etc.), this gives you regression coverage that traditional software testing never provided for AI behavior. It's especially useful for teams iterating fast on prompts or models where silent regressions are the biggest deployment risk.

Verdict

replayd addresses a genuine gap in AI deployment tooling with a well-designed, pragmatic approach. However, the credibility score sits at roughly 0.9% with only 10 stars and version 0.1.2 -- this is very early-stage. The code is clean and the concept is solid, but I'd want to see more battle-testing before staking production releases on it. Worth exploring in a non-critical project to evaluate fit, but keep an eye on the roadmap.