keyangds

keyangds / interactive_evaluation

Public

A curated resource for interactive evaluation: framework, taxonomy, design principles, and benchmark collection.

18
1
100% credibility
Found May 24, 2026 at 18 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
AI Summary

This is an academic research paper that argues for a new science of testing AI systems. Traditional AI tests check if an AI gives a correct final answer, but modern AI often works by taking many actions over time—clicking buttons, writing code, having conversations. The paper provides a framework and visual map showing how to properly test these interactive AI systems, categorizes 56 existing benchmarks, and offers five clear principles for designing fair, comprehensive evaluations. It was written by researchers from UT Austin, CMU, and other universities, and is published on arXiv with an MIT open-source license.

How It Works

1
🔍 You discover the research

You come across a paper about evaluating AI systems that interact with the world, not just answer questions.

2
📚 You read the core insight

The paper explains why traditional AI testing isn't enough when AI uses tools, browsers, or talks to people.

3
🗺️ You explore the 2D map

A visual map shows you all the different ways AI can be tested interactively, organized by what it interacts with and how it's judged.

4
You find your path
📋
Browse the benchmark collection

You discover 56 existing tests organized into three stages of evolution, from simple answers to complex interactions.

📐
Study the design principles

You learn five clear rules for building fair, comprehensive tests for interactive AI systems.

5
💡 You see the gaps

The paper highlights what's missing—like testing AI that remembers things across long conversations.

6
✍️ You apply the framework

You use these principles to design better tests for your own AI project or research.

🎯 Your evaluation improves

Your AI system gets tested more fairly, catching problems that simple answer-checking would miss.

Sign up to see the full architecture

5 more

Sign Up Free

Star Growth

See how this repo grew from 18 to 18 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is interactive_evaluation?

This is a research paper repository that proposes a design science framework for evaluating AI systems that interact with tools, environments, users, or other agents. Rather than just checking final answers, the framework focuses on what "trajectory evidence" enters evaluation and how that evidence maps to system judgments. It includes a 2D taxonomy organizing benchmarks along evaluation inputs (tools, users, agents, hybrid) and evaluation programs (task success, process quality, recoverability, safety). The repo also maintains a curated list of 56 benchmarks organized across three evolutionary stages: response-centered, task-driven, and interactive evaluation.

Why is it gaining traction?

The field is flooded with new benchmarks, but nobody has stopped to ask what "interactive evaluation" actually means or how to design it properly. This work provides conceptual scaffolding that the community desperately needs as AI systems increasingly act through multi-step trajectories rather than producing single responses. The five design principles give practitioners a concrete starting point rather than another benchmark to evaluate against. The taxonomy is particularly valuable for anyone building agentic systems who needs to understand where their evaluation approach falls in the landscape.

Who should use this?

Researchers building agent evaluation frameworks will find the taxonomy useful for positioning their work. ML engineers selecting benchmarks for agentic applications can use the curated benchmark list as a reference guide. Anyone working on multi-step AI systems where final outputs are insufficient evidence of quality will benefit from the framework's distinctions between response-centered and interactive evaluation.

Verdict

At 18 stars and 1.0% credibility, this is a research proposal dressed as a repository. The framework is intellectually valuable and the benchmark curation is thorough, but there's no code to integrate, just a paper to read. Use it as a reference for understanding evaluation design, not as a tool to add to your stack. The ideas are worth absorbing; the repo itself is mostly a link to the arXiv paper.

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.