OxRML / MADQA

What is MADQA?

MADQA is a Python benchmark for testing multimodal agentic AI systems on document QA, with 2,250 human-authored questions over 800 diverse PDFs. It evaluates how agents reason across pages, retrieve evidence, and generate attributed answers—solving the gap in benchmarks for real-world multimodal RAG and agentic frameworks. Load the dataset from Hugging Face, run baselines like BM25+MLLM agents or managed services, then score predictions via CLI with metrics like accuracy, semantic match, and page F1.

Why is it gaining traction?

It stands out by pitting agents against human performance on strategic navigation vs. stochastic search in doc collections, with a public leaderboard and arXiv paper. Developers dig the ready baselines for github multimodal llm and multimodal agentic rag setups, plus eval tools that handle JSONL outputs from any system. No more hand-rolling metrics for multimodal agentic systems.

Who should use this?

AI researchers benchmarking multimodal agentic models or frameworks, especially for github multimodal rag pipelines handling PDFs. Teams building agentic systems for stock insights or document analysis, like legal tech or finance devs needing page-level citations.

Verdict

Grab it if you're in multimodal agentic AI—solid starting point despite 19 stars and 1.0% credibility score signaling early maturity. Docs are clear with quickstarts, but expect tweaks for production scale.

(198 words)