sitodowubb

sitodowubb / spatial-vqa-bench

Public

Spatial-VQA-Bench: a focused benchmark of spatial visual reasoning for multimodal LLMs.

16
0
85% credibility
Found May 25, 2026 at 46 stars 5x -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

Spatial-VQA-Bench is a research tool that tests how well AI models understand spatial relationships in images, providing 3,200 questions across five categories (2D relations, 3D relations, rotation, occlusion, and viewpoint) along with software to run models and score their performance.

How It Works

1
💡 You discover a spatial reasoning test

You learn about a benchmark that measures how well AI understands where things are in pictures — like knowing if one object is behind another or to the left.

2
📦 You install the testing tool

With one simple command, you set up the software on your computer so it's ready to run tests whenever you want.

3
You choose which AI to test
🔍
Try an open-source AI

Test a freely available model like LLaVA or Qwen2-VL that runs on your own computer

☁️
Try a cloud AI service

Test a powerful cloud-based model like GPT-4o that thinks through your images

4
⚡ The AI examines thousands of images

The tool automatically shows each image to the AI, asks spatial questions, and records every answer the AI gives.

5
📊 You see how well the AI performed

A clear report shows the AI's accuracy across different types of spatial tasks, revealing where it excels and where it struggles.

🎯 You understand the AI's spatial abilities

You now know exactly how good this AI is at understanding positions, rotations, and what objects look like from different angles.

Sign up to see the full architecture

4 more

Sign Up Free

Star Growth

See how this repo grew from 46 to 16 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is spatial-vqa-bench?

Spatial-VQA-Bench is a Python benchmark that isolates spatial visual reasoning from other VQA capabilities. While most benchmarks wash out spatial understanding into object recognition scores, this tool tests specifically whether multimodal LLMs understand left/right, behind/in-front, near/far, and mental rotation. It ships 3,200 hand-vetted questions across five task families: 2D relations, 3D relations, rotation, occlusion, and viewpoint. The CLI lets you run evaluations against supported models like Qwen2-VL, LLaVA, and GPT-4o, then score predictions with a simple command.

Why is it gaining traction?

The benchmark fills a gap: existing VQA datasets treat spatial reasoning as a footnote, so you cannot isolate whether your model actually understands spatial relationships or just recognizes objects well. Spatial-VQA-Bench provides the granularity to answer that question. The results table is revealing—every model struggles most on rotation and viewpoint tasks, suggesting these require genuine mental simulation rather than pattern matching. The setup is minimal: install via pip, point at a model, get per-task accuracy breakdowns.

Who should use this?

ML engineers comparing multimodal LLMs for spatial applications—robotics, AR, navigation systems—will find the most value. Researchers studying visual reasoning capabilities can use the task families to drill into specific failure modes. If you are building anything where "the cup is behind the laptop" matters, this benchmark tells you which models actually get it.

Verdict

The benchmark is well-designed and the CLI is clean, but with only 16 stars it is early-stage and community validation is minimal. The 0.85% credibility score reflects reasonable engineering choices despite low visibility. Worth evaluating for serious VLM selection work, but treat it as a specialized tool rather than a mainstream standard.

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.