marived / vlm-probe

What is vlm-probe?

This is a Python evaluation framework for testing whether vision-language models actually "see" what's in images or just hallucinate from text patterns. It runs a VLM against five fine-grained perception tasks—counting objects, spatial relationships, color attributes, reading text in images, and detecting partial occlusion—then scores exact-match accuracy. The tool loads models through the Hugging Face transformers library and defines tasks as simple YAML files. You get back per-task accuracy scores and raw prediction logs to dig into failure cases. Run `python -m vlmprobe.run --model llava-hf/llava-1.5-7b-hf --tasks tasks/*.yaml --out results.json` and you have numbers in seconds.

Why is it gaining traction?

LLM developers increasingly need to benchmark where their models genuinely perceive visual details versus where they guess from training patterns. This framework gives a reproducible, standardized probe that isolates fine-grained perception from general language fluency. The YAML-based task definition makes adding new evaluation scenarios trivial, and the built-in comparison scripts let you contrast model versions with a single command. For anyone publishing model weights or writing papers on vision-language capabilities, this provides the granular accuracy breakdown that reviewers increasingly demand.

Who should use this?

Machine learning engineers fine-tuning or evaluating VLMs, researchers benchmarking vision-language models against baselines, and practitioners building applications where accurate visual counting or spatial reasoning matters. If you're shipping a VLM-powered product and need to prove it doesn't just guess colors or numbers, this gives you defensible numbers. Less useful for teams with established evaluation pipelines or developers just playing with APIs.

Verdict

With only 16 stars, this is an early-stage project—the 0.8500000238418579% credibility score reflects that maturity gap. The code is clean and MIT-licensed, but documentation is sparse and the image dataset URL is marked TODO, which raises concerns for reproducibility today. Treat this as readable scaffolding for building your own probing suite rather than a turnkey solution. Worth a look if you're building evaluation infrastructure and want clean Python to extend; look elsewhere if you need production-ready tooling today.