hongruijia / DPE

What is DPE?

DPE is a Python framework that iteratively evolves large multimodal models by diagnosing blind spots in vision-language tasks like math reasoning, charts, and OCR. You feed it a base model like Qwen-VL, run the pipeline script to spot weaknesses on benchmarks such as MMMU or MathVision, generate ~1,000 targeted image-question pairs via tool-using agents, then fine-tune with RL or SFT. Outputs ready-to-use models on Hugging Face.

Why is it gaining traction?

Unlike brute-force data scaling, DPE's adaptive diagnosis dynamically mixes training data to hit long-tail gaps—charts, formulas, dense text—boosting stability without regressions. High efficiency shines: broad gains from minimal examples, plus a model zoo for instant testing. vLLM integration keeps inference fast during eval and gen.

Who should use this?

Researchers fine-tuning LMMs for multimodal apps with blind spots in visual reasoning, like academic chart analysis or OCR-heavy docs. Teams iterating Qwen-VL variants without endless data curation.

Verdict

Grab it if you're debugging LMM weaknesses—strong paper and HF models make it low-risk to try. With 46 stars and 0.7% credibility score, it's nascent; docs are README-focused, no deep tests yet, but promising for targeted gains.

(178 words)