DCDmllm / InstructSAM

What is InstructSAM?

InstructSAM is an instruction-driven segmentation framework that lets you mask arbitrary objects in images using plain text commands. Think of it as a "segment anything" tool that actually understands what you want -- you can say "segment the person on the left" or "find all the cars" and it returns precise instance masks. Built in Python on top of Qwen3-VL and SAM3, it handles three instruction styles: simple category prompts, referring expressions, and multi-step reasoning instructions. The project ships with pre-trained weights on HuggingFace, so you can run inference immediately without training.

Why is it gaining traction?

The key differentiator is flexibility -- unlike older segmentation models locked to fixed categories, InstructSAM responds to open-ended instructions. It outputs multiple instance masks in a single pass rather than requiring repeated SAM calls or agentic loops. The reasoning-style instructions are particularly interesting: you can chain logic ("segment the vehicle that is not a car") and it handles the multi-object complexity. Pre-trained weights are available, which means researchers can evaluate immediately while still having access to full training pipelines for fine-tuning.

Who should use this?

Computer vision researchers working on multimodal instruction following will find the training code useful for experiments. Developers building image annotation tools or content moderation pipelines could integrate this for flexible object extraction. It's less suited for production deployment right now given the early star count and limited community testing.

Verdict

InstructSAM solves a real problem with a clean approach, but with only 17 stars and a 0.85% credibility score, it's very early-stage software. The arXiv paper and demo videos add legitimacy, but expect rough edges. Worth exploring for research purposes or prototyping, but wait for more community validation before building critical pipelines around it.