nomic-ai / aec-bench

What is aec-bench?

AEC-bench is a multimodal benchmark for testing agentic systems on real-world architecture, engineering, and construction documents like drawings, specs, and submittals. It packs 196 tasks across 9 types in three scopes—intrasheet, intradrawing, intraproject—challenging AI agents to handle cross-reference resolution, sheet indexing, and more via vision and reasoning. Built on Python with the Harbor framework and Docker sandboxes, you prefetch PDFs from manifests, plug in API keys for Claude, Codex, or Nomic agents, and run evals through simple Harbor CLI commands like `harbor trials start` or batch jobs.

Why is it gaining traction?

Unlike generic benchmarks, this AEC challenge on GitHub targets construction-specific pain points with actual bid sets, making agent performance directly relevant to AEC tech. Harbor's sandboxed runs ensure reproducible, secure evals without local setup hassles, and results stream to JSONL for easy analysis—perfect for iterating on multimodal models. Backed by Nomic, an arXiv paper, and a Hugging Face dataset, it's a ready-to-run AEC benchmark drawing early interest from agent builders.

Who should use this?

AI researchers tuning vision-language agents for document QA in engineering and construction. AEC tech teams validating tools against real drawing sets before deployment. Multimodal system devs needing a shell-scriptable benchmark to compare models like Claude Opus or GPT variants on tasks from detail tracing to submittal reviews.

Verdict

Grab it if you're in agentic AEC—solid foundation despite 19 stars and 1.0% credibility score signaling early days. Docs are crisp, Harbor integration shines, but expect to prefetch files and tweak agents for top scores. Worth a weekend eval for multimodal progress tracking.