demtmeder / audio-vis-align

What is audio-vis-align?

Audio-Vis-Align is a Python toolkit for training models that understand both sound and video together. Think of it as CLIP, but instead of matching images with text, it matches audio clips with their corresponding video. The system uses two separate encoders—one for audio waveforms, one for video frames—that learn to project into a shared space where matching audio-video pairs sit close together. Out of the box you get a complete pipeline: data loading with webdataset support, distributed multi-GPU training via torchrun, EMA for stable weights, and evaluation tools for retrieval metrics and zero-shot classification. The codebase is intentionally small and readable, designed for researchers who want to understand or modify the training logic without wading through abstraction layers.

Why is it gaining traction?

The hook here is reproducibility and honesty. The authors share the exact configs used to produce their published results—no hidden flags or undocumented hyperparameter tweaks. If you want to replicate their AudioSet pretraining or VGGSound finetuning numbers, the scripts walk you through it step by step. The toolkit also offers multiple loss variants, including hard-negative-aware InfoNCE, which gives researchers flexibility to experiment without rewriting boilerplate. The small footprint makes it approachable: you can read the whole training loop in one sitting.

Who should use this?

Multimodal researchers working on audio-visual alignment will find the most value here. If you're publishing on this topic and need a solid baseline or want to compare against their reported numbers, this is a clean starting point. ML engineers building audio-visual applications can use the pretrained checkpoints or fine-tune on VGGSound. The small-ablation config is practical for anyone doing rapid prototyping or testing ideas without burning GPU hours.

Verdict

This is a legitimate, well-documented research toolkit with a credibility score of 0.8999999761581421%, but at 81 stars it remains early-stage and community support is minimal. The code is clean and the results are verifiable, which is rare. Use it if you need audio-visual contrastive learning for research or product prototyping—just expect to do some of your own engineering for production-scale workflows.