facebookresearch

facebookresearch / WavFlow

Public

MultiModal Audio Generation in Raw Waveform Space.

78
4
94% credibility
Found May 23, 2026 at 81 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

WavFlow is an open-source research project from Meta AI that generates synchronized, high-quality audio from video and/or text inputs. Unlike traditional approaches that work through compressed representations, WavFlow processes raw audio waveforms directly. The system uses visual features (CLIP frames), audio-visual synchronization signals (Synchformer), and text features (CLIP text encoder) as conditioning inputs, then employs flow matching — a technique that gradually transforms random noise into audio that matches the input conditions. It supports video-to-audio generation (matching sounds to visual events), text-to-audio generation (creating sounds from descriptions), and hybrid approaches. The project is available under a non-commercial license and includes complete training code, though pre-trained checkpoints are not yet released.

How It Works

1
📚 You discover a new way to make sounds

You learn about WavFlow through its project page or research paper — a tool that creates audio directly from videos or text descriptions, working at the raw sound level rather than through compressed representations.

2
🛠️ You set up your workspace

You run a simple setup script that installs everything you need, and the tool automatically downloads the helper models it requires from the internet.

3
🎬 You prepare your source material

You create a simple spreadsheet listing your videos or text descriptions, telling the system which ones have video and which have text captions.

4
⚡ The system learns what your videos sound like

The tool analyzes each video frame-by-frame and extracts visual patterns, audio-visual sync cues, and text meanings — these become the 'blueprint' for generating matching sounds.

5
You choose your path forward
🏋️
Train your own model

You run the training script with your prepared data, and over time the system learns to generate audio that matches your specific videos or descriptions.

🔍
Wait for open checkpoints

You follow the project for future updates when the team releases models trained on publicly available data.

🎉 Your audio comes to life

You provide a video or description, and within moments the system creates matching sound effects — forest ambience, drum beats, sports sounds, whatever your input calls for.

Sign up to see the full architecture

4 more

Sign Up Free

Star Growth

See how this repo grew from 81 to 78 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is WavFlow?

WavFlow is a research project from Meta AI that generates audio directly from video clips, text captions, or both. Instead of working in a compressed latent space like most audio generation models, it processes raw waveforms end-to-end. The system uses a technique called flow matching to learn how to produce audio that matches visual and textual inputs, and it can handle video-only, text-only, or combined inputs through learned "empty" embeddings. You feed it a CSV describing your data, run feature extraction on videos, and then either train your own model or use a trained checkpoint to generate synchronized sound effects, music, or environmental audio.

Why is it gaining traction?

The main appeal is that WavFlow bypasses traditional latent compression entirely. Most audio generation models decode from a compressed representation, which can lose acoustic detail. WavFlow claims performance on par with latent-based methods while working directly on waveforms, which could mean richer, more faithful sound reproduction. The multimodal flexibility is also notable: you can generate audio from video alone, text alone, or both together. For researchers interested in raw waveform generation without the black-box nature of latent diffusion models, this is a concrete, open implementation.

Who should use this?

This is squarely aimed at AI researchers and audio ML engineers exploring multimodal generation. If you're working on foley generation, sound effect synthesis, or video-to-audio synchronization, WavFlow gives you a trainable foundation. It's not ready for production use: the team hasn't released production-trained checkpoints, so you must train your own, which requires multi-GPU setup and significant compute. If you want to experiment with flow matching on raw audio or build on top of the architecture for a research project, this is worth a look. If you need something turnkey for audio generation today, look elsewhere.

Verdict

WavFlow is an interesting research artifact with a novel architectural approach, but with only 78 stars and no released checkpoints, treat it as experimental code rather than a usable tool. The 0.95% credibility score reflects genuine early-stage status: documentation is sparse, setup is involved, and you'll need to train your own models. Meta AI's backing provides some confidence in the underlying research, but budget that compute before committing.

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.