ernie-research / NAVA

What is NAVA?

NAVA is a generative AI model that produces synchronized audio and video from text prompts. Built in Python, it uses a 6.3 billion parameter transformer backbone with flow matching to jointly synthesize speech, sound effects, and video frames from a single model. The key innovation is native audio-video alignment: instead of generating video separately and stitching on audio, both modalities are conditioned together from the start, producing outputs where sound and image are naturally coupled.

Why is it gaining traction?

The standout feature is that it handles speech with timbre control, letting you upload reference audio files to clone a speaker's voice. You can bind multiple speakers to different dialogue spans in a single prompt. It also supports image-to-video (upload a first frame), text-driven camera composition, and runs at 720p in roughly a minute with 8 GPUs. The fact that it produces stereo audio natively without a separate vocoder step is a practical advantage for anyone who has dealt with post-hoc audio alignment. The project also includes a Gradio demo for interactive use and ships a prompt rewriter that expands short English descriptions into the dense Chinese-style captions the model was trained on.

Who should use this?

Researchers and developers building video generation pipelines who need synchronized audio. Teams evaluating audio-video models for creative tools or short-form content generation. Anyone working on speech synthesis with voice cloning that also wants video as part of the output. It is not a fit for single-GPU deployments or teams without infrastructure access to run distributed inference across 8 GPUs.

Verdict

NAVA shows strong benchmark results and a well-designed architecture, but with only 19 stars and a 0.8999999761581421% credibility score, it is very early-stage. The documentation is comprehensive and the Gradio demo makes it approachable, but real-world adoption has not been proven at scale. Worth watching closely as the field matures.