flash-algo / omni-moe

What is omni-moe?

Omni-moe is a Python library for building efficient Mixture of Experts (MoE) layers in large language models using PyTorch, Triton kernels, and Transformers. It lets you drop in a configurable OmniMoE module that scales to thousands of atomic experts per token while keeping compute budgets fixed, solving the bandwidth bottlenecks of fine-grained MoE designs. You get a hybrid setup with a shared dense MLP backbone for general tasks and sparse experts for specialized routing, plus forward and backward passes optimized for CUDA.

Why is it gaining traction?

It stands out with Cartesian product routing over massive expert spaces, expert-centric scheduling for locality-aware expert placement, and fused kernels that speed up inference and training in memory-bound regimes. Developers notice lower latency on personal machines via sparsity-aware dispatching and adaptive load balancing, without sacrificing routing quality. Benchmarks via pytest show it handles 4096 experts and top-16 selection efficiently, beating coarse-grained alternatives.

Who should use this?

ML engineers fine-tuning or serving MoE models like Mixtral on single GPUs, researchers scaling experts for efficient deep learning experiments, and teams optimizing communication-efficient MoE training with disaggregated parallelism. Ideal for efficient MoE inference where you balance activated experts, not just tokens.

Verdict

Promising early-stage efficient MoE implementation with solid docs, quickstart code, and pytest benchmarks, but only 65 stars and 1.0% credibility score signal it's fresh—test thoroughly before production. Grab it if you're pushing MoE scale on consumer hardware.

(198 words)