cloudflareresearch / unweight-kernels

What is unweight-kernels?

Unweight-kernels delivers CUDA kernels for lossless compression of BF16 MLP weights in LLMs, slashing model size by about 20% overall through techniques like Huffman-coding exponents while keeping full numerical fidelity. It provides encoding, decoding, transcoding, and reconstructive matmul pipelines optimized for NVIDIA Hopper GPUs like H100 or H200, integrating with ThunderKittens for fast inference without extra HBM trips. Build it with a simple make command to get a static library ready for your LLM serving stack.

Why is it gaining traction?

Unlike generic lossless compression algorithms that bloat decode times, unweight-kernels targets BF16 entropy patterns in LLM weights for 30% MLP compression with seamless Hopper WGMMA integration and autotuned pipelines per layer. Developers notice end-to-end throughput gains on models like Llama 3.1 8B, plus hard/easy layer scheduling for better preprocess overlap—no accuracy loss, just leaner memory and faster loads. It stands out in the github lossless compression space by focusing purely on inference acceleration.

Who should use this?

LLM inference engineers deploying 7B+ models on H100/H200 clusters, especially those hitting memory walls with BF16 weights. Teams optimizing Cloudflare-style serving or custom vLLM forks will value the reconstructive matmul for persistent kernels. Hopper-only shops experimenting with weight compression techniques before scaling to production.

Verdict

Promising for Hopper-bound LLM inference but too early at 36 stars and 1.0% credibility—docs are thin, no benchmarks beyond the report, and zero tests mean you'll audit it yourself. Integrate if you're on compatible hardware and need BF16 compression now; otherwise, watch for maturity.

(198 words)