Alberto-Codes

Alberto-Codes / turboquant-vllm

Public

TurboQuant KV cache compression for consumer GPUs — 3.76x compression validated on Molmo2 + RTX 4090

alberto-codes.github.ioturboquant-vllm
19
0
100% credibility
Found Apr 01, 2026 at 19 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

TurboQuant-vLLM is a drop-in plugin for vLLM that compresses the key-value cache by up to 3.76x to reduce memory usage during AI inference while maintaining near-identical output quality.

How It Works

1
😩 Hit memory limits

You're running an AI assistant for chat or video analysis, but it crashes on long conversations or videos because your computer's graphics memory fills up.

2
🔍 Discover TurboQuant

You find turboquant-vllm, a simple add-on that squeezes your assistant's memory use by up to 4 times without losing smarts.

3
📦 Install easily

Run one quick command to add it to your setup, no complicated steps.

4
🚀 Launch with one change

Add a single flag when starting your AI server, and it automatically uses less memory.

5
📹 Handle bigger inputs

Now process longer videos or chats that used to crash, seeing the same high-quality responses.

🎉 Save memory, keep quality

Your assistant runs smoother on everyday hardware, handling more at once with no noticeable difference in results.

Sign up to see the full architecture

4 more

Sign Up Free

Star Growth

See how this repo grew from 19 to 19 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is turboquant-vllm?

Turboquant-vllm is a Python drop-in plugin for vLLM that compresses KV caches using TurboQuant, shrinking them 3.76x—from 256 to 68 bytes per token/head—while keeping output quality near-identical. Enable it with one CLI flag on models like Molmo2: `vllm serve allenai/Molmo2-4B --attention-backend CUSTOM`. Validated on RTX 4090 consumer GPUs, it tackles memory limits for long-context inference without code changes.

Why is it gaining traction?

It delivers real memory wins on consumer GPUs like the 4090, with fused Triton kernels cutting decode overhead to 1.78x via incremental dequantization. Unlike generic quantizers, TurboQuant's rotation + Lloyd-Max packs 4-bit indices efficiently, beating baselines in benchmarks (435 MiB vs 1.6 GiB for 11K tokens). The Hugging Face cache wrapper adds broad compatibility.

Who should use this?

ML engineers running vLLM on RTX consumer GPUs for vision-language models like Molmo2, especially with video or long chats hitting KV cache walls. Ideal for local inference setups needing 4K+ contexts without upgrading to A100s.

Verdict

Grab it if you're memory-constrained on consumer GPUs—3.76x cache compression is a game-changer for vLLM. At 1.0% credibility and 19 stars it's early days, but strong docs, 95% test coverage, and PyPI packaging make it low-risk to prototype.

(198 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.