quantumaikr / TurboQuant.cpp

What is TurboQuant.cpp?

TurboQuant.cpp is a standalone C inference engine that slashes KV cache memory in LLMs by up to 7x using 1-bit keys and Q4 values, letting you run 35B MoE models on 16GB laptops with 32K contexts. Built from scratch with zero dependencies, it loads GGUF files directly—no conversion needed—and delivers outputs identical to baselines on prompts like "The capital of France is Paris." Run it via `./tq_run model.gguf -p "Hello" -k turbo_kv_1b -v q4` for instant compression.

Why is it gaining traction?

Unlike llama.cpp's uniform quant, TurboQuant.cpp uses provably unbiased methods for near-zero perplexity loss (+0.03% on Gemma 3 4B), plus rare AMD GPU support via Vulkan and ROCm alongside CUDA and Metal. Developers dig the 4.9x memory win at long contexts without speed hits, plus tools like `--ppl` for quality checks and per-layer bit recommendations. It's cross-platform from Mac M3 to NVIDIA/AMD rigs.

Who should use this?

Inference engineers deploying long-context chatbots on edge hardware like laptops or AMD GPUs, where KV bloat kills 128K+ prompts. MoE model runners (Qwen3.5-35B) needing 10GB models in 5GB RSS, or anyone benchmarking KV quant before production.

Verdict

Grab it if low-mem LLM serving is your bottleneck—85 stars and 1.0% credibility score signal early days, but 31/31 tests pass cleanly with ASan/UBSan. Maturity lags on docs and examples, so prototype first.

(198 words)