ZunhaiSu

ZunhaiSu / OScaR-KV-Quant

Public

🏆 OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond — redefining the accuracy-efficiency Pareto front for X-LLMs KV quantization.

17
2
100% credibility
Found May 21, 2026 at 17 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
C++
AI Summary

OScaR is a research project from university researchers that makes AI assistants use much less memory. It works by compressing the internal memory cache that AI models use during conversations. The tool can reduce memory usage by 5x while maintaining nearly the same quality, and it makes responses generate 3x faster. It works with various types of AI models including text-only, image-understanding, and audio-capable models. The project comes from researchers at Tsinghua University, HKU, Meituan, and University of Edinburgh, and is published as an academic paper on arXiv.

How It Works

1
🔍 You discover a memory problem

You want to run an AI assistant that can handle very long conversations, but the model uses too much memory and runs out of GPU memory.

2
📄 You find OScaR online

You discover a research project from university researchers that claims to reduce AI memory usage by 5x while keeping quality intact.

3
⚙️ You set up the project

You download the code and install it on your computer following the clear instructions in the documentation.

4
🎯 You connect your AI model

You point the tool to your AI model (like Qwen3) and tell it how much you want to compress the memory (2-bit or 4-bit precision).

5
🚀 Your assistant comes to life

With one click, your AI assistant launches with the compressed memory system, ready to handle long conversations.

✨ You get fast, memory-efficient AI

Your AI assistant now uses 5x less memory, generates responses 3x faster, and handles long documents without running out of GPU memory.

Sign up to see the full architecture

4 more

Sign Up Free

Star Growth

See how this repo grew from 17 to 17 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is OScaR-KV-Quant?

OScaR is a KV cache quantization library for large language models that squeezes memory usage down to 2-bit precision without the accuracy loss you'd expect. Built in C++ with Python bindings, it attacks the Token Norm Imbalance problem that makes per-channel quantization fail at extreme compression levels. The solution: two lightweight operations called Canalized Rotation and Omni-Token Scaling that require zero training or calibration data. It plugs into the attention mechanism via custom CUDA kernels and works across text-only LLMs, multi-modal models, and the emerging class of omni-modal systems handling audio alongside text and images.

Why is it gaining traction?

The numbers are hard to ignore: 3x faster decoding, 5.3x less memory, 4.1x higher throughput compared to standard BF16 FlashDecoding. More importantly, OScaR's INT2 results sometimes beat the full 16-bit baseline on benchmarks like LongBench-E and MMAU-Pro. Unlike competing approaches like KIVI or QuaRot, there's no calibration step, no dataset to prepare, no fine-tuning overhead. You point it at your model and it works. For teams hitting GPU memory walls on long-context inference, this is the kind of unlock that changes what's possible.

Who should use this?

Inference engineers optimizing LLM deployments for memory-constrained environments will get the most value. Multi-modal teams running Vision-Language models should pay attention since OScaR handles those outlier patterns better than text-only methods. Academic researchers benchmarking quantization techniques will appreciate the clean evaluation suite. This is not yet for production serving systems—the team explicitly flags that vLLM and SGLang integration are under development.

Verdict

The research foundation is solid (Tsinghua, HKU, Meituan) and the benchmark results are compelling, but with 17 stars and a 1.0% credibility score, this is bleeding-edge territory. Installation requires compiling CUDA kernels with specific CUTLASS dependencies, and you're tied to PyTorch 2.6 with CUDA 12.4. Worth evaluating in a research context today; hold off on production until official vLLM integration ships.

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.