helgklaizar

helgklaizar / turboquant_mlx

Public

Extreme KV Cache Compression (1-3 bit) for LLMs natively on Apple Silicon (MLX). Features TurboQuant, asymmetric PolarQuant caching, and OpenAI server compatibility.

apple-silicon kv-cache llm mlx quantization
19
1
100% credibility
Found Mar 28, 2026 at 19 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

TurboQuant-MLX compresses the memory cache for language models on Apple Silicon to enable longer contexts and larger models with minimal accuracy loss.

How It Works

1
🔍 Discover TurboQuant

You hear about a handy tool that lets big AI chatbots run smoothly on your Mac without eating up all the memory.

2
📥 Get It Ready

Download the tool and set it up on your Apple computer in a few simple steps.

3
⚡ Supercharge Your AI

Pick your favorite AI model and flip on the memory-saving switch to make it use way less space while keeping responses sharp.

4
💬 Start Chatting

Ask the AI long questions or have extended conversations, and watch it handle huge amounts of text without slowing down.

5
Share or Keep Private
🏠
Local Fun

Keep generating ideas and stories right on your Mac.

🔌
Chat Server

Launch a server so you can chat through web tools or apps.

🎉 Memory Magic Unlocked

Your Mac now runs massive AI sessions effortlessly, saving gigabytes of space and keeping everything fast and accurate.

Sign up to see the full architecture

4 more

Sign Up Free

Star Growth

See how this repo grew from 19 to 19 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is turboquant_mlx?

TurboQuant-MLX delivers extreme KV cache compression down to 1-3 bits for LLMs running natively on Apple Silicon via the MLX framework. It cuts memory use by up to 81%—like shrinking a 1024MB FP16 cache at 64K tokens to 192MB—while preserving accuracy through asymmetric key-value handling and an uncompressed attention sink for prompts. Python-based, it patches mlx-lm models in two lines and offers an OpenAI-compatible server for tools like Chatbox.

Why is it gaining traction?

It hooks seamlessly into mlx-lm without model tweaks, enabling 128K+ contexts on M-series chips where stock setups OOM. Benchmarks back 5x savings on Llama 3 and Mistral Nemo, plus EXO cluster support for distributed Apple inference. Devs dig the dynamic chunking that drops VRAM for long generations.

Who should use this?

ML engineers on MacBooks or Mac Minis running local LLMs like Llama 3 8B for RAG apps with huge contexts. Inference server builders needing an extreme cache server on Apple hardware, or EXO node operators squeezing more models across devices.

Verdict

Early project at 19 stars and 1.0% credibility score—solid README and needle-in-haystack tests, but spotty model compatibility means benchmark your stack first. Grab it if you're all-in on Apple Silicon inference; skip for production without more battle-testing.

(178 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.