sharpner

sharpner / turboquant-mlx

Public

A proof of concept of googles TurboQuant Paper https://arxiv.org/abs/2504.19874

21
5
100% credibility
Found Apr 03, 2026 at 19 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

This project squeezes the memory footprint of AI language models on Apple Silicon to enable faster generation and longer contexts with minimal quality loss.

How It Works

1
🔍 Discover faster AI chats

You hear about a clever way to make AI conversations on your Mac run much faster and use less memory, even for really long talks.

2
📱 Get ready on your Mac

You grab the simple free tools needed, just a quick download for Apple computers.

3
🤖 Pick an AI buddy

You choose a smart AI model like a chatty assistant that's already tuned for your Mac.

4
Choose your vibe
⚡
Go speedy

Pick the fast path for quick replies in long chats.

⭐
Go premium

Choose top quality for the sharpest, most accurate responses.

5
💬 Start chatting

You type a question and watch the AI respond right away.

6
🚀 Feel the boost

Responses fly out super fast, using way less memory, even after thousands of words!

🎉 Chat forever

Now you can have endless, smooth AI conversations without any slowdowns.

Sign up to see the full architecture

5 more

Sign Up Free

Star Growth

See how this repo grew from 19 to 21 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is turboquant-mlx?

This Python repo is a proof-of-concept reproduction of Google's TurboQuant paper, bringing extreme KV-cache compression to LLMs on Apple Silicon via MLX. It shrinks caches 3.6-5.5x—from 969MB to as low as 177MB at 8K tokens—while keeping perplexity near fp16 baselines, tested on Llama 3, Mistral, and Gemma models from mlx-community. Users get drop-in caches for mlx-lm, plus CLI benchmarks like `python benchmark.py` for speed/quality and `run_llm.py` for instant demos.

Why is it gaining traction?

It delivers hardware-accelerated compression matching MLX's native 4-bit quant speed, with V3 paths hitting paper-correct quality via Lloyd-Max codebooks—beating affine quant at low bits. Benchmarks prove real wins: 3-bit beats fp16 perplexity on Gemma (D=256), and throughput holds at long contexts where fp16 chokes. As a concept proof github repo, it validates Google's claims on M4 Max unified memory, sparking interest in every proof github for Apple ML.

Who should use this?

ML engineers on M-series Macs running mlx-lm for Llama/Mistral inference, especially long-context RAG or agents hitting cache limits. Devs prototyping proof-of-concept studies or github proof of work for academic papers on quantization.

Verdict

Grab it for a working proof of concept deutsch-style repro of googles TurboQuant—docs, benchmarks, and tests are pro-level despite 18 stars and 1.0% credibility score. Early maturity means fork and harden for prod, but it's already usable for zero-knowledge-proof github experiments.

(198 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.