val1813 / kaiwu

What is kaiwu?

Kaiwu is a Go-based tool for deploying local LLMs with zero manual tuning—it probes your NVIDIA GPU, CPU, and RAM, then auto-selects optimal context length, KV cache type, batch sizes, and threads via a one-time warmup benchmark. Run `kaiwu run Qwen3-30B-A3B` to download and launch a model, getting an OpenAI-compatible API at localhost:11435 for tools like Continue or Cursor. Subsequent starts skip warmup, launching in 2 seconds using cached configs.

Why is it gaining traction?

Unlike Ollama or LM Studio, which require tweaking params for speed or OOM avoidance, kaiwu delivers 7x faster inference on 30B MoE models with 65% less VRAM on 8GB GPUs, plus auto 64K+ contexts where others cap at 8K. Benchmarks show real gains like 115 tok/s on dual 4090s with tensor splitting, and MoE expert offload to CPU. Developers love the "install and run" simplicity in Go, with IDE injection and status monitoring.

Who should use this?

Local AI tinkerers with NVIDIA cards (4GB+ VRAM) who hate config guesswork, especially for coding in Cursor/Continue or testing Qwen/Gemma models. MoE fans on modest hardware (8GB GPU + 16GB RAM) get the biggest wins, running 30B+ models smoothly. Skip if you're CPU-only or prefer Docker-heavy setups.

Verdict

Try kaiwu if you run GGUF models daily—its auto-optimization hooks deliver on promises, despite 47 stars and 1.0% credibility signaling early days. Solid docs and benchmarks make it worth the leap for Go-savvy devs; watch for multi-GPU polish.