raketenkater / llm-server

What is llm-server?

This GitHub llm server is a shell script that launches llama.cpp or ik_llama.cpp inference servers with zero manual config. It auto-detects NVIDIA GPUs via nvidia-smi, optimizes MoE expert placement across mixed VRAM setups, enables graph splitting for multi-GPU scaling, and includes crash recovery plus a built-in Hugging Face GGUF downloader that picks the best quantization for your VRAM and RAM. Run llm-server model.gguf to start, or add --download for llm server build from any repo.

Why is it gaining traction?

Unlike bare llama.cpp flags, it fixes library paths automatically, switches backends to avoid crashes on fused tensors, and benchmarks tok/s before exiting. The downloader scans repos, recommends quants like Q4_K_M for llm server hardware balance, and supports llm server linux with curl health checks. Developers skip hours of tuning for instant multi-GPU llm proxy server github performance.

Who should use this?

NVIDIA users running local llm inference server github on desktops or servers, like AI researchers testing Qwen models or devs building llm mcp server github prototypes. Perfect for those handling llm server cost on varied hardware without deep CUDA tweaks, or benchmarking llm server hosting setups.

Verdict

Solid docs and core smarts make it usable now, but 15 stars and 1.0% credibility score signal early maturity—expect edge cases on non-standard rigs. Try it for llm server kaufen experiments if you're in llama.cpp; contribute to boost this niche github llm server.

(198 words)