Aiziyou918 / radixInfer

What is radixInfer?

radixInfer is a Python-based, layered LLM serving system that delivers a fully runnable end-to-end control plane for handling API requests, transport, runtime scheduling, cache management, and engine execution. It exposes OpenAI-compatible endpoints like /v1/chat/completions with SSE streaming over FastAPI, plus an interactive shell for local testing. Developers get a clean separation of concerns for clarity, extensibility, and performance experimentation without the black-box feel of production engines.

Why is it gaining traction?

Its modular layers make it dead simple to swap components—like attention backends or prefix caches—while matching vLLM and SGLang throughput in benchmarks (e.g., 262 tok/s at concurrency 16). Prefix cache hits slash TTFT from 3.7s to 63ms on shared prompts, and tensor parallelism works out of the box. The included benchmark suite lets you pit it against alternatives on your hardware, hooking tinkerers who want control over execution without rebuilding from scratch.

Who should use this?

LLM researchers tweaking schedulers, cache policies, or custom engines for academic papers or prototypes. Serving engineers at startups prototyping high-throughput APIs before scaling to vLLM. Python devs diving into LLM inference internals via the shell and CLI flags like --tp-size or --num-pages.

Verdict

Grab it for experimentation if you're okay with alpha maturity (12 stars, 1.0% credibility)—docs are solid, tests pass, and benchmarks prove viability. Skip for production until FlashAttention lands and edge cases stabilize.

(198 words)