xiaguan

xiaguan / pegainfer

Public

Pure Rust + CUDA LLM inference engine

146
12
100% credibility
Found Feb 17, 2026 at 29 stars 5x -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Rust
AI Summary

pegainfer is a from-scratch inference engine that runs the Qwen3-4B language model on CUDA GPUs, providing an OpenAI-compatible web service for text completions at high speeds.

How It Works

1
🔍 Discover pegainfer

You stumble upon this project while searching for a simple way to run smart AI conversations on your own computer using its graphics power.

2
📥 Download the AI model

You grab the free model files from Hugging Face and save them in a folder called models on your computer.

3
🛠️ Prepare and launch

You follow the easy setup guide to build and start your personal AI helper with one command, watching it load onto your graphics card.

4
🚀 Your AI comes alive

Everything starts up smoothly, and your computer now hosts a fast AI ready to answer questions over the web.

5
💬 Send your first message

You type a simple question like 'What is the capital of France?' and send it to your local AI using a web tool or command.

✨ Get instant smart replies

You receive quick, accurate responses from your own AI, feeling the speed of about 70 words per second right on your machine.

Sign up to see the full architecture

4 more

Sign Up Free

Star Growth

See how this repo grew from 29 to 146 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is pegainfer?

Pegainfer is a pure Rust CUDA engine for running LLM inference on a single GPU, targeting models like Qwen3-4B at around 70 tokens per second. It spins up an OpenAI-compatible HTTP server with a /v1/completions endpoint, so you can curl prompts and get completions without PyTorch or any frameworks—just download safetensors weights and run cargo build --release followed by cargo run. Developers get a lightweight, from-scratch inference stack focused on BF16 precision and KV caching for autoregressive generation.

Why is it gaining traction?

In a sea of Python-heavy tools like vLLM or TensorRT-LLM, pegainfer stands out as a pure Rust alternative with hand-tuned CUDA kernels, delivering solid single-user throughput without external deps. The hook is its transparency: profiling traces in Chrome format let you dissect every kernel launch, appealing to those inspired by pure Rust projects like databases or GUIs. It prioritizes simplicity over batching or quantization, making it dead simple to hack on GPU ops directly.

Who should use this?

Rust systems programmers experimenting with custom LLM serving, kernel hackers porting ops to CUDA, or researchers validating inference stacks against references. Ideal for solo devs prototyping on RTX cards who want a pure Rustico-style engine without framework bloat, or educators teaching transformer guts via its quickstart and API.

Verdict

With 25 stars and a 1.0% credibility score, pegainfer is an early-stage learning tool rather than production-ready—docs are solid via README, but expect tweaks for stability. Try it if you're into pure Rust CUDA engines; fork and contribute to push toward multi-GPU or quantization.

(198 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.