SRSWTI / bodega-inference-engine

What is bodega-inference-engine?

Bodega Inference Engine runs local LLMs, multimodal vision models, and soon image generation/editing on Apple Silicon Macs via an OpenAI-compatible API server. Developers get a multi-model registry that dynamically loads/unloads models into isolated processes, supporting chat completions, streaming, JSON schema outputs, tool calls, and RAG for PDFs—all without server restarts. Built in Python with MLX, it prioritizes memory efficiency and high throughput on unified memory hardware.

Why is it gaining traction?

It delivers the fastest runtime for Apple Silicon in benchmarks, hitting 600+ tokens/second via continuous batching and speculative decoding, outpacing tools like LM Studio on concurrent loads. The dynamic model registry lets you swap models mid-session via simple curl or Python requests, with built-in health checks showing real-time RAM/GPU usage. For Apple users, it's a drop-in OpenAI server that handles vision inputs and structured outputs without leaks or restarts.

Who should use this?

Apple Silicon Mac devs building local AI apps, like agentic tools or high-throughput chat services, who need multi-model routing without downtime. Ideal for indie hackers prototyping RAG pipelines or vision apps on laptops, or teams running inference servers for internal tools. Skip if you're on non-Apple hardware or need cloud-scale deployment.

Verdict

Promising for Apple Silicon local inference with solid OpenAI compatibility and throughput wins, but at 19 stars and 1.0% credibility, it's early-stage—docs are benchmark-heavy but light on edge cases. Try the interactive setup script if you're on M-series; otherwise, stick to mature alternatives until it grows.

(198 words)