jundot / omlx

What is omlx?

oMLX runs LLM inference on Apple Silicon Macs with continuous batching for concurrent requests and SSD caching to persist KV cache across restarts, letting you reuse long contexts without recomputing. Launch via macOS menu bar app or CLI (`omlx serve --model-dir ~/models`), get OpenAI/Anthropic-compatible APIs at localhost:8000/v1, plus admin dashboard for downloading models from Hugging Face, chatting, and monitoring. Built in Python atop MLX, it handles LLMs, embeddings, and rerankers in one server.

Why is it gaining traction?

Stands out in llm github projects with dead-simple Mac menu bar control—no terminal babysitting—plus paged SSD caching that slashes llm inference time on repeated prompts, beating basic mlx-lm setups in llm inference speed benchmarks. Multi-model LRU swapping and pinning keep daily drivers loaded while auto-evicting others, with Claude Code tweaks for local tool use. Developers dig the drop-in llm inference api for llm github integration and llm github copilot alternatives.

Who should use this?

Apple Silicon devs building local llm github local apps or testing llm github download models without cloud costs. AI tinkerers needing llm-inference server for embeddings/reranking in RAG pipelines. Mac power users eyeing llm inference hardware like M-series chips for fast llm inference on cpu fallbacks or llm inference speed in coding workflows.

Verdict

Promising alpha for Mac-local LLMs (45 stars, solid README), but 1.0% credibility score flags early risks—test thoroughly before prod. Grab if you're on Apple Silicon chasing omlx 2025 vibes in llm github repository simonw-style simplicity.

(198 words)