johnmai-dev / ANE-LM

What is ANE-LM?

ANE-LM runs LLM inference directly on Apple Neural Engine using C++ and private framework APIs, targeting Qwen3.5 dense text models on Apple Silicon. It accelerates local llm github local runs, delivering high llm inference speed without GPU reliance—ideal for beating llm inference on cpu limits. The CLI offers generate for single prompts, chat for interactive sessions, and convert for BF16-to-FP16 model prep.

Why is it gaining traction?

It crushes llm inference benchmark results on ANE hardware, outpacing CPU or Metal for llm inference time in compact models, with persistent compile caching for repeat loads. Stands out in github llm-resources and llm github projects as a lightweight llm-inference server alternative for on-device use. Devs hook on the raw tps gains for llm github repo experiments, no cloud needed.

Who should use this?

Apple Silicon devs embedding llm github integration in macOS/iOS apps, researchers tweaking llm inference hardware calculator for ANE vs CPU, or tinkerers in llm github course projects chasing local llm inference api perf. Suits offline chatbots or llm github copilot-style tools where battery life matters.

Verdict

Worth testing for ANE-maxed llm inference service on M-chips—CLI simplicity shines despite 42 stars signaling early days. 0.7% credibility score flags private API risks (future macOS breakage possible), but clean README and benchmarks make it a pragmatic pick for llm github simonw fans eyeing local speedups.

(198 words)