hec-ovi

hec-ovi / vllm-awq4-qwen

Public

vLLM Qwen 3.6-27B (AWQ-INT4) + DFlash speculative decoding on AMD Strix Halo (gfx1151 iGPU, 128 GB UMA, ROCm 7.13). 24.8 t/s single-stream, vision, tool calling, 256K context, OpenAI-compatible, Docker. Matches DGX Spark FP8+DFlash+MTP at a third of the cost. No CUDA.

27b amd-strix-halo awq dflash docker
11
0
89% credibility
Found Apr 27, 2026 at 11 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

A setup to run a high-performance, quantized Qwen language model with speed boosts on specific AMD integrated GPUs.

How It Works

1
🖥️ Discover fast AI for your AMD laptop

You hear about a setup that lets a powerful AI chat super quickly on special AMD computers.

2
✅ Check your computer's graphics

See if your laptop has the right AMD graphics chip and enough memory to run it smoothly.

3
⚙️ Tweak startup settings

Make a couple simple changes in your computer's boot options to unlock full power.

4
📥 Download the AI brains

Grab the smart thinking files from the internet – it feels exciting as they arrive.

5
🚀 Launch your AI helper

Follow the guide to build and start everything with easy steps.

6
💬 Start chatting super fast

Type questions in a fun command tool and watch answers appear in seconds.

🎉 Create amazing things quickly

Generate code, describe images, or solve puzzles at blazing speeds on your own machine.

Sign up to see the full architecture

5 more

Sign Up Free

Star Growth

See how this repo grew from 11 to 11 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is vllm-awq4-qwen?

This GitHub repository delivers a Dockerized vLLM setup for Qwen 3.6-27B AWQ-INT4 with DFlash speculative decoding on AMD Strix Halo iGPUs, pushing 24.8 t/s single-stream on 128GB UMA via ROCm. Developers get OpenAI-compatible endpoints like /v1/chat/completions and /v1/responses supporting vision inputs, tool calling, and 256K context in Python. It sidesteps CUDA entirely for local inference on gfx1151 hardware.

Why is it gaining traction?

It crushes NVIDIA DGX Spark benchmarks at a third the cost, with +340% speedup over baselines on reasoning tasks—check vllm GitHub issues for upstream parallels like DFlash on Qwen3.5. The vllm GitHub Docker image includes glados.py CLI for instant REPL testing, full benches, and tunable env vars for N=8 speculation, drawing qwen 3 coder vllm fans avoiding vllm GitHub releases friction.

Who should use this?

AMD Strix Halo owners building vllm qwen3 next or omni apps with vision/tools on 256K prompts. AI devs prototyping qwen3-coder-next agents or long-context synthesis without NVIDIA, especially those hitting vllm GitHub issue stalls on ROCm.

Verdict

Grab it if you have Strix Halo—detailed README, benches, and docker-compose make setup straightforward despite 11 stars and 0.9% credibility score. Early DFlash quirks noted honestly; mature enough for single-stream workloads, less so for high concurrency.

(198 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.