songmzhang

songmzhang / KDFlow

Public

A user-friendly and efficient knowledge distillation framework for LLMs.

16
1
100% credibility
Found Mar 04, 2026 at 16 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

KDFlow is a user-friendly framework that transfers capabilities from large teacher AI models to smaller student models through efficient distillation techniques.

How It Works

1
🔍 Discover KDFlow

You learn about a helpful tool that teaches small AI models the smarts of bigger ones, making them faster and just as clever.

2
🤖 Pick your AI teachers

Choose a powerful big AI as the teacher and a smaller one as the student that will learn its skills.

3
📚 Gather conversation examples

Collect simple chat examples or prompts to help the small AI practice real talks.

4
Pick learning style
📖
Ready examples

Use existing chats for steady, reliable learning.

✨
Live practice

Let the small AI create responses on the fly for dynamic improvement.

5
🚀 Start the lesson

With one easy command, launch the training and watch the small AI absorb knowledge from the big one.

6
📊 Track the progress

Check simple updates to see the small AI getting smarter step by step.

🎉 Smart small AI ready

Your compact AI is now powerful and ready to chat quickly anywhere!

Sign up to see the full architecture

5 more

Sign Up Free

Star Growth

See how this repo grew from 16 to 16 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is KDFlow?

KDFlow is a Python framework for knowledge distillation on LLMs, letting you transfer capabilities from massive teacher models to compact students via simple CLI commands. Run off-policy KD on static datasets, on-policy with real-time rollouts, or plain SFT—all with one-liners like `python -m kdflow.cli.train_kd_off_policy --student tinyllama --teacher llama-70b --train_dataset_path data.json`. It handles cross-tokenizer distillation (Llama to Qwen) and GPU co-location, squeezing big teachers onto the same hardware as students.

Why is it gaining traction?

Claims 1.4x-6x speedups over other KD tools via SGLang teacher inference, FSDP2 training, and zero-copy hidden states—users see faster runs without extra clusters. Pluggable losses (KL, JS, skewed variants) and algorithms (Vanilla KD, DSKD) plus WandB logging make experiments quick to tweak. Colocate mode maximizes utilization on 8-GPU nodes, a real win for constrained setups.

Who should use this?

LLM engineers compressing 70B+ teachers to 7B students on single nodes. Teams doing alignment via on-policy rollouts from student-generated data. Researchers testing custom KD losses without rebuilding pipelines from scratch.

Verdict

Worth a spin for efficient LLM distillation if you have Ray/SGLang setups—solid README, CLI, and arXiv paper make it approachable despite 13 stars and 1.0% credibility score. Early-stage (v0.1), so test thoroughly before prod; fork-friendly MIT license.

(198 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.