PKU-YuanGroup / TIDE

What is TIDE?

TIDE is a Python framework for distilling massive diffusion large language models across different architectures, attention mechanisms, and tokenizers—like squeezing 8B dense or 16B MoE teachers into a 0.6B student. It provides two ready pipelines: one for cross-tokenizer transfers using chunk-level alignment, another for shared-tokenizer setups, complete with scripts for preprocessing data, running distillation, and evaluating on eight benchmarks including HumanEval and GSM8K. Users get pretrained checkpoints and datasets on Hugging Face, plus one-click training on 8 GPUs.

Why is it gaining traction?

Unlike prior diffusion distillation stuck in single architectures, TIDE handles real-world mismatches via modular tricks like progressive scheduling and complementary teacher signals, delivering +1.53 average benchmark gains and crushing code tasks (+16 on HumanEval). It slashes peak memory 22x and speeds inference 5x on H100s, making tiny models viable on consumer hardware. Solid arXiv paper, project page, and bash scripts lower the entry barrier for experiments.

Who should use this?

ML researchers tuning diffusion LLMs for code generation or math reasoning, where distilled models outperform AR baselines. Teams at startups or labs deploying edge-friendly LLMs need its cross-arch flexibility. Academic groups replicating TIDE results or extending to tide model github variants like time series forecasting.

Verdict

Grab it for diffusion distillation prototypes—prebuilt pipelines and HF assets make setup fast despite 36 stars and 1.0% credibility score signaling early maturity. Docs and scripts are crisp, but expect tweaks for production; strong paper backs the gains.

(198 words)