sail-sg / Stable-RL

What is Stable-RL?

Stable-RL rethinks the trust region in PPO for LLM reinforcement learning, replacing volatile probability ratios with stable metrics like Total Variation divergence to prevent training collapse from inference mismatches. Developers get a drop-in upgrade for RLHF/RLAIF pipelines, supporting FSDP and Megatron backends with vLLM/SGLang inference, Docker images for quick setup, and scripts to convert Hugging Face models to optimized formats. Built in Python, it delivers reliable training on math/reasoning tasks using Qwen MoE models without heuristic clipping hacks.

Why is it gaining traction?

Unlike standard PPO or GRPO, which over-penalize rare tokens and allow destabilizing updates on common ones, Stable-RL's DPPO variants enforce principled policy divergence bounds for faster convergence and higher scores on AIME24 benchmarks. Users notice immediate stability gains—no more growing mismatches or model crashes—plus easy integration via config tweaks and shell scripts for async off-policy setups. It's a practical fix for LLM RL pitfalls, backed by arXiv preprint visuals showing real volatility issues.

Who should use this?

RL engineers fine-tuning LLMs for math or tool-use tasks, especially with 7B-30B MoE models like Qwen, who hit PPO instability in colocated trainer-rollout flows. Ideal for teams using Ray for disaggregated training, needing elastic rollouts on heterogeneous hardware without rebuilding comm groups every step.

Verdict

Try it if you're rethinking trust regions in LLM RL—simple swaps yield stable baselines outperforming GRPO, but low 1.0% credibility from 31 stars signals early-stage maturity; pair with the paper for configs and expect docs/tests to evolve. Solid for experiments, not production yet.

(198 words)