poetrywanderer / RL-Projects

What is RL-Projects?

RL-Projects is a personal research repository exploring reinforcement learning post-training across three modalities: text, vision-language, and image generation. The author ran systematic experiments comparing GRPO (Group Relative Policy Optimization) against OPD (On-Policy Distillation) on the same tasks, using Qwen2.5 models and Stable Diffusion 3.5. The work spans four projects: R1-RL reproduces R1-Zero emergent reasoning on arithmetic (Countdown), Geo3K-VL-RL applies GRPO to geometry diagram reasoning, Geo3K-VL-OPD tests whether dense teacher feedback outperforms sparse rewards, and Diffusion-Flow-RL trains SD3.5 to render readable text using Flow-GRPO. All experiments run on 4-8 NVIDIA L40S GPUs with documented training scripts and evaluation pipelines.

Why is it gaining traction?

The repository stands out because it answers practical questions that papers often gloss over. The author quantifies exactly how much data efficiency OPD gains over GRPO (12x less data, 16x more efficient per sequence) and demonstrates that verifiable rewards crush learned rewards under identical conditions (77.8pp improvement vs 2.3% in the same runtime). The two-phase finding is particularly valuable: OPD aligns to teacher capability, then GRPO explores beyond it, breaking both ceilings. The work also surfaces real failure modes—group saturation, length bias, negative transfer on language tasks—that developers implementing GRPO will encounter. The documentation is unusually thorough for an experimental repo, with case studies showing exactly what behavior changes after training.

Who should use this?

ML engineers implementing RL post-training pipelines who want to understand the tradeoffs between GRPO and OPD before committing to a method. Researchers studying emergent reasoning or multimodal alignment will find the cross-project takeaways useful, particularly the signal density principle and the two-phase training strategy. Developers working with VLMs on geometry or diagram tasks should examine the Geo3K case studies to understand what visual grounding RL actually teaches. The diffusion experiments are most relevant for anyone trying to improve text rendering in generated images.

Verdict

This is a credible research-grade repository despite the low star count (44 stars, 0.8999% credibility score). The methodology is rigorous, results are quantified, and failure modes are documented honestly. However, it is experimental personal research, not a production library—expect to read the README carefully and adapt scripts for your own environment. The value lies in the insights and experimental design patterns, not in importing a polished API. If you're serious about RL post-training, the OPD vs GRPO comparison alone justifies spending an afternoon with these docs.