redai-infra / PIPO

What is PIPO?

PIPO (Pair-In, Pair-Out) is a Python implementation of a novel LLM architecture that speeds up inference by compressing consecutive token pairs during processing. The core idea: instead of processing tokens one-by-one, it folds two input tokens into one latent representation, then unfolds that hidden state to predict additional output tokens. The system uses a lightweight confidence head to decide which draft tokens to accept, eliminating the expensive verification step required by traditional speculative decoding. Built on top of SGLang and Qwen3.5 backbones (4B and 9B variants), it includes complete evaluation pipelines for math reasoning (AIME 2025, GPQA), code generation (LiveCodeBench, Codeforces), and long-context understanding (LongBench-v2).

Why is it gaining traction?

The key innovation is unifying input-side latent compression with output-side multi-token prediction—two lines of work that have been developed independently until now. The results are concrete: up to 2.64x faster first-token latency and 2.07x faster per-token latency on challenging benchmarks, with a +7.15 point improvement in pass@4 on math problems. The confidence head replaces speculative decoding's expensive verifier with a simple threshold check, making the speedup practical rather than theoretical. Researchers working on long chain-of-thought reasoning will appreciate that this approach addresses the dominant inference cost without requiring architectural changes to existing models.

Who should use this?

LLM researchers focused on inference efficiency and latency optimization will find the most value here—particularly those working on math or coding tasks where long reasoning chains are common. If you're evaluating whether to deploy Qwen3.5 in production and need better throughput, this provides a solid reference implementation. Academic researchers exploring speculative decoding variants can build on the confidence head concept without implementing the full verification pass. Teams already using SGLang for serving will have an easier integration path. However, if you need a production-ready solution with comprehensive documentation and community support, the early-stage nature of this project means you'll be doing more of the heavy lifting yourself.

Verdict

PIPO addresses a real problem with a thoughtful architecture, and the published results are compelling. With a credibility score of 0.8999999761581421% and only 15 stars, this is a research-grade implementation rather than a production tool—expect to read the code carefully and contribute your own fixes. The documentation is functional but sparse, and the Qwen3.5-only constraint limits applicability until the team adds more backbone support. Worth exploring if you're pushing the boundaries of efficient LLM inference, but not ready for production deployment without significant validation work.