marckrenn

marckrenn / pi-lab

Public

pi-lab lets you run tool-based A/B tests in various modes

ab-testing pi pi-coding-agent pi-shitty-coding-agent
12
0
100% credibility
Found Mar 27, 2026 at 12 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
TypeScript
AI Summary

pi-lab is an experimental tool for AI coding agents that runs multiple parallel versions of a task, compares their outcomes using formulas or AI grading, selects the best one, and applies it seamlessly.

How It Works

1
📖 Discover pi-lab

You hear about pi-lab, a clever way to test different strategies your AI coding helper can use for tasks like editing files or running commands.

2
🔧 Add it easily

You quickly add pi-lab to your AI coding helper so it can start running smart experiments.

3
🧪 Set up your test

You chat with your AI helper using a simple command, and it asks friendly questions to create a custom experiment comparing different approaches.

4
✏️ Work as usual

You ask your AI helper to do everyday tasks like edit code or check files, just like always.

5
✨ Smart comparison happens

Magically, it tries several options at once in safe separate spaces, picks the best one based on your rules, and uses that result.

6
📊 See the winner

You get a clear summary showing what was tried, scores for each option, and why the winner was chosen.

🎉 Better results every time

Your coding tasks now get handled more reliably with the top-performing approach automatically selected.

Sign up to see the full architecture

5 more

Sign Up Free

Star Growth

See how this repo grew from 12 to 12 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is pi-lab?

pi-lab is a TypeScript extension for the pi coding agent that lets you run tool-based A/B tests in various modes behind a single tool call. It intercepts calls to builtins like edit or custom tools, forks isolated lanes with different prompts, models, or extensions using git worktrees, then selects a winner via formula, LLM grading, or blends before applying the result to your main branch. Perfect for safely comparing agent behaviors without derailing your workflow.

Why is it gaining traction?

It stands out with seamless interception of tools like edit for transparent replacements, rich selection modes (hardcoded fallbacks to LLM tiebreakers), and built-in telemetry for runs. The /lab CLI handles creation, status, and GC, plus project-local configs make iteration fast—no more manual git juggling. Developers dig the no-git fallback and detailed summaries showing latencies, tokens, and scores.

Who should use this?

Pi agent builders A/B testing extension bundles or prompts for coding tasks. Prompt engineers at pi labs gmbh or pi-labs pune tuning lanes for better outputs. Teams running pi-labs microsoft workflows who need baselines with experimental models before production.

Verdict

Grab it if you're deep in pi and git repos for tool A/B—installs via pi install git:... and /lab create gets you running fast. At 12 stars and 1.0% credibility, it's experimental alpha with changing APIs, but strong docs and telemetry make it viable for early adopters.

(198 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.