lalitshankarch

lalitshankarch / xuniq

Public

xuniq is a blazing-fast utility to remove duplicate lines from input

19
1
100% credibility
Found Apr 14, 2026 at 19 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Rust
AI Summary

xuniq is a high-performance utility that quickly removes duplicate lines from text input without requiring the input to be sorted.

How It Works

1
📄 Spot the duplicates

You have a big text file or list full of repeated lines that you need cleaned up quickly.

2
🔍 Discover xuniq

You hear about xuniq, a superfast tool that removes duplicate lines without needing everything sorted first.

3
💻 Get xuniq ready

You add xuniq to your computer tools so it's simple to use anytime.

4
⚡ Zap the duplicates

You feed your messy list into xuniq, tweak settings like list size or safe mode if you want, and watch it fly through the job.

5
⚙️ Pick your style

Choose speedy mode for lightning results or safe mode for perfect accuracy every time.

🎉 Clean list ready

You get back a neat file with only unique lines, processed way faster than old methods, ready for whatever you need next.

Sign up to see the full architecture

4 more

Sign Up Free

Star Growth

See how this repo grew from 19 to 19 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is xuniq?

xuniq is a Rust-built CLI utility that removes duplicate lines from input streams without requiring sorted data, unlike traditional uniq. Pipe in logs, CSV, or any line-based text, and it outputs unique lines blazing-fast via stdin to stdout. Install xuniq with Cargo for quick deduping in scripts or pipelines.

Why is it gaining traction?

It crushes benchmarks, processing 1.25M duped + 2.5M unique lines in 536ms—nearly twice as fast as ripuniq and 10x quicker than sort|uniq. Options like --capacity for tuning memory and --safe for collision-free exact matching give users control without slowdowns. Developers searching xunique or xuniq grab it for real-world speedups over awk or Perl one-liners.

Who should use this?

Data engineers cleaning massive log files before analysis, DevOps folks deduping access lists in CI/CD, or backend devs streamlining ETL pipelines with unsorted inputs. If you're Nathalie xunique-style scripting large datasets or Amelia xunique-handling ad-hoc CLI tasks, this slots right in.

Verdict

Try xuniq for high-volume dedup if speed trumps bells and whistles—its Rust core delivers, but 19 stars and 1.0% credibility score signal early maturity with basic docs. Solid for prototypes, monitor for wider adoption.

(178 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.