mauriziofonte

mauriziofonte / toktoken

Public

TokToken is a fast, single-binary C codebase indexer for AI coding agents. Powered by universal-ctags and SQLite FTS5, it provides precise symbol search, dependency tracking, and an MCP server. TokToken reduces LLM context token usage by 88-99% by retrieving exact code symbols instead of reading entire files. Zero runtime dependencies.

24
5
89% credibility
Found Mar 19, 2026 at 24 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
C
AI Summary

TokToken is a tool that indexes codebases to allow AI coding agents to search symbols, trace imports, and retrieve only relevant code snippets, significantly reducing token usage as demonstrated by benchmarks on Redis and Linux kernel repositories.

Star Growth

See how this repo grew from 24 to 24 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is toktoken?

TokToken is a fast C single-binary codebase indexer for AI coding agents. It scans repos with universal-ctags and SQLite FTS5 to track symbols, dependencies, and imports across 46 languages, letting agents fetch exact code snippets instead of entire files. Run `toktoken index:create` once, then search with `search:symbols "auth"` or query via MCP server—cutting tiktoken/tiktokenizer context by 88-99%.

Why is it gaining traction?

Zero runtime dependencies, incremental updates on changed files, and GitHub repo indexing without cloning make it dead simple for big codebases. FTS5 powers relevance-ranked symbol search with dependency graphs (`find:importers`), while MCP server plugs into Cursor, Claude Code, or Copilot for native agent use. Real benchmarks on Redis/Linux kernel show token savings that agents notice immediately.

Who should use this?

Devs building AI coding agents or using Cursor/Claude for refactoring large C/Python/JS codebases. Ideal for backend teams tracing dependencies in monorepos, or frontend folks querying exact components without bloating prompts.

Verdict

Strong pick for agent-heavy workflows—88-99% context wins justify the setup. At 24 stars and 0.9% credibility, it's beta-mature with excellent docs; index a test repo to confirm fit before scaling.

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.