joshawome

joshawome / chainreason

Public

A benchmark for evaluating LLM reasoning on Ethereum and DeFi tasks

41
27
100% credibility
Found May 04, 2026 at 41 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

ChainReason is a benchmark suite for evaluating AI language models on Ethereum and DeFi reasoning tasks including protocol questions, vulnerability detection, contract classification, transaction intent, and slippage prediction.

How It Works

1
📖 Discover ChainReason

You hear about a simple tool that tests how smart AI assistants are at understanding crypto trading and smart contracts.

2
💻 Set it up

You download the tool and get it ready on your computer in just a few minutes.

3
🤖 Connect an AI helper

You link up your favorite AI service, like one from OpenAI or Anthropic, so it can start thinking about blockchain puzzles.

4
🧩 Pick a challenge

You choose one of five fun tasks, like spotting risks in code or predicting trade outcomes.

5
▶️ Run the test

You press go and watch the AI tackle a handful of real-world DeFi questions.

6
📊 Review the scores

You get clear reports with accuracy scores, showing exactly how well the AI performed on each type of reasoning.

🎉 Master AI strengths

Now you know which AIs shine at crypto analysis, helping you pick the best one for your work.

Sign up to see the full architecture

5 more

Sign Up Free

Star Growth

See how this repo grew from 41 to 41 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is chainreason?

ChainReason is a Python benchmark evaluating large language models on Ethereum and DeFi reasoning tasks, like protocol mechanics QA, Solidity vuln detection, contract classification from ABI, tx intent from action traces, and AMM slippage math. It solves the gap in testing LLMs beyond code gen or basic vulns, stressing symbolic, structural, and numeric skills via small, hand-curated datasets. Users run evals via CLI scripts, YAML configs, or Python API, hitting OpenAI, Anthropic, or local HF models with auto-caching and metrics like accuracy plus macro-F1.

Why is it gaining traction?

It stands out by consolidating underrepresented axes—protocol reasoning, tx-graph patterns, numeric grounding—absent from Solidity-focused benchmarks on GitHub. Quick-start CLI benchmarks any model in minutes, outputs JSONL results and summaries, and extends easily with custom tasks or data paths. Devs grab it for fast GitHub Copilot or local LM tests without scraping Etherscan noise.

Who should use this?

DeFi protocol engineers validating LLM assistants on swap/liquidity math, security researchers benchmarking vuln detectors on real snippets, and blockchain AI teams comparing hosted vs. open models for tx analysis. Perfect for quick evals before fine-tuning on process mining or earth science-like domain tasks.

Verdict

Grab it for lightweight DeFi LLM benchmarking—strong docs and CLI beat its 1.0% credibility score and 41 stars, though seed sizes demand your own data for production. Early maturity, but extensible enough to grow into a staple.

(178 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.