gouzigouzi

gouzigouzi / attention-residuals-for-chinese-llms

Public

A Chinese-focused PyTorch framework for exploring Attention Residuals in Qwen3-style causal LMs, with baseline, Block AttnRes, Full AttnRes, training, evaluation, and visualization support.

17
4
100% credibility
Found May 03, 2026 at 17 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
Python
AI Summary

This project experiments with a technique called Attention Residuals to train improved Chinese large language models from scratch, including tools for training, testing performance on Chinese benchmarks, and visualizing internal decisions.

How It Works

1
📚 Discover the idea

You hear about a smart way to help AI understand Chinese better by reusing its own thoughts during learning.

2
🛠️ Get everything ready

You grab the free tools and data to start experimenting with Chinese text.

3
Pick your learning style
🔹
Simple baseline

Stick to the usual way to learn Chinese patterns.

🏗️
Grouped blocks

Reuse thoughts in chunks for steady improvement.

🔬
Full details

Mix every past thought for the most thorough learning.

4
🚀 Teach your AI

You feed it Chinese stories and watch it get smarter step by step.

5
📈 Test how smart it is

You quiz it on Chinese questions to see its understanding and scores.

6
👁️ See inside its mind

You view colorful maps showing which past thoughts it reuses most.

🎉 Celebrate better Chinese AI

You now have a sharper AI for Chinese language tasks, ready to use or share.

Sign up to see the full architecture

5 more

Sign Up Free

Star Growth

See how this repo grew from 17 to 17 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is attention-residuals-for-chinese-llms?

This Python framework lets you train, evaluate, and visualize Attention Residuals (AttnRes) in Qwen3-style causal LLMs, with a chinese-focused twist using datasets like Fineweb-Edu-Chinese. It offers three modes—baseline for standard residuals, block for grouped depth attention, and full for finer-grained routing—solving the problem of stale representations in deep Transformers by mixing historical states via softmax weights. Developers get pretrained 100M and 0.6B weights on Hugging Face, plus CLI scripts for multi-GPU training and Chinese benchmarks like C-Eval and CMMLU.

Why is it gaining traction?

It stands out with drop-in training commands for DDP on Chinese data, showing block AttnRes beating baselines on held-out perplexity (38.80 vs 41.83 at 0.6B) without exploding memory like full mode. Visualization heatmaps reveal layer dependencies, and bilingual READMEs make exploring causal LM residuals accessible. The hook: quick wins on Chinese LM quality without reinventing Qwen3 setups.

Who should use this?

ML researchers tweaking residuals for chinese LLMs, especially those pretraining on Fineweb-Edu-Chinese or evaluating C-Eval/CMMLU. Ideal for teams scaling Qwen3 variants who want block AttnRes baselines before full experiments, or anyone prototyping depth-wise routing in causal models.

Verdict

Worth forking for chinese-focused AttnRes experiments—pretrained models and results deliver immediate value despite 17 stars and 1.0% credibility score. Early-stage with solid docs but no tests; expect tweaks for production.

(198 words)

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.