YutoTerashima / multilingual-llm-safety-bench

What is multilingual-llm-safety-bench?

This Python tool runs compact safety benchmarks for LLMs across English, Japanese, and Chinese, testing behaviors like refusal, over-refusal, and unsafe compliance on synthetic prompts. It solves the gap in multilingual safety evals by pulling real data from datasets like multilingual-safety-classification-dataset, running GPU-backed classification baselines (TF-IDF, MLP), and spitting out metrics, per-language breakdowns, figures, and Markdown reports. Users get quick smoke tests or full 80k-example runs via simple scripts, with redacted failure analysis to avoid sensitive content.

Why is it gaining traction?

Unlike english-japanese translators or github english dictionaries focused on vocab, this bench reveals real safety gaps in non-english github projects, like script effects in Chinese or Japanese where char n-grams beat word models. Devs dig the reproducible conda-powered GPU flows that auto-generate CSVs, JSONs, confusion matrices, and group slices (e.g., amh_Ethi unsafe recall), making it easy to spot over-refusal in low-resource languages without building from scratch.

Who should use this?

AI safety engineers tuning multilingual moderation for chat apps handling english-japanese language exchange or chinese prompts. Researchers prototyping baselines before fine-tuning mBERT on safety data. Moderation teams at Discord-like platforms auditing LLM transfer from English to Japanese names or scripts.

Verdict

Solid research prototype for multilingual safety benches, with strong docs and pytest coverage, but low maturity at 14 stars and 1.0% credibility score limits production trust. Grab it for quick per-language evals if you're experimenting with non-english LLM risks.

(198 words)