aisa-group / skill-inject

What is skill-inject?

Skill-inject is a Python benchmark for measuring agent vulnerability to skill file attacks, testing if LLM coding agents like Claude Code, Codex, or Gemini CLI execute hidden malicious instructions embedded in skill definitions. It runs isolated Docker experiments with 41 contextual injections (ambiguous harms) and 30 obvious ones (ransomware, exfiltration), across safety policies like warnings or legitimizing prompts. Developers get attack success rates (ASR), task utility baselines, and ablation results like best-of-N sampling or skill screening.

Why is it gaining traction?

It stands out with ready-to-run CLI experiments on your API keys—no custom setup beyond Docker build—and auto-evaluates via LLM judges, spitting out results in final_results folders. Unlike generic prompt injection tests, it targets real agent skills (documents, email, calendar, healthcare), mimicking alpha-skill-injector or HTB command injection skill assessments but for AI. The Docker sandbox prevents real harm while quantifying defenses.

Who should use this?

AI safety engineers benchmarking agent robustness before production. LLM agent builders evaluating skill injector risks in tools handling files, email APIs, or git. Security teams assessing vulnerability to attacks like hidden exfiltration in uploaded skills.

Verdict

Grab it if you're building secure LLM agents—strong docs, smoke tests, and HPC support via Apptainer make evaluation straightforward. But with 18 stars and 1.0% credibility score, it's early alpha; run your own baselines before trusting for high-stakes decisions.

(198 words)