Injection resistance cannot be self-evaluated - it needs a benchmark with a fixed threat model, fixed system prompt and a defined attack suite

A team believes its injection defence works because a set of hand-written attacks is refused. Every published attack they later read breaks it. The problem is that the evaluation was never comparable to anything.

JailbreakBench's motivation is that there was no clear standard of practice for jailbreak evaluation, that existing works compute costs and success rates in incomparable ways, and that many results were not reproducible because adversarial prompts were withheld, code was closed-source, or APIs evolved. Its fix is a defined threat model, fixed system prompts and chat templates, 100 behaviours, a repository of jailbreak artifacts, and a leaderboard tracking attacks AND defences. AgentDojo supplies the equivalent for agents that call tools: 97 realistic tasks across email, e-banking and travel, with 629 security test cases. Both exist because self-evaluation does not work.

Evaluate any injection or safety defence against one of these rather than an internal suite, and report the score with the threat model attached. If you must build internal tests, at minimum fix the system prompt and chat template as variables - changing either silently changes the attack surface. Include over-refusal in the score, or a defence that refuses everything will look excellent.

The chat-template point is easy to miss and matters for serving specifically: the same model with a different chat template is a different attack surface, so a defence result does not transfer across template changes. JailbreakBench also ships an over-refusal dataset, which is the metric that keeps a safety score from being gamed by blanket refusal. SCHEME RECLASSIFIED (2026-10-04): was 'measurement'.