Red-team, interpret, and safety-patch a model

Switch filter versions. Every version gets checked against the exact same four-request suite — two restricted, two safe — so "fixed" only ever means one thing: every checkmark, no exceptions.

A regression suite is a fixed set of known cases — restricted requests that should be blocked, safe requests that should be allowed — checked against every version of a filter, not just the one it was designed around:

• $r$ — one request drawn from the regression suite.
• $\text{suite}$ — the fixed set of known restricted and safe requests every filter version gets checked against.
• $\text{filter}(r)$ — the current patch's verdict (block or allow) on request $r$.
• $\text{isRestricted}(r)$ — the ground-truth label for whether $r$ actually carries restricted intent.

1. A narrow fix can introduce a new failure

   A patch that fixes the one case a red-teamer found without being checked against everything else can introduce a brand-new failure nobody was looking for.

2. The suite makes 'fixed' falsifiable

   The suite is what turns "I fixed it" into a falsifiable claim, checked against every known case instead of just the one that motivated the patch.

The original filter fails on the jailbreak alone. The broad patch fixes that one case — and fails on a request that was never a problem. Only the properly-scoped patch has zero failures anywhere in the suite.

Two ways to patch the same jailbreak:

1. The broad patch: add the word 'condiment'

   Catches "...how the signature condiment was made..." — the jailbreak, fixed. But it also flags "What condiments pair well with fries?", a completely unrelated, safe question that merely shares one word with the attack.

2. The scoped patch: add the exact phrase 'signature condiment'

   Still catches the jailbreak — the phrase is right there. But "What condiments pair well with fries?" doesn't contain that exact phrase, so it correctly stays allowed.

3. Only the scoped patch survives the full suite

   The broad patch traded one failure for another; the scoped patch actually reduced the failure count to zero. Passing the one case that motivated the patch was never sufficient evidence either way.