Adversarial Training Best Practices That Survive Evaluation

Adversarial training remains the one evasion defense that has consistently survived independent re-evaluation, and it is also the defense teams most often get wrong. The published record is blunt about this: when Croce and Hein re-tested dozens of peer-reviewed defenses with a standardized attack ensemble, robust accuracy came in lower than reported in nearly every case, often by more than ten percent, and several defenses collapsed entirely. Most adversarial training best practices are therefore not about the training loop at all. They are about threat-model discipline before training and honest evaluation after it. This guide covers both, plus the two overfitting failure modes that quietly ruin runs in between.

Start from the standard recipe, not a clever variant

The baseline is PGD adversarial training as formulated by Madry et al.: treat robustness as a min-max problem, generate worst-case perturbations with projected gradient descent inside a fixed norm ball during training, and update weights on those adversarial examples. Nearly a decade of broken alternatives later, this is still the reference implementation, and top entries on the RobustBench leaderboard are variations on it rather than departures from it.

Practical rules that follow from the formulation:

• Fix the threat model first. Norm (ℓ∞ or ℓ2), budget ε, and data domain define what you are buying. Robustness to ℓ∞ at ε=8/255 on images says nothing about ℓ2 perturbations, physical patches, or distribution shift. Write the threat model down before the first run; it is also what you will be audited against. NIST’s AI 100-2 E2025 taxonomy gives you shared vocabulary for that document, which matters when the people reading it are risk owners rather than ML engineers.
• Use enough inner-maximization steps. A weak inner attack trains against a weak adversary. Multi-step PGD with random starts is the default; single-step shortcuts need the safeguards described in the next section.
• Resist the urge to invent. Novel loss terms, detection add-ons, and gradient-obfuscating preprocessing are exactly the category of defense that adaptive attacks keep demolishing. If a variant is not holding up on RobustBench under standardized evaluation, do not bet a production system on it.

Catch the two overfitting failure modes early

Adversarial training has failure modes that clean training does not, and both are invisible if you only watch clean accuracy.

Robust overfitting. Rice, Wong, and Kolter showed that during long PGD-AT runs, robust test accuracy peaks and then degrades while training metrics keep improving, and that early stopping alone matched the gains of virtually every algorithmic improvement proposed on top of adversarial training at the time. The operational consequence: hold out a validation set, attack it with PGD every few epochs, and checkpoint on robust validation accuracy, not on loss or clean accuracy. Skipping this can silently cost you more robustness than any fancy training trick will ever add.

Catastrophic overfitting. If you use single-step FGSM training to save compute, know its failure mode: robustness to multi-step attacks can collapse to near zero mid-run while FGSM accuracy stays high. Wong, Rice, and Kolter showed FGSM training with random initialization can approach PGD-AT robustness at a fraction of the cost (their headline: 45% robust accuracy against PGD at ε=8/255 on CIFAR-10 in six minutes), but only when this collapse is avoided. Monitor PGD accuracy during training, not just the FGSM accuracy the loop optimizes; a widening gap between the two is the tell.

Evaluate with attacks you did not train against

Self-evaluation with the training attack is how broken defenses get published. The pattern to internalize from the AutoAttack work is that PGD with default hyperparameters routinely overestimates robustness, so your acceptance test needs to be stronger and more diverse than your training adversary.

Minimum bar before you call a model robust:

1. Run AutoAttack (the parameter-free white-box plus black-box ensemble RobustBench standardizes on) at your declared ε. Report that number, not your training-attack number.
2. Check for gradient masking: black-box attacks succeeding where white-box fails, or robustness that evaporates when ε grows slightly, both indicate you broke the gradients rather than the attack.
3. If your defense has any nonstandard component, budget for an adaptive attack designed against that component specifically. Attackers will.

Treat the evaluation artifacts like pentest evidence: attack configs, ε, seeds, and per-attack numbers, versioned next to the model.

Budget for the trade-offs, then keep watching in production

Adversarial training is not free and pretending otherwise sets the project up to fail. Expect a clean-accuracy drop against the standard model, a multiple of baseline training compute for multi-step PGD, and a robustness guarantee that extends exactly as far as the threat model you wrote down. NIST AI 100-2 is explicit that evasion mitigations manage rather than eliminate the attack class.

Two deployment-side practices close the loop. First, monitor robust accuracy the way you monitor drift: re-run your attack suite on production model versions on a schedule, because fine-tunes and data refreshes can quietly erode robustness that was expensive to buy. If you already have MLOps monitoring infrastructure, robustness regression belongs in it alongside drift detection; the workflow overlap with standard model monitoring practice is nearly total. Second, layer defenses. Adversarial training hardens the model itself, but input validation, rate limiting on query access (model extraction feeds evasion), and runtime guardrails cover the attack surface the norm ball does not.