Adversarial Attack Libraries: ART, Foolbox, torchattacks
ART, Foolbox, torchattacks, CleverHans and TextAttack compared on scope, framework support, attack coverage, licence and repository maintenance.
Choosing an adversarial attack library is usually treated as a taste question and it is not. The libraries differ on which threat models they can express at all, and picking the wrong one produces a robustness number that is quietly meaningless — not because the implementation is wrong, but because the attack the library ships is not the attack that would be run against the deployed system.
The confusion is worse now that “AI red teaming tool” mostly means an LLM prompt scanner. Those are a separate lineage, covered in the site’s guide to AI red teaming tools for ML models. This piece is about the other lineage: the Python libraries that implement the actual adversarial-ML attack families against classifiers, detectors and text models, where the output is a perturbed input and a robust-accuracy figure rather than a pass/fail on a prompt probe.
The comparison
Repository metadata below is read from each project’s public GitHub record as of 18 August 2026. Last activity is the most recent push to the default branch, which is the cheapest honest proxy for whether a project is still tracking the literature.
| Library | Scope | Frameworks | Attack families | Licence | Last repo activity |
|---|---|---|---|---|---|
| ART | Full ML security surface | TF/Keras, PyTorch, scikit-learn, XGBoost, LightGBM, CatBoost | Evasion, poisoning, extraction, inference, plus defences | MIT | Dec 2025 |
| Foolbox | Evasion benchmarking | PyTorch, TensorFlow, JAX (via EagerPy) | Evasion, white-box and decision/score-based black-box | MIT | Dec 2025 |
| torchattacks | PyTorch-native evasion | PyTorch only | Evasion, incl. bundled AutoAttack | MIT | Jun 2024 |
| CleverHans | Reference implementations | JAX, PyTorch, TF2 | Evasion | MIT | Apr 2024 |
| AutoAttack | Standardised evaluation | PyTorch | Parameter-free evasion ensemble | MIT | May 2024 |
| RobustBench | Leaderboard and model zoo | PyTorch | Evaluation harness, not an attack library | Not SPDX-identified | Apr 2026 |
| TextAttack | NLP | PyTorch, Hugging Face | Text evasion, augmentation, adversarial training | MIT | Aug 2026 |
| advertorch | Research toolbox | PyTorch | Evasion and defences | LGPL-3.0 | Sep 2023 |
Two things fall out of that table immediately. The licence column is not uniform — advertorch is LGPL-3.0 and RobustBench carries no machine-identifiable licence, both of which matter before anything gets vendored into a commercial codebase. And the maintenance column separates the list into projects still moving with the field and projects that are now historical references.
ART is the only one with the full threat surface
IBM’s Adversarial Robustness Toolbox, maintained under the Linux Foundation’s Trusted-AI umbrella, is the only library in the set that implements attacks outside the evasion family. It covers data poisoning and backdoors, model extraction, and membership inference alongside evasion, and it does so across a framework list that includes the tabular estimators — scikit-learn, XGBoost, LightGBM, CatBoost — that the PyTorch-only libraries cannot touch.
That breadth is the reason to reach for it, and it is also the reason it is heavier to use than the alternatives. If the model under assessment is a fraud classifier or a gradient-boosted tabular model, ART is effectively the only option. If the threat model includes anything at training time or anything privacy-related, it is again the only option, because every other library on the list assumes the model is finished and the attacker is at inference.
Foolbox and torchattacks are the ergonomic evasion tools
Foolbox exists to make the attack loop short. It wraps PyTorch, TensorFlow and JAX behind a common tensor abstraction and leans toward minimal-perturbation search — find the smallest perturbation that succeeds, rather than the strongest attack within a fixed budget — which makes it well suited to producing a robustness curve across epsilon rather than a single number. Its black-box coverage, including the decision-based attacks that need only hard labels, is the strongest of the ergonomic options.
torchattacks trades that portability for a PyTorch-only API that is close to trivial to drop into an existing training or evaluation script, and it bundles AutoAttack so the standardised evaluation is one call. The tradeoff is visible in the maintenance column: the repository’s last push predates a significant amount of recent literature, so treat it as a stable implementation of the classical attacks rather than a source of current ones.
CleverHans is where several of these attacks were first implemented and its code remains a useful reference when checking somebody else’s implementation, but its activity has slowed to the point where it should not be the primary tool for new work.
AutoAttack is the floor, not an option
Croce and Hein’s AutoAttack (arXiv:2003.01690, ICML 2020) is not a library you choose between — it is the reason most published robustness numbers before it were wrong. The paper reports that applying its parameter-free ensemble to over 50 previously published models produced lower robust accuracy than those papers claimed — in the authors’ words, “often by more than 10%” — and identified several defences as broken outright.
The ensemble combines two variants of an automatically step-size-tuned PGD, a targeted FAB attack and the black-box Square attack, and it has no hyperparameters to tune, which removes the most common way an evaluation flatters itself: an under-tuned attack. It is the harness behind RobustBench, whose leaderboard is the reference point for what robust accuracy on CIFAR-10 and ImageNet actually looks like under a standard threat model.
The practical rule is simple. Whatever library generates the attacks during development, the number that leaves the building should be produced by AutoAttack under a declared threat model, because that is the number a reader can compare against a leaderboard. Everything else is a development signal. The failure modes that make a self-run evaluation optimistic — gradient masking, too few steps, a single restart, an attack that never sees the defence it is supposed to break — are covered in evaluating adversarial robustness without fooling yourself.
Text and everything the libraries do not cover
TextAttack is the outlier in the list and the most actively maintained project in it. It targets NLP models through the Hugging Face ecosystem and implements the word- and character-level search strategies — synonym substitution under semantic constraints, importance ranking, masked-language-model-guided replacement — that make up the adversarial robustness landscape for text. Its constraint system, which bounds how far a perturbed sentence may drift in meaning, is the part with no analogue in the image libraries, because an image attack’s budget is a norm ball and a text attack’s budget is a semantics argument.
The coverage gap worth naming explicitly is multimodal. None of these libraries implements attacks against contrastive vision-language encoders, which is now one of the most consequential surfaces in deployed systems; those attacks live in the paper authors’ own repositories, as with the pre-generated universal perturbations discussed in CLIP adversarial attacks. Nor does any of them cover agent-level attacks, where the exploit targets tool invocation rather than a decision boundary.
Choosing by threat model, not by preference
- Tabular, tree-based or scikit-learn model: ART. Nothing else supports the estimator.
- Training-time, privacy or extraction threat model: ART. The other libraries stop at inference.
- PyTorch image classifier, quick internal evaluation: torchattacks for the loop, AutoAttack for the reported number.
- Robustness curve across perturbation budgets, or hard-label black-box: Foolbox.
- Text classifier or transformer NLP model: TextAttack.
- A number somebody outside the team will compare: AutoAttack, with the RobustBench leaderboard as context.
- Contrastive encoders and multimodal stacks: none of the above; go to the paper repositories.
Maintenance status belongs in this decision rather than beside it. An attack library that stopped tracking the literature will still run, still return a robust-accuracy figure, and still miss the attack class published after its last commit — which produces a robustness claim that is technically reproducible and substantively stale. To reason about which attacks are feasible against a particular access level, modality and adversary goal before choosing a tool, the site’s attack selector filters the catalogue on those constraints and names a reference implementation for each entry.
References
- Croce & Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” ICML 2020, arXiv:2003.01690
- Croce et al., “RobustBench: a standardized adversarial robustness benchmark,” 2020, arXiv:2010.09670
- Adversarial Robustness Toolbox, Trusted-AI
- Foolbox, bethgelab
- torchattacks, Harry24k
- CleverHans, cleverhans-lab
- AutoAttack, fra31
- RobustBench, RobustBench
- TextAttack, QData
- advertorch, BorealisAI
See also
Sources
Adversarial ML — in your inbox
Working adversarial ML — exploits, defenses, and the gap between — delivered when there's something worth your inbox.
No spam. Unsubscribe anytime.
Related
Best AI Red Teaming Tools for ML Models in 2026
A practitioner's guide to the tools used to red team machine learning models: PyRIT, garak, ART, ModelScan, and Giskard, and where each fits a pipeline.
Evaluating Adversarial Robustness Without Fooling Yourself
Most defenses that claim robustness are later broken — not because the idea was bad, but because the evaluation was. Adaptive attacks are non-negotiable.
Adversarial Examples vs. Data Poisoning: Timing Is Everything
Adversarial examples attack a deployed model at inference; data poisoning attacks the model before it is deployed. Different timing, different defenses.