Adversarial Transferability: How Black-Box Attacks Work

Adversarial examples transfer because models trained independently on the same task learn decision boundaries that sit close together in almost every direction, so a perturbation that crosses one model’s boundary usually crosses others too. That lets you attack a model you can only query: craft on a local surrogate and transfer, or estimate its gradient from the query outputs.

Both routes are practical against production APIs, and each has a distinct cost profile and a distinct defense. The sections below fix the numbers that get quoted loosely, give the methods that raise transfer, tabulate the query attacks that need no surrogate, and end with a transfer matrix you can reproduce and the defenses built specifically for this threat.

The substitute-model attack, with real numbers

Papernot et al.’s Practical Black-Box Attacks against Machine Learning (ACM Asia CCS 2017) turned transferability into an operational recipe against APIs that return only a label. It runs in three phases.

Train a substitute. The adversary starts from a tiny seed set (150 images from the MNIST test set in the paper), labels each seed by querying the target oracle, and trains a local model to imitate the target’s decisions. The problem is that a fixed seed set does not cover the target’s decision boundary, and querying is what the defender is watching.

Grow the training set toward the boundary. The key step is Jacobian-based dataset augmentation. After each round, the substitute set doubles by adding, for every point x, a new point x + lambda * sign(J[O(x)]), where J is the substitute’s Jacobian evaluated at the label the oracle assigned. This walks new synthetic samples in the directions where the oracle’s output changes fastest, so a handful of augmentation rounds buys a substitute that tracks the boundary where it matters. The paper uses lambda = 0.1 and six augmentation rounds.

Craft on the substitute and transfer. With white-box access to the substitute, the adversary runs FGSM, PGD, or C&W and submits the results to the target. The reported misclassification rates were 84.24% against MetaMind, 96.19% against Amazon, and 88.94% against Google, all at perturbation epsilon = 0.3. The Amazon and Google attacks used logistic-regression substitutes trained with only 800 oracle queries, a figure confirmed in the companion paper Transferability in Machine Learning. Microsoft was never in this evaluation; the “84 to 100% against Google, Amazon, Microsoft” claim that circulates is a conflation. These are also small-image tasks (MNIST, traffic signs), not ImageNet, which matters when you compare against the ImageNet transfer rates below.

Why do adversarial examples transfer?

They transfer because independently trained models place their decision boundaries very close together, and the space of perturbations that fool one model is large and shared. That is the finding of Tramer et al.’s The Space of Transferable Adversarial Examples, which measured the geometry directly rather than reasoning about it.

Two results anchor the explanation. First, using a technique that searches for mutually orthogonal adversarial directions, the authors found roughly 25 dimensions of adversarial subspace per input on MNIST (44.28 orthogonal perturbations on average, 24.87 of which also fooled a second model). A high-dimensional shared subspace is far more likely to intersect than a single shared direction. Second, for most inputs and directions the distance from a clean point to a model’s decision boundary is larger than the distance between the two models’ boundaries, so a perturbation only needs to overshoot slightly to land on the far side of both.

The complementary account is the non-robust-features hypothesis in Adversarial Examples Are Not Bugs, They Are Features: models learn predictive but brittle features from the data, “since any two models are likely to learn similar non-robust features, perturbations that manipulate such features will apply to both,” and the authors show that architectures that learn those features better are the ones adversarial examples transfer to more often. Transfer is not universal, though. Tramer et al. also construct a data distribution where two model classes learn boundaries whose adversarial directions do not overlap, and neither model’s perturbation moves the other. That is the same effect that shows up between convolutional networks and transformers below.

Reading a transfer rate

A transfer number is meaningless without its setting, because the same attack family spans nearly the whole percentage range depending on how it was run. Liu et al.’s Delving into Transferable Adversarial Examples (ICLR 2017) is the cleanest source for this, because it reports every cell of the matrix on the same 100 ImageNet validation images across ResNet-50, ResNet-101, ResNet-152, VGG-16, and GoogLeNet, with distortion measured as root-mean-square deviation (RMSD) over pixels, not as an L-infinity budget. In its non-targeted tables the cells are the target model’s accuracy on the adversarial images, so lower means more transfer; the diagonal is the white-box case.

The white-box row gives the others meaning. The gap between it and the ensemble row is what an attacker pays for not having gradients, and the honest way to read a transfer number is as a fraction of what the same attack achieves with full access. Ensembling several diverse surrogates and crafting the common direction, as in the fourth and fifth rows, is the single biggest lever in this table. When a rate appears without its setting, five things are missing and each moves the number: source architecture, target architecture, norm and budget, targeted versus untargeted, and single model versus ensemble. A ResNet-to-ResNet rate says nothing about a ResNet-to-transformer pair, and an untargeted rate says nothing about a targeted one.

Where the target instead shares a public encoder with the surrogate, the normal arrangement for multimodal systems, transfer stops being probabilistic and moves to a different regime, described in CLIP adversarial attacks.

Methods that improve transfer

Crafting a plain PGD example on one model and shipping it is the weak baseline. Six methods, all compatible with each other, close specific gaps. Budgets below are L-infinity epsilon = 16 on [0, 255] pixels unless noted.

Two of these deserve a note. Translation-invariant attacks were designed for defended targets specifically: stacking TI with input diversity (TI-DIM) crafted on a four-model ensemble reaches 78 to 85% against ensemble adversarially trained ImageNet models. And the logit loss exists because cross-entropy’s gradient vanishes as the target-class probability saturates, so targeted transfer needs far more iterations than the “20 or fewer” that older papers used; Zhao et al. show that simply running 300 iterations with the logit loss beats more elaborate targeted objectives and even moves Google Cloud Vision at an 18% targeted rate.

Choosing a surrogate

The surrogate’s architecture family decides the ceiling before any of the above methods run. Two facts govern the choice.

Transfer between convolutional networks and vision transformers is weak in both directions. Mahmood et al.’s On the Robustness of Vision Transformers to Adversarial Examples (ICCV 2021) found that “adversarial examples do not readily transfer between CNNs and transformers”: on CIFAR-10 at epsilon = 0.031, ViT-B/16 to ResNet-56 transferred at 13.9% and ResNet-56 to ViT-B/16 at 9.0% (6.6% to ViT-B/32), against far higher within-family rates. If you do not know the target family, a single CNN surrogate can quietly fail against a ViT backbone. The fix when the target is a transformer is to attack it as one: Naseer et al.’s self-ensemble with token refinement (ICLR 2022) lifts DeiT-B to ResNet-152 transfer from 44.66% to 70.18% by treating each transformer block as an ensemble member.

Slightly robust surrogates transfer better than standard ones. Springer et al.’s A Little Robustness Goes a Long Way (NeurIPS 2021) crafted targeted attacks on a ResNet-50 adversarially trained to a small L2 radius and evaluated on 1,000 ImageNet images: with the logit loss, moving the surrogate from standard (epsilon = 0) to lightly robust (epsilon = 0.1) raised targeted success against Xception from 45.9% to 77.2%, and against a ViT from 3.9% to 33.4%. Push the surrogate’s robustness too high and it discards the non-robust features the target still relies on, so the effect is a peak, not a monotone gain. A practical surrogate procedure: match the target’s likely family (use an ensemble spanning families when unsure), prefer a lightly robust checkpoint over a standard one, ensemble several diverse models, and stack MI plus DI plus TI, adding the logit loss and many iterations if the goal is targeted.

Query-based black-box attacks

Transferability is not the only route. When you can query the target repeatedly, you can attack it directly and skip the surrogate. These attacks split by what the API returns: full confidence scores (score-based) or only the top label (decision-based). Figures below are ImageNet, mostly on ResNet-50 or Inception-v3, at each paper’s stated budget.

Two patterns matter for planning. Score-based attacks are cheap once the API leaks a confidence vector; Square Attack in particular reaches near-zero failure at a median of 11 queries on ResNet-50, which is why it is the black-box member of AutoAttack. Decision-based attacks, which need only the predicted label, cost far more and are usually framed as driving perturbation size down at a fixed query budget rather than as a single success rate.

A surrogate can cut the query count sharply. Hybrid attacks seed or steer the query search with surrogate gradients.

Pick the family from three inputs. If the API returns only a label, you are in the decision-based column (HopSkipJump, Sign-OPT, or GeoDA). If it returns scores and queries are cheap, a pure score-based attack is simplest, and Square is the default first try. If queries are metered or the attack is targeted, and you can guess the architecture family well enough to stand up a surrogate, a hybrid (P-RGF, Subspace, or TREMBA) buys an order-of-magnitude query reduction. If you get no queries at all before the real attack, you are back to transfer-only, and the surrogate and stacking choices above are the whole game.

A transfer matrix you can reproduce

No public page gives first-party transfer numbers for a modern surrogate against both CNN and transformer targets. Here is the experiment to run yourself. It measures how much a transfer attack loses relative to white-box, across target families and a robust target, at two budgets.

Take 1,000 correctly classified ImageNet validation images. Use a standard ResNet-50 as the surrogate and evaluate on VGG-16 (71.6% top-1), DenseNet-121 (74.4%), ViT-B/16 (81.1%), and a robust ImageNet model from the RobustBench zoo such as Salman2020Do_R50 (evaluated on the leaderboard at L-inf 4/255). Craft with PGD, MI-FGSM, and DI-MI-FGSM at L-inf 8/255 and 16/255 on [0, 1]-scaled pixels, in both untargeted and targeted (fixed random label) modes. torchvision weights and torchattacks cover everything; the surrogate needs its normalization registered on the attack, since torchattacks expects inputs in [0, 1].

import torch, torchvision.models as tvm
from torchvision.models import (ResNet50_Weights, VGG16_Weights,
                                DenseNet121_Weights, ViT_B_16_Weights)
import torchattacks
from robustbench.utils import load_model

dev = "cuda"
mean, std = [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]
surrogate = tvm.resnet50(weights=ResNet50_Weights.IMAGENET1K_V1).eval().to(dev)

targets = {
    "vgg16": tvm.vgg16(weights=VGG16_Weights.IMAGENET1K_V1),
    "densenet121": tvm.densenet121(weights=DenseNet121_Weights.IMAGENET1K_V1),
    "vit_b_16": tvm.vit_b_16(weights=ViT_B_16_Weights.IMAGENET1K_V1),
    "robust_r50": load_model("Salman2020Do_R50", dataset="imagenet", threat_model="Linf"),
}

for eps in (8/255, 16/255):
    attacks = {
        "PGD": torchattacks.PGD(surrogate, eps=eps, alpha=eps/4, steps=20, random_start=True),
        "MI-FGSM": torchattacks.MIFGSM(surrogate, eps=eps, alpha=eps/10, steps=10, decay=1.0),
        "DI-MI-FGSM": torchattacks.DIFGSM(surrogate, eps=eps, alpha=eps/10, steps=10,
                                          decay=1.0, resize_rate=0.9, diversity_prob=0.5),
    }
    for atk in attacks.values():
        atk.set_normalization_used(mean=mean, std=std)  # surrogate expects normalized input
    # loop the 1,000 images in [0,1]; craft adv on the surrogate; then for each target
    # apply that target's own normalization and record prediction flips.

Record, for each attack and budget, the white-box success on the surrogate itself and the untargeted and targeted success on every target. Read the matrix three ways. The white-box column should sit near 100%, so every transfer cell is a fraction of that. MI-FGSM should beat PGD and DI-MI-FGSM should beat MI-FGSM on the three clean CNN and CNN-adjacent targets, matching the gains Dong et al. and Xie et al. report. ViT-B/16 should trail the CNN targets sharply, the CNN-to-transformer gap from Mahmood et al. And the RobustBench model should sit far below the clean targets and rise far less when you go from 8/255 to 16/255. Targeted success should be a small fraction of untargeted in every cell, which is the whole reason the logit loss and 300-iteration budget exist. Publishing that matrix, with the exact checkpoints and seed, is more useful than any single quoted rate.

Defenses built for the black-box threat

Generic robustness advice does not address query attacks, because the adversary here is not computing a gradient through your model. Four defenses target this threat specifically, and each has a published bypass you should test against before trusting it.

Stateful detection and Blacklight exploit the same weakness in query attacks: iterative optimization submits thousands of nearly identical images, which stand out against normal traffic. Blacklight detected all eight tested attacks, often within the first handful of queries, at a false-positive rate under 0.1%, and rejecting the flagged queries drove attack success to zero for persistent attackers. The Feng et al. result is the caution: an adaptive attacker who first probes the defense’s rejection behavior, then resamples to stay under the similarity threshold, restores near-total success at a cost of well under a dollar per example. Random noise defense trades a little clean accuracy (74.9% to 73.0% on ResNet-50 at noise scale 0.02) for a real climb in failure rates against gradient-estimation attacks, but its own analysis shows the protection is governed by the ratio between the defender’s noise and the attacker’s step size, and Square, which does not estimate a gradient, is barely affected.

The deeper caution applies to all of them. A defense can show near-zero transfer success while an adaptive white-box attacker walks through it, the Obfuscated Gradients finding. Input transforms and randomization block fixed transferred perturbations because the perturbation was crafted on a different, deterministic model, but they do not survive an attacker who models the defense. This is why standardized evaluations such as RobustBench score robustness with adaptive AutoAttack rather than transfer, and why a low transfer rate is a signal about opportunistic attackers, not a robustness certificate. For the threat model that actually matters, genuine robustness still requires adversarial training or certified methods, and the evaluation pitfalls that make a defense look better than it is are covered in evaluating adversarial robustness without fooling yourself. The reference implementations for every attack named here live in the adversarial attack libraries the field standardizes on.

FAQ

how many queries does a black-box adversarial attack need?

It depends on the output and the attack. Against a scores API, Square Attack reaches near-zero failure at a median of 11 queries on ResNet-50, and NES needs a median around 11,550 in its query-limited setting. Label-only decision attacks cost far more, tens of thousands of queries to reach small perturbations. A surrogate-guided hybrid cuts these by roughly an order of magnitude.

can you attack a model that only returns the top label?

Yes. Decision-based attacks need only the predicted label. Boundary Attack, HopSkipJump, Sign-OPT, and GeoDA start from an already-misclassified point and walk along the decision boundary to shrink the perturbation, using label flips alone as the signal. They need many more queries than score-based attacks, but they defeat the common defense of hiding confidence scores, which stops score-based estimation but not these.

does adversarial training stop black-box transfer attacks?

Only partially, and standard single-model adversarial training less than you would hope. Ensemble adversarial training was built to resist transfer by training on perturbations transferred from other models, but translation-invariant attacks combined with input diversity (TI-DIM) still reach 78 to 85% against ensemble-trained ImageNet models. Adversarial training raises the cost of transfer and query attacks, yet a strong adaptive transfer attack remains effective.

what is the best surrogate model for a transfer attack?

Match the target’s architecture family, because CNN-to-transformer transfer is weak in both directions. Prefer a lightly adversarially trained checkpoint over a standard one: a small robust radius raised targeted ImageNet transfer from a ResNet-50 to a ViT from 3.9% to 33.4% in one study. Ensemble several diverse surrogates and craft the common direction, which is the largest single lever on transfer rate.

is a low transfer attack success rate proof my model is robust?

No. Obfuscated Gradients showed defenses that block transferred and non-adaptive attacks while falling to adaptive white-box attackers who account for the defense. Input transforms and randomization destroy a perturbation crafted on a different, deterministic model, so they suppress transfer without providing real robustness. Evaluate with adaptive attacks such as AutoAttack, the standard behind RobustBench, before drawing any robustness conclusion.