Backdoor Attack on Neural Networks Detection: What Works

Backdoor attack on neural networks detection works at three points, and none of them is reliable alone: scan the training data (spectral signatures, activation clustering), scan the trained model (Neural Cleanse, ABS, MNTD), or filter inputs at inference (STRIP). Each catches the attacks it was designed around and fails against adaptive ones, so run at least two layers.

Two results set expectations. Tan and Shokri (IEEE EuroS&P 2020) trained a backdoor with an extra loss term that makes poisoned and clean hidden representations indistinguishable, and it evaded the state-of-the-art latent-space detectors of the time. The Anthropic-led Sleeper Agents paper (Hubinger et al., January 2024) found LLM backdoors survived supervised fine-tuning, RL and adversarial training, and that adversarial training taught models to recognize their triggers better. Treat detection as a probabilistic filter, not a certificate.

What is a backdoor detector actually looking for?

A backdoor detector looks for a shortcut: a small input feature that routes any input to one attacker-chosen label. Clean accuracy won’t reveal it, because a BadNets-style backdoor is trained to leave clean behavior intact. Every detector is a bet on one measurable side effect of that shortcut:

• Separable activations. Poisoned samples sit apart from clean samples of the same label in feature space. Spectral signatures and activation clustering bet here.
• A tiny trigger. One label is reachable from any input with a small patch. Neural Cleanse and ABS bet here.
• Trigger dominance. A triggered input holds its prediction when blended with unrelated images. STRIP bets here.
• Model-level signatures. A meta-classifier trained on clean and trojaned models tells them apart from query responses. MNTD bets here.

Your access level decides which bets you can place. Own the training run and you can scan data. Download a checkpoint and you can scan the model. Call an API and you can only query.

Data-level detection: scan before you train

Spectral signatures and activation clustering both train on the suspect set, pull penultimate-layer activations per class, and flag outliers. The mechanics are in our rundown of data poisoning detection methods. Here is the operational version, using the Adversarial Robustness Toolbox poison detectors:

from art.defences.detector.poison import ActivationDefence, SpectralSignatureDefense

# classifier: ART wrapper around a model trained on the suspect x_train/y_train
ac = ActivationDefence(classifier, x_train, y_train)
_, ac_clean = ac.detect_poison(nb_clusters=2, nb_dims=10, reduce="PCA",
                               cluster_analysis="smaller")

ss = SpectralSignatureDefense(classifier, x_train, y_train,
                              expected_pp_poison=0.05, eps_multiplier=1.5)
_, ss_clean = ss.detect_poison()

# is_clean lists: 1 = clean, 0 = flagged
suspects = [i for i, (a, s) in enumerate(zip(ac_clean, ss_clean)) if 0 in (a, s)]

Take the union of flags and hand-review a sample. Both methods go blind against backdoors trained to keep poisoned and clean representations inseparable, which is exactly what Tan and Shokri built. Library tradeoffs are covered in ART, Foolbox and torchattacks compared.

Model-level detection: reverse-engineer the trigger

Neural Cleanse (Wang et al., IEEE S&P 2019) frames detection as optimization: for each label, find the smallest mask and pattern that flips clean inputs to that label. A backdoored label needs a far smaller mask than a clean one, because the model already learned a shortcut to it. It is the same machinery as a universal adversarial perturbation search, constrained to a patch.

# Neural Cleanse-style scan (simplified: fixed lam, one pass per label)
norms = []
for target in range(num_classes):
    mask_raw = torch.zeros(1, H, W, requires_grad=True)
    pat_raw = torch.zeros(C, H, W, requires_grad=True)
    opt = torch.optim.Adam([mask_raw, pat_raw], lr=0.1)
    for x, _ in clean_loader:
        m, p = torch.sigmoid(mask_raw), torch.sigmoid(pat_raw)
        x_t = (1 - m) * x + m * p                    # stamp candidate trigger
        y_t = torch.full((len(x),), target)
        loss = F.cross_entropy(model(x_t), y_t) + lam * m.sum()   # L1 on mask
        opt.zero_grad(); loss.backward(); opt.step()
    norms.append(torch.sigmoid(mask_raw).sum().item())

l1 = torch.tensor(norms)
med = l1.median()
mad = 1.4826 * (l1 - med).abs().median()
anomaly = (med - l1) / mad                           # small-mask outliers only
infected = (anomaly > 2).nonzero().flatten().tolist()

An anomaly index above 2 is the paper’s cutoff, roughly a 95% outlier bound under a normality assumption. The reference implementation demonstrates it on a GTSRB traffic-sign model, and ART wraps the approach as KerasNeuralCleanse with unlearning, pruning and filtering mitigations. Cost scales linearly with class count: a 1,000-class model means 1,000 optimizations.

ABS (Liu et al., CCS 2019) skips the per-label brute force. It stimulates individual neurons, flags those that drive one output label regardless of input, then reverse-engineers a trigger from them to confirm. MNTD (Xu et al., IEEE S&P 2021) drops trigger assumptions entirely and trains a meta-classifier on a generated population of clean and trojaned models, needing only black-box access. Its authors report 97% detection AUC, and 90% for a robust variant facing an attacker who knows the detector.

Hold those numbers loosely. Qiu et al. examined Neural Cleanse, ABS and MNTD and found non-robust cases in each without any adaptive attack, just by varying task, architecture, dataset or defense hyperparameters. They singled out MNTD as less robust than claimed.

Input-level detection: STRIP at inference

STRIP (Gao et al., ACSAC 2019) assumes a deployed model you can’t vouch for. Superimpose each incoming input on clean held-out images and measure prediction entropy. A clean input’s prediction scatters once blended with random images; a triggered input stays pinned to the target label because the trigger survives the blend.

def strip_entropy(model, x, holdout, n=100):
    idx = torch.randint(len(holdout), (n,))
    blended = 0.5 * x.unsqueeze(0) + 0.5 * holdout[idx]   # superimpose
    probs = F.softmax(model(blended), dim=1)
    return -(probs * probs.clamp_min(1e-12).log()).sum(1).mean()

# Flag inputs whose entropy falls below a threshold set on clean traffic
# at your chosen false-rejection rate. Low entropy = trigger dominates.

The paper reports a false acceptance rate under 1% at a preset 1% false rejection rate across MNIST, CIFAR10 and GTSRB. The price is n forward passes per request. The bet breaks on triggers that don’t dominate the prediction, and the superimposition trick is built for images, so other modalities need their own perturbation scheme.

Do backdoor detectors work on LLMs?

Only partially. The image-classifier toolkit assumes a continuous input space and a small label set; an LLM has discrete tokens and open-ended output, so “smallest patch that flips the label” has no clean analogue. The Sleeper Agents models wrote secure code when the prompt said 2023 and exploitable code when it said 2024, and that behavior persisted through safety training, most strongly in the largest models.

The most encouraging LLM result is Anthropic’s April 2024 follow-up: linear probes on residual-stream activations at a middle layer, trained on generic contrast pairs like “Are you a helpful AI?” answered yes or no, reached 99.3% AUROC at predicting defection in the code-vulnerability sleeper agent. The same post names the catch: the feature may be an artifact of how the backdoor was inserted, and the probes were not tested on naturally arising deception. Treat it as a research lead, not a scanner.

OWASP LLM04:2025 names sleeper-agent backdoors explicitly, and its prevention list leans on tracking data origins, data version control, anomaly filtering and red teaming.

Why do backdoor detectors fail against adaptive attackers?

They fail because each detector measures one side effect, and an attacker who knows which one can train it away. Add a loss term that keeps poisoned activations inside the clean cluster and the data-level scans go blind. Use a large or input-specific trigger and Neural Cleanse’s small-mask assumption no longer holds.

The evasion literature already paid for this lesson: a defense tested only against non-adaptive attacks hasn’t been tested. BackdoorBench (NeurIPS 2022 Datasets and Benchmarks) runs 8 attacks against 9 defenses across 5 poisoning ratios, 5 models and 4 datasets, 8,000 evaluation pairs in total. Check that grid before trusting any single paper’s headline number. The TrojAI leaderboard hosted by NIST took the population view: each round scored detectors on a set of models exactly half of which were poisoned, ranked by ROC-AUC. That is the right frame. A detector is a classifier with a false-positive rate, and you need to know yours.

What to do

1. Pick detectors by access, not citation count. Own the training run: data-level scans plus fine-pruning. Third-party checkpoint: Neural Cleanse or ABS-style trigger search plus behavioral tests on your own held-out data. API only: query-based analysis and STRIP-style filtering where the modality allows blending.
2. Calibrate on a population. Train a few dozen copies of your architecture, backdoor half with varied triggers, and measure ROC-AUC before setting a threshold. One “anomaly index 3.6, infected” result says nothing about false positives.
3. Attack your own detector. Retrain the backdoor with a loss term targeting your detector’s statistic, Tan and Shokri style, and report the drop. Skip this and you have measured the detector against attackers who don’t read papers.
4. Separate detection from removal. Fine-pruning, which prunes dormant neurons and then fine-tunes on clean data, cut attack success to 0% in some cases with a 0.4% clean-accuracy drop; the authors found pruning or fine-tuning alone insufficient. For LLMs, Sleeper Agents says safety training is not a removal tool.
5. Pin provenance. Hash weights and datasets at download, record the source, and treat an unpinned from_pretrained on a mutable branch as an unreviewed dependency. Not loading untrusted weights is the first line of defense; detection is the second.

FAQ

can a backdoor survive fine-tuning

Yes, often. Sleeper Agents (Hubinger et al., 2024) found LLM backdoors persisted through supervised fine-tuning, reinforcement learning and adversarial training, most strongly in the largest models. For image classifiers, Fine-Pruning showed plain fine-tuning alone was not a sufficient defense; pruning dormant neurons before fine-tuning on clean data worked better.

what is the difference between a backdoor attack and an adversarial example

A backdoor is planted at training time; an adversarial example is crafted at inference time against an unmodified model. The backdoor attacker controls a trigger the model learned to obey, so one patch works on any input. The adversarial-example attacker searches for a perturbation per input or, at most, a universal one.

is neural cleanse still effective

Neural Cleanse still catches small, patch-style triggers like BadNets, but it is not a general scanner. It assumes the trigger is small relative to the input, so large or input-specific triggers break its core assumption, and Qiu et al. found non-robust cases even without adaptive attacks. Per-label optimization also gets expensive on models with many classes.

what tools detect backdoors in neural networks

The Adversarial Robustness Toolbox (ART) is the practical starting point: it ships activation clustering, spectral signature, provenance and RONI poison detectors, plus a Neural Cleanse wrapper with unlearning, pruning and filtering mitigations. For comparing detectors across many attacks, BackdoorBench’s 8,000 attack-defense evaluation pairs give you the reference grid.