CLIP Adversarial Attacks: One Perturbation, Every Model

Almost every multimodal system in production reads its pixels through the same component. CLIP, or a close derivative of it, is the vision encoder inside LLaVA, MiniGPT-4, OpenFlamingo, most open captioning and VQA stacks, the conditioning path of several diffusion models, and a large share of the image-retrieval and image-moderation pipelines that never get called “AI systems” at all. That concentration is convenient for builders and it is the single most useful fact an adversary has about the ecosystem.

The consequence is a threat model that classical adversarial ML does not have a good vocabulary for. In the classifier era, an attack was aimed at a model: you perturbed an input to flip one network’s decision, and transferability to other models was a bonus that decayed as architectures diverged. Against CLIP, the attack is aimed at a representation that dozens of downstream systems share verbatim. Transfer is not a bonus. It is the design.

Why the contrastive embedding is the weak point

CLIP trains an image encoder and a text encoder jointly so that matched image-caption pairs land close together in a shared space and mismatched pairs land apart. Everything downstream consumes that space: zero-shot classification compares an image embedding to a set of label embeddings, retrieval ranks by cosine similarity, and an instruction-tuned multimodal LLM projects the image embedding into its token space and generates text conditioned on it.

Two properties make this attackable in a way a plain classifier is not.

First, the target is continuous rather than discrete. An attacker does not have to flip a label; they have to move a point. The optimisation objective is a distance in embedding space, which is smooth, differentiable, and does not require knowing anything about the downstream task. The same machinery from FGSM, PGD and C&W applies directly, with the loss swapped for an embedding distance.

Second, the downstream model never sees the image. It sees the embedding. A safety filter that inspects pixels, a moderation classifier that runs on the raw upload, an OCR pass that looks for suspicious text — none of them are positioned to notice that the vector handed to the language model no longer corresponds to the picture. The perturbation is laundered through the encoder into a representation nobody audits.

Four attack surfaces, in order of how much they generalise

Targeted embedding manipulation. Solve for a perturbation that drives an image’s embedding toward a chosen target — another image’s embedding, or a text embedding for a caption the attacker picks. The result is an image that retrieves for an arbitrary query, that a zero-shot classifier assigns to an arbitrary label, and that a diffusion model conditioned on CLIP image embeddings will treat as a different reference entirely. This is per-image and requires white-box access to an encoder, but the encoder is usually an open checkpoint, so “white-box” is a weak constraint here.

Typographic attacks. Goh et al. documented in Distill’s multimodal neurons work that simply writing a word on an object causes CLIP to classify the image as the word rather than the object. No gradients, no optimisation, no model access. Any moderation or retrieval pipeline that relies on CLIP semantics inherits this, and it is the cheapest attack in the family by several orders of magnitude.

Universal perturbations. A single fixed pattern, added to any image, that degrades or redirects the encoder’s output across the whole input distribution. This is the CLIP instance of the geometry described in the universal adversarial perturbations literature: shared dominant directions in the representation that a single vector can exploit for every input rather than one at a time.

Visual jailbreaks. Qi et al. (arXiv:2306.13213) showed that an adversarial image supplied alongside an otherwise-refused text instruction causes safety-trained multimodal models to abandon their refusal behaviour. The image carries no readable content and triggers no image-content filter; its effect exists entirely in the embedding. The broader picture of that failure mode is covered in the site’s vision-language model attack overview.

X-Transfer and the “super transferability” result

The 2025 result that changes the risk calculus is X-Transfer (Huang, Erfani, Li, Ma and Bailey, arXiv:2505.05528, ICML 2025). The paper reports a single universal adversarial perturbation against CLIP that transfers simultaneously along four axes the literature had previously treated separately: across data (any input image), across domains, across models (different CLIP encoders and the large vision-language models built on them), and across tasks (zero-shot classification, image-text retrieval, captioning, visual question answering).

The mechanism the authors credit is surrogate scaling. Crafting a UAP against an ensemble of surrogate encoders is a known way to buy transferability, but the ensemble is expensive and most of it is wasted. X-Transfer searches a large pool of candidate surrogates and dynamically selects a small suitable subset during optimisation, which is what makes the scale affordable. The published comparison places the resulting perturbations ahead of both the classifier-era universal methods (GD-UAP, TRM-UAP, Meta-UAP) and the CLIP-specific ones (AdvCLIP, C-PGC, ETU), across zero-shot classification on eight datasets, image-text retrieval in both directions, captioning and VQA.

The practical significance is the release rather than the numbers. The accompanying XTransferBench toolkit ships pre-generated perturbations, indexed by surrogate-search scale, perturbation budget and targeted or non-targeted objective, as a plug-in artefact. An attack that previously required a research cluster and a white-box surrogate ensemble is now a downloaded tensor added to an image. Nothing about the attacker’s access to the victim system is required; the perturbation was computed against the shared encoder, offline, once.

For anyone maintaining a threat model, that is the same discontinuity that GCG-class suffix attacks produced on the text side. The attack stops being a capability and becomes an asset that anyone can hold.

The question that is usually asked backwards

Comparisons between CLIP and a detector such as a YOLO variant come up constantly, and they are almost always framed as “which one is more robust.” That framing does not survive contact with the threat model, because the two fail at different layers and for different reasons.

A supervised object detector is trained on a closed label set with a task-specific head. Attacks against it are the patch and physical-world family: localised, visible, printable, and specific to that model and that label set. They generally do not transfer well to a model trained on a different dataset with a different head.

CLIP has no label set and no task head. Its output is a general-purpose representation, and its failure mode is that the representation itself is steerable and that everything downstream trusts it. The relevant comparison is therefore not accuracy under attack, but blast radius: a broken detector is one broken component, while a steerable encoder is a shared dependency with a universal perturbation available for it. A pipeline that uses both should model them separately rather than picking a winner.

What actually reduces exposure

Adversarially fine-tuned encoders are the only defence with real evidence behind them. Schlarmann, Singh, Croce and Hein’s Robust CLIP (arXiv:2402.12336, ICML 2024) introduces FARE, an unsupervised adversarial fine-tuning scheme that keeps the perturbed image embedding close to the clean embedding, requiring no labels and no text encoder. The paper positions FARE against the earlier supervised TeCoA approach and reports better retention of clean performance across downstream tasks. The relevant property for a deployment is that the robust encoder is a drop-in replacement for the original checkpoint: downstream models keep working, and the attack surface shrinks without retraining the language model.

Input transformations are not a defence here. JPEG compression, resizing and random cropping have the same history against CLIP that they had against classifiers — they raise the attack cost slightly, break under expectation-over-transformation optimisation, and mainly produce robustness numbers that do not survive an adaptive attacker. The evaluation discipline in evaluating adversarial robustness without fooling yourself applies unchanged.

Certification does not extend. Randomised smoothing certifies a classifier’s label within an L2 radius. There is no label to certify in a contrastive embedding, and certifying the encoder would not bound the behaviour of a language decoder conditioned on it. Treat any certified claim about a full multimodal stack as unproven.

Architectural controls do the rest. Assume any image reaching the encoder can steer it, and place the trust boundary after the model rather than before it: constrain what the model’s output is allowed to trigger, keep tool invocation and data access behind explicit authorisation, and monitor for the retrieval and moderation anomalies that embedding manipulation produces.

Assessment checklist

• Which CLIP checkpoint is in the stack, and is it the public one an attacker can optimise against offline?
• Do user-supplied images reach the encoder, and is anything inspecting the image before it does?
• Does a downstream action — a tool call, a moderation verdict, a retrieval ranking — depend on the embedding without a second signal?
• Is the encoder replaceable with an adversarially fine-tuned variant, and has anyone costed the clean-performance delta?
• Are typographic attacks covered at all, given they need no model access whatsoever?

To work through which attacks are feasible against a specific access level, modality and adversary goal, the site’s attack selector filters the catalogue by those constraints and flags the adaptive attacks that defeat common defences.

References

• Huang et al., “X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP,” ICML 2025, arXiv:2505.05528
• Schlarmann et al., “Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models,” ICML 2024, arXiv:2402.12336
• Qi et al., “Visual Adversarial Examples Jailbreak Aligned Large Language Models,” AAAI 2024, arXiv:2306.13213
• Goh et al., “Multimodal Neurons in Artificial Neural Networks,” Distill 2021

See also

• AI attack techniques
• AI security blog