What Is Adversarial Machine Learning?

Adversarial machine learning is a field within artificial intelligence security concerned with understanding how machine learning models can be deceived, manipulated, or exploited by adversarial inputs, and with designing models and systems that are resistant to such manipulation. The field spans both offensive research, which characterizes the vulnerabilities of existing systems, and defensive research, which develops techniques for detecting, mitigating, or certifying resistance to attacks. NIST formalized a taxonomy and terminology for the field in its NIST AI 100-2 E2025 publication on adversarial machine learning attacks and mitigations, defining attack life cycle stages, attacker capabilities, and corresponding defense strategies.

Adversarial machine learning draws on statistical learning theory, optimization, cryptography, and computer security. The central observation motivating the field, established by Szegedy et al. in 2013, is that small perturbations to an input, imperceptible to a human observer, can cause a neural network classifier to produce a completely wrong output with high confidence. This fragility is not a property of a specific model but appears across many architectures and tasks.

Attack Methods

Attacks on machine learning systems are categorized by the stage of the model life cycle at which they operate and by the attacker's knowledge of and access to the target system. Evasion attacks occur at inference time: an attacker constructs a malicious input, often called an adversarial example, that is misclassified by the model while appearing benign to a human reviewer. Gradient-based methods such as the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) compute the input perturbation direction that maximally increases the model's loss, producing adversarial examples efficiently from first-order information. Data poisoning attacks target the training phase: the attacker injects carefully crafted samples into the training dataset so that the resulting model learns a hidden behavior, such as misclassifying all inputs bearing a specific trigger pattern (a backdoor attack). Privacy attacks, including model inversion and membership inference, aim to extract information about the training data rather than to alter model predictions. NIST's identification of cyberattack types that manipulate AI behavior provides a practitioner-oriented summary of the attack taxonomy.

Defense Strategies

Defenses against adversarial attacks fall into three broad categories: training-time interventions, inference-time detection, and formal verification. Adversarial training, the most widely studied training-time defense, augments the training dataset with adversarial examples generated on-the-fly so that the model learns to correctly classify perturbed inputs. While adversarial training demonstrably improves empirical robustness, it carries computational cost and can reduce accuracy on clean inputs. Certified defenses provide mathematical guarantees that no perturbation within a specified norm ball can change the model's prediction, using techniques such as randomized smoothing or interval bound propagation. At inference time, anomaly detection approaches monitor input statistics or intermediate activations for patterns characteristic of adversarial perturbations, flagging suspicious inputs before they reach the prediction layer. For large language models, defenses address prompt injection, jailbreaking, and extraction attacks that exploit the model's instruction-following behavior. Red-teaming, in which trained specialists attempt to elicit harmful or erroneous outputs, serves as an operational complement to algorithmic defenses.

Robustness Evaluation

Evaluating the robustness of a machine learning model against adversarial perturbations requires standardized benchmarks and threat models. A threat model specifies what an attacker knows (white-box access with full model knowledge, or black-box access using only output queries), what perturbation magnitude is considered realistic, and what objective the attacker pursues. RobustBench maintains a standardized leaderboard for adversarial robustness on image classification benchmarks, allowing direct comparison of defenses under consistent threat model assumptions. Robustness claims made without a stated threat model cannot be meaningfully interpreted, because a defense that succeeds against gradient-based attacks may fail against query-based or transfer attacks.

Applications

Adversarial machine learning has applications in a wide range of disciplines, including:

• Autonomous vehicle perception, ensuring object detection and classification systems resist physical-world adversarial patches
• Malware detection, hardening classifiers against evasion by attackers who can modify malware features
• Facial recognition and biometric authentication, detecting spoofing attempts using adversarial masks or printed patterns
• Natural language processing, protecting text classification and moderation systems from adversarial rephrasing
• Medical image analysis, validating that diagnostic AI is not deceived by imperceptible perturbations of clinical images