What Is Explainable AI?

Explainable AI, abbreviated XAI, is the branch of artificial intelligence research concerned with producing systems whose outputs can be accounted for in terms a human can understand and check. The problem arises because the model families that perform best on perception and language tasks, notably deep neural networks and large ensembles, distribute their decision logic across millions or billions of parameters with no readable correspondence between a parameter and a concept. Explainability is the property of being able to say why a particular output was produced, and interpretability is the related and stronger property of being able to inspect the model's mechanism directly.

The field acquired its current shape after 2016, when the DARPA Explainable Artificial Intelligence program funded work aimed at machine learning techniques that produce explainable models while retaining high predictive performance, so that users could understand, appropriately trust, and effectively manage the systems they operate. Regulatory pressure followed, with data protection and sector-specific rules in finance, medicine, and employment requiring that consequential automated decisions be capable of being explained to the people they affect.

Interpretable Models and Post Hoc Explanation

Two broad strategies exist. The first is to use a model that is interpretable by construction: linear and logistic regression with a modest number of features, shallow decision trees, rule lists, generalized additive models, and scoring systems. Their structure is the explanation, and where predictive performance is comparable, this route avoids the fidelity problems that afflict the alternative. The second strategy is post hoc explanation, in which an opaque model is trained first and a separate procedure produces an account of its behavior afterward. Post hoc methods divide into global explanations, which characterize the model overall through surrogate models, partial dependence, or feature importance rankings, and local explanations, which account for a single prediction. The central risk is infidelity: an explanation is itself an approximation, and a plausible explanation that does not reflect the model's actual computation can increase misplaced confidence rather than reduce it.

Attribution and Example-Based Methods

Feature attribution is the most widely deployed family of local methods. LIME, introduced in the 2016 paper Why Should I Trust You?, fits a simple interpretable model to samples drawn in the neighborhood of the instance being explained, and reports the resulting coefficients as the contribution of each feature. SHAP recasts attribution in terms of Shapley values from cooperative game theory, which gives the method desirable additivity and consistency properties at substantial computational cost. Gradient-based methods including integrated gradients and layer-wise relevance propagation exploit differentiability to trace an output back through the network to input pixels or tokens. Saliency maps present the result visually for images, while counterfactual explanations take a different tack, reporting the smallest change to the input that would flip the decision, which is often more actionable for an affected person. Example-based methods instead surface influential training instances or prototypes.

Evaluation and Standards

Because an explanation is produced for a human, its quality cannot be assessed by fidelity metrics alone. NIST set out four principles of explainable artificial intelligence in NISTIR 8312: a system should deliver accompanying evidence for its outputs, the explanation should be meaningful to its intended audience, it should accurately reflect the process that generated the output, and the system should operate only within the conditions for which it was designed and signal when it reaches its knowledge limits. Evaluation practice combines computational proxies, such as faithfulness under feature deletion and stability under small input perturbations, with human subject studies measuring whether an explanation improves task performance or calibrated trust. Studies repeatedly find that explanations can induce overreliance, so evaluation with actual users remains necessary.

Applications

Explainable AI is applied in fields including:

• Clinical decision support and medical imaging
• Credit scoring, underwriting, and fraud detection
• Regulatory compliance and algorithmic auditing
• Autonomous vehicles and safety-critical control
• Model debugging and data quality diagnosis
• Scientific discovery from learned models