Explainable AI

Explainable AI is AI whose results you can trace back to a reason: which information weighed most when a system turned down a loan application or ranked a job applicant low. Without that, you cannot tell whether the system works as intended, or answer a customer who asks why. For some automated decisions, EU law also gives people a right to an explanation.

What is the difference between explainability and interpretability?

Researchers do not agree on exact definitions, notes Christoph Molnar in his book Interpretable Machine Learning. A common one: interpretability is "the degree to which a human can understand the cause of a decision". Molnar treats explainability as the stronger term, interpretability plus the context a person needs. There are two routes: a model that is readable by design, such as a decision tree, or an explanation added afterwards to a model too complex to read, a black box.

How do you explain a black box?

• LIME (Ribeiro et al., 2016) varies the input of one case many times and fits a simple model that imitates the black box around that case.
• SHAP (Lundberg and Lee, 2017) uses Shapley values from game theory to divide the gap between one prediction and the average prediction among the input features.
• A counterfactual explanation names the smallest change that would have flipped the outcome, such as a lower monthly debt.
• Attention weights in a transformer are no reliable explanation: Jain and Wallace (2019) found that very different attention patterns can give the same prediction.

Can a language model explain itself?

You can ask a large language model why it answered as it did, or read its chain of thought, yet that text need not reflect how the answer came about. In a test by Anthropic (April 2025), models were slipped a hint about the answer. When they used it, Claude 3.7 Sonnet mentioned the hint in its reasoning 25% of the time and DeepSeek R1 39% of the time.

Mechanistic interpretability looks inside the model instead. In May 2024 Anthropic reported that it had extracted millions of features from Claude 3 Sonnet: patterns of neuron activity that match a concept. Amplifying the feature for the Golden Gate Bridge made the model bring up the bridge in almost every answer. Anthropic called what it found a small subset of what the model had learned.

Where does it go wrong?

• Explanations can be manipulated. Slack et al. (2019) built deliberately biased classifiers whose LIME and SHAP explanations looked harmless.
• A readable model is not always less accurate. Cynthia Rudin (2018) calls that trade-off a myth for structured data with meaningful features, such as credit data, and argues for interpretable models in high-stakes decisions.

What does the law require?

Under the GDPR, a person subject to an automated decision with significant effects can ask for meaningful information about the logic involved (Articles 15 and 22). In case C-203/22 (27 February 2025), about a woman refused a €10-a-month phone contract after an automated credit check, the EU Court of Justice ruled that this means explaining the procedure and principles actually applied, in a form she can understand. A formula or an algorithm alone is not enough; showing how different data would have changed the result can be.

The AI Act adds duties for high-risk systems such as credit scoring: they must be transparent enough for the business using them to interpret the output (Article 13), and affected people can ask for a clear explanation of the AI's role in a decision (Article 86). When these rules apply is set out under AI governance.

Example

Suppose a small lender uses a model to approve consumer loans, and a rejected applicant asks why. SHAP shows that her debt-to-income ratio pulled her score down most, followed by a short credit history. A counterfactual explanation adds that a lower monthly debt would have led to approval, the kind of answer the Court of Justice mentioned. Whether both match what the model really does is a separate test for the lender.