Explainable AI and AI Ethics
Introduction
What Question Does It Ask?
This course studies how to explain an AI system's decisions in a way people can understand, and how to make the system fair, reliable and accountable. The technical question is "why did the model decide this?"; the ethical question is "whom can this decision harm, and who is responsible?"
At a glance
human-centered evaluation
model, its data and context of use
explanation, audit report, principles
interpretability, fairness, accountability
low-medium
the model's own data
Introduction
A Short History
ProPublica publishes an analysis arguing that the COMPAS recidivism score is biased against Black defendants.
Ribeiro, Singh and Guestrin introduce LIME: explaining one prediction with a simple local model.
Lundberg and Lee publish SHAP: feature contributions from game-theoretic Shapley values.
The EU General Data Protection Regulation (GDPR) takes effect, granting a right to contest automated decisions.
The Gender Shades study shows commercial face analysis systems err far more often on darker-skinned women.
The EU AI Act is adopted: systems are classified by risk level.
The Course
Core Concepts
A model whose inner workings people cannot follow.
A model readable by design: decision tree, linear.
Explaining one prediction: LIME, SHAP values.
The model's overall behavior: feature importance, PDP.
"Had your income been this much, the loan would be approved."
Imbalance in training data carried into decisions.
Demographic parity, equal opportunity, calibration.
Auditing, logging and ownership of a decision.
The Course
A Typical Workflow
The Course
What Does the Course Cover?
- Interpretable models: linear models and trees
- Feature importance and partial dependence (PDP, ICE)
- Local explanations: LIME
- Shapley values and SHAP
- Saliency maps for deep networks
- Counterfactual explanations and recourse
- Evaluating explanations: faithfulness and stability
- Fairness metrics and the conflicts between them
- Bias mitigation: pre-, in- and post-processing
- Privacy: anonymization and differential privacy
- Accountability: model cards, datasheets, audits
- Regulation: GDPR and the EU AI Act
Skills you will gain
A typical semester; the order and coverage vary between universities.
Try It
Is the Credit Model Fair to Both Groups?
The model scores each application from 0 to 100; scores above the threshold are approved. Group B scores lower because of historical data. Move the thresholds and watch how two fairness metrics change.
Context
Prerequisites
Courses worth knowing before this one, and why they are needed.
| Course | Why it is needed |
|---|---|
| Programming | training models in Python and analyzing results |
| Algorithmic Game Theory | the Shapley value: the theoretical root of SHAP explanations |
| Philosophy or law | ethical theories, personal data rights and the notion of responsibility |
Context
Are You Ready for This Course?
Ask yourself; for every "no", the right-hand column shows where to start.
| Ask yourself | If not, start with |
|---|---|
| Can you train a classifier in Python and evaluate it on a test set? | Machine Learning |
| Can you compute false positive and false negative rates? | A short reading on confusion matrices and classification metrics |
| Can you explain conditional probability and base rates? | Probability and Statistics |
| Can you explain what the Shapley value means in game theory? | The Shapley value topic of the Algorithmic Game Theory course |
Context
Neighbouring Courses
Overlap
Overlapping Courses
| Relation | Course | Shared ground |
|---|---|---|
| shares methods | Shares methods with Machine Learning | model explanation, feature importance |
| shares methods | Shares methods with Expert Systems and Knowledge Representation | explainable decisions through readable rules |
| shares methods | Shares methods with Medical Image Processing | explainability and accountability in diagnosis |
| shares methods | Shares methods with Data Science | data bias, fairness metrics |
| shares methods | Shares methods with AI in Cybersecurity | robustness, adversarial examples |
Differences
Where It Differs (1/2)
| Criterion | Explainable AI & Ethics | Machine Learning | Data Science |
|---|---|---|---|
| Core question | When can we trust an AI system's decision, and who is responsible for the outcome? | How does a program generalize from examples? | How do we ask the right questions of data and turn the answers into decisions? |
| Approach | human-centered evaluation | statistical learning | question, data, analysis, story |
| Input | model, its data and context of use | labeled or unlabeled examples | raw, messy data |
| Output | explanation, audit report, principles | a model that makes predictions | report, visualization, model |
| Success measure | interpretability, fairness, accountability | error on unseen data | impact on decisions, accuracy |
| Math load | low-medium | high: linear algebra, probability, optimization | medium: statistics |
| Data needs | the model's own data | moderate to large | moderate to large |
- Machine Learning: ML focuses on the model predicting correctly; this course asks why it predicts and whom it affects, and how.
- Data Science: Data science draws conclusions from data; this course checks whether those conclusions are biased, fair or explainable.
Differences
Where It Differs (2/2)
| Criterion | Explainable AI & Ethics | AI in Cybersecurity | Expert Systems |
|---|---|---|---|
| Core question | When can we trust an AI system's decision, and who is responsible for the outcome? | How does AI catch attacks, and how is it attacked itself? | How is an expert's knowledge written down and used for inference? |
| Approach | human-centered evaluation | security knowledge plus ML | symbolic logic and rules |
| Input | model, its data and context of use | network traffic, logs, files | facts and expert rules |
| Output | explanation, audit report, principles | alert, class, risk score | conclusion with justification |
| Success measure | interpretability, fairness, accountability | detection rate, false alarm rate | expert agreement, explainability |
| Math load | low-medium | medium | low: propositional and predicate logic |
| Data needs | the model's own data | imbalanced and constantly changing | little; a knowledge base |
- AI in Cybersecurity: Cybersecurity protects against attackers; this course studies the harm a system does to people even without an attack.
- Expert Systems: Expert systems are already transparent through their rules; this course seeks ways to explain black-box models after the fact.
Practice
Applications
Explaining a rejection to the applicant.
Showing the physician what the model looked at.
Gender and age bias in résumé screening.
Auditing whether risk scores are fair.
Measuring error gaps between groups.
Documentation for the EU AI Act and KVKK.
Practice
Risks and Ethics
LIME and SHAP are approximations; an adversarial model can look innocent to the explainer while hiding its real decision rule.
An ethics principles document or a single fairness score can serve as a showcase in place of real auditing.
Choosing a metric is a value judgement: COMPAS could look fair under one metric and biased under another.
A plausible-looking explanation can lead users to follow a wrong decision without question.
Practice
Tools and Datasets
Tools and libraries
Well-known datasets
Practice
Common Misconceptions
| Common belief | In fact |
|---|---|
| Removing the protected attribute (gender, race) is enough. | Proxy features such as postal code or name can carry the same information; the bias remains. |
| An algorithm is neutral because it works with numbers. | A model learns the bias in past decisions from the data and scales it up. |
| An explanation shows how the model really works. | Methods such as LIME and SHAP are approximations; the explanation itself can mislead. |
| All fairness metrics can be satisfied at once. | When base rates differ between groups, calibration and equal error rates usually cannot hold together. |
Next Steps
After This Course
Courses that build on or extend this one
Related careers
Next Steps
Further Reading
- Molnar, Interpretable Machine Learning, 2nd ed., self-published (online), 2022.
- Barocas, Hardt and Narayanan, Fairness and Machine Learning: Limitations and Opportunities, MIT Press, 2023.
- O'Neil, Weapons of Math Destruction, Crown, 2016.
- Ribeiro, Singh and Guestrin, "Why Should I Trust You?: Explaining the Predictions of Any Classifier", KDD, 2016.
- Lundberg and Lee, "A Unified Approach to Interpreting Model Predictions", NeurIPS, 2017.