Bias in machine learning

Bias in machine learning means that an AI system systematically treats some groups of people worse than others, often because it learned from data in which that unequal treatment was already present. A recruitment tool trained on past applications, most of them from men, can start to favour men. For a business it matters whenever AI helps decide about people, such as job applicants or customers.

Where does bias come from?

Suresh and Guttag (MIT, 2019) describe seven sources of harm across the life of a model. Three of them sit in the data:

• Historical bias: the data reflects the world as it was, so even perfectly measured data can teach a model an unfair pattern.
• Representation bias: some groups are underrepresented, so the model works worse for them.
• Measurement bias: a feature or label is a poor stand-in for what you mean, such as arrests as a measure of crime, while some communities are policed more heavily than others.

The other four are aggregation, learning, evaluation and deployment bias: one model for groups that differ, modelling choices that widen differences, a test set that does not match the users, and use for a purpose the model was not built for.

Examples

• Amazon trained a system to rate CVs on ten years of applications, most of them from men, Reuters reported in October 2018. It penalised CVs containing the word "women's" and downgraded graduates of two all-women's colleges. Amazon disbanded the team and said recruiters never used the tool to evaluate candidates.
• Gender Shades (Buolamwini and Gebru, 2018) tested three commercial systems that classify gender from a face. Error rates reached 34.7% for darker-skinned women and stayed at most 0.8% for lighter-skinned men.
• The Dutch tax authority used applicants' nationality (Dutch or not) as an indicator in a system that automatically flagged childcare benefit applications as risky. In December 2021 the Dutch Data Protection Authority fined it €2.75 million for this and other discriminatory processing of nationality.
• Language models are not exempt. In 2024 Wilson and Caliskan (University of Washington) let embedding models rank more than 500 CVs: they favoured names associated with white people in 85.1% of the comparisons, and names associated with women in only 11.1%.

How do you test and limit it?

• Compare outcomes per group, such as the share of men and women that get through a CV screen, before and after going live, and keep a person involved in decisions about people (human-in-the-loop).
• Deleting the sensitive field is not enough, because other data can stand in for it. Amazon edited its system to be neutral to the terms it had found, and according to Reuters that was no guarantee against other ways of sorting candidates that could prove discriminatory.
• Choose a definition of fairness deliberately. Kleinberg, Mullainathan and Raghavan proved in 2016 that three common fairness conditions for risk scores cannot all hold at once, except in special cases.
• Under the EU AI Act, providers of high-risk AI such as CV screening must examine their training data for possible biases and take measures to detect, prevent and mitigate them (Article 10); those rules apply from 2 December 2027. Since the AI Omnibus of July 2026, providers and users of other AI systems may also process sensitive data such as ethnic origin to detect and correct bias, as an exception and under strict conditions.