AI systems mirror the world they were trained on - including its prejudices. How to recognise bias and what you can do about it.
In 2018 Amazon discovered that its experimental CV screener was rating women lower. The reason was simple and painful: the system had been trained on ten years of successful CVs - which were predominantly men’s. The AI did exactly what it had learned. And that was the problem.
What is bias in AI?
Bias is a systematic deviation. An AI model is biased when it consistently treats certain groups differently without a legitimate reason. Bias is almost never the result of bad intent; it is an artefact of the data, choices made during training and the context in which the model is deployed.
Five sources of bias
- 1Historical bias: the training data reflects an unequal world (as with Amazon’s CV screener).
- 2Selection bias: certain groups are under- or over-represented in the data.
- 3Measurement bias: features are measured or labelled differently across groups.
- 4Aggregation bias: a single model is applied to a diverse population while subgroups behave differently.
- 5Deployment bias: the system is used in a context it was not designed for.
Where it goes wrong in practice
Face recognition often identifies white men more accurately than women or people of colour. Speech recognition struggles with regional accents. Translation models guess gender from stereotypes (“The doctor said he…”, “The nurse said she…”). Credit models can use postcode as a proxy for ethnicity.
These are not edge cases. For someone rejected for a job, loan or home, this is the whole story.
How do you spot bias as a user?
You don’t need to be a data scientist to notice bias. Three simple checks:
- Test with variations: run the same prompt or input with different names, genders, ages or regions. Does the output change without a good reason? That is a signal.
- Ask the model to explain: a justification often reveals its assumptions. If those assumptions are stereotyped, you know where you stand.
- Look at error distributions: a model that is 95% accurate overall but 70% on a subgroup hides that gap in its average.
What the EU AI Act requires
For high-risk AI, there must be a data quality analysis including potential bias. The system must operate under human oversight and produce logs you can analyse afterwards. Bias management is no longer an ethical bonus - it is a legal requirement.
What you can do today
- 1Make the problem explicit: which groups could this system affect?
- 2Collect or request disaggregated performance figures (per subgroup, not just averages).
- 3Build a human review step into decisions with significant impact.
- 4Document what you tested and found - an audit trail protects you and your users.
- 5Train everyone involved in basic AI ethics; this is not only for engineers.
A neutral AI does not exist. A responsible AI does.
Bias does not disappear on its own
Models get better, but bias is a property of data - not of algorithms. As long as we train models on the world as it is, they learn the world as it is. It is up to us to decide what we want to repeat and what we want to correct.
In our AI literacy course you practise bias detection with real-world cases - from recruitment to healthcare.
