Computers are not automatically neutral
It is tempting to think a program must be fair because it follows rules and has no feelings. But programs are written by people, trained or tested on data that people collected, and used by people in settings with their own history. So computing innovations can reflect existing human biases. The course names two routes:
- Biases written into the algorithms: the rules, scoring formulas or decisions the developers chose.
- Biases in the data used by the innovation: who is included, who is missing, and what past decisions the data record.
And biases can be embedded at all levels of software development: in choosing the problem to solve, deciding what to measure, collecting data, writing and testing the algorithm, and deciding how results are used. Programmers should take action to reduce bias in algorithms used for computing innovations, as a way of combating existing human biases.
Here bias means a systematic unfairness in how a system treats people or groups. This topic is not about opinions or politics; it is about checking carefully whether a computing system treats groups equally and what can be done if it does not.
How biased data cause biased results
Many computing systems use data to learn patterns or to set rules. If the data are unrepresentative, the results will be too.
- Missing groups. If a face-recognition system is tested mainly on photographs of one kind of face, it may work worse on others.
- Historical patterns. If a hiring system learns from past hiring decisions, and those past decisions favoured some applicants unfairly, the system may learn to repeat them.
- Measurement choices. If a system measures something that is easy to count (for example, number of posts) instead of what it claims to measure (for example, quality of contribution), groups who differ in the easy measure may be treated differently.
As Topic 2.3 noted, bias is not eliminated by simply collecting more data if the data keep coming from the same unrepresentative source.
How algorithms can introduce bias
An algorithm can treat groups differently even when it never mentions a group. A proxy is a feature that is closely linked to a characteristic. Suppose a shortlisting rule uses “distance from the office”. If people who live far away are more likely to belong to a particular community, the rule can disadvantage that community without ever naming it.
Other sources include a poorly chosen threshold, a scoring formula that rewards what one group is more likely to have, and a program that works well in the conditions where it was tested but not elsewhere.
Detecting bias: look group by group
A single overall number can hide a problem. Imagine a system that is right 96% of the time for a large group and 60% of the time for a small group. The overall figure can still look excellent, because the large group dominates. To detect bias:
- Divide the results into groups that matter for the use.
- Compute the result for each group separately.
- Compare them and ask whether the differences are large enough to matter.
Reducing bias
Programmers and organisations can take practical steps:
- Collect representative data and say who is in and out of the data.
- Test across groups, not just overall, before and after release.
- Review features and rules for proxies of characteristics that should not matter.
- Bring in varied perspectives (Topic 1.1) at every stage, including the people who will be affected.
- Keep human review for important decisions, and give people a way to question a result.
- Monitor after release, because patterns in real use may differ from testing.
None of these removes bias with certainty. The goal is to identify it, reduce it and keep checking.
Worked example 1: the overall figure hides a gap
Scenario. A fictional recognition system is tested on 1,000 photographs: 900 from Group A and 100 from Group B. It is right on 864 of the Group A photographs and on 60 of the Group B photographs.
Task. Calculate the accuracy overall and for each group, and interpret.
Group A. 864 ÷ 900 = 0.96, so 96%.
Group B. 60 ÷ 100 = 0.60, so 60%.
Overall. The number correct is 864 + 60 = 924 of 1,000, so 924 ÷ 1,000 = 92.4%.
Interpretation. The overall figure of 92.4% sounds strong. But the system is 36 percentage points less accurate for Group B (96 - 60 = 36). The overall figure is close to Group A’s because 90% of the tests came from Group A.
What this suggests. The test set is unrepresentative (only 10% from Group B), and the training data may be too. Remedies: collect more varied training and test photographs, report accuracy for each group, and set a target for the lowest-scoring group, not just the average.
Check. The check 0.9 × 96 + 0.1 × 60 = 86.4 + 6 = 92.4 confirms the overall figure is a weighted average dominated by the larger group.
Worked example 2: a proxy rule
Scenario. A fictional company shortlists applicants with this rule, written in Marlbridge’s rendering of the course’s reference notation (AND joins two conditions; both must be true):
IF (score >= 70 AND distance < 10)
{
APPEND(shortlist, applicantNumber)
}Here score is the interview score and distance is the distance in kilometres from the applicant’s home to the office. Eight applicants:
| Applicant | Score | Distance (km) |
|---|---|---|
| 1 | 82 | 5 |
| 2 | 75 | 12 |
| 3 | 90 | 15 |
| 4 | 71 | 3 |
| 5 | 68 | 4 |
| 6 | 88 | 8 |
| 7 | 79 | 20 |
| 8 | 73 | 9 |
Task. (a) Who is shortlisted? (b) Who would be shortlisted using the score alone? (c) Evaluate the distance condition.
(a) With both conditions. Check each: 1 (82, 5): yes. 2 (75, 12): distance fails. 3 (90, 15): distance fails. 4 (71, 3): yes. 5 (68, 4): score fails. 6 (88, 8): yes. 7 (79, 20): distance fails. 8 (73, 9): yes. Shortlist: 1, 4, 6, 8 (4 applicants).
(b) Score alone (at least 70). Applicants 1, 2, 3, 4, 6, 7, 8 (7 applicants; only 5 fails).
(c) Evaluation. The distance condition removed applicants 2, 3 and 7, three people who met the score requirement, including the applicant with the highest score (90). The rule never mentions any group, but if people living far away are more likely to belong to a particular community, the rule acts as a proxy and can disadvantage that community. It may also exclude people who would be happy to travel, or work from home.
Improvements. Ask whether distance is genuinely needed for the job; if so, check which groups it excludes; if not, remove it. Test the rule’s results by group, and let applicants explain their circumstances.
Check. Both lists were produced by applying the stated rule line by line to every row, and the difference (2, 3 and 7) is exactly the applicants who fail only the distance test.
Feedback loops and other ways bias grows
Bias can strengthen itself. Imagine a fictional restaurant app that shows the most-reviewed places first. Places shown first get more visitors, so they get more reviews, so they are shown first again. A new restaurant of equal quality may never be seen, not because people disliked it but because the system’s own output changed the data it learns from. This kind of feedback loop is a reason to monitor a system after release, not only to test it before.
Other sources of bias are less obvious. A measurement choice can favour some people: a fitness band that detects steps well for people who swing their arms while walking will under-count for others. A threshold can be set at a level that suits the group the designers tested on. And testing conditions matter: a camera system tested only in bright daylight may fail at night. In each case, the lesson of this topic holds: ask who was in the data, what was measured, what was not, and how results differ by group, and keep checking after release.
Common misconceptions
- “Computers are objective, so their results cannot be biased.” They reflect the data and rules they are given.
- “If the algorithm never mentions a group, it cannot treat groups differently.” Proxy features can have the same effect.
- “A high overall accuracy means the system is fair.” Check each group.
- “Bias only comes from bad intentions.” It can arise from unrepresentative data and historical patterns.
- “More data automatically fixes bias.” Representative data and group-by-group testing are needed.
- “Bias is only a problem at the end, when the program is used.” It can be embedded at all levels.
- “Removing bias means making everything the same.” It means checking and correcting unfair differences in how people are treated.
Where this leads
Bias links back to the digital divide (5.2), since who is in the data depends on who has access, and forward to Crowdsourcing and Legal and Ethical Concerns, where fairness and responsibility are central. Practise with the practice questions, recap with the revision notes, and check your skills on the checklist.
Quick questions
What is computing bias?
Computing innovations can reflect existing human biases, either because biases are written into the algorithms or because the data they use are biased. The effect can be unfair treatment of some groups.
Can a program be biased if no one meant it to be?
Yes. Bias can come from unrepresentative data, from historical patterns in the data, or from rules that act as proxies, none of which need any intention to treat groups differently.
Is collecting more data enough to remove bias?
No. Bias is often created by the type or source of data. More data from the same source repeats the same bias. The data need to be representative, and results need to be checked across groups.
What can programmers do about bias?
They should take action to reduce it: use more representative data, test the program on different groups, review rules for proxies, include varied perspectives in the team, and monitor results after release.