Training data quality and bias explained: HSC Software Engineering Module 3
“Explain how the quality and representativeness of training data affect a model, including the risks of bias and overfitting”
A model can only learn what is in its training data. Sample bias, label bias, measurement bias and historical bias produce systematically wrong predictions for under-represented groups. Mitigations: audit and balance the data, evaluate per group, document the dataset with a datasheet, and never peek at the test set during development. Overfitting is a separate failure mode where the model memorises the training data, shown by a large training-versus-test accuracy gap; fix with more data, simpler models, regularisation or early stopping.
Imagine training a new staff member using only examples from the busiest, most chaotic day of the year. They would become very good at handling that one kind of day, and hopelessly confused by every quiet Tuesday, not because they are a bad learner, but because their "training data" never showed them what a normal day looks like. A machine learning model is exactly the same: it can only ever learn what its training examples actually contain. If those examples over-represent one group of people, one time period, or one set of conditions, the model becomes brilliant at handling that slice of reality and unreliable everywhere else, that is bias. And if you show a model so few examples, or such a strange, quirky set of examples, that it starts memorising specific details rather than the general pattern (like a student memorising the exact wording of every past exam question instead of understanding the topic), it will ace the training set and then fall apart on anything genuinely new, that is overfitting. Both failures trace back to the same root cause: the model is only ever as good, and as fair, as the data it was shown.
What this dot point is asking
NESA wants you to explain how the data used to train a model shapes its behaviour, and identify two failure modes: bias (systematic errors against specific groups) and overfitting (memorising the training data instead of learning generalisable patterns).
The answer
Garbage in, garbage out
A model can only learn what is in the data. If the training data has flaws, the model will replicate or amplify them. Common data flaws:
- Sample bias: the training data does not represent the population. A model trained on Sydney pedestrians may misbehave in Tokyo.
- Label bias: the labels themselves are wrong or reflect human prejudice. A "good employee" label assigned by biased managers reproduces their bias.
- Measurement bias: the features measure different things in different contexts (a thermometer reading differently in two cities).
- Historical bias: the data reflects past inequities. A loan approval model trained on past approvals reflects past discrimination.
- Sample bias: who or what is in the dataset does not match the deployment population.
- Label bias: the "correct answers" attached to examples are themselves wrong or prejudiced.
- Measurement bias: a feature is measured inconsistently across groups or contexts.
- Historical bias: the data is accurate, but reflects a world that was already unequal, so the model learns and continues that inequity.
A model trained on flawed data will replicate or amplify the flaw. It is not "the algorithm's fault"; the algorithm is faithfully learning whatever pattern the data actually contains.
Real examples
- Amazon's hiring tool: trained on a decade of CVs, mostly from male engineers. The model learned to downrank CVs that mentioned "women's chess club" and similar phrases. Amazon scrapped the project.
- Facial recognition disparities: commercial systems audited in 2018 had error rates under 1 percent for white men and over 30 percent for dark-skinned women, traced to the training set composition.
- Healthcare risk scores: a US algorithm using healthcare spending as a proxy for need underestimated illness in Black patients because they had less access to care, so lower past spending.
Sample bias in detail
If 90 percent of training data comes from group A and 10 percent from group B, the model's loss function rewards being right about group A more than group B. The model can achieve high overall accuracy by performing well on A and poorly on B.
Fix: oversample under-represented groups, undersample over-represented groups, or use class weights so each group contributes equally to the loss.
Overfitting and underfitting
Overfitting: the model memorises the training data. High training accuracy, low test accuracy. The model has learned noise, not patterns. Solutions: more data, smaller model, regularisation (dropout, L2 weight decay), early stopping.
Underfitting: the model is too simple to capture the patterns. Low training accuracy, low test accuracy. Solutions: more powerful model, more features, less aggressive regularisation.
from sklearn.tree import DecisionTreeClassifier
# Overfit: very deep tree memorises training data.
overfit = DecisionTreeClassifier(max_depth=None)
overfit.fit(X_train, y_train)
print("train:", overfit.score(X_train, y_train)) # very high
print("test:", overfit.score(X_test, y_test)) # much lower
# Better: limit depth to force generalisation.
fit_ok = DecisionTreeClassifier(max_depth=5)
fit_ok.fit(X_train, y_train)
print("train:", fit_ok.score(X_train, y_train)) # high
print("test:", fit_ok.score(X_test, y_test)) # close to train
Ask what the SYMPTOM is. If the model performs well overall but noticeably worse on a specific group (defined by demographic, region, or another real-world category), that is bias, a data representativeness problem. If the model performs well on the training data but noticeably worse on a held-out test set, regardless of group, that is overfitting, a generalisation problem. The two can coexist (an overfit model can also be biased), but naming the correct one, and pointing at the correct piece of evidence (per-group accuracy gap vs train/test accuracy gap), is what separates full marks from a vague "the model has a data problem" answer.
Per-group evaluation
A 95 percent overall accuracy can hide an 80 percent accuracy on one demographic. Always compute metrics per group:
import pandas as pd
results = pd.DataFrame({
"group": ["A"] * 90 + ["B"] * 10,
"correct": [True] * 88 + [False] * 2 + [True] * 7 + [False] * 3,
})
print(results.groupby("group")["correct"].mean())
# group
# A 0.977
# B 0.700
Group B's 70 percent accuracy is hidden by the 97 percent on the majority group.
Train, validation, test discipline
A clean split prevents data contamination:
- Training set (60-80 percent): used to fit the model.
- Validation set (10-20 percent): used during development to tune hyperparameters.
- Test set (10-20 percent): used once, at the end, to estimate real-world performance.
Looking at the test set during development leaks information and inflates reported accuracy.
Data documentation
Every dataset should be accompanied by:
- A datasheet explaining where the data came from, who labelled it and how.
- The demographic composition of the data.
- Known limitations (sample bias, label noise, the time period covered).
- A statement of intended use and out-of-scope use cases.
This is the "Datasheets for Datasets" standard (Gebru et al., 2018), now an industry norm.
A startup trains a model to predict secondary school graduation rates using data from 50 high-performing private schools. The model is then deployed to allocate support resources across 500 public schools.
Identify three bias risks and the likely consequences.
- Sample bias: the training schools differ from the target schools in funding, resources and student demographics. The model's predictions are unreliable for the target population.
Marker's note: name the specific mismatch (private vs public, funding, demographics), not just "the data is different".
- Outcome bias: the labels (graduated/did not) reflect what worked in the private school context. The features that predicted success there may not predict success in different conditions.
Marker's note: distinguish this from sample bias by focusing on what the LABEL means in each context, not just who is in the dataset.
- Allocation harm: if the model underestimates risk in groups under-represented in the training data, those schools get fewer resources, which makes graduation rates worse and creates a self-reinforcing cycle.
Marker's note: top answers close the loop, explaining how a biased prediction becomes a biased real-world outcome, which then further skews future data.
Mitigations: collect representative training data from a stratified sample of public schools; evaluate per school category before deployment; involve school communities in defining what success looks like; commit to ongoing monitoring with the ability to halt the system if disparities emerge.
- Treating bias as "the algorithm's fault"
- Bias usually originates in the data and the labels. The algorithm is faithfully learning patterns that the data contains.
- Reporting only overall accuracy
- A model can have 99 percent overall accuracy and 50 percent on a minority group. Markers want per-group evaluation.
- Confusing overfitting and bias
- Overfitting is when a model memorises noise in the training data. Bias is when the data systematically misrepresents the world. They are different failure modes, though both originate in the training set.
- Believing "more data" always fixes bias
- More biased data does not fix bias, it amplifies it. The data must be more representative, not just more abundant.
- Trying to remove bias by deleting the protected feature
- "Don't include gender in features" does not fix bias if other features (postcode, occupation, vocabulary) correlate with gender. Mitigation requires representative data and group-aware evaluation.
Exam-style questions
Questions in the style of NESA exam questions on this dot point, each with a worked answer. They are written by ExamExplained unless tagged "Past paper"; the year shows the paper a question is modelled on.
Practice6 marksExplain how unrepresentative training data can introduce bias into a machine learning system. Describe two methods to detect or mitigate this bias.Show worked answer →
A machine learning model can only learn patterns that exist in its training data. If the training data does not represent the population the model will be used on, the model's predictions are systematically wrong for the under-represented group. This is sample bias.
Worked example: a facial recognition model trained mostly on photos of light-skinned faces will be much less accurate on dark-skinned faces, because the training data did not represent that group well. A medical diagnosis model trained on male patients may miss conditions that present differently in female patients. A hiring screening model trained on past hires reflects past biases (e.g. preferring men if the company historically hired men).
Mitigation 1: audit the training data. Measure the representation of each demographic group in the dataset. If a group is under-represented, collect more examples from that group or use weighted sampling so under-represented examples count more during training.
Mitigation 2: evaluate per group. Compute accuracy, false positive rate and false negative rate separately for each demographic group, not just overall. A model with 95 percent overall accuracy might be 99 percent accurate on group A and 70 percent on group B - the per-group view exposes the problem.
Other valid answers: fairness-aware training objectives, removing protected attributes from features (with the caveat that proxy features can still leak the attribute), engaging affected communities in the model design.
Markers reward a clear definition of bias originating from data, a real-world example, and two distinct detection or mitigation methods (data-side and evaluation-side are the cleanest).
Practice questions
Original practice questions graded from foundation to exam level, each with a full worked solution. Try them before revealing the solution.
foundation2 marksDistinguish between sample bias and label bias, giving a one-sentence definition of each.Show worked solution →
Sample bias: the training data does not represent the population the model will be used on (e.g. mostly one demographic group).
Label bias: the labels themselves are wrong or reflect human prejudice, even if the underlying examples are representative.
Marking criteria: 1 mark for a correct definition of sample bias, 1 mark for a correct definition of label bias, that clearly distinguishes it from sample bias (labels being wrong, not the sample being unrepresentative).
foundation3 marksA car insurer trains a claims-risk model using five years of policy data from drivers in Sydney only, then deploys it nationally. Identify the type of bias most likely to occur and explain its likely consequence.Show worked solution →
This is sample bias: the training data (Sydney drivers only) does not represent the national population the model is deployed on (different road types, traffic density, weather, and driving conditions in regional and rural areas).
Consequence: the model's risk predictions will be systematically less accurate for non-Sydney drivers, e.g. it may overestimate risk for careful regional drivers whose conditions differ from Sydney traffic patterns, or underestimate risks specific to regional/rural driving (long distances, wildlife strikes, unsealed roads) that never appeared in the training data.
Marking criteria: 1 mark for correctly naming sample bias, 1 mark for identifying that the training population does not match the deployment population, 1 mark for a specific, plausible consequence (systematically wrong predictions for non-Sydney drivers).
core4 marksThe table shows a model's accuracy broken down by two demographic groups after deployment.
| Group | Number of cases | Accuracy |
|---|---|---|
| A | 900 | 98% |
| B | 100 | 62% |
(a) Calculate the overall accuracy across both groups combined. (b) Explain why reporting only the overall accuracy would hide a serious problem.Show worked solution →
(a) Overall accuracy.
(b) Why this hides a problem. The overall figure of 94.4 percent looks strong and would pass most casual quality checks, but it is dominated by Group A, which makes up 90 percent of the cases. Group B's accuracy of only 62 percent, far below an acceptable standard, is effectively averaged away. Anyone in Group B experiences a model that is wrong more than one time in three, a serious fairness and reliability issue that the single overall number completely conceals.
Marking criteria: 1 mark for correct working, 1 mark for the correct overall accuracy (94.4%), 1 mark for explaining that the larger group dominates the average, 1 mark for explaining the consequence for Group B specifically.
core5 marksA decision tree classifier is trained with no depth limit on a dataset of 1,000 labelled examples, achieving 99.5 percent accuracy on the training set but 71 percent accuracy on a held-out test set of 250 examples. (a) Identify the failure mode shown. (b) Recommend two specific changes to the model or training process to address it, explaining how each helps.Show worked solution →
(a) Failure mode. This is overfitting. The very large gap between training accuracy (99.5%) and test accuracy (71%) shows the tree has memorised the specific examples (including noise) in the training set rather than learning patterns that generalise to new data.
(b) Two mitigations.
Limit the tree's max_depth (e.g. to 5 or so). A shallower tree cannot create a separate branch for every individual training example, forcing it to capture only the broader, generalisable patterns rather than memorising noise.
Get more training data, or apply regularisation/early stopping. More diverse examples make it harder for the model to memorise the whole dataset, and stopping training before it fully fits every quirk (or penalising complexity via regularisation) directly targets the mechanism causing the overfit.
Marking criteria: 1 mark for correctly identifying overfitting, 1 mark for correctly justifying it using the train/test gap, 1 mark each for two valid, specific mitigations (2 marks), 1 mark for explaining the mechanism by which at least one mitigation reduces overfitting.
exam6 marksA public hospital network builds a triage-priority model trained on ten years of historical patient records, using length of past hospital stay as a proxy label for 'severity of illness'. Explain how this design choice could produce a biased model, and evaluate whether removing the patient's postcode from the features would fix the bias.Show worked solution →
- How the design choice introduces bias
- Length of past hospital stay is not the same thing as true severity of illness; it also reflects factors such as access to care, insurance status, and whether a patient could afford or arrange follow-up outside hospital. If historically disadvantaged groups had shorter stays not because they were less severely ill but because they were discharged earlier due to under-resourcing, cost pressure, or systemic under-treatment, the model will learn to associate that group with lower severity. This is a form of historical/label bias: the label (length of stay) is a flawed proxy that encodes past inequity rather than genuine medical need, similar to the well-documented case of a healthcare risk-scoring algorithm that used past spending as a severity proxy and underestimated illness in Black patients because they had historically received less care.
- Would removing postcode fix it
- Removing postcode alone would not reliably fix the bias. Postcode is one possible proxy feature for the disadvantage driving the bias, but the underlying problem is the flawed label (length of stay), not a single input feature. Even without postcode, other correlated features, such as insurance status, referring hospital, or even certain diagnosis codes, could still let the model reconstruct the same pattern indirectly. Removing one proxy feature can also make the bias harder to detect, since an obvious explanatory variable has been hidden without removing its underlying effect.
- Better approach
- Replace or supplement the flawed label with a more clinically direct severity measure (e.g. validated clinical severity scores recorded at admission), and evaluate the model's error rates separately across demographic and socioeconomic groups before and after deployment, rather than relying on removing a single feature.
Marking criteria: 1 mark for identifying length of stay as a flawed proxy label, 1 mark for explaining the mechanism by which this encodes historical inequity (naming or clearly describing historical/label bias), 1 mark for a concrete real-world parallel or plausible consequence, 1 mark for correctly concluding that removing postcode alone is insufficient, 1 mark for explaining why (proxy features/correlated variables can still leak the same signal), 1 mark for proposing a genuinely better mitigation (better label, per-group evaluation).
exam7 marksAssess the claim that collecting more training data is always the best way to improve a machine learning model's fairness and accuracy.Show worked solution →
This is an ASSESS question: markers reward a supported judgement, not a one-sided description.
- Plan
- Thesis: collecting more data is not always the best fix, and can actively worsen fairness, because the VALUE of additional data depends entirely on whether it improves the data's representativeness and label quality, not merely its volume.
- Case for more data helping
- More data generally helps a model learn more robust, generalisable patterns and reduces overfitting, because there are more examples to average noise away from and the model is less likely to memorise idiosyncrasies of a small dataset. For underfitting caused by too little signal, more diverse examples can also expose patterns a smaller dataset never showed.
- Case against "more data" as a universal fix
- If the additional data is collected using the same flawed process as the original set, e.g. still drawn overwhelmingly from one demographic group, or still using the same biased labelling process, then the imbalance in the dataset is simply reproduced at a larger scale. This does not fix sample bias or label bias; it can entrench them further, because the model now has even stronger statistical evidence supporting the same skewed pattern. Amazon's scrapped hiring tool is a case in point: adding more historical CVs, almost all from men, would not have fixed its bias against female-associated language, it would have reinforced it. Similarly, collecting more data does nothing to fix overfitting caused by an excessively complex model relative to a genuinely small underlying signal, nor does it fix underfitting caused by choosing too simple a model or missing features.
- Model paragraph (excerpt)
- The claim treats "more data" as a single lever, but data quantity and data quality are separate variables that a good engineer must evaluate independently. A dataset that doubles in size while keeping the same demographic skew has not become more representative, it has simply given the model twice as much evidence for the same unbalanced pattern, and per-group evaluation would likely show the accuracy gap between groups persist or widen rather than close. The genuinely effective fix is targeted: identifying which group or label process is under-represented or flawed, and correcting that specific gap, whether through stratified collection, reweighting, or fixing the labelling method, rather than an undirected increase in volume.
Marker's note: top-band answers (1) take an explicit position rather than agreeing uncritically with the claim, (2) distinguish overfitting/underfitting (where more data often does help) from bias (where it often does not), (3) use a specific named or described real-world example, and (4) close with a reasoned judgement about when more data helps versus when targeted, representative collection is required instead.