Explain how the quality and representativeness of training data affect a model, including the risks of bias and overfitting

What this dot point is asking

NESA wants you to explain how the data used to train a model shapes its behaviour, and identify two failure modes: bias (systematic errors against specific groups) and overfitting (memorising the training data instead of learning generalisable patterns).

The answer

Garbage in, garbage out

A model can only learn what is in the data. If the training data has flaws, the model will replicate or amplify them. Common data flaws:

• Sample bias: the training data does not represent the population. A model trained on Sydney pedestrians may misbehave in Tokyo.
• Label bias: the labels themselves are wrong or reflect human prejudice. A "good employee" label assigned by biased managers reproduces their bias.
• Measurement bias: the features measure different things in different contexts (a thermometer reading differently in two cities).
• Historical bias: the data reflects past inequities. A loan approval model trained on past approvals reflects past discrimination.

Real examples

• Amazon's hiring tool: trained on a decade of CVs, mostly from male engineers. The model learned to downrank CVs that mentioned "women's chess club" and similar phrases. Amazon scrapped the project.
• Facial recognition disparities: commercial systems audited in 2018 had error rates under 1 percent for white men and over 30 percent for dark-skinned women, traced to the training set composition.
• Healthcare risk scores: a US algorithm using healthcare spending as a proxy for need underestimated illness in Black patients because they had less access to care, so lower past spending.

Sample bias in detail

If 90 percent of training data comes from group A and 10 percent from group B, the model's loss function rewards being right about group A more than group B. The model can achieve high overall accuracy by performing well on A and poorly on B.

Fix: oversample under-represented groups, undersample over-represented groups, or use class weights so each group contributes equally to the loss.

Overfitting and underfitting

Overfitting: the model memorises the training data. High training accuracy, low test accuracy. The model has learned noise, not patterns. Solutions: more data, smaller model, regularisation (dropout, L2 weight decay), early stopping.

Underfitting: the model is too simple to capture the patterns. Low training accuracy, low test accuracy. Solutions: more powerful model, more features, less aggressive regularisation.

from sklearn.tree import DecisionTreeClassifier

# Overfit: very deep tree memorises training data.
overfit = DecisionTreeClassifier(max_depth=None)
overfit.fit(X_train, y_train)
print("train:", overfit.score(X_train, y_train))  # very high
print("test:", overfit.score(X_test, y_test))     # much lower

# Better: limit depth to force generalisation.
fit_ok = DecisionTreeClassifier(max_depth=5)
fit_ok.fit(X_train, y_train)
print("train:", fit_ok.score(X_train, y_train))   # high
print("test:", fit_ok.score(X_test, y_test))      # close to train

Per-group evaluation

A 95 percent overall accuracy can hide an 80 percent accuracy on one demographic. Always compute metrics per group:

import pandas as pd

results = pd.DataFrame({
    "group": ["A"] * 90 + ["B"] * 10,
    "correct": [True] * 88 + [False] * 2 + [True] * 7 + [False] * 3,
})

print(results.groupby("group")["correct"].mean())
# group
# A    0.977
# B    0.700

Group B's 70 percent accuracy is hidden by the 97 percent on the majority group.

Train, validation, test discipline

A clean split prevents data contamination:

• Training set (60-80 percent): used to fit the model.
• Validation set (10-20 percent): used during development to tune hyperparameters.
• Test set (10-20 percent): used once, at the end, to estimate real-world performance.

Looking at the test set during development leaks information and inflates reported accuracy.

Data documentation

Every dataset should be accompanied by:

• A datasheet explaining where the data came from, who labelled it and how.
• The demographic composition of the data.
• Known limitations (sample bias, label noise, the time period covered).
• A statement of intended use and out-of-scope use cases.

This is the "Datasheets for Datasets" standard (Gebru et al., 2018), now an industry norm.