Ethics in automation explained: HSC Software Engineering Module 3

Machine learning systems making decisions about people raise three big ethical issues.

Accountability: when an automated decision is wrong, who is responsible? The developer who wrote the model, the company that deployed it, or the user who relied on it? Without clear accountability, harmed individuals cannot seek redress.

Transparency: many ML models are opaque. The model can predict a refusal without anyone being able to explain why. Where decisions affect people's lives (a loan refusal, a parole denial, a job rejection), the inability to explain the decision violates the right to a reasoned outcome.

Bias and harm: a model trained on biased data perpetuates bias at scale. One biased decision affects one person; an automated biased decision affects everyone.

Case study: Robodebt (Australia, 2016-2020). Centrelink deployed an automated system to identify welfare debts by comparing reported income to ATO data, averaging annual income across pay periods. The system issued ~470,000 incorrect debt notices for over $1.7 billion. People received debts they did not owe, with little recourse and no human verification. The system was found unlawful, a Royal Commission found "venality, incompetence and cowardice", and the government settled for $1.8 billion.

Lessons: human-in-the-loop review for consequential decisions, the burden of proof should not shift to the affected individual, transparency about how the system works is essential, and external audit before deployment to high-stakes domains.

Markers reward at least two distinct ethical concerns, a named real-world case (Robodebt, COMPAS, Amazon hiring, Apple Card credit limits, Clearview AI), and at least one mitigation principle.

Accountability is being able to identify who is responsible when an automated decision causes harm or is wrong, so the affected person has someone to hold answerable and a path to redress.

Transparency is being able to explain why a system produced a particular decision, in terms a person can understand.

A system can satisfy one without the other: a company might clearly accept legal accountability (e.g. "we are responsible for our model's decisions") while still using a black-box model that no one, including the company, can actually explain, so an affected individual knows WHO to complain to but not WHY the decision was made. Conversely, a fully explainable rule-based system with no clear ownership could tell a person exactly why they were rejected, but leave them with no one accountable to appeal to.

Marking criteria: 1 mark for a correct definition of accountability, 1 mark for a correct definition of transparency, 1 mark for a valid explanation of how the two can come apart (a specific example scenario, not just restating the definitions).

The case is Robodebt (Australia, 2016-2020).

One valid safeguard (any one, fully explained):

• Human-in-the-loop verification: requiring a human caseworker to check and approve each debt notice before it was sent, rather than fully automating the decision, would have caught cases where income averaging produced an incorrect result for someone with irregular pay.
• No reverse onus of proof: not shifting the burden onto the individual to disprove an automatically generated debt would have prevented people being pursued for debts the agency itself could not properly substantiate.
• External audit before deployment: independently testing the income-averaging method against real pay patterns before rollout would likely have revealed it produced false positives for people with lumpy income.

Marking criteria: 1 mark for correctly naming Robodebt, 1 mark for a valid, specific safeguard, 1 mark for clearly explaining how that safeguard would have prevented the specific harm described (false debts from income averaging).

(a) The disparity

Group Y has a much higher false positive rate (0.21) than Group X (0.06), a gap of 0.15. This means creditworthy applicants in Group Y are denied loans they should have received far more often than creditworthy applicants in Group X, while the false negative rates are similar across both groups.

(b) Why this matters despite high overall accuracy

Overall accuracy averages across the whole population and can hide large disparities between subgroups; a model can be 90 percent accurate overall while systematically disadvantaging one group. This is unfair (a form of bias) because an individual's outcome depends partly on their group membership rather than solely on their actual creditworthiness, which may also breach anti-discrimination obligations even without any intent to discriminate.

(c) Recommended action

The company should halt or pause deployment for Group Y, investigate the source of the disparity (e.g. biased historical training data or a proxy feature correlated with group membership), and remediate the model (retraining on more representative or reweighted data, or applying a fairness constraint) before redeploying, then re-audit before resuming.

Marking criteria: 1 mark for correctly identifying the false positive rate gap, 1 mark for correctly noting the false negative rates are similar, 2 marks for a full explanation of why subgroup disparity is an ethical concern despite good overall accuracy, 1 mark for a specific, actionable remediation step (not just "fix the bias").

Common concern: lack of a human check before harm occurred. In Robodebt, automated debt notices were issued to citizens without a caseworker verifying the income-averaging calculation first. In the Amazon hiring tool, resumes were scored and effectively filtered without a recruiter checking whether the model's downranking of "women's" terms was legitimate. In both cases, an automated output was allowed to directly affect people (a debt demand; a rejected application) without a mandatory human review step that could have caught the specific flaw before it caused harm.

A key difference. Robodebt's harm came from a flawed AVERAGING METHOD applied uniformly (a design/logic flaw that produced false debts for anyone with irregular income, not tied to protected characteristics), whereas the Amazon tool's harm came from LEARNED BIAS in historical training data (a decade of mostly-male hires), which specifically and systematically disadvantaged women, a protected-characteristic-linked harm. Robodebt was ultimately found unlawful and cost the government $1.8 billion in settlement; Amazon internally identified the bias during development and cancelled the project before full deployment, so it did not cause the same scale of public harm.

Marking criteria: 1 mark for correctly identifying a valid common concern, 2 marks for explaining it clearly with reference to both cases, 1 mark for identifying a valid difference, 1 mark for explaining that difference with reference to both cases.

This is a DISCUSS question worth 7 marks: markers want breadth (multiple concerns), depth (specific reasoning), and a case study used as evidence, not decoration.

Concern 1: Bias from historical training data

Ten years of past triage records may reflect historical under-triage of certain groups (e.g. patients from communities with language barriers, or historically under-diagnosed conditions in women). A model trained on this data would learn and automate that same pattern at scale. This mirrors the Amazon hiring tool case, where a model trained on a decade of biased historical outcomes reproduced and scaled that bias rather than correcting it, even though no one intended it to discriminate.

Concern 2: Accountability when a triage score is wrong

If a patient is under-triaged and deteriorates while waiting, who is responsible: the developers, the hospital, or the triage nurse who deferred to the score? Clear accountability matters enormously in a life-or-death context, where the consequences of an unresolved "who is responsible" question are far more severe than in a typical commercial application.

Concern 3: Transparency and human-in-the-loop review

Nursing staff need to understand why a score was assigned to exercise appropriate clinical judgement rather than blindly following an opaque number. As with Robodebt, where automated debt decisions were issued with no human verification and no explanation to affected people, an unexplained and unreviewed triage score risks harming patients whose case does not fit the pattern the model learned, especially rare or atypical presentations.

Judgement

These concerns do not mean the system should not be built, but that it must be deployed as a decision-support tool, advisory only, with a human clinician as the accountable decision-maker, subgroup fairness audits performed before and after deployment, and clear escalation and override procedures, directly reflecting the lessons of Robodebt (human-in-the-loop, no automation of consequential decisions without verification) and the Amazon case (audit training data for historical bias before trusting model outputs).

Marking criteria: 1 mark per correctly identified and explained ethical concern relevant to this scenario (up to 3 concerns, 3 marks), 1 mark for accurately applying the Amazon case, 1 mark for accurately applying the Robodebt case, 1 mark for a specific mitigation tied to the scenario (human-in-the-loop, subgroup audit), 1 mark for an overall reasoned judgement rather than a list.

Why this is primarily a privacy and consent issue. Clearview AI scraped approximately 3 billion images from social media and other public websites without asking the people in those images for permission, and without any of them knowing their photo had been collected. The core wrong is not that the facial recognition matching was inaccurate or unfairly discriminatory between groups (a bias/fairness issue); it is that personal biometric data was collected and commercialised (sold to law enforcement) without lawful basis or consent, which is a privacy issue regardless of how accurate the matching later turned out to be.

Regulator action. Australia's Information Commissioner ruled in 2021 that Clearview breached the Privacy Act and ordered it to stop collecting data on Australians and to destroy existing data collected unlawfully. Equivalent regulators in the UK, Italy, France and Canada issued comparable findings and orders, showing this was treated as a data protection breach, not a bias/accuracy complaint.

Marking criteria: 1 mark for correctly identifying the absence of consent/lawful basis as the core issue, 1 mark for explicitly distinguishing this from a bias/fairness concern, 1 mark for a specific, accurate regulator action (Australian Information Commissioner ruling/order), 1 mark for noting the scale or multi-jurisdictional nature of the finding.