Identify the ethical implications of automation and artificial intelligence, including accountability, transparency, employment effects and the use of personal data

What this dot point is asking

NESA wants you to identify the major ethical concerns raised by automated decision-making, and to be able to discuss them with reference to real cases. You should know at least two cases in detail.

The answer

The big ethical concerns

Accountability and redress.

When an automated decision is wrong, who is responsible? If a self-driving car causes a crash, is it the manufacturer, the software supplier, the safety driver, or the company that operates the fleet? Affected individuals need a clear avenue to complain and to be made whole.

Transparency and explainability.

Many ML systems are black boxes. The model can predict refuse-this-loan without anyone being able to explain why. The European Union's GDPR includes a right to an explanation for automated decisions; Australian law is moving in the same direction.

Bias and fairness.

A model trained on biased data perpetuates bias at scale. See training-data-and-bias. Cases below.

Privacy and use of personal data.

ML systems are trained on personal data. Customers, patients and citizens may not have consented to that use, or even know it is happening. The General Data Protection Regulation (GDPR) and Australia's Privacy Act set baselines: lawful purpose, minimisation, retention limits, deletion rights.

Surveillance.

Facial recognition, gait analysis and behaviour prediction enable monitoring at scale. The Australian Human Rights Commission and the Office of the Australian Information Commissioner have both warned about facial recognition deployment.

Employment effects.

Automation displaces some jobs and creates others. Truck drivers, radiologists, call centre staff and translators all face changing labour markets. Industries and governments have responsibility to manage the transition (retraining, social safety nets).

Concentration of power.

A handful of companies train the largest models. Concentrated technical capability becomes concentrated economic and political power.

Case studies

Robodebt (Australia, 2016-2020).

The Australian government deployed an automated system to identify welfare debt by comparing self-reported income to ATO data, averaging annual income across pay periods. The averaging method produced false debts where income was lumpy. ~470,000 false debts totalling over $1.7 billion were issued. The Federal Court found the scheme unlawful in 2019. A Royal Commission in 2023 found "venality, incompetence and cowardice". The government paid$1.8 billion in settlement. Lessons: human-in-the-loop for consequential decisions, no reverse onus of proof, external audit before deployment, deliberate decision-making about averaging assumptions.

COMPAS recidivism scoring (US, 2016).

Northpointe's COMPAS algorithm gave US courts a risk score for criminal defendants. ProPublica's 2016 analysis found Black defendants were nearly twice as likely as white defendants to be incorrectly labelled high risk, while white defendants were more likely to be incorrectly labelled low risk. The case sparked the entire algorithmic fairness research field.

Amazon hiring tool (US, 2018).

Amazon trained a model on a decade of CVs from successful hires, mostly male. The model learned to penalise CVs that mentioned "women's chess club captain" and to downrank graduates of women-only colleges. Amazon scrapped the project. Lessons: training on biased historical data reproduces bias, even removing the protected attribute does not help because proxies leak it.

Apple Card credit limits (US, 2019).

Goldman Sachs gave women lower credit limits than their husbands on the Apple Card despite shared finances and equivalent histories. Goldman could not explain why. The case prompted a regulator investigation and ultimately a finding that the bank had not violated fair lending laws, but the inability to explain the decisions revealed how opaque such systems can be.

Clearview AI facial recognition (global).

Clearview scraped 3 billion images from social media without consent and sold facial recognition to law enforcement. Australia's Information Commissioner ruled in 2021 that Clearview had breached the Privacy Act and ordered it to stop collecting data on Australians. Multiple regulators in the UK, Italy, France and Canada have made similar findings.

Generative AI and content (current).

LLMs train on web-scale text that includes copyrighted works without consent. Image generators do the same with art. Lawsuits and regulatory action are ongoing. Workers in writing, illustration, voice acting and translation face direct labour-market effects.

Principles for responsible deployment

The Australian Government's AI Ethics Principles (2019) and the OECD AI Principles (2019) converge on roughly the same list:

1. Human, societal and environmental wellbeing is the primary goal.
2. Human-centred values: respect human rights, diversity and individual autonomy.
3. Fairness: avoid unfair discrimination.
4. Privacy protection and security.
5. Reliability and safety.
6. Transparency and explainability.
7. Contestability: people can challenge decisions.
8. Accountability: responsibility is identifiable.

A worked code example: a fairness audit

from sklearn.metrics import confusion_matrix
import pandas as pd

audit = pd.DataFrame({
    "group": ["F"] * 100 + ["M"] * 100,
    "predicted": predictions,  # from the model
    "actual": actuals,         # known ground truth
})

for group, sub in audit.groupby("group"):
    tn, fp, fn, tp = confusion_matrix(sub["actual"], sub["predicted"]).ravel()
    fpr = fp / (fp + tn)
    fnr = fn / (fn + tp)
    print(f"{group}: false positive rate={fpr:.2f}, false negative rate={fnr:.2f}")

A 5 percentage-point gap in false positive rate between groups is grounds to halt deployment, investigate the source, and remediate.