Neural network basics explained: HSC Software Engineering Module 3

Syllabus dot point

“Describe the basic structure of a neural network, including neurons, layers, weights, activation functions and training by backpropagation”

10 min readUpdated 3 Jul 2026

Quick answer

A feed-forward neural network is layers of artificial neurons. Each neuron computes a weighted sum of its inputs plus a bias and applies an activation function (typically ReLU). Training does a forward pass to compute the prediction, computes a loss, then runs backpropagation to update every weight using gradients of the loss found via the chain rule. Repeated over many epochs, the network learns to map inputs to outputs, while too many epochs on too small a network risks overfitting.

Jump to a section

What this dot point is asking
The answer
Exam-style questions
Practice questions

In plain English

Picture a factory assembly line with several workstations in a row. Each worker (a neuron) looks at what is passed along from the workers before them, decides how much attention to pay to each item (the weights), adds their own personal adjustment (the bias), and only passes something forward if it clears their own threshold (the activation function). A "deep" network is just a longer line with more workstations, so later workers can build increasingly complicated judgements out of simpler ones earlier in the line. Training the line by backpropagation is like a supervisor walking backward down the line after a bad batch of products comes out the end, telling each worker, one step at a time, exactly how much they personally contributed to the mistake and by how much to adjust their own habits for next time.

What this dot point is asking

NESA wants you to describe the basic architecture of a feed-forward neural network and the mechanics of how it is trained. You do not need to derive gradients, but you should know the components by name and what each does.

The answer

A feed-forward neural network is a stack of layers. The diagram shows a small network with an input layer of four features, one hidden layer of five neurons, and an output layer of three classes. Every neuron in one layer is connected to every neuron in the next.

The artificial neuron

The basic unit. It takes inputs $x_1, x_2, \dots, x_n$ from the previous layer, multiplies each by a weight, sums them, adds a bias, and applies an activation function:

a = f\left(\sum_{i=1}^{n} w_i x_i + b\right)

The neuron equation and activation functions

a = f\left(\sum_{i=1}^{n} w_i x_i + b\right)

$w_i$ : learned weight on input $x_i$ ; $b$ : learned bias; $f$ : activation function.
ReLU: $f(z) = \max(0, z)$ . Default for hidden layers.
Sigmoid: $f(z) = \dfrac{1}{1+e^{-z}}$ . Squashes to $(0,1)$ ; used for binary output.
Softmax: turns a score vector into probabilities summing to 1; used for multi-class output.
Tanh: $f(z) = \tanh(z)$ . Squashes to $(-1,1)$ ; older default for hidden layers.

Common activation functions:

ReLU: $f(z) = \max(0, z)$ . The default in hidden layers.
Sigmoid: $f(z) = \frac{1}{1 + e^{-z}}$ . Squashes the output to $(0, 1)$ . Used for binary classification output.
Softmax: turns a vector of scores into probabilities summing to 1. Used for multi-class classification output.
Tanh: $f(z) = \tanh(z)$ . Squashes to $(-1, 1)$ . Older default.

Layers

A neural network is a stack of layers:

Input layer: one neuron per feature. For an image, that might be 28 x 28 = 784 input neurons.
Hidden layers: one or more layers between input and output. Each neuron is connected to every neuron in the previous layer (in a fully connected network).
Output layer: one neuron for regression, $n$ neurons for $n$ -class classification.

A "deep" network has many hidden layers. Each layer learns increasingly abstract features.

Answering "describe a neuron" questions

Work through the equation in order every time: multiply each input by its weight, sum them, add the bias, then apply the activation function. Naming "weights, bias, activation function" in that order, with the correct symbol for each ( $w_i$ , $b$ , $f$ ), earns marks even before you name a specific activation function. Only then say WHICH activation is used and WHY (ReLU in hidden layers because it is cheap to compute and avoids saturating for positive inputs; softmax in a multi-class output because it produces valid probabilities that sum to 1).

Forward pass

To make a prediction, feed the input through every layer in turn. Each layer computes its weighted sums and activations. The output layer produces the prediction.

For a 784-input, 128-hidden, 10-output digit classifier:

Input: 784 pixel values, normalised to $[0, 1]$ .
Hidden layer: 128 neurons, each computing a weighted sum of the 784 inputs and applying ReLU.
Output layer: 10 neurons (one per digit 0-9), each computing a weighted sum of the 128 hidden activations and applying softmax.

The output is a probability distribution over the 10 digits. The predicted digit is the one with the highest probability.

Loss

Measures how wrong the prediction is.

Cross-entropy loss for classification: low when the predicted probability of the correct class is high.
Mean squared error for regression: low when the predicted value is close to the true value.

Backpropagation

The training algorithm. For each batch of training examples:

Forward pass: compute predictions and the loss.
Backward pass: compute the gradient of the loss with respect to every weight in the network, using the chain rule of calculus.
Update: adjust every weight by a small step opposite to its gradient. The step size is the learning rate.

After many passes through the training data (epochs), the weights settle into values that produce good predictions.

The optimiser controls how the updates are applied. Stochastic gradient descent (SGD) updates after each mini-batch. Adam is a popular adaptive variant.

A worked code example

A minimal feed-forward network in PyTorch:

import torch
import torch.nn as nn
import torch.optim as optim

class MLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        return self.fc2(x)  # softmax applied via the loss

model = MLP()
loss_fn = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

# One training step:
predictions = model(batch_x)              # forward pass
loss = loss_fn(predictions, batch_y)      # compute loss
loss.backward()                           # backpropagation
optimizer.step()                          # update weights
optimizer.zero_grad()                     # reset gradients

The framework handles the gradient calculations automatically.

Hyperparameters

Choices the developer makes that affect training:

Number of layers and neurons per layer (network architecture).
Activation functions (ReLU, sigmoid, tanh).
Learning rate (how big each weight update is).
Batch size (examples per gradient step).
Number of epochs (passes through the training data).
Regularisation (dropout, L2 weight decay) to prevent overfitting.

Overfitting

A neural network with enough parameters can memorise the training data exactly. Such a model has zero training error but performs poorly on new data. Detection: training loss keeps falling while validation loss starts rising. Prevention: more training data, smaller network, dropout, regularisation, early stopping.

Beyond feed-forward

For images and other structured inputs, specialised architectures perform far better:

Convolutional neural networks (CNNs) for images.
Recurrent neural networks (RNNs) for sequences.
Transformers for language and many other domains.

These are out of HSC scope at the architectural level, but you should recognise the names.

Worked example

A neural network is trained to classify HSC essays by predicted band (1-6). It has 50 input features (word count, sentence length, vocabulary richness, etc.), one hidden layer of 32 neurons, and 6 output neurons.

After training, training accuracy is 95 percent but validation accuracy is 60 percent. Diagnose and propose two fixes.

Diagnosis: the network has overfit the training data. It memorised idiosyncrasies of the training essays rather than learning patterns that generalise.

Fixes (any two):

More training data: collect more graded essays.
Smaller network: reduce hidden neurons (32 -> 8).
Regularisation: add dropout (randomly disable neurons during training) or L2 weight decay.
Early stopping: monitor validation loss and stop training once it starts rising.
Data augmentation: simulate more examples by paraphrasing or perturbing existing ones.

Marker's note: full marks need the diagnosis named as overfitting AND justified using the training-versus-validation accuracy gap, plus two DIFFERENT fixes, each described specifically enough to show how it reduces overfitting (not just naming a buzzword like "regularisation" with no explanation).

Common traps

Calling backpropagation "the network": Backpropagation is the training algorithm. The trained network (weights and architecture) is the model.
Forgetting the bias term: Each neuron has a learnable bias, not just weights. Without it, the network is constrained to predictions that pass through the origin.
Treating layers as identical: Different layers can have different activations. Convolutional layers, pooling layers, normalisation layers are different from fully connected layers.
Saying neural networks "think like a brain": The biological metaphor is loose. Artificial neurons are simple weighted sums; biological neurons are far more complex. Use the analogy carefully.
Confusing training and inference: Training adjusts weights; inference applies the trained weights to new data. Inference is much cheaper than training.

Exam-style questions

Questions in the style of NESA exam questions on this dot point, each with a worked answer. They are written by ExamExplained unless tagged "Past paper"; the year shows the paper a question is modelled on.

Practice6 marks

Describe the structure of a simple feed-forward neural network with one hidden layer and explain how it learns from training data.

Show worked answer →

A feed-forward neural network has three kinds of layer. The input layer has one neuron per feature. The hidden layer(s) contain artificial neurons that combine inputs from the previous layer. The output layer produces the prediction - one neuron for regression, one neuron per class for classification.

Each neuron computes a weighted sum of its inputs plus a bias, then passes the result through an activation function (typically ReLU in hidden layers, softmax or sigmoid in the output). Mathematically: $a = f(\sum_i w_i x_i + b)$ where $w_i$ are learned weights, $x_i$ are the inputs from the previous layer, $b$ is the bias and $f$ is the activation function.

Training has two phases per batch of examples.

Forward pass: feed the input through every layer in turn to produce a prediction. Compare the prediction to the true label to compute a loss (cross-entropy for classification, mean squared error for regression).

Backward pass (backpropagation): compute the gradient of the loss with respect to every weight, using the chain rule of calculus. Update each weight by a small step in the direction that reduces the loss. The optimiser (typically stochastic gradient descent or Adam) controls how large the step is.

Repeat for many epochs (passes through the training data) until the loss stops decreasing on a held-out validation set. The network then generalises to new examples.

Markers reward the three-layer structure, weights/bias/activation in the neuron description, both forward and backward pass, and recognising that backpropagation uses the chain rule to compute gradients.

Practice questions

Original practice questions graded from foundation to exam level, each with a full worked solution. Try them before revealing the solution.

foundation3 marks

A neuron has inputs

x_1 = 4

and

x_2 = 3

, weights

w_1 = 0.5

and

w_2 = -0.2

, and bias

b = 1

. Calculate the neuron's pre-activation value and its output after a ReLU activation.

Show worked solution →

Step 1: pre-activation (weighted sum plus bias).

z = w_1 x_1 + w_2 x_2 + b = (0.5)(4) + (-0.2)(3) + 1 = 2.0 - 0.6 + 1 = 2.4

Step 2: apply ReLU.

a = f(z) = \max(0, 2.4) = 2.4

Since $z$ is already positive, ReLU leaves it unchanged.

Marking criteria: 1 mark for the correct weighted sum, 1 mark for correctly adding the bias, 1 mark for correctly applying ReLU (2.4).

foundation3 marks

Explain why a neuron needs a learnable bias term as well as its weights.

Show worked solution →

Without a bias, a neuron's pre-activation is $z = \sum_i w_i x_i$ , which is forced to equal 0 whenever every input is 0, regardless of the weights chosen. A bias adds a constant, $z = \sum_i w_i x_i + b$ , letting the neuron shift its activation threshold up or down independently of the inputs. This gives the network far more flexibility to fit a relationship that does not naturally pass through the origin, in the same way a straight line $y = mx + c$ needs the $c$ term to fit data that does not pass through $(0,0)$ .

Marking criteria: 1 mark for stating that without bias, zero input forces zero pre-activation, 1 mark for explaining the bias shifts the activation threshold, 1 mark for a valid analogy or consequence (e.g. the network could not fit data offset from the origin).

core4 marks

The table below shows training and validation loss for a network over 20 epochs. | Epoch | Training loss | Validation loss | |-------|----------------|-------------------| | 1 | 0.90 | 0.95 | | 5 | 0.40 | 0.42 | | 10 | 0.15 | 0.30 | | 15 | 0.05 | 0.45 | | 20 | 0.02 | 0.60 | (a) Identify the approximate epoch at which the network begins to overfit, and justify your answer using the table. (b) Propose one specific fix.

Show worked solution →

(a) Epoch and justification. Overfitting begins at around epoch 10. Up to epoch 10, both training loss and validation loss fall together (0.90 to 0.15 and 0.95 to 0.30). After epoch 10, training loss keeps falling (to 0.02 by epoch 20) while validation loss rises again (from 0.30 back up to 0.60). This divergence, where training loss keeps improving but validation loss gets worse, is the defining signature of overfitting.

(b) Fix. Apply early stopping: monitor validation loss during training and stop (or restore the weights from) around epoch 10, before validation loss starts rising, instead of training all the way to epoch 20. (Dropout, L2 regularisation, or a smaller network would also be accepted.)

Marking criteria: 1 mark for identifying epoch 10 (or a value in that range) as the turning point, 2 marks for justifying it using the divergence between training and validation loss (not just quoting one row), 1 mark for a specific, workable fix.

core5 marks

A network has two inputs

x_1 = 1.0

and

x_2 = 2.0

, one hidden neuron with weights

w_{h1} = 0.3

w_{h2} = -0.1

, bias

b_h = 0.05

and ReLU activation, feeding a single output neuron with weight

w_o = 2.0

, bias

b_o = -0.1

and sigmoid activation. Calculate the network's output.

Show worked solution →

Step 1: hidden neuron pre-activation.

z_h = (0.3)(1.0) + (-0.1)(2.0) + 0.05 = 0.30 - 0.20 + 0.05 = 0.15

Step 2: hidden neuron activation (ReLU).

a_h = \max(0, 0.15) = 0.15

Step 3: output neuron pre-activation.

z_o = w_o \, a_h + b_o = (2.0)(0.15) + (-0.1) = 0.30 - 0.10 = 0.20

Step 4: output neuron activation (sigmoid).

a_o = \frac{1}{1 + e^{-0.20}} = \frac{1}{1 + 0.819} \approx 0.55

The network's output is approximately 0.55.

Marking criteria: 1 mark for the correct hidden pre-activation, 1 mark for correctly applying ReLU, 1 mark for the correct output pre-activation, 2 marks for correctly evaluating the sigmoid to approximately 0.55.

exam6 marks

During backpropagation, a particular weight currently has value

w = 0.80

. The gradient of the loss with respect to this weight is calculated (via the chain rule) as

0.25

, and the learning rate is

0.1

. (a) Calculate the updated weight. (b) Explain why the gradient is subtracted, not added. (c) Explain what would likely go wrong if the learning rate were set to

5.0

instead.

Show worked solution →

(a) Updated weight.

w_{new} = w_{old} - (\text{learning rate}) \times (\text{gradient}) = 0.80 - (0.1)(0.25) = 0.80 - 0.025 = 0.775

(b) Why subtract. The gradient points in the direction of steepest INCREASE of the loss. Since training wants to MINIMISE the loss, the weight is moved in the opposite direction to the gradient (subtracted), so that a positive gradient (increasing $w$ would increase the loss) correctly decreases $w$ , and a negative gradient would correctly increase $w$ .

(c) Learning rate too large. With a learning rate of 5.0, the update would be $(5.0)(0.25) = 1.25$ , giving $w_{new} = 0.80 - 1.25 = -0.45$ , a huge swing in a single step. Such large steps tend to overshoot the minimum of the loss repeatedly, causing the loss to oscillate or diverge (increase) instead of steadily decreasing, so the network fails to train.

Marking criteria: 1 mark for the correct calculation (0.775), 2 marks for correctly explaining the gradient points toward increasing loss so it is subtracted, 3 marks for explaining that too large a learning rate causes overshooting and unstable or diverging training, with a supporting calculation.

exam7 marks

Evaluate the statement "a neural network with more hidden layers always produces a better model." Refer to the forward pass, overfitting, and computational cost in your answer.

Show worked solution →

This is an EVALUATE question: markers reward a clear judgement supported by specific mechanisms, not a list of vaguely related facts.

Thesis: The statement is false as an unconditional claim. Adding hidden layers increases a network's capacity to represent complex functions, but capacity is not the only thing that determines whether a model is actually "better" on new data.
Forward pass and representational power: Each additional hidden layer lets the network compose simpler features into more abstract ones during the forward pass, in principle allowing it to model more complex relationships between inputs and outputs than a shallow network could. This is the argument in favour of more layers: greater depth can, up to a point, reduce the error a network can achieve on a genuinely complex problem.
Overfitting: However, more layers also mean more weights to learn. With a fixed amount of training data, a deeper network can fit the training set (including its noise) more and more closely, which is exactly the signature of overfitting: training loss keeps falling while validation loss stops improving or starts rising. Beyond some depth, extra layers actively make the model worse on new data, not better, because the network is memorising rather than generalising.
Computational cost: Every additional layer also increases the number of weight updates needed during backpropagation and the compute required for both training and inference. A model that is only marginally more accurate but many times slower or more expensive to run may not be "better" in any practical industrial sense, especially for real-time systems like fraud detection or predictive maintenance, where inference speed matters as much as raw accuracy.
Judgement: More hidden layers can improve a model only up to the point supported by the available training data, regularisation, and the compute budget available for training and serving; beyond that point extra depth increases overfitting risk and cost without improving real-world performance. The statement should therefore be rejected as an overgeneralisation, with depth being one hyperparameter to be tuned against validation performance, not maximised.

Marker's note: top-band answers (1) give an explicit thesis, (2) address all three named factors (forward pass, overfitting, cost) with the correct mechanism for each, (3) explain the trade-off rather than treating "more layers" as purely good or purely bad, and (4) end with a direct judgement on the original statement.

Practise this

Sources & how we know this

Software Engineering 11–12 Syllabus (2022) — NESA
Software Engineering HSC exam papers — NESA