Quick questions on Neural network basics explained: HSC Software Engineering Module 3

The basic unit. It takes inputs $x_1, x_2, \dots, x_n$ from the previous layer, multiplies each by a weight, sums them, adds a bias, and applies an activation function:

A neural network is a stack of layers:

To make a prediction, feed the input through every layer in turn. Each layer computes its weighted sums and activations. The output layer produces the prediction.

Measures how wrong the prediction is.

The training algorithm. For each batch of training examples:

A minimal feed-forward network in PyTorch:

Choices the developer makes that affect training:

A neural network with enough parameters can memorise the training data exactly. Such a model has zero training error but performs poorly on new data. Detection: training loss keeps falling while validation loss starts rising. Prevention: more training data, smaller network, dropout, regularisation, early stopping.

For images and other structured inputs, specialised architectures perform far better:

Each neuron has a learnable bias, not just weights. Without it, the network is constrained to predictions that pass through the origin.

Different layers can have different activations. Convolutional layers, pooling layers, normalisation layers are different from fully connected layers.

The biological metaphor is loose. Artificial neurons are simple weighted sums; biological neurons are far more complex. Use the analogy carefully.

Training adjusts weights; inference applies the trained weights to new data. Inference is much cheaper than training. :::