Think of the last time you practised a skill — shooting a basketball, learning a new recipe, or solving a type of maths problem you had never seen before. You tried something, noticed how far off you were, and adjusted. With every attempt your brain moved toward a better version of the action. That cycle — try, measure, adjust, repeat — is learning.

Machines follow exactly the same logic. A model begins with a random guess at how to perform a task. It measures how wrong that guess is, figures out which direction would make things better, takes a small step in that direction, and repeats until it gets good enough. That process is called gradient descent — the single most important optimisation algorithm in all of machine learning.

The four big questions that drive this entire topic are: What does it mean for a machine to learn? How does it know if it is doing well or badly? How does it improve step by step? And what can go wrong, and how do we fix it? These slides answer each in turn.

Before a machine can improve, it needs a score that tells it how badly it is performing. That score is called a loss function (also called a cost function), written J(w). The smaller J(w), the better the model is performing. The machine adjusts its parameters w to minimise J(w).

Consider the house rental example: the model predicts ₹300k; the real price is ₹350k. The error = 350 − 300 = 50 (in thousands). Using squared-error loss, J = (error)² = 2,500. Squaring has two purposes: it makes all errors positive (over-predictions and under-predictions don’t cancel each other out), and it penalises large errors far more harshly than small ones — which matches our intuition that being wildly wrong is much worse than being slightly wrong.

The machine’s entire job is to find the value of w that sits at the lowest point of the J(w) curve. Everything in gradient descent is a systematic strategy for finding that lowest point.

Not every loss function is equally useful. A good loss function must possess three key properties:

The gold standard: convex loss functions

A convex function is shaped like a bowl. Draw a straight line (called a chord) connecting any two points on the curve — that chord will lie entirely above the curve. This geometric property guarantees that only one valley exists. Gradient descent started from any point on a convex function is mathematically guaranteed to find the global minimum, regardless of starting position. Linear regression using MSE loss is convex — which is why it always converges cleanly.

Historical insight: the birth of loss functions

Real neural networks do not have smooth, bowl-shaped loss surfaces. Their landscapes contain thousands of valleys, plateaus, and saddle points. Gradient descent risks getting trapped in a local minimum — a valley that looks like the bottom from inside, but is not the deepest valley globally.

Consequences of non-convex surfaces include: multiple local minima (many false valleys); slower convergence to the true global minimum; and the possibility of never finding the global minimum at all. The problem grows worse as the model gains more parameters wᵢ.

However, modern research reveals an important nuance: in very high-dimensional spaces (millions of parameters), most local minima are actually quite close in loss value to the global minimum. The real danger in practice turns out to be saddle points — where the gradient is zero but you are not at a minimum — and flat plateaus where the gradient is near zero and learning stalls completely.

Solution 1 — smart initialisation

Where you start (w₀) determines which valley you fall into. Three strategies help escape bad starts:

• Avoid bad starts: initialise w with small random values drawn from a uniform or normal distribution centred around 0. Large initial values push the model into extreme regions of the loss surface.
• Warm start: compute a rough approximate solution first (e.g. using a simpler model), then use that as the starting point for full gradient descent.
• Multiple random restarts: run gradient descent several times from different random starting points and keep whichever run achieves the lowest final loss. Computationally expensive but effective.

Solution 2 — adding momentum

Momentum adds a memory of the previous step’s direction to the current update — like a heavy rolling ball that carries its speed through small dips. With momentum coefficient γ (gamma), the update becomes:

γ controls how much of the previous step direction is retained. A typical value is γ = 0.9, meaning 90% of the previous direction carries forward. Momentum helps in two ways: it accelerates descent along consistent directions, and it can roll through small local minima and cross flat plateaus that would trap standard gradient descent.

Solution 3 — regularisation

Regularisation modifies the loss function itself by adding a penalty term that discourages unnecessarily large weights. This smooths out the loss landscape, reducing the number and sharpness of local minima and making gradient descent more reliable.

The parameter λ controls the strength of the penalty. L2 regularisation shrinks all weights proportionally — useful when all features contribute somewhat. L1 regularisation can force many weights to exactly zero, effectively performing automatic feature selection — particularly useful in text analysis where most words are irrelevant to a specific task.

The gradient ∂J/∂w is the slope of the loss function at the current position in parameter space. It points in the direction of steepest increase. To reduce the loss we move in the opposite direction — downhill. This is precisely why we subtract the gradient, never add it.

For a single-layer perceptron specifically, the gradient of the loss with respect to the weights simplifies to ∇J = −yᵢxᵢ, where yᵢ is the true label and xᵢ is the input vector.

A beautiful built-in property of this rule: it automatically slows down near the minimum. As w approaches the lowest point, the slope ∂J/∂w approaches zero — so each step shrinks naturally, without any extra logic. Near the bottom of the bowl, the algorithm takes small, precise steps.

When to stop?

Two practical stopping criteria exist in common use:

The learning rate η is arguably the single most important hyperparameter you control when training a model. It determines the size of each downhill step. Get it wrong and the model either crawls to the answer or flies right past it.

The practical tip from the slides: start with η = 0.01 as a default and adjust based on your training loss curve. A well-behaved curve falls quickly in early epochs and then levels off smoothly near a low value. A jagged, noisy, or rising curve signals a learning rate problem. The recommended starting range is 10⁻³ to 10⁻¹.

The best-of-both-worlds strategy is to start training with a relatively high learning rate (to cover the loss landscape quickly and escape bad regions) and gradually reduce it as training progresses (for precision near the minimum). This is called a learning rate schedule.

Standard gradient descent requires you to guess (or schedule) a learning rate η. Newton’s method eliminates guessing by computing the mathematically optimal step size at every single point using the curvature of the loss function — not just its slope.

The algorithm works in three steps:

1. At the current position w, fit a quadratic curve (parabola) locally to the loss surface using both the first and second derivatives.
2. Jump directly to the minimum of that local parabola — this is the analytically exact optimal step.
3. Repeat until convergence.

Newton descent converges in far fewer steps than gradient descent because it accounts for the curvature of the surface at every point — not merely the slope. The learning rate is automatically optimal at each step and requires no tuning. However, computing and inverting H is extremely expensive for large models.

All three variants use the same core update equation. They differ only in how much data is used to compute the gradient before each weight update. This single design choice has profound consequences for speed, stability, memory usage, and GPU efficiency.

Variant 1 — stochastic gradient descent (SGD)

Rule: k = 1 (one sample per update). Weights are updated after every single training example. The data is shuffled at the start of each epoch so examples are seen in a different random order.

Variant 2 — mini-batch gradient descent

Rule: 1 < k < N (small batch per update). Weights are updated after processing a mini-batch of k examples. This is by far the most widely used approach in modern deep learning. Standard batch sizes: k = 32 to 256.

Variant 3 — batch gradient descent

Rule: k = N (entire dataset per update). The gradient is computed over all N training examples before any weight update occurs. One update per epoch.

Practical note: Batch gradient descent is rarely used for large real-world datasets. It is useful for small problems (a few thousand examples) or as a theoretical baseline to understand convergence behaviour.

The three variants make different trade-offs across seven dimensions that matter in practice. Visualising the path each method traces through parameter space (the θ₀, θ₁ space) makes their character immediately clear: batch descent moves in straight, deliberate steps; mini-batch moves in gentle arcs; SGD zigzags erratically but covers ground with great speed.

Backpropagation — how gradients are computed inside deep networks

The slides explain what to do with the gradient once you have it. They do not explain how the gradient is computed inside a multi-layer neural network. That process is called backpropagation (or backprop). It applies the chain rule of calculus repeatedly, starting from the output layer (where the loss is measured) and working backward through every layer toward the input, computing how much each parameter contributed to the final error. Without backpropagation, gradient descent cannot be applied to deep networks at all.

Adaptive optimisers — Adam, AdaGrad, RMSProp

Standard gradient descent gives every parameter the same learning rate η. Adaptive optimisers automatically give each parameter its own learning rate, adjusted dynamically based on the history of that parameter’s gradient updates. Parameters updated rarely (e.g. rare words in a language model) receive larger steps; parameters updated constantly and consistently receive smaller, more careful steps.

Overfitting and the validation loss

Gradient descent can sometimes optimise the training loss too well — the model memorises the training examples rather than learning general patterns. This is called overfitting. To detect it, practitioners always monitor a separate validation loss computed on data the model has never seen during training. When training loss continues falling but validation loss starts rising, overfitting has begun.

Vanishing and exploding gradients

In very deep networks, gradients computed by backpropagation can become either extremely small (vanishing — early layers receive near-zero gradient signals and barely learn) or extremely large (exploding — weights grow unstable and diverge). This was the primary barrier to training deep networks from the early 1990s through the mid-2000s, effectively halting deep learning research for nearly two decades. Solutions include: careful weight initialisation (Xavier / He initialisation), batch normalisation, gradient clipping, and residual connections (skip connections in ResNet-style architectures that allow gradients to flow directly through many layers unchanged).

Batch normalisation

Proposed by Ioffe and Szegedy in 2015, batch normalisation normalises the inputs to each layer during training, keeping them in a stable range. This dramatically smooths the loss landscape — imagine ironing out wrinkles in the terrain — allowing much higher learning rates and dramatically faster convergence. It also acts as a mild regulariser. Batch normalisation is now a standard component of virtually every modern deep neural network and is one of the most impactful techniques developed since backpropagation itself.