Gradient Descent: The One Loop Behind Every AI, From a Tiny Model to an LLM

The one-line idea

Every AI model is a calculation with adjustable numbers inside it. Training means changing those numbers until the model's mistakes get smaller. The method that does the changing is called gradient descent.

An LLM (large language model, the kind behind chat assistants) is trained with this same method. It is not a different trick. It is this loop, repeated an enormous number of times.

This article shows the loop on two tiny examples side by side, then explains what is different for an LLM.

Part 1: What is "loss"?

To improve something, you first need to measure how bad it is right now. That measurement is a single number called the loss. Lower is better, 0 means perfect.

Example 1. We want $w$ to be 3 (the model does not know this). We measure the loss as the squared distance from 3:

$L(w) = (w-3)^2$

The loss is 0 only at $w=3$. The model's job is to find that value.

Example 2. The model sees the words "the cat" and must guess the next word out of four options: cat, dog, sat, mat. The correct next word is sat.

The model gives each word a score (any number). To compare words fairly, scores are turned into probabilities (numbers between 0 and 1 that add up to 1). The formula is called softmax:

$\text{probability of a word} = \frac{e^{\text{its score}}}{\text{sum of } e^{\text{score}} \text{ over all words}}$

You do not need to memorise it. See what it does:

Equal scores give equal probabilities. A higher score gives a higher probability.

The loss only cares about the probability given to the correct word:

$L = -\ln(\text{probability of the correct word})$

Giving the right word a high probability means low loss. Starting with all scores at 0, the loss is $-\ln(0.25) = 1.386$.

Part 2: What is the gradient?

We know how bad the model is. Now: which way should we change the number to make the loss smaller?

The gradient answers this. It is the slope: if I increase this number a tiny bit, how much does the loss change?

Example 1: check with real numbers. At $w=0$ the loss is 9. Move $w$ up by 0.01:

Loss changed by $-0.0599$ for a $+0.01$ change, so the slope is about $-5.99$, roughly -6.

The exact formula is $2(w-3)$, which at $w=0$ gives $2(0-3) = -6$. Matches.

• Negative gradient: increasing the number lowers the loss, so increase it.
• Positive gradient: decrease the number.
• Big gradient: the loss is very sensitive to this number.

Example 2. For softmax with this loss, the gradient of each word's score has a simple form:

$\text{gradient} = \text{probability} - (1 \text{ for the correct word, else } 0)$

At the start (all probabilities 0.25):

The correct word ("sat") has a negative gradient, so raise its score. The wrong words have positive gradients, so lower their scores.

Part 3: The update (the "descent")

Move each number against its gradient:

$\text{new value} = \text{old value} - \text{lr} \times \text{gradient}$

lr is the learning rate, a step size we choose. This is the "descent" in gradient descent: the loss goes down because every step moves against the slope.

Example 1 (lr = 0.1)

Example 2 (lr = 1)

New loss: $-\ln(0.475) = 0.744$ (was 1.386). Both examples improved after one step.

Part 4: Repeat

Training is this loop, repeated:

1. Measure the loss
2. Compute the gradient
3. Update the numbers
4. Go back to 1

Example 1 (lr = 0.1)

Example 2 (lr = 1)

Notice in both: the loss falls fast at the start and slowly later. As the model improves, the gradient gets smaller, so steps get smaller. In Example 2, getting from 0.96 to 0.99 takes 80 more steps.

Part 5: The learning rate matters a lot

Example 1, same start ($w=0$), different learning rates, after 30 steps:

The learning rate is something the person training the model has to choose. Too small wastes time. Too big breaks training.

Part 6: So how is an LLM different?

An LLM does Example 2, scaled up. Here is what changes and what does not.

How training text becomes examples: take a piece of text, hide the next word, ask the model to guess it, measure the loss, update. Every position in every piece of text is a free example, because the correct answer is already in the text.

The one real addition is that the scores come from a long chain of calculations, not from free numbers. Finding the gradient for every adjustable number in that chain uses the chain rule again and again. That process has its own name, backpropagation, but it is still just "find each number's slope."

What this article does not cover

• How the model computes its scores. That part (in modern LLMs, an architecture called a transformer) is a separate topic.
• Training tricks. Real training updates using small batches of data at a time, and usually uses improved versions of gradient descent. The core step is still "move against the gradient."
• What happens after. Models are often trained further after this stage, which this article does not describe.

Glossary

Full code

import numpy as np

# Example 1: find w that minimises (w-3)^2
w = 0.; lr = 0.1
for step in range(1, 31):
    grad = 2*(w - 3)
    w -= lr*grad
    if step in (1, 3, 10, 30):
        print(step, round(w, 4), round((w-3)**2, 4))

# Example 2: guess the next word (cat, dog, sat, mat), correct = sat
softmax = lambda z: np.exp(z)/np.exp(z).sum()
scores = np.zeros(4); correct = 2; lr = 1.0
for step in range(101):
    p = softmax(scores)
    loss = -np.log(p[correct])
    if step in (0, 1, 5, 20, 100):
        print(step, p.round(3), round(loss, 4))
    grad = p.copy(); grad[correct] -= 1     # probability minus (1 for correct word)
    scores -= lr*grad

Try: change lr in Example 1 to 1.1 and watch the loss explode, like the table in Part 5.