Neural Networks From Zero: Two Tiny Examples, Every Step Explained

Part 1: The big picture

Where neural networks are used

Every row has the same shape: numbers go in, a decision or a number comes out. Photos, text and sound all get converted to numbers first.

Two kinds of output

Look at the "comes out" column. There are only two kinds:

1. A number. Price, temperature. This is called regression.
2. A choice between categories. Spam or not, you or not, which next word. This is called classification. The simplest case is yes/no.

Even the chat assistant is classification: it chooses one word out of a list of thousands.

Why this article uses exactly these two

The two kinds of output need slightly different parts inside the network. If we understand one example of each, we understand the full picture. Everything else is a bigger version.

Why such tiny data

So that every number can be checked by hand. Real networks do exactly the same steps with millions of numbers.

Part 2: The problem: learning from examples instead of rules

Normally, a programmer writes the rule: "multiply by 2, add 1."

Here the rule is hidden. We only have examples, and the computer must find the rule.

Example A: examples

Example B: examples (AND gate)

Words we will use:

• Input: the numbers we give the model.
• Output: the number the model gives back.
• Target: the correct output from the examples. We write it $y$ in A and $t$ in B.
• Prediction: what the model outputs. We write it $\hat y$ in A and $p$ in B.

Part 3: What is a "model"?

A model is a rule with adjustable numbers inside it. Start with the simplest rule that can fit Example A:

$\hat y = w \cdot x + b$

The two adjustable numbers are called parameters:

• $w$ (weight): multiplies the input. It controls how strongly the input affects the output.
• $b$ (bias): added at the end. It shifts the output up or down regardless of input.

Why we need both

Try input $x=2$ with different parameters:

With only $w$, we could not get exactly 5 and also 3 and 7 for the other inputs. The bias gives the second adjustable number. Learning means finding the right $w$ and $b$.

This is already a neuron

A neuron is this exact thing: take inputs, multiply each by its weight, add them up, add the bias. The result of that arithmetic is called the weighted sum, and we write it $z$:

$z = w_1x_1 + w_2x_2 + \dots + b$

For Example A there is one input, so $z = wx+b$. That is the entire neuron for regression.

Example B needs two changes

Change 1: two inputs. The AND gate has two inputs, so we have two weights:

$z = w_1x_1 + w_2x_2 + b$

Change 2: the output must be a probability. For yes/no we want a number between 0 and 1 (0 = surely no, 1 = surely yes, 0.5 = unsure). But $z$ can be any number, like -7 or 13. We need something that converts any number into the range 0 to 1.

That something is called an activation function: a function applied to $z$ after the weighted sum. The one we use is sigmoid:

$\sigma(z) = \frac{1}{1+e^{-z}}$

You do not need to memorise the formula. Look at what it does:

Large negative $z$ becomes close to 0. Zero becomes exactly 0.5. Large positive $z$ becomes close to 1. So sigmoid turns any number into a probability.

Example A has no activation function, because a price or a value can be any number.

Summary of the two neurons

Part 4: The learning loop

Learning is the same 5-step loop in both examples:

1. Forward pass: make a prediction with the current parameters.
2. Loss: measure how wrong it is.
3. Gradient: find which direction to change each parameter.
4. Update: change the parameters in that direction.
5. Repeat.

We will do one full pass of this loop by hand on one sample:

• A: $x=2$, target $y=5$
• B: $x=(1,1)$, target $t=1$

Step 0: Starting values

All parameters start at 0.

Why start with wrong values? The model has not seen anything yet, so no values are known to be right. Any starting point works for a single neuron. (For bigger networks, starting values need to differ, but that is a later topic.)

Step 1: Forward pass

Forward pass means: push the input through the neuron to get a prediction. "Forward" because numbers flow from input to output.

Both predictions are wrong. But "wrong" is just a feeling. The computer needs a number that says how wrong. That is the next step.

Step 2: Loss

The loss is a single number measuring how far the prediction is from the target. Lower is better, and 0 means perfect.

Example A: squared error

$L = (\hat y - y)^2$

Our values: $(0-5)^2 = 25$.

Why squared? Two reasons. The difference could be negative (predicting 8 when the target is 5 gives -(-3)), and squaring makes it positive. Squaring also makes big mistakes count much more than small ones: off by 1 costs 1, off by 4 costs 16.

Loss for sample $x=2, y=5$ with $b=0$ and different $w$:

The loss is lowest near $w=2.5$ (where the prediction would be exactly 5). That is where we want to end up.

Example B: binary cross-entropy

$L = -[t\ln p + (1-t)\ln(1-p)]$

When the target is $t=1$, this becomes $L = -\ln p$. Look at what it does:

If the model is confident and right, loss is tiny. If it is unsure (0.5), loss is 0.693. If it is confident and wrong (0.01 when the answer is 1), loss is huge.

Why not use squared error here too? Cross-entropy punishes confident wrong answers much harder, which is exactly the mistake a classifier must avoid. It also gives better learning signals with sigmoid (this shows up in Step 3).

Our values: $-\ln(0.5) = 0.693$. This is the loss of a pure coin flip. The model knows nothing yet.

Step 3: Gradient

Now we know how wrong the model is. The question is: how should we change $w$ and $b$ to reduce the loss?

We could try every possible value, but with millions of parameters that is impossible. Instead we use the gradient.

What the gradient is. The gradient of the loss with respect to a parameter says: if I increase this parameter a tiny bit, how much does the loss change? It is the slope.

Check with real numbers (Example A). Current $w=0$, loss 25. Increase $w$ by 0.01:

Loss changed by $-0.1996$ for a change of $+0.01$, so the slope is about $-0.1996/0.01 = -19.96$, roughly -20.

• A negative gradient means: increasing the parameter decreases the loss. So increase it.
• A positive gradient means the opposite: decrease the parameter.
• A large gradient means the loss is very sensitive to that parameter.

We can get this exactly, without trying values, using the chain rule:

If A depends on B, and B depends on C, then how fast A changes with C = (how fast A changes with B) × (how fast B changes with C).

In our case, loss depends on the prediction, and the prediction depends on $w$.

Example A

The -20 matches our hand-check of about -19.96.

Example B

The pattern. In both examples:

$\text{gradient of a weight} = (\text{error signal}) \times (\text{input that weight multiplies})$

A's error signal is $2(\hat y - y)$ and B's is $p-t$. The rest is identical. That is why the same learning method works for both. (The sigmoid + cross-entropy pair is chosen so the error signal is the clean $p-t$.)

Why input matters. If an input is 0, its weight gets gradient 0. That weight did not contribute to the prediction, so it is not blamed.

Step 4: Update

Move each parameter against its gradient (opposite direction, since the gradient points toward higher loss):

$\text{parameter} \leftarrow \text{parameter} - \text{lr} \times \text{gradient}$

Learning rate (lr) is a number we choose. It controls how big a step we take.

Why not just jump straight to the answer? The gradient only tells the local slope, not the exact destination. A step too large overshoots. Example A with lr = 0.3 instead of 0.01:

Too small a learning rate learns very slowly. Too large overshoots. It is a number you tune.

Step 5: Check that it worked

Run the same sample again with the new parameters.

Both losses dropped after one update. That is the whole idea.

Part 5: Repeating on all the data

One step on one sample only improves that sample. The real loop does this:

1. Run every sample through the model.
2. Compute the gradient for each sample, then average them.
3. Update the parameters once using that average.
4. Repeat.

Why average? If the update followed only one sample, it could damage the others. The average moves in a direction that helps the whole dataset.

One full pass over all samples is called an epoch.

Example A (lr = 0.05, 3 samples)

Hidden rule was $w=2, b=1$. The model reached $w=2.004$, $b=0.991$ without being told.

Notice the loss falls fast at first and then slowly. As the model gets close, gradients get smaller, so steps get smaller.

Example B (lr = 1, 4 samples)

Look at epoch 1: the weights did not move, only the bias did. Sample (1,0) pushes $w_1$ down and sample (1,1) pushes it up by the same amount, so the average cancels. After the bias changes, the outputs differ and the weights begin to move.

Part 6: Using the trained model

Example A: feed $x=4$, a value the model never saw.

Prediction: $2.004\cdot4+0.991 = 9.007$. The correct answer is 9. The model learned the rule, not a lookup table.

Example B: outputs after training are 0.000, 0.020, 0.020, 0.972. Choose a threshold of 0.5 (above it we say yes, below it no). That gives 0, 0, 0, 1, which is exactly the AND gate.

Part 7: Same and different

Part 8: What a single neuron cannot do

A single neuron separates the data with one straight line. AND is separable by one line. XOR (output 1 when inputs differ) is not. To solve XOR, we stack neurons into layers, where the first layer's outputs become the second layer's inputs. The same loop still trains it, only the chain rule gets longer. The follow-up experiment tests this.

Glossary

Full code

import numpy as np
sig = lambda z: 1/(1+np.exp(-z))

# Example A: regression, y = 2x + 1
x = np.array([1, 2, 3.]); y = np.array([3, 5, 7.])
w = b = 0.; lr = 0.05
for epoch in range(200):
    p = w*x + b                      # forward pass
    d = 2*(p - y)/len(x)             # error signal, averaged
    w -= lr*(d*x).sum()              # gradient = error x input, then update
    b -= lr*d.sum()
print(w, b, w*4 + b)

# Example B: AND gate
X = np.array([[0,0],[0,1],[1,0],[1,1.]]); t = np.array([0,0,0,1.])
w = np.zeros(2); b = 0.; lr = 1.
for epoch in range(1000):
    p = sig(X@w + b)                 # forward pass
    d = (p - t)/4                    # error signal, averaged
    w -= lr*X.T@d                    # gradient = error x input, then update
    b -= lr*d.sum()
print(w, b, sig(X@w + b).round(3))

Try: change lr to 0.3 in Example A and watch the loss blow up, like the overshoot table in Step 4.