What Is a Neuron? Trying 7 Kinds on 3 Tiny Puzzles

People hear "neuron" and think it is something complicated. It is not. Seeing 7 kinds side by side, on the same puzzles, shows what changes between simple and advanced, and what does not.

What a neuron does (in plain words)

1. Take the inputs and multiply each one by its own weight (a number we can change).
2. Add them all up, then add one extra number called the bias.
3. Optionally, push the total through a "squash" step that reshapes it.

The 7 kinds below only differ in step 3.

The three puzzles (two inputs, each 0 or 1)

• AND: answer is 1 only if both inputs are 1.
• OR: answer is 1 if at least one input is 1.
• XOR: answer is 1 if the two inputs are different.

Settings

• Kinds 2 to 5: learning speed 0.5, 2000 rounds, measuring mistakes as squared difference
• Kind 1 (step): its own simple rule, speed 0.1, 200 rounds
• Kinds 6 and 7: speed 0.5, 5000 rounds
• Starting weights: random numbers, bias starts at 0
• 10 different random starts (seeds 0 to 9) for every kind on every puzzle
• Counted as "solved" when all 4 answers are right (answer above 0.5 means 1)

Plain numpy. Each neuron is trained by nudging its weights a little bit every round to make its mistakes smaller.

import numpy as np
X = np.array([[0,0],[0,1],[1,0],[1,1.]])
tasks = {"AND": np.array([0,0,0,1.]),
         "OR":  np.array([0,1,1,1.]),
         "XOR": np.array([0,1,1,0.])}
sig = lambda z: 1/(1+np.exp(-z))

acts = {  # (squash, how fast it changes)
 "linear":  (lambda z: z,              lambda z,o: np.ones_like(z)),
 "sigmoid": (sig,                      lambda z,o: o*(1-o)),
 "tanh":    (np.tanh,                  lambda z,o: 1-o**2),
 "relu":    (lambda z: np.maximum(z,0),lambda z,o: (z>0)*1.),
}

def single(act, t, seed, lr=0.5, ep=2000):
    r = np.random.default_rng(seed); w = r.normal(0,1,2); b = 0.
    f, df = acts[act]
    for _ in range(ep):
        z = X@w + b; o = f(z); g = 2*(o-t)/4*df(z,o)
        w -= lr*X.T@g; b -= lr*g.sum()
    return np.mean((f(X@w+b) > 0.5) == t)

def step(t, seed, lr=0.1, ep=200):
    r = np.random.default_rng(seed); w = r.normal(0,1,2); b = 0.
    for _ in range(ep):
        for x, y in zip(X, t):
            o = float(x@w + b > 0)
            w += lr*(y-o)*x; b += lr*(y-o)
    return np.mean((((X@w+b) > 0)*1.) == t)

def group(h, t, seed, lr=0.5, ep=5000):
    r = np.random.default_rng(seed)
    W1 = r.normal(0,1,(2,h)); b1 = np.zeros(h)
    W2 = r.normal(0,1,(h,1)); b2 = 0.
    for _ in range(ep):
        a = np.tanh(X@W1+b1); o = sig(a@W2+b2)[:,0]
        g2 = (2*(o-t)/4*o*(1-o))[:,None]
        g1 = (g2@W2.T)*(1-a**2)
        W2 -= lr*a.T@g2; b2 -= lr*g2.sum()
        W1 -= lr*X.T@g1; b1 -= lr*g1.sum(0)
    o = sig(np.tanh(X@W1+b1)@W2+b2)[:,0]
    return np.mean((o > 0.5) == t)

def report(name, fn):
    out = [sum(fn(t,s)==1 for s in range(10)) for t in tasks.values()]
    print(f"{name:18s} AND {out[0]}/10  OR {out[1]}/10  XOR {out[2]}/10")

report("step", lambda t,s: step(t,s))
for a in acts: report(a, lambda t,s,a=a: single(a,t,s))
report("group of 2", lambda t,s: group(2,t,s))
report("group of 4", lambda t,s: group(4,t,s))

How many of the 10 random starts got all 4 answers right:

• Kinds 1 to 4 behaved the same on these puzzles. All got AND and OR every time, and none ever got XOR. So changing the squash step did not give the neuron any new power here.
• XOR only became possible when we used a group of neurons. 2 neurons got it 7 times out of 10, and 4 neurons got it 9 times out of 10.
• So the big jump from simple to advanced is not a fancier single neuron. It is using many neurons together.

• One neuron can only draw one straight line to split the answers. That is enough for AND and OR, but not for XOR.
• The squash step matters for other reasons (what kind of output you want, how smoothly it learns), but it does not make one neuron smarter.
• More neurons working together can do things one neuron cannot.

• Only 4 data points and 3 puzzles. Real problems are much bigger.
• We used one learning speed and one number of rounds per kind. Other settings could change the scores.
• 10 random starts is a small sample, so 7/10 vs 9/10 is a hint, not proof.
• We tested only 7 kinds. Real neural networks use many more (for example neurons with memory or neurons that look at small patches of an image), and we did not test those.