Chapter 26 · Artificial Neural Networks

1. The Neuron: From Biology to Arithmetic

1.1 Why copy the brain?

The human brain is the most powerful learning machine we know. Deep learning copies its structure in the hope that a machine built the same way can learn too. The copy starts small: with a single neuron.

Real neurons were first drawn in detail by the Spanish neuroscientist Santiago Ramón y Cajal in 1899. He stained brain tissue, looked at it under a microscope, and drew what he saw. Each neuron has three parts: a cell body, short branches called dendrites, and one long tail called the axon. One neuron alone can do very little — like one ant. Millions of neurons, connected, learn to see, hear and think — like an ant colony. That is the bet of deep learning: connect enough simple units and something powerful emerges.

Two roles carry over. Dendrites receive incoming signals. The axon transmits the neuron's own signal out. A signal travels down one neuron's axon and arrives at the next neuron's dendrites — and the two never touch. The connection where the signal crosses is the synapse.

Artificial neurons borrow the word. Every connection between two artificial neurons is called a synapse. This avoids an awkward question: is a given connection an axon or a dendrite — and whose? A synapse is simply where a signal passes. That is the only meaning the term carries in this book.

1.2 The artificial neuron

An artificial neuron (also called a node) receives input signals through its synapses and produces one output signal. The inputs come either from the input layer or from other hidden neurons — the mechanism is the same. In the brain, the input layer plays the role of the senses: everything you see, hear and touch arrives as electrical signals. In a network, the input layer is simply the row of feature values.

Two facts about the inputs decide whether the network works at all:

• Inputs describe one single observation — one row of your dataset. Age, salary, balance: all descriptors of the same customer. The output the network returns is for that same row. Rows are never mixed; think of one row fed to a linear regression, in and out.
• Inputs must be scaled. The neuron adds its inputs together, so all of them should sit in a similar range — otherwise a salary of 100,000 drowns a 0/1 flag in the sum. Two standard fixes:

1.3 What the output can be

The neuron's output value can take three forms:

A categorical output needs several output values, not one: the categories become dummy variables. This chapter's task is binary (churn or not), so the output layer will hold a single sigmoid neuron.

1.4 Weights: where learning lives

Every synapse carries a number called a weight. The weight decides how much of each input signal passes through: important signals get large weights, unimportant ones get weights near zero. Training a network means adjusting every weight in every synapse until the predictions fit the data. That process — gradient descent driven by backpropagation — is Section 3's subject. Whenever you hear "the network learned", read: its weights changed.

1.5 Inside the neuron: three steps

Signals arrive at the neuron. Then, in order:

1. Compute the weighted sum of the inputs — each input times its weight, plus a bias.
2. Apply the activation function to that sum.
3. Pass the result on as the neuron's output signal.

Look closely: with g = sigmoid, this is exactly the logistic regression of Chapter 11 — a weighted sum squashed to (0, 1). One neuron is one logistic unit. Stack such units in layers, each feeding the next, and the network can bend and recombine its inputs into representations no single linear boundary could reach. Why the non-linear g is essential is Section 2's warning; the calculus of adjusting the weights is Section 3's work.

1.6 The activation menu: four classic functions

The activation function decides whether — and how strongly — a neuron passes its signal on. Four classics cover most of the field.

In one sentence each. Threshold is a rigid switch: below zero it passes 0, at zero or above it passes 1 — yes or no, nothing in between, and (Section 3) no useful gradient to learn from. Sigmoid is its smooth cousin: the same 0-to-1 range with soft slopes, which is why logistic regression uses it and why gradient descent can train it. Rectifier passes 0 for negative sums and the sum itself above zero — cheap to compute and the hidden-layer default. Tanh is sigmoid-shaped but centred at zero, from −1 to +1, which keeps hidden signals balanced around 0.

Suppose the dependent variable is binary — exit or stay. Which functions fit? Two. The threshold function answers directly with 0 or 1. The sigmoid answers with the probability of getting 1 — round it for a hard answer, or keep it and apply Chapter 18's threshold logic. The standard recipe for binary classification combines two of the four: ReLU in the hidden layers, sigmoid at the output — exactly the network Section 2 builds.

2. The Network and Its Algebra

2.1 What do hidden neurons actually learn?

A property-price example makes the hidden layer concrete. Four inputs: area (sq ft), bedrooms, distance to the city and age. One output: the price. The network is already trained — the weights are fixed. What did its hidden neurons each decide to look for?

Each neuron keeps weights only on the inputs it cares about; the rest get weights of zero and are simply ignored. One neuron watches area + distance: it fires for properties close to the city that still have unusually large space — the "unfair square footage" deals. Another watches area + bedrooms + age: large, many-bedroom, new builds — exactly what young families buy. A third watches age alone: under 100 years it stays silent, but past 100 the home becomes historic and the neuron's contribution rises — the rectifier function in action: zero until a threshold, then a growing signal.

That is the trick hidden layers play: each neuron combines raw inputs into a new attribute nobody hand-designed — "family-home-ness", "historic-ness" — and fires only when its pattern is present. One neuron cannot predict a price. Many specialists, combined in the output layer, can — the ant colony again.

2.2 The churn network

The churn network, exactly as the source builds it: 12 inputs → 6 ReLU neurons → 6 ReLU neurons → 1 sigmoid output. Twelve inputs because the raw 10 features become 12 after encoding (Section 4). The output is a single number — the probability the customer leaves. Its 12 hidden detectors are trained, not narrated: some may watch Germany + inactive, others balance + products — the same specialisation, discovered by gradient descent rather than told.

2.3 Counting the parameters, by hand

A layer with n inputs and m neurons owns a weight per edge plus a bias per neuron: n × m + m parameters. The real network:

In matrix form, one layer is one line: a = g(Wᵗx + b), where Wᵗ is that layer's weight matrix. The whole forward pass is three such lines chained — a¹ = ReLU(W₁ᵗx + b₁), then a² = ReLU(W₂ᵗa¹ + b₂), then ŷ = σ(W₃ᵗa² + b₃). GPUs earn their keep by doing these matrix products at enormous speed.

3. Learning: Gradient Descent & Backprop, by Hand

There are two ways to get a program to do what you want. The first: write every rule yourself — pointy ears means cat, floppy ears means dog — and chase every exception by hand. The second: build a network, show it inputs and the correct answers, and let it find the rules on its own. Deep learning is the second road. The name of the game from here is learning: how do the weights get set without anyone telling them?

The simplest network — input layer straight into one output neuron — is the perceptron, built by Frank Rosenblatt in 1957 as the first model meant to learn and adjust itself. Everything in this section happens in the perceptron first, then scales up. One convention before starting: y is the actual value from the data; ŷ ("y-hat") is the network's prediction. Learning means making ŷ agree with y.

3.1 The learning loop and the cost function

Feed one row in. The weights scale it, the neuron activates, out comes ŷ. Compare ŷ to the actual y. The difference is the error, and the cost function is the number that measures it. The classic choice — half the squared error:

3.2 Why not just try every weight? (brute force fails)

One honest idea first: why not try thousands of candidate weights, compute the cost for each, and keep the best? For one weight, it works. For a real network it dies, and the property network of Section 2.1 shows why. Fully connected it has 25 weights (4×5 + 5×1). Test 1,000 candidate values per weight:

3.3 Gradient descent: follow the slope

The working idea is cheaper: stand on the cost curve and look which way is downhill. Pick a starting weight, work out the slope of the cost at that point (the gradient — one derivative), and step the weight the downhill way. At the new point, check the slope again. Repeat. Like a ball rolling down a valley: not a straight ride, a tightening zigzag that settles at the bottom. The bottom is the minimum of the cost — the best weights we can reach from where we started.

In symbols, one step of the walk is the update rule — and it is the nudge rule this chapter's hand example will execute:

3.4 Batch, stochastic, mini-batch — when do the weights update?

Plain gradient descent quietly assumes the cost curve is convex: one bowl, one minimum. Real cost surfaces are often non-convex (Figure 26.7, right) — a different cost function, or a squared error stretched over many dimensions, can grow extra valleys. Descent that updates carefully can then settle in a local minimum and miss the best weights. The fix is when you update:

So the full learning story over a dataset: feed every row (all through the same network — the weights are shared, not per-row), accumulate the cost, update the weights by following the slope, and start the next pass. The whole cycle — forward pass, cost, backward pass, weight update — goes by the name backpropagation.

3.5 Backpropagation by hand: the 2-2-1 network

The network starts with random weights and must learn the 127 numbers of Section 2 — but the hard part is knowing ∂L/∂w for weights buried deep in the stack. That is backpropagation proper: the chain rule applied systematically from the output backwards. Watch it work on a network small enough to hold in your head: 2 inputs → 2 ReLU neurons → 1 sigmoid output, exact numbers throughout.

3.6 Forward pass

3.7 Backward pass — the chain rule, one arrow at a time

Backpropagation's engine is the error signal δ at each neuron: how much the loss would change if that neuron's pre-activation z moved. Signals flow backwards, each layer's δ built from the next layer's:

With δ₀ in hand, the output layer's gradients are one multiplication each (∂L/∂w = δ × the input arriving on that edge):

Now push δ₀ back into the hidden layer. Each hidden neuron's error is the δ₀ flowing in through its output weight, multiplied by its own activation's slope — ReLU's derivative is 1 where z > 0 (both neurons here, z₁ = 1.0, z₂ = 1.2) and 0 where z < 0:

Hidden-layer gradients, same one-multiplication rule:

3.8 One learning step

Apply the update with learning rate η = 0.1 — subtract 0.1 × each gradient. For example w₀₁: 0.7 − 0.1(−0.4329) = 0.7433 (up, as promised), and w₁₂: −0.3 − 0.1(0.1732) = −0.3173 (down). Re-running the forward pass with all nine updated numbers gives the verdict:

3.9 The full training recipe — seven steps

Everything from this section, as one checklist. These are the steps the Keras fit() call of Section 4 runs for you — worth knowing what it is actually doing:

1. Initialize the weights to small random numbers near zero — never exactly zero (Keras does this automatically).
2. Feed in one row: each feature goes to its own input node.
3. Forward-propagate, left to right: weights scale how strongly each neuron activates, layer after layer, until ŷ comes out.
4. Compare ŷ with the actual y and measure the error — the cost.
5. Back-propagate the error, right to left, and update every weight in the same pass, each in proportion to its share of the error. The learning rate η sets the size of the nudge.
6. Repeat steps 2–5 after each row (stochastic) or after each batch (batch / mini-batch).
7. Epoch and repeat: one full pass over the training set is one epoch; run epoch after epoch until the cost stops falling.

4. Training the Churn Network

4.1 The data, prepared

Churn_Modelling.csv: 10,000 customers, 14 columns, label Exited (1 = left the bank). Chapter 3's full toolkit applies before a single neuron fires:

• Drop identifiers — RowNumber, CustomerId, Surname: unique per row, zero generalising power (Ch 3's first rule).
• Label-encode Gender — Female = 0, Male = 1 (alphabetical, sklearn's LabelEncoder convention).
• One-hot Geography — France, Germany, Spain → 3 binary columns placed first in X; 10 features become 12.
• Split 8,000 train / 2,000 test (seed 0) — test holds 1,595 stayers, 405 churners (20.25%).
• Scale — StandardScaler fit on train only. Non-negotiable for networks: salary (≈10⁵) and HasCrCard (0/1) must share a scale or the gradients drown (Ch 12's lesson, doubled).

4.2 The compile step — three choices

4.3 The real learning curve

Training loss and accuracy at eight checkpoints of the actual run (seed 0, the numbers this chapter's model produced):

Two honest readings. First, the plateau is real: by epoch 20 the network has captured nearly everything 127 parameters can hold; the remaining 80 epochs shave a sliver — diminishing returns, exactly like Chapter 6's bias-variance story told in epochs instead of degrees. Second, no memorisation: final training accuracy 0.8645 versus test accuracy 0.8620 (next section) — a gap of 0.3 points. With only 127 parameters against 8,000 examples, this network is far too small to memorise; deep learning's overfitting problem arrives with millions of parameters (Section 7).

6. Reading the Results

The test set: 2,000 customers, 405 of them churners. At the default threshold 0.5, the real confusion matrix:

Chapter 18's metrics, computed: accuracy = 1,724/2,000 = 0.862; precision = 213/297 = 0.717; recall = 213/405 = 0.526; F1 = 0.607. Context makes these numbers meaningful:

The network's advantage is not just accuracy — it found interactions the linear model could not: logistic regression catches only 96 of 405 churners; the ANN catches 213. The predicted probabilities are genuinely graded: min 0.002, mean 0.222, max 1.000 on the test set.

6.1 The threshold is a business decision

A retention campaign costs money per contact; a lost customer costs far more. Chapter 18's threshold logic, run on the real probabilities:

Note the trap: threshold 0.7 has the highest accuracy (0.866) and is the worst decision for a retention team — accuracy is dominated by the 80% majority class (Ch 18's lesson, live). If a retention offer costs little and a churner lost costs much, 0.3 is the operating point.

6.2 The homework customer, answered

The source's own test: France, credit score 600, Male, 40, tenure 3, balance $60,000, 2 products, has card, active, salary $50,000 — the real model's answer:

6.3 What did it learn? Ask the data

A neural network will not hand you a coefficient table — but the dataset's own churn rates show exactly which signals those 127 parameters absorbed:

7. ANN vs the Models So Far

After 25 chapters, the honest ledger. The ANN beat logistic regression by five accuracy points and doubled recall — but Chapter 16's decision trees and Chapter 17's forests were not run on this dataset here, and on tabular data of this size they are formidably hard to beat. What each family is actually for:

Three honest caveats for the road. Scale hunger: ANNs want thousands of examples per feature — at 10,000 rows this network is near the sensible minimum. Tuning burden: layer counts, learning rates, batch sizes, epochs — the source's 6-6 architecture is a sensible hand choice, not a derived one. Opacity: when a regulator asks "why was this customer flagged?", the 127 weights offer no sentence-level answer. The modern playbook: start with the simple model, reach for the network when the simple model demonstrably leaves value behind — or when the data is images, audio or text, where nothing else comes close.

8. From Yes/No to How Much: Regression with the ANN

The churn network answers a yes/no question: will this customer leave? Chapter 25 met three-answer questions. There is a third shape, and finance asks it constantly: how much? What will the price be tomorrow? What will demand be next week? The same network answers it — with a new head and a new loss. Nothing else changes.

8.1 Three changes

Why remove the activation? The sigmoid squeezes its output into 0–1 — a probability. A price can be any number, so the last neuron must be free to output any number: just weights, sum, done. And one practical habit: scale y too (subtract the train mean, divide by the train standard deviation), train on the scaled target, then un-scale the predictions.

8.2 The finance rules — more important than the code

• Predict returns, not prices. A network trained to output "tomorrow's price" learns one cheap trick: copy today's price. It looks accurate and predicts nothing. The honest target is the return — tomorrow's price divided by today's.
• Split by time, never randomly. Train on the past, test on the future. A random shuffle lets the network peek at tomorrow while training — Chapter 3's leakage rule, fatal in markets.
• Always race the naive baseline: "tomorrow equals today" (return = 0). If a model cannot beat doing nothing, it is not a model.
• Next week instead of next day? Same machinery — the target becomes the 5-day forward return.

8.3 The real run: can the ANN read tomorrow's AAPL?

We ran it, for real, on the course's own equity snapshot (AAPL, January 2020 – September 2026). Features: the last five daily returns. Target: tomorrow's return. Time split: train on the first 80% of days (through 2025-05-27), test on the last 338 days. Same 8-4-1 network as the mini-project — with the linear head and MSE loss:

# ---- Listing 26.8: the regression head, end to end ----
r = np.diff(np.log(close))                  # daily log returns
X = np.array([r[t-5:t] for t in range(5, len(r))])   # 5 lagged returns
y = r[5:]                                   # tomorrow's return
cut = int(0.8 * len(y))                     # TIME split — no shuffle
sc = StandardScaler()
X_train, X_test = sc.fit_transform(X[:cut]), sc.transform(X[cut:])
y_train, y_test = y[:cut], y[cut:]
ym, ys = y_train.mean(), y_train.std()      # scale y on train statistics

model = tf.keras.Sequential([
    tf.keras.layers.Dense(8, activation='relu', input_shape=(5,)),
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1)])              # linear head — any number
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(X_train, (y_train - ym) / ys, epochs=100, batch_size=32, verbose=0)
p = model.predict(X_test, verbose=0).ravel() * ys + ym   # un-scale

print(np.abs(p - y_test).mean())            # 0.01112   MAE
print(np.abs(y_test).mean())                # 0.01088   naive baseline's MAE
print(((p > 0) == (y_test > 0)).mean())     # 0.518     direction accuracy

Listing 26.8 — The regression ANN on AAPL's next-day return. Deterministic run (seed 0, op-determinism on): every number reproduces on the same TensorFlow build.

Read it honestly. The correlation between predicted and actual returns is −0.01; R² is −0.04. The network works — it outputs a number, and the last close of $333.02 got a tomorrow prediction of $334.05 — and it is beaten by doing nothing. Five lagged returns carry almost no signal. Markets price obvious information in seconds; Mini-Project 1 found the same wall with headlines, and Applied Project 1 with technical indicators. This is that lesson arriving from the ANN's direction.

What would make it a real experiment rather than a parable? Features with genuine information (order flow, fundamentals, cross-asset signals), walk-forward retraining instead of one split, and transaction costs in every number. Applied Project 3 (SVR) will meet this exact task again with a different learner — and should expect the same honest wall.