Study Guide · Neural Networks · 7 min read

How Neural Networks Learn Under the Hood

Two ideas carry all of deep learning: backpropagation, which assigns blame for every mistake, and scaling laws, which predict what more compute will buy. Here is how they fit together.

Forward pass, then backward pass

A training step has two halves. The forward pass pushes an example through the layers to produce a prediction. The loss function measures how wrong that prediction is, as a single number.

Then the backward pass runs the same network in reverse, using calculus (the chain rule) to compute exactly how much every one of millions of weights contributed to the error — each weight gets its share of the blame.

Gradient descent, billions of times

Each weight moves a tiny step in the direction that reduces error; repeat over trillions of examples in small batches and the network converges on useful behavior. The optimizer (like Adam) sets the step sizes smartly.

Activation functions between layers add the non-linearity — without them, a thousand layers would collapse into one boring linear equation. Innovations like ReLU and attention are activation-pattern innovations.

Scaling laws and the transformer era

Empirically, bigger models trained on more data keep getting predictably better — loss falls along smooth curves. That discovery justified the massive GPU investments behind modern AI.

The transformer architecture (2017) made scaling efficient through attention, which lets every token look at every other token in parallel. Nearly every famous model today — GPT, Claude, Gemini — is a transformer being scaled.

Key Points

  • Backpropagation = the chain rule assigning blame for errors to every weight.
  • Gradient descent makes tiny corrective steps across billions of examples.
  • Non-linear activations are what give depth its power.
  • Transformers + scaling laws turned neural training into forecastable engineering.

All study guides for this term: Neural Networks, Explained Simply · How Neural Networks Learn Under the Hood · Neural Networks in the Real World