Neural Networks

Eternal Term 2 of 7

Layered computation inspired by the brain — the substrate of modern AI

What It Means

A neural network is a system of interconnected simple units (neurons) organized in layers, each computing a weighted sum followed by a non-linear activation. By adjusting millions or billions of weights through backpropagation, networks learn to approximate extremely complex functions directly from data.

Why It Is Eternal

Neural networks were proposed in 1943, dismissed in the 1970s, revived in the 1980s, and triumphant in the 2010s — yet the underlying idea never changed: compose simple differentiable units into deep hierarchies that can represent almost anything.

They endure because they are universal approximators with a property no other family matches: learnable representations. Each layer builds features from the layer below — edges to faces in vision, tokens to meaning in language. Every modern breakthrough — CNNs, transformers, diffusion models, large language models — is a new neural architecture, not a departure from the paradigm.

Hardware co-evolution cements them: GPUs, TPUs, and custom accelerators exist because neural networks are parallel by nature, and their economics improve every year.

Core Ideas

Layers and representations: Depth lets networks build hierarchies of features — pixels to edges to objects in vision, tokens to syntax to plans in language. Depth is where capability compounds.
Backpropagation: The chain rule applied at scale: errors flow backward through the network and every weight learns its share of the blame. This single algorithm trains nearly all modern AI.
Activation functions: Non-linearities (ReLU, GELU, attention softmax) are what let networks model anything beyond simple linear relationships. Architecture innovation is largely connection and activation innovation.
Scaling laws: Loss falls predictably as models, data, and compute grow. This empirical law turned neural network training into an engineering discipline with forecastable returns.

Where It Shows Up

Large language models powering chat, coding, and reasoning assistants
Vision transformers for medical imaging, autonomous driving, and satellite analysis
Diffusion models generating images, video, and molecular structures
Speech recognition and synthesis in every modern voice interface

Milestones Through Time

1943 — McCulloch and Pitts model the neuron as a logical unit.
1986 — Rumelhart, Hinton, and Williams popularize backpropagation for multi-layer networks.
2017 — "Attention Is All You Need" introduces the transformer, the architecture behind modern LLMs.
2020s — Models cross hundreds of billions of parameters; scaling laws make compute the primary currency of AI progress.

The Road Ahead

Neural networks are evolving from static models into running systems — agentic networks that call tools, remember, and collaborate. Sparsity, mixture-of-experts, and neuromorphic hardware will keep pushing efficiency, but the paradigm of differentiable, layered, learnable computation is not going anywhere. It is the computational substrate of the field.

The Takeaway

Neural networks won because they turn representation learning into an engineering problem. Learn the architecture families — CNN, RNN, transformer, diffusion — and you can read any modern AI paper.