What is a neural network?
The basic building block of deep learning. How many layers do you actually need?
What is a neural network?
A neural network is a system of simple, connected processing units called neurons. Each neuron produces a sequence of real-valued outputs, or activations, from the inputs it receives. The Schmidhuber survey (2014) describes a standard neural network this way: input neurons are activated by sensors reading the environment. Every other neuron is activated through weighted connections from neurons that fired before it.
Some neurons feed back into the environment by triggering actions, such as steering a car. Learning, in this framework, means finding the weights — the numbers attached to each connection — that make the network behave as desired. Schmidhuber calls this the credit assignment problem: which of the network's many adjustable parts deserves credit, or blame, for the outcome, and how should it change.
What are neurons, layers and weights?
In the notation set out by Schmidhuber (2014), a network at any moment is a finite set of units, or neurons, linked by directed connections. Feedforward networks arrange these connections so they never loop. The first layer takes in input, and the k-th layer is the set of units reachable by a path of exactly k-1 connections from an input unit. Recurrent networks allow cycles, so a unit's output can eventually feed back into itself over time.
Each connection carries a weight, a real-valued parameter that scales the signal passing along it. A neuron typically sums its weighted inputs and passes the result through a nonlinear activation function. Schmidhuber's survey names the hyperbolic tangent, and notes that many contest-winning networks instead use a maximum, or max-pooling, operation across inputs. Networks can also reuse the same weight in many places at once, a trick called weight sharing. It cuts the number of parameters a network needs to learn, and it is used in both recurrent and convolutional networks.
How does depth help?
Depth refers to how many successive stages of transformation separate an input from an output. Schmidhuber (2014) measures it along a credit assignment path: a chain of causal links between an action and its effect. A network with a single layer of adjustable weights has a maximal path depth of one. Stacking more layers lengthens that chain. Schmidhuber proposes, for the purposes of his overview, that problems needing a path depth greater than ten count as 'very deep learning'.
Greater depth lets a network build hierarchical representations, where each layer transforms the output of the one below it, often in a nonlinear way. Complex behaviour can emerge from simple repeated stages. But depth is not a measure of difficulty on its own. Schmidhuber notes that some deep problems can be solved quickly through random weight guessing or other search methods. Finding an exact fit to a training set, by contrast, is in general computationally hard — an NP-complete problem — regardless of depth.
How is a network trained?
Training usually means gradient descent: nudging every weight in the direction that reduces a measured error between the network's output and a target value. The weights are adjusted using backpropagation, an efficient method for calculating how much each weight contributed to the error. Schmidhuber's survey traces backpropagation's mathematical roots to work on gradient descent in the 1960s. Its first explicit neural-network application came in 1981, and a 1986 paper by Rumelhart and colleagues popularised it by showing hidden layers could learn useful internal representations.
The survey by Baydin, Pearlmutter, Radul and Siskind (2015) frames backpropagation as one instance of a broader family called automatic differentiation, or autodiff. Autodiff computes exact derivatives of a program step by step, using the chain rule. It avoids both the expression bloat of symbolic differentiation and the rounding errors of numerical approximation. Its reverse mode, the authors explain, is what makes backpropagation efficient: one backward pass through the network yields the gradient with respect to every weight at once, however many there are.
A short history of deep learning
Schmidhuber's survey (2014) traces shallow learning machines back to the linear regression methods of the early 1800s, and unlearned architectures to the 1943 McCulloch-Pitts model. It identifies the Group Method of Data Handling networks of 1965 — some reaching eight layers by 1971 — as perhaps the first deep, feedforward learning systems. The Neocognitron of 1979 introduced convolutional structure, with weight replication and subsampling, years before backpropagation was first applied to a neural network in 1981.
Deep networks proved hard to train through the late 1980s, and the difficulty became an explicit research topic by the early 1990s. Schmidhuber's survey dates the discovery of the vanishing-gradient problem to 1991. Unsupervised pre-training and architectures such as Long Short-Term Memory, introduced in 1995, helped networks learn across path depths of a thousand steps or more. Schmidhuber reports that supervised deep networks began winning official international pattern recognition contests from 2009 onward. They reached the first superhuman results on a limited visual task by 2011 — gains he attributes partly to GPUs, which he says could speed up training by a factor of fifty or more.
Key terms
- Neuron
- A simple processing unit in a neural network that produces a real-valued activation from its weighted inputs.
- Weight
- A real-valued, adjustable parameter attached to a connection between two neurons, found by learning.
- Layer
- The set of neurons reachable by paths of the same length from the network's input, in a feedforward network.
- Backpropagation
- An efficient method, dating to the 1960s-70s and first applied to neural networks in 1981, for computing how each weight should change to reduce error.
- Credit assignment path
- A chain of possibly causal links between an action and its effect, used to measure whether learning in a network is deep or shallow.
- Automatic differentiation
- A family of techniques, more general than backpropagation, for computing exact numerical derivatives of a function expressed as a computer program.
- Recurrent neural network
- A network whose connections form cycles, allowing it to process and retain memories of sequences of inputs over time.
- Convolutional neural network
- A network, first introduced as the Neocognitron in 1979, in which a filter's weights are replicated across an input such as an image.
Written from
Written from the documents above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/guides@1) without human review, and updated as news on this topic publishes. How this works.