Foundations·Updated 18 Oct 2026

What is backpropagation?

Backpropagation is the algorithm that trains a neural network. It works out how much each weight contributed to an error, then nudges every weight to reduce it. The Automatic Differentiation in Machine Learning survey (Baydin et al., 2015) describes it in simplest terms as gradient descent in a network's weight space, searching for the minima of an objective function. Inputs are fed forward through the network to produce an output. That output is compared against a target, and the resulting error is sent backward through the same connections.

The backward pass uses the chain rule of calculus. For every weight, it computes the partial derivative of the error with respect to that weight. Those derivatives form a gradient, a vector pointing in the direction that most increases the error, so the training procedure moves weights the opposite way. Jürgen Schmidhuber's survey, Deep Learning in Neural Networks: An Overview (2014), frames learning as a credit assignment problem: working out which modifiable parts of a system caused success or failure, and what to change.

How does the chain rule make this work?

A network's output is a long composition of simple functions, one per layer or operation: a multiplication, an addition, a nonlinearity. The chain rule says the derivative of a composition is the product of the derivatives of its parts. Baydin et al. (2015) show this with a tiny worked example, a function built from a logarithm, a multiplication, and a sine. They track it through intermediate variables they call an evaluation trace, or Wengert list.

Backpropagation walks that trace backward. It starts from an output adjoint of one — the error's derivative with respect to itself — and multiplies through each operation's local derivative as it goes. Schmidhuber's survey gives the same idea as an explicit algorithm: an error signal at each event is computed from the errors of everything it fed into, multiplied by the local derivative of that event's activation function, then used to accumulate a weight change. Schmidhuber notes the cost of this backward pass is essentially the same as the cost of the original forward pass.

Backpropagation vs automatic differentiation

Backpropagation is, in the words of Baydin et al. (2015), a specialised counterpart of a broader technique called automatic differentiation, or autodiff. Autodiff is a family of methods for computing exact numerical derivatives of any function expressed as code, rather than as a closed-form formula. It comes in a forward mode, which propagates derivatives alongside the computation and is cheap when a function has few inputs and many outputs. Its reverse mode runs the function forward once, then propagates derivatives backward from the output.

Baydin et al. state that backpropagation is essentially equivalent to reverse-mode automatic differentiation applied to a network evaluation function composed with an objective function. Reverse mode is the efficient choice whenever there are far more inputs — weights — than outputs, which describes almost every neural network. Schmidhuber's survey traces explicit, efficient backpropagation in arbitrary networks to a 1970 master's thesis by Seppo Linnainmaa, independent of any neural network framing. The specific application to neural networks followed in 1981.

Why do gradients vanish or explode?

Backpropagation multiplies derivatives together, one per layer, as it works back through a deep network. Schmidhuber's survey calls this chaining the fundamental deep learning problem of vanishing or exploding gradients, which he dates to a 1991 analysis. If the per-layer factors are consistently smaller than one, their product shrinks toward zero as it passes through many layers. Early layers then receive almost no error signal and barely learn. If the factors are consistently larger than one, the product grows unchecked and training becomes unstable.

Schmidhuber frames the severity of this problem through what he calls the credit assignment path: the chain of causal links between an action and its effect. He defines problems with a path deeper than ten steps as requiring what he terms Very Deep Learning. A deep recurrent network stack from 1991, pre-trained with unsupervised learning, is cited in his survey as having solved benchmarks with credit assignment paths of depth 1,000 or more. Ordinary backpropagation alone had been unable to crack those benchmarks.

How does backpropagation compare with other ways to get derivatives?

Baydin et al. (2015) set reverse-mode automatic differentiation, the general technique behind backpropagation, against two older alternatives. Numerical differentiation approximates a derivative by evaluating a function at two nearby points and dividing the difference. It is easy to code but, the survey notes, costs one extra function evaluation per input dimension. It is also prone to round-off and truncation error, making it impractical for networks with millions of weights.

Symbolic differentiation manipulates expressions to produce an exact derivative formula, as computer algebra systems do. But the survey warns it suffers from expression swell: derivative expressions can grow exponentially larger than the original function. Automatic differentiation avoids both problems. It tracks numerical derivative values alongside a program's execution, giving the accuracy of symbolic methods at a small, constant overhead, and it works on code with loops and branches that symbolic methods cannot handle.

Where did backpropagation come from, and where does it stand now?

Schmidhuber's survey traces the mathematics behind backpropagation — steepest descent via the chain rule, treated as a form of dynamic programming — to papers from the early 1960s. It finds the first explicit, efficient backpropagation procedure for arbitrary networks in Linnainmaa's 1970 thesis. Werbos applied it specifically to neural networks in 1981. A 1986 paper by Rumelhart, Hinton and Williams popularised it by showing it produced useful internal representations in hidden layers.

As of Schmidhuber's 2014 survey, this same basic algorithm remained the central learning method for both feedforward and recurrent networks: iterate forward and backward passes until performance is good enough. Baydin et al. (2015) note that general-purpose automatic differentiation, the broader framework backpropagation belongs to, was only slowly being adopted in machine learning at that time. It went under names like dynamic computational graphs and differentiable programming. The algorithm is old; the framing around it keeps changing.

Key terms

Backpropagation: A specialised form of reverse-mode automatic differentiation that trains a neural network by propagating an error backward from the output to compute the gradient with respect to every weight.
Chain rule: The calculus rule stating that the derivative of a composition of functions is the product of the derivatives of its parts, used to combine per-operation derivatives into a full gradient.
Automatic differentiation (autodiff): A family of techniques for computing exact numerical derivatives of a function expressed as a computer program, more general than backpropagation, with forward and reverse modes.
Gradient: The vector of partial derivatives of an error with respect to each weight, used to decide which direction to adjust weights during training.
Vanishing/exploding gradients: The problem, described by Schmidhuber as the fundamental deep learning problem, where multiplying many per-layer derivatives together during backpropagation makes the resulting gradient shrink toward zero or grow without bound in deep networks.
Credit assignment path (CAP): Schmidhuber's term for a chain of possibly causal links between an action and its effect, whose depth measures whether a given learning problem is shallow or deep.
Reverse mode: The accumulation mode of automatic differentiation that runs a function forward once, then propagates derivatives backward from the output, which Baydin et al. identify as the mode equivalent to backpropagation.

Written from

Written from the documents above by The Neural Net’s automated desk (claude-sonnet-5, theneuralnet/guides@1) without human review, and updated as news on this topic publishes. How this works.

What is backpropagation?