Multilayer perceptrons

What Are Multilayer perceptrons?

Multilayer perceptrons (MLPs) are feedforward artificial neural networks composed of an input layer, one or more hidden layers of nonlinearly activated neurons, and an output layer, with every neuron in one layer connected to every neuron in the next. Each connection carries a learnable weight, and each neuron applies an activation function, historically a sigmoid or hyperbolic tangent, to the weighted sum of its inputs. By stacking multiple hidden layers, an MLP can represent functions of arbitrary complexity, making it the foundational architecture for deep learning before the emergence of more specialized networks such as convolutional and recurrent architectures.

The single-layer perceptron, introduced by Frank Rosenblatt in 1958, could only classify linearly separable patterns, a limitation that constrained its utility for many practical problems. The multilayer extension, combined with the backpropagation training algorithm rediscovered and popularized by Rumelhart, Hinton, and Williams in their 1986 Nature paper, resolved that limitation and established the theoretical and practical basis for modern connectionist learning.

Architecture and Activation Functions

An MLP with L hidden layers and N_l neurons per layer defines a composition of affine transformations interleaved with pointwise nonlinearities. The choice of activation function determines the gradient behavior during training and the class of functions the network can represent. Sigmoid and tanh activations were standard through the 1990s and 2000s, but they suffer from gradient saturation at large positive or negative inputs, slowing convergence. The rectified linear unit (ReLU), f(x) = max(0, x), avoids saturation in the positive regime and has become the default choice for deep MLPs because it maintains near-constant gradient magnitude for active neurons. Variants including leaky ReLU and GELU are used in contexts where the hard zero for negative inputs is undesirable.

Universal Approximation and Expressivity

A theoretical cornerstone of MLP design is the universal approximation theorem, established independently by Hornik, Stinchcombe, and White (1989) and by Cybenko in the same year. The theorem guarantees that a single-hidden-layer network with enough neurons can approximate any continuous function on a compact domain to arbitrary precision, provided the activation function is non-polynomial. This is an existence result: it says such a network exists but does not specify how to find it or how many neurons are required. An IEEE conference paper on universal approximation with non-sigmoid activations extended the result to a broader class of activation functions, confirming that the phenomenon is not tied to any particular nonlinearity.

Training via Backpropagation

Backpropagation is the algorithm that makes MLPs trainable in practice. It applies the chain rule of calculus to compute the gradient of a loss function with respect to every weight in the network in a single backward pass through the computation graph, at a cost proportional to the forward pass. Stochastic gradient descent (SGD) and its momentum-based variants then use these gradients to update weights iteratively. MIT's Foundations of Computer Vision textbook presents backpropagation as a general-purpose tool for differentiating computation graphs, not merely a neural-network training procedure. Practical training of deep MLPs also requires techniques such as batch normalization, dropout regularization, and careful weight initialization to avoid vanishing or exploding gradients in deeper architectures.

Applications

Multilayer perceptrons have applications in a wide range of fields, including: