What Is Information Entropy?

Information entropy is a quantitative measure of the uncertainty or average information content associated with a probability distribution over possible messages or symbols. Introduced by Claude Shannon in his landmark 1948 paper "A Mathematical Theory of Communication," it provides the theoretical foundation for data compression, channel capacity, and source coding. Shannon borrowed the term entropy from thermodynamics, though the two quantities are related formally rather than physically.

The concept answers a precise question: given a source that produces symbols according to known probabilities, what is the minimum average number of bits required to represent each symbol? That minimum is the entropy. A source that always produces the same symbol has zero entropy and requires no bits to encode. A source that produces two equally likely symbols has an entropy of one bit per symbol, the maximum for a binary source.

Shannon's Entropy Formula

Shannon entropy H is defined as the negative sum, over all possible outcomes, of each outcome's probability multiplied by the logarithm of that probability. When the logarithm is base 2, H is measured in bits; base e gives nats; base 10 gives hartleys. For a discrete source with n possible symbols, entropy reaches its maximum when all symbols are equally probable, a result known as the maximum-entropy principle. The arxiv tutorial on Shannon's entropy metric provides a careful derivation of the formula and its axiomatic justification, showing why it is the unique function satisfying continuity, symmetry, and recursion requirements. Joint entropy, conditional entropy, and mutual information are all derived from the same foundation, forming a consistent algebra for analyzing information flow through systems.

Relationship to Source Coding

Shannon's source coding theorem, also called the noiseless coding theorem, proves that a source with entropy H bits per symbol cannot be compressed below H bits per symbol on average without loss of information. This bound is achievable: Huffman coding and arithmetic coding are practical algorithms that approach the entropy limit for known distributions. For sources where the distribution is unknown or nonstationary, universal compression algorithms such as Lempel-Ziv-Welch use empirical frequency statistics to estimate entropy and adapt the coding accordingly. The gap between a compressed file's actual size and the theoretical entropy of its source is a widely used measure of coding efficiency. A detailed treatment of these connections appears in the introduction to information theory by Stone, which covers the relationship between entropy, compression, and channel capacity.

Relationship to Thermodynamic Entropy

Shannon's entropy formula is mathematically identical to the Boltzmann-Gibbs entropy used in statistical mechanics, a coincidence Shannon found remarkable and von Neumann reportedly suggested exploiting. Both quantify the dispersion of a probability distribution: in thermodynamics, that distribution is over microstates; in information theory, it is over messages. The two are not interchangeable without care, but the formal connection has generated productive cross-pollination. In physical information theory, work on entropy and information in dynamical systems explores the boundary between the two interpretations, showing that information entropy can characterize whether a system's evolution is regular, chaotic, or purely random.

Applications

Information entropy has applications in a wide range of fields, including:

• Data compression, where entropy bounds establish the minimum bitrate for lossless coding
• Cryptography and random number generation, where high entropy measures unpredictability of keys and sequences
• Machine learning, where cross-entropy and information gain guide decision tree construction and model training
• Communications engineering, measuring channel capacity and quantifying signal uncertainty in transmission
• Bioinformatics, analyzing sequence complexity and mutual information between positions in aligned genetic sequences