A Mathematical Theory of Communication

✨ Summary

Summary

Claude E. Shannon develops a mathematical framework for communication systems by separating the engineering problem of transmitting and reproducing messages from the semantic meaning of those messages. He models communication as a system consisting of an information source, transmitter, channel, receiver, and destination, with noise represented as a stochastic disturbance of the transmitted signal.

The paper defines information using a logarithmic measure. For a probability distribution, Shannon derives the entropy function (H=-\sum_i p_i\log p_i), showing that it quantifies uncertainty and satisfies additivity and conditional-entropy relations. For stochastic sources, entropy becomes the average information produced per symbol or per unit time. The paper also introduces ergodic sources and uses asymptotic properties of long sequences to establish that typical messages occupy a substantially smaller effective set than all formally possible messages. This provides the theoretical basis for statistical data compression and source coding.

For noiseless discrete channels, channel capacity is defined as the asymptotic logarithmic growth rate of the number of permissible signals. Shannon proves that a source with entropy rate (H) can be encoded for transmission over a channel of capacity (C) at rates approaching (C/H) source symbols per second, while rates above this limit are impossible. This establishes the noiseless source-coding limit.

For noisy channels, the paper introduces equivocation, the uncertainty about the transmitted message after observing the received signal. The information transmission rate is expressed as mutual information, (R=H(x)-H_y(x)), and channel capacity is defined as the maximum mutual information over all possible input sources. Shannon proves the noisy-channel coding theorem: information can be transmitted with arbitrarily small error probability at any rate below channel capacity, whereas rates above capacity necessarily incur a nonzero error or equivocation.

The paper extends the framework to continuous signals and defines differential entropy, entropy power, and the rate required to reproduce a continuous source subject to a fidelity criterion. This leads to the general rate–distortion formulation. For a band-limited channel with average signal power (P), bandwidth (W), and additive white thermal noise of power (N), Shannon derives the capacity limit

[ C=W\log_2\left(1+\frac{P}{N}\right). ]

The paper therefore establishes the principal theoretical limits for data compression, reliable communication, error correction, and analog or continuous-signal transmission. Its central result is that redundancy can be used either to remove statistical structure for compression or to add structure for error correction, but reliable communication is fundamentally limited by channel capacity.

Influence

The paper established information theory as a distinct field and supplied the mathematical foundations for modern digital communication. IEEE historical accounts identify Shannon’s 1948 work as foundational to communication technology and the subsequent development of the IEEE Information Theory Society. (ieeexplore.ieee.org) Its source-coding and channel-capacity results became reference points for later work in data compression, coding theory, wireless communication, and digital networks. The capacity formula for additive white Gaussian noise remains a fundamental benchmark in communications research and engineering. (ieeexplore.ieee.org) IEEE accounts also document applications of Shannon’s ideas across digital systems, including storage, telecommunications, compact discs, and deep-space communications. (spectrum.ieee.org)