Hartley had a deep insight into

In 1871, James Clerk Maxwell showed how an intelligent being could in principle sort out the disorder in a gas of randomly moving molecules, by gathering knowledge-intelligence-information about their speeds and sorting them into hot and cold gases, in apparent violation of the second law of thermodynamics. William Thomson (Lord Kelvin) called this being "Maxwell's intelligent demon."

Ludwig Boltzmann, who established the statistical physics foundation of thermodynamics, chose the logarithm of the number W of equiprobable microstates as the measure for his entropy, because he wanted entropy to be an extensive additive quantity.

where k is Boltzmann's constant. If one system can be in one thousand possible states and another system also in a thousand possible states, the combined system has a million possible states. In a base 10 system, log₁₀1000 = 3, and 3 + 3 is 6 = log₁₀1000000.

In 1929, Leo Szilard imagined a gas with but a single molecule in a container. He then devised a mechanism that could behave like Maxwell's demon. It would insert a partition into the middle of the container, then gather the information about which of the two sides of the partition the molecule was in. This was a binary decision and it allowed Szilard to develop the mathematical form for the amount of entropy S produced by a one-bit measurement, which Szilard identified as the acquisition of information and storage in the "memory" of a physical device or of a human observer.

The base-2 logarithm reflects the binary decision.

The amount of entropy generated by the measurement may, of course, always be greater than this fundamental amount of negative entropy (information) created, but not smaller, or the second law - that overall entropy must increase - would be violated.

The earlier work of Maxwell, Boltzmann, and Szilard did not figure directly in Shannon's work. Shannon studied the design of early analog computers (specifically Vannevar Bush's differential analyzer at MIT, which was used by Coolidge and James to calculate the first wave functions of the hydrogen molecule in 1936). Then, with John von Neumann and Alan Turing, Shannon helped design the first digital computers, based on the Boolean logic of 1's and 0's and binary arithmetic.

Shannon analyzed telephone switching circuits that used electromagnetic relay switches, then realized that the switches could solve some problems in Boolean algebra.

During World War II, Shannon worked at Bell Labs on cryptography and sending "intelligence" signals in the presence of noise. Alan Turing visited the labs for a couple of months and showed Shannon his 1936 ideas for a universal computer (the "Turing Machine").

Shannon's work on communications, control systems, and cryptography were initially classified, but they contained almost all of the mathematics that eventually appeared in his landmark 1948 article "A Mathematical Theory of Communication," that is considered the basis for modern information theory.

Norbert Wiener's work on probability theory in Cybernetics had an important influence on Shannon. There can be no new information in a world of certainty. Probability and statistics are at the heart of both information theory and quantum theory.

Following a suggestion by John von Neumann, Shannon developed his expression for an information (Shannon) entropy, which he showed has the same mathematical form as thermodynamic (Boltzmann) entropy. He wrote:

Suppose we have a set of possible events whose probabilities of occurrence are p₁, p₂, • • • , p_n. These probabilities are known but that is all we know concerning which event will occur. Can we find a measure of how much "choice" is involved in the selection of the event or of how uncertain we are of the outcome?

If there is such a measure, say H(p₁, p₂, • • • , p_n), it is reasonable to require of it the following properties:

1. H should be continuous in the p_n. 2. If all the p_n are equal, p_i = 1/n, then H should be a monotonic increasing function of n. With equally likely events there is more choice, or uncertainty, when there are more possible events.

3. If a choice be broken down into two successive choices, the original H should be the weighted sum of the individual values of H. The meaning of this is illustrated in Fig. 6.

Fig. 6.— Decomposition of a choice from three possibilities.

At the left we have three possibilities p₁ = 1/2, p₂ = 1/3, p₃ = 1/6. On the right we first choose between two possibilities each with probability 1/2, and if the second occurs make another choice with probabilities 2/3, 1/3. The final results have the same probabilities as before. We require, in this special case, that

H(1/2, 1/3, 1/6) = H(1/2, 1/2) + 1/2 H(2/3, 1/3)

The coefficient 1/2 is the weighting factor introduced because this second choice only occurs half the time.

The only H satisfying the three above assumptions is of the form:

H = K Σ p_i log p_i

where K is a positive constant.

Quantities, of the form H = Σ p_i log p_i (the constant K merely amounts to a choice of a unit of measure) play a central role in information theory as measures of information, choice and uncertainty. The form of H will be recognized as that of entropy as defined in certain formulations of statistical mechanics where p_i is the probability of a system being in cell i of its phase space.

H is then, for example, the H in Boltzmann's famous H theorem. We shall call H = p_i log p_i the entropy of the set of probabilities p₁, p₂, • • • , p_n.

(The Mathematical Theory of Communication, pp.48-50)

Boltzmann entropy: S = k ∑ p_i ln p_i.        Shannon information: I = - ∑ p_i ln p_i.

Shannon entropy is the average (expected) value of the information contained in a received message. If there are many possible messages, we get a lot more information than when there are only two possibilities (one bit of information). It is the base 2 logarithm of the number of possibilities. Shannon entropy thus characterizes our uncertainty about the information in an incoming message, and increases for more possibilities with greater randomness. The less likely an event is, the more information it provides when it occurs. Shannon defined his entropy or information as the negative of the logarithm of the probability distribution. He briefly considered calling it "uncertainty (after the Heisenberg uncertainty principle). but decided on "entropy" following the suggestion by von Neumann. One bit of information is also known as one "shannon."

Boltzmann entropy is maximized when the particle distribution is maximally random among positions in phase space, when the number of microstates W corresponding to a given macrostate is as large as possible. An improbable macrostate might be when every particle is in the same microstate. Finding all the particles in a corner of the possible volume is information in the same sense as receiving one of the possible messages. An equilibrium macrostate is when particles are as randomly distributed as possible. Any information is gone.

Counterintuitively, maximum Boltzmann entropy (no information) is maximal uncertainty before a message is received and then maximal Shannon entropy (information), after a message is received, making the two entropies hard to compare.

In 1877, Boltzmann proved his “H-Theorem” that the entropy or disorder in the universe always increases. He defined entropy S as the logarithm of the number W of possible states of a physical system, an equation now known as Boltzmann’s Principle,

S = k log W.

In 1929, Leo Szilard showed the mean value of the quantity of information produced by a 1-bit, two-possibility (“yes/no”) measurement as S = k log 2, where k is Boltzmann’s constant, connecting information directly to entropy.

Following Szilard, Ludwig von Bertalanffy, Erwin Schrödinger, Norbert Wiener, Claude Shannon, Warren Weaver, John von Neumann, and Leon Brillouin, all expressed similar views on the connection between physical entropy and abstract “bits” of information.

Schrödinger said the information in a living organism is the result of “feeding on negative entropy” from the sun. Wiener said “The quantity we define as amount of information is the negative of the quantity usually defined as entropy in similar situations.”

Brillouin created the term “negentropy” because he said, “One of the most interesting parts in Wiener’s Cybernetics is the discussion on “Time series, information, and communication,” in which he specifies that a certain “amount of information is the negative of the quantity usually defined as entropy in similar situations.”

Shannon, with a nudge from von Neumann, used the term entropy to describe his estimate of the amount of information that can be communicated over a channel, because his mathematical theory of the communication of information produced a mathematical formula identical to Boltzmann’s equation for entropy, except for a minus sign (the negative in negative entropy).

Shannon described a set of i messages, each with probability p_i. He then defined a quantity H,

H = k Σ p_i log p_i,

where k is a positive constant. Since H looked like the H in Boltzmann’s H-Theorem, Shannon called it the entropy of the set of probabilities p₁, p₂, . . . , p_n.

To see the connection between the two entropies, we can note that Boltzmann assumed that all his probabilities were equal. For n equal states, the probability of each state is p = 1/n.

The sum over n states, Σ p_i log p_i, is then n x 1/n x log (1/n) = log (1/n) = - log n.

If we set Shannon's number of possible messages n equal to Boltzmann's number of possible microstates W, we get Boltzmann’s entropy with a minus sign,

H = - k log W.

Shannon’s entropy H is the negative of Boltzmann’s S.

Shannon showed that a communication that is certain to tell you something you already know (one of the messages has probability unity) contains no new information.