How distinguishable states become symbols through an agreement, how halving yes-or-no questions count information in bits, and what a bigger alphabet gains in bits per symbol and costs in safety.
Using a light link with several colours, this lesson shows you how light becomes a symbol and how much information it carries, and afterwards you can:
explain why a symbol is an agreed and distinguishable state, and why an error-free transmission without a shared table still delivers nonsense
argue why halving yes-or-no questions always find one of 32 cards in five questions and on average deliver more information than a lopsided question
calculate the uncertainty with H = log₂(N) and the information of an answer with I = H₁ − H₂, both in bits, and check them with math.log2()
work out how many bits a symbol from N equally likely colours carries, how this changes the transmission time of a file, and why two symbols in groups are still enough for any message
explain why Morse code gives the frequent letter e the shortest code
weigh why more colours bring more bits per symbol but move closer together, so that only a measurement series can settle the size of your own alphabet
North Atlantic, 1942: radio traffic would give the convoy away, so the ships
keep radio silence and talk to each other anyway, with a lamp and a shutter in
front of it, short, short, short, long. The light itself knows nothing about
letters, and it means something only because both ships carry the same table.
This lesson is about the two halves of that situation: what turns a state into
a symbol, and how much a symbol can tell you. The running example is a link
that transmits data with light, where an LED sends colours and a sensor tells
them apart.
Distinguishable and agreed
A symbol is an agreed, distinguishable state, and both words carry weight.
Distinguishable is a question of physics and of the receiver: can it tell this
state apart from all the others, even with noise on the way? That question was
the subject of Analog and digital. Agreed is a question of a different
kind, which no measurement can answer, since it asks what the state stands
for, and only the people on both ends can settle that.
A raised thumb shows both halves. Nobody mistakes it for a fist or an open
hand, so it is easy to distinguish, but the thumb itself holds no meaning: in
much of Europe it says "good", in parts of West Africa and the Middle East it
is an insult, and someone from Germany who counts on their fingers raises the
thumb for "one", where someone from the United States raises the index finger.
The set of all symbols that sender and receiver have agreed on is their
alphabet. The 26 letters from A to Z are one alphabet, but four colours of
a lamp, two tones or a flag that is either raised or lowered are alphabets just
as well.
Perfect reception, perfect nonsense
What happens when the agreement is missing is easy to show. The sender spells
team with four colours, red for t, green for e, blue for a and yellow for m,
and all four colours arrive without a single error. The receiver's table,
however, swaps red and yellow:
Both tables map colours to letters, but not the same way: four colours arrive without a single error, and the receiver reads a different word.
The program decodes the same four colours with both tables:
There was no transmission error, no interference and no bad measurement, and
the message is still wrong. Worse, team turned into meat, another real
word, so nobody notices that anything went wrong. Change one entry in
receiver_table and run the program again: as long as the two tables differ,
the signal can be perfect and the meaning still wrong.
In code, the agreement is a dictionary like sender_table. It is never
transmitted; it has to sit on both sides beforehand, which makes it not data
but a promise, and whoever changes it without telling the partner builds the
nonsense above. The same dictionary can just as well map colours to bit
strings, {"red": "00", "green": "01", "blue": "10", "yellow": "11"}, and how
many digits such a string needs is the subject of the next part.
The guessing game, measured
What a message is worth does not depend on its length but on how much
uncertainty it removes. Someone draws one card from a deck of 32, and you may
only ask questions that can be answered with yes or no. Ask well, for example
"is it red?", then "is it a heart?", and you are done after five questions:
Every question halves what is left, so five questions are enough, and not by luck.
That number of necessary halvings is the measure this part is about, and its
unit is the Bit: a question that halves the field is worth one bit.
You can watch the halving at work: think of a number from 1 to 32 and let the program find it,
answering each question with y or n:
low, high = 1, 32
questions = 0while low < high:
middle = (low + high) // 2
answer = input(f"Is your number greater than {middle}? (y/n) ")
questions += 1if answer == "y":
low = middle + 1else:
high = middle
print(f"Your number is {low}, found with {questions} questions.")
Whatever number you choose, the program needs exactly five questions, because
every answer removes half of what is left.
Not every answer is worth the same
A lopsided question buys less. "Is it the ace of spades?" can settle
everything at once, but it does so in only one of 32 cases, and in the other
31 almost everything is still left. The halving question, on the other hand,
always delivers exactly one bit, whatever the answer. A rare answer tells
you a lot, an expected answer tells you little, and since you cannot choose
which one you get, what counts is the average:
Question
Chance of yes
Yes is worth
No is worth
On average
"is it red?"
1/2
1 bit
1 bit
1 bit
"is it the ace of spades?"
1/32
5 bits
0.05 bits
0.2 bits
In general, every possible answer contributes how likely it is, multiplied by
what it would be worth, and added up over all answers that is the expected
value of a question:
E = Σ pᵢ · log₂(1 / pᵢ)
Here pᵢ is the probability of an answer and log₂(1 / pᵢ) is what that answer
is worth in bits: the less likely it is, the more it carries. For a yes-or-no question
that is two terms, and the function worth() puts the numbers in:
Worked out by hand, the halving question gives ½ · 1 + ½ · 1 = 1 bit and the
question about the ace of spades 1/32 · 5 + 31/32 · 0.05 ≈ 0.16 + 0.05 ≈ 0.2
bits. A whole bit goes only to the question that really splits the field in
half, which is why halving wins on average: it takes the largest guaranteed
gain out of every answer, and the lopsided question pays off only at the very
end, when just two possibilities are left. The second row also shows that a
question whose outcome is nearly certain carries almost no information,
however long its answer, and compression rests on exactly that observation.
In the question game here you measure this yourself on sixteen faces: after
every question it shows how many bits the answer gained and what the question
was worth on average.
How to use: Tap a question to ask it, or a face to guess; a guess counts
as a question too. The game does not show how many faces would answer yes,
since counting them is your strategy. After every answer, the panel below the
questions shows how many bits it gained and what the question was worth on
average, so you can tell a well planned question from a lucky one.
One bit, and two formulas
Put generally, a bit is the amount of information in a decision between two
equally likely possibilities. A light switch holds one
bit, and it needs no computer for that. With it, both quantities can be
written down. The uncertainty of N equally likely possibilities is
H = log₂(N) bits
and measures how many halvings are still missing. Information is a
difference, the uncertainty before minus the uncertainty after,
I = H₁ − H₂
and measures what an answer has actually removed. 32 cards mean an
uncertainty of 5 bits, and each halving question removes 1 bit; the 32 is the
space of possibilities, while the uncertainty is the 5. Python calculates the
logarithm with math.log2():
import math
for n in [2, 4, 32, 26]:
print(n, "possibilities:", round(math.log2(n), 2), "bits")
Two possibilities give one bit, four give two and 32 give five. For 26 letters
the result is 4.7 bits, which is not a whole number of questions: a code made
of yes/no symbols needs five of them per letter, since four give only
2⁴ = 16 combinations and five give 32, the same kⁿ as in
Place value systems.
What a bigger alphabet buys
The same tool tells you what a symbol is worth. Each symbol from an alphabet
of N equally likely colours carries log₂(N) bits, so two colours carry one
bit per symbol, four carry two, eight carry three and sixteen carry four.
Because the same file then needs fewer symbols, it gets through sooner:
The same file at the same symbol rate: only the alphabet changes, and with it the time the file needs.
With four colours the file needs half the time it needs with two, with eight
a third and with sixteen a quarter, while the file and the symbol rate stay
the same.
At the other end, two symbols are enough for any message at all; they only
need more room. Five flashes with two possibilities each give 2⁵ = 32
combinations, enough for a whole alphabet. What a large alphabet achieves per
symbol, two symbols achieve in a group, and that is why every computer gets
by with 0 and 1 (Place value systems): a small alphabet is not a dead end but
merely slower.
When symbols are not equally likely
Everything so far assumed that all symbols are equally likely. In real text
they are not, and Morse code makes use of that: the most frequent letter
in English, e, gets the shortest code, a single dot, t gets a single dash, and
a rare letter such as q takes four signals, − − · −.
The unequal treatment pays off because it shortens the average message: what
occurs often should cost little, and compression methods still shrink files
on the same principle today. A rare symbol carries more information than a
common one, so it may also take up more room. Codes of unequal length have a
price of their own, though, since the receiver has to find out where one code
word ends; Morse code uses pauses for that, and a prefix-free code gets by
without them.
Your alphabet
So why not use 256 colours? More colours bring more bits per symbol, so the
same file needs fewer symbols and arrives sooner, but the colours also move
closer together, the noise reaches across the gap, and symbols are read
wrong, as Analog and digital and Signal and noise show. No formula
tells you where the limit lies, so the size of your alphabet is not a matter
of taste but the result of a measurement. Challenge 1: The alphabet asks exactly
this, and four steps lead to the answer:
agree on your alphabet and write it down
measure a profile for every symbol
teach the receiver to recognise every symbol
send 20 random symbols and count the errors
The first step is the one everyone wants to skip, and whoever keeps the
agreement only in their head builds the nonsense from the first part as soon
as two people work in parallel. Light means nothing until you agree what it
means, and how much it means, you can count.
Takeaways
A symbol is an agreed, distinguishable state: physics decides whether it can
be told apart, and only an agreement decides what it means.
The agreement, such as a dictionary of colours and letters, has to exist on
both sides beforehand; if it differs, perfect reception gives perfect
nonsense.
A question that halves the field is worth one bit; a lopsided question is
worth much less on average, even if it might settle everything.
With N equally likely possibilities the uncertainty is H = log₂(N) bits,
and the information of an answer is I = H₁ − H₂.
A symbol from N equally likely symbols carries log₂(N) bits, so larger
alphabets are faster, and two symbols in groups are enough for any message.
Frequent symbols deserve short codes, which is why Morse code gives the e a
single dot.
More colours move closer together, so only a measurement series settles the
size of your own alphabet.
What is a symbol in the sense of information theory?
A small picture, such as an icon on a screen.
That is the everyday meaning. Here, symbol means something more general that can also be a colour, a tone or a hand gesture.
A written character such as a letter or a digit, that is, something you can write down.
Written characters are one example of symbols, but only one. The colours of a lamp are symbols just as much, although none of them can be written down.
An agreed, distinguishable state.
Exactly, and both words matter: distinguishable, since the receiver must reliably tell it apart from all others, and agreed, since without an agreement the state carries no meaning.
A single sensor reading, such as a particular brightness.
A reading is just a number at first. A range of readings only becomes a symbol once sender and receiver jointly give it a meaning.
A symbol needs both: physics makes it distinguishable, and an agreement gives it a meaning.
Two teams send data with coloured light, and their LED shows four colours they have agreed on. What is the alphabet of their link?
The 26 letters from A to Z that the link is meant to carry.
The letters are what the messages say, but the link itself only sends colours. Its alphabet is what can be seen on the way, four colours here instead of 26 letters.
The set of the four colours that both teams have agreed on as symbols.
Exactly. How large this set may be is not decided by the agreement alone: the hardware and the surroundings have a say, which makes it a question of measurement.
The order in which the colours of a message are sent.
The order is the message itself. The alphabet is the stock from which it is assembled.
The mapping of the colours to bit sequences, such as green standing for 01.
That is the code, which builds on the alphabet. The alphabet is only the agreed set of colours themselves; how to write them in bits is a second step.
The alphabet is the stock of agreed symbols, and a code then says what each of them stands for.
A link sends letters as coloured flashes of light. Who decides that a green flash means "A"?
The physics of light: each colour has a fixed wavelength, and from that the receiver recognises which character is meant.
The wavelength makes green distinguishable, nothing more. Which character stands behind it is not in the light; it arises from agreement alone.
Sender and receiver together, by an agreement made before the transmission.
Exactly, and both sides must use exactly the same mapping. In practice this agreement is a piece of documentation and two places in the code, one in the sender and one in the receiver.
A standard for light transmission that all devices follow so that they understand each other.
Standards exist for many channels, but even they are agreed, not discovered: a standard is an agreement written down for many parties. Nothing in the light itself fixes the meaning.
The receiver alone, since it is the side that has to turn readings into characters.
A receiver deciding alone is guessing. Unless the sender uses the same mapping, the result is reliably nonsense.
Meaning is not in the signal but in an agreement that both sides made beforehand.
What is a bit?
A tiny component inside a computer that either passes or blocks current.
A bit is not a thing but a unit of measurement. It can be stored in components, but just as well in a flash of light or a hand gesture.
The amount of storage a computer needs for a single character such as "A".
That describes a byte, a group of eight bits. The mix-up is common but immediately costs a factor of eight.
The amount of information in a decision between two equally likely possibilities.
Exactly: one yes or no, one on or off. The bit is the smallest currency in which information can be counted, and every larger amount is measured in it.
The smallest brightness a sensor can still measure.
That would be a property of a particular sensor. A bit is independent of any hardware; it counts possibilities, not brightness.
One bit is the answer to one yes/no question whose two answers are equally likely.
An alphabet has four colours, all equally likely. How much information does a single received symbol carry, and why?
4 bits, because each of the four colours is a possibility of its own, and each possibility counts as one bit.
Bits do not count the possibilities themselves but the yes/no decisions needed to pin one down. That is considerably fewer: two for four possibilities.
2 bits, because two yes/no decisions suffice to determine one of four colours.
Exactly, and the reasoning is halving: the first question splits the four into two halves, the second hits the colour. Every doubling of the alphabet adds exactly one bit per symbol.
1 bit, because the receiver makes one decision per symbol: it has arrived or it has not.
Whether something arrives is not the question; which of the four colours it is, is. A symbol is not automatically one bit: how much it carries depends on how many possibilities it was chosen from.
It depends on the colour sent, because an easily recognised colour such as red carries more than a hard one such as yellow.
Recognisability affects reliability, not amount. With four equally likely colours, each carries the same information: what counts is the number of possibilities, not the actual choice.
With N equally likely symbols, each carries log₂(N) bits, so four colours carry two.
A link switches from 2 to 8 colours and still sends the same number of symbols per second. How does the transfer time of the same file change?
It drops to a quarter, because 8 divided by 2 is 4.
You do not divide the numbers of colours: information grows with doubling. From 2 to 8 is two more doublings, from 1 to 3 bits per symbol.
It drops to a third: each symbol now carries 3 bits instead of 1.
Exactly: 2 colours carry 1 bit per symbol, 8 colours carry 3. Same symbol rate, three times the bit rate, a third of the time, provided the eight colours stay distinguishable, which is the limit that the distance between regions sets.
It stays the same, since the symbol rate has not changed.
The symbol rate is the same, but each symbol now carries more. Time is saved by packing more bits into each symbol, not only by flashing faster.
It drops to an eighth, one eighth per colour.
That would require every additional colour to contribute equally. Information grows by one bit only per doubling of the alphabet, not per single colour.
Time follows bits per symbol, log₂(2) = 1 against log₂(8) = 3, so a third.
More symbols per alphabet means more bits per symbol. Why not simply make the alphabet huge, say 256 colours?
Because a lamp can mix only a limited number of distinct colours from its three channels.
An RGB LED mixes millions of shades from three channels, so sending is not the problem. The bottleneck sits on the other side.
Because the receiver would need a lookup table for 256 colours whose search per symbol takes too long.
256 entries are nothing for a computer; a single byte has that many values, and the lookup takes a fraction of a microsecond. The limit is not computational.
Because the receiver must still tell the symbols apart reliably: the more there are, the closer their measurement regions crowd together.
Exactly, and this is where two ideas interlock: the measure of information says what a larger alphabet buys, and the distance between regions says what it costs. Where the balance lies is decided by noise, ambient light and measurement.
Because an alphabet needs only as many symbols as the text has distinct characters; further colours would go unused.
Character set and alphabet are not tied like that: more colours per symbol means fewer symbols per character, hence faster. Nothing would go unused; the limit lies with the receiver, not the text.
A larger alphabet carries more per symbol until noise makes neighbouring symbols indistinguishable.
A receiver recognises every colour without error; recognition accuracy is 100 percent. Yet the resulting text is nonsense. What is the most likely cause?
The sensor delivers distorted readings, for instance because the gain is set too high.
Distorted readings would already spoil the recognition of the colours. But that is flawless, so the fault lies behind it.
Sender and receiver use different mappings from colours to characters.
Exactly: the signal arrives perfectly, but the agreement is broken. Meaning is not in the light but in the mapping, which must be identical on both sides. A classic after sender and receiver code were written separately.
Ambient light shifts the readings, so the receiver sees the colours but sorts them into the wrong regions.
Colours sorted into the wrong regions would be recognition errors, and those would not be at zero. Recognition here is flawless; the fault lies behind it.
The alphabet has fewer colours than the text has distinct characters, so several characters end up on the same colour.
Nobody maps it like that. A small alphabet maps characters to groups of symbols, which makes the transmission slower, not wrong.
Perfect reception with different tables gives perfect nonsense.
An alphabet has 16 symbols. At most how many yes/no questions are needed to determine a received symbol with certainty?
4, because halving four times leads from 16 possibilities to one.
Exactly: 16, 8, 4, 2, 1. The same calculation as guessing a card by halving, and at the same time the statement that a symbol out of 16 carries 4 bits.
8, because each question halves, and half of 16 possibilities is 8.
Halving does not mean halving the count once but counting how often you can halve. For 16 that is only four steps.
15, because in the worst case you have to rule out 15 symbols one by one.
That is asking one by one. It works, but it is the slow procedure; with clever questions, half of the candidates drop out with every answer.
2, because every question has two answers, and two times two answers cover 16.
Two questions separate only four possibilities, two times two. For 16 you need four doublings.
The number of halvings is log₂(16) = 4, and that is also the number of bits.
Which statements about bits are true?
One bit distinguishes exactly two possibilities.
Exactly, that is the definition: on or off, yes or no, bright or dark.
A bit and a byte denote the same amount, just in different contexts.
A byte is eight bits. Confusing the two throws every size estimate off by a factor of eight.
Any information can be represented as a sequence of bits.
Correct, and that is why one and the same technology can carry text, pictures and sound. How the mapping works in detail is a question of codes.
Bits exist only in computers, because only there are states stored electrically.
A light switch, a coin toss, a raised flag: each is one bit, no computer involved. Bits count possibilities, whatever they live in.
A message contains more bits if its content is more important.
Information measures possibilities, not importance. A "yes" at the altar and a "yes" at the deli counter carry the same single bit.
Bits count possibilities, independent of hardware and of importance.
An alphabet has four colours, so each symbol carries 2 bits. A 12-bit identifier is to be transmitted. How many symbols does that take at least?
12, one symbol per bit.
That would be the case with only two colours. Four colours pack two bits into each symbol, which halves the length.
6, because 12 bits divided by 2 bits per symbol.
Exactly, and this division is half of every throughput calculation: bits divided by bits per symbol gives the number of symbols, and times the seconds per symbol the duration.
3, because 12 divided by 4 colours.
You divide by bits per symbol, not by the number of colours. Four colours carry 2 bits, not 4.
24, each bit twice to be safe.
Securing by repetition is a real technique, redundancy, but a separate decision. The question asked for the minimum.
Bits divided by bits per symbol gives the number of symbols.
Why is an alphabet of just two symbols in principle enough for any message, however complicated?
It is not enough: text needs at least 26 symbols.
The way out is grouping: five two-way symbols give 32 combinations, more than enough for 26 letters.
Because two symbols lie furthest apart in the signal, so the receiver never confuses them.
True, but it does not answer the question: being recognisable robustly is not the same as being expressive. That two symbols suffice for any message is due to grouping, not to the gap.
Because every symbol of a larger alphabet can be replaced by a fixed group of two-way symbols; it only costs length.
Exactly, and that is why the bit is the universal currency: everything can be exchanged into binary decisions. For the same reason, a standard defined in bits rather than colours even survives swapping the carrier, say sound instead of light.
Because computers work with two states internally anyway, so every message is already in that form before sending.
That is an observation, not a reason: computers work that way because grouping works, not the other way round.
What a large alphabet achieves per symbol, two symbols achieve in a group.
In terms of symbols and information, what does this dictionary establish?
The four colours a lamp can produce, each with the code the manufacturer assigned to it.
A lamp can produce far more colours than listed, and the codes come from whoever wrote the table, not from a manufacturer. The dictionary selects and assigns; it describes no limit of the hardware.
The agreement: which symbol carries which group of bits. The receiver needs exactly the same mapping, only in reverse.
Exactly: this is an alphabet as a code, four symbols of 2 bits each. These lines are one half of the agreement, and if sender and receiver differ in even one entry, perfect reception becomes perfect nonsense.
The order in which the colours are sent: first red, then green, then blue, then yellow.
The order is determined by the message, not the dictionary. This only says what each colour means; when it appears is decided by the text.
How many symbols are sent per second, because two bits per symbol determine the pace of the transmission.
Bits per symbol say how much a symbol carries, not how long it lasts. The dictionary says nothing about time, and deliberately so: the meaning of the symbols and the pace of the transmission are separate decisions.
The dictionary is the agreement in code, and it has to exist identically on both sides.
Sixteen faces, one of them is the secret. Question A is "does it wear glasses?", and eight of the sixteen do. Question B is "is it face number 7?". Which question is worth more on average?
Question A: it yields a guaranteed 1 bit, question B only about a third of that on average.
Question A halves the field no matter how it turns out: 16 become 8, exactly 1 bit. Question B resolves everything in one case out of sixteen (4 bits) but leaves 15 possibilities in the other fifteen, which buys almost nothing. On average that is about 0.34 bits.
Question B: a yes solves the puzzle completely at once, which A can never do.
True for the rare lucky case, and this is the most common misconception. But you do not get to choose which answer you receive, and in fifteen cases out of sixteen you learn almost nothing. Count what a question yields on average, not at its best.
Both the same: they are both yes/no questions, and a yes/no answer is one bit.
One bit is the ceiling of a yes/no answer, not its value. It is reached only when both outcomes are equally likely, and the more lopsided the question, the less of it remains.
That depends on luck and cannot be said in advance.
The single outcome depends on luck, the expected value does not. That is exactly why questions can be compared before you ask them, and why halving is the best strategy.
Only a question that splits the field in half is worth a whole bit; the expected value of a lopsided one is far lower.
In Morse code the most common letter, "e", is also the shortest: a single dot. Rare letters like "q" are long. Why does that unequal treatment pay off?
Because short codes are less error-prone, so the most common letter should be the best protected.
If anything it is the other way round: a single dot is easier to miss than a long sequence. The gain is not in safety but in the length of the whole message.
Because "e" is the most important letter and important letters should arrive first.
Common is not the same as important, and the order of the letters does not change either way. All that matters is how often a letter occurs.
Because it makes the average message shorter: what occurs often then costs little.
Exactly the point. When letters are not equally likely, a rare letter carries more information than a common one, so it may also take up more room. Every compression scheme rests on that observation.
Because an alphabet with codes of different lengths can hold more letters than one with equal lengths.
The number of letters that can be represented depends on the length, not on its unevenness: codes of equal length with the same maximum length can represent at least as many. The gain from unequal lengths lies in the average.
Frequent symbols get short codes, so the average message gets shorter.
A link can reliably manage only two symbols, bright and dark. It is still supposed to transmit an alphabet of 32 characters by grouping several flashes. How many flashes does one group need at least?
16, half the number of characters.
The number of flashes does not grow in proportion to the number of characters but logarithmically: each extra flash doubles the number of groups instead of adding two. 16 flashes would cover 65,536 characters.
5, because two to the fifth is 32.
Each flash carries one decision between two possibilities, so one bit; five flashes carry five bits and thus tell 32 groups apart. That is the halving calculation read backwards.
32, because every character needs one flash.
That would be the case if every character had its own symbol, but there are only two symbols. Grouping several flashes is precisely the way around that.
6, because one extra place is needed to separate the groups.
Not a foolish thought, since the receiver has to separate the groups somehow. But the fixed length does that job by itself: if every group has five flashes, the receiver simply counts along. A separator is only needed when groups may differ in length.
n flashes with two possibilities give 2ⁿ groups, and 2⁵ = 32.
Claude E. Shannon: "A Mathematical Theory of Communication". Bell System
Technical Journal 27, 1948. The paper the bit, uncertainty as a logarithm and
the limit of every compression come from; its first pages get by with little
mathematics, and Claude Shannon tells how it came about.
James Gleick: The Information: A History, a Theory, a Flood. Pantheon, 2011.
A history of information from the talking drums of Africa through the
telegraph to Shannon, easy to read and free of formulas.