Bit

Definition

Bit is the standard unit of information. Given an event $E$, the amount of bit the event conveys is given as follows:

$$I(E)=\log \left(\frac{1}{p(E)}\right)=-\log (p(E))$$

Intuition

Let’s say that you want to transmit some “information” to another person. (e.g., tomorrow’s weather) However, your transmission device is so slow, so you want to transmit as less informatoin as possible for efficiency. In this scenario, is there a minimum amount of data that is sufficient enough to contain all the information you want to transmit? If so, how can you find this?

$$\cdots$$

The above definition of “bits” of information can give us some insights. According to this definition, an event $E$ conveys 1 bit of information if the information “cuts” your space of possibilities in half. In other words, the information can reduce the uncertainty by a factor of two; amongst the two equally likely possibilities, now there is just one.

Image Source: Solving Wordle using information theory

From our above scenario, suppose that it is either sunny (50%), or rainy (50%) tomorrow. Now, a weather station tells you that it will be rainy tomorrow (event $E$). This event has reduced your uncertainty of tomorrow’s weather by a factor of two. Hence, we can say that the weather station has transmit a single bit of information to you.

Image Source: A Short Introduction to Entropy, Cross-Entropy and KL-Divergence

This equivalently means that, it is possible to transmit the same information within a single bit; if the weather station used a string ("rainy\0", 48 bits) instead, then it used 48 times more physical bits than it actually needs!

References

• YouTube. (2022, February 6). Solving wordle using information theory. YouTube. https://www.youtube.com/watch?v=v68zYyaEmEA
• YouTube. (2018, February 5). A short introduction to entropy, cross-entropy and KL-divergence. YouTube. https://www.youtube.com/watch?v=ErfnhcEV1O8