Information and the logarithm

Let’s write $I(s,n)$ for the information of a string of length $n$ made up of an alphabet of size $s.$ Since we want the function $I$ to be additive, we need

$I(s,n) = k_ s n,$

where $k_ s = I(s,1).$ Our task is to find the $k_ s,$ so that we can write down $I(s,n)$ explicitly for all $n.$

Let’s fix some other positive whole number $s_1$ and declare that $I(s_1, 1) = k_1$ for some constant $k_1.$ This just means that we have fixed the value of the information in a single symbol from an alphabet of size $s_1.$

From Hartley’s second rule we know that the function $I$ must satisfy:

$\begin{equation} s_1^{n_1} = s^{n}, \end{equation}$

(1)

for some numbers $n_1$ and $n,$ then

$\begin{equation} I(s_1, n_1) = k_1 \times n_1 = k_ s \times n =I(s, n).\end{equation}$

(2)

Therefore,

$\begin{equation} k_ s = \frac{k_1n_1}{n}.\end{equation}$

(3)

Now from (1) we see that

$n_1 = \log _{s_1}{\left(s^{n}\right)}.$

Substituting this into (3) gives

$k_ s = \frac{k_1 \log _{s_1}{\left(s^{n}\right)}}{n}.$

Therefore

$I(s,n) = k_ s n = k_1 \log _{s_1}{\left(s^{n}\right)}.$

The $k_1$ was just some constant we fixed above. We can actually make it disappear from the formula by absorbing it into the base of the logarithm. Suppose we want that base to be $b.$ Let’s choose

$k_1 = \log _ b{\left( s_1 \right)}.$

Then by the rule for changing the base of logarithms we get

$I(s,n) = \log _ b{\left( s_1 \right)} \log _{s_1}{\left(s^{n}\right)} = \log _ b{\left( s_1 \right)} \frac{\log _ b{\left( s^ n\right)}}{\log _ b{\left(s_1\right)}} = \log _ b{\left( s^ n\right)},$

as required.

Return to main article