The entropy theorem via coding bounds

摘要

The entropy theorem (also known as the Shannon-McMillan-Breiman theory or the asymptotic equipartition theorem) asserts that, for a stationary ergodic finite alphabet process, the sequence-(1)log p(x/sub 1//sup n/) converges almost surely to the entropy-rate H of the process. The entropy theorem has been used to establish asymptotic bounds on the performance of noiseless codes. Here, the coding theorems are established without using the entropy theorem, and the coding theorems are then used to prove the entropy theorem. The principle feature is the direct use of coding ideas to obtain the entropy theorem.