Entropy and Compression: A Simple Proof of an Inequality of Khinchin-Ornstein-Shields

摘要

This paper concerns the folklore statement that "entropy is a lower bound for compression." More precisely, we derive from the entropy theorem a simple proof of a pointwise inequality first stated by Ornstein and Shields and which is the almost-sure version of an average inequality first stated by Khinchin in 1953. We further give an elementary proof of the original Khinchin inequality, which can be used as an exercise for information theory students, and we conclude by giving historical and technical notes of such inequality.