Conditional Information Control to Maximize and Minimize Information

摘要

In this article, we propose a new information-theoretic method called conditional information control. The method is introduced to maximize and minimize information in one network. To maximize and minimize information, we use conditional information that can take different values for different input patterns. We introduce distortion between Shannon and Renyi information functions to control information. By minimizing this distortion, conditional information can be maximized and at the same time minimized, depending upon input patterns. We applied the method to character recognition, animal classification, and grammatical inference. In all cases, experimental results confirmed that conditional information is maximized or minimized, depending upon specific input patterns, and that a limited number of important or principal hidden units can be detected. In addition, internal representation obtained by conditional information maximization and minimization can clearly be interpreted.