THE CO-INFORMATION LATTICE

摘要

In 1955, McGill published a multivariate generalisation of Shannon's mutual information. Algorithms such as Independent Component Analysis use a different generalisation, the redundancy, or multi-information. McGill's concept expresses the information shared by all of K random variables, while the multi-information expresses the information shared by any two or more of them. Partly to avoid confusion with the multi-information, I call his concept here the co-information. Co-informations, oddly, can be negative. They form a partially ordered set, or lattice, as do the entropies. Entropies and co-informations are simply and symmetrically related by Moebius inversion. The co-information lattice sheds light on the problem of approximating a joint density with a set of marginal densities, though as usual we run into the partition function. Since the marginals correspond to higher-order edges in Bayesian hypergraphs, this approach motivates new algorithms such as Dependent Component Analysis, which we describe, and (loopy) Generalised Belief Propagation on hypergraphs, which we do not. Simulations of subspace-ICA (a tractable DCA) on natural images are presented on the web. In neural computation theory, we identify the co-information of a group of neurons (possibly in space/ time staggered patterns) with the 'degree of existence' of a corresponding cell assembly.