Learning, Entropy, Free Energy, an Underlying Commonality ?

摘要

Statistical Mechanics, which is due primarily to Maxwell, Gibbs, and Boltzmann in the ninetieth century, has proven to be useful model for drawing inferences about the collective behavior of individual objects that interact according to a known force law (which for more general usage is referred to as interacting units.). Collective behavior is determined not by computing F = ma for each interacting unit because the problem is mathematically intractable. Instead, one computes the partition function for the collection of interacting units and predicts statistical behavior from the partition function. Statistical mechanics was united with Bayesian inference by Jaynes. As a continuation, Shannon demonstrated that the partition function assignment of probabilities via the interaction Hamiltonian is the solution to Bayesian assignment of probabilities (based on the maximum entropy method with known means and standard deviations). Once this technique has been applied to a variety of problems and obtained a solution, one can, of course, solve the inverse problem to determine what interaction model gives rise to a given probability assignment. The usage of statistical mechanics allows one to draw general inferences about any complex system including networks by defining "energy", "heat capacity", "temperature", and other thermodynamic characteristics of most complex systems based on the common standard of the Helmholtz free energy. Principle has noted that the aspect of entropy used in reasoning with uncertainty may not be the most appropriate entropy for learning mechanisms. Instead he has explored using Renyi entropy and derived a form of information learning dynamics that has some promising features. To fully realize the potential of the usage of a more generalized entropy to the three aspects of survival, we suggest some connections to free energy and learning. We also connect some aspects of sensing to probability distributions that suggest why certain search strategies perform better than others. In making these connections, we suggest a fundamental connection waits to be discovered between inference, learning, and related to the manner in which sensing mechanisms perform.