Non-parametric estimation of data dimensionality prior to data compression: the case of the human development index

摘要

In many applications in applied statistics, researchers reduce the complexity of a data set by combining a group of variables into a single measure using a factor analysis or an index number. We argue that such compression loses information if the data actually have high dimensionality. We advocate the use of a non-parametric estimator, commonly used in physics (the Takens estimator), to estimate the correlation dimension of the data prior to compression. The advantage of this approach over traditional linear data compression approaches is that the data do not have to be linearised. Applying our ideas to the United Nations Human Development Index, we find that the four variables that are used in its construction have dimension 3 and the index loses information.