Data Distribution Assessment and Optimal Splitting of Data Sets

摘要

A new method for assessing the quality of a data distribution based on the calculation of the Kullback-Leibler (KL) divergence is proposed. The pdf of the data is estimated by a kernel density estimator. In the case without any prior knowledge the target distribution is assumed to be uniform. Then Monte Carlo sampling of the estimated pdf allows to approximate the KL divergence as criterion for the space-filling properties of the data distribution. Applications of this KL-based criterion are manifold. Sobol sequences and maximin latin hypercubes, most frequently applied for space-filling design of experiments, are compared. Finally, strategies for optimally splitting data sets are discussed and illustrated.