Optimal Clustering and Cluster Identity in Understanding High-Dimensional Data Spaces with Tightly Distributed Points

摘要

The sensitivity of the elbow rule in determining an optimal number of clusters inhigh-dimensional spaces that are characterized by tightly distributed data points is demonstrated.The high-dimensional data samples are not artificially generated, but they are taken from a real worldevolutionary many-objective optimization. They comprise of Pareto fronts from the last 10 generationsof an evolutionary optimization computation with 14 objective functions. The choice for analyzingPareto fronts is strategic, as it is squarely intended to benefit the user who only needs one solutionto implement from the Pareto set, and therefore a systematic means of reducing the cardinality ofsolutions is imperative. As such, clustering the data and identifying the cluster from which to pickthe desired solution is covered in this manuscript, highlighting the implementation of the elbow ruleand the use of hyper-radial distances for cluster identity. The Calinski-Harabasz statistic was favoredfor determining the criteria used in the elbow rule because of its robustness. The statistic takes intoaccount the variance within clusters and also the variance between the clusters. This exercise alsoopened an opportunity to revisit the justification of using the highest Calinski-Harabasz criterionfor determining the optimal number of clusters for multivariate data. The elbow rule predictedthe maximum end of the optimal number of clusters, and the highest Calinski-Harabasz criterionmethod favored the number of clusters at the lower end. Both results are used in a unique wayfor understanding high-dimensional data, despite being inconclusive regarding which of the twomethods determine the true optimal number of clusters.