基于AP聚类的不完整大数据填充

摘要

Imputing missing values is a hot research topic in big data processing. Incomplete big data could not be clustered directly, which must be preprocessed, such as inputing missing values. Traditional algorithms, which fill missing values depending on the statistical theory of the data set, are corrupted by noise data, which decreases the imputation accuracy. The paper proposes a novel algorithm to fill missing values based on AP clustering, which clusters incomplete big data directly by proposing a new similarity metrics. The data in the same cluster is utilized to fill missing values which aims at avoiding noise corruption to improve the filling accuracy. Experiment demonstrates the proposed algorithm can cluster the incomplete big data directly and improve the accuracy for filling missing data effectively.%不完整数据的分析与填充一直是大数据处理的热点研究课题，传统的分析方法无法对不完整数据直接聚类，大部分方法先填充缺失值，然后对数据聚类。这些方法一般利用整个数据集对缺失数据进行填充，使得填充值容易受到噪声的干扰，导致填充结果不精确，进而造成聚类精度很低。提出一种不完整数据聚类算法，对不完全信息系统的相似度公式进行重新定义，给出不完整数据对象间的相似度度量方式，进而直接对不完整数据聚类。根据聚类结果将同一类对象划分到相同的簇中，通过同一类对象的属性值对缺失值进行填充，避免噪声对填充值的干扰，提高填充结果的精确性。实验结果表明，提出的方法能够对不完整数据进行聚类，并有效提高缺失数据的填充精度。