A comprehensive review of data quality management-and-insurance and respective machine learning and deep learning based techniques; case study, class imbalance (in the context of MNIST character classification)

摘要

In this paper we comprehensively investigate and discuss the most important "data quality" management related core issues. Specifically, the following issues are addressed: (a) What are the various realistic imperfections and/or sicknesses which can affect data and their respective origin? (b) Furthers, what are the appropriate diagnostic concepts (i.e. detection algorithms/schemes) w.r.t. each of the imperfections/sicknesses? (c) In addition, what are the respective machine-learning and deep learning based healing/reparation/mitigation concepts? Finally, for illustrative purposes, the effect of the class imbalance sickness is taken as a case study which is closely analyzed/studied in the context of a special dataset prepared for the case: here one just considers the comprehensive analysis of the class imbalance impact on the performance of two selected classifiers with two different architectures.