Data quality improvement in data warehouse: a framework

摘要

Data cleansing is an extremely imperative process which when carried out on the datasets, eliminates the inconsistency and duplicity from the data. It also handles null values or missing values in the data in an organised and proper manner thereby enhancing the quality of the data. In this paper, we use Kullback-Leibler divergence (KL-divergence) technique to eliminate duplicity in the datasets. Inconsistency, null values or missing values are also handled in the datasets. This is done by maintaining data marts which are made on the basis of test data. Accordingly, a framework for efficient data cleansing is suggested in order to make the data appropriate and proper for decision making purpose. A brief comparison of existing approaches of data cleansing have also been discussed. This comparison is based on various parameters such as prediction error, bias, mean square error, variance, mean absolute error, root mean square error, Theil statistics etc. These parameters are used by distance sum-based approach (DSA) to accomplish the task. The results obtained demonstrate the feasibility and validity of our method.