An Investigation of How Data Quality is Affected by Dataset Size in the Context of Big Data Analytics

摘要

In the Big Data era the volume and availability of datasets are increasing massively throughout industrialrnorganisations. These organisations, with this data, are using data analytics to provide business insights inrna way that has never been exploited before. Despite its critical role in the past, however, the problems ofrndata quality are sometimes being dismissed in this Big Data world as being irrelevant. For example, in arnlarge sample of data, will the effects of any data errors be “scaled out” as we continue to add more data?rnThe aim of this work was to determine, empirically, if and when this is the case. We investigated thernproblem of completeness on data mining classification as we increase the volume of records used to trainrnthe classifier. Our results indicate that data quality is even more important in the Big Data world ofrnincreased volume. We also found that there are opportunities for managers to improve their analyticrnresults by combining, in the correction proportions, increasing dataset size with improvements to datarnquality.