Evaluating the Quality of Datasets in Software Engineering

摘要

Research based on datasets needs to determine the quality of the data that form the basis of the results. To facilitate this type of research, the meaning of the data needs to be interpreted correctly. A dataset essentially contains data and metadata. Data are often misinterpreted dueto insufficient metadata, which causes quality issues to be associated with the datasets, such as failure to clearly identify the entity being measured and inability to clarify how the measurements were generated. The goal of this study was to determine a useful way to evaluate the qualityof datasets. We developed a quality assessment process that includes four steps: assessing the datasets model, identifying the data quality issues, evaluating the metadata, and preparing the assessment report. We introduced formal definitions of data quality issues to identify the qualityissues, as well as an evaluation scale to evaluate the quality of the metadata in datasets. We applied the quality assessment process to 92 existing datasets from real data repositories and found a number of common data quality issues in these datasets, such as duplicate, incorrect and missingdata. We also found 70 datasets containing insufficient metadata for entity and metrics. Our quality assessment process can be used to determine datasets that might carry risks of data misinterpretation because of the absence of metadata. The process also allows researchers to draw conclusionsabout whether a dataset has sufficient metadata to support correct interpretation for analysis in empirical research.