A Semi-Automatic System for Data Management and Cleaning

摘要

We present a data management system intended for semi-automatic database pre-processing. This system allows the user to obtain useable data from a large amount of records, simplifying the cleaning process with a set of built-in tools which allows the user to identify and exploit patterns in the data. This paper describes a system with set of tools including categorical definition of variables, multi-variable column split using stop characters, data cleansing based on regular expression and inclusion of tables and standards from datasets; is presented. The framework allows for operation traceability, making it feasible to create a repeatable and fast data pre-processing and cleansing procedure.