DATA MINING, STRIP MINING AND OTHER HISTORICAL DATA HIGH JINX

摘要

When one decides to embark on a data mining project there are two key tasks that must be completed at the very beginning: clearly defining the goals and expectations of the project, and preparing the data properly before any data mining or modeling is performed. When data mining with historical data sets one needs to understand several aspects of the data: 晇ariable data types, data structures, existence of potential outliers, equipment used at each operation, relationships, interactions and correlations between categorical and continuous variables, relationships between predictor and response variables, effects over time, basic assumptions about the distributions of the variables and data integrity. Using SAS Institute's JMP statistical analysis software package, several solutions will be proposed to address these data issues. The following techniques are presented: making multiple scatterplots to highlight potential outliers, constructing frequency tables to highlight missing cells and small sample sizes, using date variables to compare tools running simultaneously, changing color and symbol type to add dimensionality to the data, concatenating categorical variables to look for interactions, constructing histograms and probability plots to check data distributions, and using summary sample size tables to check data integrity. These techniques will enable the analyst to make sound, realistic and statistically correct decisions when data mining with large historical data sets.