EFFICIENT FACTOR ANALYSIS ON LARGE DATASETS USING CATEGORICAL VARIABLES

摘要

Methods and apparatus are disclosed for efficient factor analysis of a large population of data records, using factors that are categorical variables. Computation is balanced between extracting key factors by training a machine learning classifier on a reduced sample of data records, for computational efficiency, and scoring the categorical values of the key factors on the entire population, for accuracy of results. A joint factor is constructed by combining all proposed root factors, and the sample is generated by stratified sampling on the joint factor. The key factors are selected from candidate factors which can be combinations of the root factors. Original variables of a dataset, whether categorical or not, can be binned to obtain new categorical factors. Variations and user interfaces are also disclosed.