Why You Should Constrain Your Machine Learned Models

摘要

Common use of machine learning is to gather what training examples one can, train a flexible model with some smoothness regularizers, test it on a held-out set of random examples, and *hope* it works well in practice. But we will show that by adding constraints, we can prepare our models better for their futures, and be more certain of their performance. Based on 8 years of experience at Google researching, designing, training, and launching hundreds of machine-learned models, I will discuss dozens of ways that we found one can constrain ML models to produce more robust, fairer, safer, more accurate models that are easier to debug and that when they fail, do so more predictably and reasonably. This talk will focus on two classes of model constraints: shape constraints, and rate constraints. The most common shape constraint is mono-tonicity, and it has long been known how to learn monotonic functions over one input using isotonic regression. We will discuss new R&D about 6 different practically useful shape constraints, and how to impose them on flexible, mulit-layer models. The second class of constraints, rate constraints, refers to constraints on a classifiers' output statistics, and is commonly used to make classifiers act responsibly for different groups. For example, we may constrain a classifier used globally to be at least 80% accurate on training examples from India or China, as well as minimizing classification errors on average. We will point listeners to Google's open-source Tensor Flow libraries to impose these constraints, and papers with more technical detail.