Optimal constraint-based regularization for parameter estimation problems

摘要

We explore strategies to regularize ill-posed parameter estimation problems by using constraints. We argue that constraints provide a flexible and easily interpretable approach to reduce the parameter space (compared to objective-based regularization). We begin our discussion by revisiting strategies to exploit the eigenvalue structure of the Hessian matrix (inverse of the parameter covariance matrix) to design optimal constraints that minimize the variance of the parameter estimates. We show that these strategies can be used in more general settings by exploiting information from the reduced Hessian matrix. We also derive an elastic net strategy to sparsify the constraints and facilitate their interpretability (this strategy reveals parameter clusters). Our analysis also highlights that subspace parameter selection can be interpreted as a constraint regularization technique that is easy to interpret but that is suboptimal.