Machine learning and rate-distortion cost functions for enabling the optimization of device calibration

摘要

A method, non-transitory computer readable medium, and apparatus for applying a cost function to calculate an optimal value of a single variable for a calibration application are disclosed. For example, the method identifies the single variable of the calibration application, applies a cost function to the single variable, wherein the cost function comprises a function of a fit error plus a regularization weighting parameter (λ) times a regularization cost, calculates the optimal value of the single variable based upon the cost function that is applied to the single variable and uses the optimal value of the single variable to generate a calibration matrix used for the calibration application.