Numerical errors of the real time dynamic correction for the current measurement

摘要

The finite precision of arithmetic operations performed by a processor is the source of numerical errors. An algorithm of the dynamic error correction has been tested with respect to the numerical errors. A truncation operation of any bit number representation has been simulated. A floating -point processors have been tested during simulations. Two criteria ⅡⅡ_2 and ⅡⅡ_∞ have been examined.