Forecaster performance evaluation with cross-validation and variants

摘要

In time series prediction, there is currently no consensus for a best practice of how predictors should be compared and evaluated. We investigate this issue through an empirical study. First, we discuss forecast types, error calculation, and error averaging methods in use, and then we focus on model selection procedures. We consider using ordinary cross-validation techniques and the common time series approach of choosing a test set from the end of a series, as well as less common approaches such as non-dependent cross-validation or blocked cross-validation. The study uses different error measures, various machine learning methods, and synthetic time series data. The results indicate that cross-validation can be a useful tool also in time series evaluation. Theoretical problems can be prevented by using it in the blocked form.