Minimization of Variance on Controlled Markov Chain

摘要

We consider a variance criterion on a finite-stage controlled Markov chain. The variance is a sample variance for the sequence of stage rewards (random variables) on the chain. Through a nonconventinal dynamic programming approach, we minimize the expected value of the variance over some large policy class. For computational simplicity, we take the variance multiplied by the square of the total number of stages. Our invariant imbedding method expands the original state space by one dimension. We illustrate a two-state, two-action and two-stage model by stochastic decision tree-table method. This optimal solution is obtained by solving the backward recursive equation for the minimum value functions on expanded state spaces.