Many large MDPs can be represented compactly using a dynamic Bayesian network. Although the structure of the value function does not retain the structure of the process, recent work has suggested that value functions in factored MDPs can often be approximated well using a factored value function: a linear combination of restricted basis functions, each of which refers only to a small subset of variables. An approximate fac-tored value function for a particular policy can be computed using approximate dynamic pro-gramming, but this approach (and others) can only produce an approximation relative to a dis-tance metric which is weighted by the station-ary distribution of the current policy. This type of weighted projection is ill-suited to policy im-provement. We present a new approach to value determination, that uses a simple closed-form computation to compute a least-squares decom-posed approximation to the value function for any weights directly. We then use this value de-termination algorithm as a subroutine in a pol-icy iteration process. We show that, under rea-sonable restrictions, the policies induced by a factored value function can be compactly repre-sented as a decision list, and can be manipulated efficiently in a policy iteration process. We also present a method for computing error bounds for decomposed value functions using a variable-elimination algorithm for function optimization. The complexity of all of our algorithms depends on the factorization of the system dynamics and of the approximate value function.
展开▼
