AC2: A Policy Gradient Actor with Primary and Secondary Critics

摘要

We propose AC2, a policy gradient algorithm that employs a primary and a secondary critic to manage both bias and variance in policy gradients. We present through analyses and experiments that performance becomes more stable if a secondary critic concentrates on few problematic states (upper 95-percentile) that cause extreme changes in value estimates. This scheme can keep biases tolerable while lowering variances. We relate our algorithm with critic ensembles that have more components and show that ensemble averaging may not significantly reduce gradient variances in more difficult environments. We test our algorithm in a series of high-dimensional experiments and report better performance than ensembles with more critic components especially in harder environments. In addition, performance is more stable if the secondary critic trains on a few problematic states than by random sampling. Our algorithm reports better reward performance than single critic and other RL models.