Local Utility Estimation in Model-Free, Multi-Agent Environments

摘要

Software agents are an enabling technology that supports rapid, automated, distributed decision making. Many joint task environments provide reward that is based on the performance of the collective, making it difficult to assign reward accurately to individual agents based on their performance. Some method is needed to assign the proper amount of credit to each of the agents in a collective, referred to as structural credit assignment, in an effort to maximize global utility. Within the multi-credit assignment problem the objective is to accurately estimate an agent's local utility based only on a global observation or global reward. To achieve an initial local estimate for each agent a Kalman filter technique is employed. The local utility estimates created through this technique however are independent of knowledge held by other agents in the environment. This leads to the intuition that there is room to improve local utility estimation through the sharing of knowledge between agents. Hence, different communication schemes are explored in order to not only improve the local estimates provided by the Kalman filter but in an effort to allow the agents to more rapidly converge to good policies.