A Pulse Neural Network Reinforcement Learning Algorithm for Partially Observable Markov Decision Processes

摘要

This paper considers learning by a pulse neural network and proposes a new reinforcement learning algorithm focusing on the ability of pulse neuron elements to process time series. The conventional integrator neuron element is modeled in terms of the average firing rate of the biological neuron. But the pulse neuron is a modeling of the input-output relation of the time-series pulse (spike) and the decay of the internal state (internal potential). The application of such neural networks has been considered in recent engineering studies. It is known in particular that a pulse neuron with a high decay rate acts as a coincidence detector. The proposed model combines pulse neuron elements with different decay rates, which facilitates the processing of the time-series input information and the discrimination of fuzzy states in a partially observable Markov decision process. The proposed network is a four-layered feedforward network in which the pulse neuron elements forming the two hidden layers provide a pseudo-representation of the state in the environment. The elements generate a secondary reinforcement signal which results in learning similar to the conventional reinforcement scheme based on the state evaluation function. A computer experiment verifies that the proposed model works effectively in an environment which is strongly partially observable.