Parallel learning of neural networks on PC-cluster systems using mini-batch learning schema

摘要

Due to the increasing development of hardware and network technology, it has become common to have a huge data set. When processing such data by neural networks, learning takes long time. In this paper, we propose parallel learning by distributing a task into two or more CPUs. In parallel learning, mini-batch learning schema is indespensable because we must divide the entire data set into subsets and allocate each of them to the parallelized neural network. In parallel learning, the redundancy of the data is a key concept. When there are no redundancy in data, learning accuracy might decrease by the existence of the bias in each subset. On the other hand, when the data has much redundancy, the bias in each subset would be small and parallel learning might yield high efficiency. In this paper, we study a parallel learning procedure using mini-batch learning schema and investigate the relationship between the efficiency of the parallel learning and the data redundancy. It is quite common for huge data sets to have some redundancy and we expect our procedure might work in a variety of applications.