A Study on Network Size Reduction Using Sparse Input Representation in Time Delay Neural Networks

摘要

Neural networks are being increasingly used in many applications. While large deep neural networks are quickly advancing, for computing devices that lack powerful processors, it is desirable to use smaller neural networks. In this paper, we focus on smaller neural networks than common deep networks and examine how their size can be made even smaller without sacrificing the performance. We show that for some data types it is possible to make the networks smaller using sparse representation of the input. We have studied time delay neural networks (TDNN) for time series prediction using three different datasets. It is found that sparsifying input to the TDNN, using Discrete Cosine Transform (DCT), or Principal Component Analysis (PCA), can result in performance improvement. The improved performance can be traded off for network size reduction. Therefore, we can make the network smaller and maintain the same performance. It is found that for data that has more randomness and sudden changes in value (higher frequencies present), sparse representation methods using discrete cosine transform or through principal component analysis allow reducing network size by up to 40%.