Monitoring data sparsity during neural network training

摘要

An electronic device comprising a processor and a sparsity monitor is described as a processor configured to execute training iterations during a training process for a neural network, each training iteration comprising processing an individual instance of training data via the neural network. During operation, the sparsity monitor acquires, during each monitoring interval of one or more monitoring periods, intermediate data output by at least some intermediate nodes of the neural network during a training iteration that occurs during each monitoring interval. The sparsity monitor then generates, based at least in part on the intermediate data, one or more values indicative of sparsity characteristics for the intermediate data. The sparsity monitor then sends, to the processor, one or more values indicative of the sparse characteristics, the processor controlling one or more aspects of executing a subsequent training iteration based at least in part on the one or more values indicative of the sparse characteristics.