Interpretability of Deep Neural Networks has become a major area of exploration. Although these networks have achieved state-of-the-art results in many tasks, it is extremely difficult to interpret and explain their decisions. In this work, we analyze the final and penultimate layers of Deep Convolutional Networks for image classification with respect to ℓ1documentclass[12pt]{minimal} usepackage{amsmath} usepackage{wasysym} usepackage{amsfonts} usepackage{amssymb} usepackage{amsbsy} usepackage{mathrsfs} usepackage{upgreek} setlength{oddsidemargin}{-69pt} begin{document}$$ell _1$$end{document} norm and develop an algorithm for identifying subsets of features that contribute most toward the network’s decision for each class. We also develop a novel decomposed softmax to efficiently re-train the network such that the class-specific decomposition is preserved. We provide a comparison with other methods for identifying class-specific filters and show, using Pairwise Mutual Information Score, that our technique provides better decomposition. The resulting decomposed final layer provides a low-dimensional embedding (decreased by around a factor of 10) per class, which is far more interpretable. Such a low-dimensional class-specific embedding makes diagnosing issues with misclassifications of a certain class in the data easier as fewer weights contribute to the decision for the data points of that class. It also enables the network toward easier diagnostics and pruning as an entire part of the final and pre-final layer can be excluded to remove predictions for data points belonging to a particular label. The resulting layer also achieves a modest computational cost gain as compared to the final layer of the full network. Our algorithm is unsupervised in nature and can be applied to any CNN.
展开▼
