Abstract—Deep Neural Networks (DNNs) havedemonstrated impressive performance in complex learning tasks like imageclassification or voice recognition. However, because of their multilayernonlinear structure, they are not transparent, That is, it's hard to understandwhat makes them happen to a particular classification or recognition decision,given a new sample of data. Recently,several approaches have been proposed to understand and interpret the reasoningembodied in a DNN for a single test image. These methods quantify the"Importance" of individual pixels in relation to the classificationdecision and allow visualization in terms of heatmap in Pixel / input space.Although the utility of heatmaps can be judged subjectively by a human, ameasure of objective quality is missing. n this article, we present a general methodology based on theregion disruption for the evaluation of ordered collections of pixels such asheatmaps. We compare heatmaps calculated by three different the methods onSUN397, ILSVRC2012 and MIT place datasets. Our main result is that the relevance of the recentlyproposed layer-wise propagation algorithm provides qualitatively andquantitatively a better explanation of what made a DNN happen to a particularthe classification decision that the approach focused on sensitivity or themethod of devolution.
Keywords: Convolutional neural networks,explaining classification, image classification, interpretable machinelearning, relevance models