{"title": "Real Time Image Saliency for Black Box Classifiers", "book": "Advances in Neural Information Processing Systems", "page_first": 6967, "page_last": 6976, "abstract": "In this work we develop a fast saliency detection method that can be applied to any differentiable image classifier. We train a masking model to manipulate the scores of the classifier by masking salient parts of the input image. Our model generalises well to unseen images and requires a single forward pass to perform saliency detection, therefore suitable for use in real-time systems. We test our approach on CIFAR-10 and ImageNet datasets and show that the produced saliency maps are easily interpretable, sharp, and free of artifacts. We suggest a new metric for saliency and test our method on the ImageNet object localisation task. We achieve results outperforming other weakly supervised methods.", "full_text": "Real Time Image Saliency for Black Box Classi\ufb01ers\n\nPiotr Dabkowski\npd437@cam.ac.uk\n\nUniversity of Cambridge\n\nYarin Gal\n\nand Alan Turing Institute, London\n\nyarin.gal@eng.cam.ac.uk\n\nUniversity of Cambridge\n\nAbstract\n\nIn this work we develop a fast saliency detection method that can be applied to\nany differentiable image classi\ufb01er. We train a masking model to manipulate the\nscores of the classi\ufb01er by masking salient parts of the input image. Our model\ngeneralises well to unseen images and requires a single forward pass to perform\nsaliency detection, therefore suitable for use in real-time systems. We test our\napproach on CIFAR-10 and ImageNet datasets and show that the produced saliency\nmaps are easily interpretable, sharp, and free of artifacts. We suggest a new metric\nfor saliency and test our method on the ImageNet object localisation task. We\nachieve results outperforming other weakly supervised methods.\n\n1\n\nIntroduction\n\nCurrent state of the art image classi\ufb01ers rival human performance on image classi\ufb01cation tasks,\nbut often exhibit unexpected and unintuitive behaviour [6, 13]. For example, we can apply a small\nperturbation to the input image, unnoticeable to the human eye, to fool a classi\ufb01er completely [13].\nAnother example of an unexpected behaviour is when a classi\ufb01er fails to understand a given class\ndespite having high accuracy. For example, if \u201cpolar bear\u201d is the only class in the dataset that contains\nsnow, a classi\ufb01er may be able to get a 100% accuracy on this class by simply detecting the presence\nof snow and ignoring the bear completely [6]. Therefore, even with perfect accuracy, we cannot\nbe sure whether our model actually detects polar bears or just snow. One way to decouple the two\nwould be to \ufb01nd snow-only or polar-bear-only images and evaluate the model\u2019s performance on these\nimages separately. An alternative is to use an image of a polar bear with snow from the dataset and\napply a saliency detection method to test what the classi\ufb01er is really looking at [6, 11].\nSaliency detection methods show which parts of a given image are the most relevant to the model\nfor a particular input class. Such saliency maps can be obtained for example by \ufb01nding the smallest\nregion whose removal causes the classi\ufb01cation score to drop signi\ufb01cantly. This is because we expect\nthe removal of a patch which is not useful for the model not to affect the classi\ufb01cation score much.\nFinding such a salient region can be done iteratively, but this usually requires hundreds of iterations\nand is therefore a time-consuming process.\nIn this paper we lay the groundwork for a new class of fast and accurate model-based saliency\ndetectors, giving high pixel accuracy and sharp saliency maps (an example is given in \ufb01gure 1). We\npropose a fast, model agnostic, saliency detection method. Instead of iteratively obtaining saliency\nmaps for each input image separately, we train a model to predict such a map for any input image in a\nsingle feed-forward pass. We show that this approach is not only orders-of-magnitude faster than\niterative methods, but it also produces higher quality saliency masks and achieves better localisation\nresults. We assess this with standard saliency benchmarks and introduce a new saliency measure.\nOur proposed model is able to produce real-time saliency maps, enabling new applications such as\nvideo-saliency which we comment on in our Future Research section.\n\n2 Related work\n\nSince the rise of CNNs in 2012 [5] numerous methods of image saliency detection have been proposed.\nOne of the earliest such methods is a gradient-based approach introduced in [11] which computes\nthe gradient of the class with respect to the image and assumes that salient regions are at locations\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\f(a) Input Image\n\n(b) Generated saliency map\n\n(d) Image multiplied by inverted mask\nFigure 1: An example of explanations produced by our model. The top row shows the explanation for the\n\"Egyptian cat\" while the bottom row shows the explanation for the \"Beagle\". Note that produced explanations\ncan precisely both highlight and remove the selected object from the image.\n\n(c) Image multiplied by the mask\n\nwith high gradient magnitude. Other similar backpropagation-based approaches have been proposed,\nfor example Guided Backpropagation [12] or Excitation Backprop [16]. While the gradient-based\nmethods are fast enough to be applied in real-time, they produce explanations of limited quality [16]\nand they are hard to improve and build upon.\nZhou et al. [17] proposed an approach that iteratively removes patches of the input image (by setting\nthem to the mean colour) such that the class score is preserved. After a suf\ufb01cient number of iterations,\nwe are left with salient parts of the original image. The maps produced by this method are easily\ninterpretable, but unfortunately, the iterative process is very time consuming and not acceptable for\nreal-time saliency detection.\nIn another work, Cao et al. [1] introduced an optimisation method that aims to preserve only a fraction\nof network activations such that the class score is maximised. Again, after the iterative optimisation\nprocess, only activations that are relevant remain and their spatial location in the CNN feature map\nindicate salient image regions.\nVery recently (and in parallel to this work), another optimisation based method was proposed [2].\nSimilarly to Cao et al. [1], Fong and Vedaldi [2] also propose to use gradient descent to optimise for\nthe salient region, but the optimisation is done only in the image space and the classi\ufb01er model is\ntreated as a black box. Essentially Fong and Vedaldi [2]\u2019s method tries to remove as little from the\nimage as possible, and at the same time to reduce the class score as much as possible. A removed\nregion is then a minimally salient part of the image. This approach is model agnostic and the produced\nmaps are easily interpretable because the optimisation is done in the image space and the model is\ntreated as a black box.\nWe next argue what conditions a good saliency model should satisfy, and propose a new metric for\nsaliency.\n\n3\n\nImage Saliency and Introduced Evidence\n\nImage saliency is relatively hard to de\ufb01ne and there is no single obvious metric that could measure\nthe quality of the produced map. In simple terms, the saliency map is de\ufb01ned as a summarised\nexplanation of where the classi\ufb01er \u201clooks\u201d to make its prediction.\nThere are two slightly more formal de\ufb01nitions of saliency that we can use:\n\n\u2022 Smallest suf\ufb01cient region (SSR) \u2014 smallest region of the image that alone allows a con\ufb01dent\n\nclassi\ufb01cation,\n\n2\n\n\f\u2022 Smallest destroying region (SDR) \u2014 smallest region of the image that when removed,\n\nprevents a con\ufb01dent classi\ufb01cation.\n\nSimilar concepts were suggested in [2]. An example of SSR and SDR is shown in \ufb01gure 2. It can\nbe seen that SSR is very small and has only one seal visible. Given this SSR, even a human would\n\ufb01nd it dif\ufb01cult to recognise the preserved image. Nevertheless, it contains some characteristic for\n\u201cseal\u201d features such as parts of the face with whiskers, and the classi\ufb01er is over 90% con\ufb01dent that\nthis image should be labeled as a \u201cseal\u201d. On the other hand, SDR has a much stronger and larger\nregion and quite successfully removes all the evidence for seals from the image. In order to be as\ninformative as possible, we would like to \ufb01nd a region that performs well as both SSR and SDR.\n\nFigure 2: From left to right: the input image; smallest suf\ufb01cient region (SSR); smallest destroying region (SDR).\nRegions were found using the mask optimisation procedure from [2].\n\nBoth SDR and SSR remove some evidence from the image. There are few ways of removing evidence,\nfor example by blurring the evidence, setting it to a constant colour, adding noise, or by completely\ncropping out the unwanted parts. Unfortunately, each one of these methods introduces new evidence\nthat can be used by the classi\ufb01er as a side effect. For example, if we remove a part of the image by\nsetting it to the constant colour green then we may also unintentionally provide evidence for \u201cgrass\u201d\nwhich in turn may increase the probability of classes appearing often with grass (such as \u201cgiraffe\u201d).\nWe discuss this problem and ways of minimising introduced evidence next.\n\n3.1 Fighting the Introduced Evidence\nAs mentioned in the previous section, by manipulating the image we always introduce some extra\nevidence. Here, let us focus on the case of applying a mask M to the image X to obtain the edited\nimage E. In the simplest case we can simply multiply X and M element-wise:\n\nE = X  M\n\n(1)\nThis operation sets certain regions of the image to a constant \u201c0\u201d colour. While setting a larger patch\nof the image to \u201c0\u201d may sound rather harmless (perhaps following the assumption that the mean of\nall colors carries very little evidence), we may encounter problems when the mask M is not smooth.\nThe mask M, in the worst case, can be used to introduce a large amount of additional evidence by\ngenerating adversarial artifacts (a similar observation was made in [2]). An example of such a mask\nis presented in \ufb01gure 3. Adversarial artifacts generated by the mask are very small in magnitude and\nalmost imperceivable for humans, but they are able to completely destroy the original prediction of\nthe classi\ufb01er. Such adversarial masks provide very poor saliency explanations and therefore should\nbe avoided.\n\nFigure 3: The adversarial mask introduces very small perturbations, but can completely alter the classi\ufb01er\u2019s\npredictions. From left to right: an image which is correctly recognised by the classi\ufb01er with a high con\ufb01dence as\na \"tabby cat\"; a generated adversarial mask; an original image after application of the mask that is no longer\nrecognised as a \"tabby cat\".\n\n3\n\n\fE = X  M + A  (1  M )\n\nThere are a few ways to make the introduction of artifacts harder. For example, we may change the\nway we apply a mask to reduce the amount of unwanted evidence due to speci\ufb01cally-crafted masks:\n(2)\nwhere A is an alternative image. A can be chosen to be for example a highly blurred version of X.\nIn such case mask M simply selectively adds blur to the image X and therefore it is much harder\nto generate high-frequency-high-evidence artifacts. Unfortunately, applying blur does not eliminate\nexisting evidence very well, especially in the case of images with low spatial frequencies like a\nseashore or mountains.\nAnother reasonable choice of A is a random constant colour combined with high-frequency noise.\nThis makes the resulting image E more unpredictable at regions where M is low and therefore it is\nslightly harder to produce a reliable artifact.\nEven with all these measures, adversarial artifacts may still occur and therefore it is necessary to\nencourage smoothness of the mask M for example via a total variation (TV) penalty. We can also\ndirectly resize smaller masks to the required size as resizing can be seen as a smoothness mechanism.\n\n3.2 A New Saliency Metric\nA standard metric to evaluate the quality of saliency maps is the localisation accuracy of the saliency\nmap. However, it should be noted that saliency is not equivalent to localisation. For example, in order\nto recognise a dog we usually just need to see its head; legs and body are mostly irrelevant for the\nrecognition process. Therefore, saliency map for a dog will usually only include its head while the\nlocalisation box always includes a whole dog with not-salient details like legs and tail. The saliency\nof the object highly overlaps with its localisation and therefore localisation accuracy still serves as a\nuseful metric, but in order to better assess the quality and interpretability of the produced saliency\nmaps, we introduce a new, highly tuned metric.\nAccording to the SSR objective, we require that the classi\ufb01er is able to still recognise the object from\nthe produced saliency map and that the preserved region is as small as possible. In order to make sure\nthat the preserved region is free from adversarial artifacts, instead of masking we can crop the image.\nWe propose to \ufb01nd the tightest rectangular crop that contains the entire salient region and to feed that\nrectangular region to the classi\ufb01er to directly verify whether it is able to recognise the requested class.\nWe de\ufb01ne our saliency metric simply as:\n\ns(a, p) = log(\u02dca)  log(p)\n\n(3)\nwith \u02dca = max(a, 0.05). Here a is the area of the rectangular crop as a fraction of the total image size\nand p is the probability of the requested class returned by the classi\ufb01er based on the cropped region.\nThe metric is almost a direct translation of the SSR. We threshold the area at 0.05 in order to prevent\ninstabilities at low area fractions. Good saliency detectors will be able to signi\ufb01cantly reduce the\ncrop size without reducing the classi\ufb01cation probability, and therefore a low value for the saliency\nmetric is a characteristic of good saliency detectors.\nInterpreting this metric following information theory, this measure can be seen as the relative amount\nof information between an indicator variable with probability p and an indicator variable with\nprobability a \u2014 or the concentration of information in the cropped region.\nBecause most image classi\ufb01ers accept only images of a \ufb01xed size and the crop can have an arbitrary\nsize, we resize the crop to the required size disregarding aspect ratio. This seems to work well in\npractice, but it should be noted that the proposed saliency metric works best with classi\ufb01ers that are\nlargely invariant to the scale and aspect ratio of the object.\n\n3.3 The Saliency Objective\nTaking the previous conditions into consideration, we want to \ufb01nd a mask M that is smooth and\nperforms well at both SSR and SDR; examples of such masks can be seen in \ufb01gure 1. Therefore,\nmore formally, given class c of interest, and an input image X, to \ufb01nd a saliency map M for class c,\nour objective function L is given by:\n\nL(M ) = 1TV(M ) + 2AV(M )  log(fc((X, M ))) + 3fc((X, 1  M ))4\n\n(4)\nwhere fc is a softmax probability of the class c of the black box image classi\ufb01er and TV(M ) is the\ntotal variation of the mask de\ufb01ned simply as:\n\nTV(M ) =Xi,j\n\n(Mij  Mij+1)2 +Xi,j\n\n(Mij  Mi+1j)2,\n\n(5)\n\n4\n\n\fAV(M ) is the average of the mask elements, taking value between 0 and 1, and i are regularisers.\nFinally, the function  removes the evidence from the image as introduced in the previous section:\n(6)\n\n(X, M ) = X  M + A  (1  M ).\n\nIn total, the objective function is composed of 4 terms. The \ufb01rst term enforces mask smoothness,\nthe second term encourages that the region is small. The third term makes sure that the classi\ufb01er is\nable to recognise the selected class from the preserved region. Finally, the last term ensures that the\nprobability of the selected class, after the salient region is removed, is low (note that the inverted\nmask 1  M is applied). Setting 4 to a value smaller than 1 (e.g. 0.2) helps reduce this probability\nto very small values.\n\n4 Masking Model\n\nThe mask can be found iteratively for a given image-class pair by directly optimising the objective\nfunction from equation 4. In fact, this is the method used by [2] which was developed in parallel to\nthis work, with the only difference that [2] only optimises the mask iteratively and for SDR (so they\ndon\u2019t include the third term of our objective function). Unfortunately, iteratively \ufb01nding the mask is\nnot only very slow, as normally more than 100 iterations are required, but it also causes the mask to\ngreatly over\ufb01t to the image and a large TV penalty is needed to prevent adversarial artifacts from\nforming. Therefore, the produced masks are blurry, imprecise, and over\ufb01t to the speci\ufb01c image rather\nthan capturing the general behaviour of the classi\ufb01er (see \ufb01gure 2).\nFor the above reasons, we develop a trainable masking model that can produce the desired masks\nin a single forward pass without direct access to the image classi\ufb01er after training. The masking\nmodel receives an image and a class selector as inputs and learns to produce masks that minimise our\nobjective function (equation 4). In order to succeed at this task, the model must learn which parts of\nthe input image are considered salient by the black box classi\ufb01er. In theory, the model can still learn\nto develop adversarial masks that perform well on the objective function, but in practice it is not an\neasy task, because the model itself acts as some sort of a \u201cregulariser\u201d determining which patterns are\nmore likely and which are less.\n\nFigure 4: Architecture diagram of the masking model.\n\nIn order to make our masks sharp and precise, we adopt a U-Net architecture [8] so that the masking\nmodel can use feature maps from multiple resolutions. The architecture diagram can be seen in\n\ufb01gure 4. For the encoder part of the U-Net we use ResNet-50 [3] pre-trained on ImageNet [9]. It\nshould be noted that our U-Net is just a model that is trained to predict the saliency map for the given\nblack-box classi\ufb01er. We use a pre-trained ResNet as a part of this model in order to speed up the\ntraining, however, as we show in our CIFAR-10 experiment in section 5.3 the masking model can\nalso be trained completely from scratch.\nThe ResNet-50 model contains feature maps of \ufb01ve different scales, where each subsequent scale\nblock downsamples the input by a factor of two. We use the ResNet\u2019s feature map from Scale 5\n(which corresponds to downsampling by a factor of 32) and pass it through the feature \ufb01lter. The\npurpose of the feature \ufb01lter is to attenuate spatial locations which contents do not correspond to\n\n5\n\n\fthe selected class. Therefore, the feature \ufb01lter performs the initial localisation, while the following\nupsampling blocks \ufb01ne-tune the produced masks. The output of the feature \ufb01lter Y at spatial location\ni, j is given by:\n\nijCs)\n\nYij = Xij(X T\n\n(7)\nwhere Xij is the output of the Scale 5 block at spatial location i, j; Cs is the embedding of the\nselected class s and (\u00b7) is the sigmoid nonlinearity. Class embedding C can be learned as part of the\noverall objective.\nThe upsampler blocks take the lower resolution feature map as input and upsample it by a factor\nof two using transposed convolution [15], afterwards they concatenate the upsampled map with the\ncorresponding feature map from ResNet and follow that with three bottleneck blocks [3].\nFinally, to the output of the last upsampler block (Upsampler Scale 2) we apply 1x1 convolution to\nproduce a feature map with just two channels \u2014 C0, C1. The mask Ms is obtained from:\n\nMs =\n\nabs(C0)\n\nabs(C0) + abs(C1)\n\n(8)\n\nWe use this nonstandard nonlinearity because sigmoid and tanh nonlinearities did not optimise\nproperly and the extra degree of freedom from two channels greatly improved training. The mask Ms\nhas resolution four times lower than the input image and has to be upsampled by a factor of four with\nbilinear resize to obtain the \ufb01nal mask M.\nThe complexity of the model is comparable to that of ResNet-50 and it can process more than a\nhundred 224x224 images per second on a standard GPU (which is suf\ufb01cient for real-time saliency\ndetection).\n\n4.1 Training process\nWe train the masking model to directly minimise the objective function from equation 4. The weights\nof the pre-trained ResNet encoder (red blocks in \ufb01gure 4) are kept \ufb01xed during the training.\nIn order to make the training process work properly, we introduce few optimisations. First of all,\nin the naive training process, the ground truth label would always be supplied as a class selector.\nUnfortunately, under such setting, the model learns to completely ignore the class selector and simply\nalways masks the dominant object in the image. The solution to this problem is to sometimes supply\na class selector for a fake class and to apply only the area penalty term of the objective function.\nUnder this setting, the model must pay attention to the class selector, as the only way it can reduce\nloss in case of a fake label is by setting the mask to zero. During training, we set the probability of\nthe fake label occurrence to 30%. One can also greatly speed up the embedding training by ensuring\nthat the maximal value of (X T\nijCs) from equation 7 is high in case of a correct label and low in case\nof a fake label.\nFinally, let us consider again the evidence removal function (X, M ). In order to prevent the model\nfrom adapting to any single evidence removal scheme the alternative image A is randomly generated\nevery time the function  is called. In 50% of cases the image A is the blurred version of X (we use\na Gaussian blur with  = 10 to achieve a strong blur) and in the remainder of cases, A is set to a\nrandom colour image with the addition of a Gaussian noise. Such a random scheme greatly improves\nthe quality of the produced masks as the model can no longer make strong assumptions about the\n\ufb01nal look of the image.\n\n5 Experiments\n\nIn the ImageNet saliency detection experiment we use three different black-box classi\ufb01ers: AlexNet\n[5], GoogleNet [14] and ResNet-50 [3]. These models are treated as black boxes and for each one\nwe train a separate masking model. The selected parameters of the objective function are 1 = 10,\n2 = 103, 3 = 5, 4 = 0.3. The \ufb01rst upsampling block has 768 output channels and with each\nsubsequent upsampling block we reduce the number of channels by a factor of two. We train each\nmasking model as described in section 4.1 on 250,000 images from the ImageNet training set. During\nthe training process, a very meaningful class embedding was learned and we include its visualisation\nin the Appendix.\nExample masks generated by the saliency models trained on three different black box image classi\ufb01ers\ncan be seen in \ufb01gure 5, where the model is tasked to produce a saliency map for the ground truth\n\n6\n\n\f(a) Input Image\n\n(b) Model & AlexNet\n\n(c) Model & GoogleNet\n\n(d) Model & ResNet-50\n\n(e) Grad [11]\n\n(f) Mask [2]\n\nFigure 5: Saliency maps generated by different methods for the ground truth class. The ground truth classes,\nstarting from the \ufb01rst row are: Scottish terrier, chocolate syrup, standard schnauzer and sorrel. Columns b, c, d\nshow the masks generated by our masking models, each trained on a different black box classi\ufb01er (from left to\nright: AlexNet, GoogleNet, ResNet-50). Last two columns e, f show saliency maps for GoogleNet generated\nrespectively by gradient [11] and the recently introduced iterative mask optimisation approach [2].\n\nlabel. In \ufb01gure 5 it can be clearly seen that the quality of masks generated by our models clearly\noutperforms alternative approaches. The masks produced by models trained on GoogleNet and\nResNet are sharp and precise and would produce accurate object segmentations. The saliency model\ntrained on AlexNet produces much stronger and slightly larger saliency regions, possibly because\nAlexNet is a less powerful model which needs more evidence for successful classi\ufb01cation. Additional\nexamples can be seen in the appendix A.\n\n5.1 Weakly supervised object localisation\nAs discussed in section 3.2 a standard method to evaluate produced saliency maps is by object\nlocalisation accuracy. It should be noted that our model was not provided any localisation data during\ntraining and was trained using only image-class label pairs (weakly supervised training).\nWe adopt the localisation accuracy evaluation protocol from [1] and provide the ground truth label\nto the masking model. Afterwards, we threshold the produced saliency map at 0.5 and the tightest\nbounding box that contains the whole saliency map is set as the \ufb01nal localisation box. The localisation\nbox has to have IOU greater than 0.5 with any of the ground truth bounding boxes in order to consider\nthe localisation successful, otherwise, it is counted as an error. The calculated error rates for the three\nmodels are presented in table 1. The lowest localisation error of 36.7% was achieved by the saliency\nmodel trained on the ResNet-50 black box, this is a good achievement considering the fact that our\nmethod was not given any localisation training data and that a fully supervised approach employed by\nVGG [10] achieved only slightly lower error of 34.3%. The localisation error of the model trained on\nGoogleNet is very similar to the one trained on ResNet. This is not surprising because both models\nproduce very similar saliency masks (see \ufb01gure 5). The AlexNet trained model, on the other hand,\nhas a considerably higher localisation error which is probably a result of AlexNet needing larger\nimage contexts to make a successful prediction (and therefore producing saliency masks which are\nslightly less precise).\nWe also compared our object localisation errors to errors achieved by other weakly supervised\nmethods and existing saliency detection techniques. As a baseline we calculated the localisation error\n\n7\n\n\fLocalisation Err (%)\n\n39.8\n\n36.9\n\n36.7\n\nAlexnet [5] GoogleNet [14] ResNet-50 [3]\n\nTable 1: Weakly supervised bounding box localisation error on ImageNet validation set for our masking models\ntrained with different black box classi\ufb01ers.\n\nof the centrally placed rectangle which spans half of the image area \u2014 which we name \"Center\".\nThe results are presented in table 2. It can be seen that our model outperforms other approaches,\nsometimes by a signi\ufb01cant margin. It also performs signi\ufb01cantly better than the baseline (centrally\nplaced box) and iteratively optimised saliency masks. Because a big fraction of ImageNet images\nhave a large, dominant object in the center, the localisation accuracy of the centrally placed box is\nrelatively high and it managed to outperform two methods from the previous literature.\n\nCenter Grad [11] Guid [12] CAM [18] Exc [16]\n46.3\n\n41.7\n\n42.0\n\n48.1\n\n39.0\n\nFeed [1] Mask [2] This Work\n\n38.7\n\n43.1\n\n36.9\n\nTable 2: Localisation errors(%) on ImageNet validation set for popular weakly supervised methods. Error\nrates were taken from [2] which recalculated originally reported results using few different mask thresholding\ntechniques and achieved slightly lower error rates. For a fair comparison, all the methods follow the same\nevaluation protocol of [1] and produce saliency maps for GoogleNet classi\ufb01er [14].\n\n5.2 Evaluating the saliency metric\nTo better assess the interpretability of the produced masks we calculate the saliency metric introduced\nin section 3.2 for selected saliency methods and present the results in the table 3. We include a few\nbaseline approaches \u2014 the \"Central box\" introduced in the previous section, and the \"Max box\"\nwhich simply corresponds to a box spanning the whole image. We also calculate the saliency metric\nfor the ground truth bounding boxes supplied with the data, and in case the image contains more than\none ground truth box the saliency metric is set as the average over all the boxes.\nTable 3 shows that our model achieves a considerably better saliency metric than other saliency\napproaches.\nIt also signi\ufb01cantly outperforms max box and center box baselines and is on par\nwith ground truth boxes which supports the claim that the interpretability of the localisation boxes\ngenerated by our model is similar to that of the ground truth boxes.\n\nLocalisation Err (%)\n\nSaliency Metric\n\nGround truth boxes (baseline)\nMax box (baseline)\nCenter box (baseline)\nGrad [11]\nExc [16]\nMasking model (this work)\n\n0.00\n59.7\n46.3\n41.7\n39.0\n36.9\n\n0.284\n1.366\n0.645\n0.451\n0.415\n0.318\n\nTable 3: ImageNet localisation error and the saliency metric for GoogleNet.\n\n5.3 Detecting saliency of CIFAR-10\nTo verify the performance of our method on a completely different dataset we implemented our\nsaliency detection model for the CIFAR-10 dataset [4]. Because the architecture described in section\n4 speci\ufb01cally targets high-resolution images and \ufb01ve downsampling blocks would be too much for\n32x32 images, we modi\ufb01ed the architecture slightly and replaced the ResNet encoder with just 3\ndownsampling blocks with 5 convolutional layers each. We also reduced the number of bottleneck\nblocks in each upsampling block from 3 to 1. Unlike before, with this experiment, we did not use\na pre-trained masking model, but instead a randomly initialised one. We used a FitNet [7] trained\nto 92% validation accuracy as a black box classi\ufb01er to train the masking model. All the training\nparameters were used following the ImageNet model.\n\n8\n\n\fFigure 6: Saliency maps generated by our model for images from CIFAR-10 validation set.\n\nThe masking model was trained for 20 epochs. Saliency maps for sample images from the validation\nset are shown in \ufb01gure 6. It can be seen that the produced maps are clearly interpretable and a human\ncould easily recognise the original objects after masking. This con\ufb01rms that the masking model\nworks as expected even at low resolution and that FitNet model, used as a black box learned correct\nrepresentations for the CIFAR-10 classes. More interestingly, this shows that the masking model does\nnot need to rely on a pre-trained model which might inject its own biases into the generated masks.\n\n6 Conclusion and Future Research\n\nIn this work, we have presented a new, fast, and accurate saliency detection method that can be\napplied to any differentiable image classi\ufb01er. Our model is able to produce 100 saliency masks per\nsecond, suf\ufb01cient for real-time applications. We have shown that our method outperforms other\nweakly supervised techniques at the ImageNet localisation task. We have also developed a new\nsaliency metric that can be used to assess the quality of explanations produced by saliency detectors.\nUnder this new metric, the quality of explanations produced by our model outperforms other popular\nsaliency detectors and is on par with ground truth bounding boxes.\nThe model-based nature of our technique means that our work can be extended by improving the\narchitecture of the masking network, or by changing the objective function to achieve any desired\nproperties for the output mask.\nFuture work includes modifying the approach to produce high quality, weakly supervised, image\nsegmentations. Moreover, because our model can be run in real-time, it can be used for video\nsaliency detection to instantly explain decisions made by black-box classi\ufb01ers such as the ones used\nin autonomous vehicles. Lastly, our model might have biases of its own \u2014 a fact which does not\nseem to in\ufb02uence the model performance in \ufb01nding biases in other black boxes according to the\nvarious metrics we used. It would be interesting to study the biases embedded into our masking\nmodel itself, and see how these affect the generated saliency masks.\n\n9\n\n\fReferences\n[1] Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu, Jiang Wang, Zilei Wang, Yongzhen Huang, Liang\nWang, Chang Huang, Wei Xu, Deva Ramanan, and Thomas S. Huang. Look and think twice: Capturing\ntop-down visual attention with feedback convolutional neural networks. pages 2956\u20132964, 2015. doi:\n10.1109/ICCV.2015.338. URL http://dx.doi.org/10.1109/ICCV.2015.338.\n\n[2] Ruth Fong and Andrea Vedaldi. Interpretable Explanations of Black Boxes by Meaningful Perturbation.\n\narXiv preprint arXiv:1704.03296, 2017.\n\n[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.\n\nCoRR, abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385.\n\n[4] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Master\u2019s thesis, 2009. URL\n\nhttp://www.cs.toronto.edu/~{}kriz/learning-features-2009-TR.pdf.\n\n[5] Alex Krizhevsky,\n\nIlya Sutskever,\n\ndeep convolutional neural networks.\nK. Q. Weinberger, editors, Advances in Neural\n1097\u20131105. Curran Associates,\n4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf.\n\nImagenet classi\ufb01cation with\nand\nInformation Processing Systems 25, pages\nURL http://papers.nips.cc/paper/\n\nand Geoffrey E Hinton.\nIn F. Pereira, C.\n\nJ. C. Burges, L. Bottou,\n\nInc.,\n\n2012.\n\n[6] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explaining the\npredictions of any classi\ufb01er. In Proceedings of the 22nd ACM SIGKDD International Conference on\nKnowledge Discovery and Data Mining, pages 1135\u20131144. ACM, 2016.\n\n[7] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua\nBengio. FitNets: Hints for Thin Deep Nets. CoRR, abs/1412.6550, 2014. URL http://arxiv.org/\nabs/1412.6550.\n\n[8] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical\n\nimage segmentation. CoRR, abs/1505.04597, 2015. URL http://arxiv.org/abs/1505.04597.\n\n[9] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang,\nAndrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large\nScale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211\u2013252,\n2015. doi: 10.1007/s11263-015-0816-y.\n\n[10] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-\n\ntion. CoRR, abs/1409.1556, 2014. URL http://arxiv.org/abs/1409.1556.\n\n[11] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising\nimage classi\ufb01cation models and saliency maps. CoRR, abs/1312.6034, 2013. URL http://arxiv.org/\nabs/1312.6034.\n\n[12] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for\nsimplicity: The all convolutional net. CoRR, abs/1412.6806, 2014. URL http://arxiv.org/abs/1412.\n6806.\n\n[13] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow,\nand Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2013. URL http:\n//arxiv.org/abs/1312.6199.\n\n[14] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Du-\nmitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR,\nabs/1409.4842, 2014. URL http://arxiv.org/abs/1409.4842.\n\n[15] Matthew D. Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. CoRR,\n\nabs/1311.2901, 2013. URL http://arxiv.org/abs/1311.2901.\n\n[16] Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by\nexcitation backprop. 2016. URL https://www.robots.ox.ac.uk/~vgg/rg/papers/zhang_eccv16.\npdf.\n\n[17] Bolei Zhou, Aditya Khosla, \u00c0gata Lapedriza, Aude Oliva, and Antonio Torralba. Object Detectors Emerge\n\nin Deep Scene CNNs. CoRR, abs/1412.6856, 2014. URL http://arxiv.org/abs/1412.6856.\n\n[18] Bolei Zhou, Aditya Khosla, \u00c0gata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for\ndiscriminative localization. CoRR, abs/1512.04150, 2015. URL http://arxiv.org/abs/1512.04150.\n\n10\n\n\f", "award": [], "sourceid": 3492, "authors": [{"given_name": "Piotr", "family_name": "Dabkowski", "institution": "Cambridge University"}, {"given_name": "Yarin", "family_name": "Gal", "institution": "University of Oxford"}]}