{"title": "CXPlain: Causal Explanations for Model Interpretation under Uncertainty", "book": "Advances in Neural Information Processing Systems", "page_first": 10220, "page_last": 10230, "abstract": "Feature importance estimates that inform users about the degree to which given inputs influence the output of a predictive model are crucial for understanding, validating, and interpreting machine-learning models. However, providing fast and accurate estimates of feature importance for high-dimensional data, and quantifying the uncertainty of such estimates remain open challenges. Here, we frame the task of providing explanations for the decisions of machine-learning models as a causal learning task, and train causal explanation (CXPlain) models that learn to estimate to what degree certain inputs cause outputs in another machine-learning model. CXPlain can, once trained, be used to explain the target model in little time, and enables the quantification of the uncertainty associated with its feature importance estimates via bootstrap ensembling. We present experiments that demonstrate that CXPlain is significantly more accurate and faster than existing model-agnostic methods for estimating feature importance. In addition, we confirm that the uncertainty estimates provided by CXPlain ensembles are strongly correlated with their ability to accurately estimate feature importance on held-out data.", "full_text": "CXPlain: Causal Explanations for Model\n\nInterpretation under Uncertainty\n\nPatrick Schwab and Walter Karlen\n\nInstitute of Robotics and Intelligent Systems, ETH Zurich\n\npatrick.schwab@hest.ethz.ch\n\nAbstract\n\nFeature importance estimates that inform users about the degree to which given\ninputs in\ufb02uence the output of a predictive model are crucial for understanding,\nvalidating, and interpreting machine-learning models. However, providing fast and\naccurate estimates of feature importance for high-dimensional data, and quantifying\nthe uncertainty of such estimates remain open challenges. Here, we frame the task\nof providing explanations for the decisions of machine-learning models as a causal\nlearning task, and train causal explanation (CXPlain) models that learn to estimate\nto what degree certain inputs cause outputs in another machine-learning model.\nCXPlain can, once trained, be used to explain the target model in little time, and\nenables the quanti\ufb01cation of the uncertainty associated with its feature importance\nestimates via bootstrap ensembling. We present experiments that demonstrate that\nCXPlain is signi\ufb01cantly more accurate and faster than existing model-agnostic\nmethods for estimating feature importance.\nIn addition, we con\ufb01rm that the\nuncertainty estimates provided by CXPlain ensembles are strongly correlated with\ntheir ability to accurately estimate feature importance on held-out data.\n\n1\n\nIntroduction\n\nExplanation methods for machine-learning models play an important role in researching, developing,\nand using predictive models as information on what features were important for a given output enable\nus to better understand, validate, and interpret model decisions [1\u20135]. However, complex models,\nsuch as ensemble models and deep neural networks, are often dif\ufb01cult to interrogate. To address\nthis apparent dichotomy between performance and interpretability [6], researchers have developed a\nnumber of attribution methods that provide estimates of the importance of input features towards a\nmodel\u2019s output for speci\ufb01c types of models [4, 7\u201315], and for any machine-learning model [6, 16].\nHowever, providing fast and accurate feature importance estimates for any machine-learning model\nis challenging because there exists a wide variety of intricate machine-learning models with different\nunderlying model structures, algorithms, and decision functions, which makes it dif\ufb01cult to develop\nan optimised and uni\ufb01ed approach to importance attribution. Furthermore, importance estimates\nof state-of-the-art methods are typically associated with signi\ufb01cant uncertainty [3, 17\u201319], and it is\ntherefore dif\ufb01cult for users to judge when importance estimates can be expected to be accurate.\nIn this work, we present a new approach to estimating feature importance for any machine-learning\nmodel using causal explanation (CXPlain) models. CXPlain uses a causal objective to train a super-\nvised model to learn to explain another machine-learning model. This approach can be applied to any\nmachine-learning model, since it has no requirements on the predictive model to be explained. In par-\nticular, it does not require retraining or adapting the original model. We demonstrate experimentally\nthat CXPlain is signi\ufb01cantly more accurate than most existing methods, fast, and able to produce\naccurate uncertainty estimates. Source code is available at https://github.com/d909b/cxplain.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\festimate feature importance for any machine-learning model.\n\nContributions. This work contains the following contributions:\n\u2022 We introduce causal explanation (CXPlain) models, a new method for learning to accurately\n\u2022 We present a methodology based on bootstrap resampling for deriving uncertainty estimates for\n\u2022 Our experiments show that CXPlain is signi\ufb01cantly more accurate and signi\ufb01cantly faster (at\nevaluation time) than existing model-agnostic methods, and that the uncertainty estimates for\nits assigned feature importance scores are strongly correlated with the accuracy of the provided\nimportance scores on previously unseen test data.\n\nthe feature importance scores provided by CXPlain.\n\n2 Related Work\n\nFeature Importance Estimation. Existing methods for feature importance estimation can be\nsubdivided into (1) gradient-based methods, (2) methods based on sensitivity analysis, (3) methods\nthat measure the change in model con\ufb01dence when removing input features, and (4) mimic models.\nSimple Gradient (SG) [8], Integrated Gradients (IG) [10], DeepLIFT [1], and DeepSHAP [6] are\nexamples of gradient-based methods. Gradient-based methods are only applicable to differentiable\nmodels, such as neural networks, and their computation is typically fast. Methods that quantify a\nmodel\u2019s sensitivity to changes in the input, such as LIME [16] or SHAP [6], and more speci\ufb01cally\nKernel SHAP, are applicable to any machine-learning model but typically slow to compute, as large\nnumbers of model evaluations are necessary to assess a model\u2019s sensitivity. Methods based on\nmasking parts of the input and measuring the model\u2019s resulting change in con\ufb01dence [20] include\nconditional multivariate models for visualising deep neural networks [21], analysing the effects\nof erasing parts of their representations [22], image interpretation by identifying the regions for\nwhich the model most strongly responds to perturbations [23], and image masking models trained to\nmanipulate the outputs of a predictive model by occluding parts of the input [24]. The fourth main\ncategory of approaches to explaining model decisions is to train interpretable models that mimic the\ndecisions of a black-box model that we wish to explain. Tree- [25\u201327] and rule-based [28] models\nhave been used as mimic models. However, mimic models are not guaranteed to match the behavior\nof the original model. Besides these four established categories of feature importance estimation\nmethods, structural causal models (SCMs) [29] and Deep Taylor Decomposition (DTD) [30] have\nalso recently been proposed as explanation methods. However, these methods are designed for\nspeci\ufb01c types of models. In addition, the L2X method that uses a variational approximation of mutual\ninformation [31] and Bayesian nonparametrics [32] have been proposed to explain a target model.\nTsang et al. [33] detected statistical interactions by interpreting the weights learned in neural networks.\nBeyond feature attribution, testing with concept activation vectors (TCAV) [34] was proposed to\nvisualise the internal state of deep learning models, and in\ufb02uence functions [35] have been used\nto identify the training data most responsible for a given model decision. A major limitation of\nmost existing methods for feature importance estimation is that they do not inform users when their\nestimates are signi\ufb01cantly uncertain and can not be expected to be accurate.\n\nUncertainty and Reliability of Explanations. Although reliability is necessary for model expla-\nnations to be trustworthy, relatively few studies have been concerned with quantifying the uncertainty\nand robustness of explanation methods. For example, it has been shown that multiple importance\nestimation methods incorrectly attribute when a constant vector shift is applied to the input [3], that\nthe attributions provided by interpretation methods may themselves contain signi\ufb01cant uncertainty\n[18], that some explanation methods are independent of both the model and the data-generating\nprocess and, thus, can not be relied upon for important interpretation tasks [17], and that imperceptibly\nsmall perturbations of the input can signi\ufb01cantly alter the explanations provided by state-of-the-art\n\nTable 1: Comparison of CXPlain to several representative methods for feature importance estimation.\n\nAccuracy\nModel-agnostic\nUncertainty estimates\nComputation time\n\nCXPlain\nhigh\n\u0013\n\u0013\nfast\n\nSG [8] / IG [10] DeepSHAP [1, 6]\nhigh\n\u0015\n\u0015\nfast\n\nmoderate\n\u0015\n\u0015\nfast\n\nLIME [16]\nhigh\n\u0013\n\u0015\nslow\n\nSHAP [6]\nhigh\n\u0013\n\u0015\nslow\n\n2\n\n\fexplanation methods without changing the explained model\u2019s prediction [19]. These studies highlight\nthe importance of informing users when a given explanation is uncertain and should be discounted.\nIn contrast to existing works, CXPlain is an explanation model trained with a causal objective to\nlearn to explain the decisions of any machine-learning model without the need to retrain, adapt, or\nhave in-depth knowledge of the explained model. To the best of our knowledge, CXPlain is the \ufb01rst\nfeature importance estimation method that is simultaneously (1) signi\ufb01cantly more accurate than\nmost existing methods, (2) compatible with any machine-learning model and data modality, (3) able\nto provide uncertainty estimates via bootstrap resampling, and (4) fast at evaluation time (Table 1).\n\n3 Methodology\n\nProblem Setting. We consider a setting in which we are given a predictive model \u02c6f which processes\ninputs X consisting of p input features, or groups of features, xi with i \u2208 [0 . . p \u2212 1] to produce\noutputs \u02c6y \u2208 Rk of any dimensionality k. The predictive model \u02c6f is scored according to an objective\nfunction L : y \u00d7 \u02c6y \u2192 s that computes a scalar loss s \u2208 R after comparing the model\u2019s predictive\noutput \u02c6y to a ground-truth output y \u2208 Rk. The mean squared error (MSE) for regression models and\nthe categorical crossentropy for classi\ufb01cation models are commonly used examples of such objectives.\nWe note that we speci\ufb01cally do not require access to, or knowledge of, the process by which \u02c6f\nproduces its output, nor do we require \u02c6f to be differentiable or of any speci\ufb01c form. Additionally, we\nare given N \u2208 N independent and identically distributed (i.i.d.) pairs of sample covariates X and\nground-truth outputs y as training data. Given this setting, our goal is to train an explanation model\n\u02c6fexp that produces accurate estimates \u02c6A with elements \u02c6ai corresponding to the importances assigned\nto each of the p input features xi to the predictive model \u02c6f.\n\nCausal Explanations (CXPlain). The main idea be-\nhind CXPlain is to train a separate explanation model\n\u02c6fexp to explain the predictive model \u02c6f (Figure 1). This\n\ufb02exible framework has the advantage that we do not need\nto retrain or adapt the predictive model \u02c6f to explain its de-\ncisions. To train the explanation model, we utilise a causal\nobjective function that quanti\ufb01es the marginal contribution\nof either a single input feature or group of input features\ntowards the predictive model\u2019s accuracy [14, 20]. This\napproach, in essence, transforms the task of producing\nfeature importance estimates for a given predictive model\ninto a supervised learning task that we can address with existing supervised machine-learning models.\n\nFigure 1: CXPlain trains an explanation\nmodel \u02c6fexp (bottom) to learn to estimate\nimportance scores \u02c6A for a predictive tar-\nget model \u02c6f (top) given features X.\n\nCausal Objective. The core component of CXPlain is the causal objective that enables us to\noptimise explanation models to learn to explain another predictive model. The causal objective we\nbuild on was \ufb01rst introduced to jointly learn to produce accurate predictions and estimates of feature\nimportance in a single neural network model [14]. However, the original formulation of the causal\nobjective required a speci\ufb01c attentive mixture of experts architecture. In this work, we contribute\nan adapted version of the causal objective from [14] that does not require a speci\ufb01c model structure,\nand that can be used to train explanation models to learn to explain any machine-learning model.\nThe causal objective introduced in [14] was based on the Humean de\ufb01nition of causality used by\nGranger [36], who de\ufb01ned a causal relationship xi \u2192 \u02c6y between random variables xi and \u02c6y to\nexist if we are better able to predict \u02c6y using all available information than if the information apart\nfrom xi had been used [14]. i.e. if the absence of xi as a feature decreases our ability to predict\n\u02c6y. Granger [36]\u2019s de\ufb01nition of causality was based on two key assumptions: (1) That our set of\navailable variables X contains all relevant variables for the causal problem being modelled, and (2)\nthat xi temporally precedes \u02c6y [36]. In the general setting, these assumptions can not be veri\ufb01ed from\nobservational data [37]. However, in our speci\ufb01c setting, we know a priori that the inputs of the\npredictive model \u02c6f mathematically always precede its output, and that the explained model\u2019s output,\non deterministic hardware and software, is not in\ufb02uenced by variables other than those present in its\nset of input features. We can therefore use the given de\ufb01nition to quantify the degree to which an\ninput feature caused a marginal improvement in the predictive performance of the predictive model\n\n3\n\nXy^f^predictive modelfexp^explanation modelAinputmodeloutput^\f(1)\n\n\u02c6yX\\{i} = \u02c6f (X \\ {i})\n\n\u02c6f. Given input covariates X, we therefore denote \u03b5X\\{i} as the predictive model\u2019s error without\nincluding any information from the ith input feature and \u03b5X as the predictive model\u2019s error when\nconsidering all available input features. To calculate \u03b5X\\{i} and \u03b5X, we \ufb01rst compute the outputs\n\u02c6yX\\{i} and \u02c6yX of the predictive model \u02c6f without and with the ith input feature xi, respectively:\n(2)\nThere are several different approaches to obtaining X \\ {i} from the full set of input features,\ndepending on the type of input data. For most types of data, masking the respective input feature xi at\nindex i with zeroes, when the zero value has no special meaning, or replacing it with the mean value\nacross the entire data set are both valid choices [20, 21, 24]. More sophisticated feature masking\nschemes that consider the masked feature\u2019s distribution [38, 39] could be a more principled alternative\nto masking with point-wise estimates. Given X \\ {i}, we compare the predictions \u02c6yX\\{i} and \u02c6yX\nwith the ground-truth labels y using the predictive model\u2019s loss function L to calculate \u03b5X\\{i} and \u03b5X:\n(4)\nFollowing Granger [36]\u2019s de\ufb01nition of causality, we de\ufb01ne the degree \u2206\u03b5i to which the ith input\nfeature causally contributed to the predictive model\u2019s output \u02c6y as the decrease in error, as measured\nby its loss L, associated with adding that feature to the set of available information sources [14]:\n\n\u03b5X\\{i} = L(y, \u02c6yX\\{i})\n\n\u03b5X = L(y, \u02c6yX )\n\n\u02c6yX = \u02c6f (X)\n\n(3)\n\n\u2206\u03b5X,i = \u03b5X\\{i} \u2212 \u03b5X\n\n\u2206\u03b5X,i\nj=0 \u2206\u03b5X,j\n\n(cid:80)p\u22121\n(cid:80)N\u22121\n\n(5)\nLastly, we normalise the importance scores \u03c9i to relative contributions \u2208 [0, 1] with \u03a3i\u03c9i = 1 [14]:\n(6)\n\n\u03c9i(X) =\nWe then arrive at our causal objective Lcausal = 1\nl=0 KL(\u2126Xl , \u02c6AXl ) [14] that aims to minimise\nthe Kullback-Leibler (KL) divergence [40] between the target importance distribution \u2126 with \u2126(i) =\n\u03c9i(X) for a given sample X, and the distribution of importance scores \u02c6A with \u02c6A(i) = \u02c6ai as estimated\nby \u02c6fexp based on X. Using Lcausal, we can train supervised learning models to learn to explain\nany other machine-learning model based solely on its outputs, and without the need to retrain the\nmodel to be explained. Precomputing the importances \u2126 for each training sample X takes N (p + 1)\nevaluations of the target predictive model at training time. For high-dimensional images, it is sensible\nto group non-overlapping regions of adjacent pixels into feature groups, since removing single pixels\nin high-dimensional images is unlikely to strongly affect a predictive model\u2019s output [21]. This also\nsigni\ufb01cantly limits the number of feature groups p for which importances \u03c9i have to be precomputed.\nWe note that estimating \u02c6A is not necessary in situations in which ground truth labels are readily\navailable, e.g. during model development. In those situations, \u2126 can directly be used to explain \u02c6f.\n\nN\n\nExplanation Models.\nIn principle, any supervised machine learning model that can be trained with\na custom objective could be used as a causal explanation model. In this work, we focus on neural\nexplanation models. Using deep neural networks as causal explanation models has the advantage\nthat these models are able to extract high-level feature representations from high-dimensional and\nunstructured data [41], and thus remove the need to perform manual feature engineering. We leave\nthe exploration of other classes of explanation models to future work. A priori, it is not clear which\narchitectures would be most suitable to be used in neural explanation models. Absent any prior\nknowledge about the structure of the input data, multilayer perceptrons (MLPs) are likely a sensible\ndefault choice. However, since architectures that exploit the spatial or temporal structure of input\ndata have been shown to be ef\ufb01cacious, we reason that, depending on the data modality of the input\nfeatures of the model to be explained, special-purpose architectures, such as convolutional neural\nnetworks [42] for images and attentive neural networks for texts [43], could perform better than\nMLPs. In particular, U-nets [44] that have been designed for image segmentation, a task that involves\nmapping input pixels to segmentation labels, may perform well as causal explanation models for\nimages since segmentation is semantically similar to explanation, which involves mapping input\npixels to importance scores. To determine whether or not specialised model architectures can achieve\nbetter performances in neural explanation models, we experimentally evaluate both MLPs and U-nets.\n\nUncertainty of Importance Estimates.\nIn addition to producing accurate estimates of feature\nimportance, we wish to provide uncertainty estimates ui that quantify the uncertainty associated\nwith each individual feature importance estimate \u02c6ai produced by a CXPlain model. In particular, we\n\n4\n\n\fFigure 2: Comparison of the distributions of the\nchanges in log odds \u2206log-odds after masking the\ntop 10% most important pixels according to sev-\neral feature importance estimation methods across\nN = 100 MNIST test images (higher is better).\n*** = signi\ufb01cantly different (p < 0.001, MWW).\n\nFigure 3: Comparison of the distributions of the\nchanges in log odds \u2206log-odds after masking the\ntop 30% most important pixels according to sev-\neral feature importance estimation methods across\nN = 100 test ImageNet images (higher is better).\n** = signi\ufb01cantly different (p < 0.01, MWW).\n\n2\n\n2\n\n, ci,1\u2212 \u03b1\n\n2\n\n2\n\n2\n\n\u2212 ci, \u03b1\n\n2\n\n] with lower bounds ci, \u03b1\n\nand upper\nwould like to calculate con\ufb01dence intervals CIi,\u03b3 = [ci, \u03b1\nat con\ufb01dence level \u03b3 = 1 \u2212 \u03b1 for each assigned feature importance estimate \u02c6ai. The\nbounds ci,1\u2212 \u03b1\nwidth ui = ci,1\u2212 \u03b1\nof CIi,\u03b3 can subsequently be used to quantify the uncertainty of \u02c6ai. To\nderive uncertainty estimates for causal explanation models, we propose the use of bootstrap ensemble\nmethods, speci\ufb01cally using bootstrap resampling [45, 46]. To train bootstrap ensembles of causal\nexplanation models, we \ufb01rst draw N training samples X at random with repeats from the original\ntraining set. We then train an explanation model using the before-mentioned causal objective until\nconvergence on the selected subset of the training set. We repeat this process M times to obtain a\nbootstrap ensemble of M explanation models (Algorithm in Appendix B). We use the median of the\nattributions \u02c6ai of the ensemble members as the assigned importance of the bootstrap ensemble, and\nthe \u03b1\n2 quantiles as lower and upper bounds of its CI, respectively. The ef\ufb01cacy of bootstrap\nensembles for estimating the uncertainty in outputs of neural networks has been demonstrated in, e.g.,\n[47], but this work is, to the best of our knowledge, the \ufb01rst to consider using bootstrap ensembles of\nexplanation models to quantify the uncertainty in assigned importance scores. We note that Monte\nCarlo dropout [48], which uses dropout [49] at evaluation time, is an alternative method for estimating\nuncertainty for the outputs of neural networks that does not require explicitly training an ensemble of\nmodels, but may not always produce uncertainty estimates of the same quality as ensembles [47].\n\n2 and 1\u2212 \u03b1\n\n4 Experiments\n\nOur experiments aimed to answer the following questions:\n1 How does the feature importance estimation performance of CXPlain compare to that of existing\n\nstate-of-the-art methods?\n\n2 How does the computational performance of CXPlain compare to existing model-agnostic and\n\nmodel-speci\ufb01c methods for feature importance estimation?\n\n3 Are uncertainty estimates computed via bootstrap resampling of CXPlain models qualitatively and\n\nquantitatively correlated with their ability to accurately determine feature importance?\n\nTo answer these questions, we performed extensive experiments on several benchmarks that compare\nboth the computational as well as the estimation performance of CXPlain to existing state-of-the-art\nmethods for feature importance estimation. To enable a meaningful comparison, we focus most of\nour experiments on image classi\ufb01cation tasks, where we are best able to visualise and quantify the\nperformance of feature importance estimation methods, and on neural network models as models to\nbe explained, since most existing model-speci\ufb01c attribution methods that we wish to compare to were\ndeveloped exclusively for neural networks. However, we note that CXPlain as a method is compatible\nwith any machine-learning model, data modality, and both regression as well as classi\ufb01cation tasks.\nWe used Mann\u2013Whitney\u2013Wilcoxon (MWW) tests [50] to calculate p-values for the main comparisons.\n\n4.1 Determining Important Features in MNIST and ImageNet\n\nTo compare the accuracy of CXPlain to existing state-of-the-art methods for feature importance\nestimation, we evaluated its ability to identify important features in MNIST [51] and ImageNet [52]\nimages. To do so, we followed the experimental design \ufb01rst proposed by Shrikumar et al. [1], and\n\n5\n\n***0102030IntegratedGradientsSimpleGradientSHAPLIMECXPlain(MLP)Deep\u2212SHAPCXPlain(U\u2212net)D log oddsMNIST**\u22125.0\u22122.50.02.5RandomIntegratedGradientsSimpleGradientSHAPDeep\u2212SHAPLIMECXPlain(U\u2212net)D log oddsImageNet\fFigure 4: A comparison of the top 10% most im-\nportant pixels (= Mask) as identi\ufb01ed by CXPlain\n(U-net), DeepSHAP, SHAP, and LIME on the\nsame sample test set image (Source) of the 8 vs. 3\nMNIST benchmark. With accurate estimates, the\nMasked image should more closely resemble a 3\nthan an 8, since the pixels that most distinguished\nan 8 as an 8 should have been removed.\n\nFigure 5: A comparison of the feature importance\nscores (= Attribution) as estimated by CXPlain\n(U-net), SHAP, and LIME on the same sample test\nset image (Source) of the Gorilla vs. Zebra Ima-\ngeNet benchmark. We found that CXPlain (U-net)\nproduces attribution maps that are, subjectively\nand qualitatively, more semantically focused on\nthe most salient regions of the image.\n\ntrained binary classi\ufb01cation models to distinguish between two digit types (8 vs. 3) on MNIST (model\naccuracy: 99.85%), and two object categories (Gorilla vs. Zebra) on ImageNet (model accuracy:\n96.73%). As a preprocessing step, pixel values were scaled to be in the range of [0, 1] prior to training.\nWe then used several importance estimation methods to determine which input pixels were most\nimportant for the classi\ufb01cation models\u2019 decisions on N = 100 test images. We masked the top 10\nand 30% of those most important pixels for MNIST and ImageNet, respectively, and measured the\nresulting change in the classi\ufb01cation models\u2019 con\ufb01dences by computing the difference in log odds\n\n\u2206log-odds = log-odds(poriginal) \u2212 log-odds(pmasked)\n\n(7)\n1\u2212p ), and poriginal and pmasked are the classi\ufb01cation models\u2019 outputs p \u2208\nwhere log-odds(p) = log( p\n[0, 1] for the original image and the masked image with the top pixels removed, respectively. To\nensure that the explanations ei of all methods are on the same scale, we normalised them to the range\nof [0, 1] using the transformation \u02c6ai = |ei|/\u03a3N\ni=0|ei|. We plotted the assigned importances and the\nresulting masked images to qualitatively assess each methods\u2019 ability to determine the salient features\nin the original image (Figures 4 and 5). We additionally recorded the mean and standard deviation of\nthe time taken (in seconds) to compute the feature importance estimates for each method on the same\nhardware (Appendix C) over 10 and 5 runs with the same parameters and random seed for MNIST\nand ImageNet, respectively (Figures 6 and 7). Further training details are given in Appendix A.\n\n4.2 Quantifying Uncertainty in Estimates of Feature Importance\n\nTo quantitatively and qualitatively assess the accuracy of the uncertainty estimates provided by\nbootstrap ensembles of CXPlain models, we analysed whether their uncertainty estimates ui are\ncorrelated with their errors in feature importance estimation on held-out MNIST test samples. We\nevaluated several numbers M of bootstrap resampled models in order to determine how the number of\nensemble members affects the uncertainty estimation performance of bootstrap ensembles of CXPlain\nmodels. In addition, we also evaluated the performance of randomly selected uncertainty estimates\nas a baseline for comparison. In general settings, it is dif\ufb01cult to evaluate uncertainty estimates\nfor feature importance estimation methods, since we typically do not have per-feature ground-truth\nattributions to evaluate against. However, by comparing the ranking implied by the ground-truth\nchange in log-odds to the ranking implied by the explanation model we are able to de\ufb01ne a rank error\nREi for each xi. Formally, the rank error REi = |rank\u2206log-odds(i) \u2212 rank \u02c6fexp\n(i)| is the difference in\nrank between the true rank\u2206log-odds implied by \u2206log-odds, and the estimated rank \u02c6fexp\nimplied by the\nexplanation model, where rankb(i) de\ufb01nes the rank of xi from 0 to p \u2212 1 implied by b.\nAs correlation metric, we used Pearson\u2019s \u03c1 to measure the correlation between the rank error REi and\nthe uncertainty estimates ui = ci,95% \u2212 ci,5% de\ufb01ned by the bootstrap resampled \u03b3 = 90% CIs for\neach importance estimate \u02c6ai in the top 2.5% of pixels by \u2206log-odds across N = 100 unseen images\nfrom the MNIST test set. We limited the evaluation to all pixels with a \u2206log-odds greater than 0.\n\n6\n\nSourceMaskedMaskssh://d909b@ssh.schwabpatrick.com:909/usr/bin/python -u /home/d909b/bin/causal_explanations/causal_explanations/apps/main.py --output_directory=/home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty --load_existing=/home/d909b/models/cex_main_1/model.npz --load_existing_cxplain=/home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npz --dataset=mnist --num_epochs=50 --num_units=75 --batch_size=100 --num_layers=2 --dropout=0.0 --learning_rate=0.001 --l2_weight=0.000 --model_type=resnet --explanation_type=cxplain --do_not_save_attributions --source_digit=8 --target_digit=3 --attack_method=random --attack_epsilon=1 --attack_num_samples=100 --defence_method=none --discrete_attack_type=lsga --do_not_calculate_log_odds --do_not_calculate_robustness --num_explanation_samples=1000 --num_boostrap_samples=5Using TensorFlow backend./usr/local/lib/python2.7/dist-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release. from numpy.core.umath_tests import inner1dINFO: Args are: {'num_boostrap_samples': 5, 'do_adversarial_training': False, 'n_jobs': 4, 'explanation_type': 'cxplain', 'calculate_uncertainties': True, 'num_adversarial_samples': 1000, 'hyperopt_against_eval_set': False, 'num_epochs': 50, 'explanation_model_name': 'explanation.npz', 'copy_to_local': False, 'attack_iterations': 100, 'seed': 909, 'precomputed_attributions_train': '', 'num_layers': 2, 'fraction_of_data_set': 1, 'precomputed_attributions_val': '', 'calculate_log_odds': False, 'do_evaluate': False, 'with_tensorboard': False, 'do_augment': False, 'with_bn': False, 'attack_num_samples': 100, 'do_hyperopt': False, 'attack_epsilon': 1, 'save_attributions': False, 'save_predictions': True, 'target_digit': 3, 'load_existing': '/home/d909b/models/cex_main_1/model.npz', 'source_digit': 8, 'calculate_robustness': False, 'discrete_attack_type': 'lsga', 'num_hyperopt_runs': 35, 'thermometer_encoding_levels': 16, 'learning_rate': 0.001, 'batch_size': 100, 'output_directory': '/home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty', 'load_existing_cxplain': '/home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npz', 'do_merge_lsf': False, 'slic_compactness': 10, 'do_hyperopt_on_lsf': False, 'l2_weight': 0.0, 'hyperopt_o(cid:31)set': 0, 'dataset': 'mnist', 'dropout': 0.0, 'defence_method': 'none', 'num_gpus': 1, 'imagenet_folder': '/home/d909b/backup/imagenettrain-prep/', 'attack_method': 'random', 'num_layers_cxplain': -1, 'early_stopping_patience': 12, 'do_train': False, 'num_explanation_samples': 1000, 'num_units': 75, 'test_set_fraction': 0.2, 'model_type': 'resnet', 'validation_set_fraction': 0.2, 'num_units_cxplain': -1, 'model_name': 'model.npz'}INFO: Running at 2019-05-10 16:48:02.083797INFO: Seed is 9092019-05-10 16:48:02.092153: I tensor(cid:30)ow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA2019-05-10 16:48:02.200813: I tensor(cid:30)ow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero2019-05-10 16:48:02.201281: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582pciBusID: 0000:01:00.0totalMemory: 11.90GiB freeMemory: 11.55GiB2019-05-10 16:48:02.201296: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 02019-05-10 16:48:02.396536: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:2019-05-10 16:48:02.396563: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:988] 0 2019-05-10 16:48:02.396569: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2019-05-10 16:48:02.396735: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11180 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:01:00.0, compute capability: 6.1)INFO: Loading MNIST data.INFO: Run with args: {'num_boostrap_samples': 5, 'do_adversarial_training': False, 'n_jobs': 4, 'explanation_type': 'cxplain', 'calculate_uncertainties': True, 'num_adversarial_samples': 1000, 'hyperopt_against_eval_set': False, 'num_epochs': 50, 'explanation_model_name': 'explanation.npz', 'copy_to_local': False, 'attack_iterations': 100, 'seed': 909, 'precomputed_attributions_train': '', 'num_layers': 2, 'fraction_of_data_set': 1, 'precomputed_attributions_val': '', 'calculate_log_odds': False, 'do_evaluate': False, 'with_tensorboard': False, 'do_augment': False, 'with_bn': False, 'attack_num_samples': 100, 'do_hyperopt': False, 'attack_epsilon': 1, 'save_attributions': False, 'save_predictions': True, 'target_digit': 3, 'load_existing': '/home/d909b/models/cex_main_1/model.npz', 'source_digit': 8, 'calculate_robustness': False, 'discrete_attack_type': 'lsga', 'num_hyperopt_runs': 35, 'thermometer_encoding_levels': 16, 'learning_rate': 0.001, 'batch_size': 100, 'output_directory': '/home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty', 'load_existing_cxplain': '/home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npz', 'do_merge_lsf': False, 'slic_compactness': 10, 'do_hyperopt_on_lsf': False, 'l2_weight': 0.0, 'hyperopt_o(cid:31)set': 0, 'dataset': 'mnist', 'dropout': 0.0, 'defence_method': 'none', 'num_gpus': 1, 'imagenet_folder': '/home/d909b/backup/imagenettrain-prep/', 'attack_method': 'random', 'num_layers_cxplain': -1, 'early_stopping_patience': 12, 'do_train': False, 'num_explanation_samples': 1000, 'num_units': 75, 'test_set_fraction': 0.2, 'model_type': 'resnet', 'validation_set_fraction': 0.2, 'num_units_cxplain': -1, 'model_name': 'model.npz'}INFO: Loaded generator with 11982 samples. Doing 120 steps of size 100INFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Built generators with 120 training samples, 20 validation samples and 20 test samples.INFO: Started training feature extraction.__________________________________________________________________________________________________Layer (type) Output Shape Param # Connected to ==================================================================================================input_1 (InputLayer) (None, 28, 28, 1) 0 __________________________________________________________________________________________________conv2d_1 (Conv2D) (None, 28, 28, 16) 160 input_1[0][0] __________________________________________________________________________________________________activation_1 (Activation) (None, 28, 28, 16) 0 conv2d_1[0][0] __________________________________________________________________________________________________conv2d_2 (Conv2D) (None, 28, 28, 16) 2320 activation_1[0][0] __________________________________________________________________________________________________activation_2 (Activation) (None, 28, 28, 16) 0 conv2d_2[0][0] __________________________________________________________________________________________________conv2d_3 (Conv2D) (None, 28, 28, 16) 2320 activation_2[0][0] __________________________________________________________________________________________________activation_3 (Activation) (None, 28, 28, 16) 0 conv2d_3[0][0] __________________________________________________________________________________________________add_1 (Add) (None, 28, 28, 16) 0 activation_1[0][0] activation_3[0][0] __________________________________________________________________________________________________activation_4 (Activation) (None, 28, 28, 16) 0 add_1[0][0] __________________________________________________________________________________________________conv2d_4 (Conv2D) (None, 28, 28, 16) 2320 activation_4[0][0] __________________________________________________________________________________________________activation_5 (Activation) (None, 28, 28, 16) 0 conv2d_4[0][0] __________________________________________________________________________________________________conv2d_5 (Conv2D) (None, 28, 28, 16) 2320 activation_5[0][0] __________________________________________________________________________________________________activation_6 (Activation) (None, 28, 28, 16) 0 conv2d_5[0][0] __________________________________________________________________________________________________add_2 (Add) (None, 28, 28, 16) 0 activation_4[0][0] activation_6[0][0] __________________________________________________________________________________________________activation_7 (Activation) (None, 28, 28, 16) 0 add_2[0][0] __________________________________________________________________________________________________conv2d_6 (Conv2D) (None, 28, 28, 16) 2320 activation_7[0][0] __________________________________________________________________________________________________activation_8 (Activation) (None, 28, 28, 16) 0 conv2d_6[0][0] __________________________________________________________________________________________________conv2d_7 (Conv2D) (None, 28, 28, 16) 2320 activation_8[0][0] __________________________________________________________________________________________________activation_9 (Activation) (None, 28, 28, 16) 0 conv2d_7[0][0] __________________________________________________________________________________________________add_3 (Add) (None, 28, 28, 16) 0 activation_7[0][0] activation_9[0][0] __________________________________________________________________________________________________activation_10 (Activation) (None, 28, 28, 16) 0 add_3[0][0] __________________________________________________________________________________________________conv2d_8 (Conv2D) (None, 14, 14, 32) 4640 activation_10[0][0] __________________________________________________________________________________________________activation_11 (Activation) (None, 14, 14, 32) 0 conv2d_8[0][0] __________________________________________________________________________________________________conv2d_10 (Conv2D) (None, 14, 14, 32) 544 activation_10[0][0] __________________________________________________________________________________________________conv2d_9 (Conv2D) (None, 14, 14, 32) 9248 activation_11[0][0] __________________________________________________________________________________________________activation_13 (Activation) (None, 14, 14, 32) 0 conv2d_10[0][0] __________________________________________________________________________________________________activation_12 (Activation) (None, 14, 14, 32) 0 conv2d_9[0][0] __________________________________________________________________________________________________add_4 (Add) (None, 14, 14, 32) 0 activation_13[0][0] activation_12[0][0] __________________________________________________________________________________________________activation_14 (Activation) (None, 14, 14, 32) 0 add_4[0][0] __________________________________________________________________________________________________conv2d_11 (Conv2D) (None, 14, 14, 32) 9248 activation_14[0][0] __________________________________________________________________________________________________activation_15 (Activation) (None, 14, 14, 32) 0 conv2d_11[0][0] __________________________________________________________________________________________________conv2d_12 (Conv2D) (None, 14, 14, 32) 9248 activation_15[0][0] __________________________________________________________________________________________________activation_16 (Activation) (None, 14, 14, 32) 0 conv2d_12[0][0] __________________________________________________________________________________________________add_5 (Add) (None, 14, 14, 32) 0 activation_14[0][0] activation_16[0][0] __________________________________________________________________________________________________activation_17 (Activation) (None, 14, 14, 32) 0 add_5[0][0] __________________________________________________________________________________________________conv2d_13 (Conv2D) (None, 14, 14, 32) 9248 activation_17[0][0] __________________________________________________________________________________________________activation_18 (Activation) (None, 14, 14, 32) 0 conv2d_13[0][0] __________________________________________________________________________________________________conv2d_14 (Conv2D) (None, 14, 14, 32) 9248 activation_18[0][0] __________________________________________________________________________________________________activation_19 (Activation) (None, 14, 14, 32) 0 conv2d_14[0][0] __________________________________________________________________________________________________add_6 (Add) (None, 14, 14, 32) 0 activation_17[0][0] activation_19[0][0] __________________________________________________________________________________________________activation_20 (Activation) (None, 14, 14, 32) 0 add_6[0][0] __________________________________________________________________________________________________conv2d_15 (Conv2D) (None, 7, 7, 64) 18496 activation_20[0][0] __________________________________________________________________________________________________activation_21 (Activation) (None, 7, 7, 64) 0 conv2d_15[0][0] __________________________________________________________________________________________________conv2d_17 (Conv2D) (None, 7, 7, 64) 2112 activation_20[0][0] __________________________________________________________________________________________________conv2d_16 (Conv2D) (None, 7, 7, 64) 36928 activation_21[0][0] __________________________________________________________________________________________________activation_23 (Activation) (None, 7, 7, 64) 0 conv2d_17[0][0] __________________________________________________________________________________________________activation_22 (Activation) (None, 7, 7, 64) 0 conv2d_16[0][0] __________________________________________________________________________________________________add_7 (Add) (None, 7, 7, 64) 0 activation_23[0][0] activation_22[0][0] __________________________________________________________________________________________________activation_24 (Activation) (None, 7, 7, 64) 0 add_7[0][0] __________________________________________________________________________________________________conv2d_18 (Conv2D) (None, 7, 7, 64) 36928 activation_24[0][0] __________________________________________________________________________________________________activation_25 (Activation) (None, 7, 7, 64) 0 conv2d_18[0][0] __________________________________________________________________________________________________conv2d_19 (Conv2D) (None, 7, 7, 64) 36928 activation_25[0][0] __________________________________________________________________________________________________activation_26 (Activation) (None, 7, 7, 64) 0 conv2d_19[0][0] __________________________________________________________________________________________________add_8 (Add) (None, 7, 7, 64) 0 activation_24[0][0] activation_26[0][0] __________________________________________________________________________________________________activation_27 (Activation) (None, 7, 7, 64) 0 add_8[0][0] __________________________________________________________________________________________________conv2d_20 (Conv2D) (None, 7, 7, 64) 36928 activation_27[0][0] __________________________________________________________________________________________________activation_28 (Activation) (None, 7, 7, 64) 0 conv2d_20[0][0] __________________________________________________________________________________________________conv2d_21 (Conv2D) (None, 7, 7, 64) 36928 activation_28[0][0] __________________________________________________________________________________________________activation_29 (Activation) (None, 7, 7, 64) 0 conv2d_21[0][0] __________________________________________________________________________________________________add_9 (Add) (None, 7, 7, 64) 0 activation_27[0][0] activation_29[0][0] __________________________________________________________________________________________________activation_30 (Activation) (None, 7, 7, 64) 0 add_9[0][0] __________________________________________________________________________________________________average_pooling2d_1 (AveragePoo (None, 1, 1, 64) 0 activation_30[0][0] __________________________________________________________________________________________________(cid:30)atten_1 (Flatten) (None, 64) 0 average_pooling2d_1[0][0] __________________________________________________________________________________________________dense_1 (Dense) (None, 2) 130 (cid:30)atten_1[0][0] __________________________________________________________________________________________________activation_31 (Activation) (None, 2) 0 dense_1[0][0] ==================================================================================================Total params: 270,882Trainable params: 270,882Non-trainable params: 0__________________________________________________________________________________________________INFO: Loading existing model from /home/d909b/models/cex_main_1/model.npz_________________________________________________________________Layer (type) Output Shape Param # =================================================================input_3 (InputLayer) (None, 28, 28, 1) 0 _________________________________________________________________(cid:30)atten_3 (Flatten) (None, 784) 0 _________________________________________________________________dense_3 (Dense) (None, 75) 58875 _________________________________________________________________activation_63 (Activation) (None, 75) 0 _________________________________________________________________dense_4 (Dense) (None, 75) 5700 _________________________________________________________________activation_64 (Activation) (None, 75) 0 _________________________________________________________________dense_5 (Dense) (None, 196) 14896 _________________________________________________________________activation_65 (Activation) (None, 196) 0 _________________________________________________________________reshape_1 (Reshape) (None, 14, 14, 1) 0 _________________________________________________________________lambda_2 (Lambda) (None, 28, 28, 1) 0 _________________________________________________________________reshape_2 (Reshape) (None, 784) 0 =================================================================Total params: 79,471Trainable params: 79,471Non-trainable params: 0_________________________________________________________________INFO: Loading existing CXPlain model from /home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npzbuild_explanation_model : took 1.81394100189 seconds.INFO: Saving loss history to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/losses.pickleINFO: Saving model predictions.INFO: Loaded generator with 11982 samples. Doing 120 steps of size 100INFO: Saved model predictions to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/train_predictions.csvINFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Saved model predictions to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/val_predictions.csvINFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Saved model predictions to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/test_predictions.csvINFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Performance on test AUROC (weighted) = 0.999994917356212 , with AUPRC (weighted) = 0.9999949280930339 , with r^2 (weighted) = 0.9957963109416453 , with f1 (weighted) = 0.9984879166751001 , with accuracy = 0.9984879032258065 , with error = 0.0015120967741935054INFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Performance on test [n= 100 ] targets = [0.9] observed = [(1.1436718599626305, '+-', 0.5520538782615332)]INFO: Performance on test [n= 100 ] for target 0.9observed = [1.3892620290955573, 1.1377237947530967, 0.8350603218815686, 0.8713871427716444, 1.8772789577296631, 1.0421385269639882, 1.2734869085839489, 0.5600630255113783, 1.2435160902712874, 2.0858731684436953, 0.6621914099533671, 0.6070763678776874, 0.8661585671984602, 1.0517880297070337, 2.9146049149183884, 0.6235869496235044, 0.9226209148366383, 0.896227432532063, 0.2670657996896821, 0.013821244358016503, 2.189807259704378, 0.8175433055816894, 1.19488319718703, 1.8930241978639137, 0.7730931068420339, 1.27811588137019, 0.9850174716812721, 1.1216892414631758, 0.573120152159161, 1.0232491128492835, 0.8987605463776784, 1.037649214136879, 0.5118367176339356, 2.083070000264211, 1.302903762457773, 0.8204965828874214, 1.6789029079041142, 1.623794531947206, 1.2545555949371687, 1.0012906768156877, 1.2462736163330106, 0.9057037790232868, 0.9469323067092333, 0.4383414096479612, 0.8104805348408998, 0.8851755920050628, 1.132267732268079, 1.6621074239888216, 0.934013287760548, 0.8349687929106586, 0.5289153327173624, 1.4440371769826026, 2.138591203347149, 1.2270851493051227, 2.142648193073443, 0.3385335850217482, 0.46770572775341435, 0.8527171407093351, 0.47953222713471033, 1.8026274831409246, 1.1812909273907717, 0.9314364881634934, 0.2302455728277166, 1.5633736254880761, 2.3031484613717335, 2.4174470115754003, 1.1201235152098241, 2.266265571565306, 1.3994960435747492, 0.7860688071695486, 1.2776682986304249, 1.6580154383934487, 0.757406483017344, 1.0043128902083016, 0.7231292579067669, 0.740787741347223, 0.7141079559712596, 0.6734593179701929, 0.6616112633542277, 0.9468177183380312, 0.7690791699258368, 1.0155653037702255, 0.6512117579059203, 0.9884971489035145, 0.6675223757172929, 1.7234492486644164, 0.7886172116728509, 1.1807222822302033, 0.8768496962883034, 1.2127882377541048, 1.0925325778269959, 0.5464445929844274, 2.096737223622148, 0.8172319369500524, 1.8495334324613373, 1.2724899666210199, 1.0701363404910589, 1.8738202234307906, 2.195503699637992, 1.8998484264924382]Process (cid:29)nished with exit code 08 \u2192 3DeepSHAPSHAPCXPlain (U-net)LIMESourceMaskedAttributionssh://d909b@ssh.schwabpatrick.com:909/usr/bin/python -u /home/d909b/bin/causal_explanations/causal_explanations/apps/main.py --output_directory=/home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty --load_existing=/home/d909b/models/cex_main_1/model.npz --load_existing_cxplain=/home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npz --dataset=mnist --num_epochs=50 --num_units=75 --batch_size=100 --num_layers=2 --dropout=0.0 --learning_rate=0.001 --l2_weight=0.000 --model_type=resnet --explanation_type=cxplain --do_not_save_attributions --source_digit=8 --target_digit=3 --attack_method=random --attack_epsilon=1 --attack_num_samples=100 --defence_method=none --discrete_attack_type=lsga --do_not_calculate_log_odds --do_not_calculate_robustness --num_explanation_samples=1000 --num_boostrap_samples=5Using TensorFlow backend./usr/local/lib/python2.7/dist-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release. from numpy.core.umath_tests import inner1dINFO: Args are: {'num_boostrap_samples': 5, 'do_adversarial_training': False, 'n_jobs': 4, 'explanation_type': 'cxplain', 'calculate_uncertainties': True, 'num_adversarial_samples': 1000, 'hyperopt_against_eval_set': False, 'num_epochs': 50, 'explanation_model_name': 'explanation.npz', 'copy_to_local': False, 'attack_iterations': 100, 'seed': 909, 'precomputed_attributions_train': '', 'num_layers': 2, 'fraction_of_data_set': 1, 'precomputed_attributions_val': '', 'calculate_log_odds': False, 'do_evaluate': False, 'with_tensorboard': False, 'do_augment': False, 'with_bn': False, 'attack_num_samples': 100, 'do_hyperopt': False, 'attack_epsilon': 1, 'save_attributions': False, 'save_predictions': True, 'target_digit': 3, 'load_existing': '/home/d909b/models/cex_main_1/model.npz', 'source_digit': 8, 'calculate_robustness': False, 'discrete_attack_type': 'lsga', 'num_hyperopt_runs': 35, 'thermometer_encoding_levels': 16, 'learning_rate': 0.001, 'batch_size': 100, 'output_directory': '/home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty', 'load_existing_cxplain': '/home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npz', 'do_merge_lsf': False, 'slic_compactness': 10, 'do_hyperopt_on_lsf': False, 'l2_weight': 0.0, 'hyperopt_o(cid:31)set': 0, 'dataset': 'mnist', 'dropout': 0.0, 'defence_method': 'none', 'num_gpus': 1, 'imagenet_folder': '/home/d909b/backup/imagenettrain-prep/', 'attack_method': 'random', 'num_layers_cxplain': -1, 'early_stopping_patience': 12, 'do_train': False, 'num_explanation_samples': 1000, 'num_units': 75, 'test_set_fraction': 0.2, 'model_type': 'resnet', 'validation_set_fraction': 0.2, 'num_units_cxplain': -1, 'model_name': 'model.npz'}INFO: Running at 2019-05-10 16:48:02.083797INFO: Seed is 9092019-05-10 16:48:02.092153: I tensor(cid:30)ow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA2019-05-10 16:48:02.200813: I tensor(cid:30)ow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero2019-05-10 16:48:02.201281: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582pciBusID: 0000:01:00.0totalMemory: 11.90GiB freeMemory: 11.55GiB2019-05-10 16:48:02.201296: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 02019-05-10 16:48:02.396536: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:2019-05-10 16:48:02.396563: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:988] 0 2019-05-10 16:48:02.396569: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2019-05-10 16:48:02.396735: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11180 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:01:00.0, compute capability: 6.1)INFO: Loading MNIST data.INFO: Run with args: {'num_boostrap_samples': 5, 'do_adversarial_training': False, 'n_jobs': 4, 'explanation_type': 'cxplain', 'calculate_uncertainties': True, 'num_adversarial_samples': 1000, 'hyperopt_against_eval_set': False, 'num_epochs': 50, 'explanation_model_name': 'explanation.npz', 'copy_to_local': False, 'attack_iterations': 100, 'seed': 909, 'precomputed_attributions_train': '', 'num_layers': 2, 'fraction_of_data_set': 1, 'precomputed_attributions_val': '', 'calculate_log_odds': False, 'do_evaluate': False, 'with_tensorboard': False, 'do_augment': False, 'with_bn': False, 'attack_num_samples': 100, 'do_hyperopt': False, 'attack_epsilon': 1, 'save_attributions': False, 'save_predictions': True, 'target_digit': 3, 'load_existing': '/home/d909b/models/cex_main_1/model.npz', 'source_digit': 8, 'calculate_robustness': False, 'discrete_attack_type': 'lsga', 'num_hyperopt_runs': 35, 'thermometer_encoding_levels': 16, 'learning_rate': 0.001, 'batch_size': 100, 'output_directory': '/home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty', 'load_existing_cxplain': '/home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npz', 'do_merge_lsf': False, 'slic_compactness': 10, 'do_hyperopt_on_lsf': False, 'l2_weight': 0.0, 'hyperopt_o(cid:31)set': 0, 'dataset': 'mnist', 'dropout': 0.0, 'defence_method': 'none', 'num_gpus': 1, 'imagenet_folder': '/home/d909b/backup/imagenettrain-prep/', 'attack_method': 'random', 'num_layers_cxplain': -1, 'early_stopping_patience': 12, 'do_train': False, 'num_explanation_samples': 1000, 'num_units': 75, 'test_set_fraction': 0.2, 'model_type': 'resnet', 'validation_set_fraction': 0.2, 'num_units_cxplain': -1, 'model_name': 'model.npz'}INFO: Loaded generator with 11982 samples. Doing 120 steps of size 100INFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Built generators with 120 training samples, 20 validation samples and 20 test samples.INFO: Started training feature extraction.__________________________________________________________________________________________________Layer (type) Output Shape Param # Connected to ==================================================================================================input_1 (InputLayer) (None, 28, 28, 1) 0 __________________________________________________________________________________________________conv2d_1 (Conv2D) (None, 28, 28, 16) 160 input_1[0][0] __________________________________________________________________________________________________activation_1 (Activation) (None, 28, 28, 16) 0 conv2d_1[0][0] __________________________________________________________________________________________________conv2d_2 (Conv2D) (None, 28, 28, 16) 2320 activation_1[0][0] __________________________________________________________________________________________________activation_2 (Activation) (None, 28, 28, 16) 0 conv2d_2[0][0] __________________________________________________________________________________________________conv2d_3 (Conv2D) (None, 28, 28, 16) 2320 activation_2[0][0] __________________________________________________________________________________________________activation_3 (Activation) (None, 28, 28, 16) 0 conv2d_3[0][0] __________________________________________________________________________________________________add_1 (Add) (None, 28, 28, 16) 0 activation_1[0][0] activation_3[0][0] __________________________________________________________________________________________________activation_4 (Activation) (None, 28, 28, 16) 0 add_1[0][0] __________________________________________________________________________________________________conv2d_4 (Conv2D) (None, 28, 28, 16) 2320 activation_4[0][0] __________________________________________________________________________________________________activation_5 (Activation) (None, 28, 28, 16) 0 conv2d_4[0][0] __________________________________________________________________________________________________conv2d_5 (Conv2D) (None, 28, 28, 16) 2320 activation_5[0][0] __________________________________________________________________________________________________activation_6 (Activation) (None, 28, 28, 16) 0 conv2d_5[0][0] __________________________________________________________________________________________________add_2 (Add) (None, 28, 28, 16) 0 activation_4[0][0] activation_6[0][0] __________________________________________________________________________________________________activation_7 (Activation) (None, 28, 28, 16) 0 add_2[0][0] __________________________________________________________________________________________________conv2d_6 (Conv2D) (None, 28, 28, 16) 2320 activation_7[0][0] __________________________________________________________________________________________________activation_8 (Activation) (None, 28, 28, 16) 0 conv2d_6[0][0] __________________________________________________________________________________________________conv2d_7 (Conv2D) (None, 28, 28, 16) 2320 activation_8[0][0] __________________________________________________________________________________________________activation_9 (Activation) (None, 28, 28, 16) 0 conv2d_7[0][0] __________________________________________________________________________________________________add_3 (Add) (None, 28, 28, 16) 0 activation_7[0][0] activation_9[0][0] __________________________________________________________________________________________________activation_10 (Activation) (None, 28, 28, 16) 0 add_3[0][0] __________________________________________________________________________________________________conv2d_8 (Conv2D) (None, 14, 14, 32) 4640 activation_10[0][0] __________________________________________________________________________________________________activation_11 (Activation) (None, 14, 14, 32) 0 conv2d_8[0][0] __________________________________________________________________________________________________conv2d_10 (Conv2D) (None, 14, 14, 32) 544 activation_10[0][0] __________________________________________________________________________________________________conv2d_9 (Conv2D) (None, 14, 14, 32) 9248 activation_11[0][0] __________________________________________________________________________________________________activation_13 (Activation) (None, 14, 14, 32) 0 conv2d_10[0][0] __________________________________________________________________________________________________activation_12 (Activation) (None, 14, 14, 32) 0 conv2d_9[0][0] __________________________________________________________________________________________________add_4 (Add) (None, 14, 14, 32) 0 activation_13[0][0] activation_12[0][0] __________________________________________________________________________________________________activation_14 (Activation) (None, 14, 14, 32) 0 add_4[0][0] __________________________________________________________________________________________________conv2d_11 (Conv2D) (None, 14, 14, 32) 9248 activation_14[0][0] __________________________________________________________________________________________________activation_15 (Activation) (None, 14, 14, 32) 0 conv2d_11[0][0] __________________________________________________________________________________________________conv2d_12 (Conv2D) (None, 14, 14, 32) 9248 activation_15[0][0] __________________________________________________________________________________________________activation_16 (Activation) (None, 14, 14, 32) 0 conv2d_12[0][0] __________________________________________________________________________________________________add_5 (Add) (None, 14, 14, 32) 0 activation_14[0][0] activation_16[0][0] __________________________________________________________________________________________________activation_17 (Activation) (None, 14, 14, 32) 0 add_5[0][0] __________________________________________________________________________________________________conv2d_13 (Conv2D) (None, 14, 14, 32) 9248 activation_17[0][0] __________________________________________________________________________________________________activation_18 (Activation) (None, 14, 14, 32) 0 conv2d_13[0][0] __________________________________________________________________________________________________conv2d_14 (Conv2D) (None, 14, 14, 32) 9248 activation_18[0][0] __________________________________________________________________________________________________activation_19 (Activation) (None, 14, 14, 32) 0 conv2d_14[0][0] __________________________________________________________________________________________________add_6 (Add) (None, 14, 14, 32) 0 activation_17[0][0] activation_19[0][0] __________________________________________________________________________________________________activation_20 (Activation) (None, 14, 14, 32) 0 add_6[0][0] __________________________________________________________________________________________________conv2d_15 (Conv2D) (None, 7, 7, 64) 18496 activation_20[0][0] __________________________________________________________________________________________________activation_21 (Activation) (None, 7, 7, 64) 0 conv2d_15[0][0] __________________________________________________________________________________________________conv2d_17 (Conv2D) (None, 7, 7, 64) 2112 activation_20[0][0] __________________________________________________________________________________________________conv2d_16 (Conv2D) (None, 7, 7, 64) 36928 activation_21[0][0] __________________________________________________________________________________________________activation_23 (Activation) (None, 7, 7, 64) 0 conv2d_17[0][0] __________________________________________________________________________________________________activation_22 (Activation) (None, 7, 7, 64) 0 conv2d_16[0][0] __________________________________________________________________________________________________add_7 (Add) (None, 7, 7, 64) 0 activation_23[0][0] activation_22[0][0] __________________________________________________________________________________________________activation_24 (Activation) (None, 7, 7, 64) 0 add_7[0][0] __________________________________________________________________________________________________conv2d_18 (Conv2D) (None, 7, 7, 64) 36928 activation_24[0][0] __________________________________________________________________________________________________activation_25 (Activation) (None, 7, 7, 64) 0 conv2d_18[0][0] __________________________________________________________________________________________________conv2d_19 (Conv2D) (None, 7, 7, 64) 36928 activation_25[0][0] __________________________________________________________________________________________________activation_26 (Activation) (None, 7, 7, 64) 0 conv2d_19[0][0] __________________________________________________________________________________________________add_8 (Add) (None, 7, 7, 64) 0 activation_24[0][0] activation_26[0][0] __________________________________________________________________________________________________activation_27 (Activation) (None, 7, 7, 64) 0 add_8[0][0] __________________________________________________________________________________________________conv2d_20 (Conv2D) (None, 7, 7, 64) 36928 activation_27[0][0] __________________________________________________________________________________________________activation_28 (Activation) (None, 7, 7, 64) 0 conv2d_20[0][0] __________________________________________________________________________________________________conv2d_21 (Conv2D) (None, 7, 7, 64) 36928 activation_28[0][0] __________________________________________________________________________________________________activation_29 (Activation) (None, 7, 7, 64) 0 conv2d_21[0][0] __________________________________________________________________________________________________add_9 (Add) (None, 7, 7, 64) 0 activation_27[0][0] activation_29[0][0] __________________________________________________________________________________________________activation_30 (Activation) (None, 7, 7, 64) 0 add_9[0][0] __________________________________________________________________________________________________average_pooling2d_1 (AveragePoo (None, 1, 1, 64) 0 activation_30[0][0] __________________________________________________________________________________________________(cid:30)atten_1 (Flatten) (None, 64) 0 average_pooling2d_1[0][0] __________________________________________________________________________________________________dense_1 (Dense) (None, 2) 130 (cid:30)atten_1[0][0] __________________________________________________________________________________________________activation_31 (Activation) (None, 2) 0 dense_1[0][0] ==================================================================================================Total params: 270,882Trainable params: 270,882Non-trainable params: 0__________________________________________________________________________________________________INFO: Loading existing model from /home/d909b/models/cex_main_1/model.npz_________________________________________________________________Layer (type) Output Shape Param # =================================================================input_3 (InputLayer) (None, 28, 28, 1) 0 _________________________________________________________________(cid:30)atten_3 (Flatten) (None, 784) 0 _________________________________________________________________dense_3 (Dense) (None, 75) 58875 _________________________________________________________________activation_63 (Activation) (None, 75) 0 _________________________________________________________________dense_4 (Dense) (None, 75) 5700 _________________________________________________________________activation_64 (Activation) (None, 75) 0 _________________________________________________________________dense_5 (Dense) (None, 196) 14896 _________________________________________________________________activation_65 (Activation) (None, 196) 0 _________________________________________________________________reshape_1 (Reshape) (None, 14, 14, 1) 0 _________________________________________________________________lambda_2 (Lambda) (None, 28, 28, 1) 0 _________________________________________________________________reshape_2 (Reshape) (None, 784) 0 =================================================================Total params: 79,471Trainable params: 79,471Non-trainable params: 0_________________________________________________________________INFO: Loading existing CXPlain model from /home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npzbuild_explanation_model : took 1.81394100189 seconds.INFO: Saving loss history to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/losses.pickleINFO: Saving model predictions.INFO: Loaded generator with 11982 samples. Doing 120 steps of size 100INFO: Saved model predictions to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/train_predictions.csvINFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Saved model predictions to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/val_predictions.csvINFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Saved model predictions to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/test_predictions.csvINFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Performance on test AUROC (weighted) = 0.999994917356212 , with AUPRC (weighted) = 0.9999949280930339 , with r^2 (weighted) = 0.9957963109416453 , with f1 (weighted) = 0.9984879166751001 , with accuracy = 0.9984879032258065 , with error = 0.0015120967741935054INFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Performance on test [n= 100 ] targets = [0.9] observed = [(1.1436718599626305, '+-', 0.5520538782615332)]INFO: Performance on test [n= 100 ] for target 0.9observed = [1.3892620290955573, 1.1377237947530967, 0.8350603218815686, 0.8713871427716444, 1.8772789577296631, 1.0421385269639882, 1.2734869085839489, 0.5600630255113783, 1.2435160902712874, 2.0858731684436953, 0.6621914099533671, 0.6070763678776874, 0.8661585671984602, 1.0517880297070337, 2.9146049149183884, 0.6235869496235044, 0.9226209148366383, 0.896227432532063, 0.2670657996896821, 0.013821244358016503, 2.189807259704378, 0.8175433055816894, 1.19488319718703, 1.8930241978639137, 0.7730931068420339, 1.27811588137019, 0.9850174716812721, 1.1216892414631758, 0.573120152159161, 1.0232491128492835, 0.8987605463776784, 1.037649214136879, 0.5118367176339356, 2.083070000264211, 1.302903762457773, 0.8204965828874214, 1.6789029079041142, 1.623794531947206, 1.2545555949371687, 1.0012906768156877, 1.2462736163330106, 0.9057037790232868, 0.9469323067092333, 0.4383414096479612, 0.8104805348408998, 0.8851755920050628, 1.132267732268079, 1.6621074239888216, 0.934013287760548, 0.8349687929106586, 0.5289153327173624, 1.4440371769826026, 2.138591203347149, 1.2270851493051227, 2.142648193073443, 0.3385335850217482, 0.46770572775341435, 0.8527171407093351, 0.47953222713471033, 1.8026274831409246, 1.1812909273907717, 0.9314364881634934, 0.2302455728277166, 1.5633736254880761, 2.3031484613717335, 2.4174470115754003, 1.1201235152098241, 2.266265571565306, 1.3994960435747492, 0.7860688071695486, 1.2776682986304249, 1.6580154383934487, 0.757406483017344, 1.0043128902083016, 0.7231292579067669, 0.740787741347223, 0.7141079559712596, 0.6734593179701929, 0.6616112633542277, 0.9468177183380312, 0.7690791699258368, 1.0155653037702255, 0.6512117579059203, 0.9884971489035145, 0.6675223757172929, 1.7234492486644164, 0.7886172116728509, 1.1807222822302033, 0.8768496962883034, 1.2127882377541048, 1.0925325778269959, 0.5464445929844274, 2.096737223622148, 0.8172319369500524, 1.8495334324613373, 1.2724899666210199, 1.0701363404910589, 1.8738202234307906, 2.195503699637992, 1.8998484264924382]Process (cid:29)nished with exit code 0Gorilla or Zebra?SHAPCXPlain (U-net)LIME\fFigure 6: A comparison of the compute time (in\nlog2 seconds) needed to produce feature impor-\ntance estimates using several state-of-the-art fea-\nture importance estimation methods on the same\nhardware for the same N = 100 sample test im-\nages from the MNIST benchmark (lower is better).\n*** = signi\ufb01cantly different (p < 0.001, MWW)\n\nFigure 7: A comparison of the compute time (in\nlog2 seconds) needed to produce feature impor-\ntance estimates using state-of-the-art feature im-\nportance estimation methods on the same hard-\nware for the same N = 100 sample test images\nfrom the ImageNet benchmark (lower is better).\n*** = signi\ufb01cantly different (p < 0.001, MWW)\n\nIf our uncertainty estimates are well calibrated, we would expect to see a high correlation between\nthe uncertainty estimates ui and the magnitude of rank errors REi, since that would indicate that the\nuncertainty estimates ui accurately quantify how certain the feature importance estimates \u02c6ai are on\npreviously unseen sample images. For the comparison of the resulting distributions of correlation\nscores, we applied the Fisher z-transform to the correlation scores in order to correct for the skew\nin the distribution of the sample correlation [53]. Figure 9 depicts visualisations of the calculated\nground-truth log odds, the rank errors of the explanation model\u2019s importance estimates, and the\nuncertainty for each importance estimate for three test set images. We used the same hyperparameters\nas in the previous experiment to train the ensembled CXPlain (MLP) models (Appendix A).\n\n5 Results and Discussion\n\nPredictive Performance. We found that, on the MNIST benchmark, CXPlain (U-net) was compet-\nitive with the best competing state-of-the-art feature importance estimation method, DeepSHAP. We\nalso found that CXPlain (U-net) produced signi\ufb01cantly (p < 0.001, MWW) more accurate feature\nimportance estimates than CXPlain (MLP) - indicating that model architectures speci\ufb01cally tailored\nfor the image domain are more effective than MLPs in neural explanation models (Figure 2). On the\nImageNet benchmark, CXPlain signi\ufb01cantly (p < 0.01, MWW) outperformed the best competing\nfeature importance estimation method, LIME (Figure 3). We also found that the model-speci\ufb01c\nattribution methods Simple Gradient and Integrated Gradients performed relatively poorly across\nboth benchmarks, and were consistently outperformed by the model-agnostic attribution methods,\nCXPlain, and DeepSHAP. Qualitatively, we found that the estimates of feature importance provided\nby CXPlain were more focused on the subjectively more important semantic regions of the sample\nimages from both MNIST and ImageNet (Figures 4 and 5; more in Appendix D). Other methods, in\ncontrast, produced more super\ufb02uous attributions. This behavior is exhibited in Figure 5 where SHAP\nand LIME both attribute signi\ufb01cant importance to the wall behind the gorilla, whereas CXPlain\nfocused nearly all its attention on the gorilla itself, with the exception of the window frame receiving\nsome importance outside the top 30% of importances of that sample image. We believe this could be\ndue to the fact that the causal objective strongly penalises attributions outside regions of interest -\nleading to qualitatively more focused estimates of importance.\n\nComputational Performance.\nIn terms of computational performance, we found that CXPlain\ncomputed feature importance estimates signi\ufb01cantly faster than the state-of-the-art model-agnostic\nattribution methods, LIME and SHAP, on both the MNIST and ImageNet benchmarks (Figures 6\nand 7). Gradient-based attribution methods and CXPlain performed similarly. On ImageNet, the gap\nbetween LIME and SHAP and the faster methods was considerably larger than on MNIST, since the\nlarge numbers of model evaluations for LIME and SHAP were slower on higher-dimensional images.\n\nQuality of Uncertainty Estimates. We found that, quantitatively, even relatively small CXPlain\nensembles with just M = 5 bootstrap resampled models produce uncertainty estimates that are\nsigni\ufb01cantly (p < 0.001, MWW, compared to Random) correlated with its ability to accurately\nestimate feature importances on N = 100 previously unseen test images (Figure 8). We also found\n\n7\n\n***28.90(\u00b10.12)28.83(\u00b10.15)508.82(\u00b114.20)373.69(\u00b18.50)33.81(\u00b10.17)28.55(\u00b10.16)30.43(\u00b10.33)222528CXPlain(MLP)IntegratedGradientsSimpleGradientCXPlain(U\u2212net)Deep\u2212SHAPSHAPLIMEcompute time [log2 (s)]MNIST***35.87(\u00b10.11)35.78(\u00b10.05)8366.36(\u00b158.69)9230.00(\u00b124.42)69.08(\u00b10.11)34.19(\u00b10.10)222528213CXPlain(U\u2212net)IntegratedGradientsSimpleGradientDeep\u2212SHAPLIMESHAPcompute time [log2 (s)]ImageNet\fFigure 8: A comparison of the distributions of the\nz-transformed Pearson\u2019s correlations \u03c1 between\nthe uncertainty estimates ui produced by various\nnumbers M of bootstrapped ensembles of CX-\nPlain models and the Random baseline, and the\nground-truth rank errors of the top 2.5% most im-\nportant pixels across N = 100 unseen test images\nfrom the MNIST benchmark (higher is better).\n*** = signi\ufb01cantly different (p < 0.001, MWW)\n\nFigure 9: Visualisations of the calculated ground-\ntruth change in log odds \u2206log-odds, the Rank\nErrors of the explanation model\u2019s feature impor-\ntance estimates, and the Estimated Uncertainty\nfor each feature importance estimate as obtained\nvia bootstrap resampling (M = 100) for three\nunseen sample test set images (Input) from the\nMNIST benchmark. Note the visual similarity of\nthe Rank Error and the Estimated Uncertainty.\n\nthat increasing the size M of the bootstrap ensemble further signi\ufb01cantly (p < 0.001 for M = 5\nto M = 100, MWW) increases this correlation, and, thus, the quality of the provided uncertainty\nestimates. Qualitatively, there was a high visual similarity between the uncertainty estimates ui\nprovided by the CXPlain ensembles for each input feature xi and the magnitude of rank errors REi\ncommitted by its importance estimates \u02c6ai (Figure 9). The large differences in importance estimation\naccuracy between state-of-the-art feature importance estimation methods shown in the MNIST and\nImageNet benchmarks indicate that many of the importance estimates they provide are not truthful\nto the predictive model \u02c6f to be explained, and that measures of uncertainty are necessary to fully\nunderstand the expected reliability of feature importance estimates.\n\nLimitations. While they are fast at evaluation time, a limitation of CXPlain models is that they\nhave to be trained to learn to explain a predictive model. However, this one-off compute cost typically\namortises quickly, since CXPlain is signi\ufb01cantly faster at evaluation time than existing model-agnostic\nimportance estimation methods. Another important point to note is that the associations identi\ufb01ed\nby CXPlain models are only causal in the sense that they quantify the degree to which the input\nfeatures xi caused a marginal improvement in the predictive performance of the predictive model\n\u02c6f. Associations reported by CXPlain, in particular, do not in any way indicate that there is a causal\nrelationship between the explained model\u2019s input and output variables in the real world.\n\n6 Conclusion\n\nWe presented CXPlain, a new method for learning to estimate feature importance for any machine-\nlearning model. CXPlain is based on the idea of training a separate explanation model to learn\nto estimate which features are important for a given output of a target predictive model using a\ncausal objective. This approach has several advantages over existing ones: It is compatible with\nany machine-learning model, can produce estimates of feature importance quickly after training,\nand may be combined with bootstrap resampling to obtain uncertainty estimates for the provided\nfeature importance scores. We showed experimentally that CXPlain is signi\ufb01cantly more accurate in\nestimating feature importance than existing model-agnostic methods on both MNIST and ImageNet\nbenchmarks, while being orders of magnitude faster at providing importance estimates than state-\nof-the-art model-agnostic methods. We also found that, analogous to standard supervised learning\ntasks, special-purpose model architectures may improve the performance of neural explanation\nmodels in images, and that the bootstrap resampled uncertainty estimates for the importance scores\nof an explanation model are signi\ufb01cantly correlated with CXPlain\u2019s ability to accurately estimate\nfeature importance - indicating that bootstrap resampling is a suitable approach for quantifying\nthe uncertainty of importance estimates. Causal explanation models that both produce accurate\nestimates of feature importance and their uncertainties quickly for any machine-learning model and\ndata modality may enable users to better understand, validate, and interpret machine-learning models,\nwhile also informing them when their explanations can not be expected to be accurate.\n\n8\n\n******\u2212101234RandomCXPlain(M=5)CXPlain(M=10)CXPlain(M=50)CXPlain(M=100)z\u2212transformed Pearson\u00a2s rUncertainty Estimation AccuracyInputRank ErrorEstimatedUncertaintylog-oddsssh://d909b@ssh.schwabpatrick.com:909/usr/bin/python -u /home/d909b/bin/causal_explanations/causal_explanations/apps/main.py --output_directory=/home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty --load_existing=/home/d909b/models/cex_main_1/model.npz --load_existing_cxplain=/home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npz --dataset=mnist --num_epochs=50 --num_units=75 --batch_size=100 --num_layers=2 --dropout=0.0 --learning_rate=0.001 --l2_weight=0.000 --model_type=resnet --explanation_type=cxplain --do_not_save_attributions --source_digit=8 --target_digit=3 --attack_method=random --attack_epsilon=1 --attack_num_samples=100 --defence_method=none --discrete_attack_type=lsga --do_not_calculate_log_odds --do_not_calculate_robustness --num_explanation_samples=1000 --num_boostrap_samples=5Using TensorFlow backend./usr/local/lib/python2.7/dist-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release. from numpy.core.umath_tests import inner1dINFO: Args are: {'num_boostrap_samples': 5, 'do_adversarial_training': False, 'n_jobs': 4, 'explanation_type': 'cxplain', 'calculate_uncertainties': True, 'num_adversarial_samples': 1000, 'hyperopt_against_eval_set': False, 'num_epochs': 50, 'explanation_model_name': 'explanation.npz', 'copy_to_local': False, 'attack_iterations': 100, 'seed': 909, 'precomputed_attributions_train': '', 'num_layers': 2, 'fraction_of_data_set': 1, 'precomputed_attributions_val': '', 'calculate_log_odds': False, 'do_evaluate': False, 'with_tensorboard': False, 'do_augment': False, 'with_bn': False, 'attack_num_samples': 100, 'do_hyperopt': False, 'attack_epsilon': 1, 'save_attributions': False, 'save_predictions': True, 'target_digit': 3, 'load_existing': '/home/d909b/models/cex_main_1/model.npz', 'source_digit': 8, 'calculate_robustness': False, 'discrete_attack_type': 'lsga', 'num_hyperopt_runs': 35, 'thermometer_encoding_levels': 16, 'learning_rate': 0.001, 'batch_size': 100, 'output_directory': '/home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty', 'load_existing_cxplain': '/home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npz', 'do_merge_lsf': False, 'slic_compactness': 10, 'do_hyperopt_on_lsf': False, 'l2_weight': 0.0, 'hyperopt_o(cid:31)set': 0, 'dataset': 'mnist', 'dropout': 0.0, 'defence_method': 'none', 'num_gpus': 1, 'imagenet_folder': '/home/d909b/backup/imagenettrain-prep/', 'attack_method': 'random', 'num_layers_cxplain': -1, 'early_stopping_patience': 12, 'do_train': False, 'num_explanation_samples': 1000, 'num_units': 75, 'test_set_fraction': 0.2, 'model_type': 'resnet', 'validation_set_fraction': 0.2, 'num_units_cxplain': -1, 'model_name': 'model.npz'}INFO: Running at 2019-05-10 16:48:02.083797INFO: Seed is 9092019-05-10 16:48:02.092153: I tensor(cid:30)ow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA2019-05-10 16:48:02.200813: I tensor(cid:30)ow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero2019-05-10 16:48:02.201281: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582pciBusID: 0000:01:00.0totalMemory: 11.90GiB freeMemory: 11.55GiB2019-05-10 16:48:02.201296: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 02019-05-10 16:48:02.396536: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:2019-05-10 16:48:02.396563: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:988] 0 2019-05-10 16:48:02.396569: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2019-05-10 16:48:02.396735: I tensor(cid:30)ow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11180 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:01:00.0, compute capability: 6.1)INFO: Loading MNIST data.INFO: Run with args: {'num_boostrap_samples': 5, 'do_adversarial_training': False, 'n_jobs': 4, 'explanation_type': 'cxplain', 'calculate_uncertainties': True, 'num_adversarial_samples': 1000, 'hyperopt_against_eval_set': False, 'num_epochs': 50, 'explanation_model_name': 'explanation.npz', 'copy_to_local': False, 'attack_iterations': 100, 'seed': 909, 'precomputed_attributions_train': '', 'num_layers': 2, 'fraction_of_data_set': 1, 'precomputed_attributions_val': '', 'calculate_log_odds': False, 'do_evaluate': False, 'with_tensorboard': False, 'do_augment': False, 'with_bn': False, 'attack_num_samples': 100, 'do_hyperopt': False, 'attack_epsilon': 1, 'save_attributions': False, 'save_predictions': True, 'target_digit': 3, 'load_existing': '/home/d909b/models/cex_main_1/model.npz', 'source_digit': 8, 'calculate_robustness': False, 'discrete_attack_type': 'lsga', 'num_hyperopt_runs': 35, 'thermometer_encoding_levels': 16, 'learning_rate': 0.001, 'batch_size': 100, 'output_directory': '/home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty', 'load_existing_cxplain': '/home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npz', 'do_merge_lsf': False, 'slic_compactness': 10, 'do_hyperopt_on_lsf': False, 'l2_weight': 0.0, 'hyperopt_o(cid:31)set': 0, 'dataset': 'mnist', 'dropout': 0.0, 'defence_method': 'none', 'num_gpus': 1, 'imagenet_folder': '/home/d909b/backup/imagenettrain-prep/', 'attack_method': 'random', 'num_layers_cxplain': -1, 'early_stopping_patience': 12, 'do_train': False, 'num_explanation_samples': 1000, 'num_units': 75, 'test_set_fraction': 0.2, 'model_type': 'resnet', 'validation_set_fraction': 0.2, 'num_units_cxplain': -1, 'model_name': 'model.npz'}INFO: Loaded generator with 11982 samples. Doing 120 steps of size 100INFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Built generators with 120 training samples, 20 validation samples and 20 test samples.INFO: Started training feature extraction.__________________________________________________________________________________________________Layer (type) Output Shape Param # Connected to ==================================================================================================input_1 (InputLayer) (None, 28, 28, 1) 0 __________________________________________________________________________________________________conv2d_1 (Conv2D) (None, 28, 28, 16) 160 input_1[0][0] __________________________________________________________________________________________________activation_1 (Activation) (None, 28, 28, 16) 0 conv2d_1[0][0] __________________________________________________________________________________________________conv2d_2 (Conv2D) (None, 28, 28, 16) 2320 activation_1[0][0] __________________________________________________________________________________________________activation_2 (Activation) (None, 28, 28, 16) 0 conv2d_2[0][0] __________________________________________________________________________________________________conv2d_3 (Conv2D) (None, 28, 28, 16) 2320 activation_2[0][0] __________________________________________________________________________________________________activation_3 (Activation) (None, 28, 28, 16) 0 conv2d_3[0][0] __________________________________________________________________________________________________add_1 (Add) (None, 28, 28, 16) 0 activation_1[0][0] activation_3[0][0] __________________________________________________________________________________________________activation_4 (Activation) (None, 28, 28, 16) 0 add_1[0][0] __________________________________________________________________________________________________conv2d_4 (Conv2D) (None, 28, 28, 16) 2320 activation_4[0][0] __________________________________________________________________________________________________activation_5 (Activation) (None, 28, 28, 16) 0 conv2d_4[0][0] __________________________________________________________________________________________________conv2d_5 (Conv2D) (None, 28, 28, 16) 2320 activation_5[0][0] __________________________________________________________________________________________________activation_6 (Activation) (None, 28, 28, 16) 0 conv2d_5[0][0] __________________________________________________________________________________________________add_2 (Add) (None, 28, 28, 16) 0 activation_4[0][0] activation_6[0][0] __________________________________________________________________________________________________activation_7 (Activation) (None, 28, 28, 16) 0 add_2[0][0] __________________________________________________________________________________________________conv2d_6 (Conv2D) (None, 28, 28, 16) 2320 activation_7[0][0] __________________________________________________________________________________________________activation_8 (Activation) (None, 28, 28, 16) 0 conv2d_6[0][0] __________________________________________________________________________________________________conv2d_7 (Conv2D) (None, 28, 28, 16) 2320 activation_8[0][0] __________________________________________________________________________________________________activation_9 (Activation) (None, 28, 28, 16) 0 conv2d_7[0][0] __________________________________________________________________________________________________add_3 (Add) (None, 28, 28, 16) 0 activation_7[0][0] activation_9[0][0] __________________________________________________________________________________________________activation_10 (Activation) (None, 28, 28, 16) 0 add_3[0][0] __________________________________________________________________________________________________conv2d_8 (Conv2D) (None, 14, 14, 32) 4640 activation_10[0][0] __________________________________________________________________________________________________activation_11 (Activation) (None, 14, 14, 32) 0 conv2d_8[0][0] __________________________________________________________________________________________________conv2d_10 (Conv2D) (None, 14, 14, 32) 544 activation_10[0][0] __________________________________________________________________________________________________conv2d_9 (Conv2D) (None, 14, 14, 32) 9248 activation_11[0][0] __________________________________________________________________________________________________activation_13 (Activation) (None, 14, 14, 32) 0 conv2d_10[0][0] __________________________________________________________________________________________________activation_12 (Activation) (None, 14, 14, 32) 0 conv2d_9[0][0] __________________________________________________________________________________________________add_4 (Add) (None, 14, 14, 32) 0 activation_13[0][0] activation_12[0][0] __________________________________________________________________________________________________activation_14 (Activation) (None, 14, 14, 32) 0 add_4[0][0] __________________________________________________________________________________________________conv2d_11 (Conv2D) (None, 14, 14, 32) 9248 activation_14[0][0] __________________________________________________________________________________________________activation_15 (Activation) (None, 14, 14, 32) 0 conv2d_11[0][0] __________________________________________________________________________________________________conv2d_12 (Conv2D) (None, 14, 14, 32) 9248 activation_15[0][0] __________________________________________________________________________________________________activation_16 (Activation) (None, 14, 14, 32) 0 conv2d_12[0][0] __________________________________________________________________________________________________add_5 (Add) (None, 14, 14, 32) 0 activation_14[0][0] activation_16[0][0] __________________________________________________________________________________________________activation_17 (Activation) (None, 14, 14, 32) 0 add_5[0][0] __________________________________________________________________________________________________conv2d_13 (Conv2D) (None, 14, 14, 32) 9248 activation_17[0][0] __________________________________________________________________________________________________activation_18 (Activation) (None, 14, 14, 32) 0 conv2d_13[0][0] __________________________________________________________________________________________________conv2d_14 (Conv2D) (None, 14, 14, 32) 9248 activation_18[0][0] __________________________________________________________________________________________________activation_19 (Activation) (None, 14, 14, 32) 0 conv2d_14[0][0] __________________________________________________________________________________________________add_6 (Add) (None, 14, 14, 32) 0 activation_17[0][0] activation_19[0][0] __________________________________________________________________________________________________activation_20 (Activation) (None, 14, 14, 32) 0 add_6[0][0] __________________________________________________________________________________________________conv2d_15 (Conv2D) (None, 7, 7, 64) 18496 activation_20[0][0] __________________________________________________________________________________________________activation_21 (Activation) (None, 7, 7, 64) 0 conv2d_15[0][0] __________________________________________________________________________________________________conv2d_17 (Conv2D) (None, 7, 7, 64) 2112 activation_20[0][0] __________________________________________________________________________________________________conv2d_16 (Conv2D) (None, 7, 7, 64) 36928 activation_21[0][0] __________________________________________________________________________________________________activation_23 (Activation) (None, 7, 7, 64) 0 conv2d_17[0][0] __________________________________________________________________________________________________activation_22 (Activation) (None, 7, 7, 64) 0 conv2d_16[0][0] __________________________________________________________________________________________________add_7 (Add) (None, 7, 7, 64) 0 activation_23[0][0] activation_22[0][0] __________________________________________________________________________________________________activation_24 (Activation) (None, 7, 7, 64) 0 add_7[0][0] __________________________________________________________________________________________________conv2d_18 (Conv2D) (None, 7, 7, 64) 36928 activation_24[0][0] __________________________________________________________________________________________________activation_25 (Activation) (None, 7, 7, 64) 0 conv2d_18[0][0] __________________________________________________________________________________________________conv2d_19 (Conv2D) (None, 7, 7, 64) 36928 activation_25[0][0] __________________________________________________________________________________________________activation_26 (Activation) (None, 7, 7, 64) 0 conv2d_19[0][0] __________________________________________________________________________________________________add_8 (Add) (None, 7, 7, 64) 0 activation_24[0][0] activation_26[0][0] __________________________________________________________________________________________________activation_27 (Activation) (None, 7, 7, 64) 0 add_8[0][0] __________________________________________________________________________________________________conv2d_20 (Conv2D) (None, 7, 7, 64) 36928 activation_27[0][0] __________________________________________________________________________________________________activation_28 (Activation) (None, 7, 7, 64) 0 conv2d_20[0][0] __________________________________________________________________________________________________conv2d_21 (Conv2D) (None, 7, 7, 64) 36928 activation_28[0][0] __________________________________________________________________________________________________activation_29 (Activation) (None, 7, 7, 64) 0 conv2d_21[0][0] __________________________________________________________________________________________________add_9 (Add) (None, 7, 7, 64) 0 activation_27[0][0] activation_29[0][0] __________________________________________________________________________________________________activation_30 (Activation) (None, 7, 7, 64) 0 add_9[0][0] __________________________________________________________________________________________________average_pooling2d_1 (AveragePoo (None, 1, 1, 64) 0 activation_30[0][0] __________________________________________________________________________________________________(cid:30)atten_1 (Flatten) (None, 64) 0 average_pooling2d_1[0][0] __________________________________________________________________________________________________dense_1 (Dense) (None, 2) 130 (cid:30)atten_1[0][0] __________________________________________________________________________________________________activation_31 (Activation) (None, 2) 0 dense_1[0][0] ==================================================================================================Total params: 270,882Trainable params: 270,882Non-trainable params: 0__________________________________________________________________________________________________INFO: Loading existing model from /home/d909b/models/cex_main_1/model.npz_________________________________________________________________Layer (type) Output Shape Param # =================================================================input_3 (InputLayer) (None, 28, 28, 1) 0 _________________________________________________________________(cid:30)atten_3 (Flatten) (None, 784) 0 _________________________________________________________________dense_3 (Dense) (None, 75) 58875 _________________________________________________________________activation_63 (Activation) (None, 75) 0 _________________________________________________________________dense_4 (Dense) (None, 75) 5700 _________________________________________________________________activation_64 (Activation) (None, 75) 0 _________________________________________________________________dense_5 (Dense) (None, 196) 14896 _________________________________________________________________activation_65 (Activation) (None, 196) 0 _________________________________________________________________reshape_1 (Reshape) (None, 14, 14, 1) 0 _________________________________________________________________lambda_2 (Lambda) (None, 28, 28, 1) 0 _________________________________________________________________reshape_2 (Reshape) (None, 784) 0 =================================================================Total params: 79,471Trainable params: 79,471Non-trainable params: 0_________________________________________________________________INFO: Loading existing CXPlain model from /home/d909b/models/cex_main_1_cxplain_20.1h/best_explanation.npzbuild_explanation_model : took 1.81394100189 seconds.INFO: Saving loss history to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/losses.pickleINFO: Saving model predictions.INFO: Loaded generator with 11982 samples. Doing 120 steps of size 100INFO: Saved model predictions to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/train_predictions.csvINFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Saved model predictions to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/val_predictions.csvINFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Saved model predictions to /home/d909b/models/cex_main_1_cxplain_20.1h_uncertainty/test_predictions.csvINFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Performance on test AUROC (weighted) = 0.999994917356212 , with AUPRC (weighted) = 0.9999949280930339 , with r^2 (weighted) = 0.9957963109416453 , with f1 (weighted) = 0.9984879166751001 , with accuracy = 0.9984879032258065 , with error = 0.0015120967741935054INFO: Loaded generator with 1984 samples. Doing 20 steps of size 100INFO: Performance on test [n= 100 ] targets = [0.9] observed = [(1.1436718599626305, '+-', 0.5520538782615332)]INFO: Performance on test [n= 100 ] for target 0.9observed = [1.3892620290955573, 1.1377237947530967, 0.8350603218815686, 0.8713871427716444, 1.8772789577296631, 1.0421385269639882, 1.2734869085839489, 0.5600630255113783, 1.2435160902712874, 2.0858731684436953, 0.6621914099533671, 0.6070763678776874, 0.8661585671984602, 1.0517880297070337, 2.9146049149183884, 0.6235869496235044, 0.9226209148366383, 0.896227432532063, 0.2670657996896821, 0.013821244358016503, 2.189807259704378, 0.8175433055816894, 1.19488319718703, 1.8930241978639137, 0.7730931068420339, 1.27811588137019, 0.9850174716812721, 1.1216892414631758, 0.573120152159161, 1.0232491128492835, 0.8987605463776784, 1.037649214136879, 0.5118367176339356, 2.083070000264211, 1.302903762457773, 0.8204965828874214, 1.6789029079041142, 1.623794531947206, 1.2545555949371687, 1.0012906768156877, 1.2462736163330106, 0.9057037790232868, 0.9469323067092333, 0.4383414096479612, 0.8104805348408998, 0.8851755920050628, 1.132267732268079, 1.6621074239888216, 0.934013287760548, 0.8349687929106586, 0.5289153327173624, 1.4440371769826026, 2.138591203347149, 1.2270851493051227, 2.142648193073443, 0.3385335850217482, 0.46770572775341435, 0.8527171407093351, 0.47953222713471033, 1.8026274831409246, 1.1812909273907717, 0.9314364881634934, 0.2302455728277166, 1.5633736254880761, 2.3031484613717335, 2.4174470115754003, 1.1201235152098241, 2.266265571565306, 1.3994960435747492, 0.7860688071695486, 1.2776682986304249, 1.6580154383934487, 0.757406483017344, 1.0043128902083016, 0.7231292579067669, 0.740787741347223, 0.7141079559712596, 0.6734593179701929, 0.6616112633542277, 0.9468177183380312, 0.7690791699258368, 1.0155653037702255, 0.6512117579059203, 0.9884971489035145, 0.6675223757172929, 1.7234492486644164, 0.7886172116728509, 1.1807222822302033, 0.8768496962883034, 1.2127882377541048, 1.0925325778269959, 0.5464445929844274, 2.096737223622148, 0.8172319369500524, 1.8495334324613373, 1.2724899666210199, 1.0701363404910589, 1.8738202234307906, 2.195503699637992, 1.8998484264924382]Process (cid:29)nished with exit code 0\u0394Uncertainty Estimation Samples\fAcknowledgments\n\nThis work was partially funded by the Swiss National Science Foundation (SNSF) project No. 167302\nwithin the National Research Program (NRP) 75 \u201cBig Data\u201d. We gratefully acknowledge the support\nof NVIDIA Corporation with the donation of the Titan Xp GPUs used for this research. Patrick\nSchwab is an af\ufb01liated PhD fellow at the Max Planck ETH Center for Learning Systems. We\nadditionally thank the anonymous reviewers whose comments helped improve this manuscript.\n\nReferences\n[1] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Learning important features\n\nthrough propagating activation differences. International Conference of Machine Learning, 2017.\n\n[2] Zachary C Lipton. The mythos of model interpretability. arXiv preprint arXiv:1606.03490, 2016.\n\n[3] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Sch\u00fctt, Sven D\u00e4hne,\nDumitru Erhan, and Been Kim. The (un) reliability of saliency methods. arXiv preprint arXiv:1711.00867,\n2017.\n\n[4] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Vi\u00e9gas, and Martin Wattenberg. Smoothgrad:\n\nremoving noise by adding noise. arXiv preprint arXiv:1706.03825, 2017.\n\n[5] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv\n\npreprint arXiv:1702.08608, 2017.\n\n[6] Scott M Lundberg and Su-In Lee. A uni\ufb01ed approach to interpreting model predictions. In Advances in\n\nNeural Information Processing Systems, pages 4768\u20134777, 2017.\n\n[7] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert\nM\u00fcller. How to explain individual classi\ufb01cation decisions. Journal of Machine Learning Research, 11\n(Jun):1803\u20131831, 2010.\n\n[8] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising\nimage classi\ufb01cation models and saliency maps. International Conference on Learning Representations,\n2014.\n\n[9] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European\n\nConference on Computer Vision, pages 818\u2013833. Springer, 2014.\n\n[10] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. International\n\nConference on Machine Learning, 2017.\n\n[11] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel,\nand Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In\nInternational Conference on Machine Learning, pages 2048\u20132057, 2015.\n\n[12] Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart.\nRetain: An interpretable predictive model for healthcare using reverse time attention mechanism. In\nAdvances in Neural Information Processing Systems, pages 3504\u20133512, 2016.\n\n[13] Patrick Schwab, Gaetano C. Scebba, Jia Zhang, Marco Delai, and Walter Karlen. Beat by Beat: Classifying\n\nCardiac Arrhythmias with Recurrent Neural Networks. In Computing in Cardiology, 2017.\n\n[14] Patrick Schwab, Djordje Miladinovic, and Walter Karlen. Granger-causal Attentive Mixtures of Experts:\nLearning Important Features with Neural Networks. In AAAI Conference on Arti\ufb01cial Intelligence, 2019.\n\n[15] Patrick Schwab and Walter Karlen. PhoneMD: Learning to diagnose Parkinson\u2019s disease from smartphone\n\ndata. In AAAI Conference on Arti\ufb01cial Intelligence, 2019.\n\n[16] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explaining the\npredictions of any classi\ufb01er. In Proceedings of the 22nd ACM SIGKDD International Conference on\nKnowledge Discovery and Data Mining, pages 1135\u20131144. ACM, 2016.\n\n[17] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity\nchecks for saliency maps. In Advances in Neural Information Processing Systems, pages 9505\u20139515, 2018.\n\n[18] Hui Fen, Kuangyan Song, Madeilene Udell, Yiming Sun, Yujia Zhang, et al. Why should you trust my\ninterpretation? Understanding uncertainty in LIME predictions. arXiv preprint arXiv:1904.12991, 2019.\n\n9\n\n\f[19] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. AAAI\n\nConference on Arti\ufb01cial Intelligence, 2019.\n\n[20] Erik \u0160trumbelj, Igor Kononenko, and M Robnik \u0160ikonja. Explaining instance classi\ufb01cations with interac-\n\ntions of subsets of feature values. Data & Knowledge Engineering, 68(10):886\u2013904, 2009.\n\n[21] Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network\ndecisions: Prediction difference analysis. In International Conference on Learning Representations, 2017.\n\n[22] Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure.\n\narXiv preprint arXiv:1612.08220, 2016.\n\n[23] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation.\n\nIn IEEE International Conference on Computer Vision, 2017.\n\n[24] Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classi\ufb01ers. In Advances in Neural\n\nInformation Processing Systems, pages 6967\u20136976, 2017.\n\n[25] Patrick Schwab and Helmut Hlavacs. Capturing the essence: Towards the automated generation of\nIn AAAI Conference on Arti\ufb01cial Intelligence and Interactive Digital\n\ntransparent behavior models.\nEntertainment, 2015.\n\n[26] Zhengping Che, Sanjay Purushotham, Robinder Khemani, and Yan Liu. Interpretable deep models for ICU\noutcome prediction. In AMIA Annual Symposium Proceedings, volume 2016, page 371. American Medical\nInformatics Association, 2016.\n\n[27] Osbert Bastani, Carolyn Kim, and Hamsa Bastani. Interpreting blackbox models via model extraction.\n\narXiv preprint arXiv:1705.08504, 2017.\n\n[28] Robert Andrews, Joachim Diederich, and Alan B Tickle. Survey and critique of techniques for extracting\n\nrules from trained arti\ufb01cial neural networks. Knowledge-based Systems, 8(6):373\u2013389, 1995.\n\n[29] Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, and Vineeth N Balasubramanian. Neural\n\nnetwork attributions: A causal perspective. arXiv preprint arXiv:1902.02302, 2019.\n\n[30] Gr\u00e9goire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert M\u00fcller.\nExplaining nonlinear classi\ufb01cation decisions with deep taylor decomposition. Pattern Recognition, 65:\n211\u2013222, 2017.\n\n[31] Jianbo Chen, Le Song, Martin J Wainwright, and Michael I Jordan. Learning to explain: An information-\n\ntheoretic perspective on model interpretation. International Conference on Machine Learning, 2018.\n\n[32] Wenbo Guo, Sui Huang, Yunzhe Tao, Xinyu Xing, and Lin Lin. Explaining deep learning models\u2013\na bayesian non-parametric approach. In Advances in Neural Information Processing Systems, pages\n4514\u20134524, 2018.\n\n[33] Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights.\n\nInternational Conference on Learning Representations, 2017.\n\n[34] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres.\nInterpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV).\nInternational Conference on Machine Learning, 2018.\n\n[35] Pang Wei Koh and Percy Liang. Understanding black-box predictions via in\ufb02uence functions. International\n\nConference of Machine Learning, 2017.\n\n[36] Clive WJ Granger. Investigating causal relations by econometric models and cross-spectral methods.\n\nEconometrica: Journal of the Econometric Society, pages 424\u2013438, 1969.\n\n[37] Richard Stone. The assumptions on which causal inferences rest. Journal of the Royal Statistical Society.\n\nSeries B (Methodological), pages 455\u2013466, 1993.\n\n[38] Dominik Janzing, David Balduzzi, Moritz Grosse-Wentrup, and Bernhard Sch\u00f6lkopf. Quantifying causal\n\nin\ufb02uences. The Annals of Statistics, 41(5):2324\u20132358, 2013.\n\n[39] Pasha Khosravi, Yitao Liang, YooJung Choi, and Guy Van den Broeck. What to expect of classi\ufb01ers?\n\nReasoning about logistic regression with missing features. arXiv preprint arXiv:1903.01620, 2019.\n\n[40] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.\n\n10\n\n\f[41] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT Press, 2016.\n\n[42] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking\nthe Inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern\nRecognition, pages 2818\u20132826, 2016.\n\n[43] Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob\n\nUszkoreit. One Model To Learn Them All. arXiv preprint arXiv:1706.05137, 2017.\n\n[44] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical\nimage segmentation. In International Conference on Medical Image Computing and Computer-assisted\nIntervention, pages 234\u2013241. Springer, 2015.\n\n[45] Bradley Efron. The jackknife, the bootstrap, and other resampling plans, volume 38. Siam, 1982.\n\n[46] Leo Breiman. Random forests. Machine Learning, 45(1):5\u201332, 2001.\n\n[47] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive\nuncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems,\npages 6402\u20136413, 2017.\n\n[48] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty\n\nin deep learning. In International Conference on Machine Learning, pages 1050\u20131059, 2016.\n\n[49] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout:\nA simple way to prevent neural networks from over\ufb01tting. The Journal of Machine Learning Research, 15\n(1):1929\u20131958, 2014.\n\n[50] Myles Hollander and Douglas A Wolfe. Nonparametric statistical methods. Wiley New York, NY, USA,\n\n1973.\n\n[51] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. AT&T Labs [Online].\n\nAvailable: http://yann.lecun.com/exdb/mnist, 2:18, 2010.\n\n[52] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical\nimage database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248\u2013255. IEEE,\n2009.\n\n[53] N Clayton Silver and William P Dunlap. Averaging correlation coef\ufb01cients: Should Fisher\u2019s z transforma-\n\ntion be used? Journal of Applied Psychology, 72(1):146, 1987.\n\n[54] Mart\u00edn Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S\nCorrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensor\ufb02ow: Large-scale machine learning on\nheterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.\n\n[55] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing\n\ninternal covariate shift. In International Conference on Machine Learning, pages 448\u2013456, 2015.\n\n[56] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference\n\non Learning Representations, 2015.\n\n[57] G\u00fcnter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural\n\nnetworks. In Advances in Neural Information Processing Systems, pages 971\u2013980, 2017.\n\n11\n\n\f", "award": [], "sourceid": 5401, "authors": [{"given_name": "Patrick", "family_name": "Schwab", "institution": "ETH Zurich / Roche"}, {"given_name": "Walter", "family_name": "Karlen", "institution": "ETH Zurich"}]}