Paper ID: | 2018 |
---|---|

Title: | Concrete Dropout |

The author formulates dropout as a variational bayes approach (as is also done in a few recent papers). This provides a theoretically grounded way of optimizing the amount of noise per layers. Previous work uses gaussian noise to be able to use reparametrization trick. In this work, a smooth relaxation of the categorical distribution, known as concrete variable or gumbel softmax is used to reduce variance of the gradient. The final algorithm is easy to implement and can be added to any existing architecture at almost no cost.
The experimental results shed insights on the amount of noise required with respect to 1) the number of samples 2) the size of each layers 3) the position of the layer in the network.
Overall, the paper is well written except for section 3, the most important part.
1) The prior is not specified in the paper and referred to the appendix. In the appendix, there is no explicit equation of the prior.
2) Please use a less compact formulation for the posterior (instead of diag and Bernoulli^{K_l}). It could be rewritten slightly differently and be more obvious to many reader how it relates to conventional dropout.
3) in equation 3, $l$ is undefined. I'm assuming it has nothing to do with the index of the layer. Also, since the prior is not really defined, it's hard to verify this equation. Please specify the prior explicitly and write more steps on how to obtain this equation.
4) Equation 5: I thought that gumbel noise was -log(-log(u)), not +log(u). I might have misunderstood something somewhere. If this is indeed the wrong equation for gumbel noise, please rerun experiments (it should not change much the end-result).
5) The lack of comparison with [23] is disappointing and the justification for doing so is not convincing.
6) I would have loved to see experimentations with a per-weight noise parameter as is done in [23]. Can the author explain why this experiment is not in the paper?
==== post discussion ===
Overall the paper is good. Please spend the time to make the paper more clear about the issues discussed by the reviewers.

This paper attempts to incorporate the dropout rate of Bayesian neural network dropout into the standard gradient based optimization routine. The focus is in relating the dropout rate, p, with a Bernoulli likelihood. To achieve the optimization of p, this is relaxed into the continuous domain through the Concrete distribution, which in this case has a simple sigmoidal form. Experiments test the method in terms of uncertainty modeling and boost in predictive accuracy.
In general this is a nicely written paper, but the presentation can be improved further. I did enjoy the intuitive parts, but perhaps they should be a little reduced to make space for some more technical details. For example, the prior p(\omega) in Section 3 and expanding on the form and derivation of eq. (5). I also think it would be better to only refer to "epistemic", "aleatoric" uncertainty in Section 2 and onwards (eg. Fig 1) switch to referring to the proper quantity measured.
Technically, this seems like a well motivated work, although I it seems that relaxing the discrete p into (5) moves the approach closer to something like [23]. I was hoping to see a comparison and I do not understand why it has been omitted, even if it is known to underperform. As far as novelty is concerned, this work builds heavily on work by Gal et al. and (inference-wise) moves towards the direction of [23], to some extent. Having said that, there are a few important novel elements, in particular the ability to optimize p as well as some intuitive justifications.
The experiments are quite extensive and the code is given at submission time, which is great. In general, the new method doesn't seem to offer a significant boost, but by being robust means that it can speed up experiment time. It would be nice to see some more quantitative proof that the uncertainty is better calibrated, e.g. in active learning (the RL experiment was nice). Finally, the paper would be very much improved if the authors could compare the overall approach to that of tuning the dropout rate with Bayesian Optimization (which is a very common strategy).
Some smaller comments:
- in line 107- ..., this is implying that w and p have to co-adapt during optimization?
- eq. (3): please state clearly what is K
- Plots 1b, 1c actually look very similar to me, as opposed to what is stated that one has a trend and the other is flat.
- figure 1d is very interesting.
- Fig 2: what happens when the error-bar collapses? Why does that happen? Also, how is the standard "Dropout" method tuning p?
- lines 250-...: From this argumentation I can still not see why the probabilities collapse to zero.
EDIT after rebuttal: ====
I have read the reply by the authors. I believe this will be an interesting read, especially once the comments raised by the reviewers (concerning clarity etc) are addressed.

This paper proposes an approach for automatic tuning of dropout
probabilities under the variational interpretation of dropout as a Bayesian approximation. Since the dropout probability characterizes the overall
posterior uncertainty, such tuning is necessary for calibrated inferences,
but has thus far required expensive manual tuning, e.g., via offline grid
search. The proposed method replaces the fixed Bernoulli variational distribution
at training time with a concrete/gumbel-softmax distribution, allowing use of the
reparameterization trick to compute low-variance stochastic gradients
of the ELBO wrt dropout probabilities. This is evaluated on a range of tasks
ranging from toy synthetic datasets to capturing uncertainty in semantic
segmentation and model-based RL.
The motivation seems clear and the proposed solution is simple and
natural, combining dropout Bayesian inference with the concrete
distribution in the obvious way. The technical contributions of the
paper seem relatively minimal, but this is not necessarily a problem;
it's great when a simple, straightforward method turns out to
give impressive results.
That said, given the straightforwardness of the approach, the
significance of this paper rests heavily on the evaluation, and here I
am not really convinced. Although there are a lot of experiments, none
seems to show a compelling advantage and collectively they fail to
answer many natural questions about the approach. For example:
- what is the training time overhead versus standard dropout? (line
281 says 'negligible' but this is never quantified, naively I'd
expect differentiating through the gumbel-softmax to add
significant overhead)
- do the learned dropout probabilities match the (presumably optimal)
probabilities obtained via grid search?
- how to set the temperature of the concrete distribution (and does
this matter)? The code in the appendix seems to used a fixed t=0.1
but this is never mentioned or discussed in the paper.
- how does this method compare to other reasonable baselines, eg
explicit mean-field BNNs, or Gaussian dropout with tuned alpha? The
latter seems like a highly relevant comparison, since the Gaussian
approach seems simpler both theoretically and in implementation (is
directly reparameterizable without the concrete approximation), and
to dismiss it by reference to a brief workshop paper [28] claiming
the approach 'underperforms' (I don't see such a conclusion in
that paper even for the tasks it considers, let alone in any
more general sense) is lazy.
- there is a fair amount of sloppiness in the writing, including
multiple typos and cases where text and figures don't agree (some
detailed below). This does not build confidence in the
meticulousness of the experiments.
More specific comments:
Synthetic data (sec 4.1): I am not sure what
to conclude from this experiment. It reads as simply presenting the
behavior of the proposed approach, without any critical evaluation of
whether that behavior is good. Eg, Fig 1b describes aleatoric
uncertainty as following an 'increasing trend' (a bit of a stretch
given that the majority of steps are decreasing), and 1c describes
predictive uncertainty as 'mostly constant' (though the overall shape
actually looks very similar to 1b) -- but why is it obvious that
either of these behaviors is correct or desirable? Even if this
behavior qualitatively matches what you'd get from a more principled
Bayesian approach (I was expecting the paper to argue this but it
doesn't), a simple synthetic task should be an opportunity for a real
quantitative comparison: how closely do the dropout uncertainties
match what you'd get from, e.g., HMC, or even exact
Bayesian linear regression (feasible here since the true function is
linear -- so concrete dropout on a linear model should be a fair
comparison).
UCI (sec 4.2): the experimental setup is not clear to me -- is the
'dropout' baseline using some fixed probability (in which case, what
is it?) or is it tuned by grid search (how?). More fundamentally,
given that the goal of CDropout is to produce calibrated uncertainties,
why are its NLL scores (fig 2, a reasonable measure of uncertainty)
*worse* than standard dropout in the majority of cases?
MNIST (sec 4.3): again no real comparison to other approaches, no
evaluation of uncertainty quality. The accuracy 'matches' hand-tuned
dropout -- does it find the same probabilities? If so this is worth
saying. If not, seems strange to refer to the converged probabilities
as 'optimal' and analyze them for insights.
Computer vision (sec 4.4): this is to me the most promising
experiment, showing that concrete dropout matches or slightly beats
hand-tuned dropout on a real task. It's especially nice to see that
the learned probabilities are robust to initialization since this is a
natural worry in non-convex optimization. Some things that seem
confusing here:
- what is the metric "IoU"? presumably Intersection over Union but
this is never spelled out or explained.
- the differences in calibration between methods are
relatively small compared to the overall miscalibration of all the
methods - why is this?
- text says "middle layers are largely deterministic" which seems
inconsistent with Fig 8c showing convergence to around p=0.2?
(nitpick: "Table" 2 is actually a figure)
Model-based RL (sec 4.5): It's nice that the model matches our
intuition that dynamics uncertainty should decrease with
experience. But it sounds like this tuning doesn't actually improve
performance (only matches the cumulative reward of a fixed 0.1
dropout), so what's the point? Presumably there should be some
advantages to better-calibrated dynamics uncertainty; why do they not
show up in this task?
The method proposed in this paper seems promising and potentially
quite useful. If so it should be possible to put together a careful
and clear evaluation showing compelling advantages (and minor costs)
over relevant baselines. I'd like to read that paper! But as currently
submitted I don't think this meets the NIPS bar.
UPDATE: increasing my rating slightly post rebuttal. I still think this paper is marginal, but the clarifications promised in the rebuttal (describing the tuning of dropout baselines, and the problems with Gaussian dropout, among others) will make it stronger. It's worth noting that recent work (Molchanov et al., ICML 2017 https://arxiv.org/abs/1701.05369) claims to fix the variance and degeneracy issues with Gaussian dropout; this would be an interesting comparison for future work.