The paper proposes a way to learn model confidence, and shows that the proposed method better predicts mis-classifications ("failures"). All the reviewers voted to accept the paper. The reviewers raised a couple of questions initially: the authors have already addressed some of these questions in their rebuttal, which is appreciated. I encourage the authors to take into account the remaining suggestions when preparing the final version.