NeurIPS 2020

Learning Differentiable Programs with Admissible Neural Heuristics

Meta Review

Generating programs has been a long-standing problem in AI for many decades. Reviewers found valuable the fact that this approach combines prior literature on heuristic search with a modern neural networks approach to improve performance. Reviewers also found that methods which combine discrete and continuous parts of programs are in short supply, making this of wide interest and likely to spur further research. The fact that the approach is in a sense straightforward conceptually but not obvious while being able to perform when more complex methods like TerpreT are not suitable was also pointed out as a significant advance. Reviewers wished to see more related to the interpretability of the acquired models. Just because a model is not a neural network, one cannot say it is interpretable, for example a large decision tree is no more interpretable than a deep network. Evaluating this aspect would significantly strengthen the manuscript. Reviewers were concerned with the parameterization of the LSTMs, the fact that hyperparameters must be tuned to match the complexity of the expected programs for each dataset. Reviewers also wished to see more information on the type system used. Reviewers pointed out that the paper would be strengthened by a comparison against MCTS, and in particular against TerpreT, even if that latter comparison will necessarily be limited, and provided ideas on how to do so. Even the lowest-ranking reviewer agreed that with such an experiment they would be happy to accept, and even the highest-ranking reviewer agreed that such an experiment would significantly improve the paper. Differences in the review scores boiled down essentially to what one might expect these results to show and how comfortable one is to publish without them, with three out of four reviewers being comfortable doing so. We highly encourage the authors to include these experiments as they strengthen the already significant conceptual contribution.