Originality: PNS and related algorithms have not been evaluated for synthesis planning since work by Heifets and others several years ago. Revisiting this class of algorithms and proposing modifications to improve performance in multi-step synthesis planning is nice to see. Quality: The empirical evaluation is not as strong as it could be, but the conceptual contribution of this work is still important for the problem of synthesis planning. Clarity: The description of algorithms in 254-266 and elsewhere is not complete enough to reimplement the models and baselines. The dataset split, details of template extraction, network training, etc. is not provided either and the code is not available. Significance: The novelty of the modifications to the algorithm may be minor, but evaluating it in the context of this problem is important. However, the empirical evaluation is not as informative as it would need to be to support the claims of DFPN-E and MCTS being the strongest candidates for chemical synthesis planning. Specific comments: - The sentence “However, as discussed by [14], their search performance in RA is not well understood” prompted me to follow that reference, which does not really describe this. - It would make more sense for Figure 1 to show a retrosynthetic reaction rule - Reference https://doi.org/10.1021/acscentsci.9b00055 provides a different search strategy and might belong in the related work - The comment on line 189 (“However, most of the reaction rules have only one precursor”) and statistics of the rule sets described on lines 267-269 is surprising – it’s unexpected for there to be more unimolecular reactions than bimolecular reactions. - On line 235, was the data randomly split? - Line 241-242: Considering molecules in the training set as being starting materials seems overly generous. The molecules in the test set will be very similar to the training set if a random split was used, so the necessary pathways are quite short (as confirmed by the average pathway length in Table 1 – I suspect that a few outliers bring this average up). The histograms of the number of nodes shows a massive number require a single node. Given that the emphasis of this work is on multi-step synthetic planning, it would be better to exclude such simple test cases. --- Based on the authors' response (intent to release code, acknowledgement of reproducibility issue, and additional evaluation focusing targets with pathway lengths >= 3), a revised version would likely raise my score above the acceptance threshold

This reviewer really enjoyed reading this paper. PNS is a very interesting search algorithm which fell a bit behind MCTS after the introduction of progressive bias/PUCT. The difficulty of combining PNS with heuristics is a problem that also this reviewer worked on without much success, which is why this reviewer particularly appreciates this contribution. The manuscript is overall well written, and well motivated, however, a few question remain, which should be addressed before accepting the paper. Quality: In their MCTS baseline, did the authors phrase it as in Winands et al. "MCTS Solver", and Segler et al. Nature 2018, where a high or even infinite reward is given for proved states (solved and within tree) and a standard reward of 1 is given for states which are solved in the rollout? Did they consider giving partial rewards for partially solved states which is hinted at but not very well described in this 2018 Nature paper? This will likely accelerate MCTS convergence. Along these lines, do the authors stop PNS when a proof has been found, or do they continue to find more/better solutions? Is search speed the most important criterion, or are there other criteria which can be more important? How easy is it to incorporate other costs into PNS, e.g. cost of starting materials?

This paper proposes a variant of DFPN to solve the lopsided search space by applying simple heuristics. DFPN-E estimates the difficulty of finding a proof and utilizes it as the edge cost from the OR node to the AND node. Besides, two thresholds are used as constraints to the size of the search tree. The paper is well-written, especially the comprehensive overview of retrosynthetic analysis. The logic is clear and notations are clean. Nice work. Generally, I think the story is well-motivated and conclusions are well supported by the empirical experiments, but the contribution is a little incremental and engineering. It may not attract that many audience in the ML community, and personally, I would expect more explorations on the algorithm, not just two simple heuristics. Since the current model can reach the state-of-the-art performance, it would make a more interesting story by considering the more advanced strategies (like mentioned in the Conclusion Section). Other comments: (1) In line 72, should use “lose” instead of “loss”. (similarly to the remaining ones)