All the reviewers agree that the submission makes valuable algorithmic and theoretical contributions to learning (near) optimal policies for MDPs assuming a generative model is available. Thus, I am glad to recommend the paper for acceptance.