The entire review team is in agreement that this is a solid paper. It deals with a problem that is motivated, and contains results comprising strong theory and interesting experiments. Note: I have discounted the reviewer comment "Most of this paper seems very technical but the theorems do not qualify as theorems. "sufficiently large constants" and "sufficiently large n" appear in every theorem statement but the authors fail to specify how large is sufficient. These are necessary for theorems.". This is a standard practice in learning theory. We the authors to please incorporate the following in the camera ready: - Please do clarify in the main text on how to handle variable numbers of comparisons - See other excellent suggestions by reviewer 1 - Reviewer 1 brings up the important point that estimation of skills in practice may have more far reaching broader impact, and hence issues pertaining to the assumed models are important. The review team notes that the paper does an excellent job in discussing some of this in the broader impacts section pertaining to issues like injury to etc. Additionally on the technical front, we suggest the revision should recognize the limitation of using the simplistic vanilla BT models instead of models having covariates, or stochastically transitive models, or non-transitive models. These may anyways be exciting to the ML community as possible future work given the recent interest in such models in the community. - Please do clarify the issue pertaining to uniform distribution of skills. Please see "EDIT after response" in review 2. - Move some points about the motivation from the rebuttal to the main text in order to avoid concerns like what reviewers 3 and 4 brought up in their initial reviews.