NeurIPS 2019
Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
Reviewer 1
1) This paper is clearly written. The motivating example in its introduction makes me believe that tie-aware ranking is crucial for crowdsourcing problems. 2) Traditional methods merely adopt the strong signal(sparse parameter) for model prediction and structural learning. Different from this routine, their proposed method explicitly separates the strong signals and weak signals, then uses strong signals to learn a semantic structure as the outlier indicator and combines both the weak and strong signals to do a fine-grained prediction. As pointed out in the work,its helps to decouple the model selection and model prediction process.
Reviewer 2
This paper proposes a novel and unified model for individualized learning, partial ranking, and abnormal detection, which simultaneously solves important problems in personalized learning. In general, it is well-written with a good organization and easy to follow. I think this work could benefit our community in its style as it blends intuitions with solid mathematical machinery. This is what I hope to see most in the application papers. As I have mentioned in the contribution part, I particularly enjoy the optimization rules induced by the split-LBI method, which provides a dynamic way to learn and select the correct model.
Reviewer 3
The formulation of annotators’ ranking decision, i.e. integrating personalized preference, abnormal deviations, common score, and random noise simultaneously from a probabilistic view, is well-designed and novel. The proposed model can give user-specific partial ranking prediction and abnormal user detection simultaneously, which is an early trail in the direction. It is also pleased to see that iSplitLBI achieves reasonable performance in discovering incomparable pairs. The proposed model unites simplicity and effectiveness, which makes it possible to be widely adopted for future work.