NeurIPS 2019
Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
Paper ID:7294
Title:Variance Reduction in Bipartite Experiments through Correlation Clustering

Reviewer 1

This work addresses the task of designing experiments on bipartite graphs in the presence of SUTVA violation. Specifically, the authors propose an extension to the graph cluster randomization framework to the bipartite setting by casting the problem as correlation clustering. Propositions are provided that tie the variance of the causal estimate to the quality of the discovered solution. A heuristic is provided to solve the correlation clustering problem. Experiments are provided comparing the proposed method to balanced partitioning and random assignment which show the proposed method performing favorably. Overall, I think this is a nice, simple, solution to a problem that occurs a fair amount for practitioners. The paper is clearly written and motivated. The connection to correlation clustering is sensible and provides a decent interpretation of the results. I have two is that there are details missing from the experiment section that would improve reproducibility. Specifically the authors do not state how the estimation is carried out. It would be nice to see comparisons to both the estimator of Eckles, et al. (2017) and Gui, et al. (2015). A few comments / questions: 1. In the introduction the connection to optimal experimental design is alluded to. Having a more precise connection made within the paper would be both interesting and helpful. 2. It is not clear what the quality of the approximation is for the heuristic presented. It would be nice to see a discussion of this and the implications for the variance of the estimator. 3. What is the space complexity of the proposed heuristic? 4. Why are other correlation clustering heuristics not compared against in the experiment section? 5. There are no results given for the variance of the estimators / designs. Given that this is often a critical component of analysis for practitioners it would be great to estimates (and ideally coverage) for each of the shown methods, as well as a comparison that shows empirically the relationship between approximation quality of the correlation clustering heuristic and the resulting variance.

Reviewer 2

This work addresses an important and interesting setting, bipartite experiments, for causal inference in randomized experiments. In such a setting, treatment and control units are different from the outcome units. Unlike the prior work from Zigler and Papadogeorgou (2018) that considers a particular setting of partial interference, they consider the most general setting for which no immediate reduction to the standard causal setting exists. Moreover, they assume that an outcome unit’s outcome is determined by a weighted proportion of the treated diversion units in its exposure set. The main novelty is that they propose a new clustering objective, i.e., choose the clustering of diversion units that maximizes the empirical variance of the treatment exposure. To solve the optimization problem, they propose a scalable heuristic clustering algorithm. They validate the approach by running experiments on real data. The paper is well written and easy to follow. The reviewer thinks this work is interesting and insightful for future work in this direction.

Reviewer 3

The paper discusses the analysis of bipartite randomized experiments, where treatment is assigned to diversion units and outcomes are measured on outcome units. The ideas discussed propose the use of correlation clustering to identify clusters of diversion units to benefit the estimation of a treatment effect. A scalable algorithm is provided for the correlation clustering and there is a simulation example to compare the variance of estimates when using other approaches to clustering. The paper clearly demonstrates the benefits and popularity of bipartite experiments and explains the need for clustering. The level of novelty may be questioned as it is unclear of the benefits of the proposed algorithm versus other algorithms. There is some novelty to the ideas presented and the algorithm developed would be useful in various applications, but a more thorough explanation of novelty may be required. There is no conclusion section.