We thank the authors for an interesting rebuttal and paper which sparked discussion among the reviewers. The reviewers have updated their responses accordingly. Although there was discrepancy in the scores, it was agreed that the paper is well-motivated and presents an important problem domain. The main issue with this work was whether one can really learn this disentangled representation in practice. However, the paper presents an excellent proof of concept.