Batch normalization provably avoids ranks collapse for randomly initialised deep networks

Meta Review

This paper studies the effect of adding batch normalization (BN) layers on the rank of the data matrix as it passes through the network. The authors prove the rank does not collapse to rank one, which is what occurs when the network does not have BN layers. The theoretical results are accompanied by experiments. Reviewers felt that the paper made an important contribution to understanding models that use BN.