What simple trick lets neural networks train 14 times faster by normalizing each layer's input?

The answer

In 2015, Sergey Ioffe and Christian Szegedy at Google introduced batch normalization, a technique that normalizes the inputs to each layer of a neural network across a mini-batch of training examples. By reducing what they called 'internal covariate shift,' batch normalization allowed networks to use much higher learning rates and be less sensitive to initialization, cutting training time dramatically - the original paper showed their network reached the same accuracy as the previous state of the art 14 times faster. The paper has been cited over 45,000 times and batch normalization became a near-universal component in deep learning architectures.