How did ResNets solve the paradox of deeper networks performing worse - by learning to do nothing?
The secret to building 152-layer neural networks? Give each layer a shortcut to skip itself.
The answer
In December 2015, Kaiming He and colleagues at Microsoft Research introduced Residual Networks (ResNets), which use 'skip connections' that allow input to bypass one or more layers and be added directly to the output. This elegant solution addressed the degradation problem, where very deep networks paradoxically performed worse than shallower ones despite having more capacity. ResNets enabled training of networks with 152 layers - eight times deeper than previous architectures - and won the ILSVRC 2015 competition with a 3.57% top-5 error rate, surpassing human-level performance. The paper has accumulated over 200,000 citations, making it one of the most influential papers in all of computer science.
Source: arXiv (He et al., 2015)
- residual networks
- deep learning
- skip connections
More from AI & Machine Learning
- What radical question did Alan Turing ask in 1950 that launched the entire field of artificial intelligence?
- Can a machine fool you into thinking it's human - and does that mean it's truly intelligent?
- What happened at a small Dartmouth workshop in 1956 that gave artificial intelligence its name?
- How did a 1958 invention called the Perceptron convince people that machines could learn like brains?
- Could a 1966 chatbot really make people believe they were talking to a therapist?
- What was the first robot that could 'reason' about its own actions?