What is self-supervised learning and why did it unlock modern AI?

The answer

Self-supervised learning is a training paradigm where models generate their own labels from raw data, eliminating the need for expensive human annotation. In NLP, this takes the form of predicting masked words (BERT) or next tokens (GPT). In computer vision, methods like contrastive learning (SimCLR, 2020) and masked autoencoders (MAE, 2021) teach models to understand images by reconstructing corrupted or partial inputs. This approach enabled training on virtually unlimited internet data - GPT-3 consumed 570 GB of text, and CLIP learned from 400 million image-text pairs - scaling that would be impossible with manually labeled datasets.