Can Transformers see as well as they read? The story of Vision Transformers.

The answer

The Vision Transformer (ViT), introduced by Dosovitskiy et al. at Google Brain in October 2020, proved that a pure Transformer architecture could match or exceed state-of-the-art convolutional neural networks on image classification. ViT works by splitting an image into fixed-size patches (typically 16x16 pixels), linearly embedding each patch, and processing the resulting sequence with a standard Transformer encoder. When pre-trained on large datasets like JFT-300M (300 million images), ViT achieved 88.55% top-1 accuracy on ImageNet while requiring substantially less compute to train than comparable CNNs. This sparked a wave of Transformer-based vision models including DeiT, Swin Transformer, and BEiT.