Why was BERT considered a breakthrough in understanding language?
Before BERT, AI read sentences like a book - only left to right. BERT read them all at once.
The answer
Released by Google AI in October 2018, BERT (Bidirectional Encoder Representations from Transformers) was the first model to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers. Trained on 3.3 billion words from English Wikipedia and BooksCorpus, BERT achieved state-of-the-art results on 11 NLP benchmarks simultaneously. Its masked language modeling approach - randomly hiding 15% of input tokens and predicting them - gave the model a far richer understanding of language than previous left-to-right methods.
Source: arXiv
- bert
- nlp
- language models
More from AI & Machine Learning
- What radical question did Alan Turing ask in 1950 that launched the entire field of artificial intelligence?
- Can a machine fool you into thinking it's human - and does that mean it's truly intelligent?
- What happened at a small Dartmouth workshop in 1956 that gave artificial intelligence its name?
- How did a 1958 invention called the Perceptron convince people that machines could learn like brains?
- Could a 1966 chatbot really make people believe they were talking to a therapist?
- What was the first robot that could 'reason' about its own actions?