How massive is GPT-3, and why did its size matter so much?
175 billion parameters. It would take a human roughly 355 years to count them one per second.
The answer
Released by OpenAI in June 2020, GPT-3 contained 175 billion parameters - over 100 times larger than GPT-2 - and was trained on 570 GB of filtered text from Common Crawl, WebText2, Books1, Books2, and Wikipedia. Its most remarkable property was 'few-shot learning': the ability to perform tasks like translation, question-answering, and code generation from just a handful of examples in the prompt, without any fine-tuning. Training GPT-3 is estimated to have cost between $4.6 million and $12 million in compute alone, requiring roughly 3,640 petaflop-days of computation.
Source: arXiv
- gpt-3
- openai
- scaling
- large language models
More from AI & Machine Learning
- What radical question did Alan Turing ask in 1950 that launched the entire field of artificial intelligence?
- Can a machine fool you into thinking it's human - and does that mean it's truly intelligent?
- What happened at a small Dartmouth workshop in 1956 that gave artificial intelligence its name?
- How did a 1958 invention called the Perceptron convince people that machines could learn like brains?
- Could a 1966 chatbot really make people believe they were talking to a therapist?
- What was the first robot that could 'reason' about its own actions?