GPT-3 Demonstrates the Scale of Large Language Models

OpenAI introduced GPT-3 in 2020 with 175 billion parameters, more than two orders of magnitude larger than GPT-2. The model used the same basic transformer principle—predicting the next token—but scaled data, model size, and computation dramatically.

A surprising capability was few-shot learning through prompts. Instead of retraining the network for every task, users could provide a few examples in the input and ask the model to continue the pattern. GPT-3 could generate prose, translate simple text, write code-like snippets, and answer questions with no task-specific weight update.

The model was also expensive to train and serve, and it confidently generated false information. GPT-3 made clear that scale could produce broad capabilities, while raising questions about energy use, training data, cost, reliability, and access to increasingly large models.