OpenAI Releases GPT-2 in Stages

OpenAI released progressively larger versions of GPT-2 during 2019, culminating in the full 1.5-billion-parameter model in November. The organization initially withheld the largest weights because of concerns about misuse.

GPT-2 used the transformer architecture and was trained with a simple objective: predict the next token in large amounts of Internet text. Without task-specific retraining, it could continue stories, answer questions in context, summarize, and imitate styles when prompted with examples.

The model’s coherent long-form text surprised many developers at the time. The staged release also started a debate that would intensify with later AI systems: when a model is broadly useful and potentially misusable, should its weights be published freely, restricted, or exposed only through an API?