Google Releases Gemini 1.5

Google announced Gemini 1.5 in February 2024 with an experimental context window of up to one million tokens. That was far larger than the context sizes commonly offered by mainstream language models at the time.

A long context lets one request include entire books, large codebases, long transcripts, or substantial video representations. Gemini 1.5 used a mixture-of-experts architecture, where only selected parts of the network are activated for each token, increasing total model capacity without using every parameter for every computation.

Large context does not eliminate retrieval problems. A model can technically accept a million tokens yet still miss important details or spend substantial compute processing them. The release shifted competition from model size alone toward how much information a model could handle coherently in one session.