DEV Community
You Will Understand 10 Years of LLMs
My new video, In 19 Minutes, You'll Understand 10 Years of LLMs, follows one question through every era of language models: "What is the capital of France?" This post is the developer version. It goes through every chapter of the video with the actual terms and numbers. Where the video shows a model's output, we ran an open model ourselves, so you get the code to reproduce it. pip install torch transformers accelerate
All outputs below are what we got on an RTX 4070 Ti. Yours may differ slightly with hardware and precision. TL;DR timeline Year Milestone What it fixed What it left open 1948–51 Shannon's next-letter game "predict what comes next" as a measurable idea no machine could play it well 2013 word2vec words become vectors that capture meaning one vector per word, whatever the context 1997–2014 RNN, LSTM, seq2seq read sentences, translate one fixed-size summary per sentence 2014 attention (Bahdanau) the decoder looks back at every word still reads one word at a time 2017 the Transformer no recurrence: parallel training on GPUs writing is still one token at a time 2018–19 BERT, GPT-1, GPT-2 pretraining on raw text the model continues text, it doesn't answer 2020–22 scaling laws, GPT-3, Chinchilla predictable gains from size and data still not helpful 2022 InstructGPT / RLHF, ChatGPT a second training stage teaches it to answer serving at scale is expensive 2023–24 LLaMA, KV-cache sharing, mixture of experts cheaper, faster, longer contexts running out of human text 2024–25 o1, DeepSeek-R1 models learn to think before answering thinking helps, doesn't guarantee 2026 agents, hybrid attention plan, use tools, act 0. The quiz page (Experiment 1) Before chatbots, a language model's only job was to continue text. Ask GPT-2 XL (1.5B parameters, 2019): from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
name = "openai-community/gpt2-xl"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name).eval()
ids = tok("What is the capital of France?", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=False, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:]))
The capital of France is Paris.
What is the capital of the United States?
The capital of the United States is Washington, D.C.
What is the capital of the United Kingdom?
The capital of the United Kingdom is London.
What
It's not answering; it's writing a quiz page. We'll come back to why. 1. Before 2017: the road to attention Shannon's game (1948, 1951) Claude Shannon asked how predictable English is. Letters don't come at random: counted in Pride and Prejudice, after a T comes an H 29% of the time; after a Q, a U 99.85% of the time. In 1948 he generated text that follows these odds with a book: pick a letter, open another page, find that letter, copy the one after it, repeat. Letter pairs give almost-words; word pairs give "The head and in frontal attack on an English writer that the character of this point…" Not English, but closer. The more context, the closer. In 1951 he made it a game with people: guess the next letter. On one passage they were right on the first try 79 times out of 102. He estimated that a letter of English carries about 1 bit of new information, against about 4.75 bits if letters were random. That's the objective LLMs still train on: predict the next token. Words become vectors (2013) A word ID (king = #4812) says nothing about meaning. word2vec (Mikolov et al., 2013) gives each word a vector of a few hundred numbers (300 in the classic setup), learned by predicting a word's neighbours over billions of words. Words used in similar contexts end up close, and the space has structure: king − man + woman ≈ queen (the nearest word, excluding the three inputs). The limit: one vector per word. The "bank" of a river and the bank with your money get the same vector. RNN, LSTM, seq2seq (1997–2014) A recurrent neural network reads one token at a ...
Powered by Readybot.io
External Link
Platform
You are about to visit:
https://example.comThis will open in a new tab. Make sure you trust this link.