Build a 5.9M-Parameter LLM on a Free GPU
Resumo
Pixel shares a beginner-friendly tour of building a 5.9-million-parameter storytelling model from scratch on a free Google Colab GPU in about five minutes. Through a simple story analogy, the announcement explains training data, weights, loss, and gradient descent, making LLMs less mysterious and easier for the community to explore.
DEV Community
Build an LLM From Scratch in 5 Minutes on a Free GPU: I Trained a 5.9-Million-Parameter Storyteller, and Here Is What It Wrote
Part 1 of ten. Each post is a short, plain-words tour of one lesson from e=mc², a free build-an-LLM-from-scratch course; the full lesson, notebook and numbers are on the site. Most people think building a language model takes a lab, a budget and a team. Last month I built one from nothing, on a free Google Colab GPU, in about five minutes end to end. It has 5.9 million parameters, it learned to write bedtime stories, and it is sitting on the Hugging Face Hub right now where you can run it. I want to show you what it wrote, what the numbers mean, and why doing this once makes every headline about trillion-parameter models stop sounding like magic. Meet Momo and Ella The lesson is told through two characters, because the ideas are simpler than the vocabulary. Momo is a young monkey learning to tell stories. Inside his head are millions of tiny dials, all pointing somewhere random, so his first stories are nonsense. Ella is an elephant, and elephants never forget. Every night she reads Momo a story one word at a time, stops before each word, lets him guess, and tells him how wrong he was. Then she nudges his dials a little. Thousands of nights later, Momo tells stories. That is the whole of machine learning in one paragraph. Momo is the model. His dials are the weights (5.9 million of them here). Ella's stories are the training data. "How wrong were you" is the loss. Nudging the dials is gradient descent. Everything else is detail, and the lesson decodes about forty of those details one at a time. What a language model does, in one sentence Given the words so far, guess the next one. A story is that guess repeated. A chat answer is the same thing with your question as the words so far. When Momo continues "Once upon a time, a little", four things happen. The sentence is bitten into tokens, pieces of text with a number on each (our vocabulary has 4,096). Each token becomes a spot on a meaning map, a list of 256 numbers where similar words sit close together. Then every token looks back at the tokens before it and decides how much to listen to each: "capital" finds "France" and borrows some of its meaning. That is attention, from the 2017 paper that started all this, and it runs six times in a row, four heads each. Finally the last token's sharpened meaning is scored against every word in the vocabulary, and one is picked. That is it. That is also, with more stories, more blocks and more GPUs, what the big ones do. The numbers, honestly 5.9 million parameters: 6 blocks, 4 attention heads, 256-wide, a vocabulary of 4,096 pieces learned from the stories themselves. 1,500 training steps, each a batch of 32 windows of 256 tokens. 1.2 minutes of training on Colab's free T4 GPU; about five minutes for the whole notebook; about two minutes on an Apple Silicon Mac. Perplexity 9 on 200 stories the model never saw. Perplexity is "between how many words is Momo still choosing". Random guessing would be 4,096. Watching that number fall from 4,096 to 9 in front of you is the moment the field stops being mysterious. The 200 quiz stories were held out before training, so nothing the model memorised can flatter the score. An earlier draft got that wrong, and fixing it is why I trust the 9. What it wrote Given "Lily had a red balloon.", one run produced this: She ran to her mom and gave her a big bite. They both ate the balloon and had the balloon. It was shiny and yellow. They ran to the park to play with. Grammatical, kid-flavoured, slightly unhinged. That is exactly what 5.9 million parameters and two minutes buy, and I think it is wonderful. Then I gave it "Once upon a time, a little monkey", and it wrote this: Momo was sad because he could not swim like Bo the bear cub. He sat in the tall grass with his feet in the water. Ella said, "Try again, little one. Every time you try, you learn a little more." She held him up with her trunk. Momo kicked and kicked. Soon he could swim all by himself. The end. Lovely, right? It is also, character for...

Powered by Readybot.io