Build a Shakespeare GPT from Scratch in JavaScript

Summary

Pixel shares a hands-on journey building a tiny GPT from scratch in JavaScript, connecting neural networks, backpropagation, and Transformer components into a language model. The project trains on the Tiny Shakespeare corpus of roughly 1.1 million characters and a 65-character vocabulary, offering the community a practical look at how text generation works.

DEV Community

From Zero to Shakespeare GPT: I Built a Language Model in JavaScript

By the time I reached this stage, I had spent a considerable amount of time learning how neural networks work. I had implemented neurons, trained MLPs, worked through backpropagation, experimented with activation functions, and eventually built the components of a Transformer. But there was still one thing missing. I hadn't brought everything together into a language model that could learn from actual text. That was the goal from the beginning. I wanted to take everything I had learned and build a tiny GPT from scratch using JavaScript. Not a chatbot powered by an API. Not a wrapper around a pretrained model. An actual neural network with its own trainable weights, attention mechanism, training loop and text-prediction pipeline. And for the dataset, I chose Shakespeare. 1. Why Shakespeare? I needed a real text corpus to train the model. I didn't want to keep testing everything with small artificial examples. I wanted the model to learn from actual sequences of language. Shakespeare was a good starting point for a few reasons: It provides a substantial amount of continuous text. It contains dialogue, punctuation, names and varied sentence structures. Its relatively small character vocabulary makes it manageable for a first character-level language model. The generated output is fun to inspect, even when the model makes mistakes. I used the Tiny Shakespeare corpus from Andrej Karpathy's "char-rnn" project. The dataset contains roughly 1.1 million characters and, with our character-level tokeniser, produces a vocabulary of 65 unique characters. That meant our model had 65 possible character IDs to work with. This was the first time our learning experiments were connected to a meaningful language dataset. 2. Designing the project before writing more code One thing I had learned from the earlier experiments was that putting everything into one enormous file would make the project difficult to understand. I wanted the architecture to reflect the actual concepts we were learning. So I separated the implementation into modules. tiny-shakespeare-gpt/
│
├── data/
│ ├── shakespeare.txt
│ ├── nextTokenDataset.js
│ └── prepareData.js
│
├── tokenizer/
│ └── tokenizer.js
│
├── tensor/
│ ├── Tensor.js
│ ├── operations.js
│ ├── autograd.js
│ └── random.js
│
├── layers/
│ ├── embedding.js
│ ├── positionalEmbedding.js
│ ├── linear.js
│ ├── layerNorm.js
│ ├── gelu.js
│ ├── causalAttention.js
│ ├── selfAttentionHead.js
│ ├── multiHeadAttention.js
│ ├── feedForward.js
│ └── transformerBlock.js
│
├── models/
│ └── gptModel.js
│
├── training/
│ ├── loss.js
│ ├── optimizer.js
│ ├── trainStep.js
│ ├── trainLoop.js
│ └── checkpoint.js
│
└── generation/
└── generate.js
Each folder had a responsibility. The tokeniser handled text. The tensor module handled numerical operations and gradients. The layers contained the neural-network components. The model assembled the architecture. The training module handled learning. It might seem like ordinary software organisation, but it mattered a lot here. When something went wrong, I could investigate a particular operation or layer instead of debugging one massive file. The project is split into one folder per concept: data, tokenizer, tensor, layers, models, training and generation. 3. Building the tensor engine This was one of the most important implementation decisions. Machine-learning frameworks normally provide tensors and automatic differentiation. I wasn't using one. So I needed my own basic Tensor structure. At its simplest, a Tensor stores: Numerical data. Its shape. A gradient array. References to the operations that produced it. A backward function. Conceptually: class Tensor {
constructor(data, shape) {
this.data = data;
this.shape = shape;
this.grad = new Array(data.length).fill(0);

this.parent = [];
this.backward = () => {};

}
}
The parent references allow us to track the computation graph. When we perform an operation...
Build a Shakespeare GPT from Scratch in JavaScript
Powered by Readybot.io

The latest from Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …