L LearnGPT
01 / 42
VISUAL BUILD · EN

An executable course, now visible

Build GPT, one number at a time.

From raw text to a trainable decoder-only Transformer. Every transformation shows the tensors, the reasoning, and the Python code actually introduced.

Start from text
input.txt“We build GPT.”
01 Text text → [T]
02 Batches [tokens] → [B, T]
03 Bigram [B, T] → [B, T, V]
04 Embeddings [B, T] → [B, T, C]
05 Attention [B, T, C] → [B, T, C]
06 Transformer [B, T, C] ⇢ [B, T, C]
07 Training loss → weights → token
model.ptready to generate
01No jumps

Every shape follows from the previous one.

02Real code

Diffs come from all 42 snapshots.

03Readable numbers

Small examples, authentic structure.

42 CHECKPOINTS · 7 TRANSFORMATIONS

Scroll through the build. The model changes with you.

Lessons stay below the surface: what you see is one system growing.

01 Text

Read the text

Load a plain-text dataset and inspect the exact characters the model will learn from.

01_read_text.py
02 Text

Build a character tokenizer

Collect the unique characters and assign a stable integer ID to each one.

02_character_tokenizer.py
03 Text

Encode and decode

Turn text into token IDs, then reconstruct the original text from those IDs.

03_encode_decode.py
04 Text

Extract the tokenizer module

Move vocabulary, encode, and decode into reusable functions shared by later checkpoints.

tokenizer.py
05 Batches

Split training and validation data

Reserve one part of the token sequence for learning and another for measuring generalization.

tokenizer.py
06 Batches

Create inputs and targets

Shift one token sequence by a single position so every input prefix has a correct next-token target.

tokenizer.py
07 Batches

Sample random training examples

Choose valid start positions and slice many context windows from the training sequence.

tokenizer.py
08 Batches

Group examples into a Python batch

Collect multiple input and target windows so they can be processed together.

tokenizer.py
09 Batches

Move the batch into PyTorch

Convert nested Python lists into integer tensors with shape [batch, context].

tokenizer.py
10 Batches

Extract the batching module

Turn window sampling and tensor construction into one checked, reusable create_batch function.

batching.py
11 Batches

Verify the PyTorch environment

Check tensor shapes, dtypes, and the selected compute device before building the first neural model.

batching.py
12 Bigram

Build the first bigram model

Use an embedding table that maps each current token directly to logits for the next token.

batching.py
13 Bigram

Measure bigram loss

Flatten batch and time into independent predictions, then compare logits with targets using cross entropy.

batching.py
14 Bigram

Run the first training loop

Clear gradients, run the forward pass, backpropagate the loss, and update model weights.

batching.py
15 Bigram

Generate with the bigram model

Take the final logits, apply softmax, sample a token, append it, and repeat.

batching.py
16 Bigram

Expose the bigram limitation

Compare sequences that share the same final token and observe that the model predicts the same distribution.

batching.py
17 Embeddings

Introduce token embeddings

Represent every token as a learned C-dimensional vector, then project that vector back to vocabulary logits.

batching.py
18 Embeddings

Add position embeddings

Look up a learned vector for each position and add it to the corresponding token embedding.

batching.py
19 Attention

Build one causal self-attention head

Project embeddings into query, key, and value vectors, mask future positions, and compute contextual outputs.

batching.py
20 Attention

Run multiple attention heads

Compute several independent attention views and concatenate their outputs along the embedding dimension.

batching.py
21 Attention

Project multi-head attention output

Add a learned C-to-C projection after concatenating all attention heads.

batching.py
22 Transformer

Add the first residual connection

Add the original embeddings to the attention output without changing [B, T, C].

batching.py
23 Transformer

Normalize before attention

Apply LayerNorm to each token vector before sending it through multi-head attention.

batching.py
24 Transformer

Add the feed-forward network

Expand each token from C to 4C, apply GELU, and project it back to C.

batching.py
25 Transformer

Assemble a TransformerBlock

Package pre-normalized attention, feed-forward processing, and two residual additions into one module.

batching.py
26 Transformer

Stack multiple TransformerBlocks

Pass the same [B, T, C] representation through a sequence of independently learned blocks.

batching.py
27 Transformer

Add the final LayerNorm

Normalize the output of the Transformer stack before projecting it to vocabulary logits.

batching.py
28 Training

Train the Transformer

Connect batching, the full Transformer forward pass, cross entropy, backward, and optimizer updates.

batching.py
29 Training

Estimate training and validation loss

Average loss across several batches in evaluation mode without computing gradients or changing weights.

batching.py
30 Training

Save and reload a checkpoint

Persist model weights, optimizer state, configuration, tokenizer information, and the current step.

batching.py
31 Training

Generate from a checkpoint

Reconstruct the model, restore learned weights, encode a prompt, and run generation without training.

batching.py
32 Training

Control sampling

Adjust temperature and top-k to reshape the next-token distribution before multinomial sampling.

batching.py
33 Training

Keep the best checkpoint

Save a new best model only when its validation loss improves.

batching.py
34 Training

Schedule and protect optimization

Add AdamW configuration, learning-rate warmup and cosine decay, plus gradient clipping.

batching.py
35 Training

Add dropout and weight tying

Regularize intermediate activations and share parameters between token embeddings and the output head.

batching.py
36 Training

Create AdamW parameter groups

Apply weight decay to matrix-like weights while excluding biases and normalization parameters.

batching.py
37 Training

Accumulate gradients

Split one effective batch into micro-batches, scale each loss, and update weights only after accumulation.

batching.py
38 Training

Centralize config and resume training

Move scattered settings into configuration objects and continue from the step stored in a checkpoint.

batching.py
39 Training

Project only the last token during generation

Keep full logits for training but apply the output head only to the final position during inference.

batching.py
40 Training

Use optimized scaled dot-product attention

Optionally route causal attention through PyTorch’s fused scaled_dot_product_attention implementation.

batching.py
41 Training

Add performance flags and distributed training

Introduce optional mixed precision, torch.compile, and distributed data parallel controls.

batching.py
42 Training

Assemble the final LearnGPT project

Connect GPT-2 BPE tokenization, FineWeb-Edu data, the complete model, training, checkpoints, and generation.

batching.py

LIVE MODEL BLUEPRINT

The architecture assembles as the code grows.

01 Token IDs
01 Text

Read the text

Load a plain-text dataset and inspect the exact characters the model will learn from.

From language to indices

Text becomes numbers

text → [T]
Learn
L4e7a1r9n32
471932
  1. 01
    Text“Learn”
  2. 02
    SymbolsL · e · a · r · n
  3. 03
    Token IDs[T]
Why it matters

Every later tensor begins here. Seeing the raw source first keeps the path from language to numbers explicit.

study/lessons/01_read_text.py

Synchronized code

What changes

01 / 42
study/lessons/01_read_text.py
===================================================================--- /dev/null	before+++ study/lessons/01_read_text.py	lesson@@ -0,0 +1,31 @@+"""+Changes compared with the previous file:+- This lesson script uses the English project layout and imports lesson-specific+  snapshot code.+- It belongs to lesson 01 of the guided LearnGPT path.++File purpose:+- Run the lesson example in a reproducible way.+- Print the relevant intermediate values, tensor shapes, losses, or generated+  text for inspection.+"""++from pathlib import Path+++PROJECT_DIR = Path(__file__).resolve().parents[2]+DATASET_PATH = PROJECT_DIR / "data" / "raw" / "fineweb_edu_sample.txt"+++def main():+    text = DATASET_PATH.read_text(encoding="utf-8")++    print("File read:", DATASET_PATH)+    print("Number of characters:", len(text))+    print()+    print("First 500 characters:")+    print(text[:500])+++if __name__ == "__main__":+    main()
added removed context

DIDACTIC SCALE ≠ ORIGINAL WEIGHTS

The same architecture, small enough to see.

ConfigurationCHeadsBlocksV
LearnGPT smoke644250,257
GPT-2 Small768121250,257

LearnGPT reconstructs the path and components of a decoder-only GPT. Small values keep tensors and operations inspectable; GPT-2 Small uses the same family of components at a much larger scale.