I predicted this one wrong, so I chased it down
Removing positional embeddings costs about 0.04 nats, not a collapse
is all the loss went up when I removed the part that tells the model word order. Pooled over two seeds.
An ablation means removing exactly one part and measuring the damage. I removed the component that tells the model where each word sits, because the attention maths on its own has no idea about order. It is a weighted average, and averages don't care what came first.
My prediction: fluent nonsense. Right words, scrambled order.
learned_pos: false, same seed, same steps, same LR schedule, and the
same token stream: the ablation kernel mounts the baseline run's output so the data
is byte-identical rather than re-tokenized. One variable.
Predicted: severe degradation, attention being permutation-invariant.
Baseline, with positional embeddings
No positions: +0.039, and still coherent
NO POSITIONAL INFORMATION AT ALL
Tom found a shiny red box under the tree. He was very happy and said, "Wow, look at this box! It's so pretty!"
That is not word salad.
Where I was wrong
My reasoning had a hole in it. Attention really is order-blind on its own, but this model only ever looks backwards. The 5th word can see 5 words; the 50th can see 50. How much you can see is itself a clue about where you are, and the model learned to read it.
Causal masking leaks position. Prefix length varies monotonically with index, so a decoder-only model can recover positional information without any explicit encoding: the documented NoPE result (Haviv et al., 2022). It is specific to causal models; a bidirectional encoder stripped of positions really would be helpless.
Testing it instead of asserting it
Two explanations fit a small gap: the model ignores order, or it found order elsewhere. So I scored both models on normal text, then on the same text with the words shuffled. A model that ignores order barely notices.
The no-positions model is more order-sensitive than the baseline, not less. It is not order-blind in the slightest. It simply got the information another way.
And the number itself is softer than it looks
That "+0.039" is a single reading, and readings wobble. Across the last five measurements of the very same pair of runs, the gap ranges from +0.037 to +0.063, so quoting three decimal places implies a precision this experiment doesn't have. The honest claim is "roughly 0.04 to 0.06".
Eval noise is non-trivial relative to the effect: the baseline's own val loss moves ±0.013 between adjacent evals, and the measured gap spans +0.037 to +0.063 over the last five evals of one run pair. Reporting the final step alone overstates precision.
So I measured it twice
I retrained both models from scratch with a different random starting point. If the effect were a fluke of randomness, it would move around or vanish.
Both arms rerun at seed: 1338, identical otherwise, on the same token
stream.
| Seed | Baseline | No positions | Gap | Spread |
|---|---|---|---|---|
| 1337 | 1.5038 | 1.5504 | +0.0467 | 0.037 to 0.063 |
| 1338 | 1.5010 | 1.5409 | +0.0400 | 0.024 to 0.052 |
| Pooled | +0.043 | 0.024 to 0.063 |
The control that makes it believable
Look at the baseline column, not the gap. Two runs that differ only in randomness landed 0.003 apart, while the difference they're measuring is 0.04, more than ten times larger. That is what turns "a number I got once" into a real finding.
Between-seed variation on the baseline is 0.0028, an order of magnitude below the ~0.043 effect. The effect is real and reproducible; the imprecision is in the estimate, not the sign. Honest statement: positional embeddings are worth roughly 0.04 nats here. Two seeds do not give a confidence interval, but they do separate sign and rough magnitude from noise.
What it is, and what "from scratch" means
12.5M-parameter decoder-only transformer, no pretrained weights
The rule for this project: if it does the actual work, I write it. The thing that splits text into pieces, the maths that lets each word look at the words before it, the loop that nudges millions of numbers toward being less wrong. All by hand, with comments explaining why. It is deliberately tiny ("munti" is Filipino for tiny) so it trains on free hardware in under an hour.
A language model has exactly one job: given some words, put a probability on every possible next word. Chat, summarising, writing code: all of it is that single operation run in a loop, feeding its own output back in.
Nothing that performs the core computation is imported. The tokenizers
library supplies the BPE merge algorithm, but the vocabulary is trained here on
TinyStories, never downloaded. No pretrained weights, no
GPT2LMHeadModel. nanoGPT was read as a reference and reimplemented,
not vendored. Sized to fit one free session with room to spare, and kept
language-agnostic for a Tagalog re-run on the same code.
Decoder-only transformer. Token embeddings plus learned positional embeddings, six pre-LN blocks, final layernorm, and a weight-tied projection back to vocab logits. Cross-entropy against the sequence shifted by one.
Attention is the only place information moves between words. Every word asks a question ("what am I looking for?"), every earlier word advertises what it is, and the word takes a weighted blend of whatever matched. The MLP is where each word privately thinks about what it just gathered, with no talking to neighbours.
Each token projects to query, key and value; scores are QKᵀ/√d_head,
masked causally, softmaxed, and applied to V. Heads are a reshape, so it is one
batched matmul rather than a loop.
Three details that are not cosmetic
| Detail | What breaks without it |
|---|---|
1/√d_head scaling | Dot products grow with dimension, softmax saturates, gradients vanish. One division decides whether training works at all. |
| Pre-LN, not post-LN | Post-LN puts a layernorm on the residual path. Pre-LN keeps it a clean identity route from input to loss, so deep stacks train without heroics. |
| The causal mask | If position i sees position j > i, the model is reading the answer. Training loss looks better; generation is garbage. Nothing in the curve tells you. |
Prove it works before paying for it
A CPU correctness gate, run before any GPU time
Here is the trap this kind of project dies in: you train for six hours, get nonsense, and have no way to tell whether the code is broken or the model just needs more training. That single question can eat days.
The failure mode to design against is an unfalsifiable result: gibberish output with no way to separate an implementation bug from undertraining. Everything below runs on CPU in under a minute.
| Check | What it catches |
|---|---|
Loss at init ≈ ln(4096) = 8.32 | Dead head, mis-shaped output projection |
| Perturb token t, logits before t unchanged | Causal mask leakage |
| Fused attention == hand-written attention | A speed switch silently changing results |
| No-positions config is order-blind at position 0 | An ablation that doesn't ablate |
get_batch targets == inputs shifted by one | Off-by-one training a misaligned objective |
| Overfit 4 fixed batches | Everything else |
The gate that actually matters
A working model must be able to memorise four fixed examples perfectly. If it can't, it is broken, and no amount of data will rescue it. Munti drove the error to 0.0000 and then reproduced all four word-for-word from a short prompt. Only then did it touch a GPU.
Deliberate overfit to loss < 0.05 with greedy recall of the memorised sequences. Result: mean loss 0.0000, 100% greedy recall. Memorisation alone is necessary but not sufficient: the recall check is what catches a sampling bug that training loss cannot see.
The gate caught a bug in its own first version: my "loss at the start" check was wrong, not the model. Because the output layer reuses the input vocabulary table, asking the model to predict the word it was just given is far too easy, and the number came out flattering. Better found as a test failure than as a confusing result.
The first version asserted against targets=x. With weight tying, the
residual stream still carries the input embedding, so identity prediction is
trivially easy: init loss read 4.14 against an expected 8.32. The test was wrong;
independent targets fixed it.
Why the pieces aren't letters
Byte-level BPE at 4,096, trained on the corpus
The simplest choice is one box per letter. It works, and it wastes the model's
attention: the model can only look at 256 boxes at a time, so letters would let it
see a quarter as much story. Learning common chunks ( the,
once, Lily) buys four times the context for free. That is
what the switch at the top of this page shows.
Measured compression is 3.96 chars/token corpus-wide. At a fixed 256-token window, char-level would quarter the effective context. The base alphabet is the 256 byte values, so any UTF-8 string is representable and there is no unknown token, which keeps the pipeline language-agnostic for a later Tagalog corpus with no code changes.
Vocabulary: learned merges
Characters per token
Tokens, from 2.1M stories
44 minutes, and the unglamorous parts that made it survive
20k steps, and the schedule details that matter
Train and validation scores finished within 0.02 of each other, which means the model never started memorising instead of learning. That is expected, since it saw only about 60% of the stories once.
Final: train 1.486 / val 1.505 (perplexity 4.50), no overfitting gap at ~0.6 epoch. The curve is spike-free.
The boring parts are the ones that decide if a run finishes
| Mechanism | Why it's there |
|---|---|
| 200-step warmup | Adam's variance estimates are garbage at step 0; a full-size step then can wreck the initialisation permanently. |
| Cosine decay to 10% | Lets the model settle into the minimum instead of bouncing around it. |
| Gradient clipping at 1.0 | One bad batch is the usual cause of "loss went to NaN at hour three". |
| Exact checkpoint + resume | Free GPU sessions are killed at 9 hours. Optimizer state and step are saved, so a resume continues rather than restarts. |
Grammar arrives long before sense
The same prompt at three checkpoints
Identical prompt, identical sampling settings, three points in training. The order in which abilities appear is the most interesting thing the run produced.
STEP 500 · val 2.64 · sentence shapes correct, meaning absent
Once upon a time, there was a little girl called Lucy. Lucy was very kind and liked to dance. One day, while she was walking in the forest, she saw a peel on the ground and asked it it might be mad.
STEP 5,000 · val 1.77 · coherent scenes, trailing off
One day, Lily's mommy asked her to clean her up before dinner. Lily didn't want to clean her room because she wanted to play with her dolls, but she didn't want to stop playing with them.
STEP 20,000 · val 1.51 · characters persist, dialogue works
Once upon a time, there was a little girl named Lily. She loved playing with her toys, especially her dolls. One day, Lily's friend Timmy came over to play.
"Wow, your toys look so cool!" said Timmy.
"Thank you!" said Lily.
Syntax first, meaning later. At step 500 it had already learned what a sentence looks like. It just had nothing to say.
Syntactic structure is acquired well before semantic coherence; dialogue attribution and cross-paragraph entity persistence appear last.
It writes children's stories. That is the entire list
Can-do / can't-do, grounded in real generations
Everything below is a real answer from the finished model, not a strawman.
All outputs at temperature 0.8, top-k 200, from the final checkpoint.
Can
Write grammatical multi-paragraph stories with a beginning, middle and end; keep a character's name and pronouns consistent across paragraphs; produce dialogue with correct attribution; continue an unseen prompt in style.
Cannot
PROMPT · What is the capital of France?
Her Mom laughed and said "that's the container of Santa. He's coming to visit us!"
PROMPT · The mitochondria is
a friend for you. Come on, let's go and find the star in the dark room.
PROMPT · def fibonacci(n):
"Thank you for marrying me." Snow smiled and replied, "I am happy that you two are very tough."
Entity tracking also fails inside its successful samples:
He opened the box and found a box!
The cat and the cat became friends and played together every day.
None of this is a defect. It read simple children's stories and learned exactly that. A model this small, trained this briefly, on one narrow subject, has no machinery for facts or arithmetic or code, and a write-up that quoted only the good samples would be lying by omission.
Not defects: an accurate reflection of the training distribution. There is no mechanism for factual recall at this scale, corpus and compute budget. Quoting failures verbatim is the point of the section.
Four things that reported success while handing me the wrong thing
Infrastructure failures, and where each is now asserted
Every failure in this project was infrastructure, not modelling, because the gate meant no GPU time was ever spent debugging the model. All four are now checked at startup, so none can cost more than about twelve seconds again.
The free GPU was a P100
sm_60 is unsupported by the installed PyTorch. Everything before training is CPU work, so the run looked healthy for ten minutes before it would have died. Fix: assert compute capability ≥ 7.0 in cell one, and pin the T4.
is_bf16_supported() said yes on hardware with no bf16
It returns True on hardware with no bf16 at all, because it counts
emulation. Fix: gate on compute capability ≥ 8.0 instead.
A dataset upload that dropped every folder
The upload silently dropped every subdirectory and reported success. The package simply wasn't there. Fix: the archive flag on every upload, then verify the file list.
Finding the repo found the wrong copy
Each mounted run output contains a full copy of the repo, so "first match wins" picked a stale snapshot. Fix: search the dataset first, fall back to anywhere.
Three things worth carrying forward
Transferable conclusions
Buy certainty before you buy compute. A one-minute CPU test that must pass before any paid hardware runs turned every later failure into a twelve-second one.
Predict first, then measure, then publish both. The ablation is the most interesting part of this project precisely because I was wrong about it. Reporting the number without chasing the explanation would have shipped a true figure with a false story attached.
Report ranges, not decimals you didn't earn. A single evaluation is a reading, not a measurement. A sister project of mine exists largely because random variation once faked a result.
Run it yourself
Download the model
No login needed. Weights only, small enough to run on a laptop CPU.
- munti-12m.pt: the baseline model (49 MB)
- munti-12m-nopos.pt: the no-positions ablation (49 MB)
- tokenizer.json: needed by both (0.3 MB)
Train it from zero
The gate runs on a CPU in under a minute and must pass before any GPU time is spent.
git clone https://github.com/Zeref538/munti && cd munti
pip install -e .
python test_munti.py # the gate: CPU, under a minute
python -m munti.data prepare # TinyStories to token streams
python -m munti.train --config configs/munti-12m.yaml
python -m munti.sample --ckpt out/ckpt.pt --prompt "Once upon a time"
Stack: Python, PyTorch, HuggingFace tokenizers, Kaggle free T4.
Read the gate,
model.py, or the
written case study.