LiitLLM

Both models were given the same eight Taglish prompts and asked to continue them, 32 continuations each. Then the same scorer from the top of this page measured how much English came out.

Fixed committed prompt set, 32 generations per checkpoint, scored with taglish.score. Reported as a range across two seeds per arm, because a single run per arm cannot separate a real gap from seed noise.

The arms do not overlap. The gap between them is 0.120. The filtered models put roughly three times as much English into their output, and that gap is 2.8× the widest within-arm spread. Filtering for code-switching produced a model that code-switches.

What the difference actually sounds like

Same prompt, both models, at 60,000 steps.

The difference is register, not just vocabulary. The filtered model writes the way someone texts: "Gets ko na lang", "Alam mo naman kasi yung feeling", switching to English for a whole clause and back. The unfiltered model writes clean reported speech in a news register. Both are fluent Filipino. Only one of them is Taglish.

The two lists are not symmetric

The Tagalog list is almost all function words: ang, ng, sa, kasi, naman. Those are a closed class: you cannot write Tagalog syntax without them, and English text will not accidentally produce them.

The English list deliberately includes content words: office, traffic, meeting. That asymmetry is forced by the phenomenon itself: in "nag-commute ako sa office" there is not a single English function word. Score English by function words and the most characteristically Taglish sentences read as pure Tagalog.

Hyphens get split, or the most Taglish word scores nothing

nag-commute matches neither list as a single token, and neither does nagcommute, because the affix stripper only finds a stem that is itself in the Tagalog list, and commute is English. So the single most characteristically Taglish word in the sentence would contribute zero to either side.

Splitting on the hyphen fixes half of that: commute lands in the English list. The nag- half still scores nothing, since bare affixes are not lexicon entries. That is a real limitation, not a design choice, but it only ever under-counts, and it under-counts the Tagalog side of a word already surrounded by Tagalog function words, so the document is not at risk of being misread as English.

My whole pitch was that code-switching gets filtered out of web corpora. So I measured how much survives, expecting very little. Across 50,000 documents from each of two sources, 38% and 39.5% already passed the Taglish test, untouched, straight out of the corpus.

Bake-off across FineWeb-2 fil_Latn and HPLT 2.0 tgl_Latn, 50,000 documents sampled from each: keep-rates of 38.0% and 39.5%. The langid filter thins code-switched text but does not come close to removing it.

An earlier version of this page would have said something false

I first ran that bake-off on 2,000 documents and got 46.8% versus 34.4%, a clear win for FineWeb-2, and I wrote it down as one. At 50,000 documents it is 38.0% versus 39.5%: the two sources are equivalent and the ranking had inverted. Small samples do not give you a quiet, proportional error. They give you a confident answer that is backwards.

No downloaded model, no fine-tuning. The tokenizer, the transformer, the training loop are code in this repository. The only things brought in from outside are the raw text and PyTorch itself.

Architecture is inherited unchanged from Munti, a prior from-scratch model proven on TinyStories, deliberately, so that any difference in output is attributable to data rather than code. What changed is scale and context.

The 512-token context is a deliberate choice for this subject. Code-switching is a discourse-level behaviour: a writer flips register across clauses and sentences, not within a three-word window. At 256 tokens the model would rarely see a full switch cycle.

The free GPU I used stops every session at 12 hours, whether or not you are done. Each run needs about 11. There is no room for a crash, and a crash at hour 10 that loses everything is not a setback, it is the end of the project.

Kaggle: ~30 GPU-h per week per account, hard 12-hour session wall. Four runs of ~10.9h each across two accounts. Every failure mode below actually fired at least once during this project.

What makes an 11-hour run survivable

• Atomic checkpoints. Write to a temp file, then os.replace onto the target, keeping the previous one as a fallback. A crash mid-write otherwise truncates the only copy you have.
• Random state is saved too. Without it, a resumed run replays the same batches it already learned from and quietly trains on a shorter corpus than you think.
• A wall-clock guard. The loop stops itself at 11 hours and checkpoints cleanly, so I choose when the session ends rather than the platform choosing mid-write.
• A runner that assumes it will die. Re-running the entrypoint skips finished work and resumes partial work. It also detects the specific case where a run finished on the server but its output was never downloaded, which is what happens every time the laptop sleeps.
• A lockfile. Two runners once raced and pushed the same job 18 seconds apart, burning double the GPU on identical work.

The last run made the case for all of it: it stopped at step 55,000 instead of 60,000. The runner noticed, pushed a continuation from the last checkpoint, finished the remaining 5,000 steps, and moved on to scoring, with nobody awake.

Filtered (Taglish) Unfiltered (control) two seeds each

The unfiltered models reach a validation loss of 2.635. The filtered models sit at 2.745. Lower is better, so the control arm looks like the better model, and that reading is meaningless.

Each model is evaluated on a held-out slice of its own corpus. They are sitting different exams. Filtered Taglish is intrinsically harder to predict: two languages' worth of vocabulary, and a switch point that can fall anywhere. A lower score on an easier exam says nothing about which model is better at the thing I care about.

The only apples-to-apples measurement is the one in section 01: identical prompts, identical scorer, comparing outputs rather than losses. It would have been very easy to put "2.635 vs 2.745" in a results table and let a reader draw the wrong conclusion.

A 32.8M-parameter model trained on ~2B tokens of web text has grammar and register. It does not have knowledge, arithmetic, or code, and it will not tell you so.

PROMPT · Ang kabuuang populasyon ng Pilipinas noong 2020 ay

…umabot sa 3.5 milyon.

The real figure is about 109 million. The sentence is grammatical, plausibly shaped, and off by a factor of thirty.

PROMPT · 1 + 1 =

0.04 – 5.05 = 0.48 – 0.8 – 0.8 – 0.2 – 0.8 – 0.05 = 0.8 – 0.8 – 0.10 – 0.3…

PROMPT · def fibonacci(n):

Southern California, Mexico, CA 9003, Mexico, Mexico, Mexico, Ltd. (Nobel/Five: 1)…

• It repeats itself. Longer generations fall into loops: the same hashtag or phrase over and over.
• Degenerate repetition at longer horizons; no repetition penalty is applied at sampling time.
• It invents facts fluently. Names, dates and figures are shaped correctly and almost always wrong.
• It has no instruction following. There is no fine-tuning stage. It continues text; it does not answer questions.
• Its idea of Taglish is the internet's. The corpus is web text, so the register skews toward blogs, news comments and entertainment copy.
• It is under-trained, and the curve says so. Validation loss was still descending at step 60,000 in all four runs. Nothing had flattened or turned up. More compute would have produced better models. The runs stopped because the weekly GPU quota did, not because training had converged. Both arms stopped at the same step, so the comparison is unaffected; the absolute quality is not.

None of these crashed. Every one of them ran to completion and gave me something that looked like a result.

A deduplication step that deleted the corpus

Documents were deduplicated by hashing their first 200 characters. Web pages share navigation boilerplate, so thousands of distinct articles hashed identically. A 20,000-token target came out as 109 tokens. Hashing the whole document fixed it. A unit test caught this, not a person.

A config I edited but never uploaded

I changed the training length, then launched two 11-hour runs from the copy that was still sitting on the server. Both runs used the old schedule. They are still valid, because both arms were equally affected, but the config file in the repo no longer described what ran. Editing a file is not deploying it.

The fix was to set the configs to what actually ran rather than what I meant, and write a comment explaining why the obvious-looking value must not be "corrected". Seed 2 then had to reproduce the same schedule exactly, or the seed-to-seed spread would have absorbed a config change and stopped measuring noise.

…and then a second bug caused by fixing the first

That config field was read by two different things: the learning-rate schedule and the runner's "is this run finished?" test. Pinning it for the schedule made a completed 60,000-step run look 12,000 steps short, and the runner launched a pointless continuation. An assert at the top of the training notebook caught it in 55 seconds instead of after hours of compute.

A file pattern that matched two files

Every saved run contains a full copy of the repository, so a search for pyproject.toml found both the repo and the previous run's embedded copy. The fix was to scope each lookup to where the thing legitimately lives, making the ambiguity impossible rather than merely detected.

A wrong diagnosis, stated confidently

When mounted files first went missing I blamed a race condition and said so. It was a wrong path. Running one command that listed the directory would have shown that immediately. A confident wrong diagnosis costs more than saying "checking."

Download the model

No login needed. Both files are weights only, so they load with PyTorch's safe weights_only=True mode and run on a laptop CPU.

• liitllm-taglish.pt: the Taglish model (131 MB)
• liitllm-control-unfiltered.pt: the control it is compared against (131 MB)
• tokenizer.json: needed by both (0.5 MB)

git clone https://github.com/Zeref538/liitllm && cd liitllm
pip install -e .
python -m liitllm.sample --ckpt liitllm-taglish.pt --tokenizer tokenizer.json --prompt "Kahapon nag-"

Checksums and release notes: GitHub release v1.0. All four runs (both seeds per arm) are on Kaggle.

Train it from zero

The correctness gate runs on a CPU in about a minute and must pass before any GPU time is spent. It is the reason a bad generation can be blamed on data rather than on a broken attention mask.

git clone https://github.com/Zeref538/liitllm && cd liitllm
pip install -e .

python test_liitllm.py          # transformer + training loop correctness
python test_data.py             # corpus parity and dedup
python test_resume.py           # a killed session can resume

python -m liitllm.data prepare  # stream, filter, tokenize (CPU, hours)
python -m liitllm.train --config configs/liit-33m.yaml
python -m liitllm.evaluate --baseline out/ckpt.pt --ablation out-unfiltered/ckpt.pt

Stack: Python, PyTorch, HuggingFace tokenizers and datasets (streaming only), Kaggle free T4s. No pretrained weights, no paid services, no API keys.