sup computer — a small language model studio
sup computer is a research studio building small language models from scratch — small enough to train end to end on a consumer laptop, and still useful.
Our methods are LLM-assisted. A mixture of models works each step, from dataset creation to training and evaluation, under human direction. All of our research is open source.
Models
- daydream-chess-nanogptPlays chess without ever knowing the rules — learned move by move from games, not a rulebook, across three board sizes.v1 · v1-grand · v1-micro
- gatsby-nanogptBends any story toward the green light — obsession you can dial from 1 to 5.v2 · v1
- glyph-nanogptDraws the lowercase alphabet one letter at a time — a single evolving model, released chasing the twenty-six specialists that out-drew it.v1
- kenosha-kid-nanogptDreams endlessly on just six words.v2 · v1
- pona-nanogptword-level Toki Pona chat model whose whole vocabulary is the keyboard; writes more grammatically than the corpus it learned fromv1
- shakespeare-nanogptWrites Shakespeare from scratch — and gets sharper every research round.v3 · v2 · v1
Research
A month after the first model, I've trained ~60 and released 11, each starting from a question. These models are knowledge-light and shaped like their data, too small to chat with. The long-term bet is inventing harnesses that turn each one into an instrument.
Three 2.7M-param GPTs trained on 6.9M characters of Toki Pona and scored sentence-by-sentence by the community's grammar checker: the word-tokenized arms clear the span-thesis null, and the released chat model writes more grammatically than its own corpus (96.9% vs 96.0%). Mixing 4.6% oracle-filtered synthetic dialogue into the same recipe turns it into something you can talk to — 160/160 grammatical replies at temperature 0.8.
Three thousand fonts generated from Knuth's own Computer Modern source cover no more of real type design than five hundred did — the ceiling is set by the program, not by the sample count. A third to a half of independently-set parameter combinations won't compile at all, and one boolean, the single-storey g, is worth more coverage than every continuous dial combined.
A round-2 training plan built from a type-design chapter makes three predictions a model run would bet on — so all three were measured first, with zero training. One died (the chapter's craft axes explain 2.1% of what the generalist finds hard; outline complexity explains 17.9%), one passed (quantization binarized overshoot instead of erasing it), and one survived after the corpus corrected the plan itself (u is a round letter, whatever the shape grouping says).
Twenty-six 1.8M-param GPTs, one per lowercase letter, against one letter-conditioned generalist at two sizes — all trained on 82k glyph outlines from 759 open-licensed sans-serifs. The 47.8M generalist wins mean bits-per-char by 2% but fails to draw a well-formed glyph 29% of the time where the specialists fail 8% — and it ships anyway, on purpose: the studio releases one evolving instrument, with the case's numbers frozen as the yardstick every future version has to overtake.
The studio's small models are instruments — single-purpose, played rather than prompted — and sup, the studio CLI, is the accessibility argument: one greeting downloads a release and streams its voice to stdout. A handle that simple works in a shell pipe, which means it works for another model.
The shakespeare model's own likelihood is register-blind: fluent Gutenberg editorial prose scores inside any NLL band that admits verse, and the model's most inevitable text is the junk — footnotes, [Illustration] tags, speaker lists at 1.2–1.8 NLL — so the band's raised floor, not its ceiling, is the load-bearing edge. An LLM judge riding the same steer layer held verse register where the band drifted into publication history.
The studio writes to a rulebook — twelve editing operations encoded as a skill, distilled from Anthropic's research posts, Thoughtful Lab, and Ramp Labs, piloted on two files and then swept across the studio: nine model cards, ten docs, and the website's visitor copy, 130-odd edits in all. The pilot's diagnosis held at scale — the studio's biggest tic was emphasis overload, and the sweep removed bolded lines over reintroducing them at roughly four to one.
Five rounds of Token Chess — a benchmark where LLMs may not author chess moves, only steer Daydream's sampler under a token budget — asked what a token actually buys. Round one: plies, at a worsening rate (every game forfeits; legality collapses 49% → 14% out of the opening book). Round two: remove death and nothing discriminates. Round three: price batches and candidate-picking with engine adjudication, and the benchmark finally separates players — then ministral, the worst sampler on the board, beats olmo 3–1 on pure tempo. Rounds four and five close on memory: assigned a free notepad, olmo writes zero notes in 24 games; offered the choice, it picks the notepad 24 seats out of 24, explains why every time, and still writes nothing.
A single afternoon spent improving all four sup computer models at once — a larger model planned a per-model optimization, small runs executed it. Two new releases (shakespeare-nanogpt-3, kenosha-kid-nanogpt-2), one migration, one eval-only characterization, and a handful of findings that only show up when you look across projects side by side.
A three-tier chess-move GPT family (5x5, 8x8, and a custom 12x10 board) built around a single inversion: illegal moves are rendered as dim near-misses instead of being masked away by the sampler. All three tiers land in a tight band of legal-move rate (35-39% on a raw, unresampled first try) despite very different board sizes, vocabularies, and corpus sources -- and two separate facts in the original design plan turned out to be wrong when checked against the live engine instead of trusted from web research.
A repo-wide audit by a larger model found the small-model studio's engine had two advertised code paths that crashed on use, a metric that quietly flattered char models, and a resume that restarted. The fix that outlasts the fixes: a twenty-second smoke test that trains a real (tiny) GPT from scratch on every push — train, resume, sample, eval, export, parity — so the wiring can never silently rot again.
gatsby's first corpus cost ~$6 of Claude API to write. This round throws that out and has a mixture of four local open models — Olmo, Ministral, Gemma, Granite — write the corpus instead: free, unlimited, and in four different voices. The model that results matches the paid baseline's behaviour at $0. The catch, and the finding: the blend is a designed object. A granite-heavy first round broke the green-light dial; rebalancing off it and doubling the data brought the dial back.
The smallest obsession in the studio: a char-level model whose entire corpus is punctuated permutations of six words. A bot enumerates that space exactly; a learned model can't — and the blur it produces instead is the artifact. The finding: dreaminess is governed by two knobs, training progress and sampling temperature.
Don't serve a model — export only its forward pass as a static ONNX graph (tokens in, last-position logits out) and keep the autoregressive loop, sampling, and tokenization in JS, so a small model becomes a static asset that runs client-side with no server.
A char-level model built to compulsively reach for Gatsby's green light — and the $0, fully-controlled ablation that found the dial's real bottleneck: signal loudness, not corpus shape.
An LLM-assisted experiment: four rounds took held-out BPC from 2.395 to 1.919. More data was the win; regularization was the dead end.