OshonSoft AI Language Model Lab
Finding the Smallest Language Model That Can Learn Three-Digit Addition

1. Try the trained models

The download contains four projects, their initial and trained models, and complete saved replies. The primary MLP and Transformer both reproduced 3,472/3,472 exact Final evaluation replies with the current Lab training and generation code. The second-seed results are included as well.

DOWNLOAD PROJECTS AND MODELS - DOWNLOAD THE LAB

The trained model is supplied as models/selected.osai in its project. Load it through Persistence, using Choose load file and Load checkpoint. In Conversation, choose Greedy, set the reply limit shown below and clear the conversation before each independent question.

The mlp and transformer folders hold the primary recipes. Use a reply limit of 128 and ask, for example, Add 325 and 386.

Click either screenshot to open it at full size.

WATCH THE TRANSFORMER GENERATE THE REPLY (MP4)

Add 325 and 386.

5 + 6 + ↑0 -> 1 ↑1
2 + 8 + ↑1 -> 1 ↑1
3 + 3 + ↑1 -> 7 ↑0
result: 0711
[EOT]

Add 468 and 945.

8 + 5 + ↑0 -> 3 ↑1
6 + 4 + ↑1 -> 1 ↑1
4 + 9 + ↑1 -> 4 ↑1
result: 1413
[EOT]

Back to contents

2. Read the carry trace

Each line adds the two digits for that column and the incoming carry. The digit after the arrow stays in that column. The carry goes into the next column. For 325 + 386, the ones column adds 5 + 6 + 0 to give digit 1 and carry 1. The tens column uses that carry. The four-digit result is 0711.

The numeric tokenizer keeps digits, carry markers and line breaks. An exact reply has all three correct lines, the four-digit sum and the end-of-turn token. Gold answers are the reference for scoring. Each saved model reply was generated from the prompt with its own preceding tokens.

The training examples and the questions were split by operand pair, including the reversed order. TRAIN teaches the task. Training checks show progress during training. The shared random Final evaluation is scored at the planned endpoint.

Add 266 and 066.

6 + 6 + ↑0 -> 2 ↑1
6 + 6 + ↑1 -> 3 6
2 + 0 + ↑1 -> 3 ↑0
result: 0332
[EOT]

Back to contents

3. Compare saved replies during training

Choose a run, a Training checks question and a saved epoch. The two panels show the generated reply and the reference. Use the token view to see <lf>, digit joins and <eot>.

Model Question
Epoch Show tokens

Back to contents

4. See what changed with more examples and smaller models

How much practice? The three random training sets below use the same 105,584-parameter MLP and the same Training checks and Final evaluation questions. Seven epochs on more examples also mean more SGD updates. All three training sets contain the 500 local digit and incoming carry facts. The rarest fact occurs 1, 8 and 18 times in the three sets, respectively.

Chart horizontal axis EpochsSGD updates

How small? These runs used the same 7,000 TRAIN examples, numeric vocabulary and 80-token context. A successful endpoint had exact replies on all TRAIN examples and on both of its last two Training checks and known-question checkpoints. The shared Final evaluation is reported separately.

At roughly 25,000 parameters, the D4/H70 MLP reached 3,472/3,472 on the shared Final evaluation. D16/H19 reached 650/3,472 with the same seven-epoch budget. The best of the five approximately 10,000-parameter MLPs within fourteen epochs reached 750/3,472. The 13,072-parameter Transformer had 3/3,472 after seven epochs and 3,472/3,472 after fourteen.

How well did it repeat? The second seed used the same dimensions and learning rate, with its own initial weights and shuffled training order. The small Transformer gave very different outcomes in the two runs.

The separate MLP final set, supplied as test_capacity.txt, gave 3,472/3,472 for the primary model and 3,465/3,472 for its confirmation. The table above uses test_random.txt; the two final sets are scored separately.

Back to contents

5. Repeat the training

On an AMD Ryzen 7 7735HS laptop, the repeated primary MLP run took about 1 h 10 min and the primary Transformer run 2 h 9 min. These times include training and all configured checks while several projects shared the CPU. Time spent paused is excluded. Other hardware and workloads will give different timings.

Extract the projects into Documents/OshonSoft AI Language Model Lab. Choose three_digit_addition/mlp or three_digit_addition/transformer. Each already contains its training file and questions. The mlp_confirmation and transformer_confirmation folders use seed 23456.

Use the numeric tokenizer with Preserve line breaks, Global epochs and MODEL body/EOT targets. Standalone carry signs ↑0 and ↑1 each form one token. Each complete reply has 40 targets including EOT. With 7,000 training conversations, one epoch contains 280,000 updates: 1,960,000 for seven epochs or 3,920,000 for fourteen. Model dimensions and parameter counts stay the same. Train fresh models with this token sequence; checkpoints trained with split carry signs cannot be reused.

1. Open the installed OshonSoft AI Language Model Lab. On Project, click Choose project... and select the folder containing experiment.json. The saved settings fill the controls; this does not start training.
2. Review the dimensions, tokenizer, initialization, learning rate and target. Click Prepare and initialize model. Use the prepared target count to read the update budget.
3. Start a fresh run. The Lab saves checkpoints and full Prompt/Gold/Raw replies under runs/run-000001/. Use Open run folder or Read replies... to inspect them.
4. Use Pause + save before closing. Later choose the saved run.json, click Open / recover run, then Resume. Resume keeps the original target.

Use questions/ for Training checks. Select the checkpoint under the study's stated rule, then use Use checkpoint... and Final evaluation for questions/final/. Reusing these already evaluated question sets is a reproduction check; it is not a new unseen test.

For the original capacity comparison, a qualifying endpoint had exact replies on every training example and on both of its last two Training checks and known-question checkpoints. The shared benchmark and fresh MLP evaluation remained separate. Retain the second-seed failures when interpreting the smallest successful run.

Each project supplies models/initial.osai and models/selected.osai. Complete replies are grouped by update in answers/, including the full training check at the endpoint; answers/final/ keeps each final set in its own file. A fresh run creates its own runs/ folder.

Back to contents