Forecast Hero BPB versus open-weight base models

The preregistered final Hero forecast of 2.0388 dropless Paloma macro-loss converts to 0.7406 BPB. Nemotron 3 Super measures 0.7918 BPB, Qwen3.5-35B-A3B-Base measures 0.8397 BPB, and Nemotron 3 Nano measures 0.8846 BPB. They are 6.9%, 13.4%, and 19.4% higher than the Hero forecast.

Paloma subset forecasts

Each panel composes the preregistered macro-loss scaling law with that subset's loss-to-BPB calibration. The calibrations use the same 77 ladder evaluations; the lowest R² is 0.971. Open-weight models appear at 6ND training-compute estimates. The Super model card reports 12B active parameters and over 25T training tokens, giving a lower-bound coordinate of 1.8e24 FLOPs. The Qwen model card reports 3B active parameters; the 36T-token assumption gives 6.48e23 FLOPs. The Nano model card reports 3.5B active parameters and 25T training tokens, giving 5.25e23 FLOPs. Exact FLOP totals for these hybrid MoE models are unknown.

Paloma subset	Hero forecast	Super measured	Qwen measured	Nano measured
4chan	0.7788	0.7935	0.8828	0.8529
C4 100 domains	0.5923	0.6331	0.7045	0.7004
C4 English	0.6461	0.6752	0.7549	0.7472
Dolma v1.5	0.5758	0.5937	0.6582	0.6621
100 programming languages	0.2661	0.3235	0.3394	0.3926
100 subreddits	0.8176	0.8415	0.9137	0.9192
Falcon RefinedWeb	0.6842	0.6982	0.7788	0.7716
Gab	1.2833	1.3595	1.4485	1.5994
M2D2 S2ORC	0.5403	0.5734	0.6037	0.6146
M2D2 Wikipedia	0.5360	0.5100	0.6250	0.5856
Manosphere	0.8471	0.8556	0.9310	0.9208
mC4	0.5982	0.6272	0.7142	0.7094
Penn Treebank	0.5962	0.9985	0.7140	0.9696
RedPajama	0.4879	0.5029	0.5495	0.5667
Twitter AAE	2.1215	2.2443	2.2703	2.5942
WikiText-103	0.4783	0.4380	0.5465	0.5466

Loss-to-BPB calibration

The forecast in issue #8435 is expressed as macro-loss. We fit a linear conversion to 77 paired eval_dropless/paloma/macro_loss and macro_bpb observations from the four scaling-ladder rungs. The fit has R²=0.99995; it predicts the measured current Hero BPB within 0.0010. Cluster bootstrapping the four rungs gives 0.7404–0.7432 BPB. This interval covers conversion uncertainty only; issue #8435 does not assign uncertainty to the final loss forecast.

Measured checkpoint results

UncheatableEval has no preregistered final-loss trajectory, so its comparison remains a measurement at Hero step 45,837. The final forecast shown above applies only to Paloma.

Model / mode	Paloma macro BPB	UncheatableEval macro BPB
Hero final forecast	0.740620	—
Hero dropless · step 45,837	0.818798	0.536178
Nemotron 3 Super-120B-A12B-Base	0.791756	0.496361
Qwen3.5-35B-A3B-Base	0.839683	0.524489
Nemotron 3 Nano-30B-A3B-Base	0.884570	0.570292

Reproduction metadata

Input	Revision	Documents
Qwen/Qwen3.5-35B-A3B-Base	`0f0813072d2358973511097385626f21fcb6d422`
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16	`e0ac9ee3dfd02be21b5479edfc2f671ed269d0a2`	161,940
nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-Base-BF16	`b4cc202094ea3644bc4461c9fe58ec8b6a3695eb`	161,940
allenai/paloma	`65cd6fc59dba021b21db414fa5e8d7765ffbe5e6`	154,440
UncheatableEval	`185c463882a8ae0f203e51ae5852d8cf4fe299cf`	7,052
Scaling-ladder forecast script	`d23e6e9c3673435fb82d83aa6c51a607d0da6009`	77 paired evaluations

The preregistered forecast assumes the 4K context length, original data mix, and token count used by the ladder. The Hero schedule and data mix may change before the end of training, so 0.7406 BPB is a counterfactual endpoint for the preregistered recipe, not a confidence interval for the eventual checkpoint.

Evaluation metrics JSON · Macro forecast JSON · Subset forecast JSON · Qwen Evalchemy result JSON · Nano result JSON · Super result JSON · Macro Plotly JSON · Subset Plotly JSON