DiffuRefill-1B

**Status: pretraining, about half way.** Step ~100k of 200k, ~46B effective training tokens, merge round 290. The weights here are a live pretraining checkpoint: fluent English, facts still unreliable, not instruction-tuned. Use `global_ema.pt`; it is the best checkpoint at every round. A 1.08B masked-diffusion language model decoded with **confidence-targeted re-infilling**: draft the whole sequence in a handful of parallel denoising passes, then spend a few more passes rewriting only the tokens the model is least sure about. An autoregressive model needs one pass per token; this one needs ~20 passes for a whole sequence, whatever its length. Tracked every round on fixed, deterministic batches, so the change between two rounds carries no sampling noise:

译文尚未就绪，显示来源原文