Match the Canvas, Not Just the Answer: Predictability-Aware SFT for Diffusion Language Models
Published in arXiv preprint arXiv:2610.18001, 2026
Miao Li, Hanyang Jiang, Hengyu Fu, Baihe Huang, Yating Liu, Yuhang Cai, Changkun Guan, Zhongzhu Chen, Jiecheng Lu, Michael A. Boateng, Pascal Van Hentenryck. "Match the Canvas, Not Just the Answer: Predictability-Aware SFT for Diffusion Language Models." arXiv preprint arXiv:2610.18001, 2026.
Diffusion language models generate text in parallel, revealing tokens in the order the model finds them predictable rather than strictly left to right. That flexibility creates training choices that autoregressive teacher forcing never faces: which reference tokens are visible when a target is supervised, and how much generation space is allocated. Standard supervised fine-tuning (SFT) typically makes these choices by random masking, independently of how the model decodes at inference.
The paper shows how this training-inference mismatch can misdirect supervision. Random masking can hide prediction errors or penalize uncertainty that decoding would have resolved, and the visible input length can change which formulation of a response the model prefers. PACS (Predictability-Aware Canvas Selection and Supervision) uses the model’s own predictability to choose each example’s visible input length, and supervises reference tokens just before decoding would reveal them. Across reasoning and code-generation benchmarks, PACS has the highest mean accuracy among the compared SFT methods, with gains of up to 12.84 percentage points over vanilla SFT on individual benchmarks.