Predictability-Aware SFT for Diffusion Language Models
Diffusion language models decode in the order the model finds tokens predictable, not strictly left to right. Standard supervised fine-tuning typically masks tokens at random, independently of how the model decodes, which creates a mismatch between how the model is trained and how it generates.
PACS (Predictability-Aware Canvas Selection and Supervision) is designed to reduce that mismatch. It uses the model’s own predictability to choose each example’s visible input length, and supervises reference tokens just before decoding would reveal them. Across reasoning and code-generation benchmarks, PACS has the highest mean accuracy among the compared SFT methods, with gains of up to 12.84 percentage points over vanilla SFT on individual benchmarks.
Joint work with Miao Li, Hanyang Jiang, Hengyu Fu, Baihe Huang, Yating Liu, Yuhang Cai, Zhongzhu Chen, Jiecheng Lu, Michael A. Boateng, and Pascal Van Hentenryck.