Abstract
Standard Next-Token Prediction (NTP) super vises language models solely through discrete labels in the output logit space. We argue that this sparse, one-hot supervision leaves the la tent representation space under-constrained, al lowing hidden states to drift into degenerate and anisotropic configurations that limit generaliza tion. To address this issue, we propose Next Im plicit Token Prediction (NITP), which augments discrete prediction with dense, continuous super vision directly in the representation space. NITP requires the model to predict the implicit seman tic content of the next token, using shallow-layer representations from the same model as stable self-supervised targets. Theoretically, we show that NITP regularizes the optimization landscape by mitigating under-constrained degrees of free dom and enforcing a compact, structured repre sentation geometry. Empirically, across dense and MoE models ranging from 0.5B to 9B pa rameters, NITP consistently improves down stream performance with negligible computa tional overhead. Notably, on the 9B MoE model, NITP achieves a 5.7% absolute improvement on MMLU-Pro, along with gains of 6.4% on C3 and 4.3% on CommonsenseQA, with ∼2% ad ditional training FLOPs and no additional infer ence cost. Our implementation is available at https://github.com/aHapBean/NITP.
Showing the abstract — retrieve the full paper via the Exa API.