Renoise Consistency: Unlocking Efficient Self-Correction for
Diffusion Large Language Models
Motivation
Diffusion Large Language Models (DLMs) can self-correct.
Abstract
Diffusion Large Language Models (DLMs) are being actively explored as promising alternatives to autoregressive models due to their fast and flexible generation capabilities. To optimize inference in DLMs, recent studies have proposed a variety of decoding algorithms. However, the tendency of these methods to mimic autoregressive generation limits the speed and flexibility of DLMs, while also preventing the methods from fully realizing their potential. Moreover, this rigid approach leads to error propagation, while existing sampling-based correction methods incur prohibitive computational costs. To overcome this, we propose Renoise Consistency, a novel post-hoc approach that enables effective self-correction by maximizing train–inference consistency and can be applied to any decoding algorithm with a flexible computational budget. Furthermore, we introduce Adaptive Block Drafting, which leverages consistent internal representations of DLMs to significantly reduce the overall computational cost. When combined, our proposed method achieves up to a 9.00% performance improvement over the semi-autoregressive baseline, along with a 25.85% reduction in forward steps.
Observation
Findings #1: Bidirectional Inertia reveals the optimal output length or block length.
Adaptive Block Drafting (ABD), efficient drafting.
Findings #2: Scattered Anchor Context resolves the train-inference mismatch.
Renoise Consistency (ReCon), efficient self-correction.
Method
2 step train-free inference (ABD + ReCon).
Takeaway
Uniform masking (\(t \sim U[0,1]\)) yields
scattered anchor contexts (train),
while existing decoding methods (inference)
impose a mismatched left-to-right prior.
ReCon resolves this mismatch
during reconstruction.
| Forward steps | Uniform random anchors | Left-sided anchors |
|---|---|---|
| 1 | 64.57% | 26.50% |
| 5 | 86.56% | 31.40% |
| 10 | 89.21% | 48.28% |
Reconstruction success rate of initially correct GSM8K outputs (LLaDA 8B Instruct) after remasking 90% of the tokens.
Results
ReCon improves drafts.
Semi-AR: up to 7.17%p
Conf. fallback: up to 6.83%p
KLASS: up to 4.95%p
ABD: up to 5.69%p
ABD + ReCon outperforms lightweight revocable decodings.
WINO: up to 23.5%p
Saber: up to 18.1%p
BibTeX
Coming soon.