Nemotron-Labs-Diffusion-Uni: Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy Distillation

Shufan Li*, Hanrong Ye*, Greg Heinrich*, Yonggan Fu, Haotian Zhang, Sifei Liu, Aditya Grover, Pavlo Molchanov
*Core Contribution
GCE objective illustration

Nemotron-Labs-Diffusion-Uni is an unified diffusion LLM that achieves state-of-the-art performance on diverse visual understanding and generation tasks. It adopts an Elastic-MoT architecture with three elastic views.

Coastal textures

A small seashell on damp sand, delicate ridges and scattered mineral grains, soft morning coastal light. Photorealistic photograph, natural color and exposure, authentic material texture.

Alpine wildflowers

A meadow of pale wildflowers beneath distant alpine peaks, an uneven footpath, late-spring daylight and natural depth. Photorealistic photograph, natural color and exposure, authentic material texture.

Material and texture

A vintage film camera with a black leather strap on a wooden desk, realistic worn metal and textured leather, subtle window light. Photorealistic photograph, natural color and exposure, authentic material texture.

Wildlife in natural light

A kingfisher perched on a mossy branch above a calm stream, blue and orange plumage, soft diffused daylight. Photorealistic photograph, natural color and exposure, authentic material texture.

A quiet interior

A modest living room with a cream linen sofa, oak side table, paperback, and leafy plant, garden-facing window, straight architectural lines. Photorealistic photograph, natural color and exposure, authentic material texture.

A closer look

A small snail moving across a wet green leaf, fine shell texture and tiny water droplets, shallow natural depth of field. Photorealistic photograph, natural color and exposure, authentic material texture.

Output: Change the bear’s material to glass. Input image before editing InputOutput

“Change the bear’s material to glass.”

Output: Transfer the image into a dramatic charcoal-drawing style. Input image before editing InputOutput

“Transfer the image into a dramatic charcoal-drawing style.”

Reading text in a photograph

What is the brand of this camera?

Model answer

Dakota.

Understanding spatial direction

What direction is the dog facing?

A. Forward   B. Right   C. Left

Model answer

C — The dog is facing the left.

Compare values and compute an average

What is the average of longest light blue bar and shortest gray bar?

Model reasoning · excerpt

Comparing 415 and 128, the longest light blue bar is the one with the value 415.
Comparing 385 and 131, the shortest gray bar is the one with the value 131.
Average = (415 + 131) / 2
Average = 546 / 2 = 273.

Answer

273

Full saved model response
<think>The user wants me to find the average of the longest light blue bar and the shortest gray bar.

1.  **Identify the longest light blue bar:**
    *   Looking at the chart, there are two groups of "Solved homicides" and "Unsolved homicides".
    *   The legend indicates that the light blue color corresponds to the year 2012.
    *   In the "Solved homicides" group, the light blue bar has a value of 415.
    *   In the "Unsolved homicides" group, the light blue bar has a value of 128.
    *   Comparing 415 and 128, the longest light blue bar is the one with the value 415.

2.  **Identify the shortest gray bar:**
    *   The legend indicates that the gray color corresponds to the year 2014.
    *   In the "Solved homicides" group, the gray bar has a value of 385.
    *   In the "Unsolved homicides" group, the gray bar has a value of 131.
    *   Comparing 385 and 131, the shortest gray bar is the one with the value 131.

3.  **Calculate the average:**
    *   The values are 415 and 131.
    *   Average = (415 + 131) / 2
    *   Sum = 546
    *   Average = 546 / 2 = 273.

4.  **Final formatting:**
    *   The question asks for an integer answer.
    *   Format: Answer: 273.
</think>
Answer: 273
Compute a ratio from a pie chart

What's the ratio of smallest segment and second largest segment?

Model reasoning · excerpt

Ratio = Smallest Segment / Second Largest Segment
Ratio = 7 / 21
7 / 21 = 1/3
0.3333... rounded to two decimal places is 0.33.

Answer

0.33

Full saved model response
<think>The user wants me to find the ratio of the smallest segment to the second largest segment based the provided pie chart.

1.  **Identify the segments and their values:**
    *   "Strong ties with U.S.": 57%
 (This is the largest segment)
    *   "Equally close ties (VOL)": 21%
 (This is the second largest segment)
    *   "Strong ties with Russia": 15%
    *   "Don't know": 7%

 (This is the smallest segment)

2.  **Determine the smallest segment:**
    *   Looking at the percentages: 57, 21, 15, 7.
    *   The smallest value is 7% ("Don't know").

3.  **Determine the second largest segment:**
    *   Looking at the percentages: 57, 21, 15, 7.
    *   The largest is 57.
    *   The second largest is 21 (" ("Equally close ties (VOL)").

4.  **Calculate the ratio:**
    *   Ratio = Smallest Segment / Second Largest Segment
    *   Ratio = 7 / 21

5.  **Perform the division:**
    *   7 / 21 = 1/3
    *   1 / 3 ≈ 0.33333...

6.  **Format the answer:**
    *   The question asks for a floating-point number with two decimal places.
    *   0.3333... rounded to two decimal places is 0.33.

7.  **Final check:**
    *   Smallest segment: 7% ("Don't know")
    *   Second largest segment: 21% ("Equally close ties (VOL)")
    *   Ratio: 7/21 = 0.3333...
    *   Rounding to two decimal places: 0.33.
</think>
Answer: 0.33
A referring expression
a man in blue and white playing tennis
Multiple tags in one pass
dog red collar

Abstract

We propose Nemotron-Labs-Diffusion-Uni (NLD-Uni), a unified discrete diffusion large language model (DLLM) for visual understanding and generation tasks. Compared with prior unified diffusion models, NLD-Uni in- troduces two major innovations. First, we propose a unified cross-paradigm on-policy distillation (OPD) framework that successfully transfers the visual understanding ability of autoregressive VLMs and visual generation ability of continuous flow models, to a unified DLLM. Second, NLD-Uni incorporates a novel progressive self-refining diffusion process that gradually corrupts each token across multiple steps during training, as opposed to adopting the conventional masked diffusion formulation that directly corrupts a clean image token to an uninformative mask token. At inference, this design allows NLD-Uni to make intermediate commit to a token, enabling progressive self-refinement for better visual quality. Through extensive experiments, we show that NLD-Uni achieves state-of-the-art performance among unified DLLMs on diverse tasks including visual question-answering, multi-modal reasoning, object grounding, text-to-image generation, and image editing.

Model Architecture

Model architecture diagram

NLD-Uni is a unified diffusion model whose input and output are both multimodal. Images in the input stream are represented by both continuous embeddings of a semantic encoder and discrete tokens of a VQ encoder. Images in the output stream are represented solely using VQ codes. NLD-Uni is trained to predict the clean token distribution given partially masked inputs. For image generation, we incorporate an intermediate masking state.

Self-Refining Diffusion Process

Self-Refining Diffusion Process illustration

A key contribution of this work is the self-refining diffusion process, which gradually corrupts each token across multiple steps during training. This is in contrast to the conventional masked diffusion formulation that directly corrupts a clean image token to an uninformative mask token. At inference, this design allows NLD-Uni to make intermediate commit to a token, enabling progressive self-refinement for better visual quality.

Few-step generation examples

Practically, the partial corrpution is implemented by a ball-tree based noise schedule, which progressively replaces the clean token with a corrupted token sampled from its neighbors in the embedding space. Visually, this results in a gradual corruption of the image, where the corrupted tokens are semantically similar to the original tokens, allowing the model to learn a more robust denoising process. By contrast, vanilla masked diffusion replaces the clean token with a mask token, which is uninformative and can lead to a less effective denoising process.

Cross Paradigm OPD from AR teacher

Few-step generation examples

NL-Diffusion-Uni learns from an autoregressive teacher model through a cross-paradigm on-policy distillation (OPD) framework. Specifically, NLD-Uni is variable block-size diffusion model, whose block-size-one view is effectively an autoregressive model. During training, the AR view of NLD-Uni is distilled from the AR teacher model, while we indirectly train the diffusion view with a self-consistency loss that encourages the diffusion view to produce similar outputs as the AR view. This allows NLD-Uni to inherit the strong visual understanding capabilities of the AR teacher model, while also benefiting from the parallel generation capabilities of the diffusion model.

Cross Paradigm OPD from Continuous Flow teacher

Token editing examples

NL-Diffusion-Uni learns from an state-of-the-art continuous flow teacher model. While the teacher and student adopts different latent spaces and incompatible generation paradigms, we still enable on-policy learning by using the teacher to refine student's outputs. Specifically, we first generate an image from the student model, then we use the teacher model to refine the generated image. The refined image is then used as the learning target for the student model. We observe that the teacher's refinements are closely aligned to the student's output while having better visual fidelity and source image preservation. This makes the refined image a better learning target than pure teacher outputs, which are often misaligned with the student's predictions.

Performance and Speed Comparison

Speed comparison animation Speed comparison animation

NLD-Uni achieves highly competitive visual understanding and generation performance. Visual understanding performance is averaged across 11 tasks (refer to paper for details). Text-to-image and image editing reports GenEval and ImgEdit Score. Multimodal reasoning is measured on MathVista. Axes are normalized by the best model's performance on each task.

BibTeX

@misc{li2026nemotronlabsdiffusionunielasticselfrefiningunified,
      title={Nemotron-Labs-Diffusion-Uni: Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy Distillation}, 
      author={Shufan Li and Hanrong Ye and Greg Heinrich and Yonggan Fu and Haotian Zhang and Sifei Liu and Aditya Grover and Pavlo Molchanov},
      year={2026},
      eprint={2610.36466},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.36466}, 
}