SIAgentNews.com
← All news

NVIDIA releases Nemotron-Labs-Sigma continuous diffusion language models

NVIDIA launched a new family of 14 open-weights models that use a smooth embedding space to generate text, offering new ways to steer and speed up AI outputs.

· First reported by MarkTechPost

91.96Sigma-8B-instruct score on GSM8K

The short version

  • NVIDIA released 14 open checkpoints in 3B and 8B scales, including base, instruct, distilled, and self-speculative variants.
  • Unlike models that work on discrete tokens, Sigma uses a 16-D continuous embedding space to denoise text.
  • The 8B-instruct model achieved a score of 91.96 on the GSM8K math benchmark using 256 DDPM steps.
  • Self-speculative decoding on the 8B model can run up to 4.20x faster than standard autoregressive decoding on specific hardware.
  • The models require a CUDA-enabled GPU and do not currently support CPU inference or quantized versions.

Why it matters

This release introduces a different way for AI to 'think' about language by treating words as points in a smooth space rather than a list of separate items. For developers, this means better ability to steer the model's path and faster generation speeds through distillation.

A new way to generate text

NVIDIA has introduced Nemotron-Labs-Sigma, which is described as the first large-scale family of continuous diffusion language models. While most current models generate text by predicting one discrete token at a time, Sigma works by denoising information within a smooth latent space. This means the model treats language as a continuous flow rather than a series of individual jumps.

The collection includes 14 different checkpoints across two primary sizes: 3B and 8B parameters. These variants are categorized into four types: base models, instruct models for following directions, few-step distilled models for speed, and self-speculative models. Each model is designed to run on a single NVIDIA GPU using bfloat16 precision across various architectures like Ampere and Hopper.

According to MarkTechPost, the Sigma-8B base model achieved an average score of 71.16 on NVIDIA's 6-task suite. This performance puts it ahead of other diffusion models like LLaDA and Dream. It also outperformed the Qwen3-8B model when that model was running in standard autoregressive mode.

How continuous diffusion works

The underlying technology maps every token to a specific point in a learned 16-D embedding space. During the training process, the system adds Gaussian noise to these embeddings and then teaches the model how to remove that noise. This training happens alongside the learning of the embedding geometry itself, creating a unified system for understanding and reconstructing text.

Generation in Sigma happens in blocks, with the default size being 64 tokens. Each block is denoised and then committed to a KV cache before the model moves on to the next segment. This architecture uses a decoder-only Transformer backbone that incorporates grouped-query attention and rotary embeddings to manage the data flow efficiently.

The research team notes that continuous diffusion follows smooth paths rather than the jagged jumps seen in discrete models. This smoothness makes it easier to perform trajectory steering, which is a method for guiding the model's output. It also allows the models to degrade gracefully when using fewer steps during the generation process, making it more practical for real-world use.

Performance and speed gains

One of the primary goals of the Sigma family is to make high-quality text generation faster through distillation. The distilled checkpoints are designed to produce results that are very close to the high-step teacher models while using significantly fewer denoiser calls. For example, the distill-3B and distill-8B models can operate at various numbers of forward steps, such as 2, 4, 8, or 16.

The self-speculative checkpoints take this a step further by using a diffusion head to draft a 32-token block. A separate autoregressive head then verifies that draft to ensure the output matches what a standard greedy decoder would produce. On a GB200 GPU, this method allowed the 8B model to run 3.46x faster than standard autoregressive decoding on average.

While the model excels in certain areas, it still faces challenges with complex reasoning. The Sigma-8B-instruct model scored 91.96 on the GSM8K math benchmark but struggled on the AIME25 benchmark, scoring only 17.00. These results suggest that while the model is highly capable for many tasks, it still has limitations when faced with extremely difficult logic problems.

Deployment and accessibility

NVIDIA has made all 14 checkpoints available as open weights on Hugging Face. They are released under the NVIDIA Nemotron Open Model License, which explicitly states that the models are ready for commercial use. Developers can access the weights to build their own applications or integrate them into existing workflows.

To run these models, users will need specific hardware requirements, as CPU inference is not supported. The models require a CUDA GPU capable of running bfloat16, such as those from the Ampere, Lovelace, Hopper, or Blackwell series. Users will also need to ensure they have the latest versions of PyTorch and the Transformers library installed to load the models correctly.

For those looking to serve these models at scale, NVIDIA suggests using popular tools like vLLM, SGLang, or Docker Model Runner. On a single H100 GPU, the base model can generate between 120 and 140 tokens per second. This provides a high-speed option for developers who need to balance the unique benefits of continuous diffusion with production-level performance.

What to watch

  • How the 16-D embedding space scales as models get larger.
  • Whether continuous diffusion becomes the standard for steering AI behavior.
  • Future performance on harder reasoning benchmarks like AIME.

Try it

Check the NVIDIA Nemotron Open Model License on Hugging Face to see the terms for commercial use.

Sources and links

First reported by MarkTechPost. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news