SIAgentNews.com
← All news

Open models may fail at long outputs because of how words are picked, not training

A preprint tested 29 open models and found that at high randomness the word-picking rule, not the model, often decides whether long outputs hold together.

· First reported by arxiv.org

51,845Top-p candidate pool at temp 2 in one Llama-3.1-8B story

The short version

  • The study tested 29 open-weight models, from 1B to 122B parameters, on ten benchmarks plus needle retrieval to one million tokens.
  • At temperature 2, coding agents using top-p, top-k or Top-H resolved 0 of 100 GitHub issues.
  • Agents using top-nsigma resolved 46 to 62 on three Qwen3.5 models at the same temperature.
  • Across all benchmarks, top-nsigma kept a median 100% of its temperature-1 score; top-p kept 26%.
  • At the default temperature of 1, most samplers performed about the same.

Why it matters

If you run open models yourself, the sampler setting may matter as much as the model for long tasks like coding agents or long reports. Before blaming the model or waiting for a retrained version, check how it picks its words. The result is a preprint and has not been independently checked.

A familiar problem, a different suspect

Anyone who has asked an open-weight model (one whose trained settings are published for anyone to download) for a long piece of work has probably seen it drift. The text gets repetitive, strange or incoherent the longer it goes. The usual response has been to train the model more on long material.

A preprint posted on arxiv.org argues that this may be aiming at the wrong target. The authors say that, at high temperature, the part that decides whether long and agent-style generation survives is the sampler. That is the rule that picks each next token (a small chunk of text, often a word or part of one) from the model's list of probabilities.

What temperature and samplers do

A language model does not simply output a word. For every step it produces a probability for each possible next token. Temperature is the dial that controls how adventurous the pick is: at higher settings, unlikely tokens get a better chance.

A sampler sits on top of that dial. Names such as top-p, top-k, Top-H, min-p and top-nsigma are different rules for deciding which tokens stay in the pool of candidates. The paper's central point is that some rules keep that pool small and steady as temperature rises, while others let it swell with low-probability tokens.

What the tests found

The team tested 29 open-weight models, ranging from 1B to 122B parameters. The work covered ten established benchmarks plus a needle-retrieval test (finding a small planted fact in a huge text) out to one million tokens. The tasks included coding agents on SWE-bench Verified, long step-by-step reasoning, long-form writing, long structured output and long-context reading.

The pattern was stark. Samplers that kept a small, steady pool did not lose accuracy as temperature rose, while samplers that let in unlikely tokens fell apart. At temperature 2, coding agents using top-p, top-k or Top-H resolved 0 of 100 GitHub issues. Agents using top-nsigma on three Qwen3.5 models resolved 46 to 62, and they won every issue where the two approaches disagreed.

Across all benchmarks, top-nsigma kept a median 100% of its temperature-1 score. Top-p kept 26%.

Counting the candidates

To explain the gap, the authors counted how many tokens each sampler let through. In one story written by Llama-3.1-8B, moving from temperature 1 to 2 grew top-p's pool from 4 tokens to 51,845. Top-nsigma's pool stayed at 3.

Over 133 combinations of sampler setting and temperature, typical pool size tracked the quality scores from a judge, with a Spearman correlation of -0.77. That is a measure of how consistently two things move in step, and the minus sign means bigger pools went with worse quality.

The authors also tried a fix. When they chose the pool at temperature 1 and applied temperature only afterwards, Top-H, min-p and p-less stopped collapsing.

What to take from it, and what not to

The findings come with a clear limit: at the default temperature of 1, most samplers performed about the same. So this is not a claim that your everyday setup is broken. It concerns what happens when you push randomness up, perhaps to get more varied writing or more creative agent behavior.

It is also a preprint, meaning it has not been through peer review or independent replication. The authors say they release all code and data, which makes checking possible for anyone with the hardware.

For people running open models, the practical reading is modest but useful. If a long output falls apart at a high temperature, the cause may be the sampler's pool rather than the model's training. Lowering temperature or switching samplers is a cheaper experiment than retraining or rewriting prompts.

What to watch

  • Whether other researchers reproduce the results using the released code and data.
  • Whether popular open-model tools change their default sampler or temperature advice.
  • Whether the findings hold on tasks and models beyond the 29 tested.

Try it

If a long output drifts, rerun it at temperature 1 with the same prompt, then compare. If your tool offers a sampler choice, test one that keeps its candidate pool small.

Sources and links

First reported by arxiv.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news