DeepMind study finds AI chatbots can't pick truly random numbers on their own

What the researchers found

Ask a chatbot for a random number and it answers instantly. A new paper from Google DeepMind, working with the National University of Singapore, says that answer is not as random as it looks. As BlockWeeks reported, the study is titled 'The Illusion of Stochasticity in LLMs' and was published at the COLM 2026 conference.

The authors argue that reliable random sampling is a basic requirement for AI agents, meaning programs that act on a user's behalf. Today's leading models, they say, don't have it.

Knowing probability isn't doing it

The gap is between knowing and doing. The models estimate probabilities well and handle the underlying maths. But when asked to actually produce numbers that follow a given pattern, such as a uniform or a bell-curve (Gaussian) distribution, they stumble.

The team checked outputs with two standard statistical tests, Chi-Square and Kolmogorov-Smirnov. In many setups the p-values came out close to zero, which means it is very unlikely the numbers truly followed the target distribution.

The bias was easy to see. Models favoured certain familiar numbers, and results changed depending on the order in which information appeared in the prompt.

Variety is not randomness

The authors separate this from the natural variation in how models pick words, which is why the same prompt can give different answers. That variety does not mean a model can deliberately follow a specified distribution.

The fix: borrow randomness

There is good news. When a model gets a random seed from outside, for example a number generated by code, it does much better. Advanced models including Gemini-3.0-Pro and a Qwen3 model could turn that input into the right distribution.

The paper's insight is that models have the exact maths needed to reshape randomness. What they lack is a source of their own.

When bias becomes a security problem

The researchers tested game-like scenarios in which an ideal random strategy keeps an attacker's success rate down to a fixed ceiling. When agents relied on their own sampling, attackers did far better in some tests, approaching near-certain success.

For developers, the advice is direct: generate randomness in code and pass it to the model. For companies testing agents, check statistical fairness directly instead of assuming that varied output means unbiased output.