Salesforce releases a test for AI forecasts that use future context

Forecasts that only look backward

Traditional forecasting rests on one idea: the past predicts the future. Models trained on large collections of time series, called time series foundation models, study trends, seasons and variance in historical numbers, then project them forward. Moirai-2 is one example, and it can forecast new series without extra training.

But people running real operations often know things the history cannot show. An energy operator knows a heatwave is coming. A logistics manager knows a road will close. An IT lead knows a software release will spike server load. As salesforce.com reported, these insights usually sit in documents, plans or expert judgment, and the standard interfaces of these models mostly accept only numbers and predefined variables.

What GIFT-Eval-CTX actually tests

Salesforce AI Research built the benchmark to ask one question. If a system is given reliable information about future changes, can it turn that information into an accurate numerical forecast? It extends the team's earlier GIFT-Eval benchmark and spans 10,000 tasks from 20 real-world datasets in energy, logistics, weather, retail and cloud operations.

Rather than feed models loose free-form text, an automated pipeline converts future context into standardized "effects." There are two layers:

• Macro effects describe the whole forecast window: whether the trend is rising, falling or flat, any seasonal cycles, and limits on the values.
• Micro effects pin down events in specific time windows. These include temporary spikes or drops (excursions, such as Black Friday traffic), permanent level changes (mean shifts, such as opening a new facility) and changes to recurring cycles (such as altered store hours).

The model gets the shape of the future and its key events, and has to fill in the exact numbers.

How the test set was built

The benchmark deliberately picks hard windows, where traditional models that see only past data fail. The final set holds 23,476 validated micro effects, each paired with macro annotations for its forecast window.

AI agents did much of the annotation work, with checks along the way. Discovery agents scan trends and actual outcomes to find where standard models missed local events. Other agents run code to test annotations against the raw data and fix math errors. A separate critic agent then reviews the raw data and accepts, rejects or refines each annotation. Humans also spot-checked the result.

What the results show

Given only history, pure language models trail traditional forecasting models. Give them structured future context and the gap shrinks fast: GPT-5.6 Sol's error fell by 40.7%, and Claude Opus 4.8's by 29.6%.

The best results came from an agentic system called MoiraiAgent, which combines an LLM with a specialized forecasting model and code execution. It led at every forecast length, with an aggregate error score 36.3% lower than the best history-only model. Its edge grew on longer forecasts. The authors also say event context helps agents handle the transitions between several events and keep repeating patterns, such as daily electricity demand, that standard models tend to flatten out.

Why chatbots alone stumble

The tests exposed a basic weakness. Language models struggle with numerical precision and with producing an output of the right length. Without tools, GPT-5.6 Sol returned the correct forecast length in only 38.9%-40.5% of medium-to-long tasks. Claude Opus 4.8 managed it in less than 2.3%.

MoiraiAgent works around this in two steps. First it calls the Moirai-2 engine to produce a complete numerical baseline of exactly the right length. Then it uses code to compute precise targets, apply the context adjustments and check that the constraints are met before giving a final answer. The language model supplies the judgment about what the context means, and the tools handle the arithmetic.

Reading the claims with care

These results come from the benchmark's own creators, and the announcement describes the findings in strong terms. Independent teams running their own systems on the public leaderboard will show how well the numbers hold up.

The test also assumes the context is reliable and well structured. Real planners often have messier, uncertain information, and the benchmark does not tell us how forecasters cope with that. It measures a specific skill, turning known future events into numbers, and does that well.