SI Agent News
DevelopingClaudeOpenAIGeminiAgents

New benchmark shows AI agents fail when the map changes mid-task

LiveGeo tests AI agents on routes and places that shift while they work, and top models' success rates collapsed, which matters for any tool that plans around local conditions.

1.3%GPT-5.6 Sol success under shared disruptions (18.1% static)

The short version

Why it matters

Many AI tools are sold on scores from tests where nothing changes. This work suggests those scores can overstate how an agent copes with closures, delays and detours. If you rely on AI for routing, local listings or logistics, ask how the system handles information that has gone stale.

A test where the world keeps moving

As CASRAI reported, a team spanning universities in Singapore, Britain and China has released two linked tools: GeoWorld, a simulation platform, and LiveGeo, a benchmark built on top of it. A benchmark is a standardized test used to compare AI systems. The paper was submitted to arXiv, a public preprint site, on November 12, 2026 (arXiv:2611.17267). Preprints have not yet been peer reviewed.

The point is a gap in existing tests. Earlier geographic benchmarks, such as MapEval, CityBench and TravelPlanner, give an AI a frozen snapshot of maps, routes and places. Real life does not work that way. Traffic builds up, venues close, buses run late and temporary curfews shut off streets.

What GeoWorld and LiveGeo contain

GeoWorld models six cities: Shanghai, Tokyo, Singapore, London, New York and Cape Town. It covers 88 municipal regions, 1,376 verified points of interest (named places such as museums or stations) and 107,372 cached route trajectories. Conditions in this world shift on their own schedule, and the platform keeps a full history so that other researchers can rerun the same test.

LiveGeo turns this into more than 1,200 tasks with checkable answers, organized in three tiers:

Separating the tiers lets researchers tell whether an agent lacks basic spatial knowledge or simply cannot revise a plan when something breaks.

How nine leading models fared

The team tested nine frontier language models from several major labs. Even without any disruption, scores were low. On the place-pinpointing task, the best result was 37.1%, from Gemini 3.7 Flash. On the destination-finding task, the same model led with 38.0%.

Multi-stop scheduling was much harder. The top models, Gemini 3.7 Flash and Gemini 3.1 Pro Preview, each managed 8.7%. GPT-5.6 Sol reached 2.0%, and Claude Sonnet 5 solved none of its tasks (0.0%).

Disruption made things worse. On route planning without change, the leading models scored in a range from 18.1% to 26.2%. Once conditions shifted, most of them dropped to single digits:

The authors conclude that doing well on static tests does not predict reliable behavior when things change.

Why the agents break

By reading the agents' step-by-step logs, the researchers found three recurring problems.

They treat old facts as permanent. After an agent confirms that a road is open or a museum is receiving visitors, it tends to assume that stays true. It rarely goes back to check, even as time passes.

They notice trouble but do not act on it. Given an explicit alert about a blocked road or closed venue, models often acknowledge it in their written reasoning. Their next tool request, though, frequently repeats the same invalid route or makes a small tweak that still breaks a constraint.

One problem snowballs. In a multi-stop trip, a hiccup at the second stop can ruin everything after it. The models often dropped the remaining destinations rather than rebuilding the schedule.

What it means beyond the lab

The authors draw lessons for universities and other organizations that buy or deploy autonomous systems, such as campus shuttles, delivery robots or field-sampling drones. Vendor-reported scores from static tests are not enough, they argue; buyers should ask for results under changing conditions and record failure rates in their own evaluation documents.

They also point to record-keeping. Any location fact an agent relies on should carry a timestamp, the source that verified it, and a period during which it is valid. The W3C PROV-O standard, a format for recording where data came from, is one way to do that.

The team has made both the platform and the benchmark code public, so others can test their own agents. The caveats are real: this is one preprint, run in a simulation, and the scores come from the authors' own setup rather than independent repeats.

What to watch

Try it

If you use an AI assistant for routes or local hours, pick one answer today and check it against a live map or the business's own page. Note whether the assistant ever offered to re-check.

Sources and links

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

Listen to the AI Wire headlines

Listen to the AI Wire

Listen to our daily AI shows every morning: on your commute, in the shower, wherever you are

Listen to the daily shows

First reported by CASRAI · Read the original. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

More AI news