Amazon and NVIDIA test AI to help robots predict retail movements

Predicting the next move

Amazon Web Services (AWS) recently assisted a Tokyo-based company called Telexistence in testing a new way for robots to interact with humans. The goal is to move beyond robots that simply react to what is currently in front of them toward machines that can think one step ahead. By using NVIDIA's DreamZero model, the team wants robots to predict what their cameras will see a moment later and plan their arm movements accordingly.

The DreamZero model is what researchers call a world action model. This means the AI attempts to understand the physics and logic of the environment to anticipate changes. In a retail setting, this could mean a robot preparing to bag an item before a customer even finishes placing it on the counter. Amazon provided the computing power and expertise to fine-tune this model on its servers.

Despite the advanced tech, the initial results from the trial were uneven. In a simulator, the best version of the model completed a block-picking task 45% of the time. However, when the team tested the model using recordings from a physical robot in the same simulator, it succeeded zero times. The engineers attributed this failure to noise in the recordings and a mismatch between the real robot and its digital copy.

The human data bottleneck

While the AI provides the 'brain,' the robots still rely heavily on human behavior to learn how to function. Telexistence uses a method where human operators steer robots remotely through real tasks, and those sessions are turned into training data. The team gathered 183 recorded demonstrations for a specific block-picking task to teach the machine the necessary movements.

The quality of this human-provided data proved to be a significant hurdle for the engineers. Human reviewers threw out 91 of the 183 recordings, meaning almost half of the data was deemed unusable. Some recordings showed the robot sitting idle for too long, while others featured the operator picking up the wrong colored block or having their hand drift into the camera's view.

To manage this volume of data, the team built an AI reviewer to filter the recordings automatically. This system flagged 72 recordings, or 39.3%, as being close enough to work as a first filter. However, the AI still struggled with basic visual distinctions, such as telling the difference between blue blocks and purple ones. This suggests that human oversight remains a critical component of the training pipeline.

Labor gaps and retail shifts

The push for this technology is largely driven by severe labor shortages in Japan. A 2023 projection by the Recruit Works Institute estimated that the country could be short by 3.41 million workers by 2030, with that number jumping to over 11 million by 2040. Companies are looking for ways to keep stores running as the available workforce shrinks.

Retailers like Seven-Eleven Japan have already begun making concrete plans to integrate these machines. They agreed in September 2025 to develop a humanoid robot named Astra with Telexistence, with a target launch date of 2029. The stated goal is to hand off routine tasks to robots so that human employees can focus on providing high-level service that machines cannot replicate.

This shift is already visible in other markets, such as New Zealand, where checkout roles are changing. Some retail job ads now list helping customers with self-service checkouts as a primary responsibility. As robots take over the heavy lifting, the human role in retail is evolving into one of management and customer interaction, though research suggests these new roles will require careful planning to maintain employee autonomy.

The reality of robot speed

While the technology is advancing, there is still a significant gap between laboratory success and daily utility. Telexistence recently shared footage of its autonomous bagging robot, showing it moving at both normal and eight times speed. The company was transparent about the current limitations, noting that the robot is not yet fast enough for real-world use and its movements remain jerky.

Small errors in the environment can still cause significant issues for the current hardware. If a shift in an item is only a few centimeters, the robot's accuracy can drop noticeably. Genki Sano, the co-founder of Telexistence, described the current demonstrations as a starting point rather than a finished product. The team is still working to overcome the 'real-to-sim gap' where digital training doesn't perfectly translate to physical reality.

Industry analysts remain cautious about the pace of widespread adoption. Gartner expects fewer than 20 companies to put humanoid robots into production for supply chains or manufacturing through 2028. For now, the most immediate impact of this technology may not be the robots themselves, but the new jobs created to train, monitor, and manage them.