Claude Code manages to play Grand Theft Auto IV via MCP server

AI enters the game world

Just before the highly anticipated release of Grand Theft Auto 6, a developer demonstrated that Claude Code can play Grand Theft Auto IV. The project, shared on Reddit by user Wonderful-Ad-8579, shows the AI interacting with a live game environment in real time. This is not a pre-rendered video but a live demonstration of the model making decisions based on what it sees on the screen.

The AI was shown performing several distinct tasks within the game world. It successfully navigated a vehicle to a specific map waypoint and even operated a helicopter. It also demonstrated the ability to walk toward specific markers placed within the game environment. These actions show a level of spatial awareness that goes beyond simple pattern recognition.

While the AI can handle these tasks, it does so with a very deliberate and slow pace. It navigates busy streets with extreme caution and takes turns very carefully. The developer noted that the footage provided is completely unedited, highlighting the raw capabilities of the model as it processes the visual data from the game.

How the MCP server works

The technical backbone of this achievement is a Model Context Protocol (MCP) server. This setup provides Claude Code with its own set of peripherals, including a keyboard, mouse, gamepad, and a screen. By giving the AI these tools, the developer allows the model to perceive the game state and input commands just as a human player would.

The architecture involves a specific DLL that handles the capture of the screen and the input of commands. On top of this, a TypeScript MCP server runs to bridge the gap between the AI and the game software. This allows Claude to

Testing across multiple titles

The demonstration did not stop with Grand Theft Auto IV. After completing its tasks in that title, the AI spent the rest of the day playing a wide variety of other games. These included Detroit: Become Human, TABS, Halo Wars 2, Phantom Rose, and Cities: Skylines.

This variety shows that the setup is not limited to a single game engine or genre. By successfully interacting with these different titles, the project proves that the Model Context Protocol server can translate AI commands into actions across diverse software environments.

The iterative coding process

The developer used a specific methodology to improve the AI's performance. Each agent was given one game as a test bed and one specific problem to solve, such as a car that keeps stalling at a certain junction. The AI would play the game through the MCP to identify where the tools were failing.

Once a flaw was found, the AI would fix the code and then measure the results before and after on the same route. This iterative loop allows the model to refine its own logic. The developer also noted that Codex assisted with some of the coding required to build this system.

Future possibilities for game AI

While the current tool is an independent project, it suggests a new direction for interactive media. If a similar system were implemented directly into an actual game, it could create a new standard for non-player characters. This would allow for in-game AI that actively reacts to the world in dynamic ways.

The ability for an AI to perceive a screen and execute complex inputs opens up many possibilities for how digital worlds are populated. Instead of following pre-written scripts, entities could potentially navigate and solve problems in real time based on the specific state of the game environment.

Background and accessibility

The timing of this project is notable as it arrives just before the release of Grand Theft Auto 6. This upcoming title is expected to be one of the largest entertainment properties ever released, making the AI's ability to navigate the previous installment particularly timely for the community.

For those interested in exploring this technology, the project is currently open-source and available on GitLab. However, users should note that the tool is currently restricted to those running Windows 11 systems.

Seamless execution and review

The setup allows the game to run in the background, which means the user can continue to use their PC without any interruptions while the AI performs its tasks. This enables a multi-tasking environment where the model operates independently of the user's primary workflow.

Furthermore, the developer implemented a rigorous quality control process for the AI's improvements. Before any changes are made public, Claude Code reviews the modifications and runs new tests to ensure the fixes work as intended. This ensures that the automated updates to the code are reliable and functional.