SIAgentNews.com
← All news
DevelopingGemini

Google adds computer use capabilities to Gemini app for Mac

Gemini can now click, type, and scroll on macOS to automate complex workflows like data entry and photo editing.

· First reported by Appleosophy

8 GBMinimum RAM required

The short version

  • Gemini can now click, type, and scroll within apps on macOS.
  • The feature runs in an isolated background window with a live preview in the chat.
  • Users must have a Google AI Plus, Pro, or Ultra subscription to access the tool.
  • The app requires macOS Sequoia 15 or later and at least 8 GB of RAM.

Why it matters

This shift moves AI from a chat interface to an agent that can perform multi-step actions across different software. It allows for the automation of repetitive manual tasks like data extraction and batch image processing.

A new way to interact

Google has introduced a significant update to its native Gemini app for macOS. While the app previously allowed users to access AI features more efficiently, it can now perform direct actions on the computer. This new functionality is specifically designed for users who want the AI to handle multi-step tasks across different applications.

As reported by Appleosophy, the update allows Gemini to click, type, and scroll on the user's Mac. This represents a shift from simply generating text or analyzing files to actually interacting with the operating system's interface. It enables the AI to navigate software in a way that mimics human interaction with a mouse and keyboard.

This update is currently available for users in the United States who subscribe to specific premium tiers. Specifically, the new computer use capabilities are restricted to those on the Google AI Plus, Pro, and Ultra plans. This means the most advanced models are being paired with the most active automation tools.

How the technology works

The system operates by running Gemini inside an isolated background window. This design choice ensures that the AI's actions do not interfere with the user's primary workspace. A live preview is provided directly within the chat interface so the user can see exactly what the AI is doing in real time.

Because the AI works in a separate space, the user's main screen and cursor remain completely free. This allows a person to continue typing, designing, or taking calls while the AI performs background tasks. It creates a parallel workspace where the AI can execute commands without taking over the entire display.

Privacy and safety measures are integrated into this workflow to give users control over the automation. Users can choose which specific apps Gemini is allowed to access during a session. Additionally, the system allows users to watch every step live and pause the process at any moment before approving key actions.

Practical use cases

Google has highlighted several ways that this computer use capability can streamline creative and professional workflows. For example, designers can use it to batch color-grade photos in their preferred design software. It can also be used to set up complex 3D scenes, which often involve many repetitive clicks and adjustments.

The tool is also aimed at reducing manual data entry for administrative tasks. It can extract specific weights, dimensions, and addresses from local PDF files to automatically fill out multi-box shipping quotes. This moves the AI from a passive reader of documents to an active participant in data management.

Business users can also use the feature to turn live data into presentation materials. Gemini can pull metrics and screenshots from live dashboards across different regions and build a presentation in Google Slides. It can even analyze clothing photos from a local folder to research pricing and stage listings on resale platforms.

System requirements

To run the Gemini app with these new capabilities, users need to meet specific hardware and software standards. The Mac must be running macOS Sequoia 15 or a later version of the operating system. This ensures the underlying software architecture can support the app's native features.

Hardware requirements include at least 8 GB of RAM and 200MB of free disk space for the initial installation. While the app runs locally on the Mac, it still requires a stable internet connection. This is because the app must constantly access Google's servers to power the Gemini features.

The app was originally launched in April to provide a more productive way to access Google's AI on Apple hardware. This latest update builds on that foundation by adding deeper integration. It moves the app closer to becoming a primary tool for navigating the macOS environment.

The shift to agents

This update marks a transition from a chatbot that answers questions to an agent that performs actions. By giving the AI the ability to click and type, Google is enabling a new category of productivity. Users no longer have to copy and paste information between different windows manually.

The ability to approve key actions before they happen is a critical part of this rollout. It suggests that while the AI is doing the

Sources and links

First reported by Appleosophy. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news