Google brings Gemini computer use to Mac app
Google's Mac app can now perform multi-step tasks like listing items for sale or filling out PDFs by interacting with your screen.
The short version
- Gemini can now see your screen via screenshots to understand context.
- The AI can click, type, scroll, and navigate websites or programs for you.
- Users can automate multi-step workflows like batch listing items on eBay.
- The app works in a background window and does not take over your cursor.
- Gemini requires user approval for key actions and only accesses permitted apps.
Why it matters
This shifts AI from a chatbot that just generates text to an agent that can execute work across different software. It allows users to automate tedious, manual data entry and administrative tasks that previously required constant human clicking.
Expanding Gemini to Mac
Google has updated its Gemini app for Mac to include a new capability for handling repetitive tasks. While this functionality has been available to users on Windows computers for some time, Mac users can now access the same automation features. The update allows the AI to interact with various programs and websites directly.
The core of this update is the integration of what Google calls 'Computer Use.' This technology enables the AI to interpret what is happening on a user's screen and perform actions as if it were a human operator. Instead of just providing information, the tool is designed to execute sequences of actions across different applications.
As reported by Engadget, the goal is to let users complete tasks that typically require several manual steps. By using natural language commands, people can tell the AI what they want to achieve, and the system handles the navigation. This move positions Gemini as a more active assistant rather than a passive search or writing tool.
How the technology works
To perform these tasks, Gemini uses a system of screenshots to see what is currently displayed on the screen. It analyzes these images to identify buttons, text fields, and navigation menus. Once it understands the layout, it can simulate human interactions like clicking, typing, and scrolling through pages.
The AI operates within a dedicated background window to perform these actions. This means that while the AI is working on a task, it does not take over your mouse cursor or your screen. You can continue to use your Mac normally while the AI completes the requested work in its own space.
Transparency is a key part of the design, as the app provides a live preview of its progress within the chat interface. This allows the user to monitor what the AI is doing in real time. Furthermore, the AI is restricted to only interacting with the specific programs and applications that the user has granted it permission to access.
Practical use cases
Google highlighted several specific ways that users can apply this new capability to their daily workflows. One example involves e-commerce, where a user might want to list a large number of items for sale on platforms like eBay. The AI can go through a local folder of photos, research current pricing, and then stage the listings for the user.
The tool is also useful for administrative tasks that involve local files. For instance, a user could ask Gemini to fill out shipping declaration forms that are saved as PDFs. The AI can navigate the document and input the necessary details, which saves the user from manual typing across multiple forms.
Creative professionals can also use these capabilities to speed up their production pipelines. One example provided by Google is using the AI to automatically turn a pencil sketch into a 3D model. By handling the repetitive steps of conversion and modeling, the AI allows the creator to focus on the initial concept.
Safety and user control
Because the AI has the power to interact with your computer, Google has implemented several guardrails to ensure user control. The system is designed to ask for explicit approval before it performs any key actions. This ensures that the AI does not execute significant changes or deletions without a human saying yes first.
The limitation of the AI's reach is another important factor for security and privacy. Gemini can only interact with the software and apps that the user has specifically authorized. This prevents the AI from wandering into private folders or applications that are not part of the intended task.
By combining these permissions with the background window execution, Google aims to provide a balance between automation and safety. Users get the benefit of the AI doing the 'heavy lifting' of clicking and typing without losing the ability to see what is happening or intervene at any moment.
What to expect next
The rollout of 'Computer Use' to Mac marks a significant step toward more autonomous AI agents. As the technology matures, it is likely that the variety of supported actions will expand. Users will be able to automate increasingly complex workflows that span across multiple different web platforms and desktop applications.
For the average user, this means that the barrier between 'thinking' of a task and 'doing' a task is shrinking. Instead of learning how to use a specific software's interface, users can simply describe the desired outcome. The AI then figures out which buttons to press and which pages to navigate to get there.
As Google continues to refine these capabilities, the focus will likely remain on making the interaction as seamless as possible. The goal is to turn Gemini into a tool that can handle the mundane, repetitive parts of digital work. This allows people to spend less time on data entry and more time on high-level decision making.
What to watch
- How many new apps will be added to the supported list for Gemini?
- Will Google introduce more advanced 'Computer Use' features for mobile devices?
Try it
Open the Gemini app on your Mac and try asking it to perform a multi-step task, like researching prices for a list of items you want to sell.
Sources and links
- Officialaccess Gemini's "Computer Use" blog.google
- ReportingEngadget: the article we sourced
First reported by Engadget. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.