Gemini for Mac gains computer use capabilities in the US
Google is launching a feature that allows the Gemini app to navigate, click, and type within other programs on Apple silicon Macs.
The short version
- Gemini can open apps and websites to click and type through repetitive tasks.
- The feature is currently limited to English in the US for users 18 and older.
- Users can pause the agent at any time and must approve key actions before they occur.
- The agent automatically pauses for sensitive actions like logins or payments.
- Hardware requirements include an Apple silicon Mac running macOS Sequoia 15 or later.
Why it matters
This marks a shift from AI that just generates text to an agent that can perform actions on your behalf. It could automate complex workflows across multiple different software tools.
Automating your desktop
Google has introduced a new capability for its Gemini app on Mac that allows the AI to operate other programs independently. This feature, which Google calls computer use, enables the assistant to open specific apps and websites to complete tasks. Instead of just providing information, the AI can now click and type its way through repetitive workflows on your computer.
The rollout of this functionality is currently restricted to English-speaking users in the United States. To access these capabilities, users must have a personal Google account and be at least 18 years old. The service also requires a paid subscription to Google AI Plus, Pro, or Ultra.
Hardware compatibility is a specific requirement for this new tool. It is currently only available for Macs equipped with Apple silicon chips that are running macOS Sequoia 15 or later. Google has not yet provided a timeline for when these capabilities will be released in other countries.
How the agent operates
To activate the agent, users can add the specific command "@Computer use" to their prompts. Once triggered, Gemini begins its work inside an isolated background window. This design choice allows the user to continue working on other tasks while the AI performs its actions in a separate space.
While the agent works in the background, it provides a live preview directly within the chat interface. This allows the user to monitor the progress of the AI in real time. According to 9to5Google, the system is designed to navigate through the interface much like a human would by interacting with the visual elements of the software.
Google is also shipping desktop connectors for specific professional software, including Blender and Autodesk Fusion. These connectors likely help the agent interact more effectively with complex 3D modeling and design tools. This suggests a focus on making the agent useful for creative and technical professional workflows.
Permissions and safety
Users maintain a level of control over what the AI is allowed to do on their machines. You can choose which specific apps Gemini is permitted to access during a session. Furthermore, the system is designed to let you pause or stop the agent at any moment if it behaves unexpectedly or if you need to take over.
Safety measures are in place for high-stakes activities. The agent is programmed to pause automatically when it encounters prompts for logins or payment details. This prevents the AI from completing sensitive financial transactions or entering private credentials without direct human intervention.
There is currently no official word on whether Google stores the visual data the agent sees on your screen or uses that content to train its models. However, the Gemini help page warns users against entering sign-in information or payment details into task threads. Temporary backup files created during these tasks are deleted within 24 hours at the latest.
Privacy and system access
Because the agent requires the ability to interact with your system, privacy remains a primary consideration for many users. Anyone granting an AI agent access to a Mac should monitor their permissions within the System Settings. This is especially important as the AI needs to navigate various parts of the operating system to complete tasks.
Apple is also planning to implement restrictions on these types of rights for AI agents. These OS-level controls are intended to limit how much data an app can see or do. Users can currently check their Full Disk Access settings to see what permissions are granted to various applications.
Google has previously separated Gemini history from its AI training processes. However, users should remain aware of how the new computer use feature handles screen data. Keeping a close eye on permissions is the recommended way to manage the balance between automation and personal privacy.
The rollout strategy
This launch follows a pattern of gradual deployment for Google's agent features. For example, when the Gemini app for Windows first launched, users in Germany did not receive the Spark agent. Google has not provided a specific reason for why certain regions are delayed in receiving these updates.
The current rollout for the Mac version is also described as gradual. Reports from Macworld indicate that some versions of the app tested during the rollout did not have the computer use feature enabled yet. This suggests that the feature is being pushed out to users in waves rather than all at once.
As the tool becomes more common, the focus will likely shift toward refining the accuracy of the clicks and typing. For now, users in the US can begin exploring how these capabilities integrate into their daily workflows. The success of these early tests will likely determine how quickly the feature expands globally.
What to watch
- Whether Google clarifies if screen data from 'computer use' is used for AI training.
- When the feature will be released to users outside of the United States.
Try it
Check your System Settings to see which apps currently have 'Full Disk Access' permissions.
Sources and links
- Officialannounced the feature, called computer use, on November 19 blog.google
- Officialmacworld.com/article/3291400/gemini-can-now-tap-into-your-mac-for-app- macworld.com
- ReportingNotebookcheck: the article we sourced
Read the original at Notebookcheck
First reported by Notebookcheck. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.
Liked it? Say why
Add a line for other readers.
No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.