SIAgentNews.com
← All news
DevelopingGeminiAgents

Google introduces on-demand tool retrieval for Gemini API agents

A new feature allows developers to manage large libraries of AI tools without inflating costs or slowing down model responses.

· First reported by Unite.AI

Isometric diorama: picture for the story 'Google introduces on-demand tool retrieval for Gemini API agents'
AI-made isometric diorama for this story. No real people or products are shown.
23,650input and tool-use tokens for 200 tools using retrieval

The short version

  • Tool Retrieval defers full schemas, keeping only names and descriptions in the initial prompt to save on input tokens.
  • The feature is available in preview for the Gemini Interactions API, supporting Gemini 4 Argon, 3.8 Flash, and 3.6 Flash.
  • Developers can choose between server-side search, where Google handles the retrieval, or client-side search for custom ranking.
  • Google recommends using this feature once a tool library exceeds roughly 20 to 30 tools.

Why it matters

This makes it much cheaper and faster to build complex AI agents that need to choose from hundreds of different actions. It solves a scalability problem where large tool libraries previously made AI responses too expensive or less accurate.

Solving the tool library bloat

When developers build AI agents, they often give the model a set of 'tools'—specific functions the AI can call to perform tasks like checking weather or booking flights. As these libraries grow to include dozens or even hundreds of tools, a technical problem arises regarding how the model understands them.

Traditionally, every single tool's full schema, which defines its parameters and requirements, must be sent to the model with every single request. This constant inclusion inflates the number of input tokens, which increases costs, adds latency, and can actually make it harder for the model to pick the correct tool from a crowded list.

Google's new Tool Retrieval feature addresses this by deferring the full schemas. Instead of sending everything at once, the model initially only sees the name and a brief description of each tool. The full details are only loaded into the model's context when the agent specifically asks for them during a task.

How the retrieval systems work

There are two primary ways developers can implement this new functionality. In the default server-side mode, developers add a specific search tool and set a property called defer_loading to true. Google then handles the search process internally, keeping the developer's prompt prefix intact for better caching.

Client-side retrieval offers a different approach by moving the search, ranking, and filtering tasks to the developer's own application. This method allows for more customization, such as introducing new tools mid-conversation—like when a user connects a new integration—without needing to declare them at the very beginning of the interaction.

Regardless of the method chosen, the system works by inserting the retrieved schemas exactly where they are loaded rather than at the start of the prompt. This ensures that the model receives the necessary information only at the moment it becomes relevant to the current step of the conversation.

Cost and performance trade-offs

While Tool Retrieval does not have a separate fee, it changes how tokens are billed. Each deferred tool adds a small, fixed input block—roughly 26 tokens—to the request. However, every time the model successfully loads a schema, those tokens remain billable as input for all subsequent turns in that conversation.

The system also requires an additional model call for the search itself, which is charged at the input rate. This means that for very small tool libraries, the cost of performing a search might actually be higher than simply sending the full schemas every time. Google's guide highlights that for a small library of 10 tools, retrieval could actually increase the total token count.

For large-scale applications, the savings are significant. In a test case involving 200 tools, static loading required 164,000 input tokens, while the retrieval method used about 23,650. This makes the feature particularly effective for complex agents that need to navigate a massive catalog of capabilities.

Technical constraints and support

The new feature is currently available in preview specifically for the Gemini Interactions API, which is Google's recommended interface for new projects. It is not supported by the older generateContent method. The supported models for this preview include Gemini 4 Argon, Gemini 3.8 Flash, and Gemini 3.6 Flash.

There are several operational rules developers must follow. Once a tool is retrieved during a conversation, it cannot be unloaded. Additionally, the complete tools array must be resent with every request, meaning the system does not store tool declarations between separate turns or interactions.

Google suggests that developers should consider switching to this retrieval method once their tool library reaches approximately 20 to 30 tools. By using this system, developers can keep much larger catalogs on standby while maintaining a balance between performance and cost.

What to watch

  • Whether the preview API surface changes before the general availability launch.
  • How developers optimize their tool descriptions to improve the accuracy of the retrieval search.

Try it

If you are building an AI agent with more than 30 different functions, check the Gemini Interactions API documentation to see if you can implement 'defer_loading' to reduce your token costs.

Sources and links

First reported by Unite.AI. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news