Google releases framework to define subjective image categories

Defining the subjective

Many image classification tasks rely on subjective concepts that lack clear, objective boundaries. For example, determining what qualifies as 'healthy food' or 'inspiring artwork' is difficult because the criteria can vary significantly between different people and cultures.

Standard AI tools usually assume that the human user already knows exactly where the line is drawn. If the user's initial description is too vague, the resulting AI classifier will often produce inconsistent or incorrect results, making it difficult to use for professional applications like content moderation.

Google Research addressed this by releasing Agile Deliberation. This framework is designed to help users discover and refine the boundaries of a concept through a structured process. Instead of just taking a prompt at face value, the system helps humans find the specific images that sit on the edge of a definition.

How the framework works

The process begins with concept scoping, where the system breaks a broad idea into up to three smaller unit concepts. For instance, if a user wants to define 'healthy food,' the system might suggest positive subconcepts like 'salads' and negative subconcepts like 'processed food.' The user then keeps, flips, or drops these suggestions based on their specific needs.

The next phase focuses on finding borderline images. Because generative vision models can have uncalibrated confidence scores, the system does not rely on them to find difficult cases. Instead, it uses Gemini 2.5 Flash to write specific queries for ambiguous items, such as 'salads with heavy mayo dressings,' and retrieves images from a large dataset.

These images are grouped into clusters, and the system uses a multi-armed bandit algorithm to pick which ones to show the user. The user labels these specific examples and provides notes on any disagreements. The system then uses an LLM to turn those human notes into a structured written definition that serves as the final policy and the prompt for the classifier.

Testing and performance

To evaluate the framework, researchers conducted 18 live sessions with participants who did not have professional experience in content moderation. These participants spent about 45 minutes defining concepts like 'paid to play' clickbait and 'healthy food' using the framework compared to a manual deliberation process.

The results showed that Agile Deliberation significantly outperformed standard zero-shot modeling. On average, the framework provided a 10.5 point gain in F1 scores, which is a metric used to measure the accuracy of a classifier. For the 'paid to play' concept, the F1 score rose from 0.48 to 0.59.

While the framework improved overall accuracy, there were some trade-offs in specific metrics. For the 'paid to play' category, recall dropped by 17 points, and for 'healthy food,' recall dropped by 15 points. However, the precision for 'paid to play' saw a notable increase of 22 points, moving from 0.35 to 0.57.

Developer accessibility

Google has made the framework accessible to the developer community by releasing the code under the Apache 2.0 license. The repository includes a demo notebook that walks users through the 'healthy food' example, requiring only a Gemini API key to run.

The system is built to be lightweight and efficient. It uses CLIP embeddings and a FAISS index for retrieval, with the default demo index requiring about 260 MB of space. Developers can run the framework on standard hardware using either a CPU or a GPU for the indexing components.

By providing an open-source framework that handles the 'slow part' of moderation—finding the right images to label—Google is lowering the barrier for building high-quality classifiers. Users reported that the framework significantly reduced the stress and effort typically associated with manual prompt engineering and large-scale data labeling.