SIAgentNews.com
← All news

Splitting image analysis among specialist AI agents cut mannequin false alarms to 8 percent

In a forensic test, AI models mistook mannequins in uniform for soldiers, but a structured team of specialist agents fixed most of those errors.

· First reported by Bioengineer.org

8 percentmannequin false positives with LangGraph, vs 27 percent baseline

The short version

  • Three models were tested on 434 phone images: 198 of military personnel, 99 of mannequins and 137 non-military.
  • Every baseline model misclassified over 85 percent of the mannequin images, leaning on uniforms rather than signs of a real person.
  • LangGraph reached 0.90 precision and cut mannequin false positives to 8 percent, from a 27 percent baseline average.
  • The agent setups ran about 2.8 times slower and found fewer cases overall, with recall of 0.73 for LangGraph and 0.74 for CrewAI.
  • The agent setups can decline to decide and attach evidence to each finding, so a human can check the reasoning.

Why it matters

AI that looks at images is already being considered for investigations, where a confident wrong answer can do real harm. This study suggests the fix is less about a smarter single model and more about structure: separate checks, written evidence and permission to say 'not sure'. The same lesson applies to anyone who relies on an AI verdict about a photo.

The mannequin problem

A research team tested how well today's multimodal large language models handle forensic image work. These are AI systems that read both pictures and text. They tried Google's Gemini 1.5 Pro, the open-source LLAVA and OpenAI's GPT-4o, as Bioengineer.org reported from a study in the International Journal of Networked and Distributed Computing.

The models were good at finding what they were asked to find. Gemini 1.5 Pro hit a recall of 0.99, meaning it caught nearly every real case. But the three models averaged a precision of only 0.68. Precision is the share of flagged images that were actually correct, so many of the flags were wrong.

The main culprit was mannequins dressed in military gear. Each baseline model misclassified more than 85 percent of the mannequin images. The researchers concluded that the models lean on uniform patterns instead of cues that mark a real person, such as skin texture or body language.

How the test was built

The dataset held 434 images from an iOS backup. There were 198 pictures of military personnel, 99 of mannequins in military clothing and 137 non-military images as a control group. All were sent through WhatsApp and pulled out with iMazing software, to mimic how images are recovered from phones in real investigations.

Each model described every image in free text. A separate OpenAI model, o3-mini-2025-01-31, then turned those descriptions into structured JSON, a tidy data format, so the team could calculate precision, recall and accuracy.

The models differed on context. Gemini 1.5 Pro and LLAVA each scored 0.793 on spotting military personnel, with GPT-4o at 0.770. On naming the country, Gemini reached 0.787, while GPT-4o managed 0.385 and LLAVA 0.121. Even the best model reached only 0.254 on naming the military unit.

A team of specialists instead of one pass

The researchers then wrapped the models in two agentic frameworks, meaning setups where separate software roles each handle one step. CrewAI uses collaborating sub-agents. LangGraph runs a fixed, step-by-step pipeline with no loops.

The work was split into named roles. One agent checked file metadata and hashes (digital fingerprints of files). Another detected objects and read text using YOLOv8 and Tesseract. A mannequin agent looked for human-authenticity cues such as skin texture and gaze. An attribution agent compared findings against a gallery of five example images per country and branch. A cross-check agent could reject reports that did not add up.

The pipeline also used confidence thresholds, with a mannequin confidence requirement of 0.6 to 0.7, and strict format checks between steps. The aim was to turn free-form guessing into a process that can be audited.

What improved and what it cost

LangGraph reached a precision of 0.90 and held mannequin false positives to 8 percent, against a baseline average of 27 percent. CrewAI reached 0.88 precision with a 0.12 false-positive rate. The two frameworks differed in style: CrewAI wrote narrative reasoning useful for exploring a case, while LangGraph produced structured output suited to audit-ready documentation.

There were trade-offs. Recall dropped to 0.73 for LangGraph and 0.74 for CrewAI, and the LangGraph setup took about 2.8 times longer than a single pass. The researchers say the recall drop is deliberate. The systems were set to abstain when confidence was low, rather than guess.

The agent outputs also carried evidence citations and decision traces. The authors say this lines up with forensic documentation standards such as NIST SP 800-101 and ISO/IEC 27,037, because a human analyst can follow each step instead of trusting an opaque score.

Reading the result with care

This is one study on one narrow task, with a fairly small dataset and older model versions. The numbers show a clear pattern, but they do not prove the same gain would appear with other images, other models or other kinds of evidence.

The broader point is still useful. A single model that answers in one go tends to be confident and wrong in predictable ways. Splitting a job into roles, forcing each claim to cite evidence and letting the system say 'I can't tell' traded some coverage for far more trust in what it did report.

What to watch

  • Whether the gains hold with newer models and images beyond this one dataset of 434.
  • How investigators weigh the 2.8-fold slowdown and lower recall against fewer false positives at large scale.

Try it

Next time you ask an AI chatbot about a photo, ask it to list the visible evidence for its answer and what it cannot tell. Treat the answer as one input.

Sources and links

First reported by Bioengineer.org. Written by the SI Agent News Desk from the reporting above.Written with AI from the source above. Every number is checked against the source. Spot a mistake? Corrections.

Liked it? Say why

Add a line for other readers.

No account needed. We never show your email. Comments with links, ads or abuse are held and not shown. Your comment shows after a quick check.

More AI news