AI chatbots show diverging accuracy in male infertility answers

Tracking AI medical accuracy

Researchers conducted a repeated cross-sectional comparison to see how large language models (LLMs) handle sensitive medical information. They specifically chose the field of male infertility because it is highly dependent on strict clinical guidelines. Inaccurate information in this area can negatively impact a patient's diagnostic evaluation or their expectations for treatment.

The study compared three major platforms: ChatGPT, Perplexity, and DeepSeek. To ensure a fair test, the researchers used 88 prompts derived directly from the EAU Sexual and Reproductive Health Guidelines. These prompts were designed to be neutral and concise, reflecting how a real-world patient or clinician might seek rapid medical information.

Because the actual medical guidelines remained stable between 2025 and 2026, any changes in the AI responses could be attributed to the models themselves. The study looked at changes in model behavior, retrieval strategies, and how the systems selected their sources. This allowed the team to see how the underlying technology evolved over a one-year period.

How the models were tested

To simulate a real-world scenario, the researchers submitted each prompt to the platforms in fresh private chat sessions. They disabled memory functions and only kept the first complete response from each bot for analysis. This method ensures the results reflect a first-query interaction rather than a conversation where the AI might be guided by previous context.

Two academic urologists with expertise in male infertility scored every response independently. They were blinded to which model provided which answer and which year the data came from. This blinded approach helped eliminate bias and ensured that the scores reflected the actual content of the answers rather than brand loyalty.

The experts used several metrics to judge the quality of the AI responses. They used the DISCERN instrument to measure reliability, which looks at clarity, source balance, and the acknowledgment of uncertainty. They also used the Gunning Fog Index to estimate readability, which approximates the years of education needed to understand a text on the first reading.

Diverging paths for the bots

The results showed that the three models took very different paths between 2025 and 2026. ChatGPT showed significant improvements across all scored domains after its update to the 5.5 model. Its Global Quality Score and expert opinion scores both rose by one median point, while its readability also improved slightly.

DeepSeek showed the most dramatic growth of the three platforms. In 2025, it was the weakest performer, but by 2026, it improved on every single measure. Most notably, its reliability and expert opinion scores jumped from a median of 3 to a 5, showing a major leap in how well it handled clinical information.

Perplexity, however, told a different story. While it became easier to read, it actually lost ground in global quality and expert-rated clinical usefulness. Although it remained the platform most consistently associated with citations in earlier tests, its 2026 scores showed a decline in the quality of the information it provided to users.

Safety and clinical limitations

The study included a specific evaluation of patient-safety concerns. These are defined as inaccuracies or incomplete information that could lead to inappropriate treatment or delayed evaluation. In 2025, 8.0 percent of DeepSeek's responses were flagged for these concerns, but this dropped to zero in the 2026 assessment.

Despite these improvements, the researchers issued a stern warning about using these tools for medicine. They noted that fluent and well-structured answers can still contain errors or inappropriate clinical priorities. The absence of high-risk advice does not mean the platforms are safe for unsupervised clinical use by patients or doctors.

The authors also noted limitations in the study, such as the fact that they only analyzed a single response per prompt. This means the results do not account for the variability of outputs across repeated queries. Furthermore, the study only included English-language responses and did not compare the general-purpose bots against specialized clinical tools.

The future of AI health info

The findings suggest that large language models are becoming more capable of summarizing complex medical guidelines. They can serve as a tool for rapid access to information, but the data shows that model updates do not guarantee a uniform increase in quality. Some models may become more readable while simultaneously becoming less clinically accurate.

For the average user, this means that the platform you choose matters significantly depending on whether you value simplicity or expert-verified depth. The study concludes that verification against authoritative sources and human oversight remain necessary to ensure patient safety.

Moving forward, researchers will likely continue to monitor how these models handle specific clinical guidelines. As the technology evolves, the gap between linguistic fluency and clinical accuracy will remain a primary area of concern for medical professionals.