ChatGPT matches oncologist accuracy in screening cancer treatment videos

The challenge of online health info

Patients undergoing cancer immunotherapy often rely on online platforms to understand their treatment and identify potential side effects. These patients frequently search for information on immune-related adverse events and instructions on when to contact their medical teams. However, the quality of information on public video platforms varies significantly, creating a risk for those seeking reliable guidance.

For medical specialists, manually reviewing every available online video to ensure it meets clinical standards is a massive undertaking. Because of this resource-intensive process, researchers have looked into using large language models to help screen content. The goal is to create a preliminary review system that can flag high-quality resources for human experts to verify.

This specific study examined 90 English-language YouTube video transcripts focused on cancer immunotherapy. The researchers wanted to see if automated systems could align with the judgment of medical oncologists. They used a panel of three specialists to establish a gold standard for what constitutes adequate patient education.

Testing the AI models

The research team evaluated four different large language models to see how they performed against the oncologist consensus. These models included ChatGPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, and MedGemma 27B. Notably, MedGemma 27B is an open-weights model specifically adapted for medical applications, while the others are more general-purpose systems.

To ensure consistency, each AI model evaluated the entire set of 90 transcripts across three independent sessions. The researchers used four validated instruments to score the content, including the DISCERN instrument and the Global Quality Score. These tools measure various factors like understandability and how well the content provides actionable information for the patient.

The results showed that ChatGPT-5.5 was the most accurate of the group. It achieved a sensitivity of 94.3 percent and a specificity of 89.1 percent when classifying educational adequacy. This indicates that it was highly effective at identifying correct information while maintaining a relatively low rate of false positives compared to its peers.

Failures in other AI systems

In contrast to ChatGPT-5.5, the other models demonstrated a significant tendency toward leniency. These systems frequently passed deficient transcripts as clinically adequate, which could lead to patients receiving misleading information. For example, Gemini produced 43 false-positive classifications, and MedGemma produced 26 false-positives during the study.

Interestingly, the study found that being specifically adapted for medicine did not necessarily improve accuracy. MedGemma 27B failed to show better agreement with specialists than the general-purpose architectures. This suggests that the underlying architecture of the model may be more important than specific medical fine-tuning for this type of content analysis.

The researchers also noted that high-quality education was relatively rare in the sample they tested. Only 35 of the 90 transcripts, or 38.9 percent, met at least three of the four adequacy criteria. Only 26.7 percent of the videos satisfied every single threshold required for comprehensive clinical adequacy.

What the data tells us

One major finding of the study was that popularity metrics on video platforms do not correlate with content quality. The researchers confirmed that high engagement figures do not identify high-quality educational material. This means that a video with millions of views is not necessarily a safe or accurate source for cancer patients.

While ChatGPT-5.5 showed substantial statistical agreement with specialists, the researchers were careful about the implications. They noted that while specific model configurations show potential for prioritizing materials, the findings do not establish clinical validity. The AI is not yet ready to be used as an autonomous screening tool in a healthcare environment.

The study concludes that human oversight remains indispensable for evaluating online resources. Because some models are systematically lenient, they could easily miss dangerous inaccuracies. For now, automated tools can only support administrative workflows rather than replacing the final clinical review by a medical professional.