Surgeons rate ChatGPT answers on knee surgery as nearly perfect

Testing AI in the clinic

A new study published in Archives of Orthopaedic and Trauma Surgery examined how well ChatGPT handles complex medical questions from patients. Researchers focused specifically on total knee arthroplasty, which is a common procedure to replace a damaged knee joint. The goal was to see if the AI could provide reliable information to patients who are often seeking answers about their recovery.

The study evaluated 70 different questions divided into two categories. Fifty questions covered conventional knee replacement care, while 20 questions focused on fast-track recovery pathways. These modern pathways emphasize early mobilization and avoiding unnecessary hospital stays. By testing these specific areas, the researchers aimed to see if AI could keep up with contemporary surgical standards.

To ensure the questions were relevant, the researchers used a modified Delphi panel. This group included three high-volume surgeons, a physiotherapist, and an orthopedic nurse. They ranked questions based on how often they appear in clinical practice and their importance for patient safety. The 50 highest-ranked conventional questions and 20 fast-track questions were then used to prompt the AI.

How the AI was evaluated

The researchers used GPT-6 Sol to generate responses in Polish. They provided the model with standardized instructions to use patient-friendly language that matched current medical literature. To prevent the AI from being influenced by previous parts of a conversation, each question was entered into a brand-new chat session. Every response was recorded exactly as it appeared without any human editing or correction.

Three independent surgeons then reviewed every single response. They were blinded to each other's ratings and were not told which questions belonged to which category. Each surgeon scored the answers on a 5-point scale across four specific domains: accuracy, completeness, safety, and readability. This rigorous process ensured that the results reflected a high level of expert scrutiny.

The results were remarkably consistent across the board. The AI received a mean score of 4.99 for accuracy, 4.98 for completeness, and 4.98 for safety. Notably, every single one of the 20 fast-track responses received a perfect score of 5.00 from all three raters in every single category, totaling 240 out of 240 possible points.

Safety flags and nuances

Despite the high scores, the researchers noted a few rare instances where the AI's output required attention. Out of 210 individual expert assessments, only three indicated that a response needed correction by a specialist. These flags were concentrated in the conventional question group rather than the fast-track group. This suggests the AI handled the more modern recovery protocols with high precision.

One specific response was flagged as potentially unsafe by a single rater. The question concerned whether a patient needs prophylactic antibiotics before dental treatment after a knee replacement. The AI stated that routine prophylaxis is generally unnecessary but listed exceptions. The rater who flagged it preferred a more cautious approach for implant protection, though the other two raters gave it a perfect safety score.

The authors clarified that this flag was based on an interpretation of wording rather than a factual error. They noted that the AI's general advice aligned with American Dental Association guidance. Because there was no formal process to decide who was right in this instance, the authors cautioned that a single flag should not be viewed as a confirmed harmful answer.

Limitations of the study

The researchers highlighted several important limitations that readers should consider when interpreting these results. Because the conventional and fast-track questions were selected using different methods, the differences in their wording or difficulty might have influenced the ratings. Additionally, the scores were so high that it was difficult to measure the small differences between the two groups using standard statistical tests.

The study also had several structural constraints. The responses were only generated in Polish, and the evaluation was performed exclusively by surgeons. There was also no way to measure whether patients actually understood the AI's answers or if the information led to better clinical outcomes. These factors mean the study is an exploratory look at AI performance rather than a definitive proof of its clinical utility.

Furthermore, the study only captured a single moment in time for AI generation. The authors noted that future research should use the same selection process for all question types. They also suggested including a wider variety of question difficulties and more diverse raters. These steps would help determine if the AI's performance remains consistent across different types of medical scenarios.

The future of AI education

The findings suggest that AI-generated content could be a powerful tool for patient education. The researchers believe it may be most useful when integrated into supervised workflows. This could include preoperative information platforms, postoperative FAQ systems, or clinician-approved resources for patients to use at home. By providing accurate information, AI could help patients feel more prepared for their surgeries.

However, the study concludes that human oversight remains essential for safety-critical information. Because some nuances can be misunderstood, clinicians should review AI-generated content before it reaches patients. The goal is to use the AI's ability to generate conversational, readable text while ensuring that a human expert verifies the specific medical instructions for each individual case.

Moving forward, more prospective studies are needed to see if AI-supported education actually improves patient satisfaction. Researchers want to know if these tools can reduce the number of unnecessary calls to a doctor's office. Until those results are in, the message remains clear: the AI's answers were excellent by expert standards, but they still require a human touch for final verification.

try_itZinc

watch_list