TL;DR
Get health and wellness essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A study presented at the American Society for Reproductive Medicine’s annual meeting found that physicians rated ChatGPT answers about IVF less favorably than ASRM patient-education materials for accuracy, completeness, safety and overall quality. ChatGPT’s clarity ratings were similar, but half of its answers included at least one unsupported claim and 4.2% contained misinformation. Researchers cautioned that the analysis tested only one chatbot version and did not measure patient outcomes.
Physicians rated ChatGPT’s answers to common IVF questions lower than educational materials from the American Society for Reproductive Medicine (ASRM) for accuracy, completeness, safety and overall quality, according to a comparative analysis presented at the society’s annual meeting. The chatbot’s answers received ratings for clarity comparable to ASRM’s materials, but researchers also found unsupported claims and some misinformation.
In the study, four physicians scored the chatbot’s responses and two ASRM documents on a five-point scale. ChatGPT averaged 3.98 for accuracy, compared with 5.00 for ASRM materials; 3.40 for completeness versus 4.88; 2.92 for safety versus 4.88; and 3.38 for overall quality versus 5.00. Clarity ratings were close: 4.60 for ChatGPT and 4.63 for ASRM.
Researchers also checked whether responses covered key concepts in ASRM’s ReproductiveFacts.org materials. ASRM documents met 100% of the checklist criteria on average, while ChatGPT responses averaged 94.1%. Half of the chatbot answers contained at least one unsupported claim, and 4.2% contained misinformation; researchers found none in the ASRM responses.
The team used the free GPT-5.6 Luna and ran each prompt three times, producing 12 responses about the IVF process and associated risks. Physicians assessed the responses for accuracy, completeness, clarity, safety and quality, as well as checklist adherence and readability. The analysis also compared ChatGPT’s IVF success estimates by age with data from the CDC and the Society for Assisted Reproductive Technology. The chatbot generally reflected declining birth rates with age, but overestimated live birth rates at age 35 and underestimated them at ages 37 and 40.
Clear Answers Can Still Miss Key Details
The comparison highlights a practical risk for people using chatbots to understand fertility treatment: an answer can be easy to read and sound confident without being as complete or safe as a specialist society’s educational material. In IVF, details about procedures, risks and age-related outcomes can shape what patients ask clinicians and how they interpret treatment options.
Study author Nicole Fischer, a reproductive endocrinology and infertility fellow at the University of Pittsburgh, said fluent wording can coexist with omissions, outdated information and errors. The findings do not show that ChatGPT answers are uniformly wrong, nor do they establish how patients act on them. They do suggest that readability alone is not a reliable measure of medical accuracy.
The researchers’ readability assessment also found that answers generated from conversational prompts were around an eighth-grade reading level, while generic prompts produced responses closer to a 10th-grade level. ASRM materials were roughly at a 12th-grade level on the Flesch-Kincaid scale. That variation suggests the way a patient asks a question may affect how accessible the response is, though readability does not establish whether a person understands or can safely use the information.
Products Worth Considering
Fitbit Inspire 3 Health and Fitness Tracker Bundle
Fitbit Sense Health & Fitness Smartwatch (White/Gold, International Version)
Garmin Forerunner 570 GPS Running Smartwatch, 47mm, Aluminum, Black
As an affiliate, we earn on qualifying purchases.
How Researchers Compared IVF Answers
The analysis was designed to examine whether patients asking chatbots about assisted reproductive technology receive answers that are complete and safe. Researchers compared responses to common questions about the IVF process and its risks with ASRM patient-education materials, then separately examined estimated success rates across ages 20 to 45 against CDC and SART data.
Large language models generate text by predicting likely responses from training data. On technical or newer subjects, they can produce plausible-sounding material that includes unsupported details or fabricated information. Fischer said the study tested one version of one chatbot; the authors said newer or specialized systems, including ChatGPT Health, warrant separate testing. The findings were presented at the ASRM annual meeting and reported by MedPage Today on October 30, 2026.
“The key point here is that fluent, confident wording can coexist with omissions, outdated information, and errors.”
— Nicole Fischer, MD, MPH, reproductive endocrinology and infertility fellow at the University of Pittsburgh
Limits of the IVF Chatbot Test
The study had a small sample and assessed only one version of one large language model. The ratings included subjective measures, and the analysis did not track patient-level outcomes, so it cannot show whether chatbot use changed treatment decisions, understanding or health outcomes. The findings also do not establish how other chatbots or later versions would perform.
The report does not provide the full prompts, response texts or age-specific success-rate figures in the supplied account. It is also unclear how well the results apply to questions beyond those tested or to patients using different wording. The reported estimates compare with CDC and SART data, but the study’s summary does not give the exact reference periods or figures for each age.
Testing Newer Models and Patient Use
Fischer said newer and specialized language models, including ChatGPT Health, should be tested in future research. Further work could use larger samples, more chatbot versions and patient-level measures to examine whether answers are understood and how they affect conversations or decisions. No such next study or timeline was specified in the report.
For now, Fischer encouraged clinicians to ask patients whether they use chatbots for fertility information, suggest prompting systems to cite sources, and direct patients to trusted resources such as ASRM education materials. The study does not assess chatbot use as a substitute for medical care; questions about an individual’s IVF treatment and chances of success remain matters to discuss with a qualified clinician.
Key Questions
Did ChatGPT score better than ASRM materials on any measure?
No. Physicians rated ChatGPT lower on accuracy, completeness, safety and overall quality. Its clarity score was similar to the ASRM materials: 4.60 versus 4.63 on a five-point scale.
How often did the study find unsupported claims or misinformation?
Half of ChatGPT’s answers contained an unsupported claim, and 4.2% contained misinformation. The researchers reported no misinformation in the ASRM responses they assessed.
Were ChatGPT’s IVF success estimates accurate?
The researchers said estimates generally aligned with CDC and SART data and reflected declining birth rates with age. The chatbot overestimated live birth rates at age 35 and underestimated them at ages 37 and 40.
Can the findings be applied to every AI chatbot?
No. The study tested one version of one chatbot with a small set of responses. Its results do not establish how other systems, newer versions or specialized medical chatbots perform.
What did the researchers recommend for patients and clinicians?
Fischer encouraged clinicians to ask patients about chatbot use, teach them to request cited sources and point them to trusted educational materials. The study supports checking medical information against reliable sources and a qualified clinician, rather than treating a fluent chatbot response as definitive.
Source: medpagetoday.com
