OhAn artificial intelligence tool answered 83 percent of common genetic counseling questions correctly, including questions about genetic testing and genetic syndromes, a new study found.
Presented at the Society of Gynecologic Oncology’s annual meeting on women’s cancers in San Diego on March 17, 2024, the study examines the potential of a form of artificial intelligence (AI) called generative AI. Such tools predict possible options for the next word in any given sentence based on how billions of people on the Internet have used the words in context. A side effect of this next-word prediction is that creative AI chatbots like ChatGPT can generate answers to questions in realistic language and produce clear summaries of complex text.
Led by researchers at NYU Langone Health and its Perlmutter Cancer Center, the current paper explored the use of ChatGPT to answer questions about gynecologic oncology counseling for genetic syndromes.
“Our data show that this tool has the potential to reduce anxiety and answer common questions for patients to keep them informed,” said study lead investigator Jharna M. Patel, MD, of NYU Langone. I hold a gynecologic oncology fellowship. “More data input from gynecologic oncologists is needed before this tool can help inform patients about their cancer and as an adjunct to human providers alone.”
40 questions
For the study, the research team consulted ophthalmologists to select 40 questions that were frequently asked of patients and aligned with the society’s professional guidelines. They then typed the questions into ChatGPT version 3.4, seeking answers.
The authors then asked attending gynecologic oncologists to rate the chatbot responses on the following scale: 1) accurate and comprehensive, 2) accurate but not comprehensive, 3) somewhat accurate, somewhat inaccurate, and 4 ) completely wrong. The proportion of responses achieving each score was calculated overall and within each question category.
Specifically, ChatGPT provided correct and comprehensive answers to 33/40 (82.5 percent) questions, correct but not comprehensive answers to 6/40 (15 percent) questions, and partially incorrect answers to 1/40 (2.5 percent) questions. found . No answer was marked as completely incorrect.
The genetic counseling category of questions (for example, How do you know if someone has a hereditary or familial cancer syndrome?) had the highest proportion of AI responses that were both accurate and comprehensive, with ChatGPT scoring 100 percent. Answered all 20 questions correctly. ChatGPT performed equally well with questions about specific genetic disorders. Obstetricians found, for example, that 88.2 percent (15/17) of responses were correct and complete when testing for a variation in the BRCA1 or BRCA2 gene, which is known to be a driver of cancer risk, while 66.6 percent ( 2/3) were correct on Lynch syndrome, the most common form of hereditary colorectal cancer. A complete list of study questions is included below.
“We think we can further improve these results by training the AI tool on more data, and by learning to ask better sets of questions,” said study senior author Marina Stasenko, MD, an assistant professor in the Department of Obstetrics and Gynecology. is a professor and says. Gynecology at NYU Grossman School of Medicine. “The goal is to deploy it in the clinic when ready, but only as an adjunct to human providers.”
Along with Dr. Patel and Dr. Stasenko, study authors were Kathryn Herman, MD, and Whitfield B. Gruden, MD, from the Department of Obstetrics and Gynecology. Study author Emeline M. Ivicky, MD, MBA, was from the Department of Obstetrics and Gynecology at NYU Grossman Long Island School of Medicine.
Media inquiries
Greg Williams
Phone: 212-404-3500
Gregory.Williams@NYULangone.org