Artificial intelligence (AI) technologies such as large language models (LLMs) and chatbots show signs of deteriorating cognitive abilities with age akin to humans, a new study published in the journal BMJ has claimed. The study comes at a time when users are increasingly asking AI tools for medical diagnoses, owing to their ability to simplify medical jargon.
The researchers made the claim after assessing the cognitive abilities of leading LLMs – ChatGPT versions 4 and 4o, Claude 3.5 “Sonnet” (developed by Anthropic), and Gemini versions 1 and 1.5 (developed by Alphabet), using the Montreal Cognitive Assessment (MoCA) test.
While the MoCA test is mostly used to detect cognitive impairment and early signs of dementia in older adults, the study posed the questions to LLMs to test their abilities in attention, memory, language, spatial skills and executive mental function.
In humans, 26 out of 30 is considered a passing score (ie the subject has no cognitive impairment). However, the test results found that only ChatGPT 4o (26 out of 30) achieved the desired points while the likes of ChatGPT 4 and Claude (25 out of 30) failed to cross the threshold. Gemini 1.0 scored the lowest of the lot by only gaining 16 points.
“All chatbots showed poor performance in visuospatial skills and executive tasks, such as the trail making task (connecting encircled numbers and letters in ascending order) and the clock drawing test (drawing a clock face showing a specific time). Gemini models failed at the delayed recall task (remembering a five-word sequence),” the study highlighted.
Also Read | Can AI Tools Replace Human Couples Therapist? Study Says Yes
The pattern of impairment shown by the AI tools resembled that of human patients with posterior cortical atrophy, a variant of Alzheimer’s disease, the researchers added.
“These findings challenge the assumption that artificial intelligence will soon replace human doctors as the cognitive impairment evident in leading chatbots may affect their reliability in medical diagnostics and undermine patients’ confidence.”
The study results point out that it was unlikely that neurologists would be replaced by LLMs any time soon but they may soon “find themselves treating new, virtual patients” — the AI models.
