Post by Jill Dosso
This blog post summarizes results from the peer-reviewed journal article “What does ChatGPT know about dementia? A comparative analysis of information quality” published in the Journal of Alzheimer’s Disease, 2023
The need for online information about Alzheimer’s Disease and dementia
Persons living with dementia and their care partners often wish for access to more and better information about living with the condition [1–3]. While healthcare providers are a valued resource for this information, their time is limited, and access can be challenging. At least 40% of older adults seek health information online [4], though this number may be much higher in some populations [5,6]. Online information about dementia exists on many virtual platforms, including social media, and varies widely in quality [1–3,7–10].
A recent analysis identified a number of barriers to online information access for persons living with dementia: information is targeted towards care partners and medical practitioners, rather than persons with lived experience; information can be pessimistic and hard to decipher; information is inaccurate or overly simple; and information is untrustworthy [11]. There is a clear demand for easily accessible, accurate dementia information that can be customized to a variety of user information needs.
ChatGPT: a new information source
ChatGPT (Conversational Generative Pre-training Transformer) is an online tool launched by OpenAI in November 2022. Users can engage in typed back-and-forth dialogue with the system through a web browser. Unlike a typical search engine query like a Google search, the platform retains information across an interaction, which creates a more natural and conversational experience. The “machinery” of ChatGPT is a generative artificial intelligence, meaning it uses machine learning models to create novel and data-driven content. These models have been trained on a dataset created from online materials and refined through human feedback [12,13]. The exact content of this dataset has not been publicly released.
How does ChatGPT stack up as a source of online information about dementia?
In a recent study, our research team at the Neuroscience, Engagement, and Smart Tech Lab at Neuroethics Canada asked how ChatGPT compared to other sources of online information about dementia. To create a set of questions that real users would likely have about dementia, we collected Frequently Asked Questions from the webpages of three national dementia organizations in Canada, USA, and Mexico. We posed these questions to ChatGPT-3.5 in April 2023. Responses from ChatGPT were evaluated using a standard tool previously developed by the NEST lab to assess the quality of online health information [14].
Strengths of ChatGPT-3.5: We found that ChatGPT, like the Alzheimer’s organizations, provided generally accurate information, directed users to bring their questions to a physician, and did not endorse commercial products.
Strengths of Alzheimer’s organization websites: Organizations were more likely than ChatGPT to state the limits of scientific evidence explicitly and produced more readable responses (i.e., responses had a readability score corresponding to a lower grade level). They were also more likely to link to local, specific, and actionable resources for support.
Conclusion
This research represents one snapshot of behaviour from a generative artificial intelligence tool: ChatGPT-3.5. This platform, and others, will continue to change over time and may produce different responses with different prompts or in languages other than English.
This work can support:
- Persons living with dementia and their care partners in screening potential sources of dementia information online;
- Healthcare providers as they advise persons living with dementia and their care partners; and
- Non-profit providers of dementia support services as they create helpful resources for their communities.
There is an ethical imperative to include persons with lived experiences of dementia in the creation of technologies to support them [15]. These perspectives are critically important for tools at the intersection of generative artificial intelligence and digital health. Understanding the online information available to these families is a first step in prioritizing their needs and perspectives in technology research and development.
Jill Dosso, PhD is a Postdoctoral Fellow in the Neuroscience, Engagement, and Smart Tech (NEST) lab at the University of British Columbia and BC Children’s Hospital. In her work, she studies the perspectives of persons with lived experience on emerging technologies to support brain health across the lifespan.
References
[1] Allen F, Cain R, Meyer C (2020) Seeking relational information sources in the digital age: A study into information source preferences amongst family and friends of those with dementia. Dementia 19, 766–785.
[2] Montiel-Aponte MC, Bertolucci PHF (2021) Do you look for information about dementia? Knowledge of cognitive impairment in older people among their relatives. Dement Neuropsychol 15, 248–255.
[3] Washington KT, Meadows SE, Elliott SG, Koopman RJ (2011) Information needs of informal caregivers of older adults with chronic health conditions. Patient Educ Couns 83, 37–44.
[4] Yoon H, Jang Y, Vaughan PW, Garcia M (2020) Older Adults’ Internet Use for Health Information: Digital Divide by Race/Ethnicity and Socioeconomic Status. J Appl Gerontol 39, 105–110.
[5] Levy H, Janke AT, Langa KM (2015) Health Literacy and the Digital Divide Among Older Americans. J Gen Intern Med 30, 284–289.
[6] Tam MT, Dosso JA, Robillard JM (2021) The impact of a global pandemic on people living with dementia and their care partners: analysis of 417 lived experience reports. J Alzheimers Dis 80, 865–875.
[7] Robillard JM (2016) The Online Environment: A Key Variable in the Ethical Response to Complementary and Alternative Medicine for Alzheimer’s Disease. J Alzheimers Dis 51, 11–13.
[8] Robillard JM, Johnson TW, Hennessey C, Beattie BL, Illes J (2013) Aging 2.0: Health Information about Dementia on Twitter. PLOS ONE 8, e69861.
[9] Robillard JM, Illes J, Arcand M, Beattie BL, Hayden S, Lawrence P, McGrenere J, Reiner PB, Wittenberg D, Jacova C (2015) Scientific and ethical features of English-language online tests for Alzheimer’s disease. Alzheimers Dement Diagn Assess Dis Monit 1, 281–288.
[10] Robillard JM, Feng TL (2016) Health Advice in a Digital World: Quality and Content of Online Information about the Prevention of Alzheimer’s Disease. J Alzheimers Dis 55, 219–229.
[11] Dixon E, Anderson J, Blackwelder D, L. Radnofsky M, Lazar A (2022) Barriers to Online Dementia Information and Mitigation. In CHI Conference on Human Factors in Computing Systems ACM, New Orleans LA USA, pp. 1–14.
[12] Introducing ChatGPT, Last updated November 30, 2022, Accessed on November 30, 2022.
[13] Forbes, The Next Generation Of Large Language Models, Last updated February 7, 2023, Accessed on February 7, 2023.
[14] Robillard JM, Jun JH, Lai J-A, Feng TL (2018) The QUEST for quality online health information: validation of a short quantitative tool. BMC Med Inform Decis Mak 18, 87.
[15] Robillard JM, Cleland I, Hoey J, Nugent C (2018) Ethical adoption: A new imperative in the development of technology for dementia. Alzheimers Dement 14, 1104–1113.
Neuroethics at the Core 