Dr. Scott Sperling

Welcome to Meet the Authors, a podcast brought to you by a collaboration of the Society for Clinical Neuropsychology and the journal Neuropsychology. My name is Dr. Scott Sperling, and I’m grateful to be your host. In this podcast, Student Leaders in Neuropsychology will discuss prominent recently published studies with the authors who undertook the research, thereby allowing for a behind the scenes look into the development, implementation, analysis and future implications of cutting edge neuropsychology research.

Today, our student leader, Ms. Sara Pishdadian, will be discussing an exciting paper entitled “Linking Self-Perceived Cognitive Functioning Questionnaires Using Item Response Theory: The Subjective Cognitive Decline Initiatives” with three of the paper’s authors, Dr. Rabin, Dr. Elbulok-Charcape and Dr. Jones. I’d like to now introduce our student leader and esteemed guest. Ms. Sara Pishdadian is a Ph.D. candidate in Clinical Neuropsychology at York University and a pre doctoral intern at the Ottawa Hospital.

Her research interests include the study of subjective and objective memory and spatial navigation abilities in older adults and individuals with memory disorders, an examination of the clinical correlates and cognitive predictors of daily functioning and quality of life in individuals with serious mental illness. Welcome, Sarah. And for our esteemed interviewees here, Dr. Rabin is a professor of psychology at Brooklyn College and Graduate Center of the City University of New York and visiting Professor in the Department of Neurology at Albert Einstein College of Medicine in the Einstein Aging study. Her work investigates the cognitive and neurophysiological changes associated with symptomatic prodromal stages of dementia to identify the earliest markers of dementia risk. And in addition, she studies changes in judgment and problem solving along the Alzheimer’s disease continuum with implications for preventing exploitation, unsafe behaviors and functional dependance among those most vulnerable.

Dr. Elbulok-Charcape is a recent graduate of the doctoral program in Educational Psychology at the Graduate Center of the City University of New York. Her research interests include examination of the factors that underlie statistical and research literacy in undergraduate students. Her current studies focus on improving statistics, education for at risk undergraduate student populations, identifying challenges in the neuropsychological assessment of racial and ethnic minorities, and enhancing the mental health literacy of diverse undergraduate students.

And last but certainly not least, Dr. Jones is a professor of neurology at the Brown University, Warren Alpert Medical School and an epidemiologist and methodologist with expertise in the use of modern psychometric methods, including item response theory and structural equation modeling to better understand and quantify constructs in health research settings.

Welcome, Drs. Rabin, Elbulok-Charcape and Jones.

Dr. Rich Jones

Thank you for having us. Thank you.

Ms. Sara Pishdadian

Wonderful. So thank you all for being here. In this paper, the authors used item response theories to calibrate data collected from 24 studies and 40 different self-perceived cognitive functioning questionnaires in 53,000 cognitively intact older adults. For those who may not have read your paper, can you briefly summarize the main findings of your study?

Dr. Laura Rabin

Sure. Let me begin by saying that this was a huge undertaking which would not have been possible without the collaborative efforts of the 43 coauthors and everyone affiliated with the contributing studies. Special thanks to Doug Tommet, who ran the analysis and to the other primary coauthors, Rich Jones, Paul Crane, and Sietske Sikkes. So I’ll start with a brief overview.

Self-perceived cognitive functioning is assessed using numerous different approaches with little consistency or agreed upon standards, and those in the field agree that there is a need to establish a core set of items with predictive validity for cognitive and clinical outcomes. And this item analysis project was intended as the first step in this effort. We published a descriptive study a few years ago that took an in-depth look at the key properties of questionnaires and items used in studies on subjective cognitive decline or SCD.

We found little overlap with approximately 75% of questionnaires used by only one study. There was also wide variation in response options and item content, and most instruments had limited or no psychometric support. Our next step was to use item response theory methods to combine item level data across studies to identify items making the greatest contribution to measurement precision.

So-called top items. So the current study was the first to calibrate self-perceived cognitive functioning data of geographically diverse older adults. With the resulting item score as being on the same metric. And this was not an easy task as no identical item was represented across studies. In some cases, a single item provided the only linkage between studies and this link is essential for harmonization.

After many attempts we were able to pull data from 40 questionnaires with over 600 items, including over 50,000 neuropsychologically intact older adults and representing multiple languages and countries. For the analysis, we identified psychometrically robust items from two main regions on the Leyton trait. The first region captures individuals with better self-perceived cognitive functioning with few concerns. And the second captures individuals with worse self-perceived cognitive functioning or a high level of cognitive concerns.

And as a reminder, all participants were neuropsychologically intact. Some of our main findings were that there were differences in the top items from the two ranges of interest, for example, in the high concern range, almost all of the top items related to memory, while in the low concern range top items related to various domains, not just memory, but also executive functioning.

For example, decision making, shifting between activities and managing a medication schedule and language, for example, word finding and self-expression. Also in the high concern range, a lot of the items tapped into the consequences of the difficulties. For example, memory problems, making it harder to complete tasks that used to be easy. The top items for the high concern range also tended to have dichotomous response options as opposed to Likert type scales.

And another interesting finding was that while the temporal reference for top items varied comparisons to five years ago and to the present or current functioning were most common. So what does this all mean? Well, different items were likely to be endorsed by neuropsychologically intact older adults with different levels of cognitive concern. Going forward, it may be possible to select items based on the ability to capture concerns most associated with specific levels or subtypes of SCD.

This is an intriguing possibility and one that would have a big impact on questionnaire development and our understanding of SCD. Another important take home message is that considerations about time frames, response options and other key features of items in questionnaires are not trivial. They are really important and hopefully our results can be used to guide such decisions.

Ms. Sara Pishdadian

Thank you for that wonderful summary of your study findings. The study was part of a special issue in the Neuropsychology Journal on pitfalls and possibilities of harmonization of neuropsychological and other clinical endpoints. Can you speak to the benefits and challenges of using item response theory to examine large datasets and how this methodology can advance clinical research?

Dr. Rich Jones

Yeah, this is Rich. I’ll take that one. Item response theory provides a tool to link measurements across studies. The end goal is to make the data collected more comparable across study. With this tool, we can generate large datasets from multiple smaller datasets, smaller studies, and in so doing, we can better understand how to compare and contrast the findings of different studies.

We could also consider pooling the data from multiple studies to gain greater statistical power, a greater range of measurement to advance clinically interesting research questions. IRT is a very powerful tool with a specific use case laid out by the present research project. Challenges of this type of research, which I would call linking studies, are in finding the items that we can consider to be linking items across different studies.

This can be often challenging and requires a good bit of clinical judgment and requires a good collaboration between substantive researchers and data analysts in achieving those links. And also key to achieving that is that the data providers share their data along with high quality and detailed documentation regarding the data so that the judgments about linkages can be made.

Ms. Sara Pishdadian

Thank you for sharing that. In your study, the specificity of item phrasing was also shown to be important. So for example, the item ‘Compared to one year ago, do you have trouble remembering things?’ was identified as psychometrically poor, whereas the item ‘Compared to one year ago, do you feel your memory has declined substantially?’ was identified as psychometrically stronger. What are the critical clinical or research takeaways from these results?

Dr. Laura Rabin

That’s a great question because our findings suggest that phrasing can affect how people interpret and subsequently answer a question. Even if the content is similar, nuances in wording and response options can impact reporting of self perceived cognitive functioning. So when we are creating items and designing questionnaires, we should consider not only general content and psychometric issues, but also nuances and key features of items.

For this specific set of items, it’s possible that trouble remembering things was too vague relative to asking about substantial decline in memory. In fact, general trends we saw for psychometrically poor items were that they had ambiguous time comparisons, such as compared to how you used to be or how you were ever before, or had very broad content such as asking about memory for things that happened in childhood.

Some item stems were very lengthy and would probably have to be read over a few times by older adult respondents. Others used Likert scales with a lot of response options. For example nine and in some cases these options also ranged from negative to positive values. It was therefore not surprising to us that certain items didn’t fare well. Taken together, our results encourage the use of items with simple, straightforward language and clear time frames.

Ms. Sara Pishdadian

Thank you for that. And I think the clinicians and researchers listening to this podcast will resonate with that finding. One might question if there are significant variations in self-perceived cognitive functioning and or the endorsement of specific questionnaire items based on an individual’s ethnicity or other cultural factors. Do you see item response theory as a useful methodology in identifying such differences, or would you recommend other methods to address potential cross-cultural variability?

Dr. Rich Jones

That’s a great question. And you identified one of the other major uses of item response theory in applied research settings, which is what we call identifying differential item functioning or DIF. And differential item functioning is just as you described. It’s a difference in the performance on a particular item due to group membership, whether it’s an ethnicity or gender or anything like that.

And there’s pretty well-established procedures for using methods of item response theory for finding those differences. It is a limited method in that there are certain cases in which you will not be able to use differential item functioning to find biased items. And that’s when you’re giving a test where all the items are biased against one group or one ethnicity.

So just like we have to have a linking items in our linking study to bring data together across multiple studies, when you’re doing DIF detection, you have to have what we call anchor items, which are items that we assume are not affected by differential item functioning or potential bias when trying to test for differential item function. We call the situation when all the items are biased as constant DIF.

So it’s a useful method, but it does have its own challenges.

Dr. Milu Elbulok-Charcape

As Rich touched upon, there are certainly issues of culture, language and education and we do have to consider them because they’re important in SCD measurement. In fact, we know that recent research shows the prevalence of SCD may vary by demographic characteristics as well as race and ethnicity. Specifically among racial and ethnic groups, SCD has been lowest among Asian adults and highest among non-Hispanic American Indian or Alaska Native adults.

In a very recent large scale US based study, the prevalence of SCD was higher among individuals with less formal education across all racial and ethnic groups, and this is consistent with the higher risk for dementia among individuals with fewer years of education. We also see that blacks and Latinos with SCD are also disproportionately affected by social inequities and comorbid chronic conditions compared to white individuals with SCD.

So based on these findings, I would say that additional research is needed to better understand how race, education and related cultural and systemic factors impact reporting about cognition and to develop questionnaires that are appropriate for diverse groups. It’s also equally important to generate normative data that accounts for the impact of socio cultural factors on self-reported cognition. And this will require a lot of work, both locally and globally, but hopefully it will enhance the accuracy of measurement of self-reported cognition in diverse groups.

Ms. Sara Pishdadian

Thank you both for that. And the research definitely continues on. Given past research on the role of worry and memory satisfaction, can you speak to the decision to focus your analysis on the items you did in terms of the cognitive domains and specifically memory ability and change coming out, as most of your items did?

Dr. Laura Rabin

Yeah. So there are a few interesting points that you just raised. The first is that one of the top items that emerged in the study relates to worrying about memory. And this is consistent with much research out of Frank Jessen’s lab and others showing that older adults who express concern about their perceived cognitive difficulties have an increased risk of future dementia.

In fact, concern or worry associated with SCD is one of the so-called SCD plus features or characteristics of SCD for which there is a particular risk of future cognitive decline. For this reason, we feel it makes sense to include an item that taps into worry about cognitive changes on questionnaires. Something else to consider is that we used a single factor model and did not have separate models for each cognitive domain.

So the analytic model included items from various cognitive domains, and top items did not relate exclusively to memory. Also, when older adults report about their cognitive abilities, they may use the term memory, but may in fact be referring to problems with language executive function or cognition more broadly. So another take home message is that questionnaires should include items that assess multiple cognitive domains.

With respect to the other part of your question about ability and change, it turns out that across questionnaires, roughly 60% of items ask about current ability or impairment, while 40% captured change or decline. The question is, which is preferable? Our results showed that roughly two thirds of top items assessed intra-individual change and this is broadly consistent with how SCD was originally conceptualized.

Ultimately, it might not need to be one or the other. For example, the Cognitive Change Index, a questionnaire from Andy Saykin’s lab, asks respondents to report on how they are functioning now – so current ability – compared to five years ago. Assuming respondents are able to hold both current ability and change in mind when responding to CCI items, this might be a viable way forward.

Ms. Sara Pishdadian

Thank you for highlighting some of the nuances of the overlap in the cognitive domains. Getting to designing questionnaires and item selection: If you were to devise a new SCD questionnaire based on your findings, what would you recommend?

Dr. Laura Rabin

So we’ve given a lot of thought to this critical next step of devising a new questionnaire. And what we know is that self-perceived cognitive functioning is a complex construct with no single cognitive domain, time referent or even type of scale ideal for its measurement in neuropsychologically intact older adults. Also, even though we were able to identify psychometrically strong items, this does not mean that if taken together, these items would magically form a reliable and valid questionnaire.

Items with psychometric support in our study require independent validation. With these caveats in mind, these are our next steps. Right now, we’re working to validate the self-perceived cognitive functioning factor scores derived from the IRT analysis against relevant outcomes such as clinical progression. This is being overseen by Sietske Sikkes and is one of the great things about harmonization. That is, we can use our results to create a new scale and also to combine existing data from different studies to learn more about the phenomenon of SCD and how SCD relates to important outcomes.

Next, we want to use the current item bank for the construction of targeted questionnaires. Before questionnaire development, we plan to carry out focus groups with older adults to learn how they are understanding items in relation to key features of those items, such as the content and complexity of the wording, time frames being referenced and number and types of response options.

Also, we recently surveyed experts about their thoughts about the optimal way to assess self perceived cognitive functioning, and ultimately we hope to combine the information garnered from the focus groups, expert opinion and this psychometric analysis to construct a new questionnaire that we would then validate and disseminate broadly.

Dr. Milu Elbulok-Charcape

Yeah, and I also jump in to note that an important consideration in this process is to adapt the questionnaire across languages and cultures. And there should be an understanding, the subtleties and target languages and cultural differences most definitely lead participants to interpret and respond differently to items. This may account for recent findings which show that greater SCD reporting among Latinos relative to non-Hispanic whites and notably significant heterogeneity in rates of SCD among Latino subgroups in the United States.

So based on what we’ve learned from the item analysis study, in our future work, we plan to carry out focus groups and cognitive interviewing with diverse community dwelling older adults. This will help us make informed decisions about item content and questionnaire structure in light of how SCD is expressed in these specific cultural subgroups.

Ms. Sara Pishdadian

Well, thank you very much for discussing your study and sharing your expertise in this important area of research. I’ll now turn it over to our host, Dr. Sperling.

Dr. Scott Sperling

And I’d also like to thank you for this really fantastic line of research. It’s really exciting both as a clinician and a researcher to see what you’re doing here using these modern statistical methods and really moving the science forward. We talk a lot about innovation, big data and cultural neuropsychology and finding ways to do that and to do it in one fell swoop is really what you’ve been able to accomplish here.

And I think there’s a lot of very exciting ramifications, again, both on the clinical and on the clinical research side for the work that you’re doing. So on behalf of the Society for Clinical Neuropsychology and the Journal of Neuropsychology, I just want to extend my appreciation for all the work that you’ve done and for your time here today, and as well to you, Sara, for your leadership as a student, who I know also has her heart in this kind of research as well here. So thank you all again for your time and expertise. It’s well appreciated, and take care.

Ms. Sara Pishdadian

Thank you for having us.

Dr. Rich Jones

Thank you very much.