Cognitive functioning as an outcome measure in therapy  

Tracking client progress with outcome measures is a well-established precept of high-quality mental healthcare. Mental health outcome measures are tools that evaluate changes in mental health by capturing metrics across multiple areas of client functioning, symptoms, and treatment experiences at baseline, during and after treatment. Cognitive health is rarely used as an outcome measure. 

Measuring clinical outcomes offers significant benefits to therapists and clients alike. The use of outcome measures can guide treatment decisions, pinpoint the need for additional professional education and training, and help service users recognise their own improvement. 

The following content is based on currently available research. The authors have no vested interest with any organisation mentioned in this article. The authors utilise tests and batteries mentioned in the article.

Introduction

Self-report measures 

There are hundreds of well validated and reliable self-reported clinical measures related to changes in clients’ perception of their inner world, and interactions with others and the world at large. These measures are generally associated with specific medical diagnoses. For instance, the Patient Health Questionnaire 9 (PHQ-9) is used to measure depression symptoms, the Generalized Anxiety Disorder 7 (GAD-7) for anxiety, or the Post-traumatic Stress Disorder Checklist (PCL-5) for post-traumatic stress disorder (PTSD).  

Within the context of clinical practice, self-report is favoured as it gives a quantifiable insight into how clients actually feel. Clients’ perceptions of their progress are essential during therapy. There are however some limitations associated with the sole use of self-reported measures in determining progress in therapy. It has been shown that self-report and clinician-rated measures can contradict each other (Cujipers et al., 2010), which might raise questions about individuals’ self-awareness and expectations of change over time. Personal characteristics can also significantly impact self-reported outcomes (Gross & Levenson, 1993). For example, high levels of self-criticism are associated with poorer outcomes when using self-report measures (Low et al., 2020). There is also evidence proposing clients’ tendency to present in agreeable terms with their therapists which can lead to exaggerating improvement (DeVylder & Hilimire, 2015). Furthermore, there are inevitable mental state fluctuations, during the course of the day or week, that are likely to have an impact at the time of completing a specific outcome measure.   

Physiological measures 

Physiological responses, such as heart rate, have also been used to determine changes before and after treatment for decades (Boudewyns & Hyer, 1990; Rhudy et al., 2010). However, measuring physiological responses can be intrusive in everyday clinical practice and may affect the therapeutic alliance and, subsequently influence the therapeutic outcome (Baier, Kline, & Feeny, 2020).  

Neuroscience advancements 

Recent advancements in neuroscience, epigenetics and methods used to measure brain activity and its function have led to a growing body of evidence shedding light onto the complex and intrinsic relationship between brain structure, cognitive function, emotional health and overall wellbeing. We can find evidence using neuroimaging techniques, such as functional magnetic resonance imaging (fMRI), magnetic resonance imaging (MRI), positron emission tomography (PET) or computerized tomography (CT) scans to determine changes in brain activity and structure during the course of psychological therapy within research settings (Beutel et al., 2010). 

Available evidence has identified numerous interacting factors that can disrupt the cerebral and cellular networks modulating cognition. A decrease in hippocampal volume has been observed in individuals with a history of depression (Sheline et al., 1999) and in women with histories of childhood abuse and current PTSD (Bremner et al., 2003). Furthermore, Bremner et al. (1995) found that individuals with PTSD scored lower on verbal memory compared to matched controls and this was associated with smaller volume of the hippocampus. O’Brien et al. (2015) found a relationship between hippocampal volume and performance on tests of memory such as the forward digit span and spatial span tasks. 

Brain neuroplasticity, including changes in hippocampal volume may be induced by treatments such as psychotropic medication, psychedelics, and psychological therapy. Malykhin et al. (2010) found that individuals who were being medicated with anti-depressants for major depressive disorder (MDD) had a larger hippocampus compared to individuals with MDD, who were not being medicated. De Vos, Mason and Kuypers (2021) findings indicated that psychedelics induce molecular and cellular adaptations related to neuroplasticity. Bossini et al. (2011) found both an increase in hippocampal volume and improvement in PTSD symptoms following treatment with Eye Movement Desensitisation and Reprocessing (EMDR). Later, Bossini et al. (2017) found that individuals treated for post-traumatic stress disorder using EMDR had significant changes in grey matter volume, post-intervention. Moreover, Porto et al. (2009) found that the neural circuits which regulate negative emotions and fear extinction were altered in those who responded to CBT treatment.  

These exciting findings are still working their way through to clinical practice. In the future it may be possible to adjust interventions to consider such findings about individuals’ brain structures and cognitive health using some version of such technologies. For now, the use of such techniques requires a significant level of expertise. They are also time consuming, expensive and intrusive, which limit their applicability within everyday clinical practice.  

Neuropsychology has developed over the last century as a way to integrate psychological observations of behaviour with neurological observations of the nervous system. Neuropsychological tests were originally designed to measure impairment in areas of cognition including attention, visual and verbal memory, learning, and mental flexibility known to be linked to a particular brain structure or pathway. More recently, interest has also turned to understanding the complex relationship between cognition and a range of mental health symptomatology. Desai and colleagues used the PHQ-9 and paired-associate learning, verbal reasoning, spatial working memory, and digit span tests to find that depressive symptoms negatively predict learning, working memory and verbal reasoning and may be either a risk factor or prodrome for cognitive decline. They also found that decline in attention predicts depressive symptoms (Desai et al., 2020). 

This would suggest that a client presenting to therapy with depression, who is struggling to concentrate and who has reduced capacity to hold verbal information, may find limitations in their ability to benefit from therapy. In such cases, due care should be exercised in planning individually tailored sessions to facilitate the client’s learning and assimilation of new information, and in this way, maximise the benefit they can draw from therapy.  

For individuals with such difficulties, simple adaptations can be implemented during therapy to promote learning and processing of new information.  This might include, presenting new information such as psychoeducation in a clearly structured and time-limited manner; ensuring the use of simple language in short sentences and avoiding the use of complex or technical terminology.  In turn, this would be likely to translate into greater benefit for the client and their ability to engage in the process of therapy, whatever the chosen therapeutic approach.   

Measuring cognitive function 

Test batteries have been devised to measure global cognitive impairment and are mostly used to assess cognition in neurological disorders (e.g., brain injury, epilepsy, or stroke) and progressive conditions like dementias. Some of these batteries include the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) and the Wechsler Adult Intelligence Scale Fourth Edition (WAIS-IV). These tests have been typically administered individually by an expert examiner in a neutral and low stimuli environment. Many of these tests can be costly, as is the training required to be able to administer, score and interpret them.  

Brief and economical pen and paper screening tests that are easier to use and do not require significant training have also been developed. For instance, the Mini-Mental State Examination (MMSE) widely used within memory clinic services, or the Montreal Cognitive Assessment (MoCA) validated to detect mild cognitive impairment (MCI) often observed after a blow to the head. These tools have been designed to measure gross cognitive impairment and have not been validated to detect subtle cognitive changes in the brain (Molloy & Standish, 1997). Despite this, studies using such cognitive measures have yielded encouraging results. For instance, Bernhardt et al. (2021) found improvements in depression and cognition in a cohort with major depressive disorder (MDD) using the Hamilton Depression Rating Scale (HDRS), the Beck Depression Inventory (BDI) and the Cognitive Basic Assessment (COGBAT) as outcome measures.  He et al. (2019) used the MMSE together with tests measuring executive functioning, episodic memory and processing speed. They investigated the effectiveness of cognitive behavioural therapy (CBT) for improving both cognitive dysfunction and depressive symptoms associated with depression; and found improvements in cognition following CBT that remained a year post-treatment. 

Cognitive impairment is common in individuals with MDD, it affects domains such as executive function, memory and attention. Knight and Baune (2018) suggested that cognitive deficits persist even when symptoms of depression subside, and that cognition should also be a target for treatment as cognitive deficits can affect everyday functioning and therefore quality of life. The authors suggest that tools such as the THINC-it tool (McIntyre et al., 2017), which consists of objective measures of cognition alongside self-report scales such as Perceived Deficits Questionnaire (PDQ-5; Lam et al., 2018), could be used to screen for cognitive impairment. Other scales which have been developed for such purposes include the Screen for Cognitive Impairment in Psychiatry (SCIP; Tourjman et al.,2018) or the Cognitive Complaints in Bipolar Disorder Rating Assessment (COBRA; Rosa et al., 2013), a questionnaire which measures concerns of cognition in individuals with bipolar disorder. Both measures have been tested with individuals with MDD.  Moreover, the authors highlighted treatment options such as cognitive remediation programmes which involve cognitive training. They used the Trail Making Test (TMT) and the WAIS as outcome measures and found improvements in results following their suggested programme.  

Nijdam et al. (2013) compared cognitive deficits in individuals with PTSD and those with PTSD and MDD comorbidity. They used the California Verbal Learning Test (CVLT) to measure verbal learning and memory, Paragraph Recall Subtest of Rivermead Behavioural Memory Test (RBMT) for verbal learning and memory, the TMT to measure processing, planning and flexibility, and the Stroop Test for selective attention and inhibition. The results showed that individuals with PTSD and comorbid MDD performed worse on the encoding section of the CVLT compared to individuals with only PTSD, highlighting the intrinsic relationship between cognition and affect as well as their impact on functioning and wellbeing. 

Later, Nijdam et al. (2018) went on to study cognitive changes in individuals with PTSD. They measured cognition before and after trauma-focused therapy.  Using the same outcome measures, they showed small to medium sized improvements in cognition following treatment. They also found that improved PTSD symptom severity was associated with better outcomes on cognition post-treatment. 

Reliable and validated computer-based cognitive test batteries have also been developed to measure cognition. One such example is the Cambridge Neuropsychological Test Automated Battery (CANTAB – Sahakian et al., 1988). CANTAB is an online test of working memory, learning and executive function; visual, verbal and episodic memory; attention, information processing and reaction time; social and emotion recognition, decision making and response control. Giedraitiene and Kaubrys (2019) have demonstrated sensitivity to detecting changes in neuropsychological performance in these areas, whilst Fried et al. (2021) have shown CANTAB’s high validity and reliability in measuring cognition. CANTAB has a time advantage over the WAIS (Wechsler, 1955), which can take considerably longer to be completed, and can therefore, be more susceptible to interference from factors such as cognitive fatigue.  

The Brain Health Assessment (BHA; Troyer et al., 2014) is a 20-minute online assessment of memory and attention. The BHA has been found to be a reliable, valid, and accurate measure of memory and attention for both healthy and cognitively impaired individuals (Paterson et al., 2022; Troyer et al., 2014). However, it is limited in what it can measure (i.e., attention and memory). 

Dr Owen and his team at the Cambridge Brain Sciences (CBS/CREYOS Health) have developed a battery of gamified, validated tests to measure cognitive health. This assessment battery reliably measures various aspects of memory, attention, executive function, response inhibition, and reasoning using gamified tasks. There are 12 tasks to choose from, and the total completion time varies from around three minutes to 40 minutes depending on the chosen tasks. The battery has been compared to the revised WAIS. Brenkel et al. (2017) suggested that the Creyos battery provides more information on cognitive function than the MoCA (Nasreddine et al., 2005). In addition, Sternin, Burns and Owen (2019) highlighted the advantages of an online cognitive battery over pen-and-paper tests. Furthermore, gamified tasks are highly engaging (Cheong, Cheong, & Filippou, 2013) with the added benefit that clients can carry them out at a suitable time, in a comfortable environment of their choice, and online on their own device. Accordingly, these batteries have been increasingly used by clinicians to measure objective outcomes during psychological therapy.  

Cognitive measures discussed in the body of this article can also be found in Table 1. 

COGNITIVE TEST/BATTERY COGNITIVE DOMAIN 
BHA Memory, and attention
CANTAB Memory, learning, attention, processing speed and reaction time, social and emotional recognition, decision making, response control, visuospatial function and executive functioning
COBRA Working and episodic memory and perceptual speed. 
cogbat Attention, memory, executive functioning, and information processing speed
CBS/Creyos health Verbal and spatial short-term memory, response inhibition, attention, verbal reasoning, visuospatial working memory, deductive reasoning, episodic memory, visuospatial processing, mental rotation, and planning
CVLT Memory, executive functioning, attention, language, visuospatial skills, and orientation
MMSE Orientation, repetition, verbal recall, attention, calculation, language, and visual construction
MoCa Memory, executive functioning, attention, language, visuospatial skills, and orientation
Rbans Immediate and delayed memory, visuospatial/constructional, language, and attention
RBMT Memory skills related to everyday situations
SCIP Executive functioning, and non-verbal skills
THINC-IT Attention, working memory, information processing speed, and performing function
TMT Visual search speed, scanning, speed of processing, mental flexibility, and executive functioning
WAIS-IV Intellectual functioning, verbal, analogical, sequential and quantitative reasoning, working memory and psychomotor processing speed
Table 1. Cognitive measures and the corresponding cognitive domains they measure

EMDR can improve cognitive function 

The illustrations presented in figures 2 and 3 show excerpts of the report that is generated using the Creyos battery. In this case, the individual completing the tests was a 27-year-old female with a long-standing history of depression. The assessment took place 12 months after she had been involved in a road traffic accident.  

At the time of the assessment, she reported that her elevated levels of anxiety, hypervigilance, flashbacks, and underlying depression were most uncomfortable. However, her scores on the self-reported measures did not seem to reflect her verbal reports during interview. Her cognitive health report indicated mild deficits in her ability to sustain and manipulate non-verbal information. This knowledge helped in monitoring changes in perception around processing images during a combined programme of CBT/EMDR over a period of four months. 

Consistent with the initial results, the self-reported measures did not indicate a significant shift following treatment, although, trauma related symptoms decreased as assessed on the PCL-5 after 12 sessions (see Table 2).  Scores on visuospatial working memory showed a progressive improvement during the course of therapy (see Figure 2). 

Before Measures After Measures 
PHQ-933
GAD-743
PCL-5123
Table 2 Self-reported measures administered before and after intervention
Figure 1. Results of visuospatial working memory test before treatment  
Figure 2. Results of visuospatial working memory test after treatment  

These results suggest that there was a meaningful improvement on aspects of cognition related to visuospatial working memory during the course of the intervention. Further research would be valuable in determining the generalisability of these results. 

Conclusion  

Current evidence highlights the complex associations between mental and physical health, behaviour patterns and cognitive health. Technological advancements exist that facilitate access to cognitive health, but cognition is rarely measured as part of the process of psychological therapy. This is despite the fact, that cognitive diversity in our society has been well established for years now. Such evidence provides an opportunity to improve the methods that we use to determine change and effectiveness of treatments in our clinical practice. Furthermore, it provides valuable information to tailor our therapy approach to fit clients’ needs, given that we are expecting clients not only to learn new information, but to remember it and apply it on a regular basis in order to benefit from psychological therapy.  

It seems that there is value in using cognitive measures, not only to shape the approach during psychological therapy but also to measure outcome.  Research has shown that cognitive deficits can persist after the remission of symptoms associated with mental health conditions (Gorwood et al., 2008). This suggests that although we might see a significant improvement after therapy in self-reported measures, there may still be a considerable risk of relapse. Changes in cognitive function are intrinsically related to changes in brain structure and activity, and, in turn, to mental health and everyday functioning. This has strong implications for the wellbeing of clients. Further research is needed to develop our understanding about the complexities of these connections. 

Dr Lopez is a Chartered Clinical Psychologist, Associate Fellow of the British Psychological Society and EMDR Accredited Practitioner registered with the Health and Care Professions Council.

References

Baier, A. L., Kline, A. C., & Feeny, N. C. (2020). Therapeutic alliance as a mediator of change: A systematic review and evaluation of research. Clinical Psychology Review, 82, 101921.   

Bernhardt, M., Schwert, C., Aschenbrenner, S., Weisbrod, M., & Schröder, A. (2021). Longitudinal changes of cognitive deficits and treatment outcome of cognitive behavioral therapy for major depression. The Journal of Nervous and Mental Disease, 209(5), 336-342.   

Bossini, L., Santarnecchi, E., Casolaro, I., Koukouna, D., Caterini, C., Cecchini, F., Fortini, V., Vatti, G., Marino, D., Fernandez, I., Rossi, A., & Fagiolini, A. (2017). Morphovolumetric changes after EMDR treatment in drug-naïve PTSD patients. Rivista di psichiatria, 52(1), 24-31.   

Bossini, L., Tavanti, M., Calossi, S., Polizzotto, N. R., Vatti, G., Marino, D., & Castrogiovanni, P. (2011). EMDR treatment for posttraumatic stress disorder, with focus on hippocampal volumes: A pilot study. The Journal of Neuropsychiatry and Clinical Neurosciences, 23(2), E1-E2. 

Bremner, J. D., Randall, P., Scott, T. M., Bronen, R. A., Seibyl, J. P., Southwick, S. M., Delaney, R. C., McCarthy, G., Charney, D. S., & Innis, R. B. (1995). MRI-based measurement of hippocampal volume in patients with combat-related posttraumatic stress disorder. The American Journal of Psychiatry, 152(7), 973–981. https://doi.org/10.1176/ajp.152.7.973 

Bremner, J. D., Vythilingam, M., Vermetten, E., Southwick, S. M., McGlashan, T., Nazeer, A., Khan, S., Vaccarino, L. V., Soufer, R., Garg, P. K., Ng, C. K., Staib, L. H., Duncan, J. S., & Charney, D. S. (2003). MRI and PET study of deficits in hippocampal structure and function in women with childhood sexual abuse and posttraumatic stress disorder. The American Journal of Psychiatry, 160(5), 924–932. https://doi.org/10.1176/appi.ajp.160.5.924 

Brenkel, M., Shulman, K., Hazan, E., Herrmann, N., & Owen, A.M. (2017). Assessing capacity in the elderly: Comparing the MoCA with a novel computerized battery of executive function. Dementia and Geriatric Cognitive Disorders Extra, 7, 249-256.  

Beutel, M. E., Stark, R., Pan, H., Silbersweig, D., & Dietrich, S. (2010). Changes of brain activation pre-post short-term psychodynamic inpatient psychotherapy: An fMRI study of panic disorder patients. Psychiatry Research: Neuroimaging, 184(2), 96-104.  

Boudewyns, P. A., & Hyer, L. (1990). Physiological response to combat memories and preliminary treatment outcome in Vietnam veteran PTSD patients treated with direct therapeutic exposure. Behavior Therapy, 21(1), 63-87.  

Brenkel, M., Shulman, K., Hazan, E., Herrmann, N., & Owen, A.M. (2017). Assessing capacity in the elderly: comparing the MoCA with a novel computerized battery of executive function. Dementia and Geriatric Cognitive Disorders Extra, 7, 249-256.  

Cheong, C., Cheong, F., & Filippou, J. (2013). Quick quiz: A gamified approach for enhancing learning. Pacific Asia Conference on Information Systems.   

Cujipers et al., 2010 – Cuijpers, P., Li, J., Hofmann, S. G., & Andersson, G. (2010). Self-reported versus clinician-rated symptoms of depression as outcome measures in psychotherapy research on depression: a meta-analysis. Clinical Psychology Review, 30(6), 768-778. 

Desai, R., Charlesworth, G. M., Brooker, H. J., Potts, H. W., Corbett, A., Aarsland, D., & Ballard, C. G. (2020). Temporal relationship between depressive symptoms and cognition in mid and late life: A longitudinal cohort study. Journal of the American Medical Directors Association, 21(8), 1108-1113.  

De Vos C. M. H., Mason N. L., & Kuypers K. P. C. (2021). Psychedelics and neuroplasticity: A systematic review unravelling the biological underpinnings of psychedelics.  Frontiers in Psychiatry, 12. https://www.frontiersin.org/articles/10.3389/fpsyt.2021.724606 

DeVylder, J. E., & Hilimire, M. R. (2015). Screening for psychotic experiences: Social desirability biases in a non‐clinical sample. Early Intervention in Psychiatry, 9(4), 331-334.   

Folstein, M., & McHugh, P. (1975). Mini mental state: A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12, 189–198. 

Fried, R., DiSalvo, M., Kelberman, C., & Biederman, J. (2021). Can the CANTAB identify adults with attention-deficit/hyperactivity disorder? A controlled study. Applied Neuropsychology: Adult, 28(3), 318-327.    

Giedraitiene, N., & Kaubrys, G. (2019). Distinctive pattern of cognitive disorders during multiple sclerosis relapse and recovery based on computerized CANTAB tests. Frontiers in Neurology, 10, 572. 

Gorwood, P., Corruble, E., Falissard, B., & Goodwin D. (2008). Toxic effects of depression on brain function: Impairment of delayed recall and the cumulative length of depressive disorder in a large sample of depressed outpatients. American Journal of Psychiatry, 165(6), 731-739.  

Gross, J. J., & Levenson, R. W. (1993). Emotional suppression: Physiology, self-report, and expressive behavior. Journal of Personality and Social Psychology, 64(6), 970.    

He, H. L., Zhang, M., Gu, C. Z., Xue, R. R., Liu, H. X., Gao, C. F., & Duan, H. F. (2019). Effect of cognitive behavioral therapy on improving the cognitive function in major and minor depression. The Journal of Nervous and Mental Disease, 207(4), 232-238.   

Knight, Matthew J., Baune, Bernhard T. Cognitive dysfunction in major depressive disorder. Current Opinion in Psychiatry 31(1): p 26-31, January 2018. | DOI: 10.1097/YCO.0000000000000378 

Lam R. W., Lamy F. X., Danchenko N., Yarlas, A., White, M. K., Rive, B., & Saragoussi, D. (2018). Psychometric validation of the Perceived Deficits Questionnaire-Depression (PDQ-D) instrument in US and UK respondents with major depressive disorder. Neuropsychiatr Dis Treat, 14,2861–2877. 

Loew, C. A., Schauenburg, H., & Dinger, U. (2020). Self-criticism and psychotherapy outcome: A systematic review and meta-analysis. Clinical psychology review75, 101808. 

Li, M., & Caeyenberghs, K. (2018). Longitudinal assessment of chemotherapy-induced changes in brain and cognitive functioning: A systematic review. Neuroscience and Biobehavioral Reviews, 92, 304-317.    

Malykhin, N. V., Carter, R., Seres, P., & Coupland, N. J. (2010). Structural changes in the hippocampus in major depressive disorder: Contributions of disease and treatment. Journal of Psychiatry and Neuroscience, 35(5), 337-343. 

McIntyre, R. S., Best, M. W., Bowie, C. R., Carmona, N. E., Cha, D. S., Lee, Y., Subramaniapillai, M., Mansur, R. B., Barry, H., Baune, B. T., Culpepper, L., Fossati, P., Greer, T. L., Harmer, C., Klag, E., Lam, R. W., Wittchen, H-U., & Harrison, J. (2017). The THINC-Integrated tool (THINC-It) creening assessment for cogntive dysfunction: Validation in patients with major depressive disorder. Journal of Clinical Psychiatry,78(7), 873-881. 

Molloy, D. W., & Standish, T. I. (1997). A guide to the standardized Mini-Mental State Examination. International Psychogeriatrics, 9(S1), 87-94.  

Nasreddine, Z. S., Phillips, N. A., Bédirian, V., Charbonneau, S., Whitehead, V., Collin, I., Cummings, J. L., Chertkow, H. (2005). The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. J Am Geriatr Soc,53(4), 695-9. 

Nijdam et al. (2013) –Mirjam J. Nijdam, Berthold P. R. Gersons & Miranda Olff (2013) The role of major depression in neurocognitive functioning in patients with posttraumatic stress disorder, European Journal of Psychotraumatology, 4(1), DOI: 10.3402/ejpt.v4i0.19979 

Nijdam et al. (2018) – Mirjam J. NijdamIrene J. M. MartensJohannes B. ReitsmaBerthold P. R. GersonsMiranda Olff (2018) Neurocognitive functioning over the course of trauma-focused psychotherapy for PTSD: Changes in verbal memory and executive functioning. Br J Clin Psychol,  57(4), 436-452, DOI: 10.1111/bjc.12183 

O’Brien et al. (2015) – O’Brien, J.T., Lloyd A, McKeith, I, Gholkar, A. and Ferrier N. (2015 – Published online; 2004 First published) American Journal of Psychiatry, 61(11), https://ajp.psychiatryonline.org/doi/10.1176/appi.ajp.161.11.2081 

Paterson, T. S., Sivajohan, B., Gardner, S., Binns, M. A., Stokes, K. A., Freedman, M., Levine, B., & Troyer, A. K. (2022). Accuracy of a self-administered online cognitive assessment in detecting amnestic mild cognitive impairment. The Journals of Gerontology: Series B, 77(2), 341-350. 

Porto, P. R., Oliveira, L., Mari, J., Volchan, E., Figueira, I., & Ventura, P. (2009). Does cognitive behavioral therapy change the brain? A systematic review of neuroimaging in anxiety disorders. The Journal of Neuropsychiatry and Clinical Neurosciences, 21(2), 114-125.    

Randolph, C., Tierney, M. C., Mohr, E., & Chase, T. N. (1998). The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS): Preliminary clinical validity. Journal of Clinical and Experimental Neuropsychology, 20(3), 310–319.  

Rhudy, J. L., Davis, J. L., Williams, A. E., McCabe, K. M., Bartley, E. J., Byrd, P. M., & Pruiksma, K. E. (2010). Cognitive‐behavioral treatment for chronic nightmares in trauma‐exposed persons: Assessing physiological reactions to nightmare‐related fear. Journal of Clinical Psychology, 66(4), 365-382.  

Rosa, A. R., Mercadé, C., Sánchez-Moreno, J., Solé, B., Del Mar Bonnin, C., Torrent, C., Grande, I., Popovic, D., Salamero, M., Kapczinski, F., Vieta, E., & Martinez-Aran, A. (2013). Validity and reliability of a rating scale on subjective cognitive deficits in bipolar disorder (COBRA). J Affect Disord., 150(1),29–36. 

Rousseau, D., & Hassett, G. (2018, August). The Effectiveness of EMDR. Sites.bu.edu. https://sites.bu.edu/daniellerousseau/2018/08/11/the-effectiveness-of-emdr/   

Sahakian, B. J., Morris, R. G., Evenden, J. L., Heald, A., Levy, R., Philpot, M., & Robbins, T. W. (1988). A comparative study of visuospatial memory and learning in Alzheimer-type dementia and Parkinson’s disease. Brain, 111(3), 695–718.  

Santarnecchi, E., Bossini, L., Vatti, G., Fagiolini, A., La Porta, P., Di Lorenzo, G., & Rossi, A. (2019). Psychological and brain connectivity changes following trauma-focused CBT and EMDR treatment in single-episode PTSD patients. Frontiers in Psychology, 10, 129.   

Sheline, Y. I., Sanghavi, M., Mintun, M. A., & Gado, M. H. (1999). Depression duration but not age predicts hippocampal volume loss in medically healthy women with recurrent major depression. Journal of Neuroscience, 19(12), 5034-5043. 

Sternin, A., Burns, A., & Owen, A. M. (2019). Thirty-five years of computerized cognitive assessment of aging—where are we now? Diagnostics, 9(3), 114.  

Tourjman, S. V., Juster, R-P., Purdon, S., Stip, E., Kouassi, E., Potvin, S. (2018). The screen for cognitive impairment in psychiatry (SCIP) is associated with disease severity and cognitive complaints in major depression. Int J Psychiatry Clin Pract., 23(1), 49–56. 

Troyer, A. K., Rowe, G., Murphy, K. J., Levine, B., Leach, L., & Hasher, L. (2014). Development and evaluation of a self-administered on-line test of memory and attention for middle-aged and older adults. Frontiers in Aging Neuroscience, 6, 335.   

Wechsler, D. (1955). Manual for the Wechsler adult intelligence scale.  New York: Psychological Corporation.