EMDR Therapy Quarterly

In line with the Association’s strategy for developing the knowledge and skills of its members, Anthea Sutton and Beverly Coghlan revealed the dark art of critical analytic skills. For some participants, it was their first foray into this arena; for others, it was a gentle reminder of the things they had learned in previous periods of study. This was a practical webinar, with participants using two real-life examples of published clinical studies to test their knowledge.

It might not be immediately apparent why we need to be able to read a scientific paper critically. After all, if we find it in the Francine Shapiro Library or the EMDR UK Publications Database it must be valid, right?  Well actually, no. The EMDR Publications Database does go some way towards eliminating papers that we should be wary of, because unlike the Francine Shapiro Library, it only includes those that are peer reviewed; and it also automatically monitors for retractions, removes them and replaces them with a retraction notice. But, as the webinar participants discovered, peer review does not necessarily guarantee research quality and robustness.

If Ioannidis (2014) is to be believed, only 15% of health research is useful; the remainder is reported as “wasteful” due to issues with study design, irrelevance to stakeholders and other quality issues. Critical appraisal skills increase the probability of finding the useful 15%.

Critical appraisal ensures scientific rigour and reliability, helps identify biases and errors, and ultimately improves evidence-based decision-making. Figure 1 shows an accepted hierarchy of evidence generation.

At nearly every level, there is an appropriate appraisal checklist to assess the quality and robustness of a study, as well as its practical applicability to your work with clients.

How is a critical appraisal carried out?

A critical appraisal starts with a clinical or research-driven question or problem. From here, we can search for relevant articles, and the relevance can usually be discerned from the abstract. To ensure a thorough evaluation, an appropriate critical appraisal checklist, relevant to the study design of the chosen journal article, can be used. This will provide questions to guide us through the article and help us make sense of the research. These checklists are available from several different organisations, such as:

• Critical Appraisal Skills Programme (CASP).
• Centre for Evidence-based Medicine, Oxford.
• Cochrane Collaboration.
• The Queen Margaret University, Edinburgh, has curated a comprehensive library of these lists, which can be found here.

While these checklists may seem daunting at first, they do make the research more meaningful. With practice, working through the list and scoring the research becomes second nature, and we become better able to sniff out research that is not robust or applicable to our client population. It is important to note that when conducting systematic reviews or meta-analyses, it is essential to score the included research on a standardised checklist.

Research bias

Bias in research refers to a systematic error that can occur during the design, conduct, or interpretation of a study, leading to inaccurate conclusions. Essentially, the critical appraisal checklists are designed to identify potential bias. The higher the risk of bias, the less the results can be trusted.

Selection bias

Selection bias can be reduced by ensuring adequate randomisation of participants to interventions and that the randomisation is adequately concealed. This should ensure that study groups are similar at baseline, and baseline demographics and disease/disorder severity should be reported. Intention to treat (ITT) analyses should also be conducted. This means that the analyses should be run on every participant, whether they completed the intervention or dropped out for whatever reason.

Performance bias

Performance bias can be minimised by ensuring that the care provider and the participants are blinded to the interventions they receive. This is difficult to achieve in psychotherapy investigations, and we need to remember that the widely accepted gold standard for clinical investigations (the double-blind, randomised controlled trial) was developed to test pharmacological substances and not person-to-person psychological therapies.

Detection bias

Detection bias can be minimised, even when the actual investigation is not double-blinded, by blinding the results assessors. 

Attrition bias

Attrition bias refers to the overall tolerance of an intervention and should be reported for all non-completers, whatever the reason, as well as differential attrition between the study groups.

Funding bias

Funding bias can be minimised by declaring the funding source and any conflicts of interest of the investigators.

Statistics

Of course, we must have a working knowledge of statistics to understand that the statistical tests applied are appropriate for the data and to understand what the numbers mean.

p-values

In a nutshell, p-values help us determine the significance of the results. p-values of 0.05 or less indicate that the result is likely to be accurate, and p-values of greater than 0.05 are considered an indicator of an unreliable result. For example, a low p-value of 0.03 indicates that if the study was repeated 100 times, on three occasions the result would be due to random sampling error or chance, and on 97 occasions it would be due to a real effect being observed.

Confidence intervals

Confidence intervals (CIs) are often used in randomised controlled trials (RCT) and meta-analyses. They are used to indicate the variation of results and the confidence we can have in the results. CIs show the average/overall result plus the upper and lower ranges. We would not expect to get the same result if we conducted an experiment 100 times, but a confidence interval tells us how much variation we might expect and how confident we can be in it. This is usually expressed at 90, 95 or 98% confidence. This means that for a reliable, usable result, we ideally want to see a narrow confidence interval (so just a small amount of variation between studies) with a high percentage of confidence.

Power

In health research, it is important to ensure studies have a high likelihood of detecting meaningful findings, so power calculations should be undertaken to determine the number of participants required to show an effect.

Effect size

The effect size quantifies the difference between the intervention and control groups. Calculating the effect size helps us understand whether an intervention is clinically meaningful as well as statistically significant. It allows comparison across studies with different sample sizes or methods, and as we all know, in mental health research, even small effect sizes can be hugely clinically significant. Effect sizes are typically expressed as follows: small (0.2–0.5), medium (0.5–0.8) and large (greater than 0.8).

Putting critical analysis into practice

As a final exercise to consolidate their learning, webinar participants were asked to critique the most cited paper in the EMDR literature – Francine Shapiro’s 1989 study, which has been cited 544 times. The question posed was, “How does this paper stack up nowadays, given that it was published 36 years ago?”

We worked through the CASP RCT checklist and together concluded that because the p-value for EMD versus control was so striking (p<0.001 pre-post drop in SUD versus p>0.05 for the control group), we could take the results seriously enough to do further studies  – which is clearly what Shapiro did. But judging by our present-day standards, the risk of bias in this study is high, not least as Shapiro herself points out, because “the experimenter and author were one and the same.”

Critical appraisal skills are valuable to us not only as scientist-practitioners but also in our everyday lives. We are constantly bombarded with information about products/interventions/treatments with outlandish clinical claims but couched in a pseudo-scientific way. One only has to look to the ‘wellness industry’ for examples, and often our clients present us with such information as fact. Being able to neutrally and scientifically debunk the claims can be helpful. More importantly, though, these skills can be time-saving (given the amount of information available) and lead us to better clinical judgement.

Figure 3 provides additional resources to help you develop your critical analysis skills. The video is particularly helpful; it makes the whole process of reading (and analysing) a paper seem much less daunting.

Figure 3. Resources

A recording of the webinar can be found here.

Beverly Coghlan has worked as a clinical researcher for over 30 years.