Autism evidence 2. Using an evidence tool to draw your own conclusions about autism interventions

 

Please read the first Blog in this series [Who do you believe? A model for understanding the development of evidence for interventions in autism] before reading this blog.

In practice, Thornicroft et al. (see model above) point out that there are three points along the evidence continuum that they call blocks – where evidence earlier in the storyline tends to get stuck. Many basic science findings are not translated into interventions at all. This has often been pointed out in the field of autism – there is a great deal of basic and experimental research on children with autism, and much less use of this research to inform new interventions. A second translational block is when research demonstrates that an intervention CAN work, but then everyone relaxes thinking that is enough to now get on and inform service delivery. Finally, even when interventions are also tested in effectiveness studies research on scaling up and delivering interventions within typical practice and service settings is often just not done.

Using an evidence tool

By this point, I hope that you can see how you could use Thornicroft et al.’s evidence tool. What it will hopefully prompt you to do is to consider what question a particular research study is asking about an intervention. The Phases of the model help to clarify the questions.

So, is someone describing to you an early study designed to generate initial data on a recently defined new intervention (Phase 1)? If so, the research design might not be very strong or sophisticated. However, the research is still likely to be important. You should be asking about the theory underlying the intervention. The researchers or practitioners should be able to articulate this clearly and point to the evidence that supports their theory. A “good idea” is simply not enough – it has to be more properly considered.

If you are being shown results from a Randomized Controlled Trial, at what phase is this study along the continuum? Is it an initial pilot trial (in which case it is not yet definitive evidence), is it testing out whether an intervention CAN work, is it testing effectiveness in more typical settings, or is it even using a RCT design to test aspects of delivery (Phase 4) for an intervention with already established efficacy and effectiveness?

In practice, you will find that the evidence for interventions does not always follow the full continuum. Sometimes, this will be because those peddling an intervention have not fully understood what “evidence” really means. Sometimes, it suits those whose livelihood relies partly on you being interested in a particular intervention to give you only a part of the story. People may not be misleading you explicitly – they are simply not presenting the whole story, or do not have a way to easily communicate that full story to you. Now YOU have a way of looking at evidence yourselves and can start to draw your own conclusions.

There are also good pragmatic reasons why the development of evidence may not follow through the full set of phases. First, this could take a very long time in total. It is often been pointed out that the journey from basic science to practice can take two decades or more. Thus, it is perfectly appropriate to take pragmatic decisions about what evidence is most needed now. Obviously, we cannot sit around waiting for 20 years to develop evidence-based interventions for children with autism. We need them now (or soon). This is not an excuse for poorly designed research. However, researchers and practitioners could be more transparent about what they are doing and why and could use the evidence tool to explain the bigger picture and, therefore, the limitations of the evidence that they currently have.

Some examples

At this point it might be helpful to show how I might use the evidence tool to talk about the nature of evidence emerging from individual intervention studies in the field of autism. I have chosen two examples that use widely accepted strong research designs (both use RCTs), but their focus is quite different.

Howlin and colleagues (2007) carried out an evaluation of PECS (the Picture Exchange Communication System). 18 classes of children with autism were allocated randomly to PECS or education as usual. Teachers and parents received a 2 day PECS training workshop, and PECS trainers then made 6 half-day consultation visits to each class over a 5 month period. Although communication initiations and use of PECS increased in the PECS intervention classes, there were no measured increases in scores on standardized language assessments for the PECS group compared to the control group. The increases in communication initiations and use of PECS also did not continue after the consultation support ended.

I think the Howlin study of PECS is firmly in Phase 3 of the evidence continuum. This is a study of the effectiveness of PECS under typical conditions, delivered by school staff after minimal training, with some supervision, in typical school settings for children with autism. Questions that we might want to check up on, looking at this study on its own, might be about the theory underlying PECS, whether there are efficacy trials already published, and how could the intervention be scaled up and included as a standard part of practice for children with autism in educational settings (whilst also delivering outcomes)?

What is clear is that a study like Howlin’s is simply a part of a story and is not the whole story.

Another recent and well-publicized study is from Dawson, Rogers and colleagues (2010) evaluating the Early Start Denver Model (ESDM). In their research, 48 young children with autism were allocated randomly to EDSM or to “treatment as usual” for young children with autism in their locality and received intervention over 2 years. The intervention was delivered by highly trained and experienced graduate staff supervised by experts. The research team also included the developers of the intervention approach. The results were positive. For example, cognitive skills and adaptive skills were improved in the ESDM group compared to the control group.

The Dawson and Rogers evaluation of EDSM for me is a Phase 2 study – it is designed to test whether ESDM can work under ideal and fairly tightly controlled circumstances. It was an efficacy trial. The intervention is also based on basic developmental research in autism, is manualized, and has been tested previously in other smaller scale studies (see Phases 0 and 1). However, it would be a mistake to suddenly suggest that everyone must start delivering ESDM as the model for pre-school services for children with autism. Effectiveness studies are needed and, more significantly, research is needed on how ESDM could be implemented as a part of a broader pre-school educational service and on a large scale.

Again, the Dawson and Rogers study is very important and a great piece of research, but it is only a part of the story.

A call to researchers and expert practitioners

I would encourage researchers and expert practitioners to use the evidence tool to summarize the evidence for particular educational and psychological interventions for children with autism. To properly critique any intervention, evaluate research evidence from basic research findings right through to any effectiveness and implementation studies. So, when you are explaining the evidence base for an intervention, see if the evidence continuum tool helps you to provide the full story.

In a later blog, I will provide a perspective on the evidence for Applied Behaviour Analysis intervention using the evidence continuum tool.

In addition, when reporting an individual study, researchers may find it helpful to refer to the evidence continuum tool to place their research question and their findings in context.