IS ALL THE EVIDENCE IN?

 Linda Watts Jackim

A range of popular treatments subsists despite lack of science base. 

Is that damaging?

Evidence-based treatment (EBT) can be defined as clinical decision-making that integrates both individual clinical expertise and the best external evidence, writes Francis Chesley, M.D., director of the Office of Research, Review, Education and Policy at the Agency for Healthcare Research and Quality (AHRQ), the federal agency charged with funding and promoting evidence-based research. But there is most certainly some dissonance in the behavioral health community about what should be driving clinicians’ treatment decisions —the research evidence or the practitioner’s judgment in each individual situation. And the debate shows no sign of subsiding.

“Research comes first,” says Dennis P. Morrison, Ph. D., without hesitation. “There was a time when we didn’t know what worked and what didn’t and there was no choice other than the clinician’s preference. But now we have data that indicate that some treatments are effective and others are not.” Morrison is chief executive of the Center for Behavioral Health in Bloomington, Ind.

But William A. Anthony, Ph. D., executive director of the Boston University Center for Psychiatric Rehabilitation, says that several different factors should influence treatment decisions. “The system should be driven by patient preference based on the practitioner’s presentation of the evidence, or their own analysis of the evidence, then what drives the decision is what the patient is comfortable doing. So there are three things: patient preference, your clinical judgment and the evidence base.” Anthony adds that too often, the available evidence is not sufficient. “For example, the problem could be we’re not helping enough people to recover. The solution could be to use a particular evidence-based program. But maybe that evidence-based program has no evidence around helping people to recover, so then we’re using the evidence base as a solution to a problem for which it is not a solution. “I think the concept that we need to bring evidence into our decision-making is crucial,” Anthony says. “What I have been critical of is the lack of meaningful evidence we currently have in the field of behavioral health.”

By “meaningful evidence” Anthony says he means evidence about helping clients find meaningful work, decent housing and a feeling of empowerment.

 Scott Lilienfeld, Ph. D., associate professor of psychology at Emory University, says the only alternative to basing treatment decisions on controlled research is to use clinicians’ own subjective impressions of what works. “We know those are fallible. Research should always come first,” he says. Lilienfeld is lead editor of Science and Pseudoscience in Clinical Psychology (Guilford Publications, 2003).

Controversial treatments continue

The fallibility that concerns Lilienfeld and others reveals itself in the continued use of several controversial practices in the behavioral health field. One target of criticism is Critical Incident Stress Debriefing (CISD), which some say is used despite a lack of an evidence base for it. “Most studies have shown that treatment is not particularly effective, and in a couple of studies it’s actually been shown to be harmful,” says Lilienfeld. “That would be a very bad choice to use given what we know about the research literature.”

Denise Sloan, Ph. D., assistant professor of psychology at Temple University, agrees. “Empirical evidence indicates that [CISD] is not effective. In fact, it can make symptoms worse. It can be detrimental.” Sloan is secretary of the American Psychological Association’s (APA’s) section on Society for a Science of Clinical Psychology.

Eye Movement Desensitization and Reprocessing (EMDR) is another intervention in the controversial category, even though some research has shown it yields positive outcomes. “Some of the controversy over EMDR revolves around the claims that have been made on its behalf,” says Lilienfeld. “Post-Traumatic Stress Disorder is the one condition for which there is the most research, and it works better than doing nothing. Is there evidence that the supposedly specific ingredient of that treatment, namely the lateral eye movements, has anything to do with its effectiveness? No. The literature maintains that to the extent that EMDR works, it does so because it incorporates exposure, which is what all known effective treatments for that disorder already do.”

Lilienfeld says he has heard of people who still use psychoanalysis to treat schizophrenia, even though there is no evidence that it works. Rebirthing techniques still are used for children with early attachment problems, and client-centered therapy—non-directed therapy that evolves as the client reflects—is sometimes used for severe depression. “None of these techniques have been shown to work. And people continue to use them,” says Lilienfeld.

On the other side of the coin, people have identified several treatments for severe mental illness that they think are evidence-based, like supported employment, the [assertive community treatment] program, and family psychoeducation,” says Anthony. “But some of us are saying that the outcomes they produce are not what I would call recovery-oriented outcomes. “Reducing relapse, reducing symptomatology and getting people entry-level work are the outcomes we seem to be able to produce, and those are not bad outcomes,” he says. “But the outcomes that people with severe mental illnesses want are meaningful work, decent places to live, getting back into school, and feeling empowered and involved. Those are the type of outcomes that we have not demonstrated very well in randomized clinical trials.”

When is a treatment ‘evidence-based’?

The AHRQ’s Chesley lists the critical features of good evidence-based research as: (1) It identifies answerable questions within clinical decisions; (2) It locates the best evidence that is valid and applicable; (3) It evaluates the evidence for its validity and usefulness; (4) It estimates benefits and harms for individuals; (5) It evaluates clinical performance; and (6) It identifies gaps in the science. But is there a magic number of research studies with good outcomes required before a treatment intervention is considered “evidence-based”? “I don’t think there is any strict cutoff,” says Lilienfeld. “All I would say is that if a treatment has been shown to be effective in independent replications by independent investigators who do not have a clear-cut stake in the outcome, then the more such demonstrations one has, the more confident one can become.” There is no endpoint, says Anthony. Rather it is a process of becoming more and more evidence-based. “We need to build a further evidence base so that someone can say, ‘Well, 50 percent of the people that come to my program get the jobs they want.’

If you were coming in to a vocational program it would be nice to have that information.” Sloan says, “Another way of thinking about it would be, ‘Is there any support for this at all? Do we know if there is anything in the literature that would indicate that this would be an effective treatment?’ In that way even using case studies would be appropriate.” She believes that one criterion for declaring a treatment “evidence-based” is that it must be shown to be effective in a community setting.

Why isn’t everyone using EBT?

A major problem facing proponents of EBT is the gap between science and practice. The consensus is that this gap exists because many well-meaning clinicians are not used to the idea of doing clinical work with a database or measurement qualification attached to it. “There is a learning curve. There has to be a mind shift in the practitioner,” says Morrison. “Many clinicians were not raised professionally to think in terms of being data-oriented. They were taught to provide treatments because of some theoretical orientation, and so the problem is that we were trained to believe that we were good therapists if we did therapy the way we were trained to do therapy, so therefore we are good therapists when we do that. It’s a circular argument.”

Once the mind shift occurs, clinicians and organizations must learn how to implement EBT protocols, and that entails considerable expense. “You have to get trained in it, you have to get supervision in it, you have to put some resources toward doing it and making sure that you maintain some fidelity to the model,” says Morrison.

Another possibility is that EBT is not used more often because of a disjunction between practicing clinicians in the community and the people who are doing the research. “Many clinicians are resentful about the implication that researchers dictate to them what kind of practice they should use,” says Lilienfeld.

Sloan sees the plethora of treatment manuals and how-to reports that have been spawned by EBT as a disadvantage. “When you try to use a treatment manual it’s almost too rigid,” she says. “People become more like technicians than clinicians. Manuals don’t allow for the flexibilities you need when you deal with people.”

But Morrison says EBT protocols give you control of the process, and they give you demonstrable outcomes. At his Center for Behavioral Health, about two dozen EBT protocols have been implemented with high fidelity to the models, he says. However, he says, it sometimes happens that someone conducts research using a different process but achieving the same outcome at the same costs. As long as the process is benchmarked against a proven protocol, he says, it’s better to have two processes than one. “Take panic disorder, for example. Looking at the research literature, one measure used in Barlow’s protocol for panic disorder has an 85 percent success rate one year out,” says Morrison. “Using that as the gold standard, if you want to try a different kind of intervention, you may get a 90 percent success rate. That’s great. As long as you look at the outcomes that are possible using the evidence-based protocol as the gold standard, that’s exactly what we want to have happen. But you have to be held accountable to what’s achievable.”

Morrison does not believe EBT should be mandated. He thinks the decision for whether to adopt the known protocol or do your own research should be left to the clinician or the organization. “We get into trouble when systems of care such as large hospitals or state governments say you have to implement this treatment protocol, period.” That can kill the creative drive to find a better way to get the same or better outcomes, or to refine techniques, he adds.

There also is the problem of information dissemination, says Sloan. Clinicians may not always be up-to-date on what academics know to be empirically supported treatments. The reasons are legitimate: heavy caseloads, busy schedules, or not enough time or money to attend workshops to learn the most up-to-date EBTs.

“Psychology is kind of a different realm in terms of people practicing things that aren’t demonstrated to be effective,” says Sloan. “You would never do that in medicine. You would be sued. So, it’s kind of odd that there are clinicians would hold to treatments where they just ‘have a feeling’ it works.”

Improving the system

If clinicians are not using EBT because they don’t want to let go of the familiar, how can this be changed?

 Sloan says that academicians need to do a better job of disseminating the available information. “It is our responsibility to go to the practicing clinicians and make sure that they know about our information.” Clinicians don’t always have time to read journal articles, she points out.

Anthony agrees that education is crucial. “I think there have been some misconceptions, like, ‘We’ve collected all the evidence we need,’ or, ‘Evidence drives the system,’ and so forth. These misconceptions need to be cleared up.” Better training is imperative, says Sloan. “We really need to emphasize the importance of the science of clinical psychology, and if they have that training then they will carry that with them out into the field.”

Morrison adds, “It is much more important to train clinicians to be accountable to the results of their interventions, rather than the processes. After all, helping people get better is why we do this.”               

Resources

Bridging Science and Service: A Report by the National Advisory Mental Health Council’s Clinical Treatment and Services Research Workgroup. Chapter Three, “Making Connections.” National Institute of Mental Health, 2001. www.nimh.nih.gov/research/bridge.htm

Stuart, GL, Treat, TA, and Wade, WA. Effectiveness of an empirically based treatment for panic disorder delivered in a service clinic setting. Journal of Consulting and Clinical Psychology, 2000; 68(3): 506-512

This article was published in the October 2003 issue of  Behavioral Healthcare Tomorrow magazine. It is reprinted in its entirety with permission of Manisses Communications Group, Inc.  Manisses Communications Group, Inc. publishes mental health and addiction information, offering news and analysis of federal and state public policy developments, funding, cutting-edge programs and legislation in the addiction field. To learn more about the Manisses Communications Group, Inc. including how to receive a free trial subscription to Behavioral Healthcare Tomorrow contact Manisses Communications Group, Inc., 208 Governor Street, Providence, RI 02906-3246 or visit their website at http://www.manisses.com/bht/sample_issue_request.htm