Authors: Lance Dodes
A poor understanding of these issues—the need for randomization, the difference between correlation and causation, and the power of the compliance effect—has colored much of the research that has been conducted to date about the effectiveness of 12-step membership and attendance. Other studies have been bedeviled by inadequate analysis of the data itself, including sloppy omissions and statistical errors. Deborah Dawson of the National Institute on Alcohol Abuse and Alcoholism, Division of Biometry and Epidemiology, once lamented the lack of credible data in the study of addiction treatment: “Few, if any, studies have assessed the impact of different types of treatment on both the probability and rapidity of recovery, i.e. on person-years of dependence averted.”
5
Her principal complaint: the lack of controls in most AA studies.
Analyzing the available data about AA requires that we begin with a clear definition of success.
Success
, after all, can mean any number of things. Should one measure it in days of sobriety? Weeks without a binge episode? What if people who are making substantive progress slip and have one drink during an otherwise successful period of time: Should they “go back to zero,” as is the practice in many AA chapters? What if they stop drinking but acquire a gambling problem instead?
The question of prognosis is far easier to answer in the rest of medicine. Disease is usually a binary system: either you’ve got it or you don’t. Pneumonia: got it or you don’t. HIV: got it or you don’t. Multiple sclerosis, polio, emphysema—all of these are yes-or-no propositions. But alcoholism is not, in fact, a disease: it is a
behavior
, or perhaps a collection of behaviors. And because nobody can say for sure whether a behavior has ever been eliminated for good without a crystal ball, we must first establish a baseline definition of what success looks like in the treatment of addiction. I’ll propose this simple definition:
A treatment for alcoholism may be called successful if an individual no longer drinks in a way that is harmful in his or her life.
AA does not hew to a single company line on the question of its success rate; various accounts have quoted the organization as saying “upward of 75 percent of its members maintain abstinence.”
6
Here, again, is that key passage I cited from
Alcoholics Anonymous
:
Rarely have we seen a person fail who has thoroughly followed our path. Those who do not recover are people who cannot or will not completely give themselves to this simple program, usually men and women who are constitutionally incapable of being honest with themselves. There are such unfortunates. They are not at fault; they seem to have been born that way. They are naturally incapable of grasping and developing a manner of living which demands rigorous honesty. Their chances are less than average. There are those, too, who suffer from grave emotional and mental disorders, but many of them do recover if they have the capacity to be honest.
7
To understand what can actually be known about AA’s success rate, we must attempt a deep dive into the best data available. Let’s start with the controlled studies.
At least one early attempt to study Alcoholics Anonymous in a randomized experiment was run by J. M. Brandsma (of the College of Medicine at the University of Kentucky) and colleagues in 1980.
8
Eighty individuals, mainly court-referred, were randomized into three groups: AA-based treatment run by the investigators; a course of one-on-one RBT (rational behavioral therapy) run by lay people; and an open option for patients to choose any treatment they wished, which constituted a control group.
The investigators found “significantly more binge drinking at the 3-month follow-up” among the people assigned to the AA-oriented meetings. As the year mark approached, the researchers noted, “All of the lay-RBT clients reported drinking less during the last 3 months. This was significantly better than the AA or the control groups at the 0.005 level [meaning the finding was highly statistically significant].” The final data led the researchers to conclude: “In this analysis the AA group was five times more likely to binge than the control group and nine times more likely than the lay-RBT group. The AA group average was 2.4 binges in the last 3 months.”
It was a provocative result, but hardly definitive. After all, a good scientist could imagine any number of factors that might have confounded the numbers in this study. The nature of the “lay therapy” is never well defined, for instance, nor were any measures taken to ensure that this option was provided in a uniform way. The “choice” group is never broken out into subsets that might allow us to see which treatments they chose, if any. And, like almost all longitudinal studies, this one relied on self-reporting, which is a notoriously questionable metric.
The results, however, did mirror what was concluded in later trials involving AA. A review of all such reports between 1976 and 1989 was performed by C. D. Emrick (of the School of Medicine of the University of Colorado) and colleagues. The researchers concluded:
The effectiveness of AA as compared to other treatments for “alcoholism” has yet to be demonstrated. Reliable guidelines have not been established for predicting who among AA members will be successful. . . . Caution was raised against rigidly referring every alcohol-troubled person to AA.
9
It took until 1991 for another randomized study to be completed. This one found essentially the same results as the Brandsma study. In a paper published in the
New England Journal of Medicine
, the oldest continuously published medical journal in the world and widely considered the world’s most prestigious, D. C. Walsh and his co-researchers “randomly assigned a series of 227 workers newly identified as abusing alcohol to one of three rehabilitation regimens: compulsory inpatient treatment, compulsory attendance at AA meetings, and a choice of options. The findings were notable:
On seven measures of drinking and drug use . . . we found significant differences at several follow-up assessments. The hospital group fared best and that assigned to AA the least well; those allowed to choose a program had intermediate outcomes. Additional inpatient treatment was required significantly more often . . . by the AA group (63 percent) and the choice group (38 percent) than by subjects assigned to initial treatment in the hospital (23 percent).
10
These results led the researchers to issue a warning in their final recommendations: “An initial referral to AA alone or a choice of programs, although less costly than inpatient care, involves more risk than compulsory inpatient treatment and should be accompanied by close monitoring for signs of incipient relapse.”
All scientists are aware of the dangers of non-controlled studies, of course, but often they have no choice. Randomizing individuals and controlling carefully for outside factors is extremely expensive, far more so than running an observational study. Controlled experiments can be conducted only with small sample sizes and with the help of deep pockets. As a result, proper clinical data is maddeningly hard to come by in many questions of public health.
Yet one group exists solely to sort through the glut of studies and help caregivers tune out poorly designed or reported research: the Cochrane Collaboration, which comprises nearly thirty thousand researchers dedicated to pushing back against what medical pioneer David Sacket once called “the disastrous inadequacy of lesser evidence.”
11
The Collaboration’s mission is quite simply to focus only on studies with proper protocols and minimal bias and to assemble the strongest data from a rigorously defined set of criteria. No purely observational studies or uncontrolled studies are permitted in a
Cochrane Review
; the organization’s goal, simply put, is to vet all the science out there and tell us what can actually be verified.
In 2006, the Cochrane Collaboration undertook a characteristically careful and detailed look at studies of AA and 12-step recovery. First, the researchers recapped what had been determined to date:
[A] meta-analysis [historic analysis of previous studies] by Kownacki (1999) identified severe selection bias in the available studies, with the randomised studies yielding worse results [for AA] than non-randomised studies. This meta-analysis is weakened by the heterogeneity of patients and interventions that are pooled together. Emrick 1989 performed a narrative review of studies about characteristics of alcohol-dependent individuals who affiliate with AA and concluded that the effectiveness of AA as compared to other treatments for alcoholism was not clear and therefore needed to be demonstrated.
12
The Collaboration then identified eight high-quality, controlled, randomized studies, with 3,417 subjects in all.
13
Their conclusion was unambiguous: “No experimental studies unequivocally demonstrated the effectiveness of AA or TSF [Twelve Step Facilitation] approaches for reducing alcohol dependence or problems.”
Despite the fact that the best designed studies have all questioned AA’s effectiveness, there remains a body of academic articles that are very frequently cited by supporters of the 12-step movement. To understand the arguments of 12-step proponents, we must give these studies an open hearing as well.
In 1999, R. Fiorentine and colleagues ran a twenty-four-month longitudinal after-treatment study that “suggests the effectiveness of 12-step programs.” They concluded:
the findings suggest that weekly or more frequent 12-step participation is associated with drug and alcohol abstinence. Less-than-weekly participation is not associated with favorable drug and alcohol use outcomes, and participation in 12-step programs seems to be equally useful in maintaining abstinence from both illicit drug and alcohol use. These findings point to the wisdom of a general policy that recommends weekly or more frequent participation in a 12-step program as a useful and inexpensive aftercare resource for many clients.
14
The authors of this paper based their recommendations on a clear correlation that has appeared many times in the literature, namely that the longer people attend AA meetings, the more likely they are to experience better outcomes for sobriety. Here is how they summarized their findings:
In the 6-month period prior to the 24-month follow-up, approximately 27% of those participating in any 12-step meetings used an illicit drug compared to 44% of those not attending 12-step meetings. The results of the urinalysis support the same conclusion. About 28% of those attending any 12-step meetings tested positive for an illicit drug at the 24-month follow-up compared to 41% of those not attending 12-step meetings. Less than 4% of 12-step participants tested positive for alcohol at the 24-month follow-up compared to about 13% of nonattendees.
In other words, the incidence of drinking was roughly 60 percent higher among nonattendees than attendees at the first two measurements and far higher at the final data point, to the tune of a 300 percent improvement for the AA attenders.
It’s tempting to look at correlations like this and conclude, as many have, that AA must be responsible for this improvement. Yet Fiorentine and his colleagues themselves noted that their results were at odds with other recent studies like the Walsh study cited above and another by B. S. McCrady, who both found that “random assignment to AA or two other treatment condition groups did not reveal more-favorable drinking outcomes for AA participants.”
15
The researchers were also mindful of the compliance effect:
The findings suggest that 12-step programs may be an effective step in maintaining drug and alcohol abstinence. Unfortunately, the limitations of the design do not allow other variables, including the
motivational confound
, to be ruled out as possible influences on the drug and alcohol use outcomes of 12-step participants. (Emphasis added)
Hence their highly qualified final recommendation, which is not often cited by 12-step proponents:
More definitive answers to these questions may come from randomized trials involving 12-step programs and comparison groups of sufficient size that are followed over a relatively long posttreatment duration. . . . Randomized designs are the best method yet to disaggregate the effectiveness of treatment from other influences, including motivational differences. . . . [T]he findings indicate that both weekly and less-than-weekly 12-step participants had very high recovery motivation scores—scores that may be attributable, at least in part, to the sampling bias of the study.
Caveats such as these are standard practice in peer-reviewed science, so they should be taken only as possibilities, not as an indictment of the research as a whole. Yet the significance of these warnings cannot be overstated: anyone who understands the inherent difficulties with observational science would recognize this list of concerns as grounds to consider the results provisional until a controlled study can be mounted.
Fiorentine and his colleagues did attempt to minimize the effects of sampling bias by doing what researchers almost always do in epidemiological science: they applied multiple regression analysis (MRA), which involves developing a mathematical model to try to explain the data, and to account for and separate out all the known differences between the groups
—disaggregate
, in their language. MRA unquestionably has its uses, but it can no more overlay controls retroactively on an uncontrolled study than a camera can turn a single still image into a 360-degree panorama. In elegant understatement, Harvard Medical School professor and epidemiologist Jerry Avorn told the
New York Times
that MRA “doesn’t always work as well as we’d like it to.”
16
Indeed, what troubles many good scientists about research like the Fiorentine paper is that studying the people who
choose
to attend AA is an almost perfect recipe for generating the compliance effect error. AA members who frequently attend meetings may be demonstrating the same sort of self-care qualities that the placebo takers do. They may be, in effect, the Boy Scouts, or “eager patients,” of the addict population.
17
Nobody who has looked at this data would dispute that people who attend AA most often and stay the longest are more likely to improve than the dropouts. The question is whether AA is driving this outcome or benefiting from a correlation instead.