Being the ever-diligent evidence-based practitioner, a few years ago I decided to review the Cochrane database to determine what its review process had to say about the conditions I most often treat and therapies I most often use.
This experience sparked a question: What if the therapies I was offering were effective, but the traditional methodology of determining effectiveness in clinical trials was flawed in some way? Here is the result of my research into this question.
History of the RCT Paradigm
It was not uncommon in the 15th century for half of all sailors traveling around the globe to die before arriving at their destination. These men died not of battle wounds, but of an unknown disease which caused "their gums to swell, the stench of their breath to become unbearable, and the loss of all energy to even move." This condition was labeled scurvy, after the Danish word schorbect meaning ulcers in the mouth.1
An innovative thinker and surgeon by the name of Dr. James Lind devised an experiment while aboard a ship in 1747. He selected 12 ill sailors and divided them into groups of two. Isolated from the rest of the crew, the men were given the same rations, but each pair received a different scurvy treatment: cider; a few drops of a weak acid; vinegar; sea water; nutmeg and barley water; or oranges and lemons.
The two men on the citrus treatment of oranges and lemons quickly recovered and were back on their feet. The others continued to deteriorate quickly.2 The confirmation that citrus fruits provided a cure for scurvy was likely the first clinical trial ever documented.
According to Hegde,3 the history of the RCT as we know it began in agriculture, with the British statistician Ronald Fisher publishing the original works on the subject in 1925 and 1942. Fisher was interested to determine the effects of soil chemistry, weather conditions, moisture content, seed quality, fertilizer and other variables on crop growth. He began to randomize plots to isolate each variable.
Seeing the possible applications to their own fields, researchers in psychology, the social sciences and eventually medicine latched on to these methods. The first RCT published in a medical journal was by Bradford Hill in 1948 regarding the effects of streptomycin on pulmonary tuberculosis. Currently, more than 200 RCTs are being published every week.4
The RCT has long been considered the "gold standard" research paradigm in the health sciences. Since the mid 20th century, the RCT has been the accepted model of performing scientific research. According to the Centre for Evidence Based Medicine,5 at the peak of the hierarchy of evidence is the systematic review of RCTs.
Bradford Hill asserts that randomization has the capacity to ensure "neither our personal idiosyncrasies, consciously or unconsciously applied, nor our lack of judgment have entered into the construction of the two (or more) treatment groups and thus biased them in any way."7 Ronald Fisher maintained that randomization is necessary for the valid interpretation of statistical significance.6 These two claims are challenged below.
Questioning the Paradigm
There exist subgroups within the Cochrane collaboration to deal with specialty specific systematic reviews; one such subgroup is the Back & Neck Group.7 After reviewing all of the available RCT literature on the topic of back and neck therapies, the group indicated that there is no evidence of effectiveness (or lack of evidence) for the following therapies in regard to low back pain: spinal manipulation; ultrasound; electrical stimulation; individual education; orthotics; acupuncture; traction; injection therapy; surgical discectomy (poor long-term effects).
The only therapies for which any evidence is supportive are massage, multidisciplinary rehabilitation, NSAIDS (small effect size) and muscle relaxants (significant adverse effects).
What is the evidence-based spine care practitioner to do? As a chiropractor, none of the treatments available to me is supported with any "high-quality evidence." M.N. Hegde discusses the same dilemma in his profession (speech language pathology).3 He states, "In accepting RCTs as the gold standard of treatment efficacy, speech-language pathologists will face a scientific and ethical dilemma. The clinicians will have to conclude that most of their treatment procedures have not been proven effective." The physical therapy profession would also face this same dilemma, as the ranges of therapies PTs provide are quite similar to those of chiropractic.
There are also examples in the clinical trial research of very high-impact and highly cited trials being contradicted with follow-up studies. A 2005 JAMA study8 evaluated how frequently the most cited trials published in the top three scientific journals are later contradicted by follow-up research. The researchers found that in 16% of trials, the initial results were reported to have stronger effects than follow-up trials; and that an additional 16 percent were completely contradicted by follow-up studies. These are disturbing statistics for practitioners and patients who rely on these trials for treatment guidance.
A Changing Paradigm?
There has been increasing discussion recently over the conclusion that RCTs are the gold standard research paradigm in all situations. Grossman, et al.,9 state: "The RCT is the best study design in some instances; indeed, it may always be the best study design for studying primary treatment effects in large phase III pharmaceutical trials in compliant populations. In other cases, however, particularly for many public health interventions, observational studies are not only more feasible but actually give more accurate results than RCTs."
Hegde3 makes the following arguments in defense of research paradigms other than the RCT:
- Randomization is not necessary to demonstrate the internal validity of an experiment.
- Generalizing results is not a matter of randomization or experimental design- it is a matter of replication.
- The population of patients to which results of RCTs are applied is so big that the sample, no matter how large, is miniscule in comparison.
- Those who choose to participate in RCTs are self selected, not randomly selected.
- And finally (I believe most importantly), control groups work well in nonclinical research, animal research and laboratory research. In clinical treatment, control groups generate negative consequences to participants that may match or exceed its benefits.
Can other study designs be developed with rigorous methods that mimic those of clinical trials? According to a 2000 study published in The New England Journal of Medicine,10 the answer is a resounding "yes."
The authors of this study identified published reports of randomized, controlled trials and reports of observational studies with either a cohort design or a case-control design that assessed the same clinical topic to determine the similarity or difference in results for the same variable with different study designs. The results were remarkably similar for each clinical topic examined (Bacille Calmette-Guerin vaccine and TB; mammography and breast cancer; cholesterol levels and death due to trauma; treatment of hypertension and stroke; treatment of hypertension and CHD). The authors concluded: "Our results challenge the current consensus about a hierarchy of study designs in clinical research.
Challenging the Status Quo
In order to effectively participate in health science-related interdisciplinary research, we need to be well-versed in the benefits of all research design paradigms. We also need to be aware of the possible limitations and pitfalls. How can we push the boundaries of our paradigms in terms of the research in health sciences, yet maintain the rigor of the research process and the respect from those in our scientific community?
I believe this is exactly the question I have asked and hopefully answered by challenging the notion that the RCT be considered the benchmark by which we judge all research methodology.
References
- Mayberry JA. "Scurvy and Vitamin C." Class & 3L paper, Food and Drug Law, April 27, 2004.
- Collier R. Legumes, lemons and streptomycin: a short history of the clinical trial. CMAJ, Jan. 6, 2001;180(1).
- Hegde MN. A methodological review of randomized clinical trials. Communicative Disorders Rev;1(1):15-36.
- Sackett DL, Hoey J. Why randomized controlled trials fail but needn't: a new series is launched. CMAJ, May 2, 2000:162(9).
- OCEBM Levels of Evidence. Centre for Evidence-Based Medicine (CEBM), June 5, 2014.
- Marks H. Rigorous uncertainty: why RA Fisher is important. Int J Epidemiology, 2003;32:932-937.
- Cochrane Back and Neck: Our Reviews. http://back.cochrane.org/our-reviews.
- Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA, 2005;294:218-228.
- Grossman J, et al. The randomized controlled trial: gold standard or merely standard. Perspectives Biol & Med, 2005;48(4):516-534.
- Concato J, et al. Randomized, controlled trials, observational studies, and the hierarchy of research designs. NEJM, June 22, 2000;342(25):1887-1892.
Dr. Brian Anderson is an assistant professor in the Department of Clinical Practice at National University of Health Sciences and a PhD student in health sciences at Northern Illinois University. Contact him with questions and comments at
.