Clinical and experimental studies are not the only types of published research we might encounter as we look for evidence to inform our practices. One of the most useful types is the literature review, which summarizes a group of studies.
Literature reviews fall into three categories:
- Narrative reviews or surveys of the literature
- Meta-analyses
- Systematic reviews
Bibliographies of review articles are also a good source for identifying related studies. For someone looking for a synopsis of current thinking on a research question, a recent literature review is an excellent place to start. It can tell you, among other things, which articles are worth taking the time to read in their entirety.
Narrative Reviews
The narrative review examines a group of studies in a particular subject area and, as all literature reviews do, attempts to come to some conclusion based on a synthesis of the group as a whole. The author determines which studies are selected to form the group, and this feature can be both a strength and a weakness.
The primary strength of the narrative review is that the author can discuss the methods and conclusions of important studies in detail, and give an informed opinion about their relative merits. A major weakness of narrative reviews is the risk associated with author selection; in some cases, the author may be unaware of or unable to access all the relevant studies. Poorer-quality studies may be included and treated as equal to better-quality studies. The author also may choose to ignore studies with outcomes that are not consistent with the hypothesis presented, although this practice is not considered acceptable and would be unlikely to pass a peer review.
Meta-Analyses
The meta-analysis, developed as a way to improve upon the weaknesses of the narrative review, has increasingly gained respect as a form of literature review. Meta-analysis is possible as a result of the development of computerized databases, which make locating studies much easier. This review format is more necessary than ever in health care due to the expanding volume of research studies.
In a meta-analysis, studies on a particular question are grouped according to pre-established criteria, and a literature search using one or more major online databases is conducted to locate the articles that meet those criteria. This practice is analogous to specifying eligibility criteria for clinical study participants prior to their recruitment into the study. Defining the criteria for inclusion in advance reduces the potential for selection bias on the part of the investigator.
Results of the studies that meet the stated criteria are then pooled: combined so the statistical power of the group of studies is much greater than each individual study alone. Meta-analysis is often used to estimate the size of a treatment effect or to settle a research question when there are several contradictory or inconclusive studies with small numbers of participants.
An important consideration in evaluating a meta-analysis is whether the studies that have been pooled are similar enough in terms of their hypotheses tested and outcomes measured. One criticism of this method is that while it can be an improvement over the more subjective narrative review, meta-analysis can sometimes pool such disparate studies that the authors are in effect combining apples and oranges. If the research question is about fruit salad, so to speak, there is no problem. If not, the results can be misleading. This is a particular risk in integrative health care fields in which the specific combinations of techniques and modalities used in an individual treatment session can vary widely.
Another concern with meta-analysis is that publication bias has the potential to skew the results. Publication bias is the tendency for journals to reject studies with negative or inconclusive results and more likely to publish those with positive results. When studies with negative findings are not included in any subsequent analysis, their absence can prejudice the results; for example, by making an intervention appear more effective than it really is.
Systematic Reviews
The systematic review is a more refined version of the meta-analysis and has become the new “gold standard” for research evidence. It attempts to compensate for publication bias by making additional efforts to identify every relevant study on a given research question, such as locating dissertations, hand-searching journals that are not indexed in online databases, and personally communicating with authors of unpublished studies. These are then usually weighted in terms of the strength and quality of the evidence presented.
For example, a study employing random assignment of participants to an intervention or a control group would be weighted more heavily than one without these design features. Standardized methods are used for weighting studies based on internal validity and appropriateness of their statistical analysis methods. Randomized, controlled trials are usually given more weight in a systematic review because of their greater capacity to link cause and effect, which places them higher on the hierarchical model of levels of evidence.
The Cochrane Collaboration
Systematic reviews have been popularized by the Cochrane Collaboration, a voluntary group of health care providers, consumers and scientists whose name commemorates the late British physician and epidemiologist Archie Cochrane.
Cochrane was a pioneer of the evidence-based medicine movement who strongly believed in the importance of systematic reviews in shifting medical care toward practice based on research evidence. (For more information on the Cochrane Collaboration, visit www.cochrane.org.) Abstracts of all published reviews can be viewed at the Cochrane website.
Some studies use a slightly different method of systematic review called best-evidence synthesis,1 developed as a way to combine the quantitative features of meta-analysis with the advantages of the qualitative narrative review. It is used when the studies that are located turn out to be too different from one another or lack sufficient data to be pooled in the manner done in a traditional meta-analysis.
Instead, reviewers evaluate the selected studies by carefully considering the evidence presented in each one, and then draw an overall conclusion about where the weight of the evidence lies.
Systematic reviews can be expected to continue to gain popularity as a way to synthesize a large quantity of research on a disputed question and render a verdict.
Guidelines for Evaluating Literature Reviews
A major consideration when evaluating any study is to what extent the research design and methods used are a good fit with the study question. The research question may be clearly articulated and relevant, but if the design and/or methods are poorly suited to answer it, the findings can be inconclusive, misleading or not interpretable. Particularly in research on integrative health therapies, the appropriateness of the treatment protocol to the question should be thoughtfully examined.
This issue of model fit validity is important for several reasons. Many of the integrative health disciplines do not have large bodies of research, so even one or two poorly conceived studies can have a negative impact on understanding a therapy's efficacy or practices. A protocol that is inappropriate to the study question also diminishes the usefulness of that research to practitioners and makes translating research into practice more difficult.
In-depth evaluation of literature reviews is a detailed and technically challenging process whose methods continue to be refined. However, several general guidelines can be applied to assess the quality of a review:
- The research question addressed by the review should be specific and clearly defined. If possible, check out the background and expertise of the reviewers, who should represent a balanced and comprehensive team suitable to conducting a review on the subject in question. A statistician familiar with systematic review procedures is a necessity.
- The methods used to locate studies should be described, along with the criteria for inclusion or exclusion of studies from the review.
- If studies are ranked or weighted, the internal validity of the studies included should be assessed according to objective and reproducible methods, and the weighting methods should be described.
- If a meta-analysis were performed, the pooled data should be combined appropriately (apples with apples). A table that lists the individual studies with brief summaries describing the participants, methods and results of each study helps the reader to judge similarities.
- Just as in any research study, the reviewers' conclusions should be supported by the data presented.
Reference
- Slavin RE. Best evidence synthesis: an intelligent alternative to meta-analysis. J Clin Epidemiology, 1995;48(1):9-18.
Dr. Martha Brown Menard is the author of Making Sense of Research and co-executive director of the Academic Consortium for Complementary & Alternative Health Care (ACCAHC). She is also the director of the Crocker Institute and a licensed massage therapist in Kiawah Island, S.C.