Chapter 7: Meta-analysis

Intro

Introduction

Thus far, we have examined the basic types of investigations in the health research literature that are designed to compare study and control groups. These investigations often provide consistent results. At times, however, studies published in the health research literature seem to conflict with one another, making it difficult to provide definitive answers to important study questions.

It is often desirable to be able to combine data obtained in a variety of investigations and to use all the information to address a study question. Meta-analysis is a collection of methods for quantitatively combining information from different investigations to reach conclusions or address questions that were not possible on the basis of a single investigation.

Meta-analysis aims to produce its conclusion by combining data from two or more existing investigations. Traditionally, this process of research synthesis has been the review article’s role. In recent years, it has been increasingly recognized that the informal and subjective process of literature review has not always produced accurate conclusions. Meta-analysis of observational studies and randomized controlled trials has now become a standard part of health research as reflected in the publications of standard methods for their reporting with their own acronyms PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) ([1)] and MOOSE (Meta-analysis of Observational Studies in Epidemiology) ([2]).

Meta-analysis can be viewed as one method to be used as part of what is called a systematic review. Learn More 7.1 discusses systematic reviews and their relationship to meta-analysis.


Learn More 7.1: Systematic Reviews

Meta-analyses are often a component of a larger research effort known as a systematic review. Systematic reviews or overviews aim to comprehensively identify and analyze all the research literature on a given topic ranging from the etiology of a disease, to a diagnostic or screening test, to a potential preventive, curative, or rehabilitative intervention. Systematic reviews are often considered the gold standard for summarizing the literature.

They utilize methods of systematic literature review designed to identify all existing studies regardless of the type of study or whether they were published in the peer review literature. Unlike meta-analysis, systematic reviews addresses all previously researched questions related to a defined topic. The distinction between a meta-analysis and a systematic review is illustrated in the following scenario:


Mini-Study 7.1

A systematic review of the impact of exercise on coronary artery disease was conducted using a systematic literature review and examining the physiological, epidemiological, and clinical interventional research related to etiology, prevention, and treatment of coronary artery disease including the use of exercise for rehabilitation. As part of the systematic review, a meta-analysis was conducted to measure the magnitude of the impact of moderate levels of regular exercise on the occurrence of first episode of myocardial infarction

Notice that a systematic review casts a wide net. Meta-analysis as opposed to a systematic review focuses on a narrower and more clearly defined questions, often ones that have already been extensively investigated. The process and structure of systematic reviews have been formalized by the Cochrane Collaborative, an international network that produces and updates systematic review on a wide range of subjects 3. The Cochrane Collaborative has developed a standard format for developing and reporting systematic reviews. These Cochrane Reviews are far too extensive to publish in research journals. They are available on the Cochrane Collaboration Web site at http://www2.cochrane.org/reviews/. The structure of the report of a Cochrane Review includes all of the following components:

  1. Plain-language summary—A short statement summarizing the review, specifically aimed at lay people.
  2. Structured abstract—A structured summary of the review, subdivided into sections similar to the main review. This may be published independently from the review and appears on the medical bibliographic database MEDLINE.
  3. Background—An introduction to the question considered, including, for example, details on causes and incidence of a given problem, the possible mechanism of action of a proposed treatment, uncertainties about management options, and so on.
  4. Objectives—A short statement of the aim of the review.
  5. Selection criteria—A brief description of the main elements of the question under consideration.
  6. Search strategy for identification of studies.
  7. Methods of the review—A description of how studies eligible for inclusion in the review were selected, how their quality was assessed, how data were extracted from the studies, how data were analyzed, whether any subgroups were studied or any sensitivity analyses were carried out, and so on.
  8. Description of studies—A description of how many studies were found, what were their inclusion criteria, how big they were, and so on.
  9. Methodological quality of included studies—Were there any reasons to doubt the conclusions of any studies because of concerns about the study quality?
  10. Results—What do the data show? The results section may be accompanied by a graph to show a meta-analysis, if this was carried out.
  11. Discussion—This includes interpretation and assessment of results.

Systematic reviews are also becoming the starting point for development of clinical guidelines or evidence-based recommendation, as we will see in the Chapter 13.


Let us begin our approach to meta-analysis by illustrating one reason why it is important to combine the results of investigations when addressing a previously researched question. Let us examine one extreme example indicating why the conclusion reached by combining investigations might be different from those reached by examining one investigation at a time:


Mini-Study 7.2

Assume that we are interested in examining a recent innovation in the treatment of coronary artery disease known as transthoracic laser coronaryplasty (TLC). TLC is designed to treat coronary artery disease through the chest wall without using invasive techniques. The first two studies of TLC produced the following results:

Study 1

Study 1
Die Live Total
TLC 230 50 280
Control 530 210 740
Image not available

Study 2

Study 2
Die Live Total
TLC 190 405 595
Control 50 210 260
Image not available

The investigators concluded that both studies suggested that TLC produced worse outcomes than the control treatment increasing the chance of dying. Before relegating this technique to history, however, they decided to combine the results of the two studies and see what happened. Combining the data from the two studies produced the results shown below.

Combined Studies 1 and 2

Combined Studies 1 and 2
Die Live Total
TLC 420 455 875
Control 580 420 1,000
Image not available

Notice that after combining study 1 and 2 the results have changed and it now looks like TLC produces better outcomes than the control treatment. The odds ratio and relative risk now suggest that TLC reduces the chances of dying. Thus, combining studies may produce some surprising results.7.1

This process sets into motion a widespread effort to evaluate the use of TLC in a variety of settings and for a variety of indications worldwide. Most studies focused on single-vessel coronary artery disease as assessed by new noninvasive procedures. Over the next several years, dozens of studies resulted in apparently conflicting results. Thus, it was considered important to conduct a full-scale meta-analysis evaluating the effects of TLC on single-vessel coronary artery disease. Let us now look at the steps in conducting a meta-analysis using the M.A.A.R.I.E. framework.

7.1 This is known as Simpson’s Paradox. It is a very unusual situation illustrated here because of its dramatic impact. Its occurrence requires large differences between the numbers of study group and control group participants in the two studies.

Method ([4])

The process of combining information using meta-analysis can be best understood if we regard each of the studies included in the analysis as parallel to one study site in a multiple-site investigation. In a multiple-site investigation, the investigator combines the data from multiple sites to draw conclusions or interpretations. In meta-analysis, the investigator combines information from multiple studies to draw conclusions or interpretations. This parallel structure allows us to learn about meta-analysis using the M.A.A.R.I.E. framework.

As with our other uses of the M.A.A.R.I.E. framework, we start by defining the study question or study hypothesis. Meta-analysis can be used to accomplish a variety of purposes. It may begin by defining a hypothesis related to the specific purpose for conducting the meta-analysis. Meta-analysis might be used to accomplish any of the following purposes:

  • Establish statistical significance when studies are conflicting
  • Establish the best possible estimate of the magnitude of the effect
  • Evaluate harms when small numbers of adverse events are observed in earlier studies
  • Examine subgroups when the numbers in each previous investigation are not large enough to examine subgroups

As with our other types of investigations, the investigators ideally begin with a study hypothesis and proceed to test that hypothesis and draw inferences. When they do this for a therapy, for instance, they may hypothesize that the treatment has been shown to have efficacy.

The studies that should be included in a meta-analysis depend on the purpose of the analysis. Thus, the study hypothesis of the meta-analysis helps to determine the inclusion and exclusion criteria that should be used in identifying relevant studies. The following example shows how the hypothesis can help to determine which studies to include:


Mini-Study 7.3

In preparation for a meta-analysis, researchers searched the world’s literature and obtained the following 25 studies of TLC for single-vessel coronary artery disease. These investigations had characteristics which allowed them to be grouped into the following types of studies:

  • Five studies of men with single-vessel disease treated initially with coronary bypass surgery versus medication versus TLC
  • Five studies of men and women treated initially with TLC versus bypass surgery
  • Five studies of men and women treated initially with TLC versus medication
  • Five studies of men treated with TLC versus medication after previous bypass surgery
  • Five studies of women treated with repeat TLC versus medication after previous TLC

If the meta-analysis is designed to test a hypothesis, then the studies to be included are chosen because they address issues relevant to the hypothesis. For instance, if the investigator wanted to test the hypothesis that men do better than women when TLC is used to treat single-vessel coronary artery disease, then studies B and C should be used in the meta-analysis. These investigations include comparisons of the outcomes in both men and women.

If the investigators were interested in testing the hypothesis that initial TLC is better than surgery for single-vessel coronary artery disease, then studies A and B should be used in the meta-analysis because these studies compare TLC versus surgery as the initial therapy. Alternatively, if the researcher hypothesized that medication was the best treatment for single-vessel coronary artery disease, then studies A, C, D, and E would be used. In general, the studies that are used are determined by the purpose of the investigation as defined by the study hypothesis of the meta-analysis.

Despite the many similarities between meta-analysis and a multiple-site investigation, there is one important difference. In original research the investigator may define the study question and then identify settings and study participants that are suited to addressing the question and obtaining the needed sample size. In meta-analysis, the questions we may ask are often limited by the availability of previous studies. Thus, the study population and the sample size are largely outside the investigator’s control.

To try to circumvent this problem, meta-analysis researchers often define a question or issue broadly and begin by identifying all investigations related to that issue. When this is done, the investigators are conducting an exploratory meta-analysis as opposed to a hypothesis-driven meta-analysis.

In conducting an exploratory meta-analysis of TLC, for instance, the investigator might initially include all 25 studies just mentioned. Thus, the meta-analysis researcher would define the study group as consisting of those who received TLC, and the control group would consist of all the individuals receiving other therapy.

This process of using all available studies without a specific hypothesis is parallel to the process of conducting a conventional investigation without defining a study hypothesis. This type of exploratory meta-analysis can be useful, but must be conducted carefully and interpreted differently from hypothesis-driven meta-analysis. Despite the potential dangers of combining studies with very different characteristics, the limited number of available studies makes it important for meta-analysis to include techniques for combining very different types of studies.

Meta-analysis attempts to turn the diversity of studies into an advantage. Combining studies with different characteristics may allow us to harness the benefit of diversity. By including apples and oranges, we can ask whether it makes a difference if a fruit is an apple or an orange or whether it is enough that it is a fruit.

The approach for harnessing the benefit of diversity is discussed later. For now, we must recognize that there are actually two types of meta-analysis, hypothesis-based and exploratory.

It is important to remember that the fundamental difference between meta-analysis and other types of investigations is that the data have already been collected and the researcher’s choice is limited to including or excluding an existing study from the meta-analysis. Thus, the sample size in meta-analysis is limited by the existence of relevant studies.

Other types of investigations usually start by defining the question to be investigated. This question determines the types of individuals who should be included in the investigation. Similarly, the question to be addressed by a meta-analysis determines the types of studies that should be included in the meta-analysis. Thus, in hypothesis-driven meta-analysis, the first question we need to ask is whether a particular study is relevant to the meta-analysis’s study question or hypothesis.

Assignment ([4],[5])

Process of Assignment

Once the study question is defined, the investigators can determine which studies to include in a meta-analysis. This identification of studies to include is the assignment process, requires us to ask the question: Have all the relevant studies been identified?

Identifying all relevant studies is an essential step in the assignment process of a meta-analysis. It is important that the investigator describes the method used to search for research reports, including enough detail to allow subsequent investigators to obtain all the identified literature. This can even include unpublished data. Doctoral dissertations, abstracts, grant reports, and registries of studies are other possible ways to locate previous research.7.2

7.2 In performing this search, it is important to avoid double counting. Studies originally presented as abstracts, for instance, will often subsequently appear as original articles. Including the same data two or more times jeopardizes the accuracy of the results of a meta-analysis by violating the assumption that the data obtained in each of the studies are independent of the other studies