Meta-analysis of multiple outcomes: a multilevel approach

Abstract

In meta-analysis, dependent effect sizes are very common. An example is where in one or more studies the effect of an intervention is evaluated on multiple outcome variables for the same sample of participants. In this paper, we evaluate a three-level meta-analytic model to account for this kind of dependence, extending the simulation results of Van den Noortgate, López-López, Marín-Martínez, and Sánchez-Meca Behavior Research Methods, 45, 576–594 (2013) by allowing for a variation in the number of effect sizes per study, in the between-study variance, in the correlations between pairs of outcomes, and in the sample size of the studies. At the same time, we explore the performance of the approach if the outcomes used in a study can be regarded as a random sample from a population of outcomes. We conclude that although this approach is relatively simple and does not require prior estimates of the sampling covariances between effect sizes, it gives appropriate mean effect size estimates, standard error estimates, and confidence interval coverage proportions in a variety of realistic situations.

Appendices

Appendix A: Proof that the variance at the study level is the covariance

Suppose that two observed effect sizes, d _jk and d _j’k , stem from the same study, study k. According to Equation 6, d _jk  = γ ₀₀ + u _0k  + v _jk  + r _jk and d _j ' k  = γ ₀₀ + u _0k  + v _j ' k  + r _j ' k

Therefore,

$$ {\sigma}_{d_{jk}{d}_{j^{\prime }k}}={\sigma}_{\left({\gamma}_{00}+{u}_{0k}+{v}_{jk}+{r}_{jk}\right)\left({\gamma}_{00}+{u}_{0k}+{v}_{j^{\prime }k}+{r}_{j^{\prime }k}\right)} $$

Because γ ₀₀ is a constant, and adding a constant to one or both random variables does not affect their covariance, this covariance equals:

$$ {\sigma}_{d_{jk}{d}_{j^{\prime }k}}={\sigma}_{\left({u}_{0k}+{v}_{jk}+{r}_{jk}\right)\left({u}_{0k}+{v}_{j^{\prime }k}+{r}_{j^{\prime }k}\right)} $$

The covariance between two linear combinations is described by Mood, Graybill and Boes (1974, p. 179):

$$ \operatorname{cov}\left[{\displaystyle \sum_1^n{a}_i{X}_i,{\displaystyle \sum_1^m{b}_j{Y}_j}}\right]={\displaystyle \sum_1^n{\displaystyle \sum_1^m{a}_i{b}_j}}\operatorname{cov}\left[{X}_i,{Y}_j\right] $$

Hence:

$$ {\sigma}_{d_{jk}{d}_{j^{\prime }k}}={\sigma}_{r_{jk}{r}_{j^{\prime }k}}+{\sigma}_{r_{jk}{v}_{j^{\prime }k}}+{\sigma}_{r_{jk}{u}_{0k}}+{\sigma}_{v_{jk}{r}_{j^{\prime }k}}+{\sigma}_{v_{jk}{u}_{0k}}+{\sigma}_{v_{jk}{v}_{j^{\prime }k}}+{\sigma}_{u_{0k}{r}_{j^{\prime }k}}+{\sigma}_{u_{0k}{v}_{j^{\prime }k}}+{\sigma}_{u_{0k}{u}_{0k}} $$

Because in a multilevel model two residuals at the same level are assumed to be independent, as are residuals at two different levels,

$$ {\sigma}_{d_{jk}{d}_{j^{\prime }k}}={\sigma}_{u_{0k}{u}_{0k}}={\sigma}_u^2 $$

Appendix B: SAS Codes for the Example

Data set format

For the multilevel analyses, the data set should contain one row for each observed effect size. For our example, we prepared such a data set, called abuse, with the following variables (the dataset is available upon request from the first author):

• study: a study indicator with values from 1 to 39,

• outcome: an outcome indicator with values from 1 to 587,

• ES: the effect size expressed as bias corrected standardized mean differences,

• W: the inverse of the estimated sampling variance for each observed effect size, and

• X: an indicator variable for the two groups of outcomes (1 refers to the outcomes directly related to child abuse and neglect, 2 to outcomes related to risk factors).

The first ten rows are given below:

Two-level meta-analysis

For the random effects two-level analysis, the following code is run:

The Proc Mixed-command calls the mixed procedure for multilevel or linear mixed models. The data set is defined, and we ask for using the restricted maximum likelihood (REML) estimation procedure.

The Class-statement is used to define the categorical variables of our model, in our case the study and outcome indicators. In the Model-statement we define the model: the dependent variable (ES) on the left side of the equality sign, the predictor or moderator variables on the right. An intercept is included by default. In this random effects model, there are no moderator variables. The Solution-option requests the parameter regression coefficient estimates and tests in the output. The ddfm=Satterthwaite-option performs a general Satterthwaite approximation for the denominator degrees of freedom for the tests of the regression coefficients.

We use W, the inverse of the sampling variance, to weight the observed effect sizes in the analysis. However, weights that are used in the multilevel analysis will not only be based on the sampling variance, but also will automatically account for the estimated population variance(s) defined further. More specifically, the weights are equal to the inverse of the sum of the different sources of variance. The Random-statement specifies that the intercept varies randomly over outcomes. In the Parms-statement, we give starting values for the population variance of this intercept as well as for the residual variance. We use the Hold-option to fix the second parameter to the starting value of 1. In this way and by using the inverse of the sampling variance as weights, the level-one variance is automatically fixed at the sampling variances that we defined. Using a more realistic starting value for the first parameter (e.g., the estimate from a previous analysis) can speed up the estimation. Finally, we close the code using the Run-statement, and we submit the code.

For the mixed effects two-level analysis, the code is adapted as follows:

First, the X-variable is defined as a categorical variable by means of the Class-statement. Second, in the Model-statement we include the X-variable as a predictor. To get an estimate of the mean effect for each level of the X-variable, we drop the intercept, by using noint as an option in the Model-statement. Finally, to estimate and test the difference between both groups of outcomes in the expected effect, we use the Estimate-statement. The difference-parameter is labeled ‘group’, and is defined by a contrast with weights 1 and -1.1

Three-level meta-analysis

Because in the three-level random effects model we assume that the intercept might not simply vary over the 587 outcomes, but that there might be systematic differences between studies due to covariation between effect sizes from the same study, we include a second Random-statement:

We now have three sources of variance: between studies, between outcomes within studies, and sampling variance. In the Parms-statement, we define starting values for the three variances, and constrain the last one to 1.

The code for the random effects model is extended to the code for a mixed effects model in much the same way as for the two-level models.