wildmeta
Typical methods to conduct meta-analysis—pooling effect sizes or analyzing moderating effects with meta-regression—work under the assumption that the effect size estimates are independent. However, primary studies often report multiple estimates of effect sizes. Presence of multiple effect sizes leads to dependence as the estimates within each study are likely correlated (e.g., because the same participants provide multiple outcome scores). The increasingly popular method to handle such dependence, robust variance estimation (RVE), results in inflated Type 1 error rate when the number of studies is small (Hedges, Tipton & Johnson, 2010; Tipton, 2015).
Tipton (2015) and Tipton & Pustejovsky (2015) examined several small sample correction methods. Tipton (2015) recommended CR2 type correction for RVE as well as the use of Satterthwaite degrees of freedom for single coefficient tests. Tipton & Pustejovsky (2015) examined corrections for multiple-contrast hypothesis tests. The authors found that the HTZ test, which is an extension of the CR2 correction method with the Satterthwaite degrees of freedom, controlled Type 1 error rate adequately even when the number of studies was small. However, Joshi, Pustejovsky & Beretvas (2022) showed, through simulations, that the HTZ test can be conservative. The authors examined another method, cluster wild bootstrapping (CWB), that has been studied in the econometrics literature but not in the meta-analytic context. The results of the simulations from Joshi et al. (2022) showed that CWB adequately controlled for Type 1 error rate and provided higher power than the HTZ test, especially for multiple-contrast hypothesis tests.
The goal of this package is to provide applied meta-analytic researchers a set of functions with which they can conduct single coefficient tests or multiple-contrast hypothesis tests using cluster wild bootstrapping.
Installation
Install the latest release from CRAN:
install.packages("wildmeta")
You can install the development version from GitHub with:
# install.packages("devtools")
devtools::install_github("meghapsimatrix/wildmeta")
Example
The following example uses the SATCoaching
dataset from the
clubSandwich
package (Pustejovksy, 2020), originally from DerSimonian
and Laird (1983). The standardized mean differences represent the
effects of SAT coaching on SAT verbal (SATV) and/or SAT math (SATM)
scores.
The code below runs cluster wild bootstrapping to test the
multiple-contrast hypothesis that the effect of coaching does not differ
based on study type. The study_type
variable indicates whether groups
compared in primary studies were matched, randomized, or non-equivalent.
The meta-regression model also controls for hours of coaching provided
(hrs
) and whether the students took math or verbal test (test
).
Below, we run a zero-intercept meta-regression model.
Below, we use the robumeta::robu()
function to fit the full model. The
functions in our package work with models fit using robumeta::robu()
(Fisher, Tipton, & Zhipeng, 2017) and metafor::rma.mv()
(Viechtbauer,
2010).
library(wildmeta)
library(clubSandwich)
library(robumeta)
full_model <- robu(d ~ 0 + study_type + hrs + test,
studynum = study,
var.eff.size = V,
small = FALSE,
data = SATcoaching)
Wald_test_cwb(full_model = full_model,
constraints = constrain_equal(1:3),
R = 12,
seed = 20201210)
#> Test Adjustment CR_type Statistic R p_val
#> 1 CWB CR0 CR0 Naive-F 12 0.4166667
Related Work
We want to recognize other packages that provide algorithms to conduct
bootstrapping. The
fwildclusterboot
package runs the fast version of cluster wild bootstrapping proposed by
Roodman et al. (2019) and is based on the Stata’s boottest package
(Fischer & Roodman, 2021). The package runs bootstrapping for linear
regression models and fixed effects models. To the best of our
knowledge, meta-analytic models, weighted linear regression models, and
multiple contrast hypothesis tests are not supported by the package.
The multiwayvcov
package implements cluster robust variance estimation as well as
different types of cluster bootstrapping, including pair, residual, and
cluster wild bootstrapping (Graham, Arai & Hagströmer, 2016). For
cluster wild bootstrapping, Rademacher weights, Mammen weights, and
standard normal weights are available. The functions in the package
return cluster robust variance-covariance matrices. Our package, on the
other hand, outputs p-values from bootstrap hypothesis tests. Further,
our package is particularly created for applied meta-analysis whereas
the multiwayvcov
can be used with any regression analysis involving
clusters.
The boot
package can be
used to generate bootstrap replicates (Canty & Ripley, 2020). The main
function in the package, boot()
requires a dataset and a function that
calculates the estimate of interest. The function returns the bootstrap
estimate. The package also contains another function, boot.ci()
that
can be used to calculate bootstrap confidence intervals. The package
allows parallel programming. However, the boot
package does not
provide functionality for cluster wild bootstrapping.
The lmboot
package
implements pair, residual and wild bootstrapping for linear models
(Heyman, 2019). The package does not provide Rademacher or Mammen
weights for wild bootstrapping. The output of the function is a
bootstrap sampling distribution. Further, the package does not work with
clustered data.
Acknowledgments
Huge shout out to my extremely talented friend
RAZ for creating our hex sticker within minutes.
We are also extremely thankful to Wolfgang
Viechtbauer for helping us solve
this
issue
we had with the update()
function. We also thank Mikkel Vembye for
testing our package and giving us very helpful feedback, and Sangdon
Lim for helping us debug some issues.
References
Canty A. & Ripley B. (2020). boot: Bootstrap R (S-Plus) Functions. R package version 1.3-25. https://CRAN.R-project.org/package=boot
DerSimonian, R., & Laird, N. (1983). Evaluating the effect of coaching on SAT scores: A meta-analysis. Harvard Educational Review, 53(1), 1-1.
Fischer, A. & Roodman, D. (2021). fwildclusterboot: Fast Wild Cluster Bootstrap Inference for Linear Regression Models. Available from https://cran.r-project.org/package=fwildclusterboot.
Fisher, Z., Tipton, E., & Zhipeng, H. (2017). robumeta: Robust variance meta-regression. Retrieved from https://CRAN.R-project.org/package=robumeta
Graham N., Arai M., & Hagströmer, B (2016). multiwayvcov: Multi-Way Standard Error Clustering. R package version 1.2.3. https://CRAN.R-project.org/package=multiwayvcov
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65.
Heyman, M. (2019). lmboot: Bootstrap in Linear Models. R package version 0.0.1. https://CRAN.R-project.org/package=lmboot
Joshi, M., Pustejovsky, J. E., & Beretvas, S. N. (2022). Cluster wild bootstrapping to handle dependent effect sizes in meta-analysis with small number of studies. Research Synthesis Methods. https://doi.org/10.1002/jrsm.1554
Pustejovsky, J. E. (2020). clubSandwich: Cluster-robust (sandwich) variance estimators with small-sample corrections [R package version 0.4.2]. R package version 0.4.2. https://CRAN.R-project.org/package=clubSandwich
Tipton, E. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393.
Tipton, E., & Pustejovsky, J. E. (2015). Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression. Journal of Educational and Behavioral Statistics (Vol. 40).
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48.