![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance estimation (RVE) provides a method for pooling dependent effects, even when information on the exact dependence structure is not available. When the number of studies is small or moderate, however, test statistics and confidence intervals based on RVE can have inflated Type I error. This article describes and investigates several small-sample adjustments to F-statistics based on RVE. Simulation results demonstrate that one such test, which approximates the test statistic using Hotelling’s T2 distribution, is level-α and uniformly more powerful than the others. An empirical application demonstrates how results based on this test compare to the large-sample F-test.
References
|
Ahn, S., Ames, A. J., Myers, N. D. (2012). A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82, 436–476. doi:10.3102/0034654312458162 Google Scholar | SAGE Journals | |
|
Alexander, R. A., Govern, D. M. (1994). A new and simpler approximation for ANOVA under variance heterogeneity. Journal of Educational and Behavioral Statistics, 19, 91–101. doi:10.3102/10769986019002091 Google Scholar | SAGE Journals | |
|
Bell, R. M., McCaffrey, D. F. (2002). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28, 169–181. Google Scholar | |
|
Cai, L., Hayes, A. F. (2008). A new test of linear hypotheses in OLS regression under heteroscedasticity of unknown form. Journal of Educational and Behavioral Statistics, 33, 21–40. doi:10.3102/1076998607302628 Google Scholar | SAGE Journals | |
|
Cooper, H. M. (2010). Research synthesis and meta-analysis (4th ed.). Thousand Oaks, CA: SAGE. Google Scholar | |
|
De Vibe, M., Bjørndal, A., Tipton, E., Hammerstrøm, K., Kowalski, K. (2012). Mindfulness based stress reduction (MBSR) for improving health, quality of life, and social functioning in adults. Campbell Systematic Reviews, 8. doi:10.4073/csr.2012.3 Google Scholar | |
|
Fai, A. H.-T., Cornelius, P. (1996). Approximate F-tests of multiple degree of freedom hypotheses in generalized least squares analyses of unbalanced split-plot experiments. Journal of Statistical Computation and Simulation, 54, 363–378. Google Scholar | |
|
Fisher, Z., Tipton, E. (2014). robumeta: An R-package for robust variance estimation in meta-analysis. Retrieved from arXiv preprint arXiv:1503.02220 Google Scholar | |
|
Gleser, L. J., Olkin, I. (2009). Stochastically dependent effect sizes. In Cooper, H., Hedges, L. V., Valentine, J. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357–376). New York, NY: Russell Sage Foundation. Google Scholar | |
|
Hedberg, E. C. (2011). ROBUMETA: Stata module to perform robust variance estimation in meta-regression with dependent effect size estimates. Statistical Software Components S457219, Boston College Department of Economics. Google Scholar | |
|
Hedges, L. V, Tipton, E., Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65. doi:10.1002/jrsm.5 Google Scholar | Medline | |
|
Hill, G. W. (1970). Algorithm 395: Student’s t quantiles. Communications of the ACM, 13, 617–619. Google Scholar | |
|
Krishnamoorthy, K., Yu, J. (2004). Modified Nel and Van der Merwe test for the multivariate Behrens–Fisher problem. Statistics & Probability Letters, 66, 161–169. doi:10.1016/j.spl.2003.10.012 Google Scholar | |
|
MacKinnon, J. G. (2013). Thirty years of heteroskedasticity-robust inference. In Chen, X., Swanson, N. R. (Eds.), Recent advances and future directions in causality, prediction, and specification analysis (pp. 437–461). New York, NY: Springer New York. doi:10.1007/978-1-4614-1653-1 Google Scholar | |
|
McCaffrey, D. F., Bell, R. M. (2006). Improved hypothesis testing for coefficients in generalized estimating equations with small samples of clusters. Statistics in Medicine, 25, 4081–4098. doi:10.1002/sim.2502 Google Scholar | Medline | |
|
McCaffrey, D. F., Bell, R. M., Botts, C. H. (2001, August). Generalizations of biased reduced linearization. In Proceedings of the Annual Meeting of the American Statistical Association. Google Scholar | |
|
Muirhead, R. J. (1982). Aspects of multivariate statistical theory. New York, NY: John Wiley & Sons. Google Scholar | |
|
Nel, D., van der Merwe, C. (1986). A solution to the multivariate Behrens-Fisher problem. Communications in Statistics - Theory and Methods, 15, 3719–3735. Google Scholar | |
|
Pan, W., Wall, M. M. (2002). Small-sample adjustments in using the sandwich variance estimator in generalized estimating equations. Statistics in Medicine, 21, 1429–1441. doi:10.1002/sim.1142 Google Scholar | Medline | |
|
Polanin, J. R., Pigott, T. D. (2015). The use of meta-analytic statistical significance testing. Research Synthesis Methods, 6, 63–73. doi:10.1002/jrsm.1124 Google Scholar | Medline | |
|
Raudenbush, S. W., Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage. Google Scholar | |
|
Samson, J. E., Ojanen, T., Hollo, A. (2012). Social goals and youth aggression: Meta-analysis of prosocial and antisocial goals. Social Development, 21, 645–666. doi:10.1111/j.1467-9507.2012.00658.x Google Scholar | |
|
Tanner-Smith, E. E., Tipton, E. (2014). Robust variance estimation with dependent effect sizes: Practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 5, 13–30. doi:10.1002/jrsm.1091 Google Scholar | Medline | |
|
Tanner-Smith, E. E., Wilson, S. J., Lipsey, M. W. (2013). The comparative effectiveness of outpatient treatment for adolescent substance abuse: a meta-analysis. Journal of Substance Abuse Treatment, 44, 145–158. doi:10.1016/j.jsat.2012.05.006 Google Scholar | Medline | |
|
Tipton, E. (2013). Robust variance estimation in meta-regression with binary dependent effects. Research Synthesis Methods, 4, 169–187. doi:10.1002/jrsm.1070 Google Scholar | Medline | |
|
Tipton, E. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. doi:10.1037/met0000011 Google Scholar | Medline | |
|
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., Newcombe, N. S. (2013). The malleability of spatial skills: A meta-analysis of training studies. Psychological Bulletin, 139, 352–402. doi:10.1037/a0028446 Google Scholar | Medline | |
|
Webb, M., MacKinnon, J. G. (2013). Wild bootstrap inference for wildly different cluster sizes (No. 1314). Ontario, Canada: Kingston. Google Scholar | |
|
Williams, R. T. (2012). Using robust standard errors to combine multiple regression estimates with meta-analysis. Loyola University. Retrieved from http://ecommons.luc.edu/luc_diss/405/ Google Scholar | |
|
Wilson, S. J., Lipsey, M. W., Tanner-Smith, E., Huang, C. H., Steinka-Fry, K. T. (2011). Dropout prevention and intervention programs: Effects on school completion and dropout Aaong school-aged children and youth: A systematic review. Campbell Systematic Reviews, 8. doi: 10.4073/csr.2011.8 Google Scholar | |
|
Zhang, J.-T. (2012). An approximate Hotelling T2-test for heteroscedastic one-way MANOVA. Open Journal of Statistics, 2, 1–11. Google Scholar | |
|
Zhang, J.-T. (2013). Tests of linear hypotheses in the ANOVA under heteroscedasticity. International Journal of Advanced Statistics and Probability, 1, 9–24. Google Scholar |
