![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
A new approach to problems of multiple significance testing was presented in Benjamini and Hochberg (1995), which calls for controlling the expected ratio of the number of erroneous rejections to the number of rejections–the False Discovery Rate (FDR). The procedure given there was shown to control the FDR for independent test statistics. When some of the hypotheses are in fact false, that procedure is too conservative. We present here an adaptive procedure, where the number of true null hypotheses is estimated first as in Hochberg and Benjamini (1990), and this estimate is used in the procedure of Benjamini and Hochberg (1995). The result is still a simple stepwise procedure, to which we also give a graphical companion. The new procedure is used in several examples drawn from educational and behavioral studies, addressing problems in multi-center studies, subset analysis and meta-analysis. The examples vary in the number of hypotheses tested, and the implication of the new procedure on the conclusions. In a large simulation study of independent test statistics the adaptive procedure is shown to control the FDR and have substantially better power than the previously suggested FDR controlling method, which by itself is more powerful than the traditional family wise error-rate controlling methods. In cases where most of the tested hypotheses are far from being true there is hardly any penalty due to the simultaneous testing of many hypotheses.
References
| Benjamini, Y, Hochberg, Y The adaptive control of the false discovery rate in multiple independent testing problemsSeries in Statistics 93.1, Technical Report of the Department of Statistics and OR1993Tel Aviv, IsraelTel Aviv University Google Scholar | |
| Benjamini, Y, Hochberg, Y Controlling the False Discovery Rate—a new and powerful approach to multiple testingJournal of the Royal Statistical Society B199557289300 Google Scholar | |
| Benjamini, Y, Hochberg, Y, Stark, PB Confidence Intervals with more power to determine the sign: Two ends constrain the meansJournal of the American Statistical Association199893309317 Google Scholar, Crossref | |
| Cox, DR A remark on multiple comparison methodsTechnometrics19652149156 Google Scholar | |
| Harvånek & Chytil Mechanizing hypotheses formation—a way for computerized exploratory data analysis?Bulletin of the International Statistical Institution198350l104121 Google Scholar | |
| Helperin, M, Lan, GKK, Hamdy, MI Some implications of an alternative definition of the multiple comparison problemBiometrika198875773778 Google Scholar, Crossref | |
| Hochberg, Y A sharper Bonferroni procedure for multiple tests of significanceBiometrika198875800803 Google Scholar, Crossref | |
| Hochberg, Y, Benjamini, Y More powerful procedures for multiple significance testingStatistics in Medicine19909811818 Google Scholar, Crossref, Medline | |
| Hochberg, Y, Hommel, G Kotz, S Step-up multiple testing procedures: Encyclopedia for Statistical Sciences1997Supplementary Volume 2 Google Scholar | |
| Hochberg, Y, Tamhane, A Multiple Comparison Procedures1987NYWiley & Sons Google Scholar, Crossref | |
| Holm, S A simple sequentially rejective multiple test procedureScandinavian Journal of Statistics197966570 Google Scholar | |
| Hommel, G A stage wise rejective multiple test procedure based on a modified Bonferroni testBiometrika198875383386 Google Scholar, Crossref | |
| Hommel, G, Hoffmann, T Bauer, P, Hommel, G, Sonnemann, E Controlled uncertaintyMultiple hypotheses testing1987HeidelbergSpringer154161 Google Scholar | |
| Lyness, SA Predictors of differences between type A and B individuals in heart rate and blood pressure reactivityPsychological Bulletin19931142266295 Google Scholar, Crossref, Medline | |
| Raviv, A, Sadeh, A, Raviv, A, Silberstein, O The reaction of the youth in Israel to the assassination of prime minister Yizhak RabinPolitical Psychology199819255278 Google Scholar, Crossref | |
| Schweder, T, Spjøctvoll, E Plots of p-values to evaluate many tests simultaneouslyBiometrika198269493502 Google Scholar, Crossref | |
| Seeger, P A note on a method for the analysis of significances en massTechnometrics196810586593 Google Scholar, Crossref | |
| Sen, PK Some remarks on Simes-type multiple tests of significanceJournal of Statistical Planning and Inference1999a821–2139145 Google Scholar, Crossref | |
| Sen, PK Multiple comparisons in interim analysisJournal of Statistical Planning and Inference1999b821–2523 Google Scholar, Crossref | |
| Shaffer, JP Multiple hypothesis-testingAnnual Review of Psychology199546561584 Google Scholar, Crossref | |
| Simes, RJ An improved Bonferroni procedure for multiple tests of significanceBiometrika198673751754 Google Scholar, Crossref | |
| Soriç, B Statistical discoveries and effect size estimationJournal of the American Statistical Association198984608610 Google Scholar | |
| Troendle, JF Step wise normal theory multiple test procedures controlling the false discovery rateJournal of Statistical Planning and Inference2000841–2139158 Google Scholar, Crossref | |
| Tukey, JW The Philosophy of multiple comparisonsStatistical Science1991 6100116 Google Scholar, Crossref | |
| Tukey, JW Some thoughts on clinical trials, especially problems of multiplicityScience1977198697684 Google Scholar, Crossref | |
| Victor, N Exploratory data analysis and clinical researchMethods of Information in Medicine1982215354 Google Scholar, Medline | |
| Williams, VSL, Jones, LV, Tukey, JW Controlling error in multiple comparisons, with special attention to the National Assessment of Educational ProgressTechnical Report #331994Research Triangle Park, NCNational Institute of Statistical Sciences Google Scholar | |
| Williams, VSL, Jones, LV, Tukey, JW Controlling :error in multiple comparisons, with examples from state-to-state differences in educational achievementJournal of Educational and Behavioral Statistics1999244269 Google Scholar, SAGE Journals | |
| Yekutieli, D, Benjamini, Y Resampling-based false discovery rate.controlling multiple test procedures for correlated test statisticsJournal of Statistical Planning and Inference1999821–2171196 Google Scholar, Crossref |
