![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
This article proposes a flexible extension of the Fay–Herriot model for making inferences from coarsened, group-level achievement data, for example, school-level data consisting of numbers of students falling into various ordinal performance categories. The model builds on the heteroskedastic ordered probit (HETOP) framework advocated by Reardon, Shear, Castellano, and Ho by allowing group parameters to be modeled with regressions on group-level covariates, and residuals modeled using the flexible exponential family of distributions recommended by Efron. We demonstrate that the alternative modeling framework, termed the “Fay–Herriot heteroskedastic ordered probit” (FH-HETOP) model, is useful for mitigating some of the challenges with direct maximum likelihood estimators from the HETOP model. We conduct a simulation study to compare the costs and benefits of several methods for using the FH-HETOP model to estimate group parameters and functions of them, including posterior means, constrained Bayes estimators, and the “triple goal” estimators of Shen and Louis. We also provide an application of the FH-HETOP model to math proficiency data from the Early Childhood Longitudinal Study. Code for estimating the FH-HETOP model and conducting supporting calculations is provided in a new package for the R environment.
References
|
Albert, J. H., Chib, S. (1993). Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association, 88, 669–679. doi:10.1080/01621459.1993.10476321 Google Scholar | |
|
Azzalini, A., Dalla Valle, A. (1996). The multivariate skew-normal distribution. Biometrika, 83, 715–726. doi:10.1093/biomet/83.4.715 Google Scholar | |
|
Carlin, B., Louis, T. (2000). Bayes and empirical Bayes methods for data analysis (2nd ed.). Boca Raton, FL: Chapman and Hall/CRC Press. doi:10.1201/9781420057669 Google Scholar | |
|
Casella, G. (1985). An introduction to empirical Bayes data analysis. The American Statistician, 39, 83–87. doi:10.2307/2682801 Google Scholar | |
|
Devine, O., Louis, T. (1994). A constrained empirical Bayes estimator for incidence rates in areas with small populations. Statistics in Medicine, 13, 1119–1133. doi:10.1002/sim.4780131104 Google Scholar | Medline | |
|
DeYoreo, M., Kottas, A. (2017). Bayesian nonparametric modeling for multivariate ordinal regression. Journal of Computational and Graphical Statistics, 27, 71–84. doi:10.1080/10618600.2017.1316280 Google Scholar | |
|
Efron, B. (2014). Two modeling strategies for empirical Bayes estimation. Statistical Science, 29, 285–301. doi:10.1214/13-sts455 Google Scholar | Medline | |
|
Efron, B. (2016). Empirical Bayes deconvolution estimates. Biometrika, 103, 1–20. doi:10.1093/biomet/asv068 Google Scholar | |
|
Efron, B., Morris, C. (1973). Stein’s estimation rule and its competitors—An empirical Bayes approach. Journal of the American Statistical Association, 68, 117–130. doi:10.2307/2284155 Google Scholar | |
|
Efron, B., Morris, C. (1977). Stein’s paradox in statistics. Scientific American, 236, 119–127. Google Scholar | |
|
Eilers, P., Marx, B. (1996). Flexible smoothing with B-splines and penalties. Statistical Science, 11, 89–102. Google Scholar | |
|
Fahle, E. M., Reardon, S. F. (2018). How much do test scores vary among school districts? New estimates using population data, 2009-2015. Educational Researcher, 47, 221–234. doi:10.3102/0013189X18759524 Google Scholar | SAGE Journals | |
|
Fay, R., Herriot, R. (1979). Estimates of income for small places: An application of James-Stein procedures to census data. Journal of the American Statistical Association, 74, 269–277. doi:10.2307/2286322 Google Scholar | |
|
Ferguson, T. S. (1973). A Bayesian analysis of some nonparametric problems. The Annals of Statistics, 1, 209–230. doi:10.1214/aos/1176342360 Google Scholar | |
|
Fienberg, S. E., Holland, P. W. (1972). On the choice of flattening constants for estimating multinomial probabilities. Journal of Multivariate Analysis, 2, 127–134. doi:10.1016/0047-259x(72)90014-0 Google Scholar | |
|
Firth, D. (1993). Bias reduction of maximum likelihood estimates. Biometrika, 80, 27–38. doi:10.2307/2336755 Google Scholar | |
|
Gelman, A., Carlin, J., Stern, H., Rubin, D. (1995). Bayesian data analysis. London, England: Chapman & Hall. Google Scholar | Crossref | |
|
Gelman, A., Rubin, D. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472. doi:10.1214/ss/1177011136 Google Scholar | |
|
Ghosh, M. (1992). Constrained Bayes estimation with applications. Journal of the American Statistical Association, 87, 533–540. doi:10.2307/2290287 Google Scholar | |
|
Ghosh, M., Rao, J. (1994). Small area estimation: An appraisal. Statistical Science, 9, 55–76. doi:10.1214/ss/1177010647 Google Scholar | |
|
Gilks, W. R., Richardson, S., Spiegelhalter, D. J. (1996). Markov Chain Monte Carlo in practice. London, England: Chapman & Hall. Google Scholar | |
|
Gill, J., Casella, G. (2009). Nonparametric priors for ordinal Bayesian social science models: Specification and estimation. Journal of the American Statistical Association, 104, 453–454. doi:10.1198/jasa.2009.0039 Google Scholar | |
|
Gu, Y., Fiebig, D. G., Cripps, E., Kohn, R. (2009). Bayesian estimation of a random effects heteroscedastic probit model. The Econometrics Journal, 12, 324–339. doi:10.1111/j.1368-423x.2009.00283.x Google Scholar | |
|
Haberman, S. J. (1980). Discussion of “regression models for ordinal data.” Journal of the Royal Statistical Society Series B, 42, 136–137. Google Scholar | |
|
Haberman, S. J. (2005). Identifiability of parameters in item response models with unconstrained ability distributions (ETS Research Rep. No. RR-05-24). Princeton, NJ: ETS. Google Scholar | |
|
Harvey, A. C. (1976). Estimating regression models with multiplicative heteroscedasticity. Econometrica, 44, 461–465. doi:10.2307/1913974 Google Scholar | |
|
Hedeker, D., Demirtas, H., Mermelstein, R. J. (2009). A mixed ordinal location scale model for analysis of Ecological Momentary Assessment (EMA) data. Statistics and its Interface, 2, 391–401. doi:10.4310/sii.2009.v2.n4.a1 Google Scholar | Medline | |
|
Johnson, V. E. (1996). On Bayesian analysis of multirater ordinal data: An application to automated essay grading. Journal of the American Statistical Association, 91, 42–51. doi:10.2307/2291381 Google Scholar | |
|
Johnson, V. E., Albert, J. H. (1999). Ordinal data modeling. New York, NY: Springer-Verlag. Google Scholar | |
|
Kapur, K., Li, X., Blood, E. A., Hedeker, D. (2015). Bayesian mixed-effects location and scale models for multivariate longitudinal outcomes: An application to Ecological Momentary Assessment data. Statistics in Medicine, 34, 630–651. doi:10.1002/sim.6345 Google Scholar | Medline | |
|
Kim, J., Choi, K. (2008). Closing the gap: Modeling within-school variance heterogeneity in school effect studies. Asia Pacific Education Review, 9, 206–220. doi:10.1007/bf03026500 Google Scholar | |
|
Kubokawa, T., Strawderman, W. E. (2013). Dominance properties of constrained Bayes and empirical Bayes estimators. Bernoulli, 19, 2200–2221. doi:10.3150/12-bej449 Google Scholar | |
|
Laird, N. (1978). Nonparametric maximum likelihood estimation of a mixing distribution. Journal of the American Statistical Association, 73, 215–232. doi:10.2307/2286284 Google Scholar | |
|
Leckie, G., French, R., Charlton, C., Browne, W. (2014). Modeling heterogeneous variance-covariance components in two-level models. Journal of Educational and Behavioral Statistics, 39, 307–332. doi:10.3102/1076998614546494 Google Scholar | SAGE Journals | |
|
Lockwood, J. R., McCaffrey, D. F. (2014). Correcting for test score measurement error in ANCOVA models for estimating treatment effects. Journal of Educational and Behavioral Statistics, 39, 22–52. doi:10.3102/1076998613509405 Google Scholar | SAGE Journals | |
|
Lockwood, J. R., Savitsky, T. D., McCaffrey, D. F. (2015). Inferring constructs of effective teaching from classroom observations: An application of Bayesian exploratory factor analysis without restrictions. Annals of Applied Statistics, 9, 1484–1509. doi:10.1214/15-aoas833 Google Scholar | Crossref | |
|
Louis, T. (1984). Estimating a population of parameter values using Bayes and empirical Bayes methods. Journal of the American Statistical Association, 79, 393–398. doi:10.2307/2288281 Google Scholar | |
|
Lyles, R. H., Moore, R., Manatunga, A., Easley, K. (2009). Covariate-adjusted constrained Bayes predictions of random intercepts and slopes. Journal of Modern Applied Statistical Methods, 8, 81–94. doi:10.22237/jmasm/1241136360 Google Scholar | Medline | |
|
McCullagh, P. (1980). Regression models for ordinal data (with discussion). Journal of the Royal Statistical Society Series B, 42, 109–142. Google Scholar | |
|
Mislevy, R. (1984). Estimating latent distributions. Psychometrika, 49, 359–381. doi:10.1007/bf02306026 Google Scholar | |
|
Mislevy, R., Beaton, A., Kaplan, B., Sheehan, K. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29, 133–161. doi:10.1111/j.1745-3984.1992.tb00371.x Google Scholar | |
|
Monroe, S., Cai, L. (2014). Estimation of a Ramsay-curve item response theory model by the Metropolis-Hastings Robbins-Monro algorithm. Educational and Psychological Measurement, 74, 343–369. doi:10.1177/0013164413499344 Google Scholar | SAGE Journals | |
|
Morris, C. (1983). Parametric empirical Bayes inference: Theory and applications (with discussion). Journal of the American Statistical Association, 78, 47–55. doi:10.2307/2287098 Google Scholar | |
|
National Center for Education Statistics . (2002). User’s manual for the ECLS-K first grade public-use data files and electronic code book (NCES 2002-135). Retrieved from https://nces.ed.gov/pubs2002/2002135_2.pdf Google Scholar | |
|
Ohlssen, D. I., Sharples, L. D., Spiegelhalter, D. J. (2007). Flexible random-effects models using Bayesian semi-parametric models: Applications to institutional comparisons. Statistics in Medicine, 26, 2088–2112. doi:10.1002/sim.2666 Google Scholar | Medline | |
|
Paddock, S. M., Louis, T. A. (2011). Percentile-based empirical distribution function estimates for performance evaluation of healthcare providers. Journal of the Royal Statistical Society: Series C (Applied Statistics), 60, 575–589. doi:10.1111/j.1467-9876.2010.00760.x Google Scholar | Medline | |
|
Paddock, S. M., Ridgeway, G., Lin, R., Louis, T. A. (2006). Flexible distributions for triple-goal estimates in two-stage hierarchical models. Computational Statistics & Data Analysis, 50, 3243–3262. doi:10.1016/j.csda.2005.05.008 Google Scholar | Medline | |
|
Pfefferman, D. (2002). Small area estimation—New developments and directions. International Statistical Review, 70, 125–143. doi:10.2307/1403729 Google Scholar | |
|
Plummer, M . (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. Proceedings of the 3rd International Workshop on Distributed Statistical Computing (DSC 2003), Vienna, Austria. Google Scholar | |
|
Rabe-Hesketh, S., Pickles, A., Skrondal, A. (2003). Correcting for covariate measurement error in logistic regression using nonparametric maximum likelihood estimation. Statistical Modelling, 3, 215–232. doi:10.1191/1471082x03st056oa Google Scholar | SAGE Journals | |
|
Rabe-Hesketh, S., Skrondal, A. (2012). Multilevel and longitudinal modeling using Stata, Volume II: Categorical responses, counts, and survival (3rd ed.). College Station, TX: Stata Press. Google Scholar | |
|
Reardon, S. F., Ho, A. D., Shear, B. R., Fahle, E. M., Kalogrides, D., DiSalvo, R. (2017). Stanford education data archive (Version 2.0). Retrieved from http://purl.stanford.edu/db586ns4974 Google Scholar | |
|
Reardon, S. F., Shear, B. R., Castellano, K. E., Ho, A. D. (2017). Using heteroskedastic ordered probit models to recover moments of continuous test score distributions from coarsened data. Journal of Educational and Behavioral Statistics, 42, 3–45. doi:10.3102/1076998616666279 Google Scholar | SAGE Journals | |
|
Roeder, K., Carroll, R., Lindsay, B. (1996). A semiparametric mixture approach to case-control studies with errors in covariables. Journal of the American Statistical Association, 91, 722–732. doi:10.2307/2291667 Google Scholar | |
|
Savitsky, T. D., McCaffrey, D. F. (2014). Bayesian hierarchical multivariate formulation with factor analysis for nested ordinal data. Psychometrika, 79, 275–302. doi:10.1007/s11336-013-9339-z Google Scholar | Medline | |
|
Segawa, E. (2005). A growth model for multilevel ordinal data. Journal of Educational and Behavioral Statistics, 30, 369–396. doi:10.3102/10769986030004369 Google Scholar | SAGE Journals | |
|
Shen, W., Louis, T. (1998). Triple-goal estimates in two-stage, hierarchical models. Journal of the Royal Statistical Society, Series B: Statistical Methodology, 60, 455–471. doi:10.1111/1467-9868.00135 Google Scholar | |
|
Shen, W., Louis, T. (1999). Empirical Bayes estimation via the smoothing by roughening approach. Journal of Computational and Graphical Statistics, 8, 800–823. doi:10.2307/1390828 Google Scholar | |
|
Shen, W., Louis, T. (2000). Triple-goal estimates for disease mapping. Statistics in Medicine, 19, 2295–2308. Google Scholar | Medline | |
|
Spiegelhalter, D., Best, N., Carlin, B., van der Linde, A. (2002). Bayesian measures of model complexity and fit (with discussion). Journal of the Royal Statistical Society, Series B: Statistical Methodology, 64, 583–639. doi:10.1111/1467-9868.00353 Google Scholar | |
|
Vehtari, A., Gelman, A., Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4 Google Scholar | |
|
Warm, T. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427–450. doi:10.1007/bf02294627 Google Scholar | |
|
Woods, C. M., Thissen, D. (2006). Item response theory with estimation of the latent population distribution using spline-based densities. Psychometrika, 71, 281–301. doi:10.1007/s11336-004-1175-8 Google Scholar | Medline |
