![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Several states have changed their statewide achievement tests over the past 5 years. These changes may pose difficulties for educators tasked with identifying students in need of additional support. This study evaluated the stability of decision-making accuracy estimates across changes to the statewide achievement test. We analyzed extant data from a large suburban district in Wisconsin in 2014–2015 (N = 2,774) and 2015–2016 (N = 2,882). We estimated the decision-making accuracy of recommendations from the Measures of Academic Progress for predicting risk on a Common Core State Standards aligned test (2014–2015) and a new test based on updated academic standards (2015–2016) in reading and math. Findings suggest that sensitivity and specificity estimates were relatively stable in math. Changes in the criterion measure were associated with decreased sensitivity when predicting performance in reading. These results provide initial support for educators to continue existing screening practices until test vendors or state educational agencies establish cut-scores for predicting risk on the newer test. Using a lower cut-score to establish risk (increasing sensitivity while decreasing specificity) may be prudent in reading. Limitations and directions for future research are discussed.
Keywords screening/benchmarking, math, reading/literacy
References
Section:
|
Baker, D. L., Biancarosa, G., Park, B. J., Bousselot, T., Smith, J.-L., Baker, S. K., . . . Tindal, G. (2015). Validity of CBM measures of oral reading fluency and reading comprehension on high-stakes reading assessments in grades 7 and 8. Reading and Writing, 28, 57–105. doi:10.1007/s11145-014-9505-4 Google Scholar | Crossref | ISI | |
|
Catts, H. W., Fey, M. E., Zhang, X., Tomblin, J. B. (2001). Estimating the risk of future reading difficulties in kindergarten children: A research-based model and its clinical implementation. Language, Speech, and Hearing Services in Schools, 32, 38–50. Google Scholar | Crossref | Medline | ISI | |
|
Catts, H. W., Petscher, Y., Schatschneider, C., Bridges, M. S., Mendoza, K. (2009). Floor effects associated with universal screening and their impact on the early identification of reading disabilities. Journal of Learning Disabilities, 42, 163–176. Google Scholar | SAGE Journals | ISI | |
|
Center on Standards, Alignment, Instruction, and Learning . (2016). State activity around adoption and replacement of CCR standards and aligned assessments: 2011-2015. Available from http://www.c-sail.org Google Scholar | |
|
Codding, R. S., Petscher, Y., Truckenmiller, A. (2015). CBM reading, mathematics, and written expression at the secondary level: Examining latent composite relations among indices and unique predictions with a state achievement test. Journal of Educational Psychology, 107, 437–450. doi:10.1037/a0037520 Google Scholar | Crossref | Medline | |
|
Decker, D. M., Hixson, M. D., Shaw, A., Johnson, G. (2014). Classification accuracy of oral reading fluency and maze in predicting performance on large-scale reading assessments. Psychology in the Schools, 51, 625–635. doi:10.1002/pits.21773 Google Scholar | Crossref | ISI | |
|
Espin, C., Wallace, T., Lembke, E., Campbell, H., Long, J. D. (2010). Creating a progress-monitoring system in reading for middle-school students: Tracking progress toward meeting high-stakes standards. Learning Disabilities Research & Practice, 25, 60–75. doi:10.1111/j.1540-5826.2010.00304.x Google Scholar | Crossref | |
|
Every Student Succeeds Act , Pub. L., 114-95. (2015). Google Scholar | |
|
Ford, J. W., Missall, K. N., Hosp, J. L., Kuhle, J. L. (2016). Comparing two CBM maze selection tools: Considering scoring and interpretive metrics for universal screening. Journal of Applied School Psychology, 32, 329–353. doi:10.1080/15377903.2016.1207738 Google Scholar | Crossref | |
|
Fuchs, L. S., Fuchs, D., Compton, D. L. (2010). Rethinking response to intervention at middle and high school. School Psychology Review, 39, 22–28. Google Scholar | ISI | |
|
Gersten, R., Beckmann, S., Clarke, B., Foegen, A., Marsh, L., Witzel, B. (2009). Assisting students struggling with mathematics: Response to Intervention (RtI) for elementary and middle schools (NCEE 2009-4060). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from https://ies.ed.gov/ncee/wwc/practiceGuide/2 Google Scholar | |
|
Gersten, R., Clarke, B., Jordan, N. C., Newman-Gonchar, R., Haymond, K., Wilkins, C. (2012). Universal screening in mathematics for the primary grades: Beginnings of a research base. Exceptional Children, 78, 423–445. doi:10.1177/001440291207800403 Google Scholar | SAGE Journals | ISI | |
|
Graham, J. W., Olchowski, A. E., Gilreath, T. D. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8, 206–213. doi:10.1007/s11121-007-0070-9 Google Scholar | Crossref | Medline | ISI | |
|
Harper, R., Reeves, B. (1999). Reporting of precision of estimates for diagnostic accuracy: A review. British Medical Journal, 318, 1322–1323. doi:10.1136/bmj.318.7194.1322 Google Scholar | Crossref | Medline | |
|
January, S. A., Ardoin, S. P. (2015). Technical adequacy and acceptability of curriculum-based measurement and the measures of academic progress. Assessment for Effective Intervention, 41, 3–15. doi:10.1177/1534508415579095 Google Scholar | SAGE Journals | |
|
Jenkins, J. R., Hudson, R. F., Johnson, E. S. (2007). Screening for service delivery in an RTI framework: Candidate measures. School Psychology Review, 36, 560–582. Google Scholar | |
|
Kane, M. T. (2013). The argument-based approach to validation. School Psychology Review, 42, 448–457. Google Scholar | |
|
Kilgus, S. P., Methe, S. A., Maggin, D. M., Tomasula, J. L. (2014). Curriculum-based measurement of oral reading (R-CBM): A diagnostic test accuracy meta-analysis of evidence supporting use in universal screening. Journal of School Psychology, 52, 377–405. doi:10.1016/j.jsp.2014.06.002 Google Scholar | Crossref | Medline | ISI | |
|
Klingbeil, D. A., Nelson, P. M., Van Norman, E. R., Birr, C. (2017). Diagnostic accuracy of multivariate universal screening procedures for reading in upper elementary grades. Remedial and Special Education, 35, 308–320. doi:10.1177/0741932517697446 Google Scholar | SAGE Journals | |
|
Leblanc, M., Dufore, E., McDougal, J. (2012). Using general outcome measures to predict student performance on state-mandated assessments: An applied approach for establishing predictive cutscores. Journal of Applied School Psychology, 28, 1–13. doi:10.1080/15377903.2012.643753 Google Scholar | Crossref | |
|
López-Ratón, M., Rodriguez-Álvarez, M. X., Suárez, C. C., Sampedro, F. G. (2014). OptimalCutpoints: An R package for selecting optimal cutpoints in diagnostic tests. Journal of Statistical Software, 61, 1–36. Google Scholar | Crossref | ISI | |
|
Meehl, P. E., Rosen, A. (1955). Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin, 52, 194–216. doi:10.1037/h0048070 Google Scholar | Crossref | Medline | ISI | |
|
National Center for Education Statistics . (2006). School locale definitions. Retrieved from https://nces.ed.gov/surveys/urbaned/definitions.asp Google Scholar | |
|
National Center for Response to Intervention . (2016). Screening tools chart. Retrieved from https://rti4success.org/resources/tools-charts/screening-tools-chart Google Scholar | |
|
Nelson, P. M., Van Norman, E. R., Lackner, S. K. (2016). A comparison of methods to screen middle school students for reading and math difficulties. School Psychology Review, 45, 327–342. Google Scholar | Crossref | ISI | |
|
Nelson, P. M., Van Norman, E. R., VanDerHeyden, A. (2016). Reduce, reuse, recycle: The longitudinal value of local cut scores using state test data. Journal of Psychoeducational Assessment, 35, 683–694. doi:10.1177/0734282916658567 Google Scholar | SAGE Journals | |
|
Nese, J. F., Park, B. J., Alonzo, J., Tindal, G. (2011). Applied curriculum-based measurement as a predictor of high-stakes assessment: Implications for researchers and teachers. The Elementary School Journal, 111, 608–624. doi:10.1086/659034 Google Scholar | Crossref | ISI | |
|
Northwest Evaluation Association . (2011). Technical manual for measures of academic progress (MAP) and measures of academic progress for primary grades (MPG). Portland, OR: Author. Google Scholar | |
|
Northwest Evaluation Association . (2015). Smarter balanced preliminary performance levels. Portland, OR: Author. Google Scholar | |
|
Petscher, Y., Kim, Y.-S., Foorman, B. R. (2011). The importance of predictive power in early screening assessments. Assessment for Effective Intervention, 36, 158–166. doi:10.1177/1534508410396698 Google Scholar | SAGE Journals | |
|
R Core Team . (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Available from http://www.R-project.org/ Google Scholar | |
|
Shapiro, E. S. (2011). Academic skills problems (4th ed.). New York, NY: Guilford Press. Google Scholar | |
|
Shapiro, E. S., Gebhardt, S. N. (2012). Comparing computer-adaptive and curriculum-based measurement methods of assessment. School Psychology Review, 41, 295–305. Google Scholar | |
|
Smarter Balanced Assessment Consortium . (2016). 2014-2015 technical report. Los Angeles, CA: Author. Google Scholar | |
|
Stevenson, N. A., Reed, D. K., Tighe, E. L. (2016). Examining potential bias in screening measures for middle school students by special education and low socioeconomic status subgroups. Psychology in the Schools, 53, 533–547. doi:10.1002/pits.21919 Google Scholar | Crossref | ISI | |
|
Swets, J. A., Dawes, R. M., Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26. Google Scholar | SAGE Journals | |
|
Thum, Y. M., Hauser, C. H. (2015). NWEA 2015 MAP norms for student and school achievement status and growth. Portland, OR: Northwest Evaluation Association. Google Scholar | |
|
VanDerHeyden, A. M. (2011). Technical adequacy of RTI decisions. Exceptional Children, 77, 335–350. doi:10.1177/001440291107700305 Google Scholar | SAGE Journals | ISI | |
|
VanDerHeyden, A. M., Codding, R. S., Martin, R. (2017). Relative value of common screening measures in mathematics. School Psychology Review, 46, 65–87. doi:10.17105/SPR46-1.65-87 Google Scholar | Crossref | |
|
Van Norman, E. R., Nelson, P. M., Klingbeil, D. A. (2017). Single measure and gated screening approaches for identifying students at-risk for academic problems: Implications for sensitivity and specificity. School Psychology Quarterly, 32, 405–413. doi:10.1037/spq0000177 Google Scholar | Crossref | Medline | |
|
Vaughn, S., Fletcher, J. M. (2012). Response to intervention with secondary school students with reading difficulties. Journal of Learning Disabilities, 45, 244–256. doi:10.1177/0022219412442157 Google Scholar | SAGE Journals | ISI | |
|
Wisconsin Department of Public Instruction . (2016). Wisconsin forward exam: Spring 2016 technical report. Madison: Author. Google Scholar | |
|
Zhou, X., Obuchowski, N. A., McClish, D. (2011). Statistical methods in diagnostic medicine. New York, NY: John Wiley. Google Scholar | Crossref |
