Different Tests, Different Answers: The Stability of Teacher Value-Added Estimates Across Outcome Measures
Abstract
Recently, educational researchers and practitioners have turned to value-added models to evaluate teacher performance. Although value-added estimates depend on the assessment used to measure student achievement, the importance of outcome selection has received scant attention in the literature. Using data from a large, urban school district, I examine whether value-added estimates from three separate reading achievement tests provide similar answers about teacher performance. I find moderate-sized rank correlations, ranging from 0.15 to 0.58, between the estimates derived from different tests. Although the tests vary to some degree in content, scaling, and sample of students, these factors do not explain the differences in teacher effects. Instead, test timing and measurement error contribute substantially to the instability of value-added estimates across tests.
Get full access to this article
View all access and purchase options for this article.
References
Ballou D., Sanders W., Wright P. (2004). Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics, 29(1), 37–65.
Boardman A. E., Murnane R. J. (1979). Using panel data to improve estimates of the determinants of educational achievement. Sociology of Education, 52(2), 113–121.
Boyd D., Grossman P., Lankford H., Loeb S., Wycoff J. (2006). How changes in entry requirements alter the teacher workforce and affect student achievement. Education Finance and Policy, 1(2), 176–216.
Boyd D., Grossman P., Lankford H., Loeb S., Wycoff J. (2007). Who leaves? Teacher attrition and student achievement. Unpublished manuscript.
Briggs D. C., Weeks J. P., Wiley E. (2008). The sensitivity of value-added modeling to the creation of a vertical score scale (Working paper for the National Conference on Value-Added Modeling). Retrieved from http://www.wcer.wisc.edu/news/events/natConf_papers.php
Cantrell S., Fullerton J., Kane T. J., Staiger D. O. (2007). National Board Certification and teacher effectiveness: Evidence from a random assignment experiment. Unpublished manuscript.
Chester M. D. (2003). Multiple measures and high-stakes decisions: A framework for combining measures. Educational Measurement: Issues and Practice, 22(2), 32–41.
Clotfelter C. T., Ladd H. F., Vigdor J. L. (2007). How and why do teacher credentials matter for student achievement? (NBER Working Paper No. 12828). Cambridge, MA: National Bureau of Economic Research.
Croninger R. G., Valli L. (2009). “Where is the action?” Challenges to studying the teaching of reading in elementary classrooms. Educational Researcher, 38(2), 100–108.
Douglas K. M. (2007). A general method for estimating the classification reliability of complex decisions based on configural combinations of multiple assessment scores. Unpublished doctoral dissertation, University of Maryland, College Park, MD.
Gewertz C. (2007). N.Y.C. district issues “value added” grades for schools. Education Week, 27(12), 6.
Goldhaber D., Anthony E. (2007). Can teacher quality be effectively assessed? National Board Certification as a signal of effective teaching. Review of Economics and Statistics, 89(1), 134–150.
Gordon R., Kane T. J., Staiger D. O. (2006). Identifying effective teachers using performance on the job. Washington, DC: Brookings Institution.
Hanushek E. (1971). Teacher characteristics and gains in student achievement: Estimation using micro data. The American Economic Review, 61(2), 280–288.
Harris D. N., Sass T. R. (2006). Value-added models and the measurement of teacher quality. Unpublished manuscript.
Harris D. N., Sass T. R. (2009). The effects of NBPTS-certified teachers on student achievement. Journal of Policy Analysis and Management, 28(1), 55–80.
Jacob B. A., Lefgren L. (2005). Principals as agents: Subjective performance measurement in education (NBER Working Paper No. 11463). Cambridge, MA: National Bureau of Economic Research.
Kane T. J., Rockoff J. E., Staiger D. O. (2008). What does certification tell us about teacher effectiveness? Evidence from New York City. Economics of Education Review, 27(6), 615–631.
Kane T. J., Staiger D. O. (2002). Volatility in school test scores: Implications for test-based accountability systems (Brookings Papers on Education Policy). Washington, DC: Brookings Institution.
Koretz D. M. (2002). Limitations in the use of achievement tests as measures of educators’ productivity. The Journal of Human Resources, 37(4), 752–777.
Lockwood J. R., McCaffrey D. F., Hamilton L. S., Stecher B., Le V., Martinez J. F. (2007). The sensitivity of value-added teacher effect estimates to different mathematics achievement measures. Journal of Educational Measurement, 44(1), 47–67.
McCaffrey D. F., Lockwood J. R., Koretz D. M., Hamilton L. S. (2003). Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND.
McCaffrey D. F., Lockwood J. R., Koretz D., Louis T. A., Hamilton L. (2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29(1), 67.
McCaffrey D. F., Sass T. R., Lockwood J. R. (2008). The intertemporal stability of teacher effect estimates. National Center for Performance Incentives, Working Paper No. 2008-22. Retrieved from http://www.performanceincentives.org/data/files/news/PapersNews/McCaffrey_et_al_2008.pdf
McNeil M. (2009). Starting gun sounds for “Race to the Top.” Education Week, 29(12), 1, 18–19.
Murnane R. J. (1984). Selection and survival in the teacher labor market. The Review of Economics and Statistics, 66(3), 513–518.
Olson L. (2007). Houston board OKs revamped performance-pay plan. Education Week, 27(4), 11.
Raudenbush S. W. (2004). What are value-added models estimating and what does this imply for statistical practice? Journal of Educational and Behavioral Statistics, 29(1), 121–129.
Rivkin S. G., Hanushek E. A., Kain J. F. (2005). Teachers, schools, and academic achievement. Econometrica, 73(2), 417–458.
Rockoff J. E. (2004). The impact of individual teachers on student achievement: Evidence from panel data. The American Economic Review, 94(2), 247–252.
Rogosa D. R., Willett J. B. (1983). Demonstrating the reliability of the difference score in the measurement of change. Journal of Educational Measurement, 20(4), 335–343.
Rothstein J. (2007). Do value-added models add value? Tracking, fixed effects, and causal inference. Unpublished manuscript.
Rubin D. B., Stuart E. A., Zanutto E. L. (2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29(1), 103–116.
Sanders W. (2000). Value-added assessment from student achievement data: Opportunities and hurdles. Journal of Personnel Evaluation in Education, 14(4), 329–339.
Sanders W. L., Horn S. P. (1994). The Tennessee Value-Added Assessment System (TVAAS): Mixed-model methodology in educational assessment. Journal of Personnel Evaluation in Education, 8(3), 299–311.
Sass T. R. (2008). The stability of value-added measures of teacher quality and implications for teacher compensation policy. National Center for Analysis of Longitudinal Data in Education Research Brief No. 4. Retrieved from http://www.urban.org/UploadedPDF/1001266_stabilityofvalue.pdf
Tekwe C. D., Carter R. L., Ma C., Algina J., Lucas M. E., Roth J., et al. (2004). An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics, 29(1), 11–35.
Todd P. E., Wolpin K. I. (2003). On the specification and estimation of the production function for cognitive achievement. Economic Journal, 113(485), F3–33.
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: February 1, 2011
Issue published: February 2011
Keywords
Authors
Metrics and citations
Metrics
Journals metrics
This article was published in American Educational Research Journal.
View All Journal MetricsArticle usage*
Total views and downloads: 1475
*Article usage tracking started in December 2016
Altmetric
See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 116 view articles Opens in new tab
Crossref: 119
- The Palgrave Handbook of Teacher Education Research
- International Encyclopedia of Education(Fourth Edition)
- International Encyclopedia of Education(Fourth Edition)
- International Encyclopedia of Education(Fourth Edition)
- Value-added scores show limited stability over time in primary school
- The Palgrave Handbook of Teacher Education Research
- Variation, context, and inequality: comparing models of school effectiveness in two states in India
- School quality matters: A multilevel analysis of school effects on the early reading achievement of Black girls
- The Palgrave Handbook of Teacher Education Research
- View More
Figures and tables
Figures & Media
Tables
View Options
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
I am signed in as:
View my profileSign out
I can access personal subscriptions, purchases, paired institutional access and free tools such as favourite journals, email alerts and saved searches.
loading institutional access options
AERA members can access this journal content using society membership credentials.
AERA members can access this journal content using society membership credentials.
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
