![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Classroom observations increasingly inform high-stakes decisions and research in education, including the allocation of school funding and the evaluation of school-based interventions. However, trends in rater scoring tendencies over time may undermine the reliability of classroom observations. Accordingly, the present investigations, grounded in social psychology research on emotion and judgment, propose that state emotion may constitute a source of psychological bias in raters’ classroom observations. In two studies, employing independent sets of raters and approximately 5,000 videotaped fifth- and sixth-grade classroom interactions, within-rater state positive emotion was associated with favorable ratings of classroom quality using the Classroom Assessment Scoring System (CLASS). Despite various protections enacted to secure reliable and valid observations in the face of rater trends—including professional training, certification testing, and routine calibration meetings—emotional bias still emerged. Study limitations and implications for classroom observation methodology are considered.
References
Section:
|
Baker, B. D., Oluwole, J., Green, P. C. (2013). The legal consequences of mandating high-stakes decisions based on low quality information: Teacher evaluation in the race-to-the-top era. Education Policy Analysis Archives, 21, 1-71. doi:10.14507/epaa.v21n5.2013 Google Scholar | Crossref | |
|
Baron, R. A. (1987). Interviewer’s moods and reactions to job applicants: The influence of affective states on applied social judgments. Journal of Applied Social Psychology, 17, 911-926. doi:10.1111/j.1559-1816.1987.tb00298.x Google Scholar | Crossref | ISI | |
|
Baron, R. A. (1993). Interviewers’ moods and evaluations of job applicants: The role of applicant qualifications. Journal of Applied Social Psychology, 23, 253-271. doi:10.1111/j.1559-1816.1993.tb01086.x Google Scholar | Crossref | ISI | |
|
Brackett, M. A., Floman, J. L., Ashton-James, C., Cherkasskiy, L., Salovey, P. (2013). The influence of teacher emotion on grading practices: A preliminary look at the evaluation of student writing. Teachers and Teaching: Theory and Practice, 19, 634-646. doi:10.1080/13540602.2013.827453 Google Scholar | Crossref | ISI | |
|
Brennan, R. L. (2001). Generalizability theory: Statistics for social science and public policy. New York, NY: Springer-Verlag. Google Scholar | Crossref | |
|
Casabianca, J. M., Lockwood, J. R., McCaffrey, D. F. (2015). Trends in classroom observation scores. Educational and Psychological Measurement, 75, 311-337. doi:10.1177/0013164414539163 Google Scholar | SAGE Journals | ISI | |
|
Casabianca, J. M., McCaffrey, D. F., Gitomer, D. H., Bell, C. A., Hamre, B. K., Pianta, R. C. (2013). Effect of observation mode on measures of secondary mathematics teaching. Educational and Psychological Measurement, 73, 757-783. doi:10.1177/0013164413486987 Google Scholar | SAGE Journals | ISI | |
|
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159. doi:10.1037/0033-2909.112.1.155 Google Scholar | Crossref | Medline | ISI | |
|
Crawford, J. R., Henry, J. D. (2004). The Positive and Negative Affect Schedule (PANAS): Construct validity, measurement properties and normative data in a large non-clinical sample. British Journal of Clinical Psychology, 43, 245-265. doi:10.1348/0144665031752934 Google Scholar | Crossref | Medline | ISI | |
|
Cronbach, L. J., Gleser, G. C., Nanda, H., Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: John Wiley. Google Scholar | |
|
Diener, E., Kanazawa, S., Suh, E. M., Oishi, S. (2014). Why people are in a generally good mood? Personality and Social Psychology Review, 4, 1-22. doi:10.1177/1088868314544467 Google Scholar | SAGE Journals | ISI | |
|
Englich, B., Soder, K. (2009). Moody experts—How mood and expertise influence judgmental anchoring. Judgment and Decision Making, 4, 41-50. Google Scholar | ISI | |
|
Forgas, J. P. (2014). On the regulatory functions of mood: Affective influences on memory, judgments and behavior. In Forgas, J. P., Harmon-Jones, E. (Eds.), Motivation and its regulation: The control within (pp. 169-192). Sussex, UK: Psychology Press. Google Scholar | |
|
Forgas, J. P., Eich, E. (2012). Affective influences on cognition: Mood congruence, mood dependence, and mood effects on processing strategies. In Healy, A. F., Proctor, R. W. (Eds.), Handbook of psychology: Experimental psychology (Vol. 4, pp. 61-82). New York, NY: Wiley. Google Scholar | |
|
Forgas, J. P., George, J. M. (2001). Affective influences on judgments and behavior in organizations: An information processing perspective. Organizational Behavior and Human Decision Processes, 86, 3-34. doi:10.1006/obhd.2001.2971 Google Scholar | Crossref | ISI | |
|
Fromme, K., Corbin, W. R., Kruse, M. I. (2008). Behavioral risks during the transition from high school to college. Developmental Psychology, 44, 1497-1504. doi:10.1037/a0012614 Google Scholar | Crossref | Medline | ISI | |
|
Greifeneder, R., Bless, H., Pham, M. T. (2011). When do people rely on affective and cognitive feelings in judgment? A review. Personality and Social Psychology Review, 15, 107-141. doi:10.1177/1088868310367640 Google Scholar | SAGE Journals | ISI | |
|
Hafen, C. A., Allen, J. P., Mikami, A. Y., Gregory, A., Hamre, B., Pianta, R. C. (2012). The pivotal role of adolescent autonomy in secondary school classrooms. Journal of Youth and Adolescence, 41, 245-255. doi:10.1007/s10964-011-9739-2 Google Scholar | Crossref | Medline | ISI | |
|
Hagelskamp, C., Brackett, M. A., Rivers, S. E., Salovey, P. (2013). Improving classroom quality with the RULER Approach to Social and Emotional Learning: Proximal and distal outcomes. American Journal of Community Psychology, 51, 530-543. doi:10.1007/s10464-013-9570-x Google Scholar | Crossref | Medline | ISI | |
|
Hamre, B. K., Goffin, S. G., Kraft-Sayre, M. K. (2009). Classroom Assessment Scoring System implementation guide: Measuring and improving classroom interactions in early childhood settings. Charlottesville, VA: Teachstone. Google Scholar | |
|
Hamre, B. K., Pianta, R. C., Downer, J. T., DeCoster, J., Mashburn, A. J., Jones, S. M., Hamagami, A. (2013). Teaching through interactions: Testing a developmental framework of teacher effectiveness in over 4,000 classrooms. The Elementary School Journal, 113, 461-487. doi:10.1086/669616 Google Scholar | Crossref | ISI | |
|
Hamre, B. K., Pianta, R. C., Mashburn, A. J., Downer, J. T. (2007). Building a science of classrooms: Application of the CLASS framework in over 4,000 US early childhood and elementary classrooms. New York, NY: Foundation for Childhood Development. Retrieved from http://fcd-us.org/sites/default/files/BuildingAScienceOfClassroomsPiantaHamre.pdf Google Scholar | |
|
Hintze, J. M., Matthews, W. J. (2004). The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral observation. School Psychology Review, 33, 258-270. Google Scholar | ISI | |
|
Isbell, L. M., Lair, E. C. (2013). Moods, emotions, and evaluations as information. In Carlston, D. (Ed.), The Oxford handbook of social cognition (pp. 435-462). New York, NY: Oxford University Press. Google Scholar | Crossref | |
|
Kane, T. J., Cantrell, S. (2013). Ensuring fair and reliable measures of effective teaching: Culminating findings from the MET Project’s three-year study. Seattle, WA: Bill & Melinda Gates Foundation. Retrieved from http://metproject.org/downloads/MET_Ensuring_Fair_and_Reliable_Measures_Practitioner_Brief.pdf Google Scholar | |
|
McFarland, C., White, K., Newth, S. (2003). Mood acknowledgment and correction for the mood-congruency bias in social judgment. Journal of Experimental Social Psychology, 39, 483-491. doi:10.1016/S0022-1031(03)00025-8 Google Scholar | Crossref | ISI | |
|
Oishi, S., Lun, J., Sherman, G. D. (2007). Residential mobility, self-concept, and positive affect in social interactions. Journal of Personality and Social Psychology, 93, 131-141. doi:10.1037/0022-3514.93.1.131 Google Scholar | Crossref | Medline | ISI | |
|
Pashler, H., Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528-530. doi:10.1177/1745691612465253 Google Scholar | SAGE Journals | ISI | |
|
Peugh, J. L. (2010). A practical guide to multilevel modeling. Journal of School Psychology, 48, 85-112. doi:10.1016/j.jsp.2009.09.002aco Google Scholar | Crossref | Medline | ISI | |
|
Pianta, R. C. (2012). Implementing observation protocols: Lessons for K-12 education from the field of early childhood. Washington, DC: Center for American Progress. Retrieved from http://cdn.americanprogress.org/wp-content/uploads/issues/2012/05/pdf/observation_protocols.pdf Google Scholar | |
|
Pianta, R. C., Hamre, B. K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38, 109-119. doi:10.3102/0013189X09332374 Google Scholar | SAGE Journals | ISI | |
|
Pianta, R. C., La Paro, K. M., Hamre, B. K. (2008). Classroom Assessment Scoring System: Manual, K-3. Baltimore, MD: Paul H. Brookes. Google Scholar | |
|
Redelmeier, D. A., Baxter, S. D. (2009). Rainy weather and medical school admission interviews. Canadian Medical Association Journal, 181, 933. doi:10.1503/cmaj.091546 Google Scholar | Crossref | ISI | |
|
Roberts, J. K., Monaco, J. P. (2006, April). Effect size measures for the two-level linear multilevel model. Paper presented at the annual conference of the American Educational Research Association, San Francisco, CA. Google Scholar | |
|
Rosnow, R. L., Rosenthal, R., Rubin, D. B. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. New York, NY: Cambridge University Press. Google Scholar | |
|
Rudasill, K. M., Gallagher, K. C., White, J. M. (2010). Temperamental attention and activity, classroom emotional support, and academic achievement in third grade. Journal of School Psychology, 48, 113-134. doi:10.1016/j.jsp.2009.11.002 Google Scholar | Crossref | Medline | ISI | |
|
Schwarz, N., Clore, G. L. (1983). Mood, misattribution and judgments of well-being: Informative and directive functions of affective states. Journal of Personality and Social Psychology, 45, 513-523. doi:10.1037//0022-3514.45.3.513 Google Scholar | Crossref | ISI | |
|
Schwarz, N., Clore, G. L. (2003). Mood as information: 20 years later. Psychological Inquiry, 14, 296-303. doi:10.1080/1047840X.2003.9682896 Google Scholar | Crossref | ISI | |
|
Watson, D., Clark, L. A., Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54, 1063-1070. doi:10.1037/0022-3514.54.6.1063 Google Scholar | Crossref | Medline | ISI | |
|
Watson, D., Walker, L. M. (1996). The long-term stability and predictive validity of trait measures of affect. Journal of Personality and Social Psychology, 70, 567-577. doi:10.1037/0022-3514.70.3.567 Google Scholar | Crossref | Medline | ISI | |
|
Whitehurst, G. J., Chingos, M. M., Lindquist, K. M. (2014). Evaluating teachers with classroom observations: Lessons learned in four districts. Washington, DC: The Brookings Institute. Retrieved from http://www.brookings.edu/~/media/research/files/reports/2014/05/13%20teacher%20evaluation/ Google Scholar |
