![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
The purpose of this study was to extend evaluation of rater accuracy using Direct Behavior Rating–Single-Item Scales (DBR-SIS). Extension of prior research was accomplished through use of criterion ratings derived from both systematic direct observation and expert DBR-SIS scores, and also through control of the durations over which the target behavior was displayed in each clip. Rating targets included academically engaged, disruptive, and respectful behavior. Undergraduate participants (n = 113) viewed video clips of classroom instruction and subsequently rated target students’ behavior using DBR-SIS. Results indicated that both rater bias and rater error were present to some degree based on the behavior target, wording, and duration displayed, with greater bias found for the target involving respectful behavior. In addition, positive wording for academically engaged behavior resulted in improved accuracy, yet negative wording resulted in improved accuracy for disruptive and possibly respectful behavior. Implications for research and practice are discussed.
References
Section:
|
Borman, W. C. (1978). Exploring upper limits of reliability and validity in job performance ratings. Journal of Applied Psychology, 63, 135–144. Google Scholar | Crossref | ISI | |
|
Briesch, A. M., Chafouleas, S. M., Riley-Tillman, T. C. (2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and direct behavior rating. School Psychology Review, 39, 408–421. Google Scholar | ISI | |
|
Chafouleas, S. M. (2011). Direct behavior rating: A review of the issues and research in its development. Education & Treatment of Children, 34, 575–591. Google Scholar | Crossref | |
|
Chafouleas, S. M., Kilgus, S. P., Riley-Tillman, T. C., Jaffery, R., Harrison, S. (2012). Preliminary evaluation of various training components on accuracy of Direct Behavior Ratings. Journal of School Psychology, 50, 317–334. Retrieved from http://dx.doi.org/10.1016/j.jsp.2011.11.007 Google Scholar | |
|
Chafouleas, S. M., Riley-Tillman, T. C., Christ, T. J. (2009). Direct behavior rating (DBR): An emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention, 34, 195–200. Google Scholar | SAGE Journals | |
|
Chafouleas, S. M., Riley-Tillman, T. C., Sugai, G. (2007). School-based behavioral assessment. New York, NY: Guilford. Google Scholar | |
|
Chafouleas, S. M., Volpe, R. J., Gresham, F. M., Cook, C. (2010). School-based behavioral assessment within problem-solving models: Current status and future directions. School Psychology Review, 34, 343–349. Google Scholar | |
|
Chang, L. (1995). Connotatively consistent and reversed connotatively inconsistent items are not fully equivalent: Generalizability study. Educational and Psychological Measurement, 55, 991–997. Google Scholar | SAGE Journals | ISI | |
|
Christ, T. J., Riley-Tillman, T. C., Chafouleas, S. M. (2009). Foundation for the development and use of Direct Behavior Rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention, 34, 201–213. Google Scholar | SAGE Journals | |
|
Christ, T. J., Riley-Tillman, T. C., Chafouleas, S. M., Jaffery, R. (2011). Direct behavior rating (DBR): An evaluation of alternate definitions to assess classroom behaviors. School Psychology Review, 40, 181–199. Google Scholar | ISI | |
|
Edelbrock, C. (1983). Problems and issues in using rating scales to assess child personality and psychopathology. School Psychology Review, 12, 293–299. Google Scholar | |
|
Fuchs, L. S., Deno, S. L. (1991). Paradigmatic distinctions between instructionally relevant measurement models. Exceptional Children, 57, 488–500. Google Scholar | SAGE Journals | ISI | |
|
Glover, T. A., Albers, C. A. (2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117–135. Google Scholar | Crossref | ISI | |
|
Hintze, J. M. (2005). Psychometrics of direct observation. School Psychology Review, 34, 507–519. Google Scholar | ISI | |
|
Jaffery, R., Johnson, A. J., Bowler, M., Harrison, S. E., Riley-Tillman, T. C., Chafouleas, S. M. (2012). Direct behavior rating: Consensus building procedures for establishing expert ratings. Unpublished manuscript. Google Scholar | |
|
Martens, B. K. (1993). Social labeling, precision of measurement, and problem solving: Key issues in the assessment of children’s emotional problems. School Psychology Review, 22, 308–312. Google Scholar | ISI | |
|
McIntyre, R. M., Smith, D. E., Hassett, C. E. (1984). Accuracy of performance ratings as affected by rater training and perceived purpose of rating. Journal of Applied Psychology, 69, 147–156. Google Scholar | Crossref | ISI | |
|
Ory, J. C. (1982). Item placement and wording effects on overall ratings. Educational and Psychological Measurement, 42, 767–775. Google Scholar | SAGE Journals | ISI | |
|
Riley-Tillman, T. C., Chafouleas, S. M., Christ, T. J., Briesch, A. M., LeBel, T. J. (2009). The impact of item wording and behavioral specificity on the accuracy of direct behavior ratings (DBRs). School Psychology Quarterly, 24, 1–12. Google Scholar | Crossref | ISI | |
|
Riley-Tillman, T. C., Chafouleas, S. M., Sassu, K. A., Chanese, J. A. M., Glazer, A. D. (2008). Examining the agreement of direct behavior ratings and systematic direct observation for on-task and disruptive behavior. Journal of Positive Behavior Interventions, 10, 136–143. doi:10.1177/1098300707312542 Google Scholar | SAGE Journals | ISI | |
|
Tapp, J. (2004). MOOSES (Multi-Option Observation System For Experimental Studies). Retrieved from http://kc.vanderbilt.edu/mooses/mooses.html Google Scholar |
