![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Data assessment is critical for determining student behavior change in response to individualized behavior interventions in schools. This study examined the interrater agreement of the Individualized Behavior Rating Scale Tool (IBRST), a perceptual direct behavior rating tool that was used by typical school personnel to record behavior occurrence in students requiring individualized interventions. Two independent observers (teacher and data collector) used the IBRST to rate student-specific problem and appropriate behaviors during specified observation times. Data were collected across 19 students and agreement between raters was compared. Resulting linear- and quadratic-weighted kappa coefficients indicated generally adequate agreement between raters on problem behaviors and appropriate behaviors. When ratings were categorized into more or less salient behaviors, less than adequate agreement (<.60) was found for some behaviors that were less salient. Agreement remained stable from baseline to intervention. Implications for practice, limitations of the study, and directions for future research are discussed.
References
Section:
|
Batsche, G., Elliott, J., Graden, J., Grimes, J., Kovaleski, J. F., Prasse, D., Tilly, D. W.(2005). Response to intervention: Policy considerations and implementation. Alexandria, VA: National Association of State Directors of Special Education. Google Scholar | |
|
Brenner, H., Kliebsch, U. (1996). Dependence of weighted Kappa coefficients on the number of categories. Epidemiology, 7, 199–202. Google Scholar | Crossref | Medline | ISI | |
|
Burke, M. D., Vannest, K., Davis, J., Davis, C., Parker, R. (2009). Reliability of frequent retrospective behavior ratings for elementary school students with EBD. Behavioral Disorders, 34, 212–222. Google Scholar | ISI | |
|
Chafouleas, S. M., Christ, T. J., Riley-Tillman, T. C., Briesch, A. M., Chanese, J. A. M. (2007). Generalizability and dependability of Direct Behavior Ratings to assess social behavior of preschoolers. School Psychology Review, 36, 63–79. Google Scholar | ISI | |
|
Chafouleas, S. M., Kilgus, S. P., Riley-Tillman, T. C., Jaffery, R., Harrison, S. (2012). Preliminary evaluation of various training components on accuracy of direct behavior ratings. Journal of School Psychology, 50, 317–334. Google Scholar | Crossref | Medline | ISI | |
|
Chafouleas, S. M., McDougal, J. L., Riley-Tillman, T. C., Panahon, C. J., Hilt, A. M. (2005). What do daily behavior report cards (DBRCs) measure? An initial comparison of DBRCs with direct observation for off-task behavior. Psychology in the Schools, 42, 669–676. doi:10.1002/pits.20102 Google Scholar | Crossref | ISI | |
|
Chafouleas, S. M., Riley-Tillman, T. C., Christ, T. J. (2009). Direct behavior rating (DBR): An emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention, 34, 195–200. Google Scholar | SAGE Journals | |
|
Chafouleas, S. M., Riley-Tillman, T. C., McDougal, J. L. (2002). Good, bad, or in-between: How does the daily behavior report card rate? Psychology in the Schools, 39, 157–169. doi:10.1002/pits.10027 Google Scholar | Crossref | ISI | |
|
Chafouleas, S. M., Riley-Tillman, T. C., Sassu, K. A. (2006). Acceptability and reported use of daily behavior report cards among teachers. Journal of Positive Behavior Interventions, 8, 174–182. Google Scholar | SAGE Journals | ISI | |
|
Chafouleas, S. M., Sanetti, L. M. H., Jaffery, R., Fallon, L. M. (2012). An evaluation of a classwide intervention package involving self-management and a group contingency on classroom behavior of middle school students. Journal of Behavior Education, 21, 34–57. Google Scholar | Crossref | |
|
Chafouleas, S. M., Volpe, R. J., Gresham, F. M., Cook, C. (2010). School-based behavioral assessment within problem-solving models: Current status and future directions. School Psychology Review, 34, 343–349. Google Scholar | |
|
Christ, T. J., Riley-Tillman, T. C., Chafouleas, S., Jaffery, R. (2011). Direct behavior rating: An evaluation of alternate definitions to assess classroom behaviors. School Psychology Review, 40, 181–199. Google Scholar | ISI | |
|
Cicchetti, D., Bronen, R., Spencer, S., Haut, S., Berg, A., Oliver, P., Tyrer, P. (2006). Rating scales, scales of measurement, issues of reliability: Resolving some critical issues for clinicians and researchers. The Journal of Nervous and Mental Disease, 194, 557-564. Google Scholar | Crossref | Medline | ISI | |
|
Cicchetti, D. V., Allison, T. (1971). A new procedure for assessing reliability of scoring EEG sleep recordings. American Journal of EEG Technology, 11, 101–110. Google Scholar | Crossref | |
|
Cicchetti, D. V., Sparrow, S. S. (1981). Developing criteria for establishing inter-rater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, 127–137. Google Scholar | Medline | |
|
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provisions for scaled disagreement or partial credit. Psychosocial Bulletin, 70, 213–220. Google Scholar | Crossref | Medline | ISI | |
|
Cooper, J. O., Heron, T. E., Heward, W. L. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Pearson. Google Scholar | |
|
Crone, D. A., Hawken, L. S., Horner, R. H. (2010). Responding to problem behavior in schools: The behavior education program (2nd ed.). New York, NY: Guilford. Google Scholar | |
|
Crone, D. A., Horner, R. H. (2003). Building positive behavior support systems in schools: Functional behavioral assessment. New York, NY: Guilford. Google Scholar | |
|
Deno, S. L. (1989). Curriculum-based measurement and alternative special education services: A fundamental and direct relationship. In Shinn, M. R. (Ed.), Curriculum-based measurement: Assessing special children (pp. 1–17). New York, NY: Guilford. Google Scholar | |
|
Deno, S. L. (1995). School psychologist as problem solver. In Thomas, A., Grimes, J. (Eds.), Best practices in school psychology (Vol. 3, pp. 471–484). Washington, DC: National Association of School Psychologists. Google Scholar | |
|
Dunlap, G., Iovannone, R., Kincaid, D., Wilson, K., Christiansen, K., Strain, P., English, C. (2010). Prevent-teach-reinforce: A school-based model of individualized positive behavior support. Baltimore, MD: Paul H. Brookes. Google Scholar | |
|
Dwass, M. (1957). Modified randomization tests for nonparametric hypotheses. Annals of Mathematical Statistics, 28, 181–187. Google Scholar | Crossref | |
|
Fleiss, J. L., Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613–619. Google Scholar | SAGE Journals | ISI | |
|
Gall, M. D., Gall, J. P., Borg, W. R. (2007). Educational research: An introduction (8th ed.). Boston, MA: Pearson. Google Scholar | |
|
Graham, P., Jackson, R. (1993). The analysis of ordinal agreement data: Beyond weighted kappa. Journal of Clinical Epidemiology, 48, 1055–1062. Google Scholar | Crossref | ISI | |
|
Gresham, F. M. (2003). Establishing the technical adequacy of functional behavioral assessment: Conceptual and measurement challenges. Behavioral Disorders, 28, 282–298. Google Scholar | SAGE Journals | ISI | |
|
Gresham, F. M. (2004). Current status and future directions of school-based behavioral interventions. School Psychology Review, 33, 326–343. Google Scholar | ISI | |
|
Hintze, J. M., Matthews, W. J. (2004). The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral observation. School Psychology Review, 33, 258–270. Google Scholar | ISI | |
|
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179. Google Scholar | SAGE Journals | ISI | |
|
Iovannone, R., Greenbaum, P. E., Wang, W., Kincaid, D., Dunlap, G., Strain, P. (2009). Randomized controlled trial of a tertiary behavior intervention for students with problem behaviors: Preliminary outcomes. Journal of Emotional and Behavioral Disorders, 17, 213–225. Google Scholar | SAGE Journals | ISI | |
|
Jones, C., Caravaca, L., Cizek, S., Horner, R. H., Vincent, C. G. (2006). Culturally responsive schoolwide positive behavior support: A case study in one school with a high proportion of Native American students. Multiple Voices, 9, 108–119. Google Scholar | |
|
Landis, J. R., Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. Google Scholar | Crossref | Medline | ISI | |
|
Reimers, T., Wacker, D. (1988). Parents’ ratings of the acceptability of behavioral treatment recommendations made in an outpatient clinic: A preliminary analysis of the influence of treatment effectiveness. Behavioral Disorders, 14, 7–15. Google Scholar | SAGE Journals | |
|
Repp, A. C., Horner, R. H. (Eds.). (1999). Functional analysis of problem behavior: From effective assessment to effective support. Belmont, CA: Wadsworth. Google Scholar | |
|
Riley-Tillman, T. C., Chafouleas, S. M., Briesch, A. M. (2007). A school practitioner’s guide to using Daily Behavior Report Cards to monitor interventions. Psychology in the Schools, 44, 77–89. Google Scholar | Crossref | ISI | |
|
Riley-Tillman, T. C., Chafouleas, S. M., Briesch, A. M., Eckert, T. L. (2008). Daily behavior report cards and systematic direct observation: An investigation of the acceptability, reported training and use, and decision reliability among school psychologists. Journal of Behavioral Education, 17, 313–327. doi:10.1007/s10864-008-0070-5 Google Scholar | Crossref | |
|
Riley-Tillman, T. C., Methe, S. A., Weegar, K. (2009). Examining the use of direct behavior rating methodology on classwide formative assessment: A case study. Assessment for Effective Intervention, 34, 242–250. Google Scholar | SAGE Journals | |
|
Salvia, J., Ysseldyke, J. E., Bolt, S. (2007). Assessment (10th ed.). Houston, TX: Houghton Mifflin. Google Scholar | |
|
Schlientz, M. D., Riley-Tillman, T. C., Briesch, A. M., Walcott, C. M., Chafouleas, S. M. (2009). The impact of training on the accuracy of direct behavior ratings (DBR). School Psychology Quarter, 24, 73–83. Google Scholar | Crossref | ISI | |
|
Schuster, C. (2004). A note on the interpretation of weighted kappa and its relation to other rater agreement statistics for metric scales. Educational and Psychological Measurement, 64, 243–253. Google Scholar | SAGE Journals | ISI | |
|
Shapiro, E. S., Eckert, T. L. (1994). Acceptability of curriculum-based assessment by school psychologists. Journal of School Psychology, 32, 167–183. Google Scholar | Crossref | ISI | |
|
Shapiro, E. S., Heick, P. F. (2004). School psychologist assessment practices in the evaluation of students referred for social/behavioral/emotional problems. Psychology in the Schools, 41, 551–561. Google Scholar | Crossref | ISI | |
|
Shinn, M. R. (1989). Curriculum-based measurement: Assessing special children. New York, NY: Guilford. Google Scholar | |
|
Steege, M. W., Davin, T., Hathaway, M. (2001).Reliability and accuracy of a performance-based behavioral recording procedure. School Psychology Review, 30, 252–261. Google Scholar | ISI | |
|
Vanbelle, S., Albert, A. (2009). A note on the linearly weighted kappa coefficient for ordinal scales. Statistical Methodology, 6, 157–163. Google Scholar | Crossref | |
|
Vaughn, S., Linan-Thompson, S., Hickman, P. (2003). Response to instruction as a means of identifying students with reading/learning disabilities. Exceptional Children, 30, 16–19. Google Scholar | |
|
Walker, H. M., Severson, H. H. (1992). Systematic screening for behavior disorders (SSBD): User’s guide and technical manual. Longmont, CO: Sopris West. Google Scholar |
