![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
To evaluate students’ responsiveness to an intervention, both student outcome and implementer treatment integrity data are needed. Teachers are often asked to self-report treatment integrity data. However, when self-report responses are compared with those from a direct observer, it is apparent that teachers commonly overestimate the extent to which an intervention was implemented as planned. As such, more research related to teacher self-report to assess treatment integrity is needed. The objective of this preliminary single-case multiple-baseline design study was to improve interrater agreement between observers’ and teachers’ self-report ratings of treatment integrity by providing teachers with comprehensive, direct training (including an intervention description, modeling, practice, and feedback). Results indicate that after this training, agreement between observers’ and teachers’ ratings of treatment integrity improved.
Keywords treatment integrity, self-report, interrater agreement
References
Section:
|
Algozzine, B., Wang, B., Violette, A. S. (2011). Reexamining the relationship between academic achievement and social behavior. Journal of Positive Behavior Interventions, 13, 3–16. doi:10.1177/1098300709359084 Google Scholar | SAGE Journals | ISI | |
|
Arnold, H. J., Feldman, D. C. (1981). Social desirability response bias in self-report choice situations. Academy of Management Journal, 24, 377–385. doi:10.2307/255848 Google Scholar | Crossref | ISI | |
|
Baranowski, T. (1988). Validity and reliability of self-report measures of physical activity: An information processing perspective. Research Quarterly for Exercise and Sport, 59, 314–327. doi:10.1080/02701367.1988.10609379 Google Scholar | Crossref | ISI | |
|
Berk, R. A. (1979). Generalizability of behavioral observations: A clarification of interobserver agreement and interobserver reliability. American Journal of Mental Deficiency, 83, 460–472. doi:10.1037/0033-2909.86.2.376 Google Scholar | Crossref | Medline | |
|
Biggs, B. K., Vernberg, E. M., Twemlow, S. W., Fonagy, P., Dill, E. J. (2008). Teacher adherence and its relation to teacher attitudes and student outcomes in an elementary school-based violence prevention program. School Psychology Review, 37, 533–549. Google Scholar | ISI | |
|
Bruhn, A., Barron, S., Fernando, J., Balint-Langel, K. (2017). Extending the direct behavior rating: An examination of schoolwide behavior ratings and academic engagement. Journal of Positive Behavior Interventions. Advance online publication. doi:10.1177/1098300717713441 Google Scholar | SAGE Journals | |
|
Chafouleas, S. M. (2011). Direct Behavior Rating: A review of the issues and research in its development. Education and Treatment of Children, 34, 575–591. Google Scholar | Crossref | |
|
Chafouleas, S. M., Briesch, A. M., Riley-Tillman, T. C., McCoach, D. B. (2009). Moving beyond assessment of treatment acceptability: An examination of the factor structure of the Usage Rating Profile–Intervention (URP-I). School Psychology Quarterly, 24, 36–47. Google Scholar | Crossref | ISI | |
|
Chafouleas, S. M., Kilgus, S. P., Riley-Tillman, T. C., Jaffery, R., Harrison, S. (2012). Preliminary evaluation of various training components on accuracy of direct behavior ratings. Journal of School Psychology, 50, 317–334. Google Scholar | Crossref | Medline | ISI | |
|
Chafouleas, S. M., Riley-Tillman, T. C., Sugai, G. (2007). School-based behavior assessment and monitoring for informing instruction and intervention. New York, NY: Guilford. Google Scholar | |
|
Chafouleas, S. M., Sanetti, L. M. H., Jaffery, R., Fallon, L. M. (2012). An evaluation of a class-wide intervention package involving self-management and group contingency on classroom behavior of middle school students. Journal of Behavioral Education, 21, 34–57. Google Scholar | Crossref | |
|
Cochrane, W. S., Laux, J. M. (2008). A survey investigating school psychologists’ measurement of treatment integrity in school-based interventions and their beliefs about its importance. Psychology in the Schools, 46, 499–507. Google Scholar | Crossref | |
|
Dane, A. V., Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45. doi:10.1016/s0272-7358(97)00043-3 Google Scholar | Crossref | Medline | ISI | |
|
Dufrene, B. A., Lestremau, L., Zoder-Martell, K. (2014). Direct behavioral consultation: Effects on teachers’ praise and student disruptive behavior. Psychology in the Schools, 51, 567–580. Google Scholar | Crossref | |
|
Dufrene, B. A., Parker, K. M., Menousek, K., Zhou, Q., Harpole, L. L., Olmi, D. J. (2012). Direct behavioral consultation in Head Start to improve teacher use of praise and effective instruction delivery. Journal of Educational and Psychological Consultation, 22, 159–186. doi:10.1080/10474412.2011.620817 Google Scholar | Crossref | |
|
Durlak, J. A., DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. Google Scholar | Crossref | Medline | ISI | |
|
Ellis, E. S., Deshler, D. D., Lenz, B. K., Schumaker, J. B., Clark, F. L. (1991). An instructional model for teaching learning strategies. Focus on Exceptional Children, 23(6), 1–24. Google Scholar | Crossref | |
|
Erchul, W. P., Martens, B. K. (2010). School consultation: Conceptual and empirical bases and practices (3rd ed.). New York, NY: Kluwer Academic. Google Scholar | Crossref | |
|
Ermeling, B. A. (2010). Tracing the effects of teacher inquiry on classroom practice. Teaching and Teacher Education, 26, 377–388. doi:10.1016/j.tate.2009.02.019 Google Scholar | Crossref | ISI | |
|
Fallon, L. M., Kurtz, K. D., Mueller, M. R. (2017). Direct training to improve educators’ treatment integrity: A systematic review of single-case design studies. School Psychology Quarterly. http://dx.doi.org/10.1037/spq0000210 Google Scholar | |
|
Faggella-Luby, M., Graner, P. S. (2010). Literacy supports for adolescent struggling readers: Taking action through comprehension instruction. In Scruggs, T. E., Mastropieri, M. A. (Eds.), Advances in learning and behavioral disabilities (Vol. 23, pp. 61–94). Bingley, UK: Emerald Group. Google Scholar | Crossref | |
|
Fisher, R. J. (1993). Social desirability bias and the validity of indirect questioning. Journal of Consumer Research, 20, 303–315. doi:10.1086/209351 Google Scholar | Crossref | ISI | |
|
Fuchs, L. S., Fuchs, D. (1985). Effectiveness of systematic formative evaluation: A meta-analysis. Exceptional Children, 53, 199–208. doi:10.1177/001440298605300301 Google Scholar | SAGE Journals | |
|
Gresham, F. M. (1989). Assessment of treatment integrity in school consultation and prereferral intervention. School Psychology Review, 18, 37–50. Google Scholar | ISI | |
|
Gresham, F. M., MacMillan, D., Beebe-Frankenberger, M. E., Bocian, K. M. (2000). Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented? Learning Disabilities Research and Practice, 15, 198–205. doi:10.1207/sldrp1504_4 Google Scholar | Crossref | |
|
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179. doi:10.1177/001440290507100203 Google Scholar | SAGE Journals | ISI | |
|
Jaffery, R., Johnson, A. J., Bowler, M. C., Riley-Tillman, T. C., Chafouleas, S. M., Harrison, S. E. (2015). Using consensus building procedures with expert raters to establish comparison scores of behavior for direct behavior rating. Assessment for Effective Intervention, 40, 195–204. doi:10.1177/1534508415569527 Google Scholar | SAGE Journals | |
|
Jones, H. A., Clarke, A. T., Power, T. J. (2008). Expanding the concept of intervention integrity: A multidimensional model of participant engagement. In Balance, 23, 4–5. doi:10.1037/e597762009-004 Google Scholar | Crossref | |
|
King-Sears, M. E., Cummings, C. S. (1996). Inclusive practices of classroom teachers. Remedial and Special Education, 17, 217–225. doi:10.1177/074193259601700404 Google Scholar | SAGE Journals | ISI | |
|
Kline, F. M., Schumaker, J. B., Deshler, D. D. (1991). The development and validation of feedback routines for instructing students with learning disabilities. Learning Disability Quarterly, 14, 191–207. doi:10.2307/1510849 Google Scholar | SAGE Journals | ISI | |
|
Kratochwill, T. R., Bergan, J. R. (1990). Behavioral consultation in applied settings: An individual guide. New York, NY: Plenum. Google Scholar | |
|
Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., Shadish, W. R. (2010). Single-case designs technical documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf Google Scholar | |
|
Kratochwill, T. R., Levin, J. R. (2010). Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods, 15, 124–144. doi:10.1037/a0017736 Google Scholar | Crossref | Medline | ISI | |
|
Lenz, B. K., Alley, G. R., Schumaker, J. B. (1987). Activating the inactive learner: Advance organizers in the secondary content classroom. Learning Disability Quarterly, 10, 53–67. doi:10.2307/1510755 Google Scholar | SAGE Journals | ISI | |
|
MacArthur, C. A., Schwartz, S. S., Graham, S., Molloy, D., Harris, K. R. (1996). Integration of strategy instruction into a whole language classroom. Learning Disabilities Research and Practice, 11, 168–176. Google Scholar | |
|
McGivern, J. E., Walter, M. J. (2014). Legal and ethical issues related to treatment integrity in psychology and education. In Sanetti, L. M. H., Kratochwill, T. R. (Eds.), Treatment integrity: A foundation for evidence-based practice in applied psychology (pp. 229–254). Washington, DC: American Psychological Association. Google Scholar | Crossref | |
|
McIntyre, L. L., Gresham, F. M., DiGennaro, F. D., Reed, D. D. (2007). Treatment integrity of school-based interventions with children in the Journal of Applied Behavior Analysis 1991–2005. Journal of Applied Behavior Analysis, 40, 659–672. Google Scholar | Crossref | Medline | ISI | |
|
McLeod, B. D., Southam-Gerow, M. A., Weisz, J. R. (2009). Conceptual and methodological issues in treatment integrity measurement. School Psychology Review, 38, 541–546. Google Scholar | ISI | |
|
Nederhof, A. J. (1985). Methods of coping with social desirability bias: A review. European Journal of Social Psychology, 15, 263–280. Google Scholar | Crossref | ISI | |
|
Neely, L., Davis, H., Davis, J., Rispoli, M. (2015). Review of reliability and treatment integrity trends in autism-focused research. Research in Autism Spectrum Disorders, 9, 1–12. Google Scholar | Crossref | |
|
Noell, G. H. (2008). Research examining the relationships among consultation process, treatment integrity, and outcomes. In Erchul, W. P., Sheridan, S. M. (Eds.), Handbook of research in school consultation: Empirical foundations for the field (pp. 315–334). Mahwah, NJ: Erlbaum Group. Google Scholar | |
|
Noell, G. H., Gansle, K. A. (2014). Research examining the relationships among consultation process, treatment integrity, and outcomes. In Erchul, W. P., Sheridan, S. M. (Eds.), Handbook of research in school consultation: Empirical foundations for the field (2nd ed., pp. 386–408). New York, NY: Routledge. Google Scholar | |
|
Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., Williams, K. L., . . .Duhon, G. J. (2005). Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 34, 87–106. Google Scholar | ISI | |
|
Palmer, S. B., Wehmeyer, M. L., Gipson, K., Agran, M. (2004). Promoting access to the general curriculum by teaching self-determination skills. Exceptional Children, 70, 407–420. Google Scholar | SAGE Journals | ISI | |
|
Parker, R. I., Vannest, K. J., Davis, J. L., Sauber, S. B. (2010). Combining non-overlap and trend for single case research: Tau-U. Behavior Therapy, 42, 284–299. Google Scholar | Crossref | |
|
Sallis, J. F., Saelens, B. E. (2000). Assessment of physical activity by self-report: Status, limitations, and future directions. Research Quarterly for Exercise and Sport, 71, 1–14. Google Scholar | Crossref | Medline | ISI | |
|
Sanetti, L. M. H., Chafouleas, S. M., Christ, T. J., Gritter, K. L. (2009). Extending use of direct behavior rating beyond student assessment: Applications to treatment integrity assessment within a multi-tier model of school-based intervention delivery. Assessment for Effective Intervention, 34, 251–258. doi:10.1007/s10864-013-9179-z Google Scholar | SAGE Journals | |
|
Sanetti, L. M. H., Collier-Meek, M. A. (2017). Treatment integrity: Evidence-based interventions in applied settings. In Theodore, L. (Ed.), Handbook of evidence-based interventions for children and adolescents (pp. 3–16). New York, NY: Springer. Google Scholar | |
|
Sanetti, L. M. H., Collier-Meek, M. A., Long, A. C. J., Byron, J. R., Kratochwill, T. R. (2015). Increasing teacher treatment integrity of behavior support plans through consultation and implementation planning. Journal of School Psychology, 53, 209–229. Google Scholar | Crossref | Medline | ISI | |
|
Sanetti, L. M. H., Gritter, K. L., Dobey, L. (2011). Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. School Psychology Review, 40, 72–84. doi:10.1177/0143034313476399 Google Scholar | SAGE Journals | ISI | |
|
Sanetti, L. M. H., Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445–459. Google Scholar | ISI | |
|
Schumaker, J. (1989). The heart of strategy instruction: Effective modeling. Strategram: Strategies Instruction Model, 1, 1–5. Google Scholar | |
|
Sterling-Turner, H. E., Watson, T. S., Moore, J. W. (2002). The effects of direct training and treatment integrity on treatment outcomes in school consultation. School Psychology Quarterly, 17, 47–77. doi:10.1521/scpq.17.1.47.19906 Google Scholar | Crossref | ISI | |
|
Sterling-Turner, H. E., Watson, T. S., Wildmon, M., Watkins, C., Little, E. (2001). Investigating the relationship between training type and treatment integrity. School Psychology Quarterly, 16, 56–67. doi:10.1521/scpq.16.1.56.19157 Google Scholar | Crossref | ISI | |
|
Sulsky, L. M., Balzer, W. K. (1988). Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73, 497–506. doi:10.1037/0021-9010.73.3.497 Google Scholar | Crossref | ISI | |
|
Sutherland, K. S., Wehby, J. H. (2001). The effect of self-evaluation on teaching behavior in classrooms for students with emotional and behavioral disorders. The Journal of Special Education, 35, 161–171. doi:10.1177/002246690103500306 Google Scholar | SAGE Journals | ISI | |
|
Swanson, H. L., Deshler, D. D. (2003). Instructing adolescents with learning disabilities: Converting meta-analysis to practice. Journal of Learning Disabilities, 36, 124–135. Google Scholar | SAGE Journals | ISI | |
|
U.S. Department of Education, National Center for Education Statistics, Common Core of Data . (2014). CCD public school data 2012-2013, 2013-2014 school years. Retrieved from http://nces.ed.gov/ccd/schoolsearch Google Scholar | |
|
Volpe, R. J., Briesch, A. J. (2012). Generalizability and dependability of single-item and multiple-item Direct Behavior Rating Scales for engagement and disruptive behavior. School Psychology Review, 41, 246–261. Google Scholar | ISI | |
|
Zirkel, P. A., Thomas, L. B. (2010). State laws and guidelines for implementing RTI. Teaching Exceptional Children, 43, 60–73. doi:10.1177/004005991004300107 Google Scholar | SAGE Journals |
