![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
The relation between fidelity of implementation and student outcomes in a computer-based middle school mathematics curriculum was measured empirically. Participants included 485 students and 23 teachers from 11 public middle schools across seven states. Implementation fidelity was defined using two constructs: fidelity to structure and fidelity to process. Because of the nested nature of the data, we used a two-level hierarchical linear model for analysis. Four variables, all categorized as fidelity to structure variables, proved significant—total time in intervention (p <.001), concentration of time in intervention (p = .03), direct observation of intervention fidelity (p = .04), and pretest score (p <.001). Fidelity to process was found to be nonsignificant. The importance of measuring the relation between implementation fidelity and student outcomes is discussed as well as implications for researchers and teachers.
Keywords math, technology, middle school
References
Section:
|
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education . (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association. Google Scholar | |
|
Baker, E. (2001). Technology: How do we know it works? In Heineke, W., Blasi, L. (Eds.), Methods of evaluating educational technology (pp. 77-84). Greenwich, CT: Information Age. Google Scholar | |
|
Becker, H. J. (1994). How exemplary computer-using teachers differ from other teachers: Implications for realizing the potential of computers in schools. Journal of Research on Computing in Education, 26, 291–321. Google Scholar | Crossref | |
|
Campbell, D. T., Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Dallas, TX: Houghton Mifflin. Google Scholar | |
|
Cavanaugh, C. L., Kim, A., Wanzek, J., Vaughn, S. (2004). Kindergarten reading interventions for at-risk students: Twenty years of research. Learning Disabilities: A Contemporary Journal, 2, 9–21. Google Scholar | |
|
Christ, T. J. (2007). Experimental control and threats to internal validity of concurrent and nonconcurrent multiple baseline designs. Psychology in the Schools, 44, 451–459. Google Scholar | Crossref | ISI | |
|
Cohen, P. A., Kulik, J. A., Kulik, C. C. (1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19, 237–248. Google Scholar | SAGE Journals | ISI | |
|
Cook, S. B., Scruggs, T. E., Mastropieri, M. A., Casto, G. C. (1985). Handicapped students as tutors. Journal of Special Education, 19, 483–492. Google Scholar | SAGE Journals | ISI | |
|
Crawford, L. (2008, March). Effects of an online supplementary mathematics curriculum for English language learners. Paper presented at the annual meeting of the American Educational Research Association, New York, NY. Google Scholar | |
|
Digital Directions International . (n.d.). HELP Math: Summary of research findings (Technical report). Carbondale, CO: Author. Retrieved from http://www.helpprogram.net. Google Scholar | |
|
Elbaum, B., Vaughn, S. (2001). School-based interventions to enhance the self-concept of student with learning disabilities: A meta-analysis. Elementary School Journal, 101, 303–329. Google Scholar | Crossref | ISI | |
|
Fletcher, J. M., Vaughn, S. (2009). Response to intervention: Preventing and remediating academic difficulties. Child Development Perspectives, 3, 30–37. Google Scholar | Crossref | Medline | ISI | |
|
Gall, M. D., Gall, J. P., Borg, W. R. (2007). Educational research: An introduction (8th ed.). Boston, MA: Allyn & Bacon. Google Scholar | |
|
Gersten, R., Baker, S., Lloyd, J. W. (2000). Designing high-quality research in special education: Group experimental design. Journal of Special Education, 34, 2–18. Google Scholar | SAGE Journals | ISI | |
|
Gersten, R., Edyburn, D. (2007). Defining quality indicators for group design in special education technology research. Journal of Special Education Technology, 22, 3–18. Google Scholar | SAGE Journals | |
|
Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., Innocenti, M. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149–164. Google Scholar | SAGE Journals | ISI | |
|
Gresham, F. M. (1989). Assessment of treatment integrity in school consultation and prereferral intervention. School Psychology Review, 18, 37–50. Google Scholar | ISI | |
|
Gresham, F. M., Gansle, K., Noell, G. H. (1993). Treatment integrity in applied behavior analysis with children. Journal of Applied Behavior Analysis, 26, 257–263. Google Scholar | Crossref | Medline | ISI | |
|
Gresham, F. M., MacMillan, D. L., Beebe-Frankenberger, M. E., Bocian, K. M. (2000). Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented? Learning Disabilities Research & Practice, 15, 198–205. Google Scholar | Crossref | |
|
HELP Math (n.d.). Retrieved from http://www.helpprogram.net Google Scholar | |
|
Hofmann, D. A., Gavin, M. B. (1998). Centering decisions in hierarchical linear models: Implications for research in organizations. Journal of Management, 24, 623–641. Google Scholar | SAGE Journals | ISI | |
|
Hohmann, A. A., Shear, M. K. (2002). Community-based intervention research: Coping with the “noise” of real life in study design. American Journal of Psychiatry, 159, 201–207. Google Scholar | Crossref | Medline | ISI | |
|
Hox, J. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum. Google Scholar | Crossref | |
|
Hulleman, C. S., Cordray, D. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88–110. Google Scholar | Crossref | |
|
Maddux, C. D., Johnson, L., Harlow, S. (1993). The state of the art in computer education: Issues for discussion with teachers-in-training. Journal of Technology and Teacher Education, 1, 219–228. Google Scholar | |
|
Melde, C., Esbensen, F., Tusinski, K. (2006). Addressing program fidelity using onsite observations and program provider descriptions of program delivery. Evaluation Review, 30, 714–740. Google Scholar | SAGE Journals | ISI | |
|
Mellard, D. F., Johnson, E. (2008). RTI: A practitioner’s guide to implementing response to intervention. Thousand Oaks, CA: Corwin Press. Google Scholar | Crossref | |
|
Mills, S., Ragan, T. (2000). A tool for analyzing implementation fidelity of an integrated learning system. Educational Technology Research and Development, 48, 21–41. Google Scholar | Crossref | ISI | |
|
Mislevy, R. J. (2007). Validity by design. Educational Researcher, 36, 463–469. Google Scholar | SAGE Journals | |
|
Mowbray, C. T., Holter, M. C., Teague, G. B., Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340. Google Scholar | SAGE Journals | ISI | |
|
National Research Council of the National Academies: Committee for a Review of the Evaluation Data on the Effectiveness of NSF-Supported and Commercially Generated Mathematics Curriculum Materials . (2004). On evaluating curricular effectiveness: Judging the quality of K–12 mathematics evaluations. Washington, DC: Author. Google Scholar | |
|
Noell, G. H., Gresham, F. M., Gansle, K. A. (2002). Does treatment integrity matter? A preliminary investigation of instructional implementation and mathematics performance. Journal of Behavioral Education, 11, 51–67. Google Scholar | Crossref | |
|
O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relation to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33–84. Google Scholar | SAGE Journals | ISI | |
|
Pearson Education . (2008–2009). SuccessMaker 2.0 math reference guide. Upper Saddle River, NJ: Author. Google Scholar | |
|
Pierangelo, R., Giuliani, G. (2008). Frequently asked questions about response to intervention. Thousand Oaks, CA: Corwin Press. Google Scholar | |
|
Power, T. J., Blom-Hoffman, J., Clarke, A. T., Riley-Tillman, T. C., Kelleher, C., Manz, P. H. (2005). Reconceptualizing intervention integrity: A partnership-based framework for linking research with practice. Psychology in the Schools, 42, 495–507. Google Scholar | Crossref | ISI | |
|
Rohrbeck, C. A., Ginsburg-Block, M. D., Fantuzzo, J. W., Miller, T. R. (2003). Peer-assisted learning interventions with elementary school students: A meta-analytic review. Journal of Educational Psychology, 95, 240–257. Google Scholar | Crossref | ISI | |
|
Shadish, W. R., Cook, T. D., Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth. Google Scholar | |
|
Slavin, R. E. (2008). Perspectives on evidence-based research in education: What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(5), 5–14. Google Scholar | SAGE Journals | ISI | |
|
Songer, N. B., Gotwals, A. W. (2005, April). Fidelity of implementation in three sequential curricular units. In Lynch, S. (Chair) & O’Donnell, C. L. , “Fidelity of implementation” in implementation and scale-up research designs: Applications from four studies of innovative science curriculum materials and diverse populations. Symposium conducted at the annual meeting of the American Educational Research Association, Montreal, Canada. Google Scholar | |
|
Tucker, A. R., Blythe, B. (2008). Attention to treatment fidelity in social work outcomes: A review of the literature from the 1990s. Social Work Research, 32, 185–190. Google Scholar | Crossref | ISI | |
|
Weston, T. (2004). Formative evaluation for implementation: Evaluating educational technology applications and lessons. American Journal of Evaluation, 25, 51–64. Google Scholar | SAGE Journals | ISI | |
|
Ysseldyke, J., Bolt, D. M. (2007). Effect of technology-enhanced continuous progress monitoring on math achievement. School Psychology Review, 36, 453–467. Google Scholar | ISI | |
|
Ysseldyke, J., Spicuzza, R., Kosciolek, S., Teelucksingh, E., Boys, C., Lemkuil, A. (2003). Using curriculum-based instructional management system to enhance math achievement in urban schools. Journal of Education for Students Placed at Risk, 8, 247–265. Google Scholar | Crossref | |
|
Zvoch, K. (2009). Treatment fidelity in multisite evaluation. American Journal of Evaluation, 30, 44–61. Google Scholar | SAGE Journals | ISI |
