![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Cognitive or skills diagnosis models are discrete latent variable models developed specifically for the purpose of identifying the presence or absence of multiple fine-grained skills. However, applications of these models typically involve dichotomous or dichotomized data, including data from multiple-choice (MC) assessments that are scored as right or wrong. The dichotomization approach to the analysis of MC data ignores the potential diagnostic information that can be found in the distractors and is therefore deemed diagnostically suboptimal. To maximize the diagnostic value of MC assessments, this article prescribes how MC options should be constructed to make them more cognitively diagnostic and proposes a cognitive diagnosis model for analyzing such data. The article discusses the specification of the proposed model and estimation of its parameters. Moreover, results of a simulation study evaluating the viability of the model and an estimation algorithm are presented. Finally, practical considerations concerning the proposed framework are discussed.
|
Baker, F.B. (1992). Item response theory: Parameter estimation techniques . New York: Dekker. Google Scholar | |
|
Birenbaum, M. , Tatsuoka, C. , & Xin, T. (2005). Large-scale diagnostic assessment: Comparison of eighth graders' mathematics performance in the United States, Singapore and Israel. Assessment in Education Principles Policy and Practice, 12, 167-181. Google Scholar | Crossref | |
|
Briggs, D. , Alonzo, A. , Schwab, C. , & Wilson, M. (2006). Diagnostic assessment with ordered multiple-choice items. Educational Assessment, 11, 33-63. Google Scholar | Crossref | |
|
Carlin, B.P. , & Louis, T.A. (2000). Bayes and empirical Bayes methods for data analysis . New York: Chapman & Hall . Google Scholar | Crossref | |
|
de la Torre, J. (2006, June). Skills profile comparisons at the state level: An application and extension of cognitive diagnosis modeling in NAEP . Paper presented at the International Meeting of the Psychometric Society, Montreal, Canada. Google Scholar | |
|
de la Torre, J. , & Patz, R. (2005). Making the most of what we have: A practical application of multidimensional item response theory in test scoring. Journal of Educational and Behavioral Statistics, 30, 295-311. Google Scholar | SAGE Journals | ISI | |
|
Doignon, J.P. , & Falmagne, J.C. (1999). Knowledge spaces. New York: Springer-Verlag. Google Scholar | Crossref | |
|
Doornik, J.A. (2002). Object-oriented matrix programming using Ox (Version 3.1). [Computer software]. London: Timberlake Consultants Press. Google Scholar | |
|
Embretson, S. (1984). A general latent trait model for response processes . Psychometrika, 49, 175-186. Google Scholar | Crossref | ISI | |
|
Haertel, E.H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352. Google Scholar | Crossref | ISI | |
|
Haertel, E.H. , & Wiley, D.E. (1993). Representations of ability structures: Implications for testing. In N. Frederiksen , R. J. Mislevy , & I. Bejar (Eds.), Test theory for a new generation of tests (pp. 359-384). Hillsdale, NJ: Erlbaum. Google Scholar | |
|
Halford, G.S. , Andrews, G. , Dalton, C. , Boag, C. , & Zielinski, T. (2002). Young children's performance on the balance scale: The influence of relational complexity. Journal of Experimental Child Psychology, 81, 417-445. Google Scholar | Crossref | Medline | ISI | |
|
Jansen, B.R.J. , & van der Maas, H.L.J. (2002). The development of children's rule use on the balance scale task. Journal of Experimental Child Psychology, 81, 383-416. Google Scholar | Crossref | Medline | ISI | |
|
Junker, B.W. , & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272. Google Scholar | SAGE Journals | ISI | |
|
Macready, G.B. , & Dayton, C.M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 33, 379-416. Google Scholar | |
|
Mislevy, R. (1995). Probability-based inference in cognitive diagnosis . In P. D. Nichols , S. F. Chipman , & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 43-71). Hillsdale, NJ: Erlbaum. Google Scholar | |
|
National Research Council. (2001). Knowing what students know: The science and design of educational assessment . Washington, DC: National Academies Press. Google Scholar | |
|
National Research Council. (2003). Strategic education research partnership/Committee on a strategic education research partnership; M. S. Donovan, A. K. Wigdor, and C. E. Snow, Editors . Washington, DC: National Academies Press. Google Scholar | |
|
Neyman, J. , & Scott, E.L. (1948). Consistent estimates based on partially consistent observations. Econometrika, 16, 661-679. Google Scholar | Crossref | ISI | |
|
Nitko, A.J. (2001). Educational assessment of students (3rd ed.). Columbus, OH: Merrill Prentice Hall. Google Scholar | |
|
Osterlind, S.J. (1998). Constructing test items: Multiple choice, constructed-response, performance and other formats (2nd ed.). Boston: Kluwer Academic. Google Scholar | |
|
Pellegrino, J.W. , Baxter, G.P. , & Glaser, R. (1999). Addressing the ``two disciplines'' problem: Linking theories of cognition and learning with assessment and instructional practices . In A. Iran-Nejad & P. D. Pearson (Eds.), Review of research in education (pp. 307-353). Washington, DC. American Educational Research Association. Google Scholar | |
|
Sadler, P.M. (1998). Psychometric models of student conceptions in science: Reconciling qualitative studies and distractor-driven assessment instruments. Journal of Research in Science Teaching, 35, 265-296. Google Scholar | Crossref | ISI | |
|
Shultz, T.R. , Mareschal, D. , & Schmidt, W.C. (1994). Modeling cognitive development on balance scale phenomena. Machine Learning, 16, 57-86. Google Scholar | Crossref | ISI | |
|
Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8, 481-520. Google Scholar | Crossref | ISI | |
|
Siegler, R.S. (1981). Developmental sequences within and between concepts . Monographs of the Society for Research in Child Development , 46 (2, Serial No. 189). Google Scholar | Crossref | ISI | |
|
Stiggins. R.J. (2002). Assessment crisis: The absence of assessment for learning. Phi Delta Kappan, 83, 758-765. Google Scholar | SAGE Journals | ISI | |
|
Tatsuoka, C. (2002). Data-analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society Series C (Applied Statistics), 51, 337-350. Google Scholar | Crossref | ISI | |
|
Tatsuoka, K.K. (1983). Rule-space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement , 20, 345-354. Google Scholar | Crossref | ISI | |
|
Tatsuoka, K.K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen , R. Glaser , A. Lesgold , & Safto, M. (Eds.), Monitoring skills and knowledge acquisition (pp. 453-488). Hillsdale, NJ: Erlbaum. Google Scholar | |
|
Tatsuoka, K.K. , Corter, J. , & Tatsuoka, C. (2004). Patterns of diagnosed mathematical content and process skills in TIMSS-R across a sample of 20 countries. American Educational Research Journal, 41, 901-906. Google Scholar | SAGE Journals | ISI | |
|
Thissen, D. , & Steinberg, L. (1997). A response model for multiple-choice items. In W. van der Linden & R. Hambleton (Eds.), Handbook of modern item response theory (pp. 51-65). New York: Springer. Google Scholar | Crossref | |
|
van der Maas, H.L.J. , & Jansen, B R. J. (2003). What response times tell of children's behavior on the balance scale task. Journal of Experimental Child Psychology, 85, 141-177. Google Scholar | Crossref | Medline | ISI | |
|
Wainer, H. , Vevea, J.L. , Camacho, F. , Reeve, B. , Rosa, K. , Nelson, L. , et al. (2001). Augmented scores- ``Borrowing strength'' to compute scores based on small number of items. In D. Thissen & H. Wainer (Eds.), Test scoring (pp. 343-387). Mahwah, NJ: Lawrence Erlbaum. Google Scholar | |
|
Wiggins, G. (1998). Educative assessment: Designing assessment to inform and improve performance. San Francisco: Jossey-Bass. Google Scholar | |
|
Wilson, M. , & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education, 13, 181-208. Google Scholar | Crossref | ISI |
