![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Practitioners and researchers interested in understanding student achievement, its predictors, and how it relates to other student outcomes are likely unaware of how the source information about achievement may offer subtly different pictures. This study applies multitrait–multimethod (MTMM) confirmatory factor analysis (CFA) within a structural equation modeling (SEM) framework to student achievement data to demonstrate empirically how commonly used measures of student achievement may reflect different information about student performance. Using student population-level data from a single state, this study presents a robust demonstration of the similarities and differences among three commonly used achievement measures—American College Testing (ACT) scores, state test scores, and grade point average (GPA). Results show that state assessment scores and ACT scores measured a similar achievement construct, whereas student grades reflected less of the achievement construct and a higher level of method effects. Possible sources of the similarities and differences among different achievement measures are discussed, along with implications for measurement among gifted students.
References
Section:
|
Acee, T. W., Cho, Y., Kim, J., Weinstein, C. E. (2012). Relationships among properties of college students’ self-set academic goals and academic achievement. Educational Psychology, 32, 681-698. doi:10.1080/01443410.2012.712795 Google Scholar | Crossref | |
|
American College Testing . (2007). ACT Assessment Technical Manual. Iowa City, IA: Author. Google Scholar | |
|
Anderson, J. C., Gerbing, D. W. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49, 155-173. Google Scholar | Crossref | ISI | |
|
Apthorp, H. S., Igel, C., Dean, C. (2012). Using similarities and differences: A meta-analysis of its effects and emergent patterns. Social Science and Mathematics, 112, 204-216. doi:10.1111/j.1949-8594.2012.00139.x Google Scholar | Crossref | |
|
Arrasmith, D. G., Sheehan, D. S., Applebaum, W. R. (1984). A comparison of the selected-response strategy and the constructed-response strategy for assessment of a third-grade writing task. The Journal of Educational Research, 77, 172-177. doi:10.1080/00220671.1984.10885519 Google Scholar | Crossref | |
|
Atkinson, R. C., Geiser, S. (December, 2009). Reflections on a century of college admissions tests (Center for Studies in Higher Education, Research and Occasional Paper 4.09). doi:10.3102/0013189X09351981 Google Scholar | |
|
Bagozzi, R. P., Yi, Y. (1993). Multitrait-multimethod matrices in consumer research: Critique and new developments. Journal of Consumer Psychology, 2, 143-170. doi:10.1016/S1057-7408(08)80022-8 Google Scholar | Crossref | |
|
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press. Google Scholar | |
|
Campbell, D. T., Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. doi:10.1037/h0046016 Google Scholar | Crossref | Medline | ISI | |
|
Cohrs, J. C., Kämpfe-Hargrave, N., Riemann, R. (2012). Individual differences in ideological attitudes and prejudice: Evidence from peer-report data. Journal of Personality and Social Psychology, 103, 343-361. doi:10.1037/a0028706 Google Scholar | Crossref | Medline | |
|
Eid, M. (2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241-261. Google Scholar | Crossref | ISI | |
|
Ellett, L. (1993). Instructional practices in mainstreamed secondary classrooms. Journal of Learning Disabilities, 26, 57-64. doi:10.1177/002221949302600107 Google Scholar | SAGE Journals | |
|
Firestone, W. A., Mayrowetz, D., Fairman, J. (1998). Performance-based assessment and instructional change: The effects of testing in Maine and Maryland. Educational Evaluation and Policy Analysis, 20, 94-113. doi:10.3102/01623737020002095 Google Scholar | Crossref | |
|
Foreman, J. L., Gubbins, E. J. (2015). Teachers see what ability scores cannot: Predicting student performance with challenging mathematics. Journal of Advanced Academics, 26, 5-23. doi:10.1177/1932202X14552279 Google Scholar | SAGE Journals | ISI | |
|
Geiser, C., Eid, M., Nussbeck, F. W. (2008). On the meaning of the latent variables in the CT-C(M-1) model: A comment on Maydeu-Olivares and Coffman (2006). Psychological Methods, 13, 49-57. doi:10.1037/1082-989X.13.1.49 Google Scholar | Crossref | Medline | ISI | |
|
Gradwell, J. M. (2006). Teaching in spite of, rather than because of the test: A case of ambitious history teaching in New York State. In Grant, S. G. (Ed.), Measuring history: Cases of state-level testing across the United States (pp. 157-176). Greenwich, CT: Information Age. Google Scholar | |
|
Grigorenko, E. L., Geiser, C., Slobodskaya, H. R., Francis, D. J. (2010). Cross-informant symptoms from CBCL, TRF, and YSR: Trait and method variance in a normative sample of Russian youths. Psychological Assessment, 22, 893-911. doi:10.1037/a0020703 Google Scholar | Crossref | Medline | ISI | |
|
Guskey, T. R. (2000). Grading policies that work against standards . . . and how to fix them. National Association of Secondary School Principals. NASSP Bulletin, 84(620), 20-27. doi:10.1177/019263650008462003 Google Scholar | SAGE Journals | |
|
Hu, L., Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118 Google Scholar | Crossref | ISI | |
|
Joreskog, K. G. (1971). Simultaneous factor analysis in several populations. Psychometrika, 36, 409-426. Google Scholar | Crossref | ISI | |
|
Kenny, D. A., Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172. doi:10.1037/0033-2909.112.1.165 Google Scholar | Crossref | ISI | |
|
Kline, R. B. (2010). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press. Google Scholar | |
|
Kollman, D. M., Brown, T. A., Barlow, D. H. (2009). The construct validity of acceptance: A multitrait-multimethod investigation. Behavior Therapy, 40, 205-218. doi:10.1016/j.beth.2008.06.002 Google Scholar | Crossref | Medline | |
|
Kuncel, N. R., Crede, M., Thomas, L. L. (2005). The validity of self-reported grade point averages, class ranks, and test scores: A meta-analysis and review of the literature. Review of Educational Research, 75, 63-82. doi:10.3102/00346543075001063 Google Scholar | SAGE Journals | ISI | |
|
Lance, C. E., Noble, C. L., Scullen, S. E. (2002). A critique of the correlated-trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 2, 228-244. doi:10.1037/1082-989X.7.2.228 Google Scholar | Crossref | |
|
Le, V., Hamilton, L., Robyn, A. (2000). Alignment among secondary and post-secondary assessments in California. In Burr, E., Hayward, G. C., Fuller, B., Kirst, M. W. (Eds.), Crucial issues in California education 2000: Are the reform pieces fitting together? (pp. 177-206). Berkeley: Policy Analysis for California Education. Google Scholar | |
|
Linn, M. R. (1993). College entrance examinations in the United States: A brief history for college admissions counselors. The Journal of College Admissions, 140, 6-16. Google Scholar | |
|
Linn, R. L. (1998). Assessments and accountability (CSE Technical Report 490). Los Angeles, CA: Center for the Study of Evaluation. Google Scholar | |
|
Lord, F. M., Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley. Google Scholar | |
|
Marsh, H. W. (1989). Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361. doi:10.1177/014662168901300402 Google Scholar | SAGE Journals | ISI | |
|
Marsh, H. W., Bailey, M. (1991). Confirmatory factor analysis of multitrait-multimethod data: A comparison of alternative models. Applied Psychological Measurement, 15, 47-70. doi:10.1177/014662169101500106 Google Scholar | SAGE Journals | |
|
Marsh, H. W., Byrne, B. M., Craven, R. (1992). Overcoming problems in confirmatory factor analyses of MTMM data: The correlated uniqueness model and factorial invariance. Multivariate Behavioral Research, 27, 489-507. doi:10.1207/s15327906mbr2704_1 Google Scholar | Crossref | Medline | |
|
Marsh, H. W., Grayson, D. (1995). Latent variable models of multitrait-multimethod data. In Hoyle, R. H. (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 177-198). Thousand Oaks, CA: Sage. Google Scholar | |
|
McBee, M. (2010). Modeling outcomes with floor or ceiling effects: An introduction to the Tobit model. Gifted Child Quarterly, 54, 314-320. doi:10.1177/0016986210379095 Google Scholar | SAGE Journals | ISI | |
|
McBee, M. T., Peters, S. J., Waterman, C. (2014). Combining scores in multiple-criteria assessment systems: The impact of combination rules. Gifted Child Quarterly, 58, 69-89. doi:10.1177/0016986213513794 Google Scholar | SAGE Journals | |
|
Pae, H. K. (2012). Convergence and discriminant: Assessing multiple traits using multiple methods. Educational Research and Evaluation: An International Journal on Theory and Practice, 18, 571-596. doi:10.1080/13803611.2012.704167 Google Scholar | Crossref | |
|
Plucker, J. A., Callahan, C. M. (2014). Research on giftedness and gifted education: Status of the field and considerations for the future. Exceptional Children, 80, 390-406. doi:10.1177/0014402914527244 Google Scholar | SAGE Journals | ISI | |
|
Porter, A. C., Polikiff, M. S., Smithson, J. (2009). Is there a de facto national intended curriculum? Evidence from state content standards. Educational Evaluation and Policy Analysis, 31, 238-268. doi:10.3102/0162373709336465 Google Scholar | SAGE Journals | |
|
Rauch, D. P., Hartig, J. (2010). Multiple-choice versus open-ended response formats of reading test items: A two-dimensional IRT analysis. Psychological Test and Assessment Modeling, 52, 354-379. Google Scholar | |
|
Richardson, M., Abraham, C. (2009). Conscientiousness and achievement motivation predict performance. European Journal of Personality, 23, 589-605. doi:10.1002/per.732 Google Scholar | Crossref | |
|
Richardson, M., Abraham, C., Bond, R. (2012). Psychological correlates of University students’ academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138, 353-387. doi:10.1037/a0026838 Google Scholar | Crossref | Medline | ISI | |
|
Robertson, G. D. (2011, May 2). 11th graders would take ACT exam under NC plan. Community College Week, p. 3. Google Scholar | |
|
Roorda, D. L., Koomen, H. M. Y., Spilt, J. L., Oort, F. J. (2011). The influence of affective teacher-student relationships on students’ school engagement and achievement: A meta-analytic approach. Review of Educational Research, 81, 493-529. doi:10.3102/0034654311421793 Google Scholar | SAGE Journals | ISI | |
|
Skinner, E. A., Belmont, M. J. (1993). Motivation in the classroom: Reciprocal effects of teacher behavior and student engagement across the school year. Journal of Educational Psychology, 85, 571-581. doi:10.1037/0022-0663.85.4.571 Google Scholar | Crossref | ISI | |
|
Smith, L. (2007, August 15). ACT scores edge up in 2007 but suggest that many students are unprepared for college-level work. The Chronicle of Higher Education, p. 8. Google Scholar | |
|
Spearman, C. (1904). “General intelligence,” objectively determined and measured. American Journal of Psychology, 15, 201-293. Google Scholar | Crossref | |
|
Steinmetz, J., Loarer, E., Houssemand, C. (2011). Rigidity of attitudes and behaviors: A study on the validity of the concept. Individual Differences Research, 9, 84-106. Google Scholar | |
|
Swiatek, M. A. (2007). The talent search model: Past, present, and future. Gifted Child Quarterly, 51, 320-329. doi:10.1177/0016986207306318 Google Scholar | SAGE Journals | |
|
Threlfall, J., Pool, P., Homer, M., Swinnerton, B. (2007). Implicit aspects of paper and pencil mathematics assessment that come to light through the use of the computer. Educational Studies of Mathematics, 66, 335-348. doi:10.1007/s10649-006-9078-5 Google Scholar | Crossref | |
|
Tomás, J. M., Hontangas, P. M., Oliver, A. (2000). Linear confirmatory factor models to evaluate multitrait-multimethod matrices: The effects of number of indicators and correlation among methods. Multivariate Behavioral Research, 35, 469-499. doi:10.1207/S15327906MBR3504_03 Google Scholar | Crossref | Medline | |
|
Tomás, J. M., Oliver, A. (1999). Rosenberg’s self-esteem scale: Two factors or method effects. Structural Equation Modeling, 6, 84-98. doi:10.1080/10705519909540120 Google Scholar | Crossref | ISI | |
|
U.S. Department of Education . (2012). Retrieved from http://www.eddataexpress.ed.gov/index.cfm Google Scholar | |
|
Widaman, K. F. (1985). On methods for comparing apples and oranges. Multivariate Behavioral Research, 30, 101-106. doi:10.1207/s15327906mbr3001_10 Google Scholar | Crossref | |
|
Wimmers, P. F., Fung, C. (2008). The impact of case specificity and generalizable skills on clinical performance: A correlated-traits-correlated methods approach. Medical Education, 42, 580-588. doi:10.1111/j.1365-2923.2008.03089.x Google Scholar | Crossref | Medline | |
|
Wood, S., Mayo-Wilson, E. (2012). School-based mentoring for adolescents: A systematic review and meta-analysis. Research on Social Work Practice, 22, 257-269. doi:10.1177/1049731511430836 Google Scholar | SAGE Journals | |
|
Worrell, F. C. (2009). Myth 4: A single test score or indicator tells us all we need to know about giftedness. Gifted Child Quarterly, 53, 242-244. doi:10.1177/0016986209346828 Google Scholar | SAGE Journals | ISI |
