![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Advances in multilevel modeling techniques now make it possible to investigate the psychometric properties of instruments using clustered data. Factor models that overlook the clustering effect can lead to underestimated standard errors, incorrect parameter estimates, and model fit indices. In addition, factor structures may differ depending on the level of analysis. The current study illustrates the application of multilevel factor analytic techniques using a large statewide sample of middle school students (n = 39,364) from 423 schools. Both multilevel exploratory and confirmatory factor analyses were used to investigate the factor structure of the Positive Values Scale (PVS) as part of a school climate survey. Results showed that for the PVS, a two-correlated factor model at Level 1 and a one-factor model at Level 2 best fit the data. Implications and guidance for applied researchers are discussed.
Keywords multilevel CFA, multilevel EFA, school climate, values
References
Section:
|
Bear, G. G., Gaskins, C., Blank, J., Chen, F. F. (2011). Delaware School Climate Survey—Student: Its factor structure, concurrent validity, and reliability. Journal of School Psychology, 49, 157-174. doi:10.1016/j.jsp.2011.01.001 Google Scholar | Crossref | Medline | ISI | |
|
Berkowitz, M. W., Bier, M. C. (2004). Research-based character education. The Annals of the American Academy of Political and Social Science, 591, 72-85. doi:10.1177/0002716203260082 Google Scholar | SAGE Journals | ISI | |
|
Bliese, P. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In Bollen, K., Long, J. (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions and new directions (pp. 349-381). San Francisco, CA: Jossey-Bass. Google Scholar | |
|
Bronfenbrenner, U. (1974). Developmental research, public policy, and the ecology of childhood. Child Development, 45, 1-5. doi:10.2307/1127743 Google Scholar | Crossref | ISI | |
|
Browne, M. W., Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230-258. doi:10.1177/0049124192021002005 Google Scholar | SAGE Journals | ISI | |
|
Byrne, B. (2012). Structural equation modeling with Mplus: Basic concepts, applications, and programming. New York, NY: Routledge. Google Scholar | |
|
Catalano, R. F., Berglund, M. L., Ryan, J. A. M., Lonczak, H. S., Hawkins, J. D. (2002). Positive youth development in the United States: Research findings on evaluations of positive youth development programs. Prevention & Treatment, 5, doi:10.1037/1522–3736.5.1.515a Google Scholar | Crossref | |
|
Cohen, J., Cohen, P., West, S. G., Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum. Google Scholar | |
|
Cohen, J., McCabe, E., Michelli, N., Pickeral, T. (2009). School climate: Research, policy, practice, and teacher education. Teachers College Record, 111, 180-213. Google Scholar | ISI | |
|
Cornell, D., Huang, F., Konold, T., Meyer, P., Lacey, P., Nekvasil, E., . . . Skukla, K. (2013). Technical report of the Virginia secondary school climate survey: 2013 results for 7th and 8th grade students and teachers executive summary. Charlottesville: University of Virginia. Retrieved from http://curry.virginia.edu/uploads/resourceLibrary/School_Climate_Survey_State_Technical_Report_Executive_Summary_9-4-13.pdf Google Scholar | |
|
Dedrick, R. F., Greenbaum, P. E. (2011). Multilevel confirmatory factor analysis of a scale measuring interagency collaboration of children’s mental health agencies. Journal of Emotional and Behavioral Disorders, 19, 27-40. doi:10.1177/1063426610365879 Google Scholar | SAGE Journals | ISI | |
|
D’Haenens, E., Van Damme, J., Onghena, P. (2010). Multilevel exploratory factor analysis: Illustrating its surplus value in educational effectiveness research. School Effectiveness and School Improvement, 21, 209-235. doi:10.1080/09243450903581218 Google Scholar | Crossref | ISI | |
|
Domino, M. (2013). Measuring the impact of an alternative approach to school bullying. Journal of School Health, 83, 430-437. doi:10.1111/josh.12047 Google Scholar | Crossref | Medline | ISI | |
|
Dyer, N. G., Hanges, P. J., Hall, R. J. (2005). Applying multilevel confirmatory factor analysis techniques to the study of leadership. The Leadership Quarterly, 16, 149-167. doi:10.1016/j.leaqua.2004.09.009 Google Scholar | Crossref | ISI | |
|
Ender, P. (2012). EFA in a CFA framework. Retrieved from http://www.stata.com/meeting/sandiego12/materials/sd12_ender.pdf Google Scholar | |
|
Espelage, D. L., Holt, M. K., Henkel, R. R. (2003). Examination of peer-group contextual effects on aggression during early adolescence. Child Development, 74, 205-220. doi:10.1111/1467-8624.00531 Google Scholar | Crossref | Medline | ISI | |
|
Gerbing, D. W., Anderson, J. C. (1992). Monte Carlo evaluations of goodness of fit indices for structural equation models. Sociological Methods & Research, 21, 132-160. doi:10.1177/0049124192021002002 Google Scholar | SAGE Journals | ISI | |
|
Griffith, J. (1997). Student and parent perceptions of school social environment: Are they group based? The Elementary School Journal, 98, 135-150. doi:10.1086/461888 Google Scholar | Crossref | ISI | |
|
Heck, R. H., Thomas, S. L. (2008). An introduction to multilevel modeling techniques (2nd ed.). New York, NY: Routledge. Google Scholar | |
|
Hox, J. J. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum. Google Scholar | |
|
Hu, L., Bentler, P. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118 Google Scholar | Crossref | ISI | |
|
Huang, F. (2014a). Alternatives to multilevel modeling for the analysis of clustered data. Journal of Experimental Education. Advance online publication. doi:10.1080/00220973.2014.952397 Google Scholar | Crossref | ISI | |
|
Huang, F. (2014b). Analyzing group level effects with clustered data using Taylor series linearization. Practical Assessment, Research & Evaluation, 19(13). Retrieved from http://pareonline.net/getvn.asp?v=19&;n=13 Google Scholar | |
|
Julian, M. W. (2001). The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling. Structural Equation Modeling, 8, 325-352. Google Scholar | Crossref | ISI | |
|
Kaplan, D., Elliott, P. R. (1997). A didactic example of multilevel structural equation modeling applicable to the study of organizations. Structural Equation Modeling, 4, 1-24. doi:10.1080/10705519709540056 Google Scholar | Crossref | ISI | |
|
Kline, R. (2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press. Google Scholar | |
|
Konold, T., Cornell, D., Huang, F., Meyer, P., Lacey, A., Nekvasil, E., . . . Shukla, K. (2014). Multilevel multi-informant structure of the authoritative school climate survey. School Psychology Quarterly, 29, 238-255. doi:10.1037/spq0000062 Google Scholar | Crossref | Medline | ISI | |
|
Lai, M., Kwok, O. (2014). Examining the rule of thumb of not using multilevel modeling: The “design effect smaller than two” rule. The Journal of Experimental Education. Advance online publication. doi:10.1080/00220973.2014.907229 Google Scholar | Crossref | ISI | |
|
Musca, S. C., Kamiejski, R., Nugier, A., Méot, A., Er-Rafiy, A., Brauer, M. (2011). Data with hierarchical structure: Impact of intraclass correlation and sample size on type-I error. Frontiers in Psychology, 2, 1-6. doi:10.3389/fpsyg.2011.00074 Google Scholar | Crossref | Medline | ISI | |
|
Muthén, B. (1994). Multilevel covariance structure analysis. Sociological Methods & Research, 22, 376-398. doi:10.1177/0049124194022003006 Google Scholar | SAGE Journals | ISI | |
|
Muthén, B. (2005, January 20). Negative residual variance. Retrieved from http://www.statmodel.com/discussion/messages/11/555.html?1358188287 Google Scholar | |
|
Muthén, B., Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267-316. doi:10.2307/271070 Google Scholar | Crossref | ISI | |
|
Muthén, L., Muthén, B. (2009). Mplus short courses topic 1: Exploratory factor analysis, confirmatory factor analysis, and structural equation modeling for continuous outcomes. Retrieved from http://www.statmodel.com/download/Topic%201.pdf Google Scholar | |
|
Muthén, L., Muthén, B. (2010). Mplus user’s guide (6th ed.). Los Angeles, CA: Author. Google Scholar | |
|
Myung, I. J. (2000). The importance of complexity in model selection. Journal of Mathematical Psychology, 44, 190-204. doi:10.1006/jmps.1999.1283 Google Scholar | Crossref | Medline | ISI | |
|
Park, N., Peterson, C. (2006). Moral competence and character strengths among adolescents: The development and validation of the values in action inventory of strengths for youth. Journal of Adolescence, 29, 891-909. doi:10.1016/j.adolescence.2006.04.011 Google Scholar | Crossref | Medline | ISI | |
|
Preacher, K. J., Zhang, G., Kim, C., Mels, G. (2013). Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. Multivariate Behavioral Research, 48, 28-56. doi:10.1080/00273171.2012.710386 Google Scholar | Crossref | Medline | ISI | |
|
Raudenbush, S., Bryk, A. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage. Google Scholar | |
|
R Core Team . (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-project.org/ Google Scholar | |
|
Reise, S. P., Ventura, J., Nuechterlein, K. H., Kim, K. H. (2005). An illustration of multilevel factor analysis. Journal of Personality Assessment, 84, 126-136. doi:10.1207/s15327752jpa8402_02 Google Scholar | Crossref | Medline | ISI | |
|
Roth, J. L., Brooks-Gunn, J. (2003). Youth development programs: Risk, prevention, and policy. Journal of Adolescent Health, 32, 170-182. doi:10.1016/S1054-139X(02)00421-4 Google Scholar | Crossref | Medline | ISI | |
|
Schweig, J. (2013). Cross-level measurement invariance in school and classroom environment surveys: Implications for policy and practice. Educational Evaluation and Policy Analysis, 36, 259-280. doi:10.3102/0162373713509880 Google Scholar | SAGE Journals | ISI | |
|
Stapleton, L. (2006). Using multilevel structural equation modeling techniques with complex sample data. In Hancock, G., Mueller, R. (Eds.), Structural equation modeling: A second course (pp. 345-383). Greenwich, CT: Information Age. Google Scholar | |
|
Stevens, J. (2009). Applied multivariate statistics for the social sciences (5th ed.). Mahwah, NJ: Lawrence Erlbaum. Google Scholar | |
|
Thapa, A., Cohen, J., Guffey, S., Higgins-D’Alessandro, A. (2013). A review of school climate research. Review of Educational Research, 83, 357-385. doi:10.3102/0034654313483907 Google Scholar | SAGE Journals | ISI | |
|
Thomas, S. L., Heck, R. H. (2001). Analysis of large-scale secondary data in higher education research: Potential perils associated with complex sampling designs. Research in Higher Education, 42, 517-540. Google Scholar | Crossref | ISI | |
|
Van Horn, M. L. (2003). Assessing the unit of measurement for school climate through psychometric and outcome analyses of the School Climate Survey. Educational and Psychological Measurement, 63, 1002-1019. doi:10.1177/0013164403251317 Google Scholar | SAGE Journals | ISI |
