![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Multilevel modeling allows researchers to understand whether relationships between lower-level variables (e.g., individual job satisfaction and individual performance, firm capabilities and performance) change as a function of higher-order moderator variables (e.g., leadership climate, market-based conditions). We describe how to estimate such cross-level interaction effects and distill the technical literature for a general readership of management researchers, including a description of the multilevel model building process and an illustration of analyses and results with a data set grounded in substantive theory. In addition, we provide 10 specific best-practice recommendations regarding persistent and important challenges that researchers face before and after data collection to improve the accuracy of substantive conclusions involving cross-level interaction effects. Our recommendations provide guidance on how to define the cross-level interaction effect, compute statistical power and make research design decisions, test hypotheses with various types of moderator variables (e.g., continuous, categorical), rescale (i.e., center) predictors, graph the cross-level interaction effect, interpret interactions given the symmetrical nature of such effects, test multiple cross-level interaction hypotheses, test cross-level interactions involving more than two levels of nesting, compute effect-size estimates and interpret the practical importance of a cross-level interaction effect, and report results regarding the multilevel model building process.
References
|
Aguinis, H. 2004. Regression analysis for categorical moderators. New York: Guilford. Google Scholar | |
|
Aguinis, H., Beaty, J. C., Boik, R. J., Pierce, C. A. 2005. Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90: 94-107. Google Scholar | Medline | ISI | |
|
Aguinis, H., Boyd, B. K., Pierce, C. A., Short, J. C. 2011. Walking new avenues in management research methods and theories: Bridging micro and macro domains. Journal of Management, 37: 395-403. Google Scholar | SAGE Journals | ISI | |
|
Aguinis, H., Gottfredson, R. K., Wright, T. A. 2011. Best-practice recommendations for estimating interaction effects using meta-analysis. Journal of Organizational Behavior, 32: 1033-1043. Google Scholar | ISI | |
|
Aguinis, H., Pierce, C. A., Bosco, F. A., Muslin, I. S. 2009. First decade of Organizational Research Methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12: 69-112. Google Scholar | SAGE Journals | ISI | |
|
Aguinis, H., Werner, S., Abbott, J. L., Angert, C., Park, J. H., Kohlhausen, D. 2010. Customer-centric science: Reporting research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13: 515-539. Google Scholar | SAGE Journals | ISI | |
|
Aiken, L. S., West, S. G. 1991. Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage. Google Scholar | |
|
American Psychological Association . 2009. Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author. Google Scholar | |
|
APA Publications and Communications Board Working Group on Journal Article Reporting Standards . 2008. Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63: 839-851. Google Scholar | Medline | ISI | |
|
Aytug, Z. G., Rothstein, H. R., Zhou, W., Kern, M. C. 2012. Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15: 103-133. Google Scholar | SAGE Journals | ISI | |
|
Barney, J. B., Ketchen, D. J., Wright, M. 2011. The future of resource-based theory: Revitalization or decline? Journal of Management, 37: 1299-1315. Google Scholar | SAGE Journals | ISI | |
|
Bedeian, A. G., Taylor, S. G., Miller, A. N. 2010. Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning and Education, 9: 715-725. Google Scholar | ISI | |
|
Bliese, P. D. 2000. Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In Klein, K. J., Kozlowski, S. W. J. (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions: 349-381. San Francisco: Jossey-Bass. Google Scholar | |
|
Bliese, P. D. 2002. Multilevel random coefficient modeling in organizational research: Examples using SAS and S-PLUS. In Drasgow, F., Schmitt, N. (Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis: 401-445. San Francisco: Jossey-Bass. Google Scholar | |
|
Bliese, P. D., Hanges, P. J. 2004. Being both too liberal and too conservative: The perils of treating grouped data as though they were independent. Organizational Research Methods, 7: 400-417. Google Scholar | SAGE Journals | ISI | |
|
Bosker, R. J., Snijders, T. A. B., Guldemond, H. 2003. PINT (Power IN Two-level designs): Estimating standard errors of regression coefficients in hierarchical linear models for power calculations (Version 2.1). Groningen, Netherlands: Rijksuniversiteit Groningen. Google Scholar | |
|
Brutus, S., Aguinis, H., Wassmer, U. 2013. Self-reported limitations and future directions in scholarly reports: Analysis and recommendations. Journal of Management, 39: 48-75. Google Scholar | SAGE Journals | ISI | |
|
Carpenter, J. R., Goldstein, H., Rasbash, J. 2003. A novel bootstrap procedure for assessing the relationship between class size and achievement. Applied Statistics, 52: 431-443. Google Scholar | ISI | |
|
Cascio, W. F., Aguinis, H. 2008. Staffing twenty-first-century organizations. Academy of Management Annals, 2: 133-165. Google Scholar | ISI | |
|
Chen, G., Kirkman, B. L., Kanfer, R., Allen, D., Rosen, B. 2007. A multilevel study of leadership, empowerment, and performance in teams. Journal of Applied Psychology, 92: 331-346. Google Scholar | Medline | ISI | |
|
Covin, J. G., Wales, W. J. 2012. The measurement of entrepreneurial orientation. Entrepreneurship Theory and Practice, 36: 677-702. Google Scholar | SAGE Journals | ISI | |
|
Crainiceanu, C., Ruppert, D. 2004. Likelihood ratio tests in linear mixed models with one variance component. Journal of the Royal Statistical Society-B, 66: 165-185. Google Scholar | |
|
Culpepper, S. A. 2010. Studying individual differences in predictability with gamma regression and nonlinear multilevel models. Multivariate Behavioral Research, 45: 153-185. Google Scholar | Medline | ISI | |
|
Culpepper, S. A., Aguinis, H. 2011. Using analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16: 166-178. Google Scholar | Medline | ISI | |
|
Dalal, D. K., Zickar, M. J. 2012. Some common myths about centering predictor variables in moderated multiple regression and polynomial regression. Organizational Research Methods, 15: 339-362. Google Scholar | SAGE Journals | ISI | |
|
Dalton, D. R., Aguinis, H. 2013. Measurement malaise in strategic management studies: The case of corporate governance research. Organizational Research Methods, 16: 88-99. Google Scholar | SAGE Journals | ISI | |
|
Dalton, D. R., Aguinis, H., Dalton, C. A., Bosco, F. A., Pierce, C. A. 2012. Revisiting the file drawer problem in meta-analysis: An empirical assessment of published and non-published correlation matrices. Personnel Psychology, 65: 221-249. Google Scholar | ISI | |
|
Davison, M. L., Kwak, N., Seo, Y. S., Choi, J. 2002. Using hierarchical linear models to examine moderator effects: Person-by-organization interactions. Organizational Research Methods, 5: 231-254. Google Scholar | SAGE Journals | ISI | |
|
Enders, C. K., Tofighi, D. 2007. Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12: 121-138. Google Scholar | Medline | ISI | |
|
Field, C. A., Welsh, A. H. 2007. Bootstrapping clustered data. Journal of the Royal Statistical Society-B, 69: 369-390. Google Scholar | |
|
Gelfand, A. E., Sahu, S. K., Carlin, B. P. 1995. Efficient parametrisations for normal linear mixed models. Biometrika, 82: 479-488. Google Scholar | ISI | |
|
Han, J. 2005. Crossover linear modeling: Combining multilevel heterogeneities in crossover relationships. Organizational Research Methods, 8: 290-316. Google Scholar | SAGE Journals | ISI | |
|
Hedges, L. V., Hedberg, E. C. 2007. Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29: 60-87. Google Scholar | SAGE Journals | ISI | |
|
Hofmann, D. A., Gavin, M. B. 1998. Centering decisions in hierarchical linear models: Theoretical and methodological implications for organizational science. Journal of Management, 24: 623-641. Google Scholar | SAGE Journals | ISI | |
|
Hox, J. 2010. Multilevel analysis: Techniques and applications (2nd ed.). New York: Routledge. Google Scholar | |
|
Kahn, J. H. 2011. Multilevel modeling: Overview and applications to research in counseling psychology. Journal of Counseling Psychology, 58: 257-271. Google Scholar | Medline | ISI | |
|
Kenny, D. A., Judd, C. M. 1996. A general procedure for the estimation of interdependence. Psychological Bulletin, 119: 138-148. Google Scholar | Medline | ISI | |
|
Konstantopolous, S. 2008a. The power of the test for treatment effects in three-level block randomized designs. Journal of Research on Educational Effectiveness, 1: 265-288. Google Scholar | |
|
Konstantopolous, S. 2008b. The power of the test for treatment effects in three-level cluster randomized designs. Journal of Research on Educational Effectiveness, 1: 66-88. Google Scholar | |
|
Kozlowski, S. W. J., Klein, K. J. 2000. A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In Klein, K. J., Kozlowski, S. W. J. (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions: 3-90. San Francisco: Jossey-Bass. Google Scholar | |
|
Kreft, I., De Leeuw, J. 1998. Introducing multilevel modeling. Thousand Oaks, CA: Sage. Google Scholar | |
|
Liden, R. C., Antonakis, J. 2009. Considering context in psychological leadership research. Human Relations, 62: 1587-1605. Google Scholar | SAGE Journals | ISI | |
|
Maas, C. J. M., Hox, J. J. 2004. The influence of violations of assumptions on multilevel parameter estimates and their standard errors. Computational Statistics & Data Analysis, 46: 427-440. Google Scholar | ISI | |
|
Mathieu, J. E., Aguinis, H., Culpepper, S. A., Chen, G. 2012. Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. Journal of Applied Psychology, 97: 951-966. Google Scholar | Medline | ISI | |
|
Mathieu, J. E., Chen, G. 2011. The etiology of the multilevel paradigm in management research. Journal of Management, 37: 610-641. Google Scholar | SAGE Journals | ISI | |
|
Maxwell, S. E. 2004. The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9: 147-163. Google Scholar | Medline | ISI | |
|
Molloy, J. C., Ployhart, R. E., Wright, P. M. 2011. The myth of “the” micro-macro divide: Bridging system-level and disciplinary divides. Journal of Management, 37: 581-609. Google Scholar | SAGE Journals | ISI | |
|
Peugh, J. L. 2010. A practical guide to multilevel modeling. Journal of School Psychology, 48: 85-112. Google Scholar | Medline | ISI | |
|
Pierce, J. R., Aguinis, H. 2013. The too-much-of-a-good-thing effect in management. Journal of Management, 39: 313-338. Google Scholar | SAGE Journals | ISI | |
|
Preacher, K. J., Curran, P. J., Bauer, D. J. 2006. Computational tools for probing interaction effects in multiple linear regression, multilevel modeling, and latent curve analysis. Journal of Educational and Behavioral Statistics, 31: 437-448. Google Scholar | SAGE Journals | ISI | |
|
Preacher, K. J., Zyphur, M. J., Zhang, Z. 2010. A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15: 209-233. Google Scholar | Medline | ISI | |
|
Raudenbush, S. W. 1997. Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2: 173-185. Google Scholar | ISI | |
|
Raudenbush, S. W., Bryk, A. S. 2002. Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage. Google Scholar | |
|
Raudenbush, S. W., Liu, X. 2000. Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5: 199-213. Google Scholar | Medline | ISI | |
|
Raudenbush, S. W., Rowan, B., Cheong, Y. F. 1993. The pursuit of higher-order instructional goals in secondary schools: Class, teacher, and school influences. American Educational Research Journal, 30: 523-553. Google Scholar | SAGE Journals | ISI | |
|
Roberts, J. K., Monaco, J. P., Stovall, H., Foster, V. 2011. Explained variance in multilevel models. In Hox, J. J., Roberts, J. K. (Eds.), Handbook of advanced multilevel analysis: 219-230. New York: Routledge. Google Scholar | |
|
Scheipl, F., Greven, S., Kuechenhoff, H. 2008. Size and power of tests for a zero random effect variance or polynomial regression in additive and linear mixed models. Computational Statistics & Data Analysis, 52: 3283-3299. Google Scholar | ISI | |
|
Scherbaum, C. A., Ferreter, J. M. 2009. Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organizational Research Methods, 12: 347-367. Google Scholar | SAGE Journals | ISI | |
|
Singer, J. D. 1998. Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics, 24: 323-355. Google Scholar | SAGE Journals | |
|
Snijders, T. A. B. 2005. Power and sample size in multilevel linear models. In Everitt, B. S., Howell, D. C. (Eds.), Encyclopedia of statistics in behavioral science, vol. 3: 1570-1573. Chichester, UK: Wiley. Google Scholar | |
|
Snijders, T. A. B., Bosker, R. J. 2012. Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). Los Angeles, CA: Sage. Google Scholar | |
|
Spybrook, J., Bloom, H., Congdon, R., Hill, C., Martinez, A., Raudenbush, S. W. 2011. Optimal design plus empirical evidence: Documentation for the “Optimal Design” software (version 3.0). Retrieved from http://www.wtgrantfoundation.org/resources/consultation-service-and-optimal-design. Google Scholar | |
|
Stram, D., Lee, J. W. 1994. Variance components testing in the longitudinal mixed effects model. Biometrics, 50: 1171-1177. Google Scholar | Medline | ISI | |
|
Uy, M. A., Foo, M. D., Aguinis, H. 2010. Using event sampling methodology to advance entrepreneurship theory and research. Organizational Research Methods, 13: 31-54. Google Scholar | SAGE Journals | ISI | |
|
Wolfinger, R., O’Connell, M. 2007. Generalized linear mixed models: A pseudo-likelihood approach. Journal of Statistical Computation and Simulation, 48: 233-243. Google Scholar | ISI |
