Test-Retest Reliability in Metric Conjoint Experiments: A New Workflow to Evaluate Confidence in Model Results
Abstract
Metric conjoint studies are a popular research design in the entrepreneurship domain. For these studies, test-retest reliabilities of ρ > .70 or higher are an often-cited reliability criterion. Despite their widespread use, however, there is little rigorous analysis of whether test-retest reliability in metric conjoint studies relates to model efficacy. Informed by a systematic literature review, we conducted two Monte Carlo simulations to evaluate the effects of various determinants of test-retest reliability in conjoint experiments. We then illustrate a workflow for entrepreneurship researchers employing conjoint designs to better evaluate—and communicate—confidence in statistical models estimated from conjoint data.
Get full access to this article
View all access and purchase options for this article.
References
Aguinis H., Gottfredson R. K., Culpepper S. A. (2013). Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling. Journal of Management, 39(6), 1490–1528.
Aiman-Smith L., Scullen S. E., Barr S. H. (2002). Conducting studies of decision making in organizational contexts: A tutorial for policy-capturing and other regression-based techniques. Organizational Research Methods, 5(4), 388–414.
Anderson B. S., Schueler J., Baum M., Wales W. J., Gupta V. K. (2022). The chicken or the egg? Causal inference in entrepreneurial orientation–performance research. Entrepreneurship Theory and Practice, 46(6), 1569–1596.
Antonakis J., Bendahan S., Jacquart P., Lalive R. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21(6), 1086–1120.
Baron R. A. (2006). Opportunity recognition as pattern recognition: How entrepreneurs “connect the dots” to identify new business opportunities. The Academy of Management Perspectives, 20(1), 104–119.
Baumgartner T. A. (2000). Estimating the stability reliability of a score. Measurement in Physical Education and Exercise Science, 4(3), 175–178.
Bliese P. D., Chan D., Ployhart R. E. (2007). Multilevel methods. Organizational Research Methods, 10(4), 551–563.
Bowling N. A., Gibson A. M., Houpt J. W., Brower C. K. (2021). Will the questions ever end? Person-level increases in careless responding during questionnaire completion. Organizational Research Methods, 24(4), 718–738.
Busenitz L. W., Barney J. B. (1997). Differences between entrepreneurs and managers in large organizations: Biases and heuristics in strategic decision-making. Journal of Business Venturing, 12(1), 9–30.
Carmines E., Zeller R. (1979). Reliability and validity assessment. SAGE Publications.
Clogg C. C., Petkova E., Haritou A. (1995). Statistical methods for comparing regression coefficients between models. American Journal of Sociology, 100(5), 1261–1293.
Cronbach L. J. (1947). Test “reliability”: Its meaning and determination. Psychometrika, 12(1), 1–16.
DeSimone J. A. (2015). New techniques for evaluating temporal consistency. Organizational Research Methods, 18(1), 133–152.
Gelman A., Hill J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.
Gelman A., Stern H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328–331.
Goldfeld K., Wujciak-Jens J. (2020). simstudy: Illuminating research methods through data generation. Journal of Open Source Software, 5(54), 2763.
Graham M. E., Cable D. M. (2001). Consideration of the incomplete block design for policy-capturing research. Organizational Research Methods, 4(1), 26–45.
Green P. E., Srinivasan V. (1978). Conjoint analysis in consumer research: issues and outlook. The Journal of Consumer Research, 5(2), 103.
Grégoire D. A., Binder J. K., Rauch A. (2019). Navigating the validity tradeoffs of entrepreneurship research experiments: A systematic review and best-practice suggestions. Journal of Business Venturing, 34(2), 284–310.
Gunst R. F., Mason R. L. (2009). Fractional factorial design. Wiley Interdisciplinary Reviews: Computational Statistics, 1(2), 234–244.
Guttman L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10, 255–282.
Hensher D. A. (2006). How do respondents process stated choice experiments? Attribute consideration under varying information load. Journal of Applied Econometrics (Chichester, England), 21(6), 861–878.
Hess S., Hensher D. A., Daly A. (2012). Not bored yet—revisiting respondent fatigue in stated choice experiments. Transportation Research Part A: Policy and Practice, 46(3), 626–644.
Hyslop D. R., Imbens G. W. (2001). Bias from classical and other forms of measurement error. Journal of Business & Economic Statistics, 19(4), 475–481.
Kam C. C. S., Meyer J. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18(3), 512–541.
Karren R. J., Barringer M. W. (2002). A review and analysis of the policy-capturing methodology in organizational research: Guidelines for research and practice. Organizational Research Methods, 5(4), 337–361.
Koo T. K., Li M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.
Lance C. E., Butts M. M., Michels L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9(2), 202–220.
Lee B.-K., Lee W.-N. (2004). The effect of information overload on consumer choice quality in an online environment. Psychology & Marketing, 21(3), 159–183.
Leigh T. W., MacKay D. B., Summers J. O. (1984). Reliability and validity of conjoint analysis and self-explicated weights: A comparison. Journal of Marketing Research, 21(4), 456–462.
Lohrke F. T., Holloway B. B., Woolley T. W. (2010). Conjoint analysis in entrepreneurship research: A review and research agenda. Organizational Research Methods, 13(1), 16–30.
Lurie N. H. (2004). Decision making in information-rich environments: The role of information structure. The Journal of Consumer Research, 30(4), 473–486.
Maula M., Stam W. (2020). Enhancing rigor in quantitative entrepreneurship research. Entrepreneurship Theory and Practice, 44(6), 1059–1090.
McCullough J., Best R. (1979). Conjoint measurement: Temporal stability and structural reliability. Journal of Marketing Research, 16(1), 26–31.
McGraw K. O., Wong S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
Moser K. J., Tumasjan A., Welpe I. M. (2017). Small but attractive: Dimensions of new venture employer attractiveness and the moderating role of applicants’ entrepreneurial behaviors. Journal of Business Venturing, 32(5), 588–610.
Nunnally J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
Reibstein D., Bateson J. E. G., Boulding W. (1988). Conjoint analysis reliability: Empirical findings. Marketing Science, 7(3), 271–286.
Scherbaum C. A., Ferreter J. M. (2009). Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organizational Research Methods, 12(2), 347–367.
Schmidt F. L., Hunter J. E. (1999). Theory testing and measurement error. Intelligence, 27(3), 183–198.
Schwarz N., Oyserman D. (2001). Asking questions about behavior: Cognition, communication, and questionnaire construction. The American Journal of Evaluation, 22(2), 127–160.
Shane S., Venkataraman S. (2000). The promise of entrepreneurship as a field of research. Academy of Management Review, 25(1), 217–226.
Shepherd D. A., Zacharakis A. (2018). Conjoint analysis: A window of opportunity for entrepreneurship research. In Katz J. A., Corbett A. (Eds.), Reflections and extensions on key papers of the first twenty-five years of advances (pp. 149–183). Emerald Publishing Limited.
Shrout P. E., Fleiss J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.
Sireci S. G., Sukin T. (2013). Test validity. In Geisinger K. F., Bracken B. A., Carlson J. F., Hansen J.-I. C., Kuncel N. R., Reise S. P., Rodriguez M. C. (Eds.), APA handbook of testing and assessment in psychology, Vol. 1. Test theory and testing and assessment in industrial and organizational psychology (pp. 61–84). American Psychological Association. https://doi.org/10.1037/14047-004
Tay L., Drasgow F. (2012). Theoretical, statistical, and substantive issues in the assessment of construct dimensionality: Accounting for the item response process. Organizational Research Methods, 15(3), 363–384.
Tourangeau R., Lance J. R., Rasinski K. (2000). The psychology of survey response. Cambridge University Press.
Tyszka T. (1998). Two pairs of conflicting motives in decision making. Organizational Behavior and Human Decision Processes, 74(3), 189–211.
van Rosmalen J., van Herk H., Groenen P. J. F. (2010). Identifying response styles: A latent-class bilinear multinomial logit model. Journal of Marketing Research, 47(1), 157–172.
Weir J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research/National Strength & Conditioning Association, 19(1), 231–240.
Wennberg K., Anderson B. S. (2020). Editorial: Enhancing the exploration and communication of quantitative entrepreneurship research. Journal of Business Venturing, 35(3), 105938.
Zhu Z., Tomassetti A. J., Dalal R. S., Schrader S. W., Loo K., Sabat I. E., Alaybek B., Zhou Y., Jones C., Fyffe S. (2022). A test-retest reliability generalization meta-analysis of judgments via the policy-capturing technique. Organizational Research Methods, 25(3), 541–574.
Biographies
Jens Schüler is a postdoctoral researcher at the Chair of Entrepreneurship and Digital Business Models at the Department of Law, Business and Economics at the University of Bayreuth, Germany.
Brian S. Anderson is Dean’s Professor and Professor of Entrepreneurship at the University of Kansas School of Business.
Charles Y. Murnieks is the A. Gottlieb Chair of Strategic Management and Associate Professor at the Bloch School at the University of Missouri-Kansas City.
Matthias Baum is a professor and holds the Chair of Entrepreneurship and Digital Business Models at the Department of Law, Business and Economics at the University of Bayreuth, Germany.
Alexander Küsshauer is a Startup Consultant with Rheinische Friedrich-Wilhelms-Universität Bonn, Transfer Center enaCom, Bonn, Germany.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: July 4, 2023
Issue published: March 2024
Keywords
Authors
Metrics and citations
Metrics
Journals metrics
This article was published in Entrepreneurship Theory and Practice.
View All Journal MetricsArticle usage*
Total views and downloads: 853
*Article usage tracking started in December 2016
Altmetric
See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 0 view articles Opens in new tab
Crossref: 0
There are no citing articles to show.
Figures and tables
Figures & Media
Tables
View Options
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
I am signed in as:
View my profileSign out
I can access personal subscriptions, purchases, paired institutional access and free tools such as favourite journals, email alerts and saved searches.
loading institutional access options
USASBE members can access this journal content using society membership credentials.
USASBE members can access this journal content using society membership credentials.
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
