![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Multidimensional forced-choice formats can significantly reduce the impact of numerous response biases typically associated with rating scales. However, if scored with classical methodology, these questionnaires produce ipsative data, which lead to distorted scale relationships and make comparisons between individuals problematic. This research demonstrates how item response theory (IRT) modeling may be applied to overcome these problems. A multidimensional IRT model based on Thurstone’s framework for comparative data is introduced, which is suitable for use with any forced-choice questionnaire composed of items fitting the dominance response model, with any number of measured traits, and any block sizes (i.e., pairs, triplets, quads, etc.). Thurstonian IRT models are normal ogive models with structured factor loadings, structured uniquenesses, and structured local dependencies. These models can be straightforwardly estimated using structural equation modeling (SEM) software Mplus. A number of simulation studies are performed to investigate how latent traits are recovered under various forced-choice designs and provide guidelines for optimal questionnaire design. An empirical application is given to illustrate how the model may be applied in practice. It is concluded that when the recommended design guidelines are met, scores estimated from forced-choice questionnaires with the proposed methodology reproduce the latent traits well.
|
Ackerman, T.A. ( 2005). Multidimensional item response theory modeling . In A. Maydeu-Olivares & J. J. McArdle (Eds.), Contemporary psychometrics (pp. 3-26). Mahwah, NJ: Lawrence Erlbaum. Google Scholar | |
|
Baron, H. ( 1996). Strengths and limitations of ipsative measurement . Journal of Occupational and Organizational Psychology , 69, 49-56. Google Scholar | Crossref | ISI | |
|
Bartram, D. ( 2007). Increasing validity with forced-choice criterion measurement formats. International Journal of Selection and Assessment , 15, 263-272. Google Scholar | Crossref | |
|
Bock, R.D. ( 1975). Multivariate statistical methods in behavioral research . New York, NY: McGraw-Hill . Google Scholar | |
|
Cheung, M.W.L. , & Chan, W. ( 2002). Reducing uniform response bias with ipsative measurement in multiple-group confirmatory factor analysis. Structural Equation Modeling, 9, 55-77. Google Scholar | Crossref | |
|
Coombs, C.H. ( 1964). A theory of data. New York, NY: Wiley. Google Scholar | |
|
Costa, P.T. , & McCrae, R.R. (1992). NEO-PI-R professional manual . Odessa, FL: Psychological Assessment Resources. Google Scholar | |
|
Du Toit, M. (Ed.). (2003). IRT from SSI. Chicago, IL: SSI Scientific Software International. Google Scholar | |
|
Embretson, S.E. , & Reise, S. ( 2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum. Google Scholar | |
|
Forero, C.G. , Maydeu-Olivares, A. , & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation . Structural Equation Modeling, 16, 625-641. Google Scholar | Crossref | |
|
Friedman, H. , & Amoo, T. ( 1999). Rating the rating scales. Journal of Marketing Management, 9, 114-123. Google Scholar | |
|
Goldberg, L.R. ( 1992). The development of markers for the Big-Five factor structure . Psychological Assessment, 4, 26-42. Google Scholar | Crossref | |
|
Gordon, L.V. ( 1976). Survey of interpersonal values (Revised manual) . Chicago, IL: Science Research Associates. Google Scholar | |
|
Hogan, R. ( 1983). A socioanalytic theory of personality. In M. M. Page (Ed.), Nebraska symposium on motivation (pp. 336-355). Lincoln: University of Nebraska Press. Google Scholar | |
|
International Personality Item Pool: A scientific collaboratory for the development of advanced measures of personality traits and other individual differences . (n.d.). Retrieved from http://ipip.ori.org/ Google Scholar | |
|
Maydeu-Olivares, A. ( 1999). Thurstonian modeling of ranking data via mean and covariance structure analysis. Psychometrika, 64, 325-340. Google Scholar | Crossref | |
|
Maydeu-Olivares, A. , & Böckenholt, U. (2005). Structural equation modeling of paired comparisons and ranking data. Psychological Methods , 10, 285-304. Google Scholar | Crossref | Medline | |
|
Maydeu-Olivares, A. , & Brown, A. (in press). Item response modeling of paired comparison and ranking data . Multivariate Behavioural Research. Google Scholar | |
|
Maydeu-Olivares, A. , & Coffman, D.L. ( 2006). Random intercept item factor analysis. Psychological Methods, 11, 344-362. Google Scholar | Crossref | Medline | ISI | |
|
McCloy, R. , Heggestad, E. , & Reeve, C. ( 2005). A silk purse from the sow’s ear: Retrieving normative information from multidimensional forced-choice items. Organizational Research Methods, 8, 222-248. Google Scholar | SAGE Journals | |
|
McDonald, R.P. ( 1999). Test theory. A unified approach. Mahwah, NJ: Lawrence Erlbaum. Google Scholar | |
|
Meade, A. ( 2004). Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organisational Psychology, 77, 531-552. Google Scholar | Crossref | |
|
Muthén, L.K. , & Muthén, B. (1998-2007). Mplus 5. Los Angeles, CA: Muthén & Muthén . Google Scholar | |
|
Reckase, M. ( 2009). Multidimensional item response theory. New York, NY: Springer. Google Scholar | Crossref | |
|
Samejima, F. ( 1969). Calibration of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement , 17. Google Scholar | |
|
SHL (1997). Customer contact: Manual and user’s guide. Surrey, England. Author. Google Scholar | |
|
SHL. (2006). OPQ32 technical manual. Surrey, UK. Author. Google Scholar | |
|
Stark, S. , Chernyshenko, O. , & Drasgow, F. ( 2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement , 29, 184-203. Google Scholar | SAGE Journals | |
|
Stark, S. , Chernyshenko, O. , Drasgow, F. , & Williams, B. ( 2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring? Journal of Applied Psychology, 91, 25-39. Google Scholar | Crossref | Medline | ISI | |
|
Tenopyr, M.L. ( 1988). Artifactual reliability of forced-choice scales . Journal of Applied Psychology, 73, 749-751. Google Scholar | Crossref | ISI | |
|
Thurstone, L.L. ( 1927). A law of comparative judgment. Psychological Review, 79, 281-299. Google Scholar | |
|
Thurstone, L.L. ( 1931). Rank order as a psychological method. Journal of Experimental Psychology, 14, 187-201. Google Scholar | Crossref | |
|
Van Herk, H. , Poortinga, Y. , & Verhallen, T. ( 2004). Response styles in rating scales: Evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35, 346. Google Scholar | SAGE Journals | ISI |
