![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
This small n observational study used a concurrent mixed methods approach to address a void in the literature with regard to the qualitative meaningfulness of the data yielded by absolute magnitude estimation scaling (MES) used to rate subjective stimuli. We investigated whether respondents’ scales progressed from less to more and provided for meaningful discriminations, similarities and differences in how they defined their scales, and whether their definitions were congruent with how they used their scales. The respondents completed a think-aloud task while using MES and a follow-up interview regarding how they used MES and defined their scales. The respondents’ scales did not yield meaningful data. Implications for survey research and the use of mixed methods in rating scale construction are discussed.
References
Section:
|
Algom, D., Marks, L. E. (1984). Individual differences in loudness processing and loudness scales. Journal of Experimental Psychology: General, 113, 571-593. Google Scholar | Crossref | Medline | ISI | |
|
Alwin, D. F. (2001). Research on survey quality. Sociological Methods and Research, 20, 1-39. Google Scholar | |
|
Bard, E. G., Robertson, D., Sorace, A. (1996). Magnitude estimation of linguistic acceptability. Language, 72, 32-68. Google Scholar | Crossref | ISI | |
|
Bass, B. M., Cascio, W. F., O’Connor, E. J. (1974). Magnitude estimations of expressions of frequency and amount. Journal of Applied Psychology, 59, 313-320. Google Scholar | Crossref | ISI | |
|
Beltyukova, S. A., Stone, G. M., Ellis, L. W. (2008). Rasch analysis of word identification and magnitude estimation responses in measuring naïve listeners’ judgments of speech intelligibility of children with severe-to-profound hearing impairments. Journal of Speech, Language, and Hearing Research, 51, 1124-1137. Google Scholar | Crossref | Medline | ISI | |
|
Beltyukova, S. A., Stone, G. E., Fox, C. M. (2008). Magnitude estimation and categorical rating scaling in social sciences: A theoretical and psychometric controversy. Journal of Applied Measurement, 9, 151-159. Google Scholar | Medline | |
|
Bond, T. G., Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum. Google Scholar | |
|
Borsboom, D. (2005). The concept of validity. In Borsboom, D. (Ed.), Measuring the mind: Conceptual issues in contemporary psychometrics (pp. 149-172). New York, NY: Cambridge University Press. Google Scholar | Crossref | |
|
Borsboom, D., Mellenbergh, G. J., Heerden, J. V. (2004). The concept of validity. Psychological Review, 111, 1061-1071. Google Scholar | Crossref | Medline | ISI | |
|
Boverie, P. E., Scheuffele, D. J., Raymond, E. L. (1995). Multimethodological approach to examining risk-taking. Current Psychology: Developmental, Learning, Personality, & Social, 13, 289-302. Google Scholar | Crossref | ISI | |
|
Chang, L. (1996). Quantitative attitudes questionnaire: Instrument development and validation. Educational & Psychological Measurement, 56, 1037-1042. Google Scholar | SAGE Journals | ISI | |
|
Creswell, J. W. (2007). Qualitative inquiry and research design: Choosing among five approaches (2nd ed.). Thousand Oaks, CA: Sage. Google Scholar | |
|
Eadie, T. L., Doyle, P. C. (2002a). Direct magnitude estimation and interval scaling of pleasantness and severity in dysphonic and normal speakers. Journal of Acoustical Society of America, 112, 3014-3021. Google Scholar | Crossref | Medline | ISI | |
|
Eadie, T. L., Doyle, P. C. (2002b). Direct magnitude estimation and interval scaling of naturalness and severity in tracheoesophageal (TE) speakers. Journal of Speech, Language, and Hearing Research, 45, 1088-1096. Google Scholar | Crossref | Medline | ISI | |
|
Eisenberg, M., Barry, C. (1988). Order effects: A study of the possible influence of presentation order on user judgments of document relevance. Journal of the American Society for Information Science, 39, 293-300. Google Scholar | Crossref | |
|
Ellis, L., Fucci, D. (1991). Magnitude-estimation scaling of speech intelligibility: Effects of listeners' experience and semantic-syntactic context. Perceptual and Motor Skills, 73, 295-305. Google Scholar | SAGE Journals | ISI | |
|
Ericsson, K. A., Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215-251. Google Scholar | Crossref | ISI | |
|
Falmagne, J. C., Narens, L. (1983). Scales and meaningfulness of quantitative laws. Synthese, 55, 287-325. Google Scholar | Crossref | ISI | |
|
Fisher, W. P. (2004). Meaning and method in the social sciences. Human Studies, 27, 429-454. Google Scholar | Crossref | ISI | |
|
Fucci, D., Domyan, S., Ellis, L., Harris, D. (1994). Magnitude-estimation scaling: An effective method for the measurement of the quality of filtered speech. Perceptual and Motor Skills, 78, 348-350. Google Scholar | SAGE Journals | ISI | |
|
Fucci, D., Petrosino, L., Banks, M., Zaums, K., Wilcox, C. (1996). The effect of preference for three different types of music on magnitude estimation-scaling behavior in young adults. Perceptual and Motor Skills, 83, 339-347. Google Scholar | SAGE Journals | ISI | |
|
Hamblin, R. L. (1974). Social attitudes: Magnitude measurement and theory. In Blalock, H. M. (Ed.), Measurement in the social sciences (pp. 61-120). Chicago, IL: Aldine Press. Google Scholar | Crossref | |
|
Harris, D., Fucci, D., Petrosino, L., Wallace, D. (1986, September). Instrumentation for magnitude estimation and cross-modality matching of auditory and lingual vibrotactile sensations. Review of Scientific Instruments, 57, 2343-2347. Google Scholar | Crossref | ISI | |
|
Hellman, R. P., Zwislocki, J. (1963). Monaural loudness function at 1000 cps and interaural summation. Journal of the Acoustical Society of America, 35, 856-865. Google Scholar | Crossref | ISI | |
|
Hinshaw, A. S., Schepp, K. (1983). Territorial intrusion and client outcomes: The use of magnitude estimation. Communicating Nursing Research, 19, 77. Google Scholar | |
|
Horowitz, A. J. (1981). Subjective value of time in bus transit travel. Transportation, 10, 149-164. Google Scholar | Crossref | ISI | |
|
Institute of Objective Measurement (2000, December). Definition of objective measurement. Retrieved from http://www.rasch.org/define.htm Google Scholar | |
|
Kinney, M., Guzzetta, C. E. (1989). Identifying critical defining characteristics of nursing diagnoses using magnitude estimation scaling. Research in Nursing & Health, 12, 373-380. Google Scholar | Crossref | Medline | ISI | |
|
Kluge, S. (2000). Empirically grounded construction of types and typologies in qualitative Social research. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 1(1). Retrieved from http://www.qualitative-research.net/index.php/fqs/article/view/1124/2499 Google Scholar | |
|
Koskey, K. L. K. (2009). A mixed-method approach to examining the use of absolute magnitude estimation scaling using the Many-Facet Rasch model and an instrumental case study (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3383715) Google Scholar | |
|
Koskey, K. L. K., Sondergeld, T. A., Beltyukova, S. A., Fox, C. M. (2008). Is absolute magnitude estimation scaling a viable alternative to categorical rating scaling in social science? An experimental study. Paper presented at the American Educational Research Association Annual Meeting, New York, NY. Google Scholar | |
|
Lawless, H. T., Horne, J., Spiers, W. (2000). Contrast and range effects for category, magnitude and labeled magnitude scales in judgements of sweetness intensity. Chemical Senses, 25, 85-92. Google Scholar | Crossref | Medline | ISI | |
|
Linacre, M. J. (1994). Many-facet Rasch measurement. Chicago, IL: MESA Press. Google Scholar | |
|
Linacre, M. J. (1999). Investigating rating scale category utility. Journal of Outcome Measurement, 3, 103-122. Google Scholar | Medline | |
|
Linacre, M. J. (2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3, 85-106. Google Scholar | Medline | |
|
Linacre, M. J. (2005). A user’s guide to Winsteps/Ministeps Rasch-model program. Chicago, IL: MESA Press. Google Scholar | |
|
Linacre, M. J. (2006). FACETS: Rasch measurement computer program. Chicago, IL: MESA Press. Google Scholar | |
|
Linacre, M. J. (2008). A user’s guide to FACETS: Rasch-model computer programs. Chicago, IL: MESA Press. Google Scholar | |
|
Linacre, M. J., Wright, B. D. (2002). Understanding Rasch measurement: Construction of measures from many-facet data. Journal of Applied Measurement, 4, 486-512. Google Scholar | |
|
Lodge, M. (1981). Magnitude scaling: Quantitative measurement of opinions. Beverly Hills, CA: Sage. Google Scholar | Crossref | |
|
McColl, D., Fucci, D. (1999). Comparison of magnitude estimation and interval scaling of loudness. Perceptual and Motor Skills, 88, 25-30. Google Scholar | SAGE Journals | ISI | |
|
McColl, D., Fucci, D. (2006). Measurement of speech disfluency through magnitude estimation and interval scaling. Perceptual and Motor Skills, 102, 454-460. Google Scholar | SAGE Journals | ISI | |
|
Meek, P. M., Sennott-Miller, L., Ferdetich, S. L. (1992). Focus on psychometrics: Scaling stimuli with magnitude estimation. Research in Nursing & Health, 15, 77-81. Google Scholar | Crossref | Medline | ISI | |
|
Mundy, B. (1986). On the general theory of meaningful representation. Synthese, 67, 391-437. Google Scholar | Crossref | ISI | |
|
Onwuegbuzie, A. J., Collins, K. M. T. (2007). The typology of mixed methods sampling designs in social science research. Qualitative Report, 12, 281-316. Google Scholar | |
|
Opfer, J. E., Devries, J. M. (2008). Representational change and magnitude estimation: Why young children can make more accurate salary comparisons than adults. Cognition, 108, 843-849. Google Scholar | Crossref | Medline | ISI | |
|
Orth, B., Wegener, B. (1983). Scaling occupational prestige by magnitude estimation and category rating methods: A comparison with the sensory domain. European Journal of Social Psychology, 13, 417-431. Google Scholar | Crossref | ISI | |
|
Petrosino, L., Fucci, D., Ellis, L., Harris, D. (1989). Effects of auditory masking on lingual vibrotactile magnitude-estimation scaling responses of young children. Perceptual & Motor Skills, 68, 523-529. Google Scholar | SAGE Journals | ISI | |
|
Poulton, E. C. (1968). The new psychophysics: Six models for magnitude estimation. Psychological Bulletin, 69(1), 1-19. Google Scholar | Crossref | ISI | |
|
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark, Netherlands: Danmarks Paedagogiske Institut. Google Scholar | |
|
Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests (Expanded ed.). Chicago, IL: University of Chicago Press. Google Scholar | |
|
Roberts, F. S. (1979). Measurement theory. Reading, MA: Addison-Wesley. Google Scholar | |
|
Schriesheim, C. A., Gardiner, C. C. (1992). A comparative test of magnitude estimation and pair-comparison treatment of complete ranks for scaling a small number of equal interval frequency response anchors. Educational and Psychological Measurement, 52, 876-877. Google Scholar | SAGE Journals | ISI | |
|
Sennott-Miller, L., Murdaugh, C., Hinshaw, A. S. (1988). Magnitude estimation: Issues and practical applications. Western Journal of Nursing Research, 10, 414-424. Google Scholar | SAGE Journals | |
|
Siegler, R. S., Opfer, J. E. (2003). The development of numerical estimation: Evidence for multiple representation of numerical quantity. Psychological Science, 14, 237-243. Google Scholar | SAGE Journals | ISI | |
|
Southwood, M. H., Flege, J. E. (1999). Scaling foreign accent: Direct magnitude estimation versus interval scaling. Clinical Linguistics & Phonetics, 13, 335-349. Google Scholar | Crossref | ISI | |
|
Stake, R. E. (1995). The unique case. In Stake, R. E. (Ed.), The art of case study research (pp. 1-13). Thousand Oaks, CA: Sage. Google Scholar | |
|
Stevens, S. S. (1951). Mathematics, measurement, and psychophysics. In Stevens, S. S. (Ed.), Handbook of experimental psychology (pp. 1-49). New York, NY: Wiley. Google Scholar | |
|
Stevens, S. S. (1972). Psychophysics and social scaling. Morristown, NJ: General Learning Press. Google Scholar | |
|
Stevens, S. S. (1975). Psychophysics: Introduction to its perceptual, neural, and social prospects. New York, NY: Wiley. Google Scholar | |
|
Sudman, S., Bradburn, N., Schwarz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass. Google Scholar | |
|
Tashakkori, A., Teddlie, C. (1998). Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage. Google Scholar | |
|
Teddlie, C., Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks, CA: Sage. Google Scholar | |
|
Tesler, M. D., Savedra, M. C., Holzemer, W. L., Wilkie, D. J., Ward, J. A., Paul, S. M. (1991). The word-graphic rating scale as a measure of children’s and adolescents’ pain intensity. Research in Nursing & Health, 14, 361-371. Google Scholar | Crossref | Medline | ISI | |
|
Thurstone, L. L. (1974). The measurement of values. Chicago, IL: Chicago Press. (Original work published 1959) Google Scholar | |
|
Torgerson, W. S. (1958). The subjective estimate methods. In Torgerson, W. S. (Ed.), Theory and methods of scaling (pp. 61-93). New York, NY: John Wiley. Google Scholar | |
|
Viswanathan, M., Sudman, S., Johnson, M. (2004). Maximum versus meaningful discrimination in scale responses: Implications for validity of measurement of consumers’ perceptions about products. Journal of Business Research, 57, 108-124. Google Scholar | Crossref | ISI | |
|
Walker, B. (2002). Magnitude estimation of conceptual data dimensions for use in sonification. Journal of Experimental Psychology, 8, 211-221. Google Scholar | |
|
Wegener, B. (1983). Category-rating and magnitude estimation scaling techniques: An empirical comparison. Sociological methods & Research, 12, 31-75. Google Scholar | SAGE Journals | ISI | |
|
Willis, G. B. (2004). Cognitive interviewing revisited: A useful technique, in theory? In Presser, S., Rothgeb, J. M., Couper, M. P., Lesser, J. T., Martin, E., Martin, J., Singer, E. (Eds.), Methods for testing and evaluating survey questionnaires (pp. 23-43). New York, NY: John Wiley. Google Scholar | Crossref | |
|
Woolley, M. E., Bowen, G. L., Bowen, N. K. (2006). The development and evaluations of procedures to assess child self-report validity. Educational and Psychological Measurement, 66, 687-700. Google Scholar | SAGE Journals | ISI | |
|
Wright, B. D., Masters, G. N. (1982). Rating scale analysis. Chicago, IL: MESA Press. Google Scholar | |
|
Wright, B. D., Stone, M. H. (1979). Best test design: Rasch measurement. Chicago, IL: MESA Press. Google Scholar | |
|
Zraick, R. I., Liss, J. M. (2000). A comparison of equal-appearing interval scaling and direct magnitude estimation of nasal voice quality. Journal of Speech, Language, and Hearing Research, 43, 979-988. Google Scholar | Crossref | Medline | ISI | |
|
Zwislocki, J. J., Goodman, D. A. (1980). Absolute scaling of sensory magnitudes: A validation. Perceptions and Psychophysics, 28(1), 28-38. Google Scholar | Crossref | Medline |
