Skip to main content
Intended for healthcare professionals
Restricted access
Other
First published online April 27, 2010

Item Selection and Hypothesis Testing for the Adaptive Measurement of Change

Matthew D. Finkelman, David J. Weiss, and Gyenam Kim-KangView all authors and affiliations
Volume 34, Issue 4
https://doi.org/10.1177/0146621609344844

Abstract

Assessing individual change is an important topic in both psychological and educational measurement. An adaptive measurement of change (AMC) method had previously been shown to exhibit greater efficiency in detecting change than conventional nonadaptive methods. However, little work had been done to compare different procedures within the AMC framework. This study introduced a new item selection criterion and two new test statistics for detecting change with AMC that were specifically designed for the paradigm of hypothesis testing. In two simulation sets, the new methods for detecting significant change improved on existing procedures by demonstrating better adherence to Type I error rates and substantially better power for detecting relatively small change.

Get full access to this article

View all access and purchase options for this article.

Get Access

References

Agresti, A. ( 1996). An introduction to categorical data analysis. New York: John Wiley.
Google Scholar
Andersen, E.B. ( 1985). Estimating latent correlations between repeated testings . Psychometrika, 50, 3-16.
Google Scholar
Bartroff, J., Finkelman, M., & Lai, T.L. ( 2008). Modern sequential analysis and its applications to computerized adaptive testing. Psychometrika, 73, 473-486.
Google Scholar
Bereiter, C. ( 1963). Some persisting dilemmas in the measurement of change . In C. Harris (Ed.), Problems in measuring change (pp. 3-20). Madison: University of Wisconsin Press.
Google Scholar
Bock, R.D. ( 1976). Basic issues in the measurement of change. In D. N. M. de Gruijter & L. J. T. van der Kamp (Eds.), Advances in psychological and educational measurement (pp. 75-96). New York: John Wiley.
Google Scholar
Burr, J.A., & Nesselroade, J.R. (1990). Change measurement. In A. von Eye (Ed.), Statistical methods in longitudinal research (vol. 1) (pp. 3-34). Boston: Academic Press.
Google Scholar
Chang, H.H., Qian, J., & Ying, Z. ( 2001). a-stratified multistage CAT with b-blocking. Applied Psychological Measurement, 25, 333-341.
Google Scholar
Chang, H.H., & Stout, W. ( 1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37-52.
Google Scholar
Chang, H.H., & Ying, Z. ( 1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.
Google Scholar
Collins, L. M., & Sayer, A. G. (Eds.). (2001). New methods for the analysis of change. Washington, DC: American Psychological Association.
Google Scholar
Cover, T.M., & Thomas, J.A. ( 1991). Elements of information theory. New York: John Wiley.
Google Scholar
Cronbach, L.J., & Furby, L. ( 1970). How we should measure ‘‘change’’-Or should we? Psychological Bulletin, 74, 68-80.
Google Scholar
Eggen, T.J.H.M. ( 1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement , 23, 249-261.
Google Scholar
Embretson, S.E. ( 1991a). A multidimensional latent trait model for measuring learning and change. Psychometrika, 56, 495-515.
Google Scholar
Embretson, S.E. ( 1991b). Implications of a multidimensional latent trait model for measuring change. In L. M. Collins & J. L. Horn (Eds.), Best methods for the analysis of change (pp. 184-197). Washington, DC: American Psychological Association.
Google Scholar
Embretson, S.E. ( 1995). A measurement model for linking individual learning to processes and knowledge: Application to mathematical reasoning. Journal of Educational Measurement, 32, 277-294.
Google Scholar
Embretson, S.E., & Reise, S.P. ( 2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
Google Scholar
Fischer, G.H. ( 1976). Some probabilistic models for measuring change. In D. N. M. de Gruijter & L. J. T. van der Kamp (Eds.), Advances in psychological and educational measurement (pp. 97-110). New York: John Wiley.
Google Scholar
Hummel-Rossi, B., & Weinberg, S.L. (1975). Practical guidelines in applying current theories to the measurement of change. I. Problems in measuring change and recommended procedures (Ms. No. 916). JSAS Catalog of Selected Documents in Psychology , 5, 226.
Google Scholar
Kim-Kang, G., & Weiss, D.J. ( 2007). Comparison of computerized adaptive testing and classical methods for measuring individual change. In D. J. Weiss (Ed.), Proceedings of the 2007 GMAC Conference on Computerized Adaptive Testing. Available from www.psych.umn.edu/psylabs/CATCentral/
Google Scholar
Kim-Kang, G., & Weiss, D.J. ( 2008). Adaptive measurement of individual change. Zeitschrift fur Psychologie, 216, 49-58.
Google Scholar
Kingsbury, G.G., & Zara, A.R. ( 1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2, 359-375.
Google Scholar
Lai, T.L., & Shih, M.C. ( 2004). Power, sample size and adaptation considerations in the design of group sequential trials. Biometrika, 91, 507-528.
Google Scholar
Lord, F.M. ( 1963). Elementary models for measuring change. In C. W. Harris (Ed.), Problems in measuring change (pp. 21-38). Madison: University of Wisconsin Press.
Google Scholar
Lord, F.M. ( 1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
Google Scholar
Lord, F.M., & Novick, M.R. ( 1968). Statistical theories of mental test scores. Menlo Park, CA: Addison-Wesley.
Google Scholar
Manning, W.H., & DuBois, P.H. (1962). Correlation methods in research on human learning. Perceptual and Motor Skills, 15, 287-321.
Google Scholar
McDonald, R.P. ( 1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
Google Scholar
McNemar, Q. ( 1958). On growth measurement. Educational and Psychological Measurement, 18, 47-55.
Google Scholar
Muraki, E. ( 1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
Google Scholar
Samejima, F. ( 1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, No. 17.
Google Scholar
Siegmund, D. ( 1985). Sequential analysis: Tests and confidence intervals. New York: Springer.
Google Scholar
Stocking, M.L., & Lewis, C. ( 1998). Controlling item exposure conditional on ability in computerized adaptive testing. Journal of Educational and Behavioral Statistics , 23, 57-75.
Google Scholar
Sympson, J.B., & Hetter, R.D. ( 1985). Controlling item exposure rates in computerized adaptive testing. In Proceedings of the 27th annual meeting of the Military Testing Association (pp. 937-977). San Diego, CA: Navy Personnel Research and Development Center .
Google Scholar
Tucker, L.R., Damarin, F., & Messick, S. ( 1966). A base-free measure of change. Psychometrika , 31, 457-473.
Google Scholar
van der Linden, W.J. ( 2000). Constrained adaptive testing with shadow tests. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 27-52). Boston: Kluwer.
Google Scholar
van der Linden, W. J., & Glas, C. A. W. (Eds.). (2000). Computerized adaptive testing: Theory and practice. Boston: Kluwer.
Google Scholar
Wainer, H. (Ed.). (2000). Computerized adaptive testing: A primer (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
Google Scholar
Weiss, D.J. ( 1983). Introduction. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 1-9). New York: Academic Press.
Google Scholar
Weiss, D.J., & Kingsbury, G.G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21, 361-375.
Google Scholar
Willett, J.B. ( 1994). Measurement of change. In T. Husen & T. N. Postlethwaite (Eds.), The international encyclopedia of education (2nd ed., pp. 671-678). Oxford, UK: Pergamon .
Google Scholar
Willett, J.B. ( 1997). Measuring change: What individual growth modeling buys you. In E. Amsel & K. A. Renninger (Eds.), Change and development: Issues of theory, method, and application (pp. 213-243). Mahwah, NJ : Lawrence Erlbaum.
Google Scholar
Yen, W.M. ( 1986). The choice of scale for educational measurement: An IRT perspective. Journal of Educational Measurement, 23, 299-325.
Google Scholar

Cite article

Cite article

Cite article

OR

Download to reference manager

If you have citation software installed, you can download article citation data to the citation manager of your choice

Share options

Share

Share this article

Share with email
EMAIL ARTICLE LINK
Share on social media

Share access to this article

Sharing links are not relevant where the article is open access and not available if you do not have a subscription.

For more information view the SAGE Journals article sharing page.

Information, rights and permissions

Information

Published In

Volume 34, Issue 4
Pages: 238 - 254
Article first published online: April 27, 2010
Issue published: June 2010

Keywords

  1. change
  2. individual change
  3. measuring change
  4. computerized adaptive testing
  5. likelihood ratio
  6. Kullback—Leibler information

Rights and permissions

© The Author(s) 2010.
Request permissions for this article.

History

Published online: April 27, 2010
Issue published: June 2010

Authors

Affiliations

Matthew D. Finkelman
Tufts University School of Dental Medicine, Boston, Massachusetts, [email protected]
View all articles by this author
David J. Weiss
University of Minnesota, Minneapolis, Minnesota
View all articles by this author
Gyenam Kim-Kang
Measurement and Statistics Research, West Lafayette, Indiana
View all articles by this author

Metrics and citations

Metrics

Journals metrics

This article was published in Applied Psychological Measurement.

VIEW ALL JOURNAL METRICS

Article usage*

Total views and downloads: 152

*Article usage tracking started in December 2016

Articles citing this one

Web of Science: 23 view articles Opens in new tab

Crossref: 20

  1. Robustness of Adaptive Measurement of Change to Item Parameter Estimat...
    Go to citation Crossref Google Scholar
  2. Adaptive Measurement of Change: A Novel Method to Reduce Respondent Bu...
    Go to citation Crossref Google Scholar
  3. The Use of the Posterior Probability in Score Differencing
    Go to citation Crossref Google Scholar
  4. Hypothesis Testing Methods for Multivariate Multi-Occasion Intra-Indiv...
    Go to citation Crossref Google Scholar
  5. Detection of Item Preknowledge Using Response Times
    Go to citation Crossref Google Scholar
  6. Detecting test fraud using Bayes factors
    Go to citation Crossref Google Scholar
  7. Higher-Order Asymptotics and Its Application to Testing the Equality o...
    Go to citation Crossref Google Scholar
  8. Detecting Fraudulent Erasures at an Aggregate Level
    Go to citation Crossref Google Scholar
  9. Multivariate Hypothesis Testing Methods for Evaluating Significant Ind...
    Go to citation Crossref Google Scholar
  10. Application of Bayesian Methods for Detecting Fraudulent Behavior on T...
    Go to citation Crossref Google Scholar
  11. A New Statistic for Detection of Aberrant Answer Changes
    Go to citation Crossref Google Scholar
  12. Detection of Item Preknowledge Using Likelihood Ratio Test and Score T...
    Go to citation Crossref Google Scholar
  13. Comparison of Classical Test Theory and Item Response Theory in Indivi...
    Go to citation Crossref Google Scholar
  14. Person Fit Analysis in Computerized Adaptive Testing Using Tests for a...
    Go to citation Crossref Google Scholar
  15. The Asymptotic Distribution of Ability Estimates...
    Go to citation Crossref Google Scholar
  16. Assessing Individual-Level Impact of Interruptions During Online Testi...
    Go to citation Crossref Google Scholar
  17. A Comprehensive Approach for Assessing Person Fit With Test–Retest Dat...
    Go to citation Crossref Google Scholar
  18. Assessing Individual Change Using Short Tests and Questionnaires
    Go to citation Crossref Google Scholar
  19. Measuring individual significant change on the Beck Depression Invento...
    Go to citation Crossref Google Scholar
  20. A proof of principle for using adaptive testing in routine Outcome Mon...
    Go to citation Crossref Google Scholar

Figures and tables

Figures & Media

Tables

View Options

Get access

Access options

If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:

Alternatively, view purchase options below:

Purchase 24 hour online access to view and download content.

Subscribe to this journal

Access journal content via a DeepDyve subscription or find out more about this option.

Start 2 week free trial
Need help?

View options

PDF/ePub

View PDF/ePub
Download PDF

Also from SAGE Publishing