![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
This article explores an approach for test score validation that examines test takers’ strategies for taking a reading comprehension test. The authors formulated three working hypotheses about score validity pertaining to three types of test-taking strategy (comprehending meaning, test management, and test-wiseness). These hypotheses were formulated in terms of the use of three types of test-taking strategy and their relationships with performance on specific task types (testlets) and overall test performances. We illustrated the proposed method for validation using example data from the Canadian English Language Proficiency Index Program-General (CELPIP-General) reading pilot test. The findings were that (a) test takers were engaging more in processing the texts for comprehending meaning, less in test-management skills, and least in test-wiseness; (b) at the task level, task characteristics (e.g., difficulty) had implications on test takers’ engagement with different types of strategies, which, in turn, led to differences in predicting task performances; and (c) at the test level, higher engagement in comprehending meaning led to higher test performance, engagement in test management showed a small negative association with test performance, and higher engagement in test-wiseness led to poorer performance. The high congruence between the working hypotheses and the empirical results offered plausible evidence that supported the validity of CELPIP-General reading scores. Revisions to both hypotheses and research design that might improve the proposed validation method are reviewed in the “Discussion” section.
References
Section:
|
Abbott, M. L. (2006). ESL reading strategies: Differences in Arabic and Mandarin speaker test performance. Language Learning, 56, 633-670. Google Scholar | Crossref | ISI | |
|
Alderson, J. C. (2000). Assessing reading. Cambridge, UK: Cambridge University Press. Google Scholar | Crossref | |
|
Allen, S. (2003). An analytic comparison of three models of reading strategy instruction. International Review of Applied Linguistics in Language Teaching, 41(4), 319-338. Google Scholar | Crossref | |
|
American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Joint Committee on Standards for Educational, & Psychological Testing . (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Google Scholar | |
|
Anderson, N. J., Bachman, L., Perkins, K., Cohen, A. D. (1991). An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources. Language Testing, 8, 41-66. Google Scholar | SAGE Journals | |
|
Citizenship and Immigration Canada . (2012). Canadian language benchmarks: English as a second language for adults. Ottawa, Ontario: Centre for Canadian Language Benchmarks. Retrieved from http://publications.gc.ca/site/archivee-archived.html?url=http://publications.gc.ca/collections/collection_2012/cic/Ci63-26- Google Scholar | |
|
Cohen, A. D. (1984). On taking language tests what the students report. Language Testing, 1, 70-81. Google Scholar | SAGE Journals | |
|
Cohen, A. D. (2006). The coming of age of research on test-taking strategies. Language Assessment Quarterly, 3, 307-331. Google Scholar | Crossref | |
|
Cohen, A. D. (2012a). Test taker strategies and test design. In Fulcher, G., Davidson, F. (Eds.), The Routledge handbook of language testing in a nutshell (pp. 262-277). Abingdon, UK: Routledge. Google Scholar | |
|
Cohen, A. D. (2012b). Test-taking strategies. In Coombe, C., Davidson, P., O’Sullivan, B., Stoynoff, S. (Eds.), The Cambridge guide to second language assessment (pp. 96-104). Cambridge, UK: Cambridge University Press. Google Scholar | |
|
Cohen, A. D., Segal, M., Bar-Siman-Tov, R. (1984). The C-test in Hebrew. Language Testing, 1, 221-225. Google Scholar | SAGE Journals | |
|
Cohen, A. D., Upton, T. A. (2006). Strategies in responding to the new TOEFL reading tasks (TOEFL Monograph Series No. MS-33). Princeton, NJ: Educational Testing Services. Google Scholar | |
|
Cohen, A. D., Upton, T. A. (2007). “I want to go back to the text”: Response strategies on the reading subtest of the new TOEFL®. Language Testing, 24, 209-250. Google Scholar | SAGE Journals | |
|
Farr, R., Pritchard, R., Smitten, B. (1990). A description of what happens when an examinee takes a multiple-choice reading comprehension test. Journal of Educational Measurement, 27, 209-226. Google Scholar | Crossref | ISI | |
|
Horn, J. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185. Google Scholar | Crossref | Medline | ISI | |
|
Humphreys, L. G., Montanelli, R. G. (1975). An investigation of the parallel analysis criterion for determining the number of common factors. Multivariate Behavioral Research, 10, 193-205. Google Scholar | Crossref | ISI | |
|
Kline, R. B. (2011). Principles and practice of SEM (3rd ed.). New York, NY: The Guilford Press. Google Scholar | |
|
Lee, J. Y. (2011). Second language reading topic familiarity and test score: Test-taking strategies for multiple-choice comprehension questions (Unpublished doctoral dissertation). The University of Iowa, Iowa City. Google Scholar | |
|
Liu, Y., Zumbo, B. D., Wu, A. D. (2012). A demonstration of the impact of outliers on the decisions about the number of factors in exploratory factor analysis. Educational and Psychological Measurement, 72, 181-199. Google Scholar | SAGE Journals | ISI | |
|
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749. Google Scholar | Crossref | ISI | |
|
Nevo, N. (1989). Test-taking strategies on a multiple-choice test of reading comprehension. Language Testing, 6, 199-215. Google Scholar | SAGE Journals | |
|
Phakiti, A. (2003). A closer look at the relationship of cognitive and metacognitive strategy use to EFL reading achievement test performance. Language Testing, 20, 26-56. Google Scholar | SAGE Journals | |
|
Purpura, J. E. (1997). An analysis of the relationships between test takers’ cognitive and metacognitive strategy use and second language test performance. Language Learning, 47, 289-325. Google Scholar | Crossref | ISI | |
|
Purpura, J. E. (1998). Investigating the effects of strategy use and second language test performance with high-and low-ability test takers: A structural equation modelling approach. Language Testing, 15, 333-379. Google Scholar | SAGE Journals | |
|
Purpura, J. E. (1999). Learner strategy use and performance on language tests: A structural equation modeling approach. Cambridge, UK: Cambridge University Press. Google Scholar | |
|
Roizen, M. A. (1984). Test performance vis a vis test-taking strategies in reading comprehension of English as a second language. (ERIC Document Reproduction Service No. ED 224 350) Google Scholar | |
|
Rupp, A. A., Ferne, T., Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23, 441-474. Google Scholar | SAGE Journals | |
|
Salehi, M. (2011). Test taking strategies: Implications for test validation. Journal of Language Teaching and Research, 2, 850-858. Google Scholar | Crossref | |
|
Song, X. (2005). Language learner strategy use and English proficiency on the Michigan English Language Assessment Battery. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 3, 1-113. Google Scholar | |
|
Stemmer, B. (1991). What’s on a C-test taker’s mind? Mental processes in C-test taking. Bochum, Germany: University of Dr. N. Brockmeyer. Google Scholar | |
|
Storey, P. (1997). Examining the test-taking process: A cognitive perspective on the discourse cloze test. Language Testing, 14, 214-231. Google Scholar | SAGE Journals | |
|
Wu, A. D., Zumbo, B. D., Marshall, S. K. (2014). A method to aid in the interpretation of EFA results: An application of Pratt’s measures. International Journal of Behavioral Development, 38, 98-110. Google Scholar | SAGE Journals | ISI | |
|
Yoshizawa, K. (2002). Relationships among strategy use, foreign language aptitude, and second language proficiency: A structural equation modeling approach (Unpublished doctoral dissertation). Temple University, Philadelphia, PA. Google Scholar |
