A Systems Approach to Educational Testing
Abstract
Our concern in this paper is with the validity of educational tests when they are employed as critical measures of educational outcomes within a dynamic system. The problem of validity arises if an educational system adapts itself to the characteristics of the outcome measures. We introduce the concept of systemically valid tests as ones that induce curricular and instructional changes in education systems (and learning strategy changes in students) that foster the development of the cognitive traits that the tests are designed to measure. We analyze some general characteristics that contribute to or detract from a testing system's systemic validity, such as the use of direct rather than indirect assessment. We then apply these characteristics in developing a set of design principles for creating testing systems that are systemically valid. Finally, we provide an illustration of the proposed principles, by applying them to the design of a student assessment system. This design example addresses not only specifications for the tests, but also the means of teaching the process of assessment to users of the system.
Get full access to this article
View all access and purchase options for this article.
References
Anderson JA, Boyle CF, and Reiser BJ. Intelligent tutoring systems. Science 1985;228:456-468
Braun H. Calibration of essay readers 1986 Princeton, NJ Educational Testing Service (Report No. RR-86–9).
Breland HM and Jones RJ. Remote scoring of essays 1988 Princeton, NJ Educational Testing Service (Report No. 88–4).
Brown JS, Collins A, and Duguid P. Situated cognition and the culture of learning. Educational Researcher 1989;18 1 32-42
Clancey W. Guidon. Journal of Computer- Based Instruction 1983;10 1 & 2 8-15
Clement J. Students' preconceptions in elementary mechanics. American Journal of Physics 1982;50:66-71
Collins A and Brown JS. Mandl H and Lesgold A. The computer as a tool for learning through reflection. Learning issues for intelligent tutoring systems 1988 New York Springer 1-18
Collins A and Gentner DG. Gregg LW and Steinberg ER. A frame- work for a cognitive theory of writing. Cognitive processes in writing 1980 Hillsdale, NJ Erlbaum 51-72
Collins A and Frederiksen JR. Five traits of good teaching: Learning, thinking, listening, involving, helping 1989 Cambridge, MA BBN Laboratories Unpublished report.
Corbett HD and Wilson B. Raising the stakes in statewide mandatory minimum competency testing. Politics of Education Association Yearbook 1988 27-39
Darling-Hammond L and Wise A. Beyond standardization: State standards and school improvement. Elementary School Journal 1985;85:315-336
diSessa A. Unlearning Aristotelian physics: A study of knowledge-based learning. Cognitive Science 1982;6:37-76
Frederiksen JR and White BY. Frederiksen N, Glaser R, Lesgold A, and Shafto M. Intelligent tutors as intelligent testers. Diagnostics monitoring of skill and knowledge acquisition 1989 Hillsdale, NJ Erlbaum 1-25
Frederiksen N. Assessment of creativity in scientific problem solving 1978 Princeton, NJ Educational Testing Service (Research Memorandum RM–78–9).
Frederiksen N. The real test bias. American Psychologist 1984;39 3 193-202
Frederiksen N. Frederiksen N, Glaser R, Lesgold A, and Shafto M. Introduction. Diagnostic monitoring of skill and knowledge acquisition 1989 Hillsdale, NJ Erlbaum vii-xv
Johnson WL and Soloway E. PROUST: An automatic debugger for Pascal programs.. Byte 1985;10 4 179-190
Madaus G. Tanner L. The influence of testing on the curriculum. Critical issues in curriculum: 87th Yearbook of the NSSE, Part I 1988 Chicago University of Chicago Press
McCIoskey M, Caramazza A, and Green B. Curvilinear motion in the absence of external forces: Naive beliefs about the motion of objects. Science 1980;210:1139-1141
McDermott LC. Research on conceptual understanding in mechanics. Physics Today 1984;37:24-32
Messick S. Linn RL. Validity. Educational measurement 1988 3rd ed New York Macmillan 13-103
Mullis IVS. Using the primary trait system for evaluating writing. National Assessment of Educational Progress Report 1980 Denver, CO Education Commission of the States
Resnick LB and Resnick DP. Gifford BR and O'Connor MC. Assessing the thinking curriculum: New tools for educational reform. Future assessments: Changing views of aptitude, achievement, and instruction Boston Kluwer (in press).
Schoenfeld AH. Perkins DN, Segal J, and Voss J. On mathematics as sense-making: An informal attack on the unfortunate divorce of formal and informal mathematics. Informal reasoning and education Hillsdale, NJ Erlbaum (in press).
Sleeman D and Brown JS. Intelligent tutoring systems 1982 New York Academic Press
White BY. Sources of difficulty in understanding Newtonian dynamics. Cognitive Science 1983;7 1 41-65
White BY. Designing computer activities to help physics students understand Newton's laws of motion. Cognition and Instruction 1984;1:69-108
Wiggins G. A true test: Toward more authentic and equitable assessment. Phi Delta Kappan 1989 May 703-713
Wolfe DP. Opening up assessment. Educational Leadership 1987 December 24-29
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published: December 1989
Issue published: December 1989
Authors
Metrics and citations
Metrics
Article usage*
Total views and downloads: 589
*Article usage tracking started in December 2016
Altmetric
See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 0
Crossref: 348
- Impact of dental school critical thinking demonstrations carryover to practice: Survey of 5‐year graduates
- Some Impact of the English Competence Assessment Test Based on Output Standards on English Teaching at Tan Trao University
- Some Impact of the English Competence Assessment Test Based on Output Standards on English Teaching at Tan Trao University
- A review of the benefits and drawbacks of high-stakes final examinations in higher education
- Keeping Up the PACE: Evaluating Grade 8 Student Achievement Outcomes for New Hampshire’s Innovative Assessment System
- A Systematic Review of Secondary School Climate Assessments
- International Encyclopedia of Education(Fourth Edition)
- International Encyclopedia of Education(Fourth Edition)
- avaliações externas como instrumento para aprimorar a prática pedagógica
- How Multidimensional is Computational Thinking Competency? A Bi-Factor Model of the Computational Thinking Challenge
- View More
Figures and tables
Figures & Media
Tables
View Options
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
I am signed in as:
View my profileSign out
I can access personal subscriptions, purchases, paired institutional access and free tools such as favourite journals, email alerts and saved searches.
loading institutional access options
AERA members can access this journal content using society membership credentials.
AERA members can access this journal content using society membership credentials.
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
