Measuring Game Engagement: Multiple Methods and Construct Complexity
Abstract
Background. Engagement has been identified as a crucial component of learning in games research. However, the conceptualization and operationalization of engagement vary widely in the literature. Many valuable approaches illuminate ways in which presence, flow, arousal, participation, and other concepts constitute or contribute to engagement. However, few studies examine multiple conceptualizations of engagement in the same project.
Method. This article discusses the results of two experiments that measure engagement in five different ways: survey self-report, content analyses of player videos, electro-dermal activity, mouse movements, and game click logs. We examine the relationships among these measures and assess how they are affected by the technical characteristics of a 30-minute, custom-built, educational game: use of a customized character, level of narrative complexity, and level of art complexity.
Results. We found that the five measures of engagement correlated in limited ways, and that they revealed substantially different relationships with game characteristics. We conclude that engagement as a construct is more complex than is captured in any of these measures individually and that using multiple methods to assess engagement can illuminate aspects of engagement not detectable by a single method of measurement.
Get full access to this article
View all access and purchase options for this article.
References
Abbott H. P. (2002). The Cambridge introduction to narrative. Cambridge, UK: Cambridge University Press.
Annetta L., Mangrum J., Holmes S., Collazo K., Cheng M. (2009). Bridging reality to virtual reality: Investigating gender effect and student engagement on learning through video game play in an elementary school classroom. International Journal of Science Education, 31, 1091-1113.
Appleton J. J., Christenson S. L., Furlong M. J. (2008). Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in the Schools, 45, 369-386.
Bailenson J. N., Swinth K., Hoyt C., Persky S., Dimov A., Blascovich J. (2005). The independent and interactive effects of embodied-agent appearance and behavior on self-report, cognitive, and behavioral markers of copresence in immersive virtual environments. Presence: Teleoperators and Virtual Environments, 14, 379-393.
Bailey R., Wise K., Bolls P. (2009). How avatar customizability affects children’s arousal and subjective presence during junk food-sponsored online video games. Cyberpsychology, Behavior, and Social Networking, 12, 277-283.
Biocca F., Levy M. R. (1995). Communication in the age of virtual reality. Hillsdale, NJ: Lawrence Erlbaum.
Brusselle R., Bilandzic H. (2009). Measuring narrative engagement. Media Psychology, 12, 321-347.
Csikszentmihalyi M. (1997). Finding flow: The psychology of engagement with everyday life. New York, NY: Basic Books.
Dale R., Kehoe C., Spivey M. (2007). Graded motor responses in the time course of categorizing atypical exemplars. Memory & Cognition, 35, 15-28.
Dinh H. Q., Walker N., Song C., Kobayashi A., Hodges L. F. (1999). Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. In Proceedings of the IEEE Virtual Reality (VR ’99) (pp. 222-228). Washington, DC: IEEE Computer Society.
Dow S., Mehta M., Harmon E., Macintyre B., Mateas M. (2007). Presence and engagement in an interactive drama. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’07 (pp. 1475-1484). New York, NY: ACM.
Fredricks J. A., Blumenfeld P. C., Paris A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59-109.
Gee J. P. (2003). What video games have to teach us about learning and literacy. New York, NY: Palgrave Macmillan.
Gerrig R. J. (1993). Experiencing narrative worlds: On the psychological activities of reading. New Haven, CT: Yale University Press.
Green M. C. (2004). Transportation into narrative worlds: The role of prior knowledge and perceived realism. Discourse Processes, 38, 247-266.
Green M. C., Brock T. C. (2000). The role of transportation in the persuasiveness of public narratives. Journal of Personality and Social Psychology, 79, 701-721.
Green M. C., Brock T. C. (2002). In the mind’s eye: Imagery and transportation into narrative worlds. In Green M. C., Strange J. J., Brock T. C. (Eds.), Narrative impact: Social and cognitive foundations (pp. 315-341). Mahwah, NJ: Lawrence Erlbaum.
Green M. C., Brock T. C., Kaufman G. F. (2004). Understanding media enjoyment: The role of transportation into narrative worlds. Communication Theory, 14, 311-327.
Ismail E., Sidek S. N., Khan M. R., Jalaludin N. A. (2012, July). Analysis of engagement factor in trajectory tracking-based experiment. 2012 International Conference on Computer and Communication Engineering (ICCCE), Kuala Lumpur, Malaysia.
Jones M. T. (2008). Found in translation: Structural and cognitive aspects of the adaptation of comic art to film. Philadelphia, PA: Temple University.
Juul J. (2005). Half-real: Video games between real rules and fictional worlds. Cambridge, MA: The MIT Press.
Karimi A., Lim Y. P. (2010, December). Children, engagement, and enjoyment in digital narrative. Proceedings of ASCILITE 2010, Sydney, Australia. Retrieved from http://www.ascilite.org.au/conferences/sydney10/procs/Karimi-full.pdf
Klimmt C., Dorothée H., Vorderer P. (2009). The video game experience as “true” identification: A theory of enjoyable alterations of players’ self-perception. Communication Theory, 19, 351-373.
Klimmt C., Vorderer P. (2003). Media psychology “is not yet there”: Introducing theories on media entertainment to the presence debate. Presence: Teleoperators and Virtual Environments, 12, 346-359.
Krippendorff K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA: SAGE.
Lederman L. C. (1992). Debriefing: Toward a systematic assessment of theory and practice. Simulation & Gaming, 23, 145-160.
Lim S., Reeves B. (2010). Computer agents versus avatars: Responses to interactive game characters controlled by a computer or other player. International Journal of Human-Computer Studies, 68, 57-68.
Lombard M., Ditton T. (1997). At the heart of it all: The concept of presence. Journal of Computer-Mediated Communication, 3(2). http://onlinelibrary.wiley.com/doi/10.1111/j.1083-6101.1997.tb00072.x/full
Mandryk R. L., Inkpen K. M. (2004). Physiological indicators for the evaluation of co-located collaborative play. In CSCW ’04: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (pp. 102-111). New York, NY: ACM.
Mayes D. K., Cotton J. E. (2001). Measuring engagement in video games: A questionnaire. Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting, Minneapolis, Minnesota, 8-12 October 2001, (pp. 692-696).
McMahan A. (2003). Immersion, engagement, and presence: A method for analyzing 3-D video games. In Wolf M. J. P., Perron B. (Eds.), The video game theory reader (pp. 67-86). New York, NY: Routledge.
McQuiggan S., Rowe J., Lee S., Lester J. (2008). Story-based learning: The impact of narrative on learning experiences and outcomes. In Woolf B., Aïmeur E., Nkambou R., Lajoie S. (Eds.), Intelligent tutoring systems, Vol. 5091 of Lecture notes in computer science (pp. 530-539). Berlin, Germany: Springer.
Merola N. A., Peña J. (2010). The effects of avatar appearance in virtual worlds. Journal of Virtual Worlds Research, 2(5). Retrieved from https://journals.tdl.org/jvwr/article/view/843
Mott B., Callaway C., Zettlemoyer L., Lee S., Lester J. (1999). Towards narrative-centered learning environments. Proceedings of the AAAI Fall Symposium on Narrative Intelligence, Falmouth, Massachusetts, 5-7 November 1999, (pp. 78-82).
Nowak K. L., Kremar M., Farrar K. M. (2008). The causes and consequences of presence: Considering the influence of violent video games on presence and aggression. Presence: Teleoperators and Virtual Environments, 17, 256-268.
O’Brien H. L., Toms E. G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59, 938-955.
Ravaja N., Saari T., Salminen M., Laarni J., Kallinen K. (2006). Phasic emotional reactions to video game events: A psychophysiological investigation. Media Psychology, 8, 343-367.
Read J., MacFarlane S., Casey C. (2002). Endurability, engagement and expectations: Measuring children’s fun. In Proceedings of Conference on Interaction Design and Children (pp. 189-198). Eindhoven, The Netherlands: Shaker.
Renshaw T., Stevens R., Denton P. D. (2009). Towards understanding engagement in games: An eye-tracking study. On the Horizon, 17, 408-420.
Schneider E. F., Lang A., Shin M., Bradley S. D. (2004). Death with a story: How story impacts emotional, motivational, and physiological responses to first-person shooter video games. Human Communication Research, 30, 361-375.
Shaw A. (2010). Identity, identification and media representation in video game play: An audience reception study (Unpublished dissertation). University of Pennsylvania, Philadelphia.
Sundar S. S., Oeldorf-Hirsch A., Garga A. (2008). A cognitive-heuristics approach to understanding presence in virtual environments. In Spagnolli A., Gamberini L. (Eds.), PRESENCE 2008: Proceedings of the 11th Annual International Workshop on Presence (pp. 219-228). Padova, Italy: CLEUP Cooperativa Libraria Universitaria Padova.
Teixeira T., Wedel M., Pieters R. (2012). Emotion-induced engagement in Internet video advertisements. Journal of Marketing Research, 49, 144-159.
Trepte S., Reinecke L. (2011). The pleasures of success: Game-related efficacy experiences as a mediator between player performance and game enjoyment. Cyberpsychology, Behavior, and Social Networking, 14, 555-557.
Turkay S., Adinolf S. (2010). Free to be me: A survey study on customization with World of Warcraft and City of Heroes/Villains players. Procedia—Social and Behavioral Sciences, 2, 1840-1845.
Van Vugt H. C., Hoorn J. F., Konijn E. A., Dimitriadou A. (2006). Affective affordances: Improving interface character engagement through interaction. International Journal of Human-Computer Studies, 64, 874-888.
Whitbred R., Skalski P., Bracken C., Lieberman E. (2010). When richer is poorer: Understanding the influence of channel richness and presence on the introduction of a mission statement. Psychology Journal, 8, 115-139.
Wise A., Chang J., Duffy T., del Valle R. (2004). The effects of teacher social presence on student satisfaction, engagement, and learning. In Proceedings from the 6th International Conference on Learning Sciences (ICLS ’04) (pp. 568-575). New York, NY: ACM.
Wolf M. J. P. (2003). Abstraction in the video game. In Wolf M. J. P., Perron B. (Eds.), The video game theory reader (pp. 47-65). New York, NY: Routledge.
Zaichkowsky J. L. (1984). Measuring the involvement construct. Journal of Consumer Research, 12, 341-352.
Zimmons P., Panter A. (2003). The influence of rendering quality on presence and task performance in a virtual environment [Poster Abstract]. In Proceedings of IEEE Virtual Reality 2003. pp. 141–148 Los Angeles, CA: IEEE Computer Society.
Biographies
Rosa Mikeal Martey is an associate professor in the Department of Journalism and Communication at Colorado State University. Her research focuses on games, identity, social interaction, and digital media, and can be found in New Media & Society, Information, Communication & Society, and Games Studies, among others.
Contact: [email protected].
Kate Kenski (PhD, University of Pennsylvania) is an associate professor in the Department of Communication and School of Government and Public Policy at the University of Arizona. She has published more than 30 articles, books, and book chapters on political communication, public opinion, and research methods.
Contact: [email protected].
James Folkestad is an associate professor in the School of Education at Colorado State University. His work focuses on educational technology, activity systems for learning, innovative inference game-based learning, and social network analysis.
Contact: [email protected].
Laurie Feldman is a senior scientist at Haskins Laboratories in New Haven, Connecticut, and professor of psychology at the University at Albany, State University of New York. She studies how people use and understand written and spoken languages, both native and non-native.
Contact: [email protected].
Elana Gordis studies the effects of family and community violence on later development, with a particular focus on the role of physiological stress response systems, including sympathetic, parasympathetic, and hypothalamic-pituitary-adrenal axis activity. She received her PhD in clinical psychology from the University of Southern California.
Contact: [email protected].
Adrienne Shaw has been studying video games since 2005. Her research has been published in Ada: Journal of Gender, New Media, and Technology; New Media and Society; Critical Studies in Media and Communication; and Games and Culture, among others.
Contact: [email protected].
Jennifer Stromer-Galley (PhD, University of Pennsylvania, 2002) is an associate professor in information studies. Her expertise is on communication processes and effects through digital communication channels. She has more than 30 publications that focus on dimensions of digital media around influence, leadership, political communication, and training.
Contact: [email protected].
Ben Clegg is a professor of cognitive psychology at Colorado State University. His research is on aspects of human performance, particularly how to apply principles from cognitive psychology to real-world tasks and skills. His work examines issues such as training, automation, workload, and situation awareness, sequence learning, and implicit learning.
Contact: [email protected].
Hui Zhang is a doctoral student in the Department of Journalism and Technical Communication at Colorado State University. Her research focuses on health communication and information-seeking in digital media.
Contact: [email protected].
Nissim Kaufman is a PhD candidate in mathematics at the University at Albany in mathematics and statistics. His work is on parametric models, cluster analysis, and Bayesian inferences, among other topics, and he has contributed to projects funded by DARPA, IARPA, and others.
Contact: [email protected].
Ari N. Rabkin is currently a doctoral candidate with the Department of Psychology and research project assistant with the Institute for Informatics, Logics, and Security Studies (ILS) at the University at Albany, State University of New York. His research focuses on the intersection of biological/physiological measurement and psychological functioning, including user engagement during video games.
Contact: [email protected].
Samira Shaikh has a predilection for puzzle games. A computational socio-linguist by training, she holds degrees in computer engineering and computer science. Her current research focus is on natural language processing, specifically aimed at understanding social roles and phenomena in social media conversations and artificial chat agents.
Contact: [email protected].
Tomek Strzalkowski is the director of the Institute for Informatics, Logics, and Security Studies and professor of computer science at State University of New York Albany. He has done research in computational linguistics, socio-linguistics, information retrieval, question-answering, human-computer dialogue, and formal semantics of natural language. He has directed research projects sponsored by IARPA, DARPA, EC, and NSF, and published over a 100 papers and books.
Contact: [email protected].
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: November 4, 2014
Issue published: August-October 2014
Keywords
Authors
Author Contributions
All authors contributed equally to the development, review, and revision of this article and to the research that underpins it.
Metrics and citations
Metrics
Article usage*
Total views and downloads: 1731
*Article usage tracking started in December 2016
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 0
Crossref: 47
- Relationships among class climate, students' internal help‐seeking inc...
- Real-time estimation of EEG-based engagement in different tasks
- Hand Tracking with Vibrotactile Feedback Enhanced Presence, Engagement...
- Adaptive scaffolding and engagement in digital game-based learning
- Improving Behavioral-Based Safety Training in Using Verbal Command: A ...
- Présence et incarnation dans les jeux sérieux : Une revue de la littér...
- User Engagement Comparison between Advergames and Traditional Advertis...
- The Electrodermal Activity of Player Experience in Virtual Reality Gam...
- Toward Data-Driven Digital Therapeutics Analytics: Literature Review a...
- Player Engagement Classification in Mobile Games Using MUSE Headband
- Towards using Involuntary Body Gestures for Measuring the User Engagem...
- ”Every move you make in every game you play”: can game logs determine ...
- A Multimodal Approach for Real Time Recognition of Engagement towards ...
- Academical: A Choice-Based Interactive Storytelling Game for Enhancing...
- Catch Me If you Can: An Educational Serious Game to Teach Grammatical ...
- Serious Games Design Principles Using Virtual Reality to Gamify Upper ...
- Classification of Game Demand and the Presence of Experimental Pain Us...
- Improving Undergraduate Attitudes Towards Responsible Conduct of Resea...
- Effects of solo vs. collaborative play in a digital learning game on g...
- The Response to Impactful Interactivity on Spectators’ Engagement in a...
- Patterns of Engagement in an Educational Massively Multiplayer Online ...
- Getting Academical: A Choice-Based Interactive Storytelling Game for T...
- Method for Selecting an Engagement Index for a Specific Type of Game U...
- Improving Engagement Assessment in Gameplay Testing Sessions using IoT...
- Teaching Responsible Conduct of Research Through an Interactive Storyt...
- Exploring how online political quizzes boost interest in politics, pol...
- Examining the Degree of Engagement of a Participant in Economic Games ...
- Game Design Principles Influencing Stroke Survivor Engagement for VR-B...
- Towards Automatic Flow Experience Identification in Educational System...
- Modules of Interaction for ASD Children Using Rero Robot (Humanoid)
- Gamified in-store mobile marketing: The mixed effect of gamified point...
- A taxonomy and state of the art revision on affective games
- Using Computer Games to Mitigate Disaffected Emotions in the Geography...
- Piloting Two Educational Games in Five European Countries: Teachers' P...
- "You're Giving Me Mixed Signals!"
- The Temporal Attentive Observation (TAO) Scale: Development of an Inst...
- MiniColon; Teaching Kids Computational Thinking Using an Interactive S...
- “I Am a Warrior”: Self-Identification and Involvement in Massively Mul...
- A Holistic Approach to Measuring User Engagement
- Measuring Engagement in eHealth and mHealth Behavior Change Interventi...
- Learning from Social Collaboration
- Conceptualising engagement with digital behaviour change interventions...
- Neuromender∷FlexiBrains
- Learning from Social Collaboration
- From duels to classroom competition: Social competition and learning i...
- A player engagement model for an augmented reality game
- StreetWise: A Valid Ecology for a Serious Game in a Secure Forensic Me...
Figures and tables
Figures & Media
Tables
View Options
Get access
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
loading institutional access options
NASAGA members can access this journal content using society membership credentials.
NASAGA members can access this journal content using society membership credentials.
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
