The Validity of the SEEV Model as a Process Measure of Situation Awareness: The Example of a Simulated Endotracheal Intubation

Objective In the context of anesthesiology, we investigated whether the salience effort expectancy value (SEEV) model fit is associated with situation awareness and perception scores. Background The distribution of visual attention is important for situation awareness—that is, understanding what is going on—in safety-critical domains. Although the SEEV model has been suggested as a process situation awareness measure, the validity of the model as a predictor of situation awareness has not been tested. Method In a medical simulation, 31 senior and 30 junior anesthesiologists wore a mobile eye tracker and induced general anesthesia into a simulated patient. When inserting a breathing tube into the mannequin’s trachea (endotracheal intubation), the scenario included several clinically relevant events for situation awareness and general events in the environment. Both were assessed using direct awareness measures. Results The overall SEEV model fit was good with no difference between junior and senior anesthesiologists. Overall, the situation awareness scores were low. As expected, the SEEV model fits showed significant positive correlations with situation awareness level 1 scores. Conclusion The SEEV model seems to be suitable as a process situation awareness measure to predict and investigate the perception of changes in the environment (situation awareness level 1). The situation awareness scores indicated that anesthesiologists seem not to perceive the environment well during endotracheal intubation. Application The SEEV model fit can be used to capture and assess situation awareness level 1. During endotracheal intubation, anesthesiologists should be supported by technology or staff to notice changes in the environment.


INTRODUCTION
Situation awareness-knowing "what is going on"-is important for safe operation in safetycritical domains such as anesthesiology. For example, during the induction of general anesthesia, the patient's status can change quickly, and good situation awareness is important for patient safety (Schulz et al., 2013(Schulz et al., , 2016. Although research has highlighted the importance of visual attention distribution for situation awareness in aviation (e.g., Stein, 1992;Ziv, 2016), plant control (e.g., Mumaw et al., 2000), and anesthesiology (e.g., Gaba et al., 1995;Schulz et al., 2011), visual attention distribution has frequently only been analyzed time on several objects; for a review in healthcare, see Grundgeiger et al., 2015). The salience attention provides a theoretically founded way of investigating attention distribution (Wickens, 2015;Wickens et al., 2003). Wickens et al. (2008) spond to situation awareness level 1 (perception or noticing of elements in the environment). We tested this suggestion in the context of anesthesiologists inducing general anesthesia in a simwith situation awareness scores.

Eye Tracking Metrics as a Situation Awareness Measure
Situation awareness can be described as the perception or noticing of elements in the environment (SA level 1), comprehension of their meaning (SA level 2), and understanding of future implications (SA level 3; e.g., ; Wright et al., 2004). The Situation Awareness 2022, Vol. 64(7) [1181][1182][1183][1184][1185][1186][1187][1188][1189][1190][1191][1192][1193][1194] Global Assessment Technique (SAGAT, 1995) is the most common technique used to assess situation awareness (de Winter et al., 2019; ); however, there is an ongoing debate about the advantages and disadvantages of SAGAT (de Winter, 2020;de Winter et al., 2019;Stanton et al., 2017). One alternative measure is the use of eye tracking metrics, which is a so-called process measure of situation awareness (i.e., an ongoing measure Salmon et al., 2006). summarized that eye tracking has the advantage of being objective, and eye tracking includes information about order and duration of attention distribution, yet the technique does not consider auditory information, and overall there is little research that supports the validity of eye tracking as a process measure of situation awareness.
In a recent review on physiological measurements of situation awareness, Zhang et al. (in press) found 16 papers that reported eye tracking metrics and situation awareness scores but only a the association between conscious eye tracking etc.) and situation awareness scores. Moore and Gugerty (2010) used various eye tracking metrics ting to predict a direct situation awareness measure. Moore and Gugerty (2010) observed that the could explain 7% and 9% of the unique variance in SA level 1 and a composite overall SAGAT score (SA levels 1-3), respectively. In the context of healthcare, Law et al. (2020) manipulated the position of the patient monitor during a simulated neonatal resuscitation. As a secondary analysis, Law et al. (2020) relation of r s = .39 (p = .07) between a composite SAGAT score (SA levels 1-3) with the percentage of visual attention on the patient monitor. In the context of tripping hazards on construction sites, Hasanzadeh et al. (2016) correlated several eye tracking metrics of undergraduates working on a construction site task with a subjective situation awareness measure (Situational Awareness Rating Technique, SART) and observed a negawith situation awareness. However, the area of interest (AOI) description was limited and the negative association was not discussed. In a further analysis, Hasanzadeh et al. (2018) reported that the four participants with high SART scores payed more attention to feedforward-information and environmental-related AOI than seven participants with low SART scores. Finally, in a simple gage monitoring laboratory-based task, de Winter et al. (2019) observed a correlation of 0.31 task-relevant AOI and a task performance score; however, a freeze-probe score-percentage of glances correlation was only 0.10 (de Winter, personal communication).
By investigating the correlation of the tribute to and extend the above research in several ways. First, previous research with actual domain experts (e.g., Law et al., 2020;Moore & Gugerty, 2010) was limited in relation to sample size (N = 11-29). The present sample previous research considered only a single AOI icon or vital sign monitor) for the analysis of value of an AOI in relation to the main goals of an operator is assessed and combined. This as many AOI as needed in one combined analysis. Third, considering only one AOI limits any situation awareness question related to this AOI (Moore & Gugerty, 2010). Otherwise, the plausibility of associating situation awareness questions not relating to the AOI under study may on the basis of cognitive science and engineering research on visual attention and therefore provides a theoretical foundation to investigate the relation between attention distribution and situation awareness.

The Salience Effort Expectancy Value (SEEV) Model
salience, effort, expectancy, and value to predict the allocation of overt visual attention in a supervisory control task (for reviews, see Steelman et al., 2017;Wickens, 2015). The factor salience denotes the features of an AOI that possibly will attract attention. The factor information such as head turns. Salience and depend on the environment. The factor expectancy refers to the bandwidth (i.e., rate of information change) within an AOI. The factor expectancy combines the rate of information have an expectation about the event rate in a the importance of an AOI to the main goal of an down factors and depend mainly on the knowledge of the operator. equation model (Wickens et al., 2003), the faccombined to predict the so-called percentage dwell time (PDT). The PDT is the relative proportion of time for which attention is allocated top-down factors, expectancy and value, in the model. Wickens (2015) and Wickens et al. (2008) assess "optimal" attention distribution, because is driven only by factors that should guide the therefore, a gold standard against which the observed dwell time distributions can be compared. In addition, empirical studies showed (Steelman et al., 2011(Grundgeiger et al., 2020Wickens et al., 2008) Wickens et al. (2008 model and situation awareness by suggesting the so-called Attention-Situation Awareness model. In short, the model incorporates two modules. The attention module includes the of overt visual attention and therefore describes the information seeking behavior of an operator. The attended information is integrated by the belief module to update the level of situation awareness. High situation awareness enables the operator to make informed decisions (rather than relying on outdated information or preexisting knowledge) but also enables the capacity to guide attention to events of high value. In relation to situation awareness levels ( 1995), the attention module represents level 1 (perception of the environment), and the belief module represents level 2 and 3 (integration of information, anticipation of the future; Wickens et al., 2008). To the best of our knowledge, the full Attention-Situation Awareness model has so far been used only in a sensitivity analysis in which the parameters were manipulated in a computational simulation (Wickens et al., 2008).

The Present Study
In the present study, junior and senior anesthesiologists wore a mobile eye tracker and induced general anesthesia in a simulated patient (manikin). For situation awareness, we focused on the most critical and demanding phase, which is the endotracheal intubation (Weinger et al., 2000). To measure SA level 1, we implemented several events into this phase that were relevant for the immediate management of the patient such as a sudden drop in patient heart rate from 74 to 51 bpm. In addition, following Cooper et al. (2010), we implemented further events that were not relevant for immediate patient safety to investigate the general perception of environmental events, such as an ologists were led outside the simulation room, they completed a questionnaire that included questions about clinical variables and events and environmental events (Table 1) to assess the awareness of the anesthesiologist during the phase of endotracheal intubation. Finally, we asked one open-ended question addressing SA level 2 and one question addressing SA level 3.
individual anesthesiologist by correlating the observed PDT based on the eye tracking recordings of the whole procedure of inducing general model. Next, we correlated the individual model awareness scores. Based on the Attention-Situation Awareness model, we expected a posfocus on the laryngoscopy for endotracheal intubation, we also separately assessed the copy phase with perception scores. Finally, we ticipants who were scored to have overall situation awareness based on the assessment of a blinded subject-matter expert reviewing the answers to all questions (including SA levels 2 and 3) with the other participants. We expected that participants with situation awareness would other participants.

METHOD Participants
A total of 67 anesthesiologists of the Department of Anaesthesia and Critical Care at the University Hospital Würzburg participated. Due to technical failure (3), failure in the experimental procedure (1), poor eye tracking quality (1), or no consent for eye tracking (1), six participants were excluded from the analysis. Junior anesthesiologists had a mean age of M = 30 years (SD = 4; gender f/m: 19/11) and a

Design
An independent variable was experience (junior, senior). An anesthesiologist with a special 5-year training in anesthesiology (i.e., consultant) was considered to be a senior anesthesiologist. Half of the participants used a video-based device for laryngoscopy and the other half used a socalled Macintosh blade which is used as the standard device for a direct laryngoscopy. However, this comparison is not the focus of the present analyses.
As dependent variable, we measured the percalculated a SA level 1 score that included only events that were immediately relevant for managing the patient. We calculated an environment perception score that included events that happened in the simulation room environment but were not critical for immediate patient safety. We combined the SA level 1 score and the environment perception score in a global situation perception score. In addition, we used one question each to address situation awareness levels 2 and 3. For an overall situation awareness score, a blinded subject matter expert combined all SA level questions and classi-ness or not (see method section).

Procedure and Material
Participants put on the mobile eye tracker case history, precheck of equipment, preoxygenation and induction, mask ventilation and muscle (laryngoscopy), several events happened that were either triggered by the experimenter in the control room (i.e., operating room light switched used to measure the perception of the environment and SA level 1. For a detailed description of the content and timing on the events, see supplemental material. After the blade of the laryngoscopy device was removed from the mannequin's mouth (i.e., end of laryngoscopy), the experimenter entered the simulation room, indicated that the scenario was over, and guided the participant out of the simulation room. Approximately 10-15 seconds after the end naire including 13 questions about the laryngoscopy phase. The questions are provided in Table 1 and an illustration of the respective events is provided in supplemental Figure S1. Because we focused on the laryngoscopy, we did not conduct a goal directed task analysis but subject matter experts (authors CM and OH) developed situation awareness questions considering the anesthesiologist's goals for the induction of general anesthesia (King & Weavind, 2017;Weinger & Slagle, 2002). For the environmental perception, we tested to what extent the anesthesiologist perceived the changes in the environment.
Finally, participants provided demographic data. The procedure was rehearsed several times to ensure the exact timing of events and was tested with a pilot participant. All partic- The average scenario length was 7:59 min (no the whole experimental session lasted approximately 30 min.

Calculation of Perception Scores
answered question scored one point. If no answer was given, we did not consider the answer in the analysis because it is unclear whether the participants did not know the answer or accidentally skipped a question (7 instances, 96% of the answers). If only one part of a question was addressed or participants indicated with a symbol such as a question mark that they did not know the answer, we assigned zero points (12 instances, 1.64% of the answers).
We calculated the scores based on the relative proportion of correctly answered questions. We calculated the global situation perception score based on eleven questions, the environment perception score based on the seven questions that related to events in the environment but were not immediately relevant for patient safety, and a SA level 1 score based on the clinically rele-Finally, author OH was blinded in relation to any participants' demographics and assessed the answers to all questions and the two open questions regarding situation awareness levels 2 and 3 and assigned an overall situation awareness score in relation to patient clinical status of either 0 or 1. OH did not receive any information about the participants such as experience level. For 19 participants, question 8 had a misleading formulation (i.e., we did not explicitly ask whether an operating room light but any sidered in the analysis for these participants.

Calculation of Model
We provide a brief summary of the AOI analysis, the model parameters, and the model calculations which were based on Wickens and colleagues (Wickens et al., 2001, Wickens et al., 2003. For a detailed description and example calculations, see Grundgeiger et al. (2020).
First, based on previous work (Grundgeiger et al., 2017;Schulz et al., 2011), we manually the single AOI in four semantically related groups (patient, monitoring equipment, documentation, medication + general equipment) and a "Not included in analysis" group (see note of Table 2 for assignment of single AOI and Supplemental Figure S2). Second, based on the literature (King & Weavind, 2017;Weinger & Slagle, 2002), the two main goals for the induction of general anesthesia can be described as (1) rapidly, safely, and pleasantly producing amnesia, analgesia, akinesia, and autonomic and sensory block while (2) cient ventilation. Third, considering the main goals for the induction of general anesthesia, three domain experts assigned the required model parameters. Fourth, we calculated the predicted PDT for the AOI groups (for more information, see supplemental Tables S1 and S2).
As in previous research (Wickens et al., 2003(Wickens et al., , 2008, we used Pearson correlations (r) to correlate the predicted PDTs according to the model with the overall observed PDT to calcu-PDT by the model were correlated with the observed PDTs of each participant. We calculated Spearman's rho (r s ) correlations to assess awareness scores. The statistical analysis was conducted with IBM SPSS Statistics (Version 25.0. Armonk, NY: IBM Corp.). Alpha was set at .05.

Perception and Situation Awareness Scores
Overall, the situation perception scores and situation awareness scores were not high. Initially, we collected data of 50 participants and noted the general low scores. To test whether the scenario was too demanding to notice changes in relation to SA level 1 and build overall situation awareness, we ran the scenario with an additional twelve participants (six junior and six senior) but did not include any potentially distracting environmental perception events (i.e., change of clock dis-However, the SA level 1 scores were still low (M junior = .53, M senior = .47). We pooled the data for further analysis.
The analysis of the global situation perception score, the environment perception, and SA ence between junior and senior anesthesiologist (Table 3). Finally, the expert rating on the overall situation awareness of the patient based on all questions (including situation awareness levels 2 and 3) showed that 27% of the junior anesthesiologists and 26% of the senior anesthesiologists were aware of the patient status. Note. Values indicate M (SD). AOI group patient included patient's head, patient's thorax, IV access, face mask (phase 3-5, only), patient's mouth, video monitor of laryngoscope (phase 5, only), and patient's arm. AOI group monitoring equipment included anesthesia machine, patient monitor, changing settings on patient monitor, changing settings on anesthesia machine, and clock. AOI group documentation included patient record and anesthesia chart. AOI medication + general equipment group included nurse's hands, infusion, face mask (phase 1-2, only), respiratory tube, application of drugs, anesthesia trolley, video monitor of laryngoscope (phase 1-4, only), and monitoring cable. Not included in EV analysis were nurse's head, fixation due to movement in room, floor, walls, content of front pocket, IV pumps not in use, desk, entrance door to simulation, and not classified (i.e., sink, etc.). Table 2 shows the empirical dwell times on analysis. The correlation of the overall mean observed PDT of all participants and the predicted PDT based on 20 data points (5 phases correlation of r = .780, p < .001 (Figure 1, left panel).

SEEV Model Fits
ipant by correlation the observed PDT and the predicted PDT and used these correlations to thesiologists. Assessing the individual model ence between junior and senior anesthesiologists (Table 3). Considering only the laryngoscopy (phase 5), the overall mean observed PDT of all participants and predicted PDT showed a nonpoints (1 phase × 4 AOI groups) of r = .823, p = .177 (Figure 1, right panel). Again, assessing and senior anesthesiologists (Table 3).

Perception and Situation Awareness Scores, SEEV Model Fit, and PDT Correlations
each participant for all phases (1-5) and for phase 5 (laryngoscopy) with the global situation perception, SA level 1, and environment perception scores (Table 4). We observed a small r s = .293 for -tive correlation of r s laryngoscopy (phase 5) and SA level 1.
Finally, we used the expert categorization of participants regarding overall situation awareness based on the answers to all situation awarequestions regarding situation awareness levels between the participants (n = 16) who were scored to have overall situation awareness and the other participants (n = 45). Against our the group with overall situation awareness    Note. The data of 11 participants did not include any environmental events. Therefore, the n was different for the situation awareness level 1 correlations. The PDT of the AOI group documentation was zero in phase 5. Significant correlations (p < .05) are in bold font.
= .024) and without overall situation awareness (M = .821, SD = .014), t(59) = 1.417, p = .162, Hedges' g = .47. To complement our analysis, we also investigated the association of observed PDTs in phase 5 with the perception scores, as in previous research (Law et al., 2020;Moore & Gugerty, 2010). As summarized in Table 4, the PDT on ative correlations with global situation perception (r s (r s correlations with global situation perception (r s = .336), SA level 1 (r s = .257), and environmental perception (r s = .351). The PDT on the AOI icant positive correlation with global situation perception (r s = .416) and environmental perception (r s = .382). Finally, the observed PDT on the AOI group was zero in phase 5.

DISCUSSION
Overall, the hypotheses based on the Attention-Situation Awareness model (Wickens et al., 2008) were supported. Considering all participants, the positive correlations with SA level 1 and the envismall correlation with global situation perception (r s = .293). These results may be considered as support for the idea that situation perception builds up (or decreases) based on the preceding attention allocation (Wickens et al., 2008). That is, particiup situation perception in all phases of the induction and therefore showed increased scores in the situation perception scores. However, to fully support this explanation, the comparison of participants who were scored by the blinded domain expert to have overall situation awareness (levels 1-3) and the other participants should have shown the laryngoscopy (phase 5) and SA level 1 supa process situation awareness measure for level 1. was associated with higher SA level 1 scores. However, the correlation was small (r s = .255).
only addresses visual attention, and one may argue that two of our clinical events (the drop in heart rate and ventilator alarm) might have been primarily noticed via the sound of a dropping heart rate and an alarm sound. Furthermore, due to the experimental design in the present study, situation and with one SAGAT probe. Several SAGAT probes in various phases of a scenario would have resulted in more robust estimates for situation awareness scores and would have enabled an awareness scores for various phases.
between participants who were scored to have overall situation awareness (levels 1-3) and the other participants. However, the descriptive difference was in the expected direction, and the ' g = .47). Furthermore, in the Attention-Situation Awareness model (Wickens et al., 2008), the ered as the SA level 1 process measure (the attenthe overall situation awareness (the belief model). For example, in the present study, the anesthesiologists had expectations about the status of the patient based on previous phases (i.e., outdated but not necessarily wrong information) and preexisting response strategies based on anesthesiologists' general experience of inductions of general anesthesia. Direct situation awareness measures of expert participants in a representative task will response strategies, which may increase the likelihood of error but may also be correct in many situations. Future research in various safety-critical domains should aim at implementing the full Attention-Situation Awareness model to be able to make better predictions about the belief module (i.e., situation awareness levels 2 and 3). Finally, even though the senior anesthesiologists had more sicians and had knowledge and training in managing an induction of general anesthesia. This is also indicated by the similar and, in general, good M = .821; senior M = .824). For the correlation of situation in one variable poses a methodological problem. Future research addressing experience may con- (Hogan et al., 2006) or consider a more challenging scenario (Schulz et al., 2011) to increase the variability due to experience.
acceptable with an r of 0.780. Compared to a previous study on the induction of general anesthesia (r = .845, Grundgeiger et al., 2020b), the current the induction (mechanical ventilation and maintenance of general anesthesia) because we aborted the induction after the endotracheal intubation to conduct the situation awareness questionnaire. Like in a previous study (Grundgeiger et al., 2020b for junior and senior anesthesiologists. This may be due to the common and uneventful scenario (i.e., a scenario without any critical incidence but an uneventful induction of general anesthesia).
tion allocation between junior and senior anesthesiologists only in scenarios with a critical incident, such as a severe anaphylactic reaction, and not in uneventful scenarios (Schulz et al., 2011). Second, supposed to assess "optimal" attention distribution and represents a gold standard against which the observed PDT can be compared (Wickens et al., 2008). Our results further support the idea of an "optimal" attention distribution by showing and SA level 1.
Finally, we conducted a computational simpler analysis by correlating the observed PDT tion scores. The AOI patient indicated significant negative correlations between the PDT and global perception scores and environment time positive correlations were observed for correlations make sense considering that the environment perception events were not related to the AOI patient but may have been noticed when looking around the room. The positive equipment with SA level 1 makes sense because the content of the SA level 1 questions were present in the monitoring equipment. However, tion scores and environment perception scores were also positive and descriptively even larger than the correlation to monitoring equipment. Similar to the negative correlations of situation awareness and PDT on the supposing important AOI in a previous study Hasanzadeh et al. (2016) explain.
previous research investigating the association of eye tracking metrics and situation awareness measures (Law et al., 2020;Moore & Gugerty, 2010). Similar to Moore and Gugerty (2010), also similar to Moore and Gugerty (2010), the explained variance was small. One may question whether the more complex model analysis adds advantages. First, as explained in the introduction, the model can consider several AOI in one single analysis and, therefore, situation awareness questions are not limited to single AOI. Second, the model calculations are more complex compared to a pure correlation analysis Based on the above discussion, we consider the the correlation analysis. Third, the theoretical foundation of the model and the a priori parammore general predictions including several AOI compared to a correlational analysis. Similar to used to predict miss rates and response times (Steelman et al., 2013 Attention-Situation Awareness model may be used to predict situation awareness.
The study has several limitations. First, our study was powered only to detect a large expef awareness (e.g., ; Hogan et al., 2006). During laryngoscopy, experience may than we could have detected with our sample size. Second, our situation awareness measure was based only on a single SAGAT probe (and not several measurement time points, see Wright et al., 2004). For example, (2000) recommends 30-60 probes. More probes would have provided situation awareness estimates that are more reliable. However, based on the present research question, we did not see another possible approach to assess situation awareness. Third, our probe included only a limited number of questions. More items would have provided a more sensitive measure. However, the number of reasonable questions for the present procedure is limited (Dishman et al., 2020), and a recent review on situation awareness measures shows that other healthcare studies using the SAGAT have used a similar number of items ( ). Fourth, we did not conduct a goal directed task analysis to design the SAGAT items (Wright et al., 2004).
questions are in line with the goals and SAGAT items for the induction of general anesthesia Dishman et al. (2020), using a goal directed task analysis.

CONCLUSION
The results support the suggestion that the represents "optimal" scanning. If future research method to analyze eye tracking data beyond the descriptive level and to be used as an estimate ilar to the suggestions of de Winter et al. (2019), bination with the Attention-Situation Awareness model may be able to predict situation awarecould be used for training or, if the analyses could be done in real time, the design of interfaces that may increase the salience of speenough attention. Furthermore, it appears that during the brief phase from the insertion until the removal of the laryngoscope blade, anesthesiologists do not have the capacity to also perceive and understand clinically relevant elements in the environment. Critically, this includes a sudden drop in the patient's heart rate from 74 to 51 bpm, which was noticed by only 13 of 61 participants. Work place and technology designers should be aware that these situations can occur, and they should consider technology design, such as more salient information presentation or changed work practices, to empower operators to ask attending