Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements

Healthy humans have a common set of 34 muscle groups, 17 on each side of the face, that contract and relax in patterns.¹⁰ To create facial movements that are visible to the naked eye, facial muscles contract, changing the distance between facial features (Neth & Martinez, 2009) and shaping skin into folds and wrinkles on an underlying skeletal structure. Even when facial movements look the same to the naked eye, there may be differences in their execution under the skin. There are individual differences in the mechanics of making a facial movement, including variation in the anatomical details (e.g., muscle configuration and relative size vary, and some people lack certain muscle components), in the neural control of those muscles (Cattaneo & Pavesi, 2014; Hutto & Vattoth, 2015; Müri, 2015), and in the underlying skeletal structure of the face (discussed in Box 5 in the Supplemental Material).

There are three common procedures for measuring facial movements in a scientific experiment. The most sensitive, objective measure of facial movements, called facial electromyography (EMG), detects the electrical activity from actual muscular contractions (again, see Box 5 in the Supplemental Material). This is a perceiver-independent way of assessing facial movements that detects muscle contractions that are not necessarily visible to the naked eye (Tassinary & Cacioppo, 1992). The utility of facial EMG is unfortunately offset by its impracticality: It requires placing electrodes on a participant’s face in a particular configuration. In addition, a person can typically tolerate only a few electrodes on the face at one time. At the writing of this article, relatively few published articles (we identified 123) reported the use of facial EMG, the overwhelming majority of which sparsely sampled the face, measuring the electrical signals for only a small number of muscles (between one and six); none of the studies measured naturalistic facial movements as they occur outside the lab, in everyday life. Consequently, we focus our discussion on two other measurement methods: a perceiver-dependent method that describes visible facial movements, called facial actions. Human coders indicate the presence or absence of a facial action while viewing video recordings of participants. Automated methods also exist for detecting facial actions from photographs or videos.

Measuring facial movements with human coders

The Facial Action Coding System, or FACS (Ekman, Friesen, & Hager, 2002), is a systematic approach to describe what a face looks like when facial muscle movements have occurred. FACS codes describe the presence and intensity of facial movements. FACS is purely descriptive and is therefore agnostic about whether those movements might express emotions or any other mental event.¹¹ Human coders train for many weeks to reliably identify specific movements called action units (AUs). Each AU is hypothesized to correspond to the contraction of a distinct facial muscle or a distinct grouping of muscles that is visible as a specific facial movement. For example, the raising of the inner corners of the eyebrows (contracting the frontalis muscle pars medialis) corresponds to AU 1. Lowering of the inner corners of the brows (activation of the corrugator supercilii, depressor glabellae, and depressor supercilii) corresponds to AU 4. AUs are scored and analyzed as independent elements, but the underlying anatomy of many facial muscles constrains them so that they cannot move independently of one another, which generates dependencies between AUs (e.g., see Hao, Wang, Peng, & Ji, 2018). A list of facial AUs and their corresponding facial muscles can be found in Figure 5. Expert FACS coders approach interrater reliabilities of .80 for individual AUs (Jeni, Cohn, & De la Torre, 2013). The first version of FACS (Ekman & Friesen, 1978) was based largely on the work of Swedish anatomist Carl-Herman Hjortsjö, who catalogued the facial configurations described by Duchenne (Hjortsjö, 1969). In addition to the updated versions of FACS (Ekman et al., 2002), other facial coding systems have been devised for human infants (Izard et al., 1995; Oster, 2007), chimpanzees (Vick, Waller, Parr, Smith Pasqualini, & Bard, 2007) and macaque monkeys (L. A. Parr, Waller, Burrows, Gothard, & Vick, 2010; see also L. F. Barrett, 2017a). Figure 4 displays the common FACS codes for the configurations of the facial movements that have been proposed as the prototypic expressions of anger, disgust, fear, happiness, sadness, and surprise, respectively.

Download

Open in new tab

Download in PowerPoint

Fig. 5. Facial Action Coding System (FACS; Ekman & Friesen, 1978) codes for adults. AU = action unit. Images for AUs 1 to 6 are reproduced here with permission from Jeffrey Cohn. Images for AUs 7 to 46 are from the CMU-Pittsburgh AU-Coded Face Expression Image Database (Kanade, Cohn, & Tian, 2000).

Once an approach has been chosen for measuring facial movements, a clear test of the common view of emotional expressions depends on having valid measures that reliably and specifically characterize, in a generalizable way, the instances of each emotion category to which the measurements of facial muscle movements can be compared. The methods that scientists use to assess people’s emotional states vary in their dependence on human inference, however, which raises questions about the validity of the measures.

If a specific facial configuration reliably expresses instances of a certain emotion category in any given experiment, then we would expect measurements of the face (e.g., facial AU codes) to co-occur with other measures that indicate that participants are in the target emotional state. In principle, those measures might be more objective (e.g., ANS changes during an emotional event) or they might be more subjective (e.g., ratings provided by the participants themselves). In practice, however, the vast majority of experiments compare facial movements with subjective measures of emotion—a scientist’s judgment about which emotions are likely to be evoked by a particular stimulus, the judgments of other human observers about participants’ emotional states, or participants’ self-reports of emotional experience. For example, in an experiment, scientists might ask questions like these: Do the AUs that create a scowling facial configuration co-occur with self-reports of feeling angry? Do the AUs that create a pouting facial configuration co-occur with perceiver’s judgments that participants are sad? Do the AUs that create a wide-eyed, gasping facial configuration co-occur when people are exposed to an electric shock? If such observations suggest that a configuration of muscle movements is reliably observed during episodes of a given emotion category, then those movements are said to express the emotion in question. As we will see, many studies show that some facial configurations occur more often than random chance would allow but are not observed with a high degree of reliability (according to the criteria from Haidt and Keltner, 1999, explained in Table 2 of the current article).

If a facial configuration specifically (i.e., uniquely) expresses instances of a certain emotion category in any given experiment, then we would expect to observe little co-occurrence between measurements of the face and measurements indicating the presence of emotional instances from other categories (again, see Table 2 and Fig. 3).

If a configuration of facial movements is observed in instances of a certain emotion category in a reliable, specific way within an experiment, so that we can infer that the movements are expressing an instance of the emotion in that study as hypothesized, then scientists can safely infer that the facial movements in question are an expression of that emotion category’s instances in that situation. One more step is required before we can infer that the facial configuration is the expression of that emotion: We must observe a similar pattern of facial configuration–emotion co-occurrences across different experiments, to some extent generalizing across the specific measures and methods used and the participants and contexts sampled. If the facial configuration–emotion co-occurrences replicate across experiments that sample people from the same culture, then the facial configuration in question can reasonably be referred to as an emotional expression only in that culture; for example, if a scowling facial configuration co-occurs with measures of anger (and only anger) across most studies conducted on adult participants in the United States who are free from illness, then it is reasonable to refer to a scowl as an expression of anger in healthy adults in the United States. If facial configuration–emotion co-occurrences generalize across cultures—that is, if they are replicated across experiments that sample a variety of instances of that emotion category in people from different cultures—then the facial configuration in question can be said to universally express the emotion category in question.

A meta-analysis was recently conducted to test the hypothesis that the facial configurations in Figure 4 co-occur, as hypothesized, with the instances of specific emotion categories (Duran, Reisenzein, & Fernández-Dols, 2017). Thirty-seven published articles reported on how people moved their faces when exposed to objects or events that evoke emotion. Most studies included in the meta-analysis were conducted in the laboratory. The findings from these experiments were statistically summarized to assess the reliability of facial movements as expressions of emotion (see Fig. 6). In all emotion categories tested, other than fear, participants moved their facial muscles into the expected configuration more reliably than what we would expect by chance. Reliability levels were weak, however, indicating that the proposed facial configurations in Figure 4 have limited reliability (and to some extent, limited generalizability; i.e., a scowling facial configuration is an expression of anger, but not the expression of anger). More often than not, people moved their faces in ways that were not consistent with the hypotheses of the common view. An expanded version of this meta-analysis (Duran & Fernández-Dols, 2018) analyzed 131 effect sizes from 76 studies totaling 4,487 participants, with similar results: The hypothesized facial configurations were observed with average effect sizes (r) of .31 for the correlation between the intensity of a facial configuration and a measure of anger, disgust, fear, happiness, sadness, or surprise (corresponding to weak evidence of reliability; individual correlations for specific emotion categories ranged from .06 to .45, interpreted as no evidence of reliability to moderate evidence of reliability). The average proportion of the times that a facial configuration was observed during an emotional event (in one of those categories) was .22 (proportions for specific emotion categories ranged from .11 to .35, interpreted as no evidence to weak evidence of reliability).¹⁹

Download

Open in new tab

Download in PowerPoint

Fig. 6. Meta-analysis of facial movements during emotional episodes: a summary of effect sizes across studies (Duran, Reisenzein, & Fernández-Dols, 2017). Effect sizes are computed as correlations or proportions (as reported in the original experiments). Results include experiments that reported a correspondence between a facial configuration and its hypothesized emotion category as well as those that reported a correspondence between individual AUs of that facial configuration and the relevant emotion category; meta-analytic effect sizes that summarized only the effects for entire ensembles of AUs (the facial configurations specified in Fig. 4) were even lower than those reported here.

No overall assessment of specificity was reported in either the original or the expanded meta-analysis because most published studies do not report the false-positive rate (i.e., the frequency with which a facial AU is observed when an instance of the hypothesized emotion category was not present; see Fig. 3). Nonetheless, some striking examples of specificity failures have been documented in the scientific literature. For example, a certain smile, called a Duchenne smile, is defined in terms of facial muscle contractions (i.e., in terms of facial morphology): It involves movement of the orbicularis oculi, which raises the cheeks and causes wrinkles at the outer corners of the eyes, in addition to movement of the zygomaticus major, which raises the corners of the lips into a smile. A Duchenne smile is thought to be a spontaneous expression of authentic happiness. Research shows, however, that a Duchenne smile can be intentionally produced when people are not happy (Gunnery & Hall, 2014; Gunnery, Hall, & Ruben, 2013; also see Krumhuber & Manstead, 2009), consistent with evidence that Duchenne smiles often occur when people are signaling submission or affiliation rather than reflecting happiness (Rychlowska et al., 2017).

Studies of facial configuration–emotion category associations in naturalistic settings tend to yield results similar to those from studies that were conducted in more controlled laboratory settings (Fernández-Dols, 2017; Fernández-Dols & Crivelli, 2013). Some studies observe that people express emotions in real-world settings by spontaneously making the facial muscle movements proposed in Figure 4, but such observations are generally not replicable across studies (e.g., cf. Matsumoto & Willingham, 2006 and Crivelli, Carrera, & Fernández-Dols, 2015; cf. Rosenberg & Ekman, 1994 and Fernández-Dols, Sanchez, Carrera, & Ruiz-Belda, 1997). For example, two field studies of winning judo fighters recently demonstrated that so-called Duchenne smiles were better predicted by whether an athlete was interacting with an audience than the degree of happiness reported after winning their matches (Crivelli et al., 2015). Only 8 of the 55 winning fighters produced a Duchenne smile in Study 1; all occurred during a social interaction. Only 25 of 119 winning fighters produced a Duchenne smile in Study 2, documenting, at best, weak evidence for reliability.

Another source of evidence comes from asking participants sampled from various cultures to deliberately pose the facial configurations that they believe they use to express emotions. In these studies, participants are given a single emotion word or a single, brief statement to describe each emotion category and are then asked to freely pose the facial configuration that they believe they make when expressing that emotion. Such research directly examines common beliefs about emotional expressions. For example, one study provided college students from Canada and Gabon (in Central Africa) with dictionary definitions for 10 emotion categories. After practicing in front of a mirror, participants posed the facial configurations so that “their friends would be able to understand easily what they feel” (Elfenbein, Beaupre, Levesque, & Hess, 2007, p. 134) and their poses were FACS coded. Likewise, a recent study asked college students in China, India, Japan, Korea, and the United States to pose the facial movements they believe they make when expressing each of 22 emotion categories (Cordaro et al., 2018). Participants heard a brief scenario describing an event that might cause anger (“You have been insulted, and you are very angry about it”) and then were instructed to pose a facial (and nonverbal but vocal) expression of emotion, as if the events in the scenario were happening to them. Experimenters were present in the testing room as participants posed their responses. Both studies found moderate to strong evidence that participants across cultures share common beliefs about the expressive pose for anger, fear, and surprise categories; there was weak to moderate evidence for the happiness category, and weak evidence for the disgust and sadness categories (Fig. 7). Cultural variation in participants’ beliefs about emotional expressions was also observed.

Download

Open in new tab

Download in PowerPoint

Fig. 7. Comparing posed and spontaneous facial movements. Correlations or proportions are presented for anger, disgust, fear, happiness, sadness, and surprise, separately for three studies. Data are from Table 6 in Cordaro et al. (2018), from Elfenbein, Beaupre, Levesque, and Hess (2007; reliability for the anger category is for AU4 + AU5 only), and from Duran, Reisenzein, and Fernández-Dols (2017; proportion data only).

Neither study compared participants’ posed expressions (their beliefs about how they move their facial muscles to express emotions) with observations of how they actually moved their faces when expressing emotion. Nonetheless, a quick comparison of the findings from the two studies and the proportions of spontaneous facial movements made during emotional events (from the Duran et al., 2017 meta-analysis) makes it clear that posed and spontaneous movements differ, sometimes quite substantially (again, see Fig. 7). When people pose a facial configuration that they believe expresses an emotion category, they make facial movements that more reliably agree with the hypothesized facial configurations in Figure 4. The same cannot be said of people’s spontaneous facial movements during actual emotional episodes, however (for convergent evidence, see Motley & Camden, 1988; Namba, Makihara, Kabir, Miyatani, & Nakao, 2016). One possible interpretation of these findings is that posed and spontaneous facial-muscle configurations correspond to distinct communication systems. Indeed, there is some evidence that volitional and involuntary facial movements are controlled by different neural circuits (Rinn, 1984). Another factor that may contribute to the discrepancy between posed and spontaneous facial movements is that people’s beliefs about their own behavior often reflect their stereotypes and do not necessarily correspond to how they actually behave in real life (see Robinson & Clore, 2002). Indeed, if people’s beliefs, as measured by their facial poses, are influenced directly by the common view, then any observed relationship between posed facial expressions and hypothesized emotion categories is merely evidence of the beliefs themselves.

Our review of the available evidence thus far is summarized in the first through third data rows in Table 3. The hypothesized facial configurations presented in Figure 4 spontaneously occur with weak reliability during instances of the predicted emotion category, suggesting that they sometimes serve to express the predicted emotion. Furthermore, the specificity of each facial configuration as an expression of an emotion category is largely unknown (because it is typically not reported in many studies). In our view, this pattern of findings is most compatible with the interpretation that hypothesized facial configurations are not observed reliably or specifically enough to justify using them to infer a person’s emotional state, whether in the lab or in everyday life. We are not suggesting that facial movements are meaningless and devoid of information. Instead, the data suggest that the meaning of any set of facial movements may be much more variable and context-dependent than hypothesized by the common view.

Table 3. Reliability and Specificity: A Summary of the Evidence

Table 3. Reliability and Specificity: A Summary of the Evidence

View larger version

Our review of scientific studies that systematically measure the spontaneous facial movements in people of small-scale, remote cultures is brief by necessity: There are no such studies. At the time of publication, we were unable to identify even a single published report or manuscript registered on open-access, preprint services that measured facial muscle movements in people of remote cultures as they experienced emotional events. Scientists have almost exclusively observed how people from remote cultures label facial configurations as emotional expressions (i.e., studying emotion perception, not production) to test the hypothesis that certain facial configurations evolved to express certain emotion categories in a reliable, specific, and generalizable (i.e., universal) manner. Later in this article, we return to this issue and discuss the findings from these emotion-perception studies.

There are nonetheless several descriptive reports that provide support for the common view of universal emotional expressions (similar to what Valente, Theurel, & Gentaz, 2018, refer to as an “observational approach”). For example, the U.S. psychologist Paul Ekman and colleagues curated an archive of photographs of the Fore hunter-gatherers taken during his visits to Papua New Guinea in the 1960s (Ekman, 1980). The photographs were taken as people went about their daily activities in the small hamlets of the eastern highlands of Papua New Guinea. Ekman used his knowledge of the situation in which each photograph was taken to assign each facial configuration to an emotion category, leading him to conclude that the Fore expressed emotions with the proposed facial configurations shown in Figure 4. Yet different scientific methods yielded a contrasting conclusion. When Trobriand Islanders living in Papua New Guinea were asked to infer emotions in facial configurations by labeling these same photographs in their native language, both by freely offering words and by choosing the best fitting emotion word from a list of nine choices, they did not label the facial configurations as proposed by Ekman and colleagues, at above-chance levels (Crivelli, Russell, Jarillo, & Fernández-Dols, 2017).²¹ In fact, the proposed fear expression—the wide-eyed, gasping face—is reliably interpreted as an expression of threat (intent to harm) and anger by the Maori of New Zealand and by the Trobriand Islanders in Papua New Guinea (Crivelli, Jarillo & Fridlund, 2016).

A compendium of spontaneous human behavior published by the Austrian ethologist Irenäus Eibl-Eibesfeldt (1989) is sometimes cited as evidence for the hypothesis that certain facial movements are universal signals for specific emotion categories. No systematic coding procedure was used in his investigations, however. On close examination, Eibl-Eibesfeldt’s detailed descriptions appear to be more consistent with results from the studies of people living in more industrialized cultures that we reviewed above: People move their faces in a variety of ways during episodes belonging to the same emotion category. For example, as reported by Eibl-Eibesfeldt, a rapid eyebrow raise (called an eyebrow flash) is thought to express friendly recognition in some cultures but not all. Likewise, particular facial muscle movements are not specific expressions of a given emotion category. For example, an eyebrow flash would be coded with FACS AU 1 (inner brow raise) and AU 2 (outer brow raise), which are part of the proposed expressions for surprise and fear (Ekman, Levenson, & Friesen, 1983), sympathy (Haidt & Keltner, 1999), and awe (Shiota, Campos, & Keltner, 2003). Even Eibl-Eibesfeldt acknowledged that eyebrow flashes were not unique expressions of specific emotion categories, writing that they also served as a greeting, as an invitation for social contact, as a sign of thanks, as an initiation of flirting, and as a general indication of “yes” in Samoans and other Polynesians, in the Eipo and Trobriand islanders in Papua New Guinea, and in the Yanomami of South America. In Japan, eyebrow flashes are considered an impolite way for adults to greet one another. In the United States and Europe, an eyebrow flash was observed when greeting friends but not when greeting strangers.

In the only study of expression production in a natural environment that we could find, researchers read a brief emotion story to people who live in the remote Fore culture of Papua New Guinea and asked each person to “show how his face would appear” (Ekman, 1972, p. 273) if he was the person described in the emotion story (sample size was not reported). Videotapes of 9 participants were shown to 34 U.S. college students who were asked to judge which emotion was being expressed. U.S. participants were asked to infer the emotional meaning of the facial poses by choosing an emotion word from six choices provided by the experimenter (called a choice-from-array task; discussed on page 31 of this article). Participants inferred the intended emotional meaning at above-chance levels for smiling (happiness, 73%), frowning (sadness, 68%), scowling (anger, 51%), and nose-wrinkling (disgust, 46%), but not for surprise and fear (27% and 18%, respectively).

Our review of the available evidence from expression-production studies in small-scale, remote cultures is inconclusive because there are no systematic, controlled observations that examine how people who live in these cultural contexts spontaneously move their facial muscles during emotional episodes. The evidence that does exist suggests that common beliefs about emotion may share some similarities across urban and small-scale cultural contexts, but more research is needed before any interpretations are warranted. These findings are summarized in the fourth and fifth data rows of Table 3.

The most detailed research on facial movements in fetuses and newborns has focused on smiles. Human fetuses lower their brows (AU4), raise their cheeks (AU6), wrinkle their noses (AU9), crease their nasolabia (AU11), pull the corners of their lips (AU12), show their tongues (AU19), part their lips (AU25), and stretch their mouths (AU27)—all of which have been implicated, to some degree, in adult laughter. Infants sometimes produce facial movements that resemble adult laughter when other considerations suggest that they are in distress and pain (Dondi et al., 2012; Hata et al., 2013; Reissland, Francis, & Mason, 2013; Reissland, Francis, Mason, & Lincoln, 2011; Yan et al., 2006). Within 24 hr of birth, infants raise their cheek muscles in response to being touched (Cecchini, Baroni, Di Vito, & Lai, 2011). But these movements are not specific to smiling; neonates also raise their cheeks (contract the zygomatic muscle) during rapid eye movement (REM) sleep, when drowsy, and during active sleep (Dondi et al., 2007). A neonatal smile with raised cheeks is caused by brainstem activation (Rinn, 1984), and likely reflects internally generated arousal rather than expressing or communicating an emotion or even a more general feeling of pleasure (Emde & Koenig, 1969; Sroufe, 1996; Wolff, 1987). So, it remains unclear whether fetal or neonatal facial muscle movements have any relationship to specific emotional episodes as well as more generally to pleasant feelings or to other social meanings (Messinger, 2002).

In fact, it is not clear that fetal and neonatal facial movements always have a psychological meaning (consistent with a behavioral-ecology view of facial movements; Fridlund, 2017). Newborns appear to produce some combinations of facial movements for muscular reasons. For example, infants produce facial movements associated with the proposed expression for surprise (open mouth and raised eyebrows) in situations that are unsurprising, just because opening the mouth necessarily raises their eyebrows; conversely, infants do not consistently show the proposed expressive configuration for surprise in contexts that are likely to be surprising (Camras, 1992; Camras, Castro, Halberstadt, & Shuster, 2017). The facial movement that is part of the proposed expression for sadness (brows oblique and drawn together) occurs when infants attempt to lift their heads to direct their gaze (Michel, Camras, & Sullivan, 1992).

In addition, newborns produce many facial movements that co-occur with fussiness, distress, focused attention, and distaste (Oster, 2005). Newborns react to being given sweet versus sour liquids; for example, when given a sour liquid, newborns make a nose-wrinkle movement, which is part of the proposed expressive configuration for disgust (Granchrow, Steiner, & Daher, 1983). However, other studies show that newborns also make this facial movement when given sweet, salty, and bitter tastes (e.g., Rosenstein & Oster, 1988). Still other studies show that nose-wrinkling does not always occur when infants taste lemon juice (i.e., when that facial movement is expected; Bennett, Bendersky, & Lewis, 2002). More generally, infants rarely produce consistent facial movements that cleanly map onto any single emotion category. Instead, infants produce a variety of facial configurations that suggest a lack of emotional specificity (Matias & Cohn, 1993).

There are further examples that illustrate how infant facial movements lack strong reliability and specificity. In a study of 11-month-old babies from the United States, China, and Japan, infants saw a toy gorilla head that growled (to induce fear) or their arms were restrained (to induce anger; Camras et al., 2007). Observers judged the infants to be fearful or angry on the basis of their body movements, yet the infants produced the same facial movements in the two situations.²² In another study, 1-year-old infants were videotaped in situations in which they were tickled (to elicit joy), tasted sour flavors (to elicit disgust), watched a jack-in-the box (to elicit surprise), had an arm restrained (to elicit anger), and were approached by a masked stranger (to elicit fear; Bennett et al., 2002). Infants whose arms were restrained (to purportedly induce an instance of anger) produced the facial actions associated with the proposed facial configuration for an anger expression only 24% of the time (low reliability); instead, 80 infants (54%) produced the facial actions proposed as the expression of surprise, 37 infants (25%) produced the facial actions proposed as the expression of joy, 29 infants (19%) produced the facial actions proposed as the expression of fear, and 28 (18%) produced the facial actions proposed as the expression of sadness. This dramatic lack of specificity was observed for all emotion categories studied. An equal number of babies produced facial movements that are proposed as the expressions of joy, surprise, anger, disgust, and fear categories when a sour liquid was placed on infants’ tongues to elicit disgust. When infants faced a masked stranger, only 20 (13%) produced facial movements that corresponded to the proposed expression for fear, compared with 56 infants (37%) who produced facial actions associated with the proposed expression for instances of joy.²³

Taken together, these findings suggest that infant facial movements may be associated with affect (i.e., the affective features of experience, such as distress or arousal), as originally described by Bridges (1932), or may communicate a desire to approach or avoid something (e.g., Lewis, Sullivan, & Kim, 2015). Affective features such as valence (ranging from pleasantness to distress) and arousal (ranging from activated to quiescent) are continuous properties of experience, just as approach/avoidance is an affective property of action. These affective features are shared by many instances of different emotion categories, as well as with mental events that are not considered emotional (as discussed in Box 9 in the Supplemental Material), but this does not diminish their importance or effectiveness for infants.²⁴ Over time, infants likely learn to differentiate mental events with simple affective features into episodes of emotion with additional psychological features that are specific to their sociocultural contexts, making them maximally effective at eliciting needed responses from their caregivers (L. F. Barrett, 2017b; Holodynski & Friedlmeier, 2006; Weiss & Nurcombe, 1992; Witherington, Campos, & Hertenstein, 2001).

The affective meaning of an infant’s facial movements may, in fact, be what makes these movements so salient for adult observers. When infants move their lips, open their mouths, or constrict their eyes, adults view infants as feeling more pleasant or unpleasant depending on the context (Bolzani Dinehart et al., 2005). Infant expressions thus do have a reliable link to instrumental effects in the adults who observe them—playing an important role in parent–infant interaction, attachment, and the beginnings of social communication (Atzil, Gao, Fradkin, & Barrett, 2018; Feldman, 2016). For example, if an infant cries with narrowed eyes, adults infer that the infant is feeling negative, is having an unwanted experience, or is in need of help, but if the infant makes that same eye movement while smiling, adults infer that the infant is experiencing more positive emotion. These data consistently point to the usefulness of facial movements in the communication of arousal and valence, particularly when combined or with other communicative features such as vocalizations (properties of affect; see Box 9 in the Supplemental Material). Even when episodes of more specific emotions start to emerge, we do not yet have evidence that facial movements map reliably and regularly to a specific emotion category.

Young children begin to produce adult-like facial configurations after the first year of life. Even then, however, children’s facial movements continue to lack strong reliability and specificity (Bennett et al., 2002; Camras & Shutter, 2010; Matias & Cohn, 1993; Oster, 2005). Examples of a wide-eyed, gasping facial configuration, proposed as the expression of fear (see Fig. 4), have rarely been observed or reported in young infants (Witherington, Campos, Harriger, Bryan, & Margett, 2010). Nor do infants reliably produce a scowling facial configuration, proposed as the expression of anger (again, see Fig. 4). Infants scowl when they cry or are about to cry (Camras, Fatani, Fraumeni, & Shuster, 2016). A frown (mouth corner depression, AU15) is not reliably and specifically observed when infants are frustrated (Lewis & Sullivan, 2014; Sullivan and Lewis, 2003). A smile (cheek raising and lip corner pulling, AU6 and AU12) is not reliably observed when infants are in visually engaging or mastery situations, or even when they are in pleasant social interactions (Messinger, 2002).

Experiments that observe young children’s facial movements in naturalistic settings find largely the same results as those conducted in controlled laboratory settings. For example, one study trained ethnographic videographers to record a family’s daily activities over 4 days (Sears, Repetti, Reynolds, & Sperling, 2014). Coders judged whether or not the child from each participating family made a scowling facial configuration (referred to as an expression of anger), a frowning facial configuration (referred to as an expression of sadness), and so on, for the six (presumed) emotion categories included in the study—happiness, sadness, surprise, disgust, fear, and anger. During instances that were coded as anger (defined as situations that included verbal disagreements or sibling bickering, requests for compliance and/or reprimands from parents, parent refusal of child requests, during homework, and sibling provocation), a variety of facial movements were observed, including frowns, furrowed brows, and eye-rolls, as well as a variety of vocalizations, including shouts and whining, and both nonaggressive and aggressive physical behaviors.

Perhaps the most telling observation for our purposes is that expressions of anger were more often vocal than facial. During anger situations, children raised their voices 42% of the time, followed by whining about 21% of the time. By contrast, children made scowling facial configurations only 16.2% of the time.²⁵ Yet even during anger situations, the facial movements were predominantly frowning, which can be part of many different proposed facial configurations. The authors reasoned that children engage in specific behaviors to obtain specific goals, and that behaviors such as whining are more likely to attract attention and possibly change parental behavior than is a facial movement. Indeed, it is easier for parents to ignore a negative facial expression than a whining child in the room! Similar findings for low reliability and specificity of the facial configurations presented in Figure 4 were recently observed in a naturalistic study that videotaped 7- to 9-year-old children and their mothers discussing a conflict during their visit to the laboratory related to homework, chores, bedtime, or interactions with siblings (Castro, Camras, Halberstadt, & Shuster, 2018).

Newborns and infants react to the world around them with facial movements. There is not yet sufficient evidence, however, to conclude that these facial movements reliably and specifically express the instances of any emotion category (findings are summarized in the sixth data row of Table 3). When considered alongside vocalizations and body movements, there is consistent evidence that infant facial movements reliably signal distress, interest, and arousal and perhaps serve as a call for help and comfort. In young children, instances of the same emotion category appear to be expressed with a variety of different muscle movements, and the same muscle movements occur during instances of various emotion categories, and even during nonemotional instances. It may be the case that reliability and specificity emerges through learning and development (see Box 10 in the Supplemental Material), but this remains an open question that awaits future research.

In the majority of the experiments that study emotion perception, researchers ask participants to infer emotion in photographs of posed facial configurations (such as those in Fig. 4, but without the FACS codes). In most studies, the configurations have been posed by people who were not in an emotional state when the photos were taken. In a growing number of studies, the poses are created with computer-generated humans who have no actual emotional state. As a consequence, it is not possible to assess the accuracy (i.e., validity) of perceivers’ emotional inferences and, correspondingly, data from emotion-perception studies should not be interpreted as support for the validity of the common view of emotional expressions (except insofar as these are simply stipulated to be the consensus). As is the case in expression-production studies, it is more appropriate to interpret participants’ responses in terms of their agreement (or consensus) with common beliefs (which may vary by language and culture).

Even more serious is the fact that the proposed expressive facial configurations in Figure 4, which are routinely used as stimuli in emotion-perception studies, do not capture the wider range of muscle movements that are observed when people actually express instances of these emotion categories in the lab or in everyday life. A recent study that mined more than 7 million images from the Internet (Srinivasan & Martinez, 2018; for method, see Box 7 in the Supplemental Material) identified multiple facial configurations associated with the same emotion-category label and its synonyms—17 distinct facial configurations were associated with the word happiness, five with anger, four with sadness, four with surprised, two with fear, and one with disgust. The different facial configurations associated with each emotion word were more than mere variations on a universal core expression—they were distinctive sets of facial movements.²⁹

The typical emotion perception experiment takes one of several forms, summarized in Table 4. Choice-from-array tasks, in which participants are asked to match photos of facial configurations and emotion words (with or without brief stories), have dominated the study of emotion perception since the 1970s. For example, a meta-analysis of emotion-perception studies published in 2002 summarized 87 studies, 83 (95%) of which exclusively used a choice-from-array response method (Elfenbein & Ambady, 2002). This method has been widely criticized for more than 2 decades, however, because it limits the possibility of observing evidence that could disconfirm the common view. Choice-from-array tasks strongly constrain the possible meanings that participants can infer in a facial configuration, such as a photograph of a scowl, because they can choose only the options provided in the experiment (usually a small number of emotion words). In fact, the preponderance of choice-from-array tasks in the scientific study of emotion perception has been identified as one important factor that has helped perpetuate and sustain the common view (Russell, 1994). Other tasks exist for assessing emotion perception (see Table 4), including those that use a free-labeling method, where participants are asked to freely nominate words to label photographs of posed facial configurations, rather than choosing a word from a small set of predefined options. For example, on viewing a scowling configuration, participants might offer responses like “angry,” “sad,” “confused,” “hungry,” or even “wanting to avoid a social interaction.” By allowing participants more freedom in how they infer meaning in a facial configuration, free labeling makes it equally possible to observe evidence that could either support or disconfirm the common view.

Table 4. Pros and Cons of Common Tasks for Measuring Explicit Emotion Perception

Table 4. Pros and Cons of Common Tasks for Measuring Explicit Emotion Perception

View larger version

Recent innovations in measuring emotion perception use computer-generated faces or heads rather than photographs of posed human faces. One method, called reverse correlation, measures participants’ internal model of emotional expressions (i.e., their mental representations of which facial configurations are likely to express instances of emotion) by observing how participants label an avatar head that displays random combinations of animated facial action units (Yu, Garrod, & Schyns, 2012; for a review, see Jack, Crivelli, & Wheatley, 2018; Jack & Schyns, 2017). As each pattern appears on the computer screen (on a given test trial), participants infer its emotional meaning by choosing an emotion label from a set of options (a choice-from-array response). After thousands of trials, researchers estimate the statistical relationship between the dynamic patterns of facial movements and each emotion word (e.g., disgust) to reveal participants’ beliefs about which facial configurations are likely to express different emotion categories.

A second approach using computer-generated faces has participants interact with more fully developed virtual humans (Rickel et al., 2002), also known as embodied conversational agents (Cassell et al., 2000). Software-based virtual humans look like and act like people (for examples, see Fig. 9). They are similar to characters in video games in their surface appearance and are designed to interact face-to-face with humans using the same verbal and nonverbal behaviors that people use to interact with one another. The underlying technologies used to realize virtual humans vary considerably in approach and capability, but most virtual-human models can be programmed to make context-sensitive, dynamic facial actions that would, when used by a person, typically communicate emotional information to other people (see Box 11 in the Supplemental Material for discussion). The majority of the scientific studies with virtual humans were not designed to test whether human participants infer specific emotional meaning in a virtual human’s facial movements, but their design makes them useful for studying when and how facial movements take on meaning as emotional expressions: Unlike all the other ways of assessing emotion perception discussed so far, which ask participants to make explicit inferences about the emotional cause of facial configurations, interactions with virtual humans offers the possibility of learning how a participant implicitly infers emotional meaning during social interactions.

Download

Open in new tab

Download in PowerPoint

Fig. 9. Examples of virtual humans. Virtual humans are software-based artifacts that look like and act like people. (a) The system that used this virtual human is described in Feng, Jeong, Krämer, Miller, and Marsella (2017). (b) This virtual human is reproduced from Zoll, Enz, Aylett, and Paiva (2006). (c) This virtual human is reproduced from Hoyt, C., Blascovich, J., and Swinth, K. (2003). Social inhibition in immersive virtual environments. Presence, 12(2), 183–195, courtesy of The MIT Press. (d) The system that was used to create this virtual human is described in Marsella, Johnson, and LaBore (2000).

Traditionally, in most experiments, if participants reliably infer the hypothesized emotional state from a facial configuration (e.g., inferring anger from a scowling configuration) at levels that are greater than what would be expected by chance, then this is taken as evidence that people “recognize that emotional state in its facial display.” It is more scientifically correct, however, to interpret such observations as evidence that people infer an emotional state (i.e., they consistently make a reverse inference) at greater-than-chance levels. Only when reverse inferences are observed in a reliable and specific way within an experiment can scientists reasonably infer that participants are perceiving an instance of a certain emotion category in a certain facial configuration; technically, the inference holds only for emotion perception as it occurs in the particular situations contained in the experiment (because situations are never randomly sampled). If the emotion-perception evidence is replicated across experiments that sample people from the same culture, then the interpretation can be generalized to emotion perceptions in that culture. Only when the findings generalize across cultures—that is, are replicated across experiments that sample people from different cultures—is it reasonable to conclude that people universally infer a specific emotional state when perceiving a specific facial configuration. These findings can be interpreted as evidence about the reliability and specificity of producing emotional expressions if the coevolution assumption is valid (i.e., that emotional expressions and their perception coevolved as an integrated signaling system; Ekman et al., 1972; Jack et al., 2016; Shariff & Tracy, 2011). The findings can be interpreted as evidence about emotion recognition only if the reverse inference has been verified as valid (i.e., if it can be verified that the person in the photograph is, indeed, in the expected emotional state).

The most recent meta-analysis of emotion-perception studies was published by Elfenbein and Ambady (2002). It statistically summarized 87 experiments in which more than 22,000 participants from more than 20 cultures around the world inferred emotional meaning in facial configurations and other stimuli (e.g., posed vocalizations). The majority of participants were sampled from larger-scale or developed countries, including Argentina, Brazil, Canada, Chile, China, England, Estonia, Ethiopia, France, Germany, Greece, Indonesia, Ireland, Israel, Italy, Japan, Malaysia, Mexico, the Netherlands, Scotland, Singapore, Sweden, Switzerland, Turkey, the United States, Zambia, and various Caribbean countries. The majority of studies (95%) used posed facial configurations; only four studies had participants label spontaneous facial movements, a dramatic example of the challenges facing validity that we discussed earlier. All but four studies used a choice-from-array response method to measure emotion inferences, a good example of the challenges facing hypothesis disconfirmation that we discussed earlier.

The results of the meta-analysis, presented in Figure 10, reveal that perceivers inferred emotions in the facial configurations of Figure 4 in line with the common view, well above chance levels (using the criteria set out by Haidt and Keltner, 1999, presented in Table 2 of the current article). Results provided strong evidence that, when participants viewed posed facial configurations made by people from their own culture, they reliably perceived the expected emotion in those configurations: Scowling facial configurations were perceived as anger expressions, wide-eyed facial configurations were perceived as fear expressions, and so on, for all six emotion categories. Moderate levels of reliability were observed when perceivers were labeling facial configurations posed by people from other cultures; this difference in reliability between same- and cross-culture differences is referred to as an in-group advantage (see Box 12, in the Supplemental Material). The majority of emotion-perception studies did not report whether the hypothesized facial configurations were perceived with any specificity (e.g., how likely was a scowl to be perceived as expressing an instance of emotion categories other than anger, or as an instance of a mental category that is not considered emotional). Without information about specificity, no firm conclusions can be drawn about the emotional meaning of the facial configurations in Figure 4, especially for the translational purpose of inferring someone’s emotional state from their facial comportment in real life.

Download

Open in new tab

Download in PowerPoint

Fig. 10. Emotion-perception findings. Average effect sizes for perceptions of facial configurations in which 95% of the articles summarized used choice-from-array to measure participants’ emotion inferences. Data are from Elfenbein and Ambady (2002). The images presented on the x-axis are for illustrative purposes only and were not necessarily used in the articles summarized in this meta-analysis.

Download

Open in new tab

Download in PowerPoint

Fig. 11. Free labeling of facial configurations across five language groups. Data are from Srinivasan and Martinez (2018). The proportion of times participants offered emotion-category labels (or their synonyms) are reported. The facial configurations presented were chosen by researchers to represent the best match to the hypothetical facial configurations in Figure 4 on the basis of the action units (AUs) present. No configuration discovered in this study exactly matches the AU configurations proposed by Darwin or documented in prior research. According to standard scientific criteria, universal expressions of emotion should elicit agreement rates that are considerably higher than those reported here, generally in the 70% to 90% range, even when methodological constraints are relaxed (Haidt & Keltner, 1999). Specificity data are not available for the Elfenbein and Ambady (2002) meta-analysis.

Nonetheless, most of the studies cited in the Elfenbein and Ambady (2002) meta-analysis interpret their reliability findings alone (i.e., inferring anger from a scowling face, disgust from a nose-wrinkled face, fear from a wide-eyed, gasping face, etc.) as evidence of accurate reverse inferences. Such interpretations may explain why many scientists who study emotion, when surveyed, indicated that they believe compelling evidence exists for the hypothesis that certain emotion categories are each expressed with a unique, universal facial configuration (see Ekman, 2016) and interpret variation in emotional expressions to be caused by cultural learning that modifies what are presumed to be inborn universal expressive patterns (e.g., Cordaro et al., 2018; Ekman, 1972; Elfenbein, 2013). Cultural learning has also been hypothesized to modify how people “decode” facial configurations during emotion perception (Buck, 1984).

Experimental methods that place fewer constraints on participants’ inferences (Table 4) provide considerably less support for the common view of emotional expressions. In the least constrained experimental task, called free labeling, perceivers freely volunteer a word (emotion or otherwise) that they believe best captures the meaning in a facial configuration, rather than choosing from a small set of experimenter-provided options. In urban samples, participants who freely label facial configurations produce the expected emotion labels with weak reliability (when labeling spontaneously produced facial configurations) to moderate reliability (when labeling posed facial configurations). Participants’ responses usually reveal weak specificity when specificity is assessed at all (for examples and discussion, see Russell, 1994; also see Naab & Russell, 2007).

For example, participants in a study by Srinivasan and Martinez (2018) were sampled from multiple countries. They were asked to freely provide emotion words in their native languages (English, Spanish, Mandarin Chinese, Farsi, Arabic, and Russian) to label each of 35 facial configurations that had been cross-culturally identified. Their labels provided evidence of a moderately reliable correspondence between facial configurations and emotion categories, but there was no evidence of specificity (see Fig. 11).³⁰ Multiple facial configurations were associated with the same emotion category label (e.g., 17 different facial configurations were associated with the expression of happiness, five with anger, four with sadness, four with surprise, two with fear, and one with disgust). This many-to-many mapping is inconsistent with the common view that the facial configurations in Figure 4 are universally recognized as expressing the hypothesized emotion category, and they give evidence of variation that is far beyond what is proposed by the basic-emotion view. Some of this variability may come from different cultures and languages, but there is variability even within a single culture and language. Evidence of this many-to-many mapping is also apparent in free-labeling tasks in small-scale, remote samples as well (Gendron, Crivelli, & Barrett, 2018), which we discuss in the next section.

Using a choice-from-array response method with the reverse-correlation method is an inductive way to learn people’s beliefs about which facial configurations express the instances of an emotion category (for reviews, see Jack et al., 2018; Jack & Schyns, 2017). In such studies, participants view thousands of random combinations of AUs that are computer generated on an avatar head and label each one by choosing an emotion word from a set of predefined options. All of the facial configurations labeled with the same emotion word (e.g., anger) are then statistically combined for each participant to estimate that person’s belief about which facial movements express instances of the corresponding emotion category. One recent study using the reverse correlation method with participants from the United Kingdom and China found evidence of variation in the facial movements that were judged to express a single emotion category as well as similarity in the facial movements that were judged to express different categories (Jack et al., 2016). The study first identified groupings of emotion words that are widely discussed in the scientific literature (which, we should note, is dominated by English): 30 English words grouped into eight emotion categories for the sample from the United Kingdom (happy/excited/love, pride, surprise, fear, contempt/disgust, anger, sad, and shame/embarrassed) and 52 Chinese words grouped into 12 categories in the Chinese sample (joyful/excitement, pleasant surprise, great surprise/amazement, shock/alarm, fear, disgust, anger, sad, embarrassment, shame, pride, and despise). The reverse-correlation method revealed 62 separate facial configurations: The same emotion category in a given culture was associated with multiple models of facial movements because synonyms of the same emotion category were associated with distinctive models of facial movements.

Amidst this variability, Jack and colleagues also found that these 62 separate facial configurations could be summarized as four prototypes, which are presented in Figure 8 along with the corresponding emotion words with which they were frequently associated. Each prototype was described with a unique set of affective features (combinations of valence, arousal and dominance). A comparison of the four estimated configurations with the common view presented in Figure 4 and with the basic-emotion hypotheses listed in Table 1 reveals some striking similarities: Configuration 1 in Figure 8 most closely resembles the proposed expression for happiness, Configuration 2 is similar to a combination of the proposed expressions for fear and anger, Configuration 3 most closely resembles the proposed expression for surprise, and Configuration 4 is similar to a combination of the proposed expressions for disgust and anger.³¹ Taken together, these findings suggest that, at the most general level of description, participants’ beliefs about emotional expressions (i.e., their internal models of which facial movements expressed which emotions) were consistent with the common view (indeed, they could be taken to constitute part of the common view); when examined in finer detail with more granularity, however, the findings also give evidence of substantial within-category variation in beliefs about the facial movements that express instances of the same emotion category. This observation suggests that the way the common view is often described in scientific reviews, depicted in the media, and used in many applications does not, in fact, do justice to people’s more varied beliefs about facial expressions of emotion.

Designers typically study how a virtual human’s expressive movements influence an interaction with a human participant. Much of the early research modeling expressive movements in virtual humans focused on endowing them with the facial expressions proposed in Figure 4. A number of studies have endowed virtual humans with blends of these configurations (Arya, DiPaola, & Parush, 2009; Bui, Heylen, Poel, & Nijholt, 2004). Designers are also inspired by people’s beliefs about how emotions are expressed. Actors, for example, have been asked to pose facial configurations that they believe express emotions, which are then processed by graphical and machine-learning algorithms to craft the relation between emotional states and expressive movements (Alexander, Rogers, Lambeth, Chiang, & Debevec, 2009). In another study, human subjects used a specially designed software tool to craft animations of facial movements that they believed express certain mental categories, including emotion categories. Then, other human subjects judged the crafted facial configurations (Ochs, Niewiadomski, & Pelachaud, 2010). Increasingly, data-driven methods are used that place people in emotion-eliciting conditions, capture the facial and body motion, and then synthesize animations from those captured motions (Ding, Prepin, Huang, Pelachaud, & Artie`res, 2014; Niewiadomski et al., 2015; N. Wang, Marsella, & Hawkins, 2008).

In general, studies with virtual humans show nicely how the situational context influences people’s inferences about the meaning of facial movements (de Melo, Carnevale, Read, & Gratch, 2014). For example, in a game that allowed competition and cooperation (Prisoner’s Dilemma, Pruitt & Kimmel, 1977), a virtual human who smiled after making a competitive move evoked more competitive and less cooperative responses from human participants compared with a virtual human using an identical strategy in the game (tit-for-tat) but who smiled after cooperating. Virtual humans who make a verbal comment about a film that is inconsistent with their facial movements, such as saying they enjoyed the film but grimacing, quickly followed by a smile, were perceived as less reliable, trustworthy, and credible (Rehm & Andre, 2005).

The dynamics of the facial actions, including the relative timing, speed, and duration of the individual facial actions, as well as the sequence of facial muscle movements over time, offer information over and above the mere presence or absence of the movements themselves and have an important influence on how human perceivers interpret facial movements (e.g., Ambadar, Cohn, & Reed, 2009; Jack & Schyns, 2017; Keltner, 1995; Krumhuber, Kappas, & Manstead, 2013) and how much they trust a virtual human during a social interaction (Krumhuber, Manstead, Cosker, Marshall, & Rosin, 2009). Research with virtual humans has shown that the dynamics of facial muscle movements are critical for them to be perceived as emotional expressions (Niewiadomski et al., 2015; Ochs et al., 2010). These findings are consistent with research showing that the temporal dynamics carry information about the emotional meaning of facial movements that are made by real humans (e.g., Kamachi et al., 2001; Krumhuber & Kappas, 2005; Sato & Yoshikawa, 2004; for a review, see Krumhuber et al., 2013).³²

Whether people can reliably perceive emotions in the expressive configurations of Figure 4, as predicted by the common view, depends on how participants are asked to report or register their inferences (see Table 3). Hundreds of experiments have asked participants to infer the emotional meaning of posed, exaggerated facial configurations (such as those presented in Figure 4) by choosing a single emotion word from a small number of options offered by scientists, called choice-from-array-tasks. This experimental approach tends to generate moderate to strong evidence that people reliably label scowling facial configurations as angry, frowning facial configurations as sad, and so on for all six emotion categories that anchor the common view. Choice-from-array tasks severely limit the possibility of observing evidence that can disconfirm the common view of emotional expressions, however, because they restrict participants’ options for inferring the psychological meaning of facial configurations by offering them a limited set of emotion labels. (As we discuss below, when people are provided with labels other than angry, sad, afraid, and so on, they routinely choose them; also see Carroll & Russell, 1996; Crivelli et al., 2017). In addition, the specificity of emotion-perception judgments is largely unreported.

Scientists often go further and interpret the better-than-chance reliability findings from these studies as evidence that scowls are expressions of anger, frowns are expressions of sadness, and so on. Such inferences are not sound, however, because most of these studies ask participants to infer emotion from posed, static faces that are likely limited in their validity (i.e., people posing facial configurations such as those depicted in Figure 4 are unlikely to be experiencing the hypothesized emotional state). Furthermore, other ways of assessing emotion perception, such as the reverse-correlation method and free-labeling tasks, find much weaker evidence for reliability of emotion inferences. Instead, they suggest that what people actually infer and believe about facial movements incorporates considerable variability: In short, the common view depicted in many reviews, summaries, the media, and used in numerous applications is not an accurate reflection of what people believe about facial expressions of emotion, when these beliefs are probed in more detail (in a way that makes it possible to observe evidence that could disconfirm the common view). In the next section, we discuss scientific evidence from studies of emotion perception in small-scale remote cultures, which further undermines the common view.

During the period from 1969 to 1975, between five and eight small-scale samples from remote cultures in the South Pacific were studied with choice-from-array tasks; the goal was to investigate whether these participants perceived emotional expressions in facial movements in a manner similar to that of people from the United States and other industrialized countries of the Western world (see Fig. 12a). Our uncertainty in the number of samples stems from reporting inconsistencies in the published record (see note to Table 5). We present the findings here according to how the original authors reported their findings, despite the inconsistencies. Five samples performed choice-from-array tasks, three in which participants chose a photographed facial configuration to match one brief vignette that described each emotion category (Ekman, 1972; Ekman & Friesen, 1971; Sorenson, 1975) and two in which they chose a photograph to match an emotion word (Ekman, Sorenson, & Friesen, 1969). All five samples performed some version of a choice-from-array task that provided strong evidence in support of cross-cultural reliability of emotion perception in small-scale societies. Evidence for specificity was not reported. Until 2008, all claims that anger, sadness, fear, disgust, happiness, and surprise are universally recognized (and therefore are universally expressed) were based largely on three articles (two of them peer reviewed) reporting on four samples (Ekman, 1972; Ekman & Friesen, 1971; Ekman et al., 1969).³³

Download

Open in new tab

Download in PowerPoint

Fig. 12. Map of cross-cultural studies of emotion perception in small-scale societies. People in small-scale societies typically live in groupings of several hundred to several thousand people who maintain autonomy in social, political and economic spheres. (a) Epoch 1 studies, published between 1969 and 1975, were geographically constrained to societies in the South Pacific. Studies that share the same superscript letter may share the same samples. (b) Epoch 2 studies, published between 2008 and 2017, sample from a broader geographic range including Africa and South America and are more diverse in the ecological and social contexts of the societies tested. This type of diversity is a necessary condition for discovering the extent of cultural variation in psychological phenomena (Medin, Ojalehto, Marin, & Bang, 2017). Adapted from Gendron, Crivelli, and Barrett (2018).

Since 2008, 10 verifiably separate experiments observing emotional inferences in small-scale societies have been published or submitted for publication. These studies involve a greater diversity of social and ecological contexts, including sampling five small-scale societies across Africa and the South Pacific (see Fig. 12b) that were tested with a greater diversity of research methods listed in Table 4, including tasks that allow for the possibility of observing cross-cultural variation in emotion perception and therefore the possibility of disconfirming the common view. Six samples registered their emotion inferences using a choice-from-array task, in which participants were given an emotion word and asked to choose the posed facial configuration that best matched it or vice versa (Crivelli, Jarillo, Russell, & Fernández-Dols, 2016; Crivelli, Russell, Jarillo, & Fernández-Dols, 2016; Crivelli et al., 2017, Study 2; Gendron, Hoemann, et al., 2018, Study 2; Tracy & Robins, 2008).

Only one study (Tracy & Robins, 2008) reported that participants selected an emotion word to match the facial configurations similar to those in Figure 4 more reliably than what would be expected by chance, and effects ranged from weak (anger and fear) to strong (happiness) with surprise and disgust falling in the moderate range.³⁴ Information about the specificity of emotion inferences was not reported. A close examination of the evidence from four studies by Crivelli and colleagues suggest weak to moderate levels of reliability for inferring happiness in smiling facial configurations (all four studies), sadness in frowning facial configurations (all four studies), fear in gasping, wide-eyed facial configurations (three studies), anger in scowling facial configurations (two studies), and disgust in nose-wrinkled facial configurations (three studies). A detailed breakdown of findings can be found in Box 13 in the Supplemental Material. None of the studies found specificity for any facial configuration, however, except that smiling was reported as unique to happiness, but that finding was not replicated across samples.³⁵

The final study using a choice-from-array task with people from a small-scale, remote culture is important because it involves the Hadza hunter-gatherers of Tanzania (Gendron, Hoemann, et al., 2018, Study 2).³⁶ The Hadza are a high-value sample for two reasons. First, universal and innate emotional expressions are hypothesized to have evolved to solve the recurring fitness challenges of hunting and gathering in small groups on the African savanna (Pinker, 1997; Shariff & Tracy, 2011; Tooby & Cosmides, 2008); the Hadza offer a rare opportunity to study hunters and foragers who are currently living in an ecosystem that is thought to be similar to that of our Paleolithic ancestors.³⁷ Second, the population is rapidly disappearing (Gibbons, 2018). Before this study, the Hadza had not participated in any studies of emotion perception, although they have been the subject of social cognition research more broadly (H. C. Barrett et al., 2016; Bryant et al., 2016).

After listening to a brief story about a typical instance of anger, disgust, fear, happiness, sadness, and surprise, Hadza participants chose the expected facial configuration more often than chance only when the target and foil could be distinguished by the affective property referred to as valence. The finding that Hadza participants were successfully inferring pleasantness and unpleasantness is consistent with anthropological studies of emotion (Russell, 1991), linguistic studies (Osgood, May, & Miron, 1975), and findings from other recent studies of participants from small-scale societies, such as the Himba (Gendron, Roberson, van der Vyver, & Barrett, 2014a, 2014b) and the Trobriand Islanders (Crivelli, Jarillo, et al., 2016; also see Srinivasan & Martinez, 2018, described in Box 7 in the Supplemental Material); these studies showed that perceivers can reliably infer valence but not arousal in facial configurations. In addition, Hadza participants who had some contact with people from other cultures—they had some formal schooling or could speak Swahili, which is not their native language—were more consistently able to choose the hypothesized facial configuration than were those with no formal schooling who spoke minimal Swahili (for a similar finding with Fore participants in a free-labeling study, see Table 2 in Sorenson, 1975). Of the 27 Hadza participants who had minimal contact with other cultures, only 12 reliably chose the wide-eyed, gasping facial configuration at above chance levels to match the fear story. (Compare this finding with the observation that the hypothesized universal expression for fear—a wide-eyed, gasping facial configuration—is understood as an aggressive, threatening display by Trobriand Islanders; Crivelli, Jarillo, & Fridlund, 2016; Crivelli, Russell, Jarillo, & Fernández-Dols, 2016, 2017).

During the period from 1969 to 1975, between one and three small-scale samples from remote cultures in the South Pacific were studied with free labeling to investigate emotion perception (three samples were reported in Sorenson, 1975; see Table 5 in the current article). From 2008 onward, two additional studies were conducted, one asking participants from the Trobriand Islands to infer emotions in photographs of spontaneous facial configurations (Crivelli et al., 2017, Study 1) and the other asking Hadza participants to infer emotions in photographs of posed facial configurations (Gendron et al., 2018, Study 2). Taken together, these five studies provide little evidence that the facial configurations in Figure 4 are universally judged to specifically express certain emotion categories. The three free-labeling studies reported in Sorenson (1975) produced variable results. The only replicable finding appears to be that participants labeled smiling facial configurations uniquely as happiness in all studies (as the only pleasant emotion category tested). The two newer free-labeling studies both indicated that participants rarely spontaneously labeled facial configurations with the expected emotion labels (or their synonyms) above chance levels. Trobriand Islanders did not label the proposed facial configurations for happiness, sadness, anger, surprise, or disgust with the expected emotion labels (or their synonyms) at above chance levels (although they did label the faces consistently with other words; Crivelli et al., 2017, Study 1). Hadza participants labeled smiling and scowling facial configurations as happiness (44%) and anger (65%), respectively, at above chance levels (Gendron, Hoemann, et al., 2018, Study 2). The word anger was not used to uniquely label scowling facial configurations, however, and it was frequently applied to frowning, nose-wrinkled, and gasping facial configurations.

The more recent studies of people living in small-scale, remote cultures suggest two interesting and noteworthy observations. First, even though people may not routinely infer anger from scowls, sadness from frowns, and so on, they do reliably infer other social meanings for those facial configurations, because facial movements often carry important information about social motives and other psychological features (Crivelli, Jarillo, Russell, & Fernández-Dols, 2016; Crivelli et al., 2017; Rychlowska et al., 2015; Wood, Rychlowska, & Niedenthal, 2016; Yik & Russell, 1999; for a discussion, see Fridlund, 2017; J. Martin et al., 2017). For example, as we mentioned earlier, Trobriand Islanders consistently labeled wide-eyed, gasping faces (the proposed expressive facial configuration for the fear category) as signaling an intent to attack (i.e., a threat; for additional evidence in carvings and masks in a variety of cultures, including Maori, !Kung Bushmen, Himba, and Eipo, see Crivelli, Jarillo, & Fridlund, 2016; Crivelli, Jarillo, Russell, & Fernández-Dols, 2016).

Second, people do not always infer internal psychological states (emotions or otherwise) from facial movements. People who live in non-Western cultural contexts, including Himba and Hadza participants, are more likely to assume that other people’s minds are not accessible to them, a phenomenon called opacity of mind in anthropology (Danziger, 2006; Robbins & Rumsey, 2008). Instead, facial movements are perceived as actions that predict future actions in certain situations (e.g., a wide-eyed, gasping face is labeled as “looking”; Crivelli et al., 2017; Gendron, Hoemann, et al., 2018; Gendron et al., 2014b). Similar observations were unavailable for the earlier studies conducted by Ekman, Friesen, and Sorenson because, according to Sorenson (1975), they directed participants to provide emotion terms. When participants spontaneously offered an action label (e.g., “she is just looking”) or a social evaluation (e.g., “he is ugly,” or “he is stupid”), they were asked to provide an “affect term.” Such findings suggest that there may be profound cultural variation in the type of inferences human perceivers typically make when looking at other human faces in general, an observation that has been raised by a number of anthropologists and historians.

To properly interpret the scientific evidence, it is crucial to consider the constraints placed on participants by the experimental tasks that they are asked to complete, summarized in Table 4. In most urban and in some remote samples, experiments using choice-from-array tasks produce evidence supporting the common view: Participants reliably label scowling facial configurations as angry, smiling facial configurations as happy, and so on. (We do not yet know whether perceivers are uniquely labeling each facial configuration as a specific emotion because most studies do not report that information.)

It has been known for almost a century that choice-from-array tasks help participants obtain a level of reliability in their emotion perceptions that is not routinely seen in studies using methods that allow participants to respond more freely, and this is one reason they were chosen for use in the first place (for a discussion, see Gendron & Barrett, 2009, 2017; Russell, 1994; Widen & Russell, 2013). When participants are offered words for happiness, fear, surprise, anger, sadness, and disgust to register their inferences for a scowling facial configuration, they are prevented from judging a face as expressing other emotion categories (e.g., confusion or embarrassment), nonemotional mental states (e.g., a social motive, such as rejection or avoidance), or physical events (e.g., pain, illness, or gas), thus inflating reliability rates within the task. When people are provided with other options, they routinely choose them. For example, participants label scowling faces as “determined” or “puzzled,” wide-eyed faces as “hopeful,” and gasping faces as “pained” when they are provided with stories about those emotions rather than with stories of anger, surprise, and fear (Carroll & Russell, 1996; also see Crivelli et al., 2017). The problem is not with the choice-from-array task per se—it is more with failing to consider alternative explanations for the observations in an experiment and therefore drawing unwarranted conclusions from the data.

Choice-from-array tasks may do more than just limit response options, making it difficult to disconfirm common beliefs. The emotion words provided during the task may actually encourage people to see anger in scowls, sadness in pouts, and so on, or to learn associations between a word (e.g., anger) and a facial configuration (e.g., a scowl) during the experiment (e.g., Gendron, Roberson, & Barrett, 2015; Hoemann et al., in press). The potency of words is discussed in Box 14, in the Supplemental Material.

The pattern of findings from the studies conducted with remote samples replicates and underscores the pattern observed in samples of participants from larger, more urban cultural contexts: Asking perceivers to infer an emotion by matching a facial configuration to an emotion word selected from a small array of options, or telling participants a brief story about a typical instance of an emotion category and asking them to pick a facial configuration from an array of two or three photos, generally inflates agreement rates, producing evidence that is more likely to support the hypothesis of reliable emotion perception compared with data coming from less constrained response methods, such as free labeling (see Table 3). This is particularly true for studies that include only one pleasant emotion category (i.e., happiness) so that all foils differ from the target in valence. The robust reliability and specificity for inferring happiness from smiling observed in these studies may be the result of participants classifying valence rather than classifying emotion categories per se. Studies that use less constrained tasks that are designed to more freely discover how people perceive emotion instead yield evidence that generally fails to find support for the common view. Less constrained studies suggest that perceivers infer more than one emotion category from the same facial configuration, infer the same emotion category in a variety of different configurations, and often disagree about the set of emotion categories that they infer. Cultural variation in emotion perception is consistent with the variation we observed in studies of expression production (again, see Table 3) and is even consistent with the research on face perception, which itself is determined by experience and cultural factors (Caldara, 2017).

Children grow up in emotionally rich social environments, making it difficult to run experiments that are capable of testing the common view of emotion perception while also taking into account the possible roles for learning and social experience. Nonetheless, several themes have emerged in the scientific literature, all of which suggest a clear role for learning and context in children’s developing emotion-perception capacities.

One hypothesis that continues to be strongly supported by experiments is that children’s capacity to infer emotional meaning in facial movements depends on context (the conditions surrounding the face that may convey information about a face’s meaning). For example, emotion-concept learning, as a potent source of internal context, shapes emotion-perception capacity (discussed in Boxes 10 and 16 in the Supplemental Material). There are also developmental changes in how people use context to shape their emotional inferences about facial movements. Children as young as 19 months old can detect facial movements that are emotionally incongruent with a context (Walle & Campos, 2014). For example, when presented with adult facial configurations that are placed on bodies posing an emotional context (e.g., a scowling facial configuration placed on a body holding a soiled diaper), children (ages 4, 8, and 12 years) moved their eyes back and forth between faces and bodies when deciding how to label the emotional meaning of the faces, whereas adult participants directed their gaze (and overt visual attention) to the face alone, judging its emotional meaning in a way that was independent of the bodily context (Leitzke & Pollak, 2016). The youngest children were equally likely to base their labeling of the scene on face or context. The results of this experiment suggest that younger children devote greater attention to contextual information and actively cross-reference facial and contextual cues, presumably to better learn about and understand the emotional meaning those cues.³⁸

Another important source of context that shapes the development of emotion perception in children involves the broader environment in which children grow. Children who grow up in neglectful or abusive environments in which their emotional interactions with caregivers are highly atypical have a different developmental trajectory than do those growing up in more consistently nurturing environments (Bick & Nelson, 2016; Pollak, 2015). Parents from these high-risk families produce unclear or context-inconsistent expressions of emotion (Shackman et al., 2010). Neglected children, who often do not receive sufficient social feedback, show delays in perceiving emotions in the ways that adults do (Camras, Perlman, Fries, & Pollak, 2006; Pollak et al., 2000), whereas children who are physically abused learn to preferentially attend to and identify facial movements that are associated with threat, such as a scowling facial configuration (Briggs-Gowan et al., 2015; Cicchetti & Curtis, 2005; da Silva Ferreira, Crippa, & de Lima Osório, 2014; Pollak, Vardi, Bechner, & Curtin, 2005; Shackman & Pollak, 2014; Shackman, Shackman, & Pollak, 2007). Abused children require less perceptual information to infer anger in a scowling configuration (Pollak & Sinha, 2002) and more reliably track the trajectory of facial muscle activations that signal threat (Pollak, Messner, Kistler, & Cohn, 2009). Children raised in physically abusive environments also more readily infer anger and threat in ambiguous facial configurations (Pollak & Kistler, 2002) and then require greater effortful control to disengage their attention from signs of threat (Pollak & Tolley-Schell, 2003) compared with children who have not been maltreated. This close attention to scowling faces with knitted eyebrows shapes how abused children understand what facial movements mean. For example, one study found that 5-year-old abused children tended to believe that almost any kind of interpersonal situation could result in an adult becoming angry; by contrast, most nonabused children understand that anger is likely to be particular to interpersonal circumstances (Perlman, Kalish, & Pollak, 2008).

By 3 years of age, North American children not only start to show reliability in their emotion perceptions but also begin to show evidence of specificity. They understand that facial movements do not necessarily map on to emotional states, and how someone really feels can be faked or masked. Moreover, they know what facial movements are expected in a particular context and try to produce them despite their feelings. For example, the “disappointing gift” experiments developed by psychologist Pamela Cole and her colleagues demonstrate this well. In one study, preschool-age children were told they would be rewarded with a gift after they completed a task. Later, children received a beautifully wrapped package that contained a disappointing item, such as a broken pair of cheap sunglasses. When facing a smiling unfamiliar adult who had presented them with a gift, children forced themselves to smile (lip corner pull, cheek raise, and brow raise) and to thank the experimenter. Yet, although the children were smiling, they often kept their eyes focused down, slumped their shoulders, and made negative statements about the object, indicating that they did not, in fact, feel positive about the situation (Cole, 1986). Moreover, there was no difference in the behavioral responses of visually impaired children when receiving a disappointing gift (Cole, Jenkins, & Shott, 1989). Studies like this one provide a more implicit way of assessing children’s knowledge about emotion perception (i.e., it illustrates the inferences that children expect others to make from their own facial movements).

It is possible that the frequency and type of facial input that people encounter influences their emotion categorizations. To test whether the statistical distribution of emotion input would influence how people construed boundaries between emotion categories, Plate, Wood, Woodard, and Pollak (2018) manipulated the frequency of this information to perceivers. Participants were asked to categorize facial morphs (from neutral to scowling) as being either “calm” or “upset.” A third of participants saw more scowling faces, a third saw more neutral faces, and the others saw faces that were equally distributed across scowling and neutral. Both school-age children and adults adjusted their emotion categories based on the frequency of the input they encountered. Those exposed to more scowling faces increased their threshold for categorizing a face as upset (therefore narrowing their category of “anger”). Those exposed to more calm faces decreased their threshold for categorizing a face as angry. These data are consistent with the idea that the frequency or commonness of a facial configuration in an observer’s environment influences his or her conception of an emotion (Levari et al., 2018; Oakes & Ellis, 2013), as well as the more general findings that expertise with faces influences identity perception (Beale & Keil, 1995; Jenkins, White, Monfort, & Burton, 2011; McKone, Martini & Nakayama, 2001; Viviani, Binda & Bosato, 2007; for a discussion of how familiarity is important for face perception, see Young & Burton, 2017). As a result, individual differences in emotion perception may be influenced by early experience that differs according to emotional input, reflecting the malleability of these categories.

There is currently no clear evidence to support the hypothesis that infants and young children reliably and specifically infer emotion in the proposed expressive configurations for the anger, disgust, fear, happiness, sadness, and surprise categories (findings summarized in Table 3). A more plausible interpretation of the existing evidence is that young infants infer affective meaning, such as valence and arousal, from facial configurations. Data from infants and young children obtained using a variety of methods further suggest that emotion-perception abilities emerge and are shaped through learning in a social environment. These findings are consistent with the idea that the human face plays an important and privileged role to communicate importance or salience. But it is not clear that the expressive configurations proposed for specific emotion categories are similarly privileged in this way.