Do You Know What Fake News Is? An Exploration of and Intervention to Increase Youth’s Fake News Literacy

Youth should be correctly informed about what is happening in the world, but research on empowering people to identify fake news rarely targets youth. To take the first steps in increasing their fake news literacy, this study (N = 298) qualitatively looks into youth’s (10–12 years old) fake news knowledge and quantitatively tests a fake news e-learning intervention (i.e., an online lesson in which youth learn about fake news and possible solutions). Our investigation of youth’s fake news knowledge showed that, before participating in the intervention, youth already had some knowledge of what fake news is and were aware of its problem. The intervention aimed to increase youth’s knowledge, awareness, and self-efficacy toward fake news. Although it did not increase youth’s knowledge or awareness of fake news, it successfully stimulated their self-efficacy in identifying fake news.

Since former President Trump regularly accused the media of spreading "fake news," it has been a much-debated topic (BBC, 2018).Fake news, defined as the deliberate spread of incorrect information to mislead people, is nothing new (Aliaksandrau, 2017).However, with the rise of the internet and social media it has become much easier to spread such information (Allcott et al., 2019;Patterson, 2013).Youth between 10 and 12 years consume most news online (Ofcom, 2020;Smahel et al., 2020), which makes them vulnerable to encountering inaccurate information.Various initiatives are taken worldwide to combat the spread of fake news (Amazeen & Bucy, 2019;European Commission, 2020;Haciyakupoglu et al., 2018).These initiatives range from automated verification of fake news using machine learning techniques (Vosoughi et al., 2017) to making digital media literacy part of school curricula (Wardle & Derakhshan, 2017).Strikingly, research on (fake) news literacy rarely targets youth (Marchi, 2012).
The scarce research on fake news and youth shows that youth lack confidence in their information navigation skills (Notley & Dezuanni, 2019;Smahel et al., 2020).In particular, research has found that youth have difficulty distinguishing between real and fake information, demonstrated by their inability to identify spoof websites as fake (Dumitru, 2020;Loos et al., 2018;Pilgrim et al., 2019).Thus, it is essential to teach them to scrutinize information and distinguish between real and fake news (Dumitru, 2020;Loos et al., 2018).A possible solution to youth's difficulties when dealing with fake news can be found in the realm of media literacy, which aims to empower individuals in their engagements with media.However, to stimulate youth's fake news literacy, we should know more about their definitions and evaluations of fake news.Therefore, the first objective of this study is to explore youth's knowledge and evaluation of fake news.The second objective is to test whether youth's fake news literacy can be improved by a brief e-learning fake news literacy intervention.

News Media Literacy and the Specifics of Fake News
In this study, we build on literature from the field of media literacy.Critical media literacy aims to empower individuals to critically read and produce media messages to participate actively in democracy (Kellner & Share, 2019).It posits that by teaching individuals how to engage with media content critically, they can adequately understand, interact with, and use it.Over time, there has been growing attention to media literacy and interventions to stimulate media literacy (Jeong et al., 2012).News literacy, a more specific form of media literacy, focuses on accessing, analyzing, and evaluating news, and the knowledge and skills needed to do so (e.g., Ashley et al., 2013;Tully et al., 2022).
The identification of fake news is a media literate behavior (e.g., Kellner & Share, 2019;Vraga et al., 2021).Fake news is a particular kind of content which, over time, has held various meanings (from "B-quality" news to satire; Costera Meijer, 2007;Gelfert, 2018;Marchi, 2012;Scott & Eddy, 2017;Tandoc et al., 2018).Fake news, also known as disinformation, lacks authenticity, is misleading, and, importantly and deceivingly, looks like real news (e.g., Allcott & Gentzkow, 2017;Gelfert, 2018;Shu et al., 2017;Tandoc et al., 2018).The dissemination of fake news is often motivated by financial or ideological motivations, for example, to gain advertising revenue by clicks or to advance a particular political candidate (Allcott & Gentzkow, 2017).Because fake news is such a specific and deceiving kind of media content, the current study zooms in on youth's fake news literacy.
Unfortunately, the precursors of media literate behaviors can differ over different behaviors (Vraga et al., 2021).Existing media literacy interventions thus cannot simply be adjusted to target another media literate behavior.New developments require new literacies (Kellner & Share, 2019).Besides, the effectiveness of interventions that help individuals recognize fake news is still mixed (Vraga et al., 2021).Therefore, fake news literacy, empowering individuals to identify fake news, warrants further investigation.

Exploring Youth's Fake News Knowledge
The difficulties that youth have in distinguishing real from fake news (e.g., Loos et al., 2018), could be because they lack news production and fake news knowledge (Maksl et al., 2015;Pilgrim et al., 2019;Vraga & Tully, 2019).Several studies suggest that news production knowledge is a consistent predictor of relevant outcomes, or at least prerequisite for news literate behaviors, including identifying fake news and rejecting conspiracy theories (Amazeen & Bucy, 2019;Craft et al., 2017).Although a lack of knowledge has been mentioned as a possible explanation for the inability to detect fake news (e.g., Pilgrim et al., 2019), not much is known about youth's knowledge of fake news.Research on early adolescents (12-15 years old) shows that, although they do not perform well in evaluating news, they are aware of the presence of fake news and its consequences (e.g., Craft et al., 2016;Marchi, 2012;Tamboer et al., 2022).They know examples of fake news and know the motives for disseminating disinformation (Tamboer et al., 2022).To identify fake news, news consumers should, in the first place, be aware of the meaning of the concept.Exploring how youth define fake news is thus a first step toward empowering them to become more news literate.Therefore, the first research question is: RQ1: What do youth (10-12-year-olds) know and think about fake news?

Toward an Intervention to Increase Youth's Critical Analysis of Fake News
Going beyond describing youth's fake news knowledge, this study aims to test whether youth's fake news literacy can be improved via a brief e-learning intervention.Knowledge of media content, industries, and effects enables individuals to make better decisions and seek credible information (Potter, 2004).However, solely possessing knowledge of news media does not ultimately lead to more (news) media literate behavior (Jeong et al., 2012).Researchers suggest that (fake) news literacy interventions should encourage people to apply media literacy knowledge and skills in their daily lives (Tully et al., 2019;Vraga & Tully, 2015).To do so, awareness of the need for news literacy and self-efficacy-confidence in one's news literacy skills-are crucial (Chu & Lee, 2014;Rozendaal et al., 2011;Torres et al., 2018;Vraga et al., 2021).In older youth, the awareness of the need for fake news literacy and self-efficacy are quite low (McGrew et al., 2018;Wineburg et al., 2016).We would expect this to be comparable in youth between 10 and 12.To help youth become more fake news literate, interventions should thus target fake news awareness (Torres et al., 2018;Vraga et al., 2015).Youth need to be aware of the problem (fake news) and its negative consequences.Moreover, youth need to feel confident and able to use this knowledge to empower themselves against possible disinformation (Tully et al., 2019).
Some researchers and practitioners have developed lesson plans and games to increase youth's fake news literacy (e.g., Campos, 2018;Moore, 2013).However, these initiatives do not offer clear insights into their theoretical foundations and effectiveness, because they mainly focus on developing interventions (e.g., Campos, 2018) or pedagogies in lesson plans (e.g., Moore, 2013).Furthermore, most (news) media literacy interventions mainly focus on targeting news production knowledge (Jeong et al., 2012).Therefore, this study provides insight into an intervention to increase the critical analysis of fake news among youth, targeting their knowledge, awareness, and selfefficacy.We hypothesize the following: Youth who are exposed to the fake news intervention will increase their knowledge (H1), awareness (H2), and self-efficacy toward fake news (H3) over time compared to those who are not exposed to the fake news intervention.

Method
To explore RQ1, we first conduct a qualitative analysis of youth's written answers to the question: "What do you know about fake news?".To test our hypotheses, an online experiment was conducted to investigate the potential of an e-module to increase youth's knowledge, awareness, and self-efficacy for fake news.

Participants and Procedure
The study is part of a larger research project on fake news literacy and cyberbullying.The project has been reviewed independently by the Ethics Committee of the Faculty of Social Sciences at Radboud University (ECSW-2018-162).For this project, five primary schools across the Netherlands participated.In total, youth from 16 classes (grades 5-6) took part.After approval from the head of each school, information letters were sent to all parents and/ or caretakers of the youth.Of all the parents, 52 (13.8%) did not give active consent.Their children were therefore not included.
At the start of the data collection, youth with parental consent could decide whether they wanted to participate.Eight youth did not want to participate.Only youth that participated in all parts of the project were included in the final sample, resulting in a sample size of 298 youth (54.0%girls; M age = 10.92,SD age = 0.70, range 10-12 years old).
The data collection took place during school hours.After introducing the project and the consent procedure, participants were directed to an online questionnaire.All participating classes within one school were assigned to either the online fake news literacy course (n = 129; 61.2% girls, M age = 10.90, SD age = 0.71) or the control course (n = 169; 48.5% girls, M age = 10.93,SD age = 0.70).Power analysis using G*Power (Faul et al., 2007) showed that a total of 294 youth was sufficient to detect small effects (α = .05,Cohen's f = 0.19, 90% power, numerator df = 1, 2 groups, 1 covariate).

Procedure
Pre-Exposure Measurement (T0).Youth started by filling out a questionnaire that included general questions (age, sex, internet use, and news consumption via television and the internet), one open-ended knowledge question about fake news, and several questions about youth's awareness of and self-efficacy concerning detecting fake news (see Figure 1).This questionnaire also included questions about cyberbullying (Vlaanderen et al., 2020).After completing the first questionnaire, participants were automatically directed toward either the online fake news module or the anti-cyberbullying module (the control condition).Both modules were online e-learnings in which participants had to read text, complete tasks, answer questions, and view three videos.The total duration of the intervention was approximately 35 minutes.Immediately after participating in the e-learning intervention, participants filled out intervention evaluation questions.
Post-Exposure Measurement (T1).Three weeks after the e-learning, participants again completed an online questionnaire.This questionnaire was identical to the first and took approximately 15 minutes.After the last question, participants were thanked for their participation and received a certificate.Schools and parents were debriefed on the study results afterward.

Materials
The main goal of the fake news e-learning was to provide youth with fake news knowledge, awareness, and self-efficacy.The anti-cyberbullying and the fake news e-learnings were as similar as possible in terms of colors, navigation, number of questions, duration of the intervention, number of pages, and number and duration of videos.
To increase youth's fake news knowledge, the definition of fake news-(the lack of) authenticity, intent, and imitation-was introduced in the e-learning.The e-learning covered an explanation of fake news, reasons to make and distribute fake news, and participants did quizzes to match news characteristics to either real or fake news, and to distinguish real from fake news.To increase youth's awareness of fake news, while at the same time increasing their knowledge, participants were given information on the different parties involved in creating and spreading fake news.Furthermore, youth were made aware of the negative consequences of fake news by watching a video on how it can create conflicts between different groups.Afterwards, youth were asked to write down how they thought about people becoming frightened or sad because of fake news.
Participants watched a video in which a journalist explained how they could recognize fake news to increase their self-efficacy in detecting fake news.After this video, youth were asked to name as many ways to identify fake news as they remembered from the video.Finally, they saw a video explaining where fake news originated from and what they can do to stop the spread of fake news (i.e., be critical of news and stop spreading fake news).

Measures
Knowledge of Fake News.At T0, one open-ended question was asked to assess what youth already knew about fake news: "You may have heard of fake news.We would like to know what you already know about fake news.
Please type your answer below.You do not have to think too long about it; type what first comes to mind.So: What do you know about fake news?"All 298 youth answered with at least one word (e.g., "nothing," "Russia," or "fake"), no answers were left blank.In general, youth answered in one or two sentences (longest = 134 words).
Quantification of Knowledge of Fake News.To test the hypotheses, we quantified the answers to the knowledge question, using the definition of fake news.Separate from the qualitative coding for RQ1, we coded whether youth mentioned the three elements of the definition of fake news in this part, receiving one point for each element.Therefore, youth's scores on knowledge could range from 0 (none of the key features were mentioned) to 3 (all three features were correctly mentioned).Two coders coded ten percent of all items.The intercoder reliability was moderate to strong for the pre-(Krippendorff's α = .74(95% CI [0.52, 0.96]), p < .001)and post-exposure (Krippendorff's α = .80(95% CI [0.60, 1.00]), p < .001)measure.Therefore, the variable knowledge was constructed for the pre-(M = 0.56, SD = 0.56) and post-exposure measurement (M = 0.63, SD = 0.60).
Awareness of Fake News.To assess youth's awareness of fake news, four items assessing how youth thought about fake news were developed.At T0 and T1, youth were asked to rate on a VAS-scale-ranging from I certainly don't think so (0) to I certainly think so (100)-how much they agreed with items such as: "I think fake news is a big problem."A factor analysis including the four items yielded one factor (for the pre-and post-measurement).The Cronbach's α at the pre-measurement (α = .81)and the post-measure (α = .86)were sufficient.Therefore, we calculated a mean score for every child on the statements to construct the variable awareness of fake news for pre-(M = 80.45, SD = 20.35) and post-exposure measurement (M = 82.10,SD = 21.53).
Self-Efficacy to Recognize Fake News.To assess youth's self-efficacy in recognizing fake news at T0 and T1, we developed a five-item scale based on Bandura's guide for constructing self-efficacy scales (Bandura, 2006).The items included: "I can recognize fake news."Youth answered these items on a VAS-scale ranging from I certainly cannot (0) to I certainly can (100).For the pre-and post-measurement the factor analysis yielded one factor.The Cronbach's α of the pre-(α = .68)and the post-measurement (α = .72)was sufficient.Therefore, the items' mean scores for both the pre-(M = 46.09,SD = 19.67)and post-measurement (M = 52.53,SD = 19.95) were calculated.
Evaluation of the Intervention.To evaluate the intervention, we asked the participants some evaluation questions.The items included the extent to which they believed it was interesting, fun, hard, important, useless, and informational, on a VAS-scale from totally disagree (0) to totally agree (100).

Analysis Procedure
Youth's Knowledge.Youth's descriptions of what they know about fake news were analyzed using MAXQDA, a qualitative research tool used for coding (VERBI Software, 2019).We employed a thematic analysis to analyze patterns in the data (Boeije, 2010;Braun & Clarke, 2006).Coding happened in two rounds: first open, followed by axial coding (Braun & Clarke, 2013).Coding was done in close collaboration, and frequent discussions between researchers took place, contributing to the reliability of the codings (Braun & Clarke, 2013).We ordered the results in thematic maps, including counts of how often something was mentioned in parentheses (see Figure 2).Intervention Effectiveness.A randomization check showed that the experimental and control condition did not differ concerning age (t(296) = −0.362,p = .718)and grade (χ 2 (1) = 0.495, p = .482).However, the two conditions did differ in sex (χ 2 (1) = 4.765, p = .029).Therefore, sex was added as a control variable in the analyses.To test the hypotheses, 2 (within-subjects factor Figure 2. Thematic map of youth's conceptualizations of fake news, awareness of the problem of fake news and solutions to this problem, and their evaluations of fake news.

Note.
Numbers within parentheses indicate answer frequency.
time: pre-exposure and post-exposure) × 2 (between-subjects factor condition: experimental or control) mixed-model ANCOVAs were performed, one for each dependent variable (knowledge, awareness, and self-efficacy) and all controlled for sex.

Youth's Knowledge of Fake News
We made a thematic map (see Figure 2), reflecting the concepts related to fake news: youth's conceptualization of fake news, their awareness of the problem and potential solutions to fake news, and their evaluations of fake news.Some participants did not or not only write down what they knew about fake news, but came up with an example of fake news (n = 37).Some of these examples correctly referred to fake news (e.g., Trump claims that news about climate change is fake).Still, most often referred to fake messages that are not necessarily news-related (e.g., examples of phishing emails or people claiming that ghosts or aliens exist).Also, some participants (n = 28) explicitly mentioned that they did not know anything or that much about fake news, indicating that about 10% of the sample perceived themselves as not knowledgeable about fake news.
First, the qualitative analysis gave insight into youth's conceptualizations of fake news.As seen in Figure 2, youth's answers could be divided into what fake news is, who spreads it, where it is spread, how it is made, and why it is spread.According to our participants, fake news is "fake," "made-up," or contains "lies," it imitates real news, and is spread by people, both offline and online, but primarily online.These conceptualizations align with the first part of the definition of fake news: that it is verifiably false.
Youth mentioned that fake news producers often use Photoshop and spread it via manipulated websites.Some youth described the goal with which fake news is spread, of which some emphasized the intent to mislead people (the second part of the definition of fake news).Overall, the motives mentioned range from scaring people, to commercial motives, and to deceiving and influencing people.Finally, some youth said that fake news imitates real news, corresponding to the third part of the definition of fake news (imitation).
Second, the answers gave insight into youth's awareness of the problem of fake news and solutions to fake news (see Figure 2).According to our participants, fake news is a problem because it occurs often, because it is incorrect information that is hard to distinguish from correct information, because it is dangerous and illegal, but mainly because it affects people who might act on it.Solutions to fake news can be internal, something youth can do, or external, by the government or the people that spread fake news.Youth mention that they can use trustworthy sources (such as the youth's news), ask grown-ups, and avoid clicking on fake news, but mainly that they should check what they read.Regarding external solutions, youth mention that it should "just stop" and that fines against fake news and lessons to increase news literacy are necessary.
Third, the open answers gave insight into youth's evaluations of fake news.Most youth evaluated fake news negatively (see Figure 2), calling it, for example, "pathetic" and "childish."However, one child also mentioned that fake news is not too bad, and two even called it "fun to watch."

Evaluation of the Intervention
Descriptive statistics show that youth who received the fake news literacy intervention thought the intervention was interesting (M = 87.75,SD = 21.05),fun (M = 84.23,SD = 22.95), and important (M = 89.09,SD = 20.30).Furthermore, youth reported they had learned from the intervention (M = 83.27,SD = 23.05) and that it taught them some things they did not know already (M = 55.79,SD = 32.81).They did not find the intervention useless (M = 6.79,SD = 20.00) or difficult (M = 23.68,SD = 28.02).

Intervention Effectiveness
The first hypothesis predicted that youth exposed to the fake news literacy intervention would show an increase in their knowledge of fake news compared to the control intervention.There was no significant main effect of condition, F(1, 295) = 0.227, p = .634(for descriptives, see Table 1).Also, there was no significant main effect of time, F(1, 295) = 1.247, p = .265.Furthermore, we did not find a significant interaction effect of condition and time, F(1, 295) = 0.072, p = .789.Therefore, the first hypothesis was not supported.
Hypothesis 2 predicted that after exposure to the fake news literacy intervention, youth would have higher awareness levels than youth in the control intervention.There were no significant main effects for condition (F(1, 295) = 0.010, p = .920)and time (F(1, 295) = 0.000, p = .997).However, there was a significant interaction effect between condition and time (F(1, 295) = 3.997, p = .046,η 2 p = .013)on awareness.Post-hoc contrast comparisons showed a significant difference in awareness over time for the control condition (M pre-test = 79.68,SE = 1.57;M post-test = 83.07,SE = 1.66) but not for the intervention condition (M pre-test = 81.47,SE = 1.80;M post-test = 80.82, SE = 1.90).This means that youth in the control intervention reported a significant increase in their fake news awareness compared to those exposed to the fake news literacy intervention.However, the increase-and effect sizewas small.The second hypothesis was thus not supported.
The third hypothesis predicted that youth exposed to the fake news literacy intervention would increase their fake news self-efficacy compared to youth in the control intervention.There was a significant main effect of condition on self-efficacy (F(1, 295) = 7.510, p < .001,η 2 p = .025),but not of time (F(1, 295) = 0.099, p = .753).More importantly, there was a significant interaction effect between condition and time (F(1, 295) = 4.717, p = .031,η 2 p = .016)on self-efficacy (see Figure 3).Post-hoc contrast comparisons showed a significant increase in self-efficacy over time for the intervention

Discussion
This study explored youth's knowledge regarding fake news and tested whether a brief e-learning intervention can improve youth's critical analysis of (fake) news.The study showed that youth have some knowledge of fake news and awareness of its problems, although they only rarely discussed all aspects of the definition of fake news.Furthermore, we developed and tested a fake news literacy intervention to increase youth's knowledge, awareness, and self-efficacy toward fake news.The intervention was not able to increase youth's knowledge or awareness of fake news, but it successfully stimulated youth's fake news self-efficacy.This study contributed to our understanding of youth's knowledge and evaluations of fake news.The qualitative investigation showed that youth between 10 and 12 years old, like early adolescents (e.g., Craft et al., 2016;Marchi, 2012;Tamboer et al., 2022), have some knowledge of fake news.They show knowledge of the "who, what, where, why, and how" of fake news, its problems, and possible solutions, and mostly negatively evaluate fake news.The mentioned solutions include further checking the news and looking for trustworthy sources, which are strategies that are often taught to people to stimulate fake news detection (e.g., Pilgrim et al., 2019).However, these were not mentioned that often, and most youth only mentioned one of the three components of the definition of fake news (the lack of authenticity, intent, and imitation).Although they were not explicitly asked to mention three components, this does signal room for improvement.For example, youth often did not mention why people would spread fake news.On the one hand, asking youth to be knowledgeable about the specifics of fake news might be somewhat ambitious.On the other hand, if youth are insufficiently knowledgeable about fake news, this could lead to less of a need to critically engage with news, possibly explaining why youth often do not detect it (Dumitru, 2020;Loos et al., 2018;Pilgrim et al., 2019).
We developed and tested a fake news literacy intervention for youth.The e-learning intervention did not yet effectively stimulate youth's fake news knowledge and awareness.This might be because the knowledge measure focused on specific knowledge based on the definition of fake news.Youth may have learned other things, such as solutions to fake news, that were not measured.Surprisingly, not the participants in the experimental condition, but those in the control condition showed increased fake news awareness.Although the difference was small, it may be that the cyberbullying control condition more strongly affected general awareness of the need to prevent undesirable effects than the fake news condition did.Because cyberbullying is also defined as negative intentional behavior (Menesini et al., 2012), choosing this control group may have decreased the effects of the fake news intervention.However, youth's awareness of fake news was already quite high in both conditions.Future interventions could build on their awareness of fake news and emphasize the need to recognize and prevent fake news.
The increase in self-efficacy to detect fake news is an important finding, because previous studies showed that youth are not confident identifying fake news (Notley & Dezuanni, 2019;Smahel et al., 2020).Furthermore, it adds to the literature that a brief fake news literacy intervention can increase self-efficacy.Future research could build on these findings to further increase youth's self-efficacy, for example, by incorporating more practice or repetition.Another strategy is to include real-life assignments in which youth check actual news items.As such, interventions might encourage youth to apply their fake news knowledge and skills in daily life, which has been mentioned as a general challenge for news literacy interventions (e.g., Tully et al., 2019;Vraga & Tully, 2015).Going beyond knowledge, awareness, and selfefficacy, future research could look at how interventions can increase other influential factors in (fake) news literacy, such as social norms or motivation and fake news identification (e.g., Tamboer et al., 2022;Vraga et al., 2021).
To our knowledge, this study is one of the first to investigate youth's fake news knowledge and to develop and test a fake news intervention targeting knowledge, awareness, and self-efficacy.The intervention was positively evaluated by youth.Nevertheless, this study has limitations.First, we did not measure youth's actual fake news detection.Based on the literature, knowledge, awareness, and self-efficacy are essential precursors of actual news literate behavior (Jeong et al., 2012;Rozendaal et al., 2011;Vraga et al., 2021).It would be interesting to test whether increased knowledge, awareness, and self-efficacy lead to more fake news identification.Future research could, for example, use tasks like fake news detection on spoof websites (e.g., Dumitru, 2020;Loos et al., 2018;Pilgrim et al., 2019) to measure if youth perform better in detecting fake news after an intervention.Second, future research should incorporate youth's views on fake news and interventions more than we did in this study to develop more effective interventions.
This study is a first step in effectively empowering youth in their engagements with fake news.Fake news is here to stay.One way to combat fake news is to educate youth on critically evaluating news.Based on the literature, we expect that news users with greater (news) literacy skills can better recognize and combat fake news online (Jang & Kim, 2018;Vraga et al., 2021).However, developing fitting and effective interventions, especially for youth, is difficult (e.g., Tamboer et al., 2022;Vraga et al., 2021).This study adds context to media literacy research, showing that youth already display a noteworthy amount of awareness regarding fake news.However, their knowledge and self-efficacy to detect fake news can be improved.Furthermore, this study provides insights into ways to improve youth's self-efficacy skills by applying a brief and fun e-learning intervention.Future research and initiatives should build on these insights so that, hopefully in the near future, youth are better prepared to combat fake news.

Figure 3 .
Figure 3. Interaction effect of condition and time on self-efficacy to recognize fake news.