Evaluating the Impact of Aevidum on Mental Health Knowledge, Attitudes, and Help-Seeking Behaviors in High School Students: A Mixed-Methods Study

Purpose To compare Aevidum’s school mental health curriculum vs the curriculum plus Aevidum clubs in a mixed-methods study including pre/post surveys, a randomized clinical trial, and qualitative interviews. Design Concurrent mixed-methods: Aim 1) pre-post surveys evaluated curriculum only vs curriculum plus club schools separately regarding changes in knowledge, help-seeking, and school culture; Aim 2) randomized clinical trial compared curriculum only to curriculum plus club schools; Aim 3) qualitative school staff interviews enhanced understanding of school culture changes. Setting Curriculum delivered to 9th graders at ten Pennsylvania high schools; 5 schools randomized to start clubs. Subjects Students (surveys), staff (interviews). Intervention Aevidum curriculum plus/minus club. Measures Aim 1, mixed effects linear and logistic regression models for longitudinal data were used to analyze survey items at each time point. Aim 2, the same regression models were used, except models included a fixed-effect for group and group by time interaction effect. Aim 3, interviews were transcribed; a codebook was developed followed by thematic analysis. Results Pre-survey 2557 respondents; 49% female, 86% non-Hispanic white. Post-survey 737 (29% response rate). Aim 1, pre-post (Likert responses, larger numbers favorable) demonstrated increased student knowledge to identify depression (4.26 [4.19-4.33] to 4.59 [4.47-4.71], P < .001) and help a friend access support (4.30 [4.21-4.38] to 4.56 [4.40-4.71], P = .001). Help-seeking increased for phone helplines (1.61 [1.57-1.66] to 1.78 [1.70-1.86], P < .001), crisis textlines (1.60 [1.55-1.64] to 1.78 [1.70-1.86], P < .001), internet/websites (1.80 [1.75-1.85] to 1.99 [1.90-2.08], P < .001), school counselors (P = .005) and teachers (.013). Aim 2, no significant differences in knowledge, help-seeking or culture between curriculum only vs curriculum plus club schools. Aim 3, staff (n = 17) interviews supported reduced stigma and increased mental health referrals. Conclusions Aevidum’s curriculum improved mental health knowledge and help-seeking; adding the club did not significantly change responses. Staff identified positive school culture impacts. Limitations include the lower post-survey response.

][9] The University of Michigan's (UM) Peer-2-Peer (P2P) Depression Awareness Program was 1 such school-based initiative that aimed to decrease mental illness and promote well-being. 10Students were trained as peer leaders to design and implement a depression awareness campaign.Pre-post assessments demonstrated improved knowledge and attitudes regarding depression, greater confidence to identify and refer at-risk peers, improved help-seeking, and reduced stigma. 10imilar to P2P, Aevidum, meaning "I've got your back," is a student-led mental health initiative started in 2010 following a peer suicide in Lancaster, Pennsylvania (PA). 11evidum has grown from statewide to national reach, and the Aevidum mental health curriculum and club activities are currently used in over 300 schools within PA and surrounding states.Like P2P, Aevidum leverages peer networks in its clubs to expand knowledge about mental health, promote help-seeking, and reduce stigma.11 In 2017, Aevidum introduced its school mental health curriculum adopted by many PA schools to meet mandated Act 71 curriculum standards for mental health and suicide prevention.12 Despite widespread use, neither the Aevidum curriculum nor club have been formally evaluated.
The study purpose was to understand the effect of the Aevidum curriculum and club on mental health knowledge, help-seeking intentions, and school culture.The study used a mixed-methods design including pre-post surveys, a randomized clinical trial design, and qualitative interviews involving ten Pennsylvania public high schools.

Aevidum Curriculum and Clubs
The Aevidum Mental Health & Suicide Prevention Curriculum was developed with a team of school counselors, principals, and psychologists and consists of 5 lessons delivered over 3 hours. 13The content can be given in 1 threehour session or broken into smaller segments; videos pair with the lessons.School staff participated in a one-hour educator workshop, and were then asked to deliver the curriculum to 9 th graders by February 2022.Due to trimester scheduling, 1 school delivered the curriculum beginning March 2022.
The Aevidum club can be implemented independently of the curriculum.The intent is to foster cultures of caring, kindness, and advocacy. 14When starting a club, schools are requested to have approximately ten students and 2 educators attend a 1 day "Talk" training workshop facilitated by students and adult advisors from an active club.
Ten participants allow enough trained students to launch something significant, and schools can send student leaders from different grade levels and social groups.Yet, participation is limited to ten as training often includes multiple schools.
The Talk format has been endorsed by national experts (Substance Abuse & Mental Health Service Organization), and was featured on the top National Prevention Weeks campaigns. 14New clubs receive an Aevidum starter kit including banners, posters, and promotional materials, suggested campaign ideas, and wristbands with the National Suicide Prevention Lifeline number. 14Schools implementing the club participated in a virtual Talk workshop on October 29 th , 2021, and were mailed the starter kit.Additionally, schools were asked to carry out 2 club campaigns/activities during the academic year.Schools received a $1k stipend to support club activities, with no restrictions other than activities relate to mental health.Club campaigns have included "Be Kind to Your Mind," an after-school "hangout" initiative, and "Twosday," a tutu dress-up day to signify the hourly number (twenty-two) of adolescents who attempt suicide.Additional activities have included spirit weeks and participation in homecoming parades.

Design & Sample
This study took place over the August 2021-June 2022 school year and focused on evaluating Aevidum curriculum and club effectiveness to improve adolescent mental health knowledge, help-seeking intentions, and school culture.Figure 1 overviews the mixed-methods study design.With Aevidum's Executive Director, ten Pennsylvania schools were recruited via email communication.Priority was given to schools with previous interest in Aevidum and/or a prior relationship with the research team. 15,16Before the 2021-2022 academic year, schools were randomly assigned via covariate-constrained randomization to implement the Aevidum curriculum (n = 5) or the curriculum and club (n = 5).Curriculum only schools had the opportunity to start clubs the following academic year.This study received an exempt research determination by the Penn State Institutional Review Board.The randomized clinical trial was registered in ClincialTrials.gov(CONSORT diagram Figure 2).

Measures
Pre-post surveys, used with permission, were those used to evaluate P2P, the UM Depression Center P2P Depression Awareness Assessment. 10,17Questions are broadly grouped into mental health and depression knowledge, help-seeking, and school culture changes.While the survey asks no personal information, students were incentivized for pre and post-survey completion.For completing the pre-survey, 4 students per school were chosen to receive a $20 gift card (post-survey $30 gift card).Thus, to be eligible for the raffle, and to ensure post-survey distribution only to those who completed the pre-survey, an email address was required.Opt-out forms were sent to student parents at all participating schools.Prior to Aevidum curriculum implementation, all eligible students received a link to REDCap (Research Electronic Data Capture), a secure, web-based software platform designed to support data capture for research studies. 18,19This link was forwarded directly to students by the schools, thus the study team was not provided a list of student opt-outs.Post-surveys were sent via the same secure online system at the conclusion of  the school year directly to individual student email addresses supplied following completion of the presurvey. 18,19alitative Component Survey responses from curriculum only schools were compared to curriculum plus club schools to understand the added benefit of the club on school culture.However, as school culture shifts may be subtle, the student survey data was supplemented with school staff interviews to better understand school culture outcomes from Aevidum.Semi-structured interviews were conducted with up to 5 staff from each school (March 31, 2022 to May 16, 2022).Participants received a $25 gift card.A qualitative interview guide was developed, interviews were conducted via web-based meeting platform (Zoom), audio-recorded, and transcribed for analysis purposes.Transcripts were reviewed and uploaded into NVivo, a qualitative software program, for coding and analysis.

Analysis
Pre-and Post-Surveys.In primary statistical analysis (Aim 1), curriculum only and curriculum plus club schools were analyzed separately, using the same methods.Mixed effects linear and logistic regression models appropriate for longitudinal (repeated measures) data were used to analyze all survey items collected at each time point.The models contained a fixed effect for time (pre vs post) and random effects for school and student.The parameter for time was the primary parameter of interest in the model, as it indicates the change over time with intervention implementation.Parameter estimates and standard errors from the models are reported along with corresponding 95% confidence intervals and P-values.The impact on 9 th grade, to which the curriculum was delivered, was also considered separately.
In secondary statistical analysis (Aim 2), curriculum only were compared directly to curriculum plus club schools regarding changes in knowledge, help-seeking behavior, and school culture.The same mixed effects regression models were used for each survey item, except the model also included a fixed-effect for group (curriculum only vs curriculum plus club) and a group by time interaction effect.All analyses were performed using SAS statistical software version 9.4 (SAS Institute, Cary, NC).
School Staff Semi-Structured Interviews.In Aim 3, the intent was to evaluate school culture change outcomes (eg, stigma) as a result of Aevidum.An inductive, qualitative codebook was developed by the study team after reviewing all transcripts and coming to consensus on common codes and sub-codes.Two independent coders (HC & ML) coded 20% of the data and met to determine the kappa.Substantial agreement was obtained between the 2 coders with a final kappa of .78.One coder (ML) coded the remaining data for subsequent thematic analysis. 20,21As in other research this team has completed, the collaborative approach to coding and theming is described in those manuscripts. 22

Pre-Post Surveys (Aim 1)
In total, 2557 students completed the pre-survey; 1560 curriculum only students and 997 curriculum plus club arm students.Mean age was 15.6 years, 49% female, 86% identified as non-Hispanic White, and 30% were 9 th graders to whom the curriculum was delivered (Table 1).Post-surveys, sent only to those who completed the pre-survey, were completed by 392 (25%) curriculum only students and 345 (35%) curriculum plus club arm students.Post-survey nonresponse was significantly higher for males, Black and Hispanic adolescents, and 12 th graders.
Pre-to post-survey comparisons within study arms for all grades and 9 th grade alone are shown in Tables 2 and 3. Overall, curriculum only students reported improvements in mental health knowledge with significant increases in confidence to identify someone showing common signs of depression (7-point Likert scale with larger numbers more favorable, mean 4.26 [4.19-4.33] to 4.59 [4.47-4.71],P < .001)and ability to help a friend access school and/or community mental health supports (4.30 [4.21-4.38] to 4.56 [4.40-4.71],P = .001).In particular, 9 th graders indicated higher levels of confidence for both items (post-survey 4.63 [4.46-4.81]and 4.77 [4.55-4.98]respectively).Students demonstrated greater understanding that depression is not a sign of personal weakness (correct question response 77% to 86%, P < .001).Further, mental health literacy gains were significant in more students being able to identify depression symptoms, specifically, difficulty concentrating, feeling angry, frequent unexplained aches and pains, feeling tired or less energetic, and/or eating more than usual.
Improvements in help-seeking were less pronounced, though students were significantly more likely to report intention to seek help from phone helplines ( ], <.001).These negative feelings towards a peer with depression were more pronounced for 9 th graders.Finally, on the post-survey more students indicated "other" mental health problems were an issue and they would like the school to show they care about student mental health in "other" ways.Free text responses were not available for additional detail.

Curriculum Only vs Curriculum Plus Club (Aim 2)
Comparison of responses from curriculum only to curriculum plus club schools yielded essentially no differences in pre-post test responses between groups as relates to knowledge, except those in the curriculum only arm had significantly greater odds of recognizing that feeling tired or less energetic could be a symptom of depression (OR [95% CI] for curriculum 1.88 [1.12-3.15]vs curriculum plus club schools .82[.50-1.36],P = .025).Regarding help-seeking, students in the curriculum plus club arm were more likely to seek help from clergy (difference of means [95% CI] for curriculum -.06 [-.14-.02]vs curriculum plus club schools .07[-.02-.15],P = .03).Students in the curriculum only arm demonstrated more of a positive response to a new peer with depression than those in the curriculum plus club arm.
Three major themes are described below and in Table 4 where they are mapped against survey findings.Theme 1: Overall, there seemed to be an increase in mental health referrals at schools, due to Aevidum as well as increasing acceptance of speaking to counselors.While students did not demonstrate an increase in helpseeking behavior based on survey responses, school staff subjectively reported an increase in student mental health referrals suggesting an impact that was not captured by the survey alone.
Theme 2: Schools reported a noticeable difference of mental health awareness in staff and students, through openness and conversation, visual aids, and an overall positive school climate.
While students did not report many significant changes in school culture/climate from Aevidum, interviews suggest staff did observe a positive change in the school mental health dialogue.Theme 3: Schools described multiple barriers that get in the way of educating students on mental health, including community hesitancy and a focus on academics in school.
This barriers-related theme was not part of the student survey, but emerged in the staff interviews.Staff described barriers that inhibited mental health awareness efforts.Pushback from "old school" community members that schools should not be dealing with these issues was described: And so the barrier that we face a lot of the times is our community thinks and still thinks that schools should just focus on education; education and academics.You shouldn't be meddling in these kids personal lives.You shouldn't be dealing with their social, emotional, you shouldn't be dealing with their mental health.
One of the biggest barriers may be just traditional values and viewpoints, meaning more of an old school perspective, that a lot of our students get from their homes.So if you're somehow able to shift the family or parents' perspective, that's a good, but that's awful challenging.So I would think just maybe that's the biggest barrier is just people stuck in that older school mentality of not speaking up about mental health, or trying to tough things out or not recognizing that.

Time was another implementation barrier, as time for mental health takes away from educational aspects:
And then the teachers are being told, "You have to do bio remediation, you have to do this.You have to get them ready for keystones."Testing is part of life and it's part of like what we do in the high school, but it just seems like more time is allotted to all the academic and the testing than there would be to any mental health/behavior change with Positive Behavioral Interventions and Supports (PBIS), so I think we get shortchanged…[for]… mental health.
I think just because it's so new, the standardized testing, I think that's crazy.The kids are going through so much and we're hitting them with tests.And I think that some kids, academics isn't their thing.They're not worried about that if they're having mental health struggles and I don't think it's fair to give everybody the same...So I think standardized testing and a standardized test driven society is something that's... It's unfair and I think that could be a barrier.

Discussion
This mixed-methods evaluation of Aevidum's mental health curriculum and club was undertaken to evaluate benefits for knowledge, help-seeking, and school culture.Ninth graders who received the curriculum and other students in the school demonstrated both improved mental health knowledge and help-seeking intentions (Aim 1).
Addition of the club did not greatly impact knowledge or help-seeking.Students at curriculum plus club schools reported decreased embarrassment if seen going to the counselor's office, but overall support for a new peer with depression was poor (Aim 2).Yet, school staff qualitative data suggests visible improvements in school culture not well-captured on the student survey (Aim 3).The findings support mental health education in schools, confirm the value of the Aevidum curriculum to improve high school student knowledge and  help-seeking behavior, and point to benefits from a school staff perspective in a positive school culture shift.

Comparison to Prior Research
Findings are similar to those from to P2P. 10 P2P students reported higher baseline knowledge and greater baseline and post-survey willingness to seek help than students in our study. 10Yet, both programs demonstrated significant change in confidence to identify someone with signs of depression and help a friend access mental health supports.Also similar were improvements in knowledge regarding signs of depression. 10ike Aevidum, from pre to post-survey, those exposed to P2P reported help-seeking intention increased with teachers, websites, and helplines, but also for mental health professionals, doctors, and clergy members.In contrast, students exposed to P2P were more likely to positively support a new peer with depression and reported increased comfort talking about mental health issues. 10Differences may be due to several factors.First, the P2P analysis discarded all pre-survey results without a corresponding post-survey. 10In contrast, we retained all pre-survey results based on the concern that removing those who failed to complete the post-survey might result in a less representative subset of students.P2P student trainee activities required review and approval by the research team. 10In contrast, following the Talk training, Aevidum club activities were managed by individual schools.Finally, P2P evaluation was conducted during the 2015-2016 school year, 10 and our study was conducted during the 2021-2022 school year, complicated by COVID-19.It is unclear if student club participation was limited and how this impacted findings.Several schools were still working through test to stay strategies to maintain in-person learning. 23,24imilar results are reported from other school-based depression curricula.The Adolescent Depression Awareness Program (ADAP) was a three-hour curriculum delivered to 710 high school students from 6 Oklahoma schools.ADAP demonstrated improved knowledge and help-seeking, but only among those exposed directly to the programming. 25The Integrated Science Education Outreach (InSciEdOut) program was a school-based mental health intervention to reduce stigma delivered to at-risk 7 th and 8 th grade students in 2015-2016.Participants demonstrated improvements in knowledge and help-seeking, but stigma, which was low at baseline, was unchanged. 26he challenge in assessing stigma is that it relates to school culture, which may be difficult to capture in a survey.This was the rationale for inclusion of a qualitative component in this evaluation.School culture is defined as the values, attitudes, and behaviors that characterize a school. 27School culture has been shown to influence student substance use. 27Thus, it stands to reason that school culture could similarly influence the stigma related to mental health.Our school staff interviews found an overall positive experience with Aevidum programming and a sense among staff that students were more open to discussing mental health.These findings were echoed with InSciEdOut, which included teacher narratives to supplement student survey results. 26Similar to our study, teachers felt positive about the programming, and felt students were more open in discussing mental illness with greater acceptance of mental health diagnoses. 26tudy strengths include post-pandemic timing with a large, diverse student sample.Limitations include a lower than expected post-survey response rate due in part to difficulties with school e-mail servers.Although the Aevidum curriculum showed significant improvements in adolescent mental health knowledge and help-seeking, its impact on long-term clinical outcomes was not assessed.Schools had previously worked with the team or expressed interest in Aevidum, so results may not generalize to other educational settings without assessment of contextual, socioeconomic, and cultural differences.Also, students who participated may not represent the general adolescent population.

Implications and Next Steps
This study contributes to the growing body of literature on school-based depression education and peer support    programming.Future investigations should consider the longterm impact of the curriculum, especially sustained changes in knowledge and help-seeking with annual exposure (Aim 1).Response rates may be improved by including the assessment in the classroom at the end of curriculum delivery.This may also improve the ability to capture a broader spectrum of student opinions.Aevidum club did not demonstrate significantly greater impact vs the curriculum alone in the short-term, but this may change long-term as clubs become more established and grow their membership (Aim 2).Finally, while staff noted positive changes, barriers to implementation were also identified.Implementation of evidence-based mental health practices in the school setting is an emerging area of research that could support sustaining Aevidum programming in the school setting (Aim 3). 28he work.KP drafted the work and KP, EL, AM, HC, FP, HS, and DS revised it critically for important intellectual content.All authors gave final approval of the version to be published.All authors agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was supported through the Hollister Confidence Fund [Hopewell-THCF-The Pennsylvania State Unive-Subgrant-013544-2021-02-17].The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.The use of REDCap (Research Electronic Data Capture) in this project was supported by the National Center for Advancing Translational Sciences, National Institutes of Health (NIH) [Grant UL1 TR002014].

Ethical Approval
The Penn State College of Medicine Institutional Review Board determined the STUDY00017706 to meet the criteria for exempt research.

Figure 1 .
Figure 1.Overview of the components of the mixed-methods study.
Yet, these same students responded that a new student with depression would be viewed as dangerous(5-point Likert scale with lower numbers more favorable 1.98 [1.92-2.04] to 2.13 [2.04-2.23],P = .003).Students indicated they would be disinclined to help this student and more likely to stay away.Students reported the new student would be likely to be made fun of (2.77 [2.71-2.84] to 2.96 [2.87-3.06],P < .001) or ignored (2.85 [2.79-2.91] to 3.03 [2.93-3.12

Table 2 .
Comparisons of Pre-to Post-survey Responses for Knowledge and Help-seeking -Curriculum Only Schools.

Table 2 .
(continued) a Mean (95% CI) and P-values from a mixed effects linear regression model.b N (%); P-values from a mixed effects logistic regression model.Pattison et al.

Table 3 .
Comparisons of Pre-to Post-survey Responses for Culture/climate -Curriculum + Club Schools.

Table 3 .
(continued) b N (%); P-values from a mixed effects logistic regression model.Pattison et al.

Table 4 .
Comparisons of Change From Pre-to Post-survey Responses Between Study Arms (all Grades).

Table 4 .
(continued)We participated in the homecoming parade.The kids were very enthusiastic.… We bought Aevidum clothing, so a lot of them wore that.We wore black and yellow.We handed out cards that had suicide prevention hotline on one side, and our Aevidum logo on the other.That got them so pumped up that they couldn't wait to do the next parade, … We handed out candy canes that we had taped a message to it that had the Aevidum logo on one side, and then the saying, "If you needed a sign to not kill yourself, this is it."..I think Aevidum certainly has helped because it's putting out there everywhere, "It's okay.It's okay to talk about it.There are people to help you."We've always done that but we didn't have it plastered all over the walls of the hallway, particularly the ninth grade hallway.They'll put stuff up throughout the building, things that are like, "You're okay.If you need help, get help.This is how to do that."And things like that.So I think that definitely has helped increase the number of students who have come in for help or who have asked for help.