Positive impact of a telemedicine education program on practicing health care workers during the COVID-19 pandemic in Ontario, Canada: A mixed methods study of an Extension for Community Healthcare Outcomes (ECHO) program

Introduction In addition to shifting and expanding clinical responsibilities, rapidly evolving information and guidelines during the COVID-19 pandemic has made it difficult for health care workers (HCW) to synthesise and translate COVID-19 information into practice. This study evaluated whether a COVID-19-specific telemedicine education program (ECHO COVID) would impact health care workers’ self-efficacy and satisfaction in the management of patients with COVID-19. Methods A prospective mixed methods parallel-design study was conducted among ECHO COVID participants using pre-post questionnaires and a focus group discussion. Questionnaire results were examined for changes in health care workers’ self-efficacy and satisfaction. Focus group discussion data were analysed to explore health care workers’ experience in ECHO COVID and the context of their practice during the COVID-19 pandemic. Results 239 health care workers registered in ECHO COVID and 114 (47.7%) completed questionnaires and attended at least one ECHO COVID session. Median self-efficacy scores increased from 5 (IQR 4–6) to 6 (IQR 6–6) (p < 0.0001), independent of profession, years in practice, age group, or practice environment. Participants were highly satisfied with ECHO COVID sessions with a median score of 4 (IQR 4–5). Focus group discussion data indicated that health care workers gained knowledge through ECHO COVID and revealed facilitators for ECHO COVID program success, including the transition to virtual care, the practicability of knowledge provided, and a ‘perspective from the trenches.’ Discussion This study demonstrated that a telemedicine education program aimed to support health care workers in managing patients with COVID-19 had a positive impact on health care workers’ self-efficacy and satisfaction. This impact was specifically mediated by the ECHO COVID program.


Introduction
2][3][4] In addition to navigating the transition to virtual care, [5][6][7] staffing and clinic changes, [8][9][10][11][12] and vaccine distribution challenges, 13 health care workers (HCWs) have had to learn about the novel SARS-CoV-2 virus.][28] Project Extension for Community Healthcare Outcomes (ECHO) model is a virtual, telemedicine education model that provides longitudinal support and addresses the emerging needs of HCWs. 291][32][33] In April 2020, University Health Network (UHN) leveraged the pre-existing ECHO model as a model of care to support assessment and management of COVID-19 patients.'ECHO Ontario: Managing COVID-19 Patients in the Community' (ECHO for the remainder of this paper) was launched in July 2020.The goal of the program was to disseminate best practices regarding COVID-19 as they emerged and to increase HCWs' self-efficacy and knowledge of COVID-19.
The objectives of this study are: (1) to examine the impact of a telementoring education program on HCWs' self-efficacy and satisfaction and (2) to explore HCWs' experience in the program and context of practice during the COVID-19 pandemic.

Study setting
The 'ECHO Ontario: Managing COVID-19 Patients in the Community' program was represented by an interprofessional 'hub' team, including medicine (family medicine, infectious diseases, internal medicine, respirology, critical care, palliative care, and physiatry), occupational therapy, naturopathy, pharmacy, physiotherapy, social work, and information specialists.The target audience was any practicing HCWs in Ontario.
Each ECHO session ran on Tuesdays for one hour.Weekly sessions included a didactic lecture and a patient presentation. 34All sessions were recorded and available for review after sessions.Participants attended at no-cost and received Continuing Medical Education (CME) or Continuing Professional Development credits for their attendance.
A needs assessment was conducted to develop the didactic curriculum, based on available resources from the international ECHO (MetaECHO) community. 35The final curricula included: classification of COVID-19, pharmacotherapy, management of special populations, community resources, respiratory health, etc. (Appendix 1).Patient lived experiences as didactic sessions were also incorporated in each cycle, which offered a unique perspective on care.

Study design
A prospective mixed methods parallel-design study was conducted to evaluate the impact of ECHO on HCWs. 36 pre-post questionnaire was administered through Survey Monkey®.A focus group discussion (FGD) was conducted to explore the context of practice during the COVID-19 pandemic and the impact of ECHO on their practice.Data were collected from both parts of the study concurrently and analysed.

Study participants
Participants were recruited from June 2020 to March 2021.Participant eligibility for this study included (1) all HCWs (family medicine and specialist physicians, physician assistants, nurse practitioners, registered nurses, registered practical nurses, pharmacists, and other allied health professionals), (2) practicing in Ontario and (3) attended at least one ECHO session.Exclusion criteria included all HCWs practicing outside of Ontario, Canada or those not actively in practice.

Data sources: Pre-post questionnaires
The Pre-Post questionnaires were self-reported and contained sections on self-efficacy and acceptability and satisfaction.Demographics and practice characteristics were collected.
Self-efficacy is defined as an individuals' confidence in their ability to perform a particular task. 37In addition to knowledge, self-efficacy has been demonstrated to be an important mediator for behavioural change among practicing HCWs. 38,39Participants reported self-efficacy using a 17-item scale, adapted from previous ECHO programs.Each item in this section was assessed on a 7-point Likert scale which ranged from 1 = 'strongly disagree', to 7 = 'strongly agree'.Responses of 'not applicable' for any item were excluded.Self-efficacy items included ability to diagnose COVID-19, to identify which tests to order, and to manage special patient populations.
Acceptability and satisfaction with ECHO was measured using an 11-item scale.This section used a 5-point Likert scale where 1 = 'strongly disagree' and 5 = 'strongly agree', with a 'not applicable' option.Items included statements like 'I believe that the quality of care I provide my patient has improved considerably as a result of my involvement in project ECHO' and 'Project ECHO has reduced variations in care'.

Data sources: focus group discussion
One FGD was conducted in November 2020 with participants from the first cycle.The group consisted of five study participants with one facilitator (QJZ) and one note taker.The FGD was conducted and recorded over Zoom.The recording was transcribed verbatim.
Purposive sampling was used to recruit participants who had completed the pre-ECHO questionnaire.This sampling strategy was employed in order to represent a broad cross section of participants, including professions, practice types, and years in practice.
The FGD guide was developed with the research team, containing open-ended questions (Appendix 2).Two broad topics of inquiry included 1) HCWs' experience in ECHO and the impact on their practice, and 2) the context of practice during the COVID-19 pandemic.All FGD participants received a $50 honorarium for their time.

Data analysis
Statistical analysis.Statistical analyses were performed using GraphPad Prism 9.0 (GraphPad Software, Inc., Participants who completed both pre-and post-ECHO self-efficacy questionnaire.
b Will not total to 100% as participants were given the option to select more than one type of practice.
San Diego, CA).For participant characteristics, selfefficacy, and satisfaction responses, continuous and categorical variables were used including mean ± standard deviation, median (interquartile range [IQR]), frequencies (n), and percentages, as applicable.Data distribution was evaluated visually and with the use of Kolmogorov-Smirnov and Pearson omnibus normality tests.Statistical significance between groups was assessed using t-tests (parametric and non-parametric) and chisquare for categorical variables.The changes in selfefficacy scores were evaluated using paired t-tests.Those with missing data for a particular parameter or question were excluded.A p-value of <0.05 was considered significant for all analyses.
Qualitative data analysis.The qualitative descriptive method was employed to synthesise and analyse the focus group transcript.This analysis method was used because it allowed us to produce a comprehensive summary of the data in plain language and to stay close with the original meaning of the data. 40,41Two members of the study team (QJZ, SN) reviewed the questionnaire data and transcript to develop a coding framework of recurrent themes.The themes were then presented to a small working group (DR, SN, AF).Through iterative discussion during the analysis phase, all themes were discussed and discrepancies were resolved verbally.
Qualitative content analysis was conducted inductively to identify emergent themes from the data and develop a framework for coding. 42,43Codes identified the topics in the data pertinent to the research objectives.Data were then grouped into broader themes.All qualitative data was analysed using NVivo (NVivo 11 for Windows, Version 11.0.0.300 by QSR International).

Ethics approval
This study received an exemption by the UHN Research Ethics Board and was approved through the UHN Quality Improvement stream (QI ID#20-0039).

Results
Between July 2020 and February 2021, ECHO offered 24 sessions (twelve per cycle).239 HCWs registered for the program and of those, 114 (47.7%) attended at least one ECHO session and completed at least one of the pre-post questionnaires.50 (43.9%)completed both pre-post questionnaires and were grouped for subgroup analyses.Completion of the pre-ECHO questionnaire took 4 min and 19 s on average, while completion of the post-ECHO questionnaire took an average time of 4 min and 38 s.Comparing HCWs who registered but did not attend any sessions and HCWs who registered and attended at least one ECHO session with completion of questionnaires, differences were observed between groups for years in practice (p = 0.0009), age group (p = 0.0006), and number of patients with COVID-19 (p = 0.0023) (Appendix 3).

Participant characteristics
Of the 114 HCWs who were included in this study, 99 participants were female (86.8%), 42 identified as a nurse practitioner (36.8%), and nearly half reported to be in their first ten years of practice (49, 43.0%) (Table 1).42 ECHO participants reported practicing in a family health team (36.8%), while 82 worked in an urban setting (71.9%), and 45 (39.5%) reported no patients with COVID-19 in their practice.Participants in the total cohort had a median attendance of 3 ECHO sessions (IQR 2-6).The participants in the total cohort attended fewer sessions than those in the subgroup with a median of 3 [2-6] and 4 [3-9] sessions, respectively (p = 0.014).No other statistical differences between participant characteristics in the total cohort and subgroup were observed.
There was significant improvement in self-efficacy scores for all 17 items independent of underlying participant characteristics, with the exception of male sex (p = 0.094) (Table 2).Similarly, ECHO participants who had been in practice for 10 or more years were found to have higher pre-ECHO self-efficacy scores compared to those with 0-9 years in practice (p = 0.0209), but scores increased in both groups to similar values post-ECHO.
Satisfaction.All satisfaction items were evaluated for each participant (Appendix 6).The median of all satisfaction scores for all ECHO attendees was 4 [IQR, 4-5], indicating that most participants 'agreed' or 'strongly agreed' with the majority of satisfaction items (Figure 2).

Qualitative findings
Five participants participated in the 60-min FGD on November 24, 2020.Two were nurse practitioners, one family medicine physician, one pharmacist, and one chiropractor.They practiced at six practice settings: family health team, long-term care facility, independent private practice, aboriginal health access centre, hospital, and nurse practitioner-led clinic.Three participants (60%) practiced in urban centres in Southern Ontario, while two (40%) practiced at rural centres in Northern Ontario.Three participants (60%) identified as women, and the mean years in practice were 35.4 years (SD 11.3).
After analysing the FGD transcript, three themes were identified: (1) knowledge gained in ECHO, (2) context of practice during the COVID-19 pandemic, and (3) facilitators for ECHO program success.HCWs discussed how their unique context of practice changed during the COVID-19 pandemic (Table 3).Interestingly, virtual care increased attendance at ECHO sessions, allowing access to sessions where it may not have been feasible prior.

Integration
Synthesising the quantitative and qualitative data together, the data collected illustrated a rich view of frontline HCWs during the COVID-19 pandemic in the province of Ontario.The FGD explored HCWs' knowledge gained and offered facilitators to program success, which were not otherwise available through questionnaire data.Since knowledge acquisition through multiple choice testing methods could not be ascertained from questionnaires due to the evolving nature of COVID-19 information at the time of program implementation, data collected through the FGD described examples of knowledge change.

Discussion
This study demonstrated that a telemedicine education program (ECHO Ontario: Managing COVID-19 Patients in the Community) had a positive impact on HCWs in primary care.Quantitative results demonstrated that median self-efficacy scores increased significantly from 5 (IQR 4-6) to 6 (IQR 6-6) (p < 0.0001), independent of profession and other characteristics.Participants were also highly satisfied, median score of 4 out of 5 (IQR 4-5).Qualitative results revealed that knowledge was gained through session attendance and explored the context of practice in the COVID-19 pandemic in Ontario, Canada.The participants of the FGD also shared facilitators for ECHO program success, including the transition to virtual care.
As the COVID-19 pandemic has evolved, HCWs' learning needs have evolved as well.5][46] Globally, 304 ECHO programs have launched in 35 countries focusing on COVID-19. 358][49][50][51][52] Response has been overwhelmingly positive, with 500 to over 9000 attendees and reports of high satisfaction.The World Health Organization partnered with the ECHO Institute to launch an ECHO program for the continent of Africa, where there was high demand for credible sources on COVID-19. 53Finally, in Canada, ECHO Coping with COVID (ECHO-CWC) was launched to support HCWs' mental well-being, where they found that, again, participants were highly satisfied with initial sessions (mean = 4.26/5). 54he COVID-19 pandemic has been an inciting event that resulted in increased utilisation and innovation of virtual care. 5,55,568][59] Another outcome of virtual care has been increased attendance at CME events like conferences and ECHO programs.This was exemplified through this ECHO COVID program, which ran Tuesdays during clinic hours.Although attendance was high compared to other programs, findings from this study reveal that the transition to virtual care enabled participants the flexibility to attend ECHO sessions.1][62] Timing and delivery platform should be considered when planning and implementing CME events.
To our knowledge, this was one of the first studies that explored knowledge gain by HCWs in a COVID-19 telemedicine education program.Knowledge change is difficult to measure among learners, especially in CME events and with a multidisciplinary audience. 63,64onetheless, knowledge acquisition is a primary goal of education.Rather than miss this opportunity, a mixed methods study approach was used. 65,66For future program evaluation with an evolving clinical evidence base, qualitative research methods to measure knowledge change can be considered.Canada is in the midst of a fourth wave: 1.58 million people have tested positive for COVID-19 with over 27,500 deaths, individuals 12 years and older are eligible to receive a vaccine, and hospital systems in some regions of Canada remain at capacity from surging acute care hospitalisations. 25,67This study does not capture the impact of the vaccine rollout on outcome measures.Furthermore, even though participants joined from outside of Ontario and internationally (Brazil), this data was not analysed for this study.Future research will aim to collect more qualitative data from HCWs and compare the out-of-province and international cohorts with Ontario participants.Future research will also aim to collect healthcare utilisation data through provincial administrative databases, which will investigate impact of training HCWs on clinical utilisation and costs.
As the transition begins to examining the long-term socio-economic impact of COVID, understanding new vaccines and novel therapies, and preparing for future waves, the adaptability of the ECHO COVID curricula will evolve to reflect these issues.

Limitations
There are several limitations to this study.All questionnaires are self-reported and there was no control group for comparison of self-efficacy.A pre-post assessment, however, of self-efficacy scores within a subgroup of participants was able to be collected.There were differences in age group, years in practice, and number of patients assessed with COVID-19 between HCWs who registered but did not attend any ECHO sessions compared to HCWs who registered and attended at least one ECHO session.This can potentially be attributed to the larger number of participants who answered 'unknown' to the questions related to the years in practice and age group categories who attended ECHO sessions.There was also a larger observed proportion of HCWs without any COVID-19 patients in their clinical practice who registered and did not attend any ECHO sessions.Thus, the lack of perceived applicability to their practice may have accounted for the lower attendance in this group.Furthermore, qualitative data was collected only from five participants, though purposive sampling strategy was employed to ensure a representative sample participated.Given the tight time frame of this study, additional qualitative data collection was not feasible.Finally, due to the rapidly evolving knowledge base regarding COVID-19, knowledge testing using multiple choice or key feature questions was not undertaken.

Conclusion
This study demonstrated that a telemedicine education program (ECHO Ontario: Managing COVID-19 Patients in the Community) had a positive impact on HCWs' selfefficacy and satisfaction.b Will not total to 100% as participants were given the option to select more than one type of practice.

Figure 1 .
Figure 1.(A) percentage distribution of median pre-ECHO and post-ECHO self-efficacy scores across all 17 items for total cohort (n = 114), (B) percentage distribution of median pre-ECHO and post-ECHO self-efficacy scores across all 17 items for subgroup (n = 50).

Figure 2 .
Figure 2. (A) percentage distribution of median acceptability and satisfaction scores across 11 items for total cohort (n = 114), (B) percentage distribution of median post-ECHO satisfaction scores across 11 items for subgroup (n = 50).

Table 1 .
Baseline cohort characteristics of participants.

Table 2 .
Comparison of participant characteristics and pre-post ECHO self-efficacy median scores, subgroup (n = 50).

Table 3 .
Participant quotations supporting each theme.COVID] changed the whole dynamics and everything about the facility changed for everybody[…].I could hardly recognise some of my colleagues with all our gear, even without full PPEs we're all in masks and shields.[…]Alot of staff are extremely stressed, it's more work and the end isn't in sight.[…]It's a totally different work environment to anything I have worked in, in all my[37-year]career.(P_009,MD)Formyself as a chiropractor, we had a complete shut down in all the services we were providing in the province.(P_028,DC)Wehavebeen lucky because we don't have a lot of cases in the area that we cover, so for us we really dodged a bullet and I feel a nice relief.But at the same time, we have not been spurred into practice because we haven't had positive cases.(P_029,NP)Weare seeing an increase in chronic diseases that are ending up in the hospital.We have 120% capacity across Northern Ontario in many of our hospitals.Our larger centres are reaching out to smaller centres to take patients that are not COVID-related and if it's not COVID-related, it's related to people being very sick because they're not getting care early on.(P_037, NP) • Many facilitators for ECHO program success were shared: ECHO COVID offered a space for connection, community, and support amongst colleagues; ECHO offered a perspective from the trenches with other stories of frontline HCWs, and that virtual care enabled attendance at ECHO and other virtual events.
The focus group discussion revealed that knowledge was gained through sessions and explored the context of practice for HCWs during the COVID-19 pandemic in Ontario.Participants of the focus group discussion also discussed facilitators for ECHO program success, including the transition to virtual care.Even though the COVID-19 pandemic has been overwhelming and pervasive, ECHO has provided an opportunity for HCWs to gain access to rapidly evolving information, best practices, and a space for community support.3.Baseline cohort characteristics between study participants (n =114) and registrants (n =125).Study participants are HCWs who registered for ECHO COVID, attended at least one ECHO session, and completed questionnaires (the pre-ECHO questionnaire and/or the post-ECHO questionnaire).Registrants are HCWs who registered for ECHO COVID sessions but did not attend any ECHO session.