Evaluation of a new e-learning framework for teaching nuclear medicine and radiology to undergraduate medical students

Background Radiology and nuclear medicine have traditionally been taught in a series of lectures and seminars concluded by an examination testing knowledge rather than skills. Purpose To emphasize skills in the students’ learning process, we developed and evaluated a new e-learning framework for teaching medical imaging. Material and Methods The framework consists of electronic lectures, a learning management system (LMS), and a diagnostic viewing system. Students were to review positron emission tomography/computed tomography (PET/CT) examinations of five cases of lung cancer. The framework was evaluated in an objective structured clinical examination (OSCE) taken by 139 undergraduate students at the end of their third year, and in a comparative survey of two groups of 85 and 84 students in the fifth and sixth year who were taught the same oncology course with and without mandatory PET/CT exercises, respectively. Results Of the 139 third-year students, 134 (96%) passed the OSCE PET/CT station according to the predefined criteria. While 85/85 (100%) of the fifth-year students had taken exercises when they were mandatory, only 2/84 (2%) of the sixth-year students had reviewed the cases on a voluntary basis (P < 0.001). Of the 25 survey responders in the fifth year, 24 (96%) agreed that the mandatory PET/CT exercises had promoted their learning while the sixth-year students, whose course concluded with a multiple-choice examination, emphasized the utility of online lectures. Conclusion The new e-learning framework teaches students basic competency in PET/CT navigation and interpretation and is associated with a high degree of student satisfaction.

• Written instruction to the student (To be read by the student while she waits for a maximum of two minutes outside the station.) You are physician in the Lung department at Haukeland University Hospital. You are preparing yourself for the next multidisciplinary lung meeting by looking at the PET/CT study of a patient with a lung lesion. Look at the examination and demonstrate the findings.
66-year-old male patient with chronic obstructive lung disease. A computed tomography because of persistent cough had shown a newly developed lung lesion. Because the lesion was in a difficult location for primary biopsy, the patient was referred to a PET/CT examination with radioactive glucose (F-18-FDG).
You will find the PET/CT examination already loaded into the diagnostic image viewing software. Find the lung lesion. Determine its diameter in millimetres. Evaluate if the lesion has tracer uptake. Measure the intensity of the uptake with the image viewing program. What is the most likely diagnosis? Does the PET/CT examination give an indication of disease stage? If yes, show the relevant findings. What treatment do you suggest?
N. B. You are not required to remember all questions under the examination; the examiner will ask you the pertinent questions and help you move along. The student demonstrates the lung lesion on the screen using a CT lung window 2 The student correctly measures the greatest axial diameter in millimetres 3 The student gives a correct visual assessment of the lesional FDG uptake ("intense), more than in the mediastinum) 4 The student measures the correct SUVmax 5 The student concludes that the lesion is malignant 6 The student shows metastatic spread to and ipsilateral hilar lymph node 7 The student can show the lymph node in a soft tissue window 8 The student sees that there are no distant metastases 9 The student suggests curative treatment (operation or stereotactic radiation therapy depending on the patient's lung function) Each question relates to a single patient who was examined at the Lung department at Haukeland University Hospital and who underwent PET/CT. Every case will have the same 7 questions! The current case is patient ID = ***_FDG_208 67-year-old male with COPD. CT showed a progressive lesion in the lung. Because of the localization of the lesion and lung emphysema, no biopsy was taken. The patient was therefore referred to FDG-PET. Question feedback (red): In case you found something in the right lung: Are you looking at the correct patient? Check that the patient ID in Oasis (which is listed as the first item in line 2 of the header) is the same as the patient ID in the introduction to this case. -Looking up wrong patient is a classic blunder! It happens to me about twice a year, and it usually takes med som 5 -10 minutes before I find out.
Question 2: How big is the lung lesion in millimetres?
Please measure the maximum axial diameter in mm. Enter whole numbers only (no decimal separator).
Correct answers between 12.0 and 16.0. Question feedback: I measure approx. 13 mm in axial section 48. There are several possibilities why your answer could be wrong: • You need to select the lung window to obtain the correct diameter. When you measure in soft tissue window, the diameter will appear smaller than the true measurement. • Have you chosen the slice with the largest axial diameter? The correct slice is #48 of the whole-body CT series. • Did you mix up cm and mm? Question 3: When you look at the PET images, does the lesion have FDG-uptake?
Answer alternatives (only 1 correct): • No • Slight • *Intense (correct) 3 Question feedback: Compare uptake in the lung lesion with physiological soft tissue uptake in the mediastinum, which is considered "weak". Compared with the mediastinal uptake, the uptake of the lung lesion is "intense".
Question 4: Measure the lesion's FDG uptake as maximum standard uptake value (SUVmax). Type whole numbers without decimal separator.
Correct Answers -Between 9.0 and 10.0 Question feedback: SUVmax is a fairly robust parameter that is observer-independent. In routine clinical practice, we use SUVmax, not SUVmean. The SUVmean requires that one draws a precise volume of interest (VOI) around the lesion, which is time-consuming and a potential error source. SUVmax does not require a precise VOI. Therefore, we use SUVmax rather than SUVmean. Possible sources of error: ■ Have you looked at the right focus? ■ Have you read the correct numbers in Oasis? You must read the red number after "Max". A growing FDG-positive lung lesion suggested lung cancer. Surgical excision was performed. The final diagnosis was non-small cell lung cancer in the left upper lobe with two lymph node metastases in the left hilus. We take the histological findings as confirmation that our PET diagnosis was correct. That final histology reveals more lymph node metastases than seen on imaging is not unusual.

Supplementary Materials 3: Student survey responses
The user survey was conducted in Norwegian using the Moodle "Feedback" activity under Moodle 3.5. The following is a transcript of the raw survey results. Graphs were created using "R" after export of the complete survey data in comma separated value format. To test for difference between student groups, Fisher's exact test was used for proportions (number of respondence), and Kruskal-Wallis test for Likert scales (question 1 ff.) and for the time spent on the learning tasks (question 4), making no assumptions of a normal distribution. A p-value < 0.05 (two-sided) was assumed for statistical significance.
The authors' comments on selected survey results or responses are typed in italics.

Informed consent
All respondents (41 of 41) gave their informed consent to participate in the survey.
Authors' comment: The higher proportion of respondents in the 10 th term may suggest a higher level of student engagement. However, students in the 12 th term underwent their main examinations in internal medicine and surgery at the end of their studies immediately after finishing the course in radiology and nuclear medicine.
Questions 1 to 4 focused the main course activity and thus differed between 10th term and 12th term students.

Question 1: Relevance of MCQ (12 th term)/compulsory activity (10 th term)
"I perceived the questions in the multiple-choice examination as relevant for my future medical profession." (12 th term) -"I perceived the obligatory learning activity (look at 5 FDG PET/CT examinations in patients with lung cancer) as relevant for my future medical profession." (10 th term) Authors' comment: A greater proportion of students in the 12 th term doubt the relevance of the MCQ examination while a larger proportion of the 10 th term students perceives the compulsory learning activity as relevant; however the difference in responses does not meet statistical significance (n.s).
Question 2: Fairness of scoring (12 th term)/formative feedback (10 th term) "I perceived the scoring of the multiple-choice examination as fair." (12 th term) -"I perceived the formative feedback from the e-learning system as fair/constructive." (10 th term) Authors' comment: The response of "Do not know" of 23 % the 12 th term students is correct in that the results of the MCQexam were not disclosed. All of the 12 th term students passed with "A", the top grade; only 1 student passed with "B". The difference between the responses of 10 th and 12 th term students is statistically significant (p < 0.05).
Question 3: Good teaching material "I was offered good learning aids to prepare myself for the multiple-choice examination." (12 th term) -"I was offered good learning aids in order to prepare myself for the compulsory learning activity." (10 th term) Authors' comment: Students in the 10 th term appear less satisfied than students in the 12 th term (n.s.). The probable explanation is that there was very close alignment between the MCQ-questions in the teaching material on https://mitt.uib.no and the actual examination on https://vurdering.uib.no. The teaching materials in the 10 th term were similar, but they were to provide foundational knowledge and guidance for a practical task, which resided in a different domain.

Question 4: Time spent (in hours)
"How much time (hours) did you spend preparing the multiple-choice exam? How much of your time (in percent) did you use on the oncology part of the multiple-choice examination?" (12 th term) -"How much time (in hours) did you spend on doing the obligatory learning activity?" (10 th term) Authors' comment: It is very surprising that the 12 th year students report spending more time on preparing the MCQ examination than the 10 th year students on the compulsory learning activity (n.s.).

Question 9: E-learning exercises
"Exercises in the e-learning system mitt.uib.no promote my learning in medical imaging." Question 10: Medical imaging software "Working with a medical imaging program such as Oasis promotes my learning in medical imaging." Authors' comment: Students in the 10 th term give the use of dedicated medical image viewing software the highest satisfaction rating in the survey (p < 0.001). The proportion of 12 th term students who respond in the negative without ever having experienced this mode of teaching is surprising but illustrates the deficiencies of the old curriculum, which was largely lecture-based. . It is noteworthy that the students who are exposed to oral examinations answered more in the negative than those who were not (n.s.).  /mitt.uib.no). Under the new curriculum (10 th term), students undergo an MCQ examination at the end of each term (except the 10 th ). It is interesting to note that the students who were most exposed to MCQ answered more in the negative (compare also question 11) (n.s.).

Questions 15 to 17: Free-text questions: MORE, LESS, Comment
The last 3 questions were answered in free text format: "What would you like to have MORE of in teaching ("more")? What would you like to have LESS of in teaching ("less")? Do you have any further comments ("Comment")?" To enhance readability, the responses of each respondent are listed sequentially according to student term. Respondents are counted sequentially within each student term. Occasional explanatory comments by the authors are enclosed in square brackets [].

10th term students (n = 26)
Eighteen of the 26 respondents (69 %) in the 10 th term answered at least one of the three free text questions.

Student 3/26
More: Very much liked the web lecture which explained how to use Oasis. I would've liked to learn the rest of radiology in the same way! Comment: Very well-designed MCQ, Martin. One gets feedback why something is wrong. Many questions that make sure that you cannot cheat your way through.
Authors' comment: The student's comment refers to the multiple-choice questions in the teaching material on the elearning system mitt.uib.no; there was no multiple-choice examination in the 10th term.

Student 4/26
More: It is very exciting with exercises in which one really tries out Oasis [the image viewing software]. I think five compulsory teaching cases are enough, but it would be nice if we had access to some more, so we could try if we wanted to.
It would have been nice with an extra minute or two in the web lecture about how one decides on the treatment based on the imaging findings. When is surgery no longer advisable, when external beam radiation? I understand that this mainly will be taught within oncology, but a short summary would have been super.
It is very nice with web lectures so that one knows quite concretely what one needs to do in order to prepare for the seminar.
Less: not sure.
Comment: I constantly and on all occasions praise how good the teaching in nuclear medicine is. Thanks a lot that you talk yourself the time to make such a good course for us.

Student 5/26:
Comment: Teaching in nuclear medicine in this semester has been incredibly good. Short, good web lectures, concrete information on how we can prepare ourselves the seminar and incredibly instructive compulsory exercises in Oasis! The latter we could very well have much more of in radiology -e.g. diagnose fractures and the like. We would learn much in a fun and clinically relevant manner! I would recommend that you give clear advice that the compulsory exercises can be done in small groups of 2 to 4 students as there are not many computers with the software installed.

Student 7/26:
More: More practice! The most rewarding was to do the compulsory exercises after the seminar in Oasis to understand how the imaging software works.
Less: I think this was a very well-designed and good course. I think that all the subject matter is somewhat stressing and difficult to keep in mind, but I understand that we need this in order to understand the examinations performed in the patients.
Comment: I think this was a fantastic course, having a web lecture with the basal facts in the beginning and some tasks in the e-learning system (mitt.uib.no) where I got immediate feedback. I also learned a lot by working with the subject matter on my own. I would wish that all radiology teaching was designed in this way since I feel that we are a student term who has not got proper systematic radiology teaching. If we could have similar teaching in x-ray, CT and MR of different important organs (skeleton, abdomen, thorax, brain), this would've been very useful for our future clinical work. This is what I really wish before I complete my studies!!! Student 9/26:

Comment:
Absolutely fantastic teaching -if we had this in all subjects we would learn very much! I especially liked the web lectures so that we could prepare us (and look at it again if there was something that we had missed), that they were good to describe what were normal findings and pathology, and Oasis [the medical image viewing software] with feedback to the compulsory exercises (on mitt.uib.no) so that we could learn continuously. I would wish all radiology teaching was like this.

Student 10/26:
More: I like the concept that you challenge us students to sit down in front of a computer and find pathologic findings etc.
Less: Continuation from the previous question: if anything should be done, there should be more progress for the student, otherwise it gets somewhat tedious (especially if teaching is on the late afternoon).
Comment: Very cool to work with the lung cases in Oasis. This was the closest I ever came to navigating in a computer program related to imaging (I have not access to other programs under hospitalizations in peripheral hospitals). Continue with this, please make similar exercises for other areas. This is the closest many of our students will come to practical use of PET/CT before we become physicians.
Authors' comment: The "Less" comment apparently refers to the seminar. It is sometimes challenging to pace case presentations correctly so that no student is bored but every one understands. We now let students present the cases under supervision. This tends to maintain student engagement at a higher level than when the teacher keeps running the entire seminar. Most seminars took place in the morning, however one group had the seminar from 2:15 p.m to 4 p.m.

Student 13/26:
More: More about when one refers to PET, scintigraphy+++ I have no clue when one should or ought to do this.
Less: Less teaching in the claustrophobic room H113 (or so) down below. Much better to look at videos at home when it suits me.
Comment: Fantastic videos, but I would like more practical information when I should send patients to these special examinations.

Authors' comment:
We deliberately do not focus on indications for nuclear medicine examinations. Indications for imaging examinations are an endless field that requires deep knowledge of the affected specialties including alternative diagnostic approaches such as clinical chemistry or biopsies that cannot be assumed in a normal 3 rd or 5 th year student. Teaching indications would take away the focus from the images and the key image findings. In the context of practical medicine, not teaching indications does not pose a problem, since all modern diagnostic/therapeutic algorithms will contain concrete advice on which imaging modality to choose in a given clinical situation.

Student 15/26:
More: More cases, even though there are many already.

Authors' comment:
We are planning to create more cases in the context of future courses such as ELMED318 in January 2018. The teaching material in these courses will also be available made to regular students.

Student 16/26:
More: More tests on mitt.uib.no so that one could test oneself after the teaching. This has been very good in the 10th term :) I often think that teaching can be very theoretical, so therefore [I wish] more concrete and practical in respect my future work as a physician.

Comment:
There are too few computers with access to Oasis [the medical image viewing software]. We were not allowed to be in the clinical training center of the hospital because they often had courses ongoing, and we had no access to teaching room H 113. Therefore, we could only use the two computers at Armauer Hansen's house. More: The type of web lectures that you already have in both nuclear medicine and radiology is worth gold. For my own sake, one could almost have dropped the two weeks with radiology lectures and rather have done something else since the material also was posted on the UiB website as prerecorded lectures which went through EXACTLY the same, but much more effectively and better. (However I got no information that these lectures were posted on the Internet through the lecturers or the faculty, but rather by chance through a fellow student -this could have been said more clearly, but then maybe fewer student would have shown up…) Less: Plenary lectures where one just reads from a PowerPoint presentation and goes through images which one could do equally well on one's own when one wanted/was not tired.
When one should have presentations of cases in PowerPoint (especially in a subject like radiology) in the first place, it is hopeless when the PowerPoint files that are posted after the seminar with these cases do not contain explanations in PowerPoint of what findings these images actually have -with text and arrows on the images in order to explain. I understand that you want to have "incentives" so that people should meet at the lectures. But if the lectures are not good enough on their own so that people meet up, you should not "punish" those who do not show up because you have not sufficiently good lectures… Comment: I think it is incredibly good that you made web lectures, with soundtrack and PowerPoint! They are incredibly instructive, and one can look at them when one self wants to, and that they are relatively short and concise. The result is that one actually manages pay attention.
I am very happy about nuclear medicine, this could have been a rather peripheral and boring subject, but the way it has been structured with quizzes and net-based case exercises has made this very good!

Student 7/15:
Comment: thank you very much for all the work you have invested in the course, this is the best I have experienced in the whole of my medical studies. Unfortunately, the specialty is in the shadow of internal medicine and surgery, just as earlier teaching in radiology came in the shadow of other examinations. I would have rather had the whole specialty in one unit in the 4th or 5th year, then the teaching outcome would have certainly been even greater.

Student 8/15:
More: Group exercises? Probably more use for future students undergoing OSCE.
Comment: Biermann is probably the lecturer at the medical faculty of the University of Bergen who has spent most effort on a teaching setup that is well-defined, up-to-date, accessible and "multimodal" utilizing several teaching modes. Teaching and learning outcomes are largely in alignment with clinical relevance for a "general practitioner", which is what we are actually being trained for. Other subjects, hark, hark! Student 9/15: Less: Focus on particular isotopes for different examinations. As I see it, it is more relevant to learn the principles behind nuclear medicine, which applications they have, and when one potentially can have benefit of such an examination. [This is more important] than to know that it is 99m Tc-MIBI one uses for cardiac scintigraphy and parathyroid scintigraphy, because it difficult for me to see that this is something one must specify when one should refer a patient to a nuclear medicine examination sometime in the future?
Comment: I hope that radiology will be moved to an earlier part of the new curriculum. This is an incredibly useful subject that one should rather learn before the first placement at a peripheral hospital in order to increase learning outcome. When the radiology exam comes so close to the final exams, the effect is that one tries to use as little time as possible on radiology because internal medicine and surgery are looming and so close. Which is very regrettable! Otherwise I want to thank you all for very good teaching in radiology, good lecturers who also manage to differentiate between what we should know as residents and what is more advanced specialist knowledge.

Authors' comment:
We agree -the student should know the basic type of nuclear medicine examination and the principles behind it does not need to memorize the name of the radioactive drug. However, there are many different radioactive drugs that can be used for a given nuclear medicine examination so we always have to indicate which precise drug was used. For clinical referrals, it is paramount that the physician describes the clinical scenario and the clinical question, and we will choose the most appropriate study regardless which tracer was suggested in the referral. -In the new curriculum, teaching of radiology and nuclear medicine is no longer concentrated in the last 3 months before the final examination.

Student 13/15:
More: I think that the teaching a nuclear medicine has been very good, and I feel that I have good knowledge on the most important applications and I know when I shall refer.
Less: Details about names of different substances that are injected is knowledge that is soon forgotten, and I'm not sure how much focus one should put on this.
Comment: Thanks a lot for the effort that your put in to make web lectures, quizzes etc. This has been very good!

Student 14/15:
More: A larger number of relevant exercises on mitt.uib.no with images. This is a very visual specialty in which one needs to look at and read images. The web lectures were also very good and informative.
Less: Less "whiteboard/PowerPoint teaching " Authors' comment: We heartily agree with you. In the most recent redesign of the course PowerPoint is used for the web lectures only, and Face-to-face teaching is largely based on the interactive presentation of cases using dedicated nuclear medicine software such as Oasis. We resort to Powerpoint only when specific questions come from the audience that are best explained using a PowerPoint slide.