Assessing Professionalism in Medicine – A Scoping Review of Assessment Tools from 1990 to 2018

Background: Medical professionalism enhances doctor-patient relationships and advances patient-centric care. However, despite its pivotal role, the concept of medical professionalism remains diversely understood, taught and thus poorly assessed with Singapore lacking a linguistically sensitive, context specific and culturally appropriate assessment tool. A scoping review of assessments of professionalism in medicine was thus carried out to better guide its understanding. Methods: Arksey and O’Malley’s (2005) approach to scoping reviews was used to identify appropriate publications featured in four databases published between 1 January 1990 and 31 December 2018. Seven members of the research team employed thematic analysis to evaluate the selected articles. Results: 3799 abstracts were identified, 138 full-text articles reviewed and 74 studies included. The two themes identified were the context-specific nature of assessments and competency-based stages in medical professionalism. Conclusions: Prevailing assessments of professionalism in medicine must contend with differences in setting, context and levels of professional development as these explicate variances found in existing assessment criteria and approaches. However, acknowledging the significance of context-specific competency-based stages in medical professionalism will allow the forwarding of guiding principles to aid the design of a culturally-sensitive and practical approach to assessing professionalism.


The need for this review
The absence of such a context-specific, culturally appropriate and linguistically sensitive assessment tool within the Singapore setting inspired this scoping review. It is hoped that the insights proffered will guide design of a tool for the local context.

Methods
Building upon Li et al's, 35 Wilkinson et al's 4 and Veloski et al's 36 reviews, this scoping review explores the size and scope 37 of published data in peer-reviewed literature on tools used to assess medical professionalism. [38][39][40][41][42][43] Use of a scoping review also allows for the systematic extraction, synthesis and summarising of actionable and applicable information 44 in available literature 45,46 across a wide range of pedagogies, assessment contents and practice settings. [47][48][49][50][51] Levac et al's 52 adaptation of Arksey and O'Malley's 38 framework for scoping reviews was used to map key concepts, sources and types of evidences available to guide future research. 39,53,54 Levac et al's framework was also guided concurrently by the PRISMA-P 2015 checklist 45 and a 6-stage scoping review protocol was developed for this study. [38][39][40][41][42]52 Stage 1: Identifying the research question The 11-member research team was guided by a team of local clinicians, educators and researchers from Yong Loo Lin School of Medicine at National University of Singapore (YLLSoM), Duke-NUS Medical School, the University of Liverpool and the National Cancer Centre Singapore (NCCS), and two librarians from the medical libraries at YLLSoM and NCCS (henceforth the expert team). The expert and research teams discussed prevailing issues, needs, and practices pertaining to general Medical Council (gMC) The domains for professionalism include: (a) Knowledge, skills, and performance • Make the care of your patient your first concern.
• Provide a good standard of practice and care. (b) Safety and quality • Take prompt action if you think that patient safety, dignity or comfort is being compromised. • Never discriminate unfairly against patients or colleagues.
• Never abuse your patients' trust in you or the public's trust in the profession.

Canadian Medical Education Directives for Specialists (CanMEDS)
As Professionals, physicians are committed to the health and well-being of individual patients and society through ethical practice, high personal standards of behaviour, accountability to the profession and society, physician-led regulation, and maintenance of personal health.
• Commitment to patients • Commitment to society • Commitment to the profession • Commitment to self 3 assessment tools used in medical professionalism and determined the primary research question to be: 'what tools are available to assess medical professionalism among medical students or physicians?' The teams determined the secondary research questions to be: 'what domains are assessed and what approaches are used to carry out these assessments?' and 'what existing frameworks are used to guide them?' These questions were designed based on prevailing population, conceptual and contextual considerations 55 using a PICOS format (Table 2).
These collaborative consultations between the research and expert teams also took place at Stage 1, 2, 3 and 6 of the research process.

Stage 2: Identifying relevant studies
Guided by the expert team and prevailing definitions and descriptions of medical professionalism by ACGME, GMC and CanMEDS, 32,33,56 the research team developed their search strategy which may be found in Online Appendix A.
In keeping with Pham et al's 54 approach to ensuring a viable and sustainable research process, the research team confined the searches to articles published between 1 January 1990 and 31 December 2018 on PubMed, ERIC, Web of Science and Google Scholar. All qualitative and quantitative research methodologies in peer-reviewed articles published in English or had English translations were included. The independent searches of the four databases were carried out between 13 February 2019 and 24 April 2019.

Stage 3: Selecting studies to be included in the review
The nine members of the research team independently screened the titles and abstracts by exporting and organising the articles using the Endnote software. This allowed for the removal of duplicate titles and the generation of a tentative list. Sandelowski and Barroso's 57 'negotiated consensual validation' approach was used to agree on the final list of articles to be reviewed and, later, the final list of full text articles to be scrutinised.
A summary of the PRISMA process may be found in Figure 1.

Stage 4: Data characterisation and analysis
Braun and Clarke's 58 approach to thematic analysis was adopted to scrutinise articles with different goals and populations of medical students and doctors. It circumnavigates the contextspecific and socioculturally influenced nature 42 of medical professionalism [1][2][3]31,35,[58][59][60] and the wide range of research methodologies present among the included articles that prevent the use of statistical pooling and analysis. 58,[61][62][63][64][65][66] The members of the research team independently reviewed and 'actively' read the included articles to find meanings and patterns in the data to construct 'codes' from the 'surface' meaning of the text. 58,[67][68][69] The initial codes from 'open coding' were grouped into categories then into themes. The reviewers reconvened at online and face-to-face meetings to discuss their individual findings.

Stage 5: Collating, summarising, and reporting the results
A total of 3799 abstracts were identified from four databases, 138 full-text articles reviewed, and 74 full-text articles were analysed.
The narrative produced was guided by the Best Evidence Medical Education (BEME) Collaboration guide 64 72 to evaluate the quality of qualitative and quantitative studies included in this review. They met face-to-face to reconcile any differences in their assessments and forwarded a consensusbased appraisal of the included studies. A summary of these quality assessments may be found Online Appendix B.

Stage 6: Consultations with expert team and key stakeholders
Whilst the findings were well-received by the expert team, several members stressed the need to consider the impact of the diverse assessment methods, approaches, goals, and populations in relation to the local educational, healthcare and linguistic setting. This is so as to facilitate development of a cohesive, longitudinal and holistic system of assessments with appropriate integration of individual assessment tools calibrated for various training stages within the Singaporean setting.

Results
For transparency and ease of review the themes are delineated in tables. The initial themes identified were attributes and assessment criteria for professionalism (Table 3a and b); tools, approaches and modalities used to assess professionalism (Table 4); and prevailing assessment frameworks established by healthcare regulatory organisations (Table 5).
Upon recommendation from the expert team, these three initial themes were reconfigured into two in this review [132][133][134] -the context-specific and competency-based nature of assessments. The competency-based nature of assessments correspond to various stages of a medical student's or doctor's training (henceforth competency-based stages).

Context-specific nature of professionalism assessments
Although many of the included articles adopted definitions of medical professionalism outlined by frameworks established by key healthcare regulatory organisations, their assessments were revealed to be significantly context specific. For example, whilst the ACGME definition was adopted by Hochberg et al 93    Other measures • Improvement in clinical 117 and evidence-based practice 73 • Improvement in patient satisfaction and reduction in complaints 118

Journal of Medical Education and Curricular Development
transitional year residents revealed a heavier focus on participation in events such as grand rounds and sponsored conferences despite following the same ACGME definition guidelines. Differences in settings, specialties, target populations and rationales also influenced assessments of professionalism. Further examples of the context-specific nature of assessments of professionalism are featured in Table 6.

Competency-based stages in professionalism assessment
The competency-based nature of assessments is evidenced by ten studies which put forth a stepwise approach to the assessment of professionalism knowledge and skills informed by the learner's development. 23,93,[100][101][102]114,118,119,122,131

Journal of Medical Education and Curricular Development
In the initial stages, assessments focus on evaluating the understanding of concepts of professionalism as well as their key attributes and role in medical education. Next, perceptions and attitudes towards the roles and responsibilities of a professional are assessed. The final stage culminates in the sustained development and display of key professional attitudes and behaviours over time. 23,93,[100][101][102]114,118,119,122,131 The presence of such progressive and longitudinal competency-based stages highlights the need for a safe and supportive environment as well as structured training processes. 125

Discussion
In highlighting available tools used to assess medical professionalism as well as pertinent domains to consider, this systematic scoping review successfully meets its primary and secondary research objectives. The context-specific nature of assessments is evident in the fact that prevailing tools or systems of assessments are based on regnant characterisations 23,28,29,74,75,79,84,87,89,93,94,98,104,105,[119][120][121] and prioritisation of specific professional attributes of medical professionalism. 21,29,75,76,78,80,83,89,92,95,100 However, despite these contextual differences, it is equally important to acknowledge that there are consistent features to most of them. These include use of a variety of modalities, sources of assessments, and focus on knowledge, attitudes, behaviours, skills, competencies and outcomes. 3,4,23,26,30,74,75,[80][81][82][83][84]105,108,115,116,118,122,130 The findings further reinforce the notion that professionalism assessments should be competency-based, highlighting the need to carefully design their various stages. It is posited here that the stages of professionalism in the included articles correspond with the stages of Miller's Pyramid (Table 7). In turn, assessments of professionalism are consistent with Kirkpatrick's Model of Assessment (Table 8). 23,93,118,122 Here, it is inferred that well-established milestones and competency levels are able to guide the learner's progress as well as inform assessment processes. Likelihood of students to report physicians, nurses and fellow students 125 Impact of faculty development process on faculty's humanistic teaching 92,101 Medical students' professionalism, 91,111 manifest through behaviour and communication skills 80,83,104 • Self-assessment 83 • Peer-assessment 83 • Multi-source feedback 115 • Assessed by Patients 85 Physicians' professionalism 30,22,90,96,118,122 such as through behaviour 73,86 and/or demonstration of Core Competencies set out by ACgME 79,89,99 • Peer-assessment 97 This may also be carried out via evaluating the interpersonal and communication skills, 23,116 self-control 77 and empathy 102 The professionalism curriculum as a cultural change agent 93 Shows how Demonstration of Professionalism in a controlled setting such as in response to clinical vignettes and case scenarios Likelihood of students to report physicians, nurses and fellow students for unprofessional behaviour 125 Response to case scenarios of ethical issues 18,128 and unprofessional behaviour 131 Demonstration of professional behaviour using clinical vignettes 129  Perception and attitudes of professionalism 24,25,84,88,109,117,130 and professional behaviour 21,76,109 Change in attitudes towards professionalism after professionalism course 25,109,114,117,119 Residents' perceptions of their own professionalism and the professionalism of their learning environment 29 13  Tay et al. 15 The stages of assessments also reveal the longitudinal nature of professional development and the need for longitudinal assessments. There are a number of considerations.
One, reliance upon pre-existing knowledge and skills foregrounds the need for effective assessments prior to commencement of professionalism training. Whilst it is crucial that standards and codes of conduct are consistently applied, personalised training and assessments are also paramount as learners may have different baseline abilities and often require different forms of support at different junctures to achieve their desired goals.
Two, longitudinal assessments must be sensitive not only to the personalised aspect of professional development but also the appropriate stage of their learning. Indeed, emphasis on regular assessments require acknowledgement of the learner's prevailing contextual, linguistic and cultural sensitivity as well as local understanding of professionalism. These also underscore the importance of discerning appropriate tools to be employed at each stage, with the learner's movement from didactic learning to independent display of practical knowledge and skills. This stage-wise consideration is important given that the data accrued at each developmental stage will guide subsequent teaching, assessments and remediation for the learner.
Three, most tools do not incorporate multisource data required to effectively appreciate the evolving, adaptive nature of a developing professional identity in an individual and their practice. As far as possible, tools should be specific to different stages of professionalism development or be sufficiently flexible to account for differences in the learner's abilities and setting.
Four, clear and realistic milestones must be established at each competency-based stage. Remediation plans should be also made available and actively integrated to ensure that learners in need do not slip between the cracks and are offered targeted and timely support.
Five, given that assessments are largely dependent upon the supervisor, mentor, coach and/or tutor, assessors must be trained on which tool to use and how each tool is to be effectively used. To facilitate communication between assessors in different settings and acknowledging the longitudinal nature of professional development, a portfolio-based assessment method should be employed to streamline the assessment process.
Consequently, these aspects of assessing professionalism affirms the fact that medical professionalism cannot be restricted to a "standardised and reductionist" definition and assessment method. 2,31,137 Rather, these findings underline the need for a portfolio-based assessment program 4,122 where a mixture of generalised tools and context-specific 'specialised' assessment methods can be employed to assess competencies and milestones achieved. The portfolio must also consider the prevailing practice culture, availability of resources and receptivity of its implementation to ensure the sustainability of professionalism assessment.
A portfolio approach would also allow reflections on individual experiences and allow assessors to evaluate and provide feedback on these reviews of experiences and refinements in practice and thinking. An additional consideration is that learners can also provide their perspective of the assessments and may even provide their views of a particular assessment and challenge the appraisal. This introduces the need for assessments to overseen and reviewed by independent third parties who may offer a review of the overall progress of the learner or a review of a particular assessment.
Mapping these findings have generated much insight and a general guidance for the design of professionalism assessments is featured in Figure 2 where key principles corresponding to each competency-based stage is foregrounded.

Limitations
Whilst this review focused solely on medical professionalism assessments, the articles included explored broader concepts of professionalism. The limited scope of this review also saw the exclusion of closely associated concepts such as professional identity formation and ethics 31 which may have hampered a holistic picture of assessment practices in medical professionalism. 2,31,137 Due to time, manpower and resource constraint, the exclusion of other health professional literature may have also led to the omission of key ideas potentially transferable to the field of medical education.
Moreover, as the included articles were restricted to those in English, majority of the accounts originated from North America and the European countries. Although the utilisation of similar internationally acclaimed and accredited professionalism frameworks may suggest universal similarities and agreement over the domains of medical professionalism, it is evident that clinical and educational committees evaluate and adapt their guidelines in tandem with local healthcare and educational contexts and cultures. As the findings may not be representative of professionalism assessment practices beyond these countries, this review is limited in guiding educators on assessment attributes and criteria that will overcome nuanced geographical and cultural boundaries. [1][2][3]31,35,[58][59][60]

Future directions to consider
Mapping of prevailing medical professionalism assessment practices highlight the following potential areas for future research and action: • • Delineate a working definition of medical professionalism that acknowledges their longitudinal and competency-based nature and integrates local sociocultural and contextual factors • • Establish consensus on a medical professionalism framework that incorporates universally agreeable codes of conduct whilst granting flexibility for geographical and cultural considerations and specialties • • Develop principles to guide the design of professionalism assessments with more robust understanding of how various assessment methods, criteria, content, goals, and needs may be integrated with due consideration for prevailing resources, settings and target populations • • Evaluate the significance of sociocultural and linguistic idiosyncrasies in the design of assessment tools • • Carry out more detailed and context-specific systematic reviews to better determine unique factors influencing how professionalism could be best assessed in psychiatry and other medical and surgical specialities • • Adequately train and support assessors to ensure efficacious use of assessment tools and transparently convey to learners on the specifics of what they will be assessed on and how they will be assessed • • Consider the use of portfolios which will allow for longitudinal, multidimensional reviews and learner initiated commentaries and reflections -this would provide greater opportunities for personalised remediation and training, as well as more streamlined communications between assessors