The impact of disability on performance in a high-stakes postgraduate surgical examination: a retrospective cohort study

Objective Despite rising numbers of doctors in the workforce with disabilities, little is known about the impact of disabilities on postgraduate performance. To ensure all groups are treated fairly in surgical training, it is essential to know whether any attainment differences exist in markers of surgical performance. To address this gap, we assessed the impact of disabilities on performance on the Intercollegiate Membership of the Royal College of Surgeons examination (MRCS). Design Retrospective cohort study. Setting Secondary care. Participants All UK MRCS candidates attempting Part A (n = 9600) and Part B (n = 4560) between 2007 and 2017 with linked disability data in the UK Medical Education Database (https://www.ukmed.ac.uk) were included. Main outcome measures Chi-square tests and correlation coefficients established univariate associations with MRCS performance, while multiple logistic regressions identified independent predictors of success. Results Though MRCS Part B pass rates were similar (p = 0.339), candidates with registered disabilities had significantly lower first-attempt Part A pass rates (46.3% vs. 59.8%, p < 0.001). Candidates with disabilities also performed less well in examinations taken throughout school and medical school, and after adjusting for prior academic performance and sociodemographic predictors of success, logistic regression found that candidates with disabilities were no less likely to pass MRCS than their peers (odds ratio 1.04, 95% confidence interval 0.66 to 1.62). No significant variation was found in MRCS performance between type of disability or degree of limitations caused by disability (p > 0.05). Conclusion Although candidates with registered disabilities performed less well in formal, written examinations, our data indicate that they are as likely to pass MRCS at first attempt as their peers who achieved similar grades at high school and medical school. In order to enable equity in career progression, further work is needed to investigate the causes of attainment differences in early career assessments.


Introduction
Diversity and inclusivity are high on social and political agendas worldwide. Various definitions of diversity and inclusivity exist but generally, while diversity acknowledges individual differences, such as disability, gender and ethnicity, inclusivity considers how we engage with these differences to establish equity or fairness. Equality and diversity have been prioritised on medical workforce agendas to ensure fairness within the profession, and for the medical workforce to reflect and represent the population for which they provide care. 1 Some protected characteristics have received more attention in research than others. In the United Kingdom UK, ethnicity has been an area of particular focus since the landmark case of the British Association of Physicians of Indian Origin vs. the Royal College of General Practitioners and General Medical Council in 2014, which highlighted the potential legal ramifications of attainment differences in postgraduate examinations. 2 Despite this, there remains a paucity of data on the performance of doctors with other protected characteristics, such as registered disabilities, in postgraduate assessments. This is of interest given the number of doctors with disabilities (defined by the UK's Equality Act, 2010 as a 'physical or mental impairment that has a substantial and long-term adverse effect on his or her ability to carry out normal day-to-day activities') 1 is increasing. 3 Individuals with disabilities often have access to support and reasonable adjustments within school and medical school, but evidence indicates a loss of pastoral and learning support after graduation. [4][5][6] Additionally, many doctors describe facing discrimination within the workplace and worry that declaring their disability has an impact on career progression. 3,7,8 However, whether disability has an impact on postgraduate performance remains largely unknown. One study found no significant difference in pass rates for dyslexic candidates in the written Applied Knowledge Test of the Membership of the Royal College of General Practitioners examination after socio-demographic factors were taken into account. 9 Conversely, doctors with dyslexia had lower pass rates and required more attempts to pass the Membership of the Royal College of General Practitioners clinical examination. 10 Candidates with reasonable adjustments (referring to removing barriers or providing support, such as extra time in examinations) were also found to perform less well on the United States Medical Licencing Examination. 11 Unfortunately, to the best of our knowledge, no studies have investigated whether specific learning difficulties (SpLD) and other disabilities have different impacts on performance at postgraduate examinations.
More generally, within medicine, there has been concern that multiple-choice format examinations may disadvantage candidates with specific learning disabilities, especially dyslexia. 12 However, evidence from a small number of studies looking at multiplechoice performance in medical school does not support this hypothesis. 13,14 Similarly, there have been concerns regarding the impact of dyslexia on student's verbal memory and processing speed during Objective Structured Clinical Examinations with one study demonstrating poorer performance in stations assessing examination skills and data interpretation. 15 Conversely, another study found no difference in the performance of medical students with specific learning disabilities in Objective Structured Clinical Examinations. 13 In short, the evidence from medical school assessments is conflicting, again justifying the need to examine the impact of disability on medical examination performance in more detail.
In the UK, examining bodies are required under the Equality Act of 2010 to eliminate discrimination against people with protected characteristics. 16 If discrimination within surgical training is to be eliminated, it is necessary to identify whether attainment differences exist in markers of surgical performance. Thus, we examined the relationship between registered disability and success at first attempt of the Membership of the Royal College of Surgeons (MRCS) examination while also accounting for other sociodemographic factors that are known to influence performance at the examination, including gender, ethnicity and graduate status. 9,17 In the absence of a gold-standard with which to compare medical examinations, one way of validating an examination is to determine the strength of associations between similar assessments. Performance in one test should predict the performance in a future similar test and scores between each assessment are therefore expected to show significant linear correlation (predictive validity). If group-level attainment differences are identified in a single examination only, the fairness of this examination would be questionable. Conversely, if identical group-level attainment differences are identified throughout all undergraduate and postgraduate assessments, then the equitability of training and education would be brought into question. Given that prior academic attainment is known to be the strongest predictor of later success in medical education, [18][19][20] we also assessed the performance of candidates with and without disabilities in other written examinations taken earlier in their careers to identify attainment gaps and to assess the fairness of the MRCS examination.

Methods
A retrospective cohort study was undertaken using data from the UK Medical Education Database (UKMED: https://www.ukmed.ac.uk/) and four Royal Colleges of Surgeons of the UK and Ireland (Edinburgh, Glasgow, England and Ireland). Anonymised data were extracted for all UK graduates that attempted the MRCS between 2007 and 2017. Candidate first attempt results were used as they have been shown to be the best predictor of future performance in post graduate examinations. 18,21

Membership of the Royal College of Surgeons
The Intercollegiate MRCS is an internationally renowned, high-stakes postgraduate assessment, attempted by 6000 trainees from across the world every year. The MRCS Part A includes two multiple-choice papers taking 5 h to complete in total if no further time is added for reasonable adjustments (also known as testing accommodations). The MRCS Part B is a 3.5-h Objective Structured Clinical Examination during which candidates rotate through 18 stations examining knowledge and skills outlined in the MRCS syllabus.

Registered disability status
Disability data were extracted from the UKMED and MRCS databases including whether a candidate with a registered disability declared themselves limited in their day-to-day activities as a result of their disability. Disability was coded as either; 'SpLD', which includes dyslexia, dyspraxia and attention deficit hyperactivity disorder (of which dyslexia is the most common), or 'Other disability not including SpLD'. 'Other disability not including SpLD' includes any physical or mental disability, impairment or medical condition including serious visual or hearing impairment.

Markers of prior academic achievement
Individual-level linked performance data were extracted for A-Levels (high school exit examinations) as a marker of prior academic achievement and UK medical school admissions test results for the University Clinical Aptitude Test (UCAT), Biomedical Admissions Test (BMAT) or Graduate Medical School Admissions Test (GAMSAT). Total A-Level scores used in data analyses are the sum of all A-Level scores achieved, i.e. A ¼ 10 (being the highest score achievable for each A-Level), B ¼ 8, C ¼ 6, D ¼ 4, E ¼ 2, U ¼ 0 (being the lowest score for each A-Level). A small minority of candidates in the dataset (n ¼ 30) undertook A-Levels after A* grades were implemented in 2010. These were subsequently excluded for cohort homogeneity. Note that Situational Judgement Test (SJT) performance was excluded from UCAT scores as was section 3 (written task) performance from BMAT scores as the introduction and marking of these sections changed over the course of the study period.
In the absence of a medical licensing examination in the UK (such as the Medical College Admission Test [MCAT]), the performance of medical school graduates is quantified by the award of an Educational Performance Measure (EPM) decile score based on their performance decile; ranging from 34 points for the 10th (lowest) decile to 43 points for students in the 1st (highest) decile. Scores from the Foundation Programme Situational Judgement Test (FP-SJT) (a test of the behaviours and attitudes expected of doctors) are also used for the selection of graduating doctors into the UK Foundation Training Programme.

Statistical analysis
All analyses were conducted using SPSS Õ v22.0 (IBM, Armonk, New York, USA). The following standardised and anonymised data were extracted from UKMED prior to analysis by the research team: total A-Level scores, UCAT, BMAT, GAMSAT, EPM decile scores and FP-SJT scores, MRCS Part A and B first attempt results, selfdeclared gender, ethnicity and graduate status. Only candidates with MRCS and disability data were included in the analysis, negating the need for imputation of missing data. All test scores extracted from the UKMED dataset and used in the analyses were standardised according to the applicant to medical school pool and converted to Z-scores to negate changes to pass marks between each examination diet within the study period. Part A and B MRCS performance was categorised as 'pass'' or 'fail' at first attempt for logistic regression modelling. Sociodemographic variables were dichotomised to avoid subcategories with small numbers. Self-declared ethnicity was coded as 'white' or 'nonwhite' as used in similar studies to maximise statistical power. 18,20 Gender was coded as 'male' or 'female' and graduate status was coded as 'graduate' if a candidate completed a degree level qualification prior to studying medicine or 'non-graduate'.
Univariate analysis identified associations with first attempt MRCS pass/fail outcomes and the relationship between disability status and performance on measures of prior academic attainment. Logistic regression models were used to identify independent predictors of first attempt success at MRCS. Initial regression models included disability status and were adjusted for known socio demographic predictors of MRCS success (gender, ethnicity and graduate status). 9,17 A subsequent model included measures of prior academic performance from high school (A-Level scores) and medical school (EPM decile scores). Variables with p < 0.10 on univariate analysis were entered into the logistic regression model and potential predictors with p > 0.05 in the full model were then removed until only statistically significant predictors remained. Effect sizes are given as odds ratio (95% confidence interval). Correlation coefficients were calculated to examine the linear association between test scores and score relative to MRCS Part A (written paper) pass mark. Pearson correlation coefficients are presented, except for EPM decile and MRCS Part A score where Spearman's correlation coefficient was used.
When handling, storing and analysing data, the highest standards of security, governance and confidentiality were maintained. In line with the Higher Education Statistics Agency data standards (www.hesa.ac.uk), all counts presented have been rounded to the nearest 5 to ensure person-level anonymity.

Results
Between 2007 and 2017 a total of 9730 UK graduates attempted MRCS Part A: 9600 had disability data and the other 130 candidates were excluded from further analysis. Of the candidates, 59% (n ¼ 5655) passed at the first attempt. Of the candidates who attempted MRCS Part A, 7% (n ¼ 635) had a registered disability, 65% (n ¼ 415) of these candidates reported a specific learning disability and 9% (n ¼ 60) declared that their disability limited their day-to-day activities. Of all candidates attempting Part A, 63% were male (n ¼ 6090), 58% were white (n ¼ 5560) and 85% non-graduates (n ¼ 8200). Within the study period, a total of 4560 UK graduates attempted MRCS Part B, 71% (n ¼ 3230) passed at first attempt. Similarly of all Part B candidates, 6% (n ¼ 275) had a registered disability, 69% (n ¼ 190) of these candidates had a specific learning disability and 7% (n ¼ 20) declared that their disability limited their day-to-day activities. A total of 65% of Part B candidates were male (n ¼ 2975), 60% were white (n ¼ 2740) and 86% non-graduates (n ¼ 3940). Table 1 shows MRCS pass rates displayed by socio demographic group. Of all candidates without a registered disability, men, white candidates and  Table 4 shows the univariate analysis of performance on prior academic assessments for those MRCS candidates with and without disabilities. Candidates with disabilities performed worse than their peers at A-Levels, BMAT and the EPM (p < 0.05). According to Cohen's guidelines, there was a weak, but   Table 5 shows that men, white candidates and non-graduates were each significantly more likely to pass Part A at first attempt (p < 0.05) compared to their counterparts. When accounting for socio demographic factors, candidates with a registered disability were almost twice as likely to fail MRCS Part A compared to their peers (OR 0.55 [95% CI 0.46-0.64]). Subsequent models adjusting for both socio demographic factors and markers of prior academic attainment from high school (A-Levels) and medical school (EPM decile scores) first separately and then simultaneously revealed no statistically significant difference in the likelihood of success at MRCS Part A between candidates with and without disabilities (OR 1.04 [95% CI 0.66-1.62]).
As a subgroup analysis, the 1715 individuals with complete data were examined. This subgroup was comparable to the full cohort in terms of distribution of sociodemographic factors ( Table 1) and prevalence of declared disability. There was also no statistically significant difference found in Part A pass rates between 'Specific learning disability' and 'Other disabilities' (p ¼ 0.106) or between candidates with and without limitations in their day-to-day activities caused by disability (p ¼ 0.095). Mean A-Level score for doctors with disabilities in this cohort was 31.0 (SD 9.3) and the mean A-Level score for doctors without disabilities was 34.9 (SD 8.9). Mean EPM decile score for doctors with disabilities was 38.2 (SD 3.0) and the mean EPM decile score for doctors without disabilities was 38.7 (SD 2.9). Following adjustment for socio-demographic factors, candidates with a registered disability within this subgroup analysis were more likely to fail MRCS Part A compared to their peers (OR 0.66 [95% CI 0.45-0.97]), data not shown).

Discussion
MRCS candidates with disabilities had significantly lower MRCS Part A pass rates than their peers. They also performed less well in formal, written examinations on exit from high school (A-Levels) and medical school (EPM). After adjusting for prior academic attainment, no statistically significant difference was found in the likelihood of success at MRCS between candidates with and without disabilities. In short, while candidates with disabilities had lower Part A pass rates as a cohort, they were no less likely to pass Part A at first-attempt than candidates without disabilities who had similar grades at high school and medical school. Additionally, no statistically significant relationship was found between disability and success at MRCS Part B.
Significant correlations were found between both school and medical school assessment and MRCS Part A scores. Candidates with disabilities performed worse than their peers in all of the assessments included in this study. According to government statistics, this attainment gap exists in all levels of education from early school written examinations to postgraduate assessments. 22 This suggests that the MRCS is no less fair than other tests and that systemic inequalities underpin this engrained attainment gap. However, further investigation is imperative to exclude discrimination in education and training as a cause of poorer MRCS examination performance by students with registered disabilities. There was no statistically significant difference in Part A performance between candidates with specific learning disabilities and those with other registered disabilities, with both cohorts performing worse than their peers without disabilities. This is important given that candidates with non-specific learning disabilities have been underrepresented in previous studies that have tended to focus on the majority; that is, candidates who report specific learning disabilities. We hope that our data prompt others to include this breadth of disabilities when looking for associations between attainment and disability in myriad markers of success. A better understanding of the attainment differences between groups of people reporting different disabilities could drive the appropriate redistribution of support and resources throughout school, university and postgraduate training.
Interestingly, no statistically significant difference was found in Part B pass rates between candidates with and without disabilities, suggesting that while candidates with disabilities may find written examinations more difficult, they subsequently perform as well as their peers in clinical examinations. This pattern merits further investigation such as comparing the impact of disability status on performance in other clinical undergraduate and postgraduate examinations.

Reasonable adjustments
Candidates with registered disabilities are often granted reasonable adjustments for examinations, typically including extra time in which to sit the examination. These adjustments have been found to equalise performance in undergraduate medical examinations. 3,[13][14][15] The Intercollegiate Committee for Basic Surgical Examinations (ICBSE) assesses MRCS reasonable adjustment requirements on a case-by-case basis. However, reasonable adjustments for examinations are a highly contested topic in education and there is a need to balance fairness and equity with exam validity in order to maintain testing accuracy. 23 Our data suggest that the current methods used for MRCS candidates are sufficient to enable performance unrestricted by disability to at least the same degree as methods used for the other examinations.

The need for inclusivity
These data further highlight some important issues within the workplace warranting further discussion by the medical community. Similar proportions of MRCS Part A and Part B cohorts (7% and 6%, respectively) had registered disabilities, but is this a true representation of all doctors? One study has estimated the prevalence of medical students with specific learning disabilities within a medical school as high as 10%. 24 Is this difference in prevalence between medical school and surgical training due to fewer doctors with disabilities choosing to pursue a career in surgery or are those in surgery not disclosing their disabilities? We know that many do fail to disclose their disability and access the support that they are entitled to including reasonable adjustments for examinations. 1,25 This is unsurprising given ongoing reports of bullying, undermining and harassment of clinicians with disabilities within the workplace, 8,25 causing many to worry that declaring a disability may impact their careers. 7 The British Medical Association recently created 'A charter for medical schools to prevent and address racial harassment'. 26 This charter highlights the systemic discrimination of underrepresented groups and provides guidance on enacting culture change within the workplace. The creation of similar charters for other protected characteristics such as disability and the addition of equality, diversity and inclusivity training at undergraduate and postgraduate levels could increase awareness of differential attainment, discrimination and bullying and would help to drive positive culture change. Similarly, greater appreciation for the unique skills, insight and perspectives that doctors with disabilities can bring to the surgical team 1,12,27 and the elimination of the current stigmatising and bullying culture could enable greater disability disclosure and access to increased support in the postgraduate environment.

Differential attainment
Group-level attainment differences seen in this study between socio demographic groups are similar to those seen in previous studies 17 and are not unique to the MRCS. 18,[28][29][30] White healthcare professionals with disabilities are known to feel more supported than their Black, Asian or Minority Ethnic counterparts 8 and lower pass rates were observed in nonwhite candidates with disabilities in this study in comparison to their peers. It is likely that the causes of differences in performance are likely to be complex and multifaceted, social and systemic. Further work is required to ensure that bias at the question level (item level bias or unfairness) is ruled out as a source of these group-level attainment gaps, using differential item functioning.
Many candidates with disabilities use computer technology as part of their adaptive techniques to optimise learning and practice. 3,6 It is possible that recent computerisation of the MRCS Part A examination will further benefit candidates with disabilities by aiding reading fluency and enabling the optimisation of font sizes, screen colour and exam accessibility. Future studies are planned to monitor the impact of these changes in order to ensure the provision of a fair and equitable examination for all candidates.

Strengths and weaknesses
Despite the strengths of large-scale retrospective cohort studies such as this for identifying grouplevel attainment differences, further work is needed to provide a more detailed look at individual-level performance. Disability and specific learning disabilities are broad terms and reported disabilities will range in type, severity and impact on activities of daily living and perhaps educational opportunities and experiences which cannot be represented in cohort level analyses. One of the strengths of the study was that no bias was apparent when the subgroup (with complete data) were examined. Selfreported ethnicity data were dichotomised into two discrete groups as has been used in other studies 18,20 to maximise statistical power for group comparisons, rather than this being an ethical or social decision. Larger cohort sizes would have enabled a more granular analysis. The study included only UK graduates to create a homogenous study cohort within one context with an overarching definition of disability and certain approaches supporting people with disability. While thousands of non-UK graduates attempt MRCS every year and further analysis of this population may ascertain the impact of disability status on the performance of the whole group, doing so is challenging. This is because countries have different legislation regarding disability which will impact the disclosure of disabilities and the support received. The small numbers of GAMSAT candidates with disabilities (n ¼ 30) make it difficult to draw substantive conclusions in respect of this admissions test and later performance. In terms of future research, due to comparatively low numbers of candidates with registered disabilities and low associated power, large group differences are required before statistical significance is reached.

Conclusion
Candidates with registered disabilities perform less well in formal, written examinations throughout education and postgraduate training. Our data indicate that doctors with disabilities are as likely to pass MRCS A at first attempt as their peers who achieved similar grades at high school and medical school. No statistically significant relationship was found between disability and success at MRCS Part B.
Further work is needed to investigate the causes of attainment differences in early career assessments, and to drive the appropriate distribution of support and resources to enable equity in career progression. Guarantor: RE is the study guarantor.
Contributorship: RE wrote the first draft of the manuscript and performed statistical analyses with AL's supervision. All authors contributed to, edited and reviewed the manuscript. JC led the study proposal for access to UKMED data. All authors approved the final draft of the manuscript.