External validation of two prediction tools for patients at risk for recurrent Clostridioides difficile infection

Background: One in four patients with primary Clostridioides difficile infection (CDI) develops recurrent CDI (rCDI). With every recurrence, the chance of a subsequent CDI episode increases. Early identification of patients at risk for rCDI might help doctors to guide treatment. The aim of this study was to externally validate published clinical prediction tools for rCDI. Methods: The validation cohort consisted of 129 patients, diagnosed with CDI between 2018 and 2020. rCDI risk scores were calculated for each individual patient in the validation cohort using the scoring tools described in the derivation studies. Per score value, we compared the average predicted risk of rCDI with the observed number of rCDI cases. Discrimination was assessed by calculating the area under the receiver operating characteristic curve (AUC). Results: Two prediction tools were selected for validation (Cobo 2018 and Larrainzar-Coghen 2016). The two derivation studies used different definitions for rCDI. Using Cobo’s definition, rCDI occurred in 34 patients (26%) of the validation cohort: using the definition of Larrainzar-Coghen, we observed 19 recurrences (15%). The performance of both prediction tools was poor when applied to our validation cohort. The estimated AUC was 0.43 [95% confidence interval (CI); 0.32–0.54] for Cobo’s tool and 0.42 (95% CI; 0.28–0.56) for Larrainzar-Coghen’s tool. Conclusion: Performance of both prediction tools was disappointing in the external validation cohort. Currently identified clinical risk factors may not be sufficient for accurate prediction of rCDI.


Introduction
By 1978 Clostridioides difficile was considered to be a causative agent of antibiotic-associated pseudomembranous colitis. 1 Nowadays we know this toxin-producing bacterium as the most common cause of healthcare-related diarrhea in the Western world. 2,3 Primary C. difficile infection (CDI) is treated with antibiotics, either vancomycin or metronidazole. 4 Despite adequate treatment, 15-25% of patients with CDI develop recurrent disease within 2 months. 5,6 With every recurrence, the risk of a new CDI recurrence increases: the chance of developing a second recurrence is estimated at 45% and the risk of a third recurrence at 65%. 7 The healthcare burden of recurrent CDI (rCDI) is substantial, since the 180-day mortality of patients with rCDI is 33% higher than that of patients with CDI without a recurrence. 8 Multiple recurrences of CDI are treated with a tapered and/or pulsed regimen of vancomycin, fidaxomicin, or fecal microbiota transplantation (FMT). 4 It is suggested that early treatment with fidaxomicin or FMT leads to lower recurrence rates. 9,10 Early identification of patients at risk for rCDI is crucial as it permits specific preventive measures and treatment to be tailored for these patients.
Various studies have identified risk factors for rCDI. The most important contributors seem to be: older age, concomitant use of non-CDI antibiotics, antacids or immunosuppressive medication, severe underlying disease, and multiple or prolonged hospitalizations. [11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26] Several models to predict rCDI have been developed. 13,16,20,[27][28][29][30][31][32] Unfortunately, none has gained clinical acceptance due to the limited number of patients on which they are based, insufficient performance, or lack of external validation. External validation of prediction tools for patients at risk of rCDI will give insight to the applicability of these tools in clinical practice and might contribute to better, personalized treatment for patients with CDI. Therefore, we aimed to search for prediction tools in the literature and to validate the most promising ones with a cohort of patients with CDI from six hospitals in The Netherlands.

Literature search
We performed a literature search in PubMed from database inception up to December 2019 (see Appendix). Only cohort studies with rCDI as an outcome measure that provided a practical scoring tool were selected. Prediction tools developed in a specific group of patients (e.g. trauma patients, ICU patients) or that used variables that were not available in our validation cohort were excluded. The study selection process was performed by two independent researchers and conflicts were handled by consensus.

Validation cohort
For the validation cohort we used the already existing database of patients participating in an ongoing multicenter, prospective cohort study on the occurrence of rCDI, the PREDICD study (ZonMw project number 848016009). The aim of the PREDICD study is to develop a prediction model for rCDI based on a combination of clinical risk factors and fecal microbiota analysis. All adults (⩾18 years old) diagnosed with primary CDI that were hospitalized or visited the outpatient clinic of one of the participating centers between 1 March 2018 and 6 March 2020 were eligible for inclusion. Participating centers were: Amsterdam UMC location VUmc, OLVG, Spaarne Gasthuis, Haaglanden Medisch Centrum, Flevoziekenhuis, and Noordwest Ziekenhuisgroep. Primary CDI was defined as: (a) presence of diarrhea (defined as ⩾3 unformed stools within 24 h for a minimum of 2 consecutive days); (b) microbiologically confirmed CDI [using the diagnostic algorithm of the participating centre; enzyme immune assay (EIA) for glutamate dehydrogenase and/or free C. difficile toxin A and/or B, culture of toxigenic C. difficile and/or polymerase chain reaction (PCR) for detection of toxin A and/or B genes]; (c) treatment with metronidazole or vancomycin. Exclusion criteria were CDI in the preceding 3 months, microbiologically proven infectious colitis (other than CDI) in the last month, and ileostomy. This study was approved by the Medical Ethical Committee of Amsterdam UMC, location VUmc (approval number 2015.299). Written informed consent was obtained from all participants.

Data collection
Data on patient characteristics and predicting variables used in the different prediction tools were collected prospectively by (telephone) interviews, and verified and completed with electronic patient healthcare records by a small group of trained researchers. Data were captured in Castor (Castor EDC, Amsterdam, the Netherlands), a secure and GCP-compliant (FDA 21 CFR Part 11, ICH E6 Good Clinical Practice, HIPAA and GDPR) clinical data management platform. Follow-up duration was 8 weeks, starting from the first day of treatment for primary CDI. Participants were contacted by telephone at scheduled time points (5,10,14,28, and 56 days after treatment initiation). During these telephone consultations recovery and probable recurrence were evaluated. If patients were still hospitalized during the followup period, data were extracted from patient records. Participants were asked to contact the study coordinator if they developed diarrhea in between the scheduled time points.

Statistical analysis
To evaluate the predictive value of the variables of the selected prediction tools in our validation cohort, we calculated the odds ratios (ORs) for these variables in the validation cohort with multivariable logistic regression analysis using the same multivariable models as used in the original studies. In addition, an rCDI risk score was calculated for each individual patient in the validation cohort using the scoring tool described in the derivation studies. Per score value, we compared journals.sagepub.com/home/tag 3 the average predicted risk of rCDI with the observed number of rCDI cases. To quantify the ability of the prediction tools to differentiate between patients with and without rCDI (discrimination), we estimated the area under the receiver operating characteristic (ROC) curve (AUC), which ranges from 0.5 (no discrimination) to 1 (perfect discrimination). To quantify how close the calculated probabilities for rCDI were to the actual risk for rCDI, we plotted the observed number of rCDI cases versus the predicted number of rCDI cases (calibration plot).

Selection description of prediction tools
From the literature search, 54 studies were identified by title and abstract screening. A total of 10 articles were selected for full-text review of which 2 were excluded since they did not include scoring tools and were therefore not suitable for validation analysis. 13,30 We identified eight articles with a scoring tool for rCDI. 16,20,[27][28][29][31][32][33] Two articles reported the same prediction tool, therefore, only the original study was included. 16,33 The Appendix shows the predictors identified in these seven studies. Older age was identified as a predictor for rCDI in 5/7 published prediction tools, while the other 26 predictors were used in only 1 or 2 prediction tools. Of the seven remaining studies, four were excluded because they used predictors that were not available in our cohort (i.e. Horn index score, use of antidiarrheals, fidaxomicin as therapy for CDI and abdominal distension). 16,20,29,32 The prediction tool of Eyre et al. was excluded because it aimed to predict rCDI within 4 months of CDI diagnosis, instead of 2 months as used in our validation cohort. 28 Eventually, two prediction tools were selected for validation analysis: the tool of Cobo et al. and that of Larrainzar-Coghen et al. 27,31 Selected prediction models In the study of Cobo et al., rCDI was defined as: (a) ⩾3 loose stools in 24 h or ileus or pseudomembranous colitis; (b) positive free toxin testing of stool (EIA) or nucleic acid amplification test for toxins (also called PCR) or culture of toxigenic C. difficile within 2 months after the completion of treatment for CDI. If a stool sample had not been sent to the laboratory for microbiological diagnostic confirmation, the reappearance of symptoms suggestive of rCDI that resolved with vancomycin or metronidazole treatment was also considered as rCDI. If a stool sample was negative for C. difficile despite response to treatment, the reappearance of diarrhea was not considered as rCDI. 27 31 Larrainzar-Coghen et al. included four variables in their prediction tool: age (>65 years versus ⩽65 years), blood leukocyte count on the day of CDI diagnosis (⩽30× 10 9 /L versus >30 × 10 9 /L), enteral feeding 1 month preceding CDI diagnosis, and continuing proton pump inhibitor (PPI) treatment following CDI diagnosis. 31 All variables were assigned 1 point for increased risk for rCDI implying a possible minimal score of 0 and a maximum of 4 points. Based on total points, two risk categories were defined: low risk (0-1 point) and high risk (2-4 points). 31

Missing data
The only missing data in the validation cohort were on blood leukocyte count (14 patients), 1 of the variables of Larrainzar-Coghen's tool. Since we assumed that patients with severe leukocytosis would be seriously ill, and that their physicians would monitor their blood leukocyte count at least once every 3 days (we used a range of 3 days for measuring this value 'at baseline' in our validation cohort), we scored the 14 missing values as ⩽30 × 10 9 leukocytes/L.

Study and patient characteristics
Study and patient characteristics of the derivation studies and our validation cohort are shown in   27 We calculated the ORs of Cobo's predictors for rCDI in the validation cohort by using multivariable logistic regression analysis ( Table 2). None of Cobo's predictors was associated with rCDI in our validation cohort. Thereafter, we calculated rCDI risk scores for each individual patient in the validation cohort. Per score value, we compared the average predicted risk of rCDI with the observed number of rCDI cases (Figure 1(a) and Appendix). In the validation cohort, a higher score corresponded to a lower risk of rCDI; the highest risk was observed for the patients with a score of 0 (predicted as low risk Prediction tool of Larrainzar-Coghen et al. 31 The ORs of the predictors of Larrainzar-Coghen et al. 31 for rCDI in the validation cohort are shown in Table 2. Enteral feeding was not significantly associated with rCDI in our validation cohort. Only two patients in our cohort had a blood leukocyte count of >30 × 10 9 /L. Age >65 years and continuing PPI treatment were associated with the absence of rCDI with ORs of 0.22 (95% CI; 0.09-0.52) and 0.36 (95% CI; 0.16-0.82), respectively. Also for this prediction tool, the average predicted risk of rCDI per score value did not correspond well with the observed number of rCDI cases (Figure 1(b) and Appendix). In line, discrimination was poor with an estimated AUC of 0.42 (95% CI; 0.28-0.56). This finding is in contrast with the performance of the prediction tool in the original derivation cohort in which an AUC of 0.67 (95% CI; 0.59-0.75) was estimated.
Since the prediction tool of Larrainzar-Coghen et al. 31 was developed in a cohort of hospitalized patients only, we also performed a validation restricted to the hospitalized patients of our validation cohort (n = 113). This did not influence the ORs of the predictors of the Larrainzar-Coghen et al. 31 model in our cohort, neither did it substantially influence the discriminative performance of the model (AUC = 0.47: 95% CI; 0.31-0.62).
The calibration curves confirm the poor performance of both tools for predicting rCDI in the validation cohort (see Appendix).

Discussion
This study aimed to externally validate two existing prediction tools for rCDI. The tools of Cobo et al. 27 and Larrainzar-Coghen et al. 31 performed poorly in our validation cohort with estimated AUCs of 0.43 (95% CI; 0.32-0.54) and 0.42 (95% CI; 0.28-0.56), respectively. Remarkably, ROC and calibration plots of both prediction models showed a negative correlation: lower predicted probabilities for rCDI correlated with higher observed risks for rCDI, whereas higher predicted probabilities correlated with lower actual risks for rCDI. This suggests that, despite the similarities in study settings, these prediction tools are not sufficient for accurate prediction of rCDI in the general population.
The drawback of most prediction tools for rCDI is the lack of external validation. 20,28,29,31,32 To the best of our knowledge, our study is the first in which prediction tools for rCDI were validated in a setting completely independent from the setting in which the tools were developed. This might be a reason for the poor performance of these tools in our population. In only 2/7 published prediction tools for rCDI, namely those of Cobo et al. 27 and Hu et al., 16 was an 'external' validation performed. Both tools discriminated well between patients with and without rCDI in their own validation cohorts with AUCs of 0.75 (95% CI; 0.67-0.83) and 0.80 (95% CI; 0.67-0.92). However, in both studies the validation cohorts were highly similar to the derivation cohorts, because they were largely chosen from the same source population.
To determine the true robustness of a prediction model, derivation and validation cohorts should be derived from different populations.
That different study settings lead to different rCDI predictors is nicely illustrated in the Appendix: most predictors are 'unique' and included in only one or two prediction tools. This can be partially explained by the fact that not all studies collected the same variables. However, data on the 'usual suspects', such as antibiotic and PPI use, signs and symptoms of severe CDI, and immune status, were collected in the majority of these studies but generally not identified as predictors in multivariable analysis. This high variation in rCDI predictors might reflect the heterogeneity of the patient population and study designs, and could be an explanation for the low generalizability of these tools in other populations.
To explain the poor performance of the prediction tools in our validation cohort, we compared the study and patient characteristics of both derivation cohorts and the validation cohort ( year' as a predictor of rCDI. In the PREDICD study we excluded patients with CDI in the preceding 3 months. This might be the reason that Cobo's cohort comprised more patients with a CDI episode in the last year (11%) than in the validation cohort (2%), and is a limitation of our study. Since rCDI is a major risk factor for a subsequent recurrence, this might explain why 'CDI in the last year' was identified as a risk factor in Cobo's cohort but not in our population.
In the cohort of Larrainzar-Coghen et al., 31 older age and PPI continuation after CDI diagnosis were risk factors for rCDI. In our cohort these variables were inversely associated with rCDI. This is remarkable since older age is identified as risk factors for rCDI in many previous studies. 16,28,29 The literature on the association between PPI use and rCDI is less consistent. 12,23,32,34,35 Considering age, Larrainzar-Coghen et al. 31 dichotomized the variable age into ⩽65 years and >65 years of age. We scrutinized the continuous values and observed that this cut-off was quite arbitrary in our cohort: for example, when the cutoff value for age would have been >60 years old instead of >65 years, the patients with rCDI categorized as 'older' would shift from 42% to 68% and 'older' age would have been a (positive) predictor for rCDI. Therefore, we suggest the use of continuous values or multiple age categories in future prediction tools for more accurate and individualized prediction of rCDI risk.
A difference between our study and that of Larrainzar-Coghen et al. 31 is that we also included patients that visited the outpatient clinic (n = 16). Despite our expectations, rCDI occurred more frequently in outpatients (25%) than in hospitalized patients (14%). This could be a result of our active, prospective approach: patients with mild, possibly self-limiting complaints might have consulted a doctor more frequently due to our telephone consultations than they would have in a normal situation. Another explanation might be that spores are difficult to eliminate and re-exposure to spores in the home environment may be a source for relapse. However, when we performed a validation analysis restricted to the hospitalized patients of the validation cohort (n = 113), this did not increase the performance of the prediction tool.
Besides the clinical features used in prediction tools for rCDI, other variables may be predictive for rCDI. The development of antitoxin antibodies seems to be an important factor for disease resolution and the prevention of rCDI. 36,37 Furthermore, it is known that toxin production, sporulation, persistence in the host and spore germination are elevated in several hypervirulent strains such as 027/NAP1 and 078 and may influence the risk of rCDI. [38][39][40][41] Since changes in gut microbiota composition play an important role in the pathogenesis of (recurrent) CDI, Khanna et al. developed a microbiota-based risk score for rCDI that showed promising results. 42 Because many clinical factors (such as age and medication use) have an effect on the diversity of the gut microbiota and therefore on the risk of rCDI, incorporation of microbiota-related risk factors in prediction tools could lead to a more direct and accurate prediction of rCDI. We hope to confirm this hypothesis in the near future with the results of the PREDICD study. Another interesting predictor might be the virome, however, this is not yet generally considered in microbiota studies.
One of the strengths of this study is the prospective data collection by both telephone interviews and electronic health records, resulting in a few missing data. In addition, all patients in our cohort had symptomatic CDI, therefore, the risk of including patients with C. difficile colonization instead of infection was low. A limitation of our study was the relatively small sample size of 129 patients and the lack of a sample size calculation due to the use of a 'convenience sample' consisting of patients in the PREDICD study cohort. Another limitation is that we were able to validate only two of the seven prediction tools found via the literature search, mainly because they used predictors that were nonquantitative and/or variables that we did not collect for the patients in our cohort.
In conclusion, our results show poor performance of two practical prediction tools for rCDI. Accurately predicting recurrent disease remains a challenge. Possibly, prediction models with more parameters, such as microbiota composition at time of CDI diagnosis, are needed for better prediction of rCDI.
Author contributions TMR conceived the study. TMR, YHB, OMD, and CMJEV-G designed the study. TMR and LJD acquired the data. LJD, TMR, and MWH analyzed the data. LJD and TMR drafted the manuscript. All authors critically revised the manuscript and approved the final version.