Patient Acceptable Symptom State Thresholds for IKDC-SKF and KOOS at the 10-Year Follow-up After Anterior Cruciate Ligament Injury: A Study From the Delaware-Oslo ACL Cohort

Background: Clinicians need thresholds for the Patient Acceptable Symptom State (PASS) and Treatment Failure to interpret group-based patient-reported outcome measures after anterior cruciate ligament (ACL) injury. Validated thresholds that are crucial for accurately discerning patient symptom state and facilitating effective interpretation have not been determined for long-term follow-up after ACL injury. Purpose: To calculate and validate thresholds for PASS and Treatment Failure for the International Knee Documentation Committee Subjective Knee Form (IKDC-SKF) and the Knee injury and Osteoarthritis Outcome Score (KOOS) subscales at the 10-year follow-up after ACL injury. Study Design: Cohort study; Level of evidence, 3. Methods: A total of 163 participants with unilateral ACL injury (treated with reconstruction or rehabilitation alone) from the Delaware-Oslo ACL Cohort were included. Thresholds for PASS were calculated for IKDC-SKF and KOOS subscales using anchor-based predictive modeling and receiver operating characteristic (ROC) analysis. Too few participants had self-reported Treatment Failure to calculate thresholds for that outcome. Nonparametric bootstrapping was used to derive 95% CIs. The criterion validity of the predictive modeling and ROC-derived thresholds were assessed by comparing actual patient-reported PASS outcome with the calculated PASS outcome for each method of calculation and calculating their positive and negative predictive values with respect to the anchor questions. Results: A total of 127 (78%) participants reported satisfactory symptom state. Predictive modeling PASS thresholds (95% CIs) were 76.2 points (72.1-79.4 points) for IKDC-SKF, 85.4 points (80.9-89.2 points) for KOOS Pain, 76.5 points (67.8-84.7 points) for KOOS Symptoms, 93.8 points (90.1-96.9 points) for KOOS activities of daily living, 71.6 points (63.4-77.7 points) for KOOS Sports, and 59.0 points (53.7-63.9 points) for KOOS quality of life (QoL). Predictive modeling thresholds classified 81% to 93% of the participants as having satisfactory symptom state, whereas ROC-derived thresholds classified >50% as unsatisfied. The thresholds for IKDC-SKF, KOOS Sports, and KOOS QoL resulted in the most accurate percentages of PASS among all identified thresholds and therefore demonstrate the highest validity. Conclusion: Predictive modeling provided valid PASS thresholds for IKDC-SKF and KOOS at the 10-year follow-up after ACL injury. The thresholds for IKDC-SKF, KOOS Sports, and KOOS QoL should be used when determining satisfactory outcomes. ROC-derived thresholds result in substantial misclassification rates of the participants who reported satisfactory symptom state.

Patient-reported outcome measures (PROMs) are commonly used to evaluate patient-perceived knee symptoms and function after anterior cruciate ligament (ACL) injury. 9,52,53However, clinicians and policymakers struggle to interpret the outcomes of PROMs on scales from 0 to 100 -what is regarded as a satisfactory outcome for a patient? 40Knowing whether patients are satisfied with their current knee status is of greater clinical significance than understanding the value of a PROM. 5,30onverting a PROM value into a meaningful measure of satisfactory outcomes helps to better interpret outcomes of PROMs at a group level, as in national ACL registries, large cohorts, and clinical trials. 4,7,37,46,51Such measures can inform clinicians and guide patient counseling by setting realistic expectations.Hence, the Patient Acceptable Symptom State (PASS) and Treatment Failure were developed. 24PASS is a simple and clinically relevant measure to judge a symptom state that the patient describes as satisfactory. 11,47In other words, PASS refers to a state beyond which patients can be expected to feel good at a given timepoint. 40Treatment Failure identifies patients who find their treatment has failed them.Thresholds reflecting PASS or Treatment Failure can be calculated and used for a given PROM and population to identify what constitutes a satisfactory or failed outcome for patients.Currently, valid thresholds are lacking to interpret the long-term outcomes of commonly used PROMs after ACL injury. 30nsuring the validity of thresholds is crucial to accurately discern patient knee status and facilitate effective interpretation of the PROMs. 32Various methods are used to derive thresholds. 14,16,29PASS and Treatment Failure thresholds are best calculated by linking the PROM score to an anchor question, 47 with receiver operating characteristic (ROC) analysis being traditionally used and predictive modeling being recently recommended as they provide more accurate thresholds. 43,45Nevertheless, we do not know how accurately these thresholds reflect patient perception of satisfactory symptom state.
Accordingly, we aimed to calculate PASS and Treatment Failure thresholds for long-term outcomes after an ACL injury and to validate their performance in identifying the satisfactory symptom state or failed treatment.We hypothesized that predictive modeling would yield more accurate PASS thresholds than ROC analysis.

METHODS
This was a prospective cohort study from the prospective Delaware-Oslo ACL Cohort.The outcomes of interest were the International Knee Documentation Committee Subjective Knee Form (IKDC-SKF) and the Knee injury and Osteoarthritis Outcome Score (KOOS) questionnaires, 6,26,27,34,41,42 as well as single-item questions on PASS and Treatment Failure. 24The University of Delaware-Oslo ACL and the Regional Ethical Committee for South-Eastern Norway approved this study, and all participants provided written informed consent before data collection.

Participants and ACL Treatment Algorithm
A total of 300 participants were enrolled between 2006 and 2012 in the University of Delaware, Newark, Delaware, United States, and the Norwegian Sports Medicine Clinic, Oslo, Norway -150 from each site. 36The injury diagnosis was based on magnetic resonance imaging and anterior knee joint laxity measurement using the KT-1000 arthrometer (MEDmetric). 8Inclusion criteria were age between 13 and 60 years old and participation in pivoting sports (level 1 or 2) at least twice a week at the time of injury. 20,23We excluded participants with bilateral injuries, symptomatic concomitant injuries, or previous knee injuries.Among the 300 included participants, 24 had had a previous ACL reconstruction (ACLR) and came to the consultation with a graft rupture; they were excluded from the present study.Finally, 276 participants with a primary ACL injury were included at baseline.
After resolution of acute impairments at inclusion, all participants received 10 sessions of preoperative rehabilitation as described previously. 12Participants were educated about treatment alternatives, underwent functional testing, and then decided on their treatment strategy in a shared decision-making process together with their orthopaedic surgeon and physical therapist.ACLR was more likely recommended to patients experiencing dynamic knee instability after preoperative rehabilitation or planning to return to level 1 sports. 20,23,54The reason for pursuing rehabilitation alone was based mainly on achieving good knee function after rehabilitation.Delayed ACLR was performed for patients who experienced dynamic knee instability or if they changed their minds about the treatment choice. 54Further information about the treatment algorithm and rehabilitation program have been described in previous work. 19,20en years after completion of preoperative rehabilitation or after ACLR, 5 participants withdrew from the study, 11 declined to attend, 64 could not be contacted, 3 declined due to medical reasons, 3 had technical problems with the electronic questionnaire, and 27 had no reason listed.Finally, 163 (59%) participants were included in the analysis of this study (Figure 1).The rationale to include the whole cohort (including participants with different treatments) was based on previous findings from both the 5-year and 10-year follow-ups, which demonstrated that participants who followed rehabilitation alone had similar outcomes compared with participants treated with ACLR. 35,36,48

Data Collection
Questionnaires were sent to participants for the 10-year follow-up.The questionnaire included the IKDC-SKF, KOOS, PASS, and Treatment Failure anchor questions.The IKDC-SKF is scored from 0 (worst) to 100 (best). 26he IKDC-SKF has good test-retest reliability, 49 construct validity, 1,27 and responsiveness in participants after ACL injury. 27The KOOS comprises 5 separate subscales: pain, symptoms, activities of daily living (ADL), function in sport and recreation (Sports), and knee-related quality of life (QoL). 41All subscale scores were calculated using KOOS guidelines and scores ranged from 0 (worst/extreme problems) to 100 (best/no problems). 39The KOOS has good internal consistency, test-retest reliability, construct validity, and responsiveness in participants after ACL injuries and reconstruction. 6,15,28To identify PASS, we asked a first single-item anchor question 24 : ''Considering your knee function, do you feel that your current state is satisfactory?With knee function, you should take into account all activities during your daily life, sport and recreational activities, your level of pain and other symptoms, and also your knee-related quality of life.''Answer options were dichotomized as ''yes'' or ''no.''Participants answering ''no'' were asked a second anchor question related to Treatment Failure 24 : ''Would you consider your current state as being so unsatisfactory that you think the treatment has failed?''Answer options were again dichotomized as ''yes'' or ''no.''Similar PASS questions have good test-retest reliability in this patient population (k = 0.78). 33

Statistical Methods
Individual characteristics at the 10-year follow-up were presented as means and standard deviations for continuous variables and percentage distribution for categorical variables.Normality was assessed using Shapiro-Wilk tests and Q-Q plots, 50 and we considered data to be normally distributed and suitable for use with parametric tests.Group differences were assessed using Student t tests for continuous variables and with chi-square analysis of proportions or Fisher exact tests for categorical variables.Participants with missing data for the PASS question were excluded from our analysis; participants with missing data and participants with complete data were compared for sex, age, body mass index, months from injury to surgery (for ACLR only), and preoperative IKDC-SKF and KOOS scores.
Participants answering ''yes'' to the first PASS question were classified as having satisfactory symptom state.Those answering ''no'' were classified as having unsatisfactory symptom state.Participants answering ''no'' to the first question and then ''yes'' to the second question were classified as Treatment Failure.
Two modeling approaches were used to derive thresholds by anchoring IKDC-SKF and KOOS scores to the anchor questions.First, we calculated the commonly used ROC-derived thresholds using ROC analysis (no adjustment for proportion of reported answer choices or anchor question reliability). 45The optimal cutoff values were determined using the Youden index. 55Second, we calculated predictive modeling thresholds using adjusted predictive modeling, 43,44 involving logistic regression using the dichotomous anchor answer as the dependent variable and the PROM scores as the independent variable.Bias occurs when the reported proportion of satisfactory symptom state differs from 50%, 43 resulting in overestimated (or underestimated) thresholds if the proportion is .50% (or \ 50%).This effect is amplified by poor reliability of the anchor question.The item-score reliability of the anchor question was estimated by using confirmatory factor analysis (CFA). 10,18,44We adjusted the thresholds for the proportion of reported satisfactory symptom state and anchor question reliability using the following equation 44 : predictive modeling thresholds = PASS predictive -(0.08/RelPASS -0.5) 3 SD score 3 Cor) 3 log-odds(PASS).In this equation, Rel PASS is the anchor question reliability, SD score is the standard deviation of the PROM scores, Cor is the Pearson point-biserial correlation between the PROM scores and the anchor, and log-odds(PASS) is the natural logarithm of proportion having satisfactory symptom state divided by 1 minus the proportion having no satisfactory symptom state.Point-biserial correlation between the dichotomized anchors and the PROM scores were calculated as simple measures of anchor validity.We had no predefined correlation criterion due to inconsistency in literature.We used bootstrapping of 1000 random replications to obtain 95% CIs reported as 0.025 to 0.975 quantiles.
To assess criterion validity, we calculated the percentages of participants whose outcome scores exceeded these thresholds and were classified as PASS and compared them with the actual true PASS percentage based on the answer to the anchor question.In addition, we used a contingency table to calculate the sensitivity, specificity, positive predictive value, and negative predictive value of the predictive modeling thresholds and the ROC-derived thresholds for each PROM and subscale.Results are shown for the whole cohort, patients who underwent either ACLR or rehabilitation alone.In a subanalysis, we calculated thresholds for participants who underwent ACLR.All analyses were performed in R Version 4.2.3 (www.r-project.org). 38

Participants
A total of 116 participants underwent ACLR (92 underwent early ACLR 6 months after the 10-session rehabilitation program, 24 underwent delayed ACLR .6 months after the 10-session rehabilitation program), 46 followed rehabilitation alone, and 1 treatment status was unconfirmed (Table 1).The median (Q1-Q3) time from injury to surgery was 4.9 months (3.3-8.6 months); hamstring tendon autografts were used in 63 participants, patella tendon autografts were used in 32 participants, allografts were used in 19 participants, and the graft type remained unknown for 2 participants.Participants with missing data had higher body mass index (mean difference, 2.3 kg/m 2 ) at inclusion compared with those with complete data (Supplemental Table S1).

PASS and No PASS, Based on the Anchor Questions
The percentages across the whole cohort were 78% (n = 127) for PASS and 22% (n = 36) for no PASS.Only 4 participants (2.5%) reported treatment failure, while another 4 participants (2.5%) did not answer the Treatment Failure question.

Threshold Values for IKDC-SKF and KOOS Subscales
PASS thresholds were calculated with the predictive modeling and ROC analysis (Table 2).The 95% CIs indicated that predictive modeling thresholds were different from ROC analysis thresholds, with predictive modeling thresholds being lower by at least 10 points except for KOOS ADL.The point-biserial correlations between the dichotomized PASS anchor and the PROM scores were lowest (0.29) for KOOS Symptoms and highest (0.58) for KOOS QoL.We calculated the optimized CFA model fit and the estimated anchor question reliability for predictive modeling (Supplemental Table S2).Thresholds for participants who underwent ACLR were close to those from the whole cohort (results not presented).No thresholds were calculated for Treatment Failure due to the small number of participants reporting failed treatment (n = 4).

Criterion Validity of the Thresholds
The PASS thresholds derived from predictive modeling classified 81% to 93% of the participants as having PASS, whereas the ROC-derived PASS thresholds resulted in 48% to 71% (Figure 2).The difference in percentage (percentage points [pp]) to the true PASS percentage (78%) was smallest for predictive modeling thresholds in KOOS QoL (3 pp higher), KOOS Sports (5 pp higher), and IKDC-SKF (7 pp higher) (Figure 2).The ROC-derived thresholds were between 7 pp (KOOS Sports) and 30 pp (KOOS Symptoms) lower than the true PASS (Figure 2).Table 3 presents the false positives, false negatives, sensitivity, specificity, positive predictive value, and negative predictive value of the 2 methods of calculating PASS thresholds for each PROM and subscale.Of the participants who reported no PASS, predictive modeling thresholds correctly classified 73% to 86%, while ROC-derived thresholds correctly classified 40% to 61%.The predictive modeling thresholds for KOOS QoL had the highest combined positive and negative predictive values.

DISCUSSION
Our study revealed that predictive modeling PASS thresholds accurately reflected patient satisfaction, providing clinicians with a better understanding of group-based satisfactory outcomes at a 10-year follow-up after ACL injury.Predictive modeling yielded valid thresholds, with IKDC-SKF having a threshold of 76.2 points and KOOS subscales thresholds ranging from 59.0 to 93.8 points.Overall, 78% of participants reported PASS.While predictive modeling thresholds classified 81% to 93% of the participants as having satisfactory symptom state, ROCderived thresholds classified more than 50% as unsatisfied.Among the thresholds identified, IKDC-SKF, KOOS Sports, and QoL resulted in the most accurate classification of participants with satisfactory symptom state.

PASS Thresholds for IKDC-SKF and KOOS Subscales
To our knowledge, this is the first study to identify valid PASS thresholds for the IKDC-SKF and the KOOS subscales at a 10-year follow-up after ACL injury. 2,16,30Predictive modeling thresholds for IKDC-SKF and 4 KOOS subscales were considerably lower than the thresholds derived from ROC analysis, with a difference of .10points.While no other study has calculated PASS thresholds for long-term outcomes after ACL injury, 30 2 studies have proposed thresholds for short-term IKDC-SKF and KOOS using the ROC method: the proposed thresholds were lower than our predictive modeling thresholds, by up to 9 points for KOOS QoL at 6 to 24 months post-ACLR and up to 19 points for KOOS Symptoms at 1 to 5 years post-ACLR. 3,33The difference was amplified when comparing both studies' findings with our ROC-derived thresholds, with up to 22 and 38 points lower thresholds, respectively. 3,33Since thresholds are specific to the study population, the difference may be attributed to various factors of the included cohort (eg, demographics, treatment), especially the different follow-up time. 16We therefore recommend using the thresholds established in the present study in a population similar to the present study sample.

Criterion Validity of the PASS Thresholds
Predictive modeling thresholds slightly overestimated the proportions of satisfactory outcomes; the difference in

The Orthopaedic Journal of Sports Medicine
Thresholds for Satisfactory Symptom State After ACL Injury 5 percentage to the true PASS percentage was only 3 pp for KOOS QoL, 5 pp for KOOS Sports, and 7 pp for IKDC-SKF.Importantly, ROC-derived thresholds substantially underestimated the true PASS percentage of 78%, notably in KOOS Symptoms (30 pp), KOOS Pain (25 pp), and IKDC-SKF (23  pp).This finding illustrates how thresholds with poor validity can skew the clinical interpretation of outcomes.The good criterion validity of predictive modeling thresholds is attributed to the reduced bias caused by adjusting for the proportion of reported answer choices for PASS and anchor question reliability, which is particularly important when this proportion deviates from 50%. 43n contrast, ROC-derived thresholds performed poorly in classifying participants with unsatisfactory symptom state: up to 60% of participants classified as unsatisfied did in fact report true PASS, which is a substantial underestimation of satisfactory outcomes.We provided the thresholds from ROC analysis for those interested in comparison of methodologies but advise against applying them for clinical or research purposes.

Applicability of PASS Thresholds
PASS is a unidimensional measure of patient satisfaction, whereas the IKDC-SKF and KOOS assess multiple dimensions of knee symptoms and function, making it challenging to determine a single, concise threshold that indicates patients' satisfaction with their knee status.We found that predictive modeling PASS thresholds for the KOOS QoL, KOOS Sports, and IKDC-SKF resulted in percentages of participants with satisfactory symptom state that came closest to the true PASS percentage.Both KOOS subscales and the IKDC-SKF also had the strongest correlation with the PASS anchor.These findings suggest that, for KOOS, the Sports and QoL subscales capture the concerns most relevant to patients with an ACL injury. 21,25Therefore, and in line with previous recommendations from a systematic review of the psychometric properties of KOOS, 6 clinicians should prioritize these subscales when determining symptom state in participants with ACL injuries.We also recommend incorporating IKDC-SKF for determining symptom state given its high accuracy and strong correlation with the PASS anchor.
PASS thresholds are probabilistic values applicable in a population similar to our study sample.The thresholds established in this study serve as group-based reference values for estimating satisfactory symptom state, notably for ACL registries or cohorts with 10-year follow-up.However, clinicians should be aware that individual thresholds might differ from group means, as patient perceptions about satisfactory symptom state vary.In such cases, using the confidence intervals from predictive modeling proves valuable.In addition, the PASS might be influenced by patient age and change in lifestyle due to age and expectations. 31Our findings have high external validity, including participants from different geographical regions (United States and Scandinavia) who were active in jumping, pivoting, or cutting sports preinjury and received treatment that aligned with current clinical recommendations. 13revious studies that assessed group-based satisfactory symptom state at 10-year follow-up had to rely on ROCderived thresholds that were calculated for a sample of patients who were between 1 and 5 years after ACLR. 4,17,22,30,33,37,46,51 hile these thresholds may have limited validity for their given PROM and sample, they were also between 6 points higher to 19 points lower (KOOS Symptoms) when compared with our predictive modeling thresholds.For effective interpretation, we therefore recommend future studies use thresholds that are derived from predictive modeling and specific to patient characteristics, notably treatment and follow-up period. 16mitations A limitation of this study is the loss to follow-up rate of 41%, which may introduce selection bias.Participants who did not return for the 10-year follow-up had higher baseline body mass index than those who did, possibly influencing the generalizability of the findings.Another shortcoming is that the ceiling effect of the PROM scores might underestimate the thresholds.The adjustment of the predictive modeling method requires using CFA to estimate the reliability of the anchor question.Model fit for some subscales was suboptimal due to low anchor question reliability. 43When the proportion of reported answer choices lies outside 30% and 70%, as in the present study population, some residual bias will remain depending on the reliability of the anchor question; 43 this might lead to overestimated thresholds.

CONCLUSION
This study provides PASS thresholds that accurately reflect patient satisfaction on IKDC-SKF and KOOS at the 10-year follow-up after ACL injury.These thresholds can help clinicians and policymakers to interpret longterm satisfactory symptom state at a group level, such as in national ACL registries, large cohorts, or clinical trials.Predictive modeling thresholds were valid for detecting satisfactory symptom state, while ROC-derived thresholds misclassified a substantial percentage of participants with satisfactory symptom state as unsatisfactory.When determining group-based symptom state after ACL injuries, clinicians should prioritize the predictive modeling thresholds for IKDC-SKF, KOOS Sports, and KOOS QoL.

TABLE 1 Descriptive
Characteristics Overall at the 10-Year Follow-up After ACL Injury (N = 163) a ADL, activities of daily living; IKDC-SKF, International Documentation Committee Subjective Knee Form; KOOS, Knee injury and Osteoarthritis Outcome Score; PASS, Patient Acceptable Symptom State; QoL, quality of life.
a ACLR, anterior cruciate ligament reconstruction; b n (%) or mean (SD).cOf the 36 participants, 4 did not answer this question.

TABLE 2 PASS
Thresholds Calculated With Predictive Modeling and ROC Analysis a Correlation coefficients with the PASS question and PASS threshold values (95% CIs) calculated with predictive modeling and ROC for IKDC-SKF and KOOS subscales at the 10-year follow-up after ACL injury.ACL, anterior cruciate ligament; ADL, activities of daily living; IKDC-SKF, International Documentation Committee Subjective Knee Form; KOOS, Knee injury and Osteoarthritis Outcome Score; PASS, Patient Acceptable Symptom State; QoL, quality of life; ROC, receiver operating characteristic. a Percentages with 95% CIs of participants with IKDC-SKF and KOOS scores that were classified as PASS based on thresholds derived from predictive modeling (dark gray) or ROC (light gray).The horizontal line represents the true PASS percentage based on the anchor question (78%).ADL, activities of daily living; IKDC-SKF, International Documentation Committee Subjective Knee Form; KOOS, Knee injury and Osteoarthritis Outcome Score; PASS, Patient Acceptable Symptom State; QoL, quality of life; ROC, receiver operating characteristic.

TABLE 3 Predictive
Modeling and ROC-Derived PASS Thresholds a False positives (above PASS thresholds), false negatives (below PASS thresholds), sensitivity, specificity, positive predictive value, and negative predictive value of PASS thresholds calculated with predictive modeling and receiver operating characteristic (ROC) for detecting true PASS at the 10-year follow-up after ACL injury.ADL, activities of daily living; IKDC-SKF, International Documentation Committee Subjective Knee Form; KOOS, Knee injury and Osteoarthritis Outcome Score; NPV, negative predictive value; PASS, Patient Acceptable Symptom State; PPV, positive predictive value; QoL, quality of life; ROC, receiver operating characteristic. a