Measuring the impact of multiple sclerosis: Enhancing the measurement performance of the Multiple Sclerosis Impact Scale (MSIS-29) using Rasch Measurement Theory (RMT)

Background Study objectives were to evaluate the Multiple Sclerosis Impact Scale (MSIS-29) and explore an optimized scoring structure based on empirical post-hoc analyses of data from the Phase III ADVANCE clinical trial. Methods ADVANCE MSIS-29 data from six time-points were analyzed in a sample of patients with relapsing–remitting multiple sclerosis (RRMS). Rasch Measurement Theory (RMT) analysis was undertaken to examine three broad areas: sample-to-scale targeting, measurement scale properties, and sample measurement validity. Interpretation of results led to an alternative MSIS-29 scoring structure, further evaluated alongside responsiveness of the original and revised scales at Week 48. Results RMT analysis provided mixed evidence for Physical and Psychological Impact scales that were sub-optimally targeted at the lower functioning end of the scales. Their conceptual basis could also stand to improve based on item fit results. The revised MSIS-29 rescored scales improved but did not resolve the measurement scale properties and targeting of the MSIS-29. In two out of three revised scales, responsiveness analysis indicated strengthened ability to detect change. Conclusion The revised MSIS-29 provides an initial evidence-based improved patient-reported outcome (PRO) instrument for evaluating the impact of MS. Revised scoring improves conceptual clarity and interpretation of scores by refining scale structure to include Symptoms, Psychological Impact, and General Limitations. Clinical trial ADVANCE (ClinicalTrials.gov identifier NCT00906399).


Introduction
Growing interest in patient experiences of conditions like relapsingremitting multiple sclerosis (RRMS) has led to increased interest in patient-reported outcomes (PROs). 1,2 In addition to traditional endpoints, PROs are increasingly used in clinical trials to assess treatment effects from the patients' perspective. The Multiple Sclerosis Impact Scale (MSIS-29) is a PRO developed in 2001 to assess multiple sclerosis (MS) according to two scales: Physical Impact and Psychological Impact. 3 The MSIS-29 has been used in many different trials, including ADVANCE. 4,5 The pivotal ADVANCE study compared the efficacy and safety of subcutaneous peginterferon 125 mcg beta-1a dosed every two and four weeks with placebo in patients with RRMS. 4,6 Efficacy endpoints included adjusted annualized relapse rate (ARR) and confirmed disability progression (CDP), and MSIS-29 was an exploratory endpoint. Significant ARR and CDP treatment effects were demonstrated at Week 48 in both intervention groups compared with placebo. 4,6 No significant treatment effects were demonstrated on the MSIS29.
Within-scale changes from baseline indicated a significant improvement at Week 48 across all three treatment groups for Psychological Impact scores, and a worsening in Physical Impact scores (statistically significant in the placebo group only). 5 A post-hoc analysis revealed that 12-week CDP was associated with a six-point worsening of the Physical Impact scores in the placebo group (p < 0.0001) relative to a 1.9-point worsening (p ¼ 0.044) in the peginterferon beta-1a every-twoweeks group. Recent relapse (occurring within the past 29 days) was associated with a 10-point worsening of the Psychological Impact scores in the placebo group (p < 0.0001) relative to a 3.5-point worsening in the peginterferon beta-1a every-two-weeks group (p ¼ 0.031). 5 MSIS-29 and other PROs are valuable rating scales in evaluating treatment benefits from a patient's perspective, 1,7,8 measuring complex unobservable variables through self-reported questionnaires. 7,9 A well-designed PRO should be reliable and valid, and care should be taken to ensure the total score of the scale appropriately reflects the patient experience. 1,7,10,11 There are three main psychometric paradigms for developing and evaluating rating scales: Classical Test Theory (CTT), 12 Rasch Measurement Theory (RMT), 13 and Item Response Theory (IRT). 14 A detailed comparison of these paradigms is presented elsewhere displaying the sophisticated and extensive analysis potential of the RMT. 1,8 The MSIS-29 was developed in accordance with CTT criteria. 12 Item questionnaires were generated through patient interviews (with primary progressive (PPMS), secondary progressive (SPMS), and RRMS patients), expert opinion, and literature review. 3 Psychometric properties and responsiveness were evaluated in an independent sample 3 and subsequently evaluated using CTT methods by several studies. 1517 However, in order to further assess the validity of the MSIS-29, it is important to validate the scale through an independent psychometric paradigm.
The RMT offers a mathematically testable model articulating a priori criteria used to verify measurement properties or to expose and explore anomalies of any rating scale empirically. 8,13,18 An RMT of the MSIS-29 in a small sample (n ¼ 92) of SPMS, RRMS, and PPMS patients was largely supportive of its measurement properties but revealed issues with its response scale, item fit, and coverage of patients with lower psychological impact. 19 Two additional studies utilized portions of RMT analysis to assess item stability across participants with differential depression levels 16 and patient and proxy responders, 20 providing supporting evidence for MSIS-29. However, to date there have been no RMT analyses of the MSIS-29 conducted in a large sample of patients with RRMS.
The objective of this study was to evaluate measurement properties of the MSIS-29 within ADVANCE in accordance with RMT criteria. The RMT is superior to the CTT because it has interval rather than ordinal scoring, separation of item and person parameters, non-sample-dependent scale properties (e.g. reliability and validity), and the potential for individual-level measurement. 1,8 The current RMT psychometric analysis enables extensive assessment of sample-to-scale targeting within RRMS patients whose levels of disability are potentially different from those of the patients' used for the development and validation of the MSIS-29. Additionally, RMT allows for the optimization of the scoring structure and interpretation with the provision of interval-level scores.

ADVANCE study
ADVANCE was a two-year, randomized, doubleblind, parallel-group, Phase 3 study, with a one-year placebo-controlled period comparing peginterferon beta-1a 125 mcg administered every two or four weeks versus placebo in patients with RRMS. Results from the study are described in detail elsewhere. 4 ADVANCE recruited patients aged between 18 and 65 with a maximum 5.0 baseline Expanded Disability Status Scale (EDSS) score. 21 Protocol for the ADVANCE study was approved by the institutional review board at each site, and the study was conducted according to International Conference on Harmonization Guidelines for Good Clinical Practice and the Declaration of Helsinki. Every patient provided written informed consent before entering the study.

Materials
The MSIS-29 is a disease-specific PRO that measures the Physical and Psychological Impact of MS via two scales of 20 and 9 items each. 3,16 Items are scored on five-point Likert-like scales, with higher scores indicating greater impact of MS on this domain.

Data analysis
Data analysis followed three stages. First, psychometric evaluation of the MSIS-29 in line with RMT was performed. A review of these findings led to conceptual restructuring of the MSIS-29 measurement model, properties of which were evaluated using RMT at a second stage. Finally, the responsiveness of the original and rescored scales was examined. RUMM2030 22 was used to conduct the RMT and IBM SPSS 21.0 23 responsiveness analyses. Data from eight different time-points were stacked to increase sample size for the psychometric analysis.
RMT analysis RMT analysis compares observed data against the stringent criteria of the Rasch model with the broad aims described below. 8,24 How adequate is the sample-to-scale targeting? Sample-to-scale targeting concerns the match between the range of Physical or Psychological Impact measured by the MSIS-29 items, and the distribution of impact measured in the sample, subsequently influencing interpretation of all other RMT analyses. Person and item locations are plotted against the same metric, and their relative distributions are assessed. 25,26 Do the response categories work as intended? Greater changes in MSIS-29 scores signify more impact. It is therefore expected that the higher the impact of a responder, the higher the response category to be endorsed. Response thresholds are expected to be ordered in a successive manner along the measurement continuum. 8,27 Thresholds represent the point at which the probability of endorsing two adjacent response categories is equal.
To what extent do the MSIS-29 items work together to define a single measurement construct? RMT expects scale items to be cohesive and work well together clinically and statistically when summed up to a single total score. 8,27 Using a rule of thumb, fit residual (residual ¼ observed expected score) estimates for each item should be within±2.5. Chi-square tests assess the difference between each item's mean observed and expected scores within certain class intervals of the trait being measured. Item characteristic curves (ICCs) display this relationship graphically, providing context for interpreting the magnitude and pattern of numerical fit statistics.
To what extent does the response to one item bias the response to another? RMT expects that items should not be dependent on or biased by each other so as to not artificially inflate reliability. We assessed the degree of ''local dependence'' among scale items by examining item residual correlations. Residual correlations >0.30 warrant further examination, 28 as they reflect >9% of shared variance.
How has the sample been measured? Are responders in the sample separated by the MSIS-29 items? Scale items are expected to detect differences in levels of impact within a sample and changes over time. Within RMT, the person separation index (PSI) is calculated to assess this. 8,27 The PSI is a numerical indicator ranging from 0 to 1, computed as the ratio of error-corrected person variance relative to the total person variance, 29 with higher values indicating greater detection of reliable differences.
How valid is the sample measurement? Similar to item responses, it is important to assess whether the measurement of each person's total score is in line with RMT expectations. 18 This is assessed through person fit residual, with reference to the ''rule of thumb,'' expecting 99% of the sample to produce a fit residual between À2.5 and 2.5. Fit residuals outside this range indicate problematic measurements and questionable measurement validity. 8,27 What is the relationship between MSIS-29 raw scores and interval measurement? The MSIS-29 total score is ordinal, computed through the summed total of individual Likert-like items rather than an equal-interval measure of Physical or Psychological Impact. It is important to assess the extent to which ordinal raw scores approach interval measurement; one point on an ordinal scale is not necessarily the same across the breadth of the scale, 27,30 and this has implications when interpreting findings. RUMM2030 plots raw scores against estimated interval measurements, which can be used to provide a subsequent transformation on an interval 0100 score for each scale.

Responsiveness analysis
The ability of MSIS-29 scales to detect change at Week 48 was examined and compared. To increase consistency in this comparison, original and restructured scales were anchored on the same overarching scale. Interval level 0100 transformed scores were used, computed on the basis of RMT-produced interval logit for total raw scores. Responsiveness was examined using four standard indicators: Cohen's effect size (ES) 31 and standardized response mean (SRM), 32 relative efficiency using paired samples t-tests, 33 and relative precision using one-way analysis of variance (ANOVA). 34

Results
Data from a total of 1509 people with RRMS at eight time-points are presented in Table 1. Table 2) MSIS-29 scales demonstrated sub-optimal targeting, as the range of impact measured by scale items covered only 58% of those measured in the sample. Physical Impact (Figure 1(a)) also demonstrated some item bunching, whereas Psychological Impact ( Figure 1(b)) had some item gaps on the measurement scale. For both scales, person measurements and means were skewed to the floor; i.e. lower impact.

MSIS-29: RMT findings (
All 29 items displayed ordered thresholds, indicating that response categories worked as intended. Item fit residuals for 80% of physical and 89% of psychological items fell outside the recommended range; 15% and 11% failed the adjusted Chi-square criteria, respectively. ICCs reflected marginal fluctuation of observed scores from the Rasch-expected scores, suggesting all four items slightly under-discriminated impact ( Figure 2). One item pair had residual correlations >0.30 (r ¼ 0.40), suggesting dependency between item responses for ''grip things tightly'' and ''carry things.'' PSIs were 0.91 and 0.87, indicating individuals in the sample were separated adequately by MSIS-29 items. Person fit residuals indicated significant misfit for both scales, with 22% of person measurements for Physical and 19% for Psychological Impact falling outside the recommended range, suggesting problematic measurement. The relationship between raw total scores and interval logit metric was S-shaped for both scales, indicating that a one-point change in raw score is associated with a variable rate of change in the impact interval measurement; variability was highest at the two ends and lowest in the center ( Figure 3).

Revised MSIS-29 scoring structure
The MSIS-29 item content was reviewed by a multidisciplinary team that included neurologists, psychologists, and health measurement experts, and three re-conceptualized scales were proposed. The revised scales comprised: ''Symptoms,'' containing 10 of the original 20 Physical and four of the original Psychological Impact Items; ''General Limitations,'' containing 10 of the original Physical Impact scales; and ''Psychological Impacts,'' containing five of its original nine items.

Revised MSIS-29 scales
The revised MSIS-29 scales demonstrated sub-optimal but improved targeting, as the range of impact measured by the scale items covered 68% of the range of the impact measured in the sample, whereas the scales person measurements were consistently skewed to the floor of the measurement scale (i.e. lower impact; Figure 4).      . Raw score to interval metric transformation. The x-axis represents the Physical Impact construct as an interval logit score with increasing impact from left to right and the y-axis the raw score as calculated by the summed total of the Multiple Sclerosis Impact Scale (MSIS29).
fit residuals indicated significant misfit with 16%28% of person measurements across the scales falling outside the recommended range, suggesting problematic measurement. The relationship between raw total scores and interval logit metric was consistently S-shaped for both scales.

EDSS sub-sample
Examining sub-optimal targeting primarily at the floor of all scales (i.e. lower impact), scale performance was further assessed on a sub-sample of patients, excluding those with an EDSS score <2.5, reflecting no-to-little disability. Sample-to-scale targeting, the measurement scale, and sample measurement were improved in the original Physical Impact and all three revised scales when tested on the sub-sample of responders with EDSS scores >2.5 ( Table 2).

Responsiveness analysis and score interpretation
Relative precision examination suggests that the revised Psychological Impact was the most sensitive scale for detecting change (Table 3). Examining score changes from baseline within the entire sample (  (Table 4) indicated a significant reduction of Psychological Impact in the original and revised scales, and a significant increase in the General Limitations (p < 0.01). All three change scores were associated with low ESs and SRMs. Examination of minimally important difference (MID) for ES and SRM compared to mean change scores indicated that they were both higher, providing additional evidence around the magnitude of change scorers for these scales. items and 80% of the Psychological Impact items are scored on the lower end of impact (''a little''), also leaving limited room for improvement.

Discussion
The original MSIS-29 has been an important PRO instrument in MS clinical studies and trials for more than 15 years. Its straightforward scoring for two broad concepts (Physical and Psychological Impact) has allowed for wide application, and provided a strong basis for comparable data between different research endeavors. However, since it was published 3 there has been an increased use of more sophisticated psychometric methods, which provide the potential for better measurement of patient experience. 1 Our RMT findings supported previous research, 8 revealing varied evidence supporting the use of the MSIS-29. In brief, its targeting and conceptual basis could be improved; the range of impact covered by items did not match the range of impact measurement in the study sample (particularly at the floor of the scales, i.e. patients with lower impact), and item fit analyses indicated potential problems for item placement within scales.
We proffer three revised, conceptually clearer scales: ''General Symptoms'' related to range of symptoms, ''Psychological Impact'' related to emotional wellbeing, and ''General Limitations'' related to difficulties in everyday life. RMT analyses of these revised  scales indicated improved sample-to-scale targeting and item fit, although not completely resolved. Our findings also suggest the revised Psychological Impact and General Limitations scales were able to detect more change than original Psychological and Physical Impact scales, respectively. Ultimately, we recommend the MSIS-29 may be further improved by adding more complex (related to higher functioning) items in the lower range of the measurement scale to improve content coverage and floor effects, primarily relevant for patients with mild disability.
Although our findings suggest the potential for improving legacy instruments, such as the MSIS-29, it is important to consider the caveats of our empirically generated revised MSIS-29 scales. First, all post-hoc RMT psychometric analyses of the MSIS-29 are limited to the instrument's original content, which was not developed in line with RMT, nor were the items selected with clinical hierarchies in mind. Also, considering the original 5 MSIS-29 was developed with input from patients with relatively high levels of MS disability (>50% were retired because of disability), it is not surprising that the revised scales did not resolve sub-optimal targeting for patients with fewer disabilities.
Using the MSIS-29 to assess clinical change in MS populations similar to the ADVANCE cohort would require an expansion of the scale to include items that are associated with levels of symptoms, psychological impact, and limitations relevant to patients in this context. The proposed scoring structure of the revised MSIS-29, as well as the item hierarchies within each of the revised scales, represent just one way the items could be re-arranged. As this re-conceptualized scoring structure is supported by a single post-hoc psychometric analysis, it is essential that the revised scales be subjected to further psychometric testing and clinical anchoring in independent samples.
Our findings provide an initial evidence base to improve the measurement potential of the MSIS-29 as a PRO instrument in MS clinical research and trials. Articulating scores in relation to symptoms, emotional well-being, and general limitations increases conceptual clarity of MSIS-29. In trials such as ADVANCE, a more explicit and easily interpretable set of concepts can be presented in discussions regarding treatment benefits to patients, regulators, and payers. Additionally, the improved targeting of the revised MSIS-29 scales reduces the overall error associated with measurement. This improves the scales' potential to reflect the impact of clinical change in MS when it occurs. Finally, since original MSIS-29 scores are ordinal in nature, the use of the linearized (interval-level) transformed 0100 scoring would benefit the interpretation of scores and change scores, especially in patients with less disability. 35