Evidence of Validity for the Online Version of the Assessment of Adherence to Antiretroviral Therapy Questionnaire

A Patient-Reported Outcome (PRO) measure titled Cuestionario para la Evaluación de la Adhesión al Tratamiento Antiretroviral (acronym CEAT-VIH) is currently available in paper-and-pencil and digital (online assessment) formats. Due to the advantages of online assessment, the main objective of this work was to evaluate psychometric properties of the online version, in an international sample, to accumulate evidence of its validity and provide score norms for the questionnaire. A psychometric study was performed with an international sample of 1,470 participants, from 30 countries, to accumulate evidence of CEAT-VIH validity regarding internal structure and related external criterion (e.g., viral load, number of pills, and AIDS-related symptoms). Descriptive statistics and normative data for scores are also presented. The majority of participants were men (72.4%), aged 15 to 78 years old (M = 39.3, SD = 12.6). A unidimensional model with five facets occurred as the observed variables converged, which presented a good model fit (comparative fit index [CFI] = 1.000; Tucker–Lewis index [TLI] = 0.999; standardized root mean square residual [SRMR] = 0.027; and root mean square error of approximation [RMSEA] [90% confidence interval, CI] = 0.009 [0.000, 0.038], p = .995). There was a weak invariance for the CEAT-VIH structure for language versions and countries. Cronbach’s alpha values for the instrument (17 items) were acceptable across language versions (.88-.96). Evidence of validity related to external criteria was achieved by associations (e.g., Spearman and Mann–Whitney) between CEAT-VIH scores and relevant clinical (e.g., CD4+ cells, viral load, number of pills, and AIDS-related symptoms) and sociodemographic (e.g., gender, age, employment status, education level, place of residence, and participation at local AIDS association) variables. In conclusion, the overall data on the evaluated psychometric properties allow recommendation of the use of this instrument in research and applied settings.


Introduction
Adherence to HIV treatment is a complex and multifaceted behavior (Remor, 2013b), and can be understood as the extent to which one's behavior coincides with the health recommendations he or she receives (Haynes, Taylor, & Sackett, 1979). However, associations between adherence levels and health outcomes are complex; many mediators may be implicated, including time in treatment, psychological variables, quality of the provided treatment, patient age, and viral load levels (Costa, Torres, Coelho, & Luz, 2018;Marks et al., 2015;Remor, Penedo, Shen, & Schneiderman, 2007). Psychosocial aspects should also be investigated, because they may act as barriers or facilitators to the adherence process (Biello et al., 2016;Costa et al., 2018;Dima, Schweitzer, Diaconiţ, Remor, & Wanless, 2013). The combination of drug treatments and psychosocial approaches allows for various vulnerabilities to which people with HIV are exposed to be addressed. Nonadherence or low adherence to treatment is a threat to treatment effectiveness, and they contribute to elevated viral load, the incidence of opportunistic diseases, and an increase in the number of hospital admissions (e.g., Foresto et al., 2017).
To contribute with an adequate assessment of adherence, a self-report measure titled Cuestionario para la Evaluación de la Adhesión al Tratamiento Antiretroviral (CEAT-VIH, acronym in the original) (or "Assessment of Adherence to evidence with viral load, presence of side effects, AIDS symptoms, and body mass index; score responsiveness after intervention or predicting patients' abandonment were identified; and patterns of convergence and divergence with other instruments were observed (i.e., 20-item Self-Reporting Questionnaire , HIV/AIDS-targeted quality of life [HAT-QoL], Beck Depression Inventory [BDI]; see Table 1 for details).
The main objective of this article was to increase information about the psychometric properties of the instrument, in a larger international sample, by evaluating the internal structure of the online version of the CEAT-VIH questionnaire, and describe additional validity evidence (e.g., criterion related). To achieve these aims, a confirmatory factor analysis (CFA) and two multigroup confirmatory factor analyses (MGCFAs) were conducted to demonstrate the dimensionality of the instrument. In addition, to verify and accumulate validity evidence, hypotheses of association among the instrument and both relevant external clinical criteria and sociodemographic profiles were tested. Score standardization by language version and gender by country groups was also performed.

Participants
The sample consisted of 1,470 participants from different countries (15% participating in local AIDS organizations). Ages ranged from 15 to 78 years (mean [M] = 39.3, standard deviation [SD] = 12.6, median [Mdn] = 38). Furthermore, 72.4% of participants were male, 64.6% were working, and 70.7% lived in large cities or capitals. The full characterization of the sample and percentage of participants by country are summarized in detail in Table 2. Regarding HIV infection treatment, the majority of the participants were taking one (25.8%) or three (31.8%) pills per day (M = 2.9, SD = 1.7, Mdn = 3), and 82.5% were free from AIDS-related symptoms (opportunistic diseases). Participants were HIV+ from less than one to 37 years (M = 10.6, SD = 6.6, Mdn = 9). The mean CD4+ lymphocyte cells count was 506.6 (SD = 355.8, Mdn = 465), and the average viral load was 12,372.3 copies/ml (SD = 64,260.6, Mdn = 100). Almost half (45.1%) of the participants had an undetectable viral load (less than 50 copies/ml).

Instruments
CEAT-VIH online version. The online version of the CEAT-VIH 1 is a patient-reported outcome (PRO) measure, brief and easy to answer, that assesses adherence to antiretroviral therapy from a multidimensional perspective (Remor, 2013a(Remor, , 2013b. During the adaptation process of the paper-and-pencil version of the instrument to the online version, three items with a dichotomous response scale were excluded, based on the following considerations: (a) these items showed low item-total correlation in previous studies (e.g., Dima et al., 2013) and (b) their format made it difficult to score the instrument because they included qualitative responses.
Thus, the CEAT-VIH online version is composed of 17 items with a 5-point Likert-type scale for responses, distributed into five facets, according to the theoretical structure of the instrument: Compliance (three items) indicates the extent to which the behavior of the person reflects strict medication-taking adherence; Antecedents of Non-Adherence Behaviors (four items) indicates the extent to which personal or situational antecedents are related to non-adherence behavior; Doctor-Patient Communication (three items) indicates the perceived quality of the doctor-patient relationship as a motivational reinforcement to treatment adherence; Personal Beliefs/Expectancies About the Treatment (five items) indicates the extent to which the patient's beliefs and expectations affect treatment adherence behavior; and Treatment Satisfaction (two items) indicates the person's degree of satisfaction with treatment outcomes. In addition to the scores for each facet, the instrument also provides the summary adherence score that indicates the degree of overall adherence to antiretroviral treatment, which combines all five mentioned facets. Regarding the summary adherence score, the raw score can range from a minimum of 17 to a maximum of 85 points (the higher the score, the greater the treatment adherence).
The online interface allows automatic correction, and it generates standardized scores (from the raw scores) that range from 0 to 100 for each of the facets and the summary adherence score to facilitate users' interpretation of their scores. The information is shown in a graphic format and can be sent by email. These features allow professionals and patients to map which aspects promote adherence and which are barriers that need to be ameliorated with therapeutic action and/or counseling.    Table 2).

Procedures
The multilanguage CEAT-VIH online version is available at http://www.ceat-vih.info/ (compatible with mobile devices). The data used in the present study were collected through the website. Two basic types of access are observed: (a) responses directly from patients interested in their adherence self-evaluation (in some cases, patients access the CEAT-VIH website after recommendation from health professionals or non-governmental organization [NGO] personnel interested in how they are coping with adherence to medical therapy) or (b) researchers who use the instrument as a measure for adherence in their study. Researchers authorized for using the online version completed a form that explicitly granted permission for the data to be transferred to the author for psychometric analysis. Moreover, each researcher was responsible for submitting their study to their institution's ethics and research committee.

Ethical Aspects
The study was conducted in compliance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). The participants did not receive any payment for their participation. Before answering the instrument, an initial page with a digital consent form was presented, and consent to the research terms was provided by clicking the "I have read and understand the terms and conditions of this agreement" button. The study was approved by the Institutional Review Board of the Institute of Psychology, Universidade Federal do Rio Grande do Sul.

Data Analysis Plan
The statistical analyses were conducted using PASW Statistics for Windows, Version 18.0 software (SPSS, Inc., Chicago, IL, USA), for descriptive statistics, skewness and kurtosis, reliability (internal consistency was measured by Cronbach's alpha), correlations, and statistical hypothesis tests. In addition, R Software (R Core Team, R Foundation, Vienna, Austria) was used for CFA and MGCFAs, and model fit indices.
No missing values were found in the questionnaire responses. The skewness (sk) and kurtosis (ku) of all individual items showed values considered to not be normally distributed: -2.4 ≤ sk ≤ -0.2 and -0.2 ≤ ku ≤ 5.6. To confirm the theoretical structure of the CEAT-VIH online version, CFA was employed (weighted least squares means and variance adjusted [WLSMV]; Brown, 2015). The considered model fit indices were chi-square (χ 2 ) and degrees of freedom (df), incremental fit indices (comparative fit index [CFI] and Tucker-Lewis index [TLI]), and residual adjustment indices (standardized root mean square residual [SRMR] and root mean square error of approximation [RMSEA]). The models were evaluated based on CFI (>0.90), TLI (>0.90), and RMSEA (<0.08, with the 90% confidence interval [CI] not exceeding 0.10) fit indices. Measurement invariance was assessed based on the CFI difference values between the models (ΔCFI; Damásio & DeSousa, 2015). After fitting the model separately, in a second step, two MGCFAs were conducted that aimed to investigate the invariance of CEAT-VIH by the questionnaire language version and country of residence. This analysis requires at least 100 individuals per group per variable (Myers, Ahn, & Jin, 2011). To accomplish the MGCFA by country, a separate database was created with individuals only from Brazil, Mexico, Portugal, and Venezuela (n = 1,256), and for the language analysis, another database was created with individuals that responded to the questionnaire in Spanish, European Portuguese, or Brazilian Portuguese (n = 1,417). Measurement invariance was achieved if ΔCFI was <0.02 and ΔRMSEA was <0.03 for tests of metric invariance, and the traditional criteria of <-0.01 for both ΔCFI and ΔRMSEA were used for scalar invariance tests (Putnick & Bornstein, 2016;Rutkowski & Svetina, 2014). In addition, Cronbach's alpha was used to evaluate internal consistency within the complete sample. Evidence for criterion-related validity was examined for the complete sample, including relevant clinically related variables and sociodemographic characteristics. Descriptive statistics and percentile norms by language version and gender by country are also presented.

CFA to Assess Construct Validity and Dimensionality of the Instrument
The CFA was used to evaluate the goodness of fit of the theoretical model that supports the CEAT-VIH online version. The unidimensional model, composed of a single adherence factor with the five facets as observed variables, converged after 46 iterations and presented the following fit indices: χ² = 1,665.072 (df = 10, p < .001); CFI = 1.000; TLI = 0.999; RMSEA [90% CI] = 0.009 [0.000, 0.038], p = .995; SRMR = 0.027. All fit indices indicated that the final CEAT-VIH measurement model was appropriate for the data. Table 3 contains the full details of this analysis.

Multigroup CFA to Evaluate Invariance
With regard to MGCFA, the model demonstrated weak invariance, a finding that indicated the scale showed adequate configural and metric invariance (ΔCFI < 0.02, ΔRMSEA < 0.03) and also scalar non-invariance, once ΔCFI > -0.01 and ΔRMSEA > 0.01. According to the countries and language of the participants, it converged after 194 iterations, and for the questionnaire language, it converged after 165 iterations (Beaujean, 2014;Putnick & Bornstein, 2016;Rutkowski & Svetina, 2014). Table 3 shows fit indices; alternative fit indices (AFIs) for the one-factor model and invariance indices per country and per language are included in the analysis.

Reliability
Cronbach's alpha for the summary adherence score and CEAT-VIH facets was computed for the total sample (n = 1,470; α = .91) and by language version: Spanish (n = 955; α = .88), Brazilian Portuguese (n = 326; α = .88), European Portuguese (n = 136; α = .95), and English (n = 53; α = .96), as indicated in Table 4. Internal consistency for the summary adherence score (all 17 items) indicated acceptable values. For a few facets individually, and depending on the language version (i.e., Spanish and Brazilian Portuguese), there was an alpha below expectations (<.70). It is important to clarify that the facets with low alphas contain two to five items (Cronbach's alpha is calculated based on the number of items, and scales with fewer items are especially vulnerable to low alpha coefficients; Furr, 2011). On the contrary, some facets performed well, in terms of reliability, for participants who completed the European Portuguese-and Englishlanguage versions (see Table 4). The details and values for each facet and scores are presented in Table 4.

External Criterion-Related Validity: Association With Relevant Clinical-Related Variables
Clinical variables were correlated with CEAT-VIH scores to determine the evidence of clinical validity for the instrument. The hypotheses stated that the summary adherence score was expected to correlate positively with CD4+ cell count and negatively with time since diagnosis (due to treatment burden), viral load, and the number of pills prescribed to the patient. In addition, the absence of AIDS-related symptoms was expected to be associated with a higher adherence score.
The results indicated significant associations between summary adherence score and CD4+ cell count, log 10 -transformed viral load, the number of pills prescribed, and the presence of AIDS-related symptoms. Time since HIV diagnosis was not significantly associated with adherence score. Details for all facets are presented in Table 5. Note. CFA = confirmatory factor analysis; MGCFA = multigroup confirmatory factor analysis; CEAT-VIH = Cuestionario para la Evaluación de la Adhesión al Tratamiento Antiretroviral; χ 2 (df) = chi-square (degrees of freedom); CFI = comparative fit index; ΔCFI= delta comparative fit index; TLI = Tucker-Lewis index; SRMR = standardized root mean square residual; RMSEA = root mean square error of approximation; ΔRMSEA = delta root mean square error of approximation; CI = confidence interval. *p < .001.

External Criterion-Related Validity: Association With Characteristics and Sociodemographic Profile
To test the hypothesis that some sociodemographic characteristics may explain certain levels of adherence to treatment (evidence of validity related to the external criterion), the CEAT-VIH scores were correlated with gender, age, marital status, perceived SES, employment status, education level, place of residence, and participation in local AIDS organizations. The results indicated that the summary adherence score was significantly associated with gender (men scored higher), age, education level, place of residence (participants who lived in urban centers scored higher), and participation in local AIDS organizations (non-participants scored higher). The variables marital status, employment status, and perceived SES did not significantly associate with adherence scores. However, unemployed participants scored significantly higher on the compliance facet. The compliance facet indicates the extent to which the behavior of the person reflects strictly taking medication following time schedules and not forgetting doses. The unemployed apparently have fewer obstacles in terms of following time schedules or managing schedules to avoid omitting doses compared with those who work or study or combine work and study. The details for all facets are presented in Table 5.

Norms for the CEAT-VIH
To facilitate interpretation and classification of individuals evaluated in future research, the percentile norms for the CEAT-VIH were calculated. Descriptive results of the scores and percentile posts (position of the individual's score in relation to the distribution of the sample scores; Field, 2013) are available as Supplementary Material.

Discussion
The present study achieved its aims, as it demonstrated new and relevant evidence of CEAT-VIH measurement properties.
One novelty in the present study is the evidence regarding the internal structure of the instrument. The unidimensional structure that underlies the summary adherence score is a composite of five facets (i.e., Compliance, Antecedents of Non-Adherence Behaviors, Doctor-Patient Communication, Personal Beliefs/Expectancies About the Treatment, and Treatment Satisfaction) that explain individual differences in adherence behavior. Such a structure was confirmed through a CFA with goodness-of-fit indices in the acceptable range. Considering such evidence, it is recommended to use the summary adherence score as the main outcome for adherence and to evaluate or classify patients with HIV who receive treatment in terms of their adherence level (e.g., low, struggling, good, or high). The summary score can be used as a selection criterion or an outcome for interventions that address adherence improvement, and it may indicate which patients need intervention and/or counseling to overcome struggles with their antiretroviral treatment. Hence, the summary adherence score, in conjunction with the five facet scores, will help health professionals evaluate patients' difficulties in more detail and choose the most appropriate method to advise for each specific patient.
The MGCFA results evidenced that CEAT-VIH showed weak invariance. Configural and metric (ΔCFI < 0.02, ΔRMSEA < 0.03) steps of invariance were achieved for country of residence and questionnaire language. However, the scalar step of invariance was not accomplished, once ΔCFI > -0.01 and ΔRMSEA > 0.01. Regarding these findings, it is necessary to consider political and social particularities in public policies and how each country's public health systems are structured and function. In Brazil and Portugal, infectology appointments, antiretroviral medicines, complementary exams for diagnosis, and attendance and treatments directly related to HIV (AIDS) can be universally accessed. In Mexico and Venezuela, however, a certain population quota does not have consistent access to the same patterns of health care, factors that could explain the weak invariance of the construct (Biello et al., 2016;Calvetti et al., 2014;Costa et al., 2018;Padilla, 2016). In addition, adherence to HIV treatment is a complex and multifaceted behavior (Remor, 2013b), and interindividual, and consequently intercountry, differences are apparent in the construct. Another important claim is that there is very little research on the accuracy of mean-level tests for partially invariant models, and thus, much more research is required to identify the statistical and conceptual consequences of partial metric and scalar invariance. In this way, although the results showed weak invariance, CEAT-VIH demonstrated a stable structure of the model form and item loadings on the factors across samples and is a reliable and valid instrument that can be employed in different application contexts, such as multicentre studies with the online versions (Damásio & DeSousa, 2015;Putnick & Bornstein, 2016).
Although some authors (e.g., Costa et al., 2018) claim that self-reports may overestimate adherence due to social desirability (i.e., the tendency of survey respondents to answer questions in a way that will be viewed favorably by others) and recall biases, other authors (e.g., Simoni et al., 2006;Thirumurthy et al., 2012) recognize that individual selfreports can be inexpensive, easy to administer, and accurately identify medication-taking behavior. Therefore, the availability of measures that are robust in terms of psychometric properties is very much needed to minimize limitations attributed to self-report instruments. Information about validity evidence related to relevant clinical criteria variables is especially recommended (e.g., in case of HIV infection, viral load would be the gold standard).
In the authors' opinion, the CEAT-VIH overcame several limitations noted in the literature about self-report measures. For example, it had good construct validity, dimensionality invariance, acceptable reliability across samples (e.g., countries and language versions), and good criterion-related validity with relevant clinical outcomes (e.g., viral load or AIDS-related symptoms). Previous work reported responsiveness after intervention on adherence (Remor, 2013b) and convergent and divergent validity with other PRO measures (e.g., depression, anxiety, stress, psychopathology, social support, and quality of life; Remor, 2013b). Moreover, it has been used extensively in different studies and cultural contexts (36 studies published using the instrument across 12 countries) by independent researchers and always showed good performance. Regarding the online version, it is easy to apply and integrate in routine clinical settings, the questions do not exhibit comprehension problems, and it has an attractive and clean interface, an automatic scoring system, immediate graphical feedback, and the possibility of sending results by email.
Reliability coefficients for the instrument in the total sample were satisfactory (between .88 and .95 across the four language versions). However, reliability coefficients for the facets were variable across different language versions. Some facets showed reliability below expectations (<.70) in the current sample (see Table 4), meaning that these facets would not be recommended to be used separately from the conjunction of items in research. It is worth mentioning that reliability values are related to the test scores in a specific sample and are not a trait of the instrument (Urbina, 2004).
In the present study, adherence to antiretroviral treatment scores was associated with clinical variables, as expected based on the literature. Moreover, the results from the present study corroborate previous data (Remor, 2013b) regarding the relationships between external clinical variables and CEAT-VIH scores; the data highlight especially the association between viral load and AIDS-related symptoms and summary adherence score. This feature would allow researchers and clinicians working with limited resources, where biological markers are scarce, to use the instrument for follow-up for HIV infection control. Viral load control is essential to buffer HIV progression and prevent its spread (Marks et al., 2015). These results were expected along with the other associations such as CD4+ counts and the number of pills prescribed (similar results were described in Costa et al., 2018). Although these correlation coefficients were small, they appear consistently across studies and cannot be neglected.
The literature (e.g., Dunbar-Jacob & Mortimer-Stephens, 2001) is controversial about the prediction of adherence from sociodemographic variables, specifically to avoid patient stigmatization. Nevertheless, there is evidence (e.g., Rolnick, Pawloski, Hedblom, Asche, & Bruzek, 2013;World Health Organization, 2003) that some patient characteristics may indicate more risk to experience difficulties with treatment adherence. Our results indicated that age was positively associated with adherence level (similar results were reported by Biello et al., 2016), gender was associated with adherence scores (male scores were slightly higher than female scores; similar results were described in Costa et al., 2018), and education level was related to better doctor-patient communication (also reported in Pérez-Salgado et al., 2015). Patients with better formal education may connect more easily with their doctor or feel less distant or uncomfortable with health care personnel. There was an association of place of residence with personal beliefs/expectancies about the treatment, where urban participants had more positive beliefs and expectations about treatment compared with rural area individuals. In addition, individuals who participate in local AIDS organizations had slightly lower adherence scores; perhaps patients in an AIDS association are in a more vulnerable condition or are looking for help and support from the organization, potential factors that would explain their lower scores. On the contrary, marriage and occupational statuses were not related to adherence scores. In conjunction, these results may frame sociodemographic variables as risk factors (or barriers) that should be evaluated case by case to personalize health care attention. For the present study, these results support the construct validity of the CEAT-VIH, because all observed results are corroborated in the literature (e.g., Biello et al., 2016;Calvetti et al., 2014;Costa et al., 2018;Pérez-Salgado, Compean-Dardon, Staines-Orozco, & Ortiz-Hernandez, 2015), a fact that indicates the scores from the instrument behave in an expected manner.
The present study, however, is not free from limitations. The most significant limitation may be the use of a non-random sample, limited to patients or participants from countries with understanding of the languages in which the online version is available (Spanish, Portuguese, and English), with access to the Internet and the majority with a medium-to-good educational level. Further studies may be needed to test invariance across English-speaking countries or countries with other languages not included in the present study. Moreover, the design of the present study did not allow the researchers to conduct analyses of reliability test-retest or responsiveness to changes. Although previous works addressed that matter with the paper-and-pencil version (e.g., Remor, 2013b;Tafur-Valderrama, Ortiz-Alfaro, Garcıa-Jimenez, Faus-Dader, & Martinez-Martinez, 2012), it is recommended that future research address this same question with the online version.
Adherence to antiretroviral treatment is a focus of attention for multidisciplinary teams. It provides a great opportunity for psychosocial intervention that can promote self-care, adherence, healthy behaviors and lifestyles, and quality of life. The availability of reliable, valid, and culturally sensitive instruments are relevant to evaluate interventions with accuracy and precision. In conclusion, the results described in this work about the psychometric properties of the CEAT-VIH online version allow the recommendation of its use in research as a way to measure adherence to antiretroviral treatment in people with HIV infection.