Assessing the Content Validity of the Investigator-Rated ADHD Rating Scale Version IV Instrument Inattention Subscale for Use in Adults With Phenylketonuria

Content validity of the 18-item Investigator-Rated Attention-Deficit Hyperactivity Disorder (ADHD) Rating Scale IV (I-ADHD RS-IV) with adult prompts was investigated using qualitative interviews of US clinicians who had prior experience rating adults with phenylketonuria (PKU) using the I-ADHD RS-IV. Fourteen qualitative interviews were conducted to obtain key symptom experiences of adults with PKU and assessed the relevance, clarity, and administration of the I-ADHD RS-IV. Participants (n1⁄4 13, 92.9%) endorsed the inattention symptoms as key experiences by adults with PKU and endorsed the instrument as fit for purpose for adults with PKU. Participants generally reported low frequencies of occurrence for the 9 I-ADHD RS-IV hyperactivity/ impulsivity items. Despite some clinicians’ concerns for the lack of patient self-awareness, the participants reported no difficulty selecting a rating on these items. This in-depth study of the content validity of the I-ADHD RS-IV provides evidence that this clinician-reported instrument captures the severity of important inattention symptoms in adults with PKU.


Introduction
Phenylketonuria (PKU; Online Mendelian Inheritance in Man 261600 and 261630) is an autosomal recessive genetic disorder caused by a functional deficit in the phenylalanine (Phe) hydroxylase enzyme, which subsequently impairs the conversion of Phe into tyrosine. 1With diagnosis and treatment established, early symptoms can be prevented, but if untreated Phe can accumulate in the body and cause mental retardation, microcephaly, delayed speech, delayed social skills, jerking movements or seizures, psychiatric symptoms, and behavioral abnormalities. 2,3The PKU treatment consists of a Pherestrictive diet supplemented with a Phe-free mixture of amino acids; however, the diet can be difficult to adhere to and adherence to the diet is often poor. 4][7][8] The Investigator-Rated Attention-Deficit Hyperactivity Disorder (ADHD) Rating Scale IV (I-ADHD RS-IV) with adult prompts is a measure of attention-deficit and hyperactivity/ impulsivity used in a number of ADHD studies. 9The psychometric properties of this clinician-reported instrument have been previously established in a population of adults with symptoms of ADHD 10 ; however, the content validity of the I-ADHD RS-IV in an adult population with PKU has not been established.Nonetheless, this instrument could be useful for measuring key symptoms of adults with PKU.
Prior investigations of the Adult ADHD Self-Report Scale (ASRS) among adults with PKU demonstrated the content validity 11 and psychometric properties 12 of the inattention subscale of this instrument.However, among adults with PKU, full selfawareness of their symptoms and the effects of these symptoms is problematic.A recent survey of metabolic clinics in the United States noted that 50% of clinics report that patients' cognitive deficits may impact their ability to obtain regular medical care and three-quarters cited related issues, such as patient apathy and treatment noncompliance toward a continued Phe-restricted diet. 13One metabolic clinic that implemented screening for psychiatric distress during routine clinic visits found that 53% of adults screened positive for psychiatric distress and 24% for executive function impairment. 14Therefore, the use of a selfreported instrument like the ASRS to measure inattention in adults with PKU may also be problematic.

Aims
This study investigated the content validity of the I ADHD RS-IV instrument with adult prompts, with a focus on the inattention subscale, through qualitative interviews among professionals with experience working with adults with PKU.Content validity is the extent to which the instrument measures the concept of interest; supported by evidence from qualitative interviews that the items and domains of an instrument are appropriate and comprehensive relative to its intended measurement concept, population, and use. 15The interviews were designed to achieve the following goals: (1) elicit the symptom experiences of adults with PKU from the clinician perspective and (2) assess the relevance and clarity of the inattention items in the I-ADHD RS-IV and any difficulties associated with administration in adults with PKU.

Study Design
Cross-sectional qualitative interviews were conducted with 14 professionals who have experience in rating adults with PKU using the I-ADHD RS-IV.Potential participants were identified as those with an educational background in genetics, psychology, psychiatry, or a related field and who satisfy the following inclusion criteria: master's or PhD candidate or degree holder in psychology or related field, or a doctor of medicine (MD) with specialization in psychiatry; having at least 5 years of experience with research in clinical trials; experience with adult ADHD strongly recommended, with PKU-related experience preferred; at least 3 years of prior experience conducting structured or semi-structured interview research assessments; and at least 1 year of past experience evaluating ADHD-like symptoms in 18 years and older adults with PKU.Participants were selected using a sampling frame of clinicians and researchers provided by the study sponsor, all of whom had been trained to use the I-ADHD RS-IV as part of a clinical trial.The study team contacted potential participants by telephone and/or e-mail to assess eligibility and interest.
The content validity interviews followed a semi-structured guide that included a concept elicitation discussion that encouraged emergent descriptions of neuropsychiatric symptoms and impacts of PKU.Afterward, a cognitive debriefing portion asked questions about the I-ADHD RS-IV inattention items and prompts to address the clarity, comprehensiveness, relevance, recall, vocabulary, and other key aspects of the instrument.After discussion of the inattention items, the interview guide probed on the occurrence of hyperactivity/impulsivity symptoms included in the I-ADHD RS-IV.Interviews were conducted until saturation on PKU-related symptoms was achieved, as defined as the point when no new relevant or important information merges. 16,17All interviews were conducted via telephone, in English, and were digitally recorded with the consent of each participant.

Investigator-Rated ADHD RS-IV With Adult Prompts
The I-ADHD RS-IV, which assesses adult ADHD symptoms, was designed to be completed by a trained clinical psychologist or psychiatrist.The scale includes 18 items assessing the frequency, severity, and impairment of ADHD symptoms and behaviors in adults over the past month.The scale was derived directly from the diagnostic criteria as delineated in the Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition). 18The questions are divided into 2 subscales: 9 items measure symptoms of inattention and 9 items measure symptoms of hyperactivity-impulsivity. The 9 items in the inattention subscale include careless mistakes, difficulty sustaining attention on task, doesn't listen, difficulty finishing and following instructions, difficulty organizing tasks, dislikes/avoids tasks requiring attention, loses things, easily distracted, and forgetful in daily activities.The 9 items in the hyperactivity and impulsivity subscale include fidget, difficulty remaining seated, restlessness, difficulty playing quietly, on the go/acts like driven by a motor, talks excessively, burst out answers, difficulty waiting turn, and interrupts or intrudes.The items are rated using a 4-point Likert-type frequency scale (0 ¼ none, 1 ¼ mild, 2 ¼ moderate, and 3 ¼ severe).The I-ADHD RS-IV inattention and hyperactivity subscale scores range from 0 to 27, and the instrument's total score ranges from 0 to 54.Higher scores indicate a worse severity of symptoms.
The I-ADHD RS-IV was designed to be used as a semistructured interview guide during the assessment.The interviewer starts by asking the patient about each item and follows up with the relevant prompts for each item prepared by the developers.Using the prompts, the interviewer can gain clarity on the interviewee's circumstances and probe beyond a yes/no response.Ratings on the I-ADHD RS-IV are based on clinical judgment and not entirely on the patient's report.

Data Analysis
Qualitative data were analyzed using ATLAS.tiqualitative data analysis software version 7.1.6. 19All interviews were digitally recorded, and the audio recordings were transcribed.Content analysis of the output was conducted with the following steps: (1) a coding dictionary was developed based on the semistructured interview guide and relevant probes, (2) using the coding dictionary, 2 scientific team members independently coded the first structured interview transcript and then met with a senior scientific reviewer to discuss the coding dictionary and the coding process, and (3) if the coding was sufficient, then the coders independently coded the remaining transcripts with review from a senior scientific staff member.
The cognitive interview data were coded based on the participant feedback on (1) the clarity of the I-ADHD RS-IV inattention items and prompts, (2) the ease of completing the instrument, (3) the appropriateness of the format and recall period, and (4) the appropriateness of the scale in adults with PKU.Qualitative findings were summarized with exemplary quotes, frequencies, and percentages as appropriate to best summarize the results.

Study Sample
Disposition of the study sample is provided in Figure 1.Overall, 49 (57%) participants responded to the invitation to participate in the cognitive interview.A total of 37 (43%) participants did not respond to the invitation, and 35 (41%) of the invitees who responded to the invitation were either not interested or did not meet all of the cognitive interview study inclusion/exclusion criteria.A total of 14 (16%) participants expressed interest with subsequent enrollment into the study.
Demographics and the clinical experience of the study population are provided in Table 1.Approximately half of the participants stated their educational background as doctor of philosophy (PhD; n ¼ 6, 42.9%) and 2 participants indicated they were medical doctors (MD/MBBS; 14.3%).Other participants' educational backgrounds included doctorate in psychology (n ¼ 4, 28.6%), master's in social work (n ¼ 1, 7.1%), and master's in nutrition (n ¼ 1, 7.1%).The majority of the participants' professional role with adult patients with PKU  involved diagnostic and assessment work (n ¼ 11, 78.6%), and most of them worked with adult patients with PKU in an outpatient setting (n ¼ 9, 64.3%).The participant's mean years of experience working with adults with PKU was 9.9 (SD ¼ 9.2), while the mean years of experience working with adults with inattention and hyperactivity was 14.9 (SD ¼ 7.9).The mean age of adult patients with PKU treated by participants ranged from 19.5 (SD ¼ 2.6) to 47.9 (SD ¼ 9.6).The average number of adult patients with PKU seen by participants each month was 6.3 (SD ¼ 6.8), and each year was 52.7 (SD ¼ 59.3).Only 1 participant prescribed medication for patients with psychosocial issues (7.1%), and 1 participant for symptoms of ADHD (7.1%).

The PKU Symptoms
Participants were asked to discuss the symptoms observed in adult patients with PKU.If not spontaneously mentioned, participants were probed on previously reported PKU-related cognitive difficulties (eg, difficulty paying attention, difficulty concentrating), difficulty in social situations, emotional difficulties (eg, problems with mood, specific mood problems), physical problems, and hyperactivity/impulsivity. The PKU symptoms endorsed by the participants were mapped to the I-ADHD-RS IV inattention items.A total of 8 participants (57.1%) reported symptoms of hyperactivity or impulsivity, with 2 participants stating that hyperactivity/impulsivity symptoms are not really observed in adults, and 1 participant stated that hyperactivity/ impulsively symptoms are probably less common.The hyperactivity symptoms included fidgeting (n ¼ 4, 28.6%), moving around (n ¼ 1, 7.1%), interrupting conversations (n ¼ 1, 7.1%), difficulty remaining seated (n ¼ 1, 7.1%), being on the go (n ¼ 1, 7.1%), restless (n ¼ 1, 7.1%), talking too much (n ¼ 1, 7.1%), and hyperactivity with no specific details (n ¼ 3, 21.4%).

Self-Awareness of Adults With PKU
As reported by several participants, many factors contribute to the challenge of self-awareness that adults with PKU have regarding their own symptoms.Five of these professionals felt that adults with PKU, in general, are aware of their symptoms, while another 5 reported that overall adults with PKU have problems with self-awareness.Several participants commented on the spectrum of self-awareness and the contributing factors that cause some adults with PKU to be more self-aware than others.Specifically, participants felt that those adults with PKU with more symptoms, lower cognitive ability, and Phe levels that are not controlled (definition of control not provided by participant) have less self-awareness, while those with higher cognitive functioning, employment, and Phe levels that are well controlled tend to have more self-awareness.The following are 3 exemplary quotes from study participants.''I don't think they're very aware, I mean some more so than others, there's always a spectrum.I think the few who are more figuratively affected or more symptomatic have less self-awareness.''''I think their selfawareness increases as they get on the job.I think that they do not have a lot of self-awareness initially and a lot of times I find that when I do some of the initial ratings with them they say everything is fine and then when we go back once they on the drugs for a while then they really start saying oh, now I see that this was a problem before and now I see how much better it is.''''The well-controlled patients actually have quite good selfawareness.The patients with higher Phe levels will certainly be worse off, the lack of awareness is there, and paradoxically we get higher reading scores on the milder patients.''

Cognitive Interview Results: I-ADHD RS-IV Inattention Subscale
The raters in this study endorsed the I-ADHD RS-IV inattention subscale as an appropriate instrument to measure inattention symptoms in adults with PKU, with 71% (n ¼ 10) of participants stating each item was applicable to adults with PKU (Figure 2).The majority of the participants who were asked (n ¼ 11, 85%) stated a 1-month recall period was appropriate to assess the concepts.In addition to providing feedback on the items, participants were asked about the item prompts.The majority of the participants (n ¼ 10, 71%) stated that the item-level prompts were sufficient for 7 of the 9 items, with just over half of the participants (n ¼ 9) stating that item-level prompts were sufficient on the 2 items (difficulty finishing, following instructions, and dislikes/ avoids tasks requiring attention; Figure 2).Some participants had recommendations for augmentations to the prompts (additional prompt(s)) but noted that the I-ADHD RS-IV training guide and instructions allow the rater to ask the patient additional questions/prompts to elicit greater details in the symptom experience when administering the instrument.At the item level, the majority of participants stated that a patient's self-awareness of symptoms can make the rating of some items more challenging (ie, careless mistakes, doesn't listen, dislikes/avoids tasks requiring attention, and easily distracted; Figure 2), but self-awareness was less likely to affect 3 of the 9 items in the inattention domain (ie, loses things, difficulty finishing, following instructions, and difficulty organizing tasks).For 1 inattention item (forgetful in daily activities), half of the sample stated that self-awareness posed a challenge, and half of the sample stated that selfawareness is not a problem for patients.While feedback from the participants on a patient's ability to accurately reflect on their own symptom experience plays an important role in the administration and interpretation of the I-ADHD RS-IV, the participants fully understood that the rating is based on clinical judgment, and despite concern for the lack of patient self-awareness, the participants reported no difficulty selecting an appropriate rating on these items when assessing an adult with PKU.

Cognitive Interview Results: I-ADHD RS-IV Hyperactivity/ Impulsivity Subscale
Items 10 to 18 in the I-ADHD RS-IV instrument contain hyperactivity and impulsivity symptoms.Given that participants stated that hyperactivity/impulsivity symptoms are less common in adults with PKU, participants were asked about these items in aggregate with a general question about which items were applicable or not applicable to adults with PKU and which items represent symptoms frequently occurring in adults with PKU.In general, participants felt that the hyperactivity/impulsivity items are appropriate for adults with PKU, with the exception of item 13 (difficulty playing quietly).While each of the 9 items were reported by at least 1 participant as occurring in adults with PKU, the frequency of occurrence was low for each item.Moreover, each of the hyperactivity/impulsivity item was mentioned as not occurring by at least 1 participant (Table 3).

Discussion
Qualitative interviews were conducted in 14 participants with experience in rating adults with PKU using the I-ADHD RS-IV; the purpose of these interviews was to understand the PKU symptom experience and to assess the relevance of the concepts and clarity of the I-ADHD RS-IV inattention items for use in the adult population with PKU.Saturation, as defined as the point when no new relevant or important information merges and collecting additional data will not likely add to the understanding of the concepts of interest and the items in the questionnaire, of the PKU-related symptoms reported by the participants was obtained with the 14 cognitive interviews. 16,17The most common spontaneously reported symptoms of adults with PKU by these professionals included poor sustained attention, difficulty with cognitive ability, difficulty with planning and organizing skills, forgetting things/memory, and anxiety.Symptoms probed during the course of the interview included cognitive difficulties, difficulty in social situations, emotional difficulties, physical problems, and hyperactivity/impulsivity. While symptoms within these clusters were reported as observed traits in adults with PKU, inattention was the major theme and heavily endorsed by these professionals as a hallmark symptom in adults with PKU.In addition, the PKU inattention symptoms reported by the participants mapped to all 9 inattention subscale items of the I-ADHD RS-IV, further supporting the content validity of the instrument for use in adults with PKU.][22][23][24] As part of the cognitive interview, participants were asked about symptom self-awareness in adults with PKU.Accurate reflecting and reporting of the symptom experience by the patient is a critical part of the I-ADHD RS-IV assessment.While the I-ADHD RS-IV allows the rater to select a response based on their clinical judgment, the rating is dependent on the patient interview and observation of the patient.Mixed responses from the participants were reported, with half of the participants endorsing self-awareness of patients' own symptoms as problematic, and half of the participants responding that self-awareness is not a problem in eliciting the patient's symptom experience.While a patient's self-report on her or his symptom severity may differ from a rater's clinical judgment and observation of the patient's behavior, the patient is able to report on their experience and the I-ADHD RS-IV allows for the use of prompts (both scripted and not scripted), clinical judgment, and observation to make a rating for each of the instrument's items despite lack of symptom self-awareness in adults with PKU.
In qualitative research methodology, there are some innate limitations.All data were self-reported and provided through one-on-one telephone interviews that lasted 60 minutes.The interview guide was semi-scripted and intended to facilitate a fluid discussion about PKU symptoms and the I-ADHD RS-IV.The interviewer was allowed to use judgment and adapt the guide in order to cover the questions during the interview time allotted.If a participant was more verbose in his or her responses or focused on a particular area of interest, it sometimes was not possible to ask all the questions in the guide within the interview time frame.Therefore, given the fluid nature of a semi-scripted qualitative interviews, the sample size for each question that was asked was not always 14 (the total sample enrolled), and the data were summarized in a reduced sample.In conclusion, this in-depth study of the content validity of the I-ADHD RS-IV inattention subscale provides evidence that the clinician-reported instrument captures the severity of these important symptoms in adults with PKU and is appropriate for use in a PKU clinical trial to measure treatment efficacy on inattention symptoms.Additional research is required to assess the psychometric properties (ie, reliability, construct validity, measurement of change) in the context of the PKU population of interest and provide additional support for the usefulness of this measure for understanding key symptoms in adults with PKU.

Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Yinpu Chen, Rishabh Jain, Laurel Konkol, Markus Merilainen, and Haoling Weng are employees of BioMarin Pharmaceutical and receive financial support in terms of salary and stock.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Kathleen Wyrwich, Elizabeth Bacci, Katharine Gries, and Gemma Al-Jassar are employees of Evidera, which received financial support from Bio-Marin Pharmaceutical in connection with the study development and execution as well as the manuscript development.

Figure 1 .
Figure 1.Disposition of the study sample.

Figure 2 .
Figure 2. I-ADHD RS-IV inattention subscale items: cognitive interview results.I-ADHD RS-IV indicates Investigator-Rated Attention-Deficit Hyperactivity Disorder Rating Scale IV.

Table 2 .
The PKU Symptom Mapped to the I-ADHD RS-IV Inattention Subscale Items.
Abbreviation: I-ADHD RS-IV, Investigator-Rated Attention-Deficit Hyperactivity Disorder Rating Scale IV. a Categories are not mutually exclusive.b Two participants did not provide a response; percentage was calculated based on 12 participants.