Cognitive testing of the Children’s Palliative Outcome Scale (C-POS) with children, young people and their parents/carers

Background: The Children’s Palliative Outcome Scale (C-POS) is being developed using best methodological guidance on outcome measure development, This recommends cognitive testing, an established method of item improvement, prior to psychometric testing. Aim: To cognitively test C-POS within the target population to establish comprehensibility, comprehensiveness, relevance and acceptability. Design: Cross-sectional cognitive interview study following COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) methodology and Rothrock guidance on outcome measure development. Cognitive interviews were conducted using ‘think aloud’ and verbal probing techniques. Setting/participants: Children 5–⩽17 years old with life-limiting conditions and parents/carers of children with life-limiting conditions were recruited from 14 UK sites. Results: Forty-eight individuals participated (36 parents; 12 children) in cognitively testing the five versions of C-POS over two to seven rounds. Content and length were acceptable, and all questions were considered important. Refinements were made to parent/carer versions to be inclusive of non-verbal children such as changing ‘share’ to ‘express’ feelings; and ‘being able to ask questions’ to ‘having the appropriate information’. Changes to improve comprehensibility of items such as ‘living life to the fullest’ were also made. Parents reported that completing an outcome measure can be distressing but this is anticipated and that being asked is important. Conclusion: Cognitive interviewing has facilitated refinement of the C-POS, especially for non-verbal children who represent a large proportion of those with a life-limiting condition. This study has enhanced the face and content validity of the measure and provided preliminary evidence for acceptability for use in routine practice.


Background
It is estimated that each year 21 million children and young people worldwide ('children') with life-limiting or life-threatening ('life-limiting') conditions require input from palliative care services. 1Life-limiting conditions are those for which there is no hope of cure, and from which children will die.4][5] Development of the African Children's Palliative Outcome Scale (C-POS) began before recent accepted guidance on patient-reported outcome measure development was proposed and the content is informed by the African healthcare context. 67][8] Item generation and content validity were informed by a qualitative semi-structured interview study, 9,10 a systematic review to inform optimal recall period, response format and administration mode, 11 a Delphi survey to achieve consensus on priority outcomes and an item generation meeting. 12This has resulted in five versions of C-POS -two parent/proxy versions (child <2 years and ⩾2 years) and three child versions (5-7, 8-12 and 13-17 years or cognitive equivalent).The concepts explored are the same in each version, with simpler language for younger/less cognitively able children (Table 1).Child versions are named after planets to avoid stigma associated with using chronological age in children with life-limiting conditions, many of whom have developmental delay.Following item generation, it is recommended that item improvement processes are conducted to ensure relevance, comprehensiveness and comprehensibility and to reduce response error. 6,13gnitive interviewing is an established method of item improvement of PCOMs, recommended by COSMIN. 7onducting cognitive interviews has been shown to yield three times more problematic items than identified by using item non-response. 14Participants are asked to complete the measure while 'thinking aloud' with or without verbal probing. 15'Think aloud' cognitive interviews require participants to verbalise their thought processes while answering questions, and the interviewer intervenes as little as possible. 13This can provide insight into the type of information participants retrieve from memory. 16Verbal probing is often used in conjunction with 'think aloud' techniques, to locate problems with questions 15,17 and assess comprehension of a measure. 16his study aimed to use cognitive interviews to test C-POS for comprehensibility, comprehensiveness, relevance and acceptability within the target population.

Methods
Cross-sectional cognitive interview study following COSMIN and Rothrock guidance. 6,7This study is reported according to the Cognitive Interview Reporting Framework. 18

Setting
Participants were recruited from ten NHS sites and four children's hospices within England, Scotland and Northern Ireland.

Sampling and recruitment
Inclusion criteria: Children 5-⩾17 years living with a lifelimiting condition and parents/carers ('parents') of children 0-⩾17 years with a life-limiting condition who can speak and/or read English Exclusion criteria: Children unable to communicate wishes themselves or via their parent, enrolled in another What this paper adds • • This paper demonstrates that it is possible to develop a single patient-centred outcome measure for use with children of all ages with a wide range life-limiting conditions and their families.study, clinically unable or unwilling to provide consent/ assent.Parents unable or unwilling to provide consent.
We aimed to test each version with at least seven participants as per COSMIN recommendations, 7 with any amendments being made after four interviews.If amendments were required, we aimed to test the revised version with three further participants, with the final version being tested with at least three participants.Participants were purposively sampled to ensure maximum variation in children's age and diagnosis.

Data collection
Interviews were conducted face-to-face or virtually via Microsoft Teams, dependent on participant preference and COVID-19 guidance.Prior to interview, participants completed a consent process and short demographic questionnaire.Participants were then given a demonstration of the 'think aloud' technique and encouraged to take part in a practice task, whereby they were asked to count the number of windows in their house while thinking aloud. 15This task helped to build a rapport between the interviewer and participant.Interviews were conducted using the 'think aloud' technique with concurrent verbal probing. 15,19,20A Supplemental topic guide was used to ensure pre-scripted probes regarding item wording, recall and response format.After all C-POS items had been cognitively tested participants were asked questions regarding their acceptability, relevance and comprehensiveness.This included asking whether any items were inappropriate and should be left out, whether any important items were missing and should be added to C-POS, and whether the number of items and time taken to complete the measure was acceptable.Spontaneous probes were used to explore any hesitations or difficulties. 14ersion selection for child participants was guided by developmental age, allowing the child and parent to choose the most appropriate version.Previous work highlighted challenges in ascertaining at what chronological/ developmental age children can reliably use longer recall periods and more complex response formats. 11In all children >8-years-old, both a 3-and 5-point response scale format was tested, along with recall periods of the past week and yesterday and today (Figure 1).Children <8-years-old tested a 3-point response format and recall of yesterday and today.The use of emojis to anchor responses was informed by evidence from previous work 12 with a smiling emoji representing the positive outcome for the item.All child versions of C-POS had eight questions.
Proxy versions of C-POS had a recall period of the past week and a 5-point Likert response format (Never, Almost never, Sometimes, Often, All of the time).Proxy versions contained eight questions about the child (identical concepts to self-report items), and five about the family.
Interviews were conducted by LC, DB and HS (experienced qualitative researchers) and DH (new to qualitative research).Interviews were audio recorded, transcribed verbatim and pseudonymised.

Data analysis
Analysis followed the five steps of Framework analysis recommended by National Centre for Social Research for analysing cognitive interview data: familiarisation, constructing a thematic framework, indexing and sorting, charting and mapping/interpretation. [20][21][22][23] Data from all interviews was summarised into a single data set using Microsoft Excel.][22][23] Areas of investigation were based on Tourangeau's fourstage model of survey response -comprehension, retrieval, judgement and response 21 and Willis' coding system for classifying questionnaire problems. 22nterviews were analysed independently by two members of the team.Data were reviewed after each item had been tested with four participants and findings were discussed in regular research team meetings.Any problems identified and potential changes to C-POS were discussed and agreed as a group.

Ethics and consent
Ethical approval was granted by the Bloomsbury ethics committee (HRA: 21/LO/0282).Participants over 16 years

Sample characteristics
Forty-eight individuals (36 parents; 12 children) participated between June 2021 and April 2022 (Table 2).Diagnoses are classified according to ICD-10-chapter headings to preserve participant anonymity, as many lifethreatening conditions are rare. 24

Main findings
C-POS was tested over two to seven rounds dependent on version.Interview findings and subsequent changes made to C-POS are displayed in Tables 3 and 4. All participants were able to participate in the cognitive interview process after the practice task.Some children under 8 years needed direction and explanation from a parent during the first few questions.Understanding of the 'think aloud' task then improved.Participants 5-to 7-years-old (or cognitive equivalent) frequently needed an explanation of items prior to responding.

Findings related to all versions
In round 1 participants responded to items in terms of frequency, so quantifiers (how much, how often) were removed from the stem of questions, so they began with 'Have you been affected by'. ... In subsequent rounds, participants responded in terms of how much they had been affected by a symptom or concern.
The original response format in the child versions was amended so that 'most of the time' became 'always'.Children felt this fitted with 'never' at the other end of the scale and wanted definitive 'always' and 'never' response options.

Child and young person versions
The child versions of C-POS were tested over two rounds.
Comprehension.In round 1, two items posed comprehension problems.Children 5-to 12-years-old did not understand the term 'live life to the fullest'.This was amended to 'enjoy life as much as possible'.Those 5-to 7-years-old found difficulty understanding the term 'sharing feelings'.This was changed to 'been able to talk to people'.These were tested in round 2 and were understood well.

'I:
And so, when we ask about sort of live your life to the fullest, what are you thinking about in those things?
P: I don't know.' (CHI8-12d) Retrieval.All participants 13-to 17-year-old could use a recall period of the past week.They preferred this option to 'yesterday and today'.

'I think again it [yesterday and today] isn't a long enough timescale so I do prefer the past week' (CHI13-18a)
Retrieval ability varied in 8-to 12-year-olds, with some only being able to recall yesterday and today, and some responding to things that had happened since the start of the week or weekend, rather than the past 7 days.Those 5-to 7-years-old sometimes struggled with the concept of yesterday, reporting only about the current day.No changes were made to the recall period as the intention is to use the version most appropriate to the child's developmental ability.
Judgement.Participants 5-to 7-years old initially needed some support from their parent to formulate a response.The ability to respond independently improved as they moved through the measure.Items regarding 'usual activities' and 'things you enjoy' posed difficulties for some participants.There was uncertainty about whether the benchmark for responding should be in those 8-to 17-years old: activities undertaken pre-diagnosis, current activities or activities they were able to undertake before the COVID-19 pandemic.Response.All participants 13-years and over could use a 5-point response format and expressed preference for this option over the 3-point format.
'Um, because there's more options.Easier to find one that. . .

makes sense.' (CHI13-18c)
There was variability in those 8-to 12-year, with some being able to use a 5-point response format and some managing better with a 3-point format.
Participants could describe in which circumstances they would choose specific response options, suggesting understanding of how to use these: 'Because, in my week, I get, I have Monday to Friday of radiotherapy, so that's every single day, and then I get Saturday and Sunday off.So then I think, I get like, a good few days of doing things that I really enjoy.But then, if I think about yesterday or today, that could be a Sunday or a Saturday, and I would say, Yeah, that's often.Then I would say, if it was a Tuesday or Wednesday, I would say sometimes.'(CHI8-12e) Acceptability, relevance and comprehensiveness.All children found the content and number of questions acceptable.All items were reported to be important.No additional items were identified.The emojis were wellliked and made selecting a response option easier.

Parent/carer versions
The CPOS version for parents of children ⩾2 years was tested over seven rounds: the version for parents of children <2 years over four rounds.

<2 years
No problems >2 years -participants struggled to formulate an answer if their child was nonverbal.This improved when the wording was changed to 'had the appropriate information for them about their condition'.

No problems
As above Quantifiers as above.

>2 years changed to
'had the appropriate information for them about their condition'

Items about the family
Comprehension.Two items posed comprehension problems.When responding to the items regarding 'reaching full potential' (<2 years) and 'being able to "live life to the fullest"' (>2 years), participants compared their child to healthy children.It was intended that this question was answered within the context of their child's life-limiting condition.Amending this to 'live life to their fullest' across both proxy versions allowed greater comprehension.The item 'having support planning for care' was intended to ask whether participants felt supported in advance care planning decisions.However, this was interpreted to mean planning for day-to-day care needs.Amending this to 'planning for future care' allowed the item to be understood as intended.
Retrieval.Participants had no problems with retrieval.Some suggested a longer recall for the 'information needs' item as these needs are often higher at diagnosis.This was not changed as recall period is the same for all items.
Judgement.There were several issues with judgement, particularly for parents of non-verbal children.'Sharing feelings' and 'asking questions' were amended to 'express feelings' and 'having appropriate information' to be more inclusive of the range of life-limiting conditions.Judgement difficulties were also identified with the 'crying more than usual' item.Participants were unsure whether to include crying due to frustration, temper or falling over in their response.They also had difficulty deciding what 'more than usual' meant.This item was amended to 'displayed signs of worry or anxiety' which improved judgement.The item 'tiredness and fatigue' also posed difficulty.These were felt to be two different concepts requiring different responses.The item was changed to 'been able to get enough sleep'.
'So, I think tiredness and fatigue, sometimes can mean a bit different.Like I think tiredness can, like, you could be like lack of sleep.And fatigue could just be like exhaustion. ...'(PAR0-2a) Response.The 5-point Likert scale was easy to use.Some participants suggested that a 'not appropriate to my child' option was added for those who felt that their child had no understanding of, or could not articulate, concepts such as worry and information needs.
Acceptability, relevance and comprehensiveness.Measure length was acceptable.Participants found some questions upsetting to answer (particularly 'planning future care'), however none felt any items should be removed.
'You can't not ask those questions because they're important questions.' (PAR0-2g) 'Some of them [questions] could be upsetting but that goes back to what I said, this is just an upsetting situation.'(PAR2-18l) Several participants suggested adding an additional item regarding psychological or emotional support for the family.They felt this was not incorporated in the item regarding support needed to care for their child.This was added as item 14.
Finally, some participants expressed that when completing a PCOM about their child, they expected it to be emotive at times.They felt that all items should be included.They also felt that the process of cognitive interviewing made the questions more emotive than they would be in a clinical scenario.
'I think that anyone who's got a child in palliative care, who's agreed to be in a study about palliative care, should know that they might get a little bit upset while answering questions cos it's sad, you know?' (PAR2-18l) 'But it's because of the way that we're having to discuss our thought processing about why we're -that is -that -that makes it more emotive in this scenario than it might have otherwise done' (PAR2-18q)

Final C-POS versions
All versions of C-POS were tested in their final format as per COSMIN recommendations. 7Final C-POS versions are shown in Table 5. Recall period and response format for each version are informed by the results of cognitive testing.Younger children require a short recall and simple response format and children 13-to 17-year can recall the past week and use a 5-point response format.Children 8to 12-year showed variability in which recall and response format they could use, recalling the past week seemed to be easier than using a 5-point response scale.

Discussion
Cognitive testing of C-POS has demonstrated that for most participants items, response options and recall period were acceptable.All items in C-POS were found to be relevant for the population it is intended to be used with.Wording was amended for some items to improve comprehension and ensure they were interpreted as intended.Several questions posed difficulty for parents of non-verbal children.For example, concepts such as whether a child had been 'able to share their feelings' or 'ask questions'.Amending wording to use terms such as 'express feelings' and 'had appropriate information for them' allowed C-POS to be more inclusive of the range of children with life-limiting conditions, many of whom have communication difficulties. 25An option for 'not appropriate to my child' was added for those for whom a proxy response could not be determined.All of these changes have enhanced the acceptability and comprehension of C-POS items and the measure as a whole.C-POS length and content was acceptable to all participants.Initially many item stems began with 'How much. ..' or 'How often..'.Cognitive testing identified that participants were responding to these items in terms of frequency of a symptom or concern.Removing stems so questions began with 'Have you..' allowed participants to answer in terms of how a symptom or concern has affected the child, which is the intention of C-POS.
Children were able to participate in cognitive testing and provide valuable insights into the design and content of C-POS.There is strong evidence that, from the age of 8-year old, children can meaningfully self-report on their own health-outcomes. 26There is conflicting evidence on when younger children can do this. 27Evidence suggests that those 4-6 years can report on concrete domains such as pain but may have difficulty with more complex or emotional aspects. 26,28e found that children aged 5-7 years (or cognitive equivalent) often needed an adult to explain questions to them before they could choose an appropriate response and undertake the 'think aloud' technique required for the study.This finding supports recommendations that when younger children are completing PCOMs they are administered with initial adult support so that more difficult concepts can be explained. 29Previous studies that have shown that children with life-limiting conditions prefer to complete a measure in the presence of an adult in case they have questions or need support. 12e found some instances of social desirability bias in children 5-7 years (or cognitive equivalent) who gave the answers they thought the researcher wanted to hear (e.g.saying they had not experienced any pain when in fact they had).1][32] In our study, with further explanation, these participants were able to give appropriate responses.We found no evidence from the 'think aloud' responses that participants were choosing the response they liked the look of.This supports evidence that children of this age can be suggestible and may give the response they think interviewers want to hear. 28It is important children are encouraged to express their own thoughts when self-reporting their health outcomes, and clearly told that there are no right or wrong answers.
Children 5-7 years (or equivalent) sometimes responded to items in terms of the 'here and now', rather than yesterday or today.This supports evidence that young children may not always understand the difference between the past, present and future and may engage in 'scripting' whereby they respond about what usually happens and regularly occurring events. 26We found that all children were able to use a 3-point Likert response format.This contrasts with previous research which suggests that children <8 years cannot use Likert response formats and a dichotomous Yes/No response is preferable. 33We were keen to test whether children 5-7 years could use a 3-point Likert scale, as a dichotomous format limits the responsiveness of a measure. 33Future inter-rater reliability testing of C-POS will help to establish whether any additional recall and response format issues are occurring.
A small number of parents in our study found some items upsetting to answer but acknowledged that they were in a challenging and emotive situation, so this was to be expected.They did not want these items removed as they were important to address with the health care team, thus demonstrating the relevance of all items to their child's situation.Parents also reflected that the cognitive interviewing process meant they had to think about items in more detail than they would in a clinical scenario, which could have exacerbated any distress.Parents were keen to share that they expected to experience some mild distress when taking part in a palliative care research study, and this was acceptable.This finding has implications for ethical review of future studies and supports previous findings that while caregivers can find taking part in paediatric palliative care research difficult at times, they still want to participate. 34,35hildren and young people in our study had no suggestions for further C-POS items, suggesting that the measure is comprehensive and contains items on the most important symptoms and concerns in this population.Parents felt that an extra item was required asking about psychological and emotional support for themselves and the family.Ensuring that parents and family members of children with life-limiting conditions have adequate psychological and emotional support is essential to the well-being of the unwell child. 36It was initially intended that the item on 'support needed to provide care' would incorporate this.However, this was not how participants interpreted this item, which they comprehended as the more practical aspects of care provision, such as equipment needs and respite care.Therefore, an extra item was added to improve the comprehensiveness of the C-POS measure.

Strengths and limitations
During this study C-POS was tested in the target population using accepted cognitive interview techniques.We have demonstrated that with support, children with lifelimiting conditions can participate in cognitive testing of PCOMs.We have also added to the limited evidence on the age/developmental stage at which children can use more complex response formats and longer recall periods, highlighting that between the age of 8-12 years (or cognitive equivalent) ability to use these varies. 11Another strength of the study is that the C-POS version for parents of children over 2 years was tested over multiple rounds to ensure items worked for parents of non-verbal children.
Although our sample did not contain any children under 7 years old, two participants were reported by their parents to have a cognitive age of five.This may have impacted our findings, particularly regarding the comprehensibility of subjective items such as worry and the finding that children in this age group could use a 3-point Likert response format.Exploration of future psychometric data is required to affirm these findings.
The majority of participants in our study were white British, whereas the prevalence of life-limiting conditions is higher in minority ethnic groups in the UK. 24C-POS has initially been constructed in the English language, so those that could not speak or read English were excluded from participating, which may in part explain participant demographics.

Next steps
Further research is required to psychometrically test C-POS followed by development of an implementation plan.Once validated, cultural adaption can be undertaken to widen the reach of C-POS, and to enable outcomes to be measured in all children with life-limiting conditions and their families.

Conclusions
This study demonstrates the value of cognitive testing as a stage of PCOM development, as we found several issues that needed addressing.This has strengthened how problematic items are comprehended and reported.C-POS has been demonstrated to be acceptable and relevant to the target population.Comprehensiveness of the proxyreported C-POS was enhanced by the addition of an extra item.Finally, this study demonstrates that it is feasible to measure complex multidimensional outcomes for a highly vulnerable, complex population who are often excluded from research.
• • Minor adaptations were required to ensure comprehension and relevance to children of different chronological and developmental ages.A proxy version is required for non-verbal children with adaptations to some items required to ensure relevance.• • Children with life-limiting conditions are able and willing to participate in cognitive interview studies, even if they have communication difficulties or developmental delay.This paper highlights the importance of cognitive testing of patient-centred outcome measures, a step of measure development that is not always conducted.• • It is essential that parents of children with life-limiting conditions are given the opportunity to complete outcome measures, even if some items cause distress.

Table 3 .
Main findings from cognitive interviews with children and young people.
a Or cognitive equivalent.