Developing theory-based text messages to support retention in clinical trials: A mixed methods approach

Background: Returning a trial questionnaire is a behaviour affected by a range of psychological and contextual factors. Previously tested Short Message Service (SMS) messages to prompt questionnaire return have not addressed these factors, and have not been characterised by established taxonomies of behaviour change techniques (BCTs). Purpose: We aimed to develop acceptable theory-based SMS messages, with fidelity to four BCTs, to support participant understanding of the consequences of not returning trial questionnaires. Methods and Results: We initially developed 32 messages. Ten behaviour change experts assessed message fidelity to the intended BCT (Study 1a). All messages had appropriate fidelity to the intended BCT (mean ratings = 6.8/10 [SD = 0.6) to 7.5/10 [SD = 0.3]). Study 1b, a focus group with five patient representatives, recommended removing the BCT ‘comparative imagining of future outcomes’ (4 messages), two further messages be removed, and amendments to five messages. In Study 1c, 60 breast cancer survivors rated all remaining 26 messages as acceptable (mean = 3.8/5 [SD = 1.2] to 4.3/5 [SD = 0.8]). Twelve behaviour change experts rated the fidelity of the 26 messages to intended BCTs (Study 1d); all messages had appropriate fidelity (mean ratings = 6.1/10 [SD = 2.4] to 6.9/10 [SD = 1.4]). Conclusions: In these studies, we developed 26 SMS messages that were acceptable to the intended recipients and had sufficient fidelity to the intended BCTs. This approach could be taken to design interventions supporting behaviours needed for the successful delivery of clinical trials. The messages are available to research teams who can evaluate them in Studies within Trials.


Introduction
Identifying strategies to support retention of research participants in clinical trials is a recognised priority. 1 Low retention rates can limit the internal and external validity of trials, 2,3 increasing the risk of research waste and reducing the likelihood of patient benefit. Poor retention is a particular problem for some trial designs, such as 2 k factorials, where missing data can threaten the integrity of the trial. 4 Trials utilising patient-reported outcomes use a range of strategies to improve retention and questionnaire response rates. 5 These include monetary rewards, newsletters, providing a pen, among many others. Randomised comparisons of these approaches indicate some may improve retention, but the certainty of these findings is moderate at best. 6 Most existing approaches have not construed trial retention as a behaviour, and therefore interventions are often not framed in terms of behaviour change. Short Message Service (SMS) prompts are one technique that have been shown to support behaviour change across a range of other settings, including medication adherence, 7,8 smoking cessation, 9 physical activity, 10 weight loss, 11 and chronic disease management. 12 SMS messages could offer a low cost and implementable strategy for encouraging questionnaire return. However, evidence supporting behaviourally focused SMS interventions in trial retention is lacking. As such, there is a need to design effective interventions to improve retention of participants in clinical trials. 13,14 A number of studies within a trial (SWATs) have investigated the effect of electronic reminders (including SMS messages) on questionnaire return. [15][16][17][18][19][20] Despite nearly 60% of surveyed UK based Clinical Trial Units using this strategy, 21 there is mixed evidence to support the use of SMS for improving trial retention. A Cochrane review reported the relative effect of electronic reminders sent as pre-notifications and reminders for trial retention was small. 6 However, relatively few trials were included within the meta-analyses, and the certainty of the evidence was low or very low. The lack of effect may be explained by the narrow focus of the SMS message content. Examples include, 'We hope you can take a few minutes to complete this and return it to us' 20 or 'Your answers are important; so please help by returning it as soon as you can'. 19 Such messages focus on a simple prompt, and do not consider the wide range of factors that affect questionnaire return. [22][23][24][25] Efforts are underway to improve our theoretical understanding of the barriers to trial retention. 23,24 This theoretical understanding can serve as a useful basis for the focus of the intervention. For example the determinants of trial retention have been mapped to the Theoretical Domains Framework (TDF) where key barriers include poor knowledge about the frequency of questionnaires, competing priorities, and beliefs about consequences of performing and not performing the behaviour. [23][24][25] Beliefs about the consequences of a questionnaire return could be a potential target for SMS messages supporting one aspect of trial retention, and would be an advance on simple prompts. In qualitative interviews with participants across five trials, some participants could not recognise any negative consequences of not returning a questionnaire, and others were unsure what difference their contribution made to a trial. 24 In a separate qualitative study investigating barriers to questionnaire return, participants reported greater satisfaction with returning a questionnaire if they knew their questionnaire would positively contribute to research. 25 Supporting participant beliefs about positive consequences of questionnaire return and negative impacts of non-completion may improve trial retention.
A further weakness of existing interventions designed to support questionnaire return is that they do not conceptualise questionnaire return as a behaviour. As such, there is little consideration of established approaches for developing and evaluating behaviour change interventions. 26 This results in atheoretical interventions that lack generalisability. Considering questionnaire return as a behaviour, and using formal and established intervention development approaches from the field of behaviour change may encourage clinical trial specialists to specify their interventions according to existing behaviour change taxonomies. 27 Such an approach would build a cumulative evidence base which could further our understanding of the mechanisms through which this behaviour can be changed, allow interventions to be compared using a standardised taxonomy, and improve the generalisability of the intervention in the long-term.
We aimed to follow an established procedure for developing a pool of theory-based SMS messages targeting specific behaviour change techniques (BCTs) to support understanding of the importance of questionnaire return in trials. We evaluated the messages for fidelity to the intended BCT in an expert sample and their acceptability to prospective recipients.

Method
We undertook a series of iterative studies to develop the SMS messages ( Figure 1). We adapted our approach to develop the message content from Bartlett et al. 28 to accommodate social distancing requirements in place at the time of undertaking the study. Ethical approval for all studies was granted by the University of Leeds Research Ethics Committee for the School of Medicine (MREC 20-038). Data from studies 1a, 1c and 1d will be uploaded to the University of Leeds data repository after publication of the data. Data from study 1b is not available to share because membership of the group may be known to others, and therefore the data cannot be anonymised.

Identification of the behaviour change techniques
To identify candidate BCTs addressing beliefs about consequences, we drew upon a review linking BCTs to mechanisms of action derived from the TDF. 29 This review linked five BCTs to beliefs about consequences. We discarded one BCT as the trial team did not consider this appropriate in this context (Information about emotional consequences [5.6]), as there are unlikely to be emotional consequences of questionnaire completion. The four remaining BCTs were included as candidates; information about health consequences [5.1], information about social and environmental consequences [5.3], pros and cons [9.2] and comparative imagining of future outcomes [9.3]. 27

Message generation
To generate the SMS messages, the name, description and examples of each BCT (taken from 27 ) were sent to members of the research team (SS, SG, ER, LH). Guidance was given that messages should be up to 160 characters to ensure they could be sent as a single message, generic enough to implement across different trials and should aim to improve understanding of trials and trial processes. We worked independently to each generate as many SMS messages as possible. One author (ER) collated all responses and removed messages that did not fit with the intended content or the targeted BCTs. The messages were circulated to the same members of the research team, and a group consensus was reached after minor amendments and removal of messages.

Study 1a: Expert survey of initial fidelity
Aim: Using SMS messages developed by the research team (SS, SG, ER, LH) aimed at addressing beliefs about the consequences of not returning study questionnaires, our goal was to evaluate the fidelity of the SMS messages to the intended BCTs using a sample of experts in behaviour change.
Recruitment: We identified experts in behaviour change from the United Kingdom through the research team's academic networks and searching university websites and Twitter profiles. We identified 25 suitable participants, all of whom were invited via email. We emailed a single reminder to nonrespondents after 1 week. Participants were offered £100 to participate, which included attending a half-day online workshop for a related study that will be reported elsewhere.
Procedure: A survey containing the generated messages was sent to the experts in behaviour change. Following e-consent we collected basic information about their background, including demographic information. Participants were then presented with each BCT title and a description from the BCT v1 taxonomy, 27 alongside the pregenerated messages. They were asked to rate how well each message reflected the BCT in question using a single item with a 10-point rating scale: 1 (not very well) to 10 (very well). To limit the length of the survey, the message was only assessed for the BCT it was intended to target.
Analysis: Participant characteristics and backgrounds were described. The mean and standard deviations (SDs) of survey responses for each message were calculated. A priori, we considered messages rated below 5.5 by experts for removal from the message bank.

Study 1b: Patient and public involvement group
Aim: We used an established Patient and Public Involvement (PPI) group to ascertain the acceptability of the content of messages to women with experience of breast cancer.
Recruitment: Women were members of an existing PPI group for a pilot trial evaluating the feasibility of evaluating interventions to support medication adherence in women affected by breast cancer. Women were recruited into the PPI group by advertising through the Yorkshire Cancer Community, a charity supporting people affected by cancer in the Yorkshire area. Group members were paid £37.50 each to attend for 90 minutes.
Procedure: The group was attended by two members of the research team (SG, ER) who were familiar to the members. E-consent was taken prior to the meeting, and consent was confirmed verbally at the start of the meeting. All participants were sent a copy of the SMS messages 2 weeks prior to the meeting. The group were presented with the title and descriptions of the relevant BCTs and SMS messages that were retained from study 1a. The group were asked to discuss the acceptability of sending brief messages to encourage and support questionnaire completion generally, as well as comment on the specific wording of messages.
Analysis: The meeting was recorded and transcribed verbatim. One author (SG) reviewed the transcript to identify suggestions made by the PPI group on how to amend the content of the messages. Based on the suggestions, amendments to the messages were made by one author (ER) and agreed through group consensus of the research team (SS, SG, ER, LH).

Study 1c: Patient acceptability survey
Aim: We aimed to assess the acceptability of the SMS messages remaining for studies 1a and 1b to women affected by breast cancer using an online survey.
Recruitment: Sixty women with a personal history of breast cancer were recruited by a market research company (Dynata) commissioned by the research team. The women were identified on the basis that they had oestrogen receptor-positive (ER+) breast cancer, as the host trial that will evaluate these messages involves this patient group. Women received a small monetary incentive from Dynata for survey completion.
Procedure: Dynata sent out the survey link to their panel of profiled respondents who have signed up to participate in market research. People interested in participating who were sent the survey link were asked to confirm in a single screening question on the first page of the survey whether they were over 18 and have received a diagnosis of ER+ breast cancer. If they were eligible, they were asked to provide e-consent on the next page of the survey prior to being shown the survey questions. The survey contained the SMS messages that remained after study 1a and 1b. Participants were asked to rate the SMS messages using a 5point rating scale assessing how acceptable they felt the message would be to them if they were receiving it (1 = completely unacceptable, 5 = very acceptable). The order of message presentation was randomised.
Analysis: We summarised participant characteristics. Mean scores and SDs for acceptability of each SMS message were calculated. An overall acceptability score was additionally calculated for each BCT by averaging the fidelity scores of all messages relating to each BCT. A priori, we considered messages for removal from the message bank if they were rated as below the midpoint (3) on the overall acceptability score.
Study 1d: Expert reassessment survey of fidelity Aim: We aimed to assess the fidelity of the SMS messages remaining after any amendments were made in studies 1a, 1b and 1c to the intended BCTs.
Recruitment: We approached the pool of individuals identified but not approached for Study 1a, in addition to those who did not respond to the invitation, or who could not participate in study 1a. All were considered to be experts in behaviour change based on their social media or academic website profiles. We emailed 41 individuals and offered a £25 Amazon voucher upon survey completion.
Procedure: Participants were sent a link to the survey, which provided information about the study and the procedure for providing e-consent. Participants were asked to rate each remaining SMS message in terms of its fidelity to the intended BCT using a 10-point rating scale: 1 (not very well) to 10 (very well). They were presented with a short description of a BCT taken from the BCTTv1, and the corresponding SMS messages for that BCT. This was repeated for each BCT included.
Analysis: We summarised participant characteristics and backgrounds. The mean and SDs of survey responses for each message were calculated. We removed messages rated by the experts as scoring below 5.5 on the fidelity item.

Message generation
The research team generated 32 messages for the four BCTs (Table 1, Figure 1).

Study 1a: Expert survey of initial fidelity
Participants: Ten experts in behaviour change participated in the online survey. Eight were full-time research scientists, while two were healthcare professionals who also undertook research. The average time spent working in research was 16 years (SD = 5), and most (80%) described behaviour change as central to their work.
Messages and decision-making: No message scored below 5.5, and therefore all 32 messages were considered to  have adequate fidelity for the BCT they were intending to target (Table 1, Figure 1).

Study 1b: Patient and public focus group
Participants: Five women aged between 41 and 79 (mean age = 54 years, SD = 15) participated in the focus group ( Table 2). All women were daily mobile phone users, and all used SMS messages at least once a week. Messages and decision-making: We implemented all suggestions from the group where there was no disagreement. A total of six messages were discarded across four BCTs (Figure 1, Table 3), and exemplar quotes supporting these decisions are provided in Table 3. For the messages relating to the BCT 'comparative imagining of future outcomes' the group agreed this BCT should not be targeted. All four messages targeting this BCT were therefore discarded. Following suggestions from the focus group, we amended the wording of five messages. For example following a suggestion to start and finish with a positive statement where possible, 'Completing questionnaires can be time-consuming, but by doing so you are contributing to cutting-edge research. It's worth it to help improve patient care!' was amended to 'By completing your questionnaire you are contributing to cutting-edge research. We know it can be time-consuming, but it's worth it to improve patient care!'.

Study 1c: Patient acceptability survey
Participants: Sixty breast cancer survivors participated in the online acceptability survey ( Table 2). The majority were white (82%), and 42% were educated to degree level. Most women (75%) used their mobile phone at least once a day, and half (52%) used SMS messaging more than once a day.
Messages and decision-making: Table 4 shows the mean acceptability ratings for the BCTs overall, and the individual items. Overall, the messages within each BCT were considered acceptable to the breast cancer survivors. No individual messages or BCTs scored below the midpoint on the acceptability scale (3), and therefore no messages were removed as a result of this survey. Only one message scored less than four on the acceptability item 'Not receiving enough questionnaire responses could make our results less valid which could mean the research has less impact. Please return yours' (mean = 3.8, SD = 1.2).

Study 1d: Expert reassessment of fidelity
Participants: Twelve experts in behaviour change participated in the online survey, all of whom were different from those participating in Study 1a. Eleven were full-time research scientists, while one was a researcher and a healthcare professional. All participants described behaviour change interventions as central to their work (7/12) or  somewhat central (5/12). The average time spent working in research was 9 years (SD = 3).
Messages and decision-making: Table 4 shows the overall mean fidelity ratings from the behaviour change experts for each BCT. Overall, the messages were considered to have appropriate levels of fidelity to the intended BCT. No messages were rated below 5.5 out of 10 with regard to fidelity, and therefore no messages were removed for consideration. The highest rating message was from the 'Information about Health Consequences' BCT: 'Your [trial] questionnaire is due, return this to help research stay up-to-date, so that patients receive effective treatments and healthcare initiatives' (mean = 8.2, SD = 2.1). The lowest rating message was from the 'Information about social and environmental consequences' BCT: 'You may feel it doesn't matter if one person's questionnaires are incomplete, but each one ensures our research represents a wide range of people' (mean = 5.8, SD = 2.4).

Discussion
In this iterative series of studies, we developed a pool of 26 SMS messages considered to be acceptable to the intended recipients and which have sufficient fidelity to the intended BCTs as rated by experts in behaviour change. The SMS messages are available to research teams who can evaluate these messages further within SWATs to establish if these messages are effective at improving questionnaire completion, and whether specific messages are more effective than others.
Previous studies investigating SMS pre-notifications and reminders have indicated the effects of this strategy are mixed. 6,[15][16][17][18][19][20] This is surprising considering the effectiveness of SMS messages to support behaviour change across a wide range of contexts. [7][8][9][10][11][12] One explanation could be that the existing interventions tested in SWATs largely focus on basic prompts and cues, and have not explicitly made use of behaviour change theory in their development. 30 Therefore, BCTs implicitly targeted in existing interventions were unlikely to be theoretically informed. 30 Further evaluation of SMS messages targeting BCTs that have been chosen with an explicit theoretical rationale will improve our understanding of whether this strategy is broadly effective. Comparisons between messages will help establish the incremental gains that might be observed by targeting one BCT over another. Think about of how you will feel when you contribute to cutting-edge research by completing your questionnaire. We know you're busy, but your feedback is vital.
4.1 (0.9) 6.0 (3.1) A pro to completing the questionnaire may be holding up your end of the agreement. A con may be the time it takes. We hope you agree it is worth it.
4.0 (1.0) 6.0 (2.5) BCT: behaviour change technique. a Acceptability scores ranged from 1 to 5 with higher scores indicating better acceptability of the message. b Fidelity scores ranged from 1 to 10 with higher scores indicating better fidelity to the intended BCT.
We adapted an approach for developing SMS message content to support behaviour change, 28 and focused on developing SMS messages to support participant understanding of the value of questionnaire return, and the potential issues of missing data. However, given the wide range of factors affecting questionnaire return, 22 a similar approach could be undertaken to develop messages targeting other theory-informed factors. This could include targeting determinants of trial retention such as behavioural regulation, goals and memory, using BCTs such as problem solving and action planning. A process of continuous optimisation with regard to the content of these messages may help to improve participant retention, and offer a cumulative, generalisable basis for developing the evidence base in this field.
Our study had limitations. We plan to register a SWAT to evaluate three of these SMS messages within host trials involving women affected by breast cancer. As such, the content of the SMS messages was reviewed by this population in both the patient and public focus group and patient surveys. The content of these messages could therefore be more or less acceptable to different clinical populations. Our selection of BCTs was guided by evidence of previous strategies used for addressing beliefs about consequences. 29 However, this is just one factor of many that may affect questionnaire return, and further research quantifying the relative importance of known determinants of this behaviour would be a useful next step. [22][23][24][25] We evaluated the fidelity of the messages to the BCTs we had planned to target. This was a pragmatic decision due to the volume of messages we developed. However, it is likely that some of the content also targets similar BCTs, but we did not investigate this.
In summary, we used an iterative series of studies to develop a pool of SMS messages that aim to support questionnaire return within clinical trials. The messages developed use the BCTs of 'Information about Health Consequences', 'Information about social and environmental consequences' and 'Pros and Cons' and specifically focus on supporting participant understanding of the consequences of returning questionnaires to enhance validity and maximise patient benefit. The messages were found to be acceptable to the intended recipients, and faithful to the intended BCTs. SWATs are now needed to evaluate the effects of these messages on questionnaire return.