Process evaluation protocol for the BeST? Services trial

Background: Outcomes for children who have been received into state care due to maltreatment or neglect are very poor in comparison to the general population. A mental health focussed intervention is being trialled in the UK which seeks to improve outcomes for such children. Although the main study is concerned with outcome evaluation (child mental health measures), a robust process evaluation adds nuance to the findings by examining the relationship between the interventions, the participants and the context-thereby determining what works, for whom and in what context. This paper sets out the protocol for that process evaluation. Methods: The process evaluation is embedded within a Randomised Control Trial and uses Realist Evaluation as its theoretical framework. It sets out the mechanisms of change that are used to effect positive outcomes and outlines the various aspects of the context (including service provision and the legal system that provides the statutory basis for involvement with families). Data collection is primarily qualitative and takes place with a large group of stakeholders. The analysis of the interplay between context, mechanisms and outcomes will provide a richer understanding of the main trial outcomes. Discussion: The nature of the interaction between the social, legal and practice context is complex. The use of a variety of methods including case studies, focus groups and analysis of routine data are justified and it is argued that they will provide for greater understanding of the nature of the interactions within such a challenging context.


Background
The prevalence of mental health difficulties in children who have been placed in out-of-home care is significantly higher than that of children who have never been looked after (Meltzer et al., 2003;Schoemaker et al., 2020;Vasileva & Petermann, 2017). The impact on care experienced children is not only a result of maltreatment but may also be influenced by the way the support and care they receive is delivered. Factors such as the number of placements children experience or the length of time it takes to be placed can influence their wellbeing (Fawley-King et al., 2017;Perry & Price, 2017;Tarren-Sweeney, 2008). This can have significant lifelong costs to the children and to society: children who are looked after or who have been abused or neglected have poorer overall health (Halfon et al., 1995;Harden, 2004;Turney & Wildeman, 2016), poorer mental health (Engler et al., 2020;Vasileva & Petermann, 2017), are less ready for school (Geiger & Beltran, 2017;Lynch et al., 2017), poorer educational outcomes (Stapleton & Chen, 2020;Tarren-Sweeney & Goemans, 2019) and lower rates of employment (Cassarino-Perez et al., 2018;Kim et al., 2019).
The Best Services Trial (BeST ? ) is a Randomised Controlled Trial (RCT) that seeks to evaluate the clinical effectiveness and cost effectiveness of the New Orleans Intervention (NIM), a mental health intervention for children under the age of five entering care due to maltreatment (i.e. abuse and/or neglect) (Minnis et al., 2010). NIM has had a previous evaluation that compared routine data before and after(C. H. Zeanah et al., 2001), but has never been evaluated in a RCT. For this study, clinical effectiveness is defined using mental health outcomes (primarily the Strengths and Difficulties Questionnaire) and a range of relationship, psychiatric diagnostic and cognitive measures over a 2.5-year period. A health economic analysis aims to determine which intervention provides value for money (Deidda et al., 2018). The trial has two sites, Glasgow and London: two local authorities (Glasgow and Renfrewshire) feed into the Glasgow site and five Local authorities (Barking and Dagenham, Bromley, Croydon, Sutton and Tower Hamlets feed into the London site. Every child under five placed in local authority care due to abuse or neglect is eligible for inclusion in the trial (with the exception of those whose parents are unavailable to take part).
Consented families are randomised to one of two trial arms: Services as Usual (SAU-the Social Work process that children and families normally go through to assess parenting capacity) or NIM.

Process evaluation
Although the evaluation of interventions through RCTs is considered the 'gold standard' in scientific research, RCTs evaluating complex interventions require a detailed process evaluation to further explain results (Moore et al., 2015). The RCT outcome evaluation on its own does not allow for an analysis of the context(s) within which the interventions were delivered, nor of why interventions are, or are not, effective. A robust process evaluation can furnish policy-makers with additional information to help them decide (1) If an intervention is effective in one context, can we be confident that (a) another organisation (or set of professionals) will deliver it in the same way? And (b) if they do, will it produce the same outcomes in new contexts? (2) If an intervention is ineffective overall in one context, can we be confident that (a) the failure is attributable to the intervention itself, rather than to poor implementation? (b) The intervention does not benefit any of the target population? (c) If it was delivered in a different context, would it be equally ineffective?
BeST ? compares two 'complex interventions', defined as 'interventions with several interacting components' or where the implementation or interaction with the surrounding context is viewed as 'complex' (Moore et al., 2015). 'Realist' process evaluation methods are well suited for this type of intervention (Fletcher et al., 2016;Pawson & Tilley, 1997).
The Realist Evaluation framework (Pawson & Tilley, 1997) provides a theoretical framework within which to understand the relationships between context, mechanisms and outcomes. It posits that the 'world is an open system, or a constellation of structures, mechanisms and contexts' (Kazi, 2003), that 'persons are complex particulars and the events of interest-for example, programme outcomes associated with human services-are the result of complex transactions of many different kinds and cannot be explained simply in terms of a causal link between events at the surface' (Kazi, 2003). Figure 1 gives an overview of the component parts of the realist method.
The quantitative findings of BeST ? will be described alongside the Context-Mechanism-Outome Pattern Configurations (CMOCs ) in order to identify the configuration of elements necessary to replicate the results of the trial (Pawson & Tilley, 1997). This will be achieved through qualitative examination of the way contextual factors shape, and are shaped by, intervention mechanisms and outcomes, and how causal mechanisms within the context might sustain the status quo or enhance intervention effects. CMOCs will allow us to identify which elements of the interventions and context are necessary for the desired outcomes-describing what works, for whom and why.  (Pawson & Tilley, 1997).

Interventions used in BeST ?
Irrespective of whether the family is randomised to NIM or SAU, the child will always have an allocated social worker within the local authority who will oversee the management of the child's care. The outcome of the assessments/interventions undertaken by SAU or NIM will be reported to the allocated social worker and it is their responsibility to progress the child's care plan.

NIM
NIM is aimed at 'improving outcomes in young maltreated children in foster care' through provision of 'assessments and intervention to children…who are placed in foster care for abuse or neglect, and to their birth and foster families'(C. H. Zeanah et al., 2001). One of the aims of BeST ? is to establish whether NIM effects better and more timely permanent placement decisions by the legal system: the more intensive scrutiny afforded to parents by NIM might allow the child welfare and legal systems to make more timely and more robust placement decisions (Turner-Halliday et al., 2017;C. H. Zeanah et al., 2001), thereby reducing placement instability (Boyd et al., 2015). Prior to the beginning of the study two thirds of children under five in foster care in Glasgow returned to their birth parents within a year and, of those, two thirds had a further period in care (Minnis et al., 2010). Preliminary evidence from a routine data exploration of the service in New Orleans was that following intervention from NIM, the rate of re-accommodation was much lower than the rates prior to the implementation of NIM and the rates of maltreatment for subsequent siblings were also much lower(C. H. Zeanah et al., 2001).
In BeST ? , NIM is delivered by two teams: one in each trial site, comprising both healthcare (a Consultant Psychiatrist, Consultant Psychologist, Clinical Psychologist and Psychology assistant) and social care staff. Participants randomised to NIM are offered a structured attachment-based assessment involving each actual and potential caregiver. It uses structured interviews, self-report measures and observations (C. H. Zeanah et al., 2001) including The Working Model of The Child Interview (C. Zeanah & Benoit, 1995), Crowell Procedure (Crowell et al., 1988), Child Behaviour Checklist (Achenbach & Rescorla, 2000) and Ages and Stages Questionnaire (Squires et al., 1997). An intervention is then tailored for every family, drawing on a small range of relationship-based therapeutic techniques, including Circle of Security (Powell et al., 2009), Child Parent Psychotherapy (Lieberman et al., 2006) and Video Interactive Guidance (Silhánová et al., 2011), all of which comply with recommendations of a meta-analysis that examined ways of improving parental sensitivity (Bakermans-Kranenburg et al., 2003). Parents are also referred as required to other agencies for help with substance misuse, mental health issues or intrafamilial violence. At the end of the process there is a recommendation of either rehabilitation to birth family or adoption(C. H. Zeanah et al., 2001). Figure 2 outlines the NIM process.

SERVICES-AS-USUAL.
Families not randomised to NIM will receive SAU, which is the assessment and intervention that social services and others normally implement when children are removed from parental care. These SAU are delivered by local authorities who have a statutory duty to safeguard children and provide services (National Guidance for Child Protection, 2014; Great Britain. Department for Education and Skills, 2007). Social workers assess the family and help to engage them with support/clinical services. How SAU is operationalised varies across the UK ( Figure 3).
The Glasgow and Renfrewshire versions of SAU are an ideal control for NIM because they also offer a relationship-based assessment of the parental capacity to care for the child, but have a social work ethos, including more naturalistic observations of the family and unstructured assessments of case files. They do not contain an infant mental health treatment component. The timelines for both Glasgow and Renfrewshire are necessarily shorter as there is no formal intervention component. Currently SAU in London are relatively sophisticated compared to many areas of the UK because of their close relationship to South London and Maudsley NHS Trust which is a partner in this collaboration. All of the London local authorities adopt one of three different methods for assessing families. The first is an assessment delivered by a specialist team who will offer an assessment of parenting capacity (as with the Glasgow and Renfrewshire models). The second is an assessment carried out by the allocated social worker. The third is a specialist assessment carried out by an independent social worker/psychologist/psychiatrist. All three methods of assessment must be undertaken within 26 weeks as per the expectations set out by the judiciary. In contrast with NIM, the systems of care that comprise SAU have developed iteratively, borne out of local authority contexts and processes and have never been formally evaluated. Further exploration of the detailed nature of SAU is a key objective of this process evaluation. The heterogenous nature of services as usual is seen as a strength in that it will allow for a more detailed exploration of what works, for whom and why.
The key differences between the arms of the trial are summarised in Table 1 below: These processes are not necessarily linear. Often the parent/child assessment and the foster carer/child assessment occur simultaneously. Assessment/intervention activity is detailed in the pink boxes (top) and the mechanism of change is detailed in the green (bottom).

Contexts
There are many commonalities between the trial sites and New Orleans, where NIM originates, including the nature of family demographics, dysfunction and challenges (Minnis et al., 2010).
There are also several differences in context both within the UK and between the UK and New Orleans. The UK has a National Health Service, free at the point of contact, and a fully-developed welfare state that includes a preventative social work system. These are not available in New Orleans. The preventative social work approach in the UK might mean that, for those families where children  experience maltreatment and enter into foster care, family problems may be more entrenched as those families with less entrenched difficulties may already have received support and change may have been affected earlier. In contrast, in New Orleans, families receiving NIM are more likely to be previously unknown to services, so may have a greater range of severity of difficulties, some of which may respond better to intervention (Minnis et al., 2010). The use of 'dual registered' foster carers, primed to adopt the child if rehabilitation to birth family is not possible, is the norm in New Orleans but is relatively rare in the UK. The prevailing model in the UK involves children being placed with 'temporary' foster carers whose role is to care for and nurture children while their birth parents undergo assessment. A decision is then taken by social services as to whether those children should return to the care of their parents or should be placed in more permanent alternative care (of which adoption is the primary route for children in BeST ? ). This necessitates disruption in the attachment that has been formed with the foster carer in the hope that a strong relationship can be forged with birth parents (in the case of rehabilitation) or with alternative permanent carers. In New Orleans, 'dual-registered' foster carers can assume care of the children when they are removed from their parents and, if children are unable to be returned to their birth parents, can go on to become the child's permanent carers, necessitating no disruption in the child-foster carer relationship and no additional trauma or loss. The NIM intervention is undertaken with birth parents and the foster carers caring for the child at that time. In New Orleans, this means that any work done on promoting the mental health of the child is likely to be done with a carer that will ultimately assume permanent care of the child(C. H. Zeanah et al., 2001). In The UK, should the child not return to the care of their birth parents, the carers who assume permanent care of the child may not have undertaken work with NIM in respect of the child. This is a very important contextual difference between the UK and the US which does not affect the delivery of NIM but could have a profound impact on its effects.
There are also fundamental differences in the way the legal systems of Glasgow, London and New Orleans make decisions about children in foster care. In London and New Orleans the legal systems are judge led proceedings in which timescales are defined by statute, and a judge has oversight of the decisions that are taken in respect of children in foster care. In New Orleans, the Adoption and Safe Families Act decrees that if a child is in foster care for 15 of the previous 22months, the state must move to terminate parental rights and that permanence hearings for those children must be held at least every 12 months. In London, the target for making a long-term placement decision is 26 weeks (Children and Families Act 2014 s.14 (2)(ii)), although these timescales can be relaxed up to 40 weeks if a child is randomised to NIM. In contrast, Glasgow has a lay tribunal system in which most children in foster care come under the jurisdiction of the Children's Hearing System: specially trained members of the public (Members of the Children's Panel) sit in Children's Hearings and make decisions about all aspects of a child's life-from where that child lives, the duration of frequency of family contact, to whether that contact should be supervised and beyond (Norrie, 2013). The judiciary in Glasgow, known as Sheriffs, have little oversight of the day-to-day decision-making in respect of children in foster care other than establishing the facts of the circumstances that led to the reception into care and hearing any appeals of decisions made at Children's Hearings (Norrie, 2013). Unlike in New Orleans and in London, there are no statutory timescales applied in Glasgow to long term decision-making. These differences may influence how successfully or otherwise the NIM model may be applied in each of the UK sites.
Key process evaluation questions 1) What are the key components of NIM and SAU in the Glasgow and London sites? 2) Are both Interventions in both sites adhered to or do they change during the study?

Methods and data collection
The relationship between the key process evaluation questions and an overview of the data to be collected, in two phases, is summarised below and in Tables 2 and 3. Phase one ran from 2016 to 2019 and is pertinent to the bedding in of the interventions and of the trial itself. The second phase (2019-2024) will capture perceptions of the interventions and how they operate. Once the interventions and the trial are well established within their contexts.

Questions 1 and 2
Over both phases of data collection, a purposive sample of parents/foster carers, social workers, intervention team members, managers and birth parents are invited to take part in interviews and focus groups to identify the perceived key components of NIM and SAU in both sites and to capture any changes in practice or delivery during the lifetime of the study. The frequency and type of data collected are outlined in Table 3.
Quantitative data on every participant are collected regarding the use of health and social services both within and out with NIM and SAU. In addition, Professors Zeanah and Larrieu of Tullane University (where NIM originated) have agreed to rate 10 randomly selected reports from the NIM service in Glasgow to ensure model fidelity using a checklist prepared for the study. The NIM service in Glasgow will then assist with ensuring model fidelity in London with the London NIM service. At the start of phase two, a focus group with Professor Zeanah and the managers from the Glasgow and London NIM services took place to ensure adherence to the NIM intervention.

Questions 3 and 4
Because of the importance, for the success of the interventions, of positive interactions with the social work and legal systems ( synchronicity between the outputs of NIM and various legal proceedings, having NIM reports tabled at court etc.), we will need detailed understanding of these processes. A skilled social work researcher extracts routine quantitative data on the children and families coming through the trial relating their journeys through the interventions and the social work and legal systems and qualitative information from social workers and legal professionals regarding these wider systemic processes.
Qualitative methods need to be responsive to the needs and circumstances of the different participant groups: focus groups allow for both exploration of the area under discussion and observation of the group dynamics which highlight commonalities and differences between participants. Birth parents who have had children removed from their care and foster or adoptive carers warrant individual one-to-one interviews given the sensitivity of the subject matter and the need to be flexible with the participants. Given the limited number of senior managers within SAU and NIM, individual interviews may facilitate in depth discussions and more granular understanding of services and strategic perspectives.

Birth parents
Semi structured interviews have been conducted with a sample of birth parents involved each service (intervention and control) in both sites centring around their motivation to take part in the research, their experience and understanding of the randomisation process, their experience of involvement with the assessments and (if offered) any treatment and of their views of the social work and legal systems.

Foster carers/adoptive parents
A purposive sample of foster carers involved SAU and NIM across both sites are invited to take part in semi structured interviews focusing on their perceptions of their roles and the impact they have on children's longer-term outcomes. Further interviews will also take place with dual registered foster carers. All Foster Carers are asked to fill out a Use of Services Questionnaire, allowing fuller assessment of model fidelity and health economic analysis.

NIM and services as usual practitioners
Approximately annual focus groups are conducted with each of the teams delivering NIM and SAU across both sites focusing on the running of each service, and interactions with social work and health services. Time is also devoted to issues that each service feels is important, including the interface each service has with the legal system, specific legal challenges and opportunities identified by practitioners.

NIM and services as usual managers
Approximately annual semi structured interviews take place with the managers of the SAU and NIM services including discussion of operational challenges and opportunities.

Local authority social workers
Groups of case managing social workers are invited to take part in focus groups (one per phase per site), examining social worker's experiences of the care system and of working with SAU or NIM and views of the strengths and challenges of legal system for participating children There are separate focus groups for Glasgow and London social workers.

Local authority social work managers
As with social work practitioners, managers are asked to take part in semi structured interviews (one per phase per site) looking at the interface between the trial and both SAU and NIM (across both sites). Managers are asked to discuss issues brought to them by staff and also operational/strategic issues that they have identified. Interviews with senior managers and first line managers within each of the authorities, along with scrutiny of local policies and procedures, are used to draw up models of services-as-usual across the sites.

Legal personnel
Interviews and focus groups are used to describe and evaluate the legal systems in which the trial operates. In Glasgow, separate focus groups are undertaken with legal practitioners representing birth families, adoptive families and local authorities to examine perceptions of the legal system, experiences of SAU and of NIM and views on how the legal system influences outcomes for children. Focus groups are also undertaken with those legal personnel who make decisions about children in carespecifically Sheriffs and members of the Children's Panelto examine views of SAU and NIM, the interface of NIM and SAU with the legal system and how the system influences outcomes.
In London, focus groups are conducted with solicitors representing birth families, adopters and local authoritiesfocussing on the same topics as the corresponding groups in Glasgow. Focus groups with judges in London examine their interaction with SAU and NIM and more general views on how the legal system influences outcomes. A focus group with staff and managers from the Children and Families Court Advisory Service (CAFCAS) examines their interface with the trial and with SAU and NIM. CAFCAS staff are also invited to discuss their views of the system and its relationship to children's outcomes.

Case studies
Four case studies have been conducted in each site concerning participants, their social workers and/ or therapists around the time of initial involvement with either arm of the trial and a later exercise (during Phase 2) following up all participants involved in the earlier case studies. Sampling is purposive; SAU and NIM identify a number of cases where there were indications, during the process of assessment, about the nature of the final recommendation. Two cases from each service are selected; one case from each service where indications are suggestive of the child/ren being rehabilitated home, and one case from each service where it looks likely that a recommendation will be made for adoption. Participants are invited to take part in case study interviews based on their involvement in the case and their ability to provide a view on the child from their specific perspective; that is, from the angle of the parent, the foster carer, the area team social worker and the key workers involved in the case from either SAU or NIM.
Monitoring data/routine data Social work case records are used to quantify the length of time taken for children to enter into a permanent placement and will also be used to map children's legal journey's over the time they are accommodated. These data will allow for analysis of the impact of the legal system and the variety of decisions made within legal fora on children's outcomes. Health records are used to identify adverse events and will also allow researchers to quantify and describe children's health service use.
We collate data from NIM and SAU teams at each site, including monthly returns from staff regarding training and supervision received in the team and on appointments offered to families and the proportion of these attended. In addition, Use of Services diaries will be completed by foster carers and brought to the clinic at each research assessment to allow accurate completion of the Use of Services Questionnaire. These data will contribute to both the assessment of model fidelity and to the health economic analysis.

Emergent issues
During the lifetime of the trial, the process evaluation team may conduct further interviews and focus groups if required to describe and evaluate any emergent issues or unforeseen difficulties that have been identified either by our existing focus groups or through trial management activity. This could potentially involve other professional groups or stakeholders if the issues that are identified require information that is held by groups other than those we have included here. This part of the process evaluation will be iterative by definition and will seek to log and describe the interplay of these emergent issues with the services and the context.

Analysis
All the interviews and focus groups are transcribed verbatim, then is read by at least two members of the research team. Thematic Analysis (Braun & Clarke, 2006)provides the framework for this process. An initial coding of the transcripts will identify key categories in line with the realist analysis, and to answer the four key questions, then we will produce a thematic analytical framework used to analyse the remaining interviews. This is an iterative process and is continually reviewed and agreed upon as the research progresses to ensure that it continues to meet our research needs.

Discussion
Using a realist evaluation model, the mechanisms of the interventions (SAU and NIM), aspects of the wider context, and the context-mechanism-outcome patterns will be detailed. The quantitative outcomes measured by BeST ? are concerned with child mental health, the time taken for children to be placed in a permanent placement, relationship functioning and cost effectiveness. The process evaluation will allow for a richer understanding of each of these outcomes. SAU and NIM have a number of constituent parts. Although other process evaluations have detailed the theory of change underpinning trial interventions (e.g. Arthur et al. (2020); Moran and Gutman (2020); O'Brien et al. (2019)), we are unaware of other process evaluations which seek to outline which of the component parts of the interventions are considered 'key' to the delivery of the intervention and the effecting of change.
Despite conceptual frameworks existing for Process Evaluation (e.g. Moore et al. (2015); Pawson and Tilley (1997)), few published protocols of realist evaluations embedded within RCTs exist (French et al., 2020;Gilissen et al., 2020;O'Brien et al., 2019). The core of the realist method is understanding the influence of the context and the intervention mechanisms on the outcomes (Moore et al., 2015;Pawson & Tilley, 1997). Often, descriptions of the services being compared are not sufficient to unpick the complex interactions of context, mechanism and outcomes (French et al., 2020;Murdoch, 2016;Wells et al., 2012). Case studies are a useful element within a range of methods that 'capture the dynamic and complex relationship between intervention and context during implementation' (Grant et al. (2020). The frequency and timing of our data collection is intended to capture change over time (Gilissen et al., 2020;Maddison et al., 2020).
There are distinct legal and social work systems in operation over both the sites in which the trial operates. The interventions are also embedded within multi agency contexts. Using the methodology set out in this paper, the impact of both SAU and NIM on the wider context, and the impact of the context on both SAU and NIM will be interrogated. Although previous process evaluations have documented the interactions in such complex and challenging contexts (e.g. Bridges et al. (2017); Kirkpatrick et al. (2018); Segrott et al. (2017)) we are unaware of any evaluations that have sought to interrogate how the legal system and decisions made by legal tribunals interact with the interventions being tested or influence the outcomes. In this regard, BeST ? is offering a novel contribution to the literature.
Regardless of our quantitative findings, our process evaluation will offer useful information to researchers and policy-makers who may wish to test or roll out NIM in different social and legal contexts in future and will offer insight into what works, in what context and for whom.

Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This clinical trial is funded by National Institute for Health Research, Public Health Research Unit (12/211/54). Prof Ruth Dundas is supported by Medical Research Council grant MC_UU_00022/2, and Chief Scientist Office Grant SPHSU13

Ethical apporval
The study received favourable ethical opinion from the West of Scotland Ethics Committee