Evaluating the Effectiveness of a Local Primary Care Incentive Scheme: A Difference-in-Differences Study

National financial incentive schemes for improving the quality of primary care have come under criticism in the United Kingdom, leading to calls for localized alternatives. This study investigated whether a local general practice incentive-based quality improvement scheme launched in 2011 in a city in the North West of England was associated with a reduction in all-cause emergency hospital admissions. Difference-in-differences analysis was used to compare the change in emergency admission rates in the intervention city, to the change in a matched comparison population. Emergency admissions rates fell by 19 per 1,000 people in the years following the intervention (95% confidence interval [17, 21]) in the intervention city, relative to the comparison population. This effect was greater among more disadvantaged populations, narrowing socioeconomic inequalities in emergency admissions. The findings suggest that similar approaches could be an effective component of strategies to reduce unplanned hospital admissions elsewhere.


Appendix 1 Outline of the Difference-in-Differences analysis used in the study:
Difference-in-differences (DiD) analysis is an established approach increasingly used in health research to evaluate the impact of interventions, where the researcher has no role in manipulation of the intervention assignment, sometimes known as "natural experiments" 1,2 .
In DiD analyses, outcomes are observed for two groups before and after an intervention and one of the groups is exposed to an intervention in the second period but not in the first period, and the second group is not exposed to the intervention during the study time frame.
The average change in outcomes in the second (control) group is then subtracted from the average change in outcomes in the first (intervention) group. This removes biases in second period comparisons between the intervention and control group that could be the result of permanent differences between those groups, as well as biases from comparisons over time in the intervention group that could be the result of trends. Thus the differences-indifferences estimator is therefore: is the mean of the outcome variable in the intervention areas after the start of the intervention ( Intervention, ) and before the start of the intervention ( Intervention, ) and in the comparator areas after the start of the intervention ( Comparator, fter) and before the start of the intervention ( Comparator, efore).
To estimate the DiD parameter, we can run the following regression: where is the outcome variable in area a at time t, is a dummy variable taking the value 1 for the intervention area and the value 0 for the comparator area and is a dummy variable taking the 1 or the time period after the start of intervention and 0 before. The coefficient of interest is , the coefficient on the interaction term , which is the difference-in-difference parameter.
Since this analysis cannot be biased due to time invariant differences between the intervention and control groups, using a fixed effect model instead of a mixed/random effect model would offer no additional benefits in terms of controlling for unobserved confounding.
The DiD analysis could however be biased by trends in other predictors of the outcome if these diverged between treatment and intervention groups after the intervention. Therefore, potential time varying confounders are added to the model to control for such biases.

Equation for multivariable mixed effects linear regression model for emergency hospital admissions:
Where is the all-cause all-age hospital emergency admission rate per 1000 population for LSOA i at year j. The intercept ( ) contains a random effect term and thus varies across years. t is an annual time-trend term. Age is the average age of the population in years. Sex is the percent of the population who are female. Unemployment is the percent of the working age population (aged 16-64 years) claiming Jobseeker's Allowance. Group indicates whether LSOA i is an intervention or control LSOA (intervention = 1; control = 0). Period indicates whether year j occurs post or pre-intervention (post-intervention = 1; preintervention = 0). Group*Period is the difference-in-differences estimator.

Matching variables:
The following variables were included in a propensity score model to match intervention to control populations in the time period before the introduction of the intervention (2005-2010).

Matching variable Details
Age and gender profile of the population

Alcohol
The percentage of patients aged 18 years and over who have had the alcohol consumption recorded in the last 3 years

Alcohol
The percentage of patients aged 18 years and over who are drinking more than recommended units per week, have their alcohol intake recorded using the AUDIT-C or AUDIT Tool and received a brief intervention in the last 3 years (i.e. those who score 8 and above receive brief advice and those scoring 16 and above are offered specialist support, e.g. referral to LCAS)

Childhood Vaccinations and Immunisations
The combined percentage achievement (rolling 12 months) for DTaP/IPV/Hib at 1 year, MMR1 at 2 years, PCV booster at 2 years, Hib/MenC booster at 2 years The combined percentage achievement (rolling 12 months) for MMR2 at 5 years and DTaP/IPV preschool booster.

2.6
Palliative/terminal care Full participation and adherence in the gold standards framework. In order for practices to understand if this is effective then the following measures will be provided to them 1) The % of patients on the palliative care register with preferred place of care recorded on their notes 2) The % of patients on the palliative care register who have died in their preferred place of care

Dementia
The practice to establish a process to ensure that appropriate Read Codes are included into the patients' medical record upon diagnosis with Mild Cognitive Impairment and to develop an annual recall and review system for people identified with MCI; to review patients with MCI once only at 12 months post diagnosis

ACS admissions
Rate per 1000 hospital weighted population for admissions for a selection of ACS conditions (Angina, Asthma, COPD, Influenza and Pneumonia,) as primary diagnosis.

Outpatients
Rate per 1000 hospital weighted population for GP referred first Outpatient attendances to certain specialities (Dermatology, ENT, Rheumatology, Urology, Vascular Surgery, Gynaecology)

Patients on warfarin have an INR result in last 4 months
Patients on lithium should have a lithium level in the last 4 months (Lithium levels should be within a therapeutic range appropriate to the individual patient) Patients on lithium should not be prescribed a thiazide Patients with dementia should not be prescribed an antipsychotic Patients with asthma should not be prescribed a non-cardio specific beta blocker Patients with Addison's disease should not be prescribed a thiazide Prescribing for type 2 diabetes should avoid risk of hypoglycaemia -T2D on insulin with 2 or more hypo in 12 months -T2D on SU with 2 or more hypo in 12 months Resulting in a hospital admission

3.4
Reducing antibiotic use A target of 5% reduction in weighted volume of antibiotic prescribing against the practice's 2015-16 baseline or achievement of national average volume.

3.5
Prescribing of specialist and high cost analgesics A target of 5% reduction in costs for a combination of pregabalin/oxycodone/buprenorphine patches/fentanyl 4 Improving Quality

Significant Event Analysis
Practices with a list size below 3,500 (weighted) to complete 3 clinical significant events using the above format. Practices with a list size above 3,500 (weighted) to complete 5 clinical significant events using the above format.

Appendix 4 Parallel trends test:
Investigating the parallel trends assumption using regression analysis 1 Where is the all-cause all-age hospital emergency admission rate per 1000 population for If the Group*Year parameter is not statistically significant, it suggests that the emergency admissions trends pre-intervention are parallel and the control group can be used in subsequent DiD analyses.

Model outputs:
Result of parallel trend assumption test, only including the pre-intervention data, with an interaction term between intervention group and year (2005-10)     Model includes random intercept for GP practices Standard errors and confidence intervals calculated using cluster robust estimation. Model based on 92 intervention GP practices and 460 control GP practices, and 6,624 practice-years in total CI = confidence interval; DiD = Difference-in-Differences; GP = General Practitioners

Emergency hospital admissions due to chronic Ambulatory Care Sensitive Conditions (ACSCs) as the outcome of interest
The main analysis was repeated using an alternative outcome: emergency hospital admissions due to chronic ACSCs. The ICD-10 codes included in the definition of chronic ACSCs were: J45, J46, E10-E14.
The DiD analysis indicates that the LQIS was statistically significantly associated with a reduction in emergency admissions due to chronic ACSCs in the intervention compared to the control populations.

Analysis applying the synthetic control method
As matching can introduce bias related to regression to the mean, 3 we conducted additional analysis that includes a weighted combination of all of the available "untreated" units, instead of just a matched sample. In this analysis we apply the synthetic control method for microdata developed by Robbins et al. to estimate the intervention effect. 4,5 The synthetic control method is a generalisation of difference-in-differences methods, whereby an untreated version of the treated cases (i.e. a synthetic control) is created using a weighted combination of untreated cases.
To construct the synthetic control group, we derive calibration weights such that the weighted characteristics of the control LSOAs population (emergency admissions, age, unemployment, proportion of the population that were female population) match those of the intervention population in the period before the intervention (2005-2010) The weighting algorithm derives weights that meet three constraints. Firstly, the sum of weights in the control group equals the number of emergency admissions in the intervention group. Secondly, the weighted average of each of the characteristics above in the synthetic control group matches those in the intervention group. Lastly, the synthetic control and intervention group also match across all pre-intervention time points in terms of the numbers of admissions. 4 The Average Treatment Effect for the Treated (ATT) is estimated as the difference in cumulative number of admissions in the intervention group period after the intervention time point (2011-2016), compared to the (weighted) number of admissions in the synthetic control group. To estimate the 95% confidence intervals and p-values we apply a permutation procedure, through repeating the analysis through 250 placebo permutations randomly allocating the intervention LSOAs to the intervention group, to estimate the sampling distribution of the treatment effect and calculating permuted p-values and confidence intervals. 5 All analysis was performed using the Microsynth package. 4 The figure below shows the trend in the emergency admission rate in the intervention and synthetic control populations. Similarly to our main matched difference-in-differences analysis, in the intervention population the rate of admissions falls relative to the synthetic control group following the intervention. The effect size from this analysis was similar to our main analysis -indicating that the intervention was associated with a 12% decline in admissions (95% CI 11% to 14%).