Racial Disparities in Police Use of Deadly Force Against Unarmed Individuals Persist After Appropriately Benchmarking Shooting Data on Violent Crime Rates

If we condition on armed status (and consider only those incidents where armed individuals are killed by police) and then benchmark on race-specific population sizes, we get the measure:

The unbiased relative probability of police killing a violent criminal is given by $\frac{{\varphi }_B}{{\varphi }_W}$. So, the above measure is indeed biased exactly by $\frac{{\alpha }_B}{{\alpha }_W}$. Thus, applying the relative crime rate benchmark of Cesario et al. (2019)—that is, multiplying the left-hand side of Equation 13c by $\frac{{\alpha }_W}{{\alpha }_B}$—will yield an unbiased measure of $\frac{{\varphi }_B}{{\varphi }_W}$. For this correction to hold, however, we must first condition on the status of suspects as armed, before estimating the parameters of interest.

On the other hand, if we consider only those incidents where unarmed individuals are killed by police and benchmark on race-specific population sizes, we get the measure:

The relative probability of police killing an unarmed noncriminal is given by $\frac{{\theta }_B}{{\theta }_W}$. So, the above measure is biased by $\frac{1-{\alpha }_B}{1-{\alpha }_W}$. In this case, applying the relative crime rate benchmark of Cesario et al. (2019)—that is, multiplying the left-hand side of Equation 14c by $\frac{{\alpha }_W}{{\alpha }_B}$—will not yield an unbiased measure of $\frac{{\theta }_B}{{\theta }_W}$, it will yield a more biased measure, $\frac{{\theta }_B(1-{\alpha }_B){\alpha }_W}{{\theta }_W(1-{\alpha }_W){\alpha }_B}$; one that has no clear interpretation. Instead, to generate an unbiased estimate of the relative rate of police killing unarmed noncriminals, one needs to multiply by the ratio of noncriminality, $\frac{1-{\alpha }_W}{1-{\alpha }_B}$, not the ratio of criminality.

Cesario et al. (2019), however, apply the same benchmark of $\frac{{\alpha }_W}{{\alpha }_B}$ to all of their data sets—even those consisting of police shootings of unarmed, nonaggressing civilians. This leads to incorrect estimates of the quantities they claim to identify. Using the National Crime Victimization Survey (NCVS) violent crime data from 2016 to define the α parameters in the above equation, the bias introduced by the Cesario et al. (2019) methodology would result in multiplying the true anti-Black racial disparity by a scalar of approximately 0.38. This means that if crime rate differences in our theoretical model were as we find empirically, then even if we set the killing probability parameters in our causal model such that unarmed, noncriminal, Black individuals were 2.6 times more likely to be killed by police than unarmed, noncriminal, White individuals, the Cesario et al. (2019) methodology would suggest no racial disparities!

Below, we use the same data sources and statistical workflow as Cesario et al. (2019), but we break down the analysis by the status of the suspect as armed or unarmed and then apply the correct crime rate adjustment benchmark to each subpopulation. In this way, we replicate the premise of Cesario et al. (2019)—that population-level estimates of the relative risk of being killed by police can be confounded by differential rates of criminality—but correct statistical shortcomings in their methodology.

Data on the killing of civilians by police are taken from “The Counted,” an online database managed by The Guardian (2016). As Cesario et al. (2019) state, this database is more complete than official federal databases, as police departments underreport to the federal government (Feldman et al., 2017; Klinger et al., 2016; Nix et al., 2017; White, 2006). We use data directly from The Guardian (2016) in our workflow, along with the “unarmed and nonaggressing” data as presented in Cesario et al. (2019).

Following the methods described in the supplementary information of Cesario et al. (2019), we ask whether Black individuals or White individuals are more likely to be killed when benchmarking police killing data on three classes of criminal report data from 2015 to 2016: violent crimes (major), violent crimes (minor), and weapons violations. We estimate the criminal activity rates of Black and White individuals from two sources: (1) the National Incident-Based Reporting System (NIBRS; Federal Bureau of Investigation [FBI], 2016) and (2) NCVS (Bureau of Justice Statistics, 2016), as these two data sources allow for extrapolation to population-level counts. As described in Cesario et al. (2019), the NIBRS is a federal database of incidents submitted by law enforcement to the FBI (but compliance is variable and may be nonrandom), and the NCVS is a nationally representative self-report survey of criminal victimization.

The NIBRS data used herein were taken directly from Cesario et al. (2019), and the NCVS data used herein were downloaded from their official source and processed according to the methods described in Cesario et al. (2019). We used the original source data in order to take advantage of the sampling frame and weighting variables provided therein.

We use the generative stochastic models described previously as Bayesian statistical models (McElreath, 2018) coded in Stan (Carpenter et al., 2017; Stan Development Team, 2018b) and fit using R (R Core Team, 2018) and rstan (Stan Development Team, 2018a). Our complete workflow will be maintained on GitHub at https://github.com/ctross/disparitiesandbenchmarks.

We first investigate the relative risk of being the victim of a police killing using standard population size benchmarks. Figure 1 plots population-level relative risk of being the armed or unarmed victim of a police killing using data from “The Counted” (The Guardian, 2016); estimates are subdivided by year—2015 and 2016—and by encounter type. Encounter type categories include the following: (1) “All killings” which refers to all deaths by police, whether by shooting, death in custody, or other means; (2) “By gunshot” which refers to deaths caused by police gunfire; (3) “Not aggressing” which refers to deaths caused by police gunfire against unarmed and nonaggressing civilians—nonaggressing coding was done by Cesario et al. (2019); and finally (4) “Holding firearm” which refers to deaths caused by police gunfire against civilians who were themselves armed with a firearm.

Across all years, encounter types, and armed status categories, police are more likely to kill Black citizens than White citizens. However, these estimates will be affected to an unknown degree by relative crime rates (Equation 10e); to remove the effect of differential crime rates, we now apply the corrections derived in Equations 13c and 14c.

For the case of armed individuals killed by police, we apply the benchmark derived in Equation 13c to the previous estimates of racial disparities in police killings and recover an unbaised estimate of $\frac{{\varphi }_B}{{\varphi }_W}$, the relative risk of police engaging in what are normally classified as justifiable killings of armed criminals. Figure 2 plots these estimates. As expected for this subset of police killings, we recover a principle finding of Cesario et al. (2019): Racial disparities in the killing of armed suspects by police are proportional to the relative rates of violent criminality.

Our results highlight disparities between NCVS and NIBRS data sources. NCVS data suggest almost perfect proportionality between relative violent crime rates and relative police killing rates of armed suspects, whereas data from the NIBRS suggest that White suspects are killed by police at greater rates than expected relative to their violent crime rates. This might be a true empirical pattern, or it might reflect racial disparity in the reporting of NIBRS data. Cesario et al. (2019) argued that NIBRS data were unlikely to be affected by such biases and that “NCVS [data] are uncontaminated by police bias, yet…yield results consistent with the…NIBRS data” (p. 588). We, however, find it important to acknowledge the possibility that NIBRS data are biased by reporting. We contrast the relative crime rates calculated using the NCVS and the NIBRS data sets in Figure 3. Here, we find that across almost all years and crime classifications, NIBRS data show greater racial differences in crime rates than NCVS data. NCVS data are based on a randomized sampling design, making them less likely to be biased by differential policing intensity and reporting. Benchmarking approaches based on NIBRS data may underestimate the extent of anti-Black disparities in police use-of-force.

If we apply the benchmark derived in Equation 14c to the previous estimates of racial disparities in the killing of unarmed individuals by police, we can recover an unbaised estimate of $\frac{{\theta }_B}{{\theta }_W}$, the relative risk of police killing unarmed individuals. Figure 4 shows that across all crime benchmarks in all years, there is substantial evidence of anti-Black racial disparities in the killing of unarmed noncriminals by police. Here, we fail to recover the principle findings of Cesario et al. (2019). Racial disparities in the killing of unarmed citizens by police do not occur in proportion to the relative rates of noncriminality; unarmed and nonaggressing Black individuals are killed in greater numbers than would be expected given the relative populations of noncriminals.

Empirical findings aside, Cesario et al. (2019) argue convincingly that to understand racial disparities in killings by police, we have to engage in proper benchmarking, such that we compare the relative rates of police use-of-force against criminals, $\frac{{\varphi }_B}{{\varphi }_W}$, and noncriminals, $\frac{{\theta }_B}{{\theta }_W}$, independently. The push to take benchmarking seriously is important for two main reasons: (1) It is important not to confound racial disparities that might arise from justifiable shootings of armed and dangerous criminals by police with racial disparities in the killing of innocent, unarmed civilians; and (2) encounter-conditional approaches that appear to mitigate the need for benchmarks are themselves typically confounded by differential encounter rates and contexts.

With respect to the first issue, researchers interested in the topic of racial disparities in police use-of-force have accounted for status as unarmed (e.g., Ross, 2015) prior to publishing relative risk estimates, meaning that the statistical bias factor that is present if one does not also benchmark on crime rates is only on the order of $\frac{1-{\alpha }_B}{1-{\alpha }_W}\approx 1$, but some presentations of data, for example, from The Guardian (2016), have not. The formal derivations provided here validate the benchmarking methods developed in Cesario et al. (2019) when applied to the relative counts of police killings of armed suspects. As in their initial analysis, we find reliable evidence that lethal force by police occurs in direct proportion to race-specific rates of violent crime. However, our formal derivations show that the benchmarking methods developed in Cesario et al. (2019) are misleading and statistically biased when applied to the relative counts of police killings of unarmed suspects. Likewise, their methods will lead to confounding when outcome data are not, or cannot be, classified into criminality categories. Our reanalysis of their data using an appropriate crime rate–correcting benchmark reveals strong and statistically reliable anti-Black racial disparity in police killings of unarmed civilians.

Second, it is important to take benchmarking seriously because, contra Johnson et al. (2019), population considerations cannot be sidestepped when estimating racial disparities in police use-of-force (see a concise proof in Knox & Mummolo, 2019). A similar proof is presented in Ross et al. (2018), who use a generative stochastic model to show that the overall racial disparity in police use-of-force can be decomposed into the product of two terms—racial disparity in use-of-force conditional on encounter and racial disparity in the frequency of encounters. So even if encounter-conditional approaches (e.g., Fryer, 2016; Johnson et al., 2019; Worrall et al., 2018) suggest no evidence of racial disparity in the use of lethal or less-than-lethal force by police conditional on encounter, the overall per capita morbidity and mortality from police use-of-force can be higher in the Black population if the Black population is subjected to higher encounter rates with police. Both recent and decade-old data show that Black individuals are more likely to be stopped by police than White individuals (Fryer, 2016; Gelman et al., 2007; Miller et al., 2017; U.S. Department of Justice, 2016), even after a variety of statistical controls have been applied. Moreover, causal inference (e.g., Knox et al., 2019; Ross et al., 2018) approaches show that if police are biased in who they encounter, then encounter-conditional approaches can be severely confounded.

For example, consider a thought experiment in which police behavior is heterogeneous, with most officers following standard protocols and a small subset of officers engaging in additional unwarranted use of nonlethal force (like tasers) against Black individuals. Then, analysis of pooled encounter-conditional data would suggest that Black individuals are less likely to be shot rather than tased, as compared with White individuals. In other words, elevated levels of sublethal assault against innocent Black individuals by a subset of police would have the effect of diminishing the apparent severity of anti-Black racial disparities in lethal force conditional on encounter (Ross et al., 2018). Racial disparities in the frequency of taser use are consistent with such an explanation (Fryer, 2016). For this reason, anti-Black encounter bias is a confounding factor in recent encounter-conditional studies finding anti-White racial disparities (e.g., Fryer, 2016; Johnson et al., 2019; Worrall et al., 2018). For encounter-conditional analyses to be convincing, researchers would have to rule out the presence of racial bias in encounter probability (Knox & Mummolo, 2019).

Successful interventions to mitigate racial disparities in police use-of-force require that we reliably identify the drivers of such disparities. Methods leading to accurate causal inference are essential. An important step in validating our statistical tools involves applying them to simulated data sets, generated under a process that we understand explicitly—because we coded it. We should be asking ourselves: “If we knew the generative process of the data perfectly, would our statistical approach allow us to correctly detect real disparities?” In this article, we have used a simple generative model in just this way. We show that if the data generating process were such that unarmed, Black, noncriminals were more than twice as likely to be killed than unarmed, White, noncriminals (and if crime rates in our model were as we find empirically), then the statistical methodology of Cesario et al. (2019) would erroneously suggest anti-White racial disparities in the killing of unarmed noncriminals. This calls into question the validity of such methods.

As we move forward in studies of policing, new forms of data, like evidence from officer-worn cameras (Broussard et al., 2018; Willits & Makin, 2018), are becoming available. Such data raise hopes for accountability and detection of discriminatory violations of individual rights (Scheindlin, 2010) and even study of racial bias in respectfulness of language use and escalation or de-escalation of encounters (Voigt et al., 2017). However, there is also concern over how such data will be managed and protected from misuse (Ringrose, 2019). These new forms of data, however, may be able to help resolve conflicting reports about the existence of racial disparities in police behavior by recording possible disparities in both encounters and use-of-force conditional on encounter at the officer level. While such data can be powerful, they can only be appreciated in light of statistical models that must themselves be validated on inferential targets.