Constructing generalizable geographic natural experiments

When using a sample to draw causal inference, the evidence can be generalized to a target population only when that sample was randomly selected from the target population of interest. In the case of geographic natural experiments, the sample (e.g., the buffer from either side of the administrative boundary) is not constructed by randomly selecting people from the target population (e.g., the city). As a consequence, generalizability efforts must rely on an observational data analysis assumption (Stuart et al., 2018).

In randomized experiments, the most common quantity of interest is the average treatment effect (ATE). Let Y_i(1) denote the potential outcome if subject i were treated and Y_i(0) if subject i were not treated. The average treatment effect or ATE = E(Y_i(1)) − E(Y_i(0)). In observational studies, the estimand of interest is usually the average treatment effect on the treated (ATT), which can be expressed as: ATT = E(Y_i(1)|T_i = 1) − E(Y_i(0)|T_i = 1). The counterfactual control units are not observed. As a result, it is necessary to construct a control group by using two assumptions: conditional independence and common support (Hidalgo and Sekhon, 2011).

In this paper, we instead focus on a different estimand: the target average treatment effect on the treated (TATT), which will inform us about how the treatment effects operate on the target population of interest. In a sample of n units, $\mathrm{TATT}=\frac{1}{n}\sum _{i=1}^n\left(\left(Y_i\left(1\right)|T_i=1\right)-\left(Y_i\left(0\right)|T_i=1\right)\right)$. In this case, the sample of n units needs to resemble the target population of interest.

We propose a design based on template matching to extend beyond the local effects estimated when using local geographic ignorability designs and to recover the target average treatment effect on the treated (TATT). Template matching was developed by Silber et al. (2014) to make standardized comparisons based on observed characteristics. Their study randomly selected 300 patients (i.e., the template) and used them to match 300 patients at 217 hospitals, constructing a sample that resembled the template used to implement the multivariate matching.

Two assumptions are needed to claim that the matched sample resembles the population of interest and to provide causal evidence after adjusting on observables. The first, the ignorability of sample selection, states that after adjusting for the relevant observed covariates, treatment effects are the same in the matched sample and the target population (Visconti and Zubizarreta, 2018; Stuart et al., 2018). Specifically, the target average treatment effect on the treated (TATT) and the population (of interest) average treatment effect on the treated (PATT) need to be equivalent. In that case, we expect that $\frac{1}{n}\sum _{i=1}^n\left(\left(Y_i\left(1\right)|T_i=1\right)-\left(Y_i\left(0\right)|T_i=1\right)\right)=E\left(Y_i\left(1\right)|T_i=1\right)-E\left(Y_i\left(0\right)|T_i=1\right)$. Second, the conditional geographic ignorability in local neighborhood assumption, holds that within a neighborhood the potential outcomes are independent of treatment assignment conditional on observed covariates (Keele et al., 2015). In this case, every unit i has a score defined S_j = (S_j1, S_j2) that refers to the geographic location of the subject, which will be used to compute the distance to any point (b₁, b₂) located on the boundary. A collection of points within a small geographic neighborhood is defined as N(b₁, b₂). The set of covariates used to obtain covariate balance is defined as X_i. Therefore, for each point (b₁, b₂) located on the boundary, we can find a neighborhood N(b₁, b₂) where (Y_i(1), Y_i(0)) $\left|\right|$ T_i|X_i for all subjects i with score (S_j1, S_j2) in N(b₁, b₂).

A key question is how to define what is the appropriate template or target population. Recent research has advocated for a stronger connection between theory and causal identification. Scholars point to the advantages of theory-driven endeavors, which can help to better recognize undefined potential outcomes (Slough, 2022), to improve covariate balance (Resa and Zubizarreta, 2016), and to generalize a causal effect to other contexts (Gailmard et al., 2021). While the nature of causal identification strategies may require a narrow focus, the theories researchers wish to test may be far more extensive. When constructing a generalizable geographic natural experiment, we argue that researchers should ask not only what identification strategy is best to recover causal effects, but also what template or population they wish to mimic that would best test their broader theory.

For example, Posner (2004) takes advantage of the border between Zambia and Malawi to study the political salience of a cultural cleavage. Chewa and Tumbuka people live on both sides of the border. While their cultural differences are identical on both sides of the border, their political differences are more salient in Malawi than Zambia. The rationale behind exploiting this distinction is that Chewas and Tumbukas are large groups relative to the country as a whole in Malawi and, therefore, can be used as a base for coalition-building. Meanwhile, in Zambia, Chewas and Tumbukas are small relative to the country as a whole, creating little incentive to rely on them for coalition-building.

As a result, Posner (2004)’s theory directly connects with a population of interest (i.e., the entire Chewa and Tumbuka people in Malawi and Zambia) rather than just four villages along the border used in the study. If people from these villages have different distributions of observed characteristics than in the entire country,³ using a traditional geographical experiment might generate estimates that do not speak to the theory. Thus, we would advocate implementing a generalizable geographic natural experiment to improve the connection between theory and causal identification.

The utility of using template matching is also evidenced in more recent implementations of geographic natural experiments. Keele and Titiunik (2018) aim to uncover the effects of all-mail voting on turnout. To do so, they rely on data from two counties, one that used only in-person voting and one that used all-mail voting. While the resulting estimates can tell us about turnout effects at a local scale, they may not be able to extend to the true populations of interest, Colorado, and even the United States as a whole. Employing template matching in this case would provide a kind of external validity check on how well the theory underlying the paper connects with the analysis and results of the causal identification strategy.

It is important to note that we do not equate external validity and representativeness. Our goal is to show that a treatment effect can be generalized across different populations of interest (i.e., external validity). We use template matching to construct representative matched samples that are similar to the population of interest (i.e., representativeness). Using template matching to build representative matched samples can improve the limited external validity of studies that have an especially local nature, often a result of researchers’ efforts to reduce heterogeneity and decrease sensitivity to hidden biases (Rosenbaum, 2005). In observational studies, reducing heterogeneity often means decreasing the sample size to improve comparability between units (Keele, 2015). Therefore, we could end up with a treated and control group that allows us to make credible inferences but that might be substantially different from the target population.

We provide a more concrete illustration of our approach by extending Keele et al. (2015)’s study on the role of ballot initiatives in voter turnout. The authors draw on a natural experiment in the city of Milwaukee, Wisconsin, wherein a ballot initiative was established in 2008 but not implemented in the seventeen surrounding areas. This draws on a local geographic ignorability design, where units “within a narrow band around the border are assumed to be good counterfactuals for each other” (Keele and Titiunik, 2016; 3), and offers a unique opportunity to understand whether such initiatives do, in fact, foster greater political participation. Keele et al. (2015) do not find enough evidence to claim that the ballot initiative has increased turnout.

Table 1 reports the averages for the treated and control groups within a 1000 meter buffer of either side of the boundary. To compare the treatment group (i.e. residents with the ballot initiative) to the control group in their original study, Keele et al. (2015) balance on observable characteristics: age, gender, voting history, and housing values (see appendix A for details).⁵ While the unmatched treated and control groups are very similar to each other for four out of five covariates, the difference for housing prices is greater than 0.1 standard deviations. We use cardinality matching, which finds the largest matched sample that achieves the covariate balance requirements imposed by the researchers (Zubizarreta et al., 2014), to reduce imbalances. In this case, standardized differences cannot be larger than 1/10th standard deviations. Table 2 summarizes the results after matching and also compares the new means with the means for the city, state, and country (see appendix B for all of the standardized differences before and after matching).

As expected, matching generates balance for all of the covariates. This provides a compelling way to improve causal inferences by combining a natural experiment based on geography and matching to improve covariate balance. However, the results might be highly local and not necessarily reflect the effects of initiatives on voter turnout for an average American citizen or even a typical resident of Milwaukee, Wisconsin. In fact, considering the averages for the city, state, and country shown in Table 2, the matched treated and control groups do not look, on average, very similar to these potential populations of interest.

To address this concern, in this paper we combine a geographic natural experiment with template matching. Template matching is critical for achieving covariate balance not only between the treated and control groups but also with a given a target population. We use as a template the city (Milwaukee), the state (Wisconsin), and the country (United States) to match with the treated and control groups (see appendix C for more information on the city-, state-, and country-level characteristics we use).

We report results using the state as the template for the matching in Table 3 (see appendix D for the results using Milwaukee and the United States as templates). This demonstrates that the matched treated and control groups are not just similar to each other, but also similar to Wisconsin. Finally, in appendix E, we provide an illustration about how to implement a generalizable geographic natural experiment using cardinality matching in R.

We use a permutational t-test in matched pairs with an embedded sensitivity analysis (Rosenbaum, 2015).⁶ The parameter Γ represents the odds of differential assignment to the treatment due to an unobserved factor u. Γ = 1 means that two subjects in a matched pair have the same probability of getting the treatment (i.e., there are no hidden biases). Γ = 1.1 means that in a matched pair, one of the subjects is 1.1 more likely than the other to get the treatment because of an unmeasured covariate u. We provide the point estimates (when Γ = 1) and the p-values for different values of Γ.

Table 4 shows that results do not stand to a Γ = 1.2 for any of the studies. These findings illustrate that there is not enough evidence to claim that ballot initiatives can increase participation since the conclusions are sensitive to small biases and the point estimate changed direction in one of the studies. Since we understand external validity as being able to hold the conclusions of the study in other contexts or populations, the evidence when using template matching improves the external validity of the findings reported using regular matching. There is not strong evidence in any of the four analyses to claim that the ballot initiative has increased turnout, which provides extra support to Keele et al. (2015)’s main findings.