Weighting-Based Sensitivity Analysis in Causal Mediation Studies

Let Z denote treatment assignment that takes value 1 or 0 representing an experimental condition and a control condition, respectively; let M denote the focal mediator; and let Y denote the outcome. In addition, X denotes a vector of observed pretreatment covariates, P denotes a potentially omitted pretreatment covariate, and Q denotes a potentially omitted posttreatment covariate preceding the focal mediator. In the absence of P and Q, the relationships between Z, M, Y, and X are illustrated in Figure 1.

Download

Open in new tab

Download in PowerPoint

Figure 1. Causal mediation in the absence of omitted pretreatment and posttreatment confounders.

We define the causal effects of interest in terms of potential outcomes (Holland, 1988; Pearl, 2001; Robins & Greenland, 1992). Specifically, M(1) and M(0) denote the potential intermediate outcomes of an individual under the experimental condition and the control condition, respectively. The potential outcome under treatment condition z, typically written as Y(z) for z = 0, 1, can equivalently be written as $Y(z,M\left(z\right))$. Hence, $Y(1,M\left(1\right))$ and $Y(0,M\left(0\right))$ denote the individual’s potential outcomes under the experimental condition and the control condition, respectively. Additionally, $Y(1,M\left(0\right))$ denotes the individual’s potential outcome under the experimental condition when the treatment counterfactually does not change the mediator from its value under the control condition M(0) to that under the experimental condition M(1).

Following Pearl’s (2001) terminology, we define the natural indirect effect (NIE) as

This is the population average treatment effect on the outcome transmitted through the treatment-induced change in the mediator—that is, with a change in the mediator value from M(0) to M(1) when all individuals are treated. Note that M(0) and M(1) are each a random variable that can “naturally” take different values for different individuals. The natural direct effect (NDE) is defined as

This is the population average treatment effect on the outcome attributable to other unspecified pathways. Robins and Greenland (1992) called Equations 1 and 2 “the total indirect effect” and “the pure direct effect,” respectively. The sum of NIE and NDE is the total treatment effect on the outcome. This total effect can alternatively be decomposed into “the pure indirect effect” defined as $E\left[Y(0,M\left(1\right))-Y(0,M\left(0\right))\right]$ and “the total direct effect” defined as $E\left[Y(1,M\left(1\right))-Y(0,M\left(1\right))\right]$. The two decompositions are nonequivalent in the presence of a Treatment × Mediator interaction. For simplicity, the discussion in this article focuses on NIE and NDE.

Hong (2010a) first introduced a new propensity score–based weighting approach to causal mediation analysis named “RMPW.” Hong and others (Hong, 2015; Hong, Deutsch, & Hill, 2011, 2015; Hong & Nomi, 2012; Huber, 2014; Lange, Rasmussen, & Thygesen, 2014; Lange, Vansteelandt, & Bekaert, 2012; Qin & Hong, 2014, 2016, 2017; Tchetgen Tchetgen, 2013; Tchetgen Tchetgen & Shpitser, 2012) subsequently extended the RMPW approach and its several variants to quasi-experimental data to discrete and continuous mediators and outcomes, to multiple mediators, and to multisite mediation analysis.

RMPW identifies NIE and NDE under the sequential ignorability assumption summarized by Imai and colleagues (2010). Pearl (2001) stated a set of assumptions in slightly different forms:

• A1: The treatment assignment is ignorable, given pretreatment covariates X.

• B1: The mediator value assignment under each treatment condition is ignorable, given X.

A1 is satisfied if the treatment assignment is in fact randomized or is as if randomized within levels of X = x. Such a treatment assignment mechanism allows Z to be independent of the potential mediators M(1) and M(0) and of the potential outcomes $Y\left(1,m\right)$ and $Y\left(0,m\right)$, given X. It also implies that, for those with the same pretreatment characteristics x, the probability of being assigned to either treatment condition is nonzero (known as the positivity assumption).

B1 is satisfied if, under each treatment condition, the mediator value assignment is as if randomized within levels of X = x. This is plausible when individuals sharing the same pretreatment characteristics happen to display different mediator values due to random events in life. Such a mechanism of mediator value assignment under the experimental condition allows M(1) to be independent of the potential outcomes $Y\left(1,m\right)$ and $Y\left(0,m\right)$, given X. Analogously, a similar mechanism of mediator value assignment under the control condition allows M(0) to be independent of $Y\left(1,m\right)$ and $Y\left(0,m\right)$, given X. B1 also implies that the probability of being assigned to any mediator value is nonzero under each treatment condition.

When the sequential ignorability assumption holds, for $z, z\prime$z, z′ = 0, 1, and z unequal to z′,

Here, $W_z=1$ under a simple randomization of the treatment; otherwise, $W_z=\frac{pr\left(Z=z\right)}{{\theta }_z\left(x\right)}$, where ${\theta }_z\left(x\right)=pr\left(Z=z|X=x\right)$ is the propensity score for the treatment assignment. This strategy is known as inverse probability of treatment weighting (IPTW; Robins, 2000) or model-based direct adjustment (Rosenbaum, 1987). In a nonexperimental study, W_z adjusts for treatment selection associated with pretreatment covariates X.

The RMPW theorem (Hong, 2010a, 2015) states that, to identify $E\left[Y(1,M\left(0\right))\right]$, the weight for a discrete mediator is

Here, ${\theta }_{M\left(z\right)}\left(x\right)=pr\left(M=m|Z=z,X=x\right)$, for z = 0, 1, is the propensity score for the mediator value assignment under treatment condition z. This weight applies to individuals in the experimental group who displays mediator value m and pretreatment characteristics x. Once IPTW has removed treatment selection, RMPW effectively transforms the mediator distribution in the experimental group to resemble that in the control group within levels of $X=x$. If the treatment is randomized, the weight is simplified to be $W=\frac{{\theta }_{M\left(0\right)}\left(x\right)}{{\theta }_{M\left(1\right)}\left(x\right)}$. Hong (2010a, 2015) and colleagues (Hong et al., 2011, 2015; Hong & Nomi, 2012) have shown that RMPW has the flexibility of handling binary and multicategory mediators. Applying the Bayes theorem to Equation 4, it is easy to derive an alternative form of the weight that can be conveniently applied to a continuous mediator:

where ${\theta }_1\left(m,x\right)=pr\left(Z=1|M=m,X=x\right)$. This form of the weight is inherently related to the form presented by Huber (2014). Under a simple randomization of the treatment, ${\theta }_1\left(x\right)= pr\left(Z=1\right)$, Equation 5 is then simplified to be $W=\frac{pr\left(Z=1\right)}{1-pr\left(Z=1\right)}\times \frac{1-{\theta }_1\left(m,x\right)}{{\theta }_1\left(m,x\right)}$.

NIE and NDE can each be identified simply by a weighted mean contrast:

Importantly, identification through weighting does not require the analyst to specify the outcome as a function of the treatment, the mediator, and the covariates. Hence, the analyst does not have to take the risk of possibly misspecifying the functional form of the outcome model. In this way, the RMPW method conveniently accommodates Treatment × Mediator interactions, linear or nonlinear, that may naturally vary among individuals. In contrast, all regression-based identification strategies require that Treatment × Mediator interactions be either nonexistent or homogeneous. Imai and Yamamoto (2013) derive the bias when the assumption of homogeneous Treatment × Mediator interaction fails. Applications of regression-based strategies also typically invoke high-stake assumptions such as invariant covariate–outcome relationships across treatment conditions and across mediator levels. The weighting strategy avoids these model-based assumptions, although the analyst must specify the functional form of ${\theta }_{M\left(z\right)}\left(x\right)$ or the functional form of ${\theta }_1\left(m,x\right)$.

We now consider the case in which there is an additional pretreatment covariate P that is independent of the observed covariates X and may confound the mediator–outcome relationship as shown in Figure 2. The same logic applies when P represents a vector of omitted pretreatment confounders. In general, when the observed covariates X have already been adjusted for in the original analysis, an omitted confounder that is independent of X tends to be more concerning than one that is associated with X.

Download

Open in new tab

Download in PowerPoint

Figure 2. Causal mediation in the presence of an omitted pretreatment confounder.

When the treatment assignment is randomized or is as if randomized within levels of $X=x$, part (A1) of the sequential ignorability assumption is satisfied. Part (B1) of this assumption now states that the mediator value assignment under each treatment condition is ignorable given X and P. In this case, applying the weight defined in Equation 4 for a discrete mediator will no longer identify $E\left[Y(1,M\left(0\right))\right]$. We can easily prove that $E\left[Y(1,M\left(0\right))\right]$ will be identified instead by $E\left[W_{P}Y|Z=1\right]$, where the new weight is

Here, ${\theta }_{M\left(z\right)}\left(x,p\right)=pr\left(M=m|Z=z,X=x,P=p\right)$ for z = 0, 1. See Online Appendix A for a proof of this identification result. If the treatment is randomized, the weight is simplified to be $W_P=\frac{{\theta }_{M\left(0\right)}\left(x,p\right)}{{\theta }_{M\left(1\right)}\left(x,p\right)}$. The new weight W_P is distinguished from W in that W_P makes adjustment for the confounding effects of both X and P, while W adjusts for X only. The new weight for a continuous mediator is

Under a simple randomization of the treatment, we have that ${\theta }_1\left(x,p\right)={\theta }_1\left(x\right)=pr\left(Z=1\right)$. Hence, the weight becomes $W_P=\frac{pr\left(Z=1\right)}{1-pr\left(Z=1\right)}\times \frac{1-{\theta }_1\left(m,x,p\right)}{{\theta }_1\left(m,x,p\right)}$.

If P is the only omitted confounder, once P is taken into account, NIE can be identified by $E\left[W_{1}Y|Z=1\right]-E\left[W_{P}Y|Z=1\right]$ and NDE can be identified by $E\left[W_{P}Y|Z=1\right]-E\left[W_{0}Y|Z=0\right]$. In contrast, when the weight defined in Equation 4 or 5 is employed, omitting pretreatment covariate P that confounds the mediator–outcome relationship will lead to a bias. In the attempt to identify NIE, the bias is equal to

It is easy to show that the attempt to identify NDE will lead to a bias equal to $-cov\left(W_P-W,Y|Z=1\right)$. Online Appendix B shows the derivation. Let $H=\sqrt{\text{var}\left(Y|Z=1\right)}$ represent the standard deviation of the outcome in the experimental group. With H as the metric, to identify the effect size of NIE, the bias is a product of two sensitivity parameters:

The bias in identifying the effect size of NDE is simply $-B_{ES}$.

Clearly, the discrepancy between W_P and W plays a central role in assessing bias. In identifying NIE and its effect size, the bias is determined by (a) the standard deviation of $\left(W_P-W\right)$ in the experimental group denoted by σ and (b) the correlation between $\left(W_P-W\right)$ and Y in the same group denoted by ρ. For transparency, in the weighting-based sensitivity analysis, we will represent the effect size of bias in the NIE estimate and that in the NDE estimate each as a function of these two sensitivity parameters: σ and ρ. The bias is 0 when $W_P=W$ for all individuals (i.e., when $\sigma =0$) or when the correlation between $\left(W_P-W\right)$ and Y is 0 (i.e., when $\rho =0$). The bias increases with σ in the experimental group; when σ is fixed, the direction and magnitude of bias are consistent with the direction and magnitude of ρ in the experimental group. We can show that σ is associated with the degree to which P predicts mediator value assignment under either treatment condition and that ρ is additionally related to the degree to which P predicts the outcome. Online Appendix C clarifies the inherent connections between these two weighting-based sensitivity parameters and the conventional regression-based sensitivity parameters.

A posttreatment confounder preceding the focal mediator is a covariate that is an intermediate outcome of the treatment and yet at the same time is a predictor of the focal mediator and of the outcome. In Figure 3, the posttreatment covariate Q plays the role of an additional mediator (or a vector of omitted posttreatment confounders) that precedes the focal mediator M.

Download

Open in new tab

Download in PowerPoint

Figure 3. Causal mediation in the presence of an omitted posttreatment confounder preceding the focal mediator.

Viewing Q and M as consecutive mediators, we may define NIE and NDE as follows:

Note that $Y(1,Q\left(1\right), M(1,Q\left(1\right)))$ can be simplified as $Y(1, M\left(1\right))$, that $Y(0,Q\left(0\right), M(0,Q\left(0\right)))$ can be simplified as $Y(0, M\left(0\right))$, and that $Y(1,Q\left(1\right), M(0,Q\left(0\right)))$ can be simplified as $Y(1, M\left(0\right))$. Hence, seemingly Equation 8 is equivalent to Equation 1; and Equation 9 is equivalent to Equation 2. However, unlike the previous cases, here the causal effect transmitted through M includes the Z-M-Y pathway and the Z-Q-M-Y pathway; the causal effect not transmitted through M includes the Z-Q-Y pathway and the direct Z-Y pathway.

Hong (2015) explicated the identification assumptions under which the causal effects defined in Equations 8 and 9 can be identified and derived the weights needed for the identification. Specifically, the sequential ignorability assumption now consists of three ignorabilities:

1. A2: The treatment assignment Z is ignorable, given X.

2. B2: The assignment of values of the first mediator Q under each treatment condition is ignorable, given X.

3. C2: The assignment of values of the second mediator M under each treatment condition is ignorable, given X and Q.

Assumption (C2), analogous to assumption (B1), views both M(0, Q(0)) and M(1, Q(1)) to be as if randomized within levels of Q = q and X = x. Therefore, M(0, Q(0)) is assumed to be conditionally independent of all future potential outcomes including Y(1, q, m). This is a major distinction between the current set of assumptions and those invoked in past research for multiple mediators (e.g., Imai & Yamamoto, 2013). Instead of assuming that the mediator under the control condition is conditionally independent of the potential outcome under the experimental condition, the latter invokes structural assumptions such as assuming homogeneous Z-by-M interaction and assuming that Q-by-M interaction does not exist.

Under Assumptions (A2), (B2), and (C2), one can identify $E\left[Y(1,Q\left(1\right),M(0,Q\left(0\right)))\right]$ by applying the following weight to individuals in the experimental group who displays first mediator value q, second mediator value m, and pretreatment characteristics x:

Here, ${\theta }_{M\left(z\right)}\left(x,q\right)=pr\left(M=m|Z=z,Q=q,X=x\right)$, for z = 0, 1 and for all possible values of q. Online Appendix D provides a proof. If the treatment is randomized, the weight can be simplified to be $W_Q=\frac{{\theta }_{M\left(0\right)}\left(x,q\right)}{{\theta }_{M\left(1\right)}\left(x,q\right)}.$ For a continuous focal mediator M, after applying the Bayes theorem, the weight takes the following form:

Here, ${\theta }_1\left(x, q\right)=pr\left(Z=1|Q=q,X=x\right)$ and ${\theta }_1\left(m,x,q\right)=pr\left(Z=1|M=m,Q=q,X=x\right)$. Under a simple randomization of the treatment assignment, we have that ${\theta }_1\left(x\right)=pr\left(Z=1\right)$ and ${\theta }_1\left(x, q\right)={\theta }_1\left(q\right)$. The weight is then simplified to be $W_Q=\frac{{\theta }_1\left(q\right)}{1-{\theta }_1\left(q\right)}\times \frac{1-{\theta }_1\left(m,q,x\right)}{{\theta }_1\left(m,q,x\right)}$. Therefore, NIE and NDE can be respectively identified by

When Q is omitted, however, the causal effect transmitted through M will be confounded by the Z-Q-Y pathway. Applying the weights defined in Equation 4 or 5 to identify NIE will lead to a bias equal to $\text{cov}\left(W_Q-W,Y|Z=1\right)$; the bias in the identification of NDE will be $-\text{cov}\left(W_Q-W,Y|Z=1\right)$. Again with $H=\sqrt{\text{var}\left(Y|Z=1\right)}$ as the metric, in identifying the effect size of NIE, the bias is

The bias in identifying the effect size of NDE is its opposite. Hence, the bias associated with the omission of Q is determined by (a) the standard deviation of $\left(W_Q-W\right)$ in the experimental group (denoted by σ) and (b) the correlation between $\left(W_Q-W\right)$ and Y in the same group (denoted by ρ). As before, it can be shown that σ is inherently connected to the predictive relationship between Q and M and that ρ is additionally connected to the predictive relationship between Q and Y.

Sections III and IV have examined two types of covariates—an omitted pretreatment covariate P and an omitted posttreatment covariate Q—that may confound the mediator–outcome relationship. The discrepancy between the new weight that adjusts for the confounder and the initial weight that omits the confounder is central to determining the bias in identifying the natural indirect and direct effects and their effect sizes. To summarize, we now use W^# as a general form of the new weights standing for W_P or W_Q that make respective adjustments for P or Q along with X in identifying $E\left[Y(1,M\left(0\right))\right]$. In an application in which pretreatment and posttreatment confounders both exist, one may compute their aggregate bias.

When one or more confounders are omitted, the bias in identifying NIE can be represented as the covariance between the weight discrepancy and the outcome in the experimental group:

The bias in identifying NDE is its opposite. The bias in identifying the effect sizes of NIE and NDE can be further represented as a product of two sensitivity parameters: (a) the standard deviation of the weight discrepancy $\sigma =\sqrt{\text{var}\left(W^{\#}-W|Z=1\right)}$ and (b) the correlation between the weight discrepancy and the outcome $\rho =\text{corr}\left(W^{\#}-W,Y|Z=1\right)$ in the experimental group. The former is associated with the degree to which the omitted confounder is associated with the focal mediator value assignment; the latter is additionally related to the degree to which the omitted confounder predicts the outcome. We now discuss how to apply these results in practice.

Our first device is to graphically represent plausible bias in relation to the sensitivity parameters. Figure 4 illustrates a template for assessing bias in identifying the effect size of NDE and that in NIE. Here, σ has a lower bound 0 and an upper bound with some finite value. When the analyst makes additional covariance adjustment for strong predictors of the outcome, ρ would likely be bounded away from −1 and 1. The bias is 0 when σ = 0 or when ρ = 0. In this figure, the curves associated with nonzero correlation values and nonzero standard deviation values each represent a fixed amount of bias equal to ρσ. The bold dashed curve on each side corresponds to a threshold value of bias. The bias values between these two dashed curves are too small in magnitude to change the conclusion of statistical inference, while those in the outskirts of these dashed curves indicate bias values big enough in magnitude to alter the conclusion.

Download

Open in new tab

Download in PowerPoint

Figure 4. Graphical display of bias in the effect size of NDE and that in NIE. The solid curves indicate amounts of bias that may lead to a qualitative change in the conclusion of statistical inference. The number at the end of each curve denotes the bias value represented by that curve. The bold dashed curve on each side corresponds to a threshold value of bias that is just great enough to start changing the analytic conclusion. Each “*” corresponds to an observed covariate or a set of covariates that, if omitted, would contribute a bias great enough to change the analytic conclusion; each “Δ” corresponds to a covariate (or a set of covariates), the omission of which would be inconsequential. These covariates are drawn from the application example and are listed in Table 1.

To illustrate, suppose that an initial estimate of the effect size of NIE is negative but is not statistically significant and that its 95% confidence interval is $[-0.5, \hspace{0.17em}0.1]$. After the removal of a plausible positive bias, $\rho \sigma >0.1$, the new confidence interval for the effect size of NIE would shift entirely to the negative side and would lead to a new conclusion that the NIE is negative and statistically significant. After a plausible negative bias, $\rho \sigma =-0.5$, is removed, the new confidence interval would shift entirely to the positive side. The analyst would then reach a new conclusion that the NIE is positive and statistically significant.

Arguably, the entire set of observed pretreatment and posttreatment covariates supply a range of plausible reference values for the sensitivity parameters. Such information will enable applied researchers to determine the amount of plausible bias. Therefore, one may supplement Figure 4 with a sensitivity analysis table listing the estimated bias that each observed pretreatment or posttreatment covariate or each subset of the observed covariates would potentially and uniquely have contributed if it had been omitted from the analysis. Even though some of the pretreatment covariates may have been adjusted for already in the original analysis, they nonetheless provide sensitivity information should there be a comparable unmeasured confounder, while some others may have been omitted for the purpose of keeping the propensity score models parsimonious. Table 1 provides examples of omitted observed confounders in the context of the application to be discussed in the next section. Considered on the basis of these observed covariates, the plausible bias values that are great enough in magnitude to alter the initial conclusion are flagged with an “*” in Table 1 and are marked with an “*” in Figure 4. The plausible bias values due to the omission of a covariate or a subset of covariates, if inconsequential, are marked with a “Δ” in Figure 4.

Table 1. Sensitivity Analysis: National Evaluation of Welfare-to-Work Strategies Application

Table 1. Sensitivity Analysis: National Evaluation of Welfare-to-Work Strategies Application

View larger version

In some applications, researchers could argue on scientific grounds that the study has collected some of the most important confounders, pretreatment or posttreatment. This is often the case, for example, in educational, behavioral, or health research in which the list of covariates adjusted for in the initial analysis includes one or more pretest scores or detailed health records (see Shadish, Clark, & Steiner, 2008). The remaining observed and unobserved covariates, when omitted individually or collectively, might not contribute a bias great enough to qualitatively alter the conclusion. When this is the case, then by convention, one may decide that the current results are not highly sensitive to the omitted confounding. In contrast, when the existing collection of observed covariates is far from being nearly complete, one may consider the results to be relatively sensitive if an omitted covariate with a rather small value of bias would lead to a change in the conclusion.

This sensitivity analysis strategy is easy to implement in standard statistical software programs. We provide in the Online Supplemental Material a set of R code for conducting the proposed sensitivity analysis procedure when the treatment is or is not randomized and when the mediator is measured on various scales. The R code computes the effect size of bias associated with each omitted pretreatment or posttreatment covariate or their combinations; it also computes the effect size of bias due to unmeasured confounders comparable to the observed pretreatment confounders. When the omitted or unmeasured pretreatment covariates do not confound the treatment assignment, the code generates both tabular and graphical displays of results (as in Table 1 and Figure 4). Otherwise, only a sensitivity analysis table is displayed. The R code has been incorporated into the “rmpw” package available online at https://cran.r-project.org/web/packages/rmpw/index.html.

Should an omitted covariate become available for the causal analysis, there would likely be a change not only in the point estimate but also in the asymptotic variance of the estimate for the NIE and the NDE. When the focus of a sensitivity analysis is solely on bias rather than efficiency, one may consider the plausible reference values for the sensitivity parameters to be given rather than estimated. Hence, removing a plausible value of bias from an initial estimate would lead to a change in the lower and upper bounds but not the width of the 95% confidential interval for the NIE and that for the NDE. If the impact of an omitted confounder on the estimation efficiency is of interest as well, the analyst may use bootstrapping to obtain a new confidence interval accordingly.

Here, we illustrate with the National Evaluation of Welfare-to-Work Strategies (NEWWS) Riverside study. Immediately preceding the welfare reform nationwide in the mid-1990s, welfare applicants in Riverside, California, were assigned at random to either a labor force attachment (LFA) program (Z = 1) or a control condition (Z = 0) in an experimental study. The LFA program, with the goal of eventually weaning participants from the welfare system, emphasized seeking and securing employment, offered job search services, and provided incentives including a threat of sanctions should one fail to meet the program requirements, while the control group members were guaranteed cash assistance without the requirement for employment. Hong, Deutsch, and Hill (2015) investigated whether a treatment-induced increase in employment mediated the treatment impact 2 years later on maternal depression (Y) among participants, most of whom were single mothers with preschool-age children and were disproportionately depressed at the baseline. As a summary score over 12 self-administered items, the measure of depression had a mean equal to 7.49 and a standard deviation equal to 7.74. The mediator (M) was a binary indicator for whether one was employed in any quarter during the 2 years after randomization.

Although the average intention-to-treat (ITT) effect of being assigned to LFA rather than the control condition on maternal depression was indistinguishable from zero, this could possibly be due to a cancellation of the indirect effect and the direct effect. The researchers theorized that, on one hand, assignment to LFA would reduce depression on average if the program induced a large number of individuals to participate in the labor force (i.e., a negative NIE); on the other hand, assignment to LFA would increase depression on average if the employment rate remained at a relatively low level (i.e., a positive NDE). Applying the RMPW method to the data, the researchers reported a negative estimate of NIE (coefficient = −0.88, standard error [SE] = .47, t = −1.87, p = .06) and a positive estimate of NDE (coefficient = 1.26, SE = .86, t = 1.47, p = .14). With the standard deviation of the outcome in the experimental group as the metric, the estimated effect size of NIE is about −0.111; and the estimated effect size of NDE is about 0.158. The 95% confidence interval for the effect size of NIE is [−0.227, 0.006] and that for the effect size of NDE is [−0.054, 0.371]. Even though the null hypotheses are retained in both cases, the confidence interval for NIE is predominantly on the negative side while that for NDE is predominantly on the positive side. These results suggested that, should all individuals have been assigned to LFA, the treatment-induced change in employment (an increase in employment rate from 39.5% to 65.4%) would produce a considerable amount of reduction in maternal depression on average. Without this increase in employment, however, maternal depression would have increased on average by a considerable amount should all individuals have been assigned to LFA rather than to the control condition. Yet neither estimate reached the conventional statistical significance level for rejecting the null hypothesis. A subsequent reanalysis of the data that employed a generalized method-of-moments procedure to account for the sampling error in the estimated propensity scores generated similar point estimates and SE estimates (Bein et al., in press).

The NEWWS sample included 208 experimental units and 486 control units. To avoid overfitting the propensity score model for the mediator under each treatment condition, Hong, Deutsch, and Hill (2015) selected nine pretreatment covariates for each of these models. On the basis of theoretical reasoning, one may speculate that some observed pretreatment and posttreatment covariates omitted from the initial analysis as well as some unobserved covariates might bias the NIE and NDE estimates. Below are two examples of observed confounders that were omitted from the initial analysis.

The data show that individual preference for working rather than taking care of family full time, independent of the baseline employment experience and baseline depression level, predicts labor force participation under LFA. Individuals with a higher preference for working tend to be overrepresented among the ever employed and underrepresented among the never employed in the LFA group. Hence, once additional adjustment is made for this covariate, people with a higher preference and ever employed and those with a lower preference and never employed, due to their overrepresentation, are expected to receive a new weight less than the initial weight (i.e., $W^{\#}-W<0$). These people tended to experience relatively less depression at the 2-year follow-up perhaps because their preferences were seemingly satisfied. In contrast, individuals with a higher preference yet never employed and those with a lower preference yet ever employed, due to their under-representation, are expected to receive a new weight greater than the initial weight, $W^{\#}-W>0$. These people would likely experience relatively more depression. A positive correlation between weight discrepancy and the outcome ($\rho >0$) will contribute a positive bias to the original NIE estimate. The bias in the original NDE estimate will be negative and will have the same magnitude due to the treatment randomization.

A second example is whether an individual was ever in a situation of not receiving welfare during the first year after randomization. About 30.3% of the LFA group members and 21.6% of the control group members had ever been off welfare during the first year. For low-income single mothers who had a hard time to make the ends meet, losing the public safety net would likely trigger or aggravate depression; yet being off welfare might also propel an individual to make her best effort in securing employment in the job market. Therefore, having ever been off welfare during the first year is an observed posttreatment covariate that may confound the relationship between the mediator and the outcome.

For illustration, Table 1 lists the above two observed covariates along with two other covariates that were also omitted from the initial adjustment. Three of them are pretreatment and one is posttreatment. Here, X1 denotes whether an individual had been on welfare for at least 36 months during the 60 months prior to randomization; X2 measures an individual’s level of preference for working as opposed to taking care of family full time at the baseline; X3 indicates whether an individual had ever obtained a high school diploma or a General Educational Development (GED) certificate before randomization; and X4 is an indicator for whether an individual was ever off welfare during the first year after randomization. We enter each of these four variables one at a time as an additional covariate in the propensity score model and obtain a new weight accordingly. We also assess the amounts of bias when different combinations of these covariates (i.e., X1X2, X1X3, X1X4, X2X3, X2X4, X3X4, X1X2X3, X1X2X4, X1X3X4, X2X3X4, and X1X2X3X4) are omitted from the analysis.

Table 1 lists, for each of these omitted covariates, the standard deviation of the discrepancy between the new weight and the initial weight (σ) and the correlation between the weight discrepancy and the outcome (ρ) in the experimental group. Among the four covariates in this example, X2 contributes a bias of the largest magnitude when omitted. As we have reasoned earlier, this omission indeed results in a positive bias for the NIE estimate and a negative bias for the NDE estimate. Once this bias is removed, the estimated effect size of NIE is modified to be −.132 and becomes statistically significant. The estimated effect size of NDE increases to 0.179 though remains insignificant statistically. Because the initial 95% confidence interval for NIE is predominantly on the negative side, any omitted covariate with a relatively small positive bias may alter the conclusion once an adjustment is made for such a covariate. This is true with five of the covariates and covariate combinations considered in the current example (marked with “*” in Table 1). Hence, we conclude that the results of the initial NEWWS application are sensitive to the influence of potential confounders.

The weighting-based approach to sensitivity analysis is suitable for quantifying the consequences of both unadjusted treatment selection and unadjusted mediator value selection. Suppose that the treatment assignment and the mediator value assignment are ignorable, given pretreatment covariates X and P, yet P has been omitted in the initial analysis. We may apply IPTW to adjust for treatment selection, given the identification result that $E\left[Y(z,M\left(z\right))\right]=E\left[W_{z\left(p\right)}Y|Z=z\right]$ in which $W_{z\left(p\right)}=pr\left(Z=z\right)/{\theta }_z\left(x,p\right)$ for z = 0, 1. Here, ${\theta }_z\left(x,p\right)=pr\left(Z=z|X=x, P=p\right)$. Unlike W_z that adjusts for treatment selection associated with X only, $W_{z\left(p\right)}$ adjusts for treatment selection associated with both X and P. Multiplying the IPTW with the RMPW, we are able to identify $E\left[Y(1,M\left(0\right))\right]$ by $E\left[W_{P}Y|Z=1\right]$, where for a discrete mediator

For a continuous mediator, the weight is

Hence, $E\left[W_{1\left(p\right)}Y|Z=1\right]-E\left[W_{P}Y|Z=1\right]$ identifies NIE, while $E\left[W_{P}Y|Z=1\right]-E\left[W_{0\left(p\right)}Y|Z=0\right]$ identifies NDE.

Omitting a pretreatment covariate P that confounds the treatment–mediator, treatment–outcome, and mediator–outcome relationships, the analyst instead applies the weight specified in Equation 4 or 5. The bias in identifying NIE and NDE will each have an additional component. This is because, when P is omitted, there will be a bias not only in identifying $E\left[Y(1,M\left(0\right))\right]$ but also in identifying $E\left[Y(1,M\left(1\right))\right]$ and $E\left[Y(0,M\left(0\right))\right]$. The bias in identifying NIE is equal to

In identifying NDE, the omission will lead to a bias equal to

See Online Appendix E for the derivation. Importantly, when the omission of P also leads to a bias in identifying the ITT effect of the treatment on the outcome, the bias in identifying NIE and that in identifying NDE no longer sum to 0. The bias in the identification of the ITT effect is

This result indicates that the weighting approach to sensitivity analysis is applicable to a broad range of causal investigations including ITT analysis. Finally, with $H=\sqrt{\text{var}\left(Y|Z=1\right)}$ as the metric, in identifying the effect size of NIE, the bias is

The bias in identifying the effect size of NDE is

In each case, the bias increases in magnitude when the standard deviation of the discrepancy between the true weight and the ill-conceived weight increases. This quantity is associated with the degree to which P predicts Z or M after adjustment for X. The bias also increases when the correlation between this discrepancy and Y increases in the experimental group or the control group, which is additionally related to the degree to which P predicts Y.

This section reviews a number of sensitivity analysis techniques that have emerged in the recent years for causal mediation analysis and highlights their differences and connections with the weighting-based approach described above. The weighting-based bias formula is inherently related to the general form of the bias formula derived by VanderWeele (2010) under treatment randomization. This is because the discrepancy between a new weight and an initial weight due to an omitted pretreatment covariate fully captures the association between the omitted covariate and the mediator. Online Appendix F provides derivation that reveals this equivalency. However, the weighting-based approach has some important distinctions from the existing strategies for sensitivity analysis.

VanderWeele (2010) represents the amount of omitted confounding effect as a product of two hypothetical sensitivity parameters: The first parameter specifies an omitted covariate’s association with the treatment at a fixed mediator level and the second specifies its association with the outcome when the treatment and the mediator level are both fixed. In implementation, one must make simplifying assumptions (a) that the omitted covariate is binary; additionally in predicting the outcome, one must assume (b) that the omitted covariate does not interact with the treatment or the mediator and (c) that the observed pretreatment covariates do not interact with the treatment or the mediator. These simplifying model-based assumptions make it feasible and relatively convenient to implement the regression model-based sensitivity analysis techniques. However, these assumptions may prove to be scientifically unrealistic in some cases. For example, in the NEWWS data, some observed pretreatment covariates clearly interact with the treatment or the mediator in predicting the outcome. The sensitivity analysis strategies proposed within the regression framework would become overwhelmingly difficult when linear or nonlinear interactions exist between omitted covariates and the treatment or between omitted covariates and the mediator. Incorrect parametric assumptions are consequential for bias assessment. The analyst would also face challenges when an omitted covariate is multivalued. In contrast, the weighting-based bias formula does not require an omitted covariate to be binary and can be easily extended to nonexperimental treatment assignment.

In Imai and colleagues’ approach (Imai, Keele, & Tingley, 2010; Imai, Keele, & Yamamoto, 2010; Imai & Yamamoto, 2013), the key sensitivity parameter representing the severity of confounding is a hypothetical correlation between the error in the mediator model and that in the outcome model. This correlation coefficient can be related to the proportion of variance in the mediator and that in the outcome explained by a hypothetical omitted confounder. The implementation requires that the analyst explicitly specifies a parametric mediator model and a parametric outcome model consistent with the corresponding parametric models utilized for estimating NIE and NDE. Under the linear structural equation modeling framework, this strategy has a great appeal because it can accommodate some nonlinear and nonadditive terms in the parametric models and because the mediator and the omitted covariates are not constrained to be binary. Nonetheless, the model does not allow the Treatment × Mediator interaction to be heterogeneous among individuals. Additional sensitivity analysis would be required, as is the case in applications of all other regression-based methods, for assessing the consequences of outcome model misspecifications. Imai and Yamamoto (2013) propose a solution that involves two additional sensitivity parameters, one representing the correlation between the mediator and the Treatment × Mediator interaction and the other representing the variance of the Treatment × Mediator interaction. This additional sensitivity analysis becomes unnecessary when the RMPW strategy is employed for identification and when the weighting-based approach is utilized for sensitivity analysis.

Tchetgen Tchetgen and Shpitser (2012) propose to represent omitted confounding as a function of the treatment, the mediator, and the observed covariates. The sensitivity bias function is indexed by a vector of parameters possibly with a high dimension when the number of observed confounders is relatively large. This function is assumed to be known and thus must be specified by the analyst. A change in the hypothetical parameter values may lead to a change in the result of inference. To implement, the analyst would need much guidance as to how to choose the hypothetical parameter values in the selection bias function. This approach also requires that the mediator can take only a finite number of values and that the functional relationship between the mediator and the observed covariates under each treatment condition is known. The computation involves summing over the distribution of the mediator, which may become intensive or require additional parametric assumptions for multivalued mediators.

To identify NIE and NDE on the treated, Vansteelandt and VanderWeele (2012) similarly propose a selection bias function associated with an observed or unobserved posttreatment confounder. Conditioning on the posttreatment confounder, this function is meant to capture the unknown difference between treatment groups in the mediator effect on the outcome under the counterfactual experimental condition. Such a difference would indicate a confounding of the mediator–outcome relationship. The selection bias function again can take numerous possible forms; and implementation typically requires a series of additional simplifying assumptions. And finally, VanderWeele and Chiba (2014, also see VanderWeele, 2015) propose a sensitivity function for assessing bias associated with a posttreatment confounder. This sensitivity function similarly involves parameters related to the counterfactual outcomes and hence cannot be quantified by referencing to the observed posttreatment covariates.

Unlike these existing strategies, the weighting-based approach to sensitivity analysis makes no parametric assumptions about the outcome model or about the selection bias functions. Rather than relying entirely on speculation, this new approach makes use of the empirical information in the application data, including observed pretreatment and posttreatment covariates, to determine a plausible range of potential bias. This is particularly useful when researchers have measured a rich set of theoretically important covariates in a given application.

A limitation of the weighting-based approach is that, similar to the strategy proposed by Tchetgen Tchetgen and Shpitser (2012), the analyst must assume a known functional relationship between the mediator and the observed covariates under each treatment condition. This is especially true when the weight is obtained parametrically. Although the functional relationship may never be fully known, one may view a misspecified mediator model as one that has omitted certain confounders. In the NEWWS example, if the preference for taking care of family as opposed to work has a quadratic rather than a linear relationship with the logit of the mediator, the omitted quadratic form of this covariate could be entered in obtaining a new weight; the selection bias associated with this omission therefore could be quantified.

Importantly, semiparametric weighting in replacement of parametric weighting greatly reduces the reliance on the parametric assumptions that one makes in specifying propensity score models. Hong and colleagues (Hong, 2015; Hong et al., 2015) have proposed a semiparametric way of estimating RMPW by stratifying the sample on the estimated propensity scores. Researchers (Hong, 2010b, 2012, 2015; Huang, Frangakis, Dominici, Diette, & Wu, 2005) have also developed a marginal mean weighting through stratification method as an alternative to IPTW for removing bias in treatment evaluation. According to the simulation results for these methods, semiparametric weighting generates results that are considerably more robust than parametric weighting when the propensity score models are misspecified. The weighting-based sensitivity analysis can similarly take a semiparametric form.