![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
As any empirical method used for causal analysis, social experiments are prone to attrition which may flaw the validity of the results. This article considers the problem of partially missing outcomes in experiments. First, it systematically reveals under which forms of attrition—in terms of its relation to observable and/or unobservable factors—experiments do (not) yield causal parameters. Second, it shows how the various forms of attrition can be controlled for by different methods of inverse probability weighting (IPW) that are tailored to the specific missing data problem at hand. In particular, it discusses IPW methods that incorporate instrumental variables (IVs) when attrition is related to unobservables, which has been widely ignored in the experimental literature before.
References
|
Ahn, H., Powell, J. (1993). Semiparametric estimation of censored selection models with a nonparametric selection mechanism. Journal of Econometrics, 58, 3–29. Google Scholar | Crossref | |
|
Angrist, J., Bettinger, E., Kremer, M. (2006). Long-term educational consequence of secondary school vouchers: Evidence from administrative records in Colombia. American Economic Review, 96, 847–862. Google Scholar | Crossref | |
|
Angrist, J., Lavy, V. (2009). The effects of high stakes high school achievement awards: Evidence from a randomized trial. The American Economic Review, 99, 1384–1414. Google Scholar | Crossref | |
|
Barnard, J., Frangakis, C. E., Hill, J. L., Rubin, D. B. (2003). Principal stratification approach to broken randomized experiments: A case study of school choice vouchers in New York City. Journal of the American Statistical Association, 98, 299–311. Google Scholar | Crossref | |
|
Bertrand, M., Duflo, E., Mullainathan, S. (2004). How much should we trust differences-in-differences estimates?. The Quarterly Journal of Economics, 119, 249–275. Google Scholar | Crossref | |
|
Bertrand, M., Mullainathan, S. (2004). Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. The American Economic Review, 94, 991–1013. Google Scholar | Crossref | |
|
Bloom, H., Orr, L., Bell, S., Cave, G., Doolittle, F., Lin, W., Bos, J. (1997). The benefits and costs of JTPA title II-A programs: Key findings from the National Job Training Partnership Act study. Journal of Human Resources, 32, 549–576. Google Scholar | Crossref | |
|
Blundell, R., Powell, J. (2003). Endogeneity in nonparametric and semiparametric regression models. In Dewatripont, L. H. M., Turnovsky, S. (Eds.), Advances in economics and econometrics (pp. 312–357). Cambridge, UK: Cambridge University Press. Google Scholar | Crossref | |
|
Busso, M., DiNardo, J., McCrary, J. (2009a). Finite sample properties of semiparametric estimators of average treatment effects. Working paper, University of California, Berkeley Google Scholar | |
|
Busso, M., DiNardo, J., McCrary, J. (2009b). New evidence on the finite sample properties of propensity score matching and reweighting estimators. IZA Discussion Paper No. 3998 Google Scholar | |
|
Card, D. (1999). The causal effect of education on earnings. In Ashenfelter, O., Card, D. (Eds.), Handbook of labor economics (pp. 1802–1863). Amsterdam, Netherlands: North-Holland. Google Scholar | Crossref | |
|
Castiglioni, L., Pforr, K., Krieger, U. (2008). The effect of incentives on response rates and panel attrition: Results of a controlled experiment. Survey Research Methods, 2, 151–158. Google Scholar | |
|
Cochran, W. G., Chambers, S. P. (1965). The planning of observational studies of human populations. Journal of the Royal Statistical Society Series A, 128, 234–265. Google Scholar | Crossref | |
|
Crump, R. K., Hotz, V. J., Imbens, G. W., Mitnik, O. A. (2009). Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96, 187–199. Google Scholar | Crossref | |
|
Das, M., Newey, W. K., Vella, F. (2003). Nonparametric estimation of sample selection models. Review of Economic Studies, 70, 33–58. Google Scholar | Crossref | |
|
Diamond, A., Sekhon, J. S. (2006). . In Genetic matching for estimating causal effects: A general multivariate matching method for achieving balance in observational studies. Institute of Governmental Studies Working Paper. University of California, Berkeley. Google Scholar | |
|
DiNardo, J., McCrary, H., Sanbonmatsu, L. (2006). . In Constructive proposals for dealing with attrition: An empirical example. Ann Arbor: University of Michigan.. Working paper Google Scholar | |
|
Duflo, E. (2006). Field experiments in development economics. In Blundell, R., Newey, W., Persson, T. (Eds.), Advances in economics and econometrics (Vol. 2, pp. 322–48). New York, NY: Cambridge University Press Google Scholar | Crossref | |
|
Finn, H. D., Achilles, C. M. (1999). Tennessee’s class size study: Findings, implications, misconceptions. Educational Evaluation and Policy Analysis, 21, 97–109. Google Scholar | SAGE Journals | |
|
Fisher, R. (1925). . In Statistical methods for research workers. Edinburgh, UK: Oliver and Boyd. Google Scholar | |
|
Fisher, R. (1935). . In The design of experiments. Edinburgh, UK: Oliver and Boyd. Google Scholar | |
|
Fitzgerald, J., Gottschalk, P., Moffitt, R. (1998). An analysis of sample attrition in panel data: The Michigan panel study of income dynamics. The Journal of Human Resources, 33, 251–299. Google Scholar | Crossref | |
|
Frangakis, C. E., Rubin, D. B. (2002). The defining role of principal stratification and effects for comparing treatments adjusted for posttreatment variables: From treatment noncompliance to surrogate endpoints. Biometrics, 58, 191–199. Google Scholar | |
|
Freedman, D. (2006). Statistical models for causation: What inferential leverage do they provide. Evaluation Review, 30, 691–713. Google Scholar | SAGE Journals | |
|
Gertler, P. (2004). Do conditional cash transfers improve child health? Evidence form PROGRESA’s control randomized experiment. The American Economic Review, 94, 336–341. Google Scholar | Crossref | |
|
Grilo, C. M., Money, R., Barlow, D. H., Goddard, A. W., Gorman, J. M., Hofmann, S. G., , … Woods, S. W. (1998). Pretreatment patient factors predicting attrition from a multicenter randomized controlled treatment study for panic disorder. Comprehensive Psychiatry, 39, 323–332. Google Scholar | Crossref | Medline | |
|
Grogger, J. (2009). Bounding the effects of social experiments: Accounting for attrition in administrative data. mimeo, Google Scholar | |
|
Harrison, G. W., List, J. A. (2004). Field experiments. Journal of Economic Literature, 42, 1009–1055. Google Scholar | Crossref | |
|
Hausman, J. A., Wise, D. A. (1979). Attrition bias in experimental and panel date: The Gary income maintenance experiment. Econometrica, 47, 455–473. Google Scholar | Crossref | |
|
Heckman, J. J. (1974). Shadow prices, market wages and labor supply. Econometric, 42, 679–694. Google Scholar | Crossref | |
|
Heckman, J. J. (1976). The common structure of statistical models of truncation, sample selection and limited dependent variables and a simple estimator for such models. Annals of Economic and Social Measurement, 5, 475–492. Google Scholar | |
|
Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153–161. Google Scholar | Crossref | |
|
Heitjan, D. F., Basu, S. (1996). Distinguishing missing at random and missing completely at random. The American Statisticia, 50, 207–213. Google Scholar | |
|
Hirano, K., Imbens, G. W., Ridder, G. (2003). Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71, 1161–1189. Google Scholar | Crossref | |
|
Horowitz, J., Manski, C. F. (1998). Censoring of outcomes and regressors due to survey nonresponse: Identification and estimation using weights and Imputations. Journal of Econometrics, 84, 37–58. Google Scholar | Crossref | |
|
Horowitz, J., Manski, C. F. (2000). Nonparametric analysis of randomized experiments with missing covariate and outcome data. Journal of the American Statistical Association, 95, 77–84. Google Scholar | Crossref | |
|
Horvitz, D., Thompson, D. (1952). A generalization of sampling without replacement from a finite population. Journal of American Statistical Association, 47, 663–685. Google Scholar | Crossref | |
|
Huber, B. (2009). . In Treatment evaluation in the presence of sample selection (Department of Economics Discussion Paper no. 2009–07). Switzerland: University of St. Gallen. Google Scholar | |
|
Huber, M., Lechner, M., Wunsch, C. (2010). . In How to control for many covariates? Reliable estimators based on the propensity score. IZA Discussion Paper no. 5268. Institute for the Study of Labor, Bonn, Germany Google Scholar | |
|
Imai, K. (2009). Statistical analysis of randomized experiments with non-ignorable missing binary outcomes: An application to a voting experiment. Journal of the Royal Statistical Society Series C, 58, 83–104. Google Scholar | Crossref | |
|
Imbens, G. W. (2004). Nonparametric estimation of average treatment effects under exogeneity: A review. The Review of Economics and Statistics, 86, 4–29. Google Scholar | Crossref | |
|
Imbens, G. W. (2009). Better LATE than nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009). NBER Working Paper No. 14896. The National Bureau of Economic Research, Cambridge, MA Google Scholar | Crossref | |
|
Imbens, G. W., Newey, W. (2009). Identification and estimation of triangular simultaneous equations models without additivity. Econometrica, 77, 1481–1512. Google Scholar | Crossref | |
|
Imbens, G. W., Wooldridge, J. M. (2009). Recent developments in the econometrics of program evaluation. Journal of Economic Literature, 47, 5–86. Google Scholar | Crossref | |
|
Karlan, D., List, J. A. (2007). Does price matter in charitable giving? Evidence from a large-scale natural field experiment. The American Economic Review, 97, 1774–1793. Google Scholar | Crossref | |
|
Krueger, A. B. (1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114, 497–532. Google Scholar | Crossref | |
|
Krueger, A. B., Zhu, P. (2004). Another look at the New York City school voucher experiment. American Behavioral Scientist, 47, 658–698. Google Scholar | SAGE Journals | |
|
Lee, D. S. (2009). Training, wages, and sample selection: Estimating sharp bounds on treatment effects. Review of Economic Studies, 76, 1071–1102. Google Scholar | Crossref | |
|
Lee, S., Whang, Y.-J. (2010). Nonparametric tests of conditional treatment effects, . CeMMAP Working Paper CWP36/09. Centre for Microdata Methods and Practice, London, UK Google Scholar | |
|
Mealli, F., Pacini, B., (2008). Exploiting instrumental variables in causal inference with nonignorable outcome nonresponse using principal stratification. mimeo Google Scholar | |
|
Mulligan, C. B., Rubinstein, Y. (2008). Selection, investment, and women’s relative wages over time. Quarterly Journal of Economics, 123, 1061–1110. Google Scholar | Crossref | |
|
Newey, W. K. (2007). Nonparametric continuous/discrete choice models. International Economic Review, 48, 1429–1439. Google Scholar | Crossref | |
|
Newey, W. K., Powell, J. L., Vella, F. (1999). Nonparametric estimation of triangular simultaneous equations models. Econometrica, 67, 565–603. Google Scholar | Crossref | |
|
Neyman, J. (1923). On the application of probability theory to agricultural experiments. Essay on principles. Statistical Science, Reprint, 5, 463–480. Google Scholar | |
|
Robins, J., Rotnitzky, A. (1995). Semiparametric efficiency in multivariate regression models with missing data. Journal of American Statistical Association, 90, 122–129. Google Scholar | Crossref | |
|
Robins, J., Rotnitzky, A., Zhao, L. (1995). Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. Journal of American Statistical Association, 90, 106–121. Google Scholar | Crossref | |
|
Robins, J., Tsiatis, A. (1991). Correcting for non-compliance in randomized trials using rank-preserving structural failure time models. Communications in Statistics, 20, 2069–2631. Google Scholar | |
|
Rosenbaum, P. R., Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55. Google Scholar | Crossref | |
|
Rotnitzky, A., Robins, J. (1995). Semiparametric regression estimation in the presence of dependent censoring. Biometrika, 82, 805–820. Google Scholar | Crossref | |
|
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701. Google Scholar | Crossref | |
|
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581–592. Google Scholar | Crossref | |
|
Rubin, D. B. (1977). Formalizing subjective notions about the effect of nonrespondents in sample surveys. Journal of the American Statistical Association, 72, 538–543. Google Scholar | Crossref | |
|
Rubin, D. B. (1978). Multiple imputations in sample surveys—A phenomenological Bayesian approach to nonresponse. In Proceedings of the Survey Research Methods Section, American Statistical Association (pp. 20–34). Alexandria, VA: American Statistical Association. Google Scholar | |
|
Rubin, D. B. (1990). Formal modes of statistical inference for causal effects. Journal of Statistical Planning and Inference, 25, 279–292. Google Scholar | Crossref | |
|
Rubin, D. B. (1996). Multiple imputation after 18+ Years. Journal of the American Statistical Association, 91, 473–489. Google Scholar | Crossref | |
|
Rubin, D. B. (2008). For objective causal inference, design trumps analysis. The Annals of Applied Statistics, 2, 808–840. Google Scholar | Crossref | |
|
Scharfstein, D. O., Rotnitzky, A., Robins, J. M. (1999). Adjusting for nonignorable drop-out using semiparametric nonresponse models. Journal of the American Statistical Association, 94, 1096–1120. Google Scholar | Crossref | |
|
Schochet, P. Z., Burghardt, J., Glazerman, S. (2001). . In National Job Corps study: The impacts of Job Corps on participants employment and related outcomes. Report. Washington, DC: Mathematica Policy Research. Google Scholar | |
|
Shaikh, A. M., Simonsen, M., Vytlacil, E. J., Yildiz, N. (2009). A specification test for the propensity score using its distribution conditional on participation. Journal of Econometrics, 151, 33–46. Google Scholar | Crossref | |
|
Sianesi, B. (2004). An evaluation of the Swedish system of active labour market programs in the 1990s. Review of Economics and Statistics, 86, 133–155. Google Scholar | Crossref | |
|
Wooldridge, J. (2002). Inverse probability weighted M-estimators for sample selection, attrition and stratification. Portuguese Economic Journal, 1, 141–162. Google Scholar | Crossref | |
|
Wooldridge, J. (2007). Inverse probability weighted estimation for general missing data problems. Journal of Econometrics, 141, 1281–1301. Google Scholar | Crossref | |
|
Zhang, J., Rubin, D. B. (2003). Estimation of causal effects via principal stratification when some outcome are truncated by death. Journal of Educational and Behavioral Statistics, 28, 353–368. Google Scholar | SAGE Journals | |
|
Zhang, J., Rubin, D. B., Mealli, F. (2008). Evaluating the effects of job training programs on wages through principal stratification. In Millimet, D., Smith, J., Vytlacil, E. (Eds.), Advances in econometrics: Modelling and evaluating treatment effects in econometrics (Vol. 21, pp. 117–145). New York, NY: Elsevier Science Ltd. Google Scholar | Crossref |
