kellieotto / modelmatch Goto Github PK

Let's make the code more modular by splitting out the permutation test functionality.

Put the dependencies in the description file. You can't load in packages within the code and it's bad practice to use "require"

http://r-pkgs.had.co.nz/description.html

Add random forests and perhaps neural networks, "deep learning," or something similar to the simulation comparisons based on Hainmueller (2011)

To compare with model-based matching, we should implement propensity score matching.

See statsmodels/statsmodels#858
https://github.com/jburroni/statsmodels/tree/psmatch

potential data: Dehejia, Rajeev H, and Sadek Wahba. “Propensity Score-matching Methods for Nonexperimental Causal Studies.” Review of Economics and Statistics 84, no. 1 (2002): 151–161.
http://users.nber.org/~rdehejia/nswdata2.html

A Tutorial and Case Study in Propensity Score Analysis: An Application to Estimating the Effect of In-Hospital Smoking Cessation Counseling on Mortality, Peter C. Austin. Multivariate Behav Res. Jan 2011; 46(1): 119–151. Published online Feb 18, 2011. doi: 10.1080/00273171.2011.540480
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3266945/

Implement inversion of permutation test to find confidence sets for the effect. Start with the assumption that the effect is the same for all pairs. Notes from email of 10/22/2014:

1. Fit a predictive model to all cases, using the covariates but not the treatment dummy. (It might be better to fit just to the control cases, but let's defer that for now.)
2. Use any of the standard matching algorithms to match one or more control case to each treatment case, based on the predictions.
3. The null hypothesis is that, within match groups (pairs or more), the residuals from the prediction are exchangeable. Across match groups, the residuals are independent. This defines the invariance for the permutation test.
4. If the alternative is that treatment increases the response (not necessarily by a constant shift, not necessarily by the same amount in each match group, etc.), a sensible test statistic might be the sum across match groups of the difference within match groups between the mean response with treatment and without treatment. For other alternatives, we might take the sum of absolute values of differences between the means or something else.
5. The distribution of that statistic under permutation within match groups, independent across groups, lets us test. Essentially, we're doing a stratified two-sample permutation test for whether the residuals appear to come from one population or two.
6. Suppose we want to make a confidence interval for the average effect of treatment on the treated. We have some choices. I think the following are all biased towards zero because the predictive model is fitted to all cases. If it were fitted just to the controls--and if it were unbiased for the response of the controls--then the bias would go away.
   1. If we want to posit that the effect of treatment is a shift, we use the standard inversion of permutation tests (but with the stratified permutation). See, e.g., www.stat.berkeley.edu/~stark/Teach/S240/Notes/ch5.htm Search for "estimating the shift." Also see the simulations to estimate the power of the permutation test in http://www.stat.berkeley.edu/~stark/Teach/S240/Notes/ch3.htm What we need is simply to invert those calculations.
   2. Just as easily, we could posit that the effect of treatment is multiplicative, or follows any other functional form.
   3. We could find a fully nonparametric confidence bound if, within each match group, we impose an a priori bound on the effect. (One-sided bounds give one-sided confidence intervals.) Finding the confidence bound involves solving a k-dimensional optimization problem, where k is the number of match groups.