Need to reimplement do operation about dowhy HOT 6 CLOSED

Hi Adam, Amit, I just wanted to write a note of encouragement: building the ability to support general causal quantities like groupby(x).mean() is great. And a question: I can see how this implementation sketch (where the causal groupby and the aggregation function are abstracted apart) works with stratified and matched analyses, and probably also discontinuities, synthetic control methods too. Looking ahead a bit, causal trees and forests seem a bit trickier, but because they are still essentially creating (multiple duplicate) strata, I can imagine this might still be made to work. But I can’t convince myself that this can be made to work for weighting schemes without breaking the clean groupby/aggregation abstraction. I’m not a huge fan of weighting methods generally, though, but am curious to understand. Thanks, Emre From: Adam Kelleher <[email protected]> Sent: Saturday, January 26, 2019 9:19 AM To: Microsoft/dowhy <[email protected]> Cc: Subscribed <[email protected]> Subject: [Microsoft/dowhy] Need to reimplement do operation (#35) Hi @amit-sharma<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Famit-sharma&data=02%7C01%7Cemrek%40microsoft.com%7Ca49429eb3ff141f849c808d683b26ccc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636841199689143078&sdata=fzT%2FImvSZmpOk3H%2BxQhME9Vkxk5W0fmpAnR%2BSkkSgQA%3D&reserved=0> I was thinking more about our discussion around the do operation in the CausalModel, and now I'm convinced the do operation as I've implemented it is incorrect. The do operation says "mutilate the causal model by cutting all arrows into X, and then make an intervention which is effective for each unit". Instead of a well-defined do() operation, we have a quantity, Q = E[Y|do(x)], being estimated in the new causal model that results from the do operation. Of course we don't have to explicitly cut arrows if we block backdoor paths, but it's actually the "make the intervention effective" piece that I think needs the better abstraction. As I was figuring out the best way to implement the causal plot method, I'm was doing an aggregation before a plot, and then plotting the aggregate. That makes sense: we can't identify an outcome for a single unit -- only strata of units, and statistics of those outcomes (e.g. E[Y|do(x)] instead of y_i|do(x). The right way to implement this on top of pandas' abstractions would be a causal version of groupby(x).mean().plot(), and the mean() method has no special significance -- it's just the particular quantity you'd like to estimate in a stratum in the mutilated causal model. It's easy to estimate from the Robins g-formula, but so is the second moment, etc etc. Really, we should support general aggregation functions for Q, like grouby(x).agg(Q).plot(). With our current abstractions, the procedure might go something like this: The groupby is done over units for whom the intervention has been made effective, so it's not a standard groupby -- it contains all units in the dataframe, but with X set to x. These are grouped at each level of X. Then, the mean is tricky -- you replace each y_i by E[Y|do(x)] in the appropriate x stratum, and average over the result to get the group means. That second average implements the averaging over P(z), so you get the g-formula result for a first moment. I was using our do() implementation for that y_i replacement. Clearly we can get much more general than this. The causal groupby just says "make the intervention effective for each x", and the aggregation function says "and now compute the stratum-level statistics in the mutilated causal model". The problem with our current abstractions is that the quantity, Q, is coupled with the estimation procedure. There are good reasons for that -- mainly statistical efficiency. We can do better though, at least in some cases. I could definitely see a generalized procedure for sampling from P(Y|do(x)), and then computing a user-defined statistic over those samples. That would make bootstrapping errors easy, as well. I think that's a much more general approach to the do operation. i.e. do(x) returns a same-length series for Y given do(x), so we can compute df.do(y=['y'], x=['x']).mean(), df.do(y=['y'], x=['x']).std(), etc. Then,, the groupby procedure could be a light layer on top of do(), like df.causal.groupby(x=['x'], y=['y']).mean() would sample the y variables after intervening on the set x (under each stratum), and then return the mean with the multi-index x. Then we have the right abstractions for pandas to operate with, and it's pretty intuitive too. tl;dr I need to port over nonparametric conditional density estimation from causality, https://github.com/akelleh/causality/blob/master/causality/estimation/nonparametric.py<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fakelleh%2Fcausality%2Fblob%2Fmaster%2Fcausality%2Festimation%2Fnonparametric.py&data=02%7C01%7Cemrek%40microsoft.com%7Ca49429eb3ff141f849c808d683b26ccc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636841199689143078&sdata=R4I4THiaOuCuLkeKIVF%2BmqWUnJtNsk%2FEH6Ah1lvmt04%3D&reserved=0> but simplify it down to be a sampling procedure. That'll get rid of the integration over the Z cube, which should make it fast! Then we'll have a general nonparametric do, and can make proper aggregation functions. The api as it exists on the PR here is probably good enough for the time being, #34<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fdowhy%2Fpull%2F34&data=02%7C01%7Cemrek%40microsoft.com%7Ca49429eb3ff141f849c808d683b26ccc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636841199689143078&sdata=OBokngiyp5MUyGPl7i3qI36RBg%2BlIyk50dlUwXb69eM%3D&reserved=0> if you want to merge that. It's going to take a while to add the sampling process. In fact, I think it's a new object type, since we could do it with kernel density estimation, MCMC sampling on a parametric model, stratified sampling with discrete X and Z, etc. Best, A — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fdowhy%2Fissues%2F35&data=02%7C01%7Cemrek%40microsoft.com%7Ca49429eb3ff141f849c808d683b26ccc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636841199689299334&sdata=CaX9IzuIQltQpbHO%2FwyA1JYaO1OxnyJPZ7c1sV1%2BCI8%3D&reserved=0>, or mute the thread<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFtH0Fdj91Tov6DHjOY-vyQJ4zNmSvXyks5vHI4egaJpZM4aURgh&data=02%7C01%7Cemrek%40microsoft.com%7Ca49429eb3ff141f849c808d683b26ccc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636841199689299334&sdata=D9sJN7kMsSza%2BpIhx4NaeTXzy66%2FJ9CsiHvaKgLJoSI%3D&reserved=0>.

HI Adam, Ok, I can see how a weighted random sample would work for weighting approaches. That’s a clean approach. Throwing a NotImplementedException is reasonable too, as long as – in the long-term --- we aren’t essentially maintaining two separate APIs for the different kinds of methods. At that point, it might be better to just drop or deprecate the analysis methods that don’t work with the new framework. From: Adam Kelleher <[email protected]> Sent: Sunday, February 10, 2019 6:45 AM To: Microsoft/dowhy <[email protected]> Cc: Emre Kiciman <[email protected]>; Mention <[email protected]> Subject: Re: [Microsoft/dowhy] Need to reimplement do operation (#35) Hi @emrekiciman<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Femrekiciman&data=02%7C01%7Cemrek%40microsoft.com%7C63cdb9425369413be66208d68f6645c1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636854066755764530&sdata=fkicQ2sOegJfJZNYdwBv2e5ZDRdHe7das7uQcm8gDy8%3D&reserved=0> ! This computational model can support weighting by allowing you instead to draw a weighted random sample. It's certainly true that not all approaches (e.g. regression) will yield a sampling approach (though regression can be adapted into this, which does: https://gist.github.com/akelleh/7a31184ce88453599188c568b09e062d<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fakelleh%2F7a31184ce88453599188c568b09e062d&data=02%7C01%7Cemrek%40microsoft.com%7C63cdb9425369413be66208d68f6645c1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636854066755764530&sdata=UUiRbY7Wr0PrQOktsZ8ksB9v3BsTIJWm198Y%2B6K2e%2Bo%3D&reserved=0> ) but for those we can raise a NotImplemented exception, with some extra logging to inform the user that the operation isn't supported. What do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fdowhy%2Fissues%2F35%23issuecomment-462138512&data=02%7C01%7Cemrek%40microsoft.com%7C63cdb9425369413be66208d68f6645c1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636854066755764530&sdata=IqrQUz6MV2lsVGRcFCgF1nNVHu799okLvYuO3zlXhgw%3D&reserved=0>, or mute the thread<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFtH0MCpZ62A4lxg5ysV_vJGIHxBItzIks5vMDBRgaJpZM4aURgh&data=02%7C01%7Cemrek%40microsoft.com%7C63cdb9425369413be66208d68f6645c1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636854066755764530&sdata=MB3xySh3JJTtCaOz%2FKTAGGymPndPUvwo0PWPG%2BorfMo%3D&reserved=0>.

Hi Adam, This division of the interface into 3 abstractions sounds great. I’d emphasize (1) and (2). While (3) might be an interesting lower-level abstraction for methodological researchers, I think we should only really provide this if methodological investigations and improvements at this level will roll up to benefit (2) and (1). Otherwise, let’s focus on enabling methodological research as best we can directly in (2). Emre From: Adam Kelleher <[email protected]> Sent: Saturday, February 16, 2019 2:12 PM To: Microsoft/dowhy <[email protected]> Cc: Emre Kiciman <[email protected]>; Mention <[email protected]> Subject: Re: [Microsoft/dowhy] Need to reimplement do operation (#35) @emrekiciman<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Femrekiciman&data=02%7C01%7Cemrek%40microsoft.com%7C7dbf5e38151e4b814bbb08d69442a4b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636859411292409650&sdata=AbwyhW1EY%2Bwyi1yS8tYdeAV93gIcBRWi3G6e87D3bQc%3D&reserved=0> It looks like there's some good potential to support lots of workflows and philosophies with the package. In particular (1) a "high abstraction" interface, where the user interacts with the lower-level inference tools via the pandas dataframe API. It's more do-operation oriented ("Pearlian") in philosophy, and still does all the checks an warnings be operating on top of the lower-level abstractions. It builds on top of all of the abstractions you've all built the package around, and is a thin layer on top of the final DoSamplers. It allows kwargs to pass through to the underlying chain of object creation. (2) A "lower-abstraction" do operation that requires explicit calls to identify, etc. It's less user friendly due to the chain of explicit object creation, but is a more flexible and powerful workflow for the power users. (3) A "lower-abstraction" effect estimator that operates in a similar fashion to (2). That works in a more similar vein as the existing estimators and demos. Your everyday data science workflow would more likely be built around (1). Social scientists with methodological interests, or power users in quantitative fields might be more interested in (2) and (3). Methodological researchers focused on the statistics of effect estimators might be more interested in (3), with some interest in (2) as well. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fdowhy%2Fissues%2F35%23issuecomment-464373675&data=02%7C01%7Cemrek%40microsoft.com%7C7dbf5e38151e4b814bbb08d69442a4b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636859411292409650&sdata=v9aayIQcbCkH8FiX1mPZywnki7DLYOG0AI8PdtfR3%2BI%3D&reserved=0>, or mute the thread<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFtH0HEWsCCQoAGU4r5JjI9BymHlob2Qks5vOFgFgaJpZM4aURgh&data=02%7C01%7Cemrek%40microsoft.com%7C7dbf5e38151e4b814bbb08d69442a4b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636859411292565903&sdata=r100isMuG%2FLUOVFaQ%2FNhjpy5W1ejziclYbFF4eJ8dDk%3D&reserved=0>.