Synth-Validation for Estimator Selection about dowhy HOT 6 CLOSED

Thanks for your comment @MaximilianFranz ! Yes, synth-validation is an exciting idea. We are also exploring other methods for doing validation, such as bayesian model criticism and methods that introduce an interpretable parameter for confounding and then rerun the estimates.

Curious to hear your thoughts on these as we prioritize next steps for DoWhy.

And thanks for building out an R implementation of Synth-validation. Would love to integrate it with DoWhy. I am not an expert at RPy---Would you mind adding a Jupyter notebook that shows how to use the R implementation in python?

from dowhy.

@MaximilianFranz Wanted to share that we just released a roadmap for dowhy's development. Of course, synth-validation and related refutation methods are a high priority. Are you still interested in contributing to the repo?

Would also appreciate your feedback on the roadmap here: https://github.com/microsoft/dowhy/wiki/Roadmap

from dowhy.

Thanks for letting me know. I haven't forgotten about Synth-Validation, but the colleague who translated the code into R isn't sure about publishing it to the DoWhy repository yet. I will check again with him and our supervisor for further information.
If I am not mistaken, the result of their evaluation was that the Synth-Validation method is not as effective as proposed, once the setting becomes more realistic. I'll keep you posted!

from dowhy.

Ah, alright! Thanks for the update @MaximilianFranz. Curious to hear more about the evaluation, that sounds really interesting. Would you be comfortable sharing the results?

from dowhy.

Late reply, sorry for that.
My last information was that the institute was not comfortable publishing the R version of SynthValidation, as it might not be identical and is not robust.
I am currently busy with implementing a framework for academic method evaluation using parametric DGPs (JustCause), so I won't be able to work on Synth-Validation anytime soon, though it would be cool to have!

from dowhy.

Ah, alright. Thanks for the update @MaximilianFranz. Wow, JustCause looks cool---evaluation of causal models is super important! If you think that there are ways DoWhy and JustCause can work together, let me know!

from dowhy.