A statistical analysis of the impact of a nation-wide home visitation program.

This is a 2013 Data Science for Social Good Fellowship project to help the Nurse-Family Partnership evaluate its effectiveness at promoting healthy child developmental outcomes and stable families.

For a quick and gentle overview of the project, check out our blog post.

The Nurse-Family Partnership (NFP) is a nation-wide home-visitation program that pairs a nurse with an at-risk first-time mother for the duration of her pregnancy and the first two years of her child's life.

The Affordable Care Act of 2010 made new money available to home visitation programs such as NFP, but it also required that programs provide evidence of their effectiveness. Although NFP was founded on the basis of several successful randomized controlled trials, these studies were conducted decades ago and in only a handful of states. They need a way to measure their impact across the country using the observational data they have.

However, while NFP's formidable information tracking system affords them a large-scale, systematic perspective on participants' outcomes, they face two principal challenges to performing a country-wide evaluation of their program:

1. NFP's program participants differ from the general population on a range of characteristics, including some, like income or educational attainment, that probably impact the outcomes in question - for example, the wealthier you are, the more likely your kids are to be healthy. As a result, a simple comparison of means between the outcomes of NFP participants and the outcomes of the general population reflects not just NFP's impact but the impact of those factors too. You need to compare apples to apples.

2. To evaluate whether the program is helping participants, then, they would need to compare their outcomes with those of families with similar socioeconomic backgrounds who weren't in the program. But NFP does not track data on child and mother outcomes for families they don't serve.

Read more about the problem in our wiki.

To assist NFP, we used [statistical matching techniques](http://en.wikipedia.org/wiki/Matching_(statistics\)) to identify women in public national datasets who are similar on all observable characteristics (such as demographics) to the women in NFP's administrative data. We can plausibly estimate NFP's impact as the difference in outcomes between these two groups.

Our project had two goals:

1. To provide a rough evaluation of NFP's impact at a national level. Although this study will not have the same validity as a randomized controlled trial, it represents a significant improvement over the evidence basis that would otherwise be available to current decision processes.

2. To develop a methodology for basic impact evaluation of nonprofit programs where resource limitations and program size make traditional experimental evaluations impractical or impossible.

For a quick introduction to matching, read our blog post.

Read more about our methodology in our wiki.

NFP ultimately hopes to assess outcomes across a range of areas, including child development and health, mother's life course, and child welfare/intimate partner violence. For this project, we focused on two specific outcomes: immunization and breastfeeding rates. We created our comparison groups from the National Immunization Survey and the National Survey of Children's Health respectively. We also did some significant exploration in the Current Population Survey as a potential dataset for assessing mother's life outcomes like employment and educational attainment.

Read more about the datasets we used - and those we didn't - in our wiki.

• The data_preparation directory includes all of the scripts we used to clean and reshape our data. Ultimately, we needed to develop uniform datasets from both the NFP data and the comparison data files, so that we could combine these datasets for analysis. The code in this directory demonstrates the details of that process.

• The data_analysis directory contains the scripts used for the actual statistical analysis, including data exploration and matching. This directory also includes a few sample/simulation scripts we developed while exploring our methodology.

• The home_visits_calculator directory contains the scripts we used to create an online calculator for NFP nurses. When we met NFP nurses, they mentioned that one of their most tedious tasks was calculating the number of home visits they need to make with a new client before she gives birth in order to remain NFP compliant. We used the R package Shiny to create a webpage that takes the woman's weeks of gestation as input (12 in the the picture below) and gives the number of visits required as output ([1] 14 in the picture below). Click on the screenshot to access the application.

• The for_wiki directory contains image files for our GitHub READMEs and wiki pages.

Read more detailed documentation in each directory.

All of our data analysis, impact estimation, and data visualization code is written in R, a free and widely-used statistics programming language.

We chose R so our work could be widely used by the non-profit and research community. But we recognize that many potential users of this code may not have use R before. A few references and tutorials that we highly recommend:

• Code School's free, interactive tutorial in R
• Books tailored to one's programming background, such as R for SAS and SPSS Users by Robert Muenchen or R for Stata Users by Robert Muenchen and Joseph Hilbe
• Note that these books have excellent, and freely available, code samples for many common operations in R, with comparisons to the equivalent commands in SAS, SPSS and Stata.
• R in a Nutshell by Joseph Adler, published by O'Reilly
• A number of freely available "quick reference" sheets such as ones by Tom Short, and staff at the University of Auckland.
• Researchers interested in using R specifically to work with complex national surveys may be interested in Anthony D'Amico's usgsd repo, which includes sample import and analysis scripts for many popular surveys.

R is available for free via the R Project website or, with a slightly friendlier user interface, as RStudio.

Much of R's functionality comes not from the base installation but from additional packages. Details about how to install packages are available here. Our code makes use of the following packages:

• Matching
• cem
• survey
• ggplot2
• plyr
• maps
• hexbin
• mvtnorm
• sas7bdat
• car
• Hmisc
• shiny
• datasets

Because we worked with NFP's private health information, we can't make much of our data public. However, we are opening our code and documenting our methodology so it may be useful for other projects.

We would love to hear from other individuals using parts of our code, doing the same kind of work, or interested in getting involved.

Email us at [email protected].

Copyright (C) 2013 Data Science for Social Good Fellowship at the University of Chicago

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS," WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.