The National Family & Health Survey (NFHS) is a survey in India that attempts to collect information on health conditions, nutrition, family planning, domestic violence, and a host of other factors through conducting surveys on a random ("representative") sample of Indian households in all states. The fifth NFHS was conducted through 2019-21, and the reports were released to the public in 2021 and can be found at this link.

One small problem, however, is that all the reports are provided as PDFs, which are pretty neat for humans to read, but terrible for computers to parse. This repo contains scripts that will download all the district wise reports (704 of them), and extract data from the tables, and convert it into machine-friendly JSON. NFHS provides district-wise, state-wise and entire country (aggregate) reports. This repo currently contains code to download, parse and generate JSONs for district-wise reports only. Districtwise reports contain information on 104 "indicators" (or questions asked in the survey). Statewise reports seem to contain some extra information that is not reported district wise, and has approximately 130+ indicators.

Note: I tried my best to make sure the data is being parsed correctly, but there is a possibility that some data in JSON might not be 100% accurate - there is no way I could have manually verified all 704 PDF files and their outputs, so I randomly sampled and verified a couple of files, all of which looked okay. If you want to replicate the data parsing from PDFs, feel free to go through the *.py files.

All code in this repository is released under the MIT License. The data (JSON, PDFs) are available as a Kaggle Dataset

1. District wise data is available at this link (web archive link).

2. From this webpage, we get the links to each of the statewise pages, which is saved in the statewise_district_links.csv file.

3. Then, the get_districtwise_links.py script is used to compile the list of all district wise file URLs into districtwise_links.csv.

4. download_all_districts.py is used to download PDFs and save them to districtwise_data/pdfs. During this process, it appears that the webpages for one state (Telangana) and one Union Territory (Chandigarh), currently point to a 404 page. So data for these.

5. It looks like district wise data for Telangana is available in the Telangana State Compendium - we slice this file up, district wise and save the PDFs in the districtwise_data/telangana folder. Chandigarh has only one district, which covers the entire union territory, so it probably won't have any separate "district-wise" data, as such.

6. There are 704 district-wise PDF files, totalling to approximately 450MB of data.

7. With all this done, we use parse_pdf.py to parse the PDF and dump district wise data to JSON (in the directory districtwise_data/json/. This script uses Tabula and pdfminer.six for parsing PDFs.

8. In the first round of PDF parsing, we used the parse_pdf.py script at commit ce4f8ee. Out of the 704 PDFs, we could generate JSONs successfully for only 563 files. 141 PDFs resulted in errors, which are listed below, along with what was done to solve the errors:

9. The Tabula template files were generated manually by dragging and selecting the tables using the Tabula Desktop app for Linux. The saved template files are located in the directory tabula_templates in this repository. Tabula 1.2.1, which was downloaded from this link (sha256sum: fea6a5d26e2ab1abf2cc0a694d93810c59e93e0ce9190fce31541fdf6e7e6ece tabula-jar-1.2.1.zip).