Automated data scraper: points to unstructured public data sets to create a Digital Development Exchange for open data, mapped to the Digital Transformation Framework then uses GitHub Actions to process data into CSV files for the Digital Development Compass to calculate and visualize.
@idan, @mattrothenberg, Irene, & @Wattenberger (OCTO DX) got together to flesh out the tasks that are needed to make this project a success. We figured this would be the best way to split up the work. Please edit as much as you want!
Task
Notes
Captain
Technologies
1. Build data manifest
Fill out data_manifest.csv (below)
Mark
spreadsheet
2. Collecting data
Fetches data and turns it into csvs
OCTO DX
Flat, Deno
3. Processing raw data
Turning csv files into scores
Mark
??
4. Designing the website
Getting the Figma designs to a code-ready state
5. Mapping out the information architecture
A spreadsheet that lists out the pillars & sub-pillars
Data manifest details:
Matt & Amelia dug into the filename matching spreadsheet, and there's a good amount of work that needs to be done to get all of the data into clean csv files. We put together a spreadsheet that lists the necessary info for fetching that data - check out the first two rows and please let us know if anything's unclear. data_manifest.csv
What we found was that converting between formats (eg. xlsx -> csv) was straightforward, and most of the work here will revolve around figuring out where the data actually lives (eg. in the third spreadsheet in the xlsx download).
We were thinking we could use our current project that fetches data and processes it using Deno scripts. For example, this script to convert xlsx -> csv:
Drawing on the alpha and beta wireframes and the available datasets, do some exploration of what data visualizations we could create. This issue will track designs and async feedback.
Let's confirm how this page(s) will fit into https://data.undp.org/. In the case that this will be a standalone website, this issue will track creating web designs including layout, branding, etc.