theturnout / court-data-pipeline Goto Github PK

Reminder that the licensor is not The Turnout

Currently, the JSON-LD files that are scraped are saved locally using the url of the site on which they originated. As an example, a JSON hosted on https://www.courts.ca.gov/los-angeles-county.html would be saved by the scraper as https://www.courts.ca.gov/los-angeles-county.json. This file name is used during the validation process. Validated files are renamed using string manipulation to extract the just the jurisdiction name (los-angeles-county.json). This file is then used to import new data into the DB.

For the moment, this is fine, but I am using dev data with a standardized naming scheme. It's unlikely that urls encountered in the wild will be so easily parsed given the lack of standardization among court sites. So at the moment, I can think of two solutions:

1. Continue to use the URL as the filename of the JSON that is scraped but do not rename it after validation. It will be passed to the DB as {url}.json.
2. As the URLs to scrape are passed to the script as a CSV provided as an argument when the script is executed, require an additional column that provides some identifier for the courthouse or its jurisdiction and use that as the filename throughout the script.

I think (1) is probably the way to go. There was a benefit to using simpler names earlier in development that is lost now that everything is automated. While more information is always good, (2) puts additional burden on the administrators running the script and I think a goal is to make this process as easy as possible.

Regardless of approach, there is one other issue: URLS make bad filenames due to their use of the / symbol. They get parsed as directories by the file system and throw errors when they are accessed by the script. Is there a standard replacement character or can we choose one? I'm currently replacing / with . but that's also a symbol used by the file system and might have unintended consequences.

@jungshadow and @JDziurlaj, I'd appreciate your input.

Do you mind clarifying a bit on how to implement this in this context? Would using a .config or .env file in which the connection string is defined and referencing it in that section of code accomplish this?

As the definitions and SHACL file have to be kept in parity, and because the definitions will be hosted remotely, I need to add logic to download the SHACL file from the same remote source, save it locally, then use it with pyshacl in validator.py

A few things I'd like to see:

• Build application on argparse
• Split the application code into logical sections/modules (e.g., db, scraper, exporter)
• #3
• #4

You can create individual Issues on any of these for discussion as you see fit.

• Update ontology URI in validator and SHACL file when provided path by Margaret/Stanford
• Update validator to fetch SHACL file each run to ensure parity with updates to ontology (need above URI)
• Convert SPARQL query output to CSV
• Add License file
• Integrate pipeline modules into argparse
• Merge dev and main, delete dev