biglocalnews / warn-scraper Goto Github PK

This works, but there's no error logging.

https://www.ctdol.state.ct.us/progsupt/bussrvce/warnreports/warn2020.htm

These are standard HTML pages, the scrapes should be pretty similar to the ones we already have going.

There are still some hard-coded file paths in KS scraper.

Restructure repo to move all scrapers scripts into warn/scrapers/[state-abbrev].py, e.g.

warn_scraper_al.py -> warn/scrapers/al.py

Also rename the top-level function to scrape for all of these functions, which will our new cli script (scrape_warn.py #11) to dynamically load and execute each scraper.

Right now, Tennessee's scraper has a few rows of data hard-coded into it, I assume because the scraper didn't catch all of the data.

This scraper should be re-written to grab all of the data, even if we have to start from scratch because we won't know if we are missing future data by leaving it this way.

There are still some hard-coded file paths in OK scraper.

Let's move state scrapers into, e.g., a 'scrapers/` dir, and then rename to just state abbreviation:

scrapers/ne.py
scrapers/ny.py
etc.

This will set us up nicely to use meta-programming to automate scraper runs through a configurable CLI script

Right now, Michigan's scraper has a few rows of data hard-coded into it, I assume because the scraper didn't catch all of the data.

This scraper should be re-written to grab all of the data, even if we have to start from scratch because we won't know if we are missing future data by leaving it this way.

In the scrapers, pages are given a max number (in the case of ok it's 550) and this counts the number of records on the site.
The scraper runs through page by page, pulling the records until it hits that max number.
Needless to say, if the max number turns out to be higher because many records were added, it needs to be manually updated.

Talk to @zstumgoren and figure out a better way to do it programmatically so that you don't have to check the web pages and update these numbers as you go.

• re-org AZ
• re-org DE
• re-org KS
• re-org OK

It's fine to hard-code the basename of output file, but we'll need ability to pass in a directory from the CLI script to each function.

https://kcc.ky.gov/Pages/News.aspx

Downloads excel sheets.

Add scrapers from excel sheet that team didn't have a chance to complete.

Eric and Serdar shared some updated regex that would allow me to parse the data in a cleaner/more generic way.

• add the regex and run it to see what it doesn't catch
• 20 minute pomodoro to get a refresher on Regex
• 20 minute pomodoro to tweak the Regex for scraper, so that it can grab all of the data

since we'll be running this on cron on data-etl-vm

They changed URLs - site to scrape: https://dlt.ri.gov/wds/warn/

Crib and adapt the code for Slack-based alerts from covid-world-scraper.

• import slack client
• add AlertsManager Class
• add argparse argument for alerts
• implement logic for alerts
  • the script runs without breaking
  • test alerts are sent and show up on the slack channel

Need to further split ticket.

There's already a scraper for Ohio, however it only pulls 2020 data because that's what is available in tabular format on the website.

2019 and back are PDFs that need to be scraped and joined with the more recent data.

There are still some hard-coded file paths in DE scraper.

https://mn.gov/deed/programs-services/dislocated-worker/reports/

PDFs

https://www.illinoisworknet.com/LayoffRecovery/Pages/ArchivedWARNReports.aspx

May not be able to do this one. The PDF's have inconsistent structures within and the more recent files are excel files.

https://scworks.org/employer/employer-programs/at-risk-of-closing/layoff-notification-reports

This site has PDFs for each year.

Again, use covid-world-scraper as a reference. Log to /home/ubuntu/logs/warn.log but make the path configurable via CLI arg.

• add argument for cache_dir and make sure the new args work
• add the code below & integrate it to scrape_warn.py

 Path(cache_dir).mkdir(parents=True, exist_ok=True)
    logging.basicConfig(
        level=logging.INFO,
        format='%(asctime)s - %(name)-12s - %(message)s',
        datefmt='%m-%d %H:%M',
        filename=log_file,
        filemode='a'
    )
    console = logging.StreamHandler()
    console.setLevel(logging.INFO)
    formatter = logging.Formatter('%(name)-12s - %(message)s')
    console.setFormatter(formatter)
    logging.getLogger('').addHandler(console)
    logger = logging.getLogger(__name__)

• add logging to one scraper, make sure it works
• add logging to remaining scrapers

There are still some hard-coded file paths in AZ scraper.

Similar to the COVID world data

Right now I have a conditional that either runs all scrapers or runs the scrapers that were given by cli arguments.

The logic currently works in a way that depending on if you ran all scrapers or a select amount of scrapers, slack alerts you that all ran successfully even though a few may not have. Ie, if the last scraper ran successfully despite the others not running successfully, it logs the last message.

It logs the others as well, but I need to create something more like what Serdar has where it tells you overall which ones ran and which ones did not.

Thus, it's necessary to remove the current logic from where it is and re-work it.

• extract all necessary information from functions without invoking slack_messages function
• rewrite slack_messages function with new logic that tells you which scrapers ran, which did not
• integrate that function with the rest of the program

Code on dev2 runs just fine - it needs the last bit of pushing the data to the platform.

https://www.iowaworkforcedevelopment.gov/worker-adjustment-and-retraining-notification-act

Click to download an excel sheet. Similar to the California scrape.

https://www.twc.texas.gov/businesses/worker-adjustment-and-retraining-notification-warn-notices#warnNotices

These are excel sheets for every year.

• AL
• AZ
• CA
• DC
• DE
• FL
• IN
• KS
• MD
• MI
• MO
• NE
• NJ
• NY (re-run to check with new records)
• OH
• OK
• OR
• RI
• SD
• TN
• UT
• WA
• WI

2 functions in the script still have hard-coded outputs.

Set up to run on cron using pyenv (see existing examples in crontab for covid-world-scraper)

https://www.dol.state.ga.us/public/es/warn/searchwarns

This scrape is going to require selenium.
It gives you results based on different regions which you have to select from a dropdown menu.
After that, it should be pretty easy, just downloading the excel file.

Once error logging is added, we can add slack alerts to this too.

https://www.dws.state.nm.us/Rapid-Response

PDFs for each year.

https://www.vermontjoblink.com/ada/mn_warn_dsp.cfm?securitysys=on&FormID=0

This site is similar to AZ and DE, have to scrape the main site and then the links on each page for the layoff count.

https://www.colorado.gov/pacific/cdle/warn-listings

These WARN links lead to google spreadsheets. Download and convert to xlsx.

https://joblink.maine.gov/ada/mn_warn_dsp.cfm?securitysys=on&FormID=0

Similar site to AZ and NE - scrape pages then scrape link on pages.

http://wsd.dli.mt.gov/wioa/related-links/warn-notice-page

It's an excel download.

Not sure if I should leave command utility as is, or if I should use argsparse instead.

http://www.laworks.net/Downloads/Downloads_WFD.asp

Links to PDFs for each year.

Has cleaner implementation for parsing & executing args than my simple version.

https://www.nccommerce.com/data-tools-reports/labor-market-data-tools/workforce-warn-reports

These are PDFs for each year.

https://www.vec.virginia.gov/warn-notices

This is a table with many pages.

We should create a single script, e.g. scrape_warn.py that should be configurable in a few ways:

# Scrape list of specific startes
python scrape_warn.py OK

# Scrape all states and output data to custom dir
python scrape_warn.py ALL --output-dir=/tmp/foo

• python scrape_warn.py ALL
• --output-dir=/tmp/foo

https://jobs.alaska.gov/RR/WARN_notices.htm

Currently we can and do write CSVs to the data/ dir, but this causes code commits on the repository which cause the upstream to move ahead of the local code on developer machines.

Would be good to write the data to either a secondary GH repo that solely hosts the data and/or stream to Big Local GCP bucket.

https://www.labor.idaho.gov/dnn/Businesses/Layoff-Assistance#2

Links to a PDF that you can download.

Add top-level exception handling in order to trap and alert on errors for particular scrapers, without having the entire process crash for all remaining scrapers.

Depends on #2

Script should be able to run all scrapers at once, or a list of given states specified via command-line argument.

https://www.mass.gov/service-details/worker-adjustment-and-retraining-act-warn-weekly-report

Links to excel spreadsheet - can only find 2019-2020 years.