To conform with the Data Science template, we should move the getReviews.py file to the src/data folder. This folder is used for populating the data/* folders.

In this case the output file from this should go into the data/raw folder.

Acceptance

• getReviews.py resides in /src/data folder
• outputs data file to /data/raw folder

In looking at the data, there are some cases where the location does not change from one review to the next.
We should probably make sure the data structure for the review is initialized from one review to the next.

On an internal thread I discussed the purpose of FlyWhy and some approaches that didn't work. I'd should update the ReadMe to include these motivations.

In order to get a better detail about the subcategory rating, it would be helpful to explode the Category Rating into new columns.

Or find a better way.

By default the application starts at the main review page. However, due to Internet connection and other issues, it's very difficult to get the process to run to completion on all reviews successfully.

A way to make this easier is to start a process at a particular review and continue processing.

In addition, the process should log that last Url reviewed so that the next one in line can be easily found.

In an effort to adopt "good practices" and "patterns", let's setup the project folder structure using a subset of the CookieCutter Data Science Template.

For more information see this page. https://drivendata.github.io/cookiecutter-data-science/

Every run so far fails to complete. In most cases the process ends with a "selenium.common.exceptions.NoSuchElementException" when trying to find an element.

Add Exception Handling to improve likelihood of successful completion

Believe there is a Selenium timeout setting that you can set as an upper bound, but will otherwise proceed as soon as page load is complete and rendered. Would speed things up when we start to process more than a handful at a time.

When initially creating this project, I included some sample files to help me get started, or try different ideas. These are no longer valid and should be removed.

scrappy.py

The CookieCutter Data Science template sets up a Tox.ini file. Tox aims to automate and standardize testing in Python. It is part of a larger vision of easing the packaging, testing and release process of Python software.

This looks like it could be valuable, if it is, setup Tox, if not delete the Tox.ini file

The Data Science installed Tox. Please see Issue #8 for why this is being removed for now

The Reviewer.Id field is a concatenation of UID_xxx_ReviewId. See sample below

• UID_A455850D086316E0157BE50C4EB2115E-SRC_773635392

We need to determine what TA uses for the actual user id, and only store that portion in the output file.

Draw.io aka https://app.diagrams.net/ is a great tool for creating diagrams. Because it stores it's files as .xml it is very version control friendly. There is also a Visual Code plugin that makes it easy to display diagrams while your working on code.

Add a diagram illustrating the screen scraping flow.

Not required, but issue #4 should be done before working on this to avoid need to move another file around.

Write now the getReviews.py file loads up the entire review structure before writing out the file. Instead we should look at stream the reviews out to the file a page at a time. This should reduce the memory footprint needed for the python application.