We need to add the number of items shown on the screen at the bottom of the database

Add Store select option

Just need to go through and break out the groceries spiders from the url ones to make it more modular

Need a uniform way to categorize different groceries across websites

It would be cool to have a nice chart that would track the prices for different groceries that are in our database

For the Wegmans scraper, implemented a new way to track the urls scraped and not scraped.

Want to roll that out for the Lidl and Walmart scrapers

We should add a link to the page where each item is found, that way we can see more details about it.

We want a database that can support the ingestion of massive amounts of information. Cassandra seems like a good option for this task.

Initial success will look like being able to spin up a container running an instance of Cassandra.

Need to be able to search the database on the frontend - Should be able to leverage the way we did it for the Acronymdb

Add feedback and code to show the user what is being searched and sorted. Should be human readable such as: Showing all groceries, Showing groceries sorted by Grocery Name, Descending, Where Store Name like 'weg' and Grocery Name like 'tuna'.

Search Capability

It would be great if we had some sort of test framework that would help us debug problems with the backend / frontend scrapers etc

With respecting their robots.txt

We need to setup a service / framework for crawling websites and downloading groceries

Find an easy way to backup data and import data from other databases

Getting different formats for the price per units on the websites. This is making them hard to compare and see which is the better deal

It would be nice to have a url (or button you can click) to take you to the item when looking through the frontend

Was wondering if we could use elasticsearch to improve our searches. I think it would be good to spend a few cycles to at least check it out or see if it makes sense for us

Scraper for Walmart is getting redirected to walmart.com/blocked* when using docker-compose up walmart-urls However is seems like running it inside of

[dev] $ docker-compose up
[walmart] $ scrapy runspider urlScraper

Works just fine - so we need to figure that out

Need a way in the UI to be able to delete and edit entries

Need to roll in the store locations for Walmart and Lidl

Right now we can only search by one term say like deodorant but it would be nice if we could do things like mens deodorant

Need to follow the robots.txt for the webpage and ingest data from their website.

I think an easy first step would be to manually download some html and prototype a scraper.
Then move that into a chrome extension.
Then work on automating it - paying careful consideration to their robots.txt file

We will eventually need an AWS instance to run it so that other users can actually run and use it.

Here's a couple articles that would be worth poking around with -

https://blog.vikfand.com/posts/scrapy-fargate-sls-guide/#:~:text=Running%20Scrapy%20in%20AWS%20Fargate,the%2015%20minute%20execution%20limit.

https://medium.com/@mtuseeq531/how-to-run-scrapy-spiders-on-aws-ec2-instance-using-scrapyd-a6422961c017

Work on creating a spider to download webpages for Lidl

Add filters on section and sub section

Some stores report the quantity for each section. That seems like some useful data to gather to know how effective the scraper is doing

Only show 25 or 50 items and have a paging system to go through all the pages of information for groceries

Will need to shift how we do the splash middleware so that different ones can be setup to run independently without stepping on eachother

We need a way for user's to sign in and perform user management in the database

Add sorting both ascending and descending when clicking on a sortable column

Need to be able to modify the list of stores from inside the frontend