Git Product home page Git Product logo

coles_vs_woolies's Introduction

๐ŸŽ coles_vs_woolies ๐Ÿ

Python

๐Ÿ… iga now supported. (not that their online store is any good)

Receive an email every week comparing the price of products you buy often.

Scrap Aussie grocers' public APIs, schedule a GitHub Action to run weekly, and leverage MailerSend's free-tier as an email service provider.

I am working on a user-friendly website (read more). However, in the meantime, you can read below & set up your own weekly email service, or simply play with it as CLI tool.

Demo

Motivation

"I like Connoisseur ice cream, but I'll be damned if I'm paying full price for it."

I have caught myself checking the grocery catalogues for half-price choccy & ice-cream each week. So, naturally, I've sought to automate the process & have the cheapest offer across Aussie grocers, Woolies & Coles, emailed to me instead.

I looked at existing platforms, DiscountKit & PriceHipster, however they allege that Woolies, in particular, have deliberately blocked their services. Regardless, supporting the developer community is a good thing & if your website is public then it can/will be scrapped.

Requirements

  • A MailerSend account & it's API key
  • A domain verified with MailerSend (help)

MailerSend is an email & notifications SaaS with a free-tier that suits personal use. You'll need to own a domain & verify it with the platform, such that emails can be sent from that domain i.e from: [email protected]. This project also leverages MailSenders' email template features so consider that if you wish to incorporate your own email provider.

Usage

pip install .
$ python coles_vs_woolies --help
usage: coles_vs_woolies [-h] {search,email,cache} ...

Compare prices between Aussie grocers

options:
  -h, --help            show this help message and exit

actions:
  {search,email,cache}
    search              Search products
    email               Emails search results
    cache               Clears requests' cache
cp .env.example .env
# populate .env to simplify calls
cp shopping-list.example.json shopping-list.json
# populate the shopping list with your email & desired items

Install w/ GitHub Actions

n.b. I found sporadic success with GitHub-hosted runners; I would recommend setting up self-hosted GitHub runners for consistent success if you wished to continue using GitHub actions rather than running a simple cron job.

  1. Fork this repo
  2. Read the GitHub Action workflow run.yml
  3. Add GitHub Action Variables & Secrets for those in the run.yml
    • n.b. storing a shopping-list.json as a minified json string should do the job
  4. Manually invoke the GitHub Action & confirm an email was received

Getting the best results

An optimistic approach to product-search is used - equivalent to Google's "I'm feeling lucky". Consequently, there are some edge-cases and I advise for the inclusion of the product's brand, weight, and/or package size in the search-term provided.

The email will display the search-term used rather than the individual grocer's search-result product. This can vary. If a search-term is too vague, the first-item returned may not be what you're looking for. You can always find out which product the price belongs to by clicking the price which will direct you to the product's webpage. Alternatively, the display command will show verbose details to help tailor search-terms for weekly emails.

โŒ "Chocolate" - too generic
โŒ "Cadbury Chocolate" - still too generic
โœ”๏ธ "Cadbury Chocolate 180g" - now we're talking!
โœ”๏ธ "Cadbury Dairy Milk Chocolate Block 180g" - yes!

Brand Family

If it wasn't clear by now, I'm a fan of Cadbury chocolate and I have learnt that, generally, the entire product family goes on sale at the same time; that is to say most chocolate flavours go on sale: "Dairy Milk", "Marvellous Creations", "Caramilk", etc. You can avoid adding every individual product by picking the one that best represents the product family.

Product Exclusives

If you searched for a product that is exclusive to one of the grocers, the other grocer may suggest an equivalent. E.g. "Unfortunately, we couldn't find results for 'Coles Kitchen Coleslaw 200g' but here's 'Woolworths Classic Coleslaw 200g'". The suggestion system between merchants is quite good, to be honest, so you may not find this an issue.

Wacky stuff (non-food products)

Thanks, Aldi, for encouraging other grocery chains to start selling desk-chairs & circular-saws - it helps make for a lot of edge cases that I don't want to bother with. I mean, who's checking each week for the merchant with the cheapest fog-machines? Note, less wacky non-food products should be fine; e.g. batteries, sanitary products, etc.

Future plans

I have begun building a site where you can search & add specific items as a personalised weekly subscription. This will significantly drop the barrier-to-entry & my non-coder friends will be much happier. It also requires that I brush up my React.js skills, so this will take time.

If you like my work or wish to support this project going forward, the best way you can is to

matthewtimms



coles_vs_woolies's People

Contributors

matttimms avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

coles_vs_woolies's Issues

Add support for IGA

I looked at the IGA website once & it didn't seem too difficult to add as an additional merchant.

Coles `SSLCertVerificationError`

HTTPSConnectionPool(host='www.coles.com.au', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))

GitHub Actions started failing two weeks ago. This coincides with Cole's certificate change; issued on Monday, 6 May 2024 at 10:00:00.

Dropping SSL with session verify=False is not tenable as we can trigger anti-bot detection methods:

<div class="container">
<script>document.getElementsByClassName("container")[0].style.display = "none";</script>
<h1>Pardon Our Interruption</h1>
<p>As you were browsing something about your browser made us think you were a bot. There are a few reasons this might happen:</p>
<ul>
<noscript><li>You've disabled JavaScript in your web browser.</li></noscript>
<li>You're a power user moving through this website with super-human speed.</li>
<li>You've disabled cookies in your web browser.</li>
<li>A third-party browser plugin, such as Ghostery or NoScript, is preventing JavaScript from running. Additional information is available in this <a href="http://ds.tl/help-third-party-plugins" target="_blank" title="Third party browser plugins that block javascript">support article</a>.</li>
</ul>
<p>To regain access, please make sure that cookies and JavaScript are enabled before reloading the page.</p>
</div>

Add distinction between sale/non-sale items

I'd like some visual distinction in the email for items on sale vs. not on sale. Baseline prices for items may vary between grocers; usually small amounts, 50c, but it can mislead & suggest the product is on sale.

This doesn't matter if you consider the highlighted items as the "cheapest-offering" vs "what's on sale".

Tests fail on GH actions

The classic "they work on my machine".
Issue is specific to "woolies" unit tests.

Local tests are hitting Woolies' Sydney servers which appear as the main service. GH Actions are hitting the Melbourne servers.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.