Git Product home page Git Product logo

southwest-price-drop-bot's Introduction

ALERT!

Deployed versions prior to 6/30/2019 (< 3.4.0) might want to do a clean deployment - we're changing from Redis to MongoDB, and it doesn't translate, cleanly. This is able to run locally during development somewhat consistently.

Southwest Price Drop Bot

This tool lets you monitor the price of Southwest flights that you've booked. It will notify you if the price drops below what you originally paid. Then you can re-book the same flight and get Southwest credit for the price difference. This tool also lets you monitor the price of all Southwest flights on a given day. It will notify you if any flight on that day drops below the previous cheapest flight.

Note that to send text messages you need a Plivo account and to send emails you'll need a Mailgun account (or SMTP credentials). You can also send discord alerts through a webhook. You can run this tool without these accounts, but you won't get the notifications.

You can log in with either:

  • The admin username/password combo, example: admin and the-admin-password-123
  • A username/password combo, example: mom and the-admin-password-123

The second option is nice when giving out access to friends and family since it will only display alerts for the given username. Note that the password is the same for all accounts, and the admin can see all alerts.

When creating alerts, note that the email and phone numbers are optional. If those are both left blank, the user will need to manually log in to view price drops.

Deployment

  1. Click this button: deploy
  2. Create a MongoDB Atlas database and note the connection string then add this string as a config variable named MONGODB_URI
  3. Fill out the remaining config variables and click Deploy
  4. Open up the Heroku Scheduler from your app's dashboard
  5. Add an hourly task that runs npm run task:check

When updates become available, you will have to deploy them yourself using the Heroku CLI. This app follows SemVer in its versioning, so make sure to read the release notes when deploying a major version change.

Note: Deployed versions prior to 4/9/2018 using Mailgun will need to verify constants: MAILGUN_DOMAIN and MAILGUN_EMAIL.

Note: Deployed versions prior to 4/28/2018 (< 3.0.0) on Heroku will need to install the buildpack https://github.com/jontewks/puppeteer-heroku-buildpack

Note: Deployed versions prior to 7/21/2018 (< 3.2.0) on Heroku will need to verify the PROXY constant if you want to use a proxy to make the calls.

Note: Deployed versions prior to 6/30/2019 (< 3.4.0) might want to do a clean deployment - we're changing from Redis to MongoDB, and I don't think it will migrate cleanly (or at all). Otherwise, you'll need to add the mLab MongoDB add-on manually.

Screenshots

Southwest Bot Protection

Southwest has some very fancy bot protections in place.

  • Heroku IPs (which is hosted on AWS), and other hosting providers, are blocked from accessing their site. Local deployments should be permitted to access their site, and some other cloud providers may work as well. The most reliable workaround is using a residential proxy service.
  • There's also some tricky and obfuscated Javascript that detects headless browsers and is updated very frequently. There's a community of folks that implement headless chrome detection evasions, but it's a cat and mouse game.
  • Use CHROME_DEBUG=true DEBUG="puppeteer:*" combined with node inspect to debug strange chrome issues.
    • Request interception will log all URL load attempts and accept all requests.
    • slowmo is enabled and headless is disabled
    • https://infosimples.github.io/detect-headless will be opened before a Southwest URL

Proxy information

Instructions on deploying a proxy is outside the scope of this project. However, here's some information about proxies that might be useful:

  • A hosted (cheap) proxy that works is https://luminati.io. It's less than $1 each month and seems reliable. Most public proxies don't seem to work, I imagine there is some sort of public proxy block list that is in place.
  • You could use something like Squid and spin in up natively, in a container, or in a VM. Obviously you'll want to do this outside of Heroku
  • If you do use Squid, you'll want to set up port forwarding or running on a high random port, and locking down squid.conf with something like this to prevent someone from using your setup as an open proxy:
acl swa dstdomain .southwest.com
http_access allow swa
http_access deny all

To configure the Price Drop Bot to use your proxy, define a new PROXY variable within the Heroku Config. The proxy format should be http://IP:port. Example: heroku config:set PROXY='http://123.123.123.123:1234'

Development

To run the test suite:

yarn test

To run a console loaded up with Alert and Flight objects:

yarn console

When debugging chrome/puppeteer issues it's helpful to use the following command:

DEBUG="puppeteer:*" CHROME_DEBUG=true node tasks/check.js

This will send helpful chromium debugging output into your console, switch off headless mode, and enable some additional logging to help debug what might be going wrong.

Chromium Codesign Issues

If you are running into firewall notifications on macos, you'll need to sign the chromium binary:

/usr/bin/find . -name "Chromium.app" | xargs sudo codesign --sign - --force --deep

Docker Deployment

There are 3 containers in the docker setup:

  • mongo - container running mongodb
  • nodeapp - container running the frontend
  • nodescheduler - container running the check every 60 minutes

To run via docker-compose:

Create your .env file from the example. Set the mongo DB url like:

MONGODB_URI="mongodb://mongodb:27017/sw_db"

Then you can start up the docker instance.

docker-compose build
docker-compose up -d

The interface will be available on http://<dockerhost>:3000

Raspberry Pi Docker Deployment

There's a separate docker-compose and Dockerfile for the Raspberry Pi. Chrome installation on a raspberry pi device works differently, and it doesn't support mongodb by default.

docker-compose -f docker-compose.pi.yml up -d

Changelog

Attribution

This is a fork of minamhere's fork of maverick915's fork of scott113341's original project.

Downstream changes were integrated from:

Thanks to the following for their contributions:

  • @evliu - target the price list items more dynamically
  • @GC-Guy - proxy support
  • @iloveitaly - MongoDB, updated scraping/proxy support, anti-bot detection
  • @ribordy - lodash fix

southwest-price-drop-bot's People

Contributors

brussrus avatar chuckmac avatar dependabot[bot] avatar evliu avatar ibushong avatar iloveitaly avatar jaredkaczynski avatar minamhere avatar ribordy avatar samyun avatar scott113341 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

radioraheem123 violaj2011 vaidyk pmschartz chrismtb dunningdan mar5477 evliu gc-guy redbluefire stevedavison probinguranus tunganhgau ribordy fjo trickinit wangzy amshearer razzamatazm iloveitaly jglassenberg kmumm driscoll42 tanikonda5 hoopsbwc34 steadystatic staceb ch0rn kilo-watt jaredkaczynski chuckmac brussrus networkprogrammer jerbear2008 carynbear brianbacke ayofalola240 padenj cpsc310spring2017 liubrandon brogi

southwest-price-drop-bot's Issues

Frequent failures to fetch price

I've been experiencing relatively consistent failures to fetch prices, even when routing through a proxy hosted on DigitalOcean. It seems like most attempts to fetch prices result in an error page. I suspect Southwest has changed something in regards to detecting headless browsers, as I'm seeing similar results when trying to view prices (either directly via URL or via manipulating inputs and clicking "Search") via a selenium chromedriver used locally. I'm not sure what the best path forward is here; I'll spend some time over the next few weeks trying to see if there's a workaround, but if the site is detecting headless browsers, it may be the end of the road for this project.

Outdated Yarn lockfile - Can't Build on Heroku 

  -----> Building on the Heroku-20 stack
  -----> Using buildpacks:
         1. heroku/nodejs
         2. https://github.com/jontewks/puppeteer-heroku-buildpack
  -----> Node.js app detected
         
  -----> Creating runtime environment
         
         NPM_CONFIG_LOGLEVEL=error
         USE_YARN_CACHE=true
         NODE_VERBOSE=false
         NODE_ENV=production
         NODE_MODULES_CACHE=false
         
  -----> Installing binaries
         engines.node (package.json):  ^17.1.0
         engines.npm (package.json):   ^8.1.0
         engines.yarn (package.json):  ^1.22.10
         
         Resolving node version ^17.1.0...
         Downloading and installing node 17.6.0...
         Bootstrapping npm ^8.1.0 (replacing 8.5.1)...
         npm ^8.1.0 installed
         Resolving yarn version ^1.22.10...
         Downloading and installing yarn (1.22.17)
         Installed yarn 1.22.17
         
  -----> Restoring cache
         Caching has been disabled because NODE_MODULES_CACHE=false
         
  -----> Installing dependencies
         Installing node modules (yarn.lock)
         yarn install v1.22.17
         [1/4] Resolving packages...
         error Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`.
         info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
  -----> Build failed
   !     Outdated Yarn lockfile
         Your application contains a Yarn lockfile (yarn.lock) which does not
         match the dependencies in package.json. This can happen if you use npm
         to install or update a dependency instead of Yarn.
         Please run the following command in your application directory and check
         in the new yarn.lock file:
         $ yarn install
         $ git add yarn.lock
         $ git commit -m "Updated Yarn lockfile"
         $ git push heroku main
      
         https://help.heroku.com/TXYS53YJ
   !     Push rejected, failed to compile Node.js app.
   !     Push failed

Clicked the link on the README.md and couldn't build.

Proxy not working when running schedule

When running npm run task:check from the scheduler or from console it doesn't seem to be using the proxy settings. I see no connections on my proxy log. However from the app web interface, if I add a flight for the first time, or go back and edit/save, then it uses the proxy and completes the check successfully.

From the console I see Access Denied when it runs the check.

Thanks in advance for any help. This app is awesome!

Wishlist (Points)

I buy my flights exclusively with points, it would be AWESOME if it wasn't hard to integrate. However I'm assuming it is so just gonna put this here in case someone get's bored and wants to pull request it.

Prices Not Found

I just deployed the latest to heroku and my instance is not able to pull up the prices for a flight:

2018-06-13T03:11:26.289516+00:00 app[web.1]: Unable to find flight information on page: https://www.southwest.com/air/booking/select.html?adultPassengersCount=1&departureDate=2018-08-31&departureTimeOfDay=ALL_DAY&destinationAirportCode=FLL&fareType=USD&originationAirportCode=BWI&passengerType=ADULT&returnDate=&returnTimeOfDay=ALL_DAY&seniorPassengersCount=0&tripType=oneway
2018-06-13T03:11:26.297827+00:00 app[web.1]: No flights found!
2018-06-13T03:11:26.297926+00:00 app[web.1]: Min price: Infinity
2018-06-13T03:11:26.298792+00:00 app[web.1]: Got price: 8/31/2018|BWI|FLL|1569 { time: 1528859455733, price: Infinity }

Flight 1569 does exist on the page it logged above.

Grid or list layout?

Hi, would it be possibly to layout the content as a grid or list on desktop or tablet browsers? thanks!

Proxy support

Given the challenges with Heroku IPs being blocked, would it be feasible to add support for an optional proxy parameter? If we could feed in our own proxy or proxies to be used to query southwest.com, it could allow it to still run in heroku and just bounce off a proxy to bypass the IP block. This would also be useful in local mode or other cloud providers in case they get more aggressive on blocking IPs that send multiple queries.

TypeError: Plivo.RestAPI is not a function

Running into this with a fresh install on master:

TypeError: Plivo.RestAPI is not a function
    at getPlivo (/home/ubuntu/southwest-price-drop-bot/lib/bot/send-sms.js:30:16)
    at Object.sendSms (/home/ubuntu/southwest-price-drop-bot/lib/bot/send-sms.js:10:17)
    at values.map.sort.map (/home/ubuntu/southwest-price-drop-bot/tasks/check.js:77:57)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

Looks to be a result of jumping from 0.4.1 to 4.0.1 here. The API of that package changed between versions. Happy to open a PR to fix--seems that reverting to 0.4.1 will resolve things in the short term, but unsure if it's preferred to instead attempt to wholesale update all plivo references to use the new package's API

Datastore Persistence

My Redis instance was rebooted by Heroku. As a result, I lost all flights I had been tracking. I don't think Redis is usually intended to be used as a persistent data store, but the paid tiers (starting at $15/mo) of the Heroku Redis add-on do provide persistence and backups. I imagine most users looking to utilize this would be unwilling to pay that cost.

I haven't poked around much to see how data is being stored, but perhaps it's worth exploring something like Mongo or Postgres?

(Mostly looking to start a dialog to flesh out any unseen hurdles before I take some time to further investigate)

I think a postgres DB configured with an ID column and a JSON blob column could more or less serve as a drop-in replacement for Redis

Error rendering history graph

My instance successfully fetched some new prices for flights last night, but I'm now unable to view any flight page (request hangs before the app crashes entirely). Stacktrace below, seems it's coming from the history graph component. I'll poke around at this tonight, but I wanted to surface this now in case anyone else was running into a similar problem.

2019-07-18T16:36:20.026194+00:00 app[web.1]: (node:23) UnhandledPromiseRejectionWarning: TypeError: Class constructor CoreMongooseArray cannot be invoked without 'new'
2019-07-18T16:36:20.026212+00:00 app[web.1]:     at initCloneArray (/app/node_modules/lodash.clonedeep/index.js:1235:22)
2019-07-18T16:36:20.026214+00:00 app[web.1]:     at baseClone (/app/node_modules/lodash.clonedeep/index.js:854:14)
2019-07-18T16:36:20.026216+00:00 app[web.1]:     at cloneDeep (/app/node_modules/lodash.clonedeep/index.js:1399:10)
2019-07-18T16:36:20.026220+00:00 app[web.1]:     at render (/app/lib/history-graph.js:16:16)
2019-07-18T16:36:20.026223+00:00 app[web.1]:     at /app/lib/apps/app.js:137:47
2019-07-18T16:36:20.026225+00:00 app[web.1]:     at processTicksAndRejections (internal/process/task_queues.js:85:5)
2019-07-18T16:36:20.026227+00:00 app[web.1]:     at emitUnhandledRejectionWarning (internal/process/promises.js:141:15)
2019-07-18T16:36:20.026229+00:00 app[web.1]:     at processPromiseRejections (internal/process/promises.js:202:23)
2019-07-18T16:36:20.026232+00:00 app[web.1]:     at processTicksAndRejections (internal/process/task_queues.js:86:32)
2019-07-18T16:36:20.026332+00:00 app[web.1]: (node:23) TypeError: Class constructor CoreMongooseArray cannot be invoked without 'new'
2019-07-18T16:36:20.026336+00:00 app[web.1]:     at initCloneArray (/app/node_modules/lodash.clonedeep/index.js:1235:22)
2019-07-18T16:36:20.026338+00:00 app[web.1]:     at baseClone (/app/node_modules/lodash.clonedeep/index.js:854:14)
2019-07-18T16:36:20.026340+00:00 app[web.1]:     at cloneDeep (/app/node_modules/lodash.clonedeep/index.js:1399:10)
2019-07-18T16:36:20.026342+00:00 app[web.1]:     at render (/app/lib/history-graph.js:16:16)
2019-07-18T16:36:20.026345+00:00 app[web.1]:     at /app/lib/apps/app.js:137:47
2019-07-18T16:36:20.026347+00:00 app[web.1]:     at processTicksAndRejections (internal/process/task_queues.js:85:5)

EDIT: Wanted to add in that the history graph portion that's failing hasn't changed in years, so my initial suspicion is that something changed in regards to the way we're storing and/or retrieving alerts.

User:Pass Proxy error

If I'm using a proxy with USER:PASS i get "Couldn't fetch price. Double-check inputs!". Do you support proxies with USER:PASS?

Failed to return value

So 1) to make this work in DOCKER I had to run the following:

install chrome

RUN wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
RUN dpkg -i google-chrome-stable_current_amd64.deb; apt-get -fy install

  1. It pulls and checks the site
    mongo successfully connected!

found 2 alerts, checking...

lock has available permits: 5

lock has available permits: 4

Entered lock, available permits: 3

Entered lock, available permits: 3

Retrieving URL: https://www.southwest.com/air/booking/select.html?originationAirportCode=DAL&destinationAirportCode=LAS&returnAirportCode=&departureDate=2022-04-X&departureTimeOfDay=ALL_DAY&returnDate=&returnTimeOfDay=ALL_DAY&adultPassengersCount=2&seniorPassengersCount=0&fareType=USD&passengerType=ADULT&tripType=oneway&promoCode=&reset=true&redirectToVision=true&int=HOMEQBOMAIR&leapfrogRequest=true

Retrieving URL: https://www.southwest.com/air/booking/select.html?originationAirportCode=LAS&destinationAirportCode=DAL&returnAirportCode=&departureDate=2022-04-X&departureTimeOfDay=ALL_DAY&returnDate=&returnTimeOfDay=ALL_DAY&adultPassengersCount=1&seniorPassengersCount=0&fareType=USD&passengerType=ADULT&tripType=oneway&promoCode=&reset=true&redirectToVision=true&int=HOMEQBOMAIR&leapfrogRequest=true

but after about 5mins it returns alot of code/html (but doesn't seem to be the right data, when i try and search it, it doesn't have any flight info in it) and I get the following:

Error: Protocol error (Runtime.callFunctionOn): Session closed. Most likely the page has been closed.

at CDPSession.send (/home/app/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:214:35)

at next (/home/app/node_modules/puppeteer-extra-plugin-stealth/evasions/sourceurl/index.js:32:41)

at CDPSession.send (/home/app/node_modules/puppeteer-extra-plugin-stealth/evasions/sourceurl/index.js:65:16)

at ExecutionContext._evaluateInternal (/home/app/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:204:50)

at ExecutionContext.evaluate (/home/app/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:110:27)

at DOMWorld.evaluate (/home/app/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:95:24)

at runMicrotasks (<anonymous>)

at processTicksAndRejections (node:internal/process/task_queues:96:5)

No flights found!

No flights found!

Min price: Infinity

Min price: Infinity

Got price: 4/X/2022|LAS|DAL|2644 { time: 1640724203685, price: Infinity }

Got price: 4/X/2022|DAL|LAS|1826 { time: 1640724203684, price: Infinity }

4/X/2022 #2644 LAS → DAL not cheaper

4/X/2022 #1826 DAL → LAS not cheaper

I removed some dates....but I am able to take the URL that the script uses and it pulls up fine, but I guess SW see's its from a machine and gives me a "Sorry, we found some errors...
We are unable to process your request."

docker-compose doesn't work?

I receive the following error each time I try to do docker-compose up -d I am, running on a VM so I just took a screenshot.

image

Docker-Compose Errors 

$ docker-compose --version
docker-compose version 1.24.1, build 4667896b

$ docker --version
Docker version 24.0.2, build cb74dfc

$ docker-compose build
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.nodeapp.depends_on contains an invalid type, it should be an array
services.nodescheduler.depends_on contains an invalid type, it should be an array

Missing package for puppeteer chrome 

southwest_app    | /home/app/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:241
southwest_app    |             reject(new Error([
southwest_app    |                    ^
southwest_app    |
southwest_app    | Error: Failed to launch the browser process!
southwest_app    | /home/app/node_modules/puppeteer/.local-chromium/linux-982053/chrome-linux/chrome: error while loading shared libraries: libgbm.so.1: cannot open shared object file: No such file or directory
southwest_app    |

Had to add libgbm1 to the apt install in the Dockerfile.

npm run task:check -- Process exited with status 143

When the order of my buildpacks is Node.js then Puppeteer, I receive the following error:

Apr 28 19:22:09 appname app/api: Starting process with command `npm run task:check` by user [email protected] 
Apr 28 19:22:21 appname heroku/scheduler.6190: Starting process with command `npm run task:check` 
Apr 28 19:22:23 appname heroku/scheduler.6190: State changed from starting to up 
Apr 28 19:22:24 appname app/scheduler.6190: > [email protected] task:check /app 
Apr 28 19:22:24 appname app/scheduler.6190: > node --trace-warnings tasks/check.js 
Apr 28 19:22:25 appname app/scheduler.6190: checking 6 flights 
Apr 28 19:22:25 appname app/scheduler.6190: (node:20) UnhandledPromiseRejectionWarning: Error: Failed to launch chrome! 
Apr 28 19:22:25 appname app/scheduler.6190: /app/node_modules/puppeteer/.local-chromium/linux-549031/chrome-linux/chrome: error while loading shared libraries: libX11-xcb.so.1: cannot open shared object file: No such file or directory 
Apr 28 19:22:25 appname app/scheduler.6190: TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md 
Apr 28 19:22:25 appname app/scheduler.6190:     at onClose (/app/node_modules/puppeteer/lib/Launcher.js:255:14) 
Apr 28 19:22:25 appname app/scheduler.6190:     at Interface.helper.addEventListener (/app/node_modules/puppeteer/lib/Launcher.js:244:50) 
Apr 28 19:22:25 appname app/scheduler.6190:     at emitNone (events.js:111:20) 
Apr 28 19:22:25 appname app/scheduler.6190:     at Interface.emit (events.js:208:7) 
Apr 28 19:22:25 appname app/scheduler.6190:     at Interface.close (readline.js:370:8) 
Apr 28 19:22:25 appname app/scheduler.6190:     at Socket.onend (readline.js:149:10) 
Apr 28 19:22:25 appname app/scheduler.6190:     at emitNone (events.js:111:20) 
Apr 28 19:22:25 appname app/scheduler.6190:     at Socket.emit (events.js:208:7) 
Apr 28 19:22:25 appname app/scheduler.6190:     at endReadableNT (_stream_readable.js:1064:12) 
Apr 28 19:22:25 appname app/scheduler.6190:     at _combinedTickCallback (internal/process/next_tick.js:138:11) 
Apr 28 19:22:25 appname app/scheduler.6190:     at process._tickCallback (internal/process/next_tick.js:180:9) 
Apr 28 19:22:25 appname app/scheduler.6190:     at emitWarning (internal/process/promises.js:65:17) 
Apr 28 19:22:25 appname app/scheduler.6190:     at emitPendingUnhandledRejections (internal/process/promises.js:109:11) 
Apr 28 19:22:25 appname app/scheduler.6190:     at process._tickCallback (internal/process/next_tick.js:189:7) 
Apr 28 19:22:25 appname app/scheduler.6190: (node:20) Error: Failed to launch chrome! 
Apr 28 19:22:25 appname app/scheduler.6190: /app/node_modules/puppeteer/.local-chromium/linux-549031/chrome-linux/chrome: error while loading shared libraries: libX11-xcb.so.1: cannot open shared object file: No such file or directory

If I change the order of the buildpacks to Puppeteer then Node.js, I receive the following error:

Apr 28 21:49:28 appname heroku/scheduler.9651:  Starting process with command `npm run task:check` 
Apr 28 21:49:28 appname heroku/scheduler.9651:  State changed from starting to up 
Apr 28 21:49:30 appname app/scheduler.9651:  > [email protected] task:check /app 
Apr 28 21:49:30 appname app/scheduler.9651:  > node --trace-warnings tasks/check.js 
Apr 28 21:49:31 appname app/scheduler.9651:  checking 6 flights 
Apr 28 22:00:36 appname heroku/scheduler.9651: Cycling 
Apr 28 22:00:36 appname heroku/scheduler.9651: State changed from up to complete 
Apr 28 22:00:37 appname heroku/scheduler.9651: Stopping all processes with SIGTERM 
Apr 28 22:00:37 appname heroku/scheduler.9651: Process exited with status 143

I am noticing that the task doesn't run every 10 minutes even though it is scheduled, more like every 20 minutes. I believe it is because it is taking longer than 10 minutes to "complete" the task, but not sure of the cause.

Docker ERROR?

I get the following when trying to run the docker-compose build

docker-compose build
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.nodeapp.depends_on contains an invalid type, it should be an array
services.nodescheduler.depends_on contains an invalid type, it should be an array

Crash when deploying to Heroku

I'm getting the following error when deploying this new version to Heroku.

2019-07-02T15:01:39+00:00 app[heroku-redis]: source=REDIS addon=redis-concentric-47016 sample#active-connections=1 sample#load-avg-1m=0.05 sample#load-avg-5m=0.115 sample#load-avg-15m=0.115 sample#read-iops=0 sample#write-iops=0 sample#memory-total=15664228kB sample#memory-free=11924100kB sample#memory-cached=728028kB sample#memory-redis=309016bytes sample#hit-rate=0.61501 sample#evicted-keys=0 2019-07-02T15:04:07.641004+00:00 heroku[web.1]: Starting process with command npm run start2019-07-02T15:04:09.817109+00:00 app[web.1]: 2019-07-02T15:04:09.817130+00:00 app[web.1]: > [email protected] start /app 2019-07-02T15:04:09.817132+00:00 app[web.1]: > node --trace-warnings index.js 2019-07-02T15:04:09.817134+00:00 app[web.1]: 2019-07-02T15:04:11.197286+00:00 app[web.1]: 2019-07-02T15:04:11.197322+00:00 app[web.1]: /app/node_modules/mongoose/lib/connection.js:465 2019-07-02T15:04:11.197325+00:00 app[web.1]: throw new MongooseError('Theuriparameter toopenUri()must be a ' + 2019-07-02T15:04:11.197327+00:00 app[web.1]: ^ 2019-07-02T15:04:11.197340+00:00 app[web.1]: Error [MongooseError]: Theuriparameter toopenUri()must be a string, got "undefined". Make sure the first parameter tomongoose.connect()ormongoose.createConnection()is a string. 2019-07-02T15:04:11.197343+00:00 app[web.1]: at new MongooseError (/app/node_modules/mongoose/lib/error/mongooseError.js:14:11) 2019-07-02T15:04:11.197345+00:00 app[web.1]: at NativeConnection.Connection.openUri (/app/node_modules/mongoose/lib/connection.js:465:11) 2019-07-02T15:04:11.197347+00:00 app[web.1]: at Mongoose.connect (/app/node_modules/mongoose/lib/index.js:289:15) 2019-07-02T15:04:11.197349+00:00 app[web.1]: at Object.<anonymous> (/app/lib/mongo.js:4:10) 2019-07-02T15:04:11.197351+00:00 app[web.1]: at Module._compile (internal/modules/cjs/loader.js:776:30) 2019-07-02T15:04:11.197353+00:00 app[web.1]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10) 2019-07-02T15:04:11.197355+00:00 app[web.1]: at Module.load (internal/modules/cjs/loader.js:643:32) 2019-07-02T15:04:11.197357+00:00 app[web.1]: at Function.Module._load (internal/modules/cjs/loader.js:556:12) 2019-07-02T15:04:11.197359+00:00 app[web.1]: at Module.require (internal/modules/cjs/loader.js:683:19) 2019-07-02T15:04:11.197361+00:00 app[web.1]: at require (internal/modules/cjs/helpers.js:16:16) 2019-07-02T15:04:11.197363+00:00 app[web.1]: at Object.<anonymous> (/app/lib/bot/flight.js:1:18) 2019-07-02T15:04:11.197364+00:00 app[web.1]: at Module._compile (internal/modules/cjs/loader.js:776:30) 2019-07-02T15:04:11.197366+00:00 app[web.1]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10) 2019-07-02T15:04:11.197368+00:00 app[web.1]: at Module.load (internal/modules/cjs/loader.js:643:32) 2019-07-02T15:04:11.197370+00:00 app[web.1]: at Function.Module._load (internal/modules/cjs/loader.js:556:12) 2019-07-02T15:04:11.197372+00:00 app[web.1]: at Module.require (internal/modules/cjs/loader.js:683:19) 2019-07-02T15:04:11.197374+00:00 app[web.1]: at require (internal/modules/cjs/helpers.js:16:16) 2019-07-02T15:04:11.197376+00:00 app[web.1]: at Object.<anonymous> (/app/lib/bot/alert.js:6:16) 2019-07-02T15:04:11.197378+00:00 app[web.1]: at Module._compile (internal/modules/cjs/loader.js:776:30) 2019-07-02T15:04:11.197379+00:00 app[web.1]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10) 2019-07-02T15:04:11.197382+00:00 app[web.1]: at Module.load (internal/modules/cjs/loader.js:643:32) 2019-07-02T15:04:11.197384+00:00 app[web.1]: at Function.Module._load (internal/modules/cjs/loader.js:556:12) 2019-07-02T15:04:11.197386+00:00 app[web.1]: at Module.require (internal/modules/cjs/loader.js:683:19) 2019-07-02T15:04:11.197388+00:00 app[web.1]: at require (internal/modules/cjs/helpers.js:16:16) 2019-07-02T15:04:11.197390+00:00 app[web.1]: at Object.<anonymous> (/app/lib/apps/app.js:5:15) 2019-07-02T15:04:11.197392+00:00 app[web.1]: at Module._compile (internal/modules/cjs/loader.js:776:30) 2019-07-02T15:04:11.197394+00:00 app[web.1]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10) 2019-07-02T15:04:11.197396+00:00 app[web.1]: at Module.load (internal/modules/cjs/loader.js:643:32) { 2019-07-02T15:04:11.197398+00:00 app[web.1]: message: 'Theuriparameter toopenUri()must be a string, got "undefined". Make sure the first parameter tomongoose.connect()ormongoose.createConnection()is a string.', 2019-07-02T15:04:11.197401+00:00 app[web.1]: name: 'MongooseError' 2019-07-02T15:04:11.197403+00:00 app[web.1]: } 2019-07-02T15:04:11.210567+00:00 app[web.1]: npm ERR! code ELIFECYCLE 2019-07-02T15:04:11.210765+00:00 app[web.1]: npm ERR! errno 1 2019-07-02T15:04:11.211690+00:00 app[web.1]: npm ERR! [email protected] start:node --trace-warnings index.js 2019-07-02T15:04:11.211839+00:00 app[web.1]: npm ERR! Exit status 1 2019-07-02T15:04:11.212065+00:00 app[web.1]: npm ERR! 2019-07-02T15:04:11.212271+00:00 app[web.1]: npm ERR! Failed at the [email protected] start script. 2019-07-02T15:04:11.212515+00:00 app[web.1]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above. 2019-07-02T15:04:11.220007+00:00 app[web.1]: 2019-07-02T15:04:11.220229+00:00 app[web.1]: npm ERR! A complete log of this run can be found in: 2019-07-02T15:04:11.220316+00:00 app[web.1]: npm ERR! /app/.npm/_logs/2019-07-02T15_04_11_214Z-debug.log 2019-07-02T15:04:11.299612+00:00 heroku[web.1]: State changed from starting to crashed 2019-07-02T15:04:11.282421+00:00 heroku[web.1]: Process exited with status 1 2019-07-02T15:04:18.742172+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=*.herokuapp.com request_id=c3d850fd-a914-474b-970f-df695208aa57 fwd="*" dyno= connect= service= status=503 bytes= protocol=https

Added support for number of passengers

I did a recent fork and included some basic support for including the number of passengers to each of your flight alerts. I find this helpful when I'm looking at different flights for other people or for different trips where the number of people flying is different.

I did not account for existing flights you may already be tracking. That is, this could break those flights as I didn't include any logic for a previous entry in the database that doesn't include a passenger field. I just deleted the few flights I had and recreated them.

A few added notes: I find running with CHROME_DEBUG env variable set to TRUE gives me the fewest issues. I'm running this on my own machine, therefore not Heroku, etc. as a scheduled task and I've had no issues with running it this way. When I don't use the debug it often times doesn't want to load the page and pull prices appropriately. Not certain why running in debug would be different.

Also, I updated puppeteer-extra-plugin-stealth module and maybe a couple of others (acorn, react) to the latest versions. I found there were fewer errors displayed when I ran this from a console. after doing this.

Hopefully, you can include some of these changes in your version.

DOC: Entrypoint scripts need to be chmod +x

The Dockerfile uses the following line:

ADD . /home/app

This line copies all the contents of the git repo into the container which includes 2 .sh scripts that function as entrypoints for the main app container and the job container. These two scripts must be chmod +x BEFORE you build the containers otherwise on docker compose up, the containers will fail to start.

There is no mention of this critical step in the README. Please consider adding to the docs.

Mailgun alternatives?

Does anyone have any ideas for an alternative to Mailgun, now that it seems they’re getting rid of their free tier?

Local Setup help

I installed the app locally and tried to run it via previous instructions " I'm working on some instructions. In the meantime, clone the repo, install node and npm, do npm install from a terminal window in the repo directory, and node --trace-warnings index.js". i get message "app started on undefined" and nothing happened. any help is greatly appreciated. ty

mongo not found; please switch to mongosh

Healthchecks kept failing for the mongo container. Shellexec into the container to run manually:

root@c74fdc7a45b9:/# echo 'db.stats().ok' | mongo localhost:27017/sw_db
bash: mongo: command not found

This is due to mongo removing the old client and replacing with mongosh. Please fix the healthcheck to the following:

    healthcheck:
      test: echo 'db.stats().ok' | mongosh localhost:27017/sw_db --quiet

The healthcheck also runs locally so the host to connect to is always localhost; not the hostname of the container.

Stuck on "Fetching price... Reload the page in a few minutes"

I'm trying the most recent update and I'm using a residential proxy from luminati, I do receive the text message that my flight is being monitored and it looks like "npm run task:check" is running every 10 minutes but im not getting a text notification on the price drop and status stays at "Fetching price... Reload the page in a few minutes"

SMTP Email Notification on Price Drop

Hi,

I am using SMTP for email notification. I do get email for Alert being created but do not get any email on price drop, is there any kind of time lag between UI and send notification on price drop?

Thanks,
BK

App is no longer pulling updated prices from the Southwest site.

It seems that there has been a redesign on SW side of things and the app is no longer updating prices.

From the logs - it appears SW is showing the price as infinity. I purposely increased the price above the current price and it's not giving me updates:

2018-04-24T21:04:02.235741+00:00 app[scheduler.1300]: > [email protected] task:check /app

2018-04-24T21:04:02.235743+00:00 app[scheduler.1300]: > node --trace-warnings tasks/check.js

2018-04-24T21:04:02.235745+00:00 app[scheduler.1300]:

2018-04-24T21:04:04.134700+00:00 app[scheduler.1300]: checking 1 flights

2018-04-24T21:04:05.151598+00:00 app[scheduler.1300]: Infinity

2018-04-24T21:04:05.156428+00:00 app[scheduler.1300]: Got price: 10/26/2018|LAX|LAS|552 { time: 1524603844150, price: Infinity }

2018-04-24T21:04:05.162539+00:00 app[scheduler.1300]: 10/26/2018 #552 LAX → LAS not cheaper

There is a similar project that looks like they figured out the issue here: https://github.com/wcrasta/SWA-Scraper

Discord Support

Is there any chance we can have Discord webhook integration? Would be nice to just have the notifications sent there versus having to use Mailgun or Plivo. I would envision it to be similar to how another program, Tautulli for Plex, is able to post with cards too.

Not working

Followed all of the instructions and have the squid proxy setup on prem. Traffic passes through proxy fine but no flight prices return.

2022-10-03T03:55:29.079326+00:00 heroku[router]: at=info method=GET path="/633a5c84f011632a8f61f579/edit" host=southwest-bhm.herokuapp.com request_id=4ba79b7c-7991-4ab0-a115-947e28bcd40a fwd="xxx.xxx.xxx.164" dyno=web.1 connect=0ms service=113ms status=304 bytes=184 protocol=http
2022-10-03T03:55:29.144719+00:00 heroku[router]: at=info method=GET path="/logo.png" host=southwest-bhm.herokuapp.com request_id=1627bfbf-68ba-40f0-a85c-54410b19a8ee fwd="xxx.xxx.xxx.164" dyno=web.1 connect=0ms service=1ms status=304 bytes=271 protocol=http
2022-10-03T03:55:36.414490+00:00 app[web.1]: Proxy specified: http://xxx.xxx.xxx.164:9010; New proxy: http://xxx.xxx.xxx.164:9010
2022-10-03T03:55:36.507174+00:00 app[web.1]: Browser PID from check is: 116
2022-10-03T03:55:36.545771+00:00 app[web.1]: lock has available permits: 5
2022-10-03T03:55:36.545843+00:00 app[web.1]: Entered lock, available permits: 4
2022-10-03T03:55:36.556035+00:00 heroku[router]: at=info method=POST path="/633a5c84f011632a8f61f579" host=southwest-bhm.herokuapp.com request_id=b3d9c3c2-627d-4f40-8a62-784cf9c9f972 fwd="xxx.xxx.xxx.164" dyno=web.1 connect=0ms service=137ms status=303 bytes=193 protocol=http
2022-10-03T03:55:36.732717+00:00 heroku[router]: at=info method=GET path="/logo.png" host=southwest-bhm.herokuapp.com request_id=97354e19-9662-4fed-9328-f26da12b489d fwd="xxx.xxx.xxx.164" dyno=web.1 connect=0ms service=1ms status=304 bytes=271 protocol=http
2022-10-03T03:55:36.738750+00:00 heroku[router]: at=info method=GET path="/style.css" host=southwest-bhm.herokuapp.com request_id=7a06e571-c982-4472-abd7-b7bc3dfd4e4b fwd="xxx.xxx.xxx.164" dyno=web.1 connect=0ms service=1ms status=304 bytes=269 protocol=http
2022-10-03T03:55:36.654448+00:00 heroku[router]: at=info method=GET path="/633a5c84f011632a8f61f579" host=southwest-bhm.herokuapp.com request_id=683030b6-591a-4e37-b894-7e472de9cd95 fwd="xxx.xxx.xxx.164" dyno=web.1 connect=0ms service=58ms status=200 bytes=2535 protocol=http
2022-10-03T03:55:36.708396+00:00 app[web.1]: Retrieving URL: https://www.southwest.com/air/booking/select.html?int=HOMEQBOMAIR&adultPassengersCount=1&departureDate=2022-10-22&destinationAirportCode=LAS&fareType=USD&originationAirportCode=BHM&passengerType=ADULT&returnDate=&tripType=oneway&departureTimeOfDay=ALL_DAY&reset=true&returnTimeOfDay=ALL_DAY

2022-10-03T03:56:07.665136+00:00 app[web.1]: &nbsp;" class="">© 2022 <style>
2022-10-03T03:56:07.665136+00:00 app[web.1]: [data-class="swa-g-screen-reader-only"] {
2022-10-03T03:56:07.665137+00:00 app[web.1]: border: 0;
2022-10-03T03:56:07.665137+00:00 app[web.1]: clip: rect(0,0,0,0);
2022-10-03T03:56:07.665137+00:00 app[web.1]: height: 1px;
2022-10-03T03:56:07.665137+00:00 app[web.1]: margin: -1px;
2022-10-03T03:56:07.665137+00:00 app[web.1]: overflow: hidden;
2022-10-03T03:56:07.665138+00:00 app[web.1]: padding: 0;
2022-10-03T03:56:07.665138+00:00 app[web.1]: position: absolute;
2022-10-03T03:56:07.665138+00:00 app[web.1]: width: 1px;
2022-10-03T03:56:07.665138+00:00 app[web.1]: }
2022-10-03T03:56:07.665138+00:00 app[web.1]:
2022-10-03T03:56:07.665139+00:00 app[web.1]: .swa-footer--copyright-text, .footer-copyright--text {
2022-10-03T03:56:07.665139+00:00 app[web.1]: width:556px !important
2022-10-03T03:56:07.665139+00:00 app[web.1]: }
2022-10-03T03:56:07.665139+00:00 app[web.1]: #call_to_action_qdnph80bc1i, #call_to_action_oxew9f4k2gj{color:#ffffff!important;}
2022-10-03T03:56:07.665139+00:00 app[web.1]: div.footer div.flex-placement a:hover {
2022-10-03T03:56:07.665140+00:00 app[web.1]: text-decoration: underline !important;
2022-10-03T03:56:07.665140+00:00 app[web.1]: }
2022-10-03T03:56:07.665140+00:00 app[web.1]: div.footer div.flex-placement b a:hover {
2022-10-03T03:56:07.665140+00:00 app[web.1]: text-decoration: none !important;
2022-10-03T03:56:07.665140+00:00 app[web.1]: }
2022-10-03T03:56:07.665140+00:00 app[web.1]:
2022-10-03T03:56:07.665140+00:00 app[web.1]: .swa-footer--copyright-text a{
2022-10-03T03:56:07.665140+00:00 app[web.1]: text-decoration:underline;
2022-10-03T03:56:07.665141+00:00 app[web.1]: }
2022-10-03T03:56:07.665141+00:00 app[web.1]: .swa-footer--copyright-text a:hover{
2022-10-03T03:56:07.665141+00:00 app[web.1]: color:#111B40;
2022-10-03T03:56:07.665141+00:00 app[web.1]: }
2022-10-03T03:56:07.665141+00:00 app[web.1]:
2022-10-03T03:56:07.665141+00:00 app[web.1]: #FlyoutTrigger_5 > div > div > div.overlay-container--flyout-content > div > div.header-flyout--content-section > div:nth-child(3) > a:hover .link--text b * {
2022-10-03T03:56:07.665141+00:00 app[web.1]: color: #111B40;
2022-10-03T03:56:07.665142+00:00 app[web.1]: }
2022-10-03T03:56:07.665142+00:00 app[web.1]:
2022-10-03T03:56:07.665142+00:00 app[web.1]: /* Spanish Homepage footer /
2022-10-03T03:56:07.665142+00:00 app[web.1]:
2022-10-03T03:56:07.665142+00:00 app[web.1]: html[lang="es"] div.footer div.flex-placement > b:nth-child(6) {
2022-10-03T03:56:07.665142+00:00 app[web.1]: left: 277px !important;
2022-10-03T03:56:07.665142+00:00 app[web.1]: }
2022-10-03T03:56:07.665142+00:00 app[web.1]:
2022-10-03T03:56:07.665143+00:00 app[web.1]: html[lang="es"] div.footer div.flex-placement > b:nth-child(7) {
2022-10-03T03:56:07.665143+00:00 app[web.1]: left: 277px !important;
2022-10-03T03:56:07.665143+00:00 app[web.1]: }
2022-10-03T03:56:07.665143+00:00 app[web.1]:
2022-10-03T03:56:07.665143+00:00 app[web.1]: html[lang="es"] div.footer div.flex-placement > div:nth-child(8) {
2022-10-03T03:56:07.665143+00:00 app[web.1]: left: 467px !important;
2022-10-03T03:56:07.665143+00:00 app[web.1]: }
2022-10-03T03:56:07.665144+00:00 app[web.1]:
2022-10-03T03:56:07.665144+00:00 app[web.1]: html[lang="es"] div.footer div.flex-placement > div:nth-child(9) {
2022-10-03T03:56:07.665144+00:00 app[web.1]: left: 278px !important;
2022-10-03T03:56:07.665144+00:00 app[web.1]: }
2022-10-03T03:56:07.665144+00:00 app[web.1]:
2022-10-03T03:56:07.665144+00:00 app[web.1]: html[lang="es"] div.footer div.flex-placement > div:nth-child(10) {
2022-10-03T03:56:07.665144+00:00 app[web.1]: left: 482px !important;
2022-10-03T03:56:07.665145+00:00 app[web.1]: }
2022-10-03T03:56:07.665145+00:00 app[web.1]:
2022-10-03T03:56:07.665145+00:00 app[web.1]: html[lang="es"] div.footer div.flex-placement > span.swa-icon.swa-icon_small.swa-icon_external-link-box {
2022-10-03T03:56:07.665145+00:00 app[web.1]: right: 185px !important;
2022-10-03T03:56:07.665145+00:00 app[web.1]: }
2022-10-03T03:56:07.665145+00:00 app[web.1]:
2022-10-03T03:56:07.665145+00:00 app[web.1]: / Fare chart comparison flyby /
2022-10-03T03:56:07.665146+00:00 app[web.1]: .air-booking-select-compare-fares-secondary .compare-fares-secondary--flyby {
2022-10-03T03:56:07.665146+00:00 app[web.1]: font-style: normal;
2022-10-03T03:56:07.665146+00:00 app[web.1]: font-weight: bold;
2022-10-03T03:56:07.665146+00:00 app[web.1]: }
2022-10-03T03:56:07.665146+00:00 app[web.1]:
2022-10-03T03:56:07.665146+00:00 app[web.1]:
2022-10-03T03:56:07.665146+00:00 app[web.1]: / Temp - GNAV Espanol Chase ad font-sizefix /
2022-10-03T03:56:07.665147+00:00 app[web.1]: html[lang="es"] .header-flyout--promo-section .link--text div { font-size: 12px !important; }
2022-10-03T03:56:07.665147+00:00 app[web.1]: html[lang="es"] .header-flyout--promo-section .link--text div:nth-of-type(1) { font-size: ; }
2022-10-03T03:56:07.665147+00:00 app[web.1]: html[lang="es"] .header-flyout--promo-section .link--text div:nth-of-type(2) { font-size: ; }
2022-10-03T03:56:07.665147+00:00 app[web.1]:
2022-10-03T03:56:07.665147+00:00 app[web.1]: / Air CheckIn Banner Button Fix /
2022-10-03T03:56:07.665147+00:00 app[web.1]: .air-check-in-edit-contact-information-banner--link {
2022-10-03T03:56:07.665147+00:00 app[web.1]: bottom: 200px;
2022-10-03T03:56:07.665148+00:00 app[web.1]: position: absolute;
2022-10-03T03:56:07.665148+00:00 app[web.1]: right: 25px;
2022-10-03T03:56:07.665148+00:00 app[web.1]: }
2022-10-03T03:56:07.665148+00:00 app[web.1]:
2022-10-03T03:56:07.665148+00:00 app[web.1]: .facebook-share-button-command--button {
2022-10-03T03:56:07.665148+00:00 app[web.1]: background-color: #3b5998 !important;
2022-10-03T03:56:07.665148+00:00 app[web.1]: }
2022-10-03T03:56:07.665148+00:00 app[web.1]:
2022-10-03T03:56:07.665149+00:00 app[web.1]: / Air Booking Select - Fare benefit first th width /
2022-10-03T03:56:07.665149+00:00 app[web.1]: .air-booking-select-compare-fares-secondary .compare-fares-secondary--header-cell {
2022-10-03T03:56:07.665149+00:00 app[web.1]: width: 348px;
2022-10-03T03:56:07.665149+00:00 app[web.1]: }
2022-10-03T03:56:07.665149+00:00 app[web.1]: / Air Booking Select - Fare benefit sub-label workaround */
2022-10-03T03:56:07.665149+00:00 app[web.1]: .compare-fares-secondary--row:nth-of-type(5) .compare-fares-secondary--header-cell .compare-fares-secondary--label-addendum::after {
2022-10-03T03:56:07.665149+00:00 app[web.1]: content: "(That money is all yours.)";
2022-10-03T03:56:07.665150+00:00 app[web.1]: color: #636363;
2022-10-03T03:56:07.665150+00:00 app[web.1]: display: inline-block;
2022-10-03T03:56:07.665150+00:00 app[web.1]: font: italic normal 11px/1 Arial;
2022-10-03T03:56:07.665150+00:00 app[web.1]: height: 6px;
2022-10-03T03:56:07.665150+00:00 app[web.1]: white-space: normal;
2022-10-03T03:56:07.665153+00:00 app[web.1]: width: 273px;
2022-10-03T03:56:07.665153+00:00 app[web.1]: }
2022-10-03T03:56:07.665153+00:00 app[web.1]: .compare-fares-secondary--row:nth-of-type(10) .compare-fares-secondary--header-cell .compare-fares-secondary--label div::after {
2022-10-03T03:56:07.665153+00:00 app[web.1]: content: "(Temporarily suspended.)";
2022-10-03T03:56:07.665154+00:00 app[web.1]: color: #636363;
2022-10-03T03:56:07.665154+00:00 app[web.1]: display: block;
2022-10-03T03:56:07.665154+00:00 app[web.1]: font: italic normal 11px/1 Arial;
2022-10-03T03:56:07.665154+00:00 app[web.1]: height: 6px;
2022-10-03T03:56:07.665155+00:00 app[web.1]: white-space: normal;
2022-10-03T03:56:07.665155+00:00 app[web.1]: width: 273px;
2022-10-03T03:56:07.665155+00:00 app[web.1]: padding-left: 30px;
2022-10-03T03:56:07.665155+00:00 app[web.1]: padding-top: 5px;
2022-10-03T03:56:07.665155+00:00 app[web.1]: }
2022-10-03T03:56:07.665155+00:00 app[web.1]:
2022-10-03T03:56:07.665155+00:00 app[web.1]: .header-flyout-links .header-flyout-links--item .swa-icon_search { padding-left: 1px; }
2022-10-03T03:56:07.665156+00:00 app[web.1]: </style>
2022-10-03T03:56:07.665156+00:00 app[web.1]: Southwest Airlines Co. All Rights Reserved. Use of the Southwest websites and our Company Information constitutes acceptance of our Terms and Conditions. Privacy Policy Trademarks Do Not Sell My Personal Information
2022-10-03T03:56:07.665156+00:00 app[web.1]:  

1 error detected. Moving focus to error message.
<script async="" id="mpelid" src="https://southwestairlines.mpeasylink.com/mpel/mpel.js" type="text/javascript"></script><script>_satellite["_runScript2"](function(event, target, Promise) {
2022-10-03T03:56:07.665157+00:00 app[web.1]: !function(e,t,n,a,i,r,s,o,c,d){if(!e[a]||!e[a]._q){for(;o<s.length;)i(r,s[o++]);(c=t.createElement(n)).async=1,c.src="https://cdn.branch.io/branch-latest.min.js",(d=t.getElementsByTagName(n)[0]).parentNode.insertBefore(c,d),e[a]=r}}(window,document,"script","branch",(function(e,t){e[t]=function(){e._q.push([t,arguments])}}),{_q:[],_v:1},"addListener applyCode autoAppIndex banner closeBanner closeJourney creditHistory credits data deepview deepviewCta first getCode init link logout redeem referrals removeListener sendSMS setBranchViewData setIdentity track validateCode trackCommerceEvent logEvent disableTracking".split(" "),0),"swaprod"==_satellite.getVar("global: analytics report suite (CC)")?branch.init("key_live_giugWIpIXUYa4N52lEYdgbcfDyg004TU"):branch.init("key_test_kiCmZHbV9GYaXUX4au1dMeoewweY08T9");
2022-10-03T03:56:07.665158+00:00 app[web.1]: });</script>
<script type="text/javascript" src="https://zn6fetw4rxykndjoz-southwest.siteintercept.qualtrics.com/WRSiteInterceptEngine/?Q_ZID=ZN_6fEtw4RXYKndJOZ"></script><script>_satellite["_runScript3"](function(event, target, Promise) {
2022-10-03T03:56:07.665158+00:00 app[web.1]: QSI&&QSI.API&&QSI.API.unload();
2022-10-03T03:56:07.666951+00:00 app[web.1]: });</script>
<script type="text/javascript" src="https://zn6fetw4rxykndjoz-southwest.siteintercept.qualtrics.com/WRSiteInterceptEngine/?Q_ZID=ZN_6fEtw4RXYKndJOZ"></script><iframe id="LtFYLxhyISzFCZST" style="display: none;"></iframe>

2022-10-03T03:56:07.667003+00:00 app[web.1]: {
2022-10-03T03:56:07.667004+00:00 app[web.1]: 'accept-ranges': 'bytes',
2022-10-03T03:56:07.667004+00:00 app[web.1]: 'cache-control': 'max-age=0, no-cache, no-store',
2022-10-03T03:56:07.667005+00:00 app[web.1]: 'content-encoding': 'gzip',
2022-10-03T03:56:07.667005+00:00 app[web.1]: 'content-length': '2715',
2022-10-03T03:56:07.667005+00:00 app[web.1]: 'content-type': 'text/html',
2022-10-03T03:56:07.667005+00:00 app[web.1]: date: 'Mon, 03 Oct 2022 03:53:50 GMT',
2022-10-03T03:56:07.667005+00:00 app[web.1]: etag: '"ed172c32f32c448d8657cc8133c9fb4d:1661194695.670721"',
2022-10-03T03:56:07.667006+00:00 app[web.1]: expires: 'Mon, 03 Oct 2022 03:53:50 GMT',
2022-10-03T03:56:07.667006+00:00 app[web.1]: 'last-modified': 'Mon, 22 Aug 2022 18:58:15 GMT',
2022-10-03T03:56:07.667006+00:00 app[web.1]: pragma: 'no-cache',
2022-10-03T03:56:07.667007+00:00 app[web.1]: 'set-cookie': 'swa_FPID=d05e4d2f-ca89-4d11-8d05-b1e5a7b96ccb; expires=Mon, 31-Dec-2038 23:59:59 GMT; secure\n' +
2022-10-03T03:56:07.667007+00:00 app[web.1]: 'akavpau_prd_air_booking=1664769830~id=51b684828d21152db08e3247acff52eb; Domain=www.southwest.com; Path=/; HttpOnly; Secure; SameSite=None',
2022-10-03T03:56:07.667007+00:00 app[web.1]: 'strict-transport-security': 'max-age=31536000',
2022-10-03T03:56:07.667008+00:00 app[web.1]: 'terms-of-service': "Unauthorized access, display, or use of Southwest's Company Information, including fare data, is prohibited by the Terms & Conditions on Southwest.com and Swabiz.com.",
2022-10-03T03:56:07.667008+00:00 app[web.1]: vary: 'Accept-Encoding',
2022-10-03T03:56:07.667008+00:00 app[web.1]: 'x-akamai-transformed': 'hl - 0 pmb=mNONE,3',
2022-10-03T03:56:07.667008+00:00 app[web.1]: 'x-frame-options': 'SAMEORIGIN'
2022-10-03T03:56:07.667008+00:00 app[web.1]: }
2022-10-03T03:56:07.667009+00:00 app[web.1]: 200
2022-10-03T03:56:07.713147+00:00 app[web.1]: Unknown request type: ping; URL: https://p11.techlab-cdn.com/collect?t=1664769367708&st=136874&s=nG9zHilhMNlgF5nQ&ss=2&c=615dd91bee22130011c85bd8&r=PwRftwbliHO7kfzH&d=2&u=https%3A%2F%2Fwww.southwest.com%2Fair%2Fbooking%2Findex.html%3FadultPassengersCount%3D1%26departureDate%3D2022-10-20%26departureTimeOfDay%3DALL_DAY%26destinationAirportCode%3DLAS%26fareType%3DUSD%26int%3DHOMEQBOMAIR%26originationAirportCode%3DBHM%26passengerType%3DADULT%26reset%3Dtrue%26returnDate%3D%26returnTimeOfDay%3DALL_DAY%26tripType%3Doneway%26validate%3Dtrue&v=747620529&p=1&bv=9&rh=0&pi=-1&pl=-1&pwl=-1&ple=-1&psd=-1&ppu=-1&psl=-1&pfu=38&phe=15&pue=-1&pbc=12&pnu=68&pnc=1&pnr=1&fsp=0&sp=1&ah=0&sm=1&tr=1
2022-10-03T03:56:07.738178+00:00 app[web.1]: Page closed
2022-10-03T03:56:07.738311+00:00 app[web.1]: Exited lock, available permits: 5
2022-10-03T03:56:07.785378+00:00 app[web.1]: No. of departing flights found: 0
2022-10-03T03:56:07.785388+00:00 app[web.1]: No flights found!
2022-10-03T03:56:07.785424+00:00 app[web.1]: Min price: Infinity
2022-10-03T03:56:07.786056+00:00 app[web.1]: Got price: 10/20/2022|BHM|LAS|All { time: 1664769230161, price: Infinity }

Running on M1 Apple Silicon Mac

Trying to figure out how to get this working on an m1 mac. Main problem is with puppeteer.

Currently researching how this bug plays into it. puppeteer/puppeteer#6957 (comment)
Will update the ticket if I figure out how to run on m1 mac.

  1. Install chromium brew reinstall --no-quarantine chromium - this will help if you want to install chromium not in docker.

Docker Hub Build

If you want to create an account at Docker Hub I can assist with the GitHub Actions updates needed to automatically push new builds over. That way the images can be pulled instead of manually building them.

Random user agent

Seeing as how SWA blocked Heroku IPs, it's likely another step they will take is looking at User Agent strings and blocking specific ones. The default for Puppeteer is not stealthy, so it may be a good idea to get ahead of them and put in a list of the 10 or 20 most common UAs and randomly rotate through them. The function to do it is
page.setUserAgent(userAgent)
and docs are at https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagesetuseragentuseragent if someone gets bored.

SW page shows maintenance error.

I got the proxy configured with the heroku app.
When using the application it logs SW is under maintenance.
When doing ssh tunnel to the server that hosts the squid proxy the exact url that the app sends works.
I wonder if they are detecting and blocking puppet? Or did I not configure it correctly?

<h2>Thanks For Stopping By!</h2><p>Getting you to where you want to go matters to us. However, this portion of the Southwest Airlines website is undergoing maintenance and is currently unavailable.&nbsp; We are working to make it available again soon.&nbsp; We apologize for any inconvenience this may cause, but please try again later.</p>

So does this work?!

I'm still running the last build of this with a local Redis server and it appears this is no longer working. I'm confused if switching to the new build will fix anything as the Readme makes it sound like this tool doesn't work. So does it? If I run this on a local server am I being blocked because of the headless checks Southwest uses. I don't want to do a big switch to mongodb if the actual scraping of prices is still not going to work. Their website comes up fine through a normal browser so just wondering if I replace the new price grab js code will fix my problem.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.