Git Product home page Git Product logo

yogasw / olx-crawler Goto Github PK

View Code? Open in Web Editor NEW
11.0 1.0 2.0 9.83 MB

This repository contains code bot to monitor OLX sites and extract complete product data and send it to WhatsApp. support: OLX Indonesia (https://www.olx.co.id), OLX India (https://www.olx.in), OLX Pakistan (https://www.olx.com.pk), OLX South Africa (https://www.olx.co.za/). OLX Crawler , OLX Scraper , OLX Bot

PHP 75.83% Go 22.12% Shell 1.06% Dockerfile 0.98%
golang php bash docker docker-compose cron alpine rabbitmq whatsapp olx

olx-crawler's Introduction

OLX-Crawler

OLX Crawler is a bot that is used to monitor OLX sites periodically and send automatically to WhatsApp when there is new item on OLX site in detail
Support :

Installation

1. Install docker and docker compose

Make sure you've installed Docker and Docker compose because this project is using Docker all in order to minimise installation errors

2. Create a spreadsheet on Google Sheets

You can copy my spreadsheet to this: https://docs.google.com/spreadsheets/d/1a99s1jK3T-wBZI1rlpLsECdi5WFnpq8h3U6uUP8oNIs/edit?usp=sharing When you have the sheet open in your browser, the URL will look something like this: https://docs.google.com/spreadsheets/d/1-XXXXXXXXXXXXXXXXXXXSgGTwY/edit#gid=0. And in this URL, 1-XXXXXXXXXXXXXXXXXXXSgGTwY is the spreadsheet's ID and it will be different for each spreadsheet.

Save the spreadsheet ID to the. env file and change the value SPREED_SHEET_ID

3. Enable Google Sheets API in Google developers console

You can read this tutorial in step 2 and 3 https://medium.com/swlh/how-to-read-or-modify-spreadsheets-from-google-sheets-using-node-js-6f5a672bdd37#ed85

Save the Google sheets JSON credentials API to the folder crawl-olx, Change the SPREED_SHEET_AUTH value with file name credentials.json

4. Complete the contents of the .env file

MASTER_PHONE_NUMBER (Telfon number you use to send WHATSAAP messages) TARGET_WA_MESSAGE (recipients of the message), PAGE_URL_OLX (the URL of the website OLX you want to monitor)

5. Run start.sh

Run Command ./start.sh And wait for the Docker compose process to run, until it displays a QR code

If you have not had the time to scan the QR code but it has expired QR code, when run CTRL + X, and Start command ```./start.sh``

Wait 30 minutes once the bot will send a message on your WhatsApp

If successful will display the response [*] Waiting for messages. To exit press CTRL+C

Example receive messages

Beat Street 2019 akhir AB sleman (km 9rb'n.pjk panjang)

Ad description
Created At :
2020-06-01T12:48:28+07:00

Location :
https://maps.google.com/?q=-7.803,110.438

Image :
https://apollo-singapore.akamaized.net:443/v1/files/s9jg9ezk972s1-ID/image, https://apollo-singapore.akamaized.net:443/v1/files/x2uhjpq6ysyo2-ID/image, https://apollo-singapore.akamaized.net:443/v1/files/onfoxqc25k5i1-ID/image

Price : Rp 12.500.000

Parameters :
Merek : Honda
Model : Beat
Tahun : 2019
Jarak tempuh : 20.000-25.000 km
Tipe Penjual : Individu
phone : +62813927xxxxx
wa : https://wa.me/62813xxx

Description :
Jual cepat BU bgt beat street 2019 AB sleman Standar.mulus.orisinil (km 9rb'n) Pajak hidup bln november 2020..Stnk.bpkb & faktur + kunci serep (lengkap) Harga:12,5_passs/nettt (nego NO RESPON) Lokasi:Berbah (sleman) Yg serius lgsg tlp/wa aja.. Nb:Shok blkg pake punya yamaha X_ride ori empuk & lbh tinggi (yg ori lupa nyimpen)

Url :
www.olx.co.id/item/beat-street-2019-akhir-ab-sleman-km-9rbnpjk-panjang-iid-784397648

User description :
Name : Eddy Susanto

Profile :
https://www.olx.co.id/profile/65695098

Location :
https://maps.google.com/?q=-7.803,110.438

Badges : Facebook, G+, Phone number

Legal

This code is in no way affiliated with, authorized, maintained, sponsored or endorsed by OLX or any of its affiliates or subsidiaries. This is an independent and unofficial software. Use at your own risk.

olx-crawler's People

Contributors

yogasw avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

Forkers

mughu94 irfnrdh

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.