deenuu1 / meta-spy Goto Github PK

from this class all scrapers will inherit
Methods:

• _chrome_driver_configuration

• Add method to scrape data about first user
• Scraped user should has relations to users from friends list

• First fix imports
• Uncomment tests for models
• Add tests for repository.py
• Add CI/CD to run automatically tests

• Create a script to scrape next URL

Do this task after finishing all scrapers

• Save data to sqlite database
  (images - only path to the folder)

• Model QUeue
• Repository

• Private variables
• Move chrome driver configuration inside class

• Contact data
• Links to other social media
• Sex
• Date of birth

• Informations about
• pronunciation of name and surname
• Other name/surname
• Quotes

• Scrape relation ship status
• Scrape family members

• Add second command to run facebook using saved cookies

• Add pipeline to run all methods in FacebookLogin class

• Create a class Facebook login and change functions into methods
• Add first command to log in to the facebook

• Create a queue with a list of users to scrape
• Model queue
• SQLite to store queue
• After scraping all friends from 1 user it should start to scrape another user's friends and also add them to the queue

School history

• name
• date

• Save scraped users to database (avoid creating duplicates !!! based on URL )

Every class should has a one big pipeline to do all possible operations and also all extract method should has own pipeline to do 1 specified task

Move pipelines to separated functions instead of using them as a class methods

User's images and tagged

• Add script to scrape all friends from a given user facebook account
• Add command to run this script

• add .env_example
• add pre-commit

Work history

• name
• date

• Add integraiton with sqlite
• Create models
• Create shemas
• Create function to operate on db
• Save scraped data
• Add pytest
• Add CI/CD
• Add dockerfile
• Make this repo public

Make separated class to scrape friends because now driver refresh every page load

• Add more interaction in pipelines
• Progress bar

• It is not possible to scrape that amount of data in 1 session so I change the project idea to scrape only 1 person
• Save a list of friends, Save profile and background picture, Save work, school, city, relation ship status, Email, phone etc , Photos, Groups And everything what is possible