linkedin-profiles-scraping's Introduction

Web Scraping: Crawling Linkedin Profiles

Automatically scrape the web data of people profiles on Linkedin based on a specific search query

Tutorial:

English: https://youtu.be/zkfLAY2OrtI
Vietnamese: https://youtu.be/hfnBswCe4QE

Problem:

It takes 10s on average to skim through Linkedin profiles and copy that information into an excel sheet. To collect a large enough amount of data for analysis purposes, it will take time if done manually.

Approach to the problem:

To efficiently collect and cluster Linkedin profiles data, this scrip helps automatically scrape the web data of people profiles on Linkedin based on a specific search query and store the output in a CSV file

Sponsor

Scrape public LinkedIn profile data at scale with Proxycurl APIs.

Scraping Public profiles are battle tested in court in HiQ VS LinkedIn case.
GDPR, CCPA, SOC2 compliant
High rate Limit - 300 requests/minute
Fast APIs respond in ~2s
Fresh data - 88% of data is scraped real-time, other 12% are not older than 29 days
High accuracy
Tons of data points returned per profile

Built for developers, by developers.

linkedin-profiles-scraping's People

Contributors

Stargazers

Watchers

linkedin-profiles-scraping's Issues

Unable to locate element: //*[@id="ember16"]/input

- Finish importing package ...
linkedin.py:18: DeprecationWarning: use options instead of firefox_options
  driver = webdriver.Firefox(firefox_options=webdriver.FirefoxOptions())
- Finish opening a driver ...
- Finish importing the login credentials ...
- Keying in the username ...
- Keying in the password ...
- Finish logging in ...
Traceback (most recent call last):
  File "linkedin.py", line 55, in <module>
    search_field = driver.find_element_by_xpath('//*[@id="ember16"]/input')
  File "/home/dactoan/py/scrapy-linkedin/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 394, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)
  File "/home/dactoan/py/scrapy-linkedin/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 976, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "/home/dactoan/py/scrapy-linkedin/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/dactoan/py/scrapy-linkedin/venv/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: //*[@id="ember16"]/input