Add proxy options about instascrape HOT 6 OPEN

Hello! What is the error that requires the use of a proxy?

Unfortunately using proxies isn't a feature that is supported out of the box but you can probably brew up your own solution using requests builtin proxy support. instascrape can scrape data from HTML so you can use requests (with your needed proxies) to request data and then pass the response's HTML into an instascrape scraper to scrape the data.

from instascrape.

Yes, the point is to prepare "something" that you can add to request . I did it in other projects but the problem within instascrape is that I don't know how to acccess request to set my proxy credentials.

from instascrape.

@chris-greening Got some code working to scrape and validate proxies, but am not really sure how your library works. Willing to collaborate on this?

from instascrape.

@dtrillo We had to solve this problem as well. What I ended up doing was subclassing the Session object to pass in to Profile.scrape:

class Session(requests.Session):
    """Custom session subclass that allows passing through a timeout to requests.get"""
    def __init__(self, timeout=None, proxies=None):
        super().__init__()
        self.timeout = timeout
        self.proxies = proxies

    def get(self, url, **kwargs):
        kwargs.setdefault('allow_redirects', True)
        return self.request('GET', url, **kwargs, timeout=self.timeout, proxies=self.proxies)

and then using it this way:

with Session(timeout=20, proxies=self.proxies) as s:
    profile.scrape(session=s)

from instascrape.

The above is only necessary for us for authenticated proxies if you want to keep trust_env = True for the Session. Otherwise you can just create a session and add proxies to that without the need for the subclassing

from instascrape.

@chris-greening . I am trying to use proxies with InstaScrape to avoid blocking. I tried the solution of @nickhendo but can't get it to work. Any ideas?

from instascrape.