Git Product home page Git Product logo

Comments (12)

iSarabjitDhiman avatar iSarabjitDhiman commented on June 22, 2024

Hey @iSarabjitDhiman Im trying to scrape data, but every 11-12 requests to get_user_data() and get_user_id() the account gets blocked. I created 2 session ids based on the same account, with a max_requests of 9, and only limit my results to 20 of them, and yet it gets blocked, any recomendations? thanks!

insta = InstaGPy(
    use_mutiple_account=True,
    session_ids=session_ids_lst, 
    max_requests=9)

Later on it gets only 20 results of followers info, but it gets blocked and the 11-12 request done to get all the user info such as email and phone number, dont understand why if followed your recomendation.

Yes, I understand what you mean. Even I myself face the same issue. I believe its instagram api limits.
Here is how I tackle this :

  • I use minimum 3 session Ids. If I use 2, I get blocked after 10-11 requests as you mentioned. You can use more than 3 for better results.
  • I shuffle the session after 7-11 requests. Set config.MAX_REQUESTS = 11 or 10 and config.MIN_REQUESTS = 7 or 6
  • You can also try adding some sleep time between each request.

I will update the code soon if I get to know the instagram rate limits.

from instagpy.

AndresQuiVal avatar AndresQuiVal commented on June 22, 2024

@iSarabjitDhiman So if i want to inspect like 10k followers, 3 session ids is more than enough?

from instagpy.

iSarabjitDhiman avatar iSarabjitDhiman commented on June 22, 2024

@iSarabjitDhiman So if i want to inspect like 10k followers, 3 session ids is more than enough?

Yes, should be enough. I usually get away with 3-4 sessions. I easily get around 25-40k followers with 3 sessions. Make sure to modify the code to add 2-4 seconds sleep time after each request.

from instagpy.

AndresQuiVal avatar AndresQuiVal commented on June 22, 2024

@iSarabjitDhiman So if i want to inspect like 10k followers, 3 session ids is more than enough?

Yes, should be enough. I usually get away with 3-4 sessions. I easily get around 25-40k followers with 3 sessions. Make sure to modify the code to add 2-4 seconds sleep time after each request.

Should i modify your source code? or my code?

Also, is recommended to use proxies? or not neccesary?

from instagpy.

iSarabjitDhiman avatar iSarabjitDhiman commented on June 22, 2024

@iSarabjitDhiman So if i want to inspect like 10k followers, 3 session ids is more than enough?

Yes, should be enough. I usually get away with 3-4 sessions. I easily get around 25-40k followers with 3 sessions. Make sure to modify the code to add 2-4 seconds sleep time after each request.

Should i modify your source code? or my code?

The source code. I am planning to do it, but I am working on a project at the moment. I won't be able to do it until the next weekend.

Well you can also try without the sleep time, I remember there should be some sleep time in the shuffle_session function while using multiple sessions.
But if it still doesnt work for you. Add sleep time after each request.
To be honest, the code needs modification, I also forgot to include the cusor_endpoint on data return. If you can wait for like 4-5 days(or maybe less), thats great. Otherwise you gonna have to make some changes on your own.

from instagpy.

AndresQuiVal avatar AndresQuiVal commented on June 22, 2024

OK cool @iSarabjitDhiman I also will try changing user agents if possible, is there any specific norm to set user agents? in order to be more masked?

from instagpy.

iSarabjitDhiman avatar iSarabjitDhiman commented on June 22, 2024

Well, you can try using mobile app user agents. You may also want to update the other headers as well. The more you do is less. Instagram is way too smart while detecting these robots.

from instagpy.

AndresQuiVal avatar AndresQuiVal commented on June 22, 2024

ok @iSarabjitDhiman Just read the code, but dont know where to start modifying, will you provide an update soon?

from instagpy.

iSarabjitDhiman avatar iSarabjitDhiman commented on June 22, 2024

ok @iSarabjitDhiman Just read the code, but dont know where to start modifying, will you provide an update soon?

Yes I will as soon as I finish working on my current project.

from instagpy.

AndresQuiVal avatar AndresQuiVal commented on June 22, 2024

@iSarabjitDhiman ok, will be updated!

from instagpy.

AndresQuiVal avatar AndresQuiVal commented on June 22, 2024

@iSarabjitDhiman Im solving the code and if I find the solution, will submit a pull request, the problem is that i keep getting an error called: requests.exceptions.TooManyRedirects: Exceeded 30 redirects. Do you have something I can use to solve the problem? as for my investigation, it has to be with the headers

EDIT Here is the reference that explains the 30 redirect error
https://stackoverflow.com/questions/42237672/python-toomanyredirects-exceeded-30-redirects

from instagpy.

iSarabjitDhiman avatar iSarabjitDhiman commented on June 22, 2024

Hey, I just updated the code.
Now you can use pagination to handle requests. So you can modify the number of requests to avoid getting blocked.
Here is the code example.

from instagpy.

Related Issues (3)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.