Comments (12)
Hey @iSarabjitDhiman Im trying to scrape data, but every 11-12 requests to
get_user_data()
andget_user_id()
the account gets blocked. I created 2 session ids based on the same account, with amax_requests
of 9, and only limit my results to 20 of them, and yet it gets blocked, any recomendations? thanks!insta = InstaGPy( use_mutiple_account=True, session_ids=session_ids_lst, max_requests=9)
Later on it gets only 20 results of followers info, but it gets blocked and the 11-12 request done to get all the user info such as email and phone number, dont understand why if followed your recomendation.
Yes, I understand what you mean. Even I myself face the same issue. I believe its instagram api limits.
Here is how I tackle this :
- I use minimum 3 session Ids. If I use 2, I get blocked after 10-11 requests as you mentioned. You can use more than 3 for better results.
- I shuffle the session after 7-11 requests. Set config.MAX_REQUESTS = 11 or 10 and config.MIN_REQUESTS = 7 or 6
- You can also try adding some sleep time between each request.
I will update the code soon if I get to know the instagram rate limits.
from instagpy.
@iSarabjitDhiman So if i want to inspect like 10k followers, 3 session ids is more than enough?
from instagpy.
@iSarabjitDhiman So if i want to inspect like 10k followers, 3 session ids is more than enough?
Yes, should be enough. I usually get away with 3-4 sessions. I easily get around 25-40k followers with 3 sessions. Make sure to modify the code to add 2-4 seconds sleep time after each request.
from instagpy.
@iSarabjitDhiman So if i want to inspect like 10k followers, 3 session ids is more than enough?
Yes, should be enough. I usually get away with 3-4 sessions. I easily get around 25-40k followers with 3 sessions. Make sure to modify the code to add 2-4 seconds sleep time after each request.
Should i modify your source code? or my code?
Also, is recommended to use proxies? or not neccesary?
from instagpy.
@iSarabjitDhiman So if i want to inspect like 10k followers, 3 session ids is more than enough?
Yes, should be enough. I usually get away with 3-4 sessions. I easily get around 25-40k followers with 3 sessions. Make sure to modify the code to add 2-4 seconds sleep time after each request.
Should i modify your source code? or my code?
The source code. I am planning to do it, but I am working on a project at the moment. I won't be able to do it until the next weekend.
Well you can also try without the sleep time, I remember there should be some sleep time in the shuffle_session function while using multiple sessions.
But if it still doesnt work for you. Add sleep time after each request.
To be honest, the code needs modification, I also forgot to include the cusor_endpoint on data return. If you can wait for like 4-5 days(or maybe less), thats great. Otherwise you gonna have to make some changes on your own.
from instagpy.
OK cool @iSarabjitDhiman I also will try changing user agents if possible, is there any specific norm to set user agents? in order to be more masked?
from instagpy.
Well, you can try using mobile app user agents. You may also want to update the other headers as well. The more you do is less. Instagram is way too smart while detecting these robots.
from instagpy.
ok @iSarabjitDhiman Just read the code, but dont know where to start modifying, will you provide an update soon?
from instagpy.
ok @iSarabjitDhiman Just read the code, but dont know where to start modifying, will you provide an update soon?
Yes I will as soon as I finish working on my current project.
from instagpy.
@iSarabjitDhiman ok, will be updated!
from instagpy.
@iSarabjitDhiman Im solving the code and if I find the solution, will submit a pull request, the problem is that i keep getting an error called: requests.exceptions.TooManyRedirects: Exceeded 30 redirects
. Do you have something I can use to solve the problem? as for my investigation, it has to be with the headers
EDIT Here is the reference that explains the 30 redirect error
https://stackoverflow.com/questions/42237672/python-toomanyredirects-exceeded-30-redirects
from instagpy.
Hey, I just updated the code.
Now you can use pagination to handle requests. So you can modify the number of requests to avoid getting blocked.
Here is the code example.
from instagpy.
Related Issues (3)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from instagpy.