Managing proxies for scaled data scraping and other automation operations will eventually require something like ProxyNest. ProxyNest is a proxy managment API that is well-suited for mid-scale and will soon be made for large ones.

• FastAPI
• Uvicorn
• MongoDB

• add/update/delete proxies and fetch by country tag or codes.
• reset all proxies if anything goes wrong.

• reserve proxies for instances.
• refresh the reservation on request.
• clear the instance reservation.

• periodic check for proxy uptime and latency.
• periodic check for country code of the IP.
• cleanup for instances that expired.

the default username is admin and password is password :p 

Make sure you have docker installed and a mongoDB hosted. let's start by building the docker image, get into the directory using any terminal and run the docker build command.

docker build -t proxynest  .

then we'll start it and pass the right variables to your mongoDB

docker run -d -p 8000:8000 --name proxynest  \
-e DB_URL='your_db_url' \
-e DB_NAME='your_db_name' \
-e DB_USER='your_db_user' \
-e DB_PASSWORD='your_db_password' \
-e PORT=8000 \ proxynest

you'll have it running in no time serving requests !

bash setenv.sh

 pip install -r requirements.txt 

python API.py

if __name__ == "__main__":
    uvicorn.run("API:app", host="0.0.0.0", port=8042)

Including the overall routes in here, you can read indepth about the documentation when you read the webserver, or the wiki once its setup.

• fix tag system with proxy reservation as its unused there for now.
• write an actual documentation of the project.
• build a docker image automatically and publish using Github actions.
• perfecounter usage for response time instead of what is being currently used.

• Integrated rotating proxy creation.
• Better authentication.
• Considerably better task queuing instead of using asyncio task queuing for very large scale operations.
• Custom host to measure latency.
• Multi user support.
• Write more documentation
• Stress test with k6
• more profiling

This project includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.