Proxy support about suckit HOT 8 CLOSED

Nice catch, I can confirm it works with https_proxy and http_proxy.
It seems suckit makes https content retrieval and additional http requests for something else.
If I had to guess, there is some code that resolves URLs, which is responsible for the http requests. (I remember some unit test which try to resolve an invalid lwn.net URL and looks for a redirect.)

My public server IP got blocked from scraping a particular website, so I can tell it needs both kinds of proxies to circumvent the block.

For future readers:
For multithreading downloading via proxies to work, the constants MAX_EMPTY_RECEIVES and SLEEP_MILLIS may need to be adjusted upwards, otherwise all worker threads exit prematurely: They receive no work in the time interval because of the increased proxy latency.

from suckit.

Reqwest seems to support this. https://docs.rs/reqwest/latest/reqwest/struct.Proxy.html

We could read the environment variable before creating the Downloader

from suckit.

What I find strange, the environment variable is already read and processed somehow:

• the download takes significantly longer with a proxy
• bad proxy strings lead to failed downloads
• running strings on the binary shows that the strings http_proxy and HTTP_PROXY are contained

But, the downloaded page does not show the proxy ip for the ifconfig.me website.

from suckit.

My knowledge on proxy is quite limited but I agree with you, something is not right

from suckit.

From what I read here it should "just works"™

from suckit.

This works for me (with some warning and retry for the proxy connection):

https_proxy=147.135.134.57:9300 suckit https://ifconfig.me
https_proxy=147.135.134.57:9300 suckit http://ifconfig.me

The ip I get is not my real ip

Note that I'm using https_proxy and not http_proxy. If I used your http_proxy with a random https_proxy this is not working. I think that for some reason we are doing https request even when specifying http.

from suckit.

My public server IP got blocked from scraping a particular website

Typical SuckIT

Should we close this?

from suckit.

As far as I am concerned, yes.
Except you want to keep it to open for the multiple proxy feature. This was just an idea, nothing where I would contribute personally.

from suckit.