Read the project
README.mdin other languages: English, 简体中文
Many organizations on the Internet regularly publish proxy server lists, such as TheSpeedX/PROXY-List repository, etc. The authors of these repositories regularly update the latest proxy server lists, and we thank them for their outstanding work. However, not all of these proxies are available. For example, some require a username and password, and some proxies will expire in a short time.
This project can help you quickly obtain public and genuinely available proxy server resources on the Internet, including HTTP proxies, HTTP tunnels, transparent HTTP proxies, Socks4 proxies, Socks4a proxies, and Socks5 proxies.
This project is divided into 4 parts, or four command-line tools, which are downloader, verifier, server, and exporter.
downloader: Download real-time updated raw proxy files from the repository and merge them;server: An HTTP server that helps theverifierdetermine if the proxy server is available. It will accept a random string parameter passed in by the verifier, perform a hash operation on the parameter, and return the result as an HTTP response to ensure that the proxy server is indeed accessed by theverifierand that there is no cache server on the link;verifier: Verify whether the proxy servers downloaded by thedownloaderare available. Specifically, the verifier will connect to a controlled HTTP server through the proxy, and judge whether the proxy is available by judging whether the server's response is valid;exporter: Read the log files of theverifierand export the list of available proxy servers from them.
It is easy to see that the downloader has no input (the download address list is built into the tool), the output of the downloader is the input of the verifier, the output of the verifier is the input of the exporter, and the output of the exporter is the list of available proxy servers filtered by this project for you.
go install github.com/WangYihang/Proxy-Verifier/cmd/downloader@latest
go install github.com/WangYihang/Proxy-Verifier/cmd/exporter@latest
go install github.com/WangYihang/Proxy-Verifier/cmd/server@latest
go install github.com/WangYihang/Proxy-Verifier/cmd/verifier@latestUsage:
main [OPTIONS]
Application Options:
-i, --input-file= The input file in yaml format (default: -)
-o, --output-file= The output file (default: -)
-n, --num-workers= Number of workers (default: 4)
-m, --max-retries= Maximum number of retries (default: 3)
Help Options:
-h, --help Show this help messageUsage:
main [OPTIONS]
Application Options:
-b, --bind-host= The host to bind (default: 127.0.0.1)
-p, --bind-port= The port to bind (default: 80)
-l, --log-filename= The filename to log to (default: gin.log)
-s, --secret= The secret used to verify the integrity of the proxy (default:
2d7c29dd-cecb-4454-a4ec-ae2734771a60)
Help Options:
-h, --help Show this help messageUsage:
main [OPTIONS]
Application Options:
-i, --input-file= The input file
-o, --output-file= The output file
-u, --url= The target URL to connect through the proxy, e.g., http://www.google.com,
smtp://mails.tsinghua.edu.cn
-t, --timeout= Timeout in seconds (default: 16)
-n, --num-workers= Number of workers (default: 256)
-m, --monitor-interval= Interval to output the current running state (in seconds) (default: 1)
-v, --verbose Show verbose debug information
-d, --measurement-id= The measurement ID used to seperate different measurements in logs
-s, --secret= The secret used to verify the integrity of the proxy (default:
2d7c29dd-cecb-4454-a4ec-ae2734771a60)
Help Options:
-h, --help Show this help messageUsage:
main [OPTIONS]
Application Options:
-i, --input-file= The input file
-o, --output-file= The output file
-r, --require-identical If provided, the frontend IP and backend IP are required to be identical
Help Options:
-h, --help Show this help message- Define output and input filenames as environment variables
TODAY=$(date -u '+%Y-%m-%d')
FREE_OPEN_PROXIES_FILEPATH="free-open-proxies-v${TODAY}.txt"
FREE_OPEN_PROXIES_LOG_FILEPATH="free-open-proxies-v${TODAY}.log"
AVAILABLE_FREE_OPEN_PROXIES_FILEPATH="available-free-open-proxies-v${TODAY}.txt"- First, download the proxy list using the
downloader.
$ # Download the default proxies source lists yaml file
$ wget https://raw.githubusercontent.com/WangYihang/Proxy-Verifier/main/sources.yaml
$ ./downloader --input-file sources.yaml --output-file ${FREE_OPEN_PROXIES_FILEPATH}- Next, start the
serveron a machine with a public IP (e.g., 1.2.3.4).
$ ./server -b 0.0.0.0 -p 80- Once again, start the
verifierto verify the availability of the proxies.
$ ./verifier \
--input-file ${FREE_OPEN_PROXIES_FILEPATH} \
--output-file ${FREE_OPEN_PROXIES_LOG_FILEPATH} \
--url http://1.2.3.4:80/ \
--num-workers 1024 \
--timeout 8- Finally, export the available proxy servers using the
exporter.
$ ./exporter \
--input-file ${FREE_OPEN_PROXIES_LOG_FILEPATH} \
--output-file ${AVAILABLE_FREE_OPEN_PROXIES_FILEPATH}
http://20.210.113.32:8123/
http://113.254.50.31:80/
...
http://113.252.10.120:8118/
http://121.132.95.7:80/Due to the proxy data sources of this project coming from other projects, it is not possible to completely obtain all the proxy server lists on the entire internet.
- Support HTTPS proxies
- Support transparent HTTPS proxies
- Support HTTPS tunnels
- Allow the downloader to read proxy list download links through a configuration file
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent