Git Product home page Git Product logo

mubeng's Introduction


mubeng

An incredibly fast proxy checker & IP rotator with ease.

ContributingWhat's newDocumentationReport Issues



Features

  • Proxy IP rotator: Rotates your IP address for every specific request.
  • Proxy checker: Check your proxy IP which is still alive.
  • All HTTP/S methods are supported.
  • HTTP, SOCKS v4(A) & v5 proxy protocols apply.
  • All parameters & URIs are passed.
  • Easy to use: You can just run it against your proxy file, and choose the action you want!
  • Cross-platform: whether you are Windows, Linux, Mac, or even Raspberry Pi, you can run it very well.

Why mubeng?

It's fairly simple, there is no need for additional configuration.

mubeng has 2 core functionality:

1. Run proxy server as proxy IP rotation

This is useful to avoid different kinds of IP ban, i.e. bruteforce protection, API rate-limiting or WAF blocking based on IP. We also leave it entirely up to user to use proxy pool resources from anywhere.

2. Perform proxy checks

So, you don't need any extra proxy checking tools out there if you want to check your proxy pool.

Installation

Binary

Simply, download a pre-built binary from releases page and run!

Docker

Pull the Docker image by running:

▶ docker pull ghcr.io/kitabisa/mubeng:latest

Source

Using Go (v1.19) compiler:

▶ GO111MODULE=on go get -u ktbs.dev/mubeng/cmd/mubeng
NOTE: The same command above also works for updating.

— or

Manual building executable from source code:

▶ git clone https://github.com/kitabisa/mubeng
▶ cd mubeng
▶ make build
▶ (sudo) mv ./bin/mubeng /usr/local/bin
▶ make clean

Usage

For usage, it's always required to provide your proxy list, whether it is used to check or as a proxy pool for your proxy IP rotation.

kitabisa mubeng

Basic

▶ mubeng [-c|-a :8080] -f file.txt [options...]

Options

Here are all the options it supports.

▶ mubeng -h
Flag Description
-f, --file <FILE> Proxy file.
-a, --address <ADDR>:<PORT> Run proxy server.
-A, --auth <USER>:<PASS> Set authorization for proxy server.
-d, --daemon Daemonize proxy server.
-c, --check To perform proxy live check.
-g, --goroutine <N> Max. goroutine to use (default: 10).
--only-cc <AA>,<BB> Only show specific country code (comma separated).
-t, --timeout Max. time allowed for proxy server/check (default: 30s).
-r, --rotate <AFTER> Rotate proxy IP for every AFTER request (default: 1).
-m, --method <METHOD> Rotation method (sequent/random) (default: sequent).
-s, --sync Sync will wait for the previous request to complete.
-v, --verbose Dump HTTP request/responses or show died proxy on check.
-o, --output <FILE> Save output from proxy server or live check.
-u, --update Update mubeng to the latest stable version.
-w, --watch Watch proxy file, live-reload from changes.
-V, --version Show current mubeng version.

NOTES:

  • Rotations are counted for all requests, even if the request fails.
  • The proxy server runs asynchronously by default, so it doesn't guarantee that your requests after N (which is N+1 and so on) will rotate the proxy IP, instead use the -s/--sync flag to wait for requests to the previous proxy to complete.
  • Daemon mode (-d/--daemon) will install mubeng as a service on the (Linux/OSX) system/setting up callback (Windows).
    • Hence you can control service with journalctl, service or net (for Windows) command to start/stop proxy server.
    • Whenever you activate the daemon mode, it works by forcibly stop and uninstalling the existing mubeng service, then re-install and starting it up in daemon.
  • Verbose mode (-v/--verbose) and timeout (-t/--timeout) apply to both proxy check and proxy IP rotation actions.
  • HTTP traffic requests and responses is displayed when verbose mode (-v/--verbose) is enabled, but
    • We DO NOT explicitly display the request/response body, and
    • All cookie values in headers will be redacted automatically.
  • If you use output option (-o/--output) to run proxy IP rotator, request/response headers are NOT written to the log file.
  • A timeout option (-t/--timeout) value is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "5s", "300ms", "-1.5h" or "2h45m".
    • Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", and "h".

Install SSL Certificate

mubeng uses built-in certificate authority by GoProxy. With mubeng proxy server running, the generated certificate can be exported by visiting http://mubeng/cert in a browser.

Installation steps for CA certificate is similar to other proxy tools.

Examples

For example, you've proxy pool (proxies.txt) as:

http://127.0.0.1:8080
https://127.0.0.1:443
socks4://127.0.0.1:4145
socks5://127.0.0.1:2121
...
...

Because we use auto-switch transport, mubeng can accept multiple proxy protocol schemes at once.
Please refer to documentation for this package.

Proxy checker

Pass --check flag in command to perform proxy checks:

▶ mubeng -f proxies.txt --check --only-cc AU,US,UK --output live.txt

The above case also uses --output flag to save a live proxy of specific country code with --only-cc flag (ISO-3166 alpha-2) into file (live.txt) from checking result.

(Figure: Checking proxies mubeng with max. 5s timeout)

Proxy IP rotator

Furthermore, if you wish to do proxy IP rotator from proxies that are still alive earlier from the results of checking (live.txt) (or if you have your own list), you must use -a (--address) flag instead to run proxy server:

▶ mubeng -a localhost:8089 -f live.txt -r 10 -m random

The -r (--rotate) flag works to rotate your IP for every N request value you provide (10), and the -m (--method) flag will rotate the proxy sequential/randomly.

(Figure: Running mubeng as proxy IP rotator with verbose mode)

Burp Suite Upstream Proxy

In case you want to use mubeng (proxy IP rotator) as an upstream proxy in Burp Suite, acting in-between Burp Suite and mubeng to the internet, so you don't need any additional extensions in Burp Suite for that. To demonstrate this:

(Figure: Settings Burp Suite Upstream Proxy to mubeng)

In your Burp Suite instance, select Project options menu, and click Connections tab. In the Upstream Proxy Servers section, check Override user options then press Add button to add your upstream proxy rule. After that, fill required columns (Destination host, Proxy host & Proxy port) with correct details. Click OK to save settings.

OWASP ZAP Proxy Chain

It acts the same way when you using an upstream proxy. OWASP ZAP allows you to connect to another proxy for outgoing connections in OWASP ZAP session. To chain it with a mubeng proxy server:

(Figure: Settings proxy chain connection in OWASP ZAP to mubeng)

Select Tools in the menu bar in your ZAP session window, then select the Options (shortcut: Ctrl+Alt+O) submenu, and go to Connection section. In that window, scroll to Use proxy chain part then check Use an outgoing proxy server. After that, fill required columns (Address/Domain Name & Port) with correct details. Click OK to save settings.

Proxy format

Currently mubeng supports HTTP(S) & SOCKSv4(A)/v5 protocol, see examples above. But, not limited by that we also support proxy string substitution and helper functions for your proxy pool.

Templating

If you have an authenticated proxy, you definitely don't want to write credentials constantly to the proxy pool file. mubeng can evaluate environment variable with {{VARIABLE}} writing style.

For example:

  1. String substitute
$ export USERNAME="FOO"
$ export PASSWORD="BAR"
$ echo "http://{{USERNAME}}:{{PASSWORD}}@192.168.0.1:31337" > list.txt
$ mubeng -f list.txt -a :8080
  1. Helper function

Available functions currently supported:

  • uint32, and
  • uint32n N.

Those following above functions are thread-safe pseudo-randomness.

As an example of its use, we will be utilizing stream isolation over Tor SOCKS. With this method, you just need one Tor instance and each request can use a different stream with a different exit node, but that doesn't guarantee that your ass will be rotated. Thus, we have to create unique USER:PASS pair to isolate streams for every connection. In order to pass pseudo-random proxy authorization, use uint32 or uint32n function on your proxy pool, like:

$ echo "socks5://{{uint32}}:{{uint32}}@127.0.0.1:9050" > list.txt
$ while :; do mubeng -f list.txt -c 2>/dev/null; done
[LIVE] [XX] [23.**.177.2] socks5://2123347975:[email protected]:9050
[LIVE] [XX] [199.**.253.156] socks5://1646373938:[email protected]:9050
[LIVE] [XX] [185.**.101.137] socks5://814036283:[email protected]:9050
[LIVE] [XX] [185.**.83.83] socks5://2895805939:[email protected]:9050
[LIVE] [XX] [103.**.167.10] socks5://408584795:[email protected]:9050
[LIVE] [XX] [198.**.84.99] socks5://3015151335:[email protected]:9050
[LIVE] [XX] [179.**.159.197] socks5://3952852758:[email protected]:9050
^C

Limitations

Currently IP rotation runs the proxy server only as an HTTP protocol, not a SOCKSv4(A)/v5 protocol, even though the resource you have is SOCKSv4(A)/v5. In other words, the SOCKSv4(A)/v5 resource that you provide is used properly because it uses auto-switch transport on the client, but this proxy server DOES NOT switch to anything other than HTTP protocol.

Contributors

contributions

This project exists thanks to all the people who contribute. To learn how to setup a development environment and for contribution guidelines, see CONTRIBUTING.md.

Pronunciation

jv_ID/mo͞oˌbēNG/ — mubeng-mubeng nganti mumet. (ꦩꦸꦧꦺꦁ​ꦔꦤ꧀ꦠꦶ​ꦩꦸꦩꦺꦠ꧀)

Changes

For changes, see CHANGELOG.md.

License

This program is developed and maintained by members of Kitabisa Security Team, and this is not an officially supported Kitabisa product. This program is free software: you can redistribute it and/or modify it under the terms of the Apache license. Kitabisa mubeng and any contributions are copyright © by Dwi Siswanto 2021-2022.

mubeng's People

Contributors

cnbeining avatar dependabot[bot] avatar dwisiswant0 avatar github-actions[bot] avatar hz61p1 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mubeng's Issues

[FEATURE] Add another option for IP checking

Is your feature request related to a problem? Please describe.
Yes, api.myip is not a 100% stable solution, as the server sometimes goes down.
An example would be yesterday

Describe the solution you'd like
Add an option for another IP api incase of api.myip going down

Describe alternatives you've considered
I've forked this project and added a -h / --httpbin flag for using http://httpbin.org/ip and it works minus the cc feature

Additional context
Would love to create a pull request but viewed the requirements being that an issue is opened first

[BUG] Docker Unable to find proxy file

i installed the package via docker pull.

Then run command sudo docker run kitabisa/mubeng [-c|-a :8080] -f file.txt Error! no proxy file provided. the file is in the home directory tried ./file.txt and /home/ubuntu/file.txt same error. I think ignore -f flag are ignored.

then i installed via cloning the reprosotory
Everything works but i can run it as daemon get error: exit status 1

Pls help and thank you for sharing your work.

One last question, is there a way to log the response of request e.g. to log the requested address and response code 200,403,500

[FEATURE] Use proxy configuration from environment

Doesn't use proxy configuration from /etc/environment on machine with local interface only. All web proxy showing dead during check.

  • OS: Linux
  • OS version: Ubuntu 21
  • mubeng Version: 0.9.2

Tried to fix this, but I'm zero in Go.

[FEATURE] Support VMess

Please support VMess

BASE64

vmess://eyJhZGQiOiI1Ljc1LjEzNS4xMTAiLCJhaWQiOiIwIiwiYWxwbiI6IiIsImhvc3QiOiIiLCJpZCI6IjZiN2NiYzQxLWZiODAtNDFhOC04ZWI2LWI1YmRiMGU4NjE0NyIsIm5ldCI6InRjcCIsInBhdGgiOiIiLCJwb3J0IjoiNDQzIiwicHMiOiIoUzEyMynwn4ep8J+HqnQubWUvUHJpdmF0ZVZQTnMiLCJzY3kiOiJhdXRvIiwic25pIjoiIiwidGxzIjoiIiwidHlwZSI6Imh0dHAiLCJ2IjoiMiJ9

Decode

Minified JSON

{"add":"5.75.135.110","aid":"0","alpn":"","host":"","id":"6b7cbc41-fb80-41a8-8eb6-b5bdb0e86147","net":"tcp","path":"","port":"443","ps":"(S123)🇩🇪t.me/PrivateVPNs","scy":"auto","sni":"","tls":"","type":"http","v":"2"}

Unminified JSON

{
  "add": "5.75.135.110",
  "aid": "0",
  "alpn": "",
  "host": "",
  "id": "6b7cbc41-fb80-41a8-8eb6-b5bdb0e86147",
  "net": "tcp",
  "path": "",
  "port": "443",
  "ps": "(S123)🇩🇪t.me/PrivateVPNs",
  "scy": "auto",
  "sni": "",
  "tls": "",
  "type": "http",
  "v": "2"
}

[FEATURE] output only certain checked countries

Is your feature request related to a problem? Please describe.
i wonder if it's possible to have a feature to output only proxy servers on certain counties like fr de etc

Describe the solution you'd like
--country flag

Describe alternatives you've considered
none

[BUG] Unable to use with own list

Describe the bug

using mubeng gives error. [FTL] Error! open /home/x/proxies.txt: has no valid proxy URLs.

To Reproduce

run mubeng with mubeng -f /home/x/proxies.txt --check -v

Expected behavior

To check or to rotate with my list

image

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: Kali Linux on WSL
  • OS version: Linux DESKTOP-E97AU3J 4.4.0-19041-Microsoft #488-Microsoft Mon Sep 01 13:43:00 PST 2020 x86_64 GNU/Linux
  • mubeng Version: mubeng version v0.4.5

Additional context
My list is done like this:

https://x:1080
https://x:1080
https://x:1080
https://x:1080
https://x:1080

image

[BUG] remote error: tls: unknown certificate

Describe the bug

Can't open HTTPS pages over proxy.
remote error: tls: unknown certificate

To Reproduce

Start mubeng proxy and see

Environment

  • OS: Ubuntu Linux 20.04
  • Latest mubeng version

[BUG] tls: bad record MAC

Describe the bug

Some wierd things are happening when configuring an https proxy in the proxylist.

To Reproduce

Steps to reproduce the behavior:

> mubeng -f proxylist.txt -a localhost:8089

Proxylist contains one row:
https://"replace with ip":3129

This ip runs a Squid proxy with this configuration:
https_port 3129 cert=/etc/squid/certs/squid-ca-cert-key.pem

Expected behavior

Trying to run a curl against mubeng gives the following error:
curl --proxy "localhost:8089" https://api.myip.com

2023/11/06 14:42:19 [001] WARN: Cannot handshake client api.myip.com:443 local error: tls: bad record MAC

Running the same curl but pointing directly to the Squid proxy gives no errors:
curl --proxy-insecure --proxy "https://"replace with ip":3129" https://api.myip.com
{"ip":"","country":","":""}

Screenshots

image

Environment (please complete the following information):

  • OS: [e.g. mac, linux]: Mubeng on Windows, Squid on Arch Linux
  • OS version: [uname -a] Windows 11
  • mubeng Version: [mubeng -V] v0.14.1

Additional context
I am trying to get my head around how to configure secure https proxy servers that works with mubeng. E.g

Requesting client <--HTTP (or HTTPS--> Mubeng <-- HTTPS ONLY --> Squid(or any other proxy)

[ERRO] malformed HTTP response "\x00[\x00\x00\x00\x00\x00\x00"

[ERROR] 192.168.50.10:50421 Post "https://play.google.com:443/log?format=json&hasfast=true&authuser=0": malformed HTTP response "\x00[\x00\x00\x00\x00\x00\x00"

Steps to reproduce the behavior:

> ./mubeng -f p.txt -a :2828```


- OS: linux
- OS version: Linux services 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux
- mubeng Version: mubeng version v0.14.0

Error : 
imported cert to local/user pc ... done 

malformed HTTP response "\x00[\x00\x00\x00\x00\x00\x00"

[FEATURE] HTTPS SUPPORT?

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[BUG] Response errors

I have two response errors
context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Second error:

goroutine 389 [running]:
net/http.(*conn).serve.func1(0xc000153040)
        /usr/lib/go-1.15/src/net/http/server.go:1801 +0x147
panic(0x94f400, 0xdb46d0)
        /usr/lib/go-1.15/src/runtime/panic.go:975 +0x47a
ktbs.dev/mubeng/internal/server.(*Proxy).onRequest(0xc000199f90, 0xc00018d600, 0xc000405980, 0x0, 0x0)
        /home/misho/mubeng/internal/server/handler.go:83 +0x34a
github.com/elazarl/goproxy.FuncReqHandler.Handle(0xc000199fa0, 0xc00018d600, 0xc000405980, 0x10, 0x10)
        /home/misho/go/pkg/mod/github.com/elazarl/[email protected]/actions.go:19 +0x3a
github.com/elazarl/goproxy.(*ReqProxyConds).Do.func1(0xc00018d600, 0xc000405980, 0xc000045400, 0x0)
        /home/misho/go/pkg/mod/github.com/elazarl/[email protected]/dispatcher.go:216 +0xcd
github.com/elazarl/goproxy.FuncReqHandler.Handle(0xc0001a3780, 0xc00018d600, 0xc000405980, 0x40cbbb, 0x10)
        /home/misho/go/pkg/mod/github.com/elazarl/[email protected]/actions.go:19 +0x3a
github.com/elazarl/goproxy.(*ProxyHttpServer).filterRequest(0xc000152320, 0xc00018d600, 0xc000405980, 0xc0003afb90, 0x4)
        /home/misho/go/pkg/mod/github.com/elazarl/[email protected]/proxy.go:62 +0x74
github.com/elazarl/goproxy.(*ProxyHttpServer).ServeHTTP(0xc000152320, 0xaa5f60, 0xc00016a700, 0xc00018d600)
        /home/misho/go/pkg/mod/github.com/elazarl/[email protected]/proxy.go:124 +0x305
net/http.serverHandler.ServeHTTP(0xc00009a540, 0xaa5f60, 0xc00016a700, 0xc00018d600)
        /usr/lib/go-1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc000153040, 0xaa70e0, 0xc000063880)
        /usr/lib/go-1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /usr/lib/go-1.15/src/net/http/server.go:2969 +0x36c

I'm trying using this with colly but not working.
Thx

[BUG]not use https

proxies.txt as:
https://163.204.92.67:9999
https://42.7.7.38:9999
......

exe error

transport.go code :
switch proxyURL.Scheme {
case "socks5":
tr = &http.Transport{
Dial: dialer.Dial,
}
case "http":
tr = &http.Transport{
Proxy: http.ProxyURL(proxyURL),
}
default:
return nil, fmt.Errorf("unsupported proxy protocol scheme: %s", proxyURL.Scheme)
}

what is ‘https’?

[FEATURE]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Yes. It say I need say like http://ip:port and I have one list of 65k proxies and idk what proxy is http, socks5...

Describe the solution you'd like
A clear and concise description of what you want to happen.
Something like the program try to find the type
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
idk
Additional context
Add any other context or screenshots about the feature request here.

[FEATURE] Rotation methods

Is your feature request related to a problem? Please describe.
Gives the user the option to rotate by incremental or random method

Describe the solution you'd like
Something like --method=1/2.

Describe alternatives you've considered
If -r/--rotate is set to 0, then randomize.

Additional context
TBD

[BUG] Response Terminated Early

Describe the bug

Hi,

I'm currently testing mubeng with a proxy I know is up and running. Here is the output when I use it directly with curl without using mubeng:

# curl -x http://128.199.67.35:80 http://httpbin.org/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.81.0",
    "X-Amzn-Trace-Id": "Root=1-63da6436-16d97d6d19c7a9dd6665c4f5",
    "X-Proxy-Id": "714674714"
  },
  "origin": "A.B.C.D, 128.199.67.35",
  "url": "http://httpbin.org/get"
}
# curl -k -x http://128.199.67.35:80 https://httpbin.org/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.81.0",
    "X-Amzn-Trace-Id": "Root=1-63da6447-7f55d16424732d7b59533c1d"
  },
  "origin": "128.199.67.35",
  "url": "https://httpbin.org/get"
}

so I know it's up and running

now if I save it to proxies.txt:

http://128.199.67.35:80

and try to use it with mubeng, it fails for both connections to http and https sites:

# curl -x http://localhost:8080 http://httpbin.org/get
curl: (18) transfer closed with 349 bytes remaining to read

# curl -k -x http://localhost:8080 https://httpbin.org/get
curl: (18) transfer closed with outstanding read data remaining

and here is what mubeng shows when I try it with an https site:

# mubeng -a localhost:8080 -f /tmp/proxies.txt -r 10 -v

           _   v0.13.0
 _____ _ _| |_ ___ ___ ___
|     | | | . | -_|   | . |
|_|_|_|___|___|___|_|_|_  |
                      |___|
 [email protected]

2023/02/01 13:12:32 [INFO] ▶ [PID: 1245594] Starting proxy server on localhost:8080
2023/02/01 13:12:38 [DEBU] ▶ 127.0.0.1:40648 GET https://httpbin.org:443/get
* Request to https://httpbin.org:443/get
* Request from 127.0.0.1:40648
* Skipping TLS verification: connection is susceptible to man-in-the-middle attacks.
> GET /get HTTP/1.1
> Host: httpbin.org
> Accept: */*
> User-Agent: curl/7.81.0
> X-Forwarded-For: 128.199.67.35
> X-Forwarded-Proto: https

< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Length: 302
< Content-Type: application/json
< Date: Wed, 01 Feb 2023 13:12:52 GMT
< Server: gunicorn/19.9.0

2023/02/01 13:12:52 [DEBU] ▶ 127.0.0.1:40648 200 OK
2023/02/01 13:12:52 [002] WARN: Cannot write TLS response body from mitm'd client: http: read on closed response body

and here is what mubeng shows when I try it with an http site:

2023/02/01 13:13:33 [DEBU] ▶ 127.0.0.1:60014 GET http://httpbin.org/get
* Request to http://httpbin.org/get
* Request from 127.0.0.1:60014
* Skipping TLS verification: connection is susceptible to man-in-the-middle attacks.
> GET /get HTTP/1.1
> Host: httpbin.org
> Accept: */*
> User-Agent: curl/7.81.0
> X-Forwarded-For: 128.199.67.35
> X-Forwarded-Proto: http

< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Length: 349
< Content-Type: application/json
< Date: Wed, 01 Feb 2023 13:13:34 GMT
< Server: gunicorn/19.9.0

2023/02/01 13:13:35 [DEBU] ▶ 127.0.0.1:60014 200 OK

From the output above + the curl output, it looks like the connection is successful but mubeng terminates the connection before the response has been fully read (it just reads the headers and that's it)

I'm using the latest binary download:

# mubeng -V
mubeng version v0.13.0

and this is on Ubuntu 22.04 with kernel 5.15

Any ideas? Thanks

PS. This looks very similar to https://github.com/kitabisa/mubeng/issues/175

[FEATURE] Add auth in proxy server in the flag -a

Is your feature request related to a problem? Please describe.
possibility of creating a server with the flag -a that has basic authentication

Describe the solution you'd like
add the possibility of reading a configuration file the argument where are the credentials and/or the host/port to use

Additional context
This is useful when you want to use a server as an IP rotator and share it with your trusted colleagues.

http://mubeng/cert DONT WORK PLS HELP

2022/09/25 21:28:32 [DEBU] ▶ 170.187.203.113:49936 GET http://mubeng/cert
2022/09/25 21:28:33 [DEBU] ▶ 170.187.203.113:49936 500 Internal Server Error
2022/09/25 21:28:33 [DEBU] ▶ 170.187.203.113:49936 GET http://mubeng/favicon.ico
2022/09/25 21:28:33 [DEBU] ▶ 170.187.203.113:49936 503 Service Unavailable
2022/09/25 21:28:34 [DEBU] ▶ 170.187.203.113:49936 GET http://mubeng/cert
2022/09/25 21:28:34 [DEBU] ▶ 170.187.203.113:49936 500 Internal Server Error
2022/09/25 21:28:34 [DEBU] ▶ 170.187.203.113:49936 GET http://mubeng/favicon.ico
2022/09/25 21:28:34 [DEBU] ▶ 170.187.203.113:49936 503 Service Unavailable

[FEATURE] rotated proxy ip on verbose log

Is your feature request related to a problem? Please describe.
trying to use mubeng to connect to one page i needed few attempts to connect to target server, even the list of proxies was updated by one script on mubeng startup.

Describe the solution you'd like
maybe a good idea can be show what's the proxy used of the txt to allow manually remove it.

Describe alternatives you've considered
more strict checks of proxy "alive" status.

Additional context

* unexpected EOF
2022/02/14 16:13:19 [ERRO] ▶ 127.0.0.1:37108 Get "https://www.meneame.net:443/": unexpected EOF
2022/02/14 16:13:25 [DEBU] ▶ 127.0.0.1:37112 GET https://www.meneame.net:443/
* Request to https://www.meneame.net:443/
* Request from 127.0.0.1:37112
* Skipping TLS verification: connection is susceptible to man-in-the-middle attacks.
> GET / HTTP/1.1
> Host: www.meneame.net
> Accept: */*
> Accept-Language: en
> User-Agent: ELinks/0.13.2 (textmode; Linux 5.10.0-8-amd64 x86_64; 258x56-2)
> X-Forwarded-For: 51.159.24.172
> X-Forwarded-Proto: https

< HTTP/1.1 200 OK
< Cache-Control: max-age=0, must-revalidate, private
< Content-Type: text/html; charset=UTF-8
< Date: Mon, 14 Feb 2022 15:13:30 GMT
< Expires: Mon, 14 Feb 2022 15:13:30 GMT
< Pragma: no-cache
< Server: nginx
< Set-Cookie: AWSALBTG=████████████████████; Expires=Mon, 21 Feb 2022 15:13:30 GMT; Path=/
< Set-Cookie: AWSALBTGCORS=████████████████████; Expires=Mon, 21 Feb 2022 15:13:30 GMT; Path=/; SameSite=None; Secure
< Set-Cookie: AWSALB=████████████████████; Expires=Mon, 21 Feb 2022 15:13:30 GMT; Path=/
< Set-Cookie: AWSALBCORS=████████████████████; Expires=Mon, 21 Feb 2022 15:13:30 GMT; Path=/; SameSite=None; Secure
< Set-Cookie: PHPSESSID=████████████████████; path=/; secure; httponly; samesite=lax
< Vary: Accept-Encoding
< X-Cache: MISS from 3158151495b1
< X-Cache-Lookup: MISS from 3158151495b1:3128

[BUG] cant install mubeng from github

Describe the bug

I cant install mubeng from github

To Reproduce

Steps to reproduce the behavior:

go install github.com/kitabisa/mubeng/cmd/mubeng@latest

image

Expected behavior

Install mubeng from github

Environment (please complete the following information):

  • OS: Ubuntu 20.04
  • OS version: Linux nanopineo3 5.10.63-rockchip64 aarch64 GNU/Linux

[FEATURE] IP Whitelist / Proxy USER:AUTH Protection

Is your feature request related to a problem? Please describe.
IP Whitelist / Proxy USER:AUTH Protection to prevent abuse

Describe the solution you'd like
IP Whitelist / Proxy USER:AUTH Protection to prevent abuse

Describe alternatives you've considered

Additional context

[FEATURE] Socks5 for IP Rotation?

Is running the proxy server as a SOCKS5 Protocol planned?

Thanks a lot for your work on this project, very useful tool but it would become 10 times more useful if we could run the proxy server as a SOCKS5

[FEATURE] socks4 support

Many public sources of proxies offer proxy with socks4 protocol, I am aware that it is already deprecated, but it is still used.

[FEATURE] Better alternative for "--rotate" in asynchronous access to mubeng

Is your feature request related to a problem? Please describe.
Yes. I have 2 identical threads, both run an infinite loop that call a function that sends 2 GET requests. I need those pairs of GET requests to have the same proxy from mubeng used on them. Pseudocode to visualize:

class Thread:
    loop:
        // I need mubeng to use proxy 'x' from proxy list for 1st request
        request.get("http://example.com/resource1", proxy=localhost:mubeng_port)
        // I need mubeng to AGAIN use proxy 'x' from proxy list for 2nd request
        request.get("http://example.com/resource2", proxy=localhost:mubeng_port) 

thread1 = Thread()
thread2 = Thread()

Describe the solution you'd like
My proposition for implementation:

  • pass to mubeng a number (let's call it "session id") in custom header in HTTP request
  • mubeng finds that custom header in "session id", and picks a proxy in proxies array by index calculated by session_id % proxies_array_length

Describe alternatives you've considered

  • --rotate <AFTER> option would not work in my case, because of asynchronous nature of threads
  • running multiple instances of mubeng - would not work well because:
    • resource heaviness while scaling for hundreds of threads
    • I need one central proxy to be used by every thread

[BUG] Not compatible with mac?

Describe the bug

According to the read me, this works with mac, however neither the ARM binary nor the docker seemed to work

To Reproduce

Steps to reproduce the behavior:

Try to use it on mac

Expected behavior

It works.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: Mac
  • OS version: 10.15.7
  • mubeng Version: Latest

Additional context
Add any other context about the problem here. Full output log is probably a helpful thing to add here.

[FEATURE] Remove error proxy from proxy pool

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[BUG] mubeng can't run as a Windows service

Describe the bug

Trying to add mubeng as a Windows service with -d will not work since the Windows service will never start.

To Reproduce

Steps to reproduce the behavior:

> mubeng_0.14.1_windows_386.exe -f proxylist.txt -a yourip:port -d

Expected behavior

Trying to start the service will not work doesn't matter if it's run as a specific user or localsystem.

Screenshots

image

Environment (please complete the following information):

  • OS: Windows
  • OS version: 11
  • mubeng Version: 0.14.1

Additional context
Add any other context about the problem here. Full output log is probably a helpful thing to add here.

[FEATURE] shadowsocks support

hi thanks for your great tool

is it possible to add support for the shadowsocks ( ss:// ) protocol?
I'm using surfshark shadowsocks proxy for connecting to the internet and I wonder if it is possible to add shadowsocks support so guys like me can use those ss:// links as proxy

thanks

[FEATURE] Custom DNS To Stop Leaks

Is your feature request related to a problem? Please describe.
If you are using a socks 5 proxy with hostnames then the hostname would be leaked.

Describe the solution you'd like
To stop this and also to have the option can you add support for custom DNS ideally over https or tls.

This could be done using kdig but, it would need knot-dnsutils installed.
Alternatively you could use dig.

kdig -d @8.8.8.8 +tls-ca +tls-host=dns.google.com example.com

dig @9.9.9.9 google.com +tls

Describe alternatives you've considered

Additional context

[FEATURE] Monitoring for mubeng

Is your feature request related to a problem? Please describe.
In a production environment, I use mubeng for web scraping. There is a need for robust monitoring to ensure the seamless operation of the scraping processes.

Describe the solution you'd like
I propose integrating Prometheus monitoring support into Mubeng to enable comprehensive tracking of key proxy usage and performance metrics. This would involve exposing relevant metrics such as success rates, response times, and error rates to Prometheus.

Describe alternatives you've considered
Alternatively, one could manually implement external monitoring scripts to collect and analyze performance metrics.

[FEATURE] Not show debug connections on console and multi-threading support

Is your feature request related to a problem? Please describe.
I do not like that everytime I launch the rotating proxy server, it just always fills everything up with the connections being made...
I also would like multi-threading support being added for VPS 4-64+ cores.

Describe the solution you'd like
Making an option to make the console be clear from all debugging stuff and adding multi-threading for better performance.

Describe alternatives you've considered
Nope.

Additional context
Nope.

Error getting https websites

Describe the bug

Trying to get https://example.org through mubeng

To Reproduce
Run mubeng, try to curl -x http://mubenghost:mubengport example.org

./mubeng -f ../proxies.txt -a localhost:8080 -c -v
2021/03/03 22:10:15 [INFO] ▶ [PID: 29823] Starting proxy server on localhost:8080
2021/03/03 22:10:35 [001] WARN: Cannot handshake client example.org:443 local error: tls: bad record MAC

Expected behavior

Body of example.org site

[FEATURE]

Is your feature request related to a problem? Please describe.
There is a problem with the certificate. The certificate is very bad.. it is not accepted by the websites

Describe the solution you'd like
Use without a security certificate

Describe alternatives you've considered
Use without a security certificate like this: https://github.com/DKingAlpha/RPS

Additional context
The program is very good.. but the certificate causes many problems

[FEATURE] Live-reload proxy pool for proxy IP rotator

first
Gotta admit it's a great piece of work❤️
There is a suggestion that I hope can be adopted.
System: win11
-f ip.txt(Specify the txt file)
It would be better if it was dynamic
I try to use python to extract IP and modify IP.txt content.
But the new txt data is not recognized after the program runs.

A small suggestion
May good luck be with you

[BUG] Unlimited high concurrency leads to false positives in checker.

Describe the bug

Unlimited high concurrency leads to false positives in checker.

To Reproduce

Use checker to validate proxy.txt.

The last proxy is to manually verify that it is available.
image

Run mubeng's checker:

> mubeng -f .\proxy.txt -c

Expected behavior

The last proxy should be shown as alive.

Screenshots

image

Environment (please complete the following information):

  • OS: [e.g. mac, linux]
  • OS version: [uname -a]
  • mubeng Version: [mubeng -V]

Additional context
I'm guessing that the problem might be caused by unlimited waitgroup.
Maybe should to limit the frequency of requests.

wg.Add(1)
go func(address string) {
defer wg.Done()
addr, err := check(address, opt.Timeout)
if len(opt.Countries) > 0 && !isMatchCC(opt.Countries, addr.CC) {
return
}
if err != nil {
if opt.Verbose {
fmt.Printf("[%s] %s\n", aurora.Red("DIED"), address)
}
} else {
fmt.Printf("[%s] [%s] [%s] %s\n", aurora.Green("LIVE"), aurora.Magenta(addr.CC), aurora.Cyan(addr.IP), address)
if opt.Output != "" {
fmt.Fprintf(opt.Result, "%s\n", address)
}
}
}(helper.EvalFunc(proxy))
}
wg.Wait()

[FEATURE] Retry on failed proxy request

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.