hi when i run httprobe tool the internet connection is lost why .

OS : Linux Mint 19

I sure, that all is good with Go and PATH, because another Go programs work as well.
Httprobe work too, but only when try httprobe -h.
If i try: type file|httprobe - Nothing happens.

Hey Tom! Thanks a lot for this handy tool, and thanks for @RiccardoAncarani for implementing the port templates idea.

A very little mistake that I've noticed while using the templates and thought it might be worth a correction in upcoming releases if any. The xlarge template includes all ports in the large template except port 8083. Yup, that's all 😄

Cheers! 🎉

Hi!

Sometimes, after using httprobe, I also want to pass the valid/found http servers through Burp to do further analysis. Something like:

for target in $(cat targets.txt);
do
    curl -x http://localhost:8080 $target
done

I think it may be great to have the option to do such within httprobe, but I was coding this functionality and I realized that I wasn't sure on what's the best way of doing such thing.

As I want the requests to the proxy to be sent only on confirmed HTTP servers, the "proxified request" should be triggered when isListening returns (therefore once you know there is an HTTP server on that port), but this will generate 2 requests per host (first the request performed by isListening, and then, a second one being "proxified").

The other option is to add a flag that will directly make the requests called through isListening "proxified", but in this case, you'll get requests with no response on the Burp proxy (hosts that are being probed regardless of the actual existence of the http server).

Is this something that you may want to add to httprobe or you prefer to leave such out of it?

Cheers!

It would be nice if httprobe accepts massscan or nmap output, no need to 'guess' for http ports :)

hi tom facing a golang issue here when running trying to reinstall httprobe i get this

internal/race

compile: version "go1.13.8" does not match go tool version "go1.13.7"

container/list

compile: version "go1.13.8" does not match go tool version "go1.13.7"

unicode/utf8

compile: version "go1.13.8" does not match go tool version "go1.13.7"

crypto/internal/subtle

compile: version "go1.13.8" does not match go tool version "go1.13.7"

runtime/internal/sys

compile: version "go1.13.8" does not match go tool version "go1.13.7"

crypto/subtle

compile: version "go1.13.8" does not match go tool version "go1.13.7"

vendor/golang.org/x/crypto/cryptobyte/asn1

compile: version "go1.13.8" does not match go tool version "go1.13.7"

unicode

compile: version "go1.13.8" does not match go tool version "go1.13.7"

internal/nettrace

compile: version "go1.13.8" does not match go tool version "go1.13.7"

math/bits

compile: version "go1.13.8" does not match go tool version "go1.13.7"

vendor/golang.org/x/crypto/internal/subtle

compile: version "go1.13.8" does not match go tool version "go1.13.7"

runtime/internal/atomic

compile: version "go1.13.8" does not match go tool version "go1.13.7"

internal/cpu

compile: version "go1.13.8" does not match go tool version "go1.13.7"

vendor/golang.org/x/crypto/curve25519

compile: version "go1.13.8" does not match go tool version "go1.13.7"

sync/atomic

compile: version "go1.13.8" does not match go tool version "go1.13.7"

runtime/cgo

compile: version "go1.13.8" does not match go tool version "go1.13.7"
was wondering if anyone has had this issue before

Help on how to install because even after using the command:
go install github.com/tomnomnom/httprobe@latest

The is no response from the httprobe in terminal returns:
httprobe: command not found

The installation instructions on read me only downloads the latest release which is v0.1.2 from July 2019.

Consider creating a new release or updating the readme to work with master branch:

go get -u github.com/tomnomnom/httprobe@master

Hi @tomnomnom It would be good if you also add functionality to get details like status code, content length, title etc

internal/unsafeheader

compile: version "go1.15.3" does not match go tool version "go1.15.2"

runtime/internal/sys

compile: version "go1.15.3" does not match go tool version "go1.15.2"

internal/race

compile: version "go1.15.3" does not match go tool version "go1.15.2"

runtime/internal/atomic

compile: version "go1.15.3" does not match go tool version "go1.15.2"

internal/cpu

compile: version "go1.15.3" does not match go tool version "go1.15.2"

sync/atomic

compile: version "go1.15.3" does not match go tool version "go1.15.2"

unicode/utf8

compile: version "go1.15.3" does not match go tool version "go1.15.2"

unicode

compile: version "go1.15.3" does not match go tool version "go1.15.2"

container/list

compile: version "go1.15.3" does not match go tool version "go1.15.2"

math/bits

compile: version "go1.15.3" does not match go tool version "go1.15.2"

crypto/subtle

compile: version "go1.15.3" does not match go tool version "go1.15.2"

crypto/internal/subtle

compile: version "go1.15.3" does not match go tool version "go1.15.2"

unicode/utf16

compile: version "go1.15.3" does not match go tool version "go1.15.2"

vendor/golang.org/x/crypto/cryptobyte/asn1

compile: version "go1.15.3" does not match go tool version "go1.15.2"

internal/nettrace

compile: version "go1.15.3" does not match go tool version "go1.15.2"

vendor/golang.org/x/crypto/internal/subtle

compile: version "go1.15.3" does not match go tool version "go1.15.2"

runtime/cgo

compile: version "go1.15.3" does not match go tool version "go1.15.2"

bash: httprobe: command not found

Hi @tomnomnom ,
if i am going to parse (piping) the output of httprobe to input in another tool and i wanna filter the output which would be something like.
http://example.com
https://example.com
http://example1.com
and i want to make something like if statement if the same target has both http and https output => filter only https in another list aka cat urls.txt | httprobe | filter function | tee filtered_output.txt
HOW YOU CAN DO IT IN BASH ?

when i load the anew its saying not found.

┌─[muneef@parrot]─[~/Desktop/karma_v2]
└──╼ $bash karma_v2 -h

[!] Error: Unable to find anew. Make sure it installed OR
- GO111MODULE=on go get -u github.com/tomnomnom/anew
┌─[✗]─[muneef@parrot]─[/Desktop/karma_v2]
└──╼ $go install -v github.com/tomnomnom/anew@master
┌─[muneef@parrot]─[/Desktop/karma_v2]
└──╼ $anew -h
bash: anew: command not found
┌─[✗]─[muneef@parrot]─[~/Desktop/karma_v2]
└──╼ $

Hi @tomnomnom,

Thanks for the amazing tool, It really helps in HTTP probing.

As mentioned in the title, can we please have an additional argument if a user wants to print out the hosts which didn't respond or were down during the probing? Would help out in domains that point to a STALE record which is terminated etc.

Tried reinstalling it to latest but still getting:

flag provided but not defined: -prefer-https
Usage of httprobe:
-c int
set the concurrency level (default 20)
-p value
add additional probe (proto:port)
-s skip the default probes (http:80 and https:443)
-t int
timeout (milliseconds) (default 10000)
-v output errors to stderr****

Tried to use -p:443 and -p:443 -s with same results

What license is used for this software?

Hi I'm trying to install httprobe with go get in windows sub system (ubuntu), it's not working. Can I use httprobe in windows subsystem?

Hello,

I have been using httprobe for a while and everything is working fine so I decided to add it to my script, when I run the script I get an httprobe command not found. error. Every other tool in the script ( amass, nmap etc) is working fine and I can still use httprobe perfectly from the terminal in any directory (including the directory the script is in).
Any thoughts?
Thank you.

# cat amass-output.txt | httprobe --prefer-https

flag provided but not defined: -prefer-https

Usage of httprobe:
  -c int
        set the concurrency level (default 20)
  -p value
        add additional probe (proto:port)
  -s    skip the default probes (http:80 and https:443)
  -t int
        timeout (milliseconds) (default 10000)
  -v    output errors to stderr

# go version
go version go1.15.6 linux/amd64
# uname -a
Linux kali 5.9.0-kali2-amd64 #1 SMP Debian 5.9.6-1kali1 (2020-11-11) x86_64 GNU/Linux

any idea?

It's handy that you've built in a list of common ports but users have to read the code to figure out how to use that feature!
For anyone reading this you can use it by doing cat list.txt | httprobe -c 50 -p xlarge

I tried with this command subdomain.txt & httprobe but nothing happens. Pls help

I re-install go both from official and with brew installer. But there is no difference.

After command : go get -u github.com/tomnomnom/httprobe execute nothing is change. There is no output in terminal also.

whenever i run probe on large list after 50-100 subdomaiins it starts giving me temporary failure in name resolution
any permanent fix for this

I'm clone httprobe but I can't install and use it for sudomy script ??

first of all thanks for this tool. this tool is great but when I used this tool I'm getting different results for the same subdomains list every time. please see this screenshot link then you will understand
https://drive.google.com/file/d/15ivIm_Dfqwstzb_K8zc2lH99_LWMZCWZ/view?usp=sharing
https://drive.google.com/file/d/1TjY2obMhrjkBl0sxqug6-ioCksFPm91N/view?usp=sharing

httprobe doesn't discover live valid HTTP services.

It doesn't output HTTP services which return

curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104

Consider following curl request

$ curl -kiv https://xxx.xxx.xxx.xxx:443
*   Trying xxx.xxx.xxx.xxx:443...
* Connected to xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) port 443 (#0)
...
...
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
...
...
> GET / HTTP/1.1
> Host: xxx.xxx.xxx.xxx
> User-Agent: curl/7.81.0
> Accept: */*
> 
* OpenSSL SSL_read: Connection reset by peer, errno 104
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104

This is live valid HTTP service and actual behaviour of F5 reverse proxies.

The F5 returns a 200 if a valid existing virtual host is passed in the request

$ curl -kiv https://xxx.xxx.xxx.xxx -H 'Host: int-account.xxx.com'
*   Trying xxx.xxx.xxx.xxx:443...
* Connected to xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) port 443 (#0)
...
...
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
...
...
> GET / HTTP/1.1
> Host: int-account.xxx.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Tue, 31 Oct 2023 08:14:41 GMT
Date: Tue, 31 Oct 2023 08:14:41 GMT
...
...
...

If I httprobe the xxx.xxx.xxx.xxx:443 the service it's not marked as a valid live HTTP service

$ echo 'xxx.xxx.xxx.xxx:443' | httprobe 
$

Hi.

I'm a "megg" tool lover.

In "Meg", can filter status.

Filtering is not possible with the "httprobes" tool.

Therefore, all redirection domains are represented as normal.

Adding filtering options looks good.

Thanks.

The README tells to install httprobe using the command:

go get -u github.com/tomnomnom/httprobe

However, giving that command on mac terminal with the latest Go version (go version go1.17 as of Aug 2021) gives an error saying:

go get: installing executables with 'go get' in module mode is deprecated.
	Use 'go install pkg@version' instead.
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.

Using the install command like go install github.com/tomnomnom/httprobe@master works. The README needs to be updated with this command.

domain names containing Uppercase letters are not scanned

If a website is using self signed certificate and has redirect from http to https, both will be marked as failed. Just to keep this in mind.

httprobe with options whitelist status code
should be 200 only or like
cat domains.txt | httprobe -status-code 200
OR
cat domains.txt | httprobe -status-code 200,403,404

Definition

Hello Tomnomnom

Thank you for the tool I wanted check port 80 and port 443 of some subdomain that has this form
ipaddress.*.*.organisation.com
example
1046170.ostk.bm2.prod.gq1.yahoo.com or 105.240.139.98.aprez.ostk.prod.bf1.yahoo.com
but apparently httprobe doesn't like this form

Regards

Kindly add version attribute in this amazing tool, so we can track versioning.

When installing using ▶ go install github.com/tomnomnom/httprobe@latest
it installs an older version without support for the flag --prefer-https

Hi,

This tool is quite amazing. However, it lacks the functionality to save the html responses. Since, you are making GET request to all target you must be receiving the response body as well along with response code. I don't know much about go but this is what happens with python requests. If this is so, adding the functionality to save responses should not be a big deal. Please add this functionality if feasible.

When i enter $ cat domains.txt | httprobe
it suddenly exits without showing any results

Hi,

i have a problem with the Httprobe because not download..

sorry for my english XD

order for install..

Apt install golang (working done)
now
go get -u github.com/tomnomnom/httprobe (not error message, but dont download anything..)

attached screenshots.

https://ibb.co/c19P9Gb

https://ibb.co/QQYXKfF

upload photos from ImgBB.

package bufio: unrecognized import path "bufio" (import path does not begin with hostname)
package crypto/tls: unrecognized import path "crypto/tls" (import path does not begin with hostname)
package flag: unrecognized import path "flag" (import path does not begin with hostname)
package fmt: unrecognized import path "fmt" (import path does not begin with hostname)
package net/http: unrecognized import path "net/http" (import path does not begin with hostname)
package os: unrecognized import path "os" (import path does not begin with hostname)
package strings: unrecognized import path "strings" (import path does not begin with hostname)
package sync: unrecognized import path "sync" (import path does not begin with hostname)
package time: unrecognized import path "time" (import path does not begin with hostname)

Hello tomnomnom!

I have a short question, does httprobe have a max value for concurrency?

i am running it against +200k subdomains and sometimes even more, and it is a little bit slow with -c 50 provided. Is there a max value for the -c flag? any other method to speed this up without breaking everything?

Have a good day.

We've already got the 'port templates' added in #1 which are cool

It'd be good to be able to support the following cases too:

Same protocol, multiple ports:

httprobe -p http:80,81,8000 -p https:8443,9001

No protocol; checks both HTTP and HTTPS:

httprobe -p 9001
httprobe -p 80,81,8443

Hi!
I don't know why but it returns wrong results sometimes:

$ echo "genrih.resizer.mail.ru" | httprobe
https://genrih.resizer.mail.ru
http://genrih.resizer.mail.ru

Hey Tom,
Thanks for this awesome tool.
But,sadly it ain't working while using proxy.
Just wanted to give you a heads up.
Got any work-around?
Thanks

hello. thanks for this project.. i have subdomain list.. It has 134 subdomain names. When I get the protocol output with httprobe, it gets 264... adds the same subdomain name to https and http 2 times.

example my subdomain list:

videos.staging-airtableblocks.com
staging.staging-airtableblocks.com

httprobe output:

http://videos.staging-airtableblocks.com
https://videos.staging-airtableblocks.com
http://staging.staging-airtableblocks.com
https://staging.staging-airtableblocks.com

first he needs to check the https protocol. if the https protocol is used, it should add the https protocol subdomain name to the output file. If a site is added to the output file with the https protocol, the http protocol should not be added... I'll be happy if you solve this problem quickly.

go version go1.6.2 linux/amd64

PATH is set as follows:

export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

~/go$ go get -u github.com/tomnomnom/httprobe
# github.com/tomnomnom/httprobe
src/github.com/tomnomnom/httprobe/main.go:57: unknown http.Transport field 'MaxIdleConns' in struct literal
src/github.com/tomnomnom/httprobe/main.go:58: unknown http.Transport field 'IdleConnTimeout' in struct literal
src/github.com/tomnomnom/httprobe/main.go:64: unknown http.Transport field 'DialContext' in struct literal
src/github.com/tomnomnom/httprobe/main.go:68: undefined: http.ErrUseLastResponse

i can't able to use httprobe.i think this issue is due to go but can't able to the fix.

cat abc.txt | httprobe --prefer-https -c 50

the above command is showing error

cat abc.txt | httprobe
#this command works fine though

I want the result to be either http or https and not both
and if https is available I dont want http