sensepost / gowitness Goto Github PK
View Code? Open in Web Editor NEWπ gowitness - a golang, web screenshot utility using Chrome Headless
License: GNU General Public License v3.0
π gowitness - a golang, web screenshot utility using Chrome Headless
License: GNU General Public License v3.0
I don't see a way to use this on a website that has a login page. Is there a way to provide login credentials to this utility?
Getting this even with single flag
gowitness single --url=https://example.com
ERRO[2018-11-15 12:53:15] Failed to query url error="[Get https://example.com/?x=us: read tcp 192.168.31.153:37728->x.x.x.x:443: i/o timeout]" url="https://example.com." INFO[2018-11-15 12:53:15] Complete run-time=5.903457728s
gowitness constantly gives the following error after updating to 1.1.0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x14093c6]
goroutine 5183 [running]:
github.com/sensepost/gowitness/utils.getPageTitle(0xc0001bee70, 0xc000ae4d98, 0x100e448)
/Users/leonjza/Documents/git/gowitness/utils/processor.go:124 +0xa6
github.com/sensepost/gowitness/utils.getPageTitle(0xc0001bee00, 0xc000ae4de0, 0x138c945)
/Users/leonjza/Documents/git/gowitness/utils/processor.go:128 +0x7a
github.com/sensepost/gowitness/utils.getPageTitle(0xc0001bed90, 0x0, 0x0)
/Users/leonjza/Documents/git/gowitness/utils/processor.go:128 +0x7a
github.com/sensepost/gowitness/utils.getPageTitle(0xc0001becb0, 0xc000c0d0a0, 0xc0001becb0)
/Users/leonjza/Documents/git/gowitness/utils/processor.go:128 +0x7a
github.com/sensepost/gowitness/utils.ProcessURL(0xc000121480, 0x1b62de0, 0x1b60bb0, 0x9)
/Users/leonjza/Documents/git/gowitness/utils/processor.go:47 +0x51c
github.com/sensepost/gowitness/cmd.glob..func1.1(0xc00021e500, 0xc000241ac0, 0xc0002446c0, 0xc000121480)
/Users/leonjza/Documents/git/gowitness/cmd/file.go:94 +0x8a
created by github.com/sensepost/gowitness/cmd.glob..func1
/Users/leonjza/Documents/git/gowitness/cmd/file.go:90 +0x5ec
In future versions, would it be possible for the HTTP Title to be extracted when the GET response is parsed? That would be awesome.
Thanks,
Rhynorater
Love the tool, but I'm running into an edge case that might only be solvable by a big hammer. Might be overkill for this project, but I wanted to propose just in case :-)
I'm trying to capture screenshots from go tool trace -http
. Problem is, these pages can take up to 15 seconds to properly load. I've tried using --timeout 15 --chrome-time-budget 15 --chrome-timeout 15
. None of those help - Chrome considered the page loaded, and AFAIK there is no method to simply say "please wait 10 seconds before you take the screenshot, even if the page reports it is loaded before then".
A google search leads me here, where a user has a similar issue. I looked into the two recommended solutions. Both internally utilize the Chrome DevTools Protocol, which allows you to connect to a headless instance and communicate with it. As would be expected, this allows a lot more power than passing flags, e.g. you wait for DOM events, or screenshot specific page elements , or otherwise engage with the page. The more developed project is https://github.com/puppeteer/puppeteer, which is a Google project for programmatically interacting with headless Chromium.
There is already a golang project that speaks the DevTools protocol - https://github.com/chromedp/chromedp. However, it seems focused on being a golang version of puppeteer - intended for programmatic usage. For example, here is their screenshot source code. This project is clean and simply focused on getting a screenshot.
Would you be open to PRs that attempt to utilize chromedp in some circumstances? Not quite sure the proper combination of flags to indicate when it should be used, that would need to be discussed. Perhaps some "wait X seconds after specified DOM event" approach?
Misc note: The API call for taking a screenshot using DevTools is in the 1.3 release candidate, whereas 1.2 is the current stable. Doubt it's an issue, as clearly puppeteer finds it OK to expose features from the RC protocol version, but thought it was worth mentioning regardless.
Hi! So when I supply list of urls which don't have the http:// or https:// . I'm getting this error Skipping Invalid URL
so does gowitness automatically prefix urls with http:// or https:// as mentioned in this issue #11 or do we have to supply an additional argument?
The command which I used: gowitness file -s ~/Desktop/urls.txt
This can be added with the -dump-dom
Chrome flag: https://developers.google.com/web/updates/2017/04/headless-chrome#printing_the_dom
gowitness should have the option of prepending http/s like eyewitness does.
Describe the solution you'd like
Add HTTP and SOCKS5 proxy support screenshot functionality
Describe alternatives you've considered
Related to the old issue #21
Additional context
Seems like chromedp has this built in support already, just need to stub in the options and test in chrome/chrome.go
I'm attempting to batch collect images from hundreds of websites. Using gowitness file
with threads seems to work fine. However for more control I'd like to use gowitness single
as it's part of a larger bash script. However running multiple gowitness single
commands or even many gowitness single
command consecutively and I'm get panic errors as shown below.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1357950]
goroutine 1 [running]:
github.com/tidwall/buntdb.(*Tx).lock(0xc0004f8440)
/Users/austinginder/go/src/github.com/tidwall/buntdb/buntdb.go:1088 +0x30
github.com/tidwall/buntdb.(*DB).Begin(0x0, 0xc000126701, 0x115e9c5, 0xc0000881e0, 0xc0001420c0)
/Users/austinginder/go/src/github.com/tidwall/buntdb/buntdb.go:1067 +0x61
github.com/tidwall/buntdb.(*DB).managed(0x0, 0x115c901, 0xc0001268b8, 0x0, 0x0)
/Users/austinginder/go/src/github.com/tidwall/buntdb/buntdb.go:939 +0x40
github.com/tidwall/buntdb.(*DB).Update(0x0, 0xc0001268b8, 0x1, 0x1)
/Users/austinginder/go/src/github.com/tidwall/buntdb/buntdb.go:984 +0x3a
github.com/sensepost/gowitness/storage.(*Storage).SetHTTPData(0x1929de8, 0xc00017e500)
/Users/austinginder/go/src/github.com/sensepost/gowitness/storage/storage.go:53 +0x456
github.com/sensepost/gowitness/utils.ProcessURL(0xc00017e480, 0x192c0e0, 0x1929de8, 0x3c)
/Users/austinginder/go/src/github.com/sensepost/gowitness/utils/processor.go:109 +0x1c99
github.com/sensepost/gowitness/cmd.glob..func5(0x19239c0, 0xc000088a20, 0x0, 0x6)
/Users/austinginder/go/src/github.com/sensepost/gowitness/cmd/single.go:36 +0x8c
github.com/spf13/cobra.(*Command).execute(0x19239c0, 0xc000088960, 0x6, 0x6, 0x19239c0, 0xc000088960)
/Users/austinginder/go/src/github.com/spf13/cobra/command.go:766 +0x2cc
github.com/spf13/cobra.(*Command).ExecuteC(0x1923500, 0xc000127f50, 0x14447ce, 0x1923500)
/Users/austinginder/go/src/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/spf13/cobra.(*Command).Execute(0x1923500, 0xc000092058, 0x0)
/Users/austinginder/go/src/github.com/spf13/cobra/command.go:800 +0x2b
github.com/sensepost/gowitness/cmd.Execute()
/Users/austinginder/go/src/github.com/sensepost/gowitness/cmd/root.go:93 +0x31
main.main()
/Users/austinginder/go/src/github.com/sensepost/gowitness/main.go:6 +0x20
If helpful here is the format of the single
command I'm using.
gowitness single --url="$url" --resolution="1200,750" --log-format=json --log-level=warn --timeout=60 --destination="/Users/austinginder/Desktop/Screenshots/"
Current Behavior:
Take a list of 20k IP:PORT and use gowitness. From what i've seen the report output doesnt care to categories HTTP response codes. Meaning there is just a hoshposh of service responses mixed within the result. This can make it difficult to analyze even fairly small networks.
Expected Behavior
The output should be categorized by server response (at the very least).
Is your feature request related to a problem? Please describe.
I'm miss some screenshots with chrome headless
Describe the solution you'd like
I would like if chromium is available for screenshotting the sites.
Describe alternatives you've considered
Chromium
Additional context
I hope it will be in actions ASAP.
I'm currently running some rather ugly regex to get a list of vhosts out of the subject alternative name (SAN) list in a host's certificate file via way of nmap (using their default plugin set, one of which is ssl-certs). I then feed these into gowitness for screenshotting.
I noticed gowitness acknowledges DNS names from this list, but instead of browsing each one, it simply groups them under one IP. It would be nice to be able to instruct gowitness to screenshot every vhost/dns name it encounters for a given host in an nmap file as they may very well be entirely different websites.
First off, great tool. Is there a way to scan multiple CIDR ranges at once? I tried separating by commas and spaces but seems to be a no go. I guess I can just write up a quick for loop... Thanks again.
such as https://www.csdn.net
Latest version with the same problem when hitting pages that are returning just a JSON string (for example).
(...)
ERRO[2020-01-25 12:57:55] Screenshot failed destination=https-REDACTED.com.png err="exit status 1" url="https://REDACTED/"
panic: runtime error: invalid memory address or nil pointer dereference
(...)
The page returned this when using curl:
> GET / HTTP/1.1
> Host: REDACTED.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Client-IP: REDACTED
< GatewayMessage: error
< WWW-Authenticate: Bearer
< X-Backside-Transport: FAIL FAIL
< Connection: close
<
* TLSv1.2 (IN), TLS alert, close notify (256):
* Closing connection 1
* TLSv1.2 (OUT), TLS alert, close notify (256):
{ "ServiceStatus":{ "StatusType":"Error", "ServiceCode":"", "ServiceName":"
(REDACTED)
"InnerException":"", "ServerName":"" }
Originally posted by @BBerastegui in #24 (comment)
It looks like gowitness
on macOS 10.12.6 (16G1212) thinks my installed version of Google Chrome is 58.0.3029.110
, even when using the path option to point it to version 64.0.3282.140
~ % gowitness single https://google.com --chrome-path="/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome"
FATA[2018-02-05 10:44:23] Chrome version is older than v60 chromeversion=58.0.3029.110
~ % /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version
Google Chrome 64.0.3282.140
Even stops me from running commands like version
~ % gowitness version
FATA[2018-02-05 10:46:38] Chrome version is older than v60 chromeversion=58.0.3029.110
ERRO[2020-05-12 16:43:39] Screenshot failed destination=https-www.google.com.png err="exit status 1" url="https://www.google.com/"
ERRO[2020-05-12 16:43:39] Chrome process reported an error taking screenshot error="exit status 1" url="https://www.google.com/"
it is giving me an error every time I do a screenshot, I have chrome and chromium latest version installed, anyone knows whats wrong?
I like to use firefox headless mode instead of chromium to taking screenshots. Firefox users can take screenshot using --screenshot
flag via firefox binary. Check this resource to get more information:
Take a screenshot from the command line with headless
Hello,
I try include the "--fullPage" flag for chrome, instead of resolution. But with no success.
This is very useful when you have pages with different lengths. Long product lists etc.
I would be very grateful for any help.
Best Regards
Askan
Hi, First of all great tool! π
I have recently created a subdomain enumeration tool in golang and wanted to integrate gowitness as a data analysis module to my tool. But there are no clear instructions on how to do so. Therefore, anyone wanting to integrate it would have to work it out themselves.
It would be awesome if there were some instructions detailing how to integrate it into your golang project.
Thanks.
Problem: User-Agent is always "Go-http-client/1.1"
Expected behavior: According to the binary's help documentation, the default UA is supposed to be "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36", though, testing shows it is "Go-http-client/1.1" without using the --user-agent flag. Setting the --user-agent flag to something different , also results the same - "Go-http-client/1.1"
I noticed this due to many of the screenshots saying something along the lines of "prove you're not a bot".
Please let me know how to fix this, or if it is my mistake. Also, thank you for the excellent tool!
Imagine we have an nmap output file that gave us which ports are opened and which ones are closed. gowitness instead of go only for the opens goes for every possible port therefore it does not make any distinction between open or closed ports.
The expected behaviour would be to only enumerate the open ports
gowitness version: Latest, downloaded yesterday
Nmap command:
nmap -iL FILE -p80,8000 -T4 -oX OUTPUT --max-retries=2
Gowitness command:
gowitness nmap --nmap-file OUTPUT --threads 20
Hi, I get this error message while running the script on a url file list.
FATA[2018-04-21 17:10:18] fork/exec /usr/bin/chromium: too many open files
Describe the bug
The proxy server -p and nmap port selector -p are the same flag. One of them will need to be changed, could just drop the shorthand nmap -p and leave --port, or assign a random letter to nmap's -p
To Reproduce
gowitness-2.1.0-linux-amd64 nmap -h
panic: unable to redefine 'p' shorthand in "nmap" flagset: it's already used for "port" flag
Version Information:
When I move a report/db/screenshots around it breaks screenshot viewing because the report screenshot urls are absolute path'ed. This means I end up having to sed the leading path out. Can we get an option to add relative paths for screenshots?
Is your feature request related to a problem? Please describe.
Lot's of WAF screenshots
Describe the solution you'd like
accept cookies to send
It would be superb if gowitness was able to attempt screenshots for a list of default ports, on both http and https if given a text file (or stdin?) full of ips or hostnames.
as it stands now, gowitness needs to be told what protocol (http or https) and what port, but with enough threads and a low timeout, making attempts to stuff that isnt open/up goes pretty fast, so having a list of like half a dozen default ports to try (80, 8080, 443, 4433, 8888, 81, 90 etc) and to try both http and https for each as "default functionality" would dramatically increase gowitnesses portability!
Hi @leonjza Its not an issue but a request to have a feature for importing url's based on particular status code via the report viewer. Like a feature to filter screenshots based on status code and then export the urls to a text file.
Also have an option to increase number of screenshot per page and next button on the top too.
Regards,
Timeout reached while waiting for screenshot to finish [31mdestination[0m=https-www.runoob.comechartsecharts-setup.html.png
...
command is gowitness single --url=....com
version:windows10
chrome-version:latest
Would it work?
Hi guys,
Hope u are all well !
I tried to get a screenshot without the cli.
package main
import (
//"fmt"
//"os"
"net/url"
"strconv"
"strings"
"time"
//homedir "github.com/mitchellh/go-homedir"
log "github.com/sirupsen/logrus"
//"github.com/spf13/cobra"
//"github.com/spf13/viper"
chrm "github.com/sensepost/gowitness/chrome"
"github.com/sensepost/gowitness/storage"
"github.com/sensepost/gowitness/utils"
)
func main() {
// Init Google Chrome
chrome = chrm.Chrome{
Resolution: resolution,
ChromeTimeout: chromeTimeout,
Path: chromePath,
UserAgent: userAgent,
}
chrome.Setup()
screenshotURL := "https://www.zataz.com/osint-des-numeros-de-politiques-francaises-interceptes-via-whatsapp/"
u, err := url.ParseRequestURI(screenshotURL)
if err != nil {
log.WithField("url", screenshotURL).Fatal("Invalid URL specified")
}
// Process this URL
utils.ProcessURL(u, &chrome, &db, 3000)
log.WithFields(log.Fields{"run-time": time.Since(startTime)}).Info("Complete")
}
var (
cfgFile string
chrome chrm.Chrome
db storage.Storage
dbLocation string
// logging
logLevel string
logFormat string
// 'global' flags
waitTimeout int
resolution string
chromeTimeout int
chromePath string
userAgent string
// screenshot command flags
screenshotURL string
screenshotDestination string
// file scanner command flags
sourceFile string
maxThreads int
// range scanner command flags
scanCidr []string
scanFileCidr string
scanPorts string
skipHTTP bool
skipHTTPS bool
randomPermutations bool
// generate command
reportFileName string
// execution time
startTime = time.Now()
// version
version = "1.0.8"
)
// initLogging prepares the logrus logger and format.
// the --log-level and --log-format commandline args lets you
// control what and how logrus outputs data.
func initLogging() {
switch logLevel {
case "debug":
log.SetLevel(log.DebugLevel)
case "info":
log.SetLevel(log.InfoLevel)
case "warn":
log.SetLevel(log.WarnLevel)
case "error":
log.SetLevel(log.ErrorLevel)
case "fatal":
log.SetLevel(log.FatalLevel)
default:
log.WithField("log-level", logLevel).Warning("invalid log level. defaulting to info.")
log.SetLevel(log.InfoLevel)
}
// Include timestamps in the text format output
textformat := &log.TextFormatter{
FullTimestamp: true,
TimestampFormat: "2006-01-02 15:04:05",
}
switch logFormat {
case "text":
log.SetFormatter(textformat)
case "json":
log.SetFormatter(new(log.JSONFormatter))
default:
log.WithField("log-format", logFormat).Warning("invalid log format. defaulting to text.")
log.SetFormatter(textformat)
}
}
// Checks if some of the globally provided arguments are valid.
func validateFlags() {
// Check screenresolution argument values
parsedResolution := strings.Split(resolution, ",")
if len(parsedResolution) != 2 {
log.WithField("resolution", resolution).Fatal("Invalid resolution value provided")
}
if _, err := strconv.Atoi(parsedResolution[0]); err != nil {
log.WithField("resolution", resolution).Fatal("Failed to parse resolution x value")
}
if _, err := strconv.Atoi(parsedResolution[1]); err != nil {
log.WithField("resolution", resolution).Fatal("Failed to parse resolution y value")
}
}
What do I miss ?
Cheers,
L.
Hi,
Can't thank you enough for come up with this tool!
A last release, there's a but where when you specify screenshots folders and generate a report images does not reflect at db. As a result, screenshots won't load.
I'm using --source flag with urls and -d ./screenshots/ flag.
Cheers
Describe the bug
I am experiencing two errors when trying to run gowitness. One error says something about a "context deadline exceeded", and the other is "failed to save screenshot buffer"
To Reproduce
Steps to reproduce the behavior:
On the command line, type:
Expected behavior
Screenshots will be taken of the site.
Version Information:
I guess it would be a good idea to breakdown report.html into different html files rather than display it at a single file.
From time to time, when scanning a large scope, the report.html becomes unresponsive due lack of memory (yeah! my laptop suck!).
So, I think this could be avoided by segmented report into, report.html, report_01.html, report_02.html etc. Maybe 20 screenshots in each file ?
I'll leave it up to you to decide! What you think of that idea ?
Once again, thanks a lot to come up with this awesome tool.
Cheers
./gowitness report generate -D test.db -d /tmp/test/asdfasdf/
will generate an html report and put it in the current working directory, and not place the report in the designated dir based on the cmdline. this is problematic for automation when screenshotting multiple domains using a wrapper script. Also it appears you cant give the report a name other than 'report-0.html', which is also problematic if you're screenshotting several dozen or several hundred domains and each domain needs to have its own report.
Any plans to add support for RDP and VNC screenshots? There doesn't seem to be any tools out there that do this anymore and it would be nice if gowitness supported it.
Pretty simple stuff: I couldn't find examples for file
flag when checking the docs. Also, the info presented when running --help doesn't say much. Ended up checking the file.go
to get the proper syntax and then came across this:
$ gowitness file -s ~/Desktop/urls
$ gowitness file --source ~/Desktop/urls --threads -2
so maybe just add it directly to the main docs?
Thanks
It would be useful if this tool supported a file input to provide a list of urls for screenshots.
Hello,
I made a few test, but this chinese websites don't work
./gowitness single --chrome-timeout 100 -u http://www.qry.com/Home.aspx
ERRO[2019-02-24 15:55:32] Failed to query url error="[Get http://www.quippbattery.com/Home.aspx: read tcp 192.168.178.41:35752->222.76.212.44:80: i/o timeout]" url="http://www.quippbattery.com/Home.aspx"
INFO[2019-02-24 15:55:32] Complete run-time=5.266110312s
I think the "Failed to query url" Error comes to early. " --chrome-timeout 100" has no effect.
Thank you, Best Regards
Simon
Describe the bug
No such file or directory while the file exist
To Reproduce
Steps to reproduce the behavior:
~/go/bin/gowitness file domain.txt 10:22:42
FATA[2020-05-12 10:22:49] Unable to read source file error="open : no such file or directory" source=
Expected behavior
File to be read
Version Information:
Describe the bug
Trying to screen shot single url
To Reproduce
Steps to reproduce the behavior:
1.Type this in terminel and get the following error
gowitness single --url=https://www.facebook.com/
Error
gowitness single --url=https://www.facebook.com/
ERRO[2020-04-01 15:32:28] Failed to query url error="[context deadline exceeded (Client.Timeout or context cancellation while reading body)]" url="https://www.facebook.com/"
INFO[2020-04-01 15:32:28] Complete run-time=3.1479092s
Screenshots
If applicable, add screenshots to help explain your problem.
Version Information:
Ubuntu WSL (Windows)
Is your feature request related to a problem? Please describe.
When doing recon on domains containing specific words, i'm really interested in the registrant information.
Describe the solution you'd like
Having a button in the report to view whois data. As well as a button to view the HTTP headers. Something like how Aquatone generates reports.
Describe alternatives you've considered
Trying to have a seperate process fetch whois data for all the domains and edit the report template to search for the corresponding whois data.
I'm running the script using the file command, and have lots of URLs in my target file. The problem is when the URL is http://x.x.x.x and the response is a redirect to https://x.x.x.x, the script doesn't follow the redirect which results in lots of blank png files.
Describe the bug
Nmap files containing MAC addresses are included in the scan for URL's when parsing the nmap XML file
To Reproduce
Example NMAP File containing multiple addresses:
<address addr="192.168.1.1" addrtype="ipv4"/>
<address addr="00:00:00:00:00:00" addrtype="mac" vendor="FakeVendor"/>
Gowitness Result:
17 Sep 2020 10:50:00 ERR preflight failed for url error="Get \"https://00:00:00:00:00:00\": dial tcp:
lookup 00:00:00:00:00:00: no such host"
Expected behavior
Just need to modify cmd/nmap.go to exclude an address type of 'mac'
Version Information:
Additional context
I'm 99% sure MAC addresses can't be used in a URL, but not 100% sure
I'm wondering if it possible to go troufht a subdomain with out the quote http://
Great tool!
Just a word about the license, though, Creative Commons isn't really meant to be used for software. They even explicitly recommend against it:
https://creativecommons.org/faq/#can-i-apply-a-creative-commons-license-to-software
I'd love to make some changes / features / etc..., but things like the non-commercial clause of the CC license here make that problematic.
Thanks!
i clone it, then i run go build then i try to use it , but nothing happen
what should i do next ?
thank you
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.