How to spider site on internal VM

How to spider the website which is hosted on an internal virtual machine?

kali@kali:~$ gospider -v -s "https://vm-test-web/"
[0000] ERROR Failed to parse domain

Gospider pulling strange URLs

I think this is a bug
for me this works only when restarting my machine \ temp fix
gospider -s https://$domain -d 0 --sitemap -q -c 10 -a >>gospider_$domain \ my command
strange urls are getting code: 200 ok \ catch

Silent option

An option to suppress all the output would be good please
thanks

option to get just urls

Would be cool an option to get just the urls alone without the [words] and everything else. like a flag to do that

[Enchancement] Plain output format

Hi!
First, thank you for such awesome tools.

I've just started using the gospider and like the way it crawls and lists down the output. I however, miss a plain formatted output to pass the urls to other tools. Is there a possibility to use a -plain format to get the output of just the urls without the preceding information like the source of the url and http code?

One challenge I notice is the way to output information from linkfinder.

Thanks

[Enhancement] Brute force the hidden GET&POST parameters

ref:
https://github.com/m4ll0k/Aron
https://github.com/s0md3v/Arjun

[Enhancement] Raw URL Output

First off just wanted to say, love the tools man!

Could you add a flag to output all the links in raw format?

Thanks!

Include response length in output

It would be useful if the response length was included in the output, or even better, have a way to filter the output by response length (!=, <, >). This would allow the user to filter out expected responses during enumeration.

Is there any way to provide multiple cookie.?

Hi@j3ssie,
is there any way to provide multiple cookie bcoz when ever the spider hit /logout the cookie gets expire.. it will be an amazing feature if the user can provide multiple cookie!

Not able to see operations

Not able to see anything while running the tool. I tried all possibilities like for single URL, used threads but it dont get executed and no results

WARN You are not set VirusTotal API Key yet

I can't find the config file do add the API anywere.
" WARN You are not set VirusTotal API Key yet."

issue with URLs containing dashes

URLs containing dashes in a list can not be parsed:
http://ec2-XXX-XX-XX-XXX.compute-1.amazonaws.com

gospider -S test.txt -v
[0000] ERROR Failed to parse domain

[command] js files

Hi ,

is there a way that i can request for only linkfinder/.js files the --js doesnt help?

thanks
ipk1

Can't crawl localhost

Hi 👋🏻

I tried to crawl http://localhost:3000 but looks like i can't: [0000] ERROR Failed to parse domain

How could i fix that please?

Using the latest commit.

Not works with proxy option when response is HTTP/2

Hi!

There is a bug when using proxy option and the server response HTTP/2.
The crawling is immediately fail, and the debug message was this.

I used gospider -s {{MY INPUT DOMAIN}} --depth 3 -v -t 50 -c 3 -o result --proxy http://127.0.0.1:8080 --debug command.

[0007] DEBUG Error request: {{MY INPUT DOMAIN}} - Status code: 0 - Error: Get "{{MY INPUT DOMAIN}}": net/http: HTTP/1.x transport connection broken: malformed HTTP version "HTTP/2"

I think it's because of some modules version.

Missing License

Hello,
I didn't find license information.
Can you add a LICENSE file or add the license information in the README.md?

I would like to package it for Kali Linux:
https://bugs.kali.org/view.php?id=6514

Thanks.

Too many open files

Command:
gospider -S "urls files" -o outuputfles -c 5 -t 100 -d 2 --other-source -v --robots --sitemap -u web

Error:
[0024] ERROR Failed to open file to write Output: open *********/target_folder : too many open files

Linkfinder

Hey! Can you please make linkfinder default as false.

Self-defined User-Agent is case-insensitive

Some web apps bind user's cookie and user-agent header for access control, however self-defined user-agent args passed in gospider are turning into lower-case.

Help to install the script on Mac

Hi

can you please explain how can I install this tool in Mac?

Thanks

Pentest@tools ~ % sudo go get -u github.com/jaeles-project/gospider
Password:
Pentest@tools~ % gospider -s "https://google.com/" -o output -c 10 -d 1
zsh: command not found: gospider

Don't crawl similar URL

Hi guys. I want to thank you for the great tool. And there are some suggestions.
As the pic above shows, there are many similar URL in one site, can I have some method to ignore them. as just fetch one of them?

Avoid including wrong results caused by html-redirect

Try to use gospider on a site like https://vivy.com. You'll realize that due to the application "redirect" implementation all the urls are added to the output file just because the server return a 200 code.

Issue:
gospider retrieves false positive urls for applications that implement the redirect not using http codes.

To replicate:
Go to https://vivy.com/made_up_url and notice that the application returns a 200 code even though the content is an html page the redirects to the home page

<html>
<head>
	<meta http-equiv="refresh" content="0; url=https://www.vivy.com/" />
</head>
<body></body>
</html>

Possible way to solve this:
Add an option to specify a regex that, if matched in the response automatically discards the url or marks it in the output file.

Proxy settings not fully honored

It seems that proxies are not honored, by looking at Wireshark traffic I see some requests not going through any proxy.

I think this is related to gocolly/colly#392

We probably need to disable keep alives like:

c.WithTransport(&http.Transport{
  DisableKeepAlives: true,
})

in the code here

Where do I add the Virus Total API key?

I can't find the config file do add the API anywere.
" WARN You are not set VirusTotal API Key yet."

Scan with just urls and without duplicates

Hi i saw duplicates issue but still idk what is solution and have 1 errror too

How can i scan without duplicates and with just urls

some of the js file link is contained in <link href="">

Thank you so much for sharing. it helped me a lot.
and some of the js file link is contained in ,i think
crawler.C.OnHTML("[href]", func(e *colly.HTMLElement) {} can also use crawler.LinkFinderCollector

doesnt pull links out of html for bandcamp

  ~ $(go env GOPATH)/bin/gospider -s "https://dietcig.bandcamp.com/" -B --debug -v
[0000]  INFO Start crawling: https://dietcig.bandcamp.com/
[subdomains] - dietcig.bandcamp.com
[url] - [code-200] - https://dietcig.bandcamp.com/
[form] - https://dietcig.bandcamp.com/
[javascript] - https://s4.bcbits.com/bundle/bundle/1/global_head-243a7059b280df57aee82a3092d1b787.js
[javascript] - https://s4.bcbits.com/bundle/bundle/1/tralbum_head-dc42a836d10ac56b3aa730cfbe07b7d6.js
[javascript] - https://s4.bcbits.com/bundle/bundle/1/global_foot1-b86843302dee22779f9059bc9e3e5eb6.js
[javascript] - https://s4.bcbits.com/bundle/bundle/1/global_foot2-ecf4cff3c8eb7c84aaa8e0a1fd4eab75.js
[javascript] - https://s4.bcbits.com/tmpdata/cache/global_validators_bundle_8c57f6d030cddfb05cf1aae7942b766f.js
[javascript] - https://s4.bcbits.com/bundle/bundle/1/jquery_lazyload-9222bb350f055a9536b19a5494dcef8f.js
[javascript] - https://s4.bcbits.com/bundle/bundle/1/tralbum-4c810106883025cd62fc173a249d25cb.js
[javascript] - https://s4.bcbits.com/bundle/bundle/1/tralbum_templates-787d8255c41d27c39925065ea0a6b314.js
[javascript] - https://www.google-analytics.com/analytics.js
[0000]  INFO Done.

I know these links are in the raw html because I can pull them out by curl + html parser/selector

  ~ curl https://dietcig.bandcamp.com | $(go env GOPATH)/bin/pup 'a attr{href}'|grep album 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  183k    0  183k    0     0   337k      0 --:--:-- --:--:-- --:--:--  336k
/album/do-you-wonder-about-me
/album/over-easy-green-eggs-ham-edition
/album/swear-im-good-at-this
/album/sleep-talk-dinner-date
/album/over-easy
/help/downloading?from=tralbum_downloading

How to spider sites which are hosted on a virtual machine

How to spider sites which are hosted on a virtual machine e.g. https://vm-site-web:883/
gospider can't find anything...

Enhancement Request - Brute forcing

Absolutely love the tool. Having the --other-source flag is a life saver. I was thinking, you should add a flag to allow user defined directory brute forcing. Some sites don't redirect to the valid endpoints. Having a user defined seed list (similar to --other-source) will help because simply having gobuster crawl the "/" directory may not result in much data.

Example:

User develops a text file of common directories. (/login, /auth, /home, etc)
Provide a flag such as --seed or --brute to help find more endpoints.

Scope Constraints

Issue: Many requests are made to domains that were not supplied as input

Solution: Restrict requests to domains supplied as input, or provide an option to do so when desired.

Output only URLs

Hi,

At first, congratulations for this project.
I am have an issue, maybe by my mistake, but I'd want to send to stdout only URLs, without flags like [url] and [code-200].
Is it possible?

Update the release binary

Can you please update the binary to the latest commit. I don't know how to install go projects from the source itself and install it.

json output error

Hi there, I tried the --json option, and when gospider is used with --sitemap, that output is not included in json format.

If you need some output I can provide some. On the other hand if this is intentional behavior sorry for opening this issue

regards

help

what does it mean --include-other-source

Add license

There is no statement present about the licensing of the project.

Please add a LICENSE file. Thanks.

Subdomains are not shown in output

Hey, since the last update the gospider tool is not showing subdomains in output. I have checked this with multiple flags but it's not working.

The line with the pointed arrow is now missing from gospider output.

Can you please add it back?

Best,

Linkfinder issue

@j3ssie Linkfinder is not working by default in this newer version 1.1.6

Several bugs and enhancements

Hello there,

In the latest version, there are several bugs from which one of the most annoying ones is that -q flag is not quite working as intended. here is an example output:

[aws-s3] - github-cloud.s3.amazonaws.com
[url] - [code-200] - https://github.githubassets.com/assets/gist-5b974504.js
[url] - [code-200] - https://github.githubassets.com/assets/behaviors-3afe1224.js

Another issue I'm getting is that the JSON output results in an error combined with jq.

$ gospider -s https://github.com -q --json | jq

...
parse error: Invalid numeric literal at line 291, column 5

Another suggestion is that to provide a scheme-less URL support:

$ gospider -s github.com 

above command doesn't work now, and only by using HTTP scheme it works:

$ gospider -s https://github.com

Regards,
HolyBugx

RAM usage

Gospider uses a lot of RAM.
It's ram usage keeps increasing with time . And it chokes my server if it keeps running for a few hours even with low threads (2-3).

Is there any solution to this? or can you kindly solve this issue if possible?

Thanks

Can't download the tool

go get: github.com/jaeles-project/[email protected]: parsing go.mod:
module declares its path as: github.com/theblackturtle/gospider
but was required as: github.com/jaeles-project/gospider

This is the error message I am receiving and I can't move forward.

GoSpider doesn't seem to honor the delay parameter

Hi @j3ssie and team,

From what I can tell, setting the -k/--delay parameter doesn't delay anything. GoSpider still requests URL faster than expected.

Can you replicate this issue?

go get installation errors

go get -u github.com/jaeles-project/gospider
# github.com/jaeles-project/gospider/core
go/src/github.com/jaeles-project/gospider/core/crawler.go:27:17: unknown field 'MaxConnsPerHost' in struct literal of type http.Transport
go/src/github.com/jaeles-project/gospider/core/crawler.go:183:15: undefined: strings.ReplaceAll
go/src/github.com/jaeles-project/gospider/core/crawler.go:296:20: undefined: strings.ReplaceAll
go/src/github.com/jaeles-project/gospider/core/linkfinder.go:14:12: undefined: strings.ReplaceAll
go/src/github.com/jaeles-project/gospider/core/linkfinder.go:15:12: undefined: strings.ReplaceAll

This appens when downloading directly with go get -u github.com/jaeles-project/gospider

Any advice? Wrong Go version?

OS: Ubuntu 18.04.3 LTS x86_64
Kernel: 4.15.0-76-generic
Go Version: go version go1.10.4 linux/amd64

Duplicate URL's

@j3ssie any way to avoid duplicate urls as it never ends on some domains and keep continue with duplicate urls which increase the time to crawl for hours on same requests
[url] - [code-200] - https://example.com/
[url] - [code-200] - https://example.com/
[url] - [code-200] - https://example.com/
[url] - [code-200] - https://example.com/
[url] - [code-200] - https://example.com/
[url] - [code-200] - https://example.com/
[url] - [code-200] - https://example.com/

It also needed a switch to filter / skip urls throwing particular status code.

[Enhancement] Add input URL at plain output

Hi!
I was get troubled to guess input URL at plain result, and I saw the JSON option added.
I found that JSON result is writing input URL, and I think it would be really nice to add input URL at plain result too.
I think some people would get same trouble like me..
What do you think about this suggestion?

[Enhancement Request] Change Output Filename to Use . instead of replacing with _

I noticed in crawler.go there is a line of code on line #186 that replaces the filename . with _. This makes it more difficult to add this tool into a script to be chained with other tools. An easy fix would be to remove that function or allow the user to write the output filename themselves.

Colors

Hello
Thats about colored output like a dirhunt?

'-alert(3)-'

Empty output specifying HTTP(S) port

Description

Web I try to run gospider on an URL specifying also the HTTP port, sometimes I don't know why exactly it doesn't crawl the target.

Go version

go version go1.16.2 linux/amd64

Gospider Version

1.1.5 (In the last commit of https://github.com/jaeles-project/gospider/blob/2e610b3fd79e1ac0945b694385edd88028f821ce/core/version.go the version is wrong btw)

Test case 1 - Not specifying http or https port

./gospider -q -s https://shippingmanager.bpost.be/ --debug

[0000]  INFO Start crawling: https://shippingmanager.bpost.be/
[0000]  INFO Found robots.txt: https://shippingmanager.bpost.be//robots.txt
https://shippingmanager.bpost.be/ShmFrontEnd/
[0000]  INFO Done.

Test case 2 - Specifying the port:

./gospider -q -s https://shippingmanager.bpost.be:443/ --debug

Relathive path in js files makes invalid urls in linkfinder module

Hey

I found that if a relative path exists in a js file like the following:

telerik_ReportViewer({serviceUrl:"../../api/reports"

gospider will place in a loop and generates lots of invalid URLs and runs until you break the gospider manually.

You can test it on this link: https://qa-opentoweriq.bentley.com

can't install

can you tell me how to install it using "go install" please?
because the "go get -u" command is deprecated

go get: installing executables with 'go get' in module mode is deprecated.
	Use 'go install pkg@version' instead.
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.

jaeles-project / gospider Goto Github PK

gospider's People

Contributors

Stargazers

Watchers

Forkers

gospider's Issues

Description

Go version

Gospider Version

Test case 1 - Not specifying http or https port

Test case 2 - Specifying the port:

Recommend Projects

Recommend Topics

Recommend Org