eth0izzle / shhgit Goto Github PK

I had run: sudo docker-compose build which gives error:

Hello, I maintain the shhgit package in the Arch User Repository and currently the package is not working because the latest available release for shhgit is v0.2 which is almost 2 years old. This makes packaging shhgit harder as v0.2 uses an older Go version.

I´d greatly appreciate it if you could add a more updated tag. 😄
Thanks.

Hello I'm starting with shhgit
And trying to configure the integration with bitbucket.

Which session do I need to add to config.yaml? have any example?

Thank you very much

As far as I know shhgit supports:

blacklisted_strings, blacklisted_extensions, blacklisted_paths and blacklisted_entropy_extensions.

Would it be possible to add blacklisted_regex so we can properly discard things we don't want? I can do it with grep on the output, but there are side cases, like when there is more than 1 match inside the same file, that one might match the regex (being a false positive) and will also discard the valid finding.

When I follow the instructions I get an error:

$ go get github.com/eth0izzle/shhgit
# gopkg.in/src-d/go-git.v4/plumbing/transport/ssh
src/gopkg.in/src-d/go-git.v4/plumbing/transport/ssh/common.go:147:15: undefined: proxy.Dial

Any tips?

From https://shhgit.darkport.co.uk/

TODO: clean up the code and do some further testing across platforms.

Hi We have about 200 gitlab repos- could you please help us with usage instructions with gitlab.

For example, having it so that one can blacklist AKIAIOSFODNN7EXAMPLE, the AWS example key, might be useful for filtering noise out.

I don't use Slack as much as I used to, but discord I have running 24/7.

Could you please fix these broken images?

Thanks.

This is pretty rough right now but it gets the job done.

https://github.com/0xtavian/shhgit

Things changed:
Added basic auth functionally for Cloning repos which is required by GHE https://github.com/0xtavian/shhgit/blob/master/core/git.go

Implemented baseURL here https://github.com/0xtavian/shhgit/blob/master/core/session.go#L65

In order to have PRs validated a github action could be very helpful.

For a simple start:

• Build docker image
• Run docker image

I was wondering if there is a possibility to support bitbucket for enterprise usage. For instance, if a company has their private bitbucket server and would like to monitor the secrets. How difficult is it to tweak shhgit to do the job for them?

In the latest version of shhgit, there is no option to scan local GitHub repos. How to scan for a particular repo (present locally) or the repos of any particular organization?

Ran the following commands and am presented with a strange error that I cannot seem to figure out or find a solution for.

$ git clone https://github.com/eth0izzle/shhgit

Cloning into 'shhgit'...
remote: Enumerating objects: 218, done.
remote: Total 218 (delta 0), reused 0 (delta 0), pack-reused 218
Receiving objects: 100% (218/218), 3.21 MiB | 6.93 MiB/s, done.
Resolving deltas: 100% (127/127), done.

$ cd shhgit/ && vim config.yaml
Here I add my Github API key.

Run the command to get the container 'running' as in the README.md:
$ docker run -v config.yaml:/config.yaml:ro eth0izzle/shhgit

The command hands for a brief second and spits out:

read /config.yaml: is a directory

Running docker ps does not show the container at all.

Any help would be appreciated.

I downloaded the shhgit_linux_amd64_0.2.zip file in Releases and found that there is no --local option

I would like to request support for Azure Devops Cloud and On-Premise.

Here is the link to the list repository documentation and get repository documentation
https://docs.microsoft.com/en-us/rest/api/azure/devops/git/repositories/list?view=azure-devops-server-rest-5.0
https://docs.microsoft.com/en-us/rest/api/azure/devops/git/repositories/get%20repository?view=azure-devops-server-rest-5.0

shhgit should be able to output to different sources and formats, i.e. csv, json, a postgres database, UDP, elasticsearch, etc. We should take a modular approach for extensibility, i.e. struct embedding

Hi, got a panic:

Error getting GitHub events... trying again
%!(EXTRA *url.Error=Get https://api.github.com/events?per_page=300: dial tcp 140.82.118.6:443: connect: operation timed out)panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x13d8427]

goroutine 50 [running]:
github.com/eth0izzle/shhgit/core.GetRepositories(0xc0000d4750)
/Users/usr/Documents/Go/src/github.com/eth0izzle/shhgit/core/github.go:47 +0x4c7
created by main.main
/Users/usr/Documents/Go/src/github.com/eth0izzle/shhgit/main.go:143 +0x203

I've found many secrets in GitHub issue comments, i.e. people copy pasting their code asking for help without redacting the secrets/keys - you can even view comment history if they were later removed.

We can listen to the IssueCommentEvent to get a stream of real time comments (https://docs.github.com/en/developers/webhooks-and-events/github-event-types#issuecommentevent) and process the comment key within the payload as if it were code (we would need to skip file path + extension checks).

Create package for the homebrew (https://brew.sh/), I m working on this feature to create shhgit available on homebrew package manager

If I have a list of repo's say HackerOne for example, is there a variable I can use to get it to check against those specific ones?

Changing the base url for GitHub will allow support for GitHub Enterprise.

How do we run a similar instance like https://shhgit.darkport.co.uk/? Thanks!

I'm running an instance with approx. 10 access tokens. Once all have been exhausted, I receive the following log entries.

Jun 11 14:07:23 gitrecon shhgit[111502]: All GitHub tokens exchausted/rate limited. Sleeping until 16m54.566347494s
Jun 11 14:07:23 gitrecon shhgit[111502]: All GitHub tokens exchausted/rate limited. Sleeping until 16m54.566347494s

[time passes]

Jun 11 14:23:53 gitrecon shhgit[111502]: All GitHub tokens exchausted/rate limited. Sleeping until 16m54.566347494s
Jun 11 14:23:53 gitrecon shhgit[111502]: All GitHub tokens exchausted/rate limited. Sleeping until 16m54.566347494s

The same pattern of messages repeats for hours. I know at this point the tokens have API calls refreshed, because if I restart shhgit, everything starts working as normal.

Using current master branch on linux - eadd127

Hello,

I just wanted to know how many github tokens we need so we don't get rate-limited. Each token should have 5k requests and I assume shhgit iterates this tokens to not get rate-limited?

Great tool. What if I want to search for a domain or org name in combination with all the signatures? Is this possible currently?

As a developer and user of shhgit scanning local directories I most likely will have local files like for example .env files containing secrets which are protected through the full-disk-encryption of my machine and prevented to be accidentally committed through the .gitignore or ~/.gitignore_global file.

Expected behavior:

• I can specify an option to exclude those files matching gitignore-entries from the search

Actual behavior:

• I have to filter those entries from the shhgit log after the scan finished

Though this option should not be enabled by default as it reduces the visibility of secrets lingering around on the disk (which shouldn't exist in the first place) which could be committed through git add -f…

How to restrict the search to specific or list of organization ?

Hello !

I've found out that most of the signatures checks on path when working on a Windows doesn't work because of the specific path formatting.
Example:
I have a test folder containing the filepath "etc/passwd" on a Windows machine, but the path scanned by shhgit is "test\etc\passwd".

A workaround would be to modify all path checks in the config.yaml file by adding "[\/]" where there is a single "/", so it would match either "\" or "/".
Another workaround would be to modify the path before checking if it matches, with the filepath method "ToSlash()".
Example:
path = filepath.ToSlash(path)

ToSlash returns the result of replacing each separator character in path with a slash ('/') character. Multiple separators are replaced by multiple slashes.

Documentation: https://golang.org/pkg/path/filepath/#ToSlash

Hi,

is it possible to monitor certain github org or users or the repo link itself ?

Thanks

I'm getting this error :/ Could you help me remove this ?

Vicky:shhgit dv$ docker build --tag fnxpt/shhgit:latest .
\Sending build context to Docker daemon  157.2kB
Step 1/9 : FROM golang:alpine AS builder
 ---> dda4232b2bd5
Step 2/9 : WORKDIR /go/src
 ---> Using cache
 ---> 50bf36eb4bc4
Step 3/9 : ADD . .
 ---> Using cache
 ---> 1f7e4ac77b67
Step 4/9 : RUN export CGO_ENABLED=0 && go install && go build -o /
 ---> Running in e1c84a3a3020
go: errors parsing go.mod:
/go/src/go.mod:3: usage: go 1.23
The command '/bin/sh -c export CGO_ENABLED=0 && go install && go build -o /' returned a non-zero code: 1
Vicky:shhgit dv$ 

The links are are erroneously including the git hash in the url:

As an example, you can see the hash in the URL, when linking to the file:
https://github.com/<REDACTED/<REDACTED>/blob/master/2a078102<REDACTED>658bdb65a8c881/start/server/store.sqlite

I think this should either be /blob/master/$file or /blob/2a078102<REDACTED>658bdb65a8c881/$file.

Running current master 65351a7

Using this API: https://developer.github.com/v3/gists/#list-all-public-gists

Hi

I am testing out shhgit and it works really well. But it seems to be scanning its own config file as part of the local scan.

I would like to use this as part of a CI build and to ideally ignore the config.yaml file. Is this possible?

Many thanks

Simon

using the website for a long time I noted that the tool can get some AWS cred file info and none of AWS Access Key ID, AWS Account ID, AWS CLI credentials, AWS Secret Access Key and AWS Session Token. But every AWS cred file info has this another options.

Maybe some problem in the regex? I don't know much about regex, but I want to colaborate if someone get my guidance.

I.e. if you want to search for any mentions of certain domains in real time.

i have a number of pproblems but all can be solved through adding EXAMPLE instances of the apps installation process and usage.
in installing in the powershell cli i used the cloning method and run the go command stated and nothing happened.When i cloned and clicked ok It showed the cant find cmdlet error. i tried a couple of things stated in the pictures below with no progress.
i also had questions concerning which webhook is in use (assuming things went successful after cloning) is it a slack webhook url ? and if so what about the webhook payload?
also in using the app would the commands start with shhgit then a command eg:
shhgit --entropy-threshold
or
shhgit --maximum-file-size
and what about when combining them as i tried just to test and ask you guys and like how it shows in the last picture the maximum file size color shows that it is not being processed as a command like how entropy-threshold is (by looking at the yellow color)
lastly would i have to add a repository of an org/ user after those commands or in a config file or does it simply look at all repos in github. If so what would i have to do to narrow down to certain repositories?
Thank you for your time ADDING EXAMPLES would be HELPFUL to as script kiddies who value this tool. as i've searched for information about this tool elsewhere with no luck.

We were wondering if you have got any plans to add json output. It would be cool to have more information on the owner. You could put some owner information like owner's nickname, email, location, company. Most of the time that information is not available but it's better to have them.

If we have that information, it's quite easy to disclose private information. It would also be nice to index the findings on elasticsearch. In order to generate some cool graphs.

Your project is so cool.

My org is using Azure DevOps and it would be great if we could use this project.

It is though unclear what has to be done in order to integrate, hence also big challenge for me to help out. If there is any documentation or help here I would gladly try to dig in.

I've let the shhgit run for a while and seems that once all the tokens are exhausted it never wakes up again.
I had a look at the code and I think the problem is this:
client.RateLimitedUntil is a Duration, it should be decreased or reset to zero once the sleep occurs, so the recursive function that does the sleep can actually escape returning the client. But the code that re/sets the remaining duration (client.RateLimitedUntil) is done after the recursive sleeping function.

github.go:94
github.go:29

I have a very little knowledge about Go so I might be very wrong there but my shhgit did indeed sleep forever. I did repair it in my fork, exchanging the RateLimitedUntil from time.Duration to time.Time (directly using resp.Rate.Reset.Time value) and in the session.GetClient() checking if the date is after time.Now(), if not then sleep the time.Until duration. The question is if the repair makes sense.
#7

EDIT: The image is from GetGists but the same applies to GetRepositories

The formatting of the messages in Slack via the webhook seem to not be as elegant as they could be (at least in my Slack).

example:
[https://github.com/JianboLi-github/learngit.git] Matching file [33m/a3ecb1c7172fa36f6a4f215bc5cf30e39ae9fb82/TesterHorder/testerhordev3.4/testcase/geckodriver.log[0m for [32mLog file[0m

It does not handle the ansi codes well which throws off the strings and the formatting. I'd suggest for the Slack webhook either fix the coloring or remove it and just have it formatted.

Is causing issues when running this locally because it removes unexpected folders like .git, potentially breaking people's local work

How to start web view results

To replace the current yaml signatures. This will allow us to create mroe powerful rules. For example to find GitHub API keys we would regex on ([a-f\d]{40}), but currently that would produce a lot of false positives (it's a SHA1 hash). With a YARA rule we could do:

rule GitHubApikey
{
    strings:
        $re1 = /[a-f\d]{40}/
        $re2 = /Authorization: token/
        $re3 = /https://api.github.com/

    condition:
        $re1 and ($re2 or $re3)
}

Hello,

Don't know if it's a bug or not but.. this is what happens:

Starting 3a742e725a6b_shhgit.www ... done
Recreating shhgit.app            ... done
Attaching to 3a742e725a6b_shhgit.www, shhgit.app

It hangs, not even the banner shows.

Any thoughts?

via 192680d#diff-d8d0422389f03d783e32e627250fe29834bd09c6361640d1ff00661dd6820034

@@ -1,5 +1,7 @@
github_access_tokens:
  - ''
  - '4388b2658182341d61c1506bdbf249d49d5f2acc'
  - 'dcefdee459ea41ffd80b0372f056ca1a7aec49a1'
  - '26e3f7340239e28acf3a2a1b79736f6f5d81ee9a'

Does V2 version supports GHE?

Following command:
shhgit --config-path "G:\My Drive\Repositories\shhgit" --local "G:\My Drive\Repositories" --csv-path "G:\My Drive\Repositories"

leads to a deadlock, without using the "CSV export" it's working.
(The error occurs no matter where csv-path is located)

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire, locked to thread]:
sync.runtime_SemacquireMutex(0x140a438, 0x0, 0x1)
        c:/go/src/runtime/sema.go:71 +0x4e
sync.(*Mutex).lockSlow(0x140a434)
        c:/go/src/sync/mutex.go:138 +0x10f
sync.(*Mutex).Lock(...)
        c:/go/src/sync/mutex.go:81
sync.(*Once).doSlow(0x140a430, 0x1117320)
        c:/go/src/sync/once.go:62 +0x113
sync.(*Once).Do(...)
        c:/go/src/sync/once.go:57
github.com/eth0izzle/shhgit/core.GetSession(0x0)
        G:/My Drive/Repositories/shhgit/core/session.go:163 +0x65
github.com/eth0izzle/shhgit/core.LogIfError(0x11024a8, 0x1e, 0x117a100, 0xc0003b0120)
        G:/My Drive/Repositories/shhgit/core/util.go:39 +0x45
github.com/eth0izzle/shhgit/core.(*Session).InitCsvWriter(0xc0000cc5a0)
        G:/My Drive/Repositories/shhgit/core/session.go:144 +0xe5
github.com/eth0izzle/shhgit/core.(*Session).Start(0xc0000cc5a0)
        G:/My Drive/Repositories/shhgit/core/session.go:47 +0x126
github.com/eth0izzle/shhgit/core.GetSession.func1()
        G:/My Drive/Repositories/shhgit/core/session.go:181 +0x285
sync.(*Once).doSlow(0x140a430, 0x1117320)
        c:/go/src/sync/once.go:66 +0xf7
sync.(*Once).Do(...)
        c:/go/src/sync/once.go:57
github.com/eth0izzle/shhgit/core.GetSession(0x19)
        G:/My Drive/Repositories/shhgit/core/session.go:163 +0x65
main.init()
        G:/My Drive/Repositories/shhgit/main.go:27 +0x29

Hay Love this idea.

How easy would it be to do this for bitbucket as well?