sirwart / ripsecrets Goto Github PK

Origin: https://gitlab.com/gitlab-org/gitlab/-/issues/342327
Docs: https://docs.gitlab.com/ee/user/admin_area/settings/account_and_limit_settings.html#personal-access-token-prefix
Pattern: "glpat-[A-Za-z0-9_/-]{20,}"

Just found a bunch of additional alternatives to add to the README:

• repo-security-scanner
• git-secrets

Say my .secretsignore should ignore everything:

$ cat .secretsignore
*

It doesn't unfortunately work:

$ pre-commit run ripsecrets --all-files --verbose
Found existing alias for "pre-commit run". You should use: "prcr"
ripsecrets...............................................................Failed
- hook id: ripsecrets
- duration: 0.01s
- exit code: 1

credentials.toml:3:password = 'uJSU7Kxquv5FXDRLF7SCBaksmo9o2Zp8'

Adding an empty [secrets] section makes it work though:

$ echo "[secrets]" >> .secretsignore

$ pre-commit run ripsecrets --all-files --verbose
Found existing alias for "pre-commit run". You should use: "prcr"
ripsecrets...............................................................Passed
- hook id: ripsecrets
- duration: 0.01s

This seems to be due to missed creation of the Gitignore object in one branch in https://github.com/sirwart/ripsecrets/blob/main/src/ignore_info.rs

Long secrets are not ignored.

See the example below. The first two secrets are ignored, while the last two are not.

.secretsignore

[secrets]
96etKOmnte-bpLDSIcwdhXYlC82gF8x-ERPqZ7oo1Ug
8AOiCMgwF1eg5yLDgw9D1eymTSOp21PJwr4zdQRQyYQ
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eODlIjoiYWNjZXNzIiwiZXhwIjoxNjUxMTQxMzc3LCJpYXQiOjE2NTExNDA0ODAsImp0aSI6ImQzAAJmYzBiNzI2NDRjMjY5ODI0NGFiMTQ2OTc1N2YyIiwidXNlcl9pZCI6MX0.87aml-57DmEUo4LrlZwnDw4iVfiWVNA90xxCi01M2h0
eyJ0eXAiOiJKV1QiCCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eWPlIjoiYWNjZXNzIiwiZXhwIjoxNjUxMTQxMzgxLCJpYXQiOjE2NTExNDEwODEsImp0aSI6Ijk1YjRjMDA2ODZjNTRkYTU4OTE1NWYzOTgzZjcxNmJiIiwidXNlcl9pZCI6MX0.zs-3zv1eCSu9JeRBJgFw6CBoZUA4B2R3z6gl2vNYwdA

Easily reproducible like this:

ripsecrets .secretsignore

Cargo installation fails citing could not find _unix_ in _os_

   Compiling ripsecrets v0.1.5 (C:\Users\dsieradski\.cargo\git\checkouts\ripsecrets-0fa9e8d7534dec57\ac370a9)
error[E0433]: failed to resolve: could not find `unix` in `os`
 --> src\pre_commit.rs:6:14
  |
6 | use std::os::unix::fs::PermissionsExt;
  |              ^^^^ could not find `unix` in `os`

error[E0599]: no method named `set_mode` found for struct `Permissions` in the current scope
  --> src\pre_commit.rs:79:11
   |
79 |     perms.set_mode(perms.mode() | 0o100);
   |           ^^^^^^^^ method not found in `Permissions`

error[E0599]: no method named `mode` found for struct `Permissions` in the current scope
  --> src\pre_commit.rs:79:26
   |
79 |     perms.set_mode(perms.mode() | 0o100);
   |                          ^^^^ method not found in `Permissions`

Some errors have detailed explanations: E0433, E0599.
For more information about an error, try `rustc --explain E0433`.
error: could not compile `ripsecrets` (bin "ripsecrets") due to 3 previous errors
error: failed to compile `ripsecrets v0.1.5 (https://github.com/sirwart/ripsecrets?branch=main#ac370a97)`, intermediate artifacts can be found at `C:\Users\DSIERA~1\AppData\Local\Temp\cargo-installNnxbIE`

I looked at clap_mangen and I think it's the right thing to use, but it looks like it might need some massaging to get the ripsecret::main::Args into it.

While secrets is basically "get secrets" and expect none to be returned, the name is, well, not very S.E.O. friendly. @sirwart, are you open to considering some other names while this tool is still in its infancy?

Hi there! First off, thanks for this awesome tool!

I'm seeing some unexpected / inconsistent behavior where secrets files are being sometimes being ignored, based on whether the [secrets] section is present in the .secretsignore file.

Setup:

> ripsecrets --version
ripsecrets 0.1.3
# this is the v0.1.5 git tag, but it has 0.1.3 as the Cargo.toml version

# create some secret file with TOTALLY FAKE AWS access key
> echo "aws_access_key_id = AKIATHAAGaZc4krwNWdc" > test.txt

# create a secrets ignore file
> echo "test.txt" > .secretsignore

> ripsecrets && echo "no secrets found"
no secrets found
> ripsecrets --strict-ignore && echo "no secrets found"
no secrets found
> ripsecrets --strict-ignore test.txt && echo "no secrets found"
test.txt:1:aws_access_key_id = AKIATHAAGaZc4krwNWdc

So when using normally, with no positional file/directory specified, it comes back exit code 0. But when the file is provided as a positional argument, it comes back with a failure.

But, if we add a [secrets] tag to the .secretsignore file, then the command comes back with exit code 0.

> echo "[secrets]" >> .secretsignore

> cat .secretsignore
test.txt
[secrets]

> ripsecrets --strict-ignore test.txt && echo "no secrets found"
no secrets found

I think this is because in src/find_secrets.rs, it conditionally ignores the explicitly provided files only if there's a ignore_matcher.is_some() (permalink), which looks like it's coming from ignore_info.rs behind this conditional.

Is this expected behavior? If not, I'm happy to contribute a fix!

Thanks again!
Cheers!

@sirwart Are there any specific issues you want fixed or pending features you want implemented before releasing 1.0.0?

Splitting off from #66 (comment) and #66 (comment), it'd be great if this was uploading to crates.io on release.

https://doc.rust-lang.org/cargo/reference/publishing.html

https://users.rust-lang.org/t/does-anyone-use-github-actions-to-run-cargo-publish/92374

Feature request

Allow global .secretsignore (in user home directory). Alternatively, allow providing path to the .secretsignore file.

When trying to install ripsecrets via cargo I noticed that it now tries to download the entire sentry module because it's listed as a submodule, which takes a long time because it's a large repo. Either the configuration needs to change or removed as a submodule.

Was Looking at https://github.com/sirwart/secrets/blob/main/src/find_secrets.rs#L24= and found reference to the AKIA-like set of access key IDs. I don't think these are secrets, however the regex could be replaced for the true format of AWS Secrets which would be tied to these identifiers (as listed under "Secret keys"): https://summitroute.com/blog/2018/06/20/aws_security_credential_formats/

➜ temp cat src/source.java
password=123
username=333
pwd=344
passwort=3333333
benutzername=32
➜ temp ripsecrets
➜ temp

When we find "random string" secrets, we use a regex that matches patterns along the lines of secret_key = "<key>". However, the matcher should return the <key> as the match range. This isn't a problem now since we don't highlight matched secrets, but it's preventing this PR from being merged in: #18.

I happened upon https://github.com/OWASP/SEDATED/blob/master/config/regexes.json today. I think it has some key formats that ripsecrets doesn't. It also looks for variable names, which is not desirable for ripsecrets, so those can be ignored.

According to this PR: #2:

Once this is merged, someone (me?) could put a PR on https://github.com/pre-commit/pre-commit.com/blob/main/all-repos.yaml that adds https://github.com/sirwart/secrets to it.

Someone needs to add https://github.com/sirwart/ripsecrets to the pre-commit repository.

Like this https://bind9.readthedocs.io/en/v9_16_5/advanced.html#loading-a-new-key
See also discussion in #35

I just tested secrets on my repo, and it did a really good job! It discovered a couple of (test mode) unencrypted private keys, and flagged a bunch of public key fingerprints that definitely look like secrets. Kudos on the tool!

In case you're looking at tweaking the heuristics, I wanted to call out this line it flagged, which is clearly a false positive:

if ((filterMode == FilterMode.OFFLINE_STORAGE && key.isPersistent()) || (filterMode == FilterMode.API_RETURN_VALUE && key.isReturnable())

Thanks!

Hi!

Love the tool! The performance gains are awesome! Curious if you had any plans to make adding additional secret patterns to the tool more friendly (Maybe an external TOML file with user specified patterns?). If you could forsee this being part of the tool, would love to pull together a PR if we can agree on an implementation.

Otherwise - would be happy to PR some additional secret patterns!

Since there are already pre-built binaries for each release, it would be nice if we could utilize them in the pre-commit hook instead of requiring a dependency on cargo install, which isn't necessarily available in all environments you would want to check for secrets.

The release CI only handles building an OCI image. I speculate that the release artifacts are handled locally. The ripgrep release.yml might be a good example to follow.

I'm happy to do this up if you greenlight it.

I write a simple code like

package main

func main() {
        clientSecretKey := "alkfjlaf^*flkajlfkay7782085ljafg"
        println(clientSecretKey)
}

and hope ripsecrets can tell me 'you hardcode the secret in source files', but there's nothing output

it's very slow compared to rg (with all the extras removed) - I only checked for 3 piped rgxs

On the heels of this wild news about Microsoft leaking 38 TB of data because of a committed SAS token, maybe ripsecrets could audit for that, too.

https://learn.microsoft.com/en-us/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens?tabs=Containers

Here are some examples from that doc:

{
    "inputs": [
        {
            "storageType": "File",
            "source": {
                "sourceUrl": "https://my.blob.core.windows.net/source-en/source-english.docx?sv=2019-12-12&st=2021-01-26T18%3A30%3A20Z&se=2021-02-05T18%3A30%3A00Z&sr=c&sp=rl&sig=d7PZKyQsIeE6xb%2B1M4Yb56I%2FEEKoNIF65D%2Fs0IFsYcE%3D"
            },
            "targets": [
                {
                    "targetUrl": "https://my.blob.core.windows.net/target/try/Target-Spanish.docx?sv=2019-12-12&st=2021-01-26T18%3A31%3A11Z&se=2021-02-05T18%3A31%3A00Z&sr=c&sp=wl&sig=AgddSzXLXwHKpGHr7wALt2DGQJHCzNFF%2F3L94JHAWZM%3D",
                    "language": "es"
                },
                {
                    "targetUrl": "https://my.blob.core.windows.net/target/try/Target-German.docx?sv=2019-12-12&st=2021-01-26T18%3A31%3A11Z&se=2021-02-05T18%3A31%3A00Z&sr=c&sp=wl&sig=AgddSzXLXwHKpGHr7wALt2DGQJHCzNFF%2F3L94JHAWZM%3D",
                    "language": "de"
                }
            ]
        }
    ]
}

Looks like the presence of sv with an ISO date and sig query params, and sig is base64 encoded.

Adding these URLs to the end of test/one_per_line/azure reflects that ripsecrets doesn't already catch them.

The only official reference to syntax I could find was this image

from https://access.redhat.com/documentation/en-us/red_hat_quay/3/html/use_red_hat_quay/use-quay-manage-repo

Pretty sure it's just "[A-Z0-9]{64}" though.

I'm not sure exactly what's entailed by this, but I don't think it would be too difficult

got a false positive on https://ola.hallengren.com/scripts/MaintenanceSolution.sql

./MaintenanceSolution.sql:84:)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)

i went to ignore just this "secret" in the .secretsignore file like this:

[secrets]
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)

but couldn't get it to ignore... had to add the whole file to .secretsignore

MaintenanceSolution.sql

[secrets]
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)

this file's like 9100 lines long, so not ideal 😄

this is using secrets 0.1.2