Git Product home page Git Product logo

secret-shield's Introduction

This project is no longer being actively maintained. We understand the value of this, and we encourage you to seek alternative solutions, such as Gitleaks.

secret-shield Build Status

  1. 🚨 Are you being blocked from committing? 🚨
  2. About secret-shield
  3. Install
  4. Requirements
    1. Quick Install and setup
    2. Manual Install
    3. Uninstalling
  5. Using secret-shield
    1. Setting up automated searching
    2. Manually searching
    3. Additional functionality
  6. FAQ:
    1. Does secret-shield auto-update?
    2. Why secret-shield?
    3. What does it search for, exactly?
    4. It says it found a secret but it’s not, what do I do?
    5. It says it found a secret and it actually is a secret, what do I do?
    6. Does it work with my git client?
    7. I already have pre-commit hooks for something else, will secret-shield work with them?
      1. Husky + lint-staged integration

🚨 Are you being blocked from committing? 🚨

Some repositories require secret-shield to commit to them. If you don't have secret-shield you'll be blocked from committing to those repos

If secret-shield is installed on your machine, read this for common issues with secret-shield and how to fix them.

About secret-shield

secret-shield is a convenient way to protect against inadvertently committing potential secrets to GitHub. It can be set up to automatically run before each commit (if it catches something, it will stop the commit and ask you to review the findings), or you can manually run it from the command line.

Secret-shield in action

Please note: secret-shield will now be required when working with certain repositories. Learn more.

If you want to add secret-shield to your repository, take a look here.

Install

Requirements

secret-shield is a Node project tested with Node 10 & 12

secret-shield requires npm >= 6 to install globally. Previous versions of npm will not correcly install the required dependencies.

Quick Install and setup

npm install -g @mapbox/secret-shield

You still need to set it up. Easiest way: secret-shield --add-hooks global.

Manual Install

Clone this repository, then from inside it, run:

npm install
npm link

Tests are available using npm test.

You still need to set it up. Easiest way: secret-shield --add-hooks global.

Uninstalling

To uninstall secret-shield:

  1. make sure that you don't have any global hooks configured for secret-shield: secret-shield --remove-hooks global
  2. uninstall secret-shield normally (if you installed using npm, you can run npm remove -g @mapbox/secret-shield)

Using secret-shield

You can set up secret-shield to automatically search for secrets before each commit, or manually run it from anywhere.

Setting up automated searching

secret-shield uses pre-commit hooks to run before each commit and check for secrets only in whatever changes you've made. You can create these hooks either globally or on a per-repository basis.

  • To create a global pre-commit hook that will run on all repositories, run secret-shield --add-hooks global (to remove, use --remove-hooks global)
  • To create pre-commit hooks only for your current repository, run secret-shield --add-hooks local (to remove, use --remove-hooks local). Note that, if working with others, it's almost always a better idea to install secret-shield directly in your repository so that everyone who works on it uses secret-shield.

If a potential secret is found, secret-shield will abort the commit and provide you with its findings. After reviewing the findings, you can either go back and change your files or force the commit through without any checks by running git commit with the --no-verify flag.

Manually searching

You can manually use secret-shield to search through:

  • Files: secret-shield <--file|-f> <file>
  • Directories: secret-shield <--directory|-d> <directory>
  • Repositories: secret-shield <--repository|-r> <repository> [branch]
  • Strings: secret-shield <--string|-s> <string>
  • CloudFormation template files: automatically detected

Use <--redact|-R> [number] if you need to redact potentially sensitive information: output will be truncated to the specified number of characters.

Additional functionality

FAQ:

Does secret-shield auto-update?

Yes, secret-shield will automatically check for updates on average once in every 20 runs as a pre-commit hook.

Why secret-shield?

Credential leaks are frequently a problem in any organization or team. Secret-shield aims to nullify the impact of a credential leak by blocking it before the secret has a chance to get out in the first place.

When combined with documented best practices for handling secrets, secret-shield can dramatically reduce the probability of a leaked secret. Secret-shield can find already-leaked secrets much faster than searching by hand -- when combined with a robust incident response framework, this can significantly reduce the impact of a credential leak.

Security researchers can use secret-shield to find and report leaked secrets to affected teams or organizations.

What does it search for, exactly?

By default, secret-shield performs a minimal search: AWS client IDs, Mapbox secure keys, Slack tokens, and GitHub tokens.

If you perform more advanced searches, secret-shield can look for more things, such as AWS secret IDs, β€œdon’t commit” messages, and high-entropy strings.

It says it found a secret but it’s not, what do I do?

If it ran automatically before a commit, simply commit with the --no-verify flag. It won’t prompt you about those findings again, unless you change something in those lines.

If you ran it manually, you can ignore the findings.

It says it found a secret and it actually is a secret, what do I do?

If it ran automatically before a commit and the secret wasn’t there before, simply go back and remove the secret: it didn’t commit it, so you’re safe.

If the secret was there before (so it’s already been committed), or if you ran it manually and it found a secret that’s already been committed, the secret should be considered compromised -- follow your company or project's procedures for handling leaked secrets.

Does it work with my git client?

Secret-shield uses pre-commit hooks; some clients support them, others just force commits through regardless. You should check your client’s documentation on whether it supports pre-commit hooks.

I already have pre-commit hooks for something else, will secret-shield work with them?

Yes! Secret-shield will automatically detect any local hooks that you have, e.g. husky, and run them instead. If you want to run secret-shield on that repository, you should add it to those local hooks. Take a look here.

Husky + lint-staged integration

You can use secret-shield with husky and lint-staged by adding secret-shield as an npm dependency in your package.json and using the following configuration

  "husky": {
    "hooks": {
      "pre-commit": "lint-staged && secret-shield --check-and-run"
    }
  },
  "lint-staged": {
    "*": [
      "command2",
      "command2"
    ]
  }

If you are using an old version of husky

  "scripts": {
    "precommit": "lint-staged && secret-shield --check-and-run"
  },
  "lint-staged": {
    "*": [
      "command2",
      "command2"
    ]
  }

secret-shield's People

Contributors

dependabot[bot] avatar elfakyn avatar esrefdurna avatar guptabless avatar hassank-mapbox avatar iamgreut avatar jlin12358 avatar npeternel avatar thibaudlopez avatar vsmart avatar whyvez avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

secret-shield's Issues

Error on local hooks

What repository are you running secret-shield on?

tried on multiple mapbox repositories

What version of secret-shield are you running?

There is no -v flag, so I'm not sure

How did you invoke secret-shield?

`secret-shield --add-hooks local`

How is secret-shield not working as expected?

Errors with :

Could not find the local hooks to add -- is your secret-shield module correctly installed?

First reported by @chezzdev and I can reproduce the issue.

cc @zmully @k-mahoney @emilymcafee

ERROR! You must have secret-shield installed and configured globally...

What repository are you running secret-shield on?

https://github.com/mapbox/subdomain-docs/

What version of secret-shield are you running?

{
  "version": "1.0.2",
  "node": "v12.20.0",
  "os_platform": "darwin",
  "os_release": "20.4.0",
  "installed_dir": "/Users/domlet/.nvm/versions/node/v12.20.0/lib/node_modules/@mapbox/secret-shield/bin"
}

How did you invoke secret-shield?

I followed these steps:

➜  subdomain-docs git:(db-redirectstyles) βœ— npm install -g @mapbox/secret-shield
npm WARN deprecated [email protected]: This module is now under the @mapbox namespace: install @mapbox/decrypt-kms-env instead
npm WARN deprecated [email protected]: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
npm WARN deprecated [email protected]: Backported compatibility to node > 6
/Users/domlet/.nvm/versions/node/v12.20.0/bin/secret-shield -> /Users/domlet/.nvm/versions/node/v12.20.0/lib/node_modules/@mapbox/secret-shield/bin/secret-shield.js
+ @mapbox/[email protected]
updated 1 package in 6.161s
➜  subdomain-docs git:(db-redirectstyles) βœ— secret-shield --add-hooks global
Successfully added global hooks!

I also followed the instructions at:

...but I still cannot commit to this repo:

➜  subdomain-docs git:(db-redirectstyles) βœ— git commit -m "update redirects for GSG edition"
husky > pre-commit (node v12.20.0)
/bin/sh: Studio: command not found
ERROR! You must have secret-shield installed and configured globally to commit to this repository. To set up secret-shield, follow these instructions: https://github.com/mapbox/secret-shield/blob/master/docs/enabledBadge.md
husky > pre-commit hook failed (add --no-verify to bypass)

How is secret-shield not working as expected?

I cannot commit. When I try:

➜  subdomain-docs git:(db-redirectstyles) βœ— git commit -m "update redirects for GSG edition"

I get this error:

husky > pre-commit (node v12.20.0)
/bin/sh: Studio: command not found
ERROR! You must have secret-shield installed and configured globally to commit to this repository. To set up secret-shield, follow these instructions: https://github.com/mapbox/secret-shield/blob/master/docs/enabledBadge.md
husky > pre-commit hook failed (add --no-verify to bypass)

Common Falsepositive with submodule update

What repository are you running secret-shield on?

Multiple mapbox repos

What version of secret-shield are you running?

1.0.2

How did you invoke secret-shield?

pre-commit git hook

How is secret-shield not working as expected?

In projects with submodules where submodules are often changed and developed in parallel with the parent project there is common thing when submodule is in modified state.
Then secret-shield wrongly detects GitHub token secret in Subproject commit * string, e.g.
image
.

I've been able to disable this false positive by adding

    {
      "type": "remove",
      "name": "Ignore submodule commits",
      "pattern": "\\bSubproject commit [0-9a-zA-Z]+\\b"
    },

entry to the "preprocess": section of the json rule. Though it seem every rule will need to be modified similarly to avoid this falsepositive.

Initially I'd expected that it could be solved by just adding Submodule commit string to the falsePositive json here, as docs state here string should be ignored completely https://github.com/mapbox/secret-shield/blob/main/docs/writingRules.md#bulk-ignore. But it didn't work for me, and from the source code it seems that items from falsePositive json are only replaced in the string so it works differently, not ignoring whole string but just the mathching part. Not sure if that's intended of a bug and should be fixed.

SourceTree issue

What repository are you running secret-shield on?

https://github.com/mapbox/1tap-android

What version of secret-shield are you running?

{
"version": "1.0.2",
"node": "v12.22.1",
"os_platform": "darwin",
"os_release": "20.6.0",
"installed_dir": "/usr/local/lib/node_modules/@mapbox/secret-shield/bin"
}

How did you invoke secret-shield?

with AndroidStudio terminal works as expected
but with SourceTree I get error

How is secret-shield not working as expected?
SourceTree output:
/Users/nikitos/.git/hooks/pre-commit/pre-commit: line 2: secret-shield: command not found

➜  ~ cat .git/hooks/pre-commit/pre-commit
#!/bin/sh
secret-shield --pre-commit -C verydeep --enable "Mapbox Public Key" --disable "High-entropy base64 string" "Short high-entropy string" "Long high-entropy string"

Secret-shield throws exception

What repository are you running secret-shield on?

Mapbox private repo.

What version of secret-shield are you running?

How did you invoke secret-shield?

pre-commit

How is secret-shield not working as expected?

When I invoke command secret-shield --add-hooks global I got an error:

internal/modules/cjs/loader.js:1102
      throw new ERR_REQUIRE_ESM(filename, parentPath, packageJsonPath);
      ^

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /home/pavel/.local/share/fnm/node-versions/v14.20.1/installation/lib/node_modules/@mapbox/secret-shield/node_modules/p-queue/dist/index.js
require() of ES modules is not supported.
require() of /home/pavel/.local/share/fnm/node-versions/v14.20.1/installation/lib/node_modules/@mapbox/secret-shield/node_modules/p-queue/dist/index.js from /home/pavel/.local/share/fnm/node-versions/v14.20.1/installation/lib/node_modules/@mapbox/secret-shield/searchRepo.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /home/pavel/.local/share/fnm/node-versions/v14.20.1/installation/lib/node_modules/@mapbox/secret-shield/node_modules/p-queue/package.json.

    at new NodeError (internal/errors.js:322:7)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1102:13)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:101:18)
    at Object.<anonymous> (/home/pavel/.local/share/fnm/node-versions/v14.20.1/installation/lib/node_modules/@mapbox/secret-shield/searchRepo.js:3:16)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32) {
  cod
```e: 'ERR_REQUIRE_ESM'
}

Used node v14.20.1.

Update Node versions

Readme still recommends Node 4,6,8 - all of which have reached EOL πŸ˜“

We should upgrade this repo.

secretShield.checkAndRun() broken, prevents committing to repo

What repository are you running secret-shield on?

Private repo

What version of secret-shield are you running?

Unknown:

$ secret-shield --version
Unknown option: undefined
Usage: secret-shield <--string|--file|--directory|--repository> target
For detailed help run secret-shield --help or check the documentation.

How did you invoke secret-shield?

pre-commit via Husky - looks like it's running

#!/usr/bin/env node
'use strict';
const secretShield = require('@mapbox/secret-shield');
process.exit(secretShield.checkAndRun('2018-07-01'));

How is secret-shield not working as expected?

➜ my-repo (update-stuff) βœ— git commit -m 'Updates to stuff, remove dependency'
husky > pre-commit (node v8.10.0)
/bin/sh: Text.app/Contents/SharedSupport/bin:~/Applications/Sublime: No such file or directory
ERROR! You must have secret-shield installed and configured globally to commit to this repository. To set up secret-shield, follow these instructions: https://github.com/mapbox/secret-shield/blob/master/docs/enabledBadge.md
husky > pre-commit hook failed (add --no-verify to bypass)

⌦ my-repo (update-stuff) βœ— npm i -g @mapbox/secret-shield
/Users/andrewevans/.nvm/versions/node/v8.10.0/bin/secret-shield -> /Users/andrewevans/.nvm/versions/node/v8.10.0/lib/node_modules/@mapbox/secret-shield/bin/secret-shield.js
+ @mapbox/[email protected]
added 225 packages in 4.847s

➜ my-repo (update-stuff) βœ— git commit -m 'Updates to stuff, remove dependency'
husky > pre-commit (node v8.10.0)
/bin/sh: Text.app/Contents/SharedSupport/bin:~/Applications/Sublime: No such file or directory
ERROR! You must have secret-shield installed and configured globally to commit to this repository. To set up secret-shield, follow these instructions: https://github.com/mapbox/secret-shield/blob/master/docs/enabledBadge.md
husky > pre-commit hook failed (add --no-verify to bypass)

⌦ my-repo (update-stuff) βœ— which secret-shield
/Users/andrewevans/.nvm/versions/node/v8.10.0/bin/secret-shield

➜ my-repo (update-stuff) cat ~/.gitconfig | grep hooksPath
  hooksPath = /Users/andrewevans/.nvm/versions/node/v9.5.0/lib/node_modules/@mapbox/secret-shield/config/hooks

➜ my-repo (update-stuff) βœ— git commit -m 'Updates to stuff, remove dependency' --no-verify
[update-stuff 3551437] Updates to stuff, remove dependency
 3 files changed, 4522 insertions(+), 7 deletions(-)
 create mode 100644 package-lock.json

I don't know what it wants, so I gave up and used --no-verify . Error messages are totes bizarre. I expect PATH issues since I'm using nvm , but I have no idea why it would be checking Text.app/Contents... - that is for sure not on my PATH

Error: Cannot find module 'command-line-args'

What repository are you running secret-shield on?

Private repo

What version of secret-shield are you running?

npm list -g throws a bunch of errors like:

npm ERR! missing: acorn@^5.5.3, required by @mapbox/[email protected]
npm ERR! missing: command-join@^2.0.0, required by @mapbox/[email protected]
npm ERR! missing: command-line-args@^5.0.2, required by @mapbox/[email protected]
npm ERR! missing: d3-queue@^3.0.7, required by @mapbox/[email protected]
npm ERR! missing: decrypt-kms-env@^3.0.0, required by @mapbox/[email protected]
npm ERR! missing: es6-promise@^4.2.4, required by @mapbox/[email protected]
npm ERR! missing: esprima@^4.0.0, required by @mapbox/[email protected]
npm ERR! missing: fast-fuzzy@^1.5.0, required by @mapbox/[email protected]
npm ERR! missing: find-in-files@^0.5.0, required by @mapbox/[email protected]

1.0.2

How did you invoke secret-shield?

global pre-commit hook:

$ git commit -m 'Fix typo in cfn template'
module.js:471
    throw err;
    ^

Error: Cannot find module 'command-line-args'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/andrewevans/Projects/secret-shield/bin/secret-shield.js:9:14)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

$ git commit -m 'Fix typo' --no-verify
[github-apps-v2 c9059f2] Fix typo
 1 file changed, 1 insertion(+), 1 deletion(-)

$ node --version
v6.12.3

$ which secret-shield
/Users/agius/.nvm/versions/node/v6.12.3/bin/secret-shield

I have no idea what's going on with the hooks and nvm here, or even what the cwd is when running secret-shield

Prompt before pushing to a public repo

secret-shield introduces a layer of protection against accidentally committing secrets to public repos. What if we want to protect against accidentally committing to public repos at all?

Proposal

We should add a pre-push hook to the default secret-shield suites. This hook would:

  • Check if the repo is public.
  • Check if this user has never committed to this repo before. (optional: make this configurable!)

If both are true, the hook would prompt with a confirmation message.

> git push -u origin foo
WARNING!  This repository is PUBLIC, and you have never committed to this repo before.
Are you sure you want to push refs to '[email protected]:mapbox/secret-shield'?  y/N

cc/ @mapbox/security-and-compliance @ectrotter @tmpsantos

shield npm publish?

Are you likely to extend this to npm publish?

I'm embarrassed to admit I've accidentally leaked secret keys through npm not realising npm publish takes the whole directory including files not staged!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.