stackexchange / blackbox Goto Github PK
View Code? Open in Web Editor NEWSafely store secrets in Git/Mercurial/Subversion
License: MIT License
Safely store secrets in Git/Mercurial/Subversion
License: MIT License
We run into this rare enough we've been able to do it by hand, but it very quickly confuses anyone who hasn't done it properly before
The procedure is something like:
The idea is that one could use this as a git-mergetool strategy, wrapping their 'favorite' merge program.
There are probably some failure cases, etc, I'm forgetting to cover.
Currently there is a single line depending on python. It would be nice to remove this unnecessary dependency entirely or make it optional.
Hi i'm looking for a way to use git with encrypt files without losing diff tools. Is the blackbox my solution? after i made git pull and decrypt i will be able to see merge conflict in case of existing ones?
Hello,
We have a workflow requirement that consists of having all of the encrypted files unlocked, and then relocked (but not necessarily with changes).
It appears that blackbox_postdeploy
is intended to provide the "unlock all" requirement, however I don't see a mirroring "relock all" command.
Is this something supported, or how would you recommend accomplishing this?
I run blackbox_register_new_file foo
, and after it does the dance and the file is encrypted and added to the repo with a commit automatically, git-repo-root-directory/.gitignore
is updated as expected, but it isn't actually committed in the same automatic commit that foo.gpg was.
I work around it by git commit --amend .gitignore
, but it would be better if the .gitignore change was included in the automatic * registered in blackbox: path/to/foo
commit
Is that intended? If it's no longer part of the blackbox, it should show up as a file that isn't checked in.
I am using Blackbox's Production release (343b85a) on Cygwin on Windows 8 and going through the first time setup.
When running the addadmin function, I get an "ERROR: Unknown OS. Exiting.".
u0044975@U0044975-TPL-C /cygdrive/c/code/meetingtimesuggester
$ blackbox_addadmin [email protected]
ERROR: Unknown OS. Exiting.
u0044975@U0044975-TPL-C /cygdrive/c/code/meetingtimesuggester
$ uname -s
CYGWIN_NT-6.2
u0044975@U0044975-TPL-C /cygdrive/c/code/meetingtimesuggester
Using 6765988 - any attempt to get initialize a repository fails. Works fine using a much older version
$ blackbox_initialize
Enable blackbox for this git repo? (yes/no) yes
ERROR: is not a blackbox Repo
People want to be able to do:
export EDITOR="subl -nw"
Forgetting quote variables in shell script lead to many security implications. I took a look into some blackbox scripts and found some issue.
Example in blackbox_addadmin:
KEYNAME="$1"
: ${KEYNAME:?ERROR: First argument must be a keyname (email address)} ;
It looks no harm but if someone call blackbox_addadmin
like:
blackbox_addadmin '/*/*/*/*/../../../../*/*/*/*/../../../../*/*/*/*'
then it push server to heavy load, even if mightiest server. Imaging some command like this ran and what happen with server.
I have made a pull request to fix this, you can look here
"make confidence" uses gpg-agent so that the password doesn't have to be entered over and over.
However, it does ask for Alice's password once after switching Bob->Alice and vice-versa. It isn't clear to me what about become_alice and become_bob is broken that makes that happen.
It would be great if it only asked for Alice's password exactly once, and Bob's password exactly once.
'make confidence' unconditionally cd's to ~/gitwork/blackbox which may not be where the source is installed.
It also sets (rather then modifies) $PATH which may be a portability problem (though it seems to work on MacOS+Homebrew).
========== Encrypting: DONE
========== COMMITING TO HG:
abort: no repository found in '/current/repo/path' (.hg not found)!
Very few people use MacPorts anymore. It would be nice to see support for installing blackbox via HomeBrew.
You'll have to refactor the blacbox_home variable to be externally set, or be able to not put the underscore scripts in the bin
directory, that's a deal breaker for HomeBrew, they don't want any scripts going into bin
that aren't called directly.
This makes it difficult/impossible to use blackbox_cat
in a pipeline (for which I have a use case), and violates the expectation of 'cat'. Sending such diagnostics to stderr seems like a better plan - if people agree I'll try to produce an appropriate pull request.
I assumed it deletes all the files that were decrypted by post_deploy
, but it seems it doesn't work that way? Or did I mess something up?
Here's the output when I try to run it (notice no files are listed that will be shredded):
$ blackbox_shred_all_files
========== ENCRYPTED FILES THAT WERE UNLOCKED:
myproject/encrypted.py.gpg
myproject/resources/secrets.json.gpg
========== FILES THAT WILL BE SHREDDED:
========== DONE.
Last release was more than a month ago, and there are quite a few changes included.
I know I can clone the repository and use the bin folder, or even store the artifacts somewhere else, but I need some of the bug fixes and having a new release in https://github.com/StackExchange/blackbox/releases would be awesome.
Is there any idea when that will happen?
Do we have any guidelines or best-practices on how to best protect secrets in releases, such as in a .war file? one approach that comes to mind is encrypting the releases as well, such as creating an encrypted war file, however, it will involve the release management personnel be added as blackbox admins.
gpg has a --use-agent flag which uses the gpg agent if available. If it's not it does nothing, so users that don't care are not affected.
Please consider adding functionality along the lines of "blackbox_encrypt_all_files". We currently have:
However, there seems to be no function that would allow to encrypt all "blackboxed" files in the repository en masse. I understand that this is only a convenience functionality, as one can simply call blackbox_edit_end for each file before commiting. It would however great simply the workflow.
The mktemp
command in GNU coreutils, when given a -t argument, expects it to contain at least three consecutive X
characters
It looks like you added a fix for this in 56232aa then subsequently clobbered it in 02700b5
make_self_deleting_tempfile
and create_self_deleting_tempfile
(NB: Consulting manual pages on both MacOS and Linux suggests that both OSes will accept -t _stacklib_.XXXXX
so there should be no need here to switch on uname
)
If a key for a user listed in blackbox-admins.txt is not found then the re-encrypt fails leaving the unencrypted file on disk without any working.
Fixes that I can think of (not necessarily the only ones)
How to reproduce:
blackbox_edit foo.txt bar.txt
should iterate over all the files.
Currently blackbox_postdeploy prompts for the passphrase once per file. Would be good to have an option to prompt it only once.
It would be convenient to just be able to type "blackbox_edit FILE" and not have to think about the entire process:
Is the file encrypted already?
No: Ask "are you sure?" and then blackbox_register_new_file.
Yes: decrypt.
Call $EDITOR
Was "-d" option specified?"
Yes: Re-encrypt and delete the plaintext version.
No: Re-encrypt and leave the plaintext version alone. Output instructions on how to safely shred it.
GnuPG 2.1 uses by default a new keyring format. When a new admin is added to the system, its key is imported. But if no pubring.gpg is found, gpg defaults to the new keybox format and creates a pubring.kbx keybox file. When blackbox_addadmin
is called, expects a file named pubring.gpg
, see:
https://github.com/StackExchange/blackbox/blob/master/bin/blackbox_addadmin#L48
But if the repo was initialized with gpg 2.1, it fails to add the file to the vcs. With git, for example, you'll get this error:
fatal: pathspec '/home/user/project/keyrings/live/pubring.gpg' did not match any files
For more info about the new keyring format see:
I did a gem install fpm, then make packages-deb and I got an AMD64 deb.
$ ls ~/debbuild-stack_blackbox/
bin-packages.txt installroot stack-blackbox_1.0-1_amd64.deb
$ uname -m
x86_64.
Maybe this is a bug in fpm, but since you depend on it, it's your bug too!
Thanks.
This is more of a question than an issue but...
Can I host more than 1 blackbox in a repository?
From looking at the code (_blackbox_common.sh) it seems that if I set the env variable BLACKBOXDATA before invoking, I can create multiple blackboxes e.g.
export BLACKBOXDATA=keyrings/prod
(some blackbox commands)
export BLACKBOXDATA=keyrings/dev
(somre different blackbox commands)
etc. etc. Mostly, before I go using this feature, I want to insure that this functionality is intended vs. a happy accident that will be deprecated later.
Thanks!
blackbox_edit and blackbox_cat use other blackbox scripts internally, but assume that they will be found on $PATH. It's sometimes useful to be able to run these scripts by pathname without first putting them on $PATH, and all the other scripts work just fine in these circumstances.
I've created PR #56 to fix this.
Still trying to wrap my head around this problem.
After doing a git pull, running blackbox_register_new_file shows this error:
% echo nevermind > whatever.txt
% blackbox_register_new_file whatever.txt
gpg: whatever.txt: encryption failed: No public key
I double-checked that all public keys had subkeys which were not expired.
When I step back from the latest release to eab0418 the command works.
Adding "set -x" to the top of blackbox_register_new_file reveals this output:
+ gpg --use-agent --yes --trust-model=always --encrypt -o whatever.txt.gpg '-rRECIPIENT1 -rRECIPIENT2 -rRECIPIENT3' whatever.txt gpg: RECIPIENT1 -rRECIPIENT2 -rRECIPIENT3: skipped: No public key gpg: whatever.txt: encryption failed: No public key
In release eab0418 and before, output looks like this:
+ gpg --yes --trust-model=always --encrypt -o whatever.txt.gpg -rRECIPIENT1 -rRECIPIENT2 -rRECIPIENT3 whatever.txt
All the recipients are on the same line.
What baffles me is that the code in _blackbox_common.sh is the same between the two versions:
gpg --yes --trust-model=always --encrypt -o "$encrypted" $(awk '{ print "-r" $1 }' < "$BB_ADMINS") "$unencrypted"
Anybody else seeing this error?
We've had a number of git commits from Alice or Bob due to "make confidence" being overzealous about becoming Alice or Bob for the tests.
Surely this can be fixed.
This is a great project and I would love to be able to install it and keep it up to date via a package manager. It is great that you have it set up for antigen
, but not everyone uses zsh.
Please consider supporting installation via:
I am more than happy to help with this work, just let me know which direction(s) you prefer or which ones should be prioritized.
When trying to deploy via TeamCity with blackbox_postdeply, above error is emitted.
Setting up bash -x to give debug info, REPOBASE is being set as /dev/null. There isn't a .git subdirectory in the project directory.
There's manual steps for removing a file from blackbox but it would be far better to have a tool to do it so I know that I won't screw up. Is there a possibility of adding this feature?
A non-admin can gain access to the encrypted data as follows:
This can happen if a newbie runs blackbox_addadmin by mistake. This situation can be fixed by moving the change to blackbox-admins.txt out of blackbox_addadmin, and to the "step 2" part (which should be automated via a new command such as blackbox_acceptadmin).
This can happen maliciously but someone simply editing blackbox-admins.txt. However this is generally prevented by repo access controls. However, this assumes anyone with write access to the repo is trusted.
NOTE: This is safe as long as anyone with write access to the repo is also an admin (or soon to become an admit). This is the use case currently at Stack Exchange, but it should be fixed so other use-cases are safe.
Suggested solution: GPG sign the blackbox-admins.txt and only permit encryption/re-encryption to happen if the signature matches. Alternatively, I think keeping the blackbox-admins.txt file encrypted might be sufficient. I'll have to think about it.
The README file mentions that the way to remove a file from the system is to run blackbox_deregister_file
but this script isn't available in the latest release .tar.
Perhaps it's time to have a new release?
it should distinguish between the puppet directory and the blackbox install directory
when deploying independent environments, the current BASEDIR-definition assumes blackbox to be installed into every environment's bin directory
(alternatively assume blackbox/bin is in the path, ie. simply delete line 11)
Currently blackbox is not aware of blackbox enabled git submodules. The end result is a behavior that blackbox_postdeploy does not attempt to unpack files in submodules of a repository.
The work around for this currently is to have a deploy agent call blackbox on each submodule, or to register files a second time in the main repo of a given project; Both of which are less than ideal.
A nice feature would be to have blackbox_postdeploy use the .gitmodule file to check for registered modules and attempt to unpack any files registered to that submodules project.
The use of gpg-agent seems to be broken on Mac OS X.
become_alice and become_bob don't seem to work. The "a" and "b" passwords have to be entered over and over again.
Every time it asks for a password, it first prints:
can't connect to `/tmp/gpg-xhWU7m/S.gpg-agent:5356:1': No such file or directory
gpg: can't connect to `/tmp/gpg-xhWU7m/S.gpg-agent:5356:1': connect failed
gpg (GnuPG) 1.4.19
gpg-agent (GnuPG) 2.0.28
pinentry-curses (pinentry) 0.9.4
Calling blackbox_register_new_file fails, if the repository path contains spaces, eg:
========== Encrypting:
========== Encrypting: DONE
File "", line 1
import os ; print(os.path.relpath("/Users//Library/Application
^
SyntaxError: EOL while scanning string literal
In this case, the path contains in "Application Support" which includes spaces. I guess the issue is in get_unencrypted_filename?
blackbox_edit_start pops up a similar error message, but somehow still manages to decrypt the file and save it properly.
Is possible to tag a release version or a stable version? that would help creating an acceptable brew (http://brew.sh/) formula.
"make confidence" pauses to ask for passwords. It shouldn't.
GPG makes it very difficult to supply passphrase automatically... which is a very smart decision in general. However it makes automated testing a pain.
Maybe someone can make a very stupid pinentry clone that gives "a" as the password for alice, and "b" as the password for bob.
Currently, whenever a file is added to the registry, filename.txt.gpg
will be created and saved with 644 permissions. When it's decrypted, same story, 644. I'd like to be able to have whatever the permissions were at time of encryption persist through the process, so if I have something set to 600, it gets decrypted to 600.
I ran blackbox_addadmin key.asc
thinking I could do that to insert a public key. This failed, but, it still put key.asc in the blackbox-admins.txt file. This causes all further operations to fail because there is no key in the keyring with the name key.asc
.
I initialized blackbox on a branch of my project a couple of weeks ago, and now I want to finish setting it up.
Currently, in my .gitignore
file, there are these 2 lines, which were added automatically (though not sure if it was when I initialized or just recently):
pubring.gpg~
secring.gpg
Now I'm confused since the contents of keyrings/live/
directory are:
blackbox-admins.txt
blackbox-files.txt
pubring.gpg
secring.gpg
trustdb.gpg
Should I update my .gitignore to ignore the other .gpg
files in keyrings/live/
?
Add and commit should check to see if a unencrypted file is being added/committed when it is listed in the registry.
People have asked for per-file or per-subfolder access controls. I'm opening this ticket to ask for suggestions on how they'd like to see it implemented. What should it look like from the user's perspective?
@snowpong noticed that blackbox_decrypt_all_files does not work on Git for Windows. It fails with:
../blackbox/bin/blackbox_decrypt_all_files: line 19: gpg-agent: command not found
../blackbox/bin/blackbox_decrypt_all_files: line 22: exec: blackbox_postdeploy: not found
blackbox_register_new_file should add the file to .gitignore or .hgignore as appropriate.
I believe the syntax is different for each of those, so I'd appreciate pointers on what the format should be for each. In particular, we want to block path/path/foo but not path/path/foo.gpg (and not other/path/foo)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.