Showing preview only (343K chars total). Download the full file or copy to clipboard to get everything.
Repository: StackExchange/blackbox
Branch: master
Commit: 41e423410807
Files: 140
Total size: 313.1 KB
Directory structure:
gitextract_2zatdstu/
├── .circleci/
│ └── config.yml
├── .gitattributes
├── .github/
│ └── workflows/
│ ├── build.yml
│ └── release.yml
├── .gitignore
├── AUTHORS
├── CHANGELOG.md
├── DESIGN.md
├── LICENSE.txt
├── Makefile
├── README-v2.md
├── README.md
├── RELEASE_ENGINEERING.md
├── SPECS/
│ └── empty.spec
├── Version2-Ideas.md
├── bin/
│ ├── Makefile
│ ├── _blackbox_common.sh
│ ├── _blackbox_common_test.sh
│ ├── _stack_lib.sh
│ ├── blackbox_addadmin
│ ├── blackbox_cat
│ ├── blackbox_decrypt_all_files
│ ├── blackbox_decrypt_file
│ ├── blackbox_deregister_file
│ ├── blackbox_diff
│ ├── blackbox_edit
│ ├── blackbox_edit_end
│ ├── blackbox_edit_start
│ ├── blackbox_initialize
│ ├── blackbox_list_admins
│ ├── blackbox_list_files
│ ├── blackbox_listadmins
│ ├── blackbox_postdeploy
│ ├── blackbox_recurse
│ ├── blackbox_register_new_file
│ ├── blackbox_removeadmin
│ ├── blackbox_shred_all_files
│ ├── blackbox_update_all_files
│ ├── blackbox_view
│ └── blackbox_whatsnew
├── binv2/
│ ├── blackbox_addadmin
│ ├── blackbox_cat
│ ├── blackbox_decrypt_all_files
│ ├── blackbox_decrypt_file
│ ├── blackbox_deregister_file
│ ├── blackbox_diff
│ ├── blackbox_edit
│ ├── blackbox_edit_end
│ ├── blackbox_edit_start
│ ├── blackbox_initialize
│ ├── blackbox_list_admins
│ ├── blackbox_list_files
│ ├── blackbox_listadmins
│ ├── blackbox_postdeploy
│ ├── blackbox_register_new_file
│ ├── blackbox_removeadmin
│ ├── blackbox_shred_all_files
│ ├── blackbox_update_all_files
│ ├── blackbox_view
│ └── blackbox_whatsnew
├── blackbox.plugin.zsh
├── build/
│ └── build.go
├── cmd/
│ └── blackbox/
│ ├── blackbox.go
│ ├── cli.go
│ └── drive.go
├── docs/
│ ├── README.md
│ ├── admin-ops.md
│ ├── advanced.md
│ ├── alternatives.md
│ ├── backwards-compatibility.md
│ ├── compatibility.md
│ ├── dev-code-overview.md
│ ├── dev.md
│ ├── enable-repo.md
│ ├── encryption.md
│ ├── expired-keys.md
│ ├── file-ops.md
│ ├── full-command-list.md
│ ├── git-tips.md
│ ├── gnupg-tips.md
│ ├── installation.md
│ ├── role-accounts.md
│ ├── subversion-tips.md
│ ├── support.md
│ ├── user-overview.md
│ ├── why-is-this-important.md
│ ├── with-ansible.md
│ └── with-puppet.md
├── go.mod
├── go.sum
├── integrationTest/
│ ├── NOTES.txt
│ ├── README.txt
│ ├── asserts.go
│ ├── integration_test.go
│ ├── ithelpers.go
│ └── test_data/
│ ├── 000-admin-list.txt
│ ├── 000-file-list.txt
│ ├── 000-status.txt
│ ├── alice-cat-plain.txt
│ ├── basic-status.txt
│ ├── reencrypt-plain.txt
│ └── status-noreg.txt
├── models/
│ ├── crypters.go
│ └── vcs.go
├── pkg/
│ ├── bblog/
│ │ └── bblog.go
│ ├── bbutil/
│ │ ├── filestats.go
│ │ ├── rbio_test.go
│ │ ├── runbash.go
│ │ ├── shred.go
│ │ ├── sortedfile_test.go
│ │ ├── umask_posix.go
│ │ └── umask_windows.go
│ ├── box/
│ │ ├── box.go
│ │ ├── boxutils.go
│ │ ├── pretty_test.go
│ │ └── verbs.go
│ ├── commitlater/
│ │ └── commitlater.go
│ ├── crypters/
│ │ ├── _all/
│ │ │ └── all.go
│ │ ├── crypters.go
│ │ └── gnupg/
│ │ ├── gnupg.go
│ │ └── keychain.go
│ ├── makesafe/
│ │ ├── makesafe.go
│ │ └── makesafe_test.go
│ └── vcs/
│ ├── _all/
│ │ └── all.go
│ ├── git/
│ │ └── git.go
│ ├── none/
│ │ └── none.go
│ └── vcs.go
└── tools/
├── Portfile.template
├── auto_system_test
├── confidence_test.sh
├── macports_report_upgrade.sh
├── mk_deb_fpmdir
├── mk_deb_fpmdir.stack_blackbox.txt
├── mk_macports
├── mk_macports.vcs_blackbox.txt
├── mk_rpm_fpmdir
├── mk_rpm_fpmdir.stack_blackbox.txt
├── profile.d-usrblackbox-test.sh
├── profile.d-usrblackbox.sh
└── test_functions.sh
================================================
FILE CONTENTS
================================================
================================================
FILE: .circleci/config.yml
================================================
#version: 2
#
#workflows:
# version: 2
# build_and_test:
# jobs:
# - debian
# - ubuntu
#
#jobs:
#
# debian:
# docker:
# - image: debian:9.1
# steps:
# - checkout
# - run:
# name: 'Installing'
# command: |
# apt-get update -y
# apt-get install -y build-essential expect git gnupg2 pinentry-tty procps rpm ruby-dev libffi-dev
# gem install fpm
# - run:
# name: 'Cleaning'
# command: |
# rm -rf ~/.gpnupg
# make clean
# - run:
# name: 'Testing'
# command: |
# GPG=gpg2 make test
# make packages-deb
# make packages-rpm
#
# ubuntu:
# docker:
# - image: ubuntu:16.04
# steps:
# - checkout
# - run:
# name: 'Installing'
# command: |
# apt-get update -y
# apt-get install -y build-essential expect git gnupg2 pinentry-tty procps rpm ruby-dev libffi-dev
# gem install fpm
# - run:
# name: 'Cleaning'
# command: |
# rm -rf ~/.gpnupg
# make clean
# - run:
# name: 'Testing'
# command: |
# GPG=gpg2 make test
# make packages-deb
# make packages-rpm
================================================
FILE: .gitattributes
================================================
bin/** text eol=lf
tools/** text eol=lf
Makefile text eol=lf
Portfile text eol=lf
blackbox.plugin.zsh text eol=lf
================================================
FILE: .github/workflows/build.yml
================================================
name: build
on:
pull_request:
branches: [ master ]
push:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: ^1.15
- name: Build binaries
run: go run build/build.go
- name: Run unit tests
run: go test ./...
- name: Run integration tests
working-directory: integrationTest
run: umask 0027 ; rm -rf /tmp/bbhome-* && go test -long -nocleanup
================================================
FILE: .github/workflows/release.yml
================================================
on:
release:
types: [published]
name: release
jobs:
release:
name: release
runs-on: ubuntu-latest
steps:
- name: Get release
id: get_release
uses: bruceadams/get-release@v1.2.2
env:
GITHUB_TOKEN: ${{ github.token }}
- name: Checkout repo
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: ^1.15
- name: Build binaries
run: go run build/build.go
- name: Upload blackbox-Darwin
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.get_release.outputs.upload_url }}
asset_path: ./blackbox-Darwin
asset_name: blackbox-Darwin
asset_content_type: application/octet-stream
- name: Upload blackbox-Linux
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.get_release.outputs.upload_url }}
asset_path: ./blackbox-Linux
asset_name: blackbox-Linux
asset_content_type: application/octet-stream
- name: Upload blackbox.exe
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.get_release.outputs.upload_url }}
asset_path: ./blackbox.exe
asset_name: blackbox.exe
asset_content_type: application/octet-stream
================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
# C extensions
*.so
# Distribution / packaging
.Python
env/
develop-eggs/
dist/
eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.cache
nosetests.xml
coverage.xml
# Translations
*.mo
# Mr Developer
.mr.developer.cfg
.project
.pydevproject
# Rope
.ropeproject
# Django stuff:
*.log
*.pot
# Sphinx documentation
docs/_build/
# macOS
.DS_Store
# Blackbox
bbintegration
.*.swp
/integrationTest/.blackbox
# golang
/vendor/
================================================
FILE: AUTHORS
================================================
Tom Limoncelli <tlimoncelli@stackexchange.com>
================================================
FILE: CHANGELOG.md
================================================
Release v1.20220610
NOTE: I don't have a lot of time to commit to this project. I'd gladly accept help, especially
with improving the testing on various operating systems.
Major feature: macOS users rejoice! Incompatibility with macOS Monterey 12.3 is fixed! (#347)
* Add .gitattributes during repo initialization (#352)
* Update zgen reference to zgenom (#350)
* Improve test data generation (#348)
* Fix 'chmod' for macOS Monterey 12.3 (#347)
Release v1.20200429
NOTE: While there is now support for NetBSD and SunOS/SmartOS, the
release process only tests on macOS and CentOS7 because that's all I
have access to.
* Fix tools that break when ".." or "." are used in a path (#304)
* Respect PREFIX variable for copy-install (#294)
* Documentation: Add pkgsrc install instructions (#292)
* Improve support for Windows (#291)
* Clarify gpg version usage (#290)
* Many documentation fixes
* DOCUMENTATION: Promote 'getting started' to a section, enumerate steps (#283)
* Commit changes to gitignore when deregistering (#282)
* Add support for NetBSD and SunOS (SmartOS)
* Defend against ShellShock
Release v1.20181219
* New OS support: Add support for NetBSD and SunOS (SmartOS)
* Testing: Improve confidence test.
* .blackbox is now the default config directory for new repos. (#272)
* Add blackbox_decrypt_file (#270)
* Improved compatibility: change"/bin/[x]" to "/usr/bin/env [x]" (#265)
* Add blackbox_less. (#263)
* add nix method of install (#261)
* Linked setting up of GPG key (#260)
Release v1.20180618
* Restore `make manual-install` with warning. (#258)
Release v1.20180615
* Standardize on .blackbox for config. Use keyrings/live for backwards compatibility.
* Store keys in .blackbox directory (#218)
* Suggest committing changes to pubring.gpg when running blackbox_removeadmin (#248)
* Fix typo (#246)
* Improve installation instructions (#244)
* Fix replacing-expired-keys link in README (#241)
* Fix problems when gpg2 is installed next to gpg (#237)
* Many documentation corrections, updates, etc.
* Exclude default keyring from import (#223)
* .gitattributes not always updated (PR#146)
* Fix bugs related to updating .gitattributes (PR#146)
* Update readme with CircleCI link (#216)
* Run the tests on a CI (#215)
* Fixed Alpine compatibility (chmod) (#212)
* direct repobase message to stderr (#204)
* Improve Windows compatibility
* NEW: .gitattributes Set Unix-only files to eol=lf
* Silence 'not changed' output during keychain import (#200)
* Improve FreeBSD compatibility
* shred_file() outputs warning message to stderr. (#192)
* Don't complain about GPG_AGENT_INFO if using newer gpg-agent (#189)
* [FreeBSD] Fix use of chmod (#180)
* Requiring a file to be entered to finish editing (#175)
* Remove the key from the keyring when removing an admin (#173)
* Add FreeBSD support (#172)
* Add list admins commandline tool. (#170)
ignore backup files and secring.gpg in $BLACKBOXDATA (#169)
Allow parallel shredding of files (#167)
* Add/improve Mingw support
* Make "make confidence" less fragile
* And a lot, lot more.
Release v1.20170309
* "make test" is an alias for "make confidence"
* macOS: make_tempdir must create shorter paths
* Fix "make confidence" for newer version of Git
* README.md: Add info about our new mailing list
Release v1.20170611
* confidence_test.sh verifies external tools exist
* confidence_test.sh more reliable for non-UTF8 users
* "make test" no longer prompts for passwords
* blackbox works better when target directory lives in root (#194)
* Add confidence_test.sh tests for admin operations
* blackbox_list_admins fails (#193)
* confidence_test.sh works better on FreeBSD
* tools/confidence_test.sh: now works with gnupg-2.0 and gnupg-2.1
* Blackbox now officially supports both gnupg-2.0 and gnupg-2.1
* blackbox_shred_all_files: BUGFIX: Does not shred files with spaces
* blackbox_removeadmin: disable gpg's confirmation
* Sync mk_rpm_fpmdir from master
Release v1.20170127
* Starting CHANGELOG.
================================================
FILE: DESIGN.md
================================================
BlackBox Internals
==================
The goal of the Go rewrite is to improve the usability and
maintainability of Blackbox, meanwhile make it easier to implement new
The system is built in distinct layers: view, controller, model.
Suppose there is a subcommand "`foo`". `blackbox.go` parses the
user's command line args and calls `cmdFoo()`, which is given
everything it needs to do the operation. For example, it is given the
filenames the user specified exactly; even if an empty list means "all
files", at this layer the empty list is passed to the function.
`cmdFoo()` contains the business logic of how the operation should be
done: usually iterating over filenames and calling verb(s) for each
one. For example if an empty file list means "all files", this is the
layer that enumerates the files.
`cmdFoo()` is implemented in the file `cmd_foo.go`. The caller of
`cmdFoo()` should provide all data it needs to get the job done.
`cmdFoo()` doesn't refer to global flags, they are passed to the
function as parameters. Therefore the function has zero side-effects
(except possibly logging) and can be called as library functions by
other systems. This is the external (binary) API which should be
relatively stable.
`cmdFoo()` calls verbs that are in `bbutil/`. Some of those verbs are
actually interfaces. For example, any VCS-related verbs are actually a
Go interface which might be implemented one of many ways (Git,
Subversion, Mercurial), GPG-functions may be implemented by shelling
out to `gpg.exe` or by using Go's gpg library.
They layers look like this:
| View | `blackbox.go` | Parses User Commands, calls controller |
| Controller | `cmd_*.go` | The business logic. Iterates and calls verbs |
| Model | `pkg/bbutil` | Verbs |
| Interfaces | `pkg/*` | Interfaces and their implementations |
At least that's the goal. We'll see how well we can achieve this.
Version 2.0
===========
Software architecture.
We try to keep the command-line parsing separate from the business
logic and all plug-ins. This keeps things clean and easy to refactor.
In fact layer 2 could be used as a stand-alone module for projects
that want to embed blackbox actions.
Layer 1: The command itself
* cmd/blackbox/blackbox.go -- main() not much more
* cmd/blackbox/cli.go -- Set up and call the ufave/cli flag parser
* cmd/blackbox/drive.go -- Check # of arguments, conflicting flags, and then call the businss logic layer
Layer 2: The business logic
* pkg/box/box.go -- The interface to accessing .blackbox (admins, files, etc.)
* pkg/box/verbs.go -- Verbs called by Layer 1. Just the verbs
* pkg/box/boxutils.go -- Functions needed by the verbs
Layer 3: The plug-ins
* pkg/vcs/... -- Plug-ins for Git, (Mercurial, Subversion, Perforce,) and None
* pkg/crypters/... -- Plug-ins for PGP access: GnuPG, (go-openpgp, others in the future)
Layer 4: Support functions for use by Layer 3
* pkg/bbutil/filestats.go -- File manipulations
* pkg/bbutil/runbash.go -- Safely run external Linux commands
================================================
FILE: LICENSE.txt
================================================
The MIT License (MIT)
Copyright (c) 2014-2021 Stack Exchange, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
================================================
FILE: Makefile
================================================
SHELL=/bin/sh
PREFIX?=/usr/local
PKGNAME=stack_blackbox
BASEDIR?=$(HOME)
OUTPUTDIR?="$(BASEDIR)/debbuild-${PKGNAME}"
all:
@echo 'Menu:'
@echo ' make update Update any generated files'
@echo ' make packages-rpm Make RPM packages'
@echo ' make packages-deb Make DEB packages'
@echo ' make symlinks-install Make symlinks in ${PREFIX}/bin/'
@echo ' make copy-install Copy "bin" files to ${PREFIX}/bin/'
@echo ' make copy-uninstall Remove blackbox files from ${PREFIX}/bin/'
@echo ' make test Run tests'
install:
@echo 'To install, copy the files from bin to somewhere in your PATH.'
@echo 'The README.md document gives more details.'
@echo 'Or run "make" (with no options) for more info.'
# The default package type is RPM.
packages: packages-rpm
#
# RPM builds
#
# NOTE: mk_rpm_fpmdir.stack_blackbox.txt is the master list of files. All
# other packages should generate their list from it.
packages-rpm:
cd tools && PKGRELEASE="$${PKGRELEASE}" PKGDESCRIPTION="Safely store secrets in git/hg/svn repos using GPG encryption" ./mk_rpm_fpmdir stack_blackbox mk_rpm_fpmdir.stack_blackbox.txt
packages-rpm-debug:
@echo BUILD:
@PKGRELEASE=99 make packages
@echo ITEMS TO BE PACKAGED:
find $(BASEDIR)/rpmbuild-$(PKGNAME)/installroot -type f
@echo ITEMS ACTUALLY IN PACKAGE:
@rpm -qpl $$(cat $(BASEDIR)/rpmbuild-$(PKGNAME)/bin-packages.txt)
local-rpm:
@PKGRELEASE=1 make packages
-@sudo rpm -e $(PKGNAME)
sudo rpm -i $$(cat $(BASEDIR)/rpmbuild-$(PKGNAME)/bin-packages.txt)
lock-rpm:
sudo yum versionlock add $(PKGNAME)
unlock-rpm:
sudo yum versionlock clear
#
# Manual install
#
symlinks-install:
@echo "Symlinking files from ./bin to ${PREFIX}/bin"
@cd bin && for f in `find . -type f -iname "*" ! -iname "Makefile"`; do ln -fs `pwd`/$$f $(PREFIX)/bin/$$f; done
@echo 'Done.'
manual-install:
@echo '***************************************************************'
@echo '* DEPRECATED *'
@echo '* `make manual-install` is now called `make symlinks-install` *'
@echo '***************************************************************'
$(MAKE) symlinks-install
copy-install:
@echo "Copying files from ./bin to ${PREFIX}/bin"
@cd bin && for f in `find . -type f -iname "*" ! -iname "Makefile"`; do cp `pwd`/$$f $(PREFIX)/bin/$$f; done
@echo 'Done.'
copy-uninstall:
@echo "Removing blackbox files from ${PREFIX}/bin"
@cd bin && for f in `find . -type f -iname "*" ! -iname "Makefile"`; do rm $(PREFIX)/bin/$$f; done
@echo 'Done.'
#
# DEB builds
#
packages-deb: tools/mk_deb_fpmdir.stack_blackbox.txt
cd tools && OUTPUTDIR=$(OUTPUTDIR) PKGRELEASE="$${PKGRELEASE}" PKGDESCRIPTION="Safely store secrets in git/hg/svn repos using GPG encryption" ./mk_deb_fpmdir stack_blackbox mk_deb_fpmdir.stack_blackbox.txt
# Make mk_deb_fpmdir.vcs_blackbox.txt from mk_rpm_fpmdir.stack_blackbox.txt:
tools/mk_deb_fpmdir.stack_blackbox.txt: tools/mk_rpm_fpmdir.stack_blackbox.txt
sed -e '/^#/d' -e 's@/usr/blackbox/bin/@/usr/bin/@g' -e '/profile.d-usrblackbox.sh/d' <tools/mk_rpm_fpmdir.stack_blackbox.txt >$@
packages-deb-debug: tools/mk_deb_fpmdir.stack_blackbox.txt
@echo BUILD:
@PKGRELEASE=99 make packages-deb
@echo ITEMS TO BE PACKAGED:
find ~/debbuild-$(PKGNAME)/installroot -type f
@echo ITEMS ACTUALLY IN PACKAGE:
@dpkg --contents $$(cat $(BASEDIR)/debbuild-$(PKGNAME)/bin-packages.txt)
local-deb:
@PKGRELEASE=1 make packages
-@sudo dpkg -e $(PKGNAME)
sudo dpkg -i $$(cat $(BASEDIR)/rpmbuild-$(PKGNAME)/bin-packages.txt)
#
# MacPorts builds
#
# To test:
# rm -rf /tmp/foo ; mkdir -p /tmp/foo;make packages-macports DESTDIR=/tmp/foo;find /tmp/foo -ls
# Make mk_macports.vcs_blackbox.txt from mk_rpm_fpmdir.stack_blackbox.txt:
tools/mk_macports.vcs_blackbox.txt: tools/mk_rpm_fpmdir.stack_blackbox.txt
sed -e '/^#/d' -e 's@/usr/blackbox/bin/@bin/@g' -e '/profile.d-usrblackbox.sh/d' <tools/mk_rpm_fpmdir.stack_blackbox.txt >$@
# MacPorts expects to run: make packages-macports DESTDIR=${destroot}
packages-macports: tools/mk_macports.vcs_blackbox.txt
mkdir -p $(DESTDIR)/bin
cd tools && ./mk_macports mk_macports.vcs_blackbox.txt
# stow is a pretty easy way to manage simple local installs on GNU systems
install-stow:
mkdir -p /usr/local/stow/blackbox/bin
cp bin/* /usr/local/stow/blackbox/bin
rm /usr/local/stow/blackbox/bin/Makefile
cd /usr/local/stow; stow -R blackbox
uninstall-stow:
cd /usr/local/stow; stow -D blackbox
rm -rf /usr/local/stow/blackbox
# Add other package types here.
#
# Updates
#
update: tools/mk_deb_fpmdir.stack_blackbox.txt tools/mk_macports.vcs_blackbox.txt
clean:
rm -f tools/mk_deb_fpmdir.stack_blackbox.txt tools/mk_macports.vcs_blackbox.txt
#
# System Test:
#
test: confidence
confidence:
@if [ -e ~/.gnupg ]; then echo ERROR: '~/.gnupg should not exist. If it does, bugs may polute your .gnupg configuration. If the code has no bugs everything will be fine. Do you feel lucky?'; false ; fi
@if which >/dev/null gpg-agent ; then pkill gpg-agent ; rm -rf /tmp/tmp.* ; fi
@export PATH="$(PWD)/bin:$(PREFIX)/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/local/bin:/usr/local/MacGPG2/bin:/opt/homebrew/bin:$(PATH)" ; tools/auto_system_test
@if which >/dev/null gpg-agent ; then pkill gpg-agent ; fi
@if [ -e ~/.gnupg ]; then echo ERROR: '~/.gnupg was created which means the scripts might be poluting GnuPG configuration. Fix this bug.'; false ; fi
================================================
FILE: README-v2.md
================================================
BlackBox v2
===========
WARNING: v2 is still experimental. It is in the same git repo as v1
because the filenames do not overlap. Please do not mix the two. v1
is in `bin`. v2 is in `cmd/blackbox` and `binv2`.
Blackbox is an open source tool that enables you to safe store sensitive information in
Git (or other) repos by encrypting them with GPG. Only the encrypted
version of the file is available. You can be free to provide access
to the repo, as but only people with the right GPG keys can access the
encrypted data.
Things you should **never** store in a repo without encryption:
* TLS (SSL) certificates
* Passwords
* API keys
* And more!
Project Info:
* [Overview](user-overview.md)
* [Why is this important?](why-is-this-important.md)
* [Support/Community](support.md)
* [How BB encrypts](encryption.md)
* [OS Compatibility](compatibility.md)
* [Installation Instructions](installation.md)
* [Alternatives](alternatives.md)
User Info:
* [Enabling Blackbox on a Repo](enable-repo.md)
* [Enroll a file](enable-repo.md)
* [Full Command List](full-command-list.md)
* [Add/Remove users](admin-ops.md)
* [Add/Remove files](file-ops.md)
* [Advanced techiques](advanced.md)
* [Use with Role Accounts](role-accounts.md)
* [Backwards Compatibility](backwards-compatibility.md)
* [Replacing expired keys](expired-keys.md)
* [Git Tips](git-tips.md)
* [SubVersion Tips](subversion-tips.md)
* [GnuPG tips](gnupg-tips.md)
* [Use with Ansible](with-ansible.md)
* [Use with Puppet](with-puppet.md)
For contributors:
* [Developer Info](dev.md)
* [Code overview](dev-code-overview.md)
* [HOWTO: Add new OS support](dev-add-os-support.md)
* [HOWTO: Add new VCS support](dev-add-vcs-support.md)
A slide presentation about an older release [is on SlideShare](http://www.slideshare.net/TomLimoncelli/the-blackbox-project-sfae).
Join our mailing list: [https://groups.google.com/d/forum/blackbox-project](https://groups.google.com/d/forum/blackbox-project)
License
=======
This content is released under the MIT License.
See the [LICENSE.txt](LICENSE.txt) file.
================================================
FILE: README.md
================================================
BlackBox [](https://circleci.com/gh/StackExchange/workflows/blackbox) [](https://github.com/StackExchange/blackbox/actions?query=workflow%3Abuild+branch%3Amaster)
========
THIS PROJECT IS ABANDONED. DO NOT USE. DO NOT SUBMIT BUG REPORTS OR PRs.
There are plenty of other, better, projects. A partial list is listed under "Alternatives" below.
========
Safely store secrets in a VCS repo (i.e. Git, Mercurial, Subversion or Perforce). These commands make it easy for you to Gnu Privacy Guard (GPG) encrypt specific files in a repo so they are "encrypted at rest" in your repository. However, the scripts make it easy to decrypt them when you need to view or edit them, and decrypt them for use in production. Originally written for Puppet, BlackBox now works with any Git or Mercurial repository.
WARNING: The goal of this project is to be a simple wrapper around `gpg` so you and your coworkers don't have to remember its all those inscrutable and confusing flags. It is *not* intended to be a sophisticated encryption system that solves all problems or supports a large numbers of files. The ideal use-case is to keep secrets in a secure service such as Conjur, AWS KMS, Azure Key Vault or GCP KMS; then use Blackbox for safely storing the API keys needed to access that system. That way you are encrypting a single, tiny, file. Feature-requests for anything more will be rejected; do not expect or even request "enterprise features". If this disappoints you, please consider a competiting project such as https://www.agwa.name/projects/git-crypt
A slide presentation (about an older release) [is on SlideShare](http://www.slideshare.net/TomLimoncelli/the-blackbox-project-sfae).
Join our mailing list: [https://groups.google.com/d/forum/blackbox-project](https://groups.google.com/d/forum/blackbox-project)
## What blackbox is NOT:
Table of Contents
=================
- [BlackBox](#blackbox)
- [Table of Contents](#table-of-contents)
- [Overview](#overview)
- [Why is this important?](#why-is-this-important)
- [Installation Instructions](#installation-instructions)
- [Commands](#commands)
- [Compatibility](#compatibility)
- [How is the encryption done?](#how-is-the-encryption-done)
- [What does this look like to the typical user?](#what-does-this-look-like-to-the-typical-user)
- Configuration Management
- [How to use the secrets with Ansible?](#how-to-use-the-secrets-with-ansible)
- [How to use the secrets with Puppet?](#how-to-use-the-secrets-with-puppet)
- [Entire files](#entire-files)
- [Small strings](#small-strings)
- File Management
- [How to enroll a new file into the system?](#how-to-enroll-a-new-file-into-the-system)
- [How to remove a file from the system?](#how-to-remove-a-file-from-the-system)
- User Management
- [How to indoctrinate a new user into the system?](#how-to-indoctrinate-a-new-user-into-the-system)
- [How to remove a user from the system?](#how-to-remove-a-user-from-the-system)
- [Note to the Web Of Trust](#note-to-the-web-of-trust)
- Repo Management
- [Enabling BlackBox For a Repo](#enabling-blackbox-for-a-repo)
- [Set up automated users or “role accounts”](#set-up-automated-users-or-role-accounts)
- [Replacing expired keys](#replacing-expired-keys)
- [Some common errors](#some-common-errors)
- [Using BlackBox on Windows](#using-blackbox-on-windows)
- [Using BlackBox without a repo](#using-blackbox-without-a-repo)
- [Some Subversion gotchas](#some-subversion-gotchas)
- [Using Blackbox when gpg2 is installed next to gpg](#using-blackbox-when-gpg2-is-installed-next-to-gpg)
- [How to submit bugs or ask questions?](#how-to-submit-bugs-or-ask-questions)
- [Developer Info](#developer-info)
- [Alternatives](#alternatives)
- [License](#license)
Overview
========
Suppose you have a VCS repository (i.e. a Git or Mercurial repo) and certain files contain secrets such as passwords or SSL private keys. Often people just store such files "and hope that nobody finds them in the repo". That's not safe.
With BlackBox, those files are stored encrypted using GPG. Access to the VCS repo without also having the right GPG keys makes it worthless to have the files. As long as you keep your GPG keys safe, you don't have to worry about storing your VCS repo on an untrusted server. Heck, even if you trust your server, now you don't have to trust the people that do backups of that server, or the people that handle the backup tapes!
Rather than one GPG passphrase for all the files, each person with access has their own GPG keys in the system. Any file can be decrypted by anyone with their GPG key. This way, if one person leaves the company, you don't have to communicate a new password to everyone with access. Simply disable the one key that should no longer have access. The process for doing this is as easy as running 2 commands (1 to disable their key, 1 to re-encrypt all files.)
Automated processes often need access to all the decrypted files. This is easy too. For example, suppose Git is being used for Puppet files. The master needs access to the decrypted version of all the files. Simply set up a GPG key for the Puppet master (or the role account that pushes new files to the Puppet master) and have that user run `blackbox_postdeploy` after any files are updated.
Getting started
---------------
1. If you don't have a GPG key, set it up using instructions such as:
[Set up GPG key](https://help.github.com/articles/generating-a-new-gpg-key/). \
Now you are ready to go.
1. `cd` into a Git, Mercurial, Subversion or Perforce repository and run `blackbox_initialize`.
1. If a file is to be encrypted, run `blackbox_register_new_file` and you are done.
1. Add and remove keys with `blackbox_addadmin` and `blackbox_removeadmin`.
1. To view and/or edit a file, run `blackbox_edit`;
this will decrypt the file and open with whatever is specified by
your $EDITOR environment variable. \
When you close the editor the
file will automatically be encrypted again and the temporary plaintext
file will be shredded. \
If you need to leave the file decrypted while
you update you can use the`blackbox_edit_start` to decrypt the file
and `blackbox_edit_end` when you want to "put it back in the box."
Why is this important?
======================
OBVIOUSLY we don't want secret things like SSL private keys and passwords to be leaked.
NOT SO OBVIOUSLY when we store "secrets" in a VCS repo like Git or Mercurial, suddenly we are less able to share our code with other people. Communication between subteams of an organization is hurt. You can't collaborate as well. Either you find yourself emailing individual files around (yuck!), making a special repo with just the files needed by your collaborators (yuck!!), or just deciding that collaboration isn't worth all that effort (yuck!!!).
The ability to be open and transparent about our code, with the exception of a few specific files, is key to the kind of collaboration that DevOps and modern IT practitioners need to do.
Installation Instructions
=========================
- *The hard way (manual)*: Copy all the files in "bin" to your "bin".
- *The hard way (automatic)*: `make copy-install` will copy the bin files into $PREFIX/bin, default is /usr/local (uninstall with `make copy-uninstall`).
- *The symlinks way*: `make symlinks-install` will make symlinks of the bin files into $PREFIX/bin, default is /usr/local (uninstall with `make copy-uninstall`) (useful when doing development)
- *The MacPorts Way*: `sudo port install vcs_blackbox`
- *The Homebrew Way*: `brew install blackbox`
- *The RPM way*: Check out the repo and make an RPM via `make packages-rpm`; now you can distribute the RPM via local methods. (Requires [fpm](https://github.com/jordansissel/fpm).)
- *The Debian/Ubuntu way*: Check out the repo and make a DEB via `make packages-deb`; now you can distribute the DEB via local methods. (Requires [fpm](https://github.com/jordansissel/fpm).)
- *The Antigen Way*: Add `antigen bundle StackExchange/blackbox` to your .zshrc
- *The Zgenom Way*: Add `zgenom load StackExchange/blackbox` to your .zshrc where you're loading your other plugins.
- *The Nix Way*: `nix-shell -p blackbox`
- *The Pkgsrc Way*: `pkgin in scm-blackbox`
Commands
========
| Name: | Description: |
|-------------------------------------|-------------------------------------------------------------------------|
| `blackbox_edit <file>` | Decrypt, run $EDITOR, re-encrypt a file |
| `blackbox_edit_start <file>` | Decrypt a file so it can be updated |
| `blackbox_edit_end <file>` | Encrypt a file after blackbox_edit_start was used |
| `blackbox_cat <file>` | Decrypt and view the contents of a file |
| `blackbox_view <file>` | Like blackbox_cat but pipes to `less` or $PAGER |
| `blackbox_diff` | Diff decrypted files against their original crypted version |
| `blackbox_initialize` | Enable blackbox for a GIT or HG repo |
| `blackbox_register_new_file <file>` | Encrypt a file for the first time |
| `blackbox_deregister_file <file>` | Remove a file from blackbox |
| `blackbox_list_files` | List the files maintained by blackbox |
| `blackbox_list_admins` | List admins currently authorized for blackbox |
| `blackbox_decrypt_file <file>` | Decrypt a file |
| `blackbox_decrypt_all_files` | Decrypt all managed files (INTERACTIVE) |
| `blackbox_postdeploy` | Decrypt all managed files (batch) |
| `blackbox_addadmin <gpg-key>` | Add someone to the list of people that can encrypt/decrypt secrets |
| `blackbox_removeadmin <gpg-key>` | Remove someone from the list of people that can encrypt/decrypt secrets |
| `blackbox_shred_all_files` | Safely delete any decrypted files |
| `blackbox_update_all_files` | Decrypt then re-encrypt all files. Useful after keys are changed |
| `blackbox_whatsnew <file>` | show what has changed in the last commit for a given file |
Compatibility
=============
BlackBox automatically determines which VCS you are using and does the right thing. It has a plug-in architecture to make it easy to extend to work with other systems. It has been tested to work with many operating systems.
- Version Control systems
- `git` -- The Git
- `hg` -- Mercurial
- `svn` -- SubVersion (Thanks, Ben Drasin!)
- `p4` -- Perforce
- none -- The files can be decrypted outside of a repo if the `.blackbox` directory is intact
- Operating system
- CentOS / RedHat
- MacOS X
- Cygwin (Thanks, Ben Drasin!) **See Note Below**
- MinGW (git bash on windows) **See Note Below**
- NetBSD
- SmartOS
To add or fix support for a VCS system, look for code at the end of `bin/_blackbox_common.sh`
To add or fix support for a new operating system, look for the case statements in `bin/_blackbox_common.sh` and `bin/_stack_lib.sh` and maybe `tools/confidence_test.sh`
Using BlackBox on Windows
=========================
BlackBox can be used with Cygwin, MinGW or WSL2.
### Protect the line endings
BlackBox assumes that `blackbox-admins.txt` and `blackbox-files.txt` will have
LF line endings. Windows users should be careful to configure Git or other systems
to not convert or "fix" those files.
If you use Git, add the following lines to your `.gitattributes` file:
**/blackbox-admins.txt text eol=lf
**/blackbox-files.txt text eol=lf
The latest version of `blackbox_initialize` will create a `.gitattributes` file in the `$BLACKBOXDATA`
directory (usually `.blackbox`) for you.
### Cygwin
Cygwin support requires the following packages:
Normal operation:
- gnupg
- git or mercurial or subversion or perforce (as appropriate)
Development (if you will be adding code and want to run the confidence test)
- procps
- make
- git (the confidence test currently only tests git)
### MinGW
MinGW (comes with Git for Windows) support requires the following:
Normal operation:
- [Git for Windows](https://git-scm.com/) (not tested with Mercurial)
- Git Bash MINTTY returns a MinGW console. So when you install make sure you pick `MINTTY` instead of windows console. You'll be executing blackbox from the Git Bash prompt.
- You need at least version 2.8.1 of Git for Windows.
- [GnuWin32](https://sourceforge.net/projects/getgnuwin32/files/) - needed for various tools not least of which is mktemp which is used by blackbox
- after downloading the install just provides you with some batch files. Because of prior issues at sourceforge and to make sure you get the latest version of each package the batch files handle the brunt of the work of getting the correct packages and installing them for you.
- from a **windows command prompt** run `download.bat` once it has completed run `install.bat` then add the path for those tools to your PATH (ex: `PATH=%PATH%;c:\GnuWin32\bin`)
Development:
- unknown (if you develop Blackbox under MinGW, please let us know if any additional packages are required to run `make test`)
### WSL2
If you get the following error in WSL2, you can try to setup your environment with the following instructions (Tested with Ubuntu 22.04 on WSL2):
- Install [Gpg4win](https://www.gpg4win.org/) (Tested with version 4.1.0)
- Import your private key in Gpg4win (you can use Kleopatra on your Windows host if you wish).
- Edit the file `~/.gnupg/gpg-agent.conf` on WSL and add the following line: `pinentry-program "/mnt/c/Program Files (x86)/GnuPG/bin/pinentry-basic.exe"`
- Restart gpg agent on your linux system: `gpg-connect-agent reloadagent /bye`
How is the encryption done?
===========================
GPG has many different ways to encrypt a file. BlackBox uses the mode that lets you specify a list of keys that can decrypt the message.
If you have 5 people ("admins") that should be able to access the secrets, each creates a GPG key and adds their public key to the keychain. The GPG command used to encrypt the file lists all 5 key names, and therefore any 1 key can decrypt the file.
To remove someone's access, remove that admin's key name (i.e. email address) from the list of admins and re-encrypt all the files. They can still read the .gpg file (assuming they have access to the repository) but they can't decrypt it any more.
*What if they kept a copy of the old repo before you removed access?* Yes, they can decrypt old versions of the file. This is why when an admin leaves the team, you should change all your passwords, SSL certs, and so on. You should have been doing that before BlackBox, right?
*Why don't you use symmetric keys?* In other words, why mess with all this GPG key stuff and instead why don't we just encrypt all the files with a single passphrase. Yes, GPG supports that, but then we are managing a shared password, which is fraught with problems. If someone "leaves the team" we would have to communicate to everyone a new password. Now we just have to remove their key. This scales better.
*How do automated processes decrypt without asking for a password?* GPG requires a passphrase on a private key. However, it permits the creation of subkeys that have no passphrase. For automated processes, create a subkey that is only stored on the machine that needs to decrypt the files. For example, at Stack Exchange, when our Continuous Integration (CI) system pushes a code change to our Puppet masters, they run `blackbox_postdeploy` to decrypt all the files. The user that runs this code has a subkey that doesn't require a passphrase. Since we have many masters, each has its own key. And, yes, this means our Puppet Masters have to be very secure. However, they were already secure because, like, dude... if you can break into someone's puppet master you own their network.
*If you use Puppet, why didn't you just use hiera-eyaml?* There are 4 reasons:
1. This works with any Git or Mercurial repo, even if you aren't using Puppet.
2. hiera-eyaml decrypts "on demand" which means your Puppet Master now uses a lot of CPU to decrypt keys every time it is contacted. It slows down your master, which, in my case, is already slow enough.
3. This works with binary files, without having to ASCIIify them and paste them into a YAML file. Have you tried to do this with a cert that is 10K long and changes every few weeks? Ick.
4. hiera-eyaml didn't exist when I wrote this.
What does this look like to the typical user?
=============================================
- If you need to, start the GPG Agent: `eval $(gpg-agent --daemon)`
- Decrypt the file so it is editable: `blackbox_edit_start FILENAME`
- (You will need to enter your GPG passphrase.)
- Edit FILENAME as you desire: `vim FILENAME`
- Re-encrypt the file: `blackbox_edit_end FILENAME`
- Commit the changes. `git commit -a` or `hg commit`
Wait... it can be even easier than that! Run `blackbox_edit FILENAME`, and it'll decrypt the file in a temp file and call `$EDITOR` on it, re-encrypting again after the editor is closed.
How to use the secrets with Ansible?
===================================
Ansible Vault provides functionality for encrypting both entire files and strings stored within files; however,
keeping track of the password(s) required for decryption is not handled by this module.
Instead one must specify a password file when running the playbook.
Ansible example for password file: `my_secret_password.txt.gpg`
```
ansible-playbook --vault-password-file my_secret_password.txt site.yml
```
Alternatively, one can specify this in the `ANSIBLE_VAULT_PASSWORD_FILE` environment variable.
How to use the secrets with Puppet?
===================================
### Entire files:
Entire files, such as SSL certs and private keys, are treated just like regular files. You decrypt them any time you push a new release to the puppet master.
Puppet example for an encrypted file: `secret_file.key.gpg`
```
file { '/etc/my_little_secret.key':
ensure => 'file',
owner => 'root',
group => 'puppet',
mode => '0760',
source => "puppet:///modules/${module_name}/secret_file.key",
}
```
### Small strings:
Small strings, such as passwords and API keys, are stored in a hiera yaml file, which you encrypt with `blackbox_register_new_file`. For example, we use a file called `blackbox.yaml`. You can access them using the hiera() function.
*Setup:* Configure `hiera.yaml` by adding "blackbox" to the search hierarchy:
```
:hierarchy:
- ...
- blackbox
- ...
```
In blackbox.yaml specify:
```
---
module::test_password: "my secret password"
```
In your Puppet Code, access the password as you would any hiera data:
```
$the_password = hiera('module::test_password', 'fail')
file {'/tmp/debug-blackbox.txt':
content => $the_password,
owner => 'root',
group => 'root',
mode => '0600',
}
```
The variable `$the_password` will contain "my secret password" and can be used anywhere strings are used.
How to enroll a new file into the system?
=========================================
- If you need to, start the GPG Agent: `eval $(gpg-agent --daemon)`
- Add the file to the system:
```
blackbox_register_new_file path/to/file.name.key
```
Multiple file names can be specified on the command line:
Example 1: Register 2 files:
```
blackbox_register_new_file file1.txt file2.txt
```
Example 2: Register all the files in `$DIR`:
```
find $DIR -type f -not -name '*.gpg' -print0 | xargs -0 blackbox_register_new_file
```
How to remove a file from the system?
=====================================
This happens quite rarely, but we've got it covered:
```
blackbox_deregister_file path/to/file.name.key
```
How to indoctrinate a new user into the system?
===============================================
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
`.blackbox/blackbox-admins.txt` is a file that lists which users are able to decrypt files. (More pedantically, it is a list of the GnuPG key names that the file is encrypted for.)
To join the list of people that can edit the file requires three steps; You create a GPG key and add it to the key ring. Then, someone that already has access adds you to the system. Lastly, you should test your access.
### Step 1: NEW USER creates a GPG key pair on a secure machine and adds to public keychain.
If you don't already have a GPG key, here's how to generate one:
```
gpg --gen-key
```
WARNING: New versions of GPG generate keys which are not understood by
old versions of GPG. If you generate a key with a new version of GPG,
this will cause problems for users of older versions of GPG.
Therefore it is recommended that you either assure that everyone using
Blackbox have the exact same version of GPG, or generate GPG keys
using a version of GPG as old as the oldest version of GPG used by
everyone using Blackbox.
Pick defaults for encryption settings, 0 expiration. Pick a VERY GOOD passphrase. Store a backup of the private key someplace secure. For example, keep the backup copy on a USB drive that is locked in safe. Or, at least put it on a secure machine with little or no internet access, full-disk-encryption, etc. Your employer probably has rules about how to store such things.
FYI: If generating the key is slow, this is usually because the system
isn't generating enough entropy. Tip: Open another window on that
machine and run this command: `ls -R /`
Now that you have a GPG key, add yourself as an admin:
```
blackbox_addadmin KEYNAME
```
...where "KEYNAME" is the email address listed in the gpg key you created previously. For example:
```
blackbox_addadmin tal@example.com
```
When the command completes successfully, instructions on how to commit these changes will be output. Run the command as given to commit the changes. It will look like this:
```
git commit -m'NEW ADMIN: tal@example.com' .blackbox/pubring.gpg .blackbox/trustdb.gpg .blackbox/blackbox-admins.txt
```
Then push it to the repo:
```
git push
or
ht push
(or whatever is appropriate)
```
NOTE: Creating a Role Account? If you are adding the pubring.gpg of a role account, you can specify the directory where the pubring.gpg file can be found as a 2nd parameter: `blackbox_addadmin puppetmaster@puppet-master-1.example.com /path/to/the/dir`
### Step 2: EXISTING ADMIN adds new user to the system.
Ask someone that already has access to re-encrypt the data files. This gives you access. They simply decrypt and re-encrypt the data without making any changes.
Pre-check: Verify the new keys look good.
```
git pull # Or whatever is required for your system
gpg --homedir=.blackbox --list-keys
```
For example, examine the key name (email address) to make sure it conforms to corporate standards.
Import the keychain into your personal keychain and reencrypt:
```
gpg --import .blackbox/pubring.gpg
blackbox_update_all_files
```
Push the re-encrypted files:
```
git commit -a
git push
or
hg commit
hg push
```
### Step 3: NEW USER tests.
Make sure you can decrypt a file. (Suggestion: Keep a dummy file in VCS just for new people to practice on.)
How to remove a user from the system?
=====================================
Simply run `blackbox_removeadmin` with their keyname then re-encrypt:
Example:
```
blackbox_removeadmin olduser@example.com
blackbox_update_all_files
```
When the command completes, you will be given a reminder to check in the change and push it.
Note that their keys will still be in the key ring, but they will go unused. If you'd like to clean up the keyring, use the normal GPG commands and check in the file.
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
```
gpg --homedir=.blackbox --list-keys
gpg --homedir=.blackbox --delete-key olduser@example.com
git commit -m'Cleaned olduser@example.com from keyring' .blackbox/*
```
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
The key ring only has public keys. There are no secret keys to delete.
Remember that this person did have access to all the secrets at one time. They could have made a copy. Therefore, to be completely secure, you should change all passwords, generate new SSL keys, and so on just like when anyone that had privileged access leaves an organization.
Note to the Web Of Trust
========================
Validating the trustworthiness of keys is a task that can't be accomplished by Blackbox; this is a completely external topic that has to be dealt with manually (the same way as generating/managing your key is, for example) or by a dedicated mechanism (a company CA with corresponding workflows e.g.). Aside from the "common" benefits of a Web Of Trust (see [here](https://www.gnupg.org/gph/en/manual/x334.html) or [here](https://www.gnupg.org/gph/en/manual/x547.html) e.g.), it prevents several errors as well.
Historically Blackbox was using and enforcing a "trust every key" model but this has changed! Now the decision of whether and how to use the PGP/GPG trust models is left up to the user by configuration (or by the PGP/GPG defaults).
When updating Blackbox people might run into functional problems if they haven't yet dealt with the trustability of the keys they're using. It's the right time to do so and built up your Web Of Trust now!
If you have an external workflow in place that ensures the integrity of the keys Blackbox uses you might want to disable the PGP/GPG trust models and rely on this workflow.
This can be achieved by declaring "trust model always", either by passing the command line parameter `--trust-model=always` to your PGP/GPG binary when using Blackbox (by defining an alias or using the environment variable (e.g. `GPG="gpg2 --trust-model=always"`) or a combination of both) or by setting `trust-model always` in your `gpg.conf` (note that this disables the Web Of Trust everywhere, not just for Blackbox).
WARNING: It is strongly disadvised to not use any key validation at all! This opens up various ways to bypass the confidentiality of your encrypted secrets!
Where is the configuration stored? .blackbox vs. keyrings/live
==============================================================
Blackbox stores its configuration data in the `.blackbox` subdirectory. Older
repos use `keyrings/live`. For backwards compatibility either will work.
All documentation refers to `.blackbox`.
You can convert an old repo by simply renaming the directory:
```
mv keyrings/live .blackbox
rmdir keyrings
```
There is no technical reason to convert old repos except that it is less
confusing to users.
This change was made in commit 60e782a0, release v1.20180615.
The details:
- First Blackbox checks `$BLACKBOXDATA`. If this environment variable is set, this is the directory that will be used. If it lists a directory that does not exist, Blackbox will print an error and exit.
- If `$BLACKBOXDATA` is not set: (which is the typical use case)
- Blackbox will first try `keyrings/live` and use it if it exists.
- Otherwise the default `.blackbox` will be used. If `.blackbox` does not exist, Blackbox will print an error and exit.
Enabling BlackBox For a Repo
============================
Overview:
To add "blackbox" to a git or mercurial repo, you'll need to do the following:
1. Run the initialize script. This adds a few files to your repo in a directory called ".blackbox".
2. For the first user, create a GPG key and add it to the key ring.
3. Encrypt the files you want to be "secret".
4. For any automated user (one that must be able to decrypt without a passphrase), create a GPG key and create a subkey with an empty passphrase.
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
### Run the initialize script.
You'll want to include blackbox's "bin" directory in your PATH:
```
export PATH=$PATH:/the/path/to/blackbox/bin
blackbox_initialize
```
If you're using antigen, adding `antigen bundle StackExchange/blackbox` to your .zshrc will download this repository and add it to your $PATH.
### For the first user, create a GPG key and add it to the key ring.
Follow the instructions for "[How to indoctrinate a new user into the system?](#how-to-indoctrinate-a-new-user-into-the-system)". Only do Step 1.
Once that is done, is a good idea to test the system by making sure a file can be added to the system (see "How to enroll a new file into the system?"), and a different user can decrypt the file.
Make a new file and register it:
```
rm -f foo.txt.gpg foo.txt
echo This is a test. >foo.txt
blackbox_register_new_file foo.txt
```
Decrypt it:
```
blackbox_edit_start foo.txt.gpg
cat foo.txt
echo This is the new file contents. >foo.txt
```
Re-encrypt it:
```
blackbox_edit_end foo.txt.gpg
ls -l foo.txt*
```
You should only see `foo.txt.gpg` as `foo.txt` should be gone.
The next step is to commit `foo.txt.gpg` and make sure another user can check out, view, and change the contents of the file. That is left as an exercise for the reader. If you are feel like taking a risk, don't commit `foo.txt.gpg` and delete it instead.
Set up automated users or "role accounts"
=========================================
i.e. This is how a Puppet Master can have access to the unencrypted data.
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
An automated user (a "role account") is one that that must be able to decrypt without a passphrase. In general you'll want to do this for the user that pulls the files from the repo to the master. This may be automated with Jenkins CI or other CI system.
GPG keys have to have a passphrase. However, passphrases are optional on subkeys. Therefore, we will create a key with a passphrase then create a subkey without a passphrase. Since the subkey is very powerful, it should be created on a very secure machine.
There's another catch. The role account probably can't check files into Git/Mercurial. It probably only has read-only access to the repo. That's a good security policy. This means that the role account can't be used to upload the subkey public bits into the repo.
Therefore, we will create the key/subkey on a secure machine as yourself. From there we can commit the public portions into the repo. Also from this account we will export the parts that the role account needs, copy them to where the role account can access them, and import them as the role account.
ProTip: If asked to generate entropy, consider running this on the same machine in another window: `sudo dd if=/dev/sda of=/dev/null`
For the rest of this doc, you'll need to make the following substitutions:
- ROLEUSER: svc_deployacct or whatever your role account's name is.
- NEWMASTER: the machine this role account exists on.
- SECUREHOST: The machine you use to create the keys.
NOTE: This should be more automated/scripted. Patches welcome.
On SECUREHOST, create the puppet master's keys:
```
$ mkdir /tmp/NEWMASTER
$ cd /tmp/NEWMASTER
$ gpg --homedir . --gen-key
Your selection?
(1) RSA and RSA (default)
What keysize do you want? (2048) DEFAULT
Key is valid for? (0) DEFAULT
# Real name: Puppet CI Deploy Account
# Email address: svc_deployacct@hostname.domain.name
```
NOTE: Rather than a real email address, use the username@FQDN of the host the key will be used on. If you use this role account on many machines, each should have its own key. By using the FQDN of the host, you will be able to know which key is which. In this doc, we'll refer to username@FQDN as $KEYNAME
Save the passphrase somewhere safe!
Create a sub-key that has no password:
```
$ gpg --homedir . --edit-key svc_deployacct
gpg> addkey
(enter passphrase)
Please select what kind of key you want:
(3) DSA (sign only)
(4) RSA (sign only)
(5) Elgamal (encrypt only)
(6) RSA (encrypt only)
Your selection? 6
What keysize do you want? (2048)
Key is valid for? (0)
Command> key 2
(the new subkey has a "*" next to it)
Command> passwd
(enter the main key's passphrase)
(enter an empty passphrase for the subkey... confirm you want to do this)
Command> save
```
Now securely export this directory to NEWMASTER:
```
gpg --homedir . --export -a svc_sadeploy >/tmp/NEWMASTER/pubkey.txt
tar cvf /tmp/keys.tar .
rsync -avP /tmp/keys.tar NEWMASTER:/tmp/.
```
On NEWMASTER, receive the new GnuPG config:
```
sudo -u svc_deployacct bash
mkdir -m 0700 -p ~/.gnupg
cd ~/.gnupg && tar xpvf /tmp/keys.tar
```
<!---
Back on SECUREHOST, import the pubkey into the repository.
```
$ cd .blackbox
$ gpg --homedir . --import /tmp/NEWMASTER/pubkey.txt
```
-->
Back on SECUREHOST, add the new email address to .blackbox/blackbox-admins.txt:
```
cd /path/to/the/repo
blackbox_addadmin $KEYNAME /tmp/NEWMASTER
```
Verify that secring.gpg is a zero-length file. If it isn't, you have somehow added a private key to the keyring. Start over.
```
cd .blackbox
ls -l secring.gpg
```
Commit the recent changes:
```
cd .blackbox
git commit -m"Adding key for KEYNAME" pubring.gpg trustdb.gpg blackbox-admins.txt
```
Regenerate all encrypted files with the new key:
```
blackbox_update_all_files
git status
git commit -m"updated encryption" -a
git push
```
On NEWMASTER, import the keys and decrypt the files:
```
sudo -u svc_sadeploy bash # Become the role account.
gpg --import /etc/puppet/.blackbox/pubring.gpg
export PATH=$PATH:/path/to/blackbox/bin
blackbox_postdeploy
sudo -u puppet cat /etc/puppet/hieradata/blackbox.yaml # or any encrypted file.
```
ProTip: If you get "gpg: decryption failed: No secret key" then you forgot to re-encrypt blackbox.yaml with the new key.
On SECUREHOST, securely delete your files:
```
cd /tmp/NEWMASTER
# On machines with the "shred" command:
shred -u /tmp/keys.tar
find . -type f -print0 | xargs -0 shred -u
# All else:
rm -rf /tmp/NEWMASTER
```
Also shred any other temporary files you may have made.
Replacing expired keys
======================
If someone's key has already expired, blackbox will stop
encrypting. You see this error:
```
$ blackbox_edit_end modified_file.txt
--> Error: can't re-encrypt because a key has expired.
```
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
You can also detect keys that are about to expire by issuing this command and manually reviewing the "expired:" dates:
gpg --homedir=.blackbox --list-keys
or... list UIDs that will expire within 1 month from today: (Warning: this also lists keys without an expiration date)
gpg --homedir=.blackbox --list-keys --with-colons --fixed-list-mode | grep ^uid | awk -F: '$6 < '$(( $(date +%s) + 2592000))
Here's how to replace the key:
- Step 1. Administrator removes expired user:
Warning: This process will erase any unencrypted files that you were in the process of editing. Copy them elsewhere and restore the changes when done.
```
blackbox_removeadmin expired_user@example.com
# This next command overwrites any changed unencrypted files. See warning above.
blackbox_update_all_files
git commit -m "Re-encrypt all files"
gpg --homedir=.blackbox --delete-key expired_user@example.com
git commit -m 'Cleaned expired_user@example.com from keyring' .blackbox/*
git push
```
- Step 2. Expired user adds an updated key:
```
git pull
blackbox_addadmin updated_user@example.com
git commit -m'NEW ADMIN: updated_user@example.com .blackbox/pubring.gpg .blackbox/trustdb.gpg .blackbox/blackbox-admins.txt
git push
```
- Step 3. Administrator re-encrypts all files with the updated key of the expired user:
```
git pull
gpg --import .blackbox/pubring.gpg
blackbox_update_all_files
git commit -m "Re-encrypt all files"
git push
```
- Step 4: Clean up:
Any files that were temporarily copied in the first step so as to not be overwritten can now be copied back and re-encrypted with the `blackbox_edit_end` command.
(Thanks to @chishaku for finding a solution to this problem!)
### Configure git to show diffs in encrypted files
It's possible to tell Git to decrypt versions of the file before running them through `git diff` or `git log`. To achieve this do:
- Add the following to `.gitattributes` at the top of the git repository:
```
*.gpg diff=blackbox
```
- Add the following to `.git/config`:
```
[diff "blackbox"]
textconv = gpg --use-agent -q --batch --decrypt
````
And now commands like `git log -p file.gpg` will show a nice log of the changes in the encrypted file.
Some common errors
==================
`gpg: filename: skipped: No public key` -- Usually this means there is an item in `.blackbox/blackbox-admins.txt` that is not the name of the key. Either something invalid was inserted (like a filename instead of a username) or a user has left the organization and their key was removed from the keychain, but their name wasn't removed from the blackbox-admins.txt file.
`gpg: decryption failed: No secret key` -- Usually means you forgot to re-encrypt the file with the new key.
`Error: can't re-encrypt because a key has expired.` -- A user's key has expired and can't be used to encrypt any more. Follow the [Replace expired keys](#replace-expired-keys) tip.
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
Using Blackbox without a repo
=============================
If the files are copied out of a repo they can still be decrypted and edited. Obviously edits, changes to keys, and such will be lost if they are made outside the repo. Also note that commands are most likely to only work if run from the base directory (i.e. the parent to the .blackbox directory).
The following commands have been tested outside a repo:
- `blackbox_postdeploy`
- `blackbox_edit_start`
- `blackbox_edit_end`
Some Subversion gotchas
=======================
The current implementation will store the blackbox in `/keyrings` at the root of the entire repo. This will create an issue between environments that have different roots (i.e. checking out `/` on development vs `/releases/foo` in production). To get around this, you can `export BLACKBOX_REPOBASE=/path/to/repo` and set a specific base for your repo.
This was originally written for git and supports a two-phase commit, in which `commit` is a local commit and "push" sends the change upstream to the version control server when something is registered or deregistered with the system. The current implementation will immediately `commit` a file (to the upstream subversion server) when you execute a `blackbox_*` command.
Using Blackbox when gpg2 is installed next to gpg
=================================================
In some situations, team members or automated roles need to install gpg
2.x alongside the system gpg version 1.x to catch up with the team's gpg
version. On Ubuntu 16, you can ```apt-get install gnupg2``` which
installs the binary gpg2. If you want to use this gpg2 binary, run every
blackbox command with GPG=gpg2.
For example:
```
GPG=gpg2 blackbox_postdeploy
```
How to submit bugs or ask questions?
====================================
We welcome questions, bug reports and feedback!
The best place to start is to join the [blackbox-project mailing list](https://groups.google.com/d/forum/blackbox-project) and ask there.
Bugs are tracked here in Github. Please feel free to [report bugs](https://github.com/StackExchange/blackbox/issues) yourself.
Developer Info
==============
Code submissions are gladly welcomed! The code is fairly easy to read.
Get the code:
```
git clone git@github.com:StackExchange/blackbox.git
```
Test your changes:
```
make confidence
```
This runs through a number of system tests. It creates a repo, encrypts files, decrypts files, and so on. You can run these tests to verify that the changes you made didn't break anything. You can also use these tests to verify that the system works with a new operating system.
Please submit tests with code changes:
The best way to change BlackBox is via Test Driven Development. First add a test to `tools/confidence.sh`. This test should fail, and demonstrate the need for the change you are about to make. Then fix the bug or add the feature you want. When you are done, `make confidence` should pass all tests. The PR you submit should include your code as well as the new test. This way the confidence tests accumulate as the system grows as we know future changes don't break old features.
Note: The tests currently assume "git" and have been tested only on CentOS, Mac OS X, and Cygwin. Patches welcome!
Alternatives
============
Here are other open source packages that do something similar to BlackBox. If you like them better than BlackBox, please use them.
- [git-crypt](https://www.agwa.name/projects/git-crypt/)
- [Pass](http://www.zx2c4.com/projects/password-store/)
- [Transcrypt](https://github.com/elasticdog/transcrypt)
- [Keyringer](https://keyringer.pw/)
- [git-secret](https://github.com/sobolevn/git-secret)
git-crypt has the best git integration. Once set up it is nearly transparent to the users. However it only works with git.
License
=======
This content is released under the MIT License.
See the [LICENSE.txt](LICENSE.txt) file.
================================================
FILE: RELEASE_ENGINEERING.md
================================================
Table of Contents:
==================
- [Branches and Tags:](#branches-and-tags)
- [Testing:](#testing)
- [Build Tasks](#build-tasks)
- [Stable Releases](#stable-releases)
- [Production Releases](#production-releases)
- [Updating MacPorts (automatic)](#updating-macports-automatic)
- [Updating MacPorts (manual)](#updating-macports-manual)
Branches and Tags:
==================
There are 3 branches/tags:
- **HEAD:** The cutting edge of development.
- **tag stable:** Stable enough for use by most people.
- **tag production:** Burned in long enough that we are confident it can be widely adopted.
If you are packaging BlackBox for distribution, you should track the *tag production*. You might also want to provide a separate package that tracks *tag stable:* for early adopters.
Testing
=======
Tips:
* macOS: `brew install gpg pinentry`
* FreeBSD: `pkg install gpg gmake`
* CentOS7: `yum install gpg`
To run a suite of tests:
```
cd ~/src/github.com/StackExchange/blackbox
make test
```
FYI: For FreeBSD, use `gmake test`
Build Tasks
===========
Stable Releases
===============
Step 0. Test the software
Run this command to run the unit and system tests:
```
make test
```
NOTE: The tests require pinentry-tty. On macOS with NIX this
can be installed via: `nix-env -i pinentry`
Marking the software to be "stable":
Step 1. Update CHANGELOG.md
Use "git log" to see what has changed and update CHANGELOG.md.
For a new release, add:
```
echo Release v1.$(date +%Y%m%d)
```
Commit with:
```
git commit -m'Update CHANGELOG.md' CHANGELOG.md
```
Step 2. Tag it.
```
git pull
git tag -d stable
git push origin :stable
git tag stable
git push origin tag stable
```
Step 3. Mark your calendar 1 week from today to check to see if this should be promoted to production.
Production Releases
===================
If no bugs have been reported a full week after a stable tag has been pushed, mark the release to be "production".
```
git fetch
git checkout stable
git tag -d production
git push origin :production
git tag production
git push origin tag production
R="v1.$(date +%Y%m%d)"
git tag "$R"
git push origin tag "$R"
```
Step 4. Get credit!
Record the fact that you did this release in your weekly accomplishments file.
Updating MacPorts (automatic)
=============================
Step 1: Generate the Portfile
```
tools/macports_report_upgrade.sh 1.20150222
```
This script will generate a file called `Portfile-vcs_blackbox.diff` and instructions on how to submit it as a update request.
Step 2: Submit the update request.
Submit the diff file as a bug as instructed. The instructions should look like this:
- PLEASE OPEN A TICKET WITH THIS INFORMATION: https://trac.macports.org/newticket
- Summary: `vcs_blackbox @1.20150222 Update to latest upstream`
- Description: `New upstream of vcs_blackbox.
github.setup and checksums updated.`
- Type: `update`
- Component: `ports`
- Port: `vcs_blackbox`
- Keywords: `maintainer haspatch`
- Attach this file: `Portfile-vcs_blackbox.diff`
Step 3: Watch for the update to happen.
Updating MacPorts (manual)
==========================
This is the old, manual, procedure. If the automated procedure fails to work, these notes may or may not be helpful.
The ultimate result of the script should be the output of `diff -u Portfile.orig Portfile` which is sent as an attachment to MacPorts. The new `Portfile` should have these changes:
1. The `github.setup` line should have a new version number.
2. The `checksums` line(s) should have updated checksums.
How to generate the checksums?
The easiest way is to to make a Portfile with incorrect checksums, then run `sudo port -v checksum vcs_blackbox` to see what they should have been. Fix the file, and try again until the checksum command works.
Next run `port lint vcs_blackbox` and make sure it has no errors.
Some useful commands:
Change repos in sources.conf:
```
sudo vi /opt/local/etc/macports/sources.conf
Add this line early in the file:
file:///var/tmp/ports
```
Add a local repo:
```
fgrep >/dev/null -x 'file:///var/tmp/ports' /opt/local/etc/macports/sources.conf || sudo sed -i -e '1s@^@file:///var/tmp/ports\'$'\n@' /opt/local/etc/macports/sources.conf
```
Remove the local repo:
```
sudo sed -i -e '\@^file:///var/tmp/ports@d' /opt/local/etc/macports/sources.conf
```
Test a Portfile:
```
sudo port uninstall vcs_blackbox
sudo port clean --all vcs_blackbox
rm -rf ~/.macports/opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports/security/vcs_blackbox/
rm -rf /var/tmp/ports
mkdir -p /var/tmp/ports/security/vcs_blackbox
cp Portfile /var/tmp/ports/security/vcs_blackbox
cd /var/tmp/ports && portindex
sudo port -v checksum vcs_blackbox
sudo port install vcs_blackbox
```
================================================
FILE: SPECS/empty.spec
================================================
================================================
FILE: Version2-Ideas.md
================================================
# Ideas for BlackBox Version 2
I'm writing this to solicit feedback and encourage discussion.
Here are my thoughts on a "version 2" of BlackBox. This is where
I list ideas that would require major changes to the system. They
might break backwards compatibility, though usually not.
BlackBox grew from a few simple shell scripts used at StackOverflow.com
to a larger system used by dozens (hundreds?) of organizations. Not
all the design decisions were "forward looking".
These are the things I'd like to change someday.
[TOC]
## Change the commmand names
There should be one program, with subcommands that have names that make more sense:
* `blackbox admin add <key>`
* `blackbox admin list`
* `blackbox admin remove <key>`
* `blackbox cat <filename> ...`
* `blackbox decrypt <filename> ...`
* `blackbox diff <filename> ...`
* `blackbox edit <filename> ...`
* `blackbox encrypt <filename> ...`
* `blackbox file add <filename> ...`
* `blackbox file list`
* `blackbox file remove <filename> ...`
* `blackbox info`
* `blackbox init`
* `blackbox reencrypt`
* `blackbox shred --all|<filename> ...`
* `blackbox status --all|<filename> ...`
Backwards compatibility: The old scripts will be rewritten to use the new commands.
## Change the "keyrings" directory
The name `keyrings` was unfortunate. First, it should probably begin with a `.`. Second, it stores more than just keyrings. Lastly, I'm finding that in most cases we want many repos to refer to the same keyring, which is not supported very well.
A better system would be:
1. If `$BLACKBOX_CONFIG` is set, use that directory.
2. If the repo base directory has a file called `.blackbox_external`, read that file as if you are reading `$BLACKBOX_CONFIG`
3. If the repo base directory has a `keyrings` directory, use that.
4. If the repo base directory has a `.blackbox` directory, use that.
Some thoughts on `.blackbox_external`:
I'm not sure what the format should be, but I want it to be simple and expandable. It should support support `../../dir/name` and `/long/path`. However some day we may want to include a Git URL and have the system automatically get the keychain from it. That means the format has to be something like directory:../dir/name so that later we can add git:the-url.
NOTE: Maybe `.blackbox_external` should be `.blackbox/BLACKBOX_CONFIG`?
Backwards compatibility: `keyrings` would be checked before `.blackbox`.
## System Test
There needs to be a very complete system test. The `make test` we
have now is great for something written in bash.
It should be easy to make tests. Perhaps a directory of files, each
specifying a test. We could make a little language for writing tests.
# This test becomes the user "alice" and verifies that she
# can encrypt a file, and decrypt it, with full fidelity.
BECOME alice a
BASH echo "foo contents" >foo.txt
SHOULD_NOT_EXIST foo.txt.gpg
BASH blackbox encrypt foo.txt
SHOULD_NOT_EXIST foo.txt
SHOULD_EXIST foo.txt.gpg
BASH_WITH_PASSWORD a blackbox decrypt foo.txt
SHOULD_EXIST foo.txt.gpg
SHOULD_EXIST foo.txt
SHOULD_CONTAIN foo.txt "foo contents\n"
## Plug-in support
There should plug-ins support for:
Repo type:
* Git -- Using /usr/bin/git or git.exe
* Subversion
* Mercurial
* None (repoless)
* Autodetect
Encryption software:
* GnuPG -- using /usr/bin/gpg{,2} or gpg.exe
* golang.org/x/crypto/openpgp
## JSON or .txt
The files in .blackbox are mostly .txt files. Instead we should
define a .json format, and only read the .txt file is the .json file
doesn't exist.
## Repo-less mode
I can't imagine storing files that aren't in a repo. I just put everything in repos lately. I use it more than I use NFS. That said, I have received feedback that people would like the ability to disable automatic committing of files.
I prefer the file commits to be automatic because when they were manual, people often accidentally committed the plaintext file instead of the GPG file. Fixing such mistakes is a PITA and, of yeah, a big security nightmare.
That said, I'm willing to have a "repo-less" mode.
When this mode is triggered, no add/commit/ignore tasks are done. The search for the keyrings directory still uses `$BLACKBOX_CONFIG` but if that is unset it looks for `.blackbox_config` in the current directory, then recursively `..` until we hit `/`.
I think (but I'm not sure) this would benefit the entire system because it would force us to re-think what VCS actions are done when.
I think (but I'm not sure) that a simple way to implement this would be to add an environment variable that overrides the automatic VCS detection. When set to "none", all VCS operations would basically become no-ops. (This could be done by writing a plug-in that does nothing for all the `vcs_*` calls)
Backwards compatibility: This would add a `none` VCS, not remove any existing functionality.
## Is "bash" the right language?
`bash` is fairly universal. It even exists on Windows. However it is not the right language for large systems. Writing the acceptance tests is quite a bear. Managing `.gitignore` files in bash is impossible and the current implementation fails in many cases.
`python` is my second favorite language. It would make the code cleaner and more testable. However it is not installed everywhere. I would also want to write it in Python3 (why start a new project in Python2?) but sadly Python3 is less common. It is a chicken vs. egg situation.
`go` is my favorite language. I could probably rewrite this in go in a weekend. However, now the code is compiled, not interpreted. Therefore we lose the ability to just `git clone` and have the tools you want. Not everyone has a Go compiler installed on every machine.
The system is basically unusable on Windows without Cygwin or MINGW. A rewrite in python or go would make it work better on Windows, which currently requires Cygwin or MinGW (which is a bigger investment than installing Python). On the other hand, maybe Ubuntu-on-Windows makes that a non-issue.
As long as the code is in `bash` the configuration files like `blackbox-files.txt` and `blackbox-admins.txt` have problems. Filenames with carriage returns aren't supported. If this was in Python/Go/etc. those files could be json or some format with decent quoting and we could handle funny file names better. On the other hand, maybe it is best that we don't support funny filenames... we shouldn't enable bad behavior.
How important is itto blackbox users that the system is written in `bash`?
## Ditch the project and use git-crypt
People tell me that git-crypt is better because, as a plug-in, automagically supports `git diff`, `git log` and `git blame`.
However, I've never used it so I don't have any idea whether git-crypt is any better than blackbox.
Of course, git-crypt doesn't work with SVN, HG, or any other VCS. Is blackbox's strong point the fact that it support so many VCS systems? To be honest, it originally only supported HG and GIT because I was at a company that used HG but then changed to GIT. Supporting anything else was thanks to contributors. Heck, HG support hasn't even been tested recently (by me) since we've gone all git where I work.
How important is this to BlackBox users?
================================================
FILE: bin/Makefile
================================================
all: _stack_lib.sh
# Snatch _stack_lib.sh from another StackExchange project.
_stack_lib.sh: ../../scripts/misc/_stack_lib.sh
cp ../../scripts/misc/_stack_lib.sh $@
================================================
FILE: bin/_blackbox_common.sh
================================================
#!/usr/bin/env bash
#
# Common constants and functions used by the blackbox_* utilities.
#
# Usage:
#
# set -e
# source "${0%/*}/_blackbox_common.sh"
# Load additional useful functions
source "${0%/*}"/_stack_lib.sh
# Where are we?
: "${BLACKBOX_HOME:="$(cd "${0%/*}" ; pwd)"}" ;
# Make grep and sort work like it did in the 1970s:
# Without this, blackbox fails to match filenames in blackbox-files.txt
export LC_ALL='C'
export LANG='en_US.UTF-8'
# What are the candidates for the blackbox data directory?
#
# The order of candidates matter. The first entry of the array
# sets the default Blackbox directory for all new repositories.
declare -a BLACKBOXDATA_CANDIDATES
BLACKBOXDATA_CANDIDATES=(
'.blackbox'
'keyrings/live'
)
# If $EDITOR is not set, set it to "vi":
: "${EDITOR:=vi}" ;
# Allow overriding gpg command
: "${GPG:=gpg}" ;
function physical_directory_of() {
local d=$(dirname "$1")
local f=$(basename "$1")
(cd "$d" && echo "$(pwd -P | sed 's/\/$//')/$f" )
}
# Set REPOBASE to the top of the repository
# Set VCS_TYPE to 'git', 'hg', 'svn' or 'unknown'
if which >/dev/null 2>/dev/null git && git rev-parse --show-toplevel >/dev/null 2>&1 ; then
VCS_TYPE=git
REPOBASE=$(git rev-parse --show-toplevel)
elif [ -d ".svn" ] ; then
# Find topmost dir with .svn sub-dir
parent=""
grandparent="."
while [ -d "$grandparent/.svn" ]; do
parent=$grandparent
grandparent="$parent/.."
done
REPOBASE=$(cd "$parent" ; pwd)
VCS_TYPE=svn
elif which >/dev/null 2>/dev/null hg && hg root >/dev/null 2>&1 ; then
# NOTE: hg has to be tested last because it always "succeeds".
VCS_TYPE=hg
REPOBASE=$(hg root 2>/dev/null)
else
# We aren't in a repo at all. Assume the cwd is the root
# of the tree.
VCS_TYPE=unknown
REPOBASE="$(pwd)"
fi
export VCS_TYPE
export REPOBASE=$(physical_directory_of "$REPOBASE")
# FIXME: Verify this function by checking for .hg or .git
# after determining what we believe to be the answer.
if [[ -n "$BLACKBOX_REPOBASE" ]]; then
echo "Using custom repobase: $BLACKBOX_REPOBASE" >&2
export REPOBASE="$BLACKBOX_REPOBASE"
fi
if [ -z "$BLACKBOXDATA" ] ; then
BLACKBOXDATA="${BLACKBOXDATA_CANDIDATES[0]}"
for candidate in ${BLACKBOXDATA_CANDIDATES[@]} ; do
if [ -d "$REPOBASE/$candidate" ] ; then
BLACKBOXDATA="$candidate"
break
fi
done
fi
KEYRINGDIR="$REPOBASE/$BLACKBOXDATA"
BB_ADMINS_FILE="blackbox-admins.txt"
BB_ADMINS="${KEYRINGDIR}/${BB_ADMINS_FILE}"
BB_FILES_FILE="blackbox-files.txt"
BB_FILES="${KEYRINGDIR}/${BB_FILES_FILE}"
SECRING="${KEYRINGDIR}/secring.gpg"
: "${DECRYPT_UMASK:=0022}" ;
# : ${DECRYPT_UMASK:=o=} ;
# Checks if $1 is 0 bytes, and if $1/keyrings
# is a directory
function is_blackbox_repo() {
if [[ -n "$1" ]] && [[ -d "$1/keyrings" ]]; then
return 0 # Yep, its a repo
else
return 1
fi
}
# Return error if not on cryptlist.
function is_on_cryptlist() {
# Assumes $1 does NOT have the .gpg extension
file_contains_line "$BB_FILES" "$(vcs_relative_path "$1")"
}
# Exit with error if a file exists.
function fail_if_exists() {
if [[ -f "$1" ]]; then
echo ERROR: "$1" exists. "$2" >&2
echo Exiting... >&2
exit 1
fi
}
# Exit with error if a file is missing.
function fail_if_not_exists() {
if [[ ! -f "$1" ]]; then
echo ERROR: "$1" not found. "$2" >&2
echo Exiting... >&2
exit 1
fi
}
# Exit we we aren't in a VCS repo.
function fail_if_not_in_repo() {
if [[ $VCS_TYPE = "unknown" ]]; then
echo "ERROR: This must be run in a VCS repo: git, hg, or svn." >&2
echo Exiting... >&2
exit 1
fi
}
# Exit with error if filename is not registered on blackbox list.
function fail_if_not_on_cryptlist() {
# Assumes $1 does NOT have the .gpg extension
local name="$1"
if ! is_on_cryptlist "$name" ; then
echo "ERROR: $name not found in $BB_FILES" >&2
echo "PWD=$(/usr/bin/env pwd)" >&2
echo 'Exiting...' >&2
exit 1
fi
}
# Exit with error if keychain contains secret keys.
function fail_if_keychain_has_secrets() {
if [[ -s ${SECRING} ]]; then
echo 'ERROR: The file' "$SECRING" 'should be empty.' >&2
echo 'Did someone accidentally add this private key to the ring?' >&2
echo 'Exiting...' >&2
exit 1
fi
}
function get_pubring_path() {
if [[ -f "${KEYRINGDIR}/pubring.gpg" ]]; then
echo "${KEYRINGDIR}/pubring.gpg"
else
echo "${KEYRINGDIR}/pubring.kbx"
fi
}
# Output the unencrypted filename.
function get_unencrypted_filename() {
echo "$(dirname "$1")/$(basename "$1" .gpg)" | sed -e 's#^\./##'
}
# Output the encrypted filename.
function get_encrypted_filename() {
echo "$(dirname "$1")/$(basename "$1" .gpg).gpg" | sed -e 's#^\./##'
}
# Prepare keychain for use.
function prepare_keychain() {
local keyringasc
echo '========== Importing keychain: START' >&2
# Works with gpg 2.0
#$GPG --import "$(get_pubring_path)" 2>&1 | egrep -v 'not changed$' >&2
# Works with gpg 2.0 and 2.1
# NB: We must export the keys to a format that can be imported.
make_self_deleting_tempfile keyringasc
export LANG="C.UTF-8"
#if gpg2 is installed next to gpg like on ubuntu 16
if [[ "$GPG" != "gpg2" ]]; then
$GPG --export --no-default-keyring --keyring "$(get_pubring_path)" >"$keyringasc"
$GPG --import "$keyringasc" 2>&1 | egrep -v 'not changed$' >&2
else
$GPG --keyring "$(get_pubring_path)" --export | $GPG --import
fi
echo '========== Importing keychain: DONE' >&2
}
# Add file to list of encrypted files.
function add_filename_to_cryptlist() {
# If the name is already on the list, this is a no-op.
# However no matter what the datestamp is updated.
# https://github.com/koalaman/shellcheck/wiki/SC2155
local name
name=$(vcs_relative_path "$1")
if file_contains_line "$BB_FILES" "$name" ; then
echo "========== File is registered. No need to add to list."
else
echo "========== Adding file to list."
touch "$BB_FILES"
echo "$name" >> "$BB_FILES"
sort -u -o "$BB_FILES" "$BB_FILES"
fi
}
# Removes a file from the list of encrypted files
function remove_filename_from_cryptlist() {
# If the name is not already on the list, this is a no-op.
# https://github.com/koalaman/shellcheck/wiki/SC2155
local name
name=$(vcs_relative_path "$1")
if ! file_contains_line "$BB_FILES" "$name" ; then
echo "========== File is not registered. No need to remove from list."
else
echo "========== Removing file from list."
remove_line "$BB_FILES" "$name"
fi
}
# Print out who the current BB ADMINS are:
function disclose_admins() {
echo "========== blackbox administrators are:"
cat "$BB_ADMINS"
}
# Encrypt file, overwriting .gpg if it exists.
function encrypt_file() {
local unencrypted
local encrypted
unencrypted="$1"
encrypted="$2"
echo "========== Encrypting: $unencrypted" >&2
$GPG --use-agent --yes --encrypt -o "$encrypted" $(awk '{ print "-r" $1 }' < "$BB_ADMINS") "$unencrypted" >&2
echo '========== Encrypting: DONE' >&2
}
# Decrypt .gpg file, asking "yes/no" before overwriting unencrypted file.
function decrypt_file() {
local encrypted
local unencrypted
local old_umask
encrypted="$1"
unencrypted="$2"
echo "========== EXTRACTING $unencrypted" >&2
old_umask=$(umask)
umask "$DECRYPT_UMASK"
$GPG --use-agent -q --decrypt -o "$unencrypted" "$encrypted" >&2
umask "$old_umask"
}
# Decrypt .gpg file, overwriting unencrypted file if it exists.
function decrypt_file_overwrite() {
local encrypted
local unencrypted
local old_hash
local new_hash
local old_umask
encrypted="$1"
unencrypted="$2"
if [[ -f "$unencrypted" ]]; then
old_hash=$(md5sum_file "$unencrypted")
else
old_hash=unmatchable
fi
old_umask=$(umask)
umask "$DECRYPT_UMASK"
$GPG --use-agent --yes -q --decrypt -o "$unencrypted" "$encrypted" >&2
umask "$old_umask"
new_hash=$(md5sum_file "$unencrypted")
if [[ "$old_hash" != "$new_hash" ]]; then
echo "========== EXTRACTED $unencrypted" >&2
fi
}
# Shred a file. If shred binary does not exist, delete it.
function shred_file() {
local name
local CMD
local OPT
name="$1"
if which shred >/dev/null 2>/dev/null ; then
CMD=shred
OPT=-u
elif which srm >/dev/null 2>/dev/null ; then
#NOTE: srm by default uses 35-pass Gutmann algorithm
CMD=srm
OPT=-f
elif _F=$(mktemp); rm -P "${_F}" >/dev/null 2>/dev/null ; then
CMD=rm
OPT=-Pf
else
echo "shred_file: WARNING: No secure deletion utility (shred or srm) present; using insecure rm" >&2
CMD=rm
OPT=-f
fi
$CMD $OPT -- "$name"
}
# $1 is the name of a file that contains a list of files.
# For each filename, output the individual subdirectories
# leading up to that file. i.e. one one/two one/two/three
function enumerate_subdirs() {
local listfile
local dir
local filename
listfile="$1"
while read filename; do
dir=$(dirname "$filename")
while [[ $dir != '.' && $dir != '/' ]]; do
echo "$dir"
dir=$(dirname "$dir")
done
done <"$listfile" | sort -u
}
# chdir to the base of the repo.
function change_to_vcs_root() {
# if vcs_root not explicitly defined, use $REPOBASE
local rbase=${1:-$REPOBASE} # use $1 but if unset use $REPOBASE
cd "$rbase"
}
# $1 is a string pointing to a directory. Outputs a
# list of valid blackbox repos,relative to $1
function enumerate_blackbox_repos() {
if [[ -z "$1" ]]; then
echo "enumerate_blackbox_repos: ERROR: No Repo provided to Enumerate"
exit 1
fi
# https://github.com/koalaman/shellcheck/wiki/Sc2045
for dir in $1*/; do
if is_blackbox_repo "$dir"; then
echo "$dir"
fi
done
}
# Output the path of a file relative to the repo base
function vcs_relative_path() {
# Usage: vcs_relative_path file
local name="$1"
#python -c 'import os ; print(os.path.relpath("'"$(pwd -P)"'/'"$name"'", "'"$REPOBASE"'"))'
local p=$( printf "%s" "$( pwd -P )/${1}" | sed 's#//*#/#g' )
local name="${p#$REPOBASE}"
name=$( printf "%s" "$name" | sed 's#^/##g' | sed 's#/$##g' )
printf "%s" "$name"
}
# Removes a line from a text file
function remove_line() {
local tempfile
make_self_deleting_tempfile tempfile
# Ensure source file exists
touch "$1"
grep -Fsxv "$2" "$1" > "$tempfile" || true
# Using cat+rm instead of cp will preserve permissions/ownership
cat "$tempfile" > "$1"
}
# Determine if a file contains a given line
function file_contains_line() {
# $1: the file
# $2: the line
grep -xsqF "$2" "$1"
}
#
# Portability Section:
#
#
# Abstract the difference between Linux and Mac OS X:
#
function md5sum_file() {
# Portably generate the MD5 hash of file $1.
case $(uname -s) in
Darwin | FreeBSD )
md5 -r "$1" | awk '{ print $1 }'
;;
NetBSD )
md5 -q "$1"
;;
SunOS )
digest -a md5 "$1"
;;
Linux | CYGWIN* | MINGW* )
md5sum "$1" | awk '{ print $1 }'
;;
* )
echo 'ERROR: Unknown OS. Exiting. (md5sum_file)'
exit 1
;;
esac
}
function cp_permissions() {
# Copy the perms of $1 onto $2 .. end.
case $(uname -s) in
Darwin )
chmod $( stat -f '%Lp' "$1" ) "${@:2}"
;;
FreeBSD | NetBSD )
chmod $( stat -f '%p' "$1" | sed -e "s/^100//" ) "${@:2}"
;;
SunOS )
chmod $( stat -c '%a' "$1" ) "${@:2}"
;;
Linux | CYGWIN* | MINGW* | SunOS )
if [[ -e /etc/alpine-release ]]; then
chmod $( stat -c '%a' "$1" ) "${@:2}"
else
chmod --reference "$1" "${@:2}"
fi
;;
* )
echo 'ERROR: Unknown OS. Exiting. (cp_permissions)'
exit 1
;;
esac
}
#
# Abstract the difference between git and hg:
#
# Is this file in the current repo?
function is_in_vcs() {
is_in_$VCS_TYPE "$@"
}
# Mercurial
function is_in_hg() {
local filename
filename="$1"
if hg locate "$filename" ; then
echo true
else
echo false
fi
}
# Git:
function is_in_git() {
local filename
filename="$1"
if git ls-files --error-unmatch >/dev/null 2>&1 -- "$filename" ; then
echo true
else
echo false
fi
}
# Subversion
function is_in_svn() {
local filename
filename="$1"
if svn list "$filename" ; then
echo true
else
echo false
fi
}
# Perforce
function is_in_p4() {
local filename
filename="$1"
if p4 list "$filename" ; then
echo true
else
echo false
fi
}
# No repo
function is_in_unknown() {
echo true
}
# Add a file to the repo (but don't commit it).
function vcs_add() {
vcs_add_$VCS_TYPE "$@"
}
# Mercurial
function vcs_add_hg() {
hg add "$@"
}
# Git
function vcs_add_git() {
git add "$@"
}
# Subversion
function vcs_add_svn() {
svn add --parents "$@"
}
# Perfoce
function vcs_add_p4() {
p4 add "$@"
}
# No repo
function vcs_add_unknown() {
:
}
# Commit a file to the repo
function vcs_commit() {
vcs_commit_$VCS_TYPE "$@"
}
# Mercurial
function vcs_commit_hg() {
hg commit -m "$@"
}
# Git
function vcs_commit_git() {
git commit -m "$@"
}
# Subversion
function vcs_commit_svn() {
svn commit -m "$@"
}
# Perforce
function vcs_commit_p4() {
p4 submit -d "$@"
}
# No repo
function vcs_commit_unknown() {
:
}
# Remove file from repo, even if it was deleted locally already.
# If it doesn't exist yet in the repo, it should be a no-op.
function vcs_remove() {
vcs_remove_$VCS_TYPE "$@"
}
# Mercurial
function vcs_remove_hg() {
hg rm -A -- "$@"
}
# Git
function vcs_remove_git() {
git rm --ignore-unmatch -f -- "$@"
}
# Subversion
function vcs_remove_svn() {
svn delete "$@"
}
# Perforce
function vcs_remove_p4() {
p4 delete "$@"
}
# No repo
function vcs_remove_unknown() {
:
}
# Get a path for the ignore file if possible in current vcs
function vcs_ignore_file_path() {
vcs_ignore_file_path_$VCS_TYPE
}
# Mercurial
function vcs_ignore_file_path_hg() {
echo "$REPOBASE/.hgignore"
}
# Git
function vcs_ignore_file_path_git() {
echo "$REPOBASE/.gitignore"
}
# Ignore a file in a repo. If it was already ignored, this is a no-op.
function vcs_ignore() {
local file
for file in "$@"; do
vcs_ignore_$VCS_TYPE "$file"
done
}
# Mercurial
function vcs_ignore_hg() {
vcs_ignore_generic_file "$(vcs_ignore_file_path)" "$file"
}
# Git
function vcs_ignore_git() {
vcs_ignore_generic_file "$(vcs_ignore_file_path)" "$file"
git add "$REPOBASE/.gitignore"
}
# Subversion
function vcs_ignore_svn() {
svn propset svn:ignore "$file" "$(vcs_relative_path)"
}
# Perforce
function vcs_ignore_p4() {
:
}
# No repo
function vcs_ignore_unknown() {
:
}
# Generic - add line to file
function vcs_ignore_generic_file() {
local file
file="$(vcs_relative_path "$2")"
file="${file/\$\//}"
file="$(echo "/$file" | sed 's/\([\*\?]\)/\\\1/g')"
if ! file_contains_line "$1" "$file" ; then
echo "$file" >> "$1"
vcs_add "$1"
fi
}
# Notice (un-ignore) a file in a repo. If it was not ignored, this is
# a no-op
function vcs_notice() {
local file
for file in "$@"; do
vcs_notice_$VCS_TYPE "$file"
done
}
# Mercurial
function vcs_notice_hg() {
vcs_notice_generic_file "$REPOBASE/.hgignore" "$file"
}
# Git
function vcs_notice_git() {
vcs_notice_generic_file "$REPOBASE/.gitignore" "$file"
git add "$REPOBASE/.gitignore"
}
# Subversion
function vcs_notice_svn() {
svn propdel svn:ignore "$(vcs_relative_path "$file")"
}
# Perforce
function vcs_notice_p4() {
:
}
# No repo
function vcs_notice_unknown() {
:
}
# Generic - remove line to file
function vcs_notice_generic_file() {
local file
file="$(vcs_relative_path "$2")"
file="${file/\$\//}"
file="$(echo "/$file" | sed 's/\([\*\?]\)/\\\1/g')"
if file_contains_line "$1" "$file" ; then
remove_line "$1" "$file"
vcs_add "$1"
fi
if file_contains_line "$1" "${file:1}" ; then
echo "WARNING: Found a non-absolute ignore match in $1"
echo "WARNING: Confirm the pattern is intended to only exclude $file"
echo "WARNING: If so, manually update the ignore file"
fi
}
function gpg_agent_version_check() {
if ! hash 'gpg-agent' &> /dev/null; then
return 1
fi
local gpg_agent_version=$(gpg-agent --version | head -1 | awk '{ print $3 }' | tr -d '\n')
semverLT $gpg_agent_version "2.1.0"
}
function gpg_agent_notice() {
if [[ $(gpg_agent_version_check) == '0' && -z $GPG_AGENT_INFO ]];then
echo 'WARNING: You probably want to run gpg-agent as'
echo 'you will be asked for your passphrase many times.'
echo 'Example: $ eval $(gpg-agent --daemon)'
read -r -p 'Press CTRL-C now to stop. ENTER to continue: '
fi
}
================================================
FILE: bin/_blackbox_common_test.sh
================================================
#!/usr/bin/env bash
#
# _blackbox_common_test.sh -- Unit tests of functions from _blackbox_common.sh
#
set -e
. "${0%/*}/_blackbox_common.sh"
. tools/test_functions.sh
PHASE 'Test cp-permissions: TestA'
touch TestA TestB TestC TestD
chmod 0347 TestA
chmod 0700 TestB
chmod 0070 TestC
chmod 0070 TestD
cp_permissions TestA TestB TestC
# NOTE: cp_permissions is not touching TestD.
assert_file_perm '--wxr--rwx' TestA
assert_file_perm '--wxr--rwx' TestB
assert_file_perm '--wxr--rwx' TestC
assert_file_perm '----rwx---' TestD # TestD doesn't change.
rm -f TestA TestB TestC TestD
echo '========== DONE.'
================================================
FILE: bin/_stack_lib.sh
================================================
# Library functions for bash scripts at Stack Exchange.
# NOTE: This file is open sourced. Do not put Stack-proprietary code here.
# Usage:
#
# set -e
# . _stack_lib.sh
# ----- Utility Functions -----
function debugmsg() {
# Log to stderr.
echo 1>&2 LOG: "$@"
:
}
function logit() {
# Log to stderr.
echo 1>&2 LOG: "$@"
}
function fail_out() {
echo "FAILED:" "$*"
echo 'Exiting...'
exit 1
}
# on_exit and add_on_exit from http://www.linuxjournal.com/content/use-bash-trap-statement-cleanup-temporary-files
# Usage:
# add_on_exit rm -f /tmp/foo
# add_on_exit echo "I am exiting"
# tempfile=$(mktemp)
# add_on_exit rm -f "$tempfile"
function on_exit()
{
for i in "${on_exit_items[@]}"
do
eval $i
done
}
function add_on_exit()
{
local n=${#on_exit_items[*]}
on_exit_items[$n]="$*"
if [[ $n -eq 0 ]]; then
trap on_exit EXIT
fi
}
function create_self_deleting_tempfile() {
local filename
case $(uname -s) in
Darwin | FreeBSD )
: "${TMPDIR:=/tmp}" ;
filename=$(mktemp -t _stacklib_.XXXXXXXX )
;;
Linux | CYGWIN* | MINGW* | NetBSD | SunOS )
filename=$(mktemp)
;;
* )
echo 'ERROR: Unknown OS. Exiting. (create_self_deleting_tempfile)'
exit 1
;;
esac
add_on_exit rm -f "$filename"
echo "$filename"
}
function create_self_deleting_tempdir() {
local filename
case $(uname -s) in
Darwin | FreeBSD )
: "${TMPDIR:=/tmp}" ;
filename=$(mktemp -d -t _stacklib_.XXXXXXXX )
;;
Linux | CYGWIN* | MINGW* | NetBSD | SunOS )
filename=$(mktemp -d)
;;
* )
echo 'ERROR: Unknown OS. Exiting. (create_self_deleting_tempdir)'
exit 1
;;
esac
add_on_exit rm -rf "$filename"
echo "$filename"
}
# Securely and portably create a temporary file that will be deleted
# on EXIT. $1 is the variable name to store the result.
function make_self_deleting_tempfile() {
local __resultvar="$1"
local name
case $(uname -s) in
Darwin | FreeBSD )
: "${TMPDIR:=/tmp}" ;
name=$(mktemp -t _stacklib_.XXXXXXXX )
;;
Linux | CYGWIN* | MINGW* | NetBSD | SunOS )
name=$(mktemp)
;;
* )
echo 'ERROR: Unknown OS. Exiting. (make_self_deleting_tempfile)'
exit 1
;;
esac
add_on_exit rm -f "$name"
eval $__resultvar="$name"
}
function make_tempdir() {
local __resultvar="$1"
local name
case $(uname -s) in
Darwin | FreeBSD )
: "${TMPDIR:=/tmp}" ;
# The full path to the temp directory must be short.
# This is used by blackbox's testing suite to make a fake GNUPGHOME,
# which needs to fit within sockaddr_un.sun_path (see unix(7)).
name=$(mktemp -d -t SO )
;;
Linux | CYGWIN* | MINGW* | NetBSD | SunOS )
name=$(mktemp -d)
;;
* )
echo 'ERROR: Unknown OS. Exiting. (make_tempdir)'
exit 1
;;
esac
eval $__resultvar="$name"
}
function make_self_deleting_tempdir() {
local __resultvar="$1"
local dname
make_tempdir dname
add_on_exit rm -rf "$dname"
eval $__resultvar="$dname"
}
function fail_if_not_running_as_root() {
if [[ $EUID -ne 0 ]]; then
echo 'ERROR: This command should only be run as root.'
echo 'Exiting...'
exit 1
fi
}
function fail_if_in_root_directory() {
# Verify nobody has tricked us into being in "/".
case $(uname -s) in
Darwin | FreeBSD | NetBSD )
if [[ $(stat -f'%i' / ) == $(stat -f'%i' . ) ]] ; then
echo 'SECURITY ALERT: The current directory is the root directory.'
echo 'Exiting...'
exit 1
fi
;;
Linux | CYGWIN* | MINGW* | SunOS )
if [[ $(stat -c'%i' / ) == $(stat -c'%i' . ) ]] ; then
echo 'SECURITY ALERT: The current directory is the root directory.'
echo 'Exiting...'
exit 1
fi
;;
* )
echo 'ERROR: Unknown OS. Exiting. (fail_if_in_root_directory)'
exit 1
;;
esac
}
function semverParseInto() {
local RE='[^0-9]*\([0-9]*\)[.]\([0-9]*\)[.]\([0-9]*\)\([0-9A-Za-z-]*\)'
#MAJOR
eval $2=`echo $1 | sed -e "s#$RE#\1#"`
#MINOR
eval $3=`echo $1 | sed -e "s#$RE#\2#"`
#MINOR
eval $4=`echo $1 | sed -e "s#$RE#\3#"`
#SPECIAL
eval $5=`echo $1 | sed -e "s#$RE#\4#"`
}
function semverEQ() {
local MAJOR_A=0
local MINOR_A=0
local PATCH_A=0
local SPECIAL_A=0
local MAJOR_B=0
local MINOR_B=0
local PATCH_B=0
local SPECIAL_B=0
semverParseInto $1 MAJOR_A MINOR_A PATCH_A SPECIAL_A
semverParseInto $2 MAJOR_B MINOR_B PATCH_B SPECIAL_B
if [ $MAJOR_A -ne $MAJOR_B ]; then
return 1
fi
if [ $MINOR_A -ne $MINOR_B ]; then
return 1
fi
if [ $PATCH_A -ne $PATCH_B ]; then
return 1
fi
if [[ "_$SPECIAL_A" != "_$SPECIAL_B" ]]; then
return 1
fi
return 0
}
function semverLT() {
local MAJOR_A=0
local MINOR_A=0
local PATCH_A=0
local SPECIAL_A=0
local MAJOR_B=0
local MINOR_B=0
local PATCH_B=0
local SPECIAL_B=0
semverParseInto $1 MAJOR_A MINOR_A PATCH_A SPECIAL_A
semverParseInto $2 MAJOR_B MINOR_B PATCH_B SPECIAL_B
if [ $MAJOR_A -lt $MAJOR_B ]; then
return 0
fi
if [[ $MAJOR_A -le $MAJOR_B && $MINOR_A -lt $MINOR_B ]]; then
return 0
fi
if [[ $MAJOR_A -le $MAJOR_B && $MINOR_A -le $MINOR_B && $PATCH_A -lt $PATCH_B ]]; then
return 0
fi
if [[ "_$SPECIAL_A" == "_" ]] && [[ "_$SPECIAL_B" == "_" ]] ; then
return 1
fi
if [[ "_$SPECIAL_A" == "_" ]] && [[ "_$SPECIAL_B" != "_" ]] ; then
return 1
fi
if [[ "_$SPECIAL_A" != "_" ]] && [[ "_$SPECIAL_B" == "_" ]] ; then
return 0
fi
if [[ "_$SPECIAL_A" < "_$SPECIAL_B" ]]; then
return 0
fi
return 1
}
function semverGT() {
semverEQ $1 $2
local EQ=$?
semverLT $1 $2
local LT=$?
if [ $EQ -ne 0 ] && [ $LT -ne 0 ]; then
return 0
else
return 1
fi
}
================================================
FILE: bin/blackbox_addadmin
================================================
#!/usr/bin/env bash
#
# blackbox_addadmin -- Add an admin to the system
#
# Example:
# blackbox_addadmin tal@example.com
#
set -e
source "${0%/*}/_blackbox_common.sh"
fail_if_not_in_repo
KEYNAME="$1"
: "${KEYNAME:?ERROR: First argument must be a keyname (email address)}" ;
# Add the email address to the BB_ADMINS file. Remove any duplicates.
# The file must exist for sort to act as we expect.
touch "$BB_ADMINS"
echo "$1" >> "$BB_ADMINS"
sort -fdu -o "$BB_ADMINS" "$BB_ADMINS"
# Add the user's key to the keychain.
# Extract it:
make_self_deleting_tempfile pubkeyfile
# The second argument, if present, is the directory to find the GPG keys to be imported.
if [[ -z $2 ]]; then
$GPG --export -a "$KEYNAME" >"$pubkeyfile"
else
# TODO(tlim): This could probably be done with GNUPGHOME
# but that affects all commands; we just want it to affect the key export.
$GPG --homedir="$2" --export -a "$KEYNAME" >"$pubkeyfile"
fi
if [[ $(wc -l < "$pubkeyfile") = 0 ]]; then
fail_out "GPG key '$KEYNAME' not found. Please create it with: $GPG --gen-key"
exit 1
fi
# Import it:
$GPG --no-permission-warning --homedir="$KEYRINGDIR" --import "$pubkeyfile"
pubring_path=$(get_pubring_path)
vcs_add "$pubring_path" "$KEYRINGDIR/trustdb.gpg" "$BB_ADMINS"
# Make a suggestion:
echo
echo
echo 'NEXT STEP: You need to manually check these in:'
echo ' ' $VCS_TYPE commit -m\'NEW ADMIN: $KEYNAME\' "$BLACKBOXDATA/$(basename ${pubring_path})" "$BLACKBOXDATA/trustdb.gpg" "$BLACKBOXDATA/$BB_ADMINS_FILE"
================================================
FILE: bin/blackbox_cat
================================================
#!/usr/bin/env bash
#
# blackbox_cat -- Decrypt a file, cat it, shred it
#
set -e
source "${0%/*}/_blackbox_common.sh"
for param in "$@" ; do
shreddable=0
unencrypted_file=$(get_unencrypted_filename "$param")
if [[ ! -e "$unencrypted_file" ]]; then
"${BLACKBOX_HOME}/blackbox_edit_start" "$param"
shreddable=1
fi
cat "$unencrypted_file"
if [[ $shreddable = 1 ]]; then
shred_file "$unencrypted_file"
fi
done
================================================
FILE: bin/blackbox_decrypt_all_files
================================================
#!/usr/bin/env bash
#
# blackbox_decrypt_all_files -- Decrypt all blackbox files (INTERACTIVE).
#
# Usage:
# blackbox_decrypt_all_files [GROUP]
# GROUP is optional. If supplied, the resulting files
# are chgrp'ed to that group.
# Since this is often run in a security-critical situation, we
# force /usr/bin and /bin to the front of the PATH.
export PATH=/usr/bin:/bin:"$PATH"
set -e
source "${0%/*}/_blackbox_common.sh"
gpg_agent_notice
exec blackbox_postdeploy "$@"
================================================
FILE: bin/blackbox_decrypt_file
================================================
#!/usr/bin/env bash
#
# blackbox_decrypt_file -- Decrypt one or more blackbox files.
#
set -e
source "${0%/*}/_blackbox_common.sh"
if [ $# -eq 0 ]; then
echo >&2 "Please provide at least one file to decrypt"
exit 1
fi
"${BLACKBOX_HOME}/blackbox_edit_start" "$@"
================================================
FILE: bin/blackbox_deregister_file
================================================
#!/usr/bin/env bash
#
# blackbox_deregister_file -- Remove a file from the blackbox system.
#
# Takes an encrypted file and removes it from the blackbox system.
# The encrypted file will also be removed from the filesystem.
# The unencrypted file, if it exists, will be left alone.
set -e
source "${0%/*}/_blackbox_common.sh"
unencrypted_file=$(get_unencrypted_filename "$1")
encrypted_file=$(get_encrypted_filename "$1")
if [[ "$1" == "$unencrypted_file" ]]; then
echo ERROR: Please only deregister encrypted files.
exit 1
fi
echo ========== PLAINFILE "$unencrypted_file"
echo ========== ENCRYPTED "$encrypted_file"
fail_if_not_exists "$encrypted_file" "Please specify an existing file."
prepare_keychain
remove_filename_from_cryptlist "$unencrypted_file"
vcs_remove "$encrypted_file"
vcs_notice "$unencrypted_file"
vcs_add "$BB_FILES"
vcs_commit "Removing from blackbox: ${unencrypted_file}" "$BB_FILES" "$encrypted_file" "$(vcs_ignore_file_path)"
echo "========== UPDATING VCS: DONE"
echo "Local repo updated. Please push when ready."
echo " $VCS_TYPE push"
================================================
FILE: bin/blackbox_diff
================================================
#!/usr/bin/env bash
#
# blackbox_diff -- Show all differences.
#
set -e
source "${0%/*}/_blackbox_common.sh"
gpg_agent_notice
prepare_keychain
modified_files=()
modifications=()
echo '========== DIFFING FILES: START'
while IFS= read <&99 -r unencrypted_file; do
unencrypted_file=$(get_unencrypted_filename "$unencrypted_file")
encrypted_file=$(get_encrypted_filename "$unencrypted_file")
fail_if_not_on_cryptlist "$unencrypted_file"
if [[ -f "$unencrypted_file" ]]; then
out=$(diff -u <($GPG --yes -q --decrypt "$encrypted_file") "$unencrypted_file" || true)
if [ "$out" != "" ]; then
modified_files+=("$unencrypted_file")
modifications+=("$out")
fi
fi
done 99<"$BB_FILES"
modified_files_number=${#modified_files[@]}
for (( i=0; i<${modified_files_number}; i++ )); do
echo ========== PROCESSING '"'${modified_files[$i]}'"'
echo -e "${modifications[$i]}\n"
done
echo '========== DIFFING FILES: DONE'
fail_if_keychain_has_secrets
echo '========== DONE.'
if [ ${#modified_files[@]} -eq 0 ] ; then
exit 0
fi
echo 'Likely next steps:'
for f in "${modified_files[@]}" ; do
echo " blackbox_edit_end" '"'$f'"'
done
================================================
FILE: bin/blackbox_edit
================================================
#!/usr/bin/env bash
#
# blackbox_edit -- Decrypt a file temporarily for edition, then re-encrypts it again
#
set -e
source "${0%/*}/_blackbox_common.sh"
for param in "$@" ; do
unencrypted_file=$(get_unencrypted_filename "$param")
encrypted_file=$(get_encrypted_filename "$param")
echo >&2 ========== PLAINFILE '"'$unencrypted_file'"'
echo >&2 ========== ENCRYPTED '"'$encrypted_file'"'
if ! is_on_cryptlist "$encrypted_file" && ! is_on_cryptlist "$unencrypted_file" ; then
read -r -p "Encrypt file $param? (y/n) " ans
case "$ans" in
y* | Y*)
"${BLACKBOX_HOME}/blackbox_register_new_file" "$unencrypted_file"
;;
*)
echo >&2 'Skipping...'
continue
;;
esac
fi
"${BLACKBOX_HOME}/blackbox_edit_start" "$unencrypted_file"
$EDITOR "$(get_unencrypted_filename "$unencrypted_file")"
"${BLACKBOX_HOME}/blackbox_edit_end" "$unencrypted_file"
done
================================================
FILE: bin/blackbox_edit_end
================================================
#!/usr/bin/env bash
#
# blackbox_edit_end -- Re-encrypt file after edits.
#
set -e
source "${0%/*}/_blackbox_common.sh"
next_steps=()
if [ $# -eq 0 ]; then
echo >&2 "Please provide at least one file for which editing has finished"
exit 1
fi
for param in "$@" ; do
unencrypted_file=$(get_unencrypted_filename "$param")
encrypted_file=$(get_encrypted_filename "$param")
echo >&2 ========== PLAINFILE '"'$unencrypted_file'"'
echo >&2 ========== ENCRYPTED '"'$encrypted_file'"'
fail_if_not_on_cryptlist "$unencrypted_file"
fail_if_not_exists "$unencrypted_file" "No unencrypted version to encrypt!"
fail_if_keychain_has_secrets
encrypt_file "$unencrypted_file" "$encrypted_file"
shred_file "$unencrypted_file"
echo >&2 ========== UPDATED '"'$encrypted_file'"'
next_steps+=( " $VCS_TYPE commit -m\"${encrypted_file} updated\" \"$encrypted_file\"" )
done
echo >&2 "Likely next step:"
for x in "${next_steps[@]}"
do
echo >&2 "$x"
done
================================================
FILE: bin/blackbox_edit_start
================================================
#!/usr/bin/env bash
#
# blackbox_edit_start -- Decrypt a file for editing.
#
set -e
source "${0%/*}/_blackbox_common.sh"
if [ $# -eq 0 ]; then
echo >&2 "Please provide at least one file to start editing"
exit 1
fi
for param in "$@" ; do
unencrypted_file=$(get_unencrypted_filename "$param")
encrypted_file=$(get_encrypted_filename "$param")
echo >&2 ========== PLAINFILE '"'$unencrypted_file'"'
fail_if_not_on_cryptlist "$unencrypted_file"
fail_if_not_exists "$encrypted_file" "This should not happen."
if [[ ! -s "$unencrypted_file" ]]; then
rm -f "$unencrypted_file"
fi
if [[ -f "$unencrypted_file" ]]; then
echo >&2 SKIPPING: "$1" "Will not overwrite non-empty files."
continue
fi
prepare_keychain
# FIXME(tlim): prepare_keychain only needs to run once, outside of the loop.
decrypt_file "$encrypted_file" "$unencrypted_file"
done
================================================
FILE: bin/blackbox_initialize
================================================
#!/usr/bin/env bash
#
# blackbox_initialize -- Enable blackbox for a GIT or HG repo.
#
#
# Example:
# blackbox_initialize
#
set -e
source "${0%/*}/_blackbox_common.sh"
if [[ $1 != 'yes' ]]; then
read -r -p "Enable blackbox for this $VCS_TYPE repo? (yes/no) " ans
if [[ $ans = 'no' || $ans = 'n' || $ans = '' ]]; then
echo 'Exiting...'
exit 1
fi
fi
if [[ $VCS_TYPE = "unknown" ]]; then
echo 'Not in a known VCS directory'
exit 1
fi
change_to_vcs_root
echo VCS_TYPE: $VCS_TYPE
vcs_ignore "${BLACKBOXDATA}/pubring.gpg~" "${BLACKBOXDATA}/pubring.kbx~" "${BLACKBOXDATA}/secring.gpg"
# Make directories
mkdir -p "${KEYRINGDIR}"
vcs_add "${KEYRINGDIR}"
touch "$BLACKBOXDATA/$BB_ADMINS_FILE" "$BLACKBOXDATA/$BB_FILES_FILE"
vcs_add "$BLACKBOXDATA/$BB_ADMINS_FILE" "$BLACKBOXDATA/$BB_FILES_FILE"
if [[ $VCS_TYPE = "git" ]]; then
# Set .gitattributes so that Windows users don't break the admin files.
FILE="$BLACKBOXDATA/.gitattributes"
touch "$FILE"
LINE='blackbox-admins.txt text eol=lf'
grep -qF "$LINE" "$FILE" || echo "$LINE" >> "$FILE"
LINE='blackbox-files.txt text eol=lf'
grep -qF "$LINE" "$FILE" || echo "$LINE" >> "$FILE"
vcs_add "$FILE"
fi
if [[ $VCS_TYPE = "svn" ]]; then
echo
echo
echo '`subversion` automatically tracks the ignored files; you just need to commit.'
else
IGNOREFILE="$(vcs_ignore_file_path)"
test -f "$IGNOREFILE" && vcs_add "$IGNOREFILE"
# Make a suggestion:
echo
echo
echo 'NEXT STEP: You need to manually check these in:'
echo ' ' $VCS_TYPE commit -m\'INITIALIZE BLACKBOX\' "$BLACKBOXDATA" "$IGNOREFILE"
fi
================================================
FILE: bin/blackbox_list_admins
================================================
#!/usr/bin/env bash
#
# blackbox_list_admins -- List authorized admins
#
set -e
source "${0%/*}/_blackbox_common.sh"
cat "$BB_ADMINS"
================================================
FILE: bin/blackbox_list_files
================================================
#!/usr/bin/env bash
#
# blackbox_list_files -- List files that black box is tracking
#
set -e
source "${0%/*}/_blackbox_common.sh"
cat "$BB_FILES"
================================================
FILE: bin/blackbox_listadmins
================================================
#!/usr/bin/env bash
#
# blackbox_listadmins -- List active admins for keyring
#
# Example:
# blackbox_listadmins
#
set -e
source "${0%/*}/_blackbox_common.sh"
fail_if_not_in_repo
# simply display the contents of the admins file
cat "$BB_ADMINS"
================================================
FILE: bin/blackbox_postdeploy
================================================
#!/usr/bin/env bash
#
# blackbox_postdeploy -- Decrypt all blackbox files.
#
# Usage:
# blackbox_postdeploy.sh [GROUP]
# GROUP is optional. If supplied, the resulting files
# are chgrp'ed to that group.
# Since this is often run in a security-critical situation, we
# force /usr/bin and /bin to the front of the PATH.
export PATH=/usr/bin:/bin:"$PATH"
set -e
source "${0%/*}/_blackbox_common.sh"
if [[ "$1" == "" ]]; then
FILE_GROUP=""
else
FILE_GROUP="$1"
fi
change_to_vcs_root
prepare_keychain
# Decrypt:
echo '========== Decrypting new/changed files: START'
while IFS= read <&99 -r unencrypted_file; do
encrypted_file=$(get_encrypted_filename "$unencrypted_file")
decrypt_file_overwrite "$encrypted_file" "$unencrypted_file"
cp_permissions "$encrypted_file" "$unencrypted_file"
if [[ ! -z "$FILE_GROUP" ]]; then
chmod g+r "$unencrypted_file"
chgrp "$FILE_GROUP" "$unencrypted_file"
fi
done 99<"$BB_FILES"
echo '========== Decrypting new/changed files: DONE'
================================================
FILE: bin/blackbox_recurse
================================================
#!/usr/bin/env bash
# proposed space for blackbox recurion...coming soon
set -e
source "${0%/*}/_blackbox_common.sh"
echo "$REBOBASE"
================================================
FILE: bin/blackbox_register_new_file
================================================
#!/usr/bin/env bash
#
# blackbox_register_new_file -- Enroll new file(s) in the blackbox system.
#
# Takes previously unencrypted file(s) and enrolls them into the blackbox
# system. Each file will be kept in the repo as an encrypted file. On deployment
# to systems that need the plaintext (unencrypted) versions, run
# blackbox_postdeploy.sh to decrypt all the files.
set -e
source "${0%/*}/_blackbox_common.sh"
function register_new_file() {
unencrypted_file=$(get_unencrypted_filename "$1")
encrypted_file=$(get_encrypted_filename "$1")
if [[ "$1" == "$encrypted_file" ]]; then
echo ERROR: Please only register unencrypted files.
exit 1
fi
echo "========== PLAINFILE $unencrypted_file"
echo "========== ENCRYPTED $encrypted_file"
fail_if_not_exists "$unencrypted_file" "Please specify an existing file."
fail_if_exists "$encrypted_file" "Will not overwrite."
prepare_keychain
encrypt_file "$unencrypted_file" "$encrypted_file"
add_filename_to_cryptlist "$unencrypted_file"
vcs_ignore "$unencrypted_file"
# Is the unencrypted file already in HG? (ie. are we correcting a bad situation)
SECRETSEXPOSED=$(is_in_vcs "${unencrypted_file}")
echo "========== CREATED: ${encrypted_file}"
echo "========== UPDATING REPO:"
shred_file "$unencrypted_file"
if [[ "$SECRETSEXPOSED" == "true" ]] ; then
vcs_remove "$unencrypted_file"
vcs_add "$encrypted_file"
fi
echo 'NOTE: "already tracked!" messages are safe to ignore.'
vcs_add "$BB_FILES" "$encrypted_file"
vcs_commit "registered in blackbox: ${unencrypted_file}" "$BB_FILES" "$encrypted_file" "$(vcs_ignore_file_path)"
}
for target in "$@"; do
register_new_file "$target"
done
echo "========== UPDATING VCS: DONE"
if [[ $VCS_TYPE = "svn" ]]; then
echo "Local repo updated and file pushed to source control (unless an error was displayed)."
else
echo "Local repo updated. Please push when ready."
echo " $VCS_TYPE push"
fi
================================================
FILE: bin/blackbox_removeadmin
================================================
#!/usr/bin/env bash
#
# blackbox_removeadmin -- Remove an admin to the system
# NOTE: Does not remove admin from the keyring.
#
# Example:
# blackbox_removeadmin tal@example.com
#
set -e
source "${0%/*}/_blackbox_common.sh"
fail_if_not_in_repo
KEYNAME="$1"
: "${KEYNAME:?ERROR: First argument must be a keyname (email address)}" ;
# Remove the email address from the BB_ADMINS file.
remove_line "$BB_ADMINS" "$KEYNAME"
# remove the admin key from the pubring
$GPG --no-permission-warning --homedir="$KEYRINGDIR" --batch --yes --delete-key "$KEYNAME" || true
pubring_path=$(get_pubring_path)
vcs_add "$pubring_path" "$KEYRINGDIR/trustdb.gpg" "$BB_ADMINS"
# Make a suggestion:
echo
echo
echo 'NEXT STEP: Check these into the repo. Probably with a command like...'
echo $VCS_TYPE commit -m\'REMOVED ADMIN: $KEYNAME\' "$BLACKBOXDATA/$(basename ${pubring_path})" "$BLACKBOXDATA/trustdb.gpg" "$BLACKBOXDATA/$BB_ADMINS_FILE"
================================================
FILE: bin/blackbox_shred_all_files
================================================
#!/usr/bin/env bash
#
# blackbox_shred_all_files -- shred all decrypted versions of encrypted files
#
# Shred: To securely delete a file.
#
# Typical uses:
# After running blackbox_edit_start, deciding not to edit the file.
# A developer that wants to securely clean up a workspace before deleting it.
# An automated process that doesn't want to leave
# plaintext (unencrypted) files laying around.
#
# NOTE: The output lists files that were decrypted and are being
# shredded. For example, if you have many encrypted files but none
# have been decrypted for editing, you will see an empty list.
set -e
source "${0%/*}/_blackbox_common.sh"
change_to_vcs_root
echo '========== FILES BEING SHREDDED:'
exported_internal_shred_file() {
source "$1/_blackbox_common.sh"
#unencrypted_file=$(get_unencrypted_filename "$2")
unencrypted_file="$2"
if [[ -f "$unencrypted_file" ]]; then
echo " SHRED: $unencrypted_file"
shred_file "$unencrypted_file"
else
echo "NOT FOUND: $unencrypted_file"
fi
}
export -f exported_internal_shred_file
DEREFERENCED_BIN_DIR="${0%/*}"
MAX_PARALLEL_SHRED=10
bash_args=
if bash --help | grep import-functions >/dev/null 2>/dev/null; then
bash_args=--import-functions
fi
export IFS=
tr '\n' '\0' <"$BB_FILES" | xargs -0 -I{} -P $MAX_PARALLEL_SHRED bash $bash_args -c "exported_internal_shred_file $DEREFERENCED_BIN_DIR \"{}\"" $DEREFERENCED_BIN_DIR/fake
echo '========== DONE.'
================================================
FILE: bin/blackbox_update_all_files
================================================
#!/usr/bin/env bash
#
# blackbox_update_all_files -- Decrypt then re-encrypt all files. Useful after keys are changed.
#
set -e
source "${0%/*}/_blackbox_common.sh"
gpg_agent_notice
disclose_admins
prepare_keychain
echo '========== ENCRYPTED FILES TO BE RE-ENCRYPTED:'
while IFS= read <&99 -r unencrypted_file; do
echo " $unencrypted_file.gpg"
done 99<"$BB_FILES"
echo '========== FILES IN THE WAY:'
need_warning=false
while IFS= read <&99 -r unencrypted_file; do
unencrypted_file=$(get_unencrypted_filename "$unencrypted_file")
encrypted_file=$(get_encrypted_filename "$unencrypted_file")
if [[ -f "$unencrypted_file" ]]; then
need_warning=true
echo " $unencrypted_file"
fi
done 99<"$BB_FILES"
if "$need_warning" ; then
echo
echo 'WARNING: This will overwrite any unencrypted files laying about.'
read -r -p 'Press CTRL-C now to stop. ENTER to continue: '
else
echo 'All OK.'
fi
echo '========== RE-ENCRYPTING FILES:'
while IFS= read <&99 -r unencrypted_file; do
unencrypted_file=$(get_unencrypted_filename "$unencrypted_file")
encrypted_file=$(get_encrypted_filename "$unencrypted_file")
echo ========== PROCESSING '"'$unencrypted_file'"'
fail_if_not_on_cryptlist "$unencrypted_file"
decrypt_file_overwrite "$encrypted_file" "$unencrypted_file"
encrypt_file "$unencrypted_file" "$encrypted_file"
shred_file "$unencrypted_file"
done 99<"$BB_FILES"
fail_if_keychain_has_secrets
echo '========== COMMITING TO VCS:'
while IFS= read <&99 -r unencrypted_file; do
vcs_add "$unencrypted_file.gpg"
done 99<"$BB_FILES"
vcs_commit 'Re-encrypted keys'
echo '========== DONE.'
echo 'Likely next step:'
echo " $VCS_TYPE push"
================================================
FILE: bin/blackbox_view
================================================
#!/usr/bin/env bash
#
# blackbox_view -- Decrypt a file, view it, shred it
#
set -e
source "${0%/*}/_blackbox_common.sh"
for param in "$@" ; do
shreddable=0
unencrypted_file=$(get_unencrypted_filename "$param")
if [[ ! -e "$unencrypted_file" ]]; then
"${BLACKBOX_HOME}/blackbox_edit_start" "$param"
shreddable=1
fi
${PAGER:-less} "$unencrypted_file"
if [[ $shreddable = 1 ]]; then
shred_file "$unencrypted_file"
fi
done
================================================
FILE: bin/blackbox_whatsnew
================================================
#!/usr/bin/env bash
#
# blackbox_whatsnew - show what has changed in the last commit for a given file
#
set -e
source "${0%/*}/_blackbox_common.sh"
if [[ $# -ne 1 ]]
then
echo "Pass only 1 file at a time"
exit 1
fi
fail_if_not_in_repo
gpg_agent_notice
COLUMNS=`tput cols`
FILE=$1
GIT="git log --abbrev-commit --pretty=oneline"
CURR_COMMIT=`$GIT $FILE | head -1 | awk '{print $1}'`
PREV_COMMIT=`$GIT ${CURR_COMMIT}~1 $FILE | head -1 | awk '{print $1}'`
# Use colordiff if available
if which colordiff > /dev/null 2>&1
then DIFF="colordiff"
else DIFF="diff"
fi
cat_commit()
{
COMMIT=$1
git checkout $COMMIT $FILE
echo "[$COMMIT] $FILE"
echo "---------------------"
"${BLACKBOX_HOME}/blackbox_cat" $FILE | sed '/========== PLAINFILE/,/========== EXTRACTING/d'
}
CURR_CONTENT=`cat_commit $CURR_COMMIT`
PREV_CONTENT=`cat_commit $PREV_COMMIT`
clear
# For some unknown reason this command executes fine but return exit code 1
$DIFF -y --width $COLUMNS \
<(echo "CURRENT" "$CURR_CONTENT" | fold -w $(( $COLUMNS / 2 - 4 )) ) \
<(echo "PREVIOUS" "$PREV_CONTENT" | fold -w $(( $COLUMNS / 2 - 4 )) )
git checkout $CURR_COMMIT $FILE
echo
================================================
FILE: binv2/blackbox_addadmin
================================================
#!/usr/bin/env bash
exec blackbox admin add "$@"
================================================
FILE: binv2/blackbox_cat
================================================
#!/usr/bin/env bash
exec blackbox cat "$@"
================================================
FILE: binv2/blackbox_decrypt_all_files
================================================
#!/usr/bin/env bash
exec blackbox decrypt --all --agentcheck=true --overwrite "@"
================================================
FILE: binv2/blackbox_decrypt_file
================================================
#!/usr/bin/env bash
exec blackbox decrypt --overwrite "$@"
================================================
FILE: binv2/blackbox_deregister_file
================================================
#!/usr/bin/env bash
exec blackbox file remove --safe "$@"
================================================
FILE: binv2/blackbox_diff
================================================
#!/usr/bin/env bash
exec blackbox diff --diff "$@"
================================================
FILE: binv2/blackbox_edit
================================================
#!/usr/bin/env bash
exec blackbox edit "$@"
================================================
FILE: binv2/blackbox_edit_end
================================================
#!/usr/bin/env bash
exec blackbox encrypt --shred "$@"
================================================
FILE: binv2/blackbox_edit_start
================================================
#!/usr/bin/env bash
exec blackbox decrypt "$@"
================================================
FILE: binv2/blackbox_initialize
================================================
#!/usr/bin/env bash
exec blackbox init "$@"
================================================
FILE: binv2/blackbox_list_admins
================================================
#!/usr/bin/env bash
exec blackbox admin list
================================================
FILE: binv2/blackbox_list_files
================================================
#!/usr/bin/env bash
exec blackbox file list
================================================
FILE: binv2/blackbox_listadmins
================================================
#!/usr/bin/env bash
exec blackbox admin list
================================================
FILE: binv2/blackbox_postdeploy
================================================
#!/usr/bin/env bash
blackbox decrypt --all --overwrite --group "$1"
================================================
FILE: binv2/blackbox_register_new_file
================================================
#!/usr/bin/env bash
exec blackbox file add --shred "$@"
================================================
FILE: binv2/blackbox_removeadmin
================================================
#!/usr/bin/env bash
exec blackbox admin remove "$@"
================================================
FILE: binv2/blackbox_shred_all_files
================================================
#!/usr/bin/env bash
exec blackbox shred --all
================================================
FILE: binv2/blackbox_update_all_files
================================================
#!/usr/bin/env bash
exec blackbox reencrypt --all --agentcheck
================================================
FILE: binv2/blackbox_view
================================================
#!/usr/bin/env bash
blackbox cat "$@" | ${PAGER:-less}
================================================
FILE: binv2/blackbox_whatsnew
================================================
#!/usr/bin/env bash
#
# blackbox_whatsnew - show what has changed in the last commit for a given file
#
exec blackbox whatsnew "$@"
exit 0
set -e
source "${0%/*}/_blackbox_common.sh"
if [[ $# -ne 1 ]]
then
echo "Pass only 1 file at a time"
exit 1
fi
fail_if_not_in_repo
gpg_agent_notice
COLUMNS=`tput cols`
FILE=$1
GIT="git log --abbrev-commit --pretty=oneline"
CURR_COMMIT=`$GIT $FILE | head -1 | awk '{print $1}'`
PREV_COMMIT=`$GIT ${CURR_COMMIT}~1 $FILE | head -1 | awk '{print $1}'`
# Use colordiff if available
if which colordiff > /dev/null 2>&1
then DIFF="colordiff"
else DIFF="diff"
fi
cat_commit()
{
COMMIT=$1
git checkout $COMMIT $FILE
echo "[$COMMIT] $FILE"
echo "---------------------"
"${BLACKBOX_HOME}/blackbox_cat" $FILE | sed '/========== PLAINFILE/,/========== EXTRACTING/d'
}
CURR_CONTENT=`cat_commit $CURR_COMMIT`
PREV_CONTENT=`cat_commit $PREV_COMMIT`
clear
# For some unknown reason this command executes fine but return exit code 1
$DIFF -y --width $COLUMNS \
<(echo "CURRENT" "$CURR_CONTENT" | fold -w $(( $COLUMNS / 2 - 4 )) ) \
<(echo "PREVIOUS" "$PREV_CONTENT" | fold -w $(( $COLUMNS / 2 - 4 )) )
git checkout $CURR_COMMIT $FILE
echo
================================================
FILE: blackbox.plugin.zsh
================================================
#!/usr/bin/env zsh
# The MIT License (MIT)
# Copyright (c) 2014 Stack Exchange, Inc.
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
# Make it easy to install with antigen and other frameworks that can cope
# with antigen/oh-my-zsh bundles
#
# antigen bundle StackExchange/blackbox
# Add our plugin's bin diretory to user's path
PLUGIN_BIN="$(dirname $0)/bin"
export PATH="${PATH}:${PLUGIN_BIN}"
================================================
FILE: build/build.go
================================================
package main
import (
"flag"
"fmt"
"log"
"os"
"os/exec"
"strings"
"time"
)
var sha = flag.String("sha", "", "SHA of current commit")
var goos = flag.String("os", "", "OS to build (linux, windows, or darwin) Defaults to all.")
func main() {
flag.Parse()
flags := fmt.Sprintf(`-s -w -X main.SHA="%s" -X main.BuildTime=%d`, getVersion(), time.Now().Unix())
pkg := "github.com/StackExchange/blackbox/v2/cmd/blackbox"
build := func(out, goos string) {
log.Printf("Building %s", out)
cmd := exec.Command("go", "build", "-o", out, "-ldflags", flags, pkg)
os.Setenv("GOOS", goos)
os.Setenv("GO111MODULE", "on")
cmd.Stderr = os.Stderr
cmd.Stdout = os.Stdout
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
}
for _, env := range []struct {
binary, goos string
}{
{"blackbox-Linux", "linux"},
{"blackbox.exe", "windows"},
{"blackbox-Darwin", "darwin"},
} {
if *goos == "" || *goos == env.goos {
build(env.binary, env.goos)
}
}
}
func getVersion() string {
if *sha != "" {
return *sha
}
// check teamcity build version
if v := os.Getenv("BUILD_VCS_NUMBER"); v != "" {
return v
}
// check git
cmd := exec.Command("git", "rev-parse", "HEAD")
v, err := cmd.CombinedOutput()
if err != nil {
return ""
}
ver := strings.TrimSpace(string(v))
// see if dirty
cmd = exec.Command("git", "diff-index", "--quiet", "HEAD", "--")
err = cmd.Run()
// exit status 1 indicates dirty tree
if err != nil {
if err.Error() == "exit status 1" {
ver += "[dirty]"
} else {
log.Printf("!%s!", err.Error())
}
}
return ver
}
================================================
FILE: cmd/blackbox/blackbox.go
================================================
package main
import (
"fmt"
"os"
_ "github.com/StackExchange/blackbox/v2/pkg/crypters"
_ "github.com/StackExchange/blackbox/v2/pkg/crypters/_all"
_ "github.com/StackExchange/blackbox/v2/pkg/vcs"
_ "github.com/StackExchange/blackbox/v2/pkg/vcs/_all"
)
var dryRun bool
func main() {
app := flags()
err := app.Run(os.Args)
if err != nil {
fmt.Fprintf(os.Stderr, "ERROR: %s\n", err)
os.Exit(1)
}
}
================================================
FILE: cmd/blackbox/cli.go
================================================
package main
// cli.go -- Create urfave/cli datastructures and apply them.
import (
"fmt"
"github.com/urfave/cli/v2"
"github.com/StackExchange/blackbox/v2/pkg/bbutil"
)
func flags() *cli.App {
app := cli.NewApp()
app.Version = "2.0.0"
app.Usage = "Maintain encrypted files in a VCS (Git, Hg, Svn)"
defUmask := bbutil.Umask(0)
bbutil.Umask(defUmask)
defUmaskS := fmt.Sprintf("%04o", defUmask)
app.Flags = []cli.Flag{
// &cli.BoolFlag{
// Name: "dry-run",
// Aliases: []string{"n"},
// Usage: "show what would have been done",
// Destination: &dryRun,
// },
&cli.StringFlag{
Name: "vcs",
Usage: "Use this VCS (GIT, NONE) rather than autodetect",
EnvVars: []string{"BLACKBOX_VCS"},
},
&cli.StringFlag{
Name: "crypto",
Usage: "Crypto back-end plugin",
Value: "GnuPG",
EnvVars: []string{"BLACKBOX_CRYPTO"},
},
&cli.StringFlag{
Name: "config",
Usage: "Path to config",
//Value: ".blackbox",
EnvVars: []string{"BLACKBOX_CONFIGDIR", "BLACKBOXDATA"},
},
&cli.StringFlag{
Name: "team",
Usage: "Use .blackbox-$TEAM as the configdir",
EnvVars: []string{"BLACKBOX_TEAM"},
},
&cli.StringFlag{
Name: "editor",
Usage: "editor to use",
Value: "vi",
EnvVars: []string{"EDITOR", "BLACKBOX_EDITOR"},
},
&cli.StringFlag{
Name: "umask",
Usage: "umask to set when decrypting",
Value: defUmaskS,
EnvVars: []string{"BLACKBOX_UMASK", "DECRYPT_UMASK"},
},
&cli.BoolFlag{
Name: "debug",
Usage: "Show debug output",
EnvVars: []string{"BLACKBOX_DEBUG"},
},
}
app.Commands = []*cli.Command{
// List items in the order they appear in the help menu.
{
Name: "decrypt",
Aliases: []string{"de", "start"},
Usage: "Decrypt file(s)",
Flags: []cli.Flag{
&cli.BoolFlag{Name: "all", Usage: "All registered files"},
&cli.BoolFlag{Name: "agentcheck", Usage: "Do not check for gpg-agent when using --all"},
&cli.StringFlag{Name: "group", Usage: "Set group ownership"},
&cli.BoolFlag{Name: "overwrite", Usage: "Overwrite plaintext if it exists"},
},
Action: func(c *cli.Context) error { return cmdDecrypt(c) },
},
{
Name: "encrypt",
Aliases: []string{"en", "end"},
Usage: "Encrypts file(s)",
Flags: []cli.Flag{
&cli.BoolFlag{Name: "shred", Usage: "Remove plaintext afterwards"},
},
Action: func(c *cli.Context) error { return cmdEncrypt(c) },
},
{
Name: "edit",
Aliases: []string{"vi"},
Usage: "Runs $EDITOR on file(s) (decrypt if needed)",
Action: func(c *cli.Context) error { return cmdEdit(c) },
},
{
Name: "cat",
Usage: "Output plaintext to stderr (decrypt if needed)",
Action: func(c *cli.Context) error { return cmdCat(c) },
},
{
Name: "diff",
Usage: "Diffs against encrypted version",
Flags: []cli.Flag{
&cli.BoolFlag{Name: "all", Usage: "all files"},
},
Action: func(c *cli.Context) error { return cmdDiff(c) },
},
{
Name: "init",
Category: "ADMINISTRATIVE",
Usage: "Initialized blackbox for this repo",
Action: func(c *cli.Context) error { return cmdInit(c) },
},
{
Name: "admin",
Category: "ADMINISTRATIVE",
Usage: "Add/list/remove administrators",
Subcommands: []*cli.Command{
{
Name: "add",
Usage: "Adds admin(s)",
Action: func(c *cli.Context) error { return cmdAdminAdd(c) },
},
{
Name: "list",
Usage: "Lists admins",
Action: func(c *cli.Context) error { return cmdAdminList(c) },
},
{
Name: "remove",
Usage: "Remove admin(s)",
Action: func(c *cli.Context) error { return cmdAdminRemove(c) },
},
},
},
{
Name: "file",
Category: "ADMINISTRATIVE",
Usage: "Add/list/remove files from the registry",
Subcommands: []*cli.Command{
{
Name: "add",
Usage: "Registers file with the system",
Flags: []cli.Flag{
&cli.BoolFlag{Name: "shred", Usage: "Remove plaintext afterwords"},
},
Action: func(c *cli.Context) error { return cmdFileAdd(c) },
},
{
Name: "list",
Usage: "Lists the registered files",
Action: func(c *cli.Context) error { return cmdFileList(c) },
},
{
Name: "remove",
Usage: "Deregister file from the system",
Action: func(c *cli.Context) error { return cmdFileRemove(c) },
},
},
},
{
Name: "info",
Category: "DEBUG",
Usage: "Report what we know about this repo",
Action: func(c *cli.Context) error { return cmdInfo(c) },
},
{
Name: "shred",
Usage: "Shred files, or --all for all registered files",
Flags: []cli.Flag{
&cli.BoolFlag{Name: "all", Usage: "All registered files"},
},
Action: func(c *cli.Context) error { return cmdShred(c) },
},
{
Name: "status",
Category: "ADMINISTRATIVE",
Usage: "Print status of files",
Flags: []cli.Flag{
&cli.BoolFlag{Name: "name-only", Usage: "Show only names of the files"},
&cli.BoolFlag{Name: "all", Usage: "All registered files"},
&cli.StringFlag{Name: "type", Usage: "only list if status matching this string"},
},
Action: func(c *cli.Context) error { return cmdStatus(c) },
},
{
Name: "reencrypt",
Usage: "Decrypt then re-encrypt files (erases any plaintext)",
Category: "ADMINISTRATIVE",
Flags: []cli.Flag{
&cli.BoolFlag{Name: "all", Usage: "All registered files"},
&cli.BoolFlag{Name: "overwrite", Usage: "Overwrite plaintext if it exists"},
&cli.BoolFlag{Name: "agentcheck", Usage: "Do not check for gpg-agent when using --all"},
},
Action: func(c *cli.Context) error { return cmdReencrypt(c) },
},
{
Name: "testing_init",
Usage: "For use with integration test",
Category: "INTEGRATION TEST",
Action: func(c *cli.Context) error { return testingInit(c) },
},
//
}
return app
}
================================================
FILE: cmd/blackbox/drive.go
================================================
package main
// Now that cli.go has processed the flags, validate there are no
// conflicts and drive to the business logic.
import (
"fmt"
"log"
"os"
"github.com/StackExchange/blackbox/v2/pkg/bblog"
"github.com/StackExchange/blackbox/v2/pkg/box"
"github.com/urfave/cli/v2"
)
var logErr *log.Logger
func init() {
if logErr == nil {
logErr = log.New(os.Stderr, "", 0)
}
}
func allOrSomeFiles(c *cli.Context) error {
if c.Bool("all") && c.Args().Present() {
return fmt.Errorf("Can not specify filenames and --all")
}
if (!c.Args().Present()) && (!c.Bool("all")) {
return fmt.Errorf("Must specify at least one file name or --all")
}
return nil
}
const roError = `This command is disabled due to --config flag being used.
We can not determine if the flag's value is in or out of the repo, and
Blackbox can only work on one repo at a time. If the value is inside the
repo, and you'd like to suggest an algorithm that would let us determine that
automatically, please file a bug. We'd love to have this work better. In the
meanwhile, run this command without the --config flag, perhaps after cd'ing
to the base of the repo.`
// Keep these functions in alphabetical order.
func cmdAdminAdd(c *cli.Context) error {
if c.NArg() == 0 || c.NArg() > 2 {
return fmt.Errorf(
"Must specify one admin's GnuPG user-id (i.e. email address) and optionally the directory of the pubkey data (default ~/.GnuPG)")
}
bx := box.NewFromFlags(c)
if bx.ConfigRO {
return fmt.Errorf(roError)
}
err := bx.AdminAdd(c.Args().Get(0), c.Args().Get(1))
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdAdminList(c *cli.Context) error {
if c.Args().Present() {
return fmt.Errorf("This command takes zero arguments")
}
bx := box.NewFromFlags(c)
err := bx.AdminList()
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdAdminRemove(c *cli.Context) error {
if !c.Args().Present() {
return fmt.Errorf("Must specify at least one admin's GnuPG user-id (i.e. email address)")
}
bx := box.NewFromFlags(c)
if bx.ConfigRO {
return fmt.Errorf(roError)
}
err := bx.AdminRemove(c.Args().Slice())
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdCat(c *cli.Context) error {
if !c.Args().Present() {
return fmt.Errorf("Must specify at least one file name")
}
bx := box.NewFromFlags(c)
err := bx.Cat(c.Args().Slice())
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdDecrypt(c *cli.Context) error {
if err := allOrSomeFiles(c); err != nil {
return err
}
// The default for --agentcheck is off normally, and on when using --all.
pauseNeeded := c.Bool("all")
// If the user used the flag, abide by it.
if c.IsSet("agentcheck") {
pauseNeeded = c.Bool("agentcheck")
}
bx := box.NewFromFlags(c)
err := bx.Decrypt(c.Args().Slice(),
c.Bool("overwrite"),
pauseNeeded,
c.String("group"),
)
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdDiff(c *cli.Context) error {
if err := allOrSomeFiles(c); err != nil {
return err
}
bx := box.NewFromFlags(c)
err := bx.Diff(c.Args().Slice())
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdEdit(c *cli.Context) error {
if !c.Args().Present() {
return fmt.Errorf("Must specify at least one file name")
}
bx := box.NewFromFlags(c)
err := bx.Edit(c.Args().Slice())
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdEncrypt(c *cli.Context) error {
if err := allOrSomeFiles(c); err != nil {
return err
}
bx := box.NewFromFlags(c)
err := bx.Encrypt(c.Args().Slice(), c.Bool("shred"))
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdFileAdd(c *cli.Context) error {
if !c.Args().Present() {
return fmt.Errorf("Must specify at least one file name")
}
bx := box.NewFromFlags(c)
if bx.ConfigRO {
return fmt.Errorf(roError)
}
err := bx.FileAdd(c.Args().Slice(), c.Bool("shred"))
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdFileList(c *cli.Context) error {
if c.Args().Present() {
return fmt.Errorf("This command takes zero arguments")
}
bx := box.NewFromFlags(c)
err := bx.FileList()
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdFileRemove(c *cli.Context) error {
if !c.Args().Present() {
return fmt.Errorf("Must specify at least one file name")
}
bx := box.NewFromFlags(c)
if bx.ConfigRO {
return fmt.Errorf(roError)
}
err := bx.FileRemove(c.Args().Slice())
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdInfo(c *cli.Context) error {
if c.Args().Present() {
return fmt.Errorf("This command takes zero arguments")
}
bx := box.NewFromFlags(c)
err := bx.Info()
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdInit(c *cli.Context) error {
if c.Args().Len() > 1 {
return fmt.Errorf("This command takes one or two arguments")
}
bx := box.NewUninitialized(c)
if bx.ConfigRO {
return fmt.Errorf(roError)
}
err := bx.Init(c.Args().First(), c.String("vcs"))
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdReencrypt(c *cli.Context) error {
if err := allOrSomeFiles(c); err != nil {
return err
}
// The default for --agentcheck is off normally, and on when using --all.
pauseNeeded := c.Bool("all")
// If the user used the flag, abide by it.
if c.IsSet("agentcheck") {
pauseNeeded = c.Bool("agentcheck")
}
bx := box.NewFromFlags(c)
err := bx.Reencrypt(c.Args().Slice(),
c.Bool("overwrite"),
pauseNeeded,
)
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdShred(c *cli.Context) error {
if err := allOrSomeFiles(c); err != nil {
return err
}
bx := box.NewFromFlags(c)
err := bx.Shred(c.Args().Slice())
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
func cmdStatus(c *cli.Context) error {
if c.Bool("all") && c.Args().Present() {
return fmt.Errorf("Can not specify filenames and --all")
}
bx := box.NewFromFlags(c)
err := bx.Status(c.Args().Slice(), c.Bool("name-only"), c.String("type"))
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
// These are "secret" commands used by the integration tests.
func testingInit(c *cli.Context) error {
if c.Args().Present() {
return fmt.Errorf("No args required")
}
logDebug := bblog.GetDebug(c.Bool("debug"))
logDebug.Printf(
"c.String(vcs) reports %q\n",
c.String("vcs"),
)
bx := box.NewForTestingInit(c.String("vcs"))
if bx.ConfigRO {
return fmt.Errorf(roError)
}
err := bx.TestingInitRepo()
if err != nil {
return err
}
return bx.Vcs.FlushCommits()
}
================================================
FILE: docs/README.md
================================================
BlackBox
========
Blackbox is an open source tool that enables you to safe store sensitive information in
Git (or other) repos by encrypting them with GPG. Only the encrypted
version of the file is available. You can be free to provide access
to the repo, as but only people with the right GPG keys can access the
encrypted data.
Things you should **never** store in a repo without encryption:
* TLS (SSL) certificates
* Passwords
* API keys
* And more!
Project Info:
* [Overview](user-overview.md)
* [Why is this important?](why-is-this-important.md)
* [Support/Community](support.md)
* [How BB encrypts](encryption.md)
* [OS Compatibility](compatibility.md)
* [Installation Instructions](installation.md)
* [Alternatives](alternatives.md)
User Info:
* [Enabling Blackbox on a Repo](enable-repo.md)
* [Enroll a file](enable-repo.md)
* [Full Command List](full-command-list.md)
* [Add/Remove users](admin-ops.md)
* [Add/Remove files](file-ops.md)
* [Advanced techiques](advanced.md)
* [Use with Role Accounts](role-accounts.md)
* [Backwards Compatibility](backwards-compatibility.md)
* [Replacing expired keys](expired-keys.md)
* [Git Tips](git-tips.md)
* [SubVersion Tips](subversion-tips.md)
* [GnuPG tips](gnupg-tips.md)
* [Use with Ansible](with-ansible.md)
* [Use with Puppet](with-puppet.md)
For contributors:
* [Developer Info](dev.md)
* [Code overview](dev-code-overview.md)
* [HOWTO: Add new OS support](dev-add-os-support.md)
* [HOWTO: Add new VCS support](dev-add-vcs-support.md)
A slide presentation about an older release [is on SlideShare](http://www.slideshare.net/TomLimoncelli/the-blackbox-project-sfae).
Join our mailing list: [https://groups.google.com/d/forum/blackbox-project](https://groups.google.com/d/forum/blackbox-project)
License
=======
This content is released under the MIT License.
See the [LICENSE.txt](LICENSE.txt) file.
================================================
FILE: docs/admin-ops.md
================================================
User Management
===============
# Who are the current admins?
```
blackbox admin list
```
# Add a new user (admin)
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
`.blackbox/blackbox-admins.txt` is a file that lists which users are able to decrypt files. (More pedantically, it is a list of the GnuPG key names that the file is encrypted for.)
To join the list of people that can edit the file requires three steps; You create a GPG key and add it to the key ring. Then, someone that already has access adds you to the system. Lastly, you should test your access.
## Step 1: NEWPERSON creates a GPG key pair on a secure machine and add to public keychain.
If you don't already have a GPG key, here's how to generate one:
```
gpg --gen-key
```
WARNING: New versions of GPG generate keys which are not understood by
old versions of GPG. If you generate a key with a new version of GPG,
this will cause problems for users of older versions of GPG.
Therefore it is recommended that you either assure that everyone using
Blackbox have the exact same version of GPG, or generate GPG keys
using a version of GPG as old as the oldest version of GPG used by
everyone using Blackbox.
Pick defaults for encryption settings, 0 expiration. Pick a VERY GOOD
passphrase. Store a backup of the private key someplace secure. For
example, keep the backup copy on a USB drive that is locked in safe.
Or, at least put it on a secure machine with little or no internet
access, full-disk-encryption, etc. Your employer probably has rules
about how to store such things.
FYI: If generating the key is slow, this is usually because the system
isn't generating enough entropy. Tip: Open another window on that
machine and run this command: `ls -R /`
Now that you have a GPG key, add yourself as an admin:
```
blackbox admin add KEYNAME
```
...where "KEYNAME" is the email address listed in the gpg key you created previously. For example:
```
blackbox admin add tal@example.com
```
When the command completes successfully, instructions on how to commit these changes will be output. Run the command as given to commit the changes. It will look like this:
```
git commit -m'NEW ADMIN: tal@example.com' .blackbox/pubring.gpg .blackbox/trustdb.gpg .blackbox/blackbox-admins.txt
```
Then push it to the repo:
```
git push
or
ht push
(or whatever is appropriate)
```
NOTE: Creating a Role Account? If you are adding the pubring.gpg of a role account, you can specify the directory where the pubring.gpg file can be found as a 2nd parameter: `blackbox admin add puppetmaster@puppet-master-1.example.com /path/to/the/dir`
## Step 2: AN EXISTING ADMIN accepts you into the system.
Ask someone that already has access to re-encrypt the data files. This
gives you access. They simply decrypt and re-encrypt the data without
making any changes.
Pre-check: Verify the new keys look good.
```
git pull # Or whatever is required for your system
gpg --homedir=.blackbox --list-keys
```
For example, examine the key name (email address) to make sure it conforms to corporate standards.
Import the keychain into your personal keychain and reencrypt:
```
gpg --import .blackbox/pubring.gpg
blackbox reencrypt --all shred
```
Push the re-encrypted files:
```
git commit -a
git push
or
hg commit
hg push
```
### Step 3: NEWPERSON tests.
Make sure you can decrypt a file. (Suggestion: Keep a dummy file in
VCS just for new people to practice on.)
# Remove a user
Simply run `blackbox admin remove` with their keyname then re-encrypt:
Example:
```
blackbox admin remove olduser@example.com
blackbox reencrypt --all shred
```
When the command completes, you will be given a reminder to check in the change and push it.
Note that their keys will still be in the key ring, but they will go unused. If you'd like to clean up the keyring, use the normal GPG commands and check in the file.
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
```
gpg --homedir=.blackbox --list-keys
gpg --homedir=.blackbox --delete-key olduser@example.com
git commit -m'Cleaned olduser@example.com from keyring' .blackbox/*
```
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
The key ring only has public keys. There are no secret keys to delete.
Remember that this person did have access to all the secrets at one time. They could have made a copy. Therefore, to be completely secure, you should change all passwords, generate new SSL keys, and so on just like when anyone that had privileged access leaves an organization.
================================================
FILE: docs/advanced.md
================================================
Advanced Techniques
===================
# Using Blackbox without a repo
If the files are copied out of a repo they can still be decrypted and
edited. Obviously edits, changes to keys, and such will be lost if
they are made outside the repo. Also note that commands are most
likely to only work if run from the base directory (i.e. the parent to
the .blackbox directory).
Without a repo, all commands must be run from the same directory
as the ".blackbox" directory. It might work otherwise but no
promises.
# Mixing gpg 1.x/2.0 and 2.2
WARNING: Each version of GnuPG uses a different, and incompatible,
binary format to store the keychain. When Blackbox was originally
created, I didn't know this. Things are mostly upwards compatible.
That said, if you have some admins with GnuPG 1.x and others with GnuPG 2.2,
you may corrupt the keychain.
A future version will store the keychain in an GnuPG-approved
version-neutral format.
# Having gpg and gpg2 on the same machine
NOTE: This is not implemented at this time. TODO(tlim) Use GPG to find
the binary.
In some situations, team members or automated roles need to install gpg
2.x alongside the system gpg version 1.x to catch up with the team's gpg
version. On Ubuntu 16, you can ```apt-get install gnupg2``` which
installs the binary gpg2. If you want to use this gpg2 binary, run every
blackbox command with GPG=gpg2.
For example:
```
GPG=gpg2 blackbox_postdeploy
```
================================================
FILE: docs/alternatives.md
================================================
Alternatives
============
Here are other open source packages that do something similar to
BlackBox. If you like them better than BlackBox, please use them.
- [git-crypt](https://www.agwa.name/projects/git-crypt/)
- [Pass](http://www.zx2c4.com/projects/password-store/)
- [Transcrypt](https://github.com/elasticdog/transcrypt)
- [Keyringer](https://keyringer.pw/)
- [git-secret](https://github.com/sobolevn/git-secret)
git-crypt has the best git integration. Once set up it is nearly
transparent to the users. However it only works with git.
================================================
FILE: docs/backwards-compatibility.md
================================================
Backwards Compatibility
=======================
# Where is the configuration stored? .blackbox vs. keyrings/live
Blackbox stores its configuration data in the `.blackbox` subdirectory. Older
repos use `keyrings/live`. For backwards compatibility either will work.
All documentation refers to `.blackbox`.
You can convert an old repo by simply renaming the directory:
```
mv keyrings/live .blackbox
rmdir keyrings
```
There is no technical reason to convert old repos except that it is less
confusing to users.
This change was made in commit 60e782a0, release v1.20180615.
# How blackbox fines the config directory:
## Creating the repo:
`blackbox init` creates the config directory in the root
of the repo. Here's how it picks the name:
- If `$BLACKBOX_TEAM` is set, `.blackbox-$BLACKBOX_TEAM` is used.
- If the flag `--team <teamname>` is set, it uses `.blackbox-<teamname>`
- Otherwise, it uses `.blackbox`
When searching for the configuration directory, the following
locations are checked. First match wins.
- `.blackbox-$BLACKBOX_TEAM` (only if `$BLACKBOX_TEAM` is set)
- The value of `--config value` (if the flag is set)
- `$BLACKBOX_CONFIGDIR` (the preferred env. variable to use)
- `$BLACKBOXDATA` (for backwards compatibility with v1)
- `.blackbox`
- `keyrings/live` (for backwards compatibility)
NOTE: The env variables and `--config` should be set to the full path
to the config directory (i.e.: `/Users/tom/gitstuff/myrepo/.blackbox`).
If it is set to a relative directory (i.e. `.blackbox` or
`../myrepo/.blackbox`) most commands will break.
NOTE: Why the change from `$BLACKBOXDATA` to `$BLACKBOX_CONFIGDIR`? We want
all the env. variables to begin with the prefix `BLACKBOX_`. If v1
supported another name, that is still supported. If you are starting
with v2 and have no other users using v1, please use the `BLACKBOX_`
prefix.
================================================
FILE: docs/compatibility.md
================================================
Compatibility
=============
# Compatibility with Blackbox v1
The command names all changed from v1 to v2. The `binv2` directory
includes shell scripts that provide full backwards compatibility.
# Supported Architectures
Blackbox supports a plug-in archtecture to easily support multiple VCS
system. Current support is for:
## Supported VCS/DVCS systems
* git
* "none" (repo-less use is supported)
* WOULD LOVE VOLUNTEERS TO HELP ADD SUPPORT FOR: hg, svn, p4
## Supported GPG versions
* Git 1.x and 2.0
* Git 2.2 and higher
* WOULD LOVE VOLUNTEERS TO HELP ADD SUPPORT FOR:
golang.org/x/crypto/openpgp (this would make the code have no
external dependencies)
## Supported Operating systems
Blackbox should work on any Linux system with GnuPG installed.
Blackbox simply looks for `gpg` in `$PATH`.
Windows: It should work (but has not been extensively tested) on
Windows WSL2.
# Automated testing
While many combinations work, we do automated tests
on these combinations. If any of these fail it blocks the release:
* macOS: GnuPG 2.2 executables from https://gpgtools.org/
* CentOS: GnuPG 2.0.x executables from the "base" or "updates" repo.
Windows native: VOLUNTEER NEEDED to make a native Windows version
(should be rather simple as Go does most of the work)
NOTE: Version 1 worked on CentOS/RedHat, macOS, Gygwin, WinGW, NetBSD,
and SmartOS. Hopefully we can achieve that broad level of support in
the future. Any system that is supported by the Go language and
has GuPG 2.0.x or higher binaries available should be easy to achieve.
We'd also like to have automated testing for the same.
# Windows Support
BlackBox assumes that `blackbox-admins.txt` and `blackbox-files.txt` will have
LF line endings. Windows users should be careful to configure Git or other systems
to not convert or "fix" those files.
If you use Git, add the following lines to your `.gitattributes` file:
**/blackbox-admins.txt text eol=lf
**/blackbox-files.txt text eol=lf
The `blackbox init` (and newer versions of `blackbox_initialize`)
will create an appropriate `.gitattributes` file for you.
# Cygwin
TODO: List what packages are required for building the software.
TODO: List what packages are required for running the software.
# MinGW
MinGW (comes with Git for Windows) support requires the following:
TODO: FILL IN any requirements
================================================
FILE: docs/dev-code-overview.md
================================================
Code Overview
=============
Here is how the code is laid out.
TODO(tlim): Add a diagram of the layers
```
cmd/blackbox/ The command line tool.
blackbox.go main()
cli.go Definition of all subcommands and flags
drive.go Processes flags and calls functions in verbs.go
NOTE: These are the only files that are aware of the
flags. Everything else gets the flag data passed to it
as a parameter. This way the remaining system can be
used as a module.
pkg/box/ High-level functions related to "the black box".
verbs.go One function per subcommand.
box.go Functions for manipulating the files in .blackbox
boxutils.go Helper functions for the above.
pkg/bblog/ Module that provides logging facilities.
pkg/bbutil/ Functions that are useful to box, plug-ins, etc.
pkg/tainedname/ Module for printing filenames escaped for Bash.
models/vcs.go The interface that defines a VCS plug-in.
models/crypters.go The interface that defines a GPG plug-in.
pkg/crypters/ Plug-ins for GPG functionality.
pkg/crypters/gnupg Plug-in that runs an external gpg binary (found via $PATH)
pkg/vcs/ Plug-ins for VCS functionality.
pkg/vcs/none Repo-less mode.
pkg/vcs/git Git mode.
```
================================================
FILE: docs/dev.md
================================================
Developer Info
==============
Code submissions are gladly welcomed! The code is fairly easy to read.
Get the code:
```
git clone git@github.com:StackExchange/blackbox.git
```
Test your changes:
```
go test ./...
```
This runs through a number of system tests. It creates a repo,
encrypts files, decrypts files, and so on. You can run these tests to
verify that the changes you made didn't break anything. You can also
use these tests to verify that the system works with a new operating
system.
Please submit tests with code changes:
The best way to change BlackBox is via Test Driven Development. First
add a test to `tools/confidence.sh`. This test should fail, and
demonstrate the need for the change you are about to make. Then fix
the bug or add the feature you want. When you are done, `make
confidence` should pass all tests. The PR you submit should include
your code as well as the new test. This way the confidence tests
accumulate as the system grows as we know future changes don't break
old features.
Note: More info about compatibility are on the [Compatibility Page](compatibility.md)
================================================
FILE: docs/enable-repo.md
================================================
Enabling Blackbox on a Repo
===========================
Overview:
1. Run the initialization command
2. Add at least one admin.
3. Add files. (don't add files before the admins)
The long version:
1. If you don't have a GPG key, set it up using instructions such as:
[Set up GPG key](https://help.github.com/articles/generating-a-new-gpg-key/). \
Now you are ready to go.
1. `cd` into a Git, Mercurial, Subversion or Perforce repository and run `blackbox init`.
1. Add yourself with `blackbox admin add YOUR@EMAIL`
1. Commit the files as directed.
That's it!
At this point you should encrypt a file and make sure you can decrypt
it. This verifies that everything is working as expected.
1. Pick a file to be encrypted. Since this is a test, you might want
to create a test file. Call it `secret.txt` and edit the file
so that it includes your mother's maiden name. Just kidding!
Store this sentence: `This is my test file.`
2. Run `blackbox file add secret.txt`
3. Decode the encrypted version: `blackbox cat secret.txt`
The "cat" subcommand only accesses the encrypted (`.gpg`) file and is
a good way to see that the file was encrypted properly. You should
see `This is my test file.`
4 Verify that editing the file works.
To view and/or edit a file, run `blackbox edit --shred secret.txt`
Now encrypt it and shred the original:
```
blackbox encrypt --shred secret.txt
```
Now make sure you can decrypt the new file:
```
blackbox cat secret.txt
```
You should see the changed text.
Now commit and push `secret.txt.gpg` and you are done!
================================================
FILE: docs/encryption.md
================================================
How is the encryption done?
===========================
GPG has many different ways to encrypt a file. BlackBox uses the mode
that lets you specify a list of keys that can decrypt the message.
If you have 5 people ("admins") that should be able to access the
secrets, each creates a GPG key and adds their public key to the
keychain. The GPG command used to encrypt the file lists all 5 key
names, and therefore any 1 key can decrypt the file.
Blackbox stores a copy of the public keys of all admins. It never
stores the private keys.
To remove someone's access, remove that admin's key name (i.e. email
address) from the list of admins and re-encrypt all the files. They
can still read the .gpg file (assuming they have access to the
repository) but they can't decrypt it any more.
*What if they kept a copy of the old repo before you removed access?*
Yes, they can decrypt old versions of the file. This is why when an
admin leaves the team, you should change all your passwords, SSL
certs, and so on. You should have been doing that before BlackBox,
right?
*Why don't you use symmetric keys?* In other words, why mess with all
this GPG key stuff and instead why don't we just encrypt all the files
with a single passphrase. Yes, GPG supports that, but then we are
managing a shared password, which is fraught with problems. If someone
"leaves the team" we would have to communicate to everyone a new
password. Now we just have to remove their key. This scales better.
*How do automated processes decrypt without asking for a password?*
GPG requires a passphrase on a private key. However, it permits the
creation of subkeys that have no passphrase. For automated processes,
create a subkey that is only stored on the machine that needs to
decrypt the files. For example, at Stack Exchange, when our Continuous
Integration (CI) system pushes a code change to our Puppet masters,
they run `blackbox decrypt --all --overwrite` to decrypt all the files.
The user that
runs this code has a subkey that doesn't require a passphrase. Since
we have many masters, each has its own key. And, yes, this means our
Puppet Masters have to be very secure. However, they were already
secure because, like, dude... if you can break into someone's puppet
master you own their network.
*If you use Puppet, why didn't you just use hiera-eyaml?* There are 4
reasons:
1. This works with any Git or Mercurial repo, even if you aren't using Puppet.
2. hiera-eyaml decrypts "on demand" which means your Puppet Master now uses a lot of CPU to decrypt keys every time it is contacted. It slows down your master, which, in my case, is already slow enough.
3. This works with binary files, without having to ASCIIify them and paste them into a YAML file. Have you tried to do this with a cert that is 10K long and changes every few weeks? Ick.
4. hiera-eyaml didn't exist when I wrote this. (That's the real reason.)
================================================
FILE: docs/expired-keys.md
================================================
Replacing expired keys
======================
If someone's key has already expired, blackbox will stop
encrypting. You see this error:
```
$ blackbox_edit_end modified_file.txt
--> Error: can't re-encrypt because a key has expired.
```
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
You can also detect keys that are about to expire by issuing this command and manually reviewing the "expired:" dates:
gpg --homedir=.blackbox --list-keys
or... list UIDs that will expire within 1 month from today: (Warning: this also lists keys without an expiration date)
gpg --homedir=.blackbox --list-keys --with-colons --fixed-list-mode | grep ^uid | awk -F: '$6 < '$(( $(date +%s) + 2592000))
Here's how to replace the key:
- Step 1. Administrator removes expired user:
Warning: This process will erase any unencrypted files that you were in the process of editing. Copy them elsewhere and restore the changes when done.
```
blackbox_removeadmin expired_user@example.com
# This next command overwrites any changed unencrypted files. See warning above.
blackbox_update_all_files
git commit -m "Re-encrypt all files"
gpg --homedir=.blackbox --delete-key expired_user@example.com
git commit -m 'Cleaned expired_user@example.com from keyring' .blackbox/*
git push
```
- Step 2. Expired user adds an updated key:
```
git pull
blackbox_addadmin updated_user@example.com
git commit -m'NEW ADMIN: updated_user@example.com .blackbox/pubring.gpg .blackbox/trustdb.gpg .blackbox/blackbox-admins.txt
git push
```
- Step 3. Administrator re-encrypts all files with the updated key of the expired user:
```
git pull
gpg --import .blackbox/pubring.gpg
blackbox_update_all_files
git commit -m "Re-encrypt all files"
git push
```
- Step 4: Clean up:
Any files that were temporarily copied in the first step so as to not be overwritten can now be copied back and re-encrypted with the `blackbox_edit_end` command.
(Thanks to @chishaku for finding a solution to this problem!)
================================================
FILE: docs/file-ops.md
================================================
How to add/remove a file into the system?
=========================================
# Adding files:
- If you need to, start the GPG Agent: `eval $(gpg-agent --daemon)`
- Add the file to the system:
```
blackbox file add path/to/file.name.key
# If you want to delete the old plaintext:
blackbox file add --shred path/to/file.name.key
```
Multiple file names can be specified on the command line:
Example 1: Register 2 files:
```
blackbox file add --shred file1.txt file2.txt
```
Example 2: Register all the files in `$DIR`:
```
find $DIR -type f -not -name '*.gpg' -print0 | xargs -0 blackbox file add
```
# Removing files
This command
```
blackbox file remove path/to/file.name.key
```
TODO(tlim): Add examples.
# List files
To see what files are currently enrolled in the system:
```
blackbox file list
```
You can also see their status:
```
blackbox status
blackbox status just_one_file.txt
blackbox status --type ENCRYPTED
```
================================================
FILE: docs/full-command-list.md
================================================
Blackbox Command List
=====================
## Global Flags
### `--vcs`
### `--crypto`
### `--config`
### `--team`
### `--editor`
### `--umask`
### `--debug`
### `--help`
### `--help`
### `--version`
## User Commands
### `blackbox decrypt`
### `blackbox encrypt`
### `blackbox edit`
### `blackbox cat`
### `blackbox diff`
### `blackbox shred`
### `blackbox help`
## User Commands
### `blackbox init`
### `blackbox admin`
### `blackbox file`
### `blackbox status`
### `blackbox reencrypt`
## Debug
### `blackbox info`
## Integration Test (secret menu)
### `blackbox testing_init`
TODO(tlim): Can we automatically generate this? The data is all in cli.go
================================================
FILE: docs/git-tips.md
================================================
GIT tips
========
# Configure git to show diffs in encrypted files
It's possible to tell Git to decrypt versions of the file before running them through `git diff` or `git log`. To achieve this do:
- Add the following to `.gitattributes` at the top of the git repository:
```
*.gpg diff=blackbox
```
- Add the following to `.git/config`:
```
[diff "blackbox"]
textconv = gpg --use-agent -q --batch --decrypt
````
Commands like `git log -p file.gpg` and `git diff master --` will display as expected.
================================================
FILE: docs/gnupg-tips.md
================================================
GnuPG tips
==========
# Common error messages
* Message: `gpg: filename: skipped: No public key`
* Solution: Usually this means there is an item in
`.blackbox/blackbox-admins.txt` that is not the name of the key.
Either something invalid was inserted (like a filename instead of a
username) or a user has left the organization and their key was
removed from the keychain, but their name wasn't removed from the
blackbox-admins.txt file.
* Message: `gpg: decryption failed: No secret key`
* Solution: Usually means you forgot to re-encrypt the file with the new key.
* Message: `Error: can't re-encrypt because a key has expired.`
* Solution: A user's key has expired and can't be used to encrypt any more. Follow the [Replace expired keys](expired-keys.md) page.
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
# GnuPG problems
Blackbox is just a front-end to GPG. If you get into a problem with a
key or file, you'll usually have better luck asking for advice on
the gnupg users mailing list TODO: Get link to this list
The author of Blackbox is not a GnuPG expert. He wrote Blackbox
because it was better than trying to remember GPG's horrible flag
names.
================================================
FILE: docs/installation.md
================================================
Installation Instructions
=========================
Currently blackbox v2 is installed by compiling the code and
copying the binary someplace:
TODO:
```
git clone FILL IN
```
Future: We will have RPM, DEB, Chocolately packages.
Next step: [Enable on a repo](enable-repo.md)
================================================
FILE: docs/role-accounts.md
================================================
Set up automated users or "role accounts"
=========================================
TODO(tlim): I think this is overly complex. With GnuPG 2.2 and later,
you can use `--password '' --quick-generate-key userid` and you are
done. No need for subkeys. Maybe rework this?
With role accounts, you have an automated system that needs to be able
to decrypt secrets without a password. This means the security of your
repo is based on how locked down the automation system is. This
is risky, so be careful.
i.e. This is how a Puppet Master can have access to the unencrypted data.
FYI: Your repo may use `keyrings/live` instead of `.blackbox`. See "Where is the configuration stored?"
An automated user (a "role account") is one that that must be able to decrypt without a passphrase. In general you'll want to do this for the user that pulls the files from the repo to the master. This may be automated with Jenkins CI or other CI system.
GPG keys have to have a passphrase. However, passphrases are optional on subkeys. Therefore, we will create a key with a passphrase then create a subkey without a passphrase. Since the subkey is very powerful, it should be created on a very secure machine.
There's another catch. The role account probably can't check files into Git/Mercurial. It probably only has read-only access to the repo. That's a good security policy. This means that the role account can't be used to upload the subkey public bits into the repo.
Therefore, we will create the key/subkey on a secure machine as yourself. From there we can commit the public portions into the repo. Also from this account we will export the parts that the role account needs, copy them to where the role account can access them, and import them as the role account.
ProTip: If asked to generate entropy, consider running this on the same machine in another window: `sudo dd if=/dev/sda of=/dev/null`
For the rest of this doc, you'll need to make the following substitutions:
- ROLEUSER: svc_deployacct or whatever your role account's name is.
- NEWMASTER: the machine this role account exists on.
- SECUREHOST: The machine you use to create the keys.
NOTE: This should be more automated/scripted. Patches welcome.
On SECUREHOST, create the puppet master's keys:
```
$ mkdir /tmp/NEWMASTER
$ cd /tmp/NEWMASTER
$ gpg --homedir . --gen-key
Your selection?
(1) RSA and RSA (default)
What keysize do you want? (2048) DEFAULT
Key is valid for? (0) DEFAULT
# Real name: Puppet CI Deploy Account
# Email address: svc_deployacct@hostname.domain.name
```
NOTE: Rather than a real email address, use the username@FQDN of the host the key will be used on. If you use this role account on many machines, each should have its own key. By using the FQDN of the host, you will be able to know which key is which. In this doc, we'll refer to username@FQDN as $KEYNAME
Save the passphrase somewhere safe!
Create a sub-key that has no password:
```
$ gpg --homedir . --edit-key svc_deployacct
gpg> addkey
(enter passphrase)
Please select what kind of key you want:
(3) DSA (sign only)
(4) RSA (sign only)
(5) Elgamal (encrypt only)
(6) RSA (encrypt only)
Your selection? 6
What keysize do you want? (2048)
Key is valid for? (0)
Command> key 2
(the new subkey has a "*" next to it)
Command> passwd
(enter the main key's passphrase)
(enter an empty passphrase for the subkey... confirm you want to do this)
Command> save
```
Now securely export this directory to NEWMASTER:
```
gpg --homedir . --export -a svc_sadeploy >/tmp/NEWMASTER/pubkey.txt
tar cvf /tmp/keys.tar .
rsync -avP /tmp/keys.tar NEWMASTER:/tmp/.
```
On NEWMASTER, receive the new GnuPG config:
```
sudo -u svc_deployacct bash
mkdir -m 0700 -p ~/.gnupg
cd ~/.gnupg && tar xpvf /tmp/keys.tar
```
<!---
Back on SECUREHOST, import the pubkey into the repository.
```
$ cd .blackbox
$ gpg --homedir . --import /tmp/NEWMASTER/pubkey.txt
```
-->
Back on SECUREHOST, add the new email address to .blackbox/blackbox-admins.txt:
```
cd /path/to/the/repo
blackbox_addadmin $KEYNAME /tmp/NEWMASTER
```
Verify that secring.gpg is a zero-length file. If it isn't, you have somehow added a private key to the keyring. Start over.
```
cd .blackbox
ls -l secring.gpg
```
Commit the recent changes:
```
cd .blackbox
git commit -m"Adding key for KEYNAME" pubring.gpg trustdb.gpg blackbox-admins.txt
```
Regenerate all encrypted files with the new key:
```
blackbox_update_all_files
git status
git commit -m"updated encryption" -a
git push
```
On NEWMASTER, import the keys and decrypt the files:
```
sudo -u svc_sadeploy bash # Become the role account.
gpg --import /etc/puppet/.blackbox/pubring.gpg
export PATH=$PATH:/path/to/blackbox/bin
blackbox_postdeploy
sudo -u puppet cat /etc/puppet/hieradata/blackbox.yaml # or any encrypted file.
```
ProTip: If you get "gpg: decryption failed: No secret key" then you forgot to re-encrypt blackbox.yaml with the new key.
On SECUREHOST, securely delete your files:
```
cd /tmp/NEWMASTER
# On machines with the "shred" command:
shred -u /tmp/keys.tar
find . -type f -print0 | xargs -0 shred -u
# All else:
rm -rf /tmp/NEWMASTER
```
Also shred any other temporary files you may have made.
================================================
FILE: docs/subversion-tips.md
================================================
Subversion Tips
===============
NOTE: This is from v1. Can someone that uses Subversion check
this and update it?
The current implementation will store the blackbox in `/keyrings` at
the root of the entire repo. This will create an issue between
environments that have different roots (i.e. checking out `/` on
development vs `/releases/foo` in production). To get around this, you
can `export BLACKBOX_REPOBASE=/path/to/repo` and set a specific base
for your repo.
This was originally written for git and supports a two-phase commit,
in which `commit` is a local commit and "push" sends the change
upstream to the version control server when something is registered or
deregistered with the system. The current implementation will
immediately `commit` a file (to the upstream subversion server) when
you execute a `blackbox_*` command.
================================================
FILE: docs/support.md
================================================
Support
=======
# Join our community!
Join the [blackbox-project mailing list](https://groups.google.com/d/forum/blackbox-project)!
# How to submit bugs or ask questions?
We welcome questions, bug reports and feedback!
The best place to start is to join the [blackbox-project mailing list](https://groups.google.com/d/forum/blackbox-project) and ask there.
Bugs are tracked here in Github. Please feel free to [report bugs](https://github.com/StackExchange/blackbox/issues) yourself.
================================================
FILE: docs/user-overview.md
================================================
User Guide
==========
# Overview
Suppose you have a VCS repository (i.e. a Git or Mercurial repo) and
certain files contain secrets such as passwords or SSL private keys.
Often people just store such files "and hope that nobody finds them in
the repo". That's not safe. Hope is not a strategy.
With BlackBox, those files are stored encrypted using GPG. Access to
the repo without also having the right GPG keys makes those files as worthless
as random bits. As long as you keep your GPG keys safe, you don't
have to worry about storing your VCS repo on an untrusted server or
letting anyone clone the repo.
Heck, even if you trust your server, now you don't have to trust the
people that do backups of that server!
Each person ("admin") of the system can decrypt all the files using
their GPG key, which has its own passphrase. The authorized GPG keys
can decrypt any file. This is better than systems that use one
GPG key (and passphrase) that must be shared among a group of people.
It is much better than having one passphrase for each file (I don't
think anyone actually does that).
Since any admin's GPG key can decrypt the files, if one person leaves
the company, you don't have to communicate a new passphrase to everyone.
Simply disable the one key that should no longer have access.
The process for doing this is as easy as running 2 commands (1 to
disable their key, 1 to re-encrypt all files.) Obviously if they kept
a copy of the repo (and their own passphrase) before leaving the
company, they have access to the secrets. However, you should rotate
those secrets anyway. ("rotate secrets" means changing the passwords,
regenerating TLS certs, and so on).
# Sample session:
First we are going to list the files currently in the blackbox. In
this case, it is an SSH private key.
```
$ blackbox file list
modules/log_management/files/id_rsa
```
Excellent! Our coworkers have already registered a file with the
system. Let's decrypt it, edit it, and re-encrypt it.
```
$ blackbox decrypt modules/log_management/files/id_rsa
========== DECRYPTING "modules/log_management/files/id_rsa"
$ vi modules/log_management/files/id_rsa
```
That was easy so far!
When we encrypt it, Blackbox will not commit the changes, but it
will give a hint that you should. It spells out the exact command you
need to type and even proposes a commit message.
```
$ blackbox encrypt modules/log_management/files/id_rsa
========== ENCRYPTING "modules/log_management/files/id_rsa"
NEXT STEP: You need to manually check these in:
git commit -m"ENCRYPTED modules/log_management/files/id_rsa" modules/log_management/files/id_rsa.gpg
```
You can also use `blackbox edit <filename>` to decrypt a file, edit it
(it will call `$EDITOR`) and re-encrypt it.
Now let's register a new file with the blackbox system.
`data/pass.yaml` is a small file that stores a very important
password. In this example, we had just stored the unecrypted
password in our repo. That's bad. Let's encrypt it.
```
$ blackbox file add data/pass.yaml
========== SHREDDING ("/bin/rm", "-f"): "data/pass.yaml"
NEXT STEP: You need to manually check these in:
git commit -m"NEW FILES: data/pass.yaml" .gitignore keyrings/live/blackbox-files.txt modules/stacklb/pass.yaml modules/stacklb/pass.yaml.gpg
```
Before we commit the change, let's do a `git status` to see what else
has changed.
```
$ git status
On branch master
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: .gitignore
modified: keyrings/live/blackbox-files.txt
deleted: modules/stacklb/pass.yaml
new file: modules/stacklb/pass.yaml.gpg
```
Notice that a number of files were modified:
* `.gitignore`: This file is updated to include the plaintext
filename, so that you don't accidentally add it to the repo in the
future.
* `.blackbox/blackbox-files.txt`: The list of files that are registered with the system.
* `data/pass.yaml`: The file we encrypted is deleted from the repo.
* `data/pass.yaml.gpg`: The encrypted file is added to the repo.
Even though pass.yaml was deleted from the repo, it is still in the
repo's history. Anyone with an old copy of the repo, or a new copy
that knows how to view the repo's history, can see the secret
password. For that reason, you should change the password and
re-encrypt the file. This is an important point. Blackbox is not
magic and it doesn't have a "Men In Black"-style neuralizer that
can make people forget the past. If someone leaves a project, you
have to change the old passwords, etc.
Those are the basics. Your next step might be:
* TODO: How to enable Blackbox for a repo.
* TODO: How to add yourself as an admin to a repo.
* TODO: Complete list of [all blackbox commands](all-commands)
================================================
FILE: docs/why-is-this-important.md
================================================
Why encrypt your secrets?
=========================
OBVIOUSLY we don't want secret things like SSL private keys and
passwords to be leaked.
NOT SO OBVIOUSLY when we store "secrets" in a VCS repo like Git or
Mercurial, suddenly we are less able to share our code with other
people. Communication between subteams of an organization is hurt. You
can't collaborate as well. Either you find yourself emailing
individual files around (yuck!), making a special repo with just the
files needed by your collaborators (yuck!!), or just deciding that
collaboration isn't worth all that effort (yuck!!!).
The ability to be open and transparent about our code, with the
exception of a few specific files, is key to the kind of collaboration
that DevOps and modern IT practitioners need to do.
================================================
FILE: docs/with-ansible.md
================================================
How to use the secrets with Ansible?
===================================
Ansible Vault provides functionality for encrypting both entire files
and strings stored within files; however, keeping track of the
password(s) required for decryption is not handled by this module.
Instead one must specify a password file when running the playbook.
Ansible example for password file: `my_secret_password.txt.gpg`
```
ansible-playbook --vault-password-file my_secret_password.txt site.yml
```
Alternatively, one can specify this in the
`ANSIBLE_VAULT_PASSWORD_FILE` environment variable.
================================================
FILE: docs/with-puppet.md
================================================
How to use the secrets with Puppet?
===================================
# Entire files:
Entire files, such as SSL certs and private keys, are treated just
like regular files. You decrypt them any time you push a new release
to the puppet master.
Example of an encrypted file named `secret_file.key.gpg`
* Plaintext file is: `modules/${module_name}/files/secret_file.key`
* Encrypted file is: `modules/${module_name}/files/secret_file.key.gpg`
* Puppet sees it as: `puppet:///modules/${module_name}/secret_file.key`
Puppet code that stores `secret_file.key` in `/etc/my_little_secret.key`:
```
file { '/etc/my_little_secret.key':
ensure => 'file',
owner => 'root',
group => 'puppet',
mode => '0760',
source => "puppet:///modules/${module_name}/secret_file.key", # No ".gpg"
}
```
# Small strings:
For small strings such as passwords and API keys, it makes sense
to store them in an (encrypted) YAML file which is then made
available via hiera.
For example, we use a file called `blackbox.yaml`. You can access the
data in it using the hiera() function.
*Setup:*
Edit `hiera.yaml` to include "blackbox" to the search hierarchy:
```
:hierarchy:
- ...
- blackbox
- ...
```
In blackbox.yaml specify:
```
---
module::test_password: "my secret password"
```
In your Puppet Code, access the password as you would any hiera data:
```
$the_password = hiera('module::test_password', 'fail')
file {'/tmp/debug-blackbox.txt':
content => $the_password,
owner => 'root',
group => 'root',
mode => '0600',
}
```
The variable `$the_password` will contain "my secret password" and can be used anywhere strings are used.
================================================
FILE: go.mod
================================================
module github.com/StackExchange/blackbox/v2
go 1.14
require (
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883
github.com/mattn/go-runewidth v0.0.9 // indirect
github.com/olekukonko/tablewriter v0.0.4
github.com/sergi/go-diff v1.2.0 // indirect
github.com/urfave/cli/v2 v2.2.0
)
================================================
FILE: go.sum
================================================
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883 h1:bvNMNQO63//z+xNgfBlViaCIJKLlCJ6/fmUseuG0wVQ=
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d h1:U+s90UTSYgptZMwQh2aRr3LuazLJIa+Pg3Kc1ylSYVY=
github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/mattn/go-runewidth v0.0.7/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.9 h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/QdE+0=
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/olekukonko/tablewriter v0.0.4 h1:vHD/YYe1Wolo78koG299f7V/VAS08c6IpCLn+Ejf/w8=
github.com/olekukonko/tablewriter v0.0.4/go.mod h1:zq6QwlOf5SlnkVbMSr5EoBv3636FWnp+qbPhuoO21uA=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/russross/blackfriday/v2 v2.0.1 h1:lPqVAte+HuHNfhJ/0LC98ESWRz8afy9tM/0RK8m9o+Q=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ=
github.com/sergi/go-diff v1.2.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
github.com/shurcooL/sanitized_anchor_name v1.0.0 h1:PdmoCO6wvbs+7yrJyMORt4/BmY5IYyJwS/kOiWx8mHo=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/urfave/cli/v2 v2.2.0 h1:JTTnM6wKzdA0Jqodd966MVj4vWbbquZykeX1sKbe2C4=
github.com/urfave/cli/v2 v2.2.0/go.mod h1:SE9GqnLQmjVa0iPEY0f1w3ygNIYcIJ0OKPMoW2caLfQ=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
================================================
FILE: integrationTest/NOTES.txt
================================================
This should accept VCS-type and --crypto flags.
Then a shell script should run various combinations of VCS and crypters.
# Startup
* Create a repo (git, none)
# Test basic operations:
* As Alice:
* initialize blackbox, add her keys to it, see that the usual files
exist. See her name in bb-admins.txt
* encrypt a file, see that the plaintext is deleted, see the file in bb-files.txt
* decrypt the file, see the original plaintext is recovered.
* Encrypt a file --noshred.
* Decrypt the file, it should fail as the plaintext exists.
* Remove the plaintext.
* Decrypt the file, it should fail as the plaintext exists.
# Test hand-off from Alice to Bob.
* As Bob
* add himself to the admins.
* As Alice
* Update-all-files
* Create a new file. Encrypt it.
* As Bob
* Decrypt both files
* Verify contents of the new file, and the file from previous.
* Create a new file. Encrypt it.
* As Alice:
* Decrypt all files.
* Verify contents of the 3 plaintext files.
# Test a git-less directory
* Copy the old repo somewhere. Remove the .git directory.
* As Alice:
* Decrypt all
* Verify plaintext contents
# Test post-deploy with/without GID
* Back at the original repo:
* Shred all
* Run post-deploy. Verify.
* Shred all
* Run post-deploy with a custom GID. Verify.
# Test removing an admin
* As Bob:
* removes Alice. (Verify)
* Re-encrypt
* Decrypt all & verify.
* As alice
* Decrypting should fail.
# Test funny names and paths
* my/path/to/relsecrets.txt
* cwd=other/place ../../my/path/to/relsecrets.txt
* !important!.txt
* #andpounds.txt
* stars*bars?.txt
* space space.txt
* Do add/encrypt/decrypt
* Do blackbox_update_all_files
* Do remove them all
# When people start asking for commands to work with relative paths
# Test from outside the repo
* mkdir ../other/place
* cd ../other/place
* decrypt ../../secret1.txt
* encrypt ../../secret1.txt
# Test specific commands:
# blackbox admins list
# blackbox file list
# blackbox status --name-only (create 1 of each "type")
# blackbox status --type=FOO
# These should all fail:
# blackbox file list --all
# blackbox file list blah
# blackbox shred list --all
# blackbox shred list blah
rm -rf /tmp/bbhome-* && BLACKBOX_DEBUG=true go test -verbose -long -nocleanup
rm -rf /tmp/bbhome-* && go test -long -nocleanup
( gbb && cd cmd/blackbox && go install ) && blackbox
cd /tmp && rm -rf /tmp/bbhome-* ; mkdir /tmp/bbhome-1 ; cd /tmp/bbhome-1 && git init ; gitmeWork ; ( gbb && cd cmd/blackbox && go install ) && blackbox init yes && gitmeWork ; git commit -mm -a ; blackbox admin add tlimoncelli ; git commit -mnewadmin -a ; echo secrt > secret.txt ; blackbox file add secret.txt
================================================
FILE: integrationTest/README.txt
================================================
Each test does the following:
1. Copy the files from testdata/NNNN
2. Run the command in test_NNNN.sh
3.
TEST ENROLLMENT:
PHASE 'Alice creates a repo. She creates secret.txt.'
PHASE 'Alice wants to be part of the secret system.'
PHASE 'She creates a GPG key...'
PHASE 'Initializes BB...'
PHASE 'and adds herself as an admin.'
PHASE 'Bob arrives.'
PHASE 'Bob creates a gpg key.'
PHASE 'Alice does the second part to enroll bob.'
PHASE 'She enrolls bob.'
PHASE 'She enrolls secrets.txt.'
PHASE 'She decrypts secrets.txt.'
PHASE 'She edits secrets.txt.'
PHASE 'Alice copies files to a non-repo directory. (NO REPO)'
PHASE 'Alice shreds these non-repo files. (NO REPO)'
PHASE 'Alice decrypts secrets.txt (NO REPO).'
PHASE 'Alice edits secrets.txt. (NO REPO EDIT)'
PHASE 'Alice decrypts secrets.txt (NO REPO EDIT).'
PHASE 'appears.'
#PHASE 'Bob makes sure he has all new keys.'
TEST INDIVIDUAL COMMANDS:
PHASE 'Bob postdeploys... default.'
PHASE 'Bob postdeploys... with a GID.'
PHASE 'Bob cleans up the secret.'
PHASE 'Bob removes Alice.'
PHASE 'Bob reencrypts files so alice can not access them.'
PHASE 'Bob decrypts secrets.txt.'
PHASE 'Bob edits secrets.txt.'
PHASE 'Bob decrypts secrets.txt VERSION 3.'
PHASE 'Bob exposes a secret in the repo.'
PHASE 'Bob corrects it by registering it.'
PHASE 'Bob enrolls my/path/to/relsecrets.txt.'
PHASE 'Bob decrypts relsecrets.txt.'
PHASE 'Bob enrolls !important!.txt'
PHASE 'Bob enrolls #andpounds.txt'
PHASE 'Bob enrolls stars*bars?.txt'
PHASE 'Bob enrolls space space.txt'
PHASE 'Bob checks out stars*bars?.txt.'
PHASE 'Bob checks out space space.txt.'
PHASE 'Bob shreds all exposed files.'
PHASE 'Bob updates all files.'
PHASE 'Bob DEregisters mistake.txt'
PHASE 'Bob enrolls multiple files: multi1.txt and multi2.txt'
PHASE 'Alice returns. She should be locked out'
PHASE 'Alice tries to decrypt secret.txt. Is blocked.'
================================================
FILE: integrationTest/asserts.go
================================================
package main
import (
"io/ioutil"
"os"
"testing"
"github.com/andreyvit/diff"
)
func assertFileMissing(t *testing.T, name string) {
t.Helper()
_, err := os.Stat(name)
if err != nil && os.IsNotExist(err) {
return
}
if err == nil {
t.Fatalf("assertFileMissing failed: %v exists", name)
}
t.Fatalf("assertFileMissing: %q: %v", name, err)
}
func assertFileExists(t *testing.T, name string) {
t.Helper()
_, err := os.Stat(name)
if err == nil {
return
}
if os.IsNotExist(err) {
t.Fatalf("assertFileExists failed: %v not exist", name)
}
t.Fatalf("assertFileExists: file can't be accessed: %v: %v", name, err)
}
func assertFileEmpty(t *testing.T, name string) {
t.Helper()
c, err := ioutil.ReadFile(name)
if err != nil {
t.Fatal(err)
}
if len(c) != 0 {
t.Fatalf("got=%v want=%v: %v", len(c), 0, name)
}
}
func assertFileContents(t *testing.T, name string, contents string) {
t.Helper()
c, err := ioutil.ReadFile(name)
if err != nil {
t.Fatal(err)
}
if w, g := contents, string(c); w != g {
t.Errorf("assertFileContents(%q) mismatch (-got +want):\n%s",
name, diff.LineDiff(g, w))
}
}
func assertFilePerms(t *testing.T, name string, perms os.FileMode) {
t.Helper()
s, err := os.Stat(name)
if err != nil {
t.Fatal(err)
}
if s.Mode() != perms {
t.Fatalf("got=%#o want=%#o: %v", s.Mode(), perms, name)
}
}
================================================
FILE: integrationTest/integration_test.go
================================================
package main
import (
"flag"
"fmt"
"os"
"os/exec"
"path/filepath"
"testing"
"github.com/StackExchange/blackbox/v2/pkg/bblog"
_ "github.com/StackExchange/blackbox/v2/pkg/bblog"
_ "github.com/StackExchange/blackbox/v2/pkg/vcs/_all"
)
var vcsToTest = flag.String("testvcs", "GIT", "VCS to test")
var longTests = flag.Bool("long", false, "Run long version of tests")
//var crypterToTest = flag.String("crypter", "GnuPG", "crypter to test")
func init() {
testing.Init()
flag.Parse()
op, err := os.Getwd()
if err != nil {
panic(err)
}
originPath = op
}
func compile(t *testing.T) {
if PathToBlackBox() != "" {
// It's been compiled already.
return
}
// Make sure we have the latest binary
fmt.Println("========== Compiling")
cmd := exec.Command("go", "build", "-o", "../bbintegration", "../cmd/blackbox")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Run()
if err != nil {
t.Fatalf("setup_compile: %v", err)
}
cwd, err := os.Getwd()
if err != nil {
t.Fatal(err)
}
SetPathToBlackBox(filepath.Join(cwd, "../bbintegration"))
}
func setup(t *testing.T) {
logDebug := bblog.GetDebug(*verbose)
logDebug.Printf("flag.testvcs is %v", *vcsToTest)
vh := getVcs(t, *vcsToTest)
logDebug.Printf("Using BLACKBOX_VCS=%v", vh.Name())
os.Setenv("BLACKBOX_VCS", vh.Name())
}
func TestInit(t *testing.T) {
if !*longTests {
return
}
compile(t)
makeHomeDir(t, "init")
// Only zero or one args are permitted.
invalidArgs(t, "init", "one", "two")
invalidArgs(t, "init", "one", "two", "three")
runBB(t, "init", "yes")
assertFileEmpty(t, ".blackbox/blackbox-admins.txt")
assertFileEmpty(t, ".blackbox/blackbox-files.txt")
assertFilePerms(t, ".blackbox/blackbox-admins.txt", 0o640)
assertFilePerms(t, ".blackbox/blackbox-files.txt", 0o640)
}
func TestList(t *testing.T) {
if !*longTests {
return
}
compile(t)
makeHomeDir(t, "init")
runBB(t, "init", "yes")
createDummyFilesAdmin(t)
checkOutput("000-admin-list.txt", t, "admin", "list")
checkOutput("000-file-list.txt", t, "file", "list")
invalidArgs(t, "file", "list", "extra")
invalidArgs(t, "admin", "list", "extra")
}
func TestStatus(t *testing.T) {
if !*longTests {
return
}
compile(t)
makeHomeDir(t, "init")
runBB(t, "init", "yes")
createFilesStatus(t)
checkOutput("000-status.txt", t, "status")
}
func TestShred(t *testing.T) {
if !*longTests {
return
}
compile(t)
makeHomeDir(t, "shred")
runBB(t, "init", "yes")
makeFile(t, "shredme.txt", "File with SHREDME in it.\n")
assertFileExists(t, "shredme.txt")
runBB(t, "shred", "shredme.txt")
assertFileMissing(t, "shredme.txt")
}
func TestStatus_notreg(t *testing.T) {
if !*longTests {
return
}
compile(t)
makeHomeDir(t, "init")
runBB(t, "init", "yes")
createFilesStatus(t)
checkOutput("status-noreg.txt", t, "status", "status-ENCRYPTED.txt", "blah.txt")
}
// TestHard tests the functions using a fake homedir and repo.
func TestHard(t *testing.T) {
if !*longTests {
return
}
// These are basic tests that work on a fake repo.
// The repo has mostly real data, except any .gpg file
// is just garbage.
compile(t)
setup(t)
for _, cx := range []struct{ subname, prefix string }{
//{subname: ".", prefix: "."},
{subname: "mysub", prefix: ".."},
} {
subname := cx.subname
prefix := cx.prefix
_ = prefix
phase("========== SUBDIR = " + subname + " ==========")
makeHomeDir(t, "BasicAlice")
plaintextFoo := "I am the foo.txt file!\n"
plainAltered := "I am the altered file!\n"
runBB(t, "testing_init") // Runs "git init" or equiv
assertFileExists(t, ".git")
runBB(t, "init", "yes") // Creates .blackbox or equiv
if subname != "." {
err := os.Mkdir(subname, 0770)
if err != nil {
t.Fatal(fmt.Errorf("hard-mk-home %q: %v", subname, err))
}
}
olddir, err := os.Getwd()
if err != nil {
t.Fatal(err)
}
os.Chdir(subname)
os.Chdir(olddir)
phase("Alice creates a GPG key")
gpgdir := makeAdmin(t, "alice", "Alice Example", "alice@example.com")
become(t, "alice")
phase("Alice enrolls as an admin")
//os.Chdir(subname)
runBB(t, "admin", "add", "alice@example.com", gpgdir)
//os.Chdir(olddir)
// encrypt
phase("Alice registers foo.txt")
makeFile(t, "foo.txt", plaintextFoo)
//os.Chdir(subname)
//runBB(t, "file", "add", "--shred", filepath.Join(prefix, "foo.txt"))
runBB(t, "file", "add", "--shred", "foo.txt")
//os.Chdir(olddir)
// "file add" encrypts the file.
// We shred the plaintext so that we are s
gitextract_2zatdstu/
├── .circleci/
│ └── config.yml
├── .gitattributes
├── .github/
│ └── workflows/
│ ├── build.yml
│ └── release.yml
├── .gitignore
├── AUTHORS
├── CHANGELOG.md
├── DESIGN.md
├── LICENSE.txt
├── Makefile
├── README-v2.md
├── README.md
├── RELEASE_ENGINEERING.md
├── SPECS/
│ └── empty.spec
├── Version2-Ideas.md
├── bin/
│ ├── Makefile
│ ├── _blackbox_common.sh
│ ├── _blackbox_common_test.sh
│ ├── _stack_lib.sh
│ ├── blackbox_addadmin
│ ├── blackbox_cat
│ ├── blackbox_decrypt_all_files
│ ├── blackbox_decrypt_file
│ ├── blackbox_deregister_file
│ ├── blackbox_diff
│ ├── blackbox_edit
│ ├── blackbox_edit_end
│ ├── blackbox_edit_start
│ ├── blackbox_initialize
│ ├── blackbox_list_admins
│ ├── blackbox_list_files
│ ├── blackbox_listadmins
│ ├── blackbox_postdeploy
│ ├── blackbox_recurse
│ ├── blackbox_register_new_file
│ ├── blackbox_removeadmin
│ ├── blackbox_shred_all_files
│ ├── blackbox_update_all_files
│ ├── blackbox_view
│ └── blackbox_whatsnew
├── binv2/
│ ├── blackbox_addadmin
│ ├── blackbox_cat
│ ├── blackbox_decrypt_all_files
│ ├── blackbox_decrypt_file
│ ├── blackbox_deregister_file
│ ├── blackbox_diff
│ ├── blackbox_edit
│ ├── blackbox_edit_end
│ ├── blackbox_edit_start
│ ├── blackbox_initialize
│ ├── blackbox_list_admins
│ ├── blackbox_list_files
│ ├── blackbox_listadmins
│ ├── blackbox_postdeploy
│ ├── blackbox_register_new_file
│ ├── blackbox_removeadmin
│ ├── blackbox_shred_all_files
│ ├── blackbox_update_all_files
│ ├── blackbox_view
│ └── blackbox_whatsnew
├── blackbox.plugin.zsh
├── build/
│ └── build.go
├── cmd/
│ └── blackbox/
│ ├── blackbox.go
│ ├── cli.go
│ └── drive.go
├── docs/
│ ├── README.md
│ ├── admin-ops.md
│ ├── advanced.md
│ ├── alternatives.md
│ ├── backwards-compatibility.md
│ ├── compatibility.md
│ ├── dev-code-overview.md
│ ├── dev.md
│ ├── enable-repo.md
│ ├── encryption.md
│ ├── expired-keys.md
│ ├── file-ops.md
│ ├── full-command-list.md
│ ├── git-tips.md
│ ├── gnupg-tips.md
│ ├── installation.md
│ ├── role-accounts.md
│ ├── subversion-tips.md
│ ├── support.md
│ ├── user-overview.md
│ ├── why-is-this-important.md
│ ├── with-ansible.md
│ └── with-puppet.md
├── go.mod
├── go.sum
├── integrationTest/
│ ├── NOTES.txt
│ ├── README.txt
│ ├── asserts.go
│ ├── integration_test.go
│ ├── ithelpers.go
│ └── test_data/
│ ├── 000-admin-list.txt
│ ├── 000-file-list.txt
│ ├── 000-status.txt
│ ├── alice-cat-plain.txt
│ ├── basic-status.txt
│ ├── reencrypt-plain.txt
│ └── status-noreg.txt
├── models/
│ ├── crypters.go
│ └── vcs.go
├── pkg/
│ ├── bblog/
│ │ └── bblog.go
│ ├── bbutil/
│ │ ├── filestats.go
│ │ ├── rbio_test.go
│ │ ├── runbash.go
│ │ ├── shred.go
│ │ ├── sortedfile_test.go
│ │ ├── umask_posix.go
│ │ └── umask_windows.go
│ ├── box/
│ │ ├── box.go
│ │ ├── boxutils.go
│ │ ├── pretty_test.go
│ │ └── verbs.go
│ ├── commitlater/
│ │ └── commitlater.go
│ ├── crypters/
│ │ ├── _all/
│ │ │ └── all.go
│ │ ├── crypters.go
│ │ └── gnupg/
│ │ ├── gnupg.go
│ │ └── keychain.go
│ ├── makesafe/
│ │ ├── makesafe.go
│ │ └── makesafe_test.go
│ └── vcs/
│ ├── _all/
│ │ └── all.go
│ ├── git/
│ │ └── git.go
│ ├── none/
│ │ └── none.go
│ └── vcs.go
└── tools/
├── Portfile.template
├── auto_system_test
├── confidence_test.sh
├── macports_report_upgrade.sh
├── mk_deb_fpmdir
├── mk_deb_fpmdir.stack_blackbox.txt
├── mk_macports
├── mk_macports.vcs_blackbox.txt
├── mk_rpm_fpmdir
├── mk_rpm_fpmdir.stack_blackbox.txt
├── profile.d-usrblackbox-test.sh
├── profile.d-usrblackbox.sh
└── test_functions.sh
SYMBOL INDEX (205 symbols across 29 files)
FILE: build/build.go
function main (line 17) | func main() {
function getVersion (line 48) | func getVersion() string {
FILE: cmd/blackbox/blackbox.go
function main (line 15) | func main() {
FILE: cmd/blackbox/cli.go
function flags (line 13) | func flags() *cli.App {
FILE: cmd/blackbox/drive.go
function init (line 18) | func init() {
function allOrSomeFiles (line 24) | func allOrSomeFiles(c *cli.Context) error {
constant roError (line 34) | roError = `This command is disabled due to --config flag being used.
function cmdAdminAdd (line 44) | func cmdAdminAdd(c *cli.Context) error {
function cmdAdminList (line 60) | func cmdAdminList(c *cli.Context) error {
function cmdAdminRemove (line 72) | func cmdAdminRemove(c *cli.Context) error {
function cmdCat (line 87) | func cmdCat(c *cli.Context) error {
function cmdDecrypt (line 99) | func cmdDecrypt(c *cli.Context) error {
function cmdDiff (line 123) | func cmdDiff(c *cli.Context) error {
function cmdEdit (line 135) | func cmdEdit(c *cli.Context) error {
function cmdEncrypt (line 147) | func cmdEncrypt(c *cli.Context) error {
function cmdFileAdd (line 159) | func cmdFileAdd(c *cli.Context) error {
function cmdFileList (line 174) | func cmdFileList(c *cli.Context) error {
function cmdFileRemove (line 186) | func cmdFileRemove(c *cli.Context) error {
function cmdInfo (line 201) | func cmdInfo(c *cli.Context) error {
function cmdInit (line 213) | func cmdInit(c *cli.Context) error {
function cmdReencrypt (line 228) | func cmdReencrypt(c *cli.Context) error {
function cmdShred (line 251) | func cmdShred(c *cli.Context) error {
function cmdStatus (line 263) | func cmdStatus(c *cli.Context) error {
function testingInit (line 277) | func testingInit(c *cli.Context) error {
FILE: integrationTest/asserts.go
function assertFileMissing (line 11) | func assertFileMissing(t *testing.T, name string) {
function assertFileExists (line 23) | func assertFileExists(t *testing.T, name string) {
function assertFileEmpty (line 35) | func assertFileEmpty(t *testing.T, name string) {
function assertFileContents (line 46) | func assertFileContents(t *testing.T, name string, contents string) {
function assertFilePerms (line 59) | func assertFilePerms(t *testing.T, name string, perms os.FileMode) {
FILE: integrationTest/integration_test.go
function init (line 21) | func init() {
function compile (line 32) | func compile(t *testing.T) {
function setup (line 53) | func setup(t *testing.T) {
function TestInit (line 63) | func TestInit(t *testing.T) {
function TestList (line 81) | func TestList(t *testing.T) {
function TestStatus (line 97) | func TestStatus(t *testing.T) {
function TestShred (line 109) | func TestShred(t *testing.T) {
function TestStatus_notreg (line 123) | func TestStatus_notreg(t *testing.T) {
function TestHard (line 136) | func TestHard(t *testing.T) {
function TestEvilFilenames (line 271) | func TestEvilFilenames(t *testing.T) {
FILE: integrationTest/ithelpers.go
type userinfo (line 27) | type userinfo struct
function init (line 37) | func init() {
function init (line 45) | func init() {
function getVcs (line 50) | func getVcs(t *testing.T, name string) vcs.Vcs {
function makeHomeDir (line 70) | func makeHomeDir(t *testing.T, testname string) {
function createDummyFilesAdmin (line 109) | func createDummyFilesAdmin(t *testing.T) {
function createFilesStatus (line 122) | func createFilesStatus(t *testing.T) {
function addLineSorted (line 178) | func addLineSorted(t *testing.T, filename, line string) {
function removeFile (line 185) | func removeFile(t *testing.T, name string) {
function makeFile (line 189) | func makeFile(t *testing.T, name string, content string) {
function setFilePerms (line 198) | func setFilePerms(t *testing.T, name string, perms int) {
function checkOutput (line 211) | func checkOutput(name string, t *testing.T, args ...string) {
function invalidArgs (line 241) | func invalidArgs(t *testing.T, args ...string) {
function setupUser (line 261) | func setupUser(t *testing.T, user, passphrase string) {
function PathToBlackBox (line 269) | func PathToBlackBox() string { return pathToBlackBox }
function SetPathToBlackBox (line 272) | func SetPathToBlackBox(n string) {
function runBB (line 277) | func runBB(t *testing.T, args ...string) {
function phase (line 291) | func phase(msg string) {
function makeAdmin (line 299) | func makeAdmin(t *testing.T, name, fullname, email string) string {
function hasQuick (line 422) | func hasQuick(t *testing.T) bool {
function become (line 439) | func become(t *testing.T, name string) {
FILE: models/crypters.go
type Crypter (line 4) | type Crypter interface
FILE: models/vcs.go
type Vcs (line 6) | type Vcs interface
FILE: pkg/bblog/bblog.go
function GetErr (line 32) | func GetErr() *log.Logger {
function GetDebug (line 40) | func GetDebug(visible bool) *log.Logger {
FILE: pkg/bbutil/filestats.go
function DirExists (line 14) | func DirExists(path string) (bool, error) {
function FileExistsOrProblem (line 26) | func FileExistsOrProblem(path string) bool {
function Touch (line 38) | func Touch(name string) error {
function ReadFileLines (line 54) | func ReadFileLines(filename string) ([]string, error) {
function AddLinesToSortedFile (line 69) | func AddLinesToSortedFile(filename string, newlines ...string) error {
function AddLinesToFile (line 90) | func AddLinesToFile(filename string, newlines ...string) error {
function FindDirInParent (line 105) | func FindDirInParent(target string) (string, error) {
FILE: pkg/bbutil/rbio_test.go
function TestRunBashInputOutput (line 7) | func TestRunBashInputOutput(t *testing.T) {
FILE: pkg/bbutil/runbash.go
function RunBash (line 12) | func RunBash(command string, args ...string) error {
function RunBashOutput (line 29) | func RunBashOutput(command string, args ...string) (string, error) {
function RunBashOutputSilent (line 41) | func RunBashOutputSilent(command string, args ...string) (string, error) {
function RunBashInput (line 53) | func RunBashInput(input string, command string, args ...string) error {
function RunBashInputOutput (line 67) | func RunBashInputOutput(input []byte, command string, args ...string) ([...
FILE: pkg/bbutil/shred.go
function shredTemp (line 24) | func shredTemp(path, opts string) error {
function ShredInfo (line 59) | func ShredInfo() string {
function shredFile (line 64) | func shredFile(filename string) error {
function ShredFiles (line 88) | func ShredFiles(names []string) error {
FILE: pkg/bbutil/sortedfile_test.go
function TestAddLinesToSortedFile (line 9) | func TestAddLinesToSortedFile(t *testing.T) {
FILE: pkg/bbutil/umask_posix.go
function Umask (line 10) | func Umask(mask int) int {
FILE: pkg/bbutil/umask_windows.go
function Umask (line 8) | func Umask(mask int) int {
FILE: pkg/box/box.go
type Box (line 24) | type Box struct
method getAdmins (line 182) | func (bx *Box) getAdmins() error {
method getFiles (line 206) | func (bx *Box) getFiles() error {
type StatusMode (line 46) | type StatusMode
constant Itemized (line 50) | Itemized StatusMode = iota
constant All (line 52) | All
constant Unchanged (line 54) | Unchanged
constant Changed (line 56) | Changed
function NewFromFlags (line 60) | func NewFromFlags(c *cli.Context) *Box {
function NewUninitialized (line 118) | func NewUninitialized(c *cli.Context) *Box {
function NewForTestingInit (line 156) | func NewForTestingInit(vcsname string) *Box {
FILE: pkg/box/boxutils.go
function FileStatus (line 17) | func FileStatus(name string) (string, error) {
function anyGpg (line 57) | func anyGpg(names []string) error {
function parseGroup (line 117) | func parseGroup(userinput string) (int, error) {
function FindConfigDir (line 141) | func FindConfigDir(reporoot, team string) (string, error) {
function gpgAgentNotice (line 181) | func gpgAgentNotice() {
function shouldWeOverwrite (line 204) | func shouldWeOverwrite() {
function PrettyCommitMessage (line 213) | func PrettyCommitMessage(verb string, files []string) string {
FILE: pkg/box/pretty_test.go
function TestPrettyCommitMessage (line 5) | func TestPrettyCommitMessage(t *testing.T) {
FILE: pkg/box/verbs.go
method AdminAdd (line 22) | func (bx *Box) AdminAdd(nom string, sdir string) error {
method AdminList (line 57) | func (bx *Box) AdminList() error {
method AdminRemove (line 70) | func (bx *Box) AdminRemove([]string) error {
method Cat (line 75) | func (bx *Box) Cat(names []string) error {
method Decrypt (line 103) | func (bx *Box) Decrypt(names []string, overwrite bool, bulkpause bool, s...
function decryptMany (line 136) | func decryptMany(bx *Box, names []string, overwrite bool, groupchange bo...
method Diff (line 180) | func (bx *Box) Diff([]string) error {
method Edit (line 185) | func (bx *Box) Edit(names []string) error {
method Encrypt (line 214) | func (bx *Box) Encrypt(names []string, shred bool) error {
function encryptMany (line 245) | func encryptMany(bx *Box, names []string, shred bool) ([]string, error) {
method FileAdd (line 272) | func (bx *Box) FileAdd(names []string, shred bool) error {
method FileList (line 352) | func (bx *Box) FileList() error {
method FileRemove (line 364) | func (bx *Box) FileRemove(names []string) error {
method Info (line 369) | func (bx *Box) Info() error {
method Init (line 401) | func (bx *Box) Init(yes, vcsname string) error {
method Reencrypt (line 456) | func (bx *Box) Reencrypt(names []string, overwrite bool, bulkpause bool)...
method Shred (line 532) | func (bx *Box) Shred(names []string) error {
method Status (line 552) | func (bx *Box) Status(names []string, nameOnly bool, match string) error {
method TestingInitRepo (line 614) | func (bx *Box) TestingInitRepo() error {
FILE: pkg/commitlater/commitlater.go
type future (line 7) | type future struct
type List (line 15) | type List struct
method Add (line 20) | func (list *List) Add(message string, repobasedir string, files []stri...
method Flush (line 42) | func (list *List) Flush(
function sameDirs (line 29) | func sameDirs(l *List) bool {
FILE: pkg/crypters/crypters.go
type Crypter (line 11) | type Crypter interface
type NewFnSig (line 16) | type NewFnSig
type Item (line 19) | type Item struct
function SearchByName (line 30) | func SearchByName(name string, debug bool) Crypter {
function Register (line 47) | func Register(name string, priority int, newfn NewFnSig) {
FILE: pkg/crypters/gnupg/gnupg.go
function init (line 18) | func init() {
type CrypterHandle (line 23) | type CrypterHandle struct
method Name (line 50) | func (crypt CrypterHandle) Name() string {
method Decrypt (line 55) | func (crypt CrypterHandle) Decrypt(filename string, umask int, overwri...
method Cat (line 75) | func (crypt CrypterHandle) Cat(filename string) ([]byte, error) {
method Encrypt (line 101) | func (crypt CrypterHandle) Encrypt(filename string, umask int, receive...
method AddNewKey (line 129) | func (crypt CrypterHandle) AddNewKey(keyname, repobasedir, sourcedir, ...
function registerNew (line 29) | func registerNew(debug bool) (crypters.Crypter, error) {
FILE: pkg/makesafe/makesafe.go
type protection (line 21) | type protection
constant Unknown (line 25) | Unknown protection = iota
constant None (line 27) | None
constant SingleQuote (line 29) | SingleQuote
constant DoubleQuote (line 31) | DoubleQuote
constant IsAQuote (line 36) | IsAQuote = None
constant IsSpace (line 38) | IsSpace = SingleQuote
constant ShellUnsafe (line 40) | ShellUnsafe = SingleQuote
constant GlobUnsafe (line 42) | GlobUnsafe = SingleQuote
constant InterpolationUnsafe (line 44) | InterpolationUnsafe = SingleQuote
constant HasBackslash (line 46) | HasBackslash = DoubleQuote
function max (line 49) | func max(i, j protection) protection {
type tabEntry (line 57) | type tabEntry struct
function init (line 64) | func init() {
function literal (line 127) | func literal(s string) func(s rune) string {
function same (line 132) | func same() func(r rune) string {
function oct (line 137) | func oct() func(r rune) string {
function Redact (line 155) | func Redact(tainted string) string {
function RedactMany (line 196) | func RedactMany(items []string) []string {
function Shell (line 207) | func Shell(tainted string) string {
function escapeRune (line 243) | func escapeRune(r rune) string {
function ShellMany (line 261) | func ShellMany(items []string) []string {
function FirstFew (line 272) | func FirstFew(sl []string) string {
function FirstFewFlag (line 278) | func FirstFewFlag(sl []string) (string, bool) {
FILE: pkg/makesafe/makesafe_test.go
function TestRedact (line 7) | func TestRedact(t *testing.T) {
function TestRedactMany (line 33) | func TestRedactMany(t *testing.T) {
function TestShell (line 48) | func TestShell(t *testing.T) {
function TestEscapeRune (line 75) | func TestEscapeRune(t *testing.T) {
function TestShellMany (line 96) | func TestShellMany(t *testing.T) {
function TestFirstFewFlag (line 111) | func TestFirstFewFlag(t *testing.T) {
FILE: pkg/vcs/git/git.go
function init (line 16) | func init() {
type VcsHandle (line 21) | type VcsHandle struct
method Name (line 33) | func (v VcsHandle) Name() string {
method Discover (line 40) | func (v VcsHandle) Discover() (bool, string) {
method SetFileTypeUnix (line 56) | func (v VcsHandle) SetFileTypeUnix(repobasedir string, files ...string...
method IgnoreAnywhere (line 89) | func (v VcsHandle) IgnoreAnywhere(repobasedir string, files []string) ...
method IgnoreFiles (line 133) | func (v VcsHandle) IgnoreFiles(repobasedir string, files []string) err...
method Add (line 160) | func (v VcsHandle) Add(repobasedir string, files []string) error {
method CommitTitle (line 177) | func (v *VcsHandle) CommitTitle(title string) {
method NeedsCommit (line 182) | func (v *VcsHandle) NeedsCommit(message string, repobasedir string, na...
method DebugCommits (line 187) | func (v VcsHandle) DebugCommits() commitlater.List {
method FlushCommits (line 192) | func (v VcsHandle) FlushCommits() error {
method suggestCommit (line 207) | func (v *VcsHandle) suggestCommit(messages []string, repobasedir strin...
method TestingInitRepo (line 223) | func (v VcsHandle) TestingInitRepo() error {
function newGit (line 27) | func newGit() (vcs.Vcs, error) {
function ultimate (line 37) | func ultimate(s string) int { return len(s) - 1 }
function gitSafeFilename (line 110) | func gitSafeFilename(name string) string {
FILE: pkg/vcs/none/none.go
function init (line 12) | func init() {
type VcsHandle (line 17) | type VcsHandle struct
method Name (line 26) | func (v VcsHandle) Name() string {
method Discover (line 31) | func (v VcsHandle) Discover() (bool, string) {
method SetFileTypeUnix (line 41) | func (v VcsHandle) SetFileTypeUnix(repobasedir string, files ...string...
method IgnoreAnywhere (line 46) | func (v VcsHandle) IgnoreAnywhere(repobasedir string, files []string) ...
method IgnoreFiles (line 51) | func (v VcsHandle) IgnoreFiles(repobasedir string, files []string) err...
method CommitTitle (line 56) | func (v VcsHandle) CommitTitle(title string) {}
method NeedsCommit (line 59) | func (v VcsHandle) NeedsCommit(message string, repobasedir string, nam...
method DebugCommits (line 64) | func (v VcsHandle) DebugCommits() commitlater.List {
method FlushCommits (line 69) | func (v VcsHandle) FlushCommits() error {
method TestingInitRepo (line 76) | func (v VcsHandle) TestingInitRepo() error {
function newNone (line 21) | func newNone() (vcs.Vcs, error) {
FILE: pkg/vcs/vcs.go
type Vcs (line 14) | type Vcs interface
type NewFnSig (line 19) | type NewFnSig
type Item (line 22) | type Item struct
function Discover (line 34) | func Discover() (Vcs, string) {
function Register (line 71) | func Register(name string, priority int, newfn NewFnSig) {
Condensed preview — 140 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (349K chars).
[
{
"path": ".circleci/config.yml",
"chars": 1301,
"preview": "#version: 2\n#\n#workflows:\n# version: 2\n# build_and_test:\n# jobs:\n# - debian\n# - ubuntu\n#\n#jobs:\n#\n# debi"
},
{
"path": ".gitattributes",
"chars": 114,
"preview": "bin/** text eol=lf\ntools/** text eol=lf\nMakefile text eol=lf\nPortfile text eol=lf\nblackbox.plugin.zsh text eol=lf\n"
},
{
"path": ".github/workflows/build.yml",
"chars": 592,
"preview": "name: build\n\non: \n pull_request:\n branches: [ master ]\n push:\n branches: [ master ]\n\njobs:\n\n build:\n runs-on"
},
{
"path": ".github/workflows/release.yml",
"chars": 1539,
"preview": "on:\n release:\n types: [published]\n\nname: release\njobs:\n release:\n name: release\n runs-on: ubuntu-latest\n s"
},
{
"path": ".gitignore",
"chars": 629,
"preview": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\n"
},
{
"path": "AUTHORS",
"chars": 47,
"preview": "Tom Limoncelli <tlimoncelli@stackexchange.com>\n"
},
{
"path": "CHANGELOG.md",
"chars": 3979,
"preview": "Release v1.20220610\n\nNOTE: I don't have a lot of time to commit to this project. I'd gladly accept help, especially\nwit"
},
{
"path": "DESIGN.md",
"chars": 3058,
"preview": "BlackBox Internals\n==================\n\nThe goal of the Go rewrite is to improve the usability and\nmaintainability of Bla"
},
{
"path": "LICENSE.txt",
"chars": 1092,
"preview": "The MIT License (MIT)\n\nCopyright (c) 2014-2021 Stack Exchange, Inc.\n\nPermission is hereby granted, free of charge, to an"
},
{
"path": "Makefile",
"chars": 5471,
"preview": "SHELL=/bin/sh\nPREFIX?=/usr/local\nPKGNAME=stack_blackbox\nBASEDIR?=$(HOME)\nOUTPUTDIR?=\"$(BASEDIR)/debbuild-${PKGNAME}\"\n\nal"
},
{
"path": "README-v2.md",
"chars": 2069,
"preview": "BlackBox v2\n===========\n\nWARNING: v2 is still experimental. It is in the same git repo as v1\nbecause the filenames do n"
},
{
"path": "README.md",
"chars": 41981,
"preview": "BlackBox [](https://circleci.com/gh/StackExc"
},
{
"path": "RELEASE_ENGINEERING.md",
"chars": 4757,
"preview": "Table of Contents:\n==================\n\n-\t[Branches and Tags:](#branches-and-tags)\n-\t[Testing:](#testing)\n-\t[Build Tasks]"
},
{
"path": "SPECS/empty.spec",
"chars": 0,
"preview": ""
},
{
"path": "Version2-Ideas.md",
"chars": 7262,
"preview": "# Ideas for BlackBox Version 2\n\nI'm writing this to solicit feedback and encourage discussion.\n\nHere are my thoughts on "
},
{
"path": "bin/Makefile",
"chars": 167,
"preview": "all: _stack_lib.sh\n\n# Snatch _stack_lib.sh from another StackExchange project.\n_stack_lib.sh: ../../scripts/misc/_stack_"
},
{
"path": "bin/_blackbox_common.sh",
"chars": 16532,
"preview": "#!/usr/bin/env bash\n\n#\n# Common constants and functions used by the blackbox_* utilities.\n#\n\n# Usage:\n#\n# set -e\n# s"
},
{
"path": "bin/_blackbox_common_test.sh",
"chars": 607,
"preview": "#!/usr/bin/env bash\n\n#\n# _blackbox_common_test.sh -- Unit tests of functions from _blackbox_common.sh\n#\n\nset -e\n. \"${0%/"
},
{
"path": "bin/_stack_lib.sh",
"chars": 6045,
"preview": "# Library functions for bash scripts at Stack Exchange.\n\n# NOTE: This file is open sourced. Do not put Stack-proprietary"
},
{
"path": "bin/blackbox_addadmin",
"chars": 1516,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_addadmin -- Add an admin to the system\n#\n\n# Example:\n# blackbox_addadmin tal@exampl"
},
{
"path": "bin/blackbox_cat",
"chars": 435,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_cat -- Decrypt a file, cat it, shred it\n#\nset -e\nsource \"${0%/*}/_blackbox_common.sh\"\n"
},
{
"path": "bin/blackbox_decrypt_all_files",
"chars": 488,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_decrypt_all_files -- Decrypt all blackbox files (INTERACTIVE).\n#\n\n# Usage:\n# blackbo"
},
{
"path": "bin/blackbox_decrypt_file",
"chars": 270,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_decrypt_file -- Decrypt one or more blackbox files.\n#\n\nset -e\nsource \"${0%/*}/_blackbo"
},
{
"path": "bin/blackbox_deregister_file",
"chars": 1078,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_deregister_file -- Remove a file from the blackbox system.\n#\n# Takes an encrypted file"
},
{
"path": "bin/blackbox_diff",
"chars": 1160,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_diff -- Show all differences.\n#\n\nset -e\nsource \"${0%/*}/_blackbox_common.sh\"\n\ngpg_agen"
},
{
"path": "bin/blackbox_edit",
"chars": 925,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_edit -- Decrypt a file temporarily for edition, then re-encrypts it again\n#\nset -e\nsou"
},
{
"path": "bin/blackbox_edit_end",
"chars": 970,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_edit_end -- Re-encrypt file after edits.\n#\n\nset -e\nsource \"${0%/*}/_blackbox_common.sh"
},
{
"path": "bin/blackbox_edit_start",
"chars": 883,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_edit_start -- Decrypt a file for editing.\n#\n\nset -e\nsource \"${0%/*}/_blackbox_common.s"
},
{
"path": "bin/blackbox_initialize",
"chars": 1589,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_initialize -- Enable blackbox for a GIT or HG repo.\n#\n#\n# Example:\n# blackbox_initi"
},
{
"path": "bin/blackbox_list_admins",
"chars": 135,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_list_admins -- List authorized admins\n#\nset -e\nsource \"${0%/*}/_blackbox_common.sh\"\nca"
},
{
"path": "bin/blackbox_list_files",
"chars": 148,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_list_files -- List files that black box is tracking\n#\nset -e\nsource \"${0%/*}/_blackbox"
},
{
"path": "bin/blackbox_listadmins",
"chars": 254,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_listadmins -- List active admins for keyring\n#\n\n# Example:\n# blackbox_listadmins\n#\n"
},
{
"path": "bin/blackbox_postdeploy",
"chars": 1006,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_postdeploy -- Decrypt all blackbox files.\n#\n\n# Usage:\n# blackbox_postdeploy.sh [GROU"
},
{
"path": "bin/blackbox_recurse",
"chars": 135,
"preview": "#!/usr/bin/env bash\n# proposed space for blackbox recurion...coming soon\nset -e\nsource \"${0%/*}/_blackbox_common.sh\"\n\nec"
},
{
"path": "bin/blackbox_register_new_file",
"chars": 1950,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_register_new_file -- Enroll new file(s) in the blackbox system.\n#\n# Takes previously u"
},
{
"path": "bin/blackbox_removeadmin",
"chars": 933,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_removeadmin -- Remove an admin to the system\n# NOTE: Does not remove admin from the k"
},
{
"path": "bin/blackbox_shred_all_files",
"chars": 1447,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_shred_all_files -- shred all decrypted versions of encrypted files\n#\n# Shred: To secur"
},
{
"path": "bin/blackbox_update_all_files",
"chars": 1678,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_update_all_files -- Decrypt then re-encrypt all files. Useful after keys are changed.\n"
},
{
"path": "bin/blackbox_view",
"chars": 448,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_view -- Decrypt a file, view it, shred it\n#\nset -e\nsource \"${0%/*}/_blackbox_common.sh"
},
{
"path": "bin/blackbox_whatsnew",
"chars": 1176,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_whatsnew - show what has changed in the last commit for a given file\n#\n\nset -e\nsource "
},
{
"path": "binv2/blackbox_addadmin",
"chars": 49,
"preview": "#!/usr/bin/env bash\nexec blackbox admin add \"$@\"\n"
},
{
"path": "binv2/blackbox_cat",
"chars": 43,
"preview": "#!/usr/bin/env bash\nexec blackbox cat \"$@\"\n"
},
{
"path": "binv2/blackbox_decrypt_all_files",
"chars": 82,
"preview": "#!/usr/bin/env bash\nexec blackbox decrypt --all --agentcheck=true --overwrite \"@\"\n"
},
{
"path": "binv2/blackbox_decrypt_file",
"chars": 59,
"preview": "#!/usr/bin/env bash\nexec blackbox decrypt --overwrite \"$@\"\n"
},
{
"path": "binv2/blackbox_deregister_file",
"chars": 58,
"preview": "#!/usr/bin/env bash\nexec blackbox file remove --safe \"$@\"\n"
},
{
"path": "binv2/blackbox_diff",
"chars": 51,
"preview": "#!/usr/bin/env bash\nexec blackbox diff --diff \"$@\"\n"
},
{
"path": "binv2/blackbox_edit",
"chars": 44,
"preview": "#!/usr/bin/env bash\nexec blackbox edit \"$@\"\n"
},
{
"path": "binv2/blackbox_edit_end",
"chars": 55,
"preview": "#!/usr/bin/env bash\nexec blackbox encrypt --shred \"$@\"\n"
},
{
"path": "binv2/blackbox_edit_start",
"chars": 47,
"preview": "#!/usr/bin/env bash\nexec blackbox decrypt \"$@\"\n"
},
{
"path": "binv2/blackbox_initialize",
"chars": 44,
"preview": "#!/usr/bin/env bash\nexec blackbox init \"$@\"\n"
},
{
"path": "binv2/blackbox_list_admins",
"chars": 45,
"preview": "#!/usr/bin/env bash\nexec blackbox admin list\n"
},
{
"path": "binv2/blackbox_list_files",
"chars": 44,
"preview": "#!/usr/bin/env bash\nexec blackbox file list\n"
},
{
"path": "binv2/blackbox_listadmins",
"chars": 45,
"preview": "#!/usr/bin/env bash\nexec blackbox admin list\n"
},
{
"path": "binv2/blackbox_postdeploy",
"chars": 68,
"preview": "#!/usr/bin/env bash\nblackbox decrypt --all --overwrite --group \"$1\"\n"
},
{
"path": "binv2/blackbox_register_new_file",
"chars": 56,
"preview": "#!/usr/bin/env bash\nexec blackbox file add --shred \"$@\"\n"
},
{
"path": "binv2/blackbox_removeadmin",
"chars": 52,
"preview": "#!/usr/bin/env bash\nexec blackbox admin remove \"$@\"\n"
},
{
"path": "binv2/blackbox_shred_all_files",
"chars": 46,
"preview": "#!/usr/bin/env bash\nexec blackbox shred --all\n"
},
{
"path": "binv2/blackbox_update_all_files",
"chars": 63,
"preview": "#!/usr/bin/env bash\nexec blackbox reencrypt --all --agentcheck\n"
},
{
"path": "binv2/blackbox_view",
"chars": 55,
"preview": "#!/usr/bin/env bash\nblackbox cat \"$@\" | ${PAGER:-less}\n"
},
{
"path": "binv2/blackbox_whatsnew",
"chars": 1211,
"preview": "#!/usr/bin/env bash\n\n#\n# blackbox_whatsnew - show what has changed in the last commit for a given file\n#\nexec blackbox w"
},
{
"path": "blackbox.plugin.zsh",
"chars": 1405,
"preview": "#!/usr/bin/env zsh\n# The MIT License (MIT)\n\n# Copyright (c) 2014 Stack Exchange, Inc.\n\n# Permission is hereby granted, f"
},
{
"path": "build/build.go",
"chars": 1582,
"preview": "package main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"os/exec\"\n\t\"strings\"\n\t\"time\"\n)\n\nvar sha = flag.String(\"sha\", \"\", \"SH"
},
{
"path": "cmd/blackbox/blackbox.go",
"chars": 412,
"preview": "package main\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\n\t_ \"github.com/StackExchange/blackbox/v2/pkg/crypters\"\n\t_ \"github.com/StackExchange"
},
{
"path": "cmd/blackbox/cli.go",
"chars": 5955,
"preview": "package main\n\n// cli.go -- Create urfave/cli datastructures and apply them.\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/urfave/cli/v2"
},
{
"path": "cmd/blackbox/drive.go",
"chars": 6646,
"preview": "package main\n\n// Now that cli.go has processed the flags, validate there are no\n// conflicts and drive to the business l"
},
{
"path": "docs/README.md",
"chars": 1872,
"preview": "BlackBox\n========\n\nBlackbox is an open source tool that enables you to safe store sensitive information in\nGit (or other"
},
{
"path": "docs/admin-ops.md",
"chars": 4679,
"preview": "User Management\n===============\n\n\n# Who are the current admins?\n\n```\nblackbox admin list\n```\n\n\n# Add a new user (admin)\n"
},
{
"path": "docs/advanced.md",
"chars": 1438,
"preview": "Advanced Techniques\n===================\n\n\n# Using Blackbox without a repo\n\nIf the files are copied out of a repo they ca"
},
{
"path": "docs/alternatives.md",
"chars": 545,
"preview": "Alternatives\n============\n\nHere are other open source packages that do something similar to\nBlackBox. If you like them b"
},
{
"path": "docs/backwards-compatibility.md",
"chars": 1867,
"preview": "Backwards Compatibility\n=======================\n\n# Where is the configuration stored? .blackbox vs. keyrings/live\n\nBlack"
},
{
"path": "docs/compatibility.md",
"chars": 2360,
"preview": "Compatibility\n=============\n\n# Compatibility with Blackbox v1\n\nThe command names all changed from v1 to v2. The `binv2`"
},
{
"path": "docs/dev-code-overview.md",
"chars": 1406,
"preview": "Code Overview\n=============\n\nHere is how the code is laid out.\n\nTODO(tlim): Add a diagram of the layers\n\n```\ncmd/blackbo"
},
{
"path": "docs/dev.md",
"chars": 1110,
"preview": "Developer Info\n==============\n\nCode submissions are gladly welcomed! The code is fairly easy to read.\n\nGet the code:\n\n``"
},
{
"path": "docs/enable-repo.md",
"chars": 1571,
"preview": "Enabling Blackbox on a Repo\n===========================\n\nOverview:\n1. Run the initialization command\n2. Add at least one"
},
{
"path": "docs/encryption.md",
"chars": 2901,
"preview": "How is the encryption done?\n===========================\n\nGPG has many different ways to encrypt a file. BlackBox uses th"
},
{
"path": "docs/expired-keys.md",
"chars": 2036,
"preview": "Replacing expired keys\n======================\n\nIf someone's key has already expired, blackbox will stop\nencrypting. You"
},
{
"path": "docs/file-ops.md",
"chars": 949,
"preview": "How to add/remove a file into the system?\n=========================================\n\n# Adding files:\n\n- If you need to, "
},
{
"path": "docs/full-command-list.md",
"chars": 656,
"preview": "Blackbox Command List\n=====================\n\n## Global Flags\n### `--vcs`\n### `--crypto`\n### `--config`\n### `--team`\n### "
},
{
"path": "docs/git-tips.md",
"chars": 512,
"preview": "GIT tips\n========\n\n\n# Configure git to show diffs in encrypted files\n\nIt's possible to tell Git to decrypt versions of t"
},
{
"path": "docs/gnupg-tips.md",
"chars": 1233,
"preview": "GnuPG tips\n==========\n\n# Common error messages\n\n* Message: `gpg: filename: skipped: No public key`\n* Solution: Usually t"
},
{
"path": "docs/installation.md",
"chars": 282,
"preview": "Installation Instructions\n=========================\n\nCurrently blackbox v2 is installed by compiling the code and\ncopyin"
},
{
"path": "docs/role-accounts.md",
"chars": 5207,
"preview": "Set up automated users or \"role accounts\"\n=========================================\n\nTODO(tlim): I think this is overly "
},
{
"path": "docs/subversion-tips.md",
"chars": 845,
"preview": "Subversion Tips\n===============\n\nNOTE: This is from v1. Can someone that uses Subversion check\nthis and update it?\n\n\nTh"
},
{
"path": "docs/support.md",
"chars": 492,
"preview": "Support\n=======\n\n# Join our community!\n\nJoin the [blackbox-project mailing list](https://groups.google.com/d/forum/black"
},
{
"path": "docs/user-overview.md",
"chars": 4757,
"preview": "User Guide\n==========\n\n# Overview\n\nSuppose you have a VCS repository (i.e. a Git or Mercurial repo) and\ncertain files co"
},
{
"path": "docs/why-is-this-important.md",
"chars": 784,
"preview": "Why encrypt your secrets?\n=========================\n\nOBVIOUSLY we don't want secret things like SSL private keys and\npas"
},
{
"path": "docs/with-ansible.md",
"chars": 585,
"preview": "How to use the secrets with Ansible?\n===================================\n\nAnsible Vault provides functionality for encry"
},
{
"path": "docs/with-puppet.md",
"chars": 1678,
"preview": "How to use the secrets with Puppet?\n===================================\n\n# Entire files:\n\nEntire files, such as SSL cert"
},
{
"path": "go.mod",
"chars": 298,
"preview": "module github.com/StackExchange/blackbox/v2\n\ngo 1.14\n\nrequire (\n\tgithub.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee"
},
{
"path": "go.sum",
"chars": 3367,
"preview": "github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=\ngithub.com/andreyvit/diff v0.0."
},
{
"path": "integrationTest/NOTES.txt",
"chars": 2738,
"preview": "\nThis should accept VCS-type and --crypto flags.\nThen a shell script should run various combinations of VCS and crypters"
},
{
"path": "integrationTest/README.txt",
"chars": 1873,
"preview": "\nEach test does the following:\n1. Copy the files from testdata/NNNN\n2. Run the command in test_NNNN.sh\n3. \n\n\nTEST ENROLL"
},
{
"path": "integrationTest/asserts.go",
"chars": 1359,
"preview": "package main\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\t\"testing\"\n\n\t\"github.com/andreyvit/diff\"\n)\n\nfunc assertFileMissing(t *testing."
},
{
"path": "integrationTest/integration_test.go",
"chars": 8813,
"preview": "package main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"os\"\n\t\"os/exec\"\n\t\"path/filepath\"\n\t\"testing\"\n\n\t\"github.com/StackExchange/blackbox/"
},
{
"path": "integrationTest/ithelpers.go",
"chars": 16369,
"preview": "package main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"log\"\n\t\"os\"\n\t\"os/exec\"\n\t\"path/filepath\"\n\t\"runtime\"\n\t\"strings\"\n\t\"test"
},
{
"path": "integrationTest/test_data/000-admin-list.txt",
"chars": 36,
"preview": "user1@example.com\nuser2@example.com\n"
},
{
"path": "integrationTest/test_data/000-file-list.txt",
"chars": 16,
"preview": "bar.txt\nfoo.txt\n"
},
{
"path": "integrationTest/test_data/000-status.txt",
"chars": 369,
"preview": "+-------------+------------------------+\n| STATUS | NAME |\n+-------------+-----------------------"
},
{
"path": "integrationTest/test_data/alice-cat-plain.txt",
"chars": 23,
"preview": "I am the foo.txt file!\n"
},
{
"path": "integrationTest/test_data/basic-status.txt",
"chars": 120,
"preview": "+-----------+---------+\n| STATUS | NAME |\n+-----------+---------+\n| ENCRYPTED | foo.txt |\n+-----------+---------+\n"
},
{
"path": "integrationTest/test_data/reencrypt-plain.txt",
"chars": 23,
"preview": "I am the foo.txt file!\n"
},
{
"path": "integrationTest/test_data/status-noreg.txt",
"chars": 222,
"preview": "+-----------+----------------------+\n| STATUS | NAME |\n+-----------+----------------------+\n| ENCRYPT"
},
{
"path": "models/crypters.go",
"chars": 622,
"preview": "package models\n\n// Crypter is gpg binaries, go-opengpg, etc.\ntype Crypter interface {\n\t// Name returns the plug-in's can"
},
{
"path": "models/vcs.go",
"chars": 1234,
"preview": "package models\n\nimport \"github.com/StackExchange/blackbox/v2/pkg/commitlater\"\n\n// Vcs is git/hg/etc.\ntype Vcs interface "
},
{
"path": "pkg/bblog/bblog.go",
"chars": 881,
"preview": "package bblog\n\nimport (\n\t\"io/ioutil\"\n\t\"log\"\n\t\"os\"\n)\n\n/*\n\nTo use this, include the following lines in your .go file.\n\nvar"
},
{
"path": "pkg/bbutil/filestats.go",
"chars": 3306,
"preview": "package bbutil\n\nimport (\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sort\"\n\t\"strings\"\n\t\"time\"\n)\n\n// DirExists returns t"
},
{
"path": "pkg/bbutil/rbio_test.go",
"chars": 312,
"preview": "package bbutil\n\nimport (\n\t\"testing\"\n)\n\nfunc TestRunBashInputOutput(t *testing.T) {\n\n\tin := \"This is a test of the RBIO s"
},
{
"path": "pkg/bbutil/runbash.go",
"chars": 1912,
"preview": "package bbutil\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"os/exec\"\n)\n\n// RunBash runs a Bash command.\nfunc RunBash(command"
},
{
"path": "pkg/bbutil/shred.go",
"chars": 2119,
"preview": "package bbutil\n\n// Pick an appropriate secure erase command for this operating system\n// or just delete the file with os"
},
{
"path": "pkg/bbutil/sortedfile_test.go",
"chars": 1135,
"preview": "package bbutil\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\t\"testing\"\n)\n\nfunc TestAddLinesToSortedFile(t *testing.T) {\n\n\tvar tests = []"
},
{
"path": "pkg/bbutil/umask_posix.go",
"chars": 255,
"preview": "//go:build !windows\n// +build !windows\n\npackage bbutil\n\nimport \"syscall\"\n\n// Umask is a no-op on Windows, and calls sysc"
},
{
"path": "pkg/bbutil/umask_windows.go",
"chars": 221,
"preview": "//go:build windows\n// +build windows\n\npackage bbutil\n\n// Umask is a no-op on Windows, and calls syscall.Umask on all oth"
},
{
"path": "pkg/box/box.go",
"chars": 6831,
"preview": "package box\n\n// box implements the box model.\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sort\"\n\t\"strings\"\n\n\t\"githu"
},
{
"path": "pkg/box/boxutils.go",
"chars": 5797,
"preview": "package box\n\nimport (\n\t\"bufio\"\n\t\"fmt\"\n\t\"os\"\n\t\"os/user\"\n\t\"path/filepath\"\n\t\"runtime\"\n\t\"strconv\"\n\t\"strings\"\n\n\t\"github.com/S"
},
{
"path": "pkg/box/pretty_test.go",
"chars": 1208,
"preview": "package box\n\nimport \"testing\"\n\nfunc TestPrettyCommitMessage(t *testing.T) {\n\tlong := \"aVeryVeryLongLongLongStringStringS"
},
{
"path": "pkg/box/verbs.go",
"chars": 14657,
"preview": "package box\n\n// This file implements the business logic related to a black box.\n// These functions are usually called fr"
},
{
"path": "pkg/commitlater/commitlater.go",
"chars": 1853,
"preview": "package commitlater\n\nimport (\n\t\"fmt\"\n)\n\ntype future struct {\n\tmessage string // Message that describes this transactio"
},
{
"path": "pkg/crypters/_all/all.go",
"chars": 85,
"preview": "package all\n\nimport (\n\t_ \"github.com/StackExchange/blackbox/v2/pkg/crypters/gnupg\"\n)\n"
},
{
"path": "pkg/crypters/crypters.go",
"chars": 1249,
"preview": "package crypters\n\nimport (\n\t\"sort\"\n\t\"strings\"\n\n\t\"github.com/StackExchange/blackbox/v2/models\"\n)\n\n// Crypter is the handl"
},
{
"path": "pkg/crypters/gnupg/gnupg.go",
"chars": 4310,
"preview": "package gnupg\n\nimport (\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"log\"\n\t\"os\"\n\t\"os/exec\"\n\t\"path/filepath\"\n\n\t\"github.com/StackExchange/blackbo"
},
{
"path": "pkg/crypters/gnupg/keychain.go",
"chars": 4388,
"preview": "package gnupg\n\n/*\n\n# How does Blackbox manage key rings?\n\nBlackbox uses the user's .gnupg directory for most actions, su"
},
{
"path": "pkg/makesafe/makesafe.go",
"chars": 8000,
"preview": "package makesafe\n\n// untaint -- A string with a Stringer that is shell safe.\n\n// This goes to great lengths to make sure"
},
{
"path": "pkg/makesafe/makesafe_test.go",
"chars": 3663,
"preview": "package makesafe\n\nimport (\n\t\"testing\"\n)\n\nfunc TestRedact(t *testing.T) {\n\tfor i, test := range []struct{ data, expected "
},
{
"path": "pkg/vcs/_all/all.go",
"chars": 133,
"preview": "package all\n\nimport (\n\t_ \"github.com/StackExchange/blackbox/v2/pkg/vcs/git\"\n\t_ \"github.com/StackExchange/blackbox/v2/pkg"
},
{
"path": "pkg/vcs/git/git.go",
"chars": 5948,
"preview": "package git\n\nimport (\n\t\"fmt\"\n\t\"path/filepath\"\n\t\"strings\"\n\n\t\"github.com/StackExchange/blackbox/v2/pkg/bbutil\"\n\t\"github.co"
},
{
"path": "pkg/vcs/none/none.go",
"chars": 1997,
"preview": "package none\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/StackExchange/blackbox/v2/pkg/commitlater\"\n\t\"github.com/StackExchange/blackb"
},
{
"path": "pkg/vcs/vcs.go",
"chars": 2023,
"preview": "package vcs\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sort\"\n\t\"strings\"\n\n\t\"github.com/StackExchange/blackbox/v2/models\"\n)"
},
{
"path": "tools/Portfile.template",
"chars": 1186,
"preview": "# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4"
},
{
"path": "tools/auto_system_test",
"chars": 864,
"preview": "#!/usr/bin/env expect\n\n# Run the confidence test non-interactively. Since the script\n# asks for passphrases, we use \"ex"
},
{
"path": "tools/confidence_test.sh",
"chars": 14772,
"preview": "#!/usr/bin/env bash\n\nblackbox_home=$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" && pwd )/../bin\nexport PATH=\"${blackbox_home"
},
{
"path": "tools/macports_report_upgrade.sh",
"chars": 2555,
"preview": "#!/usr/bin/env bash\n\n# Turn the Portfile.template into a Portfile.\n# Usage:\n# mk_portfile.sh TEMPLATE OUTPUTFILE VERSI"
},
{
"path": "tools/mk_deb_fpmdir",
"chars": 2553,
"preview": "#! /usr/bin/env bash\n\n# Use fpm to package up files into a DEB .\n\n# Usage:\n# mk_deb_fpmdir PACKAGENAME MANIFEST1 MANIF"
},
{
"path": "tools/mk_deb_fpmdir.stack_blackbox.txt",
"chars": 1538,
"preview": "exec /usr/bin/_blackbox_common.sh ../bin/_blackbox_common.sh\nexec /usr/bin/_stack_lib.sh ../bin/_stack"
},
{
"path": "tools/mk_macports",
"chars": 787,
"preview": "#!/usr/bin/env bash\n\n# Install files into MacPorts DESTDIR\n\n# Usage:\n# mk_macports MANIFEST MANIFEST1 ...\n\n# Where \"ma"
},
{
"path": "tools/mk_macports.vcs_blackbox.txt",
"chars": 1423,
"preview": "exec bin/_blackbox_common.sh ../bin/_blackbox_common.sh\nexec bin/_stack_lib.sh ../bin/_stack_lib.sh\nex"
},
{
"path": "tools/mk_rpm_fpmdir",
"chars": 4362,
"preview": "#! /usr/bin/env bash\n\n# Use fpm to package up files into an RPM.\n\n# Usage:\n# mk_rpm_fpmdir PACKAGENAME MANIFEST1 MANIF"
},
{
"path": "tools/mk_rpm_fpmdir.stack_blackbox.txt",
"chars": 1899,
"preview": "# Update tools/mk_rpm_fpmdir.stack_blackbox.txt. Other files generate from it.\nexec /etc/profile.d/usrblackbox.sh "
},
{
"path": "tools/profile.d-usrblackbox-test.sh",
"chars": 1446,
"preview": "#!/usr/bin/env sh\n\n# Test profile.d-usrblackbox.sh\n\n# Make sure profile.d-usrblackbox.sh works.\n\n# Test variations inclu"
},
{
"path": "tools/profile.d-usrblackbox.sh",
"chars": 70,
"preview": "x=/usr/blackbox/bin\ncase \":$PATH:\" in *:$x:*);;*)PATH=\"$x:$PATH\";;esac"
},
{
"path": "tools/test_functions.sh",
"chars": 3537,
"preview": "#!/usr/bin/env bash\n\n# NB: This is copied from _blackbox_common.sh\nfunction get_pubring_path() {\n : \"${KEYRINGDIR:=keyr"
}
]
About this extraction
This page contains the full source code of the StackExchange/blackbox GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 140 files (313.1 KB), approximately 94.5k tokens, and a symbol index with 205 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.