Showing preview only (1,060K chars total). Download the full file or copy to clipboard to get everything.
Repository: VIDA-NYU/reprozip
Branch: 1.x
Commit: 44687ce73efe
Files: 178
Total size: 1004.6 KB
Directory structure:
gitextract_q6urh6ag/
├── .github/
│ └── workflows/
│ └── test.yml
├── .gitignore
├── .readthedocs.yaml
├── CHANGELOG.md
├── CITATION.cff
├── CITATION.txt
├── CONTRIBUTING.md
├── LICENSE.txt
├── README.md
├── Vagrantfile
├── allsetups.sh
├── docs/
│ ├── Makefile
│ ├── conf.py
│ ├── developerguide.rst
│ ├── faq.rst
│ ├── glossary.rst
│ ├── graph.rst
│ ├── gui.rst
│ ├── index.rst
│ ├── install.rst
│ ├── jupyter.rst
│ ├── make.bat
│ ├── man/
│ │ ├── reprounzip-docker.1
│ │ ├── reprounzip-vagrant.1
│ │ ├── reprounzip.1
│ │ └── reprozip.1
│ ├── packing.rst
│ ├── reprozip.rst
│ ├── traceschema.rst
│ ├── troubleshooting.rst
│ ├── unpacked-format.rst
│ ├── unpacking.rst
│ └── vistrails.rst
├── file-format.md
├── reprounzip/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprounzip/
│ │ ├── __init__.py
│ │ ├── common.py
│ │ ├── main.py
│ │ ├── orderedset.py
│ │ ├── pack_info.py
│ │ ├── parameters.py
│ │ ├── plugins/
│ │ │ └── __init__.py
│ │ ├── signals.py
│ │ ├── unpackers/
│ │ │ ├── __init__.py
│ │ │ ├── common/
│ │ │ │ ├── __init__.py
│ │ │ │ ├── misc.py
│ │ │ │ ├── packages.py
│ │ │ │ └── x11.py
│ │ │ ├── default.py
│ │ │ ├── graph.py
│ │ │ └── provviewer.py
│ │ └── utils.py
│ ├── setup.cfg
│ └── setup.py
├── reprounzip-docker/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprounzip/
│ │ ├── __init__.py
│ │ └── unpackers/
│ │ ├── __init__.py
│ │ └── docker.py
│ ├── setup.cfg
│ └── setup.py
├── reprounzip-qt/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprounzip_qt/
│ │ ├── __init__.py
│ │ ├── gui/
│ │ │ ├── __init__.py
│ │ │ ├── common.py
│ │ │ ├── run.py
│ │ │ └── unpack.py
│ │ ├── main.py
│ │ ├── qt_terminal.py
│ │ ├── reprounzip_interface.py
│ │ └── usage.py
│ ├── setup.cfg
│ └── setup.py
├── reprounzip-vagrant/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprounzip/
│ │ ├── __init__.py
│ │ └── unpackers/
│ │ ├── __init__.py
│ │ └── vagrant/
│ │ ├── __init__.py
│ │ ├── interaction.py
│ │ └── run_command.py
│ ├── setup.cfg
│ └── setup.py
├── reprounzip-vistrails/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprounzip/
│ │ ├── __init__.py
│ │ └── plugins/
│ │ ├── __init__.py
│ │ └── vistrails.py
│ ├── setup.cfg
│ └── setup.py
├── reprozip/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── native/
│ │ ├── config.h
│ │ ├── database.c
│ │ ├── database.h
│ │ ├── log.h
│ │ ├── ptrace_utils.c
│ │ ├── ptrace_utils.h
│ │ ├── pylog.c
│ │ ├── pytracer.c
│ │ ├── syscalls.c
│ │ ├── syscalls.h
│ │ ├── tracer.c
│ │ ├── tracer.h
│ │ ├── utils.c
│ │ └── utils.h
│ ├── reprozip/
│ │ ├── __init__.py
│ │ ├── common.py
│ │ ├── filters.py
│ │ ├── main.py
│ │ ├── pack.py
│ │ ├── tracer/
│ │ │ ├── __init__.py
│ │ │ ├── linux_pkgs.py
│ │ │ └── trace.py
│ │ ├── traceutils.py
│ │ └── utils.py
│ └── setup.py
├── reprozip-jupyter/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprozip_jupyter/
│ │ ├── __init__.py
│ │ ├── main.py
│ │ ├── notebook-extension.js
│ │ ├── run.py
│ │ ├── server_extension.py
│ │ └── trace.py
│ ├── setup.cfg
│ └── setup.py
├── scripts/
│ ├── RELEASE
│ ├── linux-wheels.sh
│ ├── macos/
│ │ ├── README.txt
│ │ ├── ReproUnzip.pkgproj
│ │ ├── app-shim/
│ │ │ ├── ReproUnzip_app/
│ │ │ │ └── Contents/
│ │ │ │ ├── Info.plist
│ │ │ │ └── Resources/
│ │ │ │ └── icon.icns
│ │ │ ├── reprounzip-qt.c
│ │ │ └── reprounzip-qt.debug.c
│ │ ├── instructions.txt
│ │ ├── reprounzip
│ │ ├── reprounzip-qt
│ │ └── reprozip-jupyter
│ ├── register-linux.sh
│ ├── test_bug_13676.py
│ ├── test_bug_23058.py
│ └── windows/
│ ├── input/
│ │ ├── reprounzip-qt.bat
│ │ ├── reprounzip.bat
│ │ └── reprozip-jupyter.bat
│ └── reprounzip.iss
├── syscalls.txt
└── tests/
├── __init__.py
├── __main__.py
├── check_images.py
├── common.py
├── connect.c
├── exec_echo.c
├── functional.py
├── readwrite.c
├── rename.c
├── segv.c
├── simple.c
├── simple_input.txt
├── simple_input2.txt
├── test_graph.py
├── test_parameters.py
├── test_rails_filter.py
├── test_reprounzip.py
├── test_reprozip.py
├── test_unpackers_common.py
├── test_utils.py
├── threads.c
├── threads2.c
└── vfork.c
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/test.yml
================================================
name: Test
on:
- push
- pull_request
jobs:
test:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
mode: [tests]
python:
- "3.11"
include:
- os: ubuntu-latest
mode: coverage
python: "3.8"
- os: ubuntu-latest
mode: checks
python: "3.8"
- os: ubuntu-latest
mode: check-images
python: "3.8"
runs-on: ${{ matrix.os }}
env:
TEST_MODE: ${{ matrix.mode }}
REPROZIP_USAGE_STATS: "off"
REPROZIP_PARAMETERS: https://stats.reprozip.org/parameters/travis/
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 20
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
- name: Install dependencies
run: |
if [ -z "${XDG_CACHE_HOME-}" ]; then
mkdir -p ~/.cache/reprozip
else
mkdir -p "$XDG_CACHE_HOME/reprozip"
fi
case "$TEST_MODE"
in
tests|coverage|check-images)
if [ "$TEST_MODE" = "coverage" ]; then
export CFLAGS="-fprofile-arcs -ftest-coverage"
fi
PKGS="libc6-dev-i386 gcc-multilib libsqlite3-dev"
if [ "$TEST_MODE" = "coverage" ]; then PKGS="$PKGS lcov"; fi
sudo apt-get update -qq
sudo apt-get install -qq $PKGS
if [ $TEST_MODE = "coverage" ]; then
pip install 'coverage<5'
# `--config-settings editable_mode=compat` works around https://github.com/pypa/setuptools/issues/3557
pip install -e ./reprozip -e ./reprounzip -e ./reprounzip-docker -e ./reprounzip-vagrant -e ./reprounzip-vistrails -e ./reprounzip-qt -e ./reprozip-jupyter --config-settings editable_mode=compat
else
pip install ./reprozip ./reprounzip ./reprounzip-docker ./reprounzip-vagrant ./reprounzip-vistrails ./reprounzip-qt ./reprozip-jupyter
fi
;;
checks)
pip install flake8 readme_renderer
;;
*)
exit 1
;;
esac
- name: Test
run: |
export LANG=C
export LC_ALL=C
export REPROZIP_TEST_PYTHON="$(which python) -Wd"
case "$TEST_MODE"
in
coverage)
export PYTHONUNBUFFERED=1
export COVER="coverage run --append --source=$PWD/reprozip/reprozip,$PWD/reprounzip/reprounzip,$PWD/reprounzip-docker/reprounzip,$PWD/reprounzip-vagrant/reprounzip,$PWD/reprounzip-vistrails/reprounzip,$PWD/tests --branch"
python -Wd -m $COVER -m tests --run-docker
;;
tests)
export PYTHONUNBUFFERED=1
python -Wd tests --run-docker
;;
check-images)
python -Wd tests --check-vagrant-images --check-docker-images
;;
checks)
flake8 --ignore=E731,W503,W504
diff -q reprozip/reprozip/common.py reprounzip/reprounzip/common.py
diff -q reprozip/reprozip/utils.py reprounzip/reprounzip/utils.py
find reprozip reprounzip reprozip-* reprounzip-* -name '*.py' -or -name '*.sh' -or -name '*.h' -or -name '*.c' | (set +x; while read i; do
T=$(file -b --mime "$i")
if ! ( echo "$T" | grep -q ascii || echo "$T" | grep -q empty ) ; then
echo "$i is not ASCII"
exit 1
fi
done)
find reprozip reprounzip reprozip-* reprounzip-* -name '*.py' -exec sh -c "grep 'logging\\.\\(debug\\|warning\\|critical\\|error\\|info\\)' \"\$@\" && exit 1; exit 0" {} +
for pkg in reprozip reprounzip reprozip-* reprounzip-*; do
(cd $pkg && python setup.py check -r -s)
done
;;
*)
exit 1
;;
esac
- name: Upload coverage
if: matrix.mode == 'coverage'
run: |
# Python
if [ -f .coverage ]; then mv .coverage .coverage.orig; fi # FIXME: useless?
coverage combine
# C
# Find the coverage file (in distutils's build directory)
OBJDIR=$(dirname "$(find . -name pytracer.gcno | head -n 1)")
(cd reprozip/native && lcov --directory ../../$OBJDIR -c -o reprozip.lcov)
curl -s -o - https://codecov.io/bash | bash -s - -X gcov
test-container:
strategy:
fail-fast: false
matrix:
image:
- python:2.7
install-python:
- false
python-cmd:
- python
include:
- image: ubuntu:20.04
install-python: true
python-cmd: python3
runs-on: ubuntu-latest
container:
image: ${{ matrix.image }}
options: "--privileged"
env:
TEST_MODE: tests
REPROZIP_USAGE_STATS: "off"
REPROZIP_PARAMETERS: https://stats.reprozip.org/parameters/travis/
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 20
- name: Install Docker client
run: |
apt-get update
apt-get install -yy curl
curl -Lo /tmp/docker.tgz https://download.docker.com/linux/static/stable/x86_64/docker-23.0.6.tgz
tar -xf /tmp/docker.tgz -C /usr/local/bin --strip-components=1
rm /tmp/docker.tgz
- name: Install Python
if: matrix.install-python == true
run: |
apt-get install -yy python3 python3-pip
- name: Install dependencies
run: |
if [ -z "${XDG_CACHE_HOME-}" ]; then
mkdir -p ~/.cache/reprozip
else
mkdir -p "$XDG_CACHE_HOME/reprozip"
fi
apt-get update -qq
apt-get install -qq libc6-dev-i386 gcc-multilib libsqlite3-dev
cat > pip.constraints.txt <<'EOF'
pyelftools<0.30
EOF
pip install -c pip.constraints.txt ./reprozip ./reprounzip ./reprounzip-docker ./reprounzip-vagrant ./reprounzip-vistrails ./reprounzip-qt ./reprozip-jupyter
- name: Test
run: |
export LANG=C
export LC_ALL=C
export REPROZIP_TEST_PYTHON="$(which ${{ matrix.python-cmd }}) -Wd"
export PYTHONUNBUFFERED=1
${{ matrix.python-cmd }} -Wd tests --run-docker
================================================
FILE: .gitignore
================================================
*.py[co]
.DS_Store
# Packages
*.egg
*.egg-info
dist
build
eggs
parts
bin
var
sdist
develop-eggs
.installed.cfg
lib
lib64
# Installer logs
pip-log.txt
# Unit test / coverage reports
.coverage
.tox
nosetests.xml
# Object files
*.o
*.ko
*.obj
*.elf
# Libraries
*.lib
*.a
# Shared objects (inc. Windows DLLs)
*.dll
*.so
*.so.*
*.dylib
# Executables
*.exe
*.out
*.i*86
*.x86_64
*.hex
# Eclipse PyDev
.project
.pydevproject
# PyCharm
.idea
# Qt Creator
*.config
*.creator
*.creator.user
*.files
*.includes
# Vagrant
.vagrant
/tests/vagrant
/docs/_build
================================================
FILE: .readthedocs.yaml
================================================
version: 2
build:
os: ubuntu-22.04
tools:
python: "3.11"
sphinx:
configuration: docs/conf.py
================================================
FILE: CHANGELOG.md
================================================
Changelog
=========
1.3.2 (2026-01-18)
------------------
(reprozip and reprounzip only)
Enhancement:
* Don't depend on deprecated Python `pkg_resources` package. Might fix installation errors on some systems, mainly done for downstream packagers
1.3.1 (2025-12-08)
------------------
(reprozip only)
Bugfixes:
* Fix build on GCC 14
* Remove fixed-size buffer for reaading paths from `/proc`
1.3 (2023-12-07)
----------------
(reprozip and reprounzip only)
Enhancements:
* Store used UNIX sockets in the trace
* Display a warning if the process connect to the systemd socket (https://docs.reprozip.org/s/systemd.html)
* Print trace warnings in red (files read then written, systemd socket)
* Improve message on interrupt, making clearer what happens if pressing twice
* Change "(d)elete" option to "(o)verwrite" in prompt when a trace exists
1.2.1 (2023-02-06)
------------------
(reprounzip and reprounzip-qt only)
Bugfixes:
* Fix typo in reprounzip, reprounzip-qt: import from reprounzip, not reprozip
1.2 (2023-02-06)
----------------
Bugfixes:
* Don't mark symlinks as input files
* Fix reprounzip-vagrant not terminating after it says that it can't install packages
* Add defense for CVE-2007-4559
* Fix OrderedSet for Python 3.10+ compatibility
Enhancements:
* Recognize Ruby gems and apps and gather the whole environment
* Don't mark Python .pth files as input files
* Accept ZIP files in addition to TAR for RPZ files (reprozip doesn't currently create ZIP files)
* Handle more Linux system calls: faccessat2, statx, execveat, clone3, openat2, fchownat, fchmodat, accept4, renameat2
1.1 (2021-07-06)
----------------
(reprounzip-vistrails didn't change)
Bugfixes:
* Fix possible crash reading `docker inspect` output
* Fix reprozip-jupyter on more recent Tornado
* Fix failure in reprounzip-docker upload if `/bin/sh` is missing
* Have `reprounzip directory` execute with the correct interpreter (e.g. packed `ld-linux.so`)
* Fix "invalid cross-device link" errors in vagrant download if the destination is not on the same device as the unpacked directory
Enhancements:
* Add PyQt5 compatibility to reprounzip-qt
* reprounzip-docker: Keep ownership of uploaded files
* Add a new certificate to download parameters, valid through 2121 (previous expires 2024)
* Always show full path of executed files in `reprozip testrun`
* Improved merging of pack on top of base image in reprounzip-docker
* Made sure the root is a mountpoint in reprounzip-vagrant, as some applications expect it (e.g. Elasticsearch)
* Disable unpacker options that won't work in reprounzip-qt (e.g. 'directory' and 'chroot' on non-Linux)
1.0.16 (2019-02-06)
-------------------
(reprozip-jupyter and reprounzip-vagrant didn't change)
Bugfixes:
* Fixed input/output file filter on Python 3 (to omit `.so`, `.pyc` etc files)
* Fixed fetching updated parameters on Python 3 (to get the correct Docker and Vagrant base images, a small JSON file is downloaded from reprozip.org)
* Fixed `--port` option of reprounzip-docker
Enhancements:
* Use the [distro](https://distro.readthedocs.io/) module instead of the deprecated `platform.linux_distribution()` function to detect the distribution (the latter will be removed in Python 3.8).
* Use dpkg-query to identify Linux packages instead of reading `dpkg/info/*.list`
1.0.15 (2018-07-31)
-------------------
(reprounzip-qt only)
Bugfixes:
* Fixed running command from reprounzip-qt on Windows
* Fixed using Jupyter from reprounzip-qt
1.0.14 (2018-07-30)
-------------------
(reprozip, reprounzip-qt and reprozip-jupyter only)
Bugfixes:
* Fixed reprounzip-qt refusing to close when an experiment is still unpacked, even after the user provided confirmation
* Fixed reprozip-jupyter on Python 3
* Fixed running gnome-terminal from reprounzip-qt, made it preferred over xterm
* Don't duplicate the latest run in the config file when the trace didn't add a run (for example because the command does not exist)
Enhancements:
* Uniformized logos and icons
* Native terminal opened by reprounzip-qt waits for a key after success before closing
* Officially support reprounzip-qt and reprozip-jupyter on Python 3
1.0.13 (2018-05-15)
-------------------
Bugfixes:
* Fix uninitialized return value making some xxx_at() calls abort the trace
* Fix some other warnings via static analysis
Enhancements:
* Show a warning when executing a file that has the set-uid or set-gid bit set, since Linux will not give it its privileges, making it confusing for users why their run failed
* Make reprounzip-docker run even without a TTY
* Correctly handle experiment returning non-0 in Docker
* The C extension now logs through Python's logging facilities ('reprozip' logger)
* Collect usage information from reprounzip-qt as well
1.0.12 (2018-03-30)
-------------------
(reprozip, reprounzip-qt and reprozip-jupyter only)
Bugfixes:
* Fix some kernel/libc issuing unrecognized `openat()` calls, resulting in files missing in the trace
* Fix `openat()` calls recording read-write as simply write
* Fix double-click on .RPZ file
1.0.11 (2017-11-05)
-------------------
Bugfixes:
* Write timestamp in config with timezone offset
* Always fix up PATH on MacOS (to pick up Docker)
* Don't have reprounzip-qt choke on MacOS's `-psn_x_xxxx` arguments
* Fix "Couldn't find reprounzip command" in the GUI on Windows if installed from the installer and opened a package by double-click
Enhancements:
* Improve DOT graph layout
* Don't silently overwrite output with `reprounzip graph`
* Add `--regex-include` to `reprounzip graph`
* `reprozip trace` now returns 125 for tracing errors (previously 1), and returns the traced program's exit code otherwise (previously always 0)
* Manpages are available
1.0.10 (2017-07-10)
-------------------
Bugfixes:
* Correctly escape shell commands containing backticks
* Overwrite tty prompt works correctly on Python 3
* Fix /proc in vagrant-chroot and chroot having outside mounts
* Fix ANSI escapes showing in Qt terminal dialog
* Fix reprozip combine crash on Python 3.6 (patch from James Clarke)
* Using `graph --packages drop` together with `--json` no longer crashes
Enhancements:
* New reprozip-jupyter tool to trace and reproduce Jupyter notebooks
* reprozip_jupyter can be registered as a Jupyter extension to trace notebooks from the Jupyter web interface
* The Qt GUI knows to run packages with reprozip-jupyter if they are notebook environments (kernels) unpacked with Docker (and reprozip-jupyter is installed)
* Add `--docker-cmd` to reprounzip-docker to select the Docker command (for example `--docker-cmd="sudo docker"`)
* Implement `--expose-port` option for Vagrant and Docker (no need for `--docker-option=-p...`)
* Add docker-machine support to GUI (select which machine to use)
* Better binaries for MacOS
* Automatically register reprounzip-qt to open .RPZ files on Linux
* Register ReproUnzip with Windows from installer
* Add icon from @heng2j
1.0.9 (2017-01-10)
------------------
Bugfixes:
* Fix CentOS Docker image versions
* Remove Fedora Docker images, they don't have tar
* Do include .pyc files in packages, so reproduction take same code path
* Don't use the experiment's resolv.conf file
* Fix handling of files opened in read-write mode
1.0.8 (2016-10-07)
------------------
Behavior changes:
* No longer default to overwriting trace directories. ReproZip will ask what to do or exit with an error if one of --continue/--overwrite is not provided
Bugfixes:
* Fix an issue identifying Debian packages when a file's in two packages
* Fix Python error `Mixing iteration and read methods would lose data`
* Fix reprounzip info showing some numbers as 0 instead of hiding them in non-verbose mode
* Another fix to X server IP determination for Docker
Enhancements:
* New GUI for reprounzip, allowing one to unpack without using the command-line
* Add filters to remove some common files types from packed files (.pyc) or detected input files (.py, .so, ...)
* Add JSON output format to `reprounzip info`
* Allow using the Virtualbox display to reproduce X11-enabled experiments
1.0.7 (2016-08-22)
------------------
Bugfixes:
* Correctly show an error message if ptrace is unavailable
* Make Docker & Vagrant setup much faster
Enhancements:
* Add support for RPM-based distributions, in addition to Debian-based
1.0.6 (2016-06-25)
------------------
(reprounzip-vistrails didn't change)
Bugfixes:
* Fixes error using Docker with `--enable-x11` on Python 3
Enhancements:
* `docker run` gets a `--detach` command, to keep the container running (useful for starting servers on remote machines)
* Restrictions on upload and download commands have been relaxed, in particular it is possible to download input files as well as output files
* Don't compress outer tar (data is still compressed); this should make some operations (like `reprounzip info`) faster
* Add `reprozip combine`, useful to combine multiple traces into one (as different runs). Handy if running distributed experiments on shared filesystem (MPI)
1.0.5 (2016-04-07)
------------------
(reprounzip-vagrant didn't change)
Bugfixes:
* Correctly download parameters from server
* More reliable way of determining X server IP without using /bin/ip
1.0.4 (2016-03-10)
------------------
Behavior changes:
* reprounzip will no longer run on Python 2.6
Bugfixes:
* Fixes file download not using cache if URL is HTTPS
* Fixes unpacking with directory or chroot for some multi-step packages
Features:
* Add `--docker-option` to pass raw options to Docker
* You can use `run` or `run -` to run every run, regardless of their number
* Allow `download <name>`, shortcut for `download <name>:./<name>` (downloads to current directory, keep name)
* Allow `download --all`
* Add `--input` and `--output` to showfiles
* Implement `vagrant suspend` command
* Do file downloads with [requests](http://python-requests.org/)
* Download a parameter file to update URLs and image names without waiting for the next ReproZip version
1.0.3 (2015-12-02)
------------------
Bugfixes:
* You could get a traceback with some unpackers (not Vagrant) on some packages that explicitely pack the / directory
* Some environment variables prevented running, such as bash exported functions.
Features:
* Remove setup directory if setup fails, so you can run setup again without gettin `target directory exists`
* Add `--set-env` and `--pass-env` to run
1.0.2 (2015-10-26)
------------------
Bugfixes:
* You can now use X11 forwarding even with a remote Docker daemon
* reprounzip-vagrant now works in paths containing spaces
1.0.1 (2015-10-12)
------------------
Bugfixes:
* Files opened through a shebang were stored with a wrong process number
* Running with Docker on non-Linux machine didn't work (e.g. docker-machine); now only X11 doesn't work.
* Some fixes to the graph command
Features:
* `--memory` option for `reprounzip vagrant setup`, to set the VM's RAM.
1.0.0 (2015-09-30)
------------------
Behavior change:
* .rpz pack format changed (version 1 -> 2). Pack is now uncompressed, data is in a nested TGZ archive; allows faster retrieval of metadata (config & trace).
* reprozip trace warnings are now info messages; won't show up without -v
Bugfixes:
* After restarting a Vagrant machine, /dev and /proc wouldn't be mounted anymore
* Files or links referenced in a shebang could be missed by the tracer
Features:
* Runs in the configuration file now have an 'id' field, that will be shown by 'reprounzip info' and can be selected when running
* Reworked `reprounzip graph`: level of details, regex filters & replace, JSON output
* Added *run* argument to `reprounzip showfiles`, to show inputs & outputs of a single run
0.7.2 (2015-08-24)
------------------
Behavior change:
* reprounzip-docker will now re-use the resulting image from the previous run when running again, instead of starting from scratch; a 'reset' command has been added to undo runs and uploads.
Bugfixes:
* Couldn't reset an input file to the original (packed) file on Python 3
* Don't show a warning about network connections when they didn't succeed
* Hide traceback when failing because Vagrant is not installed
* Fix input/output file detection assigning files to the same run
* Fix selecting multiple runs in 'docker run'
Features:
* Display the relative portion of the path when unhandled xxx_at() syscalls are used, to give an idea of what's been missed
* Add --dont-find-inputs-outputs to reprozip trace and reset, so you can clear that out if too many files would be selected (or if you don't use the feature)
* Rewrote reprounzip-vistrails plugin; uses a proper VisTrails package that now lives in the VisTrails distribution.
* Check pack format in unpackers; won't try to unpack version 2
* It is now possible to select multiple runs with `unpackername run 1-4`
0.7.1 (2015-07-14)
------------------
(reprozip only)
Bugfixes:
* Files (or links) created with rename, link or symlink then read will no longer be packed.
* A buffer overflow could happen in the log module, for instance when the experiment passes a very long argument to execve (over 4kB in a single argument) and running in debug mode (-v -v)
0.7 (2015-07-07)
----------------
Behavior change:
* No longer accept passing `-v` after the subcommand; use `reprozip -v testrun ...`, not `reprozip testrun -v`.
* Rely on `PTHREAD_EVENT_EXEC` to handle `execve()`. Makes tracing more reliable, and enable it to behave correctly on weird kernels (like UML).
* Rely on `PTRACE_EVENT_FORK` to handle `fork`/`vfork`/`clone`. Fixes vfork() deadlocking under trace.
* Completely changed the structure of input and output files (old packs will still be loaded, but new packs are not retro-compatible).
* Using one of the `run` commands without specifying a number will no longer default to running all of them; it will error out if there are multiple runs.
Bugfixes:
* Fix insertion speed in SQLite3 database
Features:
* Makes VMs (Vagrant or Docker) more resilient to massive breakage of system libraries (obliterating /lib or /usr, when using very different operating systems) by putting busybox in / and using [rpzsudo](https://github.com/remram44/static-sudo).
* No longer use `dpkg -S` to identify packages, do a single pass over internal dpkg database (this is considerably faster).
0.6.4 (2015-06-07)
------------------
(reprounzip-vistrails didn't change)
Bugfixes:
* Tracer: correctly handle `chdir()` in multi-threaded processes
* Fix leaked file descriptors, eventually making SQLite3 fail
* No longer exceed cmdline length in Dockerfile in big .RPZ pack
* Fixes `check_output` call when running Docker (not available in Python 2.6)
* Fixes installation of `sudo` failing if original machine wasn't Debian
* Don't make TAR error status fatal in Dockerfile (might not run; this is needed because Docker mount some files in the container that can't be overwritten)
0.6.3 (2015-05-06)
------------------
(reprounzip and plugins only)
Bugfixes:
* Fixes reprounzip-vistrails failing because of reporting
* Fixes reprounzip-vistrails not escaping correctly in XML in some conditions
* Fixes docker run failing to read Docker's JSON output on Python 3
* Fixes reprounzip chroot mounting too many filesystems
* Fixes typo stopping reprounzip from running on unsupported distribs
Features:
* Adds Debian 8 'Jessie' to Vagrant boxes & Docker images
* Adds Ubuntu 15.04 'Vivid' to Vagrant boxes & Docker images
0.6.2 (2015-03-16)
------------------
(reprozip only)
Bugfixes:
* Fixes installation on Python 3 with 7-bit locale
* Fixes reprozip not showing traceback on exception
* Fixes bug with multiple runs (`trace --continue`)
0.6.1 (2015-02-17)
------------------
(reprozip only)
Bugfixes:
* Fixes an overflow in _pytracer for some syscalls.
0.6 (2015-02-11)
----------------
(reprounzip-vistrails didn't change)
Bugfixes:
* Fixes `debug` log messages not being printed
* Pressing Ctrl+C wouldn't stop package identification with KeyboardInterrupt
* Fixes an error message while destroying a docker experiment
* Fixes docker not installing packages if they were packed
* Fixes docker always reporting exit status 0
Features:
* Adds `--install-pkgs` options to `docker setup`, to prefer installing
from package manager over unpacking the packed files
* With vagrant or docker, original machine hostname is restored
* X11 support for chroot, vagrant and docker unpackers
0.5.1 (2014-12-18)
------------------
(reprounzip-vistrails didn't change)
Bugfixes:
* Goes back to pack format 0.4: generates `.rpz` files readable by older reprounzip versions
* Fixes experiment not having a PTY in some conditions
* Rewrite absolute paths on command-line for directory unpacker
Features:
* Python 2.6 support for reprounzip (except 'graph')
* Makes error messages more readable
* Default trace directory renamed from `.reprozip` to `.reprozip-trace`
* Adds a log file under $HOME/.reprozip/log
* reprounzip-vagrant will use 'ssh' executable if it's available; should be more reliable, especially on Windows
* Automatically collects usage information. Nothing will be sent before you opt-in, and we made sure to only collect general details
0.5 (2014-11-24)
----------------
Features:
* All reprounzip plugins can be installed with `pip install reprounzip[all]`
* Various improvements to interactive vagrant console
* Adds support for generic plugins (alongside unpackers)
* Adds reprounzip-vistrails plugin
* Pressing Ctrl+C while tracing won't abort anymore; press it twice for SIGKILL
0.4.1 (2014-10-06)
------------------
Bugfixes:
* reprounzip showed duplicated logging messages
* Makes 'run' commands not fail if the command returns an error code
* Unicode issues with Vagrant on Windows Python 3
* Warning for files read then written didn't show the filenames
* Fixes resetted input files breaking 'showfiles'
Features:
* 'docker upload' command
* Adds signals (currently unused, needed for future plugins)
0.4 (2014-09-15)
----------------
Bugfixes:
* Copying files from host to chroot when some packages where not packed
* Don't use the full command path in directory's script
* Fixes socketcall() handling
Features:
* Displays a warning for READ_THEN_WRITTEN files
* chroot restores files' owner/group
* Adds 'reprounzip info' command
* Better error messages when trying to unpack on incompatible system
* Identifies input files, which can be replaced ('upload' operation)
* Identifies output files, which can be retrieved ('download' operation)
* New command-line interface for unpackers, with setup/run/destroy; you can now do everything through reprounzip
* Vagrant now defaults to --use-chroot`, added --no-use-chroot
* Adds --summary and --missing to installpkgs
* Adds Docker unpacker (no uploading support yet)
0.3.2 (2014-08-28)
------------------
(reprounzip only)
Bugfixes:
* Once busybox was in the local cache, setting it up could crash
* 'script.sh' files were not set as executable
0.3.1 (2014-08-26)
------------------
(reprozip only)
Bugfixes:
* Don't crash when packing an experiment that wrote in temporary directories
0.3 (2014-07-28)
----------------
Bugfixes:
* Handles Linux changing thread id to thread leader's on `execve()`
* Correctly handles processes dying from signals (e.g. SEGV)
* Fixes case of rt_sigreturn
Features:
* Database stores `is_directory` field
* Handles `mkdir()`, `symlink()`
* Forces pack to have a `.rpz` extension
* Displays a warning when the process uses `connect()` or `accept()`
* Improved logging
* Handles i386 compatibility mode on x86_64
* Handles *at() variants of system calls with AT_FDCWD
0.2.1 (2014-07-11)
------------------
Bugfixes:
* 'pack' no longer stop if a file is missing
* Do not attempt to pack files from /proc or /dev
* Stops vagrant without --use-chroot from overwriting files
* Downloads busybox instead of using the host's /bin/sh
* Correctly packs the dynamic linkers from the original machine
* The tracer no longer considers `execve()` to always happen before other accesses
* Fixes pytracer forking the Python process if the executable cannot be found
* Improves signal handling (but bugs might still exist with `SIGSTOP`)
* Fixes a bug if a process resumed before its creator (race condition)
Features:
* -v flag also controls C tracer's verbosity
* Detects (but doesn't handle yet) i386 compatibility and x32 mode on x86_64
* Stores working directories and exit codes of all processes in database
* Added `reprozip reset [-d dir]` to reset the configuration file from the database
0.2 (2014-06-18)
----------------
First version of the rewritten ReproZip, that uses ptrace. Basic functionality.
0.1 (2013-06-25)
----------------
SystemTap-based version of ReproZip.
================================================
FILE: CITATION.cff
================================================
cff-version: 1.2.0
message: If you use this software, please cite it as below.
authors:
- family-names: Rampin
given-names: Remi
affiliation: New York University
orcid: https://orcid.org/0000-0002-0524-2282
website: https://remi.rampin.org/
- family-names: Freire
given-names: Juliana
affiliation: New York University
orcid: https://orcid.org/0000-0003-3915-7075
website: https://vgc.poly.edu/~juliana/
- family-names: Chirigati
given-names: Fernando
affiliation: New York University
orcid: https://orcid.org/0000-0002-9566-5835
website: http://fchirigati.com/
- family-names: Shasha
given-names: Dennis
affiliation: New York University
orcid: https://orcid.org/0000-0002-7036-3312
website: http://cs.nyu.edu/shasha/
- family-names: Rampin
given-names: Vicky
affiliation: New York University
orcid: https://orcid.org/0000-0003-4298-168X
website: https://vicky.rampin.org/
license: BSD-3-Clause
url: https://www.reprozip.org/
repository-code: https://github.com/VIDA-NYU/reprozip
title: ReproZip
abstract: |
ReproZip is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science. It tracks operating system calls and creates a package that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step).
keywords: [python, linux, docker, reproducibility, provenance, reproducible-research, reproducible-science]
references:
- type: proceedings
doi: 10.1145/2882903.2899401
conference:
name: "SIGMOD '16"
website: https://www.sigmod2016.org/
city: San Francisco
country: US
authors:
- family-names: Rampin
given-names: Remi
affiliation: New York University
orcid: https://orcid.org/0000-0002-0524-2282
website: https://remi.rampin.org/
- family-names: Freire
given-names: Juliana
affiliation: New York University
orcid: https://orcid.org/0000-0003-3915-7075
website: https://vgc.poly.edu/~juliana/
- family-names: Chirigati
given-names: Fernando
affiliation: New York University
orcid: https://orcid.org/0000-0002-9566-5835
website: http://fchirigati.com/
- family-names: Shasha
given-names: Dennis
affiliation: New York University
orcid: https://orcid.org/0000-0002-7036-3312
website: http://cs.nyu.edu/shasha/
date-published: 2016-06-26
year: 2016
month: 6
title: "ReproZip: Computational Reproducibility With Ease"
abstract: |
We present ReproZip, the recommended packaging tool for the SIGMOD Reproducibility Review. ReproZip was designed to simplify the process of making an existing computational experiment reproducible across platforms, even when the experiment was put together without reproducibility in mind. The tool creates a self-contained package for an experiment by automatically tracking and identifying all its required dependencies. The researcher can share the package with others, who can then use ReproZip to unpack the experiment, reproduce the findings on their favorite operating system, as well as modify the original experiment for reuse in new research, all with little effort. The demo will consist of examples of non-trivial experiments, showing how these can be packed in a Linux machine and reproduced on different machines and operating systems. Demo visitors will also be able to pack and reproduce their own experiments.
- type: proceedings
doi: 10.21105/joss.00107
journal: Journal of Open Source Software
authors:
- family-names: Rampin
given-names: Remi
affiliation: New York University
orcid: https://orcid.org/0000-0002-0524-2282
website: https://remi.rampin.org/
- family-names: Freire
given-names: Juliana
affiliation: New York University
orcid: https://orcid.org/0000-0003-3915-7075
website: https://vgc.poly.edu/~juliana/
- family-names: Chirigati
given-names: Fernando
affiliation: New York University
orcid: https://orcid.org/0000-0002-9566-5835
website: http://fchirigati.com/
- family-names: Shasha
given-names: Dennis
affiliation: New York University
orcid: https://orcid.org/0000-0002-7036-3312
website: http://cs.nyu.edu/shasha/
- family-names: Rampin
given-names: Vicky
affiliation: New York University
orcid: https://orcid.org/0000-0003-4298-168X
website: https://vicky.rampin.org/
date-published: 2016-12-01
year: 2016
month: 12
title: "ReproZip: The Reproducibility Packer"
preferred-citation:
type: proceedings
doi: 10.1145/2882903.2899401
conference:
name: "SIGMOD '16"
website: https://www.sigmod2016.org/
city: San Francisco
country: US
authors:
- family-names: Rampin
given-names: Remi
affiliation: New York University
orcid: https://orcid.org/0000-0002-0524-2282
website: https://remi.rampin.org/
- family-names: Freire
given-names: Juliana
affiliation: New York University
orcid: https://orcid.org/0000-0003-3915-7075
website: https://vgc.poly.edu/~juliana/
- family-names: Chirigati
given-names: Fernando
affiliation: New York University
orcid: https://orcid.org/0000-0002-9566-5835
website: http://fchirigati.com/
- family-names: Shasha
given-names: Dennis
affiliation: New York University
orcid: https://orcid.org/0000-0002-7036-3312
website: http://cs.nyu.edu/shasha/
date-published: 2016-06-26
year: 2016
month: 6
title: "ReproZip: Computational Reproducibility With Ease"
abstract: |
We present ReproZip, the recommended packaging tool for the SIGMOD Reproducibility Review. ReproZip was designed to simplify the process of making an existing computational experiment reproducible across platforms, even when the experiment was put together without reproducibility in mind. The tool creates a self-contained package for an experiment by automatically tracking and identifying all its required dependencies. The researcher can share the package with others, who can then use ReproZip to unpack the experiment, reproduce the findings on their favorite operating system, as well as modify the original experiment for reuse in new research, all with little effort. The demo will consist of examples of non-trivial experiments, showing how these can be packed in a Linux machine and reproduced on different machines and operating systems. Demo visitors will also be able to pack and reproduce their own experiments.
version: "1.1"
date-released: 2021-07-06
doi: 10.5281/zenodo.5081097
================================================
FILE: CITATION.txt
================================================
To reference ReproZip in publications, please cite the following:
ReproZip: Computational Reproducibility With Ease, F. Chirigati, R. Rampin, D. Shasha, and J. Freire. In Proceedings of the 2016 ACM SIGMOD International Conference on Management of Data (SIGMOD), pp. 2085-2088, 2016
BibTeX:
@inproceedings{ChirigatiRSF16,
title = {ReproZip: Computational Reproducibility With Ease},
author = {Chirigati, Fernando and Rampin, R{\'e}mi and Shasha, Dennis and Freire, Juliana},
year = {2016},
isbn = {978-1-4503-3531-7},
location = {San Francisco, California, USA},
doi = {10.1145/2882903.2899401},
booktitle = {Proceedings of the 2016 International Conference on Management of Data},
series = {SIGMOD '16},
pages = {2085--2088},
numpages = {4},
publisher = {ACM},
keywords = {computational reproducibility, provenance, reprozip},
}
================================================
FILE: CONTRIBUTING.md
================================================
# Contents
* [Notes](#notes)
* [Contributing](#contributing)
* [Resolving Merge Conflicts](#resolving-merge-conflicts)
* [Best Practices for Contributing](#best-practices-for-contributing)
* [Attribution](#attribution)
# Notes
Any contributions received are assumed to be covered by the BSD 3-Clause license. We might ask you to sign a Contributor License Agreement before accepting a larger contribution. To learn more about ReproZip, see:
* [ReproZip Examples](https://examples.reprozip.org/)
* [ReproZip Documentation](https://docs.reprozip.org/)
* [ReproZip Demo Video](https://www.youtube.com/watch?v=-zLPuwCHXo0)
# Contributing
Please follow the [GitHub Community Guidelines](https://docs.github.com/en/github/site-policy/github-community-guidelines) in all your interactions with the project. If you would like to contribute to this project by modifying/adding to the code, please read the [Best Practices for Contributing](#best-practices-for-contributing) below and feel free to follow the standard GitHub workflow:
1. Fork the project.
2. Clone your fork to your computer.
* From the command line: `git clone https://github.com/<USERNAME>/reprozip.git`
3. Change into your new project folder.
* From the command line: `cd reprozip`
4. [optional] Add the upstream repository to your list of remotes.
* From the command line: `git remote add upstream https://github.com/ViDA-NYU/reprozip.git`
5. Create a branch for your new feature.
* From the command line: `git checkout -b my-feature-branch-name`
6. Make your changes.
* Avoid making changes to more files than necessary for your feature (i.e. refrain from combining your "real" pull request with incidental bug fixes). This will simplify the merging process and make your changes clearer.
7. Commit your changes. From the command line:
* `git add <FILE-NAMES>`
* `git commit -m "A descriptive commit message"`
8. While you were working some other changes might have gone in and break your stuff or vice versa. This can be a *merge conflict* but also conflicting behavior or code. Before you test, merge with upstream.
* `git fetch upstream`
* `git merge upstream/1.x`
9. Test. Run the program and do something related to your feature/fix.
10. Push the branch, uploading it to GitHub.
* `git push origin my-feature-branch-name`
11. Make a "Pull Request" from your branch here on GitHub.
# Resolving Merge Conflicts
Depending on the order that Pull Requests get processed, your PR may result in a conflict and become un-mergable. To correct this, do the following from the command line:
Switch to your branch: `git checkout my-feature-branch-name`
Pull in the latest upstream changes: `git pull upstream 1.x`
Find out what files have a conflict: `git status`
Edit the conflicting file(s) and look for a block that looks like this:
```
<<<<<<< HEAD
my awesome change
=======
some other person's less awesome change
>>>>>>> some-branch
```
Replace all five (or more) lines with the correct version (yours, theirs, or
a combination of the two). ONLY the correct content should remain (none of
that `<<<<< HEAD` stuff.)
Then re-commit and re-push the file.
```
git add the-changed-file.cs
git commit -m "Resolved conflict between this and PR #123"
git push origin my-feature-branch-name
```
The pull request should automatically update to reflect your changes.
## Best Practices for Contributing
* Before you start coding, open an issue so that the community can discuss your change to ensure it is in line with the goals of the project and not being worked on by someone else. This allows for discussion and fine tuning of your feature and results in a more succent and focused additions.
* If you are fixing a small glitch or bug, you may make a PR without opening an issue.
* If you are adding a large feature, create an issue so that we may give you feedback and agree on what makes the most sense for the project before making your change and submitting a PR (this will make sure you don't have to do major changes down the line).
* Pull Requests are eventually merged into the codebase. Please ensure they are:
* Well tested by the author. It is the author's job to ensure their code works as expected.
* Free of unnecessary log calls. Logging important for debugging, but when a PR is made, log calls should only be present when there is an actual error or to record some important piece of information or progress.
* If your code is untested, log heavy, or incomplete, prefix your PR with "[WIP]", so others know it is still being tested and shouldn't be considered for merging yet. This way we can still give you feedback or help finalize the feature even if it's not ready for prime time.
That's it! Following these guidelines will ensure that your additions are approved quickly and integrated into the project. Thanks for your contribution!
# Attribution
This CONTRIBUTING.md was adapted from [ProjectPorcupine's](https://github.com/TeamPorcupine/ProjectPorcupine)'s [CONTRIBUTING.md](https://github.com/TeamPorcupine/ProjectPorcupine/blob/master/CONTRIBUTING.md)
# Contact info
You are welcome to [subscribe to](https://groups.google.com/a/nyu.edu/g/reprozip) or contact our user mailing list [reprozip@nyu.edu](mailto:reprozip@nyu.edu) for questions, suggestions and discussions about using ReproZip.
================================================
FILE: LICENSE.txt
================================================
Copyright (C) 2014, New York University
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
================================================
FILE: README.md
================================================
[](https://github.com/VIDA-NYU/reprozip/actions)
[](https://codecov.io/github/VIDA-NYU/reprozip?branch=1.x)
[](https://docs.reprozip.org/en/1.x/)
[](https://riot.im/app/#/room/#reprozip:matrix.org)
[](http://joss.theoj.org/papers/b578b171263c73f64dfb9d040ca80fe0)
[](https://doi.org/10.5281/zenodo.10291844)
ReproZip
========
[ReproZip][web] is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science.
It tracks operating system calls and creates a package that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step). A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).
Quickstart
----------
We have an [example repository](https://github.com/VIDA-NYU/reprozip-examples) with a variety of different software. Don't hesitate to check it out, and contribute your own example if use ReproZip for something new!
### Packing
Packing experiments is only available for Linux distributions. In the environment where the experiment is originally executed, first install reprozip:
$ pip install reprozip
Then, run your experiment with reprozip. Suppose you execute your experiment by originally running the following command:
$ ./myexperiment -my --options inputs/somefile.csv other_file_here.bin
To run it with reprozip, you just need to use the prefix *reprozip trace*:
$ reprozip trace ./myexperiment -my --options inputs/somefile.csv other_file_here.bin
This command creates a *.reprozip-trace* directory, in which you'll find the configuration file, named *config.yml*. You can edit the command line and environment variables, and choose which files to pack.
If you are using Debian or Ubuntu, most of these files (library dependencies) are organized by package. You can add or remove files, or choose not to include a package by changing option *packfiles* from true to false. In this way, smaller packs can be created with reprozip (if space is an issue), and reprounzip can download these files from the package manager; however, note this is only available for Debian and Ubuntu for now, and also be aware that package versions might differ. Choosing which files to pack is also important to remove sensitive information and third-party software that is not open source and should not be distributed.
Once done editing the configuration file (or even if you did not change anything), run the following command to create a ReproZip package named *my_experiment*:
$ reprozip pack my_experiment.rpz
Voilà! Now your experiment has been packed, and you can send it to your collaborators, reviewers, and researchers around the world!
Note that you can open the help message for any reprozip command by using the flag *-h*.
### Unpacking
Do you need to unpack an experiment in a Linux machine? Easy! First, install reprounzip:
$ pip install reprounzip
Then, if you want to unpack everything in a single directory named *mydirectory* and execute the experiment from there, use the prefix *reprounzip directory*:
$ reprounzip directory setup my_experiment.rpz mydirectory
$ reprounzip directory run mydirectory
In case you prefer to build a chroot environment under *mychroot*, use the prefix *reprounzip chroot*:
$ reprounzip chroot setup my_experiment.rpz mychroot
$ reprounzip chroot run mychroot
Note that the previous options do not interfere with the original configuration of the environment, so don't worry! If you are using Debian or Ubuntu, reprounzip also has an option to install all the library dependencies directly on the machine using package managers (rather than just copying the files from the .rpz package). Be aware that this will interfere in your environment and it may update your library packages, so use it at your own risk! For this option, just use the prefix *reprounzip installpkgs*:
$ reprounzip installpkgs my_experiment.rpz
What if you want to reproduce the experiment in Windows or Mac OS X? You can build a virtual machine with the experiment! Easy as well! First, install the plugin reprounzip-vagrant:
$ pip install reprounzip-vagrant
Note that (i) you must install reprounzip first, and (ii) the plugin requires having [Vagrant][vagrant] installed. Then, use the prefix *reprounzip vagrant* to create and start a virtual machine under directory *mytemplate*:
$ reprounzip vagrant setup my_experiment.rpz mytemplate
To execute the experiment, simply run:
$ reprounzip vagrant run mytemplate
Alternatively, you may use [Docker][docker] containers to reproduce the experiment, which also works under Linux, Mac OS X, and Windows! First, install the plugin reprounzip-docker:
$ pip install reprounzip-docker
Then, assuming that you want to create the container under directory *mytemplate*, simply use the prefix *reprounzip docker*:
$ reprounzip docker setup my_experiment.rpz mytemplate
$ reprounzip docker run mytemplate
Remember that you can open the help message and learn more about other available flags and options by using the flag *-h* for any reprounzip command.
Citing ReproZip
---------------
Please use the following when citing ReproZip ([BibTeX](CITATION.txt)):
ReproZip: Computational Reproducibility With Ease
F. Chirigati, R. Rampin, D. Shasha, and J. Freire.
In Proceedings of the 2016 ACM SIGMOD International Conference on Management of Data (SIGMOD), pp. 2085-2088, 2016
Contribute
----------
Please subscribe to and contact the [reprozip@nyu.edu](https://groups.google.com/a/nyu.edu/g/reprozip) mailing list for questions, suggestions and discussions about using reprozip.
Bugs and feature plannings are tracked in the [GitHub issues](https://github.com/VIDA-NYU/reprozip/issues). Feel free to add an issue!
To suggest changes to this source code, feel free to raise a [GitHub pull request](https://github.com/VIDA-NYU/reprozip/pulls). Any contributions received are assumed to be covered by the [BSD 3-Clause license](LICENSE.txt). We might ask you to sign a _Contributor License Agreement_ before accepting a larger contribution.
License
-------
* Copyright (C) 2014, New York University
Licensed under a **BSD 3-Clause license**. See the file [LICENSE.txt](LICENSE.txt) for details.
Links and References
--------------------
For more detailed information, please refer to our [website][web], as well as to our [documentation][docs].
ReproZip is currently being developed at [NYU][nyu]. The team includes:
* [Fernando Chirigati][fc]
* [Juliana Freire][jf]
* [Rémi Rampin][rr]
* [Dennis Shasha][ds]
* [Vicky Rampin][vr]
[vagrant]: https://www.vagrantup.com/
[docker]: https://www.docker.com/
[docs]: https://docs.reprozip.org/
[web]: https://www.reprozip.org/
[pz]: https://pypi.python.org/pypi/reprozip
[puz]: https://pypi.python.org/pypi/reprounzip
[puzd]: https://pypi.python.org/pypi/reprounzip-docker
[puzv]: https://pypi.python.org/pypi/reprounzip-vagrant
[fc]: http://fchirigati.com/
[jf]: https://vgc.poly.edu/~juliana/
[rr]: https://remi.rampin.org/
[ds]: http://cs.nyu.edu/shasha/
[vr]: https://vicky.rampin.org/
[nyu]: http://engineering.nyu.edu/
================================================
FILE: Vagrantfile
================================================
# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.provision "shell",
inline: <<SCRIPT
aptitude update -y
aptitude install -y curl make gcc sqlite3 libsqlite3-dev python2.7-dev python-virtualenv libc6-dev
aptitude install -y xserver-xorg xserver-xorg-video-vesa xfwm4 x11-apps
SCRIPT
config.vm.define "x86", autostart: false do |m|
m.vm.box = "ubuntu/trusty32"
end
config.vm.define "x86_64" do |m|
m.vm.box = "remram/debian-8-amd64"
m.vm.provision "shell",
inline: <<SCRIPT
aptitude install -y libc6-dev-i386 gcc-multilib docker.io
adduser vagrant docker
SCRIPT
end
config.vm.define "travis", autostart: false do |m|
m.vm.box = "hashicorp/precise64"
end
config.vm.provider "virtualbox" do |v|
v.memory = 1024
end
end
================================================
FILE: allsetups.sh
================================================
#!/bin/sh
set -e
set -u
cd "$(dirname $0)"
PROGRAMS="reprounzip reprounzip-docker reprounzip-vagrant reprounzip-vistrails reprounzip-qt reprozip-jupyter"
if [ "$(uname -s)" = Linux ]; then
PROGRAMS="reprozip $PROGRAMS"
fi
arg="${1:-}"
if [ "$arg" = install ]; then
CMD=""
for prog in $PROGRAMS; do
CMD="$CMD ./$prog"
done
pip install -U $CMD
elif [ "$arg" = develop ]; then
# -e doesn't work with local paths before 6.0
pip install -U setuptools pip
CMD=""
for prog in $PROGRAMS; do
CMD="$CMD -e ./$prog"
done
pip install -U $CMD
elif [ "$arg" = uninstall ]; then
for prog in $PROGRAMS; do
pip uninstall -y $prog || true
done
else
echo "Usage: $(basename "$0") <install|develop|uninstall>" >&2
exit 1
fi
================================================
FILE: docs/Makefile
================================================
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = _build
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " dirhtml to make HTML files named index.html in directories"
@echo " singlehtml to make a single large HTML file"
@echo " pickle to make pickle files"
@echo " json to make JSON files"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " qthelp to make HTML files and a qthelp project"
@echo " applehelp to make an Apple Help Book"
@echo " devhelp to make HTML files and a Devhelp project"
@echo " epub to make an epub"
@echo " epub3 to make an epub3"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " texinfo to make Texinfo files"
@echo " info to make Texinfo files and run them through makeinfo"
@echo " gettext to make PO message catalogs"
@echo " changes to make an overview of all changed/added/deprecated items"
@echo " xml to make Docutils-native XML files"
@echo " pseudoxml to make pseudoxml-XML files for display purposes"
@echo " linkcheck to check all external links for integrity"
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
@echo " coverage to run coverage check of the documentation (if enabled)"
@echo " dummy to check syntax errors of document sources"
.PHONY: clean
clean:
rm -rf $(BUILDDIR)/*
.PHONY: html
html:
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
.PHONY: dirhtml
dirhtml:
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
.PHONY: singlehtml
singlehtml:
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
@echo
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
.PHONY: pickle
pickle:
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
@echo
@echo "Build finished; now you can process the pickle files."
.PHONY: json
json:
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
@echo
@echo "Build finished; now you can process the JSON files."
.PHONY: htmlhelp
htmlhelp:
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in $(BUILDDIR)/htmlhelp."
.PHONY: qthelp
qthelp:
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/ReproZip.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/ReproZip.qhc"
.PHONY: applehelp
applehelp:
$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
@echo
@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
@echo "N.B. You won't be able to view it unless you put it in" \
"~/Library/Documentation/Help or install it in your application" \
"bundle."
.PHONY: devhelp
devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/ReproZip"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/ReproZip"
@echo "# devhelp"
.PHONY: epub
epub:
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
@echo
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
.PHONY: epub3
epub3:
$(SPHINXBUILD) -b epub3 $(ALLSPHINXOPTS) $(BUILDDIR)/epub3
@echo
@echo "Build finished. The epub3 file is in $(BUILDDIR)/epub3."
.PHONY: latex
latex:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
@echo "Run \`make' in that directory to run these through (pdf)latex" \
"(use \`make latexpdf' here to do that automatically)."
.PHONY: latexpdf
latexpdf:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
.PHONY: latexpdfja
latexpdfja:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through platex and dvipdfmx..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
.PHONY: text
text:
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
@echo
@echo "Build finished. The text files are in $(BUILDDIR)/text."
.PHONY: man
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
@echo
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
.PHONY: texinfo
texinfo:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
@echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
.PHONY: info
info:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo "Running Texinfo files through makeinfo..."
make -C $(BUILDDIR)/texinfo info
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
.PHONY: gettext
gettext:
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
@echo
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
.PHONY: changes
changes:
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
@echo
@echo "The overview file is in $(BUILDDIR)/changes."
.PHONY: linkcheck
linkcheck:
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in $(BUILDDIR)/linkcheck/output.txt."
.PHONY: doctest
doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
.PHONY: coverage
coverage:
$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
@echo "Testing of coverage in the sources finished, look at the " \
"results in $(BUILDDIR)/coverage/python.txt."
.PHONY: xml
xml:
$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
@echo
@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
.PHONY: pseudoxml
pseudoxml:
$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
@echo
@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
.PHONY: dummy
dummy:
$(SPHINXBUILD) -b dummy $(ALLSPHINXOPTS) $(BUILDDIR)/dummy
@echo
@echo "Build finished. Dummy builder generates no files."
================================================
FILE: docs/conf.py
================================================
# -*- coding: utf-8 -*-
#
# ReproZip documentation build configuration file, created by
# sphinx-quickstart on Tue May 31 15:43:00 2016.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.intersphinx',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The encoding of source files.
#
# source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'ReproZip'
copyright = u'2014, New York University'
authors = [u'Fernando Chirigati', u'Remi Rampin',
u'Juliana Freire', u'Dennis Shasha']
if len(authors) <= 1:
authors_str = u', '.join(authors)
else:
authors_str = u', '.join(authors[:-1]) + u', and ' + authors[-1]
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = u'1.3'
# The full version, including alpha/beta/rc tags.
release = u'1.3'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = 'en'
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#
# today = ''
#
# Else, today_fmt is used as the format for a strftime call.
#
# today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# The reST default role (used for this markup: `text`) to use for all
# documents.
#
# default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#
# add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#
# add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#
# show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
# modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
# keep_warnings = False
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'default'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = []
# The name for this set of Sphinx documents.
# "<project> v<release> documentation" by default.
#
# html_title = u'ReproZip'
# A shorter title for the navigation bar. Default is the same as html_title.
#
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#
# html_logo = None
# The name of an image file (relative to this directory) to use as a favicon of
# the docs. This file should be a Windows icon file (.ico) being 16x16 or
# 32x32 pixels large.
#
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#
# html_extra_path = []
# If not None, a 'Last updated on:' timestamp is inserted at every page
# bottom, using the given strftime format.
# The empty string is equivalent to '%b %d, %Y'.
#
# html_last_updated_fmt = None
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#
# html_additional_pages = {}
# If false, no module index is generated.
#
# html_domain_indices = True
# If false, no index is generated.
#
# html_use_index = True
# If true, the index is split into individual pages for each letter.
#
# html_split_index = False
# If true, links to the reST sources are added to the pages.
#
# html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#
# html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#
# html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#
# html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
# html_file_suffix = None
# Language to be used for generating the HTML full-text search index.
# Sphinx supports the following languages:
# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr', 'zh'
#
# html_search_language = 'en'
# A dictionary with options for the search language support, empty by default.
# 'ja' uses this config value.
# 'zh' user can custom change `jieba` dictionary path.
#
# html_search_options = {'type': 'default'}
# The name of a javascript file (relative to the configuration directory) that
# implements a search results scorer. If empty, the default will be used.
#
# html_search_scorer = 'scorer.js'
# Output file base name for HTML help builder.
htmlhelp_basename = 'ReproZip'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'ReproZip.tex', u'ReproZip Documentation',
authors_str, 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#
# latex_use_parts = False
# If true, show page references after internal links.
#
# latex_show_pagerefs = False
# If true, show URL addresses after external links.
#
# latex_show_urls = False
# Documents to append as an appendix to all manuals.
#
# latex_appendices = []
# If false, no module index is generated.
#
# latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = []
# If true, show URL addresses after external links.
#
# man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'ReproZip', u'ReproZip Documentation',
authors_str, 'ReproZip',
u"Allows the reproducibility of command-line experiments in a different "
"environment",
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#
# texinfo_appendices = []
# If false, no module index is generated.
#
# texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#
# texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
#
# texinfo_no_detailmenu = False
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {
'python': ('https://docs.python.org/3/', None),
'rpaths': ('http://rpaths.remram.fr/en/latest/', None),
}
================================================
FILE: docs/developerguide.rst
================================================
.. _develop-plugins:
Developer's Guide
*****************
General Development Information
-------------------------------
Development happens on `GitHub <https://github.com/VIDA-NYU/reprozip>`__; bug reports and feature requests are welcome. If you are interested in giving us a hand, please do not hesitate to submit a pull request there.
Continuous testing is provided by `GitHub Actions <https://github.com/VIDA-NYU/reprozip/actions>`__. Note that ReproZip still tries to support Python 2 as well as Python 3. Test coverage is not very high because there are a lot of operations that are difficult to cover on CI (for instance, Vagrant VMs cannot be used over there).
If you have any questions or need help with the development of an unpacker or plugin, please use our development mailing-list at `reprozip@nyu.edu <https://groups.google.com/a/nyu.edu/g/reprozip>`__.
Introduction to ReproZip
------------------------
ReproZip works in two steps: tracing and packing. Under the hood, tracing is two separate steps, leading to the following workflow:
* Running the experiment under trace. During this part, the experiment is running, and the ``_pytracer`` C extension watches it through the `ptrace` mechanism, recording information in the trace SQLite3 database (``.reprozip-trace/trace.sqlite3``). This database contains raw information as it is recorded and does little else, leaving that to the next step. This part is referred to as the "C tracer".
* After the experiment is done, some additional information is computed by the Python code to generate the configuration file, by looking at the trace database and the filesystem. For example, all accesses to a file are aggregated to decide if it is read or written by the overall experiment, if it is an input or output file, resolve symlinks, etc. Additional information is written such as OS information and which distribution package each file comes from.
* Packing reads the configuration file to create the ``.rpz`` bundle, which includes a configuration file (re-written into a "canonical" version), the trace database (though it is not read at this step), and the files listed in the configuration which was possibly altered by the user.
Therefore it is important to note that the configuration file and the trace database contain distinct information, and although the configuration is inferred from the database, it contains some additional details that was obtained from the original machine afterwards.
Only the configuration file should be necessary to run unpackers. The trace database is included for information, and to support additional commands like ``reprounzip graph`` (:ref:`graph`).
Writing Unpackers
-----------------
ReproZip is divided into two steps. The first is packing, which gives a generic package containing the trace SQLite database, the YAML configuration file (which lists the paths, packages, and metadata such as command line, environment variables, and input/output files), and actual files. In the second step, a package can be run using *reprounzip*. This decoupling allows the reproducer to select the unpacker of his/her desire, and also means that when a new unpacker is released, users will be able to use it on their old packages.
Currently, different unpackers are maintained: the defaults ones (``directory`` and ``chroot``), ``vagrant`` (distributed as `reprounzip-vagrant <https://pypi.org/project/reprounzip-vagrant/>`__) and ``docker`` (distributed as `reprounzip-docker <https://pypi.org/project/reprounzip-docker/>`__). However, the interface is such that new unpackers can be easily added. While taking a look at the "official" unpackers' source is probably a good idea, this page gives some useful information about how they work.
ReproZip Bundle Format (``.rpz``)
'''''''''''''''''''''''''''''''''
An ``.rpz`` file is a ``tar.gz`` archive that contains a directory ``METADATA``, which contains meta-information from *reprozip*, and an archive ``DATA.tar.gz``, which contains the actual files that were packed and that will be unpacked to the target directory for reproducing the experiment.
The ``METADATA/version`` file marks the file as a ReproZip bundle. It always contains the string ``REPROZIP VERSION 2``. It previously contained ``REPROZIP VERSION 1`` before version 0.8 (2015), where ``DATA`` was a directory instead of being a tar.gz file.
The ``METADATA/config.yml`` file is in the same format as the configuration file generated by *reprozip*, but without the ``additional_patterns`` section (at this point, it has already been expanded to the actual list of files while packing).
The ``METADATA/trace.sqlite3`` file is the original trace generated by the C tracer and maintained in a SQLite database; it contains all the information about the experiment, in case the configuration file is insufficient in some aspect. This file is used, for instance, by the *graph* unpacker, so that it can recover the exact hierarchy of processes, together with the executable images they execute and the files they access (with the time and mode of these accesses).
.. seealso:: :ref:`Trace Database Schema <trace-schema>`
Structure
'''''''''
An unpacker is a Python module. It can be distributed separately or be a part of a bigger distribution, given that it is declared in that distribution's ``setup.py`` as an `entry_point` to be registered with `importlib` (see `setuptools' advertising behavior section <https://setuptools.pypa.io/en/latest/userguide/entry_point.html#advertising-behavior>`__). You should declare a function as `entry_point` ``reprounzip.unpackers``. The name of the entry_point (before ``=``) will be the *reprounzip* subcommand, and the value is a callable that will get called with the :class:`argparse.ArgumentParser` object for that subcommand.
The package :mod:`reprounzip.unpackers` is a namespace package, so you should be able to add your own unpackers there if you want to. Please remember to put the correct code in the ``__init__.py`` file (which you can copy from `here <https://github.com/VIDA-NYU/reprozip/blob/master/reprounzip/reprounzip/unpackers/__init__.py>`__) so that namespace packages work correctly.
The modules :mod:`reprounzip.common`, :mod:`reprounzip.utils`, and :mod:`reprounzip.unpackers.common` contain utilities that you might want to use (make sure to list *reprounzip* as a requirement in your ``setup.py``).
Example of ``setup.py``::
setup(name='reprounzip-vagrant',
namespace_packages=['reprounzip', 'reprounzip.unpackers'],
install_requires=['reprounzip>=0.4'],
entry_points={
'reprounzip.unpackers': [
'vagrant = reprounzip.unpackers.vagrant:setup'
# The setup() function sets up the parser for reprounzip vagrant
]
}
# ...
)
Usual Commands
''''''''''''''
If possible, you should try to follow the same command names that the official unpackers use, which are:
* ``setup``: to create the experiment directory and set everything for execution;
* ``run``: to reproduce the experiment;
* ``destroy``: to bring down all that setup and to prepare and delete the experiment directory safely;
* ``upload`` and ``download``: to replace input files in the experiment, and to get the output files for further examination, respectively.
If these commands can be broken down into different steps that you want to expose to the user, or if you provide completely different actions from these defaults, you can add them to the parser as well. For instance, the *vagrant* unpacker exposes ``setup/start``, which starts or resumes the virtual machine, and ``destroy/vm``, which stops and deallocates the virtual machine but leaves the template for possible reuse.
A Note on File Paths
''''''''''''''''''''
ReproZip supports Python 2 and 3, is portable to different operating systems, and is meant to accept a wide variety of configurations so that it is compatible with most experiments out there. Even trickier, `reprounzip-vagrant` needs to manipulate POSIX filenames on Windows, e.g.: in the unpacker.
Therefore, the `rpaths <https://github.com/remram44/rpaths>`__ library is used everywhere internally. You should make sure to use the correct type of path (either :class:`~rpaths.PosixPath` or :class:`~rpaths.Path`) and to cast these to the type that Python functions expect, keeping in mind 2/3 differences (most certainly either ``filename.path`` or ``str(filename)``).
Experiment Directory Format
'''''''''''''''''''''''''''
Unpackers usually create a directory with everything necessary to later run the experiment. This directory is created by the ``setup`` operation, cleaned up by ``destroy``, and is the argument to every command. For example, with `reprounzip-vagrant`::
$ reprounzip vagrant setup someexperiment.rpz mydirectory
$ reprounzip vagrant upload mydirectory /tmp/replace.txt:input_text
Unpackers unpack the config.yml file to the root of that directory, and keep status information in a ``.reprounzip`` file, which is a dict in :mod:`pickle` format. Following the same structure will allow the ``showfiles`` command, as well as :class:`~reprounzip.unpackers.common.FileUploader` and :class:`~reprounzip.unpackers.common.FileDownloader` classes, to work correctly. Please try to follow this structure.
Signals
'''''''
Since version 0.4.1, `reprounzip` has signals that can be used to hook in plugins, although no such plugin has been released at this time. To ensure that these work correctly when using your unpacker, you should emit them when appropriate. The complete list of signals is available in `signal.py <https://github.com/VIDA-NYU/reprozip/blob/master/reprounzip/reprounzip/signals.py>`__.
Final Observations
------------------
After reading this page, reading the source code of one of the "official" unpackers is probably the best way of understanding how to write your own. They should be short enough to be easy to grasp. Should you have additional questions, do not hesitate to use our mailing-list: `reprozip@nyu.edu`.
================================================
FILE: docs/faq.rst
================================================
.. _faq:
Frequently Asked Questions
**************************
Can ReproZip pack a client-server scenario?
===========================================
Yes! However, note that only tracing the client will not capture the full story: reproducibility is better achieved (and guaranteed) if the server is traced as well.
Having said that, currently, ReproZip can only trace local servers: if the server is remote (i.e., running in another machine), ReproZip cannot capture it. In this case, you can trace the client, and the experiment can only be reproduced if the remote server is still running at the moment of the reproduction.
The easiest way to pack a local client-server experiment is to write a script that starts the server, runs your client(s), and then shuts down the server; ReproZip can then trace this script. See :ref:`Capturing Connections to Servers <packing-clientserv>` for more information.
Can ReproZip pack a database?
=============================
ReproZip can trace a database server; however, because of the format it uses to store data (and also because different databases work differently), it might be hard for you to control exactly what data will be packed. You probably want to pack all the data from the databases/tables that your experiment uses and not just the pages that were touched while tracing the execution. This can be done by inspecting the configuration file and adding the relevant patterns that cover the data, e.g.: for MySQL::
additional_patterns:
- /var/lib/mysql/**
See :ref:`Capturing Connections to Servers <packing-clientserv>` for an example using a local database and additional information.
Note that ReproZip does not currently save the state of the files. Therefore, if your experiment modifies a database, ReproZip will pack the already modified data (not the one before tracing the experiment execution).
Can ReproZip pack interactive tools?
====================================
Yes! The `reprounzip` component should have no problems with experiments that interact with the user through the terminal. If your experiment runs until it receives a Ctrl+C signal, that is fine as well: ReproZip will not interfere unless you press Ctrl+C twice, stopping the experiment.
GUI tools are also supported; see :ref:`gui-tools` for more information.
.. _gui-tools:
Can ReproZip pack graphical tools?
==================================
Yes!
On Linux, graphical display is handled by the X server. Applications can connect to it as clients to display their windows and components, and to get user input.
Most unpackers now support forwarding the X connection from the experiment to the X server running on the unpacking machine. You will need a running X server to make this work, such as `Xming <https://sourceforge.net/projects/xming/>`__ for Windows or `XQuartz <https://www.xquartz.org/>`__ for Mac OS X. If you are running Linux, chances are that an X server is already configured and running.
X support is **not** enabled by default; to enable it, use the flag ``--enable-x11`` in the ``run`` command of your preferred unpacker.
.. warning::
While displaying a UI through the X protocol works fine, applications using direct rendering (DRI) to access dedicated graphic hardware might not be reproducible: the libGL library packed with the experiment is often specific to the driver of the original machine, and therefore the machine where the experiment is being reproduced would need to use the exact same hardware and driver.
Please refrain from requiring direct rendering in applications that you intend to pack with ReproZip.
If using Vagrant, you can also use the virtual machine's native display directly, by supplying the ``--use-gui`` option to ``reprounzip vagrant setup``.
How can I access the generated system or virtual machine directly?
==================================================================
If you are running ``reprounzip vagrant``, you can connect to the Vagrant virtual machine by running ``vagrant ssh`` or connecting via SSH to the destination listed by ``vagrant ssh-config`` (often ``localhost:2222``). These commands should be run from inside the directory created by the unpacker.
If you are running ``reprounzip docker``, you can inspect the Docker container directly by using ``docker``, or start a new one based on the image created by `reprounzip`; all images generated by ReproZip are named with the ``reprounzip_`` prefix. For more information on how to inspect and create Docker containers, please refer to the `Docker documentation <https://docs.docker.com/>`__.
For ``reprounzip chroot`` and ``reprounzip directory``, the filesystem is in the ``root`` subdirectory under the experiment path.
See :ref:`Structure of Unpacked Experiments <unpacked-format>` for more details.
.. warning::
Note that, in the generated system, only the files needed for running the unpacked experiment are guaranteed to work correctly. This means that you may have only parts of a software distribution (required to run the experiment), but not the software in its entirety (unless the complete software was included in the configuration file while packing). For example, you may only have a few Python files that the experiment needs, but not the ones required to run Python interactively or install new libraries. Therefore, do not expect that all the software components will run smoothly when acessing the system.
The utilities from the base system might also not work correctly (if they are not part of the experiment) because `reprounzip` overwrites the libraries with the ones from the original environment. In the worst-case scenario, the dynamic linker or the shell may not be usable. Note that some unpackers install ``/bin/busybox``, which you may find helpful.
What if my experiment runs on a distributed environment?
========================================================
ReproZip cannot trace across multiple machines. You could trace each component separately, but ReproZip cannot connect these multiple ``.rpz`` files to setup the multiple machines the right way. In particular, you will probably need to set up the same network for the components to talk to each other.
================================================
FILE: docs/glossary.rst
================================================
.. _glossary:
Glossary
********
.. glossary::
configuration (file)
A YAML file generated by ``reprozip trace`` and read by ``reprozip pack``. It can be edited before creating the package to control which files are to be included. It also contain other metadata used during reproduction. See :ref:`packing-config`.
distribution package
A software component installed by the Linux distribution's package manager. ReproZip tries to identify from which distribution package each file comes; this allows the reproducer to install the software from his distribution's package manager instead of extracting the files from the ``.rpz`` file.
bundle (or pack)
A ``.rpz`` file generated by ``reprozip pack``, containing all the files and metadata required to reproduce the experiment on another machine. See :ref:`packing`.
run
A single command line traced by ``reprozip trace [--continue]``. Multiple commands can be traced successively before creating the bundle; the reproducer will be able to run them separately using ``reprounzip <unpacker> run <directory> <run-id>``.
software package
The same as a distribution package.
unpacker
A plugin for the `reprounzip` component that reproduces an experiment from a ``.rpz`` bundle. The unpackers `chroot`, `directory`, and `installpkgs` are distributed with `reprounzip`; others come in separate packages (`reprounzip-docker` and `reprounzip-vagrant`). See :ref:`unpack-unpackers`.
================================================
FILE: docs/graph.rst
================================================
.. _graph:
Visualizing the Provenance Graph
********************************
.. note:: If you are using a Python version older than 2.7.3, this feature will not be available due to `Python bug 57885 <https://github.com/python/cpython/issues/57885>`__ related to sqlite3.
To generate a *provenance graph* related to the experiment execution, the ``reprounzip graph`` command should be used::
$ reprounzip graph graphfile.dot mypackfile.rpz
where `graphfile.dot` corresponds to the graph, and `mypackfile.rpz` corresponds to the experiment bundle.
Alternatively, you can generate the graph after running ``reprozip trace`` without creating a ``.rpz`` bundle::
$ reprounzip graph [-d tracedirectory] graphfile.dot
The graph is outputted in the `DOT <https://en.wikipedia.org/wiki/DOT_(graph_description_language)>`__ language. You can use `Graphviz <https://www.graphviz.org/>`__ to load and visualize the graph::
$ dot -Tpng graphfile.dot -o graph.png
It is also possible to output a JSON file with the flag ``--json``.
Command-Line Options
====================
Since an experiment may involve a significantly large number of file dependencies, ``reprounzip graph`` offers several command-line options to control what will be shown in the provenance graph, as described below. By default it includes all information available, which is often unreadable (see :numref:`fig-toobig`).
Filtering Out Files
+++++++++++++++++++
Files can be filtered out using a regular expression [#re]_ with the flag ``--regex-filter``. For example:
* ``--regex-filter /~[^/]*$``` will filter out files whose name begins with a tilde
* ``--regex-filter ^/usr/share`` will filter out ``/usr/share`` recursively
* ``--regex-filter \.bin$`` will filter out files with a ``.bin`` extension
These flags can be passed multiple times.
Replacing Filenames
+++++++++++++++++++
Users can remap filenames using regular expressions [#re]_ with the flag ``--regex-replace``. This can be used to:
* simplify the graph by making filenames shorter,
* aggregate multiple files to a single node by mapping them to the same name, or
* fix programs that are using some type of cache or for which the wrong access was logged, such as Python's ``.pyc`` files.
Example:
* ``--regex-replace .pyc$ \.py`` will replace accesses to bytecode cache files (.pyc) to the original source (.py)
* ``--regex-replace ^/dev(/.*)?$ /dev`` will aggregate all device files as a single path `/dev`
* ``--regex-replace ^/home/vagrant/experiment/data/(.*)\.bin data:\1`` will simplify the paths to some data files
The flag ``--aggregate`` is a shortcut allowing users to aggregate all files beginning with a given prefix. For instance, ``--aggregate /usr/somepath`` will collapse all files under ``/usr/somepath`` (this is equivalent to ``--regex-replace '^/usr/somepath' '/usr/somepath'``).
Both flags can be passed multiple times.
Controlling Levels of Detail
++++++++++++++++++++++++++++
Users can control the levels of detail for each category of items in the provenance graph.
Software Packages
.................
* ``--packages file`` will show all the files belonging to a package grouped under that package's name
* ``--packages package`` will show the package as a single item, not detailing the individual files that it contains
* ``--packages drop`` will entirely hide the packages, removing all their files from the graph
* ``--packages ignore`` will ignore the package identification, handling their files as if they had not been detected as being part of a package
Note that regex filters and replacements are applied beforehand, so files that are remapped to a package will be shown under that package name.
Processes
.........
* ``--processes thread`` will show every process and thread
* ``--processes process`` will show every process and hide threads
* ``--processes run`` will show only one node for an experiment run, even if the run is composed by multiple processes and threads
Other Files
...........
For files that are not part of a software package, or if ``--packages ignore`` is being used:
* ``--otherfiles all`` will show every file (unless filtered by ``--regex-filter``)
* ``--otherfiles io`` will show only the input and output files, as identified in the configuration file
* ``--otherfiles no`` will ignore all the files
.. [#re] Anchoring regular expressions with ``^`` and ``$`` and escaping dots (``\.``) is recommended. For more information about regular expressions, please see `here <https://en.wikipedia.org/wiki/Regular_expression>`__.
Common Recipes
==============
* Full provenance graph (likely to be unreadable for most experiments, due to the large amount of information to be presented)::
$ reprounzip graph graph.dot myexperiment.rpz
.. _fig-toobig:
.. figure:: figures/toobig.png
:width: 10in
:align: center
Provenance graph showing all the information available (full graph). This represents the default configuration.
* Mapping Python bytecode cache files to their corresponding source file (this may help attribute file accesses to software packages)::
$ reprounzip graph --regex-replace '\.pyc$' '\.py' graph.dot myexperiment.rpz
* Dataflow of the experiment, showing the runs and their corresponding input and output files::
$ reprounzip graph --packages drop --otherfiles io --processes run graph.dot myexperiment.rpz
.. _fig-digits-io:
.. figure:: figures/digits-io.png
:width: 10in
:align: center
Provenance graph showing input and output files for an experiment with 4 runs.
* Provenance graph showing only processes and threads (no file accesses)::
$ reprounzip graph --packages drop --otherfiles drop --processes thread graph.dot myexperiment.rpz
.. _fig-processes:
.. figure:: figures/ache-processes.png
:width: 10in
:align: center
Provenance graph showing only processes and threads.
================================================
FILE: docs/gui.rst
================================================
.. _unpacking-gui:
ReproUnzip GUI
**************
**reprounzip-qt** is a graphical interface (GUI) for reprounzip, allowing you to unpack and reproduce experiments from ``.rpz`` files without having to use the command-line. You can also set it as the default handler for the ``.rpz`` file extension so you can open them via double-click.
Installation
============
*reprounzip-qt* comes with the installer on `Windows <http://reprozip-files.s3-website-us-east-1.amazonaws.com/windows-installer>`_ and `Mac <http://reprozip-files.s3-website-us-east-1.amazonaws.com/mac-installer>`_. If you used one of these, you will be able to double click on any ``.rpz`` file to boot up the GUI.
If you are using Anaconda, you can install *reprounzip-qt* from anaconda.org::
$ conda install --channel conda-forge reprounzip-qt
Otherwise, you will need to `install PyQt5 <https://www.riverbankcomputing.com/software/pyqt/download>`__ (or PyQt4 ) before you can install *reprounzip-qt* from pip (on Debian or Ubuntu, you can use ``apt-get install pyqt5-dev``).
On Linux, you will need to run the application one time so that it registers
itself as the handler for ``.rpz`` files.
.. image:: figures/reprounzip-qt-linux-register.png
Usage
=====
The first tab in the window that appears is for you to set up the experiment. This will allow you to choose which `unpacker <unpacking.html#unpackers>`_ you'd like to use to reproduce the experiment, and in which directory you'd like to unpack it.
.. image:: figures/reprounzip-qt.png
After successfully unpacking, you'll be prompted to run the experiment in the second tab. You can choose which run you want to execute, though the default is to have all runs selected. ReproUnzip will detect the order of the runs and reproduce the experiment accordingly.
.. image:: figures/reprounzip-qt-1.png
Clicking "Manage Files" will bring up options to download the input and output data of the original experiment, and upload your own data to use it in the same experiment. You'll notice input files are marked ``[I]`` and output files are marked ``[O]``. ``[IO]`` are both input and output files.
.. image:: figures/reprounzip-qt-manageFiles.png
================================================
FILE: docs/index.rst
================================================
ReproZip's Documentation
************************
Welcome to ReproZip's documentation!
`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process of creating reproducible experiments from *command-line executions*. It tracks operating system calls and creates a bundle that contains all the binaries, files, and dependencies required to run a given command on the author's computational environment. A reviewer can then extract the experiment in his own environment to reproduce the results, even if the environment has a different operating system from the original one.
Currently, ReproZip can only pack experiments that originally run on Linux.
Concretely, ReproZip has two main steps:
- The :ref:`packing step <packing>` happens in the original environment, and generates a compendium of the experiment so as to make it reproducible. ReproZip tracks operating system calls while executing the experiment, and creates a ``.rpz`` file, which contains all the necessary information and components for the experiment.
- The :ref:`unpacking step <unpacking>` reproduces the experiment from the ``.rpz`` file. ReproZip offers different unpacking methods, from simply decompressing the files in a directory to starting a full virtual machine, and they can be used interchangeably from the same packed experiment. It is also possible to automatically replace input files and command-line arguments. Note that this step is also available on Windows and Mac OS X, since ReproZip can unpack the experiment in a virtual machine for further reproduction.
Contents
--------
.. toctree::
:maxdepth: 2
reprozip
install
packing
unpacking
graph
jupyter
gui
vistrails
faq
troubleshooting
unpacked-format
developerguide
glossary
Links
-----
* `Project website <https://www.reprozip.org/>`__
* `GitHub repository <https://github.com/VIDA-NYU/reprozip>`__
* Mailing list: `reprozip@nyu.edu <https://groups.google.com/a/nyu.edu/g/reprozip>`__
================================================
FILE: docs/install.rst
================================================
.. _install:
Installation
************
ReproZip is available as open source, released under the Revised BSD License. The tool is comprised of two components: **reprozip** (for the packing step) and **reprounzip** (for the unpack step). Additional components and plugins are also provided for *reprounzip*: **reprounzip-vagrant**, which unpacks the experiment in a Vagrant virtual machine; **reprounzip-docker**, which unpacks the experiment in a Docker container; and **reprounzip-vistrails**, which creates a VisTrails workflow to reproduce the experiment. More plugins may be developed in the future (and, of course, you are free to :ref:`roll your own <develop-plugins>`).
In our `website <https://www.reprozip.org/>`__, you can find links to our PyPI packages and our `GitHub repository <https://github.com/VIDA-NYU/reprozip>`__.
In the following, you will find installation instructions for :ref:`linux`, :ref:`mac`, and :ref:`windows`. ReproZip is also available for the :ref:`conda` Python distribution.
.. _linux:
Linux
=====
For Linux distributions, both *reprozip* and *reprounzip* components are available.
Required Software Packages
--------------------------
Python 2.7.3 or greater, or 3.3 or greater is required to run ReproZip [#bug]_. If you don't have Python on your machine, you can get it from `python.org <https://www.python.org/>`__. You will also need the `pip <https://pip.pypa.io/en/latest/installing/>`__ installer.
Besides Python and pip, each component or plugin to be used may have additional dependencies that you need to install (if you do not have them already installed in your environment), as described below:
+------------------------+-------------------------------------------------+
| Component / Plugin | Required Software Packages |
+========================+=================================================+
| *reprozip* | `SQLite <https://www.sqlite.org/>`__, |
| | Python headers, |
| | a working C compiler |
+------------------------+-------------------------------------------------+
| *reprounzip* | None |
+------------------------+-------------------------------------------------+
| *reprounzip-vagrant* | Python headers, |
| | a working C compiler, |
| | `Vagrant v1.1+ <https://www.vagrantup.com/>`__, |
| | `VirtualBox <https://www.virtualbox.org/>`__ |
+------------------------+-------------------------------------------------+
| *reprounzip-docker* | `Docker <https://www.docker.com/>`__ |
+------------------------+-------------------------------------------------+
| *reprounzip-vistrails* | None [#vis1]_ |
+------------------------+-------------------------------------------------+
Debian and Ubuntu
`````````````````
You can get all the required dependencies using APT::
apt-get install python3 python3-dev python3-pip gcc libsqlite3-dev libssl-dev libffi-dev
Fedora & CentOS
```````````````
You can get the dependencies using the Yum packaging manager::
yum install python3 python3-devel gcc sqlite-devel openssl-devel libffi-devel
.. [#bug] ``reprozip`` and ``reprounzip graph`` will not work before 2.7.3 due to `Python bug 13676 <https://bugs.python.org/issue13676>`__ related to sqlite3. Python 2.6 is ancient and unsupported.
.. [#vis1] `VisTrails v2.2.3+ <https://www.vistrails.org/>`__ is required to run the workflow generated by the plugin.
Installing *reprozip*
---------------------
To install or update the *reprozip* component, simply run the following command::
$ pip install -U reprozip
Installing *reprounzip*
-----------------------
You can install or update *reprounzip* with all the available components and plugins using::
$ pip install -U reprounzip[all]
Or you can install *reprounzip* and choose components manually::
# Example, this installs all the components
$ pip install -U reprounzip reprounzip-docker reprounzip-vagrant reprounzip-vistrails
.. _mac:
Mac OS X
========
For Mac OS X, only the *reprounzip* component is available.
Binaries
--------
An installer containing Python 2.7, *reprounzip*, and all the plugins can be `downloaded from GitHub <http://reprozip-files.s3-website-us-east-1.amazonaws.com/mac-installer>`__.
Required Software Packages
--------------------------
Python 2.7.3 or greater, or 3.3 or greater is required to run ReproZip [#bug2]_. If you don't have Python on your machine, you can get it from `python.org <https://www.python.org/>`__; you should prefer a 2.x release to a 3.x one. You will also need the `pip <https://pip.pypa.io/en/latest/installing/>`__ installer.
Besides Python and pip, each component or plugin to be used may have additional dependencies that you need to install (if you do not have them already installed in your environment), as described below:
+------------------------+-------------------------------------------------+
| Component / Plugin | Required Software Packages |
+========================+=================================================+
| *reprounzip* | None |
+------------------------+-------------------------------------------------+
| *reprounzip-vagrant* | Python headers, |
| | `Vagrant v1.1+ <https://www.vagrantup.com/>`__, |
| | `VirtualBox <https://www.virtualbox.org/>`__ |
+------------------------+-------------------------------------------------+
| *reprounzip-docker* | `Docker <https://www.docker.com/>`__ |
+------------------------+-------------------------------------------------+
| *reprounzip-vistrails* | None [#vis2]_ |
+------------------------+-------------------------------------------------+
You will need Xcode installed, which you can get from the Mac App Store, and the Command Line Developer Tools; instrucions on installing the latter may depend on your Mac OS X version (some information on StackOverflow `here <https://stackoverflow.com/questions/9329243/how-to-install-xcode-command-line-tools/9329325#9329325>`__).
.. seealso:: :ref:`Why does reprounzip-vagrant installation fail with error "unknown argument: -mno-fused-madd" on Mac OS X? <compiler_mac>`
.. [#bug2] ``reprozip`` and ``reprounzip graph`` will not work before 2.7.3 due to `Python bug 13676 <https://bugs.python.org/issue13676>`__ related to sqlite3. Python 2.6 is ancient and unsupported.
.. [#vis2] `VisTrails v2.2.3+ <https://www.vistrails.org/>`__ is required to run the workflow generated by the plugin.
Installing *reprounzip*
-----------------------
First, be sure to upgrade `setuptools`::
$ pip install -U setuptools
You can install or update *reprounzip* with all the available components and plugins using::
$ pip install -U reprounzip[all]
Or you can install *reprounzip* and choose components manually::
# Example, this installs all the components
$ pip install -U reprounzip reprounzip-docker reprounzip-vagrant reprounzip-vistrails
.. _windows:
Windows
=======
For Windows, only the *reprounzip* component is available.
Binaries
--------
A 32-bit installer containing Python 2.7, *reprounzip*, and all the plugins can be `downloaded from GitHub <http://reprozip-files.s3-website-us-east-1.amazonaws.com/windows-installer>`__.
Required Software Packages
--------------------------
Python 2.7.3 or greater, or 3.3 or greater is required to run ReproZip [#bug3]_. If you don't have Python on your machine, you can get it from `python.org <https://www.python.org/>`__; you should prefer a 2.x release to a 3.x one. You will also need the `pip <https://pip.pypa.io/en/latest/installing/>`__ installer.
Besides Python and pip, each component or plugin to be used may have additional dependencies that you need to install (if you do not have them already installed in your environment), as described below:
+------------------------+------------------------------------------------------------------------+
| Component / Plugin | Required Software Packages |
+========================+========================================================================+
| *reprounzip* | None |
+------------------------+------------------------------------------------------------------------+
| *reprounzip-vagrant* | `Vagrant v1.1+ <https://www.vagrantup.com/>`__, |
| | `VirtualBox <https://www.virtualbox.org/>`__ |
+------------------------+------------------------------------------------------------------------+
| *reprounzip-docker* | `Docker <https://www.docker.com/>`__ [#windowshome]_ |
+------------------------+------------------------------------------------------------------------+
| *reprounzip-vistrails* | None [#vis3]_ |
+------------------------+------------------------------------------------------------------------+
.. [#bug3] ``reprozip`` and ``reprounzip graph`` will not work before 2.7.3 due to `Python bug 13676 <https://bugs.python.org/issue13676>`__ related to sqlite3. Python 2.6 is ancient and unsupported.
.. [#vis3] `VisTrails v2.2.3+ <https://www.vistrails.org/>`__ is required to run the workflow generated by the plugin.
.. [#windowshome] Windows 10 Professional Edition is required for Docker, it will not work on Windows 10 Home Edition because it requires the Windows Hyper-V features.
Installing *reprounzip*
-----------------------
You can install or update *reprounzip* with all the available components and plugins using::
$ pip install -U reprounzip[all]
Or you can install *reprounzip* and choose components manually::
# Example, this installs all the components
$ pip install -U reprounzip reprounzip-docker reprounzip-vagrant reprounzip-vistrails
.. _conda:
Anaconda
========
*reprozip* and *reprounzip* can also be installed on the `Anaconda <https://www.anaconda.com/download>`__ Python distribution, from anaconda.org::
$ conda install --channel conda-forge reprozip reprounzip reprounzip-docker reprounzip-vagrant reprounzip-vistrails
Note, however, that *reprozip* is only available for Linux.
================================================
FILE: docs/jupyter.rst
================================================
.. _reprozip-jupyter:
Making Jupyter Notebooks Reproducible with ReproZip
***************************************************
**reprozip-jupyter** is a plugin for `Jupyter Notebooks <https://jupyter.org>`__, a popular open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. These are valuable documents for data cleaning, analysis, writing executable papers/articles, and more. However, Jupyter Notebooks are subject to dependency hell like any other application -- just the Notebook is not enough for full reproducibility. We have written a ReproZip plugin for Jupyter Notebooks to help users automatically capture dependencies (including data, environment variables, etc.) of Notebooks and also automatically set up those dependencies in another computing environment.
Installation
============
You can install *reprozip-jupyter* with pip::
$ pip install reprozip-jupyter
Or Anaconda::
$ conda install --channel conda-forge reprozip-jupyter
Once successfully installed, you should then enable the plugin for both the client and server side of Jupyter Notebooks::
$ jupyter nbextension install --py reprozip_jupyter --user
$ jupyter nbextension enable --py reprozip_jupyter --user
$ jupyter serverextension enable --py reprozip_jupyter --user
Once these steps are completed, when you start a Jupyter Notebook server, you should be able to see the ReproZip button in your notebook's toolbar.
Packing
=======
Once you have a notebook that executes the way you want, you can trace and pack all the dependencies, data, and provenance with *reprozip-jupyter* by simply clicking the button on the notebook's toolbar:
.. image:: figures/rzj-button.png
The notebook will execute from top-to-bottom and *reprozip-jupyter* traces that execution. If there are no errors in the execution, you'll see two pop-ups like this one after the other:
.. image:: figures/rzj-running.png
*reprozip-jupyter* will name the resulting ReproZip bundle (*.rpz*) as ``notebookname_datetime.rpz`` and save it to the same working directory the notebook is in:
.. image:: figures/rzj-pkg.png
Note that the notebook file itself (``.ipynb``) is not included in the bundle, so you should share or archive both of those files. The reason is that a lot of services can render notebooks (GitHub, OSF...), and they wouldn't be able to if it was in the RPZ file.
Unpacking
=========
Now, anyone can rerun the Jupyter notebook, with all dependencies automatically configured. First, they would need to install *reprounzip* and the *reprounzip-docker* plugin (see the :ref:`installation steps <install>`). Second, they need to download or otherwise acquire the ``.rpz`` file and original ``.ipynb`` notebook they'd like to reproduce.
To reproduce the notebook using the GUI, follow these steps:
1. Double-click the .rpz file.
2. The first tab in the window that appears is for you to set up how you'd like ReproUnzip to unpack and configure the contents of the *.rpz*. Choose docker as your unpacker, and choose the directory you'd like to unpack into.
3. Make sure the Jupyter Integration is checked, and click Run experiment:
.. image:: figures/rzj-setup.png
4. This second table allows you to interact with and rerun the notebook. All you need to do is click 'Run Experiment' and the Jupyter Notebook home file list should pop up in your default browser (if not, navigate to ``localhost:8888``). Open the notebook, and rerun with every dependency configured for you!
.. image:: figures/rzj-run.png
On the command line, you would:
1. Set up the experiment using *reprounzip-docker*::
$ reprounzip docker setup <bundle.rpz> <directory>
2. Rerun the notebook using *reprozip-jupyter*::
$ reprozip-jupyter run <directory>
3. The Jupyter Notebook home file list should pop up in your default browser (if not, navigate to ``localhost:8888``).
4. Open the notebook, and rerun with every dependency configured for you!
================================================
FILE: docs/make.bat
================================================
@ECHO OFF
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set BUILDDIR=_build
set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% .
set I18NSPHINXOPTS=%SPHINXOPTS% .
if NOT "%PAPER%" == "" (
set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS%
set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS%
)
if "%1" == "" goto help
if "%1" == "help" (
:help
echo.Please use `make ^<target^>` where ^<target^> is one of
echo. html to make standalone HTML files
echo. dirhtml to make HTML files named index.html in directories
echo. singlehtml to make a single large HTML file
echo. pickle to make pickle files
echo. json to make JSON files
echo. htmlhelp to make HTML files and a HTML help project
echo. qthelp to make HTML files and a qthelp project
echo. devhelp to make HTML files and a Devhelp project
echo. epub to make an epub
echo. epub3 to make an epub3
echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter
echo. text to make text files
echo. man to make manual pages
echo. texinfo to make Texinfo files
echo. gettext to make PO message catalogs
echo. changes to make an overview over all changed/added/deprecated items
echo. xml to make Docutils-native XML files
echo. pseudoxml to make pseudoxml-XML files for display purposes
echo. linkcheck to check all external links for integrity
echo. doctest to run all doctests embedded in the documentation if enabled
echo. coverage to run coverage check of the documentation if enabled
echo. dummy to check syntax errors of document sources
goto end
)
if "%1" == "clean" (
for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i
del /q /s %BUILDDIR%\*
goto end
)
REM Check if sphinx-build is available and fallback to Python version if any
%SPHINXBUILD% 1>NUL 2>NUL
if errorlevel 9009 goto sphinx_python
goto sphinx_ok
:sphinx_python
set SPHINXBUILD=python -m sphinx.__init__
%SPHINXBUILD% 2> nul
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
:sphinx_ok
if "%1" == "html" (
%SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The HTML pages are in %BUILDDIR%/html.
goto end
)
if "%1" == "dirhtml" (
%SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml.
goto end
)
if "%1" == "singlehtml" (
%SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml.
goto end
)
if "%1" == "pickle" (
%SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle
if errorlevel 1 exit /b 1
echo.
echo.Build finished; now you can process the pickle files.
goto end
)
if "%1" == "json" (
%SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json
if errorlevel 1 exit /b 1
echo.
echo.Build finished; now you can process the JSON files.
goto end
)
if "%1" == "htmlhelp" (
%SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp
if errorlevel 1 exit /b 1
echo.
echo.Build finished; now you can run HTML Help Workshop with the ^
.hhp project file in %BUILDDIR%/htmlhelp.
goto end
)
if "%1" == "qthelp" (
%SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp
if errorlevel 1 exit /b 1
echo.
echo.Build finished; now you can run "qcollectiongenerator" with the ^
.qhcp project file in %BUILDDIR%/qthelp, like this:
echo.^> qcollectiongenerator %BUILDDIR%\qthelp\ReproZip.qhcp
echo.To view the help file:
echo.^> assistant -collectionFile %BUILDDIR%\qthelp\ReproZip.ghc
goto end
)
if "%1" == "devhelp" (
%SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp
if errorlevel 1 exit /b 1
echo.
echo.Build finished.
goto end
)
if "%1" == "epub" (
%SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The epub file is in %BUILDDIR%/epub.
goto end
)
if "%1" == "epub3" (
%SPHINXBUILD% -b epub3 %ALLSPHINXOPTS% %BUILDDIR%/epub3
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The epub3 file is in %BUILDDIR%/epub3.
goto end
)
if "%1" == "latex" (
%SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
if errorlevel 1 exit /b 1
echo.
echo.Build finished; the LaTeX files are in %BUILDDIR%/latex.
goto end
)
if "%1" == "latexpdf" (
%SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
cd %BUILDDIR%/latex
make all-pdf
cd %~dp0
echo.
echo.Build finished; the PDF files are in %BUILDDIR%/latex.
goto end
)
if "%1" == "latexpdfja" (
%SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
cd %BUILDDIR%/latex
make all-pdf-ja
cd %~dp0
echo.
echo.Build finished; the PDF files are in %BUILDDIR%/latex.
goto end
)
if "%1" == "text" (
%SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The text files are in %BUILDDIR%/text.
goto end
)
if "%1" == "man" (
%SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The manual pages are in %BUILDDIR%/man.
goto end
)
if "%1" == "texinfo" (
%SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo.
goto end
)
if "%1" == "gettext" (
%SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The message catalogs are in %BUILDDIR%/locale.
goto end
)
if "%1" == "changes" (
%SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes
if errorlevel 1 exit /b 1
echo.
echo.The overview file is in %BUILDDIR%/changes.
goto end
)
if "%1" == "linkcheck" (
%SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck
if errorlevel 1 exit /b 1
echo.
echo.Link check complete; look for any errors in the above output ^
or in %BUILDDIR%/linkcheck/output.txt.
goto end
)
if "%1" == "doctest" (
%SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest
if errorlevel 1 exit /b 1
echo.
echo.Testing of doctests in the sources finished, look at the ^
results in %BUILDDIR%/doctest/output.txt.
goto end
)
if "%1" == "coverage" (
%SPHINXBUILD% -b coverage %ALLSPHINXOPTS% %BUILDDIR%/coverage
if errorlevel 1 exit /b 1
echo.
echo.Testing of coverage in the sources finished, look at the ^
results in %BUILDDIR%/coverage/python.txt.
goto end
)
if "%1" == "xml" (
%SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The XML files are in %BUILDDIR%/xml.
goto end
)
if "%1" == "pseudoxml" (
%SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml
if errorlevel 1 exit /b 1
echo.
echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml.
goto end
)
if "%1" == "dummy" (
%SPHINXBUILD% -b dummy %ALLSPHINXOPTS% %BUILDDIR%/dummy
if errorlevel 1 exit /b 1
echo.
echo.Build finished. Dummy builder generates no files.
goto end
)
:end
================================================
FILE: docs/man/reprounzip-docker.1
================================================
.\" Manpage for reprounzip
.TH man 1 "4 November 2017" "1.0.10" "reprounzip\-docker man page"
.SH NAME
reprounzip\-docker \- Docker unpacker for reprounzip
.SH SYNOPSIS
reprounzip [\-v] docker [\-h] [\-\-version] [\-\-docker\-cmd DOCKER] ...
.SH DESCRIPTION
reprounzip\-docker is the unpacker plugin for reprounzip that allows reproducing packed experiments on different environments by using a Docker container. You will need Docker to be installed on your machine to use it.
.SH OPTIONS
.SS General options:
.TP
.B \-h, \-\-help
show this help message and exit
.TP
.B \-\-version
show program's version number and exit
.TP
.B \-v, \-\-verbose
augments verbosity level
.TP
.B \-\-docker\-cmd DOCKER
changes the location or command used to run the Docker client. This option is split on spaces, for example you can use
\-\-docker\-cmd 'sudo /opt/bin/docker'
.SS Commands:
.TP
.BI setup " [\-\-base\-image IMAGE] [\-\-distribution DISTRIBUTION] [\-\-install\-pkgs] [\-\-image\-name NAME] [\-\-docker\-option OPTION] <package.rpz> <directory>"
Unpacks an experiment and sets it up for reproduction in the specified directory. A Docker image
.I NAME
will be created from the specified base
.I IMAGE
by adding the experiment's content on top of it. You can use \-\-docker\-option to pass additional options to docker build for example \-\-docker\-option=\-\-squash.
.TP
.BI run " [\-\-expose\-port PORT] [\-\-enable\-x11] [\-\-x11\-display DISPLAY] [\-\-tunneled\-x11] [\-d] [\-\-docker\-option OPTION] [\-\-pass\-env VAR] [\-\-set\-env VAR[=val]] <directory> [run] [\-\-cmdline ...]"
Runs an experiment that has been unpacked to a directory.
.TP
.BI upload " <directory> [localfile:experimentfile] [...]"
Replaces a file inside the experiment image with your own. This allows you to reproduce an experiment in its original environment, but with modified data or configuration. Running this command with only the
.I directory
will list all the named files that you can replace with this command.
.TP
.BI download " <directory> [experimentfile[:localfile]] [...]"
Download a file from the experiment container to your filesystem. If
.I localfile
is not specified, a file of the same name will be written in the current directory. Running this command with only the
.I directory
will list all the named files that you can download with this command.
.TP
.BI reset " <directory>"
Reset the experiment to the state it was when it was first unpacked. This undoes the effects of
.BR upload " and " run
commands on the experiment's environment.
.TP
.BI destroy " <directory>"
Removes an unpacked experiment directory, and the associated Docker images.
.SH SEE ALSO
.TP
The ReproZip website
https://www.reprozip.org/
.TP
ReproZip's GitHub repository
https://github.com/ViDA\-NYU/reprozip
.TP
The Docker website
https://www.docker.com/
.SH BUGS
Please report bugs on our mailing-list at reprozip@nyu.edu or on GitHub at https://github.com/VIDA\-NYU/reprozip/issues.
.SH AUTHOR
.RB "ReproZip is being developed at" " New York University" .
The team includes:
.RS
.nf
Remi Rampin
Fernando Chirigati
Vicky Rampin
Juliana Freire
Dennis Shasha
.fi
.RE
.SH COPYRIGHT
Copyright 2014 New York University.
.RB "Licensed under a" " BSD 3-Clause license." " See the LICENSE file included with the software for details."
================================================
FILE: docs/man/reprounzip-vagrant.1
================================================
.\" Manpage for reprounzip
.TH man 1 "4 November 2017" "1.0.10" "reprounzip\-vagrant man page"
.SH NAME
reprounzip\-vagrant \- Vagrant unpacker for reprounzip
.SH SYNOPSIS
reprounzip [\-v] vagrant [\-h] [\-\-version] ...
.SH DESCRIPTION
reprounzip\-vagrant is the unpacker plugin for reprounzip that allows reproducing packed experiments on different environments by using Vagrant to create a virtual machine. You will need Vagrant and VirtualBox to be installed on your machine to use it.
.SH OPTIONS
.SS General options:
.TP
.B \-h, \-\-help
show this help message and exit
.TP
.B \-\-version
show program's version number and exit
.TP
.B \-v, \-\-verbose
augments verbosity level
.SS Commands:
.TP
.BI setup " [\-\-dont\-use\-chroot] [\-\-dont\-bind\-magic\-dirs] [\-\-base\-image IMAGE] [\-\-distribution DISTRIBUTION] [\-\-memory MEMORY] [\-\-use\-gui] [\-\-expose\-port PORT] <package.rpz> <directory>"
Unpacks an experiment and sets it up for reproduction in the specified directory. A virtual machine will be created from the specified base
.I IMAGE
by adding the experiment's content on top of it. The default is to extract the experiment's files in a chroot inside the machine, and binding "magic dirs" into it
.RI ( /proc ", " /dev ", ...);"
this can be disabled using \-\-dont\-use\-chroot.
.TP
.BI run " [\-\-no\-stdin] [\-\-no\-pty] [\-\-expose\-port PORT] [\-\-enable\-x11] [\-\-x11\-display DISPLAY] [\-\-pass\-env VAR] [\-\-set\-env VAR[=val]] <directory> [run] [\-\-cmdline ...]"
Runs an experiment that has been unpacked to a directory.
.TP
.BI upload " <directory> [localfile:experimentfile] [...]"
Replaces a file inside the virtual machine with your own. This allows you to reproduce an experiment in its original environment, but with modified data or configuration. Running this command with only the
.I directory
will list all the named files that you can replace with this command.
.TP
.BI download " <directory> [experimentfile[:localfile]] [...]"
Download a file from the virtual machine to your filesystem. If
.I localfile
is not specified, a file of the same name will be written in the current directory. Running this command with only the
.I directory
will list all the named files that you can download with this command.
.TP
.BI destroy " <directory>"
Removes an unpacked experiment directory, and the associated virtual machine.
.SH SEE ALSO
.TP
The ReproZip website
https://www.reprozip.org/
.TP
ReproZip's GitHub repository
https://github.com/ViDA\-NYU/reprozip
.TP
The Vagrant website
https://www.vagrantup.com/
.SH BUGS
Please report bugs on our mailing-list at reprozip@nyu.edu or on GitHub at https://github.com/VIDA\-NYU/reprozip/issues.
.SH AUTHOR
.RB "ReproZip is being developed at" " New York University" .
The team includes:
.RS
.nf
Remi Rampin
Fernando Chirigati
Vicky Rampin
Juliana Freire
Dennis Shasha
.fi
.RE
.SH COPYRIGHT
Copyright 2014 New York University.
.RB "Licensed under a" " BSD 3-Clause license." " See the LICENSE file included with the software for details."
================================================
FILE: docs/man/reprounzip.1
================================================
.\" Manpage for reprounzip
.TH man 1 "4 November 2017" "1.0.11" "reprounzip man page"
.SH NAME
reprounzip \- the reproducibility unpacker
.SH SYNOPSIS
reprounzip [\-h] [\-\-version] [\-v] ...
.SH DESCRIPTION
reprounzip inspects or unpacks a reproducible experiment package (.rpz file) created by reprozip.
.SH OPTIONS
.SS General options:
.TP
.B \-h, \-\-help
show this help message and exit
.TP
.B \-\-version
show program's version number and exit
.TP
.B \-v, \-\-verbose
augments verbosity level
.SS Commands:
.TP
.B usage_report
Enables or disables anonymous usage reports
.TP
.B info
Prints out some information about a pack
.TP
.BI graph " <output.dot> [package.rpz]"
Generates a provenance graph from the trace data
.TP
.BI showfiles " <package>"
Prints out input and output file names
.TP
.B installpkgs " <package.rpz>"
Installs the required packages on this system
.TP
.BR directory " ..."
Unpacks the files in a directory and runs with PATH and LD_LIBRARY_PATH
.TP
.B chroot
Unpacks the files and run with chroot
.TP
.B docker
Runs the experiment in a Docker container
.TP
.B vagrant
Runs the experiment in a virtual machine created through Vagrant
.SS Unpacker commands:
Each unpacker (such as
.BR directory ", " chroot ", " docker ", or " vagrant )
has a similar set of commands, detailed below. Please refer to
.BR reprounzip\-\f(BIunpacker (1)
for detailed information about a specific unpacker.
.TP
.BI setup " <package.rpz> <directory>"
Unpacks an experiment and sets it up for reproduction in the specified directory.
.TP
.BI run " <directory> [run] [\-\-cmdline ...]"
Runs an experiment that has been unpacked to a directory.
.TP
.BI upload " <directory> [localfile:experimentfile] [...]"
Replaces a file inside the experiment with your own. This allows you to reproduce an experiment in its original environment, but with modified data or configuration. Running this command with only the
.I directory
will list all the named files that you can replace with this command.
.TP
.BI download " <directory> [experimentfile[:localfile]] [...]"
Download a file from the experiment container/virtual machine/directory to your filesystem. If
.I localfile
is not specified, a file of the same name will be written in the current directory. Running this command with only the
.I directory
will list all the named files that you can download with this command.
.TP
.BI destroy " <directory>"
Removes an unpacked experiment directory, and associated resources such as containers, virtual machines, or mount points.
.B Always use this command
to delete an unpacked experiment, otherwise resources might get left behind, or with chroot you might end up
.B deleting system files
through the mount point that has been created.
.SH FILES
.TP
.B ~/.reprozip/log
Log file where reprozip and reprounzip write messages during execution. Useful to report issues to the developers, even if you were running without
.IR \-\-verbose .
.SH ENVIRONMENT
.TP
.B REPROZIP_USAGE_STATS
If this variable is set to
.I \*(lqoff\*(rq
then usage statistics will be completely disabled (won't be reported, recorded, and won't ask you about it).
.TP
.B REPROZIP_PARAMETERS
If this variable is set to a URL, it specifies an alternate location from which to download runtime parameters. If it is set to
.IR \*(lqoff\*(rq ,
nothing will be downloaded and the bundled parameters will be used instead.
.SH EXAMPLES
.P
Get information on a package:
.IP
.nf
.RB "$" " reprounzip \-v info" " foobar0.4\-python2.rpz"
\-\-\-\-\- Pack information \-\-\-\-\-
Compressed size: 1.64 MB
Unpacked size: 4.98 MB
[...]
.fi
.P
Unpack the experiment using
.BR reprounzip\-docker (1)
under
.IR /tmp :
.IP
.nf
.RB "$" " reprounzip docker setup" " foobar0.4\-python2.rpz /tmp/foobar\-docker"
.RB "$" " reprounzip docker run" " /tmp/foobar\-docker"
.RB "$" " reprounzip docker destroy" " /tmp/foobar\-docker"
.fi
.SH SEE ALSO
.TP
The ReproZip website
https://www.reprozip.org/
.TP
ReproZip's GitHub repository
https://github.com/ViDA\-NYU/reprozip
.SH BUGS
Please report bugs on our mailing-list at reprozip@nyu.edu or on GitHub at https://github.com/VIDA\-NYU/reprozip/issues.
.SH AUTHOR
.RB "ReproZip is being developed at" " New York University" .
The team includes:
.RS
.nf
Remi Rampin
Fernando Chirigati
Vicky Rampin
Juliana Freire
Dennis Shasha
.fi
.RE
.SH COPYRIGHT
Copyright 2014 New York University.
.RB "Licensed under a" " BSD 3-Clause license." " See the LICENSE file included with the software for details."
================================================
FILE: docs/man/reprozip.1
================================================
.\" Manpage for reprozip
.TH man 1 "4 November 2017" "1.0.11" "reprozip man page"
.SH NAME
reprozip \- the reproducibility packer
.SH SYNOPSIS
reprozip [\-h] [\-\-version] [\-d DIR] [\-\-dont\-identify\-packages]
[\-\-dont\-find\-inputs\-outputs] [\-v] ...
.SH DESCRIPTION
reprozip traces and packs an experiment or program into a reproducible package (.rpz file). Those packages can be archived, and reproduced later using reprounzip(1).
.SH OPTIONS
.SS General options:
.TP
.B \-h, \-\-help
show this help message and exit
.TP
.B \-\-version
show program's version number and exit
.TP
.B \-d DIR, \-\-dir DIR
where to store database and configuration file (default: ./.reprozip\-trace)
.TP
.B \-\-dont\-identify\-packages
do not try identify which package each file comes from
.TP
.B \-\-dont\-find\-inputs\-outputs
do not try to identify input and output files
.TP
.B \-v, \-\-verbose
augments verbosity level
.SS Commands:
.TP
.B usage_report
Enables or disables anonymous usage reports
.TP
.B trace
Runs the program and writes out database and configuration file
.TP
.B testrun
Runs the program and writes out the database contents
.TP
.B reset
Resets the configuration file
.TP
.B pack
Packs the experiment according to the current configuration
.TP
.B combine
Combine multiple traces into one (possibly as subsequent runs)
.SH EXIT STATUS
reprozip returns 1 in case of internal error, otherwise it returns the same result as the traced command.
.SH FILES
.TP
.B ~/.reprozip/log
Log file where reprozip and reprounzip write messages during execution. Useful to report issues to the developers, even if you were running without
.IR \-\-verbose .
.TP
.B .reprozip\-trace/trace.sqlite3, .reprozip\-trace/config.yml
When tracing, the trace database and configuration file get written to a directory
.I .reprozip\-trace
in the current directory. This location can be overridden with
.IR \-\-dir .
It can be useful to edit the configuration file before packing, to make sure no useless or private file is to be included in the package, and to correctly label input files, output files, and run steps.
.SH ENVIRONMENT
.TP
.B REPROZIP_USAGE_STATS
If this variable is set to
.I \*(lqoff\*(rq
then usage statistics will be completely disabled (won't be reported, recorded, and won't ask you about it).
.SH EXAMPLES
.P
Trace and pack a simple Python script:
.IP
.nf
.RB "$" " reprozip trace" " python foobar.py"
.RB "$" " reprozip pack" " foobar0.4\-python2.rpz"
.fi
.P
Use testrun to show the files used by a command:
.IP
.nf
.RB "$" " reprozip testrun" " /usr/bin/id"
.fi
.SH SEE ALSO
.TP
The ReproZip website
https://www.reprozip.org/
.TP
ReproZip's GitHub repository
https://github.com/ViDA\-NYU/reprozip
.SH BUGS
Please report bugs on our mailing-list at reprozip@nyu.edu or on GitHub at https://github.com/VIDA\-NYU/reprozip/issues.
.SH AUTHOR
.RB "ReproZip is being developed at" " New York University" .
The team includes:
.RS
.nf
Remi Rampin
Fernando Chirigati
Vicky Rampin
Juliana Freire
Dennis Shasha
.fi
.RE
.SH COPYRIGHT
Copyright 2014 New York University.
.RB "Licensed under a" " BSD 3-Clause license." " See the LICENSE file included with the software for details."
================================================
FILE: docs/packing.rst
================================================
.. _packing:
Using *reprozip*
****************
The *reprozip* component is responsible for packing an experiment, which is done in three steps: :ref:`tracing the experiment <packing-trace>`, :ref:`editing the configuration file <packing-config>` (if necessary), and :ref:`creating the reproducible package <packing-pack>`. Each of these steps is explained in more details below. Please note that *reprozip* is only available for Linux distributions.
.. _packing-trace:
Tracing an Experiment
=====================
First, *reprozip* needs to trace the operating system calls used by the experiment, so as to identify all the necessary information for its future re-execution, such as binaries, files, library dependencies, and environment variables.
The following command is used to trace a command line, or a `run`, used by the experiment::
$ reprozip trace <command-line>
where `<command-line>` is the command line. By running this command, *reprozip* executes `<command-line>` and uses ``ptrace`` to trace all the system calls issued, storing them in an SQLite database.
If you run the command multiple times, *reprozip* might ask you if you want to continue with your current trace (append the new command-line to it) or replace it (throw away the previous command-line you traced). You can skip this prompt by using either the ``--continue`` or ``--overwrite`` flag, like this::
$ reprozip trace --continue <command-line>
Note that the final bundle will be able to reproduce any of the runs, and files shared by multiple runs are only stored once.
By default, if the operating system is based on Debian or RPM packages (e.g.: Ubuntu, CentOS, Fedora, ...), *reprozip* will also try to automatically identify the distribution packages from which the files come, using the available package manager of the system. This is useful to provide more detailed information about the dependencies, as well as to further help when reproducing the experiment. However, note that the ``trace`` command can take some time doing that after the experiment finishes, depending on the number of file dependencies that the experiment has. To disable this feature, users may use the flag ``--dont-identify-packages``::
$ reprozip trace --dont-identify-packages <command-line>
The database, together with a *configuration file* (see below), are placed in a directory named ``.reprozip-trace``, created under the path where the ``reprozip trace`` command was issued.
.. _packing-config:
Editing the Configuration File
==============================
The configuration file, which can be found in ``.reprozip-trace/config.yml``, contains all the information necessary for creating the experiment bundle. This file is generated by the tracer and drives the packing step.
It is very likely that you won't need to modify this file, as the automatically-generated one should be sufficient to create a working bundle. However, in some cases, you may want to edit it prior to the creation of the package to add or remove files used by your experiment. This can be particularly useful, for instance, to remove big files that can be obtained elsewhere when reproducing the experiment, to keep the size of package small, and also to remove sensitive information that the experiment may use. The configuration file can also be used to edit the main command line, to add or remove environment variables, and to edit information regarding input/output files.
.. _packing-config-general:
The first part of the configuration file gives general information with respect to the experiment and its runs, including command lines, environment variables, working directory, and machine information. Also, each run has a unique identifier (given by ``id``) that is consistently used for packing and unpacking purposes::
# Run info
version: <reprozip-version>
runs:
# Run 0
- id: <run-id>
architecture: <machine-architecture>
argv: <command-line-arguments>
binary: <command-line-binary>
distribution: <linux-distribution>
environ: <environment-variables>
exitcode: <exit-code>
gid: <group-id>
hostname: <machine-hostname>
system: <system-kernel>
uid: <user-id>
workingdir: <working-directory>
# Run 1
- id: ...
...
If necessary, users may change command line parameters by editing ``argv``, and add or remove environment variables by editing ``environ``. Users may also give a more meaningful and user-friendly identifier for a run by changing ``id``. Other attributes should not be changed in general.
.. _packing-config-inputoutput:
The next section brings information about input and output files, including their original paths and which runs read and/or wrote them. These are the files that `reprozip` identified as the main input or result of the experiment, which `reprounzip` will later be able to replace and extract from the experiment when reproducing it. You may add, remove, or edit these files in case `reprozip` fails in recognizing any important information, as well as give meaningful names to them by editing ``name``::
# Inputs are files that are only read by a run; reprounzip can replace these
# files on demand to run the experiment with custom data.
# Outputs are files that are generated by a run; reprounzip can extract these
# files from the experiment on demand, for the user to examine.
# The name field is the identifier the user will use to access these files.
inputs_outputs:
- name: <file-identifier>
path: <path-to-file>
read_by_runs: <run-ids>
written_by_runs: <run-ids>
- name: ...
...
Note that you can prevent `reprozip` from identifying which files are input or output by using the ``--dont-find-inputs-outputs`` flag in the ``reprozip trace`` command.
.. note:: To visualize the dataflow of the experiment, pleaser refer to :ref:`graph`.
.. seealso:: :ref:`Why doesn’t 'reprozip' identify my input/output file? <file_id>`
.. _packing-config-files:
The next section in the configuration file lists all the files to be packed. If the software dependencies were identified by the package manager of the system during the ``reprozip trace`` command, they will be organized in software packages and listed under ``packages``; otherwise, file dependencies will be listed under ``other_files``::
packages:
- name: <package-name>
version: <package-version>
size: <package-size>
packfiles: <include-package>
files:
# Total files used: <used-files-size>
# Installed package size: <package-size>
<files-list>
- name: ...
...
other_files:
<files-list>
The attribute ``packfiles`` can be used to control whether a software package will be packed: its default value is `true`, but users may change it to `false` to inform *reprozip* that the corresponding software package should not be included. To remove a file that was not identified as part of a package, users can simply remove it from the list under ``other_files``.
.. warning::
Note that if a software package is requested not to be included, the `reprounzip` component will try to install it from a package manager when unpacking the experiment. If the software version from the package manager is different from (and incompatible with) the one used by the experiment, the experiment may not be reproduced correctly.
.. seealso:: :ref:`Why does 'reprounzip run' fail with "no such file or directory" or similar? <nosuchfile>`
.. _packing-config-patterns:
Last, users may add file patterns under ``additional_patterns`` to include other files that they think it will be useful for a future reproduction. As an example, the following would add everything under ``/etc/apache2/`` and all the Python files of all users from LXC containers (contrived example)::
additional_patterns:
- /etc/apache2/**
- /var/lib/lxc/*/rootfs/home/**/*.py
Note that users can always reset the configuration file to its initial state by running the following command::
$ reprozip reset
.. warning::
When editing a configuration file, make sure your changes are as restrictive as possible, modifying only the necessary information. Removing important information and changing the structure of the file may cause issues while creating the bundle or unpacking the experiment.
.. _packing-pack:
Creating a Bundle
=================
After tracing all the runs from the experiment and optionally editing the configuration file, the experiment bundle can be created by using the following command::
$ reprozip pack <bundle>
where `<bundle>` is the name given to the package. This command generates a ``.rpz`` file in the current directory, which can then be sent to others so that the experiment can be reproduced. For more information regarding the unpacking step, please see :ref:`unpacking`.
Note that, by using ``reprozip pack``, files will be copied from your environment to the package; as such, you should not change any file that the experiment used before packing it, otherwise the package will contain different files from the ones the experiment used when it was originally traced.
.. warning::
Before sending your bundle to others, it is advisable to test it and ensure that the reproduction of the experiment works.
.. _packing-further:
Further Considerations
======================
Packing Multiple Command Lines
++++++++++++++++++++++++++++++
As mentioned before, ReproZip allows multiple runs (i.e., command lines) to be traced and included in the same bundle. Alternatively, users can create a simple **script** that runs all the command lines, and pass *that* to ``reprozip trace``. However, in this case, there will be no flexibility in choosing a single run to be reproduced, since the entire script will be re-executed.
Note that this flexibility has the caveat that users may reproduce the runs in a different order than the one originally used while tracing. If the order is important for the reproduction (e.g.: each run represents a step in a dataflow), please make sure to inform the correct reproduction order to whoever wants to replicate the experiment. This can also be obtained by running ``reprounzip graph``; please refer to :ref:`provenance-graph` for more information.
ReproZip can also combine multiple traces into a single one, in order to create a single bundle, using the ``reprozip combine`` command. The runs of each subsequent trace are simply appended in order.
Packing GUI and Interactive Tools
+++++++++++++++++++++++++++++++++
ReproZip is able to pack GUI tools. Additionally, there is no restriction in packing interactive experiments (i.e., experiments that require input from users). Note, however, that if entering something different can make the experiment load additional dependencies, the experiment will probably fail when reproduced on a different machine.
.. _packing-clientserv:
Capturing Connections to Servers
++++++++++++++++++++++++++++++++
When reproducing an experiment that communicates with a server, the experiment will try to connect to the same server, which may or may not fail depending on the status of the server at the moment of the reproduction. However, if the experiment uses a local server (e.g.: database) that the user has control over, this server can also be captured, together with the experiment, to ensure that the connection will succeed. Users should create a script to:
* start the server,
* execute the experiment, and
* stop the server,
and use *reprozip* to trace the script execution, rather than the experiment itself. In this way, ReproZip is able to capture the local server as well, which ensures that the server will be alive at the time of the reproduction.
For example, if you have an web app that uses MySQL and that runs until ``Ctrl+C`` is received, you can use the following script::
#!/bin/sh
if [ "$(id -u)" != 0 ]; then echo "This script needs to run as root so that it can execute MySQL" >&2; exit 1; fi
# Start MySQL
sudo -u mysql /usr/sbin/mysqld --pid-file=/run/mysqld/mysqld.pid &
sleep 5
# Don't exit the whole script on Ctrl+C
trap ' ' INT
# Execute actual experiment that uses the database
./manage.py runserver 0.0.0.0:8000
trap - INT
# Graceful shutdown
/usr/bin/mysqladmin shutdown
Note the use of ``trap`` to avoid exiting the entire script when pressing ``Ctrl+C``, to make sure that the database gets shutdown via the next command.
Excluding Sensitive and Third-Party Information
+++++++++++++++++++++++++++++++++++++++++++++++
ReproZip automatically tries to identify log and temporary files, removing them from the bundle, but the configuration file should be edited to remove any sensitive information that the experiment uses, or any third-party file/software that should not be distributed. Note that the ReproZip team is **not responsible** for personal and non-authorized files that may get distributed in a package; users should double-check the configuration file and their package before sending it to others.
Identifying Output Files
++++++++++++++++++++++++
The `reprozip` component tries to automatically identify the main output files generated by the experiment during the ``trace`` command to provide useful interfaces for users during the unpacking step. However, if the experiment creates unique names for its outputs every time it is executed (e.g.: names with current date and time), the *reprounzip* component will not be able to correctly detect these; it assumes that input and output files do not have their path names changed between different executions. In this case, handling output files will fail. It is recommended that users modify their experiment (or use a wrapper script) to generate a symbolic link (with a fixed name) that always points to the latest result, and use that as the output file's path in the configuration file (under the ``inputs_outputs`` section).
================================================
FILE: docs/reprozip.rst
================================================
Why ReproZip?
*************
Reproducibility is a core component of the scientific process: it helps researchers all around the world to verify the results and to also build on them, allowing science to move forward. In natural science, long tradition requires experiments to be described in enough detail so that they can be reproduced by researchers around the world. The same standard, however, has not been widely applied to computational science, where researchers often have to rely on plots, tables, and figures included in papers, which loosely describe the obtained results.
The truth is computational reproducibility can be very painful to achieve for a number of reasons. Take the author-reviewer scenario of a scientific paper as an example. Authors must generate a compendium that encapsulates all the inputs needed to correctly reproduce their experiments: the data, a complete specification of the experiment and its steps, and information about the originating computational environment (OS, hardware architecture, and library dependencies). Keeping track of this information manually is rarely feasible: it is both time-consuming and error-prone. First, computational environments are complex, consisting of many layers of hardware and software, and the configuration of the OS is often hidden. Second, tracking library dependencies is challenging, especially for large experiments. If authors did not plan for reproducibility since the beginning of the project, reproducibility is drastically hampered.
For reviewers, even with a compendium in their hands, it may be hard to reproduce the results. There may be no instructions about how to execute the code and explore it further; the experiment may not run on his operating system; there may be missing libraries; library versions may be different; and several issues may arise while trying to install all the required dependencies, a problem colloquially known as `dependency hell <https://en.wikipedia.org/wiki/Dependency_hell>`__.
ReproZip helps alleviate these problems by allowing the user to easily capture all the necessary components in a single, distributable bundle. Also, the tool makes it easier to reproduce an experiment by providing different unpacking methods and interfaces that avoids the need to install all the required dependencies and that makes it possible to run the experiment under different inputs.
================================================
FILE: docs/traceschema.rst
================================================
.. _trace-schema:
Trace Database Schema
*********************
The database contains three tables: ``processes``, ``opened_files``, and ``executed_files``.
``processes``
'''''''''''''
This table contains information about all the processes. A process is identified by Linux as a *pid* (process id), and is either a thread or a full-fledged process.
Note that processes are different from programs, and there is no one-to-one relationship with executions. A process is created by `clone(2) <https://linux.die.net/man/2/clone>`__ or `fork(2) <https://linux.die.net/man/2/fork>`__ and not necessarily followed by `execve(2) <https://linux.die.net/man/2/execve>`__. By contrast, a program can change its image by calling execve(2) without creating new processes (i.e., without changing *pid*).
Each entry in the ``processes`` table has the id of its parent, i.e. the process that created it by calling clone(2) or fork(2), except the original process that *reprozip* created, for which parent is NULL. There is thus exactly one process with a NULL parent per run stored in the bundle.
::
CREATE TABLE processes(
id INTEGER NOT NULL PRIMARY KEY,
run_id INTEGER NOT NULL,
parent INTEGER,
timestamp INTEGER NOT NULL,
is_thread BOOLEAN NOT NULL,
exitcode INTEGER
);
``opened_files``
''''''''''''''''
This table contains information regarding the files accessed by the processes. Note that a failed access (e.g.: trying to read a non-existing file, permission denied, etc.) is not logged. A single path might appear several times, even if accessed by the same process.
Each file has a numerical id, the canonical path name, the process that accessed it (from which you can get the executable by cross-referencing ``processes``, also using the timestamp), and the mode.
::
CREATE TABLE opened_files(
id INTEGER NOT NULL PRIMARY KEY,
run_id INTEGER NOT NULL,
name TEXT NOT NULL,
timestamp INTEGER NOT NULL,
mode INTEGER NOT NULL,
is_directory BOOLEAN NOT NULL,
process INTEGER NOT NULL
);
The *mode* attribute is a binary OR of the following values (accessible from ``reprounzip.common``)::
FILE_READ = 0x01
FILE_WRITE = 0x02
FILE_WDIR = 0x04
FILE_STAT = 0x08
FILE_LINK = 0x10
``executed_files``
''''''''''''''''''
This is a variant of ``opened_files`` for file executions, i.e. `execve(2) <https://linux.die.net/man/2/execve>`__ calls. There is no mode here (file is opened for reading by the call) and they are never directories; however, *workingdir*, *argv* (command-line arguments) and *envp* (environment variables) are added. *argv* is a list of arguments separated by null bytes (``0x00``) [#nullbytes]_, and *envp* is a list of ``VAR=value`` pairs separated by null (``0x00``) bytes [#nullbytes]_. Note that, again, failed executions (execve returns) are not logged.
::
CREATE TABLE executed_files(
id INTEGER NOT NULL PRIMARY KEY,
name TEXT NOT NULL,
run_id INTEGER NOT NULL,
timestamp INTEGER NOT NULL,
process INTEGER NOT NULL,
argv TEXT NOT NULL,
envp TEXT NOT NULL,
workingdir TEXT NOT NULL
);
.. [#nullbytes] Note that Python's sqlite3 lib is affected by `bug 13676 <https://bugs.python.org/issue13676>`__ up to Python 2.7.3, which prevents it from reading text or blob fields with embedded null bytes.
================================================
FILE: docs/troubleshooting.rst
================================================
.. _troubleshooting:
Troubleshooting
***************
The best way to start solving an issue in ReproZip is probably to look at the log messages. Some messages are not displayed by default when running ReproZip, but you can use the ``--verbose`` (or ``-v``) flag to display them. In addition, all the log messages are stored under ``$HOME/.reprozip/log``.
Please feel free to contact us at reprozip@nyu.edu if you encounter issues while using ReproZip.
------------
.. _file_id:
:Issue: **"** `reprozip` **does not identify my input/output file."**
:Diagnosis: ReproZip uses some heuristics to identify an input or output file. However, this is only intended to be a starting point, since these heuristics may fail.
:Solution: You should check the configuration file and edit the ``inputs_outputs`` section if necessary; giving readable names to input/output files also helps during reproduction. Please refer to :ref:`packing-config` for more information.
------------
.. _systemd:
:Issue: **"None of my files are packed when tracing a daemon."**
:Diagnosis: If you are starting the daemon via the `service` or `systemctl` tool, it might be calling `init` over a client/server connection. For example, this is the case if you are using SystemD. In this situation, ReproZip will successfully pack the client, but anything the server (`init`) does will not be captured, leaving out this entire daemon.
:Solution: You can still trace the binary or a non-systemd `init` script directly.
For example, instead of::
$ reprozip trace service mysql start
or::
$ reprozip trace systemctl start mysql
you can trace the binary::
$ reprozip trace /usr/bin/mysqld
Note that, if you choose to trace the binary, you need to figure out the right command line options to use.
Also, note that running the init script in ``/etc/init.d/...`` is not enough, since those scripts get subverted to call `systemctl` when systemd is installed.
------------
.. _ptrace:
:Issue: **"** `reprozip` **fails with** ``couldn't use ptrace`` **"**
:Diagnosis: ``ptrace`` is the mechanism that ReproZip uses to attach to another process and follow its system calls. Because it is so powerful, some security policies, environments or isolation mechanism may disable it.
:Solution:
* If you are using Docker, you can use the Docker option ``--cap-add=SYS_PTRACE`` (or provide your own seccomp profile that allows ptrace, by adding ``"ptrace"`` to the `default profile <https://github.com/moby/moby/blob/master/profiles/seccomp/default.json>`__; see `the Docker documentation on seccomp <https://docs.docker.com/engine/security/seccomp/>`__).
------------
.. _moving-outputs:
:Issue: **"** `reprounzip` **cannot get an output file using** ``download`` **after reproducing the experiment."**
:Diagnosis: This is probably the case where this output file does not have a fixed path name. It is common for experiments to dynamically choose where the outputs should be written, e.g.: by putting the date and time in the filename. However, ReproZip uses filenames in the ``inputs_outputs`` section of the configuration file to detect those when reproducing the experiment: if the name of the output file when reproducing is different from when it was originally packed, ReproZip cannot detect these as output files, and therefore, cannot get them through the ``download`` command.
:Solution: The easiest way to solve this issue is to re-pack the experiment: write a simple bash script that runs the experiment and either renames outputs or creates symbolic links to them with known filenames; then, trace this script (instead of the actual entry-point of your experiment) and specify these fixed path names in the ``inputs_outputs`` section of the configuration file.
------------
.. _compiler_mac:
:Issue: **"** `reprounzip-vagrant` **installation fails with error** ``unknown argument: '-mno-fused-madd'`` **on Mac OS X."**
:Diagnosis: This is an issue with the Apple LLVM compiler, which treats unrecognized command-line options as errors.
:Solution: As a workaround, before installing `reprounzip-vagrant`, run the following::
$ export CFLAGS="-Wno-error=unused-command-line-argument-hard-error-in-future"
Then, re-install `reprounzip-vagrant`::
$ pip install -I reprounzip-vagrant
Or use the following command in case you want all the available plugins::
$ pip install -I reprounzip[all]
------------
:Issue: **"The experiment fails with** ``Error: Can't open display: :0`` **when trying to reproduce it."**
:Diagnosis: The experiment probably involves running a GUI tool.
:Solution: The `reprounzip` component supports GUI tools, but it is not enabled by default; add the flag ``--enable-x11`` to the ``run`` command to enable it. See :ref:`gui-tools` for more information.
------------
.. _directory_error:
:Issue: **"The experiment run with** `reprounzip directory` **fails to find a file that has been packed."**
:Diagnosis: The `directory` unpacker does not provide any isolation from the filesystem: if the experiment being reproduced use absolute paths, these will point outside the experiment directory, and files may not be found.
:Solution: Make sure that the experiment does not use any absolute paths: if only relative paths are used internally and in the command line, ``reprounzip directory`` should work. As an alternative, you can use other unpackers (e.g.: ``reprounzip chroot`` and ``reprounzip vagrant``) that work in the presence of hardcoded absolute paths.
------------
.. _distribnotfound:
:Issue: **"** `reprounzip` **fails with** ``DistributionNotFound`` **errors."**
:Diagnosis: You probably have some plugins left over from a previous installation.
:Solution: Be sure to upgrade or remove outdated plugins when you upgrade `reprounzip`. The following command may help::
$ pip install -U reprounzip[all]
------------
:Issue: **"** `reprounzip` **shows** ``running in chroot, ignoring request`` **."**
:Diagnosis: This message comes from the systemd client, which will probably not work with ReproZip.
:Solution: In this case, the experiment should be re-packed without using systemd (see :ref:`this issue <systemd>` for more information).
------------
:Issue: **"** ``reprounzip vagrant setup`` **fails to resolve a host address."**
:Diagnosis: When running ``reprounzip vagrant setup``, if you get an error similar to this::
==> default: failed: Temporary failure in name resolution.
==> default: wget: unable to resolve host address ...
there is probably a firewall blocking the Vagrant VM to have Internet connection; the VM needs Internet connection to download required software for setting up the experiment for you.
:Solution: Make sure that your anti-virus/firewall is not causing this issue.
------------
.. _vagrant-memory:
:Issue: **"The experiment fails because of insufficient memory in Vagrant."**
:Diagnosis: It is possible that the default amount of memory allocated to the VM is insufficient for the experiment. You can see a lot of different messages there, including:
* ``Out of memory``
* ``Could not allocate memory``
* ``Killed``
:Solution: From VirtualBox, stop the machine and allocate more memory under `Settings > System > Motherboard > Memory`.
You can also use the ``--memory`` option when you run ``reprounzip vagrant setup`` to specify the amount of memory (in megabytes) at that time.
------------
.. _nosuchfile:
:Issue: **"** ``reprounzip run`` **fails with** ``no such file or directory`` **or similar."**
:Diagnosis: This error message may have different reasons, but it often means that a specific version of a library or a dynamic linker is missing:
1. If you are requesting `reprounzip` to install software using the package manager (by running ``reprounzip installpkgs``), it is possible that the software packages from the package manager are not compatible with the ones required by the experiment.
2. If, while packing, the user chose not to include some packages, `reprounzip` will try to install the ones from the package manager, which may not be compatible.
3. If you are using ``reprounzip vagrant`` or ``reprounzip docker``, ReproZip may be failing to detect the closest base system for unpacking the experiment.
:Solution:
1. Use the files inside the experiment bundle to ensure compatibility.
2. Contact the author of the ReproZip bundle to ask for a new package with all software packages included.
3. Try a different base system that you think it is closer to the original one by using the option ``--base-image`` when running these unpackers.
------------
:Issue: **"There are warnings from requests/urllib3 when running ReproZip."**
::
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:79:
InsecurePlatformWarning: A true SSLContext object is not available. This
prevents urllib3 from configuring SSL appropriately and may cause certain SSL
connections to fail. For more information, see
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
:Diagnosis: Most Python versions are insecure, because they do not validate SSL certificates, thus generating these warnings.
:Solution: If you are using Python 2.7.9 and later, you shouldn't be affected, but if you see ``InsecurePlatformWarning``, you can run ``pip install requests[security]``, which should bring in the missing components.
================================================
FILE: docs/unpacked-format.rst
================================================
.. _unpacked-format:
Structure of Unpacked Experiments
*********************************
While *reprounzip* is designed to allow users to reproduce an experiment without having to master the tool used to run it (e.g.: `Vagrant <https://www.vagrantup.com/>`__ and `Docker <https://www.docker.com/>`__), in some situations it might be useful to go behind the scenes and interact with the unpacked experiments directly.
This page describes in more details how the unpackers operate.
.. note:: Future versions of unpackers might work in a different way. No attempt is made to make unpacked experiments compatible across different versions of *reprounzip*. Bundles will always be compatible though.
.. _unpacked-common:
Common Files across Unpackers
=============================
The unpacked directory contains the original configuration file as ``config.yml``. In fact, the VisTrails integration relies on it.
A file named ``.reprounzip`` also marks the directory as an unpacked experiment. This is a Python pickle file containing a dictionary with various types of information:
* ``unpacker`` maps to the unpacker's name.
* ``input_files`` is used by the uploader/downloader machinery to keep the state of the input files inside the experiment, as they may be replaced by the user or overwritten by runs.
* Other information specific to the unpacker, as described next.
.. _unpacked-directory:
The `directory` Unpacker
========================
The experiment directory contains:
* The original configuration file ``config.yml``.
* The pickle file ``.reprounzip``.
* The tarball ``inputs.tar.gz``, which contains the original files that were identifies as input files. This tarball is used for file restoration using ``upload :<input-id>`` (see :ref:`unpacker-input-output`).
* A directory called ``root``, which contains all the bundled files in their original path, with symbolic links to absolute paths rewritten to prepend the path to ``root``.
::
unpacked-directory/
.reprounzip
config.yml
inputs.tar.gz
root/
...
When running the ``run`` command, the unpacker sets ``LD_LIBRARY_PATH`` and ``PATH`` to point inside ``root``, and optionally ``DISPLAY`` and ``XAUTHORITY`` to the host's ones.
.. _unpacked-chroot:
The `chroot` Unpacker
=====================
The experiment directory contains:
* The original configuration file ``config.yml``.
* The pickle file ``.reprounzip``, which stores whether magic directories are mounted, as explained below.
* The tarball ``inputs.tar.gz``, which contains the original files that were identifies as input files. This tarball is used for file restoration using ``upload :<input-id>`` (see :ref:`unpacker-input-output`).
* A directory called ``root``, which contains all the bundled files in their original path, with no symbolic links rewritten and file ownership restored.
::
unpacked-directory/
.reprounzip
config.yml
inputs.tar.gz
root/
dev/
dev/pts/
proc/
...
If a file is listed in the configuration file but wasn't packed (i.e.: ``pack_files`` was set to ``false`` for a software package), such file is copied from the host; if this file does not exist on the host, a warning is shown when unpacking.
Unless ``--dont-bind-magic-dirs`` is specified when unpacking, the special directories ``/dev``, ``/dev/pts``, and ``/proc`` are mounted with ``mount -o bind`` from the host.
Also, if ``/bin/sh`` or ``/usr/bin/env`` weren't both packed, a static build of `busybox <https://busybox.net/>`__ is downloaded and put under ``/bin/busybox``, and the missing binaries are created as symbolic links pointing to busybox.
Should you require a shell inside the experiment environment, you can use::
chroot root/ /bin/sh
.. _unpacked-vagrant:
The `vagrant` Unpacker
======================
The experiment directory contains:
* The original configuration file ``config.yml``.
* The pickle file ``.reprounzip``, which stores whether a chroot is used, as explained below.
* The tarball ``data.tgz``, which is part of the ``.rpz`` file and used to populate the virtual machine (VM) when it gets created.
* The setup script ``setup.sh``.
* The file ``rpz-files.list``, which contains the list of files to unpack. This list is passed to ``tar -T`` while unpacking.
* A ``Vagrantfile``, which is used to build the VM.
::
unpacked-directory/
.reprounzip
config.yml
data.tgz
busybox
Vagrantfile
setup.sh
rpz-files.list
Once ``vagrant up`` has been run by the ``setup/start`` command, a ``.vagrant`` subdirectory is created, and its content is managed by Vagrant (and appears to vary among different platforms).
Note that Vagrant drives VirtualBox or a similar virtualization software to run the VM. These will maintain state outside of the experiment directory. If you need to reconfigure or otherwise interact with the VM, you should do it from that virtualization software (e.g.: VirtualBox). The VM is named as the experiment directory with an additional suffix.
There are two modes for the virtual machine, controlled through command-line flags:
* The default mode, ``--use-chroot``, creates a chroot environment inside the VM at ``/experimentroot``. This allows ReproZip to unpack very different file system hierarchies without breaking the base system of the VM (in particular, ``ssh`` needs to keep working for the VM to be usable). In this mode, software packages that were not packed (i.e.: ``pack_files`` was set to ``false``) are installed in the VM and their required files are copied to the ``/experimentroot`` hierarchy. The software packages that were packed are simply copied over without any interaction with the VM's system.
* If ``--dont-use-chroot`` is used, no chroot environment is created. Files from software packages are never copied from the ``.rpz`` file; instead, they get installed from the package manager. Other files are simply unpacked in the VM system, possibly overwriting existing files. As long as *reprounzip-vagrant* manages to find a VM image with the same operating system as the original one, reproduction is expected to work reliably.
In the ``--use-chroot`` mode, a static build of `busybox <https://busybox.net/>`__ is downloaded and put under ``/experimentroot/busybox``, and if ``/bin/sh`` wasn't packed, it is created as a symbolic link pointing to busybox.
Uploading and downloading files from the environment is done via the shared directory ``/vagrant``, which is the experiment directory mounted in the VM by Vagrant.
Should you require a shell inside the experiment environment, you can use::
vagrant ssh
Please be aware of whether ``--use-chroot`` is in use when accessing the experiment environment: in this case, the experiment's files are located under ``/experimentroot``.
.. _unpacked-docker:
The `docker` Unpacker
=====================
The experiment directory contains:
* The original configuration file ``config.yml``.
* The pickle file ``.reprounzip``, which stores the name of the images built by the unpacker, as explained below.
* The tarball ``data.tgz``, which is part of the ``.rpz`` file and used to populate the Docker container.
* The file ``rpz-files.list``, which contains the list of files to unpack. This list is passed to ``tar -T`` while unpacking.
* A ``Dockerfile``, which is used to build the original image.
::
unpacked-directory/
.reprounzip
config.yml
data.tgz
busybox
rpzsudo
Dockerfile
rpz-files.list
Static builds of `busybox <https://busybox.net/>`__ and `rpzsudo <https://github.com/remram44/static-sudo/blob/master/rpzsudo.c>`__ are always downloaded and put into the Docker image as ``/busybox`` and ``/rpzsudo``, respectively.
Note that the ``docker`` command connects to a Docker daemon over a socket and that state will be changed there. The daemon might not be local; in particular, ``docker-machine`` might be used, which allows `reprounzip-docker` to be used on non-Linux machines, and the daemon might be in a virtual machine, on another host, or in the cloud. The `docker` unpacker will keep the environment variables set when calling Docker, notably ``DOCKER_HOST``, so these can be set accordingly before running the unpacker.
Images and containers built by the unpacker are given a random name with the prefixes ``reprounzip_image_`` and ``reprounzip_run_``, respectively; they are cleaned up when the ``destroy`` command is invoked. There are two images of which `reprounzip-docker` keeps track in the ``.reprounzip`` pickle file: the initial image, i.e., the one built by ``setup/build`` by calling ``docker build``, and the current image (initially the same as the initial image), which has been affected by a number of ``run`` and ``upload`` calls. Running the ``reset`` command returns to the initial image without having to rebuild. After each ``run`` invocation, the container is committed to a new current image so that state is kept.
A ``--detach`` option allows to start container and forget about them. reprounzip-docker leaves the container running and doesn't wait for it; this means that you can start a service on a remote machine, but note that because that container won't be committed to a new image, the side-effects of running it won't affect later executions on the same unpacked folder.
Uploading files to the environment is done by running a simple Dockerfile that builds a new image. Downloading files is done via the ``docker cp`` command.
================================================
FILE: docs/unpacking.rst
================================================
.. _unpacking:
Using *reprounzip*
******************
While *reprozip* is responsible for tracing and packing an experiment, *reprounzip* is the component used for the unpacking step. *reprounzip* is distributed with three **unpackers** for Linux (:ref:`reprounzip directory <unpack-directory>`, :ref:`reprounzip chroot <unpack-chroot>`, and :ref:`reprounzip installpkgs <unpack-installpkgs>`), but more unpackers are supported by installing additional plugins; some of these plugins are compatible with different environments as well (e.g.: :ref:`reprounzip-vagrant <unpack-vagrant>` and :ref:`reprounzip-docker <docker-plugin>`).
.. _unpack-info:
Inspecting a Bundle
===================
Showing Bundle Information
++++++++++++++++++++++++++
Before unpacking an experiment, it is often useful to have further information with respect to its bundle. The ``reprounzip info`` command allows users to do so::
$ reprounzip info <bundle>
where `<bundle>` corresponds to the experiment bundle (i.e., the ``.rpz`` file).
The output of this command has three sections. The first section, `Pack information`, contains general information about the experiment bundle, including size and total number of files::
----- Pack information -----
Compressed size: <compressed-size>
Unpacked size: <unpacked-size>
Total packed paths: <number>
The next section, `Metadata`, contains information about dependencies (i.e., software packages), machine architecture from the packing environment, and experiment runs::
----- Metadata -----
Total software packages: <total-number-software-packages>
Packed software packages: <number-packed-software-packages>
Architecture: <original-architecture> (current: <current-architecture>)
Distribution: <original-operating-system> (current: <current-operating-system>)
Runs:
<run-id>: <command-line>
<run-id>: <command-line>
...
Note that, for `Architecture` and `Distribution`, the command shows information with respect to both the original environment (i.e., the environment where the experiment was packed) and the current one (i.e., the environment where the experiment is to be unpacked). This helps users understand the differences between the environments in order to provide a better guidance in choosing the most appropriate unpacker.
If the verbose mode is used, more detailed information on the runs is provided::
$ reprounzip -v info <bundle>
...
----- Metadata -----
...
Runs:
<run-id>: <command-line>
wd: <working-directory>
exitcode: <exit-code>
<run-id>: <command-line>
wd: <working-directory>
exitcode: <exit-code>
...
Last, the section `Unpackers` shows which of the installed *reprounzip* unpackers can be successfully used in the current environment::
----- Unpackers -----
Compatible:
...
Incompatible:
...
Unknown:
...
`Compatible` lists the unpackers that can be used in the current environment, while `Incompatible` lists the unpackers that are not supported in the current environment. When using the verbose mode, an additional `Unknown` list shows the installed unpackers that may not work. As an example, for an experiment originally packed on Ubuntu and a user reproducing it on Windows, the `vagrant` unpacker (available through the :ref:`reprounzip-vagrant <unpack-vagrant>` plugin) is compatible, but :ref:`installpkgs <unpack-installpkgs>` is not; `vagrant` may also be listed under `Unknown` if ``vagrant`` is not in PATH (e.g.: if `Vagrant <https://www.vagrantup.com/>`__ is not installed).
.. _showfiles:
Showing Input and Output Files
++++++++++++++++++++++++++++++
The ``reprounzip showfiles`` command can be used to list the input and output files defined for the experiment. These files are identified by an id, which is either chosen by ReproZip or set in the configuration file before creating the ``.rpz`` file::
$ reprounzip showfiles bundle.rpz
Input files:
program_config
ipython_config
input_data
Output files:
rendered_image
logfile
Using the flag ``-v`` shows the complete path of each of these files in the experiment environment::
$ reprounzip -v showfiles bundle.rpz
Input files:
program_config (/home/user/.progrc)
ipython_config (/home/user/.ipython/profile_default/ipython_config.py)
input_data (/home/user/experiment/input.bin)
Output files:
rendered_image (/home/user/experiment/output.png)
logfile (/home/user/experiment/log.txt)
You can use the ``--input`` or ``--output`` flags to show only files that are inputs or outputs. If the bundle contains multiple runs, you can also filter files for a specific run::
$ reprounzip -v showfiles bundle.rpz preprocessing-step
Input files:
input_data (/home/user/experiment/input.bin)
Output files:
logfile (/home/user/experiment/log.txt)
where `preprocessing-step` is the run id. To see the dataflow of the experiment, please refer to :ref:`graph`.
The ``reprounzip showfiles`` command is particularly useful if you want to replace an input file with your own, or to get and save an output file for further examination. Please refer to :ref:`unpacker-input-output` for more information.
.. versionadded:: 1.0.4
The ``--input`` and ``--output`` flags.
.. _provenance-graph:
Creating a Provenance Graph
+++++++++++++++++++++++++++
ReproZip also allows users to generate a *provenance graph* related to the experiment execution by reading the metadata available in the ``.rpz`` bundle. This graph shows the experiment runs as well as the files and other dependencies they access during execution; this is particularly useful to visualize and understand the dataflow of the experiment.
See :ref:`graph` for details.
.. _unpack-unpackers:
Unpackers
=========
From the same ``.rpz`` bundle, `reprounzip` allows users to set up the experiment for reproduction in several ways by the use of different `unpackers`. Unpackers are plugins that have general interface and commands, but can also provide their own command-line syntax and options. Thanks to the decoupling between packing and unpacking steps, ``.rpz`` files from older versions of ReproZip can be used with new unpackers.
The `reprounzip` tool comes with three unpackers that are only compatible with Linux (``reprounzip directory``, ``reprounzip chroot``, and ``reprounzip installpkgs``). Additional unpackers, such as ``reprounzip vagrant`` and ``reprounzip docker``, can be installed separately. Next, each unpacker is described in more details; for more information on how to use an unpacker, please refer to :ref:`unpacker-commands`.
.. _unpack-directory:
The `directory` Unpacker: Unpacking as a Plain Directory
++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The *directory* unpacker (``reprounzip directory``) allows users to unpack the entire experiment (including library dependencies) in a single directory, and to reproduce the experiment directly from that directory. It does so by automatically setting up environment variables (e.g.: PATH, HOME, and LD_LIBRARY_PATH) that point the experiment execution to the created directory, which has the same structure as in the packing environment.
Please note that, although this unpacker is easy to use and does not require any privilege on the reproducing machine, it is **unreliable** since the directory is not isolated in any way from the remainder of the system. In particular, should the experiment use absolute paths, they will hit the host system instead. However, if the system has all the required packages (see :ref:`unpack-installpkgs`), and the experiment's files are addressed with relative paths, the use of this unpacker should not cause any problems.
.. warning:: ``reprounzip directory`` provides no isolation of the filesystem, as mentioned before. If the experiment uses absolute paths, either provided by you or hardcoded in the experiment, **they will point outside the unpacked directory**. Please be careful to use relative paths in the configuration and command line if you want this unpacker to work with your experiment. Other unpackers are more reliable in this regard.
.. note:: ``reprounzip directory`` is automatically distributed with `reprounzip`.
.. seealso:: :ref:`Why does 'reprounzip directory' fail with "IOError"? <directory_error>`
.. _unpack-chroot:
The `chroot` Unpacker: Providing Isolation with the *chroot* Mechanism
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
In the *chroot* unpacker (``reprounzip chroot``), similar to ``reprounzip directory``, a directory is created from the experiment bundle; however, a full system environment is also built, which can then be run with ``chroot(2)``, a Linux mechanism that changes the root directory ``/`` for the experiment to the experiment directory. Therefore, this unpacker addresses the limitation of the *directory* unpacker and does not fail in the presence of hardcoded absolute paths. Note as well that it **does not interfere with the current environment** since the experiment is isolated in that single directory.
.. warning:: Do **not** try to delete the experiment directory manually; **always** use ``reprounzip chroot destroy``. If ``/dev`` is mounted inside, you will also delete your system's device pseudo-files (these can be restored by rebooting or running the ``MAKEDEV`` script).
.. note:: Although *chroot* offers pretty good isolation, it is not considered completely safe: it is possible for processes owned by root to "escape" to the outer system. We recommend not running untrusted programs with this plugin.
.. note:: ``reprounzip chroot`` is automatically distributed with `reprounzip`.
.. _unpack-installpkgs:
The `installpkgs` Unpacker: Installing Software Packages
++++++++++++++++++++++++++++++++++++++++++++++++++++++++
By default, ReproZip identifies if the current environment already has the required software packages for the experiment, then using the installed ones for reproduction. For the non-installed software packages, it uses the dependencies packed in the original environment and extracted under the experiment directory.
Users may also let ReproZip try and install all the dependencies of the experiment on their machine by using the *installpkgs* unpacker (``reprounzip installpkgs``). This unpacker currently works for distribution based on Debian or RPM packages (e.g.: Ubuntu, CentOS, Fedora, ...), and uses the package manager to automatically install all the required software packages directly on the current machine, thus **interfering with your environment**.
To install the required dependencies, the following command should be used::
$ reprounzip installpkgs <bundle>
Users may use flag *y* or *assume-yes* to automatically confirm all the questions from the package manager; flag *missing* to install only the software packages that were not originally included in the experiment package (i.e.: software packages excluded in the configuration file); and flag *summary* to simply provide a summary of which software packages are installed or not in the current environment **without installing any dependency**.
.. warning:: Note that the package manager may not install the same software version as required for running the experiment, and if the versions are incompatible, the reproduction may fail.
.. note:: This unpacker is only used to install software packages. Users still need to use either ``reprounzip directory`` or ``reprounzip chroot`` to extract the experiment and execute it.
.. note:: ``reprounzip installpkgs`` is automatically distributed with `reprounzip`.
.. _unpackers:
.. _unpack-vagrant:
The `vagrant` Unpacker: Building a Virtual Machine
++++++++++++++++++++++++++++++++++++++++++++++++++
The *vagrant* unpacker (``reprounzip vagrant``) allows an experiment to be unpacked into a Virtual Machine and reproduced in that emulated environment, by automatically using `Vagrant <https://www.vagrantup.com/>`__. Therefore, the experiment can be reproduced in any environment supported by this tool, i.e., Linux, Mac OS X, and Windows. Note that the plugin assumes that Vagrant and VirtualBox are installed on your machine.
In addition to the commands listed in :ref:`unpacker-commands`, you can use ``suspend`` to save the virtual machine state to disk, and ``setup/start`` to restart a previously-created machine::
$ reprounzip vagrant suspend <path>
$ reprounzip vagrant setup/start <path>
The ``setup`` command also takes a ``--memory`` argument to explicitely select how many megabytes of RAM to allocate to the virtual machine.
.. note:: This unpacker is **not** distributed with `reprounzip`; it is a separate package that should be installed before use (see :ref:`install`).
.. versionadded:: 1.0.1
The ``--memory`` option.
.. versionadded:: 1.0.4
The ``suspend`` command.
.. _docker-plugin:
The `docker` Unpacker: Building a Docker Container
++++++++++++++++++++++++++++++++++++++++++++++++++
ReproZip can also extract and reproduce experiments as `Docker <https://www.docker.com/>`__ containers. The *docker* unpacker (``reprounzip docker``) is responsible for such integration and it assumes that Docker is already installed in the current environment.
You can pass arguments to the ``docker(1)`` program by using the ``--docker-option`` option to the ``setup`` or ``run`` commands.
Thanks to Docker's image layers feature, you can easily go back to the initial image after having run commands in the environment or replaced input files. To do that, use the ``reset`` command::
$ reprounzip docker reset <path>
.. note:: This unpacker is **not** distributed with `reprounzip`; it is a separate package that should be installed before use (see :ref:`install`).
.. _unpacker-commands:
Using an Unpacker
=================
Once you have chosen (and installed) an unpacker for your machine, you can use it to setup and run a packed experiment. An unpacker creates an **experiment directory** in which the working files are placed; these can be either the full filesystem (for *directory* or *chroot* unpackers) or other content (e.g.: a handle on a virtual machine for the *vagrant* unpacker); for the *chroot* unpacker, it might have mount points. To make sure that you free all resources and that you do not damage your environment, you should **always use the destroy command** to delete the experiment directory, not just merely delete it manually. See more information about this command below.
All the following commands need to state which unpacker is being used (i.e., ``reprounzip directory`` for the `directory` unpacker, ``reprounzip chroot`` for the `chroot` unpacker, ``reprounzip vagrant`` for the `vagrant` unpacker, and ``reprounzip docker`` for the `docker` unpacker). For the purpose of this documentation, we will use the `docker` unpacker; to use a different one, just replace ``docker`` in the following with the unpacker of your interest.
.. seealso:: :ref:`unpacked-format` provides further detailed information on unpackers.
Setting Up an Experiment Directory
++++++++++++++++++++++++++++++++++
.. note:: Some unpackers require an Internet connection during the ``setup`` command, to download some of the support software or the packages that were not packed. Make sure that you have an Internet connection, and that there is no firewall blocking the access.
To create the directory where the execution will take place, the ``setup`` command should be used::
$ reprounzip docker setup <bundle> <path>
where `<path>` is the directory where the experiment will be unpacked, i.e., the experiment directory.
Note that, once this is done, you should only remove `<path>` with the `destroy` command described below: deleting this directory manually might leave files behind, or even damage your system through bound filesystems.
The other unpacker commands take the `<path>` argument; they do not need the original bundle for the reproduction.
Reproducing the Experiment
++++++++++++++++++++++++++
After creating the directory, the experiment can be reproduced by issuing the ``run`` command::
$ reprounzip docker run <path>
which will execute the experiment inside the experiment directory. Users may also change the command line of the experiment by using ``--cmdline``::
$ reprounzip docker run <path> --cmdline <new-command-line>
where `<new-command-line>` is the modified command line. This is particularly useful to reproduce and test the experiment under different input parameter values. Using ``--cmdline`` without an argument only prints the original command line.
If the bundle contains multiple `runs` (separate commands that were packed together), all the runs are reproduced. You can also provide the id of the run or runs to be used::
$ reprounzip docker run <path> <run-id>
$ reprounzip docker run <path> <run-id> --cmdline <new-command-line>
For example::
$ reprounzip docker run unpacked-experiment 0-1,3 # First, second, and fourth runs
$ reprounzip docker run unpacked-experiment 2- # Third run and up
$ reprounzip docker run unpacked-experiment compile,test # Runs named 'compile' and 'test', in this order
If the experiment involves running a GUI tool, the graphical interface can be enable by using ``--enable-x11``::
$ reprounzip docker run <path> --enable-x11
which will forward the X connection from the experiment to the X server running on your machine. In this case, make sure you have a running X server.
If the experiment is a server, for example a website, a database management system, etc, you can expose ports from the experiment on your local machines. This is not required for the `directory` and `chroot` unpackers, since they offer no isolation of the network; for the `docker` and `vagrant` unpackers, use the ``--expose-port`` option::
$ reprounzip docker run --expose-port 8000:80 unpacked-experiment # Expose TCP port 80 (HTTP) of the experiment at http://localhost:8000/
$ reprounzip docker run --expose-port 3000 unpacked-experiment # Expose TCP port 3000 of the experiment at localhost:3000
$ reprounzip docker run --expose-port 5553:53/udp unpacked-experiment # Expose UDP port 53 of the experiment at localhost:5553
Note that in some situations, you might want to pass specific environment variables to the experiment, for example to set execution limits or parameters (such as OpenMPI information). To that effect, you can use the ``--pass-env VARNAME`` option to pass variables from the current machine, overriding the value from the original packing machine (`VARNAME` can be a regex). You can also set a variable to any value using ``--set-env VARNAME=value``. For example::
$ reprounzip docker run unpacked-experiment --pass-env 'OMPI_.*' --pass-env LANG --set-env DATA_SERVER_ADDRESS=localhost
.. versionadded:: 1.0.3
The ``--pass-env`` and ``-set-env`` options.
Removing the Experiment Directory
+++++++++++++++++++++++++++++++++
The ``destroy`` command will unmount mounted paths, destroy virtual machines, free container images, and delete the experiment directory::
$ reprounzip docker destroy <path>
Make sure you always use this command instead of simply deleting the directory manually.
.. _unpacker-input-output:
Managing Input and Output Files
+++++++++++++++++++++++++++++++
When tracing an experiment, ReproZip tries to identify which are the input and output files of the experiment. This can also be adjusted in the configuration file before packing.
If the unpacked experiment has such files, ReproZip provides some commands to manipulate them.
First, you can list these files using the ``showfiles`` command::
$ reprounzip showfiles <path>
Input files:
program_config
ipython_config
input_data
Output files:
rendered_image
logfile
To replace an input file with your own, `reprounzip`, you can use the ``upload`` command::
$ reprounzip docker upload <path> <input-path>:<input-id>
where `<input-path>` is the new file's path and `<input-id>` is the input file to be replaced (from ``showfiles``). This command overwrites the original path in the environment with the file you provided from your system. To restore the original input file, the same command, but in the following format, should be used::
$ reprounzip docker upload <path> :<input-id>
Running the ``showfiles`` command shows what the input files are currently set to::
$ reprounzip showfiles <path> --input
Input files:
program_config
(original)
ipython_config
C:\Users\Remi\Documents\ipython-config
In this example, the input `program_config` has not been changed (the one bundled in the ``.rpz`` file will be used), while the input `ipython_config` has been replaced.
After running the experiment, all the generated output files will be located under the experiment directory. To copy an output file from this directory to another desired location, use the ``download`` command::
$ reprounzip docker download <path> <output-id>:<output-path>
where `<output-id>` is the output file to be copied (from ``showfiles``) and `<output-path>` is the desired destination of the file. If an empty destination is specified, the file will be printed to stdout::
$ reprounzip docker download <path> <output-id>:
You can also omit the colon ``:`` altogether to download the file to the current directory under its original name::
$ reprounzip docker download <path> <output-id>
or even use ``--all`` to download every output file to the current directory under their original names.
Note that the ``upload`` command takes the file id on the right side of the colon (meaning that the path is the origin, and the id is the destination), while the ``download`` command takes it on the left side (meaning that the id is the origin, and the path is the destination). Both commands move data from left to right.
.. versionadded:: 1.0.4
Allow ``download <output-id>`` (no explicit destination), and add ``--all``.
.. seealso:: :ref:`Why can’t 'reprounzip' get my output files after reproducing an experiment? <moving-outputs>`
Running the Experiment in VisTrails
+++++++++++++++++++++++++++++++++++
In addition to reproducing the experiment, you may want to edit its dataflow by inserting your own processes between and around the experiment steps, or even by connecting multiple ReproZip'd experiments. However, manually managing the experiment workflow (with the help of ``reprounzip upload/download`` commands) can quickly become painful.
To allow use
gitextract_q6urh6ag/
├── .github/
│ └── workflows/
│ └── test.yml
├── .gitignore
├── .readthedocs.yaml
├── CHANGELOG.md
├── CITATION.cff
├── CITATION.txt
├── CONTRIBUTING.md
├── LICENSE.txt
├── README.md
├── Vagrantfile
├── allsetups.sh
├── docs/
│ ├── Makefile
│ ├── conf.py
│ ├── developerguide.rst
│ ├── faq.rst
│ ├── glossary.rst
│ ├── graph.rst
│ ├── gui.rst
│ ├── index.rst
│ ├── install.rst
│ ├── jupyter.rst
│ ├── make.bat
│ ├── man/
│ │ ├── reprounzip-docker.1
│ │ ├── reprounzip-vagrant.1
│ │ ├── reprounzip.1
│ │ └── reprozip.1
│ ├── packing.rst
│ ├── reprozip.rst
│ ├── traceschema.rst
│ ├── troubleshooting.rst
│ ├── unpacked-format.rst
│ ├── unpacking.rst
│ └── vistrails.rst
├── file-format.md
├── reprounzip/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprounzip/
│ │ ├── __init__.py
│ │ ├── common.py
│ │ ├── main.py
│ │ ├── orderedset.py
│ │ ├── pack_info.py
│ │ ├── parameters.py
│ │ ├── plugins/
│ │ │ └── __init__.py
│ │ ├── signals.py
│ │ ├── unpackers/
│ │ │ ├── __init__.py
│ │ │ ├── common/
│ │ │ │ ├── __init__.py
│ │ │ │ ├── misc.py
│ │ │ │ ├── packages.py
│ │ │ │ └── x11.py
│ │ │ ├── default.py
│ │ │ ├── graph.py
│ │ │ └── provviewer.py
│ │ └── utils.py
│ ├── setup.cfg
│ └── setup.py
├── reprounzip-docker/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprounzip/
│ │ ├── __init__.py
│ │ └── unpackers/
│ │ ├── __init__.py
│ │ └── docker.py
│ ├── setup.cfg
│ └── setup.py
├── reprounzip-qt/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprounzip_qt/
│ │ ├── __init__.py
│ │ ├── gui/
│ │ │ ├── __init__.py
│ │ │ ├── common.py
│ │ │ ├── run.py
│ │ │ └── unpack.py
│ │ ├── main.py
│ │ ├── qt_terminal.py
│ │ ├── reprounzip_interface.py
│ │ └── usage.py
│ ├── setup.cfg
│ └── setup.py
├── reprounzip-vagrant/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprounzip/
│ │ ├── __init__.py
│ │ └── unpackers/
│ │ ├── __init__.py
│ │ └── vagrant/
│ │ ├── __init__.py
│ │ ├── interaction.py
│ │ └── run_command.py
│ ├── setup.cfg
│ └── setup.py
├── reprounzip-vistrails/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprounzip/
│ │ ├── __init__.py
│ │ └── plugins/
│ │ ├── __init__.py
│ │ └── vistrails.py
│ ├── setup.cfg
│ └── setup.py
├── reprozip/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── native/
│ │ ├── config.h
│ │ ├── database.c
│ │ ├── database.h
│ │ ├── log.h
│ │ ├── ptrace_utils.c
│ │ ├── ptrace_utils.h
│ │ ├── pylog.c
│ │ ├── pytracer.c
│ │ ├── syscalls.c
│ │ ├── syscalls.h
│ │ ├── tracer.c
│ │ ├── tracer.h
│ │ ├── utils.c
│ │ └── utils.h
│ ├── reprozip/
│ │ ├── __init__.py
│ │ ├── common.py
│ │ ├── filters.py
│ │ ├── main.py
│ │ ├── pack.py
│ │ ├── tracer/
│ │ │ ├── __init__.py
│ │ │ ├── linux_pkgs.py
│ │ │ └── trace.py
│ │ ├── traceutils.py
│ │ └── utils.py
│ └── setup.py
├── reprozip-jupyter/
│ ├── LICENSE.txt
│ ├── MANIFEST.in
│ ├── README.rst
│ ├── reprozip_jupyter/
│ │ ├── __init__.py
│ │ ├── main.py
│ │ ├── notebook-extension.js
│ │ ├── run.py
│ │ ├── server_extension.py
│ │ └── trace.py
│ ├── setup.cfg
│ └── setup.py
├── scripts/
│ ├── RELEASE
│ ├── linux-wheels.sh
│ ├── macos/
│ │ ├── README.txt
│ │ ├── ReproUnzip.pkgproj
│ │ ├── app-shim/
│ │ │ ├── ReproUnzip_app/
│ │ │ │ └── Contents/
│ │ │ │ ├── Info.plist
│ │ │ │ └── Resources/
│ │ │ │ └── icon.icns
│ │ │ ├── reprounzip-qt.c
│ │ │ └── reprounzip-qt.debug.c
│ │ ├── instructions.txt
│ │ ├── reprounzip
│ │ ├── reprounzip-qt
│ │ └── reprozip-jupyter
│ ├── register-linux.sh
│ ├── test_bug_13676.py
│ ├── test_bug_23058.py
│ └── windows/
│ ├── input/
│ │ ├── reprounzip-qt.bat
│ │ ├── reprounzip.bat
│ │ └── reprozip-jupyter.bat
│ └── reprounzip.iss
├── syscalls.txt
└── tests/
├── __init__.py
├── __main__.py
├── check_images.py
├── common.py
├── connect.c
├── exec_echo.c
├── functional.py
├── readwrite.c
├── rename.c
├── segv.c
├── simple.c
├── simple_input.txt
├── simple_input2.txt
├── test_graph.py
├── test_parameters.py
├── test_rails_filter.py
├── test_reprounzip.py
├── test_reprozip.py
├── test_unpackers_common.py
├── test_utils.py
├── threads.c
├── threads2.c
└── vfork.c
SYMBOL INDEX (785 symbols across 72 files)
FILE: reprounzip-docker/reprounzip/unpackers/docker.py
function select_image (line 63) | def select_image(runs):
function write_dict (line 109) | def write_dict(path, dct):
function read_dict (line 113) | def read_dict(path):
function docker_setup (line 117) | def docker_setup(args):
function docker_setup_create (line 130) | def docker_setup_create(args):
function docker_setup_build (line 273) | def docker_setup_build(args):
function docker_reset (line 320) | def docker_reset(args):
function get_local_addr (line 350) | def get_local_addr():
function docker_run (line 406) | def docker_run(args):
class ContainerUploader (line 581) | class ContainerUploader(FileUploader):
method __init__ (line 582) | def __init__(self, target, input_files, files, unpacked_info,
method prepare_upload (line 598) | def prepare_upload(self, files):
method upload_file (line 607) | def upload_file(self, local_path, input_path):
method finalize (line 619) | def finalize(self):
function docker_upload (line 688) | def docker_upload(args):
class ContainerDownloader (line 703) | class ContainerDownloader(FileDownloader):
method __init__ (line 704) | def __init__(self, target, files, image, all_=False,
method prepare_download (line 710) | def prepare_download(self, files):
method download (line 719) | def download(self, remote_path, local_path):
method finalize (line 736) | def finalize(self):
function docker_download (line 745) | def docker_download(args):
function docker_destroy_docker (line 763) | def docker_destroy_docker(args):
function docker_destroy_dir (line 789) | def docker_destroy_dir(args):
function test_has_docker (line 801) | def test_has_docker(pack, **kwargs):
function setup (line 814) | def setup(parser, **kwargs):
FILE: reprounzip-qt/reprounzip_qt/gui/__init__.py
class Application (line 27) | class Application(QtWidgets.QApplication):
method __init__ (line 28) | def __init__(self, argv):
method linux_try_register_default (line 33) | def linux_try_register_default(self, window):
method linux_register_default (line 65) | def linux_register_default(self, window):
method ask_enable_usage_report (line 129) | def ask_enable_usage_report(self):
method event (line 150) | def event(self, event):
method set_first_window (line 165) | def set_first_window(self, window):
class ReprounzipUi (line 174) | class ReprounzipUi(QtWidgets.QMainWindow):
method __init__ (line 175) | def __init__(self, unpack={}, run={}, tab=None, **kwargs):
method _unpacked (line 188) | def _unpacked(self, directory, root):
method closeEvent (line 192) | def closeEvent(self, event):
method replaceable (line 199) | def replaceable(self):
FILE: reprounzip-qt/reprounzip_qt/gui/common.py
function error_msg (line 11) | def error_msg(parent, message, severity, details=None):
function handle_error (line 26) | def handle_error(parent, result):
class ResizableStack (line 34) | class ResizableStack(QtWidgets.QStackedWidget):
method __init__ (line 36) | def __init__(self, **kwargs):
method addWidget (line 41) | def addWidget(self, widget):
method _current_changed (line 46) | def _current_changed(self, idx):
class ROOT (line 54) | class ROOT(object):
function parse_ports (line 63) | def parse_ports(string, widget):
FILE: reprounzip-qt/reprounzip_qt/gui/run.py
class RunOptions (line 16) | class RunOptions(QtWidgets.QWidget):
method __init__ (line 19) | def __init__(self):
method add_row (line 23) | def add_row(self, label, widget):
method add_row_layout (line 29) | def add_row_layout(self, label, rowlayout):
method add_x11 (line 35) | def add_x11(self):
method options (line 39) | def options(self):
class DirectoryOptions (line 48) | class DirectoryOptions(RunOptions):
method __init__ (line 49) | def __init__(self):
class ChrootOptions (line 54) | class ChrootOptions(RunOptions):
method __init__ (line 55) | def __init__(self):
class DockerOptions (line 60) | class DockerOptions(RunOptions):
method __init__ (line 61) | def __init__(self):
method options (line 85) | def options(self):
class VagrantOptions (line 117) | class VagrantOptions(RunOptions):
method __init__ (line 118) | def __init__(self):
method options (line 127) | def options(self):
class FilesManager (line 141) | class FilesManager(QtWidgets.QDialog):
method __init__ (line 142) | def __init__(self, directory, unpacker=None, root=None, **kwargs):
method _file_changed (line 190) | def _file_changed(self):
method _upload (line 221) | def _upload(self):
method _download (line 234) | def _download(self):
method _reset (line 247) | def _reset(self):
class RunTab (line 257) | class RunTab(QtWidgets.QWidget):
method __init__ (line 270) | def __init__(self, unpacked_directory='', **kwargs):
method _browse (line 351) | def _browse(self):
method _directory_changed (line 360) | def _directory_changed(self, new_dir=None, force=False):
method _run (line 400) | def _run(self):
method _destroy (line 416) | def _destroy(self):
method _open_files_manager (line 422) | def _open_files_manager(self):
method set_directory (line 430) | def set_directory(self, directory, root=None):
method should_exit (line 435) | def should_exit(self):
method replaceable (line 450) | def replaceable(self):
FILE: reprounzip-qt/reprounzip_qt/gui/unpack.py
class UnpackerOptions (line 18) | class UnpackerOptions(QtWidgets.QWidget):
method __init__ (line 19) | def __init__(self):
method add_row (line 23) | def add_row(self, label, widget):
method options (line 29) | def options(self):
method available (line 33) | def available():
class DirectoryOptions (line 37) | class DirectoryOptions(UnpackerOptions):
method __init__ (line 38) | def __init__(self):
method available (line 45) | def available():
class ChrootOptions (line 49) | class ChrootOptions(UnpackerOptions):
method __init__ (line 50) | def __init__(self):
method options (line 66) | def options(self):
method available (line 88) | def available():
class DockerOptions (line 92) | class DockerOptions(UnpackerOptions):
method __init__ (line 93) | def __init__(self):
method options (line 135) | def options(self):
class VagrantOptions (line 164) | class VagrantOptions(UnpackerOptions):
method __init__ (line 165) | def __init__(self):
method options (line 197) | def options(self):
class UnpackTab (line 236) | class UnpackTab(QtWidgets.QWidget):
method __init__ (line 248) | def __init__(self, package='', **kwargs):
method replaceable (line 312) | def replaceable(self):
method _browse_pkg (line 315) | def _browse_pkg(self):
method _package_changed (line 324) | def _package_changed(self, new_pkg=None):
method _browse_dir (line 330) | def _browse_dir(self):
method _directory_changed (line 341) | def _directory_changed(self, new_dir=None):
method _unpack (line 344) | def _unpack(self):
FILE: reprounzip-qt/reprounzip_qt/main.py
function main (line 21) | def main():
FILE: reprounzip-qt/reprounzip_qt/qt_terminal.py
class Terminal (line 22) | class Terminal(QtWidgets.QWidget):
method __init__ (line 25) | def __init__(self, cmdline, env, input_enabled=False,
method _enter (line 92) | def _enter(self):
method _read_stdout (line 97) | def _read_stdout(self):
method _read_stderr (line 103) | def _read_stderr(self):
method _finished (line 109) | def _finished(self, code, status):
function run_in_builtin_terminal (line 125) | def run_in_builtin_terminal(cmd, env,
FILE: reprounzip-qt/reprounzip_qt/reprounzip_interface.py
function shell_escape (line 30) | def shell_escape(s):
function win_escape (line 50) | def win_escape(s):
function check_directory (line 69) | def check_directory(directory):
function is_jupyter (line 82) | def is_jupyter(directory):
class FileStatus (line 93) | class FileStatus(object):
method __init__ (line 94) | def __init__(self, name, path, is_input, is_output):
method __repr__ (line 101) | def __repr__(self):
class FilesStatus (line 105) | class FilesStatus(object):
method __init__ (line 106) | def __init__(self, directory):
method _refresh (line 117) | def _refresh(self):
method __getitem__ (line 124) | def __getitem__(self, item):
method __iter__ (line 128) | def __iter__(self):
function find_command (line 133) | def find_command(cmd):
function run (line 153) | def run(directory, unpacker=None, runs=None,
function docker_machine_env (line 199) | def docker_machine_env(machine):
function unpack (line 222) | def unpack(package, unpacker, directory, options=None):
function destroy (line 253) | def destroy(directory, unpacker=None, root=None):
function upload (line 273) | def upload(directory, name, path, unpacker=None, root=None):
function download (line 299) | def download(directory, name, path, unpacker=None, root=None):
function run_in_builtin_terminal_maybe (line 322) | def run_in_builtin_terminal_maybe(cmd, root, env={}, **kwargs):
function run_in_system_terminal (line 331) | def run_in_system_terminal(cmd, env={},
FILE: reprounzip-qt/reprounzip_qt/usage.py
function record_usage (line 27) | def record_usage(**kwargs):
function submit_usage_report (line 34) | def submit_usage_report(**kwargs):
FILE: reprounzip-vagrant/reprounzip/unpackers/vagrant/__init__.py
function _find_version (line 45) | def _find_version(distrib, version, architecture):
function _find_distribution (line 61) | def _find_distribution(parameter, distribution, version, architecture):
function select_box (line 76) | def select_box(runs, gui=False):
function write_dict (line 106) | def write_dict(path, dct):
function read_dict (line 110) | def read_dict(path):
function machine_setup (line 114) | def machine_setup(target):
function vagrant_setup_create (line 208) | def vagrant_setup_create(args):
function write_vagrantfile (line 416) | def write_vagrantfile(target, unpacked_info):
function vagrant_setup_start (line 454) | def vagrant_setup_start(args):
class LocalX11Handler (line 464) | class LocalX11Handler(BaseX11Handler):
method fix_env (line 469) | def fix_env(env):
function vagrant_run (line 479) | def vagrant_run(args):
class SSHUploader (line 599) | class SSHUploader(FileUploader):
method __init__ (line 600) | def __init__(self, target, input_files, files, use_chroot):
method prepare_upload (line 604) | def prepare_upload(self, files):
method upload_file (line 618) | def upload_file(self, local_path, input_path):
method finalize (line 656) | def finalize(self):
function vagrant_upload (line 661) | def vagrant_upload(args):
class SSHDownloader (line 676) | class SSHDownloader(FileDownloader):
method __init__ (line 677) | def __init__(self, target, files, use_chroot, all_=False):
method prepare_download (line 681) | def prepare_download(self, files):
method download (line 695) | def download(self, remote_path, local_path):
method finalize (line 731) | def finalize(self):
function vagrant_download (line 736) | def vagrant_download(args):
function vagrant_suspend (line 747) | def vagrant_suspend(args):
function vagrant_destroy_vm (line 759) | def vagrant_destroy_vm(args):
function vagrant_destroy_dir (line 772) | def vagrant_destroy_dir(args):
function _executable_in_path (line 783) | def _executable_in_path(executable):
function check_vagrant_version (line 794) | def check_vagrant_version():
function test_has_vagrant (line 811) | def test_has_vagrant(pack, **kwargs):
function setup (line 833) | def setup(parser, **kwargs):
FILE: reprounzip-vagrant/reprounzip/unpackers/vagrant/interaction.py
function interactive_shell (line 39) | def interactive_shell(chan, raw=True):
function posix_shell (line 46) | def posix_shell(chan, raw):
function windows_shell (line 86) | def windows_shell(chan):
FILE: reprounzip-vagrant/reprounzip/unpackers/vagrant/run_command.py
class IgnoreMissingKey (line 29) | class IgnoreMissingKey(MissingHostKeyPolicy):
method missing_host_key (line 35) | def missing_host_key(self, client, hostname, key):
function find_ssh_executable (line 39) | def find_ssh_executable(name='ssh'):
class SSHForwarder (line 58) | class SSHForwarder(BaseForwarder):
method __init__ (line 66) | def __init__(self, ssh_transport, remote_port, connector):
class _ChannelWrapper (line 71) | class _ChannelWrapper(object):
method __init__ (line 72) | def __init__(self, channel):
method sendall (line 75) | def sendall(self, data):
method recv (line 78) | def recv(self, data):
method close (line 81) | def close(self):
method _new_connection (line 84) | def _new_connection(self, channel, src_addr, dest_addr):
function run_interactive (line 93) | def run_interactive(ssh_info, interactive, cmd, request_pty, forwarded_p...
FILE: reprounzip-vistrails/reprounzip/plugins/vistrails.py
function escape_xml (line 42) | def escape_xml(s):
class IdScope (line 48) | class IdScope(object):
method __init__ (line 49) | def __init__(self):
method _add (line 61) | def _add(type_):
function split_sig (line 82) | def split_sig(sig):
class Workflow (line 87) | class Workflow(object):
method __init__ (line 88) | def __init__(self, file_, ids):
method close (line 102) | def close(self):
method add_module (line 107) | def add_module(self, sig, version, desc=None):
method add_function (line 137) | def add_function(self, mod_id, name, param_values):
method connect (line 157) | def connect(self, from_id, from_sig, from_port, to_id, to_sig, to_port):
method add_port_spec (line 183) | def add_port_spec(self, mod_id, name, type_, sigs, optional=True):
function do_vistrails (line 214) | def do_vistrails(target, pack=None, **kwargs):
function setup_vistrails (line 279) | def setup_vistrails():
function run_from_vistrails (line 285) | def run_from_vistrails():
FILE: reprounzip/reprounzip/common.py
class File (line 55) | class File(object):
method __init__ (line 60) | def __init__(self, path, size=None):
method __eq__ (line 64) | def __eq__(self, other):
method __ne__ (line 68) | def __ne__(self, other):
method __hash__ (line 71) | def __hash__(self):
class Package (line 75) | class Package(object):
method __init__ (line 78) | def __init__(self, name, version, files=None, packfiles=True, size=None):
method __eq__ (line 85) | def __eq__(self, other):
method __ne__ (line 90) | def __ne__(self, other):
method add_file (line 93) | def add_file(self, file_):
method __unicode__ (line 96) | def __unicode__(self):
class RPZPack (line 130) | class RPZPack(object):
method __init__ (line 135) | def __init__(self, pack):
method _open_tar (line 145) | def _open_tar(self):
method _open_zip (line 181) | def _open_zip(self):
method remove_data_prefix (line 225) | def remove_data_prefix(self, path):
method open_config (line 233) | def open_config(self):
method extract_config (line 241) | def extract_config(self, target):
method _extract_file (line 248) | def _extract_file(self, name, target):
method _extract_file_gz (line 260) | def _extract_file_gz(self, name, target):
method with_config (line 281) | def with_config(self):
method extract_trace (line 290) | def extract_trace(self, target):
method with_trace (line 312) | def with_trace(self):
method list_data (line 321) | def list_data(self):
method data_filenames (line 328) | def data_filenames(self):
method get_data (line 338) | def get_data(self, path):
method extract_data (line 347) | def extract_data(self, root, members):
method copy_data_tar (line 361) | def copy_data_tar(self, target):
method extensions (line 378) | def extensions(self):
method close (line 400) | def close(self):
class InvalidConfig (line 410) | class InvalidConfig(ValueError):
function read_files (line 415) | def read_files(files, File=File):
function read_packages (line 421) | def read_packages(packages, File=File, Package=Package):
class InputOutputFile (line 437) | class InputOutputFile(object):
method __init__ (line 438) | def __init__(self, path, read_runs, write_runs):
method __eq__ (line 443) | def __eq__(self, other):
method __lt__ (line 447) | def __lt__(self, other):
method __repr__ (line 450) | def __repr__(self):
function load_iofiles (line 455) | def load_iofiles(config, runs):
function load_config (line 522) | def load_config(filename, canonical, File=File, Package=Package):
function write_file (line 595) | def write_file(fp, fi, indent=0):
function write_package (line 602) | def write_package(fp, pkg, indent=0):
function save_config (line 623) | def save_config(filename, runs, packages, other_files, reprozip_version,
class LoggingDateFormatter (line 720) | class LoggingDateFormatter(logging.Formatter):
method formatTime (line 725) | def formatTime(self, record, datefmt=None):
function setup_logging (line 732) | def setup_logging(tag, verbosity):
function setup_usage_report (line 785) | def setup_usage_report(name, version):
function enable_usage_report (line 810) | def enable_usage_report(enable):
function record_usage (line 822) | def record_usage(**kwargs):
function record_usage_package (line 829) | def record_usage_package(runs, packages, other_files,
function submit_usage_report (line 852) | def submit_usage_report(**kwargs):
function get_reprozip_ca_certificate (line 861) | def get_reprozip_ca_certificate():
FILE: reprounzip/reprounzip/main.py
function get_plugins (line 44) | def get_plugins(entry_point_name):
class RPUZArgumentParser (line 64) | class RPUZArgumentParser(argparse.ArgumentParser):
method error (line 65) | def error(self, message):
function usage_report (line 71) | def usage_report(args):
function main (line 79) | def main():
FILE: reprounzip/reprounzip/orderedset.py
class OrderedSet (line 30) | class OrderedSet(MutableSet):
method __init__ (line 31) | def __init__(self, iterable=None):
method __len__ (line 38) | def __len__(self):
method __contains__ (line 41) | def __contains__(self, key):
method add (line 44) | def add(self, key):
method discard (line 50) | def discard(self, key):
method __iter__ (line 56) | def __iter__(self):
method __reversed__ (line 63) | def __reversed__(self):
method pop (line 70) | def pop(self, last=True):
method __repr__ (line 77) | def __repr__(self):
method __eq__ (line 82) | def __eq__(self, other):
method update (line 87) | def update(self, iterable):
FILE: reprounzip/reprounzip/pack_info.py
function get_package_info (line 25) | def get_package_info(pack, read_data=False):
function _print_package_info (line 139) | def _print_package_info(pack, info, verbosity=1):
function print_info (line 236) | def print_info(args):
function showfiles (line 249) | def showfiles(args):
function setup_info (line 372) | def setup_info(parser, **kwargs):
function setup_showfiles (line 381) | def setup_showfiles(parser, **kwargs):
FILE: reprounzip/reprounzip/parameters.py
function update_parameters (line 33) | def update_parameters():
function get_parameter (line 83) | def get_parameter(section):
FILE: reprounzip/reprounzip/signals.py
class SignalWarning (line 19) | class SignalWarning(UserWarning):
class Signal (line 28) | class Signal(object):
method __init__ (line 47) | def __init__(self, expected_args=[], new_args=[], old_args=[]):
method __call__ (line 57) | def __call__(self, **kwargs):
method subscribe (line 93) | def subscribe(self, func):
method unsubscribe (line 107) | def unsubscribe(self, func):
FILE: reprounzip/reprounzip/unpackers/common/misc.py
class UsageError (line 41) | class UsageError(Exception):
method __init__ (line 42) | def __init__(self, msg="Invalid command-line"):
function composite_action (line 46) | def composite_action(*functions):
function target_must_exist (line 59) | def target_must_exist(func):
function unique_names (line 72) | def unique_names():
function make_unique_name (line 87) | def make_unique_name(prefix):
function shell_escape (line 100) | def shell_escape(s):
function load_config (line 114) | def load_config(pack):
function busybox_url (line 126) | def busybox_url(arch):
function sudo_url (line 132) | def sudo_url(arch):
function rpztar_url (line 138) | def rpztar_url(arch):
class FileUploader (line 144) | class FileUploader(object):
method __init__ (line 149) | def __init__(self, target, input_files, files):
method run (line 154) | def run(self, files):
method get_config (line 227) | def get_config(self):
method prepare_upload (line 231) | def prepare_upload(self, files):
method extract_original_input (line 234) | def extract_original_input(self, input_name, input_path, temp):
method upload_file (line 247) | def upload_file(self, local_path, input_path):
method finalize (line 250) | def finalize(self):
class FileDownloader (line 254) | class FileDownloader(object):
method __init__ (line 257) | def __init__(self, target, files, all_=False):
method run (line 261) | def run(self, files, all_):
method get_config (line 325) | def get_config(self):
method prepare_download (line 329) | def prepare_download(self, files):
method download_and_print (line 332) | def download_and_print(self, remote_path):
method download (line 345) | def download(self, remote_path, local_path):
method finalize (line 348) | def finalize(self):
function get_runs (line 352) | def get_runs(runs, selected_runs, cmdline):
function add_environment_options (line 411) | def add_environment_options(parser):
function fixup_environment (line 422) | def fixup_environment(environ, args):
function pty_spawn (line 443) | def pty_spawn(*args, **kwargs):
function pty_spawn (line 448) | def pty_spawn(argv):
function interruptible_call (line 479) | def interruptible_call(cmd, **kwargs):
function metadata_read (line 515) | def metadata_read(path, type_):
function metadata_write (line 548) | def metadata_write(path, dct, type_):
function metadata_initial_iofiles (line 565) | def metadata_initial_iofiles(config, dct=None):
function metadata_update_run (line 601) | def metadata_update_run(config, dct, runs):
function parse_ports (line 627) | def parse_ports(specifications):
FILE: reprounzip/reprounzip/unpackers/common/packages.py
class CantFindInstaller (line 27) | class CantFindInstaller(UsageError):
method __init__ (line 28) | def __init__(self, msg="Can't select a package installer"):
class AptInstaller (line 32) | class AptInstaller(object):
method __init__ (line 35) | def __init__(self, binary):
method install (line 38) | def install(self, packages, assume_yes=False):
method get_packages_info (line 59) | def get_packages_info(packages):
method update_script (line 84) | def update_script(self):
method install_script (line 87) | def install_script(self, packages):
class YumInstaller (line 92) | class YumInstaller(object):
method install (line 96) | def install(cls, packages, assume_yes=False):
method get_packages_info (line 115) | def get_packages_info(packages):
method update_script (line 141) | def update_script():
method install_script (line 145) | def install_script(packages):
function select_installer (line 149) | def select_installer(pack, runs, target_distribution=THIS_DISTRIBUTION,
FILE: reprounzip/reprounzip/unpackers/common/x11.py
function ascii (line 44) | def ascii(s):
class Xauth (line 51) | class Xauth(object):
method __init__ (line 61) | def __init__(self, family, address, number, name, data):
method from_file (line 69) | def from_file(cls, fp):
method as_bytes (line 82) | def as_bytes(self):
class BaseX11Handler (line 95) | class BaseX11Handler(object):
class X11Handler (line 106) | class X11Handler(BaseX11Handler):
method __init__ (line 119) | def __init__(self, enabled, target, display=None):
method _locate_display (line 228) | def _locate_display(cls):
method port_forward (line 281) | def port_forward(self):
method fix_env (line 303) | def fix_env(self, env):
method init_cmds (line 317) | def init_cmds(self):
class BaseForwarder (line 343) | class BaseForwarder(object):
method __init__ (line 354) | def __init__(self, connector):
method _forward (line 357) | def _forward(self, client, src_addr):
class LocalForwarder (line 378) | class LocalForwarder(BaseForwarder):
method __init__ (line 386) | def __init__(self, connector, local_port=None):
method _accept (line 397) | def _accept(self, server):
FILE: reprounzip/reprounzip/unpackers/default.py
function get_elf_interpreter (line 51) | def get_elf_interpreter(file):
function installpkgs (line 62) | def installpkgs(args):
function directory_create (line 113) | def directory_create(args):
function directory_run (line 204) | def directory_run(args):
function directory_destroy (line 324) | def directory_destroy(args):
function should_restore_owner (line 336) | def should_restore_owner(param):
function should_mount_magic_dirs (line 368) | def should_mount_magic_dirs(param):
function chroot_create (line 398) | def chroot_create(args):
function chroot_mount (line 534) | def chroot_mount(args):
function chroot_run (line 561) | def chroot_run(args):
function chroot_unmount (line 625) | def chroot_unmount(target):
function chroot_destroy_unmount (line 654) | def chroot_destroy_unmount(args):
function chroot_destroy_dir (line 665) | def chroot_destroy_dir(args):
function chroot_destroy (line 682) | def chroot_destroy(args):
class LocalUploader (line 695) | class LocalUploader(FileUploader):
method __init__ (line 696) | def __init__(self, target, input_files, files, type_, param_restore_ow...
method prepare_upload (line 701) | def prepare_upload(self, files):
method extract_original_input (line 706) | def extract_original_input(self, input_name, input_path, temp):
method upload_file (line 715) | def upload_file(self, local_path, input_path):
function upload (line 728) | def upload(args):
class LocalDownloader (line 743) | class LocalDownloader(FileDownloader):
method __init__ (line 744) | def __init__(self, target, files, type_, all_=False):
method prepare_download (line 748) | def prepare_download(self, files):
method download_and_print (line 751) | def download_and_print(self, remote_path):
method download (line 763) | def download(self, remote_path, local_path):
function download (line 777) | def download(args):
function test_same_pkgmngr (line 787) | def test_same_pkgmngr(pack, config, **kwargs):
function test_linux_same_arch (line 802) | def test_linux_same_arch(pack, config, **kwargs):
function setup_installpkgs (line 819) | def setup_installpkgs(parser):
function setup_directory (line 837) | def setup_directory(parser, **kwargs):
function chroot_setup (line 908) | def chroot_setup(args):
function setup_chroot (line 917) | def setup_chroot(parser, **kwargs):
FILE: reprounzip/reprounzip/unpackers/graph.py
class Run (line 64) | class Run(object):
method __init__ (line 67) | def __init__(self, nb):
method dot (line 72) | def dot(self, fp, level_processes):
method dot_endpoint (line 85) | def dot_endpoint(self, level_processes):
method json (line 88) | def json(self, prog_map, level_processes):
class Process (line 112) | class Process(object):
method __init__ (line 117) | def __init__(self, pid, run, parent, timestamp, thread, acted, binary,
method dot (line 136) | def dot(self, fp, level_processes, indent=1):
method dot_endpoint (line 155) | def dot_endpoint(self, level_processes):
method json (line 165) | def json(self, process_map):
class Package (line 189) | class Package(object):
method __init__ (line 192) | def __init__(self, name, version=None):
method dot (line 198) | def dot(self, fp, level_pkgs):
method dot_endpoint (line 221) | def dot_endpoint(self, f, level_pkgs):
method json_endpoint (line 227) | def json_endpoint(self, f, level_pkgs):
method json (line 233) | def json(self, level_pkgs):
function parse_levels (line 245) | def parse_levels(level_pkgs, level_processes, level_other_files):
function read_events (line 288) | def read_events(database, all_forks, has_thread_flag):
function format_argv (line 437) | def format_argv(argv):
function generate (line 445) | def generate(target, configfile, database, all_forks=False, graph_format...
function graph_dot (line 594) | def graph_dot(target, runs, packages, other_files, package_map, edges,
function graph_json (line 667) | def graph_json(target, runs, packages, other_files, package_map, edges,
function graph (line 732) | def graph(args):
function disabled_bug13676 (line 755) | def disabled_bug13676(args):
function setup (line 763) | def setup(parser, **kwargs):
FILE: reprounzip/reprounzip/unpackers/provviewer.py
function xml_escape (line 30) | def xml_escape(s):
function generate (line 37) | def generate(target, configfile, database):
function provgraph (line 292) | def provgraph(args):
function disabled_bug13676 (line 311) | def disabled_bug13676(args):
function setup (line 319) | def setup(parser, **kwargs):
FILE: reprounzip/reprounzip/utils.py
class StreamWriter (line 38) | class StreamWriter(object):
method __init__ (line 39) | def __init__(self, stream):
method writelines (line 44) | def writelines(self, lines):
method write (line 47) | def write(self, obj):
method __getattr__ (line 53) | def __getattr__(self, name,
function flatten (line 96) | def flatten(n, iterable):
class UniqueNames (line 116) | class UniqueNames(object):
method __init__ (line 119) | def __init__(self):
method insert (line 122) | def insert(self, name):
method __call__ (line 126) | def __call__(self, name):
function escape (line 136) | def escape(s):
function optional_return_type (line 144) | def optional_return_type(req_args, other_args):
function tz_offset (line 206) | def tz_offset():
function isodatetime (line 211) | def isodatetime():
function hsize (line 231) | def hsize(nbytes):
function normalize_path (line 259) | def normalize_path(path):
function find_all_links_recursive (line 270) | def find_all_links_recursive(filename, files):
function find_all_links (line 294) | def find_all_links(filename, include_target=False):
function join_root (line 322) | def join_root(root, path):
function make_dir_writable (line 331) | def make_dir_writable(directory):
function rmtree_fixed (line 376) | def rmtree_fixed(path):
function copyfile (line 404) | def copyfile(source, destination, CHUNK_SIZE=4096):
function download_file (line 415) | def download_file(url, dest, cachename=None, ssl_verify=None):
FILE: reprozip-jupyter/reprozip_jupyter/__init__.py
function _jupyter_nbextension_paths (line 7) | def _jupyter_nbextension_paths():
function _jupyter_server_extension_paths (line 18) | def _jupyter_server_extension_paths():
FILE: reprozip-jupyter/reprozip_jupyter/main.py
function main (line 13) | def main():
FILE: reprozip-jupyter/reprozip_jupyter/notebook-extension.js
function fn_trace (line 9) | function fn_trace(env) {
function _on_load (line 84) | function _on_load() {
FILE: reprozip-jupyter/reprozip_jupyter/run.py
function process_connection_file (line 29) | def process_connection_file(original):
class RPZKernelManager (line 47) | class RPZKernelManager(IOLoopKernelManager):
method _launch_kernel (line 51) | def _launch_kernel(self, kernel_cmd, **kw):
class RPZMappingKernelManager (line 83) | class RPZMappingKernelManager(MappingKernelManager):
method __init__ (line 84) | def __init__(self, **kwargs):
function run_server (line 90) | def run_server(target, jupyter_args=None, verbosity=1):
function cmd_run_server (line 102) | def cmd_run_server(args):
function setup (line 111) | def setup(parser):
FILE: reprozip-jupyter/reprozip_jupyter/server_extension.py
class TraceHandler (line 21) | class TraceHandler(RequestHandler):
method initialize (line 22) | def initialize(self, nbapp=None):
method post (line 27) | def post(self):
method _trace_done (line 53) | def _trace_done(self, returncode):
method _packing_done (line 79) | def _packing_done(self, returncode):
function load_jupyter_server_extension (line 92) | def load_jupyter_server_extension(nbapp):
FILE: reprozip-jupyter/reprozip_jupyter/trace.py
class RPZOptions (line 28) | class RPZOptions(object):
method __init__ (line 29) | def __init__(self, verbosity=1, dir=None, identify_packages=True,
method trace_command_line (line 37) | def trace_command_line(self, kernel_cmd):
method config_file (line 54) | def config_file(self):
class RPZKernelManager (line 61) | class RPZKernelManager(KernelManager):
method _launch_kernel (line 64) | def _launch_kernel(self, kernel_cmd, **kw):
method finish_shutdown (line 72) | def finish_shutdown(self, *args, **kwargs):
class RPZExecutePreprocessor (line 92) | class RPZExecutePreprocessor(ExecutePreprocessor):
method __init__ (line 93) | def __init__(self, options):
method preprocess (line 97) | def preprocess(self, nb, resources):
function trace_notebook (line 145) | def trace_notebook(filename, save_notebook=True, **kwargs):
function cmd_trace_notebook (line 158) | def cmd_trace_notebook(args):
function setup (line 171) | def setup(parser):
FILE: reprozip/native/database.c
function sqlite3_uint64 (line 16) | static sqlite3_uint64 gettime(void)
function db_init (line 43) | int db_init(const char *filename)
function db_close (line 196) | int db_close(int rollback)
function db_add_process (line 225) | int db_add_process(unsigned int *id, unsigned int parent_id,
function db_add_first_process (line 262) | int db_add_first_process(unsigned int *id, const char *working_dir)
function db_add_exit (line 267) | int db_add_exit(unsigned int id, int exitcode)
function db_add_file_open (line 285) | int db_add_file_open(unsigned int process, const char *name,
function db_add_exec (line 337) | int db_add_exec(unsigned int process, const char *binary,
FILE: reprozip/native/ptrace_utils.c
function tracee_getword (line 15) | static long tracee_getword(pid_t tid, const void *addr)
function tracee_getlong (line 50) | uint64_t tracee_getlong(int mode, pid_t tid, const void *addr)
function tracee_getu64 (line 68) | uint64_t tracee_getu64(pid_t tid, const void *addr)
function tracee_getwordsize (line 76) | size_t tracee_getwordsize(int mode)
function tracee_strlen (line 86) | size_t tracee_strlen(pid_t tid, const char *str)
function tracee_read (line 109) | void tracee_read(pid_t tid, char *dst, const char *src, size_t size)
function free_strarray (line 169) | void free_strarray(char **array)
FILE: reprozip/native/pylog.c
function log_setup (line 18) | int log_setup()
function log_real_ (line 72) | void log_real_(pid_t tid, int lvl, const char *format, ...)
FILE: reprozip/native/pytracer.c
function PyObject (line 61) | static PyObject *pytracer_execute(PyObject *self, PyObject *args)
type PyModuleDef (line 159) | struct PyModuleDef
function PyMODINIT_FUNC (line 173) | PyMODINIT_FUNC PyInit__pytracer(void)
FILE: reprozip/native/syscalls.c
type syscall_table_entry (line 44) | struct syscall_table_entry {
type syscall_table (line 51) | struct syscall_table {
type syscall_table (line 56) | struct syscall_table
type Process (line 59) | struct Process
function syscall_unhandled_path1 (line 76) | static int syscall_unhandled_path1(const char *name, struct Process *pro...
function syscall_unhandled_other (line 90) | static int syscall_unhandled_other(const char *name, struct Process *pro...
function syscall_fileopening_in (line 107) | static int syscall_fileopening_in(const char *name, struct Process *proc...
function syscall_fileopening_out (line 142) | static int syscall_fileopening_out(const char *name, struct Process *pro...
function syscall_openat2_in (line 203) | static int syscall_openat2_in(const char *name, struct Process *process,
function syscall_openat2_out (line 237) | static int syscall_openat2_out(const char *name, struct Process *process,
function syscall_filecreating (line 287) | static int syscall_filecreating(const char *name, struct Process *process,
function syscall_filecreating_at (line 315) | static int syscall_filecreating_at(const char *name, struct Process *pro...
function syscall_filestat (line 354) | static int syscall_filestat(const char *name, struct Process *process,
function syscall_readlink (line 375) | static int syscall_readlink(const char *name, struct Process *process,
function syscall_mkdir (line 396) | static int syscall_mkdir(const char *name, struct Process *process,
function syscall_chdir (line 418) | static int syscall_chdir(const char *name, struct Process *process,
function record_shebangs (line 445) | static int record_shebangs(struct Process *process, const char *exec_tar...
function syscall_execve_in (line 592) | static int syscall_execve_in(const char *name, struct Process *process,
function syscall_execve_event (line 628) | int syscall_execve_event(struct Process *process)
function syscall_execve_out (line 702) | static int syscall_execve_out(const char *name, struct Process *process,
function syscall_fork_in (line 719) | static int syscall_fork_in(const char *name, struct Process *process,
function syscall_fork_out (line 726) | static int syscall_fork_out(const char *name, struct Process *process,
function syscall_fork_event (line 733) | int syscall_fork_event(struct Process *process, unsigned int event)
function handle_socket (line 840) | static int handle_socket(struct Process *process, const char *msg,
function handle_accept (line 880) | static int handle_accept(struct Process *process,
function handle_connect (line 897) | static int handle_connect(struct Process *process,
function syscall_socketcall (line 912) | static int syscall_socketcall(const char *name, struct Process *process,
function syscall_accept (line 938) | static int syscall_accept(const char *name, struct Process *process,
function syscall_connect (line 948) | static int syscall_connect(const char *name, struct Process *process,
function syscall_xxx_at (line 962) | static int syscall_xxx_at(const char *name, struct Process *process,
type unprocessed_table_entry (line 1026) | struct unprocessed_table_entry {
type syscall_table (line 1035) | struct syscall_table
type syscall_table (line 1035) | struct syscall_table
type unprocessed_table_entry (line 1036) | struct unprocessed_table_entry
type unprocessed_table_entry (line 1039) | struct unprocessed_table_entry
type syscall_table_entry (line 1052) | struct syscall_table_entry
function syscall_build_table (line 1078) | void syscall_build_table(void)
function syscall_handle (line 1371) | int syscall_handle(struct Process *process)
FILE: reprozip/native/syscalls.h
type Process (line 8) | struct Process
type Process (line 10) | struct Process
type Process (line 11) | struct Process
FILE: reprozip/native/tracer.c
type i386_regs (line 30) | struct i386_regs {
type x86_64_regs (line 51) | struct x86_64_regs {
function get_i386_reg (line 82) | static void get_i386_reg(register_type *reg, uint32_t value)
function get_x86_64_reg (line 89) | static void get_x86_64_reg(register_type *reg, uint64_t value)
function free_execve_info (line 97) | void free_execve_info(struct ExecveInfo *execi)
type Process (line 106) | struct Process
type Process (line 109) | struct Process
type Process (line 120) | struct Process
type Process (line 144) | struct Process
type ThreadGroup (line 160) | struct ThreadGroup
type ThreadGroup (line 162) | struct ThreadGroup
type ThreadGroup (line 162) | struct ThreadGroup
function trace_free_process (line 170) | void trace_free_process(struct Process *process)
function trace_count_processes (line 198) | void trace_count_processes(unsigned int *p_nproc, unsigned int *p_unknown)
function trace_add_files_from_proc (line 227) | int trace_add_files_from_proc(unsigned int process, pid_t tid,
function trace_set_options (line 333) | static void trace_set_options(pid_t tid)
function trace (line 347) | static int trace(pid_t first_proc, int *first_exit_code)
function restore_signals (line 589) | static void restore_signals(void)
function cleanup (line 603) | static void cleanup(void)
function sigint_handler (line 627) | static void sigint_handler(int signo)
function trace_init (line 644) | static void trace_init(void)
function fork_and_trace (line 669) | int fork_and_trace(const char *binary, int argc, char **argv,
FILE: reprozip/native/tracer.h
type register_type (line 13) | typedef struct S_register_type {
type ExecveInfo (line 22) | struct ExecveInfo {
type ExecveInfo (line 28) | struct ExecveInfo
type ThreadGroup (line 30) | struct ThreadGroup {
type Process (line 36) | struct Process {
type Process (line 67) | struct Process
type Process (line 71) | struct Process
type Process (line 73) | struct Process
type ThreadGroup (line 75) | struct ThreadGroup
type Process (line 77) | struct Process
FILE: reprozip/native/utils.c
function flags2mode (line 16) | unsigned int flags2mode(int flags)
function path_is_dir (line 141) | int path_is_dir(const char *pathname)
FILE: reprozip/reprozip/common.py
class File (line 55) | class File(object):
method __init__ (line 60) | def __init__(self, path, size=None):
method __eq__ (line 64) | def __eq__(self, other):
method __ne__ (line 68) | def __ne__(self, other):
method __hash__ (line 71) | def __hash__(self):
class Package (line 75) | class Package(object):
method __init__ (line 78) | def __init__(self, name, version, files=None, packfiles=True, size=None):
method __eq__ (line 85) | def __eq__(self, other):
method __ne__ (line 90) | def __ne__(self, other):
method add_file (line 93) | def add_file(self, file_):
method __unicode__ (line 96) | def __unicode__(self):
class RPZPack (line 130) | class RPZPack(object):
method __init__ (line 135) | def __init__(self, pack):
method _open_tar (line 145) | def _open_tar(self):
method _open_zip (line 181) | def _open_zip(self):
method remove_data_prefix (line 225) | def remove_data_prefix(self, path):
method open_config (line 233) | def open_config(self):
method extract_config (line 241) | def extract_config(self, target):
method _extract_file (line 248) | def _extract_file(self, name, target):
method _extract_file_gz (line 260) | def _extract_file_gz(self, name, target):
method with_config (line 281) | def with_config(self):
method extract_trace (line 290) | def extract_trace(self, target):
method with_trace (line 312) | def with_trace(self):
method list_data (line 321) | def list_data(self):
method data_filenames (line 328) | def data_filenames(self):
method get_data (line 338) | def get_data(self, path):
method extract_data (line 347) | def extract_data(self, root, members):
method copy_data_tar (line 361) | def copy_data_tar(self, target):
method extensions (line 378) | def extensions(self):
method close (line 400) | def close(self):
class InvalidConfig (line 410) | class InvalidConfig(ValueError):
function read_files (line 415) | def read_files(files, File=File):
function read_packages (line 421) | def read_packages(packages, File=File, Package=Package):
class InputOutputFile (line 437) | class InputOutputFile(object):
method __init__ (line 438) | def __init__(self, path, read_runs, write_runs):
method __eq__ (line 443) | def __eq__(self, other):
method __lt__ (line 447) | def __lt__(self, other):
method __repr__ (line 450) | def __repr__(self):
function load_iofiles (line 455) | def load_iofiles(config, runs):
function load_config (line 522) | def load_config(filename, canonical, File=File, Package=Package):
function write_file (line 595) | def write_file(fp, fi, indent=0):
function write_package (line 602) | def write_package(fp, pkg, indent=0):
function save_config (line 623) | def save_config(filename, runs, packages, other_files, reprozip_version,
class LoggingDateFormatter (line 720) | class LoggingDateFormatter(logging.Formatter):
method formatTime (line 725) | def formatTime(self, record, datefmt=None):
function setup_logging (line 732) | def setup_logging(tag, verbosity):
function setup_usage_report (line 785) | def setup_usage_report(name, version):
function enable_usage_report (line 810) | def enable_usage_report(enable):
function record_usage (line 822) | def record_usage(**kwargs):
function record_usage_package (line 829) | def record_usage_package(runs, packages, other_files,
function submit_usage_report (line 852) | def submit_usage_report(**kwargs):
function get_reprozip_ca_certificate (line 861) | def get_reprozip_ca_certificate():
FILE: reprozip/reprozip/filters.py
function builtin (line 21) | def builtin(input_files, **kwargs):
function python (line 40) | def python(files, input_files, **kwargs):
function ruby (line 77) | def ruby(files, input_files, **kwargs):
FILE: reprozip/reprozip/main.py
function shell_escape (line 48) | def shell_escape(s):
function print_db (line 62) | def print_db(database):
function testrun (line 154) | def testrun(args):
function trace (line 187) | def trace(args):
function reset (line 217) | def reset(args):
function pack (line 229) | def pack(args):
function combine (line 241) | def combine(args):
function usage_report (line 266) | def usage_report(args):
function main (line 274) | def main():
FILE: reprozip/reprozip/pack.py
function expand_patterns (line 34) | def expand_patterns(patterns):
function canonicalize_config (line 62) | def canonicalize_config(packages, other_files, additional_patterns,
function data_path (line 80) | def data_path(filename, prefix=Path('DATA')):
class PackBuilder (line 96) | class PackBuilder(object):
method __init__ (line 99) | def __init__(self, filename):
method add_data (line 103) | def add_data(self, filename):
method close (line 115) | def close(self):
function pack (line 120) | def pack(target, directory, sort_packages):
FILE: reprozip/reprozip/tracer/linux_pkgs.py
class PkgManager (line 36) | class PkgManager(object):
method __init__ (line 42) | def __init__(self):
method filter_files (line 48) | def filter_files(self, files):
method search_for_files (line 56) | def search_for_files(self, files):
method _filter (line 91) | def _filter(self, f):
method _get_packages_for_file (line 104) | def _get_packages_for_file(self, filename):
method _create_package (line 107) | def _create_package(self, pkgname):
class DpkgManager (line 115) | class DpkgManager(PkgManager):
method search_for_files (line 118) | def search_for_files(self, files):
method _get_packages_for_file (line 178) | def _get_packages_for_file(self, filename):
method _create_package (line 183) | def _create_package(self, pkgname):
class RpmManager (line 213) | class RpmManager(PkgManager):
method _get_packages_for_file (line 216) | def _get_packages_for_file(self, filename):
method _create_package (line 228) | def _create_package(self, pkgname):
function identify_packages (line 244) | def identify_packages(files):
FILE: reprozip/reprozip/tracer/trace.py
function stderr_in_red (line 44) | def stderr_in_red():
class TracedFile (line 55) | class TracedFile(File):
method __init__ (line 79) | def __init__(self, path):
method read (line 94) | def read(self, run):
method write (line 102) | def write(self, run):
function run_filter_plugins (line 115) | def run_filter_plugins(files, input_files):
function get_files (line 124) | def get_files(conn):
function tty_prompt (line 286) | def tty_prompt(prompt, chars):
function trace (line 336) | def trace(binary, argv, directory, append, verbosity='unset'):
function write_configuration (line 404) | def write_configuration(directory, sort_packages, find_inputs_outputs,
function compile_inputs_outputs (line 511) | def compile_inputs_outputs(runs, inputs, outputs):
FILE: reprozip/reprozip/traceutils.py
function create_schema (line 25) | def create_schema(conn):
function combine_files (line 76) | def combine_files(newfiles, newpackages, oldfiles, oldpackages):
function combine_traces (line 100) | def combine_traces(traces, target):
FILE: reprozip/reprozip/utils.py
class StreamWriter (line 38) | class StreamWriter(object):
method __init__ (line 39) | def __init__(self, stream):
method writelines (line 44) | def writelines(self, lines):
method write (line 47) | def write(self, obj):
method __getattr__ (line 53) | def __getattr__(self, name,
function flatten (line 96) | def flatten(n, iterable):
class UniqueNames (line 116) | class UniqueNames(object):
method __init__ (line 119) | def __init__(self):
method insert (line 122) | def insert(self, name):
method __call__ (line 126) | def __call__(self, name):
function escape (line 136) | def escape(s):
function optional_return_type (line 144) | def optional_return_type(req_args, other_args):
function tz_offset (line 206) | def tz_offset():
function isodatetime (line 211) | def isodatetime():
function hsize (line 231) | def hsize(nbytes):
function normalize_path (line 259) | def normalize_path(path):
function find_all_links_recursive (line 270) | def find_all_links_recursive(filename, files):
function find_all_links (line 294) | def find_all_links(filename, include_target=False):
function join_root (line 322) | def join_root(root, path):
function make_dir_writable (line 331) | def make_dir_writable(directory):
function rmtree_fixed (line 376) | def rmtree_fixed(path):
function copyfile (line 404) | def copyfile(source, destination, CHUNK_SIZE=4096):
function download_file (line 415) | def download_file(url, dest, cachename=None, ssl_verify=None):
FILE: scripts/macos/app-shim/reprounzip-qt.c
function main (line 5) | int main(int argc, char **argv) {
FILE: scripts/macos/app-shim/reprounzip-qt.debug.c
function main (line 8) | int main(int argc, char **argv) {
FILE: scripts/test_bug_23058.py
class Test23058 (line 7) | class Test23058(unittest.TestCase):
method do_test_verbosity (line 8) | def do_test_verbosity(self, parser, line, expected_verbosity):
method test_parents (line 15) | def test_parents(self):
method test_function (line 34) | def test_function(self):
FILE: tests/__main__.py
class Program (line 36) | class Program(unittest.TestProgram):
method createTests (line 37) | def createTests(self):
FILE: tests/check_images.py
function _vagrant_req (line 23) | def _vagrant_req(method, url, json=False):
function check_vagrant (line 41) | def check_vagrant():
function list_docker_tags (line 129) | def list_docker_tags(repository, token=None):
function check_docker (line 164) | def check_docker():
FILE: tests/common.py
function make_database (line 14) | def make_database(insert, path=None):
FILE: tests/connect.c
function main (line 22) | int main(void)
FILE: tests/exec_echo.c
function main (line 12) | int main(int argc, char **argv, char **envp)
FILE: tests/functional.py
function download_file_retry (line 30) | def download_file_retry(url, dest):
function in_temp_dir (line 56) | def in_temp_dir(f):
function print_arg_list (line 68) | def print_arg_list(f):
function call (line 82) | def call(args):
function check_call (line 89) | def check_call(args):
function call_output (line 94) | def call_output(args, stream='out'):
function check_output (line 117) | def check_output(args, stream='out'):
function build (line 124) | def build(target, sources, args=[]):
function functional_tests (line 132) | def functional_tests(raise_warnings, interactive, run_vagrant, run_docker):
FILE: tests/readwrite.c
function main (line 11) | int main(int argc, char **argv)
FILE: tests/rename.c
function main (line 13) | int main(int argc, char **argv)
FILE: tests/segv.c
function main (line 12) | int main(int argc, char **argv)
FILE: tests/simple.c
function main (line 12) | int main(int argc, char **argv)
FILE: tests/test_graph.py
class TestGraph (line 19) | class TestGraph(unittest.TestCase):
method setUpClass (line 24) | def setUpClass(cls):
method tearDownClass (line 132) | def tearDownClass(cls):
method do_dot_test (line 135) | def do_dot_test(self, expected, **kwargs):
method do_json_test (line 154) | def do_json_test(self, expected, **kwargs):
method do_tests (line 174) | def do_tests(self, expected_dot, expected_json, **kwargs):
method test_simple (line 178) | def test_simple(self):
method test_collapsed_packages (line 356) | def test_collapsed_packages(self):
FILE: tests/test_parameters.py
class TestSelection (line 14) | class TestSelection(unittest.TestCase):
method setUpClass (line 16) | def setUpClass(cls):
method tearDownClass (line 25) | def tearDownClass(cls):
method test_docker (line 28) | def test_docker(self):
method test_vagrant (line 47) | def test_vagrant(self):
FILE: tests/test_rails_filter.py
class MockTracedFile (line 13) | class MockTracedFile(File):
method __init__ (line 14) | def __init__(self, path):
class RailsFilterTest (line 18) | class RailsFilterTest(unittest.TestCase):
method setUp (line 19) | def setUp(self):
method tearDown (line 22) | def tearDown(self):
method touch (line 26) | def touch(cls, test_files):
method test_consuming_entire_gem (line 33) | def test_consuming_entire_gem(self):
method test_consuming_rails_files (line 62) | def test_consuming_rails_files(self):
FILE: tests/test_reprounzip.py
class TestSignals (line 21) | class TestSignals(unittest.TestCase):
method test_make_signal (line 22) | def test_make_signal(self):
method test_subscription (line 35) | def test_subscription(self):
method test_calls (line 51) | def test_calls(self):
class TestArgs (line 77) | class TestArgs(unittest.TestCase):
method test_argparse (line 78) | def test_argparse(self):
class TarBuilder (line 120) | class TarBuilder(object):
method __init__ (line 121) | def __init__(self, filename, mode):
method write_data (line 124) | def write_data(self, path, data, mode=None):
method add_file (line 131) | def add_file(self, name, arcname):
method close (line 134) | def close(self):
class ZipBuilder (line 139) | class ZipBuilder(object):
method __init__ (line 140) | def __init__(self, filename):
method write_data (line 143) | def write_data(self, path, data, mode=None):
method add_file (line 146) | def add_file(self, name, arcname):
method close (line 149) | def close(self):
class TestCommon (line 154) | class TestCommon(unittest.TestCase):
method test_rpzpack_v1 (line 155) | def test_rpzpack_v1(self):
method test_rpzpack_v2_tar (line 205) | def test_rpzpack_v2_tar(self):
method test_rpzpack_v2_zip (line 262) | def test_rpzpack_v2_zip(self):
FILE: tests/test_reprozip.py
class TestReprozip (line 22) | class TestReprozip(unittest.TestCase):
method test_make_dir_writable (line 24) | def test_make_dir_writable(self):
method test_make_dir_writable2 (line 45) | def test_make_dir_writable2(self):
method test_argparse (line 72) | def test_argparse(self):
class TestNames (line 110) | class TestNames(unittest.TestCase):
method test_uniquenames (line 111) | def test_uniquenames(self):
method test_label_files (line 121) | def test_label_files(self):
class TestFiles (line 134) | class TestFiles(unittest.TestCase):
method do_test (line 135) | def do_test(self, insert):
method make_paths (line 147) | def make_paths(cls, obj):
method assertEqualPaths (line 159) | def assertEqualPaths(self, objs, second):
method test_get_files (line 162) | def test_get_files(self):
method test_multiple_runs (line 185) | def test_multiple_runs(self):
class TestCombine (line 225) | class TestCombine(unittest.TestCase):
method setUp (line 226) | def setUp(self):
method tearDown (line 229) | def tearDown(self):
method test_combine (line 232) | def test_combine(self):
FILE: tests/test_unpackers_common.py
class TestCommon (line 16) | class TestCommon(unittest.TestCase):
method test_unique_names (line 17) | def test_unique_names(self):
method test_make_unique_name (line 24) | def test_make_unique_name(self):
method test_env (line 32) | def test_env(self):
class TestMisc (line 101) | class TestMisc(unittest.TestCase):
method do_ok (line 102) | def do_ok(self, arg, expected, nruns=4):
method do_fail (line 113) | def do_fail(self, arg):
method test_get_runs (line 117) | def test_get_runs(self):
FILE: tests/test_utils.py
class TestOptionalReturnType (line 10) | class TestOptionalReturnType(unittest.TestCase):
method test_namedtuple (line 11) | def test_namedtuple(self):
method test_with_opt (line 24) | def test_with_opt(self):
FILE: tests/threads.c
function main (line 35) | int main(void)
FILE: tests/threads2.c
function search (line 12) | int search(void)
function main (line 40) | int main(void)
FILE: tests/vfork.c
function main (line 13) | int main(void)
Condensed preview — 178 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (1,089K chars).
[
{
"path": ".github/workflows/test.yml",
"chars": 6487,
"preview": "name: Test\n\non:\n - push\n - pull_request\n\njobs:\n test:\n strategy:\n fail-fast: false\n matrix:\n os: "
},
{
"path": ".gitignore",
"chars": 559,
"preview": "*.py[co]\n.DS_Store\n\n# Packages\n*.egg\n*.egg-info\ndist\nbuild\neggs\nparts\nbin\nvar\nsdist\ndevelop-eggs\n.installed.cfg\nlib\nlib6"
},
{
"path": ".readthedocs.yaml",
"chars": 105,
"preview": "version: 2\n\nbuild:\n os: ubuntu-22.04\n tools:\n python: \"3.11\"\n\nsphinx:\n configuration: docs/conf.py\n"
},
{
"path": "CHANGELOG.md",
"chars": 20716,
"preview": "Changelog\n=========\n\n1.3.2 (2026-01-18)\n------------------\n\n(reprozip and reprounzip only)\n\nEnhancement:\n* Don't depend "
},
{
"path": "CITATION.cff",
"chars": 6911,
"preview": "cff-version: 1.2.0\nmessage: If you use this software, please cite it as below.\nauthors:\n - family-names: Rampin\n giv"
},
{
"path": "CITATION.txt",
"chars": 857,
"preview": "To reference ReproZip in publications, please cite the following:\n\nReproZip: Computational Reproducibility With Ease, F."
},
{
"path": "CONTRIBUTING.md",
"chars": 5323,
"preview": "# Contents\n* [Notes](#notes)\n* [Contributing](#contributing)\n* [Resolving Merge Conflicts](#resolving-merge-conflicts)\n*"
},
{
"path": "LICENSE.txt",
"chars": 1505,
"preview": "Copyright (C) 2014, New York University\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or"
},
{
"path": "README.md",
"chars": 7759,
"preview": "[](https://github.com/VIDA-NYU/reprozip/ac"
},
{
"path": "Vagrantfile",
"chars": 857,
"preview": "# -*- mode: ruby -*-\n# vi: set ft=ruby :\n\nVAGRANTFILE_API_VERSION = \"2\"\n\nVagrant.configure(VAGRANTFILE_API_VERSION) do |"
},
{
"path": "allsetups.sh",
"chars": 792,
"preview": "#!/bin/sh\n\nset -e\nset -u\n\ncd \"$(dirname $0)\"\n\nPROGRAMS=\"reprounzip reprounzip-docker reprounzip-vagrant reprounzip-vistr"
},
{
"path": "docs/Makefile",
"chars": 7614,
"preview": "# Makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line.\nSPHINXOPTS =\nSPHINXBUILD "
},
{
"path": "docs/conf.py",
"chars": 10022,
"preview": "# -*- coding: utf-8 -*-\n#\n# ReproZip documentation build configuration file, created by\n# sphinx-quickstart on Tue May 3"
},
{
"path": "docs/developerguide.rst",
"chars": 10089,
"preview": ".. _develop-plugins:\n\nDeveloper's Guide\n*****************\n\nGeneral Development Information\n----------------------------"
},
{
"path": "docs/faq.rst",
"chars": 6223,
"preview": ".. _faq:\n\nFrequently Asked Questions\n**************************\n\nCan ReproZip pack a client-server scenario?\n=========="
},
{
"path": "docs/glossary.rst",
"chars": 1503,
"preview": ".. _glossary:\n\nGlossary\n********\n\n.. glossary::\n\n configuration (file)\n A YAML file generated by ``reprozip "
},
{
"path": "docs/graph.rst",
"chars": 5903,
"preview": ".. _graph:\n\nVisualizing the Provenance Graph\n********************************\n\n.. note:: If you are using a Python ver"
},
{
"path": "docs/gui.rst",
"chars": 2191,
"preview": ".. _unpacking-gui:\n\nReproUnzip GUI\n**************\n\n**reprounzip-qt** is a graphical interface (GUI) for reprounzip, all"
},
{
"path": "docs/index.rst",
"chars": 2016,
"preview": "ReproZip's Documentation\n************************\n\nWelcome to ReproZip's documentation!\n\n`ReproZip <https://www.reprozip"
},
{
"path": "docs/install.rst",
"chars": 10719,
"preview": ".. _install:\n\nInstallation\n************\n\nReproZip is available as open source, released under the Revised BSD License. "
},
{
"path": "docs/jupyter.rst",
"chars": 4013,
"preview": ".. _reprozip-jupyter:\n\nMaking Jupyter Notebooks Reproducible with ReproZip\n********************************************"
},
{
"path": "docs/make.bat",
"chars": 7736,
"preview": "@ECHO OFF\r\n\r\nREM Command file for Sphinx documentation\r\n\r\nif \"%SPHINXBUILD%\" == \"\" (\r\n\tset SPHINXBUILD=sphinx-build\r\n)\r\n"
},
{
"path": "docs/man/reprounzip-docker.1",
"chars": 3299,
"preview": ".\\\" Manpage for reprounzip\n.TH man 1 \"4 November 2017\" \"1.0.10\" \"reprounzip\\-docker man page\"\n.SH NAME\nreprounzip\\-docke"
},
{
"path": "docs/man/reprounzip-vagrant.1",
"chars": 3026,
"preview": ".\\\" Manpage for reprounzip\n.TH man 1 \"4 November 2017\" \"1.0.10\" \"reprounzip\\-vagrant man page\"\n.SH NAME\nreprounzip\\-vagr"
},
{
"path": "docs/man/reprounzip.1",
"chars": 4484,
"preview": ".\\\" Manpage for reprounzip\n.TH man 1 \"4 November 2017\" \"1.0.11\" \"reprounzip man page\"\n.SH NAME\nreprounzip \\- the reprodu"
},
{
"path": "docs/man/reprozip.1",
"chars": 3190,
"preview": ".\\\" Manpage for reprozip\n.TH man 1 \"4 November 2017\" \"1.0.11\" \"reprozip man page\"\n.SH NAME\nreprozip \\- the reproducibili"
},
{
"path": "docs/packing.rst",
"chars": 14078,
"preview": ".. _packing:\n\nUsing *reprozip*\n****************\n\nThe *reprozip* component is responsible for packing an experiment, whi"
},
{
"path": "docs/reprozip.rst",
"chars": 2398,
"preview": "Why ReproZip?\n*************\n\nReproducibility is a core component of the scientific process: it helps researchers all aro"
},
{
"path": "docs/traceschema.rst",
"chars": 3461,
"preview": ".. _trace-schema:\n\nTrace Database Schema\n*********************\n\nThe database contains three tables: ``processes``, ``op"
},
{
"path": "docs/troubleshooting.rst",
"chars": 9762,
"preview": ".. _troubleshooting:\n\nTroubleshooting\n***************\n\nThe best way to start solving an issue in ReproZip is probably t"
},
{
"path": "docs/unpacked-format.rst",
"chars": 9613,
"preview": ".. _unpacked-format:\n\nStructure of Unpacked Experiments\n*********************************\n\nWhile *reprounzip* is design"
},
{
"path": "docs/unpacking.rst",
"chars": 23591,
"preview": ".. _unpacking:\n\nUsing *reprounzip*\n******************\n\nWhile *reprozip* is responsible for tracing and packing an exper"
},
{
"path": "docs/vistrails.rst",
"chars": 3515,
"preview": ".. _vistrails:\n\nVisTrails Plugin\n****************\n\nThe `reprounzip-vistrails` plugin is a component that interacts with"
},
{
"path": "file-format.md",
"chars": 1343,
"preview": "ReproZip pack file format\n=========================\n\nThe pack command creates an archive that contains the necessary fil"
},
{
"path": "reprounzip/LICENSE.txt",
"chars": 1505,
"preview": "Copyright (C) 2014, New York University\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or"
},
{
"path": "reprounzip/MANIFEST.in",
"chars": 39,
"preview": "include README.rst\ninclude LICENSE.txt\n"
},
{
"path": "reprounzip/README.rst",
"chars": 1540,
"preview": "ReproZip project\n================\n\n`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process o"
},
{
"path": "reprounzip/reprounzip/__init__.py",
"chars": 208,
"preview": "try: # pragma: no cover\n __import__('pkg_resources').declare_namespace(__name__)\nexcept ImportError: # pragma: no c"
},
{
"path": "reprounzip/reprounzip/common.py",
"chars": 32486,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/main.py",
"chars": 5226,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/orderedset.py",
"chars": 2987,
"preview": "# From http://code.activestate.com/recipes/576694/\n# With added update()\n\n# Copyright (C) 2009 Raymond Hettinger\n#\n# Per"
},
{
"path": "reprounzip/reprounzip/pack_info.py",
"chars": 14306,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/parameters.py",
"chars": 29269,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/plugins/__init__.py",
"chars": 208,
"preview": "try: # pragma: no cover\n __import__('pkg_resources').declare_namespace(__name__)\nexcept ImportError: # pragma: no c"
},
{
"path": "reprounzip/reprounzip/signals.py",
"chars": 4543,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/unpackers/__init__.py",
"chars": 208,
"preview": "try: # pragma: no cover\n __import__('pkg_resources').declare_namespace(__name__)\nexcept ImportError: # pragma: no c"
},
{
"path": "reprounzip/reprounzip/unpackers/common/__init__.py",
"chars": 1646,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/unpackers/common/misc.py",
"chars": 21418,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/unpackers/common/packages.py",
"chars": 6480,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/unpackers/common/x11.py",
"chars": 15399,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/unpackers/default.py",
"chars": 37484,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/unpackers/graph.py",
"chars": 30698,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/unpackers/provviewer.py",
"chars": 13104,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/reprounzip/utils.py",
"chars": 13928,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip/setup.cfg",
"chars": 28,
"preview": "[bdist_wheel]\nuniversal = 1\n"
},
{
"path": "reprounzip/setup.py",
"chars": 2729,
"preview": "import io\nimport os\nfrom setuptools import setup\n\n\n# pip workaround\nos.chdir(os.path.abspath(os.path.dirname(__file__)))"
},
{
"path": "reprounzip-docker/LICENSE.txt",
"chars": 1505,
"preview": "Copyright (C) 2014, New York University\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or"
},
{
"path": "reprounzip-docker/MANIFEST.in",
"chars": 39,
"preview": "include README.rst\ninclude LICENSE.txt\n"
},
{
"path": "reprounzip-docker/README.rst",
"chars": 1545,
"preview": "ReproZip project\n================\n\n`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process o"
},
{
"path": "reprounzip-docker/reprounzip/__init__.py",
"chars": 208,
"preview": "try: # pragma: no cover\n __import__('pkg_resources').declare_namespace(__name__)\nexcept ImportError: # pragma: no c"
},
{
"path": "reprounzip-docker/reprounzip/unpackers/__init__.py",
"chars": 208,
"preview": "try: # pragma: no cover\n __import__('pkg_resources').declare_namespace(__name__)\nexcept ImportError: # pragma: no c"
},
{
"path": "reprounzip-docker/reprounzip/unpackers/docker.py",
"chars": 37748,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-docker/setup.cfg",
"chars": 28,
"preview": "[bdist_wheel]\nuniversal = 1\n"
},
{
"path": "reprounzip-docker/setup.py",
"chars": 1956,
"preview": "import io\nimport os\nfrom setuptools import setup\n\n\n# pip workaround\nos.chdir(os.path.abspath(os.path.dirname(__file__)))"
},
{
"path": "reprounzip-qt/LICENSE.txt",
"chars": 1505,
"preview": "Copyright (C) 2014, New York University\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or"
},
{
"path": "reprounzip-qt/MANIFEST.in",
"chars": 39,
"preview": "include README.rst\ninclude LICENSE.txt\n"
},
{
"path": "reprounzip-qt/README.rst",
"chars": 1439,
"preview": "ReproZip project\n================\n\n`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process o"
},
{
"path": "reprounzip-qt/reprounzip_qt/__init__.py",
"chars": 22,
"preview": "__version__ = '1.2.1'\n"
},
{
"path": "reprounzip-qt/reprounzip_qt/gui/__init__.py",
"chars": 8021,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-qt/reprounzip_qt/gui/common.py",
"chars": 2527,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-qt/reprounzip_qt/gui/run.py",
"chars": 17644,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-qt/reprounzip_qt/gui/unpack.py",
"chars": 13607,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-qt/reprounzip_qt/main.py",
"chars": 2682,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-qt/reprounzip_qt/qt_terminal.py",
"chars": 4825,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-qt/reprounzip_qt/reprounzip_interface.py",
"chars": 13634,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-qt/reprounzip_qt/usage.py",
"chars": 1200,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-qt/setup.cfg",
"chars": 28,
"preview": "[bdist_wheel]\nuniversal = 1\n"
},
{
"path": "reprounzip-qt/setup.py",
"chars": 1939,
"preview": "import io\nimport os\nfrom setuptools import setup\n\n\n# pip workaround\nos.chdir(os.path.abspath(os.path.dirname(__file__)))"
},
{
"path": "reprounzip-vagrant/LICENSE.txt",
"chars": 1505,
"preview": "Copyright (C) 2014, New York University\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or"
},
{
"path": "reprounzip-vagrant/MANIFEST.in",
"chars": 39,
"preview": "include README.rst\ninclude LICENSE.txt\n"
},
{
"path": "reprounzip-vagrant/README.rst",
"chars": 1555,
"preview": "ReproZip project\n================\n\n`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process o"
},
{
"path": "reprounzip-vagrant/reprounzip/__init__.py",
"chars": 208,
"preview": "try: # pragma: no cover\n __import__('pkg_resources').declare_namespace(__name__)\nexcept ImportError: # pragma: no c"
},
{
"path": "reprounzip-vagrant/reprounzip/unpackers/__init__.py",
"chars": 208,
"preview": "try: # pragma: no cover\n __import__('pkg_resources').declare_namespace(__name__)\nexcept ImportError: # pragma: no c"
},
{
"path": "reprounzip-vagrant/reprounzip/unpackers/vagrant/__init__.py",
"chars": 37330,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-vagrant/reprounzip/unpackers/vagrant/interaction.py",
"chars": 3928,
"preview": "# This is paramiko/demos/interactive.py\n# Part of the Paramiko project; https://github.com/paramiko/paramiko/\n# Adapted "
},
{
"path": "reprounzip-vagrant/reprounzip/unpackers/vagrant/run_command.py",
"chars": 6386,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-vagrant/setup.cfg",
"chars": 28,
"preview": "[bdist_wheel]\nuniversal = 1\n"
},
{
"path": "reprounzip-vagrant/setup.py",
"chars": 2029,
"preview": "import io\nimport os\nfrom setuptools import setup\n\n\n# pip workaround\nos.chdir(os.path.abspath(os.path.dirname(__file__)))"
},
{
"path": "reprounzip-vistrails/LICENSE.txt",
"chars": 1505,
"preview": "Copyright (C) 2014, New York University\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or"
},
{
"path": "reprounzip-vistrails/MANIFEST.in",
"chars": 39,
"preview": "include README.rst\ninclude LICENSE.txt\n"
},
{
"path": "reprounzip-vistrails/README.rst",
"chars": 1466,
"preview": "ReproZip project\n================\n\n`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process o"
},
{
"path": "reprounzip-vistrails/reprounzip/__init__.py",
"chars": 208,
"preview": "try: # pragma: no cover\n __import__('pkg_resources').declare_namespace(__name__)\nexcept ImportError: # pragma: no c"
},
{
"path": "reprounzip-vistrails/reprounzip/plugins/__init__.py",
"chars": 208,
"preview": "try: # pragma: no cover\n __import__('pkg_resources').declare_namespace(__name__)\nexcept ImportError: # pragma: no c"
},
{
"path": "reprounzip-vistrails/reprounzip/plugins/vistrails.py",
"chars": 13790,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprounzip-vistrails/setup.cfg",
"chars": 28,
"preview": "[bdist_wheel]\nuniversal = 1\n"
},
{
"path": "reprounzip-vistrails/setup.py",
"chars": 2068,
"preview": "import io\nimport os\nfrom setuptools import setup\nimport sys\n\n\n# pip workaround\nos.chdir(os.path.abspath(os.path.dirname("
},
{
"path": "reprozip/LICENSE.txt",
"chars": 1505,
"preview": "Copyright (C) 2014, New York University\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or"
},
{
"path": "reprozip/MANIFEST.in",
"chars": 52,
"preview": "include README.rst\ninclude LICENSE.txt\ngraft native\n"
},
{
"path": "reprozip/README.rst",
"chars": 1451,
"preview": "ReproZip project\n================\n\n`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process o"
},
{
"path": "reprozip/native/config.h",
"chars": 583,
"preview": "#ifndef CONFIG_H\n#define CONFIG_H\n\n#define WORD_SIZE sizeof(int)\n\n#if !defined(X86) && !defined(X86_64)\n# if defined(_"
},
{
"path": "reprozip/native/database.c",
"chars": 11602,
"preview": "#include <errno.h>\n#include <signal.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n#include <time.h>\n\n#in"
},
{
"path": "reprozip/native/database.h",
"chars": 984,
"preview": "#ifndef DATABASE_H\n#define DATABASE_H\n\n#define FILE_READ 0x01\n#define FILE_WRITE 0x02\n#define FILE_WDIR 0x04 /* Fi"
},
{
"path": "reprozip/native/log.h",
"chars": 902,
"preview": "#ifndef LOG_H\n#define LOG_H\n\n#include <stdio.h>\n#include <time.h>\n\n#include <sys/time.h>\n#include <sys/types.h>\n\n\nextern"
},
{
"path": "reprozip/native/ptrace_utils.c",
"chars": 4295,
"preview": "#include <errno.h>\n#include <inttypes.h>\n#include <stdlib.h>\n#include <string.h>\n#include <sys/ptrace.h>\n#include <sys/t"
},
{
"path": "reprozip/native/ptrace_utils.h",
"chars": 549,
"preview": "#ifndef PTRACE_UTILS_H\n#define PTRACE_UTILS_H\n\nvoid *tracee_getptr(int mode, pid_t tid, const void *addr);\nuint64_t trac"
},
{
"path": "reprozip/native/pylog.c",
"chars": 3042,
"preview": "#include <assert.h>\n#include <errno.h>\n#include <stdarg.h>\n#include <stdlib.h>\n#include <string.h>\n#include <time.h>\n\n#i"
},
{
"path": "reprozip/native/pytracer.c",
"chars": 4552,
"preview": "#include <Python.h>\n\n#include \"database.h\"\n#include \"log.h\"\n#include \"tracer.h\"\n\n\nPyObject *Err_Base;\n\n\n/**\n * Makes a C"
},
{
"path": "reprozip/native/syscalls.c",
"chars": 52039,
"preview": "#include <errno.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n\n#include <arpa/inet.h>\n#include <fcntl.h>"
},
{
"path": "reprozip/native/syscalls.h",
"chars": 265,
"preview": "#ifndef SYSCALL_H\n#define SYSCALL_H\n\n#include \"tracer.h\"\n\nvoid syscall_build_table(void);\n\nint syscall_handle(struct Pro"
},
{
"path": "reprozip/native/tracer.c",
"chars": 22977,
"preview": "#include <errno.h>\n#include <signal.h>\n#include <stdint.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n\n#"
},
{
"path": "reprozip/native/tracer.h",
"chars": 2304,
"preview": "#ifndef TRACER_H\n#define TRACER_H\n\n#include \"config.h\"\n\n\nint fork_and_trace(const char *binary, int argc, char **argv,\n "
},
{
"path": "reprozip/native/utils.c",
"chars": 3612,
"preview": "#include <errno.h>\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n\n#include <fcntl.h>\n#include <sys/stat.h>\n"
},
{
"path": "reprozip/native/utils.h",
"chars": 349,
"preview": "#ifndef UTILS_H\n#define UTILS_H\n\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n\n#include <sys/types.h>\n#inc"
},
{
"path": "reprozip/reprozip/__init__.py",
"chars": 189,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip/reprozip/common.py",
"chars": 32486,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip/reprozip/filters.py",
"chars": 3828,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip/reprozip/main.py",
"chars": 14043,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip/reprozip/pack.py",
"chars": 7947,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip/reprozip/tracer/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "reprozip/reprozip/tracer/linux_pkgs.py",
"chars": 9370,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip/reprozip/tracer/trace.py",
"chars": 19852,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip/reprozip/traceutils.py",
"chars": 7668,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip/reprozip/utils.py",
"chars": 13928,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip/setup.py",
"chars": 2964,
"preview": "import io\nimport os\nfrom setuptools import setup, Extension\nimport sys\n\n\n# pip workaround\nos.chdir(os.path.abspath(os.pa"
},
{
"path": "reprozip-jupyter/LICENSE.txt",
"chars": 1505,
"preview": "Copyright (C) 2014, New York University\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or"
},
{
"path": "reprozip-jupyter/MANIFEST.in",
"chars": 39,
"preview": "include README.rst\ninclude LICENSE.txt\n"
},
{
"path": "reprozip-jupyter/README.rst",
"chars": 2036,
"preview": "ReproZip project\n================\n\n`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process o"
},
{
"path": "reprozip-jupyter/reprozip_jupyter/__init__.py",
"chars": 433,
"preview": "\"\"\"Traces and packs notebook environments with ReproZip.\n\"\"\"\n\n__version__ = '1.2'\n\n\ndef _jupyter_nbextension_paths():\n "
},
{
"path": "reprozip-jupyter/reprozip_jupyter/main.py",
"chars": 1598,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip-jupyter/reprozip_jupyter/notebook-extension.js",
"chars": 4647,
"preview": "define([\"jquery\",\n \"base/js/namespace\",\n \"base/js/utils\",\n \"base/js/dialog\"],\nfunction notebook_ext"
},
{
"path": "reprozip-jupyter/reprozip_jupyter/run.py",
"chars": 3645,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip-jupyter/reprozip_jupyter/server_extension.py",
"chars": 3614,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip-jupyter/reprozip_jupyter/trace.py",
"chars": 6968,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "reprozip-jupyter/setup.cfg",
"chars": 28,
"preview": "[bdist_wheel]\nuniversal = 1\n"
},
{
"path": "reprozip-jupyter/setup.py",
"chars": 2002,
"preview": "import io\nimport os\nfrom setuptools import setup\n\n\n# pip workaround\nos.chdir(os.path.abspath(os.path.dirname(__file__)))"
},
{
"path": "scripts/RELEASE",
"chars": 774,
"preview": "Update the CHANGELOG.md file\n\nUpload parameters to stats.reprozip.org\n\nBump versions in setup.py files, __init__.py file"
},
{
"path": "scripts/linux-wheels.sh",
"chars": 411,
"preview": "#!/bin/sh\n\ncd \"$(dirname \"$0\")/..\"\nif [ ! -d dist ]; then mkdir dist; fi\nfor image in manylinux2010_x86_64 manylinux2014"
},
{
"path": "scripts/macos/README.txt",
"chars": 530,
"preview": "This will install Python, ReproUnzip and all plugins to /opt/reprounzip.\n\nOnce this setup is done, you will be able to r"
},
{
"path": "scripts/macos/ReproUnzip.pkgproj",
"chars": 41820,
"preview": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/P"
},
{
"path": "scripts/macos/app-shim/ReproUnzip_app/Contents/Info.plist",
"chars": 1274,
"preview": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/P"
},
{
"path": "scripts/macos/app-shim/reprounzip-qt.c",
"chars": 341,
"preview": "#include <stdio.h>\n#include <stdlib.h>\n#include <unistd.h>\n\nint main(int argc, char **argv) {\n char **args = malloc(s"
},
{
"path": "scripts/macos/app-shim/reprounzip-qt.debug.c",
"chars": 653,
"preview": "#include <stdio.h>\n#include <stdlib.h>\n#include <unistd.h>\n\n/* This version of reprounzip-qt runs from a virtualenv, use"
},
{
"path": "scripts/macos/instructions.txt",
"chars": 934,
"preview": "Reset PATH to not include macports, etc\nexport DYLD_FRAMEWORK_PATH=/opt/reprounzip/Frameworks\nexport CFLAGS=\"-mmacosx-ve"
},
{
"path": "scripts/macos/reprounzip",
"chars": 62,
"preview": "#!/bin/sh\n\nexec \"$(dirname \"$0\")/python3/bin/reprounzip\" \"$@\"\n"
},
{
"path": "scripts/macos/reprounzip-qt",
"chars": 169,
"preview": "#!/bin/sh\n\nDYLD_FRAMEWORK_PATH=/opt/reprounzip/Frameworks; export DYLD_FRAMEWORK_PATH\nPATH=\"/opt/reprounzip:$PATH\"\nexec "
},
{
"path": "scripts/macos/reprozip-jupyter",
"chars": 68,
"preview": "#!/bin/sh\n\nexec \"$(dirname \"$0\")/python3/bin/reprozip-jupyter\" \"$@\"\n"
},
{
"path": "scripts/register-linux.sh",
"chars": 1142,
"preview": "#!/bin/sh\n\nif ! which reprounzip-qt &>/dev/null; then\n echo \"reprounzip-qt is not in PATH\" >&2\n exit 1\nfi\n\n# Insta"
},
{
"path": "scripts/test_bug_13676.py",
"chars": 964,
"preview": "from __future__ import unicode_literals\n\nimport sqlite3\nimport sys\n\n\ndata_in = \"R\\xE9mi\\0wrote\\0this\"\n\n\nif __name__ == '"
},
{
"path": "scripts/test_bug_23058.py",
"chars": 2033,
"preview": "from __future__ import unicode_literals\n\nimport argparse\nimport unittest\n\n\nclass Test23058(unittest.TestCase):\n def d"
},
{
"path": "scripts/windows/input/reprounzip-qt.bat",
"chars": 82,
"preview": "@echo off\r\nset PATH=%~dp0;%PATH%\r\n\"%~dp0\\python2.7\\Scripts\\reprounzip-qt.exe\" %*\r\n"
},
{
"path": "scripts/windows/input/reprounzip.bat",
"chars": 56,
"preview": "@echo off\r\n\"%~dp0\\python2.7\\Scripts\\reprounzip.exe\" %*\r\n"
},
{
"path": "scripts/windows/input/reprozip-jupyter.bat",
"chars": 60,
"preview": "@echo off\n\"%~dp0\\python2.7\\Scripts\\reprozip-jupyter.exe\" %*\n"
},
{
"path": "scripts/windows/reprounzip.iss",
"chars": 2381,
"preview": "; This needs:\r\n; Default installation of Python 3.9 in C:\\Python3.9\r\n; python39.dll in C:\\Python3.9 (might be in Win"
},
{
"path": "syscalls.txt",
"chars": 569,
"preview": "process syscalls\n x clone\n x fork\n x vfork\n unshare\n\n\nfile opening syscalls\n x creat\n x execve\n mkdir\n x ope"
},
{
"path": "tests/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/__main__.py",
"chars": 3443,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "tests/check_images.py",
"chars": 6749,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "tests/common.py",
"chars": 2007,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "tests/connect.c",
"chars": 815,
"preview": "/* connect.c\n *\n * Connects to a TCP server.\n *\n * usage: ./connect\n */\n\n#include <stdio.h>\n#include <string.h>\n\n#includ"
},
{
"path": "tests/exec_echo.c",
"chars": 401,
"preview": "/* exec_echo.c\n *\n * This is a very simple program that executes /bin/echo to display a message.\n * Used to try an i386 "
},
{
"path": "tests/functional.py",
"chars": 32263,
"preview": "#!/usr/bin/env python\n\n# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under "
},
{
"path": "tests/readwrite.c",
"chars": 322,
"preview": "/* readwrite.c\n *\n * This tests the readwrite access mode.\n *\n * usage: ./readwrite file\n */\n\n#include <stdio.h>\n\n\nint m"
},
{
"path": "tests/rename.c",
"chars": 869,
"preview": "/* rename.c\n *\n * This is a very simple program that creates a file, renames it, and opens the\n * renamed files. It test"
},
{
"path": "tests/segv.c",
"chars": 425,
"preview": "/* segv.c\n *\n * This program will spawn a process that will seg fault before the main\n * process finishes.\n *\n * usage: "
},
{
"path": "tests/simple.c",
"chars": 1102,
"preview": "/* simple.c\n *\n * This is a very simple program that loads data from a file and creates\n * another file with a result.\n "
},
{
"path": "tests/simple_input.txt",
"chars": 6,
"preview": "29 13\n"
},
{
"path": "tests/simple_input2.txt",
"chars": 6,
"preview": "25 11\n"
},
{
"path": "tests/test_graph.py",
"chars": 15945,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "tests/test_parameters.py",
"chars": 3191,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "tests/test_rails_filter.py",
"chars": 3311,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "tests/test_reprounzip.py",
"chars": 9297,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "tests/test_reprozip.py",
"chars": 13285,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "tests/test_unpackers_common.py",
"chars": 4971,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "tests/test_utils.py",
"chars": 1803,
"preview": "# Copyright (C) 2014 New York University\n# This file is part of ReproZip which is released under the Revised BSD License"
},
{
"path": "tests/threads.c",
"chars": 988,
"preview": "/* threads.c\n *\n * This is a very simple threaded program.\n *\n * usage: ./threads\n */\n\n#include <pthread.h>\n#include <st"
},
{
"path": "tests/threads2.c",
"chars": 770,
"preview": "/* threads2.c\n *\n * A second multithreaded program.\n *\n * usage: ./threads2\n */\n\n#include <pthread.h>\n#include <stdio.h>"
},
{
"path": "tests/vfork.c",
"chars": 542,
"preview": "/* vfork.c\n *\n * This just calls vfork() then execve().\n *\n * usage: ./vfork\n */\n\n#include <sys/types.h>\n#include <sys/w"
}
]
// ... and 1 more files (download for full content)
About this extraction
This page contains the full source code of the VIDA-NYU/reprozip GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 178 files (1004.6 KB), approximately 250.1k tokens, and a symbol index with 785 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.