[
  {
    "path": ".github/workflows/build.yaml",
    "content": "name: Check\n\non:\n  push:\n    branches:\n      - main\n      - master\n      - action\n  pull_request:\n    branches:\n      - main\n      - master\n\njobs:\n  check-numpy-import:\n    runs-on: ubuntu-latest\n\n    steps:\n      - name: Checkout repository\n        uses: actions/checkout@v2\n\n      - name: Set up Nix\n        uses: cachix/install-nix-action@v22\n\n      - name: Install Python 3.11 with Nix\n        run: nix --extra-experimental-features nix-command --extra-experimental-features flakes run nixpkgs#python311 -- -m venv .venv --copies\n\n      - name: check that using it on numpy works\n        run: |\n          echo \"Install numpy\"\n          .venv/bin/pip install numpy\n          echo \"Run fix-python\"\n          ./fix-python --venv .venv\n          echo \"Check numpy works\"\n          .venv/bin/python -c 'import numpy'\n"
  },
  {
    "path": ".gitignore",
    "content": "result\n.nix\n.venv\nlibs.nix\n"
  },
  {
    "path": "README.md",
    "content": "# fix-python\n\nWork with Python \"normally\" on NixOS in one command!\n\nTired of all these \"*.so not found\" errors?\nChange the RPATH of all the binaries in your venv!\n\n## Requirements\n\n- Nix\n- `nix-command` and `flakes` experimental features must be enabled\n\n## Install\n\nUse temporarily in a shell\n\n```\nnix shell github:GuillaumeDesforges/fix-python\n```\n\nOr add it to your profile\n\n```\nnix profile install github:GuillaumeDesforges/fix-python\n```\n\n## Usage\n\nIn your Python project, create a virtual environment `.venv` and use your preferred tool (pip, poetry, ...) to install your dependencies.\n\n> [!IMPORTANT]\n> `fix-python` will patch binary files in the virtual environment _without_ following symlinks.\n> \n> For example, if you use the `venv` module from Python's standard library, please make sure you pass `--copies` when creating the environment.\n> ```bash\n> python -m venv .venv --copies\n> ```\n\nBy default, `fix-python` patches the packages given in the following expression:\n```nix\nlet pkgs = import (builtins.getFlake \"nixpkgs\") { };\nin [\n  pkgs.gcc.cc\n  pkgs.glibc\n  pkgs.zlib\n]\n```\n\n> Note: these three packages are fundamental for most Python packages and should never be removed.\n\nIf you need to patch packages in addition to these, create a `.nix/libs.nix` file with a structure similar to the above that returns the array of packages that you want binaries to be linked with.\n\n> Note: you may add this `.nix` folder to your project `.gitignore`.\n\nFinally, call `fix-python`.\n\n```console\nfix-python --venv .venv [--libs .nix/libs.nix]\n```\n\nThe list of options is:\n\n```\n$ ./fix-python --help\nUsage: fix-python --venv .venv [--libs libs.nix] [--no-default-libs]\n--help: show this help message\n--venv: path to Python virtual environment\n--libs: path to a Nix file which returns a list of derivations\n--no-default-libs: don't patch C++ standard libraries, glibc, and zlib by default\n--gpu: enable GPU support\n--with-torch: fix pytorch dependencies issues\n--deep: looks for anything executable to patch, very slow but needed sometimes (e.g. PyQt)\n--verbose: increase verbosity\n```\n"
  },
  {
    "path": "fix-python",
    "content": "#!/usr/bin/env bash\nset -e\n\n# This script fix issues with Python binaries on NixOS\n# Usage:\n# fix-python --venv .venv [--libs libs.nix] [--no-default-libs]\n\nDEFAULT_LIBS_EXPRESSION=\"\n(\n  let pkgs = import (builtins.getFlake \\\"nixpkgs\\\") { };\n  in [\n    pkgs.gcc.cc\n    pkgs.glibc\n    pkgs.zlib\n  ]\n)\n\"\n\n# Help\nif [ \"$1\" = \"--help\" ]; then\n  echo \"Usage: fix-python --venv .venv [--libs libs.nix] [--no-default-libs]\" >&2\n  echo \"--help: show this help message\" >&2\n  echo \"--venv: path to Python virtual environment\" >&2\n  echo \"--libs: path to a Nix file which returns a list of derivations\" >&2\n  echo \"--no-default-libs: don't patch C++ standard libraries, glibc, and zlib by default\" >&2\n  echo \"--gpu: enable GPU support\" >&2\n  echo \"--with-torch: fix pytorch dependencies issues\" >&2\n  echo \"--deep: looks for anything executable to patch, very slow but needed sometimes (e.g. PyQt)\" >&2\n  echo \"--verbose: increase verbosity\" >&2\n  exit 0\nfi\n\n# arguments\nwhile [ $# -gt 0 ]; do\n  case \"$1\" in\n    --venv)\n      shift\n      VENV_PATH=\"$1\"\n      ;;\n    --libs)\n      shift\n      LIBS_PATH=\"$1\"\n      ;;\n    --no-default-libs)\n      shift\n      DEFAULT_LIBS_EXPRESSION=\"[]\"\n      ;;\n    --gpu)\n      enable_gpu=\"1\"\n      ;;\n    --with-torch)\n      enable_torch=\"1\"\n      ;;\n    --deep)\n      deep=\"1\"\n      ;;\n    --verbose)\n      verbose=\"1\"\n      ;;\n    *)\n      echo \"Unknown argument: $1\" >&2\n      echo \"Usage: fix-python --venv .venv [--libs libs.nix] [--no-default-libs]\" >&2\n      exit 1\n      ;;\n  esac\n  shift\ndone\n\n# check arguments\nif [ -z \"$VENV_PATH\" ]; then\n  echo \"Missing argument: --venv\" >&2\n  echo \"Usage: fix-python --venv .venv [--libs libs.nix] [--no-default-libs]\" >&2\n  echo \"or set VENV_PATH\" >&2\n  exit 1\nfi\n\n# check runtime dependencies are installed\nif ! command -v file &> /dev/null\nthen\n  echo \"Automatically adding \\\"file\\\" to PATH.\" >&2\n  dep_file_path=\"$(nix build --no-link --print-out-paths nixpkgs#file.out)/bin\"\n  export PATH=\"$dep_file_path:$PATH\"\n  if [ \"$verbose\" ]; then \n    echo \"dep_file_path=$dep_file_path\" >&2\n  fi\nfi\nif ! command -v patchelf &> /dev/null\nthen\n  echo \"Automatically adding \\\"patchelf\\\" to PATH.\" >&2\n  dep_patchelf_path=\"$(nix build --no-link --print-out-paths nixpkgs#patchelf)/bin\"\n  export PATH=\"$dep_patchelf_path:$PATH\"\n  if [ \"$verbose\" ]; then \n    echo \"dep_patchelf_path=$dep_patchelf_path\" >&2\n  fi\nfi\n\n# load libs from Nix file\nif [ \"$LIBS_PATH\" ];\nthen\n  # if $LIBS_PATH is just a file in the current working directory,\n  # specified without leading \"./\", we add \"./\" so that $LIBS_PATH\n  # can be interpreted directly as a Nix path in the following\n  # expression\n  if [[ ! \"$LIBS_PATH\" == *\"/\"* ]];\n  then\n    LIBS_PATH=\"./$LIBS_PATH\"\n  fi\n  custom_libs_expression=\"(import $LIBS_PATH)\"\n  mkdir -p .nix/fix-python\n  nix_libs_build_status=$(\n    nix build --impure --expr \"$custom_libs_expression\" -o .nix/fix-python/result\n    echo $?\n  )\n  if [ \"$nix_libs_build_status\" -eq \"1\" ];\n  then\n    echo \"Failed to load libs from Nix file $LIBS_PATH\" >&2\n    echo \"\" >&2\n    echo \"Try to debug this issue with the command:\" >&2\n    echo \"    nix build --impure --expr \\\"import $LIBS_PATH\\\"\" >&2\n    exit 1\n  fi\nelse\n  custom_libs_expression=\"[]\"\nfi\nall_nix_libs_expression=\"($custom_libs_expression ++ $DEFAULT_LIBS_EXPRESSION)\"\nnixos_python_nix_libs=\"$(nix eval --impure --expr \"let pkgs = import (builtins.getFlake \\\"nixpkgs\\\") {}; in pkgs.lib.strings.makeLibraryPath $all_nix_libs_expression\" | sed 's/^\"\\(.*\\)\"$/\\1/')\"\nif [ \"$verbose\" ]; then \n  echo \"nixos_python_nix_libs=$nixos_python_nix_libs\" >&2\nfi\nlibs=\"$nixos_python_nix_libs\"\n\n# load libs from virtual environment\npython_venv_libs=$(echo \"$(find \"$(realpath \"$VENV_PATH\")\" -name '*.libs'):$(find \"$(realpath \"$VENV_PATH\")\" -name 'lib')\" | tr '\\n' ':')\nif [ \"$verbose\" ]; then \n  echo \"nixos_python_venv_libs=$python_venv_libs\" >&2\nfi\nlibs=\"$libs:$python_venv_libs\"\n\n# load libs from NixOS for GPU support if requested\nif [ \"$enable_gpu\" ]; then\n  nixos_gpu_libs=\"$(readlink /run/opengl-driver)/lib\"\n  if [ \"$verbose\" ]; then \n    echo \"nixos_gpu_libs=$nixos_gpu_libs\" >&2\n  fi\n  libs=\"$libs:$nixos_gpu_libs\"\nfi\n\n# put it all together\nlibs=$(echo \"$libs\" | sed 's/:\\+/:/g' | sed 's/^://' | sed 's/:$//')\nif [ \"$verbose\" ]; then \n  echo \"libs=$libs\" >&2\nfi\n\n# patch each binary file found in the virtual environment\n# shellcheck disable=SC2156\necho \"Searching for files to patch in $VENV_PATH\" >&2\nif [ \"$deep\" ]; then\n  echo \"Deep search for binary files\" >&2\n  # For context, see #19\n  binary_files=$(find \"$(realpath \"$VENV_PATH\")\" -type f -exec sh -c \"file -i '{}' | grep -qE 'application/x-(executable|sharedlib); charset=binary'\" \\; -print)\nelse\n  echo \"Fast search for binary files\" >&2\n  binary_files=$(find \"$(realpath \"$VENV_PATH\")\" -type f -executable -exec sh -c \"file -i '{}' | grep -qE 'x-(.*); charset=binary'\" \\; -print)\nfi\nn_binary_files=$(wc -l <<< \"$binary_files\")\necho \"Found $n_binary_files binary files\" >&2\n\ncat <<< \"$binary_files\" \\\n  | while read -r file\n    do\n      echo \"Patching file: $file\" >&2\n      old_rpath=\"$(patchelf --print-rpath \"$file\" || true)\"\n      # prevent duplicates\n      new_rpath=\"$(echo \"$libs:$old_rpath\"  | sed 's/:$//' | tr ':' '\\n' | sort --unique | tr '\\n' ':' | sed 's/^://' | sed 's/:$/\\n/')\"\n      patchelf --set-rpath \"$new_rpath\" \"$file\" || true\n      old_interpreter=$(patchelf --print-interpreter \"$file\" || true)\n      if [ -n \"$old_interpreter\" ]; then\n        interpreter_name=\"$(basename \"$old_interpreter\")\"\n        new_interpreter=\"$(echo \"$new_rpath\" | tr ':' '\\n' | xargs -I {} find {} -name \"$interpreter_name\" | head -1)\"\n        patchelf --set-interpreter \"$new_interpreter\" \"$file\" || true\n      fi\n      echo >&2\n    done\n\n# `libtorch_global_deps.so` depends on libstdc++ but does not properly declare it, fix it manually see\n# https://github.com/eth-sri/lmql/blob/main/scripts/flake.d/overrides.nix#L28-L40\nif [ \"$enable_torch\" ]; then\n    torch_files=$(find \"$(realpath \"$VENV_PATH\")\" -name libtorch_global_deps.so)\n    cat <<< \"$torch_files\" \\\n      | while read -r file\n        do\n          echo \"Patching torch file: $file\" >&2\n          patchelf $file --add-needed libstdc++.so\n        done\nfi\n"
  },
  {
    "path": "flake.nix",
    "content": "{\n  outputs = { self, flake-utils, nixpkgs, ...}:\n    flake-utils.lib.eachDefaultSystem (system:\n      let\n        pkgs = nixpkgs.legacyPackages.${system};\n      in\n        {\n          packages.default =\n            pkgs.stdenv.mkDerivation {\n              name = \"fix-python\";\n              src = self;\n              phases = [ \"unpackPhase\" \"installPhase\" ];\n              installPhase = ''\n                mkdir -p $out/bin\n                cp $src/fix-python $out/bin\n              '';\n            };\n        }\n    );\n}\n"
  }
]