Repository: erikarvstedt/extra-container
Branch: master
Commit: b450bdb24fca
Files: 15
Total size: 72.5 KB

Directory structure:
gitextract_dnnhq23t/

├── CHANGELOG.md
├── LICENSE
├── Makefile
├── README.md
├── default.nix
├── eval-config.nix
├── examples/
│   └── flake/
│       ├── flake.nix
│       └── usage.sh
├── extra-container
├── flake.nix
├── run-tests-in-container.sh
├── test.sh
└── util/
    ├── generate-sudoers.rb
    ├── install.sh
    └── update-readme.rb

================================================
FILE CONTENTS
================================================

================================================
FILE: CHANGELOG.md
================================================
# 0.14 (2025-12-19)
- Enhancements
  - Support NixOS unstable
# 0.13 (2024-12-12)
- Enhancements
  - Support NixOS option `containers.<name>.autoStart`.
    This allows creating containers that are automatically started on system boot.
  - Support NixOS 24.11, NixOS unstable
# 0.12 (2023-06-15)
- Enhancements
  - Flake: Allow accessing built container configs.
    Example:
    ```bash
    nix eval ./examples/flake --apply 'sys: sys.containers.demo.config.networking.hostName'
    ```
  - Add compatibility with NixOS 23.05, NixOS unstable
- Fixes
  - Fix `extra-container destroy --all` when more than one container is installed
# 0.11 (2022-10-22)
- Enhancements
  - Support building containers via Flakes (see [examples/flake](./examples/flake)).
  - Support destroying containers from container definitions:\
    If you have installed containers with command `extra-container create ./mycontainers.nix`,
    you can now destroy these containers with the analogous command
    `extra-container destroy ./mycontainers.nix`.
- Fixes
  - Fix incomplete container state directory path in `help()` message
# 0.10 (2022-06-26)
- Enhancements
  - Support NixOS 22.05
# 0.9 (2022-04-11)
- Enhancements
  - Support NixOS unstable
- Fixes
  - Fix command `destroy` for nested declarative containers
# 0.8 (2021-09-30)
- Enhancements
  - Support NixOS unstable
- Fixes
  - Fix flake
# 0.7 (2021-08-03)
- Enhancements
  - Support NixOS 21.05 and unstable
  - Add basic [Nix flake](https://nixos.wiki/wiki/Flakes) support
    for installing and developing.\
    `extra-container` itself still uses `nix-build` internally.
# 0.6 (2021-02-05)
- Fixes
  - Add compatibility with current NixOS unstable.
  - `extra.exposeLocalhost`: don't fail when iptables lock can't be obtained immediately.
  - Fix `PATH` not being preserved in container shells.
# 0.5 (2020-11-01)
- Enhancements. (See the [README](README.md) for full documentation.)
  - Add generic support for systemd-based Linux distros.
  - Add command `shell`.
  - Add extra container options:\
    `extra.enableWAN`\
    `extra.exposeLocalhost`\
    `extra.firewallAllowHost`\
    See [eval-config.nix](eval-config.nix) for descriptions.
  - Add option `--ssh`.
  - Add option `--expr|-E`.
  - Append `pwd` to `NIX_PATH` to allow accessing the working dir in non-file configs.
  - Support nixpkgs versions > 20.03.
  - Automatically run as root via `sudo`.
- Fixes
  - Don't copy local nixpkgs sources provided via `--nixpkgs` to the nix store.

# 0.4 (2020-09-25)
- Enhancements
  - Significantly speed up container evaluation.\
    Use a reduced module set for evaluating the container host system derivation.
  - Speed up container destruction.\
    Kill the container process instead of a clean shutdown.


================================================
FILE: LICENSE
================================================
MIT License

Copyright (c) 2018 The extra-container developers

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

================================================
FILE: Makefile
================================================
all: test

test: runTests checkFlake

runTests:
	nix shell -c sudo ./run-tests-in-container.sh

checkFlake:
	nix flake check

doc:
	nix run .#updateReadme

.PHONY: runTests checkFlake test doc


================================================
FILE: README.md
================================================
# extra-container

Manage declarative NixOS containers like imperative containers, without system
rebuilds.

Each declarative container adds a full system module evaluation to every NixOS rebuild,
which can be prohibitively slow for systems with many containers or when experimenting
with single containers.

On the other hand, the faster imperative containers lack the full range of options of declarative containers.
This tool brings you the best of both worlds.

## Example

```bash

sudo extra-container create --start <<'EOF'
{
  containers.demo = {
    privateNetwork = true;
    hostAddress = "10.250.0.1";
    localAddress = "10.250.0.2";

    config = { pkgs, ... }: {
      systemd.services.hello = {
        wantedBy = [ "multi-user.target" ];
        script = ''
          while true; do
            echo hello | ${pkgs.netcat}/bin/nc -lN 50
          done
        '';
      };
      networking.firewall.allowedTCPPorts = [ 50 ];
    };
  };
}
EOF

curl --http0.9 10.250.0.2:50 # Returns 'hello' from the container

# Now change the 'hello' string in the container definition to something
# else and re-run the `extra-container create --start` command.
# The container is automatically updated via NixOS' `switch-to-configuration`.

# The container is a regular container that can be controlled
# with nixos-container
nixos-container status demo

# Remove the container
sudo extra-container destroy demo
```

#### Run command in a container and exit

```bash
cfg='{
  containers.demo.config = {
    networking.hostName = "hello";
  };
}'
extra-container shell -E "$cfg" --run c hostname # => hello
```

## Changelog

 [`CHANGELOG.md`](CHANGELOG.md)

## Install

### On NixOS

##### NixOS ≥ 21.11

Add `programs.extra-container.enable = true` to your configuration.

##### Any NixOS with `flake` support
Import `extra-container.nixosModules.default` in your configuration.

### On other systemd-based Linux distros

```bash
git clone https://github.com/erikarvstedt/extra-container
# Calls sudo during install
extra-container/util/install.sh
```
[`install.sh`](util/install.sh) installs `extra-container` to the root nix user profile
and edits `/etc/sudoers` to enable running `extra-container` with sudo.

## More features

### Shell

Command `shell` starts a container shell session.
The shell provides helper functions for interacting with the container. The container is destroyed when exiting the shell.

This config uses `extra` options that are [explained below](#private-network-helper).
```bash
read -d '' src <<'EOF' || :
{
  containers.demo = {
    extra.addressPrefix = "10.250.0"; # Sets up a private network.
    extra.enableWAN = true;
  };
}
EOF
# Provide container config via `-E` instead of stdin because the shell's stdin
# should be connected to the terminal.
extra-container shell -E "$src" --ssh
```

`extra-container` automatically runs itself via `sudo` when called as a non-root user.

An example shell session
```
...
Starting shell.
Enter "h" for documentation.

$ h
Container address: 10.250.0.2 ($ip)
Container filesystem: /var/lib/nixos-containers/demo

Run "c COMMAND" to execute a command in the container
Run "c" to start a shell session inside the container
Run "cssh" for SSH

# Container internet access, enabled via option `extra.enableWAN`
$ c curl example.com
<!doctype html>
<html>
...

# Connect with SSH, enabled by `--ssh`
$ cssh hostname
demo
```

#### Run commands

Run a command in a shell session and exit. The container is destroyed afterwards.
```bash
cfg='{ containers.demo = {}; }'
extra-container shell -E "$cfg" --run c hostname
# => demo
```

Start a shell inside the container.
```bash
cfg='{ containers.demo = {}; }'
extra-container shell -E "$cfg" --run c
```

#### Repeated calls to `extra-container shell`
When `extra-container shell` detects that it is already running in a container shell
session, it updates the running container instead of destroying and restarting it and
starting a new shell.\
To prevent `sudo` from clearing the environment variables that are needed for shell
detection, call `extra-container` without `sudo`.\
`extra-container` will automatically run itself via `sudo` only when it is first
called as a non-root user outside of a shell session.

To force container destruction inside a shell session, use `extra-container shell --destroy|-d`.

#### Disable auto-destruction
By default, `shell` destroys the shell container before starting and before exiting.
This ensures that containers start with no leftover filesystem state from
previous runs and that containers do not consume system resources after use.\
To disable auto-destructing containers, run
`extra-container shell --no-destroy|-n`


### Private network helper

Container options `extra.*` are defined by `extra-container` and help with setting up private network containers.\
See [eval-config.nix](./eval-config.nix) for full option descriptions.
```nix
containers.demo = {
  extra = {
    # Sets
    # privateNetwork = true
    # hostAddress = "${addressPrefix}.1"
    # localAddress = "${addressPrefix}.2"
    addressPrefix = "10.250.0";

    # Enable internet access for the container
    enableWAN = true;
    # Always allow connections from hostAddress
    firewallAllowHost = true;
    # Make the container's localhost reachable via localAddress
    exposeLocalhost = true;
  }
};
```

### Access working dir in non-file configs

`extra-container` appends `pwd` to `NIX_PATH` to allow configs given via `--expr|-E`
or via stdin to access the working directory.
```bash
extra-container create -E '{ imports = [ <pwd/myfile.nix> ]; ... }'
```

## Define containers via Flakes

See [examples/flake](./examples/flake).

## Usage
```
extra-container create <container-config-file>
                       [--attr|-A attrPath]
                       [--nixpkgs-path|--nixos-path path]
                       [--start|-s | --restart-changed|-r]
                       [--ssh]
                       [--build-args arg...]

    <container-config-file> is a NixOS config file with container
    definitions like 'containers.mycontainer = { ... }'

    --attr | -A attrPath
      Select an attribute from the config expression

    --nixpkgs-path
      A nix expression that returns a path to the nixpkgs source
      to use for building the containers

    --nixos-path
      Like '--nixpkgs-path', but for directly specifying the NixOS source

    --start | -s
      Start all created containers
      Update running containers that have changed or restart them if '--restart-changed' was specified

    --update-changed | -u
      Update running containers with a changed system configuration by running
      'switch-to-configuration' inside the container.
      Restart containers with a changed container configuration

    --restart-changed | -r
      Restart running containers that have changed

    --ssh
      Generate SSH keys in /tmp and enable container SSH access.
      The key files remain after exit and are reused on subsequent runs.
      Unlocks the function 'cssh' in 'extra-container shell'.
      Requires container option 'privateNetwork = true'.

    --build-args arg...
      All following args are passed to nix-build.

    Example:
      extra-container create mycontainers.nix --restart-changed

      extra-container create mycontainers.nix --nixpkgs-path \
        'fetchTarball https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz'

      extra-container create mycontainers.nix --start --build-args --builders 'ssh://worker - - 8'

echo <container-config> | extra-container create
    Read the container config from stdin

    Example:
      extra-container create --start <<EOF
        { containers.hello = { enableTun = true; config = {}; }; }
      EOF

extra-container create --expr|-E <container-config>
    Provide container config as an argument

extra-container create <store-path>
    Create containers from <store-path>/etc

    Examples:
      Create from nixos system derivation
      extra-container create /nix/store/9h..27-nixos-system-foo-18.03

      Create from nixos etc derivation
      extra-container create /nix/store/32..9j-etc

extra-container shell ...
    Start a container shell session.
    See the README for a complete documentation.
    Supports all arguments from 'create'

    Extra arguments:
      --run <cmd> <arg>...
        Run command in shell session and exit
        Must be the last option given
      --no-destroy|-n
        Do not destroy shell container before and after running
      --destroy|-d
        If running inside an existing shell session, force container to
        be destroyed before and after running

    Example:
      extra-container shell -E '{ containers.demo.config = {}; }'

extra-container build ...
    Build the container config and print the resulting NixOS system etc path

    This command can be used like 'create', but options related
    to starting are not supported

extra-container list
    List all extra containers

extra-container restart <container>...
    Fixes the broken restart command of nixos-container (nixpkgs issue #43652)

extra-container destroy <container-name>...
    Destroy containers

extra-container destroy <args for create/shell>...
    Destroy the containers defined by the args for command `create` or `shell` (see above).
    For this to work, the first arg after `destroy` must start with one of the
    following three characters: ./-

    Example:
      extra-container destroy ./containers.nix

extra-container destroy --all|-a
    Destroy all extra containers

extra-container <cmd> <arg>...
    All other commands are forwarded to nixos-container
```

## Implementation

The script works like this: Take a NixOS config with container definitions and build
the system's `config.system.build.etc` derivation. Because we're not building a full
system we can use a reduced module set (`eval-config.nix`) to improve evaluation
performance.

Now link the container files from the etc derivation to the main system, like so:
```
nixos-system/etc/systemd/system/container@CONTAINER.service -> /etc/systemd-mutable/system
nixos-system/etc/containers/CONTAINER.conf -> /etc/containers       (system.stateVersion < 22.05)
                                           -> /etc/nixos-containers (system.stateVersion ≥ 22.05)
```
Finally, add gcroots pointing to the linked files.


## Developing
All contributions and suggestions are welcome, even if they're minor or cosmetic.

### Development workflow

Run `nix develop` in the project root directory to start a development shell.\
Within the shell, you can run extra-container from the [local
source](./extra-container) via command `extra-container`.

When changing the `Usage` documentation in `extra-container`, run `make doc` to copy
these changes to `README.md`.

#### Tests

Run `make` to run the tests.\
The following tests are executed:

- Main test suite

  Can be run manually via `sudo run-tests-in-container.sh`.\
  This script creates a temporary container named `test-extra-container` in which
  the main test script [`test.sh`](./test.sh) is run.\
  `test.sh` adds and removes temporary containers named `test-*` on the host system.
  It can also be called directly, but wrapping it with `run-tests-in-container.sh`
  helps reducing interference with your main system.

- VM test

  Can be run manually via `nix build .#test`.\
  This is a basic test using the [NixOS VM test
  framework](https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/testing-python.nix).
  It is built as a Nix derivation, which makes it independent from the system
  environment.

  For debugging the VM, run `nix run .#debugTest` to start a Python test driver shell
  inside the VM.

- `nix flake check`

  Evaluates all flake outputs and builds the VM test.

#### VM

Run `nix run .#vm` to start a VM where `extra-container` is installed.\
This provides an isolated and reproducible testing environment.


================================================
FILE: default.nix
================================================
{ stdenv, lib, nixos-container, openssh, glibcLocales
, pkgSrc ? lib.cleanSource ./.
}:

stdenv.mkDerivation rec {
  pname = "extra-container";
  version = "0.14";

  src = pkgSrc;

  buildCommand = ''
    install -D $src/extra-container $out/bin/extra-container
    patchShebangs $out/bin
    share=$out/share/extra-container
    install $src/eval-config.nix -Dt $share

    # Use existing PATH for systemctl and machinectl
    scriptPath="export PATH=${lib.makeBinPath [ openssh ]}:\$PATH"

    sed -i "
      s|evalConfig=.*|evalConfig=$share/eval-config.nix|
      s|LOCALE_ARCHIVE=.*|LOCALE_ARCHIVE=${glibcLocales}/lib/locale/locale-archive|
      2i$scriptPath
      2inixosContainer=${nixos-container}/bin
    " $out/bin/extra-container
  '';

  meta = with lib; {
    description = "Run declarative containers without full system rebuilds";
    homepage = "https://github.com/erikarvstedt/extra-container";
    changelog = "https://github.com/erikarvstedt/extra-container/blob/master/CHANGELOG.md";
    license = licenses.mit;
    platforms = platforms.linux;
    maintainers = [ maintainers.erikarvstedt ];
  };
}


================================================
FILE: eval-config.nix
================================================
{ nixosPath
, systemConfig
, legacyInstallDirs
, system ? builtins.currentSystem
}:

let
  # A minimal module set for evaluating container configs.
  # This significantly reduces extra-container evaluation overhead (total eval time - container eval time)
  # Compatible with nixpkgs >= 16.09
  baseModules = [
    (nixosPath + "/modules/misc/assertions.nix")
    (nixosPath + "/modules/misc/nixpkgs.nix")
    (nixosPath + "/modules/misc/extra-arguments.nix")
    (nixosPath + "/modules/system/activation/top-level.nix")
    (nixosPath + "/modules/system/etc/etc.nix")
    (nixosPath + "/modules/system/boot/systemd.nix")
    nixosContainerModule
    dummyOptions
  ];

  nixosContainerModule = let
    new = nixosPath + "/modules/virtualisation/nixos-containers.nix";
    old = nixosPath + "/modules/virtualisation/containers.nix"; # For nixpkgs < 20.09)
  in
    if builtins.pathExists new then new else old;

  dummyOptions = { pkgs, lib, options, ... }: let
    optionValue = default: lib.mkOption { inherit default; };
    dummy = optionValue [];
  in {
    options = {
      boot.kernel.sysctl = dummy;
      boot.kernelModules = dummy;
      boot.kernelPackages.kernel.version = optionValue "";
      boot.kernelParams = dummy;
      boot.loader.systemd-boot.bootCounting.enable = optionValue false;
      environment.systemPackages = dummy;
      networking.dhcpcd.denyInterfaces = dummy;
      networking.hosts = dummy;
      networking.extraHosts = dummy;
      networking.proxy.envVars = optionValue {};
      nix.package = optionValue pkgs.nix;
      security = dummy;
      services = {
        dbus = dummy;
        logrotate = dummy;
        udev = dummy;
        rsyslogd.enable = optionValue false;
        syslog-ng.enable = optionValue false;
      };
      system.activationScripts = dummy;
      system.fsPackages = dummy;
      system.nssDatabases = dummy;
      system.nssModules = dummy;
      system.path = optionValue "";
      system.requiredKernelConfig = dummy;
      system.stateVersion = optionValue (if legacyInstallDirs then "21.11" else "22.05");
      systemd.oomd = dummy;
      systemd.user.generators = optionValue {};
      ids.gids.keys = dummy;
      ids.uids.systemd-coredump = dummy;
      ids.gids.systemd-journal = dummy;
      ids.gids.systemd-journal-gateway = dummy;
      ids.uids.systemd-journal-gateway = dummy;
      ids.gids.systemd-network = dummy;
      ids.uids.systemd-network = dummy;
      ids.uids.systemd-resolve = dummy;
      ids.gids.systemd-resolve = dummy;
      users.users.systemd-coredump = dummy;
      users.users.systemd-network.group = dummy;
      users.users.systemd-network.uid = dummy;
      users.users.systemd-resolve.group = dummy;
      users.users.systemd-resolve.uid = dummy;
      users.users.systemd-journal-gateway.group = dummy;
      users.users.systemd-journal-gateway.uid = dummy;
      users.groups.systemd-coredump = dummy;
      users.groups.systemd-network.gid = dummy;
      users.groups.systemd-resolve.gid = dummy;
      users.groups.keys.gid = dummy;
      users.groups.systemd-journal.gid = dummy;
      users.groups.systemd-journal-gateway.gid = dummy;
    };

    config = {
      systemd.timers = lib.mkForce {};
      systemd.targets = lib.mkForce {};
    } // lib.optionalAttrs (options.systemd ? managerEnvironment) {
      systemd.managerEnvironment = lib.mkForce {};
    };
  };

  containerAssert = cond: name: msg: value:
    if cond then value
    else throw "container '${name}': ${msg}'";

  assertNonNull = var:
    containerAssert (var != null);

  extraModule = { config, pkgs, lib, ... }: with lib; {
    options = {
      containers = mkOption {
        type = types.attrsOf (types.submodule (
          { config, name, ... }: {
            options = {
              extra = {
                addressPrefix = mkOption {
                  type = with types; nullOr str;
                  default = null;
                  description = ''
                    Enable privateNetwork and set
                    hostAddress = <addressPrefix>.1
                    localAddress = <addressPrefix>.2
                  '';
                };
                enableWAN = mkOption {
                  type = types.bool;
                  default = false;
                  description = ''
                    Enable WAN access inside the container by rewriting container traffic
                    to use the host's address (NAT).

                    Only active when privateNetwork == true.
                  '';
                };
                exposeLocalhost = mkOption {
                  type = types.bool;
                  default = false;
                  description = ''
                    Forward requests from the container's external interface
                    to the container's localhost.
                    Useful to test internal services from outside the container.

                    WARNING: This exposes the container's localhost to all users.
                    Only use in a trusted environment.

                    Only active when privateNetwork == true.
                  '';
                };
                firewallAllowHost = mkOption {
                  type = types.bool;
                  default = false;
                  description = ''
                    Always allow connections from the container host.

                    Only active when privateNetwork == true.
                  '';
                };
                enableSSH = mkOption {
                  type = types.bool;
                  default = builtins.getEnv("extraContainerSSH") == "1";
                  description = ''
                    Enable SSH access with an automatically generated key.
                    This enables the 'cssh' comand in extra-container shell.

                    Requires privateNetwork == true.
                  '';
                };
              };
            };

            config = mkMerge [
              (
                let
                  prefix = config.extra.addressPrefix;
                in mkIf (prefix != null) {
                  privateNetwork = true;
                  hostAddress = "${prefix}.1";
                  localAddress = "${prefix}.2";
                }
              )
              {
                config = ({ pkgs, ... }@moduleArgs: mkMerge [
                  {
                    systemd.services.forward-to-localhost = mkIf (config.extra.exposeLocalhost && config.privateNetwork) {
                      wantedBy = [ "network.target" ];
                      script = assertNonNull config.localAddress name
                        ''
                          option extra.exposeLocalhost requires localAddress to be non-null.
                        ''
                        ''
                          ${pkgs.procps}/bin/sysctl -w net.ipv4.conf.all.route_localnet=1
                          ${pkgs.iptables}/bin/iptables -w -t nat -I PREROUTING -p tcp \
                            -d ${config.localAddress} ! --dport 80 -j DNAT --to-destination 127.0.0.1
                        '';
                    };
                    networking.firewall.extraCommands = mkIf (config.extra.firewallAllowHost && config.privateNetwork) (
                      assertNonNull config.hostAddress name
                        ''
                          option extra.exposeLocalhost requires hostAddress to be non-null.
                        ''
                        ''
                          iptables -w -A nixos-fw -s ${config.hostAddress} -j ACCEPT
                        ''
                    );
                    # Silence system state warning
                    system.stateVersion = lib.mkDefault moduleArgs.config.system.nixos.release;
                  }
                  (mkIf config.extra.enableSSH {
                     services.openssh.enable = containerAssert config.privateNetwork name ''
                       option extra.enableSSH requires privateNetwork to be enabled.
                     '' true;
                     users.users.root.openssh.authorizedKeys.keyFiles = [
                       /tmp/extra-container-ssh/key.pub
                     ];
                  })
                ]);
              }
            ];
          }
        ));
      };
    };

    config = {
      systemd.services = let
        WANContainers = builtins.filter (c:
                          let cfg = config.containers.${c};
                          in cfg.privateNetwork && cfg.extra.enableWAN
                        ) (builtins.attrNames config.containers);
        iptables = "${pkgs.iptables}/bin/iptables";
        serviceCfg = c: let
          containerAddress = config.containers.${c}.localAddress;
        in
          assertNonNull containerAddress c
          ''
            option extra.enableWAN requires localAddress to be non-null
          ''
          {
            preStart = "${iptables} -w -t nat -A POSTROUTING -s ${containerAddress} -j MASQUERADE";
            postStop = "${iptables} -w -t nat -D POSTROUTING -s ${containerAddress} -j MASQUERADE || true";
          };
      in
        listToAttrs (map (c: nameValuePair "container@${c}" (serviceCfg c)) WANContainers);
    };
  };
in
import (nixosPath + "/lib/eval-config.nix") {
  inherit baseModules system;
  modules = [ extraModule systemConfig ];
}


================================================
FILE: examples/flake/flake.nix
================================================
# See how this flake is used in ./usage.sh
{
  inputs.extra-container.url = "github:erikarvstedt/extra-container";
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

  outputs = { extra-container, ... }@inputs:
    extra-container.lib.eachSupportedSystem (system: {
      packages.default = extra-container.lib.buildContainers {
        # The system of the container host
        inherit system;

        # Only set this if the `system.stateVersion` of your container
        # host is < 22.05
        # legacyInstallDirs = true;

        # Optional: Set nixpkgs.
        # If unset, the nixpkgs input of extra-container flake is used
        nixpkgs = inputs.nixpkgs;

        # Set this to disable `nix run` support
        # addRunner = false;

        config = {
          containers.demo = {
            extra.addressPrefix = "10.250.0";

            # `specialArgs` is available in nixpkgs > 22.11
            # This is useful for importing flakes from modules (see nixpkgs/lib/modules.nix).
            # specialArgs = { inherit inputs; };

            config = { pkgs, ... }: {
              systemd.services.hello = {
                wantedBy = [ "multi-user.target" ];
                script = ''
                  while true; do
                    echo hello | ${pkgs.netcat}/bin/nc -lN 50
                  done
                '';
              };
              networking.firewall.allowedTCPPorts = [ 50 ];
            };
          };
        };
      };
    });
}


================================================
FILE: examples/flake/usage.sh
================================================
# Usage via `nix run`

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
# Container lifecycle

# Create and start container defined by ./flake.nix
nix run . -- create --start
# You can use the same command to update the (running) container,
# after changing the container configuration.
#
# The arguments after `--` are passed to the `extra-container` binary in PATH,
# while the flake is used for the container definitions.

# Use `nixos-container` to control the running container
sudo nixos-container run demo -- hostname
sudo nixos-container root-login demo

# Destroy container
nix run . -- destroy

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
# Container shell
# Start an interactive shell in an ephemeral container
nix run . -- shell
nix run . # equivalent, because `shell` is used as the default command

# Run a single command in the container.
# The container is destroyed afterwards.
nix run . -- --run c hostname
nix run . -- shell --run c hostname # equivalent
nix run . -- --run bash -c 'curl --http0.9 $ip:50'

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
# Usage via `nix build`
# 1. Build container
nix build . --out-link /tmp/container
# 2. Run container
extra-container shell /tmp/container

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
# Inspect container configs
nix eval . --apply 'sys: sys.containers.demo.config.networking.hostName'


================================================
FILE: extra-container
================================================
#!/usr/bin/env bash

set -eo pipefail
shopt -s nullglob

# This script uses systemctl, machinectl and nsenter from the existing environment

showHelp() {
    echo "Usage:"
    echo
    echo "extra-container create <container-config-file>"
    echo "                       [--attr|-A attrPath]"
    echo "                       [--nixpkgs-path|--nixos-path path]"
    echo "                       [--start|-s | --restart-changed|-r]"
    echo "                       [--ssh]"
    echo "                       [--build-args arg...]"
    echo
    echo "    <container-config-file> is a NixOS config file with container"
    echo "    definitions like 'containers.mycontainer = { ... }'"
    echo
    echo "    --attr | -A attrPath"
    echo "      Select an attribute from the config expression"
    echo
    echo "    --nixpkgs-path"
    echo "      A nix expression that returns a path to the nixpkgs source"
    echo "      to use for building the containers"
    echo
    echo "    --nixos-path"
    echo "      Like '--nixpkgs-path', but for directly specifying the NixOS source"
    echo
    echo "    --start | -s"
    echo "      Start all created containers"
    echo "      Update running containers that have changed or restart them if '--restart-changed' was specified"
    echo
    echo "    --update-changed | -u"
    echo "      Update running containers with a changed system configuration by running"
    echo "      'switch-to-configuration' inside the container."
    echo "      Restart containers with a changed container configuration"
    echo
    echo "    --restart-changed | -r"
    echo "      Restart running containers that have changed"
    echo
    echo "    --ssh"
    echo "      Generate SSH keys in /tmp and enable container SSH access."
    echo "      The key files remain after exit and are reused on subsequent runs."
    echo "      Unlocks the function 'cssh' in 'extra-container shell'."
    echo "      Requires container option 'privateNetwork = true'."
    echo
    echo "    --build-args arg..."
    echo "      All following args are passed to nix-build."
    echo
    echo "    Example:"
    echo "      extra-container create mycontainers.nix --restart-changed"
    echo
    echo "      extra-container create mycontainers.nix --nixpkgs-path \\"
    echo "        'fetchTarball https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz'"
    echo
    echo "      extra-container create mycontainers.nix --start --build-args --builders 'ssh://worker - - 8'"
    echo
    echo "echo <container-config> | extra-container create"
    echo "    Read the container config from stdin"
    echo
    echo "    Example:"
    echo "      extra-container create --start <<EOF"
    echo "        { containers.hello = { enableTun = true; config = {}; }; }"
    echo "      EOF"
    echo
    echo "extra-container create --expr|-E <container-config>"
    echo "    Provide container config as an argument"
    echo
    echo "extra-container create <store-path>"
    echo "    Create containers from <store-path>/etc"
    echo
    echo "    Examples:"
    echo "      Create from nixos system derivation"
    echo "      extra-container create /nix/store/9h..27-nixos-system-foo-18.03"
    echo
    echo "      Create from nixos etc derivation"
    echo "      extra-container create /nix/store/32..9j-etc"
    echo
    echo "extra-container shell ..."
    echo "    Start a container shell session."
    echo "    See the README for a complete documentation."
    echo "    Supports all arguments from 'create'"
    echo
    echo "    Extra arguments:"
    echo "      --run <cmd> <arg>..."
    echo "        Run command in shell session and exit"
    echo "        Must be the last option given"
    echo "      --no-destroy|-n"
    echo "        Do not destroy shell container before and after running"
    echo "      --destroy|-d"
    echo "        If running inside an existing shell session, force container to"
    echo "        be destroyed before and after running"
    echo
    echo "    Example:"
    echo "      extra-container shell -E '{ containers.demo.config = {}; }'"
    echo
    echo "extra-container build ..."
    echo "    Build the container config and print the resulting NixOS system etc path"
    echo
    echo "    This command can be used like 'create', but options related"
    echo "    to starting are not supported"
    echo
    echo "extra-container list"
    echo "    List all extra containers"
    echo
    echo "extra-container restart <container>..."
    echo "    Fixes the broken restart command of nixos-container (nixpkgs issue #43652)"
    echo
    echo "extra-container destroy <container-name>..."
    echo "    Destroy containers"
    echo
    echo "extra-container destroy <args for create/shell>..."
    echo "    Destroy the containers defined by the args for command \`create\` or \`shell\` (see above)."
    echo "    For this to work, the first arg after \`destroy\` must start with one of the"
    echo "    following three characters: ./-"
    echo
    echo "    Example:"
    echo "      extra-container destroy ./containers.nix"
    echo
    echo "extra-container destroy --all|-a"
    echo "    Destroy all extra containers"
    echo
    echo "extra-container <cmd> <arg>..."
    echo "    All other commands are forwarded to nixos-container"
    exit 0
}

case $1 in
    help|-h|--help)
        showHelp
        ;;
esac

# If a container build output is given ($EXTRA_CONTAINER_ETC) and no command is
# specified in $1, use `shell` as the default command
if [[ $EXTRA_CONTAINER_ETC && ($# == 0 || $1 == -*) ]]; then
    set -- shell "$@"
elif [[ $# == 0 ]]; then
     showHelp
fi

# Run as root if needed
#------------------------------------------------------------------------------

if [[ $EUID != 0 ]]; then
    exec sudo PATH="$PATH" NIX_PATH="$NIX_PATH" EXTRA_CONTAINER_ETC="$EXTRA_CONTAINER_ETC" "${BASH_SOURCE[0]}" "$@"
fi

# Operating system specific setup
#------------------------------------------------------------------------------

errEcho() {
    >&2 echo "$@"
}

[[ -e /run/booted-system/nixos-version ]] && isNixOS=1 || isNixOS=

if [[ $isNixOS ]]; then
    mutableServicesDir=/etc/systemd-mutable/system
else
    # This is the canonical way to check for systemd
    # https://www.freedesktop.org/software/systemd/man/sd_booted.html
    if [[ ! -e /run/systemd/system ]]; then
        errEcho "extra-container requires systemd"
        exit 1
    fi
    if ! type -P machinectl > /dev/null; then
        errEcho "extra-container requires machinectl to be installed"
        exit 1
    fi
    if [[ ! -e /nix/var/nix/profiles/default ]]; then
        errEcho "extra-container requires a multi-user nix installation"
        exit 1
    fi

    mutableServicesDir=/usr/lib/systemd/system

    # For nixos-container on non-NixOS systems https://github.com/NixOS/nix/issues/599#issuecomment-153885553
    # The value of 'LOCALE_ARCHIVE' is inserted by the builder (./default.nix)
    if [[ ! -v LOCALE_ARCHIVE ]]; then
        export LOCALE_ARCHIVE=
    fi

    # Work around https://github.com/NixOS/nixpkgs/issues/28833
    createOsReleaseFile() {
        if [[ ! -f /etc/static/os-release ]]; then
            mkdir -p /etc/static
            echo "# added by extra-container" > /etc/static/os-release
        fi
    }
fi

# Use existing `nixos-container` in PATH by default because on NixOS the container
# install location differs depending on `system.stateVersion` and is hardcoded in
# the `nixos-container` binary.
# See also: function `getInstallDirs`.
if ! type -P nixos-container > /dev/null; then
    PATH=$nixosContainer:$PATH
fi

# Parse and run command
#------------------------------------------------------------------------------

trap 'echo "Error at ${BASH_SOURCE[0]}:$LINENO"' ERR

tmpDir=
onlyBuild=
shell=
list=
restart=
destroy=

case $1 in
    create|add)
        shift
        ;;
    build)
        onlyBuild=1
        shift
        ;;
    shell)
        shell=1
        shift
        ;;
    list)
        list=1
        ;;
    restart)
        restart=1
        shift
        ;;
    destroy)
        destroy=1
        shift
        ;;
    *)
        exec nixos-container "$@"
        ;;
esac

# Value is replaced by builder (./default.nix)
evalConfig=$(cd "${BASH_SOURCE[0]%/*}" && pwd)/eval-config.nix

buildContainers() {
    local containerCfg=$1
    local tmpDir=$2
    local nixosPath=$3

    local attrExpr=
    if [[ $attr ]]; then
        attrExpr=".\${''$attr''}"
    fi
    NIX_PATH=$NIX_PATH:pwd=$PWD nix-build --out-link $tmpDir/result "${buildArgs[@]}" -E \
    " let cfg = ($containerCfg)$attrExpr;
      in (import $evalConfig {
         nixosPath = $nixosPath;
         legacyInstallDirs = $legacyInstallDirs;
         systemConfig = cfg;
      }).config.system.build.etc
    " >/dev/null
}

restartContainers() {
    local services=$(getServiceNames $*)

    # `systemctl restart <container>` is broken (https://github.com/NixOS/nixpkgs/issues/43652),
    # so use a workaraound
    systemctl stop $services
    # Retry terminating the container machines until the command succeeds
    # or the machines have disappeared
    for ((i = 1;; i++)); do
        local failed=
        local output
        output=$(machinectl terminate $* 2>&1) || failed=1
        if [[ ! $failed || $output == *no*machine*known* ]]; then
            break
        fi
        echo $output
        if ((i == 20)); then
            errEcho "Failed to stop containers."
            exit 1
        fi
        sleep 0.001
    done
    systemctl start $services
}

updateContainers() {
    for container in $*; do
        local confFile=$configDirectory/$container.conf
        local systemPath=$(grep -ohP "(?<=^SYSTEM_PATH=).*" $confFile)
        # Shift output 2 spaces to the right
        echo "  Updating $container"
        nixos-container run $container -- bash -lc "${systemPath}/bin/switch-to-configuration test" |& sed 's/^/  /' || true
        echo
    done
}

getContainers() {
    for service in $mutableServicesDir/container@?*.service; do
        getContainerName $service
    done
}

getContainerName() {
    [[ $1 =~ container@(.+)\.service ]]
    echo ${BASH_REMATCH[1]}
}

getServiceNames() {
    for container in $*; do
        echo "container@$container.service "
    done
}

makeGCRootsPath() {
    # Use gcroots/auto instead of gcroots/per-user/root because
    # stale links in per-user are not automatically removed.
    # This avoids cluttering the gcroots dir in case containers
    # are manually removed.
    echo /nix/var/nix/gcroots/auto/extra-container-$1
}

# The key is reused between extra-container sessions
makeSSHKey() {
    local keyDir=/tmp/extra-container-ssh
    sshKey=$keyDir/key
    if [[ ! (-d $keyDir && $(stat -c '%a%U' $keyDir) == 700root &&
                 -f $sshKey && -f $sshKey.pub) ]]; then
        if [[ -e $keyDir ]]; then
            rm -rf $keyDir
        fi
        mkdir -m 700 $keyDir
        ssh-keygen -t ed25519 -P '' -C none -f $sshKey > /dev/null
    fi
    chmod 600 $sshKey
}

makeTmpDir() {
    if [[ ! $tmpDir ]]; then
        tmpDir=$(mktemp -d /tmp/extra-container.XXX)
    fi
}

configDirectory=
getInstallDirs() {
    if [[ ! $configDirectory ]]; then
        nixosContainer=$(type -p nixos-container)
        if grep -q '"/etc/nixos-containers"' "$nixosContainer"; then
            # NixOS ≥22.05
            configDirectory=/etc/nixos-containers
            configDirectoryName=nixos-containers
            otherConfigDirectoryName=containers
            stateDirectory=/var/lib/nixos-containers
            legacyInstallDirs=false
        else
            # NixOS <22.05
            configDirectory=/etc/containers
            configDirectoryName=containers
            otherConfigDirectoryName=nixos-containers
            stateDirectory=/var/lib/containers
            legacyInstallDirs=true
        fi
    fi
}

# This fn is used by commands `create`, `shell`, `destroy`.
# To simplify the implementation, this fn includes the whole arg parsing for commands
# `create`, `shell`. These args are not by `destroy`.
getContainersFromDefinition() {
    start=
    update=
    restart=
    ssh=
    containerExpr=
    attr=
    nixpkgsPath=
    nixosPath=
    buildArgs=()
    runCommand=()

    destroy=1
    forceDestroy=

    args=()
    while [[ $# > 0 ]]; do
        arg="$1"
        shift
        case $arg in
            --start|-s)
                start=1
                ;;
            --update-changed|-u)
                update=1
                ;;
            --restart-changed|-r)
                restart=1
                ;;
            --ssh)
                ssh=1
                ;;
            --expr|-E)
                containerExpr="$1"
                shift
                ;;
            --attr|-A)
                attr="$1"
                shift
                ;;
            --nixpkgs-path)
                nixpkgsPath="$1"
                shift
                ;;
            --nixos-path)
                nixosPath="$1"
                shift
                ;;
            --build-args)
                buildArgs=()
                # Add all following args until --run
                while [[ $# > 0 && $1 != --run ]]; do
                    buildArgs+=("$1")
                    shift
                done
                ;;
            --run)
                runCommand=("$@")
                break
                ;;
            --no-destroy|-n)
                destroy=
                ;;
            --destroy|-d)
                destroy=1
                forceDestroy=1
                ;;
            *)
                args+=("$arg")
                ;;
        esac
    done
    set -- "${args[@]}"

    if [[ ! $containerExpr ]]; then
        arg=${1:-}
        shift || true
    fi
    if [[ $# > 0 ]]; then
        errEcho "Error: Unhandled arguments: $*"
        exit 1
    fi

    ## 1. Build containers if needed

    if [[ $ssh ]]; then
        makeSSHKey
    fi
    # This env var is read by ./eval-config.nix while building
    export extraContainerSSH=$ssh

    getInstallDirs

    needToBuildContainers() {
        if [[ $containerExpr ]]; then
            return 0;
        fi
        if [[ $arg ]]; then
            if [[ -f $arg || -e $arg/default.nix ]]; then
                return 0
            fi
        elif [[ ! $EXTRA_CONTAINER_ETC ]]; then
            return 0
        fi
        return 1
    }

    if needToBuildContainers; then
        if [[ ! $containerExpr ]]; then
            if [[ ! $arg || $arg == - ]]; then
                # Read container cfg from stdin
                if [[ $shell ]]; then
                    errEcho "Reading container config from STDIN is not supported for command 'shell'"
                    exit 1
                fi
                read -d '' containerExpr || true
            else
                containerExpr="import ''$(realpath "$arg")''"
            fi
        fi
        if [[ ! $nixosPath ]]; then
            if [[ $nixpkgsPath ]]; then
                nixosPath="\"\${toString ($nixpkgsPath)}/nixos\""
            else
                nixosPath="<nixpkgs/nixos>"
            fi
        fi
        makeTmpDir
        errEcho "Building containers..."
        buildContainers "$containerExpr" $tmpDir "$nixosPath"
        nixosSystemEtc=$tmpDir/result/etc

        if [[ $onlyBuild ]]; then
            realpath $tmpDir/result
            exit 0
        fi
    else
        if [[ $arg ]]; then
            EXTRA_CONTAINER_ETC=$arg
        fi
        if [[ ! $EXTRA_CONTAINER_ETC ]]; then
            errEcho "No containers specified"
            exit 1
        fi
        nixosSystemEtc="$EXTRA_CONTAINER_ETC/etc"
        if [[ ! -e $nixosSystemEtc ]]; then
            errEcho "$nixosSystemEtc doesn't exist"
            exit 1
        fi
    fi

    ## 2. Gather containers

    services=$(echo $nixosSystemEtc/systemd/system/container@?*.service)
    if [[ ! $services ]]; then
        errEcho "No container services in $nixosSystemEtc/systemd/system"
        exit 0
    fi
    allContainers=()
    for service in $services; do
        allContainers+=($(getContainerName $service))
    done
    allContainers=($(printf '%s\n' ${allContainers[@]} | sort))
}

atExit() {
    origExitCode=$?
    set +e
    if [[ $startShellEnv && $destroy ]]; then
        [[ $runCommand ]] || echo "Destroying container."
        destroyContainers $shellContainer
    fi
    if [[ $tmpDir ]]; then
        rm -rf $tmpDir
    fi
    exit $origExitCode
}
trap atExit EXIT

# Command 'list extra containers'
#------------------------------------------------------------------------------

if [[ $list ]]; then
    getContainers
    exit 0
fi

# Command 'restart container'
#------------------------------------------------------------------------------

if [[ $restart ]]; then
    restartContainers $*
    exit 0
fi

# Command 'destroy containers'
#------------------------------------------------------------------------------

systemctlIgnoreNotLoaded() {
    local failed=
    output=$(systemctl "$@" 2>&1) || failed=1
    if [[ $failed ]]; then
        if [[ $output == *not\ loaded* ]]; then
            unitLoaded=
        else
            return 1
        fi
    fi
}

destroyContainers() {
    getInstallDirs

    if [[ ! -e "$configDirectory" ]]; then
        # No containers installed. The containers dir is needed by the code below.
        return
    fi

    local needDaemonReload=

    for container in "$@"; do
        service=container@${container}.service
        serviceFile=${mutableServicesDir}/$service
        confFile="$configDirectory/$container.conf"

        # Signal 'stop' before killing so that the killed container doesn't restart
        local output
        local unitLoaded=1
        if ! systemctlIgnoreNotLoaded stop --no-block $service; then
            errEcho "Stopping $service failed with: $output"
        fi
        if [[ $unitLoaded ]]; then
            if ! systemctlIgnoreNotLoaded kill $service; then
                errEcho "Killing $service failed with: $output"
            fi
        fi

        rm -f "$mutableServicesDir/machines.target.wants/$service"

        if [[ -L $serviceFile ]]; then
            rm $serviceFile
            needDaemonReload=1
        fi
        local gcRootsPath=$(makeGCRootsPath $container)
        rm -f $gcRootsPath
        rm -f $gcRootsPath.conf

        # Remove declarative container confFile, otherwise `nixos-container` fails
        # with 'cannot destroy declarative container'
        rm -f $confFile
        # Create dummy confFile, otherwise `nixos-container` stops before
        # destroying the container completely
        touch $confFile

        # Remove immutable attribute from nested container var/empty files so that
        # the container directory can be deleted
        for varempty in "$stateDirectory"/$container/var/lib/*containers/*/var/empty; do
            chattr -i $varempty
        done

        nixos-container destroy $container || true
    done
    if [[ $needDaemonReload ]]; then
        systemctl daemon-reload
    fi
}

if [[ $destroy ]]; then
    if  [[ $1 == --all || $1 == -a ]]; then
        getInstallDirs
        containers=($(getContainers))
    # If the first letter of the first arg is one of ./-
    # or no arg is given and EXTRA_CONTAINER_ETC is defined
    elif (($# > 0)) && [[ ${1:0:1} == [./-] ]] || [[ $EXTRA_CONTAINER_ETC ]]; then
        # Get containers to be destroyed from the container definition
        getContainersFromDefinition "$@"
        containers=("${allContainers[@]}")
        echo "Destroying containers:"
        printf '%s\n' "${containers[@]}"
    else
        if (($# == 0)); then
            errEcho "No container name specified"
            exit 1
        fi
        containers=("$@")
    fi

    destroyContainers "${containers[@]}"
    exit 0
fi

# Commands 'create', 'shell'
#------------------------------------------------------------------------------
getContainersFromDefinition "$@"

## Shell related setup

startShellEnv=
makeStartInterruptible=
if [[ $shell ]]; then
    if (( ${#allContainers[@]} > 1 )); then
        errEcho "Command 'shell' requires only one container to be defined"
        exit 1
    fi
    shellContainer=${allContainers[0]}

    [[ $shellContainer == $runningShellContainer ]] && insideContainerShell=1 || insideContainerShell=

    if [[ $runCommand || ! $insideContainerShell ]]; then
        startShellEnv=1
    fi
    if [[ $insideContainerShell && ! $forceDestroy ]]; then
        destroy=
    fi
    if [[ $startShellEnv && ! $runCommand ]]; then
        makeStartInterruptible=1
    fi
    if [[ $destroy ]]; then
        destroyContainers $shellContainer
    fi
    start=1
fi

## 4. Install containers

confWithoutSystem() {
    sed /^SYSTEM_PATH=/d $1
}

isContainerUnchanged() {
    [[ -e $serviceDest && \
       -e $confDest && \
       $(realpath $serviceFile) == $(realpath $serviceDest) ]] || return 1

    if [[ $(realpath $confFile) != $(realpath $confDest) ]]; then
        if [[ $(confWithoutSystem $confFile) == $(confWithoutSystem $confDest) ]]; then
            onlySystemChangedContainers+=($container)
        fi
        return 1
    fi
}

checkInstallationSuccess() {
    if [[ $isNixOS ]]; then
        service=container@$1.service
        # If ExecStart points to the generic 'container_-start' script of the
        # 'container@.service' service template, the installation of the
        # container service file failed.
        if [[ $(systemctl show -p ExecStart $service) == */bin/container_-start*  ]]; then
            errEcho
            errEcho 'Container service installation failed.'
            errEcho 'Please add the following to your NixOS configuration'
            errEcho 'to enable dynamically installing systemd units:'
            errEcho 'boot.extraSystemdUnitPaths = [ "/etc/systemd-mutable/system" ];'
            errEcho
            errEcho 'See also: https://github.com/erikarvstedt/extra-container/#install'
            exit 1
        fi
    fi
}

echo
echo "Installing containers:"

mkdir -p $mutableServicesDir "$configDirectory" /nix/var/nix/gcroots/auto/
if [[ ! $isNixOS ]]; then createOsReleaseFile; fi

changedContainers=()
onlySystemChangedContainers=()
for container in ${allContainers[@]} ; do
    service=container@$container.service
    serviceFile=$nixosSystemEtc/systemd/system/$service
    serviceDest=$mutableServicesDir/$service
    confFile=$nixosSystemEtc/$configDirectoryName/$container.conf
    confDest="$configDirectory/$(basename $confFile)"

    if isContainerUnchanged; then
        echo "$container (unchanged, skipped)"
    else
        echo "$container"
        changedContainers+=($container)

        if [[ ! -e $confFile ]]; then
            alternativeConfFile=$nixosSystemEtc/$otherConfigDirectoryName/$container.conf
            if [[ -e $alternativeConfFile ]]; then
                errEcho
                errEcho    'Error:'
                errEcho    'To be compatible with this container host, the container'
                errEcho -n 'must be built with `legacyInstallDirs = '
                if [[ $legacyInstallDirs == true ]]; then
                    errEcho 'true`.'
                else
                    errEcho 'false` (default).'
                fi
                errEcho
            else
                errEcho "Error: $confFile doesn't exist"
            fi
            exit 1
        fi

        ln -sf $(realpath $serviceFile) $serviceDest
        ln -sf $(realpath $confFile) $confDest
        gcRootsPath=$(makeGCRootsPath $container)
        ln -sf $serviceDest $gcRootsPath
        ln -sf $confDest $gcRootsPath.conf

        if grep -q AUTO_START=1 "$confFile"; then
            wantsDir=$mutableServicesDir/machines.target.wants
            mkdir -p "$wantsDir"
            ln -sfn "../$service" "$wantsDir"
        fi
    fi
done

if [[ $changedContainers ]]; then
    systemctl daemon-reload
    checkInstallationSuccess ${changedContainers[0]}
fi

## 3. Setup shell

if [[ $makeStartInterruptible ]]; then
    # When starting a shell, continue the script when container starting gets interrupted.
    # This way, the user can interrupt waiting for the startup of all services
    # and interact with the starting container.
    trap handleShell SIGINT
fi

handleShell() {
    trap - SIGINT

    export containerIp=$(nixos-container show-ip $shellContainer 2> /dev/null) || containerIp=
    export runningShellContainer=$shellContainer
    export containerStateDirectory=$stateDirectory
    extraContainerCmd() {
        if [[ $# > 0 ]]; then
            nixos-container run $name -- "$@" | cat;
        else
            nixos-container root-login $name
        fi
    }
    if [[ $extraContainerSSH ]]; then
        makeTmpDir
        export sshKey
        export sshControlPath=$tmpDir/ssh-connection
        extraContainerSSHCmd() {
            ssh -i $sshKey -o ConnectTimeout=2 \
                -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR \
                -o ControlMaster=auto -o ControlPath="$sshControlPath" -o ControlPersist=60 \
                root@$ip "$@"
        }
        export -f extraContainerSSHCmd

    fi
    extraContainerHelp() {
        if [[ $ip ]]; then
            echo "Container address: $ip (\$ip)"
        fi
        echo "Container filesystem: $containerStateDirectory/$name"
        echo
        echo 'Run "c COMMAND" to execute a command in the container'
        echo 'Run "c" to start a shell session inside the container'
        if [[ $extraContainerSSH ]]; then
            echo 'Run "cssh" for SSH'
        fi
    }
    export -f extraContainerCmd extraContainerHelp

    defineShortcuts='
      function h() { extraContainerHelp; }
      function c() { extraContainerCmd "$@"; }
      export -f h c
      if [[ $extraContainerSSH ]]; then
        function cssh() { extraContainerSSHCmd "$@"; }
        export -f cssh
      fi
      export name=$runningShellContainer
      export ip=$containerIp
    '

    if [[ $runCommand ]]; then
        eval "$defineShortcuts"
        echo "Running command."
        "${runCommand[0]}" "${runCommand[@]:1}"
    else
        echo 'Starting shell.'
        echo 'Enter "h" for documentation.'
        # Start interactive, non-login shell.
        # Contrary to what is stated in the manual, bash --rcfile sources
        # /etc/profile (and consequently .bashrc) before evaluating the custom rcfile.
        # This allows us to start a shell with prompt and aliases set up and add extra
        # features via --rcfile.
        bash --rcfile <(
            # Prevent aliases from overriding the helper functions
            echo "unalias h c cssh 2>/dev/null; $defineShortcuts"
            # Use PATH from the current calling environment
            printf 'export PATH=%q' "$PATH"
        )
    fi

    # Needed when called from the SIGINT handler
    exit 0
}

## 4. Start/restart containers

echo

if [[ $start || $update || $restart ]]; then
    toStart=()
    toRestart=()
    # systemctl is-active fails when some containers are not active
    statuses=($(systemctl is-active $(getServiceNames ${allContainers[@]}))) || true
    for i in ${!statuses[@]}; do
        if [[ ${statuses[$i]} == active ]]; then
            runningContainers+=(${allContainers[$i]})
        else
            toStart+=(${allContainers[$i]})
        fi
    done

    if [[ $start && ((${#toStart[@]} != 0)) ]]; then
        echo "Starting containers:"
        printf '%s\n' ${toStart[@]}
        echo
        systemctl start $(getServiceNames ${toStart[@]})
    fi

    # Update or restart changed containers that are running
    toRestart=($(comm -12 <(printf '%s\n' ${runningContainers[@]} | sort) \
                          <(printf '%s\n' ${changedContainers[@]})))
    if ((${#toRestart[@]} != 0)); then
        if [[ ! $restart ]]; then
            toUpdate=($(comm -12 <(printf '%s\n' ${toRestart[@]}) \
                                 <(printf '%s\n' ${onlySystemChangedContainers[@]})))
            toRestart=($(comm -23 <(printf '%s\n' ${toRestart[@]}) \
                                  <(printf '%s\n' ${toUpdate[@]})))
            if ((${#toUpdate[@]} != 0)); then
                echo "Updating containers:"
                printf '%s\n' ${toUpdate[@]}
                echo
                updateContainers ${toUpdate[@]}
            fi
        fi
        if ((${#toRestart[@]} != 0)); then
            echo "Restarting containers:"
            printf '%s\n' ${toRestart[@]}
            echo
            restartContainers ${toRestart[@]}
        fi
    fi
fi

if [[ $startShellEnv ]]; then
    handleShell
fi


================================================
FILE: flake.nix
================================================
{
  description = "Run declarative NixOS containers without full system rebuilds";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.11";
  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }@inputs:
    let
      supportedSystems = [ "x86_64-linux" "i686-linux" "aarch64-linux" ];
      eachSupportedSystem = flake-utils.lib.eachSystem supportedSystems;
      pkg = pkgs: pkgs.callPackage ./. { pkgSrc = ./.; };
    in
    {
      nixosModules.default = { pkgs, ... }: {
        environment.systemPackages = [ (pkg pkgs) ];
        boot.extraSystemdUnitPaths = [ "/etc/systemd-mutable/system" ];
      };

      overlays.default = final: prev: { extra-container = pkg final; };

      lib = {
        inherit supportedSystems eachSupportedSystem;

        buildContainers = {
          system
          , config
          , nixpkgs ? inputs.nixpkgs
          , legacyInstallDirs ? false
          , addRunner ? true
        }: let
          containers = self.lib.evalContainers { inherit system config nixpkgs legacyInstallDirs; };
          etc = containers.config.system.build.etc;
          withRunner = etc.overrideAttrs (old: {
            name = "container";
            buildCommand = old.buildCommand + "\n" + ''
              install -D -m700 <(printf '${''
                #!/usr/bin/env bash
                if ! type -p extra-container >/dev/null; then
                  >&2 echo "Error: extra-container is not installed"
                  >&2 echo "Docs: https://github.com/erikarvstedt/extra-container?tab=readme-ov-file#install"
                  exit 1
                fi
                EXTRA_CONTAINER_ETC=%s exec extra-container "$@"
              ''}' "$out") $out/bin/container
            '';
          });
        in
          (if addRunner then withRunner else etc) // {
            inherit (containers) config;
            inherit (containers.config) containers;
          };

        evalContainers = {
          system
          , config
          , nixpkgs ? inputs.nixpkgs
          , legacyInstallDirs ? false
        }: import ./eval-config.nix {
          inherit
            system
            legacyInstallDirs;
          nixosPath = nixpkgs + "/nixos";
          systemConfig = config;
        };
      };

    } // (eachSupportedSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
        inherit (nixpkgs) lib;
      in
      rec {
        packages.default = pkg pkgs;

        # This dev shell allows running the `extra-container` command directly from the local
        # source (./extra-container), for quick edit/test cycles.
        # This only works when `nix develop` is started from the repo root directory.
        devShells.default = let
          # Extra PATH, as defined in ./default.nix
          path = lib.makeBinPath (with pkgs; [
            openssh
          ]);
        in pkgs.stdenv.mkDerivation {
          name = "shell";

          shellHook =  ''
            PATH="${path}''${PATH:+:}$PATH"

            # Enable calling the local source (./extra-container) with command `extra-container`
            PATH="$(realpath .):$PATH"

            # Use the pinned nixpkgs for building containers when running `extra-container`
            export NIX_PATH="nixpkgs=${nixpkgs}''${NIX_PATH:+:}$NIX_PATH"

            # See comment in ./extra-container for an explanation
            export LOCALE_ARCHIVE=${pkgs.glibcLocales}/lib/locale/locale-archive
          '';
        };

        packages = {
          # Run a basic extra-container test in a NixOS VM
          test = pkgs.testers.nixosTest {
            name = "extra-container";

            nodes.machine = { config, ... }: {
              imports = [ self.nixosModules.default ];
              # memorySize = 1024 needed for evaluating the container system
              # memorySize = 1200 needed to avoid error during boot:
              #   'agetty[817]: failed to open credentials directory'
              virtualisation.memorySize = 1200;
              nix.nixPath = [ "nixpkgs=${nixpkgs}" ];
              system.stateVersion = config.system.nixos.release;
              # Pre-build the container used by testScript
              system.extraDependencies = let
                basicContainer = import ./eval-config.nix {
                  nixosPath = "${nixpkgs}/nixos";
                  legacyInstallDirs = false;
                  inherit system;
                  systemConfig = {
                    containers.test.config.environment.etc.testFile.text = "testSuccess";
                  };
                };
              in [ basicContainer.config.system.build.etc ];
            };

            testScript = ''
              config = '{ containers.test.config.environment.etc.testFile.text = "testSuccess"; }'
              output = machine.succeed(
                f"extra-container shell -E '{config}' --run c cat /etc/testFile"
              )
              if not "testSuccess" in output:
                print(f"Test failed. Output:\n{output}")
            '';
          };

          # Used by apps.vm
          vm = (import "${nixpkgs}/nixos" {
            inherit system;
            configuration = { config, pkgs, lib, modulesPath, ... }: with lib; {
              imports = [
                self.nixosModules.default
                "${modulesPath}/virtualisation/qemu-vm.nix"
              ];
              virtualisation.graphics = false;
              services.getty.autologinUser = "root";
              nix.nixPath = [ "nixpkgs=${nixpkgs}" ];
              system.stateVersion = config.system.nixos.release;
              documentation.enable = false;
              # Power off VM when the user exits the shell
              systemd.services."serial-getty@".preStop = ''
                echo o >/proc/sysrq-trigger
              '';
              # Pre-build a minimal container
              system.extraDependencies = let
                basicContainer = import ./eval-config.nix {
                  nixosPath = "${nixpkgs}/nixos";
                  legacyInstallDirs = false;
                  inherit system;
                  systemConfig = {};
                };
              in [ basicContainer.config.system.build.etc ];
            };
          }).config.system.build.vm;

          runVM = pkgs.writers.writeBash "run-vm" ''
            set -euo pipefail
            export NIX_DISK_IMAGE=/tmp/extra-container-vm-img
            rm -f $NIX_DISK_IMAGE
            trap "rm -f $NIX_DISK_IMAGE" EXIT

            export QEMU_OPTS="-smp $(nproc) -m 2000"
            ${packages.vm}/bin/run-*-vm
          '';

          debugTest = pkgs.writers.writeBash "run-debug-test" ''
            set -euo pipefail
            export TMPDIR=$(mktemp -d)
            trap "rm -rf $TMPDIR" EXIT

            export QEMU_OPTS="-smp $(nproc) -m 2000"
            ${packages.test.driver}/bin/nixos-test-driver <(
              echo "start_all(); import code; code.interact(local=globals())"
            )
          '';

          updateReadme = pkgs.writers.writeBash "update-readme" ''
            exec ${pkgs.ruby}/bin/ruby ./util/update-readme.rb
          '';
        };

        apps = {
          # Run a NixOS VM where extra-container is installed
          vm = {
            type = "app";
            program = toString packages.runVM;
          };

          # Run a Python test driver shell inside the test VM
          debugTest = {
            type = "app";
            program = toString packages.debugTest;
          };

          updateReadme = {
            type = "app";
            program = toString packages.updateReadme;
          };
        };

        checks = { inherit (packages) test; };
      }
    ));
}


================================================
FILE: run-tests-in-container.sh
================================================
#!/usr/bin/env bash

set -euo pipefail
shopt -s nullglob

scriptDir="$(dirname "$(readlink -f "$0")")"
PATH=$scriptDir:$PATH

cleanup() {
    # clean immutable files inside the container
    for f in /var/lib/*containers/test-extra-container/var/lib/*containers/*/var/empty; do
        chattr -i -a "$f"
        rm -rf "$f"
    done
    extra-container destroy test-extra-container || true
}
trap "cleanup" EXIT

trap "echo \"Error at $(realpath ${BASH_SOURCE[0]}):\$LINENO\"" ERR

cleanup

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

nixpkgs=$(nix-instantiate --eval -E '(toString <nixpkgs>)' | tr -d '"')

extra-container create -s <<EOF
{ config, pkgs, lib, ... }:
{
  containers.test-extra-container = {
    bindMounts."/extra-container".hostPath = "$scriptDir";
    bindMounts."/nixpkgs".hostPath = "$nixpkgs";
    config = { options, ... }: {
      environment = {
        systemPackages = [ pkgs.nixos-container ];
        variables.NIX_PATH = lib.mkForce "nixpkgs=/nixpkgs";
      };
      boot = lib.optionalAttrs (options.boot ? extraSystemdUnitPaths) {
        extraSystemdUnitPaths = [ "/etc/systemd-mutable/system" ];
      };
    };
  };
}
EOF

echo "Running tests"
echo
nixos-container run test-extra-container -- '/extra-container/test.sh'


================================================
FILE: test.sh
================================================
#!/usr/bin/env bash

set -euo pipefail
shopt -s nullglob

scriptDir="$(dirname "$(readlink -f "$0")")"
PATH=$scriptDir:$PATH

cleanup() {
    set +e
    for container in $(extra-container list | grep ^test-); do
        extra-container destroy $container
    done
    set -e
}
trap "cleanup" EXIT

trap "echo \"Error at $(realpath ${BASH_SOURCE[0]}):\$LINENO\"" ERR

testMatches() {
    actual="$1"
    expected="$2"
    if [[ $actual != $expected ]]; then
        echo
        echo 'Pattern does not match'
        echo 'Expected:'
        echo "$expected"
        echo
        echo 'Actual:'
        echo "$actual"
        echo
        return 1
    fi
}

# This significantly reduces eval time. Not needed for NixOS ≥ 20.03
baseConfig='config = { documentation.nixos.enable = false; }'

cleanup

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
echo "Test attr arg and container starting "

output=$(extra-container create -A 'a b' --start <<EOF
{
  "a b" = { config, pkgs, ... }: {
    containers.test-1 = {
      $baseConfig;
    };
  };
}
EOF
)

testMatches "$output" "*Installing*test-1*Starting*test-1*"

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
echo "Test starting and updating"

output=$(extra-container create -s <<EOF
{ config, pkgs, ... }:
{
  containers.test-1 = {
    $baseConfig // { environment.variables.foo = "a"; };
  };
  containers.test-2 = {
    $baseConfig;
  };
}
EOF
)

testMatches "$output" "*Starting*test-2*Updating*test-1*"

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
echo "Test unchanged"

output=$(extra-container create -s <<EOF
{ config, pkgs, ... }:
{
  containers.test-1 = {
    $baseConfig // { environment.variables.foo = "a"; };
  };
}
EOF
)

testMatches "$output" "*test-1 (unchanged, skipped)*"

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
echo "Test updating and restarting"

output=$(extra-container create -u <<EOF
{ config, pkgs, ... }:
{
  containers.test-1 = {
    $baseConfig // { environment.variables.foo = "b"; };
  };
  containers.test-2 = {
    privateNetwork = true;
    $baseConfig;
  };
}
EOF
)

testMatches "$output" "*Updating*test-1*Restarting*test-2*"

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
echo "Test list"

output=$(extra-container list | grep ^test- || true)
testMatches "$output" "test-1*test-2"

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
echo "Test destroy"

[[ $(echo /var/lib/*containers/test-*) ]]
cleanup
output=$(extra-container list | grep ^test- || true)
testMatches "$output" ""
[[ ! $(echo /var/lib/*containers/test-*) ]]

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
echo "Test shell run"

read -d '' src <<EOF || true
{ config, pkgs, ... }:
{
  containers.test-1 = {
    $baseConfig;
  };
}
EOF
output=$(extra-container shell -E "$src" --run c uname -a)
testMatches "$output" "*Linux test*"

# Container should be destroyed after running
[[ ! -e /var/lib/*containers/test-1 ]]

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
echo "Test manual build"

storePath=$(extra-container build <<EOF
{ config, pkgs, ... }:
{
  containers.test-1 = {
    $baseConfig;
  };
  containers.test-2 = {
    $baseConfig;
  };
}
EOF
)

testMatches "$storePath" "/nix/store/*"

output=$(extra-container create -s $storePath)
testMatches "$output" "*Starting*test-1*test-2*"

#―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
echo "Test destroy from container definition"
extra-container destroy $storePath
[[ ! -e /var/lib/*containers/test-1 ]]
[[ ! -e /var/lib/*containers/test-2 ]]

# TODO: Add flake tests when flakes have stabilized


================================================
FILE: util/generate-sudoers.rb
================================================
# 1. Add $HOME/.nix-profile/bin to secure_path (Defaults)
# 2. Add NIX_PATH to env_keep (Defaults)
#
def make_sudoers(old_sudoers, extra_secure_paths)
  secure_path_set = false

  sudoers = old_sudoers.sub(/(Defaults\s+secure_path\s*=\s*"?)([^"\s]+)(.*)/) do |_|
    secure_path_set = true
    prefix, path, suffix = $1, $2, $3
    paths = path.split(':')
    extra_paths = extra_secure_paths.split(':')
    new_path = (paths + extra_paths).uniq.join(':')
    "#{prefix}#{new_path}#{suffix}"
  end

  env_keep_statement = "Defaults	env_keep += NIX_PATH"

  lines_to_add = []
  lines_to_add << %(Defaults	secure_path = "#{extra_secure_paths}") if !secure_path_set
  lines_to_add << env_keep_statement if !old_sudoers.include?(env_keep_statement)

  if !lines_to_add.empty?
    sudoers << lines_to_add.join("\n") << "\n"
  end

  sudoers == old_sudoers ? nil : sudoers
end

if __FILE__ == $0
  puts make_sudoers(STDIN.read, ARGV.first)
end


================================================
FILE: util/install.sh
================================================
#!/usr/bin/env bash

set -euo pipefail

# Install extra-container on a non-NixOS systemd-based system for use with sudo.
# Requires a multi-user nix installation, because extra-container runs nix-build as root.
# This script is idempotent.

[[ -e /run/booted-system/nixos-version ]] && isNixos=1 || isNixos=
[[ -e /run/systemd/system ]] && hasSystemd=1 || hasSystemd=
scriptDir=$(cd "${BASH_SOURCE[0]%/*}" && pwd)

if [[ $EUID == 0 ]]; then
    echo "This script should NOT be run as root."
    exit 1
fi
if [[ $isNixos ]]; then
    echo "This install script is not needed on NixOS. See the README for installation instructions."
    exit 1
fi
if [[ ! $hasSystemd ]]; then
    echo "extra-container requires systemd."
    exit 1
fi
if [[ ! -e /nix/var/nix/profiles/default ]]; then
    echo "extra-container requires a multi-user nix installation."
    exit 1
fi

## 1. Build extra-container
tmpDir=$(mktemp -d)
trap "rm -rf $tmpDir" EXIT
nix-build --out-link $tmpDir/extra-container -E "(import <nixpkgs> {}).callPackage ''$scriptDir/..'' {}"

## 2. Install to root user profile
sudo $(type -P nix-env) -i $tmpDir/extra-container

## 3. Edit /etc/sudoers to enable running extra-container via sudo
# See ./edit-sudoers.rb for more details
if ! type -P ruby > /dev/null; then
    nix-build --out-link $tmpDir/ruby '<nixpkgs>' -A ruby > /dev/null
    export PATH="$tmpDir/ruby/bin${PATH:+:}$PATH"
fi

extraSecurePaths=/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin

newSudoersContent=$(sudo cat /etc/sudoers | ruby "$scriptDir"/generate-sudoers.rb "$extraSecurePaths")
if [[ $newSudoersContent ]]; then
    echo "$newSudoersContent" | sudo EDITOR="tee" visudo >/dev/null
fi


================================================
FILE: util/update-readme.rb
================================================
#!/usr/bin/env ruby

def without_warnings
  original_verbosity = $VERBOSE
  $VERBOSE = nil
  result = yield
  $VERBOSE = original_verbosity
  result
end

Dir.chdir(File.join(__dir__, '..'))
readme = File.read('README.md')
# hide `warning: Insecure world writable dir {extra-container-src-dir}`
usage = without_warnings { `extra-container`.sub(/\A.*?Usage:\s*/m, '') }
updated = readme.sub(/(?<=^## Usage\n```\n).*?(?=^```)/m, usage)
current = File.read('README.md')
if current != updated
  File.write('README.md', updated)
  puts "Updated README.md"
end