Full Code of mle-infrastructure/mle-monitor for AI

Repository: mle-infrastructure/mle-monitor
Branch: main
Commit: 8030875516d1
Files: 47
Total size: 317.2 KB

Directory structure:
gitextract_957adg6p/

├── .codecov.yml
├── .gitignore
├── CHANGELOG.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── examples/
│   ├── base_config.json
│   ├── getting_started.ipynb
│   ├── protocol_gcs.py
│   ├── protocol_local.py
│   ├── run_infrastructure.py
│   └── train.py
├── mle_monitor/
│   ├── __init__.py
│   ├── _version.py
│   ├── dashboard/
│   │   ├── __init__.py
│   │   ├── components/
│   │   │   ├── __init__.py
│   │   │   ├── plots.py
│   │   │   ├── protocol.py
│   │   │   └── usage.py
│   │   ├── layout.py
│   │   └── update.py
│   ├── mle_dashboard.py
│   ├── mle_protocol.py
│   ├── mle_resource.py
│   ├── protocol/
│   │   ├── __init__.py
│   │   ├── add.py
│   │   ├── gcs_sync.py
│   │   ├── load.py
│   │   ├── summary.py
│   │   └── tables.py
│   ├── resource/
│   │   ├── __init__.py
│   │   ├── gcp.py
│   │   ├── local.py
│   │   ├── sge.py
│   │   └── slurm.py
│   └── utils/
│       ├── __init__.py
│       ├── gcs_zip.py
│       ├── helpers.py
│       └── tracker.py
├── requirements/
│   ├── requirements-examples.txt
│   ├── requirements-test.txt
│   └── requirements.txt
├── setup.py
└── tests/
    ├── fixtures/
    │   └── base_config.json
    ├── test_dashboard.py
    ├── test_protocol.py
    └── test_resource.py

================================================
FILE CONTENTS
================================================

================================================
FILE: .codecov.yml
================================================
codecov:
  require_ci_to_pass: yes

coverage:
  precision: 2
  round: down
  range: "30...70"

coverage:
  status:
    project: # settings affecting project coverage
      default:
        threshold: 5%  # allow for 5% reduction of coverage without failing

comment:
  layout: "reach, diff, files"
  behavior: default
  require_changes: true

ignore:
  - "tests/.*"
  - "_version.py"
  - "setup.py"
  - "examples/*"
  - "docs/"


================================================
FILE: .gitignore
================================================
mle_protocol.db
*.yaml
retrieved_log_dir
logs_search
old.py
*.zip
.vim-arsync
.DS_Store
__pycache__
.sync-config.cson
.ipynb_checkpoints
*.egg-info
data/
*.key
tboards/
*.ckpt
.pytest_cache/
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
experiment_dir
# C extensions
*.so
examples/multi_seed_dir
examples/multi_config_dir
examples/every_k_dir
examples/top_k_dir
examples/post_plot_dir

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# pyenv
.python-version

# celery beat schedule file
celerybeat-schedule

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/


================================================
FILE: CHANGELOG.md
================================================
## [v0.0.2] - [03/2022]

### Changed

- Restructure requirements and installations.
- Update notebook.

## [v0.0.1] - [12/09/2021]

### Added

Basic API for `MLEProtocol`, `MLEResource` & `MLEDashboard`:

```python
from mle_monitor import MLEProtocol

# Load protocol database or create new one -> print summary
protocol_db = MLEProtocol("mle_protocol.db", verbose=False)
protocol_db.summary(tail=10, verbose=True)

# Draft data to store in protocol & add it to the protocol
meta_data = {
    "purpose": "Grid search",  # Purpose of experiment
    "project_name": "MNIST",  # Project name of experiment
    "experiment_type": "hyperparameter-search",  # Type of experiment
    "experiment_dir": "experiments/logs",  # Experiment directory
    "num_total_jobs": 10,  # Number of total jobs to run
    ...
}
new_experiment_id = protocol_db.add(meta_data)

# ... train your 10 (pseudo) networks/complete respective jobs
for i in range(10):
    protocol_db.update_progress_bar(new_experiment_id)

# Wrap up an experiment (store completion time, etc.)
protocol_db.complete(new_experiment_id)
```


================================================
FILE: CONTRIBUTING.md
================================================
# Contributing to `mle-monitor`
We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:

- Reporting a bug
- Discussing the current state of the code
- Submitting a fix
- Proposing new features
- Becoming a maintainer

## We Develop with Github
We use github to host code, to track issues and feature requests, as well as accept pull requests.

## We Use [Github Flow](https://guides.github.com/introduction/flow/index.html), So All Code Changes Happen Through Pull Requests
Pull requests are the best way to propose changes to the codebase (we use [Github Flow](https://guides.github.com/introduction/flow/index.html)). We actively welcome your pull requests:

1. Fork the repo and create your branch from `master`.
2. If you've added code that should be tested, add tests.
3. If you've changed APIs, update the documentation.
4. Ensure the test suite passes.
5. Make sure your code lints.
6. Issue that pull request!

## Any contributions you make will be under the MIT Software License
In short, when you submit code changes, your submissions are understood to be under the same [MIT License](http://choosealicense.com/licenses/mit/) that covers the project. Feel free to contact the maintainers if that's a concern.

## Report bugs using Github's [issues](https://github.com/mle-infrastructure/mle-monitor/issues)
We use GitHub issues to track public bugs. Report a bug by [opening a new issue](); it's that easy!

## Write bug reports with detail, background, and sample code

**Great Bug Reports** tend to have:

- A quick summary and/or background
- Steps to reproduce
  - Be specific!
  - Give sample code if you can.
- What you expected would happen
- What actually happens
- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)

## Use the Black Coding Style
The codebase follows the [Black](https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html) coding style. Using a autoformatter can make your life easier!

## License
By contributing, you agree that your contributions will be licensed under its MIT License.

## References
This document was adapted from the open-source contribution guidelines for [Facebook's Draft](https://github.com/facebook/draft-js/blob/a9316a723f9e918afde44dea68b5f9f39b7d9b00/CONTRIBUTING.md) and from the [Transcriptase adapted version](https://gist.github.com/briandk/3d2e8b3ec8daf5a27a62).


================================================
FILE: LICENSE
================================================
Copyright (c) 2018 The Python Packaging Authority

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


================================================
FILE: README.md
================================================
# Lightweight Experiment & Resource Monitoring 📺
[![Pyversions](https://img.shields.io/pypi/pyversions/mle-monitor.svg?style=flat-square)](https://pypi.python.org/pypi/mle-monitor)
[![PyPI version](https://badge.fury.io/py/mle-monitor.svg)](https://badge.fury.io/py/mle-monitor)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![codecov](https://codecov.io/gh/mle-infrastructure/mle-monitor/branch/main/graph/badge.svg?token=75FIYZG8BD)](https://codecov.io/gh/mle-infrastructure/mle-monitor)
[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mle-infrastructure/mle-monitor/blob/main/examples/getting_started.ipynb)
<a href="https://github.com/mle-infrastructure/mle-monitor/blob/main/docs/logo_transparent.png?raw=true"><img src="https://github.com/mle-infrastructure/mle-monitor/blob/main/docs/logo_transparent.png?raw=true" width="200" align="right" /></a>

"Did I already run this experiment before? How many resources are currently available on my cluster?" If these are common questions you encounter during your daily life as a researcher, then `mle-monitor` is made for you. It provides a lightweight API for tracking your experiments using a pickle protocol database (e.g. for hyperparameter searches and/or multi-configuration/multi-seed runs). Furthermore, it comes with built-in resource monitoring on Slurm/Grid Engine clusters and local machines/servers.

`mle-monitor` provides three core functionalities:

- **`MLEProtocol`**: A composable protocol database API for ML experiments.
- **`MLEResource`**: A tool for obtaining server/cluster usage statistics.
- **`MLEDashboard`**: A dashboard visualizing resource usage & experiment protocol.

To get started I recommend checking out the [colab notebook](https://colab.research.google.com/github/mle-infrastructure/mle-monitor/blob/main/examples/getting_started.ipynb) and an [example workflow](https://github.com/mle-infrastructure/mle-monitor/blob/main/examples/run_infrastructure.py).

<img src="https://github.com/mle-infrastructure/mle-monitor/blob/main/docs/monitor-promo-gif.gif?raw=true" alt="drawing" width="900"/>

## `MLEProtocol`: Keeping Track of Your Experiments 📝

```python
from mle_monitor import MLEProtocol

# Load protocol database or create new one -> print summary
protocol_db = MLEProtocol("mle_protocol.db", verbose=False)
protocol_db.summary(tail=10, verbose=True)

# Draft data to store in protocol & add it to the protocol
meta_data = {
    "purpose": "Grid search",  # Purpose of experiment
    "project_name": "MNIST",  # Project name of experiment
    "experiment_type": "hyperparameter-search",  # Type of experiment
    "experiment_dir": "experiments/logs",  # Experiment directory
    "num_total_jobs": 10,  # Number of total jobs to run
    ...
}
new_experiment_id = protocol_db.add(meta_data)

# ... train your 10 (pseudo) networks/complete respective jobs
for i in range(10):
    protocol_db.update_progress_bar(new_experiment_id)

# Wrap up an experiment (store completion time, etc.)
protocol_db.complete(new_experiment_id)
```

The meta data can contain the following keys:

| Search Type           | Description | Default |
|----------------------- | ----------- | --------------- |
| `purpose`          |  Purpose of experiment  | `'None provided'` |
| `project_name`        |  Project name of experiment  | `'default'` |
| `exec_resource`    |  Resource jobs are run on | `'local'` |
| `experiment_dir`  |  Experiment log storage directory   | `'experiments'` |
| `experiment_type`     | Type of experiment to run | `'single'` |
| `base_fname`     | Main code script to execute | `'main.py'` |
| `config_fname`     | Config file path of experiment | `'base_config.yaml'` |
| `num_seeds`     | Number of evaluations seeds | 1 |
| `num_total_jobs`     | Number of total jobs to run | 1 |
| `num_job_batches`     | Number of jobs in single batch | 1 |
| `num_jobs_per_batch`     | Number of sequential job batches | 1 |
| `time_per_job`     | Expected duration: days-hours-minutes | `'00:01:00'` |
| `num_cpus`     | Number of CPUs used in job | 1 |
| `num_gpus`     | Number of GPUs used in job | 0 |

Additionally you can synchronize the protocol with a Google Cloud Storage (GCS) bucket by providing `cloud_settings`. In this case also the results stored in `experiment_dir` will be uploaded to the GCS bucket, when you call `protocol.complete()`.


```python
# Define GCS settings - requires 'GOOGLE_APPLICATION_CREDENTIALS' env var.
cloud_settings = {
    "project_name": "mle-toolbox",  # GCP project name
    "bucket_name": "mle-protocol",  # GCS bucket name
    "use_protocol_sync": True,  # Whether to sync the protocol to GCS
    "use_results_storage": True,  # Whether to sync experiment_dir to GCS
}
protocol_db = MLEProtocol("mle_protocol.db", cloud_settings, verbose=True)
```

## The `MLEResource`: Keeping Track of Your Resources 📉

#### On Your Local Machine
```python
from mle_monitor import MLEResource

# Instantiate local resource and get usage data
resource = MLEResource(resource_name="local")
resource_data = resource.monitor()
```

#### On a Slurm Cluster

```python
resource = MLEResource(
    resource_name="slurm-cluster",
    monitor_config={"partitions": ["<partition-1>", "<partition-2>"]},
)
```

#### On a Grid Engine Cluster

```python
resource = MLEResource(
    resource_name="sge-cluster",
    monitor_config={"queues": ["<queue-1>", "<queue-2>"]}
)
```

## The `MLEDashboard`: Dashboard Visualization 🎞️

```python
from mle_monitor import MLEDashboard

# Instantiate dashboard with protocol and resource
dashboard = MLEDashboard(protocol, resource)

# Get a static snapshot of the protocol & resource utilisation printed in console
dashboard.snapshot()

# Run monitoring in while loop - dashboard
dashboard.live()
```

## Installation ⏳

A PyPI installation is available via:

```
pip install mle-monitor
```

If you want to get the most recent commit, please install directly from the repository:

```
pip install git+https://github.com/mle-infrastructure/mle-monitor.git@main
```

### Citing the MLE-Infrastructure ✏️

If you use `mle-monitor` in your research, please cite it as follows:

```
@software{mle_infrastructure2021github,
  author = {Robert Tjarko Lange},
  title = {{MLE-Infrastructure}: A Set of Lightweight Tools for Distributed Machine Learning Experimentation},
  url = {http://github.com/mle-infrastructure},
  year = {2021},
}
```

## Development 👷

You can run the test suite via `python -m pytest -vv tests/`. If you find a bug or are missing your favourite feature, feel free to create an issue and/or start [contributing](CONTRIBUTING.md) 🤗.


================================================
FILE: examples/base_config.json
================================================
{
"train_config": {"lrate": 0.1},
"model_config": {"num_layers": 5},
"log_config": {"time_to_track": ["step_counter"],
               "what_to_track": ["loss"],
               "time_to_print": ["step_counter"],
               "what_to_print": ["loss"],
               "print_every_k_updates": 10,
               "overwrite_experiment_dir": 1}
}


================================================
FILE: examples/getting_started.ipynb
================================================
{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "08f9d077-64ed-4dc3-b772-498569e96d2b",
   "metadata": {
    "tags": []
   },
   "source": [
    "# `mle-monitor`: Lightweight Resource Monitoring\n",
    "### Author: [@RobertTLange](https://twitter.com/RobertTLange) [Last Update: March 2023][![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mle-infrastructure/mle-monitor/blob/main/examples/getting_started.ipynb)\n",
    "\n",
    "\"Did I already run this experiment before? How many resources are currently available on my cluster?\" If these are common questions you encounter during your daily life as a researcher, then `mle-monitor` is made for you. It provides a lightweight API for tracking your experiments using a pickle protocol database (e.g. for hyperparameter searches and/or multi-configuration/multi-seed runs). Furthermore, it comes with built-in resource monitoring on Slurm/Grid Engine clusters and local machines/servers. Finally, it leverages [`rich`](https://github.com/willmcgugan/rich) in order to provide a terminal dashboard that is updated online with new protocolled experiments and the current state of resource utilization. Here is an example of a dashboard running on a Grid Engine cluster:\n",
    "\n",
    "<img src=\"https://github.com/mle-infrastructure/mle-monitor/blob/main/docs/monitor-promo-gif.gif?raw=true\" alt=\"drawing\" width=\"900\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b1ff5d35-b29e-4386-b423-9c83ad1ae169",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "try:\n",
    "    import mle_monitor\n",
    "except:\n",
    "    !pip install -q mle-monitor\n",
    "    import mle_monitor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cf5bc30-a187-4936-a57c-3023ff6b3f2a",
   "metadata": {},
   "source": [
    "`mle-monitor` comes with three core functionalities:\n",
    "\n",
    "- **`MLEProtocol`**: A composable protocol database API for ML experiments.\n",
    "- **`MLEResource`**: A tool for obtaining server/cluster usage statistics.\n",
    "- **`MLEDashboard`**: A dashboard visualizing resource usage & experiment protocol.\n",
    "\n",
    "<img src=\"https://github.com/mle-infrastructure/mle-monitor/blob/main/docs/mle_monitor_structure.png?raw=true\" alt=\"drawing\" width=\"900\"/>\n",
    "\n",
    "Finally, `mle-monitor` is part of the [`mle-infrastructure`](https://github.com/mle-infrastructure) and comes with a set of handy built-in synergies. We will wrap-up by outlining a full workflow of using the protocol together with a random search experiment using [`mle-hyperopt`](https://github.com/mle-infrastructure/mle-hyperopt), [`mle-scheduler`](https://github.com/mle-infrastructure/mle-monitor) and [`mle-logging`](https://github.com/mle-infrastructure/mle-logging)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a66df14c-b920-4926-a8c1-37c42c63e7f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if code is run in Colab: If so -- download configs from repo\n",
    "try:\n",
    "    import google.colab\n",
    "    IN_COLAB = True\n",
    "    !wget -q https://raw.githubusercontent.com/mle-infrastructure/mle-monitor/main/examples/train.py\n",
    "    !wget -q https://raw.githubusercontent.com/mle-infrastructure/mle-monitor/main/examples/base_config.json\n",
    "except:\n",
    "    IN_COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "255b0522-6c4b-440d-8dca-345ba9e8a461",
   "metadata": {},
   "source": [
    "# Experiment Management with `MLEProtocol` 📝"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "44a221b5-758b-491b-b206-65c3bddeeddb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mle_monitor import MLEProtocol\n",
    "\n",
    "# Load the protocol from a local file (create new if it doesn't exist yet)\n",
    "protocol = MLEProtocol(protocol_fname=\"mle_protocol.db\", verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef2d1797-0251-495f-8976-47a24756e79c",
   "metadata": {},
   "source": [
    "In order to add a new experiment to the protocol database you have to provide a dictionary containing the experiment meta data:\n",
    "\n",
    "| Search Type           | Description | Default |\n",
    "|----------------------- | ----------- | --------------- |\n",
    "| `purpose`          |  Purpose of experiment  | `'None provided'` |\n",
    "| `project_name`        |  Project name of experiment  | `'default'` |\n",
    "| `exec_resource`    |  Resource jobs are run on | `'local'` |\n",
    "| `experiment_dir`  |  Experiment log storage directory   | `'experiments'` |\n",
    "| `experiment_type`     | Type of experiment to run | `'single'` |\n",
    "| `base_fname`     | Main code script to execute | `'main.py'` |\n",
    "| `config_fname`     | Config file path of experiment | `'base_config.yaml'` |\n",
    "| `num_seeds`     | Number of evaluations seeds | 1 |\n",
    "| `num_total_jobs`     | Number of total jobs to run | 1 |\n",
    "| `num_job_batches`     | Number of jobs in single batch | 1 |\n",
    "| `num_jobs_per_batch`     | Number of sequential job batches | 1 |\n",
    "| `time_per_job`     | Expected duration: days-hours-minutes | `'00:01:00'` |\n",
    "| `num_cpus`     | Number of CPUs used in job | 1 |\n",
    "| `num_gpus`     | Number of GPUs used in job | 0 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d65bcced-fa6f-4865-939f-b8d35203b05f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[15:19:55] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Added experiment <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> to protocol.                       <a href=\"file:///Users/rob/anaconda3/envs/mle-toolbox/lib/python3.9/site-packages/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/anaconda3/envs/mle-toolbox/lib/python3.9/site-packages/mle_monitor/mle_protocol.py#170\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">170</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[15:19:55]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Added experiment \u001b[1;36m1\u001b[0m to protocol.                       \u001b]8;id=600704;file:///Users/rob/anaconda3/envs/mle-toolbox/lib/python3.9/site-packages/mle_monitor/mle_protocol.py\u001b\\\u001b[2mmle_protocol.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=954284;file:///Users/rob/anaconda3/envs/mle-toolbox/lib/python3.9/site-packages/mle_monitor/mle_protocol.py#170\u001b\\\u001b[2m170\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "meta_data = {\n",
    "        \"purpose\": \"Test Protocol\",  # Purpose of experiment\n",
    "        \"project_name\": \"MNIST\",  # Project name of experiment\n",
    "        \"exec_resource\": \"local\",  # Resource jobs are run on\n",
    "        \"experiment_dir\": \"log_dir\",  # Experiment log storage directory\n",
    "        \"experiment_type\": \"hyperparameter-search\",  # Type of experiment to run\n",
    "        \"base_fname\": \"train.py\",  # Main code script to execute\n",
    "        \"config_fname\": \"base_config.json\",  # Config file path of experiment\n",
    "        \"num_seeds\": 5,  # Number of evaluations seeds\n",
    "        \"num_total_jobs\": 10,  # Number of total jobs to run\n",
    "        \"num_jobs_per_batch\": 5,  # Number of jobs in single batch\n",
    "        \"num_job_batches\": 2,  # Number of sequential job batches\n",
    "        \"time_per_job\": \"00:05:00\",  # Expected duration: days-hours-minutes\n",
    "        \"num_cpus\": 2,  # Number of CPUs used in job\n",
    "        \"num_gpus\": 1,  # Number of GPUs used in job\n",
    "    }\n",
    "e_id = protocol.add(meta_data, save=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be53c0dd-f83f-4935-8d06-926a2a93088c",
   "metadata": {},
   "source": [
    "Adding the experiment will load in the configuration file (either `.json` or `.yaml`) and set the experiment status to \"running\". You can then always retrieve the provided information using `protocol.get(e_id)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "76d04dd4-5e01-45c3-b5e6-86caf6521315",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'purpose': 'Test Protocol',\n",
       " 'project_name': 'MNIST',\n",
       " 'exec_resource': 'local',\n",
       " 'experiment_dir': 'log_dir',\n",
       " 'experiment_type': 'hyperparameter-search',\n",
       " 'base_fname': 'train.py',\n",
       " 'config_fname': 'base_config.json',\n",
       " 'num_seeds': 5,\n",
       " 'num_total_jobs': 10,\n",
       " 'num_jobs_per_batch': 5,\n",
       " 'num_job_batches': 2,\n",
       " 'time_per_job': '00:05:00',\n",
       " 'num_cpus': 2,\n",
       " 'num_gpus': 1,\n",
       " 'git_hash': 'ef86aa6343db21998ba58942e3c5e5124d3463c1',\n",
       " 'loaded_config': [{'train_config': {'lrate': 0.1},\n",
       "   'model_config': {'num_layers': 5},\n",
       "   'log_config': {'time_to_track': ['step_counter'],\n",
       "    'what_to_track': ['loss'],\n",
       "    'time_to_print': ['step_counter'],\n",
       "    'what_to_print': ['loss'],\n",
       "    'print_every_k_updates': 10,\n",
       "    'overwrite_experiment_dir': 1}}],\n",
       " 'e-hash': '1795b93d0ded1d5ab9c5a63243abe649',\n",
       " 'retrieved_results': False,\n",
       " 'stored_in_cloud': False,\n",
       " 'report_generated': False,\n",
       " 'job_status': 'running',\n",
       " 'completed_jobs': 0,\n",
       " 'start_time': '03/08/23 15:19',\n",
       " 'duration': '0:10:00',\n",
       " 'stop_time': '03/09/23 01:19'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "protocol.get(e_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "683161f4-aa0e-467e-89fa-714f739815e4",
   "metadata": {},
   "source": [
    "You can also always print a summary snapshot of the last experiments using `protocol.summary()`. By providing the boolean option `full`, you also print the resources used in an experiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e13d7d30-8d6b-49d0-b3c0-427654a19383",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">                                                                              </span>               \n",
       "<span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> 🔖 </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> 🆔 </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">   🗓   </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> Project </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> Purpose       </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> Type   </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">   </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">▶</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">   </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">♻</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> CPU </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> GPU </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span>               \n",
       "<span style=\"color: #000080; text-decoration-color: #000080\"> ──────────────────────────────────────────────────────────────────────────── </span>               \n",
       "<span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #800080; text-decoration-color: #800080\">⠙ </span> <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 1  <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 03/08 <span style=\"color: #000080; text-decoration-color: #000080\"> </span> MNIST   <span style=\"color: #000080; text-decoration-color: #000080\"> </span> Test Protocol <span style=\"color: #000080; text-decoration-color: #000080\"> </span> search <span style=\"color: #000080; text-decoration-color: #000080\"> </span> Local <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 5 <span style=\"color: #000080; text-decoration-color: #000080\"> </span>  2  <span style=\"color: #000080; text-decoration-color: #000080\"> </span>  1  <span style=\"color: #000080; text-decoration-color: #000080\"> </span>               \n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">                                                                              </span>               \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m                                                                              \u001b[0m               \n",
       "\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m🔖\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m🆔\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m  🗓  \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mProject\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mPurpose      \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mType  \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m  \u001b[0m\u001b[1;33m▶\u001b[0m\u001b[1;34m  \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;33m♻\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mCPU\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mGPU\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m               \n",
       "\u001b[34m ──────────────────────────────────────────────────────────────────────────── \u001b[0m               \n",
       "\u001b[34m \u001b[0m \u001b[35m⠙ \u001b[0m \u001b[34m \u001b[0m 1  \u001b[34m \u001b[0m 03/08 \u001b[34m \u001b[0m MNIST   \u001b[34m \u001b[0m Test Protocol \u001b[34m \u001b[0m search \u001b[34m \u001b[0m Local \u001b[34m \u001b[0m 5 \u001b[34m \u001b[0m  2  \u001b[34m \u001b[0m  1  \u001b[34m \u001b[0m               \n",
       "\u001b[34m                                                                              \u001b[0m               \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">                                                                                             </span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">    </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">    </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">       </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">         </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">               </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">        </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">       </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">   </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">     </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">     </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> ⏳ Completed </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> 🔖 </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> 🆔 </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">   🗓   </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> Project </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> Purpose       </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> Type   </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">   </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">▶</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">   </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">♻</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> CPU </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> GPU </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">    Jobs </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">✔</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">    </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\"> ─────────────────────────────────────────────────────────────────────────────────────────── </span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #800080; text-decoration-color: #800080\">⠙ </span> <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 1  <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 03/08 <span style=\"color: #000080; text-decoration-color: #000080\"> </span> MNIST   <span style=\"color: #000080; text-decoration-color: #000080\"> </span> Test Protocol <span style=\"color: #000080; text-decoration-color: #000080\"> </span> search <span style=\"color: #000080; text-decoration-color: #000080\"> </span> Local <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 5 <span style=\"color: #000080; text-decoration-color: #000080\"> </span>  2  <span style=\"color: #000080; text-decoration-color: #000080\"> </span>  1  <span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0 /10 </span> <span style=\"color: #800080; text-decoration-color: #800080\">  0%</span> <span style=\"color: #000080; text-decoration-color: #000080\"> </span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">                                                                                             </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m                                                                                             \u001b[0m\n",
       "\u001b[34m \u001b[0m\u001b[1;34m    \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m    \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m       \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m         \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m               \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m        \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m       \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m   \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m     \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m     \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m⏳ Completed\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\n",
       "\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m🔖\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m🆔\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m  🗓  \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mProject\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mPurpose      \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mType  \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m  \u001b[0m\u001b[1;33m▶\u001b[0m\u001b[1;34m  \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;33m♻\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mCPU\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mGPU\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m   Jobs \u001b[0m\u001b[1;33m✔\u001b[0m\u001b[1;34m   \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\n",
       "\u001b[34m ─────────────────────────────────────────────────────────────────────────────────────────── \u001b[0m\n",
       "\u001b[34m \u001b[0m \u001b[35m⠙ \u001b[0m \u001b[34m \u001b[0m 1  \u001b[34m \u001b[0m 03/08 \u001b[34m \u001b[0m MNIST   \u001b[34m \u001b[0m Test Protocol \u001b[34m \u001b[0m search \u001b[34m \u001b[0m Local \u001b[34m \u001b[0m 5 \u001b[34m \u001b[0m  2  \u001b[34m \u001b[0m  1  \u001b[34m \u001b[0m \u001b[37m 0 /10 \u001b[0m \u001b[35m  0%\u001b[0m \u001b[34m \u001b[0m\n",
       "\u001b[34m                                                                                             \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Print a summary of the last experiments\n",
    "sub_df = protocol.summary()\n",
    "\n",
    "# ... and a more detailed version\n",
    "sub_df = protocol.summary(full=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d2c9fe6-c040-4eaf-b477-bad3df93e4e1",
   "metadata": {},
   "source": [
    "If you want to ad-hoc change any of the stored attributes of an experiment, you can do so using the `update` method. Furthermore, you can change the experiment status using `abort` or `complete`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a6ab5f12-0c06-494f-ab77-f7ea37262d0d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">                                                                              </span>               \n",
       "<span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> 🔖 </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> 🆔 </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">   🗓   </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> Project </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> Purpose       </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> Type   </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">   </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">▶</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">   </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">♻</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> CPU </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> GPU </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span>               \n",
       "<span style=\"color: #000080; text-decoration-color: #000080\"> ──────────────────────────────────────────────────────────────────────────── </span>               \n",
       "<span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">✖</span>  <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 1  <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 03/08 <span style=\"color: #000080; text-decoration-color: #000080\"> </span> MNIST   <span style=\"color: #000080; text-decoration-color: #000080\"> </span> Test Protocol <span style=\"color: #000080; text-decoration-color: #000080\"> </span> search <span style=\"color: #000080; text-decoration-color: #000080\"> </span> Slurm <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 5 <span style=\"color: #000080; text-decoration-color: #000080\"> </span>  2  <span style=\"color: #000080; text-decoration-color: #000080\"> </span>  1  <span style=\"color: #000080; text-decoration-color: #000080\"> </span>               \n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">                                                                              </span>               \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m                                                                              \u001b[0m               \n",
       "\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m🔖\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m🆔\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m  🗓  \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mProject\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mPurpose      \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mType  \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m  \u001b[0m\u001b[1;33m▶\u001b[0m\u001b[1;34m  \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;33m♻\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mCPU\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mGPU\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m               \n",
       "\u001b[34m ──────────────────────────────────────────────────────────────────────────── \u001b[0m               \n",
       "\u001b[34m \u001b[0m \u001b[31m✖\u001b[0m  \u001b[34m \u001b[0m 1  \u001b[34m \u001b[0m 03/08 \u001b[34m \u001b[0m MNIST   \u001b[34m \u001b[0m Test Protocol \u001b[34m \u001b[0m search \u001b[34m \u001b[0m Slurm \u001b[34m \u001b[0m 5 \u001b[34m \u001b[0m  2  \u001b[34m \u001b[0m  1  \u001b[34m \u001b[0m               \n",
       "\u001b[34m                                                                              \u001b[0m               \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'aborted'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Update some element in the database\n",
    "protocol.update(e_id, \"exec_resource\", \"slurm-cluster\", save=False)\n",
    "\n",
    "# Abort the experiment - changes status\n",
    "protocol.abort(e_id, save=False)\n",
    "sub_df = protocol.summary()\n",
    "\n",
    "# Get the status of the experiment\n",
    "protocol.status(e_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03d2577e-d4d0-484c-8a9a-832e44a6e10e",
   "metadata": {},
   "source": [
    "If you would like to get a summary of all reported experiments, the last experiment and its resource requirements, `protocol.monitor()` does so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "25cb30e1-a36f-4307-9c6a-65a3f5c864d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'total': '1',\n",
       " 'run': '0',\n",
       " 'done': '0',\n",
       " 'aborted': '1',\n",
       " 'sge': '0',\n",
       " 'slurm': '1',\n",
       " 'gcp': '0',\n",
       " 'local': '0',\n",
       " 'report_gen': '0',\n",
       " 'gcs_stored': '0',\n",
       " 'retrieved': '0'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the monitoring data - used later in dashboard\n",
    "protocol_data = protocol.monitor()\n",
    "protocol_data[\"total_data\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2338033-85fa-4cbd-9d09-e60395a5aef4",
   "metadata": {},
   "source": [
    "Finally, you can also store other data specific to an experiment using an additional dictionary of data as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "046554ec-c8dd-4660-a283-b8ef464d387c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[15:20:05] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Added experiment <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span> to protocol.                       <a href=\"file:///Users/rob/anaconda3/envs/mle-toolbox/lib/python3.9/site-packages/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/anaconda3/envs/mle-toolbox/lib/python3.9/site-packages/mle_monitor/mle_protocol.py#170\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">170</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[15:20:05]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Added experiment \u001b[1;36m2\u001b[0m to protocol.                       \u001b]8;id=818970;file:///Users/rob/anaconda3/envs/mle-toolbox/lib/python3.9/site-packages/mle_monitor/mle_protocol.py\u001b\\\u001b[2mmle_protocol.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=795665;file:///Users/rob/anaconda3/envs/mle-toolbox/lib/python3.9/site-packages/mle_monitor/mle_protocol.py#170\u001b\\\u001b[2m170\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'lrate': 0.0003}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extra_data = {\"extra_config\": {\"lrate\": 3e-04}}\n",
    "e_id = protocol.add(meta_data, extra_data, save=False)\n",
    "protocol.get(e_id)[\"extra_config\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ffe6828-8077-4219-9cea-9f81fb0ce10b",
   "metadata": {},
   "source": [
    "## Syncing your Protocol DB with a GCS Bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7384b977-b9b3-4482-afec-dccb899ff598",
   "metadata": {},
   "source": [
    "If you would like to keep a remote copy of your protocol, you can also automatically sync your protocol database with a Google Cloud Storage (GCS) bucket. This is especially useful when running experiments on multiple resource and will require you to have created a GCP project and a GCS bucket. Furthermore you will have to provide you `.json` authentication key path. If you don't have one yet, have a look [here](https://cloud.google.com/docs/authentication/getting-started). Alternatively, just make sure that the environment variable `GOOGLE_APPLICATION_CREDENTIALS` is set to the right path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "63324b07-e05f-4ed6-9039-f57e26fbb124",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[15:38:01] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> No DB found in GCloud Storage - mle_protocol.db            <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">gcs_sync.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py#39\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">39</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[15:38:01]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m No DB found in GCloud Storage - mle_protocol.db            \u001b]8;id=165072;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py\u001b\\\u001b[2mgcs_sync.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=723900;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py#39\u001b\\\u001b[2m39\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> New DB will be created - mle-toolbox/mle-protocol          <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">gcs_sync.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py#40\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">40</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m New DB will be created - mle-toolbox/mle-protocol          \u001b]8;id=436921;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py\u001b\\\u001b[2mgcs_sync.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=297090;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py#40\u001b\\\u001b[2m40\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Pulled protocol from GCS bucket: mle-protocol.        <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#379\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">379</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Pulled protocol from GCS bucket: mle-protocol.        \u001b]8;id=793270;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\u001b\\\u001b[2mmle_protocol.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=283709;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#379\u001b\\\u001b[2m379\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Sync your protocol with a GCS bucket\n",
    "cloud_settings = {\n",
    "    \"project_name\": \"mle-toolbox\",        # Name of your GCP project\n",
    "    \"bucket_name\": \"mle-protocol\",        # Name of your GCS bucket\n",
    "    \"protocol_fname\": \"mle_protocol.db\",  # Name of DB file in GCS bucket\n",
    "    \"use_protocol_sync\": True,            # Whether to sync the protocol\n",
    "    \"use_results_storage\": False          # Whether to upload zipped dir at completion\n",
    "}\n",
    "protocol = MLEProtocol(protocol_fname=\"mle_protocol.db\",\n",
    "                       cloud_settings=cloud_settings,\n",
    "                       verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "14065522-5cda-485a-95d9-be9109c92004",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[15:38:06] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Added experiment <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> to protocol.                       <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#162\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">162</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[15:38:06]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Added experiment \u001b[1;36m1\u001b[0m to protocol.                       \u001b]8;id=53082;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\u001b\\\u001b[2mmle_protocol.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=761624;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#162\u001b\\\u001b[2m162\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Locally stored protocol: mle_protocol.db               <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#91\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">91</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Locally stored protocol: mle_protocol.db               \u001b]8;id=284160;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\u001b\\\u001b[2mmle_protocol.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=467582;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[15:38:07] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Send to GCloud Storage - mle_protocol.db                   <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">gcs_sync.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py#70\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">70</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[15:38:07]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Send to GCloud Storage - mle_protocol.db                   \u001b]8;id=904611;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py\u001b\\\u001b[2mgcs_sync.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=895801;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/protocol/gcs_sync.py#70\u001b\\\u001b[2m70\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Send protocol to GCS bucket: mle-protocol.            <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#364\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">364</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Send protocol to GCS bucket: mle-protocol.            \u001b]8;id=688284;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\u001b\\\u001b[2mmle_protocol.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=353419;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#364\u001b\\\u001b[2m364\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> GCS synced protocol: mle_protocol.db                   <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#97\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">97</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m GCS synced protocol: mle_protocol.db                   \u001b]8;id=494795;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\u001b\\\u001b[2mmle_protocol.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=925877;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#97\u001b\\\u001b[2m97\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "e_id = protocol.add(meta_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8aa22e7c-e46f-4be8-8e0e-aeb254ad4932",
   "metadata": {},
   "source": [
    "Finally, you can also choose to store the results of an experiment in the GCS bucket. In this case the protocol will upload a zipped version of your created `experiment_dir` to the bucket whenever you call `protocol.complete()`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c74100d5-a957-4a7e-ae87-4a4626babdc4",
   "metadata": {},
   "source": [
    "# Resource Monitoring with `MLEResource` 📉\n",
    "\n",
    "You can monitor your local machine, server or clusters using the `MLEResource`. If you are running this on a Google Colab, make sure to add a GPU accelerator!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fed9e8a7-74fd-4725-9bd6-31106018069d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mle_monitor import MLEResource\n",
    "\n",
    "resource = MLEResource(resource_name=\"local\")\n",
    "resource_data = resource.monitor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0c53b8d0-0093-4de4-868b-40f0b0652322",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['pid', 'p_name', 'mem_util', 'cpu_util', 'cmdline', 'total_cpu_util', 'total_mem_util'])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resource_data[\"user_data\"].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "260c0107-21e1-4c5b-8a84-c3cd96a91536",
   "metadata": {},
   "source": [
    "You can also monitor slurm or grid engine clusters by providing the queues/partitions to monitor in `monitor_config`:\n",
    "```python\n",
    "resource = MLEResource(\n",
    "    resource_name=\"slurm-cluster\",\n",
    "    monitor_config={\"partitions\": [\"<partition-1>\", \"<partition-2>\"]},\n",
    ")\n",
    "resource = MLEResource(\n",
    "    resource_name=\"sge-cluster\",\n",
    "    monitor_config={\"queues\": [\"<queue-1>\", \"<queue-2>\"]}\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70353a53-f0cb-4eae-8c18-93b7ef07eb14",
   "metadata": {},
   "source": [
    "# Dashboard Visualization with `MLEDashboard` 🎞️"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6daefdf4-0fab-4031-bc53-fa56bb8d6bb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mle_monitor import MLEDashboard\n",
    "\n",
    "dashboard = MLEDashboard(protocol, resource)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c65b012e-d8ef-4ef9-b8a3-8ac5098b06a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\">╭───────────────────────────────────────────────────────────────────────────────────────────╮</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\">│ </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080; font-weight: bold\">General Settings 💡</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\">                               __           </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080; font-weight: bold\">Thu Dec 9 15:38:18 2021 ⏰</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\"> │</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\">│                           _ __                                                            │</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\">│ • GCS Sync Protocol: </span><span style=\"color: #008000; text-decoration-color: #008000; background-color: #000080\">✔</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\">                 ____ ___  / /__           Author: </span><a href=\"https://twitter.com/RobertTLange\"><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080; text-decoration: underline\">@RobertTLange</span></a><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\"> 🐦 │</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\">│                           ____ ___  ____  ____  (_) /_____                                │</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\">│                           _____                                                           │</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000080\">╰───────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">╭─ Local - Util ─╮</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">╭──────────── Experiment Protocol Summary ─────────────╮</span><span style=\"color: #808000; text-decoration-color: #808000\">╭─ Total Number o─╮</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\">                                                    </span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">               </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\">          </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> 🔖 </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> 🆔 </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> 🗓  </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> P… </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> P… </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> … </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">▶</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">  </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">♻</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> … </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> G… </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> … </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #008000; text-decoration-color: #008000\">✔</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> … </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 1 <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 0 <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\"> ──────── </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> … </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">✔</span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\"> </span><span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 0 <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 0 <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> </span><span style=\"color: #800000; text-decoration-color: #800000\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> </span><span style=\"color: #800000; text-decoration-color: #800000\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> </span><span style=\"color: #800000; text-decoration-color: #800000\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">  </span><span style=\"color: #800000; text-decoration-color: #800000\"> </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\"> ────────────────────────────────────────────────── </span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">-</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #800080; text-decoration-color: #800080\">⠸ </span> <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 1  <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 1… <span style=\"color: #000080; text-decoration-color: #000080\"> </span> M… <span style=\"color: #000080; text-decoration-color: #000080\"> </span> T… <span style=\"color: #000080; text-decoration-color: #000080\"> </span> … <span style=\"color: #000080; text-decoration-color: #000080\"> </span> L… <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 5 <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 2 <span style=\"color: #000080; text-decoration-color: #000080\"> </span> 1  <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> - <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 0 <span style=\"color: #808000; text-decoration-color: #808000\"> </span>  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> </span><span style=\"color: #800000; text-decoration-color: #800000\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> </span><span style=\"color: #800000; text-decoration-color: #800000\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> </span><span style=\"color: #800000; text-decoration-color: #800000\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">  </span><span style=\"color: #800000; text-decoration-color: #800000\"> </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span> P… <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span>    <span style=\"color: #000080; text-decoration-color: #000080\"> </span>   <span style=\"color: #000080; text-decoration-color: #000080\"> </span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">               </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\"> ──────── </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\">                                                    </span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\">          </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\">          </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\"> ──────── </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #800000; text-decoration-color: #800000\">          </span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">              </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\"> … </span><span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> ──────────── </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">╰─────────────────╯</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">╭─ Last Experimen─╮</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">               </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">E-ID</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 1    <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Type</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> hyp… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Dir.</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> log… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Scr…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> tra… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Con…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> bas… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Sta…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> Run… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>      <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 🏃   <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Res…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> loc… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">               </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span> … <span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> ──────────── </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span>  <span style=\"color: #800000; text-decoration-color: #800000\"> </span><span style=\"font-weight: bold\"> … </span><span style=\"color: #800000; text-decoration-color: #800000\"> </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">              </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">╰─────────────────╯</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">╭─ Experiment Com─╮</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">               </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Con…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 2/5  <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Tot…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 10   <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Jobs</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>      <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\"># </span>   <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 2    <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Bat…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>      <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Job…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 5    <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Tim…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 0:0… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Sta…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 12/… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Time</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 15:… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">~ </span>   <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 12/… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Stop</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 01:… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Time</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>      <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">~ </span>   <span style=\"color: #808000; text-decoration-color: #808000\"> </span> 0:1… <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Dur…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>      <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">---…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">---…</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> ⏳   <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #800080; text-decoration-color: #800080\">…</span> <span style=\"color: #800080; text-decoration-color: #800080\"> …</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Jobs</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span>      <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✔</span>    <span style=\"color: #808000; text-decoration-color: #808000\"> </span>      <span style=\"color: #808000; text-decoration-color: #808000\"> </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #808000; text-decoration-color: #808000\">               </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span>                                                      <span style=\"color: #0000ff; text-decoration-color: #0000ff\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                 <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">╰──────────────────────────────────────────────────────╯</span><span style=\"color: #808000; text-decoration-color: #808000\">╰─────────────────╯</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">╭─ CPU: 0/8T - Memory: ─╮╭─ Protocol Timeline: T─╮╭─ Protocol Timeline: E─╮</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">    ┌───────────────…</span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">    ┌───────────────…</span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">    ┌───────────────…</span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">1.00┤</span><span style=\"background-color: #000000\"> </span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•••</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\"> % CPU Util.</span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">2.00┤</span><span style=\"background-color: #000000\"> </span><span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">•••</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\"> Total </span><span style=\"background-color: #000000\">     </span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">1.00┤</span><span style=\"background-color: #000000\"> </span><span style=\"color: #000080; text-decoration-color: #000080; background-color: #000000\">███</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\"> SC </span><span style=\"background-color: #000000\">        </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>                     <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>                     <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">█████████████████</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>    <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.80┤</span><span style=\"background-color: #000000\"> </span><span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">•••</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\"> % Mem Util.</span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">1.60┤           </span><span style=\"background-color: #000000\">     </span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.80┤</span><span style=\"background-color: #000000\"> </span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">███</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\"> MC </span><span style=\"background-color: #000000\">        </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•</span><span style=\"background-color: #000000\">                  </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>  <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>                     <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">█████████████████</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>    <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.60┤                </span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">1.20┤</span><span style=\"background-color: #000000\">                </span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.60┤</span><span style=\"background-color: #000000\"> </span><span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">███</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\"> HS </span><span style=\"background-color: #000000\">        </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•••</span><span style=\"background-color: #000000\"> </span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•</span><span style=\"background-color: #000000\">      </span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•</span><span style=\"background-color: #000000\"> </span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•</span><span style=\"background-color: #000000\">   </span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•</span><span style=\"background-color: #000000\"> </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>  <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">•</span><span style=\"background-color: #000000\">                 </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>   <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">█████████████████</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>    <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.40┤</span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•••••••••••••••…</span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.80┤</span><span style=\"background-color: #000000\">                </span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.40┤        </span><span style=\"background-color: #000000\">        </span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.20┤</span><span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">•••••••••••••••…</span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>                     <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">█████████████████</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>    <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.00┤</span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">••••</span><span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">•</span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•••</span><span style=\"background-color: #000000\">  </span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•••</span><span style=\"background-color: #000000\">   </span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.40┤</span><span style=\"background-color: #000000\">                </span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.21┤</span><span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">███████████████…</span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">••</span><span style=\"background-color: #000000\"> </span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">••</span><span style=\"background-color: #000000\"> </span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•</span><span style=\"background-color: #000000\">      </span><span style=\"color: #800000; text-decoration-color: #800000; background-color: #000000\">•</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>       <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>                     <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">█████████████████</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>    <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">    └┬──────────────…</span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.00┤</span><span style=\"background-color: #000000\">                </span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">0.01┤</span><span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">███████████████…</span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">     12/05-20:05:03  </span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>                     <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #000000\">█████████████████</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">│</span>    <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">12/09-15:38:18 </span>       <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">    └───────────────…</span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">    └────────┬──────…</span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">                12/0…</span> <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">                    …</span> <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #000000\">15:38            </span>     <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                <span style=\"color: #800000; text-decoration-color: #800000\">│</span><span style=\"color: #808000; text-decoration-color: #808000\">│</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">││</span>                       <span style=\"color: #808000; text-decoration-color: #808000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">╰────────────────╯</span><span style=\"color: #808000; text-decoration-color: #808000\">╰───────────────────────╯╰───────────────────────╯╰───────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[37;44m╭───────────────────────────────────────────────────────────────────────────────────────────╮\u001b[0m\n",
       "\u001b[37;44m│\u001b[0m\u001b[37;44m \u001b[0m\u001b[1;37;44mGeneral Settings 💡\u001b[0m\u001b[37;44m       \u001b[0m\u001b[37;44m                        __           \u001b[0m\u001b[1;37;44mThu Dec 9 15:38:18 2021 ⏰\u001b[0m\u001b[37;44m \u001b[0m\u001b[37;44m│\u001b[0m\n",
       "\u001b[37;44m│\u001b[0m\u001b[37;44m \u001b[0m\u001b[37;44m                          \u001b[0m\u001b[37;44m_ __                                 \u001b[0m\u001b[37;44m                          \u001b[0m\u001b[37;44m \u001b[0m\u001b[37;44m│\u001b[0m\n",
       "\u001b[37;44m│\u001b[0m\u001b[37;44m \u001b[0m\u001b[37;44m• GCS Sync Protocol: \u001b[0m\u001b[32;44m✔\u001b[0m\u001b[37;44m    \u001b[0m\u001b[37;44m             ____ ___  / /__         \u001b[0m\u001b[37;44m  Author: \u001b[0m\u001b]8;id=914304;https://twitter.com/RobertTLange\u001b\\\u001b[4;37;44m@RobertTLange\u001b[0m\u001b]8;;\u001b\\\u001b[37;44m 🐦\u001b[0m\u001b[37;44m \u001b[0m\u001b[37;44m│\u001b[0m\n",
       "\u001b[37;44m│\u001b[0m\u001b[37;44m \u001b[0m\u001b[37;44m                          \u001b[0m\u001b[37;44m____ ___  ____  ____  (_) /_____     \u001b[0m\u001b[37;44m                          \u001b[0m\u001b[37;44m \u001b[0m\u001b[37;44m│\u001b[0m\n",
       "\u001b[37;44m│\u001b[0m\u001b[37;44m \u001b[0m\u001b[37;44m                          \u001b[0m\u001b[37;44m_____                                \u001b[0m\u001b[37;44m                          \u001b[0m\u001b[37;44m \u001b[0m\u001b[37;44m│\u001b[0m\n",
       "\u001b[37;44m╰───────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
       "\u001b[31m╭─\u001b[0m\u001b[31m Local - Util \u001b[0m\u001b[31m─╮\u001b[0m\u001b[94m╭─\u001b[0m\u001b[94m─────────── Experiment Protocol Summary ────────────\u001b[0m\u001b[94m─╮\u001b[0m\u001b[33m╭─\u001b[0m\u001b[33m Total Number o\u001b[0m\u001b[33m─╮\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m \u001b[34m                                                    \u001b[0m \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m               \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m          \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m \u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m🔖\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m🆔\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m🗓 \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mP…\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mP…\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m…\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;33m▶\u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;33m♻\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m…\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34mG…\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m…\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33m…\u001b[0m \u001b[33m \u001b[0m  \u001b[33m \u001b[0m \u001b[32m✔\u001b[0m \u001b[33m \u001b[0m  \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m \u001b[0m \u001b[31m \u001b[0m \u001b[31m \u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m   \u001b[34m \u001b[0m    \u001b[34m \u001b[0m   \u001b[34m \u001b[0m   \u001b[34m \u001b[0m    \u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m…\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m 1 \u001b[33m \u001b[0m  \u001b[33m \u001b[0m 0 \u001b[33m \u001b[0m  \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m ──────── \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m   \u001b[34m \u001b[0m    \u001b[34m \u001b[0m   \u001b[34m \u001b[0m   \u001b[34m \u001b[0m    \u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;34m…\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33m…\u001b[0m \u001b[33m \u001b[0m  \u001b[33m \u001b[0m \u001b[1;33m…\u001b[0m \u001b[33m \u001b[0m  \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m \u001b[0m \u001b[31m \u001b[0m \u001b[31m \u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m   \u001b[34m \u001b[0m    \u001b[34m \u001b[0m   \u001b[34m \u001b[0m   \u001b[34m \u001b[0m    \u001b[34m \u001b[0m\u001b[1;34m \u001b[0m\u001b[1;33m✔\u001b[0m\u001b[1;34m \u001b[0m\u001b[34m \u001b[0m \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m 0 \u001b[33m \u001b[0m  \u001b[33m \u001b[0m 0 \u001b[33m \u001b[0m  \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m \u001b[0m\u001b[2m \u001b[0m\u001b[31m \u001b[0m\u001b[2m \u001b[0m\u001b[31m \u001b[0m\u001b[2m \u001b[0m\u001b[31m \u001b[0m\u001b[2m  \u001b[0m\u001b[31m \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m \u001b[34m ────────────────────────────────────────────────── \u001b[0m \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33m-\u001b[0m \u001b[33m \u001b[0m  \u001b[33m \u001b[0m \u001b[1;33m…\u001b[0m \u001b[33m \u001b[0m  \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m \u001b[0m \u001b[31m \u001b[0m \u001b[31m \u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m \u001b[34m \u001b[0m \u001b[35m⠸ \u001b[0m \u001b[34m \u001b[0m 1  \u001b[34m \u001b[0m 1… \u001b[34m \u001b[0m M… \u001b[34m \u001b[0m T… \u001b[34m \u001b[0m … \u001b[34m \u001b[0m L… \u001b[34m \u001b[0m 5 \u001b[34m \u001b[0m 2 \u001b[34m \u001b[0m 1  \u001b[34m \u001b[0m   \u001b[34m \u001b[0m \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m - \u001b[33m \u001b[0m  \u001b[33m \u001b[0m 0 \u001b[33m \u001b[0m  \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m \u001b[0m\u001b[2m \u001b[0m\u001b[31m \u001b[0m\u001b[2m \u001b[0m\u001b[31m \u001b[0m\u001b[2m \u001b[0m\u001b[31m \u001b[0m\u001b[2m  \u001b[0m\u001b[31m \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m    \u001b[34m \u001b[0m P… \u001b[34m \u001b[0m   \u001b[34m \u001b[0m    \u001b[34m \u001b[0m   \u001b[34m \u001b[0m   \u001b[34m \u001b[0m    \u001b[34m \u001b[0m   \u001b[34m \u001b[0m \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m               \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m ──────── \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m \u001b[34m                                                    \u001b[0m \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m \u001b[0m \u001b[31m \u001b[0m \u001b[31m \u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m          \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m          \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m \u001b[0m \u001b[31m \u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m \u001b[31m \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m ──────── \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[31m          \u001b[0m   \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m              \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m\u001b[1;31m \u001b[0m\u001b[1;31m…\u001b[0m\u001b[1;31m \u001b[0m\u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m ──────────── \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m╰─────────────────╯\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m╭─\u001b[0m\u001b[33m Last Experimen\u001b[0m\u001b[33m─╮\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m               \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mE-ID\u001b[0m \u001b[33m \u001b[0m 1    \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mType\u001b[0m \u001b[33m \u001b[0m hyp… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mDir.\u001b[0m \u001b[33m \u001b[0m log… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mScr…\u001b[0m \u001b[33m \u001b[0m tra… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mCon…\u001b[0m \u001b[33m \u001b[0m bas… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mSta…\u001b[0m \u001b[33m \u001b[0m Run… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m      \u001b[33m \u001b[0m 🏃   \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mRes…\u001b[0m \u001b[33m \u001b[0m loc… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m               \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m … \u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m ──────────── \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m  \u001b[31m \u001b[0m\u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m\u001b[31m \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m              \u001b[0m \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m╰─────────────────╯\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m╭─\u001b[0m\u001b[33m Experiment Com\u001b[0m\u001b[33m─╮\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m               \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mCon…\u001b[0m \u001b[33m \u001b[0m 2/5  \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mTot…\u001b[0m \u001b[33m \u001b[0m 10   \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mJobs\u001b[0m \u001b[33m \u001b[0m      \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33m# \u001b[0m   \u001b[33m \u001b[0m 2    \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mBat…\u001b[0m \u001b[33m \u001b[0m      \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mJob…\u001b[0m \u001b[33m \u001b[0m 5    \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mTim…\u001b[0m \u001b[33m \u001b[0m 0:0… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mSta…\u001b[0m \u001b[33m \u001b[0m 12/… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mTime\u001b[0m \u001b[33m \u001b[0m 15:… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33m~ \u001b[0m   \u001b[33m \u001b[0m 12/… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mStop\u001b[0m \u001b[33m \u001b[0m 01:… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mTime\u001b[0m \u001b[33m \u001b[0m      \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33m~ \u001b[0m   \u001b[33m \u001b[0m 0:1… \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mDur…\u001b[0m \u001b[33m \u001b[0m      \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33m---…\u001b[0m \u001b[33m \u001b[0m \u001b[1;33m---…\u001b[0m \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m ⏳   \u001b[33m \u001b[0m \u001b[35m…\u001b[0m \u001b[35m …\u001b[0m \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;33mJobs\u001b[0m \u001b[33m \u001b[0m      \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m \u001b[0m \u001b[1;32m✔\u001b[0m    \u001b[33m \u001b[0m      \u001b[33m \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33m               \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m│\u001b[0m                                                      \u001b[94m│\u001b[0m\u001b[33m│\u001b[0m                 \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[94m╰──────────────────────────────────────────────────────╯\u001b[0m\u001b[33m╰─────────────────╯\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m╭─\u001b[0m\u001b[33m CPU: 0/8T - Memory: \u001b[0m\u001b[33m─╮\u001b[0m\u001b[33m╭─\u001b[0m\u001b[33m Protocol Timeline: T\u001b[0m\u001b[33m─╮\u001b[0m\u001b[33m╭─\u001b[0m\u001b[33m Protocol Timeline: E\u001b[0m\u001b[33m─╮\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m    ┌───────────────…\u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m    ┌───────────────…\u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m    ┌───────────────…\u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m1.00┤\u001b[0m\u001b[40m \u001b[0m\u001b[31;40m•••\u001b[0m\u001b[37;40m % CPU Util.\u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m2.00┤\u001b[0m\u001b[40m \u001b[0m\u001b[33;40m•••\u001b[0m\u001b[37;40m Total \u001b[0m\u001b[40m     \u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m1.00┤\u001b[0m\u001b[40m \u001b[0m\u001b[34;40m███\u001b[0m\u001b[37;40m SC \u001b[0m\u001b[40m        \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m│\u001b[0m                     \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m│\u001b[0m                     \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33;40m█████████████████\u001b[0m\u001b[37;40m│\u001b[0m    \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.80┤\u001b[0m\u001b[40m \u001b[0m\u001b[33;40m•••\u001b[0m\u001b[37;40m % Mem Util.\u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m1.60┤           \u001b[0m\u001b[40m     \u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.80┤\u001b[0m\u001b[40m \u001b[0m\u001b[31;40m███\u001b[0m\u001b[37;40m MC \u001b[0m\u001b[40m        \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[31;40m•\u001b[0m\u001b[40m                  \u001b[0m\u001b[37;40m│\u001b[0m  \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m│\u001b[0m                     \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33;40m█████████████████\u001b[0m\u001b[37;40m│\u001b[0m    \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.60┤                \u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m1.20┤\u001b[0m\u001b[40m                \u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.60┤\u001b[0m\u001b[40m \u001b[0m\u001b[33;40m███\u001b[0m\u001b[37;40m HS \u001b[0m\u001b[40m        \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[31;40m•••\u001b[0m\u001b[40m \u001b[0m\u001b[31;40m•\u001b[0m\u001b[40m      \u001b[0m\u001b[31;40m•\u001b[0m\u001b[40m \u001b[0m\u001b[31;40m•\u001b[0m\u001b[40m   \u001b[0m\u001b[31;40m•\u001b[0m\u001b[40m \u001b[0m\u001b[37;40m│\u001b[0m  \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33;40m•\u001b[0m\u001b[40m                 \u001b[0m\u001b[37;40m│\u001b[0m   \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33;40m█████████████████\u001b[0m\u001b[37;40m│\u001b[0m    \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.40┤\u001b[0m\u001b[31;40m•••••••••••••••…\u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.80┤\u001b[0m\u001b[40m                \u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.40┤        \u001b[0m\u001b[40m        \u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.20┤\u001b[0m\u001b[33;40m•••••••••••••••…\u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m│\u001b[0m                     \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33;40m█████████████████\u001b[0m\u001b[37;40m│\u001b[0m    \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.00┤\u001b[0m\u001b[31;40m••••\u001b[0m\u001b[33;40m•\u001b[0m\u001b[31;40m•••\u001b[0m\u001b[40m  \u001b[0m\u001b[31;40m•••\u001b[0m\u001b[40m   \u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.40┤\u001b[0m\u001b[40m                \u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.21┤\u001b[0m\u001b[33;40m███████████████…\u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[31;40m••\u001b[0m\u001b[40m \u001b[0m\u001b[31;40m••\u001b[0m\u001b[40m \u001b[0m\u001b[31;40m•\u001b[0m\u001b[40m      \u001b[0m\u001b[31;40m•\u001b[0m\u001b[37;40m│\u001b[0m       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m│\u001b[0m                     \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33;40m█████████████████\u001b[0m\u001b[37;40m│\u001b[0m    \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m    └┬──────────────…\u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.00┤\u001b[0m\u001b[40m                \u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m0.01┤\u001b[0m\u001b[33;40m███████████████…\u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m     12/05-20:05:03  \u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m│\u001b[0m                     \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[33;40m█████████████████\u001b[0m\u001b[37;40m│\u001b[0m    \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m12/09-15:38:18 \u001b[0m       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m    └───────────────…\u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m    └────────┬──────…\u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m                12/0…\u001b[0m \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m                    …\u001b[0m \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m \u001b[37;40m15:38            \u001b[0m     \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                \u001b[31m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\u001b[33m│\u001b[0m                       \u001b[33m│\u001b[0m\n",
       "\u001b[31m╰────────────────╯\u001b[0m\u001b[33m╰───────────────────────╯\u001b[0m\u001b[33m╰───────────────────────╯\u001b[0m\u001b[33m╰───────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get a static snapshot of the protocol & resource utilisation\n",
    "# Note: This will look a lot nicer in your terminal!\n",
    "dashboard.snapshot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0ca1ea3-9824-47ca-92ee-f62ed3a7a3de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run monitoring in while loop - dashboard\n",
    "dashboard.live()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d63912fe-3c40-455a-921d-4bce4422a896",
   "metadata": {},
   "source": [
    "- Add widget animation!/screenshot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8467015-0f7c-4576-95c6-2c698f5e8680",
   "metadata": {},
   "source": [
    "# Integration with the MLE-Infrastructure Ecosystem 🔺\n",
    "## Running a Hyperparameter Search for Multiple Random Seeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "277fcec3-11d3-4a13-81d6-bc261d6a8ebf",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from mle_hyperopt import RandomSearch\n",
    "    from mle_scheduler import MLEQueue\n",
    "    from mle_logging import load_meta_log\n",
    "except:\n",
    "    !pip install -q mle-hyperopt mle-scheduler mle-logging\n",
    "    !pip install --upgrade rich\n",
    "    from mle_hyperopt import RandomSearch\n",
    "    from mle_scheduler import MLEQueue\n",
    "    from mle_logging import load_meta_log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c58b0ec-af23-45a4-95d9-1e54a1040476",
   "metadata": {},
   "source": [
    "We again start by adding an experiment to the protocol at launch time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "cffe7cdd-de5e-4809-9206-a9e87b76f916",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[15:38:26] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Added experiment <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span> to protocol.                       <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#162\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">162</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[15:38:26]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Added experiment \u001b[1;36m2\u001b[0m to protocol.                       \u001b]8;id=165623;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\u001b\\\u001b[2mmle_protocol.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=63651;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#162\u001b\\\u001b[2m162\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Locally stored protocol: mle_protocol.db               <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#91\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">91</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Locally stored protocol: mle_protocol.db               \u001b]8;id=144531;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\u001b\\\u001b[2mmle_protocol.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=493805;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Load (existing) protocol database and add experiment data\n",
    "protocol_db = MLEProtocol(\"mle_protocol.db\", verbose=True)\n",
    "meta_data = {\n",
    "    \"purpose\": \"random search\",  # Purpose of experiment\n",
    "    \"project_name\": \"surrogate\",  # Project name of experiment\n",
    "    \"exec_resource\": \"local\",  # Resource jobs are run on\n",
    "    \"experiment_dir\": \"logs_search\",  # Experiment log storage directory\n",
    "    \"experiment_type\": \"hyperparameter-search\",  # Type of experiment to run\n",
    "    \"base_fname\": \"train.py\",  # Main code script to execute\n",
    "    \"config_fname\": \"base_config.json\",  # Config file path of experiment\n",
    "    \"num_seeds\": 2,  # Number of evaluations seeds\n",
    "    \"num_total_jobs\": 4,  # Number of total jobs to run\n",
    "    \"num_jobs_per_batch\": 4,  # Number of jobs in single batch\n",
    "    \"num_job_batches\": 1,  # Number of sequential job batches\n",
    "    \"time_per_job\": \"00:00:02\",  # Expected duration: days-hours-minutes\n",
    "}\n",
    "new_experiment_id = protocol_db.add(meta_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ace6804-6f2b-457a-9f6d-4dc78ed912e0",
   "metadata": {},
   "source": [
    "Afterwards, we leverage `mle-hyperopt` to instantiate a random search strategy with its parameter space. We then ask for two configurations and store them as `.yaml` files in our working directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7cd0cf2e-dce9-4e5f-8270-67ffa56809df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">    <span style=\"font-style: italic\">                MLE-Hyperopt Random Search Hyperspace 🚀                </span>    \n",
       "                                                                                \n",
       "     <span style=\"font-weight: bold\"> 🌻 Variable </span> <span style=\"font-weight: bold\"> Type        </span> <span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\"> Search Range ↔                           </span>     \n",
       "     ──────────────────────────────────────────────────────────────────────     \n",
       "      arch          categorical  <span style=\"color: #800000; text-decoration-color: #800000\"> ['mlp', 'cnn']                           </span>     \n",
       "      lrate         real         <span style=\"color: #800000; text-decoration-color: #800000\"> Begin: 0.1, End: 0.5, Prior: log-uniform </span>     \n",
       "      batch_size    integer      <span style=\"color: #800000; text-decoration-color: #800000\"> Begin: 1, End: 5, Prior: uniform         </span>     \n",
       "                                                                                \n",
       "</pre>\n"
      ],
      "text/plain": [
       "    \u001b[3m                MLE-Hyperopt Random Search Hyperspace 🚀                \u001b[0m    \n",
       "                                                                                \n",
       "     \u001b[1m \u001b[0m\u001b[1m🌻 Variable\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1mType       \u001b[0m\u001b[1m \u001b[0m \u001b[1;31m \u001b[0m\u001b[1;31mSearch Range ↔                          \u001b[0m\u001b[1;31m \u001b[0m     \n",
       "     ──────────────────────────────────────────────────────────────────────     \n",
       "      arch          categorical  \u001b[31m \u001b[0m\u001b[31m['mlp', 'cnn']                          \u001b[0m\u001b[31m \u001b[0m     \n",
       "      lrate         real         \u001b[31m \u001b[0m\u001b[31mBegin: 0.1, End: 0.5, Prior: log-uniform\u001b[0m\u001b[31m \u001b[0m     \n",
       "      batch_size    integer      \u001b[31m \u001b[0m\u001b[31mBegin: 1, End: 5, Prior: uniform        \u001b[0m\u001b[31m \u001b[0m     \n",
       "                                                                                \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'arch': 'mlp', 'lrate': 0.360379148648584, 'batch_size': 3},\n",
       " {'arch': 'cnn', 'lrate': 0.26208630215377515, 'batch_size': 2}]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Instantiate random search class\n",
    "strategy = RandomSearch(\n",
    "    real={\"lrate\": {\"begin\": 0.1, \"end\": 0.5, \"prior\": \"log-uniform\"}},\n",
    "    integer={\"batch_size\": {\"begin\": 1, \"end\": 5, \"prior\": \"uniform\"}},\n",
    "    categorical={\"arch\": [\"mlp\", \"cnn\"]},\n",
    "    verbose=True,\n",
    ")\n",
    "\n",
    "# Ask for configurations to evaluate & run parallel eval of seeds * configs\n",
    "configs, config_fnames = strategy.ask(2, store=True)\n",
    "configs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "672d70dd-ee77-40c7-a720-e0015c78e5ed",
   "metadata": {},
   "source": [
    "Next, we can use a `MLEQueue` from `mle-scheduler` to run our training script `train.py` for our two configurations and two different random seeds. `train.py` implements a simple surrogate training loop, which logs some statistics with the help of `mle-logging`. Afterwards, we merge the resulting logs into a single `meta_log.hdf5` and retrieve the mean (over seeds) test loss score for both configurations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6b700165-75b7-4b7c-a83c-1338c1645c43",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d9861ce83a384229baf7afaf9b94ee79",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[15:38:35] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Locally stored protocol: mle_protocol.db               <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#91\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">91</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[15:38:35]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Locally stored protocol: mle_protocol.db               \u001b]8;id=974722;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\u001b\\\u001b[2mmle_protocol.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=774122;file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Locally stored protocol: mle_protocol.db               <a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">mle_protocol.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/rob/Dropbox/core-code/mle-infrastructure/mle-monitor/mle_monitor/mle_protocol.py#91\">