\n \n \nmlflow_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: mlflow-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.10, pipelines.kubeflow.org/pipeline_compilation_time: '2022-01-19T14:14:11.999807',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"kernel\", \"type\":\n \"String\"}, {\"name\": \"model_name\", \"type\": \"String\"}], \"name\": \"mlflow_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.10}\nspec:\n entrypoint: mlflow-pipeline\n templates:\n - name: load-iris-data\n container:\n args: [--data, /tmp/outputs/data/data, --target, /tmp/outputs/target/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'pandas' 'scikit-learn' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location 'pandas' 'scikit-learn' --user)\n && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n\n def load_iris_data(\n data_path,\n target_path,\n ):\n import pandas as pd\n from sklearn.datasets import load_iris\n\n iris = load_iris()\n\n data = pd.DataFrame(iris[\"data\"], columns=iris[\"feature_names\"])\n target = pd.DataFrame(iris[\"target\"], columns=[\"target\"])\n\n data.to_csv(data_path, index=False)\n target.to_csv(target_path, index=False)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Load iris data', description='')\n _parser.add_argument(\"--data\", dest=\"data_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--target\", dest=\"target_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = load_iris_data(**_parsed_args)\n image: python:3.7\n outputs:\n artifacts:\n - {name: load-iris-data-data, path: /tmp/outputs/data/data}\n - {name: load-iris-data-target, path: /tmp/outputs/target/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--data\", {\"outputPath\": \"data\"}, \"--target\", {\"outputPath\": \"target\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''pandas'' ''scikit-learn'' ||\n PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n ''pandas'' ''scikit-learn'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef load_iris_data(\\n data_path,\\n target_path,\\n):\\n import\n pandas as pd\\n from sklearn.datasets import load_iris\\n\\n iris = load_iris()\\n\\n data\n = pd.DataFrame(iris[\\\"data\\\"], columns=iris[\\\"feature_names\\\"])\\n target\n = pd.DataFrame(iris[\\\"target\\\"], columns=[\\\"target\\\"])\\n\\n data.to_csv(data_path,\n index=False)\\n target.to_csv(target_path, index=False)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Load iris data'', description='''')\\n_parser.add_argument(\\\"--data\\\",\n dest=\\\"data_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--target\\\", dest=\\\"target_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = load_iris_data(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"name\": \"Load iris data\", \"outputs\": [{\"name\":\n \"data\", \"type\": \"csv\"}, {\"name\": \"target\", \"type\": \"csv\"}]}', pipelines.kubeflow.org/component_ref: '{}'}\n - name: mlflow-pipeline\n inputs:\n parameters:\n - {name: kernel}\n - {name: model_name}\n dag:\n tasks:\n - {name: load-iris-data, template: load-iris-data}\n - name: train-from-csv\n template: train-from-csv\n dependencies: [load-iris-data]\n arguments:\n parameters:\n - {name: kernel, value: '{{inputs.parameters.kernel}}'}\n artifacts:\n - {name: load-iris-data-data, from: '{{tasks.load-iris-data.outputs.artifacts.load-iris-data-data}}'}\n - {name: load-iris-data-target, from: '{{tasks.load-iris-data.outputs.artifacts.load-iris-data-target}}'}\n - name: upload-sklearn-model-to-mlflow\n template: upload-sklearn-model-to-mlflow\n dependencies: [train-from-csv]\n arguments:\n parameters:\n - {name: model_name, value: '{{inputs.parameters.model_name}}'}\n artifacts:\n - {name: train-from-csv-conda_env, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-conda_env}}'}\n - {name: train-from-csv-input_example, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-input_example}}'}\n - {name: train-from-csv-model, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-model}}'}\n - {name: train-from-csv-signature, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-signature}}'}\n - name: train-from-csv\n container:\n args: [--train-data, /tmp/inputs/train_data/data, --train-target, /tmp/inputs/train_target/data,\n --kernel, '{{inputs.parameters.kernel}}', --model, /tmp/outputs/model/data,\n --input-example, /tmp/outputs/input_example/data, --signature, /tmp/outputs/signature/data,\n --conda-env, /tmp/outputs/conda_env/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'dill' 'pandas' 'scikit-learn' 'mlflow' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location 'dill' 'pandas' 'scikit-learn'\n 'mlflow' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n\n def train_from_csv(\n train_data_path,\n train_target_path,\n model_path,\n input_example_path,\n signature_path,\n conda_env_path,\n kernel,\n ):\n import dill\n import pandas as pd\n from sklearn.svm import SVC\n\n from mlflow.models.signature import infer_signature\n from mlflow.utils.environment import _mlflow_conda_env\n\n train_data = pd.read_csv(train_data_path)\n train_target = pd.read_csv(train_target_path)\n\n clf = SVC(kernel=kernel)\n clf.fit(train_data, train_target)\n\n with open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n\n input_example = train_data.sample(1)\n with open(input_example_path, \"wb\") as file_writer:\n dill.dump(input_example, file_writer)\n\n signature = infer_signature(train_data, clf.predict(train_data))\n with open(signature_path, \"wb\") as file_writer:\n dill.dump(signature, file_writer)\n\n conda_env = _mlflow_conda_env(\n additional_pip_deps=[\"dill\", \"pandas\", \"scikit-learn\"]\n )\n with open(conda_env_path, \"wb\") as file_writer:\n dill.dump(conda_env, file_writer)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Train from csv', description='')\n _parser.add_argument(\"--train-data\", dest=\"train_data_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--train-target\", dest=\"train_target_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--kernel\", dest=\"kernel\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--model\", dest=\"model_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--input-example\", dest=\"input_example_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--signature\", dest=\"signature_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--conda-env\", dest=\"conda_env_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = train_from_csv(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: kernel}\n artifacts:\n - {name: load-iris-data-data, path: /tmp/inputs/train_data/data}\n - {name: load-iris-data-target, path: /tmp/inputs/train_target/data}\n outputs:\n artifacts:\n - {name: train-from-csv-conda_env, path: /tmp/outputs/conda_env/data}\n - {name: train-from-csv-input_example, path: /tmp/outputs/input_example/data}\n - {name: train-from-csv-model, path: /tmp/outputs/model/data}\n - {name: train-from-csv-signature, path: /tmp/outputs/signature/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--train-data\", {\"inputPath\": \"train_data\"}, \"--train-target\",\n {\"inputPath\": \"train_target\"}, \"--kernel\", {\"inputValue\": \"kernel\"}, \"--model\",\n {\"outputPath\": \"model\"}, \"--input-example\", {\"outputPath\": \"input_example\"},\n \"--signature\", {\"outputPath\": \"signature\"}, \"--conda-env\", {\"outputPath\":\n \"conda_env\"}], \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location ''dill'' ''pandas''\n ''scikit-learn'' ''mlflow'' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m\n pip install --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn''\n ''mlflow'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef train_from_csv(\\n train_data_path,\\n train_target_path,\\n model_path,\\n input_example_path,\\n signature_path,\\n conda_env_path,\\n kernel,\\n):\\n import\n dill\\n import pandas as pd\\n from sklearn.svm import SVC\\n\\n from\n mlflow.models.signature import infer_signature\\n from mlflow.utils.environment\n import _mlflow_conda_env\\n\\n train_data = pd.read_csv(train_data_path)\\n train_target\n = pd.read_csv(train_target_path)\\n\\n clf = SVC(kernel=kernel)\\n clf.fit(train_data,\n train_target)\\n\\n with open(model_path, mode=\\\"wb\\\") as file_writer:\\n dill.dump(clf,\n file_writer)\\n\\n input_example = train_data.sample(1)\\n with open(input_example_path,\n \\\"wb\\\") as file_writer:\\n dill.dump(input_example, file_writer)\\n\\n signature\n = infer_signature(train_data, clf.predict(train_data))\\n with open(signature_path,\n \\\"wb\\\") as file_writer:\\n dill.dump(signature, file_writer)\\n\\n conda_env\n = _mlflow_conda_env(\\n additional_pip_deps=[\\\"dill\\\", \\\"pandas\\\",\n \\\"scikit-learn\\\"]\\n )\\n with open(conda_env_path, \\\"wb\\\") as file_writer:\\n dill.dump(conda_env,\n file_writer)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Train\n from csv'', description='''')\\n_parser.add_argument(\\\"--train-data\\\", dest=\\\"train_data_path\\\",\n type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--train-target\\\",\n dest=\\\"train_target_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--kernel\\\",\n dest=\\\"kernel\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--model\\\",\n dest=\\\"model_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--input-example\\\", dest=\\\"input_example_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--signature\\\",\n dest=\\\"signature_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--conda-env\\\", dest=\\\"conda_env_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = train_from_csv(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"train_data\", \"type\": \"csv\"},\n {\"name\": \"train_target\", \"type\": \"csv\"}, {\"name\": \"kernel\", \"type\": \"String\"}],\n \"name\": \"Train from csv\", \"outputs\": [{\"name\": \"model\", \"type\": \"dill\"},\n {\"name\": \"input_example\", \"type\": \"dill\"}, {\"name\": \"signature\", \"type\":\n \"dill\"}, {\"name\": \"conda_env\", \"type\": \"dill\"}]}', pipelines.kubeflow.org/component_ref: '{}',\n pipelines.kubeflow.org/arguments.parameters: '{\"kernel\": \"{{inputs.parameters.kernel}}\"}'}\n - name: upload-sklearn-model-to-mlflow\n container:\n args: [--model-name, '{{inputs.parameters.model_name}}', --model, /tmp/inputs/model/data,\n --input-example, /tmp/inputs/input_example/data, --signature, /tmp/inputs/signature/data,\n --conda-env, /tmp/inputs/conda_env/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'dill' 'pandas' 'scikit-learn' 'mlflow' 'boto3' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location 'dill' 'pandas' 'scikit-learn'\n 'mlflow' 'boto3' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def upload_sklearn_model_to_mlflow(\n model_name,\n model_path,\n input_example_path,\n signature_path,\n conda_env_path,\n ):\n import os\n import dill\n from mlflow.sklearn import save_model\n\n from mlflow.tracking.client import MlflowClient\n\n os.environ[\"MLFLOW_S3_ENDPOINT_URL\"] = \"http://minio-service.kubeflow.svc:9000\"\n os.environ[\"AWS_ACCESS_KEY_ID\"] = \"minio\"\n os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"minio123\"\n\n client = MlflowClient(\"http://mlflow-server-service.mlflow-system.svc:5000\")\n\n with open(model_path, mode=\"rb\") as file_reader:\n clf = dill.load(file_reader)\n\n with open(input_example_path, \"rb\") as file_reader:\n input_example = dill.load(file_reader)\n\n with open(signature_path, \"rb\") as file_reader:\n signature = dill.load(file_reader)\n\n with open(conda_env_path, \"rb\") as file_reader:\n conda_env = dill.load(file_reader)\n\n save_model(\n sk_model=clf,\n path=model_name,\n serialization_format=\"cloudpickle\",\n conda_env=conda_env,\n signature=signature,\n input_example=input_example,\n )\n run = client.create_run(experiment_id=\"0\")\n client.log_artifact(run.info.run_id, model_name)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Upload sklearn model to mlflow', description='')\n _parser.add_argument(\"--model-name\", dest=\"model_name\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--model\", dest=\"model_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--input-example\", dest=\"input_example_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--signature\", dest=\"signature_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--conda-env\", dest=\"conda_env_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = upload_sklearn_model_to_mlflow(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: model_name}\n artifacts:\n - {name: train-from-csv-conda_env, path: /tmp/inputs/conda_env/data}\n - {name: train-from-csv-input_example, path: /tmp/inputs/input_example/data}\n - {name: train-from-csv-model, path: /tmp/inputs/model/data}\n - {name: train-from-csv-signature, path: /tmp/inputs/signature/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--model-name\", {\"inputValue\": \"model_name\"}, \"--model\", {\"inputPath\":\n \"model\"}, \"--input-example\", {\"inputPath\": \"input_example\"}, \"--signature\",\n {\"inputPath\": \"signature\"}, \"--conda-env\", {\"inputPath\": \"conda_env\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn''\n ''mlflow'' ''boto3'' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install\n --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn'' ''mlflow''\n ''boto3'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def upload_sklearn_model_to_mlflow(\\n model_name,\\n model_path,\\n input_example_path,\\n signature_path,\\n conda_env_path,\\n):\\n import\n os\\n import dill\\n from mlflow.sklearn import save_model\\n\\n from\n mlflow.tracking.client import MlflowClient\\n\\n os.environ[\\\"MLFLOW_S3_ENDPOINT_URL\\\"]\n = \\\"http://minio-service.kubeflow.svc:9000\\\"\\n os.environ[\\\"AWS_ACCESS_KEY_ID\\\"]\n = \\\"minio\\\"\\n os.environ[\\\"AWS_SECRET_ACCESS_KEY\\\"] = \\\"minio123\\\"\\n\\n client\n = MlflowClient(\\\"http://mlflow-server-service.mlflow-system.svc:5000\\\")\\n\\n with\n open(model_path, mode=\\\"rb\\\") as file_reader:\\n clf = dill.load(file_reader)\\n\\n with\n open(input_example_path, \\\"rb\\\") as file_reader:\\n input_example\n = dill.load(file_reader)\\n\\n with open(signature_path, \\\"rb\\\") as file_reader:\\n signature\n = dill.load(file_reader)\\n\\n with open(conda_env_path, \\\"rb\\\") as file_reader:\\n conda_env\n = dill.load(file_reader)\\n\\n save_model(\\n sk_model=clf,\\n path=model_name,\\n serialization_format=\\\"cloudpickle\\\",\\n conda_env=conda_env,\\n signature=signature,\\n input_example=input_example,\\n )\\n run\n = client.create_run(experiment_id=\\\"0\\\")\\n client.log_artifact(run.info.run_id,\n model_name)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Upload\n sklearn model to mlflow'', description='''')\\n_parser.add_argument(\\\"--model-name\\\",\n dest=\\\"model_name\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--model\\\",\n dest=\\\"model_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--input-example\\\",\n dest=\\\"input_example_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--signature\\\",\n dest=\\\"signature_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--conda-env\\\",\n dest=\\\"conda_env_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = upload_sklearn_model_to_mlflow(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"model_name\", \"type\": \"String\"},\n {\"name\": \"model\", \"type\": \"dill\"}, {\"name\": \"input_example\", \"type\": \"dill\"},\n {\"name\": \"signature\", \"type\": \"dill\"}, {\"name\": \"conda_env\", \"type\": \"dill\"}],\n \"name\": \"Upload sklearn model to mlflow\"}', pipelines.kubeflow.org/component_ref: '{}',\n pipelines.kubeflow.org/arguments.parameters: '{\"model_name\": \"{{inputs.parameters.model_name}}\"}'}\n arguments:\n parameters:\n - {name: kernel}\n - {name: model_name}\n serviceAccountName: pipeline-runner\n```\n\n
\n \n \nexample_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: example-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9, pipelines.kubeflow.org/pipeline_compilation_time: '2021-12-09T18:11:43.193190',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"number_1\", \"type\":\n \"Integer\"}, {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"example_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9}\nspec:\n entrypoint: example-pipeline\n templates:\n - name: example-pipeline\n inputs:\n parameters:\n - {name: number_1}\n - {name: number_2}\n dag:\n tasks:\n - name: print-and-return-number\n template: print-and-return-number\n arguments:\n parameters:\n - {name: number_1, value: '{{inputs.parameters.number_1}}'}\n - name: print-and-return-number-2\n template: print-and-return-number-2\n arguments:\n parameters:\n - {name: number_2, value: '{{inputs.parameters.number_2}}'}\n - name: sum-and-print-numbers\n template: sum-and-print-numbers\n dependencies: [print-and-return-number, print-and-return-number-2]\n arguments:\n parameters:\n - {name: print-and-return-number-2-Output, value: '{{tasks.print-and-return-number-2.outputs.parameters.print-and-return-number-2-Output}}'}\n - {name: print-and-return-number-Output, value: '{{tasks.print-and-return-number.outputs.parameters.print-and-return-number-Output}}'}\n - name: print-and-return-number\n container:\n args: [--number, '{{inputs.parameters.number_1}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(\n str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_1}\n outputs:\n parameters:\n - name: print-and-return-number-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-Output, path: /tmp/outputs/Output/data}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is number 1, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\",\n {\"outputPath\": \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(\\n str(int_value),\n str(type(int_value))))\\n return str(int_value)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Print and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_1}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n - name: print-and-return-number-2\n container:\n args: [--number, '{{inputs.parameters.number_2}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(\n str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_2}\n outputs:\n parameters:\n - name: print-and-return-number-2-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-2-Output, path: /tmp/outputs/Output/data}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is number 2, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\",\n {\"outputPath\": \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(\\n str(int_value),\n str(type(int_value))))\\n return str(int_value)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Print and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_2}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n - name: sum-and-print-numbers\n container:\n args: [--number-1, '{{inputs.parameters.print-and-return-number-Output}}', --number-2,\n '{{inputs.parameters.print-and-return-number-2-Output}}']\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def sum_and_print_numbers(number_1, number_2):\n print(number_1 + number_2)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Sum and print numbers', description='')\n _parser.add_argument(\"--number-1\", dest=\"number_1\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--number-2\", dest=\"number_2\", type=int, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = sum_and_print_numbers(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: print-and-return-number-2-Output}\n - {name: print-and-return-number-Output}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is sum of number\n 1 and number 2, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number-1\", {\"inputValue\": \"number_1\"}, \"--number-2\",\n {\"inputValue\": \"number_2\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def sum_and_print_numbers(number_1, number_2):\\n print(number_1 + number_2)\\n\\nimport\n argparse\\n_parser = argparse.ArgumentParser(prog=''Sum and print numbers'',\n description='''')\\n_parser.add_argument(\\\"--number-1\\\", dest=\\\"number_1\\\",\n type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--number-2\\\",\n dest=\\\"number_2\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = sum_and_print_numbers(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number_1\", \"type\": \"Integer\"},\n {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"Sum and print numbers\"}',\n pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number_1\":\n \"{{inputs.parameters.print-and-return-number-Output}}\", \"number_2\": \"{{inputs.parameters.print-and-return-number-2-Output}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n arguments:\n parameters:\n - {name: number_1}\n - {name: number_2}\n serviceAccountName: pipeline-runner\n```\n\n
\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n```\n\n파이프라인으로 작성하면 다음과 같이 됩니다.\n\n```python\nfrom functools import partial\n\nimport kfp\nfrom kfp.components import create_component_from_func, OutputPath\nfrom kfp.dsl import pipeline\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"matplotlib\"],\n)\ndef plot_linear(mlpipeline_ui_metadata: OutputPath(\"UI_Metadata\")):\n import base64\n import json\n from io import BytesIO\n\n import matplotlib.pyplot as plt\n\n plt.plot([1, 2, 3], [1, 2, 3])\n\n tmpfile = BytesIO()\n plt.savefig(tmpfile, format=\"png\")\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\"utf-8\")\n\n html = f\"\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n\n\n@pipeline(name=\"plot_pipeline\")\ndef plot_pipeline():\n plot_linear()\n\n\nif __name__ == \"__main__\":\n kfp.compiler.Compiler().compile(plot_pipeline, \"plot_pipeline.yaml\")\n```\n\n이 스크립트를 실행해서 나온 `plot_pipeline.yaml`을 확인하면 다음과 같습니다.\n\n\n \n \nplot_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: plot-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9, pipelines.kubeflow.org/pipeline_compilation_time: '2\n022-01-17T13:31:32.963214',\n pipelines.kubeflow.org/pipeline_spec: '{\"name\": \"plot_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9}\nspec:\n entrypoint: plot-pipeline\n templates:\n - name: plot-linear\n container:\n args: [--mlpipeline-ui-metadata, /tmp/outputs/mlpipeline_ui_metadata/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'matplotlib' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet\n --no-warn-script-location 'matplotlib' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n def plot_linear(mlpipeline_ui_metadata):\n import base64\n import json\n from io import BytesIO\n import matplotlib.pyplot as plt\n plt.plot([1, 2, 3], [1, 2, 3])\n tmpfile = BytesIO()\n plt.savefig(tmpfile, format=\"png\")\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\"utf-8\")\n html = f\"\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Plot linear', description='')\n _parser.add_argument(\"--mlpipeline-ui-metadata\", dest=\"mlpipeline_ui_metadata\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n _outputs = plot_linear(**_parsed_args)\n image: python:3.7\n outputs:\n artifacts:\n - {name: mlpipeline-ui-metadata, path: /tmp/outputs/mlpipeline_ui_metadata/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--mlpipeline-ui-metadata\", {\"outputPath\": \"mlpipeline_ui_metadata\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''matplotlib'' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location ''matplotlib''\n --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef plot_linear(mlpipeline_ui_metadata):\\n import\n base64\\n import json\\n from io import BytesIO\\n\\n import matplotlib.pyplot\n as plt\\n\\n plt.plot([1, 2, 3], [1, 2, 3])\\n\\n tmpfile = BytesIO()\\n plt.savefig(tmpfile,\n format=\\\"png\\\")\\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\\\"utf-8\\\")\\n\\n html\n = f\\\"\\\"\\n metadata = {\\n \\\"outputs\\\":\n [\\n {\\n \\\"type\\\": \\\"web-app\\\",\\n \\\"storage\\\":\n \\\"inline\\\",\\n \\\"source\\\": html,\\n },\\n ],\\n }\\n with\n open(mlpipeline_ui_metadata, \\\"w\\\") as html_writer:\\n json.dump(metadata,\n html_writer)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Plot\n linear'', description='''')\\n_parser.add_argument(\\\"--mlpipeline-ui-metadata\\\",\n dest=\\\"mlpipeline_ui_metadata\\\", type=_make_parent_dirs_and_return_path,\n required=True, default=argparse.SUPPRESS)\\n_parsed_args = vars(_parser.parse_args())\\n\\n_outputs\n = plot_linear(**_parsed_args)\\n\"], \"image\": \"python:3.7\"}}, \"name\": \"Plot\n linear\", \"outputs\": [{\"name\": \"mlpipeline_ui_metadata\", \"type\": \"UI_Metadata\"}]}',\n pipelines.kubeflow.org/component_ref: '{}'}\n - name: plot-pipeline\n dag:\n tasks:\n - {name: plot-linear, template: plot-linear}\n arguments:\n parameters: []\n serviceAccountName: pipeline-runner\n```\n\n
\n
\n"
},
{
"path": "docs/kubeflow/basic-requirements.md",
"content": "---\ntitle : \"3. Install Requirements\"\ndescription: \"\"\nsidebar_position: 3\ncontributors: [\"Jongseob Jeon\"]\n---\n\n실습을 위해 권장하는 파이썬 버전은 python>=3.7입니다. 파이썬 환경에 익숙하지 않은 분들은 다음 [Appendix 1. 파이썬 가상환경](../appendix/pyenv)을 참고하여 **클라이언트 노드**에 설치해주신 뒤 패키지 설치를 진행해주시기를 바랍니다.\n\n실습을 진행하기에서 필요한 패키지들과 버전은 다음과 같습니다.\n\n- requirements.txt\n\n ```bash\n kfp==1.8.9\n scikit-learn==1.0.1\n mlflow==1.21.0\n pandas==1.3.4\n dill==0.3.4\n ```\n\n[앞에서 만든 파이썬 가상환경](../appendix/pyenv.md#python-가상환경-생성)을 활성화합니다.\n\n```bash\npyenv activate demo\n```\n\n패키지 설치를 진행합니다.\n\n```bash\npip3 install -U pip\npip3 install kfp==1.8.9 scikit-learn==1.0.1 mlflow==1.21.0 pandas==1.3.4 dill==0.3.4\n```\n"
},
{
"path": "docs/kubeflow/basic-run.md",
"content": "---\ntitle : \"7. Pipeline - Run\"\ndescription: \"\"\nsidebar_position: 7\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Run Pipeline\n\n이제 업로드한 파이프라인을 실행시켜 보겠습니다.\n\n## Before Run\n\n### 1. Create Experiment\n\nExperiment란 Kubeflow 에서 실행되는 Run을 논리적으로 관리하는 단위입니다. \n\nKubeflow에서 namespace를 처음 들어오면 생성되어 있는 Experiment가 없습니다. 따라서 파이프라인을 실행하기 전에 미리 Experiment를 생성해두어야 합니다. Experiment이 있다면 [Run Pipeline](../kubeflow/basic-run.md#run-pipeline-1)으로 넘어가도 무방합니다.\n\nExperiment는 Create Experiment 버튼을 통해 생성할 수 있습니다.\n\n\n\n### 2. Name 입력\n\nExperiment로 사용할 이름을 입력합니다.\n\n\n## Run Pipeline\n\n### 1. Create Run 선택\n\n\n\n### 2. Experiment 선택\n\n\n\n\n\n### 3. Pipeline Config 입력\n\n파이프라인을 생성할 때 입력한 Config 값들을 채워 넣습니다.\n업로드한 파이프라인은 number_1과 number_2를 입력해야 합니다.\n\n\n\n### 4. Start\n\n입력 후 Start 버튼을 누르면 파이프라인이 실행됩니다.\n\n\n\n## Run Result\n\n실행된 파이프라인들은 Runs 탭에서 확인할 수 있습니다.\nRun을 클릭하면 실행된 파이프라인과 관련된 자세한 내용을 확인해 볼 수 있습니다.\n\n\n\n클릭하면 다음과 같은 화면이 나옵니다. 아직 실행되지 않은 컴포넌트는 회색 표시로 나옵니다.\n\n\n\n컴포넌트가 실행이 완료되면 초록색 체크 표시가 나옵니다.\n\n\n\n가장 마지막 컴포넌트를 보면 입력한 Config인 3과 5의 합인 8이 출력된 것을 확인할 수 있습니다.\n\n\n"
},
{
"path": "docs/kubeflow/how-to-debug.md",
"content": "---\ntitle : \"13. Component - Debugging\"\ndescription: \"\"\nsidebar_position: 13\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Debugging Pipeline\n\n이번 페이지에서는 Kubeflow 컴포넌트를 디버깅하는 방법에 대해서 알아봅니다.\n\n## Failed Component\n\n이번 페이지에서는 [Component - MLFlow](../kubeflow/advanced-mlflow.md#mlflow-pipeline) 에서 이용한 파이프라인을 조금 수정해서 사용합니다.\n\n우선 컴포넌트가 실패하도록 파이프라인을 변경하도록 하겠습니다.\n\n```python\nfrom functools import partial\n\nimport kfp\nfrom kfp.components import InputPath, OutputPath, create_component_from_func\nfrom kfp.dsl import pipeline\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\", \"scikit-learn\"],\n)\ndef load_iris_data(\n data_path: OutputPath(\"csv\"),\n target_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n from sklearn.datasets import load_iris\n\n iris = load_iris()\n\n data = pd.DataFrame(iris[\"data\"], columns=iris[\"feature_names\"])\n target = pd.DataFrame(iris[\"target\"], columns=[\"target\"])\n \n data[\"sepal length (cm)\"] = None\n data.to_csv(data_path, index=False)\n target.to_csv(target_path, index=False)\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\"],\n)\ndef drop_na_from_csv(\n data_path: InputPath(\"csv\"),\n output_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n\n data = pd.read_csv(data_path)\n data = data.dropna()\n data.to_csv(output_path, index=False)\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"dill\", \"pandas\", \"scikit-learn\", \"mlflow\"],\n)\ndef train_from_csv(\n train_data_path: InputPath(\"csv\"),\n train_target_path: InputPath(\"csv\"),\n model_path: OutputPath(\"dill\"),\n input_example_path: OutputPath(\"dill\"),\n signature_path: OutputPath(\"dill\"),\n conda_env_path: OutputPath(\"dill\"),\n kernel: str,\n):\n import dill\n import pandas as pd\n from sklearn.svm import SVC\n\n from mlflow.models.signature import infer_signature\n from mlflow.utils.environment import _mlflow_conda_env\n\n train_data = pd.read_csv(train_data_path)\n train_target = pd.read_csv(train_target_path)\n\n clf = SVC(kernel=kernel)\n clf.fit(train_data, train_target)\n\n with open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n\n input_example = train_data.sample(1)\n with open(input_example_path, \"wb\") as file_writer:\n dill.dump(input_example, file_writer)\n\n signature = infer_signature(train_data, clf.predict(train_data))\n with open(signature_path, \"wb\") as file_writer:\n dill.dump(signature, file_writer)\n\n conda_env = _mlflow_conda_env(\n additional_pip_deps=[\"dill\", \"pandas\", \"scikit-learn\"]\n )\n with open(conda_env_path, \"wb\") as file_writer:\n dill.dump(conda_env, file_writer)\n\n\n\n@pipeline(name=\"debugging_pipeline\")\ndef debugging_pipeline(kernel: str):\n iris_data = load_iris_data()\n drop_data = drop_na_from_csv(data=iris_data.outputs[\"data\"])\n model = train_from_csv(\n train_data=drop_data.outputs[\"output\"],\n train_target=iris_data.outputs[\"target\"],\n kernel=kernel,\n )\n\nif __name__ == \"__main__\":\n kfp.compiler.Compiler().compile(debugging_pipeline, \"debugging_pipeline.yaml\")\n\n```\n\n수정한 점은 다음과 같습니다.\n\n1. 데이터를 불러오는 `load_iris_data` 컴포넌트에서 `sepal length (cm)` 피처에 `None` 값을 주입\n2. `drop_na_from_csv` 컴포넌트에서 `drop_na()` 함수를 이용해 na 값이 포함된 `row`를 제거\n\n이제 파이프라인을 업로드하고 실행해 보겠습니다. \n실행 후 Run을 눌러서 확인해보면 `Train from csv` 컴포넌트에서 실패했다고 나옵니다.\n\n\n\n실패한 컴포넌트를 클릭하고 로그를 확인해서 실패한 이유를 확인해 보겠습니다.\n\n\n\n로그를 확인하면 데이터의 개수가 0이여서 실행되지 않았다고 나옵니다. \n분명 정상적으로 데이터를 전달했는데 왜 데이터의 개수가 0개일까요? \n\n이제 입력받은 데이터에 어떤 문제가 있었는지 확인해 보겠습니다. \n우선 컴포넌트를 클릭하고 Input/Ouput 탭에서 입력값으로 들어간 데이터들을 다운로드 받습니다. \n다운로드는 빨간색 네모로 표시된 곳의 링크를 클릭하면 됩니다.\n\n\n\n두 개의 파일을 같은 경로에 다운로드합니다. \n그리고 해당 경로로 이동해서 파일을 확인합니다.\n\n```bash\nls\n```\n\n다음과 같이 두 개의 파일이 있습니다.\n\n```bash\ndrop-na-from-csv-output.tgz load-iris-data-target.tgz\n```\n\n압축을 풀어보겠습니다.\n\n```bash\ntar -xzvf load-iris-data-target.tgz ; mv data target.csv\ntar -xzvf drop-na-from-csv-output.tgz ; mv data data.csv\n```\n\n그리고 이를 주피터 노트북을 이용해 컴포넌트 코드를 실행합니다.\n\n\n\n디버깅을 해본 결과 dropna 할 때 column을 기준으로 drop을 해야 하는데 row를 기준으로 drop을 해서 데이터가 모두 사라졌습니다.\n이제 문제의 원인을 알아냈으니 column을 기준으로 drop이 되게 컴포넌트를 수정합니다.\n\n```python\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\"],\n)\ndef drop_na_from_csv(\n data_path: InputPath(\"csv\"),\n output_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n\n data = pd.read_csv(data_path)\n data = data.dropna(axis=\"columns\")\n data.to_csv(output_path, index=False)\n```\n\n수정 후 파이프라인을 다시 업로드하고 실행하면 다음과 같이 정상적으로 수행하는 것을 확인할 수 있습니다.\n\n\n"
},
{
"path": "docs/kubeflow/kubeflow-concepts.md",
"content": "---\ntitle : \"2. Kubeflow Concepts\"\ndescription: \"\"\nsidebar_position: 2\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Component\n\n컴포넌트(Component)는 컴포넌트 콘텐츠(Component contents)와 컴포넌트 래퍼(Component wrapper)로 구성되어 있습니다.\n하나의 컴포넌트는 컴포넌트 래퍼를 통해 kubeflow에 전달되며 전달된 컴포넌트는 정의된 컴포넌트 콘텐츠를 실행(execute)하고 아티팩트(artifacts)들을 생산합니다.\n\n\n\n### Component Contents\n\n컴포넌트 콘텐츠를 구성하는 것은 총 3가지가 있습니다.\n\n\n\n1. Environemnt\n2. Python code w\\ Config\n3. Generates Artifacts\n\n예시와 함께 각 구성 요소가 어떤 것인지 알아보도록 하겠습니다.\n다음과 같이 데이터를 불러와 SVC(Support Vector Classifier)를 학습한 후 SVC 모델을 저장하는 과정을 적은 파이썬 코드가 있습니다.\n\n```python\nimport dill\nimport pandas as pd\n\nfrom sklearn.svm import SVC\n\ntrain_data = pd.read_csv(train_data_path)\ntrain_target= pd.read_csv(train_target_path)\n\nclf= SVC(\n kernel=kernel\n)\nclf.fit(train_data)\n\nwith open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n```\n\n위의 파이썬 코드는 다음과 같이 컴포넌트 콘텐츠로 나눌 수 있습니다.\n\n\n\nEnvironment는 파이썬 코드에서 사용하는 패키지들을 import하는 부분입니다. \n다음으로 Python Code w\\ Config 에서는 주어진 Config를 이용해 실제로 학습을 수행합니다. \n마지막으로 아티팩트를 저장하는 과정이 있습니다.\n\n### Component Wrapper\n\n컴포넌트 래퍼는 컴포넌트 콘텐츠에 필요한 Config를 전달하고 실행시키는 작업을 합니다.\n\n\n\nKubeflow에서는 컴포넌트 래퍼를 위의 `train_svc_from_csv`와 같이 함수의 형태로 정의합니다.\n컴포넌트 래퍼가 콘텐츠를 감싸면 다음과 같이 됩니다.\n\n\n\n### Artifacts\n\n위의 설명에서 컴포넌트는 아티팩트(Artifacts)를 생성한다고 했습니다. 아티팩트란 evaluation result, log 등 어떤 형태로든 파일로 생성되는 것을 통틀어서 칭하는 용어입니다.\n그중 우리가 관심을 두는 유의미한 것들은 다음과 같은 것들이 있습니다.\n\n\n\n- Model\n- Data\n- Metric\n- etc\n\n#### Model\n\n저희는 모델을 다음과 같이 정의 했습니다.\n\n> 모델이란 파이썬 코드와 학습된 Weights와 Network 구조 그리고 이를 실행시키기 위한 환경이 모두 포함된 형태\n\n#### Data\n\n데이터는 전 처리된 피처, 모델의 예측 값 등을 포함합니다.\n\n#### Metric\n\nMetric은 동적 지표와 정적 지표 두 가지로 나누었습니다.\n\n- 동적 지표란 train loss와 같이 학습이 진행되는 중 에폭(Epoch)마다 계속해서 변화하는 값을 의미합니다.\n- 정적 지표란 학습이 끝난 후 최종적으로 모델을 평가하는 정확도 등을 의미합니다.\n\n## Pipeline\n\n파이프라인은 컴포넌트의 집합과 컴포넌트를 실행시키는 순서도로 구성되어 있습니다. 이 때, 순서도는 방향 순환이 없는 그래프로 이루어져 있으며, 간단한 조건문을 포함할 수 있습니다.\n\n\n\n### Pipeline Config\n\n앞서 컴포넌트를 실행시키기 위해서는 Config가 필요하다고 설명했습니다. 파이프라인을 구성하는 컴포넌트의 Config 들을 모아 둔 것이 파이프라인 Config입니다.\n\n\n\n## Run\n\n파이프라인이 필요로 하는 파이프라인 Config가 주어져야지만 파이프라인을 실행할 수 있습니다. \nKubeflow에서는 실행된 파이프라인을 Run 이라고 부릅니다.\n\n\n\n파이프라인이 실행되면 각 컴포넌트가 아티팩트들을 생성합니다.\nKubeflow pipeline에서는 Run 하나당 고유한 ID 를 생성하고, Run에서 생성되는 모든 아티팩트들을 저장합니다.\n\n\n\n그러면 이제 직접 컴포넌트와 파이프라인을 작성하는 방법에 대해서 알아보도록 하겠습니다.\n"
},
{
"path": "docs/kubeflow/kubeflow-intro.md",
"content": "---\ntitle : \"1. Kubeflow Introduction\"\ndescription: \"\"\nsidebar_position: 1\ncontributors: [\"Jongseob Jeon\"]\n---\n\nKubeflow를 사용하기 위해서는 컴포넌트(Component)와 파이프라인(Pipeline)을 작성해야 합니다.\n\n*모두의 MLOps*에서 설명하는 방식은 [Kubeflow Pipeline 공식 홈페이지](https://www.kubeflow.org/docs/components/pipelines/overview/quickstart/)에서 설명하는 방식과는 다소 차이가 있습니다. 여기에서는 Kubeflow Pipeline을 워크플로(Workflow)가 아닌 앞서 설명한 [MLOps를 구성하는 요소](../kubeflow/kubeflow-concepts.md#component-contents) 중 하나의 컴포넌트로 사용하기 때문입니다.\n\n그럼 이제 컴포넌트와 파이프라인은 무엇이며 어떻게 작성할 수 있는지 알아보도록 하겠습니다.\n"
},
{
"path": "docs/kubeflow-dashboard-guide/_category_.json",
"content": "{\n \"label\": \"Kubeflow UI Guide\",\n \"position\": 5,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "docs/kubeflow-dashboard-guide/experiments-and-others.md",
"content": "---\ntitle : \"6. Kubeflow Pipeline 관련\"\ndescription: \"\"\nsidebar_position: 6\ncontributors: [\"Jaeyeon Kim\"]\n---\n\nCentral Dashboard의 왼쪽 탭의 Experiments(KFP), Pipelines, Runs, Recurring Runs, Artifacts, Executions 페이지들에서는 Kubeflow Pipeline과 Pipeline의 실행 그리고 Pipeline Run의 결과를 관리합니다.\n\n\n\nKubeflow Pipeline이 *모두의 MLOps*에서 Kubeflow를 사용하는 주된 이유이며, Kubeflow Pipeline을 만드는 방법, 실행하는 방법, 결과를 확인하는 방법 등 자세한 내용은 [3.Kubeflow](../kubeflow/kubeflow-intro)에서 다룹니다.\n"
},
{
"path": "docs/kubeflow-dashboard-guide/experiments.md",
"content": "---\ntitle : \"5. Experiments(AutoML)\"\ndescription: \"\"\nsidebar_position: 5\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n다음으로는 Central Dashboard의 왼쪽 탭의 Experiments(AutoML)을 클릭해보겠습니다.\n\n\n\n\n\nExperiments(AutoML) 페이지는 Kubeflow에서 Hyperparameter Tuning과 Neural Architecture Search를 통한 AutoML을 담당하는 [Katib](https://www.kubeflow.org/docs/components/katib/overview/)를 관리할 수 있는 페이지입니다.\n\nKatib와 Experiments(AutoML)에 대한 사용법은 *모두의 MLOps* v1.0에서는 다루지 않으며, v2.0에 추가될 예정입니다.\n"
},
{
"path": "docs/kubeflow-dashboard-guide/intro.md",
"content": "---\ntitle : \"1. Central Dashboard\"\ndescription: \"\"\nsidebar_position: 1\ncontributors: [\"Jaeyeon Kim\", \"SeungTae Kim\"]\n---\n\n[Kubeflow 설치](../setup-components/install-components-kf.md)를 완료하면, 다음 커맨드를 통해 대시보드에 접속할 수 있습니다.\n\n```bash\nkubectl port-forward --address 0.0.0.0 svc/istio-ingressgateway -n istio-system 8080:80\n```\n\n\n\nCentral Dashboard는 Kubeflow에서 제공하는 모든 기능을 통합하여 제공하는 UI입니다. Central Dashboard에서 제공하는 기능은 크게 왼쪽의 탭을 기준으로 구분할 수 있습니다.\n\n\n\n- Home\n- Notebooks\n- Tensorboards\n- Volumes\n- Models\n- Experiments(AutoML)\n- Experiments(KFP)\n- Pipelines\n- Runs\n- Recurring Runs\n- Artifacts\n- Executions\n\n그럼 이제 기능별 간단한 사용법을 알아보겠습니다.\n"
},
{
"path": "docs/kubeflow-dashboard-guide/notebooks.md",
"content": "---\ntitle : \"2. Notebooks\"\ndescription: \"\"\nsidebar_position: 2\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n## 노트북 서버(Notebook Server) 생성하기\n\n다음 Central Dashboard의 왼쪽 탭의 Notebooks를 클릭해보겠습니다.\n\n\n\n다음과 같은 화면을 볼 수 있습니다.\n\nNotebooks 탭은 JupyterHub와 비슷하게 유저별로 jupyter notebook 및 code server 환경(이하 노트북 서버)을 독립적으로 생성하고 접속할 수 있는 페이지입니다.\n\n\n\n오른쪽 위의 `+ NEW NOTEBOOK` 버튼을 클릭합니다.\n\n\n\n아래와 같은 화면이 나타나면, 이제 생성할 노트북 서버의 스펙(Spec)을 명시하여 생성합니다.\n\n\n\nexample_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: example-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline_compilation_time: '2021-12-05T13:38:51.566777',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"number_1\", \"type\":\n \"Integer\"}, {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"example_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3}\nspec:\n entrypoint: example-pipeline\n templates:\n - name: example-pipeline\n inputs:\n parameters:\n - {name: number_1}\n - {name: number_2}\n dag:\n tasks:\n - name: print-and-return-number\n template: print-and-return-number\n arguments:\n parameters:\n - {name: number_1, value: '{{inputs.parameters.number_1}}'}\n - name: print-and-return-number-2\n template: print-and-return-number-2\n arguments:\n parameters:\n - {name: number_2, value: '{{inputs.parameters.number_2}}'}\n - name: sum-and-print-numbers\n template: sum-and-print-numbers\n dependencies: [print-and-return-number, print-and-return-number-2]\n arguments:\n parameters:\n - {name: print-and-return-number-2-Output, value: '{{tasks.print-and-return-number-2.outputs.parameters.print-and-return-number-2-Output}}'}\n - {name: print-and-return-number-Output, value: '{{tasks.print-and-return-number.outputs.parameters.print-and-return-number-Output}}'}\n - name: print-and-return-number\n container:\n args: [--number, '{{inputs.parameters.number_1}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_1}\n outputs:\n parameters:\n - name: print-and-return-number-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-Output, path: /tmp/outputs/Output/data}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\", {\"outputPath\":\n \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(str(int_value), str(type(int_value))))\\n return\n str(int_value)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Print\n and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_1}}\"}'}\n - name: print-and-return-number-2\n container:\n args: [--number, '{{inputs.parameters.number_2}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_2}\n outputs:\n parameters:\n - name: print-and-return-number-2-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-2-Output, path: /tmp/outputs/Output/data}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\", {\"outputPath\":\n \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(str(int_value), str(type(int_value))))\\n return\n str(int_value)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Print\n and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_2}}\"}'}\n - name: sum-and-print-numbers\n container:\n args: [--number-1, '{{inputs.parameters.print-and-return-number-Output}}', --number-2,\n '{{inputs.parameters.print-and-return-number-2-Output}}']\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def sum_and_print_numbers(number_1, number_2):\n print(number_1 + number_2)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Sum and print numbers', description='')\n _parser.add_argument(\"--number-1\", dest=\"number_1\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--number-2\", dest=\"number_2\", type=int, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = sum_and_print_numbers(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: print-and-return-number-2-Output}\n - {name: print-and-return-number-Output}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number-1\", {\"inputValue\": \"number_1\"}, \"--number-2\", {\"inputValue\":\n \"number_2\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n sum_and_print_numbers(number_1, number_2):\\n print(number_1 + number_2)\\n\\nimport\n argparse\\n_parser = argparse.ArgumentParser(prog=''Sum and print numbers'',\n description='''')\\n_parser.add_argument(\\\"--number-1\\\", dest=\\\"number_1\\\",\n type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--number-2\\\",\n dest=\\\"number_2\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = sum_and_print_numbers(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number_1\", \"type\": \"Integer\"},\n {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"Sum and print numbers\"}',\n pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number_1\":\n \"{{inputs.parameters.print-and-return-number-Output}}\", \"number_2\": \"{{inputs.parameters.print-and-return-number-2-Output}}\"}'}\n arguments:\n parameters:\n - {name: number_1}\n - {name: number_2}\n serviceAccountName: pipeline-runner\n```\n\n\n
\n\n모두 정상적으로 입력하였다면 하단의 **LAUNCH** 버튼이 활성화되며, 버튼을 클릭하면 노트북 서버 생성이 시작됩니다.\n\n\n\n생성 후 아래와 같이 **Status** 가 초록색 체크 표시 아이콘으로 변하며, **CONNECT 버튼**이 활성화됩니다.\n\n\n\n---\n\n## 노트북 서버 접속하기\n\n**CONNECT 버튼**을 클릭하면 브라우저에 새 창이 열리며, 다음과 같은 화면이 보입니다.\n\n\n\n**Launcher**의 Notebook, Console, Terminal 아이콘을 클릭하여 사용할 수 있습니다.\n\n 생성된 Notebook 화면\n\n\n\n 생성된 Terminal 화면\n\n\n\n---\n\n## 노트북 서버 중단하기\n\n노트북 서버를 오랜 시간 사용하지 않는 경우, 쿠버네티스 클러스터의 효율적인 리소스 사용을 위해서 노트북 서버를 중단(Stop)할 수 있습니다. **단, 이 경우 노트북 서버 생성 시 Workspace Volume 또는 Data Volume으로 지정해놓은 경로 외에 저장된 데이터는 모두 초기화되는 것에 주의하시기 바랍니다.** \n노트북 서버 생성 당시 경로를 변경하지 않았다면, 디폴트(Default) Workspace Volume의 경로는 노트북 서버 내의 `/home/jovyan` 이므로, `/home/jovyan` 의 하위 경로 이외의 경로에 저장된 데이터는 모두 사라집니다.\n\n다음과 같이 `STOP` 버튼을 클릭하면 노트북 서버가 중단됩니다.\n\n\n\n중단이 완료되면 다음과 같이 `CONNECT` 버튼이 비활성화되며, `PLAY` 버튼을 클릭하면 다시 정상적으로 사용할 수 있습니다.\n\n\n"
},
{
"path": "docs/kubeflow-dashboard-guide/tensorboards.md",
"content": "---\ntitle : \"3. Tensorboards\"\ndescription: \"\"\nsidebar_position: 3\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n다음으로는 Central Dashboard의 왼쪽 탭의 Tensorboards를 클릭해보겠습니다.\n\n\n\n다음과 같은 화면을 볼 수 있습니다.\n\n\n\nTensorboards 탭은 Tensorflow, PyTorch 등의 프레임워크에서 제공하는 Tensorboard 유틸이 생성한 ML 학습 관련 데이터를 시각화하는 텐서보드 서버(Tensorboard Server)를 쿠버네티스 클러스터에 생성하는 기능을 제공합니다.\n\n이렇게 생성한 텐서보드 서버는, 일반적인 원격 텐서보드 서버의 사용법과 같이 사용할 수도 있으며, [Kubeflow 파이프라인 런에서 바로 텐서보드 서버에 데이터를 저장하는 용도](https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#tensorboard)로 활용할 수 있습니다.\n\nKubeflow 파이프라인 런의 결과를 시각화하는 방법에는 [다양한 방식](https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/)이 있으며, *모두의 MLOps*에서는 더 일반적으로 활용할 수 있도록 Kubeflow 컴포넌트의 Visualization 기능과 MLflow의 시각화 기능을 활용할 예정이므로, Tensorboards 페이지에 대한 자세한 설명은 생략하겠습니다.\n"
},
{
"path": "docs/kubeflow-dashboard-guide/volumes.md",
"content": "---\ntitle : \"4. Volumes\"\ndescription: \"\"\nsidebar_position: 4\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n## Volumes\n\n다음으로는 Central Dashboard의 왼쪽 탭의 Volumes를 클릭해보겠습니다.\n\n\n\n다음과 같은 화면을 볼 수 있습니다.\n\n\n\nVolumes 탭은 [Kubernetes의 볼륨(Volume)](https://kubernetes.io/ko/docs/concepts/storage/volumes/), 정확히는 [퍼시스턴트 볼륨 클레임(Persistent Volume Claim, 이하 pvc)](https://kubernetes.io/ko/docs/concepts/storage/persistent-volumes/) 중 현재 user의 namespace에 속한 pvc를 관리하는 기능을 제공합니다.\n\n위 스크린샷을 보면, [1. Notebooks](../kubeflow-dashboard-guide/notebooks) 페이지에서 생성한 Volume의 정보를 확인할 수 있습니다. 해당 Volume의 Storage Class는 쿠버네티스 클러스터 설치 당시 설치한 Default Storage Class인 local-path로 설정되어있음을 확인할 수 있습니다.\n\n이외에도 user namespace에 새로운 볼륨을 생성하거나, 조회하거나, 삭제하고 싶은 경우에 Volumes 페이지를 활용할 수 있습니다.\n\n---\n\n## 볼륨 생성하기\n\n오른쪽 위의 `+ NEW VOLUME` 버튼을 클릭하면 다음과 같은 화면을 볼 수 있습니다.\n\n\n\nname, size, storage class, access mode를 지정하여 생성할 수 있습니다.\n\n원하는 리소스 스펙을 지정하여 생성하면 다음과 같이 볼륨의 Status가 `Pending`으로 조회됩니다. `Status` 아이콘에 마우스 커서를 가져다 대면 *해당 볼륨은 mount하여 사용하는 first consumer가 나타날 때 실제로 생성을 진행한다(This volume will be bound when its first consumer is created.)*는 메시지를 확인할 수 있습니다. \n이는 실습을 진행하는 [StorageClass](https://kubernetes.io/ko/docs/concepts/storage/storage-classes/)인 `local-path`의 볼륨 생성 정책에 해당하며, **문제 상황이 아닙니다.** \n해당 페이지에서 Status가 `Pending` 으로 보이더라도 해당 볼륨을 사용하길 원하는 노트북 서버 혹은 파드(Pod)에서는 해당 볼륨의 이름을 지정하여 사용할 수 있으며, 그때 실제로 볼륨 생성이 진행됩니다.\n\n\n"
},
{
"path": "docs/prerequisites/_category_.json",
"content": "{\n \"label\": \"Prerequisites\",\n \"position\": 1,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "docs/prerequisites/docker/_category_.json",
"content": "{\n \"label\": \"Docker\",\n \"position\": 1,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "docs/prerequisites/docker/advanced.md",
"content": "---\ntitle : \"[Practice] Docker Advanced\"\ndescription: \"Practice to use docker more advanced way.\"\nsidebar_position: 6\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n## 도커 이미지 잘 만들기\n\n### 도커 이미지를 만들 때 고려해야 될 점\n\nDockerfile 을 활용하여 도커 이미지를 만들 때는 명령어의 **순서**가 중요합니다. \n그 이유는 도커 이미지는 여러 개의 Read-Only Layer 로 구성되어있고, 이미지를 빌드할 때 이미 존재하는 레이어는 **캐시되어** 재사용되기 때문에, 이를 생각해서 Dockerfile 을 구성한다면 **빌드 시간을 줄일 수 있습니다.**\n\nDockerfile에서 `RUN`, `ADD`, `COPY` 명령어 하나가 하나의 레이어로 저장됩니다.\n\n예를 들어서 다음과 같은 `Dockerfile`이 있습니다.\n\n```docker\n# Layer 1\nFROM ubuntu:latest\n\n# Layer 2\nRUN apt-get update && apt-get install python3 pip3 -y\n\n# Layer 3\nRUN pip3 install -U pip && pip3 install torch\n\n# Layer 4\nCOPY src/ src/\n\n# Layer 5\nCMD python src/app.py\n```\n\n위의 `Dockerfile`로 빌드된 이미지를 `docker run -it app:latest /bin/bash` 명령어로 실행하면 다음과 같은 레이어로 표현할 수 있습니다.\n\n\n\n최상단의 R/W Layer 는 이미지에 영향을 주지 않습니다. 즉, 컨테이너 내부에서 작업한 내역은 모두 휘발성입니다.\n\n하단의 레이어가 변경되면, 그 위의 레이어는 모두 새로 빌드됩니다. 그래서 Dockerfile 내장 명령어의 순서가 중요합니다. \n예를 들면, **자주 변경**되는 부분은 **최대한 뒤쪽으로** 정렬하는 것을 추천합니다. (ex. `COPY src/ app/src/`)\n\n그렇기 때문에 반대로 변경되지 않는 부분은 최대한 앞쪽으로 정렬하는게 좋습니다.\n\n만약 거의 **변경되지 않지만**, 여러 곳에서 **자주** 쓰이는 부분을 공통화할 수도 있습니다.\n해당 공통부분만 묶어서 별도의 이미지는 미리 만들어둔 다음, **베이스 이미지** 로 활용하는 것이 좋습니다.\n\n예를 들어, 다른 건 거의 똑같은데, tensorflow-cpu 를 사용하는 이미지와, tensorflow-gpu 를 사용하는 환경을 분리해서 이미지로 만들고 싶은 경우에는 다음과 같이 할 수 있습니다. \npython 과 기타 기본적인 패키지가 설치된 [`ghcr.io/makinarocks/python:3.8-base`](http://ghcr.io/makinarocks/python:3.8-base-cpu) 를 만들어두고, **tensorflow cpu 버전과 gpu 버전이** 설치된 이미지 새로 만들때는, 위의 이미지를 `FROM` 으로 불러온 다음, tensorflow install 하는 부분만 별도로 작성해서 Dockerfile 을 2 개로 관리한다면 가독성도 좋고 빌드 시간도 줄일 수 있습니다.\n\n**합칠 수 있는 Layer 는 합치는 것**이 Old version 의 도커에서는 성능 향상 효과를 이끌었습니다. 여러분의 도커 컨테이너가 어떤 도커 버전에서 실행될 것인지 보장할 수 없으며, **가독성**을 위해서도 합칠 수 있는 Layer 는 적절히 합치는 것이 좋습니다.\n\n예를 들면, 다음과 같이 작성된 `Dockerfile`이 있습니다.\n\n```docker\n# Bad Case\nRUN apt-get update\nRUN apt-get install build-essential -y\nRUN apt-get install curl -y\nRUN apt-get install jq -y\nRUN apt-get install git -y\n```\n\n이를 아래와 같이 합쳐서 적을 수 있습니다.\n\n```docker\n# Better Case\nRUN apt-get update && \\\n apt-get install -y \\\n build-essential \\\n curl \\\n jq \\\n git\n```\n\n편의를 위해서는 `.dockerignore` 도 사용하는게 좋습니다.\n`.dockerignore`는 `.gitignore` 와 비슷한 역할을 한다고 이해하면 됩니다. (git add 할 때 제외할 수 있듯이, docker build 할 때 자동으로 제외)\n\n더 많은 정보는 [Docker 공식 문서](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/)에서 확인하실 수 있습니다.\n\n### ENTRYPOINT vs CMD\n\n`ENTRYPOINT` 와 `CMD` 는 모두 컨테이너의 실행 시점에서 어떤 명령어를 실행시키고 싶을 때 사용합니다.\n그리고 이 둘 중 하나는 반드시 존재해야 합니다.\n\n- **차이점**\n - `CMD`: docker run 을 수행할 때, 쉽게 변경하여 사용할 수 있음\n - `ENTRYPOINT`: `--entrypoint` 를 사용해야 변경할 수 있음\n\n`ENTRYPOINT` 와 `CMD` 가 함께 쓰일 때는 보통 `CMD`는 `ENTRYPOINT` 에서 적은 명령의 arguments(parameters) 를 의미합니다.\n\n예를 들어서 다음과 같은 `Dockerfile` 이 있습니다.\n\n```docker\nFROM ubuntu:latest\n\n# 아래 4 가지 option 을 바꿔가며 직접 테스트해보시면 이해하기 편합니다.\n# 단, NO ENTRYPOINT 옵션은 base image 인 ubuntu:latest 에 이미 있어서 테스트해볼 수는 없고 나머지 v2, 3, 5, 6, 8, 9, 11, 12 를 테스트해볼 수 있습니다.\n# ENTRYPOINT echo \"Hello ENTRYPOINT\"\n# ENTRYPOINT [\"echo\", \"Hello ENTRYPOINT\"]\n# CMD echo \"Hello CMD\"\n# CMD [\"echo\", \"Hello CMD\"]\n```\n\n위의 `Dockerfile`에서 주석으로 표시된 부분들을 해제하며 빌드하고 실행하면 다음과 같은 결과를 얻을 수 있습니다.\n\n| | No ENTRYPOINT | ENTRYPOINT a b | ENTRYPOINT [\"a\", \"b\"] |\n| ------------------ | -------------- | -------------- | --------------------- |\n| **NO CMD** | Error! | /bin/sh -c a b | a b |\n| **CMD [\"x\", \"y\"]** | x y | /bin/sh -c a b | a b x y |\n| **CMD x y** | /bin/sh -c x y | /bin/sh -c a b | a b /bin/sh -c x y |\n\n- In Kubernetes pod\n - `ENTRYPOINT` → command\n - `CMD` → args\n\n### Docker tag 이름 짓기\n\n도커 이미지의 tag 로 **latest 는 사용하지 않는 것을 권장**합니다. \n이유는 latest 는 default tag name 이므로 **의도치 않게 overwritten** 되는 경우가 너무 많이 발생하기 때문입니다.\n\n하나의 이미지는 하나의 태그를 가짐(**uniqueness**)을 보장해야 추후 Production 단계에서 **협업/디버깅**에 용이합니다. \n내용은 다르지만, 동일한 tag 를 사용하게 되면 추후 dangling image 로 취급되어 관리하기 어려워집니다. \ndangling image는 `docker images`에는 나오지 않지만 계속해서 저장소를 차지하고 있습니다.\n\n### ETC\n\n1. log 등의 정보는 container 내부가 아닌 곳에 따로 저장합니다.\n container 내부에서 write 한 data 는 언제든지 사라질 수 있기 때문입니다.\n2. secret 한 정보, 환경(dev/prod) dependent 한 정보 등은 Dockerfile 에 직접 적는 게 아니라, env var 또는 .env config file 을 사용합니다.\n3. Dockerfile **linter** 도 존재하므로, 협업 시에는 활용하면 좋습니다.\n [https://github.com/hadolint/hadolint](https://github.com/hadolint/hadolint)\n\n## docker run 의 다양한 옵션\n\n### docker run with volume\n\nDocker container 사용 시 불편한 점이 있습니다.\n바로 Docker는 기본적으로 Docker **container 내부에서 작업한 모든 사항은 저장되지 않습니다.**\n이유는 Docker container 는 각각 격리된 파일시스템을 사용합니다. 따라서, **여러 docker container 끼리 데이터를 공유하기 어렵습니다.**\n\n이 문제를 해결하기 위해서 Docker에서 제공하는 방식은 **2 가지**가 있습니다.\n\n\n\n#### Docker volume\n\n- docker cli 를 사용해 `volume` 이라는 리소스를 직접 관리\n- host 에서 Docker area(`/var/lib/docker`) 아래에 특정 디렉토리를 생성한 다음, 해당 경로를 docker container 에 mount\n\n#### Bind mount\n\n- host 의 특정 경로를 docker container 에 mount\n\n#### How to use?\n\n사용 방식은 **동일한 인터페이스**로 `-v` 옵션을 통해 사용할 수 있습니다. \n다만, volume 을 사용할 때에는 `docker volume create`, `docker volume ls`, `docker volume rm` 등을 수행하여 직접 관리해주어야 합니다.\n\n- Docker volume\n\n ```bash\n docker run \\\n -v my_volume:/app \\\n nginx:latest\n ````\n\n- Blind mount\n\n ```bash\n docker run \\\n -v /home/user/some/path:/app \\\n nginx:latest\n ```\n\n로컬에서 개발할 때는 bind mount 가 편하긴 하지만, 환경을 깔끔하게 유지하고 싶다면 docker volume 을 사용하여 create, rm 을 명시적으로 수행하는 것도 하나의 방법입니다.\n\n쿠버네티스에서 스토리지를 제공하는 방식도 결국 docker 의 bind mount 를 활용하여 제공합니다.\n\n### docker run with resource limit\n\n기본적으로 docker container 는 **host OS 의 cpu, memory 자원을 fully 사용**할 수 있습니다. 하지만 이렇게 사용하게 되면 host OS 의 자원 상황에 따라서 **OOM** 등의 이슈로 docker container 가 비정상적으로 종료되는 상황이 발생할 수 있습니다. \n이런 문제를 다루기 위해 **docker container 실행 시, cpu 와 memory 의 사용량 제한**을 걸 수 있는 `-m` [옵션](https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory)을 제공합니다.\n\n```bash\ndocker run -d -m 512m --memory-reservation=256m --name 512-limit ubuntu sleep 3600\ndocker run -d -m 1g --memory-reservation=256m --name 1g-limit ubuntu sleep 3600\n```\n\n위의 도커를 실행 후 `docker stats` 커맨드를 통해 사용량을 확인할 수 있습니다.\n\n```bash\nCONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS\n4ea1258e2e09 1g-limit 0.00% 300KiB / 1GiB 0.03% 1kB / 0B 0B / 0B 1\n4edf94b9a3e5 512-limit 0.00% 296KiB / 512MiB 0.06% 1.11kB / 0B 0B / 0B 1\n```\n\n쿠버네티스에서 pod 라는 리소스에 cpu, memory 제한을 줄 때, 이 방식을 활용하여 제공합니다.\n\n### docker run with restart policy\n\n특정 컨테이너가 계속해서 running 상태를 유지시켜야 하는 경우가 존재합니다. 이런 경우를 위해서 해당 컨테이너가 종료되자마자 바로 재생성을 시도할 수 있는 `--restart=always` 옵션을 제공하고 있습니다.\n\n옵션 입력 후 도커를 실행합니다.\n\n```bash\ndocker run --restart=always ubuntu\n```\n\n`watch -n1 docker ps`를 통해 재실행이 되고 있는지 확인합니다.\n정상적으로 수행되고 있다면 다음과 같이 STATUS에 `Restarting (0)` 이 출력됩니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\na911850276e8 ubuntu \"bash\" 35 seconds ago Restarting (0) 6 seconds ago hungry_vaughan\n```\n\n- [https://docs.docker.com/engine/reference/commandline/run/#restart-policies---restart](https://docs.docker.com/engine/reference/commandline/run/#restart-policies---restart)\n - on-failure with max retries\n - always 등의 선택지 제공\n\n쿠버네티스에서 job 이라는 resource 의 restart 옵션을 줄 때, 이 방식을 활용하여 제공합니다.\n\n### docker run as a background process\n\n도커 컨테이너를 실행할 때는 기본적으로 foreground process 로 실행됩니다. 즉, 컨테이너를 실행한 터미널이 해당 컨테이너에 자동으로 attach 되어 있어, 다른 명령을 실행할 수 없습니다.\n\n다음과 같은 예시를 수행해봅니다. \n우선 터미널 2 개를 열어, 하나의 터미널에서는 `docker ps` 를 지켜보고, 다른 하나의 터미널에서는 다음과 같은 명령을 차례로 실행해보며 동작을 지켜봅니다.\n\n#### First Practice\n\n```bash\ndocker run -it ubuntu sleep 10\n```\n\n10 초동안 멈춰 있어야 하고, 해당 컨테이너에서 다른 명령을 수행할 수 없습니다. 10초 뒤에는 docker ps 에서 container 가 종료되는 것을 확인할 수 있습니다.\n\n#### Second Practice\n\n```bash\ndocker run -it ubuntu sleep 10\n```\n\n이후, `ctrl + p` -> `ctrl + q`\n\n해당 터미널에서 이제 다른 명령을 수행할 수 있게 되었으며, docker ps 로도 10초까지는 해당 컨테이너가 살아있는 것을 확인할 수 있습니다.\n이렇게 docker container 내부에서 빠져나온 상황을 detached 라고 부릅니다.\n도커에서는 run 을 실행함과 동시에 detached mode 로 실행시킬 수 있는 옵션을 제공합니다.\n\n#### Third Practice\n\n```bash\ndocker run -d ubuntu sleep 10\n```\n\ndetached mode 이므로 해당 명령을 실행시킨 터미널에서 다른 액션을 수행시킬 수 있습니다.\n\n상황에 따라 detached mode 를 적절히 활용하면 좋습니다. \n예를 들어, DB 와 통신하는 Backend API server 를 개발할 때 Backend API server 는 source code 를 변경시켜가면서 hot-loading 으로 계속해서 로그를 확인해봐야 하지만, DB 는 로그를 지켜볼 필요는 없는 경우라면 다음과 같이 실행할 수 있습니다. \nDB 는 docker container 를 detached mode 로 실행시키고, Backend API server 는 attached mode 로 log 를 following 하면서 실행시키면 효율적입니다.\n\n## References\n\n- [https://towardsdatascience.com/docker-storage-598e385f4efe](https://towardsdatascience.com/docker-storage-598e385f4efe)\n- [https://vsupalov.com/docker-latest-tag/](https://vsupalov.com/docker-latest-tag/)\n- [https://docs.microsoft.com/ko-kr/azure/container-registry/container-registry-image-tag-version](https://docs.microsoft.com/ko-kr/azure/container-registry/container-registry-image-tag-version)\n- [https://stevelasker.blog/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/](https://stevelasker.blog/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/)\n"
},
{
"path": "docs/prerequisites/docker/command.md",
"content": "---\ntitle : \"[Practice] Docker command\"\ndescription: \"Practice to use docker command.\"\nsidebar_position: 4\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n## 1. 정상 설치 확인\n\n```bash\ndocker run hello-world\n```\n\n정상적으로 설치된 경우 다음과 같은 메시지를 확인할 수 있습니다.\n\n```bash\nHello from Docker!\nThis message shows that your installation appears to be working correctly.\n....\n```\n\n**(For ubuntu)** sudo 없이 사용하고 싶다면 아래 사이트를 참고합니다.\n\n- [https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user)\n\n## 2. Docker Pull\n\ndocker image registry(도커 이미지를 저장하고 공유할 수 있는 저장소)로부터 Docker image 를 로컬에 다운로드 받는 커맨드입니다.\n\n아래 커맨드를 통해 docker pull에서 사용 가능한 argument들을 확인할 수 있습니다.\n\n```bash\ndocker pull --help\n```\n\n정상적으로 수행되면 아래와 같이 출력됩니다.\n\n```bash\nUsage: docker pull [OPTIONS] NAME[:TAG|@DIGEST]\n\nPull an image or a repository from a registry\n\nOptions:\n -a, --all-tags Download all tagged images in the repository\n --disable-content-trust Skip image verification (default true)\n --platform string Set platform if server is multi-platform capable\n -q, --quiet Suppress verbose output\n```\n\n여기서 알 수 있는 것은 바로 docker pull은 두 개 타입의 argument를 받는다는 것을 알 수 있습니다.\n\n1. `[OPTIONS]`\n2. `NAME[:TAG|@DIGEST]`\n\nhelp에서 나온 `-a`, -`q` 옵션을 사용하기 위해서는 NAME 앞에서 사용해야 합니다.\n\n직접 `ubuntu:18.04` 이미지를 pull 해보겠습니다.\n\n```bash\ndocker pull ubuntu:18.04\n```\n\n위 명령어를 해석하면 `ubuntu` 라는 이름을 가진 이미지 중 `18.04` 태그가 달려있는 이미지를 가져오라는 뜻입니다.\n\n만약, 정상적으로 수행된다면 다음과 비슷하게 출력됩니다.\n\n```bash\n18.04: Pulling from library/ubuntu\n20d796c36622: Pull complete \nDigest: sha256:42cd9143b6060261187a72716906187294b8b66653b50d70bc7a90ccade5c984\nStatus: Downloaded newer image for ubuntu:18.04\ndocker.io/library/ubuntu:18.04\n```\n\n위의 명령어를 수행하면 [docker.io/library](http://docker.io/library/) 라는 이름의 registry 에서 ubuntu:18.04 라는 image 를 여러분의 노트북에 다운로드 받게됩니다.\n\n- 참고사항\n - 추후 [docker.io](http://docker.io) 나 public 한 docker hub 와 같은 registry 대신에, 특정 **private** 한 registry 에서 docker image 를 가져와야 하는 경우에는, [`docker login`](https://docs.docker.com/engine/reference/commandline/login/) 을 통해서 특정 registry 를 바라보도록 한 뒤, docker pull 을 수행하는 형태로 사용합니다. 혹은 insecure registry 를 설정하는 [방안](https://stackoverflow.com/questions/42211380/add-insecure-registry-to-docker)도 활용할 수 있습니다.\n - 폐쇄망에서 docker image 를 `.tar` 파일과 같은 형태로 저장하고 공유할 수 있도록 [`docker save`](https://docs.docker.com/engine/reference/commandline/save/), [`docker load`](https://docs.docker.com/engine/reference/commandline/load/) 와 같은 명령어도 존재합니다.\n\n## 3. Docker images\n\n로컬에 존재하는 docker image 리스트를 출력하는 커맨드입니다.\n\n```bash\ndocker images --help\n```\n\ndocker images에서 사용할 수 있는 argument는 다음과 같습니다.\n\n```bash\nUsage: docker images [OPTIONS] [REPOSITORY[:TAG]]\n\nList images\n\nOptions:\n -a, --all Show all images (default hides intermediate images)\n --digests Show digests\n -f, --filter filter Filter output based on conditions provided\n --format string Pretty-print images using a Go template\n --no-trunc Don't truncate output\n -q, --quiet Only show image IDs\n```\n\n아래 명령어를 이용해 직접 실행해 보겠습니다.\n\n```bash\ndocker images\n```\n\n만약 도커를 최초 설치 후 이 실습을 진행한다면 다음과 비슷하게 출력됩니다.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\n줄 수 있는 argument중 `-q`를 사용하면 `IMAGE ID` 만 출력됩니다.\n\n```bash\ndocker images -q\n```\n\n```bash\n29e70752d7b2\n```\n\n## 4. Docker ps\n\n현재 실행 중인 도커 컨테이너 리스트를 출력하는 커맨드입니다.\n\n```bash\ndocker ps --help\n```\n\ndocker ps에서 사용할 수 있는 argument는 다음과 같습니다.\n\n```bash\nUsage: docker ps [OPTIONS]\n\nList containers\n\nOptions:\n -a, --all Show all containers (default shows just running)\n -f, --filter filter Filter output based on conditions provided\n --format string Pretty-print containers using a Go template\n -n, --last int Show n last created containers (includes all states) (default -1)\n -l, --latest Show the latest created container (includes all states)\n --no-trunc Don't truncate output\n -q, --quiet Only display container IDs\n -s, --size Display total file sizes\n```\n\n아래 명령어를 이용해 직접 실행해 보겠습니다.\n\n```bash\ndocker ps\n```\n\n현재 실행 중인 컨테이너가 없다면 다음과 같이 나옵니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n```\n\n만약 실행되는 컨테이너가 있다면 다음과 비슷하게 나옵니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nc1e8f5e89d8d ubuntu \"sleep 3600\" 13 seconds ago Up 12 seconds trusting_newton\n```\n\n## 5. Docker run\n\n도커 컨테이너를 실행시키는 커맨드입니다.\n\n```bash\ndocker run --help\n```\n\ndocker run을 실행하는 명령어는 다음과 같습니다.\n\n```bash\nUsage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]\n\nRun a command in a new container\n```\n\n여기서 우리가 확인해야 하는 것은 바로 docker run은 세 개 타입의 argument를 받는다는 것을 알 수 있습니다.\n\n1. `[OPTIONS]`\n2. `[COMMAND]`\n3. `[ARG...]`\n\n직접 도커 컨테이너를 실행해 보겠습니다.\n\n```bash\n## Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]\ndocker run -it --name demo1 ubuntu:18.04 /bin/bash\n```\n\n- `-it` : `-i` 옵션 + `-t` 옵션\n - container 를 실행시킴과 동시에 interactive 한 terminal 로 접속시켜주는 옵션\n- `--name` : name\n - 컨테이너 id 대신, 구분하기 쉽도록 지정해주는 이름\n- `/bin/bash`\n - 컨테이너를 실행시킴과 동시에 실행할 커맨드로, `/bin/bash` 는 bash 쉘을 여는 것을 의미합니다.\n\n실행 후 `exit` 명령어를 통해 컨테이너를 종료합니다.\n\n이 제 앞서 배웠던 `docker ps` 명령어를 치면 다음과 같이 나옵니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n```\n\n실행되고 있는 컨테이너가 나온다고 했지만 어째서인지 방금 실행한 컨테이너가 보이지 않습니다.\n그 이유는 `docker ps`는 기본값으로 현재 실행 중인 컨테이너를 보여주기 때문입니다.\n\n만약 종료된 컨테이너들도 보고 싶다면 `-a` 옵션을 주어야 합니다.\n\n```bash\ndocker ps -a\n```\n\n그러면 다음과 같이 종료된 컨테이너 목록도 나옵니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 2 minutes ago Exited (0) 2 minutes ago demo1\n```\n\n## 6. Docker exec\n\nDocker 컨테이너 내부에서 명령을 내리거나, 내부로 접속하는 커맨드입니다.\n\n```bash\ndocker exec --help\n```\n\n예를 들어서 다음과 같은 명령어를 실행해 보겠습니다.\n\n```bash\ndocker run -d --name demo2 ubuntu:18.04 sleep 3600\n```\n\n여기서 `-d` 옵션은 도커 컨테이너를 백그라운드에서 실행시켜서, 컨테이너에서 접속 종료를 하더라도, 계속 실행 중이 되도록 하는 커맨드입니다.\n\n`docker ps`를 통해 현재 실행중인지 확인합니다.\n\n다음과 같이 실행 중임을 확인할 수 있습니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 4 seconds ago Up 3 seconds demo2\n```\n\n이제 `docker exec` 명령어를 통해서 실행중인 도커 컨테이너에 접속해 보겠습니다.\n\n```bash\ndocker exec -it demo2 /bin/bash\n```\n\n이 전의 `docker run`과 동일하게 container 내부에 접속할 수 있습니다.\n\n`exit`을 통해 종료합니다.\n\n## 7. Docker logs\n\n도커 컨테이너의 log를 확인하는 커맨드 입니다.\n\n```bash\ndocker logs --help\n```\n\n다음과 같은 컨테이너를 실행시키도록 하겠습니다.\n\n```bash\ndocker run --name demo3 -d busybox sh -c \"while true; do $(echo date); sleep 1; done\"\n```\n\n위 명령어를 통해서 test 라는 이름의 busybox 컨테이너를 백그라운드에서 도커 컨테이너로 실행하여, 1초에 한 번씩 현재 시간을 출력하도록 했습니다.\n\n이제 아래 명령어를 통해 log를 확인해 보겠습니다.\n\n```bash\ndocker logs demo3\n```\n\n정상적으로 수행되면 아래와 비슷하게 나옵니다.\n\n```bash\nSun Mar 6 11:06:49 UTC 2022\nSun Mar 6 11:06:50 UTC 2022\nSun Mar 6 11:06:51 UTC 2022\nSun Mar 6 11:06:52 UTC 2022\nSun Mar 6 11:06:53 UTC 2022\nSun Mar 6 11:06:54 UTC 2022\n```\n\n그런데 이렇게 사용할 경우 여태까지 찍힌 log 밖에 확인할 수 없습니다. \n이 때 `-f` 옵션을 이용해 계속 watch 하며 출력할 수 있습니다.\n\n```bash\ndocker logs demo3 -f \n```\n\n## 8. Docker stop\n\n실행 중인 도커 컨테이너를 중단시키는 커맨드입니다.\n\n```bash\ndocker stop --help\n```\n\n`docker ps`를 통해 현재 실행 중인 컨테이너를 확인하면 다음과 같습니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" About a minute ago Up About a minute demo3\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 4 minutes ago Up 4 minutes demo2\n```\n\n이제 `docker stop` 을 통해 도커를 정지해 보겠습니다.\n\n```bash\ndocker stop demo2\n```\n\n실행 후 `docker ps`를 다시 입력합니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" 2 minutes ago Up 2 minutes demo3\n```\n\n위의 결과와 비교했을 때 demo2 컨테이너가 현재 실행 중인 컨테이너 목록에서 사라진 것을 확인할 수 있습니다.\n\n나머지 컨테이너도 정지합니다.\n\n```bash\ndocker stop demo3\n```\n\n## 9. Docker rm\n\n도커 컨테이너를 삭제하는 커맨드입니다.\n\n```bash\ndocker rm --help\n```\n\n도커 컨테이너는 기본적으로 종료가 된 상태로 있습니다. 그래서 `docker ps -a`를 통해서 종료된 컨테이너도 볼 수 있습니다.\n그런데 종료된 컨테이너는 왜 지워야 할까요? \n종료되어 있는 도커에는 이전에 사용한 데이터가 아직 컨테이너 내부에 남아있습니다.\n그래서 restart 등을 통해서 컨테이너를 재시작할 수 있습니다.\n그런데 이 과정에서 disk를 사용하게 됩니다.\n\n그래서 완전히 사용하지 않는 컨테이너를 지우기 위해서는 `docker rm` 명령어를 사용해야 합니다.\n\n우선 현재 컨테이너들을 확인합니다.\n\n```bash\ndocker ps -a\n```\n\n다음과 같이 3개의 컨테이너가 있습니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" 4 minutes ago Exited (137) About a minute ago demo3\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 7 minutes ago Exited (137) 2 minutes ago demo2\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 10 minutes ago Exited (0) 10 minutes ago demo1\n```\n\n아래 명령어를 통해 `demo3` 컨테이너를 삭제해 보겠습니다.\n\n```bash\ndocker rm demo3\n```\n\n`docker ps -a` 명령어를 치면 다음과 같이 2개로 줄었습니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 13 minutes ago Exited (137) 8 minutes ago demo2\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 16 minutes ago Exited (0) 16 minutes ago demo1\n```\n\n나머지 컨테이너들도 삭제합니다.\n\n```bash\ndocker rm demo2\ndocker rm demo1\n```\n\n## 10. Docker rmi\n\n도커 이미지를 삭제하는 커맨드입니다.\n\n```bash\ndocker rmi --help\n```\n\n아래 명령어를 통해 현재 어떤 이미지들이 로컬에 있는지 확인합니다.\n\n```bash\ndocker images\n```\n\n다음과 같이 출력됩니다.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nbusybox latest a8440bba1bc0 32 hours ago 1.41MB\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\n`busybox` 이미지를 삭제해 보겠습니다.\n\n```bash\ndocker rmi busybox\n```\n\n다시 `docker images`를 칠 경우 다음과 같이 나옵니다.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\n## References\n\n- [https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry)\n"
},
{
"path": "docs/prerequisites/docker/docker.md",
"content": "---\ntitle : \"What is Docker?\"\ndescription: \"Introduction to Docker.\"\nsidebar_position: 3\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n\n## 컨테이너\n\n- 컨테이너 가상화\n - 어플리케이션을 어디에서나 동일하게 실행하는 기술\n- 컨테이너 이미지\n - 어플리케이션을 실행시키기 위해 필요한 모든 파일들의 집합\n - → 붕어빵 틀\n- 컨테이너란?\n - 컨테이너 이미지를 기반으로 실행된 한 개의 프로세스\n - → 붕어빵 틀로 찍어낸 붕어빵\n\n## 도커\n\n도커는 **컨테이너를 관리**하고 사용할 수 있게 해주는 플랫폼입니다. \n이러한 도커의 슬로건은 바로 **Build Once, Run Anywhere** 로 어디에서나 동일한 실행 결과를 보장합니다.\n\n도커 내부에서 동작하는 과정을 보자면 실제로 container 를 위한 리소스를 분리하고, lifecycle 을 제어하는 기능은 linux kernel 의 cgroup 등이 수행합니다.\n하지만 이러한 인터페이스를 바로 사용하는 것은 **너무 어렵기 때문에** 다음과 같은 추상화 layer를 만들게 됩니다.\n\n\n\n이를 통해 사용자는 사용자 친화적인 API 인 **Docker CLI** 만으로 쉽게 컨테이너를 제어할 수 있습니다.\n\n## Layer 해석\n\n위에서 나온 layer들의 역할은 다음과 같습니다.\n\n1. runC: linux kernel 의 기능을 직접 사용해서, container 라는 하나의 프로세스가 사용할 네임스페이스와 cpu, memory, filesystem 등을 격리시켜주는 기능을 수행합니다.\n2. containerd: runC(OCI layer) 에게 명령을 내리기 위한 추상화 단계이며, 표준화된 인터페이스(OCI)를 사용합니다.\n3. dockerd: containerd 에게 명령을 내리는 역할만 합니다.\n4. docker cli: 사용자는 docker cli 로 dockerd (Docker daemon)에게 명령을 내리기만 하면 됩니다.\n - 이 통신 과정에서 unix socket 을 사용하기 때문에 가끔 도커 관련 에러가 나면 `/var/run/docker.sock` 가 사용 중이다, 권한이 없다 등등의 에러 메시지가 나오는 것입니다.\n\n이처럼 도커는 많은 단계를 감싸고 있지만, 흔히 도커라는 용어를 사용할 때는 Docker CLI 를 말할 때도 있고, Dockerd 를 말할 때도 있고 Docker Container 하나를 말할 때도 있어서 혼란이 생길 수 있습니다. \n앞으로 나오는 글에서도 도커가 여러가지 의미로 쓰일 수 있습니다.\n\n## For ML Engineer\n\n머신러닝 엔지니어가 도커를 사용하는 이유는 다음과 같습니다.\n\n1. 나의 ML 학습/추론 코드를 OS, python version, python 환경, 특정 python package 버전에 independent 하도록 해야 한다.\n2. 그래서 코드 뿐만이 아닌 **해당 코드가 실행되기 위해 필요한 모든 종속적인 패키지, 환경 변수, 폴더명 등등을 하나의 패키지로** 묶을 수 있는 기술이 컨테이너화 기술이다.\n3. 이 기술을 쉽게 사용하고 관리할 수 있는 소프트웨어 중 하나가 도커이며, 패키지를 도커 이미지라고 부른다.\n"
},
{
"path": "docs/prerequisites/docker/images.md",
"content": "---\ntitle : \"[Practice] Docker images\"\ndescription: \"Practice to use docker image.\"\nsidebar_position: 5\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n## 1. Dockerfile 만들기\n\n도커 이미지를 만드는 가장 쉬운 방법은 도커에서 제공하는 템플릿인 Dockerfile을 사용하는 것입니다. \n이외에는 running container 를 docker image 로 만드는 `docker commit` 등을 활용하는 방법이 있습니다.\n\n- `Dockerfile`\n - 사용자가 도커 이미지를 쉽게 만들 수 있도록, 제공하는 템플릿\n - 파일명은 꼭 `Dockerfile` 이 아니어도 상관없지만, `docker build` 수행 시, default 로 사용하는 파일명이 `Dockerfile` 입니다.\n - 도커 이미지를 만드는 `docker build` 를 수행할 때, `-f` 옵션을 주면 다른 파일명으로도 사용 가능합니다.\n - ex) `docker build -f dockerfile-asdf .` 도 가능\n\n1. 실습을 위해서 편한 디렉토리로 이동합니다.\n\n ```bash\n cd 각 스펙에 대한 자세한 내용은 아래와 같습니다.
\n\n- **name**:\n - 노트북 서버를 구분할 수 있는 이름으로 생성합니다.\n- **namespace** :\n - 따로 변경할 수 없습니다. (현재 로그인한 user 계정의 namespace이 자동으로 지정되어 있습니다.)\n- **Image**:\n - sklearn, pytorch, tensorflow 등의 파이썬 패키지가 미리 설치된 jupyter lab 이미지 중 사용할 이미지를 선택합니다.\n - 노트북 서버 내에서 GPU를 사용하여 tensorflow-cuda, pytorch-cuda 등의 이미지를 사용하는 경우, **하단의 GPUs** 부분을 확인하시기 바랍니다.\n - 추가적인 패키지나 소스코드 등을 포함한 커스텀(Custom) 노트북 서버를 사용하고 싶은 경우에는 커스텀 이미지(Custom Image)를 만들고 배포 후 사용할 수도 있습니다.\n- **CPU / RAM**\n - 필요한 자원 사용량을 입력합니다.\n - cpu : core 단위\n - 가상 core 개수 단위를 의미하며, int 형식이 아닌 `1.5`, `2.7` 등의 float 형식도 입력할 수 있습니다.\n - memory : Gi 단위\n- **GPUs**\n - 주피터 노트북에 할당할 GPU 개수를 입력합니다.\n - `None`\n - GPU 자원이 필요하지 않은 상황\n - 1, 2, 4\n - GPU 1, 2, 4 개 할당\n - GPU Vendor\n - 앞의 [(Optional) Setup GPU](../setup-kubernetes/setup-nvidia-gpu.md) 를 따라 nvidia gpu plugin을 설치하였다면 NVIDIA를 선택합니다.\n- **Workspace Volume**\n - 노트북 서버 내에서 필요한 만큼의 디스크 용량을 입력합니다.\n - Type 과 Name 은 변경하지 않고, **디스크 용량을 늘리고 싶거나** **AccessMode 를 변경하고 싶을** 때에만 변경해서 사용하시면 됩니다.\n - **\"Don't use Persistent Storage for User's home\"** 체크박스는 노트북 서버의 작업 내용을 저장하지 않아도 상관없을 때에만 클릭합니다. **일반적으로는 누르지 않는 것을 권장합니다.**\n - 기존에 미리 생성해두었던 PVC를 사용하고 싶을 때에는, Type을 \"Existing\" 으로 입력하여 해당 PVC의 이름을 입력하여 사용하시면 됩니다.\n- **Data Volumes**\n - 추가적인 스토리지 자원이 필요하다면 **\"+ ADD VOLUME\"** 버튼을 클릭하여 생성할 수 있습니다.\n- ~~Configurations, Affinity/Tolerations, Miscellaneous Settings~~\n - 일반적으로는 필요하지 않으므로 *모두의 MLOps*에서는 자세한 설명을 생략합니다.\n\n\n \n \nmlflow_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: mlflow-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.10, pipelines.kubeflow.org/pipeline_compilation_time: '2022-01-19T14:14:11.999807',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"kernel\", \"type\":\n \"String\"}, {\"name\": \"model_name\", \"type\": \"String\"}], \"name\": \"mlflow_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.10}\nspec:\n entrypoint: mlflow-pipeline\n templates:\n - name: load-iris-data\n container:\n args: [--data, /tmp/outputs/data/data, --target, /tmp/outputs/target/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'pandas' 'scikit-learn' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location 'pandas' 'scikit-learn' --user)\n && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n\n def load_iris_data(\n data_path,\n target_path,\n ):\n import pandas as pd\n from sklearn.datasets import load_iris\n\n iris = load_iris()\n\n data = pd.DataFrame(iris[\"data\"], columns=iris[\"feature_names\"])\n target = pd.DataFrame(iris[\"target\"], columns=[\"target\"])\n\n data.to_csv(data_path, index=False)\n target.to_csv(target_path, index=False)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Load iris data', description='')\n _parser.add_argument(\"--data\", dest=\"data_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--target\", dest=\"target_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = load_iris_data(**_parsed_args)\n image: python:3.7\n outputs:\n artifacts:\n - {name: load-iris-data-data, path: /tmp/outputs/data/data}\n - {name: load-iris-data-target, path: /tmp/outputs/target/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--data\", {\"outputPath\": \"data\"}, \"--target\", {\"outputPath\": \"target\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''pandas'' ''scikit-learn'' ||\n PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n ''pandas'' ''scikit-learn'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef load_iris_data(\\n data_path,\\n target_path,\\n):\\n import\n pandas as pd\\n from sklearn.datasets import load_iris\\n\\n iris = load_iris()\\n\\n data\n = pd.DataFrame(iris[\\\"data\\\"], columns=iris[\\\"feature_names\\\"])\\n target\n = pd.DataFrame(iris[\\\"target\\\"], columns=[\\\"target\\\"])\\n\\n data.to_csv(data_path,\n index=False)\\n target.to_csv(target_path, index=False)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Load iris data'', description='''')\\n_parser.add_argument(\\\"--data\\\",\n dest=\\\"data_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--target\\\", dest=\\\"target_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = load_iris_data(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"name\": \"Load iris data\", \"outputs\": [{\"name\":\n \"data\", \"type\": \"csv\"}, {\"name\": \"target\", \"type\": \"csv\"}]}', pipelines.kubeflow.org/component_ref: '{}'}\n - name: mlflow-pipeline\n inputs:\n parameters:\n - {name: kernel}\n - {name: model_name}\n dag:\n tasks:\n - {name: load-iris-data, template: load-iris-data}\n - name: train-from-csv\n template: train-from-csv\n dependencies: [load-iris-data]\n arguments:\n parameters:\n - {name: kernel, value: '{{inputs.parameters.kernel}}'}\n artifacts:\n - {name: load-iris-data-data, from: '{{tasks.load-iris-data.outputs.artifacts.load-iris-data-data}}'}\n - {name: load-iris-data-target, from: '{{tasks.load-iris-data.outputs.artifacts.load-iris-data-target}}'}\n - name: upload-sklearn-model-to-mlflow\n template: upload-sklearn-model-to-mlflow\n dependencies: [train-from-csv]\n arguments:\n parameters:\n - {name: model_name, value: '{{inputs.parameters.model_name}}'}\n artifacts:\n - {name: train-from-csv-conda_env, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-conda_env}}'}\n - {name: train-from-csv-input_example, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-input_example}}'}\n - {name: train-from-csv-model, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-model}}'}\n - {name: train-from-csv-signature, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-signature}}'}\n - name: train-from-csv\n container:\n args: [--train-data, /tmp/inputs/train_data/data, --train-target, /tmp/inputs/train_target/data,\n --kernel, '{{inputs.parameters.kernel}}', --model, /tmp/outputs/model/data,\n --input-example, /tmp/outputs/input_example/data, --signature, /tmp/outputs/signature/data,\n --conda-env, /tmp/outputs/conda_env/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'dill' 'pandas' 'scikit-learn' 'mlflow' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location 'dill' 'pandas' 'scikit-learn'\n 'mlflow' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n\n def train_from_csv(\n train_data_path,\n train_target_path,\n model_path,\n input_example_path,\n signature_path,\n conda_env_path,\n kernel,\n ):\n import dill\n import pandas as pd\n from sklearn.svm import SVC\n\n from mlflow.models.signature import infer_signature\n from mlflow.utils.environment import _mlflow_conda_env\n\n train_data = pd.read_csv(train_data_path)\n train_target = pd.read_csv(train_target_path)\n\n clf = SVC(kernel=kernel)\n clf.fit(train_data, train_target)\n\n with open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n\n input_example = train_data.sample(1)\n with open(input_example_path, \"wb\") as file_writer:\n dill.dump(input_example, file_writer)\n\n signature = infer_signature(train_data, clf.predict(train_data))\n with open(signature_path, \"wb\") as file_writer:\n dill.dump(signature, file_writer)\n\n conda_env = _mlflow_conda_env(\n additional_pip_deps=[\"dill\", \"pandas\", \"scikit-learn\"]\n )\n with open(conda_env_path, \"wb\") as file_writer:\n dill.dump(conda_env, file_writer)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Train from csv', description='')\n _parser.add_argument(\"--train-data\", dest=\"train_data_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--train-target\", dest=\"train_target_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--kernel\", dest=\"kernel\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--model\", dest=\"model_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--input-example\", dest=\"input_example_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--signature\", dest=\"signature_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--conda-env\", dest=\"conda_env_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = train_from_csv(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: kernel}\n artifacts:\n - {name: load-iris-data-data, path: /tmp/inputs/train_data/data}\n - {name: load-iris-data-target, path: /tmp/inputs/train_target/data}\n outputs:\n artifacts:\n - {name: train-from-csv-conda_env, path: /tmp/outputs/conda_env/data}\n - {name: train-from-csv-input_example, path: /tmp/outputs/input_example/data}\n - {name: train-from-csv-model, path: /tmp/outputs/model/data}\n - {name: train-from-csv-signature, path: /tmp/outputs/signature/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--train-data\", {\"inputPath\": \"train_data\"}, \"--train-target\",\n {\"inputPath\": \"train_target\"}, \"--kernel\", {\"inputValue\": \"kernel\"}, \"--model\",\n {\"outputPath\": \"model\"}, \"--input-example\", {\"outputPath\": \"input_example\"},\n \"--signature\", {\"outputPath\": \"signature\"}, \"--conda-env\", {\"outputPath\":\n \"conda_env\"}], \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location ''dill'' ''pandas''\n ''scikit-learn'' ''mlflow'' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m\n pip install --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn''\n ''mlflow'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef train_from_csv(\\n train_data_path,\\n train_target_path,\\n model_path,\\n input_example_path,\\n signature_path,\\n conda_env_path,\\n kernel,\\n):\\n import\n dill\\n import pandas as pd\\n from sklearn.svm import SVC\\n\\n from\n mlflow.models.signature import infer_signature\\n from mlflow.utils.environment\n import _mlflow_conda_env\\n\\n train_data = pd.read_csv(train_data_path)\\n train_target\n = pd.read_csv(train_target_path)\\n\\n clf = SVC(kernel=kernel)\\n clf.fit(train_data,\n train_target)\\n\\n with open(model_path, mode=\\\"wb\\\") as file_writer:\\n dill.dump(clf,\n file_writer)\\n\\n input_example = train_data.sample(1)\\n with open(input_example_path,\n \\\"wb\\\") as file_writer:\\n dill.dump(input_example, file_writer)\\n\\n signature\n = infer_signature(train_data, clf.predict(train_data))\\n with open(signature_path,\n \\\"wb\\\") as file_writer:\\n dill.dump(signature, file_writer)\\n\\n conda_env\n = _mlflow_conda_env(\\n additional_pip_deps=[\\\"dill\\\", \\\"pandas\\\",\n \\\"scikit-learn\\\"]\\n )\\n with open(conda_env_path, \\\"wb\\\") as file_writer:\\n dill.dump(conda_env,\n file_writer)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Train\n from csv'', description='''')\\n_parser.add_argument(\\\"--train-data\\\", dest=\\\"train_data_path\\\",\n type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--train-target\\\",\n dest=\\\"train_target_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--kernel\\\",\n dest=\\\"kernel\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--model\\\",\n dest=\\\"model_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--input-example\\\", dest=\\\"input_example_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--signature\\\",\n dest=\\\"signature_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--conda-env\\\", dest=\\\"conda_env_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = train_from_csv(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"train_data\", \"type\": \"csv\"},\n {\"name\": \"train_target\", \"type\": \"csv\"}, {\"name\": \"kernel\", \"type\": \"String\"}],\n \"name\": \"Train from csv\", \"outputs\": [{\"name\": \"model\", \"type\": \"dill\"},\n {\"name\": \"input_example\", \"type\": \"dill\"}, {\"name\": \"signature\", \"type\":\n \"dill\"}, {\"name\": \"conda_env\", \"type\": \"dill\"}]}', pipelines.kubeflow.org/component_ref: '{}',\n pipelines.kubeflow.org/arguments.parameters: '{\"kernel\": \"{{inputs.parameters.kernel}}\"}'}\n - name: upload-sklearn-model-to-mlflow\n container:\n args: [--model-name, '{{inputs.parameters.model_name}}', --model, /tmp/inputs/model/data,\n --input-example, /tmp/inputs/input_example/data, --signature, /tmp/inputs/signature/data,\n --conda-env, /tmp/inputs/conda_env/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'dill' 'pandas' 'scikit-learn' 'mlflow' 'boto3' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location 'dill' 'pandas' 'scikit-learn'\n 'mlflow' 'boto3' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def upload_sklearn_model_to_mlflow(\n model_name,\n model_path,\n input_example_path,\n signature_path,\n conda_env_path,\n ):\n import os\n import dill\n from mlflow.sklearn import save_model\n\n from mlflow.tracking.client import MlflowClient\n\n os.environ[\"MLFLOW_S3_ENDPOINT_URL\"] = \"http://minio-service.kubeflow.svc:9000\"\n os.environ[\"AWS_ACCESS_KEY_ID\"] = \"minio\"\n os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"minio123\"\n\n client = MlflowClient(\"http://mlflow-server-service.mlflow-system.svc:5000\")\n\n with open(model_path, mode=\"rb\") as file_reader:\n clf = dill.load(file_reader)\n\n with open(input_example_path, \"rb\") as file_reader:\n input_example = dill.load(file_reader)\n\n with open(signature_path, \"rb\") as file_reader:\n signature = dill.load(file_reader)\n\n with open(conda_env_path, \"rb\") as file_reader:\n conda_env = dill.load(file_reader)\n\n save_model(\n sk_model=clf,\n path=model_name,\n serialization_format=\"cloudpickle\",\n conda_env=conda_env,\n signature=signature,\n input_example=input_example,\n )\n run = client.create_run(experiment_id=\"0\")\n client.log_artifact(run.info.run_id, model_name)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Upload sklearn model to mlflow', description='')\n _parser.add_argument(\"--model-name\", dest=\"model_name\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--model\", dest=\"model_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--input-example\", dest=\"input_example_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--signature\", dest=\"signature_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--conda-env\", dest=\"conda_env_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = upload_sklearn_model_to_mlflow(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: model_name}\n artifacts:\n - {name: train-from-csv-conda_env, path: /tmp/inputs/conda_env/data}\n - {name: train-from-csv-input_example, path: /tmp/inputs/input_example/data}\n - {name: train-from-csv-model, path: /tmp/inputs/model/data}\n - {name: train-from-csv-signature, path: /tmp/inputs/signature/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--model-name\", {\"inputValue\": \"model_name\"}, \"--model\", {\"inputPath\":\n \"model\"}, \"--input-example\", {\"inputPath\": \"input_example\"}, \"--signature\",\n {\"inputPath\": \"signature\"}, \"--conda-env\", {\"inputPath\": \"conda_env\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn''\n ''mlflow'' ''boto3'' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install\n --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn'' ''mlflow''\n ''boto3'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def upload_sklearn_model_to_mlflow(\\n model_name,\\n model_path,\\n input_example_path,\\n signature_path,\\n conda_env_path,\\n):\\n import\n os\\n import dill\\n from mlflow.sklearn import save_model\\n\\n from\n mlflow.tracking.client import MlflowClient\\n\\n os.environ[\\\"MLFLOW_S3_ENDPOINT_URL\\\"]\n = \\\"http://minio-service.kubeflow.svc:9000\\\"\\n os.environ[\\\"AWS_ACCESS_KEY_ID\\\"]\n = \\\"minio\\\"\\n os.environ[\\\"AWS_SECRET_ACCESS_KEY\\\"] = \\\"minio123\\\"\\n\\n client\n = MlflowClient(\\\"http://mlflow-server-service.mlflow-system.svc:5000\\\")\\n\\n with\n open(model_path, mode=\\\"rb\\\") as file_reader:\\n clf = dill.load(file_reader)\\n\\n with\n open(input_example_path, \\\"rb\\\") as file_reader:\\n input_example\n = dill.load(file_reader)\\n\\n with open(signature_path, \\\"rb\\\") as file_reader:\\n signature\n = dill.load(file_reader)\\n\\n with open(conda_env_path, \\\"rb\\\") as file_reader:\\n conda_env\n = dill.load(file_reader)\\n\\n save_model(\\n sk_model=clf,\\n path=model_name,\\n serialization_format=\\\"cloudpickle\\\",\\n conda_env=conda_env,\\n signature=signature,\\n input_example=input_example,\\n )\\n run\n = client.create_run(experiment_id=\\\"0\\\")\\n client.log_artifact(run.info.run_id,\n model_name)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Upload\n sklearn model to mlflow'', description='''')\\n_parser.add_argument(\\\"--model-name\\\",\n dest=\\\"model_name\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--model\\\",\n dest=\\\"model_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--input-example\\\",\n dest=\\\"input_example_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--signature\\\",\n dest=\\\"signature_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--conda-env\\\",\n dest=\\\"conda_env_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = upload_sklearn_model_to_mlflow(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"model_name\", \"type\": \"String\"},\n {\"name\": \"model\", \"type\": \"dill\"}, {\"name\": \"input_example\", \"type\": \"dill\"},\n {\"name\": \"signature\", \"type\": \"dill\"}, {\"name\": \"conda_env\", \"type\": \"dill\"}],\n \"name\": \"Upload sklearn model to mlflow\"}', pipelines.kubeflow.org/component_ref: '{}',\n pipelines.kubeflow.org/arguments.parameters: '{\"model_name\": \"{{inputs.parameters.model_name}}\"}'}\n arguments:\n parameters:\n - {name: kernel}\n - {name: model_name}\n serviceAccountName: pipeline-runner\n```\n\n
\n \n \nexample_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: example-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9, pipelines.kubeflow.org/pipeline_compilation_time: '2021-12-09T18:11:43.193190',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"number_1\", \"type\":\n \"Integer\"}, {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"example_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9}\nspec:\n entrypoint: example-pipeline\n templates:\n - name: example-pipeline\n inputs:\n parameters:\n - {name: number_1}\n - {name: number_2}\n dag:\n tasks:\n - name: print-and-return-number\n template: print-and-return-number\n arguments:\n parameters:\n - {name: number_1, value: '{{inputs.parameters.number_1}}'}\n - name: print-and-return-number-2\n template: print-and-return-number-2\n arguments:\n parameters:\n - {name: number_2, value: '{{inputs.parameters.number_2}}'}\n - name: sum-and-print-numbers\n template: sum-and-print-numbers\n dependencies: [print-and-return-number, print-and-return-number-2]\n arguments:\n parameters:\n - {name: print-and-return-number-2-Output, value: '{{tasks.print-and-return-number-2.outputs.parameters.print-and-return-number-2-Output}}'}\n - {name: print-and-return-number-Output, value: '{{tasks.print-and-return-number.outputs.parameters.print-and-return-number-Output}}'}\n - name: print-and-return-number\n container:\n args: [--number, '{{inputs.parameters.number_1}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(\n str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_1}\n outputs:\n parameters:\n - name: print-and-return-number-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-Output, path: /tmp/outputs/Output/data}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is number 1, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\",\n {\"outputPath\": \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(\\n str(int_value),\n str(type(int_value))))\\n return str(int_value)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Print and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_1}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n - name: print-and-return-number-2\n container:\n args: [--number, '{{inputs.parameters.number_2}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(\n str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_2}\n outputs:\n parameters:\n - name: print-and-return-number-2-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-2-Output, path: /tmp/outputs/Output/data}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is number 2, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\",\n {\"outputPath\": \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(\\n str(int_value),\n str(type(int_value))))\\n return str(int_value)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Print and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_2}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n - name: sum-and-print-numbers\n container:\n args: [--number-1, '{{inputs.parameters.print-and-return-number-Output}}', --number-2,\n '{{inputs.parameters.print-and-return-number-2-Output}}']\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def sum_and_print_numbers(number_1, number_2):\n print(number_1 + number_2)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Sum and print numbers', description='')\n _parser.add_argument(\"--number-1\", dest=\"number_1\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--number-2\", dest=\"number_2\", type=int, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = sum_and_print_numbers(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: print-and-return-number-2-Output}\n - {name: print-and-return-number-Output}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is sum of number\n 1 and number 2, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number-1\", {\"inputValue\": \"number_1\"}, \"--number-2\",\n {\"inputValue\": \"number_2\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def sum_and_print_numbers(number_1, number_2):\\n print(number_1 + number_2)\\n\\nimport\n argparse\\n_parser = argparse.ArgumentParser(prog=''Sum and print numbers'',\n description='''')\\n_parser.add_argument(\\\"--number-1\\\", dest=\\\"number_1\\\",\n type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--number-2\\\",\n dest=\\\"number_2\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = sum_and_print_numbers(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number_1\", \"type\": \"Integer\"},\n {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"Sum and print numbers\"}',\n pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number_1\":\n \"{{inputs.parameters.print-and-return-number-Output}}\", \"number_2\": \"{{inputs.parameters.print-and-return-number-2-Output}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n arguments:\n parameters:\n - {name: number_1}\n - {name: number_2}\n serviceAccountName: pipeline-runner\n```\n\n
\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n```\n\nIf written in pipeline, it will be like this.\n\n```python\nfrom functools import partial\n\nimport kfp\nfrom kfp.components import create_component_from_func, OutputPath\nfrom kfp.dsl import pipeline\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"matplotlib\"],\n)\ndef plot_linear(mlpipeline_ui_metadata: OutputPath(\"UI_Metadata\")):\n import base64\n import json\n from io import BytesIO\n\n import matplotlib.pyplot as plt\n\n plt.plot([1, 2, 3], [1, 2, 3])\n\n tmpfile = BytesIO()\n plt.savefig(tmpfile, format=\"png\")\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\"utf-8\")\n\n html = f\"\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n\n\n@pipeline(name=\"plot_pipeline\")\ndef plot_pipeline():\n plot_linear()\n\n\nif __name__ == \"__main__\":\n kfp.compiler.Compiler().compile(plot_pipeline, \"plot_pipeline.yaml\")\n```\n\nIf you run this script and check the resulting `plot_pipeline.yaml`, you will see the following.\n\n\n \n \nplot_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: plot-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9, pipelines.kubeflow.org/pipeline_compilation_time: '2\n022-01-17T13:31:32.963214',\n pipelines.kubeflow.org/pipeline_spec: '{\"name\": \"plot_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9}\nspec:\n entrypoint: plot-pipeline\n templates:\n - name: plot-linear\n container:\n args: [--mlpipeline-ui-metadata, /tmp/outputs/mlpipeline_ui_metadata/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'matplotlib' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet\n --no-warn-script-location 'matplotlib' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n def plot_linear(mlpipeline_ui_metadata):\n import base64\n import json\n from io import BytesIO\n import matplotlib.pyplot as plt\n plt.plot([1, 2, 3], [1, 2, 3])\n tmpfile = BytesIO()\n plt.savefig(tmpfile, format=\"png\")\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\"utf-8\")\n html = f\"\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Plot linear', description='')\n _parser.add_argument(\"--mlpipeline-ui-metadata\", dest=\"mlpipeline_ui_metadata\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n _outputs = plot_linear(**_parsed_args)\n image: python:3.7\n outputs:\n artifacts:\n - {name: mlpipeline-ui-metadata, path: /tmp/outputs/mlpipeline_ui_metadata/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--mlpipeline-ui-metadata\", {\"outputPath\": \"mlpipeline_ui_metadata\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''matplotlib'' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location ''matplotlib''\n --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef plot_linear(mlpipeline_ui_metadata):\\n import\n base64\\n import json\\n from io import BytesIO\\n\\n import matplotlib.pyplot\n as plt\\n\\n plt.plot([1, 2, 3], [1, 2, 3])\\n\\n tmpfile = BytesIO()\\n plt.savefig(tmpfile,\n format=\\\"png\\\")\\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\\\"utf-8\\\")\\n\\n html\n = f\\\"\\\"\\n metadata = {\\n \\\"outputs\\\":\n [\\n {\\n \\\"type\\\": \\\"web-app\\\",\\n \\\"storage\\\":\n \\\"inline\\\",\\n \\\"source\\\": html,\\n },\\n ],\\n }\\n with\n open(mlpipeline_ui_metadata, \\\"w\\\") as html_writer:\\n json.dump(metadata,\n html_writer)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Plot\n linear'', description='''')\\n_parser.add_argument(\\\"--mlpipeline-ui-metadata\\\",\n dest=\\\"mlpipeline_ui_metadata\\\", type=_make_parent_dirs_and_return_path,\n required=True, default=argparse.SUPPRESS)\\n_parsed_args = vars(_parser.parse_args())\\n\\n_outputs\n = plot_linear(**_parsed_args)\\n\"], \"image\": \"python:3.7\"}}, \"name\": \"Plot\n linear\", \"outputs\": [{\"name\": \"mlpipeline_ui_metadata\", \"type\": \"UI_Metadata\"}]}',\n pipelines.kubeflow.org/component_ref: '{}'}\n - name: plot-pipeline\n dag:\n tasks:\n - {name: plot-linear, template: plot-linear}\n arguments:\n parameters: []\n serviceAccountName: pipeline-runner\n```\n\n
\n
\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow/basic-requirements.md",
"content": "---\ntitle : \"3. Install Requirements\"\ndescription: \"\"\nsidebar_position: 3\ncontributors: [\"Jongseob Jeon\"]\n---\n\nThe recommended Python version for practice is python>=3.7. For those unfamiliar with the Python environment, please refer to [Appendix 1. Python Virtual Environment](../appendix/pyenv) and install the packages on the **client node**.\n\nThe packages and versions required for the practice are as follows:\n\n- requirements.txt\n\n ```bash\n kfp==1.8.9\n scikit-learn==1.0.1\n mlflow==1.21.0\n pandas==1.3.4\n dill==0.3.4\n ```\n\nActivate the [Python virtual environment](../appendix/pyenv.md#python-가상환경-생성) created in the previous section.\n\n```bash\npyenv activate demo\n```\n\nWe are proceeding with the package installation.\n\n```bash\npip3 install -U pip\npip3 install kfp==1.8.9 scikit-learn==1.0.1 mlflow==1.21.0 pandas==1.3.4 dill==0.3.4\n```\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow/basic-run.md",
"content": "---\ntitle : \"7. Pipeline - Run\"\ndescription: \"\"\nsidebar_position: 7\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Run Pipeline\n\nNow we will run the uploaded pipeline.\n\n## Before Run\n\n### 1. Create Experiment\n\nExperiments in Kubeflow are units that logically manage runs executed within them.\n\nWhen you first enter the namespace in Kubeflow, there are no Experiments created. Therefore, you must create an Experiment beforehand in order to run the pipeline. If an Experiment already exists, you can go to [Run Pipeline](../kubeflow/basic-run.md#run-pipeline-1).\n\nExperiments can be created via the Create Experiment button.\n\n\n\n### 2. Name 입력\n\n\n\n## Run Pipeline\n\n### 1. Select Create Run\n\n\n\n### 2. Select Experiment\n\n\n\n\n\n### 3. Enter Pipeline Config\n\nFill in the values of the Config provided when creating the pipeline. The uploaded pipeline requires input values for `number_1` and `number_2`.\n\n\n\n### 4. Start\n\nClick the Start button after entering the values. The pipeline will start running.\n\n\n\n## Run Result\n\nThe executed pipelines can be viewed in the Runs tab.\nClicking on a run provides detailed information related to the executed pipeline.\n\n\n\nUpon clicking, the following screen appears. Components that have not yet executed are displayed in gray.\n\n\n\nWhen a component has completed execution, it is marked with a green checkmark.\n\n\n\nIf we look at the last component, we can see that it has outputted the sum of the input values, which in this case is 8 (the sum of 3 and 5).\n\n\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow/how-to-debug.md",
"content": "---\ntitle : \"13. Component - Debugging\"\ndescription: \"\"\nsidebar_position: 13\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Debugging Pipeline\n\nThis page covers how to debug Kubeflow components.\n\n## Failed Component\n\nWe will modify a pipeline used in [Component - MLFlow](../kubeflow/advanced-mlflow.md#mlflow-pipeline) in this page.\n\nFirst, let's modify the pipeline so that the component fails.\n\n```python\nfrom functools import partial\n\nimport kfp\nfrom kfp.components import InputPath, OutputPath, create_component_from_func\nfrom kfp.dsl import pipeline\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\", \"scikit-learn\"],\n)\ndef load_iris_data(\n data_path: OutputPath(\"csv\"),\n target_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n from sklearn.datasets import load_iris\n\n iris = load_iris()\n\n data = pd.DataFrame(iris[\"data\"], columns=iris[\"feature_names\"])\n target = pd.DataFrame(iris[\"target\"], columns=[\"target\"])\n \n data[\"sepal length (cm)\"] = None\n data.to_csv(data_path, index=False)\n target.to_csv(target_path, index=False)\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\"],\n)\ndef drop_na_from_csv(\n data_path: InputPath(\"csv\"),\n output_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n\n data = pd.read_csv(data_path)\n data = data.dropna()\n data.to_csv(output_path, index=False)\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"dill\", \"pandas\", \"scikit-learn\", \"mlflow\"],\n)\ndef train_from_csv(\n train_data_path: InputPath(\"csv\"),\n train_target_path: InputPath(\"csv\"),\n model_path: OutputPath(\"dill\"),\n input_example_path: OutputPath(\"dill\"),\n signature_path: OutputPath(\"dill\"),\n conda_env_path: OutputPath(\"dill\"),\n kernel: str,\n):\n import dill\n import pandas as pd\n from sklearn.svm import SVC\n\n from mlflow.models.signature import infer_signature\n from mlflow.utils.environment import _mlflow_conda_env\n\n train_data = pd.read_csv(train_data_path)\n train_target = pd.read_csv(train_target_path)\n\n clf = SVC(kernel=kernel)\n clf.fit(train_data, train_target)\n\n with open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n\n input_example = train_data.sample(1)\n with open(input_example_path, \"wb\") as file_writer:\n dill.dump(input_example, file_writer)\n\n signature = infer_signature(train_data, clf.predict(train_data))\n with open(signature_path, \"wb\") as file_writer:\n dill.dump(signature, file_writer)\n\n conda_env = _mlflow_conda_env(\n additional_pip_deps=[\"dill\", \"pandas\", \"scikit-learn\"]\n )\n with open(conda_env_path, \"wb\") as file_writer:\n dill.dump(conda_env, file_writer)\n\n\n\n@pipeline(name=\"debugging_pipeline\")\ndef debugging_pipeline(kernel: str):\n iris_data = load_iris_data()\n drop_data = drop_na_from_csv(data=iris_data.outputs[\"data\"])\n model = train_from_csv(\n train_data=drop_data.outputs[\"output\"],\n train_target=iris_data.outputs[\"target\"],\n kernel=kernel,\n )\n\nif __name__ == \"__main__\":\n kfp.compiler.Compiler().compile(debugging_pipeline, \"debugging_pipeline.yaml\")\n\n```\n\nThe modifications are as follows:\n\n1. In the `load_iris_data` component for loading data, `None` was injected into the `sepal length (cm)` feature.\n2. In the `drop_na_from_csv` component, use the `drop_na()` function to remove rows with na values.\n\nNow let's upload and run the pipeline. \nAfter running, if you press Run you will see that it has failed in the `Train from csv` component.\n\n\n\nClick on the failed component and check the log to see the reason for the failure.\n\n\n\nIf the log shows that the data count is 0 and the component did not run, there may be an issue with the input data. \nLet's investigate what might be the problem.\n\nFirst, click on the component and go to the Input/Output tab to download the input data. \nYou can click on the link indicated by the red square to download the data.\n\n\n\n\nDownload both files to the same location. Then navigate to the specified path and check the downloaded files.\n\n\n```bash\nls\n```\n\nThere are two files as follows.\n\n```bash\ndrop-na-from-csv-output.tgz load-iris-data-target.tgz\n```\n\nI will try to unzip it.\n\n```bash\ntar -xzvf load-iris-data-target.tgz ; mv data target.csv\ntar -xzvf drop-na-from-csv-output.tgz ; mv data data.csv\n```\n\nAnd then run the component code using a Jupyter notebook.\n\n\nDebugging revealed that dropping the data was based on rows instead of columns, resulting in all the data being removed.\nNow that we know the cause of the problem, we can modify the component to drop based on columns.\n\n```python\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\"],\n)\ndef drop_na_from_csv(\n data_path: InputPath(\"csv\"),\n output_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n\n data = pd.read_csv(data_path)\n data = data.dropna(axis=\"columns\")\n data.to_csv(output_path, index=False)\n```\n\nAfter modifying, upload the pipeline again and run it to confirm that it is running normally as follows.\n\n\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow/kubeflow-concepts.md",
"content": "---\ntitle : \"2. Kubeflow Concepts\"\ndescription: \"\"\nsidebar_position: 2\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Component\n\nA component is composed of Component contents and a Component wrapper.\nA single component is delivered to Kubeflow through a Component wrapper and the delivered component executes the defined Component contents and produces artifacts.\n\n\n\n### Component Contents\n\nThere are three components that make up the component contents:\n\n\n\n1. Environment\n2. Python code w/ Config\n3. Generates Artifacts\n\nLet's explore each component with an example.\nHere is a Python code that loads data, trains an SVC (Support Vector Classifier) model, and saves the SVC model.\n\n```python\nimport dill\nimport pandas as pd\n\nfrom sklearn.svm import SVC\n\ntrain_data = pd.read_csv(train_data_path)\ntrain_target= pd.read_csv(train_target_path)\n\nclf= SVC(\n kernel=kernel\n)\nclf.fit(train_data)\n\nwith open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n```\n\nThe above Python code can be divided into components contents as follows.\n\n\n\nEnvironment is the part of the Python code where the packages used in the code are imported. \nNext, Python Code w\\ Config is where the given Config is used to actually perform the training. \nFinally, there is a process to save the artifacts. \n\n### Component Wrapper\n\nComponent wrappers deliver the necessary Config and execute tasks for component content.\n\n\n\nIn Kubeflow, component wrappers are defined as functions, similar to the `train_svc_from_csv` example above.\nWhen a component wrapper wraps the contents, it looks like the following:\n\n\n\n### Artifacts\n\nIn the explanation above, it was mentioned that the component creates Artifacts. Artifacts is a term used to refer to any form of a file that is generated, such as evaluation results, logs, etc.\nOf the ones that we are interested in, the following are significant: Models, Data, Metrics, and etc.\n\n\n\n- Model\n- Data\n- Metric\n- etc\n\n#### Model\n\nWe defined the model as follows: \n\n> A model is a form that includes Python code, trained weights and network architecture, and an environment to run it.\n\n#### Data\n\nData includes preprocessed features, model predictions, etc. \n\n#### Metric\n\nMetric is divided into two categories: dynamic metrics and static metrics.\n\n- Dynamic metrics refer to values that continuously change during the training process, such as train loss per epoch.\n- Static metrics refer to evaluation metrics, such as accuracy, that are calculated after the training is completed.\n\n## Pipeline\n\nA pipeline consists of a collection of components and the order in which they are executed. The order forms a directed acyclic graph (DAG), which can include simple conditional statements.\n\n\n\n### Pipeline Config\n\nAs mentioned earlier, components require config to be executed. The pipeline config contains the configs for all the components in the pipeline.\n\n\n\n## Run\n\nTo execute a pipeline, the pipeline config specific to that pipeline is required. In Kubeflow, an executed pipeline is called a \"Run.\"\n\n\n\nWhen a pipeline is executed, each component generates artifacts. Kubeflow pipeline assigns a unique ID to each Run, and all artifacts generated during the Run are stored.\n\n\n\nNow, let's learn how to write components and pipelines.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow/kubeflow-intro.md",
"content": "---\ntitle : \"1. Kubeflow Introduction\"\ndescription: \"\"\nsidebar_position: 1\ncontributors: [\"Jongseob Jeon\"]\n---\n\nTo use Kubeflow, you need to write components and pipelines.\n\nThe approach described in *MLOps for ALL* differs slightly from the method described on the [Kubeflow Pipeline official website](https://www.kubeflow.org/docs/components/pipelines/overview/quickstart/). Here, Kubeflow Pipeline is used as one of the components in the [elements that make up MLOps](../kubeflow/kubeflow-concepts.md#component-contents) rather than a standalone workflow.\n\nNow, let's understand what components and pipelines are and how to write them.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow-dashboard-guide/_category_.json",
"content": "{\n \"label\": \"Kubeflow UI Guide\",\n \"position\": 5,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow-dashboard-guide/experiments-and-others.md",
"content": "---\ntitle : \"6. Kubeflow Pipeline Relates\"\ndescription: \"\"\nsidebar_position: 6\ncontributors: [\"Jaeyeon Kim\"]\n---\n\nIn the left tabs of the Central Dashboard (KFP Experiments, Pipelines, Runs, Recurring Runs, Artifacts, Executions) you can manage Kubeflow Pipelines and the results of Pipeline execution and Pipeline Runs.\n\n\n\nKubeflow Pipelines are the main reason for using Kubeflow in *MLOps for ALL*, and details on how to create, execute, and check the results of Kubeflow Pipelines can be found in [3.Kubeflow](../kubeflow/kubeflow-intro).\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow-dashboard-guide/experiments.md",
"content": "---\ntitle : \"5. Experiments(AutoML)\"\ndescription: \"\"\nsidebar_position: 5\ncontributors: [\"Jaeyeon Kim\"]\n---\n\nNext, we will click the Experiments(AutoML) tab on the left of the Central Dashboard.\n\n\n\n\n\nThe Experiments(AutoML) page is where you can manage [Katib](https://www.kubeflow.org/docs/components/katib/overview/), which is responsible for AutoML through Hyperparameter Tuning and Neural Architecture Search in Kubeflow.\n\nThe usage of Katib and Experiments(AutoML) is not covered in *MLOps for Everyone* v1.0, and will be added in v2.0.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow-dashboard-guide/intro.md",
"content": "---\ntitle : \"1. Central Dashboard\"\ndescription: \"\"\nsidebar_position: 1\ncontributors: [\"Jaeyeon Kim\", \"SeungTae Kim\"]\n---\n\nOnce you have completed [Kubeflow installation](../setup-components/install-components-kf.md), you can access the dashboard through the following command.\n\n```bash\nkubectl port-forward --address 0.0.0.0 svc/istio-ingressgateway -n istio-system 8080:80\n```\n\n\n\nThe Central Dashboard is a UI that integrates all the features provided by Kubeflow. The features provided by the Central Dashboard can be divided based on the tabs on the left side\n\n\n\n- Home\n- Notebooks\n- Tensorboards\n- Volumes\n- Models\n- Experiments(AutoML)\n- Experiments(KFP)\n- Pipelines\n- Runs\n- Recurring Runs\n- Artifacts\n- Executions\n\nLet's now look at the simple usage of each feature.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow-dashboard-guide/notebooks.md",
"content": "---\ntitle : \"2. Notebooks\"\ndescription: \"\"\nsidebar_position: 2\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n## Launch Notebook Server\n\nClick on the Notebooks tab on the left side of the Central Dashboard.\n\n\n\nYou will see a similar screen.\n\nThe Notebooks tab is a page where users can independently create and access jupyter notebook and code server environments (hereinafter referred to as a notebook server).\n\n\n\nClick the \"+ NEW NOTEBOOK\" button at the top right. \n\n\n\nWhen the screen shown below appears, now specify the spec (Spec) of the notebook server to be created.\n\n\n\n\nexample_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: example-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline_compilation_time: '2021-12-05T13:38:51.566777',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"number_1\", \"type\":\n \"Integer\"}, {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"example_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3}\nspec:\n entrypoint: example-pipeline\n templates:\n - name: example-pipeline\n inputs:\n parameters:\n - {name: number_1}\n - {name: number_2}\n dag:\n tasks:\n - name: print-and-return-number\n template: print-and-return-number\n arguments:\n parameters:\n - {name: number_1, value: '{{inputs.parameters.number_1}}'}\n - name: print-and-return-number-2\n template: print-and-return-number-2\n arguments:\n parameters:\n - {name: number_2, value: '{{inputs.parameters.number_2}}'}\n - name: sum-and-print-numbers\n template: sum-and-print-numbers\n dependencies: [print-and-return-number, print-and-return-number-2]\n arguments:\n parameters:\n - {name: print-and-return-number-2-Output, value: '{{tasks.print-and-return-number-2.outputs.parameters.print-and-return-number-2-Output}}'}\n - {name: print-and-return-number-Output, value: '{{tasks.print-and-return-number.outputs.parameters.print-and-return-number-Output}}'}\n - name: print-and-return-number\n container:\n args: [--number, '{{inputs.parameters.number_1}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_1}\n outputs:\n parameters:\n - name: print-and-return-number-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-Output, path: /tmp/outputs/Output/data}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\", {\"outputPath\":\n \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(str(int_value), str(type(int_value))))\\n return\n str(int_value)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Print\n and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_1}}\"}'}\n - name: print-and-return-number-2\n container:\n args: [--number, '{{inputs.parameters.number_2}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_2}\n outputs:\n parameters:\n - name: print-and-return-number-2-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-2-Output, path: /tmp/outputs/Output/data}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\", {\"outputPath\":\n \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(str(int_value), str(type(int_value))))\\n return\n str(int_value)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Print\n and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_2}}\"}'}\n - name: sum-and-print-numbers\n container:\n args: [--number-1, '{{inputs.parameters.print-and-return-number-Output}}', --number-2,\n '{{inputs.parameters.print-and-return-number-2-Output}}']\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def sum_and_print_numbers(number_1, number_2):\n print(number_1 + number_2)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Sum and print numbers', description='')\n _parser.add_argument(\"--number-1\", dest=\"number_1\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--number-2\", dest=\"number_2\", type=int, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = sum_and_print_numbers(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: print-and-return-number-2-Output}\n - {name: print-and-return-number-Output}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number-1\", {\"inputValue\": \"number_1\"}, \"--number-2\", {\"inputValue\":\n \"number_2\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n sum_and_print_numbers(number_1, number_2):\\n print(number_1 + number_2)\\n\\nimport\n argparse\\n_parser = argparse.ArgumentParser(prog=''Sum and print numbers'',\n description='''')\\n_parser.add_argument(\\\"--number-1\\\", dest=\\\"number_1\\\",\n type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--number-2\\\",\n dest=\\\"number_2\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = sum_and_print_numbers(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number_1\", \"type\": \"Integer\"},\n {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"Sum and print numbers\"}',\n pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number_1\":\n \"{{inputs.parameters.print-and-return-number-Output}}\", \"number_2\": \"{{inputs.parameters.print-and-return-number-2-Output}}\"}'}\n arguments:\n parameters:\n - {name: number_1}\n - {name: number_2}\n serviceAccountName: pipeline-runner\n```\n\n\n
\n\nIf you followed the [Setup GPU (Optional)](../setup-kubernetes/setup-nvidia-gpu.md), select NVIDIA if you have installed the nvidia gpu plugin.\n\n\n\nAfter creation, the **Status** will change to a green check mark icon, and the **CONNECT button** will be activated.\n\n\n---\n## Accessing the Notebook Server\n\nClicking the **CONNECT button** will open a new browser window, where you will see the following screen:\n\n\n\nYou can use the Notebook, Console, and Terminal icons in the **Launcher** to start using them.\n\n Notebook Interface\n\n\n\n Terminal Interface\n\n\n\n---\n\n## Stopping the Notebook Server\n\nIf you haven't used the notebook server for an extended period of time, you can stop it to optimize resource usage in the Kubernetes cluster. **Note that stopping the notebook server will result in the deletion of all data stored outside the Workspace Volume or Data Volume specified when creating the notebook server.** \nIf you haven't changed the path during notebook server creation, the default Workspace Volume path is `/home/jovyan` inside the notebook server, so any data stored outside the `/home/jovyan` directory will be deleted.\n\nClicking the `STOP` button as shown below will stop the notebook server:\n\n\n\nOnce the server is stopped, the `CONNECT` button will be disabled. To restart the notebook server and use it again, click the `PLAY` button.\n\n\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow-dashboard-guide/tensorboards.md",
"content": "---\ntitle : \"3. Tensorboards\"\ndescription: \"\"\nsidebar_position: 3\ncontributors: [\"Jaeyeon Kim\"]\n---\n\nLet's click on the Tensorboards tab of the left tabs of the Central Dashboard next.\n\n\n\nWe can see the following screen. \n\n\n\nThe TensorBoard server created in this way can be used just like a regular remote TensorBoard server, or it can be used for the purpose of storing data directly from a Kubeflow Pipeline run for visualization purposes.\n\nYou can refer to the [TensorBoard documentation](https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#tensorboard) for more information on using TensorBoard with Kubeflow Pipeline runs.\n\nThere are various ways to visualize the results of Kubeflow Pipeline runs, and in *MLOps for ALL*, we will utilize the Visualization feature of Kubeflow components and the visualization capabilities of MLflow to enable more general use cases. Therefore, detailed explanations of the TensorBoards page will be omitted in this context.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/kubeflow-dashboard-guide/volumes.md",
"content": "---\ntitle : \"4. Volumes\"\ndescription: \"\"\nsidebar_position: 4\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n## Volumes\n\nNext, let's click on the Volumes tab in the left of the Central Dashboard.\n\n\n\nYou will see the following screen.\n\n\n\n\nVolumes tab provides the functionality to manage the Persistent Volume Claims (PVC) belonging to the current user's namespace in Kubernetes' Volume (Volume).\n\nBy looking at the screenshot, you can see the information of the Volume created on the [1. Notebooks](../kubeflow-dashboard-guide/notebooks) page. It can be seen that the Storage Class of the Volume is set to local-path, which is the Default Storage Class installed at the time of Kubernetes cluster installation.\n\nIn addition, the Volumes page can be used if you want to create, view, or delete a new Volume in the user namespace.\n\n---\n\n## Creating a Volume\n\nBy clicking the `+ NEW VOLUME` button at the top right, you can see the following screen.\n\n\n\n\nYou can create a volume by specifying its name, size, storage class, and access mode.\n\nWhen you specify the desired resource specs to create a volume, its Status will be shown as Pending on this page. When you hover over the Status icon, you will see a message that this *(This volume will be bound when its first consumer is created.)* \nThis is according to the volume creation policy of the [StorageClass](https://kubernetes.io/ko/docs/concepts/storage/storage-classes/) used in the lab, which is local-path. **This is not a problem situation.** \nWhen the Status is shown as Pending on this page, you can still specify the name of the volume in the notebook server or pod that you want to use the volume and the volume creation will be triggered at that time.\n\n\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/prerequisites/_category_.json",
"content": "{\n \"label\": \"Prerequisites\",\n \"position\": 1,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/prerequisites/docker/_category_.json",
"content": "{\n \"label\": \"Docker\",\n \"position\": 1,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/prerequisites/docker/advanced.md",
"content": "---\ntitle : \"[Practice] Docker Advanced\"\ndescription: \"Practice to use docker more advanced way.\"\nsidebar_position: 6\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n## Making a good Docker image\n\n### Considerations to make Docker image:\n\nWhen creating a Docker image using a Dockerfile, the **order** of the commands is important. \nThis is because Docker images are composed of many Read-Only layers and when building the image, existing layers are **cached** and reused, so if you structure your Dockerfile with this in mind, you can **reduce the build time**.\n\nEach of the `RUN`, `ADD`, `COPY` commands in a Dockerfile are stored as one layer.\n\nFor example, if we have the following `Dockerfile`:\n\n```docker\n# Layer 1\nFROM ubuntu:latest\n\n# Layer 2\nRUN apt-get update && apt-get install python3 pip3 -y\n\n# Layer 3\nRUN pip3 install -U pip && pip3 install torch\n\n# Layer 4\nCOPY src/ src/\n\n# Layer 5\nCMD python src/app.py\n```\n\nIf you run the image built with the above `Dockerfile` with the command `docker run -it app:latest /bin/bash`, it can be represented in the following layers. \n\n\n\nThe topmost R/W layer does not affect the image. In other words, any changes made inside the container are volatile.\n\nWhen a lower layer is changed, all the layers above it need to be rebuilt. Therefore, the order of Dockerfile instructions is important. It is recommended to place the parts that are frequently changed towards the end. (e.g., `COPY src/ app/src/`)\n\nConversely, parts that are unlikely to change should be placed towards the beginning.\n\nIf there are parts that are rarely changed but used in multiple places, they can be consolidated. It is advisable to create a separate image for those common parts in advance and use it as a base image.\n\nFor example, if you want to create separate images for an environment that uses `tensorflow-cpu` and another environment that uses `tensorflow-gpu`, you can do the following:\nCreate a base image [`ghcr.io/makinarocks/python:3.8-base`](http://ghcr.io/makinarocks/python:3.8-base-cpu) that includes Python and other basic packages installed. Then, when creating the images with the CPU and GPU versions of TensorFlow, you can use the base image as the `FROM` instruction and write the separate instructions for installing TensorFlow in each Dockerfile. Managing two Dockerfiles in this way improves readability and reduces build time.\n\nCombining layers had performance benefits in older versions of Docker. However, since you cannot guarantee the Docker version in which your Docker containers will run, it is recommended to combine layers for readability purposes. It is best to combine layers that can be combined appropriately.\n\nHere is an example of a Dockerfile:\n\n```docker\n# Bad Case\nRUN apt-get update\nRUN apt-get install build-essential -y\nRUN apt-get install curl -y\nRUN apt-get install jq -y\nRUN apt-get install git -y\n```\n\nThis can be written by combining it as follows.\n\n```docker\n# Better Case\nRUN apt-get update && \\\n apt-get install -y \\\n build-essential \\\n curl \\\n jq \\\n git\n```\n\nFor convenience, it is better to use `.dockerignore`. \n`.dockerignore` is similar to `.gitignore` in the sense that it can be excluded when doing a `docker build` just like when doing a `git add`. \n\nMore information can be found in the [Docker Official Documentation](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/).\n\n### ENTRYPOINT vs CMD\n\n`ENTRYPOINT` and `CMD` are both used when you want to execute a command at the runtime of the container. One of them must be present in the Dockerfile.\n\n- **Difference**\n - `CMD`: Easily modifiable when running `docker run` command\n - `ENTRYPOINT`: Requires the use of `--entrypoint` to modify\n\nWhen `ENTRYPOINT` and `CMD` are used together, `CMD` typically represents the arguments (parameters) for the command specified in `ENTRYPOINT`.\n\nFor example, consider the following Dockerfile:\n\n```docker\nFROM ubuntu:latest\n\n# 아래 4 가지 option 을 바꿔가며 직접 테스트해보시면 이해하기 편합니다.\n# 단, NO ENTRYPOINT 옵션은 base image 인 ubuntu:latest 에 이미 있어서 테스트해볼 수는 없고 나머지 v2, 3, 5, 6, 8, 9, 11, 12 를 테스트해볼 수 있습니다.\n# ENTRYPOINT echo \"Hello ENTRYPOINT\"\n# ENTRYPOINT [\"echo\", \"Hello ENTRYPOINT\"]\n# CMD echo \"Hello CMD\"\n# CMD [\"echo\", \"Hello CMD\"]\n```\n\n\nIf you build and run the above `Dockerfile` with the parts marked as comments deactivated, you can get the following results: \n\n| | No ENTRYPOINT | ENTRYPOINT a b | ENTRYPOINT [\"a\", \"b\"] |\n| ------------------ | -------------- | -------------- | --------------------- |\n| **NO CMD** | Error! | /bin/sh -c a b | a b |\n| **CMD [\"x\", \"y\"]** | x y | /bin/sh -c a b | a b x y |\n| **CMD x y** | /bin/sh -c x y | /bin/sh -c a b | a b /bin/sh -c x y |\n\n- In Kubernetes pod, \n - `ENTRYPOINT` corresponds to the command\n - `CMD` corresponds to the arguments\n\n### Naming docker tag\n\nRecommend not using \"latest\" as a tag for a Docker image, as it is the default tag name and can be easily overwritten unintentionally.\n\nIt is important to ensure uniqueness of one image with one tag for the sake of collaboration and debugging in the production stage. \nUsing the same tag for different contents can lead to dangling images, which are not shown in the `docker images` but still take up storage space.\n\n### ETC\n\n1. Logs and other information are stored separately from the container, not inside it.\n This is because data written from within the container can be lost at any time.\n2. Secrets and environment-dependent information should not be written directly into the Dockerfile but should be passed in via environment variables or a .env config file.\n3. There is a **linter** for Dockerfiles, so it is useful to use it when collaborating.\n [https://github.com/hadolint/hadolint](https://github.com/hadolint/hadolint)\n\n## Several options for docker run\n\nWhen using Docker containers, there are some inconveniences.\nSpecifically, Docker does not store any of the work done within the Docker container by default.\nThis is because Docker containers use isolated file systems. Therefore, it is difficult to share data between multiple Docker containers.\n\nTo solve this problem, there are two approaches offered by Docker.\n\n\n\n#### Docker volume\n\n- Use the Docker CLI to directly manage a resource called `volume`.\n- Create a specific directory under the Docker area (`/var/lib/docker`) on the host and mount that path to a Docker container.\n\n#### Bind mount\n\n- Mount a specific path on the host to a Docker container.\n\n#### How to use?\n\nThe usage is through the same interface, using the `-v` option. \nHowever, when using volumes, you need to manage them directly by performing commands like `docker volume create`, `docker volume ls`, `docker volume rm`, etc.\n\n- Docker volume\n\n ```bash\n docker run \\\n -v my_volume:/app \\\n nginx:latest\n ````\n\n- Blind mount\n\n ```bash\n docker run \\\n -v /home/user/some/path:/app \\\n nginx:latest\n ```\n\nWhen developing locally, bind mount can be convenient, but if you want to maintain a clean environment, using Docker volume and explicitly performing create and rm operations can be another approach.\n\nThe way storage is provided in Kubernetes ultimately relies on Docker's bind mount as well.\n\n### Docker run with resource limit\n\nBasically, docker containers can **fully utilize the CPU and memory resources of the host OS**. However, when using this, depending on the resource situation of the host OS, docker containers may abnormally terminate due to issues such as **OOM**.\nTo address this problem, docker provides the `-m` [option](https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory) which allows you to **limit the usage of CPU and memory** when running the docker container.\n\n```bash\ndocker run -d -m 512m --memory-reservation=256m --name 512-limit ubuntu sleep 3600\ndocker run -d -m 1g --memory-reservation=256m --name 1g-limit ubuntu sleep 3600\n```\n\nAfter running the Docker above, you can check the usage through the 'docker stats' command.\n\n```bash\nCONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS\n4ea1258e2e09 1g-limit 0.00% 300KiB / 1GiB 0.03% 1kB / 0B 0B / 0B 1\n4edf94b9a3e5 512-limit 0.00% 296KiB / 512MiB 0.06% 1.11kB / 0B 0B / 0B 1\n```\n\nIn Kubernetes, when you limit the CPU and memory resources of a pod resource, it is provided using this technique.\n\n### docker run with restart policy\n\nIf there is a need to keep a particular container running continuously, the `--restart=always` option is provided to try to re-create the container immediately after it is terminated.\n\nAfter entering the option, run the docker.\n\n```bash\ndocker run --restart=always ubuntu\n```\n\nRun `watch -n1 docker ps` to check if it is restarting.\nIf it is running normally, `Restarting (0)` will be printed in STATUS.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\na911850276e8 ubuntu \"bash\" 35 seconds ago Restarting (0) 6 seconds ago hungry_vaughan\n```\n\n- [https://docs.docker.com/engine/reference/commandline/run/#restart-policies---restart](https://docs.docker.com/engine/reference/commandline/run/#restart-policies---restart)\n - Provides options such as \"on-failure with max retries\" and \"always\"\n\nWhen specifying the restart option for a job resource in Kubernetes, this approach is used.\n\n### Running docker run as a background process\n\nBy default, when running a Docker container, it is executed as a foreground process. This means that the terminal that launched the container is automatically attached to it, preventing you from running other commands.\n\nLet's try an example. Open two terminals, and in one terminal, continuously monitor `docker ps`, while in the other terminal, execute the following commands one by one and observe the behavior.\n\n#### First Practice\n\n```bash\ndocker run -it ubuntu sleep 10\n```\n\nYou must remain stopped for 10 seconds and you cannot perform any other commands from that container. After 10 seconds, you can check in docker ps that the container has terminated.\n\n#### Second Practice\n\n```bash\ndocker run -it ubuntu sleep 10\n```\n\nAfter that, press `ctrl + p` -> `ctrl + q`.\n\nNow you can perform other commands in that terminal, and you can also see that the container is still alive for up to 10 seconds with `docker ps`. This situation, where you exit from the Docker container, is called \"detached\". Docker provides an option to run containers in detached mode, which allows you to run the container in the background while executing the `run` command.\n\n#### Third Practice\n\n```bash\ndocker run -d ubuntu sleep 10\n```\n\nIn detached mode, you can perform other actions in the terminal that executed the command.\n\nIt is good to use detached mode appropriately according to the situation. \nFor example, when developing a backend API server that communicates with the DB, the backend API server needs to be constantly checked with hot-loading while changing the source code, but the DB does not need to be monitored, so it can be executed as follows. \nRun the DB container in detached mode, and run the backend API server in attached mode to follow the logs.\n\n\n## References\n\n- [https://towardsdatascience.com/docker-storage-598e385f4efe](https://towardsdatascience.com/docker-storage-598e385f4efe)\n- [https://vsupalov.com/docker-latest-tag/](https://vsupalov.com/docker-latest-tag/)\n- [https://docs.microsoft.com/ko-kr/azure/container-registry/container-registry-image-tag-version](https://docs.microsoft.com/ko-kr/azure/container-registry/container-registry-image-tag-version)\n- [https://stevelasker.blog/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/](https://stevelasker.blog/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/)\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/prerequisites/docker/command.md",
"content": "---\ntitle : \"[Practice] Docker command\"\ndescription: \"Practice to use docker command.\"\nsidebar_position: 4\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n## 1. Normal installation confirmation\n\n```bash\ndocker run hello-world\n```\n\nIf installed correctly, you should be able to see the following message.\n\n```bash\nHello from Docker!\nThis message shows that your installation appears to be working correctly.\n....\n```\n\n\n**(For ubuntu)** If you want to use without sudo, please refer to the following site.\n\n- [https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user)\n\n## 2. Docker Pull\n\nDocker pull is a command to download Docker images from a Docker image registry (a repository where Docker images are stored and shared).\n\nYou can check the arguments available in docker pull using the command below.\n\n```bash\ndocker pull --help\n```\n\nIf performed normally, it prints out as follows.\n\n```bash\nUsage: docker pull [OPTIONS] NAME[:TAG|@DIGEST]\n\nPull an image or a repository from a registry\n\nOptions:\n -a, --all-tags Download all tagged images in the repository\n --disable-content-trust Skip image verification (default true)\n --platform string Set platform if server is multi-platform capable\n -q, --quiet Suppress verbose output\n```\n\nIt can be seen here that docker pull takes two types of arguments. \n\n1. `[OPTIONS]`\n2. `NAME[:TAG|@DIGEST]`\n\nIn order to use the `-a` and `-q` options from help, they must be used before the NAME. \nLet's try and pull the `ubuntu:18.04` image directly.\n\n```bash\ndocker pull ubuntu:18.04\n```\n\nIf interpreted correctly, the command means to pull an image with the tag `18.04` from an image named `ubuntu`.\n\nIf performed successfully, it will produce an output similar to the following.\n\n```bash\n18.04: Pulling from library/ubuntu\n20d796c36622: Pull complete \nDigest: sha256:42cd9143b6060261187a72716906187294b8b66653b50d70bc7a90ccade5c984\nStatus: Downloaded newer image for ubuntu:18.04\ndocker.io/library/ubuntu:18.04\n```\n\nIf you perform the above command, you will download the image called 'ubuntu:18.04' from a registry named [docker.io/library](http://docker.io/library/) to your laptop.\n\n- Note that \n - in the future, if you need to get a docker image from a certain **private** registry instead of docker.io or public docker hub, you can use [`docker login`](https://docs.docker.com/engine/reference/commandline/login/) to point to the certain registry, then use `docker pull`. Alternatively, you can set up an [insecure registry](https://stackoverflow.com/questions/42211380/add-insecure-registry-to-docker). \n - Also note that [`docker save`](https://docs.docker.com/engine/reference/commandline/save/) and [`docker load`](https://docs.docker.com/engine/reference/commandline/load/) commands are available to store and share docker images in the form of `.tar` file in an intranet.\n\n\n## 3. Docker images\n\nThis is the command to list the Docker images that exist locally.\n\n```bash\ndocker images --help\n```\n\nThe arguments available for use in docker images are as follows.\n\n```bash\nUsage: docker images [OPTIONS] [REPOSITORY[:TAG]]\n\nList images\n\nOptions:\n -a, --all Show all images (default hides intermediate images)\n --digests Show digests\n -f, --filter filter Filter output based on conditions provided\n --format string Pretty-print images using a Go template\n --no-trunc Don't truncate output\n -q, --quiet Only show image IDs\n```\n\nLet's try executing the command below directly.\n\n```bash\ndocker images\n```\n\nIf you install Docker and proceed with this practice, it will output something similar to this.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\nIf you use the `-q` argument among the possible arguments, only the `IMAGE ID` will be printed.\n\n```bash\ndocker images -q\n```\n\n```bash\n29e70752d7b2\n```\n\n## 4. Docker ps\n\nCommand to output the list of currently running Docker containers.\n\n```bash\ndocker ps --help\n```\n\nUse the following arguments can be used with 'docker ps':\n\n```bash\nUsage: docker ps [OPTIONS]\n\nList containers\n\nOptions:\n -a, --all Show all containers (default shows just running)\n -f, --filter filter Filter output based on conditions provided\n --format string Pretty-print containers using a Go template\n -n, --last int Show n last created containers (includes all states) (default -1)\n -l, --latest Show the latest created container (includes all states)\n --no-trunc Don't truncate output\n -q, --quiet Only display container IDs\n -s, --size Display total file sizes\n```\n\nLet's try running the command below directly.\n\n```bash\ndocker ps\n```\n\nIf there are no currently running containers, it will be as follows.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n```\n\nIf there is a container running, it will look similar to this.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nc1e8f5e89d8d ubuntu \"sleep 3600\" 13 seconds ago Up 12 seconds trusting_newton\n```\n\n## 5. Docker run\n\nCommand to run a Docker container.\n\n```bash\ndocker run --help\n```\n\nThe command to run docker run is as follows.\n\n```bash\nUsage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]\n\nRun a command in a new container\n```\n\nWhat we need to confirm here is that the docker run command takes three types of arguments. \n\n1. `[OPTIONS]`\n2. `[COMMAND]`\n3. `[ARG...]`\n\nLet's try running a docker container ourselves.\n\n```bash\n## Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]\ndocker run -it --name demo1 ubuntu:18.04 /bin/bash\n```\n\n- `-it`: Combination of `-i` and `-t` options\n - Runs the container and connects it to an interactive terminal\n- `--name`: Assigns a name to the container for easier identification instead of using the container ID\n- `/bin/bash`: Specifies the command to be executed in the container upon startup, where `/bin/bash` opens a bash shell.\n\nAfter running the command, you can exit the container by using the `exit` command.\n\nWhen you enter the previously learned `docker ps` command, the following output will be displayed.\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n```\n\nIt was said that the container being executed was coming out, but for some reason the container that was just executed does not appear. The reason is that `docker ps` shows the currently running containers by default. If you want to see the stopped containers too, you must give the `-a` option.\n```bash\ndocker ps -a\n```\n\nThen the list of terminated containers will also be displayed.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 2 minutes ago Exited (0) 2 minutes ago demo1\n```\n\n## 6. Docker exec\n\nDocker exec is a command that is used to issue commands or access the inside of a Docker container.\n\n```bash\ndocker exec --help\n```\nFor example, let's try running the following command.\n\n```bash\ndocker run -d --name demo2 ubuntu:18.04 sleep 3600\n```\n\nHere, the `-d` option is a command that allows the Docker container to run in the background so that even if the connection ends to the container, it continues to run.\n\nUse `docker ps` to check if it is currently running.\n\nIt can be confirmed that it is running as follows.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 4 seconds ago Up 3 seconds demo2\n```\n\nNow let's connect to the running docker container through the `docker exec` command.\n\n```bash\ndocker exec -it demo2 /bin/bash\n```\n\nThis is the same as the previous `docker run` command, allowing you to access the inside of the container.\n \nYou can exit using `exit`.\n## 7. Docker logs\n\n```bash\ndocker logs --help\n```\n\nI will have the following container be executed.\n\n```bash\ndocker run --name demo3 -d busybox sh -c \"while true; do $(echo date); sleep 1; done\"\n```\n\nBy using the above command, we have set up a busybox container named \"test\" as a Docker container in the background and printed the current time once every second.\n\nNow let's check the log with the command below.\n\n```bash\ndocker logs demo3\n```\n\nIf performed normally, it will be similar to below.\n\n```bash\nSun Mar 6 11:06:49 UTC 2022\nSun Mar 6 11:06:50 UTC 2022\nSun Mar 6 11:06:51 UTC 2022\nSun Mar 6 11:06:52 UTC 2022\nSun Mar 6 11:06:53 UTC 2022\nSun Mar 6 11:06:54 UTC 2022\n```\nHowever, if used this way, you can only check the logs taken so far. \nIn this case, you can use the `-f` option to keep watching and outputting.\n\n```bash\ndocker logs demo3 -f \n```\n\n## 8. Docker stop\n\nCommand to stop a running Docker container.\n\n```bash\ndocker stop --help\n```\n\nThrough `docker ps`, you can check the containers currently running, as follows.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" About a minute ago Up About a minute demo3\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 4 minutes ago Up 4 minutes demo2\n```\nNow let's try to stop Docker with `docker stop`.\n\n```bash\ndocker stop demo2\n```\n\nAfter executing, type `docker ps` again.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" 2 minutes ago Up 2 minutes demo3\n```\n\nComparing with the above result, you can see that the demo2 container has disappeared from the list of currently running containers.\nThe rest of the containers will also be stopped.\n\n```bash\ndocker stop demo3\n```\n\nDocker rm: Command to delete a Docker container.\n\n```bash\ndocker rm --help\n```\n\nDocker containers are in a stopped state by default. That's why you can see stopped containers using `docker ps -a`.\nBut why do we have to delete the stopped containers? \nEven when stopped, the data used in the Docker remains in the container.\nSo you can restart the container through restarting. But this process will use disk.\nSo\n in order to delete the containers that are not used at all, we should use the `docker rm` command.\n \n First, let's check the current containers.\n\n```bash\ndocker ps -a\n```\n\nThere are three containers as follows.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" 4 minutes ago Exited (137) About a minute ago demo3\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 7 minutes ago Exited (137) 2 minutes ago demo2\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 10 minutes ago Exited (0) 10 minutes ago demo1\n```\n\nLet's try to delete the 'demo3' container through the following command.\n\n```bash\ndocker rm demo3\n```\n\nThe command `docker ps -a` reduced it to two lines as follows.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 13 minutes ago Exited (137) 8 minutes ago demo2\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 16 minutes ago Exited (0) 16 minutes ago demo1\n```\n\nDelete the remaining containers as well.\n\n```bash\ndocker rm demo2\ndocker rm demo1\n```\n\n## 10. Docker rmi\n\nCommand to delete a Docker image.\n\n```bash\ndocker rmi --help\n```\n\nUse the following commands to check which images are currently on the local.\n\n```bash\ndocker images\n```\n\nThe following is output.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nbusybox latest a8440bba1bc0 32 hours ago 1.41MB\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\nI will try to delete the `busybox` image.\n\n```bash\ndocker rmi busybox\n```\n\nIf you type `docker images` again, the following will appear.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\n## References\n\n- [https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry)\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/prerequisites/docker/docker.md",
"content": "---\ntitle : \"What is Docker?\"\ndescription: \"Introduction to Docker.\"\nsidebar_position: 3\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n\n## Container\n\n- Containerization:\n - A technology that allows applications to be executed uniformly anywhere.\n- Container Image:\n - A collection of all the files required to run an application.\n - → Similar to a mold for making fish-shaped bread (Bungeoppang).\n- Container:\n - A single process that is executed based on a container image.\n - → A fish-shaped bread (Bungeoppang) produced using a mold.\n\n## Docker\n\nDocker is a platform that allows you to manage and use containers. \nIts slogan is \"Build Once, Run Anywhere,\" guaranteeing the same execution results anywhere.\n\nIn the Docker, the resources for the container are separated and the lifecycle is controlled by Linux kernel's cgroups, etc. \nHowever, it is too difficult to use these interfaces directly, so an abstraction layer is created.\n\n\n\nThrough this, users can easily control containers with just the user-friendly API **Docker CLI**.\n- Users can easily control containers using the user-friendly API called **Docker CLI**.\n\n## Interpretation of Layer\n\nThe roles of the layers mentioned above are as follows:\n\n1. runC: Utilizes the functionality of the Linux kernel to isolate namespaces, CPUs, memory, filesystems, etc., for a container, which is a single process.\n2. containerd: Acts as an abstraction layer to communicate with runC (OCI layer) and uses the standardized interface (OCI).\n3. dockerd: Solely responsible for issuing commands to containerd.\n4. Docker CLI: Users only need to issue commands to dockerd (Docker daemon) using Docker CLI.\n - During this communication process, Unix socket is used, so sometimes Docker-related errors occur, such as \"the /var/run/docker.sock is in use\" or \"insufficient permissions\" error messages.\n\nAlthough Docker encompasses many stages, when the term \"Docker\" is used, it can refer to Docker CLI, Dockerd (Docker daemon), or even a single Docker container, which can lead to confusion. \nIn the upcoming text, the term \"Docker\" may be used in various contexts.\n\n## For ML Engineer\n\nML engineers use Docker for the following reasons:\n\n1. ML training/inference code needs to be independent of the underlying operating system, Python version, Python environment, and specific versions of Python packages.\n2. Therefore, the goal is to bundle not only the code but also all the dependent packages, environment variables, folder names, etc., into a single package. Containerization technology enables this.\n3. Docker is one of the software tools that makes it easy to use and manage this technology, and the packaged units are referred to as Docker images.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/current/prerequisites/docker/images.md",
"content": "---\ntitle : \"[Practice] Docker images\"\ndescription: \"Practice to use docker image.\"\nsidebar_position: 5\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n- `docker commit`\n - running container 를 docker image 로 만드는 방법\n - `docker commit -m \"message\" -a \"author\" For details for spec:
\n\n- **name**:\n - Specifies a name to identify the notebook server.\n- **namespace**:\n - Cannot be changed. (It is automatically set to the namespace of the currently logged-in user account.)\n- **Image**:\n - Selects the image to use from pre-installed JupyterLab images with Python packages like sklearn, pytorch, tensorflow, etc.\n - If you want to use an image that utilizes GPU within the notebook server, refer to the **GPUs** section below.\n - If you want to use a custom notebook server that includes additional packages or source code, you can create a custom image and deploy it for use.\n- **CPU / RAM**:\n - Specifies the amount of resources required.\n - cpu: in core units\n - Represents the number of virtual cores, and can also be specified as a float value such as `1.5`, `2.7`, etc.\n - memory: in Gi units\n- **GPUs**:\n - Specifies the number of GPUs to allocate to the Jupyter notebook.\n - `None`\n - When GPU resources are not required.\n - 1, 2, 4\n - Allocates 1, 2, or 4 GPUs.\n - GPU Vendor:\n - If you have followed the [(Optional) Setup GPU](../setup-kubernetes/setup-nvidia-gpu.md) guide and installed the NVIDIA GPU plugin, select NVIDIA.\n- **Workspace Volume**:\n - Specifies the amount of disk space required within the notebook server.\n - Do not change the Type and Name fields unless you want to increase the disk space or change the AccessMode.\n - Check the **\"Don't use Persistent Storage for User's home\"** checkbox only if it is not necessary to save the notebook server's work. **It is generally recommended not to check this option.**\n - If you want to use a pre-existing Persistent Volume Claim (PVC), select Type as \"Existing\" and enter the name of the PVC to use.\n- **Data Volumes**:\n - If additional storage resources are required, click the **\"+ ADD VOLUME\"** button to create them.\n- ~~Configurations, Affinity/Tolerations, Miscellaneous Settings~~\n - These are generally not needed, so detailed explanations are omitted in *MLOps for All*.\n\n\n \n \nmlflow_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: mlflow-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.10, pipelines.kubeflow.org/pipeline_compilation_time: '2022-01-19T14:14:11.999807',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"kernel\", \"type\":\n \"String\"}, {\"name\": \"model_name\", \"type\": \"String\"}], \"name\": \"mlflow_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.10}\nspec:\n entrypoint: mlflow-pipeline\n templates:\n - name: load-iris-data\n container:\n args: [--data, /tmp/outputs/data/data, --target, /tmp/outputs/target/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'pandas' 'scikit-learn' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location 'pandas' 'scikit-learn' --user)\n && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n\n def load_iris_data(\n data_path,\n target_path,\n ):\n import pandas as pd\n from sklearn.datasets import load_iris\n\n iris = load_iris()\n\n data = pd.DataFrame(iris[\"data\"], columns=iris[\"feature_names\"])\n target = pd.DataFrame(iris[\"target\"], columns=[\"target\"])\n\n data.to_csv(data_path, index=False)\n target.to_csv(target_path, index=False)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Load iris data', description='')\n _parser.add_argument(\"--data\", dest=\"data_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--target\", dest=\"target_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = load_iris_data(**_parsed_args)\n image: python:3.7\n outputs:\n artifacts:\n - {name: load-iris-data-data, path: /tmp/outputs/data/data}\n - {name: load-iris-data-target, path: /tmp/outputs/target/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--data\", {\"outputPath\": \"data\"}, \"--target\", {\"outputPath\": \"target\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''pandas'' ''scikit-learn'' ||\n PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n ''pandas'' ''scikit-learn'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef load_iris_data(\\n data_path,\\n target_path,\\n):\\n import\n pandas as pd\\n from sklearn.datasets import load_iris\\n\\n iris = load_iris()\\n\\n data\n = pd.DataFrame(iris[\\\"data\\\"], columns=iris[\\\"feature_names\\\"])\\n target\n = pd.DataFrame(iris[\\\"target\\\"], columns=[\\\"target\\\"])\\n\\n data.to_csv(data_path,\n index=False)\\n target.to_csv(target_path, index=False)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Load iris data'', description='''')\\n_parser.add_argument(\\\"--data\\\",\n dest=\\\"data_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--target\\\", dest=\\\"target_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = load_iris_data(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"name\": \"Load iris data\", \"outputs\": [{\"name\":\n \"data\", \"type\": \"csv\"}, {\"name\": \"target\", \"type\": \"csv\"}]}', pipelines.kubeflow.org/component_ref: '{}'}\n - name: mlflow-pipeline\n inputs:\n parameters:\n - {name: kernel}\n - {name: model_name}\n dag:\n tasks:\n - {name: load-iris-data, template: load-iris-data}\n - name: train-from-csv\n template: train-from-csv\n dependencies: [load-iris-data]\n arguments:\n parameters:\n - {name: kernel, value: '{{inputs.parameters.kernel}}'}\n artifacts:\n - {name: load-iris-data-data, from: '{{tasks.load-iris-data.outputs.artifacts.load-iris-data-data}}'}\n - {name: load-iris-data-target, from: '{{tasks.load-iris-data.outputs.artifacts.load-iris-data-target}}'}\n - name: upload-sklearn-model-to-mlflow\n template: upload-sklearn-model-to-mlflow\n dependencies: [train-from-csv]\n arguments:\n parameters:\n - {name: model_name, value: '{{inputs.parameters.model_name}}'}\n artifacts:\n - {name: train-from-csv-conda_env, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-conda_env}}'}\n - {name: train-from-csv-input_example, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-input_example}}'}\n - {name: train-from-csv-model, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-model}}'}\n - {name: train-from-csv-signature, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-signature}}'}\n - name: train-from-csv\n container:\n args: [--train-data, /tmp/inputs/train_data/data, --train-target, /tmp/inputs/train_target/data,\n --kernel, '{{inputs.parameters.kernel}}', --model, /tmp/outputs/model/data,\n --input-example, /tmp/outputs/input_example/data, --signature, /tmp/outputs/signature/data,\n --conda-env, /tmp/outputs/conda_env/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'dill' 'pandas' 'scikit-learn' 'mlflow' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location 'dill' 'pandas' 'scikit-learn'\n 'mlflow' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n\n def train_from_csv(\n train_data_path,\n train_target_path,\n model_path,\n input_example_path,\n signature_path,\n conda_env_path,\n kernel,\n ):\n import dill\n import pandas as pd\n from sklearn.svm import SVC\n\n from mlflow.models.signature import infer_signature\n from mlflow.utils.environment import _mlflow_conda_env\n\n train_data = pd.read_csv(train_data_path)\n train_target = pd.read_csv(train_target_path)\n\n clf = SVC(kernel=kernel)\n clf.fit(train_data, train_target)\n\n with open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n\n input_example = train_data.sample(1)\n with open(input_example_path, \"wb\") as file_writer:\n dill.dump(input_example, file_writer)\n\n signature = infer_signature(train_data, clf.predict(train_data))\n with open(signature_path, \"wb\") as file_writer:\n dill.dump(signature, file_writer)\n\n conda_env = _mlflow_conda_env(\n additional_pip_deps=[\"dill\", \"pandas\", \"scikit-learn\"]\n )\n with open(conda_env_path, \"wb\") as file_writer:\n dill.dump(conda_env, file_writer)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Train from csv', description='')\n _parser.add_argument(\"--train-data\", dest=\"train_data_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--train-target\", dest=\"train_target_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--kernel\", dest=\"kernel\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--model\", dest=\"model_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--input-example\", dest=\"input_example_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--signature\", dest=\"signature_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--conda-env\", dest=\"conda_env_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = train_from_csv(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: kernel}\n artifacts:\n - {name: load-iris-data-data, path: /tmp/inputs/train_data/data}\n - {name: load-iris-data-target, path: /tmp/inputs/train_target/data}\n outputs:\n artifacts:\n - {name: train-from-csv-conda_env, path: /tmp/outputs/conda_env/data}\n - {name: train-from-csv-input_example, path: /tmp/outputs/input_example/data}\n - {name: train-from-csv-model, path: /tmp/outputs/model/data}\n - {name: train-from-csv-signature, path: /tmp/outputs/signature/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--train-data\", {\"inputPath\": \"train_data\"}, \"--train-target\",\n {\"inputPath\": \"train_target\"}, \"--kernel\", {\"inputValue\": \"kernel\"}, \"--model\",\n {\"outputPath\": \"model\"}, \"--input-example\", {\"outputPath\": \"input_example\"},\n \"--signature\", {\"outputPath\": \"signature\"}, \"--conda-env\", {\"outputPath\":\n \"conda_env\"}], \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location ''dill'' ''pandas''\n ''scikit-learn'' ''mlflow'' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m\n pip install --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn''\n ''mlflow'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef train_from_csv(\\n train_data_path,\\n train_target_path,\\n model_path,\\n input_example_path,\\n signature_path,\\n conda_env_path,\\n kernel,\\n):\\n import\n dill\\n import pandas as pd\\n from sklearn.svm import SVC\\n\\n from\n mlflow.models.signature import infer_signature\\n from mlflow.utils.environment\n import _mlflow_conda_env\\n\\n train_data = pd.read_csv(train_data_path)\\n train_target\n = pd.read_csv(train_target_path)\\n\\n clf = SVC(kernel=kernel)\\n clf.fit(train_data,\n train_target)\\n\\n with open(model_path, mode=\\\"wb\\\") as file_writer:\\n dill.dump(clf,\n file_writer)\\n\\n input_example = train_data.sample(1)\\n with open(input_example_path,\n \\\"wb\\\") as file_writer:\\n dill.dump(input_example, file_writer)\\n\\n signature\n = infer_signature(train_data, clf.predict(train_data))\\n with open(signature_path,\n \\\"wb\\\") as file_writer:\\n dill.dump(signature, file_writer)\\n\\n conda_env\n = _mlflow_conda_env(\\n additional_pip_deps=[\\\"dill\\\", \\\"pandas\\\",\n \\\"scikit-learn\\\"]\\n )\\n with open(conda_env_path, \\\"wb\\\") as file_writer:\\n dill.dump(conda_env,\n file_writer)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Train\n from csv'', description='''')\\n_parser.add_argument(\\\"--train-data\\\", dest=\\\"train_data_path\\\",\n type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--train-target\\\",\n dest=\\\"train_target_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--kernel\\\",\n dest=\\\"kernel\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--model\\\",\n dest=\\\"model_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--input-example\\\", dest=\\\"input_example_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--signature\\\",\n dest=\\\"signature_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--conda-env\\\", dest=\\\"conda_env_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = train_from_csv(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"train_data\", \"type\": \"csv\"},\n {\"name\": \"train_target\", \"type\": \"csv\"}, {\"name\": \"kernel\", \"type\": \"String\"}],\n \"name\": \"Train from csv\", \"outputs\": [{\"name\": \"model\", \"type\": \"dill\"},\n {\"name\": \"input_example\", \"type\": \"dill\"}, {\"name\": \"signature\", \"type\":\n \"dill\"}, {\"name\": \"conda_env\", \"type\": \"dill\"}]}', pipelines.kubeflow.org/component_ref: '{}',\n pipelines.kubeflow.org/arguments.parameters: '{\"kernel\": \"{{inputs.parameters.kernel}}\"}'}\n - name: upload-sklearn-model-to-mlflow\n container:\n args: [--model-name, '{{inputs.parameters.model_name}}', --model, /tmp/inputs/model/data,\n --input-example, /tmp/inputs/input_example/data, --signature, /tmp/inputs/signature/data,\n --conda-env, /tmp/inputs/conda_env/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'dill' 'pandas' 'scikit-learn' 'mlflow' 'boto3' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location 'dill' 'pandas' 'scikit-learn'\n 'mlflow' 'boto3' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def upload_sklearn_model_to_mlflow(\n model_name,\n model_path,\n input_example_path,\n signature_path,\n conda_env_path,\n ):\n import os\n import dill\n from mlflow.sklearn import save_model\n\n from mlflow.tracking.client import MlflowClient\n\n os.environ[\"MLFLOW_S3_ENDPOINT_URL\"] = \"http://minio-service.kubeflow.svc:9000\"\n os.environ[\"AWS_ACCESS_KEY_ID\"] = \"minio\"\n os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"minio123\"\n\n client = MlflowClient(\"http://mlflow-server-service.mlflow-system.svc:5000\")\n\n with open(model_path, mode=\"rb\") as file_reader:\n clf = dill.load(file_reader)\n\n with open(input_example_path, \"rb\") as file_reader:\n input_example = dill.load(file_reader)\n\n with open(signature_path, \"rb\") as file_reader:\n signature = dill.load(file_reader)\n\n with open(conda_env_path, \"rb\") as file_reader:\n conda_env = dill.load(file_reader)\n\n save_model(\n sk_model=clf,\n path=model_name,\n serialization_format=\"cloudpickle\",\n conda_env=conda_env,\n signature=signature,\n input_example=input_example,\n )\n run = client.create_run(experiment_id=\"0\")\n client.log_artifact(run.info.run_id, model_name)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Upload sklearn model to mlflow', description='')\n _parser.add_argument(\"--model-name\", dest=\"model_name\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--model\", dest=\"model_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--input-example\", dest=\"input_example_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--signature\", dest=\"signature_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--conda-env\", dest=\"conda_env_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = upload_sklearn_model_to_mlflow(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: model_name}\n artifacts:\n - {name: train-from-csv-conda_env, path: /tmp/inputs/conda_env/data}\n - {name: train-from-csv-input_example, path: /tmp/inputs/input_example/data}\n - {name: train-from-csv-model, path: /tmp/inputs/model/data}\n - {name: train-from-csv-signature, path: /tmp/inputs/signature/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--model-name\", {\"inputValue\": \"model_name\"}, \"--model\", {\"inputPath\":\n \"model\"}, \"--input-example\", {\"inputPath\": \"input_example\"}, \"--signature\",\n {\"inputPath\": \"signature\"}, \"--conda-env\", {\"inputPath\": \"conda_env\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn''\n ''mlflow'' ''boto3'' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install\n --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn'' ''mlflow''\n ''boto3'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def upload_sklearn_model_to_mlflow(\\n model_name,\\n model_path,\\n input_example_path,\\n signature_path,\\n conda_env_path,\\n):\\n import\n os\\n import dill\\n from mlflow.sklearn import save_model\\n\\n from\n mlflow.tracking.client import MlflowClient\\n\\n os.environ[\\\"MLFLOW_S3_ENDPOINT_URL\\\"]\n = \\\"http://minio-service.kubeflow.svc:9000\\\"\\n os.environ[\\\"AWS_ACCESS_KEY_ID\\\"]\n = \\\"minio\\\"\\n os.environ[\\\"AWS_SECRET_ACCESS_KEY\\\"] = \\\"minio123\\\"\\n\\n client\n = MlflowClient(\\\"http://mlflow-server-service.mlflow-system.svc:5000\\\")\\n\\n with\n open(model_path, mode=\\\"rb\\\") as file_reader:\\n clf = dill.load(file_reader)\\n\\n with\n open(input_example_path, \\\"rb\\\") as file_reader:\\n input_example\n = dill.load(file_reader)\\n\\n with open(signature_path, \\\"rb\\\") as file_reader:\\n signature\n = dill.load(file_reader)\\n\\n with open(conda_env_path, \\\"rb\\\") as file_reader:\\n conda_env\n = dill.load(file_reader)\\n\\n save_model(\\n sk_model=clf,\\n path=model_name,\\n serialization_format=\\\"cloudpickle\\\",\\n conda_env=conda_env,\\n signature=signature,\\n input_example=input_example,\\n )\\n run\n = client.create_run(experiment_id=\\\"0\\\")\\n client.log_artifact(run.info.run_id,\n model_name)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Upload\n sklearn model to mlflow'', description='''')\\n_parser.add_argument(\\\"--model-name\\\",\n dest=\\\"model_name\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--model\\\",\n dest=\\\"model_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--input-example\\\",\n dest=\\\"input_example_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--signature\\\",\n dest=\\\"signature_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--conda-env\\\",\n dest=\\\"conda_env_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = upload_sklearn_model_to_mlflow(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"model_name\", \"type\": \"String\"},\n {\"name\": \"model\", \"type\": \"dill\"}, {\"name\": \"input_example\", \"type\": \"dill\"},\n {\"name\": \"signature\", \"type\": \"dill\"}, {\"name\": \"conda_env\", \"type\": \"dill\"}],\n \"name\": \"Upload sklearn model to mlflow\"}', pipelines.kubeflow.org/component_ref: '{}',\n pipelines.kubeflow.org/arguments.parameters: '{\"model_name\": \"{{inputs.parameters.model_name}}\"}'}\n arguments:\n parameters:\n - {name: kernel}\n - {name: model_name}\n serviceAccountName: pipeline-runner\n```\n\n
\n \n \nexample_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: example-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9, pipelines.kubeflow.org/pipeline_compilation_time: '2021-12-09T18:11:43.193190',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"number_1\", \"type\":\n \"Integer\"}, {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"example_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9}\nspec:\n entrypoint: example-pipeline\n templates:\n - name: example-pipeline\n inputs:\n parameters:\n - {name: number_1}\n - {name: number_2}\n dag:\n tasks:\n - name: print-and-return-number\n template: print-and-return-number\n arguments:\n parameters:\n - {name: number_1, value: '{{inputs.parameters.number_1}}'}\n - name: print-and-return-number-2\n template: print-and-return-number-2\n arguments:\n parameters:\n - {name: number_2, value: '{{inputs.parameters.number_2}}'}\n - name: sum-and-print-numbers\n template: sum-and-print-numbers\n dependencies: [print-and-return-number, print-and-return-number-2]\n arguments:\n parameters:\n - {name: print-and-return-number-2-Output, value: '{{tasks.print-and-return-number-2.outputs.parameters.print-and-return-number-2-Output}}'}\n - {name: print-and-return-number-Output, value: '{{tasks.print-and-return-number.outputs.parameters.print-and-return-number-Output}}'}\n - name: print-and-return-number\n container:\n args: [--number, '{{inputs.parameters.number_1}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(\n str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_1}\n outputs:\n parameters:\n - name: print-and-return-number-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-Output, path: /tmp/outputs/Output/data}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is number 1, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\",\n {\"outputPath\": \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(\\n str(int_value),\n str(type(int_value))))\\n return str(int_value)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Print and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_1}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n - name: print-and-return-number-2\n container:\n args: [--number, '{{inputs.parameters.number_2}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(\n str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_2}\n outputs:\n parameters:\n - name: print-and-return-number-2-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-2-Output, path: /tmp/outputs/Output/data}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is number 2, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\",\n {\"outputPath\": \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(\\n str(int_value),\n str(type(int_value))))\\n return str(int_value)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Print and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_2}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n - name: sum-and-print-numbers\n container:\n args: [--number-1, '{{inputs.parameters.print-and-return-number-Output}}', --number-2,\n '{{inputs.parameters.print-and-return-number-2-Output}}']\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def sum_and_print_numbers(number_1, number_2):\n print(number_1 + number_2)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Sum and print numbers', description='')\n _parser.add_argument(\"--number-1\", dest=\"number_1\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--number-2\", dest=\"number_2\", type=int, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = sum_and_print_numbers(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: print-and-return-number-2-Output}\n - {name: print-and-return-number-Output}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is sum of number\n 1 and number 2, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number-1\", {\"inputValue\": \"number_1\"}, \"--number-2\",\n {\"inputValue\": \"number_2\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def sum_and_print_numbers(number_1, number_2):\\n print(number_1 + number_2)\\n\\nimport\n argparse\\n_parser = argparse.ArgumentParser(prog=''Sum and print numbers'',\n description='''')\\n_parser.add_argument(\\\"--number-1\\\", dest=\\\"number_1\\\",\n type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--number-2\\\",\n dest=\\\"number_2\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = sum_and_print_numbers(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number_1\", \"type\": \"Integer\"},\n {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"Sum and print numbers\"}',\n pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number_1\":\n \"{{inputs.parameters.print-and-return-number-Output}}\", \"number_2\": \"{{inputs.parameters.print-and-return-number-2-Output}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n arguments:\n parameters:\n - {name: number_1}\n - {name: number_2}\n serviceAccountName: pipeline-runner\n```\n\n
\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n```\n\nIf written in pipeline, it will be like this.\n\n```python\nfrom functools import partial\n\nimport kfp\nfrom kfp.components import create_component_from_func, OutputPath\nfrom kfp.dsl import pipeline\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"matplotlib\"],\n)\ndef plot_linear(mlpipeline_ui_metadata: OutputPath(\"UI_Metadata\")):\n import base64\n import json\n from io import BytesIO\n\n import matplotlib.pyplot as plt\n\n plt.plot([1, 2, 3], [1, 2, 3])\n\n tmpfile = BytesIO()\n plt.savefig(tmpfile, format=\"png\")\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\"utf-8\")\n\n html = f\"\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n\n\n@pipeline(name=\"plot_pipeline\")\ndef plot_pipeline():\n plot_linear()\n\n\nif __name__ == \"__main__\":\n kfp.compiler.Compiler().compile(plot_pipeline, \"plot_pipeline.yaml\")\n```\n\nIf you run this script and check the resulting `plot_pipeline.yaml`, you will see the following.\n\n\n \n \nplot_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: plot-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9, pipelines.kubeflow.org/pipeline_compilation_time: '2\n022-01-17T13:31:32.963214',\n pipelines.kubeflow.org/pipeline_spec: '{\"name\": \"plot_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9}\nspec:\n entrypoint: plot-pipeline\n templates:\n - name: plot-linear\n container:\n args: [--mlpipeline-ui-metadata, /tmp/outputs/mlpipeline_ui_metadata/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'matplotlib' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet\n --no-warn-script-location 'matplotlib' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n def plot_linear(mlpipeline_ui_metadata):\n import base64\n import json\n from io import BytesIO\n import matplotlib.pyplot as plt\n plt.plot([1, 2, 3], [1, 2, 3])\n tmpfile = BytesIO()\n plt.savefig(tmpfile, format=\"png\")\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\"utf-8\")\n html = f\"\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Plot linear', description='')\n _parser.add_argument(\"--mlpipeline-ui-metadata\", dest=\"mlpipeline_ui_metadata\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n _outputs = plot_linear(**_parsed_args)\n image: python:3.7\n outputs:\n artifacts:\n - {name: mlpipeline-ui-metadata, path: /tmp/outputs/mlpipeline_ui_metadata/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--mlpipeline-ui-metadata\", {\"outputPath\": \"mlpipeline_ui_metadata\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''matplotlib'' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location ''matplotlib''\n --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef plot_linear(mlpipeline_ui_metadata):\\n import\n base64\\n import json\\n from io import BytesIO\\n\\n import matplotlib.pyplot\n as plt\\n\\n plt.plot([1, 2, 3], [1, 2, 3])\\n\\n tmpfile = BytesIO()\\n plt.savefig(tmpfile,\n format=\\\"png\\\")\\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\\\"utf-8\\\")\\n\\n html\n = f\\\"\\\"\\n metadata = {\\n \\\"outputs\\\":\n [\\n {\\n \\\"type\\\": \\\"web-app\\\",\\n \\\"storage\\\":\n \\\"inline\\\",\\n \\\"source\\\": html,\\n },\\n ],\\n }\\n with\n open(mlpipeline_ui_metadata, \\\"w\\\") as html_writer:\\n json.dump(metadata,\n html_writer)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Plot\n linear'', description='''')\\n_parser.add_argument(\\\"--mlpipeline-ui-metadata\\\",\n dest=\\\"mlpipeline_ui_metadata\\\", type=_make_parent_dirs_and_return_path,\n required=True, default=argparse.SUPPRESS)\\n_parsed_args = vars(_parser.parse_args())\\n\\n_outputs\n = plot_linear(**_parsed_args)\\n\"], \"image\": \"python:3.7\"}}, \"name\": \"Plot\n linear\", \"outputs\": [{\"name\": \"mlpipeline_ui_metadata\", \"type\": \"UI_Metadata\"}]}',\n pipelines.kubeflow.org/component_ref: '{}'}\n - name: plot-pipeline\n dag:\n tasks:\n - {name: plot-linear, template: plot-linear}\n arguments:\n parameters: []\n serviceAccountName: pipeline-runner\n```\n\n
\n
\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow/basic-requirements.md",
"content": "---\ntitle : \"3. Install Requirements\"\ndescription: \"\"\nsidebar_position: 3\ncontributors: [\"Jongseob Jeon\"]\n---\n\nThe recommended Python version for practice is python>=3.7. For those unfamiliar with the Python environment, please refer to [Appendix 1. Python Virtual Environment](../appendix/pyenv) and install the packages on the **client node**.\n\nThe packages and versions required for the practice are as follows:\n\n- requirements.txt\n\n ```bash\n kfp==1.8.9\n scikit-learn==1.0.1\n mlflow==1.21.0\n pandas==1.3.4\n dill==0.3.4\n ```\n\nActivate the [Python virtual environment](../appendix/pyenv.md#python-가상환경-생성) created in the previous section.\n\n```bash\npyenv activate demo\n```\n\nWe are proceeding with the package installation.\n\n```bash\npip3 install -U pip\npip3 install kfp==1.8.9 scikit-learn==1.0.1 mlflow==1.21.0 pandas==1.3.4 dill==0.3.4\n```\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow/basic-run.md",
"content": "---\ntitle : \"7. Pipeline - Run\"\ndescription: \"\"\nsidebar_position: 7\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Run Pipeline\n\nNow we will run the uploaded pipeline.\n\n## Before Run\n\n### 1. Create Experiment\n\nExperiments in Kubeflow are units that logically manage runs executed within them.\n\nWhen you first enter the namespace in Kubeflow, there are no Experiments created. Therefore, you must create an Experiment beforehand in order to run the pipeline. If an Experiment already exists, you can go to [Run Pipeline](../kubeflow/basic-run.md#run-pipeline-1).\n\nExperiments can be created via the Create Experiment button.\n\n\n\n### 2. Name 입력\n\n\n\n## Run Pipeline\n\n### 1. Select Create Run\n\n\n\n### 2. Select Experiment\n\n\n\n\n\n### 3. Enter Pipeline Config\n\nFill in the values of the Config provided when creating the pipeline. The uploaded pipeline requires input values for `number_1` and `number_2`.\n\n\n\n### 4. Start\n\nClick the Start button after entering the values. The pipeline will start running.\n\n\n\n## Run Result\n\nThe executed pipelines can be viewed in the Runs tab.\nClicking on a run provides detailed information related to the executed pipeline.\n\n\n\nUpon clicking, the following screen appears. Components that have not yet executed are displayed in gray.\n\n\n\nWhen a component has completed execution, it is marked with a green checkmark.\n\n\n\nIf we look at the last component, we can see that it has outputted the sum of the input values, which in this case is 8 (the sum of 3 and 5).\n\n\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow/how-to-debug.md",
"content": "---\ntitle : \"13. Component - Debugging\"\ndescription: \"\"\nsidebar_position: 13\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Debugging Pipeline\n\nThis page covers how to debug Kubeflow components.\n\n## Failed Component\n\nWe will modify a pipeline used in [Component - MLFlow](../kubeflow/advanced-mlflow.md#mlflow-pipeline) in this page.\n\nFirst, let's modify the pipeline so that the component fails.\n\n```python\nfrom functools import partial\n\nimport kfp\nfrom kfp.components import InputPath, OutputPath, create_component_from_func\nfrom kfp.dsl import pipeline\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\", \"scikit-learn\"],\n)\ndef load_iris_data(\n data_path: OutputPath(\"csv\"),\n target_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n from sklearn.datasets import load_iris\n\n iris = load_iris()\n\n data = pd.DataFrame(iris[\"data\"], columns=iris[\"feature_names\"])\n target = pd.DataFrame(iris[\"target\"], columns=[\"target\"])\n \n data[\"sepal length (cm)\"] = None\n data.to_csv(data_path, index=False)\n target.to_csv(target_path, index=False)\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\"],\n)\ndef drop_na_from_csv(\n data_path: InputPath(\"csv\"),\n output_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n\n data = pd.read_csv(data_path)\n data = data.dropna()\n data.to_csv(output_path, index=False)\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"dill\", \"pandas\", \"scikit-learn\", \"mlflow\"],\n)\ndef train_from_csv(\n train_data_path: InputPath(\"csv\"),\n train_target_path: InputPath(\"csv\"),\n model_path: OutputPath(\"dill\"),\n input_example_path: OutputPath(\"dill\"),\n signature_path: OutputPath(\"dill\"),\n conda_env_path: OutputPath(\"dill\"),\n kernel: str,\n):\n import dill\n import pandas as pd\n from sklearn.svm import SVC\n\n from mlflow.models.signature import infer_signature\n from mlflow.utils.environment import _mlflow_conda_env\n\n train_data = pd.read_csv(train_data_path)\n train_target = pd.read_csv(train_target_path)\n\n clf = SVC(kernel=kernel)\n clf.fit(train_data, train_target)\n\n with open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n\n input_example = train_data.sample(1)\n with open(input_example_path, \"wb\") as file_writer:\n dill.dump(input_example, file_writer)\n\n signature = infer_signature(train_data, clf.predict(train_data))\n with open(signature_path, \"wb\") as file_writer:\n dill.dump(signature, file_writer)\n\n conda_env = _mlflow_conda_env(\n additional_pip_deps=[\"dill\", \"pandas\", \"scikit-learn\"]\n )\n with open(conda_env_path, \"wb\") as file_writer:\n dill.dump(conda_env, file_writer)\n\n\n\n@pipeline(name=\"debugging_pipeline\")\ndef debugging_pipeline(kernel: str):\n iris_data = load_iris_data()\n drop_data = drop_na_from_csv(data=iris_data.outputs[\"data\"])\n model = train_from_csv(\n train_data=drop_data.outputs[\"output\"],\n train_target=iris_data.outputs[\"target\"],\n kernel=kernel,\n )\n\nif __name__ == \"__main__\":\n kfp.compiler.Compiler().compile(debugging_pipeline, \"debugging_pipeline.yaml\")\n\n```\n\nThe modifications are as follows:\n\n1. In the `load_iris_data` component for loading data, `None` was injected into the `sepal length (cm)` feature.\n2. In the `drop_na_from_csv` component, use the `drop_na()` function to remove rows with na values.\n\nNow let's upload and run the pipeline. \nAfter running, if you press Run you will see that it has failed in the `Train from csv` component.\n\n\n\nClick on the failed component and check the log to see the reason for the failure.\n\n\n\nIf the log shows that the data count is 0 and the component did not run, there may be an issue with the input data. \nLet's investigate what might be the problem.\n\nFirst, click on the component and go to the Input/Output tab to download the input data. \nYou can click on the link indicated by the red square to download the data.\n\n\n\n\nDownload both files to the same location. Then navigate to the specified path and check the downloaded files.\n\n\n```bash\nls\n```\n\nThere are two files as follows.\n\n```bash\ndrop-na-from-csv-output.tgz load-iris-data-target.tgz\n```\n\nI will try to unzip it.\n\n```bash\ntar -xzvf load-iris-data-target.tgz ; mv data target.csv\ntar -xzvf drop-na-from-csv-output.tgz ; mv data data.csv\n```\n\nAnd then run the component code using a Jupyter notebook.\n\n\nDebugging revealed that dropping the data was based on rows instead of columns, resulting in all the data being removed.\nNow that we know the cause of the problem, we can modify the component to drop based on columns.\n\n```python\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\"],\n)\ndef drop_na_from_csv(\n data_path: InputPath(\"csv\"),\n output_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n\n data = pd.read_csv(data_path)\n data = data.dropna(axis=\"columns\")\n data.to_csv(output_path, index=False)\n```\n\nAfter modifying, upload the pipeline again and run it to confirm that it is running normally as follows.\n\n\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow/kubeflow-concepts.md",
"content": "---\ntitle : \"2. Kubeflow Concepts\"\ndescription: \"\"\nsidebar_position: 2\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Component\n\nA component is composed of Component contents and a Component wrapper.\nA single component is delivered to Kubeflow through a Component wrapper and the delivered component executes the defined Component contents and produces artifacts.\n\n\n\n### Component Contents\n\nThere are three components that make up the component contents:\n\n\n\n1. Environment\n2. Python code w/ Config\n3. Generates Artifacts\n\nLet's explore each component with an example.\nHere is a Python code that loads data, trains an SVC (Support Vector Classifier) model, and saves the SVC model.\n\n```python\nimport dill\nimport pandas as pd\n\nfrom sklearn.svm import SVC\n\ntrain_data = pd.read_csv(train_data_path)\ntrain_target= pd.read_csv(train_target_path)\n\nclf= SVC(\n kernel=kernel\n)\nclf.fit(train_data)\n\nwith open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n```\n\nThe above Python code can be divided into components contents as follows.\n\n\n\nEnvironment is the part of the Python code where the packages used in the code are imported. \nNext, Python Code w\\ Config is where the given Config is used to actually perform the training. \nFinally, there is a process to save the artifacts. \n\n### Component Wrapper\n\nComponent wrappers deliver the necessary Config and execute tasks for component content.\n\n\n\nIn Kubeflow, component wrappers are defined as functions, similar to the `train_svc_from_csv` example above.\nWhen a component wrapper wraps the contents, it looks like the following:\n\n\n\n### Artifacts\n\nIn the explanation above, it was mentioned that the component creates Artifacts. Artifacts is a term used to refer to any form of a file that is generated, such as evaluation results, logs, etc.\nOf the ones that we are interested in, the following are significant: Models, Data, Metrics, and etc.\n\n\n\n- Model\n- Data\n- Metric\n- etc\n\n#### Model\n\nWe defined the model as follows: \n\n> A model is a form that includes Python code, trained weights and network architecture, and an environment to run it.\n\n#### Data\n\nData includes preprocessed features, model predictions, etc. \n\n#### Metric\n\nMetric is divided into two categories: dynamic metrics and static metrics.\n\n- Dynamic metrics refer to values that continuously change during the training process, such as train loss per epoch.\n- Static metrics refer to evaluation metrics, such as accuracy, that are calculated after the training is completed.\n\n## Pipeline\n\nA pipeline consists of a collection of components and the order in which they are executed. The order forms a directed acyclic graph (DAG), which can include simple conditional statements.\n\n\n\n### Pipeline Config\n\nAs mentioned earlier, components require config to be executed. The pipeline config contains the configs for all the components in the pipeline.\n\n\n\n## Run\n\nTo execute a pipeline, the pipeline config specific to that pipeline is required. In Kubeflow, an executed pipeline is called a \"Run.\"\n\n\n\nWhen a pipeline is executed, each component generates artifacts. Kubeflow pipeline assigns a unique ID to each Run, and all artifacts generated during the Run are stored.\n\n\n\nNow, let's learn how to write components and pipelines.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow/kubeflow-intro.md",
"content": "---\ntitle : \"1. Kubeflow Introduction\"\ndescription: \"\"\nsidebar_position: 1\ncontributors: [\"Jongseob Jeon\"]\n---\n\nTo use Kubeflow, you need to write components and pipelines.\n\nThe approach described in *MLOps for ALL* differs slightly from the method described on the [Kubeflow Pipeline official website](https://www.kubeflow.org/docs/components/pipelines/overview/quickstart/). Here, Kubeflow Pipeline is used as one of the components in the [elements that make up MLOps](../kubeflow/kubeflow-concepts.md#component-contents) rather than a standalone workflow.\n\nNow, let's understand what components and pipelines are and how to write them.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow-dashboard-guide/_category_.json",
"content": "{\n \"label\": \"Kubeflow UI Guide\",\n \"position\": 5,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow-dashboard-guide/experiments-and-others.md",
"content": "---\ntitle : \"6. Kubeflow Pipeline Relates\"\ndescription: \"\"\nsidebar_position: 6\ncontributors: [\"Jaeyeon Kim\"]\n---\n\nIn the left tabs of the Central Dashboard (KFP Experiments, Pipelines, Runs, Recurring Runs, Artifacts, Executions) you can manage Kubeflow Pipelines and the results of Pipeline execution and Pipeline Runs.\n\n\n\nKubeflow Pipelines are the main reason for using Kubeflow in *MLOps for ALL*, and details on how to create, execute, and check the results of Kubeflow Pipelines can be found in [3.Kubeflow](../kubeflow/kubeflow-intro).\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow-dashboard-guide/experiments.md",
"content": "---\ntitle : \"5. Experiments(AutoML)\"\ndescription: \"\"\nsidebar_position: 5\ncontributors: [\"Jaeyeon Kim\"]\n---\n\nNext, we will click the Experiments(AutoML) tab on the left of the Central Dashboard.\n\n\n\n\n\nThe Experiments(AutoML) page is where you can manage [Katib](https://www.kubeflow.org/docs/components/katib/overview/), which is responsible for AutoML through Hyperparameter Tuning and Neural Architecture Search in Kubeflow.\n\nThe usage of Katib and Experiments(AutoML) is not covered in *MLOps for Everyone* v1.0, and will be added in v2.0.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow-dashboard-guide/intro.md",
"content": "---\ntitle : \"1. Central Dashboard\"\ndescription: \"\"\nsidebar_position: 1\ncontributors: [\"Jaeyeon Kim\", \"SeungTae Kim\"]\n---\n\nOnce you have completed [Kubeflow installation](../setup-components/install-components-kf.md), you can access the dashboard through the following command.\n\n```bash\nkubectl port-forward --address 0.0.0.0 svc/istio-ingressgateway -n istio-system 8080:80\n```\n\n\n\nThe Central Dashboard is a UI that integrates all the features provided by Kubeflow. The features provided by the Central Dashboard can be divided based on the tabs on the left side\n\n\n\n- Home\n- Notebooks\n- Tensorboards\n- Volumes\n- Models\n- Experiments(AutoML)\n- Experiments(KFP)\n- Pipelines\n- Runs\n- Recurring Runs\n- Artifacts\n- Executions\n\nLet's now look at the simple usage of each feature.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow-dashboard-guide/notebooks.md",
"content": "---\ntitle : \"2. Notebooks\"\ndescription: \"\"\nsidebar_position: 2\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n## Launch Notebook Server\n\nClick on the Notebooks tab on the left side of the Central Dashboard.\n\n\n\nYou will see a similar screen.\n\nThe Notebooks tab is a page where users can independently create and access jupyter notebook and code server environments (hereinafter referred to as a notebook server).\n\n\n\nClick the \"+ NEW NOTEBOOK\" button at the top right. \n\n\n\nWhen the screen shown below appears, now specify the spec (Spec) of the notebook server to be created.\n\n\n\n\nexample_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: example-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline_compilation_time: '2021-12-05T13:38:51.566777',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"number_1\", \"type\":\n \"Integer\"}, {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"example_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3}\nspec:\n entrypoint: example-pipeline\n templates:\n - name: example-pipeline\n inputs:\n parameters:\n - {name: number_1}\n - {name: number_2}\n dag:\n tasks:\n - name: print-and-return-number\n template: print-and-return-number\n arguments:\n parameters:\n - {name: number_1, value: '{{inputs.parameters.number_1}}'}\n - name: print-and-return-number-2\n template: print-and-return-number-2\n arguments:\n parameters:\n - {name: number_2, value: '{{inputs.parameters.number_2}}'}\n - name: sum-and-print-numbers\n template: sum-and-print-numbers\n dependencies: [print-and-return-number, print-and-return-number-2]\n arguments:\n parameters:\n - {name: print-and-return-number-2-Output, value: '{{tasks.print-and-return-number-2.outputs.parameters.print-and-return-number-2-Output}}'}\n - {name: print-and-return-number-Output, value: '{{tasks.print-and-return-number.outputs.parameters.print-and-return-number-Output}}'}\n - name: print-and-return-number\n container:\n args: [--number, '{{inputs.parameters.number_1}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_1}\n outputs:\n parameters:\n - name: print-and-return-number-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-Output, path: /tmp/outputs/Output/data}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\", {\"outputPath\":\n \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(str(int_value), str(type(int_value))))\\n return\n str(int_value)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Print\n and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_1}}\"}'}\n - name: print-and-return-number-2\n container:\n args: [--number, '{{inputs.parameters.number_2}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_2}\n outputs:\n parameters:\n - name: print-and-return-number-2-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-2-Output, path: /tmp/outputs/Output/data}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\", {\"outputPath\":\n \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(str(int_value), str(type(int_value))))\\n return\n str(int_value)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Print\n and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_2}}\"}'}\n - name: sum-and-print-numbers\n container:\n args: [--number-1, '{{inputs.parameters.print-and-return-number-Output}}', --number-2,\n '{{inputs.parameters.print-and-return-number-2-Output}}']\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def sum_and_print_numbers(number_1, number_2):\n print(number_1 + number_2)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Sum and print numbers', description='')\n _parser.add_argument(\"--number-1\", dest=\"number_1\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--number-2\", dest=\"number_2\", type=int, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = sum_and_print_numbers(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: print-and-return-number-2-Output}\n - {name: print-and-return-number-Output}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number-1\", {\"inputValue\": \"number_1\"}, \"--number-2\", {\"inputValue\":\n \"number_2\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n sum_and_print_numbers(number_1, number_2):\\n print(number_1 + number_2)\\n\\nimport\n argparse\\n_parser = argparse.ArgumentParser(prog=''Sum and print numbers'',\n description='''')\\n_parser.add_argument(\\\"--number-1\\\", dest=\\\"number_1\\\",\n type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--number-2\\\",\n dest=\\\"number_2\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = sum_and_print_numbers(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number_1\", \"type\": \"Integer\"},\n {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"Sum and print numbers\"}',\n pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number_1\":\n \"{{inputs.parameters.print-and-return-number-Output}}\", \"number_2\": \"{{inputs.parameters.print-and-return-number-2-Output}}\"}'}\n arguments:\n parameters:\n - {name: number_1}\n - {name: number_2}\n serviceAccountName: pipeline-runner\n```\n\n\n
\n\nIf you followed the [Setup GPU (Optional)](../setup-kubernetes/setup-nvidia-gpu.md), select NVIDIA if you have installed the nvidia gpu plugin.\n\n\n\nAfter creation, the **Status** will change to a green check mark icon, and the **CONNECT button** will be activated.\n\n\n---\n## Accessing the Notebook Server\n\nClicking the **CONNECT button** will open a new browser window, where you will see the following screen:\n\n\n\nYou can use the Notebook, Console, and Terminal icons in the **Launcher** to start using them.\n\n Notebook Interface\n\n\n\n Terminal Interface\n\n\n\n---\n\n## Stopping the Notebook Server\n\nIf you haven't used the notebook server for an extended period of time, you can stop it to optimize resource usage in the Kubernetes cluster. **Note that stopping the notebook server will result in the deletion of all data stored outside the Workspace Volume or Data Volume specified when creating the notebook server.** \nIf you haven't changed the path during notebook server creation, the default Workspace Volume path is `/home/jovyan` inside the notebook server, so any data stored outside the `/home/jovyan` directory will be deleted.\n\nClicking the `STOP` button as shown below will stop the notebook server:\n\n\n\nOnce the server is stopped, the `CONNECT` button will be disabled. To restart the notebook server and use it again, click the `PLAY` button.\n\n\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow-dashboard-guide/tensorboards.md",
"content": "---\ntitle : \"3. Tensorboards\"\ndescription: \"\"\nsidebar_position: 3\ncontributors: [\"Jaeyeon Kim\"]\n---\n\nLet's click on the Tensorboards tab of the left tabs of the Central Dashboard next.\n\n\n\nWe can see the following screen. \n\n\n\nThe TensorBoard server created in this way can be used just like a regular remote TensorBoard server, or it can be used for the purpose of storing data directly from a Kubeflow Pipeline run for visualization purposes.\n\nYou can refer to the [TensorBoard documentation](https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#tensorboard) for more information on using TensorBoard with Kubeflow Pipeline runs.\n\nThere are various ways to visualize the results of Kubeflow Pipeline runs, and in *MLOps for ALL*, we will utilize the Visualization feature of Kubeflow components and the visualization capabilities of MLflow to enable more general use cases. Therefore, detailed explanations of the TensorBoards page will be omitted in this context.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/kubeflow-dashboard-guide/volumes.md",
"content": "---\ntitle : \"4. Volumes\"\ndescription: \"\"\nsidebar_position: 4\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n## Volumes\n\nNext, let's click on the Volumes tab in the left of the Central Dashboard.\n\n\n\nYou will see the following screen.\n\n\n\n\nVolumes tab provides the functionality to manage the Persistent Volume Claims (PVC) belonging to the current user's namespace in Kubernetes' Volume (Volume).\n\nBy looking at the screenshot, you can see the information of the Volume created on the [1. Notebooks](../kubeflow-dashboard-guide/notebooks) page. It can be seen that the Storage Class of the Volume is set to local-path, which is the Default Storage Class installed at the time of Kubernetes cluster installation.\n\nIn addition, the Volumes page can be used if you want to create, view, or delete a new Volume in the user namespace.\n\n---\n\n## Creating a Volume\n\nBy clicking the `+ NEW VOLUME` button at the top right, you can see the following screen.\n\n\n\n\nYou can create a volume by specifying its name, size, storage class, and access mode.\n\nWhen you specify the desired resource specs to create a volume, its Status will be shown as Pending on this page. When you hover over the Status icon, you will see a message that this *(This volume will be bound when its first consumer is created.)* \nThis is according to the volume creation policy of the [StorageClass](https://kubernetes.io/ko/docs/concepts/storage/storage-classes/) used in the lab, which is local-path. **This is not a problem situation.** \nWhen the Status is shown as Pending on this page, you can still specify the name of the volume in the notebook server or pod that you want to use the volume and the volume creation will be triggered at that time.\n\n\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/prerequisites/_category_.json",
"content": "{\n \"label\": \"Prerequisites\",\n \"position\": 1,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/prerequisites/docker/_category_.json",
"content": "{\n \"label\": \"Docker\",\n \"position\": 1,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/prerequisites/docker/advanced.md",
"content": "---\ntitle : \"[Practice] Docker Advanced\"\ndescription: \"Practice to use docker more advanced way.\"\nsidebar_position: 6\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n## Making a good Docker image\n\n### Considerations to make Docker image:\n\nWhen creating a Docker image using a Dockerfile, the **order** of the commands is important. \nThis is because Docker images are composed of many Read-Only layers and when building the image, existing layers are **cached** and reused, so if you structure your Dockerfile with this in mind, you can **reduce the build time**.\n\nEach of the `RUN`, `ADD`, `COPY` commands in a Dockerfile are stored as one layer.\n\nFor example, if we have the following `Dockerfile`:\n\n```docker\n# Layer 1\nFROM ubuntu:latest\n\n# Layer 2\nRUN apt-get update && apt-get install python3 pip3 -y\n\n# Layer 3\nRUN pip3 install -U pip && pip3 install torch\n\n# Layer 4\nCOPY src/ src/\n\n# Layer 5\nCMD python src/app.py\n```\n\nIf you run the image built with the above `Dockerfile` with the command `docker run -it app:latest /bin/bash`, it can be represented in the following layers. \n\n\n\nThe topmost R/W layer does not affect the image. In other words, any changes made inside the container are volatile.\n\nWhen a lower layer is changed, all the layers above it need to be rebuilt. Therefore, the order of Dockerfile instructions is important. It is recommended to place the parts that are frequently changed towards the end. (e.g., `COPY src/ app/src/`)\n\nConversely, parts that are unlikely to change should be placed towards the beginning.\n\nIf there are parts that are rarely changed but used in multiple places, they can be consolidated. It is advisable to create a separate image for those common parts in advance and use it as a base image.\n\nFor example, if you want to create separate images for an environment that uses `tensorflow-cpu` and another environment that uses `tensorflow-gpu`, you can do the following:\nCreate a base image [`ghcr.io/makinarocks/python:3.8-base`](http://ghcr.io/makinarocks/python:3.8-base-cpu) that includes Python and other basic packages installed. Then, when creating the images with the CPU and GPU versions of TensorFlow, you can use the base image as the `FROM` instruction and write the separate instructions for installing TensorFlow in each Dockerfile. Managing two Dockerfiles in this way improves readability and reduces build time.\n\nCombining layers had performance benefits in older versions of Docker. However, since you cannot guarantee the Docker version in which your Docker containers will run, it is recommended to combine layers for readability purposes. It is best to combine layers that can be combined appropriately.\n\nHere is an example of a Dockerfile:\n\n```docker\n# Bad Case\nRUN apt-get update\nRUN apt-get install build-essential -y\nRUN apt-get install curl -y\nRUN apt-get install jq -y\nRUN apt-get install git -y\n```\n\nThis can be written by combining it as follows.\n\n```docker\n# Better Case\nRUN apt-get update && \\\n apt-get install -y \\\n build-essential \\\n curl \\\n jq \\\n git\n```\n\nFor convenience, it is better to use `.dockerignore`. \n`.dockerignore` is similar to `.gitignore` in the sense that it can be excluded when doing a `docker build` just like when doing a `git add`. \n\nMore information can be found in the [Docker Official Documentation](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/).\n\n### ENTRYPOINT vs CMD\n\n`ENTRYPOINT` and `CMD` are both used when you want to execute a command at the runtime of the container. One of them must be present in the Dockerfile.\n\n- **Difference**\n - `CMD`: Easily modifiable when running `docker run` command\n - `ENTRYPOINT`: Requires the use of `--entrypoint` to modify\n\nWhen `ENTRYPOINT` and `CMD` are used together, `CMD` typically represents the arguments (parameters) for the command specified in `ENTRYPOINT`.\n\nFor example, consider the following Dockerfile:\n\n```docker\nFROM ubuntu:latest\n\n# 아래 4 가지 option 을 바꿔가며 직접 테스트해보시면 이해하기 편합니다.\n# 단, NO ENTRYPOINT 옵션은 base image 인 ubuntu:latest 에 이미 있어서 테스트해볼 수는 없고 나머지 v2, 3, 5, 6, 8, 9, 11, 12 를 테스트해볼 수 있습니다.\n# ENTRYPOINT echo \"Hello ENTRYPOINT\"\n# ENTRYPOINT [\"echo\", \"Hello ENTRYPOINT\"]\n# CMD echo \"Hello CMD\"\n# CMD [\"echo\", \"Hello CMD\"]\n```\n\n\nIf you build and run the above `Dockerfile` with the parts marked as comments deactivated, you can get the following results: \n\n| | No ENTRYPOINT | ENTRYPOINT a b | ENTRYPOINT [\"a\", \"b\"] |\n| ------------------ | -------------- | -------------- | --------------------- |\n| **NO CMD** | Error! | /bin/sh -c a b | a b |\n| **CMD [\"x\", \"y\"]** | x y | /bin/sh -c a b | a b x y |\n| **CMD x y** | /bin/sh -c x y | /bin/sh -c a b | a b /bin/sh -c x y |\n\n- In Kubernetes pod, \n - `ENTRYPOINT` corresponds to the command\n - `CMD` corresponds to the arguments\n\n### Naming docker tag\n\nRecommend not using \"latest\" as a tag for a Docker image, as it is the default tag name and can be easily overwritten unintentionally.\n\nIt is important to ensure uniqueness of one image with one tag for the sake of collaboration and debugging in the production stage. \nUsing the same tag for different contents can lead to dangling images, which are not shown in the `docker images` but still take up storage space.\n\n### ETC\n\n1. Logs and other information are stored separately from the container, not inside it.\n This is because data written from within the container can be lost at any time.\n2. Secrets and environment-dependent information should not be written directly into the Dockerfile but should be passed in via environment variables or a .env config file.\n3. There is a **linter** for Dockerfiles, so it is useful to use it when collaborating.\n [https://github.com/hadolint/hadolint](https://github.com/hadolint/hadolint)\n\n## Several options for docker run\n\nWhen using Docker containers, there are some inconveniences.\nSpecifically, Docker does not store any of the work done within the Docker container by default.\nThis is because Docker containers use isolated file systems. Therefore, it is difficult to share data between multiple Docker containers.\n\nTo solve this problem, there are two approaches offered by Docker.\n\n\n\n#### Docker volume\n\n- Use the Docker CLI to directly manage a resource called `volume`.\n- Create a specific directory under the Docker area (`/var/lib/docker`) on the host and mount that path to a Docker container.\n\n#### Bind mount\n\n- Mount a specific path on the host to a Docker container.\n\n#### How to use?\n\nThe usage is through the same interface, using the `-v` option. \nHowever, when using volumes, you need to manage them directly by performing commands like `docker volume create`, `docker volume ls`, `docker volume rm`, etc.\n\n- Docker volume\n\n ```bash\n docker run \\\n -v my_volume:/app \\\n nginx:latest\n ````\n\n- Blind mount\n\n ```bash\n docker run \\\n -v /home/user/some/path:/app \\\n nginx:latest\n ```\n\nWhen developing locally, bind mount can be convenient, but if you want to maintain a clean environment, using Docker volume and explicitly performing create and rm operations can be another approach.\n\nThe way storage is provided in Kubernetes ultimately relies on Docker's bind mount as well.\n\n### Docker run with resource limit\n\nBasically, docker containers can **fully utilize the CPU and memory resources of the host OS**. However, when using this, depending on the resource situation of the host OS, docker containers may abnormally terminate due to issues such as **OOM**.\nTo address this problem, docker provides the `-m` [option](https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory) which allows you to **limit the usage of CPU and memory** when running the docker container.\n\n```bash\ndocker run -d -m 512m --memory-reservation=256m --name 512-limit ubuntu sleep 3600\ndocker run -d -m 1g --memory-reservation=256m --name 1g-limit ubuntu sleep 3600\n```\n\nAfter running the Docker above, you can check the usage through the 'docker stats' command.\n\n```bash\nCONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS\n4ea1258e2e09 1g-limit 0.00% 300KiB / 1GiB 0.03% 1kB / 0B 0B / 0B 1\n4edf94b9a3e5 512-limit 0.00% 296KiB / 512MiB 0.06% 1.11kB / 0B 0B / 0B 1\n```\n\nIn Kubernetes, when you limit the CPU and memory resources of a pod resource, it is provided using this technique.\n\n### docker run with restart policy\n\nIf there is a need to keep a particular container running continuously, the `--restart=always` option is provided to try to re-create the container immediately after it is terminated.\n\nAfter entering the option, run the docker.\n\n```bash\ndocker run --restart=always ubuntu\n```\n\nRun `watch -n1 docker ps` to check if it is restarting.\nIf it is running normally, `Restarting (0)` will be printed in STATUS.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\na911850276e8 ubuntu \"bash\" 35 seconds ago Restarting (0) 6 seconds ago hungry_vaughan\n```\n\n- [https://docs.docker.com/engine/reference/commandline/run/#restart-policies---restart](https://docs.docker.com/engine/reference/commandline/run/#restart-policies---restart)\n - Provides options such as \"on-failure with max retries\" and \"always\"\n\nWhen specifying the restart option for a job resource in Kubernetes, this approach is used.\n\n### Running docker run as a background process\n\nBy default, when running a Docker container, it is executed as a foreground process. This means that the terminal that launched the container is automatically attached to it, preventing you from running other commands.\n\nLet's try an example. Open two terminals, and in one terminal, continuously monitor `docker ps`, while in the other terminal, execute the following commands one by one and observe the behavior.\n\n#### First Practice\n\n```bash\ndocker run -it ubuntu sleep 10\n```\n\nYou must remain stopped for 10 seconds and you cannot perform any other commands from that container. After 10 seconds, you can check in docker ps that the container has terminated.\n\n#### Second Practice\n\n```bash\ndocker run -it ubuntu sleep 10\n```\n\nAfter that, press `ctrl + p` -> `ctrl + q`.\n\nNow you can perform other commands in that terminal, and you can also see that the container is still alive for up to 10 seconds with `docker ps`. This situation, where you exit from the Docker container, is called \"detached\". Docker provides an option to run containers in detached mode, which allows you to run the container in the background while executing the `run` command.\n\n#### Third Practice\n\n```bash\ndocker run -d ubuntu sleep 10\n```\n\nIn detached mode, you can perform other actions in the terminal that executed the command.\n\nIt is good to use detached mode appropriately according to the situation. \nFor example, when developing a backend API server that communicates with the DB, the backend API server needs to be constantly checked with hot-loading while changing the source code, but the DB does not need to be monitored, so it can be executed as follows. \nRun the DB container in detached mode, and run the backend API server in attached mode to follow the logs.\n\n\n## References\n\n- [https://towardsdatascience.com/docker-storage-598e385f4efe](https://towardsdatascience.com/docker-storage-598e385f4efe)\n- [https://vsupalov.com/docker-latest-tag/](https://vsupalov.com/docker-latest-tag/)\n- [https://docs.microsoft.com/ko-kr/azure/container-registry/container-registry-image-tag-version](https://docs.microsoft.com/ko-kr/azure/container-registry/container-registry-image-tag-version)\n- [https://stevelasker.blog/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/](https://stevelasker.blog/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/)\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/prerequisites/docker/command.md",
"content": "---\ntitle : \"[Practice] Docker command\"\ndescription: \"Practice to use docker command.\"\nsidebar_position: 4\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n## 1. Normal installation confirmation\n\n```bash\ndocker run hello-world\n```\n\nIf installed correctly, you should be able to see the following message.\n\n```bash\nHello from Docker!\nThis message shows that your installation appears to be working correctly.\n....\n```\n\n\n**(For ubuntu)** If you want to use without sudo, please refer to the following site.\n\n- [https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user)\n\n## 2. Docker Pull\n\nDocker pull is a command to download Docker images from a Docker image registry (a repository where Docker images are stored and shared).\n\nYou can check the arguments available in docker pull using the command below.\n\n```bash\ndocker pull --help\n```\n\nIf performed normally, it prints out as follows.\n\n```bash\nUsage: docker pull [OPTIONS] NAME[:TAG|@DIGEST]\n\nPull an image or a repository from a registry\n\nOptions:\n -a, --all-tags Download all tagged images in the repository\n --disable-content-trust Skip image verification (default true)\n --platform string Set platform if server is multi-platform capable\n -q, --quiet Suppress verbose output\n```\n\nIt can be seen here that docker pull takes two types of arguments. \n\n1. `[OPTIONS]`\n2. `NAME[:TAG|@DIGEST]`\n\nIn order to use the `-a` and `-q` options from help, they must be used before the NAME. \nLet's try and pull the `ubuntu:18.04` image directly.\n\n```bash\ndocker pull ubuntu:18.04\n```\n\nIf interpreted correctly, the command means to pull an image with the tag `18.04` from an image named `ubuntu`.\n\nIf performed successfully, it will produce an output similar to the following.\n\n```bash\n18.04: Pulling from library/ubuntu\n20d796c36622: Pull complete \nDigest: sha256:42cd9143b6060261187a72716906187294b8b66653b50d70bc7a90ccade5c984\nStatus: Downloaded newer image for ubuntu:18.04\ndocker.io/library/ubuntu:18.04\n```\n\nIf you perform the above command, you will download the image called 'ubuntu:18.04' from a registry named [docker.io/library](http://docker.io/library/) to your laptop.\n\n- Note that \n - in the future, if you need to get a docker image from a certain **private** registry instead of docker.io or public docker hub, you can use [`docker login`](https://docs.docker.com/engine/reference/commandline/login/) to point to the certain registry, then use `docker pull`. Alternatively, you can set up an [insecure registry](https://stackoverflow.com/questions/42211380/add-insecure-registry-to-docker). \n - Also note that [`docker save`](https://docs.docker.com/engine/reference/commandline/save/) and [`docker load`](https://docs.docker.com/engine/reference/commandline/load/) commands are available to store and share docker images in the form of `.tar` file in an intranet.\n\n\n## 3. Docker images\n\nThis is the command to list the Docker images that exist locally.\n\n```bash\ndocker images --help\n```\n\nThe arguments available for use in docker images are as follows.\n\n```bash\nUsage: docker images [OPTIONS] [REPOSITORY[:TAG]]\n\nList images\n\nOptions:\n -a, --all Show all images (default hides intermediate images)\n --digests Show digests\n -f, --filter filter Filter output based on conditions provided\n --format string Pretty-print images using a Go template\n --no-trunc Don't truncate output\n -q, --quiet Only show image IDs\n```\n\nLet's try executing the command below directly.\n\n```bash\ndocker images\n```\n\nIf you install Docker and proceed with this practice, it will output something similar to this.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\nIf you use the `-q` argument among the possible arguments, only the `IMAGE ID` will be printed.\n\n```bash\ndocker images -q\n```\n\n```bash\n29e70752d7b2\n```\n\n## 4. Docker ps\n\nCommand to output the list of currently running Docker containers.\n\n```bash\ndocker ps --help\n```\n\nUse the following arguments can be used with 'docker ps':\n\n```bash\nUsage: docker ps [OPTIONS]\n\nList containers\n\nOptions:\n -a, --all Show all containers (default shows just running)\n -f, --filter filter Filter output based on conditions provided\n --format string Pretty-print containers using a Go template\n -n, --last int Show n last created containers (includes all states) (default -1)\n -l, --latest Show the latest created container (includes all states)\n --no-trunc Don't truncate output\n -q, --quiet Only display container IDs\n -s, --size Display total file sizes\n```\n\nLet's try running the command below directly.\n\n```bash\ndocker ps\n```\n\nIf there are no currently running containers, it will be as follows.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n```\n\nIf there is a container running, it will look similar to this.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nc1e8f5e89d8d ubuntu \"sleep 3600\" 13 seconds ago Up 12 seconds trusting_newton\n```\n\n## 5. Docker run\n\nCommand to run a Docker container.\n\n```bash\ndocker run --help\n```\n\nThe command to run docker run is as follows.\n\n```bash\nUsage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]\n\nRun a command in a new container\n```\n\nWhat we need to confirm here is that the docker run command takes three types of arguments. \n\n1. `[OPTIONS]`\n2. `[COMMAND]`\n3. `[ARG...]`\n\nLet's try running a docker container ourselves.\n\n```bash\n## Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]\ndocker run -it --name demo1 ubuntu:18.04 /bin/bash\n```\n\n- `-it`: Combination of `-i` and `-t` options\n - Runs the container and connects it to an interactive terminal\n- `--name`: Assigns a name to the container for easier identification instead of using the container ID\n- `/bin/bash`: Specifies the command to be executed in the container upon startup, where `/bin/bash` opens a bash shell.\n\nAfter running the command, you can exit the container by using the `exit` command.\n\nWhen you enter the previously learned `docker ps` command, the following output will be displayed.\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n```\n\nIt was said that the container being executed was coming out, but for some reason the container that was just executed does not appear. The reason is that `docker ps` shows the currently running containers by default. If you want to see the stopped containers too, you must give the `-a` option.\n```bash\ndocker ps -a\n```\n\nThen the list of terminated containers will also be displayed.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 2 minutes ago Exited (0) 2 minutes ago demo1\n```\n\n## 6. Docker exec\n\nDocker exec is a command that is used to issue commands or access the inside of a Docker container.\n\n```bash\ndocker exec --help\n```\nFor example, let's try running the following command.\n\n```bash\ndocker run -d --name demo2 ubuntu:18.04 sleep 3600\n```\n\nHere, the `-d` option is a command that allows the Docker container to run in the background so that even if the connection ends to the container, it continues to run.\n\nUse `docker ps` to check if it is currently running.\n\nIt can be confirmed that it is running as follows.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 4 seconds ago Up 3 seconds demo2\n```\n\nNow let's connect to the running docker container through the `docker exec` command.\n\n```bash\ndocker exec -it demo2 /bin/bash\n```\n\nThis is the same as the previous `docker run` command, allowing you to access the inside of the container.\n \nYou can exit using `exit`.\n## 7. Docker logs\n\n```bash\ndocker logs --help\n```\n\nI will have the following container be executed.\n\n```bash\ndocker run --name demo3 -d busybox sh -c \"while true; do $(echo date); sleep 1; done\"\n```\n\nBy using the above command, we have set up a busybox container named \"test\" as a Docker container in the background and printed the current time once every second.\n\nNow let's check the log with the command below.\n\n```bash\ndocker logs demo3\n```\n\nIf performed normally, it will be similar to below.\n\n```bash\nSun Mar 6 11:06:49 UTC 2022\nSun Mar 6 11:06:50 UTC 2022\nSun Mar 6 11:06:51 UTC 2022\nSun Mar 6 11:06:52 UTC 2022\nSun Mar 6 11:06:53 UTC 2022\nSun Mar 6 11:06:54 UTC 2022\n```\nHowever, if used this way, you can only check the logs taken so far. \nIn this case, you can use the `-f` option to keep watching and outputting.\n\n```bash\ndocker logs demo3 -f \n```\n\n## 8. Docker stop\n\nCommand to stop a running Docker container.\n\n```bash\ndocker stop --help\n```\n\nThrough `docker ps`, you can check the containers currently running, as follows.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" About a minute ago Up About a minute demo3\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 4 minutes ago Up 4 minutes demo2\n```\nNow let's try to stop Docker with `docker stop`.\n\n```bash\ndocker stop demo2\n```\n\nAfter executing, type `docker ps` again.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" 2 minutes ago Up 2 minutes demo3\n```\n\nComparing with the above result, you can see that the demo2 container has disappeared from the list of currently running containers.\nThe rest of the containers will also be stopped.\n\n```bash\ndocker stop demo3\n```\n\nDocker rm: Command to delete a Docker container.\n\n```bash\ndocker rm --help\n```\n\nDocker containers are in a stopped state by default. That's why you can see stopped containers using `docker ps -a`.\nBut why do we have to delete the stopped containers? \nEven when stopped, the data used in the Docker remains in the container.\nSo you can restart the container through restarting. But this process will use disk.\nSo\n in order to delete the containers that are not used at all, we should use the `docker rm` command.\n \n First, let's check the current containers.\n\n```bash\ndocker ps -a\n```\n\nThere are three containers as follows.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" 4 minutes ago Exited (137) About a minute ago demo3\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 7 minutes ago Exited (137) 2 minutes ago demo2\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 10 minutes ago Exited (0) 10 minutes ago demo1\n```\n\nLet's try to delete the 'demo3' container through the following command.\n\n```bash\ndocker rm demo3\n```\n\nThe command `docker ps -a` reduced it to two lines as follows.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 13 minutes ago Exited (137) 8 minutes ago demo2\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 16 minutes ago Exited (0) 16 minutes ago demo1\n```\n\nDelete the remaining containers as well.\n\n```bash\ndocker rm demo2\ndocker rm demo1\n```\n\n## 10. Docker rmi\n\nCommand to delete a Docker image.\n\n```bash\ndocker rmi --help\n```\n\nUse the following commands to check which images are currently on the local.\n\n```bash\ndocker images\n```\n\nThe following is output.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nbusybox latest a8440bba1bc0 32 hours ago 1.41MB\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\nI will try to delete the `busybox` image.\n\n```bash\ndocker rmi busybox\n```\n\nIf you type `docker images` again, the following will appear.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\n## References\n\n- [https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry)\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/prerequisites/docker/docker.md",
"content": "---\ntitle : \"What is Docker?\"\ndescription: \"Introduction to Docker.\"\nsidebar_position: 3\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n\n## Container\n\n- Containerization:\n - A technology that allows applications to be executed uniformly anywhere.\n- Container Image:\n - A collection of all the files required to run an application.\n - → Similar to a mold for making fish-shaped bread (Bungeoppang).\n- Container:\n - A single process that is executed based on a container image.\n - → A fish-shaped bread (Bungeoppang) produced using a mold.\n\n## Docker\n\nDocker is a platform that allows you to manage and use containers. \nIts slogan is \"Build Once, Run Anywhere,\" guaranteeing the same execution results anywhere.\n\nIn the Docker, the resources for the container are separated and the lifecycle is controlled by Linux kernel's cgroups, etc. \nHowever, it is too difficult to use these interfaces directly, so an abstraction layer is created.\n\n\n\nThrough this, users can easily control containers with just the user-friendly API **Docker CLI**.\n- Users can easily control containers using the user-friendly API called **Docker CLI**.\n\n## Interpretation of Layer\n\nThe roles of the layers mentioned above are as follows:\n\n1. runC: Utilizes the functionality of the Linux kernel to isolate namespaces, CPUs, memory, filesystems, etc., for a container, which is a single process.\n2. containerd: Acts as an abstraction layer to communicate with runC (OCI layer) and uses the standardized interface (OCI).\n3. dockerd: Solely responsible for issuing commands to containerd.\n4. Docker CLI: Users only need to issue commands to dockerd (Docker daemon) using Docker CLI.\n - During this communication process, Unix socket is used, so sometimes Docker-related errors occur, such as \"the /var/run/docker.sock is in use\" or \"insufficient permissions\" error messages.\n\nAlthough Docker encompasses many stages, when the term \"Docker\" is used, it can refer to Docker CLI, Dockerd (Docker daemon), or even a single Docker container, which can lead to confusion. \nIn the upcoming text, the term \"Docker\" may be used in various contexts.\n\n## For ML Engineer\n\nML engineers use Docker for the following reasons:\n\n1. ML training/inference code needs to be independent of the underlying operating system, Python version, Python environment, and specific versions of Python packages.\n2. Therefore, the goal is to bundle not only the code but also all the dependent packages, environment variables, folder names, etc., into a single package. Containerization technology enables this.\n3. Docker is one of the software tools that makes it easy to use and manage this technology, and the packaged units are referred to as Docker images.\n"
},
{
"path": "i18n/en/docusaurus-plugin-content-docs/version-1.0/prerequisites/docker/images.md",
"content": "---\ntitle : \"[Practice] Docker images\"\ndescription: \"Practice to use docker image.\"\nsidebar_position: 5\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n- `docker commit`\n - running container 를 docker image 로 만드는 방법\n - `docker commit -m \"message\" -a \"author\" For details for spec:
\n\n- **name**:\n - Specifies a name to identify the notebook server.\n- **namespace**:\n - Cannot be changed. (It is automatically set to the namespace of the currently logged-in user account.)\n- **Image**:\n - Selects the image to use from pre-installed JupyterLab images with Python packages like sklearn, pytorch, tensorflow, etc.\n - If you want to use an image that utilizes GPU within the notebook server, refer to the **GPUs** section below.\n - If you want to use a custom notebook server that includes additional packages or source code, you can create a custom image and deploy it for use.\n- **CPU / RAM**:\n - Specifies the amount of resources required.\n - cpu: in core units\n - Represents the number of virtual cores, and can also be specified as a float value such as `1.5`, `2.7`, etc.\n - memory: in Gi units\n- **GPUs**:\n - Specifies the number of GPUs to allocate to the Jupyter notebook.\n - `None`\n - When GPU resources are not required.\n - 1, 2, 4\n - Allocates 1, 2, or 4 GPUs.\n - GPU Vendor:\n - If you have followed the [(Optional) Setup GPU](../setup-kubernetes/setup-nvidia-gpu.md) guide and installed the NVIDIA GPU plugin, select NVIDIA.\n- **Workspace Volume**:\n - Specifies the amount of disk space required within the notebook server.\n - Do not change the Type and Name fields unless you want to increase the disk space or change the AccessMode.\n - Check the **\"Don't use Persistent Storage for User's home\"** checkbox only if it is not necessary to save the notebook server's work. **It is generally recommended not to check this option.**\n - If you want to use a pre-existing Persistent Volume Claim (PVC), select Type as \"Existing\" and enter the name of the PVC to use.\n- **Data Volumes**:\n - If additional storage resources are required, click the **\"+ ADD VOLUME\"** button to create them.\n- ~~Configurations, Affinity/Tolerations, Miscellaneous Settings~~\n - These are generally not needed, so detailed explanations are omitted in *MLOps for All*.\n\n\n Sponsored by MakinaRocks\n
\n \n 이 프로젝트는 MakinaRocks의 지원을 받아 제작되었습니다.\n \n ),\n },\n {\n title: MLOps for MLE,\n Svg: require('@site/static/img/undraw_docusaurus_mountain.svg').default,\n description: (\n <>\n\n ML Engineer를 위한 MLOps Release!\n
\n \n 구글에서 제안한 MLOps 0단계를 직접 구현하며 MLOps 가 무엇인지 공부할 수 있는 튜토리얼을 오픈했습니다!\n \n ),\n },\n];\n\nfunction Feature({title, Svg, description}: FeatureItem) {\n return (\n\n
\n );\n}\n\nexport default function HomepageFeatures(): JSX.Element {\n return (\n \n \n
\n \n
\n {title}
\n{description}
\n\n
\n \n {FeatureList.map((props, idx) => (\n
\n \n
\n );\n}\n\nfunction TeamProfileCardCol(props: ProfileProps) {\n return (\n \n
\n \n
\n \n
\n \n
\n {name}
\n\n
\n {role}
\n{children}
\n \n
\n \n {githubUrl && (\n \n GitHub\n \n )}\n {linkedinUrl && (\n \n LinkedIn\n \n )}\n
\n \n \n \n \n \n 마키나락스에서 머신러닝 엔지니어로 일하고 있습니다.\n\n 모두의 딥러닝을 통해 많은 사람들이 딥러닝을 쉽게 접했듯이 \n 모두의 MLOps를 통해 많은 사람들이 MLOps에 쉽게 접할수 있길 바랍니다.\n\n \n \n \n \n 비효율적인 작업을 자동화하는 것에 관심이 많습니다.\n \n \n \n \n 마키나락스에서 MLOps Engineer로 일하고 있습니다.\n\n 단순하게 생각하는 노력을 하고 있습니다.\n \n \n
\n );\n}\n\nexport function ContributorsRow(): JSX.Element {\n return (\n \n \n \n 마키나락스에서 ML Engineer로 일하고 있습니다.\n \n \n \n \n 마키나락스에서 CTO로 일하고 있습니다.\n 마키나락스는 머신러닝 기반의 산업용 AI 솔루션을 개발하는 스타트업입니다.\n 산업 현장의 문제 해결을 통해 사람이 본연의 일에 집중할 수 있게 만드는 것,\n 그것이 우리가 하는 일입니다.\n \n \n \n \n 3i에서 MLOps Engineer로 일하고 있습니다.\n\n kubeflow에 관심이 많습니다.\n \n \n \n \n Genesis Lab이라는 스타트업에서 Applied AI Engineer 인턴 업무를 수행하고 있습니다.\n\n 머신러닝 생태계가 우리 산업 전반에 큰 변화을 가져올 것이라 믿으며, 한 걸음씩 나아가고 있습니다.\n \n \n \n \n 백패커에서 ML 엔지니어로 일하고 있습니다.\n\n 자연어처리, 추천시스템, MLOps에 관심이 많습니다.\n \n \n
\n );\n}\n"
},
{
"path": "src/css/custom.css",
"content": "/**\n * Any CSS included here will be global. The classic template\n * bundles Infima by default. Infima is a CSS framework designed to\n * work well for content-centric websites.\n */\n\n/* You can override the default Infima variables here. */\n:root {\n --ifm-color-primary: #2e8555;\n --ifm-color-primary-dark: #29784c;\n --ifm-color-primary-darker: #277148;\n --ifm-color-primary-darkest: #205d3b;\n --ifm-color-primary-light: #33925d;\n --ifm-color-primary-lighter: #359962;\n --ifm-color-primary-lightest: #3cad6e;\n --ifm-code-font-size: 95%;\n --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1);\n}\n\n/* For readability concerns, you should choose a lighter palette in dark mode. */\n[data-theme='dark'] {\n --ifm-color-primary: #25c2a0;\n --ifm-color-primary-dark: #21af90;\n --ifm-color-primary-darker: #1fa588;\n --ifm-color-primary-darkest: #1a8870;\n --ifm-color-primary-light: #29d5b0;\n --ifm-color-primary-lighter: #32d8b4;\n --ifm-color-primary-lightest: #4fddbf;\n --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);\n}\n"
},
{
"path": "src/pages/index.module.css",
"content": "/**\n * CSS files with the .module.css suffix will be treated as CSS modules\n * and scoped locally.\n */\n\n.heroBanner {\n padding: 4rem 0;\n text-align: center;\n position: relative;\n overflow: hidden;\n}\n\n@media screen and (max-width: 996px) {\n .heroBanner {\n padding: 2rem;\n }\n}\n\n.buttons {\n display: flex;\n align-items: center;\n justify-content: center;\n}\n"
},
{
"path": "src/pages/index.tsx",
"content": "import React from \"react\";\nimport clsx from \"clsx\";\nimport Link from \"@docusaurus/Link\";\nimport useDocusaurusContext from \"@docusaurus/useDocusaurusContext\";\nimport Layout from \"@theme/Layout\";\nimport HomepageFeatures from \"@site/src/components/HomepageFeatures\";\n\nimport styles from \"./index.module.css\";\n\nfunction HomepageHeader() {\n const { siteConfig } = useDocusaurusContext();\n return (\n \n
\n {siteConfig.title}
\n{siteConfig.tagline}
\n\n \n Let's Start!\n \n
\n \n \n \nmlflow_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: mlflow-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.10, pipelines.kubeflow.org/pipeline_compilation_time: '2022-01-19T14:14:11.999807',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"kernel\", \"type\":\n \"String\"}, {\"name\": \"model_name\", \"type\": \"String\"}], \"name\": \"mlflow_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.10}\nspec:\n entrypoint: mlflow-pipeline\n templates:\n - name: load-iris-data\n container:\n args: [--data, /tmp/outputs/data/data, --target, /tmp/outputs/target/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'pandas' 'scikit-learn' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location 'pandas' 'scikit-learn' --user)\n && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n\n def load_iris_data(\n data_path,\n target_path,\n ):\n import pandas as pd\n from sklearn.datasets import load_iris\n\n iris = load_iris()\n\n data = pd.DataFrame(iris[\"data\"], columns=iris[\"feature_names\"])\n target = pd.DataFrame(iris[\"target\"], columns=[\"target\"])\n\n data.to_csv(data_path, index=False)\n target.to_csv(target_path, index=False)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Load iris data', description='')\n _parser.add_argument(\"--data\", dest=\"data_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--target\", dest=\"target_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = load_iris_data(**_parsed_args)\n image: python:3.7\n outputs:\n artifacts:\n - {name: load-iris-data-data, path: /tmp/outputs/data/data}\n - {name: load-iris-data-target, path: /tmp/outputs/target/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--data\", {\"outputPath\": \"data\"}, \"--target\", {\"outputPath\": \"target\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''pandas'' ''scikit-learn'' ||\n PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n ''pandas'' ''scikit-learn'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef load_iris_data(\\n data_path,\\n target_path,\\n):\\n import\n pandas as pd\\n from sklearn.datasets import load_iris\\n\\n iris = load_iris()\\n\\n data\n = pd.DataFrame(iris[\\\"data\\\"], columns=iris[\\\"feature_names\\\"])\\n target\n = pd.DataFrame(iris[\\\"target\\\"], columns=[\\\"target\\\"])\\n\\n data.to_csv(data_path,\n index=False)\\n target.to_csv(target_path, index=False)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Load iris data'', description='''')\\n_parser.add_argument(\\\"--data\\\",\n dest=\\\"data_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--target\\\", dest=\\\"target_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = load_iris_data(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"name\": \"Load iris data\", \"outputs\": [{\"name\":\n \"data\", \"type\": \"csv\"}, {\"name\": \"target\", \"type\": \"csv\"}]}', pipelines.kubeflow.org/component_ref: '{}'}\n - name: mlflow-pipeline\n inputs:\n parameters:\n - {name: kernel}\n - {name: model_name}\n dag:\n tasks:\n - {name: load-iris-data, template: load-iris-data}\n - name: train-from-csv\n template: train-from-csv\n dependencies: [load-iris-data]\n arguments:\n parameters:\n - {name: kernel, value: '{{inputs.parameters.kernel}}'}\n artifacts:\n - {name: load-iris-data-data, from: '{{tasks.load-iris-data.outputs.artifacts.load-iris-data-data}}'}\n - {name: load-iris-data-target, from: '{{tasks.load-iris-data.outputs.artifacts.load-iris-data-target}}'}\n - name: upload-sklearn-model-to-mlflow\n template: upload-sklearn-model-to-mlflow\n dependencies: [train-from-csv]\n arguments:\n parameters:\n - {name: model_name, value: '{{inputs.parameters.model_name}}'}\n artifacts:\n - {name: train-from-csv-conda_env, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-conda_env}}'}\n - {name: train-from-csv-input_example, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-input_example}}'}\n - {name: train-from-csv-model, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-model}}'}\n - {name: train-from-csv-signature, from: '{{tasks.train-from-csv.outputs.artifacts.train-from-csv-signature}}'}\n - name: train-from-csv\n container:\n args: [--train-data, /tmp/inputs/train_data/data, --train-target, /tmp/inputs/train_target/data,\n --kernel, '{{inputs.parameters.kernel}}', --model, /tmp/outputs/model/data,\n --input-example, /tmp/outputs/input_example/data, --signature, /tmp/outputs/signature/data,\n --conda-env, /tmp/outputs/conda_env/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'dill' 'pandas' 'scikit-learn' 'mlflow' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location 'dill' 'pandas' 'scikit-learn'\n 'mlflow' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n\n def train_from_csv(\n train_data_path,\n train_target_path,\n model_path,\n input_example_path,\n signature_path,\n conda_env_path,\n kernel,\n ):\n import dill\n import pandas as pd\n from sklearn.svm import SVC\n\n from mlflow.models.signature import infer_signature\n from mlflow.utils.environment import _mlflow_conda_env\n\n train_data = pd.read_csv(train_data_path)\n train_target = pd.read_csv(train_target_path)\n\n clf = SVC(kernel=kernel)\n clf.fit(train_data, train_target)\n\n with open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n\n input_example = train_data.sample(1)\n with open(input_example_path, \"wb\") as file_writer:\n dill.dump(input_example, file_writer)\n\n signature = infer_signature(train_data, clf.predict(train_data))\n with open(signature_path, \"wb\") as file_writer:\n dill.dump(signature, file_writer)\n\n conda_env = _mlflow_conda_env(\n additional_pip_deps=[\"dill\", \"pandas\", \"scikit-learn\"]\n )\n with open(conda_env_path, \"wb\") as file_writer:\n dill.dump(conda_env, file_writer)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Train from csv', description='')\n _parser.add_argument(\"--train-data\", dest=\"train_data_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--train-target\", dest=\"train_target_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--kernel\", dest=\"kernel\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--model\", dest=\"model_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--input-example\", dest=\"input_example_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--signature\", dest=\"signature_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--conda-env\", dest=\"conda_env_path\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = train_from_csv(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: kernel}\n artifacts:\n - {name: load-iris-data-data, path: /tmp/inputs/train_data/data}\n - {name: load-iris-data-target, path: /tmp/inputs/train_target/data}\n outputs:\n artifacts:\n - {name: train-from-csv-conda_env, path: /tmp/outputs/conda_env/data}\n - {name: train-from-csv-input_example, path: /tmp/outputs/input_example/data}\n - {name: train-from-csv-model, path: /tmp/outputs/model/data}\n - {name: train-from-csv-signature, path: /tmp/outputs/signature/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--train-data\", {\"inputPath\": \"train_data\"}, \"--train-target\",\n {\"inputPath\": \"train_target\"}, \"--kernel\", {\"inputValue\": \"kernel\"}, \"--model\",\n {\"outputPath\": \"model\"}, \"--input-example\", {\"outputPath\": \"input_example\"},\n \"--signature\", {\"outputPath\": \"signature\"}, \"--conda-env\", {\"outputPath\":\n \"conda_env\"}], \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location ''dill'' ''pandas''\n ''scikit-learn'' ''mlflow'' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m\n pip install --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn''\n ''mlflow'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef train_from_csv(\\n train_data_path,\\n train_target_path,\\n model_path,\\n input_example_path,\\n signature_path,\\n conda_env_path,\\n kernel,\\n):\\n import\n dill\\n import pandas as pd\\n from sklearn.svm import SVC\\n\\n from\n mlflow.models.signature import infer_signature\\n from mlflow.utils.environment\n import _mlflow_conda_env\\n\\n train_data = pd.read_csv(train_data_path)\\n train_target\n = pd.read_csv(train_target_path)\\n\\n clf = SVC(kernel=kernel)\\n clf.fit(train_data,\n train_target)\\n\\n with open(model_path, mode=\\\"wb\\\") as file_writer:\\n dill.dump(clf,\n file_writer)\\n\\n input_example = train_data.sample(1)\\n with open(input_example_path,\n \\\"wb\\\") as file_writer:\\n dill.dump(input_example, file_writer)\\n\\n signature\n = infer_signature(train_data, clf.predict(train_data))\\n with open(signature_path,\n \\\"wb\\\") as file_writer:\\n dill.dump(signature, file_writer)\\n\\n conda_env\n = _mlflow_conda_env(\\n additional_pip_deps=[\\\"dill\\\", \\\"pandas\\\",\n \\\"scikit-learn\\\"]\\n )\\n with open(conda_env_path, \\\"wb\\\") as file_writer:\\n dill.dump(conda_env,\n file_writer)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Train\n from csv'', description='''')\\n_parser.add_argument(\\\"--train-data\\\", dest=\\\"train_data_path\\\",\n type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--train-target\\\",\n dest=\\\"train_target_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--kernel\\\",\n dest=\\\"kernel\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--model\\\",\n dest=\\\"model_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--input-example\\\", dest=\\\"input_example_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--signature\\\",\n dest=\\\"signature_path\\\", type=_make_parent_dirs_and_return_path, required=True,\n default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--conda-env\\\", dest=\\\"conda_env_path\\\",\n type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = train_from_csv(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"train_data\", \"type\": \"csv\"},\n {\"name\": \"train_target\", \"type\": \"csv\"}, {\"name\": \"kernel\", \"type\": \"String\"}],\n \"name\": \"Train from csv\", \"outputs\": [{\"name\": \"model\", \"type\": \"dill\"},\n {\"name\": \"input_example\", \"type\": \"dill\"}, {\"name\": \"signature\", \"type\":\n \"dill\"}, {\"name\": \"conda_env\", \"type\": \"dill\"}]}', pipelines.kubeflow.org/component_ref: '{}',\n pipelines.kubeflow.org/arguments.parameters: '{\"kernel\": \"{{inputs.parameters.kernel}}\"}'}\n - name: upload-sklearn-model-to-mlflow\n container:\n args: [--model-name, '{{inputs.parameters.model_name}}', --model, /tmp/inputs/model/data,\n --input-example, /tmp/inputs/input_example/data, --signature, /tmp/inputs/signature/data,\n --conda-env, /tmp/inputs/conda_env/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'dill' 'pandas' 'scikit-learn' 'mlflow' 'boto3' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location 'dill' 'pandas' 'scikit-learn'\n 'mlflow' 'boto3' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def upload_sklearn_model_to_mlflow(\n model_name,\n model_path,\n input_example_path,\n signature_path,\n conda_env_path,\n ):\n import os\n import dill\n from mlflow.sklearn import save_model\n\n from mlflow.tracking.client import MlflowClient\n\n os.environ[\"MLFLOW_S3_ENDPOINT_URL\"] = \"http://minio-service.kubeflow.svc:9000\"\n os.environ[\"AWS_ACCESS_KEY_ID\"] = \"minio\"\n os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"minio123\"\n\n client = MlflowClient(\"http://mlflow-server-service.mlflow-system.svc:5000\")\n\n with open(model_path, mode=\"rb\") as file_reader:\n clf = dill.load(file_reader)\n\n with open(input_example_path, \"rb\") as file_reader:\n input_example = dill.load(file_reader)\n\n with open(signature_path, \"rb\") as file_reader:\n signature = dill.load(file_reader)\n\n with open(conda_env_path, \"rb\") as file_reader:\n conda_env = dill.load(file_reader)\n\n save_model(\n sk_model=clf,\n path=model_name,\n serialization_format=\"cloudpickle\",\n conda_env=conda_env,\n signature=signature,\n input_example=input_example,\n )\n run = client.create_run(experiment_id=\"0\")\n client.log_artifact(run.info.run_id, model_name)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Upload sklearn model to mlflow', description='')\n _parser.add_argument(\"--model-name\", dest=\"model_name\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--model\", dest=\"model_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--input-example\", dest=\"input_example_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--signature\", dest=\"signature_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--conda-env\", dest=\"conda_env_path\", type=str, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = upload_sklearn_model_to_mlflow(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: model_name}\n artifacts:\n - {name: train-from-csv-conda_env, path: /tmp/inputs/conda_env/data}\n - {name: train-from-csv-input_example, path: /tmp/inputs/input_example/data}\n - {name: train-from-csv-model, path: /tmp/inputs/model/data}\n - {name: train-from-csv-signature, path: /tmp/inputs/signature/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.10\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--model-name\", {\"inputValue\": \"model_name\"}, \"--model\", {\"inputPath\":\n \"model\"}, \"--input-example\", {\"inputPath\": \"input_example\"}, \"--signature\",\n {\"inputPath\": \"signature\"}, \"--conda-env\", {\"inputPath\": \"conda_env\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn''\n ''mlflow'' ''boto3'' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install\n --quiet --no-warn-script-location ''dill'' ''pandas'' ''scikit-learn'' ''mlflow''\n ''boto3'' --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def upload_sklearn_model_to_mlflow(\\n model_name,\\n model_path,\\n input_example_path,\\n signature_path,\\n conda_env_path,\\n):\\n import\n os\\n import dill\\n from mlflow.sklearn import save_model\\n\\n from\n mlflow.tracking.client import MlflowClient\\n\\n os.environ[\\\"MLFLOW_S3_ENDPOINT_URL\\\"]\n = \\\"http://minio-service.kubeflow.svc:9000\\\"\\n os.environ[\\\"AWS_ACCESS_KEY_ID\\\"]\n = \\\"minio\\\"\\n os.environ[\\\"AWS_SECRET_ACCESS_KEY\\\"] = \\\"minio123\\\"\\n\\n client\n = MlflowClient(\\\"http://mlflow-server-service.mlflow-system.svc:5000\\\")\\n\\n with\n open(model_path, mode=\\\"rb\\\") as file_reader:\\n clf = dill.load(file_reader)\\n\\n with\n open(input_example_path, \\\"rb\\\") as file_reader:\\n input_example\n = dill.load(file_reader)\\n\\n with open(signature_path, \\\"rb\\\") as file_reader:\\n signature\n = dill.load(file_reader)\\n\\n with open(conda_env_path, \\\"rb\\\") as file_reader:\\n conda_env\n = dill.load(file_reader)\\n\\n save_model(\\n sk_model=clf,\\n path=model_name,\\n serialization_format=\\\"cloudpickle\\\",\\n conda_env=conda_env,\\n signature=signature,\\n input_example=input_example,\\n )\\n run\n = client.create_run(experiment_id=\\\"0\\\")\\n client.log_artifact(run.info.run_id,\n model_name)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Upload\n sklearn model to mlflow'', description='''')\\n_parser.add_argument(\\\"--model-name\\\",\n dest=\\\"model_name\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--model\\\",\n dest=\\\"model_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--input-example\\\",\n dest=\\\"input_example_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--signature\\\",\n dest=\\\"signature_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--conda-env\\\",\n dest=\\\"conda_env_path\\\", type=str, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = upload_sklearn_model_to_mlflow(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"model_name\", \"type\": \"String\"},\n {\"name\": \"model\", \"type\": \"dill\"}, {\"name\": \"input_example\", \"type\": \"dill\"},\n {\"name\": \"signature\", \"type\": \"dill\"}, {\"name\": \"conda_env\", \"type\": \"dill\"}],\n \"name\": \"Upload sklearn model to mlflow\"}', pipelines.kubeflow.org/component_ref: '{}',\n pipelines.kubeflow.org/arguments.parameters: '{\"model_name\": \"{{inputs.parameters.model_name}}\"}'}\n arguments:\n parameters:\n - {name: kernel}\n - {name: model_name}\n serviceAccountName: pipeline-runner\n```\n\n
\n \n \nexample_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: example-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9, pipelines.kubeflow.org/pipeline_compilation_time: '2021-12-09T18:11:43.193190',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"number_1\", \"type\":\n \"Integer\"}, {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"example_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9}\nspec:\n entrypoint: example-pipeline\n templates:\n - name: example-pipeline\n inputs:\n parameters:\n - {name: number_1}\n - {name: number_2}\n dag:\n tasks:\n - name: print-and-return-number\n template: print-and-return-number\n arguments:\n parameters:\n - {name: number_1, value: '{{inputs.parameters.number_1}}'}\n - name: print-and-return-number-2\n template: print-and-return-number-2\n arguments:\n parameters:\n - {name: number_2, value: '{{inputs.parameters.number_2}}'}\n - name: sum-and-print-numbers\n template: sum-and-print-numbers\n dependencies: [print-and-return-number, print-and-return-number-2]\n arguments:\n parameters:\n - {name: print-and-return-number-2-Output, value: '{{tasks.print-and-return-number-2.outputs.parameters.print-and-return-number-2-Output}}'}\n - {name: print-and-return-number-Output, value: '{{tasks.print-and-return-number.outputs.parameters.print-and-return-number-Output}}'}\n - name: print-and-return-number\n container:\n args: [--number, '{{inputs.parameters.number_1}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(\n str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_1}\n outputs:\n parameters:\n - name: print-and-return-number-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-Output, path: /tmp/outputs/Output/data}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is number 1, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\",\n {\"outputPath\": \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(\\n str(int_value),\n str(type(int_value))))\\n return str(int_value)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Print and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_1}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n - name: print-and-return-number-2\n container:\n args: [--number, '{{inputs.parameters.number_2}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(\n str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_2}\n outputs:\n parameters:\n - name: print-and-return-number-2-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-2-Output, path: /tmp/outputs/Output/data}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is number 2, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\",\n {\"outputPath\": \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(\\n str(int_value),\n str(type(int_value))))\\n return str(int_value)\\n\\nimport argparse\\n_parser\n = argparse.ArgumentParser(prog=''Print and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_2}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n - name: sum-and-print-numbers\n container:\n args: [--number-1, '{{inputs.parameters.print-and-return-number-Output}}', --number-2,\n '{{inputs.parameters.print-and-return-number-2-Output}}']\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def sum_and_print_numbers(number_1, number_2):\n print(number_1 + number_2)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Sum and print numbers', description='')\n _parser.add_argument(\"--number-1\", dest=\"number_1\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--number-2\", dest=\"number_2\", type=int, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = sum_and_print_numbers(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: print-and-return-number-2-Output}\n - {name: print-and-return-number-Output}\n metadata:\n annotations: {pipelines.kubeflow.org/task_display_name: This is sum of number\n 1 and number 2, pipelines.kubeflow.org/component_spec: '{\"implementation\":\n {\"container\": {\"args\": [\"--number-1\", {\"inputValue\": \"number_1\"}, \"--number-2\",\n {\"inputValue\": \"number_2\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def sum_and_print_numbers(number_1, number_2):\\n print(number_1 + number_2)\\n\\nimport\n argparse\\n_parser = argparse.ArgumentParser(prog=''Sum and print numbers'',\n description='''')\\n_parser.add_argument(\\\"--number-1\\\", dest=\\\"number_1\\\",\n type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--number-2\\\",\n dest=\\\"number_2\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = sum_and_print_numbers(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number_1\", \"type\": \"Integer\"},\n {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"Sum and print numbers\"}',\n pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number_1\":\n \"{{inputs.parameters.print-and-return-number-Output}}\", \"number_2\": \"{{inputs.parameters.print-and-return-number-2-Output}}\"}'}\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n arguments:\n parameters:\n - {name: number_1}\n - {name: number_2}\n serviceAccountName: pipeline-runner\n```\n\n
\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n```\n\n파이프라인으로 작성하면 다음과 같이 됩니다.\n\n```python\nfrom functools import partial\n\nimport kfp\nfrom kfp.components import create_component_from_func, OutputPath\nfrom kfp.dsl import pipeline\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"matplotlib\"],\n)\ndef plot_linear(mlpipeline_ui_metadata: OutputPath(\"UI_Metadata\")):\n import base64\n import json\n from io import BytesIO\n\n import matplotlib.pyplot as plt\n\n plt.plot([1, 2, 3], [1, 2, 3])\n\n tmpfile = BytesIO()\n plt.savefig(tmpfile, format=\"png\")\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\"utf-8\")\n\n html = f\"\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n\n\n@pipeline(name=\"plot_pipeline\")\ndef plot_pipeline():\n plot_linear()\n\n\nif __name__ == \"__main__\":\n kfp.compiler.Compiler().compile(plot_pipeline, \"plot_pipeline.yaml\")\n```\n\n이 스크립트를 실행해서 나온 `plot_pipeline.yaml`을 확인하면 다음과 같습니다.\n\n\n \n \nplot_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: plot-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9, pipelines.kubeflow.org/pipeline_compilation_time: '2\n022-01-17T13:31:32.963214',\n pipelines.kubeflow.org/pipeline_spec: '{\"name\": \"plot_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.9}\nspec:\n entrypoint: plot-pipeline\n templates:\n - name: plot-linear\n container:\n args: [--mlpipeline-ui-metadata, /tmp/outputs/mlpipeline_ui_metadata/data]\n command:\n - sh\n - -c\n - (PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location\n 'matplotlib' || PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet\n --no-warn-script-location 'matplotlib' --user) && \"$0\" \"$@\"\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def _make_parent_dirs_and_return_path(file_path: str):\n import os\n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n return file_path\n def plot_linear(mlpipeline_ui_metadata):\n import base64\n import json\n from io import BytesIO\n import matplotlib.pyplot as plt\n plt.plot([1, 2, 3], [1, 2, 3])\n tmpfile = BytesIO()\n plt.savefig(tmpfile, format=\"png\")\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\"utf-8\")\n html = f\"\"\n metadata = {\n \"outputs\": [\n {\n \"type\": \"web-app\",\n \"storage\": \"inline\",\n \"source\": html,\n },\n ],\n }\n with open(mlpipeline_ui_metadata, \"w\") as html_writer:\n json.dump(metadata, html_writer)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Plot linear', description='')\n _parser.add_argument(\"--mlpipeline-ui-metadata\", dest=\"mlpipeline_ui_metadata\", type=_make_parent_dirs_and_return_path, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n _outputs = plot_linear(**_parsed_args)\n image: python:3.7\n outputs:\n artifacts:\n - {name: mlpipeline-ui-metadata, path: /tmp/outputs/mlpipeline_ui_metadata/data}\n metadata:\n labels:\n pipelines.kubeflow.org/kfp_sdk_version: 1.8.9\n pipelines.kubeflow.org/pipeline-sdk-type: kfp\n pipelines.kubeflow.org/enable_caching: \"true\"\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--mlpipeline-ui-metadata\", {\"outputPath\": \"mlpipeline_ui_metadata\"}],\n \"command\": [\"sh\", \"-c\", \"(PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip\n install --quiet --no-warn-script-location ''matplotlib'' || PIP_DISABLE_PIP_VERSION_CHECK=1\n python3 -m pip install --quiet --no-warn-script-location ''matplotlib''\n --user) && \\\"$0\\\" \\\"$@\\\"\", \"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf\n \\\"%s\\\" \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\",\n \"def _make_parent_dirs_and_return_path(file_path: str):\\n import os\\n os.makedirs(os.path.dirname(file_path),\n exist_ok=True)\\n return file_path\\n\\ndef plot_linear(mlpipeline_ui_metadata):\\n import\n base64\\n import json\\n from io import BytesIO\\n\\n import matplotlib.pyplot\n as plt\\n\\n plt.plot([1, 2, 3], [1, 2, 3])\\n\\n tmpfile = BytesIO()\\n plt.savefig(tmpfile,\n format=\\\"png\\\")\\n encoded = base64.b64encode(tmpfile.getvalue()).decode(\\\"utf-8\\\")\\n\\n html\n = f\\\"\\\"\\n metadata = {\\n \\\"outputs\\\":\n [\\n {\\n \\\"type\\\": \\\"web-app\\\",\\n \\\"storage\\\":\n \\\"inline\\\",\\n \\\"source\\\": html,\\n },\\n ],\\n }\\n with\n open(mlpipeline_ui_metadata, \\\"w\\\") as html_writer:\\n json.dump(metadata,\n html_writer)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Plot\n linear'', description='''')\\n_parser.add_argument(\\\"--mlpipeline-ui-metadata\\\",\n dest=\\\"mlpipeline_ui_metadata\\\", type=_make_parent_dirs_and_return_path,\n required=True, default=argparse.SUPPRESS)\\n_parsed_args = vars(_parser.parse_args())\\n\\n_outputs\n = plot_linear(**_parsed_args)\\n\"], \"image\": \"python:3.7\"}}, \"name\": \"Plot\n linear\", \"outputs\": [{\"name\": \"mlpipeline_ui_metadata\", \"type\": \"UI_Metadata\"}]}',\n pipelines.kubeflow.org/component_ref: '{}'}\n - name: plot-pipeline\n dag:\n tasks:\n - {name: plot-linear, template: plot-linear}\n arguments:\n parameters: []\n serviceAccountName: pipeline-runner\n```\n\n
\n
\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow/basic-requirements.md",
"content": "---\ntitle : \"3. Install Requirements\"\ndescription: \"\"\nsidebar_position: 3\ncontributors: [\"Jongseob Jeon\"]\n---\n\n실습을 위해 권장하는 파이썬 버전은 python>=3.7입니다. 파이썬 환경에 익숙하지 않은 분들은 다음 [Appendix 1. 파이썬 가상환경](../appendix/pyenv)을 참고하여 **클라이언트 노드**에 설치해주신 뒤 패키지 설치를 진행해주시기를 바랍니다.\n\n실습을 진행하기에서 필요한 패키지들과 버전은 다음과 같습니다.\n\n- requirements.txt\n\n ```bash\n kfp==1.8.9\n scikit-learn==1.0.1\n mlflow==1.21.0\n pandas==1.3.4\n dill==0.3.4\n ```\n\n[앞에서 만든 파이썬 가상환경](../appendix/pyenv.md#python-가상환경-생성)을 활성화합니다.\n\n```bash\npyenv activate demo\n```\n\n패키지 설치를 진행합니다.\n\n```bash\npip3 install -U pip\npip3 install kfp==1.8.9 scikit-learn==1.0.1 mlflow==1.21.0 pandas==1.3.4 dill==0.3.4\n```\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow/basic-run.md",
"content": "---\ntitle : \"7. Pipeline - Run\"\ndescription: \"\"\nsidebar_position: 7\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Run Pipeline\n\n이제 업로드한 파이프라인을 실행시켜 보겠습니다.\n\n## Before Run\n\n### 1. Create Experiment\n\nExperiment란 Kubeflow 에서 실행되는 Run을 논리적으로 관리하는 단위입니다. \n\nKubeflow에서 namespace를 처음 들어오면 생성되어 있는 Experiment가 없습니다. 따라서 파이프라인을 실행하기 전에 미리 Experiment를 생성해두어야 합니다. Experiment이 있다면 [Run Pipeline](../kubeflow/basic-run.md#run-pipeline-1)으로 넘어가도 무방합니다.\n\nExperiment는 Create Experiment 버튼을 통해 생성할 수 있습니다.\n\n\n\n### 2. Name 입력\n\nExperiment로 사용할 이름을 입력합니다.\n\n\n## Run Pipeline\n\n### 1. Create Run 선택\n\n\n\n### 2. Experiment 선택\n\n\n\n\n\n### 3. Pipeline Config 입력\n\n파이프라인을 생성할 때 입력한 Config 값들을 채워 넣습니다.\n업로드한 파이프라인은 number_1과 number_2를 입력해야 합니다.\n\n\n\n### 4. Start\n\n입력 후 Start 버튼을 누르면 파이프라인이 실행됩니다.\n\n\n\n## Run Result\n\n실행된 파이프라인들은 Runs 탭에서 확인할 수 있습니다.\nRun을 클릭하면 실행된 파이프라인과 관련된 자세한 내용을 확인해 볼 수 있습니다.\n\n\n\n클릭하면 다음과 같은 화면이 나옵니다. 아직 실행되지 않은 컴포넌트는 회색 표시로 나옵니다.\n\n\n\n컴포넌트가 실행이 완료되면 초록색 체크 표시가 나옵니다.\n\n\n\n가장 마지막 컴포넌트를 보면 입력한 Config인 3과 5의 합인 8이 출력된 것을 확인할 수 있습니다.\n\n\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow/how-to-debug.md",
"content": "---\ntitle : \"13. Component - Debugging\"\ndescription: \"\"\nsidebar_position: 13\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Debugging Pipeline\n\n이번 페이지에서는 Kubeflow 컴포넌트를 디버깅하는 방법에 대해서 알아봅니다.\n\n## Failed Component\n\n이번 페이지에서는 [Component - MLFlow](../kubeflow/advanced-mlflow.md#mlflow-pipeline) 에서 이용한 파이프라인을 조금 수정해서 사용합니다.\n\n우선 컴포넌트가 실패하도록 파이프라인을 변경하도록 하겠습니다.\n\n```python\nfrom functools import partial\n\nimport kfp\nfrom kfp.components import InputPath, OutputPath, create_component_from_func\nfrom kfp.dsl import pipeline\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\", \"scikit-learn\"],\n)\ndef load_iris_data(\n data_path: OutputPath(\"csv\"),\n target_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n from sklearn.datasets import load_iris\n\n iris = load_iris()\n\n data = pd.DataFrame(iris[\"data\"], columns=iris[\"feature_names\"])\n target = pd.DataFrame(iris[\"target\"], columns=[\"target\"])\n \n data[\"sepal length (cm)\"] = None\n data.to_csv(data_path, index=False)\n target.to_csv(target_path, index=False)\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\"],\n)\ndef drop_na_from_csv(\n data_path: InputPath(\"csv\"),\n output_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n\n data = pd.read_csv(data_path)\n data = data.dropna()\n data.to_csv(output_path, index=False)\n\n\n@partial(\n create_component_from_func,\n packages_to_install=[\"dill\", \"pandas\", \"scikit-learn\", \"mlflow\"],\n)\ndef train_from_csv(\n train_data_path: InputPath(\"csv\"),\n train_target_path: InputPath(\"csv\"),\n model_path: OutputPath(\"dill\"),\n input_example_path: OutputPath(\"dill\"),\n signature_path: OutputPath(\"dill\"),\n conda_env_path: OutputPath(\"dill\"),\n kernel: str,\n):\n import dill\n import pandas as pd\n from sklearn.svm import SVC\n\n from mlflow.models.signature import infer_signature\n from mlflow.utils.environment import _mlflow_conda_env\n\n train_data = pd.read_csv(train_data_path)\n train_target = pd.read_csv(train_target_path)\n\n clf = SVC(kernel=kernel)\n clf.fit(train_data, train_target)\n\n with open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n\n input_example = train_data.sample(1)\n with open(input_example_path, \"wb\") as file_writer:\n dill.dump(input_example, file_writer)\n\n signature = infer_signature(train_data, clf.predict(train_data))\n with open(signature_path, \"wb\") as file_writer:\n dill.dump(signature, file_writer)\n\n conda_env = _mlflow_conda_env(\n additional_pip_deps=[\"dill\", \"pandas\", \"scikit-learn\"]\n )\n with open(conda_env_path, \"wb\") as file_writer:\n dill.dump(conda_env, file_writer)\n\n\n\n@pipeline(name=\"debugging_pipeline\")\ndef debugging_pipeline(kernel: str):\n iris_data = load_iris_data()\n drop_data = drop_na_from_csv(data=iris_data.outputs[\"data\"])\n model = train_from_csv(\n train_data=drop_data.outputs[\"output\"],\n train_target=iris_data.outputs[\"target\"],\n kernel=kernel,\n )\n\nif __name__ == \"__main__\":\n kfp.compiler.Compiler().compile(debugging_pipeline, \"debugging_pipeline.yaml\")\n\n```\n\n수정한 점은 다음과 같습니다.\n\n1. 데이터를 불러오는 `load_iris_data` 컴포넌트에서 `sepal length (cm)` 피처에 `None` 값을 주입\n2. `drop_na_from_csv` 컴포넌트에서 `drop_na()` 함수를 이용해 na 값이 포함된 `row`를 제거\n\n이제 파이프라인을 업로드하고 실행해 보겠습니다. \n실행 후 Run을 눌러서 확인해보면 `Train from csv` 컴포넌트에서 실패했다고 나옵니다.\n\n\n\n실패한 컴포넌트를 클릭하고 로그를 확인해서 실패한 이유를 확인해 보겠습니다.\n\n\n\n로그를 확인하면 데이터의 개수가 0이여서 실행되지 않았다고 나옵니다. \n분명 정상적으로 데이터를 전달했는데 왜 데이터의 개수가 0개일까요? \n\n이제 입력받은 데이터에 어떤 문제가 있었는지 확인해 보겠습니다. \n우선 컴포넌트를 클릭하고 Input/Ouput 탭에서 입력값으로 들어간 데이터들을 다운로드 받습니다. \n다운로드는 빨간색 네모로 표시된 곳의 링크를 클릭하면 됩니다.\n\n\n\n두 개의 파일을 같은 경로에 다운로드합니다. \n그리고 해당 경로로 이동해서 파일을 확인합니다.\n\n```bash\nls\n```\n\n다음과 같이 두 개의 파일이 있습니다.\n\n```bash\ndrop-na-from-csv-output.tgz load-iris-data-target.tgz\n```\n\n압축을 풀어보겠습니다.\n\n```bash\ntar -xzvf load-iris-data-target.tgz ; mv data target.csv\ntar -xzvf drop-na-from-csv-output.tgz ; mv data data.csv\n```\n\n그리고 이를 주피터 노트북을 이용해 컴포넌트 코드를 실행합니다.\n\n\n\n디버깅을 해본 결과 dropna 할 때 column을 기준으로 drop을 해야 하는데 row를 기준으로 drop을 해서 데이터가 모두 사라졌습니다.\n이제 문제의 원인을 알아냈으니 column을 기준으로 drop이 되게 컴포넌트를 수정합니다.\n\n```python\n@partial(\n create_component_from_func,\n packages_to_install=[\"pandas\"],\n)\ndef drop_na_from_csv(\n data_path: InputPath(\"csv\"),\n output_path: OutputPath(\"csv\"),\n):\n import pandas as pd\n\n data = pd.read_csv(data_path)\n data = data.dropna(axis=\"columns\")\n data.to_csv(output_path, index=False)\n```\n\n수정 후 파이프라인을 다시 업로드하고 실행하면 다음과 같이 정상적으로 수행하는 것을 확인할 수 있습니다.\n\n\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow/kubeflow-concepts.md",
"content": "---\ntitle : \"2. Kubeflow Concepts\"\ndescription: \"\"\nsidebar_position: 2\ncontributors: [\"Jongseob Jeon\"]\n---\n\n## Component\n\n컴포넌트(Component)는 컴포넌트 콘텐츠(Component contents)와 컴포넌트 래퍼(Component wrapper)로 구성되어 있습니다.\n하나의 컴포넌트는 컴포넌트 래퍼를 통해 kubeflow에 전달되며 전달된 컴포넌트는 정의된 컴포넌트 콘텐츠를 실행(execute)하고 아티팩트(artifacts)들을 생산합니다.\n\n\n\n### Component Contents\n\n컴포넌트 콘텐츠를 구성하는 것은 총 3가지가 있습니다.\n\n\n\n1. Environemnt\n2. Python code w\\ Config\n3. Generates Artifacts\n\n예시와 함께 각 구성 요소가 어떤 것인지 알아보도록 하겠습니다.\n다음과 같이 데이터를 불러와 SVC(Support Vector Classifier)를 학습한 후 SVC 모델을 저장하는 과정을 적은 파이썬 코드가 있습니다.\n\n```python\nimport dill\nimport pandas as pd\n\nfrom sklearn.svm import SVC\n\ntrain_data = pd.read_csv(train_data_path)\ntrain_target= pd.read_csv(train_target_path)\n\nclf= SVC(\n kernel=kernel\n)\nclf.fit(train_data)\n\nwith open(model_path, mode=\"wb\") as file_writer:\n dill.dump(clf, file_writer)\n```\n\n위의 파이썬 코드는 다음과 같이 컴포넌트 콘텐츠로 나눌 수 있습니다.\n\n\n\nEnvironment는 파이썬 코드에서 사용하는 패키지들을 import하는 부분입니다. \n다음으로 Python Code w\\ Config 에서는 주어진 Config를 이용해 실제로 학습을 수행합니다. \n마지막으로 아티팩트를 저장하는 과정이 있습니다.\n\n### Component Wrapper\n\n컴포넌트 래퍼는 컴포넌트 콘텐츠에 필요한 Config를 전달하고 실행시키는 작업을 합니다.\n\n\n\nKubeflow에서는 컴포넌트 래퍼를 위의 `train_svc_from_csv`와 같이 함수의 형태로 정의합니다.\n컴포넌트 래퍼가 콘텐츠를 감싸면 다음과 같이 됩니다.\n\n\n\n### Artifacts\n\n위의 설명에서 컴포넌트는 아티팩트(Artifacts)를 생성한다고 했습니다. 아티팩트란 evaluation result, log 등 어떤 형태로든 파일로 생성되는 것을 통틀어서 칭하는 용어입니다.\n그중 우리가 관심을 두는 유의미한 것들은 다음과 같은 것들이 있습니다.\n\n\n\n- Model\n- Data\n- Metric\n- etc\n\n#### Model\n\n저희는 모델을 다음과 같이 정의 했습니다.\n\n> 모델이란 파이썬 코드와 학습된 Weights와 Network 구조 그리고 이를 실행시키기 위한 환경이 모두 포함된 형태\n\n#### Data\n\n데이터는 전 처리된 피처, 모델의 예측 값 등을 포함합니다.\n\n#### Metric\n\nMetric은 동적 지표와 정적 지표 두 가지로 나누었습니다.\n\n- 동적 지표란 train loss와 같이 학습이 진행되는 중 에폭(Epoch)마다 계속해서 변화하는 값을 의미합니다.\n- 정적 지표란 학습이 끝난 후 최종적으로 모델을 평가하는 정확도 등을 의미합니다.\n\n## Pipeline\n\n파이프라인은 컴포넌트의 집합과 컴포넌트를 실행시키는 순서도로 구성되어 있습니다. 이 때, 순서도는 방향 순환이 없는 그래프로 이루어져 있으며, 간단한 조건문을 포함할 수 있습니다.\n\n\n\n### Pipeline Config\n\n앞서 컴포넌트를 실행시키기 위해서는 Config가 필요하다고 설명했습니다. 파이프라인을 구성하는 컴포넌트의 Config 들을 모아 둔 것이 파이프라인 Config입니다.\n\n\n\n## Run\n\n파이프라인이 필요로 하는 파이프라인 Config가 주어져야지만 파이프라인을 실행할 수 있습니다. \nKubeflow에서는 실행된 파이프라인을 Run 이라고 부릅니다.\n\n\n\n파이프라인이 실행되면 각 컴포넌트가 아티팩트들을 생성합니다.\nKubeflow pipeline에서는 Run 하나당 고유한 ID 를 생성하고, Run에서 생성되는 모든 아티팩트들을 저장합니다.\n\n\n\n그러면 이제 직접 컴포넌트와 파이프라인을 작성하는 방법에 대해서 알아보도록 하겠습니다.\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow/kubeflow-intro.md",
"content": "---\ntitle : \"1. Kubeflow Introduction\"\ndescription: \"\"\nsidebar_position: 1\ncontributors: [\"Jongseob Jeon\"]\n---\n\nKubeflow를 사용하기 위해서는 컴포넌트(Component)와 파이프라인(Pipeline)을 작성해야 합니다.\n\n*모두의 MLOps*에서 설명하는 방식은 [Kubeflow Pipeline 공식 홈페이지](https://www.kubeflow.org/docs/components/pipelines/overview/quickstart/)에서 설명하는 방식과는 다소 차이가 있습니다. 여기에서는 Kubeflow Pipeline을 워크플로(Workflow)가 아닌 앞서 설명한 [MLOps를 구성하는 요소](../kubeflow/kubeflow-concepts.md#component-contents) 중 하나의 컴포넌트로 사용하기 때문입니다.\n\n그럼 이제 컴포넌트와 파이프라인은 무엇이며 어떻게 작성할 수 있는지 알아보도록 하겠습니다.\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow-dashboard-guide/_category_.json",
"content": "{\n \"label\": \"Kubeflow UI Guide\",\n \"position\": 5,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow-dashboard-guide/experiments-and-others.md",
"content": "---\ntitle : \"6. Kubeflow Pipeline 관련\"\ndescription: \"\"\nsidebar_position: 6\ncontributors: [\"Jaeyeon Kim\"]\n---\n\nCentral Dashboard의 왼쪽 탭의 Experiments(KFP), Pipelines, Runs, Recurring Runs, Artifacts, Executions 페이지들에서는 Kubeflow Pipeline과 Pipeline의 실행 그리고 Pipeline Run의 결과를 관리합니다.\n\n\n\nKubeflow Pipeline이 *모두의 MLOps*에서 Kubeflow를 사용하는 주된 이유이며, Kubeflow Pipeline을 만드는 방법, 실행하는 방법, 결과를 확인하는 방법 등 자세한 내용은 [3.Kubeflow](../kubeflow/kubeflow-intro)에서 다룹니다.\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow-dashboard-guide/experiments.md",
"content": "---\ntitle : \"5. Experiments(AutoML)\"\ndescription: \"\"\nsidebar_position: 5\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n다음으로는 Central Dashboard의 왼쪽 탭의 Experiments(AutoML)을 클릭해보겠습니다.\n\n\n\n\n\nExperiments(AutoML) 페이지는 Kubeflow에서 Hyperparameter Tuning과 Neural Architecture Search를 통한 AutoML을 담당하는 [Katib](https://www.kubeflow.org/docs/components/katib/overview/)를 관리할 수 있는 페이지입니다.\n\nKatib와 Experiments(AutoML)에 대한 사용법은 *모두의 MLOps* v1.0에서는 다루지 않으며, v2.0에 추가될 예정입니다.\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow-dashboard-guide/intro.md",
"content": "---\ntitle : \"1. Central Dashboard\"\ndescription: \"\"\nsidebar_position: 1\ncontributors: [\"Jaeyeon Kim\", \"SeungTae Kim\"]\n---\n\n[Kubeflow 설치](../setup-components/install-components-kf.md)를 완료하면, 다음 커맨드를 통해 대시보드에 접속할 수 있습니다.\n\n```bash\nkubectl port-forward --address 0.0.0.0 svc/istio-ingressgateway -n istio-system 8080:80\n```\n\n\n\nCentral Dashboard는 Kubeflow에서 제공하는 모든 기능을 통합하여 제공하는 UI입니다. Central Dashboard에서 제공하는 기능은 크게 왼쪽의 탭을 기준으로 구분할 수 있습니다.\n\n\n\n- Home\n- Notebooks\n- Tensorboards\n- Volumes\n- Models\n- Experiments(AutoML)\n- Experiments(KFP)\n- Pipelines\n- Runs\n- Recurring Runs\n- Artifacts\n- Executions\n\n그럼 이제 기능별 간단한 사용법을 알아보겠습니다.\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow-dashboard-guide/notebooks.md",
"content": "---\ntitle : \"2. Notebooks\"\ndescription: \"\"\nsidebar_position: 2\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n## 노트북 서버(Notebook Server) 생성하기\n\n다음 Central Dashboard의 왼쪽 탭의 Notebooks를 클릭해보겠습니다.\n\n\n\n다음과 같은 화면을 볼 수 있습니다.\n\nNotebooks 탭은 JupyterHub와 비슷하게 유저별로 jupyter notebook 및 code server 환경(이하 노트북 서버)을 독립적으로 생성하고 접속할 수 있는 페이지입니다.\n\n\n\n오른쪽 위의 `+ NEW NOTEBOOK` 버튼을 클릭합니다.\n\n\n\n아래와 같은 화면이 나타나면, 이제 생성할 노트북 서버의 스펙(Spec)을 명시하여 생성합니다.\n\n\n\nexample_pipeline.yaml
\n\n```bash\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n generateName: example-pipeline-\n annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline_compilation_time: '2021-12-05T13:38:51.566777',\n pipelines.kubeflow.org/pipeline_spec: '{\"inputs\": [{\"name\": \"number_1\", \"type\":\n \"Integer\"}, {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"example_pipeline\"}'}\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3}\nspec:\n entrypoint: example-pipeline\n templates:\n - name: example-pipeline\n inputs:\n parameters:\n - {name: number_1}\n - {name: number_2}\n dag:\n tasks:\n - name: print-and-return-number\n template: print-and-return-number\n arguments:\n parameters:\n - {name: number_1, value: '{{inputs.parameters.number_1}}'}\n - name: print-and-return-number-2\n template: print-and-return-number-2\n arguments:\n parameters:\n - {name: number_2, value: '{{inputs.parameters.number_2}}'}\n - name: sum-and-print-numbers\n template: sum-and-print-numbers\n dependencies: [print-and-return-number, print-and-return-number-2]\n arguments:\n parameters:\n - {name: print-and-return-number-2-Output, value: '{{tasks.print-and-return-number-2.outputs.parameters.print-and-return-number-2-Output}}'}\n - {name: print-and-return-number-Output, value: '{{tasks.print-and-return-number.outputs.parameters.print-and-return-number-Output}}'}\n - name: print-and-return-number\n container:\n args: [--number, '{{inputs.parameters.number_1}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_1}\n outputs:\n parameters:\n - name: print-and-return-number-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-Output, path: /tmp/outputs/Output/data}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\", {\"outputPath\":\n \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(str(int_value), str(type(int_value))))\\n return\n str(int_value)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Print\n and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_1}}\"}'}\n - name: print-and-return-number-2\n container:\n args: [--number, '{{inputs.parameters.number_2}}', '----output-paths', /tmp/outputs/Output/data]\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def print_and_return_number(number):\n print(number)\n return number\n\n def _serialize_int(int_value: int) -> str:\n if isinstance(int_value, str):\n return int_value\n if not isinstance(int_value, int):\n raise TypeError('Value \"{}\" has type \"{}\" instead of int.'.format(str(int_value), str(type(int_value))))\n return str(int_value)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Print and return number', description='')\n _parser.add_argument(\"--number\", dest=\"number\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"----output-paths\", dest=\"_output_paths\", type=str, nargs=1)\n _parsed_args = vars(_parser.parse_args())\n _output_files = _parsed_args.pop(\"_output_paths\", [])\n\n _outputs = print_and_return_number(**_parsed_args)\n\n _outputs = [_outputs]\n\n _output_serializers = [\n _serialize_int,\n\n ]\n\n import os\n for idx, output_file in enumerate(_output_files):\n try:\n os.makedirs(os.path.dirname(output_file))\n except OSError:\n pass\n with open(output_file, 'w') as f:\n f.write(_output_serializers[idx](_outputs[idx]))\n image: python:3.7\n inputs:\n parameters:\n - {name: number_2}\n outputs:\n parameters:\n - name: print-and-return-number-2-Output\n valueFrom: {path: /tmp/outputs/Output/data}\n artifacts:\n - {name: print-and-return-number-2-Output, path: /tmp/outputs/Output/data}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number\", {\"inputValue\": \"number\"}, \"----output-paths\", {\"outputPath\":\n \"Output\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n print_and_return_number(number):\\n print(number)\\n return number\\n\\ndef\n _serialize_int(int_value: int) -> str:\\n if isinstance(int_value, str):\\n return\n int_value\\n if not isinstance(int_value, int):\\n raise TypeError(''Value\n \\\"{}\\\" has type \\\"{}\\\" instead of int.''.format(str(int_value), str(type(int_value))))\\n return\n str(int_value)\\n\\nimport argparse\\n_parser = argparse.ArgumentParser(prog=''Print\n and return number'', description='''')\\n_parser.add_argument(\\\"--number\\\",\n dest=\\\"number\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"----output-paths\\\",\n dest=\\\"_output_paths\\\", type=str, nargs=1)\\n_parsed_args = vars(_parser.parse_args())\\n_output_files\n = _parsed_args.pop(\\\"_output_paths\\\", [])\\n\\n_outputs = print_and_return_number(**_parsed_args)\\n\\n_outputs\n = [_outputs]\\n\\n_output_serializers = [\\n _serialize_int,\\n\\n]\\n\\nimport\n os\\nfor idx, output_file in enumerate(_output_files):\\n try:\\n os.makedirs(os.path.dirname(output_file))\\n except\n OSError:\\n pass\\n with open(output_file, ''w'') as f:\\n f.write(_output_serializers[idx](_outputs[idx]))\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number\", \"type\": \"Integer\"}],\n \"name\": \"Print and return number\", \"outputs\": [{\"name\": \"Output\", \"type\":\n \"Integer\"}]}', pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number\":\n \"{{inputs.parameters.number_2}}\"}'}\n - name: sum-and-print-numbers\n container:\n args: [--number-1, '{{inputs.parameters.print-and-return-number-Output}}', --number-2,\n '{{inputs.parameters.print-and-return-number-2-Output}}']\n command:\n - sh\n - -ec\n - |\n program_path=$(mktemp)\n printf \"%s\" \"$0\" > \"$program_path\"\n python3 -u \"$program_path\" \"$@\"\n - |\n def sum_and_print_numbers(number_1, number_2):\n print(number_1 + number_2)\n\n import argparse\n _parser = argparse.ArgumentParser(prog='Sum and print numbers', description='')\n _parser.add_argument(\"--number-1\", dest=\"number_1\", type=int, required=True, default=argparse.SUPPRESS)\n _parser.add_argument(\"--number-2\", dest=\"number_2\", type=int, required=True, default=argparse.SUPPRESS)\n _parsed_args = vars(_parser.parse_args())\n\n _outputs = sum_and_print_numbers(**_parsed_args)\n image: python:3.7\n inputs:\n parameters:\n - {name: print-and-return-number-2-Output}\n - {name: print-and-return-number-Output}\n metadata:\n labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.6.3, pipelines.kubeflow.org/pipeline-sdk-type: kfp}\n annotations: {pipelines.kubeflow.org/component_spec: '{\"implementation\": {\"container\":\n {\"args\": [\"--number-1\", {\"inputValue\": \"number_1\"}, \"--number-2\", {\"inputValue\":\n \"number_2\"}], \"command\": [\"sh\", \"-ec\", \"program_path=$(mktemp)\\nprintf \\\"%s\\\"\n \\\"$0\\\" > \\\"$program_path\\\"\\npython3 -u \\\"$program_path\\\" \\\"$@\\\"\\n\", \"def\n sum_and_print_numbers(number_1, number_2):\\n print(number_1 + number_2)\\n\\nimport\n argparse\\n_parser = argparse.ArgumentParser(prog=''Sum and print numbers'',\n description='''')\\n_parser.add_argument(\\\"--number-1\\\", dest=\\\"number_1\\\",\n type=int, required=True, default=argparse.SUPPRESS)\\n_parser.add_argument(\\\"--number-2\\\",\n dest=\\\"number_2\\\", type=int, required=True, default=argparse.SUPPRESS)\\n_parsed_args\n = vars(_parser.parse_args())\\n\\n_outputs = sum_and_print_numbers(**_parsed_args)\\n\"],\n \"image\": \"python:3.7\"}}, \"inputs\": [{\"name\": \"number_1\", \"type\": \"Integer\"},\n {\"name\": \"number_2\", \"type\": \"Integer\"}], \"name\": \"Sum and print numbers\"}',\n pipelines.kubeflow.org/component_ref: '{}', pipelines.kubeflow.org/arguments.parameters: '{\"number_1\":\n \"{{inputs.parameters.print-and-return-number-Output}}\", \"number_2\": \"{{inputs.parameters.print-and-return-number-2-Output}}\"}'}\n arguments:\n parameters:\n - {name: number_1}\n - {name: number_2}\n serviceAccountName: pipeline-runner\n```\n\n\n
\n\n모두 정상적으로 입력하였다면 하단의 **LAUNCH** 버튼이 활성화되며, 버튼을 클릭하면 노트북 서버 생성이 시작됩니다.\n\n\n\n생성 후 아래와 같이 **Status** 가 초록색 체크 표시 아이콘으로 변하며, **CONNECT 버튼**이 활성화됩니다.\n\n\n\n---\n\n## 노트북 서버 접속하기\n\n**CONNECT 버튼**을 클릭하면 브라우저에 새 창이 열리며, 다음과 같은 화면이 보입니다.\n\n\n\n**Launcher**의 Notebook, Console, Terminal 아이콘을 클릭하여 사용할 수 있습니다.\n\n 생성된 Notebook 화면\n\n\n\n 생성된 Terminal 화면\n\n\n\n---\n\n## 노트북 서버 중단하기\n\n노트북 서버를 오랜 시간 사용하지 않는 경우, 쿠버네티스 클러스터의 효율적인 리소스 사용을 위해서 노트북 서버를 중단(Stop)할 수 있습니다. **단, 이 경우 노트북 서버 생성 시 Workspace Volume 또는 Data Volume으로 지정해놓은 경로 외에 저장된 데이터는 모두 초기화되는 것에 주의하시기 바랍니다.** \n노트북 서버 생성 당시 경로를 변경하지 않았다면, 디폴트(Default) Workspace Volume의 경로는 노트북 서버 내의 `/home/jovyan` 이므로, `/home/jovyan` 의 하위 경로 이외의 경로에 저장된 데이터는 모두 사라집니다.\n\n다음과 같이 `STOP` 버튼을 클릭하면 노트북 서버가 중단됩니다.\n\n\n\n중단이 완료되면 다음과 같이 `CONNECT` 버튼이 비활성화되며, `PLAY` 버튼을 클릭하면 다시 정상적으로 사용할 수 있습니다.\n\n\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow-dashboard-guide/tensorboards.md",
"content": "---\ntitle : \"3. Tensorboards\"\ndescription: \"\"\nsidebar_position: 3\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n다음으로는 Central Dashboard의 왼쪽 탭의 Tensorboards를 클릭해보겠습니다.\n\n\n\n다음과 같은 화면을 볼 수 있습니다.\n\n\n\nTensorboards 탭은 Tensorflow, PyTorch 등의 프레임워크에서 제공하는 Tensorboard 유틸이 생성한 ML 학습 관련 데이터를 시각화하는 텐서보드 서버(Tensorboard Server)를 쿠버네티스 클러스터에 생성하는 기능을 제공합니다.\n\n이렇게 생성한 텐서보드 서버는, 일반적인 원격 텐서보드 서버의 사용법과 같이 사용할 수도 있으며, [Kubeflow 파이프라인 런에서 바로 텐서보드 서버에 데이터를 저장하는 용도](https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#tensorboard)로 활용할 수 있습니다.\n\nKubeflow 파이프라인 런의 결과를 시각화하는 방법에는 [다양한 방식](https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/)이 있으며, *모두의 MLOps*에서는 더 일반적으로 활용할 수 있도록 Kubeflow 컴포넌트의 Visualization 기능과 MLflow의 시각화 기능을 활용할 예정이므로, Tensorboards 페이지에 대한 자세한 설명은 생략하겠습니다.\n"
},
{
"path": "versioned_docs/version-1.0/kubeflow-dashboard-guide/volumes.md",
"content": "---\ntitle : \"4. Volumes\"\ndescription: \"\"\nsidebar_position: 4\ncontributors: [\"Jaeyeon Kim\"]\n---\n\n## Volumes\n\n다음으로는 Central Dashboard의 왼쪽 탭의 Volumes를 클릭해보겠습니다.\n\n\n\n다음과 같은 화면을 볼 수 있습니다.\n\n\n\nVolumes 탭은 [Kubernetes의 볼륨(Volume)](https://kubernetes.io/ko/docs/concepts/storage/volumes/), 정확히는 [퍼시스턴트 볼륨 클레임(Persistent Volume Claim, 이하 pvc)](https://kubernetes.io/ko/docs/concepts/storage/persistent-volumes/) 중 현재 user의 namespace에 속한 pvc를 관리하는 기능을 제공합니다.\n\n위 스크린샷을 보면, [1. Notebooks](../kubeflow-dashboard-guide/notebooks) 페이지에서 생성한 Volume의 정보를 확인할 수 있습니다. 해당 Volume의 Storage Class는 쿠버네티스 클러스터 설치 당시 설치한 Default Storage Class인 local-path로 설정되어있음을 확인할 수 있습니다.\n\n이외에도 user namespace에 새로운 볼륨을 생성하거나, 조회하거나, 삭제하고 싶은 경우에 Volumes 페이지를 활용할 수 있습니다.\n\n---\n\n## 볼륨 생성하기\n\n오른쪽 위의 `+ NEW VOLUME` 버튼을 클릭하면 다음과 같은 화면을 볼 수 있습니다.\n\n\n\nname, size, storage class, access mode를 지정하여 생성할 수 있습니다.\n\n원하는 리소스 스펙을 지정하여 생성하면 다음과 같이 볼륨의 Status가 `Pending`으로 조회됩니다. `Status` 아이콘에 마우스 커서를 가져다 대면 *해당 볼륨은 mount하여 사용하는 first consumer가 나타날 때 실제로 생성을 진행한다(This volume will be bound when its first consumer is created.)*는 메시지를 확인할 수 있습니다. \n이는 실습을 진행하는 [StorageClass](https://kubernetes.io/ko/docs/concepts/storage/storage-classes/)인 `local-path`의 볼륨 생성 정책에 해당하며, **문제 상황이 아닙니다.** \n해당 페이지에서 Status가 `Pending` 으로 보이더라도 해당 볼륨을 사용하길 원하는 노트북 서버 혹은 파드(Pod)에서는 해당 볼륨의 이름을 지정하여 사용할 수 있으며, 그때 실제로 볼륨 생성이 진행됩니다.\n\n\n"
},
{
"path": "versioned_docs/version-1.0/prerequisites/_category_.json",
"content": "{\n \"label\": \"Prerequisites\",\n \"position\": 1,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "versioned_docs/version-1.0/prerequisites/docker/_category_.json",
"content": "{\n \"label\": \"Docker\",\n \"position\": 1,\n \"link\": {\n \"type\": \"generated-index\"\n }\n}\n"
},
{
"path": "versioned_docs/version-1.0/prerequisites/docker/advanced.md",
"content": "---\ntitle : \"[Practice] Docker Advanced\"\ndescription: \"Practice to use docker more advanced way.\"\nsidebar_position: 6\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n## 도커 이미지 잘 만들기\n\n### 도커 이미지를 만들 때 고려해야 될 점\n\nDockerfile 을 활용하여 도커 이미지를 만들 때는 명령어의 **순서**가 중요합니다. \n그 이유는 도커 이미지는 여러 개의 Read-Only Layer 로 구성되어있고, 이미지를 빌드할 때 이미 존재하는 레이어는 **캐시되어** 재사용되기 때문에, 이를 생각해서 Dockerfile 을 구성한다면 **빌드 시간을 줄일 수 있습니다.**\n\nDockerfile에서 `RUN`, `ADD`, `COPY` 명령어 하나가 하나의 레이어로 저장됩니다.\n\n예를 들어서 다음과 같은 `Dockerfile`이 있습니다.\n\n```docker\n# Layer 1\nFROM ubuntu:latest\n\n# Layer 2\nRUN apt-get update && apt-get install python3 pip3 -y\n\n# Layer 3\nRUN pip3 install -U pip && pip3 install torch\n\n# Layer 4\nCOPY src/ src/\n\n# Layer 5\nCMD python src/app.py\n```\n\n위의 `Dockerfile`로 빌드된 이미지를 `docker run -it app:latest /bin/bash` 명령어로 실행하면 다음과 같은 레이어로 표현할 수 있습니다.\n\n\n\n최상단의 R/W Layer 는 이미지에 영향을 주지 않습니다. 즉, 컨테이너 내부에서 작업한 내역은 모두 휘발성입니다.\n\n하단의 레이어가 변경되면, 그 위의 레이어는 모두 새로 빌드됩니다. 그래서 Dockerfile 내장 명령어의 순서가 중요합니다. \n예를 들면, **자주 변경**되는 부분은 **최대한 뒤쪽으로** 정렬하는 것을 추천합니다. (ex. `COPY src/ app/src/`)\n\n그렇기 때문에 반대로 변경되지 않는 부분은 최대한 앞쪽으로 정렬하는게 좋습니다.\n\n만약 거의 **변경되지 않지만**, 여러 곳에서 **자주** 쓰이는 부분을 공통화할 수도 있습니다.\n해당 공통부분만 묶어서 별도의 이미지는 미리 만들어둔 다음, **베이스 이미지** 로 활용하는 것이 좋습니다.\n\n예를 들어, 다른 건 거의 똑같은데, tensorflow-cpu 를 사용하는 이미지와, tensorflow-gpu 를 사용하는 환경을 분리해서 이미지로 만들고 싶은 경우에는 다음과 같이 할 수 있습니다. \npython 과 기타 기본적인 패키지가 설치된 [`ghcr.io/makinarocks/python:3.8-base`](http://ghcr.io/makinarocks/python:3.8-base-cpu) 를 만들어두고, **tensorflow cpu 버전과 gpu 버전이** 설치된 이미지 새로 만들때는, 위의 이미지를 `FROM` 으로 불러온 다음, tensorflow install 하는 부분만 별도로 작성해서 Dockerfile 을 2 개로 관리한다면 가독성도 좋고 빌드 시간도 줄일 수 있습니다.\n\n**합칠 수 있는 Layer 는 합치는 것**이 Old version 의 도커에서는 성능 향상 효과를 이끌었습니다. 여러분의 도커 컨테이너가 어떤 도커 버전에서 실행될 것인지 보장할 수 없으며, **가독성**을 위해서도 합칠 수 있는 Layer 는 적절히 합치는 것이 좋습니다.\n\n예를 들면, 다음과 같이 작성된 `Dockerfile`이 있습니다.\n\n```docker\n# Bad Case\nRUN apt-get update\nRUN apt-get install build-essential -y\nRUN apt-get install curl -y\nRUN apt-get install jq -y\nRUN apt-get install git -y\n```\n\n이를 아래와 같이 합쳐서 적을 수 있습니다.\n\n```docker\n# Better Case\nRUN apt-get update && \\\n apt-get install -y \\\n build-essential \\\n curl \\\n jq \\\n git\n```\n\n편의를 위해서는 `.dockerignore` 도 사용하는게 좋습니다.\n`.dockerignore`는 `.gitignore` 와 비슷한 역할을 한다고 이해하면 됩니다. (git add 할 때 제외할 수 있듯이, docker build 할 때 자동으로 제외)\n\n더 많은 정보는 [Docker 공식 문서](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/)에서 확인하실 수 있습니다.\n\n### ENTRYPOINT vs CMD\n\n`ENTRYPOINT` 와 `CMD` 는 모두 컨테이너의 실행 시점에서 어떤 명령어를 실행시키고 싶을 때 사용합니다.\n그리고 이 둘 중 하나는 반드시 존재해야 합니다.\n\n- **차이점**\n - `CMD`: docker run 을 수행할 때, 쉽게 변경하여 사용할 수 있음\n - `ENTRYPOINT`: `--entrypoint` 를 사용해야 변경할 수 있음\n\n`ENTRYPOINT` 와 `CMD` 가 함께 쓰일 때는 보통 `CMD`는 `ENTRYPOINT` 에서 적은 명령의 arguments(parameters) 를 의미합니다.\n\n예를 들어서 다음과 같은 `Dockerfile` 이 있습니다.\n\n```docker\nFROM ubuntu:latest\n\n# 아래 4 가지 option 을 바꿔가며 직접 테스트해보시면 이해하기 편합니다.\n# 단, NO ENTRYPOINT 옵션은 base image 인 ubuntu:latest 에 이미 있어서 테스트해볼 수는 없고 나머지 v2, 3, 5, 6, 8, 9, 11, 12 를 테스트해볼 수 있습니다.\n# ENTRYPOINT echo \"Hello ENTRYPOINT\"\n# ENTRYPOINT [\"echo\", \"Hello ENTRYPOINT\"]\n# CMD echo \"Hello CMD\"\n# CMD [\"echo\", \"Hello CMD\"]\n```\n\n위의 `Dockerfile`에서 주석으로 표시된 부분들을 해제하며 빌드하고 실행하면 다음과 같은 결과를 얻을 수 있습니다.\n\n| | No ENTRYPOINT | ENTRYPOINT a b | ENTRYPOINT [\"a\", \"b\"] |\n| ------------------ | -------------- | -------------- | --------------------- |\n| **NO CMD** | Error! | /bin/sh -c a b | a b |\n| **CMD [\"x\", \"y\"]** | x y | /bin/sh -c a b | a b x y |\n| **CMD x y** | /bin/sh -c x y | /bin/sh -c a b | a b /bin/sh -c x y |\n\n- In Kubernetes pod\n - `ENTRYPOINT` → command\n - `CMD` → args\n\n### Docker tag 이름 짓기\n\n도커 이미지의 tag 로 **latest 는 사용하지 않는 것을 권장**합니다. \n이유는 latest 는 default tag name 이므로 **의도치 않게 overwritten** 되는 경우가 너무 많이 발생하기 때문입니다.\n\n하나의 이미지는 하나의 태그를 가짐(**uniqueness**)을 보장해야 추후 Production 단계에서 **협업/디버깅**에 용이합니다. \n내용은 다르지만, 동일한 tag 를 사용하게 되면 추후 dangling image 로 취급되어 관리하기 어려워집니다. \ndangling image는 `docker images`에는 나오지 않지만 계속해서 저장소를 차지하고 있습니다.\n\n### ETC\n\n1. log 등의 정보는 container 내부가 아닌 곳에 따로 저장합니다.\n container 내부에서 write 한 data 는 언제든지 사라질 수 있기 때문입니다.\n2. secret 한 정보, 환경(dev/prod) dependent 한 정보 등은 Dockerfile 에 직접 적는 게 아니라, env var 또는 .env config file 을 사용합니다.\n3. Dockerfile **linter** 도 존재하므로, 협업 시에는 활용하면 좋습니다.\n [https://github.com/hadolint/hadolint](https://github.com/hadolint/hadolint)\n\n## docker run 의 다양한 옵션\n\n### docker run with volume\n\nDocker container 사용 시 불편한 점이 있습니다.\n바로 Docker는 기본적으로 Docker **container 내부에서 작업한 모든 사항은 저장되지 않습니다.**\n이유는 Docker container 는 각각 격리된 파일시스템을 사용합니다. 따라서, **여러 docker container 끼리 데이터를 공유하기 어렵습니다.**\n\n이 문제를 해결하기 위해서 Docker에서 제공하는 방식은 **2 가지**가 있습니다.\n\n\n\n#### Docker volume\n\n- docker cli 를 사용해 `volume` 이라는 리소스를 직접 관리\n- host 에서 Docker area(`/var/lib/docker`) 아래에 특정 디렉토리를 생성한 다음, 해당 경로를 docker container 에 mount\n\n#### Bind mount\n\n- host 의 특정 경로를 docker container 에 mount\n\n#### How to use?\n\n사용 방식은 **동일한 인터페이스**로 `-v` 옵션을 통해 사용할 수 있습니다. \n다만, volume 을 사용할 때에는 `docker volume create`, `docker volume ls`, `docker volume rm` 등을 수행하여 직접 관리해주어야 합니다.\n\n- Docker volume\n\n ```bash\n docker run \\\n -v my_volume:/app \\\n nginx:latest\n ````\n\n- Blind mount\n\n ```bash\n docker run \\\n -v /home/user/some/path:/app \\\n nginx:latest\n ```\n\n로컬에서 개발할 때는 bind mount 가 편하긴 하지만, 환경을 깔끔하게 유지하고 싶다면 docker volume 을 사용하여 create, rm 을 명시적으로 수행하는 것도 하나의 방법입니다.\n\n쿠버네티스에서 스토리지를 제공하는 방식도 결국 docker 의 bind mount 를 활용하여 제공합니다.\n\n### docker run with resource limit\n\n기본적으로 docker container 는 **host OS 의 cpu, memory 자원을 fully 사용**할 수 있습니다. 하지만 이렇게 사용하게 되면 host OS 의 자원 상황에 따라서 **OOM** 등의 이슈로 docker container 가 비정상적으로 종료되는 상황이 발생할 수 있습니다. \n이런 문제를 다루기 위해 **docker container 실행 시, cpu 와 memory 의 사용량 제한**을 걸 수 있는 `-m` [옵션](https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory)을 제공합니다.\n\n```bash\ndocker run -d -m 512m --memory-reservation=256m --name 512-limit ubuntu sleep 3600\ndocker run -d -m 1g --memory-reservation=256m --name 1g-limit ubuntu sleep 3600\n```\n\n위의 도커를 실행 후 `docker stats` 커맨드를 통해 사용량을 확인할 수 있습니다.\n\n```bash\nCONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS\n4ea1258e2e09 1g-limit 0.00% 300KiB / 1GiB 0.03% 1kB / 0B 0B / 0B 1\n4edf94b9a3e5 512-limit 0.00% 296KiB / 512MiB 0.06% 1.11kB / 0B 0B / 0B 1\n```\n\n쿠버네티스에서 pod 라는 리소스에 cpu, memory 제한을 줄 때, 이 방식을 활용하여 제공합니다.\n\n### docker run with restart policy\n\n특정 컨테이너가 계속해서 running 상태를 유지시켜야 하는 경우가 존재합니다. 이런 경우를 위해서 해당 컨테이너가 종료되자마자 바로 재생성을 시도할 수 있는 `--restart=always` 옵션을 제공하고 있습니다.\n\n옵션 입력 후 도커를 실행합니다.\n\n```bash\ndocker run --restart=always ubuntu\n```\n\n`watch -n1 docker ps`를 통해 재실행이 되고 있는지 확인합니다.\n정상적으로 수행되고 있다면 다음과 같이 STATUS에 `Restarting (0)` 이 출력됩니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\na911850276e8 ubuntu \"bash\" 35 seconds ago Restarting (0) 6 seconds ago hungry_vaughan\n```\n\n- [https://docs.docker.com/engine/reference/commandline/run/#restart-policies---restart](https://docs.docker.com/engine/reference/commandline/run/#restart-policies---restart)\n - on-failure with max retries\n - always 등의 선택지 제공\n\n쿠버네티스에서 job 이라는 resource 의 restart 옵션을 줄 때, 이 방식을 활용하여 제공합니다.\n\n### docker run as a background process\n\n도커 컨테이너를 실행할 때는 기본적으로 foreground process 로 실행됩니다. 즉, 컨테이너를 실행한 터미널이 해당 컨테이너에 자동으로 attach 되어 있어, 다른 명령을 실행할 수 없습니다.\n\n다음과 같은 예시를 수행해봅니다. \n우선 터미널 2 개를 열어, 하나의 터미널에서는 `docker ps` 를 지켜보고, 다른 하나의 터미널에서는 다음과 같은 명령을 차례로 실행해보며 동작을 지켜봅니다.\n\n#### First Practice\n\n```bash\ndocker run -it ubuntu sleep 10\n```\n\n10 초동안 멈춰 있어야 하고, 해당 컨테이너에서 다른 명령을 수행할 수 없습니다. 10초 뒤에는 docker ps 에서 container 가 종료되는 것을 확인할 수 있습니다.\n\n#### Second Practice\n\n```bash\ndocker run -it ubuntu sleep 10\n```\n\n이후, `ctrl + p` -> `ctrl + q`\n\n해당 터미널에서 이제 다른 명령을 수행할 수 있게 되었으며, docker ps 로도 10초까지는 해당 컨테이너가 살아있는 것을 확인할 수 있습니다.\n이렇게 docker container 내부에서 빠져나온 상황을 detached 라고 부릅니다.\n도커에서는 run 을 실행함과 동시에 detached mode 로 실행시킬 수 있는 옵션을 제공합니다.\n\n#### Third Practice\n\n```bash\ndocker run -d ubuntu sleep 10\n```\n\ndetached mode 이므로 해당 명령을 실행시킨 터미널에서 다른 액션을 수행시킬 수 있습니다.\n\n상황에 따라 detached mode 를 적절히 활용하면 좋습니다. \n예를 들어, DB 와 통신하는 Backend API server 를 개발할 때 Backend API server 는 source code 를 변경시켜가면서 hot-loading 으로 계속해서 로그를 확인해봐야 하지만, DB 는 로그를 지켜볼 필요는 없는 경우라면 다음과 같이 실행할 수 있습니다. \nDB 는 docker container 를 detached mode 로 실행시키고, Backend API server 는 attached mode 로 log 를 following 하면서 실행시키면 효율적입니다.\n\n## References\n\n- [https://towardsdatascience.com/docker-storage-598e385f4efe](https://towardsdatascience.com/docker-storage-598e385f4efe)\n- [https://vsupalov.com/docker-latest-tag/](https://vsupalov.com/docker-latest-tag/)\n- [https://docs.microsoft.com/ko-kr/azure/container-registry/container-registry-image-tag-version](https://docs.microsoft.com/ko-kr/azure/container-registry/container-registry-image-tag-version)\n- [https://stevelasker.blog/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/](https://stevelasker.blog/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/)\n"
},
{
"path": "versioned_docs/version-1.0/prerequisites/docker/command.md",
"content": "---\ntitle : \"[Practice] Docker command\"\ndescription: \"Practice to use docker command.\"\nsidebar_position: 4\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n## 1. 정상 설치 확인\n\n```bash\ndocker run hello-world\n```\n\n정상적으로 설치된 경우 다음과 같은 메시지를 확인할 수 있습니다.\n\n```bash\nHello from Docker!\nThis message shows that your installation appears to be working correctly.\n....\n```\n\n**(For ubuntu)** sudo 없이 사용하고 싶다면 아래 사이트를 참고합니다.\n\n- [https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user)\n\n## 2. Docker Pull\n\ndocker image registry(도커 이미지를 저장하고 공유할 수 있는 저장소)로부터 Docker image 를 로컬에 다운로드 받는 커맨드입니다.\n\n아래 커맨드를 통해 docker pull에서 사용 가능한 argument들을 확인할 수 있습니다.\n\n```bash\ndocker pull --help\n```\n\n정상적으로 수행되면 아래와 같이 출력됩니다.\n\n```bash\nUsage: docker pull [OPTIONS] NAME[:TAG|@DIGEST]\n\nPull an image or a repository from a registry\n\nOptions:\n -a, --all-tags Download all tagged images in the repository\n --disable-content-trust Skip image verification (default true)\n --platform string Set platform if server is multi-platform capable\n -q, --quiet Suppress verbose output\n```\n\n여기서 알 수 있는 것은 바로 docker pull은 두 개 타입의 argument를 받는다는 것을 알 수 있습니다.\n\n1. `[OPTIONS]`\n2. `NAME[:TAG|@DIGEST]`\n\nhelp에서 나온 `-a`, -`q` 옵션을 사용하기 위해서는 NAME 앞에서 사용해야 합니다.\n\n직접 `ubuntu:18.04` 이미지를 pull 해보겠습니다.\n\n```bash\ndocker pull ubuntu:18.04\n```\n\n위 명령어를 해석하면 `ubuntu` 라는 이름을 가진 이미지 중 `18.04` 태그가 달려있는 이미지를 가져오라는 뜻입니다.\n\n만약, 정상적으로 수행된다면 다음과 비슷하게 출력됩니다.\n\n```bash\n18.04: Pulling from library/ubuntu\n20d796c36622: Pull complete \nDigest: sha256:42cd9143b6060261187a72716906187294b8b66653b50d70bc7a90ccade5c984\nStatus: Downloaded newer image for ubuntu:18.04\ndocker.io/library/ubuntu:18.04\n```\n\n위의 명령어를 수행하면 [docker.io/library](http://docker.io/library/) 라는 이름의 registry 에서 ubuntu:18.04 라는 image 를 여러분의 노트북에 다운로드 받게됩니다.\n\n- 참고사항\n - 추후 [docker.io](http://docker.io) 나 public 한 docker hub 와 같은 registry 대신에, 특정 **private** 한 registry 에서 docker image 를 가져와야 하는 경우에는, [`docker login`](https://docs.docker.com/engine/reference/commandline/login/) 을 통해서 특정 registry 를 바라보도록 한 뒤, docker pull 을 수행하는 형태로 사용합니다. 혹은 insecure registry 를 설정하는 [방안](https://stackoverflow.com/questions/42211380/add-insecure-registry-to-docker)도 활용할 수 있습니다.\n - 폐쇄망에서 docker image 를 `.tar` 파일과 같은 형태로 저장하고 공유할 수 있도록 [`docker save`](https://docs.docker.com/engine/reference/commandline/save/), [`docker load`](https://docs.docker.com/engine/reference/commandline/load/) 와 같은 명령어도 존재합니다.\n\n## 3. Docker images\n\n로컬에 존재하는 docker image 리스트를 출력하는 커맨드입니다.\n\n```bash\ndocker images --help\n```\n\ndocker images에서 사용할 수 있는 argument는 다음과 같습니다.\n\n```bash\nUsage: docker images [OPTIONS] [REPOSITORY[:TAG]]\n\nList images\n\nOptions:\n -a, --all Show all images (default hides intermediate images)\n --digests Show digests\n -f, --filter filter Filter output based on conditions provided\n --format string Pretty-print images using a Go template\n --no-trunc Don't truncate output\n -q, --quiet Only show image IDs\n```\n\n아래 명령어를 이용해 직접 실행해 보겠습니다.\n\n```bash\ndocker images\n```\n\n만약 도커를 최초 설치 후 이 실습을 진행한다면 다음과 비슷하게 출력됩니다.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\n줄 수 있는 argument중 `-q`를 사용하면 `IMAGE ID` 만 출력됩니다.\n\n```bash\ndocker images -q\n```\n\n```bash\n29e70752d7b2\n```\n\n## 4. Docker ps\n\n현재 실행 중인 도커 컨테이너 리스트를 출력하는 커맨드입니다.\n\n```bash\ndocker ps --help\n```\n\ndocker ps에서 사용할 수 있는 argument는 다음과 같습니다.\n\n```bash\nUsage: docker ps [OPTIONS]\n\nList containers\n\nOptions:\n -a, --all Show all containers (default shows just running)\n -f, --filter filter Filter output based on conditions provided\n --format string Pretty-print containers using a Go template\n -n, --last int Show n last created containers (includes all states) (default -1)\n -l, --latest Show the latest created container (includes all states)\n --no-trunc Don't truncate output\n -q, --quiet Only display container IDs\n -s, --size Display total file sizes\n```\n\n아래 명령어를 이용해 직접 실행해 보겠습니다.\n\n```bash\ndocker ps\n```\n\n현재 실행 중인 컨테이너가 없다면 다음과 같이 나옵니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n```\n\n만약 실행되는 컨테이너가 있다면 다음과 비슷하게 나옵니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nc1e8f5e89d8d ubuntu \"sleep 3600\" 13 seconds ago Up 12 seconds trusting_newton\n```\n\n## 5. Docker run\n\n도커 컨테이너를 실행시키는 커맨드입니다.\n\n```bash\ndocker run --help\n```\n\ndocker run을 실행하는 명령어는 다음과 같습니다.\n\n```bash\nUsage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]\n\nRun a command in a new container\n```\n\n여기서 우리가 확인해야 하는 것은 바로 docker run은 세 개 타입의 argument를 받는다는 것을 알 수 있습니다.\n\n1. `[OPTIONS]`\n2. `[COMMAND]`\n3. `[ARG...]`\n\n직접 도커 컨테이너를 실행해 보겠습니다.\n\n```bash\n## Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]\ndocker run -it --name demo1 ubuntu:18.04 /bin/bash\n```\n\n- `-it` : `-i` 옵션 + `-t` 옵션\n - container 를 실행시킴과 동시에 interactive 한 terminal 로 접속시켜주는 옵션\n- `--name` : name\n - 컨테이너 id 대신, 구분하기 쉽도록 지정해주는 이름\n- `/bin/bash`\n - 컨테이너를 실행시킴과 동시에 실행할 커맨드로, `/bin/bash` 는 bash 쉘을 여는 것을 의미합니다.\n\n실행 후 `exit` 명령어를 통해 컨테이너를 종료합니다.\n\n이 제 앞서 배웠던 `docker ps` 명령어를 치면 다음과 같이 나옵니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n```\n\n실행되고 있는 컨테이너가 나온다고 했지만 어째서인지 방금 실행한 컨테이너가 보이지 않습니다.\n그 이유는 `docker ps`는 기본값으로 현재 실행 중인 컨테이너를 보여주기 때문입니다.\n\n만약 종료된 컨테이너들도 보고 싶다면 `-a` 옵션을 주어야 합니다.\n\n```bash\ndocker ps -a\n```\n\n그러면 다음과 같이 종료된 컨테이너 목록도 나옵니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 2 minutes ago Exited (0) 2 minutes ago demo1\n```\n\n## 6. Docker exec\n\nDocker 컨테이너 내부에서 명령을 내리거나, 내부로 접속하는 커맨드입니다.\n\n```bash\ndocker exec --help\n```\n\n예를 들어서 다음과 같은 명령어를 실행해 보겠습니다.\n\n```bash\ndocker run -d --name demo2 ubuntu:18.04 sleep 3600\n```\n\n여기서 `-d` 옵션은 도커 컨테이너를 백그라운드에서 실행시켜서, 컨테이너에서 접속 종료를 하더라도, 계속 실행 중이 되도록 하는 커맨드입니다.\n\n`docker ps`를 통해 현재 실행중인지 확인합니다.\n\n다음과 같이 실행 중임을 확인할 수 있습니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 4 seconds ago Up 3 seconds demo2\n```\n\n이제 `docker exec` 명령어를 통해서 실행중인 도커 컨테이너에 접속해 보겠습니다.\n\n```bash\ndocker exec -it demo2 /bin/bash\n```\n\n이 전의 `docker run`과 동일하게 container 내부에 접속할 수 있습니다.\n\n`exit`을 통해 종료합니다.\n\n## 7. Docker logs\n\n도커 컨테이너의 log를 확인하는 커맨드 입니다.\n\n```bash\ndocker logs --help\n```\n\n다음과 같은 컨테이너를 실행시키도록 하겠습니다.\n\n```bash\ndocker run --name demo3 -d busybox sh -c \"while true; do $(echo date); sleep 1; done\"\n```\n\n위 명령어를 통해서 test 라는 이름의 busybox 컨테이너를 백그라운드에서 도커 컨테이너로 실행하여, 1초에 한 번씩 현재 시간을 출력하도록 했습니다.\n\n이제 아래 명령어를 통해 log를 확인해 보겠습니다.\n\n```bash\ndocker logs demo3\n```\n\n정상적으로 수행되면 아래와 비슷하게 나옵니다.\n\n```bash\nSun Mar 6 11:06:49 UTC 2022\nSun Mar 6 11:06:50 UTC 2022\nSun Mar 6 11:06:51 UTC 2022\nSun Mar 6 11:06:52 UTC 2022\nSun Mar 6 11:06:53 UTC 2022\nSun Mar 6 11:06:54 UTC 2022\n```\n\n그런데 이렇게 사용할 경우 여태까지 찍힌 log 밖에 확인할 수 없습니다. \n이 때 `-f` 옵션을 이용해 계속 watch 하며 출력할 수 있습니다.\n\n```bash\ndocker logs demo3 -f \n```\n\n## 8. Docker stop\n\n실행 중인 도커 컨테이너를 중단시키는 커맨드입니다.\n\n```bash\ndocker stop --help\n```\n\n`docker ps`를 통해 현재 실행 중인 컨테이너를 확인하면 다음과 같습니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" About a minute ago Up About a minute demo3\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 4 minutes ago Up 4 minutes demo2\n```\n\n이제 `docker stop` 을 통해 도커를 정지해 보겠습니다.\n\n```bash\ndocker stop demo2\n```\n\n실행 후 `docker ps`를 다시 입력합니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" 2 minutes ago Up 2 minutes demo3\n```\n\n위의 결과와 비교했을 때 demo2 컨테이너가 현재 실행 중인 컨테이너 목록에서 사라진 것을 확인할 수 있습니다.\n\n나머지 컨테이너도 정지합니다.\n\n```bash\ndocker stop demo3\n```\n\n## 9. Docker rm\n\n도커 컨테이너를 삭제하는 커맨드입니다.\n\n```bash\ndocker rm --help\n```\n\n도커 컨테이너는 기본적으로 종료가 된 상태로 있습니다. 그래서 `docker ps -a`를 통해서 종료된 컨테이너도 볼 수 있습니다.\n그런데 종료된 컨테이너는 왜 지워야 할까요? \n종료되어 있는 도커에는 이전에 사용한 데이터가 아직 컨테이너 내부에 남아있습니다.\n그래서 restart 등을 통해서 컨테이너를 재시작할 수 있습니다.\n그런데 이 과정에서 disk를 사용하게 됩니다.\n\n그래서 완전히 사용하지 않는 컨테이너를 지우기 위해서는 `docker rm` 명령어를 사용해야 합니다.\n\n우선 현재 컨테이너들을 확인합니다.\n\n```bash\ndocker ps -a\n```\n\n다음과 같이 3개의 컨테이너가 있습니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n730391669c39 busybox \"sh -c 'while true; …\" 4 minutes ago Exited (137) About a minute ago demo3\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 7 minutes ago Exited (137) 2 minutes ago demo2\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 10 minutes ago Exited (0) 10 minutes ago demo1\n```\n\n아래 명령어를 통해 `demo3` 컨테이너를 삭제해 보겠습니다.\n\n```bash\ndocker rm demo3\n```\n\n`docker ps -a` 명령어를 치면 다음과 같이 2개로 줄었습니다.\n\n```bash\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nfc88a83e90f0 ubuntu:18.04 \"sleep 3600\" 13 minutes ago Exited (137) 8 minutes ago demo2\n4c1aa74a382a ubuntu:18.04 \"/bin/bash\" 16 minutes ago Exited (0) 16 minutes ago demo1\n```\n\n나머지 컨테이너들도 삭제합니다.\n\n```bash\ndocker rm demo2\ndocker rm demo1\n```\n\n## 10. Docker rmi\n\n도커 이미지를 삭제하는 커맨드입니다.\n\n```bash\ndocker rmi --help\n```\n\n아래 명령어를 통해 현재 어떤 이미지들이 로컬에 있는지 확인합니다.\n\n```bash\ndocker images\n```\n\n다음과 같이 출력됩니다.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nbusybox latest a8440bba1bc0 32 hours ago 1.41MB\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\n`busybox` 이미지를 삭제해 보겠습니다.\n\n```bash\ndocker rmi busybox\n```\n\n다시 `docker images`를 칠 경우 다음과 같이 나옵니다.\n\n```bash\nREPOSITORY TAG IMAGE ID CREATED SIZE\nubuntu 18.04 29e70752d7b2 2 days ago 56.7MB\n```\n\n## References\n\n- [https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry)\n"
},
{
"path": "versioned_docs/version-1.0/prerequisites/docker/docker.md",
"content": "---\ntitle : \"What is Docker?\"\ndescription: \"Introduction to Docker.\"\nsidebar_position: 3\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n\n## 컨테이너\n\n- 컨테이너 가상화\n - 어플리케이션을 어디에서나 동일하게 실행하는 기술\n- 컨테이너 이미지\n - 어플리케이션을 실행시키기 위해 필요한 모든 파일들의 집합\n - → 붕어빵 틀\n- 컨테이너란?\n - 컨테이너 이미지를 기반으로 실행된 한 개의 프로세스\n - → 붕어빵 틀로 찍어낸 붕어빵\n\n## 도커\n\n도커는 **컨테이너를 관리**하고 사용할 수 있게 해주는 플랫폼입니다. \n이러한 도커의 슬로건은 바로 **Build Once, Run Anywhere** 로 어디에서나 동일한 실행 결과를 보장합니다.\n\n도커 내부에서 동작하는 과정을 보자면 실제로 container 를 위한 리소스를 분리하고, lifecycle 을 제어하는 기능은 linux kernel 의 cgroup 등이 수행합니다.\n하지만 이러한 인터페이스를 바로 사용하는 것은 **너무 어렵기 때문에** 다음과 같은 추상화 layer를 만들게 됩니다.\n\n\n\n이를 통해 사용자는 사용자 친화적인 API 인 **Docker CLI** 만으로 쉽게 컨테이너를 제어할 수 있습니다.\n\n## Layer 해석\n\n위에서 나온 layer들의 역할은 다음과 같습니다.\n\n1. runC: linux kernel 의 기능을 직접 사용해서, container 라는 하나의 프로세스가 사용할 네임스페이스와 cpu, memory, filesystem 등을 격리시켜주는 기능을 수행합니다.\n2. containerd: runC(OCI layer) 에게 명령을 내리기 위한 추상화 단계이며, 표준화된 인터페이스(OCI)를 사용합니다.\n3. dockerd: containerd 에게 명령을 내리는 역할만 합니다.\n4. docker cli: 사용자는 docker cli 로 dockerd (Docker daemon)에게 명령을 내리기만 하면 됩니다.\n - 이 통신 과정에서 unix socket 을 사용하기 때문에 가끔 도커 관련 에러가 나면 `/var/run/docker.sock` 가 사용 중이다, 권한이 없다 등등의 에러 메시지가 나오는 것입니다.\n\n이처럼 도커는 많은 단계를 감싸고 있지만, 흔히 도커라는 용어를 사용할 때는 Docker CLI 를 말할 때도 있고, Dockerd 를 말할 때도 있고 Docker Container 하나를 말할 때도 있어서 혼란이 생길 수 있습니다. \n앞으로 나오는 글에서도 도커가 여러가지 의미로 쓰일 수 있습니다.\n\n## For ML Engineer\n\n머신러닝 엔지니어가 도커를 사용하는 이유는 다음과 같습니다.\n\n1. 나의 ML 학습/추론 코드를 OS, python version, python 환경, 특정 python package 버전에 independent 하도록 해야 한다.\n2. 그래서 코드 뿐만이 아닌 **해당 코드가 실행되기 위해 필요한 모든 종속적인 패키지, 환경 변수, 폴더명 등등을 하나의 패키지로** 묶을 수 있는 기술이 컨테이너화 기술이다.\n3. 이 기술을 쉽게 사용하고 관리할 수 있는 소프트웨어 중 하나가 도커이며, 패키지를 도커 이미지라고 부른다.\n"
},
{
"path": "versioned_docs/version-1.0/prerequisites/docker/images.md",
"content": "---\ntitle : \"[Practice] Docker images\"\ndescription: \"Practice to use docker image.\"\nsidebar_position: 5\ncontributors: [\"Jongseob Jeon\", \"Jaeyeon Kim\"]\n---\n\n## 1. Dockerfile 만들기\n\n도커 이미지를 만드는 가장 쉬운 방법은 도커에서 제공하는 템플릿인 Dockerfile을 사용하는 것입니다. \n이외에는 running container 를 docker image 로 만드는 `docker commit` 등을 활용하는 방법이 있습니다.\n\n- `Dockerfile`\n - 사용자가 도커 이미지를 쉽게 만들 수 있도록, 제공하는 템플릿\n - 파일명은 꼭 `Dockerfile` 이 아니어도 상관없지만, `docker build` 수행 시, default 로 사용하는 파일명이 `Dockerfile` 입니다.\n - 도커 이미지를 만드는 `docker build` 를 수행할 때, `-f` 옵션을 주면 다른 파일명으로도 사용 가능합니다.\n - ex) `docker build -f dockerfile-asdf .` 도 가능\n\n1. 실습을 위해서 편한 디렉토리로 이동합니다.\n\n ```bash\n cd