![](\"images/thumbnail-1-01.jpg\")
\n\n\n\n\n\n## 1.2 Environment preparation\n\n### 1.2.1 GitHub Codespaces\n\n ![](\"images/thumbnail-1-02-1.jpg\")
\n\n\n\nOR\n\n### 1.2.2 VM in AWS\n\n**Note**: You don't have to rent an instance in the cloud. You can follow the same instructions \nfor setting up your local environment. \n\n\n ![](\"images/thumbnail-1-02.jpg\")
\n\n\n\nCode:\n\nRecommended development environment: Linux\n\n### Step 1: Download and install the Anaconda distribution of Python\n```sh\nwget https://repo.anaconda.com/archive/Anaconda3-2022.05-Linux-x86_64.sh\nbash Anaconda3-2022.05-Linux-x86_64.sh\n```\n\n### Step 2: Update existing packages\n\n```sh\nsudo apt update\n```\n\n### Step 3: Install Docker and Docker Compose\nFollow the instructions here:\n[install-using-the-repository](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) \nSet up Docker's apt repository.\n```sh\n# Add Docker's official GPG key:\nsudo apt-get update\nsudo apt-get install ca-certificates curl\nsudo install -m 0755 -d /etc/apt/keyrings\nsudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc\nsudo chmod a+r /etc/apt/keyrings/docker.asc\n\n# Add the repository to Apt sources:\necho \\\n \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \\\n $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable\" | \\\n sudo tee /etc/apt/sources.list.d/docker.list > /dev/null\nsudo apt-get update\n```\nInstall the Docker packages.\n```sh\nsudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin\n```\nTo run docker without `sudo`:\n\n```sh\nsudo groupadd docker\nsudo usermod -aG docker $USER\n```\n\n### Step 4: Run Docker\n\n```sh\ndocker run hello-world\n```\n\nIf you get `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create\": dial unix /var/run/docker.sock: connect: permission denied.` error, restart your VM instance, or run:\n`sudo dockerd`\n\n**Note**: If you get `It is required that your private key files are NOT accessible by others. This private key will be ignored.` error, you should change permits on the downloaded file to protect your private key:\n\n ```sh\nchmod 400 name-of-your-private-key-file.pem\n```\n\n## 1.3 (Optional) Training a ride duration prediction model\n\n**Note**: The NYC taxi data is now in parquet format, not CSV.\nHere's a [video](https://www.youtube.com/watch?v=r94QjpX9vSE&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK) that explains how to \nread parquet data.\n\n\n ![](\"images/thumbnail-1-03.jpg\")
\n\n\nLinks:\n\n* [Notebook](duration-prediction.ipynb)\n\n\n## 1.4 Course overview\n\n\n ![](\"images/thumbnail-1-04.jpg\")
\n\n\n\n\n## 1.5 MLOps maturity model\n\n\n ![](\"images/thumbnail-1-05.jpg\")
\n\n\nLinks: \n\n* [MLOps Maturity model](https://docs.microsoft.com/en-us/azure/architecture/example-scenario/mlops/mlops-maturity-model)\n\n\n\n## 1.6 Homework\n\nMore information [here](../cohorts/2025/01-intro/homework.md).\n\n\n## Notes\n\nDid you take notes? Add them here:\n\n* [Local MacOS envprep by Adetbekov](https://github.com/adetbekov/mlops-zoomcamp/blob/main/01-intro/macos-local-envprep.md)\n* [Environment Setup by Ayoub](https://github.com/ayoub-berdeddouch/mlops-journey/blob/main/intro-01.md)\n* [Intro, Environment Setup, and MLOps Maturity Models by Bala](https://github.com/balapriyac/DTC-MLOps-Zoomcamp/tree/main/week1)\n* [GCP Environment Setup by Piyush](https://github.com/piyush-an/MLOps-ZoomCamp/blob/main/01-Introduction/infrastructure.md)\n* [Microsoft Azure Environment Setup by Olaide](https://github.com/josepholaide/MLOps-Practice/blob/main/Week%201/README.md)\n* [Environment Preparation using GCP and pyenv by Dani](https://github.com/syahrulhamdani/dtc-mlops/blob/main/week-1-introduction/README.md)\n* [Useful links by Zioalex](https://github.com/zioalex/mlops-zoomcamp/blob/main/My_notes_week1.md)\n* [Notes by Alvaro Navas](https://github.com/ziritrion/mlopszoomcamp/blob/main/notes/1_intro.md)\n * [Environment setup on GCP (recycled from Data Engineering Zoomcamp)](https://gist.github.com/ziritrion/3214aa570e15ae09bf72c4587cb9d686)\n * [Docker cheatsheet](https://gist.github.com/ziritrion/1842c8a4c4851602a8733bba19ab6050)\n * [Conda cheatsheet](https://gist.github.com/ziritrion/8024025672ea92b8bdeb320d6015aa0d)\n * [Git cheatsheet](https://gist.github.com/ziritrion/d73ca65bf4d19c79ca842a55853cb962)\n * [Python and data libraries cheatseet](https://gist.github.com/ziritrion/9b80e47956adc0f20ecce209d494cd0a)\n* [Cloud server setup using Deploifai (on AWS, Azure, or GCP) by 98sean98](https://github.com/98sean98/mlops-zoomcamp/blob/main/01-intro/deploifai-server/readme.md)\n* [Notes from first lesson by Neimv](https://gitlab.com/neimv/mlops/-/blob/main/lessons_weeks/notes_1.md)\n* [Course preliminaries (jupyterbook) by particle1331](https://particle1331.github.io/ok-transformer/nb/mlops/01-intro.html)\n* [Notes by Francisco Delca (environment: local ubuntu + virtualvenv)](https://github.com/FDelca/mlops_datatalks_notes/blob/main/Week1/Week1-LearningNotes.ipynb)\n* [What is MLOps? - Non-technical intro by Lorenz](https://github.com/LoHertel/Road-to-MLOps/blob/main/01-primer/README.md)\n* [Notes for Week1 by Bhagabat](https://github.com/BPrasad123/MLOps_Zoomcamp/tree/main/Week1)\n* [What is MLOps? - Storytime by Pathik Ghugare](https://github.com/pathikg/MLOps-Zoomcamp-DataTalks/blob/main/week-01-introduction/MLOps-Introduction.md)\n* [Module 1 Notes by Stephen Barrie](https://stephen137.github.io/posts/MLOps_Zoomcamp_Week_1/MLOps_Zoomcamp_Week_1.html)\n* [Module 1 Notes - MacOS + Pycharm by Hongfan (Amber)](https://github.com/Muhongfan/MLops/blob/main/01-intro/README.md)\n* [Comprehensive guide to MLOps: Theory & Concepts for Beginners by Nithish Prabhu](https://ntp3105.github.io/Comprehensive-MLOps/Week-1/Introduction%20to%20MLOps.html)\n* [Module 1 Notes -Video 1.2 Env Prep Ec2 troubleshooting notes for remote-ssh vscode to ec2](https://github.com/thatwonguy/mlops-zoomcamp/blob/personal/01-intro/README.md)\n* [Ch1 notes by Marcus](https://github.com/mleiwe/mlops-zoomcamp/blob/Ch1_Marcus/cohorts/2024/01-intro/Ch1_Notes.md)\n* [GCP setup instructions](https://github.com/mleiwe/mlops-zoomcamp/blob/Ch1_Marcus/cohorts/2024/01-intro/GoogleCloudSetUpNotes.md)\n* [Machine Learning Essentials: FAQs on Linear Regression, One-Hot Encoding & RMSE](https://github.com/niting9881/course-mlops-zoomcamp/blob/main/01-intro/README.md)\n* [GCP Environment Setup with Terraform + Cloud-Init by Artem](https://github.com/Ladn0/mlops-zoomcamp/tree/main/01-intro/notes)\n* [Week 1 - local setup hints by hannarud](https://github.com/hannarud/mlops-zoomcamp-2025/blob/main/week1_notes.md)\n* [Week-1 - Introduction to MLOps, Env Setup,Notebooks detailed Notes by Muhammad Shifa](https://github.com/MuhammadShifa/mlops-zoomcamp2025/tree/main/01-introduction#readme)\n* Send a PR, add your notes above this line\n"
},
{
"path": "01-intro/duration-prediction.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"id\": \"b2bd82d7\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Python 3.9.7\\r\\n\"\n ]\n }\n ],\n \"source\": [\n \"!python -V\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"id\": \"41062d8a\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import pandas as pd\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"id\": \"c984c564\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import pickle\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"id\": \"4add538c\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import seaborn as sns\\n\",\n \"import matplotlib.pyplot as plt\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"id\": \"8b135c2b\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from sklearn.feature_extraction import DictVectorizer\\n\",\n \"from sklearn.linear_model import LinearRegression\\n\",\n \"from sklearn.linear_model import Lasso\\n\",\n \"from sklearn.linear_model import Ridge\\n\",\n \"\\n\",\n \"from sklearn.metrics import root_mean_squared_error\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 6,\n \"id\": \"8e013caf\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df = pd.read_parquet('./data/green_tripdata_2021-01.parquet')\\n\",\n \"\\n\",\n \"df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\\n\",\n \"df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)\\n\",\n \"\\n\",\n \"df = df[(df.duration >= 1) & (df.duration <= 60)]\\n\",\n \"\\n\",\n \"categorical = ['PULocationID', 'DOLocationID']\\n\",\n \"numerical = ['trip_distance']\\n\",\n \"\\n\",\n \"df[categorical] = df[categorical].astype(str)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 7,\n \"id\": \"116ff0cc\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"9.775464208836793\"\n ]\n },\n \"execution_count\": 7,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"train_dicts = df[categorical + numerical].to_dict(orient='records')\\n\",\n \"\\n\",\n \"dv = DictVectorizer()\\n\",\n \"X_train = dv.fit_transform(train_dicts)\\n\",\n \"\\n\",\n \"target = 'duration'\\n\",\n \"y_train = df[target].values\\n\",\n \"\\n\",\n \"lr = LinearRegression()\\n\",\n \"lr.fit(X_train, y_train)\\n\",\n \"\\n\",\n \"y_pred = lr.predict(X_train)\\n\",\n \"\\n\",\n \"root_mean_squared_error(y_train, y_pred)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 8,\n \"id\": \"b134c97b\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"/home/ubuntu/anaconda3/lib/python3.9/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).\\n\",\n \" warnings.warn(msg, FutureWarning)\\n\",\n \"/home/ubuntu/anaconda3/lib/python3.9/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).\\n\",\n \" warnings.warn(msg, FutureWarning)\\n\"\n ]\n },\n {\n \"data\": {\n \"text/plain\": [\n \"![](\"images/thumbnail-2-01.jpg\")
\n\n\n\n\n## 2.2 Getting started with MLflow\n\n\n ![](\"images/thumbnail-2-02.jpg\")
\n\n\nNote: in the videos, Cristian uses Jupyter in VS code and runs everything locally\n\nBut if you set up a VM in the previous module, you can keep using it\nand use the usual Jupyter from your browser. There's no significant\ndifference between using Jupyter with VS code and without\n\n\n## 2.3 Experiment tracking with MLflow\n\n\n ![](\"images/thumbnail-2-03.jpg\")
\n\n\n\n\n## 2.4 Model management\n\n\n ![](\"images/thumbnail-2-04.jpg\")
\n\n\n\n\n## 2.5 Model registry\n\n\n ![](\"images/thumbnail-2-05.jpg\")
\n\n\n> **Starting MLflow 2.9, model registry stages are deprecated.**\nPlease use model version tags and aliases instead of stages. For example, instead of `transition_model_version_stage(name, version, stage)` use `set_registered_model_alias(name, alias, version)`. More details [here](https://github.com/mlflow/mlflow/issues/10336) and [here](https://mlflow.org/docs/latest/model-registry.html).\n\n## 2.6 MLflow in practice\n\n\n ![](\"images/thumbnail-2-06.jpg\")
\n\n\n\n## 2.7 MLflow: benefits, limitations and alternatives\n\n\n ![](\"images/thumbnail-2-07.jpg\")
\n\n\n\n## 2.7 Homework\n\nMore information [here](../cohorts/2025/02-experiment-tracking/homework.md).\n\n\n## Notes\n\nDid you take notes? Add them here:\n\n* [Notes/General Docs on MLflow by Ayoub](https://gist.github.com/Qfl3x/ccff6b0708358c040e437d52af0c2e43)\n* [Minimalist MLflow code reference by Anna V](https://github.com/annnvv/mlops_zoomcamp/blob/main/notes/module2_notes_MLflow.md)\n* [Notes from second lesson by Neimv](https://gitlab.com/neimv/mlops/-/blob/main/lessons_weeks/notes_2.md)\n* [2nd Week Experiment & Tracking notes by Ayoub.B](https://github.com/ayoub-berdeddouch/mlops-journey/blob/main/experiment_tracking_02.md)\n* [Experiment tracking (jupyterbook) by particle1331](https://particle1331.github.io/ok-transformer/nb/mlops/03-mlflow.html)\n* [Week 2: Experiment & Tracking Notes by Bengsoon Chuah](https://github.com/bengsoon/mlops-zoomcamp/blob/main/02-experiment-tracking/notes/Experiment_Tracking_notes.md)\n* [2.4 Model Management Notes by Alvaro Pena](https://github.com/alvarofps/mlops-zoomcamp/blob/main/02-experiment-tracking/my-notes/2.4%20Model%20management.md)\n* [Notes by Alvaro Navas](https://github.com/ziritrion/mlopszoomcamp/blob/main/notes/2_experiment.md)\n* [Notebook from froukje](https://github.com/froukje/ml-ops-zoomcamp/blob/master/02-experiment-tracking/week02.ipynb) and [notes](https://medium.com/@falbrechtg/getting-started-with-mlflow-tracking-46a0089d6a73)\n* [Blog post on setting up MLFlow on GCP by Isaac Kargar](https://kargarisaac.github.io/blog/mlops/data%20engineering/2022/06/15/MLFlow-on-GCP.html).\n* [Week2: Experiment tracking notes and notebook by Bhagabat](https://github.com/BPrasad123/MLOps_Zoomcamp/tree/main/Week2)\n* [Notes of ML-flow by Jaime Cabrera-Salcedo](https://github.com/jaimeh94/MLOps-Zoomcamp/tree/main/02-experiment-tracking)\n* [Experiment tracking with MLflow by Hongfan (Amber)](https://github.com/Muhongfan/MLops/blob/main/02-experiment-tracking/README.md)\n* [Running MLflow with Docker and on Minikube](https://open.substack.com/pub/asfandqazi/p/mlflow-on-minikube?r=2o17tf&utm_campaign=post&utm_medium=web)\n* [Chapter 2: Full Notes by Marcus](https://github.com/mleiwe/mlops-zoomcamp/blob/Ch2_Marcus/cohorts/2024/02-experiment-tracking/Ch2_notes.md)\n* [Experiemnt Tracking and Mlflow by Annaliese Tech](https://github.com/AnnalieseTech/MLOPS_ZOOMCAMP/blob/main/02_EXPERIMENT_TRACKING/EXPERIMENT_TRACKING_NOTES.md)\n* [Adding Hyperparameter Tuning to Your Notebook with MLflow and Hyperopt by Annaliese Tech](https://github.com/AnnalieseTech/MLOPS_ZOOMCAMP/blob/main/02_EXPERIMENT_TRACKING/Hyperparameter-Tuning.md)\n* [MLFlow setup and Experiment Tracking by Hokfu](https://github.com/Hokfu/MLOps_Zoomcamp_Study/blob/main/02-experiment-tracking/README.md)\n* [2025 Cohort | Notes on MLflow & Hyperopt by Gabi Fonseca](https://github.com/fonsecagabriella/ml_ops/blob/main/02_experiment_tracking/__notes.md)\n* [Homework 2 Experiment and Tracking article by Srikanth Ganji](https://medium.com/@srikanth.unix07/mlops-zoomcamp-2025-homework-2-experiment-tracking-with-mlflow-4ea1ed783531)\n* [Week-2 - Detailed Notes of MLFlow, Experiment Tracking, Notebooks, Homework by Muhammad Shifa](https://github.com/MuhammadShifa/mlops-zoomcamp2025/blob/main/02-experiment-tracking/README.md)\n* Send a PR, add your notes above this line\n"
},
{
"path": "02-experiment-tracking/duration-prediction.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"id\": \"b2bd82d7\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Python 3.9.12\\n\"\n ]\n }\n ],\n \"source\": [\n \"!python -V\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"id\": \"41062d8a\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import pandas as pd\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"id\": \"c984c564\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import pickle\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"id\": \"4add538c\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import seaborn as sns\\n\",\n \"import matplotlib.pyplot as plt\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"id\": \"8b135c2b\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from sklearn.feature_extraction import DictVectorizer\\n\",\n \"from sklearn.linear_model import LinearRegression\\n\",\n \"from sklearn.linear_model import Lasso\\n\",\n \"from sklearn.linear_model import Ridge\\n\",\n \"\\n\",\n \"from sklearn.metrics import mean_squared_error\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 44,\n \"id\": \"1464985f\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"![](\"images/ec2_os.png\")
\n\n![](\"images/ec2_instance_type.png\")
\n\nYou'll also need to create a new key pair so later you can connect to the new instance using SSH. Click on \"Create new key pair\" and complete the details like in the image below:\n\n![](\"images/key_pair.png\")
\n\nSelect the new key pair and then click on \"Launch Instance\".\n\n![](\"images/select_key_pair.png\")
\n\nFinally, you have to edit the security group so the EC2 instance accepts SSH (port 22) and HTTP connections (port 5000):\n\n![](\"images/security_group.png\")
\n\n3. Create an s3 bucket to be used as the artifact store.\n\nGo to s3 and click on \"Create bucket\". Fill in the bucket name as in the image below and let all the other configurations with their default values.\n\n![](\"images/s3_bucket.png\")
\n\nNote: s3 bucket names must be unique across all AWS account in all the AWS Regions within a partition, that means that once a bucket is created, the name of that bucket cannot be used by another AWS account within the same region. If you get an error saying that the bucket name was already taken you can fix it easily by just changing the name to something like `mlflow-artifacts-remote-2` or another name.\n\n4. Create a new PostgreSQL database to be used as the backend store\n\nGo to the RDS Console and click on \"Create database\". Make sure to select \"PostgreSQL\" engine type and the \"Free tier\" template.\n\n![](\"images/postgresql.png\")
\n\nSelect a name for your DB instance, set the master username as \"mlflow\" and tick the option \"Auto generate a password\" so Amazon RDS generate a password automatically.\n\n![](\"images/db_settings.png\")
\n\nFinally, on the section \"Additional configuration\" specify a database name so RDS automatically creates an initial database for you.\n\n![](\"images/db_configuration.png\")
\n\nAfter clicking on \"launch database\" you will be able to check the newly generated password, but take into account that the automatically generated password will be shown only once!\n\n![](\"images/db_password.png\")
\n\nYou can use the default values for all the other configurations.\n\nTake note of the following information:\n\n* master username\n* password \n* initial database name\n* endpoint\n\nOnce the DB instance is created, go to the RDS console, select the new db and under \"Connectivity & security\" select the VPC security group. Modify the security group by adding a new inbound rule that allows postgreSQL connections on the port 5432 from the security group of the EC2 instance. This way, the server will be able to connect to the postgres database.\n\nNote that the database name might be set to \"postgres\" by default.\n![](\"images/postgresql_inbound_rule.png\")
\n\n5. Connect to the EC2 instance and launch the tracking server.\n\nGo to the EC2 Console and find the instance launched on the step 2. Click on \"Connect\" and then follow the steps described in the tab \"SSH\". \n\nRun the following commands to install the dependencies, configure the environment and launch the server:\n* `sudo yum update`\n* `pip3 install mlflow boto3 psycopg2-binary`\n* `aws configure` # you'll need to input your AWS credentials here\n* `mlflow server -h 0.0.0.0 -p 5000 --backend-store-uri postgresql://DB_USER:DB_PASSWORD@DB_ENDPOINT:5432/DB_NAME --default-artifact-root s3://S3_BUCKET_NAME`\n\nNote: before launching the server, check that the instance can access the s3 bucket created in the step number 3. To do that, just run this command from the EC2 instance: `aws s3 ls`. You should see the bucket listed in the result.\n\n6. Access the remote tracking server from your local machine.\n\nOpen a new tab on your web browser and go to this address: `http://![](\"images/thumbnail-4-01.jpg\")
\n\n\n\n\n## 4.2 Web-services: Deploying models with Flask and Docker\n\n\n ![](\"images/thumbnail-4-02.jpg\")
\n\n\n\n[See code here](web-service/)\n\n\n## 4.3 Web-services: Getting the models from the model registry (MLflow)\n\n\n ![](\"images/thumbnail-4-03.jpg\")
\n\n\n\n[See code here](web-service-mlflow/)\n\n\n## 4.4 (Optional) Streaming: Deploying models with Kinesis and Lambda \n\n\n ![](\"images/thumbnail-4-04.jpg\")
\n\n\nNote: Since some of the steps in this video requires the use of AWS services which incur some cost on the user, it is optional to code along to this video. However, as material in Module 6 is based on the content of this video, we still highly recommended that you watch it.\n\n[See code here](streaming/)\n\n\n## 4.5 Batch: Preparing a scoring script\n\n\n ![](\"images/thumbnail-4-05.jpg\")
\n\n\n\n[See code here](batch/)\n\n\n## 4.6 MLOps Zoomcamp 4.6 - Batch scoring with Mage\n\nNo video - you already know how to do it:\n\n* Connect to MLFlow\n* Create a transformation block\n* Get the model from the registry, apply it\n\n\n## Homework\n\nMore information [here](../cohorts/2025/04-deployment/homework.md).\n\n\n## Notes\n\nDid you take notes? Add them here:\n\n* [Notes on model deployment (+ creating a modeling package) by Ron M.](https://particle1331.github.io/inefficient-networks/notebooks/mlops/04-deployment/notes.html)\n* [Notes on Model Deployment using Google Cloud Platform, by M. Ayoub C.](https://gist.github.com/Qfl3x/de2a9b98a370749a4b17a4c94ef46185)\n* [Week4: Notes on Model Deployment by Bhagabat](https://github.com/BPrasad123/MLOps_Zoomcamp/tree/main/Week4)\n* [Week 4: Deployment notes by Ayoub.B](https://github.com/ayoub-berdeddouch/mlops-journey/blob/main/deployment-04.md)\n* [Week 4: Deployment notes by Waleed](https://github.com/waleedayoub/mlops-zoomcamp/blob/main/cohorts/2023/04-deployment/module4notes.waleed.md)\n* [Week4: Deployment: Offline (Batch), Online (Web service /w MLflow, Streaming) by Hongfan (Amber)](https://github.com/Muhongfan/MLops/blob/main/04-deployment/README.md)\n* [Week 4: Deployment Notes - Marcus](https://github.com/mleiwe/mlops-zoomcamp/blob/NotesBranch/cohorts/2024/04-deployment/Ch4_Notes_ML.md)\n* [Cohort 2025| ML model deployment notes by Nitin Gupta](https://github.com/niting9881/course-mlops-zoomcamp/blob/main/04-deployment/README.md)\n* [week-4: Detailed notes on deployment,streaming module,notes,codes and homework by Muhammad Shifa](https://github.com/MuhammadShifa/mlops-zoomcamp2025/blob/main/04-deployment/README.md)\n* Send a PR, add your notes above this line\n"
},
{
"path": "04-deployment/batch/Pipfile",
"content": "[[source]]\nurl = \"https://pypi.org/simple\"\nverify_ssl = true\nname = \"pypi\"\n\n[packages]\nscikit-learn = \"==1.0.2\"\nprefect = \"==2.0b6\"\nmlflow = \"*\"\npandas = \"*\"\nboto3 = \"*\"\npyarrow = \"*\"\ns3fs = \"*\"\n\n[dev-packages]\n\n[requires]\npython_version = \"3.9\"\n"
},
{
"path": "04-deployment/batch/README.md",
"content": "## Batch deployment\n\n* Turn the notebook for training a model into a notebook for applying the model\n* Turn the notebook into a script \n* Clean it and parametrize\n"
},
{
"path": "04-deployment/batch/score.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"id\": \"2e25cf66\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"import uuid\\n\",\n \"import pickle\\n\",\n \"\\n\",\n \"import pandas as pd\\n\",\n \"\\n\",\n \"import mlflow\\n\",\n \"\\n\",\n \"from sklearn.feature_extraction import DictVectorizer\\n\",\n \"from sklearn.ensemble import RandomForestRegressor\\n\",\n \"from sklearn.metrics import mean_squared_error\\n\",\n \"from sklearn.pipeline import make_pipeline\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 7,\n \"id\": \"5887deea\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"year = 2021\\n\",\n \"month = 2\\n\",\n \"taxi_type = 'green'\\n\",\n \"\\n\",\n \"input_file = f'https://s3.amazonaws.com/nyc-tlc/trip+data/{taxi_type}_tripdata_{year:04d}-{month:02d}.parquet'\\n\",\n \"output_file = f'output/{taxi_type}/{year:04d}-{month:02d}.parquet'\\n\",\n \"\\n\",\n \"RUN_ID = os.getenv('RUN_ID', 'e1efc53e9bd149078b0c12aeaa6365df')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 8,\n \"id\": \"b9666e19\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def generate_uuids(n):\\n\",\n \" ride_ids = []\\n\",\n \" for i in range(n):\\n\",\n \" ride_ids.append(str(uuid.uuid4()))\\n\",\n \" return ride_ids\\n\",\n \"\\n\",\n \"def read_dataframe(filename: str):\\n\",\n \" df = pd.read_parquet(filename)\\n\",\n \"\\n\",\n \" df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\\n\",\n \" df.duration = df.duration.dt.total_seconds() / 60\\n\",\n \" df = df[(df.duration >= 1) & (df.duration <= 60)]\\n\",\n \" \\n\",\n \" df['ride_id'] = generate_uuids(len(df))\\n\",\n \"\\n\",\n \" return df\\n\",\n \"\\n\",\n \"\\n\",\n \"def prepare_dictionaries(df: pd.DataFrame):\\n\",\n \" categorical = ['PULocationID', 'DOLocationID']\\n\",\n \" df[categorical] = df[categorical].astype(str)\\n\",\n \" \\n\",\n \" df['PU_DO'] = df['PULocationID'] + '_' + df['DOLocationID']\\n\",\n \"\\n\",\n \" categorical = ['PU_DO']\\n\",\n \" numerical = ['trip_distance']\\n\",\n \" dicts = df[categorical + numerical].to_dict(orient='records')\\n\",\n \" return dicts\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 9,\n \"id\": \"6b5f0d80\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def load_model(run_id):\\n\",\n \" logged_model = f's3://mlflow-models-alexey/1/{RUN_ID}/artifacts/model'\\n\",\n \" model = mlflow.pyfunc.load_model(logged_model)\\n\",\n \" return model\\n\",\n \"\\n\",\n \"\\n\",\n \"def apply_model(input_file, run_id, output_file):\\n\",\n \"\\n\",\n \" df = read_dataframe(input_file)\\n\",\n \" dicts = prepare_dictionaries(df)\\n\",\n \"\\n\",\n \" \\n\",\n \" model = load_model(run_id)\\n\",\n \" y_pred = model.predict(dicts)\\n\",\n \"\\n\",\n \" df_result = pd.DataFrame()\\n\",\n \" df_result['ride_id'] = df['ride_id']\\n\",\n \" df_result['lpep_pickup_datetime'] = df['lpep_pickup_datetime']\\n\",\n \" df_result['PULocationID'] = df['PULocationID']\\n\",\n \" df_result['DOLocationID'] = df['DOLocationID']\\n\",\n \" df_result['actual_duration'] = df['duration']\\n\",\n \" df_result['predicted_duration'] = y_pred\\n\",\n \" df_result['diff'] = df_result['actual_duration'] - df_result['predicted_duration']\\n\",\n \" df_result['model_version'] = run_id\\n\",\n \" \\n\",\n \" df_result.to_parquet(output_file, index=False)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 11,\n \"id\": \"cc2899e7\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"apply_model(input_file=input_file, run_id=RUN_ID, output_file=output_file)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 12,\n \"id\": \"b75bd6c9\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"2021-02.parquet 2021-03.parquet\\r\\n\"\n ]\n }\n ],\n \"source\": [\n \"!ls output/green/\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"289d4bc4\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.7\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
},
{
"path": "04-deployment/batch/score.py",
"content": "#!/usr/bin/env python\n# coding: utf-8\n\nimport os\nimport sys\n\nimport uuid\nimport pickle\n\nfrom datetime import datetime\n\nimport pandas as pd\n\nimport mlflow\n\nfrom prefect import task, flow, get_run_logger\nfrom prefect.context import get_run_context\n\nfrom dateutil.relativedelta import relativedelta\n\nfrom sklearn.feature_extraction import DictVectorizer\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.pipeline import make_pipeline\n\n\ndef generate_uuids(n):\n ride_ids = []\n for i in range(n):\n ride_ids.append(str(uuid.uuid4()))\n return ride_ids\n\n\ndef read_dataframe(filename: str):\n df = pd.read_parquet(filename)\n\n df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\n df.duration = df.duration.dt.total_seconds() / 60\n df = df[(df.duration >= 1) & (df.duration <= 60)]\n \n df['ride_id'] = generate_uuids(len(df))\n\n return df\n\n\ndef prepare_dictionaries(df: pd.DataFrame):\n categorical = ['PULocationID', 'DOLocationID']\n df[categorical] = df[categorical].astype(str)\n \n df['PU_DO'] = df['PULocationID'] + '_' + df['DOLocationID']\n\n categorical = ['PU_DO']\n numerical = ['trip_distance']\n dicts = df[categorical + numerical].to_dict(orient='records')\n return dicts\n\n\ndef load_model(run_id):\n logged_model = f's3://mlflow-models-alexey/1/{run_id}/artifacts/model'\n model = mlflow.pyfunc.load_model(logged_model)\n return model\n\n\ndef save_results(df, y_pred, run_id, output_file):\n df_result = pd.DataFrame()\n df_result['ride_id'] = df['ride_id']\n df_result['lpep_pickup_datetime'] = df['lpep_pickup_datetime']\n df_result['PULocationID'] = df['PULocationID']\n df_result['DOLocationID'] = df['DOLocationID']\n df_result['actual_duration'] = df['duration']\n df_result['predicted_duration'] = y_pred\n df_result['diff'] = df_result['actual_duration'] - df_result['predicted_duration']\n df_result['model_version'] = run_id\n\n df_result.to_parquet(output_file, index=False)\n\n\n@task\ndef apply_model(input_file, run_id, output_file):\n logger = get_run_logger()\n\n logger.info(f'reading the data from {input_file}...')\n df = read_dataframe(input_file)\n dicts = prepare_dictionaries(df)\n\n logger.info(f'loading the model with RUN_ID={run_id}...')\n model = load_model(run_id)\n\n logger.info(f'applying the model...')\n y_pred = model.predict(dicts)\n\n logger.info(f'saving the result to {output_file}...')\n\n save_results(df, y_pred, run_id, output_file)\n return output_file\n\n\ndef get_paths(run_date, taxi_type, run_id):\n prev_month = run_date - relativedelta(months=1)\n year = prev_month.year\n month = prev_month.month \n\n input_file = f's3://nyc-tlc/trip data/{taxi_type}_tripdata_{year:04d}-{month:02d}.parquet'\n output_file = f's3://nyc-duration-prediction-alexey/taxi_type={taxi_type}/year={year:04d}/month={month:02d}/{run_id}.parquet'\n\n return input_file, output_file\n\n\n@flow\ndef ride_duration_prediction(\n taxi_type: str,\n run_id: str,\n run_date: datetime = None):\n if run_date is None:\n ctx = get_run_context()\n run_date = ctx.flow_run.expected_start_time\n \n input_file, output_file = get_paths(run_date, taxi_type, run_id)\n\n apply_model(\n input_file=input_file,\n run_id=run_id,\n output_file=output_file\n )\n\n\ndef run():\n taxi_type = sys.argv[1] # 'green'\n year = int(sys.argv[2]) # 2021\n month = int(sys.argv[3]) # 3\n\n run_id = sys.argv[4] # 'e1efc53e9bd149078b0c12aeaa6365df'\n\n ride_duration_prediction(\n taxi_type=taxi_type,\n run_id=run_id,\n run_date=datetime(year=year, month=month, day=1)\n )\n\n\nif __name__ == '__main__':\n run()\n\n\n\n\n"
},
{
"path": "04-deployment/batch/score_backfill.py",
"content": "from datetime import datetime\nfrom dateutil.relativedelta import relativedelta\n\nfrom prefect import flow\n\nimport score\n\n\n@flow\ndef ride_duration_prediction_backfill():\n start_date = datetime(year=2021, month=3, day=1)\n end_date = datetime(year=2022, month=4, day=1)\n\n d = start_date\n\n while d <= end_date:\n score.ride_duration_prediction(\n taxi_type='green',\n run_id='e1efc53e9bd149078b0c12aeaa6365df',\n run_date=d\n )\n\n d = d + relativedelta(months=1)\n\n\nif __name__ == '__main__':\n ride_duration_prediction_backfill()"
},
{
"path": "04-deployment/batch/score_deploy.py",
"content": "from prefect.deployments import Deployment\nfrom prefect.orion.schemas.schedules import CronSchedule\nfrom score import ride_duration_prediction\n\ndeployment = Deployment.build_from_flow(\n flow=ride_duration_prediction,\n name=\"ride_duration_prediction\",\n parameters={\n \"taxi_type\": \"green\",\n \"run_id\": \"e1efc53e9bd149078b0c12aeaa6365df\",\n },\n schedule=CronSchedule(cron=\"0 3 2 * *\"),\n work_queue_name=\"ml\",\n)\n\ndeployment.apply()\n"
},
{
"path": "04-deployment/load_model.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"id\": \"d84f7be2\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import mlflow\\n\",\n \"\\n\",\n \"TRACKING_URL = \\\"http://127.0.0.1:5000\\\"\\n\",\n \"mlflow.set_tracking_uri(TRACKING_URL)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"id\": \"293ac71d\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"run_id = '6dd459b11b4e48dc862f4e1019d166f6'\\n\",\n \"logged_model = f'runs:/{run_id}/model'\\n\",\n \"\\n\",\n \"loaded_model = mlflow.pyfunc.load_model(logged_model)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"id\": \"031645d6\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import mlflow\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"id\": \"e5e6825e\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"loaded_model = mlflow.pyfunc.load_model('s3://mlflow-models-alexey/1/6dd459b11b4e48dc862f4e1019d166f6/artifacts/model/')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"64f389c2\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"id\": \"aaf7ad20\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"request = {\\n\",\n \" 'lpep_pickup_datetime': '2021-01-01 00:15:56',\\n\",\n \" 'PULocationID': 43,\\n\",\n \" 'DOLocationID': 151,\\n\",\n \" 'passenger_count': 1.0,\\n\",\n \" 'trip_distance': 1.01\\n\",\n \"}\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"id\": \"6256f61d\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"features = {}\\n\",\n \"features['PU_DO'] = '%s_%s' % (request['PULocationID'], request['DOLocationID'])\\n\",\n \"features['trip_distance'] = request['trip_distance']\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"id\": \"0b3a195c\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"array([14.8285286])\"\n ]\n },\n \"execution_count\": 5,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"loaded_model.predict(features)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"6fce5f72\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
},
{
"path": "04-deployment/meta.json",
"content": "{\n \"module\": {\n \"number\": 4,\n \"title\": \"Model Deployment\"\n },\n \"units\": [\n {\n \"number\": 1,\n \"title\": \"Three ways of deploying a model\",\n \"youtube\": \"https://www.youtube.com/watch?v=JMGe4yIoBRA\"\n },\n {\n \"number\": 2,\n \"title\": \"Web-services: Deploying models with Flask and Docker\",\n \"youtube\": \"https://www.youtube.com/watch?v=D7wfMAdgdF8\"\n },\n {\n \"number\": 3,\n \"title\": \"Web-services: Getting the models from the model registry (MLflow)\",\n \"youtube\": \"https://www.youtube.com/watch?v=aewOpHSCkqI\"\n },\n {\n \"number\": 4,\n \"title\": \"(Optional) Streaming: Deploying models with Kinesis and Lambda \",\n \"youtube\": \"https://www.youtube.com/watch?v=TCqr9HNcrsI\"\n },\n {\n \"number\": 5,\n \"title\": \"Batch: Preparing a scoring script\",\n \"youtube\": \"https://www.youtube.com/watch?v=18Lbaaeigek\"\n },\n {\n \"number\": 6,\n \"title\": \"MLOps Zoomcamp 4.6 - Batch: Scheduling batch scoring jobs with Prefect\",\n \"youtube\": \"https://www.youtube.com/watch?v=ekT_JW213Tc\"\n },\n {\n \"number\": 7,\n \"title\": \"Homework\",\n \"youtube\": \"\"\n }\n ]\n}"
},
{
"path": "04-deployment/streaming/Dockerfile",
"content": "FROM public.ecr.aws/lambda/python:3.9\n\nRUN pip install -U pip\nRUN pip install pipenv \n\nCOPY [ \"Pipfile\", \"Pipfile.lock\", \"./\" ]\n\nRUN pipenv install --system --deploy\n\nCOPY [ \"lambda_function.py\", \"./\" ]\n\nCMD [ \"lambda_function.lambda_handler\" ]\n"
},
{
"path": "04-deployment/streaming/Pipfile",
"content": "[[source]]\nurl = \"https://pypi.org/simple\"\nverify_ssl = true\nname = \"pypi\"\n\n[packages]\nboto3 = \"*\"\nmlflow = \"*\"\nscikit-learn = \"==1.0.2\"\n\n[dev-packages]\n\n[requires]\npython_version = \"3.9\"\n"
},
{
"path": "04-deployment/streaming/README.md",
"content": "## Machine Learning for Streaming\n\n* Scenario\n* Creating the role \n* Create a Lambda function, test it\n* Create a Kinesis stream\n* Connect the function to the stream\n* Send the records \n\nLinks\n\n* [Tutorial: Using Amazon Lambda with Amazon Kinesis](https://docs.amazonaws.cn/en_us/lambda/latest/dg/with-kinesis-example.html)\n\n## Code snippets\n\n### Sending data\n\n\n```bash\nKINESIS_STREAM_INPUT=ride_events\naws kinesis put-record \\\n --stream-name ${KINESIS_STREAM_INPUT} \\\n --partition-key 1 \\\n --data \"Hello, this is a test.\"\n```\n\nDecoding base64\n\n```python\nbase64.b64decode(data_encoded).decode('utf-8')\n```\n\nRecord example\n\n```json\n{\n \"ride\": {\n \"PULocationID\": 130,\n \"DOLocationID\": 205,\n \"trip_distance\": 3.66\n }, \n \"ride_id\": 123\n}\n```\n\nSending this record\n\n```bash\naws kinesis put-record \\\n --stream-name ${KINESIS_STREAM_INPUT} \\\n --partition-key 1 \\\n --data '{\n \"ride\": {\n \"PULocationID\": 130,\n \"DOLocationID\": 205,\n \"trip_distance\": 3.66\n }, \n \"ride_id\": 156\n }'\n```\n\n### Test event\n\n\n```json\n{\n \"Records\": [\n {\n \"kinesis\": {\n \"kinesisSchemaVersion\": \"1.0\",\n \"partitionKey\": \"1\",\n \"sequenceNumber\": \"49630081666084879290581185630324770398608704880802529282\",\n \"data\": \"ewogICAgICAgICJyaWRlIjogewogICAgICAgICAgICAiUFVMb2NhdGlvbklEIjogMTMwLAogICAgICAgICAgICAiRE9Mb2NhdGlvbklEIjogMjA1LAogICAgICAgICAgICAidHJpcF9kaXN0YW5jZSI6IDMuNjYKICAgICAgICB9LCAKICAgICAgICAicmlkZV9pZCI6IDI1NgogICAgfQ==\",\n \"approximateArrivalTimestamp\": 1654161514.132\n },\n \"eventSource\": \"aws:kinesis\",\n \"eventVersion\": \"1.0\",\n \"eventID\": \"shardId-000000000000:49630081666084879290581185630324770398608704880802529282\",\n \"eventName\": \"aws:kinesis:record\",\n \"invokeIdentityArn\": \"arn:aws:iam::XXXXXXXXX:role/lambda-kinesis-role\",\n \"awsRegion\": \"eu-west-1\",\n \"eventSourceARN\": \"arn:aws:kinesis:eu-west-1:XXXXXXXXX:stream/ride_events\"\n }\n ]\n}\n```\n\n### Reading from the stream\n\n```bash\nKINESIS_STREAM_OUTPUT='ride_predictions'\nSHARD='shardId-000000000000'\n\nSHARD_ITERATOR=$(aws kinesis \\\n get-shard-iterator \\\n --shard-id ${SHARD} \\\n --shard-iterator-type TRIM_HORIZON \\\n --stream-name ${KINESIS_STREAM_OUTPUT} \\\n --query 'ShardIterator' \\\n)\n\nRESULT=$(aws kinesis get-records --shard-iterator $SHARD_ITERATOR)\n\necho ${RESULT} | jq -r '.Records[0].Data' | base64 --decode\n``` \n\n\n### Running the test\n\n```bash\nexport PREDICTIONS_STREAM_NAME=\"ride_predictions\"\nexport RUN_ID=\"e1efc53e9bd149078b0c12aeaa6365df\"\nexport TEST_RUN=\"True\"\n\npython test.py\n```\n\n### Putting everything to Docker\n\n```bash\ndocker build -t stream-model-duration:v1 .\n\ndocker run -it --rm \\\n -p 8080:8080 \\\n -e PREDICTIONS_STREAM_NAME=\"ride_predictions\" \\\n -e RUN_ID=\"e1efc53e9bd149078b0c12aeaa6365df\" \\\n -e TEST_RUN=\"True\" \\\n -e AWS_DEFAULT_REGION=\"eu-west-1\" \\\n stream-model-duration:v1\n```\n\nURL for testing:\n\n* http://localhost:8080/2015-03-31/functions/function/invocations\n\n\n\n### Configuring AWS CLI to run in Docker\n\nTo use AWS CLI, you may need to set the env variables:\n\n```bash\ndocker run -it --rm \\\n -p 8080:8080 \\\n -e PREDICTIONS_STREAM_NAME=\"ride_predictions\" \\\n -e RUN_ID=\"e1efc53e9bd149078b0c12aeaa6365df\" \\\n -e TEST_RUN=\"True\" \\\n -e AWS_ACCESS_KEY_ID=\"${AWS_ACCESS_KEY_ID}\" \\\n -e AWS_SECRET_ACCESS_KEY=\"${AWS_SECRET_ACCESS_KEY}\" \\\n -e AWS_DEFAULT_REGION=\"${AWS_DEFAULT_REGION}\" \\\n stream-model-duration:v1\n```\n\nAlternatively, you can mount the `.aws` folder with your credentials to the `.aws` folder in the container:\n\n```bash\ndocker run -it --rm \\\n -p 8080:8080 \\\n -e PREDICTIONS_STREAM_NAME=\"ride_predictions\" \\\n -e RUN_ID=\"e1efc53e9bd149078b0c12aeaa6365df\" \\\n -e TEST_RUN=\"True\" \\\n -v c:/Users/alexe/.aws:/root/.aws \\\n stream-model-duration:v1\n```\n\n### Publishing Docker images\n\nCreating an ECR repo\n\n```bash\naws ecr create-repository --repository-name duration-model\n```\n\nLogging in\n\n```bash\n$(aws ecr get-login --no-include-email)\n```\n\nPushing \n\n```bash\nREMOTE_URI=\"387546586013.dkr.ecr.eu-west-1.amazonaws.com/duration-model\"\nREMOTE_TAG=\"v1\"\nREMOTE_IMAGE=${REMOTE_URI}:${REMOTE_TAG}\n\nLOCAL_IMAGE=\"stream-model-duration:v1\"\ndocker tag ${LOCAL_IMAGE} ${REMOTE_IMAGE}\ndocker push ${REMOTE_IMAGE}\n```\n"
},
{
"path": "04-deployment/streaming/lambda_function.py",
"content": "import os\nimport json\nimport boto3\nimport base64\n\nimport mlflow\n\nkinesis_client = boto3.client('kinesis')\n\nPREDICTIONS_STREAM_NAME = os.getenv('PREDICTIONS_STREAM_NAME', 'ride_predictions')\n\n\nRUN_ID = os.getenv('RUN_ID')\n\nlogged_model = f's3://mlflow-models-alexey/1/{RUN_ID}/artifacts/model'\n# logged_model = f'runs:/{RUN_ID}/model'\nmodel = mlflow.pyfunc.load_model(logged_model)\n\n\nTEST_RUN = os.getenv('TEST_RUN', 'False') == 'True'\n\ndef prepare_features(ride):\n features = {}\n features['PU_DO'] = '%s_%s' % (ride['PULocationID'], ride['DOLocationID'])\n features['trip_distance'] = ride['trip_distance']\n return features\n\n\ndef predict(features):\n pred = model.predict(features)\n return float(pred[0])\n\n\ndef lambda_handler(event, context):\n # print(json.dumps(event))\n \n predictions_events = []\n \n for record in event['Records']:\n encoded_data = record['kinesis']['data']\n decoded_data = base64.b64decode(encoded_data).decode('utf-8')\n ride_event = json.loads(decoded_data)\n\n # print(ride_event)\n ride = ride_event['ride']\n ride_id = ride_event['ride_id']\n \n features = prepare_features(ride)\n prediction = predict(features)\n \n prediction_event = {\n 'model': 'ride_duration_prediction_model',\n 'version': '123',\n 'prediction': {\n 'ride_duration': prediction,\n 'ride_id': ride_id \n }\n }\n\n if not TEST_RUN:\n kinesis_client.put_record(\n StreamName=PREDICTIONS_STREAM_NAME,\n Data=json.dumps(prediction_event),\n PartitionKey=str(ride_id)\n )\n \n predictions_events.append(prediction_event)\n\n\n return {\n 'predictions': predictions_events\n }\n"
},
{
"path": "04-deployment/streaming/test.py",
"content": "\nimport lambda_function\n\nevent = {\n \"Records\": [\n {\n \"kinesis\": {\n \"kinesisSchemaVersion\": \"1.0\",\n \"partitionKey\": \"1\",\n \"sequenceNumber\": \"49630081666084879290581185630324770398608704880802529282\",\n \"data\": \"ewogICAgICAgICJyaWRlIjogewogICAgICAgICAgICAiUFVMb2NhdGlvbklEIjogMTMwLAogICAgICAgICAgICAiRE9Mb2NhdGlvbklEIjogMjA1LAogICAgICAgICAgICAidHJpcF9kaXN0YW5jZSI6IDMuNjYKICAgICAgICB9LCAKICAgICAgICAicmlkZV9pZCI6IDI1NgogICAgfQ==\",\n \"approximateArrivalTimestamp\": 1654161514.132\n },\n \"eventSource\": \"aws:kinesis\",\n \"eventVersion\": \"1.0\",\n \"eventID\": \"shardId-000000000000:49630081666084879290581185630324770398608704880802529282\",\n \"eventName\": \"aws:kinesis:record\",\n \"invokeIdentityArn\": \"arn:aws:iam::387546586013:role/lambda-kinesis-role\",\n \"awsRegion\": \"eu-west-1\",\n \"eventSourceARN\": \"arn:aws:kinesis:eu-west-1:387546586013:stream/ride_events\"\n }\n ]\n}\n\n\nresult = lambda_function.lambda_handler(event, None)\nprint(result)\n"
},
{
"path": "04-deployment/streaming/test_docker.py",
"content": "import requests \n\nevent = {\n \"Records\": [\n {\n \"kinesis\": {\n \"kinesisSchemaVersion\": \"1.0\",\n \"partitionKey\": \"1\",\n \"sequenceNumber\": \"49630081666084879290581185630324770398608704880802529282\",\n \"data\": \"ewogICAgICAgICJyaWRlIjogewogICAgICAgICAgICAiUFVMb2NhdGlvbklEIjogMTMwLAogICAgICAgICAgICAiRE9Mb2NhdGlvbklEIjogMjA1LAogICAgICAgICAgICAidHJpcF9kaXN0YW5jZSI6IDMuNjYKICAgICAgICB9LCAKICAgICAgICAicmlkZV9pZCI6IDI1NgogICAgfQ==\",\n \"approximateArrivalTimestamp\": 1654161514.132\n },\n \"eventSource\": \"aws:kinesis\",\n \"eventVersion\": \"1.0\",\n \"eventID\": \"shardId-000000000000:49630081666084879290581185630324770398608704880802529282\",\n \"eventName\": \"aws:kinesis:record\",\n \"invokeIdentityArn\": \"arn:aws:iam::387546586013:role/lambda-kinesis-role\",\n \"awsRegion\": \"eu-west-1\",\n \"eventSourceARN\": \"arn:aws:kinesis:eu-west-1:387546586013:stream/ride_events\"\n }\n ]\n}\n\n\nurl = 'http://localhost:8080/2015-03-31/functions/function/invocations'\nresponse = requests.post(url, json=event)\nprint(response.json())\n"
},
{
"path": "04-deployment/web-service/Dockerfile",
"content": "FROM python:3.9.7-slim\n\nRUN pip install -U pip\nRUN pip install pipenv \n\nWORKDIR /app\n\nCOPY [ \"Pipfile\", \"Pipfile.lock\", \"./\" ]\n\nRUN pipenv install --system --deploy\n\nCOPY [ \"predict.py\", \"lin_reg.bin\", \"./\" ]\n\nEXPOSE 9696\n\nENTRYPOINT [ \"gunicorn\", \"--bind=0.0.0.0:9696\", \"predict:app\" ]"
},
{
"path": "04-deployment/web-service/Pipfile",
"content": "[[source]]\nurl = \"https://pypi.org/simple\"\nverify_ssl = true\nname = \"pypi\"\n\n[packages]\nscikit-learn = \"==1.0.2\"\nflask = \"*\"\ngunicorn = \"*\"\n\n[dev-packages]\nrequests = \"*\"\n\n[requires]\npython_version = \"3.9\"\n"
},
{
"path": "04-deployment/web-service/README.md",
"content": "## Deploying a model as a web-service\n\n* Creating a virtual environment with Pipenv\n* Creating a script for predictiong \n* Putting the script into a Flask app\n* Packaging the app to Docker\n\n\n```bash\ndocker build -t ride-duration-prediction-service:v1 .\n```\n\n```bash\ndocker run -it --rm -p 9696:9696 ride-duration-prediction-service:v1\n```\n"
},
{
"path": "04-deployment/web-service/predict.py",
"content": "import pickle\n\nfrom flask import Flask, request, jsonify\n\nwith open('lin_reg.bin', 'rb') as f_in:\n (dv, model) = pickle.load(f_in)\n\n\ndef prepare_features(ride):\n features = {}\n features['PU_DO'] = '%s_%s' % (ride['PULocationID'], ride['DOLocationID'])\n features['trip_distance'] = ride['trip_distance']\n return features\n\n\ndef predict(features):\n X = dv.transform(features)\n preds = model.predict(X)\n return float(preds[0])\n\n\napp = Flask('duration-prediction')\n\n\n@app.route('/predict', methods=['POST'])\ndef predict_endpoint():\n ride = request.get_json()\n\n features = prepare_features(ride)\n pred = predict(features)\n\n result = {\n 'duration': pred\n }\n\n return jsonify(result)\n\n\nif __name__ == \"__main__\":\n app.run(debug=True, host='0.0.0.0', port=9696)"
},
{
"path": "04-deployment/web-service/test.py",
"content": "import requests\n\nride = {\n \"PULocationID\": 10,\n \"DOLocationID\": 50,\n \"trip_distance\": 40\n}\n\nurl = 'http://localhost:9696/predict'\nresponse = requests.post(url, json=ride)\nprint(response.json())\n"
},
{
"path": "04-deployment/web-service-mlflow/Pipfile",
"content": "[[source]]\nurl = \"https://pypi.org/simple\"\nverify_ssl = true\nname = \"pypi\"\n\n[packages]\nscikit-learn = \"==1.0.2\"\nflask = \"*\"\ngunicorn = \"*\"\nmlflow = \"*\"\nboto3 = \"*\"\n\n[dev-packages]\nrequests = \"*\"\n\n[requires]\npython_version = \"3.9\"\n"
},
{
"path": "04-deployment/web-service-mlflow/README.md",
"content": "## Getting the model for deployment from MLflow\n\n* Take the code from the previous video\n* Train another model, register with MLflow\n* Put the model into a scikit-learn pipeline\n* Model deployment with tracking server\n* Model deployment without the tracking server\n\nStarting the MLflow server with S3:\n\n```bash\nmlflow server \\\n --backend-store-uri=sqlite:///mlflow.db \\\n --default-artifact-root=s3://mlflow-models-alexey/\n```\n\nDownloading the artifact\n\n```bash\nexport MLFLOW_TRACKING_URI=\"http://127.0.0.1:5000\"\nexport MODEL_RUN_ID=\"6dd459b11b4e48dc862f4e1019d166f6\"\n\nmlflow artifacts download \\\n --run-id ${MODEL_RUN_ID} \\\n --artifact-path model \\\n --dst-path .\n```"
},
{
"path": "04-deployment/web-service-mlflow/predict.py",
"content": "import os\nimport pickle\n\nimport mlflow\nfrom flask import Flask, request, jsonify\n\n\nRUN_ID = os.getenv('RUN_ID')\n\nlogged_model = f's3://mlflow-models-alexey/1/{RUN_ID}/artifacts/model'\n# logged_model = f'runs:/{RUN_ID}/model'\nmodel = mlflow.pyfunc.load_model(logged_model)\n\n\ndef prepare_features(ride):\n features = {}\n features['PU_DO'] = '%s_%s' % (ride['PULocationID'], ride['DOLocationID'])\n features['trip_distance'] = ride['trip_distance']\n return features\n\n\ndef predict(features):\n preds = model.predict(features)\n return float(preds[0])\n\n\napp = Flask('duration-prediction')\n\n\n@app.route('/predict', methods=['POST'])\ndef predict_endpoint():\n ride = request.get_json()\n\n features = prepare_features(ride)\n pred = predict(features)\n\n result = {\n 'duration': pred,\n 'model_version': RUN_ID\n }\n\n return jsonify(result)\n\n\nif __name__ == \"__main__\":\n app.run(debug=True, host='0.0.0.0', port=9696)\n"
},
{
"path": "04-deployment/web-service-mlflow/random-forest.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"id\": \"2e25cf66\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import pickle\\n\",\n \"\\n\",\n \"import pandas as pd\\n\",\n \"\\n\",\n \"from sklearn.feature_extraction import DictVectorizer\\n\",\n \"from sklearn.ensemble import RandomForestRegressor\\n\",\n \"from sklearn.metrics import mean_squared_error\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 24,\n \"id\": \"041e9423\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from sklearn.pipeline import make_pipeline\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"id\": \"506bec39\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"2022/06/01 12:27:06 INFO mlflow.tracking.fluent: Experiment with name 'green-taxi-duration' does not exist. Creating a new experiment.\\n\"\n ]\n },\n {\n \"data\": {\n \"text/plain\": [\n \"![](\"images/thumbnail-6-1.jpg\")
\n\n\n\n### 6.2 Integration tests with docker-compose\n\n\n ![](\"images/thumbnail-6-2.jpg\")
\n\n\n\n### 6.3 Testing cloud services with LocalStack\n\n\n ![](\"images/thumbnail-6-3.jpg\")
\n\n\n\n### 6.4 Code quality: linting and formatting\n\n\n ![](\"images/thumbnail-6-4.jpg\")
\n\n\n\n### 6.5 Git pre-commit hooks\n\n\n ![](\"images/thumbnail-6-5.jpg\")
\n\n\n\n### 6.6 Makefiles and make\n\n\n ![](\"images/thumbnail-6-6.jpg\")
\n\n\n\n### 6.7 Homework\n\nMore information [here](../cohorts/2025/06-best-practices/homework.md).\n\n![](\"images/thumbnail-7-1.jpg\")
\n\n\n### Objective\n\nThe goal of this project is to apply everything we have learned\nin this course to build an end-to-end machine learning project.\n\n## Problem statement\n\nFor the project, we will ask you to build an end-to-end ML project. \n\nFor that, you will need:\n\n* Select a dataset that you're interested in (see [Datasets](#datasets))\n* Train a model on that dataset tracking your experiments\n* Create a model training pipeline\n* Deploy the model in batch, web service or streaming\n* Monitor the performance of your model\n* Follow the best practices \n\n\n## Technologies \n\nYou don't have to limit yourself to technologies covered in the course. You can use alternatives as well:\n\n* **Cloud**: AWS, GCP, Azure, ... \n* **Experiment tracking tools**: MLFlow, Weights & Biases, ... \n* **Workflow orchestration**: Prefect, Airflow, Flyte, Kubeflow, Argo, ...\n* **Monitoring**: Evidently, WhyLabs/whylogs, ...\n* **CI/CD**: Github actions, Gitlab CI/CD, ...\n* **Infrastructure as code (IaC)**: Terraform, Pulumi, Cloud Formation, ...\n\nIf you use a tool that wasn't covered in the course, be sure to explain what that tool does.\n\nIf you're not certain about some tools, ask in Slack.\n\n## Peer reviewing\n\n> [!IMPORTANT] \n> To evaluate the projects, we'll use peer reviewing. This is a great opportunity for you to learn from each other.\n> * To get points for your project, you need to evaluate 3 projects of your peers\n> * You get 3 extra points for each evaluation\n\n## Evaluation Criteria\n\n* Problem description\n * 0 points: The problem is not described\n * 1 point: The problem is described but shortly or not clearly \n * 2 points: The problem is well described and it's clear what the problem the project solves\n* Cloud\n * 0 points: Cloud is not used, things run only locally\n * 2 points: The project is developed on the cloud OR uses localstack (or similar tool) OR the project is deployed to Kubernetes or similar container management platforms\n * 4 points: The project is developed on the cloud and IaC tools are used for provisioning the infrastructure\n* Experiment tracking and model registry\n * 0 points: No experiment tracking or model registry\n * 2 points: Experiments are tracked or models are registered in the registry\n * 4 points: Both experiment tracking and model registry are used\n* Workflow orchestration\n * 0 points: No workflow orchestration\n * 2 points: Basic workflow orchestration\n * 4 points: Fully deployed workflow \n* Model deployment\n * 0 points: Model is not deployed\n * 2 points: Model is deployed but only locally\n * 4 points: The model deployment code is containerized and could be deployed to cloud or special tools for model deployment are used\n* Model monitoring\n * 0 points: No model monitoring\n * 2 points: Basic model monitoring that calculates and reports metrics\n * 4 points: Comprehensive model monitoring that sends alerts or runs a conditional workflow (e.g. retraining, generating debugging dashboard, switching to a different model) if the defined metrics threshold is violated\n* Reproducibility\n * 0 points: No instructions on how to run the code at all, the data is missing\n * 2 points: Some instructions are there, but they are not complete OR instructions are clear and complete, the code works, but the data is missing\n * 4 points: Instructions are clear, it's easy to run the code, and it works. The versions for all the dependencies are specified.\n* Best practices\n * [ ] There are unit tests (1 point)\n * [ ] There is an integration test (1 point)\n * [ ] Linter and/or code formatter are used (1 point)\n * [ ] There's a Makefile (1 point)\n * [ ] There are pre-commit hooks (1 point)\n * [ ] There's a CI/CD pipeline (2 points)\n\n> [!NOTE]\n> It's highly recommended to create a new repository for your project (not inside an existing repo) with a meaningful title, such as\n> \"Car Price Prediction\" or \"Music Genre Classification\" and include as many details as possible in the README file. ChatGPT can assist you with this. Doing so will not only make it easier to showcase your project for potential job opportunities but also have it featured on the [Projects Gallery App](#projects-gallery).\n> If you leave the README file empty or with minimal details, there may be point deductions as per the [Evaluation Criteria](#evaluation-criteria).\n\n## Resources\n\n### Datasets\n\nRefer to the provided [datasets](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/projects/datasets.md) for possible selection.\n\n### Projects Gallery\n\nExplore a collection of projects completed by members of our community. The projects cover a wide range of topics and utilize different tools and techniques. Feel free to delve into any project and see how others have tackled real-world problems with data, structured their code, and presented their findings. It's a great resource to learn and get ideas for your own projects.\n\n[](https://datatalksclub-projects.streamlit.app/)\n\n### MLOps Zoomcamp 2023\n\n* [2023 Projects](../cohorts/2023/07-project)\n\n### MLOps Zoomcamp 2022\n\n* [2022 Projects](../cohorts/2022/07-project)\n\n\n"
},
{
"path": "README.md",
"content": "\n ![\"MLOps](\"images/banner-2025.jpg\")
\n
\nMLOps (machine learning operations) is a must-know skill for many data professionals. Master the fundamentals of MLOps, from training and experimentation to deployment and monitoring.\n
\n\n\n![](\"https://user-images.githubusercontent.com/875246/185755203-17945fd1-6b64-46f2-8377-1011dcb1a444.png\")
\n
\nJoin Slack •\n#course-mlops-zoomcamp Channel •\nTelegram Announcements •\nCourse Playlist •\nFAQ •\nTweet about the Course\n
\n\n## How to Take MLOps Zoomcamp\n\n### 2025 Cohort\n- **Start Date**: May 5, 2025\n- **Register Here**: [Sign up](https://airtable.com/shrCb8y6eTbPKwSTL)\n- **Stay Updated**: Subscribe to our [Google Calendar](https://calendar.google.com/calendar/?cid=M3Jzbmg0ZDA2aHVsY2M1ZjcyNDJtODNyMTRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ) (Desktop only)\n\n### Self-Paced Learning\nAll course materials are freely available for independent study. Follow these steps:\n1. Watch the course videos.\n2. Join the [Slack community](https://datatalks.club/slack.html).\n3. Refer to the [FAQ document](https://datatalks.club/faq/mlops-zoomcamp.html) for guidance.\n\n## Syllabus\nThe course consists of structured modules, hands-on workshops, and a final project to reinforce your learning. Each module introduces core MLOps concepts and tools.\n\n### Prerequisites\nTo get the most out of this course, you should have prior experience with:\n- Python\n- Docker\n- Command line basics\n- Machine learning (e.g., through [ML Zoomcamp](https://github.com/alexeygrigorev/mlbookcamp-code/tree/master/course-zoomcamp))\n- 1+ year of programming experience\n\n## Modules\n\n### [Module 1: Introduction](01-intro)\n- What is MLOps?\n- MLOps maturity model\n- NY Taxi dataset (our running example)\n- Why MLOps is essential\n- Course structure & environment setup\n- Homework\n\n### [Module 2: Experiment Tracking & Model Management](02-experiment-tracking)\n- Introduction to experiment tracking\n- MLflow basics\n- Model saving and loading\n- Model registry\n- Hands-on MLflow exercises\n- Homework\n\n### [Module 3: Orchestration & ML Pipelines](03-orchestration)\n\n- Workflow orchestration\n- Homework\n\n### [Module 4: Model Deployment](04-deployment)\n- Deployment strategies: online (web, streaming) vs. offline (batch)\n- Deploying with Flask (web service)\n- Streaming deployment with AWS Kinesis & Lambda\n- Batch scoring for offline processing\n- Homework\n\n### [Module 5: Model Monitoring](05-monitoring)\n- Monitoring ML-based services\n- Web service monitoring with Prometheus, Evidently, and Grafana\n- Batch job monitoring with Prefect, MongoDB, and Evidently\n- Homework\n\n### [Module 6: Best Practices](06-best-practices)\n- Unit and integration testing\n- Linting, formatting, and pre-commit hooks\n- CI/CD with GitHub Actions\n- Infrastructure as Code (Terraform)\n- Homework\n\n### [Final Project](07-project/)\n- End-to-end project integrating all course concepts\n\n## Community & Support\n\n### Getting Help on Slack\n\nJoin the [`#course-mlops-zoomcamp`](https://app.slack.com/client/T01ATQK62F8/C02R98X7DS9) channel on [DataTalks.Club Slack](https://datatalks.club/slack.html) for discussions, troubleshooting, and networking.\n\nTo keep discussions organized:\n- Follow [our guidelines](asking-questions.md) when posting questions.\n- Review the [community guidelines](https://datatalks.club/slack/guidelines.html).\n\n## Instructors\n\n- [Cristian Martinez](https://www.linkedin.com/in/cristian-javier-martinez-09bb7031/)\n- [Alexey Grigorev](https://www.linkedin.com/in/agrigorev/)\n- [Emeli Dral](https://www.linkedin.com/in/emelidral/)\n\n\n## Sponsors & Supporters\n\nInterested in supporting our community? Reach out to [alexey@datatalks.club](mailto:alexey@datatalks.club).\n\n## About DataTalks.Club\n\n\n \n
\nDataTalks.Club is a global online community of data enthusiasts. It's a place to discuss data, learn, share knowledge, ask and answer questions, and support each other.\n
\n\n\nWebsite •\nJoin Slack Community •\nNewsletter •\nUpcoming Events •\nGoogle Calendar •\nYouTube •\nGitHub •\nLinkedIn •\nTwitter\n
\n\nAll the activity at DataTalks.Club mainly happens on [Slack](https://datatalks.club/slack.html). We post updates there and discuss different aspects of data, career questions, and more.\n\nAt DataTalksClub, we organize online events, community activities, and free courses. You can learn more about what we do at [DataTalksClub Community Navigation](https://www.notion.so/DataTalksClub-Community-Navigation-bf070ad27ba44bf6bbc9222082f0e5a8?pvs=21).\n" }, { "path": "after-sign-up.md", "content": "## Thank you!\n\nThanks for signining up for the course.\n\nThe process of adding you to the mailing list is not automated yet, \nbut you will hear from us closer to the course start. \n\nTo make sure you don't miss any announcements\n\n- Register in [DataTalks.Club's Slack](https://datatalks.club/slack.html) and join the [`#course-mlops-zoomcamp`](https://app.slack.com/client/T01ATQK62F8/C02R98X7DS9) channel\n- Join the [course Telegram channel with announcements](https://t.me/dtc_courses)\n- [Tweet about the course!](https://ctt.ac/fH67W)\n- Subscribe to [DataTalks.Club's YouTube channel](https://www.youtube.com/c/DataTalksClub) and check \n [the course playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK)\n- Subscribe to our [public Google Calendar](https://calendar.google.com/calendar/?cid=M3Jzbmg0ZDA2aHVsY2M1ZjcyNDJtODNyMTRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ) (it works from Desktop only)\n- Check our [Technical FAQ](https://docs.google.com/document/d/12TlBfhIiKtyBv8RnsoJR6F72bkPDGEvPOItJIxaEzE0/edit) if you have questions \n\nSee you in May!\n" }, { "path": "asking-questions.md", "content": "## Asking questions\n\nIf you have any questions, ask them \nin the [`#course-mlops-zoomcamp`](https://app.slack.com/client/T01ATQK62F8/C02R98X7DS9) channel in [DataTalks.Club](https://datatalks.club) slack.\n\nTo keep our discussion in Slack more organized, we ask you to follow these suggestions:\n\n* Before asking a question, check [FAQ](https://docs.google.com/document/d/12TlBfhIiKtyBv8RnsoJR6F72bkPDGEvPOItJIxaEzE0/edit).\n* Use threads. When you have a problem, first describe the problem shortly\n and then put the actual error in the thread - so it doesn't take the entire screen.\n* Instead of screenshots, it's better to copy-paste the error you're getting in text.\n Use ` ``` ` for formatting your code.\n It's very difficult to read text from screenshots.\n* Please don't take pictures of your code with a phone. It's even harder to read. Follow the previous suggestion,\n and in rare cases when you need to show what happens on your screen, take a screenshot.\n* You don't need to tag the instructors when you have a problem. We will see it eventually.\n* If somebody helped you with your problem and it's not in [FAQ](https://docs.google.com/document/d/12TlBfhIiKtyBv8RnsoJR6F72bkPDGEvPOItJIxaEzE0/edit), please add it there.\n It'll help other students.\n" }, { "path": "certificate.md", "content": "## Getting your certificate\n\nCongratulations on finishing the course!\n\nHere's how you can get your certificate.\n\nFirst, get your certificate id using the `compute_certificate_id` function:\n\n```python\nfrom hashlib import sha1\n\ndef compute_hash(email):\n return sha1(email.encode('utf-8')).hexdigest()\n\ndef compute_certificate_id(email):\n email_clean = email.lower().strip()\n return compute_hash(email_clean + '_')\n```\n\nThen use this hash to get the URL\n\n```python\ncohort = 2024\ncourse = 'mlops-zoomcamp'\nyour_id = compute_certificate_id('never.give.up@gmail.com')\nurl = f\"https://certificate.datatalks.club/{course}/{cohort}/{your_id}.pdf\"\nprint(url)\n```\n\nExample: https://certificate.datatalks.club/mlops-zoomcamp/2022/fe629854d45c559e9c10b3b8458ea392fdeb68a9.pdf\n\n\n## Adding to LinkedIn\n\nYou can add your certificate to LinkedIn:\n\n* Log in to your LinkedIn account, then go to your profile.\n* On the right, in the \"Add profile\" section dropdown, choose \"Background\" and then select the drop-down triangle next to \"Licenses & Certifications\".\n* In \"Name\", enter \"MLOps Zoomcamp\".\n* In \"Issuing Organization\", enter \"DataTalksClub\".\n* (Optional) In \"Issue Date\", enter the time when the certificate was created.\n* (Optional) Select the checkbox This certification does not expire. \n* Put your certificate ID.\n* In \"Certification URL\", enter the URL for your certificate.\n\n[Adapted from here](https://support.edx.org/hc/en-us/articles/206501938-How-can-I-add-my-certificate-to-my-LinkedIn-profile-)\n" }, { "path": "cohorts/2022/01-intro/homework.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 42,\n \"id\": \"7525e867\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import pandas as pd\\n\",\n \"import seaborn as sns\\n\",\n \"\\n\",\n \"from sklearn.feature_extraction import DictVectorizer\\n\",\n \"from sklearn.linear_model import LinearRegression\\n\",\n \"\\n\",\n \"from sklearn.metrics import mean_squared_error\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 35,\n \"id\": \"4c1fbc9a\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df = pd.read_parquet('./data/fhv_tripdata_2021-01.parquet')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 20,\n \"id\": \"7a70df86\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"old_len = len(df)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 36,\n \"id\": \"f17cc325\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df['duration'] = df.dropOff_datetime - df.pickup_datetime\\n\",\n \"df['duration'] = df.duration.dt.total_seconds() / 60\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 16,\n \"id\": \"1b613836\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"19.1672240937939\"\n ]\n },\n \"execution_count\": 16,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"df.duration.mean()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 37,\n \"id\": \"3092ae43\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df = df[(df.duration >= 1) & (df.duration <= 60)].copy()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 38,\n \"id\": \"2e1aa200\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"categorical = ['PUlocationID', 'DOlocationID']\\n\",\n \"\\n\",\n \"df[categorical] = df[categorical].fillna(-1).astype('int')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 52,\n \"id\": \"22c6495b\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df[categorical] = df[categorical].astype('str')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 55,\n \"id\": \"12fcfa04\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"train_dicts = df[categorical].to_dict(orient='records')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 56,\n \"id\": \"0606301f\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"dv = DictVectorizer()\\n\",\n \"X_train = dv.fit_transform(train_dicts) \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 57,\n \"id\": \"c81c6737\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"(1109826, 525)\"\n ]\n },\n \"execution_count\": 57,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"X_train.shape\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 60,\n \"id\": \"aeaf7bad\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"y_train = df.duration.values\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 59,\n \"id\": \"75199886\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"525\"\n ]\n },\n \"execution_count\": 59,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"len(dv.feature_names_)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 61,\n \"id\": \"4d7f1e4b\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"LinearRegression()\"\n ]\n },\n \"execution_count\": 61,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"lr = LinearRegression()\\n\",\n \"lr.fit(X_train, y_train)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 62,\n \"id\": \"d61318d0\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"y_pred = lr.predict(X_train)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 63,\n \"id\": \"1aa5f5c2\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"10.528519107212292\"\n ]\n },\n \"execution_count\": 63,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"mean_squared_error(y_train, y_pred, squared=False)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 64,\n \"id\": \"41c08294\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"categorical = ['PUlocationID', 'DOlocationID']\\n\",\n \"\\n\",\n \"def read_data(filename):\\n\",\n \" df = pd.read_parquet(filename)\\n\",\n \" \\n\",\n \" df['duration'] = df.dropOff_datetime - df.pickup_datetime\\n\",\n \" df['duration'] = df.duration.dt.total_seconds() / 60\\n\",\n \"\\n\",\n \" df = df[(df.duration >= 1) & (df.duration <= 60)].copy()\\n\",\n \"\\n\",\n \" df[categorical] = df[categorical].fillna(-1).astype('int').astype('str')\\n\",\n \" \\n\",\n \" return df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 65,\n \"id\": \"4854399a\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df_val = read_data('./data/fhv_tripdata_2021-02.parquet')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 68,\n \"id\": \"f9eea69f\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"val_dicts = df_val[categorical].to_dict(orient='records')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 69,\n \"id\": \"669fda0a\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"X_val = dv.transform(val_dicts) \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 70,\n \"id\": \"7f8f950d\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"y_pred = lr.predict(X_val)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 71,\n \"id\": \"8f129850\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"y_val = df_val.duration.values\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 72,\n \"id\": \"8a277ab9\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"11.014283211122269\"\n ]\n },\n \"execution_count\": 72,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"mean_squared_error(y_val, y_pred, squared=False)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"b5108416\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.7\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n" }, { "path": "cohorts/2022/01-intro/homework.md", "content": "## 1.6 Homework\n\nThe goal of this homework is to train a simple model for predicting the duration of a ride - similar to what we did in this module.\n\n\n## Q1. Downloading the data\n\nWe'll use [the same NYC taxi dataset](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page),\nbut instead of \"Green Taxi Trip Records\", we'll use \"For-Hire Vehicle Trip Records\".\n\nDownload the data for January and February 2021.\n\nNote that you need \"For-Hire Vehicle Trip Records\", not \"High Volume For-Hire Vehicle Trip Records\".\n\nRead the data for January. How many records are there?\n\n* 1054112\n* 1154112\n* 1254112\n* 1354112\n\n\n## Q2. Computing duration\n\nNow let's compute the `duration` variable. It should contain the duration of a ride in minutes. \n\nWhat's the average trip duration in January?\n\n* 15.16\n* 19.16\n* 24.16\n* 29.16\n\n## Data preparation\n\nCheck the distribution of the duration variable. There are some outliers. \n\nLet's remove them and keep only the records where the duration was between 1 and 60 minutes (inclusive).\n\nHow many records did you drop? \n\n## Q3. Missing values\n\nThe features we'll use for our model are the pickup and dropoff location IDs. \n\nBut they have a lot of missing values there. Let's replace them with \"-1\".\n\nWhat's the fractions of missing values for the pickup location ID? I.e. fraction of \"-1\"s after you filled the NAs.\n\n* 53%\n* 63%\n* 73%\n* 83%\n\n## Q4. One-hot encoding\n\nLet's apply one-hot encoding to the pickup and dropoff location IDs. We'll use only these two features for our model. \n\n* Turn the dataframe into a list of dictionaries\n* Fit a dictionary vectorizer \n* Get a feature matrix from it\n\nWhat's the dimensionality of this matrix? (The number of columns).\n\n* 2\n* 152\n* 352\n* 525\n* 725\n\n## Q5. Training a model\n\nNow let's use the feature matrix from the previous step to train a model. \n\n* Train a plain linear regression model with default parameters \n* Calculate the RMSE of the model on the training data\n\nWhat's the RMSE on train?\n\n* 5.52\n* 10.52\n* 15.52\n* 20.52\n\n\n## Q6. Evaluating the model\n\nNow let's apply this model to the validation dataset (Feb 2021). \n\nWhat's the RMSE on validation?\n\n* 6.01\n* 11.01\n* 16.01\n* 21.01\n\n## Submit the results\n\nSubmit your results here: https://forms.gle/V8q5rv7QRoZ13Sft6\n\nIt's possible that your answers won't match exactly. If it's the case, select the closest one.\n\n\n## Deadline\n\nThe deadline for submitting is 24 May 2022 (Tuesday) 23:00 CET. After that, the form will be closed.\n\n\n## Solution\n\n* [Video](https://www.youtube.com/watch?v=feH1PMLyu-Q&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK&index=9)\n* [Notebook](homework.ipynb)\n" }, { "path": "cohorts/2022/02-experiment-tracking/homework/hpo.py", "content": "import argparse\nimport os\nimport pickle\n\nimport mlflow\nimport numpy as np\nfrom hyperopt import STATUS_OK, Trials, fmin, hp, tpe\nfrom hyperopt.pyll import scope\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import mean_squared_error\n\nmlflow.set_tracking_uri(\"http://127.0.0.1:5000\")\nmlflow.set_experiment(\"random-forest-hyperopt\")\n\n\ndef load_pickle(filename):\n with open(filename, \"rb\") as f_in:\n return pickle.load(f_in)\n\n\ndef run(data_path, num_trials):\n\n X_train, y_train = load_pickle(os.path.join(data_path, \"train.pkl\"))\n X_valid, y_valid = load_pickle(os.path.join(data_path, \"valid.pkl\"))\n\n def objective(params):\n\n rf = RandomForestRegressor(**params)\n rf.fit(X_train, y_train)\n y_pred = rf.predict(X_valid)\n rmse = mean_squared_error(y_valid, y_pred, squared=False)\n\n return {'loss': rmse, 'status': STATUS_OK}\n\n search_space = {\n 'max_depth': scope.int(hp.quniform('max_depth', 1, 20, 1)),\n 'n_estimators': scope.int(hp.quniform('n_estimators', 10, 50, 1)),\n 'min_samples_split': scope.int(hp.quniform('min_samples_split', 2, 10, 1)),\n 'min_samples_leaf': scope.int(hp.quniform('min_samples_leaf', 1, 4, 1)),\n 'random_state': 42\n }\n\n rstate = np.random.default_rng(42) # for reproducible results\n fmin(\n fn=objective,\n space=search_space,\n algo=tpe.suggest,\n max_evals=num_trials,\n trials=Trials(),\n rstate=rstate\n )\n\n\nif __name__ == '__main__':\n\n parser = argparse.ArgumentParser()\n parser.add_argument(\n \"--data_path\",\n default=\"./output\",\n help=\"the location where the processed NYC taxi trip data was saved.\"\n )\n parser.add_argument(\n \"--max_evals\",\n type=int,\n default=50,\n help=\"the number of parameter evaluations for the optimizer to explore.\"\n )\n args = parser.parse_args()\n\n run(args.data_path, args.max_evals)\n" }, { "path": "cohorts/2022/02-experiment-tracking/homework/preprocess_data.py", "content": "import argparse\nimport os\nimport pickle\n\nimport pandas as pd\nfrom sklearn.feature_extraction import DictVectorizer\n\n\ndef dump_pickle(obj, filename):\n with open(filename, \"wb\") as f_out:\n return pickle.dump(obj, f_out)\n\n\ndef read_dataframe(filename: str):\n df = pd.read_parquet(filename)\n\n df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\n df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)\n df = df[(df.duration >= 1) & (df.duration <= 60)]\n\n categorical = ['PULocationID', 'DOLocationID']\n df[categorical] = df[categorical].astype(str)\n\n return df\n\n\ndef preprocess(df: pd.DataFrame, dv: DictVectorizer, fit_dv: bool = False):\n df['PU_DO'] = df['PULocationID'] + '_' + df['DOLocationID']\n categorical = ['PU_DO']\n numerical = ['trip_distance']\n dicts = df[categorical + numerical].to_dict(orient='records')\n if fit_dv:\n X = dv.fit_transform(dicts)\n else:\n X = dv.transform(dicts)\n return X, dv\n\n\ndef run(raw_data_path: str, dest_path: str, dataset: str = \"green\"):\n # load parquet files\n df_train = read_dataframe(\n os.path.join(raw_data_path, f\"{dataset}_tripdata_2021-01.parquet\")\n )\n df_valid = read_dataframe(\n os.path.join(raw_data_path, f\"{dataset}_tripdata_2021-02.parquet\")\n )\n df_test = read_dataframe(\n os.path.join(raw_data_path, f\"{dataset}_tripdata_2021-03.parquet\")\n )\n\n # extract the target\n target = 'duration'\n y_train = df_train[target].values\n y_valid = df_valid[target].values\n y_test = df_test[target].values\n\n # fit the dictvectorizer and preprocess data\n dv = DictVectorizer()\n X_train, dv = preprocess(df_train, dv, fit_dv=True)\n X_valid, _ = preprocess(df_valid, dv, fit_dv=False)\n X_test, _ = preprocess(df_test, dv, fit_dv=False)\n\n # create dest_path folder unless it already exists\n os.makedirs(dest_path, exist_ok=True)\n\n # save dictvectorizer and datasets\n dump_pickle(dv, os.path.join(dest_path, \"dv.pkl\"))\n dump_pickle((X_train, y_train), os.path.join(dest_path, \"train.pkl\"))\n dump_pickle((X_valid, y_valid), os.path.join(dest_path, \"valid.pkl\"))\n dump_pickle((X_test, y_test), os.path.join(dest_path, \"test.pkl\"))\n\n\nif __name__ == '__main__':\n\n parser = argparse.ArgumentParser()\n parser.add_argument(\n \"--raw_data_path\",\n help=\"the location where the raw NYC taxi trip data was saved\"\n )\n parser.add_argument(\n \"--dest_path\",\n help=\"the location where the resulting files will be saved.\"\n )\n args = parser.parse_args()\n\n run(args.raw_data_path, args.dest_path)\n" }, { "path": "cohorts/2022/02-experiment-tracking/homework/register_model.py", "content": "import argparse\nimport os\nimport pickle\n\nimport mlflow\nfrom hyperopt import hp, space_eval\nfrom hyperopt.pyll import scope\nfrom mlflow.entities import ViewType\nfrom mlflow.tracking import MlflowClient\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import mean_squared_error\n\nHPO_EXPERIMENT_NAME = \"random-forest-hyperopt\"\nEXPERIMENT_NAME = \"random-forest-best-models\"\n\nmlflow.set_tracking_uri(\"http://127.0.0.1:5000\")\nmlflow.set_experiment(EXPERIMENT_NAME)\nmlflow.sklearn.autolog()\n\nSPACE = {\n 'max_depth': scope.int(hp.quniform('max_depth', 1, 20, 1)),\n 'n_estimators': scope.int(hp.quniform('n_estimators', 10, 50, 1)),\n 'min_samples_split': scope.int(hp.quniform('min_samples_split', 2, 10, 1)),\n 'min_samples_leaf': scope.int(hp.quniform('min_samples_leaf', 1, 4, 1)),\n 'random_state': 42\n}\n\n\ndef load_pickle(filename):\n with open(filename, \"rb\") as f_in:\n return pickle.load(f_in)\n\n\ndef train_and_log_model(data_path, params):\n X_train, y_train = load_pickle(os.path.join(data_path, \"train.pkl\"))\n X_valid, y_valid = load_pickle(os.path.join(data_path, \"valid.pkl\"))\n X_test, y_test = load_pickle(os.path.join(data_path, \"test.pkl\"))\n\n with mlflow.start_run():\n params = space_eval(SPACE, params)\n rf = RandomForestRegressor(**params)\n rf.fit(X_train, y_train)\n\n # evaluate model on the validation and test sets\n valid_rmse = mean_squared_error(y_valid, rf.predict(X_valid), squared=False)\n mlflow.log_metric(\"valid_rmse\", valid_rmse)\n test_rmse = mean_squared_error(y_test, rf.predict(X_test), squared=False)\n mlflow.log_metric(\"test_rmse\", test_rmse)\n\n\ndef run(data_path, log_top):\n\n client = MlflowClient()\n\n # retrieve the top_n model runs and log the models to MLflow\n experiment = client.get_experiment_by_name(HPO_EXPERIMENT_NAME)\n runs = client.search_runs(\n experiment_ids=experiment.experiment_id,\n run_view_type=ViewType.ACTIVE_ONLY,\n max_results=log_top,\n order_by=[\"metrics.rmse ASC\"]\n )\n for run in runs:\n train_and_log_model(data_path=data_path, params=run.data.params)\n\n # select the model with the lowest test RMSE\n experiment = client.get_experiment_by_name(EXPERIMENT_NAME)\n # best_run = client.search_runs( ... )[0]\n\n # register the best model\n # mlflow.register_model( ... )\n\n\nif __name__ == '__main__':\n\n parser = argparse.ArgumentParser()\n parser.add_argument(\n \"--data_path\",\n default=\"./output\",\n help=\"the location where the processed NYC taxi trip data was saved.\"\n )\n parser.add_argument(\n \"--top_n\",\n default=5,\n type=int,\n help=\"the top 'top_n' models will be evaluated to decide which model to promote.\"\n )\n args = parser.parse_args()\n\n run(args.data_path, args.top_n)\n" }, { "path": "cohorts/2022/02-experiment-tracking/homework/train.py", "content": "import argparse\nimport os\nimport pickle\n\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import mean_squared_error\n\n\ndef load_pickle(filename: str):\n with open(filename, \"rb\") as f_in:\n return pickle.load(f_in)\n\n\ndef run(data_path):\n\n X_train, y_train = load_pickle(os.path.join(data_path, \"train.pkl\"))\n X_valid, y_valid = load_pickle(os.path.join(data_path, \"valid.pkl\"))\n\n rf = RandomForestRegressor(max_depth=10, random_state=0)\n rf.fit(X_train, y_train)\n y_pred = rf.predict(X_valid)\n\n rmse = mean_squared_error(y_valid, y_pred, squared=False)\n\n\nif __name__ == '__main__':\n\n parser = argparse.ArgumentParser()\n parser.add_argument(\n \"--data_path\",\n default=\"./output\",\n help=\"the location where the processed NYC taxi trip data was saved.\"\n )\n args = parser.parse_args()\n\n run(args.data_path)\n" }, { "path": "cohorts/2022/02-experiment-tracking/homework.md", "content": "## Homework\n\nThe goal of this homework is to get familiar with MLflow.\n\n\n## Q1. Install MLflow\n\nTo get started with MLflow you'll need to install the MLflow Python package.\n\nFor this we recommend creating a separate Python environment, for example, you can use [conda environments](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs), and then install the package there with `pip` or `conda`.\n\nOnce you installed the package, run the command `mlflow --version` and check the output.\n\nWhat's the version that you have?\n\n\n## Q2. Download and preprocess the data\n\nWe'll use the Green Taxi Trip Records dataset. \n\nDownload the data for January, February and March 2021 in parquet format from [here](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page).\n\nUse the script `preprocess_data.py` located in the folder [`homework`](homework) to preprocess the data.\n\nThe script will\n\n- load the data from the folder `![](\"images/thumbnail-3-01.jpg\")
\n\n\n\n\n## 3.2 Introduction to Prefect 2.0\n\n\n ![](\"images/thumbnail-3-02.jpg\")
\n\n\n\n\n## 3.3 First Prefect flow and basics\n\n\n ![](\"images/thumbnail-3-03.jpg\")
\n\n\n\n\n## 3.4 Remote Prefect Orion deployment\n\n\n ![](\"images/thumbnail-3-04.jpg\")
\n\n\n\n\n## 3.5 Deployment of Prefect flow\n\n**Note:** There are several changes to deployment in Prefect 2.3.1 since 2.0b8:\n- `DeploymentSpec` in 2.0b8 now becomes `Deployment`. \n- `work_queue_name` is used instead of `tags` to submit the deployment to the a specific work queue. \n- You don't need to create a work queue before using the work queue. A work queue will be created if it doesn't exist. \n\n```python\nfrom prefect.deployments import Deployment\nfrom prefect.orion.schemas.schedules import IntervalSchedule\nfrom datetime import timedelta\n\ndeployment = Deployment.build_from_flow(\n flow=main,\n name=\"model_training\",\n schedule=IntervalSchedule(interval=timedelta(minutes=5)),\n work_queue_name=\"ml\"\n)\n\ndeployment.apply()\n```\n\n\n ![](\"images/thumbnail-3-05.jpg\")
\n\n\nLinks:\n\n* [Instructions for Hosting Prefect Orion](https://discourse.prefect.io/t/hosting-an-orion-instance-on-a-cloud-vm/967)\n\n\n## 3.6 MLOps Zoomcamp 3.6 - (Optional) Work queues and agents\n\n\n ![](\"images/thumbnail-3-06.jpg\")
\n\n\n\n## 3.7 Homework\n\nMore information here: TBD\n\n\n## Notes\n\nDid you take notes? Add them here:\n\n* [Week 3, Prefect Introduction and S3 Bucket configuration with Prefect by M. Ayoub C.](https://gist.github.com/Qfl3x/8dd69b8173f027b9468016c118f3b6a5)\n* [Notes from froukje](https://github.com/froukje/ml-ops-zoomcamp/blob/master/03-orchestration/week03_orchestration.ipynb)\n* [Minimalist code notes from Anna V](https://github.com/annnvv/mlops_zoomcamp/blob/main/notes/module3_notes_prefect.md)\n* [Getting Started on Prefect 2.0 + Deploying worfklows for MLflow Staging by Ron Medina (Jupyter Book)](https://particle1331.github.io/inefficient-networks/notebooks/mlops/3-prefect/3-prefect.html)\n* [Quickstart your homework by Zioalex](https://github.com/zioalex/mlops-zoomcamp/blob/week3/03-orchestration/homework_quickstart.md)\n* [Notes from Maxime M](https://github.com/maxmarkov/mlops-zoomcamp/blob/master/lecture-notes/WEEK-3/03-orchestration.md)\n* [Week3: Prefect introduction and homework notes by Bhagabat](https://github.com/BPrasad123/MLOps_Zoomcamp/tree/main/Week3)\n* [Week 3: Orchestration notes by Ayoub.B](https://github.com/ayoub-berdeddouch/mlops-journey/blob/main/orchestration-03.md)\n* Send a PR, add your notes above this line\n"
},
{
"path": "cohorts/2022/03-orchestration/code/model_training.py",
"content": "from audioop import add\nimport pandas as pd\nimport pickle\n\nfrom sklearn.feature_extraction import DictVectorizer\nfrom sklearn.linear_model import LinearRegression, Lasso, Ridge\nfrom sklearn.metrics import mean_squared_error\n\nimport xgboost as xgb\n\nfrom hyperopt import fmin, tpe, hp, STATUS_OK, Trials\nfrom hyperopt.pyll import scope\n\nimport mlflow\n\nmlflow.set_tracking_uri(\"sqlite:///mlflow.db\")\nmlflow.set_experiment(\"nyc-taxi-experiment\")\n\ndef read_dataframe(filename):\n df = pd.read_parquet(filename)\n\n df.lpep_dropoff_datetime = pd.to_datetime(df.lpep_dropoff_datetime)\n df.lpep_pickup_datetime = pd.to_datetime(df.lpep_pickup_datetime)\n\n df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\n df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)\n\n df = df[(df.duration >= 1) & (df.duration <= 60)]\n\n categorical = ['PULocationID', 'DOLocationID']\n df[categorical] = df[categorical].astype(str)\n \n return df\n\ndef add_features(train_path=\"./data/green_tripdata_2021-01.parquet\",\n val_path=\"./data/green_tripdata_2021-02.parquet\"):\n df_train = read_dataframe(train_path)\n df_val = read_dataframe(val_path)\n\n print(len(df_train))\n print(len(df_val))\n\n df_train['PU_DO'] = df_train['PULocationID'] + '_' + df_train['DOLocationID']\n df_val['PU_DO'] = df_val['PULocationID'] + '_' + df_val['DOLocationID']\n\n categorical = ['PU_DO'] #'PULocationID', 'DOLocationID']\n numerical = ['trip_distance']\n\n dv = DictVectorizer()\n\n train_dicts = df_train[categorical + numerical].to_dict(orient='records')\n X_train = dv.fit_transform(train_dicts)\n\n val_dicts = df_val[categorical + numerical].to_dict(orient='records')\n X_val = dv.transform(val_dicts)\n\n target = 'duration'\n y_train = df_train[target].values\n y_val = df_val[target].values\n\n return X_train, X_val, y_train, y_val, dv\n\n# # Modelling\n\n# lr = LinearRegression()\n# lr.fit(X_train, y_train)\n\n# y_pred = lr.predict(X_val)\n\n# mean_squared_error(y_val, y_pred, squared=False)\n\n# with open('models/lin_reg.bin', 'wb') as f_out:\n# pickle.dump((dv, lr), f_out)\n\n# with mlflow.start_run():\n\n# mlflow.set_tag(\"developer\", \"cristian\")\n\n# mlflow.log_param(\"train-data-path\", \"./data/green_tripdata_2021-01.csv\")\n# mlflow.log_param(\"valid-data-path\", \"./data/green_tripdata_2021-02.csv\")\n\n# alpha = 0.1\n# mlflow.log_param(\"alpha\", alpha)\n# lr = Lasso(alpha)\n# lr.fit(X_train, y_train)\n\n# y_pred = lr.predict(X_val)\n# rmse = mean_squared_error(y_val, y_pred, squared=False)\n# mlflow.log_metric(\"rmse\", rmse)\n\n# mlflow.log_artifact(local_path=\"models/lin_reg.bin\", artifact_path=\"models_pickle\")\n\ndef train_model_search(train, valid, y_val):\n def objective(params):\n with mlflow.start_run():\n mlflow.set_tag(\"model\", \"xgboost\")\n mlflow.log_params(params)\n booster = xgb.train(\n params=params,\n dtrain=train,\n num_boost_round=1000,\n evals=[(valid, 'validation')],\n early_stopping_rounds=50\n )\n y_pred = booster.predict(valid)\n rmse = mean_squared_error(y_val, y_pred, squared=False)\n mlflow.log_metric(\"rmse\", rmse)\n\n return {'loss': rmse, 'status': STATUS_OK}\n\n search_space = {\n 'max_depth': scope.int(hp.quniform('max_depth', 4, 100, 1)),\n 'learning_rate': hp.loguniform('learning_rate', -3, 0),\n 'reg_alpha': hp.loguniform('reg_alpha', -5, -1),\n 'reg_lambda': hp.loguniform('reg_lambda', -6, -1),\n 'min_child_weight': hp.loguniform('min_child_weight', -1, 3),\n 'objective': 'reg:linear',\n 'seed': 42\n }\n\n best_result = fmin(\n fn=objective,\n space=search_space,\n algo=tpe.suggest,\n max_evals=1,\n trials=Trials()\n )\n return\n\ndef train_best_model(train, valid, y_val, dv):\n with mlflow.start_run():\n \n train = xgb.DMatrix(X_train, label=y_train)\n valid = xgb.DMatrix(X_val, label=y_val)\n\n best_params = {\n 'learning_rate': 0.09585355369315604,\n 'max_depth': 30,\n 'min_child_weight': 1.060597050922164,\n 'objective': 'reg:linear',\n 'reg_alpha': 0.018060244040060163,\n 'reg_lambda': 0.011658731377413597,\n 'seed': 42\n }\n\n mlflow.log_params(best_params)\n\n booster = xgb.train(\n params=best_params,\n dtrain=train,\n num_boost_round=1000,\n evals=[(valid, 'validation')],\n early_stopping_rounds=50\n )\n\n y_pred = booster.predict(valid)\n rmse = mean_squared_error(y_val, y_pred, squared=False)\n mlflow.log_metric(\"rmse\", rmse)\n\n with open(\"models/preprocessor.b\", \"wb\") as f_out:\n pickle.dump(dv, f_out)\n mlflow.log_artifact(\"models/preprocessor.b\", artifact_path=\"preprocessor\")\n\n mlflow.xgboost.log_model(booster, artifact_path=\"models_mlflow\")\n\nif __name__ == \"__main__\":\n X_train, X_val, y_train, y_val, dv = add_features()\n train = xgb.DMatrix(X_train, label=y_train)\n valid = xgb.DMatrix(X_val, label=y_val)\n train_model_search(train, valid, y_val)\n train_best_model(train, valid, y_val, dv)"
},
{
"path": "cohorts/2022/03-orchestration/code/orchestration.py",
"content": "import pandas as pd\nimport pickle\n\nfrom sklearn.feature_extraction import DictVectorizer\nfrom sklearn.linear_model import LinearRegression, Lasso, Ridge\nfrom sklearn.metrics import mean_squared_error\n\nimport mlflow\n\nimport xgboost as xgb\n\nfrom hyperopt import fmin, tpe, hp, STATUS_OK, Trials\nfrom hyperopt.pyll import scope\n\nfrom prefect import flow, task\n\n@task\ndef read_dataframe(filename):\n df = pd.read_parquet(filename)\n\n df.lpep_dropoff_datetime = pd.to_datetime(df.lpep_dropoff_datetime)\n df.lpep_pickup_datetime = pd.to_datetime(df.lpep_pickup_datetime)\n\n df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\n df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)\n\n df = df[(df.duration >= 1) & (df.duration <= 60)]\n\n categorical = ['PULocationID', 'DOLocationID']\n df[categorical] = df[categorical].astype(str)\n\n return df\n\n@task\ndef add_features(df_train, df_val):\n df_train['PU_DO'] = df_train['PULocationID'] + '_' + df_train['DOLocationID']\n df_val['PU_DO'] = df_val['PULocationID'] + '_' + df_val['DOLocationID']\n\n categorical = ['PU_DO'] #'PULocationID', 'DOLocationID']\n numerical = ['trip_distance']\n\n dv = DictVectorizer()\n\n train_dicts = df_train[categorical + numerical].to_dict(orient='records')\n X_train = dv.fit_transform(train_dicts)\n\n val_dicts = df_val[categorical + numerical].to_dict(orient='records')\n X_val = dv.transform(val_dicts)\n\n\n target = 'duration'\n y_train = df_train[target].values\n y_val = df_val[target].values\n return X_train, X_val, y_train, y_val, dv\n\n\n@task\ndef train_model_search(train, valid, y_val):\n def _objective(params):\n with mlflow.start_run():\n mlflow.set_tag(\"model\", \"xgboost\")\n mlflow.log_params(params)\n booster = xgb.train(\n params=params,\n dtrain=train,\n num_boost_round=1000,\n evals=[(valid, 'validation')],\n early_stopping_rounds=50\n )\n y_pred = booster.predict(valid)\n rmse = mean_squared_error(y_val, y_pred, squared=False)\n mlflow.log_metric(\"rmse\", rmse)\n\n return {'loss': rmse, 'status': STATUS_OK}\n\n search_space = {\n 'max_depth': scope.int(hp.quniform('max_depth', 4, 100, 1)),\n 'learning_rate': hp.loguniform('learning_rate', -3, 0),\n 'reg_alpha': hp.loguniform('reg_alpha', -5, -1),\n 'reg_lambda': hp.loguniform('reg_lambda', -6, -1),\n 'min_child_weight': hp.loguniform('min_child_weight', -1, 3),\n 'objective': 'reg:linear',\n 'seed': 42\n }\n\n best_result = fmin(\n fn=_objective,\n space=search_space,\n algo=tpe.suggest,\n max_evals=1,\n trials=Trials()\n )\n return best_result\n\n@task\ndef train_best_model(X_train, X_val, y_train, y_val, dv):\n with mlflow.start_run():\n \n train = xgb.DMatrix(X_train, label=y_train)\n valid = xgb.DMatrix(X_val, label=y_val)\n\n best_params = {\n 'learning_rate': 0.09585355369315604,\n 'max_depth': 30,\n 'min_child_weight': 1.060597050922164,\n 'objective': 'reg:linear',\n 'reg_alpha': 0.018060244040060163,\n 'reg_lambda': 0.011658731377413597,\n 'seed': 42\n }\n\n mlflow.log_params(best_params)\n\n booster = xgb.train(\n params=best_params,\n dtrain=train,\n num_boost_round=1000,\n evals=[(valid, 'validation')],\n early_stopping_rounds=50\n )\n\n y_pred = booster.predict(valid)\n rmse = mean_squared_error(y_val, y_pred, squared=False)\n mlflow.log_metric(\"rmse\", rmse)\n\n with open(\"models/preprocessor.b\", \"wb\") as f_out:\n pickle.dump(dv, f_out)\n mlflow.log_artifact(\"models/preprocessor.b\", artifact_path=\"preprocessor\")\n\n mlflow.xgboost.log_model(booster, artifact_path=\"models_mlflow\")\n\n@flow\ndef main_flow(train_path: str = './data/green_tripdata_2021-01.parquet', \n val_path: str = './data/green_tripdata_2021-02.parquet'):\n mlflow.set_tracking_uri(\"sqlite:///mlflow.db\")\n mlflow.set_experiment(\"nyc-taxi-experiment\")\n # Load\n df_train = read_dataframe(train_path)\n df_val = read_dataframe(val_path)\n\n # Transform\n X_train, X_val, y_train, y_val, dv = add_features(df_train, df_val).result()\n\n # Training\n train = xgb.DMatrix(X_train, label=y_train)\n valid = xgb.DMatrix(X_val, label=y_val)\n best = train_model_search(train, valid, y_val)\n train_best_model(X_train, X_val, y_train, y_val, dv, wait_for=best)\n\n# main_flow()\n\nfrom prefect.deployments import Deployment\nfrom prefect.orion.schemas.schedules import IntervalSchedule\nfrom datetime import timedelta\n\nDeployment.build_from_flow(\n flow=main_flow,\n name=\"model_training\",\n # schedule=IntervalSchedule(interval=timedelta(weeks=1)),\n work_queue_name=\"ml\",\n)"
},
{
"path": "cohorts/2022/03-orchestration/code/prefect_deploy.py",
"content": "import pandas as pd\nimport pickle\n\nfrom sklearn.feature_extraction import DictVectorizer\nfrom sklearn.metrics import mean_squared_error\n\nimport xgboost as xgb\n\nfrom hyperopt import fmin, tpe, hp, STATUS_OK, Trials\nfrom hyperopt.pyll import scope\n\nimport mlflow\n\nfrom prefect import flow, task\n\n@task\ndef read_dataframe(filename):\n df = pd.read_parquet(filename)\n\n df.lpep_dropoff_datetime = pd.to_datetime(df.lpep_dropoff_datetime)\n df.lpep_pickup_datetime = pd.to_datetime(df.lpep_pickup_datetime)\n\n df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\n df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)\n\n df = df[(df.duration >= 1) & (df.duration <= 60)]\n\n categorical = ['PULocationID', 'DOLocationID']\n df[categorical] = df[categorical].astype(str)\n \n return df\n\n@task\ndef add_features(df_train, df_val):\n # df_train = read_dataframe(train_path)\n # df_val = read_dataframe(val_path)\n\n print(len(df_train))\n print(len(df_val))\n\n df_train['PU_DO'] = df_train['PULocationID'] + '_' + df_train['DOLocationID']\n df_val['PU_DO'] = df_val['PULocationID'] + '_' + df_val['DOLocationID']\n\n categorical = ['PU_DO'] #'PULocationID', 'DOLocationID']\n numerical = ['trip_distance']\n\n dv = DictVectorizer()\n\n train_dicts = df_train[categorical + numerical].to_dict(orient='records')\n X_train = dv.fit_transform(train_dicts)\n\n val_dicts = df_val[categorical + numerical].to_dict(orient='records')\n X_val = dv.transform(val_dicts)\n\n target = 'duration'\n y_train = df_train[target].values\n y_val = df_val[target].values\n\n return X_train, X_val, y_train, y_val, dv\n\n@task\ndef train_model_search(train, valid, y_val):\n def objective(params):\n with mlflow.start_run():\n mlflow.set_tag(\"model\", \"xgboost\")\n mlflow.log_params(params)\n booster = xgb.train(\n params=params,\n dtrain=train,\n num_boost_round=100,\n evals=[(valid, 'validation')],\n early_stopping_rounds=50\n )\n y_pred = booster.predict(valid)\n rmse = mean_squared_error(y_val, y_pred, squared=False)\n mlflow.log_metric(\"rmse\", rmse)\n\n return {'loss': rmse, 'status': STATUS_OK}\n\n search_space = {\n 'max_depth': scope.int(hp.quniform('max_depth', 4, 100, 1)),\n 'learning_rate': hp.loguniform('learning_rate', -3, 0),\n 'reg_alpha': hp.loguniform('reg_alpha', -5, -1),\n 'reg_lambda': hp.loguniform('reg_lambda', -6, -1),\n 'min_child_weight': hp.loguniform('min_child_weight', -1, 3),\n 'objective': 'reg:linear',\n 'seed': 42\n }\n\n best_result = fmin(\n fn=objective,\n space=search_space,\n algo=tpe.suggest,\n max_evals=1,\n trials=Trials()\n )\n return\n\n@task\ndef train_best_model(train, valid, y_val, dv):\n with mlflow.start_run():\n\n best_params = {\n 'learning_rate': 0.09585355369315604,\n 'max_depth': 30,\n 'min_child_weight': 1.060597050922164,\n 'objective': 'reg:linear',\n 'reg_alpha': 0.018060244040060163,\n 'reg_lambda': 0.011658731377413597,\n 'seed': 42\n }\n\n mlflow.log_params(best_params)\n\n booster = xgb.train(\n params=best_params,\n dtrain=train,\n num_boost_round=100,\n evals=[(valid, 'validation')],\n early_stopping_rounds=50\n )\n\n y_pred = booster.predict(valid)\n rmse = mean_squared_error(y_val, y_pred, squared=False)\n mlflow.log_metric(\"rmse\", rmse)\n\n with open(\"models/preprocessor.b\", \"wb\") as f_out:\n pickle.dump(dv, f_out)\n mlflow.log_artifact(\"models/preprocessor.b\", artifact_path=\"preprocessor\")\n\n mlflow.xgboost.log_model(booster, artifact_path=\"models_mlflow\")\n\n@flow\ndef main(train_path: str=\"./data/green_tripdata_2021-01.parquet\",\n val_path: str=\"./data/green_tripdata_2021-02.parquet\"):\n mlflow.set_tracking_uri(\"sqlite:///mlflow.db\")\n mlflow.set_experiment(\"nyc-taxi-experiment\")\n X_train = read_dataframe(train_path)\n X_val = read_dataframe(val_path)\n X_train, X_val, y_train, y_val, dv = add_features(X_train, X_val).result()\n train = xgb.DMatrix(X_train, label=y_train)\n valid = xgb.DMatrix(X_val, label=y_val)\n train_model_search(train, valid, y_val)\n train_best_model(train, valid, y_val, dv)\n\nfrom prefect.deployments import Deployment\nfrom prefect.orion.schemas.schedules import IntervalSchedule\nfrom datetime import timedelta\n\ndeployment = Deployment.build_from_flow(\n flow=main,\n name=\"model_training\",\n schedule=IntervalSchedule(interval=timedelta(minutes=5)),\n work_queue_name=\"ml\"\n)\n\ndeployment.apply()\n\n"
},
{
"path": "cohorts/2022/03-orchestration/code/prefect_flow.py",
"content": "import pandas as pd\nimport pickle\n\nfrom sklearn.feature_extraction import DictVectorizer\nfrom sklearn.metrics import mean_squared_error\n\nimport xgboost as xgb\n\nfrom hyperopt import fmin, tpe, hp, STATUS_OK, Trials\nfrom hyperopt.pyll import scope\n\nimport mlflow\n\nfrom prefect import flow, task\n\n@task\ndef read_dataframe(filename):\n df = pd.read_parquet(filename)\n\n df.lpep_dropoff_datetime = pd.to_datetime(df.lpep_dropoff_datetime)\n df.lpep_pickup_datetime = pd.to_datetime(df.lpep_pickup_datetime)\n\n df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\n df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)\n\n df = df[(df.duration >= 1) & (df.duration <= 60)]\n\n categorical = ['PULocationID', 'DOLocationID']\n df[categorical] = df[categorical].astype(str)\n \n return df\n\n@task\ndef add_features(df_train, df_val):\n # df_train = read_dataframe(train_path)\n # df_val = read_dataframe(val_path)\n\n print(len(df_train))\n print(len(df_val))\n\n df_train['PU_DO'] = df_train['PULocationID'] + '_' + df_train['DOLocationID']\n df_val['PU_DO'] = df_val['PULocationID'] + '_' + df_val['DOLocationID']\n\n categorical = ['PU_DO'] #'PULocationID', 'DOLocationID']\n numerical = ['trip_distance']\n\n dv = DictVectorizer()\n\n train_dicts = df_train[categorical + numerical].to_dict(orient='records')\n X_train = dv.fit_transform(train_dicts)\n\n val_dicts = df_val[categorical + numerical].to_dict(orient='records')\n X_val = dv.transform(val_dicts)\n\n target = 'duration'\n y_train = df_train[target].values\n y_val = df_val[target].values\n\n return X_train, X_val, y_train, y_val, dv\n\n@task\ndef train_model_search(train, valid, y_val):\n def objective(params):\n with mlflow.start_run():\n mlflow.set_tag(\"model\", \"xgboost\")\n mlflow.log_params(params)\n booster = xgb.train(\n params=params,\n dtrain=train,\n num_boost_round=100,\n evals=[(valid, 'validation')],\n early_stopping_rounds=50\n )\n y_pred = booster.predict(valid)\n rmse = mean_squared_error(y_val, y_pred, squared=False)\n mlflow.log_metric(\"rmse\", rmse)\n\n return {'loss': rmse, 'status': STATUS_OK}\n\n search_space = {\n 'max_depth': scope.int(hp.quniform('max_depth', 4, 100, 1)),\n 'learning_rate': hp.loguniform('learning_rate', -3, 0),\n 'reg_alpha': hp.loguniform('reg_alpha', -5, -1),\n 'reg_lambda': hp.loguniform('reg_lambda', -6, -1),\n 'min_child_weight': hp.loguniform('min_child_weight', -1, 3),\n 'objective': 'reg:linear',\n 'seed': 42\n }\n\n best_result = fmin(\n fn=objective,\n space=search_space,\n algo=tpe.suggest,\n max_evals=1,\n trials=Trials()\n )\n return\n\n@task\ndef train_best_model(train, valid, y_val, dv):\n with mlflow.start_run():\n\n best_params = {\n 'learning_rate': 0.09585355369315604,\n 'max_depth': 30,\n 'min_child_weight': 1.060597050922164,\n 'objective': 'reg:linear',\n 'reg_alpha': 0.018060244040060163,\n 'reg_lambda': 0.011658731377413597,\n 'seed': 42\n }\n\n mlflow.log_params(best_params)\n\n booster = xgb.train(\n params=best_params,\n dtrain=train,\n num_boost_round=100,\n evals=[(valid, 'validation')],\n early_stopping_rounds=50\n )\n\n y_pred = booster.predict(valid)\n rmse = mean_squared_error(y_val, y_pred, squared=False)\n mlflow.log_metric(\"rmse\", rmse)\n\n with open(\"models/preprocessor.b\", \"wb\") as f_out:\n pickle.dump(dv, f_out)\n mlflow.log_artifact(\"models/preprocessor.b\", artifact_path=\"preprocessor\")\n\n mlflow.xgboost.log_model(booster, artifact_path=\"models_mlflow\")\n\n@flow\ndef main(train_path: str=\"./data/green_tripdata_2021-01.parquet\",\n val_path: str=\"./data/green_tripdata_2021-02.parquet\"):\n mlflow.set_tracking_uri(\"sqlite:///mlflow.db\")\n mlflow.set_experiment(\"nyc-taxi-experiment\")\n X_train = read_dataframe(train_path)\n X_val = read_dataframe(val_path)\n X_train, X_val, y_train, y_val, dv = add_features(X_train, X_val).result()\n train = xgb.DMatrix(X_train, label=y_train)\n valid = xgb.DMatrix(X_val, label=y_val)\n train_model_search(train, valid, y_val)\n train_best_model(train, valid, y_val, dv)\n"
},
{
"path": "cohorts/2022/03-orchestration/code/work-queue.py",
"content": "from prefect import flow\n\n@flow\ndef myflow():\n print(\"hello\")\n\nfrom prefect.deployments import Deployment\nfrom prefect.orion.schemas.schedules import IntervalSchedule\nfrom datetime import timedelta\n\ndeployment_dev = Deployment.build_from_flow(\n flow=myflow,\n name=\"model_training-dev\",\n schedule=IntervalSchedule(interval=timedelta(minutes=5)),\n work_queue_name=\"dev\"\n)\n\ndeployment_dev.apply()\n\ndeployment_prod = Deployment.build_from_flow(\n flow=myflow,\n name=\"model_training-prod\",\n schedule=IntervalSchedule(interval=timedelta(minutes=5)),\n work_queue_name=\"prod\"\n)\n\ndeployment_prod.apply()\n\n"
},
{
"path": "cohorts/2022/03-orchestration/homework.md",
"content": "## 3.6 Homework\n\nThe goal of this homework is to familiarize users with workflow orchestration. We start from the solution of homework 1. The notebook can be found below:\n\nhttps://github.com/DataTalksClub/mlops-zoomcamp/blob/main/01-intro/homework.ipynb\n\nThis has already been converted to a script called `homework.py` in the `03-orchestration` folder of this repo. \n\nYou will use the FHV dataset like in homework 1.\n\n\n## Motivation\n\nWe already have a model training script. Maybe a data scientist in your team handed it to you and your job is schedule the running of training script using a workflow orchestration - Prefect in this case. Below are the requirements. Do not implement them yet, we will do so in this exercise. Just understand the goal.\n\n1. The training flow will be run every month.\n2. The flow will take in a parameter called `date` which will be a datetime.\n * a. `date` should default to `None`.\n * b. If `date` is `None`, set `date` as the current day. Use the data from 2 months back as the training data and the data from the previous month as validation data.\n * c. If `date` is passed, get 2 months before the `date` as the training data, and the previous month as validation data.\n * d. As a concrete example, if the date passed is \"2021-03-15\", the training data should be \"fhv_tripdata_2021-01.parquet\" and the validation file will be \"fhv_trip_data_2021-02.parquet\".\n3. Save the model as \"model-{date}.bin\" where date is in `YYYY-MM-DD`. Note that `date` here is the value of the flow `parameter`. In practice, this setup makes it very easy to get the latest model to run predictions because you just need to get the most recent one.\n4. In this example we use a DictVectorizer. That is needed to run future data through our model. Save that as \"dv-{date}.b\". Similar to above, if the date is `2021-03-15`, the files output should be `model-2021-03-15.bin` and `dv-2021-03-15.b`.\n\nThis convention is not strict in industry, and in practice, you will come up with your own system to manage these training pipeline runs. For example, if we wanted to train on the whole history instead of just one month, we'd need to allow for added parameterization and logic in our flow. If the data came in weekly instead of monthly, we might need a different naming convention. But these requirements are already a simple approximation of something you could use in production.\n\nOn the deployment side, it's very easy to just pull in the latest data and predict it using the latest model and vectorizer files. Tools the MLFlow in the last chapter can simplify that process as well. This homework will focus more on the batch training.\n\nIn order, this homework assignment will be about:\n\n1. Converting the script to a Flow.\n2. Changing the parameters to take in a `date`. Making this parameter dynamic.\n3. Scheduling a batch training job that outputs the latest model somewhere.\n\n## Setup\n\nYou can use either local Prefect Orion or a VM hosted Prefect Orion instance for this. It shouldn't matter. Just note that if you use a VM hosted one, you will need to configure your local API to hit the VM.\n\nVideo 3.4 of the course will give more detailed instructions if you been run it on a VM.\n\n## Q1. Converting the script to a Prefect flow\n\nIf you want to follow the videos exactly, do:\n\n```bash\npip install prefect==2.0b5\n```\n\nIf you need Windows support, check [`windows.md`](windows.md) for installation instructions.\n\nThe current script `homework.py` is a fully functional script as long as you already have `fhv_trip_data_2021-01.parquet` and `fhv_trip_data_2021-02.parquet` inside a `data` folder. You should be able to already run it using:\n\n```bash\npython homework.py\n```\n\nWe want to bring this to workflow orchestration to add observability around it. The `main` function will be converted to a `flow` and the other functions will be `tasks`. After adding all of the decorators, there is actually one task that you will need to call `.result()` for inside the `flow` to get it to work. Which task is this?\n\n* `read_data`\n* `prepare_features`\n* `train_model`\n* `run_model`\n\nImportant: change all `print` statements to use the Prefect logger. Using the `print` statement will not appear in the Prefect UI. You have to call `get_run_logger` at the start of the task to use it.\n\n## Q2. Parameterizing the flow\n\nRight now there are two parameters for `main()` called `train_path` and `val_path`. We want to change the flow function to accept `date` instead. `date` should then be passed to a task that gives both the `train_path` and `val_path` to use.\n\nIt should look like this:\n\n```python\n@flow\ndef main(date=None):\n train_path, val_path = get_paths(date).result()\n # rest of flow below\n```\n\nWhere `get_paths` is a task that you have to implement. The specs for this are outlined in the motivation section. Listing them out again here:\n\n* a. The flow will take in a parameter called `date` which will be a datetime. `date` should default to `None`.\n* b. If `date` is `None`, use the current day. Use the data from 2 months back as the training data and the data from the previous month as validation data.\n* c. If a `date` value is supplied, get 2 months before the `date` as the training data, and the previous month as validation data.\n* d. As a concrete example, if the date passed is \"2021-03-15\", the training data should be \"fhv_tripdata_2021-01.parquet\" and the validation file will be \"fhv_trip_data_2021-02.parquet\".\n\nBecause we have two files:\n\n* fhv_tripdata_2021-01.parquet\n* fhv_tripdata_2021-02.parquet\n\nChange the `main()` flow call to the following:\n\n```python\nmain(date=\"2021-03-15\")\n```\n\nand it should use those files. This is a simplification for testing our homework.\n\nRecall the page from where we downloaded the For-Hire trip data.\n\nhttps://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page\n\nDownload the relevant files needed to run the `main` flow if `date` is 2021-08-15.\n\nFor example:\n\n```python\nmain(date=\"2021-08-15\")\n```\n\nBy setting up the logger from the previous step, we should see some logs about our training job. What is the validation MSE when running the flow with this date?\n\nNote you need to download the relevant files to run. Part of this question is understanding which files the flow should be looking for.\n\nThe validation MSE is:\n\n* 11.637\n* 11.837\n* 12.037\n* 12.237\n\n## Q3. Saving the model and artifacts\n\nAt the moment, we are not saving the model and vectorizer for future use. You don't need a new task for this, you can just add it inside the `flow`. The requirements for filenames to save it as were mentioned in the Motivation section. They are pasted again here:\n\n* Save the model as \"model-{date}.bin\" where date is in `YYYY-MM-DD`. Note that `date` here is the value of the flow `parameter`. In practice, this setup makes it very easy to get the latest model to run predictions because you just need to get the most recent one.\n* In this example we use a DictVectorizer. That is needed to run future data through our model. Save that as \"dv-{date}.b\". Similar to above, if the date is `2021-03-15`, the files output should be `model-2021-03-15.bin` and `dv-2021-03-15.b`.\n\nBy using this file name, during inference, we can just pull the latest model from our model directory and apply it. Assuming we already had a list of filenames:\n\n```python\n['model-2021-03-15.bin', 'model-2021-04-15.bin', 'model-2021-05-15.bin']\n```\n\nWe could do something like `sorted(model_list, reverse=False)[0]` to get the filename of the latest file. This is the simplest way to consistently use the latest trained model for inference. Tools like MLFlow give us more control logic to use flows.\n\nWhat is the file size of the `DictVectorizer` that we trained when the `date` is 2021-08-15?\n\n* 13,000 bytes \n* 23,000 bytes \n* 33,000 bytes \n* 43,000 bytes \n\nYou can bring up the Orion UI and see the work you've been doing. If you are using local Orion, you can start the server with:\n\n```bash\nprefect orion start\n```\n\nYou should be able to see previous Flow runs and the most recent successful runs. Navigate to some of them. Take time to explore the UI. The radar plot would be a good thing to share on social media if you participate in those posts.\n\n## Q4. Creating a deployment with a CronSchedule\n\nWe previously showed the `IntervalSchedule` in the video tutorials. In some cases, the interval is too rigid. For example, what if we wanted to run this `flow` on the 15th of every month? An interval of 30 days would not be in sync. In cases like these, the `CronSchedule` is more appropriate. The documentation for that is [here](https://orion-docs.prefect.io/concepts/schedules/#cronschedule)\n\nCron is an important part of workflow orchestration. It is used to schedule tasks, and was a predecessor for more mature orchestration frameworks. A lot of teams still use Cron in production. Even if you don't use Cron, the Cron expression is very common as a way to write a schedule, and the basics are worth learning for orchestration, even outside Prefect.\n\nFor this exercise, use a `CronSchedule` when creating a Prefect deployment.\n\nWhat is the Cron expression to run a flow at 9 AM every 15th of the month?\n\n* `* * 15 9 0`\n* `9 15 * * *`\n* `0 9 15 * *`\n* `0 15 9 1 *`\n\nHint: there are many Cron to English tools. Try looking for one to help you.\n\nCreate a deployment with `prefect deployment create` after you write your `DeploymentSpec`\n\n\n## Q5. Viewing the Deployment \n\nView the deployment in the UI. When first loading, we may not see that many flows because the default filter is 1 day back and 1 day forward. Remove the filter for 1 day forward to see the scheduled runs. \n\nHow many flow runs are scheduled by Prefect in advance? You should not be counting manually. There is a number of upcoming runs on the top right of the dashboard.\n\n* 0\n* 3\n* 10\n* 25\n\n**Note:** Your answer may be off by 1. It's okay. Select the option closest to your answer.\n\n## Q6. Creating a work-queue\n\nIn order to run this flow, you will need an agent and a work queue. Because we scheduled our flow on every month, it won't really get picked up by an agent. For this exercise, create a work-queue from the UI and view it using the CLI. \n\nFor all CLI commands with Prefect, you can use `--help` to get more information. \n\nFor example,\n\n* `prefect --help`\n* `prefect work-queue --help`\n\nWhat is the command to view the available work-queues?\n\n* `prefect work-queue inspect`\n* `prefect work-queue ls`\n* `prefect work-queue preview`\n* `prefect work-queue list`\n\n\n\n## Submit the results\n\n* Submit your results here: https://forms.gle/tYEZdrYdXHBizmtJ7\n* It's possible that your answers won't match exactly. If it's the case, select the closest one.\n* You can submit your answers multiple times. In this case, the last submission will be used for scoring.\n\n## Deadline\n\nThe deadline for submitting is 13 June 2022 (Monday) 23:00 CEST. After that, the form will be closed.\n\n\n## Solution\n\n* [Code - `homework_solution.py`](homework_solution.py)\n* [Video](https://www.youtube.com/watch?v=DfmRb7J-Wuw)\n"
},
{
"path": "cohorts/2022/03-orchestration/homework.py",
"content": "import pandas as pd\n\nfrom sklearn.feature_extraction import DictVectorizer\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_squared_error\n\ndef read_data(path):\n df = pd.read_parquet(path)\n return df\n\ndef prepare_features(df, categorical, train=True):\n df['duration'] = df.dropOff_datetime - df.pickup_datetime\n df['duration'] = df.duration.dt.total_seconds() / 60\n df = df[(df.duration >= 1) & (df.duration <= 60)].copy()\n\n mean_duration = df.duration.mean()\n if train:\n print(f\"The mean duration of training is {mean_duration}\")\n else:\n print(f\"The mean duration of validation is {mean_duration}\")\n \n df[categorical] = df[categorical].fillna(-1).astype('int').astype('str')\n return df\n\ndef train_model(df, categorical):\n\n train_dicts = df[categorical].to_dict(orient='records')\n dv = DictVectorizer()\n X_train = dv.fit_transform(train_dicts) \n y_train = df.duration.values\n\n print(f\"The shape of X_train is {X_train.shape}\")\n print(f\"The DictVectorizer has {len(dv.feature_names_)} features\")\n\n lr = LinearRegression()\n lr.fit(X_train, y_train)\n y_pred = lr.predict(X_train)\n mse = mean_squared_error(y_train, y_pred, squared=False)\n print(f\"The MSE of training is: {mse}\")\n return lr, dv\n\ndef run_model(df, categorical, dv, lr):\n val_dicts = df[categorical].to_dict(orient='records')\n X_val = dv.transform(val_dicts) \n y_pred = lr.predict(X_val)\n y_val = df.duration.values\n\n mse = mean_squared_error(y_val, y_pred, squared=False)\n print(f\"The MSE of validation is: {mse}\")\n return\n\ndef main(train_path: str = './data/fhv_tripdata_2021-01.parquet', \n val_path: str = './data/fhv_tripdata_2021-02.parquet'):\n\n categorical = ['PUlocationID', 'DOlocationID']\n\n df_train = read_data(train_path)\n df_train_processed = prepare_features(df_train, categorical)\n\n df_val = read_data(val_path)\n df_val_processed = prepare_features(df_val, categorical, False)\n\n # train the model\n lr, dv = train_model(df_train_processed, categorical)\n run_model(df_val_processed, categorical, dv, lr)\n\nmain()\n"
},
{
"path": "cohorts/2022/03-orchestration/homework_solution.py",
"content": "import pandas as pd\n\nfrom sklearn.feature_extraction import DictVectorizer\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_squared_error\n\nfrom prefect import task, flow, get_run_logger\nfrom datetime import datetime\nimport pickle\n\n@task\ndef read_data(path):\n df = pd.read_parquet(path)\n return df\n\n@task\ndef prepare_features(df, categorical, train=True):\n logger = get_run_logger()\n df['duration'] = df.dropOff_datetime - df.pickup_datetime\n df['duration'] = df.duration.dt.total_seconds() / 60\n df = df[(df.duration >= 1) & (df.duration <= 60)].copy()\n\n mean_duration = df.duration.mean()\n if train:\n logger.info(f\"The mean duration of training is {mean_duration}\")\n else:\n logger.info(f\"The mean duration of validation is {mean_duration}\")\n \n df[categorical] = df[categorical].fillna(-1).astype('int').astype('str')\n return df\n\n@task\ndef train_model(df, categorical):\n logger = get_run_logger()\n train_dicts = df[categorical].to_dict(orient='records')\n dv = DictVectorizer()\n X_train = dv.fit_transform(train_dicts) \n y_train = df.duration.values\n\n logger.info(f\"The shape of X_train is {X_train.shape}\")\n logger.info(f\"The DictVectorizer has {len(dv.feature_names_)} features\")\n\n lr = LinearRegression()\n lr.fit(X_train, y_train)\n y_pred = lr.predict(X_train)\n mse = mean_squared_error(y_train, y_pred, squared=False)\n logger.info(f\"The MSE of training is: {mse}\")\n return lr, dv\n\n@task\ndef run_model(df, categorical, dv, lr):\n logger = get_run_logger()\n val_dicts = df[categorical].to_dict(orient='records')\n X_val = dv.transform(val_dicts) \n y_pred = lr.predict(X_val)\n y_val = df.duration.values\n\n mse = mean_squared_error(y_val, y_pred, squared=False)\n logger.info(f\"The MSE of validation is: {mse}\")\n return\n\n@task\ndef get_paths(date):\n from dateutil.relativedelta import relativedelta\n if date:\n processed_date = datetime.strptime(date, \"%Y-%m-%d\")\n else:\n processed_date = datetime.today()\n train_date = processed_date - relativedelta(months=2)\n val_date = processed_date - relativedelta(months=1)\n train_path = f\"./data/fhv_tripdata_{train_date.year}-{str(train_date.month).zfill(2)}.parquet\"\n val_path = f\"./data/fhv_tripdata_{val_date.year}-{str(val_date.month).zfill(2)}.parquet\"\n return train_path, val_path\n\n@flow\ndef main(date=None):\n train_path, val_path = get_paths(date).result()\n\n categorical = ['PUlocationID', 'DOlocationID']\n\n df_train = read_data(train_path) \n df_train_processed = prepare_features(df_train, categorical)\n\n df_val = read_data(val_path)\n df_val_processed = prepare_features(df_val, categorical, False)\n\n # train the model\n lr, dv = train_model(df_train_processed, categorical).result()\n run_model(df_val_processed, categorical, dv, lr)\n\n if date is None:\n date = datetime.today.strftime(\"%Y-%m-%d\")\n with open(f'./models/dv-{date}.b', 'wb') as f_out:\n pickle.dump(dv, f_out)\n\n# main(\"2021-08-15\")\n\nfrom prefect.deployments import Deployment\nfrom prefect.orion.schemas.schedules import CronSchedule\nfrom prefect.flow_runners import SubprocessFlowRunner\n\nDeployment(\n flow=main,\n name=\"model_training\",\n schedule=CronSchedule(cron=\"0 9 15 * *\"),\n flow_runner=SubprocessFlowRunner(),\n)\n"
},
{
"path": "cohorts/2022/04-deployment/homework/Dockerfile",
"content": "FROM python:3.9.7-slim\n\nWORKDIR /app\nCOPY [ \"model2.bin\", \"model.bin\" ]\n "
},
{
"path": "cohorts/2022/04-deployment/homework/Pipfile",
"content": "[[source]]\nurl = \"https://pypi.org/simple\"\nverify_ssl = true\nname = \"pypi\"\n\n[packages]\nscikit-learn = \"==1.0.2\"\npandas = \"*\"\npyarrow = \"*\"\ns3fs = \"*\"\n\n[dev-packages]\n\n[requires]\npython_version = \"3.9\"\n"
},
{
"path": "cohorts/2022/04-deployment/homework/batch.py",
"content": "#!/usr/bin/env python\n# coding: utf-8\n\nimport sys\nimport pickle\nimport pandas as pd\n\n\nyear = int(sys.argv[1]) # 2021\nmonth = int(sys.argv[2]) #2\n\ninput_file = f's3://nyc-tlc/trip data/fhv_tripdata_{year:04d}-{month:02d}.parquet'\noutput_file = f's3://nyc-duration-prediction-alexey/taxi_type=fhv/year={year:04d}/month={month:02d}/predictions.parquet'\n\n\nwith open('model.bin', 'rb') as f_in:\n dv, lr = pickle.load(f_in)\n\n\ncategorical = ['PUlocationID', 'DOlocationID']\n\ndef read_data(filename):\n df = pd.read_parquet(filename)\n \n df['duration'] = df.dropOff_datetime - df.pickup_datetime\n df['duration'] = df.duration.dt.total_seconds() / 60\n\n df = df[(df.duration >= 1) & (df.duration <= 60)].copy()\n\n df[categorical] = df[categorical].fillna(-1).astype('int').astype('str')\n \n return df\n\n\ndf = read_data(input_file)\ndf['ride_id'] = f'{year:04d}/{month:02d}_' + df.index.astype('str')\n\n\ndicts = df[categorical].to_dict(orient='records')\nX_val = dv.transform(dicts)\ny_pred = lr.predict(X_val)\n\n\nprint('predicted mean duration:', y_pred.mean())\n\n\ndf_result = pd.DataFrame()\ndf_result['ride_id'] = df['ride_id']\ndf_result['predicted_duration'] = y_pred\n\n\ndf_result.to_parquet(\n output_file,\n engine='pyarrow',\n compression=None,\n index=False\n)"
},
{
"path": "cohorts/2022/04-deployment/homework/homework.dockerfile",
"content": "FROM agrigorev/zoomcamp-model:mlops-3.9.7-slim\n\nRUN pip install -U pip\nRUN pip install pipenv \n\nWORKDIR /app\n\nCOPY [ \"Pipfile\", \"Pipfile.lock\", \"./\" ]\n\nRUN pipenv install --system --deploy\n\nCOPY [ \"batch.py\", \"batch.py\" ]\n\nENTRYPOINT [ \"python\", \"batch.py\" ]"
},
{
"path": "cohorts/2022/04-deployment/homework/starter.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"id\": \"2c51efaa\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"scikit-learn==1.0.2\\n\",\n \"scikit-learn-intelex==2021.20210714.120553\\n\"\n ]\n }\n ],\n \"source\": [\n \"!pip freeze | grep scikit-learn\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"id\": \"0ef880a0\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import pickle\\n\",\n \"import pandas as pd\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 15,\n \"id\": \"920cff32\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"year = 2021\\n\",\n \"month = 2\\n\",\n \"\\n\",\n \"input_file = f'https://nyc-tlc.s3.amazonaws.com/trip+data/fhv_tripdata_{year:04d}-{month:02d}.parquet'\\n\",\n \"output_file = f'output/fhv_tripdata_{year:04d}-{month:02d}.parquet'\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"id\": \"7836ccfd\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"with open('model.bin', 'rb') as f_in:\\n\",\n \" dv, lr = pickle.load(f_in)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"id\": \"41c08294\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"categorical = ['PUlocationID', 'DOlocationID']\\n\",\n \"\\n\",\n \"def read_data(filename):\\n\",\n \" df = pd.read_parquet(filename)\\n\",\n \" \\n\",\n \" df['duration'] = df.dropOff_datetime - df.pickup_datetime\\n\",\n \" df['duration'] = df.duration.dt.total_seconds() / 60\\n\",\n \"\\n\",\n \" df = df[(df.duration >= 1) & (df.duration <= 60)].copy()\\n\",\n \"\\n\",\n \" df[categorical] = df[categorical].fillna(-1).astype('int').astype('str')\\n\",\n \" \\n\",\n \" return df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 10,\n \"id\": \"4854399a\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df = read_data(input_file)\\n\",\n \"df['ride_id'] = f'{year:04d}/{month:02d}_' + df.index.astype('str')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 6,\n \"id\": \"669fda0a\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"dicts = df[categorical].to_dict(orient='records')\\n\",\n \"X_val = dv.transform(dicts)\\n\",\n \"y_pred = lr.predict(X_val)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 7,\n \"id\": \"914b15a5\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"16.191691679979066\"\n ]\n },\n \"execution_count\": 7,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"y_pred.mean()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 13,\n \"id\": \"037e3d22\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df_result = pd.DataFrame()\\n\",\n \"df_result['ride_id'] = df['ride_id']\\n\",\n \"df_result['predicted_duration'] = y_pred\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 17,\n \"id\": \"7a5753be\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df_result.to_parquet(\\n\",\n \" output_file,\\n\",\n \" engine='pyarrow',\\n\",\n \" compression=None,\\n\",\n \" index=False\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 18,\n \"id\": \"f0b3b58c\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"total 19M\\r\\n\",\n \"-rw-rw-r-- 1 ubuntu ubuntu 19M Jun 30 08:43 fhv_tripdata_2021-02.parquet\\r\\n\"\n ]\n }\n ],\n \"source\": [\n \"!ls -lh output/\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"0dbe3e15\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.7\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
},
{
"path": "cohorts/2022/04-deployment/homework.md",
"content": "## 4.8 Homework\n\nIn this homework, we'll deploy the ride duration model in batch mode. Like in homework 1 and 3, we'll use the FHV data. \n\nYou'll find the starter code in the [homework](https://github.com/DataTalksClub/mlops-zoomcamp/blob/74324e4d3759e9712ce406b8b30c77cff66e6cef/04-deployment/homework/) directory.\n\n\n## Q1. Notebook\n\nWe'll start with the same notebook we ended up with in homework 1.\n\nWe cleaned it a little bit and kept only the scoring part. Now it's in [homework/starter.ipynb](https://github.com/DataTalksClub/mlops-zoomcamp/blob/74324e4d3759e9712ce406b8b30c77cff66e6cef/04-deployment/homework/starter.ipynb).\n\nRun this notebook for the February 2021 FVH data.\n\nWhat's the mean predicted duration for this dataset?\n\n* 11.19\n* 16.19\n* 21.19\n* 26.19\n\n\n## Q2. Preparing the output\n\nLike in the course videos, we want to prepare the dataframe with the output. \n\nFirst, let's create an artificial `ride_id` column:\n\n```python\ndf['ride_id'] = f'{year:04d}/{month:02d}_' + df.index.astype('str')\n```\n\nNext, write the ride id and the predictions to a dataframe with results. \n\nSave it as parquet:\n\n```python\ndf_result.to_parquet(\n output_file,\n engine='pyarrow',\n compression=None,\n index=False\n)\n```\n\nWhat's the size of the output file?\n\n* 9M\n* 19M\n* 29M\n* 39M\n\nMake sure you use the snippet above for saving the file. It should contain only these two columns. For this question, don't change the\ndtypes of the columns and use pyarrow, not fastparquet. \n\n\n## Q3. Creating the scoring script\n\nNow let's turn the notebook into a script. \n\nWhich command you need to execute for that?\n\n\n## Q4. Virtual environment\n\nNow let's put everything into a virtual environment. We'll use pipenv for that.\n\nInstall all the required libraries. Pay attention to the Scikit-Learn version:\ncheck the starter notebook for details. \n\nAfter installing the libraries, pipenv creates two files: `Pipfile`\nand `Pipfile.lock`. The `Pipfile.lock` file keeps the hashes of the\ndependencies we use for the virtual env.\n\nWhat's the first hash for the Scikit-Learn dependency?\n\n\n## Q5. Parametrize the script\n\nLet's now make the script configurable via CLI. We'll create two \nparameters: year and month.\n\nRun the script for March 2021. \n\nWhat's the mean predicted duration? \n\n* 11.29\n* 16.29\n* 21.29\n* 26.29\n\nHint: just add a print statement to your script.\n\n\n## Q6. Docker contaner \n\nFinally, we'll package the script in the docker container. \nFor that, you'll need to use a base image that we prepared. \n\nThis is how it looks like:\n\n```\nFROM python:3.9.7-slim\n\nWORKDIR /app\nCOPY [ \"model2.bin\", \"model.bin\" ]\n```\n\n(see [`homework/Dockerfile`](homework/Dockerfile))\n\nWe pushed it to [`agrigorev/zoomcamp-model:mlops-3.9.7-slim`](https://hub.docker.com/layers/zoomcamp-model/agrigorev/zoomcamp-model/mlops-3.9.7-slim/images/sha256-7fac33c783cc6018356ce16a4b408f6c977b55a4df52bdb6c4d0215edf83af5d?context=explore),\nwhich you should use as your base image.\n\nThat is, this is how your Dockerfile should start:\n\n```docker\nFROM agrigorev/zoomcamp-model:mlops-3.9.7-slim\n\n# do stuff here\n```\n\nThis image already has a pickle file with a dictionary vectorizer\nand a model. You will need to use them.\n\nImportant: don't copy the model to the docker image. You will need\nto use the pickle file already in the image. \n\nNow run the script with docker. What's the mean predicted duration\nfor April 2021? \n\n\n* 9.96\n* 16.55\n* 25.96\n* 36.55\n\n\n## Bonus: upload the result to the cloud (Not graded)\n\nJust printing the mean duration inside the docker image \ndoesn't seem very practical. Typically, after creating the output \nfile, we upload it to the cloud storage.\n\nModify your code to upload the parquet file to S3/GCS/etc.\n\n\n## Submit the results\n\n* Submit your results here: https://forms.gle/pFAYjTFqFMJELG819\n* It's possible that your answers won't match exactly. If it's the case, select the closest one.\n* You can submit your answers multiple times. In this case, the last submission will be used for scoring.\n\n## Deadline\n\nThe deadline for submitting is 27 June 2022 (Monday) 23:00 CEST. After that, the form will be closed.\n\n\n## Solution\n\nAfter the deadline, we'll post the solution here\n\n\n## Publishing the image to dockerhub\n\nThis is how we published the image to Docker hub:\n\n```bash\ndocker build -t mlops-zoomcamp-model:v1 .\ndocker tag mlops-zoomcamp-model:v1 agrigorev/zoomcamp-model:mlops-3.9.7-slim\ndocker push agrigorev/zoomcamp-model:mlops-3.9.7-slim\n```\n\n"
},
{
"path": "cohorts/2022/05-monitoring/README.md",
"content": "# 5. Model Monitoring\n\n## 5.1 Monitoring for ML-based services\n\n\n ![](\"images/thumbnail-5-01.jpg\")
\n\n\n\n[Slides](https://drive.google.com/file/d/1wcMU75ZcNNJie4ELjsKPkITIL93wHykt/view?usp=sharing)\n\n\n## 5.2 Setting up the environment\n\n\n ![](\"images/thumbnail-5-02.jpg\")
\n\n\n\n\n## 5.3 Creating a prediction service and simulating traffic\n\n\n ![](\"images/thumbnail-5-03.jpg\")
\n\n\n\n\n## 5.4 Realtime monitoring walktrough (Prometheus, Evidently, Grafana)\n\n\n ![](\"images/thumbnail-5-04.jpg\")
\n\n\n\n\n## 5.5 Batch monitoring walktrough (Prefect, MongoDB, Evidently)\n\n\n ![](\"images/thumbnail-5-05.jpg\")
\n\n\n\n\n## 5.6 Homework\n\nMore information here: TBD\n\n\n## Notes\n\nDid you take notes? Add them here:\n\n* [Week 5 notes by M. Ayoub C.](https://gist.github.com/Qfl3x/aa6b1bec35fb645ded0371c46e8aafd1)\n* [week 5: Monitoring notes Ayoub.B](https://github.com/ayoub-berdeddouch/mlops-journey/blob/main/monitoring-05.md)\n* Send a PR, add your notes above this line\n\n\n\n# Monitoring example\n\n## Prerequisites\n\nYou need following tools installed:\n- `docker`\n- `docker-compose` (included to Docker Desktop for Mac and Docker Desktop for Windows )\n\n## Preparation\n\nNote: all actions expected to be executed in repo folder.\n\n- Create virtual environment and activate it (eg. `python -m venv venv && source ./venv/bin/activate`)\n- Install required packages `pip install -r requirements.txt`\n- Run `python prepare.py` for downloading datasets\n\n## Monitoring Example\n\n### Starting services\n\nTo start all required services, execute:\n```bash\ndocker-compose up\n```\n\nIt will start following services:\n- `prometheus` - TSDB for metrics\n- `grafana` - Visual tool for metrics\n- `mongo` - MongoDB, for storing raw data, predictions, targets and profile reports\n- `evidently_service` - Evindently RT-monitoring service (draft example)\n- `prediction_service` - main service, which makes predictions\n\n### Sending data\n\nTo start sending data to service, execute:\n```bash\npython send_data.py\n```\n\nThis script will send every second single row from dataset to prediction service along with creating file `target.csv` with actual results (so it can be loaded after)\n\n## Batch Monitoring Example\n\nAfter you stop sending data to service, you can run batch monitoring pipeline (using Prefect) by running script:\n\n```bash\npython prefect_example.py\n```\n\nThis script will:\n- load `target.csv` to MongoDB\n- download dataset from MongoDB\n- Run Evidently Model Profile and Evidently Report on this data\n- Save Profile data back to MongoDB\n- Save Report to `evidently_report_example.html`\n\nYou can look at Prefect steps in Prefect Orion UI\n(to start it execute `prefect orion start`)\n"
},
{
"path": "cohorts/2022/05-monitoring/homework/docker-compose-homework-solution.yml",
"content": "version: \"3.7\"\n\nvolumes:\n mongo_data: {}\n\nnetworks:\n front-tier:\n back-tier:\n\nservices:\n prediction_service:\n build:\n context: prediction_service\n dockerfile: Dockerfile\n depends_on:\n - mongo\n environment:\n MONGO_DATABASE: \"prediction_service\"\n MONGO_ADDRESS: \"mongodb://mongo.:27017/\"\n MODEL_VERSION: \"2\"\n MODEL_FILE: \"lin_reg_V2.bin\"\n \n ports:\n - 9696:9696\n networks:\n - back-tier\n - front-tier\n\n mongo:\n image: mongo\n ports:\n - 27017:27017\n volumes:\n - mongo_data:/data/db\n networks:\n - back-tier\n - front-tier\n"
},
{
"path": "cohorts/2022/05-monitoring/homework/docker-compose-homework.yml",
"content": "version: \"3.7\"\n\nvolumes:\n mongo_data: {}\n\nnetworks:\n front-tier:\n back-tier:\n\nservices:\n prediction_service:\n build:\n context: prediction_service\n dockerfile: Dockerfile\n depends_on:\n - mongo\n environment:\n MONGO_DATABASE: \"prediction_service\"\n MONGO_ADDRESS: \"mongodb://mongo.:27017/\"\n MODEL_VERSION: \"1\"\n MODEL_FILE: \"lin_reg.bin\"\n \n ports:\n - 9696:9696\n networks:\n - back-tier\n - front-tier\n\n mongo:\n image: mongo\n ports:\n - 27017:27017\n volumes:\n - mongo_data:/data/db\n networks:\n - back-tier\n - front-tier\n"
},
{
"path": "cohorts/2022/05-monitoring/homework/model_training.py",
"content": "import pickle\n\nimport pandas as pd\nimport pyarrow.parquet as pq\nfrom sklearn.feature_extraction import DictVectorizer\nfrom sklearn.linear_model import LinearRegression\n\n\ndef read_dataframe(filename):\n df = pq.read_table(filename).to_pandas()\n\n df.lpep_dropoff_datetime = pd.to_datetime(df.lpep_dropoff_datetime)\n df.lpep_pickup_datetime = pd.to_datetime(df.lpep_pickup_datetime)\n\n df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\n df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)\n\n df = df[(df.duration >= 1) & (df.duration <= 60)]\n\n categorical = ['PULocationID', 'DOLocationID']\n df[categorical] = df[categorical].astype(str)\n \n return df\n\ndef add_features(train_data=\"./datasets/green_tripdata_2021-03.parquet\",\n additional_training_data=None):\n df_train = read_dataframe(train_data)\n\n if additional_training_data:\n extra_data = read_dataframe(additional_training_data)\n df_train = pd.concat([df_train, extra_data], axis=0, ignore_index=True)\n\n\n\n df_train['PU_DO'] = df_train['PULocationID'] + '_' + df_train['DOLocationID']\n\n categorical = ['PU_DO'] \n numerical = ['trip_distance']\n\n dv = DictVectorizer()\n\n train_dicts = df_train[categorical + numerical].to_dict(orient='records')\n X_train = dv.fit_transform(train_dicts)\n\n target = 'duration'\n y_train = df_train[target].values\n\n return X_train, y_train, dv\n\n\n\n\nif __name__ == \"__main__\":\n X_train, y_train, dv = add_features()\n \n print(\"Training model with one month of data\")\n lr = LinearRegression()\n lr.fit(X_train, y_train)\n\n \n with open('prediction_service/lin_reg.bin', 'wb') as f_out:\n pickle.dump((dv, lr), f_out)\n\n X_train, y_train, dv = add_features(additional_training_data=\"./datasets/green_tripdata_2021-04.parquet\")\n print(\"Training model with two months of data\")\n lr = LinearRegression()\n lr.fit(X_train, y_train)\n\n with open('prediction_service/lin_reg_V2.bin', 'wb') as f_out:\n pickle.dump((dv, lr), f_out)\n"
},
{
"path": "cohorts/2022/05-monitoring/homework/prediction_service/Dockerfile",
"content": "FROM python:3.8-slim-buster\n\nRUN pip install -U pip\nRUN pip install pipenv \n\nWORKDIR /app\n\nCOPY [ \"Pipfile\", \"Pipfile.lock\", \"./\" ]\n\nRUN pipenv install --system --deploy\n\nCOPY [ \"app.py\", \"lin_reg.bin\", \"lin_reg_V2.bin\", \"./\" ]\n\nEXPOSE 9696\n\nENTRYPOINT [\"gunicorn\", \"--bind=0.0.0.0:9696\", \"app:app\" ]\n"
},
{
"path": "cohorts/2022/05-monitoring/homework/prediction_service/Pipfile",
"content": "[[source]]\nurl = \"https://pypi.org/simple\"\nverify_ssl = true\nname = \"pypi\"\n\n[packages]\nscikit-learn = \"==1.0.2\"\nflask = \"==2.0.1\"\npandas = \"==1.1.5\"\nevidently = \"*\"\npymongo = \"*\"\ngunicorn = \"*\"\n\n[dev-packages]\npyarrow = \"*\"\n\n[requires]\npython_version = \"3.8\"\n"
},
{
"path": "cohorts/2022/05-monitoring/homework/prediction_service/app.py",
"content": "import logging\nimport os\nimport pickle\nimport uuid\n\nfrom flask import Flask, jsonify, request\nfrom pymongo import MongoClient\n\n\nMONGO_ADDRESS = os.getenv(\"MONGO_ADDRESS\", \"mongodb://localhost:27017/\")\nMONGO_DATABASE = os.getenv(\"MONGO_DATABASE\", \"ride_prediction\")\nLOGGED_MODEL = os.getenv(\"MODEL_FILE\", \"lin_reg.bin\")\nMODEL_VERSION = os.getenv(\"MODEL_VERSION\", \"1\")\n\nwith open(LOGGED_MODEL, 'rb') as f_in:\n dv, model = pickle.load(f_in)\n\n\nmongo_client = MongoClient(MONGO_ADDRESS)\nmongo_db = mongo_client[MONGO_DATABASE]\nmongo_collection = mongo_db.get_collection(\"data\")\n\n\napp = Flask(\"Ride-Prediction-Service\")\nlogging.basicConfig(level=logging.INFO)\n\n\ndef prepare_features(ride):\n \"\"\"Function to prepare features before making prediction\"\"\"\n\n record = ride.copy()\n record['PU_DO'] = '%s_%s' % (record['PULocationID'], record['DOLocationID'])\n\n features = dv.transform([record])\n \n return features, record\n\n\ndef save_db(record, pred_result):\n \"\"\"Save data to mongo db collection\"\"\"\n\n rec = record.copy()\n rec[\"prediction\"] = pred_result[0]\n mongo_collection.insert_one(rec)\n\n\n\n@app.route(\"/\", methods=[\"GET\"])\ndef get_info():\n \"\"\"Function to provide info about the app\"\"\"\n info = \"\"\"\"ride = {\n \"PULocationID\": 10,\n \"DOLocationID\": 50,\n \"trip_distance\": 40\n }\"\n
\n| Name | \nProject | \nSocial | \nLinks and comments | \n
|---|---|---|---|
| Ashish Lalchandani | \nProject | \n ![](\"https://user-images.githubusercontent.com/875246/192300614-2ce22ed5-bbc4-4684-8098-d8128d71aac5.png\") ![](\"https://user-images.githubusercontent.com/875246/192300611-a606521b-cb76-4090-be8e-7cc21752b996.png\") | \n\n |
| Purity Nyagweth | \nProject | \n | \n\n |
| Giovanni Pecoraro | \nProject | \n | \n\n |
| Nakul Bajaj | \nProject | \n | \n\n |
| Marcos M. Jiménez | \nProject | \n | \n\n More info\n\n\n\n> I am still improving the project. A basic model, but the important is everything around it. | \n
| Mohammed Ayoub Chettouh | \nProject | \n | \n\n More info\n\n\nLinks:\n\n
| \n
| Pedro Munoz | \nProject | \n | \n\n |
| Oluwatosin Olajide | \nProject | \n | \n\n |
| Sergei Zemskov | \nProject | \n | \n\n |
| Esteban Encina | \nProject | \n | \n\n |
| Rizdi Aprilian | \nProject | \n | \n\n |
| Karim Lulu | \nProject | \n | \n\n |
| Babatunde Ayanlola Emmanuel Ayanlowo | \n\n | | \n\n More info\n\n\nLinks:\n\n\n\n> Experts in SQL Language, Analytic Engineering &Product Management Using Scrum Framework. | \n
| Camilo A. Caceres F. | \nProject | \n | \n\n More info\n\n\nLinks:\n\n
| \n
| Yan Vlasov | \nProject | \n | \n\n |
| Low Kim Hoe | \nProject | \n | \n\n |
| Artem Glazkov | \nProject | \n | \n\n |
| Jan Weber | \nProject | \n | \n\n |
| Icaro Bombonato | \nProject | \n | \n\n More info\n\n\n\n> Feel free to contact me :) | \n
| Adam Gifford | \nProject | \n | \n\n More info\n\n\nLinks:\n\n\n\n> I am the lead data scientist at BehaVR, a digital health and wellness startup looking to translate proven science into immersive and engaging XR therapeutics for mental and behavioral health. As we move towards our long-term goal of live operational monitoring and scoring of user activity to personalize experiences dynamically, this project serves as a foundation in framework development for real-time serving and monitoring of a prediction service to classify continuous-signal data. | \n
| Artem Li | \nProject | \n | \n\n |
| Adesoba Adewale Olamide | \nProject | \n | \n\n |
| Amir | \nProject | \n | \n\n |
| Antonio Damiao Rodrigues | \nProject | \n | \n\n |
| Walid Chrimni | \nProject | \n | \n\n |
| Shivam Kurtarkar | \nProject | \n | \n\n |
| Vyacheslav Shen | \nProject | \n | \n\n More info\n\n\nLinks:\n\n | \n
| Venkat Ramakrishnan | \n\n | \n | \n |
| Deepak Ramani | \nProject | \n | \n\n More info\n\n\nLinks:\n\n
| \n
| Frauke Albrecht | \nProject | \n | \n\n |
| Rinat Gareev | \nProject | \n | \n\n More info\n\n\n\n> In my job the most of MLOps experience I have is with Kubeflow or Amazon SageMaker (AWS). I used this project to learn something new and evaluate new frameworks, for example, Ray. Thanks for the opportunity and the course! | \n
| Isaac Omolayo | \nProject | \n | \n\n |
| Antonis Stellas | \nProject | \n | \n\n |
| Ayush Bihani | \n\n | | \n\n |
| Ruslan Romanov | \nProject | \n | \n\n More info\n\n\n\n> Learning process was very exciting | \n
| Pablo Dalchiele | \nProject | \n | \n\n |
| Nahum Sá | \nProject | \n | \n\n |
| Patrick Walukagga | \nProject | \n | \n\n More info\n\n\n\n> Enthusiastic Data Sciencist/Data Engineer/ML Engineer with a strong background in backend software engineering with Python. | \n
| Moiz Sajid | \nProject | \n | \n\n More info\n\n\n\n> Thank you to the DataTalks.Club team for organizing this amazing course that covers the latest technologies and trends in MLOps! | \n
| Yusuf Yelken | \nProject | \n | \n\n |
| Alexander Gabriel Valverde Guillen | \nProject | \n | \n\n |
| Carlos Pereira | \nProject | \n | \n\n |
| Daniel Hen | \n\n | | \n\n |
| bryskulov | \nProject | \n | \n\n |
| Surawut Jirasaktavee | \nProject | \n | \n\n |
| Sai Teja Macharla | \n\n | | \n\n |
| Senthilkumar Gopal | \nProject | \n | \n\n More info\n\n\nLinks:\n\n
| \n
| Tim Becker | \n\n | | \n\n |
| Ezinwanne Aka | \n\n | | \n\n |
| Ammar Chalifah | \nProject | \n | \n\n |
| Nwoke Tochukwu | \n\n | | \n\n |
| Alessandro S. | \nProject | \n | \n\n |
| Hakeem Kazeem | \n\n | | \n\n |
| Isaac Kargar | \nProject | \n | \n\n More info\n\n\nLinks:\n\n
| \n
| Yalcin Erbasi | \nProject | \n | \n\n |
\n\n## 3.2 Introduction to Prefect\n\n\n \n\n\n## 3.3 Prefect Workflow\n\n\n \n\n## 3.4 Deploying Your Workflow\n\n\n \n\n\n## 3.5 Working with Deployments\n\n\n \n\n\n## 3.6 Prefect Cloud (optional)\n\n\n \n\n\n## 3.7 Homework\n\nMore information [here](../cohorts/2023/03-orchestration/homework.md).\n\n## Quick setup\n\n### Install packages\n\nIn a conda environment with Python 3.10.12 or similar, install all package dependencies with\n\n```bash\npip install -r requirements.txt\n```\n\n### Start the Prefect server locally\n\nCreate another window and activate your conda environment. Start the Prefect API server locally with\n\n```bash\nprefect server start\n```\n\n### Alternative to self-hosted server use Prefect Cloud for added capabilties\n\nSignup and use for free at | \\n\",\n \" | VendorID | \\n\",\n \"tpep_pickup_datetime | \\n\",\n \"tpep_dropoff_datetime | \\n\",\n \"passenger_count | \\n\",\n \"trip_distance | \\n\",\n \"RatecodeID | \\n\",\n \"store_and_fwd_flag | \\n\",\n \"PULocationID | \\n\",\n \"DOLocationID | \\n\",\n \"payment_type | \\n\",\n \"... | \\n\",\n \"extra | \\n\",\n \"mta_tax | \\n\",\n \"tip_amount | \\n\",\n \"tolls_amount | \\n\",\n \"improvement_surcharge | \\n\",\n \"total_amount | \\n\",\n \"congestion_surcharge | \\n\",\n \"Airport_fee | \\n\",\n \"duration | \\n\",\n \"ride_id | \\n\",\n \"
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \\n\",\n \"2 | \\n\",\n \"2023-03-01 00:06:43 | \\n\",\n \"2023-03-01 00:16:43 | \\n\",\n \"1.0 | \\n\",\n \"0.00 | \\n\",\n \"1.0 | \\n\",\n \"N | \\n\",\n \"238 | \\n\",\n \"42 | \\n\",\n \"2 | \\n\",\n \"... | \\n\",\n \"1.0 | \\n\",\n \"0.5 | \\n\",\n \"0.00 | \\n\",\n \"0.0 | \\n\",\n \"1.0 | \\n\",\n \"11.10 | \\n\",\n \"0.0 | \\n\",\n \"0.00 | \\n\",\n \"10.000000 | \\n\",\n \"2023/03_0 | \\n\",\n \"
| 1 | \\n\",\n \"2 | \\n\",\n \"2023-03-01 00:08:25 | \\n\",\n \"2023-03-01 00:39:30 | \\n\",\n \"2.0 | \\n\",\n \"12.40 | \\n\",\n \"1.0 | \\n\",\n \"N | \\n\",\n \"138 | \\n\",\n \"231 | \\n\",\n \"1 | \\n\",\n \"... | \\n\",\n \"6.0 | \\n\",\n \"0.5 | \\n\",\n \"12.54 | \\n\",\n \"0.0 | \\n\",\n \"1.0 | \\n\",\n \"76.49 | \\n\",\n \"2.5 | \\n\",\n \"1.25 | \\n\",\n \"31.083333 | \\n\",\n \"2023/03_1 | \\n\",\n \"
| 2 | \\n\",\n \"1 | \\n\",\n \"2023-03-01 00:15:04 | \\n\",\n \"2023-03-01 00:29:26 | \\n\",\n \"0.0 | \\n\",\n \"3.30 | \\n\",\n \"1.0 | \\n\",\n \"N | \\n\",\n \"140 | \\n\",\n \"186 | \\n\",\n \"1 | \\n\",\n \"... | \\n\",\n \"3.5 | \\n\",\n \"0.5 | \\n\",\n \"4.65 | \\n\",\n \"0.0 | \\n\",\n \"1.0 | \\n\",\n \"28.05 | \\n\",\n \"2.5 | \\n\",\n \"0.00 | \\n\",\n \"14.366667 | \\n\",\n \"2023/03_2 | \\n\",\n \"
| 3 | \\n\",\n \"1 | \\n\",\n \"2023-03-01 00:49:37 | \\n\",\n \"2023-03-01 01:01:05 | \\n\",\n \"1.0 | \\n\",\n \"2.90 | \\n\",\n \"1.0 | \\n\",\n \"N | \\n\",\n \"140 | \\n\",\n \"43 | \\n\",\n \"1 | \\n\",\n \"... | \\n\",\n \"3.5 | \\n\",\n \"0.5 | \\n\",\n \"4.10 | \\n\",\n \"0.0 | \\n\",\n \"1.0 | \\n\",\n \"24.70 | \\n\",\n \"2.5 | \\n\",\n \"0.00 | \\n\",\n \"11.466667 | \\n\",\n \"2023/03_3 | \\n\",\n \"
| 4 | \\n\",\n \"2 | \\n\",\n \"2023-03-01 00:08:04 | \\n\",\n \"2023-03-01 00:11:06 | \\n\",\n \"1.0 | \\n\",\n \"1.23 | \\n\",\n \"1.0 | \\n\",\n \"N | \\n\",\n \"79 | \\n\",\n \"137 | \\n\",\n \"1 | \\n\",\n \"... | \\n\",\n \"1.0 | \\n\",\n \"0.5 | \\n\",\n \"2.44 | \\n\",\n \"0.0 | \\n\",\n \"1.0 | \\n\",\n \"14.64 | \\n\",\n \"2.5 | \\n\",\n \"0.00 | \\n\",\n \"3.033333 | \\n\",\n \"2023/03_4 | \\n\",\n \"
5 rows × 21 columns
\\n\",\n \"| \\n\",\n \" | VendorID | \\n\",\n \"tpep_pickup_datetime | \\n\",\n \"tpep_dropoff_datetime | \\n\",\n \"passenger_count | \\n\",\n \"trip_distance | \\n\",\n \"RatecodeID | \\n\",\n \"store_and_fwd_flag | \\n\",\n \"PULocationID | \\n\",\n \"DOLocationID | \\n\",\n \"payment_type | \\n\",\n \"fare_amount | \\n\",\n \"extra | \\n\",\n \"mta_tax | \\n\",\n \"tip_amount | \\n\",\n \"tolls_amount | \\n\",\n \"improvement_surcharge | \\n\",\n \"total_amount | \\n\",\n \"congestion_surcharge | \\n\",\n \"airport_fee | \\n\",\n \"
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \\n\",\n \"2 | \\n\",\n \"2023-01-01 00:32:10 | \\n\",\n \"2023-01-01 00:40:36 | \\n\",\n \"1.0 | \\n\",\n \"0.97 | \\n\",\n \"1.0 | \\n\",\n \"N | \\n\",\n \"161 | \\n\",\n \"141 | \\n\",\n \"2 | \\n\",\n \"9.3 | \\n\",\n \"1.00 | \\n\",\n \"0.5 | \\n\",\n \"0.00 | \\n\",\n \"0.0 | \\n\",\n \"1.0 | \\n\",\n \"14.30 | \\n\",\n \"2.5 | \\n\",\n \"0.00 | \\n\",\n \"
| 1 | \\n\",\n \"2 | \\n\",\n \"2023-01-01 00:55:08 | \\n\",\n \"2023-01-01 01:01:27 | \\n\",\n \"1.0 | \\n\",\n \"1.10 | \\n\",\n \"1.0 | \\n\",\n \"N | \\n\",\n \"43 | \\n\",\n \"237 | \\n\",\n \"1 | \\n\",\n \"7.9 | \\n\",\n \"1.00 | \\n\",\n \"0.5 | \\n\",\n \"4.00 | \\n\",\n \"0.0 | \\n\",\n \"1.0 | \\n\",\n \"16.90 | \\n\",\n \"2.5 | \\n\",\n \"0.00 | \\n\",\n \"
| 2 | \\n\",\n \"2 | \\n\",\n \"2023-01-01 00:25:04 | \\n\",\n \"2023-01-01 00:37:49 | \\n\",\n \"1.0 | \\n\",\n \"2.51 | \\n\",\n \"1.0 | \\n\",\n \"N | \\n\",\n \"48 | \\n\",\n \"238 | \\n\",\n \"1 | \\n\",\n \"14.9 | \\n\",\n \"1.00 | \\n\",\n \"0.5 | \\n\",\n \"15.00 | \\n\",\n \"0.0 | \\n\",\n \"1.0 | \\n\",\n \"34.90 | \\n\",\n \"2.5 | \\n\",\n \"0.00 | \\n\",\n \"
| 3 | \\n\",\n \"1 | \\n\",\n \"2023-01-01 00:03:48 | \\n\",\n \"2023-01-01 00:13:25 | \\n\",\n \"0.0 | \\n\",\n \"1.90 | \\n\",\n \"1.0 | \\n\",\n \"N | \\n\",\n \"138 | \\n\",\n \"7 | \\n\",\n \"1 | \\n\",\n \"12.1 | \\n\",\n \"7.25 | \\n\",\n \"0.5 | \\n\",\n \"0.00 | \\n\",\n \"0.0 | \\n\",\n \"1.0 | \\n\",\n \"20.85 | \\n\",\n \"0.0 | \\n\",\n \"1.25 | \\n\",\n \"
| 4 | \\n\",\n \"2 | \\n\",\n \"2023-01-01 00:10:29 | \\n\",\n \"2023-01-01 00:21:19 | \\n\",\n \"1.0 | \\n\",\n \"1.43 | \\n\",\n \"1.0 | \\n\",\n \"N | \\n\",\n \"107 | \\n\",\n \"79 | \\n\",\n \"1 | \\n\",\n \"11.4 | \\n\",\n \"1.00 | \\n\",\n \"0.5 | \\n\",\n \"3.28 | \\n\",\n \"0.0 | \\n\",\n \"1.0 | \\n\",\n \"19.68 | \\n\",\n \"2.5 | \\n\",\n \"0.00 | \\n\",\n \"
![](\\\"images/thumbnail-5-01.jpg\\\")
\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"processing video yixA3C1xSxc...\\n\",\n \"saved to ..\\\\05-monitoring\\\\images\\\\thumbnail-5-02.jpg\\n\",\n \"## 5.2 Environment setup\\n\",\n \"\\n\",\n \"\\n\",\n \" ![](\\\"images/thumbnail-5-02.jpg\\\")
\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"processing video IjNrkqMYQeQ...\\n\",\n \"saved to ..\\\\05-monitoring\\\\images\\\\thumbnail-5-03.jpg\\n\",\n \"## 5.3 Prepare reference and model\\n\",\n \"\\n\",\n \"\\n\",\n \" ![](\\\"images/thumbnail-5-03.jpg\\\")
\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"processing video kP3lzh_HfWY...\\n\",\n \"saved to ..\\\\05-monitoring\\\\images\\\\thumbnail-5-04.jpg\\n\",\n \"## 5.4 Evidently metrics calculation\\n\",\n \"\\n\",\n \"\\n\",\n \" ![](\\\"images/thumbnail-5-04.jpg\\\")
\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"processing video s3G4PMsOMOA...\\n\",\n \"saved to ..\\\\05-monitoring\\\\images\\\\thumbnail-5-05.jpg\\n\",\n \"## 5.5 Dummy monitoring\\n\",\n \"\\n\",\n \"\\n\",\n \" ![](\\\"images/thumbnail-5-05.jpg\\\")
\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"processing video fytrmPbcLhI...\\n\",\n \"saved to ..\\\\05-monitoring\\\\images\\\\thumbnail-5-06.jpg\\n\",\n \"## 5.6 Data quality monitoring\\n\",\n \"\\n\",\n \"\\n\",\n \" ![](\\\"images/thumbnail-5-06.jpg\\\")
\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"processing video -c4iumyZMyw...\\n\",\n \"saved to ..\\\\05-monitoring\\\\images\\\\thumbnail-5-07.jpg\\n\",\n \"## 5.7 Save Grafana Dashboard\\n\",\n \"\\n\",\n \"\\n\",\n \" ![](\\\"images/thumbnail-5-07.jpg\\\")
\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"processing video sNSk3ojISh8...\\n\",\n \"saved to ..\\\\05-monitoring\\\\images\\\\thumbnail-5-08.jpg\\n\",\n \"## 5.8 Debugging with test suites and reports\\n\",\n \"\\n\",\n \"\\n\",\n \" ![](\\\"images/thumbnail-5-08.jpg\\\")
\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\"\n ]\n }\n ],\n \"source\": [\n \"module_number = module_info['number']\\n\",\n \"\\n\",\n \"parts = []\\n\",\n \"\\n\",\n \"for unit in units:\\n\",\n \" unit_number = unit['number']\\n\",\n \"\\n\",\n \" params = {\\n\",\n \" 'module_number': module_info['number'],\\n\",\n \" 'module_name': module_info['title'],\\n\",\n \" 'unit_number': unit['number'],\\n\",\n \" 'unit_title': unit['title'] \\n\",\n \" }\\n\",\n \"\\n\",\n \" if 'youtube' in unit:\\n\",\n \" youtube = unit['youtube']\\n\",\n \" if len(youtube) and youtube.startswith('https'):\\n\",\n \" thumbnail = download_thumbnail(youtube, module_number, unit_number, images_folder)\\n\",\n \" thumbnail_path = '/'.join(thumbnail.parts[2:])\\n\",\n \" params['youtube'] = f'{youtube}&list={playlist_id}'\\n\",\n \" params['thumbnail'] = thumbnail_path\\n\",\n \" \\n\",\n \" template_string = template.render(params)\\n\",\n \" print(template_string)\\n\",\n \" parts.append(template_string)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 30,\n \"id\": \"6d6e3584\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"prefix = f\\\"\\\"\\\"\\n\",\n \"# {module_info['number']}. {module_info['title']} \\n\",\n \"\\\"\\\"\\\".strip()\\n\",\n \"\\n\",\n \"\\n\",\n \"final_result = '\\\\n\\\\n'.join([prefix] + parts)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 31,\n \"id\": \"4d900666\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"clipboard.copy(final_result)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"6ff9d662\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.13\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
},
{
"path": "learning-in-public.md",
"content": "# Learning in public\n\nMost people learn in private: they consume content but don't tell\nanyone about it. There's nothing wrong with it.\n\nBut we want to encourage you to document your progress and\nshare it publicly on social media.\n\nIt helps you get noticed and will lead to:\n\n* Expanding your network: meeting new people and making new friends\n* Being invited to meetups, conferences and podcasts\n* Landing a job or getting clients\n* Many other good things\n\nHere's a more compresensive reading on why you want to do it: https://github.com/readme/guides/publishing-your-work\n\n\n## Learning in Public for Zoomcamps\n\nWhen you submit your homework or project, you can also submit\nlearning in public posts:\n\n![](\"./images/learning-in-public-links.png\")
\n\nYou can watch this video to see how your learning in public posts may look like:\n\n\n ![](\"./images/learning-in-public.png\")
\n\n\n\nSend a PR if you want to suggest improvements for this document"
}
]