## 1.2 Environment preparation
### 1.2.1 GitHub Codespaces
OR
### 1.2.2 VM in AWS
**Note**: You don't have to rent an instance in the cloud. You can follow the same instructions
for setting up your local environment.
Code:
Recommended development environment: Linux
### Step 1: Download and install the Anaconda distribution of Python
```sh
wget https://repo.anaconda.com/archive/Anaconda3-2022.05-Linux-x86_64.sh
bash Anaconda3-2022.05-Linux-x86_64.sh
```
### Step 2: Update existing packages
```sh
sudo apt update
```
### Step 3: Install Docker and Docker Compose
Follow the instructions here:
[install-using-the-repository](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository)
Set up Docker's apt repository.
```sh
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```
Install the Docker packages.
```sh
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```
To run docker without `sudo`:
```sh
sudo groupadd docker
sudo usermod -aG docker $USER
```
### Step 4: Run Docker
```sh
docker run hello-world
```
If you get `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create": dial unix /var/run/docker.sock: connect: permission denied.` error, restart your VM instance, or run:
`sudo dockerd`
**Note**: If you get `It is required that your private key files are NOT accessible by others. This private key will be ignored.` error, you should change permits on the downloaded file to protect your private key:
```sh
chmod 400 name-of-your-private-key-file.pem
```
## 1.3 (Optional) Training a ride duration prediction model
**Note**: The NYC taxi data is now in parquet format, not CSV.
Here's a [video](https://www.youtube.com/watch?v=r94QjpX9vSE&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK) that explains how to
read parquet data.
Links:
* [Notebook](duration-prediction.ipynb)
## 1.4 Course overview
## 1.5 MLOps maturity model
Links:
* [MLOps Maturity model](https://docs.microsoft.com/en-us/azure/architecture/example-scenario/mlops/mlops-maturity-model)
## 1.6 Homework
More information [here](../cohorts/2025/01-intro/homework.md).
## Notes
Did you take notes? Add them here:
* [Local MacOS envprep by Adetbekov](https://github.com/adetbekov/mlops-zoomcamp/blob/main/01-intro/macos-local-envprep.md)
* [Environment Setup by Ayoub](https://github.com/ayoub-berdeddouch/mlops-journey/blob/main/intro-01.md)
* [Intro, Environment Setup, and MLOps Maturity Models by Bala](https://github.com/balapriyac/DTC-MLOps-Zoomcamp/tree/main/week1)
* [GCP Environment Setup by Piyush](https://github.com/piyush-an/MLOps-ZoomCamp/blob/main/01-Introduction/infrastructure.md)
* [Microsoft Azure Environment Setup by Olaide](https://github.com/josepholaide/MLOps-Practice/blob/main/Week%201/README.md)
* [Environment Preparation using GCP and pyenv by Dani](https://github.com/syahrulhamdani/dtc-mlops/blob/main/week-1-introduction/README.md)
* [Useful links by Zioalex](https://github.com/zioalex/mlops-zoomcamp/blob/main/My_notes_week1.md)
* [Notes by Alvaro Navas](https://github.com/ziritrion/mlopszoomcamp/blob/main/notes/1_intro.md)
* [Environment setup on GCP (recycled from Data Engineering Zoomcamp)](https://gist.github.com/ziritrion/3214aa570e15ae09bf72c4587cb9d686)
* [Docker cheatsheet](https://gist.github.com/ziritrion/1842c8a4c4851602a8733bba19ab6050)
* [Conda cheatsheet](https://gist.github.com/ziritrion/8024025672ea92b8bdeb320d6015aa0d)
* [Git cheatsheet](https://gist.github.com/ziritrion/d73ca65bf4d19c79ca842a55853cb962)
* [Python and data libraries cheatseet](https://gist.github.com/ziritrion/9b80e47956adc0f20ecce209d494cd0a)
* [Cloud server setup using Deploifai (on AWS, Azure, or GCP) by 98sean98](https://github.com/98sean98/mlops-zoomcamp/blob/main/01-intro/deploifai-server/readme.md)
* [Notes from first lesson by Neimv](https://gitlab.com/neimv/mlops/-/blob/main/lessons_weeks/notes_1.md)
* [Course preliminaries (jupyterbook) by particle1331](https://particle1331.github.io/ok-transformer/nb/mlops/01-intro.html)
* [Notes by Francisco Delca (environment: local ubuntu + virtualvenv)](https://github.com/FDelca/mlops_datatalks_notes/blob/main/Week1/Week1-LearningNotes.ipynb)
* [What is MLOps? - Non-technical intro by Lorenz](https://github.com/LoHertel/Road-to-MLOps/blob/main/01-primer/README.md)
* [Notes for Week1 by Bhagabat](https://github.com/BPrasad123/MLOps_Zoomcamp/tree/main/Week1)
* [What is MLOps? - Storytime by Pathik Ghugare](https://github.com/pathikg/MLOps-Zoomcamp-DataTalks/blob/main/week-01-introduction/MLOps-Introduction.md)
* [Module 1 Notes by Stephen Barrie](https://stephen137.github.io/posts/MLOps_Zoomcamp_Week_1/MLOps_Zoomcamp_Week_1.html)
* [Module 1 Notes - MacOS + Pycharm by Hongfan (Amber)](https://github.com/Muhongfan/MLops/blob/main/01-intro/README.md)
* [Comprehensive guide to MLOps: Theory & Concepts for Beginners by Nithish Prabhu](https://ntp3105.github.io/Comprehensive-MLOps/Week-1/Introduction%20to%20MLOps.html)
* [Module 1 Notes -Video 1.2 Env Prep Ec2 troubleshooting notes for remote-ssh vscode to ec2](https://github.com/thatwonguy/mlops-zoomcamp/blob/personal/01-intro/README.md)
* [Ch1 notes by Marcus](https://github.com/mleiwe/mlops-zoomcamp/blob/Ch1_Marcus/cohorts/2024/01-intro/Ch1_Notes.md)
* [GCP setup instructions](https://github.com/mleiwe/mlops-zoomcamp/blob/Ch1_Marcus/cohorts/2024/01-intro/GoogleCloudSetUpNotes.md)
* [Machine Learning Essentials: FAQs on Linear Regression, One-Hot Encoding & RMSE](https://github.com/niting9881/course-mlops-zoomcamp/blob/main/01-intro/README.md)
* [GCP Environment Setup with Terraform + Cloud-Init by Artem](https://github.com/Ladn0/mlops-zoomcamp/tree/main/01-intro/notes)
* [Week 1 - local setup hints by hannarud](https://github.com/hannarud/mlops-zoomcamp-2025/blob/main/week1_notes.md)
* [Week-1 - Introduction to MLOps, Env Setup,Notebooks detailed Notes by Muhammad Shifa](https://github.com/MuhammadShifa/mlops-zoomcamp2025/tree/main/01-introduction#readme)
* Send a PR, add your notes above this line
================================================
FILE: 01-intro/duration-prediction.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "b2bd82d7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Python 3.9.7\r\n"
]
}
],
"source": [
"!python -V"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "41062d8a",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "c984c564",
"metadata": {},
"outputs": [],
"source": [
"import pickle"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "4add538c",
"metadata": {},
"outputs": [],
"source": [
"import seaborn as sns\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "8b135c2b",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.feature_extraction import DictVectorizer\n",
"from sklearn.linear_model import LinearRegression\n",
"from sklearn.linear_model import Lasso\n",
"from sklearn.linear_model import Ridge\n",
"\n",
"from sklearn.metrics import root_mean_squared_error"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "8e013caf",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_parquet('./data/green_tripdata_2021-01.parquet')\n",
"\n",
"df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\n",
"df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)\n",
"\n",
"df = df[(df.duration >= 1) & (df.duration <= 60)]\n",
"\n",
"categorical = ['PULocationID', 'DOLocationID']\n",
"numerical = ['trip_distance']\n",
"\n",
"df[categorical] = df[categorical].astype(str)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "116ff0cc",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"9.775464208836793"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_dicts = df[categorical + numerical].to_dict(orient='records')\n",
"\n",
"dv = DictVectorizer()\n",
"X_train = dv.fit_transform(train_dicts)\n",
"\n",
"target = 'duration'\n",
"y_train = df[target].values\n",
"\n",
"lr = LinearRegression()\n",
"lr.fit(X_train, y_train)\n",
"\n",
"y_pred = lr.predict(X_train)\n",
"\n",
"root_mean_squared_error(y_train, y_pred)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "b134c97b",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/ubuntu/anaconda3/lib/python3.9/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).\n",
" warnings.warn(msg, FutureWarning)\n",
"/home/ubuntu/anaconda3/lib/python3.9/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).\n",
" warnings.warn(msg, FutureWarning)\n"
]
},
{
"data": {
"text/plain": [
"
## 2.2 Getting started with MLflow
Note: in the videos, Cristian uses Jupyter in VS code and runs everything locally
But if you set up a VM in the previous module, you can keep using it
and use the usual Jupyter from your browser. There's no significant
difference between using Jupyter with VS code and without
## 2.3 Experiment tracking with MLflow
## 2.4 Model management
## 2.5 Model registry
> **Starting MLflow 2.9, model registry stages are deprecated.**
Please use model version tags and aliases instead of stages. For example, instead of `transition_model_version_stage(name, version, stage)` use `set_registered_model_alias(name, alias, version)`. More details [here](https://github.com/mlflow/mlflow/issues/10336) and [here](https://mlflow.org/docs/latest/model-registry.html).
## 2.6 MLflow in practice
## 2.7 MLflow: benefits, limitations and alternatives
## 2.7 Homework
More information [here](../cohorts/2025/02-experiment-tracking/homework.md).
## Notes
Did you take notes? Add them here:
* [Notes/General Docs on MLflow by Ayoub](https://gist.github.com/Qfl3x/ccff6b0708358c040e437d52af0c2e43)
* [Minimalist MLflow code reference by Anna V](https://github.com/annnvv/mlops_zoomcamp/blob/main/notes/module2_notes_MLflow.md)
* [Notes from second lesson by Neimv](https://gitlab.com/neimv/mlops/-/blob/main/lessons_weeks/notes_2.md)
* [2nd Week Experiment & Tracking notes by Ayoub.B](https://github.com/ayoub-berdeddouch/mlops-journey/blob/main/experiment_tracking_02.md)
* [Experiment tracking (jupyterbook) by particle1331](https://particle1331.github.io/ok-transformer/nb/mlops/03-mlflow.html)
* [Week 2: Experiment & Tracking Notes by Bengsoon Chuah](https://github.com/bengsoon/mlops-zoomcamp/blob/main/02-experiment-tracking/notes/Experiment_Tracking_notes.md)
* [2.4 Model Management Notes by Alvaro Pena](https://github.com/alvarofps/mlops-zoomcamp/blob/main/02-experiment-tracking/my-notes/2.4%20Model%20management.md)
* [Notes by Alvaro Navas](https://github.com/ziritrion/mlopszoomcamp/blob/main/notes/2_experiment.md)
* [Notebook from froukje](https://github.com/froukje/ml-ops-zoomcamp/blob/master/02-experiment-tracking/week02.ipynb) and [notes](https://medium.com/@falbrechtg/getting-started-with-mlflow-tracking-46a0089d6a73)
* [Blog post on setting up MLFlow on GCP by Isaac Kargar](https://kargarisaac.github.io/blog/mlops/data%20engineering/2022/06/15/MLFlow-on-GCP.html).
* [Week2: Experiment tracking notes and notebook by Bhagabat](https://github.com/BPrasad123/MLOps_Zoomcamp/tree/main/Week2)
* [Notes of ML-flow by Jaime Cabrera-Salcedo](https://github.com/jaimeh94/MLOps-Zoomcamp/tree/main/02-experiment-tracking)
* [Experiment tracking with MLflow by Hongfan (Amber)](https://github.com/Muhongfan/MLops/blob/main/02-experiment-tracking/README.md)
* [Running MLflow with Docker and on Minikube](https://open.substack.com/pub/asfandqazi/p/mlflow-on-minikube?r=2o17tf&utm_campaign=post&utm_medium=web)
* [Chapter 2: Full Notes by Marcus](https://github.com/mleiwe/mlops-zoomcamp/blob/Ch2_Marcus/cohorts/2024/02-experiment-tracking/Ch2_notes.md)
* [Experiemnt Tracking and Mlflow by Annaliese Tech](https://github.com/AnnalieseTech/MLOPS_ZOOMCAMP/blob/main/02_EXPERIMENT_TRACKING/EXPERIMENT_TRACKING_NOTES.md)
* [Adding Hyperparameter Tuning to Your Notebook with MLflow and Hyperopt by Annaliese Tech](https://github.com/AnnalieseTech/MLOPS_ZOOMCAMP/blob/main/02_EXPERIMENT_TRACKING/Hyperparameter-Tuning.md)
* [MLFlow setup and Experiment Tracking by Hokfu](https://github.com/Hokfu/MLOps_Zoomcamp_Study/blob/main/02-experiment-tracking/README.md)
* [2025 Cohort | Notes on MLflow & Hyperopt by Gabi Fonseca](https://github.com/fonsecagabriella/ml_ops/blob/main/02_experiment_tracking/__notes.md)
* [Homework 2 Experiment and Tracking article by Srikanth Ganji](https://medium.com/@srikanth.unix07/mlops-zoomcamp-2025-homework-2-experiment-tracking-with-mlflow-4ea1ed783531)
* [Week-2 - Detailed Notes of MLFlow, Experiment Tracking, Notebooks, Homework by Muhammad Shifa](https://github.com/MuhammadShifa/mlops-zoomcamp2025/blob/main/02-experiment-tracking/README.md)
* Send a PR, add your notes above this line
================================================
FILE: 02-experiment-tracking/duration-prediction.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "b2bd82d7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Python 3.9.12\n"
]
}
],
"source": [
"!python -V"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "41062d8a",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "c984c564",
"metadata": {},
"outputs": [],
"source": [
"import pickle"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "4add538c",
"metadata": {},
"outputs": [],
"source": [
"import seaborn as sns\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "8b135c2b",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.feature_extraction import DictVectorizer\n",
"from sklearn.linear_model import LinearRegression\n",
"from sklearn.linear_model import Lasso\n",
"from sklearn.linear_model import Ridge\n",
"\n",
"from sklearn.metrics import mean_squared_error"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "1464985f",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"
You'll also need to create a new key pair so later you can connect to the new instance using SSH. Click on "Create new key pair" and complete the details like in the image below:
Select the new key pair and then click on "Launch Instance".
Finally, you have to edit the security group so the EC2 instance accepts SSH (port 22) and HTTP connections (port 5000):
3. Create an s3 bucket to be used as the artifact store.
Go to s3 and click on "Create bucket". Fill in the bucket name as in the image below and let all the other configurations with their default values.
Note: s3 bucket names must be unique across all AWS account in all the AWS Regions within a partition, that means that once a bucket is created, the name of that bucket cannot be used by another AWS account within the same region. If you get an error saying that the bucket name was already taken you can fix it easily by just changing the name to something like `mlflow-artifacts-remote-2` or another name.
4. Create a new PostgreSQL database to be used as the backend store
Go to the RDS Console and click on "Create database". Make sure to select "PostgreSQL" engine type and the "Free tier" template.
Select a name for your DB instance, set the master username as "mlflow" and tick the option "Auto generate a password" so Amazon RDS generate a password automatically.
Finally, on the section "Additional configuration" specify a database name so RDS automatically creates an initial database for you.
After clicking on "launch database" you will be able to check the newly generated password, but take into account that the automatically generated password will be shown only once!
You can use the default values for all the other configurations.
Take note of the following information:
* master username
* password
* initial database name
* endpoint
Once the DB instance is created, go to the RDS console, select the new db and under "Connectivity & security" select the VPC security group. Modify the security group by adding a new inbound rule that allows postgreSQL connections on the port 5432 from the security group of the EC2 instance. This way, the server will be able to connect to the postgres database.
Note that the database name might be set to "postgres" by default.
5. Connect to the EC2 instance and launch the tracking server.
Go to the EC2 Console and find the instance launched on the step 2. Click on "Connect" and then follow the steps described in the tab "SSH".
Run the following commands to install the dependencies, configure the environment and launch the server:
* `sudo yum update`
* `pip3 install mlflow boto3 psycopg2-binary`
* `aws configure` # you'll need to input your AWS credentials here
* `mlflow server -h 0.0.0.0 -p 5000 --backend-store-uri postgresql://DB_USER:DB_PASSWORD@DB_ENDPOINT:5432/DB_NAME --default-artifact-root s3://S3_BUCKET_NAME`
Note: before launching the server, check that the instance can access the s3 bucket created in the step number 3. To do that, just run this command from the EC2 instance: `aws s3 ls`. You should see the bucket listed in the result.
6. Access the remote tracking server from your local machine.
Open a new tab on your web browser and go to this address: `http://
## 4.2 Web-services: Deploying models with Flask and Docker
[See code here](web-service/)
## 4.3 Web-services: Getting the models from the model registry (MLflow)
[See code here](web-service-mlflow/)
## 4.4 (Optional) Streaming: Deploying models with Kinesis and Lambda
Note: Since some of the steps in this video requires the use of AWS services which incur some cost on the user, it is optional to code along to this video. However, as material in Module 6 is based on the content of this video, we still highly recommended that you watch it.
[See code here](streaming/)
## 4.5 Batch: Preparing a scoring script
[See code here](batch/)
## 4.6 MLOps Zoomcamp 4.6 - Batch scoring with Mage
No video - you already know how to do it:
* Connect to MLFlow
* Create a transformation block
* Get the model from the registry, apply it
## Homework
More information [here](../cohorts/2025/04-deployment/homework.md).
## Notes
Did you take notes? Add them here:
* [Notes on model deployment (+ creating a modeling package) by Ron M.](https://particle1331.github.io/inefficient-networks/notebooks/mlops/04-deployment/notes.html)
* [Notes on Model Deployment using Google Cloud Platform, by M. Ayoub C.](https://gist.github.com/Qfl3x/de2a9b98a370749a4b17a4c94ef46185)
* [Week4: Notes on Model Deployment by Bhagabat](https://github.com/BPrasad123/MLOps_Zoomcamp/tree/main/Week4)
* [Week 4: Deployment notes by Ayoub.B](https://github.com/ayoub-berdeddouch/mlops-journey/blob/main/deployment-04.md)
* [Week 4: Deployment notes by Waleed](https://github.com/waleedayoub/mlops-zoomcamp/blob/main/cohorts/2023/04-deployment/module4notes.waleed.md)
* [Week4: Deployment: Offline (Batch), Online (Web service /w MLflow, Streaming) by Hongfan (Amber)](https://github.com/Muhongfan/MLops/blob/main/04-deployment/README.md)
* [Week 4: Deployment Notes - Marcus](https://github.com/mleiwe/mlops-zoomcamp/blob/NotesBranch/cohorts/2024/04-deployment/Ch4_Notes_ML.md)
* [Cohort 2025| ML model deployment notes by Nitin Gupta](https://github.com/niting9881/course-mlops-zoomcamp/blob/main/04-deployment/README.md)
* [week-4: Detailed notes on deployment,streaming module,notes,codes and homework by Muhammad Shifa](https://github.com/MuhammadShifa/mlops-zoomcamp2025/blob/main/04-deployment/README.md)
* Send a PR, add your notes above this line
================================================
FILE: 04-deployment/batch/Pipfile
================================================
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
scikit-learn = "==1.0.2"
prefect = "==2.0b6"
mlflow = "*"
pandas = "*"
boto3 = "*"
pyarrow = "*"
s3fs = "*"
[dev-packages]
[requires]
python_version = "3.9"
================================================
FILE: 04-deployment/batch/README.md
================================================
## Batch deployment
* Turn the notebook for training a model into a notebook for applying the model
* Turn the notebook into a script
* Clean it and parametrize
================================================
FILE: 04-deployment/batch/score.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "2e25cf66",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import uuid\n",
"import pickle\n",
"\n",
"import pandas as pd\n",
"\n",
"import mlflow\n",
"\n",
"from sklearn.feature_extraction import DictVectorizer\n",
"from sklearn.ensemble import RandomForestRegressor\n",
"from sklearn.metrics import mean_squared_error\n",
"from sklearn.pipeline import make_pipeline"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "5887deea",
"metadata": {},
"outputs": [],
"source": [
"year = 2021\n",
"month = 2\n",
"taxi_type = 'green'\n",
"\n",
"input_file = f'https://s3.amazonaws.com/nyc-tlc/trip+data/{taxi_type}_tripdata_{year:04d}-{month:02d}.parquet'\n",
"output_file = f'output/{taxi_type}/{year:04d}-{month:02d}.parquet'\n",
"\n",
"RUN_ID = os.getenv('RUN_ID', 'e1efc53e9bd149078b0c12aeaa6365df')"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "b9666e19",
"metadata": {},
"outputs": [],
"source": [
"def generate_uuids(n):\n",
" ride_ids = []\n",
" for i in range(n):\n",
" ride_ids.append(str(uuid.uuid4()))\n",
" return ride_ids\n",
"\n",
"def read_dataframe(filename: str):\n",
" df = pd.read_parquet(filename)\n",
"\n",
" df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime\n",
" df.duration = df.duration.dt.total_seconds() / 60\n",
" df = df[(df.duration >= 1) & (df.duration <= 60)]\n",
" \n",
" df['ride_id'] = generate_uuids(len(df))\n",
"\n",
" return df\n",
"\n",
"\n",
"def prepare_dictionaries(df: pd.DataFrame):\n",
" categorical = ['PULocationID', 'DOLocationID']\n",
" df[categorical] = df[categorical].astype(str)\n",
" \n",
" df['PU_DO'] = df['PULocationID'] + '_' + df['DOLocationID']\n",
"\n",
" categorical = ['PU_DO']\n",
" numerical = ['trip_distance']\n",
" dicts = df[categorical + numerical].to_dict(orient='records')\n",
" return dicts"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "6b5f0d80",
"metadata": {},
"outputs": [],
"source": [
"def load_model(run_id):\n",
" logged_model = f's3://mlflow-models-alexey/1/{RUN_ID}/artifacts/model'\n",
" model = mlflow.pyfunc.load_model(logged_model)\n",
" return model\n",
"\n",
"\n",
"def apply_model(input_file, run_id, output_file):\n",
"\n",
" df = read_dataframe(input_file)\n",
" dicts = prepare_dictionaries(df)\n",
"\n",
" \n",
" model = load_model(run_id)\n",
" y_pred = model.predict(dicts)\n",
"\n",
" df_result = pd.DataFrame()\n",
" df_result['ride_id'] = df['ride_id']\n",
" df_result['lpep_pickup_datetime'] = df['lpep_pickup_datetime']\n",
" df_result['PULocationID'] = df['PULocationID']\n",
" df_result['DOLocationID'] = df['DOLocationID']\n",
" df_result['actual_duration'] = df['duration']\n",
" df_result['predicted_duration'] = y_pred\n",
" df_result['diff'] = df_result['actual_duration'] - df_result['predicted_duration']\n",
" df_result['model_version'] = run_id\n",
" \n",
" df_result.to_parquet(output_file, index=False)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "cc2899e7",
"metadata": {},
"outputs": [],
"source": [
"apply_model(input_file=input_file, run_id=RUN_ID, output_file=output_file)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "b75bd6c9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-02.parquet 2021-03.parquet\r\n"
]
}
],
"source": [
"!ls output/green/"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "289d4bc4",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: 04-deployment/batch/score.py
================================================
#!/usr/bin/env python
# coding: utf-8
import os
import sys
import uuid
import pickle
from datetime import datetime
import pandas as pd
import mlflow
from prefect import task, flow, get_run_logger
from prefect.context import get_run_context
from dateutil.relativedelta import relativedelta
from sklearn.feature_extraction import DictVectorizer
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
from sklearn.pipeline import make_pipeline
def generate_uuids(n):
ride_ids = []
for i in range(n):
ride_ids.append(str(uuid.uuid4()))
return ride_ids
def read_dataframe(filename: str):
df = pd.read_parquet(filename)
df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime
df.duration = df.duration.dt.total_seconds() / 60
df = df[(df.duration >= 1) & (df.duration <= 60)]
df['ride_id'] = generate_uuids(len(df))
return df
def prepare_dictionaries(df: pd.DataFrame):
categorical = ['PULocationID', 'DOLocationID']
df[categorical] = df[categorical].astype(str)
df['PU_DO'] = df['PULocationID'] + '_' + df['DOLocationID']
categorical = ['PU_DO']
numerical = ['trip_distance']
dicts = df[categorical + numerical].to_dict(orient='records')
return dicts
def load_model(run_id):
logged_model = f's3://mlflow-models-alexey/1/{run_id}/artifacts/model'
model = mlflow.pyfunc.load_model(logged_model)
return model
def save_results(df, y_pred, run_id, output_file):
df_result = pd.DataFrame()
df_result['ride_id'] = df['ride_id']
df_result['lpep_pickup_datetime'] = df['lpep_pickup_datetime']
df_result['PULocationID'] = df['PULocationID']
df_result['DOLocationID'] = df['DOLocationID']
df_result['actual_duration'] = df['duration']
df_result['predicted_duration'] = y_pred
df_result['diff'] = df_result['actual_duration'] - df_result['predicted_duration']
df_result['model_version'] = run_id
df_result.to_parquet(output_file, index=False)
@task
def apply_model(input_file, run_id, output_file):
logger = get_run_logger()
logger.info(f'reading the data from {input_file}...')
df = read_dataframe(input_file)
dicts = prepare_dictionaries(df)
logger.info(f'loading the model with RUN_ID={run_id}...')
model = load_model(run_id)
logger.info(f'applying the model...')
y_pred = model.predict(dicts)
logger.info(f'saving the result to {output_file}...')
save_results(df, y_pred, run_id, output_file)
return output_file
def get_paths(run_date, taxi_type, run_id):
prev_month = run_date - relativedelta(months=1)
year = prev_month.year
month = prev_month.month
input_file = f's3://nyc-tlc/trip data/{taxi_type}_tripdata_{year:04d}-{month:02d}.parquet'
output_file = f's3://nyc-duration-prediction-alexey/taxi_type={taxi_type}/year={year:04d}/month={month:02d}/{run_id}.parquet'
return input_file, output_file
@flow
def ride_duration_prediction(
taxi_type: str,
run_id: str,
run_date: datetime = None):
if run_date is None:
ctx = get_run_context()
run_date = ctx.flow_run.expected_start_time
input_file, output_file = get_paths(run_date, taxi_type, run_id)
apply_model(
input_file=input_file,
run_id=run_id,
output_file=output_file
)
def run():
taxi_type = sys.argv[1] # 'green'
year = int(sys.argv[2]) # 2021
month = int(sys.argv[3]) # 3
run_id = sys.argv[4] # 'e1efc53e9bd149078b0c12aeaa6365df'
ride_duration_prediction(
taxi_type=taxi_type,
run_id=run_id,
run_date=datetime(year=year, month=month, day=1)
)
if __name__ == '__main__':
run()
================================================
FILE: 04-deployment/batch/score_backfill.py
================================================
from datetime import datetime
from dateutil.relativedelta import relativedelta
from prefect import flow
import score
@flow
def ride_duration_prediction_backfill():
start_date = datetime(year=2021, month=3, day=1)
end_date = datetime(year=2022, month=4, day=1)
d = start_date
while d <= end_date:
score.ride_duration_prediction(
taxi_type='green',
run_id='e1efc53e9bd149078b0c12aeaa6365df',
run_date=d
)
d = d + relativedelta(months=1)
if __name__ == '__main__':
ride_duration_prediction_backfill()
================================================
FILE: 04-deployment/batch/score_deploy.py
================================================
from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import CronSchedule
from score import ride_duration_prediction
deployment = Deployment.build_from_flow(
flow=ride_duration_prediction,
name="ride_duration_prediction",
parameters={
"taxi_type": "green",
"run_id": "e1efc53e9bd149078b0c12aeaa6365df",
},
schedule=CronSchedule(cron="0 3 2 * *"),
work_queue_name="ml",
)
deployment.apply()
================================================
FILE: 04-deployment/load_model.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 3,
"id": "d84f7be2",
"metadata": {},
"outputs": [],
"source": [
"import mlflow\n",
"\n",
"TRACKING_URL = \"http://127.0.0.1:5000\"\n",
"mlflow.set_tracking_uri(TRACKING_URL)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "293ac71d",
"metadata": {},
"outputs": [],
"source": [
"run_id = '6dd459b11b4e48dc862f4e1019d166f6'\n",
"logged_model = f'runs:/{run_id}/model'\n",
"\n",
"loaded_model = mlflow.pyfunc.load_model(logged_model)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "031645d6",
"metadata": {},
"outputs": [],
"source": [
"import mlflow\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "e5e6825e",
"metadata": {},
"outputs": [],
"source": [
"loaded_model = mlflow.pyfunc.load_model('s3://mlflow-models-alexey/1/6dd459b11b4e48dc862f4e1019d166f6/artifacts/model/')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "64f389c2",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 3,
"id": "aaf7ad20",
"metadata": {},
"outputs": [],
"source": [
"request = {\n",
" 'lpep_pickup_datetime': '2021-01-01 00:15:56',\n",
" 'PULocationID': 43,\n",
" 'DOLocationID': 151,\n",
" 'passenger_count': 1.0,\n",
" 'trip_distance': 1.01\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "6256f61d",
"metadata": {},
"outputs": [],
"source": [
"features = {}\n",
"features['PU_DO'] = '%s_%s' % (request['PULocationID'], request['DOLocationID'])\n",
"features['trip_distance'] = request['trip_distance']"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "0b3a195c",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([14.8285286])"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"loaded_model.predict(features)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6fce5f72",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: 04-deployment/meta.json
================================================
{
"module": {
"number": 4,
"title": "Model Deployment"
},
"units": [
{
"number": 1,
"title": "Three ways of deploying a model",
"youtube": "https://www.youtube.com/watch?v=JMGe4yIoBRA"
},
{
"number": 2,
"title": "Web-services: Deploying models with Flask and Docker",
"youtube": "https://www.youtube.com/watch?v=D7wfMAdgdF8"
},
{
"number": 3,
"title": "Web-services: Getting the models from the model registry (MLflow)",
"youtube": "https://www.youtube.com/watch?v=aewOpHSCkqI"
},
{
"number": 4,
"title": "(Optional) Streaming: Deploying models with Kinesis and Lambda ",
"youtube": "https://www.youtube.com/watch?v=TCqr9HNcrsI"
},
{
"number": 5,
"title": "Batch: Preparing a scoring script",
"youtube": "https://www.youtube.com/watch?v=18Lbaaeigek"
},
{
"number": 6,
"title": "MLOps Zoomcamp 4.6 - Batch: Scheduling batch scoring jobs with Prefect",
"youtube": "https://www.youtube.com/watch?v=ekT_JW213Tc"
},
{
"number": 7,
"title": "Homework",
"youtube": ""
}
]
}
================================================
FILE: 04-deployment/streaming/Dockerfile
================================================
FROM public.ecr.aws/lambda/python:3.9
RUN pip install -U pip
RUN pip install pipenv
COPY [ "Pipfile", "Pipfile.lock", "./" ]
RUN pipenv install --system --deploy
COPY [ "lambda_function.py", "./" ]
CMD [ "lambda_function.lambda_handler" ]
================================================
FILE: 04-deployment/streaming/Pipfile
================================================
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
boto3 = "*"
mlflow = "*"
scikit-learn = "==1.0.2"
[dev-packages]
[requires]
python_version = "3.9"
================================================
FILE: 04-deployment/streaming/README.md
================================================
## Machine Learning for Streaming
* Scenario
* Creating the role
* Create a Lambda function, test it
* Create a Kinesis stream
* Connect the function to the stream
* Send the records
Links
* [Tutorial: Using Amazon Lambda with Amazon Kinesis](https://docs.amazonaws.cn/en_us/lambda/latest/dg/with-kinesis-example.html)
## Code snippets
### Sending data
```bash
KINESIS_STREAM_INPUT=ride_events
aws kinesis put-record \
--stream-name ${KINESIS_STREAM_INPUT} \
--partition-key 1 \
--data "Hello, this is a test."
```
Decoding base64
```python
base64.b64decode(data_encoded).decode('utf-8')
```
Record example
```json
{
"ride": {
"PULocationID": 130,
"DOLocationID": 205,
"trip_distance": 3.66
},
"ride_id": 123
}
```
Sending this record
```bash
aws kinesis put-record \
--stream-name ${KINESIS_STREAM_INPUT} \
--partition-key 1 \
--data '{
"ride": {
"PULocationID": 130,
"DOLocationID": 205,
"trip_distance": 3.66
},
"ride_id": 156
}'
```
### Test event
```json
{
"Records": [
{
"kinesis": {
"kinesisSchemaVersion": "1.0",
"partitionKey": "1",
"sequenceNumber": "49630081666084879290581185630324770398608704880802529282",
"data": "ewogICAgICAgICJyaWRlIjogewogICAgICAgICAgICAiUFVMb2NhdGlvbklEIjogMTMwLAogICAgICAgICAgICAiRE9Mb2NhdGlvbklEIjogMjA1LAogICAgICAgICAgICAidHJpcF9kaXN0YW5jZSI6IDMuNjYKICAgICAgICB9LCAKICAgICAgICAicmlkZV9pZCI6IDI1NgogICAgfQ==",
"approximateArrivalTimestamp": 1654161514.132
},
"eventSource": "aws:kinesis",
"eventVersion": "1.0",
"eventID": "shardId-000000000000:49630081666084879290581185630324770398608704880802529282",
"eventName": "aws:kinesis:record",
"invokeIdentityArn": "arn:aws:iam::XXXXXXXXX:role/lambda-kinesis-role",
"awsRegion": "eu-west-1",
"eventSourceARN": "arn:aws:kinesis:eu-west-1:XXXXXXXXX:stream/ride_events"
}
]
}
```
### Reading from the stream
```bash
KINESIS_STREAM_OUTPUT='ride_predictions'
SHARD='shardId-000000000000'
SHARD_ITERATOR=$(aws kinesis \
get-shard-iterator \
--shard-id ${SHARD} \
--shard-iterator-type TRIM_HORIZON \
--stream-name ${KINESIS_STREAM_OUTPUT} \
--query 'ShardIterator' \
)
RESULT=$(aws kinesis get-records --shard-iterator $SHARD_ITERATOR)
echo ${RESULT} | jq -r '.Records[0].Data' | base64 --decode
```
### Running the test
```bash
export PREDICTIONS_STREAM_NAME="ride_predictions"
export RUN_ID="e1efc53e9bd149078b0c12aeaa6365df"
export TEST_RUN="True"
python test.py
```
### Putting everything to Docker
```bash
docker build -t stream-model-duration:v1 .
docker run -it --rm \
-p 8080:8080 \
-e PREDICTIONS_STREAM_NAME="ride_predictions" \
-e RUN_ID="e1efc53e9bd149078b0c12aeaa6365df" \
-e TEST_RUN="True" \
-e AWS_DEFAULT_REGION="eu-west-1" \
stream-model-duration:v1
```
URL for testing:
* http://localhost:8080/2015-03-31/functions/function/invocations
### Configuring AWS CLI to run in Docker
To use AWS CLI, you may need to set the env variables:
```bash
docker run -it --rm \
-p 8080:8080 \
-e PREDICTIONS_STREAM_NAME="ride_predictions" \
-e RUN_ID="e1efc53e9bd149078b0c12aeaa6365df" \
-e TEST_RUN="True" \
-e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
-e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
-e AWS_DEFAULT_REGION="${AWS_DEFAULT_REGION}" \
stream-model-duration:v1
```
Alternatively, you can mount the `.aws` folder with your credentials to the `.aws` folder in the container:
```bash
docker run -it --rm \
-p 8080:8080 \
-e PREDICTIONS_STREAM_NAME="ride_predictions" \
-e RUN_ID="e1efc53e9bd149078b0c12aeaa6365df" \
-e TEST_RUN="True" \
-v c:/Users/alexe/.aws:/root/.aws \
stream-model-duration:v1
```
### Publishing Docker images
Creating an ECR repo
```bash
aws ecr create-repository --repository-name duration-model
```
Logging in
```bash
$(aws ecr get-login --no-include-email)
```
Pushing
```bash
REMOTE_URI="387546586013.dkr.ecr.eu-west-1.amazonaws.com/duration-model"
REMOTE_TAG="v1"
REMOTE_IMAGE=${REMOTE_URI}:${REMOTE_TAG}
LOCAL_IMAGE="stream-model-duration:v1"
docker tag ${LOCAL_IMAGE} ${REMOTE_IMAGE}
docker push ${REMOTE_IMAGE}
```
================================================
FILE: 04-deployment/streaming/lambda_function.py
================================================
import os
import json
import boto3
import base64
import mlflow
kinesis_client = boto3.client('kinesis')
PREDICTIONS_STREAM_NAME = os.getenv('PREDICTIONS_STREAM_NAME', 'ride_predictions')
RUN_ID = os.getenv('RUN_ID')
logged_model = f's3://mlflow-models-alexey/1/{RUN_ID}/artifacts/model'
# logged_model = f'runs:/{RUN_ID}/model'
model = mlflow.pyfunc.load_model(logged_model)
TEST_RUN = os.getenv('TEST_RUN', 'False') == 'True'
def prepare_features(ride):
features = {}
features['PU_DO'] = '%s_%s' % (ride['PULocationID'], ride['DOLocationID'])
features['trip_distance'] = ride['trip_distance']
return features
def predict(features):
pred = model.predict(features)
return float(pred[0])
def lambda_handler(event, context):
# print(json.dumps(event))
predictions_events = []
for record in event['Records']:
encoded_data = record['kinesis']['data']
decoded_data = base64.b64decode(encoded_data).decode('utf-8')
ride_event = json.loads(decoded_data)
# print(ride_event)
ride = ride_event['ride']
ride_id = ride_event['ride_id']
features = prepare_features(ride)
prediction = predict(features)
prediction_event = {
'model': 'ride_duration_prediction_model',
'version': '123',
'prediction': {
'ride_duration': prediction,
'ride_id': ride_id
}
}
if not TEST_RUN:
kinesis_client.put_record(
StreamName=PREDICTIONS_STREAM_NAME,
Data=json.dumps(prediction_event),
PartitionKey=str(ride_id)
)
predictions_events.append(prediction_event)
return {
'predictions': predictions_events
}
================================================
FILE: 04-deployment/streaming/test.py
================================================
import lambda_function
event = {
"Records": [
{
"kinesis": {
"kinesisSchemaVersion": "1.0",
"partitionKey": "1",
"sequenceNumber": "49630081666084879290581185630324770398608704880802529282",
"data": "ewogICAgICAgICJyaWRlIjogewogICAgICAgICAgICAiUFVMb2NhdGlvbklEIjogMTMwLAogICAgICAgICAgICAiRE9Mb2NhdGlvbklEIjogMjA1LAogICAgICAgICAgICAidHJpcF9kaXN0YW5jZSI6IDMuNjYKICAgICAgICB9LCAKICAgICAgICAicmlkZV9pZCI6IDI1NgogICAgfQ==",
"approximateArrivalTimestamp": 1654161514.132
},
"eventSource": "aws:kinesis",
"eventVersion": "1.0",
"eventID": "shardId-000000000000:49630081666084879290581185630324770398608704880802529282",
"eventName": "aws:kinesis:record",
"invokeIdentityArn": "arn:aws:iam::387546586013:role/lambda-kinesis-role",
"awsRegion": "eu-west-1",
"eventSourceARN": "arn:aws:kinesis:eu-west-1:387546586013:stream/ride_events"
}
]
}
result = lambda_function.lambda_handler(event, None)
print(result)
================================================
FILE: 04-deployment/streaming/test_docker.py
================================================
import requests
event = {
"Records": [
{
"kinesis": {
"kinesisSchemaVersion": "1.0",
"partitionKey": "1",
"sequenceNumber": "49630081666084879290581185630324770398608704880802529282",
"data": "ewogICAgICAgICJyaWRlIjogewogICAgICAgICAgICAiUFVMb2NhdGlvbklEIjogMTMwLAogICAgICAgICAgICAiRE9Mb2NhdGlvbklEIjogMjA1LAogICAgICAgICAgICAidHJpcF9kaXN0YW5jZSI6IDMuNjYKICAgICAgICB9LCAKICAgICAgICAicmlkZV9pZCI6IDI1NgogICAgfQ==",
"approximateArrivalTimestamp": 1654161514.132
},
"eventSource": "aws:kinesis",
"eventVersion": "1.0",
"eventID": "shardId-000000000000:49630081666084879290581185630324770398608704880802529282",
"eventName": "aws:kinesis:record",
"invokeIdentityArn": "arn:aws:iam::387546586013:role/lambda-kinesis-role",
"awsRegion": "eu-west-1",
"eventSourceARN": "arn:aws:kinesis:eu-west-1:387546586013:stream/ride_events"
}
]
}
url = 'http://localhost:8080/2015-03-31/functions/function/invocations'
response = requests.post(url, json=event)
print(response.json())
================================================
FILE: 04-deployment/web-service/Dockerfile
================================================
FROM python:3.9.7-slim
RUN pip install -U pip
RUN pip install pipenv
WORKDIR /app
COPY [ "Pipfile", "Pipfile.lock", "./" ]
RUN pipenv install --system --deploy
COPY [ "predict.py", "lin_reg.bin", "./" ]
EXPOSE 9696
ENTRYPOINT [ "gunicorn", "--bind=0.0.0.0:9696", "predict:app" ]
================================================
FILE: 04-deployment/web-service/Pipfile
================================================
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
scikit-learn = "==1.0.2"
flask = "*"
gunicorn = "*"
[dev-packages]
requests = "*"
[requires]
python_version = "3.9"
================================================
FILE: 04-deployment/web-service/README.md
================================================
## Deploying a model as a web-service
* Creating a virtual environment with Pipenv
* Creating a script for predictiong
* Putting the script into a Flask app
* Packaging the app to Docker
```bash
docker build -t ride-duration-prediction-service:v1 .
```
```bash
docker run -it --rm -p 9696:9696 ride-duration-prediction-service:v1
```
================================================
FILE: 04-deployment/web-service/predict.py
================================================
import pickle
from flask import Flask, request, jsonify
with open('lin_reg.bin', 'rb') as f_in:
(dv, model) = pickle.load(f_in)
def prepare_features(ride):
features = {}
features['PU_DO'] = '%s_%s' % (ride['PULocationID'], ride['DOLocationID'])
features['trip_distance'] = ride['trip_distance']
return features
def predict(features):
X = dv.transform(features)
preds = model.predict(X)
return float(preds[0])
app = Flask('duration-prediction')
@app.route('/predict', methods=['POST'])
def predict_endpoint():
ride = request.get_json()
features = prepare_features(ride)
pred = predict(features)
result = {
'duration': pred
}
return jsonify(result)
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=9696)
================================================
FILE: 04-deployment/web-service/test.py
================================================
import requests
ride = {
"PULocationID": 10,
"DOLocationID": 50,
"trip_distance": 40
}
url = 'http://localhost:9696/predict'
response = requests.post(url, json=ride)
print(response.json())
================================================
FILE: 04-deployment/web-service-mlflow/Pipfile
================================================
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
scikit-learn = "==1.0.2"
flask = "*"
gunicorn = "*"
mlflow = "*"
boto3 = "*"
[dev-packages]
requests = "*"
[requires]
python_version = "3.9"
================================================
FILE: 04-deployment/web-service-mlflow/README.md
================================================
## Getting the model for deployment from MLflow
* Take the code from the previous video
* Train another model, register with MLflow
* Put the model into a scikit-learn pipeline
* Model deployment with tracking server
* Model deployment without the tracking server
Starting the MLflow server with S3:
```bash
mlflow server \
--backend-store-uri=sqlite:///mlflow.db \
--default-artifact-root=s3://mlflow-models-alexey/
```
Downloading the artifact
```bash
export MLFLOW_TRACKING_URI="http://127.0.0.1:5000"
export MODEL_RUN_ID="6dd459b11b4e48dc862f4e1019d166f6"
mlflow artifacts download \
--run-id ${MODEL_RUN_ID} \
--artifact-path model \
--dst-path .
```
================================================
FILE: 04-deployment/web-service-mlflow/predict.py
================================================
import os
import pickle
import mlflow
from flask import Flask, request, jsonify
RUN_ID = os.getenv('RUN_ID')
logged_model = f's3://mlflow-models-alexey/1/{RUN_ID}/artifacts/model'
# logged_model = f'runs:/{RUN_ID}/model'
model = mlflow.pyfunc.load_model(logged_model)
def prepare_features(ride):
features = {}
features['PU_DO'] = '%s_%s' % (ride['PULocationID'], ride['DOLocationID'])
features['trip_distance'] = ride['trip_distance']
return features
def predict(features):
preds = model.predict(features)
return float(preds[0])
app = Flask('duration-prediction')
@app.route('/predict', methods=['POST'])
def predict_endpoint():
ride = request.get_json()
features = prepare_features(ride)
pred = predict(features)
result = {
'duration': pred,
'model_version': RUN_ID
}
return jsonify(result)
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=9696)
================================================
FILE: 04-deployment/web-service-mlflow/random-forest.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "2e25cf66",
"metadata": {},
"outputs": [],
"source": [
"import pickle\n",
"\n",
"import pandas as pd\n",
"\n",
"from sklearn.feature_extraction import DictVectorizer\n",
"from sklearn.ensemble import RandomForestRegressor\n",
"from sklearn.metrics import mean_squared_error"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "041e9423",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.pipeline import make_pipeline"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "506bec39",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022/06/01 12:27:06 INFO mlflow.tracking.fluent: Experiment with name 'green-taxi-duration' does not exist. Creating a new experiment.\n"
]
},
{
"data": {
"text/plain": [
"
### 6.2 Integration tests with docker-compose
### 6.3 Testing cloud services with LocalStack
### 6.4 Code quality: linting and formatting
### 6.5 Git pre-commit hooks
### 6.6 Makefiles and make
### 6.7 Homework
More information [here](../cohorts/2025/06-best-practices/homework.md).
### Objective
The goal of this project is to apply everything we have learned
in this course to build an end-to-end machine learning project.
## Problem statement
For the project, we will ask you to build an end-to-end ML project.
For that, you will need:
* Select a dataset that you're interested in (see [Datasets](#datasets))
* Train a model on that dataset tracking your experiments
* Create a model training pipeline
* Deploy the model in batch, web service or streaming
* Monitor the performance of your model
* Follow the best practices
## Technologies
You don't have to limit yourself to technologies covered in the course. You can use alternatives as well:
* **Cloud**: AWS, GCP, Azure, ...
* **Experiment tracking tools**: MLFlow, Weights & Biases, ...
* **Workflow orchestration**: Prefect, Airflow, Flyte, Kubeflow, Argo, ...
* **Monitoring**: Evidently, WhyLabs/whylogs, ...
* **CI/CD**: Github actions, Gitlab CI/CD, ...
* **Infrastructure as code (IaC)**: Terraform, Pulumi, Cloud Formation, ...
If you use a tool that wasn't covered in the course, be sure to explain what that tool does.
If you're not certain about some tools, ask in Slack.
## Peer reviewing
> [!IMPORTANT]
> To evaluate the projects, we'll use peer reviewing. This is a great opportunity for you to learn from each other.
> * To get points for your project, you need to evaluate 3 projects of your peers
> * You get 3 extra points for each evaluation
## Evaluation Criteria
* Problem description
* 0 points: The problem is not described
* 1 point: The problem is described but shortly or not clearly
* 2 points: The problem is well described and it's clear what the problem the project solves
* Cloud
* 0 points: Cloud is not used, things run only locally
* 2 points: The project is developed on the cloud OR uses localstack (or similar tool) OR the project is deployed to Kubernetes or similar container management platforms
* 4 points: The project is developed on the cloud and IaC tools are used for provisioning the infrastructure
* Experiment tracking and model registry
* 0 points: No experiment tracking or model registry
* 2 points: Experiments are tracked or models are registered in the registry
* 4 points: Both experiment tracking and model registry are used
* Workflow orchestration
* 0 points: No workflow orchestration
* 2 points: Basic workflow orchestration
* 4 points: Fully deployed workflow
* Model deployment
* 0 points: Model is not deployed
* 2 points: Model is deployed but only locally
* 4 points: The model deployment code is containerized and could be deployed to cloud or special tools for model deployment are used
* Model monitoring
* 0 points: No model monitoring
* 2 points: Basic model monitoring that calculates and reports metrics
* 4 points: Comprehensive model monitoring that sends alerts or runs a conditional workflow (e.g. retraining, generating debugging dashboard, switching to a different model) if the defined metrics threshold is violated
* Reproducibility
* 0 points: No instructions on how to run the code at all, the data is missing
* 2 points: Some instructions are there, but they are not complete OR instructions are clear and complete, the code works, but the data is missing
* 4 points: Instructions are clear, it's easy to run the code, and it works. The versions for all the dependencies are specified.
* Best practices
* [ ] There are unit tests (1 point)
* [ ] There is an integration test (1 point)
* [ ] Linter and/or code formatter are used (1 point)
* [ ] There's a Makefile (1 point)
* [ ] There are pre-commit hooks (1 point)
* [ ] There's a CI/CD pipeline (2 points)
> [!NOTE]
> It's highly recommended to create a new repository for your project (not inside an existing repo) with a meaningful title, such as
> "Car Price Prediction" or "Music Genre Classification" and include as many details as possible in the README file. ChatGPT can assist you with this. Doing so will not only make it easier to showcase your project for potential job opportunities but also have it featured on the [Projects Gallery App](#projects-gallery).
> If you leave the README file empty or with minimal details, there may be point deductions as per the [Evaluation Criteria](#evaluation-criteria).
## Resources
### Datasets
Refer to the provided [datasets](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/projects/datasets.md) for possible selection.
### Projects Gallery
Explore a collection of projects completed by members of our community. The projects cover a wide range of topics and utilize different tools and techniques. Feel free to delve into any project and see how others have tackled real-world problems with data, structured their code, and presented their findings. It's a great resource to learn and get ideas for your own projects.
[](https://datatalksclub-projects.streamlit.app/)
### MLOps Zoomcamp 2023
* [2023 Projects](../cohorts/2023/07-project)
### MLOps Zoomcamp 2022
* [2022 Projects](../cohorts/2022/07-project)
================================================
FILE: README.md
================================================
MLOps (machine learning operations) is a must-know skill for many data professionals. Master the fundamentals of MLOps, from training and experimentation to deployment and monitoring.
Join Slack • #course-mlops-zoomcamp Channel • Telegram Announcements • Course Playlist • FAQ • Tweet about the Course
## How to Take MLOps Zoomcamp ### 2025 Cohort - **Start Date**: May 5, 2025 - **Register Here**: [Sign up](https://airtable.com/shrCb8y6eTbPKwSTL) - **Stay Updated**: Subscribe to our [Google Calendar](https://calendar.google.com/calendar/?cid=M3Jzbmg0ZDA2aHVsY2M1ZjcyNDJtODNyMTRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ) (Desktop only) ### Self-Paced Learning All course materials are freely available for independent study. Follow these steps: 1. Watch the course videos. 2. Join the [Slack community](https://datatalks.club/slack.html). 3. Refer to the [FAQ document](https://datatalks.club/faq/mlops-zoomcamp.html) for guidance. ## Syllabus The course consists of structured modules, hands-on workshops, and a final project to reinforce your learning. Each module introduces core MLOps concepts and tools. ### Prerequisites To get the most out of this course, you should have prior experience with: - Python - Docker - Command line basics - Machine learning (e.g., through [ML Zoomcamp](https://github.com/alexeygrigorev/mlbookcamp-code/tree/master/course-zoomcamp)) - 1+ year of programming experience ## Modules ### [Module 1: Introduction](01-intro) - What is MLOps? - MLOps maturity model - NY Taxi dataset (our running example) - Why MLOps is essential - Course structure & environment setup - Homework ### [Module 2: Experiment Tracking & Model Management](02-experiment-tracking) - Introduction to experiment tracking - MLflow basics - Model saving and loading - Model registry - Hands-on MLflow exercises - Homework ### [Module 3: Orchestration & ML Pipelines](03-orchestration) - Workflow orchestration - Homework ### [Module 4: Model Deployment](04-deployment) - Deployment strategies: online (web, streaming) vs. offline (batch) - Deploying with Flask (web service) - Streaming deployment with AWS Kinesis & Lambda - Batch scoring for offline processing - Homework ### [Module 5: Model Monitoring](05-monitoring) - Monitoring ML-based services - Web service monitoring with Prometheus, Evidently, and Grafana - Batch job monitoring with Prefect, MongoDB, and Evidently - Homework ### [Module 6: Best Practices](06-best-practices) - Unit and integration testing - Linting, formatting, and pre-commit hooks - CI/CD with GitHub Actions - Infrastructure as Code (Terraform) - Homework ### [Final Project](07-project/) - End-to-end project integrating all course concepts ## Community & Support ### Getting Help on Slack Join the [`#course-mlops-zoomcamp`](https://app.slack.com/client/T01ATQK62F8/C02R98X7DS9) channel on [DataTalks.Club Slack](https://datatalks.club/slack.html) for discussions, troubleshooting, and networking. To keep discussions organized: - Follow [our guidelines](asking-questions.md) when posting questions. - Review the [community guidelines](https://datatalks.club/slack/guidelines.html). ## Instructors - [Cristian Martinez](https://www.linkedin.com/in/cristian-javier-martinez-09bb7031/) - [Alexey Grigorev](https://www.linkedin.com/in/agrigorev/) - [Emeli Dral](https://www.linkedin.com/in/emelidral/) ## Sponsors & Supporters Interested in supporting our community? Reach out to [alexey@datatalks.club](mailto:alexey@datatalks.club). ## About DataTalks.Club
DataTalks.Club is a global online community of data enthusiasts. It's a place to discuss data, learn, share knowledge, ask and answer questions, and support each other.
Website • Join Slack Community • Newsletter • Upcoming Events • Google Calendar • YouTube • GitHub • LinkedIn • Twitter
All the activity at DataTalks.Club mainly happens on [Slack](https://datatalks.club/slack.html). We post updates there and discuss different aspects of data, career questions, and more. At DataTalksClub, we organize online events, community activities, and free courses. You can learn more about what we do at [DataTalksClub Community Navigation](https://www.notion.so/DataTalksClub-Community-Navigation-bf070ad27ba44bf6bbc9222082f0e5a8?pvs=21). ================================================ FILE: after-sign-up.md ================================================ ## Thank you! Thanks for signining up for the course. The process of adding you to the mailing list is not automated yet, but you will hear from us closer to the course start. To make sure you don't miss any announcements - Register in [DataTalks.Club's Slack](https://datatalks.club/slack.html) and join the [`#course-mlops-zoomcamp`](https://app.slack.com/client/T01ATQK62F8/C02R98X7DS9) channel - Join the [course Telegram channel with announcements](https://t.me/dtc_courses) - [Tweet about the course!](https://ctt.ac/fH67W) - Subscribe to [DataTalks.Club's YouTube channel](https://www.youtube.com/c/DataTalksClub) and check [the course playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK) - Subscribe to our [public Google Calendar](https://calendar.google.com/calendar/?cid=M3Jzbmg0ZDA2aHVsY2M1ZjcyNDJtODNyMTRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ) (it works from Desktop only) - Check our [Technical FAQ](https://docs.google.com/document/d/12TlBfhIiKtyBv8RnsoJR6F72bkPDGEvPOItJIxaEzE0/edit) if you have questions See you in May! ================================================ FILE: asking-questions.md ================================================ ## Asking questions If you have any questions, ask them in the [`#course-mlops-zoomcamp`](https://app.slack.com/client/T01ATQK62F8/C02R98X7DS9) channel in [DataTalks.Club](https://datatalks.club) slack. To keep our discussion in Slack more organized, we ask you to follow these suggestions: * Before asking a question, check [FAQ](https://docs.google.com/document/d/12TlBfhIiKtyBv8RnsoJR6F72bkPDGEvPOItJIxaEzE0/edit). * Use threads. When you have a problem, first describe the problem shortly and then put the actual error in the thread - so it doesn't take the entire screen. * Instead of screenshots, it's better to copy-paste the error you're getting in text. Use ` ``` ` for formatting your code. It's very difficult to read text from screenshots. * Please don't take pictures of your code with a phone. It's even harder to read. Follow the previous suggestion, and in rare cases when you need to show what happens on your screen, take a screenshot. * You don't need to tag the instructors when you have a problem. We will see it eventually. * If somebody helped you with your problem and it's not in [FAQ](https://docs.google.com/document/d/12TlBfhIiKtyBv8RnsoJR6F72bkPDGEvPOItJIxaEzE0/edit), please add it there. It'll help other students. ================================================ FILE: certificate.md ================================================ ## Getting your certificate Congratulations on finishing the course! Here's how you can get your certificate. First, get your certificate id using the `compute_certificate_id` function: ```python from hashlib import sha1 def compute_hash(email): return sha1(email.encode('utf-8')).hexdigest() def compute_certificate_id(email): email_clean = email.lower().strip() return compute_hash(email_clean + '_') ``` Then use this hash to get the URL ```python cohort = 2024 course = 'mlops-zoomcamp' your_id = compute_certificate_id('never.give.up@gmail.com') url = f"https://certificate.datatalks.club/{course}/{cohort}/{your_id}.pdf" print(url) ``` Example: https://certificate.datatalks.club/mlops-zoomcamp/2022/fe629854d45c559e9c10b3b8458ea392fdeb68a9.pdf ## Adding to LinkedIn You can add your certificate to LinkedIn: * Log in to your LinkedIn account, then go to your profile. * On the right, in the "Add profile" section dropdown, choose "Background" and then select the drop-down triangle next to "Licenses & Certifications". * In "Name", enter "MLOps Zoomcamp". * In "Issuing Organization", enter "DataTalksClub". * (Optional) In "Issue Date", enter the time when the certificate was created. * (Optional) Select the checkbox This certification does not expire. * Put your certificate ID. * In "Certification URL", enter the URL for your certificate. [Adapted from here](https://support.edx.org/hc/en-us/articles/206501938-How-can-I-add-my-certificate-to-my-LinkedIn-profile-) ================================================ FILE: cohorts/2022/01-intro/homework.ipynb ================================================ { "cells": [ { "cell_type": "code", "execution_count": 42, "id": "7525e867", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import seaborn as sns\n", "\n", "from sklearn.feature_extraction import DictVectorizer\n", "from sklearn.linear_model import LinearRegression\n", "\n", "from sklearn.metrics import mean_squared_error" ] }, { "cell_type": "code", "execution_count": 35, "id": "4c1fbc9a", "metadata": {}, "outputs": [], "source": [ "df = pd.read_parquet('./data/fhv_tripdata_2021-01.parquet')" ] }, { "cell_type": "code", "execution_count": 20, "id": "7a70df86", "metadata": {}, "outputs": [], "source": [ "old_len = len(df)" ] }, { "cell_type": "code", "execution_count": 36, "id": "f17cc325", "metadata": {}, "outputs": [], "source": [ "df['duration'] = df.dropOff_datetime - df.pickup_datetime\n", "df['duration'] = df.duration.dt.total_seconds() / 60" ] }, { "cell_type": "code", "execution_count": 16, "id": "1b613836", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "19.1672240937939" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.duration.mean()" ] }, { "cell_type": "code", "execution_count": 37, "id": "3092ae43", "metadata": {}, "outputs": [], "source": [ "df = df[(df.duration >= 1) & (df.duration <= 60)].copy()" ] }, { "cell_type": "code", "execution_count": 38, "id": "2e1aa200", "metadata": {}, "outputs": [], "source": [ "categorical = ['PUlocationID', 'DOlocationID']\n", "\n", "df[categorical] = df[categorical].fillna(-1).astype('int')" ] }, { "cell_type": "code", "execution_count": 52, "id": "22c6495b", "metadata": {}, "outputs": [], "source": [ "df[categorical] = df[categorical].astype('str')" ] }, { "cell_type": "code", "execution_count": 55, "id": "12fcfa04", "metadata": {}, "outputs": [], "source": [ "train_dicts = df[categorical].to_dict(orient='records')" ] }, { "cell_type": "code", "execution_count": 56, "id": "0606301f", "metadata": {}, "outputs": [], "source": [ "dv = DictVectorizer()\n", "X_train = dv.fit_transform(train_dicts) " ] }, { "cell_type": "code", "execution_count": 57, "id": "c81c6737", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(1109826, 525)" ] }, "execution_count": 57, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train.shape" ] }, { "cell_type": "code", "execution_count": 60, "id": "aeaf7bad", "metadata": {}, "outputs": [], "source": [ "y_train = df.duration.values" ] }, { "cell_type": "code", "execution_count": 59, "id": "75199886", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "525" ] }, "execution_count": 59, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(dv.feature_names_)" ] }, { "cell_type": "code", "execution_count": 61, "id": "4d7f1e4b", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "LinearRegression()" ] }, "execution_count": 61, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lr = LinearRegression()\n", "lr.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 62, "id": "d61318d0", "metadata": {}, "outputs": [], "source": [ "y_pred = lr.predict(X_train)" ] }, { "cell_type": "code", "execution_count": 63, "id": "1aa5f5c2", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "10.528519107212292" ] }, "execution_count": 63, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mean_squared_error(y_train, y_pred, squared=False)" ] }, { "cell_type": "code", "execution_count": 64, "id": "41c08294", "metadata": {}, "outputs": [], "source": [ "categorical = ['PUlocationID', 'DOlocationID']\n", "\n", "def read_data(filename):\n", " df = pd.read_parquet(filename)\n", " \n", " df['duration'] = df.dropOff_datetime - df.pickup_datetime\n", " df['duration'] = df.duration.dt.total_seconds() / 60\n", "\n", " df = df[(df.duration >= 1) & (df.duration <= 60)].copy()\n", "\n", " df[categorical] = df[categorical].fillna(-1).astype('int').astype('str')\n", " \n", " return df" ] }, { "cell_type": "code", "execution_count": 65, "id": "4854399a", "metadata": {}, "outputs": [], "source": [ "df_val = read_data('./data/fhv_tripdata_2021-02.parquet')" ] }, { "cell_type": "code", "execution_count": 68, "id": "f9eea69f", "metadata": {}, "outputs": [], "source": [ "val_dicts = df_val[categorical].to_dict(orient='records')" ] }, { "cell_type": "code", "execution_count": 69, "id": "669fda0a", "metadata": {}, "outputs": [], "source": [ "X_val = dv.transform(val_dicts) " ] }, { "cell_type": "code", "execution_count": 70, "id": "7f8f950d", "metadata": {}, "outputs": [], "source": [ "y_pred = lr.predict(X_val)" ] }, { "cell_type": "code", "execution_count": 71, "id": "8f129850", "metadata": {}, "outputs": [], "source": [ "y_val = df_val.duration.values" ] }, { "cell_type": "code", "execution_count": 72, "id": "8a277ab9", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "11.014283211122269" ] }, "execution_count": 72, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mean_squared_error(y_val, y_pred, squared=False)" ] }, { "cell_type": "code", "execution_count": null, "id": "b5108416", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" } }, "nbformat": 4, "nbformat_minor": 5 } ================================================ FILE: cohorts/2022/01-intro/homework.md ================================================ ## 1.6 Homework The goal of this homework is to train a simple model for predicting the duration of a ride - similar to what we did in this module. ## Q1. Downloading the data We'll use [the same NYC taxi dataset](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page), but instead of "Green Taxi Trip Records", we'll use "For-Hire Vehicle Trip Records". Download the data for January and February 2021. Note that you need "For-Hire Vehicle Trip Records", not "High Volume For-Hire Vehicle Trip Records". Read the data for January. How many records are there? * 1054112 * 1154112 * 1254112 * 1354112 ## Q2. Computing duration Now let's compute the `duration` variable. It should contain the duration of a ride in minutes. What's the average trip duration in January? * 15.16 * 19.16 * 24.16 * 29.16 ## Data preparation Check the distribution of the duration variable. There are some outliers. Let's remove them and keep only the records where the duration was between 1 and 60 minutes (inclusive). How many records did you drop? ## Q3. Missing values The features we'll use for our model are the pickup and dropoff location IDs. But they have a lot of missing values there. Let's replace them with "-1". What's the fractions of missing values for the pickup location ID? I.e. fraction of "-1"s after you filled the NAs. * 53% * 63% * 73% * 83% ## Q4. One-hot encoding Let's apply one-hot encoding to the pickup and dropoff location IDs. We'll use only these two features for our model. * Turn the dataframe into a list of dictionaries * Fit a dictionary vectorizer * Get a feature matrix from it What's the dimensionality of this matrix? (The number of columns). * 2 * 152 * 352 * 525 * 725 ## Q5. Training a model Now let's use the feature matrix from the previous step to train a model. * Train a plain linear regression model with default parameters * Calculate the RMSE of the model on the training data What's the RMSE on train? * 5.52 * 10.52 * 15.52 * 20.52 ## Q6. Evaluating the model Now let's apply this model to the validation dataset (Feb 2021). What's the RMSE on validation? * 6.01 * 11.01 * 16.01 * 21.01 ## Submit the results Submit your results here: https://forms.gle/V8q5rv7QRoZ13Sft6 It's possible that your answers won't match exactly. If it's the case, select the closest one. ## Deadline The deadline for submitting is 24 May 2022 (Tuesday) 23:00 CET. After that, the form will be closed. ## Solution * [Video](https://www.youtube.com/watch?v=feH1PMLyu-Q&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK&index=9) * [Notebook](homework.ipynb) ================================================ FILE: cohorts/2022/02-experiment-tracking/homework/hpo.py ================================================ import argparse import os import pickle import mlflow import numpy as np from hyperopt import STATUS_OK, Trials, fmin, hp, tpe from hyperopt.pyll import scope from sklearn.ensemble import RandomForestRegressor from sklearn.metrics import mean_squared_error mlflow.set_tracking_uri("http://127.0.0.1:5000") mlflow.set_experiment("random-forest-hyperopt") def load_pickle(filename): with open(filename, "rb") as f_in: return pickle.load(f_in) def run(data_path, num_trials): X_train, y_train = load_pickle(os.path.join(data_path, "train.pkl")) X_valid, y_valid = load_pickle(os.path.join(data_path, "valid.pkl")) def objective(params): rf = RandomForestRegressor(**params) rf.fit(X_train, y_train) y_pred = rf.predict(X_valid) rmse = mean_squared_error(y_valid, y_pred, squared=False) return {'loss': rmse, 'status': STATUS_OK} search_space = { 'max_depth': scope.int(hp.quniform('max_depth', 1, 20, 1)), 'n_estimators': scope.int(hp.quniform('n_estimators', 10, 50, 1)), 'min_samples_split': scope.int(hp.quniform('min_samples_split', 2, 10, 1)), 'min_samples_leaf': scope.int(hp.quniform('min_samples_leaf', 1, 4, 1)), 'random_state': 42 } rstate = np.random.default_rng(42) # for reproducible results fmin( fn=objective, space=search_space, algo=tpe.suggest, max_evals=num_trials, trials=Trials(), rstate=rstate ) if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument( "--data_path", default="./output", help="the location where the processed NYC taxi trip data was saved." ) parser.add_argument( "--max_evals", type=int, default=50, help="the number of parameter evaluations for the optimizer to explore." ) args = parser.parse_args() run(args.data_path, args.max_evals) ================================================ FILE: cohorts/2022/02-experiment-tracking/homework/preprocess_data.py ================================================ import argparse import os import pickle import pandas as pd from sklearn.feature_extraction import DictVectorizer def dump_pickle(obj, filename): with open(filename, "wb") as f_out: return pickle.dump(obj, f_out) def read_dataframe(filename: str): df = pd.read_parquet(filename) df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime df.duration = df.duration.apply(lambda td: td.total_seconds() / 60) df = df[(df.duration >= 1) & (df.duration <= 60)] categorical = ['PULocationID', 'DOLocationID'] df[categorical] = df[categorical].astype(str) return df def preprocess(df: pd.DataFrame, dv: DictVectorizer, fit_dv: bool = False): df['PU_DO'] = df['PULocationID'] + '_' + df['DOLocationID'] categorical = ['PU_DO'] numerical = ['trip_distance'] dicts = df[categorical + numerical].to_dict(orient='records') if fit_dv: X = dv.fit_transform(dicts) else: X = dv.transform(dicts) return X, dv def run(raw_data_path: str, dest_path: str, dataset: str = "green"): # load parquet files df_train = read_dataframe( os.path.join(raw_data_path, f"{dataset}_tripdata_2021-01.parquet") ) df_valid = read_dataframe( os.path.join(raw_data_path, f"{dataset}_tripdata_2021-02.parquet") ) df_test = read_dataframe( os.path.join(raw_data_path, f"{dataset}_tripdata_2021-03.parquet") ) # extract the target target = 'duration' y_train = df_train[target].values y_valid = df_valid[target].values y_test = df_test[target].values # fit the dictvectorizer and preprocess data dv = DictVectorizer() X_train, dv = preprocess(df_train, dv, fit_dv=True) X_valid, _ = preprocess(df_valid, dv, fit_dv=False) X_test, _ = preprocess(df_test, dv, fit_dv=False) # create dest_path folder unless it already exists os.makedirs(dest_path, exist_ok=True) # save dictvectorizer and datasets dump_pickle(dv, os.path.join(dest_path, "dv.pkl")) dump_pickle((X_train, y_train), os.path.join(dest_path, "train.pkl")) dump_pickle((X_valid, y_valid), os.path.join(dest_path, "valid.pkl")) dump_pickle((X_test, y_test), os.path.join(dest_path, "test.pkl")) if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument( "--raw_data_path", help="the location where the raw NYC taxi trip data was saved" ) parser.add_argument( "--dest_path", help="the location where the resulting files will be saved." ) args = parser.parse_args() run(args.raw_data_path, args.dest_path) ================================================ FILE: cohorts/2022/02-experiment-tracking/homework/register_model.py ================================================ import argparse import os import pickle import mlflow from hyperopt import hp, space_eval from hyperopt.pyll import scope from mlflow.entities import ViewType from mlflow.tracking import MlflowClient from sklearn.ensemble import RandomForestRegressor from sklearn.metrics import mean_squared_error HPO_EXPERIMENT_NAME = "random-forest-hyperopt" EXPERIMENT_NAME = "random-forest-best-models" mlflow.set_tracking_uri("http://127.0.0.1:5000") mlflow.set_experiment(EXPERIMENT_NAME) mlflow.sklearn.autolog() SPACE = { 'max_depth': scope.int(hp.quniform('max_depth', 1, 20, 1)), 'n_estimators': scope.int(hp.quniform('n_estimators', 10, 50, 1)), 'min_samples_split': scope.int(hp.quniform('min_samples_split', 2, 10, 1)), 'min_samples_leaf': scope.int(hp.quniform('min_samples_leaf', 1, 4, 1)), 'random_state': 42 } def load_pickle(filename): with open(filename, "rb") as f_in: return pickle.load(f_in) def train_and_log_model(data_path, params): X_train, y_train = load_pickle(os.path.join(data_path, "train.pkl")) X_valid, y_valid = load_pickle(os.path.join(data_path, "valid.pkl")) X_test, y_test = load_pickle(os.path.join(data_path, "test.pkl")) with mlflow.start_run(): params = space_eval(SPACE, params) rf = RandomForestRegressor(**params) rf.fit(X_train, y_train) # evaluate model on the validation and test sets valid_rmse = mean_squared_error(y_valid, rf.predict(X_valid), squared=False) mlflow.log_metric("valid_rmse", valid_rmse) test_rmse = mean_squared_error(y_test, rf.predict(X_test), squared=False) mlflow.log_metric("test_rmse", test_rmse) def run(data_path, log_top): client = MlflowClient() # retrieve the top_n model runs and log the models to MLflow experiment = client.get_experiment_by_name(HPO_EXPERIMENT_NAME) runs = client.search_runs( experiment_ids=experiment.experiment_id, run_view_type=ViewType.ACTIVE_ONLY, max_results=log_top, order_by=["metrics.rmse ASC"] ) for run in runs: train_and_log_model(data_path=data_path, params=run.data.params) # select the model with the lowest test RMSE experiment = client.get_experiment_by_name(EXPERIMENT_NAME) # best_run = client.search_runs( ... )[0] # register the best model # mlflow.register_model( ... ) if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument( "--data_path", default="./output", help="the location where the processed NYC taxi trip data was saved." ) parser.add_argument( "--top_n", default=5, type=int, help="the top 'top_n' models will be evaluated to decide which model to promote." ) args = parser.parse_args() run(args.data_path, args.top_n) ================================================ FILE: cohorts/2022/02-experiment-tracking/homework/train.py ================================================ import argparse import os import pickle from sklearn.ensemble import RandomForestRegressor from sklearn.metrics import mean_squared_error def load_pickle(filename: str): with open(filename, "rb") as f_in: return pickle.load(f_in) def run(data_path): X_train, y_train = load_pickle(os.path.join(data_path, "train.pkl")) X_valid, y_valid = load_pickle(os.path.join(data_path, "valid.pkl")) rf = RandomForestRegressor(max_depth=10, random_state=0) rf.fit(X_train, y_train) y_pred = rf.predict(X_valid) rmse = mean_squared_error(y_valid, y_pred, squared=False) if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument( "--data_path", default="./output", help="the location where the processed NYC taxi trip data was saved." ) args = parser.parse_args() run(args.data_path) ================================================ FILE: cohorts/2022/02-experiment-tracking/homework.md ================================================ ## Homework The goal of this homework is to get familiar with MLflow. ## Q1. Install MLflow To get started with MLflow you'll need to install the MLflow Python package. For this we recommend creating a separate Python environment, for example, you can use [conda environments](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs), and then install the package there with `pip` or `conda`. Once you installed the package, run the command `mlflow --version` and check the output. What's the version that you have? ## Q2. Download and preprocess the data We'll use the Green Taxi Trip Records dataset. Download the data for January, February and March 2021 in parquet format from [here](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page). Use the script `preprocess_data.py` located in the folder [`homework`](homework) to preprocess the data. The script will - load the data from the folder `
## 3.2 Introduction to Prefect 2.0
## 3.3 First Prefect flow and basics
## 3.4 Remote Prefect Orion deployment
## 3.5 Deployment of Prefect flow
**Note:** There are several changes to deployment in Prefect 2.3.1 since 2.0b8:
- `DeploymentSpec` in 2.0b8 now becomes `Deployment`.
- `work_queue_name` is used instead of `tags` to submit the deployment to the a specific work queue.
- You don't need to create a work queue before using the work queue. A work queue will be created if it doesn't exist.
```python
from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import IntervalSchedule
from datetime import timedelta
deployment = Deployment.build_from_flow(
flow=main,
name="model_training",
schedule=IntervalSchedule(interval=timedelta(minutes=5)),
work_queue_name="ml"
)
deployment.apply()
```
Links:
* [Instructions for Hosting Prefect Orion](https://discourse.prefect.io/t/hosting-an-orion-instance-on-a-cloud-vm/967)
## 3.6 MLOps Zoomcamp 3.6 - (Optional) Work queues and agents
## 3.7 Homework
More information here: TBD
## Notes
Did you take notes? Add them here:
* [Week 3, Prefect Introduction and S3 Bucket configuration with Prefect by M. Ayoub C.](https://gist.github.com/Qfl3x/8dd69b8173f027b9468016c118f3b6a5)
* [Notes from froukje](https://github.com/froukje/ml-ops-zoomcamp/blob/master/03-orchestration/week03_orchestration.ipynb)
* [Minimalist code notes from Anna V](https://github.com/annnvv/mlops_zoomcamp/blob/main/notes/module3_notes_prefect.md)
* [Getting Started on Prefect 2.0 + Deploying worfklows for MLflow Staging by Ron Medina (Jupyter Book)](https://particle1331.github.io/inefficient-networks/notebooks/mlops/3-prefect/3-prefect.html)
* [Quickstart your homework by Zioalex](https://github.com/zioalex/mlops-zoomcamp/blob/week3/03-orchestration/homework_quickstart.md)
* [Notes from Maxime M](https://github.com/maxmarkov/mlops-zoomcamp/blob/master/lecture-notes/WEEK-3/03-orchestration.md)
* [Week3: Prefect introduction and homework notes by Bhagabat](https://github.com/BPrasad123/MLOps_Zoomcamp/tree/main/Week3)
* [Week 3: Orchestration notes by Ayoub.B](https://github.com/ayoub-berdeddouch/mlops-journey/blob/main/orchestration-03.md)
* Send a PR, add your notes above this line
================================================
FILE: cohorts/2022/03-orchestration/code/model_training.py
================================================
from audioop import add
import pandas as pd
import pickle
from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LinearRegression, Lasso, Ridge
from sklearn.metrics import mean_squared_error
import xgboost as xgb
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from hyperopt.pyll import scope
import mlflow
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("nyc-taxi-experiment")
def read_dataframe(filename):
df = pd.read_parquet(filename)
df.lpep_dropoff_datetime = pd.to_datetime(df.lpep_dropoff_datetime)
df.lpep_pickup_datetime = pd.to_datetime(df.lpep_pickup_datetime)
df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime
df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)
df = df[(df.duration >= 1) & (df.duration <= 60)]
categorical = ['PULocationID', 'DOLocationID']
df[categorical] = df[categorical].astype(str)
return df
def add_features(train_path="./data/green_tripdata_2021-01.parquet",
val_path="./data/green_tripdata_2021-02.parquet"):
df_train = read_dataframe(train_path)
df_val = read_dataframe(val_path)
print(len(df_train))
print(len(df_val))
df_train['PU_DO'] = df_train['PULocationID'] + '_' + df_train['DOLocationID']
df_val['PU_DO'] = df_val['PULocationID'] + '_' + df_val['DOLocationID']
categorical = ['PU_DO'] #'PULocationID', 'DOLocationID']
numerical = ['trip_distance']
dv = DictVectorizer()
train_dicts = df_train[categorical + numerical].to_dict(orient='records')
X_train = dv.fit_transform(train_dicts)
val_dicts = df_val[categorical + numerical].to_dict(orient='records')
X_val = dv.transform(val_dicts)
target = 'duration'
y_train = df_train[target].values
y_val = df_val[target].values
return X_train, X_val, y_train, y_val, dv
# # Modelling
# lr = LinearRegression()
# lr.fit(X_train, y_train)
# y_pred = lr.predict(X_val)
# mean_squared_error(y_val, y_pred, squared=False)
# with open('models/lin_reg.bin', 'wb') as f_out:
# pickle.dump((dv, lr), f_out)
# with mlflow.start_run():
# mlflow.set_tag("developer", "cristian")
# mlflow.log_param("train-data-path", "./data/green_tripdata_2021-01.csv")
# mlflow.log_param("valid-data-path", "./data/green_tripdata_2021-02.csv")
# alpha = 0.1
# mlflow.log_param("alpha", alpha)
# lr = Lasso(alpha)
# lr.fit(X_train, y_train)
# y_pred = lr.predict(X_val)
# rmse = mean_squared_error(y_val, y_pred, squared=False)
# mlflow.log_metric("rmse", rmse)
# mlflow.log_artifact(local_path="models/lin_reg.bin", artifact_path="models_pickle")
def train_model_search(train, valid, y_val):
def objective(params):
with mlflow.start_run():
mlflow.set_tag("model", "xgboost")
mlflow.log_params(params)
booster = xgb.train(
params=params,
dtrain=train,
num_boost_round=1000,
evals=[(valid, 'validation')],
early_stopping_rounds=50
)
y_pred = booster.predict(valid)
rmse = mean_squared_error(y_val, y_pred, squared=False)
mlflow.log_metric("rmse", rmse)
return {'loss': rmse, 'status': STATUS_OK}
search_space = {
'max_depth': scope.int(hp.quniform('max_depth', 4, 100, 1)),
'learning_rate': hp.loguniform('learning_rate', -3, 0),
'reg_alpha': hp.loguniform('reg_alpha', -5, -1),
'reg_lambda': hp.loguniform('reg_lambda', -6, -1),
'min_child_weight': hp.loguniform('min_child_weight', -1, 3),
'objective': 'reg:linear',
'seed': 42
}
best_result = fmin(
fn=objective,
space=search_space,
algo=tpe.suggest,
max_evals=1,
trials=Trials()
)
return
def train_best_model(train, valid, y_val, dv):
with mlflow.start_run():
train = xgb.DMatrix(X_train, label=y_train)
valid = xgb.DMatrix(X_val, label=y_val)
best_params = {
'learning_rate': 0.09585355369315604,
'max_depth': 30,
'min_child_weight': 1.060597050922164,
'objective': 'reg:linear',
'reg_alpha': 0.018060244040060163,
'reg_lambda': 0.011658731377413597,
'seed': 42
}
mlflow.log_params(best_params)
booster = xgb.train(
params=best_params,
dtrain=train,
num_boost_round=1000,
evals=[(valid, 'validation')],
early_stopping_rounds=50
)
y_pred = booster.predict(valid)
rmse = mean_squared_error(y_val, y_pred, squared=False)
mlflow.log_metric("rmse", rmse)
with open("models/preprocessor.b", "wb") as f_out:
pickle.dump(dv, f_out)
mlflow.log_artifact("models/preprocessor.b", artifact_path="preprocessor")
mlflow.xgboost.log_model(booster, artifact_path="models_mlflow")
if __name__ == "__main__":
X_train, X_val, y_train, y_val, dv = add_features()
train = xgb.DMatrix(X_train, label=y_train)
valid = xgb.DMatrix(X_val, label=y_val)
train_model_search(train, valid, y_val)
train_best_model(train, valid, y_val, dv)
================================================
FILE: cohorts/2022/03-orchestration/code/orchestration.py
================================================
import pandas as pd
import pickle
from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LinearRegression, Lasso, Ridge
from sklearn.metrics import mean_squared_error
import mlflow
import xgboost as xgb
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from hyperopt.pyll import scope
from prefect import flow, task
@task
def read_dataframe(filename):
df = pd.read_parquet(filename)
df.lpep_dropoff_datetime = pd.to_datetime(df.lpep_dropoff_datetime)
df.lpep_pickup_datetime = pd.to_datetime(df.lpep_pickup_datetime)
df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime
df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)
df = df[(df.duration >= 1) & (df.duration <= 60)]
categorical = ['PULocationID', 'DOLocationID']
df[categorical] = df[categorical].astype(str)
return df
@task
def add_features(df_train, df_val):
df_train['PU_DO'] = df_train['PULocationID'] + '_' + df_train['DOLocationID']
df_val['PU_DO'] = df_val['PULocationID'] + '_' + df_val['DOLocationID']
categorical = ['PU_DO'] #'PULocationID', 'DOLocationID']
numerical = ['trip_distance']
dv = DictVectorizer()
train_dicts = df_train[categorical + numerical].to_dict(orient='records')
X_train = dv.fit_transform(train_dicts)
val_dicts = df_val[categorical + numerical].to_dict(orient='records')
X_val = dv.transform(val_dicts)
target = 'duration'
y_train = df_train[target].values
y_val = df_val[target].values
return X_train, X_val, y_train, y_val, dv
@task
def train_model_search(train, valid, y_val):
def _objective(params):
with mlflow.start_run():
mlflow.set_tag("model", "xgboost")
mlflow.log_params(params)
booster = xgb.train(
params=params,
dtrain=train,
num_boost_round=1000,
evals=[(valid, 'validation')],
early_stopping_rounds=50
)
y_pred = booster.predict(valid)
rmse = mean_squared_error(y_val, y_pred, squared=False)
mlflow.log_metric("rmse", rmse)
return {'loss': rmse, 'status': STATUS_OK}
search_space = {
'max_depth': scope.int(hp.quniform('max_depth', 4, 100, 1)),
'learning_rate': hp.loguniform('learning_rate', -3, 0),
'reg_alpha': hp.loguniform('reg_alpha', -5, -1),
'reg_lambda': hp.loguniform('reg_lambda', -6, -1),
'min_child_weight': hp.loguniform('min_child_weight', -1, 3),
'objective': 'reg:linear',
'seed': 42
}
best_result = fmin(
fn=_objective,
space=search_space,
algo=tpe.suggest,
max_evals=1,
trials=Trials()
)
return best_result
@task
def train_best_model(X_train, X_val, y_train, y_val, dv):
with mlflow.start_run():
train = xgb.DMatrix(X_train, label=y_train)
valid = xgb.DMatrix(X_val, label=y_val)
best_params = {
'learning_rate': 0.09585355369315604,
'max_depth': 30,
'min_child_weight': 1.060597050922164,
'objective': 'reg:linear',
'reg_alpha': 0.018060244040060163,
'reg_lambda': 0.011658731377413597,
'seed': 42
}
mlflow.log_params(best_params)
booster = xgb.train(
params=best_params,
dtrain=train,
num_boost_round=1000,
evals=[(valid, 'validation')],
early_stopping_rounds=50
)
y_pred = booster.predict(valid)
rmse = mean_squared_error(y_val, y_pred, squared=False)
mlflow.log_metric("rmse", rmse)
with open("models/preprocessor.b", "wb") as f_out:
pickle.dump(dv, f_out)
mlflow.log_artifact("models/preprocessor.b", artifact_path="preprocessor")
mlflow.xgboost.log_model(booster, artifact_path="models_mlflow")
@flow
def main_flow(train_path: str = './data/green_tripdata_2021-01.parquet',
val_path: str = './data/green_tripdata_2021-02.parquet'):
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("nyc-taxi-experiment")
# Load
df_train = read_dataframe(train_path)
df_val = read_dataframe(val_path)
# Transform
X_train, X_val, y_train, y_val, dv = add_features(df_train, df_val).result()
# Training
train = xgb.DMatrix(X_train, label=y_train)
valid = xgb.DMatrix(X_val, label=y_val)
best = train_model_search(train, valid, y_val)
train_best_model(X_train, X_val, y_train, y_val, dv, wait_for=best)
# main_flow()
from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import IntervalSchedule
from datetime import timedelta
Deployment.build_from_flow(
flow=main_flow,
name="model_training",
# schedule=IntervalSchedule(interval=timedelta(weeks=1)),
work_queue_name="ml",
)
================================================
FILE: cohorts/2022/03-orchestration/code/prefect_deploy.py
================================================
import pandas as pd
import pickle
from sklearn.feature_extraction import DictVectorizer
from sklearn.metrics import mean_squared_error
import xgboost as xgb
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from hyperopt.pyll import scope
import mlflow
from prefect import flow, task
@task
def read_dataframe(filename):
df = pd.read_parquet(filename)
df.lpep_dropoff_datetime = pd.to_datetime(df.lpep_dropoff_datetime)
df.lpep_pickup_datetime = pd.to_datetime(df.lpep_pickup_datetime)
df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime
df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)
df = df[(df.duration >= 1) & (df.duration <= 60)]
categorical = ['PULocationID', 'DOLocationID']
df[categorical] = df[categorical].astype(str)
return df
@task
def add_features(df_train, df_val):
# df_train = read_dataframe(train_path)
# df_val = read_dataframe(val_path)
print(len(df_train))
print(len(df_val))
df_train['PU_DO'] = df_train['PULocationID'] + '_' + df_train['DOLocationID']
df_val['PU_DO'] = df_val['PULocationID'] + '_' + df_val['DOLocationID']
categorical = ['PU_DO'] #'PULocationID', 'DOLocationID']
numerical = ['trip_distance']
dv = DictVectorizer()
train_dicts = df_train[categorical + numerical].to_dict(orient='records')
X_train = dv.fit_transform(train_dicts)
val_dicts = df_val[categorical + numerical].to_dict(orient='records')
X_val = dv.transform(val_dicts)
target = 'duration'
y_train = df_train[target].values
y_val = df_val[target].values
return X_train, X_val, y_train, y_val, dv
@task
def train_model_search(train, valid, y_val):
def objective(params):
with mlflow.start_run():
mlflow.set_tag("model", "xgboost")
mlflow.log_params(params)
booster = xgb.train(
params=params,
dtrain=train,
num_boost_round=100,
evals=[(valid, 'validation')],
early_stopping_rounds=50
)
y_pred = booster.predict(valid)
rmse = mean_squared_error(y_val, y_pred, squared=False)
mlflow.log_metric("rmse", rmse)
return {'loss': rmse, 'status': STATUS_OK}
search_space = {
'max_depth': scope.int(hp.quniform('max_depth', 4, 100, 1)),
'learning_rate': hp.loguniform('learning_rate', -3, 0),
'reg_alpha': hp.loguniform('reg_alpha', -5, -1),
'reg_lambda': hp.loguniform('reg_lambda', -6, -1),
'min_child_weight': hp.loguniform('min_child_weight', -1, 3),
'objective': 'reg:linear',
'seed': 42
}
best_result = fmin(
fn=objective,
space=search_space,
algo=tpe.suggest,
max_evals=1,
trials=Trials()
)
return
@task
def train_best_model(train, valid, y_val, dv):
with mlflow.start_run():
best_params = {
'learning_rate': 0.09585355369315604,
'max_depth': 30,
'min_child_weight': 1.060597050922164,
'objective': 'reg:linear',
'reg_alpha': 0.018060244040060163,
'reg_lambda': 0.011658731377413597,
'seed': 42
}
mlflow.log_params(best_params)
booster = xgb.train(
params=best_params,
dtrain=train,
num_boost_round=100,
evals=[(valid, 'validation')],
early_stopping_rounds=50
)
y_pred = booster.predict(valid)
rmse = mean_squared_error(y_val, y_pred, squared=False)
mlflow.log_metric("rmse", rmse)
with open("models/preprocessor.b", "wb") as f_out:
pickle.dump(dv, f_out)
mlflow.log_artifact("models/preprocessor.b", artifact_path="preprocessor")
mlflow.xgboost.log_model(booster, artifact_path="models_mlflow")
@flow
def main(train_path: str="./data/green_tripdata_2021-01.parquet",
val_path: str="./data/green_tripdata_2021-02.parquet"):
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("nyc-taxi-experiment")
X_train = read_dataframe(train_path)
X_val = read_dataframe(val_path)
X_train, X_val, y_train, y_val, dv = add_features(X_train, X_val).result()
train = xgb.DMatrix(X_train, label=y_train)
valid = xgb.DMatrix(X_val, label=y_val)
train_model_search(train, valid, y_val)
train_best_model(train, valid, y_val, dv)
from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import IntervalSchedule
from datetime import timedelta
deployment = Deployment.build_from_flow(
flow=main,
name="model_training",
schedule=IntervalSchedule(interval=timedelta(minutes=5)),
work_queue_name="ml"
)
deployment.apply()
================================================
FILE: cohorts/2022/03-orchestration/code/prefect_flow.py
================================================
import pandas as pd
import pickle
from sklearn.feature_extraction import DictVectorizer
from sklearn.metrics import mean_squared_error
import xgboost as xgb
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from hyperopt.pyll import scope
import mlflow
from prefect import flow, task
@task
def read_dataframe(filename):
df = pd.read_parquet(filename)
df.lpep_dropoff_datetime = pd.to_datetime(df.lpep_dropoff_datetime)
df.lpep_pickup_datetime = pd.to_datetime(df.lpep_pickup_datetime)
df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime
df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)
df = df[(df.duration >= 1) & (df.duration <= 60)]
categorical = ['PULocationID', 'DOLocationID']
df[categorical] = df[categorical].astype(str)
return df
@task
def add_features(df_train, df_val):
# df_train = read_dataframe(train_path)
# df_val = read_dataframe(val_path)
print(len(df_train))
print(len(df_val))
df_train['PU_DO'] = df_train['PULocationID'] + '_' + df_train['DOLocationID']
df_val['PU_DO'] = df_val['PULocationID'] + '_' + df_val['DOLocationID']
categorical = ['PU_DO'] #'PULocationID', 'DOLocationID']
numerical = ['trip_distance']
dv = DictVectorizer()
train_dicts = df_train[categorical + numerical].to_dict(orient='records')
X_train = dv.fit_transform(train_dicts)
val_dicts = df_val[categorical + numerical].to_dict(orient='records')
X_val = dv.transform(val_dicts)
target = 'duration'
y_train = df_train[target].values
y_val = df_val[target].values
return X_train, X_val, y_train, y_val, dv
@task
def train_model_search(train, valid, y_val):
def objective(params):
with mlflow.start_run():
mlflow.set_tag("model", "xgboost")
mlflow.log_params(params)
booster = xgb.train(
params=params,
dtrain=train,
num_boost_round=100,
evals=[(valid, 'validation')],
early_stopping_rounds=50
)
y_pred = booster.predict(valid)
rmse = mean_squared_error(y_val, y_pred, squared=False)
mlflow.log_metric("rmse", rmse)
return {'loss': rmse, 'status': STATUS_OK}
search_space = {
'max_depth': scope.int(hp.quniform('max_depth', 4, 100, 1)),
'learning_rate': hp.loguniform('learning_rate', -3, 0),
'reg_alpha': hp.loguniform('reg_alpha', -5, -1),
'reg_lambda': hp.loguniform('reg_lambda', -6, -1),
'min_child_weight': hp.loguniform('min_child_weight', -1, 3),
'objective': 'reg:linear',
'seed': 42
}
best_result = fmin(
fn=objective,
space=search_space,
algo=tpe.suggest,
max_evals=1,
trials=Trials()
)
return
@task
def train_best_model(train, valid, y_val, dv):
with mlflow.start_run():
best_params = {
'learning_rate': 0.09585355369315604,
'max_depth': 30,
'min_child_weight': 1.060597050922164,
'objective': 'reg:linear',
'reg_alpha': 0.018060244040060163,
'reg_lambda': 0.011658731377413597,
'seed': 42
}
mlflow.log_params(best_params)
booster = xgb.train(
params=best_params,
dtrain=train,
num_boost_round=100,
evals=[(valid, 'validation')],
early_stopping_rounds=50
)
y_pred = booster.predict(valid)
rmse = mean_squared_error(y_val, y_pred, squared=False)
mlflow.log_metric("rmse", rmse)
with open("models/preprocessor.b", "wb") as f_out:
pickle.dump(dv, f_out)
mlflow.log_artifact("models/preprocessor.b", artifact_path="preprocessor")
mlflow.xgboost.log_model(booster, artifact_path="models_mlflow")
@flow
def main(train_path: str="./data/green_tripdata_2021-01.parquet",
val_path: str="./data/green_tripdata_2021-02.parquet"):
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("nyc-taxi-experiment")
X_train = read_dataframe(train_path)
X_val = read_dataframe(val_path)
X_train, X_val, y_train, y_val, dv = add_features(X_train, X_val).result()
train = xgb.DMatrix(X_train, label=y_train)
valid = xgb.DMatrix(X_val, label=y_val)
train_model_search(train, valid, y_val)
train_best_model(train, valid, y_val, dv)
================================================
FILE: cohorts/2022/03-orchestration/code/work-queue.py
================================================
from prefect import flow
@flow
def myflow():
print("hello")
from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import IntervalSchedule
from datetime import timedelta
deployment_dev = Deployment.build_from_flow(
flow=myflow,
name="model_training-dev",
schedule=IntervalSchedule(interval=timedelta(minutes=5)),
work_queue_name="dev"
)
deployment_dev.apply()
deployment_prod = Deployment.build_from_flow(
flow=myflow,
name="model_training-prod",
schedule=IntervalSchedule(interval=timedelta(minutes=5)),
work_queue_name="prod"
)
deployment_prod.apply()
================================================
FILE: cohorts/2022/03-orchestration/homework.md
================================================
## 3.6 Homework
The goal of this homework is to familiarize users with workflow orchestration. We start from the solution of homework 1. The notebook can be found below:
https://github.com/DataTalksClub/mlops-zoomcamp/blob/main/01-intro/homework.ipynb
This has already been converted to a script called `homework.py` in the `03-orchestration` folder of this repo.
You will use the FHV dataset like in homework 1.
## Motivation
We already have a model training script. Maybe a data scientist in your team handed it to you and your job is schedule the running of training script using a workflow orchestration - Prefect in this case. Below are the requirements. Do not implement them yet, we will do so in this exercise. Just understand the goal.
1. The training flow will be run every month.
2. The flow will take in a parameter called `date` which will be a datetime.
* a. `date` should default to `None`.
* b. If `date` is `None`, set `date` as the current day. Use the data from 2 months back as the training data and the data from the previous month as validation data.
* c. If `date` is passed, get 2 months before the `date` as the training data, and the previous month as validation data.
* d. As a concrete example, if the date passed is "2021-03-15", the training data should be "fhv_tripdata_2021-01.parquet" and the validation file will be "fhv_trip_data_2021-02.parquet".
3. Save the model as "model-{date}.bin" where date is in `YYYY-MM-DD`. Note that `date` here is the value of the flow `parameter`. In practice, this setup makes it very easy to get the latest model to run predictions because you just need to get the most recent one.
4. In this example we use a DictVectorizer. That is needed to run future data through our model. Save that as "dv-{date}.b". Similar to above, if the date is `2021-03-15`, the files output should be `model-2021-03-15.bin` and `dv-2021-03-15.b`.
This convention is not strict in industry, and in practice, you will come up with your own system to manage these training pipeline runs. For example, if we wanted to train on the whole history instead of just one month, we'd need to allow for added parameterization and logic in our flow. If the data came in weekly instead of monthly, we might need a different naming convention. But these requirements are already a simple approximation of something you could use in production.
On the deployment side, it's very easy to just pull in the latest data and predict it using the latest model and vectorizer files. Tools the MLFlow in the last chapter can simplify that process as well. This homework will focus more on the batch training.
In order, this homework assignment will be about:
1. Converting the script to a Flow.
2. Changing the parameters to take in a `date`. Making this parameter dynamic.
3. Scheduling a batch training job that outputs the latest model somewhere.
## Setup
You can use either local Prefect Orion or a VM hosted Prefect Orion instance for this. It shouldn't matter. Just note that if you use a VM hosted one, you will need to configure your local API to hit the VM.
Video 3.4 of the course will give more detailed instructions if you been run it on a VM.
## Q1. Converting the script to a Prefect flow
If you want to follow the videos exactly, do:
```bash
pip install prefect==2.0b5
```
If you need Windows support, check [`windows.md`](windows.md) for installation instructions.
The current script `homework.py` is a fully functional script as long as you already have `fhv_trip_data_2021-01.parquet` and `fhv_trip_data_2021-02.parquet` inside a `data` folder. You should be able to already run it using:
```bash
python homework.py
```
We want to bring this to workflow orchestration to add observability around it. The `main` function will be converted to a `flow` and the other functions will be `tasks`. After adding all of the decorators, there is actually one task that you will need to call `.result()` for inside the `flow` to get it to work. Which task is this?
* `read_data`
* `prepare_features`
* `train_model`
* `run_model`
Important: change all `print` statements to use the Prefect logger. Using the `print` statement will not appear in the Prefect UI. You have to call `get_run_logger` at the start of the task to use it.
## Q2. Parameterizing the flow
Right now there are two parameters for `main()` called `train_path` and `val_path`. We want to change the flow function to accept `date` instead. `date` should then be passed to a task that gives both the `train_path` and `val_path` to use.
It should look like this:
```python
@flow
def main(date=None):
train_path, val_path = get_paths(date).result()
# rest of flow below
```
Where `get_paths` is a task that you have to implement. The specs for this are outlined in the motivation section. Listing them out again here:
* a. The flow will take in a parameter called `date` which will be a datetime. `date` should default to `None`.
* b. If `date` is `None`, use the current day. Use the data from 2 months back as the training data and the data from the previous month as validation data.
* c. If a `date` value is supplied, get 2 months before the `date` as the training data, and the previous month as validation data.
* d. As a concrete example, if the date passed is "2021-03-15", the training data should be "fhv_tripdata_2021-01.parquet" and the validation file will be "fhv_trip_data_2021-02.parquet".
Because we have two files:
* fhv_tripdata_2021-01.parquet
* fhv_tripdata_2021-02.parquet
Change the `main()` flow call to the following:
```python
main(date="2021-03-15")
```
and it should use those files. This is a simplification for testing our homework.
Recall the page from where we downloaded the For-Hire trip data.
https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page
Download the relevant files needed to run the `main` flow if `date` is 2021-08-15.
For example:
```python
main(date="2021-08-15")
```
By setting up the logger from the previous step, we should see some logs about our training job. What is the validation MSE when running the flow with this date?
Note you need to download the relevant files to run. Part of this question is understanding which files the flow should be looking for.
The validation MSE is:
* 11.637
* 11.837
* 12.037
* 12.237
## Q3. Saving the model and artifacts
At the moment, we are not saving the model and vectorizer for future use. You don't need a new task for this, you can just add it inside the `flow`. The requirements for filenames to save it as were mentioned in the Motivation section. They are pasted again here:
* Save the model as "model-{date}.bin" where date is in `YYYY-MM-DD`. Note that `date` here is the value of the flow `parameter`. In practice, this setup makes it very easy to get the latest model to run predictions because you just need to get the most recent one.
* In this example we use a DictVectorizer. That is needed to run future data through our model. Save that as "dv-{date}.b". Similar to above, if the date is `2021-03-15`, the files output should be `model-2021-03-15.bin` and `dv-2021-03-15.b`.
By using this file name, during inference, we can just pull the latest model from our model directory and apply it. Assuming we already had a list of filenames:
```python
['model-2021-03-15.bin', 'model-2021-04-15.bin', 'model-2021-05-15.bin']
```
We could do something like `sorted(model_list, reverse=False)[0]` to get the filename of the latest file. This is the simplest way to consistently use the latest trained model for inference. Tools like MLFlow give us more control logic to use flows.
What is the file size of the `DictVectorizer` that we trained when the `date` is 2021-08-15?
* 13,000 bytes
* 23,000 bytes
* 33,000 bytes
* 43,000 bytes
You can bring up the Orion UI and see the work you've been doing. If you are using local Orion, you can start the server with:
```bash
prefect orion start
```
You should be able to see previous Flow runs and the most recent successful runs. Navigate to some of them. Take time to explore the UI. The radar plot would be a good thing to share on social media if you participate in those posts.
## Q4. Creating a deployment with a CronSchedule
We previously showed the `IntervalSchedule` in the video tutorials. In some cases, the interval is too rigid. For example, what if we wanted to run this `flow` on the 15th of every month? An interval of 30 days would not be in sync. In cases like these, the `CronSchedule` is more appropriate. The documentation for that is [here](https://orion-docs.prefect.io/concepts/schedules/#cronschedule)
Cron is an important part of workflow orchestration. It is used to schedule tasks, and was a predecessor for more mature orchestration frameworks. A lot of teams still use Cron in production. Even if you don't use Cron, the Cron expression is very common as a way to write a schedule, and the basics are worth learning for orchestration, even outside Prefect.
For this exercise, use a `CronSchedule` when creating a Prefect deployment.
What is the Cron expression to run a flow at 9 AM every 15th of the month?
* `* * 15 9 0`
* `9 15 * * *`
* `0 9 15 * *`
* `0 15 9 1 *`
Hint: there are many Cron to English tools. Try looking for one to help you.
Create a deployment with `prefect deployment create` after you write your `DeploymentSpec`
## Q5. Viewing the Deployment
View the deployment in the UI. When first loading, we may not see that many flows because the default filter is 1 day back and 1 day forward. Remove the filter for 1 day forward to see the scheduled runs.
How many flow runs are scheduled by Prefect in advance? You should not be counting manually. There is a number of upcoming runs on the top right of the dashboard.
* 0
* 3
* 10
* 25
**Note:** Your answer may be off by 1. It's okay. Select the option closest to your answer.
## Q6. Creating a work-queue
In order to run this flow, you will need an agent and a work queue. Because we scheduled our flow on every month, it won't really get picked up by an agent. For this exercise, create a work-queue from the UI and view it using the CLI.
For all CLI commands with Prefect, you can use `--help` to get more information.
For example,
* `prefect --help`
* `prefect work-queue --help`
What is the command to view the available work-queues?
* `prefect work-queue inspect`
* `prefect work-queue ls`
* `prefect work-queue preview`
* `prefect work-queue list`
## Submit the results
* Submit your results here: https://forms.gle/tYEZdrYdXHBizmtJ7
* It's possible that your answers won't match exactly. If it's the case, select the closest one.
* You can submit your answers multiple times. In this case, the last submission will be used for scoring.
## Deadline
The deadline for submitting is 13 June 2022 (Monday) 23:00 CEST. After that, the form will be closed.
## Solution
* [Code - `homework_solution.py`](homework_solution.py)
* [Video](https://www.youtube.com/watch?v=DfmRb7J-Wuw)
================================================
FILE: cohorts/2022/03-orchestration/homework.py
================================================
import pandas as pd
from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
def read_data(path):
df = pd.read_parquet(path)
return df
def prepare_features(df, categorical, train=True):
df['duration'] = df.dropOff_datetime - df.pickup_datetime
df['duration'] = df.duration.dt.total_seconds() / 60
df = df[(df.duration >= 1) & (df.duration <= 60)].copy()
mean_duration = df.duration.mean()
if train:
print(f"The mean duration of training is {mean_duration}")
else:
print(f"The mean duration of validation is {mean_duration}")
df[categorical] = df[categorical].fillna(-1).astype('int').astype('str')
return df
def train_model(df, categorical):
train_dicts = df[categorical].to_dict(orient='records')
dv = DictVectorizer()
X_train = dv.fit_transform(train_dicts)
y_train = df.duration.values
print(f"The shape of X_train is {X_train.shape}")
print(f"The DictVectorizer has {len(dv.feature_names_)} features")
lr = LinearRegression()
lr.fit(X_train, y_train)
y_pred = lr.predict(X_train)
mse = mean_squared_error(y_train, y_pred, squared=False)
print(f"The MSE of training is: {mse}")
return lr, dv
def run_model(df, categorical, dv, lr):
val_dicts = df[categorical].to_dict(orient='records')
X_val = dv.transform(val_dicts)
y_pred = lr.predict(X_val)
y_val = df.duration.values
mse = mean_squared_error(y_val, y_pred, squared=False)
print(f"The MSE of validation is: {mse}")
return
def main(train_path: str = './data/fhv_tripdata_2021-01.parquet',
val_path: str = './data/fhv_tripdata_2021-02.parquet'):
categorical = ['PUlocationID', 'DOlocationID']
df_train = read_data(train_path)
df_train_processed = prepare_features(df_train, categorical)
df_val = read_data(val_path)
df_val_processed = prepare_features(df_val, categorical, False)
# train the model
lr, dv = train_model(df_train_processed, categorical)
run_model(df_val_processed, categorical, dv, lr)
main()
================================================
FILE: cohorts/2022/03-orchestration/homework_solution.py
================================================
import pandas as pd
from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from prefect import task, flow, get_run_logger
from datetime import datetime
import pickle
@task
def read_data(path):
df = pd.read_parquet(path)
return df
@task
def prepare_features(df, categorical, train=True):
logger = get_run_logger()
df['duration'] = df.dropOff_datetime - df.pickup_datetime
df['duration'] = df.duration.dt.total_seconds() / 60
df = df[(df.duration >= 1) & (df.duration <= 60)].copy()
mean_duration = df.duration.mean()
if train:
logger.info(f"The mean duration of training is {mean_duration}")
else:
logger.info(f"The mean duration of validation is {mean_duration}")
df[categorical] = df[categorical].fillna(-1).astype('int').astype('str')
return df
@task
def train_model(df, categorical):
logger = get_run_logger()
train_dicts = df[categorical].to_dict(orient='records')
dv = DictVectorizer()
X_train = dv.fit_transform(train_dicts)
y_train = df.duration.values
logger.info(f"The shape of X_train is {X_train.shape}")
logger.info(f"The DictVectorizer has {len(dv.feature_names_)} features")
lr = LinearRegression()
lr.fit(X_train, y_train)
y_pred = lr.predict(X_train)
mse = mean_squared_error(y_train, y_pred, squared=False)
logger.info(f"The MSE of training is: {mse}")
return lr, dv
@task
def run_model(df, categorical, dv, lr):
logger = get_run_logger()
val_dicts = df[categorical].to_dict(orient='records')
X_val = dv.transform(val_dicts)
y_pred = lr.predict(X_val)
y_val = df.duration.values
mse = mean_squared_error(y_val, y_pred, squared=False)
logger.info(f"The MSE of validation is: {mse}")
return
@task
def get_paths(date):
from dateutil.relativedelta import relativedelta
if date:
processed_date = datetime.strptime(date, "%Y-%m-%d")
else:
processed_date = datetime.today()
train_date = processed_date - relativedelta(months=2)
val_date = processed_date - relativedelta(months=1)
train_path = f"./data/fhv_tripdata_{train_date.year}-{str(train_date.month).zfill(2)}.parquet"
val_path = f"./data/fhv_tripdata_{val_date.year}-{str(val_date.month).zfill(2)}.parquet"
return train_path, val_path
@flow
def main(date=None):
train_path, val_path = get_paths(date).result()
categorical = ['PUlocationID', 'DOlocationID']
df_train = read_data(train_path)
df_train_processed = prepare_features(df_train, categorical)
df_val = read_data(val_path)
df_val_processed = prepare_features(df_val, categorical, False)
# train the model
lr, dv = train_model(df_train_processed, categorical).result()
run_model(df_val_processed, categorical, dv, lr)
if date is None:
date = datetime.today.strftime("%Y-%m-%d")
with open(f'./models/dv-{date}.b', 'wb') as f_out:
pickle.dump(dv, f_out)
# main("2021-08-15")
from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import CronSchedule
from prefect.flow_runners import SubprocessFlowRunner
Deployment(
flow=main,
name="model_training",
schedule=CronSchedule(cron="0 9 15 * *"),
flow_runner=SubprocessFlowRunner(),
)
================================================
FILE: cohorts/2022/04-deployment/homework/Dockerfile
================================================
FROM python:3.9.7-slim
WORKDIR /app
COPY [ "model2.bin", "model.bin" ]
================================================
FILE: cohorts/2022/04-deployment/homework/Pipfile
================================================
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
scikit-learn = "==1.0.2"
pandas = "*"
pyarrow = "*"
s3fs = "*"
[dev-packages]
[requires]
python_version = "3.9"
================================================
FILE: cohorts/2022/04-deployment/homework/batch.py
================================================
#!/usr/bin/env python
# coding: utf-8
import sys
import pickle
import pandas as pd
year = int(sys.argv[1]) # 2021
month = int(sys.argv[2]) #2
input_file = f's3://nyc-tlc/trip data/fhv_tripdata_{year:04d}-{month:02d}.parquet'
output_file = f's3://nyc-duration-prediction-alexey/taxi_type=fhv/year={year:04d}/month={month:02d}/predictions.parquet'
with open('model.bin', 'rb') as f_in:
dv, lr = pickle.load(f_in)
categorical = ['PUlocationID', 'DOlocationID']
def read_data(filename):
df = pd.read_parquet(filename)
df['duration'] = df.dropOff_datetime - df.pickup_datetime
df['duration'] = df.duration.dt.total_seconds() / 60
df = df[(df.duration >= 1) & (df.duration <= 60)].copy()
df[categorical] = df[categorical].fillna(-1).astype('int').astype('str')
return df
df = read_data(input_file)
df['ride_id'] = f'{year:04d}/{month:02d}_' + df.index.astype('str')
dicts = df[categorical].to_dict(orient='records')
X_val = dv.transform(dicts)
y_pred = lr.predict(X_val)
print('predicted mean duration:', y_pred.mean())
df_result = pd.DataFrame()
df_result['ride_id'] = df['ride_id']
df_result['predicted_duration'] = y_pred
df_result.to_parquet(
output_file,
engine='pyarrow',
compression=None,
index=False
)
================================================
FILE: cohorts/2022/04-deployment/homework/homework.dockerfile
================================================
FROM agrigorev/zoomcamp-model:mlops-3.9.7-slim
RUN pip install -U pip
RUN pip install pipenv
WORKDIR /app
COPY [ "Pipfile", "Pipfile.lock", "./" ]
RUN pipenv install --system --deploy
COPY [ "batch.py", "batch.py" ]
ENTRYPOINT [ "python", "batch.py" ]
================================================
FILE: cohorts/2022/04-deployment/homework/starter.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "2c51efaa",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"scikit-learn==1.0.2\n",
"scikit-learn-intelex==2021.20210714.120553\n"
]
}
],
"source": [
"!pip freeze | grep scikit-learn"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "0ef880a0",
"metadata": {},
"outputs": [],
"source": [
"import pickle\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "920cff32",
"metadata": {},
"outputs": [],
"source": [
"year = 2021\n",
"month = 2\n",
"\n",
"input_file = f'https://nyc-tlc.s3.amazonaws.com/trip+data/fhv_tripdata_{year:04d}-{month:02d}.parquet'\n",
"output_file = f'output/fhv_tripdata_{year:04d}-{month:02d}.parquet'"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "7836ccfd",
"metadata": {},
"outputs": [],
"source": [
"with open('model.bin', 'rb') as f_in:\n",
" dv, lr = pickle.load(f_in)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "41c08294",
"metadata": {},
"outputs": [],
"source": [
"categorical = ['PUlocationID', 'DOlocationID']\n",
"\n",
"def read_data(filename):\n",
" df = pd.read_parquet(filename)\n",
" \n",
" df['duration'] = df.dropOff_datetime - df.pickup_datetime\n",
" df['duration'] = df.duration.dt.total_seconds() / 60\n",
"\n",
" df = df[(df.duration >= 1) & (df.duration <= 60)].copy()\n",
"\n",
" df[categorical] = df[categorical].fillna(-1).astype('int').astype('str')\n",
" \n",
" return df"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "4854399a",
"metadata": {},
"outputs": [],
"source": [
"df = read_data(input_file)\n",
"df['ride_id'] = f'{year:04d}/{month:02d}_' + df.index.astype('str')"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "669fda0a",
"metadata": {},
"outputs": [],
"source": [
"dicts = df[categorical].to_dict(orient='records')\n",
"X_val = dv.transform(dicts)\n",
"y_pred = lr.predict(X_val)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "914b15a5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"16.191691679979066"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_pred.mean()"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "037e3d22",
"metadata": {},
"outputs": [],
"source": [
"df_result = pd.DataFrame()\n",
"df_result['ride_id'] = df['ride_id']\n",
"df_result['predicted_duration'] = y_pred"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "7a5753be",
"metadata": {},
"outputs": [],
"source": [
"df_result.to_parquet(\n",
" output_file,\n",
" engine='pyarrow',\n",
" compression=None,\n",
" index=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "f0b3b58c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 19M\r\n",
"-rw-rw-r-- 1 ubuntu ubuntu 19M Jun 30 08:43 fhv_tripdata_2021-02.parquet\r\n"
]
}
],
"source": [
"!ls -lh output/"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0dbe3e15",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: cohorts/2022/04-deployment/homework.md
================================================
## 4.8 Homework
In this homework, we'll deploy the ride duration model in batch mode. Like in homework 1 and 3, we'll use the FHV data.
You'll find the starter code in the [homework](https://github.com/DataTalksClub/mlops-zoomcamp/blob/74324e4d3759e9712ce406b8b30c77cff66e6cef/04-deployment/homework/) directory.
## Q1. Notebook
We'll start with the same notebook we ended up with in homework 1.
We cleaned it a little bit and kept only the scoring part. Now it's in [homework/starter.ipynb](https://github.com/DataTalksClub/mlops-zoomcamp/blob/74324e4d3759e9712ce406b8b30c77cff66e6cef/04-deployment/homework/starter.ipynb).
Run this notebook for the February 2021 FVH data.
What's the mean predicted duration for this dataset?
* 11.19
* 16.19
* 21.19
* 26.19
## Q2. Preparing the output
Like in the course videos, we want to prepare the dataframe with the output.
First, let's create an artificial `ride_id` column:
```python
df['ride_id'] = f'{year:04d}/{month:02d}_' + df.index.astype('str')
```
Next, write the ride id and the predictions to a dataframe with results.
Save it as parquet:
```python
df_result.to_parquet(
output_file,
engine='pyarrow',
compression=None,
index=False
)
```
What's the size of the output file?
* 9M
* 19M
* 29M
* 39M
Make sure you use the snippet above for saving the file. It should contain only these two columns. For this question, don't change the
dtypes of the columns and use pyarrow, not fastparquet.
## Q3. Creating the scoring script
Now let's turn the notebook into a script.
Which command you need to execute for that?
## Q4. Virtual environment
Now let's put everything into a virtual environment. We'll use pipenv for that.
Install all the required libraries. Pay attention to the Scikit-Learn version:
check the starter notebook for details.
After installing the libraries, pipenv creates two files: `Pipfile`
and `Pipfile.lock`. The `Pipfile.lock` file keeps the hashes of the
dependencies we use for the virtual env.
What's the first hash for the Scikit-Learn dependency?
## Q5. Parametrize the script
Let's now make the script configurable via CLI. We'll create two
parameters: year and month.
Run the script for March 2021.
What's the mean predicted duration?
* 11.29
* 16.29
* 21.29
* 26.29
Hint: just add a print statement to your script.
## Q6. Docker contaner
Finally, we'll package the script in the docker container.
For that, you'll need to use a base image that we prepared.
This is how it looks like:
```
FROM python:3.9.7-slim
WORKDIR /app
COPY [ "model2.bin", "model.bin" ]
```
(see [`homework/Dockerfile`](homework/Dockerfile))
We pushed it to [`agrigorev/zoomcamp-model:mlops-3.9.7-slim`](https://hub.docker.com/layers/zoomcamp-model/agrigorev/zoomcamp-model/mlops-3.9.7-slim/images/sha256-7fac33c783cc6018356ce16a4b408f6c977b55a4df52bdb6c4d0215edf83af5d?context=explore),
which you should use as your base image.
That is, this is how your Dockerfile should start:
```docker
FROM agrigorev/zoomcamp-model:mlops-3.9.7-slim
# do stuff here
```
This image already has a pickle file with a dictionary vectorizer
and a model. You will need to use them.
Important: don't copy the model to the docker image. You will need
to use the pickle file already in the image.
Now run the script with docker. What's the mean predicted duration
for April 2021?
* 9.96
* 16.55
* 25.96
* 36.55
## Bonus: upload the result to the cloud (Not graded)
Just printing the mean duration inside the docker image
doesn't seem very practical. Typically, after creating the output
file, we upload it to the cloud storage.
Modify your code to upload the parquet file to S3/GCS/etc.
## Submit the results
* Submit your results here: https://forms.gle/pFAYjTFqFMJELG819
* It's possible that your answers won't match exactly. If it's the case, select the closest one.
* You can submit your answers multiple times. In this case, the last submission will be used for scoring.
## Deadline
The deadline for submitting is 27 June 2022 (Monday) 23:00 CEST. After that, the form will be closed.
## Solution
After the deadline, we'll post the solution here
## Publishing the image to dockerhub
This is how we published the image to Docker hub:
```bash
docker build -t mlops-zoomcamp-model:v1 .
docker tag mlops-zoomcamp-model:v1 agrigorev/zoomcamp-model:mlops-3.9.7-slim
docker push agrigorev/zoomcamp-model:mlops-3.9.7-slim
```
================================================
FILE: cohorts/2022/05-monitoring/README.md
================================================
# 5. Model Monitoring
## 5.1 Monitoring for ML-based services
[Slides](https://drive.google.com/file/d/1wcMU75ZcNNJie4ELjsKPkITIL93wHykt/view?usp=sharing)
## 5.2 Setting up the environment
## 5.3 Creating a prediction service and simulating traffic
## 5.4 Realtime monitoring walktrough (Prometheus, Evidently, Grafana)
## 5.5 Batch monitoring walktrough (Prefect, MongoDB, Evidently)
## 5.6 Homework
More information here: TBD
## Notes
Did you take notes? Add them here:
* [Week 5 notes by M. Ayoub C.](https://gist.github.com/Qfl3x/aa6b1bec35fb645ded0371c46e8aafd1)
* [week 5: Monitoring notes Ayoub.B](https://github.com/ayoub-berdeddouch/mlops-journey/blob/main/monitoring-05.md)
* Send a PR, add your notes above this line
# Monitoring example
## Prerequisites
You need following tools installed:
- `docker`
- `docker-compose` (included to Docker Desktop for Mac and Docker Desktop for Windows )
## Preparation
Note: all actions expected to be executed in repo folder.
- Create virtual environment and activate it (eg. `python -m venv venv && source ./venv/bin/activate`)
- Install required packages `pip install -r requirements.txt`
- Run `python prepare.py` for downloading datasets
## Monitoring Example
### Starting services
To start all required services, execute:
```bash
docker-compose up
```
It will start following services:
- `prometheus` - TSDB for metrics
- `grafana` - Visual tool for metrics
- `mongo` - MongoDB, for storing raw data, predictions, targets and profile reports
- `evidently_service` - Evindently RT-monitoring service (draft example)
- `prediction_service` - main service, which makes predictions
### Sending data
To start sending data to service, execute:
```bash
python send_data.py
```
This script will send every second single row from dataset to prediction service along with creating file `target.csv` with actual results (so it can be loaded after)
## Batch Monitoring Example
After you stop sending data to service, you can run batch monitoring pipeline (using Prefect) by running script:
```bash
python prefect_example.py
```
This script will:
- load `target.csv` to MongoDB
- download dataset from MongoDB
- Run Evidently Model Profile and Evidently Report on this data
- Save Profile data back to MongoDB
- Save Report to `evidently_report_example.html`
You can look at Prefect steps in Prefect Orion UI
(to start it execute `prefect orion start`)
================================================
FILE: cohorts/2022/05-monitoring/homework/docker-compose-homework-solution.yml
================================================
version: "3.7"
volumes:
mongo_data: {}
networks:
front-tier:
back-tier:
services:
prediction_service:
build:
context: prediction_service
dockerfile: Dockerfile
depends_on:
- mongo
environment:
MONGO_DATABASE: "prediction_service"
MONGO_ADDRESS: "mongodb://mongo.:27017/"
MODEL_VERSION: "2"
MODEL_FILE: "lin_reg_V2.bin"
ports:
- 9696:9696
networks:
- back-tier
- front-tier
mongo:
image: mongo
ports:
- 27017:27017
volumes:
- mongo_data:/data/db
networks:
- back-tier
- front-tier
================================================
FILE: cohorts/2022/05-monitoring/homework/docker-compose-homework.yml
================================================
version: "3.7"
volumes:
mongo_data: {}
networks:
front-tier:
back-tier:
services:
prediction_service:
build:
context: prediction_service
dockerfile: Dockerfile
depends_on:
- mongo
environment:
MONGO_DATABASE: "prediction_service"
MONGO_ADDRESS: "mongodb://mongo.:27017/"
MODEL_VERSION: "1"
MODEL_FILE: "lin_reg.bin"
ports:
- 9696:9696
networks:
- back-tier
- front-tier
mongo:
image: mongo
ports:
- 27017:27017
volumes:
- mongo_data:/data/db
networks:
- back-tier
- front-tier
================================================
FILE: cohorts/2022/05-monitoring/homework/model_training.py
================================================
import pickle
import pandas as pd
import pyarrow.parquet as pq
from sklearn.feature_extraction import DictVectorizer
from sklearn.linear_model import LinearRegression
def read_dataframe(filename):
df = pq.read_table(filename).to_pandas()
df.lpep_dropoff_datetime = pd.to_datetime(df.lpep_dropoff_datetime)
df.lpep_pickup_datetime = pd.to_datetime(df.lpep_pickup_datetime)
df['duration'] = df.lpep_dropoff_datetime - df.lpep_pickup_datetime
df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)
df = df[(df.duration >= 1) & (df.duration <= 60)]
categorical = ['PULocationID', 'DOLocationID']
df[categorical] = df[categorical].astype(str)
return df
def add_features(train_data="./datasets/green_tripdata_2021-03.parquet",
additional_training_data=None):
df_train = read_dataframe(train_data)
if additional_training_data:
extra_data = read_dataframe(additional_training_data)
df_train = pd.concat([df_train, extra_data], axis=0, ignore_index=True)
df_train['PU_DO'] = df_train['PULocationID'] + '_' + df_train['DOLocationID']
categorical = ['PU_DO']
numerical = ['trip_distance']
dv = DictVectorizer()
train_dicts = df_train[categorical + numerical].to_dict(orient='records')
X_train = dv.fit_transform(train_dicts)
target = 'duration'
y_train = df_train[target].values
return X_train, y_train, dv
if __name__ == "__main__":
X_train, y_train, dv = add_features()
print("Training model with one month of data")
lr = LinearRegression()
lr.fit(X_train, y_train)
with open('prediction_service/lin_reg.bin', 'wb') as f_out:
pickle.dump((dv, lr), f_out)
X_train, y_train, dv = add_features(additional_training_data="./datasets/green_tripdata_2021-04.parquet")
print("Training model with two months of data")
lr = LinearRegression()
lr.fit(X_train, y_train)
with open('prediction_service/lin_reg_V2.bin', 'wb') as f_out:
pickle.dump((dv, lr), f_out)
================================================
FILE: cohorts/2022/05-monitoring/homework/prediction_service/Dockerfile
================================================
FROM python:3.8-slim-buster
RUN pip install -U pip
RUN pip install pipenv
WORKDIR /app
COPY [ "Pipfile", "Pipfile.lock", "./" ]
RUN pipenv install --system --deploy
COPY [ "app.py", "lin_reg.bin", "lin_reg_V2.bin", "./" ]
EXPOSE 9696
ENTRYPOINT ["gunicorn", "--bind=0.0.0.0:9696", "app:app" ]
================================================
FILE: cohorts/2022/05-monitoring/homework/prediction_service/Pipfile
================================================
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
scikit-learn = "==1.0.2"
flask = "==2.0.1"
pandas = "==1.1.5"
evidently = "*"
pymongo = "*"
gunicorn = "*"
[dev-packages]
pyarrow = "*"
[requires]
python_version = "3.8"
================================================
FILE: cohorts/2022/05-monitoring/homework/prediction_service/app.py
================================================
import logging
import os
import pickle
import uuid
from flask import Flask, jsonify, request
from pymongo import MongoClient
MONGO_ADDRESS = os.getenv("MONGO_ADDRESS", "mongodb://localhost:27017/")
MONGO_DATABASE = os.getenv("MONGO_DATABASE", "ride_prediction")
LOGGED_MODEL = os.getenv("MODEL_FILE", "lin_reg.bin")
MODEL_VERSION = os.getenv("MODEL_VERSION", "1")
with open(LOGGED_MODEL, 'rb') as f_in:
dv, model = pickle.load(f_in)
mongo_client = MongoClient(MONGO_ADDRESS)
mongo_db = mongo_client[MONGO_DATABASE]
mongo_collection = mongo_db.get_collection("data")
app = Flask("Ride-Prediction-Service")
logging.basicConfig(level=logging.INFO)
def prepare_features(ride):
"""Function to prepare features before making prediction"""
record = ride.copy()
record['PU_DO'] = '%s_%s' % (record['PULocationID'], record['DOLocationID'])
features = dv.transform([record])
return features, record
def save_db(record, pred_result):
"""Save data to mongo db collection"""
rec = record.copy()
rec["prediction"] = pred_result[0]
mongo_collection.insert_one(rec)
@app.route("/", methods=["GET"])
def get_info():
"""Function to provide info about the app"""
info = """"ride = { "PULocationID": 10, "DOLocationID": 50, "trip_distance": 40 }"
| Name | Project | Social | Links and comments |
|---|---|---|---|
| Ashish Lalchandani | Project |   |
|
| Purity Nyagweth | Project | |
|
| Giovanni Pecoraro | Project | |
|
| Nakul Bajaj | Project | |
|
| Marcos M. Jiménez | Project | |
More info> I am still improving the project. A basic model, but the important is everything around it. |
| Mohammed Ayoub Chettouh | Project | |
More infoLinks: |
| Pedro Munoz | Project | |
|
| Oluwatosin Olajide | Project | |
|
| Sergei Zemskov | Project | |
|
| Esteban Encina | Project | |
|
| Rizdi Aprilian | Project | |
|
| Karim Lulu | Project | |
|
| Babatunde Ayanlola Emmanuel Ayanlowo | |
More infoLinks: > Experts in SQL Language, Analytic Engineering &Product Management Using Scrum Framework. |
|
| Camilo A. Caceres F. | Project | |
More infoLinks: |
| Yan Vlasov | Project | |
|
| Low Kim Hoe | Project | |
|
| Artem Glazkov | Project | |
|
| Jan Weber | Project | |
|
| Icaro Bombonato | Project | |
More info> Feel free to contact me :) |
| Adam Gifford | Project | |
More infoLinks: > I am the lead data scientist at BehaVR, a digital health and wellness startup looking to translate proven science into immersive and engaging XR therapeutics for mental and behavioral health. As we move towards our long-term goal of live operational monitoring and scoring of user activity to personalize experiences dynamically, this project serves as a foundation in framework development for real-time serving and monitoring of a prediction service to classify continuous-signal data. |
| Artem Li | Project | |
|
| Adesoba Adewale Olamide | Project | |
|
| Amir | Project | |
|
| Antonio Damiao Rodrigues | Project | |
|
| Walid Chrimni | Project | |
|
| Shivam Kurtarkar | Project | |
|
| Vyacheslav Shen | Project | |
More infoLinks: |
| Venkat Ramakrishnan | |||
| Deepak Ramani | Project | |
More infoLinks: > Awesome course. Great instructor. Learned a lot. Infrastructure surrounding ML is daunting but Alexey and Co make it simpler for us to get in. Highly recommended. |
| Frauke Albrecht | Project | |
|
| Rinat Gareev | Project | |
More info> In my job the most of MLOps experience I have is with Kubeflow or Amazon SageMaker (AWS). I used this project to learn something new and evaluate new frameworks, for example, Ray. Thanks for the opportunity and the course! |
| Isaac Omolayo | Project | |
|
| Antonis Stellas | Project | |
|
| Ayush Bihani | |
||
| Ruslan Romanov | Project | |
More info> Learning process was very exciting |
| Pablo Dalchiele | Project | |
|
| Nahum Sá | Project | |
|
| Patrick Walukagga | Project | |
More info> Enthusiastic Data Sciencist/Data Engineer/ML Engineer with a strong background in backend software engineering with Python. |
| Moiz Sajid | Project | |
More info> Thank you to the DataTalks.Club team for organizing this amazing course that covers the latest technologies and trends in MLOps! |
| Yusuf Yelken | Project | |
|
| Alexander Gabriel Valverde Guillen | Project | |
|
| Carlos Pereira | Project | |
|
| Daniel Hen | |
||
| bryskulov | Project | |
|
| Surawut Jirasaktavee | Project | |
|
| Sai Teja Macharla | |
||
| Senthilkumar Gopal | Project | |
More infoLinks: |
| Tim Becker | |
||
| Ezinwanne Aka | |
||
| Ammar Chalifah | Project | |
|
| Nwoke Tochukwu | |
||
| Alessandro S. | Project | |
|
| Hakeem Kazeem | |
||
| Isaac Kargar | Project | |
More infoLinks:
|
| Yalcin Erbasi | Project | |
## 3.2 Introduction to Prefect
## 3.3 Prefect Workflow
## 3.4 Deploying Your Workflow
## 3.5 Working with Deployments
## 3.6 Prefect Cloud (optional)
## 3.7 Homework
More information [here](../cohorts/2023/03-orchestration/homework.md).
## Quick setup
### Install packages
In a conda environment with Python 3.10.12 or similar, install all package dependencies with
```bash
pip install -r requirements.txt
```
### Start the Prefect server locally
Create another window and activate your conda environment. Start the Prefect API server locally with
```bash
prefect server start
```
### Alternative to self-hosted server use Prefect Cloud for added capabilties
Signup and use for free at | \n", " | VendorID | \n", "tpep_pickup_datetime | \n", "tpep_dropoff_datetime | \n", "passenger_count | \n", "trip_distance | \n", "RatecodeID | \n", "store_and_fwd_flag | \n", "PULocationID | \n", "DOLocationID | \n", "payment_type | \n", "... | \n", "extra | \n", "mta_tax | \n", "tip_amount | \n", "tolls_amount | \n", "improvement_surcharge | \n", "total_amount | \n", "congestion_surcharge | \n", "Airport_fee | \n", "duration | \n", "ride_id | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "2 | \n", "2023-03-01 00:06:43 | \n", "2023-03-01 00:16:43 | \n", "1.0 | \n", "0.00 | \n", "1.0 | \n", "N | \n", "238 | \n", "42 | \n", "2 | \n", "... | \n", "1.0 | \n", "0.5 | \n", "0.00 | \n", "0.0 | \n", "1.0 | \n", "11.10 | \n", "0.0 | \n", "0.00 | \n", "10.000000 | \n", "2023/03_0 | \n", "
| 1 | \n", "2 | \n", "2023-03-01 00:08:25 | \n", "2023-03-01 00:39:30 | \n", "2.0 | \n", "12.40 | \n", "1.0 | \n", "N | \n", "138 | \n", "231 | \n", "1 | \n", "... | \n", "6.0 | \n", "0.5 | \n", "12.54 | \n", "0.0 | \n", "1.0 | \n", "76.49 | \n", "2.5 | \n", "1.25 | \n", "31.083333 | \n", "2023/03_1 | \n", "
| 2 | \n", "1 | \n", "2023-03-01 00:15:04 | \n", "2023-03-01 00:29:26 | \n", "0.0 | \n", "3.30 | \n", "1.0 | \n", "N | \n", "140 | \n", "186 | \n", "1 | \n", "... | \n", "3.5 | \n", "0.5 | \n", "4.65 | \n", "0.0 | \n", "1.0 | \n", "28.05 | \n", "2.5 | \n", "0.00 | \n", "14.366667 | \n", "2023/03_2 | \n", "
| 3 | \n", "1 | \n", "2023-03-01 00:49:37 | \n", "2023-03-01 01:01:05 | \n", "1.0 | \n", "2.90 | \n", "1.0 | \n", "N | \n", "140 | \n", "43 | \n", "1 | \n", "... | \n", "3.5 | \n", "0.5 | \n", "4.10 | \n", "0.0 | \n", "1.0 | \n", "24.70 | \n", "2.5 | \n", "0.00 | \n", "11.466667 | \n", "2023/03_3 | \n", "
| 4 | \n", "2 | \n", "2023-03-01 00:08:04 | \n", "2023-03-01 00:11:06 | \n", "1.0 | \n", "1.23 | \n", "1.0 | \n", "N | \n", "79 | \n", "137 | \n", "1 | \n", "... | \n", "1.0 | \n", "0.5 | \n", "2.44 | \n", "0.0 | \n", "1.0 | \n", "14.64 | \n", "2.5 | \n", "0.00 | \n", "3.033333 | \n", "2023/03_4 | \n", "
5 rows × 21 columns
\n", "| \n", " | VendorID | \n", "tpep_pickup_datetime | \n", "tpep_dropoff_datetime | \n", "passenger_count | \n", "trip_distance | \n", "RatecodeID | \n", "store_and_fwd_flag | \n", "PULocationID | \n", "DOLocationID | \n", "payment_type | \n", "fare_amount | \n", "extra | \n", "mta_tax | \n", "tip_amount | \n", "tolls_amount | \n", "improvement_surcharge | \n", "total_amount | \n", "congestion_surcharge | \n", "airport_fee | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "2 | \n", "2023-01-01 00:32:10 | \n", "2023-01-01 00:40:36 | \n", "1.0 | \n", "0.97 | \n", "1.0 | \n", "N | \n", "161 | \n", "141 | \n", "2 | \n", "9.3 | \n", "1.00 | \n", "0.5 | \n", "0.00 | \n", "0.0 | \n", "1.0 | \n", "14.30 | \n", "2.5 | \n", "0.00 | \n", "
| 1 | \n", "2 | \n", "2023-01-01 00:55:08 | \n", "2023-01-01 01:01:27 | \n", "1.0 | \n", "1.10 | \n", "1.0 | \n", "N | \n", "43 | \n", "237 | \n", "1 | \n", "7.9 | \n", "1.00 | \n", "0.5 | \n", "4.00 | \n", "0.0 | \n", "1.0 | \n", "16.90 | \n", "2.5 | \n", "0.00 | \n", "
| 2 | \n", "2 | \n", "2023-01-01 00:25:04 | \n", "2023-01-01 00:37:49 | \n", "1.0 | \n", "2.51 | \n", "1.0 | \n", "N | \n", "48 | \n", "238 | \n", "1 | \n", "14.9 | \n", "1.00 | \n", "0.5 | \n", "15.00 | \n", "0.0 | \n", "1.0 | \n", "34.90 | \n", "2.5 | \n", "0.00 | \n", "
| 3 | \n", "1 | \n", "2023-01-01 00:03:48 | \n", "2023-01-01 00:13:25 | \n", "0.0 | \n", "1.90 | \n", "1.0 | \n", "N | \n", "138 | \n", "7 | \n", "1 | \n", "12.1 | \n", "7.25 | \n", "0.5 | \n", "0.00 | \n", "0.0 | \n", "1.0 | \n", "20.85 | \n", "0.0 | \n", "1.25 | \n", "
| 4 | \n", "2 | \n", "2023-01-01 00:10:29 | \n", "2023-01-01 00:21:19 | \n", "1.0 | \n", "1.43 | \n", "1.0 | \n", "N | \n", "107 | \n", "79 | \n", "1 | \n", "11.4 | \n", "1.00 | \n", "0.5 | \n", "3.28 | \n", "0.0 | \n", "1.0 | \n", "19.68 | \n", "2.5 | \n", "0.00 | \n", "
![](\"images/thumbnail-5-01.jpg\")
\n",
"\n",
"\n",
"\n",
"processing video yixA3C1xSxc...\n",
"saved to ..\\05-monitoring\\images\\thumbnail-5-02.jpg\n",
"## 5.2 Environment setup\n",
"\n",
"\n",
" ![](\"images/thumbnail-5-02.jpg\")
\n",
"\n",
"\n",
"\n",
"processing video IjNrkqMYQeQ...\n",
"saved to ..\\05-monitoring\\images\\thumbnail-5-03.jpg\n",
"## 5.3 Prepare reference and model\n",
"\n",
"\n",
" ![](\"images/thumbnail-5-03.jpg\")
\n",
"\n",
"\n",
"\n",
"processing video kP3lzh_HfWY...\n",
"saved to ..\\05-monitoring\\images\\thumbnail-5-04.jpg\n",
"## 5.4 Evidently metrics calculation\n",
"\n",
"\n",
" ![](\"images/thumbnail-5-04.jpg\")
\n",
"\n",
"\n",
"\n",
"processing video s3G4PMsOMOA...\n",
"saved to ..\\05-monitoring\\images\\thumbnail-5-05.jpg\n",
"## 5.5 Dummy monitoring\n",
"\n",
"\n",
" ![](\"images/thumbnail-5-05.jpg\")
\n",
"\n",
"\n",
"\n",
"processing video fytrmPbcLhI...\n",
"saved to ..\\05-monitoring\\images\\thumbnail-5-06.jpg\n",
"## 5.6 Data quality monitoring\n",
"\n",
"\n",
" ![](\"images/thumbnail-5-06.jpg\")
\n",
"\n",
"\n",
"\n",
"processing video -c4iumyZMyw...\n",
"saved to ..\\05-monitoring\\images\\thumbnail-5-07.jpg\n",
"## 5.7 Save Grafana Dashboard\n",
"\n",
"\n",
" ![](\"images/thumbnail-5-07.jpg\")
\n",
"\n",
"\n",
"\n",
"processing video sNSk3ojISh8...\n",
"saved to ..\\05-monitoring\\images\\thumbnail-5-08.jpg\n",
"## 5.8 Debugging with test suites and reports\n",
"\n",
"\n",
" ![](\"images/thumbnail-5-08.jpg\")
\n",
"\n",
"\n",
"\n"
]
}
],
"source": [
"module_number = module_info['number']\n",
"\n",
"parts = []\n",
"\n",
"for unit in units:\n",
" unit_number = unit['number']\n",
"\n",
" params = {\n",
" 'module_number': module_info['number'],\n",
" 'module_name': module_info['title'],\n",
" 'unit_number': unit['number'],\n",
" 'unit_title': unit['title'] \n",
" }\n",
"\n",
" if 'youtube' in unit:\n",
" youtube = unit['youtube']\n",
" if len(youtube) and youtube.startswith('https'):\n",
" thumbnail = download_thumbnail(youtube, module_number, unit_number, images_folder)\n",
" thumbnail_path = '/'.join(thumbnail.parts[2:])\n",
" params['youtube'] = f'{youtube}&list={playlist_id}'\n",
" params['thumbnail'] = thumbnail_path\n",
" \n",
" template_string = template.render(params)\n",
" print(template_string)\n",
" parts.append(template_string)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "6d6e3584",
"metadata": {},
"outputs": [],
"source": [
"prefix = f\"\"\"\n",
"# {module_info['number']}. {module_info['title']} \n",
"\"\"\".strip()\n",
"\n",
"\n",
"final_result = '\\n\\n'.join([prefix] + parts)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "4d900666",
"metadata": {},
"outputs": [],
"source": [
"clipboard.copy(final_result)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6ff9d662",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: learning-in-public.md
================================================
# Learning in public
Most people learn in private: they consume content but don't tell
anyone about it. There's nothing wrong with it.
But we want to encourage you to document your progress and
share it publicly on social media.
It helps you get noticed and will lead to:
* Expanding your network: meeting new people and making new friends
* Being invited to meetups, conferences and podcasts
* Landing a job or getting clients
* Many other good things
Here's a more compresensive reading on why you want to do it: https://github.com/readme/guides/publishing-your-work
## Learning in Public for Zoomcamps
When you submit your homework or project, you can also submit
learning in public posts:
You can watch this video to see how your learning in public posts may look like:
Send a PR if you want to suggest improvements for this document