[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-11-introduction-to-machine-learning)
## Notes
The concept of ML is depicted with an example of predicting the price of a car. The ML model
learns from data, represented as some **features** such as year, mileage, among others, and the **target** variable, in this
case, the car's price, by extracting patterns from the data.
Then, the model is given new data (**without** the target) about cars and predicts their price (target).
In summary, ML is a process of **extracting patterns from data**, which is of two types:
* features (information about the object) and
* target (property to predict for unseen objects).
Therefore, new feature values are presented to the model, and it makes **predictions** from the learned patterns.
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-12-ml-vs-rulebased-systems)
## Notes
The difference between ML and Rule-Based systems is explained with the example of a **spam filter**.
Traditional Rule-Based systems are based on a set of **characteristics** (keywords, email length, etc.) that identify an email as spam or not. As spam emails keep changing over time the system needs to be upgraded making the process untractable due to the complexity of code maintenance as the system grows.
ML can be used to solve this problem with the following steps:
### 1. Get data
Emails from the user's spam folder and inbox give examples of spam and non-spam.
### 2. Define and calculate features
Rules/characteristics from rule-based systems can be used as a starting point to define features for the ML model. The value of the target variable for each email can be defined based on where the email was obtained from (spam folder or inbox).
Each email can be encoded (converted) to the values of its features and target.
### 3. Train and use the model
A machine learning algorithm can then be applied to the encoded emails to build a model that can predict whether a new email is spam or not spam. The **predictions are probabilities**, and to make a decision it is necessary to define a threshold to classify emails as spam or not spam.
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-13-supervised-machine-learning)
## Notes
In Supervised Machine Learning (SML) there are always labels associated with certain features.
The model is trained, and then it can make predictions on new features. In this way, the model
is taught by certain features and targets.
* **Feature matrix (X):** made of observations or objects (rows) and features (columns).
* **Target variable (y):** a vector with the target information we want to predict. For each row of X there's a value in y.
The model can be represented as a function, **g**, that takes the feature matrix, **X**, as **input** and tries to predict values as close as possible to the targets, **y**. The process of **finding** this function **g** is called **training**.
### Types of SML problems
* **Regression:** the output is a number (car's price).
* **Classification:** the output is a category (spam example).
* **Binary:** there are two categories.
* **Multiclass problems:** there are more than two categories.
* **Ranking:** the output is the top scores associated with corresponding items. It is applied in recommender systems.
In summary, SML is about teaching the model by showing it different examples, and the goal is to come up with a function, that takes the feature matrix as input, and makes predictions of values as close as possible to the **y** targets.
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-14-crispdm)
## Notes
CRISP-DM, which stands for Cross-Industry Standard Process for Data Mining, is an open standard process model that describes common approaches used by data mining experts. It is the most widely-used analytics model. Conceived in 1996, it became a European Union project under the ESPRIT funding initiative in 1997. The project was led by five companies: Integral Solutions Ltd (ISL), Teradata, Daimler AG, NCR Corporation and OHRA, an insurance company:
1. **Business understanding:** An important question is do we need ML for the project. The goal of the project has to be measurable.
2. **Data understanding:** Analyze available data sources, and decide if more data is required.
3. **Data preparation:** Clean data, remove noise applying pipelines, and convert the data to a tabular format, so we can put it into ML.
4. **Modeling:** Train different models and choose the best one. Considering the results of this step, it is proper to decide if it is required to add new features or fix data issues.
5. **Evaluation:** Measure how well the model is performing and if it solves the business problem.
6. **Deployment:** Roll out to production to all the users. The evaluation and deployment often happen together - **online evaluation**.
It is important to consider how well maintainable the project is.
In general, ML projects require many iterations.
**Iteration:**
* Start simple
* Learn from the feedback
* Improve
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-15-model-selection-process)
## Notes
### Which model to choose?
- Logistic regression
- Decision tree
- Neural Network
- Or many others
The validation dataset is not used in training. There are feature matrices and y vectors
for both training and validation datasets.
The model is fitted with training data, and it is used to predict the y values of the validation
feature matrix. Then, the predicted y values (probabilities)
are compared with the actual y values.
**Multiple comparisons problem (MCP):** just by chance one model can be lucky and obtain
good predictions because all of them are probabilistic.
The test set can help to avoid the MCP. Obtaining the best model is done with the training and validation datasets, while the test dataset is used for assuring that the proposed best model is the best.
1. Split datasets in training, validation, and test. E.g. 60%, 20% and 20% respectively
2. Train the models
3. Evaluate the models
4. Select the best model
5. Apply the best model to the test dataset
6. Compare the performance metrics of validation and test
NB: Note that it is possible to reuse the validation data. After selecting the best model (step 4), the validation and training datasets can be combined to form a single training dataset for the chosen model before testing it on the test set.
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
# Understanding Numpy: A Simple Introduction
Numpy, short for Numerical Python, is a powerful Python library that enables efficient and convenient array manipulation and mathematical operations. It forms the foundation for many scientific and data-related tasks. In this article, we'll provide a straightforward explanation of Numpy concepts and how to use them.
## Importing Numpy
Before diving into Numpy's capabilities, we need to import it. Conventionally, we import Numpy with the alias `np`, making it easier to reference its functions:
```python
import numpy as np
```
## Creating Arrays
Arrays are the building blocks of Numpy, and they can be thought of as lists but with enhanced features.
### Creating Arrays with Zeros, Ones, or Constants
You can create arrays filled with zeros, ones, or any constant using `np.zeros()`, `np.ones()`, and `np.full()`:
```python
zeros_array = np.zeros(10)
ones_array = np.ones(10)
constant_array = np.full(10, 3)
```
### Converting Lists to Arrays
To convert a Python list into a Numpy array, you can use `np.array()`:
```python
my_list = [2, 3, 4]
array_from_list = np.array(my_list)
```
### Generating Ranges of Numbers
Numpy provides functions for generating arrays of sequential numbers. For example:
```python
range_array = np.arange(10) # Creates an array from 0 to 9
```
### Creating Arrays with Linear Spacing
`np.linspace()` creates arrays with evenly spaced numbers within a specified range:
```python
linspace_array = np.linspace(0, 1, 11) # Creates 11 numbers from 0 to 1
```
### Multi-dimensional Arrays
Numpy can handle multi-dimensional arrays, often referred to as matrices. Here are some examples:
```python
zeros_matrix = np.zeros((5, 2))
ones_matrix = np.ones((5, 2))
constant_matrix = np.full((5, 2), 3)
```
## Indexing and Slicing Arrays
Like Python lists, you can access elements in Numpy arrays using indexing and slicing. For two-dimensional arrays:
```python
arr = np.array([[2, 3, 4], [4, 5, 6]])
first_row = arr[0] # Gets the first row
first_col = arr[:, 0] # Gets the first column
```
## Generating Random Arrays
Numpy can create arrays filled with random numbers. To ensure reproducibility, you can set a seed using `np.random.seed()`:
```python
np.random.seed(2) # Set the seed
random_array = np.random.rand(5, 2) # Generates random numbers between 0 and 1
```
For random numbers from a normal distribution or integers within a range:
```python
normal_distribution = np.random.randn(5, 2)
random_integers = np.random.randint(low=0, high=100, size=(5, 2))
```
## Array Operations
Numpy excels in performing mathematical operations on arrays efficiently.
### Element-wise Operations
You can perform operations on entire arrays element by element:
```python
arr = arr + 1 # Adds 1 to each element
arr = arr * 2 # Multiplies each element by 2
# Similar operations for division and exponentiation
```
### Element-wise Operations with Two Arrays
You can also perform operations between two arrays of the same shape:
```python
arr1 = np.ones(4)
arr2 = np.full(4, 3)
result = arr1 + arr2 # Element-wise addition
result = arr1 / arr2 # Element-wise division
```
### Comparison Operations
You can perform element-wise comparisons and create boolean arrays:
```python
arr = np.array([1, 2, 3, 4])
greater_than_2 = arr > 2 # Produces [False, False, True, True]
```
### Selecting Elements Based on Conditions
You can create subarrays based on certain conditions:
```python
selected_elements = arr[arr > 1] # Gets elements greater than 1
```
## Summary Operations
Numpy provides functions for summarizing array data:
```python
min_value = arr.min() # Minimum value
max_value = arr.max() # Maximum value
sum_value = arr.sum() # Sum of all elements
mean_value = arr.mean() # Mean (average) value
std_deviation = arr.std() # Standard deviation
```
In conclusion, Numpy is an essential library for anyone working with numerical data in Python. It simplifies array creation, manipulation, and mathematical operations, making it a powerful tool for scientific computing and data analysis. With the basics covered in this article, you're well on your way to harnessing Numpy's capabilities.
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-18-linear-algebra-refresher)
## Notes
### Linear Algebra Refresher
* Vector operations
* Multiplication
* Vector-vector multiplication
* Matrix-vector multiplication
* Matrix-matrix multiplication
* Identity matrix
* Inverse
### Vector operations
~~~~python
u = np.array([2, 7, 5, 6])
v = np.array([3, 4, 8, 6])
# addition
u + v
# subtraction
u - v
# scalar multiplication
2 * v
~~~~
### Multiplication
##### Vector-vector multiplication
~~~~python
def vector_vector_multiplication(u, v):
assert u.shape[0] == v.shape[0]
n = u.shape[0]
result = 0.0
for i in range(n):
result = result + u[i] * v[i]
return result
~~~~
##### Matrix-vector multiplication
~~~~python
def matrix_vector_multiplication(U, v):
assert U.shape[1] == v.shape[0]
num_rows = U.shape[0]
result = np.zeros(num_rows)
for i in range(num_rows):
result[i] = vector_vector_multiplication(U[i], v)
return result
~~~~
##### Matrix-matrix multiplication
~~~~python
def matrix_matrix_multiplication(U, V):
assert U.shape[1] == V.shape[0]
num_rows = U.shape[0]
num_cols = V.shape[1]
result = np.zeros((num_rows, num_cols))
for i in range(num_cols):
vi = V[:, i]
Uvi = matrix_vector_multiplication(U, vi)
result[:, i] = Uvi
return result
~~~~
### Identity matrix
~~~~python
I = np.eye(3)
~~~~
### Inverse
~~~~python
V = np.array([
[1, 1, 2],
[0, 0.5, 1],
[0, 2, 1],
])
inv = np.linalg.inv(V)
~~~~
Add notes here (PRs are welcome).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes here (PRs are welcome).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-110-summary)
## Notes
---
### 📚 Summary of First Session - Machine Learning Zoomcamp
1. **🚗 Introduction to Machine Learning with Cars Data**
We start with data about cars, including characteristics (features) and prices (target). A Machine Learning (ML) model can be used to extract patterns from known information (data) about some cars in order to predict car prices based on their characteristics.
2. **🧠 Rules-Based Systems vs. Machine Learning**
- **Rules-Based Systems:** It is necessary to manually convert rules into code using a programming language and apply them to data. Extracting patterns manually can become complex and challenging.
- **Machine Learning:** Instead of manually coding rules, ML models automatically extract patterns from data using Mathematics and Statistics.
3. **🔍 Supervised Machine Learning**
In supervised learning, models learn from labeled data (with known outcomes) to make predictions on unseen data.
4. **🛠️ CRISP-DM (Cross Industry Standard Process for Data Mining)**
A structured methodology for organizing ML projects, consisting of the following steps:
- 💼 **Business Understanding**
- 🔎 **Data Understanding**
- 🧹 **Data Preparation**
- 🤖 **Modeling** (choosing and training models, then selecting the best one)
- 📊 **Evaluation**
- 🚀 **Deployment**
This process is iterative, allowing for continuous improvement.
5. **🏆 Model Selection**
Split data into training, validation, and test sets. Train different models, validate them, select the best performing one, and then test it on the test set to ensure generalization.
6. **💻 Setting Up the Environment**
Install necessary tools like Python, Numpy, Pandas, Matplotlib, Scikit-learn. Anaconda is the easiest option. Eventually create an AWS account for cloud resources.
7. **🔢 Introduction to Numpy**
Numpy is crucial for manipulating numerical data, providing efficient operations on arrays and matrices.
8. **🔗 Linear Algebra**
Covering all types of multiplication with vectors and matrices, including the creation of identity matrices using functions like `np.eye()`.
9. **📊 Introduction to Pandas**
Pandas is a Python library used for processing and analyzing tabular data efficiently.
---
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Nissan | \n", "Stanza | \n", "1991 | \n", "138.0 | \n", "4 | \n", "MANUAL | \n", "sedan | \n", "2000 | \n", "
| 1 | \n", "Hyundai | \n", "Sonata | \n", "2017 | \n", "NaN | \n", "4 | \n", "AUTOMATIC | \n", "Sedan | \n", "27150 | \n", "
| 2 | \n", "Lotus | \n", "Elise | \n", "2010 | \n", "218.0 | \n", "4 | \n", "MANUAL | \n", "convertible | \n", "54990 | \n", "
| 3 | \n", "GMC | \n", "Acadia | \n", "2017 | \n", "194.0 | \n", "4 | \n", "AUTOMATIC | \n", "4dr SUV | \n", "34450 | \n", "
| 4 | \n", "Nissan | \n", "Frontier | \n", "2017 | \n", "261.0 | \n", "6 | \n", "MANUAL | \n", "Pickup | \n", "32340 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Nissan | \n", "Stanza | \n", "1991 | \n", "138.0 | \n", "4 | \n", "MANUAL | \n", "sedan | \n", "2000 | \n", "
| 1 | \n", "Hyundai | \n", "Sonata | \n", "2017 | \n", "NaN | \n", "4 | \n", "AUTOMATIC | \n", "Sedan | \n", "27150 | \n", "
| 2 | \n", "Lotus | \n", "Elise | \n", "2010 | \n", "218.0 | \n", "4 | \n", "MANUAL | \n", "convertible | \n", "54990 | \n", "
| 3 | \n", "GMC | \n", "Acadia | \n", "2017 | \n", "194.0 | \n", "4 | \n", "AUTOMATIC | \n", "4dr SUV | \n", "34450 | \n", "
| 4 | \n", "Nissan | \n", "Frontier | \n", "2017 | \n", "261.0 | \n", "6 | \n", "MANUAL | \n", "Pickup | \n", "32340 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Nissan | \n", "Stanza | \n", "1991 | \n", "138.0 | \n", "4 | \n", "MANUAL | \n", "sedan | \n", "2000 | \n", "
| 1 | \n", "Hyundai | \n", "Sonata | \n", "2017 | \n", "NaN | \n", "4 | \n", "AUTOMATIC | \n", "Sedan | \n", "27150 | \n", "
| \n", " | Make | \n", "Model | \n", "MSRP | \n", "
|---|---|---|---|
| 0 | \n", "Nissan | \n", "Stanza | \n", "2000 | \n", "
| 1 | \n", "Hyundai | \n", "Sonata | \n", "27150 | \n", "
| 2 | \n", "Lotus | \n", "Elise | \n", "54990 | \n", "
| 3 | \n", "GMC | \n", "Acadia | \n", "34450 | \n", "
| 4 | \n", "Nissan | \n", "Frontier | \n", "32340 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "id | \n", "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Nissan | \n", "Stanza | \n", "1991 | \n", "138.0 | \n", "4 | \n", "MANUAL | \n", "sedan | \n", "2000 | \n", "10 | \n", "
| 1 | \n", "Hyundai | \n", "Sonata | \n", "2017 | \n", "NaN | \n", "4 | \n", "AUTOMATIC | \n", "Sedan | \n", "27150 | \n", "20 | \n", "
| 2 | \n", "Lotus | \n", "Elise | \n", "2010 | \n", "218.0 | \n", "4 | \n", "MANUAL | \n", "convertible | \n", "54990 | \n", "30 | \n", "
| 3 | \n", "GMC | \n", "Acadia | \n", "2017 | \n", "194.0 | \n", "4 | \n", "AUTOMATIC | \n", "4dr SUV | \n", "34450 | \n", "40 | \n", "
| 4 | \n", "Nissan | \n", "Frontier | \n", "2017 | \n", "261.0 | \n", "6 | \n", "MANUAL | \n", "Pickup | \n", "32340 | \n", "50 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Nissan | \n", "Stanza | \n", "1991 | \n", "138.0 | \n", "4 | \n", "MANUAL | \n", "sedan | \n", "2000 | \n", "
| 1 | \n", "Hyundai | \n", "Sonata | \n", "2017 | \n", "NaN | \n", "4 | \n", "AUTOMATIC | \n", "Sedan | \n", "27150 | \n", "
| 2 | \n", "Lotus | \n", "Elise | \n", "2010 | \n", "218.0 | \n", "4 | \n", "MANUAL | \n", "convertible | \n", "54990 | \n", "
| 3 | \n", "GMC | \n", "Acadia | \n", "2017 | \n", "194.0 | \n", "4 | \n", "AUTOMATIC | \n", "4dr SUV | \n", "34450 | \n", "
| 4 | \n", "Nissan | \n", "Frontier | \n", "2017 | \n", "261.0 | \n", "6 | \n", "MANUAL | \n", "Pickup | \n", "32340 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|
| a | \n", "Nissan | \n", "Stanza | \n", "1991 | \n", "138.0 | \n", "4 | \n", "MANUAL | \n", "sedan | \n", "2000 | \n", "
| b | \n", "Hyundai | \n", "Sonata | \n", "2017 | \n", "NaN | \n", "4 | \n", "AUTOMATIC | \n", "Sedan | \n", "27150 | \n", "
| c | \n", "Lotus | \n", "Elise | \n", "2010 | \n", "218.0 | \n", "4 | \n", "MANUAL | \n", "convertible | \n", "54990 | \n", "
| d | \n", "GMC | \n", "Acadia | \n", "2017 | \n", "194.0 | \n", "4 | \n", "AUTOMATIC | \n", "4dr SUV | \n", "34450 | \n", "
| e | \n", "Nissan | \n", "Frontier | \n", "2017 | \n", "261.0 | \n", "6 | \n", "MANUAL | \n", "Pickup | \n", "32340 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|
| b | \n", "Hyundai | \n", "Sonata | \n", "2017 | \n", "NaN | \n", "4 | \n", "AUTOMATIC | \n", "Sedan | \n", "27150 | \n", "
| c | \n", "Lotus | \n", "Elise | \n", "2010 | \n", "218.0 | \n", "4 | \n", "MANUAL | \n", "convertible | \n", "54990 | \n", "
| e | \n", "Nissan | \n", "Frontier | \n", "2017 | \n", "261.0 | \n", "6 | \n", "MANUAL | \n", "Pickup | \n", "32340 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Nissan | \n", "Stanza | \n", "1991 | \n", "138.0 | \n", "4 | \n", "MANUAL | \n", "sedan | \n", "2000 | \n", "
| 1 | \n", "Hyundai | \n", "Sonata | \n", "2017 | \n", "NaN | \n", "4 | \n", "AUTOMATIC | \n", "Sedan | \n", "27150 | \n", "
| 2 | \n", "Lotus | \n", "Elise | \n", "2010 | \n", "218.0 | \n", "4 | \n", "MANUAL | \n", "convertible | \n", "54990 | \n", "
| 3 | \n", "GMC | \n", "Acadia | \n", "2017 | \n", "194.0 | \n", "4 | \n", "AUTOMATIC | \n", "4dr SUV | \n", "34450 | \n", "
| 4 | \n", "Nissan | \n", "Frontier | \n", "2017 | \n", "261.0 | \n", "6 | \n", "MANUAL | \n", "Pickup | \n", "32340 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Nissan | \n", "Stanza | \n", "1991 | \n", "138.0 | \n", "4 | \n", "MANUAL | \n", "sedan | \n", "2000 | \n", "
| 4 | \n", "Nissan | \n", "Frontier | \n", "2017 | \n", "261.0 | \n", "6 | \n", "MANUAL | \n", "Pickup | \n", "32340 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|
| 4 | \n", "Nissan | \n", "Frontier | \n", "2017 | \n", "261.0 | \n", "6 | \n", "MANUAL | \n", "Pickup | \n", "32340 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Vehicle_Style | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Nissan | \n", "Stanza | \n", "1991 | \n", "138.0 | \n", "4 | \n", "MANUAL | \n", "sedan | \n", "2000 | \n", "
| 1 | \n", "Hyundai | \n", "Sonata | \n", "2017 | \n", "NaN | \n", "4 | \n", "AUTOMATIC | \n", "sedan | \n", "27150 | \n", "
| 2 | \n", "Lotus | \n", "Elise | \n", "2010 | \n", "218.0 | \n", "4 | \n", "MANUAL | \n", "convertible | \n", "54990 | \n", "
| 3 | \n", "GMC | \n", "Acadia | \n", "2017 | \n", "194.0 | \n", "4 | \n", "AUTOMATIC | \n", "4dr_suv | \n", "34450 | \n", "
| 4 | \n", "Nissan | \n", "Frontier | \n", "2017 | \n", "261.0 | \n", "6 | \n", "MANUAL | \n", "pickup | \n", "32340 | \n", "
| \n", " | Year | \n", "Engine HP | \n", "Engine Cylinders | \n", "MSRP | \n", "
|---|---|---|---|---|
| count | \n", "5.00 | \n", "4.00 | \n", "5.00 | \n", "5.00 | \n", "
| mean | \n", "2010.40 | \n", "202.75 | \n", "4.40 | \n", "30186.00 | \n", "
| std | \n", "11.26 | \n", "51.30 | \n", "0.89 | \n", "18985.04 | \n", "
| min | \n", "1991.00 | \n", "138.00 | \n", "4.00 | \n", "2000.00 | \n", "
| 25% | \n", "2010.00 | \n", "180.00 | \n", "4.00 | \n", "27150.00 | \n", "
| 50% | \n", "2017.00 | \n", "206.00 | \n", "4.00 | \n", "32340.00 | \n", "
| 75% | \n", "2017.00 | \n", "228.75 | \n", "4.00 | \n", "34450.00 | \n", "
| max | \n", "2017.00 | \n", "261.00 | \n", "6.00 | \n", "54990.00 | \n", "
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-21-car-price-prediction-project)
## Notes
This project is about the creation of a model for helping users to predict car prices. The dataset was obtained from [this
kaggle competition](https://www.kaggle.com/CooperUnion/cardataset).
**Project plan:**
* Prepare data and Exploratory data analysis (EDA)
* Use linear regression for predicting price
* Understanding the internals of linear regression
* Evaluating the model with RMSE
* Feature engineering
* Regularization
* Using the model
The code and dataset are available at this [link](https://github.com/alexeygrigorev/mlbookcamp-code/tree/master/chapter-02-car-price).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
**Pandas attributes and methods:**
* `pd.read_csv(| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
**Pandas attributes and methods:**
* `df[col].unique()` -> return a list of unique values in the series
* `df[col].nunique()` -> return the number of unique values in the series
* `df.isnull().sum()` -> return the number of null values in the dataframe
**Matplotlib and seaborn methods:**
* `%matplotlib inline` -> assure that plots are displayed in jupyter notebook's cells
* `sns.histplot()` -> show the histogram of a series
**Numpy methods:**
* `np.log1p()` -> apply log transformation to a variable, after adding one to each input value.
Long-tail distributions usually confuse the ML models, so the recommendation is to transform the target variable distribution to a normal one whenever possible.
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
In general, the dataset is splitted into three parts: training, validation, and test. For each partition, we need to obtain feature matrices (X) and vectors of targets (y). First, the size of the partitions is calculated. Next, the records are shuffled to ensure that the values in the three partitions contain non-sequential records from the dataset. Finally, the partitions are created using the shuffled indices.
**Pandas attributes and methods:**
* `df.iloc[]` -> return subsets of records of a dataframe, being selected by numerical indices
* `df.reset_index()` -> restate the orginal indices
* `del df[col]` -> eliminate a column variable
**Numpy methods:**
* `np.arange()` -> return an array of numbers
* `np.random.shuffle()` -> return a shuffled array
* `np.random.seed()` -> set a seed for reproducibility
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
Model for solving regression tasks, in which the objective is to adjust a line for the data and make predictions on new values. The input of this model is the **feature matrix** `X` and a `y` **vector of predictions** is obtained, trying to be as close as possible to the **actual** `y` values. The linear regression formula is the sum of the bias term \( $w_0$ \), which refers to the predictions if there is no information, and each of the feature values times their corresponding weights as \( $x_{i1} \cdot w_1 + x_{i2} \cdot w_2 + ... + x_{in} \cdot w_n$ \).
So the simple linear regression formula looks like:
$g(x_i) = w_0 + x_{i1} \cdot w_1 + x_{i2} \cdot w_2 + ... + x_{in} \cdot w_n$.
And that can be further simplified as:
$g(x_i) = w_0 + \displaystyle\sum_{j=1}^{n} w_j \cdot x_{ij}$
Here is a simple implementation of Linear Regression in python:
~~~~python
w0 = 7.1
def linear_regression(xi):
n = len(xi)
pred = w0
w = [0.01, 0.04, 0.002]
for j in range(n):
pred = pred + w[j] * xi[j]
return pred
~~~~
If we look at the $\displaystyle\sum_{j=1}^{n} w_j \cdot x_{ij}$ part in the above equation, we know that this is nothing else but a vector-vector multiplication. Hence, we can rewrite the equation as $g(x_i) = w_0 + x_i^T \cdot w$
We need to assure that the result is shown on the untransformed scale by using the inverse function `exp()`.
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
The formula of linear regression can be synthesized with the dot product between features and weights. The feature vector includes the *bias* term with an *x* value of one, such as $w_{0}^{x_{i0}},\ where\ x_{i0} = 1\ for\ w_0$.
When all the records are included, the linear regression can be calculated with the dot product between ***feature matrix*** and ***vector of weights***, obtaining the `y` vector of predictions.
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
Obtaining predictions as close as possible to $y$ target values requires the calculation of weights from the general
LR equation. The feature matrix does not
have an inverse because it is not square, so it is required to obtain an approximate solution, which can be
obtained using the **Gram matrix**
(multiplication of feature matrix ($X$) and its transpose ($X^T$)). The vector of weights or coefficients $w$ obtained with this
formula is the closest possible solution to the LR system.
Normal Equation:
$w$ = $(X^TX)^{-1}X^Ty$
Where:
$X^TX$ is the Gram Matrix
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
* In this lesson we build a baseline model and apply the `df_train` dataset to derive weights for the bias (w0) and the features (w). For this, we use the `train_linear_regression(X, y)` function from the previous lesson.
* Linear regression only applies to numerical features. Therefore, only the numerical features from `df_train` are used for the feature matrix.
* We notice some of the features in `df_train` are `nan`. We set them to `0` for the sake of simplicity, so the model is solvable, but it will be appropriate if a non-zeo value is used as the filler (e.g. mean value of the feature).
* Once the weights are calculated, then we apply them on $$\\\\ \large g(X) = w_0 + X \cdot w$$ to derive the predicted y vector.
* Then we plot both predicted y and the actual y on the same histogram for a visual comparison.
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
* In the previous lesson we found out our predictions were a bit off from the actual target values in the training dataset. We need a way to quantify how good or bad the model is. This is where RMSE can be of help.
* Root Mean Squared Error (RMSE) is a way to evaluate regression models. It measures the error associated with the model being evaluated. This numerical figure can then be used to compare models, enabling us to choose the one that gives the best predictions.
$$RMSE = \sqrt{ \frac{1}{m} \sum_{i=1}^{m} {(g(x_i) - y_i)^2}}$$
- $g(x_i)$ is the prediction
- $y_i$ is the actual value
- $m$ is the number of observations in the dataset (i.e. cars)
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
Calculation of the RMSE on validation partition of the dataset of car price prediction. In this way, we have a metric to evaluate the model's
performance.
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
The feature age of the car was included in the dataset, obtained with the subtraction of the maximum year of cars and each of the years of cars.
This new feature improved the model performance, measured with the RMSE and comparing the distributions of y target variable and predictions.
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
Categorical variables are typically represented as strings, and pandas identifies them as object types. However, some variables that appear to be numerical may actually be categorical (e.g., the number of doors a car has). All these categorical variables need to be converted to a numerical form because ML
models can interpret only numerical features. It is possible to incorporate certain categories from a feature, not necessarily all of them.
This transformation from categorical to numerical variables is known as One-Hot encoding.
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
If the feature matrix has duplicate columns (or columns that can be expressed as a linear combination of other columns), it will not have an inverse matrix. But, sometimes this error could be passed if certain values are slightly different
between duplicated columns.
So, if we apply the normal equation with this feature matrix, the values associated with duplicated columns are very large, which decreases
the model performance. To solve this issue, one alternative is adding a small number to the diagonal of the feature matrix, which corresponds to regularization.
This technique
works because the addition of small values to the diagonal makes it less likely to have duplicated columns. The regularization value is a hyperparameter of the model. After applying
regularization the model performance improved.
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
Tuning the model consisted of finding the best regularization hyperparameter value, using the validation partition of the dataset. The model was then trained with this regularization value.
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-2-slides)
## Notes
After finding the best model and its parameters, it was trained with training and validation partitions and the final RMSE was calculated on the test partition.
Finally, the final model was used to predict the price of new cars.
The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-02-car-price/02-carprice.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
In summary, this session covered some topics, including data preparation, exploratory data analysis, the validation framework, linear regression model, LR vector and
normal forms, the baseline model, root mean squared error, feature engineering, regularization, tuning the model, and using the best model with new data. All these concepts
were explained using the problem to predict the price of cars.
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
| \n", " | make | \n", "model | \n", "year | \n", "engine_fuel_type | \n", "engine_hp | \n", "engine_cylinders | \n", "transmission_type | \n", "driven_wheels | \n", "number_of_doors | \n", "market_category | \n", "vehicle_size | \n", "vehicle_style | \n", "highway_mpg | \n", "city_mpg | \n", "popularity | \n", "msrp | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "bmw | \n", "1_series_m | \n", "2011 | \n", "premium_unleaded_(required) | \n", "335.0 | \n", "6.0 | \n", "manual | \n", "rear_wheel_drive | \n", "2.0 | \n", "factory_tuner,luxury,high-performance | \n", "compact | \n", "coupe | \n", "26 | \n", "19 | \n", "3916 | \n", "46135 | \n", "
| 1 | \n", "bmw | \n", "1_series | \n", "2011 | \n", "premium_unleaded_(required) | \n", "300.0 | \n", "6.0 | \n", "manual | \n", "rear_wheel_drive | \n", "2.0 | \n", "luxury,performance | \n", "compact | \n", "convertible | \n", "28 | \n", "19 | \n", "3916 | \n", "40650 | \n", "
| 2 | \n", "bmw | \n", "1_series | \n", "2011 | \n", "premium_unleaded_(required) | \n", "300.0 | \n", "6.0 | \n", "manual | \n", "rear_wheel_drive | \n", "2.0 | \n", "luxury,high-performance | \n", "compact | \n", "coupe | \n", "28 | \n", "20 | \n", "3916 | \n", "36350 | \n", "
| 3 | \n", "bmw | \n", "1_series | \n", "2011 | \n", "premium_unleaded_(required) | \n", "230.0 | \n", "6.0 | \n", "manual | \n", "rear_wheel_drive | \n", "2.0 | \n", "luxury,performance | \n", "compact | \n", "coupe | \n", "28 | \n", "18 | \n", "3916 | \n", "29450 | \n", "
| 4 | \n", "bmw | \n", "1_series | \n", "2011 | \n", "premium_unleaded_(required) | \n", "230.0 | \n", "6.0 | \n", "manual | \n", "rear_wheel_drive | \n", "2.0 | \n", "luxury | \n", "compact | \n", "convertible | \n", "28 | \n", "18 | \n", "3916 | \n", "34500 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 11909 | \n", "acura | \n", "zdx | \n", "2012 | \n", "premium_unleaded_(required) | \n", "300.0 | \n", "6.0 | \n", "automatic | \n", "all_wheel_drive | \n", "4.0 | \n", "crossover,hatchback,luxury | \n", "midsize | \n", "4dr_hatchback | \n", "23 | \n", "16 | \n", "204 | \n", "46120 | \n", "
| 11910 | \n", "acura | \n", "zdx | \n", "2012 | \n", "premium_unleaded_(required) | \n", "300.0 | \n", "6.0 | \n", "automatic | \n", "all_wheel_drive | \n", "4.0 | \n", "crossover,hatchback,luxury | \n", "midsize | \n", "4dr_hatchback | \n", "23 | \n", "16 | \n", "204 | \n", "56670 | \n", "
| 11911 | \n", "acura | \n", "zdx | \n", "2012 | \n", "premium_unleaded_(required) | \n", "300.0 | \n", "6.0 | \n", "automatic | \n", "all_wheel_drive | \n", "4.0 | \n", "crossover,hatchback,luxury | \n", "midsize | \n", "4dr_hatchback | \n", "23 | \n", "16 | \n", "204 | \n", "50620 | \n", "
| 11912 | \n", "acura | \n", "zdx | \n", "2013 | \n", "premium_unleaded_(recommended) | \n", "300.0 | \n", "6.0 | \n", "automatic | \n", "all_wheel_drive | \n", "4.0 | \n", "crossover,hatchback,luxury | \n", "midsize | \n", "4dr_hatchback | \n", "23 | \n", "16 | \n", "204 | \n", "50920 | \n", "
| 11913 | \n", "lincoln | \n", "zephyr | \n", "2006 | \n", "regular_unleaded | \n", "221.0 | \n", "6.0 | \n", "automatic | \n", "front_wheel_drive | \n", "4.0 | \n", "luxury | \n", "midsize | \n", "sedan | \n", "26 | \n", "17 | \n", "61 | \n", "28995 | \n", "
11914 rows × 16 columns
\n", "| \n", " | make | \n", "model | \n", "year | \n", "engine_fuel_type | \n", "engine_hp | \n", "engine_cylinders | \n", "transmission_type | \n", "driven_wheels | \n", "number_of_doors | \n", "market_category | \n", "vehicle_size | \n", "vehicle_style | \n", "highway_mpg | \n", "city_mpg | \n", "popularity | \n", "msrp | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 10 | \n", "bmw | \n", "1_series | \n", "2013 | \n", "premium_unleaded_(required) | \n", "300.0 | \n", "6.0 | \n", "manual | \n", "rear_wheel_drive | \n", "2.0 | \n", "luxury,high-performance | \n", "compact | \n", "coupe | \n", "28 | \n", "20 | \n", "3916 | \n", "39600 | \n", "
| 0 | \n", "bmw | \n", "1_series_m | \n", "2011 | \n", "premium_unleaded_(required) | \n", "335.0 | \n", "6.0 | \n", "manual | \n", "rear_wheel_drive | \n", "2.0 | \n", "factory_tuner,luxury,high-performance | \n", "compact | \n", "coupe | \n", "26 | \n", "19 | \n", "3916 | \n", "46135 | \n", "
| 3 | \n", "bmw | \n", "1_series | \n", "2011 | \n", "premium_unleaded_(required) | \n", "230.0 | \n", "6.0 | \n", "manual | \n", "rear_wheel_drive | \n", "2.0 | \n", "luxury,performance | \n", "compact | \n", "coupe | \n", "28 | \n", "18 | \n", "3916 | \n", "29450 | \n", "
| 5 | \n", "bmw | \n", "1_series | \n", "2012 | \n", "premium_unleaded_(required) | \n", "230.0 | \n", "6.0 | \n", "manual | \n", "rear_wheel_drive | \n", "2.0 | \n", "luxury,performance | \n", "compact | \n", "coupe | \n", "28 | \n", "18 | \n", "3916 | \n", "31200 | \n", "
| \n", " | make | \n", "model | \n", "year | \n", "engine_fuel_type | \n", "engine_hp | \n", "engine_cylinders | \n", "transmission_type | \n", "driven_wheels | \n", "number_of_doors | \n", "market_category | \n", "vehicle_size | \n", "vehicle_style | \n", "highway_mpg | \n", "city_mpg | \n", "popularity | \n", "msrp | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2735 | \n", "chevrolet | \n", "cobalt | \n", "2008 | \n", "regular_unleaded | \n", "148.0 | \n", "4.0 | \n", "manual | \n", "front_wheel_drive | \n", "2.0 | \n", "NaN | \n", "compact | \n", "coupe | \n", "33 | \n", "24 | \n", "1385 | \n", "14410 | \n", "
| 6720 | \n", "toyota | \n", "matrix | \n", "2012 | \n", "regular_unleaded | \n", "132.0 | \n", "4.0 | \n", "automatic | \n", "front_wheel_drive | \n", "4.0 | \n", "hatchback | \n", "compact | \n", "4dr_hatchback | \n", "32 | \n", "25 | \n", "2031 | \n", "19685 | \n", "
| 5878 | \n", "subaru | \n", "impreza | \n", "2016 | \n", "regular_unleaded | \n", "148.0 | \n", "4.0 | \n", "automatic | \n", "all_wheel_drive | \n", "4.0 | \n", "hatchback | \n", "compact | \n", "4dr_hatchback | \n", "37 | \n", "28 | \n", "640 | \n", "19795 | \n", "
| 11190 | \n", "volkswagen | \n", "vanagon | \n", "1991 | \n", "regular_unleaded | \n", "90.0 | \n", "4.0 | \n", "manual | \n", "rear_wheel_drive | \n", "3.0 | \n", "NaN | \n", "large | \n", "passenger_minivan | \n", "18 | \n", "16 | \n", "873 | \n", "2000 | \n", "
| 4554 | \n", "ford | \n", "f-150 | \n", "2017 | \n", "flex-fuel_(unleaded/e85) | \n", "385.0 | \n", "8.0 | \n", "automatic | \n", "four_wheel_drive | \n", "4.0 | \n", "flex_fuel | \n", "large | \n", "crew_cab_pickup | \n", "21 | \n", "15 | \n", "5657 | \n", "56260 | \n", "
| \n", " | make | \n", "model | \n", "year | \n", "engine_fuel_type | \n", "engine_hp | \n", "engine_cylinders | \n", "transmission_type | \n", "driven_wheels | \n", "number_of_doors | \n", "market_category | \n", "vehicle_size | \n", "vehicle_style | \n", "highway_mpg | \n", "city_mpg | \n", "popularity | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "toyota | \n", "sienna | \n", "2015 | \n", "regular_unleaded | \n", "266.0 | \n", "6.0 | \n", "automatic | \n", "front_wheel_drive | \n", "4.0 | \n", "NaN | \n", "large | \n", "passenger_minivan | \n", "25 | \n", "18 | \n", "2031 | \n", "
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification) empty?
## Notes
The project aims to identify customers that are likely to churn or stop to using a service. Each customer has a score associated with the probability of churning. Considering this data, the company would send an email with discounts or other promotions to avoid churning.
The ML strategy applied to approach this problem is binary classification, which for one instance ($i^{th}$ customer), can be expressed as:
$$\large g\left(x_{i}\right) = y_{i}$$
In the formula, $y_i$ is the model's prediction and belongs to {0,1}, with 0 being the negative value or no churning, and 1 the positive value or churning. The output corresponds to the likelihood of churning.
In brief, the main idea behind this project is to build a model with historical data from customers and assign a score of the likelihood of churning.
For this project, we used a [Kaggle dataset](https://www.kaggle.com/blastchar/telco-customer-churn).
|⚠️|The notes are written by the community.
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
This session covered data obtention and some procedures of data preparation.
**Commands, functions, and methods:**
* `!wget` - Linux shell command for downloading data
* `pd.read.csv()` - read csv files
* `df.head()` - take a look of the dataframe
* `df.head().T` - take a look of the transposed dataframe
* `df.columns` - retrieve column names of a dataframe
* `df.columns.str.lower()` - lowercase all the letters in the columns names of a dataframe
* `df.columns.str.replace(' ', '_')` - replace the space separator in the columns names of a dataframe
* `df.dtypes` - retrieve data types of all series
* `df.index` - retrieve indices of a dataframe
* `pd.to_numeric()` - convert a series values to numerical values. The `errors='coerce'` argument allows making the transformation despite some encountered errors.
* `df.fillna()` - replace NAs with some value
* `(df.x == "yes").astype(int)` - convert x series of yes-no values to numerical values.
The entire code of this project is available in [this jupyter notebook](https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/03-classification/notebook.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
Splitting the dataset with **Scikit-Learn**.
**Classes, functions, and methods:**
* `train_test_split` - Scikit-Learn class for splitting a dataset into two parts. The `test_size` argument states how large the test set should be. The `random_state` argument sets a random seed for reproducibility purposes.
* `df.reset_index(drop=True)` - reset the indices of a dataframe and delete the previous ones.
* `df.x.values` - extract the values from x series
* `del df['x']` - delete x series from a dataframe
The entire code of this project is available in [this jupyter notebook](https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/03-classification/notebook.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
The EDA for this project consisted of:
* Checking missing values
* Looking at the distribution of the target variable (churn)
* Looking at numerical and categorical variables
**Functions and methods:**
* `df.isnull().sum()` - returns the number of null values in the dataframe.
* `df.x.value_counts()` returns the number of values for each category in x series. The `normalize=True` argument retrieves the percentage of each category. In this project, the mean of churn is equal to the churn rate obtained with the value_counts method.
* `round(x, y)` - round an x number with y decimal places
* `df[x].nunique()` - returns the number of unique values in x series
The entire code of this project is available in [this jupyter notebook](https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/03-classification/notebook.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
1. **Churn rate:** Difference between global mean of the target variable and mean of the target variable for categories of a feature. If this difference is greater than 0, it means that the category is less likely to churn, and if the difference is lower than 0, the group is more likely to churn. The larger differences are indicators that a variable is more important than others.
2. **Risk ratio:** Ratio between mean of the target variable for categories of a feature and global mean of the target variable. If this ratio is greater than 1, the category is more likely to churn, and if the ratio is lower than 1, the category is less likely to churn. It expresses the feature importance in relative terms.
**Functions and methods:**
* `df.groupby('x').y.agg([mean()])` - returns a dataframe with mean of y series grouped by x series
* `display(x)` displays an output in the cell of a jupyter notebook.
The entire code of this project is available in [this jupyter notebook](https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/03-classification/notebook.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
Mutual information is a concept from information theory, which measures how much we can learn about one variable if we know the value of another. In this project, we can think of this as how much do we learn about churn if we have the information from a particular feature. So, it is a measure of the importance of a categorical variable.
**Classes, functions, and methods:**
* `mutual_info_score(x, y)` - Scikit-Learn class for calculating the mutual information between the x target variable and y feature.
* `df[x].apply(y)` - apply a y function to the x series of the df dataframe.
* ` df.sort_values(ascending=False).to_frame(name='x')` - sort values in an ascending order and called the column as x.
The entire code of this project is available in [this jupyter notebook](https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/03-classification/notebook.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
**Correlation coefficient** measures the degree of dependency between two variables. This value is negative if one variable grows while the other decreases, and it is positive if both variables increase. Depending on its size, the dependency between both variables could be low, moderate, or strong. It allows measuring the importance of numerical variables.
If `r` is correlation coefficient, then the correlation between two variables is:
- LOW when `r` is between [0, -0.2) or [0, 0.2)
- MEDIUM when `r` is between [-0.2, -0.5) or [0.2, 0.5)
- STRONG when `r` is between [-0.5, -1.0] or [0.5, 1.0]
Positive Correlation vs. Negative Correlation
* When `r` is positive, an increase in x will increase y.
* When `r` is negative, an increase in x will decrease y.
* When `r` is 0, a change in x does not affect y.
**Functions and methods:**
* `df[x].corrwith(y)` - returns the correlation between x and y series. This is a function from pandas.
The entire code of this project is available in [this jupyter notebook](https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/03-classification/notebook.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
One-Hot Encoding allows encoding categorical variables in numerical ones. This method represents each category of a variable as one column, and a 1 is assigned if the value belongs to the category or 0 otherwise.
**Classes, functions, and methods:**
* `df[x].to_dict(orient='records')` - convert x series to dictionaries, oriented by rows.
* `DictVectorizer().fit_transform(x)` - Scikit-Learn class for one-hot encoding by converting x dictionaries into a sparse matrix. It does not affect the numerical variables.
* `DictVectorizer().get_feature_names()` - return the names of the columns in the sparse matrix.
The entire code of this project is available in [this jupyter notebook](https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/03-classification/notebook.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
In general, supervised models can be represented with this formula:
$\large g\left(x_{i}\right) = y_{i}$
Depending on what is the type of target variable, the supervised task can be regression or classification (binary or multiclass). Binary classification tasks can have negative (0) or positive (1) target values. The output of these models is the probability of $x_i$ belonging to the positive class. Logistic regression is similar to linear regression because both models take into account the bias term and weighted sum of features. The difference between these models is that the output of linear regression is a real number, while logistic regression outputs a value between zero and one, applying the sigmoid function to the linear regression formula.$\large g\left(x_{i}\right) = Sigmoid\left(w_{0} + w_{1}x_{1} + w_{2}x_{2} + ... + w_{n}x_{n}\right)$
$\large Sigmoid\left(z\right)=\frac{1}{1 + exp\left( -z \right)}$
In this way, the sigmoid function allows transforming a score into a probability. The entire code of this project is available in [this jupyter notebook](https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/03-classification/notebook.ipynb).| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
This video was about training a logistic regression model with Scikit-Learn, applying it to the validation dataset, and calculating its accuracy.
**Classes, functions, and methods:**
* `LogisticRegression().fit(x)` - Scikit-Learn class for training the logistic regression model.
* `LogisticRegression().coef_[0]` - return the coefficients or weights of the LR model
* `LogisticRegression().intercept_[0]` - return the bias or intercept of the LR model
* `LogisticRegression().predict[x]` - make predictions on the x dataset
* `LogisticRegression().predict_proba[x]` - make predictions on the x dataset by returning two columns with their probabilities for the two categories - soft predictions
The entire code of this project is available in [this jupyter notebook](https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/03-classification/notebook.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
This video was about the interpretation of coefficients, and training a model with fewer features.
In the formula of the logistic regression model, only one of the one-hot encoded categories is multiplied by 1, and the other by 0. In this way, we only consider the appropriate category for each categorical feature.
**Classes, functions, and methods:**
* `zip(x,y)` - returns a new list with elements from x joined with their corresponding elements on y
The entire code of this project is available in [this jupyter notebook](https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/03-classification/notebook.ipynb).
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
We trained the logistic regression model with the full training dataset (training + validation), considering numerical and categorical features. Thus, predictions were made on the test dataset, and we evaluated the model using the accuracy metric.
In this case, the predictions of validation and test were similar, which means that the model is working well.
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-3-machine-learning-for-classification)
## Notes
In this session, we worked on a project to predict churning in customers from a company. We learned the feature importance of numerical and categorical variables, including risk ratio, mutual information, and correlation coefficient. Also, we understood one-hot encoding and implemented logistic regression with Scikit-Learn.
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
| \n", " | customerID | \n", "gender | \n", "SeniorCitizen | \n", "Partner | \n", "Dependents | \n", "tenure | \n", "PhoneService | \n", "MultipleLines | \n", "InternetService | \n", "OnlineSecurity | \n", "... | \n", "DeviceProtection | \n", "TechSupport | \n", "StreamingTV | \n", "StreamingMovies | \n", "Contract | \n", "PaperlessBilling | \n", "PaymentMethod | \n", "MonthlyCharges | \n", "TotalCharges | \n", "Churn | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "7590-VHVEG | \n", "Female | \n", "0 | \n", "Yes | \n", "No | \n", "1 | \n", "No | \n", "No phone service | \n", "DSL | \n", "No | \n", "... | \n", "No | \n", "No | \n", "No | \n", "No | \n", "Month-to-month | \n", "Yes | \n", "Electronic check | \n", "29.85 | \n", "29.85 | \n", "No | \n", "
| 1 | \n", "5575-GNVDE | \n", "Male | \n", "0 | \n", "No | \n", "No | \n", "34 | \n", "Yes | \n", "No | \n", "DSL | \n", "Yes | \n", "... | \n", "Yes | \n", "No | \n", "No | \n", "No | \n", "One year | \n", "No | \n", "Mailed check | \n", "56.95 | \n", "1889.5 | \n", "No | \n", "
| 2 | \n", "3668-QPYBK | \n", "Male | \n", "0 | \n", "No | \n", "No | \n", "2 | \n", "Yes | \n", "No | \n", "DSL | \n", "Yes | \n", "... | \n", "No | \n", "No | \n", "No | \n", "No | \n", "Month-to-month | \n", "Yes | \n", "Mailed check | \n", "53.85 | \n", "108.15 | \n", "Yes | \n", "
| 3 | \n", "7795-CFOCW | \n", "Male | \n", "0 | \n", "No | \n", "No | \n", "45 | \n", "No | \n", "No phone service | \n", "DSL | \n", "Yes | \n", "... | \n", "Yes | \n", "Yes | \n", "No | \n", "No | \n", "One year | \n", "No | \n", "Bank transfer (automatic) | \n", "42.30 | \n", "1840.75 | \n", "No | \n", "
| 4 | \n", "9237-HQITU | \n", "Female | \n", "0 | \n", "No | \n", "No | \n", "2 | \n", "Yes | \n", "No | \n", "Fiber optic | \n", "No | \n", "... | \n", "No | \n", "No | \n", "No | \n", "No | \n", "Month-to-month | \n", "Yes | \n", "Electronic check | \n", "70.70 | \n", "151.65 | \n", "Yes | \n", "
5 rows × 21 columns
\n", "| \n", " | 0 | \n", "1 | \n", "2 | \n", "3 | \n", "4 | \n", "
|---|---|---|---|---|---|
| customerid | \n", "7590-vhveg | \n", "5575-gnvde | \n", "3668-qpybk | \n", "7795-cfocw | \n", "9237-hqitu | \n", "
| gender | \n", "female | \n", "male | \n", "male | \n", "male | \n", "female | \n", "
| seniorcitizen | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| partner | \n", "yes | \n", "no | \n", "no | \n", "no | \n", "no | \n", "
| dependents | \n", "no | \n", "no | \n", "no | \n", "no | \n", "no | \n", "
| tenure | \n", "1 | \n", "34 | \n", "2 | \n", "45 | \n", "2 | \n", "
| phoneservice | \n", "no | \n", "yes | \n", "yes | \n", "no | \n", "yes | \n", "
| multiplelines | \n", "no_phone_service | \n", "no | \n", "no | \n", "no_phone_service | \n", "no | \n", "
| internetservice | \n", "dsl | \n", "dsl | \n", "dsl | \n", "dsl | \n", "fiber_optic | \n", "
| onlinesecurity | \n", "no | \n", "yes | \n", "yes | \n", "yes | \n", "no | \n", "
| onlinebackup | \n", "yes | \n", "no | \n", "yes | \n", "no | \n", "no | \n", "
| deviceprotection | \n", "no | \n", "yes | \n", "no | \n", "yes | \n", "no | \n", "
| techsupport | \n", "no | \n", "no | \n", "no | \n", "yes | \n", "no | \n", "
| streamingtv | \n", "no | \n", "no | \n", "no | \n", "no | \n", "no | \n", "
| streamingmovies | \n", "no | \n", "no | \n", "no | \n", "no | \n", "no | \n", "
| contract | \n", "month-to-month | \n", "one_year | \n", "month-to-month | \n", "one_year | \n", "month-to-month | \n", "
| paperlessbilling | \n", "yes | \n", "no | \n", "yes | \n", "no | \n", "yes | \n", "
| paymentmethod | \n", "electronic_check | \n", "mailed_check | \n", "mailed_check | \n", "bank_transfer_(automatic) | \n", "electronic_check | \n", "
| monthlycharges | \n", "29.85 | \n", "56.95 | \n", "53.85 | \n", "42.3 | \n", "70.7 | \n", "
| totalcharges | \n", "29.85 | \n", "1889.5 | \n", "108.15 | \n", "1840.75 | \n", "151.65 | \n", "
| churn | \n", "no | \n", "no | \n", "yes | \n", "no | \n", "yes | \n", "
| \n", " | customerid | \n", "gender | \n", "seniorcitizen | \n", "partner | \n", "dependents | \n", "tenure | \n", "phoneservice | \n", "multiplelines | \n", "internetservice | \n", "onlinesecurity | \n", "... | \n", "deviceprotection | \n", "techsupport | \n", "streamingtv | \n", "streamingmovies | \n", "contract | \n", "paperlessbilling | \n", "paymentmethod | \n", "monthlycharges | \n", "totalcharges | \n", "churn | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "5442-pptjy | \n", "male | \n", "0 | \n", "yes | \n", "yes | \n", "12 | \n", "yes | \n", "no | \n", "no | \n", "no_internet_service | \n", "... | \n", "no_internet_service | \n", "no_internet_service | \n", "no_internet_service | \n", "no_internet_service | \n", "two_year | \n", "no | \n", "mailed_check | \n", "19.70 | \n", "258.35 | \n", "0 | \n", "
| 1 | \n", "6261-rcvns | \n", "female | \n", "0 | \n", "no | \n", "no | \n", "42 | \n", "yes | \n", "no | \n", "dsl | \n", "yes | \n", "... | \n", "yes | \n", "yes | \n", "no | \n", "yes | \n", "one_year | \n", "no | \n", "credit_card_(automatic) | \n", "73.90 | \n", "3160.55 | \n", "1 | \n", "
| 2 | \n", "2176-osjuv | \n", "male | \n", "0 | \n", "yes | \n", "no | \n", "71 | \n", "yes | \n", "yes | \n", "dsl | \n", "yes | \n", "... | \n", "no | \n", "yes | \n", "no | \n", "no | \n", "two_year | \n", "no | \n", "bank_transfer_(automatic) | \n", "65.15 | \n", "4681.75 | \n", "0 | \n", "
| 3 | \n", "6161-erdgd | \n", "male | \n", "0 | \n", "yes | \n", "yes | \n", "71 | \n", "yes | \n", "yes | \n", "dsl | \n", "yes | \n", "... | \n", "yes | \n", "yes | \n", "yes | \n", "yes | \n", "one_year | \n", "no | \n", "electronic_check | \n", "85.45 | \n", "6300.85 | \n", "0 | \n", "
| 4 | \n", "2364-ufrom | \n", "male | \n", "0 | \n", "no | \n", "no | \n", "30 | \n", "yes | \n", "no | \n", "dsl | \n", "yes | \n", "... | \n", "no | \n", "yes | \n", "yes | \n", "no | \n", "one_year | \n", "no | \n", "electronic_check | \n", "70.40 | \n", "2044.75 | \n", "0 | \n", "
5 rows × 21 columns
\n", "| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| gender | \n", "\n", " | \n", " | \n", " | \n", " |
| female | \n", "0.276824 | \n", "2796 | \n", "0.006856 | \n", "1.025396 | \n", "
| male | \n", "0.263214 | \n", "2838 | \n", "-0.006755 | \n", "0.974980 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| seniorcitizen | \n", "\n", " | \n", " | \n", " | \n", " |
| 0 | \n", "0.242270 | \n", "4722 | \n", "-0.027698 | \n", "0.897403 | \n", "
| 1 | \n", "0.413377 | \n", "912 | \n", "0.143409 | \n", "1.531208 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| partner | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.329809 | \n", "2932 | \n", "0.059841 | \n", "1.221659 | \n", "
| yes | \n", "0.205033 | \n", "2702 | \n", "-0.064935 | \n", "0.759472 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| dependents | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.313760 | \n", "3968 | \n", "0.043792 | \n", "1.162212 | \n", "
| yes | \n", "0.165666 | \n", "1666 | \n", "-0.104302 | \n", "0.613651 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| phoneservice | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.241316 | \n", "547 | \n", "-0.028652 | \n", "0.893870 | \n", "
| yes | \n", "0.273049 | \n", "5087 | \n", "0.003081 | \n", "1.011412 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| multiplelines | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.257407 | \n", "2700 | \n", "-0.012561 | \n", "0.953474 | \n", "
| no_phone_service | \n", "0.241316 | \n", "547 | \n", "-0.028652 | \n", "0.893870 | \n", "
| yes | \n", "0.290742 | \n", "2387 | \n", "0.020773 | \n", "1.076948 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| internetservice | \n", "\n", " | \n", " | \n", " | \n", " |
| dsl | \n", "0.192347 | \n", "1934 | \n", "-0.077621 | \n", "0.712482 | \n", "
| fiber_optic | \n", "0.425171 | \n", "2479 | \n", "0.155203 | \n", "1.574895 | \n", "
| no | \n", "0.077805 | \n", "1221 | \n", "-0.192163 | \n", "0.288201 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| onlinesecurity | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.420921 | \n", "2801 | \n", "0.150953 | \n", "1.559152 | \n", "
| no_internet_service | \n", "0.077805 | \n", "1221 | \n", "-0.192163 | \n", "0.288201 | \n", "
| yes | \n", "0.153226 | \n", "1612 | \n", "-0.116742 | \n", "0.567570 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| onlinebackup | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.404323 | \n", "2498 | \n", "0.134355 | \n", "1.497672 | \n", "
| no_internet_service | \n", "0.077805 | \n", "1221 | \n", "-0.192163 | \n", "0.288201 | \n", "
| yes | \n", "0.217232 | \n", "1915 | \n", "-0.052736 | \n", "0.804660 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| deviceprotection | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.395875 | \n", "2473 | \n", "0.125907 | \n", "1.466379 | \n", "
| no_internet_service | \n", "0.077805 | \n", "1221 | \n", "-0.192163 | \n", "0.288201 | \n", "
| yes | \n", "0.230412 | \n", "1940 | \n", "-0.039556 | \n", "0.853480 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| techsupport | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.418914 | \n", "2781 | \n", "0.148946 | \n", "1.551717 | \n", "
| no_internet_service | \n", "0.077805 | \n", "1221 | \n", "-0.192163 | \n", "0.288201 | \n", "
| yes | \n", "0.159926 | \n", "1632 | \n", "-0.110042 | \n", "0.592390 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| streamingtv | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.342832 | \n", "2246 | \n", "0.072864 | \n", "1.269897 | \n", "
| no_internet_service | \n", "0.077805 | \n", "1221 | \n", "-0.192163 | \n", "0.288201 | \n", "
| yes | \n", "0.302723 | \n", "2167 | \n", "0.032755 | \n", "1.121328 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| streamingmovies | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.338906 | \n", "2213 | \n", "0.068938 | \n", "1.255358 | \n", "
| no_internet_service | \n", "0.077805 | \n", "1221 | \n", "-0.192163 | \n", "0.288201 | \n", "
| yes | \n", "0.307273 | \n", "2200 | \n", "0.037305 | \n", "1.138182 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| contract | \n", "\n", " | \n", " | \n", " | \n", " |
| month-to-month | \n", "0.431701 | \n", "3104 | \n", "0.161733 | \n", "1.599082 | \n", "
| one_year | \n", "0.120573 | \n", "1186 | \n", "-0.149395 | \n", "0.446621 | \n", "
| two_year | \n", "0.028274 | \n", "1344 | \n", "-0.241694 | \n", "0.104730 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| paperlessbilling | \n", "\n", " | \n", " | \n", " | \n", " |
| no | \n", "0.172071 | \n", "2313 | \n", "-0.097897 | \n", "0.637375 | \n", "
| yes | \n", "0.338151 | \n", "3321 | \n", "0.068183 | \n", "1.252560 | \n", "
| \n", " | mean | \n", "count | \n", "diff | \n", "risk | \n", "
|---|---|---|---|---|
| paymentmethod | \n", "\n", " | \n", " | \n", " | \n", " |
| bank_transfer_(automatic) | \n", "0.168171 | \n", "1219 | \n", "-0.101797 | \n", "0.622928 | \n", "
| credit_card_(automatic) | \n", "0.164339 | \n", "1217 | \n", "-0.105630 | \n", "0.608733 | \n", "
| electronic_check | \n", "0.455890 | \n", "1893 | \n", "0.185922 | \n", "1.688682 | \n", "
| mailed_check | \n", "0.193870 | \n", "1305 | \n", "-0.076098 | \n", "0.718121 | \n", "
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-4-evaluation-metrics-for-classification)
## Notes
The fourth week of Machine Learning Zoomcamp is about different metrics to evaluate a binary classifier. These measures include accuracy, confusion table, precision, recall, ROC curves(TPR, FRP, random model, and ideal model), AUROC, and cross-validation.
For this project, we used a [Kaggle dataset](https://www.kaggle.com/blastchar/telco-customer-churn) about churn prediction.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-4-evaluation-metrics-for-classification)
## Notes
**Accuracy** measures the fraction of correct predictions. Specifically, it is the number of correct predictions divided by the total number of predictions.
We can change the **decision threshold**, it should not be always 0.5. But, in this particular problem, the best decision cutoff, associated with the hightest accuracy (80%), was indeed 0.5.
Note that if we build a **dummy model** in which the decision cutoff is 1, so the algorithm predicts that no clients will churn, the accuracy would be 73%. Thus, we can see that the improvement of the original model with respect to the dummy model is not as high as we would expect.
Therefore, in this problem accuracy can not tell us how good is the model because the dataset is **unbalanced**, which means that there are more instances from one category than the other. This is also known as **class imbalance**.
**Classes and methods:**
* `np.linspace(x,y,z)` - returns a numpy array starting at `x` until `y` with `z` evenly spaced samples
* `Counter(x)` - collection class that counts the number of instances that satisfy the `x` condition
* `accuracy_score(x, y)` - sklearn.metrics class for calculating the accuracy of a model, given a predicted `x` dataset and a target `y` dataset.
The entire code of this project is available in [this jupyter notebook](notebook.ipynb).
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-4-evaluation-metrics-for-classification)
## Notes
Confusion table is a way of measuring different types of errors and correct decisions that binary classifiers can make. Considering this information, it is possible to evaluate the quality of the model by different strategies.
When comes to a prediction of an LR model, each falls into one of four different categories:
* Prediction is that the customer WILL churn. This is known as the **Positive class**
* And Customer actually churned - Known as a **True Positive (TP)**
* But Customer actually did not churn - Known as a **False Positive (FP)**
* Prediction is that the customer WILL NOT churn' - This is known as the **Negative class**
* Customer did not churn - **True Negative (TN)**
* Customer churned - **False Negative (FN)**
`Confusion Table` is a way to summarize the above results in a tabular format, as shown below:
| Predictions | |||
|---|---|---|---|
| Negative | Positive | ||
| Actual | Negative | TN | FP |
| Positive | FN | TP | |
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-4-evaluation-metrics-for-classification)
## Notes
**Precision** tell us the fraction of positive predictions that are correct. It takes into account only the **positive class** (TP and FP - second column of the confusion matrix), as is stated in the following formula:
$$P = \cfrac{TP}{TP + FP}$$
**Recall** measures the fraction of correctly identified positive instances. It considers parts of the **positive and negative classes** (TP and FN - second row of confusion table). The formula of this metric is presented below:
$$R = \cfrac{TP}{TP + FN}$$
In this problem, the precision and recall values were 67% and 54% respectively. So, these measures reflect some errors of our model that accuracy did not notice due to the **class imbalance**.
**MNEMONICS:**
- Precision : From the `pre`dicted positives, how many we predicted right. See how the word `pre`cision is similar to the word `pre`diction?
- Recall : From the `real` positives, how many we predicted right. See how the word `re`c`al`l is similar to the word `real`?
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-4-evaluation-metrics-for-classification)
## Notes
ROC stands for Receiver Operating Characteristic, and this idea was applied during the Second World War for evaluating the strength of radio detectors. This measure considers **False Positive Rate** (FPR) and **True Postive Rate** (TPR), which are derived from the values of the confusion matrix.
**FPR** is the fraction of false positives (FP) divided by the total number of negatives (FP and TN - the first row of confusion matrix), and we want to `minimize` it. The formula of FPR is the following:
$FPR = \large \frac{FP}{TN + FP}$
In the other hand, **TPR** or **Recall** is the fraction of true positives (TP) divided by the total number of positives (FN and TP - second row of confusion table), and we want to `maximize` this metric. The formula of this measure is presented below:$TPR =\large \frac{TP}{TP + FN}$
 ROC curves consider Recall and FPR under all the possible thresholds. If the threshold is 0 or 1, the FPR and Recall scores are the opposite of the threshold (1 and 0 respectively), but they have different meanings, as we explained before. We need to compare the ROC curves against a point of reference to evaluate its performance, so the corresponding curves of random and ideal models are required. It is possible to plot the ROC curves with FPR and Recall scores vs thresholds, or FPR vs Recall. **Classes and methods:** * `np.repeat([x,y], [z,w])` - returns a numpy array with a z number of x values first, and then a w number of y values. * `roc_curve(x, y)` - sklearn.metrics class for calculating the false positive rates, true positive rates, and thresholds, given a target x dataset and a predicted y dataset. The entire code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/course-zoomcamp/04-evaluation/notebook.ipynb). Add notes from the video (PRs are welcome)| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-4-evaluation-metrics-for-classification)
## Notes
The Area under the ROC curves can tell us how good is our model with a single value. The AUROC of a random model is 0.5, while for an ideal one is 1.
In other words, AUC can be interpreted as the probability that a randomly selected positive example has a greater score than a randomly selected negative example.
**Classes and methods:**
* `auc(x, y)` - sklearn.metrics class for calculating area under the curve of the x and y datasets. For ROC curves x would be false positive rate, and y true positive rate.
* `roc_auc_score(x, y)` - sklearn.metrics class for calculating area under the ROC curves of the x false positive rate and y true positive rate datasets.
* `randint(x, y, size=z)` - np.random class for generating random integers from the “discrete uniform” distribution; from `x` (inclusive) to `y` (exclusive) of size `z`.
The entire code of this project is available in [this jupyter notebook](notebook.ipynb).
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-4-evaluation-metrics-for-classification)
### Extra resources
In the lesson we talked about iterators and generators in Python. You can read more about them here:
- https://anandology.com/python-practice-book/iterators.html
- https://www.google.com/search?q=python+iterators+and+generators
## Notes
**Cross-validations** refers to evaluating the same model on different subsets of a dataset, getting the average prediction, and spread within predictions. This method is applied in the **parameter tuning** step, which is the process of selecting the best parameter.
In this algorithm, the full training dataset is divided into **k partitions**, we train the model in k-1 partitions of this dataset and evaluate it on the remaining subset. Then, we end up evaluating the model in all the k folds, and we calculate the average evaluation metric for all the folds.
In general, if the dataset is large, we should use the hold-out validation dataset strategy. In the other hand, if the dataset is small or we want to know the standard deviation of the model across different folds, we can use the cross-validation approach.
**Libraries, classes and methods:**
- `Kfold(k, s, x)` - sklearn.model_selection class for calculating the cross validation with k folds, s boolean attribute for shuffle decision, and an x random state
- `Kfold.split(x)` - sklearn.Kfold method for splitting the x dataset with the attributes established in the Kfold's object construction.
- `for i in tqdm()` - library for showing the progress of each i iteration in a for loop.
The code of this project is available in [this jupyter notebook](notebook.ipynb).
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-4-evaluation-metrics-for-classification)
## Notes
General definitions:
* **Metric:** A single number that describes the performance of a model
* **Accuracy:** Fraction of correct answers; sometimes misleading
* Precision and recall are less misleading when we have class imbalance
* **ROC Curve:** A way to evaluate the performance at all thresholds; okay to use with imbalance
* **K-Fold CV:** More reliable estimate for performance (mean + std)
In brief, this weeks was about different metrics to evaluate a binary classifier. These measures included accuracy, confusion table, precision, recall, ROC curves(TPR, FPR, random model, and ideal model), and AUROC. Also, we talked about a different way to estimate the performance of the model and make the parameter tuning with cross-validation.
The code of this project is available in [this jupyter notebook](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/course-zoomcamp/04-evaluation/notebook.ipynb).
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
| \n", " | threshold | \n", "tp | \n", "fp | \n", "fn | \n", "tn | \n", "tpr | \n", "fpr | \n", "
|---|---|---|---|---|---|---|---|
| 0 | \n", "0.0 | \n", "386 | \n", "1023 | \n", "0 | \n", "0 | \n", "1.000000 | \n", "1.000000 | \n", "
| 10 | \n", "0.1 | \n", "386 | \n", "882 | \n", "0 | \n", "141 | \n", "1.000000 | \n", "0.862170 | \n", "
| 20 | \n", "0.2 | \n", "386 | \n", "741 | \n", "0 | \n", "282 | \n", "1.000000 | \n", "0.724340 | \n", "
| 30 | \n", "0.3 | \n", "386 | \n", "600 | \n", "0 | \n", "423 | \n", "1.000000 | \n", "0.586510 | \n", "
| 40 | \n", "0.4 | \n", "386 | \n", "459 | \n", "0 | \n", "564 | \n", "1.000000 | \n", "0.448680 | \n", "
| 50 | \n", "0.5 | \n", "386 | \n", "319 | \n", "0 | \n", "704 | \n", "1.000000 | \n", "0.311828 | \n", "
| 60 | \n", "0.6 | \n", "386 | \n", "178 | \n", "0 | \n", "845 | \n", "1.000000 | \n", "0.173998 | \n", "
| 70 | \n", "0.7 | \n", "386 | \n", "37 | \n", "0 | \n", "986 | \n", "1.000000 | \n", "0.036168 | \n", "
| 80 | \n", "0.8 | \n", "282 | \n", "0 | \n", "104 | \n", "1023 | \n", "0.730570 | \n", "0.000000 | \n", "
| 90 | \n", "0.9 | \n", "141 | \n", "0 | \n", "245 | \n", "1023 | \n", "0.365285 | \n", "0.000000 | \n", "
| 100 | \n", "1.0 | \n", "1 | \n", "0 | \n", "385 | \n", "1023 | \n", "0.002591 | \n", "0.000000 | \n", "
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-5-model-deployment)
## Notes
In this session, we talked about the earlier model we made in chapter 3 for churn prediction. | ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-5-model-deployment)
## Notes
**In this session we'll cover the idea "How to use the model in future without training and evaluating the code"**
- To save the model we made before there is an option using the pickle library:
- First install the library with the command ```pip install pickle-mixin``` if you don't have it.
- After training the model and making it ready for the prediction process, use this code to save the model for later.
- ```python
import pickle
with open('model.bin', 'wb') as f_out: # 'wb' means write-binary
pickle.dump((dict_vectorizer, model), f_out)
```
- In the code above we'll make a binary file named model.bin, and write the dict_vectorizer for one hot encoding and the model as array in it. (We will save it as binary in case it wouldn't be readable by humans)
- To be able to use the model in future without running the code, We need to open the binary file we saved before.
- ```python
import pickle
with open('mode.bin', 'rb') as f_in: # very important to use 'rb' here, it means read-binary
dict_vectorizer, model = pickle.load(f_in)
## Note: never open a binary file you do not trust the source!
```
- With unpacking the model and the dict_vectorizer, We're able to predict again for new input values without training a new model by re-running the code.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-5-model-deployment)
## Links
* [0.0.0.0 vs localhost](https://stackoverflow.com/a/20778887/861423)
* [Top-level script enviroment](https://docs.python.org/3.9/library/__main__.html)
* [route decorator](https://flask.palletsprojects.com/en/2.2.x/api/#flask.Flask.route)
## Notes
In this session we talked about what is a web service and how to create a simple web service.
- What is actually a web service
- A web service is a method used to communicate between electronic devices.
- There are some methods in web services that we can use to satisfy our problems. Here below we would list some.
- **GET:** GET is a method used to retrieve files, For example when we are searching for a cat image in google we are actually requesting cat images with GET method.
- **POST:** POST is the second common method used in web services. It enables sending data to a server to create or update a resource. For example in a sign up process, when we are submiting our name, username, passwords, etc we are posting our data to a server that is using the web service. (Note that there is no specification where the data goes)
- **PUT:** PUT is same as POST but we are specifying where the data is going to.
- **DELETE:** DELETE is a method that is used to request to delete some data from the server.
- For more information just google the HTTP methods, You'll find useful information about this.
- To create a simple web service, there are plenty libraries available in every language. Here we would like to introduce Flask library in python.
- If you haven't installed the library just try installing it with the code ```pip install Flask```
- To create a simple web service just run the code below:
- ```python
from flask import Flask
app = Flask('ping') # give an identity to your web service
@app.route('/ping', methods=['GET']) # use decorator to add Flask's functionality to our function
def ping():
return 'PONG'
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0', port=9696) # run the code in local machine with the debugging mode true and port 9696
```
- With the code above we made a simple web server and created a route named ping that would send pong string.
- To test it, just use the `cURL` command in a new terminal by typing ```curl http://localhost:9696/ping```, or simply open your browser and search ```localhost:9696/ping```, You'll see that the 'PONG' string is received. Congrats You've made a simple web server 🥳.
- To use our web server to predict new values we must modify it. See how in the next session.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-5-model-deployment)
## Notes
In this session, we talked about implementing the functionality of prediction to our churn web service and how to make it usable in development environment.
- To make the web service predict the churn value for each customer we must modify the code in session 3 with the code we had in previous chapters. Below we can see how the code works in order to predict the churn value.
- In order to predict we need to first load the previous saved model and use a prediction function in a special route.
- To load the previous saved model we use the code below:
- ```python
import pickle
with open('churn-model.bin', 'rb') as f_in:
dv, model = pickle.load(f_in)
```
- As we had earlier to predict a value for a customer we need a function like below:
- ```python
def predict_single(customer, dv, model):
X = dv.transform([customer]) ## apply the one-hot encoding feature to the customer data
y_pred = model.predict_proba(X)[:, 1]
return y_pred[0]
```
- Then at last we make the final function used for creating the web service.
- ```python
@app.route('/predict', methods=['POST']) ## in order to send the customer information we need to post its data.
def predict():
customer = request.get_json() ## web services work best with json frame, So after the user post its data in json format we need to access the body of json.
prediction = predict_single(customer, dv, model)
churn = prediction >= 0.5
result = {
'churn_probability': float(prediction), ## we need to cast numpy float type to python native float type
'churn': bool(churn), ## same as the line above, casting the value using bool method
}
return jsonify(result) ## send back the data in json format to the user
```
- The whole code above is available in this link: [churn_serving.py](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-05-deployment/churn_serving.py)
- Finally, run your code. To see the result we can't use a simple request in web browser, because we are expecting a `POST` request in our app. We can run the code below to **post** customer data as `json` and see the response
- ```python
## a new customer informations
customer = {
'customerid': '8879-zkjof',
'gender': 'female',
'seniorcitizen': 0,
'partner': 'no',
'dependents': 'no',
'tenure': 41,
'phoneservice': 'yes',
'multiplelines': 'no',
'internetservice': 'dsl',
'onlinesecurity': 'yes',
'onlinebackup': 'no',
'deviceprotection': 'yes',
'techsupport': 'yes',
'streamingtv': 'yes',
'streamingmovies': 'yes',
'contract': 'one_year',
'paperlessbilling': 'yes',
'paymentmethod': 'bank_transfer_(automatic)',
'monthlycharges': 79.85,
'totalcharges': 3320.75
}
import requests ## to use the POST method we use a library named requests
url = 'http://localhost:9696/predict' ## this is the route we made for prediction
response = requests.post(url, json=customer) ## post the customer information in json format
result = response.json() ## get the server response
print(result)
```
- Until here we saw how we made a simple web server that predicts the churn value for every user. When you run your app you will see a warning that it is not a WGSI server and not suitable for production environmnets. To fix this issue and run this as a production server there are plenty of ways available.
- One way to create a WSGI server is to use gunicorn. To install it use the command ```pip install gunicorn```, And to run the WGSI server you can simply run it with the command ```gunicorn --bind 0.0.0.0:9696 churn:app```. Note that in __churn:app__ the name churn is the name we set for the file containing the code ```app = Flask('churn')```(for example: churn.py), You may need to change it to whatever you named your Flask app file.
- Windows users may not be able to use gunicorn library because windows system do not support some dependecies of the library. So to be able to run this on a windows machine, there is an alternative library waitress and to install it, just use the command ```pip install waitress```.
- to run the waitress wgsi server use the command ```waitress-serve --listen=0.0.0.0:9696 churn:app```.
- To test it, you can run the code above and the result will be the same.
- So until here you were able to make a production server that predicts the churn value for new customers. In the next session, we can see how to solve library version conflicts in each machine and manage dependencies for production environments.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-5-model-deployment)
## Notes
In this session we're going to make virtual environments for our project. So Let's start this session by getting to know what is a virtual environment and how to make it.
- Every time we're running a file from a directory we're using the executive files from a global directory. When we install python on our machine the executable files that are able to run our codes will go to somewhere like _/home/username/python/bin/_. The pip command, for example, may go to _/home/username/python/bin/pip_.
- Sometimes the versions of libraries conflict (the project may not run or get into massive errors). For example, we have an old project that uses sklearn library with the version of 0.24.1 and now we want to run it using sklearn version 1.0.0. We may get into errors because of the version conflict.
- To solve the conflict we can make virtual environments. A virtual environment is an operation that can separate the libraries installed in our system and the libraries with specified version we want our project to run with. There are a lot of ways to create virtual environments. One way we are going to use is a library named pipenv.
- pipenv is a library that can create a virutal environment. To install this library just use the classic method ```pip install pipenv```.
- After installing pipenv we must install the libraries we want for our project in the new virtual environment. It's really easy, Just use the command pipenv instead of pip. ```pipenv install numpy scikit-learn==0.24.1 flask```. With this command we installed the libraries we want for our project.
- Note that using the pipenv command we made two files named _Pipfile_ and _Pipfile.lock_. If we look at these files closely we can see that in Pipfile the libraries we installed are named. If we specified the library name, it's also specified in Pipfile.
- In _Pipfile.lock_ we can see that each library with its installed version is named and a hash file is there to reproduce if we move the environment to another machine.
- If we want to run the project in another machine, we can easily install the libraries we want with the command ```pipenv install```. This command will look into _Pipfile_ and _Pipfile.lock_ to install the libraries with specified version.
- After installing the required libraries we can run the project in the virtual environment with ```pipenv shell``` command. This will go to the virtual environment's shell and then any command we execute will use the virtual environment's libraries. Typing `gunicorn --bind localhost:9696 predict:app` can run the web service. However, if we prefer to use only one command to run the application in our environment, we just have to type: `pipenv run gunicorn --bind localhost:9696 predict:app`.
- Installing and using the libraries such as gunicorn is the same as the last session.
- Until here we made a virtual environment for our libraries with a required specified version. To seperate this environment more, such as making gunicorn be able to run in windows machines we need another way. The other way is using Docker. Docker allows us to seperate everything more than before and make any project able to run on any machine that supports Docker smoothly.
- In the next session we'll go in detail of how Docker works and how to use it.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-5-model-deployment)
## Installing Docker
To isolate more our project file from our system machine, there is an option named Docker. With Docker you are able to pack all your project in the system that you want and run it in any other machine system. For example if you want Ubuntu 20.4 you can have it in a mac or windows machine or other operating systems. | ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-5-model-deployment)
## Links
* [Creating an account on AWS](https://mlbookcamp.com/article/aws)
## Notes
### Deployment to the Cloud using AWS Elastic Beanstalk
AWS is the most popular cloud provider, and Elastic Beanstalk offers a simple way to deploy applications with just a few commands. It's well-suited for deploying containerized applications, like our churn prediction service packaged in a Docker container.
In this session, we explored how to deploy a machine learning model to the cloud using AWS Elastic Beanstalk, leveraging Docker to containerize the application for easy deployment and scalability.
#### Architecture Overview
1. The churn prediction service is packaged into a Docker container.
2. This container is deployed to AWS Elastic Beanstalk.
3. A marketing service sends requests to the Elastic Beanstalk environment.
4. Elastic Beanstalk forwards these requests to the Docker container.
5. The container processes the request and sends the response back to Elastic Beanstalk.
6. Elastic Beanstalk relays the response to the requesting service.
#### Scalability with Elastic Beanstalk (EB)
Elastic Beanstalk automatically scales the application based on traffic. If the churn prediction service receives a high volume of requests, EB automatically adds more instances of the service to handle the load without interruption (horizontal scaling). Similarly, when traffic decreases, EB scales down the number of instances to optimize resource utilization.
#### Deployment Steps
1. **Install the AWS EB CLI:**
```bash
pipenv install awsebcli --dev
```
Note: This installs the EB CLI as a development dependency, as it's not needed within the container itself.
2. **Initialize the EB environment:**
```bash
eb init -p docker -r eu-north-1 churn-serving
```
This command configures the EB environment with the following parameters:
* `-p docker`: Specifies the platform as Docker.
* `-r eu-north-1`: Sets the region to `eu-north-1`. You can choose a different region based on your account information.
* `churn-serving`: Defines the name of the environment.
3. **Verify the EB Configuration:**
```bash
less .elasticbeanstalk/config.yml
```
This command allows you to review the generated configuration file.
4. **Test Locally:**
```bash
eb local run --port 9696
```
This command starts the application locally using the specified port. You can use the `predict.py` script to test the service locally.
5. **Deploy to AWS:**
```bash
eb create churn-serving-env
```
This command creates the Elastic Beanstalk environment and deploys the application.
6. **Access the Deployed Service:**
After successful deployment, Elastic Beanstalk provides the endpoint URL for the service. Update the `predict-test` script with the new host address. Note that you'll no longer need the port number when accessing the deployed service.
7. **Security Considerations:**
The deployed service is publicly accessible. In a production environment, it's crucial to implement appropriate security measures to restrict access to authorized users or networks.
8. **Terminate the Service:**
```bash
eb terminate churn-serving-env
```
This command terminates the Elastic Beanstalk environment and removes all associated resources.
#### Additional Note
* Remember to configure your AWS credentials before running the EB CLI commands.
### Deployment with Heroku
As we saw how to deploy our apps in AWS Let's find it out how to deploy them in Heroku.
- First of all create your web service with flask. (example file: [churn_prediction.py](https://github.com/amindadgar/customer-churn-app/blob/main/churn_serving.py)
- Then create a file named _requirements.txt_ and pass your dependencies there. Example:
```
pickle
numpy
flask
gunicorn
```
- Create another file named _Procfile_ and add the app you want to be able to run there. Example:
```
web: gunicorn churn_serving:app
```
Note that the churn_serving name in the box above is the name of the main python file we're going to be running.
- Create your heroku profile, Go to dashboard and the Deploy tab.
- Follow the instruction to Deploy using Heroku Git.
- Great, your app is now available from global universe.
I've put my heroku app files in this repository:
https://github.com/amindadgar/customer-churn-app
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-5-model-deployment)
## Notes
In this chapter we learned these topics:
- We learned how to save the model and load it to re-use it without running the previous code.
- How to deploy the model in a web service.
- How to create a virtual environment.
- How to create a container and run our code in any operating systems.
- How to deploy our code in a public web service and access it externally from outside a local computer.
In the next chapter we will learn the algorithms such as Decision trees, Random forests and Gradient boosting as an alternative way of combining decision tress.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
LogisticRegression(solver='liblinear')In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
| \n", " | penalty | \n", "'l2' | \n", "
| \n", " | dual | \n", "False | \n", "
| \n", " | tol | \n", "0.0001 | \n", "
| \n", " | C | \n", "1.0 | \n", "
| \n", " | fit_intercept | \n", "True | \n", "
| \n", " | intercept_scaling | \n", "1 | \n", "
| \n", " | class_weight | \n", "None | \n", "
| \n", " | random_state | \n", "None | \n", "
| \n", " | solver | \n", "'liblinear' | \n", "
| \n", " | max_iter | \n", "100 | \n", "
| \n", " | multi_class | \n", "'deprecated' | \n", "
| \n", " | verbose | \n", "0 | \n", "
| \n", " | warm_start | \n", "False | \n", "
| \n", " | n_jobs | \n", "None | \n", "
| \n", " | l1_ratio | \n", "None | \n", "
LogisticRegression(solver='liblinear')In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
| \n", " | penalty | \n", "'l2' | \n", "
| \n", " | dual | \n", "False | \n", "
| \n", " | tol | \n", "0.0001 | \n", "
| \n", " | C | \n", "1.0 | \n", "
| \n", " | fit_intercept | \n", "True | \n", "
| \n", " | intercept_scaling | \n", "1 | \n", "
| \n", " | class_weight | \n", "None | \n", "
| \n", " | random_state | \n", "None | \n", "
| \n", " | solver | \n", "'liblinear' | \n", "
| \n", " | max_iter | \n", "100 | \n", "
| \n", " | multi_class | \n", "'deprecated' | \n", "
| \n", " | verbose | \n", "0 | \n", "
| \n", " | warm_start | \n", "False | \n", "
| \n", " | n_jobs | \n", "None | \n", "
| \n", " | l1_ratio | \n", "None | \n", "
Pipeline(steps=[('dictvectorizer', DictVectorizer()),\n",
" ('logisticregression', LogisticRegression(solver='liblinear'))])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. | \n", " | steps | \n", "[('dictvectorizer', ...), ('logisticregression', ...)] | \n", "
| \n", " | transform_input | \n", "None | \n", "
| \n", " | memory | \n", "None | \n", "
| \n", " | verbose | \n", "False | \n", "
| \n", " | dtype | \n", "<class 'numpy.float64'> | \n", "
| \n", " | separator | \n", "'=' | \n", "
| \n", " | sparse | \n", "True | \n", "
| \n", " | sort | \n", "True | \n", "
| \n", " | penalty | \n", "'l2' | \n", "
| \n", " | dual | \n", "False | \n", "
| \n", " | tol | \n", "0.0001 | \n", "
| \n", " | C | \n", "1.0 | \n", "
| \n", " | fit_intercept | \n", "True | \n", "
| \n", " | intercept_scaling | \n", "1 | \n", "
| \n", " | class_weight | \n", "None | \n", "
| \n", " | random_state | \n", "None | \n", "
| \n", " | solver | \n", "'liblinear' | \n", "
| \n", " | max_iter | \n", "100 | \n", "
| \n", " | multi_class | \n", "'deprecated' | \n", "
| \n", " | verbose | \n", "0 | \n", "
| \n", " | warm_start | \n", "False | \n", "
| \n", " | n_jobs | \n", "None | \n", "
| \n", " | l1_ratio | \n", "None | \n", "
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-6-decision-trees-and-ensemble-learning)
## Notes
In this session we'll learn about decision trees and ensemble learning algorithms. The questions that we try to address this week are, "What are decision trees? How are they different from ensemble algorithms? How can we implement and fine-tune these models to make binary classification predictions?"
To be specific, we'll use [credit scoring data](https://github.com/gastonstat/CreditScoring) to build a model that predicts whether a bank should lend loan to a client or not. The bank takes these decisions based on the historical record.
In the credit scoring classification problem,
- if the model returns 0, this means, the client is very likely to payback the loan and the bank will approve the loan.
- if the model returns 1, then the client is considered as a `defaulter` and the bank may not approve the loan.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-6-decision-trees-and-ensemble-learning)
## Notes
In this section we clean and prepare the [dataset](https://github.com/gastonstat/CreditScoring/raw/master/CreditScoring.csv) for the model which involves the following steps:
- Download the data from the given link.
- Reformat categorical columns (`status`, `home`, `marital`, `records`, and `job`) by mapping with appropriate values.
- Replace the maximum value of `income`, `assests`, and `debt` columns with NaNs.
- Replace the NaNs in the dataframe with `0` (*will be shown in the next lesson*).
- Extract only those rows in the column `status` who are either ok or default as value.
- Split the data in a two-step process which finally leads to the distribution of 60% train, 20% validation, and 20% test sets with random seed to `11`.
- Prepare target variable `status` by converting it from categorical to binary, where 0 represents `ok` and 1 represents `default`.
- Finally delete the target variable from the train/val/test dataframe.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-6-decision-trees-and-ensemble-learning)
## Notes
Decision Trees are powerful algorithms, capable of fitting complex datasets. The decision trees make predictions based on the bunch of *if/else* statements by splitting a node into two or more sub-nodes.
With versatility, the decision tree is also prone to overfitting. One of the reasons why this algorithm often overfits is its depth. It tends to memorize all the patterns in the train data but struggles to perform well on the unseen data (validation or test set).
To overcome the overfitting problem, we can reduce the complexity of the algorithm by reducing the depth size.
A decision tree with a depth of 1 is called `decision stump` and has only one split from the root.
**Classes, functions, and methods**:
- `DecisionTreeClassifier`: classification model from `sklearn.tree` class.
- `max_depth`: hyperparameter to control the maximum depth of decision tree algorithm.
- `export_text`: method from `sklearn.tree` class to display the text report showing the rules of a decision tree.
*Note*: we have already covered `DictVectorizer` in session 3 and `roc_auc_score` in session 4 respectively.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-6-decision-trees-and-ensemble-learning)
## Notes
This lesson first reviews the topics learned in the previous lesson about how to train a decision tree using scikit-learn, and handle a decision tree model not generalizing well due to overfitting of the data.
In this lesson, we learn about how to best split a decision tree and different classification criteria that can be used to split a tree. We dive deep in an example, splitting trees with `misclassification` criteria. Additionally, different stopping criteria to break the iterative tree split are discussed.
* **Structure of a decision tree**: A decision tree is a data structure
composed of **nodes** (which contain conditions) and **branches** (which represent the values for a condition: True or False). The tree starts with a **root node**, which is the parent of two other nodes, and each of these nodes can also be the parent of others. At the last level of the tree, there are terminal nodes, also called **leaves**.
* **Depth of a decision tree**: The **depth** of a tree is the number of levels it has, or simply the length of the longest path from the root node to a leaf node.
* **Rules & Conditions, Thresholds**: The learning algorithm for a decision tree involves determining the best **conditions** to split the data at each node in order to achieve the best possible classifier. When there are many **features**, the algorithm considers each feature with its optimized **threshold** to determine the best feature for splitting at a particular node. In essence, at each node, the algorithm evaluates all possible thresholds for every feature and calculates the resulting misclassification rate. It then selects the condition
(feature and threshold) that yields the lowest impurity.
* **Misclassification rate**: After each split, the goal is to divide the data into two sets that are as **pure** as possible. This means that the data within each set should belong predominantly to either one class, or the other. Another way to describe this is to aim for the lowest possible **misclassification rate** (impurity). The misclassification rate is a weighted average of the error rates obtained after splitting the data into two sets. The predicted class for each set is determined by the **majority class** present in this set.
* **Impurity criteria**: Common misclassification rate measurements are **GINI Impurity** and **Entropy**. It is also possible to use **MSE** for regression problems.
* **Decision trees can be used to solve regression problems**: While we focused on decision tree classifiers, it's important to note that decision trees can also be applied to regression problems using decision tree regressors.
* **Stopping Criteria**: The process of recursively splitting the data at each child node eventually stops based on certain **stopping criteria**. These criteria prevent the model from overfitting and include:
* The group is already **pure**: 0% impurity.
* The **maximum depth** has been reached.
* The group is **smaller** than the minimum size set for groups.
* The maximum number of **leaves/terminal nodes** has been reached.
#### Decision Tree Learning Algorithm in a Nutshell
* At a node, find the best split.
* Stop if max\_depth is reached.
* For each child node, if the node is sufficiently large and not pure, repeat the process from the beginning.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-6-decision-trees-and-ensemble-learning)
## Notes
In this lesson, we will discuss about different parameters used to control a Decision Tree (DT). Two of them, `max_depth` and `min_samples_leaf` have a greater importance than the others. We will further see how we first tune `max_depth` parameter and then move to tuning other parameters will help. After that, a dataframe will be created with all possible combinations of `max_depth`, `min_sample_leaf` and the auc score corresponding to them. These results will be visualized using a heatmap by pivoting the dataframe to easily determine the best possible `max_depth` and `min_samples_leaf` combination. Finally, the DT will be retrained using the identified parameter combination. The DT so trained will be viewed as a tree diagram, for visualizing decision rules.
### Steps
* **Fine-Tuning Process:** iterate to find optimal parameter settings.
* Start by tuning `max_depth` with various values to determine a subset of optimal
depths.
* Then, using this subset, fine-tune the model further by exploring different
`min_samples_leaf` values.
This method is computationally efficient for **large datasets**, though it may not be optimal for smaller ones.
* **Heatmaps for Visualization:** Store the scores (e.g., AUC) obtained during tuning in a pivot table, and create a heatmap with `seaborn` to easily identify high score areas, which helps pinpoint the optimal `max_depth` and `min_samples_leaf` combination.
**NB:** Choose parameter values that effectively control the tree's size and avoid values like 'nan' (Not a Number), even if they seem to lead to better scores.
### Importance of `max_depth` and `min_samples_leaf`
* **Controlling Overfitting:** these parameters play a critical role in preventing overfitting.
* `max_depth` limits the tree's complexity, preventing it from growing too deep and memorizing the training data.
* `min_samples_leaf` ensures that leaf nodes have a sufficient number of samples,
reducing the chance of creating nodes that are too specific to the training data.
* **Impact on Bias and Variance:** They also affect the model's bias and variance.
* Increasing `max_depth` and decreasing `min_samples_leaf` can lead to a more complex model with lower bias but higher variance.
* Decreasing `max_depth` and increasing `min_samples_leaf` results in a simpler model with higher bias but lower variance.
It's then important to find the right balance between `max_depth` and `min_samples_leaf` to achieve optimal model performance.
This involves a trade-off between bias and variance, and the best values depend on the specific dataset and problem.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-6-decision-trees-and-ensemble-learning)
## Notes
**Ensemble learning** is a machine learning paradigm where multiple models, often referred to as 'weak learners', are strategically combined to solve a particular computational intelligence problem. This approach frequently yields superior predictive performance compared to using a single model.
**Random Forest** is an example of ensemble learning where each model is a decision tree and their predictions are aggregated to identify the most popular result. Random forest only selects a random subset of features from the original data to make predictions. The 'randomness' in Random Forest stems from two key aspects:
- Each tree is potentially trained on a bootstrapped sample of the original data, introducing randomness at the row level.
- At each node during tree construction, only a random subset of features is considered for splitting. This feature randomness helps decorrelate the trees, preventing overfitting and promoting generalization to unseen data.
**Bootstrapping** is a resampling technique where numerous subsets
of the data are created by sampling the original data with replacement. This means that
some data points may appear multiple times in a single bootstrap sample, while others may
be excluded. In Random Forest, each decision tree is trained on a distinct bootstrap sample,
further contributing to the diversity and robustness of the ensemble.
**Parameter tuning** is crucial for optimizing the performance of a
Random Forest model. Two critical parameters are `max_depth`, which controls the maximum
depth of each decision tree, and `n_estimators`, which determines the number of trees in
the forest. Increasing `max_depth` allows for more complex trees, potentially leading to
overfitting. Conversely, a larger `n_estimators` generally improves model accuracy but
increases computational cost.
In random forests, the decision trees are trained independently to each other.
**Classes, functions, and methods**:
- `from sklearn.ensemble import RandomForestClassifier`: random forest classifier from sklearn ensemble class.
- `plt.plot(x, y)`: draw line plot for the values of y against x values.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-6-decision-trees-and-ensemble-learning)
## Notes
**Gradient Boosting**
Unlike Random Forest where each decision tree trains independently, in the Gradient Boosting Trees, the models are combined sequentially, where each model takes the prediction errors made by the previous model and then tries to improve the prediction. This process continues to `n` number of iterations, and in the end, all the predictions get combined to make the final prediction.
XGBoost is one of the libraries which implements the gradient boosting technique. To make use of the library, we need to install with `pip install xgboost`. To train and evaluate the model, we need to wrap our train and validation data into a special data structure from XGBoost which is called `DMatrix`. This data structure is optimized to train XGBoost models faster.
**XGBoost Training Parameters**
* `eta`: learning rate, which indicates how fast the model learns.
* `max_depth`: to control the size of the trees.
* `min_child_weight`: to control the minimum size of a child node.
* `objective`: To specify which problem we are trying to solve, either regression, or classification (binary: `'binary:logistic'`, or other).
* `nthread`: 8, used for parallelized training.
* `seed`: 1, for reproducibility.
* `verbosity`: 1 (`True`) to show warnings, if any, during model training.
**Classes, functions, and methods**:
- `xgb.train()`: method to train xgboost model.
- `xgb_params`: key-value pairs of hyperparameters to train xgboost model.
- `watchlist`: list to store training and validation data to evaluate the performance of the model after each training iteration. The list takes tuple of train and validation set from DMatrix wrapper, for example, `watchlist = [(dtrain, 'train'), (dval, 'val')]`.
- `%%capture output`: IPython magic command which captures the standard output and standard error of a cell.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-6-decision-trees-and-ensemble-learning)
## Notes
XGBoost has various tunable parameters but the three most important ones are:
- `eta` (default=0.3)
- It is also called `learning_rate` and is used to prevent overfitting by regularizing the weights of new features in each boosting step. range: [0, 1]
- `max_depth` (default=6)
- Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit. range: [0, inf]
- `min_child_weight` (default=1)
- Minimum number of samples in leaf node. range: [0, inf]
For XGBoost models, there are other ways of finding the best parameters as well but the one we implement in the notebook follows the sequence of:
- First find the best value for `eta`
- Second, find the best value for `max_depth`
- Third, find the best value for `min_child_weight`
Other useful parameter are:
- `subsample` (default=1)
- Subsample ratio of the training instances. Setting it to 0.5 means that model would randomly sample half of the training data prior to growing trees. range: (0, 1]
- `colsample_bytree` (default=1)
- This is similar to random forest, where each tree is made with the subset of randomly choosen features.
- `lambda` (default=1)
- Also called `reg_lambda`. L2 regularization term on weights. Increasing this value will make model more conservative.
- `alpha` (default=0)
- Also called `reg_alpha`. L1 regularization term on weights. Increasing this value will make model more conservative.
### Alternative: Tuning XGBoost using a loop
Instead of repeating the training process manually for each `eta` value, you can use a loop to automate it (and other parameters to be tuned):
```python
# Train XGBoost models for each eta and store AUC results
scores = {} # dictionary to store results for each eta
etas = [0.01, 0.05, 0.1, 0.3, 1.0] # list of parameter values. in this case it is 'eta'.
for eta in etas:
evals_result = {}
xgb_params = {
'eta': eta,
'max_depth': 6,
'min_child_weight': 1,
'objective': 'binary:logistic',
'eval_metric': 'auc',
'nthread': 8,
'seed': 1,
'verbosity': 1
}
model = xgb.train(
xgb_params,
dtrain,
evals=watchlist,
verbose_eval=0,
num_boost_round=200,
evals_result=evals_result
)
columns = ['iter', 'train_auc', 'val_auc']
train_aucs = list(evals_result['train'].values())[0]
val_aucs = list(evals_result['val'].values())[0]
df_results = pd.DataFrame(
list(zip(range(1, len(train_aucs) + 1), train_aucs, val_aucs)),
columns=columns
)
key = f'eta={eta}'
scores[key] = df_results
```
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-6-decision-trees-and-ensemble-learning)
## Notes
We select the final model from decision tree, random forest, or xgboost based on the best auc scores. After that we prepare the `df_full_train` and `df_test` to train and evaluate the final model. If there is not much difference between model auc scores on the train as well as test data then the model has generalized the patterns well enough.
Generally, XGBoost models perform better on tabular data than other machine learning models but the downside is that these model are easy to overfit cause of the high number of hyperparameter. Therefore, XGBoost models require a lot more attention for parameters tuning to optimize them.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-6-decision-trees-and-ensemble-learning)
## Notes
- Decision trees learn if-then-else rules from data.
- Finding the best split: select the least impure split. This algorithm can overfit, that's why we control it by limiting the max depth and the size of the group.
- Random forest is a way of combining multiple decision trees. It should have a diverse set of models to make good predictions.
- Gradient boosting trains model sequentially: each model tries to fix errors of the previous model. XGBoost is an implementation of gradient boosting.
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
| \n", " | status | \n", "seniority | \n", "home | \n", "time | \n", "age | \n", "marital | \n", "records | \n", "job | \n", "expenses | \n", "income | \n", "assets | \n", "debt | \n", "amount | \n", "price | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "ok | \n", "9 | \n", "rent | \n", "60 | \n", "30 | \n", "married | \n", "no | \n", "freelance | \n", "73 | \n", "129 | \n", "0 | \n", "0 | \n", "800 | \n", "846 | \n", "
| 1 | \n", "ok | \n", "17 | \n", "rent | \n", "60 | \n", "58 | \n", "widow | \n", "no | \n", "fixed | \n", "48 | \n", "131 | \n", "0 | \n", "0 | \n", "1000 | \n", "1658 | \n", "
| 2 | \n", "default | \n", "10 | \n", "owner | \n", "36 | \n", "46 | \n", "married | \n", "yes | \n", "freelance | \n", "90 | \n", "200 | \n", "3000 | \n", "0 | \n", "2000 | \n", "2985 | \n", "
| 3 | \n", "ok | \n", "0 | \n", "rent | \n", "60 | \n", "24 | \n", "single | \n", "no | \n", "fixed | \n", "63 | \n", "182 | \n", "2500 | \n", "0 | \n", "900 | \n", "1325 | \n", "
| 4 | \n", "ok | \n", "0 | \n", "rent | \n", "36 | \n", "26 | \n", "single | \n", "no | \n", "fixed | \n", "46 | \n", "107 | \n", "0 | \n", "0 | \n", "310 | \n", "910 | \n", "
| \n", " | seniority | \n", "time | \n", "age | \n", "expenses | \n", "income | \n", "assets | \n", "debt | \n", "amount | \n", "price | \n", "
|---|---|---|---|---|---|---|---|---|---|
| count | \n", "4455.0 | \n", "4455.0 | \n", "4455.0 | \n", "4455.0 | \n", "4455.0 | \n", "4455.0 | \n", "4455.0 | \n", "4455.0 | \n", "4455.0 | \n", "
| mean | \n", "8.0 | \n", "46.0 | \n", "37.0 | \n", "56.0 | \n", "763317.0 | \n", "1060341.0 | \n", "404382.0 | \n", "1039.0 | \n", "1463.0 | \n", "
| std | \n", "8.0 | \n", "15.0 | \n", "11.0 | \n", "20.0 | \n", "8703625.0 | \n", "10217569.0 | \n", "6344253.0 | \n", "475.0 | \n", "628.0 | \n", "
| min | \n", "0.0 | \n", "6.0 | \n", "18.0 | \n", "35.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "100.0 | \n", "105.0 | \n", "
| 25% | \n", "2.0 | \n", "36.0 | \n", "28.0 | \n", "35.0 | \n", "80.0 | \n", "0.0 | \n", "0.0 | \n", "700.0 | \n", "1118.0 | \n", "
| 50% | \n", "5.0 | \n", "48.0 | \n", "36.0 | \n", "51.0 | \n", "120.0 | \n", "3500.0 | \n", "0.0 | \n", "1000.0 | \n", "1400.0 | \n", "
| 75% | \n", "12.0 | \n", "60.0 | \n", "45.0 | \n", "72.0 | \n", "166.0 | \n", "6000.0 | \n", "0.0 | \n", "1300.0 | \n", "1692.0 | \n", "
| max | \n", "48.0 | \n", "72.0 | \n", "68.0 | \n", "180.0 | \n", "99999999.0 | \n", "99999999.0 | \n", "99999999.0 | \n", "5000.0 | \n", "11140.0 | \n", "
| \n", " | seniority | \n", "time | \n", "age | \n", "expenses | \n", "income | \n", "assets | \n", "debt | \n", "amount | \n", "price | \n", "
|---|---|---|---|---|---|---|---|---|---|
| count | \n", "4455.0 | \n", "4455.0 | \n", "4455.0 | \n", "4455.0 | \n", "4421.0 | \n", "4408.0 | \n", "4437.0 | \n", "4455.0 | \n", "4455.0 | \n", "
| mean | \n", "8.0 | \n", "46.0 | \n", "37.0 | \n", "56.0 | \n", "131.0 | \n", "5403.0 | \n", "343.0 | \n", "1039.0 | \n", "1463.0 | \n", "
| std | \n", "8.0 | \n", "15.0 | \n", "11.0 | \n", "20.0 | \n", "86.0 | \n", "11573.0 | \n", "1246.0 | \n", "475.0 | \n", "628.0 | \n", "
| min | \n", "0.0 | \n", "6.0 | \n", "18.0 | \n", "35.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "100.0 | \n", "105.0 | \n", "
| 25% | \n", "2.0 | \n", "36.0 | \n", "28.0 | \n", "35.0 | \n", "80.0 | \n", "0.0 | \n", "0.0 | \n", "700.0 | \n", "1118.0 | \n", "
| 50% | \n", "5.0 | \n", "48.0 | \n", "36.0 | \n", "51.0 | \n", "120.0 | \n", "3000.0 | \n", "0.0 | \n", "1000.0 | \n", "1400.0 | \n", "
| 75% | \n", "12.0 | \n", "60.0 | \n", "45.0 | \n", "72.0 | \n", "165.0 | \n", "6000.0 | \n", "0.0 | \n", "1300.0 | \n", "1692.0 | \n", "
| max | \n", "48.0 | \n", "72.0 | \n", "68.0 | \n", "180.0 | \n", "959.0 | \n", "300000.0 | \n", "30000.0 | \n", "5000.0 | \n", "11140.0 | \n", "
| \n", " | seniority | \n", "home | \n", "time | \n", "age | \n", "marital | \n", "records | \n", "job | \n", "expenses | \n", "income | \n", "assets | \n", "debt | \n", "amount | \n", "price | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "10 | \n", "owner | \n", "36 | \n", "36 | \n", "married | \n", "no | \n", "freelance | \n", "75 | \n", "0.0 | \n", "10000.0 | \n", "0.0 | \n", "1000 | \n", "1400 | \n", "
| 1 | \n", "6 | \n", "parents | \n", "48 | \n", "32 | \n", "single | \n", "yes | \n", "fixed | \n", "35 | \n", "85.0 | \n", "0.0 | \n", "0.0 | \n", "1100 | \n", "1330 | \n", "
| 2 | \n", "1 | \n", "parents | \n", "48 | \n", "40 | \n", "married | \n", "no | \n", "fixed | \n", "75 | \n", "121.0 | \n", "0.0 | \n", "0.0 | \n", "1320 | \n", "1600 | \n", "
| 3 | \n", "1 | \n", "parents | \n", "48 | \n", "23 | \n", "single | \n", "no | \n", "partime | \n", "35 | \n", "72.0 | \n", "0.0 | \n", "0.0 | \n", "1078 | \n", "1079 | \n", "
| 4 | \n", "5 | \n", "owner | \n", "36 | \n", "46 | \n", "married | \n", "no | \n", "freelance | \n", "60 | \n", "100.0 | \n", "4000.0 | \n", "0.0 | \n", "1100 | \n", "1897 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 2667 | \n", "18 | \n", "private | \n", "36 | \n", "45 | \n", "married | \n", "no | \n", "fixed | \n", "45 | \n", "220.0 | \n", "20000.0 | \n", "0.0 | \n", "800 | \n", "1600 | \n", "
| 2668 | \n", "7 | \n", "private | \n", "60 | \n", "29 | \n", "married | \n", "no | \n", "fixed | \n", "60 | \n", "51.0 | \n", "3500.0 | \n", "500.0 | \n", "1000 | \n", "1290 | \n", "
| 2669 | \n", "1 | \n", "parents | \n", "24 | \n", "19 | \n", "single | \n", "no | \n", "fixed | \n", "35 | \n", "28.0 | \n", "0.0 | \n", "0.0 | \n", "400 | \n", "600 | \n", "
| 2670 | \n", "15 | \n", "owner | \n", "48 | \n", "43 | \n", "married | \n", "no | \n", "freelance | \n", "60 | \n", "100.0 | \n", "18000.0 | \n", "0.0 | \n", "2500 | \n", "2976 | \n", "
| 2671 | \n", "12 | \n", "owner | \n", "48 | \n", "27 | \n", "married | \n", "yes | \n", "fixed | \n", "45 | \n", "110.0 | \n", "5000.0 | \n", "1300.0 | \n", "450 | \n", "1636 | \n", "
2672 rows × 13 columns
\n", "| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 0 | \n", "8000 | \n", "default | \n", "
| 1 | \n", "2000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "default | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 2 | \n", "0 | \n", "default | \n", "
| 1 | \n", "2000 | \n", "default | \n", "
| 7 | \n", "3000 | \n", "default | \n", "
| 5 | \n", "4000 | \n", "ok | \n", "
| 3 | \n", "5000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "ok | \n", "
| 0 | \n", "8000 | \n", "default | \n", "
| 6 | \n", "9000 | \n", "ok | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 1 | \n", "2000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "default | \n", "
| 5 | \n", "4000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "default | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 0 | \n", "8000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "ok | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 2 | \n", "0 | \n", "default | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 0 | \n", "8000 | \n", "default | \n", "
| 1 | \n", "2000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "default | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 1 | \n", "2000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "default | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 0 | \n", "8000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "default | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 1 | \n", "2000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "default | \n", "
| 7 | \n", "3000 | \n", "default | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 0 | \n", "8000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "ok | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 1 | \n", "2000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "default | \n", "
| 5 | \n", "4000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "default | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 0 | \n", "8000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "ok | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 1 | \n", "2000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "default | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 0 | \n", "8000 | \n", "default | \n", "
| 6 | \n", "9000 | \n", "ok | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 0 | \n", "8000 | \n", "default | \n", "
| 1 | \n", "2000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "default | \n", "
| \n", " | assets | \n", "status | \n", "
|---|---|---|
| 6 | \n", "9000 | \n", "ok | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 1 | \n", "2000 | \n", "1000 | \n", "default | \n", "
| 2 | \n", "0 | \n", "1000 | \n", "default | \n", "
| 3 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 4 | \n", "5000 | \n", "1000 | \n", "ok | \n", "
| 5 | \n", "4000 | \n", "1000 | \n", "ok | \n", "
| 6 | \n", "9000 | \n", "500 | \n", "ok | \n", "
| 7 | \n", "3000 | \n", "2000 | \n", "default | \n", "
| \n", " | assets | \n", "debt | \n", "status | \n", "
|---|---|---|---|
| 0 | \n", "8000 | \n", "3000 | \n", "default | \n", "
| \n", " | auc | \n", "||
|---|---|---|---|
| max_depth | \n", "4 | \n", "5 | \n", "6 | \n", "
| min_samples_leaf | \n", "\n", " | \n", " | \n", " |
| 1 | \n", "0.761 | \n", "0.767 | \n", "0.759 | \n", "
| 5 | \n", "0.761 | \n", "0.768 | \n", "0.759 | \n", "
| 10 | \n", "0.761 | \n", "0.762 | \n", "0.778 | \n", "
| 15 | \n", "0.764 | \n", "0.772 | \n", "0.785 | \n", "
| 20 | \n", "0.761 | \n", "0.774 | \n", "0.774 | \n", "
| 100 | \n", "0.756 | \n", "0.763 | \n", "0.776 | \n", "
| 200 | \n", "0.747 | \n", "0.759 | \n", "0.768 | \n", "
| 500 | \n", "0.680 | \n", "0.680 | \n", "0.680 | \n", "
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
If you have problems cloning the repository with the data, use a different command for cloning:
```bash
git clone https://github.com/alexeygrigorev/clothing-dataset-small.git
```
In this session, we'll be working on multiclass image classification with deep learning. Some deep learning frameworks like TensorFlow and Keras will be implemented on clothing dataset to classify images of clothes as t-shirts, hats, pants, etc.
The dataset has 5000 images of 20 different classes. However, we'll be using the subset which contains 10 of the most popular classes. The dataset can be downloaded from the above link.
**Useful links**:
- Full dataset: https://www.kaggle.com/agrigorev/clothing-dataset-full
- Subset: https://github.com/alexeygrigorev/clothing-dataset-small
- Corresponding Medium article: https://medium.com/data-science-insider/clothing-dataset-5b72cd7c3f1f
- CS231n CNN for Visual Recognition: https://cs231n.github.io/
## Notes
Add notes from the video (PRs are welcome)
* introduction to this weeks topics
* using images as data source instead of tabular data
* classifying images of t-shirts using a convolutional neural network
* 5000 images and 20 different classes https://www.kaggle.com/agrigorev/clothing-dataset-full
* see corresponding medium.com article https://medium.com/data-science-insider/clothing-dataset-5b72cd7c3f1f
* deep learning with tensorflow and keras
* parameters, layers, regularization
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
In the video the ```load_img``` function from Keras is imported using
```python
from tensorflow.keras.processing.image import load_img
```
If the import is not working, try using
```python
from tensorflow.keras.utils import load_img
```
[Tensorflow documentation](https://www.tensorflow.org/api_docs/python/tf/keras/utils/load_img)
Notes:
- TensorFlow is a library to train deep learning models and Keras is higher level abstraction on the top of TensorFlow. Keras used to be separate library but from tensorflow 2+ version, keras became part of the tensorflow library. The libraries can be installed using `pip install tensorflow` (for CPU and GPU). However, additional setup is required to integrate TensorFlow with GPU.
- Neural networks expect an image of a certain size, therefore, we need to provide the image size in `target_size` parameter of the `load_img` function.
- Each image consists of pixel and each of these pixels has the shape of 3 dimensions ***(height, width, color channels)***
- A typical color image consists of three color channels: `red`, `green` and `blue`. Each color channel has 8 bits or 1 byte and can represent distinct values between 0-256 (uint8 type).
**Classes, functions, and methods**:
- `import tensorflow as tf`: to import tensorflow library
- `from tensorflow import keras`: to import keras
- `from tensorflow.keras.preprocessing.image import load_img`: to import load_img function
- `load_img('path/to/image', targe_size=(150,150))`: to load the image of 150 x 150 size in PIL format
- `np.array(img)`: convert image into a numpy array of 3D shape, where each row of the array represents the value of red, green, and blue color channels of one pixel in the image.
## Notes
Add notes from the video (PRs are welcome)
* tensorflow and keras as deep learning libraries
* end-to-end open source machine learning framework
* tensorflow as library for training deep learning models
* keras as high-level abstraction on top of tensorflow
* installing tensorflow
* local vs cloud configuration
* loading and preprocessing images
* keras is part of tensorflow since version 2.0
* working with different image sizes
* processing images using the python pillow library
* encoding images as numpy arrays
* image size (i.e. 150 x 150 pixels) multiplied by number of colors (i.e. RGB) equals shape of array
* numpy array dtype as unsigned int8 (uint8) which includes the range from 0 to 255
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
> **Important**: If you rent a GPU from a cloud provider (such as AWS), don't forget to turn it
> off after you finish. It's not free and you might get a large bill at the end of the month.
### Links
* [Renting a GPU with AWS SageMaker](https://livebook.manning.com/book/machine-learning-bookcamp/appendix-e/6)
- The `keras.applications` module has different pre-trained models with different architectures. We'll use the model [Xception](https://keras.io/api/applications/xception/) which takes the input image size of `(229, 229)` and each image's pixel is scaled between `-1` and `1`.
- We create the instance of the pre-trained model using `model = Xception(weights='imagenet', input_shape=(299, 229, 3))`. Our model will use the weights from pre-trained imagenet and expect the input shape of (229, 229, 3) for images.
- Along with image size, the model also expects the `batch_size` which is the size of the batches of data (default 32). If one image is passed to the model, then the expected shape of the model should be (1, 229, 229, 3).
- The image data was peprocessed using `preprocess_input` function during `Xception` model's pre-taining. Therefore, we'll have to use this function on our data before making predictions, like so: `X = preprocess_input(X)`.
- The `pred = model.predict(X)` function returns 2D array of shape `(1, 1000)`, where 1000 is the probablity of the image classes. `decode_predictions(pred)` can be used to get the class names and their probabilities in readable format.
- In order to make the pre-trained model useful specific to our case, we'll have to do some tweak, which we'll do in the coming sections.
**Classes, functions, and methods**:
- `from tensorflow.keras.applications.xception import Xception`: import the model from keras applications
- `from tensorflow.keras.application.xception import preprocess_input`: function to perform preprocessing on images
- `from tensorflow.keras.applications.xception import decode_predictions`: extract the predictions class names in the form of tuple of list
- `model.predict(X)`: function to make predictions on the test images
**Links**:
- [Renting a GPU with AWS SageMaker](https://livebook.manning.com/book/machine-learning-bookcamp/appendix-e/23)
- [Keras Applications](https://keras.io/api/applications/) provide a list of pre-trained deep learning models
- [ImageNet](https://www.image-net.org/) is an image database that has 1,431,167 images of 1000 classes
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
### What is Convolutional Neural Network?
A convolutional neural network, also know as CNN or ConvNet, is a feed-forward neural network that is generally used to analyze visual images by processing data with grid-like topology. A CNN is used to detect and classify objects in an image. In CNNs, every image is represented in the form of an array of pixel values.
The convoluion operation forms the basis of any CNN. In a convolution operation, the arrays are multiplied element-wise, and the dot product is summed to create a new array, which represents `Wx`.
### Layers in a Convolutional Neural Network
A Convolution neural network has multiple hidden layers that help in extracting information from an image. The four important layers in CNN are:
1. Convolution layer
2. ReLU layer
3. Pooling layer
4. Fully connected layer (also called Dense layer)
**Convolution layer**
This is the first step in the process of extracting valuable features from an image. A convolution layer has several filters that perform the convolution operation. Every image is considered as a matrix of pixel values.
Consider a black and white image of 5x5 size whose pixel values are either 0 or 1 and also a filter matrix with a dimension of 3x3. Next, slide the filter matrix over the image and compute the dot product to get the convolved feature matrix.
```python
nn.Conv2d(
in_channels, # number of channels in the input image
out_channels, # number of filters to learn
kernel_size, # size of each filter (int or tuple)
stride=1, # step size for moving the filter
padding=0, # zero-padding around input
```
Explanation:
* in_channels: Input depth (e.g., 3 for RGB images).
* out_channels: Number of filters the layer will learn. Each produces one output feature map.
* kernel_size: Size of the convolutional filter. Can be a single number (square filter) or a tuple (height, width).
* stride: How many pixels the filter moves each step. Default is 1.
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
Transfer learning is a machine learning method where a model developed for a task is reused as the starting point for a model on a second task. Usually a pretrained model is trained with large volume of images and that is why the convolutional layers and vector representation of this model can be used for other tasks as well. However, the dense layers need to be retrained because they are specific to the dataset to make predictions with. In our problem, we want to keep convoluational layers but we want to train new dense layers.
Following are the steps to create train/validation data for model:
```python
# Build image generator for training (takes preprocessing input function)
train_gen = ImageDataGenerator(preprocessing_function=preprocess_input)
# Load in train dataset into train generator
train_ds = train_gen.flow_from_directory(directory=path/to/train_imgs_dir, # Train images directory
target_size=(150,150), # resize images to train faster
batch_size=32) # 32 images per batch
# Create image generator for validation
val_gen = ImageDataGenerator(preprocessing_function=preprocess_input)
# Load in image for validation
val_ds = val_gen.flow_from_directory(directory=path/to/val_imgs_dir, # Validation image directory
target_size=(150,150),
batch_size=32,
shuffle=False) # False for validation
```
Following are the steps to build model from a pretrained model:
```python
# Build base model
base_model = Xception(weights='imagenet',
include_top=False, # to create custom dense layer
input_shape=(150,150,3))
# Freeze the convolutional base by preventing the weights being updated during training
base_model.trainable = False
# Define expected image shape as input
inputs = keras.Input(shape=(150,150,3))
# Feed inputs to the base model
base = base_model(inputs, training=False) # set False because the model contains BatchNormalization layer
# Convert matrices into vectors using pooling layer
vectors = keras.layers.GlobalAveragePooling2D()(base)
# Create dense layer of 10 classes
outputs = keras.layers.Dense(10)(vectors)
# Create model for training
model = keras.Model(inputs, outputs)
```
Following are the steps to instantiate optimizer and loss function:
```python
# Define learning rate
learning_rate = 0.01
# Create optimizer
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
# Define loss function
loss = keras.losses.CategoricalCrossentropy(from_logits=True) # to keep the raw output of dense layer without applying softmax
# Compile the model
model.compile(optimizer=optimizer,
loss=loss,
metrics=['accuracy']) # evaluation metric accuracy
```
The model is ready to train once it is defined and compiled:
```python
# Train the model, validate it with validation data, and save the training history
history = model.fit(train_ds, epochs=10, validation_data=val_ds)
```
**Classes, function, and attributes**:
- `from tensorflow.keras.preprocessing.image import ImageDataGenerator`: to read the image data and make it useful for training/validation
- `flow_from_directory()`: method to read the images directly from the directory
- `next(train_ds)`: to unpack features and target variables
- `train_ds.class_indices`: attribute to get classes according to the directory structure
- `GlobalAveragePooling2D()`: accepts 4D tensor as input and operates the mean on the height and width dimensionalities for all the channels and returns vector representation of all images
- `CategoricalCrossentropy()`: method to produces a one-hot array containing the probable match for each category in multi classification
- `model.fit()`: method to train model
- `epochs`: number of iterations over all of the training data
- `history.history`: history attribute is a dictionary recording loss and metrics values (accuracy in our case) for each epoch
## Notes
Add notes from the video (PRs are welcome)
* convolutional layers convert an image into a vector representation
* dense layers use vector representations to make predictions
* using a pretrained neural network
* imagenet has 1000 different classes
* a dense layer may be specific to a certain number of classes whereas the vector representation can be applied to another dataset
* reusing the vector representation from convolutional layers means transferring knowledge and the idea behind transfer learning
* train faster on smaller size images
* the batch size
* base model vs custom model
* bottom layers vs top layers in keras
* keras optimizers
* using the adam optimizer
* weights, learning rates
* eta in xgboost
* model loss
* categorical cross entropy
* changing accuracy during several training epochs
* overfitting
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
One of the most important hyperparameters of deep learning models is the learning rate. It is a tuning parameter in an optimization function that determines the step size (how big or small) at each iteration while moving toward a mininum of a loss function.
Imagine you have a book, and you want to read it. The *learning rate* represents how fast you can read and absorb its content. If you read the book very quickly, you risk forgetting important parts and struggling to recall key details when you need to apply them. On the other hand, reading slowly allows you to study each concept thoroughly and understand it deeply, ensuring better retention. However, if you read too slowly, you might never finish the book. The goal is to find the right reading pace, or learning rate, that balances comprehension and efficiency. Reading too fast may result in superficial understanding, while reading too slowly might mean not acquiring knowledge quickly enough to meet your goals. By maintaining a moderate, balanced pace, you can maximize understanding and effectively apply what you've learned.
This analogy relates to training machine learning models. Training a model is like reading a book: you're trying to "learn" from the data. Applying that knowledge during testing or validation corresponds to validating the model. If you train the model too quickly (with a high learning rate), it may overfit, memorizing the training data without generalizing well to new data. If you train it too slowly (with a low learning rate), it may underfit, failing to learn enough patterns from the data. A balanced learning rate ensures the model acquires sufficient knowledge and performs well on both training and validation data.
We can experiement with different learning rates to find the optimal value where the model has best results. In order to try different learning rates, we should define a function to create a function first, for instance:
```python
# Function to create model
def make_model(learning_rate=0.01):
base_model = Xception(weights='imagenet',
include_top=False,
input_shape=(150,150,3))
base_model.trainable = False
#########################################
inputs = keras.Input(shape=(150,150,3))
base = base_model(inputs, training=False)
vectors = keras.layers.GlobalAveragePooling2D()(base)
outputs = keras.layers.Dense(10)(vectors)
model = keras.Model(inputs, outputs)
#########################################
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
loss = keras.losses.CategoricalCrossentropy(from_logits=True)
# Compile the model
model.compile(optimizer=optimizer,
loss=loss,
metrics=['accuracy'])
return model
```
Next, we can loop over on the list of learning rates:
```python
# Dictionary to store history with different learning rates
scores = {}
# List of learning rates
lrs = [0.0001, 0.001, 0.01, 0.1]
for lr in lrs:
print(lr)
model = make_model(learning_rate=lr)
history = model.fit(train_ds, epochs=10, validation_data=val_ds)
scores[lr] = history.history
print()
print()
```
Visualizing the training and validation accuracies help us to determine which learning rate value is the best for for the model. One typical way to determine the best value is by looking at the gap between training and validation accuracy. The smaller gap indicates the optimal value of the learning rate.
## Notes
Add notes from the video (PRs are welcome)
* learning rate analogy: the speed of reading a book
* reading fast (skimming thus missing details) vs reading slow (not much progress and leaving out books)
* finding the optimal learning rate
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
`ModelCheckpoint` callback is used with training the model to save a model or weights in a checkpoint file at some interval, so the model or weights can be loaded later to continue the training from the state saved or to use for deployment.
**Classes, function, and attributes**:
- `keras.callbacks.ModelCheckpoint`: ModelCheckpoint class from keras callbacks api
- `filepath`: path to save the model file
- `monitor`: the metric name to monitor
- `save_best_only`: only save when the model is considered the best according to the metric provided in `monitor`
- `mode`: overwrite the save file based on either maximum or the minimum scores according the metric provided in `monitor`
## Notes
Add notes from the video (PRs are welcome)
* checkpointing saves the model after each training iteration
* checkpoint conditions may include reaching the best performance
* keras callbacks
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
It is also possible to add more layers between the `vector representation layer` and the `output layer` to perform intermediate processing of the vector representation. These layers are the same dense layers as the output but the difference is that these layers use `relu` activation function for non-linearity.
Like learning rates, we should also experiment with different values of inner layer sizes:
```python
# Function to define model by adding new dense layer
def make_model(learning_rate=0.01, size_inner=100): # default layer size is 100
base_model = Xception(weights='imagenet',
include_top=False,
input_shape=(150,150,3))
base_model.trainable = False
#########################################
inputs = keras.Input(shape=(150,150,3))
base = base_model(inputs, training=False)
vectors = keras.layers.GlobalAveragePooling2D()(base)
inner = keras.layers.Dense(size_inner, activation='relu')(vectors) # activation function 'relu'
outputs = keras.layers.Dense(10)(inner)
model = keras.Model(inputs, outputs)
#########################################
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
loss = keras.losses.CategoricalCrossentropy(from_logits=True)
# Compile the model
model.compile(optimizer=optimizer,
loss=loss,
metrics=['accuracy'])
return model
```
Next, train the model with different sizes of inner layer:
```python
# Experiement different number of inner layer with best learning rate
# Note: We should've added the checkpoint for training but for simplicity we are skipping it
learning_rate = 0.001
scores = {}
# List of inner layer sizes
sizes = [10, 100, 1000]
for size in sizes:
print(size)
model = make_model(learning_rate=learning_rate, size_inner=size)
history = model.fit(train_ds, epochs=10, validation_data=val_ds)
scores[size] = history.history
print()
print()
```
Note: It may not always be possible that the model improves. Adding more layers mean introducing complexity in the model, which may not be recommended in some cases.
In the next section, we'll try different regularization technique to improve the performance with the added inner layer.
## Notes
Add notes from the video (PRs are welcome)
* softmax takes raw scores from a dense layer and transforms it into a probability
* activation functions used for output vs activation functions used for intermediate steps
* have a look at http://cs231n.stanford.edu/2017/
* sigmoid: negativ input --> zero, positive input --> straight line
* relu
* softmax
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
Dropout is a technique that prevents overfitting in neural networks by randomly dropping nodes of a layer during training. As a result, the trained model works as an ensemble model consisting of multiple neural networks.
From previous experiments we got the best values of learning rate `0.01` and layer size of `100`. We'll use these values for the next experiment along with different values of dropout rates:
```python
# Function to define model by adding new dense layer and dropout
def make_model(learning_rate=0.01, size_inner=100, droprate=0.5):
base_model = Xception(weights='imagenet',
include_top=False,
input_shape=(150,150,3))
base_model.trainable = False
#########################################
inputs = keras.Input(shape=(150,150,3))
base = base_model(inputs, training=False)
vectors = keras.layers.GlobalAveragePooling2D()(base)
inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
drop = keras.layers.Dropout(droprate)(inner) # add dropout layer
outputs = keras.layers.Dense(10)(drop)
model = keras.Model(inputs, outputs)
#########################################
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
loss = keras.losses.CategoricalCrossentropy(from_logits=True)
# Compile the model
model.compile(optimizer=optimizer,
loss=loss,
metrics=['accuracy'])
return model
# Create checkpoint to save best model for version 3
filepath = './xception_v3_{epoch:02d}_{val_accuracy:.3f}.h5'
checkpoint = keras.callbacks.ModelCheckpoint(filepath=filepath,
save_best_only=True,
monitor='val_accuracy',
mode='max')
# Set the best values of learning rate and inner layer size based on previous experiments
learning_rate = 0.001
size = 100
# Dict to store results
scores = {}
# List of dropout rates
droprates = [0.0, 0.2, 0.5, 0.8]
for droprate in droprates:
print(droprate)
model = make_model(learning_rate=learning_rate,
size_inner=size,
droprate=droprate)
# Train for longer (epochs=30) cause of dropout regularization
history = model.fit(train_ds, epochs=30, validation_data=val_ds, callbacks=[checkpoint])
scores[droprate] = history.history
print()
print()
```
Note: Because we introduce dropout in the neural networks, we will need to train our model for longer, hence, number of epochs is set to `30`.
**Classes, functions, attributes**:
- `tf.keras.layers.Dropout()`: dropout layer to randomly sets input units (i.e, nodes) to 0 with a frequency of rate at each epoch during training
- `rate`: argument to set the fraction of the input units to drop, it is a value of float between 0 and 1
## Notes
Add notes from the video (PRs are welcome)
* A neural network might learn false patterns, i.e. if it repeatedly recognizes a certain logo on a t-shirt it might learn that the logo defines the t-shirt which is wrong since the logo might also be seen on a hoodie.
* hiding parts of the images (freeze) from being seen by the learning neural network
* dropout = randomly freezing parts of the image
* comparing different performance parameters while changing dropout rate and regularization
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
In this video, I had a typo/bug: instead of using `val_gen` for generating images for validation,
I used `train_gen`. That's why adding augmentations didn't help in the video.
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
Data augmentation is a process of artifically increasing the amount of data by generating new images from existing images. This includes adding minor alterations to images by flipping, cropping, adding brightness and/or contrast, and many more.
Keras `ImageDataGenerator` class has many parameters for data augmentation that we can use for generating data. Important thing to remember that the data augmentation should only be implemented on train data, not the validation. Here's how we can generate augmented data for training the model:
```python
# Create image generator for train data and also augment the images
train_gen = ImageDataGenerator(preprocessing_function=preprocess_input,
rotation_range=30,
width_shift_range=10.0,
height_shift_range=10.0,
shear_range=10,
zoom_range=0.1,
vertical_flip=True)
train_ds = train_gen.flow_from_directory(directory=train_imgs_dir,
target_size=(150,150),
batch_size=32)
```
**How to choose augmentations?**
- First step is to use our own judgement, for example, looking at the images (both on train and validation), does it make sense to introduce horizontal flip?
- Look at the dataset, what kind of variations are there? Are objects always centered?
- Augmentations are hyperparameters: like many other hyperparameters, often times we need to test whether image augmentations are useful for the model or not. If the model doesn't improve or have same performance after certain epochs (let's say 20), in that case we don't use it.
Usually augmented data required training for longer.
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
In this section we increase the image input size from `150` to `299`, reduce the amount of data augmentation parameters and lower the learning rate. This gives us the best results than any previous experiments.
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-8-neural-networks-and-deep-learning-250592316)
Earlier we used **h5 format** to save our model when creating the checkpoint. The HDF5 format contains the model's architecture, weights values, and `compile()` information. The saved model can be loaded and used for prediction with `keras.models.load_model(path/to/saved_model)` method.
To evaluate the model and make prediction on test data, we'll need to create the same preprocessing steps for the image as we have done with train and validation data:
```python
# Create image generator for test data
test_gen = ImageDataGenerator(preprocessing_function=preprocess_input)
# Path of test images directory
test_imgs_dir = '../input/mlzoomcampimageclassification/zoomcamp-image-classification/clothing-dataset-small/test'
# Load in test images to generator
test_ds = test_gen.flow_from_directory(directory=test_imgs_dir,
target_size=(299,299),
batch_size=32,
shuffle=False)
# Path of an image to make predictions
img_path = 'path/to/image'
# Load image
img = load_img(img_path, target_size=(299,299))
# Convert image to numpy array
x = np.array(img)
# Add batch dimension to the image
X = np.array([x])
# Preprocess the image
X = preprocess_input(X)
```
The model performance can be evaluated on test data with `model.evaluate(test_ds)` and the prediction on the test image can be made using the method `model.predict(X)`. We can then zip the class names and prediction to see the likelihood.
**Classes, functions, attributes**:
- `keras.models.load_model()`: method to load saved model
- `model.evaluate()`: method to evaluate the performance of the model based on the evaluation metrics
- `model.predict()`: method to make predictions of output depending on the input
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
- We can use pre-trained models for general image classification
- Convolutional layers let us turn an image into a vector
- Dense layers use the vector to make the predictions
- Instead of training a model from scratch, we can use transfer learning and re-use already trained convolutional layers
- First, train a small model (150x150) before training a big one (299x299)
- Learning rate - how fast the model trains. Fast learner aren't always best ones
- We can save the best model using callbacks and checkpointing
- To avoid overfitting, use dropout and augmentation
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
In the last session, we built and trained a clothes classification deep learning model using `Keras` and `TensorFlow`. This session focuses on deploying it. The model categorizes images of clothing items (e.g., 👕 t-shirts, 👖 pants, etc.) uploaded by users on a website. Deployment will be done using **AWS Lambda**, a serverless solution to execute code without managing servers, and instead of `TensorFlow`, we will use `TensorFlow-lite`.
Refer to [updates.md](updates.md) for info on running TF lite
in 2024.
## Notes
Add notes from the video (PRs are welcome)
* introduction to the topic of the week: deploying a deep learning model to the cloud, aws lambda and tensorflow lite
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
AWS Lambda is a **serverless computing service** that lets you execute code without worrying about managing servers. Here's an overview of how it works and its benefits:
### **Setting Up a Lambda Function 🛠️**
1. **Accessing Lambda:**
- Go to the AWS Management Console and search for the `Lambda` service.
2. **Creating a Function:**
- Choose the `Author from scratch` option.
- Name your function (e.g., `mlzoomcamp-test`).
- Select the runtime environment (e.g., `Python 3.9`) and architecture (`x86_64`).
3. **Understanding Function Parameters:**
- **`event`:** Contains the input data passed to the function (e.g., a JSON payload).
- **`context`:** Provides details about the invocation, configuration, and execution environment.
4. **Updating the Default Function:**
- Edit `lambda_function.py` with custom logic. Example:
```python
def lambda_handler(event, context):
print("Parameters:", event) # Print input parameters
url = event["url"] # Extract URL from input
return {"prediction": "clothes"} # Sample response
```
### **Testing and Deployment 🚀**
1. **Create a Test Event:**
- Define a mock input to simulate real-world data.
2. **Deploy Changes:**
- Save and deploy the function to apply updates.
3. **Test Your Function:**
- Run the function with the test event to ensure it works as expected.
### **Advantages of AWS Lambda ✅**
- **Serverless Architecture 🖥️:** No need to provision or manage servers.
- **Cost-Effective 💰:** Pay only for requests and compute time—idle time is free!
- **Automatic Scaling 📈:** Adjusts automatically based on request volume.
- **Ease of Use 🎯:** Focus on coding; AWS handles infrastructure.
### **Dynamic Link Management Use Case 🌐**
`AWS lambda` was used to automatically redirect users to updated invite links for joining the DataTalks.Club community. This is to avoid expired links on the user side, by using a Lambda function that reads from a config file where invitation links can be update.
### Free Tier Usage
Note that `AWS Lambda` offers a free tier that includes a certain number of free requests (1 million requests per month), and free compute time (400,000 GB-seconds per month).
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
TensorFlow is a relatively large framework, with an unpacked size of approximately 1.7 GB. The size of such frameworks is an important consideration for several reasons:
- **📜 Historical Constraints:** Previously, AWS Lambda imposed a limit of `50 MB` for package sizes. While Docker has since increased these limits to 10 GB, the size of the framework still plays a crucial role in certain scenarios.
- **⚡ Performance Issues with Large Images:** Large frameworks like TensorFlow result in increased storage costs, longer initialization times (e.g., for invoking a Lambda function), slower loading times, and a significantly larger RAM footprint.
### Optimizing with TensorFlow Lite 🚀
To address these challenges, TensorFlow Lite (TF-Lite) provides a lightweight alternative designed specifically for inference tasks (i.e., making predictions with `model.predict(X)`), excluding any other functionality such as training. To use TensorFlow Lite, the original TensorFlow model needs to be converted into the TF-Lite format. This process can significantly reduce model size and improve performance.
## Notes
New URL for downloading the model:
```bash
wget https://github.com/DataTalksClub/machine-learning-zoomcamp/releases/download/chapter7-model/xception_v4_large_08_0.894.h5 -O clothing-model.h5
```
Add notes from the video (PRs are welcome)
* tensorflow has a size of approximately 1.7 GB
* there are size limits of cloud services and docker container
* tensorflow lite is small in size and limited to using a model to make predictions (inference)
* convert tensorflow keras model to a tensorflow lite model
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Refer to [updates.md](updates.md) for info on running TF lite
in 2024.
### Using `pip install` for TF-Lite binaries
When using `pip` to install the compiled binary, make sure you use the raw file, not a link to the github page.
Correct:
```bash
pip install https://github.com/alexeygrigorev/tflite-aws-lambda/raw/main/tflite/tflite_runtime-2.14.0-cp310-cp310-linux_x86_64.whl
```
(Note `/raw/` in the path)
Also correct:
```bash
pip install https://github.com/alexeygrigorev/tflite-aws-lambda/blob/main/tflite/tflite_runtime-2.14.0-cp310-cp310-linux_x86_64.whl?raw=true
```
The wheel file above is for Python 3.10. Check other available compiled TF lite versions [here](https://github.com/alexeygrigorev/tflite-aws-lambda/tree/main/tflite).
Not correct - won't work:
```bash
pip install https://github.com/alexeygrigorev/tflite-aws-lambda/blob/main/tflite/tflite_runtime-2.14.0-cp310-cp310-linux_x86_64.whl
```
If the file is incorrect, you'll get an error message like that:
```
zipfile.BadZipFile: File is not a zip file
```
### `ENTRYPOINT` vs `CMD`
This link explains the difference between them: https://stackoverflow.com/a/34245657
> `ENTRYPOINT` specifies a command that will always be executed when the container starts.
> `CMD` specifies arguments that will be fed to the `ENTRYPOINT`.
In case of the lambda base pacakge, the authors already specified the entrypoint and
we only need to overwrite the arguments passed to the entrypoint,
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-10-kubernetes)
## Notes
Add notes from the video (PRs are welcome)
* same use case as in the session before: classifying images of clothes
* using tensorflow serving, written in C++, with focus on inference
* gRPC binary protocol
* deploying to kubernetes
* 1st component: gateway (download image, resize, turn into numpy array - computationally not expensive - can be done with CPU)
* 2nd component: model (matrix multiplications - computationally expensive - thus use GPU)
* scaling the two components independently: i.e. 5 gateways handing images to 1 model
* two components in two different docker container (lesson four)
* kubernetes main concepts (lesson five)
* running kubernetes on your local machine (lesson six)
* deploy the two services to kubernetes (lesson seven)
* move from local to cloud (lesson eight)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
To build the app we need to convert the keras model `HDF5` into special format called tensorflow `SavedModel`. To do that, we download a prebuilt model and save it in the working directory:
```bash
wget https://github.com/DataTalksClub/machine-learning-zoomcamp/releases/download/chapter7-model/xception_v4_large_08_0.894.h5 -O clothing-model.h5
```
Then convert the model to `SavedModel` format:
```python
import tensorflow as tf
from tensorflow import keras
model = keras.models.load_model('./clothing-model.h5')
tf.saved_model.save(model, 'clothing-model')
```
We can inspect what's inside the saved model using the utility (saved_model_cli) from TensorFlow and the following command:
```shell
saved_model_cli show --dir clothing-model --all
```
Running the command outputs a few things but we are interested in the signature, specifically the following one. For instance:
```bash
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input_8'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 299, 299, 3)
name: serving_default_input_8:0
The given SavedModel SignatureDef contains the following output(s):
outputs['dense_7'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 10)
name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
```
Alternatively one can also use the following command to output just the desired part:
```shell
saved_model_cli show --dir clothing-model --tag_set serve --signature_def serving_default
```
We can run the model (`clothing-model`) with the prebuilt docker image `tensorflow/serving:2.7.0`:
```bash
docker run -it --rm \
-p 8500:8500 \
-v $(pwd)/clothing-model:/models/clothing-model/1 \
-e MODEL_NAME="clothing-model" \
tensorflow/serving:2.7.0
```
- docker run -it --rm (to run the docker)
- -p 8500:8500 (port mapping)
- -v $(pwd)/clothing-model:/models/clothing-model/1 (volumn mapping of *absolute model directory* to *model directory inside the docker image*)
- -e MODEL_NAME="clothing-model" (set environment variable for docker image)
- tensorflow/serving:2.7.0 (name of the image to run)
Tensorflow uses special serving called `gRPC` protocol which is optimized to use binary data format. We need to convert our prediction into `protobuf`.
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
In the previous section we created jupyter notebook to communicate with the model deployed with tensorflow. This notebook fetches an image, pre-processes it, turns it into protobuf, sends it to tensorflow-serving, does post-processing, and finally gives a human-readable answer.
In this section we convert the notebook into python script to build a flask application. To convert the notebook into script we can run the command `jupyter nbconvert --to script notebook.ipynb`. We also rename the script to `gateway.py`.
Then we create functions to prepare a request, send it, and prepare the response. For the flask app, we can reuse the code from session 5:
```python
# Create flask app
app = Flask('gateway')
@app.route('/predict', methods=['POST'])
def predict_endpoint():
data = request.get_json()
url = data['url']
result = predict(url)
return jsonify(result)
```
Our application has two components: docker container with tensorflow serving and flask application with the gateway.
We also want to put everything in the `pipenv` for deployment. For that we need to install few libraries with pipenv: `pipenv install grpcio==1.42.0 flask gunicorn keras-image-helper`.
As we discussed tensorflow is a large library and we don't want to use it in our application. Instead we can use the following script to convert numpy array into protobuf format and import the `np_to_protobuf` function into our `gateway.py` script. In order the make the script work we need to install the following libraries as well `pipenv install tensorflow-protobuf==2.7.0 protobuf==3.19`:
```python
from tensorflow.core.framework import tensor_pb2, tensor_shape_pb2, types_pb2
def dtypes_as_dtype(dtype):
if dtype == "float32":
return types_pb2.DT_FLOAT
raise Exception("dtype %s is not supported" % dtype)
def make_tensor_proto(data):
shape = data.shape
dims = [tensor_shape_pb2.TensorShapeProto.Dim(size=i) for i in shape]
proto_shape = tensor_shape_pb2.TensorShapeProto(dim=dims)
proto_dtype = dtypes_as_dtype(data.dtype)
tensor_proto = tensor_pb2.TensorProto(dtype=proto_dtype, tensor_shape=proto_shape)
tensor_proto.tensor_content = data.tostring()
return tensor_proto
def np_to_protobuf(data):
if data.dtype != "float32":
data = data.astype("float32")
return make_tensor_proto(data)
```
**Links**
- Bash script to create custom tf-serving-protobuf and compile: https://github.com/alexeygrigorev/tensorflow-protobuf/blob/main/tf-serving-proto.sh
## Notes
Add notes from the video (PRs are welcome)
* turn jupyter notebook into flask app
* the notebook communicates with the model deployed with tensorflow
* the notebook fetches an image, pre-processes it, turns it into protobuf, sends it to tensorflow-serving, does post-processing and finally gives a human-readable answer
* convert notebook into python script and call the script gateway
* prepare request, send request, prepare response
* you can reuse the flask app code from session 5
* two components: docker container with tensorflow serving and flask application with the gateway
* be aware of the library sizes: tensorflow 1.7 GB, tensorflow CPU ~400 MB, tensorflow serving
* turn numpy array into protobuf format
* tensorflow protobuf
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-10-kubernetes)
Docker Compose is a tool that helps us to define and share multi-container applications. With Compose, we can create a YAML file to define the services (in our case it is `gateway` service and `clothing-model` model) and with a single command, we can spin everything up or tear it all down. Docker compose is very useful to test the application locally.
Instead of mapping the volumn, port, and then running the docker in the terminal for our tf-serving model (clothing-model), we want to create a docker image and put everything in there. For this we want to create docker image by the name `image-model.dockerfile`:
```dockerfile
FROM tensorflow/serving:2.7.0
# Copy model in the image
COPY clothing-model /models/clothing-model/1
# Specify environmental variable
ENV MODEL_NAME="clothing-model"
```
To build the image we also need to specify the dockerfile name along with the tag, for example, `docker build -t clothing-model:xception-v4-001 -f image-model.dockerfile .`
Since we have created the dockerfile to the image, we can simply run the image with `docker run -it --rm -p 8500:8500 clothing-model:xception-v4-001`
Similarly we can do the same thing for our gateway service. The file name is `image-gateway.dockerfile`:
```dockerfile
FROM python:3.8.12-slim
RUN pip install pipenv
# Create working directory in docker image
WORKDIR /app
# Copy Pipfile and Pipfile.lock files in working dir
COPY ["Pipfile", "Pipfile.lock", "./"]
# Install required packages using pipenv
RUN pipenv install --system --deploy
# Copy gateway and protobuf scripts in the working dir
COPY ["gateway.py", "protobuf.py", "./"]
EXPOSE 9696
ENTRYPOINT ["gunicorn", "--bind=0.0.0.0:9696", "gateway:app"]
```
Build image: `docker build -t clothing-model-gateway:001 -f image-gateway.dockerfile .`
Run image: `docker run -it --rm -p 9696:9696 clothing-gateway:001`
Upon running these two containers and testing for prediction, we should expect connection error. This is because the gateway service is unable to communicate with tf-serving. In order to connect the two containers and work simultaneously we need docker compose. Docker compose require yaml file which will be executed when running the commands from docker compose, usually the file is named as `docker-compose.yaml`:
```yaml
version: "3.9"
services:
clothing-model: # tf-serving model
image: zoomcamp-10-model:xception-v4-001
gateway: # flask gateway service
image: zoomcamp-10-gateway:002 # new version
environment:
- TF_SERVING_HOST=clothing-model:8500 # look for clothing model and port 8500
ports: # map host machine with gateway
- "9696:9696"
```
Now we also need to make slight changes in the `gateway.py` to make the environment variable configurable and assign it to the host. This can be done using `host = os.getenv('TF_SERVING_HOST', 'localhost:8500')`
Running the command `docker-compose up` will establish this connection between both images, and as everything is configured properly we should have the request predictions.
**Useful commands**
- `docker-compose up`: run docker compose
- `docker-compose up -d`: run docker compose in detached mode
- `docker ps`: to see the running containers
- `docker-compose down`: stop the docker compose
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-10-kubernetes)
Kubernetes, also known as **K8s**, is an open-source system for automating the deployment, scaling, and management of containerized applications.
### 🧩 Anatomy of a Kubernetes Cluster
Imagine we have a Kubernetes cluster. Within this cluster, there are **nodes** (servers or computers running the processes). On these nodes, we find 🐳 **pods:** containers that run specific images, with allocated resources like **RAM/CPU**.
Pods are typically grouped into **deployments** 📦. All pods in a deployment share the same Docker image and configuration. Think of a deployment as a set of identical workers, ready to process requests. For example, our **gateway service** can be structured as a deployment.
- Larger pods require more resources 💪.
- In addition to pods, Kubernetes uses **services** 🛎️, such as:
- 🌐 `gateway service`: An entry point for external requests.
- 🤖 `tf-model service`: Manages communication with model-serving pods.
### 🔄 How it Works
1. When a user uploads an image 🖼️ to the website, the request first reaches the **gateway service**.
2. The gateway routes the request to one of the available pods, distributing traffic evenly (load balancing ⚖️).
3. After pre-processing, the gateway deployment forwards the request to the **model service**.
4. The model service routes the request to a pod in the `tf-serving` deployment.
5. Predictions are made 🔮 and sent back to the user, following the same path.
### 📍 Service Types
Kubernetes services act as entry points to route requests to the correct pods:
- **External Services** (`Load Balancer` 🌐): Accessible from outside the cluster. Example: `gateway service`.
- **Internal Services** (`Cluster IP` 🔒): Accessible **only** within the cluster. Example: `model service`.
At the front of the cluster, there’s an **entry point** called `INGRESS` 🚪. This directs user traffic to the appropriate external services.
### ⚙️ Scaling with Kubernetes
To handle multiple users simultaneously, Kubernetes can launch additional pods 🚀.
- As traffic increases, Kubernetes **automatically scales** the deployment up 🆙.
- When traffic decreases, it scales down 🛑 to save resources.
- This dynamic scaling is managed by the **Horizontal Pod Autoscaler (HPA)** 📊.
- If existing nodes are overwhelmed, Kubernetes can even request the creation of new nodes to handle the extra load.
Kubernetes ensures that your application stays responsive, efficient, and ready to scale at any moment!
## Notes
Add notes from the video (PRs are welcome)
* kubernetes is open source system for automating deployment scaling and management of containerized applications
* to scale up = add more instances of our application
* add more instances when load increases and remove instances when load decreases
* kubernetes cluster consists of nodes (running machines, servers)
* each node can have multiple container
* one container = one pod
* grouping pods according to type of docker image
* routing the request to the pods
* external (visible, i.e. entry point) service/client vs internal service/client
* HPA horizontal pod autoscaler = allocating resources depending on demand
* Ingress
* kubernetes configuration
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
[Slides](https://www.slideshare.net/AlexeyGrigorev/ml-zoomcamp-10-kubernetes)
## Creating a cluster with Kind
If you use WSL2 and get the following errors when creating a cluster with `kind create cluster`
```
✗ Starting control-plane :joystick:
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
the solution is to change the command:
```
Specify the node image:
```
kind create cluster --image kindest/node:v1.23.0
```
In this section, we'll deploy a simple web application to a kubernetes cluster. For that, we'll implement the following steps:
1. Create a simple ping application in Flask
- For this, we'll create a directory `ping` and for `pipenv` environment, we'll also create a separate `Pipfile` to avoid conflicts. Then we need to install `flask` and `gunicorn`.
- We'll use the app that we built in session 5 by copying `ping.py` and `Dockerfile` with slight changes and then build the image.
```python
# ping.py
from flask import Flask
app = Flask('ping-app')
@app.route('/ping', methods=['GET'])
def ping():
return 'PONG'
if __name__=="__main__":
app.run(debug=True, host='0.0.0.0', port=9696)
```
```dockerfile
# Dockerfile
FROM python:3.9-slim
RUN pip install pipenv
WORKDIR /app
COPY ["Pipfile", "Pipfile.lock", "./"]
RUN pipenv install --system --deploy
COPY "ping.py" .
EXPOSE 9696
ENTRYPOINT ["gunicorn", "--bind=0.0.0.0:9696", "ping:app"]
```
- To build the image, we need to specify the app name along with the tag, otherwise the local kubernetes setup `kind` will cause problems, `docker build -t ping:v001 .`. Now we can run on docker container and on separate terminal, use the command `curl localhost:9696/ping` to test the application.
2. Install `kubectl` and `kind` to build and test cluster locally
- We'll install kubectl from AWS because later we deploy our application on AWS: `curl -o kubectl https://s3.us-west-2.amazonaws.com/amazon-eks/1.24.7/2022-10-31/bin/linux/amd64/kubectl`.
- To install `kind` to setup local kubernetes setup (executable binaries): `wget https://kind.sigs.k8s.io/dl/v0.17.0/kind-linux-amd64 -O kind` > `chmod +x ./kind`. Once the utility is installed we need to place this into our `$PATH` at our preferred binary installation directory.
3. Setup kubernetes cluster and test it
- First thing we need to do is to create a cluster: `kind create cluster` (default cluster name is kind)
- Configure kubectl to interact with kind: `kubectl cluster-info --context kind-kind`
- Check the running services to make sure it works: `kubectl get service`
4. Create a deployment
- Kubernetes requires a lot of configuration and for that VS Code has a [handy extension](https://code.visualstudio.com/docs/azure/kubernetes) that can take a lot of hustle away.
- Create `deployment.yaml`
```yaml
apiVersion: apps/v1
kind: Deployment
metadata: # name of the deployment
name: ping-deployment
spec:
replicas: 1 # number of pods to create
selector:
matchLabels: # all pods that have the label app name 'ping' are belonged to 'ping-deployment'
app: ping
template: # template of pods (all pods have same configuration)
metadata:
labels: # each app gets the same label (i.e., ping in our case)
app: ping
spec:
containers: # name of the container
- name: ping-pod
image: ping:v001 # docker image with tag
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 9696 # port to expose
```
- We can now apply the `deployment.yaml` to our kubernetes cluster: `kubectl apply -f deployment.yaml`
- Next we need to load the docker image into our cluster: `kind load docker-image ping:v001`
- Executing the command `kubectl get pod` should give the pod status running.
- Test the pod by specifying the ports: `kubectl port-forward pod-name 9696:9696` and execute `curl localhost:9696/ping` to get the response.
5. Create service for deployment
- Create `service.yaml`
```yaml
apiVersion: v1
kind: Service
metadata: # name of the service ('ping')
name: ping
spec:
type: LoadBalancer # type of the service (external in this case)
selector: # which pods qualify for forwarding requests
app: ping
ports:
- port: 80 # port of the service
targetPort: 9696 # port of the pod
```
- Apply `service.yaml`: `kubectl apply -f service.yaml`
- Running `kubectl get service` will give us the list of external and internal services along with their *service type* and other information.
- Test the service by port forwarding and specifying the ports: `kubectl port-forward service/ping 8080:80` (using 8080 instead to avoid permission requirement) and executing `curl localhost:8080/ping` should give us the output PONG.
6. Setup and use `MetalLB` as external load-balancer
- Apply MetalLB manifest
```
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml
```
- Wait until the MetalLB pods (controller and speakers) are ready
```
kubectl wait --namespace metallb-system \
--for=condition=ready pod \
--selector=app=metallb \
--timeout=90s
```
- Setup address pool used by loadbalancers:
- Get range of IP addresses on docker kind network
```
docker network inspect -f '{{.IPAM.Config}}' kind
```
- Create Ip address pool using `metallb-config.yaml`
```yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: example
namespace: metallb-system
spec:
addresses:
- 172.20.255.200-172.20.255.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: empty
namespace: metallb-system
```
- Apply deployment and service for updates
```
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
```
- Get external LB_IP
```
kubectl get service
```
- Test using load-balancer ip address
```
curl | ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
In this section we'll deploy the tf-serving model to kubernetes. In order to do that we'll create a separate folder `kube-config` and implement the following steps:
- Create deployment for the tf-serving model `model-deployment.yaml`:
- ```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: tf-serving-clothing-model
spec:
replicas: 1
selector:
matchLabels:
app: tf-serving-clothing-model
template:
metadata:
labels:
app: tf-serving-clothing-model
spec:
containers:
- name: tf-serving-clothing-model
image: zoomcamp-10-model:xception-v4-001
resources:
limits:
memory: "512Mi"
cpu: "0.5"
ports:
- containerPort: 8500
```
- Load the model image to kind: `kind load docker-image clothing-model:xception-v4-001`
- Create model deployment: `kubectl apply -f model-deployment.yaml`
- Get the running pod id for the model: `kubectl get pod`
- Test the model deployment using the pod id: `kubectl port-forword tf-serving-clothing-model-85cd6dsb6-rfvg410m 8500:8500` and run `gateway.py` script to get the predictions.
- Create service of tf-serving model `model-service.yaml`:
- ```yaml
apiVersion: v1
kind: Service
metadata:
name: tf-serving-clothing-model
spec:
type: ClusterIP # default service type is always ClusterIP (i.e., internal service)
selector:
app: tf-serving-clothing-model
ports:
- port: 8500
targetPort: 8500
```
- Create model service: `kubectl apply -f model-service.yaml`
- Check the model service: `kubectl get service`.
- Test the model service: `kubectl port-forward service/tf-serving-clothing-model 8500:8500` and run `gateway.py` for predictions.
- Create deployment for the gateway `gateway-deployment.yaml`:
- ```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: gateway
spec:
selector:
matchLabels:
app: gateway
template:
metadata:
labels:
app: gateway
spec:
containers:
- name: gateway
image: zoomcamp-10-gateway:002
resources:
limits:
memory: "128Mi"
cpu: "100m"
ports:
- containerPort: 9696
env: # set the environment variable for model
- name: TF_SERVING_HOST
value: tf-serving-clothing-model.default.svc.cluster.local:8500 # kubernetes naming convention
```
- Load the gateway image to kind: `kind load docker-image clothing-model-gateway:002`
- Create gateway deployment `kubectl apply -f gateway-deployment.yaml` and get the running pod id `kubectl get pod`
- Test the gateway pod: `kubectl port-forward gateway-6b945f541-9gptfd 9696:9696` and execute `test.py` for getting predictions.
- Create service of tf-serving model `gateway-service.yaml`:
- ```yaml
apiVersion: v1
kind: Service
metadata:
name: gateway
spec:
type: LoadBalancer # External service to communicate with client (i.e., LoadBalancer)
selector:
app: gateway
ports:
- port: 80 # port of the service
targetPort: 9696 # port of load balancer
```
- Create gateway service: `kubectl apply -f gateway-service.yaml`
- Get service id: `kubectl get service`
- Test the gateway service: `kubectl port-forward service/gateway 8080:80` and replace the url on `test.py` to 8080 to get predictions.
**Links**
- Article to come over load balancing problem in production: https://kubernetes.io/blog/2018/11/07/grpc-load-balancing-on-kubernetes-without-tears/
## Notes
Add notes from the video (PRs are welcome)
* tensorflow serving in C++, gateway service as flask app
* gateway service: image preprocessing (i.e. resizing), prepare matrix, numpy arr, convert to protobuf, gRPC to communicate with tensorflow serving; postprocessing
* using telnet to check kubernetes pod
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
In this section, we'll create Elastic Kubernetes Service (EKS) cluster on Amazon using cli, publish images to ECR and configure kubectl.
To create cluster and manage on EKS we'll use a cli tool `eksctl` which can be downloaded from [here](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html). And next we'll follow these steps:
- In the `kube-config` folder create eks config file `eks-config.yaml`:
- ```yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: mlzoomcamp-eks
region: ap-south-1
nodeGroups: # for our case, we need only one node group (CPU)
- name: ng-m5-xlarge
instanceType: m5.xlarge
desiredCapacity: 1
```
- Create eks cluster: `eksctl create cluster -f eks-config.yaml`
- Publish local docker images to ECR:
- Create aws ecr repository for eks cluster: `aws ecr create-repository --repository-name mlzoomcamp-images`
- Bash commands to run in the terminal to push docker images to ecr repository:
- ```bash
# Registry URI
ACCOUNT_ID=22782589
REGION=ap-south-123
REGISTRY_NAME=mlzoomcamp-images
PREFIX=${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REGISTRY_NAME}
# Tag local docker images to remote tag
GATEWAY_LOCAL=zoomcamp-10-gateway:002 # gateway service
GATEWAY_REMOTE=${PREFIX}:clothing-model-gateway-002 # notice the ':' is replaced with '-' before 002
docker tag ${GATEWAY_LOCAL} ${GATEWAY_REMOTE}
MODEL_LOCAL=zoomcamp-10-model:xception-v4-001 # tf-serving model
MODEL_REMOTE=${PREFIX}:clothing-model-xception-v4-001 # same thing ':' is replaced with '-' before xception
docker tag ${MODEL_LOCAL} ${MODEL_REMOTE}
# Push tagged docker images
docker push ${MODEL_REMOTE}
docker push ${GATEWAY_REMOTE}
```
- Login to ecr and push images: `$(aws ecr get-login --no-include-email)`, first push the model and then gateway remote image.
- Get the uri of these images `echo ${MODEL_REMOTE}` and `echo ${GATEWAY_REMOTE}` and add them to `model-deployment.yaml` and `gate-deployment.yaml` respectively.
- Apply all the yaml config files to remote node coming from eks (`kubectl get nodes`):
- `kubectl apply -f model-deployment.yaml`
- `kubectl apply -f model-service.yaml`
- `kubectl apply -f gateway-deployment.yaml`
- `kubectl apply -f gateway-service.yaml`
- Testing the deployment pods and services should give us predictions.
- Executing `kubectl get service` should give us the *external port* address which need to add in the `test.py` as access url for predictions (e.g., `url = 'http://a3399e***-5180***.ap-south-123.elb.amazonaws.com/predict'`).
- To delete the remote cluster: `eksctl delete cluster --name mlzoomcamp-eks`
**NB:** EKS is not part of the Amazon Web Services Free Tier.
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
In the session 10 we covered the following concepts:
- TF-Serving is a system for deploying TensorFlow models
- When using TF-Serving, we need a component for pre-processing
- Kubernetes is a container orchestration platform
- To deploy something on Kubernetes, we need to specify a deployment and a service
- You can use Docker compose and Kind for local experiments
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
## Notes
Add notes from the video (PRs are welcome)
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
| ⚠️ |
The notes are written by the community. If you see an error here, please create a PR with a fix. |
Course platform with deadlines and submission forms for homework assignments and projects • Course Channel on Slack (#course-ml-zoomcamp) • Telegram Announcements • Course Playlist • FAQ • Tweet about the Course
Learn machine learning engineering end-to-end, from core models to deploying real applications.
Build regression and classification models in Python, work with key algorithms like linear/logistic regression, decision trees, and deep learning, and then take them to production using Docker, FastAPI, Kubernetes, and AWS Lambda.
Ready to start? Join the 2025 cohort or start with Module 1
## Syllabus | Module | Description | Topics | |--------|-------------|--------| | **[Module 1: Introduction to Machine Learning](01-intro/)** | Learn the fundamentals: what ML is, when to use it, and how to approach ML problems using the CRISP-DM framework. | • ML vs rule-based systems
A local deployment architecture using Kubernetes with Kind from one of the students' projects
Some of the course projects from past cohorts: - [Blood cell classifier for cancer prediction](https://datatalks.club/blog/how-to-build-blood-cell-classifier-for-cancer-prediction-case-study-from-ml-zoomcamp.html): an end-to-end tool that segments and classifies blood cells from microscope images to assist in detecting signs of acute lymphoblastic leukemia (ALL) - [Waste classifier](https://datatalks.club/blog/how-to-build-waste-classifier-case-study-from-ml-zoomcamp.html): an Xception-based image classifier on ~15,000 waste images, reaching 93.3% test accuracy, and serving predictions via a Flask API packaged in Docker ## Certificate
Machine Learning Zoomcamp certificate awarded upon successful completion
To receive a certificate, you'll need to complete and submit two projects: 1. **Complete two projects**: Submit either a midterm project and a capstone project, OR two capstone projects 2. **Submit on time**: Meet the project submission deadlines to qualify for certification 3. **Peer review**: Evaluate and provide feedback on 3 fellow students' projects during the peer review process ## Testimonials > Machine Learning Zoomcamp was exhaustive, with very comprehensive content that covered concepts in depth. You can learn everything from the simplest concepts to preparing and deploying an ML model for production. Additionally, the entire community behind this course is highly participative and collaborative. I would like to thank Alexey Grigorev for all the knowledge he shared with us and his team for providing the support we needed to solve each problem we faced. > > - [Alexander Daniel Rios](https://www.linkedin.com/in/alexander-daniel-rios) ([Source](https://www.linkedin.com/posts/alexander-daniel-rios_mlzoomcamp-activity-7295527609239584768-TWHh)) > Machine Learning Zoomcamp has been an incredible journey, thanks to the expert guidance of Alexey Grigorev. Hugely grateful to Alexey, Timur, and the entire DataTalksClub team for this course, and to my cohort batchmates for the invaluable support that enriched my learning experience. I’m thankful for this programme, which provided challenging coursework that is taught in a very structured and lucid way. The timely assignments & hands-on projects instill the sense of timely delivery, besides equipping us with practical acumen to solve real-life problems. > > - [Siddhartha Gogoi](https://www.linkedin.com/in/siddhartha-gogoi) ([Source](https://www.linkedin.com/posts/activity-7299906113997524994-R-oD?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)) > Balancing the intensive Machine Learning Zoomcamp with my other engagements was no easy task, but the experience deepened my expertise in machine learning engineering, reinforced my passion for ML deployment and cloud technologies, and strengthened my resilience in handling real-world ML challenges. Thank you, Alexey Grigorev, for this course! > > - [Patrick Edosoma](https://www.linkedin.com/in/patrickedosoma) ([Source](https://www.linkedin.com/posts/patrickedosoma_machinelearning-mlzoomcamp-datascience-activity-7299090071201193985-JyuC)) > Highly recommend the ML Zoomcamp for anyone wanting a structured path to production-ready machine learning. A big thank you - Alexey Grigorev and to the team at DataTalksClub for providing such a well-structured and engaging course. > > - [Abdiaziz Mohamed](https://www.linkedin.com/in/abdiaziz-mohamed) ([Source 1](https://www.linkedin.com/posts/abdiaziz-mohamed_machinelearning-deployment-docker-activity-7257086439333523456-CyK4), [Source 2](https://www.linkedin.com/posts/abdiaziz-mohamed_machinelearningzoomcamp-machinelearning-kubernetes-activity-7277039208072904704-OAiY?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)) > A huge thank you to Alexey Grigoriev for creating such an amazing course—and making it free! It’s truly inspiring. > > - [Guilherme Pereira](https://www.linkedin.com/in/guilherme-torres-pereira) ([Source](https://www.linkedin.com/posts/guilherme-torres-pereira_alexeygrigoriev-mlzoomcamp-machinelearning-activity-7396336012018356224-sK27)) > Huge thanks to Alexey Grigorev and the DataTalksClub community for the incredible support and clarity throughout. The open-source spirit and collaborative notes made the learning experience even richer. > > - [Rajendra Rawale](https://www.linkedin.com/in/rajendra1x) ([Source](https://www.linkedin.com/posts/rajendra1x_machinelearning-mlzoomcamp-datatalksclub-activity-7378450260999852032-V5Z1))Ready to start? Join the 2025 cohort or start with Module 1
## Community & Getting Help ### Where to Get Help - **Slack**: [`#course-ml-zoomcamp`](https://app.slack.com/client/T01ATQK62F8/C0288NJ5XSA) channel - **FAQ**: [Common questions and answers](https://datatalks.club/faq/machine-learning-zoomcamp.html) - **Study Groups**: Connect with other learners ### Community Guidelines - Check the [FAQ](https://datatalks.club/faq/machine-learning-zoomcamp.html) first - Follow our [question guidelines](asking-questions.md) - Be helpful and respectful - Share your learning journey ### Learning in Public We encourage sharing your progress! Write blog posts, create videos, post on social media with #mlzoomcamp. It helps you learn better and builds your professional network. **Bonus**: You can earn extra points for sharing your learning experience publicly. Learn more: [Learning in Public](learning-in-public.md) ## Sponsors Interested in sponsoring? Contact [alexey@datatalks.club](mailto:alexey@datatalks.club). ## About DataTalks.Club
DataTalks.Club is a global online community of data enthusiasts. It's a place to discuss data, learn, share knowledge, ask and answer questions, and support each other.
Website • Join Slack Community • Newsletter • Upcoming Events • YouTube • GitHub • LinkedIn • Twitter
All the activity at DataTalks.Club mainly happens on [Slack](https://datatalks.club/slack.html). We post updates there and discuss different aspects of data, career questions, and more. ================================================ FILE: after-sign-up.md ================================================ ## Thank you! Thanks for signing up for the course. The process of adding you to the mailing list is not automated yet, but you will hear from us closer to the course start. To make sure you don't miss any announcements - Register in [DataTalks.Club's Slack](https://datatalks.club/slack.html) and join the [`#course-ml-zoomcamp`](https://app.slack.com/client/T01ATQK62F8/C0288NJ5XSA) channel - Join the [course Telegram channel with announcements](https://t.me/mlzoomcamp) - [Tweet about the course!](https://clicktotweet.com/XZ6b9) - Subscribe to [DataTalks.Club's YouTube channel](https://www.youtube.com/c/DataTalksClub) and check [the course playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR) - Check our [Technical FAQ](https://datatalks.club/faq/machine-learning-zoomcamp.html) if you have questions See you in September! ================================================ FILE: article/README.md ================================================ ## Article (optional) The best way to learn about something - to teach it. In this part of the course, we'll explore some topics that weren't covered in the course. You'll need to learn about this topic and then write about what you learned in an article. (Of course if you already know something and just want to share your knowledge with us - it's also fine and more than welcome.) ## Topics You can write about pretty much anything that wasn't covered in the course, but related to machine learning and/or engineering. Here are some ideas: * Areas of Machine Learning: * Time series modelling * Machine learning for textual data * Recommender systems * Clustering * Dimensionality reduction * GANs * Tools similar to what we covered: * Using poetry instead of pipenv * LightGBM and catboost instead of xgboost * Using PyTorch instead of Tensorflow * PyCaret * FastAI * TorchServe instead of TF Serving * Some other interesting tools: * Kedro * Pandera * Evaluation metrics: * Precision/Recall curves * Pretty much everything from "explore more" sections * And more! It can be as broad as you'd like - for example, about time series in general or about some specific methods there (exponential smoothing). If you're not sure if your topic is suitable or not, feel free to ask about it in Slack. ## Requirements For the topic, the only requirement is that it wasn't covered in the course, or not covered with the same amount of details that you want to have. When learning the topic, please take a note of all the resources that you used and include them at the end in the "sources" section. ## Where can I publish it? * Medium * Github * Github pages * Wordpress * Your own blog The top voted articles can be re-publised on the DataTalks.Club website if you'd like. ## Plagiarism * Copying from others fully or in part is not allowed. * If you want to quote something from another article, mark it as a quote explicitly and point to the source immediately after the quote. ## Re-using your own work ### Allowed Re-using your own work is okay if it's something that you did as a part of this course. Examples: * You wanted to deploy something to Heroku, you learned about it and wrote a tutorial * You wanted to learn about PyTorch and used it for your project * You did a project about time series and want to share what you learned ### Not allowed It's not allowed to re-use articles that you wrote before the course started. Examples: * You were taking part in another course and wrote an article for it. Now you want to re-use the same article here > Note: you're more than welcome to share this article with the > community (e.g. in the `#shameless-content` channel in DataTalks.Club), > but this article can't be submitted as an article for ML Zoomcamp. The publication date for the article must be after the start of the course. ## Submitting the results * Submit your article to [`#course-ml-zoomcamp-articles`](https://app.slack.com/client/T01ATQK62F8/C02QXBFS1PU) - just share the link to your article there ## Evaluating We'll use voting for scoring your articles. * Check the articles in the [`#course-ml-zoomcamp-articles`](https://app.slack.com/client/T01ATQK62F8/C02QXBFS1PU) channel and put a :+1: reaction to articles that you liked * The top voted articles will get 20 points ## Past articles ### ML Zoomcamp 2023 See [here](../cohorts/2023/article.md) ### ML Zoomcamp 2022 See [here](../cohorts/2022/article.md) ### ML Zoomcamp 2021 See [here](../cohorts/2021/13-article#articles) ================================================ FILE: asking-questions.md ================================================ ## Asking questions If you have any questions, ask them in the [`#course-ml-zoomcamp`](https://app.slack.com/client/T01ATQK62F8/C0288NJ5XSA) channel in [DataTalks.Club](https://datatalks.club) slack. To keep our discussion in Slack more organized, we ask you to follow these suggestions: * Before asking a question, check [FAQ](https://datatalks.club/faq/machine-learning-zoomcamp.html). * Use threads. When you have a problem, first describe the problem shortly and then put the actual error in the thread - so it doesn't take the entire screen. * Instead of screenshots, it's better to copy-paste the error you're getting in text. Use ` ``` ` for formatting your code. It's very difficult to read text from screenshots. * Please don't take pictures of your code with a phone. It's even harder to read. Follow the previous suggestion, and in rare cases when you need to show what happens on your screen, take a screenshot. * You don't need to tag the instructors when you have a problem. We will see it eventually. * If somebody helped you with your problem and it's not in [FAQ](https://datatalks.club/faq/machine-learning-zoomcamp.html), please add it there. It'll help other students. ================================================ FILE: bento.md ================================================ ## 7. [Production-Ready Machine Learning (Bento ML)](07-bentoml-production/) - 7.1 [Intro/Session Overview](07-bentoml-production/01-intro.md) - 7.2 [Building Your Prediction Service with BentoML](07-bentoml-production/02-build-bento-service.md) - 7.3 [Deploying Your Prediction Service](07-bentoml-production/03-deploy-bento-service.md) - 7.4 [Sending, Receiving and Validating Data](07-bentoml-production/04-validation.md) - 7.5 [High-Performance Serving](07-bentoml-production/05-high-performance.md) - 7.6 [Bento Production Deployment](07-bentoml-production/06-production-deployment.md) - 7.7 [(Optional) Advanced Example: Deploying Stable Diffusion Model](07-bentoml-production/07-stable-diffusion.md) - 7.8 [Summary](07-bentoml-production/08-summary.md) - 7.9 [Homework](07-bentoml-production/homework.md) ## Supporters and partners Thanks to the course sponsors for making it possible to run this course
| \n", " | latitude | \n", "longitude | \n", "price | \n", "minimum_nights | \n", "number_of_reviews | \n", "reviews_per_month | \n", "calculated_host_listings_count | \n", "availability_365 | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "40.64749 | \n", "-73.97237 | \n", "149 | \n", "1 | \n", "9 | \n", "0.21 | \n", "6 | \n", "365 | \n", "
| 1 | \n", "40.75362 | \n", "-73.98377 | \n", "225 | \n", "1 | \n", "45 | \n", "0.38 | \n", "2 | \n", "355 | \n", "
| 2 | \n", "40.80902 | \n", "-73.94190 | \n", "150 | \n", "3 | \n", "0 | \n", "NaN | \n", "1 | \n", "365 | \n", "
| 3 | \n", "40.68514 | \n", "-73.95976 | \n", "89 | \n", "1 | \n", "270 | \n", "4.64 | \n", "1 | \n", "194 | \n", "
| 4 | \n", "40.79851 | \n", "-73.94399 | \n", "80 | \n", "10 | \n", "9 | \n", "0.10 | \n", "1 | \n", "0 | \n", "
| 5 | \n", "40.74767 | \n", "-73.97500 | \n", "200 | \n", "3 | \n", "74 | \n", "0.59 | \n", "1 | \n", "129 | \n", "
| 6 | \n", "40.68688 | \n", "-73.95596 | \n", "60 | \n", "45 | \n", "49 | \n", "0.40 | \n", "1 | \n", "0 | \n", "
| 7 | \n", "40.76489 | \n", "-73.98493 | \n", "79 | \n", "2 | \n", "430 | \n", "3.47 | \n", "1 | \n", "220 | \n", "
| 8 | \n", "40.80178 | \n", "-73.96723 | \n", "79 | \n", "2 | \n", "118 | \n", "0.99 | \n", "1 | \n", "0 | \n", "
| 9 | \n", "40.71344 | \n", "-73.99037 | \n", "150 | \n", "1 | \n", "160 | \n", "1.33 | \n", "4 | \n", "188 | \n", "
| \n", " | count | \n", "mean | \n", "std | \n", "min | \n", "0% | \n", "10% | \n", "25% | \n", "50% | \n", "75% | \n", "90% | \n", "95% | \n", "99% | \n", "max | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| latitude | \n", "48895.0 | \n", "40.728949 | \n", "0.054530 | \n", "40.49979 | \n", "40.49979 | \n", "40.66799 | \n", "40.69010 | \n", "40.72307 | \n", "40.763115 | \n", "40.80489 | \n", "40.825643 | \n", "40.864661 | \n", "40.91306 | \n", "
| longitude | \n", "48895.0 | \n", "-73.952170 | \n", "0.046157 | \n", "-74.24442 | \n", "-74.24442 | \n", "-73.99669 | \n", "-73.98307 | \n", "-73.95568 | \n", "-73.936275 | \n", "-73.90781 | \n", "-73.865771 | \n", "-73.776920 | \n", "-73.71299 | \n", "
| price | \n", "48895.0 | \n", "152.720687 | \n", "240.154170 | \n", "0.00000 | \n", "0.00000 | \n", "49.00000 | \n", "69.00000 | \n", "106.00000 | \n", "175.000000 | \n", "269.00000 | \n", "355.000000 | \n", "799.000000 | \n", "10000.00000 | \n", "
| minimum_nights | \n", "48895.0 | \n", "7.029962 | \n", "20.510550 | \n", "1.00000 | \n", "1.00000 | \n", "1.00000 | \n", "1.00000 | \n", "3.00000 | \n", "5.000000 | \n", "28.00000 | \n", "30.000000 | \n", "45.000000 | \n", "1250.00000 | \n", "
| number_of_reviews | \n", "48895.0 | \n", "23.274466 | \n", "44.550582 | \n", "0.00000 | \n", "0.00000 | \n", "0.00000 | \n", "1.00000 | \n", "5.00000 | \n", "24.000000 | \n", "70.00000 | \n", "114.000000 | \n", "214.000000 | \n", "629.00000 | \n", "
| reviews_per_month | \n", "38843.0 | \n", "1.373221 | \n", "1.680442 | \n", "0.01000 | \n", "0.01000 | \n", "0.06000 | \n", "0.19000 | \n", "0.72000 | \n", "2.020000 | \n", "3.63000 | \n", "4.640000 | \n", "7.195800 | \n", "58.50000 | \n", "
| calculated_host_listings_count | \n", "48895.0 | \n", "7.143982 | \n", "32.952519 | \n", "1.00000 | \n", "1.00000 | \n", "1.00000 | \n", "1.00000 | \n", "1.00000 | \n", "2.000000 | \n", "5.00000 | \n", "15.000000 | \n", "232.000000 | \n", "327.00000 | \n", "
| availability_365 | \n", "48895.0 | \n", "112.781327 | \n", "131.622289 | \n", "0.00000 | \n", "0.00000 | \n", "0.00000 | \n", "0.00000 | \n", "45.00000 | \n", "227.000000 | \n", "337.00000 | \n", "359.000000 | \n", "365.000000 | \n", "365.00000 | \n", "
| \n", " | neighbourhood_group | \n", "latitude | \n", "longitude | \n", "room_type | \n", "price | \n", "minimum_nights | \n", "number_of_reviews | \n", "reviews_per_month | \n", "calculated_host_listings_count | \n", "availability_365 | \n", "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Brooklyn | \n", "40.64749 | \n", "-73.97237 | \n", "Private room | \n", "149 | \n", "1 | \n", "9 | \n", "0.21 | \n", "6 | \n", "365 | \n", "
| 1 | \n", "Manhattan | \n", "40.75362 | \n", "-73.98377 | \n", "Entire home/apt | \n", "225 | \n", "1 | \n", "45 | \n", "0.38 | \n", "2 | \n", "355 | \n", "
| 2 | \n", "Manhattan | \n", "40.80902 | \n", "-73.94190 | \n", "Private room | \n", "150 | \n", "3 | \n", "0 | \n", "NaN | \n", "1 | \n", "365 | \n", "
| 3 | \n", "Brooklyn | \n", "40.68514 | \n", "-73.95976 | \n", "Entire home/apt | \n", "89 | \n", "1 | \n", "270 | \n", "4.64 | \n", "1 | \n", "194 | \n", "
| 4 | \n", "Manhattan | \n", "40.79851 | \n", "-73.94399 | \n", "Entire home/apt | \n", "80 | \n", "10 | \n", "9 | \n", "0.10 | \n", "1 | \n", "0 | \n", "
| 5 | \n", "Manhattan | \n", "40.74767 | \n", "-73.97500 | \n", "Entire home/apt | \n", "200 | \n", "3 | \n", "74 | \n", "0.59 | \n", "1 | \n", "129 | \n", "
| 6 | \n", "Brooklyn | \n", "40.68688 | \n", "-73.95596 | \n", "Private room | \n", "60 | \n", "45 | \n", "49 | \n", "0.40 | \n", "1 | \n", "0 | \n", "
| 7 | \n", "Manhattan | \n", "40.76489 | \n", "-73.98493 | \n", "Private room | \n", "79 | \n", "2 | \n", "430 | \n", "3.47 | \n", "1 | \n", "220 | \n", "
| 8 | \n", "Manhattan | \n", "40.80178 | \n", "-73.96723 | \n", "Private room | \n", "79 | \n", "2 | \n", "118 | \n", "0.99 | \n", "1 | \n", "0 | \n", "
| 9 | \n", "Manhattan | \n", "40.71344 | \n", "-73.99037 | \n", "Entire home/apt | \n", "150 | \n", "1 | \n", "160 | \n", "1.33 | \n", "4 | \n", "188 | \n", "
| \n", " | neighbourhood_group | \n", "room_type | \n", "
|---|---|---|
| count | \n", "48895 | \n", "48895 | \n", "
| unique | \n", "5 | \n", "3 | \n", "
| top | \n", "Manhattan | \n", "Entire home/apt | \n", "
| freq | \n", "21661 | \n", "25409 | \n", "
| \n", " | latitude | \n", "longitude | \n", "minimum_nights | \n", "number_of_reviews | \n", "reviews_per_month | \n", "calculated_host_listings_count | \n", "availability_365 | \n", "
|---|---|---|---|---|---|---|---|
| count | \n", "48895.000000 | \n", "48895.000000 | \n", "48895.000000 | \n", "48895.000000 | \n", "48895.000000 | \n", "48895.000000 | \n", "48895.000000 | \n", "
| mean | \n", "40.728949 | \n", "-73.952170 | \n", "7.029962 | \n", "23.274466 | \n", "1.090910 | \n", "7.143982 | \n", "112.781327 | \n", "
| std | \n", "0.054530 | \n", "0.046157 | \n", "20.510550 | \n", "44.550582 | \n", "1.597283 | \n", "32.952519 | \n", "131.622289 | \n", "
| min | \n", "40.499790 | \n", "-74.244420 | \n", "1.000000 | \n", "0.000000 | \n", "0.000000 | \n", "1.000000 | \n", "0.000000 | \n", "
| 25% | \n", "40.690100 | \n", "-73.983070 | \n", "1.000000 | \n", "1.000000 | \n", "0.040000 | \n", "1.000000 | \n", "0.000000 | \n", "
| 50% | \n", "40.723070 | \n", "-73.955680 | \n", "3.000000 | \n", "5.000000 | \n", "0.370000 | \n", "1.000000 | \n", "45.000000 | \n", "
| 75% | \n", "40.763115 | \n", "-73.936275 | \n", "5.000000 | \n", "24.000000 | \n", "1.580000 | \n", "2.000000 | \n", "227.000000 | \n", "
| max | \n", "40.913060 | \n", "-73.712990 | \n", "1250.000000 | \n", "629.000000 | \n", "58.500000 | \n", "327.000000 | \n", "365.000000 | \n", "
| \n", " | latitude | \n", "longitude | \n", "minimum_nights | \n", "number_of_reviews | \n", "reviews_per_month | \n", "calculated_host_listings_count | \n", "availability_365 | \n", "
|---|---|---|---|---|---|---|---|
| latitude | \n", "1.000000 | \n", "0.084788 | \n", "0.024869 | \n", "-0.015389 | \n", "-0.018758 | \n", "0.019517 | \n", "-0.010983 | \n", "
| longitude | \n", "0.084788 | \n", "1.000000 | \n", "-0.062747 | \n", "0.059094 | \n", "0.138516 | \n", "-0.114713 | \n", "0.082731 | \n", "
| minimum_nights | \n", "0.024869 | \n", "-0.062747 | \n", "1.000000 | \n", "-0.080116 | \n", "-0.124905 | \n", "0.127960 | \n", "0.144303 | \n", "
| number_of_reviews | \n", "-0.015389 | \n", "0.059094 | \n", "-0.080116 | \n", "1.000000 | \n", "0.589407 | \n", "-0.072376 | \n", "0.172028 | \n", "
| reviews_per_month | \n", "-0.018758 | \n", "0.138516 | \n", "-0.124905 | \n", "0.589407 | \n", "1.000000 | \n", "-0.047312 | \n", "0.163732 | \n", "
| calculated_host_listings_count | \n", "0.019517 | \n", "-0.114713 | \n", "0.127960 | \n", "-0.072376 | \n", "-0.047312 | \n", "1.000000 | \n", "0.225701 | \n", "
| availability_365 | \n", "-0.010983 | \n", "0.082731 | \n", "0.144303 | \n", "0.172028 | \n", "0.163732 | \n", "0.225701 | \n", "1.000000 | \n", "
| \n", " | MI | \n", "
|---|---|
| room_type | \n", "0.143035 | \n", "
| neighbourhood_group | \n", "0.046513 | \n", "
| \n", " | threshold | \n", "tp | \n", "fp | \n", "fn | \n", "tn | \n", "
|---|---|---|---|---|---|
| 0 | \n", "0.0 | \n", "246 | \n", "645 | \n", "0 | \n", "0 | \n", "
| 10 | \n", "0.1 | \n", "235 | \n", "445 | \n", "11 | \n", "200 | \n", "
| 20 | \n", "0.2 | \n", "215 | \n", "267 | \n", "31 | \n", "378 | \n", "
| 30 | \n", "0.3 | \n", "187 | \n", "162 | \n", "59 | \n", "483 | \n", "
| 40 | \n", "0.4 | \n", "140 | \n", "100 | \n", "106 | \n", "545 | \n", "
| 50 | \n", "0.5 | \n", "114 | \n", "64 | \n", "132 | \n", "581 | \n", "
| 60 | \n", "0.6 | \n", "78 | \n", "38 | \n", "168 | \n", "607 | \n", "
| 70 | \n", "0.7 | \n", "37 | \n", "16 | \n", "209 | \n", "629 | \n", "
| 80 | \n", "0.8 | \n", "21 | \n", "5 | \n", "225 | \n", "640 | \n", "
| 90 | \n", "0.9 | \n", "6 | \n", "2 | \n", "240 | \n", "643 | \n", "
| 100 | \n", "1.0 | \n", "0 | \n", "0 | \n", "246 | \n", "645 | \n", "
| \n", " | feature | \n", "importance | \n", "
|---|---|---|
| 0 | \n", "availability_365 | \n", "0.076503 | \n", "
| 1 | \n", "calculated_host_listings_count | \n", "0.030657 | \n", "
| 2 | \n", "latitude | \n", "0.152959 | \n", "
| 3 | \n", "longitude | \n", "0.152743 | \n", "
| 4 | \n", "minimum_nights | \n", "0.053463 | \n", "
| 5 | \n", "neighbourhood_group=Bronx | \n", "0.000361 | \n", "
| 6 | \n", "neighbourhood_group=Brooklyn | \n", "0.001282 | \n", "
| 7 | \n", "neighbourhood_group=Manhattan | \n", "0.034000 | \n", "
| 8 | \n", "neighbourhood_group=Queens | \n", "0.000973 | \n", "
| 9 | \n", "neighbourhood_group=Staten Island | \n", "0.000067 | \n", "
| 10 | \n", "number_of_reviews | \n", "0.043840 | \n", "
| 11 | \n", "reviews_per_month | \n", "0.052207 | \n", "
| 12 | \n", "room_type=Entire home/apt | \n", "0.391899 | \n", "
| 13 | \n", "room_type=Private room | \n", "0.004244 | \n", "
| 14 | \n", "room_type=Shared room | \n", "0.004801 | \n", "
| \n", " | feature | \n", "importance | \n", "
|---|---|---|
| 12 | \n", "room_type=Entire home/apt | \n", "0.391899 | \n", "
| 2 | \n", "latitude | \n", "0.152959 | \n", "
| 3 | \n", "longitude | \n", "0.152743 | \n", "
| 0 | \n", "availability_365 | \n", "0.076503 | \n", "
| 4 | \n", "minimum_nights | \n", "0.053463 | \n", "
[Notebook with solution](homework-6-solution.ipynb)
## Loading the data
* Use only the following columns:
* `'neighbourhood_group',`
* `'room_type',`
* `'latitude',`
* `'longitude',`
* `'minimum_nights',`
* `'number_of_reviews','reviews_per_month',`
* `'calculated_host_listings_count',`
* `'availability_365',`
* `'price'`
* Fill NAs with 0
* Apply the log tranform to `price`
* Do train/validation/test split with 60%/20%/20% distribution.
* Use the `train_test_split` function and set the `random_state` parameter to 1
* Use `DictVectorizer` to turn the dataframe into matrices
## Question 1
Let's train a decision tree regressor to predict the price variable.
* Train a model with `max_depth=1`
Which feature is used for splitting the data?
* `room_type`
* `neighbourhood_group`
* `number_of_reviews`
* `reviews_per_month`
## Question 2
Train a random forest model with these parameters:
* `n_estimators=10`
* `random_state=1`
* `n_jobs=-1` (optional - to make training faster)
What's the RMSE of this model on validation?
* 0.059
* 0.259
* 0.459
* 0.659
## Question 3
Now let's experiment with the `n_estimators` parameter
* Try different values of this parameter from 10 to 200 with step 10
* Set `random_state` to `1`
* Evaluate the model on the validation dataset
After which value of `n_estimators` does RMSE stop improving?
- 10
- 50
- 70
- 120
## Question 4
Let's select the best `max_depth`:
* Try different values of `max_depth`: `[10, 15, 20, 25]`
* For each of these values, try different values of `n_estimators` from 10 till 200 (with step 10)
* Fix the random seed: `random_state=1`
What's the best `max_depth`:
* 10
* 15
* 20
* 25
**Bonus question (not graded):**
Will the answer be different if we change the seed for the model?
## Question 5
We can extract feature importance information from tree-based models.
At each step of the decision tree learning algorith, it finds the best split.
When doint it, we can calculate "gain" - the reduction in impurity before and after the split.
This gain is quite useful in understanding what are the imporatant features
for tree-based models.
In Scikit-Learn, tree-based models contain this information in the
[`feature_importances_`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html#sklearn.ensemble.RandomForestRegressor.feature_importances_)
field.
For this homework question, we'll find the most important feature:
* Train the model with these parametes:
* `n_estimators=10`,
* `max_depth=20`,
* `random_state=1`,
* `n_jobs=-1` (optional)
* Get the feature importance information from this model
What's the most important feature?
* `neighbourhood_group=Manhattan`
* `room_type=Entire home/apt`
* `longitude`
* `latitude`
## Question 6
Now let's train an XGBoost model! For this question, we'll tune the `eta` parameter
* Install XGBoost
* Create DMatrix for train and validation
* Create a watchlist
* Train a model with these parameters for 100 rounds:
```
xgb_params = {
'eta': 0.3,
'max_depth': 6,
'min_child_weight': 1,
'objective': 'reg:squarederror',
'nthread': 8,
'seed': 1,
'verbosity': 1,
}
```
Now change `eta` first to `0.1` and then to `0.01`
Which eta leads to the best RMSE score on the validation dataset?
* 0.3
* 0.1
* 0.01
## Submit the results
Submit your results here: https://forms.gle/wQgFkYE6CtdDed4w8
It's possible that your answers won't match exactly. If it's the case, select the closest one. If it's exactly in between two options, select the higher value.
## Deadline
The deadline for submitting is 20 October 2021, 17:00 CET (Wednesday). After that, the form will be closed.
## Navigation
* [Machine Learning Zoomcamp course](../)
* [Session 6: Decision Trees and Ensemble Learning](./)
* Previous: [Explore more](11-explore-more.md)
================================================
FILE: cohorts/2021/07-midterm-project/README.md
================================================
## 7. Midterm Project
The idea is that you now apply everything we learned so far yourself.
That will include:
* Thinking of a problem that's interesting for you and finding a dataset for that
* Describing this problem and explaining how a model could be used
* Preparing the data and doing EDA, analyzing important features
* Training multiple models, tuning their performance and selecting the best model
* Exporting the notebook into a script
* Putting your model into a web service and deploying it locally with docker
* Bonus points for deploying the service to the cloud
### Datasets
* https://www.kaggle.com/datasets and https://www.kaggle.com/competitions
* https://archive.ics.uci.edu/ml/index.php
* https://data.europa.eu/en
* https://www.openml.org/search?type=data
* Add more data here!
### Deliverables
For this project, you repository/folder should contain the following:
* `README.md` with
* Description of the problem
* Instructions on how to run the project
* Notebook (suggested name - `notebook.ipynb`) with
* Data preparation and data clearning
* EDA, feature importance analysis
* Model selection process and parameter tuning
* Script `train.py` (suggested name)
* Training the final model
* Saving it to a file (e.g. pickle)
* Script `predict.py` (suggested name)
* Loading the model
* Serving it via a web serice (e.g. with Flask)
* `Pipenv` and `Pipenv.lock`
* or equivalents: conda environment file, requirements.txt or pyproject.toml
* `Dockerfile` for running the service
### Submit the results
Submit your results here: https://forms.gle/5JLFUwwXaUDssVKt7
### Deadline
The deadline for submitting is 3 November 2021 (Wednesday), 12:00 (noon) CET. After that, the form will be closed.
### FAQ
**Q**: Can I use poetry / virtual env for managing dependencies; catboost for boosting and FastAPI for creating a web service?
> Yes, you can use any library you want. But please make sure to document everything and clearly explain what you use.
> Think of your peers who will review it - they don't necessarily know what these libraries are.
> Please give them enough context to understand your project.
**Q**: Can multiple people use the same dataset?
> Yes, there's no way to control it or enforce. So it's totally okay if you and somebody else use the same dataset.
## Peer reviewing
To evaluate the projects, we'll use peer reviewing. This is a great opportunity for your to learn from each other.
* To get points for your project, your need to evaluate 3 projects of your peers
* You get 3 extra point for each evaluation
To find the projects you need to review, use [this table](https://docs.google.com/spreadsheets/d/e/2PACX-1vRo05lqGFkl7Jtt4o9Dqqk8JEA-U956-sxJVT-klWrKp8nTvboOkyyYVH-Z4sccFdpvfZhd40eGhU2J/pubhtml)
* Hash your email
* Find your email in the `reviewer_hash` column
* You'll have three other projects there - that's what you'll need to evaluate
Function for computing the hash of your email:
```python
from hashlib import sha1
def compute_hash(email):
return sha1(email.lower().encode('utf-8')).hexdigest()
```
Tip: you can use https://nbviewer.org/ to render notebooks if GitHub doesn't work
### Evaluation Criteria
The project will be evaluated using these criteria:
* Problem description
* EDA
* Model training
* Exporting notebook to script
* Model deployment
* Dependency and environment management
* Containerization
* Cloud deployment
[Detailed criteria](https://docs.google.com/spreadsheets/d/e/2PACX-1vQo-cOOGMA-ddbp6FgxusNBjS_HOmWaOYtvO7z-wk_TcCnPOBAza9s8Uj_eqfKGadoU0741cCGd95qI/pubhtml)
### Submitting evaluations
* To submit the evaluations, use this form: https://forms.gle/AMFvMNnfwEeTQxaj8
* You will need to submit this form three times - one for each evaluated project
* Deadline: 10 November (Wednesday), 12:00 (noon) CET. After that, the form will be closed.
### FAQ
**Q**: Do I have to run the code and make sure it works?
> It's recommended that you do that, but you don't _have_ to do it.
**Q**: What if I see an error? What if I run something and it doesn't work?
> But if you spot an error somewhere and you see that the code clearly doesn't work, then you
> give 0 points to the respective criterium. E.g. if you see an error in Dockerfile,
> then you give 0 points to the "containerization" dimension.
**Q**: Somebody submitted a project they did as a part of some other course. What to do?
> That's not good, but we didn't explicitly say it's not allowed. We'll change that for the next project,
> but for the midterm project, you can evaluate it as if it was specifically made for this course.
**Q**: Somebody just copied a Kaggle Kernel (or some other publicly available project) and
submitted this as their project.
> That's definitely not good. For the next project, we'll have an explicit policy that forbids
> doing it. For this project, you should evaluate the copied parts as 0. For example,
> if the model trianing part was copied, but the deployment wasn't, the model training part gets 0
> points, but the model deployment part gets the points.
## Office Hours 2021
We discussed the project in detail in three office hours sessions
### Office Hours Week #7
* [Video](https://www.youtube.com/watch?v=wWBm6MHu5u8)
### Office Hours Week #8
* [Video](https://www.youtube.com/watch?v=3bo7I9LaN7o)
* [Notebook](week8-office-hours.ipynb)
### Office Hours Week #9
* [Video](https://www.youtube.com/watch?v=yZ15WyKb5o4)
* [Notebook](week9-office-hours.ipynb)
### Office Hours Week #10
* [Video](https://www.youtube.com/watch?v=jT0JTlPsAQ0)
* [Notebook](week10-office-hours.ipynb)
## Community notes
Did you take notes? You can share them here (or in each unit separately)
* [Alvaro Navas' Notes](https://github.com/ziritrion/ml-zoomcamp/blob/main/notes/07_misc.md)
* Add your notes here
================================================
FILE: cohorts/2021/07-midterm-project/week10-office-hours.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"id": "d3fb0207",
"metadata": {},
"source": [
"- KNN\n",
"- Scikit-learn pipelines"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "c540d91d",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd \n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "1a2b1b5a",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv('AB_NYC_2019.csv', nrows=4000)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "7527ae35",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.neighbors import KNeighborsRegressor"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "afc0a7cd",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "ae4951cc",
"metadata": {},
"outputs": [],
"source": [
"df_train, df_val = train_test_split(df)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "bf750d64",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"3000"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(df_train)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "b19e5449",
"metadata": {},
"outputs": [],
"source": [
"X_train = df_train[['latitude', 'longitude']].fillna(0).values\n",
"y_train = df_train.price.values"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "6f47164b",
"metadata": {},
"outputs": [],
"source": [
"X_val = df_val[['latitude', 'longitude']].fillna(0).values\n",
"y_val = df_val.price.values"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "66a898c3",
"metadata": {},
"outputs": [],
"source": [
"knn = KNeighborsRegressor(n_neighbors=5)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9efc2b5e",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 13,
"id": "3660cd30",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"KNeighborsRegressor()"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"knn.fit(X_train, y_train)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "48dcfd9c",
"metadata": {},
"outputs": [],
"source": [
"y_pred = knn.predict(X_val)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "500ec7c5",
"metadata": {},
"outputs": [],
"source": [
"for i in range(1, 50):\n",
" knn = KNeighborsRegressor(n_neighbors=i)\n",
" knn.fit(X_train, y_train)\n",
" \n",
" y_pred = knn.predict(X_val)\n",
" # rmse"
]
},
{
"cell_type": "markdown",
"id": "33e96506",
"metadata": {},
"source": [
"## Pipelines"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "8ae907c3",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.feature_extraction.text import CountVectorizer\n",
"from sklearn.preprocessing import OneHotEncoder"
]
},
{
"cell_type": "code",
"execution_count": 61,
"id": "e4a80087",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv('AB_NYC_2019.csv', nrows=4000)"
]
},
{
"cell_type": "code",
"execution_count": 62,
"id": "802b904b",
"metadata": {},
"outputs": [],
"source": [
"df['number_of_reviews'] = df['number_of_reviews'].fillna(0)"
]
},
{
"cell_type": "code",
"execution_count": 63,
"id": "b43e2a7d",
"metadata": {},
"outputs": [],
"source": [
"df.name = df.name.fillna('')\n",
"df.neighbourhood_group = df.neighbourhood_group.str.lower().str.replace(' ', '_').fillna('NA')\n",
"df.neighbourhood = df.neighbourhood.str.lower().str.replace(' ', '_').fillna('NA')\n",
"df.room_type = df.room_type.str.lower().str.replace(' ', '_').fillna('NA')"
]
},
{
"cell_type": "code",
"execution_count": 64,
"id": "a14c0a72",
"metadata": {},
"outputs": [],
"source": [
"numerical = ['latitude',\n",
" 'longitude',\n",
" 'minimum_nights',\n",
" 'number_of_reviews',\n",
" 'reviews_per_month',\n",
" 'calculated_host_listings_count',\n",
" 'availability_365']"
]
},
{
"cell_type": "code",
"execution_count": 74,
"id": "be5622da",
"metadata": {},
"outputs": [],
"source": [
"df[numerical] = df[numerical].fillna(0)"
]
},
{
"cell_type": "code",
"execution_count": 75,
"id": "cdc930ca",
"metadata": {},
"outputs": [],
"source": [
"transformations = [\n",
" ('numerical', 'passthrough', numerical),\n",
" ('categories', OneHotEncoder(dtype='int32'),\n",
" ['neighbourhood_group', 'neighbourhood', 'room_type']),\n",
" ('name', CountVectorizer(min_df=100, dtype='int32'), 'name')\n",
"]\n",
"\n",
"tranformer = ColumnTransformer(transformations, remainder='drop')\n"
]
},
{
"cell_type": "code",
"execution_count": 76,
"id": "fad529b2",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 77,
"id": "2d53e04d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"ColumnTransformer(transformers=[('numerical', 'passthrough',\n",
" ['latitude', 'longitude', 'minimum_nights',\n",
" 'number_of_reviews', 'reviews_per_month',\n",
" 'calculated_host_listings_count',\n",
" 'availability_365']),\n",
" ('categories', OneHotEncoder(dtype='int32'),\n",
" ['neighbourhood_group', 'neighbourhood',\n",
" 'room_type']),\n",
" ('name',\n",
" CountVectorizer(dtype='int32', min_df=100),\n",
" 'name')])"
]
},
"execution_count": 77,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tranformer.fit(df)"
]
},
{
"cell_type": "code",
"execution_count": 79,
"id": "4d79b93b",
"metadata": {},
"outputs": [],
"source": [
"X = tranformer.transform(df)"
]
},
{
"cell_type": "code",
"execution_count": 80,
"id": "55e8021e",
"metadata": {},
"outputs": [],
"source": [
"y = df.price.values"
]
},
{
"cell_type": "code",
"execution_count": 81,
"id": "7b9666f2",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.linear_model import LinearRegression"
]
},
{
"cell_type": "code",
"execution_count": 82,
"id": "b4248462",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"LinearRegression()"
]
},
"execution_count": 82,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"lr = LinearRegression()\n",
"lr.fit(X, y)"
]
},
{
"cell_type": "code",
"execution_count": 83,
"id": "36638860",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.pipeline import Pipeline"
]
},
{
"cell_type": "code",
"execution_count": 89,
"id": "9c098d9e",
"metadata": {},
"outputs": [],
"source": [
"transformations = [\n",
" ('numerical', 'passthrough', numerical),\n",
" ('neighbourhood', CountVectorizer(token_pattern='.*', min_df=100, dtype='int32'), 'neighbourhood'),\n",
" ('ng', CountVectorizer(token_pattern='.*', min_df=50, dtype='int32'), 'neighbourhood_group'),\n",
" ('room_type', CountVectorizer(token_pattern='.*', min_df=100, dtype='int32'), 'room_type'),\n",
" ('name', CountVectorizer(min_df=100, dtype='int32'), 'name')\n",
"]\n",
"\n",
"tranformer = ColumnTransformer(transformations, remainder='drop')"
]
},
{
"cell_type": "code",
"execution_count": 90,
"id": "499242d8",
"metadata": {},
"outputs": [],
"source": [
"pipeline = Pipeline([\n",
" ('transormer', tranformer),\n",
" ('lr', LinearRegression())\n",
"])"
]
},
{
"cell_type": "code",
"execution_count": 106,
"id": "d835d0a5",
"metadata": {},
"outputs": [],
"source": [
"pipeline.fit(df, df.price.values);"
]
},
{
"cell_type": "code",
"execution_count": 92,
"id": "a5906d9a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 80.28976523, 340.61684387, 45.10203035, ..., 162.8269308 ,\n",
" 82.08441501, 111.36359434])"
]
},
"execution_count": 92,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pipeline.predict(df)"
]
},
{
"cell_type": "code",
"execution_count": 94,
"id": "94c95b39",
"metadata": {},
"outputs": [],
"source": [
"from sklearn.base import TransformerMixin"
]
},
{
"cell_type": "code",
"execution_count": 95,
"id": "d23e15f3",
"metadata": {},
"outputs": [],
"source": [
"class ConcatenatingTranformer(TransformerMixin):\n",
" \n",
" def fit(self, X, y=None):\n",
" return self\n",
"\n",
" def transform(self, X):\n",
" columns = list(X.columns)\n",
" \n",
" res = ''\n",
" \n",
" for c in columns:\n",
" res = res + ' ' + c + '=' + X[c]\n",
"\n",
" return res.str.strip()"
]
},
{
"cell_type": "code",
"execution_count": 96,
"id": "6dcde520",
"metadata": {},
"outputs": [],
"source": [
"ct = ConcatenatingTranformer()"
]
},
{
"cell_type": "code",
"execution_count": 98,
"id": "61ac82a9",
"metadata": {},
"outputs": [],
"source": [
"p2 = Pipeline([\n",
" ('concatenate', ConcatenatingTranformer()),\n",
" ('vectorize', CountVectorizer(token_pattern='\\S+', min_df=100))\n",
"])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "321d6b31",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 100,
"id": "a543f214",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<4000x17 sparse matrix of type '| \n", " | id | \n", "name | \n", "host_id | \n", "host_name | \n", "neighbourhood_group | \n", "neighbourhood | \n", "latitude | \n", "longitude | \n", "room_type | \n", "price | \n", "minimum_nights | \n", "number_of_reviews | \n", "last_review | \n", "reviews_per_month | \n", "calculated_host_listings_count | \n", "availability_365 | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "2539 | \n", "Clean & quiet apt home by the park | \n", "2787 | \n", "John | \n", "Brooklyn | \n", "Kensington | \n", "40.64749 | \n", "-73.97237 | \n", "Private room | \n", "149 | \n", "1 | \n", "9 | \n", "2018-10-19 | \n", "0.21 | \n", "6 | \n", "365 | \n", "
| 1 | \n", "2595 | \n", "Skylit Midtown Castle | \n", "2845 | \n", "Jennifer | \n", "Manhattan | \n", "Midtown | \n", "40.75362 | \n", "-73.98377 | \n", "Entire home/apt | \n", "225 | \n", "1 | \n", "45 | \n", "2019-05-21 | \n", "0.38 | \n", "2 | \n", "355 | \n", "
| 2 | \n", "3647 | \n", "THE VILLAGE OF HARLEM....NEW YORK ! | \n", "4632 | \n", "Elisabeth | \n", "Manhattan | \n", "Harlem | \n", "40.80902 | \n", "-73.94190 | \n", "Private room | \n", "150 | \n", "3 | \n", "0 | \n", "NaN | \n", "NaN | \n", "1 | \n", "365 | \n", "
| 3 | \n", "3831 | \n", "Cozy Entire Floor of Brownstone | \n", "4869 | \n", "LisaRoxanne | \n", "Brooklyn | \n", "Clinton Hill | \n", "40.68514 | \n", "-73.95976 | \n", "Entire home/apt | \n", "89 | \n", "1 | \n", "270 | \n", "2019-07-05 | \n", "4.64 | \n", "1 | \n", "194 | \n", "
| 4 | \n", "5022 | \n", "Entire Apt: Spacious Studio/Loft by central park | \n", "7192 | \n", "Laura | \n", "Manhattan | \n", "East Harlem | \n", "40.79851 | \n", "-73.94399 | \n", "Entire home/apt | \n", "80 | \n", "10 | \n", "9 | \n", "2018-11-19 | \n", "0.10 | \n", "1 | \n", "0 | \n", "
| \n", " | id | \n", "name | \n", "host_id | \n", "host_name | \n", "neighbourhood_group | \n", "neighbourhood | \n", "latitude | \n", "longitude | \n", "room_type | \n", "price | \n", "minimum_nights | \n", "number_of_reviews | \n", "last_review | \n", "reviews_per_month | \n", "calculated_host_listings_count | \n", "availability_365 | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "2539 | \n", "Clean & quiet apt home by the park | \n", "2787 | \n", "John | \n", "Brooklyn | \n", "Kensington | \n", "40.64749 | \n", "-73.97237 | \n", "Private room | \n", "149 | \n", "1 | \n", "9 | \n", "2018-10-19 | \n", "0.21 | \n", "6 | \n", "365 | \n", "
| 1 | \n", "2595 | \n", "Skylit Midtown Castle | \n", "2845 | \n", "Jennifer | \n", "Manhattan | \n", "Midtown | \n", "40.75362 | \n", "-73.98377 | \n", "Entire home/apt | \n", "225 | \n", "1 | \n", "45 | \n", "2019-05-21 | \n", "0.38 | \n", "2 | \n", "355 | \n", "
| 2 | \n", "3647 | \n", "THE VILLAGE OF HARLEM....NEW YORK ! | \n", "4632 | \n", "Elisabeth | \n", "Manhattan | \n", "Harlem | \n", "40.80902 | \n", "-73.94190 | \n", "Private room | \n", "150 | \n", "3 | \n", "0 | \n", "NaN | \n", "NaN | \n", "1 | \n", "365 | \n", "
| 3 | \n", "3831 | \n", "Cozy Entire Floor of Brownstone | \n", "4869 | \n", "LisaRoxanne | \n", "Brooklyn | \n", "Clinton Hill | \n", "40.68514 | \n", "-73.95976 | \n", "Entire home/apt | \n", "89 | \n", "1 | \n", "270 | \n", "2019-07-05 | \n", "4.64 | \n", "1 | \n", "194 | \n", "
| 4 | \n", "5022 | \n", "Entire Apt: Spacious Studio/Loft by central park | \n", "7192 | \n", "Laura | \n", "Manhattan | \n", "East Harlem | \n", "40.79851 | \n", "-73.94399 | \n", "Entire home/apt | \n", "80 | \n", "10 | \n", "9 | \n", "2018-11-19 | \n", "0.10 | \n", "1 | \n", "0 | \n", "
| \n", " | apt | \n", "by | \n", "castle | \n", "clean | \n", "harlem | \n", "home | \n", "midtown | \n", "new | \n", "of | \n", "park | \n", "quiet | \n", "skylit | \n", "the | \n", "village | \n", "york | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "1 | \n", "1 | \n", "0 | \n", "1 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "1 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "
| 1 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 2 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "1 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "1 | \n", "1 | \n", "
| \n", " | Engine HP | \n", "Engine Cylinders | \n", "
|---|---|---|
| 3912 | \n", "189.0 | \n", "4.0 | \n", "
| 3913 | \n", "218.0 | \n", "4.0 | \n", "
| 3918 | \n", "217.0 | \n", "4.0 | \n", "
| 4216 | \n", "350.0 | \n", "8.0 | \n", "
| 4257 | \n", "400.0 | \n", "6.0 | \n", "
| 4259 | \n", "276.0 | \n", "6.0 | \n", "
| 4262 | \n", "345.0 | \n", "6.0 | \n", "
| 4292 | \n", "257.0 | \n", "4.0 | \n", "
| 4293 | \n", "240.0 | \n", "4.0 | \n", "
| \n", " | longitude | \n", "latitude | \n", "housing_median_age | \n", "total_rooms | \n", "total_bedrooms | \n", "population | \n", "households | \n", "median_income | \n", "median_house_value | \n", "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "-122.23 | \n", "37.88 | \n", "41.0 | \n", "880.0 | \n", "129.0 | \n", "322.0 | \n", "126.0 | \n", "8.3252 | \n", "452600.0 | \n", "
| 1 | \n", "-122.22 | \n", "37.86 | \n", "21.0 | \n", "7099.0 | \n", "1106.0 | \n", "2401.0 | \n", "1138.0 | \n", "8.3014 | \n", "358500.0 | \n", "
| 2 | \n", "-122.24 | \n", "37.85 | \n", "52.0 | \n", "1467.0 | \n", "190.0 | \n", "496.0 | \n", "177.0 | \n", "7.2574 | \n", "352100.0 | \n", "
| 3 | \n", "-122.25 | \n", "37.85 | \n", "52.0 | \n", "1274.0 | \n", "235.0 | \n", "558.0 | \n", "219.0 | \n", "5.6431 | \n", "341300.0 | \n", "
| 4 | \n", "-122.25 | \n", "37.85 | \n", "52.0 | \n", "1627.0 | \n", "280.0 | \n", "565.0 | \n", "259.0 | \n", "3.8462 | \n", "342200.0 | \n", "
| 5 | \n", "-122.25 | \n", "37.85 | \n", "52.0 | \n", "919.0 | \n", "213.0 | \n", "413.0 | \n", "193.0 | \n", "4.0368 | \n", "269700.0 | \n", "
| 6 | \n", "-122.25 | \n", "37.84 | \n", "52.0 | \n", "2535.0 | \n", "489.0 | \n", "1094.0 | \n", "514.0 | \n", "3.6591 | \n", "299200.0 | \n", "
| 7 | \n", "-122.25 | \n", "37.84 | \n", "52.0 | \n", "3104.0 | \n", "687.0 | \n", "1157.0 | \n", "647.0 | \n", "3.1200 | \n", "241400.0 | \n", "
| 8 | \n", "-122.26 | \n", "37.84 | \n", "42.0 | \n", "2555.0 | \n", "665.0 | \n", "1206.0 | \n", "595.0 | \n", "2.0804 | \n", "226700.0 | \n", "
| 9 | \n", "-122.25 | \n", "37.84 | \n", "52.0 | \n", "3549.0 | \n", "707.0 | \n", "1551.0 | \n", "714.0 | \n", "3.6912 | \n", "261100.0 | \n", "
| \n", " | count | \n", "mean | \n", "std | \n", "min | \n", "0% | \n", "10% | \n", "25% | \n", "50% | \n", "75% | \n", "90% | \n", "95% | \n", "99% | \n", "max | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| longitude | \n", "20640.0 | \n", "-119.569704 | \n", "2.003532 | \n", "-124.3500 | \n", "-124.3500 | \n", "-122.2900 | \n", "-121.8000 | \n", "-118.4900 | \n", "-118.01000 | \n", "-117.25000 | \n", "-117.080000 | \n", "-116.29000 | \n", "-114.3100 | \n", "
| latitude | \n", "20640.0 | \n", "35.631861 | \n", "2.135952 | \n", "32.5400 | \n", "32.5400 | \n", "33.6300 | \n", "33.9300 | \n", "34.2600 | \n", "37.71000 | \n", "38.48000 | \n", "38.960000 | \n", "40.62610 | \n", "41.9500 | \n", "
| housing_median_age | \n", "20640.0 | \n", "28.639486 | \n", "12.585558 | \n", "1.0000 | \n", "1.0000 | \n", "13.0000 | \n", "18.0000 | \n", "29.0000 | \n", "37.00000 | \n", "46.00000 | \n", "52.000000 | \n", "52.00000 | \n", "52.0000 | \n", "
| total_rooms | \n", "20640.0 | \n", "2635.763081 | \n", "2181.615252 | \n", "2.0000 | \n", "2.0000 | \n", "941.0000 | \n", "1447.7500 | \n", "2127.0000 | \n", "3148.00000 | \n", "4651.50000 | \n", "6213.200000 | \n", "11212.11000 | \n", "39320.0000 | \n", "
| total_bedrooms | \n", "20433.0 | \n", "537.870553 | \n", "421.385070 | \n", "1.0000 | \n", "1.0000 | \n", "198.0000 | \n", "296.0000 | \n", "435.0000 | \n", "647.00000 | \n", "965.00000 | \n", "1275.400000 | \n", "2221.36000 | \n", "6445.0000 | \n", "
| population | \n", "20640.0 | \n", "1425.476744 | \n", "1132.462122 | \n", "3.0000 | \n", "3.0000 | \n", "510.0000 | \n", "787.0000 | \n", "1166.0000 | \n", "1725.00000 | \n", "2566.00000 | \n", "3288.000000 | \n", "5805.83000 | \n", "35682.0000 | \n", "
| households | \n", "20640.0 | \n", "499.539680 | \n", "382.329753 | \n", "1.0000 | \n", "1.0000 | \n", "184.0000 | \n", "280.0000 | \n", "409.0000 | \n", "605.00000 | \n", "890.00000 | \n", "1162.000000 | \n", "1982.66000 | \n", "6082.0000 | \n", "
| median_income | \n", "20640.0 | \n", "3.870671 | \n", "1.899822 | \n", "0.4999 | \n", "0.4999 | \n", "1.9038 | \n", "2.5634 | \n", "3.5348 | \n", "4.74325 | \n", "6.15921 | \n", "7.300305 | \n", "10.59654 | \n", "15.0001 | \n", "
| median_house_value | \n", "20640.0 | \n", "206855.816909 | \n", "115395.615874 | \n", "14999.0000 | \n", "14999.0000 | \n", "82300.0000 | \n", "119600.0000 | \n", "179700.0000 | \n", "264725.00000 | \n", "376600.00000 | \n", "489810.000000 | \n", "500001.00000 | \n", "500001.0000 | \n", "
| \n", " | longitude | \n", "latitude | \n", "housing_median_age | \n", "total_rooms | \n", "total_bedrooms | \n", "population | \n", "households | \n", "median_income | \n", "median_house_value | \n", "ocean_proximity | \n", "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "-122.23 | \n", "37.88 | \n", "41.0 | \n", "880.0 | \n", "129.0 | \n", "322.0 | \n", "126.0 | \n", "8.3252 | \n", "452600.0 | \n", "NEAR BAY | \n", "
| 1 | \n", "-122.22 | \n", "37.86 | \n", "21.0 | \n", "7099.0 | \n", "1106.0 | \n", "2401.0 | \n", "1138.0 | \n", "8.3014 | \n", "358500.0 | \n", "NEAR BAY | \n", "
| 2 | \n", "-122.24 | \n", "37.85 | \n", "52.0 | \n", "1467.0 | \n", "190.0 | \n", "496.0 | \n", "177.0 | \n", "7.2574 | \n", "352100.0 | \n", "NEAR BAY | \n", "
| 3 | \n", "-122.25 | \n", "37.85 | \n", "52.0 | \n", "1274.0 | \n", "235.0 | \n", "558.0 | \n", "219.0 | \n", "5.6431 | \n", "341300.0 | \n", "NEAR BAY | \n", "
| 4 | \n", "-122.25 | \n", "37.85 | \n", "52.0 | \n", "1627.0 | \n", "280.0 | \n", "565.0 | \n", "259.0 | \n", "3.8462 | \n", "342200.0 | \n", "NEAR BAY | \n", "
| 5 | \n", "-122.25 | \n", "37.85 | \n", "52.0 | \n", "919.0 | \n", "213.0 | \n", "413.0 | \n", "193.0 | \n", "4.0368 | \n", "269700.0 | \n", "NEAR BAY | \n", "
| 6 | \n", "-122.25 | \n", "37.84 | \n", "52.0 | \n", "2535.0 | \n", "489.0 | \n", "1094.0 | \n", "514.0 | \n", "3.6591 | \n", "299200.0 | \n", "NEAR BAY | \n", "
| 7 | \n", "-122.25 | \n", "37.84 | \n", "52.0 | \n", "3104.0 | \n", "687.0 | \n", "1157.0 | \n", "647.0 | \n", "3.1200 | \n", "241400.0 | \n", "NEAR BAY | \n", "
| 8 | \n", "-122.26 | \n", "37.84 | \n", "42.0 | \n", "2555.0 | \n", "665.0 | \n", "1206.0 | \n", "595.0 | \n", "2.0804 | \n", "226700.0 | \n", "NEAR BAY | \n", "
| 9 | \n", "-122.25 | \n", "37.84 | \n", "52.0 | \n", "3549.0 | \n", "707.0 | \n", "1551.0 | \n", "714.0 | \n", "3.6912 | \n", "261100.0 | \n", "NEAR BAY | \n", "
| \n", " | ocean_proximity | \n", "
|---|---|
| count | \n", "20640 | \n", "
| unique | \n", "5 | \n", "
| top | \n", "<1H OCEAN | \n", "
| freq | \n", "9136 | \n", "
| \n", " | longitude | \n", "latitude | \n", "housing_median_age | \n", "total_rooms | \n", "total_bedrooms | \n", "population | \n", "households | \n", "median_income | \n", "median_house_value | \n", "ocean_proximity | \n", "rooms_per_household | \n", "bedrooms_per_room | \n", "population_per_household | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "-122.23 | \n", "37.88 | \n", "41.0 | \n", "880.0 | \n", "129.0 | \n", "322.0 | \n", "126.0 | \n", "8.3252 | \n", "452600.0 | \n", "NEAR BAY | \n", "6.984127 | \n", "0.146591 | \n", "2.555556 | \n", "
| 1 | \n", "-122.22 | \n", "37.86 | \n", "21.0 | \n", "7099.0 | \n", "1106.0 | \n", "2401.0 | \n", "1138.0 | \n", "8.3014 | \n", "358500.0 | \n", "NEAR BAY | \n", "6.238137 | \n", "0.155797 | \n", "2.109842 | \n", "
| 2 | \n", "-122.24 | \n", "37.85 | \n", "52.0 | \n", "1467.0 | \n", "190.0 | \n", "496.0 | \n", "177.0 | \n", "7.2574 | \n", "352100.0 | \n", "NEAR BAY | \n", "8.288136 | \n", "0.129516 | \n", "2.802260 | \n", "
| 3 | \n", "-122.25 | \n", "37.85 | \n", "52.0 | \n", "1274.0 | \n", "235.0 | \n", "558.0 | \n", "219.0 | \n", "5.6431 | \n", "341300.0 | \n", "NEAR BAY | \n", "5.817352 | \n", "0.184458 | \n", "2.547945 | \n", "
| 4 | \n", "-122.25 | \n", "37.85 | \n", "52.0 | \n", "1627.0 | \n", "280.0 | \n", "565.0 | \n", "259.0 | \n", "3.8462 | \n", "342200.0 | \n", "NEAR BAY | \n", "6.281853 | \n", "0.172096 | \n", "2.181467 | \n", "
| \n", " | longitude | \n", "latitude | \n", "housing_median_age | \n", "total_rooms | \n", "total_bedrooms | \n", "population | \n", "households | \n", "median_income | \n", "rooms_per_household | \n", "bedrooms_per_room | \n", "population_per_household | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|
| count | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "20640.000000 | \n", "
| mean | \n", "-119.569704 | \n", "35.631861 | \n", "28.639486 | \n", "2635.763081 | \n", "536.838857 | \n", "1425.476744 | \n", "499.539680 | \n", "3.870671 | \n", "5.429000 | \n", "0.213794 | \n", "3.070655 | \n", "
| std | \n", "2.003532 | \n", "2.135952 | \n", "12.585558 | \n", "2181.615252 | \n", "419.391878 | \n", "1132.462122 | \n", "382.329753 | \n", "1.899822 | \n", "2.474173 | \n", "0.065248 | \n", "10.386050 | \n", "
| min | \n", "-124.350000 | \n", "32.540000 | \n", "1.000000 | \n", "2.000000 | \n", "1.000000 | \n", "3.000000 | \n", "1.000000 | \n", "0.499900 | \n", "0.846154 | \n", "0.037151 | \n", "0.692308 | \n", "
| 25% | \n", "-121.800000 | \n", "33.930000 | \n", "18.000000 | \n", "1447.750000 | \n", "297.000000 | \n", "787.000000 | \n", "280.000000 | \n", "2.563400 | \n", "4.440716 | \n", "0.175225 | \n", "2.429741 | \n", "
| 50% | \n", "-118.490000 | \n", "34.260000 | \n", "29.000000 | \n", "2127.000000 | \n", "435.000000 | \n", "1166.000000 | \n", "409.000000 | \n", "3.534800 | \n", "5.229129 | \n", "0.203159 | \n", "2.818116 | \n", "
| 75% | \n", "-118.010000 | \n", "37.710000 | \n", "37.000000 | \n", "3148.000000 | \n", "643.250000 | \n", "1725.000000 | \n", "605.000000 | \n", "4.743250 | \n", "6.052381 | \n", "0.240126 | \n", "3.282261 | \n", "
| max | \n", "-114.310000 | \n", "41.950000 | \n", "52.000000 | \n", "39320.000000 | \n", "6445.000000 | \n", "35682.000000 | \n", "6082.000000 | \n", "15.000100 | \n", "141.909091 | \n", "2.824675 | \n", "1243.333333 | \n", "
| \n", " | longitude | \n", "latitude | \n", "housing_median_age | \n", "total_rooms | \n", "total_bedrooms | \n", "population | \n", "households | \n", "median_income | \n", "rooms_per_household | \n", "bedrooms_per_room | \n", "population_per_household | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|
| longitude | \n", "1.000000 | \n", "-0.924664 | \n", "-0.108197 | \n", "0.044568 | \n", "0.069120 | \n", "0.099773 | \n", "0.055310 | \n", "-0.015176 | \n", "-0.027540 | \n", "0.081205 | \n", "0.002476 | \n", "
| latitude | \n", "-0.924664 | \n", "1.000000 | \n", "0.011173 | \n", "-0.036100 | \n", "-0.066484 | \n", "-0.108785 | \n", "-0.071035 | \n", "-0.079809 | \n", "0.106389 | \n", "-0.098619 | \n", "0.002366 | \n", "
| housing_median_age | \n", "-0.108197 | \n", "0.011173 | \n", "1.000000 | \n", "-0.361262 | \n", "-0.319026 | \n", "-0.296244 | \n", "-0.302916 | \n", "-0.119034 | \n", "-0.153277 | \n", "0.135622 | \n", "0.013191 | \n", "
| total_rooms | \n", "0.044568 | \n", "-0.036100 | \n", "-0.361262 | \n", "1.000000 | \n", "0.927058 | \n", "0.857126 | \n", "0.918484 | \n", "0.198050 | \n", "0.133798 | \n", "-0.187381 | \n", "-0.024581 | \n", "
| total_bedrooms | \n", "0.069120 | \n", "-0.066484 | \n", "-0.319026 | \n", "0.927058 | \n", "1.000000 | \n", "0.873535 | \n", "0.974366 | \n", "-0.007617 | \n", "0.001765 | \n", "0.071649 | \n", "-0.028325 | \n", "
| population | \n", "0.099773 | \n", "-0.108785 | \n", "-0.296244 | \n", "0.857126 | \n", "0.873535 | \n", "1.000000 | \n", "0.907222 | \n", "0.004834 | \n", "-0.072213 | \n", "0.010035 | \n", "0.069863 | \n", "
| households | \n", "0.055310 | \n", "-0.071035 | \n", "-0.302916 | \n", "0.918484 | \n", "0.974366 | \n", "0.907222 | \n", "1.000000 | \n", "0.013033 | \n", "-0.080598 | \n", "0.034498 | \n", "-0.027309 | \n", "
| median_income | \n", "-0.015176 | \n", "-0.079809 | \n", "-0.119034 | \n", "0.198050 | \n", "-0.007617 | \n", "0.004834 | \n", "0.013033 | \n", "1.000000 | \n", "0.326895 | \n", "-0.545298 | \n", "0.018766 | \n", "
| rooms_per_household | \n", "-0.027540 | \n", "0.106389 | \n", "-0.153277 | \n", "0.133798 | \n", "0.001765 | \n", "-0.072213 | \n", "-0.080598 | \n", "0.326895 | \n", "1.000000 | \n", "-0.370308 | \n", "-0.004852 | \n", "
| bedrooms_per_room | \n", "0.081205 | \n", "-0.098619 | \n", "0.135622 | \n", "-0.187381 | \n", "0.071649 | \n", "0.010035 | \n", "0.034498 | \n", "-0.545298 | \n", "-0.370308 | \n", "1.000000 | \n", "0.002601 | \n", "
| population_per_household | \n", "0.002476 | \n", "0.002366 | \n", "0.013191 | \n", "-0.024581 | \n", "-0.028325 | \n", "0.069863 | \n", "-0.027309 | \n", "0.018766 | \n", "-0.004852 | \n", "0.002601 | \n", "1.000000 | \n", "
| \n", " | MI | \n", "
|---|---|
| ocean_proximity | \n", "0.101384 | \n", "
| \n", " | card | \n", "reports | \n", "age | \n", "income | \n", "share | \n", "expenditure | \n", "owner | \n", "selfemp | \n", "dependents | \n", "months | \n", "majorcards | \n", "active | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "yes | \n", "0 | \n", "37.66667 | \n", "4.5200 | \n", "0.033270 | \n", "124.983300 | \n", "yes | \n", "no | \n", "3 | \n", "54 | \n", "1 | \n", "12 | \n", "
| 1 | \n", "yes | \n", "0 | \n", "33.25000 | \n", "2.4200 | \n", "0.005217 | \n", "9.854167 | \n", "no | \n", "no | \n", "3 | \n", "34 | \n", "1 | \n", "13 | \n", "
| 2 | \n", "yes | \n", "0 | \n", "33.66667 | \n", "4.5000 | \n", "0.004156 | \n", "15.000000 | \n", "yes | \n", "no | \n", "4 | \n", "58 | \n", "1 | \n", "5 | \n", "
| 3 | \n", "yes | \n", "0 | \n", "30.50000 | \n", "2.5400 | \n", "0.065214 | \n", "137.869200 | \n", "no | \n", "no | \n", "0 | \n", "25 | \n", "1 | \n", "7 | \n", "
| 4 | \n", "yes | \n", "0 | \n", "32.16667 | \n", "9.7867 | \n", "0.067051 | \n", "546.503300 | \n", "yes | \n", "no | \n", "2 | \n", "64 | \n", "1 | \n", "5 | \n", "
| 5 | \n", "yes | \n", "0 | \n", "23.25000 | \n", "2.5000 | \n", "0.044438 | \n", "91.996670 | \n", "no | \n", "no | \n", "0 | \n", "54 | \n", "1 | \n", "1 | \n", "
| 6 | \n", "yes | \n", "0 | \n", "27.91667 | \n", "3.9600 | \n", "0.012576 | \n", "40.833330 | \n", "no | \n", "no | \n", "2 | \n", "7 | \n", "1 | \n", "5 | \n", "
| 7 | \n", "yes | \n", "0 | \n", "29.16667 | \n", "2.3700 | \n", "0.076434 | \n", "150.790000 | \n", "yes | \n", "no | \n", "0 | \n", "77 | \n", "1 | \n", "3 | \n", "
| 8 | \n", "yes | \n", "0 | \n", "37.00000 | \n", "3.8000 | \n", "0.245628 | \n", "777.821700 | \n", "yes | \n", "no | \n", "0 | \n", "97 | \n", "1 | \n", "6 | \n", "
| 9 | \n", "yes | \n", "0 | \n", "28.41667 | \n", "3.2000 | \n", "0.019780 | \n", "52.580000 | \n", "no | \n", "no | \n", "0 | \n", "65 | \n", "1 | \n", "18 | \n", "
| \n", " | card | \n", "reports | \n", "age | \n", "income | \n", "share | \n", "expenditure | \n", "owner | \n", "selfemp | \n", "dependents | \n", "months | \n", "majorcards | \n", "active | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "1 | \n", "0 | \n", "37.66667 | \n", "4.5200 | \n", "0.033270 | \n", "124.983300 | \n", "yes | \n", "no | \n", "3 | \n", "54 | \n", "1 | \n", "12 | \n", "
| 1 | \n", "1 | \n", "0 | \n", "33.25000 | \n", "2.4200 | \n", "0.005217 | \n", "9.854167 | \n", "no | \n", "no | \n", "3 | \n", "34 | \n", "1 | \n", "13 | \n", "
| 2 | \n", "1 | \n", "0 | \n", "33.66667 | \n", "4.5000 | \n", "0.004156 | \n", "15.000000 | \n", "yes | \n", "no | \n", "4 | \n", "58 | \n", "1 | \n", "5 | \n", "
| 3 | \n", "1 | \n", "0 | \n", "30.50000 | \n", "2.5400 | \n", "0.065214 | \n", "137.869200 | \n", "no | \n", "no | \n", "0 | \n", "25 | \n", "1 | \n", "7 | \n", "
| 4 | \n", "1 | \n", "0 | \n", "32.16667 | \n", "9.7867 | \n", "0.067051 | \n", "546.503300 | \n", "yes | \n", "no | \n", "2 | \n", "64 | \n", "1 | \n", "5 | \n", "
| 5 | \n", "1 | \n", "0 | \n", "23.25000 | \n", "2.5000 | \n", "0.044438 | \n", "91.996670 | \n", "no | \n", "no | \n", "0 | \n", "54 | \n", "1 | \n", "1 | \n", "
| 6 | \n", "1 | \n", "0 | \n", "27.91667 | \n", "3.9600 | \n", "0.012576 | \n", "40.833330 | \n", "no | \n", "no | \n", "2 | \n", "7 | \n", "1 | \n", "5 | \n", "
| 7 | \n", "1 | \n", "0 | \n", "29.16667 | \n", "2.3700 | \n", "0.076434 | \n", "150.790000 | \n", "yes | \n", "no | \n", "0 | \n", "77 | \n", "1 | \n", "3 | \n", "
| 8 | \n", "1 | \n", "0 | \n", "37.00000 | \n", "3.8000 | \n", "0.245628 | \n", "777.821700 | \n", "yes | \n", "no | \n", "0 | \n", "97 | \n", "1 | \n", "6 | \n", "
| 9 | \n", "1 | \n", "0 | \n", "28.41667 | \n", "3.2000 | \n", "0.019780 | \n", "52.580000 | \n", "no | \n", "no | \n", "0 | \n", "65 | \n", "1 | \n", "18 | \n", "
| \n", " | threshold | \n", "tp | \n", "fp | \n", "fn | \n", "tn | \n", "
|---|---|---|---|---|---|
| 0 | \n", "0.0 | \n", "211 | \n", "53 | \n", "0 | \n", "0 | \n", "
| 10 | \n", "0.1 | \n", "210 | \n", "17 | \n", "1 | \n", "36 | \n", "
| 20 | \n", "0.2 | \n", "207 | \n", "5 | \n", "4 | \n", "48 | \n", "
| 30 | \n", "0.3 | \n", "205 | \n", "5 | \n", "6 | \n", "48 | \n", "
| 40 | \n", "0.4 | \n", "205 | \n", "1 | \n", "6 | \n", "52 | \n", "
| 50 | \n", "0.5 | \n", "204 | \n", "1 | \n", "7 | \n", "52 | \n", "
| 60 | \n", "0.6 | \n", "204 | \n", "1 | \n", "7 | \n", "52 | \n", "
| 70 | \n", "0.7 | \n", "204 | \n", "1 | \n", "7 | \n", "52 | \n", "
| 80 | \n", "0.8 | \n", "204 | \n", "1 | \n", "7 | \n", "52 | \n", "
| 90 | \n", "0.9 | \n", "204 | \n", "0 | \n", "7 | \n", "53 | \n", "
| 100 | \n", "1.0 | \n", "179 | \n", "0 | \n", "32 | \n", "53 | \n", "
| \n", " | feature | \n", "importance | \n", "
|---|---|---|
| 0 | \n", "households | \n", "0.016387 | \n", "
| 1 | \n", "housing_median_age | \n", "0.033145 | \n", "
| 2 | \n", "latitude | \n", "0.101256 | \n", "
| 3 | \n", "longitude | \n", "0.096470 | \n", "
| 4 | \n", "median_income | \n", "0.363326 | \n", "
| 5 | \n", "ocean_proximity=<1H OCEAN | \n", "0.002489 | \n", "
| 6 | \n", "ocean_proximity=INLAND | \n", "0.310901 | \n", "
| 7 | \n", "ocean_proximity=ISLAND | \n", "0.000357 | \n", "
| 8 | \n", "ocean_proximity=NEAR BAY | \n", "0.000480 | \n", "
| 9 | \n", "ocean_proximity=NEAR OCEAN | \n", "0.004699 | \n", "
| 10 | \n", "population | \n", "0.030777 | \n", "
| 11 | \n", "total_bedrooms | \n", "0.019172 | \n", "
| 12 | \n", "total_rooms | \n", "0.020541 | \n", "
| \n", " | feature | \n", "importance | \n", "
|---|---|---|
| 4 | \n", "median_income | \n", "0.363326 | \n", "
| 6 | \n", "ocean_proximity=INLAND | \n", "0.310901 | \n", "
| 2 | \n", "latitude | \n", "0.101256 | \n", "
| 3 | \n", "longitude | \n", "0.096470 | \n", "
| 1 | \n", "housing_median_age | \n", "0.033145 | \n", "
Configure 100 users with ramp time of 10 users per second. Click "Start Swarming" and ensure that it is working.
Now download a second model with this command:
```bash
curl -O https://s3.us-west-2.amazonaws.com/bentoml.com/mlzoomcamp/coolmodel2.bentomodel
```
Or you can download with this link as well:
[https://s3.us-west-2.amazonaws.com/bentoml.com/mlzoomcamp/coolmodel2.bentomodel](https://s3.us-west-2.amazonaws.com/bentoml.com/mlzoomcamp/coolmodel2.bentomodel)
Now import the model:
```bash
bentoml models import coolmodel2.bentomodel
```
Update your bento's runner tag and test with both models. Which model allows more traffic (more throughput) as you ramp up the traffic?
**Hint 1**: Remember to turn off and turn on your bento service between changing the model tag. Use Ctl-C to close the service in between trials.
**Hint 2**: Increase the number of concurrent users to see which one has higher throughput
Which model has better performance at higher volumes?
* The first model
* The second model
## Email from marketing
Hello ML person! I hope this email finds you well. I've heard there's this cool new ML model called Stable Diffusion.
I hear if you give it a description of a picture it will generate an image. We need a new company logo and I want it
to be fierce but also cool, think you could help out?
Thanks,
Mike Marketer
## Question 7 (optional)
Go to this Bento deployment of Stable Diffusion: https://l.bentoml.com/stable-diffusion-mlzoomcamp (or deploy it yourself)
Use the txt2image endpoint and update the prompt to: "A cartoon dragon with sunglasses".
Don't change the seed, it should be 0 by default
What is the resulting image?
### #1
### #2
### #3
### #4
## Submit the results
* Submit your results here: https://forms.gle/Hh9FWy6LGXk3wJYs8
* You can submit your solution multiple times. In this case, only the last submission will be used
* If your answer doesn't match options exactly, select the closest one
## Deadline
The deadline for submitting is **24 October 2022 (Monday), 23:00 CEST (Berlin time)**.
After that, the form will be closed.
================================================
FILE: cohorts/2022/07-bento-production/locustfile.py
================================================
from locust import task
from locust import between
from locust import HttpUser
sample = [[6.4,3.5,4.5,1.2]]
class MLZoomUser(HttpUser):
"""
Usage:
Start locust load testing client with:
locust -H http://localhost:3000
Open browser at http://0.0.0.0:8089, adjust desired number of users and spawn
rate for the load test from the Web UI and start swarming.
"""
@task
def classify(self):
self.client.post("/classify", json=sample)
wait_time = between(0.01, 2)
================================================
FILE: cohorts/2022/08-deep-learning/homework.md
================================================
## Homework
> **Solution**:
> * [Notebook](homework_8.ipynb)
### Dataset
In this homework, we'll build a model for predicting if we have an image of a dino or a dragon. For this, we will use the "Dino or Dragon?" dataset that can be downloaded from [Kaggle](https://www.kaggle.com/datasets/agrigorev/dino-or-dragon).
You can get a wget-able version [here](https://github.com/alexeygrigorev/dino-or-dragon/releases/download/data/dino-dragon.zip):
```bash
wget https://github.com/alexeygrigorev/dino-or-dragon/releases/download/data/dino-dragon.zip
unzip dino-dragon.zip
```
In the lectures we saw how to use a pre-trained neural network. In the homework, we'll train a much smaller model from scratch.
**Note:** You will need an environment with a GPU for this homework. We recommend to use [Saturn Cloud](https://bit.ly/saturn-mlzoomcamp).
You can also use a computer without a GPU (e.g. your laptop), but it will be slower.
### Data Preparation
The dataset contains around 1900 images of dinos and around 1900 images of dragons.
The dataset contains separate folders for training and test sets.
### Model
For this homework we will use Convolutional Neural Network (CNN). Like in the lectures, we'll use Keras.
You need to develop the model with following structure:
* The shape for input should be `(150, 150, 3)`
* Next, create a convolutional layer ([`Conv2D`](https://keras.io/api/layers/convolution_layers/convolution2d/)):
* Use 32 filters
* Kernel size should be `(3, 3)` (that's the size of the filter)
* Use `'relu'` as activation
* Reduce the size of the feature map with max pooling ([`MaxPooling2D`](https://keras.io/api/layers/pooling_layers/max_pooling2d/))
* Set the pooling size to `(2, 2)`
* Turn the multi-dimensional result into vectors using a [`Flatten`](https://keras.io/api/layers/reshaping_layers/flatten/) layer
* Next, add a `Dense` layer with 64 neurons and `'relu'` activation
* Finally, create the `Dense` layer with 1 neuron - this will be the output
* The output layer should have an activation - use the appropriate activation for the binary classification case
As optimizer use [`SGD`](https://keras.io/api/optimizers/sgd/) with the following parameters:
* `SGD(lr=0.002, momentum=0.8)`
For clarification about kernel size and max pooling, check [Office Hours](https://www.youtube.com/watch?v=1WRgdBTUaAc).
### Question 1
Since we have a binary classification problem, what is the best loss function for us?
- `binary crossentropy`
- `focal loss`
- `mean squared error`
- `categorical crossentropy`
Note: since we specify an activation for the output layer, we don't need to set `from_logits=True`
### Question 2
What's the total number of parameters of the model? You can use the `summary` method for that.
- 9215873
- 11215873
- 14215873
- 19215873
### Generators and Training
For the next two questions, use the following data generator for both train and test sets:
```python
ImageDataGenerator(rescale=1./255)
```
* We don't need to do any additional pre-processing for the images.
* When reading the data from train/test directories, check the `class_mode` parameter. Which value should it be for a binary classification problem?
* Use `batch_size=20`
* Use `shuffle=True` for both training and test sets.
For training use `.fit()` with the following params:
```python
model.fit(
train_generator,
epochs=10,
validation_data=test_generator
)
```
### Question 3
What is the median of training accuracy for all the epochs for this model?
- 0.40
- 0.60
- 0.90
- 0.20
### Question 4
What is the standard deviation of training loss for all the epochs for this model?
- 0.11
- 0.66
- 0.99
- 0.33
### Data Augmentation
For the next two questions, we'll generate more data using data augmentations.
Add the following augmentations to your training data generator:
* `rotation_range=40,`
* `width_shift_range=0.2,`
* `height_shift_range=0.2,`
* `shear_range=0.2,`
* `zoom_range=0.2,`
* `horizontal_flip=True,`
* `fill_mode='nearest'`
### Question 5
Let's train our model for 10 more epochs using the same code as previously.
Make sure you don't re-create the model - we want to continue training the model
we already started training.
What is the mean of test loss for all the epochs for the model trained with augmentations?
- 0.15
- 0.77
- 0.37
- 0.97
### Question 6
What's the average of test accuracy for the last 5 epochs (from 6 to 10)
for the model trained with augmentations?
- 0.84
- 0.54
- 0.44
- 0.24
## Submit the results
- Submit your results here: https://forms.gle/XdH5ztBddvTvxzpT6
- You can submit your solution multiple times. In this case, only the last submission will be used
- If your answer doesn't match options exactly, select the closest one. If the answer is exactly in between two options, select the higher value.
## Deadline
The deadline for submitting is 21 November 2022, 23:00 CEST.
After that, the form will be closed.
================================================
FILE: cohorts/2022/08-deep-learning/homework_8.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"id": "U19vNUrsczT4"
},
"outputs": [],
"source": [
"#@ IMPORTING LIBRARIES:\n",
"import os\n",
"import shutil\n",
"\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"\n",
"from tensorflow import keras\n",
"\n",
"from tensorflow.keras import layers\n",
"from tensorflow.keras import models\n",
"from tensorflow.keras import optimizers\n",
"\n",
"from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
"from tensorflow.keras.preprocessing import image"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"id": "fAOIcGU3jWsy",
"outputId": "e90e943c-e7d2-4822-a421-35648b743b64"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'2.9.2'"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#@ INSPECTION:\n",
"import tensorflow\n",
"tensorflow.__version__"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "BPF6k1g9jkpt"
},
"outputs": [],
"source": [
"#@ UNZIPPING THE DATASET:\n",
"# !unzip /content/drive/MyDrive/Data/dragon.zip"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "uCCnd0pik2hh"
},
"outputs": [],
"source": [
"#@ INITIALIZING PATH TO DATASET:\n",
"train_dir = './train'\n",
"test_dir = './test'"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "EOtAG1Oblhm1"
},
"outputs": [],
"source": [
"#@ INITIALIZING MODEL CREATION:\n",
"model = models.Sequential()\n",
"\n",
"model.add(layers.Conv2D(32, (3, 3), activation='relu',\n",
" input_shape=(150, 150, 3)))\n",
"model.add(layers.MaxPooling2D((2, 2)))\n",
"model.add(layers.Flatten())\n",
"model.add(layers.Dense(64, activation='relu'))\n",
"model.add(layers.Dense(1, activation='sigmoid'))"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "HTOuSh11ln91"
},
"outputs": [],
"source": [
"#@ COMPILING THE MODEL:\n",
"model.compile(loss='binary_crossentropy',\\\n",
" optimizer=optimizers.SGD(learning_rate=0.002, momentum=0.8),\\\n",
" metrics=['acc'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7XcouIUVmPK2"
},
"source": [
"- Answer for Question1: `binary_crossentropy`."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "i_UEvLG2mLCy",
"outputId": "bc2fda0d-5090-4521-976f-8ecdea635f0b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"sequential\"\n",
"_________________________________________________________________\n",
" Layer (type) Output Shape Param # \n",
"=================================================================\n",
" conv2d (Conv2D) (None, 148, 148, 32) 896 \n",
" \n",
" max_pooling2d (MaxPooling2D (None, 74, 74, 32) 0 \n",
" ) \n",
" \n",
" flatten (Flatten) (None, 175232) 0 \n",
" \n",
" dense (Dense) (None, 64) 11214912 \n",
" \n",
" dense_1 (Dense) (None, 1) 65 \n",
" \n",
"=================================================================\n",
"Total params: 11,215,873\n",
"Trainable params: 11,215,873\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"#@ SUMMARY:\n",
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6fJl3C7hmlk1"
},
"source": [
"- Answer for Question2: 11,215,873"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "U7CL3rmomdtN"
},
"outputs": [],
"source": [
"#@ INITIALIZING IMAGE DATAGENERATOR:\n",
"train_datagen = ImageDataGenerator(rescale=1./255)\n",
"val_datagen = ImageDataGenerator(rescale=1./255)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "NZRgtr_mnryc",
"outputId": "a371c6f3-0785-42a4-c1d2-c617a5c2c95e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 1594 images belonging to 2 classes.\n"
]
}
],
"source": [
"#@ TRAINING DATAGENERATOR:\n",
"train_generator = train_datagen.flow_from_directory(train_dir,\n",
" target_size=(150, 150),\n",
" batch_size=20,\n",
" class_mode='binary')"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "orvdanQ0n15D",
"outputId": "3d9550ed-8044-4b4b-9e25-e375382e7512"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 394 images belonging to 2 classes.\n"
]
}
],
"source": [
"#@ VALIDATION DATAGENERATOR:\n",
"validation_generator = val_datagen.flow_from_directory(test_dir,\n",
" target_size=(150, 150),\n",
" batch_size=20,\n",
" class_mode='binary')"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "IB0Amcqzn7qu",
"outputId": "e829d59b-b33f-4145-d494-b96c3308cbbc"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"data batch shape: (20, 150, 150, 3)\n",
"labels batch shape: (20,)\n"
]
}
],
"source": [
"#@ INSPECTING:\n",
"for data_batch, labels_batch in train_generator:\n",
" print('data batch shape:', data_batch.shape)\n",
" print('labels batch shape:', labels_batch.shape)\n",
" break"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "XHilosW_oFF_",
"outputId": "f551655b-a566-4234-b48c-a8b21f23928f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"80/80 [==============================] - 9s 114ms/step - loss: 0.4787 - acc: 0.7961 - val_loss: 0.4133 - val_acc: 0.8477\n",
"Epoch 2/10\n",
"80/80 [==============================] - 9s 111ms/step - loss: 0.4040 - acc: 0.8174 - val_loss: 0.3995 - val_acc: 0.8223\n",
"Epoch 3/10\n",
"80/80 [==============================] - 9s 111ms/step - loss: 0.3563 - acc: 0.8557 - val_loss: 0.3538 - val_acc: 0.8477\n",
"Epoch 4/10\n",
"80/80 [==============================] - 9s 113ms/step - loss: 0.3091 - acc: 0.8752 - val_loss: 0.3345 - val_acc: 0.8528\n",
"Epoch 5/10\n",
"80/80 [==============================] - 9s 111ms/step - loss: 0.2739 - acc: 0.9028 - val_loss: 0.3209 - val_acc: 0.8629\n",
"Epoch 6/10\n",
"80/80 [==============================] - 9s 110ms/step - loss: 0.2435 - acc: 0.9109 - val_loss: 0.3324 - val_acc: 0.8655\n",
"Epoch 7/10\n",
"80/80 [==============================] - 9s 111ms/step - loss: 0.2246 - acc: 0.9210 - val_loss: 0.3094 - val_acc: 0.8731\n",
"Epoch 8/10\n",
"80/80 [==============================] - 9s 111ms/step - loss: 0.1817 - acc: 0.9410 - val_loss: 0.2763 - val_acc: 0.8782\n",
"Epoch 9/10\n",
"80/80 [==============================] - 9s 111ms/step - loss: 0.1534 - acc: 0.9555 - val_loss: 0.2816 - val_acc: 0.8680\n",
"Epoch 10/10\n",
"80/80 [==============================] - 9s 110ms/step - loss: 0.1395 - acc: 0.9586 - val_loss: 0.3197 - val_acc: 0.8579\n"
]
}
],
"source": [
"#@ TRAINING THE MODEL:\n",
"history = model.fit(\n",
" train_generator,\n",
" epochs=10,\n",
" validation_data=validation_generator)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 545
},
"id": "Bl5GyCf-oSac",
"outputId": "35a1ad38-2143-4c96-ff01-56f8925fe90d"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV1bn/8c8XFJFRGZwIIdQLCooJEKGgKCoqogVBtISgUAeqVr3a9nq1aKX4o9qrv1vrq+rvUgcEUpFai6g4AOp1qG0JMigoiMgQcIgok4BMz++PtUNOYkJOyElOcvK8X6/zOuesPT1nB56zztprryUzwznnXOpqkOwAnHPOVS9P9M45l+I80TvnXIrzRO+ccynOE71zzqU4T/TOOZfiPNHXQ5JekjQ60esmk6TVkgZUw37fkHR19DpX0qvxrHsQx0mXtE1Sw4ON1bnyeKKvI6IkUPTYJ2lHzPvcyuzLzC4wsycTvW5tJOk2SW+WUd5G0i5JJ8e7LzPLM7PzEhRXiS8mM1trZs3MbG8i9l/G8SRplaRl1bF/V7t5oq8joiTQzMyaAWuBH8WU5RWtJ+mQ5EVZK00D+krqWKp8BPC+mX2QhJiS4QzgKOAHkk6tyQP7v8nk80Rfx0nqL6lA0n9K+hx4QtKRkl6QVCjpm+h1Wsw2sc0RYyS9Len+aN1PJV1wkOt2lPSmpK2S5kp6SNK0cuKOJ8a7Jb0T7e9VSW1ill8uaY2kjZLGlXd+zKwAeA24vNSiK4ApFcVRKuYxkt6OeX+upI8kbZb0R0Axy46X9FoU31eS8iQdES2bCqQDz0e/yG6VlCHJipKipOMkzZL0taSVkq6J2fd4STMkTYnOzVJJ2eWdg8ho4DlgdvQ69nOdJGlOdKwvJP0qKm8o6VeSPomOs0BS+9KxRuuW/nfyjqTfS9oIjD/Q+Yi2aS/p2ejvsFHSHyU1imLqFrPeUZK2S2pbwed1MTzRp4ZjgFZAB2As4e/6RPQ+HdgB/PEA2/cGlgNtgP8CHpOkg1j3z8C/gNbAeL6fXGPFE+NI4CeEmmgj4JcAkroCj0T7Py46XpnJOfJkbCySTgCyongre66K9tEGeBa4g3AuPgFOi10FuCeKrwvQnnBOMLPLKfmr7L/KOMR0oCDafjjwW0lnxywfHK1zBDDrQDFLahLtIy96jJDUKFrWHJgLvBwd69+AedGmPwdygEFAC+BKYPsBT0yx3sAq4Ghg4oHOh8J1iReANUAG0A6Ybma7os84Kma/OcA8MyuMMw4HYGb+qGMPYDUwIHrdH9gFND7A+lnANzHv3wCujl6PAVbGLGsCGHBMZdYlJMk9QJOY5dOAaXF+prJivCPm/fXAy9HrXxMSQdGyptE5GFDOvpsAW4C+0fuJwHMHea7ejl5fAfwjZj0REvPV5ez3YmBhWX/D6H1GdC4PISTBvUDzmOX3AJOj1+OBuTHLugI7DnBuRwGF0b4bA5uBodGynNi4Sm23HBhSRvn+WA9wntZW8Pfefz6APkXxlbFeb8KXoqL3+cBlyfz/VxcfXqNPDYVmtrPojaQmkv4natrYArwJHKHye3R8XvTCzIpqbM0que5xwNcxZQDrygs4zhg/j3m9PSam42L3bWbfAhvLO1YU01+AK6JfH7nAlErEUZbSMVjse0lHS5ouaX2032mEmn88is7l1piyNYSabpHS56axym8LHw3MMLM90b+Tv1LcfNOe8GukLAdaVpESf/sKzkd7YI2Z7Sm9EzP7J+Hz9Zd0IuEXx6yDjKne8kSfGkoPQfoL4ASgt5m1IFyIg5g25GrwGdAqaiYo0v4A61clxs9i9x0ds3UF2zwJXAacCzQHnq9iHKVjECU/728Jf5du0X5HldrngYaN3UA4l81jytKB9RXE9D3R9YazgVGSPle4jjMcGBQ1P60DflDO5uuA48so/zZ6jv1bH1NqndKf70DnYx2QfoAvqiej9S8Hnomt1Lj4eKJPTc0Jbc2bJLUC7qruA5rZGsLP6vHRRbQ+wI+qKcZngIsknR61NU+g4n/LbwGbgEkUt/9WJY4XgZMkDYsS1E2UTHbNgW3AZkntgP8otf0XlJNgzWwd8HfgHkmNJZ0CXEWoBVfW5cAKwpdZVvToTGhmyiG0jR8r6WZJh0lqLql3tO2jwN2SOik4RVJrC+3j6wlfHg0lXUnZXwixDnQ+/kX44rxXUtPoM8de75gGDCUk+ykHcQ7qPU/0qekB4HDgK+AfhAttNSGX0N66Efg/wNPAd+Wse9AxmtlS4GeEi6mfAd8QEteBtjFCkuhAyWRxUHGY2VfApcC9hM/bCXgnZpXfAD0I7eEvEi7cxroHuEPSJkm/LOMQOYS28A3A34C7zGxuPLGVMhp42Mw+j30A/w8YHTUPnUv4Uv4c+Bg4K9r2v4EZwKuEaxyPEc4VwDWEZL0ROInwxXQg5Z4PC/cO/IjQLLOW8Lf8cczydcB7hF8Eb1X+FLiiCxzOJZykp4GPzKzaf1G41CbpcWCDmd2R7FjqIk/0LmEUbsT5GvgUOA+YCfQxs4VJDczVaZIygEVAdzP7NLnR1E3edOMS6RhCN7ttwIPAdZ7kXVVIuhv4ALjPk/zB8xq9c86lOK/RO+dciqt1gw21adPGMjIykh2Gc87VKQsWLPjKzMocA6jWJfqMjAzy8/OTHYZzztUpktaUt8ybbpxzLsV5onfOuRTnid4551JcXG30kgYCfwAaAo+a2b2llncAHgfaEm6YGWVhwgckpRPGzGhPuIV5kJmtrkyQu3fvpqCggJ07fSyj2qpx48akpaVx6KGHJjsU51wpFSb6aLjWhwjjYRQA8yXNMrPYuSfvB6aY2ZPR5Aj3UDzRwxRgopnNkdQM2FfZIAsKCmjevDkZGRmUPx+GSxYzY+PGjRQUFNCxY+kZ+5xzyRZP000vwmQTq6x4xpchpdbpSpiuDeD1ouXRTECHmNkcADPbVmq88rjs3LmT1q1be5KvpSTRunVr/8Xl3EHKy4OMDGjQIDzn5VW0ReXEk+jbUXISgQJKToAAsBgYFr0eCjSX1JowHOqmaC7IhZLui2NChzJ5kq/d/O/j3MHJy4OxY2HNGjALz2PHJjbZJ+pi7C+BMyUtBM4kjFW9l9A01C9afiph/O0xpTeWNFZSvqT8wkKfCtI5V3+MGwfbS7VzbN8eyhMlnkS/npIz56RRaqYbM9tgZsPMrDswLirbRKj9L4qaffYQRjPsUfoAZjbJzLLNLLtt29o3ufvGjRvJysoiKyuLY445hnbt2u1/v2vXrgNum5+fz0033VThMfr27ZuocJ1zdcjatZUrPxjxJPr5QCdJHaPZfEZQas5GSW0kFe3rdkIPnKJtj5BUlL3PBmIv4laLRLd3tW7dmkWLFrFo0SKuvfZabrnllv3vGzVqxJ4935vqcr/s7GwefPDBCo/x979XNG+Dcy7RqrttPB7p6ZUrPxgVJvqoJn4D8ArwIWGS4aWSJkgaHK3WH1guaQVwNDAx2nYvodlmnqT3CXNE/ilx4X9fTbR3AYwZM4Zrr72W3r17c+utt/Kvf/2LPn360L17d/r27cvy5csBeOONN7jooosAGD9+PFdeeSX9+/fnBz/4QYkvgGbNmu1fv3///gwfPpwTTzyR3NxcikYYnT17NieeeCI9e/bkpptu2r/fWKtXr6Zfv3706NGDHj16lPgC+d3vfke3bt3IzMzktttuA2DlypUMGDCAzMxMevTowSefHOxc0M7VLTWVKyoycSI0aVKyrEmTUJ4wZlarHj179rTSli1b9r2y8nToYBb+bCUfHTrEvYsDuuuuu+y+++6z0aNH24UXXmh79uwxM7PNmzfb7t27zcxszpw5NmzYMDMze/311+3CCy/cv22fPn1s586dVlhYaK1atbJdu3aZmVnTpk33r9+iRQtbt26d7d271374wx/aW2+9ZTt27LC0tDRbtWqVmZmNGDFi/35jffvtt7Zjxw4zM1uxYoUVnc/Zs2dbnz597NtvvzUzs40bN5qZWa9evezZZ581M7MdO3bsX34wKvN3ci7ZqjtXVMa0aeG4UnieNq3y+wDyrZy8WusGNauqmmjvKnLppZfSsGHoRLR582ZGjx7Nxx9/jCR2795d5jYXXnghhx12GIcddhhHHXUUX3zxBWlpaSXW6dWr1/6yrKwsVq9eTbNmzfjBD36wv596Tk4OkyZN+t7+d+/ezQ033MCiRYto2LAhK1asAGDu3Ln85Cc/oUlUdWjVqhVbt25l/fr1DB06FAg3PTlXX9RkrqhIbm54VJeUGwKhJtq7ijRt2nT/6zvvvJOzzjqLDz74gOeff77cPuWHHXbY/tcNGzYss30/nnXK8/vf/56jjz6axYsXk5+fX+HFYufqq5rMFcmWcom+Rtq7yrB582batQu3F0yePDnh+z/hhBNYtWoVq1evBuDpp58uN45jjz2WBg0aMHXqVPbu3QvAueeeyxNPPMH2qB/X119/TfPmzUlLS2PmzJkAfPfdd/uXO5fqkpUrkiHlEn1uLkyaBB06gBSeJ02q3p9FALfeeiu333473bt3r1QNPF6HH344Dz/8MAMHDqRnz540b96cli1bfm+966+/nieffJLMzEw++uij/b86Bg4cyODBg8nOziYrK4v7778fgKlTp/Lggw9yyimn0LdvXz7//POEx+5cbZSsXJEMtW7O2OzsbCs98ciHH35Ily5dkhRR7bFt2zaaNWuGmfGzn/2MTp06ccsttyQ7rP387+Rc8khaYGbZZS1LuRp9KvvTn/5EVlYWJ510Eps3b+anP/1pskNyztUBKdfrJpXdcssttaoG75yrG7xG75xzKc4TvXPOpThP9M45l+I80TvnXIrzRB+Hs846i1deeaVE2QMPPMB1111X7jb9+/enqJvooEGD2LRp0/fWGT9+/P7+7OWZOXMmy5YVD/j561//mrlz51YmfOdcPeeJPg45OTlMnz69RNn06dPJycmJa/vZs2dzxBFHHNSxSyf6CRMmMGDAgIPal3OufvJEH4fhw4fz4osv7h83ZvXq1WzYsIF+/fpx3XXXkZ2dzUknncRdd91V5vYZGRl89dVXAEycOJHOnTtz+umn7x/KGEIf+VNPPZXMzEwuueQStm/fzt///ndmzZrFf/zHf5CVlcUnn3zCmDFjeOaZZwCYN28e3bt3p1u3blx55ZV89913+49311130aNHD7p168ZHH330vZh8OGPn6o8614/+5pth0aLE7jMrCx54oPzlrVq1olevXrz00ksMGTKE6dOnc9lllyGJiRMn0qpVK/bu3cs555zDkiVLOOWUU8rcz4IFC5g+fTqLFi1iz5499OjRg549ewIwbNgwrrnmGgDuuOMOHnvsMW688UYGDx7MRRddxPDhw0vsa+fOnYwZM4Z58+bRuXNnrrjiCh555BFuvvlmANq0acN7773Hww8/zP3338+jjz5aYvujjjqKOXPm0LhxYz7++GNycnLIz8/npZde4rnnnuOf//wnTZo04euvvwYgNzeX2267jaFDh7Jz50727dt3UOfaOVfzvEYfp9jmm9hmmxkzZtCjRw+6d+/O0qVLSzSzlPbWW28xdOhQmjRpQosWLRg8ePD+ZR988AH9+vWjW7du5OXlsXTp0gPGs3z5cjp27Ejnzp0BGD16NG+++eb+5cOGhbnae/bsuX8gtFi7d+/mmmuuoVu3blx66aX74453OOMmpUeDcq4SasPMTvVJXDV6SQOBPwANgUfN7N5SyzsQpg9sC3wNjDKzgpjlLQhTCM40sxuqEvCBat7VaciQIdxyyy289957bN++nZ49e/Lpp59y//33M3/+fI488kjGjBlT7vDEFRkzZgwzZ84kMzOTyZMn88Ybb1Qp3qKhjssb5jh2OON9+/b5WPSuxhTN7FQ0UGrRzE6QmgOK1QYV1uglNQQeAi4AugI5krqWWu1+YIqZnQJMAO4ptfxu4E3qsGbNmnHWWWdx5ZVX7q/Nb9myhaZNm9KyZUu++OILXnrppQPu44wzzmDmzJns2LGDrVu38vzzz+9ftnXrVo499lh2795NXkz1pnnz5mzduvV7+zrhhBNYvXo1K1euBMIolGeeeWbcn8eHM3bJMm5ccZIvsn17KHfVI56mm17ASjNbZWa7gOnAkFLrdAVei16/HrtcUk/CPLKvVj3c5MrJyWHx4sX7E31mZibdu3fnxBNPZOTIkZx22mkH3L5Hjx78+Mc/JjMzkwsuuIBTTz11/7K7776b3r17c9ppp3HiiSfuLx8xYgT33Xcf3bt3L3EBtHHjxjzxxBNceumldOvWjQYNGnDttdfG/Vl8OGOXLLVpZqf6osJhiiUNBwaa2dXR+8uB3rFNMJL+DPzTzP4gaRjwV6AN8A3hC2AUMADILqvpRtJYYCxAenp6zzVr1pRY7sPf1g3+d3LxyMgIzTWldegAZVxOcnGqiWGKfwmcKWkhcCawHtgLXA/Mjm2vL4uZTTKzbDPLbtu2bYJCcs7VRvVpZqfaIp6LseuB9jHv06Ky/cxsAzAMQFIz4BIz2ySpD9BP0vVAM6CRpG1mdltConfO1TlFF1zHjQvNNenpIcn7hdjqE0+inw90ktSRkOBHACNjV5DUBvjazPYBtxN64GBmuTHrjCE03RxUkjczJB3Mpq4G1LaZylztlpvrib0mVdh0Y2Z7gBuAV4APgRlmtlTSBElFHcH7A8slrSBceE3oj7DGjRuzceNGTya1lJmxceNG76LpXC1VJ+aM3b17NwUFBQfdR91Vv8aNG5OWlsahhx6a7FCcq5cOdDG2TgyBcOihh9KxY8dkh+Gcc3WSD4HgnHMpzhO9c86lOE/0zjmX4jzRO+dcivNE75xzKc4TvXPOpThP9M45l+I80TvnXIrzRO+ccynOE71zzqU4T/TOOZfiPNE751yK80TvnHMpzhO9c/VMXl6Yt7VBg/Ccl5fsiFx1iyvRSxooabmklZK+N0OUpA6S5klaIukNSWlReZakdyUtjZb9ONEfwDkXv7w8GDs2TM5tFp7HjvVkn+oqnHhEUkNgBXAuUECYWjDHzJbFrPMX4AUze1LS2cBPzOxySZ0BM7OPJR0HLAC6mNmm8o5X1sQjzrnEyMgIyb20Dh1g9eqajsYl0oEmHomnRt8LWGlmq8xsFzAdGFJqna7Aa9Hr14uWm9kKM/s4er0B+BJoW/mP4JxLhLVrK1fuUkM8ib4dsC7mfUFUFmsxMCx6PRRoLql17AqSegGNgE9KH0DSWEn5kvILCwvjjd05V0np6ZUrd6khURdjfwmcKWkhcCawHthbtFDSscBUQpPOvtIbm9kkM8s2s+y2bb3C71x1mTgRmjQpWdakSSh3qSueOWPXA+1j3qdFZftFzTLDACQ1Ay4paoeX1AJ4ERhnZv9IRNDOuYOTmxuex40LzTXp6SHJF5W71BRPop8PdJLUkZDgRwAjY1eQ1Ab4Oqqt3w48HpU3Av4GTDGzZxIZuHPu4OTmemKvbypsujGzPcANwCvAh8AMM1sqaYKkwdFq/YHlklYARwNFPwQvA84AxkhaFD2yEv0hnHPOla/C7pU1zbtXOudc5VW1e6Vzzrk6zBO9c86lOE/0ztUQH2PGJUs8vW6cc1VUNMbM9u3hfdEYM+A9YFz18xq9czVg3LjiJF9k+/ZQ7lx180TvXA3wMWZcMnmid64G+BgzLpk80TtXA3yMGZdMnuidqwG5uTBpUhj3XQrPkyb5hVhXM7zXjXM1xMeYccniNXrnnEtxnuidcy7FeaJ3Kc/vSHX1nbfRu5Tmd6Q65zV6l+L8jlTn4kz0kgZKWi5ppaTbyljeQdI8SUskvSEpLWbZaEkfR4/RiQzeuYr4HanOxZHoJTUEHgIuALoCOZK6llrtfsJ0gacAE4B7om1bAXcBvYFewF2Sjkxc+M4dmN+R6lx8NfpewEozW2Vmu4DpwJBS63QFXotevx6z/Hxgjpl9bWbfAHOAgVUP27n4+B2pzsWX6NsB62LeF0RlsRYDw6LXQ4HmklrHua1z1cbvSHUucb1ufgn8UdIY4E1gPbA33o0ljQXGAqT7b2qXYH5Hqqvv4qnRrwfax7xPi8r2M7MNZjbMzLoD46KyTfFsG607ycyyzSy7bdu2lfwIzjnnDiSeRD8f6CSpo6RGwAhgVuwKktpIKtrX7cDj0etXgPMkHRldhD0vKnPOOVdDKkz0ZrYHuIGQoD8EZpjZUkkTJA2OVusPLJe0AjgamBht+zVwN+HLYj4wISpzzjlXQ2RmyY6hhOzsbMvPz092GM45V6dIWmBm2WUt8ztjnXMuxXmid865FOeJ3jnnUpwneuecS3Ge6J1zLsV5onfOuRTnid4551KcJ3rnnEtxnuidcy7FeaJ3zrkU54neOedSnCd655xLcZ7onXMuxXmid865FOeJ3jnnUpwneuecS3FxJXpJAyUtl7RS0m1lLE+X9LqkhZKWSBoUlR8q6UlJ70v6UNLtif4AzjnnDqzCRC+pIfAQcAHQFciR1LXUancQphjsTphT9uGo/FLgMDPrBvQEfiopIzGhO+eci0c8NfpewEozW2Vmu4DpwJBS6xjQInrdEtgQU95U0iHA4cAuYEuVo3bOORe3eBJ9O2BdzPuCqCzWeGCUpAJgNnBjVP4M8C3wGbAWuL+sycEljZWULym/sLCwcp/AOefcAR2SoP3kAJPN7P9K6gNMlXQy4dfAXuA44EjgLUlzzWxV7MZmNgmYBGFy8ATF5JyrZfbuhY8/hsWLYckS+OILuOgiGDQIGjVKdnSpK55Evx5oH/M+LSqLdRUwEMDM3pXUGGgDjAReNrPdwJeS3gGygVU451Lapk0hmS9eXJzYP/gAduwIyw85BJo2hcceg1atYMQIuPxy6N0bpOTGnmriabqZD3SS1FFSI8LF1lml1lkLnAMgqQvQGCiMys+OypsCPwQ+SkzorrbLy4OMDGjQIDzn5SU7Ilcd9u0LtfRnnoE774TBg6FDBzjySDjzTLjpJnjuOWjRAq69FiZPhoULYds2+OoreOklOP98eOIJ6NMHTjgB7r4bPv002Z8sdcis4paSqLvkA0BD4HEzmyhpApBvZrOiXjh/ApoRLsDeamavSmoGPEHorSPgCTO770DHys7Otvz8/Cp9KJd8eXkwdixs315c1qQJTJoEubnJi8tVzZYtoWYeW1N///3iv3PDhiFRZ2aGxymnhOdjj624lr5lC/z1rzB1Krz+eijr1w+uuAKGD4cjjqjez1bXSVpgZtllLosn0dckT/SpISMD1qz5fnmHDrB6dU1H4ypr375Qo45tdlm8uGQt+8gjixN60aNrV2jcuOrHX7s2VBamTIGPPoLDDoMhQ0LTzvnnw6GHVv0YqcYTvatxDRpAWf+0pJBEXO2xbVuolRcl9aJa+rZtYXmDBtC5c8kaemYmtGtX/W3pZrBgQUj4Tz0VmnratoWcnJD0e/b09vwinuhdjfMafe20Zw/861/w2muhnXzxYvjkk+LlLVuWrKGfcgqcdFJodku23bvh5ZdD086sWfDdd9ClS0j4o0ZB+/YV7yOVeaJ3Nc7b6GuPTz+FV1+FV14JCX7z5lAL7tSpZA09MzMky7pQQ960Cf7yl1DTf/vtEHP//qE9/5JLoHnzZEdY8zzRu6TIy4Nx40J7a3o6TJzoSb4mbNkSLma++mp4rFwZytPTQ/v2eefB2WeHLo2pYNUqmDYt1PRXroTDD4ehQ0NNf8CA0I2zPvBE71wK27sX8vOLE/u774aypk1DQj/vvPDo1Klu1NYPlhn84x8h4U+fDt98A8ccAyNHhpp+ZmayI6xenuidSzFr1xYn9rlzQ1KTwsXJosTep0/9vdv0u+9g9uzQtPPii6F9v1u3kPBHjoTjjkt2hInnid65Om7bNnjjjeLkvnx5KG/Xrrg55pxzoE2bpIZZK23cCE8/HWr6//hH6EU0YEBI+hdfHH75pAJP9M7VMfv2hV4xRYn9nXdCrfTww8NFx6Jae5cuqd0ck2grVhS3569eDc2awbBhIen37x9u+KqrPNE7VwcUFMCcOSGxz5kTaqIAWVnFtfbTTgs3D7mq2bcvfHlOmRJ672zeDGlpobPA5ZeHLqV1jSd652qh7dvhzTdDt8dXX4Vly0L5MccU19gHDICjj05unKluxw54/vlQy3/ppXAhu0ePkPBzcurO+fdE71yS7dwZhuTdsAHeeisk9rfegl27Qg39jDNCYj//fDj5ZG+OSZYvvww9dqZMCXfkNmwY/iZXXBEGazv88GRHWD5P9M5Vg+++C4nh889DEi96Luv15s0lt+3WrbjW3q9f7U4g9dWyZaGWP21aaFZr0QIuvTTU9Pv1Cxd1axNP9M7Fadeuksk7NmmXLtu0qex9tGwZfu4fc0x4Lv26Z8/U7N6XqvbtCz2epk4NQzFv2xaG8hg1KiT9E05IdoSBJ3pX723ZEsZML6/GXfT8zTdlb9+iRfnJO7bs6KMTM3qjq52+/TaMrT9lSrhgvm8f9OoVmnZ+/OPkdm/1RO/qpV27Sg6CtWtXyeXNmx+45h372ptWXGmffQZ//nP497V4cRhq4cILQy3/ootqvndUlRO9pIHAHwgTjzxqZveWWp4OPAkcEa1zm5nNjpadAvwP0ALYB5xqZjvLO5YnelcVZmE4gClTwkW1r74KtayRI0M/6WOPLU7etWFERpcaliwJCT8vL3wBHHkkXHZZqOn36VMzF9erlOglNQRWAOcCBYSpBXPMbFnMOpOAhWb2SDTb1Gwzy5B0CPAecLmZLZbUGthkZnvLO54nencw1q4tvhGmaKKKwYPDfzSfqMLVlL17Yd68UNH4299CF9rjjy8eSvn446vv2AdK9PFcN+4FrDSzVWa2C5gODCm1jhFq7AAtgQ3R6/OAJWa2GMDMNh4oyTtXGVu2wOOPw1lnhYtj48aFSSkmTQrt7TNmhJ/QnuRdTWnYMPSkmjYt/BucPDnMzfCb38C//Rucfjr8z/+Ufy2ousST6NsB62LeF0RlscYDoyQVALOBG6PyzoBJekXSe5JuLesAksZKypeUX1hYWKkP4OqXPXvCTS1FN7JcdVXo+jZhQhiu9s034ZprfOOxcCQAABBSSURBVH5Rl3zNm8Po0WHQubVr4d57Q4K/9tpw/Wf48HBht/S1o+qQqJ6gOcBkM0sDBgFTJTUADgFOB3Kj56GSzim9sZlNMrNsM8tu27ZtgkJyqcIsjPvy85+H29QHDQo3HF15ZRiSd8UKuPNO6Ngx2ZE6V7a0NPjP/4QPPgg3Yl13Xbhh7uKLQ1fbG24IM39VV9+YeBL9eiB2kq60qCzWVcAMADN7F2gMtCHU/t80s6/MbDuhtt+jqkG7A8vLCz8XGzQIz3l5yY7o4KxfD/fdF2ZB6tED/vhH6Ns3tH1+9hk89BD88Id+F6mrO6Twb/mBB8Iv0RdeCMNcPPoo9O4NZ55ZPceNZ+6V+UAnSR0JCX4EMLLUOmuBc4DJkroQEn0h8Apwq6QmwC7gTOD3CYrdlaH0FH5r1oT3UDdmd9q2LSTyqVPDT16z0Gvh4YdDL4bWrZMdoXOJceihoTvmhReGO6efeSYMlVEd4u1eOQh4gNB18nEzmyhpApBvZrOinjZ/ApoRLszeamavRtuOAm6PymebWZnt9EW8103V1MVJuffuDVPfTZkCzz4bbkrp2LG4p0KnTsmO0Lnaz2+YqkcaNCi7nU8Kd/HVJh98UDyWyIYNYeiAor7Hp53mTTLOVcaBEn09mTa3/khPL7tGn55e87GU5Ysviu8mXLgw3E14wQWhzfJHP/LhA5yrDrVs/DVXVRMnfv+OzyZNQnmy7NgR7lIdNChMfffzn4f+xg8+GGrys2aFUQE9yTtXPbxGn2KKLriOGxf67qanhyRfUxdi9+2DTz4JY38sWRKe33gj3NzUvj3cemtoe+/SpWbicc55ok9Jubk1k9i3bIH33w/JvCixv/9+uJgK4XrBCSeEG0NGjQpdx2rbGN7O1Qee6F2F9u2DTz8trqEXPT79tHidI46AzMxwp2pmZuj7ftJJPuqjc7WBJ3pXwrZtxbX0osS+ZEkoh9ATpnNnyM4uTuqZmeHOP+8l41zt5Im+njILvXNia+hLloT29aLumS1ahCQ+ZkyooWdmhvlMfXhf5+oWT/T1wLffhj7rsU0vS5aENnYINfHjjw+J/IoripteOnTwWrpzqSBlEv3OnfCTn4SBg7Kykh1N8i1cGEbLW7QoTKFXVEtv3jwk8dzc4maXk0+GZs2SG69zrvqkTKL/7DN4+22YPTv0y66uwYHqgnnzwqh4jRuH2epHjixueika7Mw5V3+kTKLv2BH+/vcw6P/554cbdC6+ONlR1by//CV0ZezcGV55JQyB6pyr31Kqbte+fajVZ2XBJZfAY48lO6Ka9cgjYSb6U08NE3B4knfOQYolegjD2M6bB+eeC1dfDffcU32D+dcWZmGqsuuvD0OevvpqmJzYOecgBRM9QNOmoZ1+5Ej41a/C2Cq1beTGRNm7N8xOM3586Ab5t79590fnXEkp00ZfWqNGYYTENm3CyIiFhfDEE6k1UfR334XukDNmhDFk7r3Xu0M6574vrhq9pIGSlktaKem2MpanS3pd0kJJS6KJSkov3ybpl4kKPB4NGoQkP3FimHlp8ODicVjquq1b4aKLQpK/7z743e88yTvnylZhopfUEHgIuADoCuREM0rFugOYYWbdCVMNPlxq+X8DL1U93MqTQvPNn/4U2q7POQc2bkxGJIlTWAhnnx1mZZo8GX5Zo1+fzrm6Jp4afS9gpZmtMrNdwHRgSKl1DGgRvW4JbChaIOli4FNgadXDPXhXXx3mZFy0KPQtX7cumdEcvNWr4fTTYelSmDkTRo9OdkTOudounkTfDohNiwVRWazxwChJBcBs4EYASc2A/wR+U+VIE2DoUHj5ZVi/Hvr2hQ8/THZElfP++2GKvS+/hDlzQtONc85VJFG9bnKAyWaWBgwCpkpqQPgC+L2ZbTvQxpLGSsqXlF9YWJigkMrWvz/87//C7t2hZvzPf1br4RLmnXfgjDPC67feCgnfOefiEU+iXw+0j3mfFpXFugqYAWBm7wKNgTZAb+C/JK0GbgZ+JemG0gcws0lmlm1m2W3btq30h6isrKyQOI84IrR1v/JKtR+ySl54AQYMgKOOCnf/nnxysiNyztUl8ST6+UAnSR0lNSJcbJ1Vap21wDkAkroQEn2hmfUzswwzywAeAH5rZn9MWPRVcPzxIdl36hSaQJ56KtkRle3JJ8NQDiefHO767dAh2RE55+qaChO9me0BbgBeAT4k9K5ZKmmCpMHRar8ArpG0GHgKGGNW++9HPeaY0IzTt2+4uerBB5MdUUn33x9ugjrrLHjtNaiBHzvOuRSk2paPs7OzLT8/v0aPuXMn5OSEXizjxsHddye3T7pZGG75vvvgsstgyhQ47LDkxeOcq/0kLTCz7LKWpeQQCJXVuHEY9fHqq8PNVT/9aRhaIBn27Anj6t93H/zsZ/DnP3uSd85VTcoOgVBZhxwCkyaFC56//S189VVIso0b11wM27eH0SdfeCEMUnbnnX63q3Ou6jzRx5BCjf6oo+Dmm2HgQHjuOWjZsvqP/c038KMfhV41Dz8M111X/cd0ztUP3nRThn//d5g2LfTK6d8fvviieo+3YUPoIz9/Pjz9tCd551xieaIvR24uPP88rFgRbk5atap6jrNiRej1s3p1mAbx0kur5zjOufrLE/0BDBwYujV+801IxosWJXb/+fnhS2T7dnjjjTDgmnPOJZon+gr07h1uVDr00DDh+P/+b2L2O29e6B/frFloIurZMzH7dc650jzRx6FLl3CR9LjjwsTjM2dWbX9/+QsMGgQZGcV35zrnXHXxRB+nRE08XjSBd69ePoG3c65meKKvhNatYe7cg5t43CzM63r99WFsHZ/A2zlXUzzRV1KzZmHi8Zyc+CceL5rA+ze/CWPXPPssHH54jYTrnHN+w9TBaNQo9LNv27biicd9Am/nXLJ5oj9IRROPH310GAht48YwVWHTpsXrbN0aZrWaNy+MXeNzuzrnksETfRUUTTx+1FFhILRzzoEXXwxt+YWFcMEFoe/9k0+GWr1zziWDt9EnQOmJx99+O0xTuGxZGCvHk7xzLpk80SdI7MTj/fqFCbznzoULL0x2ZM65+i6uRC9poKTlklZKuq2M5emSXpe0UNISSYOi8nMlLZD0fvR8dqI/QG1SNPH48OFhAu++fZMdkXPOxdFGL6kh8BBwLlAAzJc0y8yWxax2B2GKwUckdQVmAxnAV8CPzGyDpJMJ0xG2S/BnqFWyssKdr845V1vEU6PvBaw0s1VmtguYDgwptY4BLaLXLYENAGa20Mw2ROVLgcMl+XxJzjlXg+JJ9O2AdTHvC/h+rXw8MEpSAaE2f2MZ+7kEeM/Mviu9QNJYSfmS8gsLC+MKvDbKywvj1zRoEJ7z8pIdkXPOJe5ibA4w2czSgEHAVEn79y3pJOB3wE/L2tjMJplZtpllt23bNkEh1ay8PBg7FtasCcMdrFkT3nuyd84lWzyJfj3QPuZ9WlQW6ypgBoCZvQs0BtoASEoD/gZcYWafVDXg2mrcuDCufKzt20O5c84lUzyJfj7QSVJHSY2AEcCsUuusBc4BkNSFkOgLJR0BvAjcZmbvJC7s2mft2sqVO+dcTakw0ZvZHuAGQo+ZDwm9a5ZKmiBpcLTaL4BrJC0GngLGmJlF2/0b8GtJi6LHUdXySZIsPb1y5c45V1Nk8Y6zW0Oys7MtPz8/2WFUWlEbfWzzTZMmMGlSmH/WOeeqk6QFZpZd1jK/MzZBcnNDUu/QIYyB06GDJ3nnXO3gg5olUG6uJ3bnXO3jNXrnnEtxnuidcy7FeaJ3zrkU54neOedSnCd655xLcZ7onXMuxXmid865FOeJ3jnnUpwneuecS3Ge6J1zLsV5onfOuRTnid4551KcJ3rnnEtxcSV6SQMlLZe0UtJtZSxPl/S6pIWSlkgaFLPs9mi75ZLOT2TwzjnnKlbhMMWSGgIPAecCBcB8SbPMbFnMancQZp56RFJXYDaQEb0eAZwEHAfMldTZzPYm+oM455wrWzw1+l7ASjNbZWa7gOnAkFLrGNAiet0S2BC9HgJMN7PvzOxTYGW0P+ecczUknkTfDlgX874gKos1HhglqYBQm7+xEtsiaaykfEn5hYWFcYbunHMuHom6GJsDTDazNGAQMFVS3Ps2s0lmlm1m2W3btk1QSM455yC+RL8eaB/zPi0qi3UVMAPAzN4FGgNt4tw2IfLyICMDGjQIz3l51XEU55yre+JJ9POBTpI6SmpEuLg6q9Q6a4FzACR1IST6wmi9EZIOk9QR6AT8K1HBF8nLg7FjYc0aMAvPY8d6snfOOYgj0ZvZHuAG4BXgQ0LvmqWSJkgaHK32C+AaSYuBp4AxFiwl1PSXAS8DP6uOHjfjxsH27SXLtm8P5c45V9/JzJIdQwnZ2dmWn59fqW0aNAg1+dIk2LcvQYE551wtJmmBmWWXtSwl7oxNT69cuXPO1ScpkegnToQmTUqWNWkSyp1zrr5LiUSfmwuTJkGHDqG5pkOH8D43N9mROedc8lU4BEJdkZvrid0558qSEjV655xz5fNE75xzKc4TvXPOpThP9M45l+I80TvnXIqrdXfGSioE1iQ7jipqA3yV7CBqET8fJfn5KObnoqSqnI8OZlbm8L+1LtGnAkn55d2KXB/5+SjJz0cxPxclVdf58KYb55xLcZ7onXMuxXmirx6Tkh1ALePnoyQ/H8X8XJRULefD2+idcy7FeY3eOedSnCd655xLcZ7oE0hSe0mvS1omaamkf092TMkmqaGkhZJeSHYsySbpCEnPSPpI0oeS+iQ7pmSSdEv0/+QDSU9JapzsmGqSpMclfSnpg5iyVpLmSPo4ej4yEcfyRJ9Ye4BfmFlX4IfAzyR1TXJMyfbvhLmGHfwBeNnMTgQyqcfnRVI74CYg28xOBhoCI5IbVY2bDAwsVXYbMM/MOgHzovdV5ok+gczsMzN7L3q9lfAfuV1yo0oeSWnAhcCjyY4l2SS1BM4AHgMws11mtim5USXdIcDhkg4BmgAbkhxPjTKzN4GvSxUPAZ6MXj8JXJyIY3miryaSMoDuwD+TG0lSPQDcCvgU7dARKASeiJqyHpXUNNlBJYuZrQfuB9YCnwGbzezV5EZVKxxtZp9Frz8Hjk7ETj3RVwNJzYC/Ajeb2ZZkx5MMki4CvjSzBcmOpZY4BOgBPGJm3YFvSdDP8rooanseQvgCPA5oKmlUcqOqXSz0fU9I/3dP9Akm6VBCks8zs2eTHU8SnQYMlrQamA6cLWlackNKqgKgwMyKfuE9Q0j89dUA4FMzKzSz3cCzQN8kx1QbfCHpWIDo+ctE7NQTfQJJEqEN9kMz++9kx5NMZna7maWZWQbhIttrZlZva2xm9jmwTtIJUdE5wLIkhpRsa4EfSmoS/b85h3p8cTrGLGB09Ho08FwiduqJPrFOAy4n1F4XRY9ByQ7K1Ro3AnmSlgBZwG+THE/SRL9sngHeA94n5KJ6NRyCpKeAd4ETJBVIugq4FzhX0seEXz33JuRYPgSCc86lNq/RO+dcivNE75xzKc4TvXPOpThP9M45l+I80TvnXIrzRO+ccynOE71zzqW4/w/H76R4cBiaLAAAAABJRU5ErkJggg==",
"text/plain": [
"
| \n", " | longitude | \n", "latitude | \n", "housing_median_age | \n", "total_rooms | \n", "total_bedrooms | \n", "population | \n", "households | \n", "median_income | \n", "median_house_value | \n", "ocean_proximity | \n", "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "-122.23 | \n", "37.88 | \n", "41.0 | \n", "880.0 | \n", "129.0 | \n", "322.0 | \n", "126.0 | \n", "8.3252 | \n", "452600.0 | \n", "NEAR BAY | \n", "
| 1 | \n", "-122.22 | \n", "37.86 | \n", "21.0 | \n", "7099.0 | \n", "1106.0 | \n", "2401.0 | \n", "1138.0 | \n", "8.3014 | \n", "358500.0 | \n", "NEAR BAY | \n", "
| 2 | \n", "-122.24 | \n", "37.85 | \n", "52.0 | \n", "1467.0 | \n", "190.0 | \n", "496.0 | \n", "177.0 | \n", "7.2574 | \n", "352100.0 | \n", "NEAR BAY | \n", "
| 3 | \n", "-122.25 | \n", "37.85 | \n", "52.0 | \n", "1274.0 | \n", "235.0 | \n", "558.0 | \n", "219.0 | \n", "5.6431 | \n", "341300.0 | \n", "NEAR BAY | \n", "
| 4 | \n", "-122.25 | \n", "37.85 | \n", "52.0 | \n", "1627.0 | \n", "280.0 | \n", "565.0 | \n", "259.0 | \n", "3.8462 | \n", "342200.0 | \n", "NEAR BAY | \n", "
| \n", " | min | \n", "mean | \n", "max | \n", "
|---|---|---|---|
| ocean_proximity | \n", "\n", " | \n", " | \n", " |
| <1H OCEAN | \n", "17500.0 | \n", "240084.285464 | \n", "500001.0 | \n", "
| INLAND | \n", "14999.0 | \n", "124805.392001 | \n", "500001.0 | \n", "
| ISLAND | \n", "287500.0 | \n", "380440.000000 | \n", "450000.0 | \n", "
| NEAR BAY | \n", "22500.0 | \n", "259212.311790 | \n", "500001.0 | \n", "
| NEAR OCEAN | \n", "22500.0 | \n", "249433.977427 | \n", "500001.0 | \n", "
| \n", " | housing_median_age | \n", "total_rooms | \n", "total_bedrooms | \n", "
|---|---|---|---|
| 8314 | \n", "27.0 | \n", "1675.0 | \n", "521.0 | \n", "
| 8315 | \n", "52.0 | \n", "2359.0 | \n", "591.0 | \n", "
| 8316 | \n", "52.0 | \n", "2127.0 | \n", "512.0 | \n", "
| 8317 | \n", "52.0 | \n", "996.0 | \n", "264.0 | \n", "
| 8318 | \n", "29.0 | \n", "716.0 | \n", "214.0 | \n", "
| \n", " | Make | \n", "Model | \n", "Year | \n", "Engine Fuel Type | \n", "Engine HP | \n", "Engine Cylinders | \n", "Transmission Type | \n", "Driven_Wheels | \n", "Number of Doors | \n", "Market Category | \n", "Vehicle Size | \n", "Vehicle Style | \n", "highway MPG | \n", "city mpg | \n", "Popularity | \n", "MSRP | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "BMW | \n", "1 Series M | \n", "2011 | \n", "premium unleaded (required) | \n", "335.0 | \n", "6.0 | \n", "MANUAL | \n", "rear wheel drive | \n", "2.0 | \n", "Factory Tuner,Luxury,High-Performance | \n", "Compact | \n", "Coupe | \n", "26 | \n", "19 | \n", "3916 | \n", "46135 | \n", "
| 1 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "premium unleaded (required) | \n", "300.0 | \n", "6.0 | \n", "MANUAL | \n", "rear wheel drive | \n", "2.0 | \n", "Luxury,Performance | \n", "Compact | \n", "Convertible | \n", "28 | \n", "19 | \n", "3916 | \n", "40650 | \n", "
| 2 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "premium unleaded (required) | \n", "300.0 | \n", "6.0 | \n", "MANUAL | \n", "rear wheel drive | \n", "2.0 | \n", "Luxury,High-Performance | \n", "Compact | \n", "Coupe | \n", "28 | \n", "20 | \n", "3916 | \n", "36350 | \n", "
| 3 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "premium unleaded (required) | \n", "230.0 | \n", "6.0 | \n", "MANUAL | \n", "rear wheel drive | \n", "2.0 | \n", "Luxury,Performance | \n", "Compact | \n", "Coupe | \n", "28 | \n", "18 | \n", "3916 | \n", "29450 | \n", "
| 4 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "premium unleaded (required) | \n", "230.0 | \n", "6.0 | \n", "MANUAL | \n", "rear wheel drive | \n", "2.0 | \n", "Luxury | \n", "Compact | \n", "Convertible | \n", "28 | \n", "18 | \n", "3916 | \n", "34500 | \n", "
| \n", " | make | \n", "model | \n", "year | \n", "engine_hp | \n", "engine_cylinders | \n", "transmission_type | \n", "vehicle_style | \n", "highway_mpg | \n", "city_mpg | \n", "price | \n", "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "BMW | \n", "1 Series M | \n", "2011 | \n", "335.0 | \n", "6.0 | \n", "MANUAL | \n", "Coupe | \n", "26 | \n", "19 | \n", "46135 | \n", "
| 1 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "300.0 | \n", "6.0 | \n", "MANUAL | \n", "Convertible | \n", "28 | \n", "19 | \n", "40650 | \n", "
| 2 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "300.0 | \n", "6.0 | \n", "MANUAL | \n", "Coupe | \n", "28 | \n", "20 | \n", "36350 | \n", "
| 3 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "230.0 | \n", "6.0 | \n", "MANUAL | \n", "Coupe | \n", "28 | \n", "18 | \n", "29450 | \n", "
| 4 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "230.0 | \n", "6.0 | \n", "MANUAL | \n", "Convertible | \n", "28 | \n", "18 | \n", "34500 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 11909 | \n", "Acura | \n", "ZDX | \n", "2012 | \n", "300.0 | \n", "6.0 | \n", "AUTOMATIC | \n", "4dr Hatchback | \n", "23 | \n", "16 | \n", "46120 | \n", "
| 11910 | \n", "Acura | \n", "ZDX | \n", "2012 | \n", "300.0 | \n", "6.0 | \n", "AUTOMATIC | \n", "4dr Hatchback | \n", "23 | \n", "16 | \n", "56670 | \n", "
| 11911 | \n", "Acura | \n", "ZDX | \n", "2012 | \n", "300.0 | \n", "6.0 | \n", "AUTOMATIC | \n", "4dr Hatchback | \n", "23 | \n", "16 | \n", "50620 | \n", "
| 11912 | \n", "Acura | \n", "ZDX | \n", "2013 | \n", "300.0 | \n", "6.0 | \n", "AUTOMATIC | \n", "4dr Hatchback | \n", "23 | \n", "16 | \n", "50920 | \n", "
| 11913 | \n", "Lincoln | \n", "Zephyr | \n", "2006 | \n", "221.0 | \n", "6.0 | \n", "AUTOMATIC | \n", "Sedan | \n", "26 | \n", "17 | \n", "28995 | \n", "
11914 rows × 10 columns
\n", "| \n", " | make | \n", "model | \n", "transmission_type | \n", "vehicle_style | \n", "
|---|---|---|---|---|
| count | \n", "11914 | \n", "11914 | \n", "11914 | \n", "11914 | \n", "
| unique | \n", "48 | \n", "915 | \n", "5 | \n", "16 | \n", "
| top | \n", "Chevrolet | \n", "Silverado 1500 | \n", "AUTOMATIC | \n", "Sedan | \n", "
| freq | \n", "1123 | \n", "156 | \n", "8266 | \n", "3048 | \n", "
| \n", " | year | \n", "engine_hp | \n", "engine_cylinders | \n", "highway_mpg | \n", "city_mpg | \n", "
|---|---|---|---|---|---|
| count | \n", "11914.000000 | \n", "11914.000000 | \n", "11914.000000 | \n", "11914.000000 | \n", "11914.000000 | \n", "
| mean | \n", "2010.384338 | \n", "247.941749 | \n", "5.614655 | \n", "26.637485 | \n", "19.733255 | \n", "
| std | \n", "7.579740 | \n", "110.507669 | \n", "1.800554 | \n", "8.863001 | \n", "8.987798 | \n", "
| min | \n", "1990.000000 | \n", "0.000000 | \n", "0.000000 | \n", "12.000000 | \n", "7.000000 | \n", "
| 25% | \n", "2007.000000 | \n", "170.000000 | \n", "4.000000 | \n", "22.000000 | \n", "16.000000 | \n", "
| 50% | \n", "2015.000000 | \n", "225.000000 | \n", "6.000000 | \n", "26.000000 | \n", "18.000000 | \n", "
| 75% | \n", "2016.000000 | \n", "300.000000 | \n", "6.000000 | \n", "30.000000 | \n", "22.000000 | \n", "
| max | \n", "2017.000000 | \n", "1001.000000 | \n", "16.000000 | \n", "354.000000 | \n", "137.000000 | \n", "
| \n", " | year | \n", "engine_hp | \n", "engine_cylinders | \n", "highway_mpg | \n", "city_mpg | \n", "
|---|---|---|---|---|---|
| year | \n", "1.000000 | \n", "0.338714 | \n", "-0.040708 | \n", "0.258240 | \n", "0.198171 | \n", "
| engine_hp | \n", "0.338714 | \n", "1.000000 | \n", "0.774851 | \n", "-0.415707 | \n", "-0.424918 | \n", "
| engine_cylinders | \n", "-0.040708 | \n", "0.774851 | \n", "1.000000 | \n", "-0.614541 | \n", "-0.587306 | \n", "
| highway_mpg | \n", "0.258240 | \n", "-0.415707 | \n", "-0.614541 | \n", "1.000000 | \n", "0.886829 | \n", "
| city_mpg | \n", "0.198171 | \n", "-0.424918 | \n", "-0.587306 | \n", "0.886829 | \n", "1.000000 | \n", "
| \n", " | make | \n", "model | \n", "year | \n", "engine_hp | \n", "engine_cylinders | \n", "transmission_type | \n", "vehicle_style | \n", "highway_mpg | \n", "city_mpg | \n", "above_average | \n", "
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "BMW | \n", "1 Series M | \n", "2011 | \n", "335.0 | \n", "6.0 | \n", "MANUAL | \n", "Coupe | \n", "26 | \n", "19 | \n", "1 | \n", "
| 1 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "300.0 | \n", "6.0 | \n", "MANUAL | \n", "Convertible | \n", "28 | \n", "19 | \n", "1 | \n", "
| 2 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "300.0 | \n", "6.0 | \n", "MANUAL | \n", "Coupe | \n", "28 | \n", "20 | \n", "0 | \n", "
| 3 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "230.0 | \n", "6.0 | \n", "MANUAL | \n", "Coupe | \n", "28 | \n", "18 | \n", "0 | \n", "
| 4 | \n", "BMW | \n", "1 Series | \n", "2011 | \n", "230.0 | \n", "6.0 | \n", "MANUAL | \n", "Convertible | \n", "28 | \n", "18 | \n", "0 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 11909 | \n", "Acura | \n", "ZDX | \n", "2012 | \n", "300.0 | \n", "6.0 | \n", "AUTOMATIC | \n", "4dr Hatchback | \n", "23 | \n", "16 | \n", "1 | \n", "
| 11910 | \n", "Acura | \n", "ZDX | \n", "2012 | \n", "300.0 | \n", "6.0 | \n", "AUTOMATIC | \n", "4dr Hatchback | \n", "23 | \n", "16 | \n", "1 | \n", "
| 11911 | \n", "Acura | \n", "ZDX | \n", "2012 | \n", "300.0 | \n", "6.0 | \n", "AUTOMATIC | \n", "4dr Hatchback | \n", "23 | \n", "16 | \n", "1 | \n", "
| 11912 | \n", "Acura | \n", "ZDX | \n", "2013 | \n", "300.0 | \n", "6.0 | \n", "AUTOMATIC | \n", "4dr Hatchback | \n", "23 | \n", "16 | \n", "1 | \n", "
| 11913 | \n", "Lincoln | \n", "Zephyr | \n", "2006 | \n", "221.0 | \n", "6.0 | \n", "AUTOMATIC | \n", "Sedan | \n", "26 | \n", "17 | \n", "0 | \n", "
11914 rows × 10 columns
\n", "| \n", " | MI | \n", "
|---|---|
| model | \n", "0.462344 | \n", "
| make | \n", "0.239769 | \n", "
| vehicle_style | \n", "0.084143 | \n", "
| transmission_type | \n", "0.020958 | \n", "
LogisticRegression(C=10, max_iter=1000, random_state=42, solver='liblinear')In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
LogisticRegression(C=10, max_iter=1000, random_state=42, solver='liblinear')
| \n", " | eliminated_feature | \n", "accuracy | \n", "difference | \n", "
|---|---|---|---|
| 0 | \n", "make | \n", "0.946706 | \n", "0.003294 | \n", "
| 1 | \n", "model | \n", "0.924045 | \n", "0.025955 | \n", "
| 2 | \n", "year | \n", "0.947965 | \n", "0.002035 | \n", "
| 3 | \n", "engine_hp | \n", "0.927822 | \n", "0.022178 | \n", "
| 4 | \n", "engine_cylinders | \n", "0.945867 | \n", "0.004133 | \n", "
| 5 | \n", "transmission_type | \n", "0.945027 | \n", "0.004973 | \n", "
| 6 | \n", "vehicle_style | \n", "0.932438 | \n", "0.017562 | \n", "
| 7 | \n", "highway_mpg | \n", "0.946706 | \n", "0.003294 | \n", "
| 8 | \n", "city_mpg | \n", "0.932438 | \n", "0.017562 | \n", "
| \n", " | eliminated_feature | \n", "accuracy | \n", "difference | \n", "
|---|---|---|---|
| 2 | \n", "year | \n", "0.947965 | \n", "0.002035 | \n", "
| Name | Project | Social | Links and comments |
|---|---|---|---|
| Rileen Sinha | Project |   |
More infoLinks: > Kudos & a big thanks to Alexey, Timur & the DataTalks.Club team for this wonderful course, and their generosity, patience & flexibility throughout these 5 months. Strongly recommended for anyone interested in the foundations of Machine Learning, Deep Learning, and Machine Learning Engineering. |
| Krishna Anand | Project | |
More infoLinks: |
| Peter Ernicke | Project | |
More infoLinks: > Thank you Alexey for that great journey! Feel free to connect. |
| Olga Braginskaya | Project | |
|
| Daniel Augusto Muñoz Viveros | Project | |
More infoLinks:
|
| Jorge Vladimir Abrego Arevalo | Project | |
|
| Harold Uribe Romero | Project | |
|
| Evan Hofmeister | Project | |
More infoLinks: |
| Abhijit Chakraborty | Project | |
More infoLinks: > In the lookout for challenging roles in data science/machine learning (preferably Remote) |
| Dashel Ruiz | Project | |
More infoLinks: |
| Steven Christolis | |
||
| Kabir Olawale Mohammed | Project | |
More infoLinks: |
| Inri Staka | Project | |
More infoLinks: > Hi, I am a Data Engineer with experience in Data Science and Machine Learning, you can gladly take a look at my projects and socials. |
| Daniel Takeshi | Project | |
|
| Sumedha Koranga | Project | |
|
| Mélanie Fouesnard | Project | |
More infoLinks: |
| Timur Pitsuev | |
||
| Alex Litvinov | Project | |
More infoLinks:
|
| Lokman Efe | Project | |
|
| Dese Narfa Firmansyah | Project | |
More infoLinks:
|
| Bergi Massimo | Project | |
|
| Marcos Benício de Andrade Alonso | Project | |
More infoLinks: > If you're interested in my work and would like to connect, follow my journey and contributions on LinkedIn and GitHub. |
| Ella Sahnan | Project | |
More infoLinks: |
| Sumeet Lalla | |
||
| Gabriel Galperin | Project | |
|
| Wong Chee Fah | Project | |
More infoLinks: |
| Akhyar Ahmed | Project | |
More infoLinks: |
| Phyo Min Khant | Project | |
More info> Trying to be advanced in ML and every step elevates me to a higher level. |
| David Schmid | Project | |
More info> Feel free to connect. |
| Denys Soloviov | Project | |
|
| Alla Fedorova | |
More info> That was a great course full of hands-on experience, I'm happy to recommend it and I'll be revisiting it |
|
| Alex Ionescu | Project | |
|
| Yishan Zhan | Project | |
|
| Michael Fronda | |
More info> Hello! Feel free to add me on LinkedIn :) |
|
| Mina Sonbol | Project | |
More infoLinks: > 🎊💪 |
| Sergio Andres Sosa Bautista | Project | |
More infoLinks: |
| Murat Kıran | Project | |
|
| Murat Kıran | Project | |
More infoLinks: |
| Matias Roubaud | Project | |
|
| Richard Longlands | Project | |
|
| Ekaterina Balakina | Project | |
|
| Chun Hanssen | |
||
| José Alvarado | |
||
| Cécile Guillot | Project | |
|
| Vitalii Uliantsev | |
||
| Özge Kart Tokmak | Project | |
|
| Taras Sh | |
||
| Madhav Bhatia | Project | |
More infoLinks: |
| Valery Karnevich | Project | |
|
| Aleksandra Kocot | Project | |
|
| Amit Kooner | |
||
| Murat Kahraman | Project | |
|
| Denys Savytskyi | Project | |
More infoLinks: > I have two backgrounds with master dergee's and work experience - Computer and Data Science and Cinema, TV and Art. Looking for a role where I can fully realize my potential. |
| Rahma Hayuning Astuti | |
||
| Juan Carlos Gutiérrez Concha | Project | |
More infoLinks: |
| Kevin Moore | |
More info> Available for hire! |
|
| Abhirup Ghosh | Project | |
More infoLinks: |
| Shayan Shafiee Moghadam | Project | |
|
| Mariia Boiarchuk | Project | |
More infoLinks: |
| Ryan Pramana | Project | |
More info> Email : ryan46pramana@gmail.com |
| Gonçalo Dumiense | |
||
| Adam Fieleborn | Project | |
More infoLinks: |
| Toms Kursits | |
||
| Carise F | Project | ||
| Angelo Varlotta | Project | |
More infoLinks: |
| Gopakumar Gopinathan | |
More info> Weather image classification app |
|
| Vladyslav | |
||
| Oscar Garcia | Project | |
More infoLinks: > Thanks ML Zoomcamp 2023 for such a great experience. |
| Pinku Deb Nath | Project | |
|
| David Kirwan | Project | |
|
| Isaac Ndirangu Muturi | Project | |
More infoLinks: |
| Emmanuel Eigbedion | Project | |
More info> Just grateful to your team for this amazing experience. I am also open to available job openings. |
| Avinash Vootkuri | Project | |
| \n", " | engine_displacement | \n", "num_cylinders | \n", "horsepower | \n", "vehicle_weight | \n", "acceleration | \n", "model_year | \n", "origin | \n", "fuel_type | \n", "drivetrain | \n", "num_doors | \n", "fuel_efficiency_mpg | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "170 | \n", "3.0 | \n", "159.0 | \n", "3413.433759 | \n", "17.7 | \n", "2003 | \n", "Europe | \n", "Gasoline | \n", "All-wheel drive | \n", "0.0 | \n", "13.231729 | \n", "
| 1 | \n", "130 | \n", "5.0 | \n", "97.0 | \n", "3149.664934 | \n", "17.8 | \n", "2007 | \n", "USA | \n", "Gasoline | \n", "Front-wheel drive | \n", "0.0 | \n", "13.688217 | \n", "
| 2 | \n", "170 | \n", "NaN | \n", "78.0 | \n", "3079.038997 | \n", "15.1 | \n", "2018 | \n", "Europe | \n", "Gasoline | \n", "Front-wheel drive | \n", "0.0 | \n", "14.246341 | \n", "
| 3 | \n", "220 | \n", "4.0 | \n", "NaN | \n", "2542.392402 | \n", "20.2 | \n", "2009 | \n", "USA | \n", "Diesel | \n", "All-wheel drive | \n", "2.0 | \n", "16.912736 | \n", "
| 4 | \n", "210 | \n", "1.0 | \n", "140.0 | \n", "3460.870990 | \n", "14.4 | \n", "2009 | \n", "Europe | \n", "Gasoline | \n", "All-wheel drive | \n", "2.0 | \n", "12.488369 | \n", "
| \n", " | min | \n", "mean | \n", "max | \n", "
|---|---|---|---|
| origin | \n", "\n", " | \n", " | \n", " |
| Asia | \n", "6.886245 | \n", "14.973830 | \n", "23.759123 | \n", "
| Europe | \n", "6.200971 | \n", "14.942532 | \n", "25.967222 | \n", "
| USA | \n", "6.695483 | \n", "15.040204 | \n", "24.971452 | \n", "
| \n", " | vehicle_weight | \n", "model_year | \n", "
|---|---|---|
| 8 | \n", "2714.219310 | \n", "2016 | \n", "
| 12 | \n", "2783.868974 | \n", "2010 | \n", "
| 14 | \n", "3582.687368 | \n", "2007 | \n", "
| 20 | \n", "2231.808142 | \n", "2011 | \n", "
| 21 | \n", "2659.431451 | \n", "2016 | \n", "
| 34 | \n", "2844.227534 | \n", "2014 | \n", "
| 38 | \n", "3761.994038 | \n", "2019 | \n", "
| \n", " | engine_displacement | \n", "num_cylinders | \n", "horsepower | \n", "vehicle_weight | \n", "acceleration | \n", "model_year | \n", "origin | \n", "fuel_type | \n", "drivetrain | \n", "num_doors | \n", "fuel_efficiency_mpg | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "170 | \n", "3.0 | \n", "159.0 | \n", "3413.433759 | \n", "17.7 | \n", "2003 | \n", "Europe | \n", "Gasoline | \n", "All-wheel drive | \n", "0.0 | \n", "13.231729 | \n", "
| 1 | \n", "130 | \n", "5.0 | \n", "97.0 | \n", "3149.664934 | \n", "17.8 | \n", "2007 | \n", "USA | \n", "Gasoline | \n", "Front-wheel drive | \n", "0.0 | \n", "13.688217 | \n", "
| 2 | \n", "170 | \n", "NaN | \n", "78.0 | \n", "3079.038997 | \n", "15.1 | \n", "2018 | \n", "Europe | \n", "Gasoline | \n", "Front-wheel drive | \n", "0.0 | \n", "14.246341 | \n", "
| 3 | \n", "220 | \n", "4.0 | \n", "NaN | \n", "2542.392402 | \n", "20.2 | \n", "2009 | \n", "USA | \n", "Diesel | \n", "All-wheel drive | \n", "2.0 | \n", "16.912736 | \n", "
| 4 | \n", "210 | \n", "1.0 | \n", "140.0 | \n", "3460.870990 | \n", "14.4 | \n", "2009 | \n", "Europe | \n", "Gasoline | \n", "All-wheel drive | \n", "2.0 | \n", "12.488369 | \n", "
| \n", " | engine_displacement | \n", "num_cylinders | \n", "horsepower | \n", "vehicle_weight | \n", "acceleration | \n", "model_year | \n", "num_doors | \n", "fuel_efficiency_mpg | \n", "
|---|---|---|---|---|---|---|---|---|
| count | \n", "9704.000000 | \n", "9222.000000 | \n", "8996.000000 | \n", "9704.000000 | \n", "8774.000000 | \n", "9704.000000 | \n", "9202.000000 | \n", "9704.000000 | \n", "
| mean | \n", "199.708368 | \n", "3.962481 | \n", "149.657292 | \n", "3001.280993 | \n", "15.021928 | \n", "2011.484027 | \n", "-0.006412 | \n", "14.985243 | \n", "
| std | \n", "49.455319 | \n", "1.999323 | \n", "29.879555 | \n", "497.894860 | \n", "2.510339 | \n", "6.659808 | \n", "1.048162 | \n", "2.556468 | \n", "
| min | \n", "10.000000 | \n", "0.000000 | \n", "37.000000 | \n", "952.681761 | \n", "6.000000 | \n", "2000.000000 | \n", "-4.000000 | \n", "6.200971 | \n", "
| 25% | \n", "170.000000 | \n", "3.000000 | \n", "130.000000 | \n", "2666.248985 | \n", "13.300000 | \n", "2006.000000 | \n", "-1.000000 | \n", "13.267459 | \n", "
| 50% | \n", "200.000000 | \n", "4.000000 | \n", "149.000000 | \n", "2993.226296 | \n", "15.000000 | \n", "2012.000000 | \n", "0.000000 | \n", "15.006037 | \n", "
| 75% | \n", "230.000000 | \n", "5.000000 | \n", "170.000000 | \n", "3334.957039 | \n", "16.700000 | \n", "2017.000000 | \n", "1.000000 | \n", "16.707965 | \n", "
| max | \n", "380.000000 | \n", "13.000000 | \n", "271.000000 | \n", "4739.077089 | \n", "24.300000 | \n", "2023.000000 | \n", "4.000000 | \n", "25.967222 | \n", "
| \n", " | lead_source | \n", "industry | \n", "number_of_courses_viewed | \n", "annual_income | \n", "employment_status | \n", "location | \n", "interaction_count | \n", "lead_score | \n", "converted | \n", "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "paid_ads | \n", "NaN | \n", "1 | \n", "79450.0 | \n", "unemployed | \n", "south_america | \n", "4 | \n", "0.94 | \n", "1 | \n", "
| 1 | \n", "social_media | \n", "retail | \n", "1 | \n", "46992.0 | \n", "employed | \n", "south_america | \n", "1 | \n", "0.80 | \n", "0 | \n", "
| 2 | \n", "events | \n", "healthcare | \n", "5 | \n", "78796.0 | \n", "unemployed | \n", "australia | \n", "3 | \n", "0.69 | \n", "1 | \n", "
| 3 | \n", "paid_ads | \n", "retail | \n", "2 | \n", "83843.0 | \n", "NaN | \n", "australia | \n", "1 | \n", "0.87 | \n", "0 | \n", "
| 4 | \n", "referral | \n", "education | \n", "3 | \n", "85012.0 | \n", "self_employed | \n", "europe | \n", "3 | \n", "0.62 | \n", "1 | \n", "
| \n", " | lead_source | \n", "industry | \n", "employment_status | \n", "location | \n", "
|---|---|---|---|---|
| count | \n", "1334 | \n", "1328 | \n", "1362 | \n", "1399 | \n", "
| unique | \n", "5 | \n", "7 | \n", "4 | \n", "7 | \n", "
| top | \n", "organic_search | \n", "retail | \n", "self_employed | \n", "north_america | \n", "
| freq | \n", "282 | \n", "203 | \n", "352 | \n", "225 | \n", "
| \n", " | number_of_courses_viewed | \n", "annual_income | \n", "interaction_count | \n", "lead_score | \n", "
|---|---|---|---|---|
| count | \n", "1462.000000 | \n", "1281.000000 | \n", "1462.000000 | \n", "1462.000000 | \n", "
| mean | \n", "2.031464 | \n", "59886.273224 | \n", "2.976744 | \n", "0.506108 | \n", "
| std | \n", "1.449717 | \n", "15070.140389 | \n", "1.681564 | \n", "0.288465 | \n", "
| min | \n", "0.000000 | \n", "13929.000000 | \n", "0.000000 | \n", "0.000000 | \n", "
| 25% | \n", "1.000000 | \n", "49698.000000 | \n", "2.000000 | \n", "0.262500 | \n", "
| 50% | \n", "2.000000 | \n", "60148.000000 | \n", "3.000000 | \n", "0.510000 | \n", "
| 75% | \n", "3.000000 | \n", "69639.000000 | \n", "4.000000 | \n", "0.750000 | \n", "
| max | \n", "9.000000 | \n", "109899.000000 | \n", "11.000000 | \n", "1.000000 | \n", "
| \n", " | number_of_courses_viewed | \n", "annual_income | \n", "interaction_count | \n", "lead_score | \n", "
|---|---|---|---|---|
| number_of_courses_viewed | \n", "1.000000 | \n", "0.031551 | \n", "-0.023565 | \n", "-0.004879 | \n", "
| annual_income | \n", "0.031551 | \n", "1.000000 | \n", "0.048618 | \n", "0.005334 | \n", "
| interaction_count | \n", "-0.023565 | \n", "0.048618 | \n", "1.000000 | \n", "0.009888 | \n", "
| lead_score | \n", "-0.004879 | \n", "0.005334 | \n", "0.009888 | \n", "1.000000 | \n", "
| \n", " | MI | \n", "
|---|---|
| lead_source | \n", "0.035396 | \n", "
| employment_status | \n", "0.012938 | \n", "
| industry | \n", "0.011575 | \n", "
| location | \n", "0.004464 | \n", "
LogisticRegression(max_iter=1000, random_state=42)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
LogisticRegression(max_iter=1000, random_state=42)
| \n", " | eliminated_feature | \n", "accuracy | \n", "difference | \n", "
|---|---|---|---|
| 0 | \n", "industry | \n", "0.836177 | \n", "0.020478 | \n", "
| 1 | \n", "employment_status | \n", "0.829352 | \n", "0.027304 | \n", "
| 2 | \n", "lead_score | \n", "0.815700 | \n", "0.040956 | \n", "
| \n", " | eliminated_feature | \n", "accuracy | \n", "difference | \n", "
|---|---|---|---|
| 0 | \n", "industry | \n", "0.836177 | \n", "0.020478 | \n", "
| \n", " | threshold | \n", "tp | \n", "fp | \n", "fn | \n", "tn | \n", "
|---|---|---|---|---|---|
| 0 | \n", "0.0 | \n", "171 | \n", "122 | \n", "0 | \n", "0 | \n", "
| 10 | \n", "0.1 | \n", "168 | \n", "96 | \n", "3 | \n", "26 | \n", "
| 20 | \n", "0.2 | \n", "166 | \n", "75 | \n", "5 | \n", "47 | \n", "
| 30 | \n", "0.3 | \n", "164 | \n", "50 | \n", "7 | \n", "72 | \n", "
| 40 | \n", "0.4 | \n", "158 | \n", "35 | \n", "13 | \n", "87 | \n", "
| 50 | \n", "0.5 | \n", "154 | \n", "28 | \n", "17 | \n", "94 | \n", "
| 60 | \n", "0.6 | \n", "143 | \n", "14 | \n", "28 | \n", "108 | \n", "
| 70 | \n", "0.7 | \n", "134 | \n", "11 | \n", "37 | \n", "111 | \n", "
| 80 | \n", "0.8 | \n", "125 | \n", "7 | \n", "46 | \n", "115 | \n", "
| 90 | \n", "0.9 | \n", "98 | \n", "5 | \n", "73 | \n", "117 | \n", "
| 100 | \n", "1.0 | \n", "0 | \n", "0 | \n", "171 | \n", "122 | \n", "
DecisionTreeRegressor(max_depth=1)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
DecisionTreeRegressor(max_depth=1)
RandomForestRegressor(n_estimators=10, n_jobs=-1, random_state=1)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
RandomForestRegressor(n_estimators=10, n_jobs=-1, random_state=1)
RandomForestRegressor(max_depth=20, n_estimators=10, n_jobs=-1, random_state=1)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
RandomForestRegressor(max_depth=20, n_estimators=10, n_jobs=-1, random_state=1)
| \n", " | feature | \n", "importance | \n", "
|---|---|---|
| 13 | \n", "vehicle_weight | \n", "0.959150 | \n", "
| 6 | \n", "horsepower | \n", "0.015998 | \n", "
| 0 | \n", "acceleration | \n", "0.011480 | \n", "
| 3 | \n", "engine_displacement | \n", "0.003273 | \n", "
| 7 | \n", "model_year | \n", "0.003212 | \n", "
| \n", " | lesson | \n", "name | \n", "page_name | \n", "video | \n", "slides | \n", "notebook | \n", "
|---|---|---|---|---|---|---|
| 0 | \n", "1 | \n", "Intro/Session Overview | \n", "01-intro.md | \n", "https://www.youtube.com/watch?v=2viqmJ_NpgE | \n", "\n", " | \n", " |
| 1 | \n", "2 | \n", "Building Your Prediction Service with BentoML | \n", "02-build-bento-service.md | \n", "https://www.youtube.com/watch?v=bWdEVlUw1CA | \n", "\n", " | \n", " |
| 2 | \n", "3 | \n", "Deploying Your Prediction Service | \n", "03-deploy-bento-service.md | \n", "https://www.youtube.com/watch?v=qpjLm_Lm4FA | \n", "\n", " | \n", " |
| 3 | \n", "4 | \n", "Sending, Receiving and Validating Data | \n", "04-validation.md | \n", "https://www.youtube.com/watch?v=zNYtXde0BCA | \n", "\n", " | \n", " |
| 4 | \n", "5 | \n", "High-Performance Serving | \n", "05-high-performance.md | \n", "https://www.youtube.com/watch?v=LjDYPWhJBrY | \n", "\n", " | \n", " |
| \n", " | lesson | \n", "name | \n", "page_name | \n", "video | \n", "slides | \n", "notebook | \n", "
|---|---|---|---|---|---|---|
| 0 | \n", "1 | \n", "Fashion classification | \n", "01-fashion-classification.md | \n", "https://www.youtube.com/watch?v=it1Lu7NmMpw | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 1 | \n", "2 | \n", "TensorFlow and Keras | \n", "02-tensorflow-keras.md | \n", "https://www.youtube.com/watch?v=R6o_CUmoN9Q | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 2 | \n", "3 | \n", "Pre-trained convolutional neural networks | \n", "03-pretrained-models.md | \n", "https://www.youtube.com/watch?v=qGDXEz-cr6M | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 3 | \n", "4 | \n", "Convolutional neural networks | \n", "04-conv-neural-nets.md | \n", "https://www.youtube.com/watch?v=BN-fnYzbdc8 | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 4 | \n", "5 | \n", "Transfer learning | \n", "05-transfer-learning.md | \n", "https://www.youtube.com/watch?v=WKHylqfNmq4 | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 5 | \n", "6 | \n", "Adjusting the learning rate | \n", "06-learning-rate.md | \n", "https://www.youtube.com/watch?v=2gPmRRGz0Hc | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 6 | \n", "7 | \n", "Checkpointing | \n", "07-checkpointing.md | \n", "https://www.youtube.com/watch?v=NRpGUx0o3Ps | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 7 | \n", "8 | \n", "Adding more layers | \n", "08-more-layers.md | \n", "https://www.youtube.com/watch?v=bSRRrorvAZs | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 8 | \n", "9 | \n", "Regularization and dropout | \n", "09-dropout.md | \n", "https://www.youtube.com/watch?v=74YmhVM6FTM | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 9 | \n", "10 | \n", "Data augmentation | \n", "10-augmentation.md | \n", "https://www.youtube.com/watch?v=aoPfVsS3BDE | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 10 | \n", "11 | \n", "Training a larger model | \n", "11-large-model.md | \n", "https://www.youtube.com/watch?v=_QpDGJwFjYA | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 11 | \n", "12 | \n", "Using the model | \n", "12-using-model.md | \n", "https://www.youtube.com/watch?v=cM1WHKae1wo | \n", "https://www.slideshare.net/AlexeyGrigorev/ml-z... | \n", "https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 12 | \n", "13 | \n", "Summary | \n", "13-summary.md | \n", "https://www.youtube.com/watch?v=mn0BcXJlRFM | \n", "\n", " | https://github.com/alexeygrigorev/mlbookcamp-c... | \n", "
| 13 | \n", "14 | \n", "Explore more | \n", "14-explore-more.md | \n", "\n", " | \n", " | \n", " |
| 14 | \n", "15 | \n", "Homework | \n", "homework.md | \n", "\n", " | \n", " | \n", " |
| ⚠️ | \n", "\n",
" The notes are written by the community. \n", " If you see an error here, please create a PR with a fix.\n", " | \n",
"
![](\"images/thumbnail-7-01.jpg\")
\n",
" \n",
"\n",
"\n",
"\n",
"\n",
"## Notes\n",
"\n",
"Add notes from the video (PRs are welcome)\n",
"\n",
"\n",
"| ⚠️ | \n", "\n",
" The notes are written by the community. \n", " If you see an error here, please create a PR with a fix.\n", " | \n",
"
![](\"images/thumbnail-7-02.jpg\")
\n",
" \n",
"\n",
"\n",
"\n",
"\n",
"## Notes\n",
"\n",
"Add notes from the video (PRs are welcome)\n",
"\n",
"\n",
"| ⚠️ | \n", "\n",
" The notes are written by the community. \n", " If you see an error here, please create a PR with a fix.\n", " | \n",
"
![](\"images/thumbnail-7-03.jpg\")
\n",
" \n",
"\n",
"\n",
"\n",
"\n",
"## Notes\n",
"\n",
"Add notes from the video (PRs are welcome)\n",
"\n",
"\n",
"| ⚠️ | \n", "\n",
" The notes are written by the community. \n", " If you see an error here, please create a PR with a fix.\n", " | \n",
"
![](\"images/thumbnail-7-04.jpg\")
\n",
" \n",
"\n",
"\n",
"\n",
"\n",
"## Notes\n",
"\n",
"Add notes from the video (PRs are welcome)\n",
"\n",
"\n",
"| ⚠️ | \n", "\n",
" The notes are written by the community. \n", " If you see an error here, please create a PR with a fix.\n", " | \n",
"
![](\"images/thumbnail-7-05.jpg\")
\n",
" \n",
"\n",
"\n",
"\n",
"\n",
"## Notes\n",
"\n",
"Add notes from the video (PRs are welcome)\n",
"\n",
"\n",
"| ⚠️ | \n", "\n",
" The notes are written by the community. \n", " If you see an error here, please create a PR with a fix.\n", " | \n",
"
![](\"images/thumbnail-7-06.jpg\")
\n",
" \n",
"\n",
"\n",
"\n",
"\n",
"## Notes\n",
"\n",
"Add notes from the video (PRs are welcome)\n",
"\n",
"\n",
"| ⚠️ | \n", "\n",
" The notes are written by the community. \n", " If you see an error here, please create a PR with a fix.\n", " | \n",
"
![](\"images/thumbnail-7-07.jpg\")
\n",
" \n",
"\n",
"\n",
"\n",
"\n",
"## Notes\n",
"\n",
"Add notes from the video (PRs are welcome)\n",
"\n",
"\n",
"| ⚠️ | \n", "\n",
" The notes are written by the community. \n", " If you see an error here, please create a PR with a fix.\n", " | \n",
"
![](\"images/thumbnail-7-08.jpg\")
\n",
" \n",
"\n",
"\n",
"\n",
"\n",
"## Notes\n",
"\n",
"Add notes from the video (PRs are welcome)\n",
"\n",
"\n",
"| ⚠️ | \n", "\n",
" The notes are written by the community. \n", " If you see an error here, please create a PR with a fix.\n", " | \n",
"
| ⚠️ | \n", "\n",
" The notes are written by the community. \n", " If you see an error here, please create a PR with a fix.\n", " | \n",
"
You can watch this video to see how your learning in public posts may look like:
Send a PR if you want to suggest improvements for this document
================================================
FILE: projects/README.md
================================================
## Projects
The idea is that you now apply everything we learned so far yourself.
There are three projects in this course:
* Midterm project
* Capstone 1
* Capstone 2
This is what you need to do for each project
* Think of a problem that's interesting for you and find a dataset for that
* Describe this problem and explain how a model could be used
* Prepare the data and doing EDA, analyze important features
* Train multiple models, tune their performance and select the best model
* Export the notebook into a script
* Put your model into a web service and deploy it locally with Docker
* Bonus points for deploying the service to the cloud
## Videos
[Submit and evaluate Zoomcamp projects 🎥](https://www.loom.com/share/4f5c155c550e48ddb54b71ba76516b04)
We did a few office hours sessions where we explained how you go about the projects.
Check them out:
* [ML Zoomcamp 2023 Office Hours (pre-midterm)](https://www.youtube.com/live/GuJkBzyGxyc)
* [ML Zoomcamp 2022 - Office Hours (midterm project)](https://www.youtube.com/live/AcB-Iz8h5tc)
* [2021 Office hours #7](https://www.youtube.com/live/wWBm6MHu5u8) and [2021 Office hours #8](https://www.youtube.com/live/3bo7I9LaN7o) (check #8 if you want to use text)
## Deliverables
For a project, you repository/folder should contain the following:
* `README.md` with
* Description of the problem
* Instructions on how to run the project
* Data
* You should either commit the dataset you used or have clear instructions how to download the dataset
* Notebook (suggested name - `notebook.ipynb`) with
* Data preparation and data cleaning
* EDA, feature importance analysis
* Model selection process and parameter tuning
* Script `train.py` (suggested name)
* Training the final model
* Saving it to a file (e.g. pickle) or saving it with specialized software (BentoML)
* Script `predict.py` (suggested name)
* Loading the model
* Serving it via a web service (with Flask or specialized software - BentoML, KServe, etc)
* Files with dependencies
* `Pipenv` and `Pipenv.lock` if you use Pipenv
* or equivalents: conda environment file, requirements.txt or pyproject.toml
* `Dockerfile` for running the service
* Deployment
* URL to the service you deployed or
* Video or image of how you interact with the deployed service
## Peer reviewing
> [!IMPORTANT]
> To evaluate the projects, we'll use peer reviewing. This is a great opportunity for you to learn from each other.
> * To get points for your project, you need to evaluate 3 projects of your peers
> * You get 3 extra points for each evaluation
> * If you don't evaluate your peers, you fail the project
Tip: you can use https://nbviewer.org/ to render notebooks if GitHub doesn't work
## Evaluation Criteria
The project will be evaluated using these criteria:
* Problem description
* 0 points: Problem is not described
* 1 point: Problem is described in README birefly without much details
* 2 points: Problem is described in README with enough context, so it's clear what the problem is and how the solution
will be used
* EDA
* 0 points: No EDA
* 1 point: Basic EDA (looking at min-max values, checking for missing values)
* 2 points: Extensive EDA (ranges of values, missing values, analysis of target variable, feature importance analysis)
For images: analyzing the content of the images.
For texts: frequent words, word clouds, etc
* Model training
* 0 points: No model training
* 1 point: Trained only one model, no parameter tuning
* 2 points: Trained multiple models (linear and tree-based).
For neural networks: tried multiple variations - with dropout or without, with extra inner layers or without
* 3 points: Trained multiple models and tuned their parameters.
For neural networks: same as previous, but also with tuning: adjusting learning rate, dropout rate, size of the inner layer, etc.
* Exporting notebook to script
* 0 points: No script for training a model
* 1 point: The logic for training the model is exported to a separate script
* Reproducibility
* 0 points: Not possitble to execute the notebook and the training script. Data is missing or it's not easiliy accessible
* 1 point: It's possible to re-execute the notebook and the training script without errors. The dataset is committed in the project repository or there are clear instructions on how to download the data
* Model deployment
* 0 points: Model is not deployed
* 1 point: Model is deployed (with Flask, BentoML or a similar framework)
* Dependency and enviroment management
* 0 points: No dependency management
* 1 point: Provided a file with dependencies (requirements.txt, pipfile, bentofile.yaml with dependencies, etc)
* 2 points: Provided a file with dependencies and used virtual environment. README says how to install the dependencies and how to
activate the env
* Containerization
* 0 points: No containerization
* 1 point: Dockerfile is provided or a tool that creates a docker image is used (e.g. BentoML)
* 2 points: The application is containerized and the README describes how to build a container and how to run it
* Cloud deployment
* 0 points: No deployment to the cloud
* 1 point: Docs describe clearly (with code) how to deploy the service to cloud or kubernetes cluster (local or remote)
* 2 points: There's code for deployment to cloud or kubernetes cluster (local or remote). There's a URL for testing - or video/screenshot of testing it
Total max 16 points
Or the same in table format [Criteria](https://docs.google.com/spreadsheets/d/e/2PACX-1vQCwqAtkjl07MTW-SxWUK9GUvMQ3Pv_fF8UadcuIYLgHa0PlNu9BRWtfLgivI8xSCncQs82HDwGXSm3/pubhtml)
## Cheating and plagiarism
Plagiarism in any form is not allowed. Examples of plagiarism:
* Taking somebody's else notebooks and projects (in full or partly) and using it for the capstone project
* Re-using your own projects (in full or partly) from other courses and bootcamps
* Re-using your midterm project from ML Zoomcamp in capstone
* Re-using your ML Zoomcamp from previous iterations of the course
Violating any of this will result in 0 points for this project.
## FAQ
**Q**: Can I use poetry / virtual env for managing dependencies; catboost for boosting and FastAPI for creating a web service?
> Yes, you can use any library you want. But please make sure to document everything and clearly explain what you use.
> Think of your peers who will review it - they don't necessarily know what these libraries are.
> Please give them enough context to understand your project.
**Q**: Can multiple people use the same dataset?
> Yes, there's no way to control it or enforce. So it's totally okay if you and somebody else use the same dataset.
**Q**: For peer reviewing, do I have to run the code and make sure it works?
> It's recommended that you do that, but you don't _have_ to do it.
**Q**: What if I see an error? What if I run something and it doesn't work?
> If you spot an error somewhere and you see that the code clearly doesn't work, then you
> give 0 points to the respective criterium. E.g. if you see an error in Dockerfile,
> then you give 0 points to the "containerization" dimension.
**Q**: Should I include a project title, and what happens if I leave the README file empty?
> It's highly recommended to create a new repository for your project (not inside an existing repo) with a meaningful title, such as
> "Car Price Prediction" or "Heart Risk Prediction" and including as many details as possible in the README file. ChatGPT can assist you with this. Doing so will not only make it easier to showcase your project for potential job opportunities but also have it featured on the [Projects Gallery App](#projects-gallery).
> If you leave the README file empty or with minimal details, there may be point deductions as per the [Evaluation Criteria](#evaluation-criteria).
## Resources
### Datasets
* [A list with datasets from our Data Engineering course](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/projects/datasets.md)
* [kaggle Datasets](https://www.kaggle.com/datasets) and [kaggle Competitions](https://www.kaggle.com/competitions)
* [Tensorflow Datasets](https://www.tensorflow.org/datasets/catalog/overview)
* [PyTorch - Image Datasets, Text Datasets, and Audio Datasets](https://pytorch.org/tutorials/beginner/basics/data_tutorial.html) links in first paragraph
* [UC Irvine Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php)
* [OpenML Datasets](https://www.openml.org/search?type=data)
* [OECD database](https://stats.oecd.org/index.aspx?lang=en)
* [European data](https://data.europa.eu/en)
* [Urban Data Platform Hamburg](https://www.en.urbandataplatform.hamburg/find-data)
* [New Zealand Data Sets](https://newzealand.ai/public-data-sets)
* [Open Images Dataset V7 and Extensions](https://storage.googleapis.com/openimages/web/index.html)
* [deeplake - Machine Learning Datasets](https://datasets.activeloop.ai/docs/ml/datasets/)
* [Soccer data](https://github.com/statsbomb/open-data)
* Add more data here!
### Projects Gallery
Explore a collection of projects completed by members of our community. The projects cover a wide range of topics and utilize different tools and techniques. Feel free to delve into any project and see how others have tackled real-world problems with data, structured their code, and presented their findings. It's a great resource to learn and get ideas for your own projects.
[](https://datatalksclub-projects.streamlit.app/)
### Zoomcamp 2024
* [Midterm project](../cohorts/2024/projects.md#midterm-project)
* [Capstone 1](../cohorts/2024/projects.md#capstone-1)
* [Capstone 2](../cohorts/2024/projects.md#capstone-2)
### Zoomcamp 2023
* [Midterm project](../cohorts/2023/projects.md#midterm-project)
* [Capstone 1](../cohorts/2023/projects.md#capstone-1)
* [Capstone 2](../cohorts/2023/projects.md#capstone-2)
### Zoomcamp 2022
* [Midterm project](../cohorts/2022/projects.md#midterm-project)
* [Capstone 1](../cohorts/2022/projects.md#capstone-1)
* [Capstone 2](../cohorts/2022/projects.md#capstone-2)
### Zoomcamp 2021
* [Midterm project](../cohorts/2021/07-midterm-project/)
* [Capstone 1](../cohorts/2021/12-capstone/)
* [Capstone 2](../cohorts/2021/14-project/)
================================================
FILE: projects/how-to.md
================================================
# How to Submit and Review Projects
This guide is based on [Alexey's video guide](https://www.loom.com/share/4f5c155c550e48ddb54b71ba76516b04). You can watch it for a visual guide.
## Submitting Your Project
1. Go to your course page on the course platform: https://courses.datatalks.club/
2. Open the **project submission form**.
3. Prepare your materials:
* **GitHub repo URL** (you can use a separate repo or a subfolder in your course repo).
* **Commit ID** of your final project version.
* (Optional) Up to **14 “learning in public” links** — posts where you shared your progress.
* (Optional) Link to your **FAQ contribution**, if any.
4. Review your **certificate name** — it will appear exactly as entered.
You can edit it under *Enrollment Details → Name on certificate*.
5. Submit the form.
Even after the deadline, you can still submit while the form remains open (it may be closed manually anytime).
## Reviewing Peer Projects
Once the form closes, you’ll automatically receive **3 peer review assignments**.
Each assignment includes a GitHub project link and a specific **commit hash** to review.
Peer review is part of your learning. Take time to explore how others approached their projects.
### How to review your peers?
1. Open each assigned project on GitHub (To check the correct version, use the commit link (`/commit/