Showing preview only (372K chars total). Download the full file or copy to clipboard to get everything.
Repository: alirezadir/Machine-Learning-Interviews
Branch: main
Commit: 164d43a85d86
Files: 45
Total size: 355.8 KB
Directory structure:
gitextract_pa4rkxdb/
├── LICENSE
├── README.md
└── src/
├── MLC/
│ ├── ml-coding.md
│ └── notebooks/
│ ├── .test.ipynb
│ ├── convolution.ipynb
│ ├── decision_tree.ipynb
│ ├── feedforward.ipynb
│ ├── k_means.ipynb
│ ├── k_means_2.ipynb
│ ├── k_nearest_neighbors.ipynb
│ ├── knn.ipynb
│ ├── linear_regression.ipynb
│ ├── linear_regression_md.ipynb
│ ├── logistic_regression.ipynb
│ ├── logistic_regression_md.ipynb
│ ├── numpy_practice.ipynb
│ ├── perceptron.ipynb
│ ├── softmax.ipynb
│ ├── svm.ipynb
│ └── ww_classifier.ipynb
├── MLSD/
│ ├── ml-companies.md
│ ├── ml-system-design.md
│ ├── mlsd-ads-ranking.md
│ ├── mlsd-av.md
│ ├── mlsd-event-recom.md
│ ├── mlsd-feature-eng.md
│ ├── mlsd-game-recom.md
│ ├── mlsd-harmful-content.md
│ ├── mlsd-image-search.md
│ ├── mlsd-metrics.md
│ ├── mlsd-mm-video-search.md
│ ├── mlsd-modeling-popular-archs.md
│ ├── mlsd-newsfeed.md
│ ├── mlsd-prediction.md
│ ├── mlsd-preprocessing.md
│ ├── mlsd-pymk.md
│ ├── mlsd-search.md
│ ├── mlsd-template.md
│ ├── mlsd-typeahead.md
│ ├── mlsd-video-recom.md
│ └── mlsd_obj_detection.md
├── behavior.md
├── lc-coding.md
├── ml-depth.md
└── ml-fundamental.md
================================================
FILE CONTENTS
================================================
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2021 Alireza Dirafzoon
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
[](LICENSE) [](https://github.com/psf/black) [](https://github.com/alirezadir/Machine-Learning-Interviews/stargazers) [](https://github.com/alirezadir/Machine-Learning-Interviews/network) [](https://github.com/alirezadir/Machine-Learning-Interviews/commits/main) [](https://github.com/alirezadir/Machine-Learning-Interviews/issues) [](https://github.com/alirezadir/Machine-Learning-Interviews/graphs/contributors) [](https://twitter.com/intent/tweet?text=Check%20out%20Machine%20Learning%20Interviews%20by%20%40alirezadira%20%E2%80%94%20A%20guide%20to%20prepare%20for%20ML%20interviews!&url=https%3A%2F%2Fgithub.com%2Falirezadir%2FMachine-Learning-Interviews&hashtags=MachineLearning,MLinterviews,AI)
# Machine Learning Technical Interviews :robot:
:newspaper: **News: Updated in 2025**: I have added a new repo for [Agentic AI Systems](https://github.com/alirezadir/Agentic-AI-Systems.git), including the latest trends in AI engineering and agentic systems design and development, for those who are interested. You can find a variety of resources, system design summaries, and hands-on coding examples, projects, and more.
<p align="center">
<img width="720" src="src/imgs/MLI-Book-Cover.png">
</p>
This repo aims to serve as a guide to prepare for **Machine Learning (AI) Engineering** interviews for relevant roles at big tech companies (in particular FAANG). It has compiled based on the author's personal experience and notes from his own interview preparation, when he received offers from Meta (ML Specialist), Google (ML Engineer), Amazon (Applied Scientist), Apple (Applied Scientist), and Roku (ML Engineer).
The following components are the most commonly used interview modules for technical ML roles at different companies. We will go through them one by one and share how one can prepare:
<center>
|Chapter | Content|
|---| --- |
| Chapter 1 | [General Coding (Algos and Data Structures)](src/lc-coding.md) |
| Chapter 2 | [ML Coding](src/MLC/ml-coding.md) |
| Chapter 3 | [ML Fundamentals/Breadth](src/ml-fundamental.md)|
| Chapter 4 | [ML System Design (Updated in 2023)](src/MLSD/ml-system-design.md)|
| Chapter 5 | [*Agentic AI Systems (2025)*](https://github.com/alirezadir/Agentic-AI-Systems.git)|
| Chapter 6 | [Behavioral](src/behavior.md)|
| | |
</center>
**Notes:**
* At the time I'm putting these notes together, machine learning interviews at different companies do not follow a unique structure unlike software engineering interviews. However, I found some of the components very similar to each other, although under different naming.
* The guide here is mostly focused on *Machine Learning Engineer* (and Applied Scientist) roles at big companies. Although relevant roles such as "Data Science" or "ML research scientist" have different structures in interviews, some of the modules reviewed here can be still useful.
<!-- For more understanding about different technical roles within ML umbrella you can refer to [Link](https://www.linkedin.com/pulse/machine-learning-engineer-vs-applied-scientist-whats-difference-suresh/) -->
* As a supplementary resource, you can also refer to my [Production Level Deep Learning](https://github.com/alirezadir/Production-Level-Deep-Learning) repo for further insights on how to design deep learning systems for production.
# Contribution
* Feedback and contribution are very welcome :blush:
**If you'd like to contribute**, please make a pull request with your suggested changes).
================================================
FILE: src/MLC/ml-coding.md
================================================
# <a name="ml-coding"></a> 2. ML/Data Coding :robot:
ML coding module may or may not exist in particular companies interviews. The good news is that, there are only a limited number of ML algorithms that candidates are expected to be able to code. The most common ones include:
## ML Algorithms
- Linear regression ([code](./notebooks/linear_regression.ipynb)) :white_check_mark:
- Logistic regression ([code](./notebooks/logistic_regression.ipynb)) :white_check_mark:
- K-means clustering ([code](./notebooks/k_means.ipynb)) :white_check_mark:
- K-nearest neighbors ([code 1](./notebooks/knn.ipynb) - [code 2](https://github.com/MahanFathi/CS231/blob/master/assignment1/cs231n/classifiers/k_nearest_neighbor.py)) :white_check_mark:
- Decision trees ([code](./notebooks/decision_tree.ipynb)) :white_check_mark:
<!-- (https://github.com/random-forests/tutorials/blob/master/decision_tree.py) -->
- Linear SVM ([code](./notebooks/svm.ipynb))
<!-- [link](https://towardsdatascience.com/support-vector-machine-introduction-to-machine-learning-algorithms-934a444fca47)) -->
* Neural networks
- Perceptron ([code](./notebooks/perceptron.ipynb))
- FeedForward NN ([code](./notebooks/feedforward.ipynb))
<!-- [code 1](https://github.com/alirezadir/deep-learning/blob/master/first-neural-network/my_answers.py), -->
<!-- [code 2](https://github.com/MahanFathi/CS231/blob/master/assignment1/cs231n/classifiers/neural_net.py)) -->
- Softmax ([code](./notebooks/softmax.ipynb))
- Convolution ([code](./notebooks/convolution.ipynb))
- CNN
- RNN
## Sampling
- stratified sampling ([link](https://towardsdatascience.com/the-5-sampling-algorithms-every-data-scientist-need-to-know-43c7bc11d17c))
- uniform sampling
- reservoir sampling
- sampling multinomial distribution
- random generator
## NLP algorithms
- bigrams
- tf-idf
## Other
- Random int in range ([link1](https://leetcode.com/discuss/interview-question/125347/generate-uniform-random-integer
), [link2](https://leetcode.com/articles/implement-rand10-using-rand7/))
- Triangle closing
- Meeting point
## Sample codes
- You can find some sample codes under the [Notebooks]().
================================================
FILE: src/MLC/notebooks/.test.ipynb
================================================
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Kmeans"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np \n",
"class KMeans:\n",
" def __init__(self, k, max_it=100):\n",
" self.k = k \n",
" self.max_it = max_it \n",
" # self.centroids = None \n",
" \n",
"\n",
" def fit(self, X):\n",
" # init centroids \n",
" self.centroids = X[np.random.choice(X.shape[0], size=self.k, replace=False)]\n",
" # for each it \n",
" for i in range(self.max_it):\n",
" # assign points to closest centroid \n",
" # clusters = []\n",
" # for j in range(len(X)):\n",
" # dist = np.linalg.norm(X[j] - self.centroids, axis=1)\n",
" # clusters.append(np.argmin(dist))\n",
" dist = np.linalg.norm(X[:, np.newaxis] - self.centroids, axis=2)\n",
" clusters = np.argmin(dist, axis=1)\n",
" \n",
" # update centroids (mean of clusters)\n",
" for k in range(self.k):\n",
" cluster_X = X[np.where(np.array(clusters) == k)]\n",
" if len(cluster_X) > 0 : \n",
" self.centroids[k] = np.mean(cluster_X, axis=0)\n",
" # check convergence / termination \n",
" if i > 0 and np.array_equal(self.centroids, pre_centroids): \n",
" break \n",
" pre_centroids = self.centroids \n",
" \n",
" self.clusters = clusters \n",
" \n",
" def predict(self, X):\n",
" clusters = []\n",
" for j in range(len(X)):\n",
" dist = np.linalg.norm(X[j] - self.centroids, axis=1)\n",
" clusters.append(np.argmin(dist))\n",
" return clusters \n",
" \n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0, 0, 0, 0, 0, 1, 1, 1, 1, 1]\n",
"[[ 4.62131563 5.38818365]\n",
" [-4.47889882 -4.71564167]]\n"
]
}
],
"source": [
"x1 = np.random.randn(5,2) + 5 \n",
"x2 = np.random.randn(5,2) - 5\n",
"X = np.concatenate([x1,x2], axis=0)\n",
"\n",
"\n",
"kmeans = KMeans(k=2)\n",
"kmeans.fit(X)\n",
"clusters = kmeans.predict(X)\n",
"print(clusters)\n",
"print(kmeans.centroids)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPUUlEQVR4nO3dbYisZ33H8e/vJKZ11ZBijgg5OTtKfWjqA9o1VEJta1Sihvg2sorVF0ulhgiKJi59eaDUogaUliHGNw5IiY+IT0nVQl+Yuic+NR6VELLJ8QFXoShd2hDy74uZ9Rw3Z3Zndu5zZq6z3w+EOXPPvdf9HzLnt9e55r6uK1WFJKldR+ZdgCRpNga5JDXOIJekxhnkktQ4g1ySGnfpPC565ZVXVq/Xm8elJalZJ0+e/FVVHd19fC5B3uv12NjYmMelJalZSTbPddyhFUlqnEEuSY0zyCWpcQa5JDXOIJekxhnkkg6vwQB6PThyZPg4GMy7ogOZy+2HkjR3gwGsrcH29vD55ubwOcDq6vzqOgB75JIOp/X1MyG+Y3t7eLwxBrmkw+mRR6Y7vsAMckmH0/Hj0x1fYAa5pMPpxAlYWvr9Y0tLw+ONMcglHU6rq9Dvw/IyJMPHfr+5LzrBu1YkHWarq00G9272yCWpcZ0EeZIrktyd5EdJTiV5ZRftSpL211WP/A7gK1X1QuClwKmO2pWk8+MimdUJHYyRJ7kceBXwNwBV9Rjw2KztStJ5cxHN6oRueuTPBbaATyT5TpI7kzxt90lJ1pJsJNnY2trq4LKSdEAX0axO6CbILwVeDvxzVb0M+B/gtt0nVVW/qlaqauXo0SdtOSdJF85FNKsTugny08Dpqrpv9PxuhsEuSYvpIprVCR0EeVX9Ang0yQtGh64Hfjhru5J03lxEszqhuwlBtwCDJJcBDwFv76hdSerezhea6+vD4ZTjx4ch3uAXnQCpqgt+0ZWVldrY2Ljg15WkliU5WVUru487s1OSGmeQS1LjDHJJmtBgMKDX63HkyBF6vR6DBZkN6uqHkjSBwWDA2toa26OJRJubm6yNZoOuzvlLUnvkkjSB9fX134X4ju3tbdYXYDaoQS5JE3hkzKzPcccvJINckiZwfMysz3HHLySDXJImcOLECZZ2zQZdWlrixKSzQc/jsrkGuSRNYHV1lX6/z/LyMklYXl6m3+8/+YvOcwX2zrK5m5tQdWbZ3I7C3JmdktSV3eucw3ANl6c+FX796yefv7wMDz88cfPjZnZ6+6EkdWXcOue7j+3o6ItSh1YkqSvTBnNHX5Qa5JLUlXHB/Mxnntdlcw1ySerKuHXO77gD+v3hmHgyfOz3O1s21zFySerKfuucn6ep/Aa5JHVpdfWCb1Dh0IokNc4gl6TGGeSS1DiDXJIaZ5BLUuMMcklqnEEuSY3rLMiTXJLkO0m+2FWbkrRwzuO64gfV5YSgW4FTwOUdtilJi2P3MrU764rDBZ8EdLZOeuRJjgFvBO7soj1JWkjjlqmd8wbMXQ2tfAR4H/DEuBOSrCXZSLKxtbXV0WUl6QIat0ztnDdgnjnIk9wI/LKqTu51XlX1q2qlqlaOHj0662Ul6cIbt0ztnDdg7qJHfh1wU5KHgU8Br07yyQ7alaTFMm6Z2o7WFT+omYO8qm6vqmNV1QNuBr5eVW+ZuTJJWjSrq+d1XfGDchlbSZrGHJap3U+nQV5V3wS+2WWbkqS9ObNTkhpnkEtS4wxySWqcQS5JjTPIJalxBrkkNc4gl6TGGeSS1DiDXJIaZ5BLUuMMcklqnEEuSY0zyCWpcQa5JDXOIJekxhnkktQ4g1ySGmeQS1LjDHJJapxBLkmNM8glqXEGuSQ1ziCXpMbNHORJrk7yjSSnkjyQ5NYuCpMkTebSDtp4HHhPVd2f5BnAyST3VNUPO2hbkrSPmXvkVfXzqrp/9OffAqeAq2ZtV5I0mU7HyJP0gJcB93XZriRpvM6CPMnTgU8D766q35zj9bUkG0k2tra2urqsJB16nQR5kqcwDPFBVX3mXOdUVb+qVqpq5ejRo11cVpJEN3etBPg4cKqqPjR7SZKkaXTRI78OeCvw6iTfHf33hg7alSRNYObbD6vqP4B0UIsk6QCc2SlJjTPIJalxBrkkNc4gl6TGGeSS1DiDXJIaZ5BLUuMMcklqnEEuSY0zyCWpcQa5JDXOIJekxhnkktQ4g1ySGmeQS1LjDHJJapxBLkmNM8glqXEGuSQ1ziCXpMYZ5JLUOINckhpnkEtS4zoJ8iQ3JPlxkgeT3NZFm5Kkycwc5EkuAT4GvB64BnhzkmtmbVeSNJkueuTXAg9W1UNV9RjwKeBNHbQrSZpAF0F+FfDoWc9Pj45Jki6ALoI85zhWTzopWUuykWRja2urg8tKkqCbID8NXH3W82PAz3afVFX9qlqpqpWjR492cFlJEnQT5N8GnpfkOUkuA24GvtBBu5KkCVw6awNV9XiSdwFfBS4B7qqqB2auTJI0kZmDHKCqvgR8qYu2JEnTcWanJDXOIJekxhnkktQ4g1ySGmeQS1LjDHJJapxBLkmNM8glqXEGuSQ1ziCXpMYZ5JLUOINckhpnkEtS4wxySWqcQS5JjTPIJalxBrkkNc4gl6TGGeSS1DiDXJIaZ5BLUuMMcklqnEEuSY2bKciTfDDJj5J8P8lnk1zRUV2SpAnN2iO/B3hRVb0E+Alw++wlSZKmMVOQV9XXqurx0dNvAcdmL0mSNI0ux8jfAXy5w/YkSRO4dL8TktwLPPscL61X1edH56wDjwODPdpZA9YAjh8/fqBiJUlPtm+QV9Vr9no9yduAG4Hrq6r2aKcP9AFWVlbGnidJms6+Qb6XJDcA7wf+sqq2uylJkjSNWcfIPwo8A7gnyXeT/EsHNUmSpjBTj7yq/rirQiRJB+PMTklqnEEuSY0zyCWpcQa5JDXOIJekxhnkktQ4g1ySGmeQL7DBAHo9OHJk+DgYu5KNpMNspglBOn8GA1hbg+3Rwgebm8PnAKur86tL0uKxR76g1tfPhPiO7e3hcUk6m0G+oB55ZLrjkg4vg3xBjVuy3aXcJe1mkC+oEydgaen3jy0tDY9L0tkM8gW1ugr9PiwvQzJ87Pf9olPSk3nXygJbXTW4Je3PHrkkNc4gl6TGNRPkznKUpHNrYozcWY6SNF4TPXJnOUrSeE0EubMcJWm8JoLcWY6SNF4TQe4sR0kar5MgT/LeJJXkyi7a281ZjpI03sx3rSS5GngtcF5HrJ3lKEnn1kWP/MPA+4DqoC1J0pRmCvIkNwE/rarvdVSPJGlK+w6tJLkXePY5XloHPgC8bpILJVkD1gCOe7uJJHUmVQcbEUnyYuDfgJ2pOseAnwHXVtUv9vrZlZWV2tjYONB1JemwSnKyqlZ2Hz/wl51V9QPgWWdd4GFgpap+ddA2JUnTa+I+cknSeJ0tmlVVva7akiRNzh65JDXOIJekxhnkexgMBvR6PY4cOUKv12PgbhaSFlATG0vMw2AwYG1tje3RQuibm5usjXazWHWtAEkLxB75GOvr678L8R3b29usu5uFpAVjkI/xyJhdK8Ydl6R5McjHGLeMgMsLSFo0BvkYJ06cYGnXbhZLS0uccDcLSQvGIB9jdXWVfr/P8vIySVheXqbf7/tFp6SFc+BFs2bholmSNL1xi2bZI5ekxh2qIB8MoNeDI0eGj87vkXQxODQTggYDWFuDnVvDNzeHz8G9QCW17dD0yNfXz4T4ju3t4XFJatmhCfJx83ic3yOpdYcmyMfN43F+j6TWHZogP3ECds3vYWlpeFySWnZognx1Ffp9WF6GZPjY7+//Rad3ukhadIfmrhUYhvY0d6h4p4ukFhyaHvlBeKeLpBYY5HuY5E4Xh14kzZtBvof97nTZGXrZ3ISqM0MvhrmkC8kg38N+d7o49CJpERjke9jvThcnGUlaBDMHeZJbkvw4yQNJ/rGLohbJ6io8/DA88cTw8ey7VZxkJGkRzBTkSf4aeBPwkqr6U+CfOqmqEU4ykrQIZu2RvxP4h6r6P4Cq+uXsJbXjoJOMJKlLM+0QlOS7wOeBG4D/Bd5bVd8ec+4asAZw/PjxP9vc3DzwdSXpMBq3Q9C+MzuT3As8+xwvrY9+/o+APwdeAfxrkufWOX47VFUf6MNwq7fpypckjbNvkFfVa8a9luSdwGdGwf2fSZ4ArgS2uitRkrSXWcfIPwe8GiDJ84HLgF/N2KYkaQqzLpp1F3BXkv8CHgPedq5hFUnS+TNTkFfVY8BbOqpFknQAM921cuCLJlvAIt22ciXtDwm1/h6sf/5afw+Hof7lqjq6++BcgnzRJNk41y09LWn9PVj//LX+Hg5z/a61IkmNM8glqXEG+VB/3gV0oPX3YP3z1/p7OLT1O0YuSY2zRy5JjTPIJalxBvlZLoZNMpK8N0kluXLetUwryQeT/CjJ95N8NskV865pEkluGH1uHkxy27zrmUaSq5N8I8mp0ef+1nnXdBBJLknynSRfnHctB5HkiiR3jz7/p5K8cpqfN8hHLoZNMpJcDbwWaHWzuXuAF1XVS4CfALfPuZ59JbkE+BjweuAa4M1JrplvVVN5HHhPVf0Jw1VM/66x+nfcCpyadxEzuAP4SlW9EHgpU74Xg/yMi2GTjA8D7wOa/Aa7qr5WVY+Pnn4LODbPeiZ0LfBgVT00WrLiUww7BE2oqp9X1f2jP/+WYYBcNd+qppPkGPBG4M5513IQSS4HXgV8HIZLn1TVf0/ThkF+xvOBv0hyX5J/T/KKeRc0jSQ3AT+tqu/Nu5aOvAP48ryLmMBVwKNnPT9NY0G4I0kPeBlw35xLmdZHGHZgnphzHQf1XIZLf39iNDx0Z5KnTdPArKsfNqWrTTLmZZ/6PwC87sJWNL293kNVfX50zjrDf/IPLmRtB5RzHFuYz8ykkjwd+DTw7qr6zbzrmVSSG4FfVtXJJH8153IO6lLg5cAtVXVfkjuA24C/n6aBQ6P1TTLG1Z/kxcBzgO8lgeGQxP1Jrq2qX1zAEve11/8DgCRvA24Erl+kX6J7OA1cfdbzY8DP5lTLgSR5CsMQH1TVZ+Zdz5SuA25K8gbgD4HLk3yyqlpalfU0cLqqdv4ldDfDIJ+YQytnfI5GN8moqh9U1bOqqldVPYYfjJcvWojvJ8kNwPuBm6pqe971TOjbwPOSPCfJZcDNwBfmXNPEMvzN/3HgVFV9aN71TKuqbq+qY6PP/c3A1xsLcUZ/Tx9N8oLRoeuBH07TxqHqke/DTTLm76PAHwD3jP5l8a2q+tv5lrS3qno8ybuArwKXAHdV1QNzLmsa1wFvBX4w2kwd4ANV9aX5lXQo3QIMRp2Bh4C3T/PDTtGXpMY5tCJJjTPIJalxBrkkNc4gl6TGGeSS1DiDXJIaZ5BLUuP+H8mBYH+I9lNrAAAAAElFTkSuQmCC",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"from matplotlib import pyplot as plt \n",
"\n",
"colors = ['b', 'r']\n",
"for k in range(kmeans.k):\n",
" plt.scatter(X[np.where(np.array(clusters) == k)][:,0], \n",
" X[np.where(np.array(clusters) == k)][:,1], \n",
" color=colors[k])\n",
"plt.scatter(kmeans.centroids[:,0], kmeans.centroids[:,1], color='black')\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(10, 1, 2)"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X[:, np.newaxis] "
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### KNN"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(100, 2) (100,)\n",
"[0. 0. 1. 0. 1. 1. 1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 1. 1. 0.]\n",
"[0. 0. 1. 0. 1. 1. 1. 0. 0. 0. 1. 0. 0. 0. 0. 1. 0. 1. 1. 1.]\n"
]
}
],
"source": [
"import numpy as np \n",
"from collections import Counter\n",
"class KNN:\n",
" def __init__(self, k):\n",
" self.k = k \n",
" \n",
" \n",
" def fit(self, X, y):\n",
" self.X = X\n",
" self.y = y \n",
" \n",
" def predict(self, X_test):\n",
" y_pred = []\n",
" for x in X_test: \n",
" dist = np.linalg.norm(x - self.X, axis=1)\n",
" knn_idcs = np.argsort(dist)[:self.k]\n",
" knn_labels = self.y[knn_idcs]\n",
" label = Counter(knn_labels).most_common(1)[0][0]\n",
" y_pred.append(label)\n",
" return np.array(y_pred)\n",
"\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"x1 = np.random.randn(50,2) + 1\n",
"x2 = np.random.randn(50,2) - 1\n",
"X = np.concatenate([x1, x2], axis=0)\n",
"y = np.concatenate([np.ones(50), np.zeros(50)])\n",
"print(X.shape, y.shape)\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
"\n",
"\n",
"knn = KNN(k=5)\n",
"knn.fit(X_train, y_train)\n",
"y_pred = knn.predict(X_test)\n",
"print(y_pred)\n",
"print(y_test)\n"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(40, 2)"
]
},
"execution_count": 59,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_test.shape"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0., 0.])"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.zeros(2,)"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1., 1., 1., 0., 0., 0.])"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.concatenate([np.ones(3), np.zeros(3)])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Lin Regression "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class LinearRegression: \n",
" def __init__(self):\n",
" self.m = None \n",
" self.b = None \n",
" \n",
" def fit(self, X, y):\n",
" \n",
"\n",
"\n",
" def predict(self, X):\n",
" pass "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: src/MLC/notebooks/convolution.ipynb
================================================
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Convolution "
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2D convolution "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"def convolve(signal, kernel):\n",
" output = []\n",
" kernel_size = len(kernel)\n",
" padding = kernel_size // 2 # assume zero padding\n",
" padded_signal = [0] * padding + signal + [0] * padding\n",
" \n",
" for i in range(padding, len(signal) + padding):\n",
" sum = 0\n",
" for j in range(kernel_size):\n",
" sum += kernel[j] * padded_signal[i - padding + j]\n",
" output.append(sum)\n",
" \n",
" return output\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[-2, -2, -2, -2, -2, 5]\n"
]
}
],
"source": [
"signal = [1, 2, 3, 4, 5, 6]\n",
"kernel = [1, 0, -1]\n",
"output = convolve(signal, kernel)\n",
"print(output)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3D convolution "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"def convolution(image, kernel):\n",
" # get the size of the input image and kernel\n",
" (image_height, image_width, image_channels) = image.shape\n",
" (kernel_height, kernel_width, kernel_channels) = kernel.shape\n",
" \n",
" # calculate the padding needed for 'same' convolution\n",
" pad_h = (kernel_height - 1) // 2\n",
" pad_w = (kernel_width - 1) // 2\n",
" \n",
" # pad the input image with zeros\n",
" padded_image = np.pad(image, ((pad_h, pad_h), (pad_w, pad_w), (0, 0)), 'constant')\n",
" \n",
" # create an empty output tensor\n",
" output_height = image_height\n",
" output_width = image_width\n",
" output_channels = kernel_channels\n",
" output = np.zeros((output_height, output_width, output_channels))\n",
" \n",
" # perform the convolution operation\n",
" for i in range(output_height):\n",
" for j in range(output_width):\n",
" for k in range(output_channels):\n",
" output[i, j, k] = np.sum(kernel[:, :, k] * padded_image[i:i+kernel_height, j:j+kernel_width, :])\n",
" \n",
" return output\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Input image:\n",
"[[[ 1 2]\n",
" [ 3 4]]\n",
"\n",
" [[ 5 6]\n",
" [ 7 8]]\n",
"\n",
" [[ 9 10]\n",
" [11 12]]]\n",
"\n",
"Kernel:\n",
"[[[ 1 0]\n",
" [ 0 -1]]\n",
"\n",
" [[ 0 1]\n",
" [-1 0]]]\n",
"\n",
"Output:\n",
"[[[-6. 2.]\n",
" [-2. -2.]]\n",
"\n",
" [[-6. 2.]\n",
" [-2. -2.]]\n",
"\n",
" [[-3. 1.]\n",
" [-1. -1.]]]\n"
]
}
],
"source": [
"# create an example image and kernel\n",
"image = np.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]], [[9, 10], [11, 12]]])\n",
"kernel = np.array([[[1, 0], [0, -1]], [[0, 1], [-1, 0]]])\n",
"\n",
"# perform the convolution operation\n",
"output = convolution(image, kernel)\n",
"\n",
"print('Input image:')\n",
"print(image)\n",
"\n",
"print('\\nKernel:')\n",
"print(kernel)\n",
"\n",
"print('\\nOutput:')\n",
"print(output)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: src/MLC/notebooks/decision_tree.ipynb
================================================
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"A decision tree is a type of machine learning algorithm used for classification and regression tasks. It consists of a tree-like structure where each internal node represents a feature or attribute, each branch represents a decision based on that feature, and each leaf node represents a predicted output.\n",
"\n",
"To **train** a decision tree, the algorithm uses a dataset with labeled examples to create the tree structure. It starts with the root node, which includes all the examples, and selects the feature that provides the most information gain to split the data into two subsets. It then repeats this process for each subset until it reaches a stopping criterion, such as a maximum tree depth or minimum number of examples in a leaf node.\n",
"\n",
"Once the decision tree is trained, it can be used to **predict** the output for new, unseen examples. To make a prediction, the algorithm starts at the root node and follows the branches based on the values of the input features until it reaches a leaf node. The predicted output for that example is the value associated with the leaf node.\n",
"\n",
"Decision trees have several advantages, such as being easy to interpret and visualize, handling both numerical and categorical data, and handling missing values. However, they can also suffer from overfitting if the tree is too complex or if there is noise or outliers in the data. \n",
"\n",
"To address this issue, various techniques such as pruning, ensemble methods, and regularization can be used to simplify the decision tree or combine multiple trees to improve generalization performance. Additionally, decision trees may not perform well with highly imbalanced datasets or datasets with many irrelevant features, and they may not be suitable for tasks where the relationships between features and outputs are highly nonlinear or complex."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class DecisionTree:\n",
" def __init__(self, max_depth=None):\n",
" self.max_depth = max_depth\n",
" \n",
" def fit(self, X, y):\n",
" self.n_classes_ = len(np.unique(y))\n",
" self.n_features_ = X.shape[1]\n",
" self.tree_ = self._grow_tree(X, y)\n",
" \n",
" def predict(self, X):\n",
" return [self._predict(inputs) for inputs in X]\n",
" \n",
" def _gini(self, y):\n",
" _, counts = np.unique(y, return_counts=True)\n",
" impurity = 1 - np.sum([(count / len(y)) ** 2 for count in counts])\n",
" return impurity\n",
" \n",
" def _best_split(self, X, y):\n",
" m = y.size\n",
" if m <= 1:\n",
" return None, None\n",
" \n",
" num_parent = [np.sum(y == c) for c in range(self.n_classes_)]\n",
" best_gini = 1.0 - sum((n / m) ** 2 for n in num_parent)\n",
" best_idx, best_thr = None, None\n",
" \n",
" for idx in range(self.n_features_):\n",
" thresholds, classes = zip(*sorted(zip(X[:, idx], y)))\n",
" num_left = [0] * self.n_classes_\n",
" num_right = num_parent.copy()\n",
" for i in range(1, m):\n",
" c = classes[i - 1]\n",
" num_left[c] += 1\n",
" num_right[c] -= 1\n",
" gini_left = 1.0 - sum(\n",
" (num_left[x] / i) ** 2 for x in range(self.n_classes_)\n",
" )\n",
" gini_right = 1.0 - sum(\n",
" (num_right[x] / (m - i)) ** 2 for x in range(self.n_classes_)\n",
" )\n",
" gini = (i * gini_left + (m - i) * gini_right) / m\n",
" if thresholds[i] == thresholds[i - 1]:\n",
" continue\n",
" if gini < best_gini:\n",
" best_gini = gini\n",
" best_idx = idx\n",
" best_thr = (thresholds[i] + thresholds[i - 1]) / 2\n",
" \n",
" return best_idx, best_thr\n",
" \n",
" def _grow_tree(self, X, y, depth=0):\n",
" num_samples_per_class = [np.sum(y == i) for i in range(self.n_classes_)]\n",
" predicted_class = np.argmax(num_samples_per_class)\n",
" node = Node(predicted_class=predicted_class)\n",
" if depth < self.max_depth:\n",
" idx, thr = self._best_split(X, y)\n",
" if idx is not None:\n",
" indices_left = X[:, idx] < thr\n",
" X_left, y_left = X[indices_left], y[indices_left]\n",
" X_right, y_right = X[~indices_left], y[~indices_left]\n",
" node.feature_index = idx\n",
" node.threshold = thr\n",
" node.left = self._grow_tree(X_left, y_left, depth + 1)\n",
" node.right = self._grow_tree(X_right, y_right, depth + 1)\n",
" return node\n",
" \n",
" def _predict(self, inputs):\n",
" node = self.tree_\n",
" while node.left:\n",
" if inputs[node.feature_index] < node.threshold:\n",
" node = node.left\n",
" else:\n",
" node = node.right\n",
" return node.predicted_class\n",
" \n",
"class Node:\n",
" def __init__(self, *, predicted_class):\n",
" self.predicted_class = predicted_class\n",
" self.feature_index = 0\n",
" self.threshold = 0.0 \n",
" self.left = None\n",
" self.right = None\n",
"\n",
" def is_leaf_node(self):\n",
" return self.left is None and self.right is None\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy: 1.0\n"
]
}
],
"source": [
"from sklearn.datasets import load_iris\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.metrics import accuracy_score\n",
"\n",
"# Load the iris dataset\n",
"iris = load_iris()\n",
"X = iris.data\n",
"y = iris.target\n",
"\n",
"# Split the data into training and testing sets\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
"\n",
"# Train the decision tree\n",
"tree = DecisionTree(max_depth=3)\n",
"tree.fit(X_train, y_train)\n",
"\n",
"# Make predictions on the test set\n",
"y_pred = tree.predict(X_test)\n",
"\n",
"# Compute the accuracy of the predictions\n",
"accuracy = accuracy_score(y_test, y_pred)\n",
"\n",
"print(f\"Accuracy: {accuracy}\")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: src/MLC/notebooks/feedforward.ipynb
================================================
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Forward propagation:"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"```python\n",
"Z1 = X.W1 + b1\n",
"A1 = ReLU(Z1) \n",
"Z2 = A1.W2 + b2\n",
"exp_scores = exp(Z2) \n",
"probs = exp_scores / sum(exp_scores)\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Backward propagation:\n",
"\n",
"```python \n",
"delta3 = probs\n",
"delta3[range(len(X)), y] -= 1\n",
"dW2 = A1.T.dot(delta3)\n",
"db2 = sum(delta3)\n",
"delta2 = delta3.dot(W2.T) * (A1 > 0)\n",
"dW1 = X.T.dot(delta2)\n",
"db1 = sum(delta2)\n",
"\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Here:\n",
"\n",
"X is the input data matrix of shape (num_samples, input_size), W1 is the weight matrix connecting the input layer to the hidden layer of shape (input_size, hidden_size), b1 is the bias vector for the hidden layer of shape (hidden_size,), A1 is the output of the hidden layer (also known as the hidden representation) of shape (num_samples, hidden_size), W2 is the weight matrix connecting the hidden layer to the output layer of shape (hidden_size, output_size), b2 is the bias vector for the output layer of shape (output_size,), Z2 is the weighted sum of the hidden layer output, exp_scores is the exponential of the output layer weighted sum, probs is the output probability for each class, and y is the true label vector of shape (num_samples,).\n",
"\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Code"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class TwoLayerNet:\n",
" def __init__(self, input_size, hidden_size, output_size):\n",
" self.params = {}\n",
" self.params['W1'] = np.random.randn(input_size, hidden_size)\n",
" self.params['b1'] = np.zeros(hidden_size)\n",
" self.params['W2'] = np.random.randn(hidden_size, output_size)\n",
" self.params['b2'] = np.zeros(output_size)\n",
"\n",
" def forward(self, X):\n",
" W1, b1 = self.params['W1'], self.params['b1']\n",
" W2, b2 = self.params['W2'], self.params['b2']\n",
" z1 = np.dot(X, W1) + b1\n",
" a1 = np.maximum(0, z1) # ReLU activation function\n",
" z2 = np.dot(a1, W2) + b2\n",
" # probs = 1 / (1 + np.exp(-z2)) # Sigmoid activation function\n",
" exp_z = np.exp(z2)\n",
" probs = exp_z / np.sum(exp_z, axis=1, keepdims=True)\n",
" return probs\n",
"\n",
" def loss(self, X, y):\n",
" probs = self.forward(X)\n",
" correct_logprobs = -np.log(probs[range(len(X)), y])\n",
" data_loss = np.sum(correct_logprobs)\n",
" return 1.0/len(X) * data_loss\n",
"\n",
" def train(self, X, y, num_epochs, learning_rate=0.1):\n",
" for epoch in range(num_epochs):\n",
" # Forward propagation\n",
" z1 = np.dot(X, self.params['W1']) + self.params['b1']\n",
" a1 = np.maximum(0, z1) # ReLU activation function\n",
" z2 = np.dot(a1, self.params['W2']) + self.params['b2']\n",
" # probs = 1 / (1 + np.exp(-z2)) # Sigmoid activation function\n",
" exp_z = np.exp(z2)\n",
" probs = exp_z / np.sum(exp_z, axis=1, keepdims=True)\n",
"\n",
" # Backpropagation\n",
" delta3 = probs\n",
" delta3[range(len(X)), y] -= 1\n",
" dW2 = np.dot(a1.T, delta3)\n",
" db2 = np.sum(delta3, axis=0)\n",
" delta2 = np.dot(delta3, self.params['W2'].T) * (a1 > 0) # derivative of ReLU\n",
" dW1 = np.dot(X.T, delta2)\n",
" db1 = np.sum(delta2, axis=0)\n",
"\n",
" # Update parameters\n",
" self.params['W1'] -= learning_rate * dW1\n",
" self.params['b1'] -= learning_rate * db1\n",
" self.params['W2'] -= learning_rate * dW2\n",
" self.params['b2'] -= learning_rate * db2\n",
"\n",
" # Print loss for monitoring training progress\n",
" if epoch % 100 == 0:\n",
" loss = self.loss(X, y)\n",
" print(\"Epoch {}: loss = {}\".format(epoch, loss))\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This code defines a TwoLayerNet class with an initializer that takes the input size, hidden size, and output size as arguments. The weights and biases for the two layers are initialized randomly in this function.\n",
"\n",
"The forward function takes an input X and performs the forward propagation to calculate the output probabilities for each class.\n",
"\n",
"The loss function calculates the cross-entropy loss between the predicted probabilities and the true labels y.\n",
"\n",
"The train function performs the backpropagation to update the weights and biases based on the calculated gradients. The number of training epochs and learning rate can be specified as arguments to this function."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here's an example of how to use the TwoLayerNet class to train and test the network on a toy dataset:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 0: loss = 0.8791617000548932\n",
"Epoch 100: loss = 0.03272609589944909\n",
"Epoch 200: loss = 0.010130354895034843\n",
"Epoch 300: loss = 0.005517334222420798\n",
"Epoch 400: loss = 0.0036701620853277555\n",
"Epoch 500: loss = 0.002707635703438397\n",
"Epoch 600: loss = 0.0021206045443387493\n",
"Epoch 700: loss = 0.0017317523015295431\n",
"Epoch 800: loss = 0.0014568091215886065\n",
"Epoch 900: loss = 0.0012539964886349238\n",
"Predictions: [0 1 1 0]\n"
]
}
],
"source": [
"# Generate a toy dataset\n",
"X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n",
"y = np.array([0, 1, 1, 0])\n",
"\n",
"# Initialize a neural network\n",
"net = TwoLayerNet(input_size=2, hidden_size=10, output_size=2)\n",
"\n",
"# Train the neural network\n",
"net.train(X, y, num_epochs=1000)\n",
"\n",
"# Test the neural network\n",
"probs = net.forward(X)\n",
"predictions = np.argmax(probs, axis=1)\n",
"print(\"Predictions: \", predictions)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Improvements \n",
"\n",
"There are several ways to improve the implementation of a two-layer neural network with softmax. Here are a few suggestions:\n",
"\n",
"1. Weight initialization: The current implementation initializes the weights randomly using a Gaussian distribution. However, it is recommended to use other weight initialization methods such as Xavier or He initialization to improve convergence and avoid vanishing or exploding gradients. One possible implementation for Xavier initialization of the weights is:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Xavier initialization\n",
"self.params['W1'] = np.random.randn(input_size, hidden_size) / np.sqrt(input_size)\n",
"self.params['W2'] = np.random.randn(hidden_size, output_size) / np.sqrt(hidden_size)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"2. Learning rate decay: The learning rate is a hyperparameter that determines the step size at each iteration during training. However, using a fixed learning rate may lead to suboptimal performance or slow convergence. A common technique is to gradually decrease the learning rate over time, known as learning rate decay, to fine-tune the network weights as the optimization process progresses."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Learning rate decay\n",
"learning_rate = 0.1\n",
"lr_decay = 0.95\n",
"lr_decay_epoch = 100\n",
"for epoch in range(num_epochs):\n",
" # ...\n",
" if epoch % lr_decay_epoch == 0:\n",
" learning_rate *= lr_decay"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"3. Regularization: Overfitting can occur when the model is too complex and the training data is limited. Regularization techniques such as L1 or L2 regularization can be applied to the loss function to prevent overfitting and improve the generalization performance of the model.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# L2 regularization\n",
"reg_lambda = 0.1\n",
"data_loss += 0.5 * reg_lambda * (np.sum(self.params['W1'] ** 2) + np.sum(self.params['W2'] ** 2))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"4. Mini-batch training: The current implementation updates the weights using the entire training set at each iteration, which can be computationally expensive for large datasets. An alternative is to use mini-batch training, where a random subset of the training data is used at each iteration to update the weights. This can speed up the training process and improve convergence."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Mini-batch training\n",
"batch_size = 64\n",
"num_batches = len(X) // batch_size\n",
"for epoch in range(num_epochs):\n",
" for i in range(num_batches):\n",
" # Select a random batch of data\n",
" batch_mask = np.random.choice(len(X), batch_size)\n",
" X_batch = X[batch_mask]\n",
" y_batch = y[batch_mask]\n",
"\n",
" # Forward and backward propagation using the batch data\n",
" # ...\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"5. Optimization algorithm: The current implementation uses stochastic gradient descent (SGD) as the optimization algorithm. However, there are other optimization algorithms such as Adam, Adagrad, and RMSprop that can improve the convergence speed and performance of the network."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Adam optimization\n",
"beta1, beta2 = 0.9, 0.999\n",
"eps = 1e-8\n",
"mW1, vW1 = 0, 0\n",
"mW2, vW2 = 0, 0\n",
"for epoch in range(num_epochs):\n",
" # Forward and backward propagation\n",
" # ...\n",
" # Update parameters using Adam optimization\n",
" mW1 = beta1 * mW1 + (1 - beta1) * dW1\n",
" vW1 = beta2 * vW1 + (1 - beta2) * (dW1 ** 2)\n",
" mW2 = beta1 * mW2 + (1 - beta1) * dW2\n",
" vW2 = beta2 * vW2 + (1 - beta2) * (dW2 ** 2)\n",
" self.params['W1'] -= learning_rate * mW1 / (np.sqrt(vW1) + eps)\n",
" self.params['b1'] -= learning_rate * db1\n",
" self.params['W2'] -= learning_rate * mW2 / (np.sqrt(vW2) + eps)\n",
" self.params['b2'] -= learning_rate * db2\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Other extensions: "
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"* Arbitrary activation function \n",
"* Arbitrary loss function \n",
"* Extension to multiple layers"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"\n",
"import numpy as np\n",
"\n",
"class ActivationFunction:\n",
" def __init__(self):\n",
" pass\n",
"\n",
" def __call__(self, x):\n",
" raise NotImplementedError\n",
"\n",
" def derivative(self, x):\n",
" raise NotImplementedError\n",
"\n",
"class ReLU(ActivationFunction):\n",
" def __init__(self):\n",
" super().__init__()\n",
"\n",
" def __call__(self, x):\n",
" return np.maximum(0, x)\n",
"\n",
" def derivative(self, x):\n",
" return (x > 0).astype(float)\n",
"\n",
"class Softmax(ActivationFunction):\n",
" def __init__(self):\n",
" super().__init__()\n",
"\n",
" def __call__(self, x):\n",
" exp_scores = np.exp(x - np.max(x, axis=1, keepdims=True))\n",
" probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)\n",
" return probs\n",
"\n",
" def derivative(self, x):\n",
" raise NotImplementedError\n",
"\n",
"class MultiLayerNet:\n",
" def __init__(self, input_size, hidden_sizes, output_size, activation_function, loss_function, reg_lambda=0.0):\n",
" self.params = {}\n",
" self.num_layers = 1 + len(hidden_sizes)\n",
" self.layer_sizes = [input_size] + hidden_sizes + [output_size]\n",
"\n",
" for i in range(1, self.num_layers + 1):\n",
" self.params[f'W{i}'] = np.random.randn(self.layer_sizes[i-1], self.layer_sizes[i]) / np.sqrt(self.layer_sizes[i-1])\n",
" self.params[f'b{i}'] = np.zeros(self.layer_sizes[i])\n",
"\n",
" self.activation_function = activation_function\n",
" self.activation_function_derivatives = [activation_function.derivative for _ in range(self.num_layers)]\n",
" self.loss_function = loss_function\n",
" self.reg_lambda = reg_lambda\n",
"\n",
" def forward(self, X):\n",
" layer_output = X\n",
" self.layer_inputs = []\n",
" self.layer_outputs = [X]\n",
"\n",
" for i in range(1, self.num_layers + 1):\n",
" W, b = self.params[f'W{i}'], self.params[f'b{i}']\n",
" layer_input = np.dot(layer_output, W) + b\n",
" self.layer_inputs.append(layer_input)\n",
" layer_output = self.activation_function(layer_input)\n",
" self.layer_outputs.append(layer_output)\n",
"\n",
" return layer_output\n",
"\n",
" def backward(self, X, y, output):\n",
" delta = output - y\n",
" dW = {}\n",
" db = {}\n",
" delta = delta / X.shape[0]\n",
"\n",
" for i in reversed(range(1, self.num_layers + 1)):\n",
" layer_input = self.layer_inputs[i-1]\n",
" activation_derivative = self.activation_function_derivatives[i-1](layer_input)\n",
" dW[f'W{i}'] = np.dot(self.layer_outputs[i-1].T, delta) + self.reg_lambda * self.params[f'W{i}']\n",
" db[f'b{i}'] = np.sum(delta, axis=0)\n",
" delta = np.dot(delta, self.params[f'W{i}'].T) * activation_derivative\n",
"\n",
" return dW, db\n",
"\n",
" def loss(self, X, y, output):\n",
" data_loss = self.loss_function(output, y)\n",
" reg_loss = 0.0\n",
"\n",
" for i in range(1, self.num_layers + 1):\n",
" reg_loss += 0.5 * self.reg_lambda * np.sum(self.params[f'W{i}'] ** 2)\n",
"\n",
" total_loss = data_loss + reg_loss\n",
" return total_loss\n",
"\n",
" def train(self, X, y, num_epochs, learning_rate=0.1):\n",
" for epoch in range(num_epochs):\n",
" # Forward propagation\n",
" output = self.forward(X)\n",
"\n",
" # Backward propagation\n",
" dW, db = self.backward(X, y, output)\n",
"\n",
" # Update parameters\n",
" for i in range(1, self.num_layers + 1):\n",
" self.params[f'W{i}'] -= learning_rate * dW[f'W{i}']\n",
" self.params[f'b{i}'] -= learning_rate * db[f'b{i}']\n",
"\n",
" # Print loss for monitoring training progress\n",
" if epoch % 100 == 0:\n",
" loss = self.loss(X, y, output)\n",
" print(f\"Epoch {epoch}, loss: {loss}\")\n",
"\n",
"\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from sklearn.datasets import make_classification\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"# Generate a toy classification dataset\n",
"X, y = make_classification(n_samples=1000, n_features=10, n_classes=2, random_state=42)\n",
"\n",
"# Split the dataset into training and testing sets\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
"\n",
"# Normalize the input data\n",
"mean = X_train.mean(axis=0)\n",
"std = X_train.std(axis=0)\n",
"X_train = (X_train - mean) / std\n",
"X_test = (X_test - mean) / std\n",
"\n",
"\n",
"# Define the mean squared error loss function\n",
"def mse_loss(output, y):\n",
" return np.mean((output - y) ** 2)\n",
"\n",
"# Create a multi-layer neural network with 2 hidden layers\n",
"net = MultiLayerNet(input_size=10, hidden_sizes=[20, 10], output_size=1,\n",
" activation_function=Sigmoid(), loss_function=mse_loss, reg_lambda=0.01)\n",
"\n",
"# Train the network for 1000 epochs\n",
"net.train(X_train, y_train, num_epochs=1000, learning_rate=0.01)\n",
"\n",
"# Evaluate the trained network on the test set\n",
"output = net.forward(X_test)\n",
"predicted_classes = np.round(output)\n",
"accuracy = np.mean(predicted_classes == y_test)\n",
"print(f\"Test accuracy: {accuracy}\")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: src/MLC/notebooks/k_means.ipynb
================================================
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "functional-corrections",
"metadata": {},
"source": [
"## K-means "
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "109c1cfe",
"metadata": {},
"source": [
"K-means clustering is a popular unsupervised machine learning algorithm used for grouping similar data points into k - clusters. Goal: to partition a given dataset into k (predefined) clusters.\n",
"\n",
"The k-means algorithm works by first randomly initializing k cluster centers, one for each cluster. Each data point in the dataset is then assigned to the nearest cluster center based on their distance. The distance metric used is typically Euclidean distance, but other distance measures such as Manhattan distance or cosine similarity can also be used.\n",
"\n",
"After all the data points have been assigned to a cluster, the algorithm calculates the new mean for each cluster by taking the average of all the data points assigned to that cluster. These new means become the new cluster centers. The algorithm then repeats the assignment and mean calculation steps until the cluster assignments no longer change or until a maximum number of iterations is reached.\n",
"\n",
"The final output of the k-means algorithm is a set of k clusters, where each cluster contains the data points that are most similar to each other based on the distance metric used. The algorithm is commonly used in various fields such as image segmentation, market segmentation, and customer profiling.\n",
"\n",
"\n",
"```\n",
"Initialize:\n",
"- K: number of clusters\n",
"- Data: the input dataset\n",
"- Randomly select K initial centroids\n",
"\n",
"Repeat:\n",
"- Assign each data point to the nearest centroid (based on Euclidean distance)\n",
"- Calculate the mean of each cluster to update its centroid\n",
"- Check if the centroids have converged (i.e., they no longer change)\n",
"\n",
"Until:\n",
"- The centroids have converged\n",
"- The maximum number of iterations has been reached\n",
"\n",
"Output:\n",
"- The final K clusters and their corresponding centroids\n",
"```\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "36cafa73",
"metadata": {},
"source": [
"## Code \n",
"Here's an implementation of k-means clustering algorithm in Python from scratch:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ab3cb277",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class KMeans:\n",
" def __init__(self, k, max_iterations=100):\n",
" self.k = k\n",
" self.max_iterations = max_iterations\n",
" \n",
" def fit(self, X):\n",
" # Initialize centroids randomly\n",
" self.centroids = X[np.random.choice(range(len(X)), self.k, replace=False)]\n",
" \n",
" for i in range(self.max_iterations):\n",
" # Assign each data point to the nearest centroid\n",
" cluster_assignments = []\n",
" for j in range(len(X)):\n",
" distances = np.linalg.norm(X[j] - self.centroids, axis=1)\n",
" cluster_assignments.append(np.argmin(distances))\n",
" \n",
" # Update centroids\n",
" for k in range(self.k):\n",
" cluster_data_points = X[np.where(np.array(cluster_assignments) == k)]\n",
" if len(cluster_data_points) > 0:\n",
" self.centroids[k] = np.mean(cluster_data_points, axis=0)\n",
" \n",
" # Check for convergence\n",
" if i > 0 and np.array_equal(self.centroids, previous_centroids):\n",
" break\n",
" \n",
" # Update previous centroids\n",
" previous_centroids = np.copy(self.centroids)\n",
" \n",
" # Store the final cluster assignments\n",
" self.cluster_assignments = cluster_assignments\n",
" \n",
" def predict(self, X):\n",
" # Assign each data point to the nearest centroid\n",
" cluster_assignments = []\n",
" for j in range(len(X)):\n",
" distances = np.linalg.norm(X[j] - self.centroids, axis=1)\n",
" cluster_assignments.append(np.argmin(distances))\n",
" \n",
" return cluster_assignments"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "538027c3",
"metadata": {},
"source": [
"The KMeans class has an __init__ method that takes the number of clusters (k) and the maximum number of iterations to run (max_iterations). The fit method takes the input dataset (X) and runs the k-means clustering algorithm. The predict method takes a new dataset (X) and returns the cluster assignments for each data point based on the centroids learned during training.\n",
"\n",
"Note that this implementation assumes that the input dataset X is a NumPy array with each row representing a single data point and each column representing a feature. The algorithm also uses Euclidean distance to calculate the distances between data points and centroids.\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "1724d308",
"metadata": {},
"source": [
"### Test "
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "141e9843",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1, 1, 1, 1, 1, 0, 0, 0, 0, 0]\n",
"[[-5.53443211 -5.13920695]\n",
" [ 4.46522152 5.04931144]]\n"
]
}
],
"source": [
"\n",
"x1 = np.random.randn(5,2) + 5\n",
"x2 = np.random.randn(5,2) - 5\n",
"X = np.concatenate([x1,x2], axis=0)\n",
"\n",
"# Initialize the KMeans object with k=3\n",
"kmeans = KMeans(k=2)\n",
"\n",
"# Fit the k-means model to the dataset\n",
"kmeans.fit(X)\n",
"\n",
"# Get the cluster assignments for the input dataset\n",
"cluster_assignments = kmeans.predict(X)\n",
"\n",
"# Print the cluster assignments\n",
"print(cluster_assignments)\n",
"\n",
"# Print the learned centroids\n",
"print(kmeans.centroids)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "04430ff9",
"metadata": {},
"source": [
"### Visualize"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "fa0fb8d4",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPAUlEQVR4nO3df6hkZ33H8c/n7ir11kjEvRKa3Z1JaFOaaiBlEizBWpMoUZekf/QP7UTS+sfQUEMChjTxQv+7IFrUglIZ0pSCAyForEW0mrRW6B9GZ/PDGjeREPZuNhoysQWlVxKW/faPmdvdvbl379x7njmz33PfL1jmzjNnn/M97O5nnj3Pc85xRAgAkNfCvAsAAFRDkANAcgQ5ACRHkANAcgQ5ACS3fx47PXDgQLTb7XnsGgDSOnr06CsRsbSxfS5B3m63NRwO57FrAEjL9upm7ZxaAYDkCHIASI4gB4DkCHIASI4gB4DkCHIAmKHBQGq3pYWF8etgUH4fc1l+CAB7wWAg9XrS2tr4/erq+L0kdbvl9sOIHABmZHn5TIivW1sbt5dEkAPAjJw4sbP23SLIAWBGDh/eWftuEeQAMCMrK9Li4rlti4vj9pIIcgCYkW5X6velVkuyx6/9ftmJTolVKwAwU91u+eDeqMiI3PbFtr9i+xnbx2z/YYl+AQDbKzUi/ztJ/xoRf2r7jZIWt/sNAIAyKge57bdI+iNJfy5JEfGapNeq9gsAmE6JUyuXSxpJ+kfbT9i+3/ZvbtzIds/20PZwNBoV2C0AQCoT5Psl/YGkv4+IqyX9r6R7N24UEf2I6EREZ2npdU8qAgDsUokgPynpZEQ8Nnn/FY2DHQBQg8pBHhEvSXrB9u9Omm6Q9JOq/QIAplNq1codkgaTFSvPS/qLQv0CALZRZB15RDw5Of99VUT8SUT8T4l+ATRPHffn3mu4shNAbeq6P/dew71WANSmrvtz7zUEOYDa1HV/7r2GIAdQm7ruz73XEOQAalPX/bkvRLOc5CXIAdSmrvtzX2jWJ3lXV6WIM5O8pcKcIAdQq25XOn5cOn16/JotxAeDgdrtthYWFtRutzWYIo1nPcnL8kMAmNJgMFCv19PaJJVXV1fVm6yf7J7nG2nWk7yMyAFgSsvLy/8f4uvW1ta0vM3QetaTvAQ5AEzpxBZD6K3a1816kpcgB4ApHd5iCL1V+7pZT/IS5AAwpZWVFS1uGFovLi5qZYqh9SwneQlyAJhSt9tVv99Xq9WSbbVaLfX7/fNOdNbBEVH7TjudTgyHw9r3CwCZ2T4aEZ2N7YzIASA5ghwAkiPIASA5ghwAkiPIAWBG6nqsHfdaAYAZqPOxdozIAWAG6nysHUEOADNQ52PtigW57X22n7D9jVJ9AkBWdT7WruSI/E5Jxwr2BwBp1flYuyJBbvugpA9Jur9EfwCQ3dl3PJSkffvOnCMvvXql1KqVz0u6R9JFW21guyepJ21/y0cAaIL11SmzXr1SeURu+4iklyPi6Pm2i4h+RHQiorO0tFR1twCQQh2rV0qcWrlO0s22j0t6UNL1tr9coF8ASK+O1SuVgzwi7ouIgxHRlvRhSf8eEbdWrgwAGqCO1SusIweAGapj9UrRII+I/4iIIyX7BIDMZv28Tol7rQDAzHW75e+vcjZOrQBAcgQ5ACRHkANAcgQ5ACRHkANAcgQ5ACRHkANAcgQ5ACRHkANAcgQ5ACRHkANAcgQ5ACRHkANAcgQ5ACRHkANAcgQ5ACRHkANAcgQ5ACRHkANAcgQ5ACRXOchtH7L9XdvHbD9t+84ShQEAprO/QB+nJH0iIh63fZGko7YfiYifFOgbALCNyiPyiPh5RDw++flXko5JurRqvwCA6RQ9R267LelqSY9t8lnP9tD2cDQaldwtAOxpxYLc9pslfVXSXRHxy42fR0Q/IjoR0VlaWiq1WwDY84oEue03aBzig4h4uESfAIDplFi1Ykn/IOlYRHy2ekkAgJ0oMSK/TtJHJV1v+8nJrw8W6BcAMIXKyw8j4j8luUAtAIBd4MpOAEiOIAeA5AhyAEiOIAeA5AhyAEiOIAeA5AhyAEiOIAeA5AhyAEiOIAeA5AhyAEiOIAeA5AhyAEiOIAeA5AhyAEiOIAeA5AhyAEiOIAeA5AhyAEiOIAeA5AhyAEiOIAeA5IoEue2bbD9r+znb95boEwAwncpBbnufpC9K+oCkKyV9xPaVVfsFAEynxIj8WknPRcTzEfGapAcl3VKgXwDAFEoE+aWSXjjr/clJ2zls92wPbQ9Ho1GB3QIApDJB7k3a4nUNEf2I6EREZ2lpqcBuAQBSmSA/KenQWe8PSvpZgX4BAFMoEeQ/lPQ7ti+z/UZJH5b0LwX6BQBMYX/VDiLilO2PS/q2pH2SHoiIpytXBgCYSuUgl6SI+Kakb5boCwCwM1zZCQDJEeQAkBxBDgDJEeQAkBxBDgDJEeQAkBxBDgDJEeQAkBxBDgDJEeQAkBxBDgDJEeQAkBxBDgDJEeQAkBxBDgDJEeQAkBxBDgDJEeQAkBxBDgDJEeQAkBxBDgDJVQpy25+x/YztH9n+mu2LC9UFAJhS1RH5I5LeERFXSfqppPuqlwQA2IlKQR4R34mIU5O335d0sHpJAICdKHmO/GOSvrXVh7Z7toe2h6PRqOBuAWBv27/dBrYflXTJJh8tR8TXJ9ssSzolabBVPxHRl9SXpE6nE7uqFgDwOtsGeUTceL7Pbd8m6YikGyKCgAaAmm0b5Odj+yZJfy3pPRGxVqYkAMBOVD1H/gVJF0l6xPaTtr9UoCYAwA5UGpFHxG+XKgQAsDtc2QkAyRHkAJAcQQ4AyRHkAJAcQQ4AyRHkAJAcQQ4AyRHkAJAcQQ4AyRHkAJAcQQ4AyRHkAJAcQQ4AyRHkAJAcQQ4AyRHkAJAcQQ4AyRHkAJAcQQ4AyRHkAJAcQQ4AyRHkAJBckSC3fbftsH2gRH8AgOlVDnLbhyS9T9KJ6uUAAHaqxIj8c5LukRQF+gIA7FClILd9s6QXI+KpKbbt2R7aHo5Goyq7BQCcZf92G9h+VNIlm3y0LOmTkt4/zY4ioi+pL0mdTofROwAUsm2QR8SNm7XbfqekyyQ9ZVuSDkp63Pa1EfFS0SoBAFvaNsi3EhH/Jent6+9tH5fUiYhXCtQFAJgS68gBILliQR4R7bmPxgcDqd2WFhbGr4PBXMsBgDo0Z0Q+GEi9nrS6KkWMX3u9+YQ5XygAatScIF9eltbWzm1bWxu31+lC+kIBsCc0J8hPbHFh6Vbts3KhfKEA2DOaE+SHD++sfVYulC8UAHtGc4J8ZUVaXDy3bXFx3F6nC+ULBcCe0Zwg73alfl9629vOtL3pTfXXcaF8oQDYM5oT5Ot+/eszP//iF/VPNK5/obRakj1+7ffH7QAwA46o/7YnnU4nhsNh+Y7b7fEqkY1aLen48fL7A4Aa2T4aEZ2N7c0akW8yoTiQ1F5d1cLCgtrttgYsAwTQMM0K8g0TigNJPUmrkiJCq6ur6vV6hDmARskT5NNcLblhonFZ0oYV3VpbW9Mya7oBNMiu735Yq/WrJdcvtFm/WlI6dxJx/eflZenECZ3Y4vz/CdZ0A2iQHCPynVwt2e2OJzZPn9bhVmvT7g6zphtAg+QI8l1eLbmysqLFDWu6FxcXtcKabgANkiPId3m1ZLfbVb/fV6vVkm21Wi31+311WdMNoEFyrCPfeI5cGk9qcqENgD0k9zpyrpYEgC3lWLUijUOb4AaA18kxIgcAbIkgB4DkCHIASK5ykNu+w/aztp+2/ekSRQEApldpstP2eyXdIumqiHjV9tvLlAUAmFbVEfntkj4VEa9KUkS8XL0kAMBOVA3yKyS92/Zjtr9n+5qtNrTdsz20PRyNRhV3CwBYt+2pFduPSrpkk4+WJ7//rZLeJekaSQ/Zvjw2uVw0IvqS+tL4ys4qRQMAztg2yCPixq0+s327pIcnwf0D26clHZDEkBsAalL11Mo/S7pekmxfIemNkl6p2CcAYAeqBvkDki63/WNJD0q6bbPTKrWY5glCANBAlZYfRsRrkm4tVMvuTfsEIQBooGZc2bmTJwgBQMM0I8h3+QQhAGiCZgT5Lp8gBABN0IwgX1kZPzHobIuL43YAaLhmBDlPEAKwh+V5QtB2eIIQgD2qGSNyANjDCHIASI4gB4DkCHIASI4gB4DkPI97XNkeSVqt0MUBNfsui00+Po4tryYfX5Zja0XE0sbGuQR5VbaHEdGZdx2z0uTj49jyavLxZT82Tq0AQHIEOQAklzXI+/MuYMaafHwcW15NPr7Ux5byHDkA4IysI3IAwARBDgDJpQ5y23fYftb207Y/Pe96SrN9t+2wfWDetZRk+zO2n7H9I9tfs33xvGuqyvZNk7+Lz9m+d971lGT7kO3v2j42+bd257xrKs32PttP2P7GvGvZjbRBbvu9km6RdFVE/L6kv51zSUXZPiTpfZKa+Ly6RyS9IyKukvRTSffNuZ5KbO+T9EVJH5B0paSP2L5yvlUVdUrSJyLi9yS9S9JfNez4JOlOScfmXcRupQ1ySbdL+lREvCpJEfHynOsp7XOS7pHUuNnoiPhORJyavP2+pIPzrKeAayU9FxHPR8Rrkh7UeJDRCBHx84h4fPLzrzQOvEvnW1U5tg9K+pCk++ddy25lDvIrJL3b9mO2v2f7mnkXVIrtmyW9GBFPzbuWGnxM0rfmXURFl0p64az3J9WgoDub7bakqyU9NudSSvq8xoOm03OuY9cu6CcE2X5U0iWbfLSsce1v1fi/etdIesj25ZFkPeU2x/ZJSe+vt6Kyznd8EfH1yTbLGv+3fVBnbTPgTdpS/D3cCdtvlvRVSXdFxC/nXU8Jto9Iejkijtr+4zmXs2sXdJBHxI1bfWb7dkkPT4L7B7ZPa3zjm1Fd9VWx1bHZfqekyyQ9ZVsan3Z43Pa1EfFSjSVWcr4/O0myfZukI5JuyPLlex4nJR066/1BST+bUy0zYfsNGof4ICIennc9BV0n6WbbH5T0G5LeYvvLEXHrnOvakbQXBNn+S0m/FRF/Y/sKSf8m6XADQuEcto9L6kREhjuzTcX2TZI+K+k9EZHii/d8bO/XeNL2BkkvSvqhpD+LiKfnWlghHo8o/knSf0fEXXMuZ2YmI/K7I+LInEvZscznyB+QdLntH2s8uXRb00K8wb4g6SJJj9h+0vaX5l1QFZOJ249L+rbGE4EPNSXEJ66T9FFJ10/+vJ6cjGBxgUg7IgcAjGUekQMARJADQHoEOQAkR5ADQHIEOQAkR5ADQHIEOQAk93+igTL51gL1hQAAAABJRU5ErkJggg==",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"from matplotlib import pyplot as plt\n",
"# Plot the data points with different colors based on their cluster assignments\n",
"colors = ['r', 'b']\n",
"for i in range(kmeans.k):\n",
" plt.scatter(X[np.where(np.array(cluster_assignments) == i)][:,0], \n",
" X[np.where(np.array(cluster_assignments) == i)][:,1], \n",
" color=colors[i])\n",
"\n",
"# Plot the centroids as black circles\n",
"plt.scatter(kmeans.centroids[:,0], kmeans.centroids[:,1], color='black', marker='o')\n",
"\n",
"# Show the plot\n",
"plt.show()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "69fc2d74",
"metadata": {},
"source": [
"### Optimization \n",
"Here are some ways to optimize the k-means clustering algorithm:\n",
"\n",
"Random initialization of centroids: Instead of initializing the centroids using the first k data points, we can randomly initialize them to improve the convergence of the algorithm. This can be done by selecting k random data points from the input dataset as the initial centroids.\n",
"\n",
"Early stopping: We can stop the k-means algorithm if the cluster assignments and centroids do not change after a certain number of iterations. This helps to avoid unnecessary computation.\n",
"\n",
"Vectorization: We can use numpy arrays and vectorized operations to speed up the computation. This avoids the need for loops and makes the code more efficient.\n",
"\n",
"Here's an optimized version of the k-means clustering algorithm that implements these optimizations:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "121e7b70",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class KMeans:\n",
" def __init__(self, k=3, max_iters=100, tol=1e-4):\n",
" self.k = k\n",
" self.max_iters = max_iters\n",
" self.tol = tol\n",
" \n",
" def fit(self, X):\n",
" # Initialize centroids randomly\n",
" self.centroids = X[np.random.choice(X.shape[0], self.k, replace=False)]\n",
" \n",
" # Iterate until convergence or maximum number of iterations is reached\n",
" for i in range(self.max_iters):\n",
" # Assign each data point to the closest centroid\n",
" distances = np.linalg.norm(X[:, np.newaxis] - self.centroids, axis=2)\n",
" cluster_assignments = np.argmin(distances, axis=1)\n",
" \n",
" # Update the centroids based on the new cluster assignments\n",
" new_centroids = np.array([np.mean(X[np.where(cluster_assignments == j)], axis=0) \n",
" for j in range(self.k)])\n",
" \n",
" # Check for convergence\n",
" if np.linalg.norm(new_centroids - self.centroids) < self.tol:\n",
" break\n",
" \n",
" self.centroids = new_centroids\n",
" \n",
" def predict(self, X):\n",
" # Assign each data point to the closest centroid\n",
" distances = np.linalg.norm(X[:, np.newaxis] - self.centroids, axis=2)\n",
" cluster_assignments = np.argmin(distances, axis=1)\n",
" \n",
" return cluster_assignments\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "0a8514c5",
"metadata": {},
"source": [
"This optimized version initializes the centroids randomly, uses vectorized operations for computing distances and updating the centroids, and checks for convergence after each iteration to stop the algorithm if it has converged."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "a98d4ac5",
"metadata": {},
"source": [
"Follow ups:\n",
"\n",
"* Computattional complexity: O(it * knd)\n",
"* Improve space: use index instead of copy\n",
"* Improve time: \n",
" * dim reduction\n",
" * subsample (cons?)\n",
"* mini-batch\n",
"* k-median https://mmuratarat.github.io/2019-07-23/kmeans_from_scratch"
]
},
{
"cell_type": "markdown",
"id": "a756163a",
"metadata": {},
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: src/MLC/notebooks/k_means_2.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"id": "functional-corrections",
"metadata": {},
"source": [
"## K-means with multi-dimensional data\n",
" \n",
"$X_{n \\times d}$"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "formal-antique",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import time"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "durable-horse",
"metadata": {},
"outputs": [],
"source": [
"n, d, k=1000, 20, 4\n",
"max_itr=100"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "egyptian-omaha",
"metadata": {},
"outputs": [],
"source": [
"X=np.random.random((n,d))"
]
},
{
"cell_type": "markdown",
"id": "employed-helen",
"metadata": {},
"source": [
"$$ argmin_j ||x_i - c_j||_2 $$"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "center-timer",
"metadata": {},
"outputs": [],
"source": [
"def k_means(X, k):\n",
" #Randomly Initialize Centroids\n",
" np.random.seed(0)\n",
" C= X[np.random.randint(n,size=k),:]\n",
" E=np.float('inf')\n",
" for itr in range(max_itr):\n",
" \n",
" # Find the distance of each point from the centroids \n",
" E_prev=E\n",
" E=0\n",
" center_idx=np.zeros(n)\n",
" for i in range(n):\n",
" min_d=np.float('inf')\n",
" c=0\n",
" for j in range(k):\n",
" d=np.linalg.norm(X[i,:]-C[j,:],2)\n",
" if d<min_d:\n",
" min_d=d\n",
" c=j\n",
" \n",
" E+=min_d\n",
" center_idx[i]=c\n",
" \n",
" #Find the new centers\n",
" for j in range(k):\n",
" C[j,:]=np.mean( X[center_idx==j,:] ,0)\n",
" \n",
" if itr%10==0:\n",
" print(E)\n",
" if E_prev==E:\n",
" break\n",
" \n",
" return C, E, center_idx"
]
},
{
"cell_type": "markdown",
"id": "material-hayes",
"metadata": {},
"source": [
"$$ argmin_j ||x_i - c_j||_2 $$\n",
"\n",
"$$||x_i - c_j||_2 = \\sqrt{(x_i - c_j)^T (x_i-c_j)} = \\sqrt{x_i^T x_i -2 x_i^T c_j + c_j^T c_j} $$\n",
"\n",
"- $ diag(X~X^T)$, can be used to get $x_i^T x_i$\n",
"\n",
"- $X~C^T $, can be used to get $x_i^T c_j$\n",
"\n",
"- $diag(C~C^T)$, can be used to get $c_j^T c_j$"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "colored-linux",
"metadata": {},
"outputs": [],
"source": [
"def k_means_vectorized(X, k):\n",
" \n",
" #Randomly Initialize Centroids\n",
" np.random.seed(0)\n",
" C= X[np.random.randint(n,size=k),:]\n",
" E=np.float('inf')\n",
" for itr in range(max_itr):\n",
" # Find the distance of each point from the centroids \n",
" XX= np.tile(np.diag(np.matmul(X, X.T)), (k,1) ).T\n",
" XC=np.matmul(X, C.T)\n",
" CC= np.tile(np.diag(np.matmul(C, C.T)), (n,1)) \n",
"\n",
" D= np.sqrt(XX-2*XC+CC)\n",
"\n",
" # Assign the elements to the centroids:\n",
" center_idx=np.argmin(D, axis=1)\n",
"\n",
" #Find the new centers\n",
" for j in range(k):\n",
" C[j,:]=np.mean( X[center_idx==j,:] ,0)\n",
"\n",
" #Find the error\n",
" E_prev=E\n",
" E=np.sum(D[np.arange(n),center_idx])\n",
" if itr%10==0:\n",
" print(E)\n",
" if E_prev==E:\n",
" break\n",
" \n",
" return C, E, center_idx"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "equivalent-platinum",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1517.502248752696\n",
"1218.91004301866\n",
"1217.362137659097\n",
"0.8816308975219727 seconds\n"
]
}
],
"source": [
"start=time.time()\n",
"C, E, center_idx = k_means(X, k)\n",
"print(time.time()-start,'seconds')"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "environmental-steam",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1517.502248752696\n",
"1218.9100430186547\n",
"1217.3621376590977\n",
"0.09020209312438965 seconds\n"
]
}
],
"source": [
"start=time.time()\n",
"C, E, center_idx = k_means_vectorized(X, k)\n",
"print(time.time()-start,'seconds')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "north-picking",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: src/MLC/notebooks/k_nearest_neighbors.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"id": "continuing-northern",
"metadata": {},
"source": [
"## K-nearest neighbour\n",
" \n",
"$X_{n \\times d}$\n",
"\n",
"$Y_{n \\times 1}$\n",
"\n",
"$Z_{m \\times d}$"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "recognized-seating",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import time\n",
"from collections import Counter"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "pretty-capability",
"metadata": {},
"outputs": [],
"source": [
"n, d, m=500, 20, 4\n",
"k=5"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "differential-platinum",
"metadata": {},
"outputs": [],
"source": [
"X=np.random.random((n,d))\n",
"Z=np.random.random((m,d))\n",
"Y=np.random.randint(3,size=n)"
]
},
{
"cell_type": "markdown",
"id": "alpha-lincoln",
"metadata": {},
"source": [
"$$ argmin_i ||x_i - z_j||_2 $$"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "otherwise-waste",
"metadata": {},
"outputs": [],
"source": [
"def KNN(X, Y, Z, k):\n",
" res=[]\n",
" for j in range(m):\n",
" d=np.zeros(n)\n",
" for i in range(n):\n",
" # Find the distance from each point \n",
" d[i]=np.linalg.norm(X[i,:]-Z[j,:], 2)\n",
"\n",
" c=np.argsort(d)\n",
" label=Counter(Y[c[0:k]]).most_common()[0][0]\n",
" res.append(label)\n",
" return res"
]
},
{
"cell_type": "markdown",
"id": "expected-lewis",
"metadata": {},
"source": [
"$$ argmin_j ||x_i - z_j||_2 $$\n",
"\n",
"$$||x_i - z_j||_2 = \\sqrt{(x_i - z_j)^T (x_i-z_j)} = \\sqrt{x_i^T x_i -2 x_i^T z_j + z_j^T z_j} $$\n",
"\n",
"- $ diag(X~X^T)$, can be used to get $x_i^T x_i$\n",
"\n",
"- $X~Z^T $, can be used to get $x_i^T z_j$\n",
"\n",
"- $diag(Z~Z^T)$, can be used to get $z_j^T z_j$"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "parental-method",
"metadata": {},
"outputs": [],
"source": [
"def KNN_vectorized(X, Y, Z, k):\n",
" \n",
" # Find the distance from each point \n",
" XX= np.tile(np.diag(np.matmul(X, X.T)), (m,1) ).T\n",
" XZ=np.matmul(X, Z.T)\n",
" ZZ= np.tile(np.diag(np.matmul(Z, Z.T)), (n,1)) \n",
" D= np.sqrt(XX-2*XZ+ZZ)\n",
" res=[]\n",
" for j in range(m):\n",
" c=np.argsort(D[:,j])\n",
" label=Counter(Y[c[0:k]]).most_common()[0][0]\n",
" res.append(label)\n",
" \n",
" return res"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "false-reader",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.022996902465820312 seconds\n"
]
}
],
"source": [
"start=time.time()\n",
"res = KNN(X, Y, Z, k)\n",
"print(time.time()-start,'seconds')"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "contemporary-chess",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.0029761791229248047 seconds\n"
]
}
],
"source": [
"start=time.time()\n",
"res = KNN_vectorized(X, Y, Z, k)\n",
"print(time.time()-start,'seconds')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fancy-enlargement",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: src/MLC/notebooks/knn.ipynb
================================================
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# K-Nearest Neighbors (KNN)\n",
"\n",
"K-Nearest Neighbors (KNN) is a classification algorithm that assigns a class label to a data point based on the class labels of its k nearest neighbors in the training set. It is a non-parametric algorithm, which means that it does not make any assumptions about the distribution of the data.\n",
"\n",
"The KNN algorithm works as follows:\n",
"\n",
"- Choose a value of k (the number of nearest neighbors to consider).\n",
"- For each data point in the test set, compute its distance to all data points in the training set.\n",
"- Select the k nearest neighbors based on their distances.\n",
"- Assign the class label that appears most frequently among the k nearest neighbors to the test point.\n",
"\n",
"The distance between two data points can be computed using a distance metric such as Euclidean distance, Manhattan distance, or Minkowski distance. The choice of distance metric depends on the nature of the data and the problem at hand.\n",
"\n",
"One important aspect of the KNN algorithm is the choice of the value of k. A small value of k will result in a more flexible decision boundary that can capture complex patterns in the data, but may also lead to overfitting. A large value of k will result in a smoother decision boundary that may not capture fine details in the data, but is less prone to overfitting. The value of k is typically chosen using cross-validation.\n",
"\n",
"KNN can also be used for regression tasks, where the goal is to predict a continuous value instead of a class label. In this case, the predicted value for a test point is the average of the values of its k nearest neighbors in the training set."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Code \n",
"Here is an implementation of the KNN classifier in Python:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from collections import Counter\n",
"import numpy as np\n",
"\n",
"class KNN:\n",
" def __init__(self, k=3, distance='euclidean'):\n",
" self.k = k\n",
" self.distance = distance\n",
" \n",
" def fit(self, X_train, y_train):\n",
" self.X_train = X_train\n",
" self.y_train = y_train\n",
" \n",
" def predict(self, X_test):\n",
" y_pred = []\n",
" for x in X_test:\n",
" # Compute distances between the test point and all training points\n",
" if self.distance == 'euclidean':\n",
" distances = np.linalg.norm(self.X_train - x, axis=1)\n",
" elif self.distance == 'manhattan':\n",
" distances = np.sum(np.abs(self.X_train - x), axis=1)\n",
" else:\n",
" distances = np.power(np.sum(np.power(np.abs(self.X_train - x), self.distance), axis=1), 1/self.distance)\n",
" \n",
" # Select the k nearest neighbors\n",
" nearest_indices = np.argsort(distances)[:self.k]\n",
" nearest_labels = self.y_train[nearest_indices]\n",
" \n",
" # Assign the class label that appears most frequently among the k nearest neighbors\n",
" label = Counter(nearest_labels).most_common(1)[0][0]\n",
" y_pred.append(label)\n",
" \n",
" return np.array(y_pred)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The KNN class has two main methods: fit and predict. The fit method takes the training data as input and stores it in instance variables. The predict method takes the test data as input and computes the class labels for each test point using the KNN algorithm.\n",
"\n",
"The predict method computes the distances between the test point and all training points, selects the k nearest neighbors based on the distances, and assigns the class label that appears most frequently among the k nearest neighbors to the test point.\n",
"\n",
"The distance parameter allows the user to choose the distance metric to use for computing distances. The default value is euclidean, but the user can also choose manhattan or any other value p for the Minkowski distance."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy: 1.0\n"
]
}
],
"source": [
"from sklearn.datasets import load_iris\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.metrics import accuracy_score\n",
"\n",
"# Load the iris dataset\n",
"iris = load_iris()\n",
"\n",
"# Split the data into training and test sets\n",
"X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.2, random_state=42)\n",
"\n",
"# Create a KNN classifier with k=5 and euclidean distance\n",
"knn = KNN(k=5, distance='euclidean')\n",
"\n",
"# Train the classifier on the training data\n",
"knn.fit(X_train, y_train)\n",
"\n",
"# Make predictions on the test data\n",
"y_pred = knn.predict(X_test)\n",
"\n",
"# Compute the accuracy of the classifier\n",
"accuracy = accuracy_score(y_test, y_pred)\n",
"print(f\"Accuracy: {accuracy}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualization "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEWCAYAAABmE+CbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1JklEQVR4nO3de3gU9dXA8e8JYAHlIhpvEIItVRdCuCRIvIIXLvF+gWpLVWgtolUxttZaHsuqtX2tbWP6olJeq/AqrVjU2loF8YLoq0GIcl9RqpBwaQlQAghoQs77x0zCstlsNpfZnU3O53n2ye7Mb2fOzG7m7Mxv5oyoKsYYY0xasgMwxhjjD5YQjDHGAJYQjDHGuCwhGGOMASwhGGOMcVlCMMYYA1hCMK2YiPQRERWR9h5N/2ci8kTY6ytFpExE9orIYBFZIyIjvJh3Ini9/oz/WEIwLUJENojIhWGvrxWR/4jI8LANyz8i3vOMiATd5yPcNo9GtHlXRCbEmO8pIvIXEdkuIhUislJE7hSRdi27hHWp6i9V9cawQb8BblXVo1T1I1Xtr6qLWmJe7vqpdpPNHhFZJyITW2LajYhhkYjc2HBLk6osIZgWJyI3AI8CF6vq22Gj8kTkrBhv/QK4XkT6xDmfbwBLgDJggKp2A8YBuUCXpsTeTJnAmuZOJMYv8i2qehTQFSgA/kdETm3u/IypYQnBtCgRmQT8Fhitqu9FjP418IsYb98FzAKmxTm7+4D3VPVOVd0KoKrrVPU7qrorSmwTRSTk/sL+TERuCht3rIi8LCK7RGSniLwjImnuuLtFZHPYL/ML3OFBdy/nayKyF2gHrBCRf7rja/eaRCRNRH4qIv8UkR0i8pyI9HDH1exBfV9ESoE3Yy20Ol4BdgLZcUy/oxvnDnf5lorI8ZExhi9TlHX3IHAOMN3dS5kujkIR2Ra2d5YV1ydnfMkSgmlJNwMPABeo6rIo4x8FTgnfAEXxIHB1nL98LwTmNSK+bcAlOL+wJwKFIjLEHfcjYBOQDhwP/AxQN45bgaGq2gUYDWwIn6iqfun+cgcYqKrfiDLv24ErgOHAScB/cNZHuOFAwJ1HvdyN/2XAscD6OKZ/A9ANyACOASYD+2PNI5KqTgXe4dAhsVuBUcC5wClAd+AaYEdjpmv8xRKCaUkjgWJgVT3jD+Bs8OvdS1DVfwEzgPvjmN8xwNZ4g1PVf6jqP91f2G8Dr+H86gWoBE4EMlW1UlXfUafQ10Hga0A/EemgqhtU9Z/xzjPMTcBUVd2kql8CQWBsxOGhoKp+oar1baxPEpFdOBvzF4E7VfWjOKZfibOu+qrqQVUtUdXdTViGSJU4h+ZOA0RVQzV7aiY1WUIwLWkyzq/FJ0RE6mnzP8DxInJpjOk8BIwWkYENzG8HzkY8LiKSLyLF7iGhXcBFOL+yAR7G+bX9mns46acAqroeuANnA7tNRJ4VkZPinWeYTOBF95DNLiCEk2yOD2tT1sA0tqhqd5w9nN8D58c5/aeBBcCzIrJFRH4tIh2asAyHUdU3gek4eyL/FpGZItK1udM1yWMJwbSkbcAFOL+6H4vWQFUrcY79PwBETRqqugN4xG0Ty+vA1fEEJiJfA57HORPoeHfD+kpNDKq6R1V/pKpfBy4F7qzpK1DVP6nq2TgbXcVJWI1VBuSravewR0dV3RzWJq7Sw+4ewN3AABG5oqHpu3s896lqP+BMnMNm17vv+wLoHDb5E2LNOkosv1fVHKA/zo+Bu+JZBuNPlhBMi1LVLTi/XMeISGE9zZ7GOQwzJsakfoez8QrEaDMNOFNEHhaREwBEpK/bgdo9ou0R7jzLgSoRycc5Bo77vkvc9wqwG+fX9UEROVVEzncTygGcwzUHY8RUnxnAgyKS6c4vXUQub8J0AFDVr3A673/e0PRF5DwRGSDOqbi7cQ711CzDcuBaEekgIrnA2Biz/Tfw9ZoXIjJURIa5extf4Kyfpqwb4xOWEEyLU9UynKQwVkR+FWX8QZyNeY8Y09iNc1ZSrDb/BM4A+gBrRKQCZy9gGbAnou0enI7X53A6XL8D/C2syTdx9jj2Au8Dj7nXEHwN+C9gO/Av4DicDufGKnLn95qI7MHpaxnWhOmEexLo7R5+izX9E3A633fjHEp6G6g5k+he4Bs46+Q+4E8NLMNYca4v+T3Ooav/cd+7EecQ3m+auUwmicRukGOMMQZsD8EYY4zLEoIxxhjAEoIxxhiXJQRjjDEApFxZ22OPPVb79OmT7DCMMSallJSUbFfV9FhtUi4h9OnTh2XLopXJMcYYUx8R2dhQGztkZIwxBrCEYIwxxmUJwRhjDJCCfQgmdVVWVrJp0yYOHDiQ7FBME3Ts2JFevXrRoUOzC6Uan7KEkESqSniV6MjXrUXNcm3atIkuXbqQmZlJWprtnKYSVWXHjh1s2rSJk08+OdnhtDmJ2lZ49l/p3rbvAxFZISJrROS+KG26icjfw9ok9KbhyRQMBsnPLyAzU0lLg8xMJT+/gGAwmOzQWlT4coZCB9i8uQfr129iy5YtyQ7NNIKIcMwxx9jeXRIEg0EK8vPRzExIS0MzMynIz/dkW+HlHsKXwPmqutctj/uuiLyqqsVhbX4IrFXVS0UkHVgnInPc0r6tlqpSXLyLBQuK3CGFlJYWUFpaBExpNXsKkcuZlnYTlZWbqKzcBhzXapazrbDPKvFUlV3FxRQtWABAIVBQWkpRaSlTaPk9Bc8Sgnv7wb3uyw7uI7K0qgJd3Br0R+HcNLzKq5j8QkQIhWpuFVDkPgCmEAoVtpp/vLrLOQbnBmXHceBARqtZTmO8IiIUhkJA5JYCCkOhFv8f8vRAroi0E5HlOHfSWqiqSyKaTMe5AcoWnPvwTlHV6ijTmSQiy0RkWXl5uZchJ0xZmeDk+3CF7vDWI/pyZvDVV/5fzlmzZtmhLZN0UlYWZUvhDG9pniYE94beg4BewOkikhXRZDTOHZtOAgYB06Pdk1VVZ6pqrqrmpqfHvPI6ZWRkKFAQMbTAHd56RF/OMo44wv/LaQnB+IFmZETZUjjDW1pCTvVQ1V3AIureMnEi8II61gOfA6clIqZkUlUCgQKcHcApQLX7t4hAoIDWctOiusuZiXPDsW107FjW4HLOmQN9+kBamvN3zpzmx/TFF19w8cUXM3DgQLKyspg7dy4lJSUMHz6cnJwcRo8ezdatW5k3bx7Lli1j/PjxDBo0iP379/PGG28wePBgBgwYwPe+9z2+/PJLAH7605/Sr18/srOz+fGPfwzA3//+d4YNG8bgwYO58MIL+fe//9384E2bo6oUBAJRthRQEAi0+LbCsz4Et5O4UlV3iUgn4ELq3py8FOem7O+IyPHAqcBnXsXkFyJCXl53avoMysqEjIxCAgHIy+veao6tRy4nfMwRR2TQsSMcdVT7mMs5Zw5MmgT79jmvN250XgOMH9/0mObPn89JJ53EP/7xDwAqKirIz8/npZdeIj09nblz5zJ16lSefPJJpk+fzm9+8xtyc3M5cOAAEyZM4I033uCUU07h+uuv5/HHH+f666/nxRdf5OOPP0ZE2LVrFwBnn302xcXFiAhPPPEEv/71r/ntb3/b9MBNmyQidM/LO9RnUFZGYUYGBAJ0z8tr+W2FqnryALKBj4CVwGrg5+7wycBk9/lJwGs4/Qerge82NN2cnBxtLaqrq2O+bi1qlmvt2rWHvY4lM1MV6j4yM5sXy7p167RPnz76k5/8RBcvXqyrVq3SLl266MCBA3XgwIGalZWlI0eOVFXV4cOH69KlS1VVdfny5XrOOefUTuf111/XK6+8UisrKzU7O1u/973v6fPPP69ffvmlqqquXLlSR44cqVlZWXrKKafo6NGjmxe4T9R8hiaxWmJbASzTBravXp5ltBIYHGX4jLDnW4BRXsXgd5HZvbXsGURqynKWljZueLxOOeUUSkpKeOWVV7jnnnsYOXIk/fv35/3334/5Pq1n17x9+/Z88MEHvPHGGzz77LNMnz6dN998k9tuu40777yTyy67jEWLFrW660tMYiVqW2GXixpf6t27ccPjtWXLFjp37sx3v/tdfvzjH7NkyRLKy8trE0JlZSVr1qwBoEuXLuzZsweA0047jQ0bNrB+/XoAnn76aYYPH87evXupqKjgoosu4pFHHmH58uWAcyiqZ8+eAMyePbt5QRuTIFa6wvjSgw8e3ocA0LmzM7w5Vq1axV133UVaWhodOnTg8ccfp3379tx+++1UVFRQVVXFHXfcQf/+/ZkwYQKTJ0+mU6dOvP/++zz11FOMGzeOqqoqhg4dyuTJk9m5cyeXX345Bw4cQFUpLHROEAwGg4wbN46ePXuSl5fH559/3rzAjUkAqW9X2K9yc3PVbpCTmkKhEIFAIO72c+bA1KnOYaLevZ1k0JwOZdN8jf0MjX+ISImq5sZqY3sIxrfGj7cEYEwiWR+CMcYYwBKCMcYYlyUEY4wxgCUEY4wxLksIxhhjAEsIxjTLz3/+c15//fVGv2/RokVccsklHkRkTNPZaafGNKCmzku0+0Dff//9CYmhqqqK9u3t39V4y/YQjG/NWTWHPo/0Ie2+NPo80oc5q5pX//ruu+/mscceq30dDAb57W9/y8MPP8zQoUPJzs5m2rRpAGzYsIFAIMAtt9zCkCFDKCsrY8KECWRlZTFgwIDaK5InTJjAvHnzAFi6dClnnnkmAwcO5PTTT2fPnj0cOHCAiRMnMmDAAAYPHsxbb71VJ66dO3dyxRVXkJ2dTV5eHitXrqyNb9KkSYwaNYrrr7++WctuTDwsIRhfmrNqDpP+PomNFRtRlI0VG5n090nNSgrXXnstc+fOrX393HPPkZ6ezqeffsoHH3zA8uXLKSkpYfHixQCsW7eO66+/no8++ojt27ezefNmVq9ezapVq5g4ceJh0/7qq6+45pprKCoqYsWKFbz++ut06tSJRx99FHBKZvz5z3/mhhtuqHOj+mnTpjF48GBWrlzJL3/5y8M2/iUlJbz00kv86U9/avJyGxMvSwjGl6a+MZV9lfsOG7avch9T35ja5GkOHjyYbdu2sWXLFlasWMHRRx/NypUree211xg8eDBDhgzh448/5tNPPwUgMzOTvLw8AL7+9a/z2WefcdtttzF//ny6dj38xn7r1q3jxBNPZOjQoQB07dqV9u3b8+6773LdddcBToG8zMxMPvnkk8PeG97m/PPPZ8eOHVRUVABw2WWX0alTpyYvszGNYQcljS+VVkSvc13f8HiNHTuWefPm8a9//Ytrr72WDRs2cM8993DTTTcd1m7Dhg0ceeSRta+PPvpoVqxYwYIFC3j00Ud57rnnePLJJ2vHq2rUksTx1AqL1qZmWuExGOM120MwvtS7W/Q61/UNj9e1117Ls88+y7x58xg7diyjR4/mySefZO/evQBs3ryZbdu21Xnf9u3bqa6u5uqrr+aBBx7gww8/PGz8aaedxpYtW1i6dCkAe/bsoaqqinPPPZc57r0/P/nkE0pLSzn11FMPe294m0WLFnHsscfW2QMxJhG8vIVmR2Ax8DV3PvNUdVqUdiOAR4AOwHZVHe5VTCZ1PHjBg0z6+6TDDht17tCZBy9oXv3r/v37s2fPHnr27MmJJ57IiSeeSCgU4owzzgDgqKOO4plnnqFdu3aHvW/z5s1MnDiR6upqAH71q18dNv6II45g7ty53Hbbbezfv59OnTrx+uuvc8sttzB58mQGDBhA+/btmTVrFl/72tcOe28wGGTixIlkZ2fTuXNnu3+CSRrPyl+Ls897pKruFZEOwLvAFFUtDmvTHXgPGKOqpSJynKrW/XkWxspfp65Gl79eNYepb0yltKKU3t168+AFDzJ+gJU/TSYrf526klr+2r2H5173ZQf3EZl9vgO8oKql7ntiJgPTtowfMN4SgDEJ5Gkfgoi0E5HlwDZgoaouiWhyCnC0iCwSkRIRiXqytYhMEpFlIrKsvLzcy5CNMabN8jQhqOpBVR0E9AJOF5GsiCbtgRzgYmA0cK+InBJlOjNVNVdVc9PT070M2Rhj2qyEnGWkqruARcCYiFGbgPmq+oWqbsfphB6YiJiMMcYczrOEICLpbqcxItIJuBD4OKLZS8A5ItJeRDoDw4CQVzEZY4ypn5cXpp0IzBaRdjiJ5zlVfVlEJgOo6gxVDYnIfGAlUA08oaqrPYzJGGNMPTzbQ1DVlao6WFWzVTVLVe93h89Q1Rlh7R5W1X5um0e8iseYaLZs2cLYsWMb/b6LLrqIXbt2xWzT1NLYxiSLZ9cheMWuQ0hdqXQOu5Wbji6VPkNzuHiuQ7DSFca/5syBPn0gLc35O8eb8tdZWc7Jb7NmzWLcuHFceumljBo1in379vGtb32L7OxsrrnmGoYNG0bNj5E+ffqwffv22jLZP/jBD+jfvz+jRo1i//79QMOlsTds2MA555zDkCFDGDJkCO+9916zls+Y5rKEYKKK3HNM+J7knDkwaRJs3Aiqzt9Jk5qVFKKVv66pTlrj/fffZ/bs2bz55ps89thjtRVR7733XkpKSqJO99NPP+WHP/wha9asoXv37jz//POHja+vNPZxxx3HwoUL+fDDD5k7dy633357k5fNmJZgCcHUEQwGyc8vIDNTSUuDzEwlP7+AYDCYuCCmToV9h5e/Zt8+Z3gTRSt/3bv34cXyRo4cSY8ePQCnLPW1114LQFZWFtnZ2VGne/LJJzNo0CAAcnJy2LBhw2Hj6yuNXVlZyQ9+8AMGDBjAuHHjWLt2bZOXzZiWYAdJzWFUleLiXSxYUOQOKaS0tIDS0iJgSr1lnltcaT1lrusbHqfI8teRwstNx7tXFF6srl27drWHjMKnE22dFRYWcvzxx7NixQqqq6vp2LFjvIthjCdsD8EcRkQIhQqBKUARzlfESQahUGFikgFA73rKXNc3PE6R5a9jOfvss3nuuecAWLt2LatWrWrSPOsrjV1RUcGJJ55IWloaTz/9NAcPHmzS9I1pKZYQTB1lZQIURgwtdIcnyIMPQufOhw/r3NkZ3gyR5a9jueWWWygvLyc7O5uHHnqI7OxsunXr1uh5hpfGHjhwICNHjuTAgQPccsstzJ49m7y8PD755BO7GY5JPlVNqUdOTo4ab/XuXa0wRXGq07qPKdq7d3Wzprt27drGveGZZ1QzM1VFnL/PPNOs+TdWVVWV7t+/X1VV169fr5mZmfrll18mNAa/afRnaHwDWKYNbF+tD8EcRlUJBA71GTh7CgVAEYEAqCbwsNH48c4jSfbt28d5551HZWUlqsrjjz/OEUcckbR4jPGaJQRzGBEhL687NX0GZWVCRkYhgQDk5XVPXDLwgS5dumAXQZq2xBKCqSMYDEacGSOJ3TMwxiSFdSqbqCI3/pYMjGn9LCEYY4wBLCEYY4xxWUIwbUq7du0YNGgQWVlZjBs3jn2R5TEaIbx43Y033hiz9MSiRYuaVLyupohevMOjmTVrFrfeemuLzNe0bpYQPKLJLg5XD7/GFY0XsXbq1Inly5ezevVqjjjiCGbMmHHY+KZeLfzEE0/Qr1+/esc3NSEYk0he3kKzo4h8ICIrRGSNiNwXo+1QETkoIo2/U4kP+aI4XArFFU0wGKSgoKA2CagqBQUtG+s555zD+vXrWbRoEeeddx7f+c53GDBgAAcPHuSuu+5i6NChZGdn84c//KE2hltvvZV+/fpx8cUXs23bttppjRgxovYU1fnz5zNkyBAGDhzIBRdcwIYNG5gxYwaFhYUMGjSId955h/Lycq6++mqGDh3K0KFD+b//+z8AduzYwahRoxg8eDA33XRTo5LgBx98wJlnnsngwYM588wzWbduXe24srIyxowZw6mnnsp99x36V3zmmWc4/fTTGTRoEDfddJOVz2jrGrpyrakPQICj3OcdgCVAXpR27YA3gVeAsQ1N1+9XKldXV+vo0VNqr+6FQ1f9jh49Raurm3e1byrHFe9VrtXV1TplihPblClTor5uqiOPPFJVVSsrK/Wyyy7Txx57TN966y3t3LmzfvbZZ6qq+oc//EEfeOABVVU9cOCA5uTk6GeffabPP/+8XnjhhVpVVaWbN2/Wbt266V/+8hdVVR0+fLguXbpUt23bpr169aqd1o4dO1RVddq0afrwww/XxvHtb39b33nnHVVV3bhxo5522mmqqnrbbbfpfffdp6qqL7/8sgJaXl5eZzkyMzPrDK+oqNDKykpVVV24cKFeddVVqqr61FNP6QknnKDbt2/Xffv2af/+/XXp0qW6du1aveSSS/Srr75SVdWbb75ZZ8+eXe/0Ve1K5VRGMq9UdgPY677s4D6i/dy5DXgeGBplXMo5VBwOnKJwNVVDE1wcLkXiikZEKCx0Yi0qKqKoyIl1ypQpFBY2L9b9+/fXlqo+55xz+P73v897773H6aefzsknnwzAa6+9xsqVK2v7ByoqKvj0009ZvHgx3/72t2nXrh0nnXQS559/fp3pFxcXc+6559ZOq6aUdqTXX3/9sD6H3bt3s2fPHhYvXswLL7wAwMUXX8zRRx8d97JVVFRwww038OmnnyIiVFZW1o4bOXIkxxxzDABXXXUV7777Lu3bt6ekpKS2LPf+/fs57rjj4p6faX08vTBNRNoBJUBf4FFVXRIxvidwJXA+MRKCiEwCJgF16tf70aHicEVhQxNcHC4Kv8YVTU1SqEkGQLOTARzqQ4gUWfb6v//7vxk9evRhbV555ZUG569xlgevrq7m/fffp1OnTnXGNXUZ7733Xs477zxefPFFNmzYwIgRI+qdpoigqtxwww386le/atL8TOvjaaeyqh5U1UFAL+B0EcmKaPIIcLeqxjxwqaozVTVXVXPT09O9CbYFZWQoTv2fcAXu8OTxa1zRqNtnEC68T8FLo0eP5vHHH6/9hf3JJ5/wxRdfcO655/Lss89y8OBBtm7dyltvvVXnvWeccQZvv/02n3/+OQA7d+4EnDIYe/bsqW03atQopk+fXvu6Jkmde+65zHHvCvfqq6/yn//8J+64Kyoq6NmzJ+CcWRRu4cKF7Ny5k/379/PXv/6Vs846iwsuuIB58+bV9oXs3LmTjRs3xj0/0/ok5CwjVd0FLALGRIzKBZ4VkQ3AWOAxEbkiETF5Rd3icDX3EIBqau4tEAgkZoOWSnFFU5MMioqKmDJlCtXV1UyZMoWioqKEJIUbb7yRfv36MWTIELKysrjpppuoqqriyiuv5Jvf/CYDBgzg5ptvZvjw4XXem56ezsyZM7nqqqsYOHAg11xzDQCXXnopL774Ym2n8u9//3uWLVtGdnY2/fr1qz3badq0aSxevJghQ4bw2muvxdwjzs7OplevXvTq1Ys777yTn/zkJ9xzzz2cddZZdTqHzz77bK677joGDRrE1VdfTW5uLv369eMXv/gFo0aNIjs7m5EjR7J169YWXJMm1YhX/1wikg5UquouEekEvAY8pKov19N+FvCyqs6LNd3c3Fz1e8GxYDBIcfGusOJwzsY4L697Us/oSXZcoVCIQCAQV9tgMMiuXbtqDxPVJInu3ZO7Dtu6xnyGxl9EpERVc2O28TAhZAOzcc4iSgOeU9X7RWQygKrOiGg/i1aSEKDuseR4jy17LZlxNXZj4td12JZZQkhd8SQEL88yWgkMjjJ8RpTmqOoEr2JJBr8Wh/NrXNGkUqzGtAZ2pbJJKD/1VZjGsc+u9bOEYBKmY8eO7NixwzYsKUhV2bFjBx07dkx2KMZDdoMckzC9evVi06ZNlJeXJzsU0wQdO3akV69eyQ7DeMgSgkmYDh061F7B25oEg0GK1xcTyglRtruMjK4ZBEoC5PXNszOiTEqxhGBMM6gqxeuLWTBnAawHxkDp3FJKl5TCeDszyqQWSwjGNIOIEMoJOclgifsAGAahnJAlA5NSrFPZmGYq211W9xr8Me5wY1KIJQRjmimjawbMjxg43x1uTAqxhGBMM6gqgZKAc6hoGDDN/bsEAiUBO8XWpBTrQzCmGUSEvL55MJ5DZxldk0Ggr3OWkfUhmFRiCcGYZgoGg1Z3ybQKdsjImBZgdZdMa2AJwRhjDGAJwRhjjKtNJITIMz3szA/jZ4n+vtr/h6nhWUIQkY4i8oGIrBCRNSJyX5Q240Vkpft4T0QGtnQcwWCQ/PwCMjOVtDTIzFTy8wusxozxpWAwSEF+PpqZCWlpaGYmBfn5nn1fEz0/429enmX0JXC+qu4VkQ7AuyLyqqoWh7X5HBiuqv8RkXxgJs5Z3C1CVSku3sWCBUXukEJKSwsoLXXuK2xnghg/UVV2FRdTtGABAIVAQWkpRaWlTKHlz1xK9PyM/3l5xzQF9rovO7gPjWjzXtjLYqBFa+uKCKFQofuqyH0ATCEUKrQvu/EVEaEwFAIiv61QGGr5ukiJnp/xP8/uqQwgIu2AEqAv8Kiq3h2j7Y+B01T1xijjJgGTAHr37p2zcePGuGNIS6s5Jhp+dKwaEaG6Ou7JGJMYaWmoasS31T2N1YsvbKLnZ5Imnnsqe9qprKoHVXUQzi//00UkK1o7ETkP+D4QNWGo6kxVzVXV3PT09EbFkJGhQEHE0AJ3uDH+ohkZUb6tzvDWMD/jbwk5y0hVdwGLqFsTEhHJBp4ALlfVHS08XwKBApyd4Sk4v32mAEUEAgV2NoXxFVWlIBCI8m2FgkDL10VK9PyM/3nWhyAi6UClqu4SkU7AhcBDEW16Ay8A16nqJx7EQF5ed2r6DMrKhIyMQgIByMvrbsdIja+ICN3z8g4dwy8rozAjAwIBuue1fF2kRM/P+F/MPgQR6Qqkq+o/I4Znq+rKmBN2fvnPBtrh7Ik8p6r3i8hkAFWdISJPAFcDNZ0CVQ0d48rNzdVly5Y1sFiHszozJpUk+vtq/x9tQzx9CPUmBBH5FvAIsA3nDKEJqrrUHfehqg5p2XDj05SEYIwxbV1zO5V/BuS4ncITgadF5KqaabdMiMYYY/wiVh9CO1XdCqCqH7hnAr0sIr2IuJ7AGGNM6ou1h7BHRL5R88JNDiOAy4H+HsdljDEmwWIlhJuJODSkqntwTh39npdBJUs8Rb6sEJhpq+y73/rVmxBUdYWqro8yvFJV53gbVuLFUwTPCuWZtsqK4LUNdgtN4iuCB1ihPNMmWRG8tsMSAvEXwbNCeaYtsiJ4bUdcxe3cK417q+o670OKzavrEOIpgmeF8kybZUXwUl6LFLcTkUuB5cB89/UgEflbi0ToI/EUwbNCeaatsiJ4bUM8xe2CwOnALgBVXQ708SqgZIinCJ4VyjNtlRXBazvi6UOoUtWK1nycMN4ieFYoz7RFVgSv7WiwD0FE/gi8AfwUpxDd7UAHVZ3sfXh1eVnLKJ4iX1YIzLRV9t1PbS11g5zbcK5M/hL4E1AB3NHs6Hwo8ssd7cseTxtjWiP77rd+MQ8ZubfA/JuqXghMTUxIxhhjkiHmHoKqHgT2iUi3BMVjjDEmSeLpVD4ArBKRhcAXNQNV9XbPokqS6upq0tLS6n1tWhe/HhNvybj8uozGn+LZ2v0DuBdYDJSEPWISkY4i8oGIrBCRNSJyX5Q2IiK/F5H1IrJSRJJy0x2AESNGcNxxOaSlVSMCaWnVHHdcDiNGjEhWSMZDfq3N05Jx+XUZjX81uIegqrObOO0vgfNVda+IdADeFZFXVbU4rE0+8E33MQx43P2bUNXV1axeXcGOHcuBHKAE1Rx27FjO6tWDbE+hlfFrbZ6WjMuvy2j8rcGEICKfE+WGOKr69VjvU+d81r3uyw7uI3I6lwP/67YtFpHuInJizY15EiUtLY2dO0twksFynNtAAwxi584SSwatjF9r87RkXH5dRuNv8VyHcEzYy47AOKCHqv68wYk7ZymVAH2BR1X17ojxLwP/parvuq/fAO5W1WUR7SYBkwB69+6ds3HjxoZm3WjO/0c1h5IBwEEgDbsQsxXya22elozLr8tokqJFrkNQ1R1hj82q+ghwfjwBqOpB957MvYDTRSQrMsZob4synZmqmququenp6fHMutFEqnH2EMLluMNNa+PX2jwtGZdfl9H4VzzF7YaEPXJFZDLQpTEzUdVdwCKcu62F2wSEfzt7AVsaM+2WUF1dTY8eNYeLBuHsGQwCltOjRw7V9muqVfFrbZ6WjMuvy2j8LZ7TTn8b9rwK+Bz4VkNvEpF0oFJVd7nlsy8EHopo9jfgVhF5FqczuSLR/Qfg9CFkZXVj9Wqnz0A1DZESevTIISurm/UhtDJ+rc3TknH5dRmNv8XTh/B1Vf0sYtjJqvp5A+/LBmbjHJRPA55T1fvdPQxUdYY438rpOHsO+4CJkf0HkbysZWTXIbQtfj1H365DMF6Ipw8hnj2EeUDk9QHzqHvA/TCquhIYHGX4jLDnCvwwjhgSInLjb8mgdfNrbZ6WjMuvy2j8qd6EICKn4RS16yYiV4WN6opztpExxphWJNYewqnAJUB34NKw4XuAH3gYkzHGmCSoNyGo6kvASyJyhqq+n8CYTIrw6/Fpv8ZljN/F04fwkYj8EOfwUe2hIlX9nmdRGd8LBoMUry8mlBOibHcZGV0zCJQEyOubl9RaOX6Ny5hUEE9CeBr4GBgN3A+MB0JeBmX8TVUpXl/MgjkLYD0wBkrnllK6pBTGJ7cWkB/jMiZVxJMQ+qrqOBG5XFVni8ifgAVeB2b8S0QI5YScje4S9wEwDEI5ya0F5Me4jEkV8ZxXWen+3eWWnugG9PEsIpMSynaX1b3ufIw7PIn8GpcxqSCehDBTRI7GuSfC34C1wK89jcr4XkbXDJgfMXC+OzyJ/BqXMakgnvshPOE+fRuIWfLatA2qSqAk4BybH4bzi3w+sAQCfQNJ7UPwY1zGpIp47odwPPBL4CRVzReRfsAZqvpHz6MzviQi5PXNg/EcOpvnmgwCfZ2zeZLZh+DHuIxJFfHUMnoVeAqYqqoDRaQ98JGqDkhEgJG8rGVkGsev5/v7NS5jkqlF7ocAHKuqz+FU0EVVq3DqQ5s2zq91cvwalzF+F09C+MK9a5oCiEgeUOFpVMYYYxIunusQ7sQ5u+gbIvJ/QDow1tOojDHGJFysaqe9VbVUVT8UkeE4xe4EWKeqlfW9zxhjTGqKdcjor2HP56rqGlVdbcnANFbkiQt+uX1jouOKvBWrX27N6tfPxyRerENG4T1xjb7+QEQygP8FTsDpkJ6pqkURbboBzwC93Vh+o6pPNXZexr/8Wmwu0XGNGDGCitWrKdm5kzRVqkXI6dGDbllZLFq0qMXnFy+/fj4mOWIlBK3nebyqgB+5h5y6ACUislBV14a1+SGwVlUvde/BvE5E5qjqV02Yn/EZvxabS3Rc1dXVVKxezfIdO8gBSoAcVZbv2MGg1auTdqtWv34+JnliJYSBIrIbZ0+hk/sc97WqatdYE1bVrcBW9/keEQkBPXFKX9Q2A7q491Y+CtiJk0hMK+DXYnOJjistLY2SnTvJAZbj3GQcYBA4ewxJulWrXz8fkzz1fhNVtZ2qdlXVLqra3n1e8zpmMogkIn1w7q+8JGLUdCAAbAFWAVNUtc6BVRGZJCLLRGRZeXl5Y2ZtksyvxeYSHVeaKiURw0rc4cnk18/HJIfnP01E5CjgeeAOVd0dMXo0zo+mk3B+ME0XkTrJRlVnqmququamp6d7HLFpSX4tNpfouKpFyIkYluMOTya/fj4mOTxNCCLSAScZzFHVF6I0mQi8oI71wOfAaV7GZBKnptgcS3CKzU1z/y6BQEkgaWezJDqu6upqcnr0YDnOr56D7t/lQE6PHkk728ivn49JnnguTGsSt1/gj0BIVX9XT7NS4ALgHbeI3qnAZ17FZBLLr8XmEh1XWloa3bKyGBR2llFJ2FlGyexD8OPnY5KnweJ2TZ6wyNnAOzh9AzU/gX6Gc4opqjpDRE4CZgEn4nRW/5eqPhNrulbcLvX4tdhcouOKPJsoWWcXRfLr52NaVjzF7TzbQ1DVdzn8WoZobbYAo7yKwfiDX4vNJTquyI2/H5IB+PfzMYnnj2+kMcaYpLOEYIwxBrCEYHzC6ukYk3ye9SEYEy+rp2OMP1hCMEll9XSM8Q9LCCaprJ6OMf5hfQgm6ayejjH+YAnBJJ3V0zHGHywhmKSyejrG+If1IZiksno6xviHJQSTdMFg0OrpGOMDdsjI+ILV0zEm+SwhGGOMASwhGGOMcVlCMMYYD6RifS7PEoKIZIjIWyISEpE1IjKlnnYjRGS52+Ztr+IxxphECQaDFOTno5mZkJaGZmZSkJ/v+9pcXp5lVAX8SFU/FJEuQImILFTVtTUNRKQ78BgwRlVLReQ4D+MxxhjPqSq7iospWrAAgEKgoLSUotJSpuDvM+i8vGPaVmCr+3yPiISAnsDasGbfAV5Q1VK33Tav4jHGmEQQEQpDIQCK3AfAFKAw5O/6XAnpQxCRPsBgDpUuq3EKcLSILBKREhG5vp73TxKRZSKyrLy83ONojTGmeaSsjMKIYYXucD/zPCGIyFHA88Adqro7YnR7IAe4GBgN3Csip0ROQ1Vnqmququamp6d7HbIxxjSLZmRQEDGswB3uZ54mBBHpgJMM5qjqC1GabALmq+oXqrodWAwM9DImY4zxkqpSEAhQhHOYqNr9WwQUBPxdn8uzPgRxDpT9EQip6u/qafYSMF1E2gNH4JQ1i9zTMsaYlCEidM/LO9RnUFZGYUYGBAJ0z/N3fS7xKluJyNnAO8AqnCQJ8DOgN4CqznDb3QVMdNs8oaqPxJpubm6uLlu2zJOYjTGmpfitPpeIlKhqbqw2Xp5l9C7Q4NKr6sPAw17FYYwxyZCK9bnsSmVjjDGAJQRjjDEuSwjGGGMASwimGVKxeFdbY5+RaQy7Y5ppkmAwSPH64kO3veyaQaDEue2l3wt4tRXBYJBdxcW1pz5qRgYF7qmP9hmZaCwhmEZTVYrXF7NgzgJYD4yB0rmllC4phfHJP73OpHaBNZM8lhBMo4kIoZyQkwyWcKhC1TAI5fi7eFdbkcoF1kzyWB+CaZKy3WUwJmLgGHe48YVULbBmkscSgmmSjK4ZMD9i4Hx3uPGFVC2wZpLHEoJpNFUlUBJwDhUNA6a5f5dAoMTfxbvailQusGaSx/oQTKOJCHl982A8h84yuiaDQF/nLCM7Pp18qVxgzSSPZ8XtvGLF7fzDb8W7TF32GZka8RS3s0NGpslSsXhXW2OfkWkMSwjGGGMASwjGGGNclhCMMcYAHiYEEckQkbdEJCQia0RkSoy2Q0XkoIiM9SoeY4wxsXl52mkV8CNV/VBEugAlIrJQVdeGNxKRdsBDwAIPYzHGGNMAz/YQVHWrqn7oPt8DhICeUZreBjwPbPMqFmOMMQ1LSB+CiPQBBnOoDFrN8J7AlcCMBt4/SUSWiciy8vJyz+I0xpi2zPOEICJH4ewB3KGquyNGPwLcraoHY01DVWeqaq6q5qanp3sUqTHGtG2elq4QkQ44yWCOqr4QpUku8Kx7scyxwEUiUqWqf/UyLmOMMXV5lhDE2cr/EQip6u+itVHVk8PazwJetmRgjDHJ4eUewlnAdcAqEVnuDvsZ0BtAVWP2GxhjjEkszxKCqr4LxF04RVUneBWLMcaYhtmVysYYYwBLCMYYY1yWEIwxxgCWEIwxxrgsIRhjjAEsIRhjjHFZQjDGGANYQjDGGOOyhGCMMQawhGCMMcZlCcEYYwxgCcEYY4zLEoIxxhjAEoIxxhiXJQRjjDGAJQRjYlLVmK+NaU28vIVmBvC/wAlANTBTVYsi2owH7nZf7gVuVtUVXsVkTGMEg0GK1xcTyglRtruMjK4ZBEoC5PXNIxgMJjs8Y1qcl7fQrAJ+pKofikgXoEREFqrq2rA2nwPDVfU/IpIPzASGeRiTMXFRVYrXF7NgzgJYD4yB0rmllC4phfHOeOe24ca0Hl7eQnMrsNV9vkdEQkBPYG1Ym/fC3lIM9PIqHmMaQ0QI5YScZLDEfQAMg1BOyJKBaZUS0ocgIn2AwRz6t4rm+8Cr9bx/kogsE5Fl5eXlHkRoTF1lu8tgTMTAMe5wY1ohzxOCiBwFPA/coaq762lzHk5CuDvaeFWdqaq5qpqbnp7uXbDGhMnomgHzIwbOd4cb0wp5mhBEpANOMpijqi/U0yYbeAK4XFV3eBmPMfFSVQIlAWefdhgwzf27BAIlATvbyLRKXp5lJMAfgZCq/q6eNr2BF4DrVPUTr2IxprFEhLy+eTCeQ2cZXZNBoK9zlpH1IZjWSLz6pSMiZwPvAKtwTjsF+BnQG0BVZ4jIE8DVwEZ3fJWq5saabm5uri5btsyTmI2JFHk2kZ1dZFKViJQ0tH318iyjd4GY/zmqeiNwo1cxGNNckRt/SwamNbMrlY0xxgCWEIwxxrgsIRhjjAEsIRhjjHF5dpaRV0SknENnJaWCY4HtyQ6iGVI5/lSOHVI7/lSOHVI7/vpiz1TVmFf2plxCSDUisqyhU738LJXjT+XYIbXjT+XYIbXjb07sdsjIGGMMYAnBGGOMyxKC92YmO4BmSuX4Uzl2SO34Uzl2SO34mxy79SEYY4wBbA/BGGOMyxKCMcYYwBJCixGRDSKySkSWi0idcqzi+L2IrBeRlSIyJBlxRhNH7CNEpMIdv1xEfp6MOOsjIt1FZJ6IfCwiIRE5I2K8n9d9Q7H7dt2LyKlhcS0Xkd0ickdEGz+v+3ji9/P6LxCRNSKyWkT+LCIdI8Y3ft2rqj1a4AFsAI6NMf4inFuECpAHLEl2zI2IfQTwcrLjjBHfbOBG9/kRQPcUWvcNxe7rdR8WZzvgXzgXP6XEuo8zfl+uf5z7038OdHJfPwdMaO66tz2ExLkc+F91FAPdReTEZAeV6kSkK3Auzs2YUNWvVHVXRDNfrvs4Y08VFwD/VNXIKgK+XPdR1Be/n7UHOolIe6AzsCVifKPXvSWElqPAayJSIiKToozvCYTfnX2TO8wPGood4AwRWSEir4pI/0QG14CvA+XAUyLykYg8ISJHRrTx67qPJ3bw77oPdy3w5yjD/bruI9UXP/hw/avqZuA3QCmwFahQ1dcimjV63VtCaDlnqeoQIB/4oYicGzE+2p1V/HLOb0Oxf4izKz0Q+G/grwmOL5b2wBDgcVUdDHwB/DSijV/XfTyx+3ndAyAiRwCXAX+JNjrKMD+s+1oNxO/L9S8iR+PsAZwMnAQcKSLfjWwW5a0x170lhBaiqlvcv9uAF4HTI5psAjLCXvei7i5eUjQUu6ruVtW97vNXgA4icmzCA41uE7BJVZe4r+fhbGQj2/hx3TcYu8/XfY184ENV/XeUcX5d9+Hqjd/H6/9C4HNVLVfVSpx7058Z0abR694SQgsQkSNFpEvNc2AUsDqi2d+A692e/zycXbytCQ61jnhiF5ETRJx7R4rI6Tjfmx2JjjUaVf0XUCYip7qDLgDWRjTz5bqPJ3Y/r/sw36b+wy2+XPcR6o3fx+u/FMgTkc5ufBcAoYg2jV73nt1TuY05HnjR/d60B/6kqvNFZDKAqs4AXsHp9V8P7AMmJinWSPHEPha4WUSqgP3AteqexuATtwFz3F3/z4CJKbLuoeHYfb3uRaQzMBK4KWxYqqz7eOL35fpX1SUiMg/nkFYV8BEws7nr3kpXGGOMAeyQkTHGGJclBGOMMYAlBGOMMS5LCMYYYwBLCMYYY1yWEEybIyIHI6pc9mnCNK4QkX4ehFcz/fkisktEXvZqHsZEsusQTFu0X1UHNXMaVwAvU/ciuHqJSHtVrYqz+cM4BctuaqihMS3F9hCMAUQkR0Tedgv8LaipCikiPxCRpW5xs+fdK0PPxKl987C7h/ENEVkkIrnue44VkQ3u8wki8hcR+TtOAcEjReRJd5oficjl0eJR1TeAPQlZeGNclhBMW9Qp7HDRiyLSAadw2VhVzQGeBB50276gqkPd4mYh4Puq+h5OWYC7VHWQqv6zgfmdAdygqucDU4E3VXUocB5OUolW4dSYhLNDRqYtOuyQkYhkAVnAQreERzucksIAWSLyC6A7cBSwoAnzW6iqO93no4DLROTH7uuOQG/q1qExJuEsIRjjlAleo6pnRBk3C7hCVVeIyAScO2hFU8WhPe6OEeO+iJjX1aq6rsnRGuMRO2RkDKwD0sW9n7GIdAi7EUoXYKt7WGl82Hv2uONqbABy3OdjY8xrAXBbWAXNwc0P35iWYQnBtHmq+hXORvwhEVkBLOdQbfl7gSXAQuDjsLc9C9zldgx/A+fuVTeLyHtArHr5DwAdgJUistp9XYeIvINzw5YLRGSTiIxu6vIZEy+rdmqMMQawPQRjjDEuSwjGGGMASwjGGGNclhCMMcYAlhCMMca4LCEYY4wBLCEYY4xx/T88XoahIYBZQAAAAABJRU5ErkJggg==",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"from sklearn.datasets import load_iris\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"# Load the iris dataset\n",
"iris = load_iris()\n",
"\n",
"# Split the data into training and test sets\n",
"X_train, X_test, y_train, y_test = train_test_split(iris.data[:, :2], iris.target, test_size=0.2, random_state=42)\n",
"\n",
"# Create a KNN classifier with k=5 and euclidean distance\n",
"knn = KNN(k=5, distance='euclidean')\n",
"\n",
"# Train the classifier on the training data\n",
"knn.fit(X_train, y_train)\n",
"\n",
"# Make predictions on the test data\n",
"y_pred = knn.predict(X_test)\n",
"\n",
"# Create scatter plots of the test data with colored points representing the true and predicted labels\n",
"fig, ax = plt.subplots()\n",
"scatter1 = ax.scatter(X_test[y_test==0, 0], X_test[y_test==0, 1], c='b', cmap='viridis', label=iris.target_names[0])\n",
"scatter2 = ax.scatter(X_test[y_test==1, 0], X_test[y_test==1, 1], c='g', cmap='viridis', label=iris.target_names[1])\n",
"scatter3 = ax.scatter(X_test[y_test==2, 0], X_test[y_test==2, 1], c='r', cmap='viridis', label=iris.target_names[2])\n",
"scatter4 = ax.scatter(X_test[:, 0], X_test[:, 1], c='k', cmap='viridis', marker='x', label='Predicted Label')\n",
"ax.set_xlabel('Feature 1')\n",
"ax.set_ylabel('Feature 2')\n",
"ax.set_title('KNN Classifier Results')\n",
"handles = [scatter1, scatter2, scatter3, scatter4]\n",
"labels = [h.get_label() for h in handles]\n",
"ax.legend(handles=handles, labels=labels)\n",
"plt.show()\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: src/MLC/notebooks/linear_regression.ipynb
================================================
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Linear Regression \n",
"\n",
"Linear regression is a statistical method used to model the relationship between a dependent variable (often denoted as \"y\") and one or more independent variables (often denoted as \"x\"). The basic idea of linear regression is to find the straight line that best fits the data points in a scatter plot.\n",
"\n",
"The most common form of linear regression is simple linear regression, which models the relationship between two variables:\n",
"\n",
"$y = mx + b$\n",
"\n",
"where y is the dependent variable, x is the independent variable, m is the slope, and b is the intercept. \n",
"\n",
"Given a set of input data ($\\{x_i, y_i\\}$), the goal of linear regression is to find the values of m and b that best fit the data\n",
"\n",
"\n",
"The values of m and b are chosen to minimize the \"sum of squared errors\" (SSE) $(\\sum (y - \\hat{y})^2)$.\n",
"\n",
"Taking the partial derivatives with respect to m and b, set them equal to 0, and solve for m and b, we get:\n",
"\n",
"m = sum((x - x_mean) * (y - y_mean)) / sum((x - x_mean)^2) \n",
"b = y_mean - m * x_mean\n",
"\n",
"\n",
"Multiple linear regression is a more general form of linear regression that models the relationship between multiple independent variables and one dependent variable. The formula for the best-fit hyperplane in multiple linear regression is:\n",
"\n",
"$y = w_0 + w_1.x_1 + w_2.x_2 + ... + w_n.x_n = X^T. W$"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Code \n",
"### Simple linear regression \n",
"Here is a basic implementation of simple linear regression in Python using the least squares method:"
]
},
{
"cell_type": "code",
"execution_count": 107,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class LinearRegression:\n",
" def __init__(self):\n",
" self.slope = None\n",
" self.intercept = None\n",
"\n",
" def fit(self, X, y):\n",
" n = len(X)\n",
" x_mean = np.mean(X)\n",
" y_mean = np.mean(y)\n",
" numerator = 0\n",
" denominator = 0\n",
" for i in range(n):\n",
" numerator += (X[i] - x_mean) * (y[i] - y_mean)\n",
" denominator += (X[i] - x_mean) ** 2\n",
" self.slope = numerator / denominator\n",
" self.intercept = y_mean - self.slope * x_mean\n",
"\n",
" def predict(self, X):\n",
" y_pred = []\n",
" for x in X:\n",
" y_pred.append(self.slope * x + self.intercept)\n",
" return y_pred\n"
]
},
{
"cell_type": "code",
"execution_count": 109,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.6\n",
"2.2\n",
"[2.8000000000000003, 3.4000000000000004, 4.0, 4.6, 5.2]\n"
]
}
],
"source": [
"X = np.array([1, 2, 3, 4, 5])\n",
"y = np.array([2, 4, 5, 4, 5])\n",
"lr = LinearRegression()\n",
"lr.fit(X, y)\n",
"print(lr.slope) # Output: 0.6\n",
"print(lr.intercept) # Output: 2.2\n",
"y_pred = lr.predict(X)\n",
"print(y_pred) # Output: [2.8, 3.4, 4.0, 4.6, 5.2]\n",
"\n",
"\n",
"# print(f\"The value of x is {x:.2f}\")\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Vectorized "
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"$y = X.W$ \n",
"$W = (X^T.X)^{-1}X^T.y $"
]
},
{
"cell_type": "code",
"execution_count": 110,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class LinearRegression:\n",
" def __init__(self):\n",
" self.W = None\n",
"\n",
" def fit(self, X, y):\n",
" '''\n",
" X: n x d \n",
" '''\n",
" # Add bias term to X -> [1 X]\n",
" n = X.shape[0]\n",
" X = np.hstack([np.ones((n, 1)), X])\n",
" self.W = np.linalg.inv(X.T @ X) @ X.T @ y\n",
"\n",
" def predict(self, X):\n",
" n = X.shape[0]\n",
# while predicting as well the bias must be prepended
" X = np.hstack([np.ones((n, 1)), X])\n",
" return X @ self.W\n",
" \n"
]
},
{
"cell_type": "code",
"execution_count": 111,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[3. 1. 2.]\n",
"[43. 55.]\n"
]
}
],
"source": [
"# Create example input data\n",
"X = np.array([[2, 2], [4, 5], [7, 8]])\n",
"y = np.array([9, 17, 26])\n",
"\n",
"# Fit linear regression model\n",
"lr = LinearRegression()\n",
"lr.fit(X, y)\n",
"print(lr.W) # [3. 1. 2.]\n",
"\n",
"# Make predictions on new data\n",
"X_new = np.array([[10, 11], [13, 14]])\n",
"y_pred = lr.predict(X_new)\n",
"print(y_pred) # Output: [43. 55.]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Improvements \n",
"here are some improvements to the simple linear regression implementation to make it more robust:\n",
"\n",
"1. Add input **validation**: Add input validation to check that the input arrays X and y have the same length and are not empty.\n",
"\n",
"2. Use NumPy broadcasting: Instead of looping through the data to calculate the numerator and denominator, we can use NumPy broadcasting to perform the calculations in a vectorized way. This will make the code faster and more efficient.\n",
"\n",
"3. Add **regularization**: Regularization can help prevent overfitting by adding a penalty term to the cost function. One common regularization technique is L2 regularization, which adds the sum of squares of the coefficients to the cost function. This can be easily added to the code by adding a regularization parameter to the constructor.\n",
"\n",
"4. Use **gradient descent**: For large datasets, calculating the inverse of the matrix in the normal equation can be computationally expensive. To overcome this, we can use gradient descent to minimize the cost function. This can be implemented by adding a method that updates the coefficients iteratively using the gradient descent algorithm.\n",
"\n",
"Here's the updated code that incorporates these improvements:"
]
},
{
"cell_type": "code",
"execution_count": 105,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"\n",
"class LinearRegressionGD:\n",
" def __init__(self, regul=0):\n",
" self.regul = regul\n",
" self.W = None\n",
"\n",
" def fit(self, X, y, lr=0.01, num_iter=1000):\n",
" # Input validation\n",
" if len(X) != len(y) or len(X) == 0:\n",
" raise ValueError(\"X and y must have the same length and cannot be empty\")\n",
" \n",
" # Add bias term to X -> [1 X]\n",
" X = np.hstack([np.ones((len(X), 1)), X])\n",
"\n",
" # Initialize W to zeros\n",
" self.W = np.zeros(X.shape[1])\n",
"\n",
" # Use gradient descent to minimize cost function\n",
" for i in range(num_iter):\n",
" # Calculate predicted values\n",
" y_pred = np.dot(X, self.W)\n",
"\n",
" # Calculate cost function\n",
" cost = np.sum((y_pred - y) ** 2) + self.regul * np.sum(self.W ** 2)\n",
"\n",
" # Calculate gradients\n",
" gradients = 2 * np.dot(X.T, (y_pred - y)) + 2 * self.regul * self.W\n",
"\n",
" # Update W\n",
" self.W = self.W - lr * gradients\n",
"\n",
" if (i % 1000 == 0 ): print(cost)\n",
"\n",
" def predict(self, X):\n",
" # Add bias term to X\n",
" X = np.hstack([np.ones((len(X), 1)), X])\n",
"\n",
" # Calculate predicted values\n",
" y_pred = np.dot(X, self.W)\n",
" return y_pred\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test "
]
},
{
"cell_type": "code",
"execution_count": 103,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"86.0\n",
"2.8791287270130335\n",
"2.8791287270130344\n",
"2.8791287270130344\n",
"2.8791287270130344\n",
"2.8791287270130344\n",
"2.8791287270130344\n",
"2.8791287270130344\n",
"2.8791287270130344\n",
"2.8791287270130344\n",
"[1.99964292 0.65345474]\n",
"[2.65309766 3.3065524 3.96000714 4.61346188 5.26691662]\n"
]
}
],
"source": [
"X = np.array([[1, 2, 3, 4, 5]]).T\n",
"y = np.array([2, 4, 5, 4, 5])\n",
"lr = LinearRegressionGD(regul=0.1)\n",
"lr.fit(X, y, lr=0.01, num_iter=10000)\n",
"print(lr.W) # Output: [ 1.99964292 0.65345474 ]\n",
"y_pred = lr.predict(X)\n",
"print(y_pred) # # Output: [2.65309766, 3.3065524, 3.96000714, 4.61346188, 5.26691662]\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualize "
]
},
{
"cell_type": "code",
"execution_count": 104,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlVElEQVR4nO3deZyVdfn/8ddbxAXFqJiSAMHUvqYmQpPivqQmYNJiZqKWWoRLuZtLqagkuScuRC5hoonmFqFpCeauAyLKouKCEMSmgAQiy/X743P4NQ0zMMjc5z5nzvv5eJzHnHvh3NfcnLmve/l8ro8iAjMzq1wb5B2AmZnly4nAzKzCORGYmVU4JwIzswrnRGBmVuGcCMzMKpwTgZUcSXtLej3vOJoDSRMk7Zd3HFbanAgsN5LelXRg3fkR8VRE/F8eMdUl6WJJyyQtkjRf0rOSds87rsaKiB0jYnTecVhpcyIwK5C0YQOL7omIzYG2wCjg3gy2LUn+e7Rc+ItnJUfSfpKm15p+V9JZksZLWiDpHkmb1Fp+qKRxtc7Yd6617FxJb0n6UNJESd+utexHkp6RdK2k94GL1xRXRCwHhgHtJVUVPuNTkm6VNFPSvyRdJqlFYVkLSVdLmivpHUmnSIpVCUfSaEkDJD0DLAa+KGl7SY9Lel/S65KOqBVvz8Lv8GFhW2cV5reVNKLw+78v6alVSaX2VZekjSVdJ2lG4XWdpI1r73NJZ0qaXfh9jvtk/4NWbpwIrFwcARwCbA3sDPwIQFI34Dbgp8Bngd8BD686wAFvAXsDnwL6A3dKalfrc3cD3gY+BwxYUwCSNgKOBeYBHxRmDwWWA9sCXYGDgR8Xlv0E6AHsAnQDvlXPxx4D9AVaA3OAx4G7CvH8ALhJ0o6FdW8FfhoRrYGdgCcK888EpgNVwOeB84H6asdcAHQvxNMF2BX4Za3lW5L2U3vgBOBGSZ9ewy6xZsKJwMrF9RExIyLeB/5COphBOtj+LiJeiIgVETEUWEo64BER9xb+3cqIuAd4k3QAXGVGRAyKiOURsaSBbR8haT6wpLC9wyNiuaTPkw70p0XEfyJiNnAtcOSqfwf8NiKmR8QHwMB6PvsPETGhcLVxCPBuRNxeiGcs8Gfg8MK6y4AdJG0RER8Ulq+a3w7oFBHLCs9Y6ksEfYBLImJ2RMwhJcZjai1fVli+LCJGAouAknhWY9lyIrBy8e9a7xcDmxfedwLOLNwWmV84YHcEvgAg6dhat43mk86k29b6rGmN2PbwiGhDOtt+DfhqrW23BGbW+vzfkc7mKcRQ+/Pr21bteZ2A3er8Ln1IZ+oA3wV6AlMlPVnrofWVwBTgMUlvSzq3gd/jC8DUWtNTC/NWmVdISKvU3s/WjDX0cMysXEwDBkTEard1JHUCfg98HXguIlZIGgeo1mqNLr8bEXMl/RR4SdJdhW0vBdrWOYCuMhPoUGu6Y30fW+d3eTIiDmpg+y8BvSW1BE4BhgMdI+JD0u2hMwu3kUZJeiki/lHnI2aQks2EwvRWhXlW4XxFYHlrKWmTWq91PTn5PdBP0m6FljebSeolqTWwGelAOweg8PBzp/UJNiImA38DzomImcBjwNWStpC0gaRtJO1bWH04cKqk9pLaAL9Yy8ePAL4k6RhJLQuvr0n6sqSNJPWR9KmIWAYsBFYUfq9DJW0rSbXmr6jn8+8GfimpSlJb4ELgzvXZH9Y8OBFY3kaS7r2vel28Lv84ImpI9+1vID3AnULhQXJETASuBp4DZgFfAZ5pgpivBPpK+hzp4fFGwMTC9u8j3a+HlKQeA8YDL5N+1+XUf5CmcGZ/MOkZwwzS7bDfAKsefB8DvCtpIdAPOLowfzvg76R7+s8BNzXQd+AyoKYQz6vA2MI8q3DywDRmxSGpBzA4IjrlHYtZbb4iMMuIpE0Lbf83lNQeuAh4IO+4zOryFYFZRiS1Ap4Etifd9vorcGpELMw1MLM6nAjMzCqcbw2ZmVW4sutH0LZt2+jcuXPeYZiZlZUxY8bMjYiq+paVXSLo3LkzNTU1eYdhZlZWJE1taJlvDZmZVTgnAjOzCudEYGZW4ZwIzMwqnBOBmVmFcyIwM6twTgRmZhXOicDMrNQtXgxXXAHPNEUV9dU5EZiZlaqlS+GGG2CbbeAXv4ARIzLZTNn1LDYza/aWL4c77oD+/eG992CffeDee2GvvTLZnK8IzMxKxcqV8Kc/wY47wgknwOc+B3/7G4wenVkSACcCM7P8RcDDD0PXrvCDH8BGG8EDD8CLL8LBB4OU6eadCMzM8hIBf/877L479O6dHgoPGwbjxsG3vpV5AljFicDMLA/PPgsHHAAHHQQzZsDvfw8TJ8JRR0GLFkUNxYnAzKyYXn4ZevWCPfdMB/7f/hbeeAN+/GNo2TKXkJwIzMyKYdIk+N73oFs3eO45uPxyePtt+PnPYZNNcg3NzUfNzLL0zjtw8cVw553QqhX86ldwxhnQpk3ekf1/TgRmZln417/gssvglltgww3h9NNTp7CqekeLzJUTgZlZU5ozBwYOhJtuSh3DfvITuOACaN8+78ga5ERgZtYUFiyAq6+Ga69NzUCPOQYuugi23jrvyNbKicDMbH385z8waFAqCvfBB+mBcP/+8OUv5x1Zo2XaakjSu5JelTROUk09yyXpeklTJI2X1C3LeMyyMmwYdO4MG2yQfg4blndElrmPPkpNP7/4RTjvPNhjDxg7FoYPb/IkkPX3qxhXBPtHxNwGlvUAtiu8dgNuLvw0KxvDhkHfvuluAMDUqWkaoE+f/OKyjCxbBkOHwiWXwLRpsN9+qRzEHntksrlifL/y7kfQG7gjkueBNpLa5RyT2Tq54IL//pGusnhxmm/NyMqVcNddsMMO6QHwF76QykM88URmSQCK8/3KOhEE8JikMZL61rO8PTCt1vT0wrz/IamvpBpJNXPmzMkoVLNP5r331m2+lZkIePBB6NIlnYK3apUKxD33HHz965nXAyrG9yvrRLBnRHQj3QI6WdI+dZbXtwdjtRkRQyKiOiKqq0qwDa5Vtq22Wrf5ViYi4LHHYLfd4Nvfho8/hrvvTiUivvnNohWEK8b3K9NEEBEzCj9nAw8Au9ZZZTrQsdZ0B2BGljGZNbUBA9JJYm2tWqX5Vqaefjrd+//GN2DWLLj1VpgwAY48Mj2xLaJifL8y+40kbSap9ar3wMHAa3VWexg4ttB6qDuwICJmZhWTWRb69IEhQ6BTp3SS2KlTmvaD4jI0Zgz06AF7750KwQ0alH4ef3zqHZyDYny/FLHanZim+WDpi6SrAEitk+6KiAGS+gFExGBJAm4ADgEWA8dFxGrNTGurrq6Ompo1rmJmtm4mTIALL4T774fPfCaVgjjllNVPxcuYpDERUV3fssxSXES8DXSpZ/7gWu8DODmrGMzM1uitt1JBuGHDYPPNU0/g00+HT30q78iKyj2LzazyTJ8Ol14Kt92WxgA466x0FfDZz+YdWS6cCMyscsyencYBuPnm1C/gpz9NDfLbVXb3JScCM2v+PvgArroqlYRYsgR++MP0TKBz57wjKwlOBGbWfC1alA7+V10F8+fD97+fCsL93//lHVlJcSIws+bno4/S7Z/LL0/jA3zzm+mZQJfV2q8Y+dcaMjNrOsuWwe9+B9tum4aD3HnnVAri4YedBNbAicDMyt+KFWlM4O23h379Uv2FJ55IReG6d887upLnRGBm5SsidQLbeec0ItgWW8CIEfDMM7D//nlHVzacCMys/ETAI49AdTV897upKejw4alERK9eRSsI11w4EZhZefnnP2GffaBnT3j/fbj9dnj11TREZJELwjUX3mtmVh5eeilVA91331Qa4qab4PXX4Uc/yq0gXHPhRGBmpe3VV9N4ALvumm79XHVVSgQnnggbbZR3dM2C06iZlaYpU1IRuLvvhtatU0ew005LD4StSTkRmFlpee+91Pnr9tth441TMbizz07loS0TTgRmVhpmzYJf/xoGFyrVn3QSnH8+bLllvnFVACcCM8vX++/DlVfC9dfD0qXp4e+FF3rQ5yJyIjCzfHz4IVx3XXr4++GHaTzg/v1hu+3yjqziOBGYWXEtWZKafg4cCHPnQu/e6ZnAV76Sd2QVy81Hzaw4Pv44VQTddts0IljXrvDCC/Dgg04COXMiMLNsrVgBQ4emgnAnnQRbbw2jR8Njj6W+AZY7JwIzy8bKlXDvvbDTTukB8Kc/DSNHwlNPpd7BVjIyTwSSWkh6WdKIepbtJ2mBpHGF14VZx2NmGYuAv/41FYQ74ohUAO6++6CmBnr0cEG4ElSMh8WnApOAhroDPhURhxYhDjPL2qhR8MtfwrPPwhe/CHfcAUcdBS1a5B2ZrUGmVwSSOgC9gFuy3I6Z5eyFF+DAA+GAA2Dq1NQpbPLkNEaAk0DJy/rW0HXAOcDKNayzu6RXJD0iaceM4zGzpjR+PBx2WBoFbPx4uOYaePNN+OlPoWXLvKOzRsosEUg6FJgdEWPWsNpYoFNEdAEGAQ828Fl9JdVIqpkzZ07TB2tm6+b111MHsC5d0vgAl10Gb78Np58Om26ad3S2jrK8ItgTOEzSu8CfgAMk3Vl7hYhYGBGLCu9HAi0lta37QRExJCKqI6K6qqoqw5DNbI2mToXjj4cddkhDQp5/PrzzDlxwAWy+ed7R2SeUWSKIiPMiokNEdAaOBJ6IiKNrryNpSyk1IZC0ayGeeVnFZGaf0MyZ8LOfpfIPd90FP/95ugIYMCA1C7WyVvQSE5L6AUTEYOBw4ERJy4ElwJEREcWOycwaMG8e/OY3cMMNqWfwCSekVkEdO+YdmTUhldtxt7q6OmpqavIOw6x5W7gQrr0Wrr4aFi1KTUAvvjiVh7CyJGlMRFTXt8xF58zsvxYvhhtvTFcB8+bBd74Dl1wCO7pBX3PmEhNmlsYBuPFG2GYbOOec1Cv4pZfgz392EqgAviIwq2TLl8Mf/5jGAZg6FfbeG4YPTz+tYviKwKwSrVwJ99yTCsIdfzxUVcGjj8KTTzoJVCAnArNKEgF/+Qt065Y6hG24Idx/P7z4InzjGy4IV6GcCMwqxT/+AbvvnkpCLFoEd94Jr7wC3/62E0CFcyIwa+6eey4VgzvwQPjXv2DIEJg0Cfr0cUE4A5wIzJqvcePg0ENhjz1gwoQ0UPybb8JPfuKCcPY/nAjMmpvJk9OAMF27wjPPwK9/DW+9BaeeCptsknd0VoLcfNSsuXjnndQM9I9/hFatUimIM8+ENm3yjsxKnBOBWbmbMSOVgb7lFthgAzjtNDj33NQk1KwRnAjMytXcuTBwYOoRvHw5/PjH6Sqgffu8I7My40RgVm4WLEjF4K69NtUGOvpouOiiNEaw2SfgRGBWLv7zHxg0CK64Aj74AA4/PD0T2GGHvCOzMudEYFbqli5Nbf8HDIBZs6BnT7j00tQ72KwJOBGYlarly+EPf0hloKdNg333TdVA99wz78ismXE/ArNSs3Il3H03fPnLqfNXu3bw2GMwapSTgGXCicCsVETAQw/BLrukEcE23TRNP/88HHSQ6wFZZpwIzPIWkc74d9sNvvUt+OijdEUwblwqEOcEYBlzIjDL0zPPwP77pxLQs2alTmETJ6YS0Rv4z9OKw980szyMHZta/+y1V6oNNGgQvPEGnHBCGiPArIicCMyKaeLE1P7/q19N9/4HDkwF4U45BTbeOO/orEJlnggktZD0sqQR9SyTpOslTZE0XpIbRpeIYcOgc+d0d6Jz5zRt6+Htt+HYY9PQkH/7G1x4YSoS94tfwGab5R1d0fn7VVqKcQ16KjAJ2KKeZT2A7Qqv3YCbCz8tR8OGQd++qXoBpDHN+/ZN7/v0yS+usjR9eioId+ut6ZbPmWemg3/btnlHlht/v0pPplcEkjoAvYBbGlilN3BHJM8DbSS1yzImW7sLLvjvH+kqixen+dZIs2fDGWfAttvCbbelI91bb8GVV1Z0EgB/v0pR1lcE1wHnAK0bWN4emFZrenph3szaK0nqC/QF2GqrrZo8SPtf7723bvOtlvnz4aqr0mhgS5ak20EXXZTufxjg71cpyuyKQNKhwOyIGLOm1eqZF6vNiBgSEdURUV3lGuuZayjXOgevwaJFaSSwrbdONYF69UrDQ95+u5NAHf5+lZ4sbw3tCRwm6V3gT8ABku6ss850oGOt6Q7AjAxjskYYMCANcFVbq1ZpvtXx0Ufp7H+bbdK9jb32gpdfhnvuge23zzu6kuTvV+nJLBFExHkR0SEiOgNHAk9ExNF1VnsYOLbQeqg7sCAiZtb9LCuuPn1SsctOnVKn1k6d0rQf5NWybFnaKdttB6efnloDPfss/OUvqUSENcjfr9JT9J4rkvoBRMRgYCTQE5gCLAaOK3Y8Vr8+ffyHWa8VK1L5h4svTg9/u3eHoUPhgAPyjqys+PtVWoqSCCJiNDC68H5wrfkBnFyMGMzWSwQ88EBq/z9hAnTpks7+e/VyLSAre+5ZbLYmEfDoo/C1r8F3v5vGCLjnnlQi4tBDnQSsWXAiMGvIP/+ZBoPp0QPmzUstgF57DY44wgXhrFnxt9msrpqaVA10331hyhS48UZ4/XX40Y9cEM6aJScCs1Veew2+8510G2jMmNQLeMoUOOkk2GijvKMzy4xPb8ymTEm9f+++GzbfHPr3h9NOgy3qK49l1vw4EVjlmjYNLr001QLaaCM4+2w45xz47GfzjsysqJwIrPLMmgWXXw4335ymTzoJzj8fttwy37jMcuJEYJXjgw/Sff/f/haWLoUf/jD1C+jUKe/IzHLlRGDN34cfpoP/VVfBggVpPOD+/eFLX8o7MrOS4ERgzdeSJen2z+WXw9y5cNhh6ZnAzjvnHZlZSXHzUWt+Pv4YBg9Og8KceWYqAvf88/DQQ04CZvVwIrDmY8UKuOOOVP75xBPTOACjRsHjj8NuHgHVrCFOBFb+Vq6E++6Dr3wlPQBu0wb++ld4+mnYb7+8ozMreU4EVr4iYORIqK6G730vzbv33lQiomdPF4QzayQnAitPo0en0cB69UrjBA8dCq++Cocf7oJwZutorX8xkk6R9OliBGO2Vi++CAcdBPvvD+++m1oFTZ6cBolv0SLv6MzKUmNOnbYEXpI0XNIhkq+3LQfjx0Pv3umh77hxcPXVqUZQv34uCGe2ntaaCCLil8B2wK3Aj4A3Jf1a0jYZx2YGb7wBP/hBagL65JOpH8Dbb8MZZ8Cmm+YdnVmz0KibqYUhJf9deC0HPg3cJ+mKDGOzSjZ1KpxwAuywAzz8MJx7bkoAv/wltG6dd3RmzcpaexZL+jnwQ2AucAtwdkQsk7QB8CZwTrYhWkX5979hwAAYMiRNn3IKnHcefP7z+cZl1ow1psREW+A7ETG19syIWCnp0GzCsoozb14qCHf99aln8PHHw69+BR075h2ZWbO31kQQEReuYdmkhpZJ2gT4J7BxYTv3RcRFddbZD3gIeKcw6/6IuGStUVvzsXAhXHstXHNNKg531FFw8cWpPISZFUWWReeWAgdExCJJLYGnJT0SEc/XWe+piPCVRaVZsiSNBTxwYLoa+Pa34ZJLYKed8o7MrOJk1vMmkkWFyZaFV2S1PSsTH38MN90E22yTRgSrroaXXoL773cSMMtJpl0wJbWQNA6YDTweES/Us9rukl6R9IikHRv4nL6SaiTVzJkzJ8uQLSvLl8Ptt6cxAE4+Od36efJJePTRlAzMLDeZJoKIWBERuwAdgF0l1T3lGwt0ioguwCDgwQY+Z0hEVEdEdVVVVZYhW1NbuRKGD09n+8cfD23bpoP/k0/CPvvkHZ2ZUaRaQxExHxgNHFJn/sJVt48iYiTQUlLbYsRkGYuAESOgWzf4/vdhww3T7Z+XXoJvfMMF4cxKSGaJQFKVpDaF95sCBwKT66yz5aqSFZJ2LcQzL6uYrEieeAL22AO++c3UEujOO+GVV9IDYScAs5KTZauhdsBQSS1IB/jhETFCUj+AiBgMHA6cKGk5sAQ4stCL2crR88/DBRekRNChA/zud3DccdCyZd6RmdkaZJYIImI80LWe+YNrvb8BuCGrGKxIxo1Lnb9GjICqqtQvoF8/2GSTvCMzs0Zw4Xb75CZPTvf/u3ZNo4ENGJDqAZ12mpOAWRnJ8taQNVfvvgv9+6fxgTfdNN0OOuusNESkmZUdJwJrvBkz0ln/73+fRgE79dRUFfRzn8s7MjNbD04EtnZz58JvfgM33JA6hp1wQioH3aFD3pGZWRNwIrCGLViQisFdey0sWgRHHw0XXZTKQ5hZs+FEYKv7z3/S2f8VV8D778N3v5sKwu2wQ96RmVkGnAjsv5YuTQPCDBgAs2ZBjx5w2WWpd7CZNVtOBJbu+w8dms7633sP9t0X7rsP9tor78jMrAjcj6CSrVwJd9+dbvn8+MdpOMjHHoNRo5wEzCqIE0ElioCHHoJddkkjgm28MTz4ILzwAhx0kOsBmVUYJ4JKEgGPPw7du8O3vgUffQR33ZUKwvXu7QRgVqGcCCrFM8/A/vvDwQfDzJlwyy0wcSL84Aepc5iZVSwfAZq7sWOhZ890z3/yZLj+enjzzdQpbEO3FTAzJ4Lma+JEOPxw+OpXU3nogQPhrbfgZz9LzwTMzAp8StjcvP02XHwxDBsGrVrBhRfCGWfApz6Vd2RmVqKcCJqLf/0LLr0Ubr013fI54wz4xS/SGMFmZmvgRFDu5syByy+Hm25K/QL69k1lob/whbwjM7My4URQrubPh6uvTgXhliyBY49Nt4G23jrvyMyszDgRlJtFi2DQoFQQbv58OOKINEjM9tvnHZmZlSkngnLx0UcweHC6DTR7Nhx6aHomsMsueUdmZmXOiaDULVsGf/hDKgg3fXrqFPbgg7D77nlHZmbNRGb9CCRtIulFSa9ImiCpfz3rSNL1kqZIGi/J9Y5XWbEiNQH98pfTA+AOHeAf/4AnnnASMLMmlWWHsqXAARHRBdgFOERS9zrr9AC2K7z6AjdnGE95iIAHHoAuXdKIYJttBn/5Czz7LBxwQN7RmVkzlFkiiGRRYbJl4RV1VusN3FFY93mgjaR2WcVU0iLgb3+DXXeF73wn3RL605/g5ZfT8wAXhDOzjGRaYkJSC0njgNnA4xHxQp1V2gPTak1PL8yr+zl9JdVIqpkzZ05m8ebmqafSYDCHHJL6Bdx2G0yYAN//vgvCmVnmMj3KRMSKiNgF6ADsKmmnOqvUd5pb96qBiBgSEdURUV1VVZVBpDmpqUkH/332gSlT4MYb4Y034LjjXBDOzIqmKKebETEfGA0cUmfRdKBjrekOwIxixJSr115Lt3++9jV46SW48sqUCE46CTbaKO/ozKzCZNlqqEpSm8L7TYEDgcl1VnsYOLbQeqg7sCAiZmYVU+6mTEkPgHfeGf7+91Qc7p134KyzUoE4M7McZHn/oR0wVFILUsIZHhEjJPUDiIjBwEigJzAFWAwcl2E8+Zk2LXX+uu22dMZ/9tlwzjnw2c/mHZmZWXaJICLGA13rmT+41vsATs4qhtzNmpV6At98c2oVdOKJcP750K4yG0aZWWnyE8ksfPBBuu//29/C0qXwwx+mgnCdOuUdmZnZapwImtKHH6aD/1VXwYIFcOSRqSDcl76Ud2RmZg1yImgKS5ak2z+XXw5z58Jhh6VnAjvvnHdkZmZr5d5K6+Pjj1NF0G23hTPPTJVAn38eHnrIScDMyoYTwSexYgXccUcaA+DEE6FzZxg1Ch5/HHbbLe/ozMzWiRPBuli5Eu67D77ylfQAuE0b+Otf4emnYb/98o7OzOwTcSJojAgYORKqq+F730vz7r03lYjo2dMF4cysrDkRrM3o0bD33tCrVxoacuhQePVVOPxwF4Qzs2bBR7KGvPgiHHRQGhHsnXdSq6DJk9Mg8S1a5B2dmVmTcSKoa/x46N07PfQdNw6uvjrVCOrXzwXhzKxZcj+CVd54Ay66CO65B7bYIvUDOPVUaN0678jMzDLlRDB1ahoYfuhQ2HhjOPfcVA30M5/JOzIzs6Ko3ETw73/DgAEwZEiaPuUUOO88+Pzn843LzKzIKi8RzJsHV1wBgwalnsHHHw+/+hV07Lj2f2tm1gxVTiJYuBCuvRauuSYVhzvqqDQwzLbb5h2ZmVmuKqfV0AMPpAP/17+eWgbdeaeTgJkZlXRF0KdPKg3RrVvekZiZlZTKuSLYcEMnATOzelROIjAzs3o5EZiZVTgnAjOzCudEYGZW4TJLBJI6SholaZKkCZJOrWed/SQtkDSu8Lowq3jMzKx+WTYfXQ6cGRFjJbUGxkh6PCIm1lnvqYg4NMM4zMxsDTK7IoiImRExtvD+Q2AS0D6r7ZmZ2SdTlGcEkjoDXYEX6lm8u6RXJD0iaccG/n1fSTWSaubMmZNlqGZmFSfzRCBpc+DPwGkRsbDO4rFAp4joAgwCHqzvMyJiSERUR0R1VVVVpvGamVWaTBOBpJakJDAsIu6vuzwiFkbEosL7kUBLSW2zjMnMzP5Xlq2GBNwKTIqIaxpYZ8vCekjatRDPvKxiMjOz1WXZamhP4BjgVUnjCvPOB7YCiIjBwOHAiZKWA0uAIyMiMozJzMzqyCwRRMTTgNayzg3ADVnFYGZma+eexWZmFc6JwMyswjkRmJlVOCcCM7MK50RgZlbhnAjMzCqcE4GZWYVzIjAzq3BOBGZmFc6JwMyswjkRmJlVOCcCM7MK50RgZlbhnAjMzCqcE4GZWYVzIjAzq3BOBGZmFc6JwMyswjkRmJlVOCcCM7MK50RgZlbhMksEkjpKGiVpkqQJkk6tZx1Jul7SFEnjJXXLIpZhw6BzZ9hgg/Rz2LAstmJmVp42zPCzlwNnRsRYSa2BMZIej4iJtdbpAWxXeO0G3Fz42WSGDYO+fWHx4jQ9dWqaBujTpym3ZGZWnjK7IoiImRExtvD+Q2AS0L7Oar2BOyJ5HmgjqV1TxnHBBf9NAqssXpzmm5lZkZ4RSOoMdAVeqLOoPTCt1vR0Vk8WSOorqUZSzZw5c9Zp2++9t27zzcwqTeaJQNLmwJ+B0yJiYd3F9fyTWG1GxJCIqI6I6qqqqnXa/lZbrdt8M7NKk2kikNSSlASGRcT99awyHehYa7oDMKMpYxgwAFq1+t95rVql+WZmlm2rIQG3ApMi4poGVnsYOLbQeqg7sCAiZjZlHH36wJAh0KkTSOnnkCF+UGxmtkqWrYb2BI4BXpU0rjDvfGArgIgYDIwEegJTgMXAcVkE0qePD/xmZg3JLBFExNPU/wyg9joBnJxVDGZmtnbuWWxmVuGcCMzMKpwTgZlZhXMiMDOrcErPa8uHpDnA1E/4z9sCc5swnKZSqnFB6cbmuNaN41o3zTGuThFRb4/csksE60NSTURU5x1HXaUaF5RubI5r3TiudVNpcfnWkJlZhXMiMDOrcJWWCIbkHUADSjUuKN3YHNe6cVzrpqLiqqhnBGZmtrpKuyIwM7M6nAjMzCpcs0wEkm6TNFvSaw0sl6TrJU2RNF5StxKJaz9JCySNK7wuLEJMHSWNkjRJ0gRJp9azTtH3VyPjymN/bSLpRUmvFOLqX886eeyvxsRV9P1Va9stJL0saUQ9y3L5e2xEXHnur3clvVrYbk09y5t2n0VEs3sB+wDdgNcaWN4TeIRUHbU78EKJxLUfMKLI+6od0K3wvjXwBrBD3vurkXHlsb8EbF5435I0/Gr3EthfjYmr6Pur1rbPAO6qb/t5/T02Iq4899e7QNs1LG/SfdYsrwgi4p/A+2tYpTdwRyTPA20ktSuBuIouImZGxNjC+w+BSaw+bnTR91cj4yq6wj5YVJhsWXjVbXGRx/5qTFy5kNQB6AXc0sAqufw9NiKuUtak+6xZJoJGaA9MqzU9nRI4yBTsXri8f0TSjsXcsKTOQFfS2WRtue6vNcQFOeyvwu2EccBs4PGIKIn91Yi4IJ/v13XAOcDKBpbn9f26jjXHBfn9PQbwmKQxkvrWs7xJ91mlJoL6BswphbOnsaR6IF2AQcCDxdqwpM1J40ufFhEL6y6u558UZX+tJa5c9ldErIiIXUhjbO8qaac6q+SyvxoRV9H3l6RDgdkRMWZNq9UzL9P91ci4cvt7BPaMiG5AD+BkSfvUWd6k+6xSE8F0oGOt6Q7AjJxi+f8iYuGqy/uIGAm0lNQ26+1Kakk62A6LiPvrWSWX/bW2uPLaX7W2Px8YDRxSZ1Gu36+G4sppf+0JHCbpXeBPwAGS7qyzTh77a61x5fn9iogZhZ+zgQeAXeus0qT7rFITwcPAsYUn792BBRExM++gJG0pSYX3u5L+f+ZlvE0BtwKTIuKaBlYr+v5qTFw57a8qSW0K7zcFDgQm11ktj/211rjy2F8RcV5EdIiIzsCRwBMRcXSd1Yq+vxoTVx77q7CtzSS1XvUeOBio29KwSfdZloPX50bS3aQn/m0lTQcuIj08IyIGAyNJT92nAIuB40okrsOBEyUtB5YAR0ahiUCG9gSOAV4t3F8GOB/YqlZceeyvxsSVx/5qBwyV1IJ0YBgeESMk9asVVx77qzFx5bG/6lUC+6sxceW1vz4PPFDIQRsCd0XEo1nuM5eYMDOrcJV6a8jMzAqcCMzMKpwTgZlZhXMiMDOrcE4EZmYVzonAbD0oVUl9R9JnCtOfLkx3yjs2s8ZyIjBbDxExDbgZGFiYNRAYEhFT84vKbN24H4HZeiqUwhgD3Ab8BOgaER/nG5VZ4zXLnsVmxRQRyySdDTwKHOwkYOXGt4bMmkYPYCZQt+KnWclzIjBbT5J2AQ4ijRR1+voMEGKWBycCs/VQqE55M2m8hPeAK4Gr8o3KbN04EZitn58A70XE44Xpm4DtJe2bY0xm68SthszMKpyvCMzMKpwTgZlZhXMiMDOrcE4EZmYVzonAzKzCORGYmVU4JwIzswr3/wDvQGQVKb6w6QAAAABJRU5ErkJggg==",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt \n",
"\n",
"# Plot the data and the linear regression line\n",
"plt.scatter(X, y, color='blue')\n",
"plt.plot(X, y_pred, color='red')\n",
"plt.xlabel('X')\n",
"plt.ylabel('y')\n",
"plt.title('Linear Regression')\n",
"plt.show()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: src/MLC/notebooks/linear_regression_md.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Linear regression python multi-dimensional data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Linear Regression with two variables in one dimensional data\n",
"\n",
" \n",
" \n",
" $$ F(X)=X \\times W $$\n",
" $$ C=|| F(X) - Y ||_2^2 + \\lambda ||W||_2^2$$\n",
"\n",
"$X_{n \\times k}$\n",
"\n",
"$W_{k \\times p}$\n",
"\n",
"$Y_{n \\times p}$"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import random"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"n, k, p=100, 8, 3 \n",
"X=np.random.random([n,k])\n",
"W=np.random.random([k,p])\n",
"Y=np.random.random([n,p])\n",
"max_itr=1000\n",
"alpha=0.0001\n",
"Lambda=0.01"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Gradient is as follows:\n",
"$$ X^T 2 E + \\lambda 2 W$$"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# F(x)= w[0]*x + w[1]\n",
"def F(X, W):\n",
" return np.matmul(X,W)\n",
"\n",
"def cost(Y_est, Y, W, Lambda):\n",
" E=Y_est-Y\n",
" return E, np.linalg.norm(E,2)+ Lambda * np.linalg.norm(W,2)\n",
"\n",
"def gradient(E,X, W, Lambda):\n",
" return 2* np.matmul(X.T, E) + Lambda* 2* W"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"def fit(W, X, Y, alpha, Lambda, max_itr):\n",
" for i in range(max_itr):\n",
" \n",
" Y_est=F(X,W)\n",
" E, c= cost(Y_est, Y, W, Lambda)\n",
" Wg=gradient(E, X, W, Lambda)\n",
" W=W - alpha * Wg\n",
" if i%100==0:\n",
" print(c)\n",
" \n",
" return W"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To take into account for the biases, we concatenate X by a 1 column, and increase the number of rows in W by one"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"34.3004759224227\n",
"4.265835757989014\n",
"4.052505749060854\n",
"3.8807845759072968\n",
"3.7422281683979812\n",
"3.6303399157863434\n",
"3.5398708528835554\n",
"3.4665749938168915\n",
"3.4070257924246747\n",
"3.3584711183863862\n"
]
}
],
"source": [
"X=np.concatenate( (X, np.ones((n,1))), axis=1 ) \n",
"W=np.concatenate( (W, np.random.random((1,p)) ), axis=0 )\n",
"\n",
"W = fit(W, X, Y, alpha, Lambda, max_itr)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
================================================
FILE: src/MLC/notebooks/logistic_regression.ipynb
================================================
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Logistic Regression \n",
"\n",
"Logistic regression is a statistical method used for binary classification, which means it is used to predict the probability of an event occurring or not. It is a type of generalized linear model that is used when the dependent variable is binary or categorical.\n",
"\n",
"In logistic regression, the dependent variable is binary (i.e., it can take on one of two values, usually 0 or 1), and the independent variables can be either continuous or categorical. The goal of logistic regression is to find the relationship between the independent variables and the dependent variable by estimating the probability of the dependent variable being 1 given the values of the independent variables.\n",
"\n",
"The logistic regression model uses a logistic function (also known as the sigmoid function) to map the input values of the independent variables to a value between 0 and 1, which represents the probability of the dependent variable being 1. The logistic function is defined as:\n",
"\n",
"css\n",
"Copy code\n",
"p = 1 / (1 + e^(-z))\n",
"where p is the predicted probability of the dependent variable being 1, e is the base of the natural logarithm, and z is the linear combination of the independent variables.\n",
"\n",
"The logistic regression model estimates the values of the coefficients of the independent variables that maximize the likelihood of observing the data given the model. This is typically done using maximum likelihood estimation or gradient descent optimization.\n",
"\n",
"Once the model is trained, it can be used to make predictions on new data by inputting the values of the independent variables into the logistic function and obtaining the predicted probability of the dependent variable being 1. The model can then classify the new observation as 1 or 0 based on a threshold probability value that is chosen by the user.\n",
"\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Code \n",
" Here's an example implementation using gradient descent optimization:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class LogisticRegression:\n",
" \n",
" def __init__(self, learning_rate=0.01, n_iters=1000):\n",
" self.learning_rate = learning_rate\n",
" self.n_iters = n_iters\n",
" self.weights = None\n",
" self.bias = None\n",
" \n",
" def fit(self, X, y):\n",
" # initialize weights and bias to zeros\n",
" n_samples, n_features = X.shape\n",
" self.weights = np.zeros(n_features)\n",
" self.bias = 0\n",
" \n",
" # gradient descent optimization\n",
" for i in range(self.n_iters):\n",
" # calculate predicted probabilities and cost\n",
" z = np.dot(X, self.weights) + self.bias\n",
" y_pred = self._sigmoid(z)\n",
" cost = (-1 / n_samples) * np.sum(y * np.log(y_pred) + (1 - y) * np.log(1 - y_pred))\n",
" \n",
" # calculate gradients\n",
" dw = (1 / n_samples) * np.dot(X.T, (y_pred - y))\n",
" db = (1 / n_samples) * np.sum(y_pred - y)\n",
" \n",
" # update weights and bias\n",
" self.weights -= self.learning_rate * dw\n",
" self.bias -= self.learning_rate * db\n",
" \n",
" def predict(self, X):\n",
" # calculate predicted probabilities\n",
" z = np.dot(X, self.weights) + self.bias\n",
" y_pred = self._sigmoid(z)\n",
" # convert probabilities to binary predictions\n",
" return np.round(y_pred).astype(int)\n",
" \n",
" def _sigmoid(self, z):\n",
" return 1 / (1 + np.exp(-z))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1 1]\n"
]
}
],
"source": [
"# create sample dataset\n",
"X = np.array([[1, 2], [2, 3], [3, 4], [4, 5], [5, 6]])\n",
"y = np.array([0, 0, 1, 1, 1])\n",
"\n",
"# initialize logistic regression model\n",
"lr = LogisticRegression()\n",
"\n",
"# train model on sample dataset\n",
"lr.fit(X, y)\n",
"\n",
"# make predictions on new data\n",
"X_new = np.array([[6, 7], [7, 8]])\n",
"y_pred = lr.predict(X_new)\n",
"\n",
"print(y_pred) # [1, 1]\n",
"\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Improvements \n",
"here are some possible improvements you could make to the code:\n",
"\n",
"1. Add regularization: Regularization can help prevent overfitting and improve the generalization performance of the model. You could add L1 or L2 regularization to the cost function and adjust the regularization strength with a hyperparameter. Here's an example of how to add L2 regularization to the code:"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"2. Use a more sophisticated optimization algorithm: Gradient descent is a simple and effective optimization algorithm, but it may not be the most efficient or accurate for large or complex datasets. You could try using a more sophisticated algorithm, such as stochastic gradient descent (SGD), mini-batch SGD, or Adam, which can converge faster and find better optima. Here's an example of how to use mini-batch SGD:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class LogisticRegression:\n",
" \n",
" def __init__(self, learning_rate=0.01, n_iters=1000, regularization='l2', reg_strength=0.1, batch_size=32):\n",
" self.learning_rate = learning_rate\n",
" self.n_iters = n_iters\n",
" self.regularization = regularization\n",
" self.reg_strength = reg_strength\n",
" self.batch_size = batch_size\n",
" self.weights = None\n",
" self.bias = None\n",
" \n",
" def fit(self, X, y):\n",
" n_samples, n_features = X.shape\n",
" self.weights = np.zeros(n_features)\n",
" self.bias = 0\n",
" n_batches = n_samples // self.batch_size\n",
" for i in range(self.n_iters):\n",
" batch_indices = np.random.choice(n_samples, self.batch_size)\n",
" X_batch = X[batch_indices]\n",
" y_batch = y[batch_indices]\n",
" z = np.dot(X_batch, self.weights) + self.bias\n",
" y_pred = self._sigmoid(z)\n",
" cost = (-1 / self.batch_size) * np.sum(y_batch * np.log(y_pred) + (1 - y_batch) * np.log(1 - y_pred))\n",
" if self.regularization == 'l2':\n",
" reg_cost = (self.reg_strength / (2 * n_samples)) * np.sum(self.weights ** 2)\n",
" cost += reg_cost\n",
" elif self.regularization == 'l1':\n",
" reg_cost = (self.reg_strength / (2 * n_samples)) * np.sum(np.abs(self.weights))\n",
" cost += reg_cost\n",
" dw = (1 / self.batch_size) * np.dot(X_batch.T, (y_pred - y_batch))\n",
" db = (1 / self.batch_size) * np.sum(y_pred - y_batch)\n",
" if self.regularization == 'l2':\n",
" dw += (self.reg_strength / n_samples) * self.weights\n",
" elif self.regularization == 'l1':\n",
" dw += (self.reg_strength / n_samples) * np.sign(self.weights)\n",
" self.weights -= self.learning_rate * dw\n",
" self.bias -= self.learning_rate * db\n",
" \n",
" def predict(self, X):\n",
" z = np.dot(X, self.weights) + self.bias\n",
" y_pred = self._sigmoid(z)\n",
" return np.round(y_pred).astype(int)\n",
" \n",
" def _sigmoid(self, z):\n",
" return 1 / (1 + np.exp(-z))\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This implementation includes the following improvements:\n",
"\n",
"1. Regularization: You can choose between L1 or L2 regularization by setting the regularization parameter to either 'l1' or 'l2', and adjust the regularization strength with the reg_strength parameter.\n",
"\n",
"2. Mini-batch stochastic gradient descent: The model uses mini-batch SGD (instead of simple gradient descent) to update the weights and bias, which can converge faster and find better optima.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1 1]\n"
]
}
],
"source": [
"# create sample dataset\n",
"X = np.array([[1, 2], [2, 3], [3, 4], [4, 5], [5, 6]])\n",
"y = np.array([0, 0, 1, 1, 1])\n",
"\n",
"# initialize logistic regression model\n",
"lr = LogisticRegression(learning_rate=0.01, n_iters=1000, regularization='l2', reg_strength=0.1, batch_size=2)\n",
"\n",
"# train model on sample dataset\n",
"lr.fit(X, y)\n",
"\n",
"# make predictions on new data\n",
"X_new = np.array([[6, 7], [7, 8]])\n",
"y_pred = lr.predict(X_new)\n",
"\n",
"print(y_pred) # [1, 1]\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualize "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is difficult to visualize logistic regression since it is a high-dimensional problem. However, we can visualize the decision boundary of a logistic regression model for a two-dimensional dataset.\n",
"\n",
"Here's an example of how to visualize the decision boundary of the LogisticRegression class on a 2D dataset using the matplotlib library:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAW0AAAD8CAYAAAC8TPVwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAASPUlEQVR4nO3dbWyd9XnH8e91jh3jJHZiCE0CyZp0aukoK4G6dAWVbdBWRGXdeLMVqRWgSXnT9WEPQqVv0Dq1UqWpaidVoBTaUZUHMQJSVXUdoI21ldaUhLJSCLSBZSUQyAOBkAec2L72wocsD3bOMTnHt//H349kxT7ntvM7ivLLnf+5//cVmYkkqQy1qgNIklpnaUtSQSxtSSqIpS1JBbG0JakglrYkFaSl0o6Iv46IJyPiVxFxd0Sc0elgkqSTNS3tiDgX+CwwnJkXAHXgE50OJkk6WavLIz1Af0T0APOBFzsXSZI0lZ5mB2TmCxHxj8BvgUPAg5n54InHRcQ6YB1Af733fasHh9qdVW3S2ztCzzlL2DEyzoF9dXpr9aojSXPe7uef2Z2ZZzc7LpptY4+IIWAD8BfAq8C/APdl5vem+p73nLk07/rItdMKrJmzctk2hv7her689SCPPryYpf2DVUeS5rxvffZDmzNzuNlxrSyPfBj4n8zclZlHgPuBS083oCRp+lop7d8CfxAR8yMigCuBLZ2NJUmaTNPSzsyNwH3AY8ATje9Z3+FckqRJNH0jEiAzbwZu7nAWSVIT7oiUpIJY2pJUEEtbkgpiac8xK5dtY/GXruPLv9nPxocWVh1H0jS19EakusP5NwRb338N191aBwapRY8ba6TCWNpzxMpl2+i59Ho2bD1oWUsFc3lEkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSCWNqSVBBLW5IKYmlLUkEsbUkqiKUtSQVpWtoRcV5EPH7Mx76I+PwMZJMknaDpXf4y8xlgDUBE1IEXgAc6G0uSNJnpLo9cCTybmf/biTDqnIG1q3n6yGsOPpAKN937aX8CuLsTQdQZb06q+crWA2y8bRAI76UtFazl0o6IecDHgZumeH4dsA5g+fyBtoTT6Tl+Us0ils8fqjqSpNM0neWRtcBjmfnyZE9m5vrMHM7M4aG+/vak0+mLAMLClrrEdEr7WlwakaRKtVTaETEf+Ahwf2fjSJJOpaU17cw8CJzV4SySpCbcESlJBbG0JakglrYkFcTSlqSCWNqSVBBLW5IKYmlLUkEsbUkqiKUtSQWxtCWpIJa2JBXE0pakgljaklQQS7tLTUytWcOGbfWqo0hqo+nOiNQsN9lMSKfWSN3D0u4ixxX2w4PUoschvlKXcXmky0QERM3ClrqUpS1JBXF5RNKslgePMLrtVXLfCDG/l/qqxdQG+6qOVZlWB/sujoj7IuLpiNgSER/sdDBJGt9/mMM/f4HxHfvJA0cY33WQI5t3ML7nYNXRKtPq8sg3gB9l5ruBC4EtnYskSRNGt74CY3n8g+PJkaf3kJmTf1OXa7o8EhGDwOXA9QCZeRg43NlYkgT56huTPzEyOlHmPTGzgWaBVs603wHsAr4TEb+IiNsiYsGJB0XEuojYFBGb9o4cantQSXNQ7xSbwyKgNvcKG1or7R7gYuCWzLwIOAB84cSDMnN9Zg5n5vBQX3+bY0qai+q/M3hyOdegtnwhYWlPaTuwPTM3Nr6+j4kSl6SOqq8YpLZiYKK46xNn17UlC+h555lVR6tM0zXtzHwpIp6PiPMy8xngSuCpzkeTNNdFBL3vPItctZg8NEr09RB9c/t+Oq1ep/0Z4M6ImAc8B9zQuUiSdLzorRNTrW/PMS2VdmY+Dgx3NookqRm3sUtSQSxtSSqIpS1JBfGGUV1iYlLNNVx3Sw1YOFf3HUhdz9IunJNqpLnF0i7cwNrVPDO6j40PDzn4QJoDXNPuBhFAWNjSHGBpS1JBLG1JKoilLUkFsbQlqSCWtiQVxNKWpIJY2pJUEEtbkgpiaUtSQSxtSSqI9x6RCpfjyfjug4zvOQi9dXrOGSDm91YdSx3SUmlHxDbgdWAMGM1MR49Js0COJ0ce20HuPwxjCQGHn99Hz3uWUH/bwqrjqQOmc6b9x5m5u2NJJE3b2Iuvk68fhvGceCCBTEaf2k1tyQLCG6t3Hde0pYKNv3zg/wv7BPnaGzOcRjOh1dJO4MGI2BwR6yY7ICLWRcSmiNi0d+RQ+xJKmlLUT3EmXfOcrBu1+qd6WWZeDKwFPh0Rl594QGauz8zhzBwe6utva0hJk6udOwCTFXdPjRicN/OB1HEtlXZmvtj4dSfwAHBJJ0OpuZXLtvH76/+Qr579Pm781uKq46gitSXzqZ0zALWY+KgH9NbovXApEa5nd6Omb0RGxAKglpmvNz7/KPCljifTlI4O8b21DixyJuQcFhH0vusscuUg43vfgN4atbPm+wZkF2vl6pGlwAONf7V7gLsy80cdTaXmGiPGLGwBRH8v9X6vzZ4LmpZ2Zj4HXDgDWSRJTfj2siQVxNKWpIJY2pJUEEtbkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSCWNqSVBBLW5IKYmlLUkEs7cJM3Et7jYMPpDlqOtPYVaGVy7ax+EvX8ZWtB9h42yC16GFp/2DVsSTNMEu7AEcL+9mDbHzYSTXSXObySCGiMammFvWqo0iqkGfamhPy8Bjjuw9CTAzDjV7/8VOZWi7tiKgDm4AXMvPqzkWS2mv0hX2M/foVCCAB9lD/vSX0LFtYcTJp+qazPPI5YEungkidkIeOTBT2eMJYTvw6noxt2U2OjFYdT5q2lko7IlYAHwNu62wcqb3GXj4AmZM+N77r4AynkU5fq2faXwduBManOiAi1kXEpojYtHfkUDuySadvPBtLIifIJMcnL3NpNmta2hFxNbAzMzef6rjMXJ+Zw5k5PNTX37aA0umonT0fanHyExHUl8yf+UDSaWrlTPsy4OMRsQ24B7giIr7X0VRSm9QG+qidO3B8cdeC+tsXEfN7qwsmvUVNrx7JzJuAmwAi4o+Av8vMT3Y2ltQ+ve86i/GlCybWt4H6soXUBvsqTiW9NV6nrTmhtugMaovOqDqGdNqmVdqZ+QjwSEeSSJKachu7JBXE0pakgljaklQQS1uSCmJpS1JBLG1JKojXac9yEzMhr+G6W2rAwkl3ZEuaOyztWerEmZAQjhmTZGnPVgNrV/PM6D42PjzkEF9JR7mmPZs15kJa2JLeZGlLUkEsbUkqiKUtSQWxtCWpIJa2JBXE0pakgljaklQQS1sTMul/aRcLt20nRkerTiNpCk13REbEGcCPgb7G8fdl5s2dDqaZ07dnL+9efxd9r+wlazXI5Lk//xP2DL+36miSTtDKNvYR4IrM3B8RvcBPI+JfM/NnHc6mmZDJ+d+8g3mvvEot8+jDv3vP9zm0/G0cPHdZheEknajp8khO2N/4srfxkaf4FhVk4bbt9Ow/cFxhA8ToKMt+8vOKUkmaSktr2hFRj4jHgZ3AQ5m5cZJj1kXEpojYtHfkUJtjqlN69x9o3OPkeLVM5r22r4JEkk6lpdLOzLHMXAOsAC6JiAsmOWZ9Zg5n5vBQX3+bY6pTXl+1gtro2EmPj/X2svf8d1WQSNKpTOvqkcx8FXgEuKoTYTTzRgcW8sIVlzI2r/foY2O9PRweWsSuD6ypLpikSbVy9cjZwJHMfDUi+oEPA1/teDLNmO0fu5IDb1/Bsv/8GT2H3mDPhefz0ocuYXzevKqjSTpBK1ePLAfuiIg6E2fm92bmDzobSzNt7wXnsfeC86qOIamJpqWdmb8ELpqBLOKEMWPfcviBpOM5bmwWOTrE99Y6MOiYMUknsbRniZXLttFz6fVs2HrQspY0Je89IkkFsbQlqSCWtiQVxNKWpIJY2pJUEEtbkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSCWNqSVBBLW5IKYmnPEgNrV/P0kdfY+NDCqqNImsW8n/Y0ZCbjY+PU6jUioi0/87hJNbcNAuG9tCVNqZXBviuB7wLLgHFgfWZ+o9PBZpPMZMeTL7PzN3sYHxun94weVly4nKGVi0/r5x4t7GcPsvHhRSyfP9SewJK6Vitn2qPA32bmYxExAGyOiIcy86kOZ5s1XvjlS+x6dg85lgAcOTTKtke3U59XZ3DpwFv+uQNrVxPz+oBD1KLeprSSulnTNe3M3JGZjzU+fx3YApzb6WCzxfjYOLuPKew35Viy48mdFaWSNFdN643IiFjFxGT2jZM8ty4iNkXEpr0jh9oUr3qjI6PkFM+N7B+Z0SyS1HJpR8RCYAPw+czcd+Lzmbk+M4czc3ior7+dGSvVe0bvlG869i/untcpqQwtlXZE9DJR2Hdm5v2djTS7RC1Yfv7bqNWPL+6oB+dcsLSiVJLmqlauHgngdmBLZn6t85Fmn6XnnU29r87LT+3iyBtH6F/cz7nvXcaCM+dXHU3SHNPK1SOXAZ8CnoiIxxuPfTEzf9ixVLPQklVnsmTVmVXHkDTHNS3tzPwp0J6dJJKk0+I2dkkqiKUtSQWxtCWpIJa2JBXE0pakgljaklQQS1uSCmJpS1JBLG1JKojjxipy/g3B1vev4cZ/GgUGqLnnVFILLO0ZNtlMSMeMSWqVpT3DBtau5pnRfWx8eIha9DjEV9K0uKZdhQicui7prbC0JakglrYkFcTSlqSCWNqSVBBLW5IK0rS0I+LbEbEzIn41E4EkSVNr5Uz7n4GrOpxDktSCpqWdmT8GXpmBLJKkJlzTlqSCtK20I2JdRGyKiE17Rw6168dKko7RttLOzPWZOZyZw0N9/e36sZKkY7g8IkkFaeWSv7uB/wLOi4jtEfGXnY8lSZpM01uzZua1MxFkLli5bBv1D17HhmcPVh1FUqG8n/YMmZhUcw3X3VpnYlJNvepIkgpkaXeYk2oktZOlPQMiAqLmpBpJp82rRySpIJa2JBXE0pakgljaklQQS1uSCmJpS1JBLG1JKoilLUkFsbQlqSCWtiQVxNKWpIJY2pJUEEtbkgpiaUtSQSxtSSqIpS1JBWmptCPiqoh4JiK2RsQXOh2qmwysXc3TR15j40MLq44iqQs0nVwTEXXgm8BHgO3AoxHx/cx8qtPhSjbZmDGn1kg6Xa2MG7sE2JqZzwFExD3AnwKW9hSOH+K7yJmQktqmldI+F3j+mK+3Ax848aCIWAesa3w5subeb/zq9OPNSkuA3ac84t6ZCdIhzV9f2Xx9Zevm13deKwe1UtoxyWN50gOZ64H1ABGxKTOHWwlQmm5+beDrK52vr1wRsamV41p5I3I7sPKYr1cAL76VUJKk09NKaT8KvDMiVkfEPOATwPc7G0uSNJmmyyOZORoRfwX8G1AHvp2ZTzb5tvXtCDdLdfNrA19f6Xx95WrptUXmScvTkqRZyh2RklQQS1uSCtLW0u7m7e4R8e2I2BkRXXn9eUSsjIj/iIgtEfFkRHyu6kztFBFnRMTPI+K/G6/v76vO1G4RUY+IX0TED6rO0m4RsS0inoiIx1u9NK4kEbE4Iu6LiKcbfwc/OOWx7VrTbmx3/zXHbHcHru2W7e4RcTmwH/huZl5QdZ52i4jlwPLMfCwiBoDNwJ910Z9fAAsyc39E9AI/BT6XmT+rOFrbRMTfAMPAYGZeXXWedoqIbcBwZnblxpqIuAP4SWbe1rhKb35mvjrZse080z663T0zDwNvbnfvCpn5Y+CVqnN0SmbuyMzHGp+/DmxhYjdsV8gJ+xtf9jY+uuZd+IhYAXwMuK3qLJqeiBgELgduB8jMw1MVNrS3tCfb7t41f+nnkohYBVwEbKw4Sls1lg8eB3YCD2VmN72+rwM3AuMV5+iUBB6MiM2NW2Z0k3cAu4DvNJa3bouIBVMd3M7Sbmm7u2a3iFgIbAA+n5n7qs7TTpk5lplrmNjVe0lEdMUyV0RcDezMzM1VZ+mgyzLzYmAt8OnGcmW36AEuBm7JzIuAA8CU7wm2s7Td7l64xlrvBuDOzLy/6jyd0viv5yPAVdUmaZvLgI831n3vAa6IiO9VG6m9MvPFxq87gQeYWI7tFtuB7cf8z+8+Jkp8Uu0sbbe7F6zxRt3twJbM/FrVedotIs6OiMWNz/uBDwNPVxqqTTLzpsxckZmrmPh79++Z+cmKY7VNRCxovDlOY9ngo0DXXMWVmS8Bz0fEm3f5u5JT3Pq6lbv8tfobv5Xt7sWIiLuBPwKWRMR24ObMvL3aVG11GfAp4InGui/AFzPzh9VFaqvlwB2Nq5xqwL2Z2XWXxnWppcADE+cV9AB3ZeaPqo3Udp8B7myc8D4H3DDVgW5jl6SCuCNSkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSC/B/dAeYbl24tJgAAAABJRU5ErkJggg==",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"# create 2D dataset\n",
"X = np.array([[1, 2], [2, 3], [3, 4], [4, 5], [5, 6]])\n",
"y = np.array([0, 0, 1, 1, 1])\n",
"\n",
"# initialize logistic regression model\n",
"lr = LogisticRegression(learning_rate=0.01, n_iters=1000, regularization='l2', reg_strength=0.1, batch_size=2)\n",
"\n",
"# train model on dataset\n",
"lr.fit(X, y)\n",
"\n",
"# plot decision boundary\n",
"x1 = np.linspace(0, 6, 100)\n",
"x2 = np.linspace(0, 8, 100)\n",
"xx, yy = np.meshgrid(x1, x2)\n",
"Z = lr.predict(np.c_[xx.ravel(), yy.ravel()])\n",
"Z = Z.reshape(xx.shape)\n",
"plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral, alpha=0.8)\n",
"\n",
"# plot data points\n",
"plt.scatter(X[:,0], X[:,1], c=y, cmap=plt.cm.Spectral)\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: src/MLC/notebooks/logistic_regression_md.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## logistic regression multi-dimensional data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" logistic regression multi-dimensional data\n",
" \n",
" \n",
" $$ F(X)=X \\times W $$\n",
" $$ H(x)= \\frac{1}{1+ e ^{-F(x)}} $$\n",
" $$ C= -\\frac{1}{n} \\sum_{i,j} (Y \\odot log(H(x)) + (1-Y) \\odot log(1-H(x)) ) $$\n",
"\n",
"$X_{n \\times k}$\n",
"\n",
"$W_{k \\times p}$\n",
"\n",
"$Y_{n \\times p}$"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import random"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"n, k, p=100, 8, 3 "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"X=np.random.random([n,k])\n",
"W=np.random.random([k,p])\n",
"\n",
"y=np.random.randint(p, size=(1,n))\n",
"Y=np.zeros((n,p))\n",
"Y[np.arange(n), y]=1\n",
"\n",
"max_itr=5000\n",
"alpha=0.01\n",
"Lambda=0.01"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Gradient is as follows:\n",
"$$ X^T (H(x)-Y) + \\lambda 2 W$$"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# F(x)= w[0]*x + w[1]\n",
"def F(X, W):\n",
" return np.matmul(X,W)\n",
"\n",
"def H(F):\n",
" return 1/(1+np.exp(-F))\n",
"\n",
"def cost(Y_est, Y):\n",
" E= - (1/n) * (np.sum(Y*np.log(Y_est) + (1-Y)*np.log(1-Y_est))) + np.linalg.norm(W,2)\n",
" return E, np.sum(np.argmax(Y_est,1)==y)/n\n",
"\n",
"def gradient(Y_est, Y, X):\n",
" return (1/n) * np.matmul(X.T, (Y_est - Y) ) + Lambda* 2* W"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def fit(W, X, Y, alpha, max_itr):\n",
" for i in range(max_itr):\n",
" \n",
" F_x=F(X,W)\n",
" Y_est=H(F_x)\n",
" E, c= cost(Y_est, Y)\n",
" Wg=gradient(Y_est, Y, X)\n",
" W=W - alpha * Wg\n",
" if i%1000==0:\n",
" print(E, c)\n",
" \n",
" return W, Y_est"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To take into account for the biases, we concatenate X by a 1 column, and increase the number of rows in W by one"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"9.368653735228364 0.31\n",
"4.994251188297815 0.43\n",
"4.951873226767272 0.48\n",
"4.922370610237865 0.47\n",
"4.901694423284286 0.48\n"
]
}
],
"source": [
"X=np.concatenate( (X, np.ones((n,1))), axis=1 ) \n",
"W=np.concatenate( (W, np.random.random((1,p)) ), axis=0 )\n",
"\n",
"W, Y_est = fit(W, X, Y, alpha, max_itr)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
================================================
FILE: src/MLC/notebooks/numpy_practice.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Numpy Practice\n",
"- Author: Alireza Dirafzoon\n",
"- Contributions are welcome :) "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1, 2, 3])"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"### array()\n",
"a = [1, 2, 3]\n",
"x = np.array(a) \n",
"x = np.asarray(a)\n",
"x"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[1, 2, 3]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x.tolist()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1., 2., 3.], dtype=float32)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x.astype(np.float32)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([3, 2, 1, 0])"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"### arange()\n",
"np.arange(3) \n",
"np.arange(0,7,2) \n",
"np.arange(3, -1, -1)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0., 0., 0.],\n",
" [0., 0., 0.],\n",
" [0., 0., 0.]])"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"### zeros, ones, eye, linspace\n",
"np.zeros(3) \n",
"np.zeros((3,3)) "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1., 1., 1.],\n",
" [1., 1., 1.],\n",
" [1., 1., 1.]])"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.ones(3)\n",
"np.ones((3,3))"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1., 0., 0.],\n",
" [0., 1., 0.],\n",
" [0., 0., 1.]])"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.eye(3)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0. , 3.5, 7. ])"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.linspace(0,10,3) # 3 points, from 0 to 10, inclusive \n",
"np.linspace(0,7,3) "
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.7771434 , 0.08427174, 0.84780602],\n",
" [0.6069425 , 0.72381233, 0.54255502]])"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"### np.random\n",
"\n",
"# random.rand(): uniform distr over [0, 1)\n",
"np.random.rand() \n",
"np.random.rand(2)\n",
"np.random.rand(2,3)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 2.2473354 , -1.27775236, -0.70635289],\n",
" [-1.56768889, 0.33955847, -0.16860601]])"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# random.randn(): normal distr.\n",
"np.random.randn(2,3)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[2, 3],\n",
" [3, 1]])"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# random.randint: int in [low,high) / [0, high)\n",
"np.random.randint(1,4)\n",
"np.random.randint(1,4, (2,2))"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"3"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.random.randint(4)"
]
},
{
"cell_type": "code",
"execution_count": 124,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1, 2, 3],\n",
" [4, 5, 6]])"
]
},
"execution_count": 124,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"## array methods \n",
"\n",
"### reshape\n",
"a = np.arange(1,7)\n",
"a = a.reshape(2,3)\n",
"a"
]
},
{
"cell_type": "code",
"execution_count": 125,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([4, 5, 6])"
]
},
"execution_count": 125,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"### max, min, atgmax, argmin \n",
"a.max(axis = 0)"
]
},
{
"cell_type": "code",
"execution_count": 126,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1, 1, 1])"
]
},
"execution_count": 126,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a.argmax(axis=0)"
]
},
{
"cell_type": "code",
"execution_count": 127,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([3, 6])"
]
},
"execution_count": 127,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a.max(axis = 1)"
]
},
{
"cell_type": "code",
"execution_count": 128,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"6"
]
},
"execution_count": 128,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a.max()"
]
},
{
"cell_type": "code",
"execution_count": 129,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"5"
]
},
"execution_count": 129,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a.argmax()"
]
},
{
"cell_type": "code",
"execution_count": 130,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(2, 3)"
]
},
"execution_count": 130,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"### shape and dtype \n",
"a.shape"
]
},
{
"cell_type": "code",
"execution_count": 131,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dtype('int64')"
]
},
"execution_count": 131,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a.dtype"
]
},
{
"cell_type": "code",
"execution_count": 132,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"48"
]
},
"execution_count": 132,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a.nbytes"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0, 1, 2, 3],\n",
" [ 4, 5, 6, 7],\n",
" [ 8, 9, 10, 11]])"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"### 2D array/matrix \n",
"m = np.arange(12).reshape(3,4)\n",
"m"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([4., 5., 6., 7.])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m.mean(axis=0)"
]
},
{
"cell_type": "code",
"execution_count": 139,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1.11803399, 1.11803399, 1.11803399])"
]
},
"execution_count": 139,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m.std(axis=1)"
]
},
{
"cell_type": "code",
"execution_count": 140,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0, 4, 8],\n",
" [ 1, 5, 9],\n",
" [ 2, 6, 10],\n",
" [ 3, 7, 11]])"
]
},
"execution_count": 140,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m.T # or m.transpose"
]
},
{
"cell_type": "code",
"execution_count": 142,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]])"
]
},
"execution_count": 142,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m.reshape((-1,12))"
]
},
{
"cell_type": "code",
"execution_count": 141,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])"
]
},
"execution_count": 141,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m.reshape(-1)"
]
},
{
"cell_type": "code",
"execution_count": 145,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])"
]
},
"execution_count": 145,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m.ravel()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0, 1, 2, 3],\n",
" [ 4, 5, 6, 7],\n",
" [ 8, 9, 10, 11]])"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"## indexing and selection"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"6\n",
"6\n",
"[4 5 6 7]\n"
]
}
],
"source": [
"print(m[1][2])\n",
"print(m[1,2])\n",
"print(m[1,:])"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0., 0., 0., 0., 0., 5., 5., 5., 0., 0.])"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"## boradcasting\n",
"a = np.zeros(10)\n",
"a[5:8] = 5 # note we can't do this with list \n",
"a"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([2., 2., 2.])"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"suba = a[:3]\n",
"suba[:] = 2\n",
"suba"
]
},
{
"cell_type": "code",
"execution_count": 135,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([2., 2., 2., 0., 0., 5., 5., 5., 0., 0.])"
]
},
"execution_count": 135,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a # note that suba is not a copy, just points to a slice of a"
]
},
{
"cell_type": "code",
"execution_count": 136,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([2., 2., 2.])"
]
},
"execution_count": 136,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"suba = np.copy(a[:3])\n",
"suba"
]
},
{
"cell_type": "code",
"execution_count": 101,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0., 0., 0., 0.],\n",
" [2., 2., 2., 2.],\n",
" [0., 0., 0., 0.],\n",
" [0., 0., 0., 0.]])"
]
},
"execution_count": 101,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m = np.zeros((4,4))\n",
"m[1] = 2\n",
"m"
]
},
{
"cell_type": "code",
"execution_count": 104,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[2., 2., 2., 2.],\n",
" [0., 0., 0., 0.]])"
]
},
"execution_count": 104,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# fancy indexing\n",
"m[[1,3]]"
]
},
{
"cell_type": "code",
"execution_count": 153,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([False, False, False, True])"
]
},
"execution_count": 153,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"## selection \n",
"a = np.arange(4)\n",
"a > 2 # note we can't do this with list "
]
},
{
"cell_type": "code",
"execution_count": 156,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1"
]
},
"execution_count": 156,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"(a == 2).astype(np.int16).sum()"
]
},
{
"cell_type": "code",
"execution_count": 154,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(array([1, 2, 3]),)"
]
},
"execution_count": 154,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a.nonzero()"
]
},
{
"cell_type": "code",
"execution_count": 108,
"metadata": {},
"outputs": [],
"source": [
"## Operations"
]
},
{
"cell_type": "code",
"execution_count": 159,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0. , 0.25, 0.4 ])"
]
},
"execution_count": 159,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a, b = np.arange(0,3), np.arange(3,6)\n",
"a + b\n",
"a - b \n",
"a * b # element-wise \n",
"a/b # element-wise "
]
},
{
"cell_type": "code",
"execution_count": 114,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 0, 4, 10])"
]
},
"execution_count": 114,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.multiply(a,b) # element-wise "
]
},
{
"cell_type": "code",
"execution_count": 163,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"14"
]
},
"execution_count": 163,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# dot product of arrays\n",
"np.dot(a,b)"
]
},
{
"cell_type": "code",
"execution_count": 164,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([-3, 6, -3])"
]
},
"execution_count": 164,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# cross product \n",
"np.cross(a,b)"
]
},
{
"cell_type": "code",
"execution_count": 162,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"14"
]
},
"execution_count": 162,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# matrix multiplication\n",
"np.matmul(a,b.T)"
]
},
{
"cell_type": "code",
"execution_count": 177,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[0 1 2]\n",
" [3 4 5]\n",
" [6 7 8]] [0 1 0]\n"
]
},
{
"data": {
"text/plain": [
"array([3, 4, 5])"
]
},
"execution_count": 177,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a = np.arange(9).reshape((3,3)) #2D\n",
"b = np.array([0,1,0]) # 1D\n",
"print(a,b) \n",
"np.matmul(a,b) # 2D * 1D -> broadcasts the 1D array, treating it as a col "
]
},
{
"cell_type": "code",
"execution_count": 120,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0, 1, 2])"
]
},
"execution_count": 120,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.power(a,2) # element-wise \n",
"np.power(a,b) # element-wise \n",
"np.mod(a,b)"
]
},
{
"cell_type": "code",
"execution_count": 178,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/alirezadirafzoon/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:4: RuntimeWarning: divide by zero encountered in log\n",
" after removing the cwd from sys.path.\n"
]
},
{
"data": {
"text/plain": [
"array([[ -inf, 0. , 0.69314718],\n",
" [1.09861229, 1.38629436, 1.60943791],\n",
" [1.79175947, 1.94591015, 2.07944154]])"
]
},
"execution_count": 178,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.sqrt(a)\n",
"np.exp(a)\n",
"np.sin(a)\n",
"np.log(a)"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [],
"source": [
"## Kmeans "
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [
{
"ename": "TypeError",
"evalue": "Population must be a sequence or set. For dicts, use list(d).",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-67-a1d78b800ed7>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0mx2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconcatenate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mx1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mx2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mmu\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclusters\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfind_centers\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m<ipython-input-52-11ad8b933b9e>\u001b[0m in \u001b[0;36mfind_centers\u001b[0;34m(X, K)\u001b[0m\n\u001b[1;32m 24\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfind_centers\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 25\u001b[0m \u001b[0;31m# Initialize to K random centers\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m \u001b[0moldmu\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msample\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 27\u001b[0m \u001b[0mmu\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msample\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 28\u001b[0m \u001b[0;32mwhile\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhas_converged\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmu\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moldmu\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/opt/anaconda3/lib/python3.7/random.py\u001b[0m in \u001b[0;36msample\u001b[0;34m(self, population, k)\u001b[0m\n\u001b[1;32m 315\u001b[0m \u001b[0mpopulation\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpopulation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 316\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpopulation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_Sequence\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 317\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Population must be a sequence or set. For dicts, use list(d).\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 318\u001b[0m \u001b[0mrandbelow\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_randbelow\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 319\u001b[0m \u001b[0mn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpopulation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mTypeError\u001b[0m: Population must be a sequence or set. For dicts, use list(d)."
]
}
],
"source": [
"x1 = np.add(np.random.randn(10,2), 5)\n",
"x2 = np.add(np.random.randn(10,2), -5)\n",
"X = np.concatenate([x1,x2], axis=0)\n",
"mu, clusters = kmeans(X,2)"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[[4.8649386655349955, 3.9952475402226817],\n",
" [5.498489206113001, 5.069951322563478],\n",
" [4.929449898684354, 5.719151512307626],\n",
" [4.595440437145644, 4.810271477510138],\n",
" [5.285073437207049, 5.922053828848186],\n",
" [3.2378112256065865, 4.595935658934975],\n",
" [3.8231073755832887, 6.144586325794659],\n",
" [4.1009988278675245, 6.559105478655928],\n",
" [3.9976386132206, 4.424471531025596],\n",
" [4.691876028371731, 5.345908717367563],\n",
" [-5.720985281350966, -5.68922383985498],\n",
" [-5.4201288230000815, -4.431411907717413],\n",
" [-3.6983426126902725, -4.636565625778152],\n",
" [-5.342010805119905, -6.095419133835849],\n",
" [-4.2666049359220235, -3.284073438471302],\n",
" [-6.469221214094414, -7.369070651238069],\n",
" [-3.284553291631532, -5.672466183383029],\n",
" [-3.4845642662555996, -5.40312458836927],\n",
" [-5.6863731385517005, -5.30056130289524],\n",
" [-5.194321373602274, -5.935463756358125]]"
]
},
"execution_count": 65,
"metadata": {},
"output_type": "execute_result"
}
],
"source": []
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 4.61084009, 3.81951528, 4.80554869, 4.89529018, 4.32093641,\n",
" 4.23753206, 2.72005748, 6.7060486 , 4.02801539, 4.04573508,\n",
" -6.07068165, -7.1437371 , -6.49431954, -4.90412879, -5.25460504,\n",
" -3.86646858, -6.98290866, -4.82434449, -6.14940609, -6.55090156],\n",
" [ 4.79319455, 4.07846865, 6.5072268 , 4.9865201 , 4.66317278,\n",
" 2.8773762 , 3.67165213, 4.9436099 , 3.93374775, 1.65634458,\n",
" -5.16876443, -5.99996567, -4.70701913, -5.61868297, -5.08846978,\n",
" -6.31017567, -3.46475003, -4.53046633, -4.35615641, -7.20485829]])"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import random\n",
"def dist():\n",
" pass \n",
"def assign_clusters(X, mu):\n",
" \n",
"def kmeans(X,k):\n",
" mu = random.sample(X,k)\n",
" it = 1 \n",
" max_it = 100\n",
" while it < max_it: \n",
" # assign clusters to centers \n",
" clusters = assign_clusters(X, mu)\n",
" # calculate new centers \n",
" mu = calculate_centers(mu, clusters)\n",
" return mu, clusters"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"mu = np.random.rand(5,2)\n",
"x = np.random.rand(2)"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[array([-0.722217 , -0.95781472]),\n",
" array([-0.43986246, -0.58463802]),\n",
" array([-0.16399214, -0.27117604]),\n",
" array([-0.88255848, -0.98718324]),\n",
" array([-0.53056903, -0.60690576])]"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[-mu[i[0]] for i in enumerate(mu)]"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"ename": "TypeError",
"evalue": "'int' object is not subscriptable",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-48-7924ee5c2f3f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmu_i\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmu\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mmu_i\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m<ipython-input-48-7924ee5c2f3f>\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmu_i\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmu\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mmu_i\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m: 'int' object is not subscriptable"
]
}
],
"source": [
"for i, mu_i in enumerate(mu):\n",
" print(min(i, np.linalg.norm(x - mu_i), key=lambda x:x[1]))"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0 [1, 2, 3]\n",
"1 [4, 5, 6]\n"
]
}
],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: src/MLC/notebooks/perceptron.ipynb
================================================
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The perceptron algorithm is a type of linear classification algorithm used to classify data into two categories. It is a simple algorithm that learns from the mistakes made during the classification process and adjusts the weights of the input features to improve the accuracy of the classification. \n",
"\n",
"```python \n",
"y_pred = sign(w0 + w1*x1 + w2*x2 + ... + wn*xn)\n",
"wi = wi + learning_rate * (target - y_pred) * xi\n",
"```\n",
"\n",
"Here is an implementation of the perceptron algorithm in Python:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class Perceptron:\n",
" def __init__(self, lr=0.01, n_iter=100):\n",
" self.lr = lr\n",
" self.n_iter = n_iter\n",
"\n",
" def fit(self, X, y):\n",
" self.weights = np.zeros(1 + X.shape[1])\n",
" self.errors = []\n",
"\n",
" for _ in range(self.n_iter):\n",
" errors = 0\n",
" for xi, target in zip(X, y):\n",
" update = self.lr * (target - self.predict(xi))\n",
" self.weights[1:] += update * xi\n",
" self.weights[0] += update\n",
" errors += int(update != 0.0)\n",
" self.errors.append(errors)\n",
" return self\n",
"\n",
" def net_input(self, X):\n",
" return np.dot(X, self.weights[1:]) + self.weights[0]\n",
"\n",
" def predict(self, X):\n",
" return np.where(self.net_input(X) >= 0.0, 1, -1)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The Perceptron class has the following methods:\n",
"\n",
"__init__(self, lr=0.01, n_iter=100): Initializes the perceptron with a learning rate (lr) and number of iterations (n_iter) to perform during training.\n",
"\n",
"fit(self, X, y): Trains the perceptron on the input data X and target labels y. The method initializes the weights to zero and iterates through the data n_iter times, adjusting the weights after each misclassification. The method returns the trained perceptron.\n",
"\n",
"net_input(self, X): Computes the weighted sum of inputs and bias.\n",
"\n",
"predict(self, X): Predicts the class label for a given input X based on the current weights.\n",
"\n",
"To use the perceptron algorithm, you can create an instance of the Perceptron class, and then call the fit method with your input data X and target labels y. Here is an example usage:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([-1, 1])"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X = np.array([[2.0, 1.0], [3.0, 4.0], [4.0, 2.0], [3.0, 1.0]])\n",
"y = np.array([-1, 1, 1, -1])\n",
"perceptron = Perceptron()\n",
"perceptron.fit(X, y)\n",
"\n",
"new_X = np.array([[5.0, 2.0], [1.0, 3.0]])\n",
"perceptron.predict(new_X)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: src/MLC/notebooks/softmax.ipynb
================================================
================================================
FILE: src/MLC/notebooks/svm.ipynb
================================================
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Support Vector Machines (SVMs)\n",
"\n",
"Support Vector Machines (SVMs) are a type of machine learning algorithm used for classification and regression analysis. In particular, linear SVMs are used for binary classification problems where the goal is to separate two classes by a hyperplane.\n",
"\n",
"The hyperplane is a line that divides the feature space into two regions. The SVM algorithm tries to find the hyperplane that maximizes the margin, which is the distance between the hyperplane and the closest points from each class. The points closest to the hyperplane are called support vectors and play a crucial role in the algorithm's optimization process.\n",
"\n",
"In linear SVMs, the hyperplane is defined by a linear function of the input features. The algorithm tries to find the optimal values of the coefficients of this function, called weights, that maximize the margin. This optimization problem can be formulated as a quadratic programming problem, which can be efficiently solved using standard optimization techniques.\n",
"\n",
"In addition to finding the optimal hyperplane, SVMs can also handle non-linearly separable data by using a kernel trick. This technique maps the input features into a higher-dimensional space, where they might become linearly separable. The SVM algorithm then finds the optimal hyperplane in this transformed feature space, which corresponds to a non-linear decision boundary in the original feature space.\n",
"\n",
"Linear SVMs have been widely used in many applications, including text classification, image classification, and bioinformatics. They have the advantage of being computationally efficient and easy to interpret. However, they may not perform well in highly non-linearly separable datasets, where non-linear SVMs may be a better choice."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Code "
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class SVM:\n",
" def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iters=1000):\n",
" self.lr = learning_rate\n",
" self.lambda_param = lambda_param\n",
" self.n_iters = n_iters\n",
" self.w = None\n",
" self.b = None\n",
"\n",
" def fit(self, X, y):\n",
" n_samples, n_features = X.shape\n",
" y_ = np.where(y <= 0, -1, 1)\n",
" self.w = np.zeros(n_features)\n",
" self.b = 0\n",
"\n",
" # Gradient descent\n",
" for _ in range(self.n_iters):\n",
" for idx, x_i in enumerate(X):\n",
" condition = y_[idx] * (np.dot(x_i, self.w) - self.b) >= 1\n",
" if condition:\n",
" self.w -= self.lr * (2 * self.lambda_param * self.w)\n",
" else:\n",
" self.w -= self.lr * (2 * self.lambda_param * self.w - np.dot(x_i, y_[idx]))\n",
" self.b -= self.lr * y_[idx]\n",
"\n",
" def predict(self, X):\n",
" linear_output = np.dot(X, self.w) - self.b\n",
" return np.sign(linear_output)\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy: 1.0\n"
]
}
],
"source": [
"# Example usage\n",
"from sklearn import datasets\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"X, y = datasets.make_blobs(n_samples=100, centers=2, random_state=42)\n",
"y = np.where(y == 0, -1, 1)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
"\n",
"svm = SVM()\n",
"svm.fit(X_train, y_train)\n",
"y_pred = svm.predict(X_test)\n",
"\n",
"\n",
"# Evaluate model\n",
"accuracy = accuracy_score(y_test, y_pred)\n",
"print(\"Accuracy:\", accuracy)"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy: 0.5\n"
]
}
],
"source": [
"# Generate data\n",
"X, y = make_classification(n_features=5, n_samples=100, n_informative=5, n_redundant=0, n_classes=2, random_state=1)\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)\n",
"\n",
"# Initialize SVM model\n",
"svm = SVM()\n",
"\n",
"# Train model\n",
"svm.fit(X_train, y_train)\n",
"\n",
"# Make predictions\n",
"y_pred = svm.predict(X_test)\n",
"\n",
"# Evaluate model\n",
"accuracy = accuracy_score(y_test, y_pred)\n",
"print(\"Accuracy:\", accuracy)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: src/MLC/notebooks/ww_classifier.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## CS 224N Lecture 3: Word Window Classification\n",
"\n",
"### Pytorch Exploration\n",
"\n",
"### Author: Matthew Lamm"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"import pprint\n",
"import torch\n",
"import torch.nn as nn\n",
"pp = pprint.PrettyPrinter()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Our Data\n",
"\n",
"The task at hand is to assign a label of 1 to words in a sentence that correspond with a LOCATION, and a label of 0 to everything else. \n",
"\n",
"In this simplified example, we only ever see spans of length 1."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"train_sents = [s.lower().split() for s in [\"we 'll always have Paris\",\n",
" \"I live
gitextract_pa4rkxdb/
├── LICENSE
├── README.md
└── src/
├── MLC/
│ ├── ml-coding.md
│ └── notebooks/
│ ├── .test.ipynb
│ ├── convolution.ipynb
│ ├── decision_tree.ipynb
│ ├── feedforward.ipynb
│ ├── k_means.ipynb
│ ├── k_means_2.ipynb
│ ├── k_nearest_neighbors.ipynb
│ ├── knn.ipynb
│ ├── linear_regression.ipynb
│ ├── linear_regression_md.ipynb
│ ├── logistic_regression.ipynb
│ ├── logistic_regression_md.ipynb
│ ├── numpy_practice.ipynb
│ ├── perceptron.ipynb
│ ├── softmax.ipynb
│ ├── svm.ipynb
│ └── ww_classifier.ipynb
├── MLSD/
│ ├── ml-companies.md
│ ├── ml-system-design.md
│ ├── mlsd-ads-ranking.md
│ ├── mlsd-av.md
│ ├── mlsd-event-recom.md
│ ├── mlsd-feature-eng.md
│ ├── mlsd-game-recom.md
│ ├── mlsd-harmful-content.md
│ ├── mlsd-image-search.md
│ ├── mlsd-metrics.md
│ ├── mlsd-mm-video-search.md
│ ├── mlsd-modeling-popular-archs.md
│ ├── mlsd-newsfeed.md
│ ├── mlsd-prediction.md
│ ├── mlsd-preprocessing.md
│ ├── mlsd-pymk.md
│ ├── mlsd-search.md
│ ├── mlsd-template.md
│ ├── mlsd-typeahead.md
│ ├── mlsd-video-recom.md
│ └── mlsd_obj_detection.md
├── behavior.md
├── lc-coding.md
├── ml-depth.md
└── ml-fundamental.md
Condensed preview — 45 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (389K chars).
[
{
"path": "LICENSE",
"chars": 1074,
"preview": "MIT License\n\nCopyright (c) 2021 Alireza Dirafzoon\n\nPermission is hereby granted, free of charge, to any person obtaining"
},
{
"path": "README.md",
"chars": 4281,
"preview": "[](LICENSE) [![Code style: black](https://img.shields."
},
{
"path": "src/MLC/ml-coding.md",
"chars": 2188,
"preview": "# <a name=\"ml-coding\"></a> 2. ML/Data Coding :robot:\nML coding module may or may not exist in particular companies inter"
},
{
"path": "src/MLC/notebooks/.test.ipynb",
"chars": 12881,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Kmeans\"\n "
},
{
"path": "src/MLC/notebooks/convolution.ipynb",
"chars": 4489,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Convolution"
},
{
"path": "src/MLC/notebooks/decision_tree.ipynb",
"chars": 7837,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"A decision tr"
},
{
"path": "src/MLC/notebooks/feedforward.ipynb",
"chars": 19296,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Forward p"
},
{
"path": "src/MLC/notebooks/k_means.ipynb",
"chars": 17042,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"id\": \"functional-corrections\",\n \"metadata\": {}"
},
{
"path": "src/MLC/notebooks/k_means_2.ipynb",
"chars": 5399,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"functional-corrections\",\n \"metadata\": {},\n \"source\": [\n \""
},
{
"path": "src/MLC/notebooks/k_nearest_neighbors.ipynb",
"chars": 4092,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"continuing-northern\",\n \"metadata\": {},\n \"source\": [\n \"## "
},
{
"path": "src/MLC/notebooks/knn.ipynb",
"chars": 26855,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# K-Nearest N"
},
{
"path": "src/MLC/notebooks/linear_regression.ipynb",
"chars": 23645,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Linear Regr"
},
{
"path": "src/MLC/notebooks/linear_regression_md.ipynb",
"chars": 3602,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Linear regression python multi-d"
},
{
"path": "src/MLC/notebooks/logistic_regression.ipynb",
"chars": 18858,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Logistic Re"
},
{
"path": "src/MLC/notebooks/logistic_regression_md.ipynb",
"chars": 3755,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## logistic regression multi-dimens"
},
{
"path": "src/MLC/notebooks/numpy_practice.ipynb",
"chars": 31421,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Numpy Practice\\n\",\n \"- Author:"
},
{
"path": "src/MLC/notebooks/perceptron.ipynb",
"chars": 3916,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"The perceptro"
},
{
"path": "src/MLC/notebooks/softmax.ipynb",
"chars": 0,
"preview": ""
},
{
"path": "src/MLC/notebooks/svm.ipynb",
"chars": 5548,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Support Vec"
},
{
"path": "src/MLC/notebooks/ww_classifier.ipynb",
"chars": 29686,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## CS 224N Lecture 3: Word Window C"
},
{
"path": "src/MLSD/ml-companies.md",
"chars": 7145,
"preview": "## ML Systems at Big Companies\n\n- LinkedIn\n - [Learning to be Relevant](http://www.shivanirao.info/uploads/3/1/2/8/3128"
},
{
"path": "src/MLSD/ml-system-design.md",
"chars": 17837,
"preview": "# <a name=\"ml-sys\"></a> 4. Machine Learning System Design\n\n||\n| --- |\n| 1. [The 9-Step ML System Design Formula](#1-the"
},
{
"path": "src/MLSD/mlsd-ads-ranking.md",
"chars": 6923,
"preview": "# Ads Click Prediction \n\n### 1. Problem Formulation\n* Clarifying questions\n * What is the primary business objective of"
},
{
"path": "src/MLSD/mlsd-av.md",
"chars": 15308,
"preview": "\n# Self-driving cars\n- drives itself, with little or no human intervention \n- different levels of authonomy \n\n## Hardwar"
},
{
"path": "src/MLSD/mlsd-event-recom.md",
"chars": 4331,
"preview": "\n# Design an event recommendation system \n\n## 1. Problem Formulation \n\n* Clarifying questions \n - Use case? \n - even"
},
{
"path": "src/MLSD/mlsd-feature-eng.md",
"chars": 3854,
"preview": "\n# Feature preprocessing \n\n## Text preprocessing \nnormalization -> tokenization -> token to ids\n* normalization \n* token"
},
{
"path": "src/MLSD/mlsd-game-recom.md",
"chars": 11939,
"preview": "\n# Design a game recommendation engine \n\n## 1. Problem Formulation \nUser-game interaction \n\nSome existing data examples:"
},
{
"path": "src/MLSD/mlsd-harmful-content.md",
"chars": 5232,
"preview": "# Harmful content detection on social media\n\n### 1. Problem Formulation\n* Clarifying questions\n * What types of harmful"
},
{
"path": "src/MLSD/mlsd-image-search.md",
"chars": 5541,
"preview": "# Image Search System (Pinterest)\n\n### 1. Problem Formulation\n* Clarifying questions\n - What is the primary (business"
},
{
"path": "src/MLSD/mlsd-metrics.md",
"chars": 7372,
"preview": "# Offline Metrics \n\nThese offline metrics are commonly used in search, information retrieval, and recommendation systems"
},
{
"path": "src/MLSD/mlsd-mm-video-search.md",
"chars": 4007,
"preview": "# Multimodal Video Search System \n\n### 1. Problem Formulation\n* Clarifying questions\n - What is the primary (business"
},
{
"path": "src/MLSD/mlsd-modeling-popular-archs.md",
"chars": 486,
"preview": "\n# Popular Neural Network Architectures\n\n* ## Two stage funnel architecture \n * candidate generation + ranking\n\n* ## Tw"
},
{
"path": "src/MLSD/mlsd-newsfeed.md",
"chars": 6879,
"preview": "# News Feed System \n\n### 1. Problem Formulation\nshow feed (recent posts and activities from other users) on a social net"
},
{
"path": "src/MLSD/mlsd-prediction.md",
"chars": 445,
"preview": "# Prediction Service \n\n## Embedding Generation \n\n## Indexing Service \n\n- Index text, image, video by their embeddings \n-"
},
{
"path": "src/MLSD/mlsd-preprocessing.md",
"chars": 271,
"preview": "## Preprocessing Text: \n\n* ### Normalization -> Tokenization [Pre-Tokenization -> Tokenizer Model -> Post-processing] ->"
},
{
"path": "src/MLSD/mlsd-pymk.md",
"chars": 4262,
"preview": "# Friends/Follower recommendation (People you may know)\n\n### 1. Problem Formulation\nRecommend a list of users that you m"
},
{
"path": "src/MLSD/mlsd-search.md",
"chars": 10189,
"preview": "# Search System \n\n### 1. Problem Formulation\n* Clarifying questions\n - Is it a generalized search engine (like google"
},
{
"path": "src/MLSD/mlsd-template.md",
"chars": 981,
"preview": "# The 9 Step ML System Design Formula Template\n\n1. Problem Formulation\n * Clarifying questions\n * Use case(s) and "
},
{
"path": "src/MLSD/mlsd-typeahead.md",
"chars": 0,
"preview": ""
},
{
"path": "src/MLSD/mlsd-video-recom.md",
"chars": 11432,
"preview": "\n# Design a video recommendation system \n\n## 1. Problem Formulation \nUser-video interaction \n\nSome existing data example"
},
{
"path": "src/MLSD/mlsd_obj_detection.md",
"chars": 3032,
"preview": "## 2D object detectors\n### Two stage detectors\nTwo-stage object detectors are a type of deep learning model used for obj"
},
{
"path": "src/behavior.md",
"chars": 1004,
"preview": "# Behavioral Interviews \n\n## STAR Method \n[How to Answer Common Situational Interview Questions](https://www.interviewki"
},
{
"path": "src/lc-coding.md",
"chars": 2086,
"preview": "# <a name=\"coding\"></a> General Coding Interview (Algorithms and Data Structures) :computer: \n\nAs an ML engineer, you're"
},
{
"path": "src/ml-depth.md",
"chars": 1059,
"preview": "# <a name=\"depth\"></a> 3. ML Depth\nML depth interviews typically aim to measure the depth of your knowledge in both theo"
},
{
"path": "src/ml-fundamental.md",
"chars": 6887,
"preview": "# <a name=\"breadth\"></a> 3. ML Fundamentals (Breadth)\nAs the name suggests, this interview is intended to evaluate your "
}
]
About this extraction
This page contains the full source code of the alirezadir/Machine-Learning-Interviews GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 45 files (355.8 KB), approximately 123.2k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.