Showing preview only (885K chars total). Download the full file or copy to clipboard to get everything.
Repository: rickwierenga/CS229-Python
Branch: master
Commit: 2b749ee78829
Files: 38
Total size: 11.4 MB
Directory structure:
gitextract_5_24ji07/
├── .gitignore
├── README.md
├── ex1/
│ ├── PE1 - Linear Regression (Exercises).ipynb
│ ├── ex1data1.txt
│ └── ex1data2.txt
├── ex2/
│ ├── PE2 - Logistic Regression (Exercises).ipynb
│ ├── ex2data1.txt
│ └── ex2data2.txt
├── ex3/
│ ├── PE3 - Multi-class Classification and Neural Networks (Exercises).ipynb
│ ├── ex3data1.mat
│ └── ex3weights.mat
├── ex4/
│ ├── PE4 - Learning Neural Networks (Exercises).ipynb
│ ├── ex4data1.mat
│ └── ex4weights.mat
├── ex5/
│ ├── PE5 - Logistic Regression (Exercises).ipynb
│ └── ex5data1.mat
├── ex6/
│ ├── Support Vector Machines (Exercises).ipynb
│ ├── emailSample1.txt
│ ├── ex6data1.mat
│ ├── ex6data2.mat
│ ├── ex6data3.mat
│ ├── process_email.py
│ ├── spamTest.mat
│ ├── spamTrain.mat
│ └── vocab.txt
├── ex7/
│ ├── K-means Clustering and Principal Component Analysis (Exercises).ipynb
│ ├── ex7data1.mat
│ ├── ex7data2.mat
│ └── ex7faces.mat
├── ex8/
│ ├── Anomaly Detection and Recommender Systems (Exercises).ipynb
│ ├── ex8_movieParams.mat
│ ├── ex8_movies.mat
│ ├── ex8data1.mat
│ ├── ex8data2.mat
│ ├── load_movie_list.py
│ └── movie_ids.txt
├── requirements.txt
└── week6/
└── 1-Evaluating-a-Learning-Algorithm.md
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
*.DS_Store
octave/
exercises/
__pycache__/
*.pyc
*.swp
*.swo
.ipynb_checkpoints
env/
================================================
FILE: README.md
================================================
# Stanford CS229 Machine Learning in Python
This repository contains the problem sets for Stanford CS229 (Machine Learning) on Coursera translated to Python 3. It also contains some of my notes.
Check out the [course website](http://cs229.stanford.edu/) and the [Coursera course](https://www.coursera.org/learn/machine-learning). Please note that your solutions won't be graded and this repo is not affiliated with Coursera or Stanford in any way.
## Installation
Make sure you have jupyter notebooks installed. You can find instructions [here](https://jupyter.org/install).
The following Python packages are used:
- [Numpy](https://www.numpy.org)
- [Scipy](https://www.scipy.org)
- [Matplotlib](https://matplotlib.org)
- [Pandas](https://pandas.pydata.org)
- [Pillow](https://python-pillow.org)
- [Natural Language Toolkit](http://www.nltk.org)
You can install all dependencies using:
```shell
python3 -m pip install -r requirements.txt
```
## Instructions
1. Please download the exercises (pdf) from the Coursera course. Some instructions are included in the Notebooks.
2. Complete the exercises in the exercises Notebook.
3. Compare your answers to the code in solutions Notebook.
## Contents
1. Linear Regression
2. Logistic Regression & Regularization
3. Multiclass Classifcation & Neural Networks
4. Neueral Networks Learning
5. Regularized Linear Regression and Bias v.s. Variance
6. Support Vector Machines
7. K-means Clustering and Principal Component Analysis
8. Anomaly Detection and Recommender Systems
## Copyright Notice
All code, exercises, data and other files in this repo are ©Stanford University. If you are unhappy about me hosting these files on GitHub for educational purposes, please send me an email.
The code was 'translated' to Python by Rick Wierenga. Some of the instructions are modified to better fit the Python ecosystem by me too. The data, background information and the intended exercise are the same.
### Solutions
At the request of Stanford staff, I removed the notebooks with solutions from this repository. The exercises and notes are still available.
---
©2020 Rick Wierenga
================================================
FILE: ex1/PE1 - Linear Regression (Exercises).ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Linear Regression\n",
"\n",
"Stanford CS229 - Machine Learning by Andrew Ng. Programming exercise 1.\n",
"\n",
"Please check out [the repository on GitHub](https://github.com/rickwierenga/CS229-Python/). If you spot any mistakes or inconcistencies, please create an issue. For questions you can find me on Twitter: [@rickwierenga](https://twitter.com/rickwierenga). Starring the project on GitHub means a ton to me!"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pylab as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Linear Regression with a single variable\n",
"---\n",
"In this part of this exercise, you will implement linear regression with one variable to predict profits for a food truck. Suppose you are the CEO of a restaurant franchise and are considering different cities for opening a new outlet. The chain already has trucks in various cities and you have data for profits and populations from the cities. You would like to use this data to help you select which city to expand to next. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Population</th>\n",
" <th>Profit</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>6.1101</td>\n",
" <td>17.5920</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>5.5277</td>\n",
" <td>9.1302</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>8.5186</td>\n",
" <td>13.6620</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>7.0032</td>\n",
" <td>11.8540</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>5.8598</td>\n",
" <td>6.8233</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Population Profit\n",
"0 6.1101 17.5920\n",
"1 5.5277 9.1302\n",
"2 8.5186 13.6620\n",
"3 7.0032 11.8540\n",
"4 5.8598 6.8233"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# start by loading the data\n",
"data = pd.read_csv('ex1data1.txt', header=None, names=['Population', 'Profit'])\n",
"\n",
"# initialize some useful variables\n",
"m = len(data) # the number of training examples\n",
"X = np.append(np.ones((m, 1)), np.array(data[\"Population\"]).reshape((m,1)), axis=1) # Add x0, a vector of 1's, to X.\n",
"y = np.array(data[\"Profit\"]).reshape(m, 1)\n",
"\n",
"data.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualising the data\n",
"Plotting helps us get insight in the data we are working with. Using the `'bx'` option, we get blue crosses. You can read more about markers [here](https://matplotlib.org/api/markers_api.html)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Text(0.5, 1.0, 'Relation between profit and population')"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xm8XVV99/HPF8IYIYCJlCmkIvbRSgW5AbFguSAWKCpQy6AGHCNRHqWRRtAnEUktD5FgUSs+KsqkhrQiAmIVyBWHFkxAZlRAQUAIYZBJKgK/54+1N3ffkzPse++Zz/f9eu3XOWeP65y77/rttdbeaykiMDOzwbVOpxNgZmad5UBgZjbgHAjMzAacA4GZ2YBzIDAzG3AOBGZmA86BoEdI2lvSvZPY/ouSFjYzTdl+T5J0frP3268kbSnpR5KekLRU0sckfaVDaZklKSRN6cTxa5H0Q0nvneC2MyU9KWndZqern3XVCdDvJN0FbAk8BzwJ/CdwbEQ82eTjvBN4b0Tsmc+LiGOaeYxmkHQS8LKIeEen09JGc4GHgE2j4iEeSbOA3wDrRcSz7U9a78n+p94bEVcARMRvgRd1NFE9yCWC9ntTRLwI2BnYBTixw+mxJil5Zb09cGtlEDDrJAeCDomIB4DvkwICAJI2kHSapN9KWp1V52xUbXtJJ0i6M6tiuFXSIdn8VwBfBPbIisi/z+afLemfC9u/T9Idkh6RdLGkrQvLQtIxkm6X9HtJ/yZJdb7OhpIuyNJynaRXF/a1taRvSVoj6TeSPpTN3x/4GHB4ls4bJA1Luqmw7eWSVhY+/1jSwfX2my1bp/D7PCxpuaQtsmV5dcjR2e/8kKSP1/pi2e/2xSwtT0i6StL2Fb/VByXdDtyezXudpJWSHsteX5fvCzgaWJB95zdUVK39KHv9fbZ8jyrp2U3Sf2d/l/slfV7S+mX+dpLWzc6vhyT9Gvi7On9TJN0l6cTs/HpU0tckbVhY3ugc+pCkX2fH+7SkdbJlY6oTVaeKStIOklZkf8eHJH1d0mbZsvOAmcAl2e+1oHJf2XlycZbGOyS9r7Dvk7Jz49zsb3uLpKF6v0nfighPbZqAu4A3ZO+3BW4Czigs/wxwMbAFsAlwCXBKtmxv4N7Cuv8AbE0K5ocDTwFbZcveCfyk4thnA/+cvd+HVD3xGmAD4HPAjwrrBnApsBnpH20NsH+N73QS8CfgrcB6wPFk1RtZ2q4FFgHrAy8Ffg38bWHb8wv72gj4H2B6tv1q4L7st9gIeBp4cYn9fhi4OvuNNwD+H/DNbNms7Pt9Odvnq4E/Aq+o8f3OBp4AXp/t64zib5vt6/Lsb7ZR9vooMIdU9Xpk9vnFlX+Hyt+gkLYpdc6hXYHXZvueBdwGHFfmbwccA/wC2C5L50i945HO15sL6/+U8Z1DI9l2M4Ffkapwqv3dx3xv4IeFdV8G7JcdYwYpWP5rtf+pGvv6EfAFYEPSRdcaYJ9COv4HOBBYFzgFuLrT+URH8qZOJ2CQpuykfTLLWAK4EtgsWyZSZr5DYf09gN9k7/emEAiq7Pt64C3Z+3dSPxCcBSwpLHsRKTOflX0OYM/C8uXACTWOe1Lxn4eUSd8P7AXsDvy2Yv0Tga8Vtj2/YvmPgUNJmd0PsmPvDwwDN2brNNrvbcC+hWVbZd8vzzwD2Law/GfAETW+39nAsorf6jlgu8JvtU9h+RzgZxX7+G/gnZV/h8rfgBKBoEr6jgO+Xfhc828HrACOKSx7Y73jkc7X4voHAneO4xzav7D8A8CV1f7uld+bQiCokqaDgZ9XpLFqICAFsOeATQrLTwHOLqTjisKyVwJPN+N/vdcmNxa338ERcYWkvwG+Qbr6/T3pamdj4NpCLYxIVyprkXQUMJ904kP6R5xeMg1bA9flHyLiSUkPA9uQ/rEAHiis/wfqN8DdU9jX80p3N21N+ofcWln1VGZdUmZfy1VkQS97/yjwN6Sr9quydbZvsN/tgW9Ler6w/DlSQ31uot/vSUmPZN/vnsrl2fy7K7a/m/TbTpqklwOnA0Ok82UKqXRUVOu7FdOcp6uRyvXz6p8y51CtbUuTtCWpFLYXqWS4DumcKGNr4JGIeKIiHcXqn8rfakNJU2LAGuvdRtAhEXEV6erwtGzWQ6Sqj7+MiM2yaVqkhuUxsjrqLwPHkqocNiMV4fMI0qgh8nekzDLf31RSlct9E/w62xX2tQ6pSuZ3pIzgN4Xvs1lEbBIRB9ZJZx4IXp+9v4oUCP6G0UDQaL/3AAdULN8wIprx/V5Equ74XWF58XuM+W0zMyn325ZpQD6TVL2zY0RsSmpnqdd+U3Q/he+SpauRyvXz713mHKq17VOkIJb7szrH/xfS77JT9n3fwdjvW+83+x2whaRNKtIx0fOgbzkQdNa/AvtJenVEPE/K3D8j6SUAkraR9LdVtptK+gdYk633LuBVheWrgW2LjYgVvgm8S9LOkjYg/bNdExF3TfB77Crp0KyB7jjS1fvVpCqXJyR9VNJGWWPlqyTNLqRzVt6ImPkv4C+A3UhVLLeQMpzdGW1MbbTfLwKfyht1Jc2Q9JYJfjeAAyXtmf2ei0lVYffUWPcy4OWS3iZpiqTDSVUOl5Y4zhrgeVKbRy2bAI8DT0r6X8C80t8iVRN9SNK2kjYHTiixzQez9bcAPg5ckM0vcw79k6TNJW1HarfJt70eeL3SPf/TqH/n3Cak6tTHJG0D/FPF8tXU+L2yv9F/AadI2lDSXwHvAfzcSwUHgg6KiDXAuaRGT4CPAncAV0t6HLiClClWbncrsJRU97wa2InUkJdbAdwCPCDpoSrbXwEsBL5FukrcAThiEl/lO6QG67yR9NCI+FNEPAccRGqk+w2p1PMVYFq23b9nrw9Lui5L21OkKodbIuKZbPl/A3dHxIPZOo32ewap0f0Hkp4gBaXdJ/H9vgF8AniE1Fhb87mHiHg4S9tHgIeBBcBBEbHW36HKtn8APgX8NLvj57VVVjseeBupnenLjGauZXyZdKfaDaTf+MIS23yD1Fbza+BO4J+ztJY5h75Dqra6HvguqV2BiLg8S/eN2fJ6QfKTpAbpx7J9VKb5FOD/ZL/X8VW2P5JUffo74NvAJ7K0W4GyRhIzq0Lpls97I+L/dDot7aaKh7XGuW2Qqq/uaHrCrOlcIjAzG3AOBGZmA85VQ2ZmA84lAjOzAdcTD5RNnz49Zs2a1elkmJn1lGuvvfahiJjRaL2WBYLs3uFzSU9zBvCliDhDqevh95HdAw98LCIuq7evWbNmsWrVqlYl1cysL0kq8/R4S0sEzwIfiYjrsif7rpV0ebbsMxFxWp1tzcysTVoWCCLiftKDJkTEE5Juo0n9rZiZWfO0pbFYaeSlXYBrslnHSrpR0lezR92rbTNX0ipJq9asWVNtFTMza4KWB4Ksk65vkfpMf5zUadYOpO4B7id1lbCWiPhSRAxFxNCMGQ3bOszMbIJaGggkrUcKAl+PiAsBImJ1RDxX6GRtt1amwczM6mtZIFDqVP8s4LaIOL0wf6vCaoeQuk82M7PMkiUwMjJ23shImt8KrSwR/DWpJ8p9JF2fTQcCSyTdJOlG0qhT/9jCNJiZ9ZzZs+Gww0aDwchI+jx7dv3tJqqVdw39hOoDZtR9ZsDMbNAND8Py5SnznzcPzjwzfR4ebs3x3MWEmVkXGh5OQWDx4vTaqiAADgRmZl1pZCSVBBYuTK+VbQbN5EBgZtZl8jaB5cvh5JNHq4laFQwcCMzMuszKlWPbBPI2g5UrW3O8nhiPYGhoKNzpnJnZ+Ei6NiKGGq3nEoGZ2YBzIDAzG3AOBGZmA86BwMx6Rru7XhgUDgRm1jPa3fXCoOiJMYvNzKD9XS8MCpcIzKyntLPrhUHhQGBmPaWdXS8MCgcCM+sZ7e56YVA4EJhZz2h31wuDwl1MmJmRbkGdPXtsm8PISAoyCxZ0Ll2T4S4mzMzGYZBvTfXto2ZmDPatqS4RmJllBvXWVAcCM7PMoN6a6qohMzNG2wQOPTSVBIaHR29Vhd5uNG7EJQIzM0ZvTT3iiBQAIH1etqz/G41dIjAzY+zVfrHR+MIL+7/R2CUCM7MKg9Zo7EBgZlZh0BqNHQjMzAoGsT8jBwIzs4JB7M/IfQ2ZmfWpjvc1JGk7SSOSbpV0i6QPZ/O3kHS5pNuz181blQYzM2uslVVDzwIfiYhXAq8FPijplcAJwJURsSNwZfbZzMw6pGWBICLuj4jrsvdPALcB2wBvAc7JVjsHOLhVaTAzs8ba0lgsaRawC3ANsGVE3J8tegDYssY2cyWtkrRqzZo17UimmdlAankgkPQi4FvAcRHxeHFZpJbqqq3VEfGliBiKiKEZM2a0OplmZgOrpYFA0nqkIPD1iLgwm71a0lbZ8q2AB1uZBjMzq6+Vdw0JOAu4LSJOLyy6GDg6e3808J1WpcHMzBprZadzfw3MAW6SdH0272PA/wWWS3oPcDdwWAvTYGZmDbQsEETETwDVWLxvq45rZmbj4y4mzMwGnAOBmdmAcyAwMxtwDgRmZgPOgcDMbMA5EJiZDTgHAjOzJlqyZO3RzEZG0vxu5UBgZtZEs2ePHdoyH/py9uzOpqueVj5ZbGY2cPKhLQ87DObNgzPPHDv0ZTdyiaCBXizmmVlnDQ+nILB4cXrt5iAADgQN9WIxz8w6a2QklQQWLkyvlReT3caBoIFiMW/RovTa7cU8M+uc/GJx+XI4+eTR/KObg4EDQQm9Vswzs85ZuXLsxWJ+MblyZWfTVY/SIGHdbWhoKFatWtWx4+cRvlcafsz61ZIlqVq2+P83MpIy2QULOpeubiXp2ogYarSeSwQN9GIxz6xfuc2uNRwIGujFYp5Zv3KbXWu4asjMes6iRanNbuHCVFK36lw1ZGZ9qdduzewFDgRm1jPcZtcaDgRm1jPcZtcabiMwM+tTbiMwM7NSGvY+KknAbsA22az7gJ9FLxQlzMysobqBQNIbgS8At5MCAMC2wMskfSAiftDi9JmZWYs1KhGcAbwhIu4qzpT058BlwCtalC6znufuEKxXNGojmALcW2X+fcB6zU+OWf9wdwjWKxqVCL4KrJS0DLgnm7cdcARwVisTZtbrenGkKhtMdUsEEXEK8HZAwB7ZJODt2bKaJH1V0oOSbi7MO0nSfZKuz6YDJ/8VzLqXuzC3XtDwrqGIuBW4VdIW2edHSu77bODzwLkV8z8TEaeNJ5FmvaqyO4ThYQcD6z51SwSSZkpaJulB4BrgZ9lV/jJJs+ptGxE/AsoGDbO+4+4QrFc0aiy+APg2sFVE7BgRLwO2Ai4Clk3wmMdKujGrOtp8gvswa6klS9bOsEdG0vyy3B2C9Yq6XUxIuj0idhzvssI6s4BLI+JV2ectgYeAABaTAsy7a2w7F5gLMHPmzF3vvvvuhl/GrFmKV/PDw2t/NusFzepi4lpJX5C0u6Sts2l3SV8Afj7eREXE6oh4LiKeB75MemK51rpfioihiBiaMWPGeA9lNikeAMUGSaPG4qOA9wCfZGwXExczgdtHJW0VEfdnHw8Bbq63vlknFe/4WbjQQcD6V91AEBHPAGdm07hI+iawNzBd0r3AJ4C9Je1Mqhq6C3j/ePdr1i6+48cGRaO+hqaQSgQHM7ZE8B3grIj4U61tI+LIKrP9EJr1hMo2geFhVw9Z/2rURnAesDOpaujAbPok8Grg/NYmzaxzfMePDZJGdw39KiJePt5lzeaBacpxJ2dmVtSsu4YekfQPkl5YT9I6kg4HHp1sIq253MmZmU1Eo0BwBPBWYLWkX0n6FfAAcGi2zLqIb3kcn2Y8NGbWDxp1OndXRBweETPIOp2LiJdk837TniTaeLiTs/JcgjJLSo9ZHBEPR8TDAJKGJG3dumTZRFXe8uh+bWpzCcosmejg9f8b+K6kC5qZGJscd3I2fi5BmU0wEETE0RGxC/DeJqfHJsG3PI6fS1BmDW4fBZA0DdifsQ+UfT8ift/itL3At49aK7hjOet3Tbl9VNJRwHWkriI2zqZhUmd0RzUhnWYd4xKUWdLogbJfArtXXv1n4whc4wfKzMy6V7MeKBOpg7hKz2fLzMysxzXqhvpTwHWSfgDck82bCexHGljGzMx6XKMHys4BhoCrgD9m0w+BoYg4u9WJs9bz07XdxX8P64SGt49GxKMRsSwilmbTsohwP0NdarwZiZ+u7S7+e1gnTPSBMiTd1MyE9Kt2XOEVj5FnJKefPjq/Xkbip2u7i/8e1hERUXMidS5Xbfp7YE29bZs57brrrtGrVqyImD49vVb73IpjLF0aIUXMmVP+WAsXRkB6tc7z38OaAVgVJfLYRoHgT8DZwNeqTE+UOUAzpl4OBBGjGfXChc0PArWOMWdO+YykHemz8vz3sGZpViC4FnhVjWX3lDlAM6ZeDwQR7bnCy4+RlwTKZCTtKLFYef57WDOVDQSN2giOAx6vseyQiVRFDaJ29GeTH2POHDj/fDjxxHIdz/np2u7iv4d1QsO+hrpBLz9Z3I7+bIr7XLkSpkyBU04Ze0wPV2k2eMo+WdzogTIkvQR4KiKekrQRMB/YBDgjIu6ffFL7W70rvGYFguIx8n3ussvoMYrzzcwqlel9dAXwzoj4raQlwAzgF8D+EdGW7KWXSwRmZp3SrN5HjwZ2APbO3h8OrCKNW7y9pKMk/VUzEmxmZp3RqGroh8BTwI3Ai4HVwCWkDuc+mC1/rHXJMzOzVqsbCCLibkmfA75P6nH0fVkV0Uzg4Yj4bTsSaWZmrdOwsTgizpR0HvB8RPwhm/0wcGRLU2ZmZm3RMBAARMSTFZ+fak1yzMys3Sbc6Vwjkr4q6UFJNxfmbSHpckm3Z6+bt+r4ZmW422ezFgYCUh9F+1fMOwG4MiJ2BK7MPpt1jLt9NmthIIiIHwGPVMx+C3BO9v4c4OBWHd+sDHf7bFYyEEg6NKvOeUzS45KekFSrD6J6tiw8jfwAsGWdY86VtErSqjVr1kzgUGblDA/DvHmweHF6dRCwQVO2RLAEeHNETIuITSNik4jYdDIHznrGq/lYc0R8KSKGImJoxowZkzmUWV3t6BTQrJuVDQSrI+K2JhxvtaStALLXB5uwT7MJK3bYV6a3VrN+VDYQrJJ0gaQjs2qiQyUdOoHjXQwcnb0/GvjOBPZh1jTu9tmsZDfUkr5WZXZExLvrbPNNYG9gOqlrik8AFwHLgZnA3cBhEVHZoLyWVnQ6t2RJujOkWB/s7pqtGp8r1qua1g01QES8a7wJiIhaTx7vO959tUJ+22C1cQLMinyuWL+rGwgkLYiIJVl/Q2sVHSLiQy1LWYsVbxucNy81Evq2QavG54r1u0YlgryBuC8HAyjeNrhwof+xrTafK9bPGvU+ekn2ek699XpV5W2DHsnLavG5Yv2slV1MdDXfNmhl+VyxfjewgcC3DVpZPles35W9ffSvI+Knjea1iscsNjMbv6aMWVzwuZLzzFrGXUabtUaj20f3AF4HzJA0v7BoU2DdVibMrJLv5zdrjUYlgvWBF5ECxiaF6XHgra1NmvWKdl2pu8tos9ZodPvoVcBVks6OiLvblCbrMe28Uvf9/GbNV7dEIOlfs7efl3Rx5dSG9FkXaHTF384rdXcZbdZ8jZ4sPjd7Pa3VCbHuVeaKvx1X6sXj5g90uXrIbPIatRF8Ons9MCKuqpxanbhO6MU7U1qd5jJX/O24Uvf9/GYtEhE1J+BW0l1DtwG7AK8pTvW2bea06667RrusWBExfXp6rfa5G7UrzQsXRkB6nezxTz117eUrVqT5ZtYcwKookcc2CgRvBb4HPAGMVEwryhygGVM7A0HEaEa2cGH3B4Fcq9Ncb/8TydR7MeCa9ZqmBIIXVoKFZdZr1dTOQJBnasWr33ZcqTbjCrnWFftkj1km055MMOilgGvWS5oaCNL+eDOp0fg04KCy2zVjanfV0KabRkybljKoadPS51ZnUtUy26lTI5YuXXu9apnrRDLVslfl9TL5fFlx2xUrIubOLZeO8QYvMyuv2SWCU4ArgXdn0+XAv5TZthlTuwNBnvkvXDgaFKplaM2u567MzJcurZ1RF4+dz1+6dHT+eIPBRK/KKwPAtGkRG21U+zdr5rHNrL5mB4IbgXUKn9cFbiyzbTOmbq0aakU9d+UVcq3MsjIoVAsaZQPSZK/Ki2ncaKNy+3IbgVnrtSIQbFH4vEW/BoKI8V2pNvOqtta+Gt2tM5ljNyv9eRo33rjcvnzXkFnrNTsQHAncDZwNnAP8Bji8zLbNmLr99tFm1HPXOm5+pV8rc53MsZt1VZ63q2y88Wh7iq/wzTqvaYEAELAdsFXWYPxm4M/K7LxZUyeqhorqXamuWJEywDlzJl41U+u4S5emfdfKqCd7Nd+Mq/I8DXPnVm809hW+Wec0u0RwU5n1WjW1u2qorMqr9srXyV4N18uou6WO3VU8Zt2rbCAoO0LZOcDnI6IjD/OPd4SyJUtS/ziVXSCsXAkLFjQvXcXj5P3gHHAAXHghXHJJa/u/WbIE7rwTjjhi9DgjI7BsGeywQ3O/p5n1pmaPULY7cLWkOyXdKOkmSTdOLomtk3eSlvd3k2fSs2c39zgLFozt92bePDjvPJg/v/WdoC1YkIJA5SDqF17Y/O9pZn2uTLEB2L7aVGbbZkwTqRpq5T3q9erzm9UFQ1nN/p7dXtXT7ekz6yY0qa+hDYHjgM8D7wemlNlps6eJthG06qnVuXNHH5g69dSIefMipIiDDkrLyzbyNis4NfN7dkvbQy3dnj6zbtKsQHABcH4WBC4Cziiz02ZPEwkEeWZdvFJuxpVj/vBW/sTxnDnpV1x33bF3zeRP+eZaVUJptN9+7AOo29Nn1i2aFQhuKryfAlxXZqcNDwp3ATcB15dJ6HgDQd7VQfGe9mb1GVTM6POnaCH1C9QoY2p05T6RW1cbXR1P9Aq62/sA6vb0mXWDZgWC6+p9nuiUBYLpZdcfbyCo7Agt7zNo7txx7aam/AGqKVPSL7jBBqMlg1oZU56WffddOyBN9JbQ8fYeWvYKutuvuLs9fWbdolmB4Dng8Wx6Ani28P7xMgeosd+WBoKiVlw5rliRMn+I2Guv1B4grf1QWXH9yo7Zaj2B26pMruzv0O118N2ePrNu0pRA0KqJ1EXFdcC1wNwa68wFVgGrZs6cOaEfoUymOpE69L/7u/TLzZmTMvSpU1MwyJ+ubXTXUB4M9t23+d1GVDOe4NLtd+V0e/rMukm3B4JtsteXADcAr6+3/mRuH2105TjeK8wVK8aOEzB3bgoGxcbhMhnTeDqSm0zm5ytos8HV1YFgTALgJOD4eutMJBCMJ/Ns9xVz5fFq9dOTD+5Sb1yCRnwFbTa4ujYQAFOBTQrv/wvYv942ze5rqFrm2Kixt1kqM/G5c1MJY9NNI3bbLWX6xecQli6NOOCA1lVzmVn/6uZA8NKsOugG4Bbg4422aXYgqMyMly6t39jbTNXaCzbdNLU7bLrpaCP00qVrpzOvTpozZ+3v000d0ZlZd+jaQDCRqRW9j+aZ5Jw5KQjkdf7NyjwbXZ1XG2pyp53ihdtRa41I1ii9E7nryCUJs/408IGgTObW6Ap7MhpdnVd+3nHHlJaddhp9UG3OnOpX+o1KMOO968glCbP+NPCBoGxG3MpBXRodI1++337pL7HeeqltYOrUiPXXH73yH0+bxkS/V6ueXzCzzhn4QBAx/rtzxpP5lb2KbnR1nmfo++03eqW//vopGBx0UP0AUm9A+3ppqsXdNpj1FweCTDFzyzPGPCDkzwDkASG/8i5bLVT2ir/e8qlTR6t45s4dDQz77ju6TrVSRrXMvhnPG7hEYNY/HAiieuZWnJf3IDqZ8XYbPRhWmWHXK5Xk6Wn3LaJuIzDrTwMfCOplbnnmve++qTpm2rTUgVz+xHDxQa5qd/nkJjIYTb7vyoCQpyPvGK+dmbHvGjLrTwMfCBplxMUSQV4dk9+6mXcZUe8qOa/Pn8htp9VKKs6MzazZHAjqXMEXM/e859C99hoNBnvt1bjOv9jfUHF52YzbDbNm1moDHwiqVQ1VZt4rVqQSwS67pF9i/fVHxxiofLYg14wM3A2zZtYOAx8IIhpnuPmwk9OmpYe48u6kX/OasdU+Zfc3njS5YdbMWq1sIFiHPjY8DPPmweLF6XV4eOzy2bPhlFPg8MNhn31SxdB668Fpp6Xp4x+H978/rTsyAgcdBCeeCCefDMuXw2GHwemnw5Il5dO0cmXaNk/L8HD6vHJlc76zmdl4Tel0AlppZATOPBMWLkyvw8Njg0ExU37/+1MQWLQozZ89GzbYYOy6ixenwLHLLmmbE09M619ySfk0LViw9rzKdJmZtVWZYkOnp1YOTFNtm3pVP67fN7NewaBXDa1cCYceOvo5r4JZtqx2VU6jqqSy65iZ9ZK+DQQLFsARR6R6/JGRlPn//Odw4YWp2gdG5+cqq5JGRtbeb5l1zMx6SV+3EeSlgMMOg+22g+uvT43Aw8MpAz/kkNRQvGQJTJmSMvd99knLN9ssNQ4vXgy//CXssEMKIIcdNtquMDw89rOZWS9SqkbqbkNDQ7Fq1aoJb79oUcrQ118fNtoIPvQh+Oxn011CF12U1nnTm+Coo+Ab34A//Sk1HL/tbfC1r6XtLrpotBG5mOmPjKT51RqBzcw6SdK1ETHUcL1+DwQjI+mqfd68lPk//TQ880wKCN/97mimnq93wAFw/vkpEEyZkqaLLvIVv5n1nrKBoG/bCGA0c1++PN37v2hRCgIA0th180bg886DPfdM6/3hD/DhD48NAkuWrN0uUNnWYGbWS/o6EBSfExgZScFg441h333Tlf4hh4xm6nkj8Jw58JOfpOqg9daDpUvHZvxTpqS2g+J2hx022gBtZtZr+joQLFgwejW/bFlqE7j0UrjiilTdE5Hm55n5iSfCxRenaqONNoL3vjdVJeUZ/8hIeqBs8eK0/qJF5RuLXZIws27V14GgaIcdxtb1f/rT8Pa3p/l5yWFkBDbdNAWLww+H555Ldxk9/zx86lNw8MHp2YT588f/LEF+x5FLEmbWdco8ddbpaTJDVdZSOZ5AvfEF8h5H8y6sPUC8mfUCSj5Z3NdmgfyPAAAMTUlEQVTPEdQzf356/chH4Oyz4eab09X//Pmjt4QuX57aEZ55JrUtTJmSHkpbuDCVBubPH32W4MQT4dln699GWnwqeeFC34lkZt2hr6uGqtXL7757emYAUka+005w000wdepoEChW2TzzTGon+MhHUtXSokVw9NGprWBkZGznc42qefxUspl1o74uEVQ+CTwykq78n346dR8NKQgAPPkkDA3B3XePrv+mN6UG5WLvpZdcMlpayJ9POPPMNL/eFX7xVlY/lWxmXaVM/VGnp8m0EVSrl1+6NF4YlhLS5113Te/zQ5UZkzhvO6gczazakJUek9jM2g2PUJaceuro4PT58JIrVowOSbnXXqOZ/MtfPpqxb7xx/TGJ823mzJn4IPaT5eBiZvV0dSAA9gd+CdwBnNBo/ckEgvzKfs6clEEvXZqGpITRYJBn+tOnR+y3X/Wr/KLKzL7yGO26G8jDXppZPV0bCIB1gTuBlwLrAzcAr6y3zWTHLM4z+bxkkFcHrVgRsd566XM+sH2+3tSptTPUalfilaWOdvEtqWZWSzcHgj2A7xc+nwicWG+biQaCYoad1+dvvXXEQQeNrlOcP9Gr605nxvl3aHcQMrPu1s2B4K3AVwqf5wCfr7LeXGAVsGrmzJmT+jFqZdTF+XmJoHK7RvXtna6e6XQQMrPu1fOBoDg1466hyow6rwaabAbeyQbbTgchM+tuZQNBJx4ouw/YrvB522xeSxR7IIXRUcuuuKL6/JUrx7f/Ysd2ueHh9gxUU+u7jfc7mNlg68QDZSuBHSX9OSkAHAG8rd2J2Hvv6hl4Lz3cVS3Y9Np3MLPOa3uJICKeBY4Fvg/cBiyPiFtadTz3+mlmVl9HupiIiMuAy9pxrOIA9nl3EO7WwcxsVF93OpdbuTKNRVwcP8CDwpiZJX3d6VxuypQ0IP2cOalEsNlmqffQ5cs7nTIzs87r+xJBPrzkaafB976XSgbHH5+6jp5o9ZCHnTSzftL3gSC/xTIfXvK88+Ad70iDyEyUG6DNrJ/0fSDI7/MvDgrzve9NLtMuNkCPZwB7M7Nu1PeBAMYOCnPyyaOZ+GRGCCsOO1l2AHszs240EIGgFU/gethJM+sXSt1RdLehoaFYtWpVp5PxgsphJys/m5l1A0nXRsRQo/UGokTQbO7jx8z6iUsEZmZ9yiUCMzMrpS8DgR/4MjMrry8DgR/4MjMrry/7GnKPo2Zm5fVliQD8wJeZWVl9Gwj8wJeZWTl9GQha0aWEmVm/6stA4Ae+zMzK8wNlZmZ9yg+UmZlZKQ4EZmYDzoHAzGzAORCYmQ04BwIzswHXE3cNSVoD3D3BzacDDzUxOa3m9LZer6XZ6W2tXksvlE/z9hExo9FKPREIJkPSqjK3T3ULp7f1ei3NTm9r9Vp6oflpdtWQmdmAcyAwMxtwgxAIvtTpBIyT09t6vZZmp7e1ei290OQ0930bgZmZ1TcIJQIzM6vDgcDMbMD1TSCQdJekmyRdL2mtrkqVfFbSHZJulPSaTqQzS8tfZOnMp8clHVexzt6SHiuss6jNafyqpAcl3VyYt4WkyyXdnr1uXmPbo7N1bpd0dIfT/GlJv8j+5t+WtFmNbeueP21M70mS7iv83Q+sse3+kn6Znc8ndDC9FxTSepek62ts24nfdztJI5JulXSLpA9n87vyPK6T3tafwxHRFxNwFzC9zvIDge8BAl4LXNPpNGfpWhd4gPTgR3H+3sClHUzX64HXADcX5i0BTsjenwCcWmW7LYBfZ6+bZ+8372Ca3whMyd6fWi3NZc6fNqb3JOD4EufMncBLgfWBG4BXdiK9FcuXAou66PfdCnhN9n4T4FfAK7v1PK6T3pafw31TIijhLcC5kVwNbCZpq04nCtgXuDMiJvrkdEtExI+ARypmvwU4J3t/DnBwlU3/Frg8Ih6JiEeBy4H9W5bQgmppjogfRMSz2cergW3bkZYyavzGZewG3BERv46IZ4BlpL9NS9VLryQBhwHfbHU6yoqI+yPiuuz9E8BtwDZ06XlcK73tOIf7KRAE8ANJ10qaW2X5NsA9hc/3ZvM67Qhq//PsIekGSd+T9JftTFQNW0bE/dn7B4Atq6zTrb8zwLtJpcJqGp0/7XRsVg3w1RrVFt34G+8FrI6I22ss7+jvK2kWsAtwDT1wHlekt6gl5/CU8Sawi+0ZEfdJeglwuaRfZFcwXUvS+sCbgROrLL6OVF30ZFZPfBGwYzvTV09EhKSeufdY0seBZ4Gv11ilW86fM4HFpH/qxaTqlnd3IB3jdST1SwMd+30lvQj4FnBcRDyeCi9JN57HlektzG/ZOdw3JYKIuC97fRD4Nqn4XHQfsF3h87bZvE46ALguIlZXLoiIxyPiyez9ZcB6kqa3O4EVVufVadnrg1XW6brfWdI7gYOAt0dWmVqpxPnTFhGxOiKei4jngS/XSEdX/caSpgCHAhfUWqdTv6+k9UiZ6tcj4sJsdteexzXS2/JzuC8CgaSpkjbJ35MaV26uWO1i4CglrwUeKxQPO6XmVZSkP8vqXZG0G+lv9XAb01bNxUB+98TRwHeqrPN94I2SNs+qNd6YzesISfsDC4A3R8QfaqxT5vxpi4p2q0NqpGMlsKOkP89KlUeQ/jad8gbgFxFxb7WFnfp9s/+fs4DbIuL0wqKuPI9rpbct53ArW8HbNZHunrghm24BPp7NPwY4Jnsv4N9Id1vcBAx1OM1TSRn7tMK8YnqPzb7LDaQGote1OX3fBO4H/kSqH30P8GLgSuB24Apgi2zdIeArhW3fDdyRTe/qcJrvINX1Xp9NX8zW3Rq4rN7506H0npednzeSMqytKtObfT6QdFfJnZ1Mbzb/7Py8LazbDb/vnqQqthsLf/8Du/U8rpPelp/D7mLCzGzA9UXVkJmZTZwDgZnZgHMgMDMbcA4EZmYDzoHAzGzAORBYR0h6Lusl8WZJ/y5p4ybv/52SPt9gnb0lva7w+RhJRzXh2FtL+o9xbnOsUk+iUXxwMHvupWGvuarRG2n2rME12fwLsucOkLRB9vmObPmsiX1b6wcOBNYpT0fEzhHxKuAZ0jMU7bY38EIgiIgvRsS5k91pRPwuIt46zs1+Snowq7LzwQNIXYvsCMwldUExhqR1Sc/IHEDqrfJISa/MFp8KfCYiXgY8SnpWgez10Wz+Z7L1bEA5EFg3+DHwMgBJ87NSws3KxmiQNEupP/avS7pN0n/kJQilPtinZ++HJP2wcueS3pRd9f5c0hWStsyugI8B/jErmeylNBbA8dk2O0u6WqN9wG+ezf+hpFMl/UzSryTtVeV4s5T12Z+VTC6U9J9K/dovqfYDRMTPI+KuKovK9JpbtTfS7EnVfYC8dFLsabPYA+d/APvmT7Lb4HEgsI5S6qfmAOAmSbsC7wJ2J40Z8T5Ju2Sr/gXwhYh4BfA48IFxHOYnwGsjYhdSJrkgy3S/SLpa3jkiflyxzbnARyPir0hP+n6isGxKROwGHFcxv5adgcOBnYDDJW3XYP2iMr1g1lrnxcDvY7QL4+K2L2yTLX8sW98GkAOBdcpGSqNZrQJ+S+pjZU/g2xHxVKQO9y4kdW8McE9E/DR7f362blnbAt+XdBPwT0DdLr0lTQM2i4irslnnkAZlyeWdgV0LzCpx/Csj4rGI+B/gVmD7caTdrOX6qRtq6y1PR8TOxRkNaiYq+0LJPz/L6AXNhjW2/RxwekRcLGlv0ihgk/HH7PU5yv0P/bHwvuw2uTK9YNZa52FSVdKU7Kq/uG2+zb1ZqWwane/U0DrEJQLrJj8GDpa0cdaD4iHZPICZkvbI3r+NVN0DaXi+XbP3f19jv9MYzQCLY88+QRoScIyIeAx4tFD/Pwe4qnK9NqnZa66kX2TrVO2NNFJHYiNA3nBd7Gmz2APnW4EV4Y7HBpYDgXWNSMP0nQ38jDQy01ci4ufZ4l8CH5R0G2kM2fzumU8CZygN1v1cjV2fBPy7pGuBhwrzLwEOyRuLK7Y5Gvi0pBtJdfwnT+a7NSLpQ5LuJV213yjpK9miy0jj5d5BGp/gA9n600k96uZ1/MeSukm+DVgeEbdk238UmC/pDlIbwFnZ/LOAF2fz55PG7rUB5d5Hretld/hcmt1qaoCkg4CXRsRnO50W631uIzDrQRFxaafTYP3DJQIzswHnNgIzswHnQGBmNuAcCMzMBpwDgZnZgHMgMDMbcP8fLM+P7w87bZMAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(data['Population'], data['Profit'], 'bx')\n",
"plt.xlabel('Population in 10,000')\n",
"plt.ylabel('Profit in $10,000')\n",
"plt.title('Relation between profit and population')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The hypotheses function\n",
"Our hypothesis function has the general form:\n",
"$y= h_\\theta(x)= \\theta_0 + \\theta_1x$\n",
"Note that this is like the equation of a straight line. We give to $h_\\theta(x)$ values for $\\theta_0$ and $\\theta_1$ to get our estimated output y. In other words, we are trying to create a function called $h_\\theta$ that is trying to map our input data (the x's) to our output data (the y's).\n",
"\n",
"### Cost function\n",
"\n",
"The cost functions yields \"how far off\" our hypotheses $h_\\theta$ is. It takes the avarage of the distance between our hypothesis and the actual point and squares it. Formally, the cost function has the following definition:\n",
"\n",
"$J(\\theta) = \\frac{1}{2m} \\displaystyle\\sum_{i = 0}^{m}(h_θ(x^{(i)}) - y^{(i)})^2$\n",
"\n",
"#### Vectorization\n",
"Vectorizations is the act of replacing the loops in a computer program with matrix operations. If you have a good linear algebra library (like numpy), the library will optimize the code automatically for the computer the code runs on. Mathematically, the 'regular' function should mean the same as the vectorized function.\n",
"\n",
"Gradient descent vectorized:\n",
"$\\theta = \\frac{1}{2m}(X\\theta - \\vec{y})^T(X\\theta-\\vec{y})$\n",
"\n",
"**Exercise**: Implement a vectorized implementation of the cost function."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"def cost_function(X, y, theta):\n",
" \"\"\" Computes the cost of using theta as the parameter for linear gression to fit the data in X and y. \"\"\"\n",
" \n",
" return 0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With $\\theta = \\begin{bmatrix}0 & 0\\end{bmatrix}$, $J(\\theta)$ should return 32.07."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0\n"
]
}
],
"source": [
"initial_theta = np.zeros((2,1))\n",
"print(cost_function(X, y, initial_theta))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Gradient descent\n",
"We want are hypothesis $h_\\theta(x)$ to function as good as possibly. Therefore, we want to minimalize the cost function $J(\\theta)$. Gradient descent is an algorithm used to do that. \n",
"\n",
"The formal definition of gradient descent:\n",
"\n",
"$repeat \\ \\{ \\\\ \\enspace \\theta_j := \\theta_j - \\alpha \\frac{1}{m}\\displaystyle\\sum_{i = 1}^{m}(h_\\theta(x^{(i)})-y^{(i)})x_j^{(i)}\\\\\\}$\n",
"\n",
"An illustration of gradient descent on a single variable:\n",
"<div>\n",
" <img style='max-width:50%;' src='notes/gradientdescent.png'>\n",
"</div>\n",
"\n",
"**Exercise**: Implement the gradient descent algorithm in Python."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def gradient_descent(X, y, theta, alpha, iterations):\n",
" \"\"\" Performs gradient descent to learn theta. \n",
" Returns the found value for theta and the history of the cost function.\n",
" \"\"\"\n",
" J_history = []\n",
" return theta, J_history"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Gradient descent should have found approximately the following: $\\theta = \\begin{bmatrix}-3.6303\\\\1.1664\\end{bmatrix}$"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.],\n",
" [0.]])"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# You can change different values for these variables\n",
"alpha = 0.01\n",
"iterations = 1500\n",
"\n",
"theta, J_history = gradient_descent(X, y, initial_theta, alpha, iterations)\n",
"theta"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using the results\n",
"#### Plotting the regularization line"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x11426d940>]"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAG2FJREFUeJzt3X+MHOV9x/HPtzk7sk1soHapS0jdUBKJCimGM01bXGVxSIE6DaDq4vy40Cbppaei5kejEy46C2FVlAvnqlErV6TQEEjjWIpLICFKgFspitpGd6YGnJIUaEEFOXD5oZIolRLg2z+eme7c3v6Y3duZnZl9v6TV7s7O7j63N/e5Z595fpi7CwBQfj837AIAAAaDQAeAiiDQAaAiCHQAqAgCHQAqgkAHgIog0AGgIgh0AKgIAh0AKmKs2w5mdo6kz0g6S5JLus3d/9rMbpT0R5KWo13/3N3v7/RaW7du9R07dqypwAAwao4fP/49d9/Wbb+ugS7pJUl/5u4Pm9lrJB03sweix/7K3W9NW6gdO3ZoaWkp7e4AAElm9kya/boGurufknQquv0jM3tc0tlrKx4AYNB6akM3sx2Sdkr6ZrTpOjN71MzuMLMzBlw2AEAPUge6mZ0m6QuSPuLuL0o6LOlcSW9SqMHPt3nelJktmdnS8vJyq10AAAOQKtDNbJ1CmH/W3Y9Jkrs/7+4vu/srkj4l6eJWz3X329x93N3Ht23r2qYPAOhT10A3M5N0u6TH3f1QYvv2xG5XSzo5+OIBANJKU0P/LUmTki41sxPR5UpJc2b2mJk9Kqkm6aNZFhQASmduTqrXV26r18P2DKTp5fINSdbioY59zgFg5O3aJU1MSEePSrVaCPP4fgbS9EMHAPSjVgvhPTEhTU9Lhw83wj0DDP0HgCzVaiHMDx4M1xmFuUSgA0C26vVQM5+dDdfNbeoDRKADQFaSbeY33dRofsko1Al0AMjK4uLKNvO4TX1xMZO3M3fP5IVbGR8fdybnAoDemNlxdx/vth81dACoCAIdACqCQAeAiiDQAeQv5yHxo4JAB5C/eEh8HOpx975du4ZbrpJj6D+A/OU8JH5UUEMHMBw5DokfFQQ6gOHIcUj8qCDQAeQv5yHxo4JAB5C/nIfEjwqG/gOolrm50Fsm2SZfr4d/FjMzwyvXGjD0H8BoGuEukXRbBFAtI9wlkho6gOoZ0S6RBDqA6hnRLpE0uQColrjN/JprQs28Vmt0kZRKfXK0G2roAKol7hK5b18IcincP3Kk8idHqaEDqJZk7Tt5cvTYscqfHKWGDqC6RuzkKIEOoLpG7OQogQ6gmkZwvhgCHUA1jeB8MV3ncjGzcyR9RtJZklzSbe7+12Z2pqTPS9oh6WlJE+7+w06vxVwuANC7Qc7l8pKkP3P38yW9WdKfmNn5kq6X9JC7nyfpoeg+AGBIuga6u59y94ej2z+S9LiksyW9Q9Kd0W53Sroqq0ICALrrqQ3dzHZI2inpm5LOcvdT0UPfVWiSAQAMSepAN7PTJH1B0kfc/cXkYx4a4ls2xpvZlJktmdnS8vLymgoLAGgvVaCb2TqFMP+sux+LNj9vZtujx7dLeqHVc939Nncfd/fxbdu2DaLMAIAWuga6mZmk2yU97u6HEg/dK+na6Pa1kr44+OIBANJKM5fLb0malPSYmZ2Itv25pL+UdNTMPiDpGUkT2RQRAJBG10B3929IsjYP7xlscQAA/WKkKABUBIEOABVBoANARRDoAFARBDoAVASBDgAVQaADQEUQ6ADQytzc6tWN6vWwvaAIdABoZdeulUvWxUva7do13HJ1kGboPwCMnnjJuokJaXo6LDKdXNKugEanhl7Cr08AhqxWC2F+8GC4LnCYS6MU6CX8+gRgyOr1UDOfnQ3XzZXCghmdQE9+fTpwIFwX/OsTgCGKK31Hj0o33dTIjwKH+ugEulS6r08AhmhxcWWlL64ULi4Ot1wdWFg9Lh/j4+O+tLSU2/utEv/HLckJDqCy5uZCc2fy769eD2E5MzO8chWUmR139/Fu+41ODb2EX5+AyuKcViZGJ9BL+PUJqCzOaWVitJpcABTLgQPhnNbsbPjmjJZocgFQbCXrElgGBDqA/HFOKxMEOoD8cU4rE7ShA0DB0YYOACOGQAeAiiDQAaAiCHSgG6ZeRkkQ6EA3DFNHSbBiEdBNCVeuwWjqWkM3szvM7AUzO5nYdqOZPWdmJ6LLldkWExgypl5GCaRpcvm0pMtbbP8rd39TdLl/sMUCCoZh6iiBroHu7l+X9IMcygIUE8PUURJrOSl6nZk9GjXJnDGwEgGDNIgeKgxTR0n0G+iHJZ0r6U2STkmab7ejmU2Z2ZKZLS0vL/f5dkCfBtFDZWZmdZt5rcbKOiicvgLd3Z9395fd/RVJn5J0cYd9b3P3cXcf37ZtW7/lBPrDQgoYIX0FupltT9y9WtLJdvsCQ0cPFYyINN0WPyfpXyS90cyeNbMPSJozs8fM7FFJNUkfzbicQP/ooYIR0XVgkbu/q8Xm2zMoCzB4yR4qtVq40OyCimLoP6qNHioYISxwUURzc6EXRrIGWa+HEKJnBTByWOCizJgMCkAfCPQioqtdb5jeFpBEoBcXXe3S4xsNIIlALy662qXHNxpAEoFeTEwG1Tu+0QAEeiHR1a53fKMB6LaICmgePNR8Hyg5ui1idPCNBpBEDR0ACo8aOgCMGAIdACqCQK8KRksWC78PDAGBXlS9BgKjJYuF3weGgEBPK48aV/I94kA4dKixvVMgMFqyWPh9YBjcPbfLRRdd5KW1sOC+dWu4bnU/i/eYn3c3c5+cTP9es7PuUrjG8PH7wABIWvIUGUug9yIO3NnZwYd5u/eYnEwfCHmUD+nx+8CAEOhZyaPGFb9HXDNPEwh5fINAevw+MEBpA5029F7kMV9I/B6Tk9Ldd0v796eboIvRksXC7wNDwEjRtPKYLyT5mouL0tiYdPPNK9+TZeiAkZN2pOhYHoWphE41rkEFevI94tfcubPxHsntANCEGjoAFBxzuQDAiCHQAaAiCHQAqAgCHQAqgkAHgIroGuhmdoeZvWBmJxPbzjSzB8zsiej6jGyLCXTBdLVAqhr6pyVd3rTtekkPuft5kh6K7gPDw3S1QPdAd/evS/pB0+Z3SLozun2npKsGXC6gN0xXC/Tdhn6Wu5+Kbn9X0lntdjSzKTNbMrOl5eXlPt8OSKFWk6anpYMHwzVhjhGz5pOi0UxgbYebuvtt7j7u7uPbtm1b69sB7eUxeRpQYP0G+vNmtl2SousXBlckoA/Jic3SzE4JVFC/gX6vpGuj29dK+uJgigP0ielqgVTdFj8n6V8kvdHMnjWzD0j6S0mXmdkTkt4a3R8OuqtBClMKN7eZ12orpxrmWEHFpenl8i533+7u69z9te5+u7t/3933uPt57v5Wd2/uBZMfuqshLY4VVFz550NPdlebng4nw+iuhlY4VlBx1Rj6T3c1pMWxggqrRqDTXQ1pcaygwsof6HRXQ1ocK6i48gc63dWQFscKKo41RQGg4FhTFMVFf3AgEwQ68kd/cCATBDoa8qo5M9UtkAkCHQ151pzpDw4MHIE+SrrVwPOsOdMfHBg4An2UpKmB51Fzpj84kInqBnoZe1JkXeY0NfA8as70Bwey4e65XS666CLPzcKC+9at4brV/SLKq8yzs+5SuF7r+99yy+rHFxbCdgADIWnJU2RsdQPdvRFIs7PFD/NY1mXu9Pr9hHMZ/3ECJUOgx+GUrI3mUXMcRI21XQ16re+ZJnzXEupl+scJlAiBvrDgvnmz+5YtIWi2bAn3sw6bVqG5aZP7/Pzq/VqFZD/hmLaW3Cms48eSz11YcJ+aSleOXv8JAUiNQF9YaIT47Gwj3FsF06DbgZtDeX6+feAm3zvePj/f2N5rqPdbS24O8i1b3DdsaP+ZDfK9AXREoPfS5JJFO3BzjbVd6DWHe6vwT/uPZa215GQZN2xI91q0oQOZI9Dde6s5DrKW2e61uvUuWct7D6r8cRk3bkz3WvRyATJHoPdTcxxEO3C7941r3u1Cci3vPahacnzeYePGxvkGatzA0BHovdYcFxZCkE1O9t/k0e595+fDa7cL3LXWrgdRS47LMDXV+uQoNW5gaAj0XjTXopuv11o77RS4RWmDpukEKKy0gV7sFYvm5sI8I81D0xcXpZmZwRUs+T7xPCNXXCEdOybdd1+2MwHOzUlPPSXt29d4n3pdOnJEOvfcwf6cAEqpGisW5TWd68zMynlFpqelu+6SPvax7Kd1nZkJYd48OdWxYyz4AKAnxQ70rKdzbTUZ1qFD0vx868mpspo8K4ufs+iTkxW9fEAZpWmXGdSl7zb0rEYhTk01Bs7ccov79LS7mfveveHxtCczB9XePcifsyht8+0UvXxAgagyJ0Xj0E32ABnEybp4EE88gnRyMnwcr3rVyl4e8ajNWFajIru9bhXnWCl6+YCCyCXQJT0t6TFJJ9K8Yc+Bnhy+nxySPog5WZKBHY+KlMK8K90CpltNup8uk91qq/3WaIs+x0rRywcUQJ6BvjXt/j0HevOEUfGcLFNT/Xwmq8UDacbGwkfx6lc3aurtAiYuy549q/+x9NsVsdfZEtPWaIteAy56+YCCqEagJ2VRk1tYCCEuue/eHdrLzVYPLkru3zyBVbsRlVmFVdrPoeht1EUvH1AgeQX6f0l6WNJxSVNt9pmStCRp6XWve11/P02acOynjfl3fzd8BJOTIZg3bQqhHo+W7DZXeBzqe/YMfjh/K738kyj6QKGilw8okLwC/ezo+hckPSLptzvt31cNPW1Nrtca38LCynnKp6ZCqCdPgqYJmF4m3FpLiFGjBUZW7r1cJN0o6eOd9ukr0HsJwbxrsM3v124elHiRiE7zondDjRYYWZkHuqRNkl6TuP3Pki7v9JyBz+XSKuS6ndQclOYwnpoKNf7Nm90vvjiEd7If+/y8+xVXZNd8BKCy8gj010fNLI9I+pakG7o9Z+CB3hyq8/OdT2oOUqv29M2bQ7v85s2Nk63z86vLGTfTTE6u/nmKNGEXgEKozsCibuKwm5wMYR63iQ8qBLvVllstIXfBBf7/3SDbrVDUrbz99JKhZg9UUnUCPU1IdavxrkW32nLz/fPOC2W54ILGgKXJydY1727fKHrtJUPNHqik6gR62kDNcnGIbu8RP37ZZeEjXbcutJ1v2uS+fn2jJt5Lm3+/P1dW/d8BDE11At29994kvYRY2lptt9pyHMyXXdaoea9fH0J9797O/wg6LRzdqUztMJweqJRqBbr7ypCKAy4O9rgPeRzscU04bXNL2hp4p8c3bWo0nUxNNQJ+z57GPq1q/a1CexD91amhA5VRrUBvFVLJbfGMiWtZD7PbAKHm4O30LSEuT95dE2lDByqpOoHeKaTiEN6zJzRzbNkSJtqKR4AmB/S06pUSi/uLtwrgdsEbv3ZzsMfliCcQyzNU6eUCVFJ1Ar1boCZr6HEzR9xlMB7K36nWGrd399PdsdU3B0IVwIBVO9BbrSQUz5S4e3cj1Hfv7t4mnpzPJfl42gDmBCSAjFUn0Fs1uTSH8MJCqKHv3Bl+pPXrG3OcN/dNjw0iiDkBCSAH1Ql09+7BGS8nt2VLGMwTT4N74YUrm1PSvl4vZeIEJICMpQ30n1vjGtP5qNWk6Wnp4MFwXautfHzXLunmm6V3vlO69NLQ4LJunXTrreFyww3Shz4U9q3Xpb17pf37pZtuko4elSYmpEOHeltxfnExPDcuS60W7i8uDuZnBoAejQ27AKnU69Lhw9LsbLiu1VaGejJcP/ShEOYHDoTtu3ZJr371yn0PHgz/AHbuDM/Zvz/sf9996cs0M7N6W3O5ACBPaarxg7pkusBFq+d0alKh/RtASagyTS6Li9I11zTux00bR460byLp1kSTdh8AKJHiB/rMjLRvX2jnrtdDiP/bv0nHjoXmFKmxPdbcRFOvr37dNPsAQImUow09rpVPTEjnnCOdOBFOdtZqIYivvjqcEJ2bk8bGQkhfeml4/PTTw0nQgwel73xHOvfc8I9gYqLR7l6rrbwPACVkoXkmH+Pj4760tNT/Cxw4EIJ5/XppwwbpT/9U+uQnQ6+We+4J+7z97dL73if94z9KP/tZOEH67ndL//AP4Xn33NM4WZoM73o9bG91shMAhsjMjrv7eNf9ShPo9XqoRU9PhxD/3/+VfvrTEOxf/nIjnOP9rrhCuvvuEOhjY+Fyzz3UwAGUTtpAL34butQI6aNHQ9/xAwdCmEuS2cp945Odd90lXXJJ2O8nP5E+/OGVYT43t7rdvLktHgBKpByBnuxnXq+HUN+4UdqzJ9S8r766Ec7xyc7JSekb3wjNLOvWSfPzKwN8bCy0rSefNzHRONEKACVTjkCfmWnUro8cCW3mX/qS9OCDoRnFPWyPQ3n/funee0NzzIYN0gc/GJpo4gCv18PAooMHw/4HDqQ/KUrNHkBBlSPQk849d2Vb+Cc+Ib3nPWF7XJOv16XNm0Pov/Od0ssvh14xr7wi/cVfSFddFfq2f+xjvfdFj3vIULMHUDRpRh8N6rKmJejaaZ7PvNP85vEMi/HUuyzEDKAEVKnZFruZnw8/ygUXrA7zeD71eCbGjRsbC19s3Lg6+JOrG3XCPOgAclKtQG+1yMXFF7vv3du4f8EF4cc57bRwv3mtzw0bGgEcz6k+Pb169aJNm7rXuKmhA8hR2kAvx0jR5pGd9bp08mQ40XnoUNjnscfC9Y9/LI2PS88809j/7W8PJ06TszXed1+jzT3u3374cNjeqS092YWSUaYAiiRN6g/qsqYml1a14ripJb7Mz7tfdFG4Hb9XmjVD4+aT5tWNWi1Fx5qhAHKmSjW5uIfAjBeBjtutFxYaS83t3t0I6ze8oRHQyXbyWDKA4+dMTva/WPRa8U8CQAe5BLqkyyV9R9KTkq7vtv+aAj2uaU9ONk5ebtoUfoQ41OPw3rrV/bLLWte6k5pDu/k98mobZzk7AB1kHuiSXiXpKUmvl7Re0iOSzu/0nDWvKRqHdVxTj5tZFhbc160L9+MFpOP9Op3kbFUzbv4WkBdOtAJoI22g9z05l5n9hqQb3f13ovv7ozb5m9s9p9/JuXZc/+W+yggARXHJr27V3R/89b6em8fkXGdL+u/E/Wejbc0FmTKzJTNbWl5eXsPbAQA6SlONb3WR9PuS/j5xf1LS33R6zkB6uTS3M8fNK2ttfx7miUna0AF0oBzWFH1O0jmJ+6+NtmUjOeOi1FjF6MEHW29fXOzt9ZMTgMVqtXwWvGj3s/X6MwAYaWsZWLQo6Twz+xWFIN8n6d0DKVUv3vKW1kFcpkE+rf5plO1nADB0fdfQ3f0lSddJ+qqkxyUddfdvDapgqzDLIQB0tKah/+5+v6T7B1SWzpILRcfD9BluDwD/r1zzoS8uhrVCk/OXs7gEAEhaYw09d2NjYeHnyclQQz/99LDy0NGjwy4ZAAxdeWro8bJxt94qfeUroab+8Y+H5eb6bXZhOTkAFVKeQI+79sXLxt11l/Te90ovvdT/a3KiFUCFlCfQ437i9XpobpmdDTX1tYRv8kRrLwtFA0ABlSfQpZWLS9x0UyOMm5tNelGr9b5QNAAUULkCPYsRlcka/+HDa/vnAABD1Pdsi/3od7bFzDQvJ9d8HwAKII/ZFsuPOVQAVMho19ABoASooQPAiCl2oDPwBwBSK3agM/AHAFIr9lwuzLAIAKkVu4YuMfAHAFIqfqAz8AcAUil2oGcx1B8AKqrYgc7AHwBIjYFFAFBwDCwCgBFDoANARRDoAFARBDoAVASBDgAVkWsvFzNblvRMn0/fKul7AyxO1ihv9spWZsqbrbKVV0pf5l92923ddso10NfCzJbSdNspCsqbvbKVmfJmq2zllQZfZppcAKAiCHQAqIgyBfptwy5Ajyhv9spWZsqbrbKVVxpwmUvThg4A6KxMNXQAQAeFC3Qze9rMHjOzE2a2aiYvCz5pZk+a2aNmduEwyhmV5Y1ROePLi2b2kaZ93mJm/5PY50DOZbzDzF4ws5OJbWea2QNm9kR0fUab514b7fOEmV075DJ/wsy+Hf3O/8nMTm/z3I7HT47lvdHMnkv83q9s89zLzew70fF8/RDL+/lEWZ82sxNtnjuMz/ccM6ub2b+b2bfM7MPR9kIexx3Km/0x7O6Fukh6WtLWDo9fKekrkkzSmyV9c9hljsr1KknfVegvmtz+FklfGmK5flvShZJOJrbNSbo+un29pFtaPO9MSf8ZXZ8R3T5jiGV+m6Sx6PYtrcqc5vjJsbw3Svp4imPmKUmvl7Re0iOSzh9GeZsen5d0oECf73ZJF0a3XyPpPySdX9TjuEN5Mz+GC1dDT+Edkj7jwb9KOt3Mtg+7UJL2SHrK3fsdOJUJd/+6pB80bX6HpDuj23dKuqrFU39H0gPu/gN3/6GkByRdnllBE1qV2d2/5u4vRXf/VdJr8yhLGm0+4zQulvSku/+nu/9U0hGF302mOpXXzEzShKTPZV2OtNz9lLs/HN3+kaTHJZ2tgh7H7cqbxzFcxEB3SV8zs+NmNtXi8bMl/Xfi/rPRtmHbp/Z/BL9hZo+Y2VfM7NfyLFQbZ7n7qej2dyWd1WKfon7OkvR+hW9prXQ7fvJ0XfT1+o42zQFF/Ix3S3re3Z9o8/hQP18z2yFpp6RvqgTHcVN5kzI5hsd6LWAOLnH358zsFyQ9YGbfjmoUhWVm6yX9nqT9LR5+WKEZ5sdRO+o9ks7Ls3yduLubWWm6OpnZDZJekvTZNrsU5fg5LOmgwh/nQYVmjPcPoRy9epc6186H9vma2WmSviDpI+7+YvgyERTxOG4ub2J7Zsdw4Wro7v5cdP2CpH9S+Fqa9JykcxL3XxttG6YrJD3s7s83P+DuL7r7j6Pb90taZ2Zb8y5gk+fjZqro+oUW+xTuczazP5C0V9J7PGpsbJbi+MmFuz/v7i+7+yuSPtWmHIX6jM1sTNI1kj7fbp9hfb5mtk4hHD/r7seizYU9jtuUN/NjuFCBbmabzOw18W2Fkwgnm3a7V9L7LHizpP9JfO0alra1GjP7xahdUmZ2scJn/v0cy9bKvZLis/3XSvpii32+KultZnZG1FzwtmjbUJjZ5ZJmJP2eu/+kzT5pjp9cNJ3XubpNORYlnWdmvxJ9y9un8LsZlrdK+ra7P9vqwWF9vtHfz+2SHnf3Q4mHCnkctytvLsdwlmd7+zg7/HqFM/2PSPqWpBui7X8s6Y+j2ybpbxV6BzwmaXzIZd6kENBbEtuS5b0u+lkeUTgR8ps5l+9zkk5J+plC++EHJP28pIckPSHpQUlnRvuOS/r7xHPfL+nJ6PKHQy7zkwptoSeiy99F+/6SpPs7HT9DKu9d0fH5qELwbG8ub3T/SoVeEE8Ns7zR9k/Hx21i3yJ8vpcoNF09mvj9X1nU47hDeTM/hhkpCgAVUagmFwBA/wh0AKgIAh0AKoJAB4CKINABoCIIdACoCAIdACqCQAeAivg/vMyNKB4jqBIAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(X[:,1], y, 'rx', label='Training data')\n",
"plt.plot(X[:,1], X.dot(theta), label='Linear regression')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Plotting the cost history\n",
"A plot of how $J(\\theta)$ decreases over time. This is are model learning."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x11438eeb8>]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAADqFJREFUeJzt23+o3fV9x/Hnq7k0axE00WitMbu2CiNu0MJBKdvA1V9x0EZa/7D7o2FryR+rf6yl0BTHtOof6tZZSruN0BZCYdXOURqQItFWGGNYT6yjzdo0t7HFpLZNjQhOqmR974/7dTufy4k3ud9z78nR5wMO93y/38+99/3xgs97zvcmVYUkSa9607QHkCSdWQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ15qY9wEqcd955NT8/P+0xJGmm7N+//9dVtWm5dTMZhvn5eYbD4bTHkKSZkuRnp7LOt5IkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpMZEwJNmW5GCShSS7xlxfn+SB7vrjSeaXXN+S5MUkn5zEPJKklesdhiTrgC8CNwBbgQ8l2bpk2UeA56vqUuA+4J4l1/8e+FbfWSRJ/U3iFcMVwEJVHa6qV4D7ge1L1mwH9nTPHwSuThKAJDcCTwMHJjCLJKmnSYThIuCZkeMj3bmxa6rqBPACcG6Ss4BPAZ+ZwBySpAmY9s3n24H7qurF5RYm2ZlkmGR47Nix1Z9Mkt6g5ibwNY4CF48cb+7OjVtzJMkccDbwHHAlcFOSe4FzgN8m+U1VfWHpN6mq3cBugMFgUBOYW5I0xiTC8ARwWZJLWAzAzcCfLVmzF9gB/AdwE/Dtqirgj19dkOR24MVxUZAkrZ3eYaiqE0luAR4G1gFfqaoDSe4AhlW1F/gy8NUkC8BxFuMhSToDZfEX99kyGAxqOBxOewxJmilJ9lfVYLl10775LEk6wxgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpMZEwJNmW5GCShSS7xlxfn+SB7vrjSea789cm2Z/k+93H905iHknSyvUOQ5J1wBeBG4CtwIeSbF2y7CPA81V1KXAfcE93/tfA+6rqD4AdwFf7ziNJ6mcSrxiuABaq6nBVvQLcD2xfsmY7sKd7/iBwdZJU1feq6ufd+QPAW5Ksn8BMkqQVmkQYLgKeGTk+0p0bu6aqTgAvAOcuWfNB4MmqenkCM0mSVmhu2gMAJLmcxbeXrnuNNTuBnQBbtmxZo8kk6Y1nEq8YjgIXjxxv7s6NXZNkDjgbeK473gx8A/hwVf3kZN+kqnZX1aCqBps2bZrA2JKkcSYRhieAy5JckuTNwM3A3iVr9rJ4cxngJuDbVVVJzgEeAnZV1b9PYBZJUk+9w9DdM7gFeBj4IfD1qjqQ5I4k7++WfRk4N8kC8Ang1T9pvQW4FPibJE91j/P7ziRJWrlU1bRnOG2DwaCGw+G0x5CkmZJkf1UNllvnv3yWJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIaEwlDkm1JDiZZSLJrzPX1SR7orj+eZH7k2qe78weTXD+JeSRJK9c7DEnWAV8EbgC2Ah9KsnXJso8Az1fVpcB9wD3d524FbgYuB7YB/9B9PUnSlEziFcMVwEJVHa6qV4D7ge1L1mwH9nTPHwSuTpLu/P1V9XJVPQ0sdF9PkjQlkwjDRcAzI8dHunNj11TVCeAF4NxT/FxJ0hqamZvPSXYmGSYZHjt2bNrjSNLr1iTCcBS4eOR4c3du7Jokc8DZwHOn+LkAVNXuqhpU1WDTpk0TGFuSNM4kwvAEcFmSS5K8mcWbyXuXrNkL7Oie3wR8u6qqO39z91dLlwCXAd+dwEySpBWa6/sFqupEkluAh4F1wFeq6kCSO4BhVe0Fvgx8NckCcJzFeNCt+zrwX8AJ4GNV9T99Z5IkrVwWf3GfLYPBoIbD4bTHkKSZkmR/VQ2WWzczN58lSWvDMEiSGoZBktQwDJKkhmGQJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpYRgkSY1eYUiyMcm+JIe6jxtOsm5Ht+ZQkh3dubcmeSjJj5IcSHJ3n1kkSZPR9xXDLuDRqroMeLQ7biTZCNwGXAlcAdw2EpC/q6rfA94N/GGSG3rOI0nqqW8YtgN7uud7gBvHrLke2FdVx6vqeWAfsK2qXqqq7wBU1SvAk8DmnvNIknrqG4YLqurZ7vkvgAvGrLkIeGbk+Eh37v8kOQd4H4uvOiRJUzS33IIkjwBvG3Pp1tGDqqokdboDJJkDvgZ8vqoOv8a6ncBOgC1btpzut5EknaJlw1BV15zsWpJfJrmwqp5NciHwqzHLjgJXjRxvBh4bOd4NHKqqzy0zx+5uLYPB4LQDJEk6NX3fStoL7Oie7wC+OWbNw8B1STZ0N52v686R5C7gbOCves4hSZqQvmG4G7g2ySHgmu6YJIMkXwKoquPAncAT3eOOqjqeZDOLb0dtBZ5M8lSSj/acR5LUU6pm712ZwWBQw+Fw2mNI0kxJsr+qBsut818+S5IahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJjV5hSLIxyb4kh7qPG06ybke35lCSHWOu703ygz6zSJImo+8rhl3Ao1V1GfBod9xIshG4DbgSuAK4bTQgST4AvNhzDknShPQNw3ZgT/d8D3DjmDXXA/uq6nhVPQ/sA7YBJDkL+ARwV885JEkT0jcMF1TVs93zXwAXjFlzEfDMyPGR7hzAncBngZd6ziFJmpC55RYkeQR425hLt44eVFUlqVP9xkneBbyzqj6eZP4U1u8EdgJs2bLlVL+NJOk0LRuGqrrmZNeS/DLJhVX1bJILgV+NWXYUuGrkeDPwGPAeYJDkp90c5yd5rKquYoyq2g3sBhgMBqccIEnS6en7VtJe4NW/MtoBfHPMmoeB65Js6G46Xwc8XFX/WFVvr6p54I+AH58sCpKktdM3DHcD1yY5BFzTHZNkkORLAFV1nMV7CU90jzu6c5KkM1CqZu9dmcFgUMPhcNpjSNJMSbK/qgbLrfNfPkuSGoZBktQwDJKkhmGQJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGqmqac9w2pIcA3427TlO03nAr6c9xBpzz28M7nl2/G5VbVpu0UyGYRYlGVbVYNpzrCX3/Mbgnl9/fCtJktQwDJKkhmFYO7unPcAUuOc3Bvf8OuM9BklSw1cMkqSGYZigJBuT7EtyqPu44STrdnRrDiXZMeb63iQ/WP2J++uz5yRvTfJQkh8lOZDk7rWd/vQk2ZbkYJKFJLvGXF+f5IHu+uNJ5keufbo7fzDJ9Ws5dx8r3XOSa5PsT/L97uN713r2lejzM+6ub0nyYpJPrtXMq6KqfEzoAdwL7Oqe7wLuGbNmI3C4+7ihe75h5PoHgH8GfjDt/az2noG3An/SrXkz8G/ADdPe00n2uQ74CfCObtb/BLYuWfOXwD91z28GHuieb+3Wrwcu6b7OumnvaZX3/G7g7d3z3weOTns/q7nfkesPAv8CfHLa++nz8BXDZG0H9nTP9wA3jllzPbCvqo5X1fPAPmAbQJKzgE8Ad63BrJOy4j1X1UtV9R2AqnoFeBLYvAYzr8QVwEJVHe5mvZ/FvY8a/W/xIHB1knTn76+ql6vqaWCh+3pnuhXvuaq+V1U/784fAN6SZP2aTL1yfX7GJLkReJrF/c40wzBZF1TVs93zXwAXjFlzEfDMyPGR7hzAncBngZdWbcLJ67tnAJKcA7wPeHQ1hpyAZfcwuqaqTgAvAOee4ueeifrsedQHgSer6uVVmnNSVrzf7pe6TwGfWYM5V93ctAeYNUkeAd425tKtowdVVUlO+U++krwLeGdVfXzp+5bTtlp7Hvn6c8DXgM9X1eGVTakzUZLLgXuA66Y9yyq7Hbivql7sXkDMNMNwmqrqmpNdS/LLJBdW1bNJLgR+NWbZUeCqkePNwGPAe4BBkp+y+HM5P8ljVXUVU7aKe37VbuBQVX1uAuOulqPAxSPHm7tz49Yc6WJ3NvDcKX7umajPnkmyGfgG8OGq+snqj9tbn/1eCdyU5F7gHOC3SX5TVV9Y/bFXwbRvcryeHsDf0t6IvXfMmo0svg+5oXs8DWxcsmae2bn53GvPLN5P+VfgTdPeyzL7nGPxpvkl/P+NycuXrPkY7Y3Jr3fPL6e9+XyY2bj53GfP53TrPzDtfazFfpesuZ0Zv/k89QFeTw8W31t9FDgEPDLyP78B8KWRdX/B4g3IBeDPx3ydWQrDivfM4m9kBfwQeKp7fHTae3qNvf4p8GMW/3Ll1u7cHcD7u+e/w+JfpCwA3wXeMfK5t3afd5Az9C+vJrln4K+B/x75uT4FnD/t/azmz3jka8x8GPyXz5Kkhn+VJElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJjf8FFDYZsBaypoYAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(J_history)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Making a prediction using the model\n",
"The model can be used by calculating the dot product of the input and $\\theta$."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'In a city with a population of 35000, we predict a profit of $0.00'"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prediction = np.array([1, 3.5]).dot(theta) * 10000 # don't forget to multiply the prediction by 10000\n",
"'In a city with a population of 35000, we predict a profit of $%.2f' % prediction"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Multivariate Linear Regression\n",
"\n",
"---\n",
"In this part, you will implement linear regression with multiple variables to predict the prices of houses. Suppose you are selling your house and you want to know what a good market price would be. One way to do this is to first collect information on recent houses sold and make a model of housing prices."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Size</th>\n",
" <th>Bedrooms</th>\n",
" <th>Price</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2104</td>\n",
" <td>3</td>\n",
" <td>399900</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1600</td>\n",
" <td>3</td>\n",
" <td>329900</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2400</td>\n",
" <td>3</td>\n",
" <td>369000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1416</td>\n",
" <td>2</td>\n",
" <td>232000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>3000</td>\n",
" <td>4</td>\n",
" <td>539900</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Size Bedrooms Price\n",
"0 2104 3 399900\n",
"1 1600 3 329900\n",
"2 2400 3 369000\n",
"3 1416 2 232000\n",
"4 3000 4 539900"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# load data\n",
"data = pd.read_csv(\"ex1data2.txt\", header = None, names=[\"Size\", \"Bedrooms\",\"Price\"])\n",
"m = len(data)\n",
"\n",
"# Initialize X, y and theta\n",
"x0 = np.ones(m)\n",
"size = np.array((data[\"Size\"]))\n",
"bedrooms = np.array((data[\"Bedrooms\"]))\n",
"X = np.array([x0, size, bedrooms]).T\n",
"y = np.array(data[\"Price\"]).reshape(len(data.index), 1)\n",
"theta_init = np.zeros((3,1))\n",
"\n",
"data.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Feature Normalization\n",
"When features differ by order of magnitude, first performing feature scaling can make gradient descent converge much more quickly. Formally:\n",
"\n",
"$x := \\frac{x - \\mu}{\\sigma}$\n",
"\n",
"Where $\\mu$ is the average and $\\sigma$ the standard deviation.\n",
"\n",
"**Important**: It is crucial to store $\\mu$ and $\\sigma$ if you want to make predictions using the model later.\n",
"\n",
"**Exercise**: Perform feature normalization on the following dataset."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1.000e+00, 2.104e+03, 3.000e+00],\n",
" [1.000e+00, 1.600e+03, 3.000e+00],\n",
" [1.000e+00, 2.400e+03, 3.000e+00],\n",
" [1.000e+00, 1.416e+03, 2.000e+00],\n",
" [1.000e+00, 3.000e+03, 4.000e+00]])"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# perform normalization\n",
"def normalize(X):\n",
" \"\"\" Normalizes the features in X\n",
" \n",
" returns a normalized version of X where\n",
" the mean value of each feature is 0 and the standard deviation\n",
" is 1. This is often a good preprocessing step to do when\n",
" working with learning algorithms.\n",
" \"\"\"\n",
" mu = np.zeros(len(X))\n",
" sigma = np.zeros(len(X))\n",
" \n",
" return X, mu, sigma\n",
"\n",
"X, mu, sigma = normalize(X)\n",
"X[0:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Gradient Descent\n",
"\n",
"Remember the algorithm for gradient descent:\n",
"\n",
"$repeat \\ \\{ \\\\ \\enspace \\theta_j := \\theta_j - \\alpha \\frac{1}{m}\\displaystyle\\sum_{i = 1}^{m}(h_\\theta(x^{(i)})-y^{(i)})x_j^{(i)}\\\\\\}$\n",
"\n",
"The vectorization for multivariate gradient descent:\n",
"\n",
"$\\theta := \\theta - \\frac{\\alpha}{m}X^T(X\\theta - \\vec{y})$\n",
"\n",
"**Exercise**: Implement gradient descent for multiple features. Make sure your solution is vectorized and supports any number of features."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.],\n",
" [0.],\n",
" [0.]])"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def gradient_descent_multi(X, y, theta, alpha, iterations):\n",
" J_history = []\n",
" return theta, J_history\n",
"\n",
"alpha = 0.01\n",
"iterations = 1500\n",
"initial_theta = np.zeros((3,1))\n",
"theta, J_history = gradient_descent_multi(X, y, initial_theta, alpha, iterations)\n",
"theta"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As before we see how the cost decreases over time."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Text(0, 0.5, 'cost')"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEWCAYAAABBvWFzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAFYFJREFUeJzt3X+0XWV95/H3x0SpDjUkEBAIMSjM1NBpYdYt1FXtUEF+OKNhIY7YdhlHWUw70i5xuWqQLkHUDmgpanWmk1FH/AUIDm1mtKYRy9RhLHKDYImICaEMhACBRGykgpHv/HF29HB7knvJfe49ubnv11pnnb2f/ey9vw93kc/d+zln31QVkiRN1rOGXYAkad9goEiSmjBQJElNGCiSpCYMFElSEwaKJKkJA0WaQkneleTjQ67hL5MsH2YNmh3i91A0GyS5EfhsVQ3tH/ckS4B7gGdX1Y4pOsfFwFFV9dtTcXxpd7xCkRpJMncmH1+aLANFs1qSNyW5KclHkzyW5LtJTurbPi/JJ5JsTrIpyfuSzBmz7xVJHgUuHnD8i5N8tlv9m+79+0m2J3lp1+fNSe5Msi3J6iQv7Nu/krw1yXpgfdf24ST3JflBkrVJXt61nwa8C3h9d/zbu/Ybk5zTLT8ryR8muTfJw0k+nWRet21Jd77lSf5fkkeSXNjwP7f2cQaKBCcAdwMHARcB/yPJgm7bp4AdwFHAccApwDlj9t0IHAK8f5zz/Hr3fkBV7V9V30iyjF4InAksBL4OXDVmvzO68yzt1m8BjgUWAJ8Hrk3yc1X1FeCPgGu64//ygBre1L1+A3gRsD/w0TF9Xgb8C+Ak4N1JXjLOuCTAQJEAHgY+VFU/rqprgLuAf5PkEOBVwNuq6odV9TBwBXB2374PVNWfVtWOqvrHPTj37wD/qaru7OZV/gg4tv8qpdu+defxq+qzVfVod87Lgf3oBcBE/BbwJ1W1saq2AxcAZ4+5nfaeqvrHqroduB0YFEzSP2GgSLCpnv7plHuBw4AXAs8GNif5fpLvA/8VOLiv732TPPcLgQ/3HX8rEODwXZ0jyTu6W2SPdfvMo3d1NRGH0RvfTvcCc+ldYe30YN/y4/SuYqRxOcknweFJ0hcqi4FV9P4hfwI4aDefynomH5Mc1Pc+4P1V9bmJ7NfNl/wBvdtR66rqqSTb6IXQROp5gF6I7bSY3i29h4BF4+wr7ZZXKFLviuP3kzw7yeuAlwBfrqrNwF8Blyd5fjeh/eIk/3oPz7MFeIre3MVOfwZckOQY+OmHAF63m2P8PL0A2ALMTfJu4Pl92x8CliTZ1f/bVwHnJzkyyf78bM5lSj7GrNnFQNFssqvf3m8GjgYeoTexflZVPdpteyPwHOA7wDbgOuDQPTp51ePd8W/qbnH9alVdD1wGXJ3kB8AdwOm7Ocxq4CvA9+jdrvoRT78ldm33/miSWwfs/0ngM/Q+cXZPt//v7cl4pLH8YqNmhe4f10uq6s/HtL8JOKeqXjaUwqR9iFco2ud1t5NeAnxr2LVI+zIDRfu0JJfRmwd5Z1XdO15/SXvOW16SpCa8QpEkNTGrvody0EEH1ZIlS4ZdhiTNKGvXrn2kqhaO129WBcqSJUsYHR0ddhmSNKMkmdD8o7e8JElNGCiSpCYMFElSEwaKJKkJA0WS1ISBIklqwkCRJDVhoEiSmjBQJElNGCiSpCYMFElSEwaKJKkJA0WS1ISBIklqwkCRJDVhoEiSmjBQJElNGCiSpCYMFElSEwaKJKkJA0WS1ISBIklqwkCRJDVhoEiSmjBQJElNDDVQkpyW5K4kG5KsGLB9vyTXdNtvTrJkzPbFSbYnecd01SxJGmxogZJkDvAx4HRgKfCGJEvHdHsLsK2qjgKuAC4bs/1PgL+c6lolSeMb5hXK8cCGqtpYVU8CVwPLxvRZBlzZLV8HnJQkAEnOAO4B1k1TvZKk3RhmoBwO3Ne3fn/XNrBPVe0AHgMOTLI/8E7gPeOdJMm5SUaTjG7ZsqVJ4ZKkf2qmTspfDFxRVdvH61hVK6tqpKpGFi5cOPWVSdIsNXeI594EHNG3vqhrG9Tn/iRzgXnAo8AJwFlJPgAcADyV5EdV9dGpL1uSNMgwA+UW4OgkR9ILjrOB3xzTZxWwHPgGcBbwtaoq4OU7OyS5GNhumEjScA0tUKpqR5LzgNXAHOCTVbUuySXAaFWtAj4BfCbJBmArvdCRJO2F0vuFf3YYGRmp0dHRYZchSTNKkrVVNTJev5k6KS9J2ssYKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWpiqIGS5LQkdyXZkGTFgO37Jbmm235zkiVd+yuTrE3yd937K6a7dknS0w0tUJLMAT4GnA4sBd6QZOmYbm8BtlXVUcAVwGVd+yPAq6vqXwLLgc9MT9WSpF0Z5hXK8cCGqtpYVU8CVwPLxvRZBlzZLV8HnJQkVfWtqnqga18HPDfJftNStSRpoGEGyuHAfX3r93dtA/tU1Q7gMeDAMX1eC9xaVU9MUZ2SpAmYO+wCJiPJMfRug52ymz7nAucCLF68eJoqk6TZZ5hXKJuAI/rWF3VtA/skmQvMAx7t1hcB1wNvrKq7d3WSqlpZVSNVNbJw4cKG5UuS+g0zUG4Bjk5yZJLnAGcDq8b0WUVv0h3gLOBrVVVJDgC+BKyoqpumrWJJ0i4NLVC6OZHzgNXAncAXqmpdkkuSvKbr9gngwCQbgLcDOz9afB5wFPDuJLd1r4OneQiSpD6pqmHXMG1GRkZqdHR02GVI0oySZG1VjYzXz2/KS5KaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU1MKFCSvG4ibZKk2WuiVygXTLBNkjRLzd3dxiSnA68CDk/ykb5Nzwd2TGVhkqSZZbeBAjwAjAKvAdb2tf8DcP5UFSVJmnl2GyhVdTtwe5LPV9WPAZLMB46oqm3TUaAkaWaY6BzKmiTPT7IAuBX4b0mumOzJk5yW5K4kG5KsGLB9vyTXdNtvTrKkb9sFXftdSU6dbC2SpMmZaKDMq6ofAGcCn66qE4CTJnPiJHOAjwGnA0uBNyRZOqbbW4BtVXUUcAVwWbfvUuBs4BjgNOA/d8eTJA3JRANlbpJDgX8H/K9G5z4e2FBVG6vqSeBqYNmYPsuAK7vl64CTkqRrv7qqnqiqe4AN3fEkSUMy0UC5BFgN3F1VtyR5EbB+kuc+HLivb/3+rm1gn6raATwGHDjBfQFIcm6S0SSjW7ZsmWTJkqRdmVCgVNW1VfVLVfW73frGqnrt1JbWRlWtrKqRqhpZuHDhsMuRpH3WRL8pvyjJ9Uke7l5fTLJokufeBBzRt76oaxvYJ8lcYB7w6AT3lSRNo4ne8vrvwCrgsO71P7u2ybgFODrJkUmeQ2+SfdWYPquA5d3yWcDXqqq69rO7T4EdCRwNfHOS9UiSJmG8LzbutLCq+gPkU0neNpkTV9WOJOfRm5uZA3yyqtYluQQYrapVwCeAzyTZAGylFzp0/b4AfIfeN/bfWlU/mUw9kqTJmWigPJrkt4GruvU30Lv1NClV9WXgy2Pa3t23/CNg4EMoq+r9wPsnW4MkqY2J3vJ6M72PDD8IbKZ3++lNU1STJGkGmugVyiXA8p2PW+m+Mf/H9IJGkqQJX6H8Uv+zu6pqK3Dc1JQkSZqJJhooz+oeCgn89Aplolc3kqRZYKKhcDnwjSTXduuvwwlxSVKfCQVKVX06ySjwiq7pzKr6ztSVJUmaaSZ826oLEENEkjTQROdQJEnaLQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmhhIoSRYkWZNkffc+fxf9lnd91idZ3rU9L8mXknw3ybokl05v9ZKkQYZ1hbICuKGqjgZu6NafJskC4CLgBOB44KK+4PnjqvoF4Djg15KcPj1lS5J2ZViBsgy4slu+EjhjQJ9TgTVVtbWqtgFrgNOq6vGq+muAqnoSuBVYNA01S5J2Y1iBckhVbe6WHwQOGdDncOC+vvX7u7afSnIA8Gp6VzmSpCGaO1UHTvJV4AUDNl3Yv1JVlaT24PhzgauAj1TVxt30Oxc4F2Dx4sXP9DSSpAmaskCpqpN3tS3JQ0kOrarNSQ4FHh7QbRNwYt/6IuDGvvWVwPqq+tA4dazs+jIyMvKMg0uSNDHDuuW1CljeLS8H/mJAn9XAKUnmd5Pxp3RtJHkfMA942zTUKkmagGEFyqXAK5OsB07u1kkykuTjAFW1FXgvcEv3uqSqtiZZRO+22VLg1iS3JTlnGIOQJP1MqmbPXaCRkZEaHR0ddhmSNKMkWVtVI+P185vykqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpoYSqAkWZBkTZL13fv8XfRb3vVZn2T5gO2rktwx9RVLksYzrCuUFcANVXU0cEO3/jRJFgAXAScAxwMX9QdPkjOB7dNTriRpPMMKlGXAld3ylcAZA/qcCqypqq1VtQ1YA5wGkGR/4O3A+6ahVknSBAwrUA6pqs3d8oPAIQP6HA7c17d+f9cG8F7gcuDx8U6U5Nwko0lGt2zZMomSJUm7M3eqDpzkq8ALBmy6sH+lqipJPYPjHgu8uKrOT7JkvP5VtRJYCTAyMjLh80iSnpkpC5SqOnlX25I8lOTQqtqc5FDg4QHdNgEn9q0vAm4EXgqMJPl7evUfnOTGqjoRSdLQDOuW1ypg56e2lgN/MaDPauCUJPO7yfhTgNVV9V+q6rCqWgK8DPieYSJJwzesQLkUeGWS9cDJ3TpJRpJ8HKCqttKbK7mle13StUmS9kKpmj3TCiMjIzU6OjrsMiRpRkmytqpGxuvnN+UlSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaSFUNu4Zpk2QLcO+w63iGDgIeGXYR08wxzw6OeeZ4YVUtHK/TrAqUmSjJaFWNDLuO6eSYZwfHvO/xlpckqQkDRZLUhIGy91s57AKGwDHPDo55H+MciiSpCa9QJElNGCiSpCYMlL1AkgVJ1iRZ373P30W/5V2f9UmWD9i+KskdU1/x5E1mzEmel+RLSb6bZF2SS6e3+mcmyWlJ7kqyIcmKAdv3S3JNt/3mJEv6tl3Qtd+V5NTprHsy9nTMSV6ZZG2Sv+veXzHdte+JyfyMu+2Lk2xP8o7pqnlKVJWvIb+ADwAruuUVwGUD+iwANnbv87vl+X3bzwQ+D9wx7PFM9ZiB5wG/0fV5DvB14PRhj2kX45wD3A28qKv1dmDpmD7/Efizbvls4JpueWnXfz/gyO44c4Y9pike83HAYd3yLwKbhj2eqRxv3/brgGuBdwx7PJN5eYWyd1gGXNktXwmcMaDPqcCaqtpaVduANcBpAEn2B94OvG8aam1lj8dcVY9X1V8DVNWTwK3AommoeU8cD2yoqo1drVfTG3u//v8W1wEnJUnXfnVVPVFV9wAbuuPt7fZ4zFX1rap6oGtfBzw3yX7TUvWem8zPmCRnAPfQG++MZqDsHQ6pqs3d8oPAIQP6HA7c17d+f9cG8F7gcuDxKauwvcmOGYAkBwCvBm6YiiIbGHcM/X2qagfwGHDgBPfdG01mzP1eC9xaVU9MUZ2t7PF4u18G3wm8ZxrqnHJzh13AbJHkq8ALBmy6sH+lqirJhD/LneRY4MVVdf7Y+7LDNlVj7jv+XOAq4CNVtXHPqtTeKMkxwGXAKcOuZYpdDFxRVdu7C5YZzUCZJlV18q62JXkoyaFVtTnJocDDA7ptAk7sW18E3Ai8FBhJ8vf0fp4HJ7mxqk5kyKZwzDutBNZX1YcalDtVNgFH9K0v6toG9bm/C8l5wKMT3HdvNJkxk2QRcD3wxqq6e+rLnbTJjPcE4KwkHwAOAJ5K8qOq+ujUlz0Fhj2J46sAPsjTJ6g/MKDPAnr3Wed3r3uABWP6LGHmTMpPasz05ou+CDxr2GMZZ5xz6X2Y4Eh+NmF7zJg+b+XpE7Zf6JaP4emT8huZGZPykxnzAV3/M4c9jukY75g+FzPDJ+WHXoCvgt694xuA9cBX+/7RHAE+3tfvzfQmZjcA/37AcWZSoOzxmOn9BljAncBt3eucYY9pN2N9FfA9ep8EurBruwR4Tbf8c/Q+4bMB+Cbwor59L+z2u4u99JNsLccM/CHww76f623AwcMez1T+jPuOMeMDxUevSJKa8FNekqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkfZAkv/bvS9J8puNj/2uQeeS9nZ+bFiahCQn0vvuwL99BvvMrd7znHa1fXtV7d+iPmk6eYUi7YEk27vFS4GXJ7ktyflJ5iT5YJJbknw7yX/o+p+Y5OtJVgHf6dr+vPubH+uSnNu1XUrvCbu3Jflc/7nS88Ekd3R/L+T1fce+Mcl13d+I+dzOJ9lK08lneUmTs4K+K5QuGB6rql/pHrt+U5K/6vr+K+AXq/coeoA3V9XWJM8FbknyxapakeS8qjp2wLnOBI4Ffhk4qNvnb7ptx9F7VMsDwE3ArwH/p/1wpV3zCkVq6xTgjUluA26m94iZo7tt3+wLE4DfT3I78Lf0Hhx4NLv3MuCqqvpJVT0E/G/gV/qOfX9VPUXvcSVLmoxGega8QpHaCvB7VbX6aY29uZYfjlk/GXhpVT2e5EZ6z3vaU/1/M+Qn+P+2hsArFGly/gH4+b711cDvJnk2QJJ/nuSfDdhvHrCtC5NfAH61b9uPd+4/xteB13fzNAuBX6f3oEFpr+BvMdLkfBv4SXfr6lPAh+ndbrq1mxjfwuA/b/wV4HeS3EnvScJ/27dtJfDtJLdW1W/1tV9P7+/f3E7vact/UFUPdoEkDZ0fG5YkNeEtL0lSEwaKJKkJA0WS1ISBIklqwkCRJDVhoEiSmjBQJElN/H+7X9GYyN1Q+AAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(J_history)\n",
"plt.title('J per iteration')\n",
"plt.xlabel('iteration')\n",
"plt.ylabel('cost')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we want to make a prediction on a normalized dataset, we have to normalize our input too."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:1: RuntimeWarning: divide by zero encountered in double_scalars\n",
" \"\"\"Entry point for launching an IPython kernel.\n"
]
},
{
"data": {
"text/plain": [
"'In a house of 1650 square feet with 3 rooms, we predict a price of $nan'"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"price = theta.transpose() @ np.array([1, (1650-mu[1])/sigma[1], (3-mu[2])/sigma[2]]) # normalize the input\n",
"'In a house of 1650 square feet with 3 rooms, we predict a price of $%.2f' % price"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using normal equations\n",
"We can use normal equations to get the exact solution in only one calculation. Although using normal equations is very fast for a small datasets with a small number of features, it can be inefficient for larger datasets because the complexity of matrix multiplication is $O(n^3)$.\n",
"\n",
"The normal equation for linear regression is:\n",
"\n",
"$\\theta = (X^TX)^{−1}X^T\\vec{y}$\n",
"\n",
"**Exercise**: Find theta using normal equations."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[89597.9095428 ],\n",
" [ 139.21067402],\n",
" [-8738.01911233]])"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"theta"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:1: RuntimeWarning: divide by zero encountered in double_scalars\n",
" \"\"\"Entry point for launching an IPython kernel.\n"
]
},
{
"data": {
"text/plain": [
"'In a house of 1650 square feet with 3 rooms, we predict a price of $nan'"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"price = theta.transpose() @ np.array([1, (1650-mu[1])/sigma[1], (3-mu[2])/sigma[2]]) # normalize the input\n",
"'In a house of 1650 square feet with 3 rooms, we predict a price of $%.2f' % price"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: ex1/ex1data1.txt
================================================
6.1101,17.592
5.5277,9.1302
8.5186,13.662
7.0032,11.854
5.8598,6.8233
8.3829,11.886
7.4764,4.3483
8.5781,12
6.4862,6.5987
5.0546,3.8166
5.7107,3.2522
14.164,15.505
5.734,3.1551
8.4084,7.2258
5.6407,0.71618
5.3794,3.5129
6.3654,5.3048
5.1301,0.56077
6.4296,3.6518
7.0708,5.3893
6.1891,3.1386
20.27,21.767
5.4901,4.263
6.3261,5.1875
5.5649,3.0825
18.945,22.638
12.828,13.501
10.957,7.0467
13.176,14.692
22.203,24.147
5.2524,-1.22
6.5894,5.9966
9.2482,12.134
5.8918,1.8495
8.2111,6.5426
7.9334,4.5623
8.0959,4.1164
5.6063,3.3928
12.836,10.117
6.3534,5.4974
5.4069,0.55657
6.8825,3.9115
11.708,5.3854
5.7737,2.4406
7.8247,6.7318
7.0931,1.0463
5.0702,5.1337
5.8014,1.844
11.7,8.0043
5.5416,1.0179
7.5402,6.7504
5.3077,1.8396
7.4239,4.2885
7.6031,4.9981
6.3328,1.4233
6.3589,-1.4211
6.2742,2.4756
5.6397,4.6042
9.3102,3.9624
9.4536,5.4141
8.8254,5.1694
5.1793,-0.74279
21.279,17.929
14.908,12.054
18.959,17.054
7.2182,4.8852
8.2951,5.7442
10.236,7.7754
5.4994,1.0173
20.341,20.992
10.136,6.6799
7.3345,4.0259
6.0062,1.2784
7.2259,3.3411
5.0269,-2.6807
6.5479,0.29678
7.5386,3.8845
5.0365,5.7014
10.274,6.7526
5.1077,2.0576
5.7292,0.47953
5.1884,0.20421
6.3557,0.67861
9.7687,7.5435
6.5159,5.3436
8.5172,4.2415
9.1802,6.7981
6.002,0.92695
5.5204,0.152
5.0594,2.8214
5.7077,1.8451
7.6366,4.2959
5.8707,7.2029
5.3054,1.9869
8.2934,0.14454
13.394,9.0551
5.4369,0.61705
================================================
FILE: ex1/ex1data2.txt
================================================
2104,3,399900
1600,3,329900
2400,3,369000
1416,2,232000
3000,4,539900
1985,4,299900
1534,3,314900
1427,3,198999
1380,3,212000
1494,3,242500
1940,4,239999
2000,3,347000
1890,3,329999
4478,5,699900
1268,3,259900
2300,4,449900
1320,2,299900
1236,3,199900
2609,4,499998
3031,4,599000
1767,3,252900
1888,2,255000
1604,3,242900
1962,4,259900
3890,3,573900
1100,3,249900
1458,3,464500
2526,3,469000
2200,3,475000
2637,3,299900
1839,2,349900
1000,1,169900
2040,4,314900
3137,3,579900
1811,4,285900
1437,3,249900
1239,3,229900
2132,4,345000
4215,4,549000
2162,4,287000
1664,2,368500
2238,3,329900
2567,4,314000
1200,3,299000
852,2,179900
1852,4,299900
1203,3,239500
================================================
FILE: ex2/PE2 - Logistic Regression (Exercises).ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Linear Regression\n",
"\n",
"Stanford CS229 - Machine Learning by Andrew Ng. Programming exercise 2 with solutions.\n",
"\n",
"Please check out [the repository on GitHub](https://github.com/rickwierenga/CS229-Python/). If you spot any mistakes or inconcistencies, please create an issue. For questions you can find me on Twitter: [@rickwierenga](https://twitter.com/rickwierenga). Starring the project on GitHub means a ton to me!"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pylab as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Logistic Regression\n",
"\n",
"---\n",
"Logistic regression is predicting in which category a given data point is. In binary classifiction, there are only two categories:\n",
"\n",
"$$y \\in \\{0,1\\}$$\n",
"\n",
"In this exercise, you will implement logistic regression and apply it to two different datasets. Before starting on the programming exercise, we strongly recommend watching the video lectures and completing the review questions for the associated topics."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 1. , 34.62365962, 78.02469282],\n",
" [ 1. , 30.28671077, 43.89499752],\n",
" [ 1. , 35.84740877, 72.90219803],\n",
" [ 1. , 60.18259939, 86.3085521 ],\n",
" [ 1. , 79.03273605, 75.34437644]])"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# start by loading the data\n",
"data = pd.read_csv(\"ex2data1.txt\", header = None, \n",
" names = [\"Exam 1 Score\", \"Exam 2 Score\", \"Accepted\"])\n",
"\n",
"# initialize some useful variables\n",
"m = len(data[\"Accepted\"])\n",
"x0 = np.ones(m)\n",
"size = np.array((data[\"Exam 1 Score\"]))\n",
"bedrooms = np.array((data[\"Exam 2 Score\"]))\n",
"X = np.array([x0, size, bedrooms]).T\n",
"y = np.array(data[\"Accepted\"]).reshape((m,1))\n",
"m, n = X.shape\n",
"\n",
"X[:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualising the data\n",
"A blue cross means accepted. A yellow oval means rejected."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3XuQVOW57/HvA+gQCAmC7AlKYCAiKiKjjCYTjRLxlsSo2V6i4Sgasymr1O01Xso6MqSOVVqmyq0mFUPCVk4K0UjhJSbmqETijZgMhgiCbA0OiMIwQSHBCwrznD969TgMPZe+rnet/n2qpqZ7TU+vp9f0POvt570sc3dERCS9+sUdgIiIlJcSvYhIyinRi4iknBK9iEjKKdGLiKScEr2ISMop0YuIpJwSvYhIyinRi4ik3IC4AwDYd999va6uLu4wREQSZdmyZf9w9xG9PS6IRF9XV0dzc3PcYYiIJIqZrevL41S6ERFJuV4TvZn9t5ltNrOVnbYNM7OnzOz16Ps+0XYzs7vM7A0ze8XMjihn8CIi0ru+tOjvA07psu0GYLG7jwcWR/cBvgGMj75mAj8rTZgiIlKoXhO9uz8LvNtl8+nAvOj2POCMTtv/r2f8CRhqZiNLFayIiOSv0Bp9rbtvjG5vAmqj2/sDb3V63IZom4iIxKTozljPXLkk76uXmNlMM2s2s+a2trZiwyhIa+t8li6tY8mSfixdWkdr6/xY4hARKadCE31rtiQTfd8cbX8b+GKnx42Ktu3B3ee4e4O7N4wY0esw0JJrbZ3PmjUz2bFjHeDs2LGONWtmKtmLSOoUmugfA2ZEt2cAj3bafkE0+uYrwLZOJZ6grF17E+3tH+y2rb39A9auvSmmiEREyqPXCVNmtgCYCuxrZhuAWcCtwK/N7GJgHXBO9PDfAd8E3gA+AC4qQ8wlsWPH+ry2i4gkVV9G3Zzn7iPdfS93H+Xuc919i7tPc/fx7n6Cu78bPdbd/VJ3/5K7T3L3YKe71tSMzmt7qTU1VWQ3Ehj93SUOVTszdty4W+jXb9Bu2/r1G8S4cbdUZP+zZ1dkNxIY/d0lDlWb6GtrpzNhwhxqasYARk3NGCZMmENt7fS4Q5MyUotaqlHVJnrIJPvGxhamTm2nsbGl7Em+qQnMMl/w6W0ln/LpemzjaFEn6e8eYkxSPMsMg49XQ0ODV9vqlWYQwKFPva7HOe7jXsj+m5oql4DjPj6SHzNb5u4NvT2uqlv0Uh2S1KLORXV9KVZVJPoQZ8DOmhV3BOnVNbFnE2X2mLtnvuJI9CH+3ZN+IpTepb50k50B23lyVL9+g9TxWiVCK930VVNT7pb8rFnlTcBJOT6SodJNRDNgpbMQW9S5NDV9+skD4v0UIsmX+kSvGbDVrWtiV6LsWVJOhJKf1Cf6uGfASrzSkNgrmXzTcLxkT6lP9HHPgBUplpKvFCv1iV4zYEWk2vW6emUa1NZOV2KXYFVyQpRUp9S36EVCpwlRUm5K9CIiKadELxKDJM5GDTk26VnqZ8aKhC4ps1GTEmc10cxYEREBlOhFYhfybNQklphkTyrdiEifZJN9AClDIhUp3ZjZFWa20sxeNbMro23DzOwpM3s9+r5PMfsQkfypxS2dFZzozexQ4D+Ao4DJwKlmdgBwA7DY3ccDi6P7IlJBpRybny3fZKl8kzzFtOgPBl5y9w/cfSfwR+DfgdOBedFj5gFnFBeiiPRFuRKvlkxOvmIS/Urga2Y23MwGAd8EvgjUuvvG6DGbgNoiY5SEU0KojNmz1XEquRW81o27rzaz24AngfeB5cCuLo9xM8vZdWNmM4GZAKNHa8ngNJs9W8mmUrKt7nKNeQ95hJB0r6jOWHef6+5T3P1Y4D3gf4BWMxsJEH3f3M3vznH3BndvGDFiRDFhSB8p2aZPd8Mfy7m/pEvDa8hXsaNu/i36PppMff5+4DFgRvSQGcCjxexDSqeSi2dp/HVldFc/V8u7e9W4iFxR4+jN7DlgOPAJcLW7Lzaz4cCvgdHAOuAcd3+3p+fROPrKiGsKu6bOV4aOc9+k6ThVZBy9u3/N3Q9x98nuvjjatsXdp7n7eHc/obckL+WllnX16K4VX81/6+xrr/b/A82MDVQ5LkYRV0tGF9aIV5pasPnK9drTdDy0qFnCpamOqCQv1SqU974SfRVRB131qOZSRW+vvZL/B6E02FS6CUhTU+43xqxZ1fEPWgoqE+0pTaWKfMX92su9f5VuEkhTzYsXSgsqbnrPxCfET1NK9CJlFNc/d+cTXjWX7OJ47SE22JToA1XN/5z5CrEFlRXCJ4wQjkNcqvm1d6ZEHyi9QfsuxBZUpWX7JkI94VWrUBps6oyVVIm78w3i6VTv+rpDOA5Sfn3tjC149UqREIXQguo88kcJV0Kg0o2kSjWVKXoq1YRwwpNwqEUvUkblTLj65CB9pRa9SBlV0ycMCZcSvUgKqFQjPVGiF0kBfXKQnijRi4iknBK9iEjKKdGLiKScEr1IF6p3S9oo0Yt0EcJCZCKlVFSiN7OrzOxVM1tpZgvMbKCZjTWzl8zsDTN70Mz2LlWwIiKSv4ITvZntD/wn0ODuhwL9gXOB24A73P0A4D3g4lIEKlJOWvlR0qzY0s0A4DNmNgAYBGwEjgcWRj+fB5xR5D5Eyk5LHUuaFZzo3f1t4MfAejIJfhuwDNjq7jujh20A9i82SBERKVwxpZt9gNOBscB+wGDglDx+f6aZNZtZc1tbW6FhpEpr63yWLq1jyZJ+LF1aR2vr/LhDqkpaTkDSppjSzQnAm+7e5u6fAIuAo4GhUSkHYBTwdq5fdvc57t7g7g0jRowoIox0aG2dz5o1M9mxYx3g7NixjjVrZirZxyCEck0IMUh6FJPo1wNfMbNBZmbANGAV8AxwVvSYGcCjxYVYHdauvYn29g9229be/gFr194UU0QSJw3xlFIqpkb/EplO15eBFdFzzQGuB642szeA4cDcEsRZdnGXTXbsWJ/XdikftaYlbYoadePus9z9IHc/1N3Pd/cd7r7W3Y9y9wPc/Wx331GqYMslhLJJTc3ovLZL+cTVmtYQTykXzYwljLLJuHG30K/foN229es3iHHjbqlYDBIvDfEMT1qOvRI9YZRNamunM2HCHGpqxgBGTc0YJkyYQ23t9IrFUM3UmpZc0tJXomvGkimPZMo2e26vpNra6UrsMQnt+qtJHeLZ+ThKONSip7xlk7g7eSWZkpos09ACTuOnOyV6ylc2CaGTtxRCfIOXM6aktqalNNLYV2Ie92dUoKGhwZubm+MOo+SWLq3rpiQ0hsbGlsoHVKAQShldhRhTmuRTgmlqyt2SnzUr2ckRwn+fmdkyd2/o7XFV36IvZ2klhE5ekULkU4JJYws4Ky2f7qo60Ze7tJLksfEh1ilDjEnSLS3vrapO9OUeP5/ksfEhttJCjClNSnEiTUsLOG2qOtGXu7SisfGSJD2dSPOp10t4qnocfSXGz6dhbHyIrbQQY0qz2bOVxJOsqlv0SS6tVFKI/+AhxpQmOpGmS1Un+r6UVjThKXl0Eihetlyjzu900Dj6HmRH5XTusO3Xb5Dq7IHrOvZZ0/KLF/p48mqlcfQlEMKqllK8NEzLl/SqRCNEib4HmvCUHCozlJdq9uVTiYaIEn0Pkjzhqdp0HRqYTUzZfyIl/uLouCWbEn0PSj0qRx27laPJVRKySn8Crepx9L3JdriuXXsTO3asp6ZmNOPG3VJQR2zXjt3scgud9yOloTKDhK7S1z/QqJsKSctKlkmkUTcSsmISfdlH3ZjZBDNb3unrn2Z2pZkNM7OnzOz16Ps+he4jTdSxGx8l+T3pmISjEp9AC0707r7G3evdvR6YAnwAPAzcACx29/HA4uh+1VPHbjjSluQKeT1pGnKa9L9nkoZXTgP+7u7rgNOBedH2ecAZJdpHoqV1uYUk/pOlKclB+l5Pvqr99fdFqRL9ucCC6Hatu2+Mbm8Caku0j0RL60qW+idLDs01qF5FJ3oz2xs4DXio688809Obs5vBzGaaWbOZNbe1tRUbRiLU1k6nsbGFqVPbaWxsSXyST5K0JblCXk93Q06TKAl/z97+FpVU9KgbMzsduNTdT4rurwGmuvtGMxsJLHH3CT09RzWMukmTpF8jNG3rthTyejr/TtKPR6jx9xRXqWKu5Fo35/Fp2QbgMWBGdHsG8GgJ9iEB0WSk5NNcg+pSVKI3s8HAicCiTptvBU40s9eBE6L7IsFIW5Ir9PWEXvroq5D+nj2VlOIsN2nClBRFk5GSL9TSR9KlrXQjVUxJXiR8SvQiVS6k0kea9HRcK33MVbpJqNbW+SVZbE1KR2UsqTSVblIsuxJmZpE071gJU8sex0uTxyRUSvQJpEsciuSmT1S5KdEnkFbCDEcSZmhWk3J8qkrD31KJvsQqcRUprYQZDk0eS780lOSU6EuoUrXztKyEqWQopaBPVb1Toi+hStXO07ISZhpaSp1pmGI8yvGpKm0nDw2vLKElS/qRe7FOY+rU9kqHEzzNyJRSK8d7KuT3qYZXxkC1896lraUkYdGnqtyU6EsoLbXzclLnpZRTOd5HaTh5KNGXUFpq531RidFFIiFIQyNkQNwBpE1t7fRUJvbOsqOLsh3P2dFFQF6vPQ0tJZEkUIte8laq0UVpaCmJJIESveRNM3NFkkWJXvKm0UUiyaJEL3nT6CKRZFGil7xV0+ii3qifQQpR6feNZsZKUar9Aighz5qUcCXqmrFmNtTMFprZa2a22swazWyYmT1lZq9H3/cpZh+VonHh+dMFUESSodjSzZ3A7939IGAysBq4AVjs7uOBxdH9oClhZeR7sqvWC6BoGQcpRJzvm4JLN2b2eWA5MM47PYmZrQGmuvtGMxsJLHH3CT09V9ylm6VL66Ikv7uamjE0NrZUPqAYdJ0EBZkO1p5q7+VexC0J12BV6UYKkaTSzVigDbjXzP5qZr80s8FArbtvjB6zCagtYh8VoXHhhbXOyz3MMm3LGIvEpZhEPwA4AviZux8OvE+XMk3U0s953jKzmWbWbGbNbW1tRYRRPI0LL+xkp2GWWsZBClPp900xiX4DsMHdX4ruLyST+Fujkg3R9825ftnd57h7g7s3jBgxoogwiqeEVdjJrhzDLJNW/w41LglbooZXmtlzwA/cfY2ZNQGDox9tcfdbzewGYJi7X9fT88RdowcNEyykRl9uqn+L9KyvNfpiV6+8HJhvZnsDa4GLyHxK+LWZXQysA84pch8VUQ2rTvYk+9qr+WQnklZFJXp3Xw7kOptMK+Z5JR6hnexU/xYpDS2BIMFS/bt66G9dXkr0IhI7DaUtLyX6hNPSDZIUarXHR4k+wbR0gyRJ11Z70obSJplWr0wwLd0gSdLTcFkNpS1MRVavlHhp6QYJnVrtYVCiTzAt3SCha2rKtNSzrfXs7a6JXkNpy0uJPsG0dIOkhVr45aVEn2C6pJ8kiVrt8VFnrIhIQqkzVkREACV6EZHUU6IXEUk5JXqRCtCoEomTEr1URLWvyaNFuyROSvRS9iQcypo81X6ykeqlRF/lKpGE1669abdLFAK0t3/A2rU3lWwfvYnjZKPp/xIKJfoqV4kkHMKaPHGcbPo6/V+k3JToq1wlknAIa/KEcLIRiYsSfZWrRBIOYU2euE82mv4vcSoq0ZtZi5mtMLPlZtYcbRtmZk+Z2evR931KE6qUQyWScNxr8rS2zmfnzu17bK/kyUblGonTgBI8x9fd/R+d7t8ALHb3W83shuj+9SXYj5RBNtmuXXsTO3asp6ZmNOPG3UJt7XRaW+fn3F7ofuJYbC3bCdu1Pj9gwHDGj79TC8BJVShFou/qdGBqdHsesIQyJPpSJqFqlysJd02Q2VEq2ccnRa5OWID+/T+bqNchUoxia/QOPGlmy8xsZrSt1t03Rrc3AbVF7mMPoYzLTrMQhkSWgjphRYpP9Me4+xHAN4BLzezYzj/0zBrIOddBNrOZZtZsZs1tbW157TQtSShk3SfIPa9RG7K4O2FFQlBUonf3t6Pvm4GHgaOAVjMbCRB939zN785x9wZ3bxgxYkRe+1Urrfy6T4SWqE9OIYz4qXbqiI5fwYnezAab2ZDsbeAkYCXwGDAjetgM4NFig+xKrbTyyyRCy/ETT9Qnp7hH/IjW+QlBMZ2xtcDDlpnfPQC4391/b2Z/AX5tZhcD64Bzig9zd+PG3bLHSAq10kqrtnY6q1f/r5w/S9onp7hG/IiEouAWvbuvdffJ0ddEd78l2r7F3ae5+3h3P8Hd3y1duBlqpVVG5vjm2q5PTklWiVKK1vkJi64Zm4OGbmbkGoPer98gnVQTzuzT9XfSuL9qomvGFkhDNz+V65PTF74wg7Vrb9JSv93QUsgSIiX6LjR0c3e1tdNpbGxh6tR2xo27hU2b5ukk2I0QGgndlUbiLKVonZ/4qXTTxZIl/cg99N+YOrW90uEEZenSupzj6GtqxtDY2FL5gAITwvHpS5lEpZT0UOmmQBq6mVtr6/xuJ0slbRROuWh+R7oluSNZib4LTbDZU7Yk0Z1qPwlmxdVIyLcsk/ZSSrkScpLnA6h0k4NG3eyuu5IEaBROZyGMUlJZpnzHIMRjq9JNETp3QDY2tlR9Euup9KAk/6m0z+9IcumiUGmZD6AWvfQqhE5G6ZumpvIloRBbtFlNTblLK7Nmle54hPj6q6JFX4oxyxr33Dv1WyRH0lqapaILsfcssYm+FGOWQxj3nARpL0lI98pRukhq8k1yJ3ZiSzf5lhNydbBm7qskIdIXpSpdlLsE0rl8Vc5SVgj6WrpJbKLPZ2JTd6Mhcl1irrvnEOlOtYzSSkqij2tfcUh9jT6fMcvdLWsA/fN6bglbHP0t1VT+66l00duxT8volaRKbKLPp4Ow++GBu9TJmBJxJdxqWhupu6Tcl2Nfyc5SnVT2lNhEn08HYfet/zHqZEyJuBKulj0I72SnETh7KuYKU7Hr65WDeroila4+lA5xJdyamtHddOhXrvwXdx9Bvsc+yaNXkiqxLfp8aHhg+sW1zkzccwxC6CPI99hXsmWtk0pGYkfdiHRW6XVmOreiBwwYhjvs2vVuxVvUIcxaDmGNn2rV11E3iS7diGRlE0olShhdE9vOnVvo128QBx/8q4onthD6CGprp7Nt2wu8884cYBfQny98YYaSfECKTvRm1h9oBt5291PNbCzwADAcWAac7+4fF7sfkd5Uqr+lp87HSie3UPoINm2aRybJA+xi06Z5fP7zRyvZB6IUNforgNWd7t8G3OHuBwDvAReXYB8iwQihFZ0Vdx8BhDfqRvZUVKI3s1HAt4BfRvcNOB5YGD1kHnBGMfsQCU1IVyELYaBBSCc+ya3Y0s1/AdcBQ6L7w4Gt7r4zur8B2L+QJ/7kk0/YsGEDH330UZEhSqkMHDiQUaNGsddee5X0eeMeHpivnobrxiHuIcIhlI+kZwUnejM7Fdjs7svMbGoBvz8TmAkwevSeb4gNGzYwZMgQ6urqsOwUN4mNu7NlyxY2bNjA2LFjS/a8XTs2s8MDgWCTfSU7fpMgtBNfvpLW0ChEMS36o4HTzOybwEDgc8CdwFAzGxC16kcBb+f6ZXefA8yBzPDKrj//6KOPlOQDYmYMHz6ctra2kj5vSB2b+Yi7FR2SJJ/4ktjQKETBid7dbwRuBIha9Ne6+3Qzewg4i8zImxnAo4XuQ0k+LOX4e6i+mw5JPfEV0tBI4ieAcsyMvR642szeIFOzn1uGfVTMI488gpnx2muv5fz5hRdeyMKFC3P+LJd33nmHs846C4Dly5fzu9/9ruNnS5Ys4cUXX8w7xrq6Ov7xj3/k/XshCKljU6pPvg2NEGYiF6Ikid7dl7j7qdHtte5+lLsf4O5nu/uOUuyjr0o9vXrBggUcc8wxLFiwoCTPt99++3WcGEqV6JMshOGBUr3ybWgkdShp6ta6yXWB4EJt376d559/nrlz5/LAAw8AmU7Jyy67jAkTJnDCCSewefPmjsfX1dVx4403Ul9fT0NDAy+//DInn3wyX/rSl7jnnnsAaGlp4dBDD+Xjjz/m5ptv5sEHH6S+vp7bbruNe+65hzvuuIP6+nqee+452traOPPMMznyyCM58sgjeeGFFwDYsmULJ510EhMnTuQHP/gBISxjUagQhgdK9cq3oZHUUqOWQOjBo48+yimnnMKBBx7I8OHDWbZsGevWrWPNmjWsWrWK1tZWDjnkEL7//e93/M7o0aNZvnw5V111FRdeeCEvvPACH330EYceeiiXXHJJx+P23ntvfvSjH9Hc3MxPfvITAD788EM++9nPcu211wLwve99j6uuuopjjjmG9evXc/LJJ7N69Wpmz57NMcccw80338xvf/tb5s5NdHUssfVdSb58O5KTOpQ0FYm+qWn3lny2z3DWrOJKOQsWLOCKK64A4Nxzz2XBggXs3LmT8847j/79+7Pffvtx/PHH7/Y7p512GgCTJk1i+/btDBkyhCFDhlBTU8PWrVvz2v/TTz/NqlWrOu7/85//ZPv27Tz77LMsWrQIgG9961vss88+hb9IkSqXT0MjqUNJU5Poswm9VNeIfPfdd/nDH/7AihUrMDN27dqFmfGd73ynx9+rqakBoF+/fh23s/d37tzZ3a/l1N7ezp/+9CcGDhyY/wsQkZJL6lDS1NXoS2XhwoWcf/75rFu3jpaWFt566y3Gjh3L8OHDefDBB9m1axcbN27kmWeeKXgfQ4YM4V//+le390866STuvvvujvvLly8H4Nhjj+X+++8H4IknnuC9994rOAYRyU9t7XQaG1uYOrWdxsaW4JM8pDDRl+pCAwsWLNij9X7mmWeyceNGxo8fzyGHHMIFF1xAY2Njwfv4+te/zqpVq6ivr+fBBx/k29/+Ng8//HBHZ+xdd91Fc3Mzhx12GIccckhHh+6sWbN49tlnmThxIosWLco5szhkcVzEW6SaBXvhkdWrV3PwwQfHFJF0p9i/iy5SIVI6fb3wSOpa9BK2pI5DFkkyJXqpqKSOQxZJMiV6qSgteSBSeUr0UlFa8kCk8pTopaK05IFI5aViwpQki5Y8EKksteh7YGZcc801Hfd//OMf09TLmgqPPPLIbssWFCLfZYcfe+wxbr311pz7v++++3jnnXfy2n924TURSYfUJPpyTMKpqalh0aJFeSXdUiT6fJ122mnccMMNOfdfSKIXkXRJRaIv18UABgwYwMyZM7njjjv2+FlLSwvHH388hx12GNOmTWP9+vW8+OKLPPbYY/zwhz+kvr6ev//977v9zm9+8xu+/OUvc/jhh3PCCSfQ2toKdL/scEtLCwcddBAXXnghBx54INOnT+fpp5/m6KOPZvz48fz5z38GMsn8sssu22P/t912G83NzUyfPp36+no+/PBDli1bxnHHHceUKVM4+eST2bhxIwDLli1j8uTJTJ48mZ/+9KdFHTcRCYy7x/41ZcoU72rVqlV7bOvOiy+O8WeeYY+vF18c0+fnyGXw4MG+bds2HzNmjG/dutVvv/12nzVrlru7n3rqqX7fffe5u/vcuXP99NNPd3f3GTNm+EMPPZTz+d59911vb293d/df/OIXfvXVV7u7++WXX+6zZ892d/fHH3/cAW9ra/M333zT+/fv76+88orv2rXLjzjiCL/ooou8vb3dH3nkkY593nvvvX7ppZfm3P9xxx3nf/nLX9zd/eOPP/bGxkbfvHmzu7s/8MADftFFF7m7+6RJk/yPf/yju7tfe+21PnHixJyvIZ+/i4iUF9DsfcixqeiMLecknM997nNccMEF3HXXXXzmM5/p2L506dKOpYLPP/98rrvuul6fa8OGDXz3u99l48aNfPzxx4wdOxagx2WHx44dy6RJkwCYOHEi06ZNw8yYNGkSLS0teb2WNWvWsHLlSk488UQAdu3axciRI9m6dStbt27l2GOP7Xg9TzzxRF7PLSLhSkXpptyTcK688krmzp3L+++/X9TzXH755Vx22WWsWLGCn//853z00Ue9/k7XpY47L4Oc77LH7s7EiRNZvnw5y5cvZ8WKFTz55JP5vQgRSZxUJPpyT8IZNmwY55xzzm5XcvrqV7/acXnB+fPn87WvfQ3Yc6nhzrZt28b+++8PwLx58zq2l3LZ4Z6WPp4wYQJtbW0sXboUgE8++YRXX32VoUOHMnToUJ5//vmO1yPJohVBpScFJ3ozG2hmfzazv5nZq2Y2O9o+1sxeMrM3zOxBM9u7dOHmVolJONdcc81uo2/uvvtu7r33Xg477DB+9atfceeddwKZK1HdfvvtHH744Xt0xjY1NXH22WczZcoU9t13347tpVx2uOv+L7zwQi655BLq6+vZtWsXCxcu5Prrr2fy5MnU19d3XIz83nvv5dJLL6W+vj7R16CtRuUajCDpUfAyxWZmwGB3325mewHPA1cAVwOL3P0BM7sH+Ju7/6yn59Iyxcmhv0t4li6t6+Y6pmNobGypfEBSMWVfpjjq9N0e3d0r+nLgeGBhtH0ecEah+xCR3mlFUOlNUTV6M+tvZsuBzcBTwN+Bre6e7SXcAOxfXIgi0hOtCCq9KSrRu/sud68HRgFHAQf19XfNbKaZNZtZc1tbWzFhiFQ1rQgqvSnJqBt33wo8AzQCQ80sOz5/FPB2N78zx90b3L1hxIgR3T1vKcKTEtHfI0xaEVR6U/CEKTMbAXzi7lvN7DPAicBtZBL+WcADwAzg0UKef+DAgWzZsoXhw4eT6feVOLk7W7ZsYeDAgXGHIjloRVDpSTEzY0cC88ysP5lPBr9298fNbBXwgJn9H+CvwNyenqQ7o0aNYsOGDaisE46BAwcyatSouMMQkTwVnOjd/RXg8Bzb15Kp1xdlr7326lgiQERECpeKmbEiItI9JXoRkZRTohcRSbmCl0AoaRBmbcCec7j7Zl+g75eAil+S4k1SrKB4yylJsUKy4i0m1jHunnt8eidBJPpimFlzX9Z6CEWS4k1SrKB4yylJsUKy4q1ErCrdiIiknBK9iEjKpSHRz4k7gDwlKd4kxQqKt5ySFCskK96yx5r4Gr2IiPQsDS16ERHpQaISfUiXL+yraM3+v5rZ49H9kGNtMbMVZrZW8sXAAAAD8UlEQVTczJqjbcPM7Ckzez36vk/ccQKY2VAzW2hmr5nZajNrDDjWCdExzX7908yuDDVeADO7KvofW2lmC6L/vSDfu2Z2RRTnq2Z2ZbQtmGNrZv9tZpvNbGWnbTnjs4y7omP8ipkdUYoYEpXogR3A8e4+GagHTjGzr5BZNfMOdz8AeA+4OMYYu7oCWN3pfsixAnzd3es7Dfe6AVjs7uOBxdH9ENwJ/N7dDwImkznGQcbq7muiY1oPTAE+AB4m0HjNbH/gP4EGdz8U6A+cS4DvXTM7FPgPMutrTQZONbMDCOvY3gec0mVbd/F9Axgffc0EerwMa5+5eyK/gEHAy8CXyUw2GBBtbwT+X9zxRbGMiv6IxwOPAxZqrFE8LcC+XbatAUZGt0cCawKI8/PAm0R9TCHHmiP2k4AXQo6XzFXh3gKGkVn48HHg5BDfu8DZwNxO9/83cF1oxxaoA1Z2up8zPuDnwHm5HlfMV9Ja9Em7fOF/kXnTtUf3hxNurJC55u+TZrbMzGZG22rdfWN0exNQG09ouxkLtAH3RmWxX5rZYMKMtatzgQXR7SDjdfe3gR8D64GNwDZgGWG+d1cCXzOz4WY2CPgm8EUCPbaddBdf9iSbVZLjnLhE70VcvrCSzOxUYLO7L4s7ljwc4+5HkPn4eKmZHdv5h55pYoQwTGsAcATwM3c/HHifLh/NA4q1Q1TTPg14qOvPQoo3qhefTuaEuh8wmD1LD0Fw99VkSkpPAr8HlgO7ujwmmGObSyXiS1yiz/ICLl9YYUcDp5lZC5mrbR1Ppq4cYqxAR0sOd99MpoZ8FNBqZiMBou+b44uwwwZgg7u/FN1fSCbxhxhrZ98AXnb31uh+qPGeALzp7m3u/gmwiMz7Ocj3rrvPdfcp7n4smb6D/yHcY5vVXXxvk/lEklWS45yoRG9mI8xsaHQ7e/nC1Xx6+UIo4vKFpeTuN7r7KHevI/Nx/Q/uPp0AYwUws8FmNiR7m0wteSXwGJk4IZB43X0T8JaZTYg2TQNWEWCsXZzHp2UbCDfe9cBXzGyQmRmfHt9Q37v/Fn0fDfw7cD/hHtus7uJ7DLggGn3zFWBbpxJP4eLsoCigQ+MwMpcnfIVMEro52j4O+DPwBpmPxTVxx9ol7qnA4yHHGsX1t+jrVeCmaPtwMh3KrwNPA8PijjWKqx5ojt4LjwD7hBprFO9gYAvw+U7bQo53NvBa9H/2K6Am4Pfuc2RORH8DpoV2bMmc3DcCn5D5NHpxd/GRGbDxUzJ9jyvIjHwqOgbNjBURSblElW5ERCR/SvQiIimnRC8iknJK9CIiKadELyKSckr0IiIpp0QvIpJySvQiIin3/wGmIbjdghZsigAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Find indices of positive and negative examples\n",
"pos = np.where(y==1)[0]\n",
"neg = np.where(y==0)[0]\n",
"\n",
"# Plot examples\n",
"plt.plot(X[pos, 1], X[pos, 2], 'b+', label='Admitted')\n",
"plt.plot(X[neg, 1], X[neg, 2], 'yo', label='Not admitted')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Sigmoid\n",
"The sigmoid function, or logistic function, is a function that asymptotes at 0 and 1. The value at 0 is $\\frac{1}{2}$.\n",
"\n",
"$h_\\theta(x) = g(\\theta^Tx) = g(z) = \\frac{1}{1+ e^{-z}} = \\frac{1}{1+ e^{-\\theta^Tx}}$\n",
"\n",
"A plot of the sigmoid function:\n",
"\n",
"\n",
"We are going to use the sigmoid function to predict how likely it is that a given data point is in category 0. Our hypothesis:\n",
"\n",
"$h_\\theta(x) = P(y = 0|x;\\theta)$\n",
"\n",
"Because there are only two categories (in this case), we can derrive that:\n",
"\n",
"$P(y = 0|x;\\theta) + P(y = 1|x;\\theta)= 1$\n",
"\n",
"**Exercise**: Implement the sigmoid function in Python."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"def sigmoid(z):\n",
" return z"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Compute the Cost and Gradient\n",
"\n",
"#### The cost function\n",
"\n",
"The cost function in logistic regression differs from the one used in linear regression. The cost function in logistic regression:\n",
"\n",
"$$J(\\theta) = - \\begin{bmatrix}\\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}-y^{(i)}\\log h(x^{(i)}-(1-y^{(i)})\\log(1-h_\\theta(x^{(i)}))\\end{bmatrix}$$\n",
"\n",
"Assume our hypothesis for an example is wrong, the higher probability $h_\\theta$ had predicted, the higher the penatly.\n",
"\n",
"A vectorized version of the cost function:\n",
"\n",
"$$J(\\theta) = \\frac{1}{m} ⋅(−y^T \\log(h)−(1−y)^T \\log(1−h))$$\n",
"\n",
"**Exercise**: Implement the vectorized cost function in Python."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def compute_cost(theta, X, y):\n",
" return 0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### The gradient\n",
"\n",
"The gradient is the step a minimization algorithm, like gradient descent, takes to get to the (local) minimum. Note that this step can be taken in a higher dimension and hence the gradient is a vector. In the previous programming exercise we used gradient descent. This time, we are going to use an algorithm called conjugate gradient to find the minimum. How that algorithm works is beyond the scope if this course. If you are interested, you can learn more about it [here](https://en.wikipedia.org/wiki/Conjugate_gradient_method).\n",
"\n",
"The partial derrivative or $J(\\theta)$:\n",
"\n",
"$$\\frac{\\delta}{\\delta\\theta_J} = \\frac{1}{m}\\displaystyle\\sum_{i = 1}^{m} \\begin{bmatrix}(h_\\theta(x^{(i)}) - y^{(i)}\\end{bmatrix}x_j^{(i)}$$\n",
"\n",
"Vectorized:\n",
"\n",
"$$\\frac{\\delta}{\\delta\\theta_J} = \\frac{1}{m} \\cdot X^T \\cdot (g(X\\cdot\\theta)-\\vec{y})$$\n",
"\n",
"**Exercise**: Write a function to compute the gradient."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def compute_gradient(theta, X, y):\n",
" return np.zeros(len(theta))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Test the cost and gradient function. We expect $J \\approx 0.693$ and $\\frac{\\delta}{\\delta\\theta_J} \\approx \\begin{bmatrix}-0.1000 & -12.0092 & -11.2628 \\end{bmatrix}$"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cost: \n",
"0.6931471805599453\n",
"\n",
"Gradient: \n",
"[ -0.1 -12.00921659 -11.26284221]\n"
]
}
],
"source": [
"initial_theta = np.zeros(n)\n",
"y = y.reshape(m)\n",
"print('Cost: \\n{}\\n'.format(compute_cost(initial_theta, X, y)))\n",
"print('Gradient: \\n{}'.format(compute_gradient(initial_theta, X, y)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Learning $\\theta$ using conjugate gradient\n",
"\n",
"The optimization library we are going to use is `scipy.optimize`. We have to provide the algorithm with our cost function, initial guess, gradient among with some other (optional) configuration options.\n",
"\n",
"Please scan [the docs](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html) of `minimize` first."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Optimization terminated successfully.\n",
" Current function value: 0.203498\n",
" Iterations: 42\n",
" Function evaluations: 100\n",
" Gradient evaluations: 100\n",
"Conjugate gradient found the following values for theta: [-25.15522426 0.20618292 0.20142209]\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:3: RuntimeWarning: divide by zero encountered in log\n",
" This is separate from the ipykernel package so we can avoid doing imports until\n"
]
}
],
"source": [
"from scipy.optimize import minimize\n",
"result = minimize(compute_cost, initial_theta, args = (X, y),\n",
" method = 'CG', jac = compute_gradient, \n",
" options = {\"maxiter\": 400, \"disp\" : 1})\n",
"theta = result.x\n",
"print('Conjugate gradient found the following values for theta: {}'.format(theta))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The decision boundary\n",
"The decision boundary is a line (in case of a 2d plane) that seperates the area where we predict $y=1$ and $y=0$."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.legend.Legend at 0x120189cc0>"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xd4VFX6wPHvm0JCLyGEThLpJBAg9BKkKyhFRZGlWVhWURB0rUtxxbIKKOiKKIvoTxFlaaJIJ1SBoJFAIIAQakgiGDoh5fz+yCQbIH3KvTNzPs+TJzM3M/e+czPzzrnvOfdcUUqhaZqmuS4PowPQNE3T7Esnek3TNBenE72maZqL04le0zTNxelEr2ma5uJ0otc0TXNxOtFrmqa5OJ3oNU3TXJxO9JqmaS7Oy+gAAKpWraoCAwONDkPTNM2p7N279w+llH9hjzNFog8MDCQqKsroMDRN05yKiJwoyuN06UbTNM3F6USvaZrm4gpN9CLyHxFJEpH9uZZVEZF1InLE8ruyZbmIyGwROSoi+0SklT2D1zRN0wpXlBb950Df25a9BGxQSjUANljuA9wDNLD8jAE+tk2YmqZpWkkVmuiVUluAC7ctHgAstNxeCAzMtfwLleVnoJKI1LBVsJqmaVrxlbRGH6CUSrDcPgcEWG7XAk7letxpy7I7iMgYEYkSkajk5OQShmE/iYlfsXNnIJs3e7BzZyCJiV8ZHZKmaVqJWN0Zq7IuUVXsy1QppeYppcKVUuH+/oUOA3WoxMSviIsbQ2rqCUCRmnqCuLgxOtlrmuaUSproE7NLMpbfSZblZ4A6uR5X27LMqRw79iqZmdduWZaZeY1jx141KCJN07SSK2miXwmMtNweCazItXyEZfRNe+BirhKP00hNPVms5ZpWFFOnGh2B5q6KMrxyEbATaCQip0XkceBtoJeIHAF6Wu4D/AgcA44CnwJP2SVqO/PxqVus5baiE4FrmzbN6Ag0dyVZJXZjhYeHKzNNgZBdo89dvvHwKEOjRvMICBhmt+2KgAn+HZqd6P+vZmsislcpFV7Y4/SZsXkICBhGo0bz8PGpBwg+PvXsnuQ1+7n9SMmRR05Tp2YleJGs+9m3zXT0ZqZYNPvQLXqDTZ2a9yH9lCn6A2grt7ekjWpZF3e7U6c65j2gjzScV1Fb9DrRm4j+wNmHsyZ6R8Wp33fOyy1KN7uPX2Dsl3s5d/GGVevRJ0e5nvxKJkaWUKZMcdy2CuMMJSXNdpw60Z84f5XNh5PoOTOSz7cfJyOz+M0SM50cZaZE4OymTs1qpWa3VLNv337f0fX6ojzGEQk4v/2jE71rcvrSzcnz13h1eQxbj/xBi9oVeXNwKM1qVizy83fuDLQk+Vv5+NSjQ4f4EsWkmYtZSjfFpUs3WmHconQDUNevDF881pbZQ1tyJuU693+4nTd/PMi1m+lFer4+Ocr13X6kpI+cbqX3h+tz+hZ9bhevpfH2TwdZtPsUtSqV5o2BIdzduFqBz9Etes2sHDXqRnNebtOiz61iGW/eGtyc78Z2oEwpT0Z/voenv/qFpEv5d9YGB0/Hw6PMLcs8PMoQHDzd3uFqWoF0ktdsxaUSfbY2gVX44dkuPN+7IesOJtJjRiRf/nyCzDw6a/XJUZqRdDLXHMGlSjd5if/jKq8uj2H70fO0rFuJtwaH0rh6BbtsS9OKS3eEatZwy9JNXgKrluX/Hm/HzCEtOHH+Gv1nb+Odnw5x/WaG0aFpmunpIw7X4PKJHkBEGNyqNhsmRjC4VS0+3vw7vd+PJPKw+a5spbk+ZzpZSc+46RpcvnSTl5+PneeVZTEcS77K/S1q8o/+TfEv7+Ow7WtaNrOXbswen7vTpZsCtA/2Y/X4Lkzo2YCf9p+jx4zNLNp9Ms/OWk1zN850xKEVjVu26HM7mnSFV5fFsOv4BdoEVubNQaE0CChvSCya+zH7WHndojc3h7ToRWS8iOwXkQMiMsGyrIqIrBORI5bfla3Zhr3Vr1aOb8a0518PNudI0hXunb2VGWvjuJGmO2s1+7PHHDaadrsSJ3oRCQGeBNoCLYD+IlIfeAnYoJRqAGyw3Dc1EWFIeB02TIzgvuY1mbPxKH3f38L2o38YHZqmFYstOk916cb1WNOibwLsUkpdU0qlA5HAYGAAsNDymIXAQOtCdBy/cj7MfDiMr55oB8Cwz3Yx8dtozl9JNTgyTcufntlSK4w1iX4/0EVE/ESkDHAvUAcIUEolWB5zDgiwMkaH61S/Kj9N6Moz3evz/W9n6TEzkm+jTmGG/gxnphOFfUybplvgWsFKnOiVUgeBd4C1wE9ANJBx22MUkGd2FJExIhIlIlHJyeYbz+7r7cmk3o348dku1Pcvx9+X7OOReT/ze/IVo0NzWnpMtv3YqwXurDNb6i+5W9ls1I2IvAmcBsYD3ZRSCSJSA9islGpU0HPNfinBzEzF4qhTvPXjQW6kZfLU3Xfxt2534ePlaXRoVnPkqA89gsN28rvWMOh9DO7zXnPUqJtqlt91yarPfw2sBEZaHjISWGHNNszAw0MY2rYuGyZ1o29Idd5ff4R7PtjKz8fOGx2a1ezdytYde/aRXx3dWVvgmn1Z1aIXka2AH5AGTFRKbRARP+BboC5wAhiilLpQ0HrM3qK/XeThZF5bHsOpC9d5qHVtXrm3CZXLljI6rBJxZMvHXVpZjpbffjX7GH1byn6deTVcpkxx3f1Q1Ba9258wVVLXb2Ywe+MRPt1yjAqlvflH/yYMDKuFZDdd7cBWH9z8Dvvt/YHQid4+8ntfuNP+dtbLRVpLJ3oHOXTuEi8vjeHXkyl0qu/HGwNDCapa1i7bsseb15EfCHdqYZqBuyQ7MC7RG/2e1nPdOEjj6hX479iO/HNgCPtOXaTP+1v4cOMRbqZnGh2a6TjqA+HOXybu1CdS0Gt1VF+Fs4wk04neBjw8hOHt67FhUgS9mgTw3trD9Ju9lT3xBXZNFIm9P7iu2HnnLB8+e3GXk50KOrHLFV+vNXTpxg42HkrkH8sPcCblOkPb1uWlvo2pWMbb6vW606G4Ncyyn4xIOLlfu1n2gyM4ugRplk5fXboxUPfGAayb2JUnuwTxbdQpesyMZOVvZ/WZtXZkxpKF0UcWrni0lh9HvlZnnCJCJ3o7KVPKi1f7NWXF052oVcmXZxf9ysgFezh14VqJ1+lOH9zicsYPny1kHzXk9SXnTlz9/2wtXbpxgIxMxZc743l3TRwZSjG+R0Oe6BKEt6f+nrUHI0sWjj6sd9dhhWZhdH+AHl5pQgkXrzN15QHWHEikcfXyvDk4lFZ1TT1dv1My+sOXzRFJVyd696Zr9CZUo2JpPhkezrzhrbl4PY0HPt7Ba8tjuHQjzejQXIoZkrw9mWFYoeZcdIveIFdS05mxNo6FO+KpWs6Hqfc3456Q6nY9s1ZzLEccWegWvHvTpRsnse90Ci8vjeHA2Ut0b1yN1wc0o3blMkaHpTkJnejdmy7dOInmtSux4ulOvNavCT8fO0+vmVv4dMsx0jP0mbVa4XSpRisK3aI3kdN/XmPKigNsOJRE0xoVeGtwKC3qVDI6LE3TTEq36J1Q7cpl+GxkOB8Pa8UfV1IZ9O/tTF15gCup6UaHpmmaE9OJ3mREhHtCa7B+UgR/aV+PhTvj6TkjkjUHzhkdmqZpTkonepOq4OvN6wNCWPq3jlQq481fv9zLk19EcTblutGhuTxXH56puR9rrzD1HPAEWRcAjwFGAzWAb8i68tReYLhS6mZB69E1+oKlZWTyn23HmbX+MJ4iTOrdiJEdA/H00EMx7UGPZNGchd1r9CJSC3gWCFdKhQCewCPAO8AspVR94E/g8ZJuQ8vi7enBXyPuYt1zEYQHVuH1VbEM+vd29p+5aHRomqY5AWtLN15AaRHxAsoACUB3YInl7wuBgVZuQ7OoU6UMn49uw5yhLTmbcoP7P9zGG6tiuao7a61mxtkvNc1WrC3djAemA9eBtcB44GdLax4RqQOstrT486VLN8V38Xoa7/x0iK93naRWpdK8PqAZPZoEGB2WS9ClG81ZOKJ0UxkYAAQBNYGyQN9iPH+MiESJSFRycnJJw3B5iYlfsXNnIJs3e7BzZyCJiV8BULG0N28OCuW/f+tAWR9PHl8YxVNf7SXx0g2DI9Y0zWysKd30BI4rpZKVUmnAUqATUMlSygGoDZzJ68lKqXlKqXClVLi/v78VYbiuxMSviIsbQ2rqCUCRmnqCuLgxOckeoHW9Kqx6pgsv9GnEhoNJ9JwRyZc748nI1E3SkjL6bFNdLtJszZpEfxJoLyJlJGsmrh5ALLAJeNDymJHACutCNF5+rWp7O3bsVTIzb71QSWbmNY4de/WWZaW8PHj67vqsmdCVsLqV+MeKAzzw8Q4OJlxySJyabRl9ZSrN9Vhbo58GPAykA7+SNdSyFlnDK6tYlv1FKZVa0HrMXKPPblXnTrgeHmVo1GgeAQHD7LrtzZs9yBq5ejuhW7e858JRSrEi+iz/XBVLyvU0nugSxIQeDSldytOusboSo2v0Rm9fcx4OmQJBKTVFKdVYKRWilBqulEpVSh1TSrVVStVXSj1UWJI3u6K2qu3Bx6dusZZD1pm1A1vWYsOkCB5sVZtPIo/Ra1Ykm+OS7BWmZgN61I9x3GEf6zNjC5GaerJYy20pOHg6Hh63Tlns4VGG4ODphT63UplSvPNgcxaPaY+PlwejFuzhmUW/knRZd9bmxehE667XvDUDdyiV6URfiJK0qm0lIGAYjRrNw8enHiD4+NQrdsmoXbAfP47vwsReDVmz/xw9Z0Ty9a6TZOrO2lvoRFsyev84B53oC2FNq7ogRe3gDQgYRocO8XTrlkmHDvEl6hfw8fLk2R4N+GlCF5rVrMgry2IY8slODidetuo1GM1Vk4zRo36Kw1lbw0YfwTmano++CBITv+LYsVdJTT2Jj09dgoOnW9URa2QHr1KK//5yhuk/xHL5Rjp/jQjmme4N8PUuemetq1982yyvzxm4QsexM78GPR+9jdg6yYOxHbwiwoOta7NhUjcGhNXio02/0+f9LWw78keR1+GsrbiictckX9TX7W6tYVegE30BinLCUkkY2cGbrUrZUswY0oKvn2yHhwh/mb+L5xZH88cVcw+S0knGfor6Be5q/RnOVCorKZ3oC2CvlreRHby363hXVVaP78Kz3euzat9Zes6M5Ns9p7i9pGeWBOtqSUYznju8d3SiL4C9Wt726uAtKV9vTyb2bsTq8V1oWK08f//vPh6e9zNHk67kPEYnWNeU3xd4t25Fe747tIZdgU70BbBXy9sWwybtoX618nwzpj3vPBBK3LnL3PPBFmauO8yNtAxD48qPTjLWy+8LPDKy6M/XzE+PuimAkaNjjPbHlVTeWBXL8uizBFctyxuDQuh4V1XAOUalOEOMZpN79Ikzj0RxJ3rUjQ0U1PI2aqIzR6lazof3H2nJF4+1JT1T8einu3j+u9+4cPWmUyTQ2zsWnSFmo0VEmKMfRrM93aIvAXdr6d9Iy2D2hiPM23KM8r5evNavKYNb1UKyM4IJ3d4i1S3U4tH7yzbsfWSpW/R2ZOQ4eCP4envy976N+eHZLgT7l2PSd78x7LNdHP/jqtGh3cIsI4M0LZtZzjnRib4EzDAO3giNqpfnu792YPqgEGLOXKTP+1uYveEIqenm6Ky9vWMxu7M2+8OmE3/R6Y5u16ITfQnYejSOM9X7PTyEYe3qsWFiBL2aBjBz3WH6zd7G7uMXjA7tDnpIaMnpfVRyZjyy1Im+BGw5Dt5eZ9/aW7UKvnz0aCsWjGrD9ZsZDPlkJy/9dx8p124aHRqgW6SacczYwNCJvgRsOQ7e2ev9dzeuxrqJXflr12C+23uanjMjWRF95o4zax3t9g+VTvy6le7OSjzqRkQaAYtzLQoGJgNfWJYHAvHAEKXUnwWty9lG3dhSSS4XaFaxZy/x8rIYfjuVQpcGVXljYAj1/MoaHdYd3HWMvR5J43hOP+pGKRWnlApTSoUBrYFrwDLgJWCDUqoBsMFyX8uHmea9sVbTmhVY+reOvD6gGb+eTKH3rC18tOkoaRnm+sIyy0gIa7nbl5Uzvl6zxGyr0k0P4Hel1AlgALDQsnwhMNBG23BJZpv3xhpTp4KnhzCiQyDrJ0bQvXE13l0TR//Z29h7osCDOq0EivKFZcaOwZJylS9oI9gq0T8CLLLcDlBKJVhunwMC8nqCiIwRkSgRiUpOTrZRGM7HrPPelETuD2L1ir58/JfWfDYinMs30nhw7g5eXRbDxetphsTmSgmvOPLrGNSsV9B7x2zvK6vPjBWRUsBZoJlSKlFEUpRSlXL9/U+lVOWC1uHONXpXkl8N+GpqOjPXHWbB9uP4lfNhyn1N6Rdaw7Aza525Vj11at4t2ylTCk8uzjiXjTWv194K2oeO2r+OPDP2HuAXpVSi5X6iiNSwBFEDSLLBNjSTKkpLuayPF//o35SV4zpTvYIv477+lcc+38OpC9fyWqVWAGuG7jnjyCMzDlV0RrZI9EP5X9kGYCUw0nJ7JLDCBtvQTKo4H8SQWhVZ9lRH/tG/KbuOX6D3rC3M2/I76Q7urHXGhGcr7li+sqWCGjZmLg9aVboRkbLASSBYKXXRsswP+BaoC5wga3hlgadN6tKNayjO4erZlOtMXnGA9QcTaVKjAm8NDiWsTqXCn6jlsGbonrOUbnIz27BYZyrd6NkrNZsp7gdRKcWaA4lMXXmAxMs3GNG+Hs/3aUR5X297hahZOGOiNxtnSvT6zFgXYvScOcVtbYkIfUOqs25iV0Z2COSLn0/Qc2YkP+1PMPzM2uIyU0uzKNy5fGUrBe1Ds+1f3aJ3Ea4wR370qRReXhrDwYRL9GwSwLQBzahVqbTRYRWJbiFrRtAtejfj7HPmAITVqcT34zrx6r1N2H70D3rNjGT+tuMO76zVnIMtj6Kc7YisuHSidxGuMke+l6cHT3YNZu1zXWkXVIV/ropl4L+3E3P6otGh3cHMoyzcgS3PlHX1s251oncge9bQnXnOnLwSY50qZfjPqDZ89GgrEi+lMuCjbbz+fSxXU9MdHl9+9BhvzVnoRO8g9p533pnnzMmvNSUi9Gteg/UTI3i0XV0W7DhOr5mRrItNzPsJmsuz5VGUOx2R6c5YB9m5M9CS5G/l41OPDh3ibbKNxMSvOHbsVVJTT+LjU5fg4OlO0RFb1I7MvSf+5JWlMcQlXqZvs+pMvb8Z1Sv62j/AIjDbGG93YMsOcGftTNfj6E3Gleadt4WSzmGSlpHJp1uP8cH6I3h7evBCn0b8pX09PD2MmTdHM45O9HrUjek4cw3dHkpa3/b29OCpbvVZ91wELetWYsrKAwz+eAcHzpqvs1azL1uOVTfbuHdb0y16B3GFce4FsaZsVNLWlFKKlb+d5Z+rYvnzWhqPdw5iQs8GlCnlVfyVaZoT0i16k3GleedvZ21Hc0lbUyLCgLBarJ8YwZDw2szbcoxeM7ew6ZCeMFXTctMtes1qjuhoLordxy/wyrIYjiZdoV/zGkzp35RqFczRWatp9qBb9JrDmOVkrbZBVfjx2S5M6tWQdbGJ9JgZyf/9fILMTOMbM5pmJJ3oNauZqaO5lJcHz/RowJoJXQmtVZHXlu/nwbk7iDt32eGxaJpZ6ESvWc2MJ2sFVS3LV0+0Y+aQFsSfv0a/2Vv510+HuJGWYdPt6LHzWl7M9r7Qib4IjJ7+1+yyO5o9Pf1ylnl4GD/rpIgwuFVt1k+MYGDLWvx78+/0nrWFLYdtdzF6V58jRSsZs70vrEr0IlJJRJaIyCEROSgiHUSkioisE5Ejlt8FXhjc7Ow9dYErUep6zu309POm2U9VypbivYdasOjJ9nh5CCP+s5vx3/xK8uVUo0PTNIewtkX/AfCTUqox0AI4CLwEbFBKNQA2WO47LVeY/rekinMk4wz7qcNdfqye0IXxPRqwOuYcPWdG8s3uk8XurHWnOVK0ojPz+6LEwytFpCIQTdb1YlWu5XFAN6VUgojUADYrpRoVtC4zD69016kLinuClz33kz3mkTmadIVXl8Ww6/gF2gZW4c3BIdSvVr7Y63HWU+c1+3KlSwkGAcnAAhH5VUQ+s1wsPEAplWB5zDkgIJ8Ax4hIlIhEJSfbrmZqa2YaUeJIxW2h23M/2aPeWb9aOb4Z055/PdCcuMTL3PPBVmaujbN5Z62mmYE1id4LaAV8rJRqCVzltjKNpaWf5/eaUmqeUipcKRXu7+9vRRj2ZcYRJY5Q3LHxzrifRIQhbeqwYVIE9zWvyeyNR7nng63sOPpHkdfh6nOkaCVjtveFNYn+NHBaKbXLcn8JWYk/0VKywfLbqc9Hd+WpCwpS3Ba6rfeTI+udVcv5MPPhMP7v8XZkKsWjn+1i4rfRXLh6s0hxatrtzPa+sGoKBBHZCjyhlIoTkalAWcufziul3haRl4AqSqm/F7QeM9fo3ZWZJmFzZB38RloGH248ytzI3ynv68Ur9zbhwda1EdHTIGvm45D56EUkDPgMKAUcA0aTdZTwLVAXOAEMUUpdKGg9OtGbk1kuZGJEh+fhxMu8sjSGqBN/0j64CtMHhXKXfznHBqFphdAXHtFchlFXb8rMVCyOOsVbPx7kRlomT99dn7HdgvHx8nR8MC5EX43LdnSi1zQbSbp8g3+uOsj3v53lLv+yvDkolHbBfoU/UcuTHpJqO3r2Sjemp2ywrWrlfZkztCWfj27DzYxMHp73My8u2UfKtcI7a92ZbrWbh070LkZP2WA/3RpVY+2ECMZG3MWSX07TY0Yky349jRmOis0o9/kPZj5r1B3o0o2LMctFQFzdwYRLvLw0huhTKXSuX5U3BoYQWLVs4U90I/mVaHTpxnZ06cZNmeUiIK6uSY0K/PdvHfnngGb8diqFPu9v4aNNR7mZ7rrTYhSFbrmbk070LsZdp2wwgqeHMLxDIOsnRdCzSQDvromj/5ytRMUXOJrYpU2dmtVaz26xZ9/OnejNdtaoO9CJ3sU441QEzi6ggi8fDWvF/JHhXE3N4MG5O3l5aQwXr6UZHZop6da94+lE72LcdcoGM+jRJIC1z3XlyS5BLN5zkh4zI/n+t7Nu21mrW+7moTtjNc0O9p+5yCvLYth3+iIRDf15Y2AIdaqUKfyJmlYMujNW0wwUUqsiy57qxJT7mhIVf4FesyKZG/k7aRnu3VmrGUMnek2zE08PYXSnINZPiqBahj9vrz7EfXO28evJP40OTXMzOtFrDuWOZ+3WqFiarW+F88nw1qRcS2PwxzuYvGI/l27ozlrNMXSi1+5gr2Rs9Fm7Rn/J9GlWnfWTIhjVMZD/+/kEPWdE8mNMgtt21mqOoxO9dgt7JmMjLyBuxJdMXicPlff1Qu1txvKnO+Ff3oenvvqFJxZGcSblut3i0DQ96ka7hT2nUDDyQutGTw2R12n/6RmZfL4jnhlrDyMCE3s1ZFTHQLw8dftLKxo96kYrEXtOoWDkWbtmnBrCy9ODJ7oEs25iV9oH+/HGDwcZ8NF29p1OMSwmzTVZlehFJF5EYkQkWkSiLMuqiMg6ETli+V3ZNqFqjmDPZGzkWbteXlXyXO6oqSEKOnmoduUyzB8ZzsfDWpF8OZWBH21n2vcHuJKa7pDYNNdnixb93UqpsFyHDy8BG5RSDYANlvuak8gvGfv53Wt1R6ZRZ+0mJn5FevqlO5aLlHLY1BCFnfYvItwTWoP1kyIY1q4en++Ip9fMSNYeOOeQ+DTXZu01Y+OBcKXUH7mWxQHdlFIJIlID2KyUalTQeuxVozfLNU+dze37zc/vXs6dW2iKC4WXRH71eU9PP7p0+SOPZxjvl5N/8srSGA6du0zvpgFMG9CMGhVLGx2WZjKOujj4ceBPsnrYPlFKzRORFKVUJcvfBfgz+35+7JHos0dZOGtyMhOjOzKtZWQnsDXSMjKZv+04768/jKcIz/dpxIgOgXh6iNGhaSbhqM7YzkqpVsA9wNMi0jX3H1XWt0ie3yQiMkZEokQkKjk52cow7mTkUD5Xk39H5p3J34ycdepmb08PxkbcxbrnIggPrMK072MZ9O/t7D9z0ejQNCdjVaJXSp2x/E4ClgFtgURLyQbL76R8njtPKRWulAr39/e3Jow8mXGUhbPKPyGKU5zZ6uxTN9epUobPR7dh9tCWnE25wYCPtjP9h1iumrSzVk9DbD4lTvQiUlZEymffBnoD+4GVwEjLw0YCK6wNsiSctRVnRlkJMa9ygXKKIyRXmLpZRLi/RU02TIxgSHgdPt16nN6ztrDxUKLRod0h97ViNXMocY1eRILJasUDeAFfK6Wmi4gf8C1QFzgBDFFKFXjJHV2jN7/Nm/OrC5u7zu2qouIv8PLSGI4kXeHe0OpMua8ZARV883381KmOa2nra8I6jt1r9EqpY0qpFpafZkqp6Zbl55VSPZRSDZRSPQtL8vZir1ac0fOlGCVrP+a1XB8hGSE8sAo/PNuFF/o0Yv3BJHrOiOTLnfFkZuadYe3dytbXijU3PQVCMbjzUUJerx288fKqQHr6BT18FeOG88b/cZWhM2NIyDxPy7qVeHNQKE1qVLjlMY5sZesWvePoKRDswJ1H8tx+hOTp6YeIkJ5+HiNmojQbI2fmDKxalp/fasesh1tw4vw17puzjbdXH+K1KRm6la0BOtEXi7uP5AkIGEaHDvE0afIlGRkpKHXzlr+7y5deXoxvBAiDWtZmw8QIBreqxdzI34ksG8nmuOSc1rVSWT/2TvT6WrHmoxN9MeiRPP9ruUJGnn93ly+92xnRCMirLl6lXCnK7G/BN2Pa4+3pwcj/7ObZRb/iUfaG3eLIKy5n3Y6rHu3oGn0xuHONPlt+Z8lmc5azZW3N6LOH86qLp6ZnMHfzMT7adBQyPJg6qAmPtKmDh4ucWWuPvgBn61/QNXo7cIXx2NYqqIXqTCch2ZoZT8ry8fJkfM8GrJ7QhVbBFXhlWQxDPtnJ4cTLRXq+q7Zu3ZFO9IW4fTglQIcO8XTrlkmHDvFuleShoDKVp9t96eVmdCOgoLr4Xf7lWPRke959sDlHk6/Qb/ZW3lsTx420vMtv2cx44pM9hnG6w9BQly7dWDsHF+jIAAAegElEQVTcTZdq7qT3iXM7fyWV6T8eZOkvZwj0K8MbA0Pp3KBqno81exlDl2506cYmw92MH0lhPka3XDXr+JXzYeaQML5+oh0iwl/m7+K5xdGcv5IK2K/FbG+u1Pq2B5dt0Re3cyyv1v/Bg8NxxultNa0obqRl8O9NR/k48nfK+njxyr1NeKh1bcSS5W3VurVXKzn3tA622oYjp4qwBYfMR28r9kj0xZmDPL9yhEhpMjLO37EGdx1Zot3JFS5ucyTxMq8si2FP/J+0C6rC9EGh1K9WrsDkWZzX7YhyiLOVXGzF7Us3xRnznl+JRgTTjaTQCubIuYiMPBvWlhoElGfxmA68PTiUgwmXuPeDrcxad5hXJ+fdWVuU1+2IDk536ES1FZdt0Ren07Cg1n+TJl86fYvNXTi6o9josfP2kHw5lTd+iGVF9FmCq5Zl+qBQOtzld8tjivu6dYvefty+RV+cTsOCWv/Zp/2763BKZ+LoznOjp8Swx9GLf3kfPnikJV881pb0TMXQT3/mhe9+48+r/5vuwujXrRWfl9EB2FNAwLAiJebg4Ol5tgR1ica5OCoBZden87lKpkOmxLj96CW7fALYpDHStaE/ayZ0ZfbGI3y65RgbDiXxWr8mDGpZCx+fuvm06PN+3Y6Y+0bPr1Mwl23RF4ceMugaHDEX0a316Ts5qoHgiKOX0qU8ebFvY1Y925lAvzJM/PY3/jJ/F16V30Kk1C2PFSmV7+vWwyuNZ3WiFxFPEflVRFZZ7geJyC4ROSoii+X2d4RJ6RKN83PENAR5JdhsjmwgOLJ80rh6BZaM7cgbA0PYd/oiw74sz4qjD5Ce+b+CgBn6+rT82aJFPx44mOv+O8AspVR94E/gcRtsQ9MK5Ygjs/wTqTi0geDomVQ9PIS/tK/HhokRtAr4haVHhjF5+2wO/9nU8og0tz6R0OysSvQiUhvoB3xmuS9Ad2CJ5SELgYHWbEPTisPeR2ZmmaraqEnUqlXwZWzzqTzXeiqpGT68uetf/Gf/M1xNK6s7Y03M2hb9+8DfgewzkPyAFKVUuuX+aaCWldvQ3IQzXI/XLLNUGtmv5ONTlxb+UbzZ+SnuCfov28705OWtc4lKHqRLOCZV4kQvIv2BJKXU3hI+f4yIRIlIVHJycknD0FyEs5x8ZKaOe6P6lbK/7Hy8Unm40QKmdJhA1dLn+XDvY4xcsIeT5/PuwzCaMzQk7KXEJ0yJyFvAcCAd8AUqAMuAPkB1pVS6iHQApiql+hS0Lme58IhmP6548pEru30KhHqB01l3vBPvrokjLSOT8T0b8GSXYLw9zTGwz1VnXXXoXDci0g14XinVX0S+A/6rlPpGROYC+5RS/y7o+TrRa8WZm0gzr4SL15m2MpafDpyjUUB53hwcSut6lY0OyyaTHJrxC8HIM2NfBCaKyFGyavbz7bANzcWYpZNTs06NiqWZO7w1n44I59KNNB6cu4PXlsdw8XqaoXEVZziqs5QRi8MmiV4ptVkp1d9y+5hSqq1Sqr5S6iGlVKottqG5NrN0cmq20atpAOsmRjC6YxBf7zpJz5mR/LAvwbDOWltMcujMw0fNUUDT3J6ZOjk12yjn48Xk+5qy4unOBFTw4emvf+Gxz/dw6oLjO2uL05Bwxbl8TDt7ZVpaGqdPn+bGjRsGRaUZydfXl9q1a+Pt7W10KJoNpGdksnDnCWasjUMpeK5XAx7rFISXAztri1p3d6aBAU5/4ZHjx49Tvnx5/Pz8cq54o7kHpRTnz5/n8uXLBAUFGR2OZkNnUq4zZcV+1h9MommNCrw1OJQWdSoZHdYtnGmEjtNPU3zjxg2d5F1cWtp5rlzZx+XLUVy5so+0tKyreYkIfn5++mjOBdWqVJpPR4Qz9y+tOH81lYH/3s7UlQe4fMPYztrcXLGMaOppinWSd11paee5ceME2SdVK3XTch+8vfUXvCsTEfqG1KBT/aq8tyaOhTvj+Wn/Oabe34y+IdWNDg8o+hTnzsK0LXqzWL58OSLCoUOH8vz7qFGjWLJkSZ5/y8vZs2d58MEHAYiOjubHH3/M+dvmzZvZsWNHsWMMDAzkjz/+KPbzjJSaeob/zZyRLdOyXHMH5X29mTYghGVPdaJy2VKM/b+9PPlFFGdTrhsdmstxuURv63mpFy1aROfOnVm0aJFN1lezZs2cLwZbJXpnpNTNYi3XXFdYnUqsHNeJl+9pzNYjyfSaGcl/th0nI9P4/kNX4XKJfto0263rypUrbNu2jfnz5/PNN98AWR2F48aNo1GjRvTs2ZOkpKScxwcGBvLyyy8TFhZGeHg4v/zyC3369OGuu+5i7ty5AMTHxxMSEsLNmzeZPHkyixcvJiwsjHfeeYe5c+cya9YswsLC2Lp1K8nJyTzwwAO0adOGNm3asH37dgDOnz9P7969adasGU888YRTTiSV32UKnOTyBZqNeXt68NeIu1j3XARtgqrw+qpYBn60nf1nLhodmkswdY3eaCtWrKBv3740bNgQPz8/9u7dy4kTJ4iLiyM2NpbExESaNm3KY489lvOcunXrEh0dzXPPPceoUaPYvn07N27cICQkhLFjx+Y8rlSpUrz++utERUXx4YcfAnD9+nXKlSvH888/D8Cjjz7Kc889R+fOnTl58iR9+vTh4MGDTJs2jc6dOzN58mR++OEH5s93vpOPfXxq3VKjz+KBj4+e7NSd1alShgWj2vBDTALTvo/l/g+3MbpTEBN7NaSsj05XJeUSe27q1Ftb8tn9eFOmWFfKWbRoEePHjwfgkUceYdGiRaSnpzN06FA8PT2pWbMm3bt3v+U5999/PwChoaFcuXKF8uXLU758eXx8fEhJSSnW9tevX09sbGzO/UuXLnHlyhW2bNnC0qVLAejXrx+VKxs/l0hxeXv7AVm1eqVuIlIKH59aOcs19yUi9G9eky4N/PnXT4eYv+04q2MSeH1ACD2bBhgdnlNymUSfndBFwBaVjAsXLrBx40ZiYmIQETIyMhARBg0aVODzfHx8APDw8Mi5nX0/PT09v6flKTMzk59//hlfX9/ivwAn4O3tpxO7lq+Kpb2ZPiiUwa1q8fLSGJ74Iop7Qqoz5b5mVK/omp8Je3G5Gr2tLFmyhOHDh3PixAni4+M5deoUQUFB+Pn5sXjxYjIyMkhISGDTpk0l3kb58uW5fPlyvvd79+7NnDlzcu5HR0cD0LVrV77++msAVq9ezZ9//lniGDTN7FrXq8KqZ7rw976N2HgoiZ4zI/liZ7zurC0Gl0v0U6bYZj2LFi26o/X+wAMPkJCQQIMGDWjatCkjRoygQ4cOJd7G3XffTWxsLGFhYSxevJj77ruPZcuW5XTGzp49m6ioKJo3b07Tpk1zOnSnTJnCli1baNasGUuXLqVuXT3Do+baSnl58FS3+qx9rist61Zi8ooDPPDxDmLPXjI6NKdg2ikQDh48SJMmTQyKSDMD/R7Q8qKUYuVvZ3n9+1hSrqfxROcgxvdsQJlSLlGJLhannwJB0zQtLyLCgLBabJgUwUOta/PJlmP0nrWFTXFJhT/ZTelEr2maU6pUphRvP9Ccb//aAV9vT0Yv2MO4r38h6bKeI+l2OtFrmhNz5wteZ2sbVIUfnu3MxF4NWRubSI8ZkXy16wSZurM2R4kTvYj4ishuEflNRA6IyDTL8iAR2SUiR0VksehTHTXNLlzxkncl5ePlybM9GvDT+C6E1KzIq8v289AnO4k7d7nwJ7sBa1r0qUB3pVQLIAzoKyLtgXeAWUqp+sCfwOPWh6lp2u1c8ZJ31gr2L8fXT7ZjxkMtOJZ8hX6zt/Kvnw5xIy3D6NAMVeJEr7Jcsdz1tvwooDuQPZ3jQmCgVRFqmpYnV7zknS2ICA+0rs2GSd0Y2LIW/978O71nbWHrkWSjQzOMVTV6EfEUkWggCVgH/A6kKKWyTwE9DeQ5eYmIjBGRKBGJSk425z9ARJg0aVLO/ffee4+phcypsHz58lumLSiJ4k47vHLlSt5+++08t//5559z9uzZYm0/e+I1zdyKc8Frd1SlbCnee6gFXz/ZDk8PYfj83Uz45lf+uJJqdGgOZ1WiV0plKKXCgNpAW6BxMZ47TykVrpQK9/f3tyYMwD6dUj4+PixdurRYSdcWib647r//fl566aU8t1+SRK85h+Jc8NqddbyrKqvHd+HZHg34ISaBHjMiWbznpFt11tpk1I1SKgXYBHQAKolI9pkLtQG7X0nCXp1SXl5ejBkzhlmzZt3xt/j4eLp3707z5s3p0aMHJ0+eZMeOHaxcuZIXXniBsLAwfv/991ue8/3339OuXTtatmxJz549SUxMBPKfdjg+Pp7GjRszatQoGjZsyLBhw1i/fj2dOnWiQYMG7N69G8hK5uPGjbtj+++88w5RUVEMGzaMsLAwrl+/zt69e4mIiKB169b06dOHhIQEAPbu3UuLFi1o0aIFH330kVX7TXMMV7zknb34ensysVdDVo/vSqPq5XnxvzE8Mu9njia5SWetUqpEP4A/UMlyuzSwFegPfAc8Ylk+F3iqsHW1bt1a3S42NvaOZfnZsaOe2rSJO3527KhX5HXkpWzZsurixYuqXr16KiUlRb377rtqypQpSiml+vfvrz7//HOllFLz589XAwYMUEopNXLkSPXdd9/lub4LFy6ozMxMpZRSn376qZo4caJSSqlnnnlGTZs2TSml1KpVqxSgkpOT1fHjx5Wnp6fat2+fysjIUK1atVKjR49WmZmZavny5TnbXLBggXr66afz3H5ERITas2ePUkqpmzdvqg4dOqikpCSllFLffPONGj16tFJKqdDQUBUZGamUUur5559XzZo1s2rf2UJx3gOaVlQZGZlq8e6TqvnUNar+Kz+oGWsOqes3040Oq0SAKFWEfG3NOcM1gIUi4knWkcG3SqlVIhILfCMibwC/AnafLN2enVIVKlRgxIgRzJ49m9KlS+cs37lzZ85UwcOHD+fvf/97oes6ffo0Dz/8MAkJCdy8eZOgoCCAAqcdDgoKIjQ0FIBmzZrRo0cPRITQ0FDi4+OL9Vri4uLYv38/vXr1AiAjI4MaNWqQkpJCSkoKXbt2zXk9q1evLta6Nc1ZeHgIQ9rUoXuTaryxKpbZG4/y/b4Epg8MoWP9qkaHZxfWjLrZp5RqqZRqrpQKUUq9bll+TCnVVilVXyn1kFLK7j0f9u6UmjBhAvPnz+fq1atWreeZZ55h3LhxxMTE8Mknn3DjRuFn8N0+1XHuaZCLO+2xUopmzZoRHR1NdHQ0MTExrF27tngvQtNcRNVyPrz/SEu+fLwtmUrx6Ge7mPTtb1y46nqXs3SJM2Pt3SlVpUoVhgwZcsuVnDp27JhzecGvvvqKLl26AHdONZzbxYsXqVUraxDSwoULc5bbctrhgqY+btSoEcnJyezcuROAtLQ0Dhw4QKVKlahUqRLbtm3LeT2a5i66NPBnzYSuPH33XayIPkOPGZtZsve0U16iMz8ukegd0Sk1adKkW0bfzJkzhwULFtC8eXO+/PJLPvjgAyDrSlTvvvsuLVu2vKMzdurUqTz00EO0bt2aqlX/d4hoy2mHb9/+qFGjGDt2LGFhYWRkZLBkyRJefPFFWrRoQVhYWM7FyBcsWMDTTz9NWFiYS73BNa0ofL09eaFPY34c34Vg/3I8/91vPPrpLo4lXyn8yU5AT1OsmZZ+D2hGyMxUfLPnFG+tPkhqWiZP312fsd2C8fHyNDq0O+hpijVN00rAw0N4tF1dNkyKoE9IdWatP8y9H2xl9/ELRodWYjrRa5qm5aFaeV/mDG3JgtFtSE3PZMgnO3lxyT5SrjlfZ61O9JqmaQW4u1E11j7Xlb9GBLPkl9P0mBHJ8l/POFVflk70mqZphShTyouX72nC9+M6U6dKGSYsjmbEf3Zz4rx1Q64dRSd6TdO0ImpaswL//VtH/jmgGdEnU+g9awsfbTrKzfRMo0MrkE70mqZpxeDpIQzvEMj6SRF0b1yNd9fEcd+cbew9Yd7OWp3oC+Dp6UlYWBjNmjWjRYsWzJgxg8zMkn1zT548mfXr1+f797lz5/LFF1+UNNQcmzdvpn///lavpzhGjRrFkiVLCn+gprmQgAq+fPyX1nw2Ipwrqek88PFOXlkWw8XraUaHdgdr5rpxeaVLlyY6OhqApKQkHn30US5dusS0adOKva7XX3+9wL+PHTu2RDE6o/T0dLy89FtPcw09mwbQ4S4/Zq07zH+2H2ftgUSm3NeU/s1rICJGhwfoFn2RVatWjXnz5vHhhx+ilCIjI4MXXniBNm3a0Lx5cz755JOcx77zzjuEhobSokWLnHnic7d6X3rpJZo2bUrz5s15/vnngayzZt977z0AoqOjad++Pc2bN2fQoEE5UyJ069aNF198kbZt29KwYUO2bt2aZ6yXLl2iX79+NGrUiLFjx+YchSxatIjQ0FBCQkJ48cUXcx5frly5nNtLlixh1KhROTE/++yzdOzYkeDg4Jz4lVKMGzeORo0a0bNnT5KSknKe//rrr9OmTRtCQkIYM2ZMzsiEbt26MWHCBMLDw5k+fTpBQUGkpaXlxJv7vqY5m7I+XrzWvykrx3WmRkVfnln0K6M/38OpC9cKf7IDOEWzatr3B4g9e8mm62xaswJT7mtWrOcEBweTkZFBUlISK1asoGLFiuzZs4fU1FQ6depE7969OXToECtWrGDXrl2UKVOGCxdurdudP3+eZcuWcejQIUSElJSUO7YzYsQI5syZQ0REBJMnT2batGm8//77QFZrePfu3fz4449MmzYtz3LQ7t27iY2NpV69evTt25elS5fSsWNHXnzxRfbu3UvlypXp3bs3y5cvZ+DAgq/0mJCQwLZt2zh06BD3338/Dz74IMuWLSMuLo7Y2FgSExNp2rQpjz32GADjxo1j8uTJQNYsmKtWreK+++4D4ObNm2SfAR0fH88PP/zAwIED+eabbxg8eDDe3t7F+n9omtmE1KrI8qc78cXOeN5bE0evWZFM6NmQxzsH4e1pXLtat+hLaO3atXzxxReEhYXRrl07zp8/z5EjR1i/fj2jR4+mTJmsSdaqVKlyy/MqVqyIr68vjz/+OEuXLs15XLaLFy+SkpJCREQEACNHjmTLli05fx88eDAArVu3znea4rZt2xIcHIynpydDhw5l27Zt7Nmzh27duuHv74+XlxfDhg27Zb35GThwIB4eHjRt2jTnQilbtmxh6NCheHp6UrNmTbp3757z+E2bNtGuXTtCQ0PZuHEjBw4cyPnbww8/nHP7iSeeYMGCBUDWPDujR48uNBZNcwaeHsLoTkGsmxhBlwb+vL36EPfN2cavJ0s+WaG1nKJFX9yWt70cO3YMT09PqlWrhlKKOXPm0KdPn1ses2bNmgLX4eXlxe7du9mwYQNLlizhww8/ZOPGjUWOIXuaYk9Pz3ynKb69LlhYnTD332+fOjn3NMmFnSBy48YNnnrqKaKioqhTpw5Tp069ZX1ly5bNud2pUyfi4+PZvHkzGRkZ+hq1msupWak0n44IZ82Bc0xZcYDBH+9gePt6vNCnEeV9HXv0qlv0RZScnMzYsWMZN24cIkKfPn34+OOPc+rKhw8f5urVq/Tq1YsFCxZw7VpWbe720s2VK1e4ePEi9957L7NmzeK333675e8VK1akcuXKOfX3L7/8Mqd1X1S7d+/m+PHjZGZmsnjxYjp37kzbtm2JjIzkjz/+ICMjg0WLFuWsNyAggIMHD5KZmcmyZcsKXX/Xrl1ZvHgxGRkZJCQksGnTJuB/XxJVq1blypUrhY7EGTFiBI8++qhuzWsurU+z6qyb2JWRHQL58ucT9JwZyeqYBIeeWVviFr2I1AG+AAIABcxTSn0gIlWAxUAgEA8MUUoZd8xihevXrxMWFkZaWhpeXl4MHz6ciRMnAlmlh/j4eFq1aoVSCn9/f5YvX07fvn2Jjo4mPDycUqVKce+99/Lmm2/mrPPy5csMGDCAGzduoJRi5syZd2x34cKFjB07lmvXrhEcHJxT4iiqNm3aMG7cOI4ePcrdd9/NoEGD8PDw4O233+buu+9GKUW/fv0YMGAAAG+//Tb9+/fH39+f8PBwrlwpeGrWQYMGsXHjRpo2bUrdunXp0KEDAJUqVeLJJ58kJCSE6tWr06ZNmwLXM2zYMF577TWGDh1arNenac6mvK83U+9vxqCWtXh5aQx/++oXejapxrQBIdSqVLrwFVipxNMUi0gNoIZS6hcRKQ/sBQYCo4ALSqm3ReQloLJS6sUCVqWnKXZTS5YsYcWKFXz55Zd5/l2/BzRXlJ6RyYLt8cxcdxgReOeB5tzXomaJ1lXUaYpL3KJXSiUACZbbl0XkIFALGAB0szxsIbAZKDDRa+7nmWeeYfXq1fz4449Gh6JpDuXl6cGTXYO5J7Q6U1fGElS1bOFPsnabtliJiAQCLYFdQIDlSwDgHFmlHU27xZw5c4wOQdMMVbtyGT4bWWhj3Cas7owVkXLAf4EJSqlbBrurrLpQnrUhERkjIlEiEpWcnGxtGJqmaVo+rEr0IuJNVpL/Sim11LI40VK/z67jJ+X1XKXUPKVUuFIq3N/fP8/1O9N8z5pt6f+9ptlOiRO9ZA2+ng8cVErlHjqyEhhpuT0SWFGS9fv6+nL+/Hn9gXdDSinOnz+Pr6+v0aFomkuwpkbfCRgOxIhItGXZK8DbwLci8jhwAhhSkpXXrl2b06dPo8s67snX15fatWsbHYamuQRrRt1sA/I75bJHSdebzdvbm6CgIGtXo2ma5vb0mbGapmkuTid6TdM0F6cTvaZpmosr8RQINg1CJJmsjlt7qAr8Yad125IzxKljtA1niBGcI053j7GeUirv8em5mCLR25OIRBVlLgijOUOcOkbbcIYYwTni1DEWjS7daJqmuTid6DVN01ycOyT6eUYHUETOEKeO0TacIUZwjjh1jEXg8jV6TdM0d+cOLXpN0zS35lKJXkR8RWS3iPwmIgdEZJpleZCI7BKRoyKyWERKmSBWTxH5VURWmTFGEYkXkRgRiRaRKMuyKiKyTkSOWH5XNjJGS0yVRGSJiBwSkYMi0sFMcYpII8s+zP65JCITzBSjJc7nLJ+Z/SKyyPJZMtt7crwlvgMiMsGyzPD9KCL/EZEkEdmfa1mecUmW2ZZ9uk9EWjkiRpdK9EAq0F0p1QIIA/qKSHvgHWCWUqo+8CfwuIExZhsPHMx134wx3q2UCss1NOwlYINSqgGwwXLfaB8APymlGgMtyNqnpolTKRVn2YdhQGvgGrDMTDGKSC3gWSBcKRUCeAKPYKL3pIiEAE8Cbcn6P/cXkfqYYz9+DvS9bVl+cd0DNLD8jAE+dkiESimX/AHKAL8A7cg6WcHLsrwDsMbg2Gpb/vndgVVkTQ5nthjjgaq3LYsj6zrBADWAOINjrAgcx9LXZNY4c8XVG9huthjJugToKaAKWRMdrgL6mOk9CTwEzM91/x/A382yH4FAYH+u+3nGBXwCDM3rcfb8cbUWfXZJJJqsC56sA34HUpRS6ZaHnCbrjW2k98l6k2Za7vthvhgVsFZE9orIGMsys10mMghIBhZYymCfiUhZzBdntkeARZbbpolRKXUGeA84SdZ1oC8CezHXe3I/0EVE/ESkDHAvUAcT7cfb5BdX9pdqNofsV5dL9EqpDJV1mFybrMO8xgaHdAsR6Q8kKaX2Gh1LITorpVqRdaj5tIh0zf1HldUcMXrIlhfQCvhYKdUSuMpth+4miRNLfft+4Lvb/2Z0jJb68QCyvjhrAmW5sxRhKKXUQbJKSWuBn4BoIOO2x5jif307M8Tlcok+m1IqBdhE1iFnJRHJnnu/NnDGsMCyLthyv4jEA9+QVb75AHPFmN3KQymVRFZNuS1FvEykA50GTiuldlnuLyEr8ZstTsj6wvxFKZVouW+mGHsCx5VSyUqpNGApWe9Ts70n5yulWiulupLVZ3AYc+3H3PKL6wxZRyLZHLJfXSrRi4i/iFSy3C4N9CKrc24T8KDlYSW+vKEtKKVeVkrVVkoFknUov1EpNQwTxSgiZUWkfPZtsmrL+7HRZSJtRSl1DjglIo0si3oAsZgsTouh/K9sA+aK8STQXkTKiIjwv/1omvckgIhUs/yuCwwGvsZc+zG3/OJaCYywjL5pD1zMVeKxHyM6LuzYIdIc+BXYR1ZimmxZHgzsBo6SdejsY3Sslri6AavMFqMllt8sPweAVy3L/cjqRD4CrAeqmGAfhgFRlv/5cqCy2eIkqxRyHqiYa5nZYpwGHLJ8br4EfMz0nrTEuJWsL6DfgB5m2Y9kfYEnAGlkHWU+nl9cZA28+IisvsMYskY62T1GfWaspmmai3Op0o2maZp2J53oNU3TXJxO9JqmaS5OJ3pN0zQXpxO9pmmai9OJXtM0zcXpRK9pmubidKLXNE1zcf8PRSdlK5otj3MAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Find indices of positive and negative examples\n",
"pos = np.where(y==1)[0]\n",
"neg = np.where(y==0)[0]\n",
"\n",
"# Plot examples\n",
"plt.plot(X[pos, 1], X[pos, 2], 'b+', label='Admitted')\n",
"plt.plot(X[neg, 1], X[neg, 2], 'yo', label='Not admitted')\n",
"plt.legend()\n",
"\n",
"# Plot the decision boundary\n",
"plot_x = np.array([min(X[:,1])-2, max(X[:,1])+2])\n",
"plot_y = (-1./theta[2]) * ((theta[1] * (plot_x) + theta[0]))\n",
"\n",
"plt.plot(plot_x, plot_y, label='Decision boundary')\n",
"\n",
"# Legend, specific for the exercise\n",
"plt.legend()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Evaluation\n",
"#### Probability that a given student will be admitted"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"For a student with scores 45 and 85, we predict an admission probability of 0.78\n"
]
}
],
"source": [
"prob = sigmoid(np.array([1, 45, 85]).dot(theta))\n",
"print('For a student with scores 45 and 85, we predict an admission probability of {:.2}'.format(prob))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Accuracy\n",
"It's often a good idea to see how well your model trained. You can do that by checking how much datapoints we can predict correctly using $\\theta$. In a real world application, you should consider splitting your data (eg 80% - 20%) and test on data the model has not seen before. This gives you more realistic insight in how your model would perform in the real world - and that's your ultimate goal ;)\n",
"\n",
"In this example, we expect a training accuracy of 89.0%."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training Accuracy: 89.0%\n"
]
}
],
"source": [
"p = np.zeros((m, 1))\n",
"for (i, example) in enumerate(X):\n",
" prob = sigmoid(np.array(example.dot(theta)))\n",
" if prob >= 0.5:\n",
" p[i] = 1\n",
" else:\n",
" p[i] = 0\n",
"print('Training Accuracy: {}%'.format(np.mean(p == y.reshape((m, 1))) * 100))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Regularized linear regression\n",
"\n",
"---\n",
"Regularization is a meganism for preventing overfitting. _Overfitting_ means that our model works extremely well on the training set but bad in the real world. It's focussed on the training data. _Underfitting_ is either a not well trained model or the feature mapping is not done (correctly). We will use regularization in this exercise. Furtermore, we are going to look at a more complex decision boundary\n",
"\n",
"In this part of the exercise, you will implement regularized logistic regression to predict whether microchips from a fabrication plant passes quality assur- ance (QA). During QA, each microchip goes through various tests to ensure it is functioning correctly.\n",
"\n",
"Suppose you are the product manager of the factory and you have the test results for some microchips on two different tests. From these two tests, you would like to determine whether the microchips should be accepted or rejected. To help you make the decision, you have a dataset of test results on past microchips, from which you can build a logistic regression model."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0.051267, 0.69956 ],\n",
" [-0.092742, 0.68494 ],\n",
" [-0.21371 , 0.69225 ],\n",
" [-0.375 , 0.50219 ],\n",
" [-0.51325 , 0.46564 ]])"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# start by loading the data\n",
"data = pd.read_csv(\"ex2data2.txt\", header = None, \n",
" names = [\"Test 1\", \"Test 2\", \"Status\"])\n",
"\n",
"# initialize some useful variables\n",
"m = len(data[\"Status\"])\n",
"size = np.array((data[\"Test 1\"]))\n",
"bedrooms = np.array((data[\"Test 2\"]))\n",
"X = np.array([size, bedrooms]).T # don't add a column of ones yet.\n",
"y = np.array(data[\"Status\"])\n",
"\n",
"X[:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualising the data"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3XuQFPW58PHvA8oSozkC8iKKXDZBFEQXRc1GoyhEyA1MaRSDCl6KF49Yb7wk4ksdWTxFRY+pItGY1xgJkMQACcfLJtEyXiBEwcQlZyOIISIuiiJsQEyMAgLP+8f0rj2zMzsz2/fu51M1tTN9mf5N70w//buLqmKMMca06RZ1AowxxsSLBQZjjDF5LDAYY4zJY4HBGGNMHgsMxhhj8lhgMMYYk8cCgzHGmDwWGIwxxuSxwGCMMSbPIVEnoCuOOuooHTx4cNTJMMaYRFm7du3fVbVvue0SGRgGDx5MU1NT1MkwxphEEZEtlWxnRUnGGGPyWGAwxhiTxwKDMcaYPBYYjDHG5LHAYIwxJo8FBpMK27c/xJo1g1m5shtr1gxm+/aHok6SMYmVyOaqxrht3/4QGzdO5+DBDwDYu3cLGzdOB6BfvylRJs2YRLIcg0m8zZtntweFNgcPfsDmzbMjSpExyWaBwSTe3r1vVLU86azYzATNAoNJvJqagVUtT7K2YrO9e7cA2l5sZsHB+MkCg0m82tp5dOt2WN6ybt0Oo7Z2XkQpCo4Vm5kwWGAwidev3xSGDXuAmppBgFBTM4hhwx5IZcVz1orNTDSsVZJJhX79pqQyEBSqqRnoFCN1XG6MXyzHYEyCZKnYzETHAoMxCZKlYjMTHStKMiZhslJsZqLjS45BRH4iIjtEZH2J9SIi94jIJhF5SUROda2bKiKvOo+pfqTHGGNM1/lVlLQImNDJ+i8CQ53HdOD/AYhIb2AOcCZwBjBHRHr5lKbMsA5P1bNzZkxpvgQGVV0F7Opkk0nATzXnBeBIEekPjAeeUtVdqvou8BSdBxhTIKsdnrxc2LN6zoypVFiVz8cCb7peb3WWlVpuKpTFDk9eL+xZPGfGVCMxrZJEZLqINIlIU2tra9TJiY0sdnjyemHP4jkzphphBYa3gONcrwc4y0ot70BVH1DV0ao6um/fvoElNGniOk5QQ0Nw7+31wh7Xc2ZMXIQVGBqBK53WSZ8F3lPVbcCTwAUi0supdL7AWWYqFNcOT3PnBvfeXi/scT1nxsSFX81VlwBrgGEislVErhGRGSIyw9nkcWAzsAn4MfDvAKq6C/hP4EXncYezzFQoix2evF7Ys3jOjKmGqGrUaaja6NGjtampKepkmAINDcVzCnPm+F+0tH37Q2zePJu9e9+gpmYgtbXzEnFhT2q6TTqIyFpVHV12OwsMBvy/YIlAAr9aHfh5XgqnIIVcTsdyKyYslQaGxLRKMsGxdv3F+X1erJmsSQoLDAnlZ8/dIC5Yc+Z0edfY8Pu8WDNZkxQWGBLI7zvZIC5YQTZXDYvf58WayVbGhiuJngWGBPL7TtYuWMX5fV6smWx5VqwZDxYYEsjvO1m7YBXn93mxZrLlWT1MPNh8DAnk9/SObRcma0aZL4jzYnMpdM7qYeLBAkMC1dbOK9rs0csdvl2wirPzEi6b0zoerCgpgaxIwqSVFWvGg+UYEsruZE0aWbFmPFhgMCaD4jw0h930RM+KkkzqpaFPhZ+sSagpxwKDSb0ghwBPImsSasqxwGBiy+70g2FNQk05FhhMbHm5029oyI3wKpJ73fbcgo31dDflWWAwvojbBbehITfsd9vQ323P/Uyn32P6hDVGkDUJNeVYYDC+8KscPyl3+n5X4IZZIWz9YEw5vkzUIyITgO8D3YEHVfXOgvXzgfOcl4cB/0tVj3TWHQDWOeveUNWJ5Y5nE/XETxAT8/j1ng0N/geWNWsGl+ihO4j6+pbI3y8scW72ajoKbaIeEekO3Ad8ERgOXCYiw93bqOqNqlqnqnXAvcDDrtUftq2rJCiY+EjK3X0Q6fG7AjeJFcLW7DW9/ChKOgPYpKqbVXUfsBSY1Mn2lwFLfDiuiVjQ5fhxnuzH7wrcJFYIW7PX9PIjMBwLvOl6vdVZ1oGIDAKGAM+6FvcUkSYReUFELvQhPSYl4pbzcPO7AjeJFcJ+5nJscp54CXtIjMnAclU94Fo2SFXfEpFa4FkRWaeqrxXuKCLTgekAAwfG9y4qq+J8dx8Ev8f0SeIYQX6NhNpWJNWW+2grkgJi/fnTzHPls4jUAw2qOt55fRuAqn6nyLb/A1yvqqtLvNci4DequryzY1rlszHRK7ygQy6XU20Lp6RWvCdRaJXPwIvAUBEZIiI9yOUKGosk6ASgF7DGtayXiNQ4z48CzgI2+JCmVAgyex3nrHvcipDilp648KvZaxIr3tPOc2BQ1f3ATOBJ4BXgl6r6sojcISLuVkaTgaWan0U5EWgSkb8AK4A7VdUCA8G2+IhLa5JSF9y4jW0Ut/SUE2Yg69dvCvX1LYwZc5D6+pYuFf0kseI97XzpxxC2uBQlBdmGO8jsdVyy7qX6KXSl/0IQfRXaBNFHI0hJS69fRVKmvDCLkjIp6LvuILPXccy6e+0T4fddfVL6aAQtjM9rPbHjxwJDFwXdhjvI7HWUWfdSF1wIfmyjaoQx1pKfggpkYRWj+VEk5VWc693CZoGhi4K+6w6yXXuUbeb9vODaXf3HkhbI4iYu9W5xYYGhi4K+6w4yex33rPu551a2XVgXw6z10YhrwA3yjt56ceezyucuimuFWZCVsH4rldauVJ4mrcI1SH5+B+JyXoP+va1c2Q0o9kGFMWMOen7/uLDK54DF9a47SU0r/QxgWbur70xSbgyqkeQ6vSSywOBBHCrM0sJr8UUaL4blhFFZGpeAm+Q6vSSywJACcS0TroZVnlYnrMrSuJz/JNfpJZHVMUTM7zqBuJQJe5GGzxC0uHRS9KrSTqJxrdNLGqtjSIgk1QmEJS7FF3EWx06K1aom12N39OGywJAyabioxqX4wqsgP0dnRStJOX/VVihbnV54LDBEIMg6gaRcFLIgyNxgZ5WlScmFpiHXk1YWGCJgFa3GqzQUrVgT0fiywGASo7PAGZegGmYLMXfRypNPtnD00VMS1TLNmojGl7VKqpLfQ20nqady1DprrRTHlkzF0hTG/zuO56KUIIeuNx1V2irJAkMVrMlctNIQGMJIZxzPhYkHa64aABtoK3ydFc3EvWNfV1uIeU1/GlqmmWhZjqEKWRloK66SlmNo09BQvIXSnDn+DSJoTCVCzTGIyAQR2Sgim0RkVpH100SkVUSance1rnVTReRV5zHVj/QEJSutKOJyx50W1gotejYJT3U8BwYR6Q7cB3wRGA5cJiLDi2y6TFXrnMeDzr69gTnAmcAZwBwR6eU1TUEJohVFHL+wcW0H31kRSdKLT+JeLJZkNglP9fzIMZwBbFLVzaq6D1gKTKpw3/HAU6q6S1XfBZ4CJviQpkD43XbcvrDVSUJz1XJKBbC05irikH6rG6yeH4HhWOBN1+utzrJCF4nISyKyXESOq3JfRGS6iDSJSFNra6sPye4aP7vlx+kL69cdaxwuBHGWtfMTh9yn9bCuXlitkn4NDFbVk8nlChZX+waq+oCqjlbV0X379vU9gVHw+oX18yLj1x1rHC4ESZf0YrG4yUrdoJ/8CAxvAce5Xg9wlrVT1Z2qutd5+SBwWqX7ppnXL6xdhNMp6bmKuNWXWA/r6vkRGF4EhorIEBHpAUwGGt0biEh/18uJwCvO8yeBC0Skl1PpfIGzLBPi+oWt9o41bhcCE62o60sKj5OGcaXC5ks/BhH5EvA9oDvwE1WdJyJ3AE2q2igi3yEXEPYDu4DrVPWvzr5XA//Xeat5qrqw3PHSNFFPtUMCVNsmPmzWBt+4RfF9sO9gaTYkRgbE8QcQxzSZ6EQxFph9B0uzITFMJKzi1LiFWXxkxZn+sRxDgtnIrMZ0ZDmG0izHkAEWFLyzc2hMRxYYTKZZk9/0seJM7ywwGGNSxXKB3llgMJljFZXGdM4Cg4lUFBfjqDtgmXSK40jJXWWBwUTKyvhNGqRtpGQLDCbTrKLS+CFOIyX7wQJDyKy4Il5l/Pb/MH5I29De1sEtZNb5Jp+dD5MGa9YMdoqR8tXUDKK+viX8BJVgHdwKpKliKCx2N21MZeI6UnJXZSIwRF0xFKeik2qEUTFsZfwmDdI2tHcmipK6ks2rdjjsSiWp6CRJaTXGlGdFSS7VVgxFncOIUlJzN8YY/2QiMFQ7hWaQTc/iXnRinb9MVlk95McyERiqrRgKsumZXWBN1sXxN5DlUoJifAkMIjJBRDaKyCYRmVVk/U0iskFEXhKRZ0RkkGvdARFpdh6Nhfv6odqKoWpzGGkV99yNSaY49nZPWwc1rw7x+gYi0h24D/gCsBV4UUQaVXWDa7P/AUar6gcich3wX8ClzroPVbXOazrK6ddvSsWVx7W189i4cXreFyXJTc+6Ko53dsYEIW0d1LzyI8dwBrBJVTer6j5gKTDJvYGqrlDVtqvsC8AAH44bmLQ1PTMmanFv1GClBPk85xiAY4E3Xa+3Amd2sv01wBOu1z1FpAnYD9ypqo/6kCbPqslhGGM6556GNo7NoK2UIF+olc8icjkwGrjbtXiQ0672G8D3ROTTJfadLiJNItLU2toaQmrTJy53Z8bEjZUS5PMjMLwFHOd6PcBZlkdExgGzgYmqurdtuaq+5fzdDKwERhU7iKo+oKqjVXV03759fUh2cnX1Ah/HSj+TPXFt1NCv3xTq61sYM+Yg9fUtmQ0K4E9geBEYKiJDRKQHMBnIa10kIqOAH5ELCjtcy3uJSI3z/CjgLMBdaW2KsAu8STLLucaf58CgqvuBmcCTwCvAL1X1ZRG5Q0QmOpvdDRwO/KqgWeqJQJOI/AVYQa6OwQKDj+Je6WeMiZ9MjJWUBg0NxXMKc+ZUfpGPY6WfMSY8NlZSyqR1qIqkp9+YNLLAkCFxrPSz+hJTyG4WomeBIYG6eoG3H5xJArtZiJ4FhgRK+gXeKsSNiTcLDCZ0aa0vMV1nNwvxYq2STKSspZQpZN+J4FirJJMIcawQNybrLDCYSFlRgSlkNwsdhT27nB+jqxpjjG/sZiFf2+xybSO/ts0uBwQ2npPlGIwxJkBe7/ajmF3OcgzGGBMQP+72o5hdznIMxhgTED/u9qOYXc4CgzHGBMSPu/3a2nl063ZY3rKgZ5ezwGCMMQHx424/itnlLDB4EHYTsqBYKxBjguHX3X7Ys8tZYOiitkqlvXu3ANpeqZTE4OB10DILLMYUl9S5pG1IjC5as2awExTy1dQMor6+JfwEeeB1CAIbwsCYZAh1SAwRmSAiG0Vkk4jMKrK+RkSWOev/KCKDXetuc5ZvFJHxfqQnDH43IQv7rtsGLTPVsu9GZdJQxOw5MIhId+A+4IvAcOAyERlesNk1wLuq+hlgPnCXs+9wYDIwApgA/NB5v9jzuwlZ2GPQex3h1AJL9tg8CeWlpYjZjxzDGcAmVd2sqvuApcCkgm0mAYud58uBsSIizvKlqrpXVV8HNjnvF3tRNCGLExs625iOouilHAQ/AsOxwJuu11udZUW3UdX9wHtAnwr3jSU/KpXictcdxaBlFkCSIS7f0aSIopdyEBIzJIaITAemAwwcGFyPv2r06zfFU+uChoaPf2BRVuB6/ZF3JbDMnWsXlyQI+zu6fftDbN48m71736CmZiC1tfNi34LHraZmYIlGKfG4ZlXKjxzDW8BxrtcDnGVFtxGRQ4B/A3ZWuC8AqvqAqo5W1dF9+/b1IdnGL3aBN35IQ/l8WoqY/QgMLwJDRWSIiPQgV5ncWLBNIzDVeX4x8Kzm2sk2ApOdVktDgKHAn3xIU+JkYQx6K5ZItqC/o2kon09qv4VCvvRjEJEvAd8DugM/UdV5InIH0KSqjSLSE/gZMArYBUxW1c3OvrOBq4H9wDdV9Ylyx4tDPwbzsa5k/63vgym0cmU3oNiXQhgz5mDYyUmlSvsx+FLHoKqPA48XLLvd9XwP8PUS+84DkpXP8ijp5ahuUUwiYtIpLeXzaWBDYoQsDeWobl3N/meh6MxUJy3l82lggaEKfvRoTEM5qltXm+dZvYIpFNfy+TT0ZK5WYpqrRs2vIpO0tHNuY9l/4yevTcD9ltWiUssxVMivO/0oZmMKkmX/TZqlLYdfKQsMFfLrTj9tF9K4Zv+N8UPacviVsqKkCvlVZNJ2wUxLqySIX/bfGL9ktajUAkOFamvn5ZU1Qtfv9IO8kKapKawxUfPzd58kVpRUoSQUmaStKawxUbdeS8LvPgg2g1uKpGlWOWPAesj7LdQZ3Ew8ZLWiLCxR370aExYLDCmStqawcWMzmIXDBluMngWGFElbU1iv7EISnCDPrV+zA9r/v+ssMKRIFBVlcf7x+XGHb3evxSUh95SENMaVVT4bT+JcOeh32uL8WcMW1rlwzyBXLft/dWSVzxHK4qBblQjjLtvu8IMTxbntSvFRWGlM8+/ccgw+Kxx0C3Ll/Glq+9zQUDybPmdO5z/AsO/g/D6el7vXtEnC3XiQaUzq77zSHIMFBp9lrS9BNT++pAeGOIoqWCXh3AaZxqT+zq0oKSLWlyBflEU7WZgMKKoK1iSc2yDTmPbfuafAICK9ReQpEXnV+duryDZ1IrJGRF4WkZdE5FLXukUi8rqINDuPOi/piYOs9SUo9+Pzq+lhV1ixT3CS8P8LMo1p/517zTHMAp5R1aHAM87rQh8AV6rqCGAC8D0ROdK1/luqWuc8mj2mx3fVVjAlrS+B1wo0u/iGLysV7HFubpq033m1vAaGScBi5/li4MLCDVT1b6r6qvP8bWAH0NfjcUPRlUHpkjToVtiD7iWh+CEJosyFmZwk/c67wlPls4jsVtUjnecCvNv2usT2Z5ALICNU9aCILALqgb04OQ5V3Vti3+nAdICBAweetmVLx4ofvyW1gqkSDQ0wfnyyPp+1CuooCZXA1ehqizdTGd9aJYnI08DRRVbNBha7A4GIvKuqHeoZnHX9gZXAVFV9wbXsHaAH8ADwmqreUS7RYbVKWrmyG1Ds/AhjxhwM/PhBEoEVK5L1+dJ2EfRDmoOl/b/9V2lgKDtRj6qO6+Qg20Wkv6pucy7yO0ps9yngt8DstqDgvPc25+leEVkI3FIuPWFK++xNaf98WZDWoGCi5bWOoRGY6jyfCjxWuIGI9AAeAX6qqssL1vV3/gq5+on1HtPjqyRUMFVTeVxYafkf/zGPPXvi/fmyUtFqOrI6qeh4rWPoA/wSGAhsAS5R1V0iMhqYoarXisjlwELgZdeu01S1WUSeJVcRLUCzs8/75Y4bZge3OE+V6aX3ZVs2Pc6fr5AVLRjjjfV8zgAvleNJvMgmMc3GxIn1fM4AL70vk5hNT2KajUkiCwwJ5qX3ZRLL6JOYZmOSyAJDgiWhctwYkzwWGBIs7b0vjTHRKNuPwcRbv35TUhsIktRiyqRPlr9/FhhMLBU2xW0bxwnIzI/TRCfr3z8rSjKxtHnz7Lz+GQAHD37A5s2zI0qRyZKsf/8sMJhYKtcUN6gWStbyyUD6J+IpxwJDBiVhEvNyTXGDGqs/LnMAWICKVton4inHAkOEorhAhz0HQ1dlvSluXAJUmyTcTPgp698/CwwRieoCnZSy02JNcdevf4Cjj57i+4B6NlBf55JyM+GnrDcFt7GSIhLVJEBpmWMiqHGTohyPKa6T1KR5wqqssbGSYi6qyq2klp0WFmWMHZu+u9W4TtmZ9YrYLLLAEJGoLtBJLDstVpRx223BFGXYQH0dJfVmwnSdBYaIRHWBTmLZabF6ke7dg6kXifruvE2cAlQSbyaMN9bzOSJtF+IoutwnbRiNLBZlxCVAQbTfVRMNCwwRStoFOio2N3X07LuaLZ6KkkSkt4g8JSKvOn97ldjugIg0O49G1/IhIvJHEdkkIsuc+aGNyWNFGcaEy2sdwyzgGVUdCjzjvC7mQ1Wtcx4TXcvvAuar6meAd4FrPKbHpFAS60WMSTJP/RhEZCMwRlW3iUh/YKWqDiuy3fuqenjBMgFagaNVdb+I1AMNqjq+3HHT0I/BpFtDQ7zqCYyB8Pox9FPVbc7zd4B+JbbrKSJNIvKCiFzoLOsD7FbV/c7rrcCxHtNjTCzEbUiLsCU1KGZt6I9SylY+i8jTwNFFVuW1FVRVFZFS2Y9BqvqWiNQCz4rIOuC9ahIqItOB6QADB3asdPzoo4/YunUre/bsqeZtTYB69uzJgAEDOPTQQ6NOivFJpTmhuXODHQE3iPfO+hwMbqEUJRXsswj4DfDf+FiU9Prrr3PEEUfQp08fpG3QGxMZVWXnzp3885//ZMiQIVEnJxRxHdLCT5UOGRLk0CJBvXcWhv4IqyipEZjqPJ8KPFYkIb1EpMZ5fhRwFrBBcxFpBXBxZ/tXas+ePRYUYkRE6NOnT6ZycHEd0iIsSR+MMIv9ZUrxGhjuBL4gIq8C45zXiMhoEXnQ2eZEoElE/kIuENypqhucdbcCN4nIJnJ1Dgu8JMaCQrzY/yPeKr1gV3rBDzIw+h10itUl2NAfH/MUGFR1p6qOVdWhqjpOVXc5y5tU9Vrn+WpVHamqpzh/F7j236yqZ6jqZ1T166q619vHid6jjz6KiPDXv/616Ppp06axfPnyit/v7bff5uKLc5mq5uZmHn/88fZ1K1euZPXq1VWncfDgwfz973+vej9TuTgNaVFKpRXkccgJ+ZmGUsOI9+nzJesv48j8WEl+f7mXLFnC2WefzZIlS3x5v2OOOaY9kPgVGLIiyhYmDQ3Rt3CJ8vhxDoyl5iTZufNx6y/jyHxg8LNZ4fvvv89zzz3HggULWLp0KZCrhJ05cybDhg1j3Lhx7Nixo337wYMHc9ttt1FXV8fo0aP585//zPjx4/n0pz/N/fffD0BLSwsnnXQS+/bt4/bbb2fZsmXU1dVx1113cf/99zN//nzq6ur4wx/+QGtrKxdddBGnn346p59+Os8//zwAO3fu5IILLmDEiBFce+21JHEOjmpFPblMXI//wx8+5KlIptQFvzAIXXddcJ/Ta9DprC6hX78p1Ne3MGbMQerrWzIZFMDGSvLVY489xoQJEzj++OPp06cPa9euZcuWLWzcuJENGzawfft2hg8fztVXX92+z8CBA2lububGG29k2rRpPP/88+zZs4eTTjqJGTNmtG/Xo0cP7rjjDpqamvjBD34AwIcffsjhhx/OLbfcAsA3vvENbrzxRs4++2zeeOMNxo8fzyuvvMLcuXM5++yzuf322/ntb3/LggWeqnISobOZ6sL4sVdz/CCaX5Y6/qhRs1HNHb8rrXuKpTPsZp5ez5WNvVVeJnMMQbWeWLJkCZMnTwZg8uTJLFmyhFWrVnHZZZfRvXt3jjnmGM4///y8fSZOzI0QMnLkSM4880yOOOII+vbtS01NDbt3767q+E8//TQzZ86krq6OiRMn8o9//IP333+fVatWcfnllwPw5S9/mV69ig5plSpRtzCp5vhBdIYL8/MnZbrYNjb2VnmZzDG479D8ahO9a9cunn32WdatW4eIcODAAUSEr33ta53uV1NTA0C3bt3an7e93r9/f6ndijp48CAvvPACPXv2rP4DpEzUd4VJOL5f9QBRB+Fq2TDi5WUyxxCE5cuXc8UVV7BlyxZaWlp48803GTJkCH369GHZsmUcOHCAbdu2sWLFii4f44gjjuCf//xnydcXXHAB9957b/vr5uZmAM455xx+8YtfAPDEE0/w7rvvdjkNSRH1XWG54wfd5r+Sz+/XsZLYzNPqEjqX+cDg113TkiVLOuQOLrroIrZt28bQoUMZPnw4V155JfX19V0+xnnnnceGDRuoq6tj2bJlfPWrX+WRRx5pr3y+5557aGpq4uSTT2b48OHtFdhz5sxh1apVjBgxgocffrjokCJBiqJ1TNQjspY7ftBNQMP8/FEHYeM/T0NiRKXYkBivvPIKJ554YkQpMqWsX9/Erl3n5pVBd+t2WGabARYT5PARYdm+/SErmkmASofEyGQdgwnP/v3vRto6KAni3Oa/UjbDW7pkvijJBEv1QNHlca2YjEJSxhIy2WGBwQRKpHvR5XGumDQm6ywwmEAdckgvq5g0JmEsMJhAde/+SRt/xpiEscpnEzirmDQmWSzH4CMR4eabb25//d3vfpeGMjWLjz76KBs2bOh0m3KqHUa7sbGRO++8s+jxFy1axNtvv13V8dsG+jPGpENmA0MQna5qamp4+OGHq7pI+xEYqjVx4kRmzZpV9PhdCQzGmHTJZGAIakjkQw45hOnTpzN//vwO61paWjj//PM5+eSTGTt2LG+88QarV6+msbGRb33rW9TV1fHaa6/l7fPrX/+aM888k1GjRjFu3Di2b98OlB5Gu6WlhRNOOIFp06Zx/PHHM2XKFJ5++mnOOusshg4dyp/+9Ccgd/GfOXNmh+PfddddNDU1MWXKFOrq6vjwww9Zu3Yt5557Lqeddhrjx49n27ZtAKxdu5ZTTjmFU045hfvuu8/TeTPGxIyqdvkB9AaeAl51/vYqss15QLPrsQe40Fm3CHjdta6ukuOedtppWmjDhg0dlpWyevUgXbGCDo/VqwdV/B7FfPKTn9T33ntPBw0apLt379a7775b58yZo6qqX/nKV3TRokWqqrpgwQKdNGmSqqpOnTpVf/WrXxV9v127dunBgwdVVfXHP/6x3nTTTaqqesMNN+jcuXNVVfU3v/mNAtra2qqvv/66du/eXV966SU9cOCAnnrqqXrVVVfpwYMH9dFHH20/5sKFC/X6668vevxzzz1XX3zxRVVV3bdvn9bX1+uOHTtUVXXp0qV61VVXqarqyJEj9fe//72qqt5yyy06YsSIop+hmv+LyY533vm58zsUXb16kL7zzs+jTlImAE1awTXWa+XzLOAZVb1TRGY5r28tCDwrgDoAEekNbAJ+59rkW6pa+VyXPghyNMhPfepTXHnlldxzzz184hOfaF++Zs0aHn74YQCuuOIKvv3tb5d9r61bt3LppZeybds29u3bx5AhQwBYtWpV+3sVDqM9ZMgQRo4PSX/fAAAI5UlEQVQcCcCIESMYO3YsIsLIkSNpaWmp6rNs3LiR9evX84UvfAGAAwcO0L9/f3bv3s3u3bs555xz2j/PE088UdV7J5kN/+BN2PM3mOp5LUqaBCx2ni8GLiyz/cXAE6r6QZntAhX0aJDf/OY3WbBgAf/61788vc8NN9zAzJkzWbduHT/60Y/Ys2dP2X0Kh+52D+td7TDeqsqIESNobm6mubmZdevW8bvf/a78jikW9cxsaZC0+RuyyGtg6Keq25zn7wD9ymw/GSicDHmeiLwkIvNFpKbYTn4LejTI3r17c8kll+TNlPa5z32ufbrPhx56iM9//vNAx6Gz3d577z2OPfZYABYvXty+3M9htDsbynvYsGG0trayZs0aAD766CNefvlljjzySI488kiee+659s+TFXG4qEU9l7RXSZu/IYvKBgYReVpE1hd5THJv55RflRwjUkT6AyOBJ12LbwNOAE4nV19xa5Fd2/afLiJNItLU2tpaLtmdCmNI4ptvvjmvddK9997LwoULOfnkk/nZz37G97//fSA309vdd9/NqFGjOlQ+NzQ08PWvf53TTjuNo446qn25n8NoFx5/2rRpzJgxg7q6Og4cOMDy5cu59dZbOeWUU6irq2P16tUALFy4kOuvv566urpMzCHdJuqLWhpyLEmcvyFrPA27LSIbgTGqus258K9U1WEltv0/wAhVnV5i/RjgFlX9Srnj2rDbyZG2/8uaNYNLzIw2iPr6ltQf3w+FdQxgQ7GHpdJht70WJTUCU53nU4HHOtn2MgqKkZxggogIufqJ9R7TY0ygop6UJuocix+inkTJlOe1VdKdwC9F5BpgC3AJgIiMBmao6rXO68HAccDvC/Z/SET6AkKuueoMj+kxJlBRzxcc9VzSfrFhUuLNU2BQ1Z3A2CLLm4BrXa9bgGOLbHe+l+MbE4UoL2q1tfOKFsPYaLXGT6nq+ZylStAksP+H/6wYxoQhNaOr9uzZk507d9KnTx9yVRYmSqrKzp076dmzZ9RJSR0rhjFBS01gGDBgAFu3bsVrU1bjn549ezJgwICok2GMqVJqAsOhhx7aPmSEMcaYrktVHYMxxhjvLDAYY4zJY4HBGGNMHk9DYkRFRFrJdagLy1FA5dOyhc/S542lzxtLnzdhpm+QqvYtt1EiA0PYRKSpkvFFomLp88bS542lz5s4ps+KkowxxuSxwGCMMSaPBYbKPBB1Asqw9Hlj6fPG0udN7NJndQzGGGPyWI7BGGNMHgsMDhHpLSJPicirzt9eRbY5T0SaXY89InKhs26RiLzuWlcXdvqc7Q640tDoWj5ERP4oIptEZJmI9Ag7fSJSJyJrRORlZ57vS13rAjl/IjJBRDY6n3tWkfU1zvnY5Jyfwa51tznLN4rIeD/SU2XabhKRDc65ekZEBrnWFf0/R5DGaSLS6krLta51U53vw6siMrVw35DSN9+Vtr+JyG7XukDPoYj8RER2iEjRCcgk5x4n7S+JyKmudYGfu06pqj1yxWn/Bcxyns8C7iqzfW9gF3CY83oRcHHU6QPeL7H8l8Bk5/n9wHVhpw84HhjqPD8G2AYcGdT5A7oDrwG1QA/gL8Dwgm3+HbjfeT4ZWOY8H+5sXwMMcd6ne8hpO8/1/bquLW2d/Z8jOH/TgB8U2bc3sNn528t53ivs9BVsfwPwk7DOIXAOcCqwvsT6LwFPkJuo7LPAH8M6d+UelmP42CRgsfN8MbmpRjtzMfCEqn5QZju/VJu+dpIbh/x8YHlX9q9Q2fSp6t9U9VXn+dvADqBsZxsPzgA2qepmVd0HLHXS6eZO93JgrHO+JgFLVXWvqr4ObHLeL7S0qeoK1/frBSDsoWorOX+ljAeeUtVdqvou8BQwIeL0dZheOEiquorczWMpk4Cfas4LwJGSm+44jHPXKQsMH+unqtuc5+8A/cpsP5mOX7J5TpZwvojURJS+niLSJCIvtBVzAX2A3aq633m9lSIz6oWUPgBE5Axyd3mvuRb7ff6OBd50vS72udu3cc7Pe+TOVyX7Bp02t2vI3V22KfZ/9lulabzI+b8tF5Hjqtw3jPThFMMNAZ51LQ7jHHamVPrDOHedSs2w25UQkaeBo4usmu1+oaoqIiWbazlRfSTwpGvxbeQuiD3INT+7FbgjgvQNUtW3RKQWeFZE1pG72Hnm8/n7GTBVVQ86iz2fv7QSkcuB0cC5rsUd/s+q+lrxdwjUr4ElqrpXRP43udxXHKfsnQwsV9UDrmVxOYexk6nAoKrjSq0Tke0i0l9VtzkXrh2dvNUlwCOq+pHrvdvulveKyELglijSp6pvOX83i8hKYBTw3+SyqYc4d8UDgLeiSJ+IfAr4LTDbyT63vbfn81fEW8BxrtfFPnfbNltF5BDg34CdFe4bdNoQkXHkAu+5qrq3bXmJ/7PfF7WyadTcvO9tHiRX19S275iCfVeGnT6XycD17gUhncPOlEp/GOeuU1aU9LFGoK32fyrwWCfbdiirdC6GbeX5FwJFWyIEmT4R6dVWBCMiRwFnARs0V6O1gly9SMn9Q0hfD+ARcuWqywvWBXH+XgSGSq5FVg9yF4fC1ifudF8MPOucr0ZgsuRaLQ0BhgJ/8iFNFadNREYBPwImquoO1/Ki/2cf01ZNGvu7Xk4EXnGePwlc4KS1F3AB+TnsUNLnpPEEcpW4a1zLwjqHnWkErnRaJ30WeM+5QQrj3HUuzJruOD/IlSs/A7wKPA30dpaPBh50bTeYXETvVrD/s8A6che0nwOHh50+4HNOGv7i/L3GtX8tuQvbJuBXQE0E6bsc+Ahodj3qgjx/5Fp+/I3cneBsZ9kd5C62AD2d87HJOT+1rn1nO/ttBL4YwHeuXNqeBra7zlVjuf9zBGn8DvCyk5YVwAmufa92zusm4Koo0ue8bgDuLNgv8HNI7uZxm/Od30qunmgGMMNZL8B9TtrXAaPDPHedPaznszHGmDxWlGSMMSaPBQZjjDF5LDAYY4zJY4HBGGNMHgsMxhhj8lhgMMYYk8cCgzHGmDwWGIwxxuT5//WTskO2mhV/AAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Find indices of positive and negative examples\n",
"pos = np.where(y==1)[0]\n",
"neg = np.where(y==0)[0]\n",
"\n",
"# Plot examples\n",
"plt.plot(X[pos, 0], X[pos, 1], 'b+', label='Admitted')\n",
"plt.plot(X[neg, 0], X[neg, 1], 'yo', label='Not admitted')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Feature Mapping\n",
"\n",
"One way to fit the data better is to create more features from each data point. While the feature mapping allows us to build a more expressive classifier, it also more susceptible to overfitting. This will also add $x_0$."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"def map_feature(X1, X2, degree):\n",
" if not type(X1) == np.ndarray:\n",
" X1 = np.array([X1])\n",
"\n",
" if not type(X2) == np.ndarray:\n",
" X2 = np.array([X2])\n",
"\n",
" assert X1.shape == X2.shape\n",
" \n",
" out = np.ones((len(X1), 1))\n",
" for i in range(1, degree+1):\n",
" for j in range(i + 1):\n",
" new = (X1 ** (i-j) * X2 ** j).reshape(len(X1), 1)\n",
" out = np.hstack((out, new))\n",
" return out\n",
"\n",
"X = map_feature(X[:,0], X[:,1], degree=6)\n",
"m, n = X.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Computing Cost and Gradient\n",
"#### Cost function\n",
"Regularization works by penalizing theta. Theta values can be high when an overfit occurs so we prefer to keep them low. $\\lambda$ is the regularization parameter. If $\\lambda=0$, no regularization happens. If $\\lambda$ is some big number, $\\theta$ has a very high penalty. Just like the learning rate $\\alpha$ you have to try out certain values and see which work.\n",
"\n",
"The cost function with regularization:\n",
"\n",
"$$J(\\theta) = \\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}-\\begin{bmatrix}y^{(i)}\\log h(x^{(i)}+(1-y^{(i)}\\log(1-h_\\theta(x^{(i)})) \\end{bmatrix} + \\frac{\\lambda}{m}\\displaystyle\\sum_{j=1}^{n}{\\theta_j}^2$$\n",
"\n",
"**Exercise**: Implement the regularized cost function."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"def compute_regularized_cost(theta, X, y, _lambda):\n",
" m = len(y)\n",
" regularization = _lambda / (2 * m) * np.sum(theta[1:] ** 2) #np.squared()???/\n",
" cost = 1/m * (-y @ np.log(sigmoid(X @ theta)) - (1 - y) @ np.log(1 - sigmoid(X@theta)))\n",
" return cost + regularization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Gradient descent\n",
"Gradient descent, just like the cost function, is slightly modified for regularization.\n",
"\n",
"For $\\theta_{j}$ where $j = 0$: \n",
"\n",
"$$\\theta_j := \\theta_j -\\alpha \\begin{bmatrix}\\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}(h_\\theta(x^{(i)}-y^{(i)}){x_0}^{(i)}\\end{bmatrix}$$\n",
"\n",
"For $\\theta_{j}$ where $j \\in \\{1, 2, ..., n\\} $: \n",
"\n",
"$$\\theta_j := \\theta_j -\\alpha \\begin{bmatrix}\\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}(h_\\theta(x^{(i)}-y^{(i)})x_j^{(i)} + \\frac{\\lambda}{m}{\\theta_j} \\end{bmatrix}$$\n",
"\n",
"Note that we don't penalize our bias vector $X_1$.\n",
"\n",
"**Exercise**: Implement `compute_regularized_gradient`."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"def compute_regularized_gradient(theta, X, y, _lambda):\n",
" return np.zeros(len(theta))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We expect: $J \\approx 0.693$ and the first five values of $\\theta$: $\\begin{bmatrix} 0.0085 && 0.0188 && 0.0001 && 0.0503 && 0.0115 \\end{bmatrix}$"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cost at initial theta (zeros): 2.13\n",
"Gradient at initial theta (zeros) - first five values only: \n",
"[0.34604507 0.08508073 0.11852457 0.1505916 0.01591449]\n"
]
}
],
"source": [
"_lambda = 1\n",
"initial_theta = np.ones(n)\n",
"\n",
"cost = compute_regularized_cost(initial_theta, X, y, _lambda)\n",
"grad = compute_regularized_gradient(initial_theta, X, y, _lambda)\n",
"\n",
"print('Cost at initial theta (zeros): {:.3}'.format(cost))\n",
"print('Gradient at initial theta (zeros) - first five values only: \\n{}'.format(grad[:5]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Find an optimimal value for theta using conjugate gradient."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Optimization terminated successfully.\n",
" Current function value: 0.529003\n",
" Iterations: 28\n",
" Function evaluations: 76\n",
" Gradient evaluations: 76\n",
"Conjugate gradient found the following values for theta - first five values only: [ 1.27278161 0.62533883 1.18105652 -2.02009622 -0.91762258]\n"
]
}
],
"source": [
"from scipy.optimize import minimize\n",
"result = minimize(compute_regularized_cost, initial_theta, args = (X, y, _lambda),\n",
" method = 'CG', jac = compute_regularized_gradient, \n",
" options = {\"maxiter\": 400, \"disp\" : 1})\n",
"theta = result.x\n",
"print('Conjugate gradient found the following values for theta - first five values only: {}'.format(theta[:5]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Plotting the decision boundary\n",
"We can plot a more complex decision boundary using `np.linspace` and `contour`."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.contour.QuadContourSet at 0x112c08588>"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl4VOXd8PHvnW2yExJCCEsIYQk7UUAWNxQURIT6uEtVbL2oPmJtq09rX1tBn4fXWtundWvVStX6IqJIEReqRUUEwhIU2UGWJARCCNlIyJ653z8yCSFkmeXMmTOT3+e6ciWZOXPOPSeT87vv370cpbVGCCGECPJ1AYQQQliDBAQhhBCABAQhhBAOEhCEEEIAEhCEEEI4SEAQQggBGBQQlFJ/V0qdUkrtbuf5KUqpMqXUDsfXE0YcVwghhHFCDNrPG8CLwD862OZrrfUsg44nhBDCYIa0ELTW64FiI/YlhBDCN4xqIThjklLqO+AE8KjWek/rDZRS84H5AFFRUWOHDh1qYvGEEML/bd++/bTWOtGd15oVEL4B+mutK5RSM4FVwODWG2mtXwVeBRg3bpzOysoyqXhCCBEYlFI57r7WlFFGWuszWusKx8+fAKFKqR5mHFsIIYRzTAkISqleSinl+PkSx3GLzDi2EEII5xiSMlJKLQOmAD2UUnnAQiAUQGv9MnAz8IBSqh6oAm7XssyqEEJYiiEBQWt9RyfPv0jjsFQhhJ+qq6sjLy+P6upqXxdFAOHh4fTt25fQ0FDD9mnmKCMhhB/Ly8sjJiaG1NRUHBlg4SNaa4qKisjLy2PAgAGG7VeWrhBCOKW6upqEhAQJBhaglCIhIcHw1poEBCGE0yQYWIc3/hYSEIQQQgASEIQQfmbVqlUopdi/f3+bz8+bN48VK1Y4vb8TJ05w8803A7Bjxw4++eST5ufWrVvHpk2bXC5jamoqp0+fdvl1viYBQQjhVYsWGbu/ZcuWcdlll7Fs2TJD9te7d+/mAGJUQPBXEhCEEF715JPG7auiooINGzawZMkS3nnnHaBxxM2CBQtIT09n2rRpnDp1qnn71NRUfv3rX5ORkcG4ceP45ptvmD59OgMHDuTll18GIDs7m5EjR1JbW8sTTzzB8uXLycjI4JlnnuHll1/mT3/6ExkZGXz99dcUFhZy0003MX78eMaPH8/GjRsBKCoq4tprr2XEiBHcd999+Os0Kxl2KoTwGx988AEzZsxgyJAhJCQksH37dnJycjhw4AB79+6loKCA4cOH86Mf/aj5NSkpKezYsYOf//znzJs3j40bN1JdXc3IkSO5//77m7cLCwvjqaeeIisrixdfbJw2VVVVRXR0NI8++igAd955Jz//+c+57LLLyM3NZfr06ezbt48nn3ySyy67jCeeeIKPP/6YJUuWmHtiDCIBQQhhuEWLzm8ZNA2IWbjQsxTSsmXLePjhhwG4/fbbWbZsGfX19dxxxx0EBwfTu3dvrr766vNeM3v2bABGjRpFRUUFMTExxMTEYLPZKC0tden4a9euZe/evc2/nzlzhoqKCtavX8/KlSsBuP766+nevbv7b9KHJCAIIQy3aNG5C79SYEQGpbi4mC+++IJdu3ahlKKhoQGlFDfeeGOHr7PZbAAEBQU1/9z0e319vUtlsNvtbN68mfDwcNffgB+QPgQhhF9YsWIFd911Fzk5OWRnZ3Ps2DEGDBhAQkICy5cvp6Ghgfz8fL788ku3jxETE0N5eXm7v1977bW88MILzb/v2LEDgCuuuIK3334bgDVr1lBSUuJ2GXxJAoIQwqsWLjRmP8uWLbugNXDTTTeRn5/P4MGDGT58OHfffTeTJk1y+xhXXXUVe/fuJSMjg+XLl3PDDTfwz3/+s7lT+fnnnycrK4vRo0czfPjw5o7phQsXsn79ekaMGMHKlStJSUnx6L36irJqb7jcIEcIa9m3bx/Dhg3zdTFEC239TZRS27XW49zZn7QQhBBCABIQhBBCOEhAEEIIAUhAEEII4SABQQghBCABQQghhIMEBCGEX1BK8cgjjzT//oc//IFFnayDsWrVqvOWmnCHq0tZr169mt/97ndtHv+NN97gxIkTLh2/afE9M0hAEEJ4RUHBUjIzU1m3LojMzFQKCpZ6tD+bzcbKlStdujgbERBcNXv2bB577LE2j+9OQDCTBAQhhOEKCpZy4MB8ampyAE1NTQ4HDsz3KCiEhIQwf/58/vSnP13wXHZ2NldffTWjR49m6tSp5ObmsmnTJlavXs1//dd/kZGRweHDh897zYcffsiECRO46KKLmDZtGgUFBUD7S1lnZ2czdOhQ5s2bx5AhQ5g7dy5r167l0ksvZfDgwWzduhVovOgvWLDgguM/88wzZGVlMXfuXDIyMqiqqmL79u1ceeWVjB07lunTp5Ofnw/A9u3bGTNmDGPGjOGll15y+5y5TGttya+xY8dqIYR17N271+ltN23qr7/8kgu+Nm3q7/bxo6KidFlZme7fv78uLS3Vzz77rF64cKHWWutZs2bpN954Q2ut9ZIlS/ScOXO01lrfc889+r333mtzf8XFxdput2uttf7b3/6mf/GLX2ittX7ooYf0k08+qbXW+qOPPtKALiws1EePHtXBwcF6586duqGhQV988cX63nvv1Xa7Xa9atar5mK+//rp+8MEH2zz+lVdeqbdt26a11rq2tlZPmjRJnzp1Smut9TvvvKPvvfderbXWo0aN0l999ZXWWutHH31Ujxgxos330NbfBMjSbl53ZbVTIYThampyXXrcWbGxsdx99908//zzREREND+emZnZvPz0XXfdxS9/+ctO95WXl8dtt91Gfn4+tbW1DBgwAKDDpawHDBjAqFGjABgxYgRTp05FKcWoUaPIzs526b0cOHCA3bt3c8011wDQ0NBAcnIypaWllJaWcsUVVzS/nzVr1ri0b3dJykgIYTibre3F3dp73BU/+9nPWLJkCWfPnvVoPw899BALFixg165dvPLKK1RXV3f6mtbLZ7dcWtvVpbS11owYMYIdO3awY8cOdu3axWeffebamzCYBAThM0Z3OgrrSEtbTFBQ5HmPBQVFkpa22ON9x8fHc+utt553V7LJkyc331Jz6dKlXH755cCFy1e3VFZWRp8+fQB48803mx83cinrjpbTTk9Pp7CwkMzMTADq6urYs2cPcXFxxMXFsWHDhub3YxYJCMInvNHpKKwjKWku6emvYrP1BxQ2W3/S018lKWmuIft/5JFHzhtt9MILL/D6668zevRo3nrrLZ577jmg8a5qzz77LBdddNEFncqLFi3illtuYezYsfTo0aP5cSOXsm59/Hnz5nH//feTkZFBQ0MDK1as4Fe/+hVjxowhIyODTZs2AfD666/z4IMPkpGRYer9mWX5a+ETmZmpjmBwPputP5MmZZtfIA8UFCzlyJHHqanJxWZLIS1tsWEXPiuR5a+tx+jlr6VTWfiEtzodzdbU0rHbKwGaWzpAQAYFEdgkZSR8wpudjmY6cuTx5mDQxG6v5MiRx31UIiHcJwFB+IQ3Ox3NFCgtHWdZNcXcFXnjbyEBQfiEtzsdzRIoLR1nhIeHU1RUJEHBArTWFBUVER4ebuh+pQ9B+ExS0ly/CwCtpaUtPq8PAfyzpeOMvn37kpeXR2Fhoa+LImgM0H379jV0nxIQhPBAU0DrCqOMQkNDm2fzisAkAUEIDwVCS0cIMKgPQSn1d6XUKaXU7naeV0qp55VSh5RSO5VSFxtx3K5CZvTKORDCDEZ1Kr8BzOjg+euAwY6v+cBfDTpuwAu0Gb3uXNgD7RwIYVWGBASt9XqguINN5gD/cKzOuhmIU0olG3HsQBdI49zdvbAH0jkQwsrMGnbaBzjW4vc8x2PnUUrNV0plKaWyZCRDIyuMc+/kLoVOc/fCboVzIERXYKl5CFrrV7XW47TW4xITE31dHEuwwjj3J580Zj/uXtitcA6E6ArMCgjHgX4tfu/reEx0IlBm9IL7F/ZAOgdCWJlZAWE1cLdjtNFEoExrnW/Ssf2ar2b0LloESjV+wbmfPUkfuXthN+scyEgm0dUZsvy1UmoZMAXoARQAC4FQAK31y0opBbxI40ikSuBerXWHa1vL8teeM2pZZqXAqNUKvLVUtKf7bb1qKTQGK39cTkN0bZ4sfy33Q7AoK13gjAwI3mDEew2k+zOIrs2TgGCpTmXRyIhx90YO1Vy40OWXmMqI99pVRzJJmky0JAHBgqx2gTNq2Km3GPFeu+JIJpnwJ1qTgGBBcoFzjRHvtSuOZJIJf6I1CQgWJBc41xjxXgPl/gyu6KppMtE+We3UgoxYY78rLcts1HvtaquW2mwp7XSkB14rUjhHRhlZlLeGZwrRRIbaBiZPRhlJC8GirFBbrWtooLS6muLqKoorKymprqKoqorymhoq6+qorKvlbMvvtXXUNjRQr+3U2+002Bu/N30pBcEqiCClUEoRrBRBji9bSAjhwSGEhzR+hYUEEx4SQmRIKNFhNqLDwogOCyPG1vhzTJiNWJuN+IgI4sIjCAsO9um5cpcvA39XakUK50hA6MLqGhrIKz/D0ZISsksbv3LLyjhRfoZTlWc5U1PT7muDlSIqLIyo0FAiQ8OIDA0lMjSUGFsYIUHBhAQpQoKCCQ5ShKgggoMau6vsWmPXmgZtR2tNg73x59qGBqrr6zlTU8OpyrPse28UvW/YRlV9HeU1NTR00pKNDgsjPjyCuIgIuodH0Cs6mj4xsfSOiXF8j6VXdDShFgocrWvoTaN8AFODggQA0URSRl1ERW0t+06fYs+pU+wpPMXuUwUcLimm3m5v3iYmzEb/uDj6xMSSFBVFfEQk8RERxDsusvGRkcSHRxBrsxEWHIxqWteiHYsWuT9kteVkOK011fX1VNTWUl5bQ3ltLeU1NZypqaakupqSqiqKq6soqWr8Kq6q5OTZCk5Xnj+CRgFJUdH069aN1Lju9O8WR2pcHP27xdE/rjvRYWHuFdZNMhlOeIPMVBbn0VpzpKSYzLxjbDtxnD2FBRwtKaHpL90jMpKRPZMY1iORtO7xpMbFkdqtO/EREZ1e5F3hyQxnI2ZHV9fXcaK8nBPl5RwvP8OJ8jOcKC8np6yUnNJSCivPnrd9j8hIBnaPZ1B8AoPjExgUn8Cg+HgSI6MMPS9N1q0LAtp6k4opU+xtPC5E56QPoYvTWnO0tITNecfYfPwYW/Lymi92SVHRjE5KYk76MEYkJjGyZ096RkV3uD9PavaeWLTo/KW2m67BCxe6V57wkFCiatcQfPxxkmpySbGlkDb6XI78bG0tuWWlHC0tJaeshOzSUg4XF7H6wH7KaxvTZRPitnNL8r/oHlJCNT2piv4ZqX3vIT2hB1EetihklI+wGmkh+KkGu52Nx3L59PD3fHn0CCfPVgCNAWBi335M7NOXCX370b9bnMu1W09q560v6k1cvagb0UJwdxSN1prCyrPsz1lC/anHCKK6+bkaeyhv5N3M1tKxpHSLY1iPREb07Mnonr24OLm3S0FCRvkIb5CUURdyovwM7+7ZzXt7d5FfUUFUaChX9B/A5Sn9mdC3H6luBIDWjFrMztcpI09z9O29Xgf3ZlfYCvadLmT/6UJyykqBxo720Um9mNCnHxP79mOsEwHC3VFGMixZtEdSRgFOa826nKMs3fkd63KOorXm8pRUfnPFVVydmoYtxPM/o9HpGk8ZsaCepzNx29tONeTz0wmTmn8/U1PDdyfz2XI8jy3Hj/Hat1m8vH0rwUqR0SuZy1L6c3lKKqOTehESdP7iAO6M8rHC6CQRmKSFYGF2rfnXoYO8tG0L+04XkhgZxa0jRnLr8FH069bNa8c1qoXgq76IJt5qIXT2+sq6OrbnH2dLXh4bj+Wws+AkGoi12ZjcN4Ur+qdyeUoqfWJjXXg3nperibQuApukjAJMvd3ORwf385dtWzhUUkxa9+48OG4is4akmzKO3ur3P3CWpzl6o3L8pdVVbMzNZX1uNhtys8mvaOzvGRyfwNQBA5mWNpCMXskEOZnq82R0kvRbBD5JGfm5ljU2HZzMypMz+Th/KOkJPXhhxixmDBrcPLHLlX25W/uz+v0PnOXpTFyjZvLGhUdw/ZB0rh+SjtaaQ8XFrM/N5oujR/jbN9t4eftWekRGMnPQEG4cOpzRSb067AfyZHRSRyucSkAQ0kLwsbZqbLX2MIJ7PM1Vo37udK2xvX35ovbny2GrVr93Q2tl1dWsyznKZ4cP8fnRw9Q2NDAgrjs3Dx/BzcNHkhgZdcFrPPk7y9yHwCd3TDOZkXeZaqvGFhZUS+TZ510KBu3ty9vr27d1AW5r2Kkrr3eXK8c1gzPvrVt4OHPSh/HSzBvYet/9PD31WnpERvLspg1c9vdXWfDJh2zIzcHeouLmyVLdXek+GcJ10kJwkZG1cK01674KRhlUY/NF7a+t/gZX+iCM7K+wWt+HJ+U5XFzEO3t28f6+PZRWV9O/Wxy3jxzFTcNG0iMysvMdtMMqrUjhPdJCMJFRtfCztbU8/OnHFNW2PVrInRqbL2t/ixY1XgCbGjVNP3s7heOr43rbwPgEHr98CjPzHuBP02eSFBXNMxu/5tK/v8Ijn63hYNFpt/brixsByX2b/Ye0EFxkRC28oraWH61eyTf5J1g0tpyU+t8bUmMzq/bX2WzkzmrGRs1mbs0KLQSj31vL93S4uIilu77j3b27qaqr4wdDh/OzCZO9OgTZU9IiMZ8MOzWRp2PAK2prufeD99lxMp8/T7+e64ekGzou3Owx5pIyap8R5WlrHyVVVbyyfStvfrcDu7Zz+8jRLBg/kcSoCzug3WHkZ0hWdDWfBAQTeVLjqW1o4N4PVrL1+DGem3E9Mwenu3x8q42ksVJA8Idz4wxnWxknK8p5Yetm3tu7m9CgIOZlXMz9Yy8hxmZzt8iG1+hlVJP5pA/BRO7mYLXWPPHlWjLzcnlm2nS3ggFYbyRN05wFd3P5Rs558GUwaCtP7u57W7SoMZCcux9E41fr99crOobFV1/DZz+cx7S0Qfw1ayvT3nqdlfv2nDcqyRVGj1STUU3+RVoIJnlr5w4Wrvuchy6ZyM8nXur2fqyWFmmLP5TRSN7Mk7tyLr87mc+TX33JjoJ8xvfuw7PXzCClW5xLxzO6Ri99COaTFoLF5ZeX8+zGr4nZdB0PT5js8usDdSRNoPDm/A9XWhljeiWz4tY7+N3Ua9l3upDr3/4Hy3fvpKnS58xoH6Nr9L4Y1STcJy0EEzzw8WrWZR9l/4KHvdLJaDVWy+V7W3u1aq0VV13lmzz58fIz/PLfn5KZl8tVqWk8NqaE/JyHOq2pS43e/0kLwcLWHjnEp4e/56cTJvq6KKbxl2BgVDnbqz0XFPguT94nJpa3bryZJ664ik3Hctl18FGnWjFSo+/apIXgRWdraxl6yy7yVo+94Dl3x6V3tdq3NxnV2mqvVv3UU6+ydq3vL6SHios49l2P5pTj+WS0T6CRFoJFvbt3N6FT17ElL6/TESPO8tdg0F65/fX9tNSyVq214uTJ/jz11Kt8/vlcS/T3DIpPwBYuo31E56SF4CRXJ+s02O1M/cffSYyK4r1b7gD8I//vLe29d7PPSUdj/JueN4qV/t7SN9B1SAvBy5r+mRpnXOrmWxZ2tCbLl9lHyD1TxrwxFzc/Fij3GfBnHY3x72iOh7+3ZM5rxaA4XRvHp6X3EhF3k6+LJixEAoIT3BlW+PqOb0mOjmH6oMHNj/n7RcVV7Q2XnTLF/4bRujMh0GoVgKSkuUyalM1VU+yEp2ayIncgt65YzsmKcl8XTViEIQFBKTVDKXVAKXVIKfVYG8/PU0oVKqV2OL7uM+K4ZnH1Zu1HS0vIzMvlrtEZF9xU3Vd8dcOatmrj69Y5NxPX21ouxueN4GTlADdryFCWzP4Pjp8p447336W0usrtfclqpoHD46uVUioYeAm4DhgO3KGUGt7Gpsu11hmOr9c8Pa6ZXJ2s81X2UQBmDTl/eQpf/uNYbckLK2gasdVecPLnCYHOlPGylP68Pucm8svL+emaj6m3uz8T2ZV0qrAuI6qvlwCHtNZHtNa1wDvAHAP2axlpaYsJCjr/piRBQZGkpS1uc/ttJ47TNzaWvrHnliXu6v847aVPrJZWacnZNYWsyNkKwLjefXjyqqlsOJbD7zeud/k4vrhLn/AeIwJCH+BYi9/zHI+1dpNSaqdSaoVSql9bO1JKzVdKZSmlsgoLCw0omjFcmayjtSbrxHHGJp9/Ctz9x/Hk4uNJDdfoi54/DDu1cnDypttGjOLu0Rm89u12/rlvr0uvdTWdKqzNrAT3h0Cq1no08G/gzbY20lq/qrUep7Uel5iYaFLRnNPUITdlip1Jk7LbHap37EwZhZVnGdf7/IDg7j+OJ6keT2q4XTHF1NF58Ydg4UkF4PHLpzCxTz9+/cVn7DpV4PQxZTXTwGJEQDgOtKzx93U81kxrXaS1rnH8+hpw4dTdAHHgdOOtDUf2TDrvcfnH8W9Wasm0x5MKQGhwMC/OnEVCRCQ/+9fHVNbVdXicJq6mU4W1GREQtgGDlVIDlFJhwO3A6pYbKKWSW/w6G9hnwHEt6UTFGaBxLZmWXPnH8UZnpjM1XH/uRBWei4+I5A/XzOBoaQlPb/iq3e1ath5l7aPAYshMZaXUTODPQDDwd631YqXUU0CW1nq1UuppGgNBPVAMPKC13t/RPq02U9lZT2/4ije/+5Z9//kwqtXiMe7cmtBXs12tNMtWuM6TNa8Wf72OJd9uZ8nsG7kqNe2C5+WzYW1yC00LWfDJh+w9XcgXd//IkP1JQBBmq6mv5wfLl1JUVcmaO+8hITLS6dt6Ct+TpSsspOBsBcnR0Ybtz1edmVbtRJWLj/fZQkL43+kzOVNdw+8cQ1H9eQiucJ4EBIPV1NcTHhJq2P589Q9n1X/0rjj6yReG9Ujkh6MzWLV/L9mlJb4ujjCJBASD1dnthAbLaRX+7ydjxxMSFMxfsrac97hVW4/Cc3LlMlidvYFQi6xfFChk9JNvJEZFcceo0fxz315yy0qbHzfjvMv6SL4hVy6D1TfYCQkK9nUxTOXtC4Tkr33nJxePJzgoiL9mbTXtmF19mRdfkoBgsNDgYOrtDb4uhqkkrx+4kqKjuWnYCFbt3+fRiqiukPWRfEcCgsFibTbO1NQ0/y61WGNJ/tp8Pxw1hpqGet53cZ0jd8n6SL4jAcFgrQNCoNaefZXXlwBrvmGJPbm4VzJv7/oOM+YtyTIvvtOlAoIZHVUxrQKC1Rh1QZW8ftcyd1SG48ZPxzrf2EOyPpLvdJmAYFZHVZwtnD3vjrTsqJhAbbEI75o5eAhx4eG8t3e3148l6yP5ToivC2CWjjqqWn/Q3FlzqEnf2G5EXruesvfHEGuzdYklICSvH/hsISFcmzaIT74/SE19PbYQ7146kpLmSgDwgS7TQnC2o8rTlkS/bo13STvWYty2r3k732+Flo9on1F/n5mD06moq+Xr3GyXXidzCvxHlwkIznZUeTrkLcVx28zcM2WANWrPku/v2oxKE07q249utnA++f6g06+ROQX+pcsEBGc7qjwd8pbSLQ6AnNLGFoJcdEWgCA0O5pqBA1l79DC1Dc7NtZE5Bf6lywQEZzuqPB3yFmOz0SsqmgNFpz0tsldYocUivM9bacKrUwdSUVvLdwX5Tm0vcwr8S5fpVAbnOqrS0hZz4MD882o1rg55G9GzJ3sLT7ldTm+SFkvX0PIGOUYObJjQpy8K2Jx3jPG9+3a6vc2W4kgXXfi4sJ4u00JwlhFD3kYkJnG4pJiK2lrvFbQVudALM3SPiGB4Yk8yjzk3H0HmFPgXCQhtSEqay6RJ2UyZYmfSpGyXh79d1CsZu9ZON6vb4uoFXuYXiPYYnSac1Lcf3+SfoLq+rtNtZU6Bf5GA4AUXJfcmWCmna1FtkQu8MIrRrccJffpRa29gZ0GBU9t7WsES5pGA4AWxNhsXJ/dmXfYRrx5H7hMgfGFEz54A7DttzX4y4T4JCF5yVWoae08XcrKi3OnXuHqBt8r8AglAXUtSVDTdbOF8X1zs66IIg0lA8JIpqQMAWJd91OnXWOUC7ypJb/kvdz5bSikGdu/OEQkIAUcCgpekJ/Sgd0wMXxz1btqoicwvEO5wN5inxcdzuEQCQqCRgOAlSimmDxzM+pxsztRUu/x6Vy/wvkgTSf9F19UnJpbCyrPUOTljWfgHCQhedMOQodTaG/j08CGXX2v1C6u/preEMcG8Z1Q0AKcrKzvZUvgTCQheNCapFymx3fjwwH5fF0WIZkYE855RUQCcOlthePmE70hA8CKlFLOGDGVTXi6FlWd9XRyvkf6LrifJ0UIo8JOAIEtwO0cCgpfNSR+GXWs+2L/P10XxGkkT+S93g3l8RAQAxVVVBpamY+5e1GUJbudJQPCywQkJjE3uzTt7dppyg3IhXOFuMI8OCwPgbF3ny1cYwZOLuizB7TwJCCa4feRojpSUsO3EcV8XRQhDRIY2BoTKOnMWcPTkoi5LcDtPAoIJZg4aQkyYjWW7d/q6KEIYIiQoCFtwiGktBE8u6p7e46QrkYDgBldzmRGhodw4dBhrDh2UYXoiYESGhlBp0hLvnlzUZQlu50lAcJG7ucy7RmdQ29DA27u+M6eguJcflg5i4SyFwqxeMU8u6rIEt/MkILjI3VzmwPgEpqQO4K2dO6ipr/dmEZu5syyBrEsknKXRKJOO5elFXZbgdo4EBBd5ksv88UVjKaqq5IcPyxowwjVWbLlpGufaeFPL9OyRI4+TlrZYLupeZEhAUErNUEodUEodUko91sbzNqXUcsfzW5RSqUYc1xc8yWVO7pvC0B6JrPhLTxrsdqOLBri3LIGsS2R9Vmy5aY1XWwgyf8B8HgcEpVQw8BJwHTAcuEMpNbzVZj8GSrTWg4A/Ac94elxf8SSXqZTiwXETAFixb49XyufOsgTuLmUgAaNrq22oJyw42Gv7l/kD5jOihXAJcEhrfURrXQu8A8xptc0c4E3HzyuAqcrbbU0vcTeX2VQLn5WeDsDtI0f5fS3cirXWQOLtlpsnyznU2+1U1dcTY7MZU5g2yPwB84Ws65xCAAAW4ElEQVQYsI8+QMubB+cBE9rbRmtdr5QqAxKA0y03UkrNB+YDpKRYd4xwUtJcl/OXixad+0dWCgY890ceHD+BRyZdZnj5mrizLIGsS2QdrT8zRk50b0rHNNXAm9IxgFOf7bOO4abRYd4LCDZbiiNddOHjwjss1amstX5Vaz1Oaz0uMTHR18XxqhuGDOW1b7ZzvPyM147hjWGn0t8QGDxNx1Q0B4Qww8vWROYPmM+IgHAc6Nfi976Ox9rcRikVAnQDigw4tl9auBB+OflyAJ7++isfl+Z8naUR5D4IvmF0y83TdEyZ46ZPMV5sIcj8AfMZkTLaBgxWSg2g8cJ/O3Bnq21WA/cAmcDNwBe6C630VlCwlCNHHqemJhebLYUHHlhMUuxcHhh3CX/esomvc7K5vH+qr4vpcRpBeI/RAdfTdEzTstdN90XwFnfSs8J9HrcQtNb1wALgU2Af8K7Weo9S6iml1GzHZkuABKXUIeAXwAVDU63Mk863jobO/WTseFLjuvPEus9Nm6zWEVfTCNLf4L88TccUVDQGhKToaLeOL/cnsCZD+hC01p9orYdorQdqrRc7HntCa73a8XO11voWrfUgrfUlWmtz7jxvAE/HQnd0kbWFhPDklKvJKSvlle3bvFB617iaRpA0kf/yNB2TX1GOAnpGut5CkPkF1mWpTmUr8rTzrbOL7OUpqcwanM5fsrZwtLTEs8J6SFaF7Fo8Wc6hoKKCHpFRhLoxD0HmF1iXBIROeNr55sxF9jdXTMEWHMKv136G3YddKzKqQzgrp6yUfrGxbr1W5hdYlwSETnhaa3bmItszKprfXDGFrSfyeGvntx3uz5u5VxnVIZx1uKSYgfEJbr1WWqLWJQGhE57Wmp29yN48bARTUgfw9Ib17DlV0Oa+zMi9yqqQXZezfUKnKys5XVnJYDcDgrRErUsCQieMqDU7c5FVSvHstBnEh0ewYM1HnKmpuWAbyb0Kb3J2KZKmCsvInkluHUdaotZlxDyEgGfWWOiEyEieu+567nz/XR79bA0vz5pDUIslnyT36pyWSz4I4+0oyEcBwxN7ur0PmV9gTdJCsJjxvfvy+OVTWHv0MM9vyTzvOcm9OkcW3XOeO0uRbDqWy4ieScR6cWE74RsSECzonjEXcdOwETy/NZPPDn/f/HhXyL1Kzb5zRp4jV5ciqayrY8fJfCb3S5G/VQCSgGBBSin+56ppjE7qxSOfreH7osZln7yde7XCP7i7tfuutOieL1tAWSeOU2e3c2nfFGmJBSAJCAYxejioLSSEv86cTURoKPM/WkVxVWNnsjdHATn7D27Fi6wsuuc5Z5Yi2ZCbTVhQMON69zH8+LKche9JQDCAt4aDJsfE8PL1c8ivKOcnH31gifWOwPgaaleq3bvLjHPU2b601rz6x1gOPPQzIsNCDS2HLGdhDRIQDODN4aAXJ/fmj9dcx/b8Ezz8r4+pa2jweJ8tWeFibHTt3peL7nnrvFmhBbSn8BRBV3/JO7t3GV4OGVJtDRIQDODt4aDXD0ln4ZVX8dmRQ/z800+ot9sN2S84f6GxQuBwli/L5K95dWfO2SffHyRYKa5NG2T48WVItTVIQOiAszlNM4aD3jPmYv7PZVfyyaGD/OKzzoOC0flYs2qosqR257xxjjoLZFpr1hw6yOR+KXSPiDC8HDKk2hokILTDlZymWcNB77t4HL+69HI+OniAn675iNp20kfu5mOtcDG2YoujM2a3nnxxjr45eYKcslJmDk73Sjm6wpBqfyABoR2u5DTNnIr/k7GX8JvLp/Cvw9/zwMerL+hoXrTI/Xyss//g7gQOf7zQO8sK+X13uBLIlu3aSXRoGLNaBAQjyXIW1qCseifLcePG6aysLJ8df926IKCtc6OYMsW4HL673t71Hb/9ci2T+qXw6qwfEBl6btTHl19ar+xKnbtgBjJ/fZ8dlbu0uoqJS17h1uEjeeqqaeYWTLhMKbVdaz3OnddKC6EdVsppttUfcOeoMfzhmuvYnHeMe1at4IzjpucdlVHysd5nhbSb0d7ft5fahgbuGDXG10URXiYBoR1WyWl21B9w47DhPD/jer54PZVu4eHNTf/f/nYx1dW+L7s/jUwyir++t/YCWYPdztJd33FRr2SG9Ug0t1DCdJIy6kBBwVKOHHmcmppcbLYU0tIWm57TzMxMdQSD89ls/Zk0KRuAzXnHeGjNh9TUN7DrPx9Ca2uUvSV/TaV0dR8fPMBD//qIF6+7gZmDh/i6OMIJnqSMZPnrDlhhiV5nxmdP7NuPD27/IQ98vJpdwP9mbuThCXf4vOzCv9m15sVtmxnUPZ4Zgwb7ujjCBJIysjhn+wN6x8Ty7s23M/muQ7y4bTP3fbiKsurqNl/rC4GYWw90nx85zIGi0/zn+Ann3ZdDBC4JCBbnSl+GLSSEDW8O5KkpU9l0LIcfLF/KvsJTZhW1Q/6aW++q6u12ntuyiZTYbswaMtTXxREmkYBgca6Oz1ZK8cPRGbx9061U1ddx47tv83LWVkOXu3CGrFzp317atpm9pwv55aWXExJk3GVCPhfWJp3KAayospLHv/w3nx0+xOikXjwzbTrpCT28ftymkVEtJ8cFBUXKRCM/sT3/OLetWM6c9GH88drrDNuvfC7MIfMQRJsSIiP568zZPD/jevLKypi97C1e2rbZ8BVTW+tsprSkj6yrvKaGX3y6hj4xsSy68mpD9y0rmlqfBIQAp5Ri1pChfPrDeVw7cBB/zNzIf7z7tlf7FjobGWXUiqASWIz35FdfcLz8DP87/TpiDL5nsqxoan0SEExiVu60veMkREbywnU38JeZszlZUcHsd/4f//frdefNcDaKWTOlzV5q2lcByKzPzvI9u1i5fy8Lxk9kbLLxd0STGfTWJwHBBGbdDcqZ48wYNJjPfjiPm4ePZMm325ny5hL+/u32dldOdUdbI6MaGiL57W8X+/WsZV/c68Csz87W43k88eVarkhJZcElEw3ddxOrzP4X7ZOAYAKzcqfOHqd7RARPT72WD++4i5GJSfzP1+u49q3X+ejgfowYZNDWyKiRI19l7dq5Hq8I2tWWwzDjs3OsrIwHPv6Aft268fx11xs6qqglWdHU+mSUkQnMWjnV3eOsz8nm6Q1fcaDoNGOSevHry67kkj59DStX62U0fvvbxaxd6/lFwIzlMBYtartlsHChOUHI25+d8poabnlvGQVnz7LytjsZENfd430K35JRRhZnVu7U3eNc0T+Vj+64i99Pm05BRQW3v7+cH61eydbjeR63GNpKefz61/5z83Rf3+vAm5+dsupq7v/4Aw6XFPPSzBskGAgJCGYwK3fqyXGCg4K4efhIPr/7Rzw66TJ2njzJ7e8v58Z33+bjgwfcntjWVsojONiYlEdXWA7DW5+dw8VF3Pju22SdOM6z11zH5H7SsSskZWQas1YfNeo41fV1rNy3l9e+3U52aQl9YmL50UVjuWX4SKLDwpzej9VvNOSKRYt801dh9GdnXfZRfvqvj7AFh/DX62czrrfxI4qE73iSMvIoICil4oHlQCqQDdyqtS5pY7sGYJfj11yt9ezO9h1oAcFf2bXm8yOH+du3WWSdOE6szcZtI0Zx+8jRTqUYnFm+W5hDa81r32bxuw3rGdYjkVdu+AF9YmJ9XSxhMF8GhN8DxVrr3ymlHgO6a61/1cZ2FVrraFf2LQHBenaczOe1b7L47Mgh6u12JvdL4c6Ro5k6YCC2kLZXUpflCqyhur6O33yxlpX793LdoCE8e82M5tuuisDiy4BwAJiitc5XSiUD67TWF9yFWwJCYDl1toL39u5m2e6dnCgvp5stnFlD0rlp2AjGJPVCtVoq2Wo36wHfpX+8qb339F3BSX619lMOFp3mZxMm89AlEy/4G3mTFf/+gcyXAaFUax3n+FkBJU2/t9quHtgB1AO/01qvamd/84H5ACkpKWNzci5MNQjraLDb2Xgsl/f37eGzw4eoaagnJbYbMwYPYeagIYzqmWTqhccV/ngHt86CWOv3lHemjGc3beDDg/tJjIzi99Omc2XqAMOO5wxpIZrPqwFBKbUW6NXGU48Db7YMAEqpEq31BYllpVQfrfVxpVQa8AUwVWt9uKPjSgvBv5ypqWHNoYOs+f4gm/Jyqbfb6Rsby3WDhnDdoCFtthx8yR8DQmdlbnq+rLqav2Rt4c0d3xIUpPjxRWOZf/F4l9cmMuIcSR+S+SyfMmr1mjeAj7TWKzraTgKC/yqtruLfRw6z5tBBNubmUGe3kxwdw9QBaVw1II2JffoR4YP8ta8nmXmqrQt0e+8pbsYm7vvFGX4x8VKSY2IMO54zWqaI2h5hBv44ysxf+DIgPAsUtehUjtda/7LVNt2BSq11jVKqB5AJzNFa7+1o3xIQAsOZmmo+P3KEfx0+yIbcHKrq67EFh3BJnz5M7pfCpf36Mzyxp+m3aLRKC6GztIwzQSy/vJwPDuzjgfGXMOC5P3JpvxT+z2VXMiyxp1vl8SRotpUiaou0ELzHlwEhAXgXSAFyaBx2WqyUGgfcr7W+Tyk1GXgFsNM4Ee7PWuslne1bAkLgqamvZ+vxPL7MOcrG3By+Ly4CIC48nIt79WZs796MjtqKKv0DtTXHvNoBOW3aUv77v83p6OyoU9WVwNRy24raWj49/D0r9+1lc14uGjj68COsO3qUK/qnGpKecydotpciakn6ELzLZwHBmyQgBL6Cigoy83LJzDvG9vzjJOq1zOu7AltQXfM2WoXTM+V5hqfeZ1gfREHBUnbvnk9wsPc7OjvrVHU1IKw7epRVB/bx6eHvqa5v7MS/cdhwfpA+nNf/HGdo6sudgND+RERoXNBORhl5mwQEERA2bkqhrvbYBY+fro3jf47+NyMSkxjZsycjEnsysmcSqXHd3Uo1OdvRacQom/aOdfJkf+64I/u8x1qnZRrsdvaeLmRjbg4bj+Ww5m/9iJ2xkVibjesHp/Mfw4bTR63j6FHvtHTcef/Siex7EhBEQGivdqlRrLV/yp7CU+w/Xdh874aIkBAGdo9nYHwCg+LjGRDXndS47vTvFkdUB8trOLuchhH9DJ0dq+kYWmtOV1ZyqLiI74uL2HI8j8y8XEqrG29glJ7Qg0v79Xf0u6RgCwmx5JBOK5apq/EkILQ9vVQIH7DZUtqsXYbbUlg86RoA6hoaOFRcxK5TBRwoOs3h4mK2ncjjgwP7zntNQkQEyTGxJEdHkxwdQ3JMDMnRMfSKjiE4tA8NdXltHt+s96SDk/nbN9uA8dz87tscKinmTE1N8/PJ0dFMHTCQS/v159J+KSRGRV2wj47uleCri2/TcWUimn+SgCCc5u0Zp2lpi9usXbZc2TM0OJhhiT0vGEFztraWnLJSsktLyCkr5VhZGfkVFeSUlbE5L4/y2nMX2wlxl1/QV1Gnw8gsu4lH7vqezP83uPnxpozUzQ8UcNuCwhaPn0tVNdjtVNXXcba2rvF7XR2VtbVU1tcR3zCHSyNeJjSotnn7Gnsob+ReyZbS9STPanxPs4YMZVD3eAbGxzOoewK9oqM77TOx6j2Kk5LmSgDwU5IyEk4xKxXgraBTUVvLyYpy8svLKaqqpKZsJXFVLxKmT1Gpe5B59jayysZytq6Oqro6Kuvr2Pfgwwx47o8uHSdYKSJCQ4kKDSMyNJRYm43x3bK4yPY24aqQhqBeBHX/L3om3Unf2FjiwiPcfk+SrxdtkT4E4XVd8eKjFJytraOyro6a+nq0oy+g9b9MkFJEhoYSGRpKWHCwaTOyJV8v2iJ9CMLrrJqe8KaFC2m+0FuR5OuF0SQgCKe01znqjY5Yq/CH5SwkXy+MJLfQFE4x6zagQgjfkYAgnJKUNJf09Fex2frTOOO0v+SqhQgwkjISTpP0hBCBTVoIQgghAAkIQnQJBQVLycxMZd26IDIzUykoWOrrIgkLkpSREAGu9XyFmpocDhyYDyApQHEeaSGIgCQ14nM6WvNIiJYkIIiA01Qjbpw3oZtrxN4KClYPPl1xUqFwjwQEEXDMrBGbHXzc0d7kwUCeVCjcIwFBBBwza8T+kI6RSYXCWRIQRMAxs0bsD+kYmVQonCWjjETAcea+CkbxlzWeZFKhcIa0EETAMbNGLOkYEUikhSACklk1YlmCWgQSCQhCeEjSMSJQSMpICCEEIAFBCCGEgwQEIYQQgAQEIYQQDhIQhBBCABIQhBBCOEhAEEIIAUhAEEII4SABQQghBCABQQghhIMEBCGEEIAEBCGEEA4eBQSl1C1KqT1KKbtSalwH281QSh1QSh1SSj3myTGFEEJ4h6cthN3AfwDr29tAKRUMvARcBwwH7lBKDffwuEIIIQzm0fLXWut9AEqpjja7BDiktT7i2PYdYA6w15NjCyGEMJYZ90PoAxxr8XseMKGtDZVS84H5jl9rlFK7vVw2f9EDOO3rQliEnItz5FycI+finHR3X9hpQFBKrQV6tfHU41rrD9w9cFu01q8CrzqOm6W1brdfoiuRc3GOnItz5FycI+fiHKVUlruv7TQgaK2nubtzh+NAvxa/93U8JoQQwkLMGHa6DRislBqglAoDbgdWm3BcIYQQLvB02OmNSqk8YBLwsVLqU8fjvZVSnwBoreuBBcCnwD7gXa31Hid2/6onZQswci7OkXNxjpyLc+RcnOP2uVBaayMLIoQQwk/JTGUhhBCABAQhhBAOlgkIsgzGOUqpeKXUv5VS3zu+d29nuwal1A7HV0B11Hf2d1ZK2ZRSyx3Pb1FKpZpfSnM4cS7mKaUKW3wW7vNFOb1NKfV3pdSp9uYnqUbPO87TTqXUxWaX0SxOnIspSqmyFp+JJ5zasdbaEl/AMBonVKwDxrWzTTBwGEgDwoDvgOG+LrsXzsXvgcccPz8GPNPOdhW+LquX3n+nf2fgP4GXHT/fDiz3dbl9eC7mAS/6uqwmnIsrgIuB3e08PxNYAyhgIrDF12X24bmYAnzk6n4t00LQWu/TWh/oZLPmZTC01rVA0zIYgWYO8Kbj5zeBH/iwLL7gzN+55TlaAUxVnayh4qe6yme+U1rr9UBxB5vMAf6hG20G4pRSyeaUzlxOnAu3WCYgOKmtZTD6+Kgs3pSktc53/HwSSGpnu3ClVJZSarNSKpCChjN/5+ZtdOPQ5jIgwZTSmcvZz/xNjjTJCqVUvzae7wq6yvXBWZOUUt8ppdYopUY48wIz1jJqZuYyGFbX0blo+YvWWiul2hsb3F9rfVwplQZ8oZTapbU+bHRZheV9CCzTWtcopX5CY8vpah+XSfjWNzReHyqUUjOBVcDgzl5kakDQsgxGs47OhVKqQCmVrLXOdzR5T7Wzj+OO70eUUuuAi2jMN/s7Z/7OTdvkKaVCgG5AkTnFM1Wn50Jr3fJ9v0ZjH1RXFDDXB09prc+0+PkTpdRflFI9tNYdLgDobymjrrIMxmrgHsfP9wAXtJ6UUt2VUjbHzz2ASwmcJcWd+Tu3PEc3A19oR29agOn0XLTKk8+mcUWArmg1cLdjtNFEoKxF6rVLUUr1aupTU0pdQuO1vvMKk697y1v0it9IY86vBigAPnU83hv4pMV2M4GDNNaEH/d1ub10LhKAz4HvgbVAvOPxccBrjp8nA7toHHWyC/ixr8tt8Dm44O8MPAXMdvwcDrwHHAK2Amm+LrMPz8XTwB7HZ+FLYKivy+yl87AMyAfqHNeKHwP3A/c7nlc03ozrsON/os3RioHw5cS5WNDiM7EZmOzMfmXpCiGEEID/pYyEEEJ4iQQEIYQQgAQEIYQQDhIQhBBCABIQhBBCOEhAEEIIAUhAEEII4fD/ATlMOPI7GFFmAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Find indices of positive and negative examples\n",
"pos = np.where(y==1)[0]\n",
"neg = np.where(y==0)[0]\n",
"\n",
"# Plot examples\n",
"plt.plot(X[pos, 1], X[pos, 2], 'b+', label='Admitted')\n",
"plt.plot(X[neg, 1], X[neg, 2], 'yo', label='Not admitted')\n",
"plt.legend()\n",
"\n",
"# Here is the grid range\n",
"u = np.linspace(-1, 1.5, 50).reshape(50)\n",
"v = np.linspace(-1, 1.5, 50).reshape(50)\n",
"z = np.zeros((len(u), len(v)))\n",
"\n",
"# Evaluate z = theta*x over the grid\n",
"for i in range(len(u)):\n",
" for j in range(len(v)):\n",
" z[i,j] = map_feature(u[i], v[j], degree=6).dot(theta)\n",
"\n",
"# Plot z = 0\n",
"# Notice you need to specify the range [0, 0]\n",
"#z = z.reshape(len(u), len(v))\n",
"plt.contour(u, v, z.T, 0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exploring regularization\n",
"As mentioned before, $\\lambda$ is used to control the problem of overfitting. In the following graphs, models trained with different values of lambda are shown to give you some intuition on the problem of over/underfitting."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"def create_plot_for_lambda(X, y, _lambda):\n",
" from scipy.optimize import minimize\n",
" result = minimize(compute_regularized_cost, initial_theta, args = (X, y, _lambda),\n",
" method = 'CG', jac = compute_regularized_gradient, \n",
" options = {\"maxiter\": 400, \"disp\" : 1})\n",
" theta = result.x\n",
" print('Conjugate gradient found the following values for theta - first five values only: {}'.format(theta[:5]))\n",
"\n",
" # Find indices of positive and negative examples\n",
" pos = np.where(y==1)[0]\n",
" neg = np.where(y==0)[0]\n",
"\n",
" # Plot examples\n",
" plt.plot(X[pos, 1], X[pos, 2], 'b+', label='Admitted')\n",
" plt.plot(X[neg, 1], X[neg, 2], 'yo', label='Not admitted')\n",
" plt.legend()\n",
"\n",
" # Here is the grid range\n",
" u = np.linspace(-1, 1.5, 50).reshape(50)\n",
" v = np.linspace(-1, 1.5, 50).reshape(50)\n",
" z = np.zeros((len(u), len(v)))\n",
"\n",
" # Evaluate z = theta*x over the grid\n",
" for i in range(len(u)):\n",
" for j in range(len(v)):\n",
" z[i,j] = map_feature(u[i], v[j], degree=6).dot(theta)\n",
"\n",
" # Plot z = 0\n",
" # Notice you need to specify the range [0, 0]\n",
" #z = z.reshape(len(u), len(v))\n",
" plt.contour(u, v, z.T, 0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Overfitting\n",
"Remember: overfitting means the model probably only works on the training set and not well in the real world. If you model is overfit, consider adding regularization or choosing a higher value for $\\lambda$.\n",
"\n",
"You can check out other values of $\\lambda$ yourself."
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Warning: Maximum number of iterations has been exceeded.\n",
" Current function value: 0.281767\n",
" Iterations: 400\n",
" Function evaluations: 1435\n",
" Gradient evaluations: 1435\n",
"Conjugate gradient found the following values for theta - first five values only: [ 2.53231573 -1.3591363 2.44932572 -6.28390865 -8.54930906]\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xd8VFX6+PHPSaeEFkihhtB7KIpBkCpVQEURRFRcF93f6uquurrrroCu311Xd+2uDRRXOouIICCINAE1IL0TAgRCSKEkJJkkk/P7I8UQUqbcmTszed6vFy8yM3fuPXOTuc897TlKa40QQgjhZ3YBhBBCeAYJCEIIIQAJCEIIIYpJQBBCCAFIQBBCCFFMAoIQQgjAoICglJqjlLqglNpfyeuDlFKXlVK7i/+9YMRxhRBCGCfAoP18CrwDfFbFNlu01rcZdDwhhBAGM6SGoLXeDGQYsS8hhBDmMKqGYIs4pdQe4BzwtNb6QPkNlFLTgekAderU6d2xY0c3Fu9aFquVkxcziKwbSoOQENPK4avOXLlMncAgGtWqZXZRhPApO3fuTNNaN3Hkve4KCLuAVlrrLKXUaGA50K78RlrrD4EPAfr06aPj4+PdVLzrXbFYeGTlcn44m8SkPn15om8cgf7+ppXHl5zISOfWzz/l2ZsH8EjvG80ujhA+RSl1ytH3umWUkdb6itY6q/jnr4FApVRjdxzbUfWCg/l0/ATu6tyF9+J/4K4lC0i4KK1iRlh0YB8Bfn7c2amL2UURQpThloCglIpUSqnin28sPm66O47tjOCAAP45bCTvjh7LmSuXGbvgv8zftwdJCOi4PKuVZYcOMrR1G5rUrmN2cYQQZRjSZKSUWgAMAhorpZKAGUAggNb6feAu4DdKqQIgB5ikveiqOqpte3pGRvHMujX85bv1bDtzmteGjyQkINDsonmd9QnHycjNYVKXbmYXRQhRjiEBQWs9uZrX36FoWKrXiqwbytzb7+LDnT/x6rYtJGdl8uFttxNWu7bZRfMqC/fvI6puKP1btjK7KMJO+fn5JCUlkZuba3ZRBBASEkLz5s0JDDTuxtSdo4y8np9SPNrnRlo1aMAf1q7mzsXzeWPEaHpGNTW7aF7hzOXLbD1ziif6xuHvJ5PkvU1SUhKhoaFER0dT3AIsTKK1Jj09naSkJFq3bm3YfuVb6YBRbduzYMJECgoLuWvJAv62eSM5+flmF8vjLT64Dz+luLtzV7OLIhyQm5tLWFiYBAMPoJQiLCzM8NqaBAQHxUZGsWbKA9zbrQdzdu9k1Ly5xJ87a3axPFZBYSFLDx5gYKvWNA2tZ3ZxhIMkGHgOV/wuJCA4ITQ4mJcGD2PBnRNRSnHvssUs2L/X7GJ5pC2nE0m5msU9XaR2IISnkoBggL7NW/DlpCnc3KIlz29YxwvfrSffajW7WB7li8MHaRgSwqDoGLOLIrzc8uXLUUpx+PDhCl9/8MEHWbp0qc37O3fuHHfddRcAu3fv5uuvvy59bePGjWzbts3uMkZHR5OWlmb3+8wmAcEg9YJD+HjsHUzv1YfP9+1h6hdLOXnpotnF8ghXLLmsO3GCMe06ECSzvWucmTON3d+CBQvo378/CxYsMGR/TZs2LQ0gRgUEbyUBwUD+fn48138g/x4+ioNpFxj1+Vxe3/E9uQU1u8P54107sVgLmNS1u9lFESaYNcu4fWVlZbF161Zmz57NwoULgaIRN4899hgdOnRg2LBhXLhwoXT76Oho/vSnPxEbG0ufPn3YtWsXI0aMoE2bNrz//vsAJCYm0rVrV/Ly8njhhRdYtGgRsbGxvPLKK7z//vu8/vrrxMbGsmXLFlJTU5kwYQI33HADN9xwA99//z0A6enpDB8+nC5duvDwww977eRVGXbqArd37Ey/Fi35+9bNvP3jDr4+dpT3Ro+jXViY2UVzu4ycbObs3slt7TrQuUm42cURXu7LL79k5MiRtG/fnrCwMHbu3MmpU6c4cuQIBw8eJCUlhc6dO/PQQw+Vvqdly5bs3r2b3//+9zz44IN8//335Obm0rVrVx599NHS7YKCgnjxxReJj4/nnXeKpk3l5ORQt25dnn76aQDuvfdefv/739O/f39Onz7NiBEjOHToELNmzaJ///688MILrFq1itmzZ7v3xBhEagguEl6nLq+PGM1nt9/Fpdxc7lg8j5VHK27z9GVz9/xMdn4+j98YZ3ZRhBvNnAlKFf2DX352tvlowYIFTJo0CYBJkyaxYMECNm/ezOTJk/H396dp06YMGTLkmveMGzcOgG7dutG3b19CQ0Np0qQJwcHBXLp0ya7jr1+/nscee4zY2FjGjRvHlStXyMrKYvPmzdx3330AjBkzhoYNGzr3QU0iNQQX69+yFV9Nvo/HV6/kd2tWset8Ms/dfEuNaEu/mpfHZ3t2MzymbY2sHdVkM2f+cvFXCoxoQcnIyGDDhg3s27cPpRRWqxWlFHfccUeV7wsODgbAz8+v9OeSxwUFBXaVobCwkB07dhDioynxpYbgBpF1Q5l/50Smxfbi0927uHvJAnYlnzO7WC638MA+LltyeaT3DWYXRfiApUuXMnXqVE6dOkViYiJnzpyhdevWhIWFsWjRIqxWK8nJyXz33XcOHyM0NJTMzMxKHw8fPpy333679PHu3bsBuOWWW5g/fz4Aq1ev5uJF7xxQIgHBTQL9/fnrLYN5Z9RYUq5mcdeSBfxuzUou5uSYXTSXyLNamb0rnpuatZDUHjXcjBnG7GfBggXX1QYmTJhAcnIy7dq1o3Pnztx///3ExTnePDl48GAOHjxIbGwsixYtYuzYsXzxxRelncpvvfUW8fHxdO/enc6dO5d2TM+YMYPNmzfTpUsXli1bRsuWLZ36rGZRntobbvYCOa50NS+PD3f9xAfxPxFWuxZvjBzDDU2bm10sQy09uJ8/rl/LJ+PuZGC0cblWhHkOHTpEp06dzC6GKKOi34lSaqfWuo8j+5MaggnqBAXx+5tuZsnEyQT5B3Dv/xbz7k8/UOihwdlehVrz0a54OjVuwi2tos0ujhDCRhIQTNQtPIIVk+5jVLv2/Gv7Vh5YvpTUq1fNLpbTvjuZwLGMdKb3vkFy3wjhRSQgmCw0OJg3R4zh70NuJf7cOYZ8NpvXd3zPFYvF7KI57INdP9EstB6j27Y3uyhCCDtIQPAASinu6dqdVfdOZWCr1rz94w5GzZvLj2eTzC6a3XafTyb+3Fke6tmbwBowtFYIXyIBwYPENGzEO6PHsmzivQT5+3PvssX8e/v3XpUo79M9u6gbFCRrHgjhhSQgeKDYyChWTp7KnZ06885PO5j0v0WcvmzfjEozpGRl8fWxo9zduSt1g4LMLo4Qwk4SEDxUnaAg/jlsJG+OHMPxjAxum/9fZv+806NXZpu/fw/WwkKmdo81uyjCBymleOqpp0ofv/baa8ysJhfG8uXLOXjwoFPHtTeV9YoVK/jHP/5R4fE//fRTzp2zb1JqSfI9d5CA4OHGtu/I1/feT/fISF7espEBn3zEkoP7PS6b4uXcXObt3cPg6BiiG3hnHhdhrJSUeWzfHs3GjX5s3x5NSso8p/YXHBzMsmXL7Lo4GxEQ7DVu3Diee+65Co/vSEBwJwkIXqBZvXp8fsfdLLrrHto0asSz69fy26+/4lKu58xyfvenHVzMzeHJm/qZXRThAVJS5nHkyHQsllOAxmI5xZEj050KCgEBAUyfPp3XX3/9utcSExMZMmQI3bt3Z+jQoZw+fZpt27axYsUKnnnmGWJjYzlx4sQ17/nqq6/o27cvPXv2ZNiwYaSkpACVp7JOTEykY8eOPPjgg7Rv354pU6awfv16br75Ztq1a8ePP/4IFF30H3vsseuO/8orrxAfH8+UKVOIjY0lJyeHnTt3MnDgQHr37s2IESNITk4GYOfOnfTo0YMePXrw7rvvOnzO7Ka19sh/vXv31uJ61sJC/UH8j7r92//WcR+/r78/fcrsIulTly7qDm+/rp9Zt9rsoggXOnjwoM3bbtvWSn/3Hdf927atlcPHr1Onjr58+bJu1aqVvnTpkn711Vf1jBkztNZa33bbbfrTTz/VWms9e/ZsPX78eK211g888IBesmRJhfvLyMjQhYWFWmutP/roI/2HP/xBa631448/rmfNmqW11nrlypUa0KmpqfrkyZPa399f7927V1utVt2rVy89bdo0XVhYqJcvX156zE8++UT/9re/rfD4AwcO1D/99JPWWuu8vDwdFxenL1y4oLXWeuHChXratGlaa627deumN23apLXW+umnn9ZdunSp8DNU9DsB4rWD113Jdupl/JRieu8b6NeiJU+uXcV9Xyzh4Z69efKmm6kdGGhKmV7bthU/P8UfbrrZlOMLz2OxnLbreVvVq1eP+++/n7feeotatWqVPr99+3aWLVsGwNSpU/njH/9Y7b6SkpK45557SE5OJi8vj9ati1KsbN68uXRf5VNZt27dmm7dugHQpUsXhg4dilKKbt26kZiYaNdnOXLkCPv37+fWW28FwGq1EhUVxaVLl7h06RK33HJL6edZvXq1Xft2lDQZeamu4RF8NWkqU7r14OOfd3LznA95ddsWMnKy3VqO3eeTWXnsCL/u1YfIuqFuPbbwXMHBFSd3q+x5ezz55JPMnj2bq07O6n/88cd57LHH2LdvHx988AG5ubnVvqd8+uyyqbXtTaWttaZLly7s3r2b3bt3s2/fPr755hv7PoTBJCB4sVqBgbw0eBhL755MXPMWfLDzJ0bP/4xtZ5y7C7PH6zu+J6xWbX7dy/4U10Z3OgrPERPzMn5+ta95zs+vNjExLzu970aNGjFx4sRrViXr169f6ZKa8+bNY8CAAcD16avLunz5Ms2aNQNg7ty5pc8bmcq6qnTaHTp0IDU1le3btwOQn5/PgQMHaNCgAQ0aNGDr1q2ln8ddJCD4gF5RTXlvzDi+mnQfoUHBTP1iCa9t20pBYaFLj3sw9QJbTp/ioZ697J534IpOR+E5IiKm0KHDhwQHtwIUwcGt6NDhQyIiphiy/6eeeuqa0UZvv/02n3zyCd27d+e///0vb775JlC0qtqrr75Kz549r+tUnjlzJnfffTe9e/emcePGpc8bmcq6/PEffPBBHn30UWJjY7FarSxdupRnn32WHj16EBsby7Zt2wD45JNP+O1vf0tsbKxbRxRK+msfk52fz6xNG1hycD+9o5ryxogxNKtXzyXHenLtKr5NOMH3D02nXrB9K0ht3x5dHAyuFRzciri4RINK6B4pKfNISHgei+U0wcEtiYl52bALnyeR9NeeR9JfiyrVDgzklWEjeGPEaI6kpTFq/lz+vnUTp+xcO7Y6SVcus+roESZ17W53MADXdTq6m9R0hC+RgOCjxnXoxFeTp9K/RSvm/LyTwZ/N5tn1a7HY2fFVmc/2/IxSimmxvRx6vys7Hd0pIeF5Cguv7cgvLMwmIeF5k0okhOMkIPiwVg0a8N6YcWyZ9mse7tmbJQf3M3X5UkNGIu1IOsMNTZvTNNSx5ihXdjq6k6/UdGzlqU3MNZErfhcSEGqAyLqh/HnAIN4aOYa9KeeZsHgBCRczHN6fpaCAI+lpdIuIcHgfru50dBdfqenYIiQkhPT0dAkKHkBrTXp6OiEh9jfXVkUmptUgt7XvSNPQejyycjkTFi/g3dFj6dfC/gvX0Yx08gsL6dbE8YAARUHB2wJAeTExL3PkyPRrmo28saZji+bNm5OUlERqaqrZRREUBejmzY1di10CQg3TK6opyyZO4VcrlnHfF0voGRnFQ7G9GdO+g8372H+hKOdL13DnAoIvKAloNWGUUWBgYOlsXuGbJCDUQC3q1+d/E+9l4YG9LDmwn8fXrKRpaCg9o5ra9P7s4hTcl3JzaEUDVxbVK/hCTUcIMKgPQSk1Ryl1QSm1v5LXlVLqLaXUcaXUXqWUY0NTaihXzOgNDQ7m171u4It7phBZpy4vbPwWq40T2SZ27kqT2nWYtek7Ct3UniyzmoVwPaM6lT8FRlbx+iigXfG/6cB/DDquz3P1OPc6QUE8P2AQB1IvMH//XpveExoczLM3D2B3SjJfHLYv17wjF3YZ6y+EexgSELTWm4Gqhq2MBz4rzs66A2iglIoy4ti+zh3j3Ee3a0+/Fi351/atpGXbNiT19o6d6RkZxSvfbybTYrHpPY5e2GWsvxDu4a5hp82AM2UeJxU/dw2l1HSlVLxSKl5GMhRxxzh3pRSzBg7hisXCZ3t+vu71ilYp9FOKFwYOIT07m6nLl7Ix8WS1wxEdvbDXtLH+QpjFo+YhaK0/1Fr30Vr3adKkidnF8QjuGueeXTyDObJu3etemzWr4vf0iIjktVtHkZZ9lYdWLGP8ws/Zevr6/EQlHL2w16Sx/kKYyV0B4SzQoszj5sXPiWq4a0bvF4cOEOTnz5h2tg8/BbijU2c23P8rXhk2gsy8PO5fvpRn16/lcgW55R29sPvKrGYhPJ27AsIK4P7i0UY3AZe11sluOrZXc8eM3nyrla+OHmZoTBvqF898nDkTlCr6B7/8XFHzUZC/P3d37sqaKQ/wmz43suzQAYZ//ilrjh+7ZjtHL+zumtUsI5lETWdI+mul1AJgENAYSAFmAIEAWuv3lVIKeIeikUjZwDStdZW5rSX9tfNsTcu87sRxHln1JR/ddjtDY9pc97pSYM+fyYELKTy7fi0H01IZ3qYtj91wU+kkNlelinZ2vyUd3uVnHHtjOg1RszmT/lrWQ/BQ7rrAnc28wp2L5hMSEMD6qdMI9Pe/bl/2BgQoqnV8/HM87/y4g5yCAnpGRvGrnn0Y0aYt/n7GVkyNuJj70voMomaT9RB8jBHj7m0Z0ZNpsfDwii/IKcjno7G3VxgMAGbMsP8zBPr785s+fdn+q0f46y2DycjJ4bHVXzFi3qcsPbiffKvV/p1WwohhqTV1JJM0k4myJCB4IHdc4AoKC/ndmpUcz0jn3dFjaR/WuMLtoeJ+A1vVCw5hWmwv1k+dxtsjbyPEP4A/rl/LoLmzmbtnFznFaTCcYcTFvCaOZJIJf6I8CQgeyB0XuPfjf2TTqUReGjyMAS2j7S6jvfz9/BjTvgNfTZ7KnHF30rxePWZt+o5bPv2Id3/6gSs2Tm6riBEX85o4kkkm/InyJCB4IFdf4I6kp/H+zh8ZHtOWSV27O1VWeymlGBTdmkV3TWLhhHvoGh7Jv7Zvpf+cD3lp83ecvmz/Up9GXMx9ZX0Ge9TUZjJROcl26oGMyLFfWVpm/9A7+PXiedQNCmLGwCGGl90eNzZrzo3NmrP/Qgqzf97Jf/fu5tPdu7g1pi3TYntxY7PmqJJxr1UwKgV1TctaGhzcspKOdN9tJhNVk1FGHsoVwzMtBQXc98US9l+4wMK77qFHRKRBpTVGSlYWn+/bzfx9e7iYm0vnxk14MLYXt7XvQEhAoNnF8zky1NY3ybBTUS2tNc+sW8Oywwd5e+Rtdi2I4265Bfl8efgQn+zexdGMdOoFB3Nnpy7c27U7bRuFmV08Q7lqXoa3HF8Yz5mAIE1GNYDWmrd/3MGywwd5om+cRwcDgJCAQA4t7c7qGd344WwS8/fvYV5xc9KNTZszqWt3RrZt6/W1hvJ36CWjfAC3XZRrWjOZqJrUEHxcodb8bctGPt29izs6dua1W0fa1C5vhJkzHR+yWn4yXFp2NksP7mfh/r2cvnKZesHBjO/QiTs7daF7eITbPpORZDKccAVpMhIVshQU8PS61aw6dpSHYnvz5wED8XPjhdORGc7VvbdQa35IOsPCA/v45sRxLNYC2jUK446Onbm9Yyci64Y6V2g32rjRD6joBCkGDbJt9TohypOZyuI6BYWF/H7t16w6dpQ/9x/IX24ZZHMwcGYimjNsSajnpxRxLVry5sgx/PDwI/zfkFupFxzMP7dtof8nH/HA8qV8eeRQ6YQ3Z2fiunImb02cDCc8m9QQfFCh1jy7fi3/O3SAv94ymGmx9i1h7cyd/cyZFa+fMGOGfYHG3jIkXrrIF4cPsuzQQc5mXqF2YCD3tznNDcHvovQvqbjtGUXj6lE4MspHuII0GYlS2fn5PP3NatacOMYTfeN4om8/u/fhTEAwaj+OvrdQa348m8RXRw/T23oPjQIvXreNrW307mjjd3SUj4wOEpWRJiMBwPmsTCYtXcjaE8f4c/+B/O7GOJvfa8/6B+7gSEI9KGpSuql5C14eciuNAiue9ZxrOc2W04nkVZNgzx0zeSMiphAXl8igQYXExSXaVXORHETCaDLs1EfsTTnP9JXLuZqXx4djb2do6+vXNahK2RFBRtUQHL2ol5THWZXNxM3Ia8Azy/9H3aAgBrVqzdCYNgyObk294BCb3m92G39VOYikdiGcIQHBB6w6eoSn162hce3aLLl7Mh0be8Z61GbVLkpUlgLkxi5v8FHbm1mXcJxvTyaw8tgRAvz8uKFpcwa2iubmFi3p1CTckBQiruBMzcUT5j4IzyUBwQM4esdWqDX/if+Bf23/nt5RTfnPmPFYM79g+3bn7v6cubP3JFXlOGoBDI1pg7WwkD0p51mXcJwNJxP4x/ebAWgYEsKAVtEMD59J/Zy3yc9L8pi7aWdqLs7WLoRvk05lkzk60uSKxcIz61azLuEE4zp05JWhI7iUvsgjRq04MyHN7OOmZGWxPek0W0+fYtOpk6Tn5OCvFL2imtKvRUvimrekR0QkwQHm3Us5MzpJ5j74Phll5GZGtsE6MpLlSHoav1m1gqQrl/lT/4E82KMnSilTZr5WdBG2pw/CyOBhVN9HiUKt2Ztynm9PnmBT4kkOpF5AAyEBAfSOaspNzVtwU/MWdA+PrHC1OVcGRkf/BmV2tO+TgOBGRo8dt/eO7aujh3lu/VrqBgXz9qjbuLFZc4f3ZYSKLsL2XJiNvIgbHRDKu5ybyw9nz7Aj6Qzbk85wJD0NKAoQnRs3oVtEJN3CI+gaHkGbho0I8PdzaXkcIXMffJ8MO3Ujo1eZsnW2ap7Vykubv+OJNavo3CScrybfd00wsGdfrmDWsFV3Hrd+SAjD27TjhYFDWD3lAX56+De8M2osk7p2x8/Pj8UH9vH0ujWMnDeX7u+/DcCsTRtYfGAfh9NSsRbaH5SN/hxmLAQk6zZ7D6kh2Mnou3Bb7tgSL13kd6tXsj/1Ag/06Mmf+g8kqIImCnfd/VU3G7m6O3WjZjOX5+oaQnWshYU8+Vwu77xa+7rXGozcRvNx8XSPiKRHRBSxkZF0C48ksm7dKhPzmf2ZnCU1EveTJiM3ckUbbFXtwSuPHubP367D38+Pfwwbzog27RzelyvUpCYjeykF1kJN4qWL7Dl/nt0pyew+n8yhtFQKimsLDUJC6NQ4nE6Nm9CpSRM6NW5C20ZhpQHfjM9kdh+ZcI4EBDdy1x2PpaCAl7ZsZP6+PfSKjOLNUbfRLLSeaSN4KuNJAcEbzg0U/W4PpF5g/4UUDqelcjAtlSNpaVisBQBcXtOPjNXXzzL/6wuaF2e5Nlut2X1kwnkSENzM1XfhCRczeHz1Sg6lpTK99w08ddPNpaNYPO0uuOQi7GgzkKddxB1V0d/Ef/4zxebPVlBYSOKlixxKS+VQaipH09M4lpHO5mkP0/rNfwEQ5OdP64YNaR/WmPZhjekQFkaHsCY0q1fPsLTmRt/RSw3B/SQg+AitNUsPHeDFTRsI8vfnteGjGBwdc802nhYQKuINZTSSK2uNSsHelBSOFQeIo+npHMtII+nKldJt6gQG0j0iktjIKG5q3oI+Uc2oFejYanJm9JEJY8kSml6iqrvhK5Zcnl3/DWtPHOOmZi341/BRRIWGlr6v7N13yc2gs52wwhiunP07YwZ0C4+gW3jENc9nWiwcz0jnSHoah9JS2X0+mY92xfOf+B8J8vOnZ1QU/Vq0pF+LlqXzJGyp2Rqdv6mq2eLC80gNwY0qu3M+cCGF3379FeeyMnmmX39+1bNPpU0A3nD37SvNQLaq7K5aa8Xgwe5rJ7+al0f8ubNsSzrNtjOnOVg8ka5OYCD3tDpJv9rv40fVa0PIHb33kxqCl9Jas/DAPmZt2kBYrVosmDCR3lHNzC6W07wlGBgVuCq7q05JcW9W1DpBQQyMbs3A6NYAXMzJYXvSGbYlnaarddY1wQAqrsXIHX3NJjUEF6uss/XPf7GSO+Abvjh8kAEtW/H6iNE0qnX9+PWK9uctF1xPZ1Rtq7K76hdf/JD16z3jQlp5LQZ219rKQ7G9qR8Scv0bhdeRmcoebObMoi9dyYVHaziSlsZP7T9j+eGDPNm3H3PG3WlTMCjZnzeqrNze+nnKKjv7V2vF+fOtePHFD/n22ymmLzRUorI+gKuFjXnnxx0Mmjubj3fFYykocHPJhCeRGoKNjBhqqhQsP3yIP337DXUCg3hj5GhubtHKRSX2LJXdjbu7T6Sq4bElrxvFk/p7quobyPAfzj+/38KmUydpFlqPp+L6M65DR8OGsgr3kmGnLmZER5vWmtumn+VQt0Xc2LQ5b40aQ3iduq4qssfxlIBQ1bGrKosjTXWeFBCg+pua78+c4pWtm9mfeoEuTcJ59uZb6N+yZtyw+BIJCC7m7OSagsJCZm7awPx9e0rXLjAzn767VHY3PnAgbNp0/fPuHkZrT0Bw5OLujf09hVrz1dHDvLZtK2czrzCiTTv+MXS49C94EdP7EJRSI5VSR5RSx5VSz1Xw+oNKqVSl1O7ifw8bcVx3cWbJQktBAb/9egXz9+3h0d438u/ho00JBmYtWFO+/0Rr2Lix4ufdXcayyfhckS3V24IBgJ9SjO/QifVTp/F0XH82nDzB7YvmcT4rs9L3SDZT3+F0QFBK+QPvAqOAzsBkpVTnCjZdpLWOLf73sbPHdSdH00pnWiw8tGIZ6xJOMGPgYB5oe5ofdrQ25YtT0Z16TVdyB19ZcDIrpbcRnC1jcEAA/++Gvsy7cyLp2dncu2wJKVlZ121X0pxaVIPWpWs0S1DwTkbUEG4EjmutE7TWecBCYLwB+/UYMTEv4+d37Sig6hZbT8/OZsoXS/jxbBL/Hj6akZGHavQXp7J1mj15/eaqgoWnM+oGoE/TZnxy+52kXs3i3mUCY3ujAAAgAElEQVSLuXD12qBg9PogwlxGBIRmwJkyj5OKnytvglJqr1JqqVKqRUU7UkpNV0rFK6XiU1NTDSiaMexdVCQ5M5NJ/1vEsfR0Prjtdm7v2MnhL44zFx9n7nCNvuh5w7BTTw5OZuod1YxPxk8gpYKg4ExzqvA8TncqK6XuAkZqrR8ufjwV6Ku1fqzMNmFAltbaopR6BLhHaz2kqv16UqeyPbLz85mweD5nr1zh43F3lK5q5mjSMKNGqti7H08bIWM2b+ggdtXCQyV+OpfEtC+XEVU3lCV3T6JBSC3JZuqBzO5UPguUveNvXvxcKa11utbaUvzwY6C3Acf1SC9v2cjR9DTeHT32miUuzVzeUjjP04MBuL6J64amzZkz7k52LejIjI0bAMeaU4XnMiIg/AS0U0q1VkoFAZOAFWU3UEpFlXk4DjhkwHE9zoaTCSzYv5df9+rDgFbR17xmzxfHFZ2ZtjSHeHMnqnCPG5s1J2N1HF8dPcw3J46ZskazcB1D5iEopUYDbwD+wByt9ctKqReBeK31CqXU3ykKBAVABvAbrfXhqvbpbU1G6dnZjJo/l8a16/DFxHsrHFrqyGxns5pupMnIu7myiUspGPbZJwT6+7Nq8tQq14QW7icT00ymtebRVV+yKTGR5ZOm0LFxE8P2LQFBeILK+iceeOIin77R0O3lEZUzuw+hxltz4hjrEk7wdL/+hgYDMG/ki6eOuJHmK3OU75+wFFjpN/sD/IZ8Z2q5hLEkIDipoLCQf23fStuGjZgW28vw/Zt1AfTUC69MsPMMQf7+DItpyw9JZ8izWs0ujjCIBAQnLTt0gISLF/lDXH/8/eR0Ct9XUnvs16IlOQUF7D6fbG6BhGHkCuYES0EBb/6wne4RkYxo09bs4vgsGf3kWUrO+03NWuCnFNvOGD8JTfIjmUMCghMWHdhHclYmT8f1r9EjLVx9YfbmFBK+rH5ICF3DI9iWZGxAkPxI5pGA4IQVRw7RtUl4jc8ZL+36NVfX8AgSMjIM3afkRzKPBAQHZeRk8/P5ZIa0blPldnIXayxPHf1UU4XXrkNGbo6hHcuSH8k8EhActCPpDBoYFN26yu189e7ZrHZ9CbCeJbxOHQDSsq8atk9J82KeGhUQjOyo2ptyniA/fzo3CTewhK5n1AVV2vUFQJPigJB61biAIPmRzFNjAoLRHVXHMjJo06gRQf7+173myaNifLXGIswRVqvowp2Rm2PYPiU/knl8f2HfYlV1VJX/Q7Ml51BuQQF1goIqPFbZPDI1IQWEtOvXXAWFRanbA5Sx95YREVMkAJigxtQQbO2osrUmkWctqLB24IlcXWPxhJqPqJwrfz8lnclVfRdkToH3qDEBwdaOKluHvOVZrQT6VR8QPOHuWdr7azZXNhNWFxBkToF3qTEBwdaOKltrEvmFhQT5V3/65KIrfFmetQCoPCDInALvUmMCgq0dVbbWJBoEh5CRY1xHmrt4Qo1FuJ67BjaUfAca1qpV4esyp8C71JhOZbCtoyom5mWOHJl+zV1NRTWJlvXrsyExwSXldCWpsdQM7hrYkJyViZ9ShNepW+HrwcEtK1lzWeYUeKIaU0Owla01iZb1G5CWnU1WXp45BS1HLvTCDMlZmYTXrkNAJZl+ZU6Bd5GAUIGIiCnExSUyaFAhcXGJFdYqohs0AODUpYsuKYO9F3iZXyAq48pmwuTMTCLrhlb6uswp8C41qsnISN3CIwHYdCqRLuERhu9/1iy56xfGcOXf0dnMTDpXs0qgzCnwHlJDcFCL+vW5qVkLFh7Yi7V4co67efKMaOH78qxWzly+ROuGsqayr5CA4IQp3XqQdOUKW05f32nmCHsv8J4yv0ACUM2UdOUyVq2JadDI7KIIg0hAcMKtbdrSuHZt5u/bY8j+POUCby/pv/BezvxtnbxY1H8mNQTfIQHBCUH+/kzs3I0NiQkkuqhz2VYyv0A4wplgnnCpaGGc1g0kIPgKCQhOur9HLCEBAby2bauh+7X3Am9GM5H0X9RsCRcvElarFvVDQswuijCIBAQnhdepy8M9+/D18aP8nHzOsP16+oXVW5u3hHHBPOFiBq0bSv+BL5GAYIBf9+pDk9p1+L+tm9C+nutaeD2jgnnCxYvESHORT5GAYIA6QUE8eVM/diafY+2J42YXx+2k/6LmuWLJJT0n22s6lCUFt20kIBjk7s5daR/WmL9t+Y6rHpLOwl2kmch7ORrMT1++DECr+u4LCI5e1CUFt+0kIBgkwM+P/xtyK8mZmfxrx/dmF0cImzgazM9lXgGgWb16xhWmCs5c1CUFt+0kIBioV1RT7usey9zdu9hzPtns4gjhMslZmQBEVZHHyEjOXNQlBbftJCAY7Om4/kTUqctz335Ddn6+2cURwiXOZWYS5O9PWCXrIBjNmYu6rWucCAkIDqmqLTM0OJi/Dx3OsYx0Hlm5nNwCCQrC95zPyiKqbiiqZOyqizlzUZcU3LaTgGAnW9oyB0a35p/DRrDtzGl+s2oFloICU8rqSPuwdBALW1yxWKgXHOy24zlzUZcU3LZTnjpuvk+fPjo+Pt7sYlxn+/boSlaAakVcXOI1zy0+sI/nvv2GwdExvDd6LMEB7s027shKWa5cXUv4jolLFxKg/Jg/YaLbjpmSMo+EhOexWE4THNySmJiX5aJeAaXUTq11H0feKzUEO9nTljmxSzdeHnIr3yUm8Ls1K8mzWgG5Cxf287S/mat5edQJCnT5cco2zyYkPE9MzMtVLlwlnGNIQFBKjVRKHVFKHVdKPVfB68FKqUXFr/+glIo24rhmsLctc3LX7swaNJR1CSd4Ys0q8q1Wl2YHdSQtgeQl8nyellE2Oz+fOkFBLj2GzB9wP6cDglLKH3gXGAV0BiYrpTqX2+xXwEWtdVvgdeAVZ49rFkfaMqd2j+WFWwaz9sQxnlz7tUvL50haAkdTGUjAqLmsuhA/XNuhLPMH3M+IGsKNwHGtdYLWOg9YCIwvt814YG7xz0uBocpdwxMM5mgHVeLyXpx84ineGzMW8I27cE+7a/U1rq65OZPOoVZAILkuHiwh8wfcz4iA0Aw4U+ZxUvFzFW6jtS4ALgNh5XeklJqulIpXSsWnpqYaUDTXiIiYQlxcol1tmSV34XN+3gXAkLlzOJia6tKA4EhaAslL5DlcmVHW2eaYkIAAcq2uDQgyf8D9PKpTWWv9oda6j9a6T5MmVS/c7a2mxfYC4Gp+Hncums+i/XtdliHVFcNOpb/BNzjbHFMrIJDcfNcGBJk/4H5GBISzQIsyj5sXP1fhNkqpAKA+kG7Asb3SjBmwcvL93NC0GX/asI5n1q0hx0NmNVfXjCDrIJjD6Jqbs80xtYMCycyzGFmk68j8AfczYmD8T0A7pVRrii78k4B7y22zAngA2A7cBWzQnjoBwgXKj5/+zW9epnHtKXx6+wTe/nE7b/2wnYNpqbw3eizRJuaXL2lGKLlzLGlGAORLaDKjA25wcMtK5tPY1hzTPLQeO8+dQ2vt0tnKERFT5G/PjZyuIRT3CTwGrAUOAYu11geUUi8qpcYVbzYbCFNKHQf+AFw3NNWTOdP5VlVbrZ9SPNG3H3PG3cn5rEzGLfycVUePmLbIjr3NCNLf4L2cbY5p1aAhmXkWLuXmOnR8WZ/AM8lM5WqUv2uGoi+OrVVXW2c2J125zGOrV7I35TwDW7Vm1qAhtKzfwJDPYKuNG/2Aiv4eFIMGFbq1LML1nJn5+23CCX69cjnLJt5LbGSU3cd15jslqiYzlV3I2c43W9tqm9erz9K7J/OXAYOIP5fEiM/n8t5PP5BfPLvZHWRUR83iyGi5Eq0aFN2snLx40e7jyvwCzyUBoRrOdr7Zc5EN8PPjoZ69WTd1GoOjW/Pa9q3cvmgee1PO215gJ8ioDmGrVvUbUDswkD0p9q/7IfMLPJcEhGo4e9fsyEU2sm4o740Zx/tjxpGRk8Odi+fz8paNZOfnu7TtVUZ1CFsF+vvTK7IpO84m2f1eqYl6LgkI1XD2rtmZi+zwNu1Ye9+D3NOlG7N/3skflj3CgUMPuzS3izPNCMK72TuSqW/zFhxNTyMjJ7v6jcuQmqjnkk5lG3hC2t2dyWc5ezCWev5p171WUeptIexlb+rz+HNnmbh0If8ZM44RbdrZdSxP+E75Kmc6ld2boN9LecJY6N5Rzcg8UvFcPml7vdbMmTJRzh26R0RSOzCQTYkn7Q4InvCdEteTJiMvUlkbay7hXLE4Nh7cF0nSPds5k4okyN+foa3bsPbEMbeOhhOuIwHBi1TU9lqgg/ns9BAGzZ3Nx7viTVuu0yhyZ189I8+RM6lIZs6E29p14GJuLtuTzlS7vfB8EhC8SEUd1N06z+Yvo96gW3gE/7d1E8P++wnLDx+k0IG+IU+4GDt6d1+Tku55Sg1o1iy4pVU0dYOCWHnssNnFEQaQTmWDeEIn2ZbTifzz+y0cSL1A58ZNeH7AIOJa2D6Uz9ZORVe20RuxprOvrwvtqs9n7++1pBxPf7OadQkn+PHhR51aN9wTvkO+QGYqm8xTlvob0DKaLyfdx7+Hj+aSJZcpXyxh+lfLSbiYYehxjL5DrUl3945yxzmytZmofDn+NWIUp7/sxZoTxxw+tqd8h2o6qSEYwNZ8Re6UW5DPnJ938Z/4H7BYrdzXPZbf3XgTDUJqXbPdzJkVX+BnzKj8AuHKO3Aj9m3mKCN3HNtTakAl5SjUmiFzZ9M0tB7zJ0x0aF+e+B3yVlJDMJknTsUPCQjk/93Qlw0P/Iq7O3flsz0/M3juHOb8vJO8MiNCbO1U9Ka7eDPL5Cnt+/Zy5pz5KcXELt3YcfaMw7VRT/wO1UQSEKpga5oIT5yKX1L2Az+GclutX7N4VG26hUfwty0bGTlvLutOHLcrzba7FsaRlNrVc8U5ciSQlS3HXZ274K8Uiw/sc+j4nvgdqokkIFTCnjZNT5uKX1HZs849xT/jLMwedwf+SvHIqi+574ulHEn/ZeazJ1yMPbHGUR1315485RyVLUd4nboMbd2G/x066NCcBE/7DtVUEhAqYU+KXk9KCjdzZuVlP3nyeQZHx/D1vfczc+AQDqVd4Lb5n/Hi5u+4Ysm1+ULjSODwlIuYK3jrsqJGB7K7O3clPSebjYkn7X6vJ32HajLpVK6Ety4WoxR8951tZb+Yk8O/dnzPgn17aFSrNn+8eQATOnXBzwVLInpKR6ireevnNKLcBYWF3DznQ2IjI/ngttuNKZiwm3Qqu4AntWnam/La1rI3rFWLvw0expeT7qNV/fo8u34tdy1e4Lb1F3yRJzS7mSXAz487OnZiw8kEUrOvml0c4QAJCJXwlDZNW/oyylf9//rXl8nNtb3sXcMjWHz3ZF67dSRJmZe5Y9E8nl2/lrRs+9Ial+dNI5OM4q2fzahAdmenLli1ZkPCCWN2KNxKmoyq4AkzJ+0dn11S9Xe07FcsFt79aQef7N5F7cBAno7rz+Su3fH3c+7ewVubUoR9tNZ0e/9tJnbpxgu3DDa7ODWSM01GEhA8nL19GUZdeI9npDNj47dsTzpD1/AIXho0lB52LqbuinIJzzd+4efUDw7hszvuMrsoNZL0Ifgwe/syjKr6t20Uxud33M2bI8dw4WoWdy6ez5+//YaLOTkO7a8mt63XNG0bhXHiYsVrdwjPJgHBw9nbl2FkG7ZSirHtO7Luvmk81LM3Sw7uZ+h/5zBv3x6shfaNtPLWtnVhvzYNG5GclUWmxWJ2UYSdJCB4OE8Ynx0aHMzzAwax8t77ad+oMX/9bj23Lfgv285UnlbA3pFRwnc0DQ0FIK2CtZbl78KzyRKaXsBTlhvsENaYBRMmsvr4Mf7x/Sbu+2IJw2Pa8qf+A2nVoEHpdiUjo0omx5WMjAI84nMI1yqZx1JYrhYpfxeeT2oIwi5KKUa3a8+6+6bxVFx/tp45xYjPP+Uf328ubSKobpa3NB/5toDiEWnWcqMI7Jn9L8whAUE4JDgggN/e0Jdvpz7E2A4d+XDnTwz5bA5LDu6vNnOlURlBJbB4Jj9VHBDK1RAko6nnk4DgJu5qO3V3G21E3bq8eutIlt8zpXS28+WCRhVua/Qsb3enmjYrAHlbu3tBYXFyu3IpUDxp9r+omAQEN3DXalBmrjrVPSKSxXdP5o0Ro1l/6Q4shYHXvG611uavf33Zq2ctm7HWgTeuJLb/QgpBfv7ENGh4zfOeMvtfVE4Cghu4q+3U7DZaP6UY16ETr9/5AZb6L3GpoBFawxVrY4Ki/s769VOczgha09JhmP07dcSu88l0CQ+/bn1lTxgxJ6omAcEN3NV26ilttAF+fozr/SyjB1/gXOPdrE4bS0rSDL77zo/NW1swdKjjd7fuTjVtdgDylN+prfKsVvamnKdXVNMKX4+ImEJcXCKDBhUSF5cowcDDSEBwA3e1nXpaG22Qvz/DwvczOWoRjYMuoZSmsCCJPz77MNsOv23Xim1mMXutA0/7nVZnU+JJ8qxW+jRtZnZRhAMkILiBu9pOPbGNtqImj6DAXM6dfoHxCz9n2aEDWAoKHNp3TUiH4Ym/08ocz0jnmfVraNOwEYNatTa7OMIBEhDcwF1tp57YRltZ00ZY0GVyCwp4et0aBnz6EW/+sM3udNvu7jcwIwB54u+0IqnZV3loxTIC/fyZM+7O6/oPhHdwKtupUqoRsAiIBhKBiVrrixVsZwVKVt8+rbUeV92+Jdupb6gqffdNN51k65lTfLJ7FxsTTxLk78/4Dp2YFtuLjo2bmFBa4Yjs/Hwm/28RxzPSmT/hHnpERJpdpBrNmWynzobx54Bvtdb/UEo9V/z42Qq2y9Faxzp5LOGFYmJeviZdAfzS5KGUYkDLaAa0jOZERjqf7PmZZYcOsOTgfm5u0ZIHe/TillbRBPr7m/gJRFXyrVaeXLOKA6kXeH/MOAkGXs7ZJqPxwNzin+cCspCquIatTR5tGoXxt8HD2PbQdJ7p158TGRn8euVy+s5+nz+uX8O3J0843NdQEV8cpuruz3QiI527lixg/ckTvHDLYIbFtK1wO2+bWFeTOdtkdElr3aD4ZwVcLHlcbrsCYDdQAPxDa728kv1NB6YDtGzZsvepU9c3NYiaId9qZfOpRFYeO8KGkwlk5lmoGxjEoNatGdmmHQNbtaZOUJDD+/fGBXtmzqz6om/0Z6rseDn5+Xy+bzf/3r6N2oEBvDT4Vka3a1/hPsontIOiGqIn9oP4CpeumKaUWg9UVA98HphbNgAopS5qrRuW31Ap1UxrfVYpFQNsAIZqratcdFX6EESJPKuV7WdOs/bEMdYlHCc9J4dg/wAGtGzF0Jg2DGkdQ5PadezapzcGhOrKbPRnKr+/81mZ/Hfvbhbs38ul3FyGRMfw96HDaVKn8nNv7xKwwnku7UPQWg+r4sApSqkorXWyUioKuFDJPs4W/5+glNoI9ARkFW5hkyB/fwZGt2ZgdGteGjyMncnnWHP8KN8kHGf9yRMooGdkFMNi2jIspg1tGjZClcujA0V3u2XTT5RsMmOG9zYhueMz7U05zye7d7Hq2BEKtebWmLY81LMXfaKaVXiey67nXfHyr547sa6mc7bJ6FUgvUynciOt9R/LbdMQyNZaW5RSjYHtwHit9cGq9i01BFEdrTWH01JZl3CC9SdPsP9CCgAt69WnZ1RTujQJp3OTcLo0Cad+SMg17/WUGkJ1zUDlL/glKrrgG/GZZszQvPji9Rf58NE/8Niz2TzQoyct61/XKlyqoiaiikgNwXVc2mRUzYHDgMVAS+AURcNOM5RSfYBHtdYPK6X6AR8AhRR1Yr+htZ5d3b4lINRMZe8ug4NbEhPzss1tzcmZmXx78gRbTieyLyWF81ezSl9rFlqvNEB0aNyYVx/5kZde+jMWyxm7j2PkZ7LnIm50k9HVvDzOXLnM8Yx0DqRe4MCFCxxITeFibi4AJ594ivEL5zG2fUfu7tyVesHB1e6zsiaisqQPwbVMCwiuJAGh5jG6AzItO5tDqRc4kHqBg8X/J166yI0NdvJA1DJCAn9Z89dKMFmhM2kSfi9RoaE0rRtqyOSq6j6TkQGhbG3DUlBAWk42adnZpGdnk3I1izOXL5N05TJnrlzmzOXLZOTmlL430M+PDmGN6RIeQZfiWlWvpk3trnFs3OhHZc1ERaPMXBt8hQQE4SPc0QF5NS+PnT+2obAg6brX0vIa8MfDfyl9nLvuFvrce5imofWIrFuXhrVq0SAkhAYhtWgQHEKDWkX/1wsOJiQggOCAgNLlI6v7TOfPt2Ly5Gs/U0kzkLWwkJyCArLz88jOzycnP5+r+fm8+8863PNYKpl5Fq5YLFyx5BKSu4pWeja1SCWzMIx1F29nc1oPMvOuX+A+wM+PZqH1aF6vHi3rN6B5vXq0qFef1g0a0i6sMUHl5ntU15xVEelENp8EBOETKr+7VAwaVFjB88YfJ7hNIueyMknOzOTxvjfx4PL/kZyZSXJWVoUX2fKC/PwJDggoDhD+zGo1vfw6MUDRnf6sUx+xadrD9PnwPfKsheRZreQXWim08TvZt8FOpjVfSpBffulzBTqYo+oP6DrjCKtVm8a1a9O4dh2a1KlDZJ26+Pu5NluNDDM1n5kzlYUwTHBwy0ruLo3PClvZceJa/HKsx4FPxk8ofVxQWMjl3Fwu5eZwyZLLpZxcLllyuWKxYCkoILegAIu1AEtBARarldyCArJ1E+qo1OuOla2b0CuqKZuAEW3bE+TvT5CfH4H+/gT6+VMrMIBaAYHUCQyiVmAgtQMDqRUYQO2AQOoFh1A/JJgDuzpgseRfs98AZaFn8Hzibvg/w86XPUou+o72AwlzSUAQNnOmw9cWVaW5MFJVx6l6GKcfYbVrE1b72uyjVUlJeb3CY/Xp9DpjIkZTbwbMHFzpyO4qWSxnKnne3CGdERFTJAB4KWkyEjZxV1OAq4OOPccxamiqqz6TtNeLikgfgnC5mnjx8ZS5CpWR9npREWcCgqyHIGzibUs5GsHTF+DxlrUShPeQPgRhE3d1+HoSb0hnIe31wkhSQxA28aalHIUQjpGAIGwizRNC+D5pMhI2k+YJIXyb1BCEEEIAEhCEqBFkGUthC2kyEsLHlZ+vYLGc4siR6QDSBCiuITUE4ZPkjvgXCQnPX7dgTWFhNgkJz5tUIuGpJCAIn1NyR1w0b0KX3hG7Kih4evCpiZMKhWMkIAif4847YncHH0dUNnnQlycVCsdIQBA+x513xN7QHCOTCoWtJCAIn+POO2JvaI6RSYXCVjLKSPgcd62rAN6T40kmFQpbSA1B+Bx33hFLc4zwJVJDED7JXXfEsmSk8CUSEIRwkjTHCF8hTUZCCCEACQhCCCGKSUAQQggBSEAQQghRTAKCEEIIQAKCEEKIYhIQhBBCABIQhBBCFJOAIIQQApCAIIQQopgEBCGEEIAEBCGEEMWcCghKqbuVUgeUUoVKqT5VbDdSKXVEKXVcKfWcM8cUQgjhGs7WEPYDdwKbK9tAKeUPvAuMAjoDk5VSnZ08rhBCCIM5lf5aa30IQClV1WY3Ase11gnF2y4ExgMHnTm2EEIIY7ljPYRmwJkyj5OAvhVtqJSaDkwvfmhRSu13cdm8RWMgzexCeAg5F7+Qc/ELORe/6ODoG6sNCEqp9UBkBS89r7X+0tEDV0Rr/SHwYfFx47XWlfZL1CRyLn4h5+IXci5+IefiF0qpeEffW21A0FoPc3Tnxc4CLco8bl78nBBCCA/ijmGnPwHtlFKtlVJBwCRghRuOK4QQwg7ODju9QymVBMQBq5RSa4ufb6qU+hpAa10APAasBQ4Bi7XWB2zY/YfOlM3HyLn4hZyLX8i5+IWci184fC6U1trIggghhPBSMlNZCCEEIAFBCCFEMY8JCJIG4xdKqUZKqXVKqWPF/zesZDurUmp38T+f6qiv7veslApWSi0qfv0HpVS0+0vpHjaciweVUqll/hYeNqOcrqaUmqOUulDZ/CRV5K3i87RXKdXL3WV0FxvOxSCl1OUyfxMv2LRjrbVH/AM6UTShYiPQp5Jt/IETQAwQBOwBOptddheci38CzxX//BzwSiXbZZldVhd9/mp/z8D/A94v/nkSsMjscpt4Lh4E3jG7rG44F7cAvYD9lbw+GlgNKOAm4Aezy2ziuRgErLR3vx5TQ9BaH9JaH6lms9I0GFrrPKAkDYavGQ/MLf55LnC7iWUxgy2/57LnaCkwVFWTQ8VL1ZS/+WpprTcDGVVsMh74TBfZATRQSkW5p3TuZcO5cIjHBAQbVZQGo5lJZXGlCK11cvHP54GISrYLUUrFK6V2KKV8KWjY8nsu3UYXDW2+DIS5pXTuZevf/ITiZpKlSqkWFbxeE9SU64Ot4pRSe5RSq5VSXWx5gztyGZVyZxoMT1fVuSj7QGutlVKVjQ1upbU+q5SKATYopfZprU8YXVbh8b4CFmitLUqpRyiqOQ0xuUzCXLsouj5kKaVGA8uBdtW9ya0BQUsajFJVnQulVIpSKkprnVxc5b1QyT7OFv+foJTaCPSkqL3Z29nyey7ZJkkpFQDUB9LdUzy3qvZcaK3Lfu6PKeqDqol85vrgLK31lTI/f62Uek8p1VhrXWUCQG9rMqopaTBWAA8U//wAcF3tSSnVUCkVXPxzY+BmfCeluC2/57Ln6C5ggy7uTfMx1Z6Lcu3k4yjKCFATrQDuLx5tdBNwuUzTa42ilIos6VNTSt1I0bW++hsms3vLy/SK30FRm58FSAHWFj/fFPi6zHajgaMU3Qk/b3a5XXQuwoBvgWPAeqBR8fN9gI+Lf+4H7KNo1Mk+4Fdml9vgc3Dd7xl4ERhX/HMIsAQ4DvwIxJhdZhPPxd+BA8V/C98BHc0us4vOwwIgGcgvvlb8CngUeLT4dUXRYlwnir8TFY5W9IV/NpyLx8r8TewA+tmyX0ldIYQQAvC+Jq18H9AAAAAxSURBVCMhhBAuIgFBCCEEIAFBCCFEMQkIQgghAAkIQgghiklAEEIIAUhAEEIIUez/A2YjI/zb6oF1AAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"create_plot_for_lambda(X, y, 0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Underfitting\n",
"Underfitting means the model is not well trained. Having a high $\\lambda$ can cause underfitting."
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Optimization terminated successfully.\n",
" Current function value: 0.648216\n",
" Iterations: 14\n",
" Function evaluations: 29\n",
" Gradient evaluations: 29\n",
"Conjugate gradient found the following values for theta - first five values only: [ 0.326144 -0.00815789 0.16580133 -0.44666092 -0.11177511]\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl8VOW9+PHPk20SskICCRBCEmTfguACooCg4r5vpSq2vVxb7e2ibW2tAt7SW6/9XVvtYlXcEReqiAsuKIggKItIWGSLLIFsJJB9n+f3RyYhCTPJZObMnDMz3/frlVeSyZlznjkzeb7n2b5Haa0RQgghwswugBBCCGuQgCCEEAKQgCCEEMJBAoIQQghAAoIQQggHCQhCCCEAgwKCUupZpVSxUmqHi79PV0qVK6W2Ob4eMuK4QgghjBNh0H6eB/4GvNjFNp9rra8w6HhCCCEMZkgLQWu9FigzYl9CCCHMYVQLwR2TlVLfAMeA+7TWOztvoJSaB8wDiI2NnThixAg/Fk8IIQLfli1bjmut+3ryXH8FhK3AYK11lVLqMmA5MLTzRlrrp4CnACZNmqQ3b97sp+IJIURwUEod8vS5fpllpLWu0FpXOX5+H4hUSqX449hCCCHc45eAoJRKU0opx89nO45b6o9jCyGEcI8hXUZKqaXAdCBFKZUPzAciAbTWTwI3AD9WSjUBtcAtWtKsCiGEpRgSELTWt3bz97/RMi1VCBGgGhsbyc/Pp66uzuyiCCA6Opr09HQiIyMN26c/ZxkJIQJYfn4+8fHxZGZm4ugBFibRWlNaWkp+fj5ZWVmG7VdSVwgh3FJXV0dycrIEAwtQSpGcnGx4a00CghDCbRIMrMMX74UEBCGEEIAEBCFEgFm+fDlKKb799lunf587dy7Lli1ze3/Hjh3jhhtuAGDbtm28//77bX9bs2YNX3zxRY/LmJmZyfHjx3v8PLNJQBBC+NSCBcbub+nSpUydOpWlS5casr8BAwa0BRCjAkKgkoAghPCphQuN21dVVRXr1q1j8eLFvPrqq0DLjJt77rmH4cOHM2vWLIqLi9u2z8zM5Le//S05OTlMmjSJrVu3cskllzBkyBCefPJJAA4ePMiYMWNoaGjgoYce4rXXXiMnJ4dHHnmEJ598kscee4ycnBw+//xzSkpKuP766znrrLM466yzWL9+PQClpaVcfPHFjB49mh/96EcE6jIrmXYqhAgYb7/9NrNnz2bYsGEkJyezZcsWDh06xJ49e9i1axdFRUWMGjWKH/zgB23PycjIYNu2bfziF79g7ty5rF+/nrq6OsaMGcNdd93Vtl1UVBQPP/wwmzdv5m9/a1k2VVtbS1xcHPfddx8A3/ve9/jFL37B1KlTOXz4MJdccgm7d+9m4cKFTJ06lYceeoj33nuPxYsX+/fEGEQCghDCcAsWdGwZtE6ImT/fuy6kpUuX8rOf/QyAW265haVLl9LU1MStt95KeHg4AwYM4MILL+zwnKuuugqAsWPHUlVVRXx8PPHx8dhsNk6ePNmj469atYpdu3a1/V5RUUFVVRVr167lzTffBODyyy+nd+/enr9IE0lAEEIYbsGCUxW/UmBED0pZWRmffvopubm5KKVobm5GKcW1117b5fNsNhsAYWFhbT+3/t7U1NSjMtjtdjZu3Eh0dHTPX0AAkDEEIURAWLZsGbfddhuHDh3i4MGDHDlyhKysLJKTk3nttddobm6moKCA1atXe3yM+Ph4KisrXf5+8cUX88QTT7T9vm3bNgAuuOACXnnlFQBWrlzJiRMnPC6DmSQgCCF8av58Y/azdOnS01oD119/PQUFBQwdOpRRo0Zx++23M3nyZI+PMWPGDHbt2kVOTg6vvfYaV155JW+99VbboPLjjz/O5s2bGTduHKNGjWobmJ4/fz5r165l9OjRvPnmm2RkZHj1Ws2irDoaLjfIEcJadu/ezciRI80uhmjH2XuilNqitZ7kyf6khSCEEAKQgCCEEMJBAoIQQghAAoIQQggHCQhCCCEACQhCCCEcJCAIIQKCUop777237fc///nPLOgmD8by5cs7pJrwRE9TWa9YsYI//elPTo///PPPc+zYsR4dvzX5nj9IQBBC+ERR0RI2bMhkzZowNmzIpKhoiVf7s9lsvPnmmz2qnI0ICD111VVXcf/99zs9vicBwZ8kIAghDFdUtIQ9e+ZRX38I0NTXH2LPnnleBYWIiAjmzZvHY489dtrfDh48yIUXXsi4ceOYOXMmhw8f5osvvmDFihX86le/IicnhwMHDnR4zjvvvMM555zDhAkTmDVrFkVFRYDrVNYHDx5kxIgRzJ07l2HDhjFnzhxWrVrFeeedx9ChQ/nqq6+Alkr/nnvuOe34jzzyCJs3b2bOnDnk5ORQW1vLli1bmDZtGhMnTuSSSy6hoKAAgC1btjB+/HjGjx/P3//+d4/PWY9prS35NXHiRC2EsI5du3a5ve0XXwzWq1dz2tcXXwz2+PixsbG6vLxcDx48WJ88eVI/+uijev78+Vprra+44gr9/PPPa621Xrx4sb766qu11lrfcccd+o033nC6v7KyMm2327XWWj/99NP6l7/8pdZa65/+9Kd64cKFWmut3333XQ3okpIS/d133+nw8HC9fft23dzcrM8880x95513arvdrpcvX952zOeee07ffffdTo8/bdo0vWnTJq211g0NDXry5Mm6uLhYa631q6++qu+8806ttdZjx47Vn332mdZa6/vuu0+PHj3a6Wtw9p4Am7WH9a5kOxVCGK6+/nCPHndXQkICt99+O48//jgxMTFtj2/YsKEt/fRtt93Gr3/96273lZ+fz80330xBQQENDQ1kZWUBdJnKOisri7FjxwIwevRoZs6ciVKKsWPHcvDgwR69lj179rBjxw4uuugiAJqbm+nfvz8nT57k5MmTXHDBBW2vZ+XKlT3at6eky0gIYTibzXlyN1eP98TPf/5zFi9eTHV1tVf7+elPf8o999xDbm4u//rXv6irq+v2OZ3TZ7dPrd3TVNpaa0aPHs22bdvYtm0bubm5fPTRRz17EQaTgCBMY/Sgo7CO7OxFhIX16vBYWFgvsrMXeb3vPn36cNNNN3W4K9mUKVPabqm5ZMkSzj//fOD09NXtlZeXM3DgQABeeOGFtseNTGXdVTrt4cOHU1JSwoYNGwBobGxk586dJCUlkZSUxLp169pej79IQBCm8MWgo7CO1NQ5DB/+FDbbYEBhsw1m+PCnSE2dY8j+77333g6zjZ544gmee+45xo0bx0svvcRf//pXoOWuao8++igTJkw4bVB5wYIF3HjjjUycOJGUlJS2x41MZd35+HPnzuWuu+4iJyeH5uZmli1bxm9+8xvGjx9PTk4OX3zxBQDPPfccd999Nzk5OX69P7Okvxam2LAh0xEMOrLZBjN58kH/F8gLRUVLyMt7gPr6w9hsGWRnLzKs4rMSSX9tPUanv5ZBZWEKXw06+ltrS8durwFoa+kAQRkURHCTLiNhCl8OOvpTXt4DbcGgld1eQ17eAyaVSAjPSUAQpvDloKM/BUtLx11W7WIORb54LyQgCFP4etDRX4KlpeOO6OhoSktLJShYgNaa0tJSoqOjDd2vjCEI06Smzgm4ANBZdvaiDmMIEJgtHXekp6eTn59PSUmJ2UURtATo9PR0Q/cpAUEIL7QGtFCYZRQZGdm2mlcEJwkIQngpGFo6QoBBYwhKqWeVUsVKqR0u/q6UUo8rpfYrpbYrpc404rihQlb0yjkQwh+MGlR+Hpjdxd8vBYY6vuYB/zTouEEv2Fb0elKxB9s5EMKqDAkIWuu1QFkXm1wNvOjIzroRSFJK9Tfi2MEumOa5e1qxB9M5EMLK/DXtdCBwpN3v+Y7HOlBKzVNKbVZKbZaZDC2sMM+9m7sUus3Tit0K50CIUGCpdQha66e01pO01pP69u1rdnEswQrz3BcuNGY/nlbsVjgHQoQCfwWEo8Cgdr+nOx4T3QiWFb3gecUeTOdACCvzV0BYAdzumG10LlCutS7w07EDmlkrehcsAKVavuDUz950H3lasfvrHMhMJhHqDEl/rZRaCkwHUoAiYD4QCaC1flIppYC/0TITqQa4U2vdZW5rSX/tPaPSMisFRmUr8FWqaG/32zlrKbQEq0BMpyFCmzfpr+V+CBZlpQrOyIDgC0a81mC6P4MIbd4EBEsNKosWRsy7N3Kq5vz5PX6KXxnxWkN1JpN0k4n2JCBYkNUqOKOmnfqKEa81FGcyyYI/0ZkEBAuSCq5njHitoTiTSRb8ic4kIFiQVHA9Y8RrDZb7M/REqHaTCdck26kFGZFjP5TSMhv1WkMta6nNluFiID34WpHCPTLLyKJ8NT1TiFYy1TY4eTPLSFoIFhVqV6uhyszAH0qtSOEeCQjCkhYssP7sJm91vkJvneUD+DUoSAAQrWRQWfiMNxW6UQn1rExm+QirkYAgfCYUKnVvyCwfYTUSEMRpzOqq8UVCPW9X4vpyJW8orRURgUFmGQkA6poaKamuoaSmmokDBrBs106qGxqoamigurGB6oYGqhsbqWpooK6piYbmZhqam6hvbnb83PL7obfOJP+d0yc4pF7+FQOv2kR4WBjhKowwpYgIO/XdFh6OLSKC6IgIosIjeP6a6/jZB+8RHR5BbFQUcVFRxEZGEev4HhcVSYItmuSYGJJ79SLBFk1YayRx8HYWja9n4cgsH+ELMstIdKmxuZnCqiqOVJSTX1FOfkUF+RXlHKuspKSmmpKaaqoaGto9415+9fEHbb+FK9WuIo5yVNrhRIVHEBdlwxYRTlR4OLbwCKb+VynqZx8RphR/nHkRD366CqUUCrDr4TRrTbPdTrPW2LWmyW6n2W6nobmZuqYm6pubqG6sBeCbwkLqmpraAlJXly4RYWH0iYkhOaYXyTG96Bcby0VR92LDeR+9OxVuV338RlTY3szykWnJwhekhRBETtbVsr+sjP1lpW3fD5woo7CqkuZ273OYUvSPi2dAfDz9YmPp2yuWL18exvvPpJ+2zwcebOa/F4ahOl19u8ObLKmdZxlpralpbKS6sbXV0kh5XR2ltTWU1tR0+H68pobi6mr+kPWfOCu21vBixTLSExLJSExkUEISGYmJpCckkmCztW23Zk0YOA1DiunT7Z69MANIy0J0RVoIIUZrzZGKcnKLisgtLiS3uJh9Zcc5XnOqgoiOiOCM3n2YOGAAGQlJpCckkJ6QyKCERNLi4ogMD++402nA0y0/dqzIO23XA95kSe08bqBaWylRUfSLdW8fGzYscjpAW6P7Ulpby7bCQsrr6zr8LaVXL4b07kN27z5MD08lUhee9nyz+/i9bblI60K4IgEhAJyorWXTsXy+KSp0BIGitoosKiyc4SkpTM/MYmifZIb0TmZon2QGJiSc1qfub2avI8jO/qPTK+lJIx/j8gtbKsCK+jqOlJdzuKKcw+UnyTtxgrwTZazcv5e86BnMTV+GLayx7flN2kZx2H/AkcOMSOlL75gYv78ub2YnWWHtg7AuCQgW0PmKLStrESVhs/j4wH5WfXeAb4+XAC395MOTU7j0jKGMTU1jbL9UhiWnENXuar+oaAl5ux7ggBdXf1a//4G73OmjT7BFM7pfNKP7pZ72/LLaO/n20GRqi/+HMHshVc3JvFN8OatKooE3Wo4RG8e41FTOTc/g3PRBDE9O8Xkg9iYHka/HRURgkzEEkznrD26wR/Fc/vVsKp/EWQMGMjUjk3MGpjO2Xyq2CNcx3Cp9y2atMvbXcUtqqvn2eAnfHi9hd0kJWwuOcbiiHIA+0TGcmz6Ic9MHMWVQBllJvT0af+mKN++zVcdFhHHkFpp+ZmQf7Lr1GTQ1Hjnt8eaw/kw4az99Yno5eZZzZtwG0lkl3JPBZCMrcTNv9Xm0ooIN+YfZkH+EDUcOU1hdxYmVkxl1Yy7TM7OZPjiLKYMyiI2KMuR4nn4G5VahwU8Cgh8ZcRVe3dDABwf28e9dO/mPPjc7nQnjyRWbGVd/zirhnlTMRlbiVrn3s9aag+Unye7dm7veXcH6w4eoamwgKiycswemMz0zixlZ2WQl9fZ72azSihS+I/dU9iNv8s8cKS/ngU8/5pzFT/Krjz+goKqSxrA0p9t6MpPFzJWvvlhlbOXjdkUp1VbZ//Pyq9g87ycsufZGbh+fQ2FVJX/4fA0zX3yWi156jv/bsJ7vTp5wuS+jX4cZNwKS+zYHDmkh9JAnV+GV9fX8Y/OXPPf1VlBw5bAR3DR6DJP6D6S4+BXDrtj8dfW3YIHzPEXz55+qoLv6WHX3fE9ZoYXgzms7Ul7O6oN5fHhgH18ezceuNVMGZfC9MeOYlX1Gh0kCVnhN3pAWif9Jl5Ef9aQPttluZ9muHfx5w3pKa2u4bsQo7psylbS4+A7bGTkm4e855tJl5Jo75SmuruKNXTt4dUcuRysrSOnVixtHjWHO2PEMiE8w5TUZ+RmSMQv/k4DgR+5e8Ww4cpg/fL6G3cdLmNh/AL+/YAbjU513D/WE1e4TYKWAEAjnxpVmu521hw/ySu43/PsfaZz4YPJp23jbgnKH0Vf0MqvJ/2QMwY+664M9UVvLzz98jzlvvUFFfT1PzL6C12+4xZBgANZLKd26ZsHTvnwj1zyYGQyc9ZP35LWFh4UxIzObp6+8ltzXRvPwZ6sZ/Y/HAbjjrX+z7vAh5s/3/cWb0fdokIyugUVaCAbKLS5i3jvLKa2t4SeTzuHHk87uct2AJ6zWLeJMIJTRSL7qJy+vqyMpJpqznv4nx2tqyEntz+/On8akAQONKLZTRl/RyxiC/0kLwQI+yTvALcteJSIsjLdu+h4/P3fKacHA0ytYK86kEaf46s5nidHRzJ8Pn8/9D/4wYxbHqiq4admr3PfRSspqa7rfQSfuzPYx+orejFlNwnPSQjDAC99s5b/XrmFMv1SevuIa+sY6z75mxJVzIFx9W60v39dcXVVrrZgxw7h+8prGRv6x6Uue2rqJhCgbv79gBlcPH+HWSmh3r9Tlij7wSQvBJM12Ow+vXc3Cz1YzK2sIS6+7yWUwCCWBEgyMKqerq+eiImP7yXtFRnLflKmsuOX7ZCQm8cuP3ufOt98k35E2oyvutmLkij60SUDwUH1TEz95fwXPb9vKnTln8vfLriQmMvK07Yzu7gmWxHNWYNQAfXb2IsLCOqYYCQvrxTPPLDLmAJ2MSOnLGzfewvxpM9hScJRLXn6eZ7/eQlet/Z5kSE1NncPkyQeZPt3O5MkHJRiEEAkIHlr0+Ro+zjvA/GkzePCCGYSHOT+VCxa0dPG0/q+2/uzNeEIgclXuQH097bW/qtZaUVg4mIcffopPPpnjs/Ge8LAw7hh/Jh9+fy7npmfwh8/X8OP3VnS6890pMttHuEPGENzUfrGOPbw/T383jXHZ/8lvp05zex+B0P/vK65eu7/PSVcriVv/bhR/vTatNc9u28qf1n3GyJS+LL7qutO6LmVsIHTIGIKPtf4ztay41IQ1H+MHg/7N7UO6vyFJe9LdY76uWmxddSFZuSWjlOKHEybyryuu4cCJMm54Yyl5J8o6bCNjA8IdEhDc4GxALlI1cPjggz3aj5UrFV9wNX4yfXrgTaP1ZLzB3xcAF2Zl88p1N1Hd0MCNbyzl64JjHf4uYwOiO4YEBKXUbKXUHqXUfqXU/U7+PlcpVaKU2ub4+pERx/UXb25ZaBVm3bDG2dX4mjXGjqt4qn0yPl8EJzPO+fi0/iy76VYSbNF8/6032HzsqM+PKdlMg4fXAUEpFQ78HbgUGAXcqpQa5WTT17TWOY6vZ7w9rj8ZNSBn5j+O1VJeWEHreglXwSlQFwRmJvVm0t7bSIuL54cr3mJ3SbHPjtW5O7X1Hs0SFAKTES2Es4H9Wus8rXUD8CpwtQH7tYzs7EU0aVuHx8LCepGd7f60wlD/x3HVfWLlcRWjZ4j505//GMWL19xAXFQkd7z9bw52cc8Fb/hqlbYwhxEBYSDQ/h6Q+Y7HOrteKbVdKbVMKTXI2Y6UUvOUUpuVUptLSkoMKJoxCvWFLD5yHfWk4umAnKf/ON5UPt5c4Rpd6QXCtFMrBydPDExI4MVrbsBu19y+fBmlNT1Pd9GdYOhOFaf4a1D5HSBTaz0O+Bh4wdlGWuuntNaTtNaT+vbt66eidU1rzR/Xfcb++vM5b8phjwfkPP3H8aarx5sr3FDsYurqvARCsHB2AXBGcjKT9t5GcXU1//XBuzTbjU05LesbgosRAeEo0P6KP93xWButdanWut7x6zPARAOO6xe5xUVsOnaUu886hzgvbpAu/ziBzUotGVdcXQA8+ed4Hp4+kw35R3hp+zZDjtPK1SrtnnSnCuswIiBsAoYqpbKUUlHALcCK9hsopfq3+/UqYLcBx/WLV3dsJyYigutGjvZqPz35x/HFYKY7V7iBOogqunfjqDFMG5zFo1987lbuo660bz3K+obgYshKZaXUZcBfgHDgWa31IqXUw8BmrfUKpdT/0BIImoAy4Mda62+72qcVVipXNTRw7uInuXzocB6ZdYnX+/Pk1oRmrW4O5VXVwcBZxtmjlRXMfvl5JqQN4IVrrncrS6oz8tmwNrmFpo8s/3Y3v/zofV674WbOGpBuShkkIAgjvbx9Gw+t+YT/nXUJN4wa4/bzukr5IS1Ia5HUFT7ycd5+0mLjmNjfd3eo6o5Zg5lWHUSVysc73xs7nrMGDOSP6z6jor7O7ecF8hRc4T4JCC5ordlScJRz0gcR5mHT2ghm/cNZ9R89FGc/GSlMKR66YAYn6+p4eqt1kkcKa5CA4MKxykqKq6s5s/8As4sihKFG90vliqHDeW7bVkpqqnv8fKu2HoX3JCC4sKWgZebsmWn9u9lS+JrMfjLeLyafR31TE//Y9GWPn+uP8y75kcwhAcGFrwsL6BUZyfAUayyQszJfVxDSf228rKTe3DhqDK/kfsPRigqzi9NBqKd5MZMEBBf2lpYyrE8KES7uhCZOkX79wHTP2edi15qXcr1frGYkyY9kHqntXDhScZKMpESv9yNXscaS/mvjDIhPYGbWEP69awdNBqe08IbkRzKPBAQnmux2CiorGZTgfUAI1qtns/r1JcAa67qRoyitrWX94UNmF6WNpHkxT0gFBHcHqgoqK2nW2pCAYDVGVajSrx8cpg3OIsFmY8XeLhMH+JXkRzJPyASEngxUFVRVAi1Nak9YeVZMsLZYhGdsERHMHjKUjw/sp7G52eziAJIfyUwhExB6MlBVVfoG/zviDzTlZXk05S3Urp6lXz+wTc/MpqqxgW1FBWYXpY3c/9kcIRMQ3B2oKipagir7HSlRJwmWKW++brEEa6ALFt29P1MGtazGX+ejcQRZUxA4QiYguDtQlZf3AIqOOV68mfJmhavnUGuxiI666yZMsEUzPjWNzw8ZHxBkTUFgCZmA4O5AldFT3qTSFYFgasZgthcXUl7nfsI7d8iagsASMgHB3YGqYJ/yZoUWi/C9nnYTnj0wHbvWbC8uNLQcsqYgsESYXQB/Sk2d0+3gVHb2Inbu/hFh7bqNgmnKm7RYQkP7G+S4c2+LMX1TAdhRXMT5GZmGlcNmy3B0F53+uLCekGkhuCs1dQ4lMQ9wvCGJQJryJhW98EZidDSDE5PILSoydL+ypiCwSEBwoi76Mn797e8Zd06VaVPeelrBy/oC4Yq73YRj+6WSW2xsQJA1BYElpLqM3BUV3nJa6pvMW6izcKFc9QtjuPs5GtW3H+/u20NFfT0JNpthx3enq1ZYg7QQnEi0RQNQ3oNbDJrByiuiReDJTOoNwOHykyaXRJhFAoITSdGOgGDwFLzu9LSCt8r6AglAwWFwUhIAh05KQAhVEhCcaA0IJ/3cQrBKBd9TMn4RuNp/tjIcyRwPSQshZElAcKI1IJyorTW5JO6T9QXCE+2DeWxUFCm9eklACGESEJxI6RWL4lTWUzP0tII3o5tIxi+CT2psHCU11WYXQ5hEAoITUeHhpMbGcazSvIBg9Yo1ULu3RNfBPDmmF2UB1DIWxpKA4MLAhATL3XxcCCN0Fcz7xMRQVlvT1dNFEJOA4MKA+ASOVUpAcIeMXwSPPkHaQpAU3O6RgOBCRmIiRysraLDIXaSsTLqJAlfnYJ4YbaOmsZEmu92cAnXB00pdUnC7TwKCC2f0SaZZaw6ePGF2UYTwmc7B3OZYpW+1CyFvKnVJwe0+CQguDO2TDMD+slKTSyKE/9giwgGob2oyuSQdeVOpSwpu90lAcCG7d28UsE8CggghrS2E+mZrBQRvKvVgv8eJkSQguBAdEUlGYhJ7jh8/7W8yQCWCVVR4awvBWl1G3lTqkoLbfRIQujC6bz92lhR3eCyQBqg8GeyVAeLQ1uyYixoepkwuSUfeVOqSgtt9EhC6MDY1lSMV5R1SWATSAJUnOYYkL1Foa51dFBFmrarB20o9NXUOkycfZPp0u2n3OAkE1nrXLWZcvzSADjcNMWKASq7CRU/56zPT6JhdFBEW7p8DdqN992xe3gNkZy+SSt2HDAkISqnZSqk9Sqn9Sqn7nfzdppR6zfH3L5VSmUYc19fGpqYRphRbCo62PWbEAJUvr8I9yTEkeYmsz18tt9YWQqQFWgiB1D0bLLx+15VS4cDfgUuBUcCtSqlRnTb7IXBCa30G8BjwiLfH9Ye4qCjG9ktl/eFTNwm3+gCVJzmGPM1LJAEj+NQ2NQIQHWH+zRQDqXs2WBhxGXA2sF9rnae1bgBeBa7utM3VwAuOn5cBM5VS1hq1cuGCwZlsKyrkZF3LOIKnfZnBeBUu4w2+5evPjLPZcuV1dURHRGCzQECQ9QP+Z0RAGAgcafd7vuMxp9torZuAciC5846UUvOUUpuVUptLSkoMKJr3pg/Owq4169q1EjwZoDIjO6gnOYYkL5F1+PIz46o7plfDyrZbyJpN1g/4n/kdhe1orZ/SWk/SWk/q27ev2cUBYFxqGknR0Xx26KDZRekxX0w7DcaWTihy1R0zVD1Hos1mUqk6snr3bDAyIiAcBQa1+z3d8ZjTbZRSEUAiEBBLgMPDwjg/I5M1B7+j2aCEX1a+Cu9u0Z3cB8EcRn9mXHW79FIzHKdoAAAXUklEQVTHSYy2RgtB1g/4nxEdhZuAoUqpLFoq/luA73XaZgVwB7ABuAH4VOvWKsX6Ls4+g3f2fsuWgmOcPTC9x88vKlpCXt4D1NcfxmbL4Mc/XgRY70Pd2o3QeuXY2o0AyD+hyYwOuDZbhqO7qKPy5t6kxsUZezAvpKbOkc+eH3ndQnCMCdwDfAjsBl7XWu9USj2slLrKsdliIFkptR/4JXDa1FQrGxO7kUdHLKJ6X0aPU1UE0tS5ns7qsHJLR3TNVXfMvwsuZUBcvM+PL+lfrMmQqQRa6/eB9zs99lC7n+uAG404lr8VFS3h0IGfkBzl2VVzV5Ws1a58ejqrQ7qJAlfrZ699y7XvwPms31bGzLG+DQjSErUuSw0qW5G3c6EDaeqczOoILZ1ny9VFXwZAmo9bCLK+wLokIHTD2wo9kCpZmdUR2g6dPAm03C3QlwLpIinUSEDohrcVutGVrC/7XmVWR2jLO1mGArKSevv0OIF0kRRqzF+OaHHZ2Ys69HcCaBXjdoXurK82O3uRR5WsP/peZVZH6FryeDIDZyUQExnp0+M4+5+Slqg1SEDoRucKvawxif36B8zoQaVpVCUbSAPUIvB8uWQYc2/M9flxjLxIEsaSgOCG9hX6Q6tX8caunfznBXV+X8Ajfa/uWbBAZkD1VEuW0zCG9D4to4xPSEvUmmQMoYduHj2W+uYmlu/Z5fdjS9+reyTpnvtaU5FEhrdUBQ9Omy6pSEKYBIQeGt0vlbH9Unl1Ry7+XmwdCrOApCLqnpHnqDUVybJdOwHYV1rqdioSea+CjwQED9w6Zhx7So+zrbDAr8f19SwgK/yDe3p1H0pJ93zRAsotKgR6NsNIWmLBR1k1pdCkSZP05s2bzS6GU1UNDUxe/CSzzxjGoxfNBk7PVxSIg2RKnUpa1xVf9tG7WwZf78PKfPH6bnj9FXYvG8fO18eYVo5g+B+yAqXUFq31JE+eKy0ED8RFRXHtiFG8s/dbSmqqAypfkRGMvjIMpat7T/nyHNU2NpJbXMRtPyszrRyh9j9kVRIQPHTnhIk02e38a/OmgF6Kb4XK2OiU2mYm3fPVefNl2vGthcdotNs5x41Mvr4qRyD/DwUTCQgeykrqzbUjRrEk95uAng7q7j+4FQKHu8wsUyD2q391NJ+TK6cwqX/nGx36TyD/DwUTCQhd6C5NxH+dPZlmbadWO7+7m5nTQY1OceGvG+NISu3uGX2OvszP58QHk4nv4Z3SjCyHTKm2BgkILrjTpzkoMZEbR41hSf5FKBXT4flmTgf1tD/WCpWxFVsc3fF368nI/VY3NHg8W87IcoTClOpAIAHBBXf7NH9+zhS+qTqbL2p/bImkcAsWeN4f6+4/uCeBIxArencF6m1FFyyAOFsUe376c8DcbkBJrGgNMu3UhTVrwgBn50YxfXrHeyv/7auN/N/G9bx6/c0e3WLTSErB6tXul91fgn0qaKtAe52/WfUhK/fvJffHPw2ocgvXZNqpD/SkT/OHEyaSFhvHH9d9ht0H/1U9HQ+Q/ljzWKHbzV31TU18sH8fs7KGmF0UYRESEFzoSZ9mTGQk906ZyvaiQlbs2W1oOdwZD+jch/3gg4uoqzO/PzaQZiYZJZBe2yff5VHZUM/Vw0cGVCATviNdRl3oycpJu9Zc//orHK2sYNVtPyChhzM2XNmwIdMRDDqy2QYzefLB0x5v7bKw2qrPQOtKCQVz3nydQ+Un+eyOHxEeJteGwcKbLiNJf92FnqToDVOKhdNncu1rS/jLxvU8NO1CQ8rg6fxsSS8surKvtJQN+Ue4b/JUCQaijXwSDDQuNY3vjR3Pi9u3sbuk2JB99nQ8wKpNf6uWK1S9nLuNqLBwbh491uyiCAuRgGCw+yZPJckWzYOrV9Fs935GT0/nZ1u1D9uq5QpFVQ0NvLV7F5cNHUZyr17dP0GEDAkIBkuMjub3F0xna2EBj3+1wev9Ber8bKNXSgvjvJL7DVWNDdwxfoLfjy2fC2uTMQQfuGbEKL7IP8wTX21kQtoApmdmebW/QBsPaJ0Z1bo4rnVmFBBQryMY1TU18szXmzlvUAbj0/r79djyubA+aSH4yMPTZzIipS+//Oh9jlZUmF0cv+pupbR0H5nn9Z07OF5Tw91nnev3Y0tGU+uTgOAj0RGR/OOyK2my27l75TvUNzWZXSS/6W5mlFEZQSWw9ExDczP/2rKJif0HuJXq2miS0dT6JCD4UGZSbx69aDbbiwr552cPsGHDYJ/3nVqhj9ZfK6X9nWrarABk1Hv6xq4dFFRVcvdZ56JaVwv6kaygtz4JCD52yZChPDihgpHqMceVkO/uBmWVu045mxnV3NyLBx9cFNCrls2414FR72llfT1/2fgFZw0YyLTBmT4pa3cko6n1SUDwgxERz2MLa+zwmC/6Tq3SR+tsZtSYMU+xatUcrzOChlo6DKPe06e2bqK0tobfnT/dlNYBBO6MuVAis4z8oL7+iIvHje07tVIfbfuZUa1pNHbvvg2bLYOZMxcBnlUCCxacqvz9kQ5jwYKOLYPWunT+fP8EISPe04LKSp7ZuoWrho9gfGqaUUXzSKDNmAs10kLwA3/1nVqxj9ZZl8dvfxs4N083+14H3r6nWmsWfPYJGs19k6caWTQRhCQg+IGzvtMGexQxfe/3+XHM7qN11uURHm5MN1YopMPw9j1d/PUWPs47wK+mnE96QqIviiiCiAQEP+jcdxoemc6bJXP4wafNfFNU6LPjWKGP1pfdWP4eNzAjAHnznn51NJ9H1q/lkiFD+UHOmb4vrAh4XqW/Vkr1AV4DMoGDwE1a6xNOtmsGch2/HtZaX9Xdvq2Q/tqXDpef5La3llFWW8PTV17LuemDzC6ST/Q0fbcwRkl1NVcsfYnYqCjevnkO8QalYxfWZ+Yd0+4HPtFaDwU+cfzuTK3WOsfx1W0wCAUZiUm8dsPNDIhP4M633+ST7w6YXSSfsGI3VrBrstv56cp3qWqo55+XXyXBQLjN24BwNfCC4+cXgGu83F9ISYuLZ+n1NzEsOZm73n2bl7dvM7tIhrNiNxYE5zTVlgFwzUOrV/HVsXwWXXgRw5NTzC6WJRZLCvd422V0Umud5PhZASdaf++0XROwDWgC/qS1Xu5if/OAeQAZGRkTDx06vashGFU1NPCzD95j9cE8bhuXw++mTsMWITOCfSkQ7+DWfsqtM0rBE19u5P82rufus87hXi9nFXV3PHd0TmgHLS1EK1wUBCtvuoy6DQhKqVWAs8nLDwAvtA8ASqkTWuveTvYxUGt9VCmVDXwKzNRad9lHEuxjCJ012+08sn4tz3y9hdF9+/HX2ZeT3buP2cUKWoEYELors1KQ9df/x3UjRvHoRbO9XoBmxDmSMST/8+kYgtZ6ltZ6jJOvt4EipVR/RyH6A05vE6a1Pur4ngesAfyfiN3iwsPC+N3503n6ims4WlnBVa++zFu7d5ldrKASjKucO7+m7352L//vkktZuNCc1cjQsYvIWTAASWhnVd6OIawA7nD8fAfwducNlFK9lVI2x88pwHmA1HQuzMwewnu33s7ovv249+OV/PzD9zhRW2t2sYKC2YvMXJWpu793FcTmz9e8sO1rhjz+fwBU1NV79Zq8DZqdFyK6IgntrMnbMYRk4HUgAzhEy7TTMqXUJOAurfWPlFJTgH8BdloC0F+01ou723eodRl11mS3849NX/K3TRtJio7mDzNmcfGQoWYXy+da01zU1x/GZssgO3uRT/qaZ81awn//t++PA12/pp50y3Tetr6piflrPuH1XTu4MDObxVdfa2g3mCddRq66iNqTMQTf8ukYgllCPSC02lVSzK8//oBdx0u4OPsMfjt1GoOTThu3Dwr+GoAsKlrCjh3zCA/3/UBnd6/J04BQXF3FT95bwdbCAu4+6xx+ce55PLxQGdra8SQgrFkThuuWgfJ58BXmrkMQPjaqbz/eunkO902eyrojh7jk5ef50/q1VNbXm100w/krW2te3gMdgoGr4xhRubp6TatXP9DjbpnWldLfFBZw9atL2H28hH9eoJnC91n7WTiXXGLslE5PVma7zr00mOnT7UyefFCCgYVJCyGAFFVV8ecN6/j37p0kx/Til5PP44aRo4kMDze7aIZwfXWpmD7d7vfjGDHLprtj9eQYWmuW7d7Jg6tX0S82lr+c10RVwX2WmtIp00zNJy2EEJEaF8ejF81m+c1zyExK4oFPP+bil5/nzd07aWxuNrt4XgvGrLBGHWv38RK+/9Yb/GbVh0zsP4DlN8+h/vgjlrj/RXtWXYgo3CMBIQCNS03j9Rtu4ekrryEuKor7Pv6A6S8s5l9bvqK8rs5nx/X1ilN/pbno6jhGT03t7jV11y1TWlPDA59+zJVLX2L38RIWTLuQF6+5gT4xvSx1/4v2UlPnMHnyQekiCkDSZRTgtNZ8ejCPZ7/eyob8w/SKjOT6kaO5Y/wEQxe2+XPA1x+zjNw5jlGL1zx5TQ3Nzby0fRuPf7mBmsYGbhuXw3+dM5mk6Ji2bWTRl3BGZhkJAHaXFPPstq28s+dbGu3NzMjM5s4JZzIlPcPrVauhWPmYsZq5NcD/z7rPyDtxggsyMvn9BdM5o0/yadtKf71wxpuAIAlzgsjIvv149KLZ/HrK+bycu41Xcr/htreWMTA+gUvPGMplQ4czPjXNo+Bg1e4JX/Ln/Q8q6ut469tdLM3dzt6yUrKSevPMldcyIzPL5fvVWun7o0UlQoO0EIJYfVMTK/fv5Z29e1h3+CCNdjsD4uO59IxhXHbGMHLS+rsdHEKxheBrWmu2FxXyyo7tvLP3W+qamhjbL5Xvj8vh6uEjiQqS2WPCv6TLSHSror6OVXkHeH/fXtYdPkSDvZm02DimZWYxbXAWUwZlkNBF3nzpnjBOcXUVH+zfxxu7drCzpJhekZFcPXwkt44Zx5h+qWYXTwQ4CQiiRyrq6/kk7wAf5e1n/ZFDVDU0EBEWxplpA5iWmcm0wVmMTOl7WuvBXwO+wehoRQUfHNjHB/v3srXgGBoYmdKXW8eM4+rhI+UmNsIwEhCExxqbm/m6sIA1B79j7aHv2HW8BIDkmF5MGjCQM/v3Z1y/NMb0SyU2Ksrk0gaO+qYmdpYUsyH/MB8d2E9ucRHQEgRmnzGU2UOGMTT59IFiIbwlAUEYpri6is8OHWRj/hE2Hcsnv6ICAAWc0SeZcalpjO7bj6HJyQxLTiElppfXM5gCndaao5UVfF1YwNcFx/i6sIBdJcU02ltWPY9LTWP2kKHMPmMomUmn3S5ECENJQBA+c7ymhu1FhS1fxYXkFhVS2i4dd5/omLbgMLRPMlm9ezM4MYn+cfGEhwXfusfK+nr2l5Wyt6yU/WWl7CstZdfxYo7XtIytREdEMK5fGjn9+3NmWn8m9B9A316xJpdauvtCiQQE4Tdaa47X1LCn9Dj7ykrZW3qcvaXH2VdaSlVjQ9t2kWFhDIhPYHBiEhmJiWQkJpEWF0e/2DhSY+PoFxtLTGSkia/EObvWFFdXcbSygqMVFRyrrORoZQVHysvZV1ZKQVVl27a28AjO6NOH4ckp5KT1Z0Jaf4Ylp1gut5RMCAgtEhCE6bTWHKuq5NDJkxwpP8mh8nIOl5/kcEXL9won2VkTbDZSY+NI6RVLUnQ0iTYbidHRJNqiHd9txEfZsEVEEB0RgS0iAlt4OFHh4djCI4hwtEC0I3lc241v0BwvfpXi/Pk0NR4lLGIg4X1+RWOvK6lpbKSivo6y2lpO1NVyoraWstbvtbUUVlW2dfW0SoqOJj0+gSF9khnaJ5lhyckM7ZNCekJCQLSCZMpwaJGFacJ0SikGxicwMD4BBp2euK28ro6i6iqKqqsorq6mqKqK4uoqiqqrKamuYm9pFeX1dVTU1dNg9y5R3zlJW5ibvgxbWCMA9qZ8agvv4/n8jXx5cmLbdr0iI+kdHUPvmBj6RMeQmdSb/nHxDIiPZ2BCQtvr6W4w3erdMaG4qFB4RgKC8IvE6Jar/mHJKV1up7WmrqmJ8vo6yuvrqayvp66pifrmJuqbmqhraqKhuZmG5mbsWrcllm4d1lYKBpb/mQh7Y4f92sIauSt7LQ+N+CuJNhu9o2MM6bLq3B1TX3+IPXvmAVgmKNhsGS5aCHIbS9GRBARhKUopYiIjiYmMJC0u3qN9rFlT5PRx3XSMkSl9vSneabq6qY9VAkJ29iKnYwhGZ5EVgc/6HaBC9JA/73cQCN0xco8C4S5pIYig488r4kDpjklNnSMBQHRLWggi6PjzithfN/URwh+khSCCkr+uiCUFtQgmEhCE8JJ0x4hgIV1GQgghAAkIQgghHCQgCCGEACQgCCGEcJCAIIQQApCAIIQQwkECghBCCEACghBCCAcJCEIIIQAJCEIIIRwkIAghhAAkIAghhHDwKiAopW5USu1UStmVUi5v6qyUmq2U2qOU2q+Uut+bYwohhPANb1sIO4DrgLWuNlBKhQN/By4FRgG3KqVGeXlcIYQQBvMq/bXWeje03Ae3C2cD+7XWeY5tXwWuBnZ5c2whhBDG8sf9EAYCR9r9ng+c42xDpdQ8YJ7j13ql1A4fly1QpADHzS6ERci5OEXOxSlyLk4Z7ukTuw0ISqlVQJqTPz2gtX7b0wM7o7V+CnjKcdzNWmuX4xKhRM7FKXIuTpFzcYqci1OUUps9fW63AUFrPcvTnTscBQa1+z3d8ZgQQggL8ce0003AUKVUllIqCrgFWOGH4wohhOgBb6edXquUygcmA+8ppT50PD5AKfU+gNa6CbgH+BDYDbyutd7pxu6f8qZsQUbOxSlyLk6Rc3GKnItTPD4XSmttZEGEEEIEKFmpLIQQApCAIIQQwsEyAUHSYJyilOqjlPpYKbXP8b23i+2alVLbHF9BNVDf3fuslLIppV5z/P1LpVSm/0vpH26ci7lKqZJ2n4UfmVFOX1NKPauUKna1Pkm1eNxxnrYrpc70dxn9xY1zMV0pVd7uM/GQWzvWWlviCxhJy4KKNcAkF9uEAweAbCAK+AYYZXbZfXAu/he43/Hz/cAjLrarMrusPnr93b7PwE+AJx0/3wK8Zna5TTwXc4G/mV1WP5yLC4AzgR0u/n4ZsBJQwLnAl2aX2cRzMR14t6f7tUwLQWu9W2u9p5vN2tJgaK0bgNY0GMHmauAFx88vANeYWBYzuPM+tz9Hy4CZqpscKgEqVD7z3dJarwXKutjkauBF3WIjkKSU6u+f0vmXG+fCI5YJCG5ylgZjoEll8aVUrXWB4+dCINXFdtFKqc1KqY1KqWAKGu68z23b6JapzeVAsl9K51/ufuavd3STLFNKDXLy91AQKvWDuyYrpb5RSq1USo125wn+yGXUxp9pMKyuq3PR/hettVZKuZobPFhrfVQplQ18qpTK1VofMLqswvLeAZZqreuVUv9JS8vpQpPLJMy1lZb6oUopdRmwHBja3ZP8GhC0pMFo09W5UEoVKaX6a60LHE3eYhf7OOr4nqeUWgNMoKW/OdC58z63bpOvlIoAEoFS/xTPr7o9F1rr9q/7GVrGoEJR0NQP3tJaV7T7+X2l1D+UUila6y4TAAZal1GopMFYAdzh+PkO4LTWk1Kqt1LK5vg5BTiP4Ekp7s773P4c3QB8qh2jaUGm23PRqZ/8KloyAoSiFcDtjtlG5wLl7bpeQ4pSKq11TE0pdTYtdX33F0xmj5a3GxW/lpY+v3qgCPjQ8fgA4P12210G7KXlSvgBs8vto3ORDHwC7ANWAX0cj08CnnH8PAXIpWXWSS7wQ7PLbfA5OO19Bh4GrnL8HA28AewHvgKyzS6ziefif4Cdjs/CamCE2WX20XlYChQAjY664ofAXcBdjr8rWm7GdcDxP+F0tmIwfLlxLu5p95nYCExxZ7+SukIIIQQQeF1GQgghfEQCghBCCEACghBCCAcJCEIIIQAJCEIIIRwkIAghhAAkIAghhHD4/+A6zvTVcV0vAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"create_plot_for_lambda(X, y, 10)"
]
},
{
gitextract_5_24ji07/
├── .gitignore
├── README.md
├── ex1/
│ ├── PE1 - Linear Regression (Exercises).ipynb
│ ├── ex1data1.txt
│ └── ex1data2.txt
├── ex2/
│ ├── PE2 - Logistic Regression (Exercises).ipynb
│ ├── ex2data1.txt
│ └── ex2data2.txt
├── ex3/
│ ├── PE3 - Multi-class Classification and Neural Networks (Exercises).ipynb
│ ├── ex3data1.mat
│ └── ex3weights.mat
├── ex4/
│ ├── PE4 - Learning Neural Networks (Exercises).ipynb
│ ├── ex4data1.mat
│ └── ex4weights.mat
├── ex5/
│ ├── PE5 - Logistic Regression (Exercises).ipynb
│ └── ex5data1.mat
├── ex6/
│ ├── Support Vector Machines (Exercises).ipynb
│ ├── emailSample1.txt
│ ├── ex6data1.mat
│ ├── ex6data2.mat
│ ├── ex6data3.mat
│ ├── process_email.py
│ ├── spamTest.mat
│ ├── spamTrain.mat
│ └── vocab.txt
├── ex7/
│ ├── K-means Clustering and Principal Component Analysis (Exercises).ipynb
│ ├── ex7data1.mat
│ ├── ex7data2.mat
│ └── ex7faces.mat
├── ex8/
│ ├── Anomaly Detection and Recommender Systems (Exercises).ipynb
│ ├── ex8_movieParams.mat
│ ├── ex8_movies.mat
│ ├── ex8data1.mat
│ ├── ex8data2.mat
│ ├── load_movie_list.py
│ └── movie_ids.txt
├── requirements.txt
└── week6/
└── 1-Evaluating-a-Learning-Algorithm.md
SYMBOL INDEX (2 symbols across 2 files) FILE: ex6/process_email.py function process_email (line 10) | def process_email(email_contents): FILE: ex8/load_movie_list.py function load_movie_list (line 1) | def load_movie_list():
Condensed preview — 38 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (906K chars).
[
{
"path": ".gitignore",
"chars": 85,
"preview": "*.DS_Store\noctave/\nexercises/\n__pycache__/\n*.pyc\n*.swp\n*.swo\n.ipynb_checkpoints\nenv/\n"
},
{
"path": "README.md",
"chars": 2138,
"preview": "# Stanford CS229 Machine Learning in Python\n\nThis repository contains the problem sets for Stanford CS229 (Machine Learn"
},
{
"path": "ex1/PE1 - Linear Regression (Exercises).ipynb",
"chars": 59507,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Linear Regression\\n\",\n \"\\n\",\n "
},
{
"path": "ex1/ex1data1.txt",
"chars": 1359,
"preview": "6.1101,17.592\n5.5277,9.1302\n8.5186,13.662\n7.0032,11.854\n5.8598,6.8233\n8.3829,11.886\n7.4764,4.3483\n8.5781,12\n6.4862,6.598"
},
{
"path": "ex1/ex1data2.txt",
"chars": 657,
"preview": "2104,3,399900\n1600,3,329900\n2400,3,369000\n1416,2,232000\n3000,4,539900\n1985,4,299900\n1534,3,314900\n1427,3,198999\n1380,3,2"
},
{
"path": "ex2/PE2 - Logistic Regression (Exercises).ipynb",
"chars": 155530,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Linear Regression\\n\",\n \"\\n\",\n "
},
{
"path": "ex2/ex2data1.txt",
"chars": 3775,
"preview": "34.62365962451697,78.0246928153624,0\n30.28671076822607,43.89499752400101,0\n35.84740876993872,72.90219802708364,0\n60.1825"
},
{
"path": "ex2/ex2data2.txt",
"chars": 2233,
"preview": "0.051267,0.69956,1\n-0.092742,0.68494,1\n-0.21371,0.69225,1\n-0.375,0.50219,1\n-0.51325,0.46564,1\n-0.52477,0.2098,1\n-0.39804"
},
{
"path": "ex3/PE3 - Multi-class Classification and Neural Networks (Exercises).ipynb",
"chars": 73750,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Multi-class Classification and Ne"
},
{
"path": "ex4/PE4 - Learning Neural Networks (Exercises).ipynb",
"chars": 91969,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Neural Networks Learning\\n\",\n "
},
{
"path": "ex5/PE5 - Logistic Regression (Exercises).ipynb",
"chars": 106056,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Regularized Linear Regression and"
},
{
"path": "ex6/Support Vector Machines (Exercises).ipynb",
"chars": 158073,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Support Vector Machines\\n\",\n \""
},
{
"path": "ex6/emailSample1.txt",
"chars": 393,
"preview": "> Anyone knows how much it costs to host a web portal ?\n>\nWell, it depends on how many visitors you're expecting.\nThis c"
},
{
"path": "ex6/process_email.py",
"chars": 1889,
"preview": "import re\nfrom nltk.stem import PorterStemmer\n \n# Load vocabulary\nvocabulary = []\nwith open('vocab.txt', 'r') as f:\n "
},
{
"path": "ex6/vocab.txt",
"chars": 11852,
"preview": "aa\nab\nabil\nabl\nabout\nabov\nabsolut\nabus\nac\naccept\naccess\naccord\naccount\nachiev\nacquir\nacross\nact\naction\nactiv\nactual\nad\na"
},
{
"path": "ex7/K-means Clustering and Principal Component Analysis (Exercises).ipynb",
"chars": 95810,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# K-means Clustering and Principal "
},
{
"path": "ex8/Anomaly Detection and Recommender Systems (Exercises).ipynb",
"chars": 61873,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Anomaly Detection and Recommender"
},
{
"path": "ex8/load_movie_list.py",
"chars": 323,
"preview": "def load_movie_list():\n movies = []\n with open('movie_ids.txt', 'r', encoding = \"ISO-8859-1\") as f:\n for li"
},
{
"path": "ex8/movie_ids.txt",
"chars": 48432,
"preview": "1 Toy Story (1995)\n2 GoldenEye (1995)\n3 Four Rooms (1995)\n4 Get Shorty (1995)\n5 Copycat (1995)\n6 Shanghai Triad (Yao a y"
},
{
"path": "requirements.txt",
"chars": 184,
"preview": "cycler==0.10.0\nkiwisolver==1.1.0\nmatplotlib==3.1.1\nnltk==3.4.5\nnumpy==1.16.4\npandas==0.24.2\nPillow==8.1.1\npyparsing==2.4"
},
{
"path": "week6/1-Evaluating-a-Learning-Algorithm.md",
"chars": 3929,
"preview": "# Advice for Applying Machine Learning\n\n## Evaluating a Learning Algorithm\n\n### What to do next?\n\nIf you trained your mo"
}
]
// ... and 17 more files (download for full content)
About this extraction
This page contains the full source code of the rickwierenga/CS229-Python GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 38 files (11.4 MB), approximately 505.6k tokens, and a symbol index with 2 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.