Repository: rickwierenga/CS229-Python Branch: master Commit: 2b749ee78829 Files: 38 Total size: 11.4 MB Directory structure: gitextract_5_24ji07/ ├── .gitignore ├── README.md ├── ex1/ │ ├── PE1 - Linear Regression (Exercises).ipynb │ ├── ex1data1.txt │ └── ex1data2.txt ├── ex2/ │ ├── PE2 - Logistic Regression (Exercises).ipynb │ ├── ex2data1.txt │ └── ex2data2.txt ├── ex3/ │ ├── PE3 - Multi-class Classification and Neural Networks (Exercises).ipynb │ ├── ex3data1.mat │ └── ex3weights.mat ├── ex4/ │ ├── PE4 - Learning Neural Networks (Exercises).ipynb │ ├── ex4data1.mat │ └── ex4weights.mat ├── ex5/ │ ├── PE5 - Logistic Regression (Exercises).ipynb │ └── ex5data1.mat ├── ex6/ │ ├── Support Vector Machines (Exercises).ipynb │ ├── emailSample1.txt │ ├── ex6data1.mat │ ├── ex6data2.mat │ ├── ex6data3.mat │ ├── process_email.py │ ├── spamTest.mat │ ├── spamTrain.mat │ └── vocab.txt ├── ex7/ │ ├── K-means Clustering and Principal Component Analysis (Exercises).ipynb │ ├── ex7data1.mat │ ├── ex7data2.mat │ └── ex7faces.mat ├── ex8/ │ ├── Anomaly Detection and Recommender Systems (Exercises).ipynb │ ├── ex8_movieParams.mat │ ├── ex8_movies.mat │ ├── ex8data1.mat │ ├── ex8data2.mat │ ├── load_movie_list.py │ └── movie_ids.txt ├── requirements.txt └── week6/ └── 1-Evaluating-a-Learning-Algorithm.md ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ *.DS_Store octave/ exercises/ __pycache__/ *.pyc *.swp *.swo .ipynb_checkpoints env/ ================================================ FILE: README.md ================================================ # Stanford CS229 Machine Learning in Python This repository contains the problem sets for Stanford CS229 (Machine Learning) on Coursera translated to Python 3. It also contains some of my notes. Check out the [course website](http://cs229.stanford.edu/) and the [Coursera course](https://www.coursera.org/learn/machine-learning). Please note that your solutions won't be graded and this repo is not affiliated with Coursera or Stanford in any way. ## Installation Make sure you have jupyter notebooks installed. You can find instructions [here](https://jupyter.org/install). The following Python packages are used: - [Numpy](https://www.numpy.org) - [Scipy](https://www.scipy.org) - [Matplotlib](https://matplotlib.org) - [Pandas](https://pandas.pydata.org) - [Pillow](https://python-pillow.org) - [Natural Language Toolkit](http://www.nltk.org) You can install all dependencies using: ```shell python3 -m pip install -r requirements.txt ``` ## Instructions 1. Please download the exercises (pdf) from the Coursera course. Some instructions are included in the Notebooks. 2. Complete the exercises in the exercises Notebook. 3. Compare your answers to the code in solutions Notebook. ## Contents 1. Linear Regression 2. Logistic Regression & Regularization 3. Multiclass Classifcation & Neural Networks 4. Neueral Networks Learning 5. Regularized Linear Regression and Bias v.s. Variance 6. Support Vector Machines 7. K-means Clustering and Principal Component Analysis 8. Anomaly Detection and Recommender Systems ## Copyright Notice All code, exercises, data and other files in this repo are ©Stanford University. If you are unhappy about me hosting these files on GitHub for educational purposes, please send me an email. The code was 'translated' to Python by Rick Wierenga. Some of the instructions are modified to better fit the Python ecosystem by me too. The data, background information and the intended exercise are the same. ### Solutions At the request of Stanford staff, I removed the notebooks with solutions from this repository. The exercises and notes are still available. --- ©2020 Rick Wierenga ================================================ FILE: ex1/PE1 - Linear Regression (Exercises).ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Linear Regression\n", "\n", "Stanford CS229 - Machine Learning by Andrew Ng. Programming exercise 1.\n", "\n", "Please check out [the repository on GitHub](https://github.com/rickwierenga/CS229-Python/). If you spot any mistakes or inconcistencies, please create an issue. For questions you can find me on Twitter: [@rickwierenga](https://twitter.com/rickwierenga). Starring the project on GitHub means a ton to me!" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pylab as plt\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Linear Regression with a single variable\n", "---\n", "In this part of this exercise, you will implement linear regression with one variable to predict profits for a food truck. Suppose you are the CEO of a restaurant franchise and are considering different cities for opening a new outlet. The chain already has trucks in various cities and you have data for profits and populations from the cities. You would like to use this data to help you select which city to expand to next. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PopulationProfit
06.110117.5920
15.52779.1302
28.518613.6620
37.003211.8540
45.85986.8233
\n", "
" ], "text/plain": [ " Population Profit\n", "0 6.1101 17.5920\n", "1 5.5277 9.1302\n", "2 8.5186 13.6620\n", "3 7.0032 11.8540\n", "4 5.8598 6.8233" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# start by loading the data\n", "data = pd.read_csv('ex1data1.txt', header=None, names=['Population', 'Profit'])\n", "\n", "# initialize some useful variables\n", "m = len(data) # the number of training examples\n", "X = np.append(np.ones((m, 1)), np.array(data[\"Population\"]).reshape((m,1)), axis=1) # Add x0, a vector of 1's, to X.\n", "y = np.array(data[\"Profit\"]).reshape(m, 1)\n", "\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualising the data\n", "Plotting helps us get insight in the data we are working with. Using the `'bx'` option, we get blue crosses. You can read more about markers [here](https://matplotlib.org/api/markers_api.html)." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Text(0.5, 1.0, 'Relation between profit and population')" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xm8XVV99/HPF8IYIYCJlCmkIvbRSgW5AbFguSAWKCpQy6AGHCNRHqWRRtAnEUktD5FgUSs+KsqkhrQiAmIVyBWHFkxAZlRAQUAIYZBJKgK/54+1N3ffkzPse++Zz/f9eu3XOWeP65y77/rttdbeaykiMDOzwbVOpxNgZmad5UBgZjbgHAjMzAacA4GZ2YBzIDAzG3AOBGZmA86BoEdI2lvSvZPY/ouSFjYzTdl+T5J0frP3268kbSnpR5KekLRU0sckfaVDaZklKSRN6cTxa5H0Q0nvneC2MyU9KWndZqern3XVCdDvJN0FbAk8BzwJ/CdwbEQ82eTjvBN4b0Tsmc+LiGOaeYxmkHQS8LKIeEen09JGc4GHgE2j4iEeSbOA3wDrRcSz7U9a78n+p94bEVcARMRvgRd1NFE9yCWC9ntTRLwI2BnYBTixw+mxJil5Zb09cGtlEDDrJAeCDomIB4DvkwICAJI2kHSapN9KWp1V52xUbXtJJ0i6M6tiuFXSIdn8VwBfBPbIisi/z+afLemfC9u/T9Idkh6RdLGkrQvLQtIxkm6X9HtJ/yZJdb7OhpIuyNJynaRXF/a1taRvSVoj6TeSPpTN3x/4GHB4ls4bJA1Luqmw7eWSVhY+/1jSwfX2my1bp/D7PCxpuaQtsmV5dcjR2e/8kKSP1/pi2e/2xSwtT0i6StL2Fb/VByXdDtyezXudpJWSHsteX5fvCzgaWJB95zdUVK39KHv9fbZ8jyrp2U3Sf2d/l/slfV7S+mX+dpLWzc6vhyT9Gvi7On9TJN0l6cTs/HpU0tckbVhY3ugc+pCkX2fH+7SkdbJlY6oTVaeKStIOklZkf8eHJH1d0mbZsvOAmcAl2e+1oHJf2XlycZbGOyS9r7Dvk7Jz49zsb3uLpKF6v0nfighPbZqAu4A3ZO+3BW4Czigs/wxwMbAFsAlwCXBKtmxv4N7Cuv8AbE0K5ocDTwFbZcveCfyk4thnA/+cvd+HVD3xGmAD4HPAjwrrBnApsBnpH20NsH+N73QS8CfgrcB6wPFk1RtZ2q4FFgHrAy8Ffg38bWHb8wv72gj4H2B6tv1q4L7st9gIeBp4cYn9fhi4OvuNNwD+H/DNbNms7Pt9Odvnq4E/Aq+o8f3OBp4AXp/t64zib5vt6/Lsb7ZR9vooMIdU9Xpk9vnFlX+Hyt+gkLYpdc6hXYHXZvueBdwGHFfmbwccA/wC2C5L50i945HO15sL6/+U8Z1DI9l2M4Ffkapwqv3dx3xv4IeFdV8G7JcdYwYpWP5rtf+pGvv6EfAFYEPSRdcaYJ9COv4HOBBYFzgFuLrT+URH8qZOJ2CQpuykfTLLWAK4EtgsWyZSZr5DYf09gN9k7/emEAiq7Pt64C3Z+3dSPxCcBSwpLHsRKTOflX0OYM/C8uXACTWOe1Lxn4eUSd8P7AXsDvy2Yv0Tga8Vtj2/YvmPgUNJmd0PsmPvDwwDN2brNNrvbcC+hWVbZd8vzzwD2Law/GfAETW+39nAsorf6jlgu8JvtU9h+RzgZxX7+G/gnZV/h8rfgBKBoEr6jgO+Xfhc828HrACOKSx7Y73jkc7X4voHAneO4xzav7D8A8CV1f7uld+bQiCokqaDgZ9XpLFqICAFsOeATQrLTwHOLqTjisKyVwJPN+N/vdcmNxa338ERcYWkvwG+Qbr6/T3pamdj4NpCLYxIVyprkXQUMJ904kP6R5xeMg1bA9flHyLiSUkPA9uQ/rEAHiis/wfqN8DdU9jX80p3N21N+ofcWln1VGZdUmZfy1VkQS97/yjwN6Sr9quydbZvsN/tgW9Ler6w/DlSQ31uot/vSUmPZN/vnsrl2fy7K7a/m/TbTpqklwOnA0Ok82UKqXRUVOu7FdOcp6uRyvXz6p8y51CtbUuTtCWpFLYXqWS4DumcKGNr4JGIeKIiHcXqn8rfakNJU2LAGuvdRtAhEXEV6erwtGzWQ6Sqj7+MiM2yaVqkhuUxsjrqLwPHkqocNiMV4fMI0qgh8nekzDLf31RSlct9E/w62xX2tQ6pSuZ3pIzgN4Xvs1lEbBIRB9ZJZx4IXp+9v4oUCP6G0UDQaL/3AAdULN8wIprx/V5Equ74XWF58XuM+W0zMyn325ZpQD6TVL2zY0RsSmpnqdd+U3Q/he+SpauRyvXz713mHKq17VOkIJb7szrH/xfS77JT9n3fwdjvW+83+x2whaRNKtIx0fOgbzkQdNa/AvtJenVEPE/K3D8j6SUAkraR9LdVtptK+gdYk633LuBVheWrgW2LjYgVvgm8S9LOkjYg/bNdExF3TfB77Crp0KyB7jjS1fvVpCqXJyR9VNJGWWPlqyTNLqRzVt6ImPkv4C+A3UhVLLeQMpzdGW1MbbTfLwKfyht1Jc2Q9JYJfjeAAyXtmf2ei0lVYffUWPcy4OWS3iZpiqTDSVUOl5Y4zhrgeVKbRy2bAI8DT0r6X8C80t8iVRN9SNK2kjYHTiixzQez9bcAPg5ckM0vcw79k6TNJW1HarfJt70eeL3SPf/TqH/n3Cak6tTHJG0D/FPF8tXU+L2yv9F/AadI2lDSXwHvAfzcSwUHgg6KiDXAuaRGT4CPAncAV0t6HLiClClWbncrsJRU97wa2InUkJdbAdwCPCDpoSrbXwEsBL5FukrcAThiEl/lO6QG67yR9NCI+FNEPAccRGqk+w2p1PMVYFq23b9nrw9Lui5L21OkKodbIuKZbPl/A3dHxIPZOo32ewap0f0Hkp4gBaXdJ/H9vgF8AniE1Fhb87mHiHg4S9tHgIeBBcBBEbHW36HKtn8APgX8NLvj57VVVjseeBupnenLjGauZXyZdKfaDaTf+MIS23yD1Fbza+BO4J+ztJY5h75Dqra6HvguqV2BiLg8S/eN2fJ6QfKTpAbpx7J9VKb5FOD/ZL/X8VW2P5JUffo74NvAJ7K0W4GyRhIzq0Lpls97I+L/dDot7aaKh7XGuW2Qqq/uaHrCrOlcIjAzG3AOBGZmA85VQ2ZmA84lAjOzAdcTD5RNnz49Zs2a1elkmJn1lGuvvfahiJjRaL2WBYLs3uFzSU9zBvCliDhDqevh95HdAw98LCIuq7evWbNmsWrVqlYl1cysL0kq8/R4S0sEzwIfiYjrsif7rpV0ebbsMxFxWp1tzcysTVoWCCLiftKDJkTEE5Juo0n9rZiZWfO0pbFYaeSlXYBrslnHSrpR0lezR92rbTNX0ipJq9asWVNtFTMza4KWB4Ksk65vkfpMf5zUadYOpO4B7id1lbCWiPhSRAxFxNCMGQ3bOszMbIJaGggkrUcKAl+PiAsBImJ1RDxX6GRtt1amwczM6mtZIFDqVP8s4LaIOL0wf6vCaoeQuk82M7PMkiUwMjJ23shImt8KrSwR/DWpJ8p9JF2fTQcCSyTdJOlG0qhT/9jCNJiZ9ZzZs+Gww0aDwchI+jx7dv3tJqqVdw39hOoDZtR9ZsDMbNAND8Py5SnznzcPzjwzfR4ebs3x3MWEmVkXGh5OQWDx4vTaqiAADgRmZl1pZCSVBBYuTK+VbQbN5EBgZtZl8jaB5cvh5JNHq4laFQwcCMzMuszKlWPbBPI2g5UrW3O8nhiPYGhoKNzpnJnZ+Ei6NiKGGq3nEoGZ2YBzIDAzG3AOBGZmA86BwMx6Rru7XhgUDgRm1jPa3fXCoOiJMYvNzKD9XS8MCpcIzKyntLPrhUHhQGBmPaWdXS8MCgcCM+sZ7e56YVA4EJhZz2h31wuDwl1MmJmRbkGdPXtsm8PISAoyCxZ0Ll2T4S4mzMzGYZBvTfXto2ZmDPatqS4RmJllBvXWVAcCM7PMoN6a6qohMzNG2wQOPTSVBIaHR29Vhd5uNG7EJQIzM0ZvTT3iiBQAIH1etqz/G41dIjAzY+zVfrHR+MIL+7/R2CUCM7MKg9Zo7EBgZlZh0BqNHQjMzAoGsT8jBwIzs4JB7M/IfQ2ZmfWpjvc1JGk7SSOSbpV0i6QPZ/O3kHS5pNuz181blQYzM2uslVVDzwIfiYhXAq8FPijplcAJwJURsSNwZfbZzMw6pGWBICLuj4jrsvdPALcB2wBvAc7JVjsHOLhVaTAzs8ba0lgsaRawC3ANsGVE3J8tegDYssY2cyWtkrRqzZo17UimmdlAankgkPQi4FvAcRHxeHFZpJbqqq3VEfGliBiKiKEZM2a0OplmZgOrpYFA0nqkIPD1iLgwm71a0lbZ8q2AB1uZBjMzq6+Vdw0JOAu4LSJOLyy6GDg6e3808J1WpcHMzBprZadzfw3MAW6SdH0272PA/wWWS3oPcDdwWAvTYGZmDbQsEETETwDVWLxvq45rZmbj4y4mzMwGnAOBmdmAcyAwMxtwDgRmZgPOgcDMbMA5EJiZDTgHAjOzJlqyZO3RzEZG0vxu5UBgZtZEs2ePHdoyH/py9uzOpqueVj5ZbGY2cPKhLQ87DObNgzPPHDv0ZTdyiaCBXizmmVlnDQ+nILB4cXrt5iAADgQN9WIxz8w6a2QklQQWLkyvlReT3caBoIFiMW/RovTa7cU8M+uc/GJx+XI4+eTR/KObg4EDQQm9Vswzs85ZuXLsxWJ+MblyZWfTVY/SIGHdbWhoKFatWtWx4+cRvlcafsz61ZIlqVq2+P83MpIy2QULOpeubiXp2ogYarSeSwQN9GIxz6xfuc2uNRwIGujFYp5Zv3KbXWu4asjMes6iRanNbuHCVFK36lw1ZGZ9qdduzewFDgRm1jPcZtcaDgRm1jPcZtcabiMwM+tTbiMwM7NSGvY+KknAbsA22az7gJ9FLxQlzMysobqBQNIbgS8At5MCAMC2wMskfSAiftDi9JmZWYs1KhGcAbwhIu4qzpT058BlwCtalC6znufuEKxXNGojmALcW2X+fcB6zU+OWf9wdwjWKxqVCL4KrJS0DLgnm7cdcARwVisTZtbrenGkKhtMdUsEEXEK8HZAwB7ZJODt2bKaJH1V0oOSbi7MO0nSfZKuz6YDJ/8VzLqXuzC3XtDwrqGIuBW4VdIW2edHSu77bODzwLkV8z8TEaeNJ5FmvaqyO4ThYQcD6z51SwSSZkpaJulB4BrgZ9lV/jJJs+ptGxE/AsoGDbO+4+4QrFc0aiy+APg2sFVE7BgRLwO2Ai4Clk3wmMdKujGrOtp8gvswa6klS9bOsEdG0vyy3B2C9Yq6XUxIuj0idhzvssI6s4BLI+JV2ectgYeAABaTAsy7a2w7F5gLMHPmzF3vvvvuhl/GrFmKV/PDw2t/NusFzepi4lpJX5C0u6Sts2l3SV8Afj7eREXE6oh4LiKeB75MemK51rpfioihiBiaMWPGeA9lNikeAMUGSaPG4qOA9wCfZGwXExczgdtHJW0VEfdnHw8Bbq63vlknFe/4WbjQQcD6V91AEBHPAGdm07hI+iawNzBd0r3AJ4C9Je1Mqhq6C3j/ePdr1i6+48cGRaO+hqaQSgQHM7ZE8B3grIj4U61tI+LIKrP9EJr1hMo2geFhVw9Z/2rURnAesDOpaujAbPok8Grg/NYmzaxzfMePDZJGdw39KiJePt5lzeaBacpxJ2dmVtSsu4YekfQPkl5YT9I6kg4HHp1sIq253MmZmU1Eo0BwBPBWYLWkX0n6FfAAcGi2zLqIb3kcn2Y8NGbWDxp1OndXRBweETPIOp2LiJdk837TniTaeLiTs/JcgjJLSo9ZHBEPR8TDAJKGJG3dumTZRFXe8uh+bWpzCcosmejg9f8b+K6kC5qZGJscd3I2fi5BmU0wEETE0RGxC/DeJqfHJsG3PI6fS1BmDW4fBZA0DdifsQ+UfT8ift/itL3At49aK7hjOet3Tbl9VNJRwHWkriI2zqZhUmd0RzUhnWYd4xKUWdLogbJfArtXXv1n4whc4wfKzMy6V7MeKBOpg7hKz2fLzMysxzXqhvpTwHWSfgDck82bCexHGljGzMx6XKMHys4BhoCrgD9m0w+BoYg4u9WJs9bz07XdxX8P64SGt49GxKMRsSwilmbTsohwP0NdarwZiZ+u7S7+e1gnTPSBMiTd1MyE9Kt2XOEVj5FnJKefPjq/Xkbip2u7i/8e1hERUXMidS5Xbfp7YE29bZs57brrrtGrVqyImD49vVb73IpjLF0aIUXMmVP+WAsXRkB6tc7z38OaAVgVJfLYRoHgT8DZwNeqTE+UOUAzpl4OBBGjGfXChc0PArWOMWdO+YykHemz8vz3sGZpViC4FnhVjWX3lDlAM6ZeDwQR7bnCy4+RlwTKZCTtKLFYef57WDOVDQSN2giOAx6vseyQiVRFDaJ29GeTH2POHDj/fDjxxHIdz/np2u7iv4d1QsO+hrpBLz9Z3I7+bIr7XLkSpkyBU04Ze0wPV2k2eMo+WdzogTIkvQR4KiKekrQRMB/YBDgjIu6ffFL7W70rvGYFguIx8n3ussvoMYrzzcwqlel9dAXwzoj4raQlwAzgF8D+EdGW7KWXSwRmZp3SrN5HjwZ2APbO3h8OrCKNW7y9pKMk/VUzEmxmZp3RqGroh8BTwI3Ai4HVwCWkDuc+mC1/rHXJMzOzVqsbCCLibkmfA75P6nH0fVkV0Uzg4Yj4bTsSaWZmrdOwsTgizpR0HvB8RPwhm/0wcGRLU2ZmZm3RMBAARMSTFZ+fak1yzMys3Sbc6Vwjkr4q6UFJNxfmbSHpckm3Z6+bt+r4ZmW422ezFgYCUh9F+1fMOwG4MiJ2BK7MPpt1jLt9NmthIIiIHwGPVMx+C3BO9v4c4OBWHd+sDHf7bFYyEEg6NKvOeUzS45KekFSrD6J6tiw8jfwAsGWdY86VtErSqjVr1kzgUGblDA/DvHmweHF6dRCwQVO2RLAEeHNETIuITSNik4jYdDIHznrGq/lYc0R8KSKGImJoxowZkzmUWV3t6BTQrJuVDQSrI+K2JhxvtaStALLXB5uwT7MJK3bYV6a3VrN+VDYQrJJ0gaQjs2qiQyUdOoHjXQwcnb0/GvjOBPZh1jTu9tmsZDfUkr5WZXZExLvrbPNNYG9gOqlrik8AFwHLgZnA3cBhEVHZoLyWVnQ6t2RJujOkWB/s7pqtGp8r1qua1g01QES8a7wJiIhaTx7vO959tUJ+22C1cQLMinyuWL+rGwgkLYiIJVl/Q2sVHSLiQy1LWYsVbxucNy81Evq2QavG54r1u0YlgryBuC8HAyjeNrhwof+xrTafK9bPGvU+ekn2ek699XpV5W2DHsnLavG5Yv2slV1MdDXfNmhl+VyxfjewgcC3DVpZPles35W9ffSvI+Knjea1iscsNjMbv6aMWVzwuZLzzFrGXUabtUaj20f3AF4HzJA0v7BoU2DdVibMrJLv5zdrjUYlgvWBF5ECxiaF6XHgra1NmvWKdl2pu8tos9ZodPvoVcBVks6OiLvblCbrMe28Uvf9/GbNV7dEIOlfs7efl3Rx5dSG9FkXaHTF384rdXcZbdZ8jZ4sPjd7Pa3VCbHuVeaKvx1X6sXj5g90uXrIbPIatRF8Ons9MCKuqpxanbhO6MU7U1qd5jJX/O24Uvf9/GYtEhE1J+BW0l1DtwG7AK8pTvW2bea06667RrusWBExfXp6rfa5G7UrzQsXRkB6nezxTz117eUrVqT5ZtYcwKookcc2CgRvBb4HPAGMVEwryhygGVM7A0HEaEa2cGH3B4Fcq9Ncb/8TydR7MeCa9ZqmBIIXVoKFZdZr1dTOQJBnasWr33ZcqTbjCrnWFftkj1km055MMOilgGvWS5oaCNL+eDOp0fg04KCy2zVjanfV0KabRkybljKoadPS51ZnUtUy26lTI5YuXXu9apnrRDLVslfl9TL5fFlx2xUrIubOLZeO8QYvMyuv2SWCU4ArgXdn0+XAv5TZthlTuwNBnvkvXDgaFKplaM2u567MzJcurZ1RF4+dz1+6dHT+eIPBRK/KKwPAtGkRG21U+zdr5rHNrL5mB4IbgXUKn9cFbiyzbTOmbq0aakU9d+UVcq3MsjIoVAsaZQPSZK/Ki2ncaKNy+3IbgVnrtSIQbFH4vEW/BoKI8V2pNvOqtta+Gt2tM5ljNyv9eRo33rjcvnzXkFnrNTsQHAncDZwNnAP8Bji8zLbNmLr99tFm1HPXOm5+pV8rc53MsZt1VZ63q2y88Wh7iq/wzTqvaYEAELAdsFXWYPxm4M/K7LxZUyeqhorqXamuWJEywDlzJl41U+u4S5emfdfKqCd7Nd+Mq/I8DXPnVm809hW+Wec0u0RwU5n1WjW1u2qorMqr9srXyV4N18uou6WO3VU8Zt2rbCAoO0LZOcDnI6IjD/OPd4SyJUtS/ziVXSCsXAkLFjQvXcXj5P3gHHAAXHghXHJJa/u/WbIE7rwTjjhi9DgjI7BsGeywQ3O/p5n1pmaPULY7cLWkOyXdKOkmSTdOLomtk3eSlvd3k2fSs2c39zgLFozt92bePDjvPJg/v/WdoC1YkIJA5SDqF17Y/O9pZn2uTLEB2L7aVGbbZkwTqRpq5T3q9erzm9UFQ1nN/p7dXtXT7ekz6yY0qa+hDYHjgM8D7wemlNlps6eJthG06qnVuXNHH5g69dSIefMipIiDDkrLyzbyNis4NfN7dkvbQy3dnj6zbtKsQHABcH4WBC4Cziiz02ZPEwkEeWZdvFJuxpVj/vBW/sTxnDnpV1x33bF3zeRP+eZaVUJptN9+7AOo29Nn1i2aFQhuKryfAlxXZqcNDwp3ATcB15dJ6HgDQd7VQfGe9mb1GVTM6POnaCH1C9QoY2p05T6RW1cbXR1P9Aq62/sA6vb0mXWDZgWC6+p9nuiUBYLpZdcfbyCo7Agt7zNo7txx7aam/AGqKVPSL7jBBqMlg1oZU56WffddOyBN9JbQ8fYeWvYKutuvuLs9fWbdolmB4Dng8Wx6Ani28P7xMgeosd+WBoKiVlw5rliRMn+I2Guv1B4grf1QWXH9yo7Zaj2B26pMruzv0O118N2ePrNu0pRA0KqJ1EXFdcC1wNwa68wFVgGrZs6cOaEfoUymOpE69L/7u/TLzZmTMvSpU1MwyJ+ubXTXUB4M9t23+d1GVDOe4NLtd+V0e/rMukm3B4JtsteXADcAr6+3/mRuH2105TjeK8wVK8aOEzB3bgoGxcbhMhnTeDqSm0zm5ytos8HV1YFgTALgJOD4eutMJBCMJ/Ns9xVz5fFq9dOTD+5Sb1yCRnwFbTa4ujYQAFOBTQrv/wvYv942ze5rqFrm2Kixt1kqM/G5c1MJY9NNI3bbLWX6xecQli6NOOCA1lVzmVn/6uZA8NKsOugG4Bbg4422aXYgqMyMly6t39jbTNXaCzbdNLU7bLrpaCP00qVrpzOvTpozZ+3v000d0ZlZd+jaQDCRqRW9j+aZ5Jw5KQjkdf7NyjwbXZ1XG2pyp53ihdtRa41I1ii9E7nryCUJs/408IGgTObW6Ap7MhpdnVd+3nHHlJaddhp9UG3OnOpX+o1KMOO968glCbP+NPCBoGxG3MpBXRodI1++337pL7HeeqltYOrUiPXXH73yH0+bxkS/V6ueXzCzzhn4QBAx/rtzxpP5lb2KbnR1nmfo++03eqW//vopGBx0UP0AUm9A+3ppqsXdNpj1FweCTDFzyzPGPCDkzwDkASG/8i5bLVT2ir/e8qlTR6t45s4dDQz77ju6TrVSRrXMvhnPG7hEYNY/HAiieuZWnJf3IDqZ8XYbPRhWmWHXK5Xk6Wn3LaJuIzDrTwMfCOplbnnmve++qTpm2rTUgVz+xHDxQa5qd/nkJjIYTb7vyoCQpyPvGK+dmbHvGjLrTwMfCBplxMUSQV4dk9+6mXcZUe8qOa/Pn8htp9VKKs6MzazZHAjqXMEXM/e859C99hoNBnvt1bjOv9jfUHF52YzbDbNm1moDHwiqVQ1VZt4rVqQSwS67pF9i/fVHxxiofLYg14wM3A2zZtYOAx8IIhpnuPmwk9OmpYe48u6kX/OasdU+Zfc3njS5YdbMWq1sIFiHPjY8DPPmweLF6XV4eOzy2bPhlFPg8MNhn31SxdB668Fpp6Xp4x+H978/rTsyAgcdBCeeCCefDMuXw2GHwemnw5Il5dO0cmXaNk/L8HD6vHJlc76zmdl4Tel0AlppZATOPBMWLkyvw8Njg0ExU37/+1MQWLQozZ89GzbYYOy6ixenwLHLLmmbE09M619ySfk0LViw9rzKdJmZtVWZYkOnp1YOTFNtm3pVP67fN7NewaBXDa1cCYceOvo5r4JZtqx2VU6jqqSy65iZ9ZK+DQQLFsARR6R6/JGRlPn//Odw4YWp2gdG5+cqq5JGRtbeb5l1zMx6SV+3EeSlgMMOg+22g+uvT43Aw8MpAz/kkNRQvGQJTJmSMvd99knLN9ssNQ4vXgy//CXssEMKIIcdNtquMDw89rOZWS9SqkbqbkNDQ7Fq1aoJb79oUcrQ118fNtoIPvQh+Oxn011CF12U1nnTm+Coo+Ab34A//Sk1HL/tbfC1r6XtLrpotBG5mOmPjKT51RqBzcw6SdK1ETHUcL1+DwQjI+mqfd68lPk//TQ880wKCN/97mimnq93wAFw/vkpEEyZkqaLLvIVv5n1nrKBoG/bCGA0c1++PN37v2hRCgIA0th180bg886DPfdM6/3hD/DhD48NAkuWrN0uUNnWYGbWS/o6EBSfExgZScFg441h333Tlf4hh4xm6nkj8Jw58JOfpOqg9daDpUvHZvxTpqS2g+J2hx022gBtZtZr+joQLFgwejW/bFlqE7j0UrjiilTdE5Hm55n5iSfCxRenaqONNoL3vjdVJeUZ/8hIeqBs8eK0/qJF5RuLXZIws27V14GgaIcdxtb1f/rT8Pa3p/l5yWFkBDbdNAWLww+H555Ldxk9/zx86lNw8MHp2YT588f/LEF+x5FLEmbWdco8ddbpaTJDVdZSOZ5AvfEF8h5H8y6sPUC8mfUCSj5Z3NdmgfyPAAAMTUlEQVTPEdQzf356/chH4Oyz4eab09X//Pmjt4QuX57aEZ55JrUtTJmSHkpbuDCVBubPH32W4MQT4dln699GWnwqeeFC34lkZt2hr6uGqtXL7757emYAUka+005w000wdepoEChW2TzzTGon+MhHUtXSokVw9NGprWBkZGznc42qefxUspl1o74uEVQ+CTwykq78n346dR8NKQgAPPkkDA3B3XePrv+mN6UG5WLvpZdcMlpayJ9POPPMNL/eFX7xVlY/lWxmXaVM/VGnp8m0EVSrl1+6NF4YlhLS5113Te/zQ5UZkzhvO6gczazakJUek9jM2g2PUJaceuro4PT58JIrVowOSbnXXqOZ/MtfPpqxb7xx/TGJ823mzJn4IPaT5eBiZvV0dSAA9gd+CdwBnNBo/ckEgvzKfs6clEEvXZqGpITRYJBn+tOnR+y3X/Wr/KLKzL7yGO26G8jDXppZPV0bCIB1gTuBlwLrAzcAr6y3zWTHLM4z+bxkkFcHrVgRsd566XM+sH2+3tSptTPUalfilaWOdvEtqWZWSzcHgj2A7xc+nwicWG+biQaCYoad1+dvvXXEQQeNrlOcP9Gr605nxvl3aHcQMrPu1s2B4K3AVwqf5wCfr7LeXGAVsGrmzJmT+jFqZdTF+XmJoHK7RvXtna6e6XQQMrPu1fOBoDg1466hyow6rwaabAbeyQbbTgchM+tuZQNBJx4ouw/YrvB522xeSxR7IIXRUcuuuKL6/JUrx7f/Ysd2ueHh9gxUU+u7jfc7mNlg68QDZSuBHSX9OSkAHAG8rd2J2Hvv6hl4Lz3cVS3Y9Np3MLPOa3uJICKeBY4Fvg/cBiyPiFtadTz3+mlmVl9HupiIiMuAy9pxrOIA9nl3EO7WwcxsVF93OpdbuTKNRVwcP8CDwpiZJX3d6VxuypQ0IP2cOalEsNlmqffQ5cs7nTIzs87r+xJBPrzkaafB976XSgbHH5+6jp5o9ZCHnTSzftL3gSC/xTIfXvK88+Ad70iDyEyUG6DNrJ/0fSDI7/MvDgrzve9NLtMuNkCPZwB7M7Nu1PeBAMYOCnPyyaOZ+GRGCCsOO1l2AHszs240EIGgFU/gethJM+sXSt1RdLehoaFYtWpVp5PxgsphJys/m5l1A0nXRsRQo/UGokTQbO7jx8z6iUsEZmZ9yiUCMzMrpS8DgR/4MjMrry8DgR/4MjMrry/7GnKPo2Zm5fVliQD8wJeZWVl9Gwj8wJeZWTl9GQha0aWEmVm/6stA4Ae+zMzK8wNlZmZ9yg+UmZlZKQ4EZmYDzoHAzGzAORCYmQ04BwIzswHXE3cNSVoD3D3BzacDDzUxOa3m9LZer6XZ6W2tXksvlE/z9hExo9FKPREIJkPSqjK3T3ULp7f1ei3NTm9r9Vp6oflpdtWQmdmAcyAwMxtwgxAIvtTpBIyT09t6vZZmp7e1ei290OQ0930bgZmZ1TcIJQIzM6vDgcDMbMD1TSCQdJekmyRdL2mtrkqVfFbSHZJulPSaTqQzS8tfZOnMp8clHVexzt6SHiuss6jNafyqpAcl3VyYt4WkyyXdnr1uXmPbo7N1bpd0dIfT/GlJv8j+5t+WtFmNbeueP21M70mS7iv83Q+sse3+kn6Znc8ndDC9FxTSepek62ts24nfdztJI5JulXSLpA9n87vyPK6T3tafwxHRFxNwFzC9zvIDge8BAl4LXNPpNGfpWhd4gPTgR3H+3sClHUzX64HXADcX5i0BTsjenwCcWmW7LYBfZ6+bZ+8372Ca3whMyd6fWi3NZc6fNqb3JOD4EufMncBLgfWBG4BXdiK9FcuXAou66PfdCnhN9n4T4FfAK7v1PK6T3pafw31TIijhLcC5kVwNbCZpq04nCtgXuDMiJvrkdEtExI+ARypmvwU4J3t/DnBwlU3/Frg8Ih6JiEeBy4H9W5bQgmppjogfRMSz2cergW3bkZYyavzGZewG3BERv46IZ4BlpL9NS9VLryQBhwHfbHU6yoqI+yPiuuz9E8BtwDZ06XlcK73tOIf7KRAE8ANJ10qaW2X5NsA9hc/3ZvM67Qhq//PsIekGSd+T9JftTFQNW0bE/dn7B4Atq6zTrb8zwLtJpcJqGp0/7XRsVg3w1RrVFt34G+8FrI6I22ss7+jvK2kWsAtwDT1wHlekt6gl5/CU8Sawi+0ZEfdJeglwuaRfZFcwXUvS+sCbgROrLL6OVF30ZFZPfBGwYzvTV09EhKSeufdY0seBZ4Gv11ilW86fM4HFpH/qxaTqlnd3IB3jdST1SwMd+30lvQj4FnBcRDyeCi9JN57HlektzG/ZOdw3JYKIuC97fRD4Nqn4XHQfsF3h87bZvE46ALguIlZXLoiIxyPiyez9ZcB6kqa3O4EVVufVadnrg1XW6brfWdI7gYOAt0dWmVqpxPnTFhGxOiKei4jngS/XSEdX/caSpgCHAhfUWqdTv6+k9UiZ6tcj4sJsdteexzXS2/JzuC8CgaSpkjbJ35MaV26uWO1i4CglrwUeKxQPO6XmVZSkP8vqXZG0G+lv9XAb01bNxUB+98TRwHeqrPN94I2SNs+qNd6YzesISfsDC4A3R8QfaqxT5vxpi4p2q0NqpGMlsKOkP89KlUeQ/jad8gbgFxFxb7WFnfp9s/+fs4DbIuL0wqKuPI9rpbct53ArW8HbNZHunrghm24BPp7NPwY4Jnsv4N9Id1vcBAx1OM1TSRn7tMK8YnqPzb7LDaQGote1OX3fBO4H/kSqH30P8GLgSuB24Apgi2zdIeArhW3fDdyRTe/qcJrvINX1Xp9NX8zW3Rq4rN7506H0npednzeSMqytKtObfT6QdFfJnZ1Mbzb/7Py8LazbDb/vnqQqthsLf/8Du/U8rpPelp/D7mLCzGzA9UXVkJmZTZwDgZnZgHMgMDMbcA4EZmYDzoHAzGzAORBYR0h6Lusl8WZJ/y5p4ybv/52SPt9gnb0lva7w+RhJRzXh2FtL+o9xbnOsUk+iUXxwMHvupWGvuarRG2n2rME12fwLsucOkLRB9vmObPmsiX1b6wcOBNYpT0fEzhHxKuAZ0jMU7bY38EIgiIgvRsS5k91pRPwuIt46zs1+Snowq7LzwQNIXYvsCMwldUExhqR1Sc/IHEDqrfJISa/MFp8KfCYiXgY8SnpWgez10Wz+Z7L1bEA5EFg3+DHwMgBJ87NSws3KxmiQNEupP/avS7pN0n/kJQilPtinZ++HJP2wcueS3pRd9f5c0hWStsyugI8B/jErmeylNBbA8dk2O0u6WqN9wG+ezf+hpFMl/UzSryTtVeV4s5T12Z+VTC6U9J9K/dovqfYDRMTPI+KuKovK9JpbtTfS7EnVfYC8dFLsabPYA+d/APvmT7Lb4HEgsI5S6qfmAOAmSbsC7wJ2J40Z8T5Ju2Sr/gXwhYh4BfA48IFxHOYnwGsjYhdSJrkgy3S/SLpa3jkiflyxzbnARyPir0hP+n6isGxKROwGHFcxv5adgcOBnYDDJW3XYP2iMr1g1lrnxcDvY7QL4+K2L2yTLX8sW98GkAOBdcpGSqNZrQJ+S+pjZU/g2xHxVKQO9y4kdW8McE9E/DR7f362blnbAt+XdBPwT0DdLr0lTQM2i4irslnnkAZlyeWdgV0LzCpx/Csj4rGI+B/gVmD7caTdrOX6qRtq6y1PR8TOxRkNaiYq+0LJPz/L6AXNhjW2/RxwekRcLGlv0ihgk/HH7PU5yv0P/bHwvuw2uTK9YNZa52FSVdKU7Kq/uG2+zb1ZqWwane/U0DrEJQLrJj8GDpa0cdaD4iHZPICZkvbI3r+NVN0DaXi+XbP3f19jv9MYzQCLY88+QRoScIyIeAx4tFD/Pwe4qnK9NqnZa66kX2TrVO2NNFJHYiNA3nBd7Gmz2APnW4EV4Y7HBpYDgXWNSMP0nQ38jDQy01ci4ufZ4l8CH5R0G2kM2fzumU8CZygN1v1cjV2fBPy7pGuBhwrzLwEOyRuLK7Y5Gvi0pBtJdfwnT+a7NSLpQ5LuJV213yjpK9miy0jj5d5BGp/gA9n600k96uZ1/MeSukm+DVgeEbdk238UmC/pDlIbwFnZ/LOAF2fz55PG7rUB5d5Hretld/hcmt1qaoCkg4CXRsRnO50W631uIzDrQRFxaafTYP3DJQIzswHnNgIzswHnQGBmNuAcCMzMBpwDgZnZgHMgMDMbcP8fLM+P7w87bZMAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(data['Population'], data['Profit'], 'bx')\n", "plt.xlabel('Population in 10,000')\n", "plt.ylabel('Profit in $10,000')\n", "plt.title('Relation between profit and population')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The hypotheses function\n", "Our hypothesis function has the general form:\n", "$y= h_\\theta(x)= \\theta_0 + \\theta_1x$\n", "Note that this is like the equation of a straight line. We give to $h_\\theta(x)$ values for $\\theta_0$ and $\\theta_1$ to get our estimated output y. In other words, we are trying to create a function called $h_\\theta$ that is trying to map our input data (the x's) to our output data (the y's).\n", "\n", "### Cost function\n", "\n", "The cost functions yields \"how far off\" our hypotheses $h_\\theta$ is. It takes the avarage of the distance between our hypothesis and the actual point and squares it. Formally, the cost function has the following definition:\n", "\n", "$J(\\theta) = \\frac{1}{2m} \\displaystyle\\sum_{i = 0}^{m}(h_θ(x^{(i)}) - y^{(i)})^2$\n", "\n", "#### Vectorization\n", "Vectorizations is the act of replacing the loops in a computer program with matrix operations. If you have a good linear algebra library (like numpy), the library will optimize the code automatically for the computer the code runs on. Mathematically, the 'regular' function should mean the same as the vectorized function.\n", "\n", "Gradient descent vectorized:\n", "$\\theta = \\frac{1}{2m}(X\\theta - \\vec{y})^T(X\\theta-\\vec{y})$\n", "\n", "**Exercise**: Implement a vectorized implementation of the cost function." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def cost_function(X, y, theta):\n", " \"\"\" Computes the cost of using theta as the parameter for linear gression to fit the data in X and y. \"\"\"\n", " \n", " return 0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With $\\theta = \\begin{bmatrix}0 & 0\\end{bmatrix}$, $J(\\theta)$ should return 32.07." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\n" ] } ], "source": [ "initial_theta = np.zeros((2,1))\n", "print(cost_function(X, y, initial_theta))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Gradient descent\n", "We want are hypothesis $h_\\theta(x)$ to function as good as possibly. Therefore, we want to minimalize the cost function $J(\\theta)$. Gradient descent is an algorithm used to do that. \n", "\n", "The formal definition of gradient descent:\n", "\n", "$repeat \\ \\{ \\\\ \\enspace \\theta_j := \\theta_j - \\alpha \\frac{1}{m}\\displaystyle\\sum_{i = 1}^{m}(h_\\theta(x^{(i)})-y^{(i)})x_j^{(i)}\\\\\\}$\n", "\n", "An illustration of gradient descent on a single variable:\n", "
\n", " \n", "
\n", "\n", "**Exercise**: Implement the gradient descent algorithm in Python." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def gradient_descent(X, y, theta, alpha, iterations):\n", " \"\"\" Performs gradient descent to learn theta. \n", " Returns the found value for theta and the history of the cost function.\n", " \"\"\"\n", " J_history = []\n", " return theta, J_history" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Gradient descent should have found approximately the following: $\\theta = \\begin{bmatrix}-3.6303\\\\1.1664\\end{bmatrix}$" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "array([[0.],\n", " [0.]])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# You can change different values for these variables\n", "alpha = 0.01\n", "iterations = 1500\n", "\n", "theta, J_history = gradient_descent(X, y, initial_theta, alpha, iterations)\n", "theta" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using the results\n", "#### Plotting the regularization line" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAG2FJREFUeJzt3X+MHOV9x/HPtzk7sk1soHapS0jdUBKJCimGM01bXGVxSIE6DaDq4vy40Cbppaei5kejEy46C2FVlAvnqlErV6TQEEjjWIpLICFKgFspitpGd6YGnJIUaEEFOXD5oZIolRLg2z+eme7c3v6Y3duZnZl9v6TV7s7O7j63N/e5Z595fpi7CwBQfj837AIAAAaDQAeAiiDQAaAiCHQAqAgCHQAqgkAHgIog0AGgIgh0AKgIAh0AKmKs2w5mdo6kz0g6S5JLus3d/9rMbpT0R5KWo13/3N3v7/RaW7du9R07dqypwAAwao4fP/49d9/Wbb+ugS7pJUl/5u4Pm9lrJB03sweix/7K3W9NW6gdO3ZoaWkp7e4AAElm9kya/boGurufknQquv0jM3tc0tlrKx4AYNB6akM3sx2Sdkr6ZrTpOjN71MzuMLMzBlw2AEAPUge6mZ0m6QuSPuLuL0o6LOlcSW9SqMHPt3nelJktmdnS8vJyq10AAAOQKtDNbJ1CmH/W3Y9Jkrs/7+4vu/srkj4l6eJWz3X329x93N3Ht23r2qYPAOhT10A3M5N0u6TH3f1QYvv2xG5XSzo5+OIBANJKU0P/LUmTki41sxPR5UpJc2b2mJk9Kqkm6aNZFhQASmduTqrXV26r18P2DKTp5fINSdbioY59zgFg5O3aJU1MSEePSrVaCPP4fgbS9EMHAPSjVgvhPTEhTU9Lhw83wj0DDP0HgCzVaiHMDx4M1xmFuUSgA0C26vVQM5+dDdfNbeoDRKADQFaSbeY33dRofsko1Al0AMjK4uLKNvO4TX1xMZO3M3fP5IVbGR8fdybnAoDemNlxdx/vth81dACoCAIdACqCQAeAiiDQAeQv5yHxo4JAB5C/eEh8HOpx975du4ZbrpJj6D+A/OU8JH5UUEMHMBw5DokfFQQ6gOHIcUj8qCDQAeQv5yHxo4JAB5C/nIfEjwqG/gOolrm50Fsm2SZfr4d/FjMzwyvXGjD0H8BoGuEukXRbBFAtI9wlkho6gOoZ0S6RBDqA6hnRLpE0uQColrjN/JprQs28Vmt0kZRKfXK0G2roAKol7hK5b18IcincP3Kk8idHqaEDqJZk7Tt5cvTYscqfHKWGDqC6RuzkKIEOoLpG7OQogQ6gmkZwvhgCHUA1jeB8MV3ncjGzcyR9RtJZklzSbe7+12Z2pqTPS9oh6WlJE+7+w06vxVwuANC7Qc7l8pKkP3P38yW9WdKfmNn5kq6X9JC7nyfpoeg+AGBIuga6u59y94ej2z+S9LiksyW9Q9Kd0W53Sroqq0ICALrrqQ3dzHZI2inpm5LOcvdT0UPfVWiSAQAMSepAN7PTJH1B0kfc/cXkYx4a4ls2xpvZlJktmdnS8vLymgoLAGgvVaCb2TqFMP+sux+LNj9vZtujx7dLeqHVc939Nncfd/fxbdu2DaLMAIAWuga6mZmk2yU97u6HEg/dK+na6Pa1kr44+OIBANJKM5fLb0malPSYmZ2Itv25pL+UdNTMPiDpGUkT2RQRAJBG10B3929IsjYP7xlscQAA/WKkKABUBIEOABVBoANARRDoAFARBDoAVASBDgAVQaADQEUQ6ADQytzc6tWN6vWwvaAIdABoZdeulUvWxUva7do13HJ1kGboPwCMnnjJuokJaXo6LDKdXNKugEanhl7Cr08AhqxWC2F+8GC4LnCYS6MU6CX8+gRgyOr1UDOfnQ3XzZXCghmdQE9+fTpwIFwX/OsTgCGKK31Hj0o33dTIjwKH+ugEulS6r08AhmhxcWWlL64ULi4Ot1wdWFg9Lh/j4+O+tLSU2/utEv/HLckJDqCy5uZCc2fy769eD2E5MzO8chWUmR139/Fu+41ODb2EX5+AyuKcViZGJ9BL+PUJqCzOaWVitJpcABTLgQPhnNbsbPjmjJZocgFQbCXrElgGBDqA/HFOKxMEOoD8cU4rE7ShA0DB0YYOACOGQAeAiiDQAaAiCHSgG6ZeRkkQ6EA3DFNHSbBiEdBNCVeuwWjqWkM3szvM7AUzO5nYdqOZPWdmJ6LLldkWExgypl5GCaRpcvm0pMtbbP8rd39TdLl/sMUCCoZh6iiBroHu7l+X9IMcygIUE8PUURJrOSl6nZk9GjXJnDGwEgGDNIgeKgxTR0n0G+iHJZ0r6U2STkmab7ejmU2Z2ZKZLS0vL/f5dkCfBtFDZWZmdZt5rcbKOiicvgLd3Z9395fd/RVJn5J0cYd9b3P3cXcf37ZtW7/lBPrDQgoYIX0FupltT9y9WtLJdvsCQ0cPFYyINN0WPyfpXyS90cyeNbMPSJozs8fM7FFJNUkfzbicQP/ooYIR0XVgkbu/q8Xm2zMoCzB4yR4qtVq40OyCimLoP6qNHioYISxwUURzc6EXRrIGWa+HEKJnBTByWOCizJgMCkAfCPQioqtdb5jeFpBEoBcXXe3S4xsNIIlALy662qXHNxpAEoFeTEwG1Tu+0QAEeiHR1a53fKMB6LaICmgePNR8Hyg5ui1idPCNBpBEDR0ACo8aOgCMGAIdACqCQK8KRksWC78PDAGBXlS9BgKjJYuF3weGgEBPK48aV/I94kA4dKixvVMgMFqyWPh9YBjcPbfLRRdd5KW1sOC+dWu4bnU/i/eYn3c3c5+cTP9es7PuUrjG8PH7wABIWvIUGUug9yIO3NnZwYd5u/eYnEwfCHmUD+nx+8CAEOhZyaPGFb9HXDNPEwh5fINAevw+MEBpA5029F7kMV9I/B6Tk9Ldd0v796eboIvRksXC7wNDwEjRtPKYLyT5mouL0tiYdPPNK9+TZeiAkZN2pOhYHoWphE41rkEFevI94tfcubPxHsntANCEGjoAFBxzuQDAiCHQAaAiCHQAqAgCHQAqgkAHgIroGuhmdoeZvWBmJxPbzjSzB8zsiej6jGyLCXTBdLVAqhr6pyVd3rTtekkPuft5kh6K7gPDw3S1QPdAd/evS/pB0+Z3SLozun2npKsGXC6gN0xXC/Tdhn6Wu5+Kbn9X0lntdjSzKTNbMrOl5eXlPt8OSKFWk6anpYMHwzVhjhGz5pOi0UxgbYebuvtt7j7u7uPbtm1b69sB7eUxeRpQYP0G+vNmtl2SousXBlckoA/Jic3SzE4JVFC/gX6vpGuj29dK+uJgigP0ielqgVTdFj8n6V8kvdHMnjWzD0j6S0mXmdkTkt4a3R8OuqtBClMKN7eZ12orpxrmWEHFpenl8i533+7u69z9te5+u7t/3933uPt57v5Wd2/uBZMfuqshLY4VVFz550NPdlebng4nw+iuhlY4VlBx1Rj6T3c1pMWxggqrRqDTXQ1pcaygwsof6HRXQ1ocK6i48gc63dWQFscKKo41RQGg4FhTFMVFf3AgEwQ68kd/cCATBDoa8qo5M9UtkAkCHQ151pzpDw4MHIE+SrrVwPOsOdMfHBg4An2UpKmB51Fzpj84kInqBnoZe1JkXeY0NfA8as70Bwey4e65XS666CLPzcKC+9at4brV/SLKq8yzs+5SuF7r+99yy+rHFxbCdgADIWnJU2RsdQPdvRFIs7PFD/NY1mXu9Pr9hHMZ/3ECJUOgx+GUrI3mUXMcRI21XQ16re+ZJnzXEupl+scJlAiBvrDgvnmz+5YtIWi2bAn3sw6bVqG5aZP7/Pzq/VqFZD/hmLaW3Cms48eSz11YcJ+aSleOXv8JAUiNQF9YaIT47Gwj3FsF06DbgZtDeX6+feAm3zvePj/f2N5rqPdbS24O8i1b3DdsaP+ZDfK9AXREoPfS5JJFO3BzjbVd6DWHe6vwT/uPZa215GQZN2xI91q0oQOZI9Dde6s5DrKW2e61uvUuWct7D6r8cRk3bkz3WvRyATJHoPdTcxxEO3C7941r3u1Cci3vPahacnzeYePGxvkGatzA0BHovdYcFxZCkE1O9t/k0e595+fDa7cL3LXWrgdRS47LMDXV+uQoNW5gaAj0XjTXopuv11o77RS4RWmDpukEKKy0gV7sFYvm5sI8I81D0xcXpZmZwRUs+T7xPCNXXCEdOybdd1+2MwHOzUlPPSXt29d4n3pdOnJEOvfcwf6cAEqpGisW5TWd68zMynlFpqelu+6SPvax7Kd1nZkJYd48OdWxYyz4AKAnxQ70rKdzbTUZ1qFD0vx868mpspo8K4ufs+iTkxW9fEAZpWmXGdSl7zb0rEYhTk01Bs7ccov79LS7mfveveHxtCczB9XePcifsyht8+0UvXxAgagyJ0Xj0E32ABnEybp4EE88gnRyMnwcr3rVyl4e8ajNWFajIru9bhXnWCl6+YCCyCXQJT0t6TFJJ9K8Yc+Bnhy+nxySPog5WZKBHY+KlMK8K90CpltNup8uk91qq/3WaIs+x0rRywcUQJ6BvjXt/j0HevOEUfGcLFNT/Xwmq8UDacbGwkfx6lc3aurtAiYuy549q/+x9NsVsdfZEtPWaIteAy56+YCCqEagJ2VRk1tYCCEuue/eHdrLzVYPLkru3zyBVbsRlVmFVdrPoeht1EUvH1AgeQX6f0l6WNJxSVNt9pmStCRp6XWve11/P02acOynjfl3fzd8BJOTIZg3bQqhHo+W7DZXeBzqe/YMfjh/K738kyj6QKGilw8okLwC/ezo+hckPSLptzvt31cNPW1Nrtca38LCynnKp6ZCqCdPgqYJmF4m3FpLiFGjBUZW7r1cJN0o6eOd9ukr0HsJwbxrsM3v124elHiRiE7zondDjRYYWZkHuqRNkl6TuP3Pki7v9JyBz+XSKuS6ndQclOYwnpoKNf7Nm90vvjiEd7If+/y8+xVXZNd8BKCy8gj010fNLI9I+pakG7o9Z+CB3hyq8/OdT2oOUqv29M2bQ7v85s2Nk63z86vLGTfTTE6u/nmKNGEXgEKozsCibuKwm5wMYR63iQ8qBLvVllstIXfBBf7/3SDbrVDUrbz99JKhZg9UUnUCPU1IdavxrkW32nLz/fPOC2W54ILGgKXJydY1727fKHrtJUPNHqik6gR62kDNcnGIbu8RP37ZZeEjXbcutJ1v2uS+fn2jJt5Lm3+/P1dW/d8BDE11At29994kvYRY2lptt9pyHMyXXdaoea9fH0J9797O/wg6LRzdqUztMJweqJRqBbr7ypCKAy4O9rgPeRzscU04bXNL2hp4p8c3bWo0nUxNNQJ+z57GPq1q/a1CexD91amhA5VRrUBvFVLJbfGMiWtZD7PbAKHm4O30LSEuT95dE2lDByqpOoHeKaTiEN6zJzRzbNkSJtqKR4AmB/S06pUSi/uLtwrgdsEbv3ZzsMfliCcQyzNU6eUCVFJ1Ar1boCZr6HEzR9xlMB7K36nWGrd399PdsdU3B0IVwIBVO9BbrSQUz5S4e3cj1Hfv7t4mnpzPJfl42gDmBCSAjFUn0Fs1uTSH8MJCqKHv3Bl+pPXrG3OcN/dNjw0iiDkBCSAH1Ql09+7BGS8nt2VLGMwTT4N74YUrm1PSvl4vZeIEJICMpQ30n1vjGtP5qNWk6Wnp4MFwXautfHzXLunmm6V3vlO69NLQ4LJunXTrreFyww3Shz4U9q3Xpb17pf37pZtuko4elSYmpEOHeltxfnExPDcuS60W7i8uDuZnBoAejQ27AKnU69Lhw9LsbLiu1VaGejJcP/ShEOYHDoTtu3ZJr371yn0PHgz/AHbuDM/Zvz/sf9996cs0M7N6W3O5ACBPaarxg7pkusBFq+d0alKh/RtASagyTS6Li9I11zTux00bR460byLp1kSTdh8AKJHiB/rMjLRvX2jnrtdDiP/bv0nHjoXmFKmxPdbcRFOvr37dNPsAQImUow09rpVPTEjnnCOdOBFOdtZqIYivvjqcEJ2bk8bGQkhfeml4/PTTw0nQgwel73xHOvfc8I9gYqLR7l6rrbwPACVkoXkmH+Pj4760tNT/Cxw4EIJ5/XppwwbpT/9U+uQnQ6+We+4J+7z97dL73if94z9KP/tZOEH67ndL//AP4Xn33NM4WZoM73o9bG91shMAhsjMjrv7eNf9ShPo9XqoRU9PhxD/3/+VfvrTEOxf/nIjnOP9rrhCuvvuEOhjY+Fyzz3UwAGUTtpAL34butQI6aNHQ9/xAwdCmEuS2cp945Odd90lXXJJ2O8nP5E+/OGVYT43t7rdvLktHgBKpByBnuxnXq+HUN+4UdqzJ9S8r766Ec7xyc7JSekb3wjNLOvWSfPzKwN8bCy0rSefNzHRONEKACVTjkCfmWnUro8cCW3mX/qS9OCDoRnFPWyPQ3n/funee0NzzIYN0gc/GJpo4gCv18PAooMHw/4HDqQ/KUrNHkBBlSPQk849d2Vb+Cc+Ib3nPWF7XJOv16XNm0Pov/Od0ssvh14xr7wi/cVfSFddFfq2f+xjvfdFj3vIULMHUDRpRh8N6rKmJejaaZ7PvNP85vEMi/HUuyzEDKAEVKnZFruZnw8/ygUXrA7zeD71eCbGjRsbC19s3Lg6+JOrG3XCPOgAclKtQG+1yMXFF7vv3du4f8EF4cc57bRwv3mtzw0bGgEcz6k+Pb169aJNm7rXuKmhA8hR2kAvx0jR5pGd9bp08mQ40XnoUNjnscfC9Y9/LI2PS88809j/7W8PJ06TszXed1+jzT3u3374cNjeqS092YWSUaYAiiRN6g/qsqYml1a14ripJb7Mz7tfdFG4Hb9XmjVD4+aT5tWNWi1Fx5qhAHKmSjW5uIfAjBeBjtutFxYaS83t3t0I6ze8oRHQyXbyWDKA4+dMTva/WPRa8U8CQAe5BLqkyyV9R9KTkq7vtv+aAj2uaU9ONk5ebtoUfoQ41OPw3rrV/bLLWte6k5pDu/k98mobZzk7AB1kHuiSXiXpKUmvl7Re0iOSzu/0nDWvKRqHdVxTj5tZFhbc160L9+MFpOP9Op3kbFUzbv4WkBdOtAJoI22g9z05l5n9hqQb3f13ovv7ozb5m9s9p9/JuXZc/+W+yggARXHJr27V3R/89b6em8fkXGdL+u/E/Wejbc0FmTKzJTNbWl5eXsPbAQA6SlONb3WR9PuS/j5xf1LS33R6zkB6uTS3M8fNK2ttfx7miUna0AF0oBzWFH1O0jmJ+6+NtmUjOeOi1FjF6MEHW29fXOzt9ZMTgMVqtXwWvGj3s/X6MwAYaWsZWLQo6Twz+xWFIN8n6d0DKVUv3vKW1kFcpkE+rf5plO1nADB0fdfQ3f0lSddJ+qqkxyUddfdvDapgqzDLIQB0tKah/+5+v6T7B1SWzpILRcfD9BluDwD/r1zzoS8uhrVCk/OXs7gEAEhaYw09d2NjYeHnyclQQz/99LDy0NGjwy4ZAAxdeWro8bJxt94qfeUroab+8Y+H5eb6bXZhOTkAFVKeQI+79sXLxt11l/Te90ovvdT/a3KiFUCFlCfQ437i9XpobpmdDTX1tYRv8kRrLwtFA0ABlSfQpZWLS9x0UyOMm5tNelGr9b5QNAAUULkCPYsRlcka/+HDa/vnAABD1Pdsi/3od7bFzDQvJ9d8HwAKII/ZFsuPOVQAVMho19ABoASooQPAiCl2oDPwBwBSK3agM/AHAFIr9lwuzLAIAKkVu4YuMfAHAFIqfqAz8AcAUil2oGcx1B8AKqrYgc7AHwBIjYFFAFBwDCwCgBFDoANARRDoAFARBDoAVASBDgAVkWsvFzNblvRMn0/fKul7AyxO1ihv9spWZsqbrbKVV0pf5l92923ddso10NfCzJbSdNspCsqbvbKVmfJmq2zllQZfZppcAKAiCHQAqIgyBfptwy5Ajyhv9spWZsqbrbKVVxpwmUvThg4A6KxMNXQAQAeFC3Qze9rMHjOzE2a2aiYvCz5pZk+a2aNmduEwyhmV5Y1ROePLi2b2kaZ93mJm/5PY50DOZbzDzF4ws5OJbWea2QNm9kR0fUab514b7fOEmV075DJ/wsy+Hf3O/8nMTm/z3I7HT47lvdHMnkv83q9s89zLzew70fF8/RDL+/lEWZ82sxNtnjuMz/ccM6ub2b+b2bfM7MPR9kIexx3Km/0x7O6Fukh6WtLWDo9fKekrkkzSmyV9c9hljsr1KknfVegvmtz+FklfGmK5flvShZJOJrbNSbo+un29pFtaPO9MSf8ZXZ8R3T5jiGV+m6Sx6PYtrcqc5vjJsbw3Svp4imPmKUmvl7Re0iOSzh9GeZsen5d0oECf73ZJF0a3XyPpPySdX9TjuEN5Mz+GC1dDT+Edkj7jwb9KOt3Mtg+7UJL2SHrK3fsdOJUJd/+6pB80bX6HpDuj23dKuqrFU39H0gPu/gN3/6GkByRdnllBE1qV2d2/5u4vRXf/VdJr8yhLGm0+4zQulvSku/+nu/9U0hGF302mOpXXzEzShKTPZV2OtNz9lLs/HN3+kaTHJZ2tgh7H7cqbxzFcxEB3SV8zs+NmNtXi8bMl/Xfi/rPRtmHbp/Z/BL9hZo+Y2VfM7NfyLFQbZ7n7qej2dyWd1WKfon7OkvR+hW9prXQ7fvJ0XfT1+o42zQFF/Ix3S3re3Z9o8/hQP18z2yFpp6RvqgTHcVN5kzI5hsd6LWAOLnH358zsFyQ9YGbfjmoUhWVm6yX9nqT9LR5+WKEZ5sdRO+o9ks7Ls3yduLubWWm6OpnZDZJekvTZNrsU5fg5LOmgwh/nQYVmjPcPoRy9epc6186H9vma2WmSviDpI+7+YvgyERTxOG4ub2J7Zsdw4Wro7v5cdP2CpH9S+Fqa9JykcxL3XxttG6YrJD3s7s83P+DuL7r7j6Pb90taZ2Zb8y5gk+fjZqro+oUW+xTuczazP5C0V9J7PGpsbJbi+MmFuz/v7i+7+yuSPtWmHIX6jM1sTNI1kj7fbp9hfb5mtk4hHD/r7seizYU9jtuUN/NjuFCBbmabzOw18W2Fkwgnm3a7V9L7LHizpP9JfO0alra1GjP7xahdUmZ2scJn/v0cy9bKvZLis/3XSvpii32+KultZnZG1FzwtmjbUJjZ5ZJmJP2eu/+kzT5pjp9cNJ3XubpNORYlnWdmvxJ9y9un8LsZlrdK+ra7P9vqwWF9vtHfz+2SHnf3Q4mHCnkctytvLsdwlmd7+zg7/HqFM/2PSPqWpBui7X8s6Y+j2ybpbxV6BzwmaXzIZd6kENBbEtuS5b0u+lkeUTgR8ps5l+9zkk5J+plC++EHJP28pIckPSHpQUlnRvuOS/r7xHPfL+nJ6PKHQy7zkwptoSeiy99F+/6SpPs7HT9DKu9d0fH5qELwbG8ub3T/SoVeEE8Ns7zR9k/Hx21i3yJ8vpcoNF09mvj9X1nU47hDeTM/hhkpCgAVUagmFwBA/wh0AKgIAh0AKoJAB4CKINABoCIIdACoCAIdACqCQAeAivg/vMyNKB4jqBIAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(X[:,1], y, 'rx', label='Training data')\n", "plt.plot(X[:,1], X.dot(theta), label='Linear regression')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Plotting the cost history\n", "A plot of how $J(\\theta)$ decreases over time. This is are model learning." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAADqFJREFUeJzt23+o3fV9x/Hnq7k0axE00WitMbu2CiNu0MJBKdvA1V9x0EZa/7D7o2FryR+rf6yl0BTHtOof6tZZSruN0BZCYdXOURqQItFWGGNYT6yjzdo0t7HFpLZNjQhOqmR974/7dTufy4k3ud9z78nR5wMO93y/38+99/3xgs97zvcmVYUkSa9607QHkCSdWQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ15qY9wEqcd955NT8/P+0xJGmm7N+//9dVtWm5dTMZhvn5eYbD4bTHkKSZkuRnp7LOt5IkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpMZEwJNmW5GCShSS7xlxfn+SB7vrjSeaXXN+S5MUkn5zEPJKklesdhiTrgC8CNwBbgQ8l2bpk2UeA56vqUuA+4J4l1/8e+FbfWSRJ/U3iFcMVwEJVHa6qV4D7ge1L1mwH9nTPHwSuThKAJDcCTwMHJjCLJKmnSYThIuCZkeMj3bmxa6rqBPACcG6Ss4BPAZ+ZwBySpAmY9s3n24H7qurF5RYm2ZlkmGR47Nix1Z9Mkt6g5ibwNY4CF48cb+7OjVtzJMkccDbwHHAlcFOSe4FzgN8m+U1VfWHpN6mq3cBugMFgUBOYW5I0xiTC8ARwWZJLWAzAzcCfLVmzF9gB/AdwE/Dtqirgj19dkOR24MVxUZAkrZ3eYaiqE0luAR4G1gFfqaoDSe4AhlW1F/gy8NUkC8BxFuMhSToDZfEX99kyGAxqOBxOewxJmilJ9lfVYLl10775LEk6wxgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpMZEwJNmW5GCShSS7xlxfn+SB7vrjSea789cm2Z/k+93H905iHknSyvUOQ5J1wBeBG4CtwIeSbF2y7CPA81V1KXAfcE93/tfA+6rqD4AdwFf7ziNJ6mcSrxiuABaq6nBVvQLcD2xfsmY7sKd7/iBwdZJU1feq6ufd+QPAW5Ksn8BMkqQVmkQYLgKeGTk+0p0bu6aqTgAvAOcuWfNB4MmqenkCM0mSVmhu2gMAJLmcxbeXrnuNNTuBnQBbtmxZo8kk6Y1nEq8YjgIXjxxv7s6NXZNkDjgbeK473gx8A/hwVf3kZN+kqnZX1aCqBps2bZrA2JKkcSYRhieAy5JckuTNwM3A3iVr9rJ4cxngJuDbVVVJzgEeAnZV1b9PYBZJUk+9w9DdM7gFeBj4IfD1qjqQ5I4k7++WfRk4N8kC8Ang1T9pvQW4FPibJE91j/P7ziRJWrlU1bRnOG2DwaCGw+G0x5CkmZJkf1UNllvnv3yWJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIaEwlDkm1JDiZZSLJrzPX1SR7orj+eZH7k2qe78weTXD+JeSRJK9c7DEnWAV8EbgC2Ah9KsnXJso8Az1fVpcB9wD3d524FbgYuB7YB/9B9PUnSlEziFcMVwEJVHa6qV4D7ge1L1mwH9nTPHwSuTpLu/P1V9XJVPQ0sdF9PkjQlkwjDRcAzI8dHunNj11TVCeAF4NxT/FxJ0hqamZvPSXYmGSYZHjt2bNrjSNLr1iTCcBS4eOR4c3du7Jokc8DZwHOn+LkAVNXuqhpU1WDTpk0TGFuSNM4kwvAEcFmSS5K8mcWbyXuXrNkL7Oie3wR8u6qqO39z91dLlwCXAd+dwEySpBWa6/sFqupEkluAh4F1wFeq6kCSO4BhVe0Fvgx8NckCcJzFeNCt+zrwX8AJ4GNV9T99Z5IkrVwWf3GfLYPBoIbD4bTHkKSZkmR/VQ2WWzczN58lSWvDMEiSGoZBktQwDJKkhmGQJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpYRgkSY1eYUiyMcm+JIe6jxtOsm5Ht+ZQkh3dubcmeSjJj5IcSHJ3n1kkSZPR9xXDLuDRqroMeLQ7biTZCNwGXAlcAdw2EpC/q6rfA94N/GGSG3rOI0nqqW8YtgN7uud7gBvHrLke2FdVx6vqeWAfsK2qXqqq7wBU1SvAk8DmnvNIknrqG4YLqurZ7vkvgAvGrLkIeGbk+Eh37v8kOQd4H4uvOiRJUzS33IIkjwBvG3Pp1tGDqqokdboDJJkDvgZ8vqoOv8a6ncBOgC1btpzut5EknaJlw1BV15zsWpJfJrmwqp5NciHwqzHLjgJXjRxvBh4bOd4NHKqqzy0zx+5uLYPB4LQDJEk6NX3fStoL7Oie7wC+OWbNw8B1STZ0N52v686R5C7gbOCves4hSZqQvmG4G7g2ySHgmu6YJIMkXwKoquPAncAT3eOOqjqeZDOLb0dtBZ5M8lSSj/acR5LUU6pm712ZwWBQw+Fw2mNI0kxJsr+qBsut818+S5IahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGoZBktQwDJKkhmGQJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJjV5hSLIxyb4kh7qPG06ybke35lCSHWOu703ygz6zSJImo+8rhl3Ao1V1GfBod9xIshG4DbgSuAK4bTQgST4AvNhzDknShPQNw3ZgT/d8D3DjmDXXA/uq6nhVPQ/sA7YBJDkL+ARwV885JEkT0jcMF1TVs93zXwAXjFlzEfDMyPGR7hzAncBngZd6ziFJmpC55RYkeQR425hLt44eVFUlqVP9xkneBbyzqj6eZP4U1u8EdgJs2bLlVL+NJOk0LRuGqrrmZNeS/DLJhVX1bJILgV+NWXYUuGrkeDPwGPAeYJDkp90c5yd5rKquYoyq2g3sBhgMBqccIEnS6en7VtJe4NW/MtoBfHPMmoeB65Js6G46Xwc8XFX/WFVvr6p54I+AH58sCpKktdM3DHcD1yY5BFzTHZNkkORLAFV1nMV7CU90jzu6c5KkM1CqZu9dmcFgUMPhcNpjSNJMSbK/qgbLrfNfPkuSGoZBktQwDJKkhmGQJDUMgySpYRgkSQ3DIElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJDcMgSWoYBklSwzBIkhqGQZLUMAySpIZhkCQ1DIMkqWEYJEkNwyBJahgGSVLDMEiSGqmqac9w2pIcA3427TlO03nAr6c9xBpzz28M7nl2/G5VbVpu0UyGYRYlGVbVYNpzrCX3/Mbgnl9/fCtJktQwDJKkhmFYO7unPcAUuOc3Bvf8OuM9BklSw1cMkqSGYZigJBuT7EtyqPu44STrdnRrDiXZMeb63iQ/WP2J++uz5yRvTfJQkh8lOZDk7rWd/vQk2ZbkYJKFJLvGXF+f5IHu+uNJ5keufbo7fzDJ9Ws5dx8r3XOSa5PsT/L97uN713r2lejzM+6ub0nyYpJPrtXMq6KqfEzoAdwL7Oqe7wLuGbNmI3C4+7ihe75h5PoHgH8GfjDt/az2noG3An/SrXkz8G/ADdPe00n2uQ74CfCObtb/BLYuWfOXwD91z28GHuieb+3Wrwcu6b7OumnvaZX3/G7g7d3z3weOTns/q7nfkesPAv8CfHLa++nz8BXDZG0H9nTP9wA3jllzPbCvqo5X1fPAPmAbQJKzgE8Ad63BrJOy4j1X1UtV9R2AqnoFeBLYvAYzr8QVwEJVHe5mvZ/FvY8a/W/xIHB1knTn76+ql6vqaWCh+3pnuhXvuaq+V1U/784fAN6SZP2aTL1yfX7GJLkReJrF/c40wzBZF1TVs93zXwAXjFlzEfDMyPGR7hzAncBngZdWbcLJ67tnAJKcA7wPeHQ1hpyAZfcwuqaqTgAvAOee4ueeifrsedQHgSer6uVVmnNSVrzf7pe6TwGfWYM5V93ctAeYNUkeAd425tKtowdVVUlO+U++krwLeGdVfXzp+5bTtlp7Hvn6c8DXgM9X1eGVTakzUZLLgXuA66Y9yyq7Hbivql7sXkDMNMNwmqrqmpNdS/LLJBdW1bNJLgR+NWbZUeCqkePNwGPAe4BBkp+y+HM5P8ljVXUVU7aKe37VbuBQVX1uAuOulqPAxSPHm7tz49Yc6WJ3NvDcKX7umajPnkmyGfgG8OGq+snqj9tbn/1eCdyU5F7gHOC3SX5TVV9Y/bFXwbRvcryeHsDf0t6IvXfMmo0svg+5oXs8DWxcsmae2bn53GvPLN5P+VfgTdPeyzL7nGPxpvkl/P+NycuXrPkY7Y3Jr3fPL6e9+XyY2bj53GfP53TrPzDtfazFfpesuZ0Zv/k89QFeTw8W31t9FDgEPDLyP78B8KWRdX/B4g3IBeDPx3ydWQrDivfM4m9kBfwQeKp7fHTae3qNvf4p8GMW/3Ll1u7cHcD7u+e/w+JfpCwA3wXeMfK5t3afd5Az9C+vJrln4K+B/x75uT4FnD/t/azmz3jka8x8GPyXz5Kkhn+VJElqGAZJUsMwSJIahkGS1DAMkqSGYZAkNQyDJKlhGCRJjf8FFDYZsBaypoYAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(J_history)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Making a prediction using the model\n", "The model can be used by calculating the dot product of the input and $\\theta$." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'In a city with a population of 35000, we predict a profit of $0.00'" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "prediction = np.array([1, 3.5]).dot(theta) * 10000 # don't forget to multiply the prediction by 10000\n", "'In a city with a population of 35000, we predict a profit of $%.2f' % prediction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Multivariate Linear Regression\n", "\n", "---\n", "In this part, you will implement linear regression with multiple variables to predict the prices of houses. Suppose you are selling your house and you want to know what a good market price would be. One way to do this is to first collect information on recent houses sold and make a model of housing prices." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SizeBedroomsPrice
021043399900
116003329900
224003369000
314162232000
430004539900
\n", "
" ], "text/plain": [ " Size Bedrooms Price\n", "0 2104 3 399900\n", "1 1600 3 329900\n", "2 2400 3 369000\n", "3 1416 2 232000\n", "4 3000 4 539900" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# load data\n", "data = pd.read_csv(\"ex1data2.txt\", header = None, names=[\"Size\", \"Bedrooms\",\"Price\"])\n", "m = len(data)\n", "\n", "# Initialize X, y and theta\n", "x0 = np.ones(m)\n", "size = np.array((data[\"Size\"]))\n", "bedrooms = np.array((data[\"Bedrooms\"]))\n", "X = np.array([x0, size, bedrooms]).T\n", "y = np.array(data[\"Price\"]).reshape(len(data.index), 1)\n", "theta_init = np.zeros((3,1))\n", "\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Feature Normalization\n", "When features differ by order of magnitude, first performing feature scaling can make gradient descent converge much more quickly. Formally:\n", "\n", "$x := \\frac{x - \\mu}{\\sigma}$\n", "\n", "Where $\\mu$ is the average and $\\sigma$ the standard deviation.\n", "\n", "**Important**: It is crucial to store $\\mu$ and $\\sigma$ if you want to make predictions using the model later.\n", "\n", "**Exercise**: Perform feature normalization on the following dataset." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[1.000e+00, 2.104e+03, 3.000e+00],\n", " [1.000e+00, 1.600e+03, 3.000e+00],\n", " [1.000e+00, 2.400e+03, 3.000e+00],\n", " [1.000e+00, 1.416e+03, 2.000e+00],\n", " [1.000e+00, 3.000e+03, 4.000e+00]])" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# perform normalization\n", "def normalize(X):\n", " \"\"\" Normalizes the features in X\n", " \n", " returns a normalized version of X where\n", " the mean value of each feature is 0 and the standard deviation\n", " is 1. This is often a good preprocessing step to do when\n", " working with learning algorithms.\n", " \"\"\"\n", " mu = np.zeros(len(X))\n", " sigma = np.zeros(len(X))\n", " \n", " return X, mu, sigma\n", "\n", "X, mu, sigma = normalize(X)\n", "X[0:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Gradient Descent\n", "\n", "Remember the algorithm for gradient descent:\n", "\n", "$repeat \\ \\{ \\\\ \\enspace \\theta_j := \\theta_j - \\alpha \\frac{1}{m}\\displaystyle\\sum_{i = 1}^{m}(h_\\theta(x^{(i)})-y^{(i)})x_j^{(i)}\\\\\\}$\n", "\n", "The vectorization for multivariate gradient descent:\n", "\n", "$\\theta := \\theta - \\frac{\\alpha}{m}X^T(X\\theta - \\vec{y})$\n", "\n", "**Exercise**: Implement gradient descent for multiple features. Make sure your solution is vectorized and supports any number of features." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.],\n", " [0.],\n", " [0.]])" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def gradient_descent_multi(X, y, theta, alpha, iterations):\n", " J_history = []\n", " return theta, J_history\n", "\n", "alpha = 0.01\n", "iterations = 1500\n", "initial_theta = np.zeros((3,1))\n", "theta, J_history = gradient_descent_multi(X, y, initial_theta, alpha, iterations)\n", "theta" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As before we see how the cost decreases over time." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Text(0, 0.5, 'cost')" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEWCAYAAABBvWFzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAFYFJREFUeJzt3X+0XWV95/H3x0SpDjUkEBAIMSjM1NBpYdYt1FXtUEF+OKNhIY7YdhlHWUw70i5xuWqQLkHUDmgpanWmk1FH/AUIDm1mtKYRy9RhLHKDYImICaEMhACBRGykgpHv/HF29HB7knvJfe49ubnv11pnnb2f/ey9vw93kc/d+zln31QVkiRN1rOGXYAkad9goEiSmjBQJElNGCiSpCYMFElSEwaKJKkJA0WaQkneleTjQ67hL5MsH2YNmh3i91A0GyS5EfhsVQ3tH/ckS4B7gGdX1Y4pOsfFwFFV9dtTcXxpd7xCkRpJMncmH1+aLANFs1qSNyW5KclHkzyW5LtJTurbPi/JJ5JsTrIpyfuSzBmz7xVJHgUuHnD8i5N8tlv9m+79+0m2J3lp1+fNSe5Msi3J6iQv7Nu/krw1yXpgfdf24ST3JflBkrVJXt61nwa8C3h9d/zbu/Ybk5zTLT8ryR8muTfJw0k+nWRet21Jd77lSf5fkkeSXNjwP7f2cQaKBCcAdwMHARcB/yPJgm7bp4AdwFHAccApwDlj9t0IHAK8f5zz/Hr3fkBV7V9V30iyjF4InAksBL4OXDVmvzO68yzt1m8BjgUWAJ8Hrk3yc1X1FeCPgGu64//ygBre1L1+A3gRsD/w0TF9Xgb8C+Ak4N1JXjLOuCTAQJEAHgY+VFU/rqprgLuAf5PkEOBVwNuq6odV9TBwBXB2374PVNWfVtWOqvrHPTj37wD/qaru7OZV/gg4tv8qpdu+defxq+qzVfVod87Lgf3oBcBE/BbwJ1W1saq2AxcAZ4+5nfaeqvrHqroduB0YFEzSP2GgSLCpnv7plHuBw4AXAs8GNif5fpLvA/8VOLiv732TPPcLgQ/3HX8rEODwXZ0jyTu6W2SPdfvMo3d1NRGH0RvfTvcCc+ldYe30YN/y4/SuYqRxOcknweFJ0hcqi4FV9P4hfwI4aDefynomH5Mc1Pc+4P1V9bmJ7NfNl/wBvdtR66rqqSTb6IXQROp5gF6I7bSY3i29h4BF4+wr7ZZXKFLviuP3kzw7yeuAlwBfrqrNwF8Blyd5fjeh/eIk/3oPz7MFeIre3MVOfwZckOQY+OmHAF63m2P8PL0A2ALMTfJu4Pl92x8CliTZ1f/bVwHnJzkyyf78bM5lSj7GrNnFQNFssqvf3m8GjgYeoTexflZVPdpteyPwHOA7wDbgOuDQPTp51ePd8W/qbnH9alVdD1wGXJ3kB8AdwOm7Ocxq4CvA9+jdrvoRT78ldm33/miSWwfs/0ngM/Q+cXZPt//v7cl4pLH8YqNmhe4f10uq6s/HtL8JOKeqXjaUwqR9iFco2ud1t5NeAnxr2LVI+zIDRfu0JJfRmwd5Z1XdO15/SXvOW16SpCa8QpEkNTGrvody0EEH1ZIlS4ZdhiTNKGvXrn2kqhaO129WBcqSJUsYHR0ddhmSNKMkmdD8o7e8JElNGCiSpCYMFElSEwaKJKkJA0WS1ISBIklqwkCRJDVhoEiSmjBQJElNGCiSpCYMFElSEwaKJKkJA0WS1ISBIklqwkCRJDVhoEiSmjBQJElNGCiSpCYMFElSEwaKJKkJA0WS1ISBIklqwkCRJDVhoEiSmjBQJElNDDVQkpyW5K4kG5KsGLB9vyTXdNtvTrJkzPbFSbYnecd01SxJGmxogZJkDvAx4HRgKfCGJEvHdHsLsK2qjgKuAC4bs/1PgL+c6lolSeMb5hXK8cCGqtpYVU8CVwPLxvRZBlzZLV8HnJQkAEnOAO4B1k1TvZKk3RhmoBwO3Ne3fn/XNrBPVe0AHgMOTLI/8E7gPeOdJMm5SUaTjG7ZsqVJ4ZKkf2qmTspfDFxRVdvH61hVK6tqpKpGFi5cOPWVSdIsNXeI594EHNG3vqhrG9Tn/iRzgXnAo8AJwFlJPgAcADyV5EdV9dGpL1uSNMgwA+UW4OgkR9ILjrOB3xzTZxWwHPgGcBbwtaoq4OU7OyS5GNhumEjScA0tUKpqR5LzgNXAHOCTVbUuySXAaFWtAj4BfCbJBmArvdCRJO2F0vuFf3YYGRmp0dHRYZchSTNKkrVVNTJev5k6KS9J2ssYKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWpiqIGS5LQkdyXZkGTFgO37Jbmm235zkiVd+yuTrE3yd937K6a7dknS0w0tUJLMAT4GnA4sBd6QZOmYbm8BtlXVUcAVwGVd+yPAq6vqXwLLgc9MT9WSpF0Z5hXK8cCGqtpYVU8CVwPLxvRZBlzZLV8HnJQkVfWtqnqga18HPDfJftNStSRpoGEGyuHAfX3r93dtA/tU1Q7gMeDAMX1eC9xaVU9MUZ2SpAmYO+wCJiPJMfRug52ymz7nAucCLF68eJoqk6TZZ5hXKJuAI/rWF3VtA/skmQvMAx7t1hcB1wNvrKq7d3WSqlpZVSNVNbJw4cKG5UuS+g0zUG4Bjk5yZJLnAGcDq8b0WUVv0h3gLOBrVVVJDgC+BKyoqpumrWJJ0i4NLVC6OZHzgNXAncAXqmpdkkuSvKbr9gngwCQbgLcDOz9afB5wFPDuJLd1r4OneQiSpD6pqmHXMG1GRkZqdHR02GVI0oySZG1VjYzXz2/KS5KaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU1MKFCSvG4ibZKk2WuiVygXTLBNkjRLzd3dxiSnA68CDk/ykb5Nzwd2TGVhkqSZZbeBAjwAjAKvAdb2tf8DcP5UFSVJmnl2GyhVdTtwe5LPV9WPAZLMB46oqm3TUaAkaWaY6BzKmiTPT7IAuBX4b0mumOzJk5yW5K4kG5KsGLB9vyTXdNtvTrKkb9sFXftdSU6dbC2SpMmZaKDMq6ofAGcCn66qE4CTJnPiJHOAjwGnA0uBNyRZOqbbW4BtVXUUcAVwWbfvUuBs4BjgNOA/d8eTJA3JRANlbpJDgX8H/K9G5z4e2FBVG6vqSeBqYNmYPsuAK7vl64CTkqRrv7qqnqiqe4AN3fEkSUMy0UC5BFgN3F1VtyR5EbB+kuc+HLivb/3+rm1gn6raATwGHDjBfQFIcm6S0SSjW7ZsmWTJkqRdmVCgVNW1VfVLVfW73frGqnrt1JbWRlWtrKqRqhpZuHDhsMuRpH3WRL8pvyjJ9Uke7l5fTLJokufeBBzRt76oaxvYJ8lcYB7w6AT3lSRNo4ne8vrvwCrgsO71P7u2ybgFODrJkUmeQ2+SfdWYPquA5d3yWcDXqqq69rO7T4EdCRwNfHOS9UiSJmG8LzbutLCq+gPkU0neNpkTV9WOJOfRm5uZA3yyqtYluQQYrapVwCeAzyTZAGylFzp0/b4AfIfeN/bfWlU/mUw9kqTJmWigPJrkt4GruvU30Lv1NClV9WXgy2Pa3t23/CNg4EMoq+r9wPsnW4MkqY2J3vJ6M72PDD8IbKZ3++lNU1STJGkGmugVyiXA8p2PW+m+Mf/H9IJGkqQJX6H8Uv+zu6pqK3Dc1JQkSZqJJhooz+oeCgn89Aplolc3kqRZYKKhcDnwjSTXduuvwwlxSVKfCQVKVX06ySjwiq7pzKr6ztSVJUmaaSZ826oLEENEkjTQROdQJEnaLQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmhhIoSRYkWZNkffc+fxf9lnd91idZ3rU9L8mXknw3ybokl05v9ZKkQYZ1hbICuKGqjgZu6NafJskC4CLgBOB44KK+4PnjqvoF4Djg15KcPj1lS5J2ZViBsgy4slu+EjhjQJ9TgTVVtbWqtgFrgNOq6vGq+muAqnoSuBVYNA01S5J2Y1iBckhVbe6WHwQOGdDncOC+vvX7u7afSnIA8Gp6VzmSpCGaO1UHTvJV4AUDNl3Yv1JVlaT24PhzgauAj1TVxt30Oxc4F2Dx4sXP9DSSpAmaskCpqpN3tS3JQ0kOrarNSQ4FHh7QbRNwYt/6IuDGvvWVwPqq+tA4dazs+jIyMvKMg0uSNDHDuuW1CljeLS8H/mJAn9XAKUnmd5Pxp3RtJHkfMA942zTUKkmagGEFyqXAK5OsB07u1kkykuTjAFW1FXgvcEv3uqSqtiZZRO+22VLg1iS3JTlnGIOQJP1MqmbPXaCRkZEaHR0ddhmSNKMkWVtVI+P185vykqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpowUCRJTRgokqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkSQ1YaBIkpoYSqAkWZBkTZL13fv8XfRb3vVZn2T5gO2rktwx9RVLksYzrCuUFcANVXU0cEO3/jRJFgAXAScAxwMX9QdPkjOB7dNTriRpPMMKlGXAld3ylcAZA/qcCqypqq1VtQ1YA5wGkGR/4O3A+6ahVknSBAwrUA6pqs3d8oPAIQP6HA7c17d+f9cG8F7gcuDx8U6U5Nwko0lGt2zZMomSJUm7M3eqDpzkq8ALBmy6sH+lqipJPYPjHgu8uKrOT7JkvP5VtRJYCTAyMjLh80iSnpkpC5SqOnlX25I8lOTQqtqc5FDg4QHdNgEn9q0vAm4EXgqMJPl7evUfnOTGqjoRSdLQDOuW1ypg56e2lgN/MaDPauCUJPO7yfhTgNVV9V+q6rCqWgK8DPieYSJJwzesQLkUeGWS9cDJ3TpJRpJ8HKCqttKbK7mle13StUmS9kKpmj3TCiMjIzU6OjrsMiRpRkmytqpGxuvnN+UlSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaMFAkSU0YKJKkJgwUSVITBookqQkDRZLUhIEiSWrCQJEkNWGgSJKaSFUNu4Zpk2QLcO+w63iGDgIeGXYR08wxzw6OeeZ4YVUtHK/TrAqUmSjJaFWNDLuO6eSYZwfHvO/xlpckqQkDRZLUhIGy91s57AKGwDHPDo55H+MciiSpCa9QJElNGCiSpCYMlL1AkgVJ1iRZ373P30W/5V2f9UmWD9i+KskdU1/x5E1mzEmel+RLSb6bZF2SS6e3+mcmyWlJ7kqyIcmKAdv3S3JNt/3mJEv6tl3Qtd+V5NTprHsy9nTMSV6ZZG2Sv+veXzHdte+JyfyMu+2Lk2xP8o7pqnlKVJWvIb+ADwAruuUVwGUD+iwANnbv87vl+X3bzwQ+D9wx7PFM9ZiB5wG/0fV5DvB14PRhj2kX45wD3A28qKv1dmDpmD7/Efizbvls4JpueWnXfz/gyO44c4Y9pike83HAYd3yLwKbhj2eqRxv3/brgGuBdwx7PJN5eYWyd1gGXNktXwmcMaDPqcCaqtpaVduANcBpAEn2B94OvG8aam1lj8dcVY9X1V8DVNWTwK3AommoeU8cD2yoqo1drVfTG3u//v8W1wEnJUnXfnVVPVFV9wAbuuPt7fZ4zFX1rap6oGtfBzw3yX7TUvWem8zPmCRnAPfQG++MZqDsHQ6pqs3d8oPAIQP6HA7c17d+f9cG8F7gcuDxKauwvcmOGYAkBwCvBm6YiiIbGHcM/X2qagfwGHDgBPfdG01mzP1eC9xaVU9MUZ2t7PF4u18G3wm8ZxrqnHJzh13AbJHkq8ALBmy6sH+lqirJhD/LneRY4MVVdf7Y+7LDNlVj7jv+XOAq4CNVtXHPqtTeKMkxwGXAKcOuZYpdDFxRVdu7C5YZzUCZJlV18q62JXkoyaFVtTnJocDDA7ptAk7sW18E3Ai8FBhJ8vf0fp4HJ7mxqk5kyKZwzDutBNZX1YcalDtVNgFH9K0v6toG9bm/C8l5wKMT3HdvNJkxk2QRcD3wxqq6e+rLnbTJjPcE4KwkHwAOAJ5K8qOq+ujUlz0Fhj2J46sAPsjTJ6g/MKDPAnr3Wed3r3uABWP6LGHmTMpPasz05ou+CDxr2GMZZ5xz6X2Y4Eh+NmF7zJg+b+XpE7Zf6JaP4emT8huZGZPykxnzAV3/M4c9jukY75g+FzPDJ+WHXoCvgt694xuA9cBX+/7RHAE+3tfvzfQmZjcA/37AcWZSoOzxmOn9BljAncBt3eucYY9pN2N9FfA9ep8EurBruwR4Tbf8c/Q+4bMB+Cbwor59L+z2u4u99JNsLccM/CHww76f623AwcMez1T+jPuOMeMDxUevSJKa8FNekqQmDBRJUhMGiiSpCQNFktSEgSJJasJAkfZAkv/bvS9J8puNj/2uQeeS9nZ+bFiahCQn0vvuwL99BvvMrd7znHa1fXtV7d+iPmk6eYUi7YEk27vFS4GXJ7ktyflJ5iT5YJJbknw7yX/o+p+Y5OtJVgHf6dr+vPubH+uSnNu1XUrvCbu3Jflc/7nS88Ekd3R/L+T1fce+Mcl13d+I+dzOJ9lK08lneUmTs4K+K5QuGB6rql/pHrt+U5K/6vr+K+AXq/coeoA3V9XWJM8FbknyxapakeS8qjp2wLnOBI4Ffhk4qNvnb7ptx9F7VMsDwE3ArwH/p/1wpV3zCkVq6xTgjUluA26m94iZo7tt3+wLE4DfT3I78Lf0Hhx4NLv3MuCqqvpJVT0E/G/gV/qOfX9VPUXvcSVLmoxGega8QpHaCvB7VbX6aY29uZYfjlk/GXhpVT2e5EZ6z3vaU/1/M+Qn+P+2hsArFGly/gH4+b711cDvJnk2QJJ/nuSfDdhvHrCtC5NfAH61b9uPd+4/xteB13fzNAuBX6f3oEFpr+BvMdLkfBv4SXfr6lPAh+ndbrq1mxjfwuA/b/wV4HeS3EnvScJ/27dtJfDtJLdW1W/1tV9P7+/f3E7vact/UFUPdoEkDZ0fG5YkNeEtL0lSEwaKJKkJA0WS1ISBIklqwkCRJDVhoEiSmjBQJElN/H+7X9GYyN1Q+AAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(J_history)\n", "plt.title('J per iteration')\n", "plt.xlabel('iteration')\n", "plt.ylabel('cost')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we want to make a prediction on a normalized dataset, we have to normalize our input too." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:1: RuntimeWarning: divide by zero encountered in double_scalars\n", " \"\"\"Entry point for launching an IPython kernel.\n" ] }, { "data": { "text/plain": [ "'In a house of 1650 square feet with 3 rooms, we predict a price of $nan'" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "price = theta.transpose() @ np.array([1, (1650-mu[1])/sigma[1], (3-mu[2])/sigma[2]]) # normalize the input\n", "'In a house of 1650 square feet with 3 rooms, we predict a price of $%.2f' % price" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using normal equations\n", "We can use normal equations to get the exact solution in only one calculation. Although using normal equations is very fast for a small datasets with a small number of features, it can be inefficient for larger datasets because the complexity of matrix multiplication is $O(n^3)$.\n", "\n", "The normal equation for linear regression is:\n", "\n", "$\\theta = 􏰅 (X^TX)^{−1}X^T\\vec{y}$\n", "\n", "**Exercise**: Find theta using normal equations." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[89597.9095428 ],\n", " [ 139.21067402],\n", " [-8738.01911233]])" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "theta" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:1: RuntimeWarning: divide by zero encountered in double_scalars\n", " \"\"\"Entry point for launching an IPython kernel.\n" ] }, { "data": { "text/plain": [ "'In a house of 1650 square feet with 3 rooms, we predict a price of $nan'" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "price = theta.transpose() @ np.array([1, (1650-mu[1])/sigma[1], (3-mu[2])/sigma[2]]) # normalize the input\n", "'In a house of 1650 square feet with 3 rooms, we predict a price of $%.2f' % price" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: ex1/ex1data1.txt ================================================ 6.1101,17.592 5.5277,9.1302 8.5186,13.662 7.0032,11.854 5.8598,6.8233 8.3829,11.886 7.4764,4.3483 8.5781,12 6.4862,6.5987 5.0546,3.8166 5.7107,3.2522 14.164,15.505 5.734,3.1551 8.4084,7.2258 5.6407,0.71618 5.3794,3.5129 6.3654,5.3048 5.1301,0.56077 6.4296,3.6518 7.0708,5.3893 6.1891,3.1386 20.27,21.767 5.4901,4.263 6.3261,5.1875 5.5649,3.0825 18.945,22.638 12.828,13.501 10.957,7.0467 13.176,14.692 22.203,24.147 5.2524,-1.22 6.5894,5.9966 9.2482,12.134 5.8918,1.8495 8.2111,6.5426 7.9334,4.5623 8.0959,4.1164 5.6063,3.3928 12.836,10.117 6.3534,5.4974 5.4069,0.55657 6.8825,3.9115 11.708,5.3854 5.7737,2.4406 7.8247,6.7318 7.0931,1.0463 5.0702,5.1337 5.8014,1.844 11.7,8.0043 5.5416,1.0179 7.5402,6.7504 5.3077,1.8396 7.4239,4.2885 7.6031,4.9981 6.3328,1.4233 6.3589,-1.4211 6.2742,2.4756 5.6397,4.6042 9.3102,3.9624 9.4536,5.4141 8.8254,5.1694 5.1793,-0.74279 21.279,17.929 14.908,12.054 18.959,17.054 7.2182,4.8852 8.2951,5.7442 10.236,7.7754 5.4994,1.0173 20.341,20.992 10.136,6.6799 7.3345,4.0259 6.0062,1.2784 7.2259,3.3411 5.0269,-2.6807 6.5479,0.29678 7.5386,3.8845 5.0365,5.7014 10.274,6.7526 5.1077,2.0576 5.7292,0.47953 5.1884,0.20421 6.3557,0.67861 9.7687,7.5435 6.5159,5.3436 8.5172,4.2415 9.1802,6.7981 6.002,0.92695 5.5204,0.152 5.0594,2.8214 5.7077,1.8451 7.6366,4.2959 5.8707,7.2029 5.3054,1.9869 8.2934,0.14454 13.394,9.0551 5.4369,0.61705 ================================================ FILE: ex1/ex1data2.txt ================================================ 2104,3,399900 1600,3,329900 2400,3,369000 1416,2,232000 3000,4,539900 1985,4,299900 1534,3,314900 1427,3,198999 1380,3,212000 1494,3,242500 1940,4,239999 2000,3,347000 1890,3,329999 4478,5,699900 1268,3,259900 2300,4,449900 1320,2,299900 1236,3,199900 2609,4,499998 3031,4,599000 1767,3,252900 1888,2,255000 1604,3,242900 1962,4,259900 3890,3,573900 1100,3,249900 1458,3,464500 2526,3,469000 2200,3,475000 2637,3,299900 1839,2,349900 1000,1,169900 2040,4,314900 3137,3,579900 1811,4,285900 1437,3,249900 1239,3,229900 2132,4,345000 4215,4,549000 2162,4,287000 1664,2,368500 2238,3,329900 2567,4,314000 1200,3,299000 852,2,179900 1852,4,299900 1203,3,239500 ================================================ FILE: ex2/PE2 - Logistic Regression (Exercises).ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Linear Regression\n", "\n", "Stanford CS229 - Machine Learning by Andrew Ng. Programming exercise 2 with solutions.\n", "\n", "Please check out [the repository on GitHub](https://github.com/rickwierenga/CS229-Python/). If you spot any mistakes or inconcistencies, please create an issue. For questions you can find me on Twitter: [@rickwierenga](https://twitter.com/rickwierenga). Starring the project on GitHub means a ton to me!" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pylab as plt\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Logistic Regression\n", "\n", "---\n", "Logistic regression is predicting in which category a given data point is. In binary classifiction, there are only two categories:\n", "\n", "$$y \\in \\{0,1\\}$$\n", "\n", "In this exercise, you will implement logistic regression and apply it to two different datasets. Before starting on the programming exercise, we strongly recommend watching the video lectures and completing the review questions for the associated topics." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[ 1. , 34.62365962, 78.02469282],\n", " [ 1. , 30.28671077, 43.89499752],\n", " [ 1. , 35.84740877, 72.90219803],\n", " [ 1. , 60.18259939, 86.3085521 ],\n", " [ 1. , 79.03273605, 75.34437644]])" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# start by loading the data\n", "data = pd.read_csv(\"ex2data1.txt\", header = None, \n", " names = [\"Exam 1 Score\", \"Exam 2 Score\", \"Accepted\"])\n", "\n", "# initialize some useful variables\n", "m = len(data[\"Accepted\"])\n", "x0 = np.ones(m)\n", "size = np.array((data[\"Exam 1 Score\"]))\n", "bedrooms = np.array((data[\"Exam 2 Score\"]))\n", "X = np.array([x0, size, bedrooms]).T\n", "y = np.array(data[\"Accepted\"]).reshape((m,1))\n", "m, n = X.shape\n", "\n", "X[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualising the data\n", "A blue cross means accepted. A yellow oval means rejected." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3XuQVOW57/HvA+gQCAmC7AlKYCAiKiKjjCYTjRLxlsSo2V6i4Sgasymr1O01Xso6MqSOVVqmyq0mFUPCVk4K0UjhJSbmqETijZgMhgiCbA0OiMIwQSHBCwrznD969TgMPZe+rnet/n2qpqZ7TU+vp9f0POvt570sc3dERCS9+sUdgIiIlJcSvYhIyinRi4iknBK9iEjKKdGLiKScEr2ISMop0YuIpJwSvYhIyinRi4ik3IC4AwDYd999va6uLu4wREQSZdmyZf9w9xG9PS6IRF9XV0dzc3PcYYiIJIqZrevL41S6ERFJuV4TvZn9t5ltNrOVnbYNM7OnzOz16Ps+0XYzs7vM7A0ze8XMjihn8CIi0ru+tOjvA07psu0GYLG7jwcWR/cBvgGMj75mAj8rTZgiIlKoXhO9uz8LvNtl8+nAvOj2POCMTtv/r2f8CRhqZiNLFayIiOSv0Bp9rbtvjG5vAmqj2/sDb3V63IZom4iIxKTozljPXLkk76uXmNlMM2s2s+a2trZiwyhIa+t8li6tY8mSfixdWkdr6/xY4hARKadCE31rtiQTfd8cbX8b+GKnx42Ktu3B3ee4e4O7N4wY0esw0JJrbZ3PmjUz2bFjHeDs2LGONWtmKtmLSOoUmugfA2ZEt2cAj3bafkE0+uYrwLZOJZ6grF17E+3tH+y2rb39A9auvSmmiEREyqPXCVNmtgCYCuxrZhuAWcCtwK/N7GJgHXBO9PDfAd8E3gA+AC4qQ8wlsWPH+ry2i4gkVV9G3Zzn7iPdfS93H+Xuc919i7tPc/fx7n6Cu78bPdbd/VJ3/5K7T3L3YKe71tSMzmt7qTU1VWQ3Ehj93SUOVTszdty4W+jXb9Bu2/r1G8S4cbdUZP+zZ1dkNxIY/d0lDlWb6GtrpzNhwhxqasYARk3NGCZMmENt7fS4Q5MyUotaqlHVJnrIJPvGxhamTm2nsbGl7Em+qQnMMl/w6W0ln/LpemzjaFEn6e8eYkxSPMsMg49XQ0ODV9vqlWYQwKFPva7HOe7jXsj+m5oql4DjPj6SHzNb5u4NvT2uqlv0Uh2S1KLORXV9KVZVJPoQZ8DOmhV3BOnVNbFnE2X2mLtnvuJI9CH+3ZN+IpTepb50k50B23lyVL9+g9TxWiVCK930VVNT7pb8rFnlTcBJOT6SodJNRDNgpbMQW9S5NDV9+skD4v0UIsmX+kSvGbDVrWtiV6LsWVJOhJKf1Cf6uGfASrzSkNgrmXzTcLxkT6lP9HHPgBUplpKvFCv1iV4zYEWk2vW6emUa1NZOV2KXYFVyQpRUp9S36EVCpwlRUm5K9CIiKadELxKDJM5GDTk26VnqZ8aKhC4ps1GTEmc10cxYEREBlOhFYhfybNQklphkTyrdiEifZJN9AClDIhUp3ZjZFWa20sxeNbMro23DzOwpM3s9+r5PMfsQkfypxS2dFZzozexQ4D+Ao4DJwKlmdgBwA7DY3ccDi6P7IlJBpRybny3fZKl8kzzFtOgPBl5y9w/cfSfwR+DfgdOBedFj5gFnFBeiiPRFuRKvlkxOvmIS/Urga2Y23MwGAd8EvgjUuvvG6DGbgNoiY5SEU0KojNmz1XEquRW81o27rzaz24AngfeB5cCuLo9xM8vZdWNmM4GZAKNHa8ngNJs9W8mmUrKt7nKNeQ95hJB0r6jOWHef6+5T3P1Y4D3gf4BWMxsJEH3f3M3vznH3BndvGDFiRDFhSB8p2aZPd8Mfy7m/pEvDa8hXsaNu/i36PppMff5+4DFgRvSQGcCjxexDSqeSi2dp/HVldFc/V8u7e9W4iFxR4+jN7DlgOPAJcLW7Lzaz4cCvgdHAOuAcd3+3p+fROPrKiGsKu6bOV4aOc9+k6ThVZBy9u3/N3Q9x98nuvjjatsXdp7n7eHc/obckL+WllnX16K4VX81/6+xrr/b/A82MDVQ5LkYRV0tGF9aIV5pasPnK9drTdDy0qFnCpamOqCQv1SqU974SfRVRB131qOZSRW+vvZL/B6E02FS6CUhTU+43xqxZ1fEPWgoqE+0pTaWKfMX92su9f5VuEkhTzYsXSgsqbnrPxCfET1NK9CJlFNc/d+cTXjWX7OJ47SE22JToA1XN/5z5CrEFlRXCJ4wQjkNcqvm1d6ZEHyi9QfsuxBZUpWX7JkI94VWrUBps6oyVVIm78w3i6VTv+rpDOA5Sfn3tjC149UqREIXQguo88kcJV0Kg0o2kSjWVKXoq1YRwwpNwqEUvUkblTLj65CB9pRa9SBlV0ycMCZcSvUgKqFQjPVGiF0kBfXKQnijRi4iknBK9iEjKKdGLiKScEr1IF6p3S9oo0Yt0EcJCZCKlVFSiN7OrzOxVM1tpZgvMbKCZjTWzl8zsDTN70Mz2LlWwIiKSv4ITvZntD/wn0ODuhwL9gXOB24A73P0A4D3g4lIEKlJOWvlR0qzY0s0A4DNmNgAYBGwEjgcWRj+fB5xR5D5Eyk5LHUuaFZzo3f1t4MfAejIJfhuwDNjq7jujh20A9i82SBERKVwxpZt9gNOBscB+wGDglDx+f6aZNZtZc1tbW6FhpEpr63yWLq1jyZJ+LF1aR2vr/LhDqkpaTkDSppjSzQnAm+7e5u6fAIuAo4GhUSkHYBTwdq5fdvc57t7g7g0jRowoIox0aG2dz5o1M9mxYx3g7NixjjVrZirZxyCEck0IMUh6FJPo1wNfMbNBZmbANGAV8AxwVvSYGcCjxYVYHdauvYn29g9229be/gFr194UU0QSJw3xlFIqpkb/EplO15eBFdFzzQGuB642szeA4cDcEsRZdnGXTXbsWJ/XdikftaYlbYoadePus9z9IHc/1N3Pd/cd7r7W3Y9y9wPc/Wx331GqYMslhLJJTc3ovLZL+cTVmtYQTykXzYwljLLJuHG30K/foN229es3iHHjbqlYDBIvDfEMT1qOvRI9YZRNamunM2HCHGpqxgBGTc0YJkyYQ23t9IrFUM3UmpZc0tJXomvGkimPZMo2e26vpNra6UrsMQnt+qtJHeLZ+ThKONSip7xlk7g7eSWZkpos09ACTuOnOyV6ylc2CaGTtxRCfIOXM6aktqalNNLYV2Ie92dUoKGhwZubm+MOo+SWLq3rpiQ0hsbGlsoHVKAQShldhRhTmuRTgmlqyt2SnzUr2ckRwn+fmdkyd2/o7XFV36IvZ2klhE5ekULkU4JJYws4Ky2f7qo60Ze7tJLksfEh1ilDjEnSLS3vrapO9OUeP5/ksfEhttJCjClNSnEiTUsLOG2qOtGXu7SisfGSJD2dSPOp10t4qnocfSXGz6dhbHyIrbQQY0qz2bOVxJOsqlv0SS6tVFKI/+AhxpQmOpGmS1Un+r6UVjThKXl0Eihetlyjzu900Dj6HmRH5XTusO3Xb5Dq7IHrOvZZ0/KLF/p48mqlcfQlEMKqllK8NEzLl/SqRCNEib4HmvCUHCozlJdq9uVTiYaIEn0Pkjzhqdp0HRqYTUzZfyIl/uLouCWbEn0PSj0qRx27laPJVRKySn8Crepx9L3JdriuXXsTO3asp6ZmNOPG3VJQR2zXjt3scgud9yOloTKDhK7S1z/QqJsKSctKlkmkUTcSsmISfdlH3ZjZBDNb3unrn2Z2pZkNM7OnzOz16Ps+he4jTdSxGx8l+T3pmISjEp9AC0707r7G3evdvR6YAnwAPAzcACx29/HA4uh+1VPHbjjSluQKeT1pGnKa9L9nkoZXTgP+7u7rgNOBedH2ecAZJdpHoqV1uYUk/pOlKclB+l5Pvqr99fdFqRL9ucCC6Hatu2+Mbm8Caku0j0RL60qW+idLDs01qF5FJ3oz2xs4DXio688809Obs5vBzGaaWbOZNbe1tRUbRiLU1k6nsbGFqVPbaWxsSXyST5K0JblCXk93Q06TKAl/z97+FpVU9KgbMzsduNTdT4rurwGmuvtGMxsJLHH3CT09RzWMukmTpF8jNG3rthTyejr/TtKPR6jx9xRXqWKu5Fo35/Fp2QbgMWBGdHsG8GgJ9iEB0WSk5NNcg+pSVKI3s8HAicCiTptvBU40s9eBE6L7IsFIW5Ir9PWEXvroq5D+nj2VlOIsN2nClBRFk5GSL9TSR9KlrXQjVUxJXiR8SvQiVS6k0kea9HRcK33MVbpJqNbW+SVZbE1KR2UsqTSVblIsuxJmZpE071gJU8sex0uTxyRUSvQJpEsciuSmT1S5KdEnkFbCDEcSZmhWk3J8qkrD31KJvsQqcRUprYQZDk0eS780lOSU6EuoUrXztKyEqWQopaBPVb1Toi+hStXO07ISZhpaSp1pmGI8yvGpKm0nDw2vLKElS/qRe7FOY+rU9kqHEzzNyJRSK8d7KuT3qYZXxkC1896lraUkYdGnqtyU6EsoLbXzclLnpZRTOd5HaTh5KNGXUFpq531RidFFIiFIQyNkQNwBpE1t7fRUJvbOsqOLsh3P2dFFQF6vPQ0tJZEkUIte8laq0UVpaCmJJIESveRNM3NFkkWJXvKm0UUiyaJEL3nT6CKRZFGil7xV0+ii3qifQQpR6feNZsZKUar9Aighz5qUcCXqmrFmNtTMFprZa2a22swazWyYmT1lZq9H3/cpZh+VonHh+dMFUESSodjSzZ3A7939IGAysBq4AVjs7uOBxdH9oClhZeR7sqvWC6BoGQcpRJzvm4JLN2b2eWA5MM47PYmZrQGmuvtGMxsJLHH3CT09V9ylm6VL66Ikv7uamjE0NrZUPqAYdJ0EBZkO1p5q7+VexC0J12BV6UYKkaTSzVigDbjXzP5qZr80s8FArbtvjB6zCagtYh8VoXHhhbXOyz3MMm3LGIvEpZhEPwA4AviZux8OvE+XMk3U0s953jKzmWbWbGbNbW1tRYRRPI0LL+xkp2GWWsZBClPp900xiX4DsMHdX4ruLyST+Fujkg3R9825ftnd57h7g7s3jBgxoogwiqeEVdjJrhzDLJNW/w41LglbooZXmtlzwA/cfY2ZNQGDox9tcfdbzewGYJi7X9fT88RdowcNEyykRl9uqn+L9KyvNfpiV6+8HJhvZnsDa4GLyHxK+LWZXQysA84pch8VUQ2rTvYk+9qr+WQnklZFJXp3Xw7kOptMK+Z5JR6hnexU/xYpDS2BIMFS/bt66G9dXkr0IhI7DaUtLyX6hNPSDZIUarXHR4k+wbR0gyRJ11Z70obSJplWr0wwLd0gSdLTcFkNpS1MRVavlHhp6QYJnVrtYVCiTzAt3SCha2rKtNSzrfXs7a6JXkNpy0uJPsG0dIOkhVr45aVEn2C6pJ8kiVrt8VFnrIhIQqkzVkREACV6EZHUU6IXEUk5JXqRCtCoEomTEr1URLWvyaNFuyROSvRS9iQcypo81X6ykeqlRF/lKpGE1669abdLFAK0t3/A2rU3lWwfvYnjZKPp/xIKJfoqV4kkHMKaPHGcbPo6/V+k3JToq1wlknAIa/KEcLIRiYsSfZWrRBIOYU2euE82mv4vcSoq0ZtZi5mtMLPlZtYcbRtmZk+Z2evR931KE6qUQyWScNxr8rS2zmfnzu17bK/kyUblGonTgBI8x9fd/R+d7t8ALHb3W83shuj+9SXYj5RBNtmuXXsTO3asp6ZmNOPG3UJt7XRaW+fn3F7ofuJYbC3bCdu1Pj9gwHDGj79TC8BJVShFou/qdGBqdHsesIQyJPpSJqFqlysJd02Q2VEq2ccnRa5OWID+/T+bqNchUoxia/QOPGlmy8xsZrSt1t03Rrc3AbVF7mMPoYzLTrMQhkSWgjphRYpP9Me4+xHAN4BLzezYzj/0zBrIOddBNrOZZtZsZs1tbW157TQtSShk3SfIPa9RG7K4O2FFQlBUonf3t6Pvm4GHgaOAVjMbCRB939zN785x9wZ3bxgxYkRe+1Urrfy6T4SWqE9OIYz4qXbqiI5fwYnezAab2ZDsbeAkYCXwGDAjetgM4NFig+xKrbTyyyRCy/ETT9Qnp7hH/IjW+QlBMZ2xtcDDlpnfPQC4391/b2Z/AX5tZhcD64Bzig9zd+PG3bLHSAq10kqrtnY6q1f/r5w/S9onp7hG/IiEouAWvbuvdffJ0ddEd78l2r7F3ae5+3h3P8Hd3y1duBlqpVVG5vjm2q5PTklWiVKK1vkJi64Zm4OGbmbkGoPer98gnVQTzuzT9XfSuL9qomvGFkhDNz+V65PTF74wg7Vrb9JSv93QUsgSIiX6LjR0c3e1tdNpbGxh6tR2xo27hU2b5ukk2I0QGgndlUbiLKVonZ/4qXTTxZIl/cg99N+YOrW90uEEZenSupzj6GtqxtDY2FL5gAITwvHpS5lEpZT0UOmmQBq6mVtr6/xuJ0slbRROuWh+R7oluSNZib4LTbDZU7Yk0Z1qPwlmxdVIyLcsk/ZSSrkScpLnA6h0k4NG3eyuu5IEaBROZyGMUlJZpnzHIMRjq9JNETp3QDY2tlR9Euup9KAk/6m0z+9IcumiUGmZD6AWvfQqhE5G6ZumpvIloRBbtFlNTblLK7Nmle54hPj6q6JFX4oxyxr33Dv1WyRH0lqapaILsfcssYm+FGOWQxj3nARpL0lI98pRukhq8k1yJ3ZiSzf5lhNydbBm7qskIdIXpSpdlLsE0rl8Vc5SVgj6WrpJbKLPZ2JTd6Mhcl1irrvnEOlOtYzSSkqij2tfcUh9jT6fMcvdLWsA/fN6bglbHP0t1VT+66l00duxT8volaRKbKLPp4Ow++GBu9TJmBJxJdxqWhupu6Tcl2Nfyc5SnVT2lNhEn08HYfet/zHqZEyJuBKulj0I72SnETh7KuYKU7Hr65WDeroila4+lA5xJdyamtHddOhXrvwXdx9Bvsc+yaNXkiqxLfp8aHhg+sW1zkzccwxC6CPI99hXsmWtk0pGYkfdiHRW6XVmOreiBwwYhjvs2vVuxVvUIcxaDmGNn2rV11E3iS7diGRlE0olShhdE9vOnVvo128QBx/8q4onthD6CGprp7Nt2wu8884cYBfQny98YYaSfECKTvRm1h9oBt5291PNbCzwADAcWAac7+4fF7sfkd5Uqr+lp87HSie3UPoINm2aRybJA+xi06Z5fP7zRyvZB6IUNforgNWd7t8G3OHuBwDvAReXYB8iwQihFZ0Vdx8BhDfqRvZUVKI3s1HAt4BfRvcNOB5YGD1kHnBGMfsQCU1IVyELYaBBSCc+ya3Y0s1/AdcBQ6L7w4Gt7r4zur8B2L+QJ/7kk0/YsGEDH330UZEhSqkMHDiQUaNGsddee5X0eeMeHpivnobrxiHuIcIhlI+kZwUnejM7Fdjs7svMbGoBvz8TmAkwevSeb4gNGzYwZMgQ6urqsOwUN4mNu7NlyxY2bNjA2LFjS/a8XTs2s8MDgWCTfSU7fpMgtBNfvpLW0ChEMS36o4HTzOybwEDgc8CdwFAzGxC16kcBb+f6ZXefA8yBzPDKrj//6KOPlOQDYmYMHz6ctra2kj5vSB2b+Yi7FR2SJJ/4ktjQKETBid7dbwRuBIha9Ne6+3Qzewg4i8zImxnAo4XuQ0k+LOX4e6i+mw5JPfEV0tBI4ieAcsyMvR642szeIFOzn1uGfVTMI488gpnx2muv5fz5hRdeyMKFC3P+LJd33nmHs846C4Dly5fzu9/9ruNnS5Ys4cUXX8w7xrq6Ov7xj3/k/XshCKljU6pPvg2NEGYiF6Ikid7dl7j7qdHtte5+lLsf4O5nu/uOUuyjr0o9vXrBggUcc8wxLFiwoCTPt99++3WcGEqV6JMshOGBUr3ybWgkdShp6ta6yXWB4EJt376d559/nrlz5/LAAw8AmU7Jyy67jAkTJnDCCSewefPmjsfX1dVx4403Ul9fT0NDAy+//DInn3wyX/rSl7jnnnsAaGlp4dBDD+Xjjz/m5ptv5sEHH6S+vp7bbruNe+65hzvuuIP6+nqee+452traOPPMMznyyCM58sgjeeGFFwDYsmULJ510EhMnTuQHP/gBISxjUagQhgdK9cq3oZHUUqOWQOjBo48+yimnnMKBBx7I8OHDWbZsGevWrWPNmjWsWrWK1tZWDjnkEL7//e93/M7o0aNZvnw5V111FRdeeCEvvPACH330EYceeiiXXHJJx+P23ntvfvSjH9Hc3MxPfvITAD788EM++9nPcu211wLwve99j6uuuopjjjmG9evXc/LJJ7N69Wpmz57NMcccw80338xvf/tb5s5NdHUssfVdSb58O5KTOpQ0FYm+qWn3lny2z3DWrOJKOQsWLOCKK64A4Nxzz2XBggXs3LmT8847j/79+7Pffvtx/PHH7/Y7p512GgCTJk1i+/btDBkyhCFDhlBTU8PWrVvz2v/TTz/NqlWrOu7/85//ZPv27Tz77LMsWrQIgG9961vss88+hb9IkSqXT0MjqUNJU5Poswm9VNeIfPfdd/nDH/7AihUrMDN27dqFmfGd73ynx9+rqakBoF+/fh23s/d37tzZ3a/l1N7ezp/+9CcGDhyY/wsQkZJL6lDS1NXoS2XhwoWcf/75rFu3jpaWFt566y3Gjh3L8OHDefDBB9m1axcbN27kmWeeKXgfQ4YM4V//+le390866STuvvvujvvLly8H4Nhjj+X+++8H4IknnuC9994rOAYRyU9t7XQaG1uYOrWdxsaW4JM8pDDRl+pCAwsWLNij9X7mmWeyceNGxo8fzyGHHMIFF1xAY2Njwfv4+te/zqpVq6ivr+fBBx/k29/+Ng8//HBHZ+xdd91Fc3Mzhx12GIccckhHh+6sWbN49tlnmThxIosWLco5szhkcVzEW6SaBXvhkdWrV3PwwQfHFJF0p9i/iy5SIVI6fb3wSOpa9BK2pI5DFkkyJXqpqKSOQxZJMiV6qSgteSBSeUr0UlFa8kCk8pTopaK05IFI5aViwpQki5Y8EKksteh7YGZcc801Hfd//OMf09TLmgqPPPLIbssWFCLfZYcfe+wxbr311pz7v++++3jnnXfy2n924TURSYfUJPpyTMKpqalh0aJFeSXdUiT6fJ122mnccMMNOfdfSKIXkXRJRaIv18UABgwYwMyZM7njjjv2+FlLSwvHH388hx12GNOmTWP9+vW8+OKLPPbYY/zwhz+kvr6ev//977v9zm9+8xu+/OUvc/jhh3PCCSfQ2toKdL/scEtLCwcddBAXXnghBx54INOnT+fpp5/m6KOPZvz48fz5z38GMsn8sssu22P/t912G83NzUyfPp36+no+/PBDli1bxnHHHceUKVM4+eST2bhxIwDLli1j8uTJTJ48mZ/+9KdFHTcRCYy7x/41ZcoU72rVqlV7bOvOiy+O8WeeYY+vF18c0+fnyGXw4MG+bds2HzNmjG/dutVvv/12nzVrlru7n3rqqX7fffe5u/vcuXP99NNPd3f3GTNm+EMPPZTz+d59911vb293d/df/OIXfvXVV7u7++WXX+6zZ892d/fHH3/cAW9ra/M333zT+/fv76+88orv2rXLjzjiCL/ooou8vb3dH3nkkY593nvvvX7ppZfm3P9xxx3nf/nLX9zd/eOPP/bGxkbfvHmzu7s/8MADftFFF7m7+6RJk/yPf/yju7tfe+21PnHixJyvIZ+/i4iUF9DsfcixqeiMLecknM997nNccMEF3HXXXXzmM5/p2L506dKOpYLPP/98rrvuul6fa8OGDXz3u99l48aNfPzxx4wdOxagx2WHx44dy6RJkwCYOHEi06ZNw8yYNGkSLS0teb2WNWvWsHLlSk488UQAdu3axciRI9m6dStbt27l2GOP7Xg9TzzxRF7PLSLhSkXpptyTcK688krmzp3L+++/X9TzXH755Vx22WWsWLGCn//853z00Ue9/k7XpY47L4Oc77LH7s7EiRNZvnw5y5cvZ8WKFTz55JP5vQgRSZxUJPpyT8IZNmwY55xzzm5XcvrqV7/acXnB+fPn87WvfQ3Yc6nhzrZt28b+++8PwLx58zq2l3LZ4Z6WPp4wYQJtbW0sXboUgE8++YRXX32VoUOHMnToUJ5//vmO1yPJohVBpScFJ3ozG2hmfzazv5nZq2Y2O9o+1sxeMrM3zOxBM9u7dOHmVolJONdcc81uo2/uvvtu7r33Xg477DB+9atfceeddwKZK1HdfvvtHH744Xt0xjY1NXH22WczZcoU9t13347tpVx2uOv+L7zwQi655BLq6+vZtWsXCxcu5Prrr2fy5MnU19d3XIz83nvv5dJLL6W+vj7R16CtRuUajCDpUfAyxWZmwGB3325mewHPA1cAVwOL3P0BM7sH+Ju7/6yn59Iyxcmhv0t4li6t6+Y6pmNobGypfEBSMWVfpjjq9N0e3d0r+nLgeGBhtH0ecEah+xCR3mlFUOlNUTV6M+tvZsuBzcBTwN+Bre6e7SXcAOxfXIgi0hOtCCq9KSrRu/sud68HRgFHAQf19XfNbKaZNZtZc1tbWzFhiFQ1rQgqvSnJqBt33wo8AzQCQ80sOz5/FPB2N78zx90b3L1hxIgR3T1vKcKTEtHfI0xaEVR6U/CEKTMbAXzi7lvN7DPAicBtZBL+WcADwAzg0UKef+DAgWzZsoXhw4eT6feVOLk7W7ZsYeDAgXGHIjloRVDpSTEzY0cC88ysP5lPBr9298fNbBXwgJn9H+CvwNyenqQ7o0aNYsOGDaisE46BAwcyatSouMMQkTwVnOjd/RXg8Bzb15Kp1xdlr7326lgiQERECpeKmbEiItI9JXoRkZRTohcRSbmCl0AoaRBmbcCec7j7Zl+g75eAil+S4k1SrKB4yylJsUKy4i0m1jHunnt8eidBJPpimFlzX9Z6CEWS4k1SrKB4yylJsUKy4q1ErCrdiIiknBK9iEjKpSHRz4k7gDwlKd4kxQqKt5ySFCskK96yx5r4Gr2IiPQsDS16ERHpQaISfUiXL+yraM3+v5rZ49H9kGNtMbMVZrZW8sXAAAAD8UlEQVTczJqjbcPM7Ckzez36vk/ccQKY2VAzW2hmr5nZajNrDDjWCdExzX7908yuDDVeADO7KvofW2lmC6L/vSDfu2Z2RRTnq2Z2ZbQtmGNrZv9tZpvNbGWnbTnjs4y7omP8ipkdUYoYEpXogR3A8e4+GagHTjGzr5BZNfMOdz8AeA+4OMYYu7oCWN3pfsixAnzd3es7Dfe6AVjs7uOBxdH9ENwJ/N7dDwImkznGQcbq7muiY1oPTAE+AB4m0HjNbH/gP4EGdz8U6A+cS4DvXTM7FPgPMutrTQZONbMDCOvY3gec0mVbd/F9Axgffc0EerwMa5+5eyK/gEHAy8CXyUw2GBBtbwT+X9zxRbGMiv6IxwOPAxZqrFE8LcC+XbatAUZGt0cCawKI8/PAm0R9TCHHmiP2k4AXQo6XzFXh3gKGkVn48HHg5BDfu8DZwNxO9/83cF1oxxaoA1Z2up8zPuDnwHm5HlfMV9Ja9Em7fOF/kXnTtUf3hxNurJC55u+TZrbMzGZG22rdfWN0exNQG09ouxkLtAH3RmWxX5rZYMKMtatzgQXR7SDjdfe3gR8D64GNwDZgGWG+d1cCXzOz4WY2CPgm8EUCPbaddBdf9iSbVZLjnLhE70VcvrCSzOxUYLO7L4s7ljwc4+5HkPn4eKmZHdv5h55pYoQwTGsAcATwM3c/HHifLh/NA4q1Q1TTPg14qOvPQoo3qhefTuaEuh8wmD1LD0Fw99VkSkpPAr8HlgO7ujwmmGObSyXiS1yiz/ICLl9YYUcDp5lZC5mrbR1Ppq4cYqxAR0sOd99MpoZ8FNBqZiMBou+b44uwwwZgg7u/FN1fSCbxhxhrZ98AXnb31uh+qPGeALzp7m3u/gmwiMz7Ocj3rrvPdfcp7n4smb6D/yHcY5vVXXxvk/lEklWS45yoRG9mI8xsaHQ7e/nC1Xx6+UIo4vKFpeTuN7r7KHevI/Nx/Q/uPp0AYwUws8FmNiR7m0wteSXwGJk4IZB43X0T8JaZTYg2TQNWEWCsXZzHp2UbCDfe9cBXzGyQmRmfHt9Q37v/Fn0fDfw7cD/hHtus7uJ7DLggGn3zFWBbpxJP4eLsoCigQ+MwMpcnfIVMEro52j4O+DPwBpmPxTVxx9ol7qnA4yHHGsX1t+jrVeCmaPtwMh3KrwNPA8PijjWKqx5ojt4LjwD7hBprFO9gYAvw+U7bQo53NvBa9H/2K6Am4Pfuc2RORH8DpoV2bMmc3DcCn5D5NHpxd/GRGbDxUzJ9jyvIjHwqOgbNjBURSblElW5ERCR/SvQiIimnRC8iknJK9CIiKadELyKSckr0IiIpp0QvIpJySvQiIin3/wGmIbjdghZsigAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Find indices of positive and negative examples\n", "pos = np.where(y==1)[0]\n", "neg = np.where(y==0)[0]\n", "\n", "# Plot examples\n", "plt.plot(X[pos, 1], X[pos, 2], 'b+', label='Admitted')\n", "plt.plot(X[neg, 1], X[neg, 2], 'yo', label='Not admitted')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Sigmoid\n", "The sigmoid function, or logistic function, is a function that asymptotes at 0 and 1. The value at 0 is $\\frac{1}{2}$.\n", "\n", "$h_\\theta(x) = g(\\theta^Tx) = g(z) = \\frac{1}{1+ e^{-z}} = \\frac{1}{1+ e^{-\\theta^Tx}}$\n", "\n", "A plot of the sigmoid function:\n", "![sigmoid](notes/sigmoid.png)\n", "\n", "We are going to use the sigmoid function to predict how likely it is that a given data point is in category 0. Our hypothesis:\n", "\n", "$h_\\theta(x) = P(y = 0|x;\\theta)$\n", "\n", "Because there are only two categories (in this case), we can derrive that:\n", "\n", "$P(y = 0|x;\\theta) + P(y = 1|x;\\theta)= 1$\n", "\n", "**Exercise**: Implement the sigmoid function in Python." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def sigmoid(z):\n", " return z" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Compute the Cost and Gradient\n", "\n", "#### The cost function\n", "\n", "The cost function in logistic regression differs from the one used in linear regression. The cost function in logistic regression:\n", "\n", "$$J(\\theta) = - \\begin{bmatrix}\\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}-y^{(i)}\\log h(x^{(i)}-(1-y^{(i)})\\log(1-h_\\theta(x^{(i)}))\\end{bmatrix}$$\n", "\n", "Assume our hypothesis for an example is wrong, the higher probability $h_\\theta$ had predicted, the higher the penatly.\n", "\n", "A vectorized version of the cost function:\n", "\n", "$$J(\\theta) = \\frac{1}{m} ⋅(−y^T \\log(h)−(1−y)^T \\log(1−h))$$\n", "\n", "**Exercise**: Implement the vectorized cost function in Python." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def compute_cost(theta, X, y):\n", " return 0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### The gradient\n", "\n", "The gradient is the step a minimization algorithm, like gradient descent, takes to get to the (local) minimum. Note that this step can be taken in a higher dimension and hence the gradient is a vector. In the previous programming exercise we used gradient descent. This time, we are going to use an algorithm called conjugate gradient to find the minimum. How that algorithm works is beyond the scope if this course. If you are interested, you can learn more about it [here](https://en.wikipedia.org/wiki/Conjugate_gradient_method).\n", "\n", "The partial derrivative or $J(\\theta)$:\n", "\n", "$$\\frac{\\delta}{\\delta\\theta_J} = \\frac{1}{m}\\displaystyle\\sum_{i = 1}^{m} \\begin{bmatrix}(h_\\theta(x^{(i)}) - y^{(i)}\\end{bmatrix}x_j^{(i)}$$\n", "\n", "Vectorized:\n", "\n", "$$\\frac{\\delta}{\\delta\\theta_J} = \\frac{1}{m} \\cdot X^T \\cdot (g(X\\cdot\\theta)-\\vec{y})$$\n", "\n", "**Exercise**: Write a function to compute the gradient." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def compute_gradient(theta, X, y):\n", " return np.zeros(len(theta))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Test the cost and gradient function. We expect $J \\approx 0.693$ and $\\frac{\\delta}{\\delta\\theta_J} \\approx \\begin{bmatrix}-0.1000 & -12.0092 & -11.2628 \\end{bmatrix}$" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cost: \n", "0.6931471805599453\n", "\n", "Gradient: \n", "[ -0.1 -12.00921659 -11.26284221]\n" ] } ], "source": [ "initial_theta = np.zeros(n)\n", "y = y.reshape(m)\n", "print('Cost: \\n{}\\n'.format(compute_cost(initial_theta, X, y)))\n", "print('Gradient: \\n{}'.format(compute_gradient(initial_theta, X, y)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Learning $\\theta$ using conjugate gradient\n", "\n", "The optimization library we are going to use is `scipy.optimize`. We have to provide the algorithm with our cost function, initial guess, gradient among with some other (optional) configuration options.\n", "\n", "Please scan [the docs](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html) of `minimize` first." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Optimization terminated successfully.\n", " Current function value: 0.203498\n", " Iterations: 42\n", " Function evaluations: 100\n", " Gradient evaluations: 100\n", "Conjugate gradient found the following values for theta: [-25.15522426 0.20618292 0.20142209]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:3: RuntimeWarning: divide by zero encountered in log\n", " This is separate from the ipykernel package so we can avoid doing imports until\n" ] } ], "source": [ "from scipy.optimize import minimize\n", "result = minimize(compute_cost, initial_theta, args = (X, y),\n", " method = 'CG', jac = compute_gradient, \n", " options = {\"maxiter\": 400, \"disp\" : 1})\n", "theta = result.x\n", "print('Conjugate gradient found the following values for theta: {}'.format(theta))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The decision boundary\n", "The decision boundary is a line (in case of a 2d plane) that seperates the area where we predict $y=1$ and $y=0$." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xd4VFX6wPHvm0JCLyGEThLpJBAg9BKkKyhFRZGlWVhWURB0rUtxxbIKKOiKKIvoTxFlaaJIJ1SBoJFAIIAQakgiGDoh5fz+yCQbIH3KvTNzPs+TJzM3M/e+czPzzrnvOfdcUUqhaZqmuS4PowPQNE3T7Esnek3TNBenE72maZqL04le0zTNxelEr2ma5uJ0otc0TXNxOtFrmqa5OJ3oNU3TXJxO9JqmaS7Oy+gAAKpWraoCAwONDkPTNM2p7N279w+llH9hjzNFog8MDCQqKsroMDRN05yKiJwoyuN06UbTNM3F6USvaZrm4gpN9CLyHxFJEpH9uZZVEZF1InLE8ruyZbmIyGwROSoi+0SklT2D1zRN0wpXlBb950Df25a9BGxQSjUANljuA9wDNLD8jAE+tk2YmqZpWkkVmuiVUluAC7ctHgAstNxeCAzMtfwLleVnoJKI1LBVsJqmaVrxlbRGH6CUSrDcPgcEWG7XAk7letxpy7I7iMgYEYkSkajk5OQShmE/iYlfsXNnIJs3e7BzZyCJiV8ZHZKmaVqJWN0Zq7IuUVXsy1QppeYppcKVUuH+/oUOA3WoxMSviIsbQ2rqCUCRmnqCuLgxOtlrmuaUSproE7NLMpbfSZblZ4A6uR5X27LMqRw79iqZmdduWZaZeY1jx141KCJN07SSK2miXwmMtNweCazItXyEZfRNe+BirhKP00hNPVms5ZpWFFOnGh2B5q6KMrxyEbATaCQip0XkceBtoJeIHAF6Wu4D/AgcA44CnwJP2SVqO/PxqVus5baiE4FrmzbN6Ag0dyVZJXZjhYeHKzNNgZBdo89dvvHwKEOjRvMICBhmt+2KgAn+HZqd6P+vZmsislcpFV7Y4/SZsXkICBhGo0bz8PGpBwg+PvXsnuQ1+7n9SMmRR05Tp2YleJGs+9m3zXT0ZqZYNPvQLXqDTZ2a9yH9lCn6A2grt7ekjWpZF3e7U6c65j2gjzScV1Fb9DrRm4j+wNmHsyZ6R8Wp33fOyy1KN7uPX2Dsl3s5d/GGVevRJ0e5nvxKJkaWUKZMcdy2CuMMJSXNdpw60Z84f5XNh5PoOTOSz7cfJyOz+M0SM50cZaZE4OymTs1qpWa3VLNv337f0fX6ojzGEQk4v/2jE71rcvrSzcnz13h1eQxbj/xBi9oVeXNwKM1qVizy83fuDLQk+Vv5+NSjQ4f4EsWkmYtZSjfFpUs3WmHconQDUNevDF881pbZQ1tyJuU693+4nTd/PMi1m+lFer4+Ocr13X6kpI+cbqX3h+tz+hZ9bhevpfH2TwdZtPsUtSqV5o2BIdzduFqBz9Etes2sHDXqRnNebtOiz61iGW/eGtyc78Z2oEwpT0Z/voenv/qFpEv5d9YGB0/Hw6PMLcs8PMoQHDzd3uFqWoF0ktdsxaUSfbY2gVX44dkuPN+7IesOJtJjRiRf/nyCzDw6a/XJUZqRdDLXHMGlSjd5if/jKq8uj2H70fO0rFuJtwaH0rh6BbtsS9OKS3eEatZwy9JNXgKrluX/Hm/HzCEtOHH+Gv1nb+Odnw5x/WaG0aFpmunpIw7X4PKJHkBEGNyqNhsmRjC4VS0+3vw7vd+PJPKw+a5spbk+ZzpZSc+46RpcvnSTl5+PneeVZTEcS77K/S1q8o/+TfEv7+Ow7WtaNrOXbswen7vTpZsCtA/2Y/X4Lkzo2YCf9p+jx4zNLNp9Ms/OWk1zN850xKEVjVu26HM7mnSFV5fFsOv4BdoEVubNQaE0CChvSCya+zH7WHndojc3h7ToRWS8iOwXkQMiMsGyrIqIrBORI5bfla3Zhr3Vr1aOb8a0518PNudI0hXunb2VGWvjuJGmO2s1+7PHHDaadrsSJ3oRCQGeBNoCLYD+IlIfeAnYoJRqAGyw3Dc1EWFIeB02TIzgvuY1mbPxKH3f38L2o38YHZqmFYstOk916cb1WNOibwLsUkpdU0qlA5HAYGAAsNDymIXAQOtCdBy/cj7MfDiMr55oB8Cwz3Yx8dtozl9JNTgyTcufntlSK4w1iX4/0EVE/ESkDHAvUAcIUEolWB5zDgiwMkaH61S/Kj9N6Moz3evz/W9n6TEzkm+jTmGG/gxnphOFfUybplvgWsFKnOiVUgeBd4C1wE9ANJBx22MUkGd2FJExIhIlIlHJyeYbz+7r7cmk3o348dku1Pcvx9+X7OOReT/ze/IVo0NzWnpMtv3YqwXurDNb6i+5W9ls1I2IvAmcBsYD3ZRSCSJSA9islGpU0HPNfinBzEzF4qhTvPXjQW6kZfLU3Xfxt2534ePlaXRoVnPkqA89gsN28rvWMOh9DO7zXnPUqJtqlt91yarPfw2sBEZaHjISWGHNNszAw0MY2rYuGyZ1o29Idd5ff4R7PtjKz8fOGx2a1ezdytYde/aRXx3dWVvgmn1Z1aIXka2AH5AGTFRKbRARP+BboC5wAhiilLpQ0HrM3qK/XeThZF5bHsOpC9d5qHVtXrm3CZXLljI6rBJxZMvHXVpZjpbffjX7GH1byn6deTVcpkxx3f1Q1Ba9258wVVLXb2Ywe+MRPt1yjAqlvflH/yYMDKuFZDdd7cBWH9z8Dvvt/YHQid4+8ntfuNP+dtbLRVpLJ3oHOXTuEi8vjeHXkyl0qu/HGwNDCapa1i7bsseb15EfCHdqYZqBuyQ7MC7RG/2e1nPdOEjj6hX479iO/HNgCPtOXaTP+1v4cOMRbqZnGh2a6TjqA+HOXybu1CdS0Gt1VF+Fs4wk04neBjw8hOHt67FhUgS9mgTw3trD9Ju9lT3xBXZNFIm9P7iu2HnnLB8+e3GXk50KOrHLFV+vNXTpxg42HkrkH8sPcCblOkPb1uWlvo2pWMbb6vW606G4Ncyyn4xIOLlfu1n2gyM4ugRplk5fXboxUPfGAayb2JUnuwTxbdQpesyMZOVvZ/WZtXZkxpKF0UcWrni0lh9HvlZnnCJCJ3o7KVPKi1f7NWXF052oVcmXZxf9ysgFezh14VqJ1+lOH9zicsYPny1kHzXk9SXnTlz9/2wtXbpxgIxMxZc743l3TRwZSjG+R0Oe6BKEt6f+nrUHI0sWjj6sd9dhhWZhdH+AHl5pQgkXrzN15QHWHEikcfXyvDk4lFZ1TT1dv1My+sOXzRFJVyd696Zr9CZUo2JpPhkezrzhrbl4PY0HPt7Ba8tjuHQjzejQXIoZkrw9mWFYoeZcdIveIFdS05mxNo6FO+KpWs6Hqfc3456Q6nY9s1ZzLEccWegWvHvTpRsnse90Ci8vjeHA2Ut0b1yN1wc0o3blMkaHpTkJnejdmy7dOInmtSux4ulOvNavCT8fO0+vmVv4dMsx0jP0mbVa4XSpRisK3aI3kdN/XmPKigNsOJRE0xoVeGtwKC3qVDI6LE3TTEq36J1Q7cpl+GxkOB8Pa8UfV1IZ9O/tTF15gCup6UaHpmmaE9OJ3mREhHtCa7B+UgR/aV+PhTvj6TkjkjUHzhkdmqZpTkonepOq4OvN6wNCWPq3jlQq481fv9zLk19EcTblutGhuTxXH56puR9rrzD1HPAEWRcAjwFGAzWAb8i68tReYLhS6mZB69E1+oKlZWTyn23HmbX+MJ4iTOrdiJEdA/H00EMx7UGPZNGchd1r9CJSC3gWCFdKhQCewCPAO8AspVR94E/g8ZJuQ8vi7enBXyPuYt1zEYQHVuH1VbEM+vd29p+5aHRomqY5AWtLN15AaRHxAsoACUB3YInl7wuBgVZuQ7OoU6UMn49uw5yhLTmbcoP7P9zGG6tiuao7a61mxtkvNc1WrC3djAemA9eBtcB44GdLax4RqQOstrT486VLN8V38Xoa7/x0iK93naRWpdK8PqAZPZoEGB2WS9ClG81ZOKJ0UxkYAAQBNYGyQN9iPH+MiESJSFRycnJJw3B5iYlfsXNnIJs3e7BzZyCJiV8BULG0N28OCuW/f+tAWR9PHl8YxVNf7SXx0g2DI9Y0zWysKd30BI4rpZKVUmnAUqATUMlSygGoDZzJ68lKqXlKqXClVLi/v78VYbiuxMSviIsbQ2rqCUCRmnqCuLgxOckeoHW9Kqx6pgsv9GnEhoNJ9JwRyZc748nI1E3SkjL6bFNdLtJszZpEfxJoLyJlJGsmrh5ALLAJeNDymJHACutCNF5+rWp7O3bsVTIzb71QSWbmNY4de/WWZaW8PHj67vqsmdCVsLqV+MeKAzzw8Q4OJlxySJyabRl9ZSrN9Vhbo58GPAykA7+SNdSyFlnDK6tYlv1FKZVa0HrMXKPPblXnTrgeHmVo1GgeAQHD7LrtzZs9yBq5ejuhW7e858JRSrEi+iz/XBVLyvU0nugSxIQeDSldytOusboSo2v0Rm9fcx4OmQJBKTVFKdVYKRWilBqulEpVSh1TSrVVStVXSj1UWJI3u6K2qu3Bx6dusZZD1pm1A1vWYsOkCB5sVZtPIo/Ra1Ykm+OS7BWmZgN61I9x3GEf6zNjC5GaerJYy20pOHg6Hh63Tlns4VGG4ODphT63UplSvPNgcxaPaY+PlwejFuzhmUW/knRZd9bmxehE667XvDUDdyiV6URfiJK0qm0lIGAYjRrNw8enHiD4+NQrdsmoXbAfP47vwsReDVmz/xw9Z0Ty9a6TZOrO2lvoRFsyev84B53oC2FNq7ogRe3gDQgYRocO8XTrlkmHDvEl6hfw8fLk2R4N+GlCF5rVrMgry2IY8slODidetuo1GM1Vk4zRo36Kw1lbw0YfwTmano++CBITv+LYsVdJTT2Jj09dgoOnW9URa2QHr1KK//5yhuk/xHL5Rjp/jQjmme4N8PUuemetq1982yyvzxm4QsexM78GPR+9jdg6yYOxHbwiwoOta7NhUjcGhNXio02/0+f9LWw78keR1+GsrbiictckX9TX7W6tYVegE30BinLCUkkY2cGbrUrZUswY0oKvn2yHhwh/mb+L5xZH88cVcw+S0knGfor6Be5q/RnOVCorKZ3oC2CvlreRHby363hXVVaP78Kz3euzat9Zes6M5Ns9p7i9pGeWBOtqSUYznju8d3SiL4C9Wt726uAtKV9vTyb2bsTq8V1oWK08f//vPh6e9zNHk67kPEYnWNeU3xd4t25Fe747tIZdgU70BbBXy9sWwybtoX618nwzpj3vPBBK3LnL3PPBFmauO8yNtAxD48qPTjLWy+8LPDKy6M/XzE+PuimAkaNjjPbHlVTeWBXL8uizBFctyxuDQuh4V1XAOUalOEOMZpN79Ikzj0RxJ3rUjQ0U1PI2aqIzR6lazof3H2nJF4+1JT1T8einu3j+u9+4cPWmUyTQ2zsWnSFmo0VEmKMfRrM93aIvAXdr6d9Iy2D2hiPM23KM8r5evNavKYNb1UKyM4IJ3d4i1S3U4tH7yzbsfWSpW/R2ZOQ4eCP4envy976N+eHZLgT7l2PSd78x7LNdHP/jqtGh3cIsI4M0LZtZzjnRib4EzDAO3giNqpfnu792YPqgEGLOXKTP+1uYveEIqenm6Ky9vWMxu7M2+8OmE3/R6Y5u16ITfQnYejSOM9X7PTyEYe3qsWFiBL2aBjBz3WH6zd7G7uMXjA7tDnpIaMnpfVRyZjyy1Im+BGw5Dt5eZ9/aW7UKvnz0aCsWjGrD9ZsZDPlkJy/9dx8p124aHRqgW6SacczYwNCJvgRsOQ7e2ev9dzeuxrqJXflr12C+23uanjMjWRF95o4zax3t9g+VTvy6le7OSjzqRkQaAYtzLQoGJgNfWJYHAvHAEKXUnwWty9lG3dhSSS4XaFaxZy/x8rIYfjuVQpcGVXljYAj1/MoaHdYd3HWMvR5J43hOP+pGKRWnlApTSoUBrYFrwDLgJWCDUqoBsMFyX8uHmea9sVbTmhVY+reOvD6gGb+eTKH3rC18tOkoaRnm+sIyy0gIa7nbl5Uzvl6zxGyr0k0P4Hel1AlgALDQsnwhMNBG23BJZpv3xhpTp4KnhzCiQyDrJ0bQvXE13l0TR//Z29h7osCDOq0EivKFZcaOwZJylS9oI9gq0T8CLLLcDlBKJVhunwMC8nqCiIwRkSgRiUpOTrZRGM7HrPPelETuD2L1ir58/JfWfDYinMs30nhw7g5eXRbDxetphsTmSgmvOPLrGNSsV9B7x2zvK6vPjBWRUsBZoJlSKlFEUpRSlXL9/U+lVOWC1uHONXpXkl8N+GpqOjPXHWbB9uP4lfNhyn1N6Rdaw7Aza525Vj11at4t2ylTCk8uzjiXjTWv194K2oeO2r+OPDP2HuAXpVSi5X6iiNSwBFEDSLLBNjSTKkpLuayPF//o35SV4zpTvYIv477+lcc+38OpC9fyWqVWAGuG7jnjyCMzDlV0RrZI9EP5X9kGYCUw0nJ7JLDCBtvQTKo4H8SQWhVZ9lRH/tG/KbuOX6D3rC3M2/I76Q7urHXGhGcr7li+sqWCGjZmLg9aVboRkbLASSBYKXXRsswP+BaoC5wga3hlgadN6tKNayjO4erZlOtMXnGA9QcTaVKjAm8NDiWsTqXCn6jlsGbonrOUbnIz27BYZyrd6NkrNZsp7gdRKcWaA4lMXXmAxMs3GNG+Hs/3aUR5X297hahZOGOiNxtnSvT6zFgXYvScOcVtbYkIfUOqs25iV0Z2COSLn0/Qc2YkP+1PMPzM2uIyU0uzKNy5fGUrBe1Ds+1f3aJ3Ea4wR370qRReXhrDwYRL9GwSwLQBzahVqbTRYRWJbiFrRtAtejfj7HPmAITVqcT34zrx6r1N2H70D3rNjGT+tuMO76zVnIMtj6Kc7YisuHSidxGuMke+l6cHT3YNZu1zXWkXVIV/ropl4L+3E3P6otGh3cHMoyzcgS3PlHX1s251oncge9bQnXnOnLwSY50qZfjPqDZ89GgrEi+lMuCjbbz+fSxXU9MdHl9+9BhvzVnoRO8g9p533pnnzMmvNSUi9Gteg/UTI3i0XV0W7DhOr5mRrItNzPsJmsuz5VGUOx2R6c5YB9m5M9CS5G/l41OPDh3ibbKNxMSvOHbsVVJTT+LjU5fg4OlO0RFb1I7MvSf+5JWlMcQlXqZvs+pMvb8Z1Sv62j/AIjDbGG93YMsOcGftTNfj6E3Gleadt4WSzmGSlpHJp1uP8cH6I3h7evBCn0b8pX09PD2MmTdHM45O9HrUjek4cw3dHkpa3/b29OCpbvVZ91wELetWYsrKAwz+eAcHzpqvs1azL1uOVTfbuHdb0y16B3GFce4FsaZsVNLWlFKKlb+d5Z+rYvnzWhqPdw5iQs8GlCnlVfyVaZoT0i16k3GleedvZ21Hc0lbUyLCgLBarJ8YwZDw2szbcoxeM7ew6ZCeMFXTctMtes1qjuhoLordxy/wyrIYjiZdoV/zGkzp35RqFczRWatp9qBb9JrDmOVkrbZBVfjx2S5M6tWQdbGJ9JgZyf/9fILMTOMbM5pmJJ3oNauZqaO5lJcHz/RowJoJXQmtVZHXlu/nwbk7iDt32eGxaJpZ6ESvWc2MJ2sFVS3LV0+0Y+aQFsSfv0a/2Vv510+HuJGWYdPt6LHzWl7M9r7Qib4IjJ7+1+yyO5o9Pf1ylnl4GD/rpIgwuFVt1k+MYGDLWvx78+/0nrWFLYdtdzF6V58jRSsZs70vrEr0IlJJRJaIyCEROSgiHUSkioisE5Ejlt8FXhjc7Ow9dYErUep6zu309POm2U9VypbivYdasOjJ9nh5CCP+s5vx3/xK8uVUo0PTNIewtkX/AfCTUqox0AI4CLwEbFBKNQA2WO47LVeY/rekinMk4wz7qcNdfqye0IXxPRqwOuYcPWdG8s3uk8XurHWnOVK0ojPz+6LEwytFpCIQTdb1YlWu5XFAN6VUgojUADYrpRoVtC4zD69016kLinuClz33kz3mkTmadIVXl8Ww6/gF2gZW4c3BIdSvVr7Y63HWU+c1+3KlSwkGAcnAAhH5VUQ+s1wsPEAplWB5zDkgIJ8Ax4hIlIhEJSfbrmZqa2YaUeJIxW2h23M/2aPeWb9aOb4Z055/PdCcuMTL3PPBVmaujbN5Z62mmYE1id4LaAV8rJRqCVzltjKNpaWf5/eaUmqeUipcKRXu7+9vRRj2ZcYRJY5Q3LHxzrifRIQhbeqwYVIE9zWvyeyNR7nng63sOPpHkdfh6nOkaCVjtveFNYn+NHBaKbXLcn8JWYk/0VKywfLbqc9Hd+WpCwpS3Ba6rfeTI+udVcv5MPPhMP7v8XZkKsWjn+1i4rfRXLh6s0hxatrtzPa+sGoKBBHZCjyhlIoTkalAWcufziul3haRl4AqSqm/F7QeM9fo3ZWZJmFzZB38RloGH248ytzI3ynv68Ur9zbhwda1EdHTIGvm45D56EUkDPgMKAUcA0aTdZTwLVAXOAEMUUpdKGg9OtGbk1kuZGJEh+fhxMu8sjSGqBN/0j64CtMHhXKXfznHBqFphdAXHtFchlFXb8rMVCyOOsVbPx7kRlomT99dn7HdgvHx8nR8MC5EX43LdnSi1zQbSbp8g3+uOsj3v53lLv+yvDkolHbBfoU/UcuTHpJqO3r2Sjemp2ywrWrlfZkztCWfj27DzYxMHp73My8u2UfKtcI7a92ZbrWbh070LkZP2WA/3RpVY+2ECMZG3MWSX07TY0Yky349jRmOis0o9/kPZj5r1B3o0o2LMctFQFzdwYRLvLw0huhTKXSuX5U3BoYQWLVs4U90I/mVaHTpxnZ06cZNmeUiIK6uSY0K/PdvHfnngGb8diqFPu9v4aNNR7mZ7rrTYhSFbrmbk070LsZdp2wwgqeHMLxDIOsnRdCzSQDvromj/5ytRMUXOJrYpU2dmtVaz26xZ9/OnejNdtaoO9CJ3sU441QEzi6ggi8fDWvF/JHhXE3N4MG5O3l5aQwXr6UZHZop6da94+lE72LcdcoGM+jRJIC1z3XlyS5BLN5zkh4zI/n+t7Nu21mrW+7moTtjNc0O9p+5yCvLYth3+iIRDf15Y2AIdaqUKfyJmlYMujNW0wwUUqsiy57qxJT7mhIVf4FesyKZG/k7aRnu3VmrGUMnek2zE08PYXSnINZPiqBahj9vrz7EfXO28evJP40OTXMzOtFrDuWOZ+3WqFiarW+F88nw1qRcS2PwxzuYvGI/l27ozlrNMXSi1+5gr2Rs9Fm7Rn/J9GlWnfWTIhjVMZD/+/kEPWdE8mNMgtt21mqOoxO9dgt7JmMjLyBuxJdMXicPlff1Qu1txvKnO+Ff3oenvvqFJxZGcSblut3i0DQ96ka7hT2nUDDyQutGTw2R12n/6RmZfL4jnhlrDyMCE3s1ZFTHQLw8dftLKxo96kYrEXtOoWDkWbtmnBrCy9ODJ7oEs25iV9oH+/HGDwcZ8NF29p1OMSwmzTVZlehFJF5EYkQkWkSiLMuqiMg6ETli+V3ZNqFqjmDPZGzkWbteXlXyXO6oqSEKOnmoduUyzB8ZzsfDWpF8OZWBH21n2vcHuJKa7pDYNNdnixb93UqpsFyHDy8BG5RSDYANlvuak8gvGfv53Wt1R6ZRZ+0mJn5FevqlO5aLlHLY1BCFnfYvItwTWoP1kyIY1q4en++Ip9fMSNYeOOeQ+DTXZu01Y+OBcKXUH7mWxQHdlFIJIlID2KyUalTQeuxVozfLNU+dze37zc/vXs6dW2iKC4WXRH71eU9PP7p0+SOPZxjvl5N/8srSGA6du0zvpgFMG9CMGhVLGx2WZjKOujj4ceBPsnrYPlFKzRORFKVUJcvfBfgz+35+7JHos0dZOGtyMhOjOzKtZWQnsDXSMjKZv+04768/jKcIz/dpxIgOgXh6iNGhaSbhqM7YzkqpVsA9wNMi0jX3H1XWt0ie3yQiMkZEokQkKjk52cow7mTkUD5Xk39H5p3J34ycdepmb08PxkbcxbrnIggPrMK072MZ9O/t7D9z0ejQNCdjVaJXSp2x/E4ClgFtgURLyQbL76R8njtPKRWulAr39/e3Jow8mXGUhbPKPyGKU5zZ6uxTN9epUobPR7dh9tCWnE25wYCPtjP9h1iumrSzVk9DbD4lTvQiUlZEymffBnoD+4GVwEjLw0YCK6wNsiSctRVnRlkJMa9ygXKKIyRXmLpZRLi/RU02TIxgSHgdPt16nN6ztrDxUKLRod0h97ViNXMocY1eRILJasUDeAFfK6Wmi4gf8C1QFzgBDFFKFXjJHV2jN7/Nm/OrC5u7zu2qouIv8PLSGI4kXeHe0OpMua8ZARV883381KmOa2nra8I6jt1r9EqpY0qpFpafZkqp6Zbl55VSPZRSDZRSPQtL8vZir1ac0fOlGCVrP+a1XB8hGSE8sAo/PNuFF/o0Yv3BJHrOiOTLnfFkZuadYe3dytbXijU3PQVCMbjzUUJerx288fKqQHr6BT18FeOG88b/cZWhM2NIyDxPy7qVeHNQKE1qVLjlMY5sZesWvePoKRDswJ1H8tx+hOTp6YeIkJ5+HiNmojQbI2fmDKxalp/fasesh1tw4vw17puzjbdXH+K1KRm6la0BOtEXi7uP5AkIGEaHDvE0afIlGRkpKHXzlr+7y5deXoxvBAiDWtZmw8QIBreqxdzI34ksG8nmuOSc1rVSWT/2TvT6WrHmoxN9MeiRPP9ruUJGnn93ly+92xnRCMirLl6lXCnK7G/BN2Pa4+3pwcj/7ObZRb/iUfaG3eLIKy5n3Y6rHu3oGn0xuHONPlt+Z8lmc5azZW3N6LOH86qLp6ZnMHfzMT7adBQyPJg6qAmPtKmDh4ucWWuPvgBn61/QNXo7cIXx2NYqqIXqTCch2ZoZT8ry8fJkfM8GrJ7QhVbBFXhlWQxDPtnJ4cTLRXq+q7Zu3ZFO9IW4fTglQIcO8XTrlkmHDvFuleShoDKVp9t96eVmdCOgoLr4Xf7lWPRke959sDlHk6/Qb/ZW3lsTx420vMtv2cx44pM9hnG6w9BQly7dWDsHF+jIAAAegElEQVTcTZdq7qT3iXM7fyWV6T8eZOkvZwj0K8MbA0Pp3KBqno81exlDl2506cYmw92MH0lhPka3XDXr+JXzYeaQML5+oh0iwl/m7+K5xdGcv5IK2K/FbG+u1Pq2B5dt0Re3cyyv1v/Bg8NxxultNa0obqRl8O9NR/k48nfK+njxyr1NeKh1bcSS5W3VurVXKzn3tA622oYjp4qwBYfMR28r9kj0xZmDPL9yhEhpMjLO37EGdx1Zot3JFS5ucyTxMq8si2FP/J+0C6rC9EGh1K9WrsDkWZzX7YhyiLOVXGzF7Us3xRnznl+JRgTTjaTQCubIuYiMPBvWlhoElGfxmA68PTiUgwmXuPeDrcxad5hXJ+fdWVuU1+2IDk536ES1FZdt0Ren07Cg1n+TJl86fYvNXTi6o9josfP2kHw5lTd+iGVF9FmCq5Zl+qBQOtzld8tjivu6dYvefty+RV+cTsOCWv/Zp/2763BKZ+LoznOjp8Swx9GLf3kfPnikJV881pb0TMXQT3/mhe9+48+r/5vuwujXrRWfl9EB2FNAwLAiJebg4Ol5tgR1ica5OCoBZden87lKpkOmxLj96CW7fALYpDHStaE/ayZ0ZfbGI3y65RgbDiXxWr8mDGpZCx+fuvm06PN+3Y6Y+0bPr1Mwl23RF4ceMugaHDEX0a316Ts5qoHgiKOX0qU8ebFvY1Y925lAvzJM/PY3/jJ/F16V30Kk1C2PFSmV7+vWwyuNZ3WiFxFPEflVRFZZ7geJyC4ROSoii+X2d4RJ6RKN83PENAR5JdhsjmwgOLJ80rh6BZaM7cgbA0PYd/oiw74sz4qjD5Ce+b+CgBn6+rT82aJFPx44mOv+O8AspVR94E/gcRtsQ9MK5Ygjs/wTqTi0geDomVQ9PIS/tK/HhokRtAr4haVHhjF5+2wO/9nU8og0tz6R0OysSvQiUhvoB3xmuS9Ad2CJ5SELgYHWbEPTisPeR2ZmmaraqEnUqlXwZWzzqTzXeiqpGT68uetf/Gf/M1xNK6s7Y03M2hb9+8DfgewzkPyAFKVUuuX+aaCWldvQ3IQzXI/XLLNUGtmv5ONTlxb+UbzZ+SnuCfov28705OWtc4lKHqRLOCZV4kQvIv2BJKXU3hI+f4yIRIlIVHJycknD0FyEs5x8ZKaOe6P6lbK/7Hy8Unm40QKmdJhA1dLn+XDvY4xcsIeT5/PuwzCaMzQk7KXEJ0yJyFvAcCAd8AUqAMuAPkB1pVS6iHQApiql+hS0Lme58IhmP6548pEru30KhHqB01l3vBPvrokjLSOT8T0b8GSXYLw9zTGwz1VnXXXoXDci0g14XinVX0S+A/6rlPpGROYC+5RS/y7o+TrRa8WZm0gzr4SL15m2MpafDpyjUUB53hwcSut6lY0OyyaTHJrxC8HIM2NfBCaKyFGyavbz7bANzcWYpZNTs06NiqWZO7w1n44I59KNNB6cu4PXlsdw8XqaoXEVZziqs5QRi8MmiV4ptVkp1d9y+5hSqq1Sqr5S6iGlVKottqG5NrN0cmq20atpAOsmRjC6YxBf7zpJz5mR/LAvwbDOWltMcujMw0fNUUDT3J6ZOjk12yjn48Xk+5qy4unOBFTw4emvf+Gxz/dw6oLjO2uL05Bwxbl8TDt7ZVpaGqdPn+bGjRsGRaUZydfXl9q1a+Pt7W10KJoNpGdksnDnCWasjUMpeK5XAx7rFISXAztri1p3d6aBAU5/4ZHjx49Tvnx5/Pz8cq54o7kHpRTnz5/n8uXLBAUFGR2OZkNnUq4zZcV+1h9MommNCrw1OJQWdSoZHdYtnGmEjtNPU3zjxg2d5F1cWtp5rlzZx+XLUVy5so+0tKyreYkIfn5++mjOBdWqVJpPR4Qz9y+tOH81lYH/3s7UlQe4fMPYztrcXLGMaOppinWSd11paee5ceME2SdVK3XTch+8vfUXvCsTEfqG1KBT/aq8tyaOhTvj+Wn/Oabe34y+IdWNDg8o+hTnzsK0LXqzWL58OSLCoUOH8vz7qFGjWLJkSZ5/y8vZs2d58MEHAYiOjubHH3/M+dvmzZvZsWNHsWMMDAzkjz/+KPbzjJSaeob/zZyRLdOyXHMH5X29mTYghGVPdaJy2VKM/b+9PPlFFGdTrhsdmstxuURv63mpFy1aROfOnVm0aJFN1lezZs2cLwZbJXpnpNTNYi3XXFdYnUqsHNeJl+9pzNYjyfSaGcl/th0nI9P4/kNX4XKJfto0263rypUrbNu2jfnz5/PNN98AWR2F48aNo1GjRvTs2ZOkpKScxwcGBvLyyy8TFhZGeHg4v/zyC3369OGuu+5i7ty5AMTHxxMSEsLNmzeZPHkyixcvJiwsjHfeeYe5c+cya9YswsLC2Lp1K8nJyTzwwAO0adOGNm3asH37dgDOnz9P7969adasGU888YRTTiSV32UKnOTyBZqNeXt68NeIu1j3XARtgqrw+qpYBn60nf1nLhodmkswdY3eaCtWrKBv3740bNgQPz8/9u7dy4kTJ4iLiyM2NpbExESaNm3KY489lvOcunXrEh0dzXPPPceoUaPYvn07N27cICQkhLFjx+Y8rlSpUrz++utERUXx4YcfAnD9+nXKlSvH888/D8Cjjz7Kc889R+fOnTl58iR9+vTh4MGDTJs2jc6dOzN58mR++OEH5s93vpOPfXxq3VKjz+KBj4+e7NSd1alShgWj2vBDTALTvo/l/g+3MbpTEBN7NaSsj05XJeUSe27q1Ftb8tn9eFOmWFfKWbRoEePHjwfgkUceYdGiRaSnpzN06FA8PT2pWbMm3bt3v+U5999/PwChoaFcuXKF8uXLU758eXx8fEhJSSnW9tevX09sbGzO/UuXLnHlyhW2bNnC0qVLAejXrx+VKxs/l0hxeXv7AVm1eqVuIlIKH59aOcs19yUi9G9eky4N/PnXT4eYv+04q2MSeH1ACD2bBhgdnlNymUSfndBFwBaVjAsXLrBx40ZiYmIQETIyMhARBg0aVODzfHx8APDw8Mi5nX0/PT09v6flKTMzk59//hlfX9/ivwAn4O3tpxO7lq+Kpb2ZPiiUwa1q8fLSGJ74Iop7Qqoz5b5mVK/omp8Je3G5Gr2tLFmyhOHDh3PixAni4+M5deoUQUFB+Pn5sXjxYjIyMkhISGDTpk0l3kb58uW5fPlyvvd79+7NnDlzcu5HR0cD0LVrV77++msAVq9ezZ9//lniGDTN7FrXq8KqZ7rw976N2HgoiZ4zI/liZ7zurC0Gl0v0U6bYZj2LFi26o/X+wAMPkJCQQIMGDWjatCkjRoygQ4cOJd7G3XffTWxsLGFhYSxevJj77ruPZcuW5XTGzp49m6ioKJo3b07Tpk1zOnSnTJnCli1baNasGUuXLqVuXT3Do+baSnl58FS3+qx9rist61Zi8ooDPPDxDmLPXjI6NKdg2ikQDh48SJMmTQyKSDMD/R7Q8qKUYuVvZ3n9+1hSrqfxROcgxvdsQJlSLlGJLhannwJB0zQtLyLCgLBabJgUwUOta/PJlmP0nrWFTXFJhT/ZTelEr2maU6pUphRvP9Ccb//aAV9vT0Yv2MO4r38h6bKeI+l2OtFrmhNz5wteZ2sbVIUfnu3MxF4NWRubSI8ZkXy16wSZurM2R4kTvYj4ishuEflNRA6IyDTL8iAR2SUiR0VksehTHTXNLlzxkncl5ePlybM9GvDT+C6E1KzIq8v289AnO4k7d7nwJ7sBa1r0qUB3pVQLIAzoKyLtgXeAWUqp+sCfwOPWh6lp2u1c8ZJ31gr2L8fXT7ZjxkMtOJZ8hX6zt/Kvnw5xIy3D6NAMVeJEr7Jcsdz1tvwooDuQPZ3jQmCgVRFqmpYnV7zknS2ICA+0rs2GSd0Y2LIW/978O71nbWHrkWSjQzOMVTV6EfEUkWggCVgH/A6kKKWyTwE9DeQ5eYmIjBGRKBGJSk425z9ARJg0aVLO/ffee4+phcypsHz58lumLSiJ4k47vHLlSt5+++08t//5559z9uzZYm0/e+I1zdyKc8Frd1SlbCnee6gFXz/ZDk8PYfj83Uz45lf+uJJqdGgOZ1WiV0plKKXCgNpAW6BxMZ47TykVrpQK9/f3tyYMwD6dUj4+PixdurRYSdcWib647r//fl566aU8t1+SRK85h+Jc8NqddbyrKqvHd+HZHg34ISaBHjMiWbznpFt11tpk1I1SKgXYBHQAKolI9pkLtQG7X0nCXp1SXl5ejBkzhlmzZt3xt/j4eLp3707z5s3p0aMHJ0+eZMeOHaxcuZIXXniBsLAwfv/991ue8/3339OuXTtatmxJz549SUxMBPKfdjg+Pp7GjRszatQoGjZsyLBhw1i/fj2dOnWiQYMG7N69G8hK5uPGjbtj+++88w5RUVEMGzaMsLAwrl+/zt69e4mIiKB169b06dOHhIQEAPbu3UuLFi1o0aIFH330kVX7TXMMV7zknb34ensysVdDVo/vSqPq5XnxvzE8Mu9njia5SWetUqpEP4A/UMlyuzSwFegPfAc8Ylk+F3iqsHW1bt1a3S42NvaOZfnZsaOe2rSJO3527KhX5HXkpWzZsurixYuqXr16KiUlRb377rtqypQpSiml+vfvrz7//HOllFLz589XAwYMUEopNXLkSPXdd9/lub4LFy6ozMxMpZRSn376qZo4caJSSqlnnnlGTZs2TSml1KpVqxSgkpOT1fHjx5Wnp6fat2+fysjIUK1atVKjR49WmZmZavny5TnbXLBggXr66afz3H5ERITas2ePUkqpmzdvqg4dOqikpCSllFLffPONGj16tFJKqdDQUBUZGamUUur5559XzZo1s2rf2UJx3gOaVlQZGZlq8e6TqvnUNar+Kz+oGWsOqes3040Oq0SAKFWEfG3NOcM1gIUi4knWkcG3SqlVIhILfCMibwC/AnafLN2enVIVKlRgxIgRzJ49m9KlS+cs37lzZ85UwcOHD+fvf/97oes6ffo0Dz/8MAkJCdy8eZOgoCCAAqcdDgoKIjQ0FIBmzZrRo0cPRITQ0FDi4+OL9Vri4uLYv38/vXr1AiAjI4MaNWqQkpJCSkoKXbt2zXk9q1evLta6Nc1ZeHgIQ9rUoXuTaryxKpbZG4/y/b4Epg8MoWP9qkaHZxfWjLrZp5RqqZRqrpQKUUq9bll+TCnVVilVXyn1kFLK7j0f9u6UmjBhAvPnz+fq1atWreeZZ55h3LhxxMTE8Mknn3DjRuFn8N0+1XHuaZCLO+2xUopmzZoRHR1NdHQ0MTExrF27tngvQtNcRNVyPrz/SEu+fLwtmUrx6Ge7mPTtb1y46nqXs3SJM2Pt3SlVpUoVhgwZcsuVnDp27JhzecGvvvqKLl26AHdONZzbxYsXqVUraxDSwoULc5bbctrhgqY+btSoEcnJyezcuROAtLQ0Dhw4QKVKlahUqRLbtm3LeT2a5i66NPBnzYSuPH33XayIPkOPGZtZsve0U16iMz8ukegd0Sk1adKkW0bfzJkzhwULFtC8eXO+/PJLPvjgAyDrSlTvvvsuLVu2vKMzdurUqTz00EO0bt2aqlX/d4hoy2mHb9/+qFGjGDt2LGFhYWRkZLBkyRJefPFFWrRoQVhYWM7FyBcsWMDTTz9NWFiYS73BNa0ofL09eaFPY34c34Vg/3I8/91vPPrpLo4lXyn8yU5AT1OsmZZ+D2hGyMxUfLPnFG+tPkhqWiZP312fsd2C8fHyNDq0O+hpijVN00rAw0N4tF1dNkyKoE9IdWatP8y9H2xl9/ELRodWYjrRa5qm5aFaeV/mDG3JgtFtSE3PZMgnO3lxyT5SrjlfZ61O9JqmaQW4u1E11j7Xlb9GBLPkl9P0mBHJ8l/POFVflk70mqZphShTyouX72nC9+M6U6dKGSYsjmbEf3Zz4rx1Q64dRSd6TdO0ImpaswL//VtH/jmgGdEnU+g9awsfbTrKzfRMo0MrkE70mqZpxeDpIQzvEMj6SRF0b1yNd9fEcd+cbew9Yd7OWp3oC+Dp6UlYWBjNmjWjRYsWzJgxg8zMkn1zT548mfXr1+f797lz5/LFF1+UNNQcmzdvpn///lavpzhGjRrFkiVLCn+gprmQgAq+fPyX1nw2Ipwrqek88PFOXlkWw8XraUaHdgdr5rpxeaVLlyY6OhqApKQkHn30US5dusS0adOKva7XX3+9wL+PHTu2RDE6o/T0dLy89FtPcw09mwbQ4S4/Zq07zH+2H2ftgUSm3NeU/s1rICJGhwfoFn2RVatWjXnz5vHhhx+ilCIjI4MXXniBNm3a0Lx5cz755JOcx77zzjuEhobSokWLnHnic7d6X3rpJZo2bUrz5s15/vnngayzZt977z0AoqOjad++Pc2bN2fQoEE5UyJ069aNF198kbZt29KwYUO2bt2aZ6yXLl2iX79+NGrUiLFjx+YchSxatIjQ0FBCQkJ48cUXcx5frly5nNtLlixh1KhROTE/++yzdOzYkeDg4Jz4lVKMGzeORo0a0bNnT5KSknKe//rrr9OmTRtCQkIYM2ZMzsiEbt26MWHCBMLDw5k+fTpBQUGkpaXlxJv7vqY5m7I+XrzWvykrx3WmRkVfnln0K6M/38OpC9cKf7IDOEWzatr3B4g9e8mm62xaswJT7mtWrOcEBweTkZFBUlISK1asoGLFiuzZs4fU1FQ6depE7969OXToECtWrGDXrl2UKVOGCxdurdudP3+eZcuWcejQIUSElJSUO7YzYsQI5syZQ0REBJMnT2batGm8//77QFZrePfu3fz4449MmzYtz3LQ7t27iY2NpV69evTt25elS5fSsWNHXnzxRfbu3UvlypXp3bs3y5cvZ+DAgq/0mJCQwLZt2zh06BD3338/Dz74IMuWLSMuLo7Y2FgSExNp2rQpjz32GADjxo1j8uTJQNYsmKtWreK+++4D4ObNm2SfAR0fH88PP/zAwIED+eabbxg8eDDe3t7F+n9omtmE1KrI8qc78cXOeN5bE0evWZFM6NmQxzsH4e1pXLtat+hLaO3atXzxxReEhYXRrl07zp8/z5EjR1i/fj2jR4+mTJmsSdaqVKlyy/MqVqyIr68vjz/+OEuXLs15XLaLFy+SkpJCREQEACNHjmTLli05fx88eDAArVu3znea4rZt2xIcHIynpydDhw5l27Zt7Nmzh27duuHv74+XlxfDhg27Zb35GThwIB4eHjRt2jTnQilbtmxh6NCheHp6UrNmTbp3757z+E2bNtGuXTtCQ0PZuHEjBw4cyPnbww8/nHP7iSeeYMGCBUDWPDujR48uNBZNcwaeHsLoTkGsmxhBlwb+vL36EPfN2cavJ0s+WaG1nKJFX9yWt70cO3YMT09PqlWrhlKKOXPm0KdPn1ses2bNmgLX4eXlxe7du9mwYQNLlizhww8/ZOPGjUWOIXuaYk9Pz3ynKb69LlhYnTD332+fOjn3NMmFnSBy48YNnnrqKaKioqhTpw5Tp069ZX1ly5bNud2pUyfi4+PZvHkzGRkZ+hq1msupWak0n44IZ82Bc0xZcYDBH+9gePt6vNCnEeV9HXv0qlv0RZScnMzYsWMZN24cIkKfPn34+OOPc+rKhw8f5urVq/Tq1YsFCxZw7VpWbe720s2VK1e4ePEi9957L7NmzeK333675e8VK1akcuXKOfX3L7/8Mqd1X1S7d+/m+PHjZGZmsnjxYjp37kzbtm2JjIzkjz/+ICMjg0WLFuWsNyAggIMHD5KZmcmyZcsKXX/Xrl1ZvHgxGRkZJCQksGnTJuB/XxJVq1blypUrhY7EGTFiBI8++qhuzWsurU+z6qyb2JWRHQL58ucT9JwZyeqYBIeeWVviFr2I1AG+AAIABcxTSn0gIlWAxUAgEA8MUUoZd8xihevXrxMWFkZaWhpeXl4MHz6ciRMnAlmlh/j4eFq1aoVSCn9/f5YvX07fvn2Jjo4mPDycUqVKce+99/Lmm2/mrPPy5csMGDCAGzduoJRi5syZd2x34cKFjB07lmvXrhEcHJxT4iiqNm3aMG7cOI4ePcrdd9/NoEGD8PDw4O233+buu+9GKUW/fv0YMGAAAG+//Tb9+/fH39+f8PBwrlwpeGrWQYMGsXHjRpo2bUrdunXp0KEDAJUqVeLJJ58kJCSE6tWr06ZNmwLXM2zYMF577TWGDh1arNenac6mvK83U+9vxqCWtXh5aQx/++oXejapxrQBIdSqVLrwFVipxNMUi0gNoIZS6hcRKQ/sBQYCo4ALSqm3ReQloLJS6sUCVqWnKXZTS5YsYcWKFXz55Zd5/l2/BzRXlJ6RyYLt8cxcdxgReOeB5tzXomaJ1lXUaYpL3KJXSiUACZbbl0XkIFALGAB0szxsIbAZKDDRa+7nmWeeYfXq1fz4449Gh6JpDuXl6cGTXYO5J7Q6U1fGElS1bOFPsnabtliJiAQCLYFdQIDlSwDgHFmlHU27xZw5c4wOQdMMVbtyGT4bWWhj3Cas7owVkXLAf4EJSqlbBrurrLpQnrUhERkjIlEiEpWcnGxtGJqmaVo+rEr0IuJNVpL/Sim11LI40VK/z67jJ+X1XKXUPKVUuFIq3N/fP8/1O9N8z5pt6f+9ptlOiRO9ZA2+ng8cVErlHjqyEhhpuT0SWFGS9fv6+nL+/Hn9gXdDSinOnz+Pr6+v0aFomkuwpkbfCRgOxIhItGXZK8DbwLci8jhwAhhSkpXXrl2b06dPo8s67snX15fatWsbHYamuQRrRt1sA/I75bJHSdebzdvbm6CgIGtXo2ma5vb0mbGapmkuTid6TdM0F6cTvaZpmosr8RQINg1CJJmsjlt7qAr8Yad125IzxKljtA1niBGcI053j7GeUirv8em5mCLR25OIRBVlLgijOUOcOkbbcIYYwTni1DEWjS7daJqmuTid6DVN01ycOyT6eUYHUETOEKeO0TacIUZwjjh1jEXg8jV6TdM0d+cOLXpN0zS35lKJXkR8RWS3iPwmIgdEZJpleZCI7BKRoyKyWERKmSBWTxH5VURWmTFGEYkXkRgRiRaRKMuyKiKyTkSOWH5XNjJGS0yVRGSJiBwSkYMi0sFMcYpII8s+zP65JCITzBSjJc7nLJ+Z/SKyyPJZMtt7crwlvgMiMsGyzPD9KCL/EZEkEdmfa1mecUmW2ZZ9uk9EWjkiRpdK9EAq0F0p1QIIA/qKSHvgHWCWUqo+8CfwuIExZhsPHMx134wx3q2UCss1NOwlYINSqgGwwXLfaB8APymlGgMtyNqnpolTKRVn2YdhQGvgGrDMTDGKSC3gWSBcKRUCeAKPYKL3pIiEAE8Cbcn6P/cXkfqYYz9+DvS9bVl+cd0DNLD8jAE+dkiESimX/AHKAL8A7cg6WcHLsrwDsMbg2Gpb/vndgVVkTQ5nthjjgaq3LYsj6zrBADWAOINjrAgcx9LXZNY4c8XVG9huthjJugToKaAKWRMdrgL6mOk9CTwEzM91/x/A382yH4FAYH+u+3nGBXwCDM3rcfb8cbUWfXZJJJqsC56sA34HUpRS6ZaHnCbrjW2k98l6k2Za7vthvhgVsFZE9orIGMsys10mMghIBhZYymCfiUhZzBdntkeARZbbpolRKXUGeA84SdZ1oC8CezHXe3I/0EVE/ESkDHAvUAcT7cfb5BdX9pdqNofsV5dL9EqpDJV1mFybrMO8xgaHdAsR6Q8kKaX2Gh1LITorpVqRdaj5tIh0zf1HldUcMXrIlhfQCvhYKdUSuMpth+4miRNLfft+4Lvb/2Z0jJb68QCyvjhrAmW5sxRhKKXUQbJKSWuBn4BoIOO2x5jif307M8Tlcok+m1IqBdhE1iFnJRHJnnu/NnDGsMCyLthyv4jEA9+QVb75AHPFmN3KQymVRFZNuS1FvEykA50GTiuldlnuLyEr8ZstTsj6wvxFKZVouW+mGHsCx5VSyUqpNGApWe9Ts70n5yulWiulupLVZ3AYc+3H3PKL6wxZRyLZHLJfXSrRi4i/iFSy3C4N9CKrc24T8KDlYSW+vKEtKKVeVkrVVkoFknUov1EpNQwTxSgiZUWkfPZtsmrL+7HRZSJtRSl1DjglIo0si3oAsZgsTouh/K9sA+aK8STQXkTKiIjwv/1omvckgIhUs/yuCwwGvsZc+zG3/OJaCYywjL5pD1zMVeKxHyM6LuzYIdIc+BXYR1ZimmxZHgzsBo6SdejsY3Sslri6AavMFqMllt8sPweAVy3L/cjqRD4CrAeqmGAfhgFRlv/5cqCy2eIkqxRyHqiYa5nZYpwGHLJ8br4EfMz0nrTEuJWsL6DfgB5m2Y9kfYEnAGlkHWU+nl9cZA28+IisvsMYskY62T1GfWaspmmai3Op0o2maZp2J53oNU3TXJxO9JqmaS5OJ3pN0zQXpxO9pmmai9OJXtM0zcXpRK9pmubidKLXNE1zcf8PRSdlK5otj3MAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Find indices of positive and negative examples\n", "pos = np.where(y==1)[0]\n", "neg = np.where(y==0)[0]\n", "\n", "# Plot examples\n", "plt.plot(X[pos, 1], X[pos, 2], 'b+', label='Admitted')\n", "plt.plot(X[neg, 1], X[neg, 2], 'yo', label='Not admitted')\n", "plt.legend()\n", "\n", "# Plot the decision boundary\n", "plot_x = np.array([min(X[:,1])-2, max(X[:,1])+2])\n", "plot_y = (-1./theta[2]) * ((theta[1] * (plot_x) + theta[0]))\n", "\n", "plt.plot(plot_x, plot_y, label='Decision boundary')\n", "\n", "# Legend, specific for the exercise\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Evaluation\n", "#### Probability that a given student will be admitted" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "For a student with scores 45 and 85, we predict an admission probability of 0.78\n" ] } ], "source": [ "prob = sigmoid(np.array([1, 45, 85]).dot(theta))\n", "print('For a student with scores 45 and 85, we predict an admission probability of {:.2}'.format(prob))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Accuracy\n", "It's often a good idea to see how well your model trained. You can do that by checking how much datapoints we can predict correctly using $\\theta$. In a real world application, you should consider splitting your data (eg 80% - 20%) and test on data the model has not seen before. This gives you more realistic insight in how your model would perform in the real world - and that's your ultimate goal ;)\n", "\n", "In this example, we expect a training accuracy of 89.0%." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training Accuracy: 89.0%\n" ] } ], "source": [ "p = np.zeros((m, 1))\n", "for (i, example) in enumerate(X):\n", " prob = sigmoid(np.array(example.dot(theta)))\n", " if prob >= 0.5:\n", " p[i] = 1\n", " else:\n", " p[i] = 0\n", "print('Training Accuracy: {}%'.format(np.mean(p == y.reshape((m, 1))) * 100))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Regularized linear regression\n", "\n", "---\n", "Regularization is a meganism for preventing overfitting. _Overfitting_ means that our model works extremely well on the training set but bad in the real world. It's focussed on the training data. _Underfitting_ is either a not well trained model or the feature mapping is not done (correctly). We will use regularization in this exercise. Furtermore, we are going to look at a more complex decision boundary\n", "\n", "In this part of the exercise, you will implement regularized logistic regression to predict whether microchips from a fabrication plant passes quality assur- ance (QA). During QA, each microchip goes through various tests to ensure it is functioning correctly.\n", "\n", "Suppose you are the product manager of the factory and you have the test results for some microchips on two different tests. From these two tests, you would like to determine whether the microchips should be accepted or rejected. To help you make the decision, you have a dataset of test results on past microchips, from which you can build a logistic regression model." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[ 0.051267, 0.69956 ],\n", " [-0.092742, 0.68494 ],\n", " [-0.21371 , 0.69225 ],\n", " [-0.375 , 0.50219 ],\n", " [-0.51325 , 0.46564 ]])" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# start by loading the data\n", "data = pd.read_csv(\"ex2data2.txt\", header = None, \n", " names = [\"Test 1\", \"Test 2\", \"Status\"])\n", "\n", "# initialize some useful variables\n", "m = len(data[\"Status\"])\n", "size = np.array((data[\"Test 1\"]))\n", "bedrooms = np.array((data[\"Test 2\"]))\n", "X = np.array([size, bedrooms]).T # don't add a column of ones yet.\n", "y = np.array(data[\"Status\"])\n", "\n", "X[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualising the data" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3XuQFPW58PHvA8oSozkC8iKKXDZBFEQXRc1GoyhEyA1MaRSDCl6KF49Yb7wk4ksdWTxFRY+pItGY1xgJkMQACcfLJtEyXiBEwcQlZyOIISIuiiJsQEyMAgLP+8f0rj2zMzsz2/fu51M1tTN9mf5N70w//buLqmKMMca06RZ1AowxxsSLBQZjjDF5LDAYY4zJY4HBGGNMHgsMxhhj8lhgMMYYk8cCgzHGmDwWGIwxxuSxwGCMMSbPIVEnoCuOOuooHTx4cNTJMMaYRFm7du3fVbVvue0SGRgGDx5MU1NT1MkwxphEEZEtlWxnRUnGGGPyWGAwxhiTxwKDMcaYPBYYjDHG5LHAYIwxJo8FBpMK27c/xJo1g1m5shtr1gxm+/aHok6SMYmVyOaqxrht3/4QGzdO5+DBDwDYu3cLGzdOB6BfvylRJs2YRLIcg0m8zZtntweFNgcPfsDmzbMjSpExyWaBwSTe3r1vVLU86azYzATNAoNJvJqagVUtT7K2YrO9e7cA2l5sZsHB+MkCg0m82tp5dOt2WN6ybt0Oo7Z2XkQpCo4Vm5kwWGAwidev3xSGDXuAmppBgFBTM4hhwx5IZcVz1orNTDSsVZJJhX79pqQyEBSqqRnoFCN1XG6MXyzHYEyCZKnYzETHAoMxCZKlYjMTHStKMiZhslJsZqLjS45BRH4iIjtEZH2J9SIi94jIJhF5SUROda2bKiKvOo+pfqTHGGNM1/lVlLQImNDJ+i8CQ53HdOD/AYhIb2AOcCZwBjBHRHr5lKbMsA5P1bNzZkxpvgQGVV0F7Opkk0nATzXnBeBIEekPjAeeUtVdqvou8BSdBxhTIKsdnrxc2LN6zoypVFiVz8cCb7peb3WWlVpuKpTFDk9eL+xZPGfGVCMxrZJEZLqINIlIU2tra9TJiY0sdnjyemHP4jkzphphBYa3gONcrwc4y0ot70BVH1DV0ao6um/fvoElNGniOk5QQ0Nw7+31wh7Xc2ZMXIQVGBqBK53WSZ8F3lPVbcCTwAUi0supdL7AWWYqFNcOT3PnBvfeXi/scT1nxsSFX81VlwBrgGEislVErhGRGSIyw9nkcWAzsAn4MfDvAKq6C/hP4EXncYezzFQoix2evF7Ys3jOjKmGqGrUaaja6NGjtampKepkmAINDcVzCnPm+F+0tH37Q2zePJu9e9+gpmYgtbXzEnFhT2q6TTqIyFpVHV12OwsMBvy/YIlAAr9aHfh5XgqnIIVcTsdyKyYslQaGxLRKMsGxdv3F+X1erJmsSQoLDAnlZ8/dIC5Yc+Z0edfY8Pu8WDNZkxQWGBLI7zvZIC5YQTZXDYvf58WayVbGhiuJngWGBPL7TtYuWMX5fV6smWx5VqwZDxYYEsjvO1m7YBXn93mxZrLlWT1MPNh8DAnk9/SObRcma0aZL4jzYnMpdM7qYeLBAkMC1dbOK9rs0csdvl2wirPzEi6b0zoerCgpgaxIwqSVFWvGg+UYEsruZE0aWbFmPFhgMCaD4jw0h930RM+KkkzqpaFPhZ+sSagpxwKDSb0ghwBPImsSasqxwGBiy+70g2FNQk05FhhMbHm5029oyI3wKpJ73fbcgo31dDflWWAwvojbBbehITfsd9vQ323P/Uyn32P6hDVGkDUJNeVYYDC+8KscPyl3+n5X4IZZIWz9YEw5vkzUIyITgO8D3YEHVfXOgvXzgfOcl4cB/0tVj3TWHQDWOeveUNWJ5Y5nE/XETxAT8/j1ng0N/geWNWsGl+ihO4j6+pbI3y8scW72ajoKbaIeEekO3Ad8ERgOXCYiw93bqOqNqlqnqnXAvcDDrtUftq2rJCiY+EjK3X0Q6fG7AjeJFcLW7DW9/ChKOgPYpKqbVXUfsBSY1Mn2lwFLfDiuiVjQ5fhxnuzH7wrcJFYIW7PX9PIjMBwLvOl6vdVZ1oGIDAKGAM+6FvcUkSYReUFELvQhPSYl4pbzcPO7AjeJFcJ+5nJscp54CXtIjMnAclU94Fo2SFXfEpFa4FkRWaeqrxXuKCLTgekAAwfG9y4qq+J8dx8Ev8f0SeIYQX6NhNpWJNWW+2grkgJi/fnTzHPls4jUAw2qOt55fRuAqn6nyLb/A1yvqqtLvNci4DequryzY1rlszHRK7ygQy6XU20Lp6RWvCdRaJXPwIvAUBEZIiI9yOUKGosk6ASgF7DGtayXiNQ4z48CzgI2+JCmVAgyex3nrHvcipDilp648KvZaxIr3tPOc2BQ1f3ATOBJ4BXgl6r6sojcISLuVkaTgaWan0U5EWgSkb8AK4A7VdUCA8G2+IhLa5JSF9y4jW0Ut/SUE2Yg69dvCvX1LYwZc5D6+pYuFf0kseI97XzpxxC2uBQlBdmGO8jsdVyy7qX6KXSl/0IQfRXaBNFHI0hJS69fRVKmvDCLkjIp6LvuILPXccy6e+0T4fddfVL6aAQtjM9rPbHjxwJDFwXdhjvI7HWUWfdSF1wIfmyjaoQx1pKfggpkYRWj+VEk5VWc693CZoGhi4K+6w6yXXuUbeb9vODaXf3HkhbI4iYu9W5xYYGhi4K+6w4yex33rPu551a2XVgXw6z10YhrwA3yjt56ceezyucuimuFWZCVsH4rldauVJ4mrcI1SH5+B+JyXoP+va1c2Q0o9kGFMWMOen7/uLDK54DF9a47SU0r/QxgWbur70xSbgyqkeQ6vSSywOBBHCrM0sJr8UUaL4blhFFZGpeAm+Q6vSSywJACcS0TroZVnlYnrMrSuJz/JNfpJZHVMUTM7zqBuJQJe5GGzxC0uHRS9KrSTqJxrdNLGqtjSIgk1QmEJS7FF3EWx06K1aom12N39OGywJAyabioxqX4wqsgP0dnRStJOX/VVihbnV54LDBEIMg6gaRcFLIgyNxgZ5WlScmFpiHXk1YWGCJgFa3GqzQUrVgT0fiywGASo7PAGZegGmYLMXfRypNPtnD00VMS1TLNmojGl7VKqpLfQ20nqady1DprrRTHlkzF0hTG/zuO56KUIIeuNx1V2irJAkMVrMlctNIQGMJIZxzPhYkHa64aABtoK3ydFc3EvWNfV1uIeU1/GlqmmWhZjqEKWRloK66SlmNo09BQvIXSnDn+DSJoTCVCzTGIyAQR2Sgim0RkVpH100SkVUSance1rnVTReRV5zHVj/QEJSutKOJyx50W1gotejYJT3U8BwYR6Q7cB3wRGA5cJiLDi2y6TFXrnMeDzr69gTnAmcAZwBwR6eU1TUEJohVFHL+wcW0H31kRSdKLT+JeLJZkNglP9fzIMZwBbFLVzaq6D1gKTKpw3/HAU6q6S1XfBZ4CJviQpkD43XbcvrDVSUJz1XJKBbC05irikH6rG6yeH4HhWOBN1+utzrJCF4nISyKyXESOq3JfRGS6iDSJSFNra6sPye4aP7vlx+kL69cdaxwuBHGWtfMTh9yn9bCuXlitkn4NDFbVk8nlChZX+waq+oCqjlbV0X379vU9gVHw+oX18yLj1x1rHC4ESZf0YrG4yUrdoJ/8CAxvAce5Xg9wlrVT1Z2qutd5+SBwWqX7ppnXL6xdhNMp6bmKuNWXWA/r6vkRGF4EhorIEBHpAUwGGt0biEh/18uJwCvO8yeBC0Skl1PpfIGzLBPi+oWt9o41bhcCE62o60sKj5OGcaXC5ks/BhH5EvA9oDvwE1WdJyJ3AE2q2igi3yEXEPYDu4DrVPWvzr5XA//Xeat5qrqw3PHSNFFPtUMCVNsmPmzWBt+4RfF9sO9gaTYkRgbE8QcQxzSZ6EQxFph9B0uzITFMJKzi1LiFWXxkxZn+sRxDgtnIrMZ0ZDmG0izHkAEWFLyzc2hMRxYYTKZZk9/0seJM7ywwGGNSxXKB3llgMJljFZXGdM4Cg4lUFBfjqDtgmXSK40jJXWWBwUTKyvhNGqRtpGQLDCbTrKLS+CFOIyX7wQJDyKy4Il5l/Pb/MH5I29De1sEtZNb5Jp+dD5MGa9YMdoqR8tXUDKK+viX8BJVgHdwKpKliKCx2N21MZeI6UnJXZSIwRF0xFKeik2qEUTFsZfwmDdI2tHcmipK6ks2rdjjsSiWp6CRJaTXGlGdFSS7VVgxFncOIUlJzN8YY/2QiMFQ7hWaQTc/iXnRinb9MVlk95McyERiqrRgKsumZXWBN1sXxN5DlUoJifAkMIjJBRDaKyCYRmVVk/U0iskFEXhKRZ0RkkGvdARFpdh6Nhfv6odqKoWpzGGkV99yNSaY49nZPWwc1rw7x+gYi0h24D/gCsBV4UUQaVXWDa7P/AUar6gcich3wX8ClzroPVbXOazrK6ddvSsWVx7W189i4cXreFyXJTc+6Ko53dsYEIW0d1LzyI8dwBrBJVTer6j5gKTDJvYGqrlDVtqvsC8AAH44bmLQ1PTMmanFv1GClBPk85xiAY4E3Xa+3Amd2sv01wBOu1z1FpAnYD9ypqo/6kCbPqslhGGM6556GNo7NoK2UIF+olc8icjkwGrjbtXiQ0672G8D3ROTTJfadLiJNItLU2toaQmrTJy53Z8bEjZUS5PMjMLwFHOd6PcBZlkdExgGzgYmqurdtuaq+5fzdDKwERhU7iKo+oKqjVXV03759fUh2cnX1Ah/HSj+TPXFt1NCv3xTq61sYM+Yg9fUtmQ0K4E9geBEYKiJDRKQHMBnIa10kIqOAH5ELCjtcy3uJSI3z/CjgLMBdaW2KsAu8STLLucaf58CgqvuBmcCTwCvAL1X1ZRG5Q0QmOpvdDRwO/KqgWeqJQJOI/AVYQa6OwQKDj+Je6WeMiZ9MjJWUBg0NxXMKc+ZUfpGPY6WfMSY8NlZSyqR1qIqkp9+YNLLAkCFxrPSz+hJTyG4WomeBIYG6eoG3H5xJArtZiJ4FhgRK+gXeKsSNiTcLDCZ0aa0vMV1nNwvxYq2STKSspZQpZN+J4FirJJMIcawQNybrLDCYSFlRgSlkNwsdhT27nB+jqxpjjG/sZiFf2+xybSO/ts0uBwQ2npPlGIwxJkBe7/ajmF3OcgzGGBMQP+72o5hdznIMxhgTED/u9qOYXc4CgzHGBMSPu/3a2nl063ZY3rKgZ5ezwGCMMQHx424/itnlLDB4EHYTsqBYKxBjguHX3X7Ys8tZYOiitkqlvXu3ANpeqZTE4OB10DILLMYUl9S5pG1IjC5as2awExTy1dQMor6+JfwEeeB1CAIbwsCYZAh1SAwRmSAiG0Vkk4jMKrK+RkSWOev/KCKDXetuc5ZvFJHxfqQnDH43IQv7rtsGLTPVsu9GZdJQxOw5MIhId+A+4IvAcOAyERlesNk1wLuq+hlgPnCXs+9wYDIwApgA/NB5v9jzuwlZ2GPQex3h1AJL9tg8CeWlpYjZjxzDGcAmVd2sqvuApcCkgm0mAYud58uBsSIizvKlqrpXVV8HNjnvF3tRNCGLExs625iOouilHAQ/AsOxwJuu11udZUW3UdX9wHtAnwr3jSU/KpXictcdxaBlFkCSIS7f0aSIopdyEBIzJIaITAemAwwcGFyPv2r06zfFU+uChoaPf2BRVuB6/ZF3JbDMnWsXlyQI+zu6fftDbN48m71736CmZiC1tfNi34LHraZmYIlGKfG4ZlXKjxzDW8BxrtcDnGVFtxGRQ4B/A3ZWuC8AqvqAqo5W1dF9+/b1IdnGL3aBN35IQ/l8WoqY/QgMLwJDRWSIiPQgV5ncWLBNIzDVeX4x8Kzm2sk2ApOdVktDgKHAn3xIU+JkYQx6K5ZItqC/o2kon09qv4VCvvRjEJEvAd8DugM/UdV5InIH0KSqjSLSE/gZMArYBUxW1c3OvrOBq4H9wDdV9Ylyx4tDPwbzsa5k/63vgym0cmU3oNiXQhgz5mDYyUmlSvsx+FLHoKqPA48XLLvd9XwP8PUS+84DkpXP8ijp5ahuUUwiYtIpLeXzaWBDYoQsDeWobl3N/meh6MxUJy3l82lggaEKfvRoTEM5qltXm+dZvYIpFNfy+TT0ZK5WYpqrRs2vIpO0tHNuY9l/4yevTcD9ltWiUssxVMivO/0oZmMKkmX/TZqlLYdfKQsMFfLrTj9tF9K4Zv+N8UPacviVsqKkCvlVZNJ2wUxLqySIX/bfGL9ktajUAkOFamvn5ZU1Qtfv9IO8kKapKawxUfPzd58kVpRUoSQUmaStKawxUbdeS8LvPgg2g1uKpGlWOWPAesj7LdQZ3Ew8ZLWiLCxR370aExYLDCmStqawcWMzmIXDBluMngWGFElbU1iv7EISnCDPrV+zA9r/v+ssMKRIFBVlcf7x+XGHb3evxSUh95SENMaVVT4bT+JcOeh32uL8WcMW1rlwzyBXLft/dWSVzxHK4qBblQjjLtvu8IMTxbntSvFRWGlM8+/ccgw+Kxx0C3Ll/Glq+9zQUDybPmdO5z/AsO/g/D6el7vXtEnC3XiQaUzq77zSHIMFBp9lrS9BNT++pAeGOIoqWCXh3AaZxqT+zq0oKSLWlyBflEU7WZgMKKoK1iSc2yDTmPbfuafAICK9ReQpEXnV+duryDZ1IrJGRF4WkZdE5FLXukUi8rqINDuPOi/piYOs9SUo9+Pzq+lhV1ixT3CS8P8LMo1p/517zTHMAp5R1aHAM87rQh8AV6rqCGAC8D0ROdK1/luqWuc8mj2mx3fVVjAlrS+B1wo0u/iGLysV7HFubpq033m1vAaGScBi5/li4MLCDVT1b6r6qvP8bWAH0NfjcUPRlUHpkjToVtiD7iWh+CEJosyFmZwk/c67wlPls4jsVtUjnecCvNv2usT2Z5ALICNU9aCILALqgb04OQ5V3Vti3+nAdICBAweetmVLx4ofvyW1gqkSDQ0wfnyyPp+1CuooCZXA1ehqizdTGd9aJYnI08DRRVbNBha7A4GIvKuqHeoZnHX9gZXAVFV9wbXsHaAH8ADwmqreUS7RYbVKWrmyG1Ds/AhjxhwM/PhBEoEVK5L1+dJ2EfRDmoOl/b/9V2lgKDtRj6qO6+Qg20Wkv6pucy7yO0ps9yngt8DstqDgvPc25+leEVkI3FIuPWFK++xNaf98WZDWoGCi5bWOoRGY6jyfCjxWuIGI9AAeAX6qqssL1vV3/gq5+on1HtPjqyRUMFVTeVxYafkf/zGPPXvi/fmyUtFqOrI6qeh4rWPoA/wSGAhsAS5R1V0iMhqYoarXisjlwELgZdeu01S1WUSeJVcRLUCzs8/75Y4bZge3OE+V6aX3ZVs2Pc6fr5AVLRjjjfV8zgAvleNJvMgmMc3GxIn1fM4AL70vk5hNT2KajUkiCwwJ5qX3ZRLL6JOYZmOSyAJDgiWhctwYkzwWGBIs7b0vjTHRKNuPwcRbv35TUhsIktRiyqRPlr9/FhhMLBU2xW0bxwnIzI/TRCfr3z8rSjKxtHnz7Lz+GQAHD37A5s2zI0qRyZKsf/8sMJhYKtcUN6gWStbyyUD6J+IpxwJDBiVhEvNyTXGDGqs/LnMAWICKVton4inHAkOEorhAhz0HQ1dlvSluXAJUmyTcTPgp698/CwwRieoCnZSy02JNcdevf4Cjj57i+4B6NlBf55JyM+GnrDcFt7GSIhLVJEBpmWMiqHGTohyPKa6T1KR5wqqssbGSYi6qyq2klp0WFmWMHZu+u9W4TtmZ9YrYLLLAEJGoLtBJLDstVpRx223BFGXYQH0dJfVmwnSdBYaIRHWBTmLZabF6ke7dg6kXifruvE2cAlQSbyaMN9bzOSJtF+IoutwnbRiNLBZlxCVAQbTfVRMNCwwRStoFOio2N3X07LuaLZ6KkkSkt4g8JSKvOn97ldjugIg0O49G1/IhIvJHEdkkIsuc+aGNyWNFGcaEy2sdwyzgGVUdCjzjvC7mQ1Wtcx4TXcvvAuar6meAd4FrPKbHpFAS60WMSTJP/RhEZCMwRlW3iUh/YKWqDiuy3fuqenjBMgFagaNVdb+I1AMNqjq+3HHT0I/BpFtDQ7zqCYyB8Pox9FPVbc7zd4B+JbbrKSJNIvKCiFzoLOsD7FbV/c7rrcCxHtNjTCzEbUiLsCU1KGZt6I9SylY+i8jTwNFFVuW1FVRVFZFS2Y9BqvqWiNQCz4rIOuC9ahIqItOB6QADB3asdPzoo4/YunUre/bsqeZtTYB69uzJgAEDOPTQQ6NOivFJpTmhuXODHQE3iPfO+hwMbqEUJRXsswj4DfDf+FiU9Prrr3PEEUfQp08fpG3QGxMZVWXnzp3885//ZMiQIVEnJxRxHdLCT5UOGRLk0CJBvXcWhv4IqyipEZjqPJ8KPFYkIb1EpMZ5fhRwFrBBcxFpBXBxZ/tXas+ePRYUYkRE6NOnT6ZycHEd0iIsSR+MMIv9ZUrxGhjuBL4gIq8C45zXiMhoEXnQ2eZEoElE/kIuENypqhucdbcCN4nIJnJ1Dgu8JMaCQrzY/yPeKr1gV3rBDzIw+h10itUl2NAfH/MUGFR1p6qOVdWhqjpOVXc5y5tU9Vrn+WpVHamqpzh/F7j236yqZ6jqZ1T166q619vHid6jjz6KiPDXv/616Ppp06axfPnyit/v7bff5uKLc5mq5uZmHn/88fZ1K1euZPXq1VWncfDgwfz973+vej9TuTgNaVFKpRXkccgJ+ZmGUsOI9+nzJesv48j8WEl+f7mXLFnC2WefzZIlS3x5v2OOOaY9kPgVGLIiyhYmDQ3Rt3CJ8vhxDoyl5iTZufNx6y/jyHxg8LNZ4fvvv89zzz3HggULWLp0KZCrhJ05cybDhg1j3Lhx7Nixo337wYMHc9ttt1FXV8fo0aP585//zPjx4/n0pz/N/fffD0BLSwsnnXQS+/bt4/bbb2fZsmXU1dVx1113cf/99zN//nzq6ur4wx/+QGtrKxdddBGnn346p59+Os8//zwAO3fu5IILLmDEiBFce+21JHEOjmpFPblMXI//wx8+5KlIptQFvzAIXXddcJ/Ta9DprC6hX78p1Ne3MGbMQerrWzIZFMDGSvLVY489xoQJEzj++OPp06cPa9euZcuWLWzcuJENGzawfft2hg8fztVXX92+z8CBA2lububGG29k2rRpPP/88+zZs4eTTjqJGTNmtG/Xo0cP7rjjDpqamvjBD34AwIcffsjhhx/OLbfcAsA3vvENbrzxRs4++2zeeOMNxo8fzyuvvMLcuXM5++yzuf322/ntb3/LggWeqnISobOZ6sL4sVdz/CCaX5Y6/qhRs1HNHb8rrXuKpTPsZp5ez5WNvVVeJnMMQbWeWLJkCZMnTwZg8uTJLFmyhFWrVnHZZZfRvXt3jjnmGM4///y8fSZOzI0QMnLkSM4880yOOOII+vbtS01NDbt3767q+E8//TQzZ86krq6OiRMn8o9//IP333+fVatWcfnllwPw5S9/mV69ig5plSpRtzCp5vhBdIYL8/MnZbrYNjb2VnmZzDG479D8ahO9a9cunn32WdatW4eIcODAAUSEr33ta53uV1NTA0C3bt3an7e93r9/f6ndijp48CAvvPACPXv2rP4DpEzUd4VJOL5f9QBRB+Fq2TDi5WUyxxCE5cuXc8UVV7BlyxZaWlp48803GTJkCH369GHZsmUcOHCAbdu2sWLFii4f44gjjuCf//xnydcXXHAB9957b/vr5uZmAM455xx+8YtfAPDEE0/w7rvvdjkNSRH1XWG54wfd5r+Sz+/XsZLYzNPqEjqX+cDg113TkiVLOuQOLrroIrZt28bQoUMZPnw4V155JfX19V0+xnnnnceGDRuoq6tj2bJlfPWrX+WRRx5pr3y+5557aGpq4uSTT2b48OHtFdhz5sxh1apVjBgxgocffrjokCJBiqJ1TNQjspY7ftBNQMP8/FEHYeM/T0NiRKXYkBivvPIKJ554YkQpMqWsX9/Erl3n5pVBd+t2WGabARYT5PARYdm+/SErmkmASofEyGQdgwnP/v3vRto6KAni3Oa/UjbDW7pkvijJBEv1QNHlca2YjEJSxhIy2WGBwQRKpHvR5XGumDQm6ywwmEAdckgvq5g0JmEsMJhAde/+SRt/xpiEscpnEzirmDQmWSzH4CMR4eabb25//d3vfpeGMjWLjz76KBs2bOh0m3KqHUa7sbGRO++8s+jxFy1axNtvv13V8dsG+jPGpENmA0MQna5qamp4+OGHq7pI+xEYqjVx4kRmzZpV9PhdCQzGmHTJZGAIakjkQw45hOnTpzN//vwO61paWjj//PM5+eSTGTt2LG+88QarV6+msbGRb33rW9TV1fHaa6/l7fPrX/+aM888k1GjRjFu3Di2b98OlB5Gu6WlhRNOOIFp06Zx/PHHM2XKFJ5++mnOOusshg4dyp/+9Ccgd/GfOXNmh+PfddddNDU1MWXKFOrq6vjwww9Zu3Yt5557Lqeddhrjx49n27ZtAKxdu5ZTTjmFU045hfvuu8/TeTPGxIyqdvkB9AaeAl51/vYqss15QLPrsQe40Fm3CHjdta6ukuOedtppWmjDhg0dlpWyevUgXbGCDo/VqwdV/B7FfPKTn9T33ntPBw0apLt379a7775b58yZo6qqX/nKV3TRokWqqrpgwQKdNGmSqqpOnTpVf/WrXxV9v127dunBgwdVVfXHP/6x3nTTTaqqesMNN+jcuXNVVfU3v/mNAtra2qqvv/66du/eXV966SU9cOCAnnrqqXrVVVfpwYMH9dFHH20/5sKFC/X6668vevxzzz1XX3zxRVVV3bdvn9bX1+uOHTtUVXXp0qV61VVXqarqyJEj9fe//72qqt5yyy06YsSIop+hmv+LyY533vm58zsUXb16kL7zzs+jTlImAE1awTXWa+XzLOAZVb1TRGY5r28tCDwrgDoAEekNbAJ+59rkW6pa+VyXPghyNMhPfepTXHnlldxzzz184hOfaF++Zs0aHn74YQCuuOIKvv3tb5d9r61bt3LppZeybds29u3bx5AhQwBYtWpV+3sVDqM9ZMgQRo4PSX/fAAAI5UlEQVQcCcCIESMYO3YsIsLIkSNpaWmp6rNs3LiR9evX84UvfAGAAwcO0L9/f3bv3s3u3bs555xz2j/PE088UdV7J5kN/+BN2PM3mOp5LUqaBCx2ni8GLiyz/cXAE6r6QZntAhX0aJDf/OY3WbBgAf/61788vc8NN9zAzJkzWbduHT/60Y/Ys2dP2X0Kh+52D+td7TDeqsqIESNobm6mubmZdevW8bvf/a78jikW9cxsaZC0+RuyyGtg6Keq25zn7wD9ymw/GSicDHmeiLwkIvNFpKbYTn4LejTI3r17c8kll+TNlPa5z32ufbrPhx56iM9//vNAx6Gz3d577z2OPfZYABYvXty+3M9htDsbynvYsGG0trayZs0aAD766CNefvlljjzySI488kiee+659s+TFXG4qEU9l7RXSZu/IYvKBgYReVpE1hd5THJv55RflRwjUkT6AyOBJ12LbwNOAE4nV19xa5Fd2/afLiJNItLU2tpaLtmdCmNI4ptvvjmvddK9997LwoULOfnkk/nZz37G97//fSA309vdd9/NqFGjOlQ+NzQ08PWvf53TTjuNo446qn25n8NoFx5/2rRpzJgxg7q6Og4cOMDy5cu59dZbOeWUU6irq2P16tUALFy4kOuvv566urpMzCHdJuqLWhpyLEmcvyFrPA27LSIbgTGqus258K9U1WEltv0/wAhVnV5i/RjgFlX9Srnj2rDbyZG2/8uaNYNLzIw2iPr6ltQf3w+FdQxgQ7GHpdJht70WJTUCU53nU4HHOtn2MgqKkZxggogIufqJ9R7TY0ygop6UJuocix+inkTJlOe1VdKdwC9F5BpgC3AJgIiMBmao6rXO68HAccDvC/Z/SET6AkKuueoMj+kxJlBRzxcc9VzSfrFhUuLNU2BQ1Z3A2CLLm4BrXa9bgGOLbHe+l+MbE4UoL2q1tfOKFsPYaLXGT6nq+ZylStAksP+H/6wYxoQhNaOr9uzZk507d9KnTx9yVRYmSqrKzp076dmzZ9RJSR0rhjFBS01gGDBgAFu3bsVrU1bjn549ezJgwICok2GMqVJqAsOhhx7aPmSEMcaYrktVHYMxxhjvLDAYY4zJY4HBGGNMHk9DYkRFRFrJdagLy1FA5dOyhc/S542lzxtLnzdhpm+QqvYtt1EiA0PYRKSpkvFFomLp88bS542lz5s4ps+KkowxxuSxwGCMMSaPBYbKPBB1Asqw9Hlj6fPG0udN7NJndQzGGGPyWI7BGGNMHgsMDhHpLSJPicirzt9eRbY5T0SaXY89InKhs26RiLzuWlcXdvqc7Q640tDoWj5ERP4oIptEZJmI9Ag7fSJSJyJrRORlZ57vS13rAjl/IjJBRDY6n3tWkfU1zvnY5Jyfwa51tznLN4rIeD/SU2XabhKRDc65ekZEBrnWFf0/R5DGaSLS6krLta51U53vw6siMrVw35DSN9+Vtr+JyG7XukDPoYj8RER2iEjRCcgk5x4n7S+JyKmudYGfu06pqj1yxWn/Bcxyns8C7iqzfW9gF3CY83oRcHHU6QPeL7H8l8Bk5/n9wHVhpw84HhjqPD8G2AYcGdT5A7oDrwG1QA/gL8Dwgm3+HbjfeT4ZWOY8H+5sXwMMcd6ne8hpO8/1/bquLW2d/Z8jOH/TgB8U2bc3sNn528t53ivs9BVsfwPwk7DOIXAOcCqwvsT6LwFPkJuo7LPAH8M6d+UelmP42CRgsfN8MbmpRjtzMfCEqn5QZju/VJu+dpIbh/x8YHlX9q9Q2fSp6t9U9VXn+dvADqBsZxsPzgA2qepmVd0HLHXS6eZO93JgrHO+JgFLVXWvqr4ObHLeL7S0qeoK1/frBSDsoWorOX+ljAeeUtVdqvou8BQwIeL0dZheOEiquorczWMpk4Cfas4LwJGSm+44jHPXKQsMH+unqtuc5+8A/cpsP5mOX7J5TpZwvojURJS+niLSJCIvtBVzAX2A3aq633m9lSIz6oWUPgBE5Axyd3mvuRb7ff6OBd50vS72udu3cc7Pe+TOVyX7Bp02t2vI3V22KfZ/9lulabzI+b8tF5Hjqtw3jPThFMMNAZ51LQ7jHHamVPrDOHedSs2w25UQkaeBo4usmu1+oaoqIiWbazlRfSTwpGvxbeQuiD3INT+7FbgjgvQNUtW3RKQWeFZE1pG72Hnm8/n7GTBVVQ86iz2fv7QSkcuB0cC5rsUd/s+q+lrxdwjUr4ElqrpXRP43udxXHKfsnQwsV9UDrmVxOYexk6nAoKrjSq0Tke0i0l9VtzkXrh2dvNUlwCOq+pHrvdvulveKyELglijSp6pvOX83i8hKYBTw3+SyqYc4d8UDgLeiSJ+IfAr4LTDbyT63vbfn81fEW8BxrtfFPnfbNltF5BDg34CdFe4bdNoQkXHkAu+5qrq3bXmJ/7PfF7WyadTcvO9tHiRX19S275iCfVeGnT6XycD17gUhncPOlEp/GOeuU1aU9LFGoK32fyrwWCfbdiirdC6GbeX5FwJFWyIEmT4R6dVWBCMiRwFnARs0V6O1gly9SMn9Q0hfD+ARcuWqywvWBXH+XgSGSq5FVg9yF4fC1ifudF8MPOucr0ZgsuRaLQ0BhgJ/8iFNFadNREYBPwImquoO1/Ki/2cf01ZNGvu7Xk4EXnGePwlc4KS1F3AB+TnsUNLnpPEEcpW4a1zLwjqHnWkErnRaJ30WeM+5QQrj3HUuzJruOD/IlSs/A7wKPA30dpaPBh50bTeYXETvVrD/s8A6che0nwOHh50+4HNOGv7i/L3GtX8tuQvbJuBXQE0E6bsc+Ahodj3qgjx/5Fp+/I3cneBsZ9kd5C62AD2d87HJOT+1rn1nO/ttBL4YwHeuXNqeBra7zlVjuf9zBGn8DvCyk5YVwAmufa92zusm4Koo0ue8bgDuLNgv8HNI7uZxm/Od30qunmgGMMNZL8B9TtrXAaPDPHedPaznszHGmDxWlGSMMSaPBQZjjDF5LDAYY4zJY4HBGGNMHgsMxhhj8lhgMMYYk8cCgzHGmDwWGIwxxuT5//WTskO2mhV/AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Find indices of positive and negative examples\n", "pos = np.where(y==1)[0]\n", "neg = np.where(y==0)[0]\n", "\n", "# Plot examples\n", "plt.plot(X[pos, 0], X[pos, 1], 'b+', label='Admitted')\n", "plt.plot(X[neg, 0], X[neg, 1], 'yo', label='Not admitted')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Feature Mapping\n", "\n", "One way to fit the data better is to create more features from each data point. While the feature mapping allows us to build a more expressive classifier, it also more susceptible to overfitting. This will also add $x_0$." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "def map_feature(X1, X2, degree):\n", " if not type(X1) == np.ndarray:\n", " X1 = np.array([X1])\n", "\n", " if not type(X2) == np.ndarray:\n", " X2 = np.array([X2])\n", "\n", " assert X1.shape == X2.shape\n", " \n", " out = np.ones((len(X1), 1))\n", " for i in range(1, degree+1):\n", " for j in range(i + 1):\n", " new = (X1 ** (i-j) * X2 ** j).reshape(len(X1), 1)\n", " out = np.hstack((out, new))\n", " return out\n", "\n", "X = map_feature(X[:,0], X[:,1], degree=6)\n", "m, n = X.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Computing Cost and Gradient\n", "#### Cost function\n", "Regularization works by penalizing theta. Theta values can be high when an overfit occurs so we prefer to keep them low. $\\lambda$ is the regularization parameter. If $\\lambda=0$, no regularization happens. If $\\lambda$ is some big number, $\\theta$ has a very high penalty. Just like the learning rate $\\alpha$ you have to try out certain values and see which work.\n", "\n", "The cost function with regularization:\n", "\n", "$$J(\\theta) = \\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}-\\begin{bmatrix}y^{(i)}\\log h(x^{(i)}+(1-y^{(i)}\\log(1-h_\\theta(x^{(i)})) \\end{bmatrix} + \\frac{\\lambda}{m}\\displaystyle\\sum_{j=1}^{n}{\\theta_j}^2$$\n", "\n", "**Exercise**: Implement the regularized cost function." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "def compute_regularized_cost(theta, X, y, _lambda):\n", " m = len(y)\n", " regularization = _lambda / (2 * m) * np.sum(theta[1:] ** 2) #np.squared()???/\n", " cost = 1/m * (-y @ np.log(sigmoid(X @ theta)) - (1 - y) @ np.log(1 - sigmoid(X@theta)))\n", " return cost + regularization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Gradient descent\n", "Gradient descent, just like the cost function, is slightly modified for regularization.\n", "\n", "For $\\theta_{j}$ where $j = 0$: \n", "\n", "$$\\theta_j := \\theta_j -\\alpha \\begin{bmatrix}\\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}(h_\\theta(x^{(i)}-y^{(i)}){x_0}^{(i)}\\end{bmatrix}$$\n", "\n", "For $\\theta_{j}$ where $j \\in \\{1, 2, ..., n\\} $: \n", "\n", "$$\\theta_j := \\theta_j -\\alpha \\begin{bmatrix}\\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}(h_\\theta(x^{(i)}-y^{(i)})x_j^{(i)} + \\frac{\\lambda}{m}{\\theta_j} \\end{bmatrix}$$\n", "\n", "Note that we don't penalize our bias vector $X_1$.\n", "\n", "**Exercise**: Implement `compute_regularized_gradient`." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "def compute_regularized_gradient(theta, X, y, _lambda):\n", " return np.zeros(len(theta))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We expect: $J \\approx 0.693$ and the first five values of $\\theta$: $\\begin{bmatrix} 0.0085 && 0.0188 && 0.0001 && 0.0503 && 0.0115 \\end{bmatrix}$" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cost at initial theta (zeros): 2.13\n", "Gradient at initial theta (zeros) - first five values only: \n", "[0.34604507 0.08508073 0.11852457 0.1505916 0.01591449]\n" ] } ], "source": [ "_lambda = 1\n", "initial_theta = np.ones(n)\n", "\n", "cost = compute_regularized_cost(initial_theta, X, y, _lambda)\n", "grad = compute_regularized_gradient(initial_theta, X, y, _lambda)\n", "\n", "print('Cost at initial theta (zeros): {:.3}'.format(cost))\n", "print('Gradient at initial theta (zeros) - first five values only: \\n{}'.format(grad[:5]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Find an optimimal value for theta using conjugate gradient." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Optimization terminated successfully.\n", " Current function value: 0.529003\n", " Iterations: 28\n", " Function evaluations: 76\n", " Gradient evaluations: 76\n", "Conjugate gradient found the following values for theta - first five values only: [ 1.27278161 0.62533883 1.18105652 -2.02009622 -0.91762258]\n" ] } ], "source": [ "from scipy.optimize import minimize\n", "result = minimize(compute_regularized_cost, initial_theta, args = (X, y, _lambda),\n", " method = 'CG', jac = compute_regularized_gradient, \n", " options = {\"maxiter\": 400, \"disp\" : 1})\n", "theta = result.x\n", "print('Conjugate gradient found the following values for theta - first five values only: {}'.format(theta[:5]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plotting the decision boundary\n", "We can plot a more complex decision boundary using `np.linspace` and `contour`." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl4VOXd8PHvnW2yExJCCEsIYQk7UUAWNxQURIT6uEtVbL2oPmJtq09rX1tBn4fXWtundWvVStX6IqJIEReqRUUEwhIU2UGWJARCCNlIyJ653z8yCSFkmeXMmTOT3+e6ciWZOXPOPSeT87vv370cpbVGCCGECPJ1AYQQQliDBAQhhBCABAQhhBAOEhCEEEIAEhCEEEI4SEAQQggBGBQQlFJ/V0qdUkrtbuf5KUqpMqXUDsfXE0YcVwghhHFCDNrPG8CLwD862OZrrfUsg44nhBDCYIa0ELTW64FiI/YlhBDCN4xqIThjklLqO+AE8KjWek/rDZRS84H5AFFRUWOHDh1qYvGEEML/bd++/bTWOtGd15oVEL4B+mutK5RSM4FVwODWG2mtXwVeBRg3bpzOysoyqXhCCBEYlFI57r7WlFFGWuszWusKx8+fAKFKqR5mHFsIIYRzTAkISqleSinl+PkSx3GLzDi2EEII5xiSMlJKLQOmAD2UUnnAQiAUQGv9MnAz8IBSqh6oAm7XssyqEEJYiiEBQWt9RyfPv0jjsFQhhJ+qq6sjLy+P6upqXxdFAOHh4fTt25fQ0FDD9mnmKCMhhB/Ly8sjJiaG1NRUHBlg4SNaa4qKisjLy2PAgAGG7VeWrhBCOKW6upqEhAQJBhaglCIhIcHw1poEBCGE0yQYWIc3/hYSEIQQQgASEIQQfmbVqlUopdi/f3+bz8+bN48VK1Y4vb8TJ05w8803A7Bjxw4++eST5ufWrVvHpk2bXC5jamoqp0+fdvl1viYBQQjhVYsWGbu/ZcuWcdlll7Fs2TJD9te7d+/mAGJUQPBXEhCEEF715JPG7auiooINGzawZMkS3nnnHaBxxM2CBQtIT09n2rRpnDp1qnn71NRUfv3rX5ORkcG4ceP45ptvmD59OgMHDuTll18GIDs7m5EjR1JbW8sTTzzB8uXLycjI4JlnnuHll1/mT3/6ExkZGXz99dcUFhZy0003MX78eMaPH8/GjRsBKCoq4tprr2XEiBHcd999+Os0Kxl2KoTwGx988AEzZsxgyJAhJCQksH37dnJycjhw4AB79+6loKCA4cOH86Mf/aj5NSkpKezYsYOf//znzJs3j40bN1JdXc3IkSO5//77m7cLCwvjqaeeIisrixdfbJw2VVVVRXR0NI8++igAd955Jz//+c+57LLLyM3NZfr06ezbt48nn3ySyy67jCeeeIKPP/6YJUuWmHtiDCIBQQhhuEWLzm8ZNA2IWbjQsxTSsmXLePjhhwG4/fbbWbZsGfX19dxxxx0EBwfTu3dvrr766vNeM3v2bABGjRpFRUUFMTExxMTEYLPZKC0tden4a9euZe/evc2/nzlzhoqKCtavX8/KlSsBuP766+nevbv7b9KHJCAIIQy3aNG5C79SYEQGpbi4mC+++IJdu3ahlKKhoQGlFDfeeGOHr7PZbAAEBQU1/9z0e319vUtlsNvtbN68mfDwcNffgB+QPgQhhF9YsWIFd911Fzk5OWRnZ3Ps2DEGDBhAQkICy5cvp6Ghgfz8fL788ku3jxETE0N5eXm7v1977bW88MILzb/v2LEDgCuuuIK3334bgDVr1lBSUuJ2GXxJAoIQwqsWLjRmP8uWLbugNXDTTTeRn5/P4MGDGT58OHfffTeTJk1y+xhXXXUVe/fuJSMjg+XLl3PDDTfwz3/+s7lT+fnnnycrK4vRo0czfPjw5o7phQsXsn79ekaMGMHKlStJSUnx6L36irJqb7jcIEcIa9m3bx/Dhg3zdTFEC239TZRS27XW49zZn7QQhBBCABIQhBBCOEhAEEIIAUhAEEII4SABQQghBCABQQghhIMEBCGEX1BK8cgjjzT//oc//IFFnayDsWrVqvOWmnCHq0tZr169mt/97ndtHv+NN97gxIkTLh2/afE9M0hAEEJ4RUHBUjIzU1m3LojMzFQKCpZ6tD+bzcbKlStdujgbERBcNXv2bB577LE2j+9OQDCTBAQhhOEKCpZy4MB8ampyAE1NTQ4HDsz3KCiEhIQwf/58/vSnP13wXHZ2NldffTWjR49m6tSp5ObmsmnTJlavXs1//dd/kZGRweHDh897zYcffsiECRO46KKLmDZtGgUFBUD7S1lnZ2czdOhQ5s2bx5AhQ5g7dy5r167l0ksvZfDgwWzduhVovOgvWLDgguM/88wzZGVlMXfuXDIyMqiqqmL79u1ceeWVjB07lunTp5Ofnw/A9u3bGTNmDGPGjOGll15y+5y5TGttya+xY8dqIYR17N271+ltN23qr7/8kgu+Nm3q7/bxo6KidFlZme7fv78uLS3Vzz77rF64cKHWWutZs2bpN954Q2ut9ZIlS/ScOXO01lrfc889+r333mtzf8XFxdput2uttf7b3/6mf/GLX2ittX7ooYf0k08+qbXW+qOPPtKALiws1EePHtXBwcF6586duqGhQV988cX63nvv1Xa7Xa9atar5mK+//rp+8MEH2zz+lVdeqbdt26a11rq2tlZPmjRJnzp1Smut9TvvvKPvvfderbXWo0aN0l999ZXWWutHH31Ujxgxos330NbfBMjSbl53ZbVTIYThampyXXrcWbGxsdx99908//zzREREND+emZnZvPz0XXfdxS9/+ctO95WXl8dtt91Gfn4+tbW1DBgwAKDDpawHDBjAqFGjABgxYgRTp05FKcWoUaPIzs526b0cOHCA3bt3c8011wDQ0NBAcnIypaWllJaWcsUVVzS/nzVr1ri0b3dJykgIYTibre3F3dp73BU/+9nPWLJkCWfPnvVoPw899BALFixg165dvPLKK1RXV3f6mtbLZ7dcWtvVpbS11owYMYIdO3awY8cOdu3axWeffebamzCYBAThM0Z3OgrrSEtbTFBQ5HmPBQVFkpa22ON9x8fHc+utt553V7LJkyc331Jz6dKlXH755cCFy1e3VFZWRp8+fQB48803mx83cinrjpbTTk9Pp7CwkMzMTADq6urYs2cPcXFxxMXFsWHDhub3YxYJCMInvNHpKKwjKWku6emvYrP1BxQ2W3/S018lKWmuIft/5JFHzhtt9MILL/D6668zevRo3nrrLZ577jmg8a5qzz77LBdddNEFncqLFi3illtuYezYsfTo0aP5cSOXsm59/Hnz5nH//feTkZFBQ0MDK1as4Fe/+hVjxowhIyODTZs2AfD666/z4IMPkpGRYer9mWX5a+ETmZmpjmBwPputP5MmZZtfIA8UFCzlyJHHqanJxWZLIS1tsWEXPiuR5a+tx+jlr6VTWfiEtzodzdbU0rHbKwGaWzpAQAYFEdgkZSR8wpudjmY6cuTx5mDQxG6v5MiRx31UIiHcJwFB+IQ3Ox3NFCgtHWdZNcXcFXnjbyEBQfiEtzsdzRIoLR1nhIeHU1RUJEHBArTWFBUVER4ebuh+pQ9B+ExS0ly/CwCtpaUtPq8PAfyzpeOMvn37kpeXR2Fhoa+LImgM0H379jV0nxIQhPBAU0DrCqOMQkNDm2fzisAkAUEIDwVCS0cIMKgPQSn1d6XUKaXU7naeV0qp55VSh5RSO5VSFxtx3K5CZvTKORDCDEZ1Kr8BzOjg+euAwY6v+cBfDTpuwAu0Gb3uXNgD7RwIYVWGBASt9XqguINN5gD/cKzOuhmIU0olG3HsQBdI49zdvbAH0jkQwsrMGnbaBzjW4vc8x2PnUUrNV0plKaWyZCRDIyuMc+/kLoVOc/fCboVzIERXYKl5CFrrV7XW47TW4xITE31dHEuwwjj3J580Zj/uXtitcA6E6ArMCgjHgX4tfu/reEx0IlBm9IL7F/ZAOgdCWJlZAWE1cLdjtNFEoExrnW/Ssf2ar2b0LloESjV+wbmfPUkfuXthN+scyEgm0dUZsvy1UmoZMAXoARQAC4FQAK31y0opBbxI40ikSuBerXWHa1vL8teeM2pZZqXAqNUKvLVUtKf7bb1qKTQGK39cTkN0bZ4sfy33Q7AoK13gjAwI3mDEew2k+zOIrs2TgGCpTmXRyIhx90YO1Vy40OWXmMqI99pVRzJJmky0JAHBgqx2gTNq2Km3GPFeu+JIJpnwJ1qTgGBBcoFzjRHvtSuOZJIJf6I1CQgWJBc41xjxXgPl/gyu6KppMtE+We3UgoxYY78rLcts1HvtaquW2mwp7XSkB14rUjhHRhlZlLeGZwrRRIbaBiZPRhlJC8GirFBbrWtooLS6muLqKoorKymprqKoqorymhoq6+qorKvlbMvvtXXUNjRQr+3U2+002Bu/N30pBcEqiCClUEoRrBRBji9bSAjhwSGEhzR+hYUEEx4SQmRIKNFhNqLDwogOCyPG1vhzTJiNWJuN+IgI4sIjCAsO9um5cpcvA39XakUK50hA6MLqGhrIKz/D0ZISsksbv3LLyjhRfoZTlWc5U1PT7muDlSIqLIyo0FAiQ8OIDA0lMjSUGFsYIUHBhAQpQoKCCQ5ShKgggoMau6vsWmPXmgZtR2tNg73x59qGBqrr6zlTU8OpyrPse28UvW/YRlV9HeU1NTR00pKNDgsjPjyCuIgIuodH0Cs6mj4xsfSOiXF8j6VXdDShFgocrWvoTaN8AFODggQA0URSRl1ERW0t+06fYs+pU+wpPMXuUwUcLimm3m5v3iYmzEb/uDj6xMSSFBVFfEQk8RERxDsusvGRkcSHRxBrsxEWHIxqWteiHYsWuT9kteVkOK011fX1VNTWUl5bQ3ltLeU1NZypqaakupqSqiqKq6soqWr8Kq6q5OTZCk5Xnj+CRgFJUdH069aN1Lju9O8WR2pcHP27xdE/rjvRYWHuFdZNMhlOeIPMVBbn0VpzpKSYzLxjbDtxnD2FBRwtKaHpL90jMpKRPZMY1iORtO7xpMbFkdqtO/EREZ1e5F3hyQxnI2ZHV9fXcaK8nBPl5RwvP8OJ8jOcKC8np6yUnNJSCivPnrd9j8hIBnaPZ1B8AoPjExgUn8Cg+HgSI6MMPS9N1q0LAtp6k4opU+xtPC5E56QPoYvTWnO0tITNecfYfPwYW/Lymi92SVHRjE5KYk76MEYkJjGyZ096RkV3uD9PavaeWLTo/KW2m67BCxe6V57wkFCiatcQfPxxkmpySbGlkDb6XI78bG0tuWWlHC0tJaeshOzSUg4XF7H6wH7KaxvTZRPitnNL8r/oHlJCNT2piv4ZqX3vIT2hB1EetihklI+wGmkh+KkGu52Nx3L59PD3fHn0CCfPVgCNAWBi335M7NOXCX370b9bnMu1W09q560v6k1cvagb0UJwdxSN1prCyrPsz1lC/anHCKK6+bkaeyhv5N3M1tKxpHSLY1iPREb07Mnonr24OLm3S0FCRvkIb5CUURdyovwM7+7ZzXt7d5FfUUFUaChX9B/A5Sn9mdC3H6luBIDWjFrMztcpI09z9O29Xgf3ZlfYCvadLmT/6UJyykqBxo720Um9mNCnHxP79mOsEwHC3VFGMixZtEdSRgFOa826nKMs3fkd63KOorXm8pRUfnPFVVydmoYtxPM/o9HpGk8ZsaCepzNx29tONeTz0wmTmn8/U1PDdyfz2XI8jy3Hj/Hat1m8vH0rwUqR0SuZy1L6c3lKKqOTehESdP7iAO6M8rHC6CQRmKSFYGF2rfnXoYO8tG0L+04XkhgZxa0jRnLr8FH069bNa8c1qoXgq76IJt5qIXT2+sq6OrbnH2dLXh4bj+Wws+AkGoi12ZjcN4Ur+qdyeUoqfWJjXXg3nperibQuApukjAJMvd3ORwf385dtWzhUUkxa9+48OG4is4akmzKO3ur3P3CWpzl6o3L8pdVVbMzNZX1uNhtys8mvaOzvGRyfwNQBA5mWNpCMXskEOZnq82R0kvRbBD5JGfm5ljU2HZzMypMz+Th/KOkJPXhhxixmDBrcPLHLlX25W/uz+v0PnOXpTFyjZvLGhUdw/ZB0rh+SjtaaQ8XFrM/N5oujR/jbN9t4eftWekRGMnPQEG4cOpzRSb067AfyZHRSRyucSkAQ0kLwsbZqbLX2MIJ7PM1Vo37udK2xvX35ovbny2GrVr93Q2tl1dWsyznKZ4cP8fnRw9Q2NDAgrjs3Dx/BzcNHkhgZdcFrPPk7y9yHwCd3TDOZkXeZaqvGFhZUS+TZ510KBu3ty9vr27d1AW5r2Kkrr3eXK8c1gzPvrVt4OHPSh/HSzBvYet/9PD31WnpERvLspg1c9vdXWfDJh2zIzcHeouLmyVLdXek+GcJ10kJwkZG1cK01674KRhlUY/NF7a+t/gZX+iCM7K+wWt+HJ+U5XFzEO3t28f6+PZRWV9O/Wxy3jxzFTcNG0iMysvMdtMMqrUjhPdJCMJFRtfCztbU8/OnHFNW2PVrInRqbL2t/ixY1XgCbGjVNP3s7heOr43rbwPgEHr98CjPzHuBP02eSFBXNMxu/5tK/v8Ijn63hYNFpt/brixsByX2b/Ye0EFxkRC28oraWH61eyTf5J1g0tpyU+t8bUmMzq/bX2WzkzmrGRs1mbs0KLQSj31vL93S4uIilu77j3b27qaqr4wdDh/OzCZO9OgTZU9IiMZ8MOzWRp2PAK2prufeD99lxMp8/T7+e64ekGzou3Owx5pIyap8R5WlrHyVVVbyyfStvfrcDu7Zz+8jRLBg/kcSoCzug3WHkZ0hWdDWfBAQTeVLjqW1o4N4PVrL1+DGem3E9Mwenu3x8q42ksVJA8Idz4wxnWxknK8p5Yetm3tu7m9CgIOZlXMz9Yy8hxmZzt8iG1+hlVJP5pA/BRO7mYLXWPPHlWjLzcnlm2nS3ggFYbyRN05wFd3P5Rs558GUwaCtP7u57W7SoMZCcux9E41fr99crOobFV1/DZz+cx7S0Qfw1ayvT3nqdlfv2nDcqyRVGj1STUU3+RVoIJnlr5w4Wrvuchy6ZyM8nXur2fqyWFmmLP5TRSN7Mk7tyLr87mc+TX33JjoJ8xvfuw7PXzCClW5xLxzO6Ri99COaTFoLF5ZeX8+zGr4nZdB0PT5js8usDdSRNoPDm/A9XWhljeiWz4tY7+N3Ua9l3upDr3/4Hy3fvpKnS58xoH6Nr9L4Y1STcJy0EEzzw8WrWZR9l/4KHvdLJaDVWy+V7W3u1aq0VV13lmzz58fIz/PLfn5KZl8tVqWk8NqaE/JyHOq2pS43e/0kLwcLWHjnEp4e/56cTJvq6KKbxl2BgVDnbqz0XFPguT94nJpa3bryZJ664ik3Hctl18FGnWjFSo+/apIXgRWdraxl6yy7yVo+94Dl3x6V3tdq3NxnV2mqvVv3UU6+ydq3vL6SHios49l2P5pTj+WS0T6CRFoJFvbt3N6FT17ElL6/TESPO8tdg0F65/fX9tNSyVq214uTJ/jz11Kt8/vlcS/T3DIpPwBYuo31E56SF4CRXJ+s02O1M/cffSYyK4r1b7gD8I//vLe29d7PPSUdj/JueN4qV/t7SN9B1SAvBy5r+mRpnXOrmWxZ2tCbLl9lHyD1TxrwxFzc/Fij3GfBnHY3x72iOh7+3ZM5rxaA4XRvHp6X3EhF3k6+LJixEAoIT3BlW+PqOb0mOjmH6oMHNj/n7RcVV7Q2XnTLF/4bRujMh0GoVgKSkuUyalM1VU+yEp2ayIncgt65YzsmKcl8XTViEIQFBKTVDKXVAKXVIKfVYG8/PU0oVKqV2OL7uM+K4ZnH1Zu1HS0vIzMvlrtEZF9xU3Vd8dcOatmrj69Y5NxPX21ouxueN4GTlADdryFCWzP4Pjp8p447336W0usrtfclqpoHD46uVUioYeAm4DhgO3KGUGt7Gpsu11hmOr9c8Pa6ZXJ2s81X2UQBmDTl/eQpf/uNYbckLK2gasdVecPLnCYHOlPGylP68Pucm8svL+emaj6m3uz8T2ZV0qrAuI6qvlwCHtNZHtNa1wDvAHAP2axlpaYsJCjr/piRBQZGkpS1uc/ttJ47TNzaWvrHnliXu6v847aVPrJZWacnZNYWsyNkKwLjefXjyqqlsOJbD7zeud/k4vrhLn/AeIwJCH+BYi9/zHI+1dpNSaqdSaoVSql9bO1JKzVdKZSmlsgoLCw0omjFcmayjtSbrxHHGJp9/Ctz9x/Hk4uNJDdfoi54/DDu1cnDypttGjOLu0Rm89u12/rlvr0uvdTWdKqzNrAT3h0Cq1no08G/gzbY20lq/qrUep7Uel5iYaFLRnNPUITdlip1Jk7LbHap37EwZhZVnGdf7/IDg7j+OJ6keT2q4XTHF1NF58Ydg4UkF4PHLpzCxTz9+/cVn7DpV4PQxZTXTwGJEQDgOtKzx93U81kxrXaS1rnH8+hpw4dTdAHHgdOOtDUf2TDrvcfnH8W9Wasm0x5MKQGhwMC/OnEVCRCQ/+9fHVNbVdXicJq6mU4W1GREQtgGDlVIDlFJhwO3A6pYbKKWSW/w6G9hnwHEt6UTFGaBxLZmWXPnH8UZnpjM1XH/uRBWei4+I5A/XzOBoaQlPb/iq3e1ath5l7aPAYshMZaXUTODPQDDwd631YqXUU0CW1nq1UuppGgNBPVAMPKC13t/RPq02U9lZT2/4ije/+5Z9//kwqtXiMe7cmtBXs12tNMtWuM6TNa8Wf72OJd9uZ8nsG7kqNe2C5+WzYW1yC00LWfDJh+w9XcgXd//IkP1JQBBmq6mv5wfLl1JUVcmaO+8hITLS6dt6Ct+TpSsspOBsBcnR0Ybtz1edmVbtRJWLj/fZQkL43+kzOVNdw+8cQ1H9eQiucJ4EBIPV1NcTHhJq2P589Q9n1X/0rjj6yReG9Ujkh6MzWLV/L9mlJb4ujjCJBASD1dnthAbLaRX+7ydjxxMSFMxfsrac97hVW4/Cc3LlMlidvYFQi6xfFChk9JNvJEZFcceo0fxz315yy0qbHzfjvMv6SL4hVy6D1TfYCQkK9nUxTOXtC4Tkr33nJxePJzgoiL9mbTXtmF19mRdfkoBgsNDgYOrtDb4uhqkkrx+4kqKjuWnYCFbt3+fRiqiukPWRfEcCgsFibTbO1NQ0/y61WGNJ/tp8Pxw1hpqGet53cZ0jd8n6SL4jAcFgrQNCoNaefZXXlwBrvmGJPbm4VzJv7/oOM+YtyTIvvtOlAoIZHVUxrQKC1Rh1QZW8ftcyd1SG48ZPxzrf2EOyPpLvdJmAYFZHVZwtnD3vjrTsqJhAbbEI75o5eAhx4eG8t3e3148l6yP5ToivC2CWjjqqWn/Q3FlzqEnf2G5EXruesvfHEGuzdYklICSvH/hsISFcmzaIT74/SE19PbYQ7146kpLmSgDwgS7TQnC2o8rTlkS/bo13STvWYty2r3k732+Flo9on1F/n5mD06moq+Xr3GyXXidzCvxHlwkIznZUeTrkLcVx28zcM2WANWrPku/v2oxKE07q249utnA++f6g06+ROQX+pcsEBGc7qjwd8pbSLQ6AnNLGFoJcdEWgCA0O5pqBA1l79DC1Dc7NtZE5Bf6lywQEZzuqPB3yFmOz0SsqmgNFpz0tsldYocUivM9bacKrUwdSUVvLdwX5Tm0vcwr8S5fpVAbnOqrS0hZz4MD882o1rg55G9GzJ3sLT7ldTm+SFkvX0PIGOUYObJjQpy8K2Jx3jPG9+3a6vc2W4kgXXfi4sJ4u00JwlhFD3kYkJnG4pJiK2lrvFbQVudALM3SPiGB4Yk8yjzk3H0HmFPgXCQhtSEqay6RJ2UyZYmfSpGyXh79d1CsZu9ZON6vb4uoFXuYXiPYYnSac1Lcf3+SfoLq+rtNtZU6Bf5GA4AUXJfcmWCmna1FtkQu8MIrRrccJffpRa29gZ0GBU9t7WsES5pGA4AWxNhsXJ/dmXfYRrx5H7hMgfGFEz54A7DttzX4y4T4JCF5yVWoae08XcrKi3OnXuHqBt8r8AglAXUtSVDTdbOF8X1zs66IIg0lA8JIpqQMAWJd91OnXWOUC7ypJb/kvdz5bSikGdu/OEQkIAUcCgpekJ/Sgd0wMXxz1btqoicwvEO5wN5inxcdzuEQCQqCRgOAlSimmDxzM+pxsztRUu/x6Vy/wvkgTSf9F19UnJpbCyrPUOTljWfgHCQhedMOQodTaG/j08CGXX2v1C6u/preEMcG8Z1Q0AKcrKzvZUvgTCQheNCapFymx3fjwwH5fF0WIZkYE855RUQCcOlthePmE70hA8CKlFLOGDGVTXi6FlWd9XRyvkf6LrifJ0UIo8JOAIEtwO0cCgpfNSR+GXWs+2L/P10XxGkkT+S93g3l8RAQAxVVVBpamY+5e1GUJbudJQPCywQkJjE3uzTt7dppyg3IhXOFuMI8OCwPgbF3ny1cYwZOLuizB7TwJCCa4feRojpSUsO3EcV8XRQhDRIY2BoTKOnMWcPTkoi5LcDtPAoIJZg4aQkyYjWW7d/q6KEIYIiQoCFtwiGktBE8u6p7e46QrkYDgBldzmRGhodw4dBhrDh2UYXoiYESGhlBp0hLvnlzUZQlu50lAcJG7ucy7RmdQ29DA27u+M6eguJcflg5i4SyFwqxeMU8u6rIEt/MkILjI3VzmwPgEpqQO4K2dO6ipr/dmEZu5syyBrEsknKXRKJOO5elFXZbgdo4EBBd5ksv88UVjKaqq5IcPyxowwjVWbLlpGufaeFPL9OyRI4+TlrZYLupeZEhAUErNUEodUEodUko91sbzNqXUcsfzW5RSqUYc1xc8yWVO7pvC0B6JrPhLTxrsdqOLBri3LIGsS2R9Vmy5aY1XWwgyf8B8HgcEpVQw8BJwHTAcuEMpNbzVZj8GSrTWg4A/Ac94elxf8SSXqZTiwXETAFixb49XyufOsgTuLmUgAaNrq22oJyw42Gv7l/kD5jOihXAJcEhrfURrXQu8A8xptc0c4E3HzyuAqcrbbU0vcTeX2VQLn5WeDsDtI0f5fS3cirXWQOLtlpsnyznU2+1U1dcTY7MZU5g2yPwB84Ws65xCAAAW4ElEQVQYsI8+QMubB+cBE9rbRmtdr5QqAxKA0y03UkrNB+YDpKRYd4xwUtJcl/OXixad+0dWCgY890ceHD+BRyZdZnj5mrizLIGsS2QdrT8zRk50b0rHNNXAm9IxgFOf7bOO4abRYd4LCDZbiiNddOHjwjss1amstX5Vaz1Oaz0uMTHR18XxqhuGDOW1b7ZzvPyM147hjWGn0t8QGDxNx1Q0B4Qww8vWROYPmM+IgHAc6Nfi976Ox9rcRikVAnQDigw4tl9auBB+OflyAJ7++isfl+Z8naUR5D4IvmF0y83TdEyZ46ZPMV5sIcj8AfMZkTLaBgxWSg2g8cJ/O3Bnq21WA/cAmcDNwBe6C630VlCwlCNHHqemJhebLYUHHlhMUuxcHhh3CX/esomvc7K5vH+qr4vpcRpBeI/RAdfTdEzTstdN90XwFnfSs8J9HrcQtNb1wALgU2Af8K7Weo9S6iml1GzHZkuABKXUIeAXwAVDU63Mk863jobO/WTseFLjuvPEus9Nm6zWEVfTCNLf4L88TccUVDQGhKToaLeOL/cnsCZD+hC01p9orYdorQdqrRc7HntCa73a8XO11voWrfUgrfUlWmtz7jxvAE/HQnd0kbWFhPDklKvJKSvlle3bvFB617iaRpA0kf/yNB2TX1GOAnpGut5CkPkF1mWpTmUr8rTzrbOL7OUpqcwanM5fsrZwtLTEs8J6SFaF7Fo8Wc6hoKKCHpFRhLoxD0HmF1iXBIROeNr55sxF9jdXTMEWHMKv136G3YddKzKqQzgrp6yUfrGxbr1W5hdYlwSETnhaa3bmItszKprfXDGFrSfyeGvntx3uz5u5VxnVIZx1uKSYgfEJbr1WWqLWJQGhE57Wmp29yN48bARTUgfw9Ib17DlV0Oa+zMi9yqqQXZezfUKnKys5XVnJYDcDgrRErUsCQieMqDU7c5FVSvHstBnEh0ewYM1HnKmpuWAbyb0Kb3J2KZKmCsvInkluHUdaotZlxDyEgGfWWOiEyEieu+567nz/XR79bA0vz5pDUIslnyT36pyWSz4I4+0oyEcBwxN7ur0PmV9gTdJCsJjxvfvy+OVTWHv0MM9vyTzvOcm9OkcW3XOeO0uRbDqWy4ieScR6cWE74RsSECzonjEXcdOwETy/NZPPDn/f/HhXyL1Kzb5zRp4jV5ciqayrY8fJfCb3S5G/VQCSgGBBSin+56ppjE7qxSOfreH7osZln7yde7XCP7i7tfuutOieL1tAWSeOU2e3c2nfFGmJBSAJCAYxejioLSSEv86cTURoKPM/WkVxVWNnsjdHATn7D27Fi6wsuuc5Z5Yi2ZCbTVhQMON69zH8+LKche9JQDCAt4aDJsfE8PL1c8ivKOcnH31gifWOwPgaaleq3bvLjHPU2b601rz6x1gOPPQzIsNCDS2HLGdhDRIQDODN4aAXJ/fmj9dcx/b8Ezz8r4+pa2jweJ8tWeFibHTt3peL7nnrvFmhBbSn8BRBV3/JO7t3GV4OGVJtDRIQDODt4aDXD0ln4ZVX8dmRQ/z800+ot9sN2S84f6GxQuBwli/L5K95dWfO2SffHyRYKa5NG2T48WVItTVIQOiAszlNM4aD3jPmYv7PZVfyyaGD/OKzzoOC0flYs2qosqR257xxjjoLZFpr1hw6yOR+KXSPiDC8HDKk2hokILTDlZymWcNB77t4HL+69HI+OniAn675iNp20kfu5mOtcDG2YoujM2a3nnxxjr45eYKcslJmDk73Sjm6wpBqfyABoR2u5DTNnIr/k7GX8JvLp/Cvw9/zwMerL+hoXrTI/Xyss//g7gQOf7zQO8sK+X13uBLIlu3aSXRoGLNaBAQjyXIW1qCseifLcePG6aysLJ8df926IKCtc6OYMsW4HL673t71Hb/9ci2T+qXw6qwfEBl6btTHl19ar+xKnbtgBjJ/fZ8dlbu0uoqJS17h1uEjeeqqaeYWTLhMKbVdaz3OnddKC6EdVsppttUfcOeoMfzhmuvYnHeMe1at4IzjpucdlVHysd5nhbSb0d7ft5fahgbuGDXG10URXiYBoR1WyWl21B9w47DhPD/jer54PZVu4eHNTf/f/nYx1dW+L7s/jUwyir++t/YCWYPdztJd33FRr2SG9Ug0t1DCdJIy6kBBwVKOHHmcmppcbLYU0tIWm57TzMxMdQSD89ls/Zk0KRuAzXnHeGjNh9TUN7DrPx9Ca2uUvSV/TaV0dR8fPMBD//qIF6+7gZmDh/i6OMIJnqSMZPnrDlhhiV5nxmdP7NuPD27/IQ98vJpdwP9mbuThCXf4vOzCv9m15sVtmxnUPZ4Zgwb7ujjCBJIysjhn+wN6x8Ty7s23M/muQ7y4bTP3fbiKsurqNl/rC4GYWw90nx85zIGi0/zn+Ann3ZdDBC4JCBbnSl+GLSSEDW8O5KkpU9l0LIcfLF/KvsJTZhW1Q/6aW++q6u12ntuyiZTYbswaMtTXxREmkYBgca6Oz1ZK8cPRGbx9061U1ddx47tv83LWVkOXu3CGrFzp317atpm9pwv55aWXExJk3GVCPhfWJp3KAayospLHv/w3nx0+xOikXjwzbTrpCT28ftymkVEtJ8cFBUXKRCM/sT3/OLetWM6c9GH88drrDNuvfC7MIfMQRJsSIiP568zZPD/jevLKypi97C1e2rbZ8BVTW+tsprSkj6yrvKaGX3y6hj4xsSy68mpD9y0rmlqfBIQAp5Ri1pChfPrDeVw7cBB/zNzIf7z7tlf7FjobGWXUiqASWIz35FdfcLz8DP87/TpiDL5nsqxoan0SEExiVu60veMkREbywnU38JeZszlZUcHsd/4f//frdefNcDaKWTOlzV5q2lcByKzPzvI9u1i5fy8Lxk9kbLLxd0STGfTWJwHBBGbdDcqZ48wYNJjPfjiPm4ePZMm325ny5hL+/u32dldOdUdbI6MaGiL57W8X+/WsZV/c68Csz87W43k88eVarkhJZcElEw3ddxOrzP4X7ZOAYAKzcqfOHqd7RARPT72WD++4i5GJSfzP1+u49q3X+ejgfowYZNDWyKiRI19l7dq5Hq8I2tWWwzDjs3OsrIwHPv6Aft268fx11xs6qqglWdHU+mSUkQnMWjnV3eOsz8nm6Q1fcaDoNGOSevHry67kkj59DStX62U0fvvbxaxd6/lFwIzlMBYtartlsHChOUHI25+d8poabnlvGQVnz7LytjsZENfd430K35JRRhZnVu7U3eNc0T+Vj+64i99Pm05BRQW3v7+cH61eydbjeR63GNpKefz61/5z83Rf3+vAm5+dsupq7v/4Aw6XFPPSzBskGAgJCGYwK3fqyXGCg4K4efhIPr/7Rzw66TJ2njzJ7e8v58Z33+bjgwfcntjWVsojONiYlEdXWA7DW5+dw8VF3Pju22SdOM6z11zH5H7SsSskZWQas1YfNeo41fV1rNy3l9e+3U52aQl9YmL50UVjuWX4SKLDwpzej9VvNOSKRYt801dh9GdnXfZRfvqvj7AFh/DX62czrrfxI4qE73iSMvIoICil4oHlQCqQDdyqtS5pY7sGYJfj11yt9ezO9h1oAcFf2bXm8yOH+du3WWSdOE6szcZtI0Zx+8jRTqUYnFm+W5hDa81r32bxuw3rGdYjkVdu+AF9YmJ9XSxhMF8GhN8DxVrr3ymlHgO6a61/1cZ2FVrraFf2LQHBenaczOe1b7L47Mgh6u12JvdL4c6Ro5k6YCC2kLZXUpflCqyhur6O33yxlpX793LdoCE8e82M5tuuisDiy4BwAJiitc5XSiUD67TWF9yFWwJCYDl1toL39u5m2e6dnCgvp5stnFlD0rlp2AjGJPVCtVoq2Wo36wHfpX+8qb339F3BSX619lMOFp3mZxMm89AlEy/4G3mTFf/+gcyXAaFUax3n+FkBJU2/t9quHtgB1AO/01qvamd/84H5ACkpKWNzci5MNQjraLDb2Xgsl/f37eGzw4eoaagnJbYbMwYPYeagIYzqmWTqhccV/ngHt86CWOv3lHemjGc3beDDg/tJjIzi99Omc2XqAMOO5wxpIZrPqwFBKbUW6NXGU48Db7YMAEqpEq31BYllpVQfrfVxpVQa8AUwVWt9uKPjSgvBv5ypqWHNoYOs+f4gm/Jyqbfb6Rsby3WDhnDdoCFtthx8yR8DQmdlbnq+rLqav2Rt4c0d3xIUpPjxRWOZf/F4l9cmMuIcSR+S+SyfMmr1mjeAj7TWKzraTgKC/yqtruLfRw6z5tBBNubmUGe3kxwdw9QBaVw1II2JffoR4YP8ta8nmXmqrQt0e+8pbsYm7vvFGX4x8VKSY2IMO54zWqaI2h5hBv44ysxf+DIgPAsUtehUjtda/7LVNt2BSq11jVKqB5AJzNFa7+1o3xIQAsOZmmo+P3KEfx0+yIbcHKrq67EFh3BJnz5M7pfCpf36Mzyxp+m3aLRKC6GztIwzQSy/vJwPDuzjgfGXMOC5P3JpvxT+z2VXMiyxp1vl8SRotpUiaou0ELzHlwEhAXgXSAFyaBx2WqyUGgfcr7W+Tyk1GXgFsNM4Ee7PWuslne1bAkLgqamvZ+vxPL7MOcrG3By+Ly4CIC48nIt79WZs796MjtqKKv0DtTXHvNoBOW3aUv77v83p6OyoU9WVwNRy24raWj49/D0r9+1lc14uGjj68COsO3qUK/qnGpKecydotpciakn6ELzLZwHBmyQgBL6Cigoy83LJzDvG9vzjJOq1zOu7AltQXfM2WoXTM+V5hqfeZ1gfREHBUnbvnk9wsPc7OjvrVHU1IKw7epRVB/bx6eHvqa5v7MS/cdhwfpA+nNf/HGdo6sudgND+RERoXNBORhl5mwQEERA2bkqhrvbYBY+fro3jf47+NyMSkxjZsycjEnsysmcSqXHd3Uo1OdvRacQom/aOdfJkf+64I/u8x1qnZRrsdvaeLmRjbg4bj+Ww5m/9iJ2xkVibjesHp/Mfw4bTR63j6FHvtHTcef/Siex7EhBEQGivdqlRrLV/yp7CU+w/Xdh874aIkBAGdo9nYHwCg+LjGRDXndS47vTvFkdUB8trOLuchhH9DJ0dq+kYWmtOV1ZyqLiI74uL2HI8j8y8XEqrG29glJ7Qg0v79Xf0u6RgCwmx5JBOK5apq/EkILQ9vVQIH7DZUtqsXYbbUlg86RoA6hoaOFRcxK5TBRwoOs3h4mK2ncjjgwP7zntNQkQEyTGxJEdHkxwdQ3JMDMnRMfSKjiE4tA8NdXltHt+s96SDk/nbN9uA8dz87tscKinmTE1N8/PJ0dFMHTCQS/v159J+KSRGRV2wj47uleCri2/TcWUimn+SgCCc5u0Zp2lpi9usXbZc2TM0OJhhiT0vGEFztraWnLJSsktLyCkr5VhZGfkVFeSUlbE5L4/y2nMX2wlxl1/QV1Gnw8gsu4lH7vqezP83uPnxpozUzQ8UcNuCwhaPn0tVNdjtVNXXcba2rvF7XR2VtbVU1tcR3zCHSyNeJjSotnn7Gnsob+ReyZbS9STPanxPs4YMZVD3eAbGxzOoewK9oqM77TOx6j2Kk5LmSgDwU5IyEk4xKxXgraBTUVvLyYpy8svLKaqqpKZsJXFVLxKmT1Gpe5B59jayysZytq6Oqro6Kuvr2Pfgwwx47o8uHSdYKSJCQ4kKDSMyNJRYm43x3bK4yPY24aqQhqBeBHX/L3om3Unf2FjiwiPcfk+SrxdtkT4E4XVd8eKjFJytraOyro6a+nq0oy+g9b9MkFJEhoYSGRpKWHCwaTOyJV8v2iJ9CMLrrJqe8KaFC2m+0FuR5OuF0SQgCKe01znqjY5Yq/CH5SwkXy+MJLfQFE4x6zagQgjfkYAgnJKUNJf09Fex2frTOOO0v+SqhQgwkjISTpP0hBCBTVoIQgghAAkIQnQJBQVLycxMZd26IDIzUykoWOrrIgkLkpSREAGu9XyFmpocDhyYDyApQHEeaSGIgCQ14nM6WvNIiJYkIIiA01Qjbpw3oZtrxN4KClYPPl1xUqFwjwQEEXDMrBGbHXzc0d7kwUCeVCjcIwFBBBwza8T+kI6RSYXCWRIQRMAxs0bsD+kYmVQonCWjjETAcea+CkbxlzWeZFKhcIa0EETAMbNGLOkYEUikhSACklk1YlmCWgQSCQhCeEjSMSJQSMpICCEEIAFBCCGEgwQEIYQQgAQEIYQQDhIQhBBCABIQhBBCOEhAEEIIAUhAEEII4SABQQghBCABQQghhIMEBCGEEIAEBCGEEA4eBQSl1C1KqT1KKbtSalwH281QSh1QSh1SSj3myTGFEEJ4h6cthN3AfwDr29tAKRUMvARcBwwH7lBKDffwuEIIIQzm0fLXWut9AEqpjja7BDiktT7i2PYdYA6w15NjCyGEMJYZ90PoAxxr8XseMKGtDZVS84H5jl9rlFK7vVw2f9EDOO3rQliEnItz5FycI+finHR3X9hpQFBKrQV6tfHU41rrD9w9cFu01q8CrzqOm6W1brdfoiuRc3GOnItz5FycI+fiHKVUlruv7TQgaK2nubtzh+NAvxa/93U8JoQQwkLMGHa6DRislBqglAoDbgdWm3BcIYQQLvB02OmNSqk8YBLwsVLqU8fjvZVSnwBoreuBBcCnwD7gXa31Hid2/6onZQswci7OkXNxjpyLc+RcnOP2uVBaayMLIoQQwk/JTGUhhBCABAQhhBAOlgkIsgzGOUqpeKXUv5VS3zu+d29nuwal1A7HV0B11Hf2d1ZK2ZRSyx3Pb1FKpZpfSnM4cS7mKaUKW3wW7vNFOb1NKfV3pdSp9uYnqUbPO87TTqXUxWaX0SxOnIspSqmyFp+JJ5zasdbaEl/AMBonVKwDxrWzTTBwGEgDwoDvgOG+LrsXzsXvgcccPz8GPNPOdhW+LquX3n+nf2fgP4GXHT/fDiz3dbl9eC7mAS/6uqwmnIsrgIuB3e08PxNYAyhgIrDF12X24bmYAnzk6n4t00LQWu/TWh/oZLPmZTC01rVA0zIYgWYO8Kbj5zeBH/iwLL7gzN+55TlaAUxVnayh4qe6yme+U1rr9UBxB5vMAf6hG20G4pRSyeaUzlxOnAu3WCYgOKmtZTD6+Kgs3pSktc53/HwSSGpnu3ClVJZSarNSKpCChjN/5+ZtdOPQ5jIgwZTSmcvZz/xNjjTJCqVUvzae7wq6yvXBWZOUUt8ppdYopUY48wIz1jJqZuYyGFbX0blo+YvWWiul2hsb3F9rfVwplQZ8oZTapbU+bHRZheV9CCzTWtcopX5CY8vpah+XSfjWNzReHyqUUjOBVcDgzl5kakDQsgxGs47OhVKqQCmVrLXOdzR5T7Wzj+OO70eUUuuAi2jMN/s7Z/7OTdvkKaVCgG5AkTnFM1Wn50Jr3fJ9v0ZjH1RXFDDXB09prc+0+PkTpdRflFI9tNYdLgDobymjrrIMxmrgHsfP9wAXtJ6UUt2VUjbHzz2ASwmcJcWd+Tu3PEc3A19oR29agOn0XLTKk8+mcUWArmg1cLdjtNFEoKxF6rVLUUr1aupTU0pdQuO1vvMKk697y1v0it9IY86vBigAPnU83hv4pMV2M4GDNNaEH/d1ub10LhKAz4HvgbVAvOPxccBrjp8nA7toHHWyC/ixr8tt8Dm44O8MPAXMdvwcDrwHHAK2Amm+LrMPz8XTwB7HZ+FLYKivy+yl87AMyAfqHNeKHwP3A/c7nlc03ozrsON/os3RioHw5cS5WNDiM7EZmOzMfmXpCiGEEID/pYyEEEJ4iQQEIYQQgAQEIYQQDhIQhBBCABIQhBBCOEhAEEIIAUhAEEII4fD/ATlMOPI7GFFmAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Find indices of positive and negative examples\n", "pos = np.where(y==1)[0]\n", "neg = np.where(y==0)[0]\n", "\n", "# Plot examples\n", "plt.plot(X[pos, 1], X[pos, 2], 'b+', label='Admitted')\n", "plt.plot(X[neg, 1], X[neg, 2], 'yo', label='Not admitted')\n", "plt.legend()\n", "\n", "# Here is the grid range\n", "u = np.linspace(-1, 1.5, 50).reshape(50)\n", "v = np.linspace(-1, 1.5, 50).reshape(50)\n", "z = np.zeros((len(u), len(v)))\n", "\n", "# Evaluate z = theta*x over the grid\n", "for i in range(len(u)):\n", " for j in range(len(v)):\n", " z[i,j] = map_feature(u[i], v[j], degree=6).dot(theta)\n", "\n", "# Plot z = 0\n", "# Notice you need to specify the range [0, 0]\n", "#z = z.reshape(len(u), len(v))\n", "plt.contour(u, v, z.T, 0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exploring regularization\n", "As mentioned before, $\\lambda$ is used to control the problem of overfitting. In the following graphs, models trained with different values of lambda are shown to give you some intuition on the problem of over/underfitting." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "def create_plot_for_lambda(X, y, _lambda):\n", " from scipy.optimize import minimize\n", " result = minimize(compute_regularized_cost, initial_theta, args = (X, y, _lambda),\n", " method = 'CG', jac = compute_regularized_gradient, \n", " options = {\"maxiter\": 400, \"disp\" : 1})\n", " theta = result.x\n", " print('Conjugate gradient found the following values for theta - first five values only: {}'.format(theta[:5]))\n", "\n", " # Find indices of positive and negative examples\n", " pos = np.where(y==1)[0]\n", " neg = np.where(y==0)[0]\n", "\n", " # Plot examples\n", " plt.plot(X[pos, 1], X[pos, 2], 'b+', label='Admitted')\n", " plt.plot(X[neg, 1], X[neg, 2], 'yo', label='Not admitted')\n", " plt.legend()\n", "\n", " # Here is the grid range\n", " u = np.linspace(-1, 1.5, 50).reshape(50)\n", " v = np.linspace(-1, 1.5, 50).reshape(50)\n", " z = np.zeros((len(u), len(v)))\n", "\n", " # Evaluate z = theta*x over the grid\n", " for i in range(len(u)):\n", " for j in range(len(v)):\n", " z[i,j] = map_feature(u[i], v[j], degree=6).dot(theta)\n", "\n", " # Plot z = 0\n", " # Notice you need to specify the range [0, 0]\n", " #z = z.reshape(len(u), len(v))\n", " plt.contour(u, v, z.T, 0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Overfitting\n", "Remember: overfitting means the model probably only works on the training set and not well in the real world. If you model is overfit, consider adding regularization or choosing a higher value for $\\lambda$.\n", "\n", "You can check out other values of $\\lambda$ yourself." ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Warning: Maximum number of iterations has been exceeded.\n", " Current function value: 0.281767\n", " Iterations: 400\n", " Function evaluations: 1435\n", " Gradient evaluations: 1435\n", "Conjugate gradient found the following values for theta - first five values only: [ 2.53231573 -1.3591363 2.44932572 -6.28390865 -8.54930906]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xd8VFX6+PHPSaeEFkihhtB7KIpBkCpVQEURRFRcF93f6uquurrrroCu311Xd+2uDRRXOouIICCINAE1IL0TAgRCSKEkJJkkk/P7I8UQUqbcmTszed6vFy8yM3fuPXOTuc897TlKa40QQgjhZ3YBhBBCeAYJCEIIIQAJCEIIIYpJQBBCCAFIQBBCCFFMAoIQQgjAoICglJqjlLqglNpfyeuDlFKXlVK7i/+9YMRxhRBCGCfAoP18CrwDfFbFNlu01rcZdDwhhBAGM6SGoLXeDGQYsS8hhBDmMKqGYIs4pdQe4BzwtNb6QPkNlFLTgekAderU6d2xY0c3Fu9aFquVkxcziKwbSoOQENPK4avOXLlMncAgGtWqZXZRhPApO3fuTNNaN3Hkve4KCLuAVlrrLKXUaGA50K78RlrrD4EPAfr06aPj4+PdVLzrXbFYeGTlcn44m8SkPn15om8cgf7+ppXHl5zISOfWzz/l2ZsH8EjvG80ujhA+RSl1ytH3umWUkdb6itY6q/jnr4FApVRjdxzbUfWCg/l0/ATu6tyF9+J/4K4lC0i4KK1iRlh0YB8Bfn7c2amL2UURQpThloCglIpUSqnin28sPm66O47tjOCAAP45bCTvjh7LmSuXGbvgv8zftwdJCOi4PKuVZYcOMrR1G5rUrmN2cYQQZRjSZKSUWgAMAhorpZKAGUAggNb6feAu4DdKqQIgB5ikveiqOqpte3pGRvHMujX85bv1bDtzmteGjyQkINDsonmd9QnHycjNYVKXbmYXRQhRjiEBQWs9uZrX36FoWKrXiqwbytzb7+LDnT/x6rYtJGdl8uFttxNWu7bZRfMqC/fvI6puKP1btjK7KMJO+fn5JCUlkZuba3ZRBBASEkLz5s0JDDTuxtSdo4y8np9SPNrnRlo1aMAf1q7mzsXzeWPEaHpGNTW7aF7hzOXLbD1ziif6xuHvJ5PkvU1SUhKhoaFER0dT3AIsTKK1Jj09naSkJFq3bm3YfuVb6YBRbduzYMJECgoLuWvJAv62eSM5+flmF8vjLT64Dz+luLtzV7OLIhyQm5tLWFiYBAMPoJQiLCzM8NqaBAQHxUZGsWbKA9zbrQdzdu9k1Ly5xJ87a3axPFZBYSFLDx5gYKvWNA2tZ3ZxhIMkGHgOV/wuJCA4ITQ4mJcGD2PBnRNRSnHvssUs2L/X7GJ5pC2nE0m5msU9XaR2IISnkoBggL7NW/DlpCnc3KIlz29YxwvfrSffajW7WB7li8MHaRgSwqDoGLOLIrzc8uXLUUpx+PDhCl9/8MEHWbp0qc37O3fuHHfddRcAu3fv5uuvvy59bePGjWzbts3uMkZHR5OWlmb3+8wmAcEg9YJD+HjsHUzv1YfP9+1h6hdLOXnpotnF8ghXLLmsO3GCMe06ECSzvWucmTON3d+CBQvo378/CxYsMGR/TZs2LQ0gRgUEbyUBwUD+fn48138g/x4+ioNpFxj1+Vxe3/E9uQU1u8P54107sVgLmNS1u9lFESaYNcu4fWVlZbF161Zmz57NwoULgaIRN4899hgdOnRg2LBhXLhwoXT76Oho/vSnPxEbG0ufPn3YtWsXI0aMoE2bNrz//vsAJCYm0rVrV/Ly8njhhRdYtGgRsbGxvPLKK7z//vu8/vrrxMbGsmXLFlJTU5kwYQI33HADN9xwA99//z0A6enpDB8+nC5duvDwww977eRVGXbqArd37Ey/Fi35+9bNvP3jDr4+dpT3Ro+jXViY2UVzu4ycbObs3slt7TrQuUm42cURXu7LL79k5MiRtG/fnrCwMHbu3MmpU6c4cuQIBw8eJCUlhc6dO/PQQw+Vvqdly5bs3r2b3//+9zz44IN8//335Obm0rVrVx599NHS7YKCgnjxxReJj4/nnXeKpk3l5ORQt25dnn76aQDuvfdefv/739O/f39Onz7NiBEjOHToELNmzaJ///688MILrFq1itmzZ7v3xBhEagguEl6nLq+PGM1nt9/Fpdxc7lg8j5VHK27z9GVz9/xMdn4+j98YZ3ZRhBvNnAlKFf2DX352tvlowYIFTJo0CYBJkyaxYMECNm/ezOTJk/H396dp06YMGTLkmveMGzcOgG7dutG3b19CQ0Np0qQJwcHBXLp0ya7jr1+/nscee4zY2FjGjRvHlStXyMrKYvPmzdx3330AjBkzhoYNGzr3QU0iNQQX69+yFV9Nvo/HV6/kd2tWset8Ms/dfEuNaEu/mpfHZ3t2MzymbY2sHdVkM2f+cvFXCoxoQcnIyGDDhg3s27cPpRRWqxWlFHfccUeV7wsODgbAz8+v9OeSxwUFBXaVobCwkB07dhDioynxpYbgBpF1Q5l/50Smxfbi0927uHvJAnYlnzO7WC638MA+LltyeaT3DWYXRfiApUuXMnXqVE6dOkViYiJnzpyhdevWhIWFsWjRIqxWK8nJyXz33XcOHyM0NJTMzMxKHw8fPpy333679PHu3bsBuOWWW5g/fz4Aq1ev5uJF7xxQIgHBTQL9/fnrLYN5Z9RYUq5mcdeSBfxuzUou5uSYXTSXyLNamb0rnpuatZDUHjXcjBnG7GfBggXX1QYmTJhAcnIy7dq1o3Pnztx///3ExTnePDl48GAOHjxIbGwsixYtYuzYsXzxxRelncpvvfUW8fHxdO/enc6dO5d2TM+YMYPNmzfTpUsXli1bRsuWLZ36rGZRntobbvYCOa50NS+PD3f9xAfxPxFWuxZvjBzDDU2bm10sQy09uJ8/rl/LJ+PuZGC0cblWhHkOHTpEp06dzC6GKKOi34lSaqfWuo8j+5MaggnqBAXx+5tuZsnEyQT5B3Dv/xbz7k8/UOihwdlehVrz0a54OjVuwi2tos0ujhDCRhIQTNQtPIIVk+5jVLv2/Gv7Vh5YvpTUq1fNLpbTvjuZwLGMdKb3vkFy3wjhRSQgmCw0OJg3R4zh70NuJf7cOYZ8NpvXd3zPFYvF7KI57INdP9EstB6j27Y3uyhCCDtIQPAASinu6dqdVfdOZWCr1rz94w5GzZvLj2eTzC6a3XafTyb+3Fke6tmbwBowtFYIXyIBwYPENGzEO6PHsmzivQT5+3PvssX8e/v3XpUo79M9u6gbFCRrHgjhhSQgeKDYyChWTp7KnZ06885PO5j0v0WcvmzfjEozpGRl8fWxo9zduSt1g4LMLo4Qwk4SEDxUnaAg/jlsJG+OHMPxjAxum/9fZv+806NXZpu/fw/WwkKmdo81uyjCBymleOqpp0ofv/baa8ysJhfG8uXLOXjwoFPHtTeV9YoVK/jHP/5R4fE//fRTzp2zb1JqSfI9d5CA4OHGtu/I1/feT/fISF7espEBn3zEkoP7PS6b4uXcXObt3cPg6BiiG3hnHhdhrJSUeWzfHs3GjX5s3x5NSso8p/YXHBzMsmXL7Lo4GxEQ7DVu3Diee+65Co/vSEBwJwkIXqBZvXp8fsfdLLrrHto0asSz69fy26+/4lKu58xyfvenHVzMzeHJm/qZXRThAVJS5nHkyHQsllOAxmI5xZEj050KCgEBAUyfPp3XX3/9utcSExMZMmQI3bt3Z+jQoZw+fZpt27axYsUKnnnmGWJjYzlx4sQ17/nqq6/o27cvPXv2ZNiwYaSkpACVp7JOTEykY8eOPPjgg7Rv354pU6awfv16br75Ztq1a8ePP/4IFF30H3vsseuO/8orrxAfH8+UKVOIjY0lJyeHnTt3MnDgQHr37s2IESNITk4GYOfOnfTo0YMePXrw7rvvOnzO7Ka19sh/vXv31uJ61sJC/UH8j7r92//WcR+/r78/fcrsIulTly7qDm+/rp9Zt9rsoggXOnjwoM3bbtvWSn/3Hdf927atlcPHr1Onjr58+bJu1aqVvnTpkn711Vf1jBkztNZa33bbbfrTTz/VWms9e/ZsPX78eK211g888IBesmRJhfvLyMjQhYWFWmutP/roI/2HP/xBa631448/rmfNmqW11nrlypUa0KmpqfrkyZPa399f7927V1utVt2rVy89bdo0XVhYqJcvX156zE8++UT/9re/rfD4AwcO1D/99JPWWuu8vDwdFxenL1y4oLXWeuHChXratGlaa627deumN23apLXW+umnn9ZdunSp8DNU9DsB4rWD113Jdupl/JRieu8b6NeiJU+uXcV9Xyzh4Z69efKmm6kdGGhKmV7bthU/P8UfbrrZlOMLz2OxnLbreVvVq1eP+++/n7feeotatWqVPr99+3aWLVsGwNSpU/njH/9Y7b6SkpK45557SE5OJi8vj9ati1KsbN68uXRf5VNZt27dmm7dugHQpUsXhg4dilKKbt26kZiYaNdnOXLkCPv37+fWW28FwGq1EhUVxaVLl7h06RK33HJL6edZvXq1Xft2lDQZeamu4RF8NWkqU7r14OOfd3LznA95ddsWMnKy3VqO3eeTWXnsCL/u1YfIuqFuPbbwXMHBFSd3q+x5ezz55JPMnj2bq07O6n/88cd57LHH2LdvHx988AG5ubnVvqd8+uyyqbXtTaWttaZLly7s3r2b3bt3s2/fPr755hv7PoTBJCB4sVqBgbw0eBhL755MXPMWfLDzJ0bP/4xtZ5y7C7PH6zu+J6xWbX7dy/4U10Z3OgrPERPzMn5+ta95zs+vNjExLzu970aNGjFx4sRrViXr169f6ZKa8+bNY8CAAcD16avLunz5Ms2aNQNg7ty5pc8bmcq6qnTaHTp0IDU1le3btwOQn5/PgQMHaNCgAQ0aNGDr1q2ln8ddJCD4gF5RTXlvzDi+mnQfoUHBTP1iCa9t20pBYaFLj3sw9QJbTp/ioZ697J534IpOR+E5IiKm0KHDhwQHtwIUwcGt6NDhQyIiphiy/6eeeuqa0UZvv/02n3zyCd27d+e///0vb775JlC0qtqrr75Kz549r+tUnjlzJnfffTe9e/emcePGpc8bmcq6/PEffPBBHn30UWJjY7FarSxdupRnn32WHj16EBsby7Zt2wD45JNP+O1vf0tsbKxbRxRK+msfk52fz6xNG1hycD+9o5ryxogxNKtXzyXHenLtKr5NOMH3D02nXrB9K0ht3x5dHAyuFRzciri4RINK6B4pKfNISHgei+U0wcEtiYl52bALnyeR9NeeR9JfiyrVDgzklWEjeGPEaI6kpTFq/lz+vnUTp+xcO7Y6SVcus+roESZ17W53MADXdTq6m9R0hC+RgOCjxnXoxFeTp9K/RSvm/LyTwZ/N5tn1a7HY2fFVmc/2/IxSimmxvRx6vys7Hd0pIeF5Cguv7cgvLMwmIeF5k0okhOMkIPiwVg0a8N6YcWyZ9mse7tmbJQf3M3X5UkNGIu1IOsMNTZvTNNSx5ihXdjq6k6/UdGzlqU3MNZErfhcSEGqAyLqh/HnAIN4aOYa9KeeZsHgBCRczHN6fpaCAI+lpdIuIcHgfru50dBdfqenYIiQkhPT0dAkKHkBrTXp6OiEh9jfXVkUmptUgt7XvSNPQejyycjkTFi/g3dFj6dfC/gvX0Yx08gsL6dbE8YAARUHB2wJAeTExL3PkyPRrmo28saZji+bNm5OUlERqaqrZRREUBejmzY1di10CQg3TK6opyyZO4VcrlnHfF0voGRnFQ7G9GdO+g8372H+hKOdL13DnAoIvKAloNWGUUWBgYOlsXuGbJCDUQC3q1+d/E+9l4YG9LDmwn8fXrKRpaCg9o5ra9P7s4hTcl3JzaEUDVxbVK/hCTUcIMKgPQSk1Ryl1QSm1v5LXlVLqLaXUcaXUXqWUY0NTaihXzOgNDQ7m171u4It7phBZpy4vbPwWq40T2SZ27kqT2nWYtek7Ct3UniyzmoVwPaM6lT8FRlbx+iigXfG/6cB/DDquz3P1OPc6QUE8P2AQB1IvMH//XpveExoczLM3D2B3SjJfHLYv17wjF3YZ6y+EexgSELTWm4Gqhq2MBz4rzs66A2iglIoy4ti+zh3j3Ee3a0+/Fi351/atpGXbNiT19o6d6RkZxSvfbybTYrHpPY5e2GWsvxDu4a5hp82AM2UeJxU/dw2l1HSlVLxSKl5GMhRxxzh3pRSzBg7hisXCZ3t+vu71ilYp9FOKFwYOIT07m6nLl7Ix8WS1wxEdvbDXtLH+QpjFo+YhaK0/1Fr30Vr3adKkidnF8QjuGueeXTyDObJu3etemzWr4vf0iIjktVtHkZZ9lYdWLGP8ws/Zevr6/EQlHL2w16Sx/kKYyV0B4SzQoszj5sXPiWq4a0bvF4cOEOTnz5h2tg8/BbijU2c23P8rXhk2gsy8PO5fvpRn16/lcgW55R29sPvKrGYhPJ27AsIK4P7i0UY3AZe11sluOrZXc8eM3nyrla+OHmZoTBvqF898nDkTlCr6B7/8XFHzUZC/P3d37sqaKQ/wmz43suzQAYZ//ilrjh+7ZjtHL+zumtUsI5lETWdI+mul1AJgENAYSAFmAIEAWuv3lVIKeIeikUjZwDStdZW5rSX9tfNsTcu87sRxHln1JR/ddjtDY9pc97pSYM+fyYELKTy7fi0H01IZ3qYtj91wU+kkNlelinZ2vyUd3uVnHHtjOg1RszmT/lrWQ/BQ7rrAnc28wp2L5hMSEMD6qdMI9Pe/bl/2BgQoqnV8/HM87/y4g5yCAnpGRvGrnn0Y0aYt/n7GVkyNuJj70voMomaT9RB8jBHj7m0Z0ZNpsfDwii/IKcjno7G3VxgMAGbMsP8zBPr785s+fdn+q0f46y2DycjJ4bHVXzFi3qcsPbiffKvV/p1WwohhqTV1JJM0k4myJCB4IHdc4AoKC/ndmpUcz0jn3dFjaR/WuMLtoeJ+A1vVCw5hWmwv1k+dxtsjbyPEP4A/rl/LoLmzmbtnFznFaTCcYcTFvCaOZJIJf6I8CQgeyB0XuPfjf2TTqUReGjyMAS2j7S6jvfz9/BjTvgNfTZ7KnHF30rxePWZt+o5bPv2Id3/6gSs2Tm6riBEX85o4kkkm/InyJCB4IFdf4I6kp/H+zh8ZHtOWSV27O1VWeymlGBTdmkV3TWLhhHvoGh7Jv7Zvpf+cD3lp83ecvmz/Up9GXMx9ZX0Ge9TUZjJROcl26oGMyLFfWVpm/9A7+PXiedQNCmLGwCGGl90eNzZrzo3NmrP/Qgqzf97Jf/fu5tPdu7g1pi3TYntxY7PmqJJxr1UwKgV1TctaGhzcspKOdN9tJhNVk1FGHsoVwzMtBQXc98US9l+4wMK77qFHRKRBpTVGSlYWn+/bzfx9e7iYm0vnxk14MLYXt7XvQEhAoNnF8zky1NY3ybBTUS2tNc+sW8Oywwd5e+Rtdi2I4265Bfl8efgQn+zexdGMdOoFB3Nnpy7c27U7bRuFmV08Q7lqXoa3HF8Yz5mAIE1GNYDWmrd/3MGywwd5om+cRwcDgJCAQA4t7c7qGd344WwS8/fvYV5xc9KNTZszqWt3RrZt6/W1hvJ36CWjfAC3XZRrWjOZqJrUEHxcodb8bctGPt29izs6dua1W0fa1C5vhJkzHR+yWn4yXFp2NksP7mfh/r2cvnKZesHBjO/QiTs7daF7eITbPpORZDKccAVpMhIVshQU8PS61aw6dpSHYnvz5wED8XPjhdORGc7VvbdQa35IOsPCA/v45sRxLNYC2jUK446Onbm9Yyci64Y6V2g32rjRD6joBCkGDbJt9TohypOZyuI6BYWF/H7t16w6dpQ/9x/IX24ZZHMwcGYimjNsSajnpxRxLVry5sgx/PDwI/zfkFupFxzMP7dtof8nH/HA8qV8eeRQ6YQ3Z2fiunImb02cDCc8m9QQfFCh1jy7fi3/O3SAv94ymGmx9i1h7cyd/cyZFa+fMGOGfYHG3jIkXrrIF4cPsuzQQc5mXqF2YCD3tznNDcHvovQvqbjtGUXj6lE4MspHuII0GYlS2fn5PP3NatacOMYTfeN4om8/u/fhTEAwaj+OvrdQa348m8RXRw/T23oPjQIvXreNrW307mjjd3SUj4wOEpWRJiMBwPmsTCYtXcjaE8f4c/+B/O7GOJvfa8/6B+7gSEI9KGpSuql5C14eciuNAiue9ZxrOc2W04nkVZNgzx0zeSMiphAXl8igQYXExSXaVXORHETCaDLs1EfsTTnP9JXLuZqXx4djb2do6+vXNahK2RFBRtUQHL2ol5THWZXNxM3Ia8Azy/9H3aAgBrVqzdCYNgyObk294BCb3m92G39VOYikdiGcIQHBB6w6eoSn162hce3aLLl7Mh0be8Z61GbVLkpUlgLkxi5v8FHbm1mXcJxvTyaw8tgRAvz8uKFpcwa2iubmFi3p1CTckBQiruBMzcUT5j4IzyUBwQM4esdWqDX/if+Bf23/nt5RTfnPmPFYM79g+3bn7v6cubP3JFXlOGoBDI1pg7WwkD0p51mXcJwNJxP4x/ebAWgYEsKAVtEMD59J/Zy3yc9L8pi7aWdqLs7WLoRvk05lkzk60uSKxcIz61azLuEE4zp05JWhI7iUvsgjRq04MyHN7OOmZGWxPek0W0+fYtOpk6Tn5OCvFL2imtKvRUvimrekR0QkwQHm3Us5MzpJ5j74Phll5GZGtsE6MpLlSHoav1m1gqQrl/lT/4E82KMnSilTZr5WdBG2pw/CyOBhVN9HiUKt2Ztynm9PnmBT4kkOpF5AAyEBAfSOaspNzVtwU/MWdA+PrHC1OVcGRkf/BmV2tO+TgOBGRo8dt/eO7aujh3lu/VrqBgXz9qjbuLFZc4f3ZYSKLsL2XJiNvIgbHRDKu5ybyw9nz7Aj6Qzbk85wJD0NKAoQnRs3oVtEJN3CI+gaHkGbho0I8PdzaXkcIXMffJ8MO3Ujo1eZsnW2ap7Vykubv+OJNavo3CScrybfd00wsGdfrmDWsFV3Hrd+SAjD27TjhYFDWD3lAX56+De8M2osk7p2x8/Pj8UH9vH0ujWMnDeX7u+/DcCsTRtYfGAfh9NSsRbaH5SN/hxmLAQk6zZ7D6kh2Mnou3Bb7tgSL13kd6tXsj/1Ag/06Mmf+g8kqIImCnfd/VU3G7m6O3WjZjOX5+oaQnWshYU8+Vwu77xa+7rXGozcRvNx8XSPiKRHRBSxkZF0C48ksm7dKhPzmf2ZnCU1EveTJiM3ckUbbFXtwSuPHubP367D38+Pfwwbzog27RzelyvUpCYjeykF1kJN4qWL7Dl/nt0pyew+n8yhtFQKimsLDUJC6NQ4nE6Nm9CpSRM6NW5C20ZhpQHfjM9kdh+ZcI4EBDdy1x2PpaCAl7ZsZP6+PfSKjOLNUbfRLLSeaSN4KuNJAcEbzg0U/W4PpF5g/4UUDqelcjAtlSNpaVisBQBcXtOPjNXXzzL/6wuaF2e5Nlut2X1kwnkSENzM1XfhCRczeHz1Sg6lpTK99w08ddPNpaNYPO0uuOQi7GgzkKddxB1V0d/Ef/4zxebPVlBYSOKlixxKS+VQaipH09M4lpHO5mkP0/rNfwEQ5OdP64YNaR/WmPZhjekQFkaHsCY0q1fPsLTmRt/RSw3B/SQg+AitNUsPHeDFTRsI8vfnteGjGBwdc802nhYQKuINZTSSK2uNSsHelBSOFQeIo+npHMtII+nKldJt6gQG0j0iktjIKG5q3oI+Uc2oFejYanJm9JEJY8kSml6iqrvhK5Zcnl3/DWtPHOOmZi341/BRRIWGlr6v7N13yc2gs52wwhiunP07YwZ0C4+gW3jENc9nWiwcz0jnSHoah9JS2X0+mY92xfOf+B8J8vOnZ1QU/Vq0pF+LlqXzJGyp2Rqdv6mq2eLC80gNwY0qu3M+cCGF3379FeeyMnmmX39+1bNPpU0A3nD37SvNQLaq7K5aa8Xgwe5rJ7+al0f8ubNsSzrNtjOnOVg8ka5OYCD3tDpJv9rv40fVa0PIHb33kxqCl9Jas/DAPmZt2kBYrVosmDCR3lHNzC6W07wlGBgVuCq7q05JcW9W1DpBQQyMbs3A6NYAXMzJYXvSGbYlnaarddY1wQAqrsXIHX3NJjUEF6uss/XPf7GSO+Abvjh8kAEtW/H6iNE0qnX9+PWK9uctF1xPZ1Rtq7K76hdf/JD16z3jQlp5LQZ219rKQ7G9qR8Scv0bhdeRmcoebObMoi9dyYVHaziSlsZP7T9j+eGDPNm3H3PG3WlTMCjZnzeqrNze+nnKKjv7V2vF+fOtePHFD/n22ymmLzRUorI+gKuFjXnnxx0Mmjubj3fFYykocHPJhCeRGoKNjBhqqhQsP3yIP337DXUCg3hj5GhubtHKRSX2LJXdjbu7T6Sq4bElrxvFk/p7quobyPAfzj+/38KmUydpFlqPp+L6M65DR8OGsgr3kmGnLmZER5vWmtumn+VQt0Xc2LQ5b40aQ3iduq4qssfxlIBQ1bGrKosjTXWeFBCg+pua78+c4pWtm9mfeoEuTcJ59uZb6N+yZtyw+BIJCC7m7OSagsJCZm7awPx9e0rXLjAzn767VHY3PnAgbNp0/fPuHkZrT0Bw5OLujf09hVrz1dHDvLZtK2czrzCiTTv+MXS49C94EdP7EJRSI5VSR5RSx5VSz1Xw+oNKqVSl1O7ifw8bcVx3cWbJQktBAb/9egXz9+3h0d438u/ho00JBmYtWFO+/0Rr2Lix4ufdXcayyfhckS3V24IBgJ9SjO/QifVTp/F0XH82nDzB7YvmcT4rs9L3SDZT3+F0QFBK+QPvAqOAzsBkpVTnCjZdpLWOLf73sbPHdSdH00pnWiw8tGIZ6xJOMGPgYB5oe5ofdrQ25YtT0Z16TVdyB19ZcDIrpbcRnC1jcEAA/++Gvsy7cyLp2dncu2wJKVlZ121X0pxaVIPWpWs0S1DwTkbUEG4EjmutE7TWecBCYLwB+/UYMTEv4+d37Sig6hZbT8/OZsoXS/jxbBL/Hj6akZGHavQXp7J1mj15/eaqgoWnM+oGoE/TZnxy+52kXs3i3mUCY3ujAAAgAElEQVSLuXD12qBg9PogwlxGBIRmwJkyj5OKnytvglJqr1JqqVKqRUU7UkpNV0rFK6XiU1NTDSiaMexdVCQ5M5NJ/1vEsfR0Prjtdm7v2MnhL44zFx9n7nCNvuh5w7BTTw5OZuod1YxPxk8gpYKg4ExzqvA8TncqK6XuAkZqrR8ufjwV6Ku1fqzMNmFAltbaopR6BLhHaz2kqv16UqeyPbLz85mweD5nr1zh43F3lK5q5mjSMKNGqti7H08bIWM2b+ggdtXCQyV+OpfEtC+XEVU3lCV3T6JBSC3JZuqBzO5UPguUveNvXvxcKa11utbaUvzwY6C3Acf1SC9v2cjR9DTeHT32miUuzVzeUjjP04MBuL6J64amzZkz7k52LejIjI0bAMeaU4XnMiIg/AS0U0q1VkoFAZOAFWU3UEpFlXk4DjhkwHE9zoaTCSzYv5df9+rDgFbR17xmzxfHFZ2ZtjSHeHMnqnCPG5s1J2N1HF8dPcw3J46ZskazcB1D5iEopUYDbwD+wByt9ctKqReBeK31CqXU3ykKBAVABvAbrfXhqvbpbU1G6dnZjJo/l8a16/DFxHsrHFrqyGxns5pupMnIu7myiUspGPbZJwT6+7Nq8tQq14QW7icT00ymtebRVV+yKTGR5ZOm0LFxE8P2LQFBeILK+iceeOIin77R0O3lEZUzuw+hxltz4hjrEk7wdL/+hgYDMG/ki6eOuJHmK3OU75+wFFjpN/sD/IZ8Z2q5hLEkIDipoLCQf23fStuGjZgW28vw/Zt1AfTUC69MsPMMQf7+DItpyw9JZ8izWs0ujjCIBAQnLTt0gISLF/lDXH/8/eR0Ct9XUnvs16IlOQUF7D6fbG6BhGHkCuYES0EBb/6wne4RkYxo09bs4vgsGf3kWUrO+03NWuCnFNvOGD8JTfIjmUMCghMWHdhHclYmT8f1r9EjLVx9YfbmFBK+rH5ICF3DI9iWZGxAkPxI5pGA4IQVRw7RtUl4jc8ZL+36NVfX8AgSMjIM3afkRzKPBAQHZeRk8/P5ZIa0blPldnIXayxPHf1UU4XXrkNGbo6hHcuSH8k8EhActCPpDBoYFN26yu189e7ZrHZ9CbCeJbxOHQDSsq8atk9J82KeGhUQjOyo2ptyniA/fzo3CTewhK5n1AVV2vUFQJPigJB61biAIPmRzFNjAoLRHVXHMjJo06gRQf7+173myaNifLXGIswRVqvowp2Rm2PYPiU/knl8f2HfYlV1VJX/Q7Ml51BuQQF1goIqPFbZPDI1IQWEtOvXXAWFRanbA5Sx95YREVMkAJigxtQQbO2osrUmkWctqLB24IlcXWPxhJqPqJwrfz8lnclVfRdkToH3qDEBwdaOKluHvOVZrQT6VR8QPOHuWdr7azZXNhNWFxBkToF3qTEBwdaOKltrEvmFhQT5V3/65KIrfFmetQCoPCDInALvUmMCgq0dVbbWJBoEh5CRY1xHmrt4Qo1FuJ67BjaUfAca1qpV4esyp8C71JhOZbCtoyom5mWOHJl+zV1NRTWJlvXrsyExwSXldCWpsdQM7hrYkJyViZ9ShNepW+HrwcEtK1lzWeYUeKIaU0Owla01iZb1G5CWnU1WXp45BS1HLvTCDMlZmYTXrkNAJZl+ZU6Bd5GAUIGIiCnExSUyaFAhcXGJFdYqohs0AODUpYsuKYO9F3iZXyAq48pmwuTMTCLrhlb6uswp8C41qsnISN3CIwHYdCqRLuERhu9/1iy56xfGcOXf0dnMTDpXs0qgzCnwHlJDcFCL+vW5qVkLFh7Yi7V4co67efKMaOH78qxWzly+ROuGsqayr5CA4IQp3XqQdOUKW05f32nmCHsv8J4yv0ACUM2UdOUyVq2JadDI7KIIg0hAcMKtbdrSuHZt5u/bY8j+POUCby/pv/BezvxtnbxY1H8mNQTfIQHBCUH+/kzs3I0NiQkkuqhz2VYyv0A4wplgnnCpaGGc1g0kIPgKCQhOur9HLCEBAby2bauh+7X3Am9GM5H0X9RsCRcvElarFvVDQswuijCIBAQnhdepy8M9+/D18aP8nHzOsP16+oXVW5u3hHHBPOFiBq0bSv+BL5GAYIBf9+pDk9p1+L+tm9C+nutaeD2jgnnCxYvESHORT5GAYIA6QUE8eVM/diafY+2J42YXx+2k/6LmuWLJJT0n22s6lCUFt20kIBjk7s5daR/WmL9t+Y6rHpLOwl2kmch7ORrMT1++DECr+u4LCI5e1CUFt+0kIBgkwM+P/xtyK8mZmfxrx/dmF0cImzgazM9lXgGgWb16xhWmCs5c1CUFt+0kIBioV1RT7usey9zdu9hzPtns4gjhMslZmQBEVZHHyEjOXNQlBbftJCAY7Om4/kTUqctz335Ddn6+2cURwiXOZWYS5O9PWCXrIBjNmYu6rWucCAkIDqmqLTM0OJi/Dx3OsYx0Hlm5nNwCCQrC95zPyiKqbiiqZOyqizlzUZcU3LaTgGAnW9oyB0a35p/DRrDtzGl+s2oFloICU8rqSPuwdBALW1yxWKgXHOy24zlzUZcU3LZTnjpuvk+fPjo+Pt7sYlxn+/boSlaAakVcXOI1zy0+sI/nvv2GwdExvDd6LMEB7s027shKWa5cXUv4jolLFxKg/Jg/YaLbjpmSMo+EhOexWE4THNySmJiX5aJeAaXUTq11H0feKzUEO9nTljmxSzdeHnIr3yUm8Ls1K8mzWgG5Cxf287S/mat5edQJCnT5cco2zyYkPE9MzMtVLlwlnGNIQFBKjVRKHVFKHVdKPVfB68FKqUXFr/+glIo24rhmsLctc3LX7swaNJR1CSd4Ys0q8q1Wl2YHdSQtgeQl8nyellE2Oz+fOkFBLj2GzB9wP6cDglLKH3gXGAV0BiYrpTqX2+xXwEWtdVvgdeAVZ49rFkfaMqd2j+WFWwaz9sQxnlz7tUvL50haAkdTGUjAqLmsuhA/XNuhLPMH3M+IGsKNwHGtdYLWOg9YCIwvt814YG7xz0uBocpdwxMM5mgHVeLyXpx84ineGzMW8I27cE+7a/U1rq65OZPOoVZAILkuHiwh8wfcz4iA0Aw4U+ZxUvFzFW6jtS4ALgNh5XeklJqulIpXSsWnpqYaUDTXiIiYQlxcol1tmSV34XN+3gXAkLlzOJia6tKA4EhaAslL5DlcmVHW2eaYkIAAcq2uDQgyf8D9PKpTWWv9oda6j9a6T5MmVS/c7a2mxfYC4Gp+Hncums+i/XtdliHVFcNOpb/BNzjbHFMrIJDcfNcGBJk/4H5GBISzQIsyj5sXP1fhNkqpAKA+kG7Asb3SjBmwcvL93NC0GX/asI5n1q0hx0NmNVfXjCDrIJjD6Jqbs80xtYMCycyzGFmk68j8AfczYmD8T0A7pVRrii78k4B7y22zAngA2A7cBWzQnjoBwgXKj5/+zW9epnHtKXx6+wTe/nE7b/2wnYNpqbw3eizRJuaXL2lGKLlzLGlGAORLaDKjA25wcMtK5tPY1hzTPLQeO8+dQ2vt0tnKERFT5G/PjZyuIRT3CTwGrAUOAYu11geUUi8qpcYVbzYbCFNKHQf+AFw3NNWTOdP5VlVbrZ9SPNG3H3PG3cn5rEzGLfycVUePmLbIjr3NCNLf4L2cbY5p1aAhmXkWLuXmOnR8WZ/AM8lM5WqUv2uGoi+OrVVXW2c2J125zGOrV7I35TwDW7Vm1qAhtKzfwJDPYKuNG/2Aiv4eFIMGFbq1LML1nJn5+23CCX69cjnLJt5LbGSU3cd15jslqiYzlV3I2c43W9tqm9erz9K7J/OXAYOIP5fEiM/n8t5PP5BfPLvZHWRUR83iyGi5Eq0aFN2snLx40e7jyvwCzyUBoRrOdr7Zc5EN8PPjoZ69WTd1GoOjW/Pa9q3cvmgee1PO215gJ8ioDmGrVvUbUDswkD0p9q/7IfMLPJcEhGo4e9fsyEU2sm4o740Zx/tjxpGRk8Odi+fz8paNZOfnu7TtVUZ1CFsF+vvTK7IpO84m2f1eqYl6LgkI1XD2rtmZi+zwNu1Ye9+D3NOlG7N/3skflj3CgUMPuzS3izPNCMK72TuSqW/zFhxNTyMjJ7v6jcuQmqjnkk5lG3hC2t2dyWc5ezCWev5p171WUeptIexlb+rz+HNnmbh0If8ZM44RbdrZdSxP+E75Kmc6ld2boN9LecJY6N5Rzcg8UvFcPml7vdbMmTJRzh26R0RSOzCQTYkn7Q4InvCdEteTJiMvUlkbay7hXLE4Nh7cF0nSPds5k4okyN+foa3bsPbEMbeOhhOuIwHBi1TU9lqgg/ns9BAGzZ3Nx7viTVuu0yhyZ189I8+RM6lIZs6E29p14GJuLtuTzlS7vfB8EhC8SEUd1N06z+Yvo96gW3gE/7d1E8P++wnLDx+k0IG+IU+4GDt6d1+Tku55Sg1o1iy4pVU0dYOCWHnssNnFEQaQTmWDeEIn2ZbTifzz+y0cSL1A58ZNeH7AIOJa2D6Uz9ZORVe20RuxprOvrwvtqs9n7++1pBxPf7OadQkn+PHhR51aN9wTvkO+QGYqm8xTlvob0DKaLyfdx7+Hj+aSJZcpXyxh+lfLSbiYYehxjL5DrUl3945yxzmytZmofDn+NWIUp7/sxZoTxxw+tqd8h2o6qSEYwNZ8Re6UW5DPnJ938Z/4H7BYrdzXPZbf3XgTDUJqXbPdzJkVX+BnzKj8AuHKO3Aj9m3mKCN3HNtTakAl5SjUmiFzZ9M0tB7zJ0x0aF+e+B3yVlJDMJknTsUPCQjk/93Qlw0P/Iq7O3flsz0/M3juHOb8vJO8MiNCbO1U9Ka7eDPL5Cnt+/Zy5pz5KcXELt3YcfaMw7VRT/wO1UQSEKpga5oIT5yKX1L2Az+GclutX7N4VG26hUfwty0bGTlvLutOHLcrzba7FsaRlNrVc8U5ciSQlS3HXZ274K8Uiw/sc+j4nvgdqokkIFTCnjZNT5uKX1HZs849xT/jLMwedwf+SvHIqi+574ulHEn/ZeazJ1yMPbHGUR1315485RyVLUd4nboMbd2G/x066NCcBE/7DtVUEhAqYU+KXk9KCjdzZuVlP3nyeQZHx/D1vfczc+AQDqVd4Lb5n/Hi5u+4Ysm1+ULjSODwlIuYK3jrsqJGB7K7O3clPSebjYkn7X6vJ32HajLpVK6Ety4WoxR8951tZb+Yk8O/dnzPgn17aFSrNn+8eQATOnXBzwVLInpKR6ireevnNKLcBYWF3DznQ2IjI/ngttuNKZiwm3Qqu4AntWnam/La1rI3rFWLvw0expeT7qNV/fo8u34tdy1e4Lb1F3yRJzS7mSXAz487OnZiw8kEUrOvml0c4QAJCJXwlDZNW/oyylf9//rXl8nNtb3sXcMjWHz3ZF67dSRJmZe5Y9E8nl2/lrRs+9Ial+dNI5OM4q2fzahAdmenLli1ZkPCCWN2KNxKmoyq4AkzJ+0dn11S9Xe07FcsFt79aQef7N5F7cBAno7rz+Su3fH3c+7ewVubUoR9tNZ0e/9tJnbpxgu3DDa7ODWSM01GEhA8nL19GUZdeI9npDNj47dsTzpD1/AIXho0lB52LqbuinIJzzd+4efUDw7hszvuMrsoNZL0Ifgwe/syjKr6t20Uxud33M2bI8dw4WoWdy6ez5+//YaLOTkO7a8mt63XNG0bhXHiYsVrdwjPJgHBw9nbl2FkG7ZSirHtO7Luvmk81LM3Sw7uZ+h/5zBv3x6shfaNtPLWtnVhvzYNG5GclUWmxWJ2UYSdJCB4OE8Ynx0aHMzzAwax8t77ad+oMX/9bj23Lfgv285UnlbA3pFRwnc0DQ0FIK2CtZbl78KzyRKaXsBTlhvsENaYBRMmsvr4Mf7x/Sbu+2IJw2Pa8qf+A2nVoEHpdiUjo0omx5WMjAI84nMI1yqZx1JYrhYpfxeeT2oIwi5KKUa3a8+6+6bxVFx/tp45xYjPP+Uf328ubSKobpa3NB/5toDiEWnWcqMI7Jn9L8whAUE4JDgggN/e0Jdvpz7E2A4d+XDnTwz5bA5LDu6vNnOlURlBJbB4Jj9VHBDK1RAko6nnk4DgJu5qO3V3G21E3bq8eutIlt8zpXS28+WCRhVua/Qsb3enmjYrAHlbu3tBYXFyu3IpUDxp9r+omAQEN3DXalBmrjrVPSKSxXdP5o0Ro1l/6Q4shYHXvG611uavf33Zq2ctm7HWgTeuJLb/QgpBfv7ENGh4zfOeMvtfVE4Cghu4q+3U7DZaP6UY16ETr9/5AZb6L3GpoBFawxVrY4Ki/s769VOczgha09JhmP07dcSu88l0CQ+/bn1lTxgxJ6omAcEN3NV26ilttAF+fozr/SyjB1/gXOPdrE4bS0rSDL77zo/NW1swdKjjd7fuTjVtdgDylN+prfKsVvamnKdXVNMKX4+ImEJcXCKDBhUSF5cowcDDSEBwA3e1nXpaG22Qvz/DwvczOWoRjYMuoZSmsCCJPz77MNsOv23Xim1mMXutA0/7nVZnU+JJ8qxW+jRtZnZRhAMkILiBu9pOPbGNtqImj6DAXM6dfoHxCz9n2aEDWAoKHNp3TUiH4Ym/08ocz0jnmfVraNOwEYNatTa7OMIBEhDcwF1tp57YRltZ00ZY0GVyCwp4et0aBnz6EW/+sM3udNvu7jcwIwB54u+0IqnZV3loxTIC/fyZM+7O6/oPhHdwKtupUqoRsAiIBhKBiVrrixVsZwVKVt8+rbUeV92+Jdupb6gqffdNN51k65lTfLJ7FxsTTxLk78/4Dp2YFtuLjo2bmFBa4Yjs/Hwm/28RxzPSmT/hHnpERJpdpBrNmWynzobx54Bvtdb/UEo9V/z42Qq2y9Faxzp5LOGFYmJeviZdAfzS5KGUYkDLaAa0jOZERjqf7PmZZYcOsOTgfm5u0ZIHe/TillbRBPr7m/gJRFXyrVaeXLOKA6kXeH/MOAkGXs7ZJqPxwNzin+cCspCquIatTR5tGoXxt8HD2PbQdJ7p158TGRn8euVy+s5+nz+uX8O3J0843NdQEV8cpuruz3QiI527lixg/ckTvHDLYIbFtK1wO2+bWFeTOdtkdElr3aD4ZwVcLHlcbrsCYDdQAPxDa728kv1NB6YDtGzZsvepU9c3NYiaId9qZfOpRFYeO8KGkwlk5lmoGxjEoNatGdmmHQNbtaZOUJDD+/fGBXtmzqz6om/0Z6rseDn5+Xy+bzf/3r6N2oEBvDT4Vka3a1/hPsontIOiGqIn9oP4CpeumKaUWg9UVA98HphbNgAopS5qrRuW31Ap1UxrfVYpFQNsAIZqratcdFX6EESJPKuV7WdOs/bEMdYlHCc9J4dg/wAGtGzF0Jg2DGkdQ5PadezapzcGhOrKbPRnKr+/81mZ/Hfvbhbs38ul3FyGRMfw96HDaVKn8nNv7xKwwnku7UPQWg+r4sApSqkorXWyUioKuFDJPs4W/5+glNoI9ARkFW5hkyB/fwZGt2ZgdGteGjyMncnnWHP8KN8kHGf9yRMooGdkFMNi2jIspg1tGjZClcujA0V3u2XTT5RsMmOG9zYhueMz7U05zye7d7Hq2BEKtebWmLY81LMXfaKaVXiey67nXfHyr547sa6mc7bJ6FUgvUynciOt9R/LbdMQyNZaW5RSjYHtwHit9cGq9i01BFEdrTWH01JZl3CC9SdPsP9CCgAt69WnZ1RTujQJp3OTcLo0Cad+SMg17/WUGkJ1zUDlL/glKrrgG/GZZszQvPji9Rf58NE/8Niz2TzQoyct61/XKlyqoiaiikgNwXVc2mRUzYHDgMVAS+AURcNOM5RSfYBHtdYPK6X6AR8AhRR1Yr+htZ5d3b4lINRMZe8ug4NbEhPzss1tzcmZmXx78gRbTieyLyWF81ezSl9rFlqvNEB0aNyYVx/5kZde+jMWyxm7j2PkZ7LnIm50k9HVvDzOXLnM8Yx0DqRe4MCFCxxITeFibi4AJ594ivEL5zG2fUfu7tyVesHB1e6zsiaisqQPwbVMCwiuJAGh5jG6AzItO5tDqRc4kHqBg8X/J166yI0NdvJA1DJCAn9Z89dKMFmhM2kSfi9RoaE0rRtqyOSq6j6TkQGhbG3DUlBAWk42adnZpGdnk3I1izOXL5N05TJnrlzmzOXLZOTmlL430M+PDmGN6RIeQZfiWlWvpk3trnFs3OhHZc1ERaPMXBt8hQQE4SPc0QF5NS+PnT+2obAg6brX0vIa8MfDfyl9nLvuFvrce5imofWIrFuXhrVq0SAkhAYhtWgQHEKDWkX/1wsOJiQggOCAgNLlI6v7TOfPt2Ly5Gs/U0kzkLWwkJyCArLz88jOzycnP5+r+fm8+8863PNYKpl5Fq5YLFyx5BKSu4pWeja1SCWzMIx1F29nc1oPMvOuX+A+wM+PZqH1aF6vHi3rN6B5vXq0qFef1g0a0i6sMUHl5ntU15xVEelENp8EBOETKr+7VAwaVFjB88YfJ7hNIueyMknOzOTxvjfx4PL/kZyZSXJWVoUX2fKC/PwJDggoDhD+zGo1vfw6MUDRnf6sUx+xadrD9PnwPfKsheRZreQXWim08TvZt8FOpjVfSpBffulzBTqYo+oP6DrjCKtVm8a1a9O4dh2a1KlDZJ26+Pu5NluNDDM1n5kzlYUwTHBwy0ruLo3PClvZceJa/HKsx4FPxk8ofVxQWMjl3Fwu5eZwyZLLpZxcLllyuWKxYCkoILegAIu1AEtBARarldyCArJ1E+qo1OuOla2b0CuqKZuAEW3bE+TvT5CfH4H+/gT6+VMrMIBaAYHUCQyiVmAgtQMDqRUYQO2AQOoFh1A/JJgDuzpgseRfs98AZaFn8Hzibvg/w86XPUou+o72AwlzSUAQNnOmw9cWVaW5MFJVx6l6GKcfYbVrE1b72uyjVUlJeb3CY/Xp9DpjIkZTbwbMHFzpyO4qWSxnKnne3CGdERFTJAB4KWkyEjZxV1OAq4OOPccxamiqqz6TtNeLikgfgnC5mnjx8ZS5CpWR9npREWcCgqyHIGzibUs5GsHTF+DxlrUShPeQPgRhE3d1+HoSb0hnIe31wkhSQxA28aalHIUQjpGAIGwizRNC+D5pMhI2k+YJIXyb1BCEEEIAEhCEqBFkGUthC2kyEsLHlZ+vYLGc4siR6QDSBCiuITUE4ZPkjvgXCQnPX7dgTWFhNgkJz5tUIuGpJCAIn1NyR1w0b0KX3hG7Kih4evCpiZMKhWMkIAif4847YncHH0dUNnnQlycVCsdIQBA+x513xN7QHCOTCoWtJCAIn+POO2JvaI6RSYXCVjLKSPgcd62rAN6T40kmFQpbSA1B+Bx33hFLc4zwJVJDED7JXXfEsmSk8CUSEIRwkjTHCF8hTUZCCCEACQhCCCGKSUAQQggBSEAQQghRTAKCEEIIQAKCEEKIYhIQhBBCABIQhBBCFJOAIIQQApCAIIQQopgEBCGEEIAEBCGEEMWcCghKqbuVUgeUUoVKqT5VbDdSKXVEKXVcKfWcM8cUQgjhGs7WEPYDdwKbK9tAKeUPvAuMAjoDk5VSnZ08rhBCCIM5lf5aa30IQClV1WY3Ase11gnF2y4ExgMHnTm2EEIIY7ljPYRmwJkyj5OAvhVtqJSaDkwvfmhRSu13cdm8RWMgzexCeAg5F7+Qc/ELORe/6ODoG6sNCEqp9UBkBS89r7X+0tEDV0Rr/SHwYfFx47XWlfZL1CRyLn4h5+IXci5+IefiF0qpeEffW21A0FoPc3Tnxc4CLco8bl78nBBCCA/ijmGnPwHtlFKtlVJBwCRghRuOK4QQwg7ODju9QymVBMQBq5RSa4ufb6qU+hpAa10APAasBQ4Bi7XWB2zY/YfOlM3HyLn4hZyLX8i5+IWci184fC6U1trIggghhPBSMlNZCCEEIAFBCCFEMY8JCJIG4xdKqUZKqXVKqWPF/zesZDurUmp38T+f6qiv7veslApWSi0qfv0HpVS0+0vpHjaciweVUqll/hYeNqOcrqaUmqOUulDZ/CRV5K3i87RXKdXL3WV0FxvOxSCl1OUyfxMv2LRjrbVH/AM6UTShYiPQp5Jt/IETQAwQBOwBOptddheci38CzxX//BzwSiXbZZldVhd9/mp/z8D/A94v/nkSsMjscpt4Lh4E3jG7rG44F7cAvYD9lbw+GlgNKOAm4Aezy2ziuRgErLR3vx5TQ9BaH9JaH6lms9I0GFrrPKAkDYavGQ/MLf55LnC7iWUxgy2/57LnaCkwVFWTQ8VL1ZS/+WpprTcDGVVsMh74TBfZATRQSkW5p3TuZcO5cIjHBAQbVZQGo5lJZXGlCK11cvHP54GISrYLUUrFK6V2KKV8KWjY8nsu3UYXDW2+DIS5pXTuZevf/ITiZpKlSqkWFbxeE9SU64Ot4pRSe5RSq5VSXWx5gztyGZVyZxoMT1fVuSj7QGutlVKVjQ1upbU+q5SKATYopfZprU8YXVbh8b4CFmitLUqpRyiqOQ0xuUzCXLsouj5kKaVGA8uBdtW9ya0BQUsajFJVnQulVIpSKkprnVxc5b1QyT7OFv+foJTaCPSkqL3Z29nyey7ZJkkpFQDUB9LdUzy3qvZcaK3Lfu6PKeqDqol85vrgLK31lTI/f62Uek8p1VhrXWUCQG9rMqopaTBWAA8U//wAcF3tSSnVUCkVXPxzY+BmfCeluC2/57Ln6C5ggy7uTfMx1Z6Lcu3k4yjKCFATrQDuLx5tdBNwuUzTa42ilIos6VNTSt1I0bW++hsms3vLy/SK30FRm58FSAHWFj/fFPi6zHajgaMU3Qk/b3a5XXQuwoBvgWPAeqBR8fN9gI+Lf+4H7KNo1Mk+4Fdml9vgc3Dd7xl4ERhX/HMIsAQ4DvwIxJhdZhPPxd+BA8V/C98BHc0us4vOwwIgGcgvvlb8CngUeLT4dUXRYlwnir8TFY5W9IV/NpyLx8r8TewA+tmyX0ldIYQQAvC+Jq18H9AAAAAxSURBVCMhhBAuIgFBCCEEIAFBCCFEMQkIQgghAAkIQgghiklAEEIIAUhAEEIIUez/A2YjI/zb6oF1AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "create_plot_for_lambda(X, y, 0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Underfitting\n", "Underfitting means the model is not well trained. Having a high $\\lambda$ can cause underfitting." ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Optimization terminated successfully.\n", " Current function value: 0.648216\n", " Iterations: 14\n", " Function evaluations: 29\n", " Gradient evaluations: 29\n", "Conjugate gradient found the following values for theta - first five values only: [ 0.326144 -0.00815789 0.16580133 -0.44666092 -0.11177511]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl8VOW9+PHPk20SskICCRBCEmTfguACooCg4r5vpSq2vVxb7e2ibW2tAt7SW6/9XVvtYlXcEReqiAsuKIggKItIWGSLLIFsJJB9n+f3RyYhCTPJZObMnDMz3/frlVeSyZlznjkzeb7n2b5Haa0RQgghwswugBBCCGuQgCCEEAKQgCCEEMJBAoIQQghAAoIQQggHCQhCCCEAgwKCUupZpVSxUmqHi79PV0qVK6W2Ob4eMuK4QgghjBNh0H6eB/4GvNjFNp9rra8w6HhCCCEMZkgLQWu9FigzYl9CCCHMYVQLwR2TlVLfAMeA+7TWOztvoJSaB8wDiI2NnThixAg/Fk8IIQLfli1bjmut+3ryXH8FhK3AYK11lVLqMmA5MLTzRlrrp4CnACZNmqQ3b97sp+IJIURwUEod8vS5fpllpLWu0FpXOX5+H4hUSqX449hCCCHc45eAoJRKU0opx89nO45b6o9jCyGEcI8hXUZKqaXAdCBFKZUPzAciAbTWTwI3AD9WSjUBtcAtWtKsCiGEpRgSELTWt3bz97/RMi1VCBGgGhsbyc/Pp66uzuyiCCA6Opr09HQiIyMN26c/ZxkJIQJYfn4+8fHxZGZm4ugBFibRWlNaWkp+fj5ZWVmG7VdSVwgh3FJXV0dycrIEAwtQSpGcnGx4a00CghDCbRIMrMMX74UEBCGEEIAEBCFEgFm+fDlKKb799lunf587dy7Lli1ze3/Hjh3jhhtuAGDbtm28//77bX9bs2YNX3zxRY/LmJmZyfHjx3v8PLNJQBBC+NSCBcbub+nSpUydOpWlS5casr8BAwa0BRCjAkKgkoAghPCphQuN21dVVRXr1q1j8eLFvPrqq0DLjJt77rmH4cOHM2vWLIqLi9u2z8zM5Le//S05OTlMmjSJrVu3cskllzBkyBCefPJJAA4ePMiYMWNoaGjgoYce4rXXXiMnJ4dHHnmEJ598kscee4ycnBw+//xzSkpKuP766znrrLM466yzWL9+PQClpaVcfPHFjB49mh/96EcE6jIrmXYqhAgYb7/9NrNnz2bYsGEkJyezZcsWDh06xJ49e9i1axdFRUWMGjWKH/zgB23PycjIYNu2bfziF79g7ty5rF+/nrq6OsaMGcNdd93Vtl1UVBQPP/wwmzdv5m9/a1k2VVtbS1xcHPfddx8A3/ve9/jFL37B1KlTOXz4MJdccgm7d+9m4cKFTJ06lYceeoj33nuPxYsX+/fEGEQCghDCcAsWdGwZtE6ImT/fuy6kpUuX8rOf/QyAW265haVLl9LU1MStt95KeHg4AwYM4MILL+zwnKuuugqAsWPHUlVVRXx8PPHx8dhsNk6ePNmj469atYpdu3a1/V5RUUFVVRVr167lzTffBODyyy+nd+/enr9IE0lAEEIYbsGCUxW/UmBED0pZWRmffvopubm5KKVobm5GKcW1117b5fNsNhsAYWFhbT+3/t7U1NSjMtjtdjZu3Eh0dHTPX0AAkDEEIURAWLZsGbfddhuHDh3i4MGDHDlyhKysLJKTk3nttddobm6moKCA1atXe3yM+Ph4KisrXf5+8cUX88QTT7T9vm3bNgAuuOACXnnlFQBWrlzJiRMnPC6DmSQgCCF8av58Y/azdOnS01oD119/PQUFBQwdOpRRo0Zx++23M3nyZI+PMWPGDHbt2kVOTg6vvfYaV155JW+99VbboPLjjz/O5s2bGTduHKNGjWobmJ4/fz5r165l9OjRvPnmm2RkZHj1Ws2irDoaLjfIEcJadu/ezciRI80uhmjH2XuilNqitZ7kyf6khSCEEAKQgCCEEMJBAoIQQghAAoIQQggHCQhCCCEACQhCCCEcJCAIIQKCUop777237fc///nPLOgmD8by5cs7pJrwRE9TWa9YsYI//elPTo///PPPc+zYsR4dvzX5nj9IQBBC+ERR0RI2bMhkzZowNmzIpKhoiVf7s9lsvPnmmz2qnI0ICD111VVXcf/99zs9vicBwZ8kIAghDFdUtIQ9e+ZRX38I0NTXH2LPnnleBYWIiAjmzZvHY489dtrfDh48yIUXXsi4ceOYOXMmhw8f5osvvmDFihX86le/IicnhwMHDnR4zjvvvMM555zDhAkTmDVrFkVFRYDrVNYHDx5kxIgRzJ07l2HDhjFnzhxWrVrFeeedx9ChQ/nqq6+Alkr/nnvuOe34jzzyCJs3b2bOnDnk5ORQW1vLli1bmDZtGhMnTuSSSy6hoKAAgC1btjB+/HjGjx/P3//+d4/PWY9prS35NXHiRC2EsI5du3a5ve0XXwzWq1dz2tcXXwz2+PixsbG6vLxcDx48WJ88eVI/+uijev78+Vprra+44gr9/PPPa621Xrx4sb766qu11lrfcccd+o033nC6v7KyMm2327XWWj/99NP6l7/8pdZa65/+9Kd64cKFWmut3333XQ3okpIS/d133+nw8HC9fft23dzcrM8880x95513arvdrpcvX952zOeee07ffffdTo8/bdo0vWnTJq211g0NDXry5Mm6uLhYa631q6++qu+8806ttdZjx47Vn332mdZa6/vuu0+PHj3a6Wtw9p4Am7WH9a5kOxVCGK6+/nCPHndXQkICt99+O48//jgxMTFtj2/YsKEt/fRtt93Gr3/96273lZ+fz80330xBQQENDQ1kZWUBdJnKOisri7FjxwIwevRoZs6ciVKKsWPHcvDgwR69lj179rBjxw4uuugiAJqbm+nfvz8nT57k5MmTXHDBBW2vZ+XKlT3at6eky0gIYTibzXlyN1eP98TPf/5zFi9eTHV1tVf7+elPf8o999xDbm4u//rXv6irq+v2OZ3TZ7dPrd3TVNpaa0aPHs22bdvYtm0bubm5fPTRRz17EQaTgCBMY/Sgo7CO7OxFhIX16vBYWFgvsrMXeb3vPn36cNNNN3W4K9mUKVPabqm5ZMkSzj//fOD09NXtlZeXM3DgQABeeOGFtseNTGXdVTrt4cOHU1JSwoYNGwBobGxk586dJCUlkZSUxLp169pej79IQBCm8MWgo7CO1NQ5DB/+FDbbYEBhsw1m+PCnSE2dY8j+77333g6zjZ544gmee+45xo0bx0svvcRf//pXoOWuao8++igTJkw4bVB5wYIF3HjjjUycOJGUlJS2x41MZd35+HPnzuWuu+4iJyeH5uZmli1bxm9+8xvGjx9PTk4OX3zxBQDPPfccd999Nzk5OX69P7Okvxam2LAh0xEMOrLZBjN58kH/F8gLRUVLyMt7gPr6w9hsGWRnLzKs4rMSSX9tPUanv5ZBZWEKXw06+ltrS8durwFoa+kAQRkURHCTLiNhCl8OOvpTXt4DbcGgld1eQ17eAyaVSAjPSUAQpvDloKM/BUtLx11W7WIORb54LyQgCFP4etDRX4KlpeOO6OhoSktLJShYgNaa0tJSoqOjDd2vjCEI06Smzgm4ANBZdvaiDmMIEJgtHXekp6eTn59PSUmJ2UURtATo9PR0Q/cpAUEIL7QGtFCYZRQZGdm2mlcEJwkIQngpGFo6QoBBYwhKqWeVUsVKqR0u/q6UUo8rpfYrpbYrpc404rihQlb0yjkQwh+MGlR+Hpjdxd8vBYY6vuYB/zTouEEv2Fb0elKxB9s5EMKqDAkIWuu1QFkXm1wNvOjIzroRSFJK9Tfi2MEumOa5e1qxB9M5EMLK/DXtdCBwpN3v+Y7HOlBKzVNKbVZKbZaZDC2sMM+9m7sUus3Tit0K50CIUGCpdQha66e01pO01pP69u1rdnEswQrz3BcuNGY/nlbsVjgHQoQCfwWEo8Cgdr+nOx4T3QiWFb3gecUeTOdACCvzV0BYAdzumG10LlCutS7w07EDmlkrehcsAKVavuDUz950H3lasfvrHMhMJhHqDEl/rZRaCkwHUoAiYD4QCaC1flIppYC/0TITqQa4U2vdZW5rSX/tPaPSMisFRmUr8FWqaG/32zlrKbQEq0BMpyFCmzfpr+V+CBZlpQrOyIDgC0a81mC6P4MIbd4EBEsNKosWRsy7N3Kq5vz5PX6KXxnxWkN1JpN0k4n2JCBYkNUqOKOmnfqKEa81FGcyyYI/0ZkEBAuSCq5njHitoTiTSRb8ic4kIFiQVHA9Y8RrDZb7M/REqHaTCdck26kFGZFjP5TSMhv1WkMta6nNluFiID34WpHCPTLLyKJ8NT1TiFYy1TY4eTPLSFoIFhVqV6uhyszAH0qtSOEeCQjCkhYssP7sJm91vkJvneUD+DUoSAAQrWRQWfiMNxW6UQn1rExm+QirkYAgfCYUKnVvyCwfYTUSEMRpzOqq8UVCPW9X4vpyJW8orRURgUFmGQkA6poaKamuoaSmmokDBrBs106qGxqoamigurGB6oYGqhsbqWpooK6piYbmZhqam6hvbnb83PL7obfOJP+d0yc4pF7+FQOv2kR4WBjhKowwpYgIO/XdFh6OLSKC6IgIosIjeP6a6/jZB+8RHR5BbFQUcVFRxEZGEev4HhcVSYItmuSYGJJ79SLBFk1YayRx8HYWja9n4cgsH+ELMstIdKmxuZnCqiqOVJSTX1FOfkUF+RXlHKuspKSmmpKaaqoaGto9415+9fEHbb+FK9WuIo5yVNrhRIVHEBdlwxYRTlR4OLbwCKb+VynqZx8RphR/nHkRD366CqUUCrDr4TRrTbPdTrPW2LWmyW6n2W6nobmZuqYm6pubqG6sBeCbwkLqmpraAlJXly4RYWH0iYkhOaYXyTG96Bcby0VR92LDeR+9OxVuV338RlTY3szykWnJwhekhRBETtbVsr+sjP1lpW3fD5woo7CqkuZ273OYUvSPi2dAfDz9YmPp2yuWL18exvvPpJ+2zwcebOa/F4ahOl19u8ObLKmdZxlpralpbKS6sbXV0kh5XR2ltTWU1tR0+H68pobi6mr+kPWfOCu21vBixTLSExLJSExkUEISGYmJpCckkmCztW23Zk0YOA1DiunT7Z69MANIy0J0RVoIIUZrzZGKcnKLisgtLiS3uJh9Zcc5XnOqgoiOiOCM3n2YOGAAGQlJpCckkJ6QyKCERNLi4ogMD++402nA0y0/dqzIO23XA95kSe08bqBaWylRUfSLdW8fGzYscjpAW6P7Ulpby7bCQsrr6zr8LaVXL4b07kN27z5MD08lUhee9nyz+/i9bblI60K4IgEhAJyorWXTsXy+KSp0BIGitoosKiyc4SkpTM/MYmifZIb0TmZon2QGJiSc1qfub2avI8jO/qPTK+lJIx/j8gtbKsCK+jqOlJdzuKKcw+UnyTtxgrwTZazcv5e86BnMTV+GLayx7flN2kZx2H/AkcOMSOlL75gYv78ub2YnWWHtg7AuCQgW0PmKLStrESVhs/j4wH5WfXeAb4+XAC395MOTU7j0jKGMTU1jbL9UhiWnENXuar+oaAl5ux7ggBdXf1a//4G73OmjT7BFM7pfNKP7pZ72/LLaO/n20GRqi/+HMHshVc3JvFN8OatKooE3Wo4RG8e41FTOTc/g3PRBDE9O8Xkg9iYHka/HRURgkzEEkznrD26wR/Fc/vVsKp/EWQMGMjUjk3MGpjO2Xyq2CNcx3Cp9y2atMvbXcUtqqvn2eAnfHi9hd0kJWwuOcbiiHIA+0TGcmz6Ic9MHMWVQBllJvT0af+mKN++zVcdFhHHkFpp+ZmQf7Lr1GTQ1Hjnt8eaw/kw4az99Yno5eZZzZtwG0lkl3JPBZCMrcTNv9Xm0ooIN+YfZkH+EDUcOU1hdxYmVkxl1Yy7TM7OZPjiLKYMyiI2KMuR4nn4G5VahwU8Cgh8ZcRVe3dDABwf28e9dO/mPPjc7nQnjyRWbGVd/zirhnlTMRlbiVrn3s9aag+Unye7dm7veXcH6w4eoamwgKiycswemMz0zixlZ2WQl9fZ72azSihS+I/dU9iNv8s8cKS/ngU8/5pzFT/Krjz+goKqSxrA0p9t6MpPFzJWvvlhlbOXjdkUp1VbZ//Pyq9g87ycsufZGbh+fQ2FVJX/4fA0zX3yWi156jv/bsJ7vTp5wuS+jX4cZNwKS+zYHDmkh9JAnV+GV9fX8Y/OXPPf1VlBw5bAR3DR6DJP6D6S4+BXDrtj8dfW3YIHzPEXz55+qoLv6WHX3fE9ZoYXgzms7Ul7O6oN5fHhgH18ezceuNVMGZfC9MeOYlX1Gh0kCVnhN3pAWif9Jl5Ef9aQPttluZ9muHfx5w3pKa2u4bsQo7psylbS4+A7bGTkm4e855tJl5Jo75SmuruKNXTt4dUcuRysrSOnVixtHjWHO2PEMiE8w5TUZ+RmSMQv/k4DgR+5e8Ww4cpg/fL6G3cdLmNh/AL+/YAbjU513D/WE1e4TYKWAEAjnxpVmu521hw/ySu43/PsfaZz4YPJp23jbgnKH0Vf0MqvJ/2QMwY+664M9UVvLzz98jzlvvUFFfT1PzL6C12+4xZBgANZLKd26ZsHTvnwj1zyYGQyc9ZP35LWFh4UxIzObp6+8ltzXRvPwZ6sZ/Y/HAbjjrX+z7vAh5s/3/cWb0fdokIyugUVaCAbKLS5i3jvLKa2t4SeTzuHHk87uct2AJ6zWLeJMIJTRSL7qJy+vqyMpJpqznv4nx2tqyEntz+/On8akAQONKLZTRl/RyxiC/0kLwQI+yTvALcteJSIsjLdu+h4/P3fKacHA0ytYK86kEaf46s5nidHRzJ8Pn8/9D/4wYxbHqiq4admr3PfRSspqa7rfQSfuzPYx+orejFlNwnPSQjDAC99s5b/XrmFMv1SevuIa+sY6z75mxJVzIFx9W60v39dcXVVrrZgxw7h+8prGRv6x6Uue2rqJhCgbv79gBlcPH+HWSmh3r9Tlij7wSQvBJM12Ow+vXc3Cz1YzK2sIS6+7yWUwCCWBEgyMKqerq+eiImP7yXtFRnLflKmsuOX7ZCQm8cuP3ufOt98k35E2oyvutmLkij60SUDwUH1TEz95fwXPb9vKnTln8vfLriQmMvK07Yzu7gmWxHNWYNQAfXb2IsLCOqYYCQvrxTPPLDLmAJ2MSOnLGzfewvxpM9hScJRLXn6eZ7/eQlet/Z5kSE1NncPkyQeZPt3O5MkHJRiEEAkIHlr0+Ro+zjvA/GkzePCCGYSHOT+VCxa0dPG0/q+2/uzNeEIgclXuQH097bW/qtZaUVg4mIcffopPPpnjs/Ge8LAw7hh/Jh9+fy7npmfwh8/X8OP3VnS6890pMttHuEPGENzUfrGOPbw/T383jXHZ/8lvp05zex+B0P/vK65eu7/PSVcriVv/bhR/vTatNc9u28qf1n3GyJS+LL7qutO6LmVsIHTIGIKPtf4ztay41IQ1H+MHg/7N7UO6vyFJe9LdY76uWmxddSFZuSWjlOKHEybyryuu4cCJMm54Yyl5J8o6bCNjA8IdEhDc4GxALlI1cPjggz3aj5UrFV9wNX4yfXrgTaP1ZLzB3xcAF2Zl88p1N1Hd0MCNbyzl64JjHf4uYwOiO4YEBKXUbKXUHqXUfqXU/U7+PlcpVaKU2ub4+pERx/UXb25ZaBVm3bDG2dX4mjXGjqt4qn0yPl8EJzPO+fi0/iy76VYSbNF8/6032HzsqM+PKdlMg4fXAUEpFQ78HbgUGAXcqpQa5WTT17TWOY6vZ7w9rj8ZNSBn5j+O1VJeWEHreglXwSlQFwRmJvVm0t7bSIuL54cr3mJ3SbHPjtW5O7X1Hs0SFAKTES2Es4H9Wus8rXUD8CpwtQH7tYzs7EU0aVuHx8LCepGd7f60wlD/x3HVfWLlcRWjZ4j505//GMWL19xAXFQkd7z9bw52cc8Fb/hqlbYwhxEBYSDQ/h6Q+Y7HOrteKbVdKbVMKTXI2Y6UUvOUUpuVUptLSkoMKJoxCvWFLD5yHfWk4umAnKf/ON5UPt5c4Rpd6QXCtFMrBydPDExI4MVrbsBu19y+fBmlNT1Pd9GdYOhOFaf4a1D5HSBTaz0O+Bh4wdlGWuuntNaTtNaT+vbt66eidU1rzR/Xfcb++vM5b8phjwfkPP3H8aarx5sr3FDsYurqvARCsHB2AXBGcjKT9t5GcXU1//XBuzTbjU05LesbgosRAeEo0P6KP93xWButdanWut7x6zPARAOO6xe5xUVsOnaUu886hzgvbpAu/ziBzUotGVdcXQA8+ed4Hp4+kw35R3hp+zZDjtPK1SrtnnSnCuswIiBsAoYqpbKUUlHALcCK9hsopfq3+/UqYLcBx/WLV3dsJyYigutGjvZqPz35x/HFYKY7V7iBOogqunfjqDFMG5zFo1987lbuo660bz3K+obgYshKZaXUZcBfgHDgWa31IqXUw8BmrfUKpdT/0BIImoAy4Mda62+72qcVVipXNTRw7uInuXzocB6ZdYnX+/Pk1oRmrW4O5VXVwcBZxtmjlRXMfvl5JqQN4IVrrncrS6oz8tmwNrmFpo8s/3Y3v/zofV674WbOGpBuShkkIAgjvbx9Gw+t+YT/nXUJN4wa4/bzukr5IS1Ia5HUFT7ycd5+0mLjmNjfd3eo6o5Zg5lWHUSVysc73xs7nrMGDOSP6z6jor7O7ecF8hRc4T4JCC5ordlScJRz0gcR5mHT2ghm/cNZ9R89FGc/GSlMKR66YAYn6+p4eqt1kkcKa5CA4MKxykqKq6s5s/8As4sihKFG90vliqHDeW7bVkpqqnv8fKu2HoX3JCC4sKWgZebsmWn9u9lS+JrMfjLeLyafR31TE//Y9GWPn+uP8y75kcwhAcGFrwsL6BUZyfAUayyQszJfVxDSf228rKTe3DhqDK/kfsPRigqzi9NBqKd5MZMEBBf2lpYyrE8KES7uhCZOkX79wHTP2edi15qXcr1frGYkyY9kHqntXDhScZKMpESv9yNXscaS/mvjDIhPYGbWEP69awdNBqe08IbkRzKPBAQnmux2CiorGZTgfUAI1qtns/r1JcAa67qRoyitrWX94UNmF6WNpHkxT0gFBHcHqgoqK2nW2pCAYDVGVajSrx8cpg3OIsFmY8XeLhMH+JXkRzJPyASEngxUFVRVAi1Nak9YeVZMsLZYhGdsERHMHjKUjw/sp7G52eziAJIfyUwhExB6MlBVVfoG/zviDzTlZXk05S3Urp6lXz+wTc/MpqqxgW1FBWYXpY3c/9kcIRMQ3B2oKipagir7HSlRJwmWKW++brEEa6ALFt29P1MGtazGX+ejcQRZUxA4QiYguDtQlZf3AIqOOV68mfJmhavnUGuxiI666yZMsEUzPjWNzw8ZHxBkTUFgCZmA4O5AldFT3qTSFYFgasZgthcXUl7nfsI7d8iagsASMgHB3YGqYJ/yZoUWi/C9nnYTnj0wHbvWbC8uNLQcsqYgsESYXQB/Sk2d0+3gVHb2Inbu/hFh7bqNgmnKm7RYQkP7G+S4c2+LMX1TAdhRXMT5GZmGlcNmy3B0F53+uLCekGkhuCs1dQ4lMQ9wvCGJQJryJhW98EZidDSDE5PILSoydL+ypiCwSEBwoi76Mn797e8Zd06VaVPeelrBy/oC4Yq73YRj+6WSW2xsQJA1BYElpLqM3BUV3nJa6pvMW6izcKFc9QtjuPs5GtW3H+/u20NFfT0JNpthx3enq1ZYg7QQnEi0RQNQ3oNbDJrByiuiReDJTOoNwOHykyaXRJhFAoITSdGOgGDwFLzu9LSCt8r6AglAwWFwUhIAh05KQAhVEhCcaA0IJ/3cQrBKBd9TMn4RuNp/tjIcyRwPSQshZElAcKI1IJyorTW5JO6T9QXCE+2DeWxUFCm9eklACGESEJxI6RWL4lTWUzP0tII3o5tIxi+CT2psHCU11WYXQ5hEAoITUeHhpMbGcazSvIBg9Yo1ULu3RNfBPDmmF2UB1DIWxpKA4MLAhATL3XxcCCN0Fcz7xMRQVlvT1dNFEJOA4MKA+ASOVUpAcIeMXwSPPkHaQpAU3O6RgOBCRmIiRysraLDIXaSsTLqJAlfnYJ4YbaOmsZEmu92cAnXB00pdUnC7TwKCC2f0SaZZaw6ePGF2UYTwmc7B3OZYpW+1CyFvKnVJwe0+CQguDO2TDMD+slKTSyKE/9giwgGob2oyuSQdeVOpSwpu90lAcCG7d28UsE8CggghrS2E+mZrBQRvKvVgv8eJkSQguBAdEUlGYhJ7jh8/7W8yQCWCVVR4awvBWl1G3lTqkoLbfRIQujC6bz92lhR3eCyQBqg8GeyVAeLQ1uyYixoepkwuSUfeVOqSgtt9EhC6MDY1lSMV5R1SWATSAJUnOYYkL1Foa51dFBFmrarB20o9NXUOkycfZPp0u2n3OAkE1nrXLWZcvzSADjcNMWKASq7CRU/56zPT6JhdFBEW7p8DdqN992xe3gNkZy+SSt2HDAkISqnZSqk9Sqn9Sqn7nfzdppR6zfH3L5VSmUYc19fGpqYRphRbCo62PWbEAJUvr8I9yTEkeYmsz18tt9YWQqQFWgiB1D0bLLx+15VS4cDfgUuBUcCtSqlRnTb7IXBCa30G8BjwiLfH9Ye4qCjG9ktl/eFTNwm3+gCVJzmGPM1LJAEj+NQ2NQIQHWH+zRQDqXs2WBhxGXA2sF9rnae1bgBeBa7utM3VwAuOn5cBM5VS1hq1cuGCwZlsKyrkZF3LOIKnfZnBeBUu4w2+5evPjLPZcuV1dURHRGCzQECQ9QP+Z0RAGAgcafd7vuMxp9torZuAciC5846UUvOUUpuVUptLSkoMKJr3pg/Owq4169q1EjwZoDIjO6gnOYYkL5F1+PIz46o7plfDyrZbyJpN1g/4n/kdhe1orZ/SWk/SWk/q27ev2cUBYFxqGknR0Xx26KDZRekxX0w7DcaWTihy1R0zVD1Hos1mUqk6snr3bDAyIiAcBQa1+z3d8ZjTbZRSEUAiEBBLgMPDwjg/I5M1B7+j2aCEX1a+Cu9u0Z3cB8EcRn9mXHW79FIzHKdoAAAXUklEQVTHSYy2RgtB1g/4nxEdhZuAoUqpLFoq/luA73XaZgVwB7ABuAH4VOvWKsX6Ls4+g3f2fsuWgmOcPTC9x88vKlpCXt4D1NcfxmbL4Mc/XgRY70Pd2o3QeuXY2o0AyD+hyYwOuDZbhqO7qKPy5t6kxsUZezAvpKbOkc+eH3ndQnCMCdwDfAjsBl7XWu9USj2slLrKsdliIFkptR/4JXDa1FQrGxO7kUdHLKJ6X0aPU1UE0tS5ns7qsHJLR3TNVXfMvwsuZUBcvM+PL+lfrMmQqQRa6/eB9zs99lC7n+uAG404lr8VFS3h0IGfkBzl2VVzV5Ws1a58ejqrQ7qJAlfrZ699y7XvwPms31bGzLG+DQjSErUuSw0qW5G3c6EDaeqczOoILZ1ny9VFXwZAmo9bCLK+wLokIHTD2wo9kCpZmdUR2g6dPAm03C3QlwLpIinUSEDohrcVutGVrC/7XmVWR2jLO1mGArKSevv0OIF0kRRqzF+OaHHZ2Ys69HcCaBXjdoXurK82O3uRR5WsP/peZVZH6FryeDIDZyUQExnp0+M4+5+Slqg1SEDoRucKvawxif36B8zoQaVpVCUbSAPUIvB8uWQYc2/M9flxjLxIEsaSgOCG9hX6Q6tX8caunfznBXV+X8Ajfa/uWbBAZkD1VEuW0zCG9D4to4xPSEvUmmQMoYduHj2W+uYmlu/Z5fdjS9+reyTpnvtaU5FEhrdUBQ9Omy6pSEKYBIQeGt0vlbH9Unl1Ry7+XmwdCrOApCLqnpHnqDUVybJdOwHYV1rqdioSea+CjwQED9w6Zhx7So+zrbDAr8f19SwgK/yDe3p1H0pJ93zRAsotKgR6NsNIWmLBR1k1pdCkSZP05s2bzS6GU1UNDUxe/CSzzxjGoxfNBk7PVxSIg2RKnUpa1xVf9tG7WwZf78PKfPH6bnj9FXYvG8fO18eYVo5g+B+yAqXUFq31JE+eKy0ED8RFRXHtiFG8s/dbSmqqAypfkRGMvjIMpat7T/nyHNU2NpJbXMRtPyszrRyh9j9kVRIQPHTnhIk02e38a/OmgF6Kb4XK2OiU2mYm3fPVefNl2vGthcdotNs5x41Mvr4qRyD/DwUTCQgeykrqzbUjRrEk95uAng7q7j+4FQKHu8wsUyD2q391NJ+TK6cwqX/nGx36TyD/DwUTCQhd6C5NxH+dPZlmbadWO7+7m5nTQY1OceGvG+NISu3uGX2OvszP58QHk4nv4Z3SjCyHTKm2BgkILrjTpzkoMZEbR41hSf5FKBXT4flmTgf1tD/WCpWxFVsc3fF368nI/VY3NHg8W87IcoTClOpAIAHBBXf7NH9+zhS+qTqbL2p/bImkcAsWeN4f6+4/uCeBIxArencF6m1FFyyAOFsUe376c8DcbkBJrGgNMu3UhTVrwgBn50YxfXrHeyv/7auN/N/G9bx6/c0e3WLTSErB6tXul91fgn0qaKtAe52/WfUhK/fvJffHPw2ocgvXZNqpD/SkT/OHEyaSFhvHH9d9ht0H/1U9HQ+Q/ljzWKHbzV31TU18sH8fs7KGmF0UYRESEFzoSZ9mTGQk906ZyvaiQlbs2W1oOdwZD+jch/3gg4uoqzO/PzaQZiYZJZBe2yff5VHZUM/Vw0cGVCATviNdRl3oycpJu9Zc//orHK2sYNVtPyChhzM2XNmwIdMRDDqy2QYzefLB0x5v7bKw2qrPQOtKCQVz3nydQ+Un+eyOHxEeJteGwcKbLiNJf92FnqToDVOKhdNncu1rS/jLxvU8NO1CQ8rg6fxsSS8surKvtJQN+Ue4b/JUCQaijXwSDDQuNY3vjR3Pi9u3sbuk2JB99nQ8wKpNf6uWK1S9nLuNqLBwbh491uyiCAuRgGCw+yZPJckWzYOrV9Fs935GT0/nZ1u1D9uq5QpFVQ0NvLV7F5cNHUZyr17dP0GEDAkIBkuMjub3F0xna2EBj3+1wev9Ber8bKNXSgvjvJL7DVWNDdwxfoLfjy2fC2uTMQQfuGbEKL7IP8wTX21kQtoApmdmebW/QBsPaJ0Z1bo4rnVmFBBQryMY1TU18szXmzlvUAbj0/r79djyubA+aSH4yMPTZzIipS+//Oh9jlZUmF0cv+pupbR0H5nn9Z07OF5Tw91nnev3Y0tGU+uTgOAj0RGR/OOyK2my27l75TvUNzWZXSS/6W5mlFEZQSWw9ExDczP/2rKJif0HuJXq2miS0dT6JCD4UGZSbx69aDbbiwr552cPsGHDYJ/3nVqhj9ZfK6X9nWrarABk1Hv6xq4dFFRVcvdZ56JaVwv6kaygtz4JCD52yZChPDihgpHqMceVkO/uBmWVu045mxnV3NyLBx9cFNCrls2414FR72llfT1/2fgFZw0YyLTBmT4pa3cko6n1SUDwgxERz2MLa+zwmC/6Tq3SR+tsZtSYMU+xatUcrzOChlo6DKPe06e2bqK0tobfnT/dlNYBBO6MuVAis4z8oL7+iIvHje07tVIfbfuZUa1pNHbvvg2bLYOZMxcBnlUCCxacqvz9kQ5jwYKOLYPWunT+fP8EISPe04LKSp7ZuoWrho9gfGqaUUXzSKDNmAs10kLwA3/1nVqxj9ZZl8dvfxs4N083+14H3r6nWmsWfPYJGs19k6caWTQRhCQg+IGzvtMGexQxfe/3+XHM7qN11uURHm5MN1YopMPw9j1d/PUWPs47wK+mnE96QqIviiiCiAQEP+jcdxoemc6bJXP4wafNfFNU6LPjWKGP1pfdWP4eNzAjAHnznn51NJ9H1q/lkiFD+UHOmb4vrAh4XqW/Vkr1AV4DMoGDwE1a6xNOtmsGch2/HtZaX9Xdvq2Q/tqXDpef5La3llFWW8PTV17LuemDzC6ST/Q0fbcwRkl1NVcsfYnYqCjevnkO8QalYxfWZ+Yd0+4HPtFaDwU+cfzuTK3WOsfx1W0wCAUZiUm8dsPNDIhP4M633+ST7w6YXSSfsGI3VrBrstv56cp3qWqo55+XXyXBQLjN24BwNfCC4+cXgGu83F9ISYuLZ+n1NzEsOZm73n2bl7dvM7tIhrNiNxYE5zTVlgFwzUOrV/HVsXwWXXgRw5NTzC6WJRZLCvd422V0Umud5PhZASdaf++0XROwDWgC/qS1Xu5if/OAeQAZGRkTDx06vashGFU1NPCzD95j9cE8bhuXw++mTsMWITOCfSkQ7+DWfsqtM0rBE19u5P82rufus87hXi9nFXV3PHd0TmgHLS1EK1wUBCtvuoy6DQhKqVWAs8nLDwAvtA8ASqkTWuveTvYxUGt9VCmVDXwKzNRad9lHEuxjCJ012+08sn4tz3y9hdF9+/HX2ZeT3buP2cUKWoEYELors1KQ9df/x3UjRvHoRbO9XoBmxDmSMST/8+kYgtZ6ltZ6jJOvt4EipVR/RyH6A05vE6a1Pur4ngesAfyfiN3iwsPC+N3503n6ims4WlnBVa++zFu7d5ldrKASjKucO7+m7352L//vkktZuNCc1cjQsYvIWTAASWhnVd6OIawA7nD8fAfwducNlFK9lVI2x88pwHmA1HQuzMwewnu33s7ovv249+OV/PzD9zhRW2t2sYKC2YvMXJWpu793FcTmz9e8sO1rhjz+fwBU1NV79Zq8DZqdFyK6IgntrMnbMYRk4HUgAzhEy7TTMqXUJOAurfWPlFJTgH8BdloC0F+01ou723eodRl11mS3849NX/K3TRtJio7mDzNmcfGQoWYXy+da01zU1x/GZssgO3uRT/qaZ81awn//t++PA12/pp50y3Tetr6piflrPuH1XTu4MDObxVdfa2g3mCddRq66iNqTMQTf8ukYgllCPSC02lVSzK8//oBdx0u4OPsMfjt1GoOTThu3Dwr+GoAsKlrCjh3zCA/3/UBnd6/J04BQXF3FT95bwdbCAu4+6xx+ce55PLxQGdra8SQgrFkThuuWgfJ58BXmrkMQPjaqbz/eunkO902eyrojh7jk5ef50/q1VNbXm100w/krW2te3gMdgoGr4xhRubp6TatXP9DjbpnWldLfFBZw9atL2H28hH9eoJnC91n7WTiXXGLslE5PVma7zr00mOnT7UyefFCCgYVJCyGAFFVV8ecN6/j37p0kx/Til5PP44aRo4kMDze7aIZwfXWpmD7d7vfjGDHLprtj9eQYWmuW7d7Jg6tX0S82lr+c10RVwX2WmtIp00zNJy2EEJEaF8ejF81m+c1zyExK4oFPP+bil5/nzd07aWxuNrt4XgvGrLBGHWv38RK+/9Yb/GbVh0zsP4DlN8+h/vgjlrj/RXtWXYgo3CMBIQCNS03j9Rtu4ekrryEuKor7Pv6A6S8s5l9bvqK8rs5nx/X1ilN/pbno6jhGT03t7jV11y1TWlPDA59+zJVLX2L38RIWTLuQF6+5gT4xvSx1/4v2UlPnMHnyQekiCkDSZRTgtNZ8ejCPZ7/eyob8w/SKjOT6kaO5Y/wEQxe2+XPA1x+zjNw5jlGL1zx5TQ3Nzby0fRuPf7mBmsYGbhuXw3+dM5mk6Ji2bWTRl3BGZhkJAHaXFPPstq28s+dbGu3NzMjM5s4JZzIlPcPrVauhWPmYsZq5NcD/z7rPyDtxggsyMvn9BdM5o0/yadtKf71wxpuAIAlzgsjIvv149KLZ/HrK+bycu41Xcr/htreWMTA+gUvPGMplQ4czPjXNo+Bg1e4JX/Ln/Q8q6ut469tdLM3dzt6yUrKSevPMldcyIzPL5fvVWun7o0UlQoO0EIJYfVMTK/fv5Z29e1h3+CCNdjsD4uO59IxhXHbGMHLS+rsdHEKxheBrWmu2FxXyyo7tvLP3W+qamhjbL5Xvj8vh6uEjiQqS2WPCv6TLSHSror6OVXkHeH/fXtYdPkSDvZm02DimZWYxbXAWUwZlkNBF3nzpnjBOcXUVH+zfxxu7drCzpJhekZFcPXwkt44Zx5h+qWYXTwQ4CQiiRyrq6/kk7wAf5e1n/ZFDVDU0EBEWxplpA5iWmcm0wVmMTOl7WuvBXwO+wehoRQUfHNjHB/v3srXgGBoYmdKXW8eM4+rhI+UmNsIwEhCExxqbm/m6sIA1B79j7aHv2HW8BIDkmF5MGjCQM/v3Z1y/NMb0SyU2Ksrk0gaO+qYmdpYUsyH/MB8d2E9ucRHQEgRmnzGU2UOGMTT59IFiIbwlAUEYpri6is8OHWRj/hE2Hcsnv6ICAAWc0SeZcalpjO7bj6HJyQxLTiElppfXM5gCndaao5UVfF1YwNcFx/i6sIBdJcU02ltWPY9LTWP2kKHMPmMomUmn3S5ECENJQBA+c7ymhu1FhS1fxYXkFhVS2i4dd5/omLbgMLRPMlm9ezM4MYn+cfGEhwXfusfK+nr2l5Wyt6yU/WWl7CstZdfxYo7XtIytREdEMK5fGjn9+3NmWn8m9B9A316xJpdauvtCiQQE4Tdaa47X1LCn9Dj7ykrZW3qcvaXH2VdaSlVjQ9t2kWFhDIhPYHBiEhmJiWQkJpEWF0e/2DhSY+PoFxtLTGSkia/EObvWFFdXcbSygqMVFRyrrORoZQVHysvZV1ZKQVVl27a28AjO6NOH4ckp5KT1Z0Jaf4Ylp1gut5RMCAgtEhCE6bTWHKuq5NDJkxwpP8mh8nIOl5/kcEXL9won2VkTbDZSY+NI6RVLUnQ0iTYbidHRJNqiHd9txEfZsEVEEB0RgS0iAlt4OFHh4djCI4hwtEC0I3lc241v0BwvfpXi/Pk0NR4lLGIg4X1+RWOvK6lpbKSivo6y2lpO1NVyoraWstbvtbUUVlW2dfW0SoqOJj0+gSF9khnaJ5lhyckM7ZNCekJCQLSCZMpwaJGFacJ0SikGxicwMD4BBp2euK28ro6i6iqKqqsorq6mqKqK4uoqiqqrKamuYm9pFeX1dVTU1dNg9y5R3zlJW5ibvgxbWCMA9qZ8agvv4/n8jXx5cmLbdr0iI+kdHUPvmBj6RMeQmdSb/nHxDIiPZ2BCQtvr6W4w3erdMaG4qFB4RgKC8IvE6Jar/mHJKV1up7WmrqmJ8vo6yuvrqayvp66pifrmJuqbmqhraqKhuZmG5mbsWrcllm4d1lYKBpb/mQh7Y4f92sIauSt7LQ+N+CuJNhu9o2MM6bLq3B1TX3+IPXvmAVgmKNhsGS5aCHIbS9GRBARhKUopYiIjiYmMJC0u3qN9rFlT5PRx3XSMkSl9vSneabq6qY9VAkJ29iKnYwhGZ5EVgc/6HaBC9JA/73cQCN0xco8C4S5pIYig488r4kDpjklNnSMBQHRLWggi6PjzithfN/URwh+khSCCkr+uiCUFtQgmEhCE8JJ0x4hgIV1GQgghAAkIQgghHCQgCCGEACQgCCGEcJCAIIQQApCAIIQQwkECghBCCEACghBCCAcJCEIIIQAJCEIIIRwkIAghhAAkIAghhHDwKiAopW5USu1UStmVUi5v6qyUmq2U2qOU2q+Uut+bYwohhPANb1sIO4DrgLWuNlBKhQN/By4FRgG3KqVGeXlcIYQQBvMq/bXWeje03Ae3C2cD+7XWeY5tXwWuBnZ5c2whhBDG8sf9EAYCR9r9ng+c42xDpdQ8YJ7j13ql1A4fly1QpADHzS6ERci5OEXOxSlyLk4Z7ukTuw0ISqlVQJqTPz2gtX7b0wM7o7V+CnjKcdzNWmuX4xKhRM7FKXIuTpFzcYqci1OUUps9fW63AUFrPcvTnTscBQa1+z3d8ZgQQggL8ce0003AUKVUllIqCrgFWOGH4wohhOgBb6edXquUygcmA+8ppT50PD5AKfU+gNa6CbgH+BDYDbyutd7pxu6f8qZsQUbOxSlyLk6Rc3GKnItTPD4XSmttZEGEEEIEKFmpLIQQApCAIIQQwsEyAUHSYJyilOqjlPpYKbXP8b23i+2alVLbHF9BNVDf3fuslLIppV5z/P1LpVSm/0vpH26ci7lKqZJ2n4UfmVFOX1NKPauUKna1Pkm1eNxxnrYrpc70dxn9xY1zMV0pVd7uM/GQWzvWWlviCxhJy4KKNcAkF9uEAweAbCAK+AYYZXbZfXAu/he43/Hz/cAjLrarMrusPnr93b7PwE+AJx0/3wK8Zna5TTwXc4G/mV1WP5yLC4AzgR0u/n4ZsBJQwLnAl2aX2cRzMR14t6f7tUwLQWu9W2u9p5vN2tJgaK0bgNY0GMHmauAFx88vANeYWBYzuPM+tz9Hy4CZqpscKgEqVD7z3dJarwXKutjkauBF3WIjkKSU6u+f0vmXG+fCI5YJCG5ylgZjoEll8aVUrXWB4+dCINXFdtFKqc1KqY1KqWAKGu68z23b6JapzeVAsl9K51/ufuavd3STLFNKDXLy91AQKvWDuyYrpb5RSq1USo125wn+yGXUxp9pMKyuq3PR/hettVZKuZobPFhrfVQplQ18qpTK1VofMLqswvLeAZZqreuVUv9JS8vpQpPLJMy1lZb6oUopdRmwHBja3ZP8GhC0pMFo09W5UEoVKaX6a60LHE3eYhf7OOr4nqeUWgNMoKW/OdC58z63bpOvlIoAEoFS/xTPr7o9F1rr9q/7GVrGoEJR0NQP3tJaV7T7+X2l1D+UUila6y4TAAZal1GopMFYAdzh+PkO4LTWk1Kqt1LK5vg5BTiP4Ekp7s773P4c3QB8qh2jaUGm23PRqZ/8KloyAoSiFcDtjtlG5wLl7bpeQ4pSKq11TE0pdTYtdX33F0xmj5a3GxW/lpY+v3qgCPjQ8fgA4P12210G7KXlSvgBs8vto3ORDHwC7ANWAX0cj08CnnH8PAXIpWXWSS7wQ7PLbfA5OO19Bh4GrnL8HA28AewHvgKyzS6ziefif4Cdjs/CamCE2WX20XlYChQAjY664ofAXcBdjr8rWm7GdcDxP+F0tmIwfLlxLu5p95nYCExxZ7+SukIIIQQQeF1GQgghfEQCghBCCEACghBCCAcJCEIIIQAJCEIIIRwkIAghhAAkIAghhHD4/+A6zvTVcV0vAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "create_plot_for_lambda(X, y, 10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Underfitting$^2$\n", "This is what model trained with an extremely high value of $\\lambda$ looks like:" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Optimization terminated successfully.\n", " Current function value: 0.687913\n", " Iterations: 10\n", " Function evaluations: 23\n", " Gradient evaluations: 23\n", "Conjugate gradient found the following values for theta - first five values only: [ 0.00980668 -0.01412537 0.00377853 -0.042739 -0.01022205]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl4VdW5+PHvykASSCAhhBCGQKJhHsIgGkVFQUFkqBVblati9VJ7xbZW22q9Ctpyq7X32jr0Z21xLEWUomKFqigUGVQGw0yQIUAgJCEQIGTOWb8/chJCyHCGPZ1z3s/z5MlJss/e6+xzst+937XWu5XWGiGEECLM7gYIIYRwBgkIQgghAAkIQggh3CQgCCGEACQgCCGEcJOAIIQQAjAoICilXlVKFSqltrfw97FKqVNKqWz31xNGbFcIIYRxIgxaz+vAi8CbrSzzhdZ6skHbE0IIYTBDrhC01quBE0asSwghhD2MukLwRJZSagtwFHhYa72j6QJKqVnALIAOHTqM7N+/v4XNE0KIwLdp06bjWuskX55rVUDYDPTWWpcqpSYB7wMZTRfSWr8CvAIwatQovXHjRouaJ4QQwUEpddDX51oyykhrfVprXep+vAyIVEp1sWLbQgghPGNJQFBKdVNKKffj0e7tFluxbSGEEJ4xJGWklFoIjAW6KKXygDlAJIDW+mVgOvAjpVQNUA7cqqXMqhBCOIohAUFrfVsbf3+RumGpQogAVV1dTV5eHhUVFXY3RQDR0dH07NmTyMhIw9Zp5SgjIUQAy8vLIy4ujj59+uDOAAubaK0pLi4mLy+PtLQ0w9YrpSuEEB6pqKggMTFRgoEDKKVITEw0/GpNAoIQwmMSDJzDjPdCAoIQQghAAoIQIsC8//77KKXYvXt3s3+fOXMmixcv9nh9R48eZfr06QBkZ2ezbNmyhr+tWrWKdevWed3GPn36cPz4ca+fZzcJCEIIU82da+z6Fi5cyJgxY1i4cKEh6+vevXtDADEqIAQqCQhCCFM9+aRx6yotLWXNmjXMnz+ft99+G6gbcTN79mz69evH+PHjKSwsbFi+T58+PProo2RmZjJq1Cg2b97MhAkTuOiii3j55ZcByM3NZfDgwVRVVfHEE0+waNEiMjMzeeaZZ3j55Zd57rnnyMzM5IsvvqCoqIibb76ZSy65hEsuuYS1a9cCUFxczPXXX8+gQYO49957CdRpVjLsVAgRMD744AMmTpxI3759SUxMZNOmTRw8eJCcnBx27txJQUEBAwcO5Ac/+EHDc1JTU8nOzubBBx9k5syZrF27loqKCgYPHsx9993XsFy7du146qmn2LhxIy++WDdtqry8nNjYWB5++GEAbr/9dh588EHGjBnDoUOHmDBhArt27eLJJ59kzJgxPPHEE3z00UfMnz/f2h1jEAkIQgjDzZ17/pVB/YCYOXP8SyEtXLiQn/zkJwDceuutLFy4kJqaGm677TbCw8Pp3r0711577XnPmTp1KgBDhgyhtLSUuLg44uLiiIqKoqSkxKvtr1ixgp07dzb8fPr0aUpLS1m9ejVLliwB4MYbbyQhIcH3F2kjCQhCCMPNnXvuwK8UGJFBOXHiBJ9//jnbtm1DKUVtbS1KKW666aZWnxcVFQVAWFhYw+P6n2tqarxqg8vl4ssvvyQ6Otr7FxAApA9BCBEQFi9ezB133MHBgwfJzc3l8OHDpKWlkZiYyKJFi6itrSU/P5+VK1f6vI24uDjOnDnT4s/XX389L7zwQsPP2dnZAFx11VX8/e9/B2D58uWcPHnS5zbYSQKCEMJUc+YYs56FCxdecDVw8803k5+fT0ZGBgMHDuTOO+8kKyvL521cc8017Ny5k8zMTBYtWsSUKVN47733GjqVn3/+eTZu3MjQoUMZOHBgQ8f0nDlzWL16NYMGDWLJkiWkpqb69VrtopzaGy43yBHCWXbt2sWAAQPsboZopLn3RCm1SWs9ypf1yRWCEEIIQAKCEEIINwkIQgghAAkIQggh3CQgCCGEACQgCCGEcJOAIIQICEopHnrooYaff//73zO3jToY77///nmlJnzhbSnrpUuX8vTTTze7/ddff52jR496tf364ntWkIAghDBFQcEC1q/vw6pVYaxf34eCggV+rS8qKoolS5Z4dXA2IiB4a+rUqTzyyCPNbt+XgGAlCQhCCMMVFCwgJ2cWlZUHAU1l5UFycmb5FRQiIiKYNWsWzz333AV/y83N5dprr2Xo0KGMGzeOQ4cOsW7dOpYuXcrPf/5zMjMz2bdv33nP+fDDD7n00ksZPnw448ePp6CgAGi5lHVubi79+/dn5syZ9O3blxkzZrBixQquuOIKMjIy+Prrr4G6g/7s2bMv2P4zzzzDxo0bmTFjBpmZmZSXl7Np0yauvvpqRo4cyYQJE8jPzwdg06ZNDBs2jGHDhvHSSy/5vM+8prV25NfIkSO1EMI5du7c6fGy69b11itXcsHXunW9fd5+hw4d9KlTp3Tv3r11SUmJfvbZZ/WcOXO01lpPnjxZv/7661prrefPn6+nTZumtdb6rrvu0u+++26z6ztx4oR2uVxaa63/8pe/6J/97Gdaa60feOAB/eSTT2qttf7nP/+pAV1UVKQPHDigw8PD9datW3Vtba0eMWKEvvvuu7XL5dLvv/9+wzZfe+01ff/99ze7/auvvlpv2LBBa611VVWVzsrK0oWFhVprrd9++2199913a621HjJkiP73v/+ttdb64Ycf1oMGDWr2NTT3ngAbtY/HXal2KoQwXGXlIa9+76mOHTty55138vzzzxMTE9Pw+/Xr1zeUn77jjjv4xS9+0ea68vLy+P73v09+fj5VVVWkpaUBtFrKOi0tjSFDhgAwaNAgxo0bh1KKIUOGkJub69VrycnJYfv27Vx33XUA1NbWkpKSQklJCSUlJVx11VUNr2f58uVerdtXkjISQhguKqr54m4t/d4bP/3pT5k/fz5nz571az0PPPAAs2fPZtu2bfz5z3+moqKizec0LZ/duLS2t6W0tdYMGjSI7OxssrOz2bZtG5988ol3L8JgEhCEbYzudBTOkZ4+j7Cw9uf9LiysPenp8/xed+fOnfne97533l3JLr/88oZbai5YsIArr7wSuLB8dWOnTp2iR48eALzxxhsNvzeylHVr5bT79etHUVER69evB6C6upodO3YQHx9PfHw8a9asaXg9VpGAIGxhRqejcI7k5Bn06/cKUVG9AUVUVG/69XuF5OQZhqz/oYceOm+00QsvvMBrr73G0KFDeeutt/jjH/8I1N1V7dlnn2X48OEXdCrPnTuXW265hZEjR9KlS5eG3xtZyrrp9mfOnMl9991HZmYmtbW1LF68mF/+8pcMGzaMzMxM1q1bB8Brr73G/fffT2ZmpqX3Z5by18IW69f3cQeD80VF9SYrK9f6BvmhoGAB+/c/RmXlIaKiUklPn2fYgc9JpPy18xhd/lo6lYUtzOp0tFr9lY7LVQbQcKUDBGVQEMFNUkbCFmZ2Olpp//7HGoJBPZerjP37H7OpRUL4TgKCsIWZnY5WCpYrHU85NcUcisx4LyQgCFuY3elolWC50vFEdHQ0xcXFEhQcQGtNcXEx0dHRhq5X+hCEbZKTZwRcAGgqPX3eeX0IEJhXOp7o2bMneXl5FBUV2d0UQV2A7tmzp6HrlIAghB/qA1oojDKKjIxsmM0rgpMEBCH8FAxXOkKAQX0ISqlXlVKFSqntLfxdKaWeV0rtVUptVUqNMGK7oUJm9Mo+EMIKRnUqvw5MbOXvNwAZ7q9ZwP8zaLtBL9hm9PpyYA+2fSCEUxkSELTWq4ETrSwyDXjTXZ31SyBeKZVixLaDXTCNc/f1wB5M+0AIJ7Nq2GkP4HCjn/PcvzuPUmqWUmqjUmqjjGSo44Rx7m3cpdBjvh7YnbAPhAgFjpqHoLV+RWs9Sms9Kikpye7mOIITxrk/+aQx6/H1wO6EfSBEKLAqIBwBejX6uaf7d6INwTKjF3w/sAfTPhDCyawKCEuBO92jjS4DTmmt8y3adkCza0bv3LmgVN0XnHvsT/rI1wO7VftARjKJUGdI+Wul1EJgLNAFKADmAJEAWuuXlVIKeJG6kUhlwN1a61ZrW0v5a/8ZVZZZKTCqWoFZpaL9XW/TqqVQF6wCsZyGCG3+lL+W+yE4lJMOcEYGBDMY8VqD6f4MIrT5ExAc1aks6hgx7t7IoZpz5nj9FEsZ8VpDdSSTpMlEYxIQHMhpBzijhp2axYjXGoojmWTCn2hKAoIDyQHOO0a81lAcySQT/kRTEhAcSA5w3jHitQbL/Rm8EappMtEyqXbqQEbU2A+lssxGvdZQq1oaFZXaQkd68F1FCs/IKCOHMmt4phD1ZKhtcPJnlJFcIThUqJ2thio7A38oXUUKz0hAEI40d67zRzf5q+kZev0oH8DSoCABQNSTTmVhGn8O6EYV1HMyGeUjnEYCgjBNKBzU/SGjfITTSEAQF7ArVWNGQT1/Z+KaOZM3lOaKiMAgAUFcwJ8ze38O6nPn1tVMqh/4Vv/Y14Dg70xcs2fyhtJcEREYZNipuIBRxez8WY8RbfC3YJ0VBe98HWUkw5JFS6S4nfCbGekafxhRUM/fHL0VOf7k5BlkZeUydqyLrKxcj4OB1CASZpCAIADj0zXg30HdiEDkb47eqTl+f0cnSYVT0RIJCMI0ds8j8DdH79Qcvz9XLnJ1IVojAcEBjDxjM2JdTr//gaf8LVjn1IJ3/ly5yNwH0RrpVLaZkfVknFKbxq5ZxqEwuxn8e59XrQoDmvufV4wd6zK2ocIW0qlsMSPP6I08Y7Pj7K+5A7A3w1aNPIA7bSKcWcHJnysXp/aLCGeQKwQvGX0WbuQZmx1nf80ND/VmyKiR92t22r2fndYecM5VpDCPXCFYyOizcCPP2Ow8+7Nr2KrThssazejXYUe/iIxqChwSELxk9Nh0I0eyWDUqprmD8JNP1nVGezJs1ciDuBnDZf1hdIAyIw3my9wHX8mopsAiKSMvmTF71chZp1bPYJWUUcuMaI8dr8nIz5AVs73F+SRlZCEzzsK9OWNr60zTyrM/pwmW4bJ2psGMPqOXiq6BRQKCl+wem+60kTT1B2FfD2JGHsTt7DdoLk/u62uzMw3m5D4yYT5JGQUYp6VFmhMIbTSSmSN3rN6XRo9Uk1FN1pOUUYDw9Qwv2EfSBDoz538YeQXlyWgfo8/o7b6iFt6RKwQLBWono7dCZcZwvZbOqrVWXHONM2b/enqmLmf0gU+uEISjBEowMKqdLZ09FxQ4J0/u6VWMnNGHNgkIJjM63RMsI2mcwKgO+pZGnv31r86585k3o31CeaRaqJOAYDKjR4wEytl3Uy21O1BfT2ONz6q1Vhw71punnnqFzz6b4Zj+HhntIzwhfQgeMmKyTiDk/83S0mu3ep/Mndv8lUHj4bNGcdL7LX0DoUP6EExm1GQdSffYr7UrttZSSHaf4ftL+gaEJyQgeMCoYYWBflDxVkv9J2PHBt4wWl/6G5x2AiB9A6IthgQEpdREpVSOUmqvUuqRZv4+UylVpJTKdn/da8R2rRIM0+/tumFNc2fjq1Y5oyDdnDnmzvFwcoAzklQzDR5+9yEopcKBPcB1QB6wAbhNa72z0TIzgVFa69mertdJfQhGFeiyuvBcY63ls6tqazlRXsaJ8nKKy8ooLi/nTFUllTU1VNXWUllbQ2VtLVU1dd8BIsLCCA8LIzIsjAj3V2RYOB3ataNjVBQd20XVfY+KomN0ND07dsTl0qj6I68H7bJa07a01t/g9IO9VXNBpG/CefzpQzAiIGQBc7XWE9w/Pwqgtf5to2VmEsABwYgPvZ3/OGXV1XRoF8m/9n7LwZISDp6q+zpy+jTF5WWUVlW1uY524eG0Cw8nKjwcpRQ1LtcFX605uTyL7lM2kBwbR7cOsXSLjSU5Npb1b2XwXz8vpXd8Ar07dSI6ItKol+211oKTkwKXJ6xqr1QzdR5/AkKEAdvvARxu9HMecGkzy92slLqKuquJB7XWh5suoJSaBcwCSE11znC4+gO2P2f3rfVDtFXd1JszvYLSUrYXFrC18Bhv/rEz2W8PAOoOshMvzgAgZfIGsu4oYkhyMl1i2tM5pj2dY2IavifGxPDy7+N4fI4mKjycyPBwwpqc2TeltabG5aKsuprTlZWcqqzgdGVlw+NTV1RQeHYYBWfPcKy0lA1Hj1B4tpTqgRu476O6dSggJS6OtPiEuq+EzqTHJzAwqStJHTp4vhN85LScfyAIhnSqOMeIK4TpwESt9b3un+8ALm18NaCUSgRKtdaVSqkfAt/XWl/b2nqddIVgBF+LhrV1pnf0zGmWfbuHr44cZlthAYVnzwIQphQXd05kcFJX0hI6M3v0pWwtKKB3p050jIpus71WnGG6tOZEeTlHz5zm4KkSDpw8yYES99fJk5ypqmxYtkv79gxK6srQ5G5c0r0nw7ul0KFdO3Mb2EgglOOwI8UlVwjO4/iUUZPlw4ETWutOra032AKCr/84zR2YT1VUsHzvHj7I2cXXR/LQQFp8ApndUhjSNZnBXZMZmNSV9pGRra6nNXanSLQ7WHx7ophdx4vYWVTIzqJCcoqP49KacKUYmtyNsX3SuLp3GoO7Jrd5FRNKzHz/GgdH6UNwHrsDQgR1aaBxwBHqOpVv11rvaLRMitY63/34JuCXWuvLWltvsAUEb/5xWjrTu312MXET1rAq9wBVrlrSExKY1m8AU/sOoHd8fKvb9+QMNxA6Uc9UVvLNsXw2HM1j7aFDbCnIRwOJMe0Z2yeNsb3TGJPam07RbV8FBTMzA0LTdds5WEJcyNaA4G7AJOAPQDjwqtZ6nlLqKWCj1nqpUuq3wFSgBjgB/Ehrvbu1dQZbQADf/nGUgt+vW8Pftm7hVGUFSe07MKVvf6b1H8DgpK4XjNoxit1XCJ4qLivji0O5rMw9wBeHcimpqCBcKUb36MmkjH5MvCiDxPbt215RkDEzxRUon41QZXtAMEMwBgRvHCg5yV83b+R/xl1H+h//l+suupgZQ4Zxec9UwsPMn08YiP/0tS4XWwqOsTJ3P8v37mH/yZOEK0VWz1RuzOjLxIv7hvyVg68C4epR1JGAEETOVlXxwtfreTV7M2FKkfT1FN74QwLpCZ0tbYdTO1E9bZfWmt3Fx/loTw4ffZvDwVMlxERE8J3+A7ln+EjL92cwCcSThVAiASFIrNi/lzmrPiO/tJTvDRzMQ5ePIam9+cMtA4kvByOtNdsKC/j7ti18kLObqtoaJvftz+xLLiMjMdGchgYxCQjOZvc8BOGnqtpanlm7mteyNzOgSxLP3zCZkSk97G5W0FDuEUlDk7vx88uvZP43m3hr6zf8c89ubri4L/ePvowBXZLsbmbAkPkawUuuEGx29MxpHlj+T745ls/MYcN5ZMzVtAsPt7tZjmJG/vpkeTmvZm/ijexvKK2u4vqLLubHo7MYmNTVn6YKg8jIJd9JyihAbc4/yn9++B7VtS6eHj+BSRl97W6ST6zsbzA6XXGqooLXsjfzWvZmzlRV8p1+A5hz9bXS+WwjmdvgHwkIAWhLwTHueO9durTvwPypN5EWn2B3k3xmZU7ZrG2drqzkL5s38PLGr+naIZb/u/4GLu3Zy/gNiTbJ7Gf/yA1yHKy5M+edRYXc9f5iEqJjWHDTLQEdDKxmVv66Y1QUD2WNYfEtt9EuPJzbl7zD79etodpd3VVYR+oj2UcCgsma5r73FB/njvfepUNkOxZ89xZS4uLsaZifzLyPQFvbNdOwbin887Y7uGXgYP608Sumv7uQAyUnzd2oOI/c/9k+IRUQ7L6Rx8nycu798D0iw8NZ8N1b6Nmx1XJOpjDqgNrarSgDXYd27Xh6/ARemjSFQ6dOMfnvb7Joxza7mxUy0tPnERZ2/uzysLD2pKfPs6lFoSNkAoJR90X2REtnz5N/eISC0lL+fOM0+tiUJvLlVpCh6oaL+7Ls9jvJ7NadRz/7hD9t+MruJoUEuf+zfUKmU9mbjiojh7zVd4LuKipk8sK3uGf4SH515Vif1mUEMzplnTqr2Si1LhcPf/ovPsjZxX9fOZYfDB9pd5OEaJF0KnvA044qM64ktNb8z5p/0yk6mvsvabXIqynMzvcHczAACA8L49nrJjLxogx+88UqFmzbYneTvGL3+2N3qlZ4LmQCgqcdVa3d2cwXc+bAqoMHWHv4EA+MzrJlfHsw5/utEhEWxh8m3si1fdJ5fOUKFu/cbneTPGZnmtDKVK3wX8gEBE87qowe8jZnjuaZtV/Qu1M8M4YM82kdwhnahYfz0qQpjOnVm0c++4TP9u+zu0mOZ/QJljBXyAQETzuqjB7y9s2xfPYUH+dHo0Y7oiSF1KHxT1REBH+ePI2BSV156NPlHD51yu4mNcuuYcFNyZyCwBIyncqeMnra/GOff8p7u3fy9b0/ItbCewALcx06VcKUhX8jLSGBd6bf6ohg3xI7q5PKrGPrSaeygYwc8lZZU8NH3+Yw4aIM04OB9AdYK7VTPL+7bgJbC47x2zX/trs5jiVzCgKLBIRmJCfPICsrl7FjXWRl5fo85PSzA/s4XVnJdwcM9Pq53h7gZX6B9SZclMHdmSN4Y8s3fHEo1+7mtMjONKHMKQgskjIy0c8+XsYXhw7y5T0/9Pq2l95e5stNS+xRWVPDDX9/k6jwcD66/U7CTLrHtRCekpSRQ2UXHGNESopp90B2SsdhKIuKiODHo7PIKT7Op/v32t0cIfwiAcEkJRXl5JacZFhyisfP8fYA75T5BaEegCb37UfvTvG8+PWXOPWKWwhPSEAwydaCAgAyu3kXEJxwgPdWqPdfRISFcf8ll7KjqJDPc/fb3RyvOP2zJawlAcEk2wqPATCka7Il25P5Bfaa1m8APTt25OWNX9vdFK+EejAX55OAYJJjpaV0jo4hLirKp+d7e4C3I00k/RfnRIaHM2PIMDblH+XQqRK7myOETyQgmORkeTkJMTE+P9/pB9ZATW+ZaXLf/gB8uCfH5pa0ToK5aIkEBJOc8DMgiMDTI64jo7r3YGnOLkd3LkswFy2RgGCSExXldI4OjYAg/RfnTO3bn29PFLP7eJHdTRGNSAluz0hAMElFdTXRkRF2N8MScmZ5zoSLMwD498FcexvioUAK5r4e1KUEt+ckIJgkOiKCqppau5shLJbUvgN94hP45thRu5vikUAJ5v4c1KUEt+ckIJikXUQEFbU1djdD2GBEtxQ25+c7uh8h0PhzUJcS3J6TgGCS6IgIKmskIISi4SndKS4v45BD75UQiPw5qBt9j5NgJgHBB57kMmMiIjhbXW1D64TdRrhnp28pyLe5JcHDn4O6lOD2nAQEL3may+wWG0f+mTP2NNLNl/xwoOSUnSy1UzwAR21+/4OJPwd1KcHtOQkIXvI0l5naKZ6isrOU2XiV4EtZAill4L8O7doR264dx0olIBjF34O6Ufc4CXYSELzkaS4ztVMnAA6fvjCPLGfhwS8lNo5jZ0sNW1+ofmYap2f373+M9PR5clA3kSEBQSk1USmVo5Taq5R6pJm/RymlFrn//pVSqo8R27WDp7nM+rTBoZIL69qYeRbuS1kCKWVgvOQOsRSUGhcQQvHKTeYPWM/vgKCUCgdeAm4ABgK3KaWa3jPyHuCk1vpi4DngGX+3axdPc5np8QkoYJfFM1Z9KUvgaykDCRgti4uKorSqyu5mBDSZP2A9I64QRgN7tdb7tdZVwNvAtCbLTAPecD9eDIxTKjDvNehpLjMuKoqMxC4NE5SC8Sw8FM9aPaW19vt2mmZ/ZpxezkHmD1jPiNoKPYDDjX7OAy5taRmtdY1S6hSQCBxvvJBSahYwCyA11bljhJOTZ3iUvxzRLYVle/fg0pq5c1XDP7JV9z/2pSxBIJUycDIN+HvGM3cupn1m6tMx9Wfg9ekYwDG5+aioVHe66MLfC3M4qlNZa/2K1nqU1npUUlKS3c3xW2a3FE5XVrL/5Albtm/GsNNgvNIxg0af20kOFAjpGJk/YD0jAsIRoFejn3u6f9fsMkqpCKATUGzAth1tZEp3AL4+knfe7518Ft5WGkFKJ3umptZFhIEBwejPTCCkY2T+gPWMSBltADKUUmnUHfhvBW5vssxS4C5gPTAd+FyHQKGX9ITO9OzYkf2HX2d96XtUVh4iKiqVH/1oHuC8D3UgpBECRWHZWZI6xBq2PqMDbqCkYzxNzwpj+H2FoLWuAWYDHwO7gHe01juUUk8ppaa6F5sPJCql9gI/Ay4Ymupkvna+KaW4tc8BLol+KSCGznmbRnDylY7d8s+cISXWuIBgNLvTMU7v0A5VhhTs11ovA5Y1+d0TjR5XALcYsS2r+XvWPDDiTag5f7Zy/UHWaWc+3qYRJE3UvMqaGorLy+gWG2d3U1pU/9nbv/+xhivX9PR5lnwm5UrUuULjDi5+aO2s2aMPb03zdfGdlKutFyhpBKcrcM9Q7ubgKwSwLx3j9/+UMI2jRhk5kb+db4FUetfuNEKwqL995sWdE21uiTMFQod2qJKA0AZ/D+jp6fNAnX9vZX8OsmbmXmVUhzGyjx0jMiyMQUld7W6KIwXSSVKokYDQBn/PmpOTZ9Cv7yucrO6M1v4dZK2o7SJVIf23pSCf/l2SiIoIrIysVX1CciXqXBIQ2mDEWXNKyn+wP/YD/nP770kfst3ng2wgTCYKdbUuF9sKChiW3M3upnjNqlIkciXqXIF1CmMTIzrfvjtgEC9t+IpFO7bx40uzfFqH5F4907jkg9V2FBVSWl3FCPekRNE8mV/gTHKFYJG0+ATG9knjra3ZPt9rWXKvnrGz6N4n+/YSrhRj+6TZ1wgvSCkS0ZgEBAv9IHMkxeVlLN2z26fnh0LuNdAPRB/v+5bRPXoRHx3T9sI+MnIf+VOKJNDfK3EhCQgWuqJXKv0Su/DXzRupdbm8fr7ZuVcn/IP7enbvhDPdvSeK2XfyBBMuutjU7Til7LhT2iGMIwHBIJ4MB1VKMfuSy/j2RDGLd273aTtmjgLy9B/cCYGjKScU3fvX3m8BuC7d3IBgFrtLkUg5C/tJQDCAN8NBJ2X0ZVT3Hjyz7gsMqf14AAARSklEQVSKy8ouXFkAMPrM0Aln9/6qcbl4e8dWsnqmkhJnfMkKK/aRp2kiM9oht8t0BgkIBvBmOKhSit9cM57SqiqeXrvaqia2yAkHY6PP7u040/3swD6OnjlD2KprTVm/E66AzGyHDKl2BgkIBvB2OGjfxC7MGnEJ/9i1g/WH7R0y6uk/uBMCh6fsaNObW7JJiY3jby8EZrkKu99HGVLtDBIQWuFpTtOX4aCzR19KasdO/PfKFT4PQ22N0flYq85Q7c5j++Lb4mLW5x1ixpBhlmzPjH3kSxrQyHbIkGpnkIDQAm9ymr4MB42OiOTX14znQMlJ/nf9Gtva3pgTDsZ2n6n64o6fFHPgJw9x/+i6W4mbffXklH1kZDtCYUh1IJCA0AJvcpq+Dge9sncf/mPIMP76zSbWHr6w7LQv5s71PR/r6T+4L4HDKQcxo+06XkTR6A95Zs0Xtuf3veWkNKCUs3AG5dQ7WY4aNUpv3LjRtu2vWhUGNLdvFGPHej+HoCXl1dVMefstyqqqWT7jLjpFR/u1PqVg5Upr2u4Npc6lm4KFS2tu/ccivi0uZtVd99ApOjpgX2egtltcSCm1SWs9ypfnyhVCC6zKacZERvLchBs5Xl7Gf6/8lOYCtLf9AZKPtcbb27ey8egRfnXl1Q2B3AlpNyF8JQGhBVbmNId0Teanl17OR9/u4e0d2877myf9AU0v/R9/fB4VFfbnY52UkjBabslJnl67mqyeqUwfMKjh94H62iSQCZCUUasKChZYds/ZWpeLe5a+x5rDB3l+4o1MyugHwPr1fVq4rWVvsrJyL/h9/aW/lW33RDClJMqrq7n53YUcKz3Dh7feQY+OHe1ukhAN/EkZSfnrVlhZojc8LIw/3TiVmR/8g59+vIyo8AjGpV/k8/hsKS9sDq01j69cQc7xIl6d+l0JBiKoSMrIQdpHRjJ/yk0MTOrK/cs+5ItDuV73Bzj10t+p7fLWwu1bWbJ7Jz++NIurA6TEtRCekpSRA5VUlHP7knfJLTnJK2MVruOPnjeMNCysvQzJs0H2sXxuXbyIrF69mD/1u4TVd44I4SAyyijIxEfH8OZ3ptOrYyf+c6WmJuE3ATc+O9gqV+49Ucw9S5fQLTaW/7t+kgQDHwXb5yLYSEBwqC7t27Pw5u/RNzGR+/4NhZ2XB8yN74OtcuXBkhLufG8x4WFhvPGd6STEmHfzm2AWbJ+LYCQBwcE6x7TnbzfdwqiUHvzs42W8smlDs/MUnKatmdKBNDRza8Expr+7kIraGt6YdjO94+PtblLAkoqmzicBweHioqJ4ddpNTLgog6fXruZHy5ZyurLS7ma1qq2RUUbdT8HswLIq9wC3L3mHmMgIFt9yGwOSupq7wSAnFU2dTwKCRfzJnUZHRPLSpCn8aszVfH5gPzf+/c0Wy2Y7IUdr1Uxps27h6NKalzZ8xb0fvkdafAL/uOV20hM623Zl44T31Agyg975JCBYwIjcqVKKe0eM4p3ptxIZHs6M997lqdUrqaipNnQ7RmhulndtbXsef3ye42ctF5eV8YMPlvC/69dwY0Y/3r75+yR16ADYcw9hp7ynRpCKps4nAcECRuZOM7ul8NFtd3DXsOG8nr2ZyQvfYsuxfMO344/mKlcOHvwKK1bM8LsiqJnlMDYczWPKwrf48shhfnPNeP4wYRId2rXzf8V+cMp7agSpaOp8Mg/BAmZVTl17+CC/+PRfFJ49y4whwxjLdaZsx19Ny2g8/vg8Vqzw/yBgVDmMM5WV/OGrdby55Rt6duzEizdMZlDXZKAu0DR3ZTBnjjVXN1ZV3RXBQ0pXOFxUVGoL9Yj8y51e0as3y2fcxe/WrWHBti0M7R9P58iThm/HH/Upj/qz3MrKgzz66CwKCrD9zLC6tpYlu3bw3FfrKDp7llsHD+WXV1xFx6iohmXmzj134LejHpNZnx0hmiMpIwuYmTvtGBXNb64Zz/IZd7G79i4qXZGmbMdXzaU8wsONSXn4Wg7DpTUf5Ozi+r+9zqOff0r32I4s+f4M5l173XnBwAkk7y6sJFcIFqg/Ezaz+ujFnRP51cTn2LQng2N5c2mvijhZnUBJzGwubj/FsO14y8yhht6mbGpcLj7dv5c/frWePcXH6d8lib9M/g7XpqWjPJh5bEc9Jis+O0LU86sPQSnVGVgE9AFyge9prS/IWSilaoH6Qv+HtNZT21p3MPUh2GHX8SJe+vpLlu/dA8BVvdP4/qAhjEtLJzI83LJ2eFu+2wzHSs/wzo7tLNqxlfzSUtLiE3jwssuZlNFPSlCIoONPH4K/AeF3wAmt9dNKqUeABK31L5tZrlRrHevNuiUgGOPwqVO8u3M77+7cTsHZUhJj2nPzgIHcPGAwGYmJpm+/aR8CWFOcr8blYt3hQyzcvpUV+/dSqzVXpvbm9iHDGJd2ERFhki0VwcnOgJADjNVa5yulUoBVWut+zSwnAcFmNS4Xqw/m8s6ObXx2YB+1WtO3cyJjUvtwea9URvfoSaxJQyytulnPyfJy1ucdZu3hg3yyby/F5WV0jo5h+qDB3DZo6HllJxp3FgcLp74mp92sKdjZGRBKtNbx7scKOFn/c5PlaoBsoAZ4Wmv9fgvrmwXMAkhNTR158OCFqQbhv6KzZ1m+dw+f7N/LpqNHqaytISIsjGHJ3bi8VyqX90wls1sKURHO7mIqq65mw5E81uUdYt3hQ+wsKkQDsZHtuLJ3H6b07c81fdKafR2BeAe3tg74Rr8mIwKMXVeIoczUgKCUWgF0a+ZPjwFvNA4ASqmTWuuEZtbRQ2t9RCmVDnwOjNNa72ttu3KFYI3Kmho25R9l3eG6g+rWwmO4tCYiLIyLOycyoEtS3VdS3ffOMe3bXqkJTpaXs/t4EbuLj9d9d39Vu1y0CwtnREr3umDWK5Whyd3aTAkFYkBoq81GvyYj1ueEPqRQY+o8BK31+FY2XKCUSmmUMipsYR1H3N/3K6VWAcOBVgOCsEZURETDgRTgdGUlXx85zDfH8tlZVMS6w4d4b/fOhuW7dYglLaEz3ePiSImNI+W877HEtYvyaMROY1przlRVcqy0lILSUgrOur9KSzl8+jQ5x4s4dra0YfnEmBj6d0niB8NHcnmvVEal9CAmMrKVLdRpOsmsvplWTTIzgxNfU+MUUfOT6qSgnVP5mzJ6Fihu1KncWWv9iybLJABlWutKpVQXYD0wTWu9s5lVNpArBOcoLitj1/Eidh0vZFdREYdOlXD0zBkKy87iavL5UUBMZCQxEZHEREYQHR5BdEQE4e4z9qraWqpra6lq9HW2uprK2poLttspKprucXH0S+zCgKQk+icm0b9LUkNtIX845QqhrbSMNzOljXhN/s7Mbi5F1By5QjCPnX0IicA7QCpwkLphpyeUUqOA+7TW9yqlLgf+DLiomwj3B631/LbWLQHB+WpcLgrOlpJ/5gz5pWc4VnqGM5VVlFVXU15TTUVNTcOXS7vQGtqFhxMZHu7+HtYQMLp2iCU5NpbOtZ/iOvE7aquPmNoBOX78An79a2s6OlvrVPXmIB4IKaOWUkSNSR+CuWwrXaG1LgbGNfP7jcC97sfrgCH+bEc4U0RYGD3iOtIjrqMh66s7u3zkvDIXOTmzAGPLXBQULODRR2dRWWnuduq31bR0h1nbsmPiXFOtp4KUjDJyOBmMLRzDqsqe+/c/Rnh429sxIgff0mtaufIxryu2NnfAb3yvhAkTjL1Xgi8BpuV7HvQOmFvAhjIJCMIxrLqjlqfbMeL+By1tq1u3Q16XAm/6d7PvleBLQJTaS4FNAoJwDKvuqGXlnbvM3JYT75Ug9zwIbBIQhMfMvpWjVWeXrW3H6BvwtPWa/Mn7O/UexcnJM8jKypUUUQCSG+QIj1g149SqMgeebMeoUTtmvSaZ9CWaY9uwUzNJQHCWUDz4OGWuQkukLIRojj8BQVJGwiNOTU+YyQnDOFsj+XphNGdXLxOOEYq3cgyEchbJyTMkAAjDyBWC8IgMJxQi+ElAEB6R9IQQwU9SRsJjkp4QIrjJFYIQQghAAoIQIcHsSYUiOEjKSIggZ2XFVRHY5ApBBCU5Iz7HiTWPhDNJQBBBx+wqoM1tz8nBJxQnFQrfSEAQQcfKM2Krg48vrKzuKgKbBAQRdKw8Iw6EdIxMKhSekoAggo6VZ8SBkI6RSYXCUzLKSASd9PR5zVYBNeOMOFBqPMmkQuEJuUIQQcfKM2JJx4hgIlcIIihZdUZcvw0rbuojhNkkIAjhJ0nHiGAhKSMhhBCABAQhhBBuEhCEEEIAEhCEEEK4SUAQQggBSEAQQgjhJgFBCCEEIAFBCCGEmwQEIYQQgAQEIYQQbhIQhBBCABIQhBBCuPkVEJRStyildiilXEqpUa0sN1EplaOU2quUesSfbQohhDCHv1cI24HvAqtbWkApFQ68BNwADARuU0oN9HO7QgghDOZX+Wut9S4ApVRri40G9mqt97uXfRuYBuz0Z9tCCCGMZcX9EHoAhxv9nAdc2tyCSqlZwCz3j5VKqe0mty1QdAGO290Ih5B9cY7si3NkX5zTz9cnthkQlFIrgG7N/OkxrfUHvm64OVrrV4BX3NvdqLVusV8ilMi+OEf2xTmyL86RfXGOUmqjr89tMyBorcf7unK3I0CvRj/3dP9OCCGEg1gx7HQDkKGUSlNKtQNuBZZasF0hhBBe8HfY6U1KqTwgC/hIKfWx+/fdlVLLALTWNcBs4GNgF/CO1nqHB6t/xZ+2BRnZF+fIvjhH9sU5si/O8XlfKK21kQ0RQggRoGSmshBCCEACghBCCDfHBAQpg3GOUqqzUupTpdS37u8JLSxXq5TKdn8FVUd9W++zUipKKbXI/fevlFJ9rG+lNTzYFzOVUkWNPgv32tFOsymlXlVKFbY0P0nVed69n7YqpUZY3UareLAvxiqlTjX6TDzh0Yq11o74AgZQN6FiFTCqhWXCgX1AOtAO2AIMtLvtJuyL3wGPuB8/AjzTwnKldrfVpNff5vsM/BfwsvvxrcAiu9tt476YCbxod1st2BdXASOA7S38fRKwHFDAZcBXdrfZxn0xFvint+t1zBWC1nqX1jqnjcUaymBorauA+jIYwWYa8Ib78RvAd2xsix08eZ8b76PFwDjVRg2VABUqn/k2aa1XAydaWWQa8Kau8yUQr5RKsaZ11vJgX/jEMQHBQ82VwehhU1vMlKy1znc/PgYkt7BctFJqo1LqS6VUMAUNT97nhmV03dDmU0CiJa2zlqef+ZvdaZLFSqlezfw9FITK8cFTWUqpLUqp5UqpQZ48wYpaRg2sLIPhdK3ti8Y/aK21UqqlscG9tdZHlFLpwOdKqW1a631Gt1U43ofAQq11pVLqh9RdOV1rc5uEvTZTd3woVUpNAt4HMtp6kqUBQUsZjAat7QulVIFSKkVrne++5C1sYR1H3N/3K6VWAcOpyzcHOk/e5/pl8pRSEUAnoNia5lmqzX2htW78uv9KXR9UKAqa44O/tNanGz1eppT6k1Kqi9a61QKAgZYyCpUyGEuBu9yP7wIuuHpSSiUopaLcj7sAVxA8JcU9eZ8b76PpwOfa3ZsWZNrcF03y5FOpqwgQipYCd7pHG10GnGqUeg0pSqlu9X1qSqnR1B3r2z5hsru3vFGv+E3U5fwqgQLgY/fvuwPLGi03CdhD3ZnwY3a326R9kQh8BnwLrAA6u38/Cvir+/HlwDbqRp1sA+6xu90G74ML3mfgKWCq+3E08C6wF/gaSLe7zTbui98CO9yfhZVAf7vbbNJ+WAjkA9XuY8U9wH3Afe6/K+puxrXP/T/R7GjFYPjyYF/MbvSZ+BK43JP1SukKIYQQQOCljIQQQphEAoIQQghAAoIQQgg3CQhCCCEACQhCCCHcJCAIIYQAJCAIIYRw+/960DobB2ZrEQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "create_plot_for_lambda(X, y, 130)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Adjusted normal equation\n", "As you would have expected, the normal equation changes when you add normalization to your machine learning model. The adjusted version of the normal equation: \n", "\n", "$$\\theta=(X^TX+\\lambda\\begin{bmatrix} 0&\\dotsi&\\dotsi&\\dotsi &0\\\\\\vdots &1&0& 0 & \\vdots&\\\\\\vdots&0&1 &0 & \\vdots\\\\\\vdots&0&0&1&\\vdots\\\\0&\\dotsi&\\dotsi&\\dotsi&1 \\end{bmatrix})^{-1}X^Ty$$\n", "\n", "If $m < n$, $X$ is [non-invertable](https://www.quora.com/What-is-a-non-invertible-matrix-What-are-some-examples)." ] }, { "cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([ 0.94416708, 0.2315894 , 0.41142826, -0.7380916 , -0.50116657,\n", " -0.69193639, 0.1003575 , -0.10718673, -0.06667559, -0.05556887,\n", " -0.45471969, 0.09321721, -0.23956536, -0.07088082, -0.40279421,\n", " 0.04618662, -0.01432296, 0.04153635, -0.10356044, -0.02654812,\n", " 0.05619122, -0.18296629, 0.12363998, -0.11445861, 0.06788982,\n", " -0.10602358, 0.04149463, -0.05781815])" ] }, "execution_count": 60, "metadata": {}, "output_type": "execute_result" } ], "source": [ "L = np.eye(n)\n", "L[0] = 0\n", "_lambda = 1\n", "theta = np.linalg.inv(X.T @ X + _lambda * L)@X.T@y\n", "theta" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: ex2/ex2data1.txt ================================================ 34.62365962451697,78.0246928153624,0 30.28671076822607,43.89499752400101,0 35.84740876993872,72.90219802708364,0 60.18259938620976,86.30855209546826,1 79.0327360507101,75.3443764369103,1 45.08327747668339,56.3163717815305,0 61.10666453684766,96.51142588489624,1 75.02474556738889,46.55401354116538,1 76.09878670226257,87.42056971926803,1 84.43281996120035,43.53339331072109,1 95.86155507093572,38.22527805795094,0 75.01365838958247,30.60326323428011,0 82.30705337399482,76.48196330235604,1 69.36458875970939,97.71869196188608,1 39.53833914367223,76.03681085115882,0 53.9710521485623,89.20735013750205,1 69.07014406283025,52.74046973016765,1 67.94685547711617,46.67857410673128,0 70.66150955499435,92.92713789364831,1 76.97878372747498,47.57596364975532,1 67.37202754570876,42.83843832029179,0 89.67677575072079,65.79936592745237,1 50.534788289883,48.85581152764205,0 34.21206097786789,44.20952859866288,0 77.9240914545704,68.9723599933059,1 62.27101367004632,69.95445795447587,1 80.1901807509566,44.82162893218353,1 93.114388797442,38.80067033713209,0 61.83020602312595,50.25610789244621,0 38.78580379679423,64.99568095539578,0 61.379289447425,72.80788731317097,1 85.40451939411645,57.05198397627122,1 52.10797973193984,63.12762376881715,0 52.04540476831827,69.43286012045222,1 40.23689373545111,71.16774802184875,0 54.63510555424817,52.21388588061123,0 33.91550010906887,98.86943574220611,0 64.17698887494485,80.90806058670817,1 74.78925295941542,41.57341522824434,0 34.1836400264419,75.2377203360134,0 83.90239366249155,56.30804621605327,1 51.54772026906181,46.85629026349976,0 94.44336776917852,65.56892160559052,1 82.36875375713919,40.61825515970618,0 51.04775177128865,45.82270145776001,0 62.22267576120188,52.06099194836679,0 77.19303492601364,70.45820000180959,1 97.77159928000232,86.7278223300282,1 62.07306379667647,96.76882412413983,1 91.56497449807442,88.69629254546599,1 79.94481794066932,74.16311935043758,1 99.2725269292572,60.99903099844988,1 90.54671411399852,43.39060180650027,1 34.52451385320009,60.39634245837173,0 50.2864961189907,49.80453881323059,0 49.58667721632031,59.80895099453265,0 97.64563396007767,68.86157272420604,1 32.57720016809309,95.59854761387875,0 74.24869136721598,69.82457122657193,1 71.79646205863379,78.45356224515052,1 75.3956114656803,85.75993667331619,1 35.28611281526193,47.02051394723416,0 56.25381749711624,39.26147251058019,0 30.05882244669796,49.59297386723685,0 44.66826172480893,66.45008614558913,0 66.56089447242954,41.09209807936973,0 40.45755098375164,97.53518548909936,1 49.07256321908844,51.88321182073966,0 80.27957401466998,92.11606081344084,1 66.74671856944039,60.99139402740988,1 32.72283304060323,43.30717306430063,0 64.0393204150601,78.03168802018232,1 72.34649422579923,96.22759296761404,1 60.45788573918959,73.09499809758037,1 58.84095621726802,75.85844831279042,1 99.82785779692128,72.36925193383885,1 47.26426910848174,88.47586499559782,1 50.45815980285988,75.80985952982456,1 60.45555629271532,42.50840943572217,0 82.22666157785568,42.71987853716458,0 88.9138964166533,69.80378889835472,1 94.83450672430196,45.69430680250754,1 67.31925746917527,66.58935317747915,1 57.23870631569862,59.51428198012956,1 80.36675600171273,90.96014789746954,1 68.46852178591112,85.59430710452014,1 42.0754545384731,78.84478600148043,0 75.47770200533905,90.42453899753964,1 78.63542434898018,96.64742716885644,1 52.34800398794107,60.76950525602592,0 94.09433112516793,77.15910509073893,1 90.44855097096364,87.50879176484702,1 55.48216114069585,35.57070347228866,0 74.49269241843041,84.84513684930135,1 89.84580670720979,45.35828361091658,1 83.48916274498238,48.38028579728175,1 42.2617008099817,87.10385094025457,1 99.31500880510394,68.77540947206617,1 55.34001756003703,64.9319380069486,1 74.77589300092767,89.52981289513276,1 ================================================ FILE: ex2/ex2data2.txt ================================================ 0.051267,0.69956,1 -0.092742,0.68494,1 -0.21371,0.69225,1 -0.375,0.50219,1 -0.51325,0.46564,1 -0.52477,0.2098,1 -0.39804,0.034357,1 -0.30588,-0.19225,1 0.016705,-0.40424,1 0.13191,-0.51389,1 0.38537,-0.56506,1 0.52938,-0.5212,1 0.63882,-0.24342,1 0.73675,-0.18494,1 0.54666,0.48757,1 0.322,0.5826,1 0.16647,0.53874,1 -0.046659,0.81652,1 -0.17339,0.69956,1 -0.47869,0.63377,1 -0.60541,0.59722,1 -0.62846,0.33406,1 -0.59389,0.005117,1 -0.42108,-0.27266,1 -0.11578,-0.39693,1 0.20104,-0.60161,1 0.46601,-0.53582,1 0.67339,-0.53582,1 -0.13882,0.54605,1 -0.29435,0.77997,1 -0.26555,0.96272,1 -0.16187,0.8019,1 -0.17339,0.64839,1 -0.28283,0.47295,1 -0.36348,0.31213,1 -0.30012,0.027047,1 -0.23675,-0.21418,1 -0.06394,-0.18494,1 0.062788,-0.16301,1 0.22984,-0.41155,1 0.2932,-0.2288,1 0.48329,-0.18494,1 0.64459,-0.14108,1 0.46025,0.012427,1 0.6273,0.15863,1 0.57546,0.26827,1 0.72523,0.44371,1 0.22408,0.52412,1 0.44297,0.67032,1 0.322,0.69225,1 0.13767,0.57529,1 -0.0063364,0.39985,1 -0.092742,0.55336,1 -0.20795,0.35599,1 -0.20795,0.17325,1 -0.43836,0.21711,1 -0.21947,-0.016813,1 -0.13882,-0.27266,1 0.18376,0.93348,0 0.22408,0.77997,0 0.29896,0.61915,0 0.50634,0.75804,0 0.61578,0.7288,0 0.60426,0.59722,0 0.76555,0.50219,0 0.92684,0.3633,0 0.82316,0.27558,0 0.96141,0.085526,0 0.93836,0.012427,0 0.86348,-0.082602,0 0.89804,-0.20687,0 0.85196,-0.36769,0 0.82892,-0.5212,0 0.79435,-0.55775,0 0.59274,-0.7405,0 0.51786,-0.5943,0 0.46601,-0.41886,0 0.35081,-0.57968,0 0.28744,-0.76974,0 0.085829,-0.75512,0 0.14919,-0.57968,0 -0.13306,-0.4481,0 -0.40956,-0.41155,0 -0.39228,-0.25804,0 -0.74366,-0.25804,0 -0.69758,0.041667,0 -0.75518,0.2902,0 -0.69758,0.68494,0 -0.4038,0.70687,0 -0.38076,0.91886,0 -0.50749,0.90424,0 -0.54781,0.70687,0 0.10311,0.77997,0 0.057028,0.91886,0 -0.10426,0.99196,0 -0.081221,1.1089,0 0.28744,1.087,0 0.39689,0.82383,0 0.63882,0.88962,0 0.82316,0.66301,0 0.67339,0.64108,0 1.0709,0.10015,0 -0.046659,-0.57968,0 -0.23675,-0.63816,0 -0.15035,-0.36769,0 -0.49021,-0.3019,0 -0.46717,-0.13377,0 -0.28859,-0.060673,0 -0.61118,-0.067982,0 -0.66302,-0.21418,0 -0.59965,-0.41886,0 -0.72638,-0.082602,0 -0.83007,0.31213,0 -0.72062,0.53874,0 -0.59389,0.49488,0 -0.48445,0.99927,0 -0.0063364,0.99927,0 0.63265,-0.030612,0 ================================================ FILE: ex3/PE3 - Multi-class Classification and Neural Networks (Exercises).ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Multi-class Classification and Neural Networks\n", "\n", "Stanford CS229 - Machine Learning by Andrew Ng. Programming exercise 3.\n", "\n", "Please check out [the repository on GitHub](https://github.com/rickwierenga/CS229-Python/). If you spot any mistakes or inconcistencies, please create an issue. For questions you can find me on Twitter: [@rickwierenga](https://twitter.com/rickwierenga). Starring the project on GitHub means a ton to me!" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pylab as plt\n", "from scipy.optimize import minimize\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Multi-class Classification\n", "\n", "---\n", "For this exercise, you will use logistic regression and neural networks to recognize handwritten digits (from 0 to 9). Automated handwritten digit recognition is widely used today - from recognizing zip codes (postal codes) on mail envelopes to recognizing amounts written on bank checks. This exercise will show you how the methods you’ve learned can be used for this classification task." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# start by loading the data\n", "import scipy.io as sio\n", "\n", "data = sio.loadmat(\"ex3data1.mat\")\n", "X = data[\"X\"]\n", "y = data[\"y\"]\n", "y = y.reshape(len(y))\n", "\n", "m, n = X.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualising the data\n", "Draw 100 random samples from the dataset." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQUAAAEICAYAAABWCOFPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzsXXdYVNfTfs82+gJSREVBAQsodsXEEgv2EguWaBRDbInGGltiiQ17VBQ1Ro0FFY29RiKxRVCwYBciRZC2lAV2l858f+xyP7DAFoz6y77Pcx7Ye8+dM7fNPWXmHUZE0EMPPfQoBe99K6CHHnp8WNAbBT300KMc9EZBDz30KAe9UdBDDz3KQW8U9NBDj3LQGwU99NCjHPRG4QMFY8ybMXb9HchdxhhLY4wlV7VsbcAYW8wY2/+OZL+Ta/i/jv+kUWCMxTLGchljsjJl8/vW612DMVYHwEwArkRk9x7a/4wxlvBvt6sO3qVxeh/t6ALB+1bgPaIfEf35vpX4l1EHQDoRpb5pJ2NMQERF/7JOenxg+E/2FCoCY2wrY+xomd+rGGOXmBKWjLEzjDEJYyxT9b99mbqXVd3zG6rex2nGmBVjLIAxls0YC2OMOZapT4yx7xhj0aou/RrG2BvvCWOsIWMsiDGWwRh7xhgbWmZfb8bYY8ZYDmPsJWNs1huO7wYgCEBNlW6/McYcVTr4MMZeAAhW1e3PGHvEGJOqzqlRGTmxjLHvGWP3GWNyxthOxlh1xth5Vft/MsYs39C+CYDzZdqXMcZqqnaLGGN7Vcc/Yoy1KnNcTcbYUdU1j2GMfVfBvbNijJ1SXetbAJxe2b+RMRav2n+bMdZBtb0ngPkAhqn0ilBtH8sYe6LSK5oxNqGMLGvV/Zeq7sm10nv3Np3f1s4HByL6zxUAsQC6vWWfMYBIAN4AOgBIA2Cv2mcFYLCqjhmAIwBOlDn2MoB/oHwYzQE8VsnqBmWvbC+A3WXqE4C/AFSD8iseCeBr1T5vANdV/5sAiAcwViWnuUovV9X+JAAdVP9bAmjxlnP7DEBCmd+OKh32qtowAlAfgByAJwAhgNmqcxKVuXahAKoDqAUgFcAdlU6GUBqWReq0r9q2GEAegN4A+AB8AYSq9vEA3AawEIAIQD0A0QB6vEX+IQCHVefSGMDL0muo2j9KdQ8FUA6jkgEYltFj/yvy+qjuJQPQCYCi9Nqq9NymukZC1bPCKtP5Te18aOW9K/BeTlr5YMsASMuUcWX2twWQASAOwIgK5DQDkFnm92UAP5T5vQ7A+TK/+wG4V+Y3AehZ5vc3AC6p/vfG/xuFYQCuvdL29tKXD8ALABMAiCs573IvJf7fKNQrs20BgMNlfvNUL9dnZa7dyDL7jwLYWub3FJQxlBW1r9q2GMCfZX67Asgtcx9evFJ/HsoY1jLb+QAKATQss20FyhiFNxyTCaBpGT0qfFkBnAAwVfX/EgAnATi/UqdCnT8Go/BfHj58TkQWZcqO0h1EdBNK686g/PIAABhjxoyx7YyxOMZYNoCrACwYY/wyclPK/J/7ht+mr+gRX+b/OAA18TocALRVdVWljDEpgJEASicLB0P5pY1jjF1hjLWr/PTfqkNNlR4AACIqUe2vVaaOpudYGcquhCgAGDLGBFCed81Xzns+lL2UV2EDZQ/g1evJgTE2SzUcyFLJMgdg/TalGGO9GGOhquGBFMprXFp/DZQ9qIuqocVc1XZNdP4g8V+eaHwrGGPfAjAAkAhl99lXtWsmgAYA2hJRMmOsGYC7UBoPbVEbwCPV/3VUbb6KeABXiMjzTQKIKAzAAMaYEMBkKA1ZbQ10KBsqmwigSekPxhhTyXqpgTx12lEH8QBiiMhFjboSAEVQ6vpUta1O6U7V/MFsAF0BPCKiEsZYJv7/3pXTjTFmAGUvaDSAk0RUyBg7UVqfiHKgfB5mMsYaAwhmjIWpofMHH5b8X+4pvBGMsfoAlkE5/vwSwGzVyw8o5xFyAUgZY9UALKqCJr9nygnM2gCmAgh8Q50zAOozxr5kjAlVpTVjrBFjTMQYG8kYMyeiQgDZAEp00OcwgD6Msa4qIzMTQD6AGzrILEUKACvGmLma9W8ByGGMzWGMGTHG+Iyxxoyx1q9WJKJiAMcALFb16FwBjClTxQxKoyEBIGCMLQQgfkU3R/b/E70iKD8MEgBFjLFeALqXVmaM9WWMOauMZhaAYiive2U6v9rOB4cPVrF/AadZeT+F46ou634Aq4gogoiioOz67VN9OTZAORmXBuVk24Uq0OMklBNT9wCcBbDz1Qqqr1J3AMOh/JInA1gF5UMLKI1XrGpIMxHKoYVWIKJnUBpEPyjPsx+Uy7cF2sosI/spgIMAolVd6zcNlcrWLwbQF8q5mxiVPr9C2e1/EyZDOXRJBvAbgN1l9v0B5f2KhHJYkYfyQ40jqr/pjLE7qmv+HZRGMhPAFwBOlanvAuBPKOemQgD4E9Ffauhcrp2Kzv99gakmP/R4D2CMEQAXIvrnfeuihx6l+C/3FPTQQ483QG8U9NBDj3J4Z0aBMdaTKT3v/imzXKNHGRAR0w8d9PjQ8E7mFFTr9pFQesUlAAiD0gnocZU3poceelQp3pWfQhsA/xBRNAAwxg4BGACl2+9rUE246aGHHu8WaURkU1mldzV8qIXyyz0JKO8RB8bYeMZYOGMs/B3poIceepRHXOVV3uNEIxH9QkStiKhV5bU/TggEAggEAtSsWROTJ0/GmTNncODAAVhYWGglz9zcHA0aNEDDhg1Ro0YNGBoawtDQsIq1/jhgZGSEgIAABAUFwdXV9X2r86+DMQaxWAxLS0vweFX7Gr+r4cNLlHeztUfVuMl+NKhbty5mzVJGMPfr1w9isRjFxcUAgN69e+PAgQMayTMxMcHhw4fRrl07REVFQS6XQyKRAACWL1+OO3c+SD+Yd4Zx48ahb9++MDQ0xJQpUzBp0qQqb6PUqOfl5VW5bG1hamqKIUOGoGnTpmjQoAEYY7hz5w7Onz+P69erhmTqXRmFMAAujLG6UBqD4VB6hGkNNzc3dOvWDZ07dwaPx0Nubi5++eUXXLp0SWdl69evj86dO6NFixZwcHBAcHAwtm/fjqysLK1lzpgxA9988w0AQCKRICMjAyYmJiAilJRo7oVco0YNWFlZYcKECbh+/TpsbGwwefJkAMCmTZvw5ZdfIiYmRmt91QGPx4NQKERhYaHa52BsbAxra2soFAqkpaVViR41a9aEl5cXCgsLUVxcjNatW8Pe3h4JCbqTOpmbKx0P27dvjzFjxiA3NxeLFy9G9erVwefzcevWLRQWFurcjjZwcXHBsmXL0LNnT/D5fBARiouL0bZtW3h5ecHf3x87dijj+uRyufYNvavwSygjyiIBPEeZcOK31KW3FRMTE1qwYAFJJBLKz8+ngoICysnJoaKiIrp37x45Ozu/9diKiq2tLdna2pK/vz9JpVJKSEigR48e0d27d6moqIhWrlypldzS4urqSkeOHKEjR46Qt7c3tWjRgtLS0ig5OZlGjRqlsTyRSET16tUj1aRsuXOIiYmhtWvX6qTvm4qdnR3Z2dlRu3bt6IsvvqAjR47QuXPnqG7dupXq2rVrVzp06BClpaVRVlYWZWZm0u+//06Ojo4669WwYUO6f/8+zZo1i5YsWULFxcU0ffp0nWS6ubnR7NmzKTIykiIjI0mhUNCTJ09o3759tHPnTrp16xb5+/uToaFhlV5jQ0ND8vDwoBUrVlDPnj3fWk8oFNKSJUtIoVBQRkYGZWVlUUREBN26dYuysrIoOzubpFIpbdu2jbZt20ZmZmZvkhOuzrv7zqIkiegcgHPvSr4eeujxbvDBhk4bGChjfRYvXozp06cjISEBmzZtQkREBJydnTFnzhw4OjrCxsYG//yjuf/PlClTAAA9e/bEokWLEBoaCoVCgS+//BLu7u5ISUmpRELFePz4Mby8vLjf3t7e4PF4KCoqwt27dzWWV1BQgOjo6DfuKykp4a6XNrCzs4OHhwcyMzORnJyM1q1bo3379nBzcwOgHF7Z2toiKSkJP//8M+Li3jyJzecraSUWLVqEadOmISgoCHPmzEFmZiaKi4vx7bffws/PD6NGjdJpaFZYWIiioiLIZDLcv38fGRkZEAi0f5RHjBiBpUuXwtTUFH/88QcA4MqVKwgODoadnR0GDx6M/fv3Y8+ePVU2v9C4cWN8+eWXaNeuHRwcHGBtbY2wsLC31h84cCC++OILFBQUICEhAUlJSdi7dy9u3ryJzp07Y9y4cahfvz6GDBkCALhx4wb27t2rnXLvm+XlbcMHZ2dncnZ2pidPnlB8fDx98skn3L7Ro0dTXl4eBQYGvq2bVGlxdHQkR0dHcnd3JwBUu3Zt2r59O8nlcjpx4gTZ2tpqLJMxRmZmZmRra0tisZjbzuPx6OrVqySVSikoKKjcEECX4ufnR35+fkRENGjQIK1k9O3bl8LCwig6Opru3btHISEhFBERQU+ePKG1a9fS2rVradq0aTRixAhycnIiPp//VlmbN2+mzZs308uXL6lPnz6vnae5uTn5+fnR1q1bSSQSaX3ejDHq0aMHWVlZkbe3N8lkMpo7d67W8rp27UqTJk0iV1fXcm18++23dPjwYerVq5dO96xHjx507NgxOnLkCEVERFBERATduXOHjh8/Tt7e3uTg4EA1atSosI2lS5dSXl4e7dy5k9q1a0dmZmblruGkSZMoPj6eMjMzKTMzkw4dOkR2dnavynm/wwddIRYrQ90tLS2Rk5OD9PR0AECDBg0wYcIEZGZmYsuWLcjJydFKfmxsLADA2toakyZNwsyZM1FcXIz58+dj165dGskVi8UYMGAAmjRpgoYNG8LBwQHx8fG4evUqAOWXtmHDhhAIBDhz5kypIdQJX3zxBby9vQEAfn5+CAoK0liGo6MjFixYgMePH2P//v1IT08HESEhIQF5eXkaXYOxY8di3LhxAJRftXPnXh85ZmVlYcGCBRgxYgSMjIxQUKBdNDYRcV90V1dXFBcXIyoqSitZAHDp0iX8/fffqF+/PhYuXAgA6NChA6ytrTFgwAC8ePFCa9mDBg3CggULcOvWLdy4cQP5+fkAgGvXruHly8oX5EqXrz/55BPw+Xzcv38fISEhr9XbunUr7OzsMHv2bABA8+bN4eDggORkLdJ7vO9ewtt6CqampmRqakqHDx8mIqKHDx/SvHnz6MCBA1RUVEQrVqzQ2nJbW1uTj48P+fj40J07dyg3N5cKCwspKiqK9u7dW+GEz6tFLBbT9u3bKTc3l2QyGb148YICAgLo5s2bJJfLSS6XU2ZmJmVkZNCZM2fI1NRU5x7C6NGjKSEhgdasWUNr1qwp1ytRt5iZmdHevXtJKpXSrl27yMvLi6ytrbXSx9bWliIjI2nXrl20a9cutY8zMDAgAwMDra+DlZUVnT17lk6cOKHTl9zd3Z1u3LhBz58/pxs3btCNGzdo+/btFBkZSWPHjtVa7k8//URERPn5+eTn50dWVlYay6hevTpVr16drl69SmlpaTRkyJC31u3Rowe9ePGCXrx4QampqTR48OBX66jVU3jvBuFtRqG01K5dm3777TfKyckhhUJBMpmM4uLiqGHDhhpfYBMTE5oxYwZFRUVRKfLz8+nFixd0584dCgoKon/++YcePXqk1qoGj8ejNWvWUE5ODmVlZdHp06e5F6tOnTrcDUpJSaGsrCw6deoUtWzZUuuHDAANGTKEJBIJzZgxQyc5jRs3ptjYWJLL5RQdHU3p6el05coV8vDw0FhWnz59KCcnh2rWrEk1a9ZU65i2bdtSYGAgBQYGkqWlpcZt8ng82rFjBxERLV68WKdrsXz5coqOjqaJEycSn8/nhkiDBg2i2NhYatWqlcYyGWO0Y8cO+umnn2jUqFF0+vRpWr9+vcZyvv76a/r6669JKpWSTCarcJWlWrVqFBwcTMHBwVRUVETffffdq3U+7uFDKeLj4+Ht7Y2bN29i3rx5sLGxgVgsRvv27fH06dPKBZSBlZUVJk+eDD6fjxUrVgAAwsPDkZKSgqSkJMTExGD58uUYO3YsjI2NK5VnZGQEBwcHEBHi4+Px448/Ii0tDfXq1cOKFSs4b0M+n4/4+Hh07NgRAQEB6NKlCxIT30TFWDHatm2LxYsXY//+/fD399f4+LKIiorCkiVLYGhoiLt378Ld3R1z587FvHnz8MUXX2i0zi0Wi2FqaspNNFYEPp+Prl27YtGiRbC0fC09xFvBGIODgwP3f4MGDdC3b18cO3YMmzZtUlvOm7Br1y4cOnQIDx48KLf92rVrKCgoQL169RAerpk3PhFxwykAyM/Px6RJk7hnQt0JS6lUytUv9aF4G2QyGSeXz+dDyRSnBd53L6GynkJpGTp0KKWlpVFISAglJyfTs2fPyMnJSWPrXdHXrFmzZhQXF0cnTpxQ6+tlYmJChw8fJplMRr6+vgSAnJyc6PLly5SVlUUSiYQkEgn99NNPVL9+fbp8+TLl5OTQ0aNHNe6q9+/fnyQSCR04cECnSToAJBAIaMuWLTRy5MjXvkoSiYRcXFw0kufi4kLPnz+nM2fO0JkzZ6hjx45Uu3ZtMjMzI1NTU6pevTo5OTnR6NGj6fLly3Tt2jVasGAB/fnnn/Tnn39S7dq1K5Rvb29Pfn5+nA/Bs2fPKDY2loqKimjevHk6XYuKioGBAV2/fp1mz56t9jFGRkbUokULqlWrVrnt3t7edOvWLTIxMSETExO15dnb25O9vT1dv36d5HI5hYaGUuPGjd9Yd8SIEZSYmEiJiYkUGRn5pl7f/8bwobT8/PPPlJ+fTwMHDqSVK1eSTCaj4cOHV9kD0KpVKwoLC6PIyEhq3bq1WseIRCLauXMnKRQKunz5Mn333XcUEhJCmZmZJJfL6cCBA3TgwAFuvOvl5UUZGRmUmZlJw4YNU1u3Ll26UFJSEoWHh3MPBGOMatWqRbVq1SIej6fRufL5fDp79ixFRUXRl19+SXw+n0xMTGjr1q0UHR1dqXPSm4qnpyddu3aNrl27RsnJyRQfH09nz56lPXv2UEBAAF26dIn27t1LgwYNIkNDQxKJRHT58mW6fPlypUOhbdu2UWxsLP3999/0999/k0wmI4lEQlFRUSSRSGjChAlqv2hmZma0cOFCWrJkCXl7e5Onpyc1atSIBg0aREOHDiUXFxdycXEhJycn+vrrryksLOxNY/O3lk2bNtGNGzfKPUMGBgb0559/0vLly7V+PseNG0cJCQmUlZVFt2/fps6dO5NQKCSRSERCoZBGjhxJjx8/JplMRjKZjPz9/d8k539j+FCKFy9eIDc3F+Hh4bC0tERRUVGVBQN99tln2LZtG/Lz8zF27Fi1/QgKCgqwf/9+dOrUCc2bN0fLli1BREhNTcXu3buxf78yj6jK8CEmJgaZmZkwMDBQa+YZANq1a4etW7ciPz8fkydPRnR0NLy8vNC5c2ccOaLkAFVXVimKi4uxYMEC/PLLL9i4cSO8vb0hFArRokULHDx4UCt34aCgIISGhgIA6tSpAwsLCxgYGCAvLw+ZmZmQSqVISkoqd0zpEKpfv35Yv379G+V+8cUXGDhwIFJSUiASiQAoXXhXrVqFsLAwrF27FuvWrUPbtm2xfv16PHz4sEI9+Xw+2rdvD09PTxQXF0Mmk0EikaB27drg8XjlfF5q1KgBPz8/HD16tAKJ5ZGQkIAuXbpgwIABMDIyQnZ2NqZNm4b8/Py3nqM6+O2331CvXj1MmDABdevWxfr16/Hy5UvIZDKYmJigadOmsLCwgEwmAwCcPn1a67Y+GqPw6NEj5OXlYdy4cTA2NoZQKNQqhuBVfPvtt/D19UVISAimT5+Ox48144H566+/sGvXLkybNg1mZmaIiYnB6tWr8dtvv71WVyKRICgoCFlZWWoZHqFQiJ9//hkuLi64c+cOevXqhTlz5iAuLg579uzBzZs3NdK1LO7cuYPu3btj5MiR6NOnD2xtbbF161Zs3LhRa9/+0iXMR48eVVJTiVJjWdH8BRFBIBDAxcWFO98RI0bg8uXLKCkpweDBgzFv3jwMGzYM3bp1g4eHR4XzNVKpFF5eXujTpw/atGkDR0dHmJqaIikpCS9fvuSOLSkpwenTpzV2NFu9ejUePXqE4cOHY9GiRRCJRIiMjMSIESOQnZ2tkayyKCwsxMqVK+Ho6IjPP/8cTk5OqFevHnd9RCIR4uPjuSXV0iVbbfBBsDmrQ7JSo0YNnDp1Ck5OTlAoFMjKykLfvn21CgLq1q0bNwnUunVrBAUF4YcfftApYKdPnz6wt7fH33//XenXShPMmzcP48ePR3x8PO7du4cLFy680QdAFxgYGKBGjRqIi4vDv/k8DBgwAAAgEom4Xs+r4PF46NevH8zNzXHlyhUAeM2jUiQSoXPnznB0dMTBgwc1fvlEIhEEAgEUCoUWZ/F2MMbA4/G46NiqQIMGDeDj4wNPT084OzuDiPD8+XMEBgYiLCyssgDB26QGVYGeuFUPPfQoj/c9yajuRCMAzqVVLpdr7FRibm5O3bp1o7Nnz1JKSgqFhIRQSEgItW/fXuOJun+zlK6YWFhYvHdd9OXDKIwxql69OtWrV4/q1atH1atXV/dYtSYaP5rhA6DsSvr4+CA3NxcHDx7UqFvWtm1b9O3bF+bm5ggPD+e6q7m5udoprYceHx/UGj58VEZBDz300An6OQU99NBDc3zURqFx48ZYsGABt36tK4YOHYqVK1fCxUWdzOd66PG/iY/GT+FViMViLF26FM2bN8fq1at1lte+fXts2LABtra2ePDggdahuJaWligpKdGJROTfhlAoBJ/PR0lJCSwtLWFqalp2EhjJycn/6bkXCwsLWFlZITMzExkZGVrLMTIyQrVq1QAo57J0kfUuobVRYIzVBrAXQHUoZzZ/IaKNjLHFAMYBkKiqziclNVuVQSAQYO3atfDw8ICPjw8Xo64tGjVqhKVLl8La2hqpqakc14I2+Prrr9GpUyeMGTOG44AQiUTo3bs3/vjjjw/u5TI2NsacOXPg5OSE/Px8tGzZEi4uLigqKuICnH755RcsWrRIK+4KExMTfPLJJygoKMCtW7fe6fl36NAB9vb2OHHihNbtlDJYubq6olmzZmjYsCFatmyJrl27wtfXF/Pnz9dIXp06deDu7o7GjRvD3d0d7dq1AwBkZGRg2LBharOGiUQieHl5wdraGsXFxSgqKgKgnHxPTk7GqVOnuG06Q4dlxBoAWqj+N4OSpNUVwGIAs97FkiSg9NtfsWIFpaena0WA+moxNjam48ePU25uLuXn51NwcDAZGRlpLa9jx45EROXi3s3MzOjevXs0bdq0976c9WoZN24cpaWlUV5eHhUVFVFeXh7nP19UVERFRUV08+ZNsrGxUVumqakp+fj4UEBAAJ09e5YSExMpISGBjh49+tZgHl1L586dKS4ujo4ePao1uWqXLl3o3LlzdO7cOXr48CFlZWWRXC6ne/fu0d69e6lbt25qyxowYAAFBgbS7du3KSMjg4qKikihUNDDhw/p4cOHdOTIEXJwcFBbnlAopDlz5tDOnTvp/PnzRESUl5dHubm5JJfLK+RZKFPebewDESUBSFL9n8MYe4JXskBVNQQCAVasWIEJEyZgxYoVXGyBLujYsSPatm0LxhgkEglmzJih09dMLBaDiMqFEdeuXRvm5uYaLaGKRCLY2trC2NgYYrEYZmZmHMdjQkICZDIZUlNTAUBrL0QnJyf4+PiAz+fj6dOniIyMLLe/lPfwypUryMzMrFSejY0yI9mUKVMwdepUCIVCKBQKHDlyBElJSRgzZgyWLFmCb775RjtGoLegWrVq2LBhA3g8HiZNmqQ1j6KVlRVatGgBADAzM8OtW7fg5+eH4OBgLoRZHfTv3x9btmyBra0tYmNj8fz5c0RHRyMoKAi///47AGgkD1C6Oa9atQoAYGtri44dOyIpKQnm5ubYs2cPJk+ejOPHj1eN92QVOR85AngBQAxlTyEWwH0AuwBYvuWY8QDCVUUta/nVV18REdGaNWt0/rIYGxuTsbExnT9/ngoKCigzM5MmTJigs9zvv/+e0tPTqV27dty2Pn360NOnTyuNvmzTpg21adOGli1bRgcPHqTQ0FB6+PAhpaSkUFFREREpSWEiIiLo8uXLHNPRkCFDSCAQaKSnqakpbdmyhXJycujWrVtakauULVZWVhQQEEABAQGUk5ND9+/fp0WLFlG3bt1IKBSSQCCgHTt2UH5+Pu3YsUMnpiQbGxuaPHkymZubEwBq3bo1KRQK2rRpk07n4ObmRrGxsRQbG0tBQUFa0dEbGhpSaGgopaen0/r168nd3V2jHoGmhcfjkb+/PyUkJKhDPPTvhE4DMAVwG8Ag1e/qAPhQrmwsB7CrKoYPXl5elJqaSoGBgdzDoEvx9vbmPCQVCgVt3LhR4xfr1cLn8+nPP/+kmzdvlounnzNnDsXHx5Obm1uFx69evZpWr15NOTk5FBwcTD/88APNnz+f5s6dS3PmzOHKvHnzyN/fnzIyMigjI4OioqIqlf1q6dChA718+ZJSU1PJx8eH267ttW3UqBElJCRQQkICXb169TW2IsYYTZ06leLj4yk2NlYrGrZSst3g4GAqLi6mvn37EgAKDAykx48fa8yvUbaYmZlRYGAgpaWlUVpaGg0YMEBjGcbGxuTn50dJSUlvJJIViUQcvZqLi0uVedJ++eWXlJeXp07uD7WMgk5LkowxIYCjAAKI6BgAEFEKERUTUQmAHVBmoNZDDz0+Euiy+sAA7ATwhIjWl9leQzXfAAADAegcMtiwYUNs2rQJV65cwdixY3WOZhOJRFxKN6FQiIyMDPj7++s8e+vi4oKmTZvi+PHj5TgOTExMoFAoKuUpKKUVCwoKwqNHjyqlbLO1tQWgZIvWZBzt7OyM5cuXQywW4+XLl7CwsMDYsWPh5OSEtm3bIjg4GOnp6dwcw+XLlyuVWRoyDgCBgYGv5c0gIpw+fRpz5swBYwxmZmYarRpVq1YNO3fuBKBcZTh8+DAuXLiA+vXro3379ti/fz+eP3+utryy4PF4mDZtGnoJlNnaAAAgAElEQVT06IEbN24AAO7fv6+xHB8fH4wfPx7379/HkydPMGzYMBQUFMDOzg4uLi6wtLRE06ZNASijfo8fPw5/f3+do2pv376NrKwsjgFdZ+gwbGgPZZfkPoB7qtIbwD4AD1TbTwGooevwYfPmzRQSEkL29vYEQKduPp/Pp/nz53P8+K92u8zNzalnz57Us2dPatSokUayR40aRVKplL755pty21etWkWPHz/Wibn41WJmZka3b9+m27dv0+nTpzVidP7pp58oOzub0tPTKTs7m5uvkMlklJ6eTjKZjEpKSujx48f0+PHjcvkQdCkeHh4kk8koMTGRy7ehThEKhbRhwwYqKSmhkpISOnr0KFWrVo0A0Jo1ayg6OlqncbutrS09fPiQ5HI5paSkUEpKCoWHh9O6des0krts2TIiIsrKyqLU1FTKy8ujtLQ0ys7OprS0NEpPT6fdu3fT7t27KTAwkPLz8+n8+fPcuWhb2rVrRxkZGbR9+/bK6r7z1YfrAN7EDFmlPgl2dnbo27cvIiMjMXXqVLRv3x7FxcUICAjA3r17NU6k6eHhgfHjx8PMzAwAEBYWhnXr1sHa2hrDhg3DuHHjULduXTDGkJqainHjxuGvv/6qVK5AIECPHj2Ql5eHW7duAQBH/tq4cWMA0NmfoizatGnDEZk+f/5co14OYwxGRkYgUsbil5SU4NmzZ9i/fz9SUlJgZGSEb775Bn369AGgzONQGfmMSCRCkyZNAABNmjSBXC6Hubk5YmNjIZfL8eTJEygUCm5lRpNZcjc3NwwYMABPnjwBoPQFyczMhIuLC4YMGYL4+Hi4u7tjz549mDZtGu7du6e2bABITU3FihUr4O3tza3kmJubY9y4cRAIBJg+fbpahD4XLlxAkyZNIBQKER0djYKCAsTGxiIgIAAymQw8Ho/r0RERFi5ciGnTpqF9+/Y4depUhbINDAy4xL6vXr+WLVtCLBZXWQLfD96jcciQIbCxsYGdnR3atGmDe/fuobCwEOvWrYOlpSXHyqwO+Hw+fHx8YG1tzTnirFmzBklJSQgMDETXrl1hbGyMoqIiMMZQu3ZtNGvWTC2jYGBgAAsLC8TGxuKff/5By5Yt0aBBAwDKxBy5ubn49ttvIZFI8Pvvv+vMGmVsbMwte8bExGg0pNq+fTtHb3f37l0UFRXhxYsX5YYgLi4u6NevHwBUytLcoEEDzJo1C927dwegzAqdm5uLvLw8MMY4o5CbmwsiwuPHjzVi4ra2tkb16tU5Z7AlS5YAABwcHFC7dm0IhUJs2bIFpqamWnuSHjhwAMePH+dYpzp27Ij9+/fD2dkZYrFYrSXE69ev49atW+Dz+Wota1+8eBETJ04Ej1f51N4PP/yAGjVqIDU1FTY2Nrh//z4YYygoKMBnn32G/Pz8KsuG/cEbBTMzM4hEIuTk5GDVqlXYuHEjjIyMcOHCBfTp0we//PKL2hayVq1a6NOnD3g8HsdlZ2xsjJkzZ6JXr17g8XjchSUiiEQimJqaqiXb3Nwc1tbWcHd3x507dzh/BUDp3ioWi+Hr64v4+HjcvHnzrfkY1UWtWrW4mA9NvxAvX77Er7/+WqHskSNHcn4KlRmcMWPG4KuvvkJwcDAAYPz48YiLi0N2djZGjBiBcePGoVu3bhy70a5duzTqKdy7dw979uzBsGHDAAATJ06EQCBASUkJCgoKIBAI8PDhQ+zZs0ctJi4+nw8+n/9ahqrc3FwIhUIAgJeXFywtLREbG6uRJ6cmWa9yc3PB4/HUosZv3LgxXF1dOVf00jylfD4fPB4PRAQPDw/UrFkTALRKIVCKD94onD17Ft999x2ysrLw119/oaSkBNOnT4ebmxv8/Pw0cgIpKCiAQqGAWCzm0nH98ssvAJRkpiUlJZzVNjAwQGpqqtqTQOnp6QgODoaFhQXS09Nx/fp17ss2ePBgREdHY/v27Xj+/LnGRKtvgpOTE/dl//vvv3WWV4p27dphwYIFaN68OZcuraKurZ2dHYYOHYrff/8do0aNAqB0tOHz+bC0tERubi5MTU0RFhaG0NBQeHh4oHbt2hrplJaWhkmTJnFkpNbW1vjuu+/g7OyMvXv34vfff8f169fV7n0NGDAAAwYMgK+vb7keC5/Px+LFiwEAo0aNgkQiwW+//aaxQ5CpqSmcnJwQERHx1jrVq1fH/PnzkZ6e/hqh7Zswd+5cpKamolq1ajA1NUXjxo1RXFwMGxsbfPfddzAwMMDcuXN1MgYcqsJ5qQp8HSqcINmxYwcVFxdTZGQkXbt2jSQSCe3fv1+rdGleXl4UERHBrfFnZ2dzrr3Z2dlcrgY/Pz/q0KGDRpODZmZm5OzszCX2bNq0KTVt2pQyMjJo/PjxVTJZBygdj06cOEFRUVEUFRWlNitTKX35r7/+SgEBAeTj40N9+/alfv36kY+PDx07doyePXvGpbtbvnx5pbTkIpGI1q1bRzdu3KDFixfT4sWLadmyZXT06FEKCwujtLQ02rp1K+e3MWjQIIqKiqK2bdtqff4mJiYUGhpK165d08qvYuLEiVRQUECHDh3iqOFdXV3J39+f81PIyMigWbNmaeVL0Lp1a3r69Clt2LCB+vXrR/Xr1ydXV1dq0KABzZgxg2bMmEG3b98muVxOU6ZM0elZMDAwoDNnztA///yjjnv3/07eBysrK1q1ahXt3buXpkyZQm3btq0w+7E6L5WDgwM5ODjQ2LFj6ZdffqHZs2fT2LFjydnZmaysrKrEscTT05M8PT1JJpNRixYtdJZXWj755BNKS0ujX3/9lX799Ve1jxs6dCgNHTqU8vPzSS6XU1paGqWmplJqaiplZmZSYWEh5ebmUmhoKE2fPp3Mzc3Veulq1apF27Zt4/z6U1NT6cKFC7Ry5UoaN27ca4bV19eXW0HQZuZ9ypQplJCQoNEKRtkiFArpxIkTpFAo6MiRI+Tn50fR0dEkl8u5FZ1JkyZpfX8sLCxoz549VFhYSFKplJKSkig1NZUkEgnl5eVRXl4epaSk0Jo1a3RO7OPu7k4JCQl07949MjY2rqz+/45R+FhL27ZtuZyJVeGFWVpGjx5N+fn5NHPmTJo5c6baxzk7O5OzszPt2rWLsrKyKDs7m7KysignJ4eio6Np3759tGHDBqpfv75Werm6upKrqyv16dOnwgAqS0tLkkqltHDhQlq4cKHa8u3s7MjOzo4iIiJ0dnXv0qULnTx5kl68eEEFBQX05MkTmj9/PpeRSdd7ZGVlRStWrKCwsDBKT0+nR48e0Y0bN7ikwJ999hkJhUKd2+nRowfJ5XLat2+fOvX1RuF9F6FQSEKhkCwsLHTy9S9bShOX5ufn04YNG2jDhg0ayzAwMKC6detSvXr1uL+1atUiQ0NDnXpgmpR58+bRsWPH6NixY2onmHV3dyd3d3c6efJkpanm1CmmpqZUq1YtcnJy4oZ8VVkYY2RtbU2Ojo5kZ2enVdbpykrr1q3p4sWL1Lx5c3Xq/+8Rt+qhnAwLDAxE//79uSSz06ZNe89a6fGRQM/RqIceemiOD35JUo/yKC4uxl9//QVXV1ccPHjwfaujx/8g9MMHPfT470A/fNBDDz00h374oMdHC2NjY1hbWwNQpoCviizkHxvMzMxQvXp1pKSkaEWs+ybojcK/CBcXF7Ru3RpisRjnz5/XOf5BVxgZGaFWrVpwcXFBnTp1AAAvXrxAZGQkoqOj8S6GlgKBAGKxGPb29lycgCbBUaVo1qwZpk2bht69e0OhUKBTp07v/Xq+DTY2NmjZsiX++OMP7pra2tqiuLiYc4XXBvXr18e6devQt29f/Pjjj1i+fHmV6PvRGQVbW1s4ODigadOmMDAwQIsWLWBsbIzjx48DAI4dO6ZWGHFp4ItQKERRUZFGgSzawMfHB9OmTUPDhg2Rk5ODBw8evPeH2NXVFT///DMaNWoEKysrMMaQnJyMuLg4xMbG4uzZswCAo0ePqhWFWRrYIxaLYWBgACJlmHTnzp3Rvn17GBoaws7ODra2trC0tOSCz3bu3KlR7o7OnTtj9erVcHd3BwCsXLmyHBEsj8eDs7MzFxylS/Tg1KlT4enpiejoaPj6+qoVp1AWPB4Pq1evhoWFBS5cuMBtHz9+PEecq20Pp1WrVujSpQvkcjnatWsHHo9XJb2lj8Io2Nvbo1atWmjXrh3HECQQCJCbmwuBQACRSIQePXoAAEJDQ7lAnjehTp06cHZ2xsKFCwEoY/UTExPx8OFDhIeH48KFC1zcflWhbdu2WLp0KWxtbcEYw5IlS6o0iOlVMMZgb28PiURSISNTcXExqlevjvj4eAQFBSE1NRVdu3ZFw4YN4ebmhtatWwMAQkJCKsxPwOPx0KdPH8yaNQuAMlir7JdQLpdDJBKBz+cjMTERV69ehVQq5XoIFQUOlUUp09SqVavQqlUrXLlyBVevXsW2bdvKcVX069cPJiYmWiX0URKKKZ+53r174/vvv0etWrVw//59jhlLE3Tq1Ak9evSAr68vAGU0LQD07t2bC8DT9kVu06YNGGMgIjg6OkIkEmnNZF0WH7xRKA05HjRoEIgI4eHh2L17N1JSUiAUCjF+/HgYGBhwN+xtltzU1BQTJkzA8OHDUbduXY5khTGGCxcuICoqCm5ubhg9ejQWL16MkydPVtk5dO3aFdWqVUNhYSGOHDmC3bt3V5nsUpiamqJRo0ZwcHBAjx49ULduXUybNq3CKM/Hjx9j0qRJiImJQVxcHMzMzODk5MSFZf/2228AUCnNmZubGzZt2sR9mTdu3IhmzZrhyZMnCA0NRXJystJTjjEkJSVp3WX+9ttvufZWrlyJ1atXv0Y9P3LkSHz//ff4+uuvNR7+mJmZYfTo0QCUPTszMzP8+uuv6NixIywtLTXW18DAAD4+PigqKsKhQ4cAKLv8pecQGBioNQWgubk5XF1dwRgDYwwpKSn/HT4FmUyGw4cP49KlS7h27RpSUlJQXFyMHj16YPz48SgoKMC6des4foC3XZhevXph4cKFEIvFkEgkOH/+PADg0KFDOHHiBPLy8sDj8XD+/HmsXbsW8fHxuHPnjk66e3h4AAC++uorCAQCSKVSbNq0SWsikFatWqF79+4gIvB4PLi5uQFQTrhZWFiAz+cjKysL9+/fx/79+xEdHV2hvIKCAo4DoUOHDti4cSMcHBxARNi8eTM2b94MAJW+XE2aNIGFhQXHe/n06dMq68qWRWnPhYgQGBhYziCIRCIMGDAAq1atws8//4zw8HC15X722Wdwc3PD4MGDOQ7Fly9fwsfHByEhIWjRogUsLCw0fulmzZqFgQMHYsWKFZBIlAnTSvNKiMVitbNDvQkdOnRA8+bNUVhYCB6Ph7i4uKrJ+YAqMAqMsVgAOQCKARQRUSvGWDUAgVDmg4gFMJSIKs8m8gaUlJRwcfRGRkbo2bMnvvjiC3To0AHXr1/H8OHDERISUqEMkUiE/v37gzGGrVu34ty5czhz5swb2zp69Cg2b96MZcuWYdCgQVp3x8RiMWbPng0AqFu3LrKysjBv3jw8evRIY1l8Ph9ff/01PD098fz5czg5OaGgoIB7KZ4/f47w8HBEREQgLi5Ooy8kn8/H3Llz8e2338LExARpaWk4fvw4li1bhuzsbLVkZGRkQCAQYO7cuQCUcxARERFQKBRVRhEG/H8CFT6fD0dHR452zcXFBXPnzkW/fv1w8uRJbNmyRS15PB4PAwcOxOrVq2FnZ4fs7GwEBAQAANavX4/Y2FgMHDgQHh4e2LFjh0ZzQE2bNsWUKVNw+/Zt7oNlZGSEwYMHAwCysrIqfW4rglgshlAo5O51VRkEAKiKYKZYANavbFsNYK7q/7kAVukSECUSiahfv3506tQpSk5OptDQUJo0aRKZmZmpFTRibW1N4eHhFBkZSSNGjKiwbu/evUkul1NycjI1adJE60CVjRs3cmnXiouL6ccff9RKjoWFBa1cuZIuX77MpVyzsrKqsgAePp9P/v7+lJubS5mZmXTp0iVydnbWSIaxsTGNGTOG7t69S3fv3qXk5GS6c+cOhYeH04ULFyggIIC++uorqlevnk66enl5kZeXF8nlcvL39ycHBwfy9fWlp0+fUnJyMo0ZM0btZ8LGxobmzp1LycnJpFAoKDY2lhYvXswlCSp7H9XJ2VG2MMZo+/btpFAoyuXUKJsb49mzZ+okb3lrWbp0KeXm5pJUKqWcnByaPn26Osf9a8lgYvG6UXgGFYszlDknn2lqFMRiMYnFYpo4cSIdP36cUlJSSC6XU3Z2Nn3//fcaXUATExM6efIkyeXySslOLl68SAqFgqKjo7V6iAUCAc2ePZvL6lRUVETXr19Xmwjl1bJw4UIiItq3bx8tWrSIRo4cSZaWllUWdQkow4gfP35MMpmMJBIJhYWFqRt1V66UvlBubm7UoUMH8vX1pQsXLlBYWBglJiaSXC6n1atXa/0ymJiYkImJCf3111+Uk5NDKSkpJJFIKDAwkHr16qW2HHd3d7pw4QJlZWVRfn4+3b17l4YOHfpavXbt2lFMTAz5+vpqpGevXr0oLS2NYmJiyNvbm7y8vGjEiBG0efNmys7OpuzsbHrw4IHWIep16tShGzdukFwup5ycHHrx4oW6H7B3nwxGBQJwkTF2mzE2XrWtOv1/7odkKLNGlQNjbDxjLJwxpv7gTw899Hj3qIKeQi3VX1sAEQA6ApC+UidTk55Cy5Yt6fTp03T69GmSyWSUn59PMpmMcnJyKCsrixITE99o2Ssqffr0oZSUFNq8efNb67Rs2ZLi4+NJLpdTaGioVvklRo8eTVKplAoLC+natWt07do1nbqJHh4etH79evrxxx9p3rx5dPXqVdqyZYtOmbHfVDp06EDHjx8nqVRKGRkZdODAAbKxsdEo2/TbiomJCX3yySe0ZMkSevr0KUVHR9P69eupfv36Gn0t27dvT+3bt6cnT56QQqGgxMREjenMxGIxBQUFUUFBASUnJ9Pq1avfqEPz5s3p5s2bJJVKqX379hq1sWHDBioqKiKZTEa5ublUUFBABQUFJJfLOeal4OBgqlGjhlbXs1u3btxwr/R+TZ06VZ1j/32SFajS0EPH4YObmxvHP3jt2jWaN28ede7cmWrUqEEzZ84kIqJ169ZpfDE//fRTCgwMpLCwMPL19SVfX1/q378/tWrVinr06EF37twhqVRKUqmUwsLCNKbKatasGcXExFBBQQHl5eVx9GdV8dKWliFDhtDff/9NNWvWrFK5gDIpyqFDh0gqlVJqaip169ZNo/Tr6rbh7+9PRETDhw+n4cOHq3WcsbExXbx4kS5evEiZmZmUkpJC//zzz2s5Kysr7u7ulJaWRhEREW+kXKtZsybVrFmT/vrrLyopKaGVK1dqTM03bNgwunv3LkVHR9Pt27cpJCSEJBIJyWQyOnXqFJ06dUprgwCAPvvsM5LJZJxRyM3NVZeJ6t0mgwEAxpgJAB4pU9GbAOgOYAmUmaHGAFip+qvRov/Tp0/RpUsXACjngNOpUyeMHz8eCQkJOHbsmMb6/v3337h79y4GDhwIT09PAMD8+fPRqFEj5OfnIz09HUuWLMHEiRMhl8tLDZZacHR0xPTp01GtWjXw+Xz8/vvvlSb4qAx8Ph9Dhw5FcXExiAguLi7o0qULCgsL1coroClSU1Nx8+ZNeHp6wszMTKu1eXXamDlzJho3boxu3boBUKaZq+xajx07llsuXLFiBYqLizFnzhzUrVtXo+VHqVSKn3/+GWfPnn0taYyJiQnngNWyZUv4+/vD399f46XVwMBAnD59Go6Ojvjnn39gbW2N69evw8TEhFuJ0NQzsiwsLCw4/wQ+n4+MjAwcOHBAa3mvQceeQT0ohwwRAB4B+EG13QrAJQBRAP4EUE3TicZXS5s2bejBgweUlZVF8+fPr7Ivl729PXXp0oU+++wzatCgAdnY2FBkZCRt3bpVIzlLly6lkpISysvLo5iYGGratKnOuvH5fAoICKDk5GRKSkqizMxMiouLKzejrav8Uvo1JycnmjRpEj148ICys7Pp5cuX1KlTJ+rUqZPa8j7//HP6/PPPK/16N2zYkKKjo2nt2rXqZEomHo9HBw8eJJlMRjKZjHr37k2TJk2ioqIidWfd1SrDhw/nJgJv3bpVZfRpQ4YMoZycHAoNDSVra2uytrbWWpaJiQkdOXKEZDIZ10vYuHGjuhPP776nQETRAJq+YXs6gK66yC6Fk5MTPv30UyxZsgTp6ekYMGCAWglP1UVCQkK5xK92dnbg8Xho1qwZBAKBWh5nlpaW6N69O4qKiiCXyzF58mS1XXcrQnFxMUaNGoW6deuiVq1aAJQ+Cbpy+zdu3Bjdu3fHJ598ApFIBLlcDg8PD1hYWMDAwABJSUn48ccfceXKFY3ktmzZEgDQrVs35OTkcMlk8vLyYGBggOrVq+Obb75Bt27dkJiYiMOHD6slt6SkBA8fPkT//v0BAOvWrYOxsTEyMzMRGxurkY5vg7W1NSZMmMDpfOnSJY1yirwNAoEAPj4+MDIywm+//aaz30bHjh3RuXNn8Hg8GBoaIjw8HMuXL9eoV1sZPiiPRkNDQ7Ru3Rq2trZo1qwZAGXijrp16+LAgQNYs2aNTl5g6kChUCA6Ohp16tSBWCxGRkZGpcc4Ozujfv36EAqFiIiIqNK4BiJCdHR0pd6JmqB169ZYvnw5SkpKQEQoLCyEXC5HbGwsLl26hHPnznGejpqg9Lw3b96MoUOHorCwEEQEhUIBU1NTMMZQWFiIQ4cOYfPmzRo5cm3btg2Ojo4AlIla+Hw+9u3bh4sXL2qs55swdepUeHh44I8//gAAbN26tUocgj799FM0bdoU4eHhnGxdkJGRgfDwcC4pzK5du5Camqqz3LL4oJiXHBwccPnyZfD5fM5i3759Gxs3bkRwcPC/Fi8/a9Ys/PDDD+jatatars5CoRALFixAx44dMXnyZJ1Ti79r2NraYuLEiWCMIT09HampqQgLC4NMJkN6errO19nBwQEtWrRAs2bNuMCo+/fvIywsDFKpVK3Ubm9CacJeOzs7AMrMUep6XVaEb775BitXrsT169fh7e0NAFX2ok2cOBETJ07EjBkztDK0r0IgEMDU1BQCgQB8Ph8SiUST+6UW89IHZRRMTEzQs2dPlJSUcC6lERERVevCqQYcHR1x+vRpbNu2TW2XWT0+XsyYMQN16tRBYGCgTq7HHwE+PqPwIaFp06bIy8vDs2fP3rcqeuhRVdAbBT300KMc9MSteuihh+bQGwU99NCjHD5qo2BjYwOxWPzO5FtbW6NatWpaH+/u7g53d3ds27YNwcHB3DKrrqhZsyY2btyIFi1acKQdHzKEQiGEQiEEAgGqVasGJycnmJiYvG+19HgLPig/BU1gZGSE06dPY+fOndixY0eVyOTz+bC3t0fjxo1hbW2NmTNn4tGjRxg9erTGrDvOzs44cuQIAKUDFhGhefPmr7nWagoDAwP88MMP8Pb25pidfvrpJ/zxxx//+ipNRWjZsiXq1q0LW1tbtGjRAjye8vvTpEkT1K5dGyNHjsSlS5fes5bvFkZGRnB1dQWg/MAYGRlx7um5ubnIzc3FnTt3Pqj7BgBVGhClg7u0xu6ePXv2pMLCQuratatOLqhCoZD69+9PGzZsoK1bt1JoaChlZWVRbm4uXbhwgTw9PTVOGe7g4EBBQUGUn5/Plapwx61VqxZt3ryZsrKyKDU1lXPJTUpKou+++67K3H11Lc7OznT79m2SSqWUmZlJhYWFlJubSydPnqS1a9dSly5dyhGZaCLX2dmZZsyYQVu3bqWDBw+Sl5cXmZiYaBy8VlkxNDQkQ0NDnWR8+umn9OLFC3r+/DmlpqZSamoqR7Ly8uVLev78OQUEBJCLi8u/dW/evZvz+8TChQtx48YNhIaGanV8w4YNAfw/yaitrS2KiooQExOD48ePY/fu3bh9+zZkMplGcs3MzLBq1Sp06dKFo0XftWuXVkzAZdG8eXNs2rQJrVq1QlRUFNatW4devXpx59KyZUsIhUKNezTm5ubIzs6GhYUFBAIBx7wsFotRr149AEB2djYePXpUjjG5IlSrVg329vZQKBQwMzNDXl4epk6dil27dml20q/o6efnB0Dp6pudnQ0zMzP06tULmZmZUCgUOHToEJYuXaqRXAMDA7i4uKB27dpwdHSEg4MDAMDb2xs8Hg/Tp0/HgQMHtHIjtra2Rnh4OF6+fIns7GzcunWL2+fh4YFWrVqhR48e2LNnj9rM03w+n3PeMjQ0RJ06dTgGagBcj0wqleLu3bta9UI+SqPQs2dPNGnSBD4+PpDL5Rof37BhQ2zcuBEA4OnpiaKiIly5cgU7d+5EeHi41hTvJiYm2LJlC4YMGYKSkhKcO3cOADBnzhydu4hfffUVN/xYtWoVTpw4wXFXWllZVcrma2pqCgsLCwBKotLS/AvOzs6Ijo5Go0aNYGhoiMTERFhYWMDW1hZNmjQBoOzqjh07Vu2oz+joaCxYsACjRo1CixYtsG7dOp2j+BwdHbmu+IMHD7BixQqYm5uje/fuaNiwIerVq4eePXti5cqVlRpGHo+HBg0aoEePHujSpQtatWoFsViMrKwsjnZ+x44d8Pb2xogRI3D06FGtuDpPnjyJc+fOvVGfkpISdOrUCQ8fPqzweatduzYAJfGwk5MTbGxs4OTkBMYYTE1N4erqCh6PB8YYiouLOaOQmJiIESNGaOeM9b6HDpoOHywsLCgkJITOnj2rddd28+bNXHduw4YN1KpVK7K3t9eJ4kwkEtGvv/5KxcXFlJubS2fPniVLS0uytLTUudtna2tLd+7coezsbJo5c6ZWMtavX0/JycmUnJxMUqmUJBIJSSQSUigUpFAoKD09nTIyMig9PZ2Sk5MpLi6Oi0qUy+U0cuRIjdqbPWC0oOIAACAASURBVHs2ZWVlkUQioS5dupCdnR0ZGBhofQ28vb0pJiaGYmJiqEuXLtx2Ho9Ho0aNIqlUShcvXlSLGGf69OkUHR1NT58+pYiICAoMDCRPT0+qWbMmGRgYkIGBAbm6ulJCQgJt2LChSrruQqGQPD096fLly3T58mVKTk6mkJCQCklmjIyMaPfu3bR7927KyckhIqLCwkJKSUmhxMREiouLo+joaLpz5w5dunSJnj17xvGBxMfHU/fu3V+V+b85fJgyZQo8PDzQuXNnjY91dHTEkiVLMGzYMC6jUFFREdq0aYMGDRrA1tYWq1evxpMnTzT+MgwePJhj6g0NDcWoUaNey0ng6OiIgoICjaMc5XI5MjIywOfz4ePjg6ioKI25Gs6dO8etpERFRSE8PBwlJSVwcnKCSCTCo0ePwOfzuQCpMWPGYMyYMQCAgwcPapz23s3NDWZmZigoKMC5c+fw8OFDREdHY9++fVwPRxMEBQVxw5ey8SglJSVo06YNSkpKcPjw4UqjWhljMDIyws6dO7F9+/a3Ri0yxjjqdG1ha2uLXr16wcDAAF27dkXHjh057odJkybh5s2bFT4LPB6PW10rLCzEzz//jISEBDx58gS5ubmQyWTg8/mIi4tDeno69u7di0GDBgFQ3m+to4nfdy9Bk55C/fr1SSqVkr+/v1bWesuWLVRYWEhyuZybACwuLiaFQkEpKSmchZ09e7ZGcgUCAT158oTy8vIoPT2dPv/883L77e3taenSpXT37l26desWTZs2jcRisUZtuLq60tatWykxMZFSU1PJ19eXzM3NydzcvEq+ZGVLx44dOVo6uVxOo0aN0lhGy5YtacmSJbRs2TJKSEggiURCBQUF9PLlS1q2bFmVEc96e3tTTk4OHTt2TO1rYWhoyPFIvK1069aNsrOzadGiRVrrtm/fPiopKaHi4v9j77rDoji779ll6U1AQAFBihI1Kkqwx4IaS4yRGLsxUSNJDCbRGBWNwcSGGlHjZ6xgi8aOHRvWWKJiQ4KISO+9LruwnN8fyPwkom4z+ZKP8zz3YZmdufvu7Mydt9x7joIVFRWUyWQcP348x48fr9TxYrGYLVu2ZMuWLTlo0KAXnrP+/fszIyODSUlJTEpKYs+ePeva76+nY3uVQUEkEnHbtm2MiopSmzdw/PjxTElJYWZmJq9du8Zr165x7dq19Pf3p7e3N3v06MGffvqJ9+/fV4mXz9nZmXl5eSwvL+eGDRuE9k6YMIETJkyoRdFWWVnJxMREduzYUa3v4Onpybt377K8vJwHDhzggQMHlKY1V8ZcXFx47949yuVy7tmzh3v27FE5gP3ZDA0Nqaury5kzZ7KsrIwxMTEaDSVqzNramufPn2dJSclLWbpVNW9vb6anpyt9A9dln376KZcuXcrFixdz4cKFXLduncDbOWrUKFpaWmqtvQsXLqRMJuOaNWu4Zs2a5+337woKPj4+rKio0PjH79evHwcNGkRbW1va2to+8765uTmjo6O5aNEipX2OHz+excXFLC8v59SpU2lubs7Vq1ezoqJC6JnExMSwqKiIZWVlTElJ4Ztvvqn2d2jTpg3DwsKYm5vL3Nxczpo1Sys3GQB+9tlnLCwspFwuZ9++fdm3b1+tXbiGhobcv38/Hz58qJX2fv3111QoFAwNDVWbQv959sknnzA2NlYjlqS67IMPPuAHH3zABw8ePNOjVNe6devGpKQkZmRksF+/fuzXr9/z9v33BAULCwveuHGD4eHhajEsq2I2NjaMiYnh+fPnaWxsrNQxnp6eLCwsZGlpKaOjo3n27FnKZDKBuffcuXM8deoUS0tLSZKHDx9WegJy2LBh3Lp16zMUZzVd/OTkZMbFxWkkXFNjAwcOZHJyMmUyGVevXi3oLLzsOD09PU6cOPGl7M8SiYQHDx7ko0ePNA4KdnZ2fPToETMzM1WeBFXGFi9ezKNHj2rdbw0F3i+//KKVduvq6nLdunWsqKjg7t27X5Zf8WonGkUikTuqpeFq4ALgOwANAEwCkP1k+2ySx9X9nHrUox5/MbT0pNdBteiLE57QvGuzp7B8+XLKZDJ26dJFo6gqEoleOhnVuHFj3rt3j4WFhXR2dlbKr0Qi4fXr1ymVSllaWsqysjIWFxcLGYc1w4bCwkIePnxYack3PT093rp1i6mpqezUqRN1dHSop6fH1q1bc/HixUxPT2d6ejoTEhLo4eGh0bmpeYqT5Llz51Qa70okEp47d47ff/89v//++zr3MTMzo7e3N+Pj43njxg2Vs0T//DuuXbuWCoVCWWpzpa2md3To0CFu27ZN6eNsbW2V6lV5eHjQw8ODW7du1fh6Bqr1OpKSkpidnc0RI0a8bP+/bviAamr3y09ez4MWg0Lv3r0plUr53XffaXwCdXR02LlzZw4dOpT9+/dn//79aWNjQ7FYTDMzM7799ts8cOAAKyoquGDBApV89+3bl2lpaUJKb3FxsWAymYzp6emcPHkyTUxMlPapq6vL8PBw5ubm8vfff+fx48d59OhR3rt3j6WlpUJQmDJlikp+67IvvviCpaWlfPDgATt06KDSsWKxmKGhoUxNTWVqair9/f3ZrFkzWlhY0MjIiO3bt+fu3buZlpbGpKQkDh48WKO2+vr6CroPnTt31vi6eNpqBGpiY2NVCgrBwcFctmzZc38HkUjEzp07C5O3mp6DGtu0aRPLy8u5f/9+ZfQp/tKgEALA76mgkADg3pPtFs85xhfAzSf2zBeoGZ9GRESoJczyPHN3d+eZM2cEnceEhASeP3+eMTExrKioYFFREVeuXKmWzt8777zDS5cuMTo6utay5+7du+np6alWe93c3DhjxgzevHmTmZmZJMmUlBSeOHGCM2fO5MyZM1UWK/mz2dvb89atWywsLFS7hsLd3Z2//fYbf/vtN8pkMpaWlvLq1as8ffo0CwsLSZKZmZl1CrAoax07dmTHjh0ZExPD/Pz8l4oFq2MtWrRgixYtmJiYqNLS97Zt21hSUsKLFy9y9OjRbNu2LT08PNi7d29+/vnn3LFjB2NiYjh16lSt0dIPGzaMmZmZTElJUfZc/GUCs3oAclCtHwlU60bqoLoseyGAEHV6Ct988w2/+eYbymQydu/eXas/fLdu3YSIHRMTw8rKSj5+/Jhbt27V+EKztLRk69atOWrUKI4dO5Zjx47V+CkOVC8VvvXWW/T19WXv3r21ugw5bNgwFhYWct26dRrlDzg6OtLR0ZFz5szh8ePHWVFRQblczmvXrjE4OLiuDDulzdrammfOnOGZM2dYVlbGpUuXar0ICgDbt2/P9u3bMzExsVbm5MusRYsWPHDgAEmysLBQ6MUVFBTw0aNH3LZtGwcOHKi1drq4uPDixYssKSnhwoULlT3uLwsK7wI49Zz3mgK4r05QaNSoERs1asSmTZtqVWG5xmpmaRs1akRXV1c2btxYo3HuP9VMTEwYGhrKu3fv0sHBQat+XV1d6eLiQisrK41WjUQiEQMDA1mD8PBw2tvba/1cGBkZMSgoiEFBQbx06ZLSq081ZmVlxR49erBXr1708/PjDz/8wB9++IE9e/bUuDf3Z5sxYwbLysqYmpr6oiXIP9tfFhR2ARj/1P+Nn3o9FcAudYJCvf019tZbb7GsrEylvIy/w3x8fLhw4UIuXLhQbQn3l9nQoUOFwDNv3rxX8jDSln3//feUyWS8e/euKnNAf5mWZF8Anzy1ealIJPJ40oiEP71Xj/8yODo64s6dOxqVNf8VCA0NRWho6Cv9jLNnz6J79+4AgDt37tQ8sP5roVAoEBISgoiICK36rWdzrkc9/ndQz+Zcj3rUQ3X8I4OCiYkJ7OzsXpn/r776ChEREZg5c6ZW/JmamsLJyQk6Ojpa8VePZ2FpaSkwRf2vQFdXF1ZWVtq/rjSdaNSGQcVJllWrVjExMfGlkueqmp6eHr/88ksWFRUxLCyMgwYN0shfr1692KtXLx4/fpy5ubn09/fXWuFSvf2/icVirl+/nuXl5WzSpMnf3p6/yqZOncpLly6xXbt2yh6j1ETjP7KnUI961OPV4R/HvNSuXTsMGzYMhoaGWh9C+Pj44Pvvv0dQUBCWLFki0HGriqZNm2L27Nno27cvgGoGnqqqKkyYMAGHDh3CH3/8oZX21nQbJRKJ0qSqNXjzzTcxc+ZM5Obm4uTJk5BKpSguLoZcLoeFhQUuX778XFYiddqpr68PhUIBmUwGXV1dVFZWam1238nJCe+++y6OHj2qsgq1l5cXvv76a8TFxSE/Px/Z2dV1fNHR0cL51dHRQatWrdCwYUPI5XKcOXMGt2/f1rjdbdu2xbRp07BlyxacO3dOpWNNTEwwePBgtG7dGjY2Nhq35Wn844KCoaEhGjRogOjoaPz2229a8ztkyBD8+OOP+OOPPzBv3jyNfI0dOxaTJk0SgopcLodEIkFiYiLy8vLU8mlgYAA7OzsoFAqYmpqia9euGDhwIIBqXYmdO3di2bJlSrM5KxQKeHl5wcbGBoMHD0ZlZSUqKiqgUChgZGSE33//HV9++SUAKM00XBcGDhwIPz8/dOnSBSkpKQgODsbw4cNx8uRJjc+zSCSCnZ0djh49ioyMDEybNg2FhYUq+YiNjUVubi58fHzg4OAAPT09ANU0bzVBLCcnB3K5HElJSTh16tRLKdrs7Ozg5OSE5OTkWg8WPT092NraAgC6du2KCRMmoH379khMTFQ5KOjo6EAikaCyslIVKXql8I8LCjY2NtDR0cGjR4/UvsH+DE9PTyxfvhwNGzbEjz/+qLE/U1NTVFVVCZTeZmZmaNu2LR48eCBwQyoLExMTeHh44MMPP4SXlxeqqqpgYWGBpk2bIjMzE0B1oPTz88P27duRnJyslN/r169j7ty5aN++Pdzd3dGsWTOIxWIYGRmhqqoKffv2FdSn1AkKr732GkaOHIlx48bBwsICmZmZsLOzQ0BAAIyNjWvRnasLOzs7bN26FU5OTpg8eTKSkpJUOl5XVxctWrTA999/j5UrV8LDw0Og/jcyMkJhYSEeP36MlJQU5OXlISEhQSnuTj8/P3z88ceIiYlBcXExRCIRSMLQ0BBNmzYFAIEv8+rVq2oxXSsUCiEYaDut4B8VFHR1dTF+/HhIJBKtJWyIxWKMHj0atra22Lt3L4KDgzX2uXr1asTHx+Py5csAgPnz58PDw0O4iZVFv379EBgYCAcHBxgbG6OsrAwnT57E/fv38eDBA+GpGBwcjOzsbJW6zpWVldiwYQN0dHRgZmYGQ0NDBAYGYvDgwQJh6ePHj1Vqr4mJCQBg1qxZGD16NMzMzHD//n3s2rULnp6esLGxgUQiQXp6Onbs2KGS77owc+ZM9OrVC6dPn8b27dtVPn7evHmYOHEipk2bhp07d2rUI3oasbGxyM7ORpMmTSCVSiESiSCRSCCXy4XhiUgkgp6eHr766iuBVl4VuLu7w87OrjoDUaLl2/jvXnlQZfWhS5cuLCgoYEpKCrt27aqVGdz33ntPoCL38/PT2sywiYkJw8LCGBYWxtLS0heRab7w++7Zs4dBQUH09fVlt27datVn1JR/V1ZWalxa3rx5c6anpws073PnzqVYLFYpZ9/f35/+/v4sLi6mXC7nw4cPefbsWRYVFVEqlbKgoIByuZw//PCDxue3a9euLCwsZExMjFqUaR06dGBycjLDw8O1TrkGVBeHtW/fng0bNqSNjQ2dnZ1pZmYmFMllZmYyLCzspQSydZlYLGZQUBClUimLiopUqTz9d1G8SyQSjB07Fubm5jhy5IjaylA1qBHNGDBgACQSCXR0dDB79my0aNECn3/+ucbt/eSTT4SU2YqKCojFYmG8qiyuXLmCmzdvQi6X1/l+//79AQAREREapym/+eab0NPTg0gkQlZWFvbv36/SWFVPTw9vvfUWAKCwsBAVFRUwNDSEjY0N1q5di4YNG2L48OGIi4vDwYMHNWorUE31b2pqik2bNiE3N1fl4wcOHAgHBwfEx8ejT58+OHbsGIqLizVuVw2SkpKeGc7o6upi3LhxAKrniDZu3KiWSJCFhQW6dOkCktDX14e1tbVW2lyDf0xQ6NGjB3x8fJCZmYmQkBCNFZc6duwIABg+fDjy8vIQHx+PBg0aYNSoUTh69CjCwsJU9lnTJZTJZCgsLBRmrysqKmBhYQEvLy+cOnVKJZ/PCwgDBw6Ej48PACAoKAgpKSkqt7cGFhYW6N27N3R0dCAWi3HhwgWVV0jkcrkQpJ6WMauoqICpqSm2bt0KY2NjhISE1NJtUAfvv/8++vfvj5iYGGzatEmtMXVISAiaNGmC1q1bY+PGjYiJicG+fftw9uxZABB0MVSBhYUFHBwcoKurC3t7e+jr6wtPXz09PXTv3h1t2rQBUD0noI66WQ0UCgUqKioERW+t4u8eOigzfDA1NeXZs2dZVVXFLVu2aKV7N2vWLM6aNYulpaXcsWMHTUxMGBAQwOLiYg4dOlRlf8bGxpw3bx5XrVr1zGekpqayoKCAH3/8sVba7urqyitXrnDHjh3csWOHxmW5fn5+zMnJYUFBAePj4zWmdvuzDRkyhAqFghEREXR1dVXbj6GhIQ0NDXn06FFWVFRw0aJFGn93Jycnjh8/nufPn6dUKhWUw1asWMGmTZsq7cfAwIBr165lYmIiExISBMatGvLe0tJS5ufnMy8vj3l5eczJyVGl5LmW6evrs0+fPty1axcVCoUqw7F/z/ChZcuW8PLyQnp6OpYvX64VnzU5BCKRCOnp6Wjbti2GDx+O5ORktbq3Y8eOxZw5c/Dw4UNh2y+//AIA6NWrFzp06KCyMtTz0KtXL7i5uQldUU2WpFq1aoWJEydCIpGAJAoLCzXqdfwZxsbGmDRpEsRiMQ4ePIi4uDi1fdX0jHr16oVTp05hwYIFGi/HJSYmYvPmzdi3bx969+6NMWPGAAA+/PBDuLq64qOPPlJqlYskHjx4gOLiYmRlZaGiogKlpaUwNTWFjo4ORo8eDRcXF6GXtH37dvz+++9qtVkmk+HMmTPo0qULxGKxMBTWGv7uXsLLegp6eno8cuQIFQoF9+zZo7Wn182bN3nz5k2WlpZy3bp1TE9PZ2lp6YuENF5oa9eupVQqZWJiImfMmMGAgABGR0czOjqaZWVljIiIoIuLi8btbtu2LZOTk7XCWQmAK1asYFFREbOzs1lcXMydO3fS0NBQa+d57NixLC8vZ0REBB0dHdX2Y2dnx4cPH/Lhw4eUy+Va+/7Ps8WLF1Mmk/Grr77Sir+NGzcyOTmZPXr0YI8ePbTi84cffmBFRQVDQ0OVJYT5d/QUnJyc0KZNG6SmpgpS5NpAzZxEZWUlevbsCRMTEyQkJCAwMFAtfzVJQ2ZmZpg/fz5ICp+RlJSEzz//XOUlvj9DV1cX3333HTIzM7F+/XqNfAGAtbU1unXrBoVCAYlEgsePH+PHH39UO5Pzz2jevDm++uorAMCuXbtUziN4Gj4+PoIEO0mtLR8+D+Xl5aiqqtLK5KOlpSU6d+6MmJgYrXIfKBQKKBQK2Nvbw9DQUKM5iqdRX/tQj3rUozb+7qHDy4YPgwYNYmFhIY8ePapVstKJEydy4sSJJMnHjx9zx44d7NSpk9r+vLy8mJGRIci6l5SUsLCwkIWFhXzvvfe00ubJkyczIyODI0eO1Iq/rl27Mj4+nnl5eSwpKeHhw4e1dn51dHS4ZMkSVlZW8uTJkxpxNDZv3pxRUVGswcqVK7XOeVhj3t7e9Pb2ZnR0NG/fvq2VId/06dNZVlbGzZs3q5z78SLr0qULCwsLef36dWVzLf4dw4f79+9DLpcjPj5eq+vINdLqenp6uHv3Lq5cuaKRvxs3bsDPzw8TJ05Er169cOPGDWGiURs0Yh4eHpg2bRr279+PvXv3auwPqO4i1wxxSEJHR0dIydUUtra2mDBhAjIzM/HTTz+9VCL+RTAwMICZmRl++uknANDKBKOzszMkEgmys7NRVVUFZ2dn+Pr6YtCgQQCqz82sWbM0HvIBgLm5OaqqqtCkSRP07t0bAHD69GmN/WZnZ6O8vBwFBQUqF8S9EEo+yUMAZOEpZmYAlgBOA4h98tfiyXYRgJ8APEK19kN7TXoKOjo6bNq0Ka2srF7Jk0HbZmpqSldXV6W1IpW1bdu28dq1ayotk73MRCIRt2/fTplMxsrKSm7atElrvj/66COWl5dzx44dWmln48aNKZFItKYlOmnSJF64cIERERG8fv0679y5w8TERIaEhDAkJIStWrXS2rnw8/NjdnY2KyoquHnzZm7evFkrfsViMR0dHWljY6PsMUr1FJTiaBSJRN0BlADYRvL1J9uWAsgjGSgSiWahOijMFIlEAwFMATAQQEcAq0h2fIn/lzfifxyXLl1CbGwsJkyYoFW/9vb2mDRpEiQSCdasWYP09HSNfRoZGeHYsWNo2bIl3n//fVy6dEkLLdU+mjZtivbt26OyshK5ublIS0sTKiC1WXloYGCAYcOGwdHRUVjujoqK0pp/FaAUR6Mq4/6mqN1TiMETOncAjQHEPHm9HsCouvZTp6dQb9Xm6enJli1b/u3tUNaaN29OT0/PVzb2rze17JXPKdiSrHmsZKBaGQoA7AE8Xb+b8mSb5o+g/2Fom8b7VePpJK56/LOglSVJVj/uqcoxIpHIVyQS3RSJRDfV+UwnJyfMnz8fLVu2VOfwetSjHs+BJkEhUyQSNQaAJ3+znmxPBdDkqf0cnmyrBZIbSL6h1BjnT2jQoAG2bt2Kdu3aqcy08zLo6urCzc0NRkZGah1fw7qjakWkup9lZ2eHU6dO4c0339TYn6mpKVxcXODs7AxDQ0MttLAaurq6cHJygqOjo9Z8vgrUfH8XFxe4urrC2Nj4727S3wMN5hSWAZj15PUsAEufvH4bQBiqVyE6AbiuyepDXbZo0SIePnyYBgYGWh1zmZqa8ttvv6VCoeDPP/9MOzs7lY5v3rw579+/z/j4eLZp0+aVjg/79+/PCxcu8MKFC7x58yadnZ3V9mVjY8M+ffrw2LFjlMlklEql/PHHH4UCJE3b+tprrzEhIYHR0dF/5fhZJTMwMODGjRspl8spl8spk8l46NAh+vv7a3Ru/woTi8X09fXl6tWrOXr0aI4ePfp5+2pPSxLAr6ieE6hA9RzBRABWAMJRvSR5BoDlU0uSawDEAYgE8IY2g0Lr1q155coVtm/fXqsn1srKinPmzGFSUhILCwuZn5/PjRs3qqQn2KtXLyYlJTEmJoatW7eu9V6nTp201ub33nuPOTk5/OWXX/jLL7+oLbbasmVLBgUF8cqVK0xLS2NWVha3bdvGY8eOMT09nf369VO7ku9p69KlC1NTU3nv3r1XdmMYGBhopP3Ys2dPJiYmsqSkhCUlJSwtLWV5eTnLysq4b98+jYhYrK2tOWHCBIaEhGjlnOro6LBBgwY0MTHh66+/zg0bNjAzM5MkmZmZyczMTA4ZMqSuY/8agVltmLInQywWc9++fZw5c6bWLiZ7e3uOHz+e9+/fZ05ODpOTk7ljxw4mJyfz5MmTKjHj9OjRg4mJic8EBXd3d0ZGRjIsLIzW1tZqt1UkEnHq1KnMyMjgjz/+KChnq+OrSZMmvHTpEouKihgREcHFixezdevWNDExobu7O1NTU3nkyBEeOXJELXagp23dunWsrKzkrVu3NPJT831dXFzYokULduzYkX369OHq1auZmZnJzp07q+377bffZkJCAnNycpiTk8Ndu3bx9u3bLCoqokwm4+rVq9XyO3ToUCYlJTElJYWRkZFMSUlhSkqK2lmpjRo14tq1a5mbm8uMjAzm5ORQKpWyuLiYRUVFQqn28ePH6ypAq9d9qEc96qE6/uvTnJ9Gnz59YGxsjI0bN2rsq0OHDgCAZcuWoX379hCJRKisrERZWRl27tyJZs2a1WIQUgZVVVU1PZ9aMDY2hqGhIdq1awdnZ2eBvFNV+Pr6IiAgAIsXL8aSJUvU8lGD7OxsgZvi5s2bSE9Ph66uLsrLy1FaWgq5XC6kJtf1nZRFTeJOZWWl2nX/DRo0wPjx49GvXz8AgIuLizAJKJVK0bBhQ5ibm6vNQNSpUyfMnz8fTk5OOH/+PABg5MiRmDx5MpYsWYLKykq4ubnB3NxcpYntPn36ICgoCBEREVi5ciXi4uLw9ddfAwDGjx+P8PBwla6Fvn37Ys6cOfD09IRIJIK+vr5wTmv4MGp+M29vb0ycOBEBAQFK+6/BPyYo6OjoYMKECTh48KDG1O5eXl5Ys2YNgGqSkZs3b6JBgwZwc3NDbm4u5HI5dHV1Vb4Z2rZtCxMTE5SXl9c6tqZbpglH/7hx47B48WJs3rwZS5cuVcvH0ygvL3+GTEahUEBPTw+LFi1Co0aNhPc1ye4zNDQULlR1S3vd3Nwwbdo0JCQkAAA2bNiAqKgoPHr0CNnZ2fjxxx/RunVrtViRJRIJJk6ciJYtWyI2Nhbz588X3gsODoaHhwfGjx+PNm3a4K233lK67sTR0RGrVq3CiRMnMH36dKFu5+bN6hX4QYMGwcrKSumgMH36dEyfPh0NGjRAZWUl9PX1UVlZiQcPHuDKlSuoqqqCmZkZevToAaBaCmHy5MkIDg5WvWT9755PUHZOYcCAAbxx44bKKwJ/tk6dOjEiIkIYex09epS2trb09/dnZGQku3fvThMTEx4/fpwnTpxQaTy9Zs0aVlRU8MCBAzQ3Nxe2t2vXjrGxsUxJSVFL/9LS0pKPHz/myZMn66wUNTU1paOjo9rzFc7OzhwwYAAHDhzIn3/+mYWFhQwODqa5uXmt76GOTZkyRaAmmzZtmlo+xGIx9fT0nvt+eHg4Q0JCVPZrZGTEhQsXsqioiCkpKXz33Xef2cfX15cFBQUsLy/nnj17lF6NWb58Oa9evVqrZqdt27YMDw9neHg4QXDkngAAIABJREFUz507p9TkpYmJCU1MTLh7927KZDKWlZUxLy+PoaGhnDt3Ll977bVa+wcFBTEoKIhlZWWUyWR/pr/7d1RJAtWVjBMnTkRoaCjS0tIECipVK+9EIhEWLVoENzc3oaewbNkyZGZm4syZM7Uk3SIjI9G2bVuV/FdVVUEikSA+Pl6r+RMTJ06Eo6MjNm7ciOLiYkgkEgwdOhTDhg0DUE0Yqq+vD5lMhmvXriEwMFClitKJEyfiyy+/hEwmg5mZGS5fvoypU6eqLMH2ZzRq1Ajvvfce9PT0kJGRobaiV1VV1XMJbHv27Ak3Nze1CHjatWuHTz/9FPr6+igoKKizHuG3335DQkICWrVqpXTPsUmTJujRowd+/fVXgWnazMwMAQEB8Pb2BlDNRq2MLJ+fnx+AaqLeyspKxMXFYffu3QgKCnqGDEdXVxf6+vrC/88bzr4M/4ig0LlzZ9jY2GDLli1o0qQJVq9ejdjYWHzzzTcq+enTpw/at2+POXPm4D//+U+t927cuFHrf5FIpNKcgkgkgoGBATIyMgRG4Lr2UQd2dna1uCM/+OADhISECOXev/zyC1JSUrBnzx7cuHFDqYBQIzkGVJfx1gx7evTogWbNmuHdd98VSr/VnVN4++230aNHD8jlckRFRSE6OlotPy/CsGHDQFLlgGNsbIxvv/0WxsbGOHv2LFatWoVHjx49s19RURHkcjnEYjGkUqlSQ6kahuX4+HghCWrGjBkYPHiwMPxQhge0cePG+PjjjwEA+vr6SE5OxtSpUxEeHl7n/s7OzujatSuA6mtNLBard8393UMHZYYPa9as4cyZM2lubs4DBw6QJL/55huVuopt27bl7du3qVAoOGnSpBfua2pqymPHjvH06dNKDx/c3Nx4+/ZthoWFPVPe26ZNG8bGxjIhIUHlpKYGDRrw2LFjXLJkCQHws88+Y2pqKjdv3swGDRqwQYMGBMBPP/2UWVlZ7N+//3N9NW7cmI0bN+Z3333Hbdu21blv06ZNGRUVxcuXL9PW1pa2trZqdfmBajZrhUJBkvz666/V9vM8a9GiBTMyMujv76/ysd7e3kxKSqJUKuVHH3300v3u3Lmj9JKnjo4ON2zYwNzcXMbExDAlJYXl5eV89OgR3dzc6ObmppQfV1dXIZlKKpXyzp07L0yk6tKlCx89esRHjx5RJpMxMTHxz8Ptf0+eQnBwMMeOHcutW7dSoVDw7t27dHJyUukiGDRoEHNzc3nv3r2X1srPmjWLUqmUP/30k9L+W7ZsyQcPHvDgwYO0tLRk48aN2bBhQzZs2JB9+/ZlQkICr127pnICU6dOnZibm0sPDw/OmDGDKSkpAlW8nZ0d7ezsuGXLFiYlJb2UQt7Hx4c+Pj5CclZERAQ7duz4zH41iTwDBgzggAED1LphBw8ezLi4OEqlUiYkJNT5OZra1q1bGRYWpixpqWDGxsbct28f5XI5r169+lyOii5duvCPP/5gZWUlg4ODVfoMc3NzDhs2jFOnTuWiRYt47949bt26VSVOCFdXV1ZUVLCiooIlJSVMS0vj1q1bOXToUHbq1IkmJia0t7dnt27d6O3tzaVLlwpzZVKplIGBgX9O6Pp3BAU9PT2ePXuW165dY3l5OVNSUujt7a3yBTRo0CDm5OTw8ePHz71AnZ2duWjRIubn53P79u3CU1gZq5kMys7OZlRUFKOjo/ngwQM+ePCA8fHxzMzMZG5uLmNjY1XKQOzTpw8rKio4YcIE5uXl0dfXl25ubvz+++8F/zt37mS3bt1e6qt9+/Zs37494+LieOPGDfbr16/OiUtzc3PGxsYyMDCQgYGBat2wkZGRQsr08ePHVTqXytjQoUOZnJzMN998U+VjGzRowAsXLrCiooInT54UJjEtLS3ZvXt3du/enXPnzmVMTAxlMhmjo6PZu3dvtdsqEokYERFBHx8flY4zNzfnwYMHefDgQSHTsqysjEVFRczJyeHp06eZkZHBgoICFhQUsLi4mFVVVayqquKRI0doaWn5Z5//jolGuVyOkpIS9OzZE2VlZfj1119Vlu0GqkktIiMj0bNnT8ybN09Yj75+/TpatWqFxo0bo1u3bmjevDkCAwOxadMmFBQUKO2/SZMmaNu2LYyMjCCRSJCWloasrOoasTt37mD8+PFISEjApk2bkJ+fr7TfpKQkxMTE4JtvvkFxcTEGDRoEPz8/5ObmCmvQBw4cUEqCvkZzYP369QgICMDHH3+M9PR0PHjwQJjIk0gk+OCDD2Bra6u0gvWf0bBhQ+jq6goThDNnzlTpXL4M1tbWmDdvHkJDQ9UmcKmZM3J1dcV//vMfSKVSODs7o3Xr1gCqGZgNDAwQFxeH6dOnP3ccrwy8vLygq6tb55zFi1BYWIjJkycDANasWYM+ffpAJBIJcxY9e/aEQqEAWU2ll5SUhF27dgEATpw4of7S/d/dS1Bm+ODh4cHFixdz9OjRGpF2NGvWjKGhoczOzhYib2ZmpvD60KFDKovA1pi5uTkDAwO5fv16tmnTptZchLm5OS9cuKA2DdeYMWOYn5/P9PR0btq0id7e3hqdBz09PS5fvpwxMTHMzs7m1atX+dVXX/Gjjz7i7t27WVZWxvDwcDZt2lQt+rdRo0YxLy+PUqmUly9fVrudz7Off/6Zjx49UrseQU9PjwEBAXz8+LFQ51BaWsqSkhJWVlaysrKSxcXFDA4OZpcuXTRub1BQEENDQzXyIZFIOHDgQC5btoy//vorY2Ji+Mcff3DLli0MDAzksmXL2KdPn5f5+XcMH7RtBgYG9PPzY2xsLGNjY5mbm8t9+/ZxypQpGk2qvcy2b9+ukdqyJuP7ukwsFrNVq1b87LPPeObMGZaUlJAkMzIyuHz5co3k3T744APK5XJeu3aNzZo101qbvby86OXlxczMTI4YMUIjX/r6+uzZsydv3bpFmUzG0tJSFhUVCRyNX3zxBc3MzDRus4WFBa9evcoFCxZo7TyYmpqyXbt2bNu2LfX19VU5tr72oR71qIfqUIq49ZU34m8gbm3cuDGA6vXfrKwslJWVvdLPs7a2hlQqRUlJySv9HHVgZmYGCwsLSCQSyGQypKWlaZTabGJiAltbWxQUFKglE18XLCwshPyPyMhIQUdTU1hbW8PMzEx4StboaCozR6MMnJyccPjwYezcuVPjehUtQLvErf+W4UO9/TPtp59+EpbntK2K/SrNwsKCkyZNYvPmzf/2tkCbFO+vGvUU7/V4ESQSCdq3by9Q5NWsHNVDZSjVU6gPCvWox/8OlAoK//V5Cv/r6NOnD/r16weSSE1NRUhIiFbl8+pC06ZNMXToUOzevRsAhHF2Pf5H8HfPJ9TPKbzYAgICSJJlZWUsLCzkjh07tCJ6+jzT09Pjvn37eO3aNdrb26vN//hXmL29Pffv38/Bgwe/ss/Q0dGhubk5LSwsVF3++2807WQ0ikSiEACDAGTx/yXjlgF4B4Ac1QSt40kWiESipgCiUa0KBQDXSH76ss/4N+KNN95Au3btAABisRiZmZmIiopCbGysSn5SUlJQXFwMkqioqMDbb7+NgoICLF++XCvip0/DxsYGQUFB8PLywogRI5Ca+gwz/38NjIyMsHbtWrRt2xZz587Vml9LS0sA1SXZHh4esLW1hbu7O3R0dHD37l3MmjVLqRUkAwMDtG7dGg8fPhTK6Fu2bClQ8aekpOD48eM1D0WV0bVrV3h4eCAiIgLXrl1Ty8dzocRTvDuA9qhN7/4WAMmT10sALHnyuunT+2mzp2BpaclVq1Zxz549Got/6urqcty4cRw3bhyDgoJ48uRJrl69mkOGDNEKpfmkSZP46NEjoTglOTmZsbGxvH37tspJN+bm5uzUqROXL1/OgoIC5uTksKioiIGBgVqVNTc0NGR4eDhzcnL4zjvvKH2cSCSimZkZLS0taWVlRSsrK1paWtLExERjwtcX2ZdffkmSDAgI0JpPNzc3Hj9+nMePH2dOTg7Lysp47do1rl69mlFRUTx37lxd9QR12uzZs5mZmcmVK1eyVatWXLFiBe/cucOMjAyBcHXlypVqEeN07dqV0dHRJMlbt25x5syZ/PDDD19IRPPEtErx3hTPudkB+ADY8SqDgqOjI0+dOsX79+8zNjaWO3fupI6ODiUSCW1sbFSm9p40aRIVCgUVCgXT09N58eJF3rx5k+np6bx165ZSxUXPMxcXFz548IBxcXEcM2YMx4wZQxsbG7Zp04ZpaWlcunSpWn719PS4evVqZmVlsbi4mOvXr9eaCrOFhQX37dvHoqIilbvi1tbWPHfuHLOzs/n48WM+fvyYKSkpTEhI4MWLF7ls2TJ6e3vXWXilrtUUh23evFlrXXpPT09ev35dWPY8evQoBwwYIDwkLC0tlc5wHDBgAJOTkxkSEsK4uDjGx8dz3bp17Nu3L83MzGhmZsaxY8cyKSlJZWZykUjE4OBgIS07Pz9fsAULFlBHR+dFwfgvCwpHAIx9ar9SALcBXADw5gt8+gK4+cReeCL27dvHM2fO0MrKikOGDBGEYD7++GNu2LBB5RujU6dOTE1NZWpqKn///Xc6OTnRyMiIXbt2ZUBAAE+ePKlMHnmd1rNnT5aXl3PRokW1to8cOZKPHz/mpUuX1BaK7dy5Mx89esT8/HxGRUUJ/Aia3Ax2dnY8cuQIZTIZJ06cWOs9PT29lz59xGIx3dzc6O7uLsxBODs7s1+/fvziiy+4bt063r9/n9evX2eLFi00amvXrl3ZtWtXJiYmMiIiQhUJ9heaoaEhDx06RJlMxtDQUIaGhtaiUVPFxGIx9+zZw02bNhGors795JNP6tx3+/bt3LNnj0r+vb29mZ6eztjYWK5YsYKZmZnMy8tjfn4+4+Li2K1btxc91F59UAAwB0Ao/n9pUx+A1ZPXnqgWmjXTpKfwzjvvMCYmRuCia9OmDRctWsR58+YJE2/q/Hjt2rVju3bteOHCBf7+++90dXWlSCSira0t+/Xrx2HDhqnl97XXXmNSUhI3btwobPPw8GBUVJSgKzF06FC1b4pHjx4xNzeXJSUldHV11ahGwcDAgFu3bmVZWRknT54sbLe3t+eGDRsYHx/P+Ph4btq0iSYmJmp/TuPGjRkYGMiUlBR6eXmp5aNNmzb87bff+NtvvzEhIYEDBw58Zh87Ozu6uroq042uZT4+PiwqKmJSUhJHjRrFUaNGcfDgwfT09FTruxYWFnLu3Lkv3M/BwUEoRFPFf0BAAKVSKa9cuUJXV1du2LCBycnJzM3NZVFREUeOHPkiTYlXW/sgEok+QvUE5BjW3NmkjGTuk9cRqJ6EbK7uZ9SjHvX4G6BOTwFAfwB/ALD+037WAHSevHZBtbCspSY9ha1bt9Yi+nB1deXevXv5+PFjknwp29DLbOTIkSwvL+eDBw946NAhJiUlMSEhgffu3aO/v7/KY2F9fX1u2bKFsbGxAhuyn58f8/PzmZWVxc8//1zttn700UdMS0tjbm4u09PT6ejoWJcKkFKmp6fH//znP8zKyuKHH34obO/cubNAK7do0SIuWrSIcXFx9PX11eg8A9UlxA8ePFC55Pn1118XKMZkMtkzT9dmzZrxhx9+4Pnz55menq7SNWFtbc0LFy6wpKSEhYWFLCoqYlFREcnqitHnyK8915o0acLc3NzntsHY2JjGxsYMDg5mWloa3d3dlfZtb2/P8PBwlpWV8erVq8Jvv379espkMh47dkxg43qOD60tSf4KoCeAhiKRKAVAAAB/VA8VTj8hhqxZeuwO4AeRSFQBoArApyTVYnqQSKqbZmpqip07dwIAzM3NBbKRGzduoLS0FKdOnVLHPV5//XUAwIwZMyAWi+Hu7g43NzcsX74cZ86cwRtvvAFjY2M0bNhQpWQhmUyG1atXw9PTEzt27ABQLWZiaGiIsLAwhISEqNVeGxsbfPzxxzA2NkZxcTFWr16t0ZLhiBEjMH78eAQEBGDr1q0AAE9PT2zduhWnT5/G9OnTBbZgd3d3ODg4qP1ZNZg9eza6deuGwYMHq3QerKys4OjoiC1btgAANm3aVOv9jz/+GFOnTkVFRQUMDQ1ha2urtO8uXbqgVatWkMlkiIuLw+nTpwEAV65cgb+/P1asWIGcnByliWHbtm2LyspKZGRk1Pn+unXrAADDhw9HaWkp1qxZg2+//VapZUWJRAJDQ0OQ1aQqOjo66NOnD7p3747y8nKcOXMGaWlpSn7zF0CVVYJXZXhBdFy9ejUXL17Md955h1evXuXSpUsFOq3ly5er9cSysLDgpUuXeOnSJd69e5dTpkzh8uXLKZfLnyHVUFe0dMeOHcKTLSsri3v27FH7qd66dWseO3aM+fn5LCws5IkTJ55pl7GxMTt37sxevXrRyMjohf50dHR4/vx5njlzRuA3fP311/nw4UPu379fmNH39PSkp6cnExIS6tREUMd27tzJqVOnKr2/SCTimjVrmJ2dzQ4dOrBDhw613vvuu++YlpbGyMhIFhcXMyoqSiVtjQYNGnDp0qWcMGHCM1yPY8aMYWlpKQ8fPqz08uqnn37K3NzcZ+Y8XFxcuH79ekFLcubMmRw2bBivXLnCy5cvK0VXZ2hoyODgYBYXFzMvL4979uzhsWPHWFRUxNzcXGWWZ/8dJCtt2rThhQsXeP36dYHBedy4cUxNTVVbULRfv35MSkpiUlISu3fvTqBaRj4xMVErF7+bmxtPnjzJsrIylpWV8e7du2pnIU6YMIF3795lTk4Os7OzmZuby6ioKP74449cuHAhFy5cyAULFnDPnj0CH+SECRNe6NPKyoppaWlctmyZsO3QoUN8/PixkANiamrKo0ePCqbpOQGqcy7i4+NfyJ5c102blJTE4ODgZ5bbBg8ezJycHGZkZPDu3bvCRKE22gpUM4Dn5uby/v37Si99jhs3jiUlJZw+fTr19fX5+uuvc86cObx//z7Ly8s5bNiwWpPYM2fOZEZGhtIiRx999JGgjF5DwFsjipuYmMgVK1ZwxYoVz3uY/TuCQs1F3KhRIwLVs7YxMTE8f/682mv0EyZMYHR0NKOjowW2JUNDQx49elRjRWtdXV2uWrWKjx8/5uXLl3n58mWmpqY+I02vjA0cOJCpqaksKChgVlaWYDk5OSwtLRWCTg2ZZ80M9Llz517o18HBgTKZTFg28/HxYV5eHt9++20C1Wvya9asET7vRbTxqpinpyfj4uJUYmNycHBgUVERZ8+eXWt7y5YtGRUVJSguh4WFsWvXrkr7tbS0ZNOmTV+4UrFmzRrK5XKVEqQsLCx4+vRp5ufnMzIykomJiSTJ9PT0Z1a0DA0Nefr0aa5du1bpJDQ9PT3279+f169fZ0ZGBvPy8piXl8ecnBwWFBQwMjKSkZGRzwti/w7iVgC1iDp69uyJRo0awd/fX2WFqBpkZGQIwpyWlpbIzMyEVCpFbm4uhg4dqjYZhpGREYYNGwYbGxsEBwcjLCwMALBgwQLIZDKVfLm6usLPzw9GRkaQy+W1RD1IQiaTCdueBFbo6OhAX18fv//++wt9p6WlYevWrRg4cCB8fX3Ro0cPGBgYwMXFBT///DM6dOgAfX19gcjkxIkTKrX9eZg3bx4iIyNVSvV2dnaGQqGAg4ODoIj15ptvokePHnBxcUFaWhrWrVuH1atXo7y8/KX+RCIR3n33XXz99dcIDQ1FUFBQrfdrroupU6di5MiRiI+Px549e5Rub35+PsaNG4fJkydj6NChMDExwe7du7Fnzx4cOHCg1r4fffQRHB0dMWfOHKVJbeRyOU6cOIGkpCRs374dDg4OCAsLQ5cuXWBjYyOQw9RcE+rgHxEUamBoaIiZM2fi9u3bak8wAtXMPTVMS82bNxeUi8rKymrJbqmCRo0aYfHixejWrRuCgoKwe/duvPXWWwCqmY0MDAxU8teyZUt06NABenp6kEgkKC8vR2VlJUQiESoqKvDgwQOBrVcsFkNHRwfZ2dk4duwYLl++/ELfVVVVCAgIgK2tLVatWgWFQgFDQ0MEBQWhvLwc586dQ0BAAG7fvq3WuagLPj4+GDRoEPr27avScXFxcUhNTcXw4cMxdOhQANXBV0dHB7/++iuWLFmCmJiYl3j5fxgZGeHDDz/EG2+8gZMnT6Jbt25ISEiAtbU1Ro0ahVatWgEAunfvjrKyMixcuFBlZav09HTMnTsXjo6OcHBwwCeffCLUP9QEnU8++QSTJ0/GokWLcP36dZX8A9UT2iShUChgbGwMPT096OjoCKzZmjBn/e1DB2WGDzXm6+tLhUKhsjpUXTZ//nzOnz+fBw4cEBJzdu7cycjISLX8+fj4kCSvXLlCiUTCjh07cu3atVy7di3feOMN6urqquRPV1eXn332Gffu3cu9e/fS19eXXbt2Zffu3dmlSxdaWVlRJBJRJBIJY21VJ0VNTEw4evRorlu3jocPH+bXX3/Ntm3bqj25+jzr2LEjy8rK1K5TaNCgAXv27ClkTA4fPpwlJSX85Zdf1PLXuXNnXr9+XSBsLSwsZHFxMeVyOdPS0piWlsZjx45x0KBBGn3vPXv2cO/evcLQwNramtu2beO2bdv4+PFjjh07Vm3f77//PlNTU4VsxqKiIl6+fFlIynvOcf8+5qWtW7eiW7duGDhwoEpPh7rQvn17AMDJkyfxww8/YPv27di1axcaN26ssrBsjb8tW7bAzMwMYWFh6N+/v8Dz//7772tVcPafhKZNmyIsLAy3bt3Chx9+qPaQ72kYGxtj8ODBkEqlSmky/hlisRjt27fHkCFDIBaLQRJisRhZWVmIiIgAAFy8eFHjdi5btgxjx47FsWPHIJVK8cYbbyAzMxMAsGTJEly9elVt32+99RY2btwIKysrAMD9+/fx6aef4s6dOy867N/H0bh//34mJCSovbRXly1ZsoRFRUV8+PAhExISVFbxedratWvH3bt3Mzo6mvPnz6e7u7tKySn/NnN0dOSdO3d44MABrRRu/dPMysqK8+fP571793j+/Hn6+fnV0v/UxOzt7bl3717euXOHp0+fVjZ9/N/XU+jduzfeeOMNrF+/XmuKQ6ampnj33XdhYGCAyMjIl07S1UM5GBsb48iRIygvL8eYMWNUUsWqxyuDUj2Fet2HetSjHrXwj+op1OOfAz09Pbzxxhu4e/cuSktL/+7m1KMa9WzO9ahHPWqhfvigLOzs7BAQECCsSNSjHv/L+MckL4lEIlhZWcHc3BwikQhZWVkoKirS2K++vj6mTJmCWbNmQV9fX5Br/2+FSCSChYUFjIyMkJ6eDgBQKBR/c6vq8W/CPyIoeHt7Y/jw4fDy8kLz5s2hq6uLw4cPY+LEiRprIIwfPx6TJk1CSUmJRmvoYrEYrVq1QmRkpEbteZH/li1bYuTIkejbty+cnJxw6NAhAMDevXsRHh6uUWrrX4Xhw4cjNTX1pVmX/01wdnaGk5PT/4wy1X99ULC3t8eyZcuQkZGBn376Cffv30ffvn3h5+eHbt26CfUFqsLDwwMAMGnSJEgkEuTk5KhNlS0SibB06VKMHj0acXFxkMvlePDgAR4+fAgASE1NRXFxMa5evapW70ZXVxdz5szB2LFjYW1tjVu3buHcuXMYM2YMAKBz587w8fFBXFyc0j5fe+01TJs2DXK5HLdu3cKdO3cQGRmpNWHVuvDdd99h9OjRmDBhgtZ86uvrq1xXogpqflupVKpRUKhJb34a6qQi6+rqQldXV3gAKBQKyOVytdtVF/7rg4KtrS2Ki4sxdOhQoeDF3d0dCoVCo4uhX79+AIBmzZpBJpNh7969QjabqiCJuLg4REREwMDAACKRCC4uLnB3dwcAVFZWwtnZGVKpFMOHDxeChbKYMWMGZs+ejfT0dHz11VfYuXMnZDKZkL1mYWEBY2NjlXyam5tj9OjRMDY2RklJCaRSKaKiohAYGIiTJ0+q5EsZTJw4Ed988w3GjRuHK1euqHy8paUlzMzMAACtW7eGk5MTLCwsMGLECBw9ehSBgYEa5a60bdsWHTt2FMhfanqNAwcORPfu3eHr66uyz0aNGsHOzg7NmjXDoEGDhOxJoDqYff/997h//77S/ho3boxNmzahY8eOwpAxOTkZ4eHhyMzMrHVe7969K5DkqIy/O5vxZRmNxsbG/OCDDwQKLxcXF168eJFHjhx5KZnI88zDw0MoMS0oKODNmzeF0mxt2tO6DJ06dWJ2djanT5+ukg+RSMRp06Zx7969tYg7TE1NGRMTw5iYGMbHx/P1119Xya++vj4nTpzII0eOMDExkfHx8SwoKKhFsqIt69y5MwsKCvjZZ5+pdby1tTU3b97M+/fvC7wEJKlQKFhaWsry8nIuXLhQI52Jn376ib/99lst2nxdXV2GhoYyPDxcpYxMT09PBgUF8ezZs0xLS2NWVhZLSkpYWloqWFVVFYcPH65SG4cNG8aCggKmpqby7NmzjI+PZ3Z2tlD7UFBQIJRSf/vtt3Wdj38PnwIANmzYkKNGjeK1a9dIkv7+/ioXGdXcZCtWrBB+nOTk5Bex32rNzMzMmJ6ezm+//VblY/X09GhgYCC039bWlitXrmRJSQlLSkq4aNEitUVhjI2NaW9vT2tra06YMIFFRUVqs03XZQYGBjx//jzPnTun9k2rr6/PjRs3CsVK58+fZ2BgIEeMGMFJkyYxKytLkABQx7+bmxuLioqeoWIfM2YMy8rKlKb7NzMz46hRo3jnzh2SZEVFBaVSqXCzFhQUCBoNBQUFTEhI4LJly2hhYaGU/y+++IJyuZxXr16lg4MDW7Vqxbfeeot9+/blu+++y9mzZzM9PV3QMqmDX1Q7QQFACIAs1CZunYdqUtY7T2zgU+/5A3iEaum4ftoKCk2bNuWhQ4d44sQJHjx4kCkpKdy2bZvK3P8ODg68ePEipVIppVIpQ0JCatGP+fr6snvGKPtjAAAgAElEQVT37moFnBfZ8OHDWVBQ8KIKNqVsxowZfPjwIaVSKe/du8d79+5prZfTtm1bpqWlqaxF8CIbMmQISXLAgAEa+bGwsKC1tTWtra1rVXGOHz+eUqm0FrmvKqajo8ONGzfy0aNHtdixxGIxr169yp07dyp9LTg6OvLWrVssLy9nWVkZz507x0uXLrGsrIwFBQUCPV+N1ZDjfPrpp0q18+DBg5TL5QwPD6/zIdChQweB7u3SpUs0Nzf/8z5aCwp1ycbNAzC9jn1bAriLalJXZ1RTvOtoIyjo6urS3t6eDRo0oLGxMYcMGcKsrKxnBExeZl26dBG6XPn5+QJLsbu7O2/fvs3y8nImJSWxR48eWrsxDA0NGRMTw6CgII19hYSEMC8vj2lpacJT4ZtvvlFZ66DGevfuzVGjRnHEiBHcuXMnS0tLGRYWprXvvmPHDh47dkzrQRaoLv3et28fCwsL1abR8/b2ZlZWFmfNmlVr+7hx45iRkcFevXop5UdHR4dz585lbm4uMzIyuHv3bjo7O9PNzY1RUVG8e/cu3377bfbq1Yu9evVily5deOHCBcrlckZHRysV2L/44gtevXqVH3zwwTPl7dbW1ty3b5/QA34Of+mrE4PB84OCPwD/p/4/CaCzNoJCXbZgwQKePXtW6ForYx07dmRWVhb/+OMP/vHHH3RyciJQza9QUlLCzMxMFhQU8MiRIxrrSpqamtLU1JQnTpzg7du31e7ePm0SiYSGhoaUSCS8c+cO79y5w6SkJJXnFABw8uTJzMzMZGFhIQsKClhSUkKpVMqDBw8+V/zF3Nxc6TkHDw8P5ufnK/UkVMdcXFwYFRWlNodkw4YNef78eebk5LBr1660srJiy5Yt2bJlS965c4cXLlxQ+hpwdnZmaWkpZTIZ9+/fL2wXiUTP8DLW2K5du1heXk6pVKq0qM/zhhrDhw9nWVmZIN33HMq7VysGA8BPJBLdE4lEISKRyOLJNntUq0LVIOXJtmcgEol8RSLRTZFIdFODNtSjHvXQMtRdklwLYD6qo898AMsBqLT4THIDgA3As7UPIpEIJiYmAPDC5KTLly9jzJgxsLW1RWJiolKfKxaLay0NlZWVwcHBAT4+PsKSp1wuR8eOHTFo0CDs3bv3pT6NjY3xzjvv4LXXXkNWVhbu3bsHuVyOH374AUA1p2CvXr1qcU2qi8rKSmG5LCcnB0C1rsTTHI4vg0gkwoIFCzBhwgQYGxtDoVBAIpFALBajqKgIb775Jn755RfcvXsXQPVvIBaLYWRkBENDQ6xcuVLIpnzRZ0yaNAnJycn49ddf1fy2L0afPn3QrFkz7Nu3T63jhw8fjm7duqGqqgqbNm1CQUGBcF20aNECsbGx6NOnD44dO/bSnIIaSryqqqpaGaYk67yG7O3t4eLiAoVCAR0dHeFzX4a6StD19PQwbNgw6OrqCudCFR7MP0OtoEAys+a1SCTaCODok39TATR5aleHJ9tUgoODg8Cok56ejh07dmD37t3P/DCNGzeGRCJR+oQCQFRUFK5cuQIvLy8A1aIo3bt3h4ODAyoqKoSby9DQEF5eXkoFhUWLFuHDDz/E7du3oaenh9mzZ8PExARGRkYAgMLCQmzbtg0GBga4fPky9uzZg3PnzmmUpi0SiWBtbS38r0oijKGhIfz8/IRzp6+vj9TUVCxcuBC3b99G3759MWXKFCHBy8zMDAUFBUhISMCiRYteGhAAwNraGsOHD8eWLVteCeuUWCxG7969UVFRgePHj6vlIz09HYGBgTh16hSKi4thb2+PI0eOAKgWbVm6dCny8/OVOrelpaV49OgRnJ2d0bNnT4SGhiIkJEQgBI6KioJEIsGUKVMAAL6+vrC3t0dJSQmWLl2KhIQEtb4DAHz55Zd45513cOHCBaxcuVJtPwLUnFNo/NTrqQB2PXndCrUnGh9DjYnGJk2aMDs7m9nZ2dywYQN9fHyeWV5p0qQJIyMjuWrVKpXHkp988glzc3OZm5vLgoIC5ubmMisri9nZ2czJyWFubi7j4+OVFkO9ePEiT5w4IYzT8/LyePbsWY4YMYIjRozg+++/z7lz5zIwMJAHDx5kVlYWo6Ki+M4779TpTyKRcPDgwc8dZxoYGHDp0qXCkuTmzZtVYvPR09PjjRs3BA2JjRs30tvbu9Y+Tk5OdHNzo5ubG4cMGcJmzZqpJDLbqFEjFhQU0M/PT63x/stMLBbz9OnTPHz4sMZzPzXm7+/P5ORkJicn08PDQ+XjO3TowKtXr1Iul7O0tJTFxcUsKytjTEwMN2/ezF27dgnXtVwu540bN/jFF19oxEo1ZcoUJiQkMCYmRhkdFK2tPvwKIB1ABarnCCYC2A4gEsA9AIdRO0jMQfWqQwyAAUoGnVqNr1km2rhxI3fv3v3Ml/Py8uLp06cFtWhVT6RIJKK/vz/9/f2ZlZXF4uJiFhUVMT09neHh4Xz48CGnTJmitL+lS5cyLS2NJ06cIEmGhYU9lzJOT0+Pbdq04eeff86kpCSOGzfumX3s7OwYHx/PadOmPfNeq1ateOjQIUF5+MqVK2zatKnK56BZs2bs1KkT3d3d/6+9Mw+rqmrb+L05cJgRGTQFhcRUTI3ETEl7UYMATevNpAwScyoVh3xTEzXjtTT7zDnLsTQlM0RRKRVEEHBCNAREJgcGZR4PKgL39wec/aGBnqlQv/O7rucCNmc/rLXP5jl7rfWs+/lbpNI6dOjAnJwcjRZnefAa3bx5s9lrpIq1a9eOZ86c4datW7l161aV/fTu3ZvLly8XE4qqqqpYXV3Ne/fusba2Vgzk4eHh7Nmzp1oiucOHDxcLwyhYvVozdR9IvtfM4a0Pef2XAL58lN+HUVdXJ46Ndu/ejdWrV2P9+vXo3bs3Xn/9dXh5eSEjIwMfffSRUvn+Tdoo1naorq6Gq6srKioqcOTIEYSEhIj1BBRlxYoV6N69O+zt7TF16lTs2LGjRWGRmpoaJCYmIjExEZaWlqKk+IOvSUpKwuLFi2Fra4usrCxIpVI4OzvD3d0dJBEVFYWZM2cCgEqPnunp6WqNOx+FTCZDVlaWUnMdyjBp0iQYGhoqnTLeEu+99x4cHBzw7rvvquXn0qVLmD9/PrZv3w5fX18YGRmJ/2wSiURMRf7tt9+UGvY+iJubG1atWoVOnTrh22+/xZo1a9Rq930oEjn+bkMLn6hSqZTTp09nREQEk5OTmZ2dzf3793PmzJkKZ4H9U2ZsbEwzMzOlzjEzMxOXRB+05557jj/88ANv3rzJuro6kmRWVhaXLFnCYcOGPXb9f9AkEgkPHz7MuXPn/i3+ly9fzrS0NPbp00cjbf3555+bfSp9HM3e3p6nTp1iRUUFt2zZokxJwqcnzblNmzbs2rUrO3furLHx45NgUqmUNjY27NKlCx0cHNihQ4dWb5OiJpFI+P333z+yrqWqtnv3bqanp6uUn9Hcdf7vf/+rcDpza5qFhQVDQ0N5+/ZtpqamKlVMF09TUNCa1h60GTNm8MqVK//vJPRHjhxJmUzGI0eO0NnZWdnznz6Jdy1a5Jibm6Nbt274888//1Y9hccNKysr9OrVCxkZGcjJyVH2dK1wqxYtWu5DK9yqRYsW5dEGBS1atNzHYxsUjIyMYGRkBBMTE5ibm8Pc3By6uo+9etw/grGxMdasWYO0tDSkpaVh/Pjxrd2kJxIjIyPx3jI3N2/t5jw2PHb/ZW3btoWfnx9effVVAA15+oaGhgAaEm4iIyMRHx+vdtVpOdbW1nj99deRnp6Oa9euick2JMUKwaoirwjs7u4OY2NjnD59WilNvuZo06YNvvnmG3h7e4t5+upUL26KmZkZ3N3d0a5dO0RHR6vd1kfRtWtXAEBOTo64GU1RTE1NMWbMGOzatUvpczt27AgPDw+MHDkS1tbWqK+vR319PQ4cOIDg4GAAUHiDnSL07dsX/fv3R2FhIQ4datgmpOrkqIWFBQYOHAipVAqSyM7OhrW1NSIiIjQnutvay5EPLkn27t2bERERDAkJYUhICH18fPjFF1/wm2++YUREBDMzM5mSksKAgACNVO9dsWIF6+rqePXqVZ46dUpMHT59+jT37NnDwYMHq+TXzc2NsbGxjI2N5d27d1lfX8/MzEz++OOPfOGFF9Rq77179xgXF8d27doprTzVnPXu3ZsLFy5kWFgY09PTSZJLlizRyBKaIAiUSCTU0dGhmZkZ+/bty9mzZ/PgwYOiTuaqVauU9mtlZUWSKlUJHzduHIuLixkREcGvvvqKQUFBTElJYVVVFYOCghgUFKTR5LAFCxaQJOPj42lpaamSroahoSH9/f0ZFhbG+Ph4pqamcvXq1Tx48CALCgoUlfl7MvMUBEGgVCq9T/RUbjo6OnR0dOS6det46dIlxsTEsH///iq9UYMGDeKgQYN48+ZNXrt2jampqSwpKWFNTQ3r6upYVVXFvLy8v+j2Pco++eQTrlmzhunp6eKmqx9//JGffPIJDx8+zMrKSv7+++8qaSr6+fmxtLSUN27cUDlYNTWpVMrAwEAeP36cGzZs4L/+9S8OGzaMly5d4rhx49Ty3b59e3p7e3P79u08ePAgQ0JCePbsWZaUlLC6upokRZWgkJAQlf7G2rVruXTpUqXPs7CwYLdu3e7b82Fubs6ffvqJBQUFLCgo4LBhw9S+vnL7z3/+w/LyckZFRdHCwoIWFhZK+5g/fz737dtHLy8vmpqaikl8FhYW3L59OwsLCzlo0KBH+Xkyg4KiNnToUJaUlDAyMlKp3XtAQ9QNDw9neHg4y8rK6OPjQ0dHR3p6evLjjz/mJ598wtGjR7Nfv35KKRu/8cYbLC4uJklGRkaK0lvyACAIAufPn89bt24pHWzkCk41NTVctmyZRm5WY2Nj+vj4sHfv3uKxTZs2cd++fWr5dXFx4e+//06ZTMaamhreuXOHd+/eZVVVFf/8809GRUXxf/7nfzhmzBiOGTOG7du3V+nvLFu2jJs3b9bItQAaEqJu3brFW7du0d3dXWN+NREU2rVr15wQK01NTXno0CHm5OS0mDLfxDSzIaq1sba2hq2tLSwtLSGRSPDqq6/Czs4Ozs7OqKysRHx8vNL69n5+fnj55ZcBACRx7do1XL58GZcvX1a5nXp6enj//fdhbm6OzMxMBAYGIjIy8r7XkER8fDxMTEwwfvx4/PbbbwoLr3h5eWHgwIG4evUqdu7cqXI7myKTyfDzzz+LP/v7+8PFxQV+fn4q+ZPX0li3bh06d+6M3Nxc7N+/H5WVldDX10dERARSUlJQXV2N8vJy+QeCygiCoLaPpvTp00ecR1JH36A5SKJTp0545ZVXAECcD1KUgoKCZo9Pnz4dbm5uOH36tMbmQR7boCCfUZ89ezasra3FoiXFxcW4evUqQkJCcPjwYZw5c0apWopdunTBhAkTxIo99fX1eO2111BaWork5GSV2/vuu+9ixIgRqKqqwvLly/8SEORIpVLcu3cPnTp1gomJicJBwdXVFWZmZpg7dy5SUlJUbmdLjBs3DgsWLMCiRYsQH6+8Qp5EIhEFRDp16oT6+nqkpqZiz549SEhIUKskX0vU19c3W3lJFdzc3DBq1Chxd66mdl8CDbt+BUFAdna2RsvlvfjiiwgICEBRURE+/fRTjfl9LIOCjo4OvLy8AAB2dnaor6+HIAiorKxEWloaDh8+jJCQkGalqR6Fq6srnJ2dxacLkpg7dy58fHywaNEilaXDRowYAUNDQ2zYsOG+T98HIQldXV0kJiYq3H4nJyd4eXkhNTVVvKmsrKzEsnF2dnbYtWuXyhWu5Fu+f/nlF2zd2uKu+IdC/t9qjUwmg1Qqhbu7O/r27YvY2FgsWbJE46sZNjY2aj3dNeXll1+GlZUVoqOjNeJPjoWFBZycnDS+hdzExAQBAQEwMjLCtm3bcO7cOc05b+35hJbmFJ555hk+88wzHD58OGfMmMF169bx119/5cmTJ3n9+nXm5eXxq6++4r/+9S+lZuC9vLyYkJDA3bt3c/fu3Zw5cybXr1/PW7duMS0tjT169FB6vPfaa6+xtLSU4eHhtLGxeehrR4wYQZlMxtDQUBobGyvkf9KkSayqqmJgYCCBBhXjiIgIcSJTJpNx586dKgt2+Pr6UiaTNSv4ooy1adOGbdq04UsvvcT58+dz//79vHHjBuvq6hgbG6uRHY1yk0gkPHLkiEaUok1MTHjixAlGR0fTyMhI5cpjzVmfPn34559/sqqqSq05haZmYGDAjRs3UiaTMTY2tiXl5ubs6Zto1NPTY6dOnejh4cFp06bxjz/+YFZWFiMiItirVy+Fbjq5jwePjx8/njKZjL/88ovS/1zBwcGsr6+nr6/vI1+7YsUKlpaWcurUqQr7nzRpEsvLy0Vps23btjE3N5f+/v709/dnQkICQ0NDVa7A5OHhwfz8fObm5nL69OlqqQE1NX19fbq6ujIuLo61tbU8efKkxnY1GhoaMjU1VenSa82Zl5cXa2pqNLYM29T69OnDxMREVlVVMTo6WiNBYdGiRZTJZMzLy6Ojo6My5z6ZQcHMzEzh5TojIyO+8sorTE1NZXR0NKOjo9VaX/7666+Zl5fHAQMGKHXegQMHWFdXx7Fjxz70dc8//zyzsrIYGxur1KeRPCh8/PHHHDRoEEtLS/nFF1+In8ypqalcvHixyv2WSCRs164dFy1axEuXLol1O5W58Xfu3MmXXnqpWV3LXr168eTJk6ytreWBAwc0Uv/CxsaGdXV17Nu3r9q+Zs2axbt372okwDxojo6OPH36NCsrKxkbG6tynoLcPD09WVRUxJycHPr5+dHU1JQuLi58//33OWfOHO7Zs4d79uzhwYMHGRYWxsOHD/P999+Xn6+Zug+NdR0KBEFIanJsjyAIFxvtmiAIFxuP2wuCcLvJ775/lH8tWrQ8Xigy0fgjgPUAdsgPkPSWfy8IwkoATTW8M0k6qdIYa2trBAUFYcOGDQgJCXnk66urqxEbG4udO3di6dKlABomdlSZgASAyMhIjB07Fj179kRCQgJqamoUOq/xaeeRvPHGG7Czs0NERASqq6sVbpdEIoGenh46d+6MoUOHIiUlBVu2bME777wDoCH1WV6jQRXq6upQUFCA+Ph4vPPOO0rP6FtaWsLT0xNubm4AgPXr12Pt2rWihH1SUhImTJiA/fv3Y8iQIejevbtK5eiboqenh7y8PI1MNMonnd3d3dGjRw8AaHafjUQiQUlJCVauXKmw79TUVISHh6NHjx5ISkpqUbtTEUaOHIlNmzbB3NwcpaWlmDp1KubOnYtnn30WBgYGqKurQ1ZWFgDgypUruHLlCgoLC5W+NxQRbo0WBMG+ud8JDVOqYwAMVeqvtsDt27fF+gOurq4AGoRbz507B0EQ/rL0KM9/9/X1FWeNH7bEp6Ojg3HjxsHS0lKcZW8aQFJTU1FbW4sxY8bAxMQEGzduVCifXL468rB/Jh8fH8yaNQvp6elYt27dI302JT4+HgUFBfD19RXf+GXLlsHDwwMA8PPPP7e4BKoMLi4u0NPTUzjIyTl9+jQCAwMREBAAAFi8eDFcXV2xYcMG/PHHH7h9+zbS0tJw69Yt2NjYKO2/OUgiPT1d6RyV5ggLC8PIkSPx/vvvQ09P76Gv3bBhg1K+SYo5GWfPnlV6n0ZT7ty5g6ysLGRmZkJHRwcpKSno0KED9u3bh4qKCuTk5CAxMREAcPXqVZUDkLpLkoMB5JNsKgv8rCAIFwBUAFhI8mRzJwqCMBnA5KbHqqqq4Ofnhzlz5sDX1xcAMHz4cCQnJ8PIyAjXrl1DUVERSMLQ0BAeHh6wtbVFQkICZs+eDQAoKytrsbEODg74+uuvYWxsjNGjRwMAzp49i/z8fKSkpMDR0REGBgZwcXFBVFSUwhtMVq5ciSFDhmDq1Kk4d+4crly5Al1dXXHDz4wZM/DBBx8gNTUV/v7+4hunKPHx8Zg7dy6WLl2Krl27wtLSEs888wy++OILAA03qjLFYJrStm1blJeXo0+fPnjrrbcQFxf30GvYHLdv38batWuRkZEBAJg7dy6GDBmCl156CYmJiThz5gyMjIzQo0cPsUKXutTV1WkkIABAdnY2vL298eyzz2Lw4MEA8Jc2SiQSpKamIiIiQmn/JKGjowM/Pz8cOXIEAFRRTcLRo0dx/PhxcXnz3r17kEgkSuXpKISCE4H2aFIMpsnxjQDmNPlZH4Bl4/fOaKgraabK6oO9vT3t7e35zjvvMDAwkMuXL2doaCijoqIYGRnJqKgo/vrrr5w+fXqzqwnNma6uLjdv3sw7d+6IVlNTw9raWtbX17O+vp7FxcUMCAhQeLlQbv7+/szNzWV8fDyXLl3KXbt2MS0tjWlpaayvrxerECvj80F74YUXOGXKFPr6+rJnz55qT4L179+fBw4c4KZNm5iamsqwsDCV6mg8aLa2tvz000+ZkpLCmpoa0aqqqnj06FG1r4O87REREWr7+Sds7ty5rKysZHFxMSdOnMiJEye2Vls0p9HYOHw4RLJXk2O6aCgJ50yy2bAnCMIJNFSnfmiKnKJybMbGxuJYTxAE3L17V+lPC0tLS/j5+YmfCPb29sjMzEReXh6Kiorwxx9/4OzZsyo94trb28PHxwcjR46Eubm5mJq6ceNG7Nu3T2OfbJrC1dUV33zzDfT19bF3716sXr36obU7laVjx45wcXGBo6MjdHR0cPz4cSQlJak859OUzp07Y+LEiVi8eLEGWvr3MnHiREyZMgW7d+/Gnj17AECpuiIaRCE5NpWfFAB4AIh64Jg1GsvEAeiChqBhoak8Ba1pTWtqmcaWJIMAnALQXRCEHEEQJjT+6l00lJRryqsAEhuXKH8D8BHJkkf9DS1atDw+aNWctWj5/4NWzVmLFkVxcXHBRx99BFNTU5iamrZ2c1qVx3KXZEsYGxvj7bffRrdu3ZCTk4Pvv3+8EyatrKwANBQuyc3N1dhEo62tLaysrMTkqpycHDFRSItq9OrVC+vWrRP1QFetWtXKLWo9nqjhw+eff44FCxaIGXgBAQHYu3evwpmHLWFqaophw4YhJSVFI/vo27Zti3//+9/4+OOPAQDOzs4ICgrC4sWLxbV8VejUqROmTJmC7t27o7S0FIMGDQIAxMXFYdKkSUqvmEilUpibm6NHjx5iQszQoUNx+fJlXLx4Ue3r2pRnn30W/fr1Q9u2bUUBXk3g4OCAnj17ws7ODmfOnFF5C7GxsTFCQkJgYWEBABg1ahRyc3M10kZ9fX2MHDkSFhYWuHXrFgAgIiICVVVVavvu2LEjBgwYIArQyu+BEydONHevaYcPWrRoUYF/cjekukuSo0aN4qZNm3jq1CkWFxezpKSEW7ZsUSnhRhAECoLAt956i8ePH2dNTQ3j4+OV3iH5oPXv35/h4eGsqqri+fPnef78ee7du5dVVVXctm0bDQwMVPL74osvMiYmhl9//bWoxefu7k53d3dev35daR2IN998k8ePH2dcXBxTU1MZHh7O6Oho3rp1i5cvX2ZISAj79Omjdqn3Nm3acOHChUxJSaFMJiNJZmRkKKV92ZxZWVlxyZIlTEpK4o0bN3jjxg3Gx8fT2tpaZZ9Tp04VE63ee+89tdoHgF27dmX//v3p4+PDkpISkhSFYUNDQ9US3+3ZsyfXr1/Ps2fPsri4mFVVVSwuLmZ5eTnLy8uZkJBALy+vB897MrdOK2JmZmZ8++23eeDAAd6+fZsbNmxQ+qJOnjyZkydPZnZ2NqOiohgTE8Pi4mLGxcUpnCHZnB04cEAUPbGxsaGNjQ0FQeDFixcZFxfHjh07quTX29ubvr6+9ykQOzk50cnJifn5+UrJxnt6ejInJ4ehoaEcO3Ysu3btyg4dOogBQL49+9tvv+W3336r8rWwt7fn2bNneefOHYaEhDA8PJzV1dXcsWOHSpoNEomEEomE48aNY0JCAgsLC7lt2zY6OzvT09OTeXl5DA4OVjngtG/fXvynnT17tsr9ltvGjRtZVFTE69evMyUlhYmJiaIwzt27d3n58mW6uLgofQ1GjhzJ5ORk3rhxg0eOHOG8efM4cOBAWltb84cffuAPP/xAkvzPf/7z4PlPh3Brc1RUVCA4OBjR0dH47bffMHjwYDz33HMAGgrGKEK/fg1Dq+DgYMyZMwcWFhbYvn078vLy1Nq0EhgYiK1bt+Lo0aP3+amtrVUpS1KOPBOuKSYmJir56ty5My5duoS33nrrvrz5mzdvwsnJCaNHjwZJxMTEqNxeqVSKefPmwcHBAZ999hlWrVqFYcOGISQkBMeOHVPpWkydOhUAsGLFCiQnJ+PDDz/EwYMHYWJigrVr10IqlWL37t0qt7mkpATl5Q0bfo2MjFT2I0dPTw8GBga4cOECFixYgLt376JLly4AgLFjx+KNN96Am5ubwjtGjY2NMWfOHMyaNQupqan45JNPEBUVJd5n06ZNw5AhQwA07AFSNWvyiQwKcgoLC3Hw4EF89dVXYtqyokFBTllZGerq6lBeXg49PT3o6OigsLBQ5TadP3++Wa1EHR0dCIKgUa0++aauO3fuKDUpeOHCBVhbW/9lI82AAQMQFBQEHR0dzJs3D/v371e5bZaWlhgxYgR27twpzuR7eXmhurpape3O3bp1w6xZswA0bHGfNm0arl69CgBYsmQJnJ2d8cEHH4gVnlTh3r174j+otbW1yn7kZGVlQUdHBwYGBkhJSYFMJhM3w9nb2+Pf//63UsHR3NwcHh4eqKiowJQpU3Dp0iUADdu8AwMDMW3aNGRmZgIAfH19ceHCBdUa3tpDB1WGD3Lr0qUL4+LiSJJTpkxRqpbChx9+yA8//JAXLlxgjx496O3tzaSkpGaVg9S1gQMHMj8/n4cPH2abNm004lNPT4/JyclMTk7mrl27lD636eN7x44dOXHiRB47digCWmQAAA1NSURBVIwrV65kly5d1G5fp06dWFJSwrfffpsA+M4777C8vJxffvmlSrJxM2fOZElJCUtKSvjmm2+Kx11dXZmTk8PVq1erLEfX1JYuXcqlS5cyJyeHHTp0UMuXjY0No6KiWFFRwVmzZtHAwIBubm50c3PjkSNHmJubq1TRHRsbG8bExDApKUmsdSKVSrl06VJRr9HR0fFhEm1P9pyCoaEhDQ0NaWpqyu7du9PQ0JDdu3eno6Mj7ezs6OHhwWPHjvH27dvMz8+nl5dXcxMrLZqZmRnNzMwYERHBS5cuMSsrS20RUKlUSqlU+pfjCxcuZHl5uUZERuVmZWXFrKwsZmVlqbzrztramp999hmTkpJIUhyTjx49+r65C1WsTZs2PHHiBCMjIzlv3jwWFBQwMjJSZbk8T09PcTy+Y8cOGhgYsEePHrx48SJTUlLUmgdqaoGBgQwMDOSdO3doa2urtr8333yT+fn5vHHjBsPDw5mdnc3s7GwmJCTQ399fqUJGRkZG3Lx5MysrK7lw4UIxIJSWljIkJIROTk6P8vHkzSnY2dmJegMjR44E0LCPvVu3bsjNzYWDgwMkEglqamrQtm1b6OnpoaCgAFu2bMGxY8eU+lvyZJ8lS5Zg3759sLKygpWVFaRSqdLr82ZmZggICMDw4cNx7949xMTEIDg4GN27dwcAzJw5E5mZmTh69KhSfh/GzJkzxXX0LVu2KH2+iYkJvv/+e7i5uSEjIwPR0dFITU2Fvr4+tmzZAk9PT8yZMwfAwzUqWqK8vBxxcXH49NNP0a9fP+Tl5WHKlCkq75A8duwYNm/eDACYMmUKgoKCYGlpibZt28Lb2xvZ2dkq+QUa8gg8PT0xaNAgvPjiiwAaduH27t1bJd2DppSWliI/Px/29vbo06ePODcUFBSktPpUdXU1AgMD4ejoCH9/f7zyyivo378/wsPDMXnyZI3sPgWAVn9KaPqk8Pnnn1Mmk7GsrIylpaWiFRcXi1/l35eVlbGiooIbNmygra0t9fT0qKenp3QkX7hwISsrK3ny5ElmZGRw3bp1SvsYO3YsKyoqGBwczDVr1vDmzZssKChgUVERi4qKKJPJOGLECLU/deT969evH69evcrx48dz/PjxKvkyNDTkqFGj6OHhwXbt2t2nHzF79mwWFxfT1dWVrq6uKvk3MDAQVa5jYmLYuXNntfuvq6tLXV1dzpgxg/n5+SSp0vv1oHl7e7OsrIzXrl1jRkYGMzIyeO/ePV65coVr1qzh66+/rrRPe3t7BgYG8sKFC+J9e+3aNXbv3l1tReuXX36Zly9fZlVVFfPz85UpcffkDR+OHz/O8vJyFhYWNmslJSUsLy9nZmYmQ0JCWFhYyIyMDK5cuZL6+vpKLUXJhw+RkZE8c+YMzc3N6erqyoyMDH788cdKvUnLli1jTk4OXV1d6eDgwJiYGN67d08MCrdv32ZwcLDK0t69e/fmd999x9DQUP700088ceIEt23bpnIgfJQZGRkxKipKVMhWthiusbEx9+zZQ5lMxoyMDJ46dUojj+Jyk1eNLisr49mzZ9UWbVm1ahVTUlLYq1cvcfhAkpcuXWJeXh7T0tL42muvKezPysqKBw4c4N27dxkbG8sPP/yQWVlZrKqqoouLi9LLkM3dD/LCyPn5+QwNDVVUdOfJCwqzZs0Sky+aPilUVlayvLycN2/eZEhICF1dXdmmTRu+9957PHToEFesWKH0P4hcHj0mJoa7d+8Wj3/xxRfct2+fUklG//3vf1lZWcnDhw8zOzubZWVl3LVrF729vent7c2jR4+yoqJCpTkFIyMjHj16lKGhoRw6dCgvX75MkkrNn6hikyZNYmZmJjMzM5XOrZg9ezZramq4Zs0a9urVizk5OWIhG3XM2dmZzs7OzMvL48mTJzlt2jTeuHFDpVL2Te3UqVPcu3cvpVIpt27dyq1bt/LcuXNidWpnZ2eFA7quri5XrFjByspKrlu3jn379qW9vT2Tk5NZUVHBgQMHcuDAgSq3VV4cOSEhgcOGDePUqVOZmZnJy5cvs1+/fo86/8mbU/jpp5+QnZ0NLy8vtGvXDkCDKGpsbCwKCwuRnp6OlJQUlJQ0SDQEBQXhyJEjsLa2VnqpTy5qmZaWJm6CAYAzZ86gf//+0NPTUzhfYceOHejduze6du2KpKQkbN++HQcPHhQ3QEkkEgwePBgvvfSS0pu4hg4digEDBqB///5wdnaGqakpSMLJyQlhYWFK+XoQXV1dvPDCC80uodbU1MDS0hJAQ3k2Rda8pVIpAOCVV17BjRs3sGjRIlRUVKCwsPCRgqiPwsTEBMuXLwfQMLb29/dHTU0N5s2bh+eff14t3ydPnsTw4cPx6aefitqgAQEBKCkpEe81RRk2bBgmTZqEI0eOYP78+ZDJZOjXr594LdXBwMAAs2bNQseOHeHj44OEhAREREQgNTUVW7ZsQWBgICZMaJA7uXnzpsp/R7v3QYsWLffxWD0plJaWIjg4GPv27YNEIgEAkHyoWq0q0RyAWAV527ZtmD17NiwtLVFRUQEXFxdcv35dKa3C9PR0vP3226Ky7oPtra2tVVl9OD8/H3l5eVi0aBG6du0qKiVPmDABmzZtAgAUFRUp7bdjx44IDAzEoUOHxCcFMzMzjB49GmPGjMGAAQNw4MABAFC4boCxsTEAiDsh79y5I27zVjdpq0OHDnBwcADQIPvv4OCAL7/8Enfu3MGyZcvU8v3LL7+gXbt2cHFxwd69ewEAhw4dUsmXi4sLjIyMkJubCysrK9ja2qJr166QSqXIyMhQS5vRwsICw4cPR1ZWFhISEqCnpwc7Ozt07twZgiDAxMREI8lxj1VQkEPybyld3hznzp1DcXExdu3ahdraWri4uGDixIlK+2kuGMh54YUXUFRUpFIK7vnz5zF9+nT069cPX331FZKTkyGTyTBixAhxyVOVoNCrVy/4+PhAX18ftra2yM7OxtixY+Hu7o4TJ07A399fzGhUdIlWviQWFRWFBQsWICQkBAYGBrCwsMD169eVbmNTOnToIA5P3n33XUyfPh1paWmYMWMGTp8+rZbvhIQE+Pn5qeVDTlJSEsrKyuDr64shQ4ZAX18fhoaGqKmpwbp169S6DoIgoL6+Ht27d8eWLVtgYGCAvn37olOnTsjIyMD69es1Iwjb2pOMTScaW8uMjY05ZcoUHj16VKMJRnLr0KHDI6tRK2O6urq0tbWlqakpTU1NVfJhYmLCMWPG8LvvvuPhw4d55swZnjx5km5ubmonLhkZGTEgIIDp6elMSUnhhAkT1F4lad++PcPCwhgWFsakpCSuXr1aoysamjKJREJvb2+GhYXx4sWLTEhI4O+//04fHx+V36umvkePHi1mNSYmJnL//v0cM2YMra2tFdlkpjmJ978brUajFi3/CFqRFS1atCjP4zKnUARA1vj1acMKT2e/gKe3b09rv+wUedFjMXwAAEEQ4hV5tHnSeFr7BTy9fXta+6Uo2uGDFi1a7kMbFLRo0XIfj1NQ2NTaDfibeFr7BTy9fXta+6UQj82cghYtWh4PHqcnBS1atDwGaIOCFi1a7qPVg4IgCB6CIFwRBCFDEIT5rd0edREE4ZogCJcEQbgoCEJ84zELQRCOCYKQ3vi1bWu381EIgrBNEIQCQRCSmhxrth9CA2sb38NEQRD6tl7LH00LfVsiCEJu4/t2URAErya/+6yxb1cEQXi9dVr9z9GqQUEQBAmADQA8AfQE8J4gCD1bs00aYghJpyZr3fMBRJB8DkBE48+POz8C8HjgWEv98ATwXKNNBrDxH2qjqvyIv/YNAFY1vm9OJMMAoPF+fBfA843nfNd43z61tPaTQn8AGSSzSNYA+AXAqFZu09/BKAA/NX7/E4A3W7EtCkEyGsCDe9Jb6scoADvYwGkA5oIgdPhnWqo8LfStJUYB+IXkXZJXAWSg4b59amntoGADoKkMb07jsScZAjgqCMJ5QRAmNx5rT1IuhXMLQPvWaZratNSPp+V9nN44/NnWZIj3tPRNYVo7KDyNDCLZFw2P1NMEQXi16S/ZsAb8xK8DPy39aMJGAA4AnADcBLCydZvTerR2UMgF0KnJz7aNx55YSOY2fi0AEIKGR818+eN049eC1muhWrTUjyf+fSSZT7KOZD2Azfi/IcIT3zdlae2gcA7Ac4IgPCsIghQNEzqhrdwmlREEwVgQBFP59wDcASShoU/jGl82DsCB1mmh2rTUj1AAHzSuQgwAUN5kmPFE8MAcyFtoeN+Ahr69KwiCviAIz6JhMvXsP92+f5JW3TpNslYQhOkAjgCQANhGMrk126Qm7QGENOrk6QLYTfIPQRDOAfhVEIQJAK4DGNOKbVQIQRCCALgCsBIEIQfA5wCWo/l+hAHwQsMkXDWA8f94g5Wghb65CoLghIYh0TUAUwCAZLIgCL8CSAFQC2AayZZFQ58CtGnOWrRouY/WHj5o0aLlMUMbFLRo0XIf2qCgRYuW+9AGBS1atNyHNiho0aLlPrRBQYsWLfehDQpatGi5j/8FEIchWqqlybIAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import random\n", "from PIL import Image\n", "\n", "# get 100 random images from the dataset\n", "num_samples = 100\n", "samples = random.sample(list(X), num_samples)\n", "display_img = Image.new('RGB', (200, 200))\n", "\n", "# loop over the images, turn them into a PIL image\n", "i = 0\n", "for col in range(10):\n", " for row in range(10):\n", " array = samples[i]\n", " array = ((array / max(array)) * 255).reshape((20, 20)).transpose() # redistribute values\n", " img = Image.fromarray(array)\n", " display_img.paste(img, (col*20, row*20))\n", " i += 1\n", "\n", "# present display_img\n", "plt.title('Examples from the dataset')\n", "plt.imshow(display_img, interpolation='nearest')" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Add a bias column to X\n", "bias = np.ones((m,1))\n", "X = np.append(bias, X, axis=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Compute the Cost and Gradient\n", "The following functions are copied from [ex2](https://github.com/rickwierenga/CS229-Python/tree/master/ex2)." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def sigmoid(z):\n", " return 1 / (1 + np.exp(-z))" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def compute_regularized_cost(theta, X, y, _lambda):\n", " m = len(y)\n", " regularization = _lambda / (2 * m) * np.sum(theta[1:] ** 2) #np.squared()???/\n", " cost = 1/m * (-y @ np.log(sigmoid(X @ theta)) - (1 - y) @ np.log(1 - sigmoid(X@theta)))\n", " return cost + regularization" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def compute_regularized_gradient(theta, X, y, _lambda):\n", " m = len(y)\n", " n = len(theta)\n", " gradient = np.ones(n)\n", " hx = sigmoid(X @ theta)\n", " gradient = (1 / m) * X.T @ (hx - y)\n", " regularization = (_lambda / m) * theta\n", " regularization[0] = 0\n", " \n", " return gradient + regularization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### One-vs-all classification\n", "One-vs-all classification works by finding a decision boundary between every class (denoted $k$) and every example that is not in the class (0). We will store the found values for theta as rows in our matrix $\\Theta$ (capital) where every column is are the values of $\\theta$ like we had in binary classification.\n", "\n", "In this part of the exercise, you will implement one-vs-all classification by training multiple regularized logistic regression classifiers, one for each of the K classes in our dataset (numbers 0 through 9). In the handwritten digits dataset, $K = 10$, but your code should work for any value of $K$.\n", "\n", "**Exercise**: Implement one-vs-all logistic regression. Your code should return all the classifier parameters in a matrix $\\Theta \\in \\mathbb{R}^{K\\times(N+1)}$, where each row of Θ corresponds to the learned logistic regression parameters for one class. `scipy.optimize.minimize` is already imported as `optimize`." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "def train(X, y, K, _lambda):\n", " n = X.shape[1]\n", " theta = np.zeros((K, n))\n", " \n", " return theta" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "K = 10\n", "_lambda = 0.1\n", "theta = train(X, y, K, _lambda)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Evaluation\n", "According to the exercise, we should have a 96,46% accuracy over the entire training set using multi-class classification." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Training set accuracy using multi-class classification: 10.0%'" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Make sure to add 1 to the result as `y` is one indexed while the prediction is 0 indexed.\n", "accuracy = np.mean(np.argmax(sigmoid(X @ theta.T), axis = 1) + 1 == y) * 100\n", "'Training set accuracy using multi-class classification: {:2}%'.format(accuracy)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Multi-class Classification with Neural Networks\n", "\n", "---\n", "Neural networks are a more advanced model for doing logistic regression. Neural networks are based on the brain. Neurons in the brain are nodes in the network. $x_1, x_2, ... x_n$ are the _input layers_ of the network and the hypothesis $h_\\theta(x)$ is the _output layer_. The output layer exists of $\\theta_0, \\theta_1, ... \\theta_n$.\n", "\n", "Between the input layer and output layer are _hidden layer(s)_. Each connection in the network has a weight we previously called \"theta parameters\". The hidden layers' values are denoted by $a$. $a^j$ is the activation of layer $j$ in the network and $a_i^{(j)}$ is the activation (value) for neuron $i$ in layer $j$. Although networks in theory can have an infinte number of neurons, it is often impraticial to use too many layers at it slows down learning and training dramatically. You should choose the number of hidden layers depending on the complexity of your dataset.\n", "\n", "Each neuron has an _activation function_. In this case we use the sigmoid function.\n", "\n", "For a network consisting of a single layer with 3 neurons in the input and hidden layer with one neuron in the output layer, the proces would like something like: (note that $a_0$ os not shown - it will always be equal to a vector of 1's) \n", "\n", "$$\\begin{bmatrix}x_0\\\\x_1\\\\x_2\\end{bmatrix} \\rightarrow \\begin{bmatrix}a_0^{(2)} \\\\a_1^{(2)} \\\\a_2^{(2)} \\end{bmatrix} \\rightarrow h_\\theta(x)$$\n", "\n", "With a network wired as shown below and the sigmoid function, $g$, as the activation function, we would optain the values for node $a_i^j$ as follows:\n", "\n", "$$a_i^{(j)} = g(\\Theta_{i,0}^{(i-j)}x_0 + \\Theta_{i,1}^{(i-j)}x_1 + ... + \\Theta_{i,n}^{(i-j)}x_n)$$\n", "\n", "We can vectorize that as:\n", "\n", "$$a_i^{(j)} = g(\\theta^{(j-1)}x^{(j-1)})$$\n", "\n", "Our neural network is shown in the image below. Since the images are of size 20×20, this gives us 400 input layer units (excluding the extra bias unit which always outputs +1)\n", "\n", "
\n", " \n", "
\n", "\n", "In this part of the exercise, you will implement a neural network to recognize handwritten digits using the same training set as before. The neural network will be able to represent complex models that form non-linear hypotheses. For this week, you will be using parameters from a neural network that we have already trained. Your goal is to implement the feedforward propagation algorithm to use our weights for prediction. In next week’s exercise, you will write the backpropagation algorithm for learning the neural network parameters." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((25, 401), (10, 26))" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# load the pretrained weights\n", "theta = sio.loadmat(\"ex3weights.mat\")\n", "theta_1 = theta['Theta1']\n", "theta_2 = theta['Theta2']\n", "theta_1.shape, theta_2.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Feedforward propogation\n", "\n", "Feedforward propogation is the routing of the data throught the neueral network to get a prediction. In multi-class classificiation, the output layer of the network exists of multiple neurons. The output of the network, therefore, is a vector consisisting of probalities for each class. Remember the formula for forwarding data described above.\n", "\n", "**Exercise**: Implement the feedforward computation that computes $h_\\theta(x^{(i)})$ for every example i and returns the associated predictions. Similar to the one-vs-all classification strategy, the prediction from the neural network will be the label that has the largest output $(h_\\theta(x))_k$." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "def add_bias(X):\n", " bias = np.ones((m,1))\n", " X = np.append(bias, X, axis=1)\n", " return X" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "def forward(theta, X):\n", " return X" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "layer2_activation = add_bias(forward(theta_1, X))\n", "predictions = forward(theta_2, layer2_activation)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Evaluation\n", "According to the exercise, we should have a 97,52% accuracy over the entire training set using neural networks." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Training set accuracy using neural networks: 0.96%'" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Make sure to add 1 to the result as `y` is one indexed while `predictions` is 0 indexed.\n", "accuracy = np.mean(np.argmax(sigmoid(predictions), axis = 1) + 1 == y) * 100\n", "'Training set accuracy using neural networks: {:2}%'.format(accuracy)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's make a couple of predictions using the neural network. Most predictions should be correct." ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "'The neural network predicts 110 and the correct answer is 7. This means that it got the answer wrong.'" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "i = random.randint(0, m)\n", "prediction = np.argmax(sigmoid(predictions[i])) + 1\n", "answer = y[i]\n", "\n", "'The neural network predicts {} and the correct answer is {}. This means that it got the answer {}.' \\\n", ".format(prediction, answer, 'right' if prediction == answer else 'wrong')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: ex4/PE4 - Learning Neural Networks (Exercises).ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Neural Networks Learning\n", "\n", "Stanford CS229 - Machine Learning by Andrew Ng. Programming exercise 4.\n", "\n", "Please check out [the repository on GitHub](https://github.com/rickwierenga/CS229-Python/). If you spot any mistakes or inconcistencies, please create an issue. For questions you can find me on Twitter: [@rickwierenga](https://twitter.com/rickwierenga). Starring the project on GitHub means a ton to me!" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pylab as plt\n", "from scipy.optimize import minimize\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Neural networks\n", "\n", "---\n", "In the previous exercise, you implemented feedforward propagation for neural networks and used it to predict handwritten digits with the weights we provided. In this exercise, you will implement the backpropagation algorithm to learn the parameters for the neural network.\n", "\n", "Load the data and view some samples in the same way as [ex3](https://github.com/rickwierenga/CS229-Python/tree/master/ex3).\n", "\n", "Remember the output of a neural network: $h_\\Theta(x) \\in \\mathbb{R}^K$. We want y to be a 2 dimensional vector in the form that are network should output. For example, we would represent the output 1 as:\n", "\n", "$\\begin{bmatrix}0\\\\1\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\\\0\\end{bmatrix}$" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "def remap(y, K):\n", " m = len(y)\n", " out = np.zeros((m, K))\n", " for index in range(m):\n", " out[index][y[index] - 1] = 1\n", " return out" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import scipy.io as sio\n", "\n", "# Load data\n", "data = sio.loadmat(\"ex4data1.mat\")\n", "X = data[\"X\"]\n", "y = data[\"y\"]\n", "y = y.reshape(len(y))\n", "\n", "# Initialize some useful variables\n", "m, n = X.shape\n", "input_layer_size = 400\n", "hidden_layer_size = 25\n", "K = 10 # number of classes / output_layer_size\n", "\n", "# remap y\n", "mapped_y = remap(y, K)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQUAAAEICAYAAABWCOFPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzsXXdYFFf3fu/Sm0pVQewVeyzYY2yxp5hiYsMWNYlGsSZRPzVGExM1tlhiFHuLvQJ2jaCiAooKIiIgHRZYFhYE3t8fu+wPlLIFo/m+fZ/nPLAzd86cuTNz5pZz3yNIwgADDDCgEJLXbYABBhjwZsHgFAwwwIBiMDgFAwwwoBgMTsEAAwwoBoNTMMAAA4rB4BQMMMCAYjA4hTcUQggPIcTVV6B3sRAiWQgRX9G6dYEQYoEQYucr0v1K6vC/Hf+TTkEIESmEyBZCZBaRta/brlcNIURNANMBuJGs9hrO310IEfNPn1cTvErn9DrOow+MX7cBrxGDSJ593Ub8w6gJIIVkYkk7hRDGJPP+YZsMeMPwP9lSKAtCiPVCiINFfv8shDgnlLAVQpwQQiQJIaSq/2sUKXtR1Ty/pmp9HBdC2AshdgkhMoQQN4UQtYuUpxBiihAiQtWk/0UIUeI9EUI0FkL4CiFShRChQohPiuzrL4S4L4SQCSGeCSFmlHB8LwC+AJxVtnkJIWqrbBgrhIgCcF5VdrAQIkQIkaa6piZF9EQKIWYKIYKFEHIhxJ9CiKpCiNOq858VQtiWcH4rAKeLnD9TCOGs2m0qhNiuOj5ECNG2yHHOQoiDqjp/IoSYUsa9sxdCHFPV9Q0A9V7Yv0oIEa3af0sI0VW1vS+A7wB8qrIrSLV9tBDigcquCCHEhCK6HFT3P011T64U3rvSbC7tPG8cSP7PCYBIAL1K2WcJIAyAB4CuAJIB1FDtswcwRFXGBsABAEeKHHsRQDiUD2NlAPdVunpB2SrbDmBrkfIEcAGAHZRf8TAA41T7PABcVf1vBSAawGiVntYqu9xU++MAdFX9bwvgrVKurTuAmCK/a6ts2K46hwWAhgDkAHoDMAEwS3VNpkXqzh9AVQAuABIB3FbZZA6lY/mPJudXbVsAQAGgPwAjAEsB+Kv2SQDcAjAfgCmAugAiALxbiv69AParrqUZgGeFdajaP1x1D42h7EbFAzAvYsfOF/QNUN1LAeBtAFmFdauyc4OqjkxUz4ooz+aSzvOmyWs34LVctPLBzgSQVkTGF9nvDiAVwFMAn5WhpxUAaZHfFwF8X+T3cgCni/weBCCwyG8C6Fvk95cAzqn+98D/O4VPAVx54dwbC18+AFEAJgCoVM51F3sp8f9OoW6RbfMA7C/yW6J6uboXqbthRfYfBLC+yO/JKOIoyzq/atsCAGeL/HYDkF3kPkS9UP5bFHGsRbYbAXgOoHGRbUtQxCmUcIwUQMsidpT5sgI4AuAb1f+LABwFUP+FMmXa/G9wCv/L3Yf3SVYpIn8U7iB5HUrvLqD88gAAhBCWQoiNQoinQogMAJcBVBFCGBXRm1Dk/+wSflu/YEd0kf+fAnDGy6gFwF3VVE0TQqQBGAagcLBwCJRf2qdCiEtCiI7lX36pNjir7AAAkCxQ7XcpUkbbaywPRWdCsgCYCyGMobxu5xeu+zsoWykvwhHKFsCL9amGEGKGqjuQrtJVGYBDaUYJIfoJIfxV3YM0KOu4sPwvULagfFRdizmq7drY/Ebif3mgsVQIIb4CYAYgFsrm81LVrukAGgFwJxkvhGgF4A6UzkNXuAIIUf1fU3XOFxEN4BLJ3iUpIHkTwHtCCBMAX0PpyFy1sKHoUtlYAM0LfwghhErXMy30aXIeTRAN4AnJBhqUTQKQB6WtD1XbahbuVI0fzALQE0AIyQIhhBT/f++K2SaEMIOyFTQSwFGSz4UQRwrLk5RB+TxMF0I0A3BeCHFTA5vf+GXJ/8sthRIhhGgIYDGU/c8RAGapXn5AOY6QDSBNCGEH4D8VcMqZQjmA6QrgGwD7SihzAkBDIcQIIYSJStoJIZoIIUyFEMOEEJVJPgeQAaBAD3v2AxgghOipcjLTAeQAuKaHzkIkALAXQlTWsPwNADIhxGwhhIUQwkgI0UwI0e7FgiTzARwCsEDVonMDMKpIERsonUYSAGMhxHwAlV6wrbb4/4FeUyg/DEkA8oQQ/QD0KSwshBgohKivcprpAPKhrPfybH7xPG8c3ljD/gEcF8XjFA6rmqw7AfxMMojkIyibfjtUX47foByMS4ZysO1MBdhxFMqBqUAAJwH8+WIB1VepD4ChUH7J4wH8DOVDCyidV6SqSzMRyq6FTiAZCqVDXAPldQ6Ccvo2V1edRXQ/BLAHQISqaV1SV6lo+XwAA6Ecu3mismczlM3+kvA1lF2XeABeALYW2ecN5f0Kg7JboUDxrsYB1d8UIcRtVZ1PgdJJSgF8DuBYkfINAJyFcmzKD8DvJC9oYHOx85R1/a8LQjX4YcBrgBCCABqQDH/dthhgQCH+l1sKBhhgQAkwOAUDDDCgGF6ZUxBC9BXKyLvwItM1BhQBSWHoOhjwpuGVjCmo5u3DoIyKiwFwE8ogoPsVfjIDDDCgQvGq4hTaAwgnGQEAQoi9AN6DMuz3JagG3AwwwIBXi2SSjuUVelVOwQXFp3tioAz/VEMI8QWAL17R+XVC7dq14ebmBlNTU/j5+SEhIaH8g8pA5crKWajRo0fD19cXISEh5RxhgAGvFE/LL/IaBxpJbiLZlmTb8kuXjmrVqsHc3FwvW4QQmD59Oo4ePYqBAwfizp07kEqleukEABcXF7i4uGDq1KnYvn07XF21CTIsHSYmJnB0dET9+vXVYmlpWSG69YGRkRGMjIxga2sLZUzPvxfm5uaoVasWXFxcyi/8mmFsbIzatWvDycmpYvRViJaX8QzFw2xroGLCZIvByckJ586dw7Rp0+Dj46Oznm7dumHy5MkYO3Yszp07V2H2ZWRkAABiYmLQpk0bfP/99/jmm2+Qk5OjlR4hBKpWrYo6deqgQYMG6NOnD5o1a4Zq1ZRLHxwcHPDpp5/i4MGD5Wh6tTAzU8ZSTZo0CXfu3MHp06f/0fObmJigZs2aaN68OYQQOHZMGWuUn59f5nESiQR169ZFy5Yt1dtat26NadOmQSqVYuPGjVi9ejXS09M1tkUIgaZNm6Jr166Fi6IgkUjUtgQHB+PGjRvl2lYeTExMMH/+fEybNg1Pnz7Fli1bAACrVq1CXp6O1BivYpUVlM4mAkAdKMNFgwA0LaM8dRFPT0/m5eWxSZMmOh0PgGZmZvTy8uLFixdpbW2ts56SRCi5Cvjtt99SKpUyPT2dGzZsoJGRkcY66tSpwx07dvDGjRt89OgRHzx4wJUrV3LIkCHs1q0bu3XrxmXLlulVBwBoZGTEUaNG8fjx47xw4QJ//vln2tvb097eXmMdVapUYZUqVfjXX3/xxIkTFVqX5Z138uTJPHz4MO/cuUO5XM6zZ89qfLybmxtv3bpFhULBiIgIRkRE8OjRozx9+jQfP37MuLg4NmvWTCub+vbty7t37zIvL49yuZxSqZQhISFMTU1lamoqo6Oj+eGHH+p13Y6OjtywYQNlMhkVCgXlcjnT09OZnp7O/v37l3RMgEbv76tafgnlirIwAI9RZDlxRTmFatWq8datW5wxY4ZeFdu9e3dGRERw8ODBr/TBvXDhAmUyGVNSUvjWW29pdIyRkRGPHz/OW7ducfz48XzrrbdYq1atl8qZmJjoZVvr1q156tQpZmZm0tfXl9u3b2d+fj4nTJjACRMmaKzHzMyMZmZm3L9/P2/dulVqOUtLS7Zr104vm42NjVmnTh3OnTuXDx48oFwuJ0kmJCTw6tWrnDRpksa6atWqxb/++osHDhxg79692bt3b5qbm7Ny5cr08fFhbGwsmzZtqpV9S5cuJUnOnz+flpaWNDMzoxCCzZs3Z/PmzRkXF8fvv/+eAGhqaqr19derV4/Hjh0jSQYHB3P27NkcNWoUY2JiGBMTw5UrV5Z0nEZO4ZWtkiR5CsCpV6XfAAMMeEV4VS0FLVsVWnvKP//8kzdv3tS7ye/p6cmgoCBaWVnppac82bx5M2UyGQMDA1mtWjWNjjExMeGDBw84atSoV2ZXhw4dGBYWxgsXLnDw4MGUSCS0trZmXFwcZ8yYoVVLrLC7cfbsWV64cKHUcl27duX27dt1trl9+/bcv38/79y5w6ysLMpkMv79999csmQJe/fuTVtbW630SSQSuri40MzMrNj2n3/+mdnZ2bx37x4dHR210tmlSxcmJCTw/PnzbNiwIQFly++nn37iTz/9xEuXLrFFixb84IMPuGLFCq2ev6pVq/LIkSNMTU2lXC7n2rVrCShbYP7+/vT39+fly5dfuh687u7Dq3QKY8eOZXp6Ovv166fzg+Xi4kIXFxfeu3ePZ86coUQieWUvXuPGjXn37l2S5LRp07Sy8enTpxwxYkSF2iOEoL29PU1NTXnt2jVu2rSpWBO2X79+TE9PZ5cuXdilSxeN9bZs2ZItW7ZkeHg4x4wZU2q5b7/9llFRUezZsyd79uyptf0rVqzg8+fPeeXKFc6cOZNvv/221o6gPJk4cSKzs7MZEhLCzp0766Tjm2++YU5ODi9dusTq1avT3t6e48eP5/jx41mzZk0C4Pz58ymXy9W/yxNzc3Pu2bOHqampnDhxIidNmsRDhw7RxcWFAOjj40MfHx8GBQWxevXqLx7/3+kUBg8ezKioKM6aNUuvm96pUyd26tSJJMt8gHUVS0tLDhkyhEOGDKG/vz/lcjl9fX1LHBMoSSQSCTdu3EiSPHr0KMeNG8ehQ4eyV69e/PTTT9m9e3e6urrS1dVVZxuHDx/OR48e0dnZWb3NzMyM//nPfzh69GgaGRlpNSi6ZcsWbtmyhXfu3KGTk1Op5fbs2cP8/HzOnj2bs2fP1lh/YX2GhYVx/PjxFX7PmjVrxoULF3LhwoVMSUnh/fv32b17d710fvXVV0xLS+Pu3btLbNWuWrWKSUlJrFGjhkb65s2bx7y8PC5fvly9zcLCggBoZ2fHGzdu8MaNGwwMDNTZKfyrmJdcXV2xY8cOXLlyBcuWLdNLl0SiDNGIjo6Gv78/AOU8u0QiwfPnz3XWa2lpCQ8PD/Tp0wddunQBAFSpUgWPHz/G5MmT8fSpRvEjsLGxgYuLC8LCwmBqaophw4bByckJEokEcrkcRkZGKChQcqlcvnwZ69atQ3i4dssoXF1doVAoEBv7/2RP9vb2SE9Px9WrV7WaLvvqq6/w4YcfAgCWLl2KxMQSWeRRs2ZN1K1bFxKJRKvYCkdHR/zwww8AgIsXL+KPP/4o5wjtIJFIMH36dIwaNQoAoFAoYG5ujqpV9WNRW7duHYyNjbF48WIsXLgQ3377LQCgYcOGmDBhAkaOHImzZ88iJSWlXF29evXC1KlTcfPmTaxYsUK9PTs7GwBgZ2cHGxsbAEB4eHip96A8/Gucgr29PY4dO4aUlBR4enrqrc/YWHnpYWFhUCgU+PDDD/Hll1/C0dERu3fvxp9//onk5GSNddnZ2aFly5YYOnQohgwZAhMTE8jlcgDA6dOnsXTpUjx8+LAcTf+P9PR0jBgxArm5uVAoFAAAa2tr2NnZIS4uDvb29ujQoQMA4PPPP8fJkycxevRoXLumOUFSQkIC6tati/nz5+PQoUNo2LAhfvjhB7i5uYEkVq1apZGeWrVqYfTo0WrnumnTplLLVq1aFba2toiJicHly5c1trVmzZqws7MDALz11lto27YtAgICND6+PBQUFODYsWNqp00SX3/9NXr27Injx48jKytLZ92rVq2Cra0tvvvuO9SqVQsA0LJlS9StWxfe3t744Ycf1C92Wfjqq68AAHPmzMGzZy+H/djb28PCwgIAcOHCBd1jIF5310GT7oOxsTH37dvHzMxMuru7V0hTsXPnzuzcuTNjY2N55coVpqSk8NChQ/zpp5/4+PFjjhs3TiM9Dg4O/PXXX3nhwgVGREQwMzOTKSkpTExM5Hvvvcf33nvvpWM+/PBDrlq1ig4ODhVyLRYWFpw/fz4vXrxYZrP9RalcuTJnzpzJ9PR0xsXF0cvLi2fPnqVCoSjR7tJk3rx5TEtLU3fJiu6zs7NjixYt+NFHH3HJkiU8c+YMpVJpseavJlKpUiV6enrS09OTqampvHXrFmvXrl0h9VeStGrVis+ePaOfn19JzXCNxdXVlR4eHvT29mZOTg4VCgUVCgUvXrxIDw8PddO/PGnbti0zMjK4c+fOUrt0q1atYnR0NKOjo9m8efOSyvz3dB+mT5+OTz75BMOGDcP169crRGdh07tatWowMjLC9OnT4eXlBSsrK3z88cdo0qRJORqUTfwNGzagX79+KCgowPPnz5GbmwupVIorV66gZk0lb+jMmTNBEnZ2dujcuTNatmwJf39/dQtAX2RnZ2PdunXw8PDAoEGD8OefLzG6lYj09HT88ssv2LdvHyQSCUhix44dOHv2rDoaUBMIIWBhYYHhw4cDAHr37g0rKyu0bt0a1atXh0QigRAC0dHRePz4MRo0aIAHDx5odY0ZGRnqJvPz58/xn//8B2PGjMGCBQvU91JTVKpUCXl5eSV+/QtbI8OGDYOxsTGCgoJ0uk/9+vXDtGnT4ObmBgCQSqXw9fVF69atAQBbt27Ftm3bNNY3bNgwGBkZ4dSpUyW2APr16wcPDw8cOKBke7t7967WNqvxulsJ5bUU6tevT7lcznnz5lXol6BNmzZs06YNFQoFw8PD2bRpU7q6unLBggVMS0srLSKsmDg6OvLSpUvMyMigXC6nTCZjWloa09LSmJKSwoyMDLXI5XJmZ2czLy+Ply9fZt26dcvUrekXpKj8/vvvXL16tc51MmXKFMpkMq0Di5o1a8Zly5bxyJEjPHLkCP38/Hj58mVu376d8+fP56effsqWLVsSUA7wRkRE6DVQaGxsTG9vb968eVPrqUIhBJcvX66exntRpk6dyqlTpzItLY2XLl3SeiC3Zs2a3LhxI2UyGePj47l161Z269aNAPjRRx8xPj6e8fHx/PTTTzXS5+DgQAcHBwYEBDAkJKTE1lH9+vV5/fp1hoaGslGjRmzUqFFp+v7dsw8SiYQSiYRbtmzh0aNHS5pz1UtMTU1pamrKRYsWMSMjg0+ePGFoaChTUlI4adIkjc9XtWpVdu/enRMmTODChQt58OBBnjx5knFxceoHICkpib6+vlyzZg1nzpyp0QzERx99xJ9++onNmjXTKOLNysqKf//9N+fOnatTfdSsWZOJiYlct26dznVqZWVFKysrurq6snr16lQtiS8mw4YNY3JyMlu0aKGRThMTExobG7+0fdmyZbx+/XqxmRNNZfHixZTJZFy1ahVbt25NiURCW1tb9u/fn2FhYQwLC2NaWhq//fZbrXUXTpdu27aNrVq1Um/v3LkzHz9+zNu3b/P27dvqKcTyxMnJiU5OTrx9+zYfPHjA+vXrq/fZ2Nhw8uTJDAoKYmJioiZT1/9up+Do6EhHR0eGhITw0KFDGocG6yK9evXiyZMn+ddff7Fv37566SqcxrOwsGDXrl3ZtWtX9unTh+bm5lrpMTEx4fjx43nu3DleunSJ48ePZ4cOHdiuXTu6u7uzZcuWbNWqFVu1asVx48bx2rVrTEtL09n+VatWMSIigvXq1Xtl9QwoW2j37t1jhw4dNCo/btw4ent7s1evXrS2tlZP623ZsoXh4eE6PRetWrXiuXPnSJJRUVH08fGhv78/ZTIZ4+Li1IFb2n6InJ2dGRgYyCtXrtDFxYXm5uZs2rQpJ0+ezLi4ON6+fZtvv/023377ba1t3rRpExUKBXfv3s158+Zxw4YNjIyMZE5ODr29vTVt3f27nUJhHP2vv/7KzMxMDh069JU+rG+qVK9enVOmTKGPjw+vXbvGx48fMzQ0lPHx8UxISGBCQgIjIyO5Y8cO9uvXT6u4gkJ5++23KZVK6enp+cqvx8rKihs2bOCGDRs0Kt+iRQvGxsYyNjaWJ0+e5MmTJ7lnzx7ev3+fnp6eOnWzAOV6jxMnTjAkJIShoaHMycnhlStX2KNHD/bo0UMnnQ4ODrx+/TqfPXvGU6dO0dvbm3FxcUxPT+f+/fu1Xj9RVOrXr8/Lly8zKyuLubm5fPToEU+fPs0pU6ZoE7ilkVMwELcaYIABxfBG5H0oi47NzMwMtra2iI+PL63I/wSsrKxQqVIlmJubo6CgAGZmZoWtLMjlciQlJekUdGVtbY2DBw8iMzMTQ4cO1StwS1NYWVmhcuXKxYKmysLHH38MT09PdWCOEAILFy7E/v37yzmybNjY2KgDqKysrCCVSvUm13Fzc8OYMWPQtWtXPH/+HOfOncOBAwcQFhaG3Fz98unY2tqicuXKEEIgKysLcrkcmZmZ2qi4RQ1Ijd54p2DAq0W1atUwduxYbN68WW/6OQPeeBicggEGGFAMGjkFw5iCAQYYUAwGp2CAAQYUg8Ep/MtgZGQENzc3rF27FhcvXsTFixfRoEGD123Wfx2srKzg6ur6xrJSCyFga2sLe3t72Nvbw9TUtMJ067z2QQjhCmA7gKpQzoFuIrlKCLEAwHgASaqi31FJzfZGo5DR18LCAo0bN4aDgwMKCgpQUFCAa9euaZyzwc7ODv369YOTk5N6duD58+e4ceMGbt26pXWcflFYWlpi5syZ+Oyzz9Sr7QCgf//+Gq9oLA/Ozs5o2rQp7OzscOfOHQDKlaTlQSKRwNFRmWektAFLiUSCwYMH4+7du3j8+LFWdllaWmLAgAEAAF9fX6SlpRXTq0+9FkVhvS5duhQSiQRDhw7VW1+PHj1gZmaGgoICBAUFAQACAwO1ZvUuhKWlJWbNmoXu3bvDxMQEQgiEhYUhODgYe/bsAQDExcXpbrQeAUfVAbyl+t8GSpJWNwALAMzQN3ipLKloliRbW1t+++23vHfvHkNDQ5mamkqZTKYmA/X29i5Xh6mpKUeNGsXbt2+TJHNycpiZmcnMzEzm5OQwLi6OW7du1TpWv1CaNGnC3377jVKplFlZWUxPT1evuNu9e7fOYeCWlpbs1q0bZ82axV27djEgIIDx8fGUyWTcvn07t2/frlF9CyHUbM6llXF0dKRCoeCQIUO0trN169ZqUtJ333232L7x48frpLOouLi40NPTk8HBwQwODmZBQQE3b95cYqi2Jvdq7ty53LFjBwMDA5mbm0uSzM/PZ2RkpDrYTFPCnZLqMSQkhPn5+VQoFMzJyWFeXh6zs7PVAV5Vq1Yt6dhXu0qSZByAONX/MiHEAygzQ+kNGxsb1K5dG4CSf9/NzU1NipKfn48GDRogNDQUq1ev1nsarWrVqti4cSN69+6NyMhI+Pn54dSpU3j48CEWLVqEfv36qfMrlAVLS0v06dMHUVFRWLJkCfz8/NRfs44dO+LIkSMYMmQIVq5ciaSkpHK0FUenTp2wbt06NGjQAPn5+Wq9hU3bwhV5y5Yt0/iL2bRpUwwaNAgDBgxA69atIYTAs2fPkJiYiJMnT6J79+546623AChjGQpzWJQGksW+3iWhdu3aiIiI0GkFX8+ePdV5JLy9vYvta9++Pe7du6e1TkB538aOHYspU6agUqVK6hbMlStXMHjwYNja2mLChAkacWvUrVsXnp6eGDRokDqGIDg4GJs3bwagfHa7d+8OQLnq8cGDB1iyZInWNiclJWHRokUYOXIkACA1NRWOjo5o27YtevXqBUDZ0vnyyy91W4lbQWHKtQFEAagEZUshEkAwgC0AbEs55gsAASpRezNra2uuXr1avS48LS2NcXFx3L17N/fu3ctdu3bx6tWrlMlk/O677/RuJcybN4/Z2dk8fPgwGzdurN7u6elJmUzG1NRUTp48WSNd9vb2xb6qLVq0YIsWLejr68v09HTOnDlTa6JZd3d3BgcHU6FQMDMzk6mpqRw4cCD79evHc+fO8dy5c1QoFHzw4AH/85//0MbGplyd7733Hu/du8fc3FwmJydz3bp1HD9+PJs3b05LS0sOGzaMCQkJ3LRpEzdt2qRT6HRJsm3bNvr6+pa4wKk8WbFiBZcvX/4SD4O1tTVDQkL45Zdf6vzV9ff356lTpzhw4EBWqlSJlSpVopWVFT09PZmfn68RXbwQgrNnz+ahQ4f41Vdf0c3NrcScGYXErQqFQuOVkqWJmZmZ+n7b2tpy06ZN6lW5T548KYlb8p9Z+wDAGsAtAB+qflcFYATlIOaPALZo031o27YtHz16xEIsXryYVatWpampqbqJ7OHhwYyMDB45coSVK1fWqUJr1KjBGjVq8P79+zx58iQrVaqk3ieRSLhv3z6mpKRwyZIlWj/Ezs7OnDlzJqOiohgVFcWQkBCdSGbNzc3p5eXF7Oxspqen08fHh71791bvb9q0KZs2bcqQkBDm5eUxLCxMvUS5LNm0aRNDQkI4ffp0Nm3atJgjGzduHKVSKY8dO6Ymt9XnwS2UJk2aMCsrS6cl8Obm5jx+/Dg/+eQTfvLJJ8X22dnZMSoqil988YX65dRGt0QiYbVq1UpkUy7k8NQ0aYuVlVWpC9+qV6/O77//ntnZ2czOzuahQ4d0fnZLk8aNG6u7P3FxcSWt4Xj1ax+EECYADgLYRfIQAJBMIJlPsgDAH1BmoDbAAAP+LdCjhSCgnH347YXt1Yv8Pw3AXm1aCu7u7oyIiGBycjKTk5NLXArcrFkzRkdH8/jx4zp728Kl2X5+fvTz81N/KVxdXblv3z7GxMTotDKzQ4cOvH37NuPi4rhs2TIuW7ZMp8FFKysr/vjjj5TJZMzMzGRMTExpFFucMmUKk5KSGBMToxGNmqur60u5J6ytrfndd98xOjqahw8frvAl1F5eXoyNjdWJir1Ro0Z89uwZe/XqxV69ehXb5+DgwOjoaPr4+HDjxo3F+Ab0kZYtW/Lo0aMMDw8vi7SkXHFzc+Ps2bMZFhbG7OxsnjhxgidOnNCLhbs0qVSpEq9evcqrV68yMjKSbdq0ebHMq+0+AOiiOlEwgECV9AewA8Bd1fZjKOIkNHEKDRs2pK+vLxctWsRFixaVSjASHBxMPz8/jal8WOToAAAgAElEQVSxXxRjY2MaGxtz06ZNzM3N5ZgxY1itWjWePHmSGRkZnDJlik56T548SW9vb3UCEF2lsBsll8uZlJTE6dOnl1rWycmJfn5+LCgo4C+//KL1uTp27MitW7cyKSmJMpmswpepd+zYkQUFBfz88891Or5WrVp89OgRvb296e3tzbZt27J9+/ZcunQpL126RJKMiIjgvHnzinUDtX0ezMzMOGnSJE6aNImhoaGMiIhQsybpIp06dVLPZISHh3POnDnqj1FF1m+hTJkyhVlZWczKyqKXl1dJ3d5/L59C1apV1cxLJV28kZERb9++TYVCoRNhRVH55JNPmJaWxoiICIaGhjI6OpqDBg3SSVfDhg2ZlJSkd04KAJwxYwalUilzcnL4008/lVnW3t5e/XKsXLlS435127ZtuWvXLiYmJvL58+dMSkpiRkYG4+PjOXnyZHWCXH2uw8jIiHv37uWVK1f0ynk5YsQIPnz4kA8fPmRaWhrj4+Pp5+fH2NhYxsXFsU6dOjrprVSpEn/88Uf+/ffffPDgAfPy8tTjM25ubnpd+5AhQxgZGcns7GwGBwfrlUPCyMiITZo0Yc+ePTlixAgOHz6cb7/9Nhs1akQLCwu2adOGjx8/VieDKYXU9t9L3FreNKPKkUAIoXfEWXx8PNLT01GtWjU8fPgQs2bNwvHjx3XSFRsbCz8/P3z33XdwcHDA+fPnAQABAQEa08UXwtzcHBKJBEZGRuVSw9vb26Ny5cooKCjQuD4qV66M+fPnY9CgQQgICMCaNWsQGBgIV1dXTJkyBaNGjVLXQ2RkpFa2F8XIkSPx6aef4v3339drWXYhoSygpHjPzMzE9evXsXr1avTq1QtPnjzRSa+RkRF69OgBKysrHD58WJ3noXfv3qhSpYrO9gLAoUOH8PjxY3z88cf4+uuvsXPnTjWp7qJFizSiYLeysgIATJ06FSNGjECjRo0glUohhIClpSXCwsIQFhaGevXqwczMDJMnTwag3z177a2EkloK5Unnzp0ZGRnJ3NxcnVlyGjZsyIYNG/LIkSPMyspiaGgoO3bsqNeXAVCOVezdu5fJycmUSqWUSqWMiYnhxo0b2alTJ40Drzp37sxHjx4xJyeHhw8fVgejFH69hRDqazhw4ADlcjljYmI0DuKpUaMGN27cyLlz577UnF29ejWDgoJYv359vfrodevWZWxsLHfu3Kl3vZYkZmZm9PHx4d27d/XS4+TkpKbbr1mzJmvWrMm7d+9WWBYqIQTd3d25c+dOdUvkl19+KZd7s0qVKjx8+DAPHz6snmrctWsXP//8c/7000/Mzc2lXC6nXC5nfn4+09PT1cFLnp6ebNWq1YstvX9vS6E81K5dGzY2NoiOjtb6C+zk5ISxY8di2LBhal0kERQUVCHJRZKSkvD555+jQYMGaNtWuUrV3d0d48ePx+DBg/Huu+8iODi4XD15eXkwMjJCbm4u3n33XezevRuBgYHqIC5TU1N89NFHAJSU5cbGxggODoavr69GdsbExGDChAkvbTc1NUXdunWRkpJSbjBSefD09ISZmRkWLlyol57SUBjerE0CnEJ07NgROTk5uH37drFMSoWkK9bW1jqRolhbW+OTTz7B9evX1aHxJHH9+nVcv35dTeIyfPhwREVFYc2aNaXqsre3V4d2KxQKJCcnQyKRYOTIkahduzbS09Oxe/du3Lx5E/Xr11enFwCANm3aoFGjRpg7d67WwXKvvZWgS0th9OjROsUpODg48MCBA+ow4UKJjIzUKpGqtlLI3RcfH1/SiHCJUrlyZW7dupVZWVmUy+XMzMxkdna2Oqw1Ozu7GH38sWPH2LZtW73sNDY25sKFCxkXF8fRo0frpatz585MT0/nyJEjX1m9WlhY8MyZM9y4caPW9+PGjRs8fPhwse0SiYTr1q3junXrmJCQUCyYTVP55JNPGBISUupMUceOHdmxY0dmZGTQy8urzDGbmjVrqhnB09LSKJVKmZuby7CwMJ44cYLffvttmeM0lStXfnH/f2dLwdzcHO+88w7MzMzw8OFDyGQyjY6rVKkSfvvtNwwYMAA5OTnqvndBQQGOHz+Oq1ev6mWXEEI91mFmZgZnZ2cMGTIEAPD111/DysoKM2bMwK1btzTSl56ejqlTp0KhUKBDhw5o1KgRTExMQBJCCBgbGyMqKgoAsHnzZly6dEnnlo4QAvXq1YOHhwc8PDxw5MgR7Nq1SyddAGBhYYHp06fj8OHD2Llzp856yoMQAkZGRqhXr55Wx6Wnp8Pb2xsTJ07EwoULcebMGVSqVAnjxo1Dt27dAAA//vijVmn+CpGdnQ07OztMmDABhw4dQmhoKIyNjWFkZISqVavixx9/BKBs5Xh5eamfmZIQHR0NDw8PAMq0eRkZGahTpw58fHxw+/btMo8tvE5d8K9zCgqFAkFBQRg8eDDkcrnGsf516tRB+/btIZFIYGJioj5uw4YN+OWXX/SyydjYGDNmzFBnF2rQoAFatmwJJycnAMCuXbuwb98+9cCjpkhPT8e0adPg7OyMrl27wtzcXP0gSCQSdZalS5cu6WS3vb092rZti4EDB6Jbt25o2LAhdu7ciVmzZunFJ/juu++iTZs26NmzZ4WtXiwJWVlZiI+PR9++fWFhYaFRPkZAuWp13rx5KCgowNSpUzFo0CDIZDJkZGRgzJgxAKBeZ6EtTp48iV9//RWenp4YM2YMgoODYWxsDBMTE7i4uMDIyAgAsGLFCty4caNMXSRx5swZnezQB/86pwAol4Xm5eVp5QktLCwQFxeHzMxMSCQSdWV///33uifiVKFwkdKQIUOQk5MDPz8/+Pr6qvv3jx490lm3QqFAREQEIiIi9LIRALp27QoAGDNmDCQSCaysrODo6AhbW1tkZWXB09MTBw4c0Lj1VRoaNGiANWvWaJ0FWxccPXoUNWrUQOXKlTV2CoVYtGgRtmzZAoVCgefPn0OhUOiVSBZQtjyXL1+OY8eO4auvvoK7uzvi4uLw/PlzHD9+XD3+cebMmVfqMPXC6x5P0GVM4d133+WzZ8/0Xi5rEIP8j4kh74MBBhigPf6VbM7m5uZo0aIF7t+/ry3vvQEG/C/DQPFugAEGFIOB4t0AAwzQHm/U7IOZmRlq1KiB3NxcREdH/6PndnJygrW1NWQymfYRYAaUCRsbG9ja2sLExERdv29CC/XfBisrK1SpUgWxsbGvtP7eKKfg4eGBtWvXIjExEfv27QMA3Lx5E0FBQZBKpUhMTESjRo0QGxurcwiuubk53NzcULNmTQBKDkgzMzN8+umnMDU1xZ07d7B06VL8/fffFXZd+kAikaBOnTpIS0tDSkpKmWUdHBxgbGysU95NFxcXNGrUCA0aNEBSUhJiYmLUjjk+Pl7nh7B+/fpYuXIlOnXqBDs7O5w5cwajRo1C7dq14ebmBgD466+/dB4b6tmzJ9zc3JCfn49Dhw7plXPU0tISn376KQAgODhY40CzfwqLFy9Gv3790Llz53KfBb3wuqcji05Jzp49m4mJiUxISGAhEhMTGRoaSm9vb65evZr379/n5s2by2QNLku+/vprxsXFMSUlhSkpKYyOjuaGDRv48ccfs2PHjhw0aBC9vLxKY8PVSgo5G+zt7eno6KgTIcz48eMZHBysEWPzkiVL+Oeff2rE01golpaWXLx4Mf39/fn06VOmp6czJiaG9+7d46VLl3jp0iWdOQVatGjBS5cuMS8vj/v37+fKlSt5+fJljh49Ws0RmZuby7feektr3Q0aNODatWsZFRXFBw8eMCIighs3btRrqbenp6f6uSuLv0Jbsba25owZM7h//3727NmTPXv21FqHiYkJAwICGB0d/RJBjhby7+NTMDMzo729PRs1asQVK1ZwxYoVXLduHdevX881a9bw5s2bjIqKYkJCAu/du8dZs2axZs2aWlXMoEGD6O/vzzlz5nDOnDkvrVSTSCTcvXs3FyxYoPNDULVqVQ4dOpT79u1TszglJCTwyZMn3LBhg8Yvweeff87ExEQqFApKpVIOHDiw1LLt2rVjTEwMpVKpVus4Vq5cyezsbEqlUq5bt47VqlVjy5YtuX79ejWFvC78ksbGxly7di1zc3O5bds2NePSV199xcePHzMxMZG///47f//9d625ECZMmMDk5GTeuHGD/fv3JwD26NGDsbGxOq/Z6N+/P2NjY7l582Zu3rxZK8dakjg7O3PYsGHcs2cPg4KCGBAQwJiYGC5dupRLly7VWp+JiQlv3brFZ8+esXr16rra9e9zCuVJ/fr12aJFC65fv57Z2dnMycnhhg0btK6csr64VapU4a1bt7h//36dvjpWVlY8duwYMzIymJOTw5ycHGZlZTEzM5P+/v68d+8eY2JiOHHixHJfqkOHDjE7O5tyuZzXrl0rdRmzi4sLz58/T5I8depUiSSkJT1kY8aM4YMHD7h06VIOHjxY/SLY2dnR29ubO3bs4I4dO7RmoAaU7MKXLl1iSkqK2gl2796dYWFhlMvl/PHHH2lkZKQxU3SVKlXU9HbZ2dncu3dvMWcikUjo7e3N69eva3T9RaVTp068du0aIyMj2bt372LkuICS/q9v374akcS4ubnx999/Z2BgIIODg7lw4UJ++OGHrFmzJnfs2MGRI0fqtEjMxMSEgYGBPH/+fIn3Y8iQIRw3blyZ5ET4b3QK7u7u3L17N+/cuUOZTMacnByeOnWqwijIAbBXr17Mzc3l2rVrtT7WwsKCs2fPZmpqKm/fvs01a9ZwzZo1HDJkCDt06EAnJyc6OTlxxYoVzMjIKDMic/jw4erVkQEBASU6hEI68lOnTlGhUPDOnTsar5RctWoVw8PDX2IqNjIy4i+//MInT56oV/TpUo9VqlTh+fPnmZiYyLZt23LgwIG8d+8e4+LiuHnzZjo7O2ulb9WqVepVoaNHj37pBTU2NuaJEycYExPDdu3aaazXzs6OFy5cYF5eXomtw7p16/L8+fPcs2dPuaze1atX5++//86tW7dy/PjxdHR0pJmZGVu2bMn9+/fz3LlzrF69uk5fehMTE969e5dbt2596aW3sbFhQEAASfKdd97hO++8U5qef4ziPRJKTsbAwpMCsAPgC+CR6m+JuR+0dQonTpxQk0ncv3+fPj4+HDFihE4PbUnSsGFDXrx4kdHR0VpRcZmYmHDixIk8evQo/fz8OHTo0DK5Ap2dnRkQEEB/f/9Sy0ycOJEFBQWUy+X89ddfSyzTrVs3duvWjbGxsZRKpezatatG9jZq1IiJiYmcMGHCS/uGDh3KhIQEvevVycmJd+7cYVxcHBcuXMj4+HhGRESwT58+Wuvq0aMHo6OjOXXqVE6dOrXEMu+99566RaXNeNPChQuZm5vLJ0+evNTtcnR05PHjxxkTE8Nhw4aVS4pSKDY2Nvz444+5cOFCnj17lvn5+Xz06BGbNGmic32amJgwJCSEXl5eLzmFX3/9Vb2keufOnWWR2vyjTsHhhW3LAMxR/T8HwM8V4RS++OILdSq2y5cvs0WLFno9uEXl3Xff5c2bN5mZmUkPDw+NjxNCcP78+ZTL5dy+fbtG/WMTExPu27ePDx48KPNa8/PzKZfLuXfvXn7zzTd0d3dXP5hubm68cuUKr1y5Qrlczvj4+NJ4+V6SmjVrMjAwkPv37y/Wd65duzavXr3KzZs3l5q/QFMxNzfnxo0bKZVKKZPJmJyczFGjRmmtx8LCgpcvX6a3tzfNzc1LtevPP/9kQUEBhw8frpX+lStXMj8/n4cOHaKTk1OxfYsXL6ZCoaCnp6dWOtu3b8/09HRmZGQwOzububm5jI+P586dOzl+/HiOHz+erq6uWqc/3LVrF2/evFmse1ToDP/44w8OHz5cnUTJwsKiJB2v1SmEQsXiDGXOydCKcApOTk787bffGB0dzZSUFC5cuFDjF6Es6dy5s5oMdNasWVrlZWzfvj2joqL4119/adz3rlOnDgMDA0v8UheKq6srT548yaysLKampjIzM5OPHj3iyZMn+ccff9DPz09NxZWWlsa0tDT6+vryiy++eOnhLkmmTZvGuLg4Hjp0iM2bN6epqSnXr1/Pc+fOVViSkoEDB/LZs2dUKBTcuXOnTsSttWrVYlJSUplkuh988AFlMhlXrFihtf7Vq1czNzeXSUlJPHXqFGfPns3Zs2dz/vz5TElJYVRU1EsJdoyNjcvsslaqVIldunRh1apV2a5dO3722WfctWsXr127pib2SU5O5oEDB7Siu5swYQIzMjLYrVs32tvbc+DAgZRKpYyIiGDDhg3ZtWtXxsbGMjY2lnZ2diXp+McWRBGAjxDilhDiC9W2qlTmmgSAeCizRhWDEOILIUSAEEJ/DjQDDDCg4lABLQUX1V8nAEEAugFIe6GMtCJaCoUyfPhwnj9/nrm5uTx+/LjW05KAMku0qakphw0bxpCQECYmJr6UkkwTGTduHJ8/f86xY8dqfMzixYt55cqVclsWvXv3ZmhoKLOzs5mZmcmMjAxmZmYyKyuLMplM3UKQyWTMyspifn4+/f39NRoPEULws88+47Vr17hz507u3buX9+7dqzBauh49evDKlStqKrGbN2+qyVG1EQcHBz558qTEfBaFeRoSExN58eJFnfIpfPDBB0xKSlIP6ha2vsoa5O3bt69OA7BVqlThgAEDOGDAAB44cIAkefbsWY3rxdnZmYGBgXz27BkDAgL4/PlzxsTEqKnj69Spw6CgIAYFBb2UNEcl//zsA1Rp6KFn96FJkyZqVt3SKqhWrVq8du0ac3NzuXfvXlpaWtLS0lLjG1TYt3v48CEfPXrEMWPG6JTiftKkSSTJcePGaVR+wIABWuUA6NixIzdv3sywsDBmZWUxIyODUqmU6enp6of3+vXrHDt2LGfPnq11IJC1tTXnzp1Lknz8+DE/+uijEhOjaiKF02EjR45keHg4T58+zXfeeYc3btxgSkpKqbyF5cnatWupUCh47NgxHjt2jO+//z6nT59OX19fNYu1LgFQhTYPHz6c8+bN46VLl4o52rCwMHbq1OmlYzw8PDhgwACdzlcoRkZGHDNmDG/cuKGVs2zXrh2PHTtGqVRKHx+fYs7Jzs6Ot2/f5u3bt0uLhXj1TgGAFQCbIv9fA9AXwC8oPtC4TBunsGfPHl64cIEXLlxggwYNSqwcV1dXXr9+nTk5Obx48SIrV66scV94zJgx6gGZo0eP0t3dXeebO3jwYKanp9PLy6u0fhwBZfzCzJkzGR4eXuZYQmnSvHlzTp8+nSdOnGBGRgbj4uJ49OhRHj16lE2bNiVUL7guL8Uff/zBoKAgLlq0iAEBAbx//z6///57Ojs7azV1WBhwFhcXV2x67+zZs5TJZKV9vcoVBwcHfvPNN7x27RqvXbvGmJgYRkRE8Pbt2yTJH374Qa8XFFC2nHbs2KGOLYmMjCw1saytra3OmaiKypw5c3jw4EGdjnV0dHxpXMPBwYEhISEMCQl5iZRWJf+IU6gLZZchCEAIgO9V2+0BnINySvIsADttnMK6devU0XTnzp3jqlWruGbNGq5du5Zr1qzhqlWrePz4caakpDA3N1crpzBgwABKpVL1F+Gbb77RqoXxolhaWnL16tXMzMzk1atXOXPmTDZv3pxubm5s164d27Vrx+nTp9Pb25thYWEapTUvS6ZMmcLMzEympKRwypQpOqe3K5TatWvzxo0b6rTo7u7u9PHxoVwuV4c5lzcT4eLiwp9//lk9yDVv3jza29vznXfe4eLFi5mSksLLly+zbt26etlaGOfRrFkzNm/enLt372ZMTAxr1aql9ws6bNgwSqVSdS5GXfOJaCqjRo3i/fv3S0oXr7NYWVmp7Q8KCirJcf17g5caN26sDlRRKBTqacjs7GwWFBSoJSkpiefOnePo0aNpZmam0azBu+++y5SUFBYiKyuLPj4+3Lx5M7du3coJEyZoPR1nZWXF6dOn88yZM4yOjmZUVBQjIyN58+ZN3rx5k5cvX+aiRYsqZD3FsmXLmJiYyEWLFqmDl/TR17t3b3UkH6AcaymciitsrpcXtNOhQ4diSYHv3LnDO3fuMCUlhQqFgjdu3NArJ2NJMmnSJCoUCvr4+OgdkiyEUK+jaNq0qbrlVdHi5OTEL7/8kl9++SXDwsJ0itkoT1q2bMmWLVtSoVDwhx9+ePFZ/vdSvCcmJqoZlrt164ZOnToBUBK27t+/H/n5+TA2NkZ4eDhOnz6NmJgYjXVfvHgRY8eORatWrQAAJNGkSRO4uLjAwsICjRs3xtWrV9WJPDSBXC7H8uXLsXbtWri7u8PGxgb5+fl4+vQpAGU6OV3ptouiTp066NmzJ1JTU3H8+HFkZGTorTM5ORmWlpZYvXo1Ll26hGrVqqFbt244d+4cPD09ASgT05QFExMTGBkZqVOc2dvbw8jICKGhoZDJZPj5559x+fJlvW0thLu7O2bOnIlnz57h559/1ptotkaNGujTpw+8vLy0uu+awMzMDO+88w66deuGvn37wthY+crNmTMHPj4+FXouAAgKCgIArFmzBrNnz0ZqaipWrlyplY43nnmpcuXKsLe3BwDk5OTg2bNnFX5+MzMzNTe/qakpZDIZcnJyKvw8+sLKygo9e/ZEcnIysrOzcefOHb11SiQSDBw4EFOmTIGzszMyMzPh5eWFPXv2qLMNlQdTU1NUq1YNJiYmAKDOq5GVlYW8vDxkZmbqzZhdFOPGjcPYsWMxe/bsCnE2bdu2xbx58+Dp6YnHjx9XgIVA+/btMXr0aLi4uKB27dqQSqVIS0vD4sWLASgpAV4lCpfcy+XyosvJDXRsBhjwulCpUiW8//77ePjwIRISEpCUlKQ3fXwFwOAUDDDAgGIwcDQaYIAB2sPgFAwwwIBiMDgFAwwwoBjeyCnJ0mBsbAwHBwdYWVkhMzMTCQkJFarf1tYWNjY2yMzMRGpqql66jI2N1Qlmk5KS8Pz584ow8ZXD2toaxsbGOhPjvi5YWlrC1NS0Qu12dnYGoJwKfxPG3v4xvO7ApZKCl0qTnj178sGDB8zNzeXdu3fZvn17vYM9hBB0dnbmuHHj6OfnR5lMxoCAAJ1CkYtK/fr1+eDBAz548ECnhVavQxo3bswDBw7w2rVr7NOnj1Z0afqILkuqi4qZmRkPHz6sNe9BWWJpacnr168zNja2Qnk7XqVYWFjwgw8+4AcffFDa8vl/b0RjaTJq1ChmZGQwLS2Ncrmcfn5+eoe41q1blydPniRJRkVF8ZdffmFgYCCTkpL0ehgsLCx44MABHjhwgNu3b9f7wa9atSr37dvHVq1aVfjDZGVlxc8++4y3b99mbGwsb968yeTkZA4ZMkTnJL7W1tacNGlSqYvazM3NuWDBAi5YsIDffPONXvZ/8sknTE1NLXHxkrZS6Ajnz59PmUzGhw8f6sSYZGtry+HDh/Ozzz5j165dOWXKFG7evJl16tTRmqhWU3F1dVVHk7799tsllfnvcgoWFhZctWoVZTIZ09PTKZPJKJVKOXToUJ0qsJDFZ/ny5czPz6efnx+bNWtGQMl6lJubq9Vy6JKkkDH6yJEjejEZ1a9fn3///TcjIiL0ofcuUTp37szr168zMzOTFy9eZPfu3Vm5cmWeOHGCBw8e5MGDB3UisB0+fDhTUlJKDW9u1aoVHz16xEePHnHbtm06229ubs6QkBD++eefFVIf9vb2tLe3VxPh+vv7a/3hcXNzY2BgIDMzM5mWlsaUlBTKZDI+f/6c27Zt47Zt20pd6KepSCQSTpgwgevXr1evEJ04cSKzs7OZnZ1dGinNvzfMuSQMGTIEo0ePRkFBAQ4ePAhjY2N8/PHHGD58OA4cOKB1xJyLiwsA4KOPPoKvry+mTZuGBw8eAFCGLSsUCkRGRuplc9OmTQGUHyZcFt5//33s378fT58+Rf/+/fVKdlIS+vfvjypVqmDu3LnYtWuXOjtWVFSUOhTcyMhIq2twcHCAh4cHEhMTERcX99J+MzMzzJs3D7a2tgCAP//8U2f733vvPVhZWemloyi6desGQBlKnZ+fj+joaCQmJmql45133kGTJk2gUChgbGwMMzMzFBQUQC6XY+TIkQAAqVSKqVOn6mxnv379sGjRIkRERKB58+Z4+PAhPvjgA0RERADQM2LydbcSNGkpODs788SJEyTJ3bt3E1By08lkMt65c0djQs2iYmdnRzs7O/bp0+elBTULFy5kdna2VuStL4qxsTH9/f3p7+/P/fv369RS6NixI3Nzc7lx40Z13oSSpGnTptywYQPj4+P5999/a8VbUKVKlZf6n05OTgwODuaSJUu4ZMkSre0u5Gf47bffStw/ceJEyuVyrl+/nuvXr9eax6KwK1avXj3evXuXp06d0uurWyhCCB4+fJiHDx9mTk4OMzIyOGfOHK11rFixQr2QLyEhgQsWLFCvPC2Uu3fvakXFVlQkEgl37NjB5ORkddemS5cuJMmvv/6aX3/9dWnH/vd0H5YsWcKcnBw+fPiQrVu3JgC2adOGsbGxvHXrlk5OoTRxc3NjVFQUN2/eXO7qwLLE0tJSvUrywIEDWjsFIyMj3r9/n15eXqWWKcwhkJCQwBMnTnDAgAFcv349b9++rVcduLu7kyRnzZrFWbNmaXxcp06d2KlTJ0ZERDAyMrLE1Ybu7u4MCwvjvXv32KBBA62a0ePGjeO4ceM4d+5cGhsb08vLi48ePdK7KV4oNjY26qXHCoWCkZGRGjNkF0rTpk0ZGxvLrKws3rt3T318ly5dmJCQQJlMRplMRrlcrjNRS/fu3Zmfn881a9aot61atYqJiYl0d3cvix/kv8MpODg40N/fn7m5ucW89ocffkiZTFZhTkEIQTMzM27cuJGPHz/Wq5UAKElRClPTrV+/XutR/MmTJ/Pu3bsl0pULIfjtt9+yEEU5FX766bfgUBQAACAASURBVCeGh4frbLeZmRlXr17NJ0+eqPkgNDnO2dmZFy9e5MWLF5mbm8uZM2eW+NKdOnWKOTk5HDNmjFZ2denSRV2fK1as4JAhQ/j06VN+9tlnBFBeEhSNZN68eUxNTWVqaqqavEfbZ6tjx45MSUlhdna2mkClcePGPHjwIBMTE9XErXK5nPv379eaGMfOzo5///03nz17pl7u3rJlS8bHx3Pr1q3lHf/fMaYwa9YstGrVCqmpqfD19VVvl0gkMDU1RU5OTqFj0QoWFhYAlKvZBg8ejBo1asDIyAh9+/bF0aNHcf/+fb3sbt26tXopsZ+fn9ZjHnXq1MG2bdtKnHefP38+pk6dii5dugCAOhlu9erVMWLECMyePVtnuz08PDBixAjMnz9f436pk5MTVq1ahfbt2wMAMjMzYWtri/79+yM2NhYSiQRCCMyYMQO9e/fG6tWrsWXLFo1tqlWrFn777Tf1suPw8HB4eHjAwsICPXr0QNeuXdGsWTMsX74cR48e1f6iATRr1gzDhw+HtbU1AEAmk+H48ePIzc3VSk/h6tq8vDx07NgRfn5+cHZ2hpOTE3Jzc9XPan5+Pnr37o2mTZvi+vXrGusfM2YM3nrrLXz33Xfw9fWFlZUVfvzxR9jY2MDPzw+1a9cGADRq1Ah2dnY4cuQIsrOztbqGN9YpVKlSBYBysM7U1BRHjhxBYGAgAKVD6Ny5MwoKCnDq1CmtA4NatWqFWbNmAQB69+6NJ0+ewNHREbVq1YJMJkPv3r2xZcsWLF++HOHh4Toto27Xrh3MzMwAAAUFBVofn56ejvfeew+//fZbsUE+U1NT9OrVC59//nmxzNg2NjZYt24dEhMTceTIkXL1Ozo6wt3dHYDypXNyckJBQQHGjBkDIQSaNWuGH3/8EQDw6NEj+Pr6lrpsvX///vjwww+hUCjUNs6cOROTJ09WL5sWQsDe3h7nz5/HsmXLtKoLa2tr3L9/X/1MrFu3Dnl5eUhOTkbr1q0REBCA06dPIzw8XCu9RdGmTRvUqFFD/dJeu3YNGzZs0FpPaGgovL298dFHHwEAmjdvDpLqZ1QIAUD5DKelpWnFs9GrVy94eHggOTkZAQEBqF+/PkaNGoWuXbsiPz8fo0aNwowZMwAon7m4uDh4e3tr7RRee9ehtO5DYdP16dOnzMrKKpYZaPDgwUxMTGRUVJTW8/bjxo1jdHQ0vby86OXlxdatW3PQoEFMSEjgX3/9xbZt2/Lnn39mZGQkN27cWFpSjXJl8+bN6jljXealXV1dGRoayitXrtDd3Z1WVla0srKitbU169evz6pVq7JevXqsV68ep0+fzujoaAYEBJQ5fValShVWqVKF33zzDX18fHj37t3/Y++6w5o82+9JCEsQkA0uxC1OxD3rrHXiqKNWceOq9VdHFXe1DtRaa23de4Ja96wDUVwIqCigoCwlbEIgEMb5/YF5PwcjCdjxfZzrui/Cmzd3HpKX532ee5xDX19fvnz5krm5uVQoFAwPD+fTp08ZERHByMhIRkZGMj09nbdu3SqS0LVFixacPHmywFTcp08f9unTh3379mXv3r0Fodnw8PBSFZy1a9eO7dq1Y1JSErds2cLKlSsXG4BV12rXri2kZVWBwAULFmjtr2HDhgwMDGRWVhaVSqWQRk9NTRUYxeLi4jSi0hs9ejSlUimzsrKYmZnJiIgI3rp1i1KpVNiifvnllwLzUvXq1QsjhP202weRSFQXwJF3DjkCWATADMAEAAlvj88neU7b9ylHOcrxF6OM7vQ6KBB9qY63NO+lXSnMnj2bs2fPFqLAKuHU5s2bMyQkRBCB1SQQNGzYMEZHR3P+/PnCMUdHR968eZO3bt1ilSpVhOMmJiZaKyU5OzszKipKSLlp4wMoqIrbuHEjL1y4wEuXLjEoKIhhYWH08/PjpUuXGB8fz/j4eN66dYtDhw4tcVXj4eFBDw8PPn78mEOGDKGNjQ2dnZ156tQpZmRkcMOGDbSzs6OpqSnNzc1paWlJS0tL/t///R/XrVunNR/k6dOnKZfLtZaJV9mOHTu4Y8cOBgYGaixQW5x5enpSqVRSoVAI2YfSkszWrFmTy5cv59GjR3ny5EkmJyczLS1N4BuNiIjQiAty0qRJfPjwIV+8eMHDhw/z22+/5bBhwxgZGcnLly+ru6L967IPAHoAuPX28RKUclKQSCS8fPmywOv/8uVL9urVi25ubnz48CGVSiWPHTvGypUrq/2hmpmZ8d69ezx79izFYjE7derETp068dWrVwwNDdVak6AwW716NRUKBUeMGMERI0aU2p+trS179uzJkSNHcvz48Rw6dChbtWpFZ2dnOjs7q3VBWFlZ8dKlS7x06RL79etHGxsbzpgxgw8fPmRwcDAnTJhQ6lLswqxRo0bMyMjg2bNnNZaIf9dcXFwEWv6yFBWuW7cuY2NjmZmZybS0NPbq1Yu9evUqM/8SiYRffvmloNWhYim/fv26RtWpYrGY1apVY8OGDQWC4h9++IHx8fGapDb/0uzDMACH3vl9mkgkGgXgAYDvSH5E9vdWYm7ih8eBggo6Ozs7AAVRWlNTU+zevRvGxsYwMDDA9evX8c0332jE16ivrw+lUonnz59jyZIlmDZtGoACItcVK1bg8ePHavsqDoaGhqhTpw5iYmLw559/lonPuLi4Ulcy5uTk4MKFCwAKAoGTJk0SPteTJ08KJLNlCSMjI6xcuRI6Ojo4cOAAMjIytPbVv39/3L59GwCwb9++shoi2rVrh4oVKwIANm3aJHxGZYXc3FyBv5KkQDK7adMmjb7T/Px8REVFCb9bW1ujR48e+PPPP8t8zKWeFEQikR6AfgDmvT30G4AfUDAz/QBgHYCxH76O5FYAW9/64LvPZWdn49SpUwCA6dOnQ19fH4aGhvD19cWePXvg7++vMYGrVCpFcHAwBg0ahCdPnmDJkiUAgD179pQJ07IKOTk5ePToEWrXri2kJP8JSE1Nxfr16//S95RIJDA0NERAQECpmIvNzMzg7OwsZEPKEmKxGBUqVMDjx4+xefNmrdLbJcHJyQm6urrIy8vDtWvXAEDr1KkK5ubmMDMzQ0xMTJmS4gJAWWwd+gO4VMRzDgCeaBNT0NXVpa6uLmvUqEFHR0c6OjqWWg3Z3Nyc1apVK9UyVh2rUKEC7e3tKZFISlUV+d9gjo6OpS4E09XVZeXKlT9JK7eJiQlr1aqlllK3tubq6sqUlBQqlUpB96G0PsViMTdu3KhplkSt7UOpiVtFItFhABdJ7nr7ux3fKk6LRKKZAFqRHFaCj9INohzl+AdDLBZj5MiRsLKywq5duwCg1CQ+WuLTszmLRCIjAFEAHEmmvT22D0BTFMxMrwBM4n9k6YvyUz4plKMcnx7lFO/lKEc53kM5xXs5ylEOzfGP7X1QwdDQEI6OjmjRogW6du0KZ2dnPHv2DGvWrMG9e/f+7uEVCxMTEwAFfQYkIRKJkJ2djTdv3pRpxNjc3BwSiURjMpD/dZiamiI3Nxc6OjqQyWSwtrYWpPJycnJgYGAAkv9ICcFPiX/89mHZsmVYsGABXrx4gadPnyIiIgJ2dnaoXr063NzcEBYW9lcOVS3UqlULX331Fdq1aweggIlHNQmkpqZizZo1ZZIebNCgAQBg165dePz4McaPH19qn58SZmZmaNu2La5du6Z5k04RqFmzJho0aIDq1avj7t27and2GhoaYt++fdDX14eOjg7OnDkDd3d3XLx4EQDw8OFDdOzYESKRCLt27dKok/EfjPLtQznKUQ7N8Y/dPqh65ytWrIjJkyfj0qVLeP36tbCUmzZtGiZOnIjZs2erVXAiEolgYGAAGxsbobcdAGJjYwVewrKAlZUV9u/fjyZNmsDf3x8AcOXKFYSHh+PGjRuYPXs2vvnmG/j4+ODBgweleq9x48YBKOCE2LhxY6nHXhQMDQ1Rr149JCQkICYm5qPn7ezsUKlSpRI5KDp27AgPDw/4+/uXeqVgaWmJadOmYfDgwcI2LT4+Hr169VLr+8zPz0fFihXRpUsXyGQytGjRAqGhoWjcuDGAgjblmjVrQiaT4ebNm6VeKZibm6NLly4YMGAAgIJ28GfPnmH9+vUaXX+9e/eGm5sbUlNT4evrC29vb2RmZpZt0VVZ9D6UQQFUkQUXRdGY1ahRgydOnFCrfrxJkybctm0bL1y4wIsXLzIoKIhxcXGMi4tjeHg4V61aVWqqeJXp6elx06ZN9Pb2ppOTE52cnN5rJNqxYwdzcnK4fv36UjMFqdq/nz9/zrp162rtRyQSsUOHDhw6dCgrVapEKysruri40M3NjW5ubgwICGB6ejonT55c6Ovnzp3LiIgIGhoaFtuH0aVLF8bFxWndPq0qXho9ejQfPnzIhIQE7ty5k82bN2evXr34+vVrHjt2TOgNKMmcnJw4dOhQOjg40Nra+r3ejx49ejA6Opo5OTkaUdJ9aMbGxpw0aRJDQkIYGhrKq1ev8urVq4yLiyNJzps3T21fjRs35qtXr6hQKEiSeXl59PLyokQi4fz58/n7779z0qRJnDRpUlEM3P8dzEsq4o4PYW9vD3Nzc7V89OnTB0FBQdizZw9iYmKgr6+PqlWrAgB69uyJuXPngiTmzZtXgqeSoVQqsWrVKiQlJX10N2zQoAGaNGkChUKBmJiYIslXDA0NYWRkhMTExCLfx8zMTGCLfv36tdbKSFZWVpg9ezZGjx4NQ0ND+Pn5QSKRCEzOAODl5YXTp0/jxIkThfq4ffs2xowZg/r16wMo2I8XhmfPnoEk6tevr1WQeMqUKQCANWvWIDg4GGPHjsXp06dhbGyMjRs3Qk9PDwcPHlTbX3BwMIKDg4Xf7ezsMGnSJADA559/LqwotSVvqVmzJhYuXIiBAwciJCQE8+bNE/o3OnToAGdnZ9y5c0dtf/Xr14ednR0iIyOxcuVK6Orqwt/fHzo6OhgxYgTq1KkjfEfbtm3TfvXwd68SSlopFGWzZs3i1atX1Sp9LowHT6X7sHPnTgYFBZXqTquOGRgY8MCBA1QoFLx37957bdoqMzU1pampKXfv3s3x48cX68/W1pZPnz7l06dPKZPJtBJCqVSpEg8dOiQQtAQFBfHy5cvcvn07x44dy6ZNm7Jp06Ylrmjs7e0ZGBjIlStXcuXKlUWeV7lyZcbHx/PEiRMar5Lq1KnD8PBwhoeH89y5c+8R16xdu5ZZWVlaKXHp6OjQ2dmZW7ZsYVBQEFWQyWS8cuUKXV1dteoebdOmDQMCAhgeHk53d/ePyGBq1qxJc3Nztf01adKET58+ZUZGBhcuXPjec+7u7pRKpZTJZFy7di3Xrl1blJ//jpVCYbC0tESPHj1w/fp1tZqZ5HL5e79XrFgRhw4VNHU6OTlhwIABCA0N/SRjVcHR0RHt27eHWCxGQEDAR3tzfX19rFy5UnhcUsNMXFwcAgICABRkO3R1dTUe05QpUzBs2DCcPn0aCxYsQHBwcJGpUj09PeTm5ha6uhGLxdDR0RFWLmKxuNDzVFRkhoaGwmN10atXL0EnYuvWrXj58iUAoHPnzhg2bBh+//13HDt2TCOfAODu7o5Vq1YJXYw3b94EAGzYsAHHjx/X2B9Q0B06Z84cGBsbY+zYsbhx44bw3NixBb2Ba9euxcmTJzF27Fi17ugNGzaEvb09wsPD36Pbs7GxwdChQ2FgYCCsHEqLf8WkYG1tjSVLlsDAwACLFy/GuHHj0LRpU3z33Xda+ZszZw66desGoICoNCgoqCyH+xF69uwJDw8PWFlZ4fLly4V2+zk7O8PNzQ1AQWdoUlISJBIJSEJXVxcVKlSAhYUFoqKiIBaLoVAohJz6OysujeDt7Y2qVauiX79+2L17N7777juhi+9DFPceIpEIOjo6sLS0BADo6uoWmttPTU1FeHg4xGLNk15hYWHC+w8cOBAXLlyAg4MDNmzYAJlMhnXr1mlV+xEXF4f09HSB/1E1ufbo0QP16tXDtm3bNA5Ed+zYEZ06dcKePXvg4+MDIyMjtG7dGj169MCgQYOE93n58qVG35tIJEJISIggWgQUpLtr164NkggKCiqT1Ok/flKoV68evLy8YGZmhpSUFNy8eRN6enpYs2aNVhwI1tbW6NWrl/Dl9+rVC82aNUNUVBTCwsIQFhamFbeAgYEBWrZsCSMjI+Tm5qJ58+YCs26rVq1Qs2ZNREdHY/369e/1xasQFhYmtBf/9ttvWL9+Pe7du4f8/Hy0b98e+fn50NHRQV5eHvLz8/Hbb78JnBOqwihNERoaCnd3d/z+++9Yt24dPD09MWLEiEJrP4ojx5XL5cjMzBQyRjo6OsJzVlZWMDExQX5+PoyMjIqMEZWEy5cvY9u2bQCASZMm4dChQ7CwsEClSpUwdOhQREdHa+X32LFjSElJQatWrWBqaooKFSoAKCDedXNzw9y5c7FixQrs3btXbf6DpKQkxMXFYfjw4WjdujV0dXVRpUqV91ZIgYGBGjFaqzIMXbp0wcyZM3Hr1i1069YNU6ZMgYGBAdLS0rBw4UJBIao0+McWL6mWikeOHIGdnR2GDBkCJycneHt74/Xr12jZsqXGnApAwdJ15MiRArFGbm4uGjVqhEaNGqF69epISUnBvXv3cOnSJZw9e1ZtroW5c+di7ty5MDY2Rn5+vvDPq4JUKoWbmxtu3bpV5N1BVYzUu3dvNGvWDNbW1ggLC0NERIRwd83Pz4eZmRkGDRqEatWqASiYFL744gv4+Pho/Hmo4OjoiMOHD+PIkSNYt26dxq9fvXq1kCI9dOgQZDIZHBwcUK1aNVhZWSEvLw96enqoVasWzpw5g/79+2vMcq2adKZMmQIPDw9YW1tj06ZNmD59usbjLQkWFhYYM2YMpk6dCkNDQwQGBmL69Ol4/vy5Wq93dnZGp06d8Nlnn+H+/ftIT0/HjBkzhElnzpw52LNnj9rjMTMzw88//yywROvo6CA/Px8KhQIikQgJCQkYOHDge4HTQqBW8dLfHmQsKtA4YcIETpgwQVB/7tixI58/f05/f39ev36dv/76a5kFAcViMU1NTWlnZ8exY8fyzJkzDAsL44MHD0oUQxGLxfz++++ZmJjItLQ0Xrt2jXfu3BFEP1RiuI8ePfpInq44MzAwoImJSZEBuWrVqgkSZ1lZWWopGalSepaWloUGz86cOcNdu3ZpxQFRo0YNIfD5+vVrBgQE0MvLi4sWLeKSJUu4ePFiLl68mLdv36ZUKi1U5EZdGz16NJOSkpiamsp79+59MhVnoECM6ODBg5RKpUUpORdrqiD3F198wcTERN64cYM3btxghQoVNPZVpUoVHj16lJGRkQwPD+eJEycYFhbG5ORkRkREqBNs/ncrRM2YMYMzZszgwYMHOWPGDCYmJvLKlSt0cHBg1apVeffuXU6YMOGTXQxVqlTh2bNn+d133xV73pQpU5iQkMC0tDQmJSXx1atXjImJoUwmY2BgIAMDAxkdHc3Y2FgOGTKkTMe4detWbt26lUqlkl27di3xfBUv5b179z66wFu2bMmoqCiNlZveterVq7N69erFSrcPHTqU6enpbNy4scb+mzdvLsgF3rx5k1OnTmVUVBR/+umnT3YdODg48MKFC3z16pUgWaip6erq8vjx45TL5XR1daWrq2upxtS6dWu6uLhQR0eHGzZsYHp6Ol+9esXPPvuspNf+u7MPqmDK/PnzMWTIEJw+fRpTpkwR9nVLly7Fzz//jAYNGmDNmjVC0E2pVGolvvIhDA0NYWpqWmgFnwoSiQQuLi4wNTVFZmYmDAwMIJFIEB0djUuXLmH79u0AChSdunbtigkTJuD69etlVkGpyvWr+ixK4oRUVXGamZlh2bJlWL16Ne7cuQMHBwd4enpCKpXi6tWrWo9HnVhMZGQk5HI5Jk6cKPBkqgNjY2OsWrUKQMH+evr06VAqlZg7d66Q9VDXj5GRERQKBYyNjWFoaAiZTAYbGxvo6+sLWzIbGxs0adIEnTp1gqOjI3744YeSluZFok+fPujYsSPu3r37XiZCW7xb2xAdHa1V4LY4lPc+lKMc5Xgff/fWoajtQ9WqVVm1alXu3r2bc+bMKVTfoW/fvvT39+fLly+F8tHS6gF06dKFO3bsYHh4OD09PUs8v2/fvgwODuarV6945coVzps3T9jjmpiY0MTEhKdOnWJOTg6vXLmiEa13Sabyf+3aNR44cKCo0taPrEWLFty/fz9DQkL48uVLvnnzhjdv3tR6eayJWVhYMDAwkF5eXhq9rnbt2oyIiGBERASXL1/OQYMGMSQkhGFhYeosmwUbPXo0nz9/zufPnzMpKYnp6emUSqVMSEhgamqqoBCVlZXFnJwc3r59m8uXL9ea17Nq1aq8du0aMzIyBDHcsrQuXbowOzv7f2P7oEoxqXL3heH06dMIDAxE06ZNBepsTTkFVDnkmjVrIjs7GzNmzEBSUhIWLlyoFuPu6dOn8fr1axgbG+P+/fvIzMwUnlNFy/X09CCRSPDw4cNSU7W/C5lMJoxh2rRpMDAwUKvR6P79+xgzZgycnZ1hY2MDHR0d+Pn5lenYikJSUhKCg4PRpEkT1KhRAwCEQqTiYGdnBz09PQDAsGHDMG3aNISFheGbb77RqFTYy8sLCoUCY8aMgYGBAZ4+fYphw4bBxMQEwcHBQkbr7t27CAsLw6NHj0olNty9e3e0bNkSV69eLXMqdgDIyMiATCaDSCQqMqulMdS8k+8EEI93mJkBmAO4DOD525+V3h4XAdgI4AWARwCctVkpfGqztbWlra0tPT09+csvv7B///60tbVl5cqVtYoMF2YqeXQ7OzvWrFlTo+yDJmZoaMgaNWqovVL4u83NzY23bt2ii4uLoPxVktnY2PDcuXM8d+4cnzx5wg0bNhRaKq6uGRsb08zMjGKxmNWrV2fNmjVpZWVFY2NjjeXhi7IKFSrw5MmTzM7O5ujRoz/Zd79r1y4qlcoSS+NRltkHAB0BOOP9SWENgO/fPv4ewOq3j78AcB4Fk0NrAHf/iZNCuZXbpzZdXV2OGjWKc+fO1VqoWB1zdHTkkiVL2K5du5LOLduUJD7QcAAQCsDu7WM7AKFvH28BMLyw88onhXIrt7/V1JoUSpN9sOF/qNvjANi8fVwZwLs1pzFvj5WjHOX4F6BMAo0kqSlNe3FakuUoRzn+PpRmpSAViUR2QIEqFAoCkQAQC6DqO+dVeXvsPZDcStKF6tRi/wXQ1dWFra0tjI2N/+6hlOMfAJFIhCpVqgjdk/9LKM1K4RSA0QBWvf158p3j097KybUCkMYSFKKKg66uLurWrYvY2Fg4OzvDwcEB6enpOH/+PLKzs4vs8VcXFhYWGD58OFxcXNCpUyccPHgQHh4eWvt7F6rmFycnJzRu3BjZ2dk4e/asUH35T4GjoyOcnJxgbW0NkUiE0NBQPHny5B83zk8NAwMDtG/fHkBBFeLQoUPx7NkzTJgwAeHh4WXyHqpJpn///tDX18epU6c0TgVXq1YNzs7OsLOzQ15eHkQiEbKysnDu3LmyqZZVM8h4CMAbADkoiBGMA2AB4E8UpCSvADB/JyX5K4BwAI8BuGiTfVCxEI0ePZr+/v68c+cO4+LiqFQqmZaWRi8vL+7fv59z584tVSpu8uTJlMlkzMnJYWBgIN3d3cskqNO1a1ceOnSIhw4dYkhICJVKJRUKBTdt2kQ7OzuNfOnp6VEikVAkElFXV1fgQtTX11ebj/BDMzMz44gRI7h9+3Y+ePCAUqmU2dnZzM3N5atXr+jl5cVGjRqxUaNGWn8GBgYG7N+/P8+cOUNvb2/WqlWrzINnYrGYBgYGQuOYSCTSivuyZ8+e9Pb2ZmRkJCMjIxkSEsIDBw4wLS2NI0eOLJOxGhgY0MvLi15eXszPz2d2djYHDBigkY927drx/v37TE9PZ25uLrOyspifn0+pVKpOevff3RBlbW1Na2trnjp1irGxsfT19eXFixcZFxfHtLQ0yuVyoZLLyspK4y9INencvn2bWVlZzMzM5LZt24okilXX9PT0OGXKFIaHhzMzM5OZmZlMS0vjixcv+OrVK2ZkZPD+/fvFfoFGRkZctGgR582bx99++413797l+fPnuXXrVvr4+DAmJoavX78WKvP69++v0RhVOf+UlBQmJSXx4cOH9PPzo6+vL8+cOcOHDx8yOTmZsbGxjI2N5dGjR9mwYUON3qNRo0Y8e/YsfXx8uHz5cn7//fesVq1amU0Genp6HDFiBM+cOcOUlBRu2rSJ+vr67Ny5M+fOnau2H319fa5cuZIxMTE8ePAgW7duzdatW1NXV5fTp09neHg4e/fuXerxikQirlixgkqlkkqlklKplG5ubhrf0ObNm0cfHx9u3ryZY8aM4fnz55mRkcHIyEg2b968TCaF8t6HcpSjHO/j714lFLVSEIlEFIlEbNiwIZs3b87GjRtzy5YtjI+PZ3p6OuVyOSMiIujh4aHVElrVRiyXyymTyZiens6MjAwOGjRI67uBmZkZV6xYwRcvXjAjI0NonZ4/fz7bt2/PHj16cMOGDVQoFFyyZEmRfipXrsyYmBjm5OQwMzOTcrmcGRkZVCgUzMjIYHp6OlNSUoSVyKlTp9QisFXZnDlzKJPJeOvWLQ4bNoy1atWig4MDq1evTn19fTZq1IhXrlyhVCqlVCplWloaT58+/RH5aFHWtm1bBgcHc/PmzR+Rk9auXZtjxozRirNBZdbW1ty+fTuTk5N58eJF/vDDD/T396ednR2rVq2qEe/B+PHjmZSUxG+//faj54KCgjh79uxSrxKAAr6JwMBApqSkMCUlhV9//bVW2xyVXECVKlU4e/Zsvnz5knK5nBcuXFBnxfzv3j58+IFeuHCB2dnZjI+P54sXL7hgwYJSaTUsXbqUS5cuZXZ2NmUyGRMTE5mcnMxHjx6xwSsIFgAAIABJREFUZs2aWn1ZO3bsoFwup1Qq5b59+4SmrnfPa9++PZOTk3n06NEiy6n19PQ4ZMgQzps3jydPnuTQoUNZs2ZN1q9fn/Xq1aOjoyMXLFjA9PR0pqen88yZMxoxA3t6ejIqKoq9evUq8pzBgwdz7NixHDt2LCMjIxkXF1ciB0Ljxo3ZuHFjRkREcMaMGR89LxKJePDgQW7ZskWr78zBwUHgN0hOTuaXX35JfX19fvHFF0xKSlK7ZPpdW7RoERMTE9mtW7f3jvfp04dPnz4tVUzlXRsxYgQzMzOFxr3CGvzUtSlTpvDx48eUyWSUyWSMiopSd5z/7oYoFdzc3DBr1ixYWVnh0KFDOH78OB49eoRXr16Vyq9KczA+Ph4WFhZITk5GXl4e0tLStNJQEIlEsLCwgI6ODuLj47Ft27ZCeQMHDhyIChUqICMjo0iiUaVSCS8vLwAFEfHCeA0tLCxgZGQEoKDhShM2Z3NzcxgYGAjcAYXB29sbEycWlJHo6urC29u7RCoyFR2bRCIplGqsZcuW6NKlC3r16qX2WN+FiiauRo0a6NevH3x9fTFlyhT89NNP8PX11Yqf8PDhwxg0aBB++OEHJCcnC9fVrFmzEBAQoBUP6IfQ1dXFsGHDoKuri9WrVwP4D7eFJhCJRJg6dSqWLl0q0P7l5uYiLy9PK39F4R8/KTRs2BBOTk7YvHkzpk6dWmZ+z5w5A6CAInzy5MmwtLSErq4uHj58qFUqztzcHPb29sjOzoajoyNmzZolTDyqzkUDAwPY29tDV1cXERERaqkZF0V0KhaLVassiMVitYhbVV2bL168gFgsxrx582BpaYnLly8jKioKiYmJyM3Nhb6+Pr755ht8++23AICIiAisWbOmxA5MlXybra0tOnfuLFCR6+npoVOnTlizZg0OHTokUNNrClUnbEJCAhQKBRYuXIg5c+ZAIpHgxo0bSE5O1thnWFgY5syZgy1btmDnzp1CetDIyKjMUtNVqlSBi4sLIiMjixTKUQcGBgZo27YtdHR0cOjQITx58gTDhg1DnTp14OnpiYkTJ5ZNp+vfvXUoafswYMAASqVS/vbbb2zdujV79erF2rVrl1knW6VKlXjixAlhvx4XF/fRUlIdmzZtGmUymbAFWbFixUcyas2aNWNISAijo6M1zhh8aJs2baIKJ0+e1JjzsG/fvjx9+jQjIiIYFxfHp0+f8vz58/z++++5efNmpqSkCDERddNm3bp1Y7du3ZiRkcHs7Gz+8ccf3LZtG/39/UmSUqmUTZo00fpvrlWrFmvVqsXTp0/z3r17lEqlJEkfH59SLceBArGVwMBA4TNNTEzkrFmzyqSRaevWrczKyuL06dNL7cvZ2ZkuLi5CTKZJkyYMDg6mTCZTpxPzvyOmYGBgwFmzZjE8PJwvX75kbGwsnz17xrFjx7J+/fqlbkfW19fnkSNHmJWVRblczpycHB45ckSji8HGxob+/v5MS0vjn3/+WeiFb2pqyjVr1lAmk/Gbb74p1ZgrVarEM2fOMCsri1lZWVy6dKlWfsRiMdu2bcupU6cyODhYIJmVy+X09/dnixYtSiSufddUwWFXV1eeP3+er1694osXL7hp0yZeuXKFz58/L/U/hcocHBx4+/Ztvn79WiOSlaKusQkTJvDFixc8e/Ysz549y127djE1NZU//vhjqYKin332GSMjI7lz585P1jo/ZMgQJiUl8dy5czQzMyvuBvHfMSmoLrbGjRvT1dWVY8aMYb9+/dizZ09evnyZx48f17qAR/WBJiQkMDExkdHR0czOzub9+/fp4OCgto+mTZsKUfqZM2cWek737t2ZnJzMly9fFktsqo61bNlSCP7FxcWxR48eWvtq0KABvby8mJCQwJCQEJ48eZJBQUGMjY3lggULuGDBAq38GhkZ0dbWljY2NgTAdevW8dWrV2X2jzBr1ixKpdIyIcNdvHgxMzMzGRoaynr16rFevXoEChjF09LS2K9fP638ikQi7tq1i/7+/u+JDGtjurq6RcrXmZiY8OnTp3z16pUQ7C3Cz3/PpFCUubu7Uy6X09HRUePXVqtWjdWqVWNISAizs7N55coVXr58mfn5+fT19WXlypXV9uXh4cGUlBSmpqZy6tSpHz3fsWNH7tq1i2lpaVy+fHmpL+LZs2czKyuLAQEBDAgI0KooqGvXrvT09GRkZCQfPXrEH374QbiYunXrxqSkJL58+ZIvX74sE0XuFStWMCwsrNR+gIK7b0pKCleuXFlqYhlzc3OGhoYyOjr6I5ZlExMT3rlzh/v379f4fRo1asS1a9cyIiKCX3zxRan/5mnTpnH37t1s2rTpR89VrFiRjx8/ZmZmJjt06FAc3f+/N/ugq6srMNQWFozT1dXFlClT4O7ujsuXL0MqlWr8Hq6urgAABwcHZGVloW7dujA2NoZIJIKfn59GQjOPHz9GXl4edHV1YW1tLRy3srICAKxatQpt2rTBunXrsHz5co3H+i7EYjE6dOiAvLw8eHt7A0ChilPFoXHjxtixYwdMTU2xY8cO7Ny58z3KMT8/Pxw6dEiQOJsyZQqWLVuGjIwMrcf98uVLtajiioNKZdzDwwPnz5/HsmXLVDcVraGvrw8dHR1s3br1I1VtlSKXigZOE0yYMAHTp09HTExMserh6iI0NBRLly5FixYtcOfOHZw9exZXrlxBeno6mjdvDhMTEyiVymKVvNTFP3JSaNiwIWbPng0AuH79Oh49egQzMzP06dMHKSkpaNSoEVq0aIHU1FStLlZ9fX1BS5IkJBIJ7O3tkZiYiG3btuHHH3/UyN+jR4+QmJgIOzs7jB49GvXq1UNWVpYQDa9cuTI8PT2xaNEirWXTVKhYsSKqV68OkkJ2Q1NMnDgRZmZm2L59O77//nvk5ua+93xGRgZWrFghyNEPHz4c586dKxU9eXZ2NgwNDVGjRg21OBkLg7u7OwDA1NQUX331VaknGaBAuevRo0do1aoVBg4cKEywRkZGGDVqFOrVq6fx5FOxYkXk5eXh0KFDuHr16ns3Cm1x+fJlHDx4ENOmTUOtWrUwcOBABAUFISkpCa1atYKVlRX27dtXJinUv33rUNj2wdLSktevX+f169e5fv16oY9AqVQyJyeHubm5vHz5MmvWrKlVEEgsFnPv3r3cu3cv8/PzGRwczH379vGrr77Syp9EImH//v0ZHBwssAFnZmYyISGBCQkJ/L//+78yWTYDBYIob968YXx8PNu0acM2bdpo7GP37t2MjIxkq1atijxHR0eHR44c4ZEjR0iS48aNK9W4Fy5cyJiYGK37Hxo2bChUWJYk0KOpVa9enT/99BMfPXrExMREJiYmMj09ncHBwRw1apTG18T06dOZkZFBHx+fMuP7BAqYoZcuXcqbN29SLpdToVAwKyuLKSkp3L59uzqNduW9D+UoRzk0xz9y+5CYmIilS5cCKKhoTExMxIULF5CamgqJRILIyEgcO3ZMq1gCUCDSqhIlXbZsGdLT05GSkqJ1VVhubi5OnjwJkUiEmTNnwsjICDk5OVi2bBkAlCm1d5s2bWBsbIyYmBitBHaBghhI/fr10bRpU9y7d6/QpXFeXh4ePHgAoGA7p6KT1xY6OjqwsrJC48aNNY6BAMCAAQMERazff/+9VGP5EJGRkZg5cyasra2FAiySkMlkWvMT3L59GwsWLHiP8r+0iI6OxuLFi/Hzzz+jU6dO6NGjByIjI+Hr64s7d+58tA3UGn/31qE02Yf/RWvYsKHAz9C1a1e1NCT/CdawYUNOmTJFq/Sxg4MDfXx82LRp00Kj7+Wmtqm1ffjHStGXo2g0atQItra2QrlwWUS3/8kwNjZGjRo1BH3RMrsj/u9BLSn68kmhHOX434Fak8I/MqZQjuJhZmaGyZMnC4rY+/fvL3W+XoWhQ4eiYsWKgmJ2Of738K/LPlSrVg3Lly/HtWvX0LBhw0/yHpaWlrC3t1er8/CvhEgkQteuXXH48GFMnz4dGRkZyMjIKLMJoVOnTti2bRu6dOlSJv6MjY3h6uqK33//HT4+Pti4cSOaN29eJr7/KqgCpLVq1YKFhcXfPZwSYWVlBSsrK1SqVElrHyWuFEQi0U4AfQDEk2z49pgngL4AlCggaB1DMlUkEjkAeIYCVSgAuEPSXevRvUXFihXRoUMHdOrUCa6urrC0tISJiQnmzZuHcePGlbogCCho73VxcUH37t3Rt29fmJiYoFmzZhoXRhkYGOCLL76AlZWVwDItFosRFRWF8+fPaz0+kUgEd3d3eHh4oGLFivjll19w6tQprf19CFNTUyxevBiGhoYCl0Np0LRpU6xcuRKdO3fG/fv38fDhQ1hYWGD//v2YPXu20LpeWujo6KBNmzaoU6cOdHR08OzZMzx48KDU14SJiQm6dOmC3r17o1WrVqhRowaCg4Px1VdflRmz86fAli1bAACXLl3SPkujRmagMB3JHgAkbx+vxn90JB3ePa8ssg8ikYi//PILExMTqVQqefPmTfbs2ZO7d+9mQEAAa9euXeqorIWFBffs2cOIiAhmZ2dToVDwxo0b1NHR0chPvXr1uHTpUiYkJJAk8/LymJeXx5ycHJ44caJUY2zXrh2jo6Mpl8u5bNmyUrcKf2jjx4+nUqnk8ePHaWFhUSpf1atXp5+fHzMzM7l8+fL3aMKWLl3Kx48fa8Vu9a7Z2tpyyJAhPH78OGNjY5mdnc2cnBy+ePGCnTt31tqvqnfgxIkTfPPmDW/evEl3d3du3LiRSqWSa9as0dp3aRr31LE+ffoIZLsdO3Ys7JwyFZh1QBH/7ABcARz4lJPC4MGDOX36dHbo0EFoPz116hRv376tUeNSYWZjY8M//viDOTk5lMvlTE9PZ3Z2Nt+8eaMOO+57/wg3b95kamoq09PT+eDBA+7fv5/79+/nwIEDWbduXVauXJktW7bk6NGjNaL5MjIyore3N5OSkrhz506NJ6uSrHnz5oyIiGBgYGCpJlkrKytaWVnx2rVrjI+PL5QzokGDBnz+/Dl/+OEHrd+nTp06DAwMpFKp5OXLl7lq1Sp+8cUXnDlzJtPS0jh48GCNfZqamnL+/PkMDw9neHg4//zzT7q4uFBPT489evTg8+fPefjwYbVp6s3Nzeng4MC+ffty2rRpXLhwIZ88ecJVq1Zx1apVGvNfqGNr164VODCKuGn8ZZPCaQAj3zkvA0AAgBsAOhTjcyKAB29Noz9eLBYzNDSUx48f15oEw8LCghYWFjx27BgzMzMZFhbGuLg4PnjwgL/++itTUlK4d+9etf01a9aM0dHRTE9P54ULF+jo6EgjIyMaGRnx888/56pVq3jnzh0+efKEX331FW1tbdX23bBhQwYHBzMxMZF9+/Yt0wvJ3NycR44coVQqLZazUR0bPnw4hw8fztTU1EI5GlX2+++/Mzg4mDVr1tRqxWBjY8MlS5Zw+vTpwipEX1+fv/32G9+8eVPUXbJY8/DwYExMDKdOncqpU6cKrc6jR49mdHQ0f/75Z7XJca2srLhr1y4+efKEWVlZwqoxIyND4MBYsWJFmU7uVapU4ePHjzl69OjiyFY+/aQAwAPACfwntakPwOLt4+YoEJo1Kc1KoTCrVasWY2JieOTIEa2W0cbGxjx+/DiPHz/O/Px8+vj4sH///ty7dy87duxIMzMz3r9/n3fv3lVbU2Ls2LFMSEjg+fPnaWVlRTMzM27fvp3bt2/n69evmZOTw4CAAPbp00ejsUokEi5YsIByuZz79u2jtbX1R+cMGjSIkyZN0opkdO/evczNzeXvv/9eqovSzMyMDx8+5MOHD+nr61vsP9CIESMYFRVVpsVXQ4cOZUZGBnft2qXxMt3S0pIBAQH08PAQjuno6HDRokWMiYnh6tWri+QyKMz09fW5bds2vn79mtevX+eqVas4dOhQTpgwgfHx8YyPj+eVK1e03qYZGxvTxcVF0CgRi8Vcs2YNnz59ytq1axe32vu0vQ8ikcgNBQHIr6j6zyazSSa9feyPgiBkHW3foxzlKMffAG1WCgA+B/AUgNUH51kB0Hn72BEFwrLmZb1SWL58OeVyOSdOnKjxLFuhQgV6enoKy7jHjx+zdevWwnN4e9e7ffs279+/LzAHFWdisZiHDx+mXC7ngQMH2LVrV0FyLC0tjampqVy+fLlWZCVOTk4MCgrinTt3hBiHkZERBw8ezF9//ZW//vorX716RZIabXcA8Msvv6RMJmNgYOBHqww9PT1WqlRJba2HOXPmMDU1lampqRw1alSx59rZ2VEqlXLChAmcMGGCVnfLd61nz54MCwujr6+vxkpWQAFJycWLF/nLL7/Q2dmZzs7OPHr0KENDQz8iXlHXKlWqRCsrq/fIWcaMGUOFQkGFQsFVq1Zp/fcuWrSIvr6+AjtY7969KZfL1WHJKpvtAwrXkXyBgq1B4Fv7/e25gwAEvz32EEBfNSedYv8YlT6gRCLhjBkzGBUVxV27dtHIyEjjD3TIkCFMT08XAjIq6q0PL1p/f38+ePBA7b2/q6srpVIpU1JSGBcXx+TkZKENNyUlhevXr9doCaqyrl27Mj09/T1Kr169elGpVAoT24sXLxgaGsr79++zbt26avnV09Ojt7c3c3NzCyUUHTlyJH18fOjj40M3N7difdna2vLOnTv08/Ojn59fiRoUenp6jIqK4rZt27ht2za1xVtU2pmqfzSJRMKRI0cyOjqavr6+rFOnjtb/aFOmTGFsbKygJfnixQuNRGVKMmNjY3p7ews3Cm2Je/v06cO4uDguW7aMQMGN7MSJEwwPD1eH5q9smJdIDi/k8I4izj0G4FhJPtWBs7MzgIKuwPbt20MsFsPIyAidO3eGrq4u3rx5o5HWAQBUqlQJP/74I3Jzc+Hp6QkACAkJ+eg8V1dX1K1bF/fu3VO7y+3KlSv4448/0K1bN5iamiI/Px86OjoACmogRo0aBTs7O3h6eqpN821gYIDx48dDLBbj+vXrsLa2xqRJk+Dm5oanT59i06ZNAICbN2/is88+w1dffaUWbTxQ0PnYtGlTBAQE4NChQwAK6ilsbW0xbNgwzJw5E1WqVAEAkMTx48eL7JTs0qULGjVqhFmzZgFAiVTrPXv2hLGxMUaOHAkAsLGxKZHAZfjw4QJr1a1bt3D27Fm0bdsW7u7uuHHjBkaOHIn4+Hi1/vYPIRaLBXYlFXGLQqEoGwXnt7C2tkb9+vVx8+ZNAMDJkydLeMXHqFSpEsaNGwcLCws0aNAAAwcOhFgsRrdu3bBu3TqhN6TUUGfm+NSGQmbtkJAQhoSEMDAwkNeuXRMk0s6dO8dLly4xMzOTe/bs0Si1M3r0aCoUCp4/f17IDHx4jr29Pf39/ZmZmamxhJyxsTHnz5/P5ORkpqSk8OTJkzx58iTnzZvHFy9eUKlUakRWYmFhwTt37lAmk3HQoEH09vYWBGvfvbsPGTKEr1+/5unTp9VePY0aNYpZWVlcvXq1cKxatWq8fPkynz17xlOnTjE5OVlQsyoueDd58mQqlUp27NixxMj/0KFDGRAQwKSkJEGVu7DVmsocHR25ceNGnj17lhcuXBCUwnJzc0mSx44dKzWHpCq/v379ekGFasWKFfT29i6zepCJEydSqVQKymTa+Bg2bBhJ0s/Pj9euXWNqaipjY2MZExPDwYMH08HBgWKxuDg5un8nR2Pnzp2xcOFCHD16FABw4sQJrF69Grm5uVixYgU2b94s8CF4eHhAJpNh9uzZJVawSSQS9O3bFwYGBrh69WqRlYru7u5o1KgRTp48iePHj2s0drlcDqlUCiMjI3z99dcC519OTg5cXV1RtWpVocpRHeTk5EChUEAsFsPNzQ1t27YFULAyyM/PF/gOqlevDolEgp9++kktijKJRILhwwsWgKqVkIODA06ePAkHBweMGTMG9evXR4cOHQAAe/fuLXYFouIFNDQ0BFBw59XX1xfGUqFCBbRu3RrTpk1Dp06dYGxsjLi4OGGlU9hqDQAGDRqE+fPnIzk5GVevXkWtWrUAFFR3yuVyGBkZQSwWCz0gFStWhJWVldpKUfXq1cPixYvRrFkzrF69Ghs3bhSe09HRgbGxcZkoL4nFYnTt2hU5OTk4d+6c1n6ePXuGCRMm4PTp05DJZJg1axbmz5+PgIAADBgwAHK5vNTKacA/sCGqZs2a0NXVFUhPV65cCTMzM8ycORP79u0TvqQ1a9bAzs4O48aNw5MnT4TyzqJgaWmJKlWqQKFQCBfRu9DX18fMmTPx3XffwdfXF3PnzlWtYjSCWCxGfn4+dHV13yPRzM/P19ifTCbDxYsXUbduXXz22WdQKpXIyspCs2bN0LBhQ1StWhVAgQTe5s2bcePGDbUnHUNDQ5BEhw4dMGDAAIwaNQq1a9dGfHw8XF1d0bJlS2zYsAEASixJvnjxInx8fARezeHDhyM/Px9HjhxBnTp10LlzZ3Tu3BkGBgbIz89HTk4O9u3bV+w2SjVx+fn5YfHixZg8eTI6duwIALh69SoOHjyIMWPGoGvXrpg7dy6uXbuGzMzM9whoi4JIJMKgQYOwZMkSWFtbw8PDAxcvXsTAgQPRu3dvAEDt2rUxc+ZMtT5LdWBubo4///wTjx490tpHUFAQgoKCABTwfvbv3x9+fn4YOHCgVlKHReLv3jp8uH1YunQpSQrimceOHWOzZs0KXQ41b96cUVFR9PHxoaWlJS0tLYtcellbW/PWrVvMzc3l0aNH6ejoSEdHRzo4OLBfv37cv38/09PTefv2ba0o41U2adIkymQyBgUFCcrWHTt25MuXLymTyfj1119r5M/KyoqrV69mQkIC09LSmJmZyYcPH/L8+fMcN24cx40bV+zyuygbMmQIX758ydzcXMrlcmZnZwuCOCkpKVyxYoUQ2FPHX40aNejt7U1vb2/KZDKmpqYKCk5KpZKZmZnMyMhgcnIyPT09Swy6DhgwgLGxsdyzZw8vXbrEhIQELlq0iIsWLRK4CNu0acMHDx4wLS2NDx8+VLuwq169enzy5AkVCgVjY2Pp5eVFf39/ymQyvnnzhm/evCmVlsaHZm9vzzdv3pQpV2ffvn2ZlJRUYqbnA/t36j60aNGCP//8MwcPHqxWueqGDRuoVCrZv3//EiO6rq6ujIqKEtSr4+PjKZVKmZWVRZlMxqNHj2pNLKoyNzc3JicnMzU1VRBriYuLY0ZGBs+ePat2MdSH1rZtW86fP5/Hjh3TSLWppH88b29vHj9+nN7e3vzjjz+4ePFiIUWrqakUolq2bMn+/ftz9uzZvHLlipBS3blzp9pjX7VqFV+/fs2nT59yz549RcYqDA0N2aRJEzZq1EjtCaxixYq8cOECc3NzhbJ2kgwPD+fAgQM5cODAMvl8VbZ48WKmpqZqXLhWlFWoUIGnTp1iSEiIpoVa/85JQVOrW7cuR48ezcqVK6vVBzF06FCePXuWr169EszT05PDhw8vE+bdypUrc9++fUJ9QmpqKlNSUqhQKHj27FmNJOP/G8zY2Jj16tWjnZ1dcQGwQj/Hjh070snJqczHJBKJ2K5dO27ZsoX37t3jiRMn+OOPP7Jly5af5DNYtWoVw8LCilNu0shsbW356NGjElPFhVg5HVtRMDY2hqmpqfC7tgSoRaFSpUrYuHEjnJycABSQoIaFhWHNmjV4+vRpmQh2lKP0kEgkMDMzg0KhKJXQTUk4ePAgWrRoAVdXVzx58qTU/nR0dGBjY4P4+HhNqenKmZeKglwuh1wu/2T+U1JS8PXXX38y/+UoG+Tm5v4l/JZ37txB8+bNy+xmkJeXh9evX5eJr8Lwr2NeKkc5yvFp8T+5fShHOf5KmJmZoU6dOggKClK74vQToZzNuRzlKMd7UGtSKN8+lKMc5XgP/5OBxr8CIpEI9vb2AApUnMu04ux/HCKRCJUrV4ZIJEJycjIsLS2Rn5+P2NhYjcrI/0oYGRnB1NQUFSpUAFAQLIyKikJeXl6xr6tYsSIAwNPTE7Vq1UJoaCiqVKkCX19fbN68+dNkTf7uGoXS1il8ClNRtWnLjKOvr89Zs2YxKiqKUVFRvH79+j9W3k0ikbBPnz50d3fnyJEjWbVq1TLxW6FCBXbo0IFdunRhly5dOGjQoFJVir5rX375JWNjYxkdHU0vLy9KpdLiyEpLND09PbZt27bYilhtzcDAgC4uLjx27BgjIiKYk5PDnJwcymQybty4scQGNhWnxZkzZ5ibmysoTScmJvLkyZOatov/bxQvldZEIhGdnJw4e/Zsfv/99zx69KhAxbZ//36NJ4ZevXrx1KlTTEtLo1KppFKpZG5uLsPDwzls2DCtOCUlEgkrVKhAPT09oQuutPx+IpGIvXv35oEDB5iUlESSVCgUvHv3Ll1cXErl28jIiD///DOjoqIYExPDV69eMTk5mYsWLSqVXxMTE5qYmPDixYtMT09ncnIy5XI5ExMTKZfLOXfuXK38zpgxg3FxcR9VHEokEtapU6dURW1VqlTh+fPnmZOTw7CwMIEYViaTMT09XW2OymXLllGhUDAjI4MZGRnMzs5mXl4eL168qAnP5b+zS1IFlRBLkyZNoKOjA39//0/yPhKJBD/++CM6d+6MiIgIxMfHIzQ0FC1btkSfPn1w+PBhtTUKxo0bh5UrV8LAwAABAQFCt15SUhLs7e2xdetWiEQigb9AXYwZMwZr1qxBZGQkwsLChN7/u3fvAihoEAoJCUFaWpraf/PKlSsxatQo5OXl4cmTJ7hz5w4aNWqEjh074sCBAxgxYoTGn7lEUnA5TZs2DW5uboiNjcXVq1fRsmVLWFhYoH379hr5+xCqv9vc3Bx6enrIzs5GdHQ0LCwskJubi+7du2P9+vUa1wPUq1cPNjY2Av+FCi1btsTx48cxcOBA3L59W6sxx8TEYOHChUID2LfffgsAGD16NN68eaN4NTzfAAAgAElEQVS2PsWKFSuQkpIiiL00atQItWrVQseOHeHv749u3boJXbOlxt+9SihqpaBqWHr27BkTEhK4YMGCQklLS2tisZiDBw/mwIED6ejoKDTqHDp0iGFhYWrTew0fPpzh4eG8fPkyBw0a9BGNm56eHs+fP89NmzZpPMYGDRrQz8+PCoWCCQkJfPDgAaVSKbOzs4U+Di8vL7VZl+rVq8eYmBg+fvyY/fr1EyjXTExM6OrqyvDwcB44cIDGxsY0NjZWe5w1atRgjRo1+ODBAwYHBwt9Drdu3aJMJuOtW7feoyfT1HR1damrq0sPDw9u376dX331FTdt2sT4+HimpaXx+vXrGrNbqTgrIiMjP2q8W7t2LTMzM9m+ffsyudZmzJhBuVwuNJ1NnjxZKz/6+vq0srLi999/z8zMTGZlZXHlypXqvPbfvX1wcXGhi4sLExMT+fr1a/r6+vLatWscMGAA69WrR2tra5qYmJTqIivK+vXrxzdv3vD27dslNjCZmprS09OTkZGR/Pbbb4vdI7q7uzM0NFQr2jAnJycOGTKEtra2tLCwoL29Pdu0acM2bdrw1KlTJMnhw4erfXEmJydz5MiRhT6/cuVKPnnyhHXq1NForKqJ/OHDh/Ty8iJQwIocHBzM1NTUMqv/F4vFNDY25rhx4xgSEsKkpCRmZGRw/vz5GvtauHAh8/Pz+eOPP3703KlTp5iRkVEmk0L//v0ZHx8vUPQtWbJEa3kClRkZGXHJkiXMy8vj/v371dlSls32oQjZuCUAJgBQ8VXNJ3nu7XPzUMDjmAfgG5IXS3qPwmBjYwOgIGqblJQENzc3dO/eHePGjYOtrS2ysrKQlJSES5cuYfPmzdq8RaFo3Lgx3N3dYWtri3Xr1pVIyVW7dm08ffoU8fHxePHiRbHR4KNHj2LgwIFo2bIlwsLCNBpXcHAwgoOD3zsWFxcHAMjKykJqaqraBBv5+fnIz8+HkZERRCKRamIWYGlpCUBzyXexWCz8dHBwQPPmzZGUlAQ/Pz8MGzYMCoWiTMpz8/Pz0a5dO6xatQo6OjogiYyMjCLJWoqCoaEhmjZtivT0dPj4+Hz0fG5u7rs3Lo0gEolQp04dtG/fHi4uLhgwYAAqVqwoLPFXrVpVamm7jIwMPH/+HGKxGHXr1oW9vT2io6NL5RNQLyW5G8AmAHs/OP4TybXvHhCJRA0ADAPgBMAewBWRSFSHZPF5l0LQpEkTAAU8hSYmJrC3t8dvv/2GvXv3wt7eHqampmjWrBmGDh2KXbt2qcU4pIJEIhEu4Pz8fOHib9CgAXbs2AEnJyds3boVW7duLdGXv78/cnNzUa1aNcyaNQu3b98ukqMwOTkZUqkUw4YNw5kzZ0qdpvzuu+8AAJ9//jn27t0LPz8/tV53+PBhtGvXDgsWLEDfvn1x6dIlpKenQ1dXF02bNhVYfAYPHgwA2LBhg1oMRKoL8sKFCxg/fjz279+PrKwsWFpaIjc3F9nZ2WXWa2BsbCwQ2hgaGuLYsWMasxo1bNgQHTt2RFpaGmxsbNClSxchrpCXlwdLS0uQ/CjWUBJ0dHTg6uoKDw8PNGzYECKRCAqFAnl5eahcuTIA4LPPPhO0RQubmNVB/fr1sWDBAmRnZyMyMlK4SZQW6hC3+rwVjlUH/QEcJpkN4KVIJHoBoCUA9a7Wd6CauaVSKcRisRB4VM2OABAVFYXu3btDT09PrUlBJBJhyJAhmDp1KkxMTAAAiYmJOHDgAF6/fo01a9agTp062LVrF+bMmaNWDpgkKleuLKxoSpr909LS8Pnnn6N58+b4888/S/RfFIYPH4558+YBKBATXbRokdqvTUhIwPTp0zFu3Di0aNECX375JRwdHWFoaAilUomkpCTo6elh4cKFAAomzrVr15bgFUIJ76JFi3Dz5k3UrVsXTk5OAl1Y/fr10blzZ1y/fl2jv9XIyAgKheK9GoTs7GzI5XIhj//u5K4uhg8fDmNjY+jr6+PXX38F8J8Ad35+PvLy8kAS5ubmGvm1t7fH+PHjUa1aNQQFBeHGjRt48uQJJk6ciFatWgEApk+fjrt37yI5ORk1atSAVCrVuOagQ4cOqFq1KnJzc9GmTRt069ZNIMBt3bo1MjMz4e/vr3HgtTQVjdNEItEjkUi0UyQSqXSvK6OA+l2FmLfHPoJIJJooEokeiESiMgqZlqMc5SgTqBkIdMD7YjA2AHRQMKmsALDz7fFNeKsr+fb3HQAGaxNorFq1KqtWrcqAgAAmJiYWKvZqaGjI06dPq8XmY2JiwvXr1zMmJuY9ZuDExEQqFArGxMQwKyuLv/32m8ayY71792ZmZiY3bNhQYuBzxowZjI2NZbdu3bQOMHXt2pUREREMDg5mcHBwqdiMRSIRv/vuO6akpPD06dMcMGAAmzdvzq5du3LPnj3cs2cPQ0JC2KFDh1IFxVR59u+//16j13Xv3p27d+/+iGzFxMSEAwYMoL+/P9PT0/no0SONaOl0dXV59+5dymQy3rt3j1u2bOG2bdu4detWbt26lYcOHaJUKmVaWhrbtm2r8d9bo0YNdu/e/T0mrw0bNlCFly9fCkFXY2NjSiQStX1XqFCBP/30E9+8eUO5XC7Q9IWGhvLEiRM8ceIE8/PzGRAQ8GGdzaerUyApVT0WiUTbAKgS+bEAqr5zapW3xzSGivhky5Yt6NmzJ0JDQz86R6FQQCqVom3btrh//36RviQSCb777ju4uLhg8ODBiIiIENiMmzdvjtzcXFhZWYEkGjdujAEDBuDYsWNqL0fDw8Mhl8vRpEkTmJqaFhkrEIvFaN26NYyNjVGpUqVCzykJ3bp1w+7du5GZmYmxY8cCACIjI7XyBRToa3zzzTdYsWIFfv311/e2YartQL9+/TB58mQ8ePBAo9jNu/Dz84NcLkfNmjWFWEBJMDMzw6xZs9CjRw9ER0djxYoVAArYo/X09EBSiC1kZGSordGh8jF16lSYmpri8ePHSEr6//bOOyyqa2vj755KlyIQEFRARMoFo0g0iF1BwBZ7wahcW9SrsRvjjT363ShYwRpbMGhugoYYjYmiYrBBEEUUC4gFQeltgIH1/THMuaCC07wod37Psx/gnDmbvWfOrLPLWu/KAWOMczt2dHREx44dOcVoZUlLS0NaWhr3t5GREdq2bcutHSQnJ3P3uDLaHi4uLti9ezenz1BVVQXGGCorK2FjYwM7OzsAssVnVSMyVTIKjDErIsqs+XMIALmczHEAEYyxjZAtNDoCuKLK/5DfNDt27MDhw4frfeNiY2PRvXv3Buvq2rUrRo4ciS+//BKmpqb46quvOGVgqVSKiIgI3Lt3Dx07dkTnzp3h7e2N48ePK2wU0tLSUFlZCQMDg3oXjExNTbFmzRoMGDAAd+/eRWJiokJ118bT0xNbt25FaWkpxo8fzzkvqcPw4cNBRDhx4sQrX3i5w45EIoGBgYFSC27m5ubIzc3lvmRZWVkoLy+HUCjk5u1vwsPDA23btkVhYSGCg4Ph5uYGQLbr4unpCUdHR1RXV6OiogL//ve/kZGRoXD7ADTo7FNRUQHGGEpLSxX6cllaWmLs2LEICwt7reEMCAhAhw4duIQ1mzZtQk5OjlLtBYBx48ahU6dOkEgkdQwrYwzV1dVcW+XrMMquswAKGAXG2GEAPQA0Z4w9BvAVgB6MsfaQDUnSAUwFACJKZowdgSzPpBTADFV2HmpDRA166iny1Pnoo4/QunVrrFy5EhYWFtDX18eNGzcAyKTiT506haKiIhgaGsLW1hbPnz9X6okolUpx7do1dO/eHSEhIQgNDYVEIuF2UOzt7dG3b1907twZVVVV2LJlC7dYqiiBgYHYunUrxGIxRo8ejUuXLil1fX2YmJggJSXltVuFgwcPBgAIhUL89NNPCj/R2rdvj1WrVmHjxo04e/Ysd8zY2BgAFF5pr6qq4rZPxWIxevXqBUC2ui+VSiGVSiEWi5GVlYXff/9doToVhTEGoVCI+/fvIzMz842vb9asGdauXQtHR0fs3r0bBQUF3C7Zxx9/jKCgIBgaGnILwqdPn1a6TUKhkBsJAP/x8JQbxvLycm5RMTQ0FDExMQp7udahsR2X6ltTULT079+fdu7c2eBrWrRoQTt27KCIiAhas2YNjR49mgwNDcnQ0FCteXLt8sknn1Bubi4REeXl5VF2djaXYam4uJiqq6vp2bNnNHPmTKXrHj58OOXl5VFqaqrKSsv1leDgYMrOzq4TsGVkZETjxo2j1NRUSk1NpRMnTigVKDVnzhwiIoqJiaE+ffpQt27d6OzZs1RYWEifffaZUu2bP38+FRcXU35+Pj1//pwrcmHc69ev0/DhwzX6ngCgzp0707Nnzyg0NFSh11tbW9ODBw9IKpVSTk4OPXnyhIqKiqiyspJKS0spMTGRFi9eTDo6OlwKeVVKUFAQF1AVHx9PFy5coJCQEAoICCCxWPym+t/v2AdFuX///huHSE+ePMHUqVPfajt+/fVXzJ49Gz4+PrCxsUG7du24Pem0tDScPXsWP//8M06ePKlUvQMHDkRYWBgSEhIwd+5cLhmIJts9YMAArF69Gp988glKSkrg4uICb29vXLx4EYAslkEZp5jLly8jMjIS3bp1Q2RkJCQSCQwNDRETE4N9+/Yp1b7Dhw/DyckJ7du3R7t27QDInuJ//vknzpw5g+PHjyuUAEZZTE1Noaenh7y8PIVen5mZiWHDhmHkyJFwcXHhph6XL19GZmYmbty4geTkZLVDu48fP47JkyejqKgI165dQ3l5ObKyslTyc6iXxh4lqDtSaNeuHW3btk3jTwpVC5/PJ0NDQ7K2tiYHBwdycHAgKysrlaIafX19KSMjg44dO6Zw9mtVir6+Ps2fP5/LxH3u3DkaPHiwWqMpsVhM7u7utGzZMoqOjqbNmzdzqdOVLSKRiMzNzTk3agcHB2rWrNlb/RwtLCxow4YN9NFHHyl9rTxmRF035rdQ3u/YB0VLu3bt6PDhw1ywzDvwxmusrFixguLj4xXKZ6Et2qJA+d+YPuTm5oLP58PR0REA3spQsrE4ePAgIiMjNZ6XQouWhtAKt2rR8r+DNhlMY9OhQwcAsi3RPXv2aCStuRYtbxutmrMWLVrq8F5NH8RiMZo1a8Z5hWkagUAAGxsb8Pl8ZGVlqZVarl27doiKigIAWFhYwMPDQyOx7oDM1XXWrFlo3bo1AODrr79+rR6AKjDG0LJlS7x48eKt5lfUFGKxGBUVFZrdkmu6NL28D/PmzcMPP/zAhctqCiMjI4wZMwZbt25FUlIS7t27hxUrVqhcn0AgwOrVq+Hk5AQnJyecPn1aozkLW7ZsiYKCApw5cwZnzpzBv/71Lzg5Oaldr7OzM7Zv347r169jypQpKtfj5OSEiRMncou/cvT19TkvPE3QqlUrHDp0CA4ODhqr831FIBDgww8/rFM6dOig0vv9Xq0peHl5wcXFBS4uLhrx+wdk8/0vvvgCvr6+SE9PR0REBJ4/f65SbIIcXV1dmJmZcWIrO3fuVDmQqDZyMY6TJ0/WcYL68MMP0bFjx9cGjSnKgAEDsH79ejg7O6OwsLCO4pRQKFQoJt/CwgIAEBISgv79+yMiIgKzZ8/GixcvEBAQgKVLl+LkyZNYuXKlyu2szcSJE/Hhhx+qnYrN1tYWvXr1goeHBxdfkZKSgqioKFy8eFHttSCBQACRSITKykqNJJk1MTFBv3790L9/f5iamkIqlUJHRwd2dnZ11K8EAgGWLVuGQ4cOKddetVv4X6SiogIikYhLqKEqLVq04HzQhwwZAiLCli1b8M033yArK+sNV7+Zrl27wt3dHadOyZTo1BFTkTNixAj4+/vj7t27iI6OruPZeO/ePfj4+CAiIkKlutu3b49169aBMYZZs2ahbdu2ePbsGed92KxZMyxevPiNRucf//gHAJmqUHBwMA4dOoSKigro6upi4sSJcHNzw759+1RWGqqNUCiEp6cn0tPT1VKwcnNzw7fffgsrKys8efIEjx8/BgC4uroiMDAQKSkpmDRpkspT1m7dumHFihXw9PREUlISJ+by/fffq+TdqKenh7CwMPj6+kIoFEIsFkMgECAvLw9lZWXc9nVZWRn3HjVpo8Dj8Wo7PKmMv78/xo0bB0CmhDR//vxXvlB8Ph/NmzdX2kjo6Ohg/PjxEIlE2LJli1rtlOPo6IjVq1cjOjoad+7cecUoZmVloWXLlirVLRAIMGnSJLRq1Qpjx47FsWPH4OnpiZCQEE6das2aNfVKzMkRiUQYP348ACAqKgr79+/nIiQ/+OADODo6IiYmRiGJO0UQCoWwsbHBixcvFI66fB2enp64du0aIiMjkZqaygWGiUQiLFmyBKNGjVJ5HWvBggWYN28ezM3Nce/ePRgbGyM8PByALKpyy5YtSkUxyiUA+vTpAz6fj8zMTMTFxeHu3btITk5GQUEBF2hXWFjIhZQrTWN7Myrj0bh9+3bKy8ujHj16qOXZ5ebmRvHx8RQfH08lJSW0bds2TuZcKBSSv78//frrr7R06VKl627VqhVlZ2dTZGSkxjzRJkyYQImJifW6Ss+fP1/lRCu6urr073//mwoKCsjS0pJatWpFly5dogcPHpCXlxd5eXkpVE+XLl1IIpGQRCKhSZMm1Tnn4OBAN27coKioKI29J05OTvTs2TP6448/1HJ5FolEr31fe/fuTbdu3aI9e/YoJXMPyFzdFy1aROXl5XT37l368ssvycjIiCZPnkxlZWVUVlZGCQkJdQRYFClWVlZ08eJFKigooEWLFpGZmZmyauZNz6PxypUrGDp0qFpPBgC4efMmPvvsMwDAhg0bEBwcDGtraxw9ehT9+/fHgAED8Pz5c+zdu1fpuoOCgmBsbIzvv/9erTbWprq6Gnw+H/r6+igsLOSO6+vrAwB8fX0RGhqqUt1lZWWIj49Hz549MXXqVHh5ecHe3h4zZ87ElSuKS2FYWFhwT73XzcGJSO3PrTa9e/eGmZkZbt++jaKiIpXrqd3WNm3aICgoCAAwefJkJCQkYPHixUrvQrm5uWH27NmIjY1FcHAw0tPTYW1tjcGDB3MJc/Ly8pReXxg/fjycnZ1RUlICU1NTDBw4EElJSZpPlNTYowRlRgqTJk2irKws6tmzp8aeOB988AFFR0dzKbkKCwvp0KFDryQGUbRcuHCBUlNTqU2bNtyx9u3bU79+/VROZmNmZkZxcXG0atWqOscnTpxIEydOpPT0dHJ2dlb5PTA3N6ezZ89yiUpmzJihdB3u7u7cSGHatGl1zjk4OFBSUpJGRwrTpk2j6upqldpau1hZWdGYMWMoJCSErl+/TnKioqJUDkLr0aMHVVZW0oYNG6hz584UHBxM8fHxJJFIqLCwkAoLC2nWrFlK1cnn87k8FEVFRSSRSEgqlVJycjKtWbNG0WCzpjdSkD9pNLkn7e3tDRcXF1RVVaGiogIFBQUIDQ3FX3/9pXRdH3/8MZycnHD06FHw+Xzs378fgGzhzcjICDk5OZgxY4bS4dM5OTmYO3cu9u3bh8LCQmzZsgXjx4/H/PnzAQCbN29GSkqK0u2VU1ZWhoqKCujp6aG0tFQhUZGXuXv3LvfUtbGxqXNO7kfAGINIJIK1tTUEAgH3BFY19Jcxxj15VUEkEmH9+vUYO3YsHj58iN9++41L9WdgYPDGjND1kZqaiujoaAQHB2PChAnQ19dHbm4uioqKOBl2ZVMHVlVVYc2aNcjMzETbtm1haWkJfX19fPDBB5g7dy66du2K0aNHaySvxntlFEpKSsDj8SAWi9WqRyQS4YsvvgAg29YqKyvDtGnT4OrqiunTp2Pw4MEq5eXr3bs3AJl+w7Zt29CpUycAQH5+PvLy8mBlZYVx48bh1KlTSn8J4uLi8I9//AObNm2Cn58fjI2NuT788MMPSrdVDo/Hw//93//By8sL27ZtQ48ePbBgwQJcuXKFW4lXhOrqak57oHfv3oiKisKtW7fg5OSEzp07w9jYGK6urvj+++9hZ2cHAwMDHDx4EACwatUqpdutiQXnyspKHD16FKmpqfj9999x6dIlfPjhhwDAfanXrVundL1Pnz7F1KlT0bVrVxgYGEBPTw/Z2dlYsWIFZ7xV8Vu5fPkyrl69CmNjY9jb28PIyAitWrXC7Nmz4enpiTlz5mDp0qVc31TlvTIKsbGxMDIywqBBg3D69GmVBSumTZuGxYsXA5BFVU6fPh2XL1+Gra0tgoKC0Lp1a4XFRWtja2uLvLw8GBoawtPTE2PGjAEAXLp0CZ06dUJUVBQePnyo0s1sZGSEgoIC5Ofno1evXjhw4IBaxgAADA0NMX/+fAwbNgxLlixBeHg4Pv30U2zevBljx47F+vXrFa6rvLwcISEhAICFCxfi119/RUlJCZd4x9DQkNM7PHToEO7fv48LFy4AUG3kJ9+BUccwEBF+/vln/Pzzz6+cY4xxWcpUITs7Gz/++CP394gRI9CmTRu1ZeOqq6uRm5tbZzfI1tYWS5cuRYcOHThBYHW8ft/o0ViT1yGbMXaz1rFIxlhiTUlnjCXWHG/NGCurdS5c5ZZp0aKlUVApbRwRjZT/zhjbAKC2OuR9ImqvqQbWRiKRQCQSIT09XeVRgpWVFYYPH87FIQQHB+Ovv/5Cq1atEBwcDBMTE+Tl5alUf05ODiwtLfHJJ5+Ax+NxqddjY2Ph4uKC0tJSLlWYMsj9BgQCAY4cOYLw8HAsXLgQHTt2BACVVp/19fWxfft2DB06FJ9//jl27NgBQKbgXFxcjFatWildp3wH5Nq1axgwYADEYjHS09NRWFiIefPm4eHDh5y0vrp4e3sjPz8f9+7d00h9cuTrCIwxjYrBdunSBYWFhZxgsKLIXcX79u2LO3fu4Nq1a3XEWHk8HmxsbMDj8fDw4UO1UxECaqaNY7KVvxEAeqndEgWQ6/KbmZmpXIeZmRlatGiBpKQkALKcCbNnz+Z89RMTE7F7926V6g4LC4Onpye6dOkCHo+HOXPmAJDNsc3NzXHx4kWlk6BaW1vjX//6F/Ly8hAUFISCggI4ODjgyy+/5L64qhiFhQsXIjAwECtXrsSuXbvAGIOrqyuWLFkCY2Nj3L9/X+k65cTGxiI2NrbOMScnJ/j7+3NGV12uX78Od3d3lRZFG8Ld3R0AUFRUpHQS4IZo3rw57ty5o/SUT75ou2LFClRWViI+Ph4nT57EH3/8gebNm6Nv377w8/NDfn4+oqKiNBKer+6agg+ALCKqrVduxxj7C0AhgC+J6MLrLmSMTQGgdNSNVCqFm5sb9PX1VfLWKikpQV5eHvcUv379OiwsLPDs2TNERkZi3bp1Kt8MGRkZGDNmDPz9/eHj48Nlb87NzcWPP/6IU6dOKb0A1KdPH7i7u2PJkiVwd3eHo6MjJk6ciFu3bnF5GZSlU6dOmD17Nh49eoSqqips3boV3t7eaN26NaRSKU6cOKFRPwtAJmyqo6MDoVCokfry8vKQn5+v8UAzeX7O2t6N6tKuXTsEBAQgMTFRaV8NuUT+smXLsGjRIvj5+aFnz57czo2Ojg709PRw5coVjYn6qmsURgOovbeSCaAlEeUwxjoCiGKMuRJR4csXEtFOADsB5ZSXhEKhWttQaWlpOHr0KGbNmgUAuHjxIlJSUnDq1ClcvXpV5W0oOc+fP8f+/fu57Uh1iYmJweLFi8EYg5OTE0QiEfbu3Yvo6Og6QUvKIHeH9fDwwLp161BZWYnExERs27YNMTExOHfunNpBRi8jlUphaGiINm3aaCT0XUdHB0+fPlU7HH3QoEF48uQJysrKsHbtWi6qcOXKlRoLHR8yZAhMTEyQlpamVBar2oSHhyM5ORleXl7w8vKCq6sr7OzskJmZiTNnziAsLEzpZDj1oqBzUWvUyiVZc0wAIAuATQPXxQDw1JTzEp/PJwcHB7K2tlbWvbNOEYvFZGNjQzY2Nu+i4u5bL4wxMjc3JwcHB7K3tyc7OzsyNTV9q//TzMyM9u3bRwEBARqpz9jYmKysrNSuZ+LEiXTv3j0ub2RQUBAFBQVptO/r1q0jiURCY8aM0Uh9urq6ZGtrSx4eHmRnZ6dMHsq37rzUB8BtIuI2sxlj5gByiaiKMWYPWdq4B2r8jzpUVVWpNdeVU15ertQefFODiPD8+XOVRxqqkJOTgwkTJmisvvz8fI0sqkVHR8PIyAj29va4cOGC2tu8r6OiogISiURj6x9lZWV49OiRxkR7XkGBp/hhyKYFlZCllg+uOb4PwLSXXjsUQDKARAAJAAZo0s1ZW7TlfSwtW7YkHx8fMjIyauy2KDRSeK/k2LRo0aIWTU+OTcvbx8vLC+vXr+eycmv53+O9cnMGZFJn1tbWKCwsRGFhodKr5Do6OrC0tKyzg/Gy/Nj/Ki1atMDmzZvh4eGh+XDcdxzGGGxsbLi4moKCgv/de+Jth0Vrek1h+vTpJJVK6caNG7Rv3z7q2LGjQtfxeDzy9vamgwcPUn5+PlVXV1N1dTVJpVJKSEigzz///K3ma5QXT09PGjt2LJmZmSl1HY/HIz09vXqLWCxWq11GRka0atUqLnT8bb8P71KxsrKijRs3UmZmJslJTEyk5cuXk6Wlpdp1Dxs2jAICAjS+w+Pr60tTp06lv/3tb4peo9Cagnb6oEWLljq8V9MHPz8/fP7553j+/Dlu3bqFzp07QyQSYezYsQ1GywmFQnz55ZeYNGkSjI2N8eeff3I+6E5OTvD29sbKlSuRmZmpcU++2gQFBWHVqlVgjOHmzZvIycl54zX29vbw9fWFm5sb2rVrV6ef8t8ZY8jJycGyZctU8sY0MTFBaGgoRo4ciYyMDJXChWvz0UcfYfz48cjOzkZhYSF0dXVx8OBBjW6hWVlZYf369fjzzz+xc+dOEBHEYjEYY2CMwdPTk3MUaigM3tLSElu3bsXAgQNRUtiSc8cAABH6SURBVFKCM2fOAJC5Jf/zn/9EmzZtsGjRIqXzeYrFYgwcOBALFixA27ZtUVlZiRs3biA6OhoAsGfPnjoxDKowbdo0DB48GJGRkZgwYQIkEola9XE09tRBkekDj8ejcePGUUZGBp06dYo8PT0JAH322Wd0586dN271WFtb09WrV2nu3Llka2tbR3NPV1eX7O3tKS8vjw4fPkw6OjoqDeU8PT1p1apV9aorWVhY0Pnz56miooJ27NihkO7f6NGj6cmTJ5wiUmlpKZWWllJZWRkVFxdTSUkJlZSUUFlZGVVWVtLGjRuVbrdIJKJvv/2WysrK6OnTp6/oKypTli1bRsuWLaMLFy5QYGAg6enpkY6ODvXq1Yv27t1LnTp10siwmcfj0e7du6miooJyc3Pp8OHDFBERQY8fP6Znz55RYmIifffdd+Tp6cndK/WVwMBAevHiBRUVFVFRUREFBgZSYGAgGRgYUGhoKJWXl9OmTZuUalu/fv3oxIkTlJeXR5WVlVypqKjgNBoPHDhA+vr6ar0P3333HeXm5lJ2draiU4imkYrewMCANm7cSAUFBfTLL7/UmeP17duXHj169EYhV6FQSK1bt65X+NTNzY2Kiopo3759JBKJVPqAVq5cSURE06dPf+357du3U2lpKSUkJHAisW8qoaGhVF1dTVevXqU5c+bQN998QwMHDqSBAwdScHAw3b59m27fvk0lJSWUlJREDg4OSrd7zpw5lJubSy9evKC///3vKt+gfn5+lJqaSqmpqdSrV69Xznfq1In279+vkXWbmTNnUnZ2NhUUFFBRUREVFBTQpUuXKCwsjKZNm0bdunUjHo+nUF0hISEkkUgoMzOTtm3bRhYWFpxhF4lEFB4eTo8fPyZ3d/c31iUSiWjq1Kn06NEjzgCkpKTQ119/Tbt376bk5OQ6hlwdAwzIjEJOTg5lZ2fTnDlzFLmmaRgF+U27Z88esra2rnPO09OTUlNTadSoUUq/ofIFusmTJ1NKSgo9fvyY/P39VfpwTExM6M6dO1RSUkKDBg165TxjjJKSkkgqldInn3yicL0jRoygTZs2cU9YucFycXGhPXv2UH5+PuXn51NpaSmFhoYq3W5zc3NKTk6mFy9e0NatW9VyHf/xxx9py5YttGXLlnpfEx4e/op+ozJFKBSSUCik3bt3U1lZGZ07d452795NgwcPJjMzs3qNfn2lRYsWdP78eaqurqZff/2VjI2NX3lNnz59KDc3l44cOfLGJ/vw4cMpOzubKisrKTk5mYKDg8nc3Jz77A4ePMiNGoqLiyklJUVpRefaRW4UsrKyaPny5Ypc0zQ0Gn/44QckJyfj7Nmzr2jkd+nSBSYmJkoHghgbG3MqQWPGjEFVVRV27NjBzSeVZdSoUfjggw/w4MEDxMXFvXI+KCgILVq0wOPHj5WK/z9y5AiOHDlS59jatWsxfvx4mJubc26+3377rUpZl6ZMmQJ7e3tER0dj3rx5ICIYGhqia9euKC4uRnp6OgCZS/GbFJMrKyvxyy+/NPgaGxsbtGnTBpGRkSqFT9c8QFBVVYXKykqsXr0ap0+fVroeOXZ2dvDw8EBhYSEuXLjw2jn+tWvXkJiYCHd3d1hbW3N5FV5GJBIhMDAQZmZmSE9Px9y5c7lkQIBMOUsgEHDzfiKChYWFSvfv61BVX+R1vPNG4fHjx6+NU9DT08PQoUORmJiotJ6ijo4OnJ2dAcii94gIgwYNQsuWLRETE4OjR49yAptvwtDQEH5+fhCLxbh27dprIwB79eoFExMTnDp1CmlpaRCLxaiqqlIqEYiuri5WrFiBGTNmQCAQ4ObNm1i7di0A4Oeff1Y6jt7NzQ0TJ04EACQkJKC8vByWlpZYvnw5RowYgaKiIs5XPz8/H998802Dma5ycnLQrFmz154TCoVYtGgRfH198csvv2gk5p+I1E7FJ5VKwRjDixcvcOTIEc7o1CY/Px9btmzBt99+i08//RTLli177euMjIzg7OwMIsL58+frGAR5PXPnzuXk0rp3747KysrX1qUo8kVVTUrnA++BUaiPAQMGwN3dHQsWLFD6Jnv27BlWr14NQKb137ZtW7Rv3x4ff/wxhgwZgqFDh+LYsWM4cODAG3cIhg8fjn79+kEqlSI3Nxcff/wxxGIxpFIpp5rj4+OD4uJi2NraYu/evejduzenzvwmbGxs0K1bN4wcORK+vr64efMmzp49i7CwME55WBWCg4NhY2ODuLg47NixAz4+PggPD4epqSnu3LkDAwMDuLq6ApAZvsTExAaNQlxcHBfwFBMTUyezVrNmzdCxY0cIBAJkZmaqHJJc++YXi8Xo2rXrK2IuymBpaQk+nw+JRNLgyr18xOTv74/Q0NDXajhUVlZyOTmsra1hYWFR5wEhlUqRmZnJOdvJd4zelHmrIV6agmuM98Io8Hg88Hg82Nvbo3nz5mjRogU2bdqEc+fOKZ0nT458awiQpYgzMjKCi4sLpk6dioCAAPj4+MDGxgbz5s1rsB6BQACBQIDS0lJMmTIFM2bM4M7JRwISiQTl5eXw8PBAhw4doKurq5AidcuWLREeHo7+/ftDKpUiLi4OU6dOrVfOXSAQoHv37rh582aD6e4sLS3Rt29f5OXlYcOGDZgwYQJmzpyJ9PR0LFiwAHfu3MGaNWtgb28PQCY48ttvvzXY1qNHj3JKUGFhYfjtt98QHx8PExMTTJkyhZO8u3Xr1hv7XR9yrYvy8nLweDy0adMGYrFYZe0Hb29viEQipKSkNCiowufzUVVVBQMDg3pFYgoKChAdHQ1vb2/4+Pjg0KFDWL16Nc6fPw9ANtJbuHAhOnfuDEA23M/MzNRY5KQ8sawmeCeNgkAg4J6yI0aMwKBBg1BeXg49PT0YGxvDwMAApqamMDc3x4gRI3D48GGlhuIvU1VVhby8PFy8eBFXr17lxEeCgoIQFxfXYDhtcXExcnJyUF5eXsdiM8Y4xWHGGHg8Hh48eICSkhKsWLECFy9ebLBN5ubmCAkJQZ8+fXD37l1899132L9/PzIyMmBtbY3mzZtzo5g2bdqgb9++cHJyQp8+fRAVFcVNDV6Hra0tzM3NUVxcDKFQiMWLF6OqqgrffPMNdHR0sHPnTnTs2JF70m3bto1TAKoPiUTCjb66dOmCIUOGoHv37mCMYf/+/bhw4QICAgK4p64qyOfN+/btQ7du3TBs2DBcv35d5Zyd8kS3tra2MDU1rVfFqbq6GiKRCHFxcQ2OHH/88Ud4e3tjwIAB6NGjB9zc3PDo0SNcuXIFXl5eXEZrQPaUlz/s1BH2kee++Omnn1Su42XeOaNgbGyMf/7zn/j73/8OAJxDUWxsLDw8PLBixQo8fPgQYWFhqKioQHFxsdpqSbWpqKjAo0ePkJ2dDXNzcwQGBuLYsWP1yqgdO3YMaWlp0NXV5YwCkcyRZs+ePQBkw+8zZ85g7NixCivv9OjRA/369UN1dTWuX7+OhIQE+Pv7o0uXLnB1dYWzszOnLWFvbw8ej8cNUa9evdpg3UZGRuDz+dDV1cXIkSPB5/PB4/Fw6NAhiMVilJSUIDw8nBuF3bx5s8H6XiYuLu6VBVdHR0fw+XyNSJwlJCTgzz//hIeHB+zs7FSu5+nTp6iuroaZmRmMjIxeaxR4PB4CAgIAyLQwG5qqZmRkYMKECVi8eDEmTJgAKysrWFhYoEOHDlzf5U/0+tZfFEVHRweGhoYAZHlGFF0DU4R3ziiMHDkSkyZN4sRTt2/fjnv37sHV1RWff/45Hj16hLlz5yqdZQkA93Q0MTGpExBVVVUFS0tLBAQEoFmzZvD29oaHhwckEgkKCgoaXNktKSl57Y6DSCTirmOMYdWqVUpJceXn5yMzMxNmZmbw8/NDYGAgeDweGGMoLy9HeXk5J2D7/Plz5Obmcjso9a2Qy0lKSkJaWhqcnZ0xdOhQlJeXgzGGiooK3Lp1CyEhISqnta8PebsV8eKsD/lnNnToUPj4+CA/P19pg1WbixcvIj8/H+bm5ggODsbatWu59Y7mzZvD09MTgwYNwpgxY5CcnKzQ07ikpATLli1DdHQ0evbsiXbt2qG6uhq3bt3C2bNnsXz5cgBAv379VG43ADg7O8PW1haMMcTHx6vtHVkbbeyDFi1a6vBOjRR4PB48PDyQmprKDV0lEgm++OILTJo0CRkZGRg1ahQSExOVqpfP52Py5MlYsGABkpOT0b59exgbGwP4zzxVnslIvih45swZREREIDY2VqXpyeDBg6GrqwtANp1Qdofk9OnT+Nvf/oZu3brBz88PHh4e4PF4uHv3LuLj48Hj8XDixAkA/8nFqOj/ePHiBUaOHInp06fDwcGB25r7/fffceTIEY1IsNeHKivlenp6sLCw4NLkjR49GhkZGVi4cKFCOzj1ceXKFWzfvh1Lly7FrFmz4OnpyU0hXFxc4OTkBB0dHdy+fRvz5s1TKnbj8uXLuHz58iv5T+UqzOpuI/L5fPD5fDDGuG11TfFOGYXq6mqEhoZi9erVOH78OABwTjO7du1CRESEykE1jDHExsbC1dUVT5484b5QAoGAWwh8+vQprly5gvz8fCQkJKiV4lwkEnEr1UKhkFt0VIby8nKcPn0ap0+fhq6uLjcE18QayoMHD7BgwQK161EGIyMj2NjY4OHDhwpfM2zYMIwZMwa2trZwcXEBIMsr8dVXX+HSpUtqt2nnzp0wMTGBr68vevXqVWfKd+7cOdy6dQsHDhxQWV/i5S9r7XtCHVVy+RqXuvW8jndSjk1PTw/W1tYAZBYxPz+/wS02RZAvphkZGYGIuDlY7a0cTVpcc3NzLuJy586diIyM1Ei97ysikQjfffcddHR08MMPP+Dhw4eIiYl543W7du3CyJEjkZ2djV27dgEAduzYoRHRVjl8Ph/NmzeHgYEB9/nLnZoKCws1+hSWZ9Hq27cvkpKSEBQUpPLO2bZt2+Dk5ITNmzcjOjpaEa9GheTY3kmjoKVp0qxZM0yZMgWGhoYICQlRaJpiaWkJV1dX3LlzR+nwZS2voDUKWrRoqYNCRuFdWVN4AaCk5mdTozmaZr+Aptu3ptqvVoq86J0YKQAAY+yaIlbsfaOp9gtoun1rqv1SFK2fghYtWuqgNQpatGipw7tkFHY2dgPeEk21X0DT7VtT7ZdCvDNrClq0aHk3eJdGClq0aHkH0BoFLVq01KHRjQJjzI8xdocxdo8xtrix26MujLF0xtgNxlgiY+xazTFTxthpxtjdmp8mjd3ON8EY28sYy2aM3ax17LX9YDI213yGSYyxDo3X8jdTT9+WM8ae1HxuiYwx/1rnltT07Q5jzLdxWv3fo1GNAmOMD2AbgP4AXACMZoy5NGabNERPImpfa697MYA/iMgRwB81f7/r7APg99Kx+vrRH4BjTZkCIOy/1EZV2YdX+wYAITWfW3siOgEANffjKACuNddsr7lvmyyNPVLwAnCPiB4QUQWA7wEMauQ2vQ0GAdhf8/t+AIMbsS0KQUTnAbysKlpfPwYBOEAyLgEwZoxZ/Xdaqjz19K0+BgH4nojKiSgNwD3I7tsmS2MbhRYAasdCP6459j5DAH5jjMUzxqbUHLMkIrlC5zMAlo3TNLWprx9N5XOcWTP92VtritdU+qYwjW0UmiJdiagDZEPqGYyxbrVPkmwP+L3fB24q/ahFGAAHAO0BZALY0LjNaTwa2yg8AWBb62+bmmPvLUT0pOZnNoCfIBtqZsmH0zU/X80Y835QXz/e+8+RiLKIqIqIqgHswn+mCO9935SlsY3CVQCOjDE7xpgIsgWd443cJpVhjOkzxgzlvwPoB+AmZH36tOZlnwI41jgtVJv6+nEcwPiaXYjOAApqTTPeC15aAxkC2ecGyPo2ijEmZozZQbaYeuW/3b7/Jo0aOk1EUsbYTACnAPAB7CWi5MZsk5pYAvipRn9PACCCiE4yxq4COMIYCwbwEMCIRmyjQjDGDgPoAaA5Y+wxgK8ArMPr+3ECgD9ki3ClAOpPOvEOUE/fejDG2kM2JUoHMBUAiCiZMXYEwC0AUgAziEhzOQXeQbRuzlq0aKlDY08ftGjR8o6hNQpatGipg9YoaNGipQ5ao6BFi5Y6aI2CFi1a6qA1Clq0aKmD1iho0aKlDv8PzuoSBZEg/BIAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import random\n", "from PIL import Image\n", "\n", "# get 100 random images from the dataset\n", "num_samples = 100\n", "samples = random.sample(list(X), num_samples)\n", "display_img = Image.new('RGB', (200, 200))\n", "\n", "# loop over the images, turn them into a PIL image\n", "i = 0\n", "for col in range(10):\n", " for row in range(10):\n", " array = samples[i]\n", " array = ((array / max(array)) * 255).reshape((20, 20)).transpose() # redistribute values\n", " img = Image.fromarray(array)\n", " display_img.paste(img, (col*20, row*20))\n", " i += 1\n", "\n", "# present display_img\n", "plt.title('Examples from the dataset')\n", "plt.imshow(display_img, interpolation='nearest')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Load the provided weights." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# load the pretrained weights\n", "theta = sio.loadmat(\"ex4weights.mat\")\n", "theta_1 = theta['Theta1']\n", "theta_2 = theta['Theta2']\n", "nn_params = np.concatenate([theta_1.flatten(), theta_2.flatten()])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Feedforward\n", "These are the functions for doing feedforward as written [ex3](https://github.com/rickwierenga/CS229-Python/tree/master/ex3)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def sigmoid(z):\n", " return 1 / (1 + np.exp(-z))\n", "\n", "def add_bias(X):\n", " m = len(X)\n", " bias = np.ones(m)\n", " X = np.vstack((bias, X.T)).T\n", " return X\n", "\n", "def forward(theta, X):\n", " return sigmoid(theta @ X)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Cost Function\n", "\n", "Remember the following variables from the lectures: \n", "\n", "* $L$: Total number of layers in the network\n", "* $s_l$: number of units (not counting bias unit) in layer $l$.\n", "* $K$: number of output classes\n", "\n", "The cost function for neural networks without regularization:\n", "\n", "$$J(\\theta) = \\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}\\displaystyle\\sum_{k=1}^{K}\\begin{bmatrix} -y^{(i)}_k \\log (h_\\theta(x^{(i)}) - (1 - y^{(i)})_k \\log(1-(h_\\theta(x^{(i)}))_k)\\end{bmatrix}$$\n", "\n", "And with regularization:\n", "\n", "$$J(\\theta) = -\\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}\\displaystyle\\sum_{k=1}^{K}\\begin{bmatrix} -y^{(i)}_k \\log ((h_\\theta(x^{(i)}-(1-y^{(i)})_k) \\log(1-(h_\\theta(x^{(i)}))_k)\\end{bmatrix} + \\frac{\\lambda}{2m} \\displaystyle\\sum_{l=1}^{L-1}\\displaystyle\\sum_{i=1}^{s_l}\\displaystyle\\sum_{j=1}^{s_l+1}(\\Theta^{(l)}_{ji})^2$$\n", "\n", "The double sum adds up the costs for each cell in the output layer. The triple sum adds up the squares of all $\\Theta$s in the network.\n", "\n", "**Exercise**: Implement the cost function for neural networks, `compute_nn_cost`, in Python. There are some structural comments to help you." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def compute_nn_cost(nn_params, X, y, input_layer_size, hidden_layer_size, K, _lambda=0):\n", " m = len(y)\n", " \n", " # Extract theta_1 and theta_2 from nn_params\n", " \n", " # Feed forward the network to get the predictions\n", " \n", " # Compute the cost of the current prediction\n", " network_cost = 0\n", " regularization = 0\n", "\n", " return network_cost + regularization" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cost without regularization: 0% (0.29 approx)\n", "Cost with regularization: 0% (0.38 approx)\n" ] } ], "source": [ "J = compute_nn_cost(nn_params, X, mapped_y,\n", " input_layer_size=input_layer_size,\n", " hidden_layer_size=hidden_layer_size,\n", " K=10)\n", "_lambda = 1\n", "J_reg = compute_nn_cost(nn_params, X, mapped_y,\n", " input_layer_size=input_layer_size,\n", " hidden_layer_size=hidden_layer_size,\n", " K=10,\n", " _lambda=_lambda)\n", "\n", "print('Cost without regularization: {:2}% (0.29 approx)'.format(J))\n", "print('Cost with regularization: {:2}% (0.38 approx)'.format(J_reg))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Backpropogation\n", "\n", "---\n", "\n", "### The Algorithm\n", "As the name suggests, backpropogation is roughly the opposite of feedforward propogation. Backpropogation is an algorithm that trains neural networks by computing the gradient and thereupon applying it to the neural network.\n", "\n", "Backpropogation (Backprop) starts at the end of the network. It finds the difference between the output of the neural network and the desired output. This value gets stored in $\\delta_j^{(l)}$, the error/cost for $a_j^{(l)}$. Also, $\\frac{\\delta}{\\delta z_j^{(l)}} = \\delta_j^{(l)}$. The process formaly:\n", "\n", "$$\\delta_j^{(l)} = a_j^{(l)} - y_j$$ for $l = L$\n", "\n", "$$\\delta_j^{(l)} = (\\Theta^{(l)})^T\\delta^{(l+1)} .* g'(z^{(l)}) = (\\Theta^{(l)})^T\\delta^{(l+1)} .* \\delta^{(l)}$$ for $L > l > 1$\n", "\n", "Also: \n", "\n", "$$D^{(l)}_{ij} = \\frac{\\delta}{\\delta \\Theta_j^{(l)}}J(\\Theta)$$\n", "\n", "As you would probably have expected, we don't apply the gradient to the input layer, layer 1, because we don't want to change our input in order to get a better output.\n", "\n", "The complete algorithm:\n", "\n", "Set $\\Delta^{(l)}_{ij} = 0$ for all ($l$, $i$, $j$)\n", "\n", "for $i = $ to $m$\n", "1. Perform forward propogation to compute $a^{(l)}$ for $l = 2, 3, ..., L$\n", "2. Using $y^{(l)}$, compute $\\delta^{(L)} = a^{(L)} - y^{(i)}$\n", "3. Compute $\\delta^{(L-2)}$, ..., $\\delta^{(2)}$\n", "4. $\\Delta^{(l)}_{ij} := \\Delta^{(l)}_{ij} + a^{(l)}_j\\delta^{(l+1)}_i$\n", "\n", "$D^{(l)}_{i0} = \\frac{1}{m}\\Delta^{(l)}_{i0}$\n", "\n", "$D^{(l)}_{ij} = \\frac{1}{m}\\Delta^{(l)}_{ij} + \\lambda\\Theta^{(l)}_{ij}$ if y $\\neq 0$\n", "\n", "### Gradient Checking\n", "After you've implemented code to compute the gradient, it's often a good idea to validate your code by comparing the gradient to an approximation of it. The approximiation is defined as: \n", "\n", "$$\\frac{J(\\theta+\\epsilon) - J(\\theta-\\epsilon)}{2\\epsilon} \\approx D$$\n", "\n", "### Random Initialization\n", "\n", "If all values in the neural networks are the same, the neural network will fail to develop advanced patterns and it will not function. This is the reason we use a random value for theta as the initial input (break the symmetry).\n", "\n", "### Neural Network Traning\n", "Follow these steps when training a neural network:\n", "1. Randomly initialize weights (avoid **symmetric ... **)\n", "2. Implement forward propogation to get $h_\\Theta(x^{(i)})$ from any $x^{(i)}$.\n", "3. Implement code to compute cost function $J(\\Theta)$.\n", "4. Implement backprop to compute partial derrivatives $\\frac{\\delta}{\\delta \\Theta^{(l)}_{jk}}J(\\Theta)$. \n", " * Usually with a for loop over the training examples:\n", " * Perform forward and backward propogation using one example\n", " * Get activations $a^{(l)}$ and delta terms $d^{(l)}$ for $l = 2, ..., L$\n", "5. Use gradient checking to compare $\\frac{\\delta}{\\delta \\Theta^{(l)}_{jk}}J(\\Theta)$ computed using back propogation vs. using numerical estate of gradient of $J(\\Theta)$. Then disable gradient checking code.\n", "6. Use gradient descent or advanced optimization method with backpropogation to try to minimize $J(\\Theta)$ as a function of parameters $\\Theta$.\n", "\n", "---\n", "\n", "In this part of the exercise, you will implement the backpropagation algorithm to compute the gradient for the neural network cost function. Once you have computed the gradient, you will be able to train the neural network by minimizing the cost function $J(\\theta)$ using an advanced optimization algorithm such as cg." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Sigmoid Gradient\n", "**Exercise**: Implement the gradient for the sigmoid function." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "def sigmoid_gradient(z):\n", " return z" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Initializing Parameters\n", "**Exercise**: Initialize random weights." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "def initialize_random_weights(L_in, L_out):\n", " epsilon = 0.12\n", " W = np.zeros((L_out, L_in + 1))\n", " return W" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "initial_theta_1 = initialize_random_weights(input_layer_size, hidden_layer_size)\n", "initial_theta_2 = initialize_random_weights(hidden_layer_size, K)\n", "initial_nn_parameters = np.concatenate([initial_theta_1.flatten(), initial_theta_2.flatten()])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Implement backpropogation\n", "**Exercise**: Implement back propogation in Python" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "def backprop_gradient(nn_params, X, y, input_layer_size, hidden_layer_size, K, _lambda=None):\n", " return nn_params" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "# This cell contains functions for testing the gradient. You do not have to understand them.\n", "def debug_initialize_weights(fan_out, fan_in):\n", " W = np.sin(np.arange(1, (fan_in + 1) * fan_out + 1)) / 10\n", " return W.reshape(fan_out, fan_in + 1)\n", "\n", "def compute_numerical_gradient(cost_function, nn_params, X, y, input_layer_size, hidden_layer_size, K, _lambda):\n", " numgrad = np.zeros(nn_params.shape)\n", " perturb = np.zeros(nn_params.shape)\n", " e = 1e-4\n", " for p in range(len(nn_params)):\n", " # Set pertubation vector\n", " perturb[p] = e\n", " loss_1 = cost_function(nn_params-perturb, X, y, \n", " input_layer_size=input_layer_size, \n", " hidden_layer_size=hidden_layer_size, \n", " K=K, \n", " _lambda=_lambda)\n", " loss_2 = cost_function(nn_params+perturb, X, y, \n", " input_layer_size=input_layer_size, \n", " hidden_layer_size=hidden_layer_size, \n", " K=K, \n", " _lambda=_lambda)\n", " \n", " # Compute numerical gradient\n", " numgrad[p] = (loss_2 - loss_1) / (2*e)\n", " perturb[p] = 0\n", " return numgrad\n", "\n", "def check_gradient(cost_function, gradient_function, _lambda=0):\n", " \"\"\" Check the gradient function \"\"\"\n", "\n", " # Initialize test values\n", " input_layer_size = 3\n", " hidden_layer_size = 5\n", " K = 3\n", " m = 5\n", " \n", " theta_1 = debug_initialize_weights(hidden_layer_size, input_layer_size)\n", " theta_2 = debug_initialize_weights(K, hidden_layer_size)\n", " X = debug_initialize_weights(m, input_layer_size - 1)\n", " y = 1 + np.mod(np.arange(1, m+1), K)\n", " out = np.zeros((m, K))\n", " for index in range(m):\n", " out[index][y[index] - 1] = 1\n", " y = out\n", " \n", " # Unroll parameters\n", " nn_params = np.concatenate([theta_1.flatten(), theta_2.flatten()])\n", " \n", " # Compute gradient via backprop\n", " backprop_gradient = gradient_function(nn_params, X, y,\n", " input_layer_size=input_layer_size,\n", " hidden_layer_size=hidden_layer_size,\n", " K=K,\n", " _lambda=_lambda)\n", " \n", " # Compute numerical gradient\n", " numerical_gradient = compute_numerical_gradient(cost_function, nn_params, X, y,\n", " input_layer_size=input_layer_size,\n", " hidden_layer_size=hidden_layer_size,\n", " K=K,\n", " _lambda=_lambda)\n", " \n", " # Compare the backprop and numerical gradient\n", " gradients = pd.DataFrame({'Backprop': backprop_gradient, \n", " 'Numerical': numerical_gradient, \n", " 'Difference':np.abs(backprop_gradient - numerical_gradient)})\n", " pd.options.display.max_rows = 5\n", " print(gradients)\n", " \n", " # Compute the difference\n", " diff = np.linalg.norm(numerical_gradient - backprop_gradient) / np.linalg.norm(backprop_gradient + numerical_gradient)\n", " print('If the backprop gradient is computed well, the relative diffrence will be no more than 1e-9: {}'.format(diff))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Test the backpropogation algorithm (with and without regularization)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The gradients without regularization: \n", " Backprop Numerical Difference\n", "0 0.084147 0.0 0.084147\n", "1 0.090930 0.0 0.090930\n", ".. ... ... ...\n", "36 -0.096140 0.0 0.096140\n", "37 -0.075099 0.0 0.075099\n", "\n", "[38 rows x 3 columns]\n", "If the backprop gradient is computed well, the relative diffrence will be no more than 1e-9: 1.0\n", "\n", "-------------\n", "\n", "The gradients with regularization (lambda=3): \n", " Backprop Numerical Difference\n", "0 0.084147 0.0 0.084147\n", "1 0.090930 0.0 0.090930\n", ".. ... ... ...\n", "36 -0.096140 0.0 0.096140\n", "37 -0.075099 0.0 0.075099\n", "\n", "[38 rows x 3 columns]\n", "If the backprop gradient is computed well, the relative diffrence will be no more than 1e-9: 1.0\n" ] } ], "source": [ "print('The gradients without regularization: ')\n", "check_gradient(compute_nn_cost, backprop_gradient)\n", "\n", "print('\\n-------------\\n')\n", "\n", "print('The gradients with regularization (lambda=3): ')\n", "check_gradient(compute_nn_cost, backprop_gradient, _lambda=3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training the neural network\n", "\n", "The neural network will now be trained using your functions." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Optimization terminated successfully.\n", " Current function value: 0.000000\n", " Iterations: 0\n", " Function evaluations: 1\n", " Gradient evaluations: 1\n" ] } ], "source": [ "# Get random initial values for theta\n", "initial_theta_1 = initialize_random_weights(input_layer_size, hidden_layer_size)\n", "initial_theta_2 = initialize_random_weights(hidden_layer_size, K)\n", "initial_nn_parameters = np.concatenate([initial_theta_1.flatten(), initial_theta_2.flatten()])\n", "\n", "# Set config\n", "_lambda = 1\n", "args = (X, mapped_y, input_layer_size, hidden_layer_size, K, _lambda)\n", "\n", "# Train NN\n", "result = minimize(compute_nn_cost, initial_nn_parameters, args=args,\n", " method='CG', jac=backprop_gradient, \n", " options={\"maxiter\": 50, \"disp\" : 1})\n", "nn_params = result.x\n", "theta_1 = nn_params[:hidden_layer_size * (input_layer_size + 1)].reshape((hidden_layer_size, (input_layer_size + 1)))\n", "theta_2 = nn_params[(hidden_layer_size * (input_layer_size + 1)):].reshape((K, (hidden_layer_size + 1)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualising the hidden layer\n", "\n", "You can now \"visualize\" what the neural network is learning by displaying the hidden units to see what features they are capturing in the data." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:11: RuntimeWarning: invalid value encountered in true_divide\n", " # This is added back by InteractiveShellApp.init_path()\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAEICAYAAAB/KknhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAEmBJREFUeJzt3Hu0XGV9xvHvQ8JFrklAY0i4hItaRAWbKohWF6CmiMKyFFGogaJZ9iYqVcEuW8VLddXKpdoqhWK0yMXAAoyWe2jVaiSAVUhAIgokBAKEAAJSUp7+sd8jk9NzzgwnM+ec4X0+a2XNzN579v6dd+bZ7/vus09km4ioyybjXUBEjL0EP6JCCX5EhRL8iAol+BEVSvAjKvScDL6kr0j6eI+PcZ2k95TnR0u6sgfH6Ml+OzjuAZJul/RrSYcPsf5Xkg4e5r2vk3TbCPv+mqRPj7DekvYYXeXDa/28og+DL+lySacMsfwwSfdKmmz7fbY/NVY12T7X9ps2Zh+Sdi1f+snd3O8onQJ8yfbWti95Nm+0/T3bL+5RXX1P0t6SrpD0gKRxu4mm74IPLACOkaRBy/8YONf2+nGo6blmF+CW8S6i37WexFs8BVwIHD/G5WygH4N/CbA98LqBBZKmAocCXy+vfzuclLSDpEWS1klaK+l7kjYp6zYYVg5639TyvvslPVSezxqqIEnHSvp+eS5Jp0paI+kRST+TtHdZ9xZJN5Xld0v6RMtu/rM8ritD7P1b91ve/xpJ10t6uDy+pmXddZI+JekHkh6VdKWkHYZrREnvlbSitMllknYsy38B7AZ8u9Sx+TC72EfST0stF0jaorz/DZJWthxnX0k3lpouALYYVMeHJa2WdI+kPxm0bnNJX5B0l6T7yhTuea3HkXRiaevVko4b7ucdtN/dJV0r6cHS854raUpLPRcN2v4MSaeX59tJOrscb5WkT0uaVNYdW9r/VEkPAp8YfGzbt9k+m3E+sfZd8G0/QXPGfHfL4iOBW23/9xBvORFYCTwfmA58DOhkiLUJcA5N77cz8ATwpQ7e9ybg94EXAduV2h4s6x4rdU8B3gL8qZ6ZQ/9+eZxShtg/bN2ppGnAd4AzaE58XwS+I2n7ls3eBRwHvADYDPiroQqUdCDwd6W2GcCdwPkAtncH7gLeWup4cpif80hgLjAbeDlw7BDH2YzmRP0NYBrwLeAPW9bPLTW+EdgTGHzd4HM07bgPsAcwE/iblvUvpGnjmTQ96JdLJ9COys+/I/A7wE48E9J/A+a2nAgmA0dROhXga8D6Us++NJ9367WDVwN30HzXPtNBLeOi74JfLACOGOhlaMK0YJhtn6L5cu9i+6kyB20bfNsP2r7I9uO2H6X5EF/fQW1PAdsALwFke7nt1WWf19n+me2nbf8UOK/DfUJzorjd9jdsr7d9HnAr8NaWbc6x/fOWk+M+w+zraOBfbd9Ygn0ysL+kXTusBeAM2/fYXgt8e5hj7QdsCpxW2n4hcH3L+iNLzTfbfoyWHrJM5eYDH7S9tnwGn6UJ4YCngFPKvr8L/Bpoe33B9grbV9l+0vb9NCfR15d1q2lGX39UNp8LPGD7BknTgUOAD9h+zPYa4NRBNd1j+x/LZ/REu1rGS18G3/b3gQeAwyXtDrwK+OYwm/89sAK4UtIdkk7q5BiStpT0VUl3SnqE5sswZWBYN0Jt19KMDL4MrJF0pqRtyz5fLWlxmT48DLwPGHY4PsiOND1zqztpersB97Y8fxzYupN92f41zahk5jDbD6WTY+0IrBp0or1z0Pq7h1n3fGBL4IYyTVsHXF6WD3hw0DWdkX7m35I0XdL5Zaj+CE0v3/o5LACOKc+PoRmxQDP62xRY3VLTV2lGWANaf54Jqy+DX3ydpqc/BrjC9n1DbWT7Udsn2t4NeBvwIUkHldWP03y5Bryw5fmJNL3Hq21vyzND8cEXFYc65hm2fxfYi2ao+uGy6pvAZcBOtrcDvtKyv3ajkHtovnitdgZWtaun3b4kbUUzfRjNvkayGphZeu8BOw9av9Mw6x6gmV691PaU8m87222D3YHP0rT3y8pnewwbfq6XAC8v12YOBc4ty+8GngR2aKlpW9svbXlvX/y5a78H/2DgvQw/zEfSoZL2KF++h4H/BZ4uq38CvEvSpDLfbB12b0PzxVtX5td/20lRkn6v9Oyb0szpf9NyvG2AtbZ/I+lVNHPyAfeX7XYbZtffBV4k6V2SJkt6B82JZVEndQ1yHnCcpH3KxbvPAkts/2oU+xrJD2nmw++XtKmkt9OMzgZcCBwraS9JW9LSxrafBv4FOFXSCwAkzZT05i7UtQ3NtOBhSTN55sQ8cOzfAAtpTtQ/tn1XWb4auBL4B0nbStqkXCjsdLo2cPF3C5prMEjaYoQLqD3Tt8EvX9L/Arai6UWHsydwNc0H/UPgn2wvLutOoJkjr6OZ97b+zvo04Hk0Pc+PaIaZndiW5gv7EM3Q9UGa6QbAnwGnSHqU5iLVhS0/z+M01xF+UIaR+w36eR+k6X1OLPv8CHCo7Qc6rKt1X1cDHwcuoul1d2fDeWpX2P4f4O00F/7WAu8ALm5Z/+807XwtzXTs2kG7+GhZ/qMyJL+aDubwHfgk8EqajuA7rTW1WAC8jGeG+QPeTRPaZTSf8UKaa0id2oWmQxm4qv8EMOwNT72i/EccEf+fpJ1pLp6+0PYj411Pt/Vtjx/RK2ru8/gQcP5zMfQAQ91ZFFGtcqHzPppp2txxLqdnNqrHlzRX0m1q7gDr6NdkERNZ+f381rZfarsvfjU3GqOe45ffZ/+c5q6rlTQ3ZrzT9rLulRcRvbAxQ/1XASts3wEg6XzgMJqrnUPSOP41UkQtbLe912Rjhvoz2fAupZUMceeXpPmSlkpauhHHiogu6vnFPdtnAmdCevyIiWJjevxVbHi75Sy6f8tnRPTAxgT/emBPSbPLn18exch30EXEBDHqob7t9ZL+ArgCmETzZ575X1si+sCY3rKbOX5E7/X6qn5E9KkEP6JCCX5EhRL8iAol+BEVSvAjKpTgR1QowY+oUIIfUaEEP6JCCX5EhRL8iAol+BEVSvAjKpTgR1QowY+oUIIfUaEEP6JCCX5EhRL8iAol+BEVSvAjKpTgR1QowY+oUIIfUaEEP6JCCX5EhRL8iAol+BEVSvAjKpTgR1QowY+oUIIfUaEEP6JCCX5EhdoGX9JOkhZLWibpFkknlOXTJF0l6fbyOLX35UZEN8j2yBtIM4AZtm+UtA1wA3A4cCyw1vbnJJ0ETLX90Tb7GvlgEbHRbKvdNm17fNurbd9Ynj8KLAdmAocBC8pmC2hOBhHRByY/m40l7QrsCywBptteXVbdC0wf5j3zgfmjLzEiuq3tUP+3G0pbA/8BfMb2xZLW2Z7Ssv4h2yPO8zPUj+i9rgz1ASRtClwEnGv74rL4vjL/H7gOsGa0hUbE2Orkqr6As4Hltr/YsuoyYF55Pg+4tPvlRUQvdHJV/7XA94CfAU+XxR+jmedfCOwM3AkcaXttm31lqB/RY50M9Tue43dDgh/Re12b40fEc0uCH1GhBD+iQgl+RIUS/IgKJfgRFUrwIyqU4EdUKMGPqFCCH1GhBD+iQgl+RIUS/IgKJfgRFUrwIyqU4EdUKMGPqFCCH1GhBD+iQgl+RIUS/IgKJfgRFUrwIyqU4EdUKMGPqFCCH1GhBD+iQgl+RIUS/IgKJfgRFUrwIyqU4EdUKMGPqFCCH1GhjoMvaZKkmyQtKq9nS1oiaYWkCyRt1rsyI6Kbnk2PfwKwvOX154FTbe8BPAQc383CIqJ3Ogq+pFnAW4CzymsBBwILyyYLgMN7UWBEdF+nPf5pwEeAp8vr7YF1tteX1yuBmUO9UdJ8SUslLd2oSiOia9oGX9KhwBrbN4zmALbPtD3H9pzRvD8ium9yB9scALxN0iHAFsC2wOnAFEmTS68/C1jVuzIjopva9vi2T7Y9y/auwFHAtbaPBhYDR5TN5gGX9qzKiOiqjfk9/keBD0laQTPnP7s7JUVEr8n22B1MGruDRVTKttptkzv3IiqU4EdUKMGPqFCCH1GhBD+iQgl+RIUS/IgKJfgRFUrwIyqU4EdUKMGPqFCCH1GhBD+iQgl+RIUS/IgKJfgRFUrwIyqU4EdUKMGPqFCCH1GhBD+iQgl+RIUS/IgKJfgRFUrwIyqU4EdUKMGPqFCCH1GhBD+iQgl+RIUS/IgKJfgRFUrwIyqU4EdUKMGPqFBHwZc0RdJCSbdKWi5pf0nTJF0l6fbyOLXXxUZEd3Ta458OXG77JcArgOXAScA1tvcErimvI6IPyPbIG0jbAT8BdnPLxpJuA95ge7WkGcB1tl/cZl8jHywiNppttdumkx5/NnA/cI6kmySdJWkrYLrt1WWbe4HpQ71Z0nxJSyUt7bTwiOitTnr8OcCPgANsL5F0OvAI8Je2p7Rs95DtEef56fEjeq9bPf5KYKXtJeX1QuCVwH1liE95XDPaQiNibLUNvu17gbslDczfDwKWAZcB88qyecClPakwIrqu7VAfQNI+wFnAZsAdwHE0J40LgZ2BO4Ejba9ts58M9SN6rJOhfkfB75YEP6L3ujXHj4jnmAQ/okIJfkSFEvyICiX4ERVK8CMqlOBHVCjBj6hQgh9RoQQ/okIJfkSFEvyICiX4ERVK8CMqlOBHVCjBj6hQgh9RoQQ/okIJfkSFEvyICiX4ERVK8CMqlOBHVCjBj6hQgh9RoQQ/okIJfkSFEvyICiX4ERVK8CMqlOBHVCjBj6hQgh9RoQQ/okIdBV/SByXdIulmSedJ2kLSbElLJK2QdIGkzXpdbER0R9vgS5oJvB+YY3tvYBJwFPB54FTbewAPAcf3stCI6J5Oh/qTgedJmgxsCawGDgQWlvULgMO7X15E9ELb4NteBXwBuIsm8A8DNwDrbK8vm60EZg71fknzJS2VtLQ7JUfExupkqD8VOAyYDewIbAXM7fQAts+0Pcf2nFFXGRFd1clQ/2Dgl7bvt/0UcDFwADClDP0BZgGrelRjRHRZJ8G/C9hP0paSBBwELAMWA0eUbeYBl/amxIjoNtluv5H0SeAdwHrgJuA9NHP684FpZdkxtp9ss5/2B4uIjWJb7bbpKPjdkuBH9F4nwc+dexEVSvAjKpTgR1QowY+oUIIfUaEEP6JCCX5EhRL8iAol+BEVSvAjKpTgR1QowY+oUIIfUaEEP6JCCX5EhRL8iAol+BEVSvAjKpTgR1QowY+oUIIfUaEEP6JCCX5EhRL8iAol+BEVSvAjKpTgR1QowY+oUIIfUaEEP6JCCX5EhRL8iAol+BEVSvAjKpTgR1QowY+oUIIfUaHJY3y8B4DHymM/2IH+qRX6q95+qhX6p95dOtlItntdyIYHlJbanjOmBx2lfqoV+qvefqoV+q/edjLUj6hQgh9RofEI/pnjcMzR6qdaob/q7adaof/qHdGYz/EjYvxlqB9RoQQ/okJjFnxJcyXdJmmFpJPG6ridkrSTpMWSlkm6RdIJZfk0SVdJur08Th3vWgdImiTpJkmLyuvZkpaUNr5A0mbjXeMASVMkLZR0q6TlkvafqG0r6YPlO3CzpPMkbTGR23Y0xiT4kiYBXwb+ANgLeKekvcbi2M/CeuBE23sB+wF/Xmo8CbjG9p7ANeX1RHECsLzl9eeBU23vATwEHD8uVQ3tdOBy2y8BXkFT94RrW0kzgfcDc2zvDUwCjmJit+2zZ7vn/4D9gStaXp8MnDwWx96Imi8F3gjcBswoy2YAt413baWWWTRhORBYBIjmzrLJQ7X5ONe6HfBLysXkluUTrm2BmcDdwDSaO1sXAW+eqG072n9jNdQfaMwBK8uyCUnSrsC+wBJguu3VZdW9wPRxKmuw04CPAE+X19sD62yvL68nUhvPBu4HzilTk7MkbcUEbFvbq4AvAHcBq4GHgRuYuG07Krm4N4ikrYGLgA/YfqR1nZvT/bj//lPSocAa2zeMdy0dmgy8Evhn2/vS/L3GBsP6CdS2U4HDaE5WOwJbAXPHtageGKvgrwJ2ank9qyybUCRtShP6c21fXBbfJ2lGWT8DWDNe9bU4AHibpF8B59MM908Hpkga+MOridTGK4GVtpeU1wtpTgQTsW0PBn5p+37bTwEX07T3RG3bURmr4F8P7FmujG5Gc7HksjE6dkckCTgbWG77iy2rLgPmlefzaOb+48r2ybZn2d6Vpi2vtX00sBg4omw2IWoFsH0vcLekF5dFBwHLmIBtSzPE30/SluU7MVDrhGzbURvDiyaHAD8HfgH89Xhf3BiivtfSDDV/Cvyk/DuEZu58DXA7cDUwbbxrHVT3G4BF5fluwI+BFcC3gM3Hu76WOvcBlpb2vQSYOlHbFvgkcCtwM/ANYPOJ3Laj+ZdbdiMqlIt7ERVK8CMqlOBHVCjBj6hQgh9RoQQ/okIJfkSF/g8+mnBqooZ4FwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# get 100 random images from the dataset\n", "num_samples = 100\n", "hidden_unit_visual = theta_1[:, 1:]\n", "display_img = Image.new('RGB', (100, 100))\n", "\n", "# loop over the images, turn them into a PIL image\n", "i = 0\n", "for col in range(5):\n", " for row in range(5):\n", " array = hidden_unit_visual[i]\n", " array = ((array / max(array)) * 255).reshape((20, 20)).transpose() # redistribute values\n", " img = Image.fromarray(array)\n", " display_img.paste(img, (col*20, row*20))\n", " i += 1\n", "\n", "# present display_img\n", "plt.title('Visualisation of hidden layer 1')\n", "plt.imshow(display_img, interpolation='nearest')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Evaluating the model\n", "\n", "Get the accuracy on the training set for the trained values of theta. According to the exercise, you should have an accuracy of about 95%. However, this may vary due to the random initalization." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Training set accuracy using the a neural network with the trained values for theta: 10.0%'" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Make sure to add 1 to the result as `y` is one indexed while the prediction is 0 indexed.\n", "layer2_activation = add_bias(forward(theta_1, add_bias(X).T).T).T\n", "predictions = forward(theta_2, layer2_activation).T\n", "\n", "accuracy = np.mean(np.argmax(predictions, axis = 1) + 1 == y) * 100\n", "'Training set accuracy using the a neural network with the trained values for theta: {:2}%'.format(accuracy)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: ex5/PE5 - Logistic Regression (Exercises).ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Regularized Linear Regression and Bias v.s. Variance\n", "\n", "Stanford CS229 - Machine Learning by Andrew Ng. Programming exercise 5.\n", "\n", "Please check out [the repository on GitHub](https://github.com/rickwierenga/CS229-Python/). If you spot any mistakes or inconcistencies, please create an issue. For questions you can find me on Twitter: [@rickwierenga](https://twitter.com/rickwierenga). Starring the project on GitHub means a ton to me!" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pylab as plt\n", "from scipy.optimize import minimize\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Regularized Linear Regression\n", "\n", "---\n", "In the first half of the exercise, you will implement regularized linear regres- sion to predict the amount of water flowing out of a dam using the change of water level in a reservoir. In the next half, you will go through some diag- nostics of debugging learning algorithms and examine the effects of bias v.s. variance.\n", "\n", "Start by loading the data." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import scipy.io as sio\n", "\n", "# Load data\n", "data = sio.loadmat(\"ex5data1.mat\")\n", "X = data[\"X\"]\n", "m, n = X.shape\n", "y = data[\"y\"].reshape(m)\n", "Xval = data['Xval']\n", "mval, nval = Xval.shape\n", "yval = data['yval'].reshape(mval)\n", "\n", "# Add bias to X\n", "X = np.hstack((np.ones((m, 1)), X))\n", "Xval = np.hstack((np.ones((mval, 1)), Xval))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualising the data" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Text(0, 0.5, 'Water flowing out of the dam (y)')" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAHwtJREFUeJzt3XmYHFW9//H3h00UUOAyYARC2PkBIssMcMVtUBBX1EeiqMgVfkajXFFQBL2JsrgQFZSfkp+5gEYviOPChYu4RBgElGUmEJKwCSJBEE1QQBRFwO/945w2nTDTXZPpqp6e/ryep5/uOlVd9e16evo755yqcxQRmJlZ91qr3QGYmVl7ORGYmXU5JwIzsy7nRGBm1uWcCMzMupwTgZlZl3MiMDPrck4EZmZdzonAzKzLrdPuAIrYbLPNYtq0ae0Ow8ysoyxcuPDBiOhptl1HJIJp06YxPDzc7jDMzDqKpGVFtnPTkJlZl3MiMDPrck4EZmZdzonAzKzLORGYmXU5JwIzs4lmzhwYHFy1bHAwlZfAicDMbKLp64Pp01cmg8HBtNzXV8rhOuI+AjOzrtLfDwMD6cd/5kyYOzct9/eXcjjXCMzMJqL+/pQETj01PZeUBMCJwMxsYhocTDWBWbPS8+p9Bi3kRGBmNtHU+gQGBuCUU1Y2E5WUDJwIzMwmmqGhVfsEan0GQ0OlHE4RUcqOW6m3tzc86JyZ2dhIWhgRvc22c43AzKzLORGYmXW50hKBpPUl3SDpZkm3SDo5l39d0q8lLcqPPcuKwczMmivzhrLHgQMj4s+S1gWukfTDvO4jEfHdEo9tZmYFlZYIIvVC/zkvrpsfE79n2sysy5TaRyBpbUmLgOXAgoi4Pq/6lKTFks6U9IxR3jtD0rCk4RUrVpQZpplZVys1EUTEUxGxJ7AVsK+k3YGTgF2APmBT4KOjvHdeRPRGRG9PT9O5l83MOlPFI42OpJKrhiLiYWAQOCQiHojkceBrwL5VxGBmNiFVPNLoSMq8aqhH0sb59TOBg4DbJU3JZQLeACwtKwYzswmvfqTR2bNXDi1R4iBzqyvzqqEpwHxJa5MSzkBEXCrpCkk9gIBFwHtLjMHMbOKrH2l01qxKkwCUe9XQYmCvEcoPLOuYZmYdafWRRvv7K00GvrPYzKydKh5pdCROBGZm7VTxSKMj8eijZmaTVNHRR5v2EUhaC3gB8Dzgr8DSiFg+/hDNzGwiGDURSNqedLPXK4A7gRXA+sBOkh4DvgrMj4h/VBGomZmVo1GN4DRgLvCeWK39SNLmwNuAI4D55YVnZmZlGzURRMThDdYtB75YSkRmZlapplcNSVoo6f2SNqkiIDMzq1aRy0ffQuooHpJ0oaRX5uEhzMxsEmiaCCLiroj4OLATcAFwHrBM0smSNi07QDMzK1ehG8ok7QF8Afgc8D3gMOBPwBXlhWZmZlUoch/BQuBh4FzgxDx8NMD1kg4oMzgzMytfkUHnDouIu0daERFvanE8ZmZWsVGbhiS9Q9JaoyUBSdtLelF5oZmZWRUa1Qj+BbgpNw0tZOWdxTsALwUeBE4sPUIzMytVoxvKviTpy8CBwAHAHqSxhm4DjoiIe6sJ0czMytSwjyAingIW5IeZmU1Cno/AzKzLlTl5/fqSbpB0s6RbJJ2cy7eVdL2kuyR9W9J6ZcVgZmbNlVkjeBw4MCJeAOwJHCJpf+B04MyI2AF4CDi6xBjMzKyJIjeUbQy8E5hWv31EfKDR+/LQ1X/Oi+vmR5A6n9+Wy+cDnyQNd21mZm1Q5Iayy4DrgCXAmCahkbQ26dLTHYCvAL8CHo6IJ/Mm9wFbjmWfZmbWWkUSwfoRcdya7DxfdbRnrlVcBOxS9L2SZgAzAKZOnbomhzczswKK9BF8U9K7JU2RtGntMZaDRMTDwCDwr8DGkmoJaCvg/lHeMy8ieiOit6enZyyHMzOzMSiSCP5OGnX0WlIzz0JguNmbJPXkmgCSngkcRLoZbRB4c97sSODisYdtZmatUqRp6Hhgh4h4cIz7ngLMz/0EawEDEXGppFuBCyWdBtxEGtXUzMzapEgiuAt4bKw7jojFwF4jlN8N7DvW/ZmZWTmKJIK/AIskDZLuDQCaXz5qZmadoUgi+O/8MDOzSahpIoiI+VUEYmZm7VHkzuIdgc8Au5LmIwAgIrYrMS4zM6tIkctHv0YaAuJJoB/4BvBfZQZlZmbVKZIInhkRlwOKiGUR8UngNeWGZWZmVSnSWfy4pLWAOyUdQ7oTeMNywzIzs6oUqREcCzwL+ACwD3AE6Y5gMzObBIpcNTSUX/4ZeFe54ZiZWdVGTQSS/oc0f8CIIuL1pURkZmaValQj+Hx+fhPwXFZeKXQ48PsygzIzs+qMmggi4mcAkr4QEb11q/5HUtPRR83MrDMU6SzeQNI/bx6TtC2wQXkhmZlZlYpcPvoh4EpJdwMCtiHPHGZmZp2vyFVDP8rDTNSmmbw9Ih5v9B4zM+scRWoE5B/+m0uOxczM2qBIH4GZmU1iTgRmZl2uaSJQ8g5Js/PyVEmeatLMbJIoUiM4G/hX0o1kAI8CX2n2JklbSxqUdKukWyQdm8s/Kel+SYvy49VrHL2ZmY1bkc7i/SJib0k3AUTEQ5LWK/C+J4HjI+JGSRsBCyUtyOvOjIjPN3ivmZlVpEgieELS2uRxhyT1AP9o9qaIeAB4IL9+VNJtwJbjiNXMzEpQpGnoLOAiYHNJnwKuAT49loNImgbsBVyfi46RtFjSeZI2Gcu+zMystRQx6gCjKzeSdgFeTrqz+PKIuK3wAaQNgZ8Bn4qI70vaAniQVMM4FZgSEUeN8L4Z5DuYp06dus+yZcuKHtLMzABJC1cbK27k7QomgrWBLahrSoqIewu8b13gUuDHEXHGCOunAZdGxO6N9tPb2xvDwx7nzsxsLIomgqZ9BJL+HfgEaejpp0i1ggD2aPI+AecCt9UnAUlTcv8BwBuBpc1iMDOz8hTpLD4W2Dki/jDGfR9AmtZyiaRFuexjwOGS9iQlk3uA94xxv2Zm1kJFEsFvgEfGuuOIuIZUe1jdZWPdl5mZlafRVJXH5Zd3k4ah/gHwz1FHR2rzNzOzztOoRrBRfr43P9bLD2gwl7GZmXWWRlNVngwg6bCI+E79OkmHlR2YmZlVo8gNZScVLDMzsw7UqI/gVcCrgS0lnVW36tmkcYTMzNpvzhzo64P+/pVlg4MwNAQnnNC+uDpIoxrBb4Fh4G/AwrrHJcAryw/NzKyAvj6YPj39+EN6nj49lVshjfoIbgZulnRBRDxRYUxmZsX198PAQPrxnzkT5s5Ny/U1BGuoaR+Bk4CZTXj9/SkJnHpqenYSGBNPVWlmnW9wMNUEZs1Kz7VmIitk1EQg6Zv5+djqwjEzG6Nan8DAAJxyyspmIieDwhrVCPaR9DzgKEmbSNq0/lFVgGZmDQ0NrdonUOszGBpqb1wdZNRhqCV9AJgJbAfcz6rjBkVEbFd+eImHoTYzG7uiw1CPWiOIiLMi4v8A50XEdhGxbd2jsiRgZmblajr6aETMlPQC4MW56KqIWFxuWGZmVpWmVw3lJqLzgc3z4/w8WY2ZmU0CReYj+L/AfhHxFwBJpwPXAv+vzMDMzKwaRe4jEGmKypradJVmZjYJFKkRfA24XtJFefkNpLmIzcxsEijSWXyGpCuBF+Wid0XETaVGZWZmlSlSIyAibgRuHMuOJW0NfAPYgjSj2byI+FK+Ge3bwDTS5PXTI+KhsezbzMxap8yxhp4Ejo+IXYH9gfdL2hU4Ebg8InYELs/LZmbWJqUlgoh4INckiIhHgduALYFDgfl5s/mkPgczM2uTIvcRnF6krMk+pgF7AdcDW0TEA3nV70hNR2Zm1iZFagQHjVD2qqIHkLQh8D3ggxHxp/p1kQY6GnGwI0kzJA1LGl6xYkXRw5mZ2Rg1GoZ6pqQlwM6SFtc9fg0UGmJC0rqkJHB+RHw/F/9e0pS8fgqwfKT3RsS8iOiNiN6enp6xfCYzMxuDRlcNXQD8EPgMq3boPhoRf2y2Y0ki3W9wW0ScUbfqEuBI4LP5+eKxBm1mZq3TaM7iR4BHJH10tVUbStowIu5tsu8DgCOAJZIW5bKPkRLAgKSjgWXA9DUL3czMWqHIfQQ/ILXjC1gf2Ba4A9it0Zsi4hpGH4ri5WOI0czMSlTkzuLn1y9L2ht4X2kRmZlZpcZ8H0G+N2C/EmIxM7M2aFojkHRc3eJawN7Ab0uLyMzMKlWkj2CjutdPkvoMvldOOGZmVrUifQQnwz9vDCMi/lx2UGZmVp0iQ0zsLukm4BbgFkkLJe1efmhmZlaFIp3F84DjImKbiNgGOD6XmZnZJFAkEWwQEYO1hYi4EtigtIjMzKxSRTqL75Y0C/hmXn4HcHd5IZmZWZWK1AiOAnqA75OuFtosl5mZ2SRQ5Kqhh4APVBCLmZm1QZlTVZqZWQdwIjAz63JOBGZmXa7IWENnjVD8CDAcEZ5UxsyswxWpEawP7AncmR97AFsBR0v6YomxmZlZBYrcR7AHcEBEPAUgaS5wNfAiYEmJsZmZWQWK1Ag2ATasW94A2DQnhsdLicrMzCpTpEYwB1gk6UrS1JMvAT4taQPgpyXGZmZmFWhaI4iIc4EXAv8NXAS8KCLOiYi/RMRHRnufpPMkLZe0tK7sk5Lul7QoP17dig9hZmZrrujlo2sBK4CHgB0kvaTAe74OHDJC+ZkRsWd+XFbw+GZmVpIil4+eDryFNB/BP3JxAFc1el9EXCVp2jjjM7NONWcO9PVBf//KssFBGBqCE05oX1z2NEVqBG8Ado6I10TE6/Lj9eM45jGSFuemo01G20jSDEnDkoZXrFgxjsOZWVv09cH06enHH9Lz9Omp3CaUIongbmDdFh1vLrA96b6EB4AvjLZhRMyLiN6I6O3p6WnR4c2sMv39MDCQfvxnz07PAwOr1hBsQihy1dBjpKuGLqfuctGIGPOIpBHx+9prSf8JXDrWfZhZB+nvh5kz4dRTYdYsJ4EJqkgiuCQ/xk3SlIh4IC++EVjaaHsz63CDgzB3bkoCc+emROBkMOEUmY9g/prsWNK3gJcBm0m6D/gE8DJJe5I6m+8B3rMm+zazDlDrE6g1B/X3u3logho1EUgaiIjpkpaQfrhXERF7NNpxRBw+QvG5Yw/RzDrS0NCqP/q1PoOhISeCCUYRT/uNTytyM46kbUZaHxHLSo2sTm9vbwwPD1d1ODOzSUHSwojobbbdqDWCurb8VwBXRcSdrQrOzMwmjiKdxVOBr+abwxaSbiS7OiIWlRiXmZlVpMhYQ5+IiAOB3UjDT3+ElBDMzGwSKDLExH8AB5CGor4J+DApIZiZ2SRQpGnoTcCTwA+AnwHXRoTnITAzmySKNA3tTeowvgE4CFgi6ZqyAzMzs2oUaRraHXgx8FKgF/gNbhoyM5s0ijQNfZb0w38WMBQRT5QbkpmZVanIEBOvlbQesBOws6Q7nAzMzCaPIk1DLwW+QRobSMDWko6MiIYT05iZWWco0jR0BnBwRNwBIGkn4FvAPmUGZmZm1SgyMc26tSQAEBG/pHUT1ZiZWZsVqREMSzoH+K+8/HbAI8CZmU0SRRLBTOD9QG1GsquBs0uLyMzMKlXkqqHHSf0EZ5QfjpmZVa3RxDQjTkhT02xiGjMz6wyNagSHAX+tKhAzM2uPRlcNXZBnITstIpat/mi2Y0nnSVouaWld2aaSFki6Mz9v0ooPYWZma65RIlhP0tuAF0p60+qPAvv+OnDIamUnApdHxI7A5XnZzMzaqFHT0HtJl4puDLxutXUBfL/RjiPiqjyrWb1DgZfl1/OBK4GPForUzMxK0WjO4muAayQNR8S5LTreFnVzIf8O2KJF+zUzszVUZD6CViWB1fcbNLgqSdIMScOShlesWFFGCGZmRrEhJlrp95KmAOTn5aNtGBHzIqI3Inp7enoqC9DMrNs0TARKtm7h8S4BjsyvjwQubuG+zcxsDTRMBLn55rI12bGkbwHXkuYwuE/S0aRJbg6SdCdp+svPrsm+zcysdYqMNXSjpL6IGBrLjiPi8FFWvXws+zEzs3IVSQT7AW+XtAz4C2lymvAQE2Zmk0ORRPDK0qMwM7O2KXL56DJga+DA/PqxIu8zM7PO0PQHXdInSHf/npSL1mXlJDVmZtbhivxn/0bg9aT+ASLit8BGZQZlZmbVKZII/l5/F7CkDcoNycxaas4cGBxctWxwMJWbUSwRDEj6KrCxpHcDPwXOKTcsM2uZvj6YPn1lMhgcTMt9fe2NyyaMIlNVfl7SQcCfgJ2B2RGxoPTIzKw1+vthYCD9+M+cCXPnpuX+/nZHZhNE00Qg6fSI+CiwYIQyM+sE/f0pCZx6Ksya5SRgqyjSNHTQCGWvanUgZlaiwcFUE5g1Kz2v3mdgXa3R5PUzgfcB20laXLdqI+DnZQdmZi1S6xOoNQf196+6bF2vUdPQBcAPgc+w6pSSj0bEH0uNysxaZ2ho1R/9Wp/B0JATgQGgdGVogQ2lzYH1a8sRcW9ZQa2ut7c3hoeHqzqcmdmkIGlhRPQ2267IncWvy8NG/xr4GXAPqaZgZmaTQJHO4tOA/YFfRsS2pGGkrys1KjMzq0yRRPBERPwBWEvSWhExCDStapiZWWcoMgz1w5I2BK4Czpe0nDzukJmZdb4iNYJDgb8CHwJ+BPwKeF2ZQZmZWXUa3UfwQeAXwI0R8VQunl9JVGZmVplGTUNbAV8EdpG0hHQT2S+AX4z3PgJJ9wCPAk8BTxa5vMnMzMoxaiKIiA8DSFqP1Dn8QuBdwDxJD0fEruM8dn9EPDjOfZiZ2TgV6Sx+JvBs4Dn58VtgSZlBmZlZdRr1EcwDdiM14VxPahY6IyIeasFxA/iJpAC+GhHzWrBPMzNbA41qBFOBZwB3AvcD9wEPt+i4L4qI+/OwFQsk3R4RV9VvIGkGMANg6tSpLTqsmZmtbtTLRyPiEKAP+HwuOh4YkvQTSSeP56ARcX9+Xg5cBOw7wjbzIqI3Inp7enrGczgzM2ug4X0EkSwFLiONL/RzYHvg2DU9oKQNJG1Uew0cDCxd0/2Zmdn4NOoj+ADpSqEXAk+QLx0FzmN8ncVbABdJqh3/goj40Tj2Z2Zm49Coj2Aa8B3gQxHxQKsOGBF3Ay9o1f5aZs6cNJl3/fjsg4NpzPYTTmhfXNZ9/F20ijXqIzguIr7XyiQwofX1pVmbalP41WZ16utrb1zWffxdtIoVuY+gO9RmbZo+PU3yPXeup/Kz9vB30SpWZNC57tHfn/7wTj01PfsPz9rF30WrkBNBvcHB9N/XrFnpuVY1N6uav4tWocmZCObMefofzuBgKh9NrR12YABOOWVl1dx/gFY1fxetYpMzEaxJZ9vQ0KrtsLV22qGh8uNd3ZokMps8JtJ30bqCIqLdMTTV29sbw8PDY3tT7ce/Ezvb6v8j7O9/+rKZWQGSFhYZ5n9y1gigszvb6q8amT3bScDMSjV5E0Gnd7Z1ciJrlSqayNwMZzZJE8Fk6Gzr9ETWClXcWOWbt8wgIib8Y5999okxOf30iCuuWLXsiitSeSe44oqIzTZb+RlWX+4mtc8+a1Z556CKY5i1ATAcBX5jJ29ncSfzWDOrmj07NZHNmpVqeJ16DLOKFe0sdiKwia2Kq786+QozswZ81ZAlndwZWkVfz2ToTzIbJyeCya6TO0OruLHKN2+ZuWmoK7jpw6wruWnIVvI9CWbWgBNBN/A9CWbWgBPBZOfOUDNroi2JQNIhku6QdJekE9sRQ9dwZ6iZNVF5Z7GktYFfAgcB9wFDwOERceto73FnsZnZ2E3kzuJ9gbsi4u6I+DtwIXBoG+IwMzPakwi2BH5Tt3xfLluFpBmShiUNr1ixorLgzMy6zYTtLI6IeRHRGxG9PT097Q7HzGzSakciuB/Yum55q1xmZmZt0I5EMATsKGlbSesBbwUuaUMcZmZGm4aYkPRq4IvA2sB5EfGpJtuvAJZVEVuLbAY82O4gJiifm9H53IzO52Z0jc7NNhHRtG29I8Ya6jSShotcstWNfG5G53MzOp+b0bXi3EzYzmIzM6uGE4GZWZdzIijHvHYHMIH53IzO52Z0PjejG/e5cR+BmVmXc43AzKzLORGUQNLxkkLSZnlZks7Ko60ulrR3u2OsmqTPSbo9f/6LJG1ct+6kfG7ukPTKdsbZLh6RdyVJW0salHSrpFskHZvLN5W0QNKd+XmTdsfaLpLWlnSTpEvz8raSrs/fn2/ne7QKcyJoMUlbAwcD99YVvwrYMT9mAHPbEFq7LQB2j4g9SKPPngQgaVfSTYW7AYcAZ+cRartG/rxfIX1PdgUOz+elWz0JHB8RuwL7A+/P5+NE4PKI2BG4PC93q2OB2+qWTwfOjIgdgIeAo8eyMyeC1jsTOAGo73w5FPhGJNcBG0ua0pbo2iQifhIRT+bF60hDi0A6NxdGxOMR8WvgLtIItd3EI/LWiYgHIuLG/PpR0g/elqRzMj9vNh94Q3sibC9JWwGvAc7JywIOBL6bNxnzuXEiaCFJhwL3R8TNq60qNOJqFzkK+GF+7XPjczAqSdOAvYDrgS0i4oG86nfAFm0Kq92+SPpn8x95+V+Ah+v+0Rrz92ed1sXWHST9FHjuCKs+DnyM1CzUlRqdm4i4OG/zcVLV//wqY7POI2lD4HvAByPiT+kf3yQiQlLXXfIo6bXA8ohYKOllrdqvE8EYRcQrRiqX9HxgW+Dm/IXdCrhR0r50yYiro52bGkn/BrwWeHmsvG65K85NEz4Hq5G0LikJnB8R38/Fv5c0JSIeyE2ry9sXYdscALw+j9e2PvBs4Euk5uZ1cq1gzN8fNw21SEQsiYjNI2JaREwjVc/2jojfkUZXfWe+emh/4JG6Km5XkHQIqTr7+oh4rG7VJcBbJT1D0rakDvUb2hFjG3lE3jq5zftc4LaIOKNu1SXAkfn1kcDFVcfWbhFxUkRslX9j3gpcERFvBwaBN+fNxnxuXCOoxmXAq0kdoY8B72pvOG3xZeAZwIJcY7ouIt4bEbdIGgBuJTUZvT8inmpjnJWLiCclHQP8mJUj8t7S5rDa6QDgCGCJpEW57GPAZ4EBSUeTRiOe3qb4JqKPAhdKOg24iZRIC/OdxWZmXc5NQ2ZmXc6JwMysyzkRmJl1OScCM7Mu50RgZtblnAhs3CQ9V9KFkn4laaGkyyTtJOlltdER203SKZIa3vDWouNsLOl9LdjPlZLaMkevpL0kNbz8UNIxko6qKiYrlxOBjUu++eci4MqI2D4i9iGNLDqhxoGJiNkR8dMKDrUxMKZEkG80nEh/ix8DzmqyzXnAv1cQi1VgIn35rDP1A09ExP+vFUTEzRFxdV7cUNJ381wE5+fEgaTZkoYkLZU0r678SkmnS7pB0i8lvTiXP0vSQB6j/qI89npvXnewpGsl3SjpO3mMmlVI+rqkN+fX90g6OW+/RNIuI2z/A0l75Nc3SZqdX58i6d2SNpR0ed0+aqOFfhbYXtIiSZ/L7/lI/qyLJZ2cy6YpzT/wDWApqw4xsXosT/t8SvMXfKdum3/WvkY7HwU/90bAHrWBEyV9qe6zv1LSVZLWyneH36M0hIp1OCcCG6/dgYUN1u8FfJA0zv52pLtGAb4cEX0RsTvwTNIYRDXrRMS++X2fyGXvAx7KY9TPAvYBUJr85z+AV0TE3sAwcFyBuB/M288FPjzC+quBF0t6DumO51rcLwauAv4GvDHvox/4Qk5mJwK/iog9I+Ijkg4mDZuxL7AnsI+kl+R97QicHRG7RcSykYJs8Pl+CuwnaYO86VtId5Y2Ox/NPncvKTHVnAS8RVI/qZbwroiojXo5nM+HdTgnAivbDRFxX/7xWARMy+X9+b/6JaSx1Here09tkLGFddu/iDROPxGxFFicy/cnJZmf5+EIjgS2KRDXSMeodzXwElIC+AGpZvMsYNuIuAMQ8GlJi0k/ylsycnPYwflxE3AjsAspAQAsy/NTNDLi58uDi/0IeJ2kdUjj01882vZj+NxTgBW1hfyf/7tJEwt9OSJ+VbftcuB5TeK3DuCxhmy8bmHlYFcjebzu9VPAOpLWB84GeiPiN5I+SRpJcfX3PEXz76iABRFx+Jiibn6MIdJ/x3eTfgQ3I/0g1mo/bwd6gH0i4glJ96z2Gerj+0xEfHWVwjTO/l8KxNno810IHAP8ERiOiEdzraTR+Wj2uf/K0z/H84E/8PQf/fXz9tbhXCOw8boCeIakGbUCSXvU2vZHUfuheTC3XzdKJDU/Jw8ypjRt4fNz+XXAAZJ2yOs2kLTTGD/D0+SZwn4DHAZcS6ohfJjULATwHNK48E/kZpPaf92PAhvV7erHwFF17fRbStp8DKE0+nw/A/YmJagLC2xfxG3ADrUFSdsAx5Oa+F4lab+6bXdi1WYk61BOBDYueV6BNwKvULp89BbgM6QZpEZ7z8PAf5J+RH5M+u+7mbOBHkm3AqeRaiKPRMQK4N+Ab+VmmmtJzS+tcDXpx/6v+fVW+RnSxDq9uWnrncDtABHxB1KzzFJJn4uInwAXANfmbb/LqomioUafL4/SeilpruNLm21f8Hi3A8+RtFGuXZwLfDgifkuaB/ecXKOD1Gy2oOi+beLy6KPWEZQmeF83Iv4maXtSu/zO+T93ayFJHwIejYhzGmyzF3BcRBxRXWRWFvcRWKd4FjCoNHOVgPc5CZRmLqlJrJHNSFdv2STgGoGZWZdzH4GZWZdzIjAz63JOBGZmXc6JwMysyzkRmJl1OScCM7Mu97/G/GxfBmS6PAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(X[:, 1], y, 'rx')\n", "plt.xlabel('Change in water leven (x)')\n", "plt.ylabel('Water flowing out of the dam (y)')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Recall the cost function for regularized linear regression: \n", "\n", "$$J(\\theta) = \\frac{1}{m}(\\displaystyle\\sum_{i=1}^{m}(h_\\theta(x^{(i)}) - y^{(i)})^2) + \\frac{\\lambda}{m}\\displaystyle\\sum_{j=1}^{n}{\\theta_j}^2$$\n", "\n", "**Exercise**: Write a regularized vectorized linear regression function." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def linear_reg_cost_function(theta, X, y, _lambda):\n", " m = len(y)\n", " cost = 0\n", " regularization = 0\n", " return cost + regularization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "According to the exercise, using ones for theta should return 303.993 as intial cost." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "initial_theta = np.ones(2)\n", "linear_reg_cost_function(initial_theta, X, y, _lambda=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Regularized linear regression gradient\n", "\n", "The partial derrivatives for $\\theta_j$:\n", "\n", "$$\\frac{\\delta J(\\theta)}{\\delta\\theta_j} = \\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}(h_\\theta(x^{(i)}) - y^{(i)})x_i^{(i)}$$ for $j=0$\n", "\n", "$$\\frac{\\delta J(\\theta)}{\\delta\\theta_j} = (\\frac{1}{m}\\displaystyle\\sum_{i=1}^{m}(h_\\theta(x^{(i)}) - y^{(i)})x_i^{(i)}) + \\frac{\\lambda}{m}\\theta_j$$ for $j\\geqslant 1$\n", "\n", "Vectorized: \n", "\n", "$$\\frac{\\delta J(\\theta)}{\\delta\\theta_j} = \\frac{1}{m} \\cdot X^T \\cdot (X\\theta - \\vec{y}) $$\n", "\n", "**Exercise**: Find the partial derrivatives of $J(\\theta)$. Your code should not contain any loops." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def compute_gradient(theta, X, y, _lambda):\n", " hx = X @ theta\n", " cost = (1/m) * X.T @ (hx - y)\n", " regularization = (_lambda/m) * np.concatenate(([0], theta[1:]))\n", " return cost + regularization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should get the following values: $\\begin{bmatrix}-15.30 && 598.250 \\end{bmatrix}$." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([-15.30301567, 598.25074417])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "compute_gradient(initial_theta, X, y, _lambda=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training the mdoel\n", "Now we can train the model using `scipy.optimize.minimize`. In this implementation, we set $\\lambda$ to 0." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Warning: Desired error not necessarily achieved due to precision loss.\n", " Current function value: 0.000000\n", " Iterations: 0\n", " Function evaluations: 113\n", " Gradient evaluations: 101\n" ] }, { "data": { "text/plain": [ "array([1., 1.])" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "_lambda = 0\n", "args = (X, y, _lambda)\n", "result = minimize(linear_reg_cost_function, initial_theta, args=args,\n", " method='CG', jac=compute_gradient,\n", " options={'maxiter': 50, 'disp': True})\n", "theta = result.x\n", "theta" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualize the found $\\theta$\n", "\n", "Althought the found value is not a good value, it's the best conjugate gradient could find using our model." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xd4VNXWwOHfAoHQlGalGESKoYcExSAaRUKxohdFbFi4KlYsgIgoqKAoCJ+IF7tXilwbKgiKgCAK0jsSqlLupSgdJIH1/XFOhpmQTCZkJmeSrPd55pnZe86cWXMIs2bvfc7eoqoYY4wx2SnmdQDGGGOimyUKY4wxQVmiMMYYE5QlCmOMMUFZojDGGBOUJQpjjDFBWaIwxhgTlCUKY4wxQVmiMMYYE9QpXgcQDlWqVNHY2FivwzDGmAJlwYIFO1X19Jy2KxSJIjY2lvnz53sdhjHGFCgisimU7azryRhjTFCeJwoRKS4ii0TkG7dcU0TmishaEflEREp6HaMxxhRlnicK4BFglV/5ZWCoqp4P/AXc7UlUxhhjAI/HKESkGtABeBHoISICXA7c4m7yIfAcMDK3+05LS2Pz5s0cPnw4TNGacIiJiaFatWqUKFHC61CMMSHyejD7deApoLxbrgzsVtV0t7wZqHoyO968eTPly5cnNjYWJ/8Yr6kqu3btYvPmzdSsWdPrcIwxIfKs60lErgK2q+qCk3x9NxGZLyLzd+zYccLzhw8fpnLlypYkooiIULlyZWvlGVPAeDlGkQRcIyIbgXE4XU7DgAoiktHSqQZsyerFqjpKVRNUNeH007M+DdiSRPSxfxNjCh7PEoWq9lbVaqoaC9wMTFPVLsB04EZ3szuACR6FaIwxUetw2lGGfL+GrbsPRfy9ouGsp8x64gxsr8UZs3jX43hOyq5du2jSpAlNmjThrLPOomrVqr7ykSNHQtpH165d+e2334JuM2LECEaPHh2OkANMnTqV6667Lug2CxcuZPLkyWF/b2NMcOPn/0G9vpMZ/kMqM9ec2PUebl4PZgOgqjOAGe7j9UDzfA3glVcgMRGSk4/XTZ8O8+bBU0+d1C4rV67M4sWLAXjuuecoV64cTzzxRMA2qoqqUqxY1vn6/fffz/F9unfvflLxhcPChQtZvnw5bdu29SwGY4qSPYfSaPz8d77ydU3O4ebmNZxCHr+zgonGFkX+S0yETp2cAw3OfadOTn2YrV27lri4OLp06UL9+vXZtm0b3bp1IyEhgfr169O/f3/fti1btmTx4sWkp6dToUIFevXqRePGjWnRogXbt28H4JlnnuH111/3bd+rVy+aN29O3bp1+fnnnwE4cOAAN9xwA3Fxcdx4440kJCT4kpi/iRMnUrduXeLj45kw4XiP35w5c2jRogVNmzYlKSmJ1NRUDh06RP/+/Rk9ejRNmjTh008/zXI7Y0x4vPXjuoAkMXP8k7x+5m6nEMHvLOD4r9qCfGvWrJlmtnLlyhPqgpo2TbVKFdW+fZ37adNy9/og+vXrp4MHD1ZV1dTUVBURnTdvnu/5Xbt2qapqWlqatmzZUlesWKGqqklJSbpo0SJNS0tTQCdNmqSqqo899pgOHDhQVVX79OmjQ4cO9W3/1FNPqarqhAkTNCUlRVVVBw4cqA888ICqqi5evFiLFSumixYtCojxwIEDWrVqVV27dq0eO3ZMO3bsqNdee62qqu7evVvT0tJUVfXbb7/VTp06qarq22+/rY888ohvH9ltl1mu/22MKcL+t+eQntvzG9/tpYnu/58wfGcB8zWE79io6HqKCsnJcP/9MGAA9O0b2A0VZrVq1SIhIcFXHjt2LO+++y7p6els3bqVlStXEhcXF/Ca0qVL065dOwCaNWvGrFmzstx3x44dfdts3LgRgJ9++omePXsC0LhxY+rXr3/C61auXEmdOnWoVasWAF26dOGjjz4CYPfu3dx+++2sW7cu6OcKdTtjTGgGfLOSd3/a4CvP69Oa08uXcgr5+J1lXU8Zpk+HkSOdAz5y5PFuqAgoW7as73FqairDhg1j2rRpLF26lLZt22Z5nUHJksenvCpevDjp6eknbANQqlSpHLfJrT59+pCSksLy5cv58ssvs70OItTtjDHBbdx5gNheE31Jok/7C9g4qMPxJAH5+p1liQKO9++NHw/9+zv3/mMWEbR3717Kly/PqaeeyrZt25gyZUrY3yMpKYnx48cDsGzZMlauXHnCNnFxcaSmprJhwwZUlbFjx/qe27NnD1WrOhfIf/DBB7768uXLs2/fvhy3M8aE7qGxi7js1Rm+8tLn2nBvq/MCN8rn7yxLFOCcKTB+/PGmW3KyU543L+JvHR8fT1xcHPXq1eP2228nKSkp7O/x0EMPsWXLFuLi4nj++eeJi4vjtNNOC9imTJkyvPXWW7Rr146EhATOPvts33M9e/bkySefJD4+Hqdb03H55ZezZMkSmjZtyqeffprtdsaYnC3fsofYXhP5eslWAF79R2M2DurAqcOHnpgAxo2Djh3z7TtLCsN/6ISEBM28cNGqVau44IILPIoouqSnp5Oenk5MTAypqam0adOG1NRUTjnFmyEq+7cx5rhjx5Sb357Drxv+BKBimRL80vsKYkoUdzbwbz0kJ59YzgMRWaCqCTltZ4PZRcD+/fu54oorSE9PR1X517/+5VmSMMYc9/O6ndzy9lxf+b07E7i83pmBG2W0Fjp1cgavR44MS5LIDfu2KAIqVKjAggUnNfeiMSYC0o4eo/WQH9m06yAA9c4qz8SHL6F4sWzmQsvHM5yyYonCGGPy0eTl27jv44W+8qf3tSAhtlLwF2U+wyk52VoUxhhT2Bw6cpSmA77jcNoxAFrVOZ0PuybmPKNy5jGJ5OSwjVGEyhKFMcZE2Ji5v/P0F8t85SmPtqLuWeWDvMJPsLMyLVEYY0zBtudgGo37H5+f6cZm1Xj1H41zt5OsJvnL564nu44igooXL06TJk1o0KABV199Nbt37/Y6pADt27ePupiMKSzemJYakCRmPZWc+yQRJSxRRFDp0qVZvHgxy5cvp1KlSowYMSIs+w3X1ByTJk2iQoUKYdmXMcbx3z2Hie01kVe/WwNA9+RabBzUgeqVyngc2cmzRJFPWrRowZYtx1d1HTx4MImJiTRq1Ih+/fr56gcMGEDdunVp2bIlnTt35tVXXwXgsssu49FHHyUhIYFhw4axY8cObrjhBhITE0lMTGT27NkA/Pjjj74Fkpo2bcq+ffvYtm0brVq18rVuMiYUjI2NZefOnQAMGTKEBg0a0KBBA9+05Rs3buSCCy7g3nvvpX79+rRp04ZDhyK/mpYxBVW/Ccu5aOAPvvKCZ1rzZEo9DyMKjyIxRvH81ytYuXVvWPcZd86p9Lv6xFlYs3L06FF++OEH7r77bgC+++47UlNT+fXXX1FVrrnmGmbOnEnp0qX57LPPWLJkCWlpacTHx9OsWTPffo4cOULGFei33HILjz32GC1btuT3338nJSWFVatW8eqrrzJixAiSkpLYv38/MTExjBo1ipSUFPr06cPRo0c5ePBgQHwLFizg/fffZ+7cuagqF154IZdeeikVK1YkNTWVsWPH8vbbb9OpUyc+++wzbr311jAdRWMKh3U79nPFaz/6ys9eFcddLWt6GFF45ZgoRKQY0Bg4BzgELFfV7ZEOrDA4dOgQTZo0YcuWLVxwwQVceeWVgJMovvvuO5o2bQo4V06npqayb98+rr32WmJiYoiJieHqq68O2N9NN93kezx16tSAyf327t3L/v37SUpKokePHnTp0oWOHTtSrVo1EhMTueuuu0hLS+O6666jSZMmAfv96aefuP76632z2nbs2JFZs2ZxzTXXULNmTd/2/lOXG2Oc9XweGL2Qb5f/11e3/PkUypUqXL/Bs/00IlILZ/3q1kAqsAOIAeqIyEHgX8CHqnosPwLNi1B/+YdbxhjFwYMHSUlJYcSIETz88MOoKr179+af//xnwPYZXT7Z8Z+e/NixY8yZM4eYmJiAbXr16kWHDh2YNGkSSUlJTJkyhVatWjFz5kwmTpzInXfeSY8ePbj99ttD+gwZ05aDMzhvXU/GOJZu3s01b8z2lV+/qQnXNa3qYUSRE2yM4gXgY6CWqqao6q2qeqOqNgKuAU4DbsuPIAu6MmXKMHz4cF577TXS09NJSUnhvffeY//+/QBs2bKF7du3k5SUxNdff83hw4fZv38/33zzTbb7bNOmDf/3f//nK2csbbpu3ToaNmxIz549SUxMZPXq1WzatIkzzzyTe++9l3vuuYeFCxcG7OuSSy7hyy+/5ODBgxw4cIAvvviCSy65JAJHwpiC79gx5boRs31J4ozypfjthbaFNklAkBaFqnYO8tx2IPjPXxOgadOmNGrUiLFjx3LbbbexatUqWrRoAUC5cuX4+OOPSUxM5JprrqFRo0aceeaZNGzY8ITpwDMMHz6c7t2706hRI9LT02nVqhVvvfUWr7/+OtOnT6dYsWLUr1+fdu3aMW7cOAYPHkyJEiUoV66cb+W6DPHx8dx55500b94cgHvuuYemTZtaN5MxmfyUupNb3z0+id8HXRO5rO4ZHkaUP3KcZlxEFgDvAWNU9a98iSqXCtM04/v376dcuXIcPHiQVq1aMWrUKOLj470OK6wK6r+NKbqOpB/jssHT2brHWbWxYdXT+LJ7UvaT+BUQ4Zxm/CagKzBPROYD7wPfaWFYyCIKdevWjZUrV3L48GHuuOOOQpckjClovlm6lQfHLPKVP3/gYuJrVPQwovyXY6JQ1bVAHxHpC1yF07o4KiLvA8NU9c8Ix1ikjBkzxusQjDHAwSPpNHzuO44ec34Tt77gDN6+PSHnSfwKoZDO4RKRRjitivbAZ8BooCUwDWgS5KWeUtUi+Y8azawhagqCf/+ykb4TVvjKU3u04vwzQpzErxAK5TqKBcBu4F2gl6r+7T41V0TCv8BzmMTExLBr1y4qV65sySJKqCq7du064ZReY6LFXweO0HTA975y5+Y1GNixoYcRRYdQWhT/UNX1WT2hqh3DHE/YVKtWjc2bN7Njxw6vQzF+YmJiqFatmtdhGHOCod+vYdgPqb7yz70u55wKpT2MKHoEu+DuVpwznbJMEu4FeWer6k+RCi4vSpQoQc2ahecSemNMZGzdfYiLB03zlR++ojY9rqzjYUTRJ1iLojKwyO16WsDxK7PPBy4FdgK9Ih6hMcZEyNNfLGPM3N995YV9r6RS2ZIeRhSdgl1wN0xE3gAuB5KARjhzPa0CblPV37N7rTHGRLO12/fReshMX7n/tfW5vUWsdwFFuaBjFKp6FPjevRljTIGmqtz70XymrnLmNS1eTFjarw1lC9kkfuFmR8cYUyR8u2wb948+Ps/ZG7c05apG53gYUcFhicIYU6ilv/wK5/8VOIP0mitLU3Lyx9Aoi/WozQlshTtjTKH14c8bA5LES9c3ZGNKGUrefBMkJnoYWcESygV3FYDbgVj/7VX14ciFZYwxJ+/QkaNc8OzkgLp173WleMn7YORIGD8ekpM9iq7gCaXraRIwB1gGRP0iRcaYou3lyasZOWOdrzzqtma0qX8WlLwPBgyAvn0tSeRSKIkiRlV7RDwSY4zJg8zTbwBsGNjemcJn+nSnJdG3r3OfnGzJIhdCSRT/FpF7gW+AjHmeyOussSJSHfgIOBNQYJR77UYl4BOcrq6NQKdoXQfDGBMdHh67iK+WbPWVP7v/Ypqd604FPn06dOp0vLspOTmwbHIUymD2EWAw8AvOFdoLgPlBXxGadOBxVY0DLgK6i0gcztXeP6hqbeAH7OpvY0w2/vjzILG9JvqSRI1KZdg4qMPxJAEwb15gUkhOdsrz5nkQccEUygp364HmqrozooGITADecG+Xqeo2ETkbmKGqdYO9NqsV7owxhdt1I2az+I/dvvLUHpdy/hnlPIyo4AnnCndrgYN5Dyl7IhILNAXmAmeq6jb3qf/idE0ZYwwAK7fupf3wWb7yJbWr8O+7L/QwosIvlERxAFgsItMJHKMIy+mxIlIOZzGkR1V1r//aEaqqIpJlk0dEugHdAGrUqBGOUIwxUa5p/+/462Carzz36Ss481Rb3yTSQkkUX7q3sBORErgr5qnq5271/0TkbL+up+1ZvVZVRwGjwOl6ikR8xpjo8PPandzyzlxfuXPz6gzs2MjDiIqWUNbM/jASbyxO0+FdYJWqDvF76ivgDmCQez8hEu9vjIl+qkrN3pMC6pY+14ZTY0p4FFHRFMqV2bWBgUAcznoUAKjqeXl87yTgNmCZiCx2657GSRDjReRuYBPQKY/vY4wpgL5aspWHxy7ylZ9oU4cHL6/tYURFVyhdT+8D/YChQDLQlTDMEeWujJfdYtZX5HX/xpiCKe3oMWr3+Tag7rcX2lLqlOIeRWRC+cIvrao/4JxKu0lVnwM6RDYsY0xR9M6s9QFJ4pUbG7FxUAdLEh4LpUXxt4gUA1JF5EFgC2AnKxtjwubA3+nU7zcloG79S+0pViy7TgeTn0JJFI8AZYCHgQE4S6PeEcmgjDFFx4NjFvLN0m2+8vt3JpJc7wwPIzKZhXLWU8Z17vtxxieMMSbPNu06wKWDZ/jKJYsXY82L7bwLyGQr20QhIl/jTNaXJVW9JiIRGWMKvdheEwPKb3aJp33Ds50J/ObNg6ds5bloEqxF8ap73xE4C/jYLXcG/hfJoIwxhdOCTX9xw8ifA+o2ppSBjCSRMauriSrZJgpV/RFARF7LNGnU1yJiM/AZY3Ilcyvik24XceGmpU5yuP9+W3kuioUymF1WRM5T1fUAIlITKBvZsIwxhcWkZdt4YPTCgLqNg9wz7M9LdpKErTwX1UJJFI8BM9zpxgU4F3cyPmOMCSZzK+KEqcBt5bkCIZSznia703jUc6tWq+rfwV5jjCnaRs1cx0uTVvvKpUsUZ9WAtoEb2cpzBUYoLQrcxLAkwrEYYwq4Y8eU854OnMRv/jOtqVKu1IkbB1t5zhJFVMlxhbuCwFa4M8Z7vT9fxthff/eVm51bkc/uv9jDiExOwrnCnTHGZOtw2lHq9Z0cULd6QFtiStj8TIVFKNOMC9AFOE9V+4tIDeAsVf014tEZY6LajSN/Zv6mv3xlW1CocAqlRfEmcAxnjqf+wD6cVekSIxiXMSaK7dz/NwkvTA2os0n8Cq9QEsWFqhovIosAVPUvESkZ4biMMVEq7tnJHDxy1Ffu3a4e/7y0locRmUgLJVGkiUhx3HmfROR0nBaGMaYIWbdjP1e89mNAne/COVOohZIohgNfAGeIyIvAjcAzEY3KGBNVsp3EzxQJoVxwN1pEFuAsTyrAdaq6KuKRGWM8N3f9Lm4aNSegzloRRU+op8emAnsztheRGqr6e/CXGGMKssytiM/uv5hm51b0KBrjpVBOj30I6IcztfhRnFaFAnYOnDGF0ITFW3hk3OKAOmtFFG2hLoVaV1V3RToYY4wHXnkFEhPRyy6jZu/A6Td+fPIyzq1sk0UXdaEkij+APZEOxBjjkcRE3njx37w65aCvqvKhvSy47iywJGEIvhRqD/fhepxpxicCvlljVXVIhGMzxkTY0WNKrSkHIeEGX92ij7tT8d/v28R8xidYi6K8e/+7eyvp3iDIWtrGmIKhxyeL+XzRFl+5pezh40FdbAEhc4JgS6E+DyAi/1DV//g/JyL/iHRgxpjIOHgknbhnpwTUrW5dmpjOd9oCQiZLxULYpneIdcaYKNdh+KyAJHHnxbFsTClDTOebnLUg+vd37jt1chYWMobgYxTtgPZAVREZ7vfUqUB6pAMzxoTP9r2Haf7SDwF1vkn8XploCwiZoIKNUWwF5gPXAAv86vfhrKNtjCkAavaeiP/6ZP2ujqNrUs3jFU89deKLrOvJ+Ak2RrEEWCIiY1Q1LR9jMsaEwZr/7aPN0JkBdXbhnDkZOY5RWJJwvfLKiX2206c79cbkpxD+FmN7TQxIEm/fnmBJwpy0UAazDUBiYuAA3/TpTjnR1m8y+SzI3+LstTtPmKNp46AOXBl3pgeBmsIi20QhIv927x/Jv3Dyycm0DjIG+Dp1gmefde79BwDzk7VuirZs/hZjpxykyztzfZt99WCStSJMWARrUTQTkXOAu0SkoohU8r/lV4ARcbKtg+RkuP9+GDDAufdqsM9aN8flR9KMxsTs97f4n259ifWbfgOcVkSjahU8Cs4UOqqa5Q14GFiFM23HemCD3219dq/z4tasWTPNtWnTVKtUUe3b17mfNi0yr4mUaIrFSxnHIePzZy4XlPc4iZiOVami5/b8JuD2x58HvIvJFDjAfA3hOzbnDWBkKDvy8nZSiULV+ZIF5z4n0fhlkZv4g3n55RM/x7RpTn1BkB9JM5oS87RpesctLwYkiIsf+KDo/lgwJy3URBHKWU/3i0hjEXnQvRWOdSimT3emKsiYsiCnq1Dnzcv+oiQv5Db+YAp6V1Z+dAlGSbfjkfRjxE45yIzqjX11S/q1YfaNNbz7WzSFX06ZBKcLajnQ370tAx4KJQvl1y3XLYpobB3kRiTij6ZfzLlVRFoUjZ6bEtCKOP/pifkegylcCGPX01KgrF+5LLA0lJ3n5Qa0BX4D1gK9gm2b60RR0LtaIhV/uLqy8lMRGKP4c//fJ4xFHDqSni/vbQq3UBOFONtmT0SWAYmqetgtxwDzVLVhmBs3/u9ZHFgDXAlsBuYBnVV1ZVbbJyQk6Pz58yMVTtGQ0d10//1OV5ZXp/7mlrs6W0Cs06c73TBZTU0Rre+RjczXRFxYsxKf/LNFRN/TFB0iskBVE3LcLoRE0QO4A/jCrboO+EBVX89zlNm/ZwvgOVVNccu9AVR1YFbbW6LIo4wkkZEcMpdNvlu3Yz9XvPZjQN2Gge0REY8iMoVRqIkix6VQVXWIiMwAWrpVXVV1UR7jy0lVnCVYM2wGLvTfQES6Ad0AatSoEeFwCrlgA/WWKPJd5lbE3S1r0veqOI+iMSaEFoUXRORGoK2q3uOWbwMuVNUHs9reWhSmMJi9dmfAldVgk/iZyApbi8IjW4DqfuVqbp0xhVLmVsTAjg3p3NxayiY6RGuimAfUFpGaOAniZuAWb0MyJvzGzP2dp79YFlBnrQgTbXJMFCLysqr2zKkunFQ1XUQeBKYAxYH3VHVFpN7PGC9kbkWMufdCLq5VxaNojMleKC2KK4HMSaFdFnVhpaqTgEmRfA9jvPD81yt4f/bGgDprRZhoFmzN7PuBB4DzRGSp31PlgdmRDsyYwkZVqdk78LfPtMcv5bzTy3kUkTGhCdaiGAN8CwwEevnV71PVPyMalTGFTKe3fuHXjYH/bawVYQqKYGtm7wH2iEjmLqZyIlJOVX+PbGjGFHyH045Sr+/kgLpFfa+kYtmSHkVkTO6FMkYxEVBAgBigJs4cTPUjGJcxBd75T08i/djx65QqlCnB4mfbeBiRMScnlCuzA+Z0EpF4nLELY0wWduz7m8QXpwbUrXmhHSVPsSXqTcGU6+soVHWhiFyY85bGFD2ZT3lNrns673dt7lE0xoRHKNdR9PArFgPiga0Ri8iYAmj1f/fS9vVZAXU2iZ8pLEJpUZT3e5yOM2bxWWTCMabgydyK6J5ciydT6nkUjTHhF8oYxfMAIlLOLe+PdFDGFATTV2+n6weBy4/aKa+mMAql66kB8G+gklveCdyhqssjHJsxUStzK2JIp8Z0jK/mUTTGRFYoXU+jgB6qOh1ARC5z6y6OYFzGRKX3ftpA/28CF1q0VoQp7EJJFGUzkgSAqs4QkbIRjMmYqJS5FfGf+1qQGFvJo2iMyT+hJIr1ItIXp/sJ4FZgfeRCMia69P58KWN//SOgzloRpigJJVHcBTwPfI5zhfYst86YQu3YMeW8pwMn8Zv5ZDI1KpfxKCJjvBHKWU9/AQ/nQyzGRI0Ow2exYuvegDprRZiiKlpXuDPGEwePpBP37JSAuqXPteHUmBIeRWSM9yxRGOPKPFhdtUJpZve63KNojIkelihMkfffPYe5aOAPAXVrX2zHKcVtEj9jILQL7oZnUb0HmK+qE8IfkjH5J3Mron3Ds3izSzOPojEmOoXSoogB6gH/ccs3ABuAxiKSrKqPRio4YyJl+ZY9XPV/PwXU2WC1MVkLJVE0ApJU9SiAiIzEOUW2JbAsgrEZExGZWxGPX1mHh66o7VE0xkS/UBJFRaAcTncTQFmgkqoeFZG/IxaZMWE2efl/ue/jBQF11oowJmehJIpXgMUiMgNnOdRWwEvuNB5Tg73QmGiRuRUx4pZ4OjQ626NojClYQrng7l0RmQRkLNP1tKpmLFz0ZMQiMyYMRs5Yx8uTVwfUWSvCmNwJ9fTYYsAOd/vzReR8VZ0ZubCMybvMrYgvuyfRpHoFj6IxpuAK5fTYl4GbgBXAMbdaAUsUJio9Om4RXy4OXK3XWhHGnLxQWhTXAXVV1QauTVQ7ekyplWkSv9m9LqdqhdIeRWRM4RDSNONACcAShYla9fp+y+G0YwF11oowJjxCSRQHcc56+gG/ZKGqNqOs8dyeQ2k0fv67gLrlz6dQrpTNTmNMuITyv+kr92ZMVMk8WF2u1Cksfz7Fo2iMKbxCOT32w/wIxJhQbdp1gEsHzwioW/dSe4oXE28CMqaQyzZRiMh4Ve0kIstwznIKoKqNIhqZMVnI3IpoVed0PrqreTZbG2PCIViL4hH3/qr8CMSYYH5Zt4vOb88JqLPBamPyR7aJQlW3uQ9bAzNVNTV/QjImUOZWRPfkWjyZUs+jaIwpekIZzK4B/EtEYoEFOBfazVLVxRGMyxjG/vo7vT8PnKDYWhHG5L9QBrP7AYhIaeBenPmdXgeKRzY0U5RlbkUMu7kJ1zap6lE0xhRtoUzh8QyQhDPV+CLgCZz1KIwJu+e+WsEHP28MqLNWhDHeCqXrqSOQDkwEfgR+yet0HiIyGLgaOAKsA7qq6m73ud7A3cBR4GFVnZKX9zIFg6pSs3fg9BtfPHAxTWtU9CgiY0yGHFePV9V4nAHtX4ErgWUi8lPwV+Xoe6CBe4rtGqA3gIjEATcD9YG2wJsiYl1chdz1b84+IUlsHNTBkoQxUSKUrqcGwCXApUAC8AfG0hSfAAASiUlEQVR57HpSVf85F+YAN7qPrwXGuS2WDSKyFmcdjF/y8n4mOh1JP0adZ74NqPu51+WcY5P4GRNVQul6GoSTGIYD81Q1Lcwx3AV84j6uipM4Mmx2604gIt2AbgA1atQIc0gm0jIPVoONRRgTrUI56+kqESkJ1AHqishvoSQLEZkKnJXFU31UdYK7TR+c8Y/RuQsbVHUUMAogISHhhCvHTXT668ARmg74PqBuZf8UypS0SfyMiVahdD1dCnwEbMRZM7u6iNyR0wp3qto6h/3eiXPV9xWqmvFFvwWo7rdZNbfOFAKZWxFnlC/Fr32C/pkYY6JAKD/jhgBtVPU3ABGpA4wFmp3sm4pIW+Ap4FJVPej31FfAGBEZApwD1MYZRDcF2Nrt+2g9JPB3xfqX2lPMJvEzpkAIJVGUyEgSAKq6RkRK5PF93wBKAd+LCMAcVb1PVVeIyHhgJU6XVHdVPZrH9zIeytyKaNfgLEbeetK/MYwxHgglUcwXkXeAj91yF2B+Xt5UVc8P8tyLwIt52b/x3sw1O7j9vcDGoA1WG1MwhZIo7ge6Axkr2s0C3oxYRKbAy9yKeKJNHR68vLZH0Rhj8iqUs57+xhmnGBL5cExB9sHsDTz39cqAOmtFGFPwBVu4KMsFizLYwkXGX+ZWxFu3NqNtg6zOjjbGFDTBWhT/AA7lVyCmYOr56VI+mf9HQJ21IowpXIIlijGqGi8i/1bV2/ItIlMgZDWJ3zcPtaRB1dM8isgYEynBEkVJEbkFuFhEOmZ+UlU/j1xYJpqlDJ3Jb//bF1BnrQhjCq9gieI+nFNhK+BMCe5PAUsURczf6Uep+8zkgLpf+1zBGeVjPIrIGJMfgq2Z/RPwk4jMV9V38zEmE4VsEj9jiq5QTo+1JFGE7dj3N4kvTg2oWz2gLTElbJkQY4oKm7LTZCtzK+K8KmWZ9sRl3gRjjPFM0EQhzkRM1VT1j2DbmcJl1ba9tBsWuDbVhoHtceflMsYUMUEThaqqiEwCGuZTPMZjmVsRHeOrMqRTE4+iMcZEg1C6nhaKSKKqzot4NMYz8zf+yY1vBa44a4PVxhgILVFcCHQRkU3AAZzFi9Sm8Cg8MrcinulwAfdccp5H0Rhjok0oiSIl4lEYT0xYvIVHxi0OqLNWhDEms1BOj90kIi2B2qr6voicDpSLfGgmkjK3Ij5/4GLia1T0KBpjTDQLZc3sfkACUBd4HyiBs4hRUmRDM5EwbGoqQ6euCaizVoQxJphQup6uB5oCCwFUdauIlI9oVCbssprEb9ZTyVSvVMajiIwxBUUoieKIe5qsAohI2QjHZMKs+5iFTFy6zVcWgQ0DrRVhjAlNKIlivIj8C6ggIvcCdwHvRDYsEw6H045Sr2/gJH5L+rXhtNIlPIrIGFMQhTKY/aqIXAnsxRmneFZVv494ZCZPkl+dwYadB3zlxtUrMKG7DSsZY3IvlMHsl1W1J/B9FnUmyvx54AjxAwLzeOqL7ShRvJhHERljCrpQvj2uzKKuXbgDMXkX22tiQJLo3Lw6Gwd1sCRhjMmTbFsUInI/8ABwnogs9XuqPDA70oGZ0K3bsZ8rXvsxoM4m8TPGhEvQNbOBb4GBQC+/+n2q+mdEozIhs+k3jDGRFmyFuz3AHqAzgIicAcQA5USknKr+nj8hmqys+d8+2gydGVBnF84ZYyIhlMHsq4EhwDnAduBcYBVQP7KhmexkbkWMuq0Zbeqf5VE0xpjCLpTrKF4ALgKmqmpTEUkGbo1sWCYrv6zbRee35/jKpUsUZ9WAth5GZIwpCkJJFGmquktEiolIMVWdLiKvRzwyEyBzK2Lmk8nUqGzTbxhjIi+URLFbRMoBM4HRIrIdZ10Kkw8yTwVuF84ZY/JbKIniWuAw8BjQBTgN6B/JoAwcO6ac93TgJH6L+l5JxbIlPYrIGFNUBbuO4lHgZ2Chqh51qz/Ml6iKuJEz1vHy5NW+8vVNqzL0Jlu32hjjjWAtimrA60A9EVmGc5Hdz8DPdh1FZPydfpS6zwRO4rd6QFtiShT3KCJjjAl+HcUTACJSEmfhoouBrsAoEdmtqnH5E2LR8PQXyxgz9/ilKY9cUZvHrqzjYUTGGOMIZYyiNHAqztjEacBWYFkkgypK9hxKo/Hz3wXUrXupPcWL2fQbxpjoEGyMYhTORXX7gLk43U5DVPWvfIqt0Os8ag6/rN/lK79yYyM6JVT3MCJjjDlRsBZFDaAUkApsATYDu/MjqMJu6+5DXDxoWkCdTb9hjIlWwcYo2ooz/Wh9nPGJx4EGIvIn8Iuq9svrm4vI48CrwOmqutN9v2FAe+AgcKeqLszr+0SThBemsnP/377yh3c159I6p3sYkTHGBBd0jEJVFVguIrtxJgjcA1wFNAfylChEpDrQBvCfXLAdUNu9XQiMdO8LvJVb99J++KyAOmtFGGMKgmBjFA/jtCQuBtJwT40F3iM8g9lDgaeACX511wIfuQlqjohUEJGzVXVbGN7PM5mn3/jmoZY0qHqaR9EYY0zuBGtRxAL/AR4L9xe1iFwLbFHVJZkW16kK/OFX3uzWFchEMSt1B7e9+6uvXKlsSRb2zWrBQGOMiV7Bxih65GXHIjIVyGru6z7A0zjdTnnZfzegG0CNGjXysquIyNyK+KlnMtUq2iR+xpiCJ5TrKE6KqrbOql5EGgI1gYzWRDVgoYg0xzm7yv/80GpuXVb7HwWMAkhISNDwRZ43ny7YzBP/WeIrN69ZifH/bOFhRMYYkzcRSxTZUdVlwBkZZRHZCCS4Zz19BTwoIuNwBrH3FJTxiawm8VvybBtOK1PCo4iMMSY88j1R5GASzqmxa3FOj+3qbTihGTY1laFT1/jKNydWZ9ANjTyMyBhjwsfzRKGqsX6PFejuXTS5czjtKPX6Bk7i99sLbSl1ik3iZ4wpPDxPFAXV4+OX8NnCzb7ykyl16Z58vocRGWNMZFiiyKVDR45ywbOBrYj1L7WnmE3iZ4wppCxR5MIn836n52fHrzUcelNjrm9azcOIjDEm8ixRhGDPwTQa9z8+FXjrC87knTsSPIzIGGPyjyWKHIyYvpbBU37zlWc9lUz1SnbhnDGm6LBEkY3/7T3MhS/94Cvfd2kterWr52FExhjjDUsUWXjuqxV88PNGX3len9acXr6UdwEZY4yHLFH42bDzAMmvzvCVn+lwAfdccp53ARljTBSwRAGoKg+OWcTEZcdnC1n2XBvKx9j0G8YYU+QTxbLNe7j6jZ985SGdGtMx3k55NcaYDEU6Ufzx50FfkqhctiSze11OTAmbfsMYY/wV6URRrtQpJJ1fmbtb1uTyemd6HY4xxkSlIp0oKpYtyeh7LvI6DGOMiWrFvA7AGGNMdLNEYYwxJihLFMYYY4KyRGGMMSYoSxTGGGOCskRhjDEmKEsUxhhjgrJEYYwxJihRVa9jyDMR2QFs8jqOXKgC7PQ6iChlxyZ7dmyCs+OTveyOzbmqenpOLy4UiaKgEZH5qmprqWbBjk327NgEZ8cne3k9Ntb1ZIwxJihLFMYYY4KyROGNUV4HEMXs2GTPjk1wdnyyl6djY2MUxhhjgrIWhTHGmKAsUXhARB4XERWRKm5ZRGS4iKwVkaUiEu91jPlNRAaLyGr3838hIhX8nuvtHpvfRCTFyzi9IiJt3c+/VkR6eR2Pl0SkuohMF5GVIrJCRB5x6yuJyPcikureV/Q6Vq+ISHERWSQi37jlmiIy1/37+URESuZmf5Yo8pmIVAfaAL/7VbcDaru3bsBID0Lz2vdAA1VtBKwBegOISBxwM1AfaAu8KSJFar1a9/OOwPk7iQM6u8elqEoHHlfVOOAioLt7PHoBP6hqbeAHt1xUPQKs8iu/DAxV1fOBv4C7c7MzSxT5byjwFOA/OHQt8JE65gAVRORsT6LziKp+p6rpbnEOUM19fC0wTlX/VtUNwFqguRcxeqg5sFZV16vqEWAcznEpklR1m6oudB/vw/lCrIpzTD50N/sQuM6bCL0lItWADsA7blmAy4FP3U1yfWwsUeQjEbkW2KKqSzI9VRX4w6+82a0rqu4CvnUf27GxY5AtEYkFmgJzgTNVdZv71H+BMz0Ky2uv4/wYPeaWKwO7/X6I5frvp0ivmR0JIjIVOCuLp/oAT+N0OxVJwY6Nqk5wt+mD07UwOj9jMwWPiJQDPgMeVdW9zg9nh6qqiBS5UzpF5Cpgu6ouEJHLwrVfSxRhpqqts6oXkYZATWCJ+wddDVgoIs2BLUB1v82ruXWFSnbHJoOI3AlcBVyhx8/bLhLHJgd2DDIRkRI4SWK0qn7uVv9PRM5W1W1u1+127yL0TBJwjYi0B2KAU4FhON3Zp7itilz//VjXUz5R1WWqeoaqxqpqLE7zL15V/wt8Bdzunv10EbDHrwldJIhIW5zm8jWqetDvqa+Am0WklIjUxBnw/9WLGD00D6jtnrlSEmdw/yuPY/KM2+f+LrBKVYf4PfUVcIf7+A5gQn7H5jVV7a2q1dzvmJuBaaraBZgO3OhulutjYy2K6DAJaI8zUHsQ6OptOJ54AygFfO+2uOao6n2qukJExgMrcbqkuqvqUQ/jzHeqmi4iDwJTgOLAe6q6wuOwvJQE3AYsE5HFbt3TwCBgvIjcjTObdCeP4otGPYFxIvICsAgn0YbMrsw2xhgTlHU9GWOMCcoShTHGmKAsURhjjAnKEoUxxpigLFEYY4wJyhKFiTgROUtExonIOhFZICKTRKSOiFyWMbul10Skv4gEvSAwTO9TQUQeCMN+ZoiIJ+tDi0hTEQl6eqWIPCgid+VXTCayLFGYiHIvjvoCmKGqtVS1Gc7MsFE1D4+qPquqU/PhrSoAuUoU7oWY0fR/9WlgeA7bvAc8lA+xmHwQTX98pnBKBtJU9a2MClVdoqqz3GI5EfnUXYtitJtYEJFnRWSeiCwXkVF+9TNE5GUR+VVE1ojIJW59GREZ765R8IU7936C+1wbEflFRBaKyH/cOYICiMgHInKj+3ijiDzvbr9MROplsf1EEWnkPl4kIs+6j/uLyL0iUk5EfvDbR8Zsr4OAWiKyWEQGu6950v2sS0XkebcuVpz1Jz4ClhM4hUfmWE74fOKsX/Efv218rbfsjkeIn7s80ChjYksRGeb32VNEZKaIFHOvrt8ozhQ1poCzRGEirQGwIMjzTYFHcdZZOA/nqluAN1Q1UVUbAKVx5oDKcIqqNndf18+tewD4y12joC/QDECcxaGeAVqrajwwH+gRQtw73e1HAk9k8fws4BIROQ3nivGMuC8BZgKHgevdfSQDr7nJrhewTlWbqOqTItIGZ1qS5kAToJmItHL3VRt4U1Xrq+qmrIIM8vmmAheKSFl305twrszN6Xjk9LkTcBJXht7ATSKSjNPK6KqqGbOWznePhyngLFEYr/2qqpvdL5fFQKxbn+y2CpbhzKVf3+81GZPALfDbviXOOg2o6nJgqVt/EU4Smu1O93AHcG4IcWX1Hv5mAa1wEsREnJZRGaCmqv4GCPCSiCzF+dKuStbdbW3c2yJgIVAPJ0EAbHLXJwkmy8/nTv42GbhaRE7BWZ9gQnbb5+Jznw3syCi4LYd7cRaeekNV1/ltux04J4f4TQFgcz2ZSFvB8cnIsvK33+OjwCkiEgO8CSSo6h8i8hzOTJiZX3OUnP+GBfheVTvnKuqc32Mezq/r9ThfklVwvjAzWk9dgNOBZqqaJiIbM30G//gGquq/AiqddRYOhBBnsM83DngQ+BOYr6r73FZNsOOR0+c+xImfoyGwixOTQoy7vSngrEVhIm0aUEpEumVUiEijjLGFbGR8Ee10+8+DJZoMs3EngRNnWcyGbv0cIElEznefKysidXL5GU7grjT3B/AP4BecFsYTON1OAKfhrAuQ5nbLZPxq3weU99vVFOAuv3GCqiJyRi5CCfb5fgTicRLYuBC2D8Uq4PyMgoicCzyO04XYTkQu9Nu2DoHdVKaAskRhIspdV+J6oLU4p8euAAbirECW3Wt2A2/jfMlMwfn1npM3gdNFZCXwAk5LZo+q7gDuBMa63UC/4HTvhMMsnGRwyH1czb0HZ+GlBLfr7HZgNYCq7sLp9lkuIoNV9TtgDPCLu+2nBCaSoIJ9PneW3W9w1tr+JqftQ3y/1cBpIlLebZ28Czyhqltx1mF+x20RgtMt932o+zbRy2aPNYWCiBQHSqjqYRGphTMuUNf95W/CSEQeA/ap6jtBtmkK9FDV2/IvMhMpNkZhCosywHRxVj4T4AFLEhEzEqfLLZgqOGefmULAWhTGGGOCsjEKY4wxQVmiMMYYE5QlCmOMMUFZojDGGBOUJQpjjDFBWaIwxhgT1P8DuDu7jhRHVakAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(X[:,1], y, 'rx', label='Training data')\n", "plt.plot(X[:,1], X.dot(theta), label='Regression')\n", "plt.legend()\n", "plt.xlabel('Change in water leven (x)')\n", "plt.ylabel('Water flowing out of the dam (y)')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bias Variance\n", "Models with high bias are not complex enough for the data and tend to underfit, while models with high variance overfit to the training data.\n", "\n", "### Learning Curves\n", "You will now implement code to generate the learning curves that will be useful in debugging learning algorithms. Recall that a learning curve plots training and cross validation error as a function of training set size.\n", "\n", "This model has a high bias problem.\n", "\n", "**Exercise**: Implement the learning curve function." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "def learning_curve(X, y, Xval, yval, _lambda):\n", " \"\"\" Get the learning curves for each value of m\n", " \n", " Returns:\n", " :error_train: The training error of the dataset until i\n", " :error_val: The error of the _entire_ cross validation set\n", " \"\"\"\n", " m, n = X.shape\n", " error_train = np.zeros((m, 1))\n", " error_val = np.zeros((m, 1))\n", " \n", " for i in range(1, m+1):\n", " pass # remove this line\n", " \n", " return error_train, error_val" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAE+hJREFUeJzt3X2QVfWd5/H3d4GIgNEGH6Flm9pYsQGJ4A0yS0xwMAaSVTTRQDbZxVQSqlxddWa3dhmnanAcU2WmXMOmRk2RxKyVNRqLjIHdMcuqgUqyiS6NMQREA/FhaFAEVHyCMWa/+0dfmf4xjUDf01y7+/2qovo8/M65nwNUf/qcc+/pyEwkSXrHP2t2AEnSe4vFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpMLQZgfojRNPPDHb2tqaHUOS+pV169btysyTDjWuXxZDW1sbHR0dzY4hSf1KRDx3OOO8lCRJKlgMkqSCxSBJKvTLewySBo7f//73dHZ2sm/fvmZHGTCGDx9Oa2srw4YN69X2FoOkpurs7OS4446jra2NiGh2nH4vM9m9ezednZ1MmDChV/vwUpKkptq3bx9jxoyxFCoSEYwZM6ahMzCLQVLTWQrVavTv02KQJBUsBkmD2iuvvMLtt99+xNt98pOf5JVXXumDRM1nMUga1A5WDG+//fa7bvfAAw9wwgkn9FWspvJdSZIGtcWLF/O73/2Os88+m2HDhjF8+HBaWlp48skn+e1vf8sll1zC1q1b2bdvH9deey2LFi0C/vHRPK+//jpz587lIx/5CL/4xS8YN24cK1as4Nhjj23ykfWexSDpPeMv/8dGntj+aqX7nDj2/Sy5aNJB1998881s2LCBxx9/nDVr1vCpT32KDRs27H+r55133sno0aPZu3cvH/7wh/nMZz7DmDFjin1s3ryZe+65h29961t89rOf5Yc//CFf+MIXKj2Oo8likKRupk+fXrz//xvf+Ab3338/AFu3bmXz5s3/pBgmTJjA2WefDcA555zDs88+e9Ty9gWLQdJ7xrv9ZH+0jBw5cv/0mjVreOihh/jlL3/JiBEjmDVrVo+fDzjmmGP2Tw8ZMoS9e/celax9xZvPkga14447jtdee63HdXv27KGlpYURI0bw5JNP8sgjjxzldM3hGYOkQW3MmDHMnDmTyZMnc+yxx3LKKafsXzdnzhy++c1v0t7ezgc/+EFmzJjRxKRHT2RmszMcsVqtlv6iHmlg2LRpE+3t7c2OMeD09PcaEesys3aobb2UJEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIElHYNSoUQBs376dyy67rMcxs2bN4lBvqV+6dClvvvnm/vn30mO8KymGiJgTEU9FxJaIWNzD+mMi4gf19Y9GRNsB68dHxOsR8R+ryCNJfW3s2LEsX76819sfWAzvpcd4N1wMETEEuA2YC0wEPhcREw8Y9iXg5cz8APB14GsHrL8V+HGjWSTpSC1evJjbbrtt//wNN9zATTfdxOzZs5k2bRpnnXUWK1as+CfbPfvss0yePBmAvXv3smDBAtrb27n00kuLZyVdeeWV1Go1Jk2axJIlS4CuB/Nt376d888/n/PPPx/oeoz3rl27ALj11luZPHkykydPZunSpftfr729na985StMmjSJCy+8sM+eyVTFIzGmA1sy82mAiLgXmAc80W3MPOCG+vRy4G8iIjIzI+IS4BngjQqySOrPfrwYXvhNtfs89SyYe/NBV8+fP5/rrruOq666CoD77ruPVatWcc011/D+97+fXbt2MWPGDC6++OKD/i7lO+64gxEjRrBp0ybWr1/PtGnT9q/76le/yujRo/nDH/7A7NmzWb9+Pddccw233norq1ev5sQTTyz2tW7dOr773e/y6KOPkpmce+65fOxjH6OlpeWoPd67iktJ44Ct3eY768t6HJOZbwN7gDERMQr4z8BfVpBDko7Y1KlTefHFF9m+fTu//vWvaWlp4dRTT+X6669nypQpXHDBBWzbto0dO3YcdB8//elP93+DnjJlClOmTNm/7r777mPatGlMnTqVjRs38sQTTxxsNwD8/Oc/59JLL2XkyJGMGjWKT3/60/zsZz8Djt7jvZv9EL0bgK9n5usHa+J3RMQiYBHA+PHj+z6ZpKPvXX6y70uXX345y5cv54UXXmD+/Pncfffd7Ny5k3Xr1jFs2DDa2tp6fNz2oTzzzDPccsstrF27lpaWFq644ope7ecdR+vx3lWcMWwDTu8231pf1uOYiBgKHA/sBs4F/joingWuA66PiKt7epHMXJaZtcysnXTSSRXElqQu8+fP595772X58uVcfvnl7Nmzh5NPPplhw4axevVqnnvuuXfd/qMf/Sjf//73AdiwYQPr168H4NVXX2XkyJEcf/zx7Nixgx//+B9vpR7scd/nnXceP/rRj3jzzTd54403uP/++znvvPMqPNpDq+KMYS1wRkRMoKsAFgD/+oAxK4GFwC+By4CfZNdjXfcfbUTcALyemX9TQSZJOmyTJk3itddeY9y4cZx22ml8/vOf56KLLuKss86iVqtx5plnvuv2V155JV/84hdpb2+nvb2dc845B4APfehDTJ06lTPPPJPTTz+dmTNn7t9m0aJFzJkzh7Fjx7J69er9y6dNm8YVV1zB9OnTAfjyl7/M1KlTj+pvhavksdsR8UlgKTAEuDMzvxoRNwIdmbkyIoYD3wOmAi8BC965Wd1tHzfQVQy3HOr1fOy2NHD42O2+0chjtyu5x5CZDwAPHLDsL7pN7wMuP8Q+bqgiiySpMX7yWZJUsBgkNV1//E2S72WN/n1aDJKaavjw4ezevdtyqEhmsnv3boYPH97rfTT7cwySBrnW1lY6OzvZuXNns6MMGMOHD6e1tbXX21sMkppq2LBhTJgwodkx1I2XkiRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSwGCRJBYtBklSopBgiYk5EPBURWyJicQ/rj4mIH9TXPxoRbfXlH4+IdRHxm/rXP64ijySp9xouhogYAtwGzAUmAp+LiIkHDPsS8HJmfgD4OvC1+vJdwEWZeRawEPheo3kkSY2p4oxhOrAlM5/OzLeAe4F5B4yZB9xVn14OzI6IyMxfZeb2+vKNwLERcUwFmSRJvVRFMYwDtnab76wv63FMZr4N7AHGHDDmM8BjmfkPFWSSJPXS0GYHAIiISXRdXrrwXcYsAhYBjB8//iglk6TBp4ozhm3A6d3mW+vLehwTEUOB44Hd9flW4H7g32bm7w72Ipm5LDNrmVk76aSTKogtSepJFcWwFjgjIiZExPuABcDKA8aspOvmMsBlwE8yMyPiBODvgMWZ+X8qyCJJalDDxVC/Z3A1sArYBNyXmRsj4saIuLg+7DvAmIjYAvwp8M5bWq8GPgD8RUQ8Xv9zcqOZJEm9F5nZ7AxHrFarZUdHR7NjSFK/EhHrMrN2qHF+8lmSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEkFi0GSVLAYJEmFSoohIuZExFMRsSUiFvew/piI+EF9/aMR0dZt3Z/Vlz8VEZ+oIo8kqfcaLoaIGALcBswFJgKfi4iJBwz7EvByZn4A+Drwtfq2E4EFwCRgDnB7fX+SpCap4oxhOrAlM5/OzLeAe4F5B4yZB9xVn14OzI6IqC+/NzP/ITOfAbbU9ydJapKhFexjHLC123wncO7BxmTm2xGxBxhTX/7IAduOqyBTjx65/Ssc98qmvtq9JPWp105oZ8a/+1afv06/ufkcEYsioiMiOnbu3NnsOJI0YFVxxrANOL3bfGt9WU9jOiNiKHA8sPswtwUgM5cBywBqtVr2JujRaFpJ6u+qOGNYC5wRERMi4n103UxeecCYlcDC+vRlwE8yM+vLF9TftTQBOAP4vxVkkiT1UsNnDPV7BlcDq4AhwJ2ZuTEibgQ6MnMl8B3gexGxBXiJrvKgPu4+4AngbeCqzPxDo5kkSb0XXT+49y+1Wi07OjqaHUOS+pWIWJeZtUON6zc3nyVJR4fFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpILFIEkqWAySpEJDxRARoyPiwYjYXP/acpBxC+tjNkfEwvqyERHxdxHxZERsjIibG8kiSapGo2cMi4GHM/MM4OH6fCEiRgNLgHOB6cCSbgVyS2aeCUwFZkbE3AbzSJIa1GgxzAPuqk/fBVzSw5hPAA9m5kuZ+TLwIDAnM9/MzNUAmfkW8BjQ2mAeSVKDGi2GUzLz+fr0C8ApPYwZB2ztNt9ZX7ZfRJwAXETXWYckqYmGHmpARDwEnNrDqj/vPpOZGRF5pAEiYihwD/CNzHz6XcYtAhYBjB8//khfRpJ0mA5ZDJl5wcHWRcSOiDgtM5+PiNOAF3sYtg2Y1W2+FVjTbX4ZsDkzlx4ix7L6WGq12hEXkCTp8DR6KWklsLA+vRBY0cOYVcCFEdFSv+l8YX0ZEXETcDxwXYM5JEkVabQYbgY+HhGbgQvq80RELSK+DZCZLwF/Bayt/7kxM1+KiFa6LkdNBB6LiMcj4ssN5pEkNSgy+99VmVqtlh0dHc2OIUn9SkSsy8zaocb5yWdJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVLAZJUsFikCQVGiqGiBgdEQ9GxOb615aDjFtYH7M5Ihb2sH5lRGxoJIskqRqNnjEsBh7OzDOAh+vzhYgYDSwBzgWmA0u6F0hEfBp4vcEckqSKNFoM84C76tN3AZf0MOYTwIOZ+VJmvgw8CMwBiIhRwJ8CNzWYQ5JUkUaL4ZTMfL4+/QJwSg9jxgFbu8131pcB/BXwX4A3G8whSarI0EMNiIiHgFN7WPXn3WcyMyMiD/eFI+Js4F9k5p9ERNthjF8ELAIYP3784b6MJOkIHbIYMvOCg62LiB0RcVpmPh8RpwEv9jBsGzCr23wrsAb4I6AWEc/Wc5wcEWsycxY9yMxlwDKAWq122AUkSToyjV5KWgm88y6jhcCKHsasAi6MiJb6TecLgVWZeUdmjs3MNuAjwG8PVgqSpKOn0WK4Gfh4RGwGLqjPExG1iPg2QGa+RNe9hLX1PzfWl0mS3oMis/9dlanVatnR0dHsGJLUr0TEusysHWqcn3yWJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBUiM5ud4YhFxE7guV5ufiKwq8I47yUD+dhgYB+fx9Z/9afj++eZedKhBvXLYmhERHRkZq3ZOfrCQD42GNjH57H1XwPx+LyUJEkqWAySpMJgLIZlzQ7QhwbyscHAPj6Prf8acMc36O4xSJLe3WA8Y5AkvYtBUwwRMScinoqILRGxuNl5qhQRp0fE6oh4IiI2RsS1zc5UtYgYEhG/ioj/2ewsVYqIEyJieUQ8GRGbIuKPmp2pShHxJ/X/kxsi4p6IGN7sTL0VEXdGxIsRsaHbstER8WBEbK5/bWlmxqoMimKIiCHAbcBcYCLwuYiY2NxUlXob+A+ZORGYAVw1wI4P4FpgU7ND9IH/CvyvzDwT+BAD6BgjYhxwDVDLzMnAEGBBc1M15L8Bcw5Ythh4ODPPAB6uz/d7g6IYgOnAlsx8OjPfAu4F5jU5U2Uy8/nMfKw+/Rpd31zGNTdVdSKiFfgU8O1mZ6lSRBwPfBT4DkBmvpWZrzQ3VeWGAsdGxFBgBLC9yXl6LTN/Crx0wOJ5wF316buAS45qqD4yWIphHLC123wnA+gbZ3cR0QZMBR5tbpJKLQX+E/D/mh2kYhOAncB365fJvh0RI5sdqiqZuQ24Bfh74HlgT2b+7+amqtwpmfl8ffoF4JRmhqnKYCmGQSEiRgE/BK7LzFebnacKEfGvgBczc12zs/SBocA04I7MnAq8wQC5FAFQv94+j64CHAuMjIgvNDdV38mut3gOiLd5DpZi2Aac3m2+tb5swIiIYXSVwt2Z+bfNzlOhmcDFEfEsXZcA/zgi/ntzI1WmE+jMzHfO7pbTVRQDxQXAM5m5MzN/D/wt8C+bnKlqOyLiNID61xebnKcSg6UY1gJnRMSEiHgfXTfAVjY5U2UiIui6Tr0pM29tdp4qZeafZWZrZrbR9e/2k8wcED91ZuYLwNaI+GB90WzgiSZGqtrfAzMiYkT9/+hsBtDN9bqVwML69EJgRROzVGZoswMcDZn5dkRcDayi650Rd2bmxibHqtJM4N8Av4mIx+vLrs/MB5qYSYfn3wN3139geRr4YpPzVCYzH42I5cBjdL1z7lf0408JR8Q9wCzgxIjoBJYANwP3RcSX6Hri82ebl7A6fvJZklQYLJeSJEmHyWKQJBUsBklSwWKQJBUsBklSwWKQJBUsBklSwWKQJBX+P+A9Uj20na7OAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "error_train, error_val = learning_curve(X, y, Xval, yval, _lambda)\n", "plt.plot(error_train, label='train')\n", "plt.plot(error_val, label='validation')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Polynomial Regression\n", "The problem with our linear model was that it was too simple for the data and resulted in underfitting (high bias). In this part of the exercise, you will address this problem by adding more features.\n", "\n", "**Exercise**: Implement the function that maps the original training set X of size $m \\times 1$ into its higher powers. Specifically, when a training set X of size m × 1 is passed into the function, the function should return a $m \\times p$ matrix." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "def polynomial_features(X, p):\n", " return X" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Remember feature scaling:\n", "\n", "$$X := \\frac{X - \\mu}{\\sigma}$$\n", "\n", "Where $\\mu$ is the average value of $X$ and $\\sigma$ is the standard deviation.\n", "\n", "**Exercise**: Implement feature scaling." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "def scale_features(X):\n", " return X, 1, 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Add polynomial features to $X$ and normalize it." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "p = 8\n", "\n", "# X\n", "X_poly = polynomial_features(X[:, 1:], p) # ignore the bias column when adding polynomial features.\n", "X_poly, mu, sigma = scale_features(X_poly)\n", "X_poly = np.hstack((np.ones((m, 1)), X_poly))\n", "\n", "# X_val\n", "X_val_poly = polynomial_features(X[:, 1:], p) # ignore the bias column when adding polynomial features.\n", "X_val_poly = (X_val_poly - mu) / sigma\n", "X_val_poly = np.hstack((np.ones((m, 1)), X_val_poly))\n", "\n", "# X_test\n", "X_test_poly = polynomial_features(X[:, 1:], p) # ignore the bias column when adding polynomial features.\n", "X_test_poly = (X_test_poly - mu) / sigma\n", "X_test_poly = np.hstack((np.ones((m, 1)), X_test_poly))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Learning polynomial regression\n", "\n", "This example shows the learning curve and fit without regularization (high variance)." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "ename": "ValueError", "evalue": "shapes (12,2) and (9,) not aligned: 2 (dim 1) != 9 (dim 0)", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 4\u001b[0m result = minimize(linear_reg_cost_function, initial_theta, args=args,\n\u001b[1;32m 5\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'CG'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcompute_gradient\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m options={'maxiter': 1000, 'disp': True})\n\u001b[0m\u001b[1;32m 7\u001b[0m \u001b[0mtheta\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/optimize/_minimize.py\u001b[0m in \u001b[0;36mminimize\u001b[0;34m(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)\u001b[0m\n\u001b[1;32m 591\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_powell\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 592\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mmeth\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'cg'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 593\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_cg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 594\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mmeth\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'bfgs'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 595\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_bfgs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36m_minimize_cg\u001b[0;34m(fun, x0, args, jac, callback, gtol, norm, eps, maxiter, disp, return_all, **unknown_options)\u001b[0m\n\u001b[1;32m 1270\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1271\u001b[0m \u001b[0mgrad_calls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmyfprime\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwrap_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfprime\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1272\u001b[0;31m \u001b[0mgfk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmyfprime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1273\u001b[0m \u001b[0mk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1274\u001b[0m \u001b[0mxk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36mfunction_wrapper\u001b[0;34m(*wrapper_args)\u001b[0m\n\u001b[1;32m 298\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfunction_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mwrapper_args\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 299\u001b[0m \u001b[0mncalls\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 300\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrapper_args\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 301\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 302\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mncalls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunction_wrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m\u001b[0m in \u001b[0;36mcompute_gradient\u001b[0;34m(theta, X, y, _lambda)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcompute_gradient\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtheta\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_lambda\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mhx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m \u001b[0;34m@\u001b[0m \u001b[0mtheta\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0mcost\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m \u001b[0;34m@\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhx\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mregularization\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0m_lambda\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconcatenate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtheta\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mcost\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mregularization\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mValueError\u001b[0m: shapes (12,2) and (9,) not aligned: 2 (dim 1) != 9 (dim 0)" ] } ], "source": [ "_lambda = 0\n", "args = (X_poly, y, _lambda)\n", "initial_theta = np.ones(p+1)\n", "result = minimize(linear_reg_cost_function, initial_theta, args=args,\n", " method='CG', jac=compute_gradient,\n", " options={'maxiter': 1000, 'disp': True})\n", "theta = result.x" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xd4VGX2wPHvCRACSehIC70nIC2gCAhRFFQEREVdFCwr9oYKKGJh2dXVFcvPFcV1VRQFRBFXsNDBghJ6E0OvSg8lBFLO7497EwMkkwlkMiXn8zzzZO6dO/eeS8Kced/33vOKqmKMMcbkJczfARhjjAlsliiMMcZ4ZInCGGOMR5YojDHGeGSJwhhjjEeWKIwxxnhkicIYY4xHliiMMcZ4ZInCGGOMRyX9HUBhqFKlitarV8/fYRhjTFBZsmTJPlWtmt92IZEo6tWrR2Jior/DMMaYoCIiW73ZzrqejDHGeGSJwhhjjEeWKIwxxngUEmMUuUlLS2PHjh2kpqb6OxRzmoiICGJiYihVqpS/QzHGeCFkE8WOHTuIjo6mXr16iIi/wzEuVWX//v3s2LGD+vXr+zscY4wXQrbrKTU1lcqVK1uSCDAiQuXKla2lZ0wQCdlEAViSCFD2ezEmuIR0ojDGmFCVlpHJpr1Hi+RYlih8ZP/+/bRu3ZrWrVtTvXp1atWqlb188uRJr/Zx2223sX79eo/b/Pvf/2bChAmFEfIpZs2aRd++fT1us3TpUr755ptCP7YxxrPVO5Pp88YP3PTOIlJOpvv8eCE7mF0gL74I7dtDQsKf6+bOhcWLYejQs9pl5cqVWb58OQDPPvssUVFRPPbYY6dso6qoKmFhuefr9957L9/j3HfffWcVX2FYunQpq1evpmfPnn6LwZjiJDUtg9dmJzFuwSYqlg1ndN84yob7/mPc7y0KESkhIstE5Ct3ub6I/CwiG0RkkoiE+zyI9u2hf38nOYDzs39/Z30h27BhA7GxsQwYMIC4uDh2797N4MGDiY+PJy4ujlGjRmVv27lzZ5YvX056ejoVKlRg+PDhtGrVio4dO7Jnzx4AnnrqKV599dXs7YcPH06HDh1o2rQpP/74IwDHjh3j2muvJTY2luuuu474+PjsJJbT9OnTadq0KW3btmXatGnZ6xctWkTHjh1p06YNnTp1IikpiePHjzNq1CgmTJhA69atmTJlSq7bGWMKz+APlzB23kb6tanF7CFd6dmiRtEcOOtbrb8ewBDgY+Ard3kycKP7/C3gnvz20a5dOz3d2rVrz1jn0Zw5qlWqqI4c6fycM6dg7/fgmWee0ZdeeklVVZOSklREdPHixdmv79+/X1VV09LStHPnzrpmzRpVVe3UqZMuW7ZM09LSFNAZM2aoquojjzyizz//vKqqjhgxQl955ZXs7YcOHaqqqtOmTdMePXqoqurzzz+v9957r6qqLl++XMPCwnTZsmWnxHjs2DGtVauWbtiwQTMzM7Vfv37ap08fVVU9dOiQpqWlqarq119/rf3791dV1XfeeUcfeuih7H3ktV1uCvz7MaaYOpKapsdPpquq6o8b9umC3/YU2r6BRPXic9qvXU8iEgNcBfwdGCLO5TCXAH9xN/kAeBYY6/NgEhLgnnvgb3+DkSNP7YYqZA0bNiQ+Pj57+ZNPPuHdd98lPT2dXbt2sXbtWmJjY095T5kyZbjiiisAaNeuHQsXLsx13/369cveZsuWLQB8//33DBs2DIBWrVoRFxd3xvvWrl1LkyZNaNiwIQADBgxg/PjxABw6dIiBAweyceNGj+fl7XbGGO/M/20vT36+ir5tavJ4j2Z0bFjZL3H4u+vpVWAokOkuVwYOqWrW6MwOoFZubxSRwSKSKCKJe/fuPfdI5s6FsWOdJDF27J/dUD4QGRmZ/TwpKYnXXnuNOXPmsHLlSnr27JnrPQbh4X/2wJUoUYL09NwHsEqXLp3vNgU1YsQIevTowerVq/niiy/yvAfC2+2MMZ4dSjnJo5NXMOi/vxBRKoxLmp3n13j8lihEpBewR1WXnM37VXWcqsaranzVqvmWU/csa0xi8mQYNcr5mXPMwocOHz5MdHQ05cqVY/fu3Xz77beFfoxOnToxefJkAFatWsXatWvP2CY2NpakpCQ2b96MqvLJJ59kv5acnEytWk6+fv/997PXR0dHc+TIkXy3M8Z474cN++g+ZgHTlu/k/oRGTH+wC+3qVvJrTP5sUXQCeovIFmAiTpfTa0AFEcnqEosBdvo8ksWLneSQ1d2UkOAsL17s80O3bduW2NhYmjVrxsCBA+nUqVOhH+OBBx5g586dxMbG8txzzxEbG0v58uVP2aZs2bK89dZbXHHFFcTHx1Ojxp+DZMOGDePxxx+nbdu2WeNKAFxyySWsWLGCNm3aMGXKlDy3M8Z4r3JUOLUrlWHa/Z14rEdTIkqV8HdISCD8hxaRbsBjqtpLRD4FPlPViSLyFrBSVd/09P74+Hg9feKidevW0bx5c5/FHEzS09NJT08nIiKCpKQkLr/8cpKSkihZ0n9DVPb7McahqkxZsoM1uw7zbO+47HVFUcFARJaoanx+2wXifRTDgIkiMhpYBrzr53iC3tGjR7n00ktJT09HVXn77bf9miSMKfbce7e2t7qAJ6euYmHSPjpUDCN13Qwihj0ecGVuAuLTQlXnAfPc55uADv6MJ9RUqFCBJUvOaijIGOMDGfHxjH92HC92TiGsRAn+1rwUAx67mbDJk/wdWq4CIlEYY0xxcqB9J8Z0Oc4Fm1fw9waZ1HrstVPHSQOMvy+PNcaYYiEtI5NPE7eTmalUjS7N9EcSeK9BKrVGj3Tu4QrQJAGWKIwxxudW7Ujm6v/7nsenrGThhn0A1Fn5M/JW0dy7da4sURhjjI+kpmXwwte/0veNhRw4eJS3b2lH1yZVnaRwzTXQr1+R37t1NixR+FCJEiVo3bo1LVq04PrrryclJcXj9lFRUUUU2akSExN58MEHPW4zb948evXqletrN910E+effz6vvPIKTz/9NLNmzQLg1VdfzfecjQlld45P5K35G7m+Vklm/uceeuxZ57wwcSKowo03OstFeO/W2bDBbB8qU6ZMdpXWAQMG8NZbbzFkyBA/R3Wm+Pj4U2pPFcTvv//O4sWL2bBhwxmvvfrqq9x8882ULVv2XEM0JmgcSU2jVIkwIkqV4L6ERtzdtSGdGlWBFu87rYZ77oHPP4cvvjh1XCIhIWDHKaxFUUS6dOmS/WE6ZswYWrRoQYsWLbJLhOc0cOBAvvjii+zlAQMGMG3aNN5//3369etHz549ady4MUNzzJXxySef0LJlS1q0aJFdABCcVsrjjz9OXFwc3bt355dffqFbt240aNCAL7/8Eji1tfDLL79klwq/6KKL8p046fLLL2fnzp20bt2ahQsXcuuttzJlyhRef/11du3aRUJCAgkB+sdvTGGb++seeryygNdnOyX2L2xQ2UkScGrh0QAfvD6DNyVmA/3hTZnx/m/9eMZj/I+bVVU15UR6rq9PXrxNVVX3Hz1xxmveiIyMVFWnfHjv3r31zTff1MTERG3RooUePXpUjxw5orGxsbp06dJTtp83b94pJb7r1aunaWlp+t5772n9+vX10KFDevz4ca1Tp45u27ZNd+7cqbVr19Y9e/ZoWlqaJiQk6NSpU1VVTylP3rdvX73sssv05MmTunz5cm3VqpWqqs6dO1evuuoqVVVNTk7OLhU+c+ZM7dev3xnb5LR582aNi4vLXh40aJB++umnqqpat25d3bt3b67/NlZm3ISS/UdP6MMTl2ndYV9p95fn6ZKtB87cyIdTGZwtCqvMuIiEAa2AmsBxYLWq7vFt+goNx48fp3Xr1oDTorjjjjsYO3Ys11xzTXYF2X79+rFw4ULatGmT/b6uXbty7733snfvXj777DOuvfba7DupL7300uw6TbGxsWzdupX9+/fTrVs3soojDhgwgAULFtC3b1/Cw8OzZ6Br2bIlpUuXplSpUrRs2TK7DHlOycnJDBo0iKSkJESEtLQ0n/37GBMKFibt5eGJy0k+nsaDlzbmvoSGlC55Wn2mnIVHs7qYci4HuDwThYg0xCmn0R1IAvYCEUATEUkB3gY+UNXMvPYRSCbd1THP18qEl/D4eqXIcI+v57nfHGMUBTVw4EA++ugjJk6ceMqUqFllxMG7UuKlSpXKLgcQFhaW/f6wsLBc3zty5EgSEhKYOnUqW7ZsoVu3bmcVvzHFxXnREdSvEsnoa1rQrHq53DfyVHg0CBKFpzGK0cBHQENV7aGqN6vqdap6PtAbKA/cUhRBhpIuXbrwxRdfkJKSwrFjx5g6dSpdunQ5Y7tbb701e/zi9EmMTtehQwfmz5/Pvn37yMjI4JNPPqFr165nFV9hlgo/vQy5MaFAVZn4yzZGfrEagKbVo/n07o55JwmAoUPPTAgJCc76IJBni0JVb/Lw2h6cSYdMAbVt25Zbb72VDh2cclZ//etfT+l2ylKtWjWaN29O3759891njRo1eOGFF0hISEBVueqqq+jTp89ZxTd06FAGDRrE6NGjueqqq85qH1kGDx5Mz549qVmzJnMD9PpwYwpi2/4Uhn++kh837ufCBpVITcsgolSJgCviV9jyLTMuIkuA/wIfq+rBIomqgEKxzHhKSgotW7Zk6dKlZ8wdEQqC/fdjipeMTOW9Hzbzr+/WUzIsjCevbM6N7WsTFhbcCcLbMuPeXB57A85A9mIRmSgiPSTU06efzZo1i+bNm/PAAw+EZJIwJtgcOHaS12Yn0alhFWYOuZi/XFAn6JNEQeR71ZOqbgBGiMhIoBdO6yJDRN4DXlPVAz6Osdjp3r07W7du9XcYxhRrJ9Mz+WLZTq5rF0PV6NLMeLALMRXLhHw3U268ujNbRM4HbgOuBD4DJgCdgTlAa59Fd460iGaJMgWTX3enMf62Yvshhk5Zyfo/jlC9fAQXN6lK7UrFt8KAN/dRLAEO4cw0N1xVT7gv/SwihT/BcyGJiIhg//79VK5c2ZJFAFFV9u/fT0REhL9DMeYMx09mMGbmet79fjPnRUfwn4HxXNykqr/D8jtvWhTXqzPr3BlUtV8hx1NoYmJi2LFjB3v37vV3KOY0ERERxMTE+DsMY85w5/hEvt+wj5s61OGJK5tRLqKUv0MKCHle9SQiN+Nc6ZTrDXXuDXk1VPV7H8bnldyuejLGGG8cTk0j3C3i9/Om/WSoclHDKv4Oq0h4e9WTpxZFZWCZ2/W0hD/vzG4EdAX2AcMLIVZjjPGL2ev+YMTU1VzTthbDejbjggaV/R1SQPJ0w91rIvIGcAnQCTgfp9bTOuAWVd1WNCEaY0zh2n/0BM/9by1frthFs+rR9Iyr7u+QAprHMQpVzQBmug9jjAl6C37by8OTlnMkNY1Hujfhnm4NCS9pMy54YhMXGWOKlerlI2hUNYrR17SgSbVof4cTFCyNGmNCWmam8vHP2xgxdRUATapFM/nujpYkCsBaFMaYkLVl3zGGf76SRZsO0LFB5ewifqZgvLnhrgIwEKiXc3tVfdB3YRljzNnLyFT++/1mXp65nlJhYbzQryU3tK9tN9+eJW9aFDOARcAqICgmKTLGFG8Hjp3k/+Yk0blRVUb3bUH18lYJ4Fx4kygiVHWIzyMxxpiCePFFaN8+e0KgE+kZfP7xHG7YvYyqw4Yy46Eu1KpQPIv4FTZvBrM/FJE7RaSGiFTKevg8MmOM8aR9e2fe6blzWbbtIFe/8C1PrD3J9w3aAhBTsawliULiTYviJPASMALIqvehQANfBWWMMflKSCDl40m8/PLn/LfFMaofPcB7F9fk4uu7+zuykONNongUaKSq+3wdjDHGFMTgrZF836InNy+dzrALqhHdf6C/QwpJ3nQ9bQBSfB2IMcZ4I/l4GqlpGQA8WOkIk6b/g9EXViH6rTfA5mb3CW9aFMeA5SIyF8iai+KcL48VkdrAeKAaTlfWOLe+VCVgEs7luFuA/oE6V7cxpmjNXPsHT32ximvaxDA8Yjcd7roJJk92BrQTEpwxi6xlU2i8aVF8Afwd+BGnimzW41ylA4+qaixwIXCfiMTiVKSdraqNgdlYhVpjir19R09w/8dLuXN8IhXLhnNly+qwePGpSSEhwVlevNi/wYagPOejKGoiMg14w310U9XdIlIDmKeqTT291+ajMCZ0zVu/h4cnLSflRAYPXNKIu7s1pFQJqz5UGApjPoqsHTUGngdiceajAEBVC+2qJxGpB7QBfgaqqepu96XfcbqmjDHFVM0KZWhaLZrRfVvQ2Ooz+YU3afk9YCxOV1ECzrjCR4UVgIhEAZ8BD6vq4ZyvqdPcybXJIyKDRSRRRBJtulNjQkdmpvLhoq088fmfRfwm3dXRkoQfeZMoyqjqbJxuqq2q+ixwVWEcXERK4SSJCar6ubv6D7fLCffnntzeq6rjVDVeVeOrVrXJz40JBZv2HuXGcYsY+cVqdhxMyb66yfiXN1c9nRCRMCBJRO4HdgJR53pgcW6ZfBdYp6pjcrz0JTAIeMH9Oe1cj2WMCWzpGZm8s3Azr8z6jYiSYbx03flc1y7G7qwOEN4kioeAssCDwN9wpkYdVAjH7gTcAqwSkeXuuidxEsRkEbkD2Ar0L4RjGWMC2MGUNN6av5GEplX5W58WnFfOivgFkoC56ulc2FVPxgSfE+kZTFmyg5va1yEsTNh16Dg1K5Txd1jFyjlf9SQi/yOPgWQAVe19lrEZY4q5JVsPMuyzlWzYc5S6lSLp3LiKJYkA5qnr6V/uz35Adf680ukm4A9fBmWMCU3HTqTzr+/W8/6PW6hZvgwf3N6Bzo2r+Dssk488E4WqzgcQkZdPa5r8T0Ssn8cYU2CDP0zkhw37GdSxLo/3bEZUaZuNORh4c3lspIhk31wnIvWBSN+FZIwJCS++CHPnkpzyZxG/hysd5dPozTzXp4UliSDizW/qEWCeiGwCBKgLDPZpVMaY4Ne+Pd88/k9GXplCv44NeCLid9pnFfEzQSXfRKGq37hlPJq5q35V1ROe3mOMKd72HEnlmZ3l+Lr7A8Tu3srVP6yCt1+2yq5Byqu2n5sYVvg4FmNMCJi7fg8PT1zO8bQMHu/RlMFzfqHU6FEwcqQliSBlJRiNMYUqpkIZ4mqWY8aDXbiP7ZR6600nSYwdaxMLBSlLFMaYc5KZqXzw4xaGf7YSgMbVovn4zgtptGbxnxMJjRrl/Ozf35JFEMo3UYjjZhF52l2uIyIdfB+aMSbQbdx7lP5v/8QzX65hV3LqqUX8bGKhkJFvCQ8RGQtkApeoanMRqQh8p6rtiyJAb1gJD2OKVlpGJuMWbOK12UmUKVWCkb1iubZtLSviF2QKbeIi4AJVbSsiywBU9aCIhJ9zhMaYoJV8PI1xCzbRvfl5PNs7jvOirYhfKPMmUaSJSAncuk8iUhWnhWGMKUZS0zL4NHE7Ay6oS5Wo0nzzcBdqlLf6TMWBN4nidWAqcJ6I/B24DnjKp1EZYwLK4i0HGDZlJZv2HaN+lSg6N65iSaIY8eaGuwkisgS4FOfO7L6qus7nkRlj/O7oiXRe/OZXxv+0lZiKZfjwDiviVxx5W2wlCTictb2I1FHVbT6LyhgTEAaPT+SnTfu5rVM9Hru8KZFWn6lYyve3LiIPAM/glBbPwGlVKHC+b0MzxvjDoZSTlC5ZgjLhJXj08iaA0K5uRX+HZfzI26lQm6rqfl8HY4wpAi++CO3bn1pOY+5cWLyYGVfcwtPTVnNt2xieuLI57epW8l+cJmB4c2f2diDZ14EYY4pI+/an3iE9dy57Bt3JXSXP594JS6lRvgx9Wtfyb4wmoHiaCnWI+3QTTpnx6UB21VhVHePj2IwxvpB1h3T//nDPPcz56kceHvgaJw4ow69oxl8716dkCavuY/7kqesp2v25zX2Euw/wMJe2MSYIJCTAPffA3/5Gnaf+Tqt6VXiudxwNqkb5OzITgDxNhfocgIhcr6qf5nxNRK73dWDGGN/IyFQ+GD+TX1ed4MWRI2k09hU+nNwRLEmYPHjTvnzCy3XGmACX9McRrn/xG0b9msbeTt1IHfmMVXU1+fI0RnEFcCVQS0Rez/FSOSDd14EZYwrPyfRM3p6/kf+bs4HIzJO82rIsff7S3Snil7Oqq00sZHLhaYxiF5AI9AaW5Fh/BGcebWNMkDicmsa7P2zm8rhqPNs7jipRpU/dICHBkoTJk6cxihXAChH5WFXTijCm0OPhunWGDvVfXCa4FPDvKDUtg0mLt3PLhU4Rv28fvphq5azKqym4fMcoLEkUglyuW6d/f2e9Md4qwN/Rz5v2c8VrC3nmyzX8tMm5V9aShDlbdrH02XjxxTMH/ubOddbnJud1608//ef0kNbUNwXhxd/RkdQ0nvpiFTeMW0R6ZiYT/noBnRpZET9zbvJMFCLyofvzoaILJ0icTQshx3Xr3HNP0SWJgiY1E9jy+TsaPH4JE37exh2d6/PtwxdbkjCFQ1VzfQBrgZrACqAiUCnnI6/3+ePRrl07LXJz5qhWqaI6cqTzc86cwt2+sOPMOt7py6Hmn/8889zmzHHWB+J+CyqXv6P9R09oyol0VVVN3HJAl2w9ULQxmaAFJKoXn7GeEsWDwDqcsh2bgM05Hpu82XlRPfySKFSd/6zg/PTE3x/W/kpS/uCrf2t//w5zOWbm7Nk6rcNV2uapr/Tv09cWXRwmZJxzosjeAMZ6syN/PgK+RREI30a9TWp5CYRz8JavEqO/E26O38HuQ8f1jvcXa91hX2nvJyfrut3JRRuLCQmFliicfdEKuN99nO/Ne871AfQE1gMbgOGeti3yRBEI3y4LojA+4ILtnM81MRb1fgtg1trftcXT32jTp2bouPkbNT0j02+xmOBWmC2KB4HVwCj3sQp4wJudn+0DKAFsBBrgFCJcAcTmtX2RJ4pg/HZdGB/w/v5G7a1QbVG4kv44ogPf/Vk37z3ql+Ob0FGYiWIlEJljORJY6c3Oz/YBdAS+zbH8BPBEXtv7bYwiGBR2UguAb9QeheAYRXpGpr6zYKMOmbTc58cyxYu3icKb+ygEZwrULFnTofpSLZwJk7LscNeZgho69MxLcRMSzu6O8LlzYexYGDnS+RmIReQWLz713oKcdYwCcb/5+O2PI1w79kdGT1/HwZSTpKZl5P8mYwqZOEnFwwbOBEaDgKnuqr7A+6r6qs+CErkO6Kmqf3WXbwEuUNX7c2wzGBgMUKdOnXZbt271VTgG/rxXJOvD8vRlU6hOpmcydt5G3pibRHREKZ65OpberWo6RfyMKSQiskRV4/PbzpsSHmOA24AD7uM2XyYJ106gdo7lGHddzrjGqWq8qsZXrVrVx+EYf32jLq4Op6bx/o+bubJlDWY+cjF9WteyJGH8Jt8WhT+ISEngN+BSnASxGPiLqq7Jbfv4+HhNTEwswgiNKXzHT2bwyS/bGHRRPUqECXsOp3Ke1WcyPuRti8JTmXG/UdV0Ebkf+BbnCqj/5pUkjAkFP27cx/DPVrHtQApNq0fTqVEVSxImYARkogBQ1RnADH/HYYwvHU5N4/kZv/LJL9uoW7ksn9x5IR0bVvZ3WMacIt8xChH5pzfrjDEFN3h8IpMWb+OuixvwzUMXW5IwAcmbFsVlwLDT1l2RyzpjjBf2Hz1B2fCSlAkvwdCezSghQqvaFfwdljF58lRm/B4RWQU0FZGVOR6bcW7CM8YUgKoybflOuo+ZzyuzfgOgbZ2KliRMwPPUovgY+Bp4HhieY/0RVT3g06iMCTG7k4/z1NTVzP51D61rV+C6djH+DskYr3maMzsZSBaR07uYokQkSlW3+TY0Y0LDzLV/8Mik5WRkKiN7xXKre/mrMcHCmzGK6YDilO2IAOrjVHWN82FcxoSM+lUiia9XkVG9W1Cncll/h2NMgeWbKFS1Zc5lEWkL3OuziIwJcukZmfz3h838uvsIY25oTaPzonj/tg7+DsuYs1bg+yhUdamIXOCLYIwJdut2H2bYZytZuSOZy2KrkZqWQUSpEv4Oy5hzkm+icIsCZgkD2gK7fBaRMUHoRHoG/567kTfnbqBC2VL8+y9tubJldavPZEKCNy2K6BzP03HGLD7zTTjGBKejqel8tGgrvVvVZGSvWCpGhvs7JGMKjTdjFM8BiEiUu3zU10EZEwxSTqbz8c/buK1TfSpHlebbhy+manRpf4dlTKHzpuupBfAhUMld3gcMUtXVPo7NmID1w4Z9DP98JdsPHCe2RjkualTFkoQJWd50PY0DhqjqXAAR6eauu8iHcRkTkJKPp/GP6euYlLid+lUimTT4Qi5oYPWZTGjzJlFEZiUJAFWdJyKRPozJmIB114eJLN5ykLu7NuTh7o3tiiZTLHiTKDaJyEic7ieAm4FNvgvJmMCy98gJIkuXoGx4SYb1bEbJsDBaxpT3d1jGFJl8y4wDtwNVgc9xrnaq4q4zJqSpKp8v3cFlr8znlZlOEb82dSpakjDFjjdXPR0EHiyCWIwJGDsPHWfE1FXMW7+XtnUqcEP72vm/yZgQFbAz3BnjL9+t+Z1HJi1HgWevjuWWjlbEzxRvliiMcakqIkLD86K4sEFlnu0dR+1KVsTPGG/GKIwJaekZmYydt5FHJi0HoGHVKN69tb0lCWNc3txw93ouq5OBRFWdVvghGVN01u46zNDPVrB652F6xFkRP2Ny403XUwTQDPjUXb4W2Ay0EpEEVX3YV8EZ4yupaRm8MWcDb83fSIWy4Ywd0JYrWtbwd1jGBCRvEsX5QCdVzQAQkbHAQqAzsMqHsRnjM8dOpPPxL9vo07oWI3s1p0JZK+JnTF68SRQVgSic7iaASKCSqmaIyAmfRWZMITt2Ip0JP2/ljs4NqBxVmpmPXEzlKKvPZEx+vEkULwLLRWQeznSoFwP/cMt4zPJhbMYUmgW/7eWJz1exK/k4LWqV56KGVSxJGOMlb264e1dEZgBZczk+qapZExc97rPIjCkEh1JOMnr6OqYs2UGDqpF8eldH4utV8ndYxgQVb++jCAP2uts3EpFGqrrAd2EZUzgGf7iEJVsPcl9CQx64xIr4GXM2vLk89p/ADcAaINNdrYAlChNbZUtgAAAWUklEQVSQ9hxJJap0ScqGl+TJK5tTqoQQV9PqMxlztrxpUfQFmqqqDVybgKaqTFmyg9HT13F9uxie6hVL69oV/B2WMUHPqzLjQCnAEoUJWNsPpPDk1FUsTNpH+3oVuemCOv4OyZiQ4U2iSMG56mk2OZKFqlpFWRMQvln9O0MmL0eAUX3iuPmCuoRZET9jCo03ieJL92FMQMkq4tekWhSdGlXhmatjialo9ZmMKWzeXB77QVEEYoy30jIyGbdgE+t/P8LrN7WhQdUo3hkY7++wjAlZeVaPFZHJ7s9VIrLy9Me5HFREXhKRX919TRWRCjlee0JENojIehHpcS7HMaFn9c5k+rzxAy99u54MVU6kZ/g7JGNCnqcWxUPuz14+OO5M4AlVTXcvv30CGCYiscCNQBxQE5glIk2y6kyZ4is1LYPXZicxbsEmKkWG8/Yt7egRV93fYRlTLOSZKFR1t/u0O7BAVZMK66Cq+l2OxUXAde7zPsBE91LczSKyAeeO8J8K69gmOKWczGDy4u1c27YWI66MpXzZUv4OyZhiw5vB7DrA2yJSD1iCc6PdQlVdXkgx3A5Mcp/XwkkcWXa460wxdPREOh8t2sqdXRpQKTKcmUO6UinSqrwaU9S8Gcx+BkBEygB34tR3ehXwWAtBRGYBufUNjMia8EhERgDpwISChQ0iMhgYDFCnjl0zH2rmrd/DiKmr2ZV8nFYxFejYsLIlCWP8xJsSHk8BnXBKjS8DHsOZj8IjVe2ez35vxRn/uFRV1V29E6idY7MYd11u+x8HjAOIj4/X3LYxwefgsZP8bfpaPl+6k0bnRTHl7otoV7eiv8MypljzpuupH863/unAfOCncy3nISI9gaFAV1VNyfHSl8DHIjIGZzC7MfDLuRzLBJe7PlrC0q0HefCSRtx3SSNKl7Qifsb4mzddT21FpBxOq+IyYJyI7FHVzudw3DeA0sBMEQFYpKp3q+oa97LctTjJ6T674in07TmcSmTpkkSWLsmIK5tTqkQYsTXL+TssY4zLm66nFkAXoCsQD2zHi64nT1S1kYfX/g78/Vz2b4KDqvJp4g7+Nn0t/eNrM7JXLK2siJ8xAcebrqcXcBLD68BiVU3zbUimONi23yni9/2GfXSoX4kBVsTPmIDlTddTLxEJB5oATUVkvSULcy6+Wb2bRyatoESYMLpvC/7SoY4V8TMmgHnT9dQVGA9swZkzu7aIDLIZ7kxBZRXxa1q9HF2bVOXpq2OpWaGMv8MyxuTDm66nMcDlqroeQESaAJ8A7XwZmAkdJ9MzeXv+Rn7bc5TXb2xN/SqRvHWL/fkYEyzyLAqYQ6msJAGgqr/hTGRkTL5W7jhE7ze+5+WZvwFwMiMzn3cYYwKNNy2KRBH5D/CRuzwASPRdSCYUpKZl8MrM33hn4SaqRpfmnYHxXBZbzd9hGWPOgjeJ4h7gPiBrRruFwJs+i8iEhJSTGUxZsoMb2tdm+BXNKV/GGqHGBCv5s3pG8IqPj9fERGvk+NuR1DQ+XLSVuy5uSIkw4eCxk1S0+kzGBCwRWaKq+c76lWeLQkRWAXlmEVU9/yxjMyFozq9/MGLqav44nEqb2hXp2LCyJQljQoSnrqfrgeNFFYgJTvuPnmDUV2uZtnwXTapF8eaAi2hTx4r4GRNKPCWKj906Tx+q6i1FFpEJKvd8tJRl2w/ycPfG3NutEeElvbmQzhgTTDwlinAR+QtwkYj0O/1FVf3cd2GZQPZ7cirREU4Rv5G9YgkvGUbT6tH+DssY4yOeEsXdOJfCVgCuPu01BSxRFDOqysTF2/nH9HX0b+8U8WsZU97fYRljfMzTnNnfA9+LSKKqvluEMZkAtHX/MYZ/toqfNu2nY4PKDOxY198hGWOKiDdFAS1JFHMzVu1myOTllAoL4/l+LbmxfW3ceUSMMcWANzfcmWIqq4hf8xrluKTZeYzsFUuN8lbEz5jixuMlKuKo7WkbE3pOpmfy6qzfuP+TZagq9atE8uaAdpYkjCmmPCYKdW7bnlFEsZgAsHz7Ia7+v+95dVYSJcPEivgZY7zqeloqIu1VdbHPozF+c/xkBmNmrufd7zdzXnQE7w6K59LmVsTPGONdorgAGCAiW4FjOJMXqZXwCC2paRlMXbaLmzrUYfgVzYiOsCJ+xhiHN4mih8+jMH5xODWN8T9u4e6uDakYGc7sIV0pX9YShDHmVPnWW1DVrUBt4BL3eYo37zOBbdbaP7hszHzGzPyNxVsOAliSMMbkyps5s58B4oGmwHs4s9t9BHTybWjGF/YfPcGz/1vL/1bsoln1aN4ZGM/5MRX8HZYxJoB50/V0DdAGWAqgqrtExAr7BKmsIn5DLmvC3V0bWhE/Y0y+vEkUJ1VVRUQBRCTSxzGZQrY7+TjlIkoRWbokT1/tFPFrUs1yvTHGO958nZwsIm8DFUTkTmAW8B/fhmUKQ2amMuHnrVw2ZgEvf/cbAC1qlbckYYwpEG9qPf1LRC4DDuOMUzytqjN9Hpk5J5v3HWP4Zyv5efMBOjWqzK0X1fN3SMaYIOXNYPY/VXUYMDOXdSYATV/pFPELLxnGi9eez/XxMVbEzxhz1rzperosl3VXFHYg5tw5FVcgrmY5LoutxqwhXelvlV6NMecozxaFiNwD3As0EJGVOV6KBn7wdWDGeyfSM/j3nA1s2HuUf/+lLfWqRPLGX9r6OyxjTIjwOGc28DXwPDA8x/ojqnrAp1EZry3ddpBhU1aStOco/drU4mRGJqVLlvB3WMaYEOJphrtkIBm4CUBEzgMigCgRiVLVbUUToslNysl0/vXtb7z342ZqlIvgvdvak9D0PH+HZYwJQfmOUYjI1SKSBGwG5gNbcFoa50xEHhURFZEq7rKIyOsiskFEVoqI9Z/k4URaJv9buYtbLqzLd0O6WpIwxviMN4PZo4ELgd9UtT5wKbDoXA/sToh0OZCzZXIF0Nh9DAbGnutxQkny8TRen51EekYmFSPDmTWkK6P6tCCqtE1UaIzxHW8SRZqq7gfCRCRMVefi1H46V68AQwHNsa4PMF4di3Bu8qtRCMcKet+u+Z3LxszntdlJLNnqFvErY0X8jDG+581X0UMiEgUsACaIyB6ceSnOmoj0AXaq6orTLt2sBWzPsbzDXbf7XI4XzPYeOcGzX65h+qrdNK9RjncHtadlTHl/h2WMKUa8SRR9gFTgEWAAUB4Yld+bRGQWUD2Xl0YAT+J0O501ERmM0z1FnTp1zmVXAe3eCUtYsT2Zxy5vwl1dG1KqhBXxM8YULU/3UTwM/AgsVdUMd/UH3u5YVbvnsd+WQH0gqzURgzPdagdgJ87cF1li3HW57X8cMA4gPj5ec9smWO08dJzyZUoRVbokz1wdR+mSYTS2+kzGGD/x9PU0BngV2CMi80XkHyLSS0QqncsBVXWVqp6nqvVUtR5O91JbVf0d+BIY6F79dCGQrKrFptspM1MZ/9MWLh8znzE5ivhZkjDG+JOn+ygeAxCRcJzB64uA24BxInJIVWN9EM8M4EpgA85Merf54BgBaePeowz/bCWLtxykS+Mq3Napnr9DMsYYwLsxijJAOZyxifLALmBVYQXgtiqynitwX2HtO1h8tXIXQyavIKJkGC9ddz7XtbMifsaYwOFpjGIcEAccAX7GGa8Yo6oHiyi2kKeqiAgta5WnZ1x1nurVnPOiI/wdljHGnMLTGEUdoDTwO86A8g7gUFEEFepS0zJ46dtfueejpagqdStH8vpNbSxJGGMCkqcxip7i9H/E4YxPPAq0EJEDwE+q+kwRxRhSlmw9wNApK9m49xjXto2xIn7GmIDncYzCHTNYLSKHcAoEJgO9gA6AJYoCOHYinZe+Xc8HP22hZvkyfHB7B7o2qervsIwxJl+exigexGlJXASk4YxR/Aj8l0IczC4u0jIymbFqNwMvrMvjPZtZfSZjTNDw9GlVD/gUeKQ43ctQmA6lnOS9H7bwwCWNqFA2nFmPdqVchNVnMsYEF09jFEOKMpBQ8/Wq3YyctoaDKSe5qGFlLmhQ2ZKEMSYoWf9HIdtzOJWnp63hmzW/E1ezHB/c3p64mlbEzxgTvCxRFLL7Pl7Kih3JDOvZjDu71KekFfEzxgQ5SxSFYMfBFCqUDSeqdEme7R1HRKkSNKwa5e+wjDGmUNjX3XOQmam8/8NmLn9lAS9/tx6AuJrlLUkYY0KKtSjO0oY9ThG/xK0H6dqkKnd0ru/vkIwxxicsUZyFL1fs4rHJKyhbugRj+rfimja1rIifMSZkWaIogMxMJSxMaBVTnitbVmfEVbFUjS7t77CMMcanbIzCC6lpGbzw9a/c/dGS7CJ+r97YxpKEMaZYsESRj182H+DK1xby1vyNVCwbTlpGSM26aowx+bKupzwcPZHOP7/+lQ8XbaV2pTJ8dMcFdG5cxd9hGWNMkbNEkYf0jEy+W/s7t3eqz2M9mlA23P6pjDHFk3365XDw2Ene+2EzD17amAplw5n9aDer8mqMKfbsUxBnStIZq37nmS9Xcygljc6Nq9KhfiVLEsYYgyUK/jicysgvVvPd2j9oWas842+/gNia5fwdljHGBIxinyjum7CUVTuTeeKKZtzR2Yr4GWPM6Yp9ohjVpwURpcJoYPWZjDEmV8U+UVg3kzHGeGb9LMYYYzyyRGGMMcYjSxTGGGM8skRhjDHGI0sUxhhjPLJEYYwxxiNLFMYYYzyyRGGMMcYjUQ3+iXhEZC+w1d9x+FgVYJ+/g/Cj4nz+xfncwc7fl+dfV1Wr5rdRSCSK4kBEElU13t9x+EtxPv/ifO5g5x8I529dT8YYYzyyRGGMMcYjSxTBY5y/A/Cz4nz+xfncwc7f7+dvYxTGGGM8shaFMcYYjyxRBAkReVREVESquMsiIq+LyAYRWSkibf0dY2ETkZdE5Ff3/KaKSIUcrz3hnvt6Eenhzzh9SUR6uue4QUSG+zseXxOR2iIyV0TWisgaEXnIXV9JRGaKSJL7s6K/Y/UVESkhIstE5Ct3ub6I/Oz+DUwSkfCijskSRRAQkdrA5cC2HKuvABq7j8HAWD+E5mszgRaqej7wG/AEgIjEAjcCcUBP4E0RKeG3KH3EPad/4/yuY4Gb3HMPZenAo6oaC1wI3Oee83Bgtqo2Bma7y6HqIWBdjuV/Aq+oaiPgIHBHUQdkiSI4vAIMBXIOKPUBxqtjEVBBRGr4JTofUdXvVDXdXVwExLjP+wATVfWEqm4GNgAd/BGjj3UANqjqJlU9CUzEOfeQpaq7VXWp+/wIzgdmLZzz/sDd7AOgr38i9C0RiQGuAv7jLgtwCTDF3cQv526JIsCJSB9gp6quOO2lWsD2HMs73HWh6nbga/d5cTn34nKeuRKRekAb4Gegmqrudl/6Hajmp7B87VWcL4WZ7nJl4FCOL0x++Rso9nNmBwIRmQVUz+WlEcCTON1OIcnTuavqNHebEThdEhOKMjbjPyISBXwGPKyqh50v1g5VVREJucs1RaQXsEdVl4hIN3/Hk5MligCgqt1zWy8iLYH6wAr3P0oMsFREOgA7gdo5No9x1wWVvM49i4jcCvQCLtU/r+UOiXP3QnE5z1OISCmcJDFBVT93V/8hIjVUdbfbxbrHfxH6TCegt4hcCUQA5YDXcLqVS7qtCr/8DVjXUwBT1VWqep6q1lPVejjNzraq+jvwJTDQvfrpQiA5R9M8JIhIT5xmeG9VTcnx0pfAjSJSWkTq4wzo/+KPGH1sMdDYveolHGcA/0s/x+RTbp/8u8A6VR2T46UvgUHu80HAtKKOzddU9QlVjXH/r98IzFHVAcBc4Dp3M7+cu7UogtcM4EqcgdwU4Db/huMTbwClgZlui2qRqt6tqmtEZDKwFqdL6j5VzfBjnD6hqukicj/wLVAC+K+qrvFzWL7WCbgFWCUiy911TwIvAJNF5A6cStH9/RSfPwwDJorIaGAZTiItUnZntjHGGI+s68kYY4xHliiMMcZ4ZInCGGOMR5YojDHGeGSJwhhjjEeWKIzPiUh1EZkoIhtFZImIzBCRJiLSLatCpr+JyCgR8XjzXyEdp4KI3FsI+5knIn6ZR1lE2oiIx0s0ReR+Ebm9qGIyvmWJwviUewPVVGCeqjZU1XY4VWADqlaPqj6tqrOK4FAVgAIlCvemykD6v/ok8Ho+2/wXeKAIYjFFIJD++ExoSgDSVPWtrBWqukJVF7qLUSIyxZ13YoKbWBCRp0VksYisFpFxOdbPE5F/isgvIvKbiHRx15cVkcnuPAZT3fr98e5rl4vITyKyVEQ+desInUJE3heR69znW0TkOXf7VSLSLJftp4vI+e7zZSLytPt8lIjcKSJRIjI7xz6yqr6+ADQUkeUi8pL7nsfdc10pIs+56+qJMw/FeGA1p5byOD2WM85PnHksPs2xTXbrLa9/Dy/POxo4P6tIpYi8luPce4jIAhEJc++k3yJOuRkT5CxRGF9rASzx8Hob4GGc+RYa4NyZC/CGqrZX1RZAGZx6T1lKqmoH933PuOvuBQ668xiMBNoBiDPR01NAd1VtCyQCQ7yIe5+7/VjgsVxeXwh0EZHyOHeHZ8XdBVgApALXuPtIAF52k91wYKOqtlbVx0XkcpwSJB2A1kA7EbnY3Vdj4E1VjVPVrbkF6eH8ZgEXiEiku+kNOHf35vfvkd95x+MkrixPADeISAJOK+M2Vc2qfJro/nuYIGeJwvjbL6q6w/1wWQ7Uc9cnuK2CVTj1+ONyvCerUNySHNt3xpmvAVVdDax011+Ik4R+cEtCDALqehFXbsfIaSFwMU6CmI7TMioL1FfV9YAA/xCRlTgf2rXIvbvtcvexDFgKNMNJEABb3blGPMn1/NwCct8AV4tISZw5DqbltX0BzrsGsDdrwW053IkzydQbqroxx7Z7gJr5xG+CgNV6Mr62hj8LmuXmRI7nGUBJEYkA3gTiVXW7iDyLU03z9PdkkP/fsAAzVfWmAkWd/zEW43y73oTzIVkF5wMzq/U0AKgKtFPVNBHZcto55IzveVV9+5SVzlwMx7yI09P5TQTuBw4Aiap6xG3VePr3yO+8j3PmebQE9nNmUohwtzdBzloUxtfmAKVFZHDWChE5P2tsIQ9ZH0T73P5zT4kmyw+4heLEmTqzpbt+EdBJRBq5r0WKSJMCnsMZ3BnntgPXAz/htDAew+l2AiiPM7dAmtstk/Wt/QgQnWNX3wK35xgnqCUi5xUgFE/nNx9oi5PAJnqxvTfWAY2yFkSkLvAoThfiFSJyQY5tm3BqN5UJUpYojE+5c0hcA3QX5/LYNcDzOLOU5fWeQ8A7OB8y3+J8e8/Pm0BVEVkLjMZpySSr6l7gVuATtxvoJ5zuncKwECcZHHefx7g/wZlkKd7tOhsI/Aqgqvtxun1Wi8hLqvod8DHwk7vtFE5NJB55Oj+3ou5XOHNuf5Xf9l4e71egvIhEu62Td4HHVHUXzlzO/3FbhOB0y830dt8mcFn1WBMSRKQEUEpVU0WkIc64QFP3m78pRCLyCHBEVf/jYZs2wBBVvaXoIjO+YmMUJlSUBeaKMzuaAPdakvCZsThdbp5Uwbn6zIQAa1EYY4zxyMYojDHGeGSJwhhjjEeWKIwxxnhkicIYY4xHliiMMcZ4ZInCGGOMR/8PavRTnX0lYd0AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Plot training data\n", "plt.plot(X[:,1], y, 'rx', label='Training data')\n", "\n", "# Plot polynomial fit\n", "space = np.array(np.arange(np.min(X) - 5, np.max(X) + 5, 0.05)).reshape((1912, 1))\n", "X_poly_plt = polynomial_features(space, p)\n", "X_poly_plt = (X_poly_plt - mu) / sigma\n", "X_poly_plt = np.hstack((np.ones((X_poly_plt.shape[0], 1)), X_poly_plt))\n", "plt.plot(space, X_poly_plt @ theta, '--', label='Polynomial fit')\n", "\n", "# Plot\n", "plt.legend()\n", "plt.xlabel('Change in water leven (x)')\n", "plt.ylabel('Water flowing out of the dam (y)')\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEICAYAAABbOlNNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAHBBJREFUeJzt3Xu8VXWd//HXOw7KVbkZys3DpMk9wR3YmIlBBjWGmgpOTeBDZcbR1Gn6zTDVBJn+st/DUeuR2lhZZqYylMlMNqYGqXmJg6mBqOCFOKDcFASBFPv8/lgL2t/T3hxgb9icc97Px2M/zrp811qftfY++73Xd+2zjiICMzOzHd5V6wLMzOzA4mAwM7OEg8HMzBIOBjMzSzgYzMws4WAwM7OEg8FKkvQLSVNrXceBRtJiSWN3MX++pPP3Y0k1IellSeMrWH6zpL+qck0D8vW2q+Z62yIHwwGm0l+4aomIiRFxS63rONBExNCImA8gaZakH9W4pBYpIrpExIuVrKPp70pE/CFf7zuVV9i2ORjaIEl1ta6hUq1hH/alA/X4HKh1WcrB0IJI+htJT0raIOkRSSOK5s2Q9IKkTZKekXR60bxpkn4j6VpJ64FZ+bSHJV0t6XVJL0maWLTMzi6R3Wg7UNKD+bbvl3T9rj5JS5qU78cbec0T8unJJ8DiT+SS6iWFpPMk/QH4Vd7ddXGTdT8l6Yx8eJCk+yS9Juk5SWeXqedkSb8vGr9P0oKi8YcknVZcY17zF4DJeffFU0WrPDI/3psk/VJSrzLbHSupUdI/S1oj6RVJ5xbNP1TSDyWtlbRc0pcklfydzY/VHEk/kvQGME3Su4peF+slzZbUo2iZz+TrXS/p34uPv6QfSLqiaa1ltj1a0qP56/IVSd+SdFDR/JB0kaSlwNKiaUdJ6pMfvx2PLZIib/MeSb/K61sn6TZJ3fJ5twIDgP/Ol/uXotdIXd6mj6S5+fO/TNIFTY7X7Pz4blLWRVgotX9tUkT4cQA9gJeB8SWmjwTWAGOAdsDUvO3B+fyzgD5kYT8ZeBM4Ip83DdgOfBaoAzrm094GLsjXdyGwClC+zHzg/KLld9X2UeBq4CDgg8AbwI/K7N9oYCPwkbzWvsCgUvsOzNqxHqAeCOCHQOd8Hz4D/Kao/RBgA3Bw3mYFcG6+zyOBdcCQEjV1BLYBvYD2wGpgJdA1n7cV6Nm0xuL6itY1H3gBeG++7HzgqjLHYmz+vFyeb/djwBagez7/h8DdeR31wPPAeWXWNSt/jk7Lj2tH4FLgMaBffkz+E7i96Fhtzp+vg/Ln7+2iffsBcEWTWhtLvU6B44Dj8+NcDywBLitqG8B9QA+gY9G0o0rsx21FNR5F9jo5GDgMeBC4rtzvStFrpC4ffxC4AegAHAusBT5cdLy25ce8HfA14LFa//4fKI+aF+BHkyekfDDcCHy1ybTngJPKrOdJYFI+PA34Q5P504BlReOd8l+qw/Px+aTBULIt2ae27UCnovk/onww/Cdw7e7sO6WD4a+K5nclC8Aj8/ErgZvz4cnAQyW2PbPMth8Czsjf4H4JzAYmACcDT5eqkfLB8KWi8X8E/rfMNseShU5d0bQ1eQ3tgLcoCjLg74H5ZdY1C3iwybQlwLii8SPI3vzrgC+TvwEXPadvsRfBUKKWy4C7isaD/A25ybSjmkz7V2AheXiUWO9pwO928XrZ8RqpA/oD7wBdi+Z/DfhB0fG6v2jeEGBrJb+7renh/r6W40hgqqTPFk07iOwsAUmfAT5H9ssB0IXsE/AOK0qs89UdAxGxRdKO5Uop17YX8FpEbGmyrf5l1tMfuKfMvN2xcz8iYpOknwNTgK8D55Cd1UB2vMZI2lC0bB1wa5n1/pr8zS8ffh04CfhjPr4nXi0a3kL5YwqwPiK2l2i/4+xledG85WRnWOU0fY6PBO6S9Keiae8AvcleN8XHcouybsY9Jum9wDVAgSxg6sje4HdVW9N1TCQ7wxkTEVvzab2BbwAnkn0IeBfZ87I7+pC9LjcVTVue17hD0+epg6S6Js9Hm+RrDC3HCuDKiOhW9OgUEbdLOhL4DnAxWZdHN2ARoKLl99VtdF8BekjqVDStXChAth/vKTPvTbI3lh0OL9Gm6X7cDpwj6QNkXQbzirbz6ybHq0tEXFhm2zuC4UP58K/JguEkygfDvrw18TqyT/dHFk0bQNbFVU7TelYAE5scgw4RsZLseeu3o6GkjkDPomV357nY4UbgWeDoiDiE7NqLmrQpe6wkHQPcApwdEcUB8n/z5Ybn6/00u/+aXkX2uuxaNK2542c5B8OBqb2kDkWPOrI3/n+QNEaZzpI+nr/wO5P9kqwFyC9gDtsfhUbEcqCB7IL2Qfkb9Km7WOR7wLmSxuUXR/tKGpTPexKYIql9fiHwzN0o4R6yN8/LgTsjYsen4/8B3ivp7/L1tZf0fkmDy6znEeAYsmsgv42Ixfl6x5D1VZeyGqgvd0G4EpF95XI2cKWkrnn4f46sm253fTtf/kgASYdJmpTPmwOcKumv8wvFs0jfdJ8EPiaph6TDybqHyulKdl1pc/5clgvfvyDpELLrKF+MiIdLrHczsFFSX+D/NJm/Gij5txB5wDwCfC3/HRoBnMeeHb82y8FwYLqHrO95x2NWRDSQdZN8i+x0ehlZ3z8R8QzwH2QXgVcDw4Hf7Md6PwV8AFgPXAHcSdYF8xci4rdkF4SvJbsI/Wv+/Kn438nOJl4HvgL8uLkNR8QfgZ8C44vb510Ip5B1M60i6zb4OtmFzFLreRN4AlgcEW/lkx8FlkfEmjKb/6/853pJTzRX6174LNkn9xeBh8n27+Y9WP4bwFzgl5I2kV2IHgOQB99ngTvIzh42k13f2PG83Qo8RdaP/0uy57SczwN/C2wi+wCzq7ZNjSIL5GuLv52Uz/tKPn8j8HOy57nY14Av5d+G+nyJdZ9D1rW6CriL7PrS/XtQW5u141slZlUj6U7g2YiYWetabPdI6kL2ja6jI+KlWtdjteUzBqtY3kXznrxraAIwCfhZreuyXZN0qqROkjqTfV3192RnCNbGORisGg4n+5rmZuCbwIUR8buaVmS7YxJZN8sq4GhgSrgLwXBXkpmZNeEzBjMzS7TIP3Dr1atX1NfX17oMM7MWZeHChesi4rDm2rXIYKivr6ehoaHWZZiZtSiSljffyl1JZmbWhIPBzMwSDgYzM0u0yGsMZtZ6vP322zQ2NrJt27Zal9JqdOjQgX79+tG+ffu9Wt7BYGY11djYSNeuXamvrye/nbtVICJYv349jY2NDBw4cK/W4a4kM6upbdu20bNnT4dClUiiZ8+eFZ2BORjMrOYcCtVV6fF0MJiZWcLBYGZt2oYNG7jhhhv2eLmPfexjbNiwofmGLZCDwczatHLBsH37rv/18z333EO3bt32VVk15W8lmVmbNmPGDF544QWOPfZY2rdvT4cOHejevTvPPvsszz//PKeddhorVqxg27ZtXHrppUyfPh348615Nm/ezMSJE/ngBz/II488Qt++fbn77rvp2LFjjfds7zkYzOyA8ZX/Xswzq96o6jqH9DmEmacOLTv/qquuYtGiRTz55JPMnz+fj3/84yxatGjnVz1vvvlmevTowdatW3n/+9/PJz/5SXr27JmsY+nSpdx+++185zvf4eyzz+YnP/kJn/70p6u6H/uTg8HMrMjo0aOT7/9/85vf5K677gJgxYoVLF269C+CYeDAgRx77LEAHHfccbz88sv7rd59wcFgZgeMXX2y3186d+68c3j+/Pncf//9PProo3Tq1ImxY8eW/PuAgw8+eOdwu3bt2Lp1636pdV/xxWcza9O6du3Kpk2bSs7buHEj3bt3p1OnTjz77LM89thj+7m62vAZg5m1aT179uSEE05g2LBhdOzYkd69e++cN2HCBL797W8zePBgjjnmGI4//vgaVrr/tMj/+VwoFML/qMesdViyZAmDBw+udRmtTqnjKmlhRBSaW9ZdSWZmlnAwmJlZwsFgZmYJB4OZmSUcDGZmlnAwmJlZwsFgZrYHunTpAsCqVas488wzS7YZO3YszX2l/rrrrmPLli07xw+k23g7GMzM9kKfPn2YM2fOXi/fNBgOpNt4VyUYJE2Q9JykZZJmlJh/sKQ78/mPS6pvMn+ApM2SPl+NeszMdteMGTO4/vrrd47PmjWLK664gnHjxjFq1CiGDx/O3Xff/RfLvfzyywwbNgyArVu3MmXKFAYPHszpp5+e3CvpwgsvpFAoMHToUGbOnAlkN+ZbtWoVJ598MieffDKQ3cZ73bp1AFxzzTUMGzaMYcOGcd111+3c3uDBg7ngggsYOnQop5xyyj67J1PFt8SQ1A64HvgI0AgskDQ3Ip4panYe8HpEHCVpCvB1YHLR/GuAX1Rai5m1cL+YAa/+vrrrPHw4TLyq7OzJkydz2WWXcdFFFwEwe/Zs7r33Xi655BIOOeQQ1q1bx/HHH88nPvGJsv9L+cYbb6RTp04sWbKEp59+mlGjRu2cd+WVV9KjRw/eeecdxo0bx9NPP80ll1zCNddcw7x58+jVq1eyroULF/L973+fxx9/nIhgzJgxnHTSSXTv3n2/3d67GmcMo4FlEfFiRLwF3AFMatJmEnBLPjwHGKf8CEs6DXgJWFyFWszM9sjIkSNZs2YNq1at4qmnnqJ79+4cfvjhfOELX2DEiBGMHz+elStXsnr16rLrePDBB3e+QY8YMYIRI0bsnDd79mxGjRrFyJEjWbx4Mc8880y51QDw8MMPc/rpp9O5c2e6dOnCGWecwUMPPQTsv9t7V+Mmen2BFUXjjcCYcm0iYrukjUBPSduAfyU729hlN5Kk6cB0gAEDBlShbDM74Ozik/2+dNZZZzFnzhxeffVVJk+ezG233cbatWtZuHAh7du3p76+vuTttpvz0ksvcfXVV7NgwQK6d+/OtGnT9mo9O+yv23vX+uLzLODaiNjcXMOIuCkiChFROOyww/Z9ZWbWZkyePJk77riDOXPmcNZZZ7Fx40be/e530759e+bNm8fy5ct3ufyHPvQhfvzjHwOwaNEinn76aQDeeOMNOnfuzKGHHsrq1av5xS/+3GNe7nbfJ554Ij/72c/YsmULb775JnfddRcnnnhiFfe2edU4Y1gJ9C8a75dPK9WmUVIdcCiwnuzM4kxJ/w/oBvxJ0raI+FYV6jIz2y1Dhw5l06ZN9O3blyOOOIJPfepTnHrqqQwfPpxCocCgQYN2ufyFF17Iueeey+DBgxk8eDDHHXccAO973/sYOXIkgwYNon///pxwwgk7l5k+fToTJkygT58+zJs3b+f0UaNGMW3aNEaPHg3A+eefz8iRI/frf4Wr+Lbb+Rv988A4sgBYAPxtRCwuanMRMDwi/iG/+HxGRJzdZD2zgM0RcXVz2/Rtt81aD992e9+o5LbbFZ8x5NcMLgbuBdoBN0fEYkmXAw0RMRf4HnCrpGXAa8CUSrdrZmb7RlX+g1tE3APc02Tal4uGtwFnNbOOWdWoxczMKlPri89mZrTE/yR5IKv0eDoYzKymOnTowPr16x0OVRIRrF+/ng4dOuz1OqrSlWRmtrf69etHY2Mja9eurXUprUaHDh3o16/fXi/vYDCzmmrfvj0DBw6sdRlWxF1JZmaWcDCYmVnCwWBmZgkHg5mZJRwMZmaWcDCYmVnCwWBmZgkHg5mZJRwMZmaWcDCYmVnCwWBmZgkHg5mZJRwMZmaWcDCYmVnCwWBmZgkHg5mZJRwMZmaWcDCYmVnCwWBmZgkHg5mZJRwMZmaWcDCYmVnCwWBmZgkHg5mZJRwMZmaWcDCYmVmiKsEgaYKk5yQtkzSjxPyDJd2Zz39cUn0+/SOSFkr6ff7zw9Wox8zM9l7FwSCpHXA9MBEYApwjaUiTZucBr0fEUcC1wNfz6euAUyNiODAVuLXSeszMrDLVOGMYDSyLiBcj4i3gDmBSkzaTgFvy4TnAOEmKiN9FxKp8+mKgo6SDq1CTmZntpWoEQ19gRdF4Yz6tZJuI2A5sBHo2afNJ4ImI+GMVajIzs71UV+sCACQNJeteOmUXbaYD0wEGDBiwnyozM2t7qnHGsBLoXzTeL59Wso2kOuBQYH0+3g+4C/hMRLxQbiMRcVNEFCKicNhhh1WhbDMzK6UawbAAOFrSQEkHAVOAuU3azCW7uAxwJvCriAhJ3YCfAzMi4jdVqMXMzCpUcTDk1wwuBu4FlgCzI2KxpMslfSJv9j2gp6RlwOeAHV9pvRg4CviypCfzx7srrcnMzPaeIqLWNeyxQqEQDQ0NtS7DzKxFkbQwIgrNtfNfPpuZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklqhIMkiZIek7SMkkzSsw/WNKd+fzHJdUXzfu3fPpzkj5ajXrMzGzvVRwMktoB1wMTgSHAOZKGNGl2HvB6RBwFXAt8PV92CDAFGApMAG7I12dmZjVSV4V1jAaWRcSLAJLuACYBzxS1mQTMyofnAN+SpHz6HRHxR+AlScvy9T1ahbr+wmM3XEDXDUv2xarNzPa5Td0Gc/w/fmefb6caXUl9gRVF4435tJJtImI7sBHouZvLAiBpuqQGSQ1r166tQtlmZlZKNc4Y9ouIuAm4CaBQKMTerGN/JK2ZWUtXjTOGlUD/ovF++bSSbSTVAYcC63dzWTMz24+qEQwLgKMlDZR0ENnF5LlN2swFpubDZwK/iojIp0/Jv7U0EDga+G0VajIzs71UcVdSRGyXdDFwL9AOuDkiFku6HGiIiLnA94Bb84vLr5GFB3m72WQXqrcDF0XEO5XWZGZme0/ZB/eWpVAoRENDQ63LMDNrUSQtjIhCc+38l89mZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWcLBYGZmCQeDmZklHAxmZpZwMJiZWaKiYJDUQ9J9kpbmP7uXaTc1b7NU0tR8WidJP5f0rKTFkq6qpBYzM6uOSs8YZgAPRMTRwAP5eEJSD2AmMAYYDcwsCpCrI2IQMBI4QdLECusxM7MKVRoMk4Bb8uFbgNNKtPkocF9EvBYRrwP3ARMiYktEzAOIiLeAJ4B+FdZjZmYVqjQYekfEK/nwq0DvEm36AiuKxhvzaTtJ6gacSnbWYWZmNVTXXANJ9wOHl5j1xeKRiAhJsacFSKoDbge+GREv7qLddGA6wIABA/Z0M2ZmtpuaDYaIGF9unqTVko6IiFckHQGsKdFsJTC2aLwfML9o/CZgaURc10wdN+VtKRQKexxAZma2eyrtSpoLTM2HpwJ3l2hzL3CKpO75RedT8mlIugI4FLiswjrMzKxKKg2Gq4CPSFoKjM/HkVSQ9F2AiHgN+CqwIH9cHhGvSepH1h01BHhC0pOSzq+wHjMzq5AiWl6vTKFQiIaGhlqXYWbWokhaGBGF5tr5L5/NzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEhUFg6Qeku6TtDT/2b1Mu6l5m6WSppaYP1fSokpqMTOz6qj0jGEG8EBEHA08kI8nJPUAZgJjgNHAzOIAkXQGsLnCOszMrEoqDYZJwC358C3AaSXafBS4LyJei4jXgfuACQCSugCfA66osA4zM6uSSoOhd0S8kg+/CvQu0aYvsKJovDGfBvBV4D+ALc1tSNJ0SQ2SGtauXVtByWZmtit1zTWQdD9weIlZXyweiYiQFLu7YUnHAu+JiH+SVN9c+4i4CbgJoFAo7PZ2zMxszzQbDBExvtw8SaslHRERr0g6AlhTotlKYGzReD9gPvABoCDp5byOd0uaHxFjMTOzmqm0K2kusONbRlOBu0u0uRc4RVL3/KLzKcC9EXFjRPSJiHrgg8DzDgUzs9qrNBiuAj4iaSkwPh9HUkHSdwEi4jWyawkL8sfl+TQzMzsAKaLlddcXCoVoaGiodRlmZi2KpIURUWiunf/y2czMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4SDwczMEg4GMzNLOBjMzCzhYDAzs4QiotY17DFJa4Hle7l4L2BdFcs5kLTmfYPWvX/et5arJe3fkRFxWHONWmQwVEJSQ0QUal3HvtCa9w1a9/5531qu1rh/7koyM7OEg8HMzBJtMRhuqnUB+1Br3jdo3fvnfWu5Wt3+tblrDGZmtmtt8YzBzMx2wcFgZmaJNhMMkiZIek7SMkkzal1PNUnqL2mepGckLZZ0aa1rqjZJ7ST9TtL/1LqWapLUTdIcSc9KWiLpA7WuqZok/VP+mlwk6XZJHWpd096SdLOkNZIWFU3rIek+SUvzn91rWWO1tIlgkNQOuB6YCAwBzpE0pLZVVdV24J8jYghwPHBRK9s/gEuBJbUuYh/4BvC/ETEIeB+taB8l9QUuAQoRMQxoB0ypbVUV+QEwocm0GcADEXE08EA+3uK1iWAARgPLIuLFiHgLuAOYVOOaqiYiXomIJ/LhTWRvLn1rW1X1SOoHfBz4bq1rqSZJhwIfAr4HEBFvRcSG2lZVdXVAR0l1QCdgVY3r2WsR8SDwWpPJk4Bb8uFbgNP2a1H7SFsJhr7AiqLxRlrRG2cxSfXASODx2lZSVdcB/wL8qdaFVNlAYC3w/byb7LuSOte6qGqJiJXA1cAfgFeAjRHxy9pWVXW9I+KVfPhVoHcti6mWthIMbYKkLsBPgMsi4o1a11MNkv4GWBMRC2tdyz5QB4wCboyIkcCbtJKuCIC8v30SWQD2ATpL+nRtq9p3Ivvuf6v4/n9bCYaVQP+i8X75tFZDUnuyULgtIn5a63qq6ATgE5JeJusC/LCkH9W2pKppBBojYsfZ3RyyoGgtxgMvRcTaiHgb+Cnw1zWuqdpWSzoCIP+5psb1VEVbCYYFwNGSBko6iOwC2Nwa11Q1kkTWT70kIq6pdT3VFBH/FhH9IqKe7Hn7VUS0ik+dEfEqsELSMfmkccAzNSyp2v4AHC+pU/4aHUcruriemwtMzYenAnfXsJaqqat1AftDRGyXdDFwL9k3I26OiMU1LquaTgD+Dvi9pCfzaV+IiHtqWJPtns8Ct+UfWF4Ezq1xPVUTEY9LmgM8QfbNud/Rgm8fIel2YCzQS1IjMBO4Cpgt6TyyfwVwdu0qrB7fEsPMzBJtpSvJzMx2k4PBzMwSDgYzM0s4GMzMLOFgMDOzhIPBzMwSDgYzM0v8f37trl81ujZiAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "error_train, error_val = learning_curve(X_poly, y, X_val_poly, y, _lambda)\n", "plt.plot(error_train, label='train')\n", "plt.plot(error_val, label='validation')\n", "plt.title('Learning curve with no regularization')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Adjusting the regularization parameter" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "def plot_regularization(_lambda):\n", " \"\"\" Plot the fit and learning curve for a regularization parameter _lambda \"\"\"\n", " \n", " # 1. Fit\n", " \n", " # Find theta\n", " args = (X_poly, y, _lambda)\n", " initial_theta = np.ones(p+1)\n", " result = minimize(linear_reg_cost_function, initial_theta, args=args,\n", " method='CG', jac=compute_gradient,\n", " options={'maxiter': 1000, 'disp': False})\n", " theta = result.x\n", " \n", " # Plot training data\n", " plt.plot(X[:,1], y, 'rx', label='Training data')\n", "\n", " # Plot polynomial fit\n", " space = np.array(np.arange(np.min(X) - 5, np.max(X) + 5, 0.05)).reshape((1912, 1))\n", " X_poly_plt = polynomial_features(space, p)\n", " X_poly_plt = (X_poly_plt - mu) / sigma\n", " X_poly_plt = np.hstack((np.ones((X_poly_plt.shape[0], 1)), X_poly_plt))\n", " plt.plot(space, X_poly_plt @ theta, '--', label='Polynomial fit')\n", "\n", " # Plot\n", " plt.legend()\n", " plt.xlabel('Change in water leven (x)')\n", " plt.ylabel('Water flowing out of the dam (y)')\n", " plt.show()\n", " \n", " \n", " # 2. Learning curve\n", " \n", " # Plot learning curve\n", " error_train, error_val = learning_curve(X_poly, y, X_val_poly, y, _lambda)\n", " plt.plot(error_train, label='train')\n", " plt.plot(error_val, label='validation')\n", " plt.title('Learning curve with no regularization')\n", " plt.legend()\n", " \n", " plt.show()" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "ename": "ValueError", "evalue": "shapes (12,2) and (9,) not aligned: 2 (dim 1) != 9 (dim 0)", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;31m# Good regularization\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mplot_regularization\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m\u001b[0m in \u001b[0;36mplot_regularization\u001b[0;34m(_lambda)\u001b[0m\n\u001b[1;32m 9\u001b[0m result = minimize(linear_reg_cost_function, initial_theta, args=args,\n\u001b[1;32m 10\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'CG'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcompute_gradient\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m options={'maxiter': 1000, 'disp': False})\n\u001b[0m\u001b[1;32m 12\u001b[0m \u001b[0mtheta\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/optimize/_minimize.py\u001b[0m in \u001b[0;36mminimize\u001b[0;34m(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)\u001b[0m\n\u001b[1;32m 591\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_powell\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 592\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mmeth\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'cg'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 593\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_cg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 594\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mmeth\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'bfgs'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 595\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_bfgs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36m_minimize_cg\u001b[0;34m(fun, x0, args, jac, callback, gtol, norm, eps, maxiter, disp, return_all, **unknown_options)\u001b[0m\n\u001b[1;32m 1270\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1271\u001b[0m \u001b[0mgrad_calls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmyfprime\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwrap_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfprime\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1272\u001b[0;31m \u001b[0mgfk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmyfprime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1273\u001b[0m \u001b[0mk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1274\u001b[0m \u001b[0mxk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36mfunction_wrapper\u001b[0;34m(*wrapper_args)\u001b[0m\n\u001b[1;32m 298\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfunction_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mwrapper_args\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 299\u001b[0m \u001b[0mncalls\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 300\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrapper_args\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 301\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 302\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mncalls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunction_wrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m\u001b[0m in \u001b[0;36mcompute_gradient\u001b[0;34m(theta, X, y, _lambda)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcompute_gradient\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtheta\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_lambda\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mhx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m \u001b[0;34m@\u001b[0m \u001b[0mtheta\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0mcost\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m \u001b[0;34m@\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhx\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mregularization\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0m_lambda\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconcatenate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtheta\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mcost\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mregularization\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mValueError\u001b[0m: shapes (12,2) and (9,) not aligned: 2 (dim 1) != 9 (dim 0)" ] } ], "source": [ "# Good regularization\n", "plot_regularization(1)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "ename": "ValueError", "evalue": "shapes (12,2) and (9,) not aligned: 2 (dim 1) != 9 (dim 0)", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;31m# (Way) too much regularization\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mplot_regularization\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m\u001b[0m in \u001b[0;36mplot_regularization\u001b[0;34m(_lambda)\u001b[0m\n\u001b[1;32m 9\u001b[0m result = minimize(linear_reg_cost_function, initial_theta, args=args,\n\u001b[1;32m 10\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'CG'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcompute_gradient\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m options={'maxiter': 1000, 'disp': False})\n\u001b[0m\u001b[1;32m 12\u001b[0m \u001b[0mtheta\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/optimize/_minimize.py\u001b[0m in \u001b[0;36mminimize\u001b[0;34m(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)\u001b[0m\n\u001b[1;32m 591\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_powell\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 592\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mmeth\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'cg'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 593\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_cg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 594\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mmeth\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'bfgs'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 595\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0m_minimize_bfgs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjac\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36m_minimize_cg\u001b[0;34m(fun, x0, args, jac, callback, gtol, norm, eps, maxiter, disp, return_all, **unknown_options)\u001b[0m\n\u001b[1;32m 1270\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1271\u001b[0m \u001b[0mgrad_calls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmyfprime\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwrap_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfprime\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1272\u001b[0;31m \u001b[0mgfk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmyfprime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1273\u001b[0m \u001b[0mk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1274\u001b[0m \u001b[0mxk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mx0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scipy/optimize/optimize.py\u001b[0m in \u001b[0;36mfunction_wrapper\u001b[0;34m(*wrapper_args)\u001b[0m\n\u001b[1;32m 298\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfunction_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mwrapper_args\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 299\u001b[0m \u001b[0mncalls\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 300\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrapper_args\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 301\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 302\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mncalls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunction_wrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m\u001b[0m in \u001b[0;36mcompute_gradient\u001b[0;34m(theta, X, y, _lambda)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcompute_gradient\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtheta\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_lambda\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mhx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m \u001b[0;34m@\u001b[0m \u001b[0mtheta\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0mcost\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m \u001b[0;34m@\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhx\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mregularization\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0m_lambda\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconcatenate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtheta\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mcost\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mregularization\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mValueError\u001b[0m: shapes (12,2) and (9,) not aligned: 2 (dim 1) != 9 (dim 0)" ] } ], "source": [ "# (Way) too much regularization\n", "plot_regularization(100)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: ex6/Support Vector Machines (Exercises).ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Support Vector Machines\n", "\n", "Stanford CS229 - Machine Learning by Andrew Ng. Programming exercise 6.\n", "\n", "Please check out [the repository on GitHub](https://github.com/rickwierenga/CS229-Python/). If you spot any mistakes or inconcistencies, please create an issue. For questions you can find me on Twitter: [@rickwierenga](https://twitter.com/rickwierenga). Starring the project on GitHub means a ton to me!" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pylab as plt\n", "import scipy.io as sio\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should use a support vector machine when " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Support Vector Machines\n", "---\n", "In the first half of this exercise, you will be using support vector machines (SVMs) with various example 2D datasets. Experimenting with these datasets will help you gain an intuition of how SVMs work and how to use a Gaussian kernel with SVMs. In the next half of the exercise, you will be using support vector machines to build a spam classifier." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# Load data\n", "data = sio.loadmat(\"ex6data1.mat\")\n", "X = data[\"X\"]\n", "y = data[\"y\"].flatten()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualising the data" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAF4BJREFUeJzt3X+MXWWdx/H3p9AdJLiS0EltKNPZCaSJmhXpgExINi3GhEVS/pBVNsiK0dSgLhjddFXSutY0G/uHGpeNbANmQbtKg6zbbTCGhGnUpCItFgRqN7XbIoYdfqhggzub2u/+cc/QmTt35t65P85zfnxeyU3vvefp3O+c9n7Pc56figjMzKxalqUOwMzM+s/J3cysgpzczcwqyMndzKyCnNzNzCrIyd3MrIKc3M3MKsjJ3cysgpzczcwq6OxUH7xixYoYHR1N9fFmZqV08ODBlyJiuF25ZMl9dHSUAwcOpPp4M7NSknSik3JuljEzqyAndzOzCnJyNzOrICd3M7MKcnI3M6sgJ3ezgtuxAyYn5743Odl432whTu5mBXf55fC+951J8JOTjdeXX542Lis2J3erlTLWgjdsgN27Gwl969bGn7t3N943W4iTu9VKWWvBGzbArbfCF7/Y+NOJ3dpxcrdaKWsteHISvv512LKl8Wfz3ceglfGOp+6c3K12ylYLnrm72L0btm07c3HKM8GX9Y6nzpzcrXZS14KX6rHH5t5dzNx9PPZYfjGU9Y6n1iKiowdwFvAzYG+LY7cALwKHssdH2v28devWhVneHnkkYsWKxp+tXtvitmyJgMaflgZwIDrI2Uupud8OHF7k+P0RcWn2uLuL64zZwBWhFlxWZbvjqbuOlvyVtBp4D7Ad+NRAIzIboM2b57+3YYObF9qZ3e4/c77cNFNsndbcvwpsBk4vUua9kp6U9ICki3oPzcyKwnc85aNGE84iBaTrgGsj4mOS1gN/FxHXNZW5ADgZEdOSPgq8PyKubvGzNgGbAEZGRtadONHRmvNmZpaRdDAixtuV66TmfhWwUdJx4DvA1ZK+NbtARLwcEdPZy7uBda1+UETsjIjxiBgfHm67S5SZmXWpbXKPiM9GxOqIGAVuBB6JiA/MLiNp1ayXG1m849XMzAas6z1UJW2jMSRnD3CbpI3AKeA3NIZGmplZIm3b3AdlfHw8vEG2mdnS9LPN3cw64PVXrEic3M36xOuvWJF03eZuZnPNXn/l1lsbszg9ycdScc3drI/KtuKkVZeTu1kfef0VKwond7M+KcK662YznNzN+sTrryzOo4ny5eRu1iebN89vY9+wofVKlHXk0UT58mgZM8uFRxPlyzV3M8uNRxPlx8ndzHLj0UT5cXI3s1x4NFG+nNzNLBceTZQvrwppZtYHO3Y0Rv7M7keYnGxcvPo5YsqrQpqZ5ahoQz09FNLMrA+KNtTTNXczsz4p0lBPJ3czsz4p0lDP0iR3r0thZkVWtKGepUnuReusMDObrWhDPUs1FHImoRehs8LMLIVKDoUsUmeFlYOb86yuSpXci9RZYeXg5rwzfKGrl9Ik96J1Vlg5zB57vHXrmf9Ddbzr84WuXjpO7pLOkvQzSXtbHBuSdL+ko5IelTTazyCheJ0VVh5uzmvwha5ellJzvx04vMCxDwO/jYiLga8AX+o1sGbe5ca65ea8M3yhq4+Okruk1cB7gLsXKHI9cG/2/AHgXZLUe3hmvXFz3ly+0NVHpzX3rwKbgdMLHL8Q+BVARJwCXgEu6Dk6sx65Oe8MX+jqpe3CYZKuA16IiIOS1vfyYZI2AZsARkZGevlRZh1p1Wy3YUM9myMWu9DV8XxUXdtJTJL+EbgZOAWcA/wp8GBEfGBWmR8A/xAR+yWdDfwPMByL/HCv525mtnR9m8QUEZ+NiNURMQrcCDwyO7Fn9gAfzJ7fkJVJM/XVzMy6H+cuaZukjdnLe4ALJB0FPgV8ph/BmZl58lV3lpTcI2JfRFyXPd8aEXuy5/8bEX8VERdHxBURcWwQwZpZ/XjyVXdKM0PVzOppUJOvqn5H4ORu1kdVTxipDGLyVdXvCJzczfqo6gkjlUFMvqr6cgxO7mZ9VPWEkcIgJ19VeTkGJ3ezPqtywkhhkLOMq7wcQ6l2YjIrA+8YVg6z7wg2bJj/uqgquROTWdF5/ZbyqPq6Q665m/XRjh2NztPZNb/JyUbC8PLU1g+d1tyd3M3MSsTNMmZmNebkbmZWQU7uZmYV5ORuZlZBTu5mZhXk5G5mVkFO7mZmFeTkbmZWQU7uZmYV5ORuZlZBTu5mZhXk5G5mVkFO7mbWkveDLTcndzNryfvBltvZqQMws2KavR+sd5Uqn7Y1d0nnSPqppCckPS3pCy3K3CLpRUmHssdHBhOupTQ1tYv9+0fZt28Z+/ePMjW1K3VINmDeD7a8OmmWmQaujoi3A5cC10i6skW5+yPi0uxxd1+jtOSmpnZx5MgmpqdPAMH09AmOHNnkBF8Ag2wbr/IG0lXXNrlHw8ns5fLskWb7Jkvm2LE7OH36tTnvnT79GseO3ZEoIpsxqLZx7wdbbh11qEo6S9Ih4AXg4Yh4tEWx90p6UtIDki7qa5SW3PT0s0t63/Izu21869YzCbnXJpSqbyBddUvaQ1XS+cC/A38bEU/Nev8C4GRETEv6KPD+iLi6xd/fBGwCGBkZWXfixIle47ec7N8/mjXJzDU0tIaJieP5B2TzbN3aaBvfsqVR07ZqGsgeqhHxO2ASuKbp/ZcjYjp7eTewboG/vzMixiNifHh4eCkfbYmNjW1n2bJz57y3bNm5jI1tTxSRzea2cWvWyWiZ4azGjqQ3AO8GftFUZtWslxuBw/0M0tJbufIm1q7dydDQGkAMDa1h7dqdrFx5U+rQaq8ObeNlmFBVtBg7qbmvAiYlPQk8RqPNfa+kbZI2ZmVuy4ZJPgHcBtwymHAtpZUrb2Ji4jjr159mYuK4E3tB1KFtvAwTqooW45La3PtpfHw8Dhw4kOSzq2RqahfHjt3B9PSzDA2NMDa23UnXKmkmWRZ5QlUeMQ6kzd2KxWPPrU7KMKGqSDE6uZdY2ceee8arLUUZOo2LFKPXlimxMo89n7nrmLk4zdx1AG5Wsnlmdxpv2NB49Gs8f78ULUbX3EtsaGhkSe8XSdnvOixfZeg0LlqM7lAtsebaLzTGnpdhiOK+fctovYqFWL/+dN7hWBd27GiMBJldK52cbCSzzZvTxVV17lCtgTKPPS/zXYc1FG3on83lNveSW7nyplIk82ZjY9tb3nV4xmt5eL33YnPN3ZIo812HnVGkoX82l2vulkxZ7zrsjOahfzOjRCw919zNrCt1WNOmzJzczawrRRv6Z3N5KKTVmtfmsbLxUEjrSR2WBvDaPJanvJcEdnK3eeqS9DxL1vKU97wAJ3ebpy5Jr8xr81j5DGqv24U4uds8dUl6niVrectzXoCTu81Tl6TnfWEtb3kuCezkbvPUJel5lmy9pN7jNO95AZ6havPMJLc6DBH0LNn6mOnQnGnnnp1s87DYvIBBNM94nLuZ1UYZ9mFtx+PcrdDqMI6+jlI3fbRTp4XOnNwtd3UZR19HRV/jvUh7nA6ak7vlri7j6Oso77HcS1G3hc6c3EukKk0ZdRlHX1dFbfqo20JnbUfLSDoH+CEwlJV/ICI+31RmCLgPWAe8DLw/Io73Pdoaa94vdaYpAyjdaI+hoZGsSWb++1Z+RV3jvdW+rkWJbRA6qblPA1dHxNuBS4FrJF3ZVObDwG8j4mLgK8CX+htmfSxUO69SU0ZdxtHXUd2aPoqsbXKPhpPZy+XZo3n85PXAvdnzB4B3SVLfoqyJxToaq9SU4clD1VW3po8i62icu6SzgIPAxcA/R8TfNx1/CrgmIp7LXv8SeGdEvLTQz/Q49/n27x9doLliDcCCxyYmjg86NDMriL6Oc4+IP0bEpcBq4ApJb+syqE2SDkg68OKLL3bzIyptsdq5mzLKqSqd4FY+SxotExG/AyaBa5oO/Rq4CEDS2cCbaHSsNv/9nRExHhHjw8PD3UVcYYst2OWmjPLxeH5LqW1ylzQs6fzs+RuAdwO/aCq2B/hg9vwG4JFIta5BibWrna9ceRMTE8dZv/40ExPHe07sdahVpvwdq9QJbuXTycJhq4B7s3b3ZcDuiNgraRtwICL2APcA35R0FPgNcOPAIq6wPBfsqtLQyoWk/h2r1Alu5eOFw2pqsc7bqnTQpv4d+/n53sjbZnjhMFtUHWqVqX/HfnWCu+3euuHkXlN12G0p9e/Yr05wt933pugrVQ6Kk3tN1WFoZRF+x350gqe+Aym7oq9UOShO7jVVh6GVVfkdU9+BlF2RV6ocJHeomhVc86gfaNyBlPFCldLWrY2VKrdsaax7U1buUDWriKrcgaRUp006ZniDbKuVsg4p9Ebe3Zu9UuXMEr91aJpxzd1qw0MK66muK1W6zd1qI/WkJrN+cJu7WRMPKbQ6cXK32vCQQqsTJ3erjbwmNdVhtU0rPid3y12q5JfHkEJ32lpRuEO1xMo4rK/qE3LcaWuD5g7ViitrDbHqi2C509aKwsm9pMqaJKue/LrttHU7vfWbk3tJlTVJVn3ESjedtinvwnxRqS4n95Iqa5IswjK8g9RNp22qu7CyNu1ZZ7y2TEmNjW1v2TFZ9CSZ5z6xqSx1HZhUd2GLXVSq9O9RV07uiXU74qXMSdKLYM01NDSywAibwd6FlbVpzzrj5J5Q87DAmdtioOME7yRZfqnuwlJdVCwfbnNPqKwjXqy/Uq3XXvX+j7pzzT0h3xbbjBR3YWVu2rP2nNwT8m2xpeamvepq2ywj6SJJk5KekfS0pNtblFkv6RVJh7LH1sGEWy2+LTazQemk5n4K+HREPC7pjcBBSQ9HxDNN5X4UEdf1P8Tq8m2xmQ1K2+QeEc8Dz2fPfy/pMHAh0JzcrQu+LTazQVjSaBlJo8A7gEdbHJ6Q9ISk70t6ax9iMzOzLnXcoSrpPOC7wCcj4tWmw48DayLipKRrge8Bl7T4GZuATQAjI+40NDMblI5q7pKW00jsuyLiwebjEfFqRJzMnj8ELJe0okW5nRExHhHjw8PDPYZuZmYL6WS0jIB7gMMR8eUFyrw5K4ekK7Kf+3I/AzUzs8510ixzFXAz8HNJh7L3PgeMAETEXcANwK2STgF/AG6MVFs8WWWUcacps6LoZLTMjwG1KXMncGe/gjLrdd2dKvDFzXrhtWWskOq+7o7XWrdeOblbIeW17k5RdyKq+8XNeufkboWUx05TRa4de1E565WTu81RlJpsHuvuFLl2XNZtFK04nNztdUWqyeaxxnmRa8deVM565SV/7XVF21Nz0OvuFHnJ5TItKudRPcXk5G6vK3JNdhCKvsl4GRaV85DV4nKzjL2ubu28qba3q5Ii91vUnWvu9rqi12QHoQy14yKr291embjmbq9zTdaWqm53e2XimrvN4ZqsLUUd7/bKwjV3sx4VZW5ACr7bKy7X3M164NEivtsrKtfczXrg0SJWVE7uZj3waBErKid3sx54tIgVlZO7WQ+8BowVlZO7WQ88WsSKyqNlzHrk0SJWRK65m5lVkJO7mVkFOblbJdV51qgZOLlbBRVpR6lu+MJk/eDkbgOVIlGVedZo2S9MVhxtk7ukiyRNSnpG0tOSbm9RRpK+JumopCclXTaYcK1MUiWqMs8aLfOFyYqlk5r7KeDTEfEW4Erg45Le0lTmL4FLsscm4Ot9jdJKKVWiKvOs0TJfmKxY2ib3iHg+Ih7Pnv8eOAxc2FTseuC+aPgJcL6kVX2P1kolVaIq86zRMl+YrFiW1OYuaRR4B/Bo06ELgV/Nev0c8y8AVjOpElWZZ43mcWFyh209dDxDVdJ5wHeBT0bEq918mKRNNJptGBlxTaTqUu7SU9ZZozMxHzt2B9PTzzI0NMLY2Pa+/S5ef74+FBHtC0nLgb3ADyLiyy2O/wuwLyK+nb0+AqyPiOcX+pnj4+Nx4MCBrgO3cpia2jWwRGVLt3//aNbBPdfQ0BomJo7nH5AtmaSDETHerlzbmrskAfcAh1sl9swe4BOSvgO8E3hlscRu9VHWGnRVucO2PjpplrkKuBn4uaRD2XufA0YAIuIu4CHgWuAo8Brwof6Hama9GhoaWaDm7mbSqmmb3CPix4DalAng4/0KyswGI2U/iOXLM1TNaqTMI4lsabyeu1nNuB+kHlxzNzOrICd3M7MKcnI3M6sgJ3czswpycjczqyAndzOzCnJyNzOrICd3M7MKcnI3M6sgJ3czswpycjczqyAndzOzCnJyNzOrICd3M7MKcnI3M6sgJ3ezBUxN7WL//lH27VvG/v2jTE3tSh2SWce8WYdZC1NTu+ZsRzc9fYIjRzYBeKMLKwXX3M1aOHbsjjn7jAKcPv0ax47dkSgis6VxcjdrYXr62SW9b1Y0Tu5mLQwNjSzpfbOicXI3a2FsbDvLlp07571ly85lbGx7oojMlsbJ3ayFlStvYu3anQwNrQHE0NAa1q7d6c5UK422o2UkfQO4DnghIt7W4vh64D+A/87eejAitvUzSLMUVq68ycncSquToZD/CtwJ3LdImR9FxHV9icjMzHrWtlkmIn4I/CaHWMzMrE/61eY+IekJSd+X9NY+/UwzM+tSP2aoPg6siYiTkq4Fvgdc0qqgpE3AJoCREQ8pMzMblJ5r7hHxakSczJ4/BCyXtGKBsjsjYjwixoeHh3v9aDMzW0DPNXdJbwamIiIkXUHjgvFyu7938ODBlySdaFNsBfBSrzEOQFHjAsfWraLGVtS4wLF1q9fY1nRSqJOhkN8G1gMrJD0HfB5YDhARdwE3ALdKOgX8AbgxIqLdz42ItlV3SQciYrxdubwVNS5wbN0qamxFjQscW7fyiq1tco+Iv25z/E4aQyXNzKwgPEPVzKyCip7cd6YOYAFFjQscW7eKGltR4wLH1q1cYlMHzeNmZlYyRa+5m5lZF5Ind0nXSDoi6aikz7Q4PiTp/uz4o5JGCxTbLZJelHQoe3wkp7i+IekFSU8tcFySvpbF/aSky/KIq8PY1kt6ZdY525pTXBdJmpT0jKSnJd3eokyS89ZhbKnO2zmSfprNQH9a0hdalEnyHe0wtiTf0eyzz5L0M0l7Wxwb/DmLiGQP4Czgl8AY8CfAE8Bbmsp8DLgre34jcH+BYrsFuDPBefsL4DLgqQWOXwt8HxBwJfBogWJbD+xNcM5WAZdlz98I/FeLf88k563D2FKdNwHnZc+XA48CVzaVSfUd7SS2JN/R7LM/Bfxbq3+3PM5Z6pr7FcDRiDgWEf8HfAe4vqnM9cC92fMHgHdJUkFiSyLaL+Z2PXBfNPwEOF/SqoLElkREPB8Rj2fPfw8cBi5sKpbkvHUYWxLZuTiZvVyePZo76pJ8RzuMLQlJq4H3AHcvUGTg5yx1cr8Q+NWs188x/z/162Ui4hTwCnBBQWIDeG92C/+ApItyiKsTncaeStKF5rJb4HfQqOnNlvy8LRIbJDpvWfPCIeAF4OGIWPC85fwd7SQ2SPMd/SqwGTi9wPGBn7PUyb3s/hMYjYg/Bx7mzJXYFjaz0NzbgX+isdBcbiSdB3wX+GREvJrnZ7fTJrZk5y0i/hgRlwKrgSskzdu0J5UOYsv9OyppZnOjg4P+rMWkTu6/BmZfSVdn77UsI+ls4E10sHZNHrFFxMsRMZ29vBtYl0NcnejkvCYRS1hort8kLaeRPHdFxIMtiiQ7b+1iS3neZsXwO2ASuKbpUKrvaNvYEn1HrwI2SjpOozn3aknfaioz8HOWOrk/Blwi6c8k/QmNjoU9TWX2AB/Mnt8APBJZL0Tq2JraYzfSaCstgj3A32SjP64EXomI51MHBY2F5mbaFrWEheb68LkC7gEOR8SXFyiW5Lx1ElvC8zYs6fzs+RuAdwO/aCqW5DvaSWwpvqMR8dmIWB0RozTyxiMR8YGmYgM/Z/1Yz71rEXFK0ieAH9AYnfKNiHha0jbgQETsofGf/puSjtLoqLuxQLHdJmkjcCqL7ZY8YlP7xdweojHy4yjwGvChPOLqMLauFprrg6uAm4GfZ220AJ8DRmbFluq8dRJbqvO2CrhX0lk0Lii7I2JvEb6jHcaW5DvaSt7nzDNUzcwqKHWzjJmZDYCTu5lZBTm5m5lVkJO7mVkFObmbmVWQk7uZWQU5uZuZVZCTu5lZBf0/AlcGsNl4/QUAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "pos = np.where(y==1)[0]\n", "neg = np.where(y==0)[0]\n", "\n", "plt.plot(X[pos, 0], X[pos, 1], 'bx')\n", "plt.plot(X[neg, 0], X[neg, 1], 'yo')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training a linear SVM\n", "Remember: high variance = overfit, high bias = underfit.\n", "\n", "$C$ is the regularization parameter for SVMs where $\\lambda$ was the regularization parameter for logistic regression.\n", "\n", "$$C = \\frac{1}{\\lambda}$$\n", "\n", "Training an SVM in Python can be done using `SciKit Learn`. How this works internally, is beyond the scope of these notebooks." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from sklearn.svm import SVC" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def train_svm(X, y, **params):\n", " \"\"\" Train an SVM with data X, y and regularization parameter C \"\"\"\n", " \n", " model = SVC(**params)\n", " clf = model.fit(X, y)\n", " return clf" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def plot_svm_decision_boundary(clf):\n", " \"\"\" Plot the decision boundary for an SVM \"\"\"\n", " w = clf.coef_[0]\n", " a = -w[0] / w[1]\n", "\n", " xp = np.array(np.linspace(np.min(X[:, 0]), np.max(X[:, 0]), 100))\n", " yp = a * xp - (clf.intercept_[0]) / w[1]\n", " plt.plot(xp, yp, '-k')\n", " plt.plot(X[pos, 0], X[pos, 1], 'bx')\n", " plt.plot(X[neg, 0], X[neg, 1], 'yo')" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl8VNX5x/HPw9IgsqiQIi4Qk7AjBBKRSFUWqVRx4YdaVKwoGhEoIFFEICxRQHEDi0qpIoiRgohsoogmWKqIhC2CYCAjCi6AFhW0hiXn90cGCyEhk2Rmzr13nvfrNa/Mcp35ekOee+bcc84VYwxKKaW8pZLtAEoppYJPi7tSSnmQFnellPIgLe5KKeVBWtyVUsqDtLgrpZQHaXFXSikP0uKulFIepMVdKaU8qIqtD65bt66JiYmx9fFKKeVK69at+84YE13adtaKe0xMDNnZ2bY+XimlXElEvghkO+2WUUopD9LirpRSHqTFXSmlPEiLu1JKeZAWd6WU8iAt7kop5UFa3JVSyoNcV9z37NlDamoqe/futR1FKaUcy3XFPSsriylTphAXF8eYMWP46aefbEdSSinHcV1x79WrF1u2bOFPf/oT6enpxMbG8tRTT/Hrr7/ajqaUUo7huuIO0KRJE+bNm0d2djZt27YlNTWVxo0bM2PGDI4cOWI7nlJKWefK4n5MYmIi77zzDu+99x7169enb9++XHjhhbz++usYY2zHU0opa1xd3I/p3LkzH330EQsWLEBEuOGGG2jXrh3vvvuu7WhKKWWFJ4o7gIjQo0cPPvnkE1566SX27t1L165dueKKK1i7dq3teEopFVaeKe7HVK5cmT59+pCbm8vkyZPZtGkT7dq1o2fPnmzdutV2PKWUCgvPFfdjoqKiGDx4MD6fj7Fjx7JixQpatmzJnXfeyZdffmk7nlJKhZRni/sxNWvWZMyYMeTl5TF48GAyMjJo1KgRQ4cOZd++fbbjKVWqSZMgK+vE57KyCp9XqiSeL+7HREdH89RTT7F9+3Z69+7NlClTiI2NZdy4cRw4cMB2PKVKdNFFcNNN/yvwWVmFjy+6yG4u5WwRU9yPadCgAS+++CKbN2+ma9eujB07ltjYWCZPnkx+fr7teCrE3NgK7tQJ5s0rLOijRxf+nDev8HmlShJxxf2YZs2asWDBAj766CNatWrFfffdR+PGjZk5cyZHjx61HU+FiFtbwZ06wb33wsMPF/4Md2F340Ex4hljrNwSExONk7zzzjsmMTHRAKZZs2ZmwYIFpqCgwHYsFQKZmcbUrWtMWlrhz8xM24lKZzvzsc8/9rlFH6vwAbJNADVWi/txCgoKzPz5802TJk0MYNq1a2cy9V+vJ6WlFf7rT0uznaR0Timstg8wqlCgxT1iu2WKIyL07NmTzZs388ILL/D111/TuXNnrrzyStatW2c7ngqSrCx4/nlISyv8WbS7wWnWrj2xj/1YH3y45+bZ7hpSZRTIEaDwYEFlYAOwtJjX+gD7gI3+212lvZ8TW+5F/fe//zVPPvmkqVOnjgHMjTfeaLZt22Y7lqoAp7SC3Uhb7s5ACFrug4FTTfGca4xJ8N9eKPthxnmqVavG0KFD8fl8jB49mmXLltGiRQvuvvtudu/ebTueKgentILd5tiJ53nzID39f6N3nP6tJ5IFVNxF5DzgasATRbusatWqxbhx4/D5fAwcOJCXX36Z+Ph47r//fr7//nvb8VQZDBt2cndCp06Fz6uS6UHRfcQEsDSuiMwHJgI1gfuNMd2LvN7H//o+IBe4zxizq5j3SQFSABo0aJD4xRdfVDS/FTt37mTs2LHMnj2bGjVq8MADDzBkyBBq1KhhO5pSyuNEZJ0xJqm07UptuYtId2CvMeZUZxSXADHGmFbACmBWcRsZY6YbY5KMMUnR0dGlfbRjxcTEMHPmTHJycujcuTNpaWnExcUxdepUDh06ZDueUkoF1C3TAbhWRHYC/wQ6i8grx29gjPneGHNseucLQGJQUzpUixYteOONN1i9ejXNmzfnr3/9K02aNGH27Nk6EUopZVWpxd0Y85Ax5jxjTAzQC8g0xvQ+fhsRqX/cw2s59YlXz2nfvj2ZmZksX76cs846i7/85S+0bt2aRYsW6RWhlFJWlHucu4iki8i1/oeDRGSLiGwCBlE4NDKiiAh//OMfWbt2LXPnzuXw4cNcf/31dOjQgffff992PKVUhAnohGooJCUlmezsbCufHQ5Hjhxh5syZjB07lq+++oorr7ySCRMm0LZtW9vRlFIuFrQTqqp8qlSpwl133cX27dt5/PHHWbt2LYmJifTq1Yvt27fbjqdCQBfXUk6ixT3ETjvtNO6//358Ph+jRo1i6dKlNGvWjHvuuYevvvrKdjwVRG5dcVJ5kxb3MKlduzYPP/wweXl59O/fn5deeon4+HiGDRvGf/7zH9vxVBDouuvKSbS4h1m9evV45pln+Oyzz7jpppt44okniI2NZcKECfz888+246kK0sW1lFNocbfkggsuYNasWeTk5HD55ZczcuRI4uLiePbZZ3UilIu5bcVJ5V1a3C1r2bIlixYt4sMPP6RJkyYMHDiQpk2b8sorr+hEKJfRxbVOTU84h5cWd4dITk5m5cqVLFu2jNq1a3PbbbfRpk0blixZohOhXEIX1zo1PeEcXjrO3YEKCgqYN28eaWlp7Nixgw4dOjBx4kQuvfRS29GUqpBjBf3eewu7rfSEc9npOHcXq1SpEr169eLTTz/l73//O59//jmXXXYZV199NZs2bbIdT6ly0xPO4aPF3cGqVq1KSkoK27dv57HHHmP16tUkJCRwyy23kJeXZzueUmWmJ5zDR4u7C1SvXp1hw4bh8/kYMWIEixYtomnTpvTv359vvvnGdjylAqInnMNLi7uLnHHGGYwfP568vDzuuece/vGPfxAXF8fw4cPZv3+/7XhKnZKecA4vPaHqYnl5eYwePZo5c+ZQu3ZtHnzwQQYNGkT16tVtR1Mq4kyaVDjy5/jzCFlZhQevYF7GUU+oRoC4uDgyMjLYsGEDHTp04KGHHiI+Pp7nn3+ew4cP246nVERx2lBPLe4e0Lp1a5YuXcqqVauIjY2lf//+NGvWjDlz5lBQUGA7nlIRwWlrC2lx95A//OEPrFq1iqVLl3L66adzyy230LZtW5YtW6YToZQKAycN9XRNcdepy4EREa6++mo2bNhARkYGBw4c4Oqrr+byyy/ngw8+sB1PKU9z0lBP1xR3p/VnOV2lSpW45ZZb2Lp1K88++yzbt2/nD3/4A9dccw05OTm24ynlOU4b6uma4u60/iy3+N3vfkf//v3ZsWMHEyZMYNWqVSQkJHDbbbfh8/lsx1PKMxw31NMYY+WWmJhoyiMtzRgo/KnK7vvvvzcPPvigqVatmqlataoZMGCA+eabb2zHCpnHHjMmM/PE5zIzC59Xyo2AbBNAjXVNyx2c1Z/lVmeddRaPPvooeXl59O3bl2nTphEXF8fIkSP54YcfbMcLOu3O+x89bxVhAjkChOJW1pZ7ZqYxdev+rxVW9LEqn9zcXNOrVy8DmDPPPNNMmjTJ/PLLL7ZjBdWxfytpaZH9b0b/hryBAFvuARdjoDKwAVhazGtRwFxgB7AGiCnt/cpa3PXrdWitX7/edOvWzQDmnHPOMX//+9/NoUOHbMcKGu3OK6QHOvcLRXEfCrxaQnHvD0zz3+8FzC3t/crb565C6/333zeXXHKJAUx8fLz55z//aY4ePWo7VoVoQTuRHujcLajFHTgPeA/oXEJxXw4k++9XAb7Dv25NSTct7s5VUFBglixZYlq2bGkA06ZNG/PWW2+ZgoIC29HKTLsiTqQHOvcLtLgHekJ1MjAMKGku+7nALn8f/hHgR6BOgO+tHEZE6N69Oxs3bmT27Nn88MMP/OlPf6JTp06sXr3adrwycdzwNIucNg5bhVapxV1EugN7jTHrKvphIpIiItkikr1v376Kvp0KscqVK9O7d2+2bdvG1KlT2bZtG5dccgnXXXcdmzdvth0vIMOGnTwXolOn4K7S5xZ6oIsspS75KyITgduAI0A1oBawwBjT+7htlgNjjTGrRaQK8C0QbU7x5rrkr/scPHiQKVOmMGnSJA4cOEDv3r1JT08nJibGdjSlIkbQlvw1xjxkjDnPGBND4cnSzOMLu99i4Hb//Rv82+hKVR5To0YNRo4cic/n4/777+e1116jcePGDBo0iD179tiOpzxKx+eXT7knMYlIuohc63/4IlBHRHZQOKpmeDDCKWeqU6cOkyZNYvv27fTp04fnnnuOuLg40tLS+PHHH23HUx6jE9HKR6/EpCosNzeXtLQ05s2bR506dXjooYcYMGAA1apVsx1NecSxgn7vvYWz0yN5XSm9EpMKm8aNGzN37lzWrVtHUlIS999/P40aNeLFF1/kyJEjtuOFlXYhhEYo1kn3+u9Ki7sKmrZt2/L222+TlZXFeeedx1133UXLli2ZP38+kXIKRrsQQiMU60p5/ncVyGD4UNx0EpO3FRQUmIULF5rmzZsbwCQlJZkVK1bYjhUWOlEouEI5Ec2Nvyu8uCqkcg8R4brrriMnJ4eZM2eyb98+unbtSpcuXfj4449txwspJ11qzQtCOT7f07+rQI4Aobhpyz2y/Prrr2bKlCkmOjraAKZHjx5my5YttmOFhBtbg5HKjb8rgr1wWLBvWtwj008//WTS09NNzZo1TaVKlUyfPn3Mzp07bccKGl3Lxj3c+rsKtLhrt4wKq5o1a5KWlobP52PIkCHMmTOHxo0bM2TIEPbu3Ws7XoXpFH/38PrvSse5K6t27dpFeno6M2bMoHr16gwdOpTU1FRq1aplO5pSjqTj3JUrnH/++fzjH/9gy5YtdOvWjfT0dGJjY3n66af59ddfbcdTyrW0uCtHaNq0Ka+99hpr166lbdu2DB06lMaNGzNjxoyImwilVDBocVeOkpSUxDvvvMO7777L2WefTd++fbnwwgtZsGBBxEyEUioYtLgrR+rSpQtr1qxhwYIFiAg9e/bk4osv5r333rMdTSlX0OKuHEtE6NGjBzk5OcyYMYNvv/2WK664gq5du7LWK0MalAoRLe7K8apUqcIdd9xBbm4uTz/9NBs3bqRdu3bccMMNbNu2zXY8pRxJi7tyjWrVqjFkyBDy8vIYM2YMy5cvp0WLFvTt25ddu3bZjqeUo2hxV65Tq1Ytxo4di8/nY9CgQbzyyis0atSI1NRUvvvuO9vxlHIELe7KtaKjo3n66afZvn07t9xyC5MnTyY2Npb09HQOHDhgO55SVmlxV67XoEEDZsyYwebNm+natStjxowhLi6OZ555hvz8fNvxlLJCi7vyjGbNmvH666+zZs0aLrzwQgYPHkyTJk2YNWsWR48etR1PqbDS4q48p127drz33nusWLGC6Oho+vTpQ6tWrVi4cKFOhCoDr1+Gzuu0uCvPuuKKK/j444+ZP38+R48epUePHiQnJ7Ny5Urb0VzB85eh8zgt7srTjs1u3bx5My+88AJfffUVnTp14sorr2TdunW24znasSVwb7oJRo8u/Hn8ErnK2Uot7iJSTUQ+FpFNIrJFRMYVs00fEdknIhv9t7tCE1fZtGdPBqtXx7ByZSVWr45hz54M25ECVqVKFfr27cv27dt58sknyc7OJikpiZtuuonc3Fzb8RzL05eh87hAWu75QGdjTGsgAegmIu2L2W6uMSbBf3shqCmVdXv2ZPDZZynk538BGPLzv+Czz1JcVeChcCLU0KFD8fl8pKWlsWzZMpo3b05KSgq7d++2Ha9cQtk3npUFzz8PaWmFP4t+jnKuUou7/8pOB/0Pq/pvelYqwvh8Iyko+OWE5woKfsHnG2kpUcXUrl2b9PR0fD4fAwYMYObMmTRq1Ihhw4bx/fff245XJqHqGz/2PvPmQXr6/7potMC7Q0B97iJSWUQ2AnuBFcaYNcVs1lNEckRkvoicH9SUyrr8/C/L9Lxb/P73v2fKlCnk5uby5z//mSeeeILY2FgeeeQRDh48WPobOECo+sa9fhk6ryvTZfZE5AzgDeCvxpjNxz1fBzhojMkXkXuAPxtjOhfz36cAKQANGjRI/OKLLyqaX4XJ6tUx/i6ZE0VFNSQ5eWf4A4XI5s2bGTVqFIsWLeL3v/89o0aNIiUlhaioKNvRSjV6dGHfeFpaYUtbeVNILrNnjPkByAK6FXn+e2PMsamALwCJJfz3040xScaYpOjo6LJ8tLIsNnY8lSpVP+G5SpWqExs73lKi0GjZsiULFy5k9erVNG/enEGDBtG0aVNefvllR0+E8nrfuBvG3DstYyCjZaL9LXZE5DSgK7CtyDb1j3t4LbA1mCGVffXq3UqTJtOJimoICFFRDWnSZDr16t1qO1pItG/fnszMTJYvX85ZZ53F7bffTuvWrVm8eLHjJkJFQt+4G8bcOy6jMeaUN6AVsAHIATYDo/3PpwPX+u9PBLYAmyhs2Tct7X0TExONUm5w9OhRM3fuXNOoUSMDmOTkZLNy5UrbsX7z2GPGZGae+FxmZuHzXpKZaUzdusakpRX+LPr/7AThyAhkm1LqqzGm9OIeqpsW9+D49ttXzIcfNjRZWWI+/LCh+fbbV2xH8qxDhw6Z6dOnm3POOccAplu3bmb9+vW2Y0WUtLTCqpWWZjtJyUKdMdDirjNUXcwrY8/domrVqtx9993s2LGDSZMmsWbNGtq2bcvNN9/M9u3bbcfzPDecV3BSRi3uLub2sedunfF62mmn8cADD/D5558zcuRIFi9eTPPmzenXrx9ff/217Xie5IbzCk7LqMXdxdw89twL3zpq167NI488Ql5eHv369WPGjBnEx8czfPhw9u/fbzuep7hhzL3TMpZpnHswJSUlmezsbCuf7RVuHnvu5uwl+fzzzxk9ejQZGRnUqlWLBx98kEGDBnH66afbjhYSkyYVjgQ5frJUVlZhMRs2zF4urwvJOHflLG4ee+7mbx0lueCCC5g9ezabNm3isssuY8SIEcTFxfHcc89x6NAh2/GCznFD/9QJtLi7mJvHnkdFNSjT825y4YUXsnjxYj744AMaN27MgAEDaNasGRkZGRQUFNiOFzS6JLCzaXF3uXr1biU5eScdOxaQnLzTFYUd3P2tI1CXXHIJ77//Pm+++SY1a9akd+/eJCQksHTpUsdNhCovXRLYubS4Kyvc/K2jLESEq666ivXr1/Pqq6/yyy+/cM0113DppZeyatUq2/EqzElD/1QRgQyGD8VNJzGpSHTo0CHz/PPPm/r16xvAXHXVVWbTpk22Y5XLsdmYx2ZhFn2sQgOdxKSU81StWpV+/fqxY8cOHn30UT788EMSEhK49dZbycvLsx2vTJw29E+dSIdCqoi2Z08GPt9I8vO/JCqqAbGx48PaNbR//34ef/xxJk+ezOHDh0lJSWHUqFHUr1+/9P9YRSQdCqkqxK2zR8vCCROpzjzzTCZMmEBeXh53330306dPJy4ujhEjRuhEKI8J95LAWtzVSZxQ9MLBScs31K9fn+eee45t27bRo0cPHn30UWJjY3nsscf45ZdfSn8D5XjhnhegxV2dxElFL5ScOJEqLi6OjIwMNmzYQIcOHRg+fDjx8fFMmzaNw4cPW8ulKi7c8wK0uKuTOLHohYKTJ1K1bt2apUuX8q9//YvY2FjuvfdemjVrxpw5czw1ESrShHNegBZ3dRInF71gcsNEqmPj4ZcsWcLpp5/OLbfcQtu2bXnrrbc8MxEqXJxwGbxwzgvQ4q5O4oaiFwxumUglInTv3p0NGzaQkZHBgQMHuOqqq7j88sv54IMPbMdzDdtr4YR9SeBABsOH4qaTmJxNr/DkXPn5+ebZZ581Z599tgHMNddcY3JycmzHcgWbl+oL1uUQ0cvsKSfTg0fFHTx40EyYMMHUrl3biIjp3bu3ycvLs5rJDddzdcOl+k4l0OKu3TIq7CJlqGWonX766Tz00EP4fD6GDRvG/Pnzadq0KQMHDuTbb7+1ksl210dpImktHC3uKuwiZahluJx11lk8+uij5OXlceeddzJt2jTi4uIYNWoUP/74Y1izOHkZYKddBi/UtLirsIuUoZbhds455zBt2jS2bt3Ktddey/jx44mNjeXxxx/nv//9b9hyOHUZ4EhbC6fU4i4i1UTkYxHZJCJbRGRcMdtEichcEdkhImtEJCYUYSOdV5YEiJShlrY0atSIOXPmsH79ei6++GKGDRtGo0aNmD59elgmQjm162PYsJMPNJ06efeSgIG03POBzsaY1kAC0E1E2hfZpi+w3xgTDzwNPBbcmJGjpALupX7qSBlqaVubNm1YtmwZ77//Pg0bNuSee+6hRYsWzJ07N2QToSKt68PJSi3u/hO0B/0Pq/pvRWdPXAfM8t+fD3QREQlayghxqgLupX5qt4wv94rLLruMf//73yxatIioqCh69epFUlISy5cvD/pEqEjr+nCygJb8FZHKwDogHnjWGPNgkdc3A92MMbv9j/OAi40x35X0nrrk78lWr47xF/YTRUU19PdHF/e7Ejp21OnoTmV7SeGijh49yquvvsro0aPZuXMnl19+ORMnTiQ5OdlaJlU2QV3y1xhz1BiTAJwHtBORluUMlSIi2SKSvW/fvvK8haed6kSj9lO7jxO70ipXrsxtt93Gtm3b+Nvf/sbWrVu55JJLuP7669m8ebO1XCr4yjRaxhjzA5AFdCvy0lfA+QAiUgWoDXxfzH8/3RiTZIxJio6OLl9iDztVAQ9FP7VXTtA6lZO70qKiohg4cCB5eXk8/PDDZGVl0apVK26//XZ27txpO54KgkBGy0SLyBn++6cBXYFtRTZbDNzuv38DkGmC3ZkXAU5VwIPdT+3EVmUo2DyAuWHIZ40aNRg1ahQ+n4/U1FTmzp1L48aNGTRoEHv27LEdT1VAqX3uItKKwpOllSk8GMwzxqSLSDqF02AXi0g1YDbQBvgP0MsY4zvV+2qfe/HC1Ud7qv795OSdQf88G44dwI5vPVeqVD1sJ2/duI93795Neno6M2bMoFq1agwdOpTU1FRq165tO5ryC7TPXa+hGqFWrqyE10/Q2i6uwTy4hPvE7GeffUZaWhqvvfYaderUYcSIEfTv359q1aqF7DNVYPQaquqUIuEEre1ukWB1pdnoQmvSpAnz5s0jOzubxMREUlNTadSoES+++CJHjhwJ2eeq4NHiHqEiYSKREw5g9erdSnLyTjp2LCA5eWe5Wts2T8wmJiayfPlyMjMzOffcc7nrrrto2bIl8+fPd83FQpxwkQ4btLhHqEiYSOSVA5jtbyAAnTp1YvXq1bzxxhtUrlyZG2+8kXbt2vHuu++GLUN5OX2lylDR4h7BgtGqdDKvHMCc8A0ECq8Idf3115OTk8PMmTPZu3cvXbt2pUuXLnz88cdhzVIWTl6pMpS0uCtP88IBzGnfQCpXrsztt99Obm4ukydP5pNPPuHiiy/m//7v/9i6dauVTKVx6kqVoaTFXUUUN07ccuo3kKioKAYPHkxeXh7jxo3j3XffpWXLltx55518+aVzxvKDc1eqDCUdCqkihu1x71733XffMXHiRJ599lmMMfTv358RI0Zgezb68StVdup08mO30aGQShXh5OUAvKBu3bo8+eSTbN++nd69e/PMM88QGxvLuHHjOHDggLVckbpSpbbcVcSIhIlbTrJt2zbS0tKYP38+devWZeTIkfTr108nQlWQttyVKsIpo04iRdOmTXnttdf4+OOPSUhI4L777qNx48a89NJLOhEqDLS4q4gRrlEnbjxpG0oXXXQRK1as4N133+Xss8/mzjvvpFWrVixYsMA1E6HcSIu7CjtbxS8co04iZbXN8ujSpQtr1qzh9ddfxxhDz549ad++PZmZmbajeZL2ubuY067yEwivj1ixvViZWxw5coTZs2czZswYdu3axRVXXMHEiRNJSiq1KzniaZ+7x7m1hej1EStOWCrADapUqcIdd9xBbm4uTz31FBs3buSiiy7ixhtvZNu2opeLUOWhxd2l3FokvV78ynvSNlL76atVq8Z9991HXl4eY8aM4e2336ZFixbcdddd7Nq1y3Y8V9Pi7lJuLZJeH7FSnpO2Nr+FOeWgUqtWLcaOHYvP52PQoEHMnj2bRo0akZqaynfffWclk9tpcXcptxZJp62TEmzlOWlr61uYE7v2oqOjefrpp8nNzeXmm29m8uTJxMbGkp6ebnUilBtpcbesvC0ntxZJp66TEkxlXazM1rcwJ3ftNWzYkJdeeolPPvmELl26MGbMGOLi4pgyZQr5+fm247mCFneLKtJycnOR9MJKjcFk61uYG7r2mjdvzhtvvMFHH31Ey5YtGTJkCE2aNGHWrFkcPXrUdjxH0+JuUUVbTlokvcHWtzA3de1dfPHFvPfeeyxfvpy6devSp08fWrVqxcKFC3UiVAm0uFvkhpaTCj1b38Lc1rUnIvzxj39k7dq1vPbaaxw5coQePXpwySWX8P7779uO5zha3C1yU8tJhZaNb2Fu7doTEW644Qa2bNnCCy+8wO7du+nYsSPdunVj/fr1tuM5RqnFXUTOF5EsEflURLaIyOBitukoIj+KyEb/bXRo4nqL21pOynvc3LVXpUoV+vbtS25uLk888QRr164lMTGRP//5z+Tm5tqOZ10gLfcjQKoxpjnQHhggIs2L2W6VMSbBf0sPakqPcmvLSSknOe2000hNTcXn85GWlsabb75J8+bNSUlJYffu3bbjWVPmtWVEZBEw1Riz4rjnOgL3G2O6B/o+uraMUioU9uzZw4QJE3j++eepXLkyAwcOZPjw4dSpU8d2tKAIydoyIhIDtAHWFPNysohsEpG3RKRFWd5XKaWCpV69ekyZMoXc3FxuuukmnnzySWJjYxk/fjwHDx60HS9sAi7uIlIDeB0YYoz5qcjL64GGxpjWwN+AhSW8R4qIZItI9r59+8qbWSmlShUTE8OsWbPIycmhU6dOjBo1ivj4eKZOncqhQ4dsxwu5gIq7iFSlsLBnGGMWFH3dGPOTMeag//4yoKqI1C1mu+nGmCRjTJLti+YqpSJDy5YtWbhwIR9++CFNmzZkyCpWAAALZ0lEQVTlr3/9K02bNmX27NmenggVyGgZAV4Ethpjniphm7P92yEi7fzv+30wgyqlVEUkJyeTlZXF22+/zRlnnMFf/vIXEhISWLJkiScnQgXScu8A3AZ0Pm6o41Ui0k9E+vm3uQHYLCKbgGeAXsaLe0uFlVNWLFTeISJceeWVZGdnM3fuXPLz87n22mvp0KED//rXv2zHCyq9EpNyJK9fsSkQbrzSltscPnyYl156iXHjxvH111/TrVs3JkyYQJs2bWxHK5FeiUm5mpNXLAwHJy7H60VVq1YlJSWFHTt2MGnSJNasWUPbtm25+eab2bFjh+14FaLFXTlSuNbdcWrXT6Qf3MLttNNO44EHHsDn8zFy5EgWL15Ms2bN6NevH19//bXteOWixV05UjjW3XFy61gXlbPjjDPO4JFHHiEvL4977rmHGTNmEB8fz/Dhw9m/f7/teGWixV2dwCkt2XCsu+Pk1rEuKmfX2WefzdSpU9m2bRs9e/Zk0qRJXHDBBUycOJGff/7ZdryAaHFXv3FSSzYc6+44uXWsi8o5Q2xsLLNnz2bjxo1ceumljBgxgvj4eJ577jnHT4TS4q5+47SWbKhXLHRy69hNi8o55dteKLVq1YolS5bw73//m/j4eAYMGECzZs3IyMigoKDAdrxiaXFXv3FySzYUnN46dsNyvE76thcOx8bDv/nmm9SsWZPevXvTpk0b3nzzTcdNhNLirn7j5JZsKLipdexUTvu2Fw4iwlVXXcX69et59dVX+fnnn+nevTuXXnopq1atsh3vN1rc1W+c3pINBTe0jp0s0r7tHa9SpUrcfPPNfPrppzz33HPk5eVx2WWX0b17d3JycmzH0+Ku/kdbsqqsIu3bXnF+97vfce+995KXl8fEiRP54IMPSEhIoHfv3vh8Pmu5dPkBpVS56TIRJ9u/fz+TJk1iypQpHD58mJSUFEaNGkX9+vWD8v66/IBSYRIJo0VKot/2TnbmmWcyceJE8vLyuPvuu5k+fTrx8fGMGDGCH374IWw5tOWuVAVoy1WVZseOHYwePZo5c+Zw5plnMnz4cAYOHEj16tVL/4+LoS13pcIgEkeLqLKJj4/n1VdfZcOGDSQnJ/Pggw+Smpoa8s+tEvJPUMrDInm0iCqbhIQE3nzzTVatWsV5550X8s/T4q5UBURFNfBP4Dn5eaWKc+mll4blc7RbRqkKiMS5AcodtLgrVQE6WkQ5lXbLKFVB9erdqsVcOY623JVSyoO0uCtPiuSJRUqBFncVYjaKrNuXodUDkwqGUou7iJwvIlki8qmIbBGRwcVsIyLyjIjsEJEcEWkbmrjKTWwVWTdPLHL7gUk5RyAt9yNAqjGmOdAeGCAizYts8yegkf+WAjwf1JTKlWwVWTdPLHLzgUk5S6nF3RjzjTFmvf/+AWArcG6Rza4DXjaFPgLOEJHgLIGmXMtWkXXzMrRuPjApZylTn7uIxABtgDVFXjoX2HXc492cfABQEcZWkXXzxKJw7DPt048MARd3EakBvA4MMcb8VJ4PE5EUEckWkex9+/aV5y2Ui9gqsm6eWBTqfaZ9+pEjoCV/RaQqsBRYbox5qpjX/w6sNMbM8T/+DOhojPmmpPfUJX8jw549Gfh8I8nP/5KoqAbExo53RZG1KZT7bPXqmBLWwmlIcvLOoHyGCq1Al/wtdYaqiAjwIrC1uMLutxgYKCL/BC4GfjxVYVeRQ2dvll0o95n26UeOQJYf6ADcBnwiIhv9z40AGgAYY6YBy4CrgB3AL8AdwY+qlKooXcUycpRa3I0x/waklG0MMCBYoZRSoREbO77YK0e54WSzKhudoapUBHHzyWZVNroqpFIRRs+DRAZtuSullAdpcVdKKQ/S4q6UUh6kxV0ppTxIi7tSSnmQFnellPIgLe5KKeVBWtyVUsqDtLgrpZQHaXFXSikP0uKulFIepMVdKaU8SIu7Ukp5kBZ3pZTyIC3uSpVgz54MVq+OYeXKSqxeHaMXkVauouu5K1WMPXsyTrhiUX7+F3z2WQqAroWuXEFb7koVw+cbecKl6AAKCn7B5xtpKZFSZaPFXali5Od/WabnlXIaLe5KFSMqqkGZnlfKabS4K1WM2NjxVKpU/YTnKlWqTmzseEuJlCobLe5KFaNevVtp0mQ6UVENASEqqiFNmkzXk6nKNUodLSMiM4DuwF5jTMtiXu8ILAI+9z+1wBiTHsyQStlQr96tWsyVawUyFHImMBV4+RTbrDLGdA9KIqWUUhVWareMMeZfwH/CkEUppVSQBKvPPVlENonIWyLSoqSNRCRFRLJFJHvfvn1B+millFJFBaO4rwcaGmNaA38DFpa0oTFmujEmyRiTFB0dHYSPVkopVZwKF3djzE/GmIP++8uAqiJSt8LJlFJKlVuF15YRkbOBPcYYIyLtKDxgfF/af7du3brvROSLUjarC3xX0Ywh4NRcoNnKy6nZnJoLNFt5VTRbw0A2CmQo5BygI1BXRHYDY4CqAMaYacANwL0icgT4L9DLGGNKe19jTKn9MiKSbYxJKm27cHNqLtBs5eXUbE7NBZqtvMKVrdTiboy5uZTXp1I4VFIppZRD6AxVpZTyIKcX9+m2A5TAqblAs5WXU7M5NRdotvIKSzYJoHtcKaWUyzi95a6UUqocrBd3EekmIp+JyA4RGV7M61EiMtf/+hoRiXFQtj4isk9ENvpvd4Up1wwR2Ssim0t4XUTkGX/uHBFpG45cAWbrKCI/HrfPRocp1/kikiUin4rIFhEZXMw2VvZbgNls7bdqIvKxfwb6FhEZV8w2Vv5GA8xm5W/U/9mVRWSDiCwt5rXQ7zNjjLUbUBnIA2KB3wGbgOZFtukPTPPf7wXMdVC2PsBUC/vtMqAtsLmE168C3gIEaA+scVC2jsBSC/usPtDWf78mkFvM79PKfgswm639JkAN//2qwBqgfZFtbP2NBpLNyt+o/7OHAq8W93sLxz6z3XJvB+wwxviMMYeAfwLXFdnmOmCW//58oIuIiEOyWWFKX8ztOuBlU+gj4AwRqe+QbFYYY74xxqz33z8AbAXOLbKZlf0WYDYr/PvioP9hVf+t6Ik6K3+jAWazQkTOA64GXihhk5DvM9vF/Vxg13GPd3PyP+rftjHGHAF+BOo4JBtAT/9X+Pkicn4YcgUi0Oy2BLTQXKj4vwK3obCldzzr++0U2cDSfvN3L2wE9gIrjDEl7rcw/40Gkg3s/I1OBoYBBSW8HvJ9Zru4u90SIMYY0wpYwf+OxKpkAS80FwoiUgN4HRhijPkpnJ9dmlKyWdtvxpijxpgE4DygnYicdNEeWwLIFva/URE5dnGjdaH+rFOxXdy/Ao4/kp7nf67YbUSkClCbANauCUc2Y8z3xph8/8MXgMQw5ApEIPvVCmNxoTkRqUph8cwwxiwoZhNr+620bDb323EZfgCygG5FXrL1N1pqNkt/ox2Aa0VkJ4XduZ1F5JUi24R8n9ku7muBRiJygYj8jsITC4uLbLMYuN1//wYg0/jPQtjOVqQ/9loK+0qdYDHwF//oj/bAj8aYb2yHgsKF5o71LUoZFpoLwucK8CKw1RjzVAmbWdlvgWSzuN+iReQM//3TgK7AtiKbWfkbDSSbjb9RY8xDxpjzjDExFNaNTGNM7yKbhXyfVXhVyIowxhwRkYHAcgpHp8wwxmwRkXQg2xizmMJ/9LNFZAeFJ+p6OSjbIBG5Fjjiz9YnHNmk9MXcllE48mMH8AtwRzhyBZitXAvNBUEH4DbgE38fLcAIoMFx2Wztt0Cy2dpv9YFZIlKZwgPKPGPMUif8jQaYzcrfaHHCvc90hqpSSnmQ7W4ZpZRSIaDFXSmlPEiLu1JKeZAWd6WU8iAt7kop5UFa3JVSyoO0uCullAdpcVdKKQ/6fwclyYxCRY7AAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Decision boundary with C=1. Notice that one example is not classified correctly.\n", "clf = train_svm(X, y, kernel='linear')\n", "plot_svm_decision_boundary(clf)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1] (51, 2) (51,)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3X9w1PW97/HnO4ipLV69NUgVDLlUxoo9tpjgj9G5BluLYrLM7Q/k1torbYdTWnq09Yz1x8CxUMfROWJtvVN1hN5WuIc6tppN/IU2wSOCCFRE0HLFCK0MBIGGlspEMO/7x+7GJCTZTbK73x/7esxksj++fvedlX3v5/v5vr/vj7k7IiISL2VBByAiIvmn5C4iEkNK7iIiMaTkLiISQ0ruIiIxpOQuIhJDSu4iIjGk5C4iEkNK7iIiMXRcUC9cUVHhVVVVQb28iEgkbdy4cZ+7j862XWDJvaqqig0bNgT18iIikWRmO3PZTtMyIiIxpOQuIhJDSu4iIjGk5C4iEkNK7iIiMaTkLhJyd98NLS09H2tpST0u0h8ld5GQmzIFZs78KMG3tKTuT5kSbFwSbkruUlKiOAqeOhUefTSV0BcsSP1+9NHU4yL9UXKXkhLVUfDUqTB3LixalPqtxC7ZKLlLSYnqKLilBX75S5g/P/W799FHoUXxiKfUKblLyYnaKDhzdPHoo7Bw4UdfTsVM8FE94illSu5ScoIeBQ/W+vU9jy4yRx/r1xcvhqge8ZQ0d8/pBxgBvAo09fHcdcB7wKb0z3ey7a+6utpFiq252b2iIvW7r/sysPnz3SH1W4IBbPAccvZgRu7XA28O8Pxv3f3z6Z+Hh/A9I1JwYRgFR1XUjnhKXU4tf81sHHAVcAfwo4JGJFJAN9107GNTp2p6IZvu8/6Z90tTM+GW68j9Z8BNQOcA23zFzDab2WNmdsbwQxORsNART/RYagpngA3M6oDp7v49M6sF/tXd63ptcwpwyN07zOyfgavd/bI+9jUHmANQWVlZvXNnTj3nRUQkzcw2untNtu1yGblfDCTMbAewArjMzJZ138Dd97t7R/ruw0B1Xzty94fcvcbda0aPzrpKlIiIDFHW5O7ut7j7OHevAmYBze7+je7bmNlp3e4mGPjEq4iIFNiQ11A1s4WkSnKSwL+YWQI4ChwgVRopIiIByTrnXig1NTWuBbJFRAYnn3PuIpID9V+RMFFyF8kT9V+RMBnynLuI9NS9/8rcuamrOHWRjwRFI3eRPIpax0mJLyV3kTxS/xUJCyV3kTwJQ991kQwld5E8Uf+VgamaqLiU3EXy5Kabjp1jnzq1706UpUjVRMWlahkRKQpVExWXRu4iUjSqJioeJXcRKRpVExWPkruIFIWqiYpLyV1EikLVRMWlrpAiInlw992pyp/u5xFaWlJfXvmsmFJXSBGRIgpbqadKIUVE8iBspZ4auYuI5EmYSj2V3EVE8iRMpZ6RSe7qSyEiYRa2Us/IJPewnawQEekubKWekSqFzCT0MJysEBEJQixLIcN0skKiQdN5UqoildzDdLJCokHTeR/RF11piUxyD9vJComG7rXHCxZ89G+oFI/69EVXWnJO7mY2wsxeNbOmPp4rN7Pfmtl2M1tnZlX5DBLCd7JCokPTeSn6oistgxm5Xw+82c9z3wb+6u5nAvcCdw03sN4yq9ysX7+e2tpaFi9ezBlnbNcqN5KVpvM+oi+60pFTcjezccBVwMP9bDID+HX69mPAF8zMhh/esdrb2zlw4AA33ngjEydOZNKkSdx8882sWbOGDz/8sBAvKRGm6bye9EVXOnIduf8MuAno7Of5scBfANz9KHAQOGXY0fXh8ssvZ/PmzbS2tnLfffdx+umnc88993DxxRdz2mmnMXv2bB5//HH+8Y9/FOLlJWI0nfcRfdGVlqx17mZWB0x39++ZWS3wr+5e12ubLcAV7v5u+v7bwAXuvq/XdnOAOQCVlZXVO3fuzMsf0d7ezjPPPEMymeTpp5+mvb2d8vJyvvCFL5BIJKirq2Ps2LF5eS2RqCpWS1oprFzr3HNJ7ncC1wJHgY8B/wX4vbt/o9s2zwK3u/taMzsO2AOM9gF2Xqh+7keOHGH16tU0NDSQTCZ55513Mq9HIpEgkUhw7rnnUqBZIxGRgspbcu+101r6Hrl/H/gnd/+umc0CvuzuMwfaVzEW63B33njjDRoaGmhsbGTdunW4O5WVldTX15NIJKitreX4448vaBwiIvlS8CtUzWyhmSXSd5cAp5jZduBHwM1D3W8+mRnnnHMOt956K2vXrmX37t08/PDDTJ48maVLlzJt2jQqKiqYOXMmy5Yt48CBA0GHLCK96OKroYlUb5l8Onz4MM8//zyNjY00NjayZ88eRowYwSWXXNI1fXPmmWcGFp+IpHQ/ETx16rH3S01BpmXyKejk3l1nZycbNmwgmUySTCZ5/fXXATj77LO7Ev0FF1zAiBEjAo5UpDQVomlgVE8w55rccfdAfqqrqz2sWltb/b777vMvfvGLftxxxzngo0eP9tmzZ/vjjz/uhw4dCjpECam77nJvbu75WHNz6nEZnvnz3SH1Ox+am90rKj76/9X7flgBGzyHHKvknkV7e7uvWLHCv/71r/vJJ5/sgJeXl/v06dP9gQce8F27dgUdooRIVBNG2GXex/nz8/t+Fmq/haTkXgAffPCBNzc3+w033OATJkxwwAGvqanxhQsX+qZNm7yzszPoMCVgUUwYYVboL8x8HxEUmpJ7gXV2dvqWLVv8zjvv9IsuusjNzAGvrKz0efPm+cqVK72joyPoMCUgUUsYYVbIqa4ofhEruRfZnj17fMmSJT5jxgw/4YQTHPATTzzRv/a1r/kjjzzi+/fvDzpEKZIoJoxSFNUptFyTu6plCuDw4cP84Q9/IJlMqsyyxKhsLzriXi2j5F5g3cssGxoa2LJlC6Ayy7iKasKQ6FByD6kdO3bQ2NhIQ0MDL7zwAkePHqWiooK6ujoSiQSXX345o0aNCjpMEQkpJfcIOHjwIM888wwNDQ19drOsr6/n9NNPDzpMEQkRJfeIyXSzzFwl29raCqibpYj0pOQeYe7Om2++2dXN8uWXX8a9ZzfLSy+9lPLy8qBDFZEiU3KPkba2NpqammhsbGTlypUcPnyYE088kSuuuIJEIsH06dP55Cc/GXSYIlIESu4xpW6WIqVNyb0EZMosGxsbSSaTbN68GfiozLK+vp4LL7xQZZYiMaLkXoIyZZbJZJJVq1apzFIkhpTcS1x7ezvPPvssyWSSp556SmWWIjGh5C5djhw5wosvvtg1qleZpUh0KblLn9xTi4Zn2iG88sorXWWWmRG9Fg0XCS8ld8lJW1sbTz75JMlkskeZ5ZVXXkl9fb3KLEVCRsldBi1bN8v6+nomTpwYdJgiJU3JXYale5llQ0ND16Lhn/nMZ7rm6VVmGW/qcBlOSu6SVzt27Oga0WfKLEePHk1dXR319fUqs4wh9aYPJyV3KZhs3Szr6uoYO3Zs0GFKHmQS+ty58MtfKrGHQa7JvSyHHX3MzF4xs9fMbKuZ/aSPba4zs/fMbFP65ztDDVzCq61tOWvXVvHqq/+Vysofc++9V7F3716am5uZO3cuf/rTn/jud7/LuHHjmDJlCosWLeK1114jqAGEDN/UqanEvmhR6rcSe3RkHblbqvj5E+5+yMxGAquB69395W7bXAfUuPu8XF9YI/doaWtbzrZtc+jsfL/rsbKyj3PWWQ8xZsw1QKrMcuvWrV19b7p3s1SZZeEUcm5cI/fwydvIPb0m66H03ZHpHw3FSkxr6209EjtAZ+f7tLbe1nXfzPjsZz/LLbfcwpo1a9i9ezcPP/wwkydPZsmSJUybNo2KigpmzpzJsmXLOHDgQLH/jFiaMiWVgFtaUvczCXnKlOHtt/sc+8KFqd/dX0fCLWtyBzCzEWa2CdgLPOfu6/rY7CtmttnMHjOzM/IapQSuo+PPg3ocYMyYMXz729/miSeeYN++fTQ2NjJr1ixefPFFrr32Wk499VRqa2tZvHgxb731VqFCj72pUz9KvAsW5O+k5/r1PfeTeZ3164cfsxTeoE6omtnJwOPAD9x9S7fHTwEOuXuHmf0zcLW7X9bHfz8HmANQWVlZvXPnzuHGL0Wydm0VHR3H/v8qLx/PRRftGNS++utmqTLL4VmwIDU3Pn9+aqQt8VSwahkzWwC87+7/3s/zI4AD7n7SQPvRnHu05DLnPlTqZjl8mhsvHbkmd9x9wB9gNHBy+vYJwItAXa9tTut2+38AL2fbb3V1tUu07NmzzNesGe8tLeZr1oz3PXuW5f012tvbfcWKFX7NNdf4ySef7ICXl5f79OnT/YEHHvBdu3bl/TWjrrnZvaIi9buv+3Fw113H/j3NzanHw6JYMQIbPEt+dfeckvu5wKvAZmALsCD9+EIgkb59J7AVeA1oAT6Tbb9K7pLNBx984C0tLf7DH/7QJ0yY4KRO5Ht1dbX/5Cc/8VdffdU7OzuDDjNwUUh8wxWFL7BixZhrctdFTBHX1rac1tbb6Oj4M+XllUyYcMewp0nCyNOLhieTSZLJZFeZ5RlnnNE1T68yy3iLwtRTMWLUFaoloJDz4GGX6WbZ0NDAc889p0XDS0QUThoXOkYl9xKQzwqWIOTrqCPTzbKhoYGmpqYe3Szr6+tJJBLqZhkDGrmnKLmXgFWryuj7ejKjtraz2OEMSqGOOtTNMp6i0MSsWDHm7QpVCa/y8spBPR4muVzxOhRlZWWcf/75LFq0iM2bN/POO+/w85//nLFjx7J48WIuueQSPvWpTzF79mwef/xxDh06lH2nErgoXFAVthg1co+wKM+5B3HUkelm2XvR8Msuu6yr9426WeZO/d6DoZF7CRgz5hrOOushysvHA0Z5+fhIJHYI5qjjpJNO4uqrr2b58uU9ullu27aNuXPnMm7cOGpqali4cCGbNm1SN8ssCtXTRvJDI3cJRJiOOjJllg0NDT26WarMMrsonOSMG51QldALa41+W1sbTU1NNDY29lg0XGWWfYtCeWKcKLmL5MHhw4d5/vnnaWxspKmpid27d2vR8G40ci8+JXeRPFM3y56iUJ4YR0ruIgW2c+fOrnr67ouGX3XVVSQSCb70pS/xiU98IugwC0bVMsFQchfJQb7m/fsrs9Si4ZJvSu4yLGE92ZlPharYOXLkCC+99BLJZJKGhgZaW1sBqK6u7pq++dznPkdqeWIpFfk60lFylyELU5liIRWjN08u3SwvvfRSysvL8/J6El75Okeh5C5DFvWGZLkK4ipZdbMsbfmoLtIVqjJkQ1kMO4qCuEp2zJgxfOtb36KhoYH9+/cPuGj49u3bCxaHBGPq1FRiX7Qo9buQVUVK7nKMKDckG4wJE+6grOzjPR4rK/s4EybcUZTXP+GEE6irq+Ohhx5i165dvPLKK9x888389a9/5cYbb2TixImcffbZ/PjHP+all17iww8/LEpcUjgtLakR+/z5qd+Z1g2FoOQuxwg66RVLmHrzlJWVMWXKFH7605/y2muv8c4773Dfffcxbtw4dbPMk7vvPjaZtrSkHi+G7nPsCxemfnfvzZNvmnOXPpVCtUxUqJtlfgR90ZWqZUSkX5kyy4aGBpLJZFeZZU1NTdeqUyqz7F8c2iUouUuo6chg+HLpZlnsMssoXLUa9UZnqpaR0MrU0afKLZ2Ojp1s2zaHtrblQYcWKWbGpEmTuOWWW1izZg27d+9myZIlTJ48maVLlzJt2jRGjx7NzJkzWbZsGQcOHCh4TGHv8V7ME5pB08hdiq5U6uiDlFk0vLGxkcbGxq5ulhdffDEzZswgkUhw5plnFuS1wzr1EfSce75o5B5DbW3LWbu2ilWryli7tiqyI91SqaMPUqbM8sEHH+Tdd99l3bp13HLLLbS3tx9TZrl69eq8llkWs5Z7MMK2xmmhZR25m9nHgP8EyoHjgMfc/d96bVMO/AaoBvYDV7v7joH2q5H74MSpJYBG7sHasWNHV9viTDfLiooK6urqSCQSXH755YwaNWrI+w/ryD0u8jly7wAuc/fPAZ8HrjCzC3tt823gr+5+JnAvcNdgA5aU/kbnra239UjsAJ2d79PaelsQYQ5LqdTRh1VVVRU/+MEPeO6559i3bx8rVqxg2rRpPPHEE3z5y1+moqKCq666igcffJBdu3YNat/FruWW/h2XbQNPDe0zV0yMTP/0Hu7PAG5P334MuN/MzLXC8KD0Hp1nTjSmbsdnKiNzpKFqmeBlFg2/+uqrOXLkCKtXr+7qUf/UU08Bg+tmOdDUh0bvxZXTCVUzGwFsBM4E/re7/7jX81uAK9z93fT9t4EL3H1ff/vUtMyxBpquADSVIUXj7rzxxhtd0zdhKLOUlLyeUHX3D93988A44Hwz++wQg5pjZhvMbMN77703lF3E2kCjc01lRFNUT4KbGeeccw4333xzV5nl0qVLOe+88wIrs5TBGXQppJktAN5393/v9tizwO3uvtbMjgP2AKMHmpbRyP1Y2U406sKfaInTSfDuDh8+THNzM8lkskeZpRYNL468XaFqZqOBI+7ebmYnACuBu9y9qds23wf+yd2/a2azgC+7+8yB9qvkfqxiJ4NS+LII8m8shaqgzs5ONm7c2LXq1Ouvvw6U7qLhxZDP5H4u8GtgBKlpnEfdfaGZLQQ2uHsyXS75CDAZOADMcvfWgfar5N63YiWjuI4quwv6bwxiMZCgFbrMUtRbRrIohVFl0H9jPl8/ikdZBw8e5Nlnn+2qvFE3y/zQFaoyoDiVVvYn6L8xXyfBo9qL56STTmLmzJksX76cvXv30tLSwty5c9m2bRtz585l3Lhx1NTUsHDhQjZt2oQqp/NLyb1ElcJqS0H/jflaDCQOF7CNHDmS2tpa7r33XrZv387WrVu58847GTlyJLfffjuTJ09m/PjxzJs3j5UrV9LR0ZG31w56kY6gaFqmRAU9H10Mcfkb4z53n23R8CuvvJJTTjllyPuPS8OwDM25S1ZRnMcdrDj8jUGfOyimTJllQ0MDTU1N7N69m7Kysq4yy0QiMaQyyzj1u1FyF4mJuByBDFamzDKzGMnmzZuBoZdZRn2Rjgwld5EYicMRyHDt3LmTZDI5pDJLjdyLSMldgqAkGQ+ZRcMbGxt58sknByyz1Jx7kSm5S7GV6vRG3PXuZplZNDzTzfLAge+QSJzGZZd91M0ybOu6DoaSu0gvpXRislQN1M2yvr6eRCJBbW1tpLtZKrmL9BL3kkI51t69e3nyySdJJpOsXLmS999/n1GjRnWVWU6fPn1YZZZBUHIX6UUj99LWVzfLfJRZFpuSu0gvxZpz10nb8ItyN0sldwmtIJNfoV9bJ22jqXs3yxdeeIEjR46EtpulknsJiOIIMe7JT1M/0ZfpZplMJrOWWQZByT3mopok4578dNI2Xo4cOcJLL73UdfHU22+/DQxu0fB8U8vfmItqp8Cg2/AW2lA7UUZ1rdW4y3SzXLx4MW+99VZXN8vjjz++4N0sh0vJPaKimiSDbsNbaEPp4R5kv3Z9qeTOzJg0aVLXouF79uzpWjT8V7/6VY9Fwx955BH2798faLxK7hEV1SSZrwUswmooPdyDOgqL6iIgYXHqqacye/ZsnnjiCfbt20dTUxOzZs1i9erVfPOb3+TUU0/l0ksv5Z577uGtt94qenyac4+oqM65QzRPBBdSUPP0cT//EZR8d7PsTSdUI2I4iU5JMh6CSrI6+VscmTLLxsZGVq1axZEjR5g3bx6/+MUvhrQ/JfcIiPLoW/InqH8HGrkXX6bM8tOf/jTV1dVD2oeqZSIgqhUvkl/5Wmt1sOJ+/iOMMouGDzWxD8ZxBX8F6VdUK14k/8aMuaboR2uZ19PUXjwpuQeovLyyn8PicFe8SHwE8aUixZF1WsbMzjCzFjN7w8y2mtn1fWxTa2YHzWxT+mdBYcKNFx0Wi0ih5DJyPwrc6O5/NLMTgY1m9py7v9FruxfdvS7/IcaXDotFpFCyJnd33w3sTt/+u5m9CYwFeid3GQIdFotIIQyqWsbMqoDJwLo+nr7IzF4zs6fN7Jw8xCYiIkOU8wlVMxsF/A64wd3/1uvpPwLj3f2QmU0HngCOWdLEzOYAcwAqK3XSUESkUHIauZvZSFKJfbm7/7738+7+N3c/lL79FDDSzCr62O4hd69x95rRo0cPM3QREelPLtUyBiwB3nT3xf1s86n0dpjZ+en9BtsSTUSkhOUyLXMxcC3wupltSj92K1AJ4O4PAF8F5prZUeAwMMuD6msgsaHeOSJDl0u1zGpgwGVG3P1+4P58BSXSu99Kph0tUDIJXl9uMhzqLSOhVOp9d9RrXYZLyV1CqVh9d8K6ElGpf7nJ8Cm5SygVY6WpMI+O1VROhkvJXXoIy0i2GH13wjw6juoyihIeSu7SJUwj2WL0OA/z6FhN5WS41PJXugw0kg2iSqPQfXfC3HI5Sk3lVNUTTkru0iXMI9lCmDDhjj6XtwvL6DgKTeVUshpempaRLqU2zxvU8nZxEubzFqVOI3fpEvaRbCFEYXQcZqV2tBclGrlLF41kZbBK7WgvSjRylx40kpXBKMWjvajQyF1kmMJybUAQdLQXXhq5iwyDqkV0tBdWGrmLDIOqRSSslNxFhkHVIhJWSu4iw6BqEQkrJXeRYVAPGAkrJXeRYVC1iISVqmVEhknVIhJGGrmLiMSQkruISAwpuUsslfJVoyKg5C4xFKYVpYZCX0ySD0ruUlBBJKooXzUa9S8mCY+syd3MzjCzFjN7w8y2mtn1fWxjZvZzM9tuZpvN7LzChCtRElSiivJVo1H+YpJwyWXkfhS40d0nARcC3zezSb22uRKYmP6ZA/wyr1FKJAWVqKJ81WiUv5gkXLImd3ff7e5/TN/+O/AmMLbXZjOA33jKy8DJZnZa3qOVSAkqUUX5qtEofzFJuAxqzt3MqoDJwLpeT40F/tLt/rsc+wUgJSaoRBXlq0aL8cWkE7alIecrVM1sFPA74AZ3/9tQXszM5pCatqGyUiORuAtylZ6oXjWaibm19TY6Ov5MeXklEybckbe/Rf3nS4e5e/aNzEYCTcCz7r64j+cfBFa5+3+k728Dat19d3/7rKmp8Q0bNgw5cImGtrblBUtUMnhr11alT3D3VF4+nosu2lH8gGTQzGyju9dk2y7ryN3MDFgCvNlXYk9LAvPMbAVwAXBwoMQupSOqI+i40gnb0pHLtMzFwLXA62a2Kf3YrUAlgLs/ADwFTAe2A+8Ds/MfqogMV3l5ZT8jd02Txk3W5O7uqwHLso0D389XUCJSGEGeB5Hi0hWqIiUkypVEMjjq5y5SYnQepDRo5C4iEkNK7iIiMaTkLiISQ0ruIiIxpOQuIhJDSu4iIjGk5C4iEkNK7iIiMaTkLiISQ0ruIiIxpOQuIhJDSu4iIjGk5C4iEkNK7iIiMaTkLiISQ0ruIv1oa1vO2rVVrFpVxtq1VbS1LQ86JJGcabEOkT60tS3vsRxdR8dOtm2bA6CFLiQSNHIX6UNr62091hkF6Ox8n9bW2wKKSGRwlNxF+tDR8edBPS4SNkruIn0oL68c1OMiYaPkLtKHCRPuoKzs4z0eKyv7OBMm3BFQRCKDo+Qu0ocxY67hrLMeorx8PGCUl4/nrLMe0slUiYys1TJmthSoA/a6+2f7eL4WaADeST/0e3dfmM8gRYIwZsw1SuYSWbmUQv4f4H7gNwNs86K71+UlIhERGbas0zLu/p/AgSLEIiIieZKvOfeLzOw1M3vazM7J0z5FRGSI8nGF6h+B8e5+yMymA08AE/va0MzmAHMAKitVUiYiUijDHrm7+9/c/VD69lPASDOr6Gfbh9y9xt1rRo8ePdyXFhGRfgx75G5mnwLa3N3N7HxSXxj7s/13Gzdu3GdmO7NsVgHsG26MBRDWuECxDVVYYwtrXKDYhmq4sY3PZaNcSiH/A6gFKszsXeDfgJEA7v4A8FVgrpkdBQ4Ds9zds+3X3bMO3c1sg7vXZNuu2MIaFyi2oQprbGGNCxTbUBUrtqzJ3d3/Z5bn7ydVKikiIiGhK1RFRGIo7Mn9oaAD6EdY4wLFNlRhjS2scYFiG6qixGY5TI+LiEjEhH3kLiIiQxB4cjezK8xsm5ltN7Ob+3i+3Mx+m35+nZlVhSi268zsPTPblP75TpHiWmpme81sSz/Pm5n9PB33ZjM7rxhx5RhbrZkd7PaeLShSXGeYWYuZvWFmW83s+j62CeR9yzG2oN63j5nZK+kr0Lea2U/62CaQz2iOsQXyGU2/9ggze9XMmvp4rvDvmbsH9gOMAN4GJgDHA68Bk3pt8z3ggfTtWcBvQxTbdcD9Abxv/x04D9jSz/PTgacBAy4E1oUotlqgKYD37DTgvPTtE4H/18f/z0DetxxjC+p9M2BU+vZIYB1wYa9tgvqM5hJbIJ/R9Gv/CPi/ff1/K8Z7FvTI/Xxgu7u3uvsHwApgRq9tZgC/Tt9+DPiCmVlIYguEZ2/mNgP4jae8DJxsZqeFJLZAuPtud/9j+vbfgTeBsb02C+R9yzG2QKTfi0PpuyPTP71P1AXyGc0xtkCY2TjgKuDhfjYp+HsWdHIfC/yl2/13OfYfddc27n4UOAicEpLYAL6SPoR/zMzOKEJcucg19qAE2mgufQg8mdRIr7vA37cBYoOA3rf09MImYC/wnLv3+74V+TOaS2wQzGf0Z8BNQGc/zxf8PQs6uUddI1Dl7ucCz/HRN7H0L9No7nPAL0g1misaMxsF/A64wd3/VszXziZLbIG9b+7+obt/HhgHnG9mxyzaE5QcYiv6Z9TMMosbbSz0aw0k6OS+C+j+TTou/Vif25jZccBJ5NC7phixuft+d+9I330YqC5CXLnI5X0NhA+i0Vy+mdlIUslzubv/vo9NAnvfssUW5PvWLYZ2oAW4otdTQX1Gs8YW0Gf0YiBhZjtITedeZmbLem1T8Pcs6OS+HphoZv/NzI4ndWIh2WubJPC/0re/CjR7+ixE0LH1mo9NkJorDYMk8M109ceFwEF33x10UJBqNJeZW7RBNJrLw+sasAR4090X97NZIO9bLrEF+L6NNrOT07fYcHUuAAAA20lEQVRPAC4H/tRrs0A+o7nEFsRn1N1vcfdx7l5FKm80u/s3em1W8PcsH/3ch8zdj5rZPOBZUtUpS919q5ktBDa4e5LUP/pHzGw7qRN1s0IU27+YWQI4mo7tumLEZtmbuT1FqvJjO/A+MLsYceUY25AazeXBxcC1wOvpOVqAW4HKbrEF9b7lEltQ79tpwK/NbASpL5RH3b0pDJ/RHGML5DPal2K/Z7pCVUQkhoKelhERkQJQchcRiSEldxGRGFJyFxGJISV3EZEYUnIXEYkhJXcRkRhSchcRiaH/D/9hoFvdgDBDAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Decision boundary with C=100. All examples are classified correctly but the boundary does not seem a natural fit.\n", "clf = train_svm(X, y, C=100, kernel='linear')\n", "print(clf.predict(X[0].reshape(1, 2)), X.shape, y.shape)\n", "plot_svm_decision_boundary(clf)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## SVM with Gaussian Kernels\n", "---\n", "To find non-linear decision boundaries with the SVM, we need to first im- plement a Gaussian kernel. You can think of the Gaussian kernel as a sim- ilarity function that measures the “distance” between a pair of examples, $(x^{(i)},x^{(j)})$. The Gaussian kernel is also parameterized by a bandwidth parameter, $\\sigma$, which determines how fast the similarity metric decreases (to 0) as the examples are further apart.\n", "\n", "The Gaussian Kernel is defined as:\n", "\n", "$$K_{gaussian}(x^{(i)}, x^{(j)}) = \\exp\\left(-\\frac{\\left\\lVert x^{(i)} - x^{(j)} \\right\\rVert ^ 2}{2\\sigma^2}\\right) = \\exp\\left(-\\frac{\\displaystyle\\sum_{k=1}^{n}(x_k^{(i)} - x_k^{(j)})^2}{2\\sigma^2}\\right)$$.\n", "\n", "**Exercise**: Implement the Gaussian kernal function." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics.pairwise import euclidean_distances\n", "\n", "def gaussian_wrapper(sigma):\n", " def gaussian_kernel(x1, x2):\n", " # Placeholder. Remove this line.\n", " return x1 @ x2.T\n", " \n", " return gaussian_kernel" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following code should return approximately $0.324652$." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[7]])" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x1 = np.array([[1, 2, 1]])\n", "x2 = np.array([[0, 4, -1]])\n", "gaussian_wrapper(2)(x1, x2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Dataset 2" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "data = sio.loadmat(\"ex6data2.mat\")\n", "X = data[\"X\"]\n", "y = data[\"y\"].flatten()" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJztvX2UVdWZJ/zbVZcqKQw4FgmGTsSQBqxM0olGktgtGeot7bYIdoR0m0pLhU66p0hlemTeijphsTAdWC4atOodXN0afbNiK9BdkgkYk8CbjtZFsDU9RQbzWYLKGGNsaMU2HVsD3mK/fzy1Ofvsuz/POfejbp3fWqzi3nvOPh9772c/+/d8Mc45cuTIkSNHY6Gp1jeQI0eOHDmyRy7cc+TIkaMBkQv3HDly5GhA5MI9R44cORoQuXDPkSNHjgZELtxz5MiRowGRC/ccOXLkaEDkwj1Hjhw5GhC5cM+RI0eOBkShVheePXs2v+iii2p1+Rw5cuSYlPjBD37wMuf8ra7jaibcL7roIhw6dKhWl8+RI0eOSQnG2M99jstpmRw5cuRoQDiFO2Psa4yxf2GM/cTwO2OM3cEYe4Yx9iPG2KXZ32aOHDly5AiBj+b+twCutvzeDWDBxL8+AHelv60cOXLkyJEGTuHOOT8A4BXLIR8HcD8nfB/AeYyxt2d1gzly5MiRIxxZcO6/BeAX0ucXJr4rA2OsjzF2iDF26KWXXsrg0jly5MiRQ4eqGlQ55/dwzi/jnF/21rc6PXlyTDJs3QoUi/HvikX6PkeOHNVFFsL9lwDeKX1+x8R3OaYYFi8GrrsuEvDFIn1evLi295UjX3inIrIQ7g8B+PSE18xHAPyKc/7PGbSbY5KhsxPYtYsE+i230N9du+j7HLVFvvBOPTiDmBhjfw9gKYDZjLEXAHwJwDQA4Jx/BcBeAMsAPAPgdQCfqdTN5qh/dHYC/f3Apk3Ahg25YK8XyAtvfz9w113AypXlxxWLwOgocPPN1b/HHNnCKdw5559y/M4B/JfM7ijHpEaxSIJjwwb629mZC/h6gW7hlXdXQpvftavWd5ojC+QRqjkygywcNm6MNEWV681RG6gLL5DTaI2MXLjnyAyjo3HhIKiA0VH7eWmMfbmh0A+mhReItPn+/lywNxJy4Z6j5lCNfWvWACtWxI19JoE9lQ2FIQubaeEdHo5r8/kuq4HAOa/Jvw9+8IM8R2NhZITz2bPpr+6zCVu2cD44SMdu2MD5rFmcz5jBeV+fXzvi9w0b/K7XKEj6vrM6P0dtAOAQ95CxuXDPYcWWLeWTfWSEvtf9PjJCwrmry19QCKHS20sjsreX85kzqR1fgb1hA527YUPY87ngev5aI83CVu/PlkOPXLjnyAQjI5y3tZFmLT7Pnk2fhXBQtb/p08MF7eAg54xxvmQJ/R0cLBfYsjAS/x8ZIQ1fLA5tbdlpnvKOQrQ5OEi7inoSomkWtlzATz7kwj1HGZJOZCF4e3sjwa7bzm/YEK5xy+fLmvusWdSW3I68kIgdQlsb/RX3pN5bmnchL2Ti/sTC4wN5AZLvWyxGWSxCou2uLnoP6i7KtMMSv6v3klMz9Y9cuOcoQxqOVQjeJUv05wjtUdack3LuM2eaOXd1ITnnnGjRka/rWrB834X4fsmSaOHxhW4xmj69XAir78J3AVbbFwuruqC4nneq2iwmK3Lh3sBIs5VOMpFdAk7WHmfONGuPrvbFeX199nZkGkL+f+h78X0XroXN59k2bPCjq0IW4O7u+C5iZIQWxd/+bftCpXteH2rH5/3mNE/lkQv3BkZaL4cQjtZFTaS9F86Taas6CshEF9nuxfUuXJSUz3N0dUW7Gp9F1XfRSWLv0D1vmuupx2cxHnLYkQv3BkfSrXToeS6jYjU1NR3NITR8dRHSPV+oZ8/IiN2YbLs/uf22Nvqn3muaRUe9po+9Q9f3ocLYZ/zkNE9lkQv3BoUsoIQA6O0No2RCtaq0AjyrBcDkLSPaEf83CUafxcG2GPjctyzYRPt9ffT9hz4U2SREO4ODRK+Y2ghxA21ri66lPqvJcCqOD3lGn4WnUq6pOXLh3rBw0SQ2hPqsq7+nvWfXopLF9V2CUfze1UXGWJWz7utL/7xCsHV1xa8vKJ7ly+Of1XtIokkLe4e8y5KFukmId3eHvfOREdq12YzYueZeWeTCvYHh4oGTCkkfwZKkbdUbxkRtqNqkqn1m5QEj73hCDLk+cAk2nT+/eEcqzSV/9qWBVGpqxozyhV8niH0WE7V9nftp0t1hDn/kwr2BsWVL5MEhvERkYTkyEtEB4rPMHQvohIav5hsyecUxsh+7TYAIl0GhWfsK+RDfddUIK3PWSQWT77sRXkdLloSf6/u8pgVM166vpq0uPKJPZffX3Fum8siFewPDJKDkiacGAYV4kgjvDtWjIum2e8sWzvv74xprf795wgvB1NISboiUr6nTmnV++PJCaXq2rNwATZq7fD9p6QzbAmZqNwlHnvPqtUEu3OsYWfip27bi4nd18vkID1lznjWLtH/dwqBbAEwQAu2qq+icq64y2wnke2xr40EuhK73pKMohMCXffR1hmqbZu3bnyrHruPc0wpM033KC5jpnCSxD+KcJIbZHMmQC/c6Rhpe0uYxohrydBPWJjzk++jrI6HX2krHC3qkr0+fGsAm4Fyauy5Mf3CQrt3WxnlTU/k99/VFtJN6PfV5fCgmsaiJdAY6Q7WpPd/+VIOOOI97y2Shueu4e7GA6XLvuO7dtAOS8+uMjKSntHL4IxfudQ4x+F05QXzakIXT9Ok08WTDoE27tyXjEprzpZdSe21t7rZ1k1vVHk08sND+ZI12cJDzadPoPCFQQoSJbTFTBZe47iWX6J9J1566MAkXSHXhCelL8dnXTmJrz2b81L0D9Roh9yU8c3IvmcoiF+6TAEJQTJ+eXOMRgk7mpoV2rQouVduyCeXBwajd6dMjDV517xPnyBkidXy1y1tGPlfQJ6omf8458dwsvpp5iG1ApS90gk5uT/awkW0FsnBPyteb+ixkbLjcFl3w9XQSyIqHzw2zZjSccG+0zlY1PZXmCIHLAMq5/f2pQksW+GKyurxcBGzh7aFatnrPMu1ku57rmlu2lHPEIyPkf27i903tDQ7Go1BbWtL7rsvPLveN7AFlQ5pANxVi8VJz7Yv7MHnQhKZFVpHknU0VNJxwz3rbKlCLRcP0LEk0nlDt1ARdAi6V6hEC0WaMTeppkkQL133n4y+u0jpCw7XlzLE9w+BgtACKMWlaHHz7Sb6+HIHq07/qriw0VbHa1owZ/KzrpkyP6a5lS70cOteyGtuNhoYT7pzbNUz595BBUAsNQZfjZObMsOpF4rws7t0mOH0Cikz5Z0KFkfoctlB6E+fuOybEOxe0kxqtKo5xRXBu2UIaf0sLP0uxDQ7qI11DKYu+PhKk06dHtJuNEpERkvDMBvGeBC3X2qqP7D3nnMhWoS6q8nGh4zV3tyxHQwp3zpO597lQSw0hjYDOMlzfJFh92hbanW+CLfXeTXli1NB5+V5t3jI+/Snz67KPuymoy9Q/QojOmBHZC3RacpIxJu8INmzQu06aYLMfhIwbsXCLbJPTp5Onkxov4Ft9K+Q95Jq7HpkKdwBXAzgC4BkAX9T8Pg/AIwB+BGA/gHe42sxCcxedncXqXisNoda2hKyun4Z20H1O2q6Aqz+F4Gxt1XsY+d6D0K5FNka1uHfIs6pIo7nb7jnkfoTmLj+fnMNGjuz17SOfuVaLHfVkQWbCHUAzgGcBzAfQAuCHAN6jHPN1AKsn/v9/AdjuajcLzl3ebk5WzV2Fj0abJvy+kghdIIXG7/LmkA2pOkOoSoHo+lPl4EUVJ1HJSeWSBcR58rOptIztd/kY9dlt/SKP96Scu00w+o551fNqZCRy9RT31dqq/912b67r1nos1zOyFO6XA/iu9HkdgHXKMT8F8M6J/zMA/+ZqN623jAiMkSekL8+rs/CbCj5XG/L1Za1JpSVc5+o+65B2EqXx6uA87oVhewbRrqxhC0Gi87s3KQHi2T72sXIvEB1PLl9D1lKTCMoQZOUtI99jEhuAqS1BmV16KT8b5CZ+M92nbXzmwtwfWQr3PwLwVelzL4C/Vo75OwBrJ/6/EgAH0G5rN62fu9D6fHlek0AXUYM2zbHakCd0qJvkyEgUjejzTGm3v6qA1Qlc13MKbw5XkishOFSqQl30TUJCFcI+uz4TLeFaTJIIeJOhXVdLNi3SLEjqfajG2yTaeJbvsdFRbeE+F8BuAIcBbAPwAoDzNG31ATgE4NCFF16Y+iFDBmjSAWlDUm3D5zxZqwqJsuQ8yuFi0oZVpNU8xdZd9bbxpR04L9fgTc82MhL59Sexkch+3z7CRCz+ch/IKQOy1DjVe9AFo/mkXQi5js7TydWe7pltuWtC76se6NF6RlVpGeX4cwG84Go3qwjVEJ5XHTwhAzLLCEKXlhKiuZsWLZGcy3fxSmtQDj1fDiSSNfjWVjv1IHYGJs3dBvm9hqRAdtEyWcIl4LLQcF20pPxufOw+YhedJhJWIHd9dCNL4V4AcAzAuySD6n9UjpkNoGni/7cC2OhqNwvhrpsILk1K1dySeHfI2o4cKOJr8DLdu3odmRKQhaBpwqvBKjptOORefJH0fJXCUT+nPV53j0kWYtEHXV363DpZU3guAZemv0xKihyZqxbfsJUjlBcH3V/XvVUqwrWRkbUr5DIARye8ZtZPfLcRwB/yiLp5euKYrwJodbWZRrgL4Spz7uKzamRVhaVOCIZq3PL5W7boaRAfjxZdaHhSbxm5LfVebRMlrSaY9vwQSseUNsCnNJ5OqPlQHKo3jAhSqpTx3Vdwh2i4Pg4EOp942ZBrcnU0Ceek0bShi8NURMMGMXEebR37++MDob+fMgiK72VNRC4QnIRnFFA1fzlHuWlQ6iaACARJExouvw+dodBH4GblLZP0fM5rtxX3XZjk95s2D1DW95OEBjRFr9riR3z7KElf9vVFLqnyAp9FPdtGREMLd87LtWh1oKrpZUMiLl3uX2ISqILdJKjVydXf714QQt+DLBBcdTNrjbTaXpZwCUrd+02aB8gFn92FoIlCPWjU53SlX04SpJSULhoZiUe4VmJH1EhoeOHOeaQliIRGYrLZSpn5QDehVW5XRDjKgt2WfU8e+DNmRAuDGhouw0crnoz+wZXaiid9FyHeSEK4huYBSgp1LOo8aJLuOsVY1BUQkXPnz5pFhu6PfaycjxfadRp6TlwjiZF8KqLhhbvQUFWvkI98JB512NubrniC0EJkzV/lLX3dKkONuWn57HqG6L8sPCzkNkPfV4i2Gdr+8eM7+OOPz+PFIuOPPz6PHznSH/t8/PgOr/NGRnakMnjL9+pjb9LZfUQ+/VmzIppTNvYntRXJ1w+NxJ2qaGjhLvPWomSb+AwQ7y4HnbS1hQt3zs0aXZII15DJpTuvEX1/K8G1V1JYh+wMjh/fwR99tI0XizD+e/TRtjIBrzvv0Ufb+F//9Y7E70p+rjT2JrFrEZk0TS6hIe81bYTzVERDC3cxIOSBKjjb/n7SMHz9kn35ddf5qsBXv9dNLtnP26bhNKLvb9pFyyZofd9XJSkt0rzNgl38e/zxeV7n7do1ryLvKhTi3fom/vJxU7Zl/6x3irEWaGjhrkKezKrrWm+vXXjqtAyxIISkmk2Snlb24OE8XrKNc7pP4VUje8IkGezqVt9ECVSjrSzoJlMbgi820T3VslEUi8xLuBeLzOu8kRGmfe40CH0Xvpq7gLrImvrMlre/0XarWWDKCHedhqCjQGzbUbWNJEUixDFqPVMf2kW4RKoBSx/5CPVQfz8d65PPWyd4TVv9JAI+i7ZsQiVE4Kj9ZjLQqkbAatgxstbc5eOyWoxC3oWs9Mhj3UXNmIL0XN5JjUhDZoUpIdx1g1Pn1SKH45u4btNW3rZ46AagbBTyHaCy8Vc8g6gOpPPZN01sk+A9eLDdIFyag7VvH+GTBi5tzkZZ2VwsdQbxSrpfZs25i+NMu6akuylfYeoTQKby5yYN3EWbNSINmSWmhHD3jTpUfXpDtQndYNN9J7cTWplG9tdXz/Ud7L7aoq+g0cFMNxBtkAX947sDCuk3+fzQqOKkyMpbRhbgOsF/5Eh/qt1UVsJU7EKXL4/3mVx2MNfc02NKCHcfqIPFFGZt2pr6au6qtiK0b52rmICaf17cW2trOdXgM9j9eV4/ikAHm+aeJf3j2gGpwtpUw1XXRz5RxfUI8+LdnLg/XcI0lJe31W71mWty0jJ5cQgJNmx0A2wu3Hn54NH5o9sGiGrgHBkhN8uWlnIq6EMfsm9JdXy/mo9eFCI+5xz6rHLsLn7YNPkPHGh3UgQ6454ONgGeFWXjswOyuaPa3ptYQF1RxfWI8MXb3p8+nLvPMSrEO16yJH6syxe+r4+0fnXOmipk6Rb0qZBwzFe4N6GBMToK7NoFdHYCxSKweTNw++3Ae99L3193HbB4Mf0uo7MTuPlm+j9j5e1yHv986hTw1rfSeeKa4vp3302fS6XomrfcQn+/9S1gYIDu7brrgJ4eYN8+YO9e+nzkCN1vqRTd165d1K4O8+ffiqamtth3TU1tWLhwGxYtugetrfNAhbKatee3tl5ofpkTmDPn+lhbra3zsGjRPZgz53qcOvW89hzT9zqIdyH6raUFmD4duOMO+k3g5pujfhPvZfNm4LXX6P3efju9W/n34WFg926gtxc4fBjo7ga2bwdWrYrecVKcOLETTzxxEfbvb8ITT1yEEyd2Zno84Nc/IcfL82PrVvpOHl/FYnSMPG7FOToUizSGlywBDh6kdyyOlfts8WJqS3xfLFLfDAzEr7d5M80T9XqLF9Nv69bRsZ/+NHDjjcDGjeZ7m3LwWQEq8a9atIyAawsX4u+ubmVDEnXp+M0st5c+nLfe2DdNMrw2x6gWX2Shufsa5XQQ77arS0+BqSUZ0xZrEQilo5LSVz5G2qR0mElDF0ZU1X5h816SY09suyIbJeSTEkKcL9KPXHqp9+NOaiCnZfRwCXGdgPYxqPoYgurJWCQvAgcOtPP9+1tSC4ksOfckPtji3ercVW1crg/VYINpUTt4sF270IYsgseP74h5O+3fP4MXi00GoR59f+BAe+L4A1WR8S1WEkKVqPEo8qLhmic6mq1QCMtJM5n5+ly4G2DjEE2D29cVUpR/Uz1o0iZWkpHEG8V1jsvLxiR0snTJSwPdu/XNQS63kXRi+3LhYqFzeRwJHD++w7joVkJ751yvtNhqyKrwzblvatd3FyxsVsIBQXeuDVnNx1rAV7gzOrb6uOyyy/ihQ4dqcm3B6/b3A3fdFecdf/IT4mE3bKDvVqwAPvlJ4s6LRfrMOfDggxGXf911xP19+cvE3RYK9DsQcZSjo+X8vuA0Bb/vwokTO3HkSB/OnHn97HdNTW1nOe+k5+zf3wSqaW4Cw9KlZ1LdRyWwdSu9U/ndine6eDFw663AI49QX27cWLn7eOKJi3Dq1M+9jiVbBbTHt7bOw+WXP5eoXdv15DZt0M0L8S6LRWDTpmhe2MatajdRP4tjVqwAfvMb+u2b3yT71ic/CSxaRPNItK+bJ8Ui8Pu/T8eJexLzUD439HknA1/PGPsB5/wy13ENbVA1obOTOnTTJvorOrRQAHbsIIPbXXcBg4MkyHt6ovM++Un6LBvz1q2jAfbgg8C3v02D9GMfo8ErBoxsTJLvw1ewA8CxY+tjAhUAzpx5HceOrS87VhjsxsZWOc9xGd7U3033MTa22sswmBWEUU4W7OIzAPzwh9Qvd90VN8ZmDZ0h24RTp543Gr7nz7+17Ni08G1DFsAbN8YdDgB6h+JdAvZxKwzYNiPs6CiwZw/9tn078PGP02eADKXiuqI99XrDw8CMGfF7Eo4LvnPKJAcaBQ0j3LduLZ/AxWLkBaB+Lw/WYjHuTbNvH1n59+4FvvSleKfffTf9k1EqkVDv7KR/N9wAvPEGcNll2Q4YX28UoVnbtD75HJtwChM64zhypK9qAt4kRAC9oKqUgNd5EDU3t2uPbW29UHv8BResxrFj62PeM7ZFt1Bo976eD2TPGSDuYZTkXboEpxDA+/aRMrVjB3DvveQx49KghWfNnj3xe5Lb9YEsBwYHgaGh8t918mOyoGGEu9DixKBbs4Y0Z1kDKBbpe91gHR6m/w8M0GAMcZGTtXJ5wPzwh9kKFNNE9dGsbefEhQ0gXCVlN0ef+wD0O4kkbn++0AkRk6AyuZBmgTlzrsfllz+HpUvP4PLLn8PChdus2rl8/Pz5t+L48fsmFmOOU6d+jiNH+tDevgyMtWiuNg0LFmwLup4KtU9Wr96p3Vm++93J3qVOgVJ/F/Pw/vtprm3fHnedNCGL/lV3Kps2kSulEPDqLlCGrEiK/8sLQb0sCpOecxe8q7wt7+4Gvv518pFWufGVK+O0ChDn9NLwcMUisHw5DRTZfz2EB7TBl+t2cegh/PiJEztx7Nh6nDr1PFpbL8T8+bfiV7/6R7z44l2WsyKOPkt+Xu5rgaEh0toHBuqPN9W9O90zm7j11tZ5mD//Vjz99FqUSicBAM3N7Vi4cJu2Hd31AGi/S9Inuvdv4sNdnLtsLykUaNfc3U0a+caN2cwX271v3UrXla/jO5bk5wGAZcuApibavQPZznkdfDn3Se8to1q51SRcIa6HLl9f9VhdWbxKR83ZvFFsUaKy50ta18YDB0yJyMq9a7JMNKb2jy0S1dcjptoxBvrj7RGmST2QQhPJufrE18Oku1tfuq+7u7xNtQ99Mp8mge+9++bZkWVLW5s55UIlgKnkLSNr7Dt20BZv3z5aWWUrv/CYMGkgt90G3HRT5DkjtmQislGsxoICcq3sSbR/X21Pd56qjclIoi0n8dbw98SJe+D4Qn63g4PRLkn+XXh42LRHuX9lLUz0degOwHeHEvXvz0HRwrb514y5c/tw/Ph9iXY+4f3n7hOfse2juQsIDXrz5qjNSmm9rnsPnbe33BLJlueeI1ppyRJgbKyyO0hfzb0hhDtA4cfbt5Nx5v77426La9fGO8vXVUv+bmiIODl54VC3mHJnivsJccNLQ2HYJrLY3gPlW3Rbu24XSRXN6Oi4L9amjXLwddFTIU8q27u1TVZ1a71iBXD6NFF5e/aET0yf53QtwHroFwCf9xfaf7594vP+0wjKSrqtmq4TsiDJx/f3A9u2kYfc7/wOpVwQMqhSmFKukMJ63ttLglcYO4Qbo2rl93HVUo/ZvDky+qjWf9WYOzQUd6n0NaqGuDqqMHuwsLMTNvKgiYx2snFTNbIVCudrWywU2rXGO1WwA+Z8NyZDnwsuQ50Mm8eG3L/FIgn2N94gT6ckGpePJ5OPobsceuHs4+JoMnw3N+v7z6dPfN9/iJthSJ+mge06IUZaVfAzBrz5JvCjH0WeP6rnTS0w6YW7eNHf+hatlrL3y4MPRm6Lamf5DD75mO5uWjh0A0MWFCKB0e23x+/HZ8D6CAiT54nLk8bsm74KTzxxEY4e/XyZ8C+V/q3MW6OpqQ0LFmwzJg9TYUs0FgrVw2HlSuDaa+PvVvVasAkNuX8ZSy5cqA/0U0nulyz81gWam6OF1zQmfBPJ+faJyRde9758BXZIm0CYy3PIdULiUOSFYHSUlL9p0ygG5v77ae7fcktlYyt84CXcGWNXM8aOMMaeYYx9UfP7hYyxImPsMGPsR4yxZdnfqh6mFfcXv7Cf5zP4xDFiNV63zjwAhaAQLpRqRkIfNy2XgI77r8e1b5eGbBMsp079HC+++BWNVvkmmpreohUCquufTTCEHGuD2tc9PSSUh4fps+y+5iM0ikXaUre1Ee+ravM+EH0CjJf9pmrDbp9z5ndRAGfO/BonTuy0jok5c67HBResRpQFtBkXXLA6uP8EfLXbEIEd6tao7pJtLotJruOzeMgLwc03Aw8/TMJcKJIDA/Tct91mv6dKw8m5M8aaARwFcBWAFwCMAvgU5/xn0jH3ADjMOb+LMfYeAHs55xfZ2q10+gEbhwaEce6yu5Z8juwClsaIKuDi3M28ejOAM2CsDZy/AeAMhDFu4cI7AaQJZS9PPWDi7UOMwUkNxypM793ltqe6xgLx8eGbFsLWJypNpefciVNvbZ2H9vZlWuMpY00YH3+t7AquVAbz599akzQR8ruXDdfina5ZQ5/lYECdS6UJrrnm67Jpa9uXe096ThpkZlBljF0O4C85538w8XkdAHDON0vH3A3gGOd8y8Txg5zz37W1W43cMkknPhA2QLLs3BMnduLo0bUYHye/5kKhHQsWkF9zqIFMnsgnTuzE2Fhv0PmA2yAorgH4+05nnZvGZCSz9SGQPtcP4DJasuAFUPe7vd9MHjd07ayN2aHQzQ1TfqaQ+WIzwKadj0kUtSyUO19kKdz/CMDVnPM/n/jcC+DDnPO/kI55O4B/APAfAMwAcCXn/Ae2dquVOKwaVvg0moIKm+CLXOj8IU/k/ftd2/64oFAFrs0jBPBLhuVqJ1To+HrEVEqjOnhw9tmF2IS02rJ5d8BQKJx/NsBJRmvrvAkqLjs31KRQ+2jlSkoOltT9McQdM6mwTSI3quXxU21vmU8B+FvO+TsALAOwnTFW1jZjrI8xdogxduill17K6NJmhFrhly0rt3IPDdH3NmSRFEzA5jETkqBKQObao/QC5WhqasPcuZ+zGtlsBt+QKkxZVGwC3Nyuj1dUWugqdanw9XgygXh73YU4OIfR1uKbrsIXSY2ZnZ3A+98fOS/09JBgv+SSyFlBTRZmgi+fH+Kto7tGqPdOtTx+QuAj3H8J4J3S53dMfCfjzwDsAgDO+RMAzgEwW22Ic34P5/wyzvllb33rW5PdsSdCrfAAcOWV8fwSwrf9yivDr590Ipg081Onfl7meWIqlydDnsimxaFQaMeiRfdg4cI7rUY2m7AIESRm4cKDcs+YjGS33RYX8GKSv//92W+VS6VXvI5L4ylD/aDfYY+Pv2L0fMnaDTWpMbNYBA4disolAsAf/zHwve8B73tf5Kzg0ze33RY/trOTPqvGy6TCNoncSHJONeAj3EcBLGCMvYuRX1wPgIeUY56VpqKMAAAgAElEQVQH0AUAjLEOkHCvvGpuQZLkQgMD5MZ0443ARz8auTTKEZACLuGddCKYBTZ9L3s5dHTcZ9Xk5YksuFzaFUSJwTo6duCKK172ogxswiJEkNh2IDr/exNMO6abborevewRMzqa/YTz1YJNx/kmVTPtumzXz9INFUi2ExLjfs8e4DvfIa69u5sE7lVXAT/+MXmXbd7s1zfvfCfVTZDn1Ze/TN+r10wibJPIjVokqvOBV4TqhGvj/wBJha9xzm9ljG0E5Th4aMJD5v8FcC5IxbiZc/4PtjZrWazDhY9+lCLNliwBDhzQH+PD5ybh/Wy8+NKl5X119Ojn8eKL94Bc8ZrA2HRw/nrMWJelATN7bxn9TiWt0U8Y7uSoUyB7asYn6jSpYVl+n8St/xuAN2PHXnDB6sTpCZIihFtW7VHi3AULgH/913DOXTbIrl0bRYfKUcVZ2sDqEVMu/UBWEFTMFVcAjz1m1twBP+EdamQJMTaSB8xnIE94YBo6Ou7NJAVAVu6KNmSde0bGlVeWV2KqxCRX31N7+zKcPLk3dTZIVfAz1oKmprdgfPyVs+2aFshKecSkMVSKc0UOKDG3TIZuk5AeHqbzX3+ddmSilsJUwZRKP5AVhGC//XbS2AVFYwolHh2lgSobbXwjJEOjCnX0xtGjaxEX7ADw5sT3EZIYMG3BMabjk+Rsz9roJ1As6isxJTV0h2DWrN+z2i7EuzLbV57XGtY5P41C4dxYu7a+zTqPfhq6Qz73ve+luSWoGBONYaI2Fy0izR2gv8PDyexbWSOpna1SaFjhrnvRa9ZEARQC8st/+OG4pj4wQOXyHnhAf45alm9oyC9C0hVV6MuTmlzw1O9tAtQkAEJL+oUsBDKyNvoBwIc/DFxzTfzdX3MNfZ8W6vvSpW1wLYKuKlnkn+63IJvzx5yfuE9MSMMty+fefDPNLTUdiLro6jh+Uau4pYUW7pYWYOfOeBoKl30rjRC2nZvczlYZNKxwl1/01q0keIeHSVALQ5uoyiRe/t695RTMwABw7Fh5h4mIVbks3403RpZ820RwCc6swvUFTAK0vX2ZUQCEaPtpEp5lbfQDgLe9jbbshw/T58OH6fPb3pa4SQD6RUyXtsH27K7kYaFujKa+ZQyJ+8SENC6/Sc9VXRqPHIk49o0b6W+hQHYyX0NvGiFsO7carrchaGjOXcfxXXIJrfKlEg0KESXn046uKnxnZ8Sr9/bSltM1YLPimR97bLY2gKVQaMcVV7wc+07Hn9v4WsA/KMkWpdnRsaOioe4mhNhOfBGWwkHfl7Z3JVdZCjGCh0a16gzz9QpdAJSpktprr/nbt7KwHZjOrXQw05Tj3HXbJYB8m0UyL8HxlUqkya1d69ehuoAIoYnIvPq+fX6rf1Y884IF28qyNjLWggULtgGIUwgiCErsBgCbT/3zQXSJKTUwgKoVzFb7f2CAFtqDB0nApxXsQKivepP2ue1Fr889K7jFjqZQiApfMzZde55up2e+DsusP2wURRb8s47a3L27/LjOTpp3IX7taYKcbOfWUzDTpBTuuoFTKFD9Unm7tGIFrehC8ArjJ+f+L19QOnKHDQ1F95DEwBQiOG1GsTlzrsfFF38tRmlcfPHXYpqfjnKJMhnq0dp6oRddcuLETuPuQSAtFWCDPA7Edln0zec/Tz7U73sfae5Z5Ne2CcxyjGsXNpstQbd4nDnzRtTi+MkgW4YpqjWr/rBRFFnwz5XIQimfk1QIm86tt2CmSUnLqK5Ta9YQly6KavT302QeHycevbMz2qa3tADnnOPv9yx70AwMxD+LkntJ/Gl93AwrUZnJRrmEtB9WVagyuUzUcSD65tJLgR/8gMbBnXeW92FSmPrjggtWS7EGcehoLNOCqB6bNgePOWYiu/6wURRpqI8QhPq1+8SomGA7V6Zqfe4jKRrez10eOHfcEWWZKxZJOy8UgC1b4n60H/kIMHcucXa+6V3T1nhM4yvu8oO2tWvj9QnpOPIQ/rmSWQhVAdLdTTTcVVcB/yCF0Q0NkTfU3r3prmfqTxuXrnLc+oVxGgqFmSiVXvHgzf2EcyVKHKrYuhX4yU/iJSVlgVatZFohSBPkVA8BUg0v3IH4wOnsjAylnMdrYbryS/t0jm6Qujo6bWSoTWA0NbVZ202quYuFw3V/vumH00RL+k4k2aC9b1910q6qsGVu7OjYXkZnyWmdGZsB4E1wfvrsMeTxMl3r7moSzrpgqkpHr6q1hdeti+oeANVLgzuV0PAGVZX3Onw4MpTeeCMJdsF3yW5YggsEosIBy5aRdi63vWaNPhhpcDDib1WeV+UV07gIAnbjm6vd0DwwAr7+0HbjL5s4Jp1bow9vK/rGp1pWJeHLcYsFXxbanL8eE+wA9Sdj5oyPoi1hj3nssdkYG/tMzMZy/Ph9mDnzcugqMWWBYtHsDgxExVBkF0HhhlyrwJ6phEmpuet4r2XLgKYm4AtfiLQEwF5cQ1A6pRLQ3Ey0DhAvJgDoeV2Vgxeai6ydpHV5DOO1CSKPtysMPm1uF9O9ye58WcDG2wpby549UbWsjRspgvGv/oqOCdmdpYUPxx3qTtnRsV1LA/mPDQZbjv40kHdWqjswEC3CMgU6PEweL7kWnxwNrbmrVnSAaJhVq+JaG6CfzLIr0w03UHFtxigadfnyeJUY+Vpbt5KfvCiAe8stdL0PfIA4R9UtKq3Lo/BY8UntS2Blmtv8+beio2M7AGBsrPesx41wnzPV7XS5/em8aTo6dmDJEr8Mk75wuawJ3eTmm6lvOAfe+tZ4/1crUtAnc2OIO6XwWlLdHCmn0GrPRT+uXGTpvWRzBxa/qVp7Ltirh0mpuauwcbP791MCKdlL4vOfB776VeCLX4y0QWGIBczGH3nHII5vaaE81TfcUK5ZZpWN0Y/fZtpjmpvbwfkbWg8P0urt3Hu1a6KqcHlcmH6vlqeGDJ/+9tXcGWs569Yqg7KAfgWh5RKV1jP3lnF5ntSjYXWywldzL7gOmAwwaeednbRVv/FG+m5ggAT7XXfRpN+4kY5ZsYKomenTSYPfti06X21z167IcNvSArz5JvGOAwP0uzywxcRMK/RMtTBFYWzz7/r8M2fOvG4VEHJqAiGoBBcPwOmyaTs2BKqgUN8vENfshWHd9n0lEe/vnwNoLksrYcr2yHkJVNicoFO6TpzYmYFgj+8kfF1yTcfYfNFll0jZPqabWzmyR0No7i7IoegHD0b+zwANvGuuIWG9bx99pxbwlVEsEnXz+utAVxdx/cI7QGgulUgpm5VG6EJcY/dzo6uUy52Pt0w9ae4CNn/4aLfUDGAcra3zUCq95uUVk0Ufq/niXeMq7e4zjU95Dj2mhCtkCEwFOLZuBZ59Np6volgkw8+7310upNesod/WrnUbbrOES8MyTcKmpunWCFIVwic7xBhcyZzsNugEx/LlwOrVwNe/HjeCb9hQvbzfNrdInXHT15/d1/3UBjmOwWdRTrtw14NfeKMhF+4S1CRSH/tYRKMI+Ay4rLWQtDy1T5EIAEEeN4VCOxYs2FYXmrsLOsExNES2lL/6q3gAW0jgmQm+/RUihEOStKXV3NX2fBblWi3cOcxoaG+ZEOgKcHznO+V5aHy8KbKslejKge4qtKA7X3jHyJ4VOq8WG0olyl/S3r4sVU3UtDnZfaBLIzswAHz3u0SVyWlXBwbSC3bf/OghCeBCkrS1ty8LuGPVC4o8qeSx5OPNValiKjkqj4bX3JctI28ZOQ/M0BD5Rz/1FH33wx9WnwO0abvt7cvKDGdxzvZ50Lrsl8vE99pqO/XgLZMUWXtnhJY/9N0tifPVqFWxg/Ipl2hrm47X00FA+a4ua849R/bIaRkFJn729ddr455l37rr3RrN38ePcW2XyU/anO/bt516RSWMqa7+sgeJ6fst1LgZSvdcfvlzzkUprbdMjuojp2UUqFVSVqwgN8la5V02b2ubYZ7A7oldKJzvrJs5Z871mDv3czAFMNnvr75RqbSr9vdRTtOI4COiwfT9JqcC8ElV4d8n7Cyl46qoJe5TF+gmkHVlsBzVwZQR7kDc9/n0aXJ1dAmAShW9NfGsOqrFF4y1oFR6NcYLP/XUZ7UCfuHCO9HRsR3Nze1lv/nw5VkXX84KWdpFZNjy8Qjooj9tEaknT+71KJb987PvWGcH0YOfFcA+nHmaGrg56hdTSriLYIquLgpAErAJgEoVvTUVw3AZPMvRjMhQOg3q4sD5aTz99FrjPSxZ8jI6OnYE1TCtZ2GQps6nDWp/mSALc3of5ikm3pubR48M5hdcsNprjIgF18dYmzbBXY76xJQR7vJ2/eGH41kjAbMAqGTRW912V68hMpx3Xpd2knZ03IelS89g/vxbwfm/a68T4ufug6kqDOT+cuWRiapd2XZizUFJ4c6ceR0nT+712kXIUcKuilohxdBzTB40jHB30Sejo5R+VEAI7eFhN8Ui0znd3eX+8VmnL21qimplNje3o6NjOz7wgYeNk9RVNs+GJFr4VBYGcRolrsHLGrFuASxHOAV36tTznm3TYjA2thpjY6tw6tQLoP59AWNjq4JdInNMPngJd8bY1YyxI4yxZxhjX9T8/v8wxp6c+HeUMfZq9rdqhw99smhR/JjDhykHuItiUXOGi3zuWWcbFIJW1rQ5j2pomgxbrsmu49UFkmjhNmHg459fj1y9D+ILIUCGUhLwqkbst9D5ZvuMQHmEQhbRce3fU6d+jrGxXhw9+nnjblH4xR89+vlJ22dTGU5XSMZYM4CjAK4C8AKAUQCf4pz/zHD8fwVwCef8s7Z2K+EK6VPPUVSKeec7gSefjNfV1EWpmup06vK3p4XZsBYlCNO5odld5Kaho+PeslQFwrXN5t5ncoW05U6xVf45cWInnnrqs7HCFKbsh/WIEF93X590taKW69hFi+6x5uEPBzvrKeNy3VTvYzL0WSMiS1fIDwF4hnN+jNOsHAbwccvxnwLw9363GQ4b/WLL/S1omM2bgY4O0tpbWigHuGhDp4Wr3hcDAyTYdfnb08KskY3DRpnY3Cp1gl2mYUywbclNxuCTJ/dadwFPP722rOKQzeBbbwiho3x4cX8jepyG82nbH1Qpysd1U2Aq2FcaAT7C/bcA/EL6/MLEd2VgjM0D8C4AI+lvTQ8b/aKmFlUXgc5O4sxFArHWVvJ3txlKVe+LYpE0dpd/fBIXSh+OUzexTB4RHR33lWlXPnytjyukjiIyu/OR8DMZdrM2+FYKIdx0fAEETPx8XKjq2p5XRsO52g6FvDj5Uj6nTv08FT1jo+cmM3VXT8jaoNoD4H9yzrWWIsZYH2PsEGPs0EsvvZToAibvFcAdvDI0RJx5by8wNgZ86Uvk726q8qMiJEAmiQuluQ5nHOoENGnSum2za/L6ukLS5GPYv7+A/fsZDh6cbWmzeoa5SgoGU24X0/fRAsjR0bHd2j+h+XlCNG0XfHLJ6JDU/dVmxPc18OcLgBs+xTp+CeCd0ud3THynQw+A/2JqiHN+D4B7AOLcPe+xDLpCDFu32osGFIu0GAiOvVgkrb2pifzefYoI+BQmkO9RCH/fMPg5c67H2Ngq5/OLKFQ1HNzFgUZ+1yYvDebMS1POt1Nbunzkok0hoJqb27XH2Qy+IahU0RCBkyf3Gr83heir33d0bNfei6uwi6598YxpoBbcLpVe8z5X7CJN9hyTjchlxDf9Zsp3k3U/Nwp8DKoFkEG1CyTURwH8Cef8p8pxFwP4/wC8i3skrElbIPuaa8i1UTZq2tL2qulhi0WqqNTTA9x9d6RZr1wZz+0ujk2afzo0gZXLEEdVeziAN89+52Pg8klmlVXSMRUiRzzltPlM7N51Bt+kCE09HJozxWa4Vg2jPgZmF2z5aXTjwAVdMrgoTbTbkKpHZHj3NZjb0ggT7CmGa5Viul6QmUGVU/2vvwDwXQBjAHZxzn/KGNvIGPtD6dAeAMM+gj0NhBDeuJEE+7p19HloyE57qNz56CilH7j7bvosNO0nnyShL9Mp115LBT2S3KvNBqCDyS0NoMHb1PQWqBOa/JlXWbenLq7dRAGo299QwS5zyXPmXI+OjnuVotrZCHYgzOApBJFPqgYBm+Fap22++OI9iYO99G6XEUiA+gt2sYOSbSXz59+K48fvM17DB/I78TWY22wXPnaNqRxnEYJJlxVS1sCFoO/uBr7xjWwq7Qi6hnOqtrRtG9VV3bMnrO00hT1sGqUrM6BJM7SdJzQ6V21Ugr92V22XuRCN7rHHZhsMufSefKtdJYM742a6whzTEBf8DHPnfg4LF96Z4hrTwBiLCe/yrJVme5HYvQH2NMKAOw2x6b6bm9uxZMnLns8Tx2TKfNmwWSFlDVxw79u3A1/4QjZuiZ2dJMhLJaJTSqVwwQ6kS2Bly8LnMniZNMNC4Xzt8YVCuzHTn17bjwJ3yjENhUI7QnLUuIxivoYzE19s2pGYPXTOQGjyY2O92L+fnb2uMFyntxFwHD36eesRSbVQsRuK7462lwl2uoafYBdtXnzx14JyEJlgcwDwcQ6g/pxW1u6ZM7/OzMCr9v1khI9BtW6hq6o+OppNzUaxoZE3NiH1IE15atIuQPPn3+rUHnWCwbRBs23czAKGS4UgokLPIcU82tuXxfhonVHM13Bm0qh1BS/CwLXXPXZsvcWA7IcXX7wLALRCFxCRqOG2lzDjumsXRr+XSq/h6afXolR6xWoUNhnMAdK25fGhGpCFQiLu3ZVTnp4/TktxfrrMwOsDsxIzuY21k05zFzC5JRYKdhdEl/+54NhbWmjRaGmJOPhKZYgMQbmPczl02v34+CvaY03f2zIaCppj6VKOpUtLWLqUW/N86zSjF1/8ipGPPnFiJw4enI2xsVVenLXJntDcfK7xnkK1b/m6WXG7L754j/E3l+3l4ou/Vqahh2jS9CxuwQ6QJxTtdOy5hxYu3AadRg2g7LyQnEa6Y01J8pL0jeucyRq0NWmFu4n2KJXsWRxdAnp4OOLYN26kv4zR95XMEBkCEaVIFEgcJhoiJADHntGwvBanCzbNSAVtiT9j1YzVyWgzsJloHZsgcl03O799c+IwHT3R0bE9tpCmKaJhE2i08JkFv0nYxQ3m9vNCchr5JkoDkvWNzzmT0Vg7aWkZF+2h+sHLx9j8z9/9bhLoghsXHPzoaETBmNquFkw0RHNzOxYu1NMQOjrHtBDYJ1P4djVsYjTB5QWiTkYThdHcfL6T1hFb/ebm83HmzK/LvD10150//9Yyl79ksCcO86FXfKHSGoXC+Vq7Q6HQ7hUxbOpTcc8mA744L8TjxXf8JC3K7kN1TsYMmZNWc7fBJw2BKQeNMNjKGr78uVAId28MhcuIaBK+jNFvuvOyjGIV8HHBBEInht2LRDeBTdGdjJkDYoC44XrJkpclg6H9unPmXD/hkupGU1MbzjuvS/vb3Ln+aZrTRGTqaI1S6d/AWEvsuKamNqsNRoarT107xZCdpOnYQqE9Rq/JqbJD4JsqYrKh4YS7T4oAH/9zHQUjMkpmXZ9Thg8XaRK+pdJJ63m+2/hQLcWVAz67RFfN2gXJtHCVSnp7gk3rNHHdcr1TwGyriN1tczsWLboHH/jAw5g7tx+Rpt6MuXP7jcZUFWkrX+mVgTfR1PSWsnfm81yAOeWCgCudgt7jZZpWiJraetvbroulxC6VTiZOiSD6PkrnQH2Vxiuo1ph0fu4u6Dxa1qyhv3Ik6rp1xM8LjdzEncsRpueea29bwOWdY/Op9fHVDvFPThK1l9S/3XYt9ZlPn35ZaxQjj4t/g0rNJEkNbHtPJu8eX195nz7IKmIybUSmLSJU9bcPHVshHlJqOgUdtWWiFnVtmVIfZzXm6zW1ccP6ubugq6HZ0wPs3h0JXaGBC0Ft8j9XNXxVsKtti3NsHjQuLcyHiwzRhOUCy74ajU4Tnjv3c16l3XyuNT7+Gjj/jeaXaVi4cBs6Ou6NbbcLhXa8/e1/ZqScTLC9J5P268sF+5W6y8YIlzYiM4QCCR1bY2Or8Nhjs7X9YdspHju2XmuzGB/Xa9/6LKTZRao2YunISWtQdUHW4Ds7KWfMNdcAH/kI8MMfRpkkt27VLwhqRGlnp17DD00QZhpETz+9FnPmXG80DsoTUZdkqlR6zeJhEi0i8vk26Ix5s2b9nkehiPJrqVqRyWBXKMws84M+cWInjh5de9YvHPA35sbfU/k9q94bJBT0idVUQehqW3dOUviMCRva25fF3p/8vQp1bNkTzREEHSKf74JNAOsSkumQ9r343E/oQlFPka4Np7kLqC6PixYB//7vwCOPkBAGgOXLyUAqQ/i8h0SY2gy0Kmx8uW+1eqBck1m4cJtT4zJpIr7GOnHNjo4dQdd6+um1Xq5sKkcuFgXdouWrVYl7NkXVioUiKl5SLshMBjXB06qGSYKeP06C0HTAKmzZLHWQxxYZfc1pBQRCtVyXADbNE3mslkqvaY3CSd57FnVk9ZGun8Fjj82uSWrihhXuqkF040Zgxgxg+nRgcJACkzZtInpGR6mo2rwIclJL8ImgKF8PGttgEdqKr1eLCh9vAXXSJDHWqfdouxalBfCL5lTfjcu/2aZVqQtWc7M+/YIu6Zf43uf9m+gFeReSFmnGBOCvlarv7OjRz+P48fvgm0tItOejLLjoH1v8hRir4+MnwTkPSnlhQtoFFDAbrn0CwCqBhjOoqhAG0enTge98hwTvpk1AWxslGgP8KBVTIjDZg8YnQRilvTXlbHcnlNIhxADqaxgMMUrZ2gD8cpiIFLmUfvZ5Z/i97R5178MUrm9ePPz6IsRYWS2o1MD4+GvaBVZ+f2mTxIn2TPEUOqF74sTOibQG8XszHV/pVL++OflNVIsrqZ9AodCOK65IluAMmMIGVRlCo+7qojQChw9HGnahEEWd+lAqpuhUERHrmyBszpzrjaHvri2gSSPyTfCl00Sy4BptFYrckZCkcYnc5/IOwrYrsGlVuvfB+WkUCjPLtF9zeTu/7XgW2/ksEeLTLr+/kChiHUR7IYbJOXOuxxVXvIyOjh2p4i+yMlzrjLYhO1vfPhcUbKXRMAZVXTGOFSuAT36S3BSHhoAbb4wqMQkD6aJF5cnHbAJejU7VHetKELZw4TbvaFEBWxItd4Ivs8aRhVHKxumao0fj6VmfeOIiywIVFzKuhGBmu8YrMY0pNJOkDiGRv9WA2ae9HYXCucaxkFZACoE8Ntar/d3Wvm8kbpYGVF/YFiv1nn0iXeV2K21obRjNXTWgDg9TxsOeHvpcKpFgL5Xoc2cnUSq33FIelLRmjT652Jo12USnyhwq5wycx7UVXSFt2yAza4/znEFLWXCNNo3K1D7ldnG3IRaoKL/KDlxxxcsxbxp1N+OjTZuMtYVCexBvm5YPzxqm9zg+/op1LKQTkNEOq5I7mSzGamikb8huQYwFV1oJW7tZomGEu0qb7N5NlZaEBn3zzaSxywbRUgn41rfKKRWgPLnYihW0YJiiU13ZJlVEXhxncN11z+FnP4sEu85PPokA9Rn0ocIpVJj6tp9kgTJtmdvblznfR5JMkiakSeCVNZIKVz//dpO44GdplywEsAlpF9IkzgOh75PuxW1rqQZt1zAGVUHLCINpby/wmc9EkaKhOd2FkBWGVldtVVvlJVeOefVaOmOsy5gUr7cZz7EOmAsvh8AUxZe2VqitbVsbtnei1gpVn7keDaFZIE2kZXwM6WAzsMZrqeqKedfa/zuJQTbLcel7vgtTzqC6eDFp19u2kWDfsYOCloTAD827rhpa7767XOB2dkaLhQiUuvbauMEVoPqrQsvfulVf7/X977cbdV0aUTwvCvlqCz9btVaoLW92aMKyM2dex8mTe1NTE0m0MttuxqVNV8sQatrRLVsWttPzRRrtVrwzm5HZxwCtvnsAqXLj+MI1fpMYZJO8T1cu/mrRdg1jUAWIY2cMuOgicnV8/XXg3nupkHZo3nVdlSfX+T09tKgIgysQCfmenqje644dxP8LDV/UbNVdS9aCCoXzwdh0jI+/otV+TMY0dXOmMwj5VD1yCVN/zVCvvYWmuE1jYKuWIfTZZ4Fbb40oQlEMZskS804vLdKmCna9m9D3FmKUTAqf8Zt0vIS+T10EeS12Kg0j3EdHaQIJWmbDBuC556i+amjedd/UAzoUCrSw3H47cMcd8fqrYifQ20u+8a++SsdwHk1++VrveU952H5TU5uxzFmIkUYU3JDTF7gmYBph6lsyz9WGrVQfEAkan4UEqPwE7OkBHniABLpccP0LX6DfTXRcSEnHJLC9H593E/LeKu3CKO7HNX6r6dmUdoHNAg3DuQNx7lpMohtucOd7UZFkYskLglhgROCUrJmJidzdTQtPVxewfr3+WkuWmLi7ZnR03Ocd5KGHb4AKQ0fHdomLjZ/nyx/68p22QBIT3y8HPtm0y1p5sRSLlOri9dej4DnR33LW0Y0b4+eYNPu0BWKqnQGx0sFHgL8NpZ5yvySFL+feMJq7uq0VGnGo5g0kK24tctEAEZ1zxx3kYQPEr3/eeeRz39tLlJHpWvv3mzSbca3Wq/eznQbGmBIi7x95qFYzkn3PXSlfgSjxlympmay92bR7G9+vCgidz3zWNEAodAXXbdRfaEK6EFSDJpFRDY3Zd1dZDxp1tdAwBlU50dfoKNEhDz5I/3dFjWYBsSDIhUL27CGXTOFCKTSwzZuJtnnve+0FP2x0hy7qT2f86ei4V6owxBAVI3DDVM1IFDMQVetNhjFKtWCvhyo/o03oZFGWLQsaINTl1VRwXRjVbYVfRkdphycb2rMwuoa8nzQVoASqEQtQSRfMyYqG0dxlbVv+v6wJqYnAsuY0bZkkxXfqMUD5MQKuiDdTIIVu0sjfmbbJVLYsHsVoijiUPXJM3DktPuZ6qOrkswkdm2ambrWbm8/XLihZeMKoxfemAFsAACAASURBVF1chlC54LoYgytWEA/vGiuFAhnfe3tJcz/vvCiPURr4arlZ2EkEKq0x14sRs57gpbkzxq5mjB1hjD3DGPui4ZjrGGM/Y4z9lDH2d9neZvZQI1qTuEuq0OWFl90lfY8RcEW8JRVWtrJlSa5hyh3i0pRV7c3mnmi65/b2ZWVudmfO/BpqCbestDhTjiETXSIKrstCfM8e4BOfiBtPRa1eMQ6Ghqj9228n6q67m6i8devSUzO+Wu5kK2BRT8Fk9QCncGeMNQP4GwDdAN4D4FOMsfcoxywAsA7A73HO/yOA/1aBe80UoZO0Vpgz53p0dNyX6ZZTt03WJe4yRXvqoBPktoWhtXWeNjeH6TlNW/uTJ/c6koQBIq2vjUIKQUj+fp/FXKdoCAPrwABdY/t2YNWqKH1GGvjSJGnprSwonRzJ4aO5fwjAM5zzY5yscsMAPq4c858B/A3n/F8BgHP+L9nepj9COFHfSaq2KQKR5Dbla4Tysi5UgrNUtRydkNQFKIXsIvRFkCn9rqn4he05dZqZLUmYLqgri+AZOdvobbfRWBBYs4Y8Y0xjAygfHyLP0fLlkaLx7W+TYJeNrvv2kd98FmPLR8s1L85NzneYtqh3jvTwEe6/BeAX0ucXJr6TsRDAQsbYPzLGvs8Yu1rXEGOsjzF2iDF26KWXXkp2xw6E0C2+RTbUNgsF2iKLKk7qNSpB+VR6y+kb7Rmyi6BdR3k9VFHoWqfZ6Z7TpgHaBJCuAlRaWkHm2NevB5qbyWd9aIh+27mT3F9NYwPQj4/Nm4mq2bSJKBi1ZsC559I1h4eJs89ybJlgzjcz7hTUk43SEchaMaslsjKoFgAsALAUwDsAHGCMvY9z/qp8EOf8HgD3AOTnntG1Y/B1IfMJVJKNrqLN7m7ygLn99igQSb2GOH75cpqwcoRsloEoWSLElQzwN1yZDGm+xjrXcWaj87ixAlQarxnVIP6tb1Gai//+38kTplCgSl+msQHox6go+iJSZ3zgA0TBrFtHysemTXTegw9GAv6yy6J6wJWgE0U/jI2thlp+0OU6WY3ApUog1GBez/DR3H8J4J3S53dMfCfjBQAPcc7f5Jz/HwBHQcK+JtDRLeqKPDpKE0e4R4oJd9tt0XGio4eGIre07dsph4zgQk2UTmcnCfbt28s1sUpoWWlh47tVzRlA6l2Er2bnOi4kzapAGq8ZlUPv7KSxUCpRkNLate6xIc4Tx3R3R14w999PisONNwI/+Ql9L5eD7OykqNfTp6N6wFkLdrm/6T3rC2TbBHXS3D2hPH3WvP5kscX5wEe4jwJYwBh7F6NyLj0AHlKOeRCktYMxNhtE0xzL8D6DoKNb1K3w4sU0YWRB29kJ3HRTdJzgQm+8EXj44cgtbd8+Evg2SqdYpOOEJvbpT8cHSr1t/0x8N2BP+pR0cvlqdj7H+aZZBbL3fS4WKVht+nSKPt22zT02xHnimN27414wAwNkPN2+nYT3wEBc4Mh+82nqCuig48pNFbFMdU4jV9v4eb4FaXx5+krx+iEG83qGk5bhnJcYY38B4Lsg9ehrnPOfMsY2AjjEOX9o4rffZ4z9DLTM38Q596uKnDFsdIsPXaPbMl95JfC975Ggvv/+eFUnsXWWrync2ORc8du30/nicz1u/3QUii3aE4CRMgHs1I0vDZT2uOZmewWiNJCTvn3nO/TdNdcQBz84GK/4JY810xi95JJoLOzbVx65KgSOnMIgNPraBXvJxnjaCVVQl6c1CItmDo2crVSkrW/SQFusDFDZ3EA+8OLcOed7AexVvrtF+j8HMDDxr6YYHSXaREAI6+Fh8jlWy+TpIE+k3l7gG9+INPZiMV7VSQhpmeKROVJZg9+9m3LMi8FSqfDyLGHTnE2TiwyZb1j5dN+Q9LTHLVxoLscXCnUyU/4fYO7c6Lvrrwd++Uva6QlhLUdHi7+2urs6wb9uXeShI0da64Kf0iBNyUbTwuCbQ8Y81n5+1tjud3xyXj8kaaBLQau18tZQicMETEmXhNHKJUzF8XJ6XuGWputoU2EPIN6hw8Mk4OXzTYmj6gW2pE80ifzHj2+SMBVZH5cUIcm8kib+0mmD11xDi8XevVFbcn3gLJEmyVfaAij2xHfTUCjMRKkUpbs2FRZJk5AsNHLdVmjHpwhPEky5xGEybN4IphVZdCoQfT88DHzsY3SeqoWpRjV1R3DddSTk1VW8pyc6P0nO+GrDpjnbq/aUQ9WodDSQSUD7FpvI2kVUjAtRTWvXLuK8Fy8GDh0iAWsymCbZmekEyNy5xLHLqJROlibJV9oC1vZ0G2+e9X4SO0FTBbBq5pPRzX2f36qBhkkcpkI1ipRK9q2w2GKJJF/Dw/RPGLNGR80ruCqkATpn9279Nk8ty6cmjqonY6stsMjkYSP7tcvw8ZSot8AXMS4KBfp7+DCNpUceIY8VsUPTISvD3N13kwuk7MHx4IPZa+1AuoC5tMm7Is8nN7KqAKbCFKNiCh5bs8ZsPPeNo6kYOOc1+ffBD36QVxIjI5zPns35hg30d2Qk7JyZMzmfNSs6T/ymtqN+L3/esIFzgP6q2LJF35b43tRmrXH8+A7++OPzeLHI+OOPz+NHjvTHPh8/voMfP76DP/poGy8Wcfbfo4+28ePHd1jbpnZQ9u/xx+dV5+EMEO+/t5dzxjhvbeV8+vT4+LCdZxqDtjGggzqeurs5HxyMHzM4SN/XCur4cPW5DqZxUP6PVeAJ9P2mm5OzZpGc6OsrP2ZwkPMZMyozh0GOLE4Z25DCPY1wlCeQzwJhmqB9feGLi+4Zkp5fCfgI7WhygxeLzWeFs24RUFEssqpO4hCIcVEolI8PndAeHIz/NjjIeVtb/Fj1/L4+EhbqMUKIy+NhcJDz5ctpsRECfnAw/nmyQjfOqr3o6xQzdU7qhLqY+21t5f1gW7hDMKWFe6hGJB+jClSb9u1qJ+2qneTalYRLszYJ/yNH+r00+XrX3C+9lIRnb298MqvjamSEtDYxucX5g4P6Y127xf5+sxAX/1+ypDEEu4C8AzhwoJ3v398SvBM0wSUfbIqVaU5WUxmb0sI9CXQCWUy20A5Lurjo7qeeNHeXZm3eTjd7Ce2kdE4lIQtm3V9Tv4T0nxAYXV16Dd2kuYvxtGQJnb9kSfbPz3lyqiULiqYSbclzvbubFk95sW5r4/xDHyo/Vvy/q4uoOXkhHRmhxb8aylgu3AOhCmTBqfX1RZ+rJWRtmn8WC0dSuDRrs/D340yPH9/BDxxoP/v7wYPtNRXsnEfvW37v8mfbe1e1PF3fCbpG1tx7e+k8eYega0+cX0nNXU+RsLP9buqfLBbqLAW6CjGnrrqK3ml/P32vUl3iWEGzCiE/Y0Z0nFgQ1F1dpeAr3BvSzz0LVLr6fNJr2wInKu1q5SqsbPZTboYuP4nsj1ztos2Vhs7HGSiPZBaRziKOYvly4I03gCuuAB57rDzGQm7v8OH4+Wp7WcBVdN3UR2mLYldjPIgYk6uuojgC9Z3LUOdksQj8wR+QS2pTE/Dmm3TeJZfo41myhK+fe8O5QtaTG2FS2Ao81DKxkS3/jC2fyNy5fU4XucmUItY1xkxurkC870SlJVmQTJtGAuLgQcovs3mzud7qAw/Ezx8YoM8PP5zds7qiPc1VuPQLgm9cRKXHg+ymePgw1TM+eJAEvG5h1CWM6+kht9jTp6mvLrmE+qWnp/I1m33QcMI9SS513WQtFEiL8mmn2gtKLRMbqfnWATmxGBDlE8FZ4b9w4Z1Of+TJlCLWNcZctXRF3w0MxAXJ6CgJ/Oefj4pzrFtHwlrX3ic+US6IBgYokjUr+AQg6fvIlKUz/r0p8Vwlx4O6+P7xHwM//jHwvveR5i4XX7G18c1vAq2t9FkEt6nxLLVEwwn3JJqtqXjCpk1+7bgmu6/w9z1O1jpqEhwxgRMndmJsbLU1n4iuipKIbpUndNIUsbWAbYy56Dxb34lMpbKGvnkzZSo17eQqDXPBDhlNmqyg+jTB8ve2oLVKjgd58R0aAr7yFVpwV62K0i3bBHyxSKlJxsejvFGnThGdNjyc+vYyQ8MJdyBcszVNVp+83Lbz1QyQrl2Az3GmLX+1BbyYmKG5vk0TWler1Te6sRa1Ok1jzNaHrr6zafy1QpyKA/Tpf8dRLpznaY6j3Zzor7GxVUbqJW20qw2CYtm6NaK27ryTvr/kEmDZMju1NTpKmWKbm4nS2bePePs336SkcfWChhTuSTRb3WQV7XR1UZ5uuR1Vo7YtKL67CZ/jZAEgri8LgGrZF/QZACOYNCwTl3rixC40NU0/+11zc7vReCYL84MHZ+Oppz6bacqCrVsprFzub7U2arFIY6KrKz7GbH3oEt4mW4u4ngzRz9WgBKNdF0dHx3ZrPV2XcG5vX6bQeOUQpR2zTi2gYvFi4NgxEuhAtPh+4Qt2auvmmynl8pe+RFp+dzcJ+cFB4J/+qXY7aRUNJ9yTarbqgiAbsdavBxgjTq1YNGvUtgXFdzchKj6pi4yYrLIAEFqi+N7HvpAVbNynTcMynTc+fjJWFo/zN7THqZr/+PhJUN32CGkNb4sXk0Yn9/ff/m1UG7VYpN/GxymNtDrGTH1tM5S77se0G6hEvV4bZHrNVCDFJpx1hdhVtLZeWPEMn0B654RSSV9UpdaGVIGGc4VM4sKouhMWi5RmdePGyGBVLJrrVorzTal+ZRdGV4ZA4c62alVkUJOzWZruvdo54c0ucgyFwvkTgppcIOVCDS7XOhVqkQf/8/3SzJog+vv0aVrY2QQb0dxMffn975Og37OH3veaNfT73XfHU0bv3h0v2uID3RgWBWAGBuJppWV32KTXSwqbu2OUNTQunM1pgSPMnduvzfZYKbdY4RIpivEImOSGmkG2v592cT09lUnmpmLKukIm0Y5uuy1e5qyzkwS7zLt1dgI33KCvWym22z09cZe34eF4RkF5N7F8ebnRZmiINP/bbyfB3t1Ngl6+NxXV8JzRcdp6Q9s0MDZN0sCJj5epEj8DXQSVZvH1lkhreBP9/cYbVBt1YICEpsgIyXkk2AHq+927ox3funXUhxs30uc1a8qpnmKRvlMpFJOBf+XKqJ/FWCsW6R7k+r6uMZAVleNHvRBVNjbWi/37GVwip7m5Xavdp92NmZ75wx+mMomiHObQUNQvph3Q4sW08MveMYzRfK8XSgZoQOGeBDfdFBUgBqLJdNNN0TE22kUsKPI2r1iMAhl06YY3bSKNQb6m0CCEIXf7dtLgSyXzvVfac8ZkAAVQtu0uFGaWUSQCcvmz8vP0KYLVcwE/oZ2F4a1YLK+NevgwafJAPJ+60PB27aJ+kAtei606QJN/xYqI6rn2WqJ/VAGiowvEYqGmlb7uOqrPu307sGABuefJY8p38UhC5YRRL+KFmbxoospZlXCD1D3ztdcCP/0p9eVnPhN5ynR3R6m/Tbn6P/lJEuji3e3ZQ2mY64WSAZCnHxCw5QIJTQTmE3Zuyxzpm5ekGqmBQ5J5udMP6LM7+mUBZJZjp/GDB9szC1OX07nK4eYA5+ecQ6HmM2bQ72qOGVuyN5GvqK3NL2WwaEsNaZf7WaQquOoqalu+LzURmXovlcpd5JeGQhwTZQ4V/WZL+Zumf11ZHTdsoJwxvvlhapXYD57pB3LNfQI2eiPERU2nSZs0pZ6e6Jrd3XH+dN064Nxz7QbharjO2epaqm6HLq3a9rvsKWM7V6ctdnTciyuuePlsYFUIL6vbrg8PA4sWkSYmdmSdnRQ9+ulPk6dEoUC+zevXx201ul2UuEZnJ7B2LdE8b7xB9hsThSK39Y1vlNOGckGZq66iXcWXvkQ2gTfeoF0nY3HqSH5mID7e5e/Two8S41i6lGPp0hKWLuWxfrNRd2m8odQ5fvfdUd+K75ua/HbBIZ50NYua91kBKvFvMmnuoW3oNCxd++I7UQRCZPpTtcFqJQbTwVU4QU4G5dLAdYnAfLT2SmaG9N39qLsvOZOjqx05s+TMmZF2KBdzCL0nU7ZKkQjLplHKOxNdquG08M3H7mrDpcGHwjTHxfsoFKI+8UnTLI5Td3mmvjLlfw+d28izQvojK3pDFQBCUIvOkwWCeg2R3a8aWeVC4DNR5YkWn5RNTkFtTxOcfTZAHXwXdtG/8vGiEo8rW6fo39ZWEgY22sQn86eapVIW7LZKUWJMyumsZ8ygc7Iccy7hfPBgu3Kcvq+zKuBimuNiUezrK+8TtZqSgCmDbFeXefyIY+R+SSpnfIV7w7lCJoHqerZ1K227hSFTzhki/Ml9skMKikUY1IRrXUsLGWSEK5uAMIxt2EBeFvWCyOfY5IJIboeqb/L4+Gsx33UBOTOg2TUunStjKIQx2/bu16wBdu4k6uPBB+m7a66hcbJvn91LRURD/u//HV2jWCRa5d3vTp9KQIyd1la6F4AMhio1U5TcfF99lZ65rY3+lkrZpzQ4cWInnnrqs5AN7Yy14OKLvwYAzsyPIdklbW7QgP63226Lp3cQhtbFi8tdnm3wGT/imOnTyXCb1HXZ1xUy19w1ULdN8moeutrKRjSfrVs9FedQYTOu+m7FVa2rHqovhRiwzzmHtDmZztBt3ZNeI+n9z5hBxSLUSk5iV6EeL7RIeVxWCibt3KfvQ/LCZ7UDDzWU+vStfExbW1j7KpAlLQPgagBHADwD4Iua3/8UwEsAnpz49+euNutZuHMe7wyZn0wyWLq6yjtT3mZnNSgrDdtE8y9qnHzyZg3fWqcyRkaiyWk7Tj2nUv2bpO2RERLsrlqwlYAs6H0Wf/UcF02XdhENPT/EPiIri2kWVV/h7vSWYYw1A/gbAN0A3gPgU4yx92gOfYBz/oGJf191by7qG7IF/YYbyMshSaBQsUjbO9UCLwdW1WPCKB1s+T58fZCFD3qUPKoXTU3T0dzcXtZmpbF4MfDFL1LKV+HtsnkzsHo1bddNMPm6m7wfKtm/SdoeHiZqUPWZr/R4U2MmTEgTgJYmqE+mUTdupICw7u54sGFRiR3wef/iGIDaf/BB8riSA9EqApf0B3A5gO9Kn9cBWKcc86cA/tpnNRH/poLmLrQ8n0LJkx0mzf3AgfYyraueaqWqhmxbbVShdc2YYfd1r2fUcpfos7tTx0HoWEmjuesMpTNmEBWno2hDkVWJTGRlUGWM/RGAqznnfz7xuRfAhznnfyEd86cANoOomaMA/m/O+S9s7daTQVWFagiVDVOAf4IhYZiVc8OIHCHVyv9RLYSURUtbgi1rXHop+YkvWQKMjcVzAskGxjVryCgqxsGKFcBvfkNjY+/eydGftSwfac8rw7QJwkLGijxv5ZgR37mqey8ilqBUop1aS4s+dqCaqHZumW8BuIhz/jsAvgfgPsNN9THGDjHGDr300ksZXTp7yFut0VHaRu3ZE1XS2bWLtu3qdkrdmt98cxR+LkLJN29uPMEOlFM2zc3taGqajrGx3rJgp3qqulQsAs8+S4vwwYO0DQf04fjvfnc0DgCi606dAn7v9+jzZCjlmDQzZRYwF+CYZwxACxkraegvW6ChHHh2ww36oDCXLKgJXKo9PGgZ5fhmAL9ytVtvtEzolil0e1urUOVawLWVrgcPGc4j2qy/n7baIrjonHPIy8R2XiWDgBoVeo8qFvO4UlHNsaKjdGQDqCl2oNpUFzJMPzAKYAFj7F2MsRYAPQAekg9gjL1d+viHAMbSLjrVhKBP5JV7aIgyN5qSKYXkgi5WOLlXvcFUkOPpp9cCMGcTzKLKjg+EpjU6SsbTr3wF+JM/oeLIAFEtzUodClUT4zxKA8xY3LiaQw99VSd6caa0AtUYK3J6CDkdyPBwRMl++9uUz5/zKPmbQNq88BWDzwoAYBmIS38WwPqJ7zYC+MOJ/28G8FMAPwRQBHCxq8160tzVaDU5HYALLo3ctapnZWSpJ9jc3ORUBb7ubVlD7oMtW0hzZ4zzlhbS0FpbIyOaejznUZ/JfT/Z+6zaCNHIKzVW1IjjwUHSzC+9lMbD8uVRygABU+wA59XbnSNPPxAG0cFLlvCzmfh8z7FZ5rds0Q+QD30o+l7eAopMkZN5i591TpBKQPWGammJC2o5PF/XH5UMStIJs1ouhpWAr497JSHPPTk9RBIPqGoGIebCPQFE+tQlS/wCQXx5Nt2xatSrmndiMuP48R11MXldEJrWJZeUC/KREX3wGee0AMsBKKI/bTy9L0wpjffvb6kL19EscORIf90s/mJudnWVp/v13Y1NZs59SmBoiCqx9PaSO9y6dfYAgxDLvI6TEx444hqnT5ut8ZMNc+ZcPxGUVA7VY0JX5akaEHaQ3l7gySepX+Sau4cPU/BZVxcV7ZDHwYsvRoFMAoJ/Twt94fE3YaoTa3p/tXqvLpw4sRMvvvgVw6+sanYXAcGzP/JIebpfHy+iZctIdsiy4PBhYP782gch5onDUF4zVbhBrVuXbTIlXXIh8V1bG1Vdr2Yd1DRwFTD28XsP8Y3PErL/8+ioPhZhwwYyogH6OAdR2zbL2rUnTuzE2NiqoHOamtrK3t8FF6yuag1SE3RjxJ6ADli6tLryqFiMJ/QzxbKY/OCHhsjQevvtJDtEDWTxuRLw9XPPhTuSB3aEnCcEiiwMABIcpRIJGJFpsG6s7Qb4CmXXAlCrYCZXv6m/Fw2ZAn0yAfpC907doALkvt9XM0jMNEZsz1ftIDYxJ22F7cU8tgVIHT5MAv2KK4DHHqusYAdy4V5xyNGnYnAcPhxFnwLxFMHqwBDaoEj9C5gHVr0hK6FcrXS/WURlqoJct1inWYxN79QEl6B0obm5HQsXbquYJm9+HtOCxNDRsR0ArApBlli2DLjyyrggHhoCHn6YIo5V2Pr8ox+lILglS4ADBypyu2dR7QjVKYfFi0mwr1tHIend3bR6Cw1OjnDU8fM9PSTY5VJfgiaoVsRgUmQVYWqOWEyeOEoHV0FoV4ShGqcwNBRPMGUrheiLkHcnkqtF/uLhGB8/ibGxz1SMizc/z7imhB7D3LmfAwBtMfZK3eNNN9EclsfF5s30vQ6mpGRDQ6SxL1lCf+VEY7VELtwTQgjjzZuB3/kdCkNvaaECCCqtogv5vvtu+qe2Wc9CXSAroVzpABU5OEUI4E9/moLT5P6Rhf/WrZHwXrw42mWtXBkJ8ltu0dc0TWNA83l3TU1t6OjYcTZMf/r039YcFWLZfRPHjq0PON4fzc3nG75v19TA3Y6FC+80Br9V6h5Dg490wYgyx37gAP298cb6EPC5cHfAptV1dpLGLrZjzc3JUo1ONvgKZZfHhi2FsAkheTxkoS36avt24BOfiPePPMl/8hOanEJ4Dw8TfSaos85Oot1ElS65jTQLs74o9DQUCpQKuVBoB2NRrp4nn7wSr776iKalMJq1Uvl8TN5DjFG/X375c2X5ZGqRc8g3RbBMrcq7tQceiHPsAwP0+eGHK3bL3sg5dwd8DCmrVgHf/CYwPk7hycI42sgCXmcsBSK+tLn5fJw58+uYC18WHhuhmf/E793d5Oq6ahWVodMdL3j13l46Jis+3WVYdh2XzNjqhxA7ie9zAMnsKdU2sMt2M9HXJg+5ajhd+CI3qGYIk6fL8uUkDC65hNypOAe+9CXgyBFg9+5yoVDLdKuVhq8AEhM1RFCoCDVmivqivb3A/ffT+aLvhMalesQILT+tJ0wW7p5pja2MtYDzcZQbMqeho+Ner/vQPQdjLWhqegvGx18p68MkgrrarrGq22Il3BhDlREf5AbVDKHbuo2Okh/0wAD9f88e0tZLJeLSdRysy7A3maEPvinHqVPPl1XkCTWc+W6lBX++e3ekjQ8NUb9s2kSaerEY915av560NxHQljbRm41H9g00CqMlmsuorosv/ho6Ou6boHikI5tnerVI/very56D89MYHz8JXR8msackoenSoFQiQb55c5SO+/bbyym3NKhlUrEpp7kn0Z6zdHvL2oWuXmAvxBBBeHik2X77vkOXZibaef/7gUOHyoOUSqVoQU7aT7b3ogtACiluosPcuf1YuPBO7W9JNONQSkjuwzS7s6ygm+9r1tBf4dAg6Liurspx5VnGROSauwG+2rMw3KnbqJUr07m9panxWM/w9faYP//WVIYzk2FL1x8uzUwOPRdpH4Tb6sAALfajoyTo5V2YyYCrg/m9NHl7hpi04PPO6wL5jQNAs1WwA/ZdhA4mjd2GShg/06RS0M33Bx4gQ7mY33fcAUyfTgt8ml2aydi/Zk1tUn4XqnOZ+oG8TbJpfmJQrFwZL267axd5Tgh/9FCo7lTCx32yY/78WzUa3jQUCjNRKsU5WVMIus8CYcvpo75HsRN79dVIa5K5VF1fqLs3WXMX58ifXZg//1Y89dRny3LDAHqjok44Cm03em/NOHPmdbzxxjPo6LjPWxsOWVSFxq4PODJD9KGq8QvaRn4eH6RtRzffxQ5Njg7/znfouzS7NHWXJ1IbcB45WHR2Vo+amXKaO+CnPYtBsXt3ufae1O0tROucDJA1qmPH1uOCC1Yr/sv34oorXj7r8gbIFEPcV87Xv91VJk7VnopFYNs22nLLWpPcF+eeW54oTnZ3TcOZzplzPZqa3uJ3MMwL3Jw510saPAncUFtFSHyC24bSBGBa/BupD7PyWc+iHd187+wkYfz661RGTw0kVMeRsN/IO7ZikaJcxXHi/BUrKPL1mmvIRVr1nFu5sjpJxaakcNcFI+gQSqG4fLDT1HisN+iMoseP34f582/V1sOMHw8QD00CPkvDmbwNVw2l8mIq94UcbTw6Wk7VJaXSxOJHRkc3XAtcWkEXYuS00SsUTHU/OjruNRo/s/JZz6Id3XwvFskrSpUBQlFQ6ZxCgew1hULU5nXXkRBXFbTTp4nqW7kS+P734/ch6rJWwztuytEyOi3cpI2FUii6bZm8hdd16GSlZWyCRs76KAxqpEeoGDhFZgAAC0pJREFUW3yeuQ+zrGm///1RNkfxjsViKveFunVXx0MSKs3HENnc3I5C4Vxvg2NaQRend+zXbG290JgbRhbi6rmiz01G5NAoZtN9+Lajm+8+VImOzhH2m1dfjdO5l1wSHbdtG0Wq33hj5DdfKweKKae5+2rPLgpFp6UDkcFV3cLXbYX0hHAJGlWzN3G3lTDA6Qyl8m+mRVannSel0ly0RlNTGxYu3KaN1DQhi7QPpuhQFSYt38bxl+/OVEwLTi2RNkWFbr6LZH0uGaCOiYGB8jEi5q/4vlSiuX/uudTmhg0k/OVzqjXvp5xwd3G2Aq5FwOR109OjFxKN4OMuc+ymoSMEja/fuykHSRrotuGhycHEsUmpNNuipaOhfDxCqllYPInPuavPWUBFE/E+xsZ6wdj0s2kYXPeh9rNsixHwzeukSxinjpHFi4n6u+MOsuswBnz5y5H75erVwPe+F8VMyHmLKo0pR8v4wkWhmLxuAPp/Vxdt0cQ5nZ20RVu+fHIV5RAopxnKNXFZ0Phq5GfO/BonTuzMhG+Xw8nFuz3vvCga1USZ2ai6pFSamU4op6F8PUJCaJUsMGfO9UFtu/qc89Mx2s4E9X2Mj5+c2DVsd57rokZ9oY6J886Lx0iIMbJuHQl0zoHf/V1a9MUaViwCX/86MDhIY1Jkjr399urM+ymnuWcJddsGRANi/Xrq5GuvjQw4mzdT0qrJ6ONu1sqaodOofKkCzk9jbGxVJqXgnn02ijAVE3v9esraWSqZM0Mm0c5dmnaIlh1iKPWlVbJCiI+5T5/7LPppDMdZRYSqY0LETMgxErt2UdDTnj3kcbNpE/3ds4fOl2Mm+vspncWqVdlGwNow5SJUs4QaKSmKdsh87YoVwGWXkWW+EqXZqoXQRFBJkl2lzSMie8fccANtlUslytYpjGdqnpkk0D8b5SSXg4h8IzSrVbQkFKERrT597mNAz+J9ZBkRaoKIfgUiOXDHHVGdBgGRy+gTn4gnrUuaVyqPUK0wdIa23bvjx3R2kpB55BHakgm6YDL6uJu0MhNnruNs1dwmKkLc+kwG7Z4eckXbtIn+futbJNiFxi5yxuzbl/zd63cxHC++eBcee2z2WS0XgJeWXa2iJaEI1aDjfQ4kjWVI+z58XZ1DoBtvhQL5ua9YEQlszikCVrWtbdpEY054z1SDe8+Fe0KYtvK33Rbv2LvuImEyPJx9gYdqgibltLLvBWeug0ohLFiwTZOzPA7Xtl1MMtWffc0a+rxoUcR5ir+dnVGWx1WrSGNPs7ja7rFU0ifSsqGahtIQJHG9jPqco6Nje6IkYEnfh1xoRShRws7l288mw/uzz5Y7RGzeTLY1ziMh/uCDES0DxKkZUdynu5t2FhXfuXPOa/Lvgx/8IG9EjIxwPns254OD+r8jI7W+w+Q4eLCdF4so+/f44/O82zh+fAd//PF52nZ82hLvd2SE/s2cyXlbG/0dHOR81iz6/4YN9HfWLPp+xgzOe3vjfTAywvmWLeHvwXb/Sd9N9F4Yf/zxefz48R3hN5YxTM8Z0t9JkeR9jIxQPw8ORp/F3PPtZ3l8qZ/F/zdsiB+zYQPnAP11IeRYEwAc4h4y1ksQA7gawBEAzwD4ouW4T4DIsstcbU5G4b5lS7lw1gmIkRESOFkJk3pBscgMQowFt3X8+A7+6KNtsXYefbTNexKLSdbWFk2Wvj4S6PL7njGD89ZW/WRNiuPHd3gL9yTvpl6Qpo9qBZMAzqoNVTiHXC+Le+M8Q+EOcod4FsB8AC0AfgjgPZrj3gLgAIDvN6pwt63qKrJYoesNWWtyabRV8X6nT48mS19feV/09dE/GVkssgcOnJup5l6vqMcdhQtZzD1dG6pwVnfjNnkQIjtcyFK4Xw7gu9LndQDWaY77HwA+BmB/owp3zv1W36xW6HpDvWhyIyNEt0yfTn/lLXO13rV5FzN5tNxGRKU0d51wbmuLKCD5XJ3i4Lvr90GWwv2PAHxV+twL4K+VYy4F8I2J/ze0cOfcrhlkuULXI2qtyYn3KbR0lROtFu1l5t2bJ5WW20jIYu6Z2tDtCmtFs/oK99QRqoyxJgBDAP7U49g+AH0AcOGFtXXzSgrVzUqNVgzJNz4ZERq1mDXU9wvEk4FV6x3r8tdXst5nDjeymHshbdR70j9nEBNj7HIAf8k5/4OJz+sAgHO+eeLzLBAn/9rEKRcAeAXAH3LOjVFKkzGISQ1JVj/nmFqohzJyOaYefIOYfDT3UQALGGPvAvBLAD0A/kT8yDn/FYDZ0oX3A7jRJtgnKxpdK88RhlrvYnLksMEp3DnnJcbYXwD4Lshz5muc858yxjaCuJ+HKn2T9YJGyseeI0eOxoYX58453wtgr/LdLYZjl6a/rRw5cuTIkQZ5+oEcOXLkaEDkwj1Hjhw5GhC5cM+RI0eOBkQu3HPkyJGjAVGzYh2MsZcA/DuAl2tyA/WB2ciff6o+/1R+diB//jTPP49z/lbXQTUT7gDAGDvk44zfqMiff+o+/1R+diB//mo8f07L5MiRI0cDIhfuOXLkyNGAqLVwv6fG16818uefupjKzw7kz1/x568p554jR44cOSqDWmvuOXLkyJGjAqiKcGeMXc0YO8IYe4Yx9kXN762MsQcmfv8nxthF1bivasDj2QcYYz9jjP2IMfYIY2xeLe6zUnA9v3TcJxhjnDHWUB4UPs/PGLtuYgz8lDH2d9W+x0rCY/xfyBgrMsYOT8yBZbW4z0qAMfY1xti/MMZ+YvidMcbumHg3P2KMXZrpDfhU9EjzDx41WAF8HsBXJv7fA+CBSt9XNf55PnsngLaJ//c3yrP7Pv/EcUH1dyfLP8/+XwDgMID/MPH5bbW+7yo//z0A+if+/x4Az9X6vjN8/o+CqtT9xPD7MgD7ADAAHwHwT1levxqa+4cAPMM5P8Y5Pw1gGMDHlWM+DuC+if//TwBdjDFWhXurNJzPzjkvcs5FOZ/vA3hHle+xkvDpewDYBGALgN9U8+aqAJ/n/88A/oZz/q8AwDn/lyrfYyXh8/wcwMyJ/88C8GIV76+i4JwfABUuMuHjAO7nhO8DOI8x9vasrl8N4f5bAH4hfX5h4jvtMZzzEoBfAWivwr1VGj7PLuPPQCt5o8D5/BNb0Xdyzr9TzRurEnz6fyGAhYyxf2SMfZ8xdnXV7q7y8Hn+vwSwijH2Aiit+H+tzq3VBULlQxBS11DNkQ0YY6sAXAbgP9X6XqqFkPq7DYwCiJpZCtq1HWCMvY9z/mpN76p6+BSAv+WcD06U9NzOGHsv5/xMrW9ssqMamvsvAbxT+vyOie+0xzDGCqDt2ckq3Ful4fPsYIxdCWA9qO7sqSrdWzXgev63AHgvgP2MsedAvONDDWRU9en/FwA8xDl/k3P+fwAcBQn7RoDP8/8ZgF0AwDl/AsA5kMp2Nji85ENSVEO4n63ByhhrARlM1dJ8DwFYPfH/PwIwwicsDpMczmdnjF0C4G6QYG8kvhVwPD/n/Fec89mc84s45xeBbA7WwuqTDD5j/0GQ1g7G2GwQTXOsmjdZQfg8//MAugCAMdYBEu4vVfUua4eHAHx6wmvmIwB+xTn/58xar5LVeBlII3kWwPqJ7zaCJjJAHfp1AM8A+F8A5tfa0l3FZ38YwAkAT078e6jW91zN51eO3Y8G8pbx7H8GoqZ+BuDHAHpqfc9Vfv73APhHkCfNkwB+v9b3nOGz/z2AfwbwJmiH9mcAPgfgc1Lf/83Eu/lx1mM/j1DNkSNHjgZEHqGaI0eOHA2IXLjnyJEjRwMiF+45cuTI0YDIhXuOHDlyNCBy4Z4jR44cDYhcuOfIkSNHAyIX7jly5MjRgMiFe44cOXI0IP5/GY5eXKHQ3OkAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "pos = np.where(y==1)[0]\n", "neg = np.where(y==0)[0]\n", "\n", "plt.plot(X[pos, 0], X[pos, 1], 'bx')\n", "plt.plot(X[neg, 0], X[neg, 1], 'yo')" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "SVC(C=1, cache_size=200, class_weight=None, coef0=0.0,\n", " decision_function_shape='ovr', degree=3, gamma='auto_deprecated',\n", " kernel=.gaussian_kernel at 0x121dda048>,\n", " max_iter=-1, probability=False, random_state=None, shrinking=True,\n", " tol=0.001, verbose=False)" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf = SVC(C=1, kernel = gaussian_wrapper(0.1))\n", "clf.fit(X, y)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzsfXt8VMXZ//fsbhISLkGDclUQBIzWWgq00kpL3ogW1BZpRVpIrb2A9CL9oVJTX2yLry+Ckr74Vqu8rZeCNWorXrGtmiBQvASLFyz3eAUJGEAuCSG7O78/JpOdnZ2ZM3PO2c0G9vv55AO7e86cy8w888zzfJ/ncQghyCGHHHLI4fhHqKNvIIcccsghh8wgJ/BzyCGHHE4Q5AR+DjnkkMMJgpzAzyGHHHI4QZAT+DnkkEMOJwhyAj+HHHLI4QRBTuDnkEMOOZwgyAn8HHLIIYcTBDmBn0MOOeRwgiDSURfu1asXGTRoUEddPoeAsGsX8PHHQN++QL9+HX03Odji0CGgvh445RRg715g8GDgyBGga1ege/fk444cAfr06bh7zYHi9ddf/4QQcoqnkwkhHfI3cuRIkkPnRk0NIb16ETJvHv23pqaj7ygHL5g3jxCA/ktIol9Zf4qfc+hYAFhPPMrdnEknB0+orQWmTAEefRSYP5/+O2UK/T6HzoPaWuD3vwfmzaP/1tYCZWWJ/rz55kQ/l5V19N3m4Bc5gZ+DJ9TVJQsBJiTq6jJz/UWLUheX2lr6fQ5m0C3aZWXArFnALbfQf3PC/vhATuDn4Alz56YKgbIy+r0bghDWo0cn7yiY8Bo92ryNEx26RVum+edwHMCrLcjvX86G3/mxcGGqXbemhn6vg8xG3LUrIYsX27WV8yGkwmufiMfnbPjZC/iw4ecEfg6e4UcwzJhBSHFxQlgvXkw/z5hh15bocDzREYSwDmLRyCF9yAn8HDoMXrXsmhpCCgsTwrqmhpAePZIXAbe2MqHhd0bhl9v5HN/wI/BzNvwcAJjZ1WXHAMB553lz7uXnA4WFwOLFwKRJwBNPANdeq2+L3QPvcCwrAyZPDp4ltGgREIkkt1tVBVx2WXb7CoJwuOac4scpvK4Ufv9yGn52oaaGkKKihB2daYmLFye0WZm5wFYrF9thJhl2bZlmymvZ7NxLL6XmH74tG83bRHPn30GvXoRUVBDiOKm+BlNkarcQhIafs+NnL5Az6eTA4EeoLF5MBVpFRbKg49vjhQkT9rZCgd0j31ZRESFdusjbEtsV7zOdAo19P3YsnS0VFfbXEp+X/X/GjOCFKH+NhQtT+1AcC7rxkjMNZSdyAv84gx+h7Vczq6igo2LsWPV5TCsvLw/uPmfMoAuIuLiIuwsmfNh9+nHWmgo0k3dich0m6IuLqf+CXyxl8DIOZLshtkuTjQW38WLjFDe5387oE8k25AT+cQa/QtuPI9VNmw1K6/My8ZnwYZo9uwdm2rFpS2xTJdBMdj0m4N8b76w2OcdmHIjvlS0w5eXqc1V9atvXJvebMxX5R07gH4fwK1ht6Yom9uqOnKzsWuJ91dSYmZZkC8zixZT/r3rHNTXufg0ZVItZeTlp91eki4kk6yOTBSaofDom95szFflDTuAfp/DKMfcyoWT2XiYQ2eeO2o6b2KWZPVwnvGV+AFGYuy0SJs8rE5bFxVTQFxUlTFemQtTr4s37WWwFsJ++NrnfXPyEd+QE/nEEmUOTD0hygx8tPAiBno5FwbRNNyHCv1Necw/qPlXXKi6mgpeZnmpqCMnLI2TWrORrLl5MyIQJ6nZstGGe/cRfV2Q0Bb1ry2n46UdO4B9HEE0UMpOFDm7CMd1auq0ACep+TIVIEA5nU+iuNWsW/Y0tOuKOg38mr2aV8nI6dvgdEb8b0mnyEybYvx923UwuMCcicgL/OANjrPDCKyihbTPhvLJERD69zuZtIiSCeiaZ1s2fE6TWb7IAMSE/dmyysGfvXWTc8N/bvguxT9wc3V4Eswn9NMfS8Y+cwD8OwZsnZMwLZiJgn22Elak27GXSs2MYldGEK19Tk6AqdulCn8NG8NsEUfHPwoQ+vzAFoX3avDfGiBo71tv5InTvgh9TJtfwY3rJmW3Sh5zAP84gThaZIPIrrHiKo3htmaA0nbgLF1JzBa+5zprlrsGx+8nP9+bYFO9BPH7GjFQ/SE1NgjkjslNkz2uqnZoep9Lw3e7DC2TtmVzDj3M155hND3ICP0uQztS0srQDKiqdqeYuozjKzrOZuEyIjR9Pzxk/3j0dAX/fRUXEmrqoas/GxMPvkPjF0G2XwH+26X/RZi+z4QclMHX3rbtGTsPPTuQEfpbAz1acQWe/5aNLVRPKlKnCjr/0UipoRoxI1qrZ9d3ojrL7t9HwRdNNjx5U+Obn+xN2bgJHfA9M4M6aJV8MTdq16f8JE1Lb5lk6QQpM1UKk61s/Y1l2brpZUScScgI/i8AGe3l5KrPGZoCrBFJFhdzhqNoFiJD5AwoKElo1a4Plp5ddR+fw43cPOhu+uLAxYT9jBr12ly6ERCLJcQCEqE0zsveqW/xkQpAxZ9yiasV2ZekMKipS790UKmHrJ6JYdZ+qtAt+0iTIGD5iTIcXZSgHipzAzzIwgVBY6G+Aq0wvrHgI35bbhNJpebxWzacRUAkYMekXfy1Tlo54f+yZeCHLBD+/8zBN2OZFQxZ3UawdN5+GeA/8YucFbhq51zGl6idxJ2kKcRzW1OhjRtJh5jkRWT85gZ9F4Ae16Fj1MsBNnatuA1+mNfIaPLtOJKLWilXPqRNCOqEka8MkF4yb4PBqjvDaLtuV8Ivz+PF0EVXtbMR2bROiVVTI23drg39GnunlBTNm0HtgkbwsmljXZtCOXK993ZnRqQV+kKtxR6/2ssHHnJBeBnjQGpHYnsh9Z8JqxAiz68kmr20feA3DtzXXsHtQ+UjYTkdXYtFN8+Yd1SqTkBcBFcQOgm+Dj8L1M6bYYsz8Lfn5qTtPW3OXlzmcjp1DNqPTCnxeQwpCUHf0aq/iy+syFaqQrmdRMXt4273Ofi3en59JpmrDTaM12Rmwc1WmJPE5L71U7XMxaXvxYvpezz03lfmkugebd1dTQ4UkYz3xfgYbvxCLd+Cpr37Anpv98T4Bds1evah/hBWq140vr+P+RKKAdkqB37fvSKUN1I9wy5bV3u8zpWO3ohOUplGdMtvv4sX22qLu/egWIZUN3yYOQdzRuAWHufUl28kxDd9EGHsRUPwOghA5lVOHmprUOsJ+5ohKw5eND37nI44zWbs2czhb5nym0CkFPjAyabAH2WnZsNp3tHlJRFCLKtM0bVMGE5L8TnibMu9n4P/ftWuqMNaxdGzGEAu4YpGuTAC60UdVAVmMisrul2m0ugUn0xo+c7LKUnZ4Ga86G774jLYFa0zncEfv6jsCnVLg8xo+QxCC+kRb7U0R5ALk9R3bTk4v48HkHF4zVVFdbdoWdxm62rdeBJR4jF8bvpfPMrixdHjCQbo09mxTrDKBtAt8AF8DsAXAdgA3Sn4fCOBFAG8BWAVggFubvA2f38YHYRPO1tXeRsN1O5+howa318VZp2Xy4McDSyXMwyaVsMqm3KVLIkc901RlY8WNasq3r4rS5duy7cMgWTq669rOQV2brK0RI1J9GibUzWydw9mAtAp8AGEAOwAMBpAP4E0AZwvHPAbgqrb//weAZW7tjhw5sn0i8fzimhq74s5+hWgmIS5wYhpkW03PZjIEsVgEQe2rqXG3I4vfsYAoJvRtUwnz/2djjgWZMQHNUhh7TYPAXzNdu8tMCMMgd9lsLnftmhpLoRozbuM0m5SejkK6Bf4YAH/nPlcCqBSOeQfAaW3/dwAcdGtXxtIRJ6mqE20iBrMNvFDwwtOvqaHCSrRtuw36IIQF0874yWtiCpG1oWOK6KJgZYnGVOeIC78ojE0FtEkwWdDCWPY8NlHGXhDUgmUSS+H3Hk/kHUC6Bf63APyB+1wB4HfCMX8GMLvt/5MBEAAlkrZmAFgPYP3pp59OCPE2yMROFgtNZ3vn81qUF41KtOGaDvogJrSpSUZ3/Zoab1xwWSphG/Dv2kZwMAEmnm8T+GaLTAs2sX1ZNLef5wmSSJHunVS2IxsEfj8AjwPYAGAJgI8A9NS1y0faehkMYqfbsgBs8oCYDnRbO6mJhq/i9n/+894WuSAmntc2/JiEdKmETSCOF5vYD34nw+7ZtAKZH5gItqAWGtU4U9VcsDGf1tTIGVfZsoB0NnS4SUc4vhuAj9za5SNtva7WXlkA/HVlpiGvmpWbViaarExs+LpzbNkaQWhGfttgEa38O3ET+qY2dLd79qot8++9vDzZJs1+T5cN2U2wBbUTkC0czNfBm7H4sSia93Tj1zawT4echp9egR8BUA/gDM5pe45wTC8Aobb/3wpgvlu7/fuPlAZoMO6yqU3WNKe7DOxYVvuTtc8CfJjN0Y3rbMqi8Opg5u9TXCBMszIGIRiCaMM24RYh7qmE3eDXHi6adAoLM+MzMhVsQS7ksnGsqrnAFkHTHYhubujQmX126UAmaJkTAWxtY+vc1PbdfABfJwmzz7a2Y/4AoMCtzWHDRpKiIhrSzgswxtrRZQbk/y+L/LTRuERbMtPmWMpgnclENFGw/48YYad529ynrf1ZvFcetpppUOaDbNDQbN8hf89BJMUL+v5sTRw2Gr34nH58UH7Mt0HM9+MBnTLwirF0GGND1Fz5DIQiM8JG8JhwhUVtpWtXomWE8O2IW9yCguAdyKKADCr3UEciG2ywtho0L2D8JMUzgY2PSRTUNrtblUYvs7eLi4CJhi+7ppeFsqYmeD9AZ0WnFfiEJG+TZZQ5m2LYMqgGtkxLYPzwwkLz6/KDmN8VyK7tBUGYUToaKodgEDQ9vzBZeLLp/sX+F/0ZNuNDFMC8MOfTZzDwJlem4JjWHg5iHHtlpx1v6LQCnx9wMo3JLzODQaZZqLjCQ4YknHIqzrUIJjTy8sw1ENNdyvEQaCLuhGwDztzg9R150TizYQHm71smmG3Gh0h80D2XygfFdh28X4SfY15z9cie2TTh3fGMTinwhw0bmWS6YVWXevSgn7/wBSoU+FXdzbmng06T8yOQ2PF8EJGJEMgGwZFJsOfzW/pR17bNu/T6/oNYgHfvXk7WrRtIamsdsm7dQLJ793Lzk9sQdN4pPwuHuGt2Y+LYvkOxb/xWFOvs6JQCv39/quHzW9KaGkIuuYR+Pv98+2o6Krhpcn7ZM7aRwqb3dbwhnXZ723eZyZ0TL+BXry4hq1blk9patP+99FKRUujLFocgxk3QCoeNBm577SCYPscTOqXAZzZ8Vdg6o+/ZBLuomAdeiyd7zethE7yVDc7LTCCoxU3XJ9n4LnfvXk5eeqkoScDL/tatG2h0bk1NEZk0ablvQZ2OBY+9fz7ltAqq8eBGsuCDv/idRWcycfpFpxb4DOJkNQlnFyHTHNyiA03yq9tOLrfzZBGnfsxVDEGYCtLRZpDapKotpjToTEYd4Q+h704v7OmfY3xuTc3AtD6Dl/dko+EzyBZo3ViRUTJFhe5EQKcX+KrVnrf78oFROpOLTpOUDSZVBSVx0fCSrIydJ2olrHBEly6JZ/Ebri/TBnWmgky2ma48M6oIUJUPJmgzhglqax0jgS/T8NXnpi4OQcL2PYmLrkk0rck8Ve0Gg9otdlZ0aoHvprGxicrMO2KZO9lgNHHQyiigqgHklQ7GzhOLWi9enMjFbpuETKVxq7RBmSAxRTralMHLgsD3sQlt0k8eHz8w0fBVi6jJ+1eNB787MxuhyrNxRIVI1o8mC4qbeS4bzXeZQqcW+KZ2cCb02ZZdFeloMlBlg0U1gLxsVWXniaXpamrsB61O49Zrkt4mfaY0TPauTJ3fXvqYb1OMrk4n5Db8PLJmTYlrv7jtsFS/b9kyK5CdWdBC1XRhyGn4enRqgW8Dt5BuE83BRsMXzzelg6nOO/dckuSPsB20Oo3Pjybp5XqEBOszYFq5G73Vax+z74uLCYlEaJBcphKg+XlPunPVfR72vTNLh1BlZk6d09WtfzvCLJdtOCEEPj8AVRq+acI1frDobPgqOhjvJHJLzMXOGz8+sViIRUNMB61O4/bDBlFBp2Gmw2fAa95eKbQszYDs3dbUJEdTB5G5sSNh6h+w3ZmZCFWvfhmxdoX47k2ZcabmIz/3mq047gW+OGm9RmrKsi6edRb94yFmYtRNAJPfRMFy6aXego9sNG6/k54hUz4DfkHnSyCanMO/e10AEV+8hS26ptlGsxHp0vBNBKQfTZvteMeO1Z9jkgfL5PpeMrRmM457ge81MEqEbJDItG1RaDA6mDjg3WyOXrQRHWy06nQ7XIO074uLpJhQz+TcbEmA5sV849Xko9rVrVrV1Sq4SwfT5IOmwp6dw7j6OvOoqTnHpO/5RG+2ZTmzDce9wA8SskEifuelCEqmWAOmwkHn0EssBuH2RcBWGAS5oIgMGvZ/02L2OsYOIfSzjAQQdAI0L2Yuv6ax3buXtzmAxb4wcwy7wU3o2ox7cccr1rLQnaMS6qbXF815nVXYE3KCCnyd5uG2JTVh6dhoL140nUxAXBxk7A2vGmA6bPh+An5sFms/5ggdVIvgmjUl1o5X1cIpCvjVq0vI6tUyga+/rg1k41sVIa/rK6+BU24MOt2845UJtqMrKOi85hxCTlCB79eubsLSMdEe0iU80gE3Jo+tdp6OyF4bqN69qnAHQ7qceKZOVH5htDGN7d69PMVUY/PnZ0EW54LXspPMlyIuwDNmyN8/73fR7f5U846ZCFkGXLarsPXbZJPj14/AD6GToqwMePRRYMoU4Oab6b+PPkq/V/0GJP4/fz799/LLgUmTkr+bMgWoqgJ+/3ugvBxYsgSorU1cu7YWWLSI/r+uLnFd/r7q6uyfqaHhIbz88iCsWhXCyy8PQkPDQ4Ge29LygbYN2e+6dnv3noYxY97DuHFxjBnzHnr3nmZ8v36waBHtA/7d85+jUWDWLOCWW+i/rG8Y5s5N/a6sjH7vBwUFpxsdF483ob7+Ju05su/r628CIcc83x9/XRvU1tK5MG8e/be2lr7jO+4AFiygc2zBAvo5GtW3NXcuMHUqnWP8nHr8cWD06NTrPvIIcPQo0LNn8nzduROYPNl93pWVAWPHAk1NwDe+ATz3HL3PcBiorjZ/B6NHJ99zbS39LN5ztiPS0TfgB2VliYk9b17yJK6rAyZMSP5t5szUQXLllYn/s38rK+k5zzxDv2OD7Ikn6Gd+AZEJCbbo2KCh4SFs2TID8XgTAKCl5X1s2TIDAKSCtKHhIdTX34SWlg8QiZyMaPQggFbtuQUFp6Ol5X3lPYhCxvaeMgU2+Xhhz38G6GcmoLz0hxcMHnxr0vvSgS2usnNCoSIMHnyr8hw/sG1DfLdlZcmfDxxIzLE5c8za5BWyWbNoH/F9x1BXB6xYAWzYAFx/PTB9OkAIFeCvvpqYg3y7sn4eOxY46SRg2bLEfY4YYaeUmd5ztiOrNHymufHgtWkRMs2DIRIBli8HKirob1VVVIuYOjW5jXvvpX88olEq7NkAWrECcBzg1ltTBUtQqK+/KUVQqDQyJoip8CaIRhvBhL3u3MGDb0UoVCS9vkzIqO5p06bp1juQIKHb3fECit+xieMqHejdexqGD1+KgoKBABwUFAxEOFwiPZYtrrJzhg9fKl1QdTuISKTE6rqm0O1gdfPPDbyyJtuFAYmd2Jw5VNgvWwacd15C2JvOwdGjqWbP36eXHR1/z+edl/q7TlZlDbzagvz+yWz4NvZwEzu9DRvADelm4djYcv1kYEzY3d1ZOm42ab9OWr+Q9Uk22VoJ8c7cUeXHkdvw81LaS4dTnYdf35UXUoQJlTPo+1Tds1vSxXQCPmz4WWXSkW2bJk9OPa62Frj9dr3tXNxyVlS42xdVEDWZdJgIVOYWmUZmui2Xndu79zRjc4ybCYjtIsT2eHNTQcHpGDz41sBNQKo+CcrEFhTYc5u+DxMz2rZts9t2dUA4XIJhw5aktGd7Xf76JufoNH+3d+1mJpIdW1lJ/QQVFXTn/rnPmZmQ/Nyn2z1PmkTNvddea2biWbSI7jbYMYsWUUtENJoYt8wX5defpITXlcLvn6jh85oZX2fT1BsvQxB0SUbnSnfRhSCCqvzSLEWtcsuWWda7iEzQNRnzQkxfkY3MKFtkKkOprL/99p3p7spmF7ZwIa2CxwdKsWyz6aRWiveoCr4sLzff+Yvj9JJL6Lm8bDHJ4IrjgaXDHHGMHcNW8uHD1fZaHXS2XBtfQV0d3SEsWJCw/VVW0nsJ0kNvYstljBmqdTtJ5ztOfpvdVm8HlkH0CTCtsqHhUddzxV2EjS/CFCJDorqaOu+YP0bc3dn0r63fyA1emVbJfZsKflfnh83Fzpf197Zts331XTqYLHPnAv37Ux8aw4gRQEGB9zZNID7L6NFUBojP8uab5j4M0fe0Zg3QtSvwm9/Qz5Mm0ecU/YxBwqELRuYxatQosn79+qTvqqoS3vjnnkts4x59lL5MxgaYPz9xjrhNAhImnxtukG+fWGdWViZ/1i0kbPB69dD7NXOIW30KBwBBQcFAX2YTnaDRIRQqSllYVq0KAZCNKQfjxsU93R9g9/7F7beMxcPGDZDMuqqups59L455WR/J3hF/PB0TbAHXzcUwSksfBACra8hg39/mfWfSTyb946XdoKG7ppdnYLj55mT24KWXUtpoUVGCLKKD4zivE0JGeXmmrNHwASp8mTd+1ixqo3v0UToJVWwAlVbBC3t2HFuhmZZ+/fXAxo1yYSBbrc87T88qUEGtQZtrZjLNmQl7xoH3qvl5ofupdhE2vHIbmLA6+GPddoVs3AAJfvcll1Det1dh4p1pBeiFPQDEAtHCAfv+tuk7k34y6R8v7QYN3TW9xt+IvqcNG+huFUj8m05klcCX0aeAhMYlo9mZDh7xuAULkhcXmTDgF5HLL6ed6YWCFoSZQzVJ2femi4psUVBN6EikJIXGGQoVobR0uTLQSkb9VPHKbWBLAXQTEPx4qK0Fjh0DmpupA86rMHHrIx7yBVyPeLyp3Vlrem0ZVP0dDsv726bvTPvJVoD7oYB6he6aXoL3RDNzZSVw3XVAKESvkZ9PzTrpfLasEfgqm3t1tftKajp4+OMmTJAvLvw12OJw+eV09X3iCW/cblNBoNPQ3TRnk0VFtSiUlEyUTvShQ5cYc8QZbHjlphDHxuTJqRNDtLubCAh+PDiOf2Fis7sJIojK5NoyqBblYcPs+5uHzm8mO9ZUgNu0y+DXN+Plmm4QdwVbtlAb/rRp9Bos3scmAtgWWSPwVVukIUPcV1LTwcOOYw7hykp1Z/LCYNQoKuxtt28MJoLATUN305zVi8r77QuIyhzQ2LhSOdG9pE8IOuWCODamTk2eGKJz0HSy1tbStBlFRdS/I2r9NmhoeAix2OGU71UasruAdqTfyrRwwEFJycSU+9GlxOjT5yoAYdYq+vS5ynN/M5iaOWyFqRfziV8Hss01TRcXcVcwZAjw9NOJwE8W5DlkiNk9eoGR09ZxnK8BWAI6Qv5ACLlN+P10AA8C6Nl2zI2EkJW6NmVOWxlUTlnGVTV1nvDf19XRCc4cwuw8nv8apJPIxJmncqRFIiUIh7uhpeUDOE4RCGkGEAcQRr9+MzBs2N3a881g71DNBNdeB13/uI0Z/vzJkxOsCN55a8OFljvU1Rx59TkJJ3xJyUTs3v2gdMx8+uk/sWvXPeDt/vx4chtvts7loGHSP0HAdA77vR+vDlyv102r09ZxnDCAuwBMAHA2gG87jnO2cNh/AniUEDICwFQAd3u5GRncVmrTlZg/bu7chEOYHcfvGoLezjEzBx/uHgoVJh2j0tCj0cZ2rZ+QI6DCHgBi2L37Qe0OwBQybVOnIXpxQvulEorQmfFM7KtsPNx7byKQhg/IsRE8Knt8JNJNKUBlpq/S0mUYN45gzJj3MGzY3cpdV2PjSohOXt5852beSwd11gZ8/zDtWJx/Eyf6p8uamnr97ga8OKGDuK4XuGr4juOMAfBrQsjFbZ8rAYAQsoA75l4A9YSQhW3HLyaEfEnXrqmGD2SekpUODcRNq/KqoTOWDrvGpk1XAYgZny/T7LzeK38vPNKhUbqNiUxpkQCwapXc/ALQdxL0LkhNfQXGjSOu1Nh0UWe9QKUd85RsW9qj2LYNldePjOHpljx1PKh7ZEg3LbM/gA+5zx+1fcfj1wCmO47zEYCVAH4ma8hxnBmO46x3HGf93r17jW/SKyXLq+MmHelzVVrVtm2zAXjX0PmdARUm+gkbDpe4OuXcNEAbNopJe7Yw2YFlVnsKK3/xQ8VVQW3/d7SsK/Z9uqizXuabSjuORqnQ57+vrDT3m9nu0v3SPr2yiDJNNw3KafttAA8QQgYAmAhgmeM4KW0TQpYSQkYRQkadcsopxo17fZk7diSzOWpr6ecdO4wvHRh0JpuGhoesMi3yECepbtIyJoabU85NoNsKDNsFwg0mZjyv22xvMNtRBWU2oU5g2a6CoL7+JlcHf7qos14XWZE9V1aWiJsZMSLxvSzSVQVbR68f2qcfE3Cm6aYmAn8ngNO4zwPavuPxAwCPAgAh5GUAXQD0CuIG/bxMxuaYNCm40GWvuwadIGZCQGRIDBu2RKv1yyapaqcQiZQYm1DcBLqtwFA/O/Fkz1ftwIBUphWriZBO7Yku0mYIgo5J+1Bu0mlp+cCVGpsO6izgfZEV2XNVVfScK64Ann8eOPfcBKsuHX3o12fnJwgr06m8TQR+HYChjuOc4ThOPqhT9inhmA8AlAOA4ziloALf3Gaju7iPilKM5hSN0kkfjdLPKk+9iSD3qsXotCeVEBAnZjhcgkhEnS+HMWeo+YSaGagzcDkuuOAT4wntJtBtBYbOXBWkqUPsm6qq5JoI6ZpI8ueT2/V1C7+NY1u1yPB59nU7uXRVK/MSUMWE3p/+RKtRXX89cNFFwD33AOPHA2+/TYMkWT4rE9js7v1WrfNqAg6yWp4xTDKsgZpptgLYAeCmtu/mA/h62//PBvBPAG8CeAPARW5t+q1pawrTavVecvHbZuFUFZvWFawjYD4oAAAgAElEQVQ2rRkbdJbKoOvVJufhT182SNY3Yg2EdGfUNCkYr+sPk/7jr0ELmecpj++oesO2c0OWObOigs7X8ePtiqSL91FcTLNPyvLXd0Zs+OQFsuTtmQQ+smVmVfK0oMFWdcehIfN33pmImJVpHjYecy8eeRu2ioqn3a/fNe3cex62zBnxWpni1GeCIcL6pqKCao0M6WLpqGDzXt36TzYeHCcfoVB3xGL7ktrvKJ69Vz66rI0JE+ju7I47KIVa15aKkVVdTduwSUyWLdh2YD2e+fA36BKhwXwOCAojx3A4WoCfnr3KM0snqwqgBI3qairsmRmnrIymSaiulnd8WVkiQRpfI1cWtOOlIIpNUQpVsrRdu+5BcfGXU87x6hj1UrfWzwJhU+jFC2ROMH7LnK4Jr3on5gVH5JRc1n+y8UDIMUQi3TB27CdJ35vw8NOxuPstNiIGR7Ii6SNG6NtS1TiurExNTJZJuq4NFi0CImfUIFT6W4ScVnTPO4gP/jUab/1rBM6/agUA4GBLEV57djiAVZ6vkzWpFVRQ2dZNAjOGDEm22dfVUe2PD13mz2EdX1REQ+5ra+3D9t3ssKa2U7WgJlKmh87RqrsnW8qk38yf6WKIAMAXvwhcdlly31x2Gf0+KMjepdd3kpoxMxWsX20WdF2aDb9ZW3XwS2e2CY4UryE6iysraZ75/PzkxGSRiL0PLoiaCbI2XqyJ4Tu/qMFvN34LH/S6Eb+eOQI76obicCyEv784Hr/78X8j/lYh3l34Fby78CvouqwcqypvNL+oBFmv4etWb9X2kUEcHHxbQPI57P9PPEF/Y+ly8/NTFw2VFnP22fbasgq68oKyCT148K3SbXxJyUTtPQXJqTfNs8PaCVrLPPVU2g8bNtB+2bCBbudPPdV30wDUuyHHKfT0TtwyZvILoc3OSD12wr76Lt3wU56SdxbPm0cTk8l291u2JBYH02AnlQziZY0bWBvLHj6M3af/F/71zxLcd92P8NMljyMc2ofBX1yPst/chXt+cgtOO+MN7NoxAiseAS79xpXmFzFAp7Dhsxd83nl0QjMbPLPRjx5NK8+YUsBknS1u9ZgduLwceOEFs2fyY0cXQaNmKyCzd+siWkVBqjIXsDZs71kX5em3EEsQYEV0LrgAWLs2YQMOAkEXDtG9SzEPj1//TyhUpFlcMh9hGzTEec1yJKlMN7Y+OD+RuLubduCFXQ/grXV98fuffwdl334ctQ9PxvcWz8cn/Q9j+18GIe+Yg68Wn4pw96tx660h7X35ibTNOg1fZmMDqLB/8UVqbuERjdLveZu7DqImwM5xy75p0naQAUa9e09TJsliWp9MwItCmi4a6ntS7QxUJpZI5GRNTnbvO5qgMGcOVQjWrAHGjg1O2ANe+jHUHlQng24XJ+bhkRUwd5xC6bmqXZRq8Y9ETjZ/JAtkOkkaE8JlZfSzGG/DfvPig1PJDREHWj7GP/f8FTFCg/H2NH8IB68gEoqj9HxgzJXd8fTd30efL6zFK6+U4GefGY/vLp2Q9By2vkEbdJgNf/duuV1sxw598ZFIhH5meeojEbsoNbfINj/BEDYRqCac62HD7kZp6TIp393Ebkz/L+9inq9twqlvaHgIa9f2Ugp7hkwk4eLtoez/zKZaVUWF/aBBVMOvqgruujaFYihiWhu5l9iMeLw50XqsUdm+zFc0ePCtoKE0yYhGDwZmx+ehi1kJspawDZ/d6/zWyY2Pj2xG3d6/4JH6W/HYe1PwydE/Y3/LI9jf8gjyQuvw8dFi/F/tONz637NQ+8Bk/GD6HkTrL8CiqTfhuz9LFvbpDsLqMJPO8OGjyL5969s7auZMWl5uBXVIt1OzHnuM2tF5M84ll9DqRIWFwLPPmlPAZs6kDB3RJDR1aiIntR+txHTbHQRtzguNz+u1dG3JkV4TgWhDZQVqpk2jk7FrV5pnfMMGat4Jyqyj6zcAysR1OpOeahGVnROEyXDNml6IxcyuFwRUppAgKJxe4GV+y+71W1OiWPLAZhw64//QJVKHUFucXcPR7nh+42dweB/dfcXjDopfK8LMS67BNT87Sfm8NvfVKU063bvTSDo2GB55JEGdYiHxy5YBn/88nbD8i8jPB/r1Az7h2GimFDDH0X/24zgydUiaOD7dqI9u5iO1QzBszce2LccXFMVSBZ6VMWsWHTeOA/zjHwlhz/fZCy8EI/Dd+ldtPntfadoZOnSJsUktCJNhLLbPdxs2qKtLpLdgphAmyGydp0HAy/yuqwOqH4lh5Jc/xu6mT7GhZD5+VHUKnn6pFBOH1WHrwVNRt20IosfCaH2pB/7vp9Nw2mDKFAiFQyj8WSEWLdJTVv3IHRt0uNNWrODOB11Mn07LEAapFfhxvjD4DVRyCz7yUzCFaWpBBjjpnIsiMllIgx87gH0gXNDQOXV170UcTyUlE9HYuDJlfAWh4QdJLDABc6KzuSymPfYSwJhuEEIQJU3tCui7h97C3z76NXoWHAIAxIiD1z85HTs/KcGRwwXo8Vxf3PGLK9ClqAv6nHEqHFGLDBjpTo+cNoh2MSCh2U+fTqMkeVsWb6tjtj7eVmdiA3TL9eFmW/TLQweCqU/rxmcPMmGZTmN3nPy2rJ7BJeEyse/yY2fJEhpFnckC1zLocgapfBsyYb9794PS8WUbwyDzE6UzDkJEbS0V7nfcQYX9hAlU+LMkaG7+tI7AziNb8D8bv44/7xiPh+vp3yt7r0NBpBlrPz4Tz79figdXj0Xhw+W4IXw1fv+Za/HXZdfjjM8MRN/BvdMu7P0ia2z4zDl79CgV8KJmL9qyVNq+Gx3Ljb4l2vXF6wShIbmVt1NT/5K1c91Ow83ubqOJeynh5wduOzmVDZ/FUGTCFqwCpdNOV/ya2n+qcSCC982Y7C7d/A2ZSKXB26X5dBef+Uwyt92k5Gg6sX7v3/DynsUoijQjEorjWDyMt/f1R0uMWrxj8RA2rhuEKYdL0b/PSfjSRefhtGF9039jCvjR8DtM4A8YMIosW7beWNDKIBPew4cnD5yqKqpBPPMMPUcUHCzXDgvQYAsPIcDs2almn6BMJckh9eIk1096b9dIhe0ilckatjrT28SJwIUXUrs8EyobNgBLlyZ2ivwCn+mweVOlwI7X7z6++D6im3e1AzmT/Snry7q6xGLAB1NGo6mBTkGDkDhWvPc77Gx6GpFQFEWRFhyMFuC9QyVojUfw+mtDcP7G03BqcTcAQJcuefjezyeiuKR78DfjAZ1S4IuBV17ZMTIfALMTigmYdEmWHn88eUDW1spti0HbQNWTPlnoh0JF6NPnKqlt1w3ZVNLOBir7rkrjD6IsXhAwZWHZ+EZ046uh4SFs3Tpbyr5JhYPS0mUZS65m6ncLwremQ2usGfdvnYem2CbkhY6iW14z9rZ0xcHWQuxv7opXnyrF6F0noVthAeb85+XofZp5gaZM47gQ+G6QCWumvV93XWKQAHTglJZSPraYMVHVJr/lLCxMXQDYdYPOROgWuSradlXXlWlsAALT8DMNNwHgRvfLJPNDBhMN2max1zl87SizYagqdKVjPNgockE5cOMkiod3LMDeoxvbviHID+9F10gL9rR0RzQewtbG3tj60Jnod7AAQ045Cb+5owIFhQXeL5pBnBACn+fpM1PN9ddTTv7TTydrDvffTx2/Y8cCmzapJ72oGbIdQVFRgton00iC3A773f6zdAay1Lm0b1ul13VbLHTPk25zgKlWqBIQ2cj8kEGlPPToMQYHDqwCFcxh9Os3Q5oSG1Dz+L0hsePLJhOeCZqi+/Fo/UI0RfchRt5Fj/wjONjaBfG2QjTH4hGsrR+K/Y/0Q0EshEvPPhPXzvtm1jtZZTghBD5vb7/2WuD224FQKDnHdW0tsHgxsHKlmgYmQqSNnX8+3RnwCdPSaQf2v/13tCH6MvA5b2x3LJnItW6iFWa7hs/gJjhVLB3TnDlqB7EItWbPoAva092D34XBK91655EtWN3wMFpiR3A09hoKQq04Fo+gJR7BP98fivoVAxFq03cKWh3856QyXPSt863uLRtxQgh8QB1ly/9+2WVUq+OLJjBnkExgL1oEbNxIdwRMI8y0o8/P9p+ZfUxtwaLd3tYnkWketwzZbsNnUAlOnS/G5v3aOn11Y4QX6Kb3ENTib2P22Xzgn6g/9AZ2HtmBcOhVhB36TI3HuuJvb52Lvet7IXKEYM6IL+DbM8cb30NnQqeMtPUKtgOT7cTq6hKmGCA5mk0lvEePprsFMWFRJgWESaEMXZIznZ1ehOMkc7BtozeDTBDnFbLcKZMnU5OfKNwnTzYvwBE0VPEUfEI8P+mqbd65LumdmOXU9B78pspmUEWZln5xEzY0bgMAEBCs2fU4enahn/PDwPtHTsbfX/ssmvYWost7Idx91WSM+Hap8XVPRHQagc/okixZ2pIl9POVVyZ49Px2nwl5nfBWZdnzqxEGsc2VtTF8+FJlu6aOO0KOYO3aXhg6lPLnbStQpbtilQlkAmLqVOpoZ0iXdm/Tt7oiNjx4IRlM3vtURKMH2/w6x9q/U2njpvcQ9OJPCMHeo1vQHDuMZ9+/H0V5G5IUux4FwJv7BuC1t85E7FgIPV/phidum4FT+qcn2+fxiE4j8Kurk+vRlpVRm/7Ond6LE/gtySaDSclAE7uurI3hw5dKzSayHC86QRCNNrbfk216ZNvjMwUxv0467Pe25SBtBLKXdNUlJROxa9fvDe++FaFQCSKRbkn+gvr6m7BpU0XSODS9hyAW/ziJoSm6B4eO7cPDO36FksJdAICu+cCmT3vjjR2DQAiV+oc/7YLPrO+Lp26Zjvwu+ej2k67G18mBotPY8JmdTwzYYFF6Ij2zoxx1bmwaPp85g2jXdQua8XMfsvayjaXjB+lk6Nj6L2wok3wbIq8+Eilp35GZ3I8aySwcnf3dpI+92vDjJIY4iWLbp+vx/M7foLiAFuqOkhBe33s6Gvb3xMGDhSj5ez/c8vPJCIVpBpiep3RHSd+cNn9C2PD5bby4VZ8yBfjmN92LE2QCbvVEZZNftOuq2BQ2W+XBg29VVswS2zPxIfCwPT5T8FLYwga2JozknZcsoppCpj0Tksh7z+/ITDKmqsBr3m72d5M+ti1XSQjB8zuXof7QH1EYofSZvEge1n58Jo5Fw3h3e29c8M4wTPnqOeh/dm8M/c4gq+fLwR2dRuAzyLbujJ2RzkoxprCpJ5oM950WK0huMsFUFbPE9o4XpMsfw0PdtyGsWhWS9gcTnGptPJSiEZs6Q+3ouE7SomK6eLmNN7eFIU6ieHDrr3GMrEHEiSESiqMp3hVv7emP1lgEb790Bq44OhynnNwd//GN0Rgwo4/h8+TgBZ1O4APJGS8rKpKpeOmY6DZQ2T9t8snLYFKQXMSwYXejuPjL0rB7U7t7NptveKTDHyNC1rcUdEem6w+1Ni5LhicX4mIb6vuRgRgtFrwSYOuzICSOx+p/i4ajf4fj0OfKC7WiS7gVnzT1xIFjhdh3pBv+/diZ+Fp4AHp074Lfzv0GuhXLM4zmEDw6jQ2fBx9cU1WV4N3zv8uomJmqsWlTTJxCxZEOA4gbFyS3vSc3wZ2JIKvOBpsEZTx09nZVtLSIcDjZ6ZqaPkMXXBVGaemDVkXRTXwWLbHDeGDrPDTHtiMSakb3vCbsbemGw600TQEBsGl3P+z78yAMIUUoPbMPfjbvWwiHw8rnzEGPEybwCvBXBKWjyqoB+hTDvXtPMYquXLVKHQY+blzw/ZgNQVbZDNOkdO7JzUyipfPgOI6UVgmY0XLFMeWmBOie751Tf4Ro/BDyw7tQFDmGPS3dECMhbNrbD9uXDcGpR/LoNeHgygs/hykzjs8gqI7AceW0ddPC2dadFT3ht+6AXlvPBHVPh1CosH1Sirnki4u/7ErVVMNdW/Ki4WdDkFW2wUzDV5tFZKDCXvdOwwiHCxCLHU76li+qYmLWicebsHXrbGWSvU2bKlBff1P72FAtQq2hfHSJvIMj0XwcjuXjuU3nYv+KvsiLhvDNzw7HA/dN6pQ5ak4EZJ3AF3Nhi7x6N7aOG/+et/+Xl6f+ni4TjzjpeRYGoHd+sfPV0OdIsbXFMmRDkFU2IbUfU9+76BsxqQecKGmo0vBjKcKewXbxjcUa23caLS3vp+TiaWl5H5u3/BCv7HkCpOtI9Di2CyGSSMAXRwjvFg7H8pcuwFkN9Zhw3vMYddIL6PJjpkRcbnU/OWQWRiUOHcf5muM4WxzH2e44zo2S33/rOM4bbX9bHcc54PWGeC385ptThTorf8cf993vApdeam7WYdS99etp8BYrrcYWl9Gjvd69HCYlC23P51FQMDAt1zcphycro8fD7ffOBF1heFWJRxOB3Ni4UlseUYeCgtMDX4BJ/Ci67XsCnxZ+jMbigWgN54MAONJaiKZjBRh6+C3cMvReTBv7JE7qfgCO463UZw6Zh6uG7zhOGMBdAMYD+AhAneM4TxFC/s2OIYT8P+74nwEY4eemeC1c5NWLOwBWA3f8eHdnrIy6d/nlVOjLqlsFBR03X0XnMzkfULNtkk0Pcvu+mzBy41m77RwaGh7C5s3fb7c7t7S8j82bv5/UdmeCjmmjKiRjQp1saflAwtl3B9/3tumxXduOR7Hmpb7Y0HgxSMzBxGObcVHZE4i37UwJDqQMKy95dHLILEw0/C8A2E4IqSd05lYD+Ibm+G8DeNit0d271YWqdcWNRc1++XIq7F94gTJ2WDsyTV1G3Vuxgh6nKmoeBPQamLt2pD4/rExZyxda93ZfFL17T8OYMe9h3Lg4xox5z5gvDgDbts1OcjICACHHsG3bbNfrZiPcis/LYKK5s/PZu1bt2MLhkrbfkncTvXtPw/DhS5N+69v3BwiHe7SfG4mUwHHMUxE4DvCDc9Zj/dyb8XrlPFxy8bp2Ya/Diezf6QwwEfj9AXzIff6o7bsUOI4zEMAZAGoUv89wHGe94zjro9FPMWVKqjklEklo4fPnJ4S7KPSZZj99OvCPf9AyhtdfTxcBFfOGJVMT8eab8sVFBFuMeLBFSgeTSa8zsahMKzzNjoeJ3dhv/puGhodc+eKq7IzBFezILOT96KCkZKLynGRhTI/nIesHVX8PG7ZEufjyC/Pgwbdi9+4Hk1hB0eg+xMkR84cF0NLyIRzHgeM4FoKcBGa6O5HMhZmCkQ3fAlMB/IUQIvUiEkKWEkJGEUJGDRhQLLXVR6PqABqG2lqaGbGighYtqa2lPPzp0+kiYKqp8yYe1eLCg5mTbG3+bNK7sWl0IfqiBqfjwrtNTrfzeSQmlYNVqyJYtcrBmjW92k0z8vYz79TNxOTv3Xsa+vS5CslCm2D37ge110sIY4LS0mWu/Wjb3+13QuKIkyi2bL9OsuATYalxB9+PNn0ahD1f3KWKbbr9LraVWxgoXHn4juOMAfBrQsjFbZ8rAYAQskBy7AYAPyGErHO7MOPh2ya7cit+YUO39BKI5aeiklvRahaA4yeylVZBugpq5o554XL7eqnJXO81a3pJuefhcAnGjv3EuE0v95muADG32AQV/TXoiOXk9k7DoeKL8MnRNTj1yLsIx45ZC3e3Orpbt/7YIjMnhSxew/Q9uL3nTBdpySakNfDKcZwIgK0AygHsBFAH4DuEkHeE484C8DcAZxCDaK5Ro0aRkSPX45FHaMlCvgi5TuDqipmzcofpDqjympFRF23JMmaalreTwYzzHWzGTRGlpcuTnLqbNl2NZMdhHkpL7w90snkNEPMihHXBSKWly5QVrvz0q+y+U2i+cEBArLfsqoUqQRX9AOHwyYjHD6X4Y9yRGoDGO/EB6lw+66z7Ut6DW1CbadDb8Rg86Efgu44PQkgUwE8B/B3AJgCPEkLecRxnvuM4X+cOnQqg2kTYA8ChQ7RCESEJsw1jzOhMJDI7fDSaXNuWtXf77d5s7jroHMpuUNnyI5ESDB++FI2NK6VO0E2bphttRd1s9zq7vWzba5+NcWCKXbm09P4k00TQwh7wFiDGhA9vEti8+fuu71jnuFVXuFrqi5bL8P7Oe/CP1d3w703TU9pzPAh7PqGazAfA3k0s1ugi7OWmSvFd2Tjx3Rzkpg70XPBgMowCrwghKwGsFL67Wfj8a5sLHzlCGTJAwkRCSKJ6lQ1UJdKqq+kCwoqmsELoU6fatc/gNyOjG82RpjOWwyRYSqeNi2XseKjolbqyeCJUi0kmUil7CRDTCR/d/apy3sRihzXvyj7d9dHYITyw9T/RHHsPAHDS0Y9w+uE3kA8zcxyPSKQE0ehBJO+0HPTrd41npz+DbgcjjgcbJ75bERa/RVrCYf959TtLYkEeQTttjdGnT0JoMs797NnAvfcGd42pUym9bNIkaoaZNIl+9irwdRkZTaGjObo5xnRaIdVM5ZZbtn1VDUaVZkoINOyiPEQiJbBxKqbDedbQ8BCi0dQoVDcWkk746O6POVTD4RJJeyrLucpZn8xo2dP8Hu7+9zW4850K3L/1MnQJ1yGGJkTRjL5N/0bIg7AvKBiICy74RLLTWoZhw+6WnmOu/VJa8LBhd3tyMuvg5rg2dWzTMZCX0n48fiitTuVsRYcnT/PjBDVBbS2Nwm1qAoqKkk0/mcqeaQozJ6nc6aq2t1Pbsm7yudmlxWyMut0C/yyiTdjUjm2qOanel6pCVPIzu7s1HScfoVB3xGL7Uu5D977Fd0n5761Ks0gcDnZ1H4ej3Y7BcYDmWB6aY3lYtbkUB57sjTBx8Jsf/BK69DSyQCuvfgKVs12EmA/Ke7v0wYJ3Zp+urDJHr+fdjm+S/TRd2n6nTZ6mM5HwpQz545kwthHWbE0T1za3vD2Zhkm0pWoXoCuY7TbwdCYRLyYZmYlIVohFFplpk/dHZXoIh7u53nM4XOIq1Ag5lpR3hr8P3fumJpRE24QcAdsRyQRPCAR9Dq3GK5GL8I+Xz8ORN7qj6AiwYNoEXLCMBq2//PK9rgIGMK8+pUJDw0OIxw9JfgkhHC5KyukTizVi06ar2wRq6qIoYtiwJRInPsDGhaqvvSoAuipz9Hfvdnzduaa5qjoCHWbSAfQmEjfOuwknntns8/OpkzU/Pzl3jlveno4AM/mUli53zWPDQ+3E0ufZ8WoS0UEuiPXpHRoaHsKaNb2wSeKQVJmy/Djkhg1bAtlWXwf+PnTvOxzuJvmlFa0gSlJumMTw0u1j8ecLZ+P1e36JNct+iQu+lshQog6+W95urtOZC01RX3+TdCcSiZzUZsJLfS66iLmbNUQnvszUJfa1jelEZZo0dSrbwI/5tSPRoQJfxrgpK0t8rxPGJsK6upra7FesoPTJFSvo5+rq5OsxH0K6Uit4RShU2P7/cLhEuz03SXQmgk0mUdN1nK4IhQqxaVOFJ1u7jebEyjZu2nS1VuOWtenG1ND5DEyEj+4+VFG3J508Qf380X3Kdh0HeH55JQacKS/x5zUYyxaqe49G9xlRdN0EHb8oQeGT4O/BJvGfetzFrOeGG0yi57ORCdShAt8NbsLY7fchQ6iQLytLUDFXrKDfA1TTnznTO80yXWCCONksoM9j4kUgqEwihDQZa20y2GhOJSUT2yavPsmXrE3dImeiGfLCp7T0QaNslXzeG1nU7a6PlyLmyBePT6M90NRaKP1Nrj0nIwgNnodsQVT1XSRyMtRO6WSYCjoTaqXNLk6360qvU1mObEwjntUC343z7vY7v4NgJiD2PTP3PPKIeWoFvzBlqai0mm3bZmvPtxUIOju0eG3TWABArf3KwIJ7dNBRPlUT2TYldOoETp0a/H1E40fxSePTEN9VCHE48WiKv4jEC3B234X47LC74Dj5Sb85Tj6GDl2ifgES+GU8qRbEkpKJ0kWUPo8ZwcNU0JnsSm0S1una6917GgYPvrW92Ex9/U2+GTVeza8diawV+G55bmzz4MhMQFOnJnYA/DE2NEtT2Ngi1dvqxkBpYLYaiOk1ZYJYZ8PX34c8Iyh/Ldki58W+z4QCnbzJ5gYWHHfKqVPx5+0L8OC2C9HS8pG0nZCDJEZNOHwyzj7njxh2zjU4beDVOOus+5LejSzSVIcgKIGqBbGxcaV0EY3F1OYoHo6TbyzoTHalclplnrUCkC4aJXMo876CdJnbgkCH0zJVkLFwZs6k/957b+J3IMHMMaFUuqVF8ErVdGMS2IR426Q08EMtk9MaU2mFQVxTRcljDBMZe0MVdm8Cr7Q51XnHQt2ws89nEXLou/moqSfO278a3eCegTLoMP4g0gWYpiZwu6bsfHovwQQiydIxAEC/frOUcQQypCPFQkfl6UlraoWOgsyhO3UqzZJZW5sQvjwzhzl8VTBJi+AlI6aJ9mCjcdpUP2JFVLxs62UaUb9+1wTijOJNDmvX9kIsJiuCltcuFEpL708KZopEStC37w9QX3+Tp+fTvUMvu6u8+GHsONwL/2o8Dc/VfwY1VRdg3ztXIx4vcL2XoJ13QaQLsM3tbz4mCdgc2LSpAqtWOb6C7FSsoV277rFqMx0pFvxWsusIZF1NWx3KyoDJk2nOHTHh2qJF7sLeJC2Cl0Lnuo5nK71N+L8sBUM0eljDYkksMvz5JpDx7BMF1eUaXTh8cpvGpC64zms+qqjWSKRHUuSkVz6+6rkAdUwD3z/vfnQ3ttXPRV6causyb8PBWHdsuf3LGNK9J745ajC+9UA5HMdBQ8P5VkXNg0AQtYbluyq5qQRIHZOqZ02Gnl9vAp2fyaa6VjrqM3fGPD1Zq+EziEVHpk4FmpsTzBzALCe9TVoEW6qmroQh00JsaZOibXrYsCW+iqjYQOeMAvIQjx/S7mZMc7FENTTFbdtm+9ae2HOoHMZHW97HA+tH4d3tP0N+/AictiNFK2e0NQ8nxW/EYw/PxW1LZ+CKGRfCaTPS8/3Ur9+MlGulw3nnhYIrgyOE7oqfRXhhNTF4HeqFGDgAACAASURBVJs6gWwjWIN6Zyb35nURER3xW7f+OPBUJFkv8EUTy4YNQGsrDaJavJgybZjg1mXG1HH+RdhmxNR1MBOGfnnU4vkqqCaBF1aH7J4jkR4pW2xxMpv7H+TvjQaDyXcFXrQn1XWioQL0bXonJUcNlXuJwuTnfvZ+fGX8L7XXaGh4CLt3P4hku7iDPn2uCtyeGwQnX2YqIeSYsVBO0FLN4hcAb31HBbIqR5R63onjHUDg1MwgFxGZWXjXrt8H7mTOepMOb2KZMIHWsL3jDuDAAaqBF7W9b9s0CSrnbHU19RPYZMRUZVAEkk0HfjJHik5hlYlHNgn8mEfEe6bOvlTwEbMmjl/dxNAJHRPtqaHhIWzZfh1irQ2IhrujKX8wuuEjhDkTRAxh/HXrBFw55AmFPFEXJlfdsyy6uLFxpfR4v/CbhdTGHCEjJABoW+DczDoJ8AFxpikgeveehk8//WdKag63VN+y8T58+NJAnedu2W9tYLIrDmIHn/UaPpAwsbAatiNGJDTwSIQKads0CSrnLGCfEZNpXCr4tenJVn+a7ySZrqaaBEE6l9y2sbRNubCnTtlUupxNHn7VJCeE4JU9j+PZjd/BO5uuQry1AQ6AvNghdD/6Ng4WnoIjoHzyfUd64vmaK3DbpAfRpYs8cMaesprd9lzxPdNAqlSIz60iJMhMbjrYBMSJGDbsbqPSkAyZdKbqYl9sdtWm48S2IJGIrNfwgWQTy513Ak8+mchxzzRwlkef2d7nzdPb3m2cs+w6OrBgH6+OIZ3WIxvAhBxDJFKCcLibq3YRpDAqKZkoLXXHCnnr2hTLGqo0sXD4ZGVpRP75WmNH8ea+59AcO4y6vX9BSZe96L//dYQErTNE4iD7mvHC07/F7Uu+j0heBJMvob8NjpnlVXdDOpyCQUH2noG8tgybCbOO7LlVwtNG2POxFC+/PMiV4CCDzY4mGxZf2121avykwkFxMfGczD+rNHzRQQvQ8oWXXZYIsLryyuSgFl4Dt7W9B51Hx6tNz03r0eU3MYmsDdK5pDJRsO9tkriphInjpObhD4WKMGzYEjRF92PbwTVYu7sa926+BO8cuB31h36PHvmNWL1zKMJReQrikwsP4rd3z0AkL1nHCSpHTTqcgkFBbi5oRSjU3fW5gxGScV8BcbYI2pnqBba7DFXe/lQQlJSgv9f7yiqBLzOzsAApJozvvZdGx/ImlrKyZBs+H3k7c6bameunXKEMXoWH2+DwO4CDFEZuE9bmWrqFjL7H0wA4yCvojwGDb8VrR/fjoe1fx7qGX2DHoTvhhGJ4dttn8eArF+D/nizH8NqvtZ2TCpVv4+WXB7VXGistXeY5R02mkpt5geo9x2LuCoN+jJnl1uGrS2VCGGfD4mu7sPXuPQ2RSA+jtiMR5LsfJUdWCXyZHf6ZZ4A5c1KPE9k1KtolILfVRyJ2qRlMwWx6r70Wx9GjyZNIVU83SCGquqeghJHbhLW5lqqt/IIB+HfcwRs9B+P9/udje6/Tsab5Uew79mcciBbhqS2fw8Yt3dDvvTcwq3Ap5g96GI9NGYS5v/k2hgxZYPSu0hFqH3Rys6DgR8jqA64IEpG1A9sKvaSC35FnQhgHOd695izy8s51NOXk42BbTb4dWZlawS39gS1kVbXcCqwA7mkWdL/rWEOi6cgk7Dth438fsspT6aivqWJnBBVOLgtNjyOExp6D0dT1FOw/VoR/7T4dsTjVS4625KPhiX6Y//UYQt1vA5yj0nsweRfpCLXPVvhNAZA89lLB3plpugZV/2RbjVg/783LuSbpK0KhInz9603vHjhABls8SjuySuAvWkQ17wULgPPOo4LzV78ColHzXDkqeFlERCGt+1xXl7h39ntVFb3eddfpncKmg0N1nKqItB+Tgu6egGCoaM3Rg3jsja+jb9MriMRbEAvnY3tBKf7Vci6aW/Kw/ekz8JNBI9G1K01f0LNXd3x10ii8+urgjOeSSRcyVWYzCGHq9s78LKIdlZdGB79Kge07l+e2ykMk0iOpolifPtM959LJKoFfVQVcfz3l2Y8YQZ21R47QAKsRI7xXpPJTN3fmTEr7nD07OZUDn7BNjBGYMyf5+2XL3BcaP1op0/hFyAam6SBMlwa8u2kH/rzjF+gSaURBqBUhJ47th09BU2sBPtp3Ej59eCAu7n8Gep9ajG//7GKEI6mBPUEI60xr+BMnAhdemGyerKqi6bnr6812gh0Nt3fmR2hnuj9M5kFHKAUm99Vpa9qKiEapwFywgArncBjo2hVYuTJZc7aBaQ4dFaZOpYKc7Q6A5IAunulTUUHv88ABujhUVtLPzCnM0ztlHes2sHUVfUyOt6GKBcGm4J/xWKgLPi4ailiPbuieH8IHR05GjISwYfsgdHuiHwYUdsXULw3HN/9U7tpuULlkgqBjmuLCC6kyA1ChLyo3XhWSTMLtnfkJRMokldJ0HnQE1dZvQJ0bskrgsy0si6JlAtaEV6+CLoeOaXuRCI3oveMOGgfA59AXmT4TJiQLf9lCc/bZ3iJf1VxduYYvJjmLRg8bc6C9DvatB17Dcx8tQfHRbTj98L/aOfH58WYMOLwR/45+Do++fjF6v9QTXUIRzJ9xEUY//Bltm+LiWFIyUWrCYoLHREsKMkrSBEyzv/56GkOydm1iNwi4x49kwvTj9t5M3plXgZVJ4WqS7BDIvFKQCWSVSQdINr8sWUI9/HxmzExqPvzuoLaWTsjCQuDZZ+Xbb6a1TZ9O0zPMn5+8hWcTdOxYb7nabWz4NLDGkaaWTUVii5rsoEtOkaDanq/e/Rje2vd3xOJN6JH/HlrjEQzaU4f8+FGIOHSoJ0Z9dgv6Djw16blUQkT3zKxaluj4yzZbMI+vfAVYswYYOxZYvZp+Z2JydPMn+UVHv7dMXt/GVJNtjmTAn0knqwS+mP/m8stp5sInnqCfM23b5IussAl55500+IsvwsJPwMpKapoSWTo81AOOQjfQTRkOsdhhZQIyETobLBP6/EIUJ1H87aP/w96jH2Df0a04ucvHaImHESMhtO4/gsGH/o2uec2QJ19MZWzoJrqtbTeb2TdMIbjggoSGL/qmdILcjy/KDdnw3jIlXLPhWf3guBH4vAD1WtEqaJhqVjZbbhP6ld/B57aoMJgIV2oyioNETkJTz4uxHY3oWXAIhABREsK/Pjkdr6wsxedPehtTP/dXRCJqh5b4XG6Tz9ZxlglHmxfzCm+z5234l1xCP5u0tWgRsHFjMgkgqDnh5b1lo/Zrgo7ezfjFceO05Qct/3/e/p5p+6apD0B2HdX96rJrMvh1Vqlsorr8O25OYSe6D4WNj6BX96F4ZudXsOnV0+G0OLj4cG/ULb4ar9UNQUuLWqjK7J9uzjqdbVcmcDJhC7bNzAoAL7yQbLNn/77wgjxtt2zcRCKUQFBRQTX8nj0TfiK/sH1vfgvUdCQy7b/JJhhp+I7jfA3AElBV7w+EkNskx0wB8GtQNeFNQsh3dG261bS1Qbrtm+mAaTCLn/ZtbN9N0X14/bVzEDu227Xtfc3FaKp/BNN/cnHS9267itLS5dYBULaxB+mISZAhneYV3fUY80ukAfuFrdbb2c0inRlprWnrOE4YwF0AJgA4G8C3Hcc5WzhmKIBKAF8mhJwD4OdebsYrbFMjZwN0VaWCYALIwsuZMOTTCWze8iM8s3E6HtrxDewu6oa4455t4+TCgynCHtBr0QUFA6WCwy3UXhUm39i4Usq0aGxcyR0PAOF2BkYQFYMYgk68xyBLIFhbC9x+Ox3Tc+YkpwqPRoO5rm06gmzISJmDPUxy6XwBwHZCSD2hlI9qAN8QjvkRgLsIIfsBgBCyx++NqQa+LBcNYDcBbdv2eo4J0pl0S8ztIhOSJN6MLvsfw75jRajeeSH+dqQc+5uLQYgDVTUjXaFrWcY/x8nXlnJ0e35ZjhqdwOndexq3kMTavg+mYhADT8etqqJ/4u8mY0McV6NH0ypuM2cm2pkyBbjhhsTulV33ueeAHTuCG5c2uYCCSoLmNVdNDt5gIvD7A/iQ+/xR23c8hgEY5jjOPx3HeaXNBOQLqgIlqtq1NpkvxbZnzqSTjG9bnDS292ODTCTdOtCyC0dVRcmjx/Dsb8di5uEKTA4twiVle1BWJq9b6laHt7T0/rZCJxSRSAnOOus+11znts9vUoglXUUweJNht27Ad79LHbBM6FdVARdfTG3u4nmiIBbHFUCpyNXVqbtV/ros4V91NWWzpWNc6qBKqhaLHTYW2ulIYNcRSJcymA4E5bSNABgKYByAAQBWO45zLiHkAH+Q4zgzAMwAgNNP12sCNgVKbKNpxbarq5Mz+smccDb309Foiu7H0k3/D93yd8Bps6lHQnH0CxcgL9aSekK8N176400pRay9OLd0gTc2rA63Y+WObwctLe9rWVBBmBxER/7ttwPXXEOVjTfeoLb1a66htvYRI/SOXdm4WrEiEfdRXp56XYAKk7lzKWW5ujrz45L1xdats5OK1USjjcbOW9MAqGyHFyd+R8FE4O8EwCcZH9D2HY+PALxKCGkF8K7jOFtBF4CkwoCEkKUAlgLUaet2YdMKViomze23Jz4DieRsLBkba7u8HLjpJvdJU1ZGk7qJ99MRdFERO49sxqP1v0R++FMUhFtQXEBz1ByNUhNLnISw/uMSfOPkvyMSaW0/LxQqwvDSxSnCniGoUG8bVofJscmLUXKQmCxojCEIto7IIGMCe+RIaluvqADuvhu44grzimr8OAeSK7xNmkQFO6MmX3opPZadW1YGNDf7i0g3gWwRjkS6pVQnMxXamU6nkC5WTmdSBk1MOnUAhjqOc4bjOPkApgJ4SjjmCVDtHo7j9AI18dT7vTmZmUa2fRo9OrXmbFkZtXvy2+VIhG69IxH63Z130shZRhZy8wEwwV5URKOAa2szt4WWoW7P3/DbjZOwZONE/H3nD9Atfy8aWrri3SO9UP36+di66AK03DUOLXeNA7l3HGaffRfOPfd+pb3czZ7qx95qY2IxPZaZgujziMI9kaudIV1h8WVllDXDImife46OC1O/kljCc9KkhNlmxQq6+5w0KWHiueUWuntg47qqKpmu6bemgwwq84ufnZRXP4DtOMyE6ShdTvyg4arhE0KijuP8FMDfQb149xFC3nEcZz6A9YSQp9p+u8hxnH+DesluIISYhXkqoDLTVFaab59kK+8ddwC/+Q3V8iMRmiYBoBPKceSJzvj7YVG/l19Og2by85Nz66QzJoCQOJ764G68e2gtgGacVLAXBeF8HIx2wZ7Dp2DVmnPQ+5/F6BKK4LafTMCI6lJJK3KN3U2r9su7ttHmbDU/tXChEcLp5lrzAve55xJjtLIyWWGR8evFcd7QQDNoMpSV0fF1660JDX7OnESEripLa9AapmoRVmdqVfP3maZNC6nnAUjeceoWZS/jMBOmI1E5VcVSdDSMbPiEkJUAVgrf3cz9nwCY0/YXCHQBTzbbJ3G7PGcOzb754ouJ7W9tLRX2V15JnXCyRYXR4th1rr2WtvmlLyVfOyh7HiEEq3ZXY9OBl9q/Oxb7ACcVHEBeuABREsIbjQOw9snPoMd7eTjFyceKRd/DqbNLNK2q4TYp3LRut+2yTWCPbRCQ+vj0c8Jra6nmLQrcK66g3z/9tN6vJI7ze++lGVrFwL4330wVJnyWVsbF95Ic0AS6oLxQqMgowZgorKPRRjhOPkKhEsRi+4wWZS/CO92mI1sfop/CSn6VxqyKtDWFqW0fSF15e/ZMnTx1dQktnQ9wYWYisfNkpiZxYfJiz4uRVvxj5x+wv+VjNDS9jZO7NCAvFEacUNNEXjiE2o+G462/nIm8JgcjI8VY+78/QkEXzyUu2+E2KdS/v2+kcdlkHrTNUpiprIayiVhdDUyblipwb7+dCns2hnhBDCQmr1uEtm6nqxuDQWuXukV18OBbjezjMmFNyDFEIt0wduwnRvehG4cvvzzIt7LhBbYZed2UwnQ6gbMqlw4PXfQs4I29I+YzUW1/dSUR2bXZfVRX08yYYhumFbaaovuwbs9f0Rw9jPqDz6FnwWEAQGs8hPV7B+HVv50FHKUCP9TiYPopZ+Ln865QOlm9wi1yMpPFV2yP9XK8F3iJ6PYbBS5bZC67jKZkWLky0ebllyeS+qUDQeSfSWfhmgRSK0QBwZXlDApukdq634+bXDo8VJoy4L59YpOEX3lraxP5TFh0omolVu0gpkwBJk9OvQ9xC+5mz2s8+j4+aHoLHx6qx67mx1EYpjbM/Egentl+LrZtGACnCbikeQDWL/ouwmF5AFSQcNOSVb+r8gHJNDEV40clrG0mY7oKR/ACt66OatYsZuPNNxM7QZ1J0Q+DQ7YD6NeP+o54pFtvCyL/TLoK1ySjtT1LLNttDh++FMOHL02bQuDFBONmpbCxYliBENIhfyNHjiQmmDePEID+SwghCxcSUlOTfExNDf2e/9yrV+K4mhpCiooIWbxYf554/rx5qe3IvpedK56z4m+7yAeHXif3bf4l+eOWL5EHto4hD2wdQxZvLCcT/jKbfOneSjLqxvnkkaXPG72XdGD37uVk3bqBpLbWIevWDSS7dy93/Z1+RsrfunUDja/50ktFSee+9FJRyrU7Cnx/1tQQ0qMHIQUFdExWVKjHgQhxHAd1X7qxmG0Iqq91487PWPQK1Zxn8kUmr2bM0Pefrn9ByTKe5G5WC3w/g1o8d/FidafIzlMd5zZxFy4k5G/PHyKHju0ih47tIo1Ht5Mf/2EBmXLD78gDW8eQKTf8jly19EYy4bHZZPyf55AxP55Pfr94B/n1vCbSeqzV/AGzBH4nsd8FIxPgx1LXrrT/x44lxHFSlQi381Xj2ESRESEbixMmpN7T4sX0+2yAm1JhA3Oh7wT4BHK4KYn85+JiqjjMmJFQJHilgi0GqjaOS4HvJnhNIE4IvxNPd3401kKaWw+Q3238Mbl/y5h2Df6BrWPIfVvGkP/3yhVk8jPXkNG/+l+SV3CE3HfXh6T+7ffI889Hs1JDM52Yu3cvJ2vWlLRPrtWrS8iWLbOMJ3VtrdNhk9QGbCwVFlLN3lTDNx3HNuN94cKEAsMrNOx7fiESPx9PkCkbHak8qJRBUW64CXr2u9gGW/yPS4HvReMRj5UJZ69ba7WppoEsfuubSQK+cv0k8t0Xv9f+V/6n68mPfnAnefGxdeStf27O+u24qdYun3B5ZNWqfGONvzNp+OXlVMPv0SNV0MqwcGHq5GUT282UqDNBugl19tlmF9JZwSsmq1eXWI09U9iYkVVz2nQxMJEFx6XA9wOVcBa1IhtBy3f6pv3ryP+8/TUy54GfkSuu/x35w+YvkzmvfIvMWvttMvEv15Irpt9OHlj0ZPvf5td3pLQXtE03SJgK4SDsqJ3Rhl9cnKqhuZ0r+ywDGxef/zy9Fn8uMwPoNHyGsWNJu+kpXQjSPJPN9+TWj6afbRcDFfwI/KylZfqBzGteVZUcCGNDjyOE4B87H8SWTx9G2ImiW14zmmJ5ePdQL8TiIax/awjO/md/9CvujvGXjMD5F52nbc+NktXRMKXPmZZRlJ3LI5tL5fktu6mj+IqsDkbxnTWLpu6IxYBwGJg9m352nOSobhX1V1Y7N4giKTyCKhOYzX3Pg+/H224DfvhDmi8JoOPivffo38qVieNZPzM5U1dHC9evWUMj9hlV/KabgC9/mbK+TGTBcVPTNp2woU6ta3gSr+59AE4bvzzsHEWP/MPYf6wQB1qLcLClEOv+UYpzNp+Eovw8XP+LSRh4lpgxWg43XnY6o+xMYVrNyKQ2r+pchs4y4f1AFMyyMXD55ZRayQRBbS2lf7a00L+iIuCZZ1IDskSlQVU7N2ih71b/2DR2YtOmq8GnVgDyUFp6f1YuGqwfx49PLlmpe8f8fGZ9GovRYL3hw+l5RUVUEQXMlNCcwPeJOInhifd/hw8PvwaCwzipYC8OR/NxJFYAgE7E+gOn4M3qs9BnTx4G9OiG26quRrfi1HzgbnAT6H4DdYKAqfYmOw7Ig+M4oLVy1OfaXKczQyWYxe8nT6bxHOKu9MYbgdZWmuTv2WflY4L/fPvtwIUXJgueqqpEoFZQMNndufXlmjW9UjJtAkA4XGIceZupMST21xVXAPfcY7+Lqq0FJk6kMiUWAwoKEsKe7QjclLvjUuAHremK7bXEDmP+w49j0xs98NXv3YueBZ/icDQfMRLCjoOn4OXHz0HRR4nqTWX9++M/b5uWkSCobDD5qLQm8fuSkokpNXIBswCdzlwX1WR8ui3eumhspg06Ds3bdOediR2AyiSUDbtAEbq+XLVKHS0+bpyZXMrEGFL1Y9++wNtv0wypq1ebt/fd79I02kAiqMpGqTsuI2295pNQTcQdO4BFt8fw0yXVGPbFLXh19SHcf93N+NFvf4W8cDNe/OAsbFxxJsLNQP7G/8B/zR+Jiy7OS2pj8eLMTKi0RdlZQBa5KstUuHv3g1JtKugMmtkGk/Gpy7EC6KOxWVEeZrMvK6Nmn+pqefqEdOTP0cE94pUiiL7UmWwyMYZk/XjFFbTfxo6lGn5VlbmG/+STVLNvaaHyasmShCkv3TDJh98h4EPSbQqTs4m48vlPUfvx/bjjsWdx+beacfIFf8T37vg5qn46EY/8dhD+eN2vcPbs5XhxU2/88fcX4YvrL8Pr/1uJuj/+Ev912/mYNj0vsLJxtiXQaiXJ2Toa1N56laeygar85UHVRe0IuI1P3Q6AXyxYqUKxzOGQIckO2rIy+nnIkMw9ow5iLWJV/eNw+GRl7vpIRJ7Zlf/eLZd9JsbQ3LmpprZ77qEK4OrV1JzDl7hUobaWprOOxWga7YoKKvSbm+lCnglkrcAHkjVdk6ICjUffwymf+ztm3/kYrryS4H9uJZg/80u45n9uwFnn/xElI7bhpLGv46m7vo/iIW9hwefH4I3bbsLrd/4SP/nF5e0JyUwXG1NBblMPl9cUVcIg02CTTpYkDdBrU7oJK6uLapPlsqMLYOvGp67P3TR/IFXIsOM6sqqaCL4Wsaz+MZCHePyQUlgPHboEtKZSAo6Tj6FDl7T37aZN07VKht8x5AW8wxag/95xB/1eh7o66l8Jh4ENG6jQHz+e+mh2ijUE04SsteEDZrbs/S3v4WjsIFZ++BAcZy3CDn2e5Yt/jBfunY7TLnsRgya9gHjUwaG/noOP6qbicyNCWL8eeOIJJ4n1INpA3TJe2jhYTe3yfmmA6YCbvVZnL1WdGw6XIBLp1vYbzbjJUu2amIPS6azjk+/xDAvmEGUlMpmdnSVSs8l4GMT9qWz4HcX0Ek0vsdjh9kRmPPjxIjPXAKnZLVORoPlmkukVxLtlrJ7p0xMFcxYsyIwNP2sDr1TBCy++GCdHWveSxuZ6UvXWtKQI13n/uoxM/Mu1ZNR/3kUiXQ6TH313Dyk5OUZWPPopefKvh0ivXvH2gBmWz0IVQGMaAWcTKWcbYBFEeokgoE5/4B4kpTvXph0xoIZP5xB0hK4YqMf+Pf/8ROQqC4Tq2jURTRtEUI3N/XkNBMoUvKbNMAnoY/2c6eCvIN7twoWJ9Bx82hfTLAI4HgOv2Er61XEJM8IjK9/Hn19YhW/9+A8AgBhx8Na+Adi1vycOHSlE+Mm+mDRqKub+egAe+4uD8vJQu5Yl0t4Y93nUqFTtzJYaaZL7vraWFp/+5jfpqs63bRu4k2knro5zXVr6oFabCoqrb+IgpDDPra4De++shOD06cBTT1HtPhymY/OVV2iZTL54joylk46+c2s7m8eNG4PGnfLpoLSU0lw6gtbr9936Pd+Php+1Nvzv/+w9bOg1Gcu2j23/OzZsOi6b+QBe3XMGXto5FMtf+RKi930V399XgYV9r8Ezy36BfS0D8dfHwygvp4/G7KNDhqTmvL/2WlrqULS/8jZWZo/nbay8nd7Ewco6+JZbkmueVlW5O4Nt/RjpgMpOWlr6IABg7dpeWLXKwapVDtas6ZVkS5edq0JLy/tSe7ysUpIKQTnr2HtftoxyrZcto+Pl6aep0H/xRUqT5B2rfJEc1ueVlbRspldfjMpPVFeXPC7q6pKPKyuji1U2jpvBg2/V+l/c+5AYld1MF/g5OWFC6kKrImMsWpSY88xHV1lJFcFM+eiySuC/vW81lmz8Gu7d/FU8/eF30CN/D97a3x/rPxmI9Z8MxCsNZ+CBlV/BKY9diC+unYR7zvsJfrf0x/jatAvw2S+fZXUtnaDmHWbM+ca+551vMgfrpZemeuurq+kOY84cesyCBXSg3Hyz++qeScaOahKKjIyCgoEYPnwpAGDz5u8n2WljsUZs2nS19lwVO4Mi1blnSrEL0lnH3ntFBaXdVVTQzxs2JIqNiJtj3lHLiqUsWJCw+bLAqJkzk/uxtpZ+JxMUKudvJJI8LiKR5OP4wuom48aWSWYC3bgRHfmbNlVg1SoHL788CCUlE7VKAm0vvZRM3fuYOZPGRVRU0HdcVZXoQ50CN3o0nfOVlYkd4YIFdOHgHfbpRIeadOrq6rDywz9g+8HHEQ5F0T3vCI5E8/HBkZPQGg+j7l9Dcc7L/dCrK+38Ll3ycM3cy1DS5yRluybmGFuTjWoL5jVnj2n5Q9v79AMvTlCduUa3bTc10biVV0w4foN11vHa+YIFiX9ZdGUkQs2DTz2VCIYCkimXOlMLC6hasYJ+x382GX+ik0+83wkT6I5k1ixg0CB6v+w5olF5NGcmx5pJmcJ+/X6IPXseTXH68mMynUFXqvdRWUnnLOv3DRuoAzY/n3Lr3fj0QZjaOmWk7WnnnExueHQ0ivMPYf+xQhxsLcT+liKsW3k2ztpWjG55eaic9y0MOLOPddtuL9WLp10U0iY8a9n1bTo8k2wLL5NHb2vV29J5ZoVbG5lOwaBi6dx4I/DOO8C4ccCrryYm/9ixDmo40AAAGs1JREFU1J5vEkELJIR+NEoFR36+Wtgz8G1266YeF4cPJ/K9bNiQvFj96U/0flSMkEzZ/U3SMrD0CjoGjlxxcAAQK8aXCrL3Ida2njWLjpeWFncFjsFU4VOhU7J0+p3dgyx6+yLyw5cqyPnX3EIu/dZt5Kff/19y5FCTqbNbiyDZETImjpu3nl2/oiK5neJiyuqQndOR0LFpVOwHHZtCx5YRmRWrV7szbtLFxrCpu8Afy/quooIydUxT4bI22PhgY0TH0LBljJWXUwYRYxcxRsjnP68/3y97xBSmabVNkFzuMHkMB5FmWydH+KI4pinXTWseMMjGJ1C8hXiUux0m8Lud3J985Tv/TW65cRmJxWL6t2SJIAuM6AS76jq8MOALUMyYkZzjnB0b9ITyArdJqCqAIhaccFsoZPnvaRt5HZIT3w/NTlVRTdcWW/SLiuhffr6+SInp/YnjktGOmQCPRNwVIFY4hVXyEsuCBgXTSlU2SEchHZ0cYe84Ekks+DydV1fghrUjFq7R0cP5KlnA51pJZxP46SqAEjQHWVxhWfEJ1qFs0peXy68nTqJs0OZlMJmEsskjljh0WyhUE3PNmpIOK6bhRUGQnWNaGamoKDEmmPBnRVVEmO5AxONqaqggysszK7ouxh6IykrQcCtEvmZNieRY9dgIulSmiaI3YwZ9P3xRnMWLk3d8PGR9yY7XjT22uBQW0n/9aPhZy8P3CtHuvWgRdVpFo/Sz38hV3pkDUC7/sWMJO6wskyHLjufVZpcpJOylKoeaPrpRdS7vBzAtrpJpmNhV+chnfgywwiUmNu9Fi2giv+bmxJgoK6NtDBkSnG+mtha45BJ6ncJC4L/+K9kBLd7rxImJtMp83vdIJNi0yiIaGh7C5s3fB59O23HycdZZ97VnZzXx3wTtwNX5z4Dk35hPRhVx7QaTsceOKSwEmptP/5iQD/pZPxSyjJYZBMQcJKNHJ6hxo0fTjrn88mRapU1SNEavmzSJUjAJoXnKV6xItCUOkueey65EaCqw3CiM9iaC8aNV+XFUCwVPk8vGhGm1htTXHTtov1dXJ4T9pEn0XzEXjgpz51KGDz8mAJoBM0hHfHU1VULKy+m/I0bQe4xG5fd6ww10nlRVJeioL7xAF4F0onfvaTjrrPuSqJtM2APyGAwZ1942p44bDVWXy0j8rayMViWTxfS4wWTs8cfQdF+n9jW/ggCvWwO/f+msaSuC334z26Zf+355eapNVLZ9z4YQd1u41ZlVb8XDRs7XbKhhy7bX4lZ9xgz3rTUbP/xW3hSZGBNer5EpG74NbEw1No79IPvBq8/Q1Ocj1lQGPh8j6bThA/gagC0AtgO4UfL79wDsBfBG298P3drMpMAnJNnB5pfBY9rBNgyQbINu8rjl1pEJc769NWtK2pg5HVf8WuYM4yeWqo+YDR6g/9oKiEyMCa/XyBRLxw38WDFRIrwiCHKHSmizcSUe68bAUR3DXyetLB3QVIY7AAwGkA/gTQBnC8d8D8DvbC7cWTX8zqq1BwkdI0K2UGSLVi+CsaZEuq0bRbKwkM6c/PxUbawzLOYqBCEA/cKEPKAaO16ou36VPxWpg98puu0cba+DdCZPcxxnDIBfE0Iubvtc2WYKWsAd8z0AowghPzU1JWWqpq3oZBWjHG2jCZnTTUzEFrTTLZthGwiVraUMa2upH6apidqsr746eazIolHZ+Pn612lYfWEhLS4OZL72cJDg50m6I2118Foc3Utw3v9v79pjrDrO++/jLktZ8O6WtbGNDY6xTL0RVkIKTRyF1mtsK4sMDqbCW4lH0qiAqzZWqRt1hbAbUISKAQlLqQuN7GKcFuyKIKJiRYm9laWoVIvk2LFzY8ds8Gu96wQX5LQu+C7TP+YOZ+7szDlz5rzuY37Sau/j3HPnzuObb77H7xO/Mc1Es6igjjT6MmvytOsAvCM9f7f6morVRPQKEf0bEc11aUwWkInQhod56rOIptEVnoiCcLqpRFhHj7pXxGo0mDhSTAurnksZtrXxlPhDh4AVK4KFqnPmy2UHn3qKF734+GOeWt/Iwh6wK8qSB8xz4hJuv/0SbrvtjHaexSVSkwWza6EhneMX4LxZ4l4XL/I58o1v1MnciDoCAPhjAN+Vnq+DYr4B0ANgWvXxJgAvGO61EcApAKfmzZvndrbJCHHtnvVw/K0XRB2ls0iKSQrZFKdmvZrGU54j4rFs8250k049wHWuxI3DT8OXEmbeFXOqoyM66zpuG5DApGOj4b8HQNbYr6++Jm8aZxljF6pPvwvg9w2bywHG2GLG2OKrrrrK4qvd4ML8F6cMIVAftMX1gKiao0AxZeiiIDTaw4c58+G2bYGm39+v12xVFtVVq3hB6m3beCFqEe7r4Q49nTbhwoW3QstYxg33TaOEpKkUKsDnQ0cHP0HK18lyKa7MSQM2An8YwM1EdCPxApQDAI7LFxCRHBe6EkA5vSbGg0i0Uqli77knmnc+TtF029jtZofpKF0ub4ikV86ySIUJQhkQC/vIER6b/tJLXOALoX/mTO3ndAoDYyIumv+PcId5WKB2rgCCDA2AVpkQyFupEPNI5cY/fJjLjoEB7ts5diygV1dNZHFlTiqwOQYAWA7gDfBona3V17YDWFl9vBPAa+ARPEMAbom6Z9bUCq4p4jZee5tInUYOyYyDJOUPi4A8ViKiYsaMIP68s7O2lKH6GQGVBM2bdNJHXPNOnuUOTXLmnnvir/u4kUJIYNJx+lAaf1mGZYrBWLo0sMvG+ZxNfL0aZytCr/r74yf0NDKiSNeKtNOboI7zxo2T489tOU68Hyc7pM2PkxSqEicS1RYtck9Uc5lDXuBrIBbw0qXxaEtt4+tN18uD/sILtaRHzSgQouOmi1mcUVA1c92iC9O8VOZTkQUpqK/ThCm3oSiyuTwwNvY0GxqaUldKhLrmN24MiOlcEtVcc3qSCPym49IBasu7lctBDdkwO3vcsDST/U2UMazbsKyUIWyuPE56MnROs7B6pnlA9r8IZ6sanic4ZcJ8NMJ+b3qeBnRO8XJ5LcrlDUZHual/i+53WwhCNUBHpje1MGe/uuaPHOE+H3mO2Dp+d+0KOJlkuXDffdmGwrZld+tiMDTEB2P3bi58hed7cDCIvddBN0h9feFCWnbYCNZD9fWOjmBCRN2vHhFWcUhAPNclvqiLU02QEcJKvk+WUBOKxsf5whPo6+NzZds27nQT16xaVZs4MzDA/8uJO2kl1gA2zKUTNc/kmHNd/54//xOMjR0srN/jYGRkK2T2TBltbZ2Ftlde29Onc+JEsa7jOF1Pn+bzTswjkdQ3MJBt8mbTafjDw7ye7JYt/LnYlSuV9DvSFKkzNGQXllXvsAm5FLCNxImbIJM21JPc/v08kkLWqiqVQNgDfBEyFmwMYtMYGMgmNLe23+1x4cLbxv4dHT1QaL/rYDpxhCXkVSof5tU8LcSaFyykAiaLgClEHOAnwq98hSuoIoNbbABZoen48JPCto6sKRVd8I3fd1+tFmhK169nZEGJkDcfflp1gXVp+EA2NWCji3zrMW3aDRE1gsNQAjCRSi1YG4RRIYSdbET7ok6dWUBQcezYMdl6YFIowygrgIDao6OjVskIQ9bUCi0DEcO/YgW34QL8/4oV/HU5Dttk8//xj/n//fuDo57Y+eMmdhSNLCgR8ubDTyu5RU20A5Kn5pvg0r/CfGbux6ilPlH9bvMpLk2EnfTmz/82eMqPiqno6VlufepMG8PDfPx37gzs9YODXEMPS9AMi7UX+nZeercX+BJEsZT16zk/yt138//r1wdFVARMmXonTiTP4KsXZCGc806QsUluscnMVs13qsMtTe4Zm/4lakep1APVfKbrX73wNCMPU0+YMiGKorS19Vx+vVTqQW/vkzh79kRhpqlvfrM2KOPhh7lc+MEP7H19wvQnbPbt7XxOtbfz51mbfJvOaZsEsnBYuBD40Y+AW28Fnn22sYmxXDF//retHLFxIFcyyvJILpty5AW3bNnkcRSnACGw29qCMoAAP+XJTlzhoFPtrWk55XX9DkxFW1snKpUPjX32xht/jtHRA5AdutOm3YBK5beYmDgbqw1ZE9uVSrO0bSqVZgHg80Q3J8rlddr75UnEZwrWMEFVFkRJS0HEJ+bNqlX89SzlTEtp+DaaXF8fT5H+2c+AuXP5//7+1hP2QDxKhDghf6KUYhj7oYyk3EiyE314ePK95I3+1Vf5qW5wMNDERD3RvJgkdf3e2/skvvSl31zuMwA1/f3Tn96J0dHHoUbv9PQsx8REfEdn1iUnTSGsUaGt9VAiMw6tiomVE6iN6urr489vuinbtreUhi9rcjoHChDE8N91F7fH33UXf/7ZzwaRP60Ek6YlI+tQS5txUyGEssxHHsZbL2tt69ZxDf/cOb6gdUf2rENsw/pd198mJ+fo6AFMmzbP8H7AUyMjD2I7U7RNVBROFqdOWwgfn1wEvrubO15NDtew/J6oOWUKOADmXO36G1pKw4+y5w4N8V1782ZOprV7N/+/eTN/XbeTu2ifzQBZoy+XN0TaVZMk/biSTPX1AYsX1ya+iXs9+mjtuIlTwLJlvMB4f3824ZZpJD7pHJ5mTBj9JnPmbJZIynjinCuxne63hf1eV029SCK+JUv4/JNPfzt38nliOu0lYeU0BRwA/2M7+JPQUho+EG5/Gx7mO7W8Ky9aVPt6mP3XVvtsdEwOqZvQXifsqmmcAOLaTYU29vLLwdG7u7s2fE5XCW3rVr7JP/QQ1/TTSpiL6gObBDeBePbqkpXfRP5+sVHbjo3ut/3iF39aJV78RPt7k2jqNqfOLNDXx097a9YEp78sfXuyoiOH/t5xx/mPXO/ZsHH4rvHVunjqpAOWxT3rGbZx4iJeP414/rh9vHcvF9oi41p9Lt/zM58BTp2qLXspYqvVDd0VYX1gEn5xS0bqMGfOA1iw4B9Cr3EpDyh/tlzeANOmr0Ie8zibXF6wkSvCp7NtG7fLZw31+1oyDj9ufPWuXXzRyw6UwUEeY580FEoXdtXMsNEwZW0taTy/yfEVNm6VChfuO3cG4XO7d/PXBcS4Pf98YPIRp7stW4LjeBoOWnMfvBU7+9hkounuXoaA06hkJewB9+xnsVHYCnugth+Eps99DPxkkSSePg2TmUmunD4dBAAIh+1jjwGbNjk31wpp191oWJOO6bhjErZLlgRZcrL9bfv2cI4dG+jCrppZ6JudgPpC06brbSMr4ji+BIQ2du5coB2pTnfduOlOh8PDkxWJuNm6YY5Tk7Zu2iTSDm112ZDjavYC8pin6exP614muQIEZj9xEty3jxOoDQwkX++6k4UpHBjousL1expWwwfiadZ9fbzjhMYns1smSYpy0T4bDarm1NOzXKth9vYe1IZaJk22snF86Zzne/cCe/botSN53GbOnMyoKjve08jW5b9VF3PIEIdpVECEtvb2HgLA49Ndtdq4DlQbzZ4ne02teU0d8zR5ldK8l06u9PUFnEpi/I8d48JfnP6SBHCoc2zTJu5PksOBAU7ZAszo0N3DBg0t8OMed7IwvcSlVW406AjUxsYO4pprNkRGSoiNolxehylTpmszQ9OCumCEzX7HDv1GLI+byLAWjKqqQHeNEpLBf6vJXzbhtCHGIbcLQ9wNOTpKqIRbbnkCvb1Phs6RNKk70ryXSa7s3w88+ODkjUAoHjrFQNCyqPfftat2g5DDiO+8k58c2tt50Ij4TJDsNzoe+0cJuBLpJ/1LWgDFpXiAS3WZVilVaELcMnMCusIoWZc8lMd3xozJZS3Dxs1mbsQtRScQFCsxVwVzKWjiOjbhbYz+/rTKWqbZ/rTuFSZXbOaIeo1aBUt3P/m96dOjC/MgQQGUhrXhx7Xrqqx1wh4Wpam1YtilDFvNSY24mJj4rfGInVUkhhq6qdrsw3wrUWGfrn4aXQSMDKFJu4QapqnVxvn+MB9O1MlNnidtbbPAzT6fXH7fNYkqrYQsk1w5fBg4ejRafujm4KJFZl+j7C/Yt49r9Q89FMyxOKHINmhYk07chAYb04vOBgdwu5npON/siVc29l2daaFS0XO3ZMl5kiSiIeyzSfw0YeaPpKatomgGTCag3t6DkcJenid8jgTCvlTqce6PtBKyTHLlppvsTLe6eWQyJQsZId6rVLiMmTmT33v58sk+qKSZtg2r4ceFTUWrMG3+6qv1O22znwBsNKc4mZ+CHCttuJ7gbD7rEiUkYN7gyJiDYBufXhTNgGuUUNQ8YezjxO3K6vRoIz9M82hwUH86XLIkiPxZtgw4eRL41re4MxgApkwBLl0KrhfXJsm0bVgNPwuYnHNAMGB79gRc+eIzg4M85NPVoVdvkKNyRka2Rjpo42jtly59lDp3uU190LCTWNTpL0l6vGsETJpVxrJAXAI8IHqeuETVJI29T/OErptHolym6XRIxCN/vvhFoFQKyONEdv+JE8H1RMD99wOAe6Zty2j4tlDtZkCtEO/u5jY2IKh6s3MnsHp1ura2oqCLZx4bOxgqSMw23SlQC1EzdjF1O/6SJZwbR64PumYNF/gi0kacxGwzaNPKpYirhYeFF+r6rCiaAReY50mAOMpDGrH3aZ7QdQqAWi5TVSa+/33+nbLsUPM7ZHm0fTtw4ED8tgk0LLVCVlBT+EWpQjUh4uGHucB//PGgrGEzUCu40CCYUvPDzTyUWjq9zGIoxmBwEHj99VpHmwjVXLuWE6QlHafx8e/hl7988LK/olTqwYIF+7Thqbbmj7xLQCZBXGqEKAc2EI9uI60SnEVSo0R999AQtx6sXh3M2TvucKdW8Bq+BJMNTi10sWVLkMEpqHRdbMf1CJfID5NNN6w2qWyukO/hAqGlCYZLeUwGBmoX1Nq1wKFDyU9i4+PfqxKEXbz82sTEWZTLX5v0e9KIgMmT790GLtp17Tx5Cyo9c1z/Q1pRSllEw4RBZNUCtaeJ8XH9aUOUVRTJgcC1s12/29vwJdgmUcme+KNHA7rUsM80CsJqosYtaqKL5lBhY7eV7azisWpn/fzned2CpUv5fzEm8mLu7+daUhq8JCMjW2uEfYBPUC5vcLYp510C0hWuma3BPGHo7T2UyP+QVpRS2nw1MnQ+AlE3W/idgECxlGWHzOv0zDNc6Pf3A8A117m2x5t0LCDzXIhdN202xXpB2LHblkFRvZ/Q/M2ZpuHmCtWuqnKarFrFHV/33su193XrguMvEGj/Tz8dsGWqp7m4MJteapG0z+qFRVJFPZiekrB8mrRsNd7eBmEMmyYfgYsZWLBmAvPeZ+ztOXatq4XX8C0gp0wPDweDJQa5kTV6FSLyQ8fv4hJFIWv+QbGNWkRpZHL0lIhWkDlN7r8feOSRQHt/7jk+RocPB4tt4cKAPVPERicZN1stMmmf2UbA5I16KDWYJEpJrOkoLdsGYVxLpsi/LVvi0bzIpxCg5yr71ilwTdFN+peUWiENxKFNcKFlaGSY0+fJ+Z5J6RZkagP5sSkdfuPG7GgxxsaeNtILpNln9YoiqDPSRpprOupeKi1HnO9W5zfQ9TpzlLstLfDj8vG4cqk0ItLkOZHhwhnDWO0C6epirLMzWCxZCvYw2Ar8pH1Wr3Ady3pCmmvadK84/Do6qIopEnDptLTAZ8x+p201Db+eNDiVcKqzkwt9HQFVnggjRCu6zzyikYeGr1MqOzriEfupSCLwC3PaEtGvAYRnYeSGuXOA2dcCH7wPvDM6+f2uK4Ab5wO/GuFZburzSbgSwG+ybnXW6OrCrJ4eXNfWhvZKBRfPnsV758/jwwS3dOyXOVfzdPLzHwWPAc4LPjrOx0M8zg9dXZg1ezZuIAp8YdV1NTFlCtpi9FlTzJcMkGG/xF7Tjvea0RHMXfn6RPP19xhjTkVQCovDZ4y5Ox7qHER0ijkmRjQzfL/o4ftFD98vehCRc3ijj9Lx8PDwaBF4ge/h4eHRIvACPxskoDdqavh+0cP3ix6+X/Rw7pfCnLYeHh4eHvnCa/geHh4eLQIv8B1BRF8moteJ6E0i+lvN+1uI6OdE9AoRPU9Eel6BJkNUv0jXrSYiRkQtE4Vh0zdEtKY6b14jon/Ju41FwGItzSOiISJ6qbqelhfRzjxBRE8Q0QdE9KrhfSKix6p99goRfc7qxq4B/K38B040cxrAfADtAF4G8Gnlmj4AHdXHDwA4UnS766FfqtddAeBFACcBLC663fXSNwBuBvASgN+tPp9ddLvrpF8OAHig+vjTAM4U3e4c+uUPAXwOwKuG95cDeA6cY/oLAP7L5r5ew3fDHwB4kzE2wjhH7mEA98oXMMaGGGOCxu8kgOtzbmMRiOyXKnYA+HsA/5dn4wqGTd/8GYDvMMb+GwAYYx/k3MYiYNMvDEBn9XEXAE1yZHOBMfYiEJqwdy+ApxjHSQDdRHRt1H29wHfDdQDekZ6/W33NhK+D78bNjsh+qR495zLG/j3PhtUBbObMAgALiOgnRHSSiL6cW+uKg02//B2AtUT0LoATAP4yn6bVNeLKIAC+4lXmIKK1ABYD+KOi21I0iGgKgL0AvlpwU+oVbeBmndvBT4QvEtGtjLFzhbaqePwJgH9mjO0hotsAHCKihYyx+qr52ADwGr4b3gMwV3p+ffW1GhDRnQC2AljJGLuQU9uKRFS/XAFgIYD/IKIz4LbH4y3iuLWZM+8COM4Y+4Qx9isAb4BvAM0Mm375OoBnAIAx9p8AfgecZ6eVYSWDVHiB74ZhADcT0Y1E1A5gAMBx+QIiWgRgP7iwbwVbLBDRL4yx84yxKxljn2KMfQrct7GSMdYYpc+SIXLOADgGrt2DiK4EN/GM5NnIAmDTL28DWAYARNQLLvB/nWsr6w/HAayvRut8AcB5xtj7UR/yJh0HMMYqRPQXAH4IHmXwBGPsNSLaDk5dehzAowBmAniWiADgbcbYysIanQMs+6UlYdk3PwRwNxH9HMAEgL9hjJ0trtXZw7Jf/hrAPxHRX4E7cL/KqqEqzQoi+lfwzf/Kqu/iEQBTAYAx9o/gvozlAN4E8L8AvmZ13ybvNw8PDw+PKrxJx8PDw6NF4AW+h4eHR4vAC3wPDw+PFoEX+B4eHh4tAi/wPTw8PFoEXuB7eHh4tAi8wPfw8PBoEXiB7+Hh4dEi+H8gk9ORt5FtRwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "xp1 = np.array(np.linspace(np.min(X[:, 0]), np.max(X[:, 0]), 200))\n", "xp2 = np.array(np.linspace(np.min(X[:, 1]), np.max(X[:, 1]), 200))\n", "xp1_mesh, xp2_mesh = np.meshgrid(xp1, xp2)\n", "grid = np.array(list(zip(xp1_mesh.flatten(), xp2_mesh.flatten())))\n", "prediction_grid = clf.predict(grid).reshape((200, 200))\n", "plt.contour(xp1, xp2, prediction_grid)\n", "plt.plot(X[pos, 0], X[pos, 1], 'bx')\n", "plt.plot(X[neg, 0], X[neg, 1], 'yo')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Dataset 3" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "data = sio.loadmat(\"ex6data3.mat\")\n", "X = data[\"X\"]\n", "y = data[\"y\"].flatten()" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "pos = np.where(y==1)[0]\n", "neg = np.where(y==0)[0]" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "SVC(C=2, cache_size=200, class_weight=None, coef0=0.0,\n", " decision_function_shape='ovr', degree=3, gamma='auto_deprecated',\n", " kernel=.gaussian_kernel at 0x12253b730>,\n", " max_iter=-1, probability=False, random_state=None, shrinking=True,\n", " tol=0.001, verbose=False)" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf = SVC(C=2, kernel = gaussian_wrapper(0.1))\n", "clf.fit(X, y)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJztnXd4VFX6+D8nCUlAiiCIIgKiYldcgxoBhZ9lXV1l97u2FRV3VRTLFkXFvqvrWlbEXgDpuApYwC6QgCDogqL0BKS30GtInff3RyaahJnMnbnt3JnzeZ55kpmcufedMzfvee/bjhIRDAaDwZBapPktgMFgMBi8xyh/g8FgSEGM8jcYDIYUxCh/g8FgSEGM8jcYDIYUxCh/g8FgSEGM8jcYDIYUxCh/g8FgSEGM8jcYDIYUJMNvAaLRsmVL6dChg99iGAyBYMMG2LgRDj8c2rTxW5rUoCy0j20lm9hdnE2zkoYc1raF3yIB8N13320VkVaxxmmr/Dt06MDcuXP9FsNgAODZZ6FLF+jZ85fX8vNhzhy47z7/5KqW46qr4JFH4PXX4T//qS2nDug8f4mwaPtMvip6hOz05gyZcx5nTTyUQeP+7rdYACilVlsZZ9w+BoMFunSpUrD5+VXPqxVuly6x3/vss7+8r5r8/KrX7VItx7hx8PjjVT9ryqkLduZPJ8oqi3l10R3M2XofEOLjZafR8rODeey1m/wWLW6M8jcYLNCz5y+K9dFHf1G4VixsNxXfnDm15aiWc84c+8d2EjvzpxMTVj9I48x5LN19GCPH9aD3lquZNOZ+Dm7Z1G/R4kdEtHycccYZYjDoxiOPiEDVz3jIyxNp2bLqfS1bVj1PRRKdPx1Yvus7GbK0qzy38AI58aGnZfr73/otUkSAuWJBxxrL32CwSH5+lU+92rcej2ulZ0/o1w+eeKLqZ9AsXiewM39+UlFZwuAl/flq011USBozV3Wi0xIh99LT/RbNFtoGfA0GnajpW+/Zs+oRj+uiruKrPkaqYHf+/GL+9ulM3/gETTOLWb6nFVO+PI3bG5/D9e9d7LdotjGWv8FgATu+9aAEZa2QaPA6KLGJmny5/lW+3/oA6WkVTCrozIJBXfn8lnu5/s7gK34AJZru5JWTkyMm1dOQDCRTmmNdC77u82QhJBUMLfg1JSFh+Oc9+Uv5qfzp/l5+i2UJpdR3IpITa5yx/A0Gl7nvvgMVY8+e0RV/vNa1m6mkdUmWrJ36KNw5h5cXXUpW+n6W72lFgznp9Opzrt9iOY5R/gaDZsSbGup1Dn0yB6/HLH+Erzf/layMEj796RSWv3I6nw68jRaHNfdbNMcxAV+DQTNqWtf9+lUFiOuzruMdb5dkDV7vKy+iUqaybn9zPv68C39peg69R1/kt1iuYSx/gyECXrpSIhGvde2VNZ5MweuarNm7kLcK+gDw4+Yjafmlone/5FX8YJS/wRARv9sRxJsT71UOfRCzduqjUsoZXvAQUzbcSnZGMZNXn8COMe0YPORWv0VzHyuVYH48TIWvwW/8qsqtPm/1+eo+tzveUMWSHbPlhQUXyojCXHns+99K7nMPyRvPfiihUMhv0WyBqfA1WMVvF4eu+BXYjNe6TjZr3Av2V2xn1uZ7yMoo5dPlpzBnYFc+6n0Pt97bC6WU3+J5gsnzN6RM7na8VM+DF0FUg3eEpJK3f3qISvmKz9eeyJanjmTK5Mf8FssxrOb5m2wfg+fZIkEgqO0IDPWzcs98Jq66n+bZu1hbfDCF37bjgSvO8VssXzDK3wDUdnE88ohRcPW5UlJ9boLK0p15zNr8CA0bpPPlqhPZPLojHz1+M206HOq3aL5glL8BSN7c7USJVH2b6nMSZERCfF30FkrBiDnd6fJRS8a8+7eU8e9HwgR8DUmbu22wRrIH/DcWL2PQwl5kpK1kTXELiuc15k+3X5TSih+M8jcQrGwRnRWVzrLVh981DW4yef0QPl17I00yd5K/7jjyXzybSX1v4PQeJ/ktmv9YyQf142Hy/A2R0DmnXWfZYpGMO42FQiEZsrS7PL/wfDn79QflmYfGBj6H3wqYPH9DMqJzV0mdZYtFsjVr21qyhkEL/kCDtApW7GmFfNmQO+//fcq7empilL8hcHilqBJx4yQqm5Muo0SOFdQtFusiEmL8iuf5cPW1NM3azMyNx/D98JMY8fC1NGrS0G/x9MLK7YEfD+P28Ydnnjnwlj8vr+p1XfDKRZGIGydR2Zx0GaVqe4h1e5fI8/MvlRGFufLvH38t5755vzxx/yiprKz0WzRPwaLbx3clH+1hlL8/6K4IvJYvHmVuVzYnF7V4jhWEBd8Kbyy5UAYv7SY3TL1RzuvzpKwuWOe3SL7gqfIHLgYKgOXAgHrG/QEQICfWMY3y9w+dg39+KKpHHqn6T3nkEfdls3our4+lM6FQSCatel1GFObK3f/7g+Rc86Ts213st1i+4ZnyB9KBn4COQCbwI3BihHFNgK+Ab4zy159UURyx8HIh9Mvyt4KudwdbilfKwB97yYjCXHl2wUVyzhsPyjMDxvgrlM94qfxzgS9qPH8AeCDCuBeAS4FpRvnrjc6Wv5d46WLy0+fv1zHtsr1kjQxd2lWGLO0qN027Xvo9cYV8Nf0Iyc9XMmtWe9m0KTUXAavK34lsnyOAtTWerwu/9jNKqV8BR4rIJw6cz+Aiptr3F7wsfnPyXG7IrVsaq4gwbeMQMtJCfLzyVJq9v49rzv2UytB6QCgtXU1BQV+Kisb6I2AAsN3SWSl1BXCxiNwcfn49cJaI3Bl+ngbkATeKyCql1DSgv4gc0K9ZKdUX6AvQrl27M1avXm1LNkP8PPtsVWVnzX/q/PwqxRGp340htXj00V+a/z3+uD8y7CzdwLCCv3FIw3VsLT2IMZ925z+dXyEjc8sBY7Oy2pObu8p7IX3EaktnJyz/9cCRNZ63Db9WTRPgZGCaUmoVcDYwSSl1gHAiMlhEckQkp1WrVg6IZoiX++470Jrr2VNfxa9LSwVd5HATv2sBRISJq15l/KqraZa9nm+KjmLC6+cyvNtVZGRujfie0tI13goZIJxQ/nOAY5VSRymlMoFrgEnVfxSRXSLSUkQ6iEgHqgK+l0ey/A2GeNGlL40ucriF3+7A8tA+Xln8B3aWj2VHWSPGfN2N5u+dz8yXHqTzOceTldUu4vuivW5wQPmLSAVwJ/AFsAQYJyKLlFKPK6Uut3t8gzski6Wqiy9aFzncwu/mfwt2fETTzE18s6UDE17oxpAud/DvQX8mPT0dgI4dnyQtrVGt96SlNaJjxye9ETCIWIkK+/Ew2T7uomP2hh10SU3VRY5kIRQKyedrh8uQpd1kyNKu0mXYw/Lwn16POHbTpjEya1Z7k+1jKnztoUtes5tyJEtKpy6fQxc5gkgkxb2rdKM8P/8KGVGYK88tuEAuGH2P/OW2N6S8rNxvcbXGKH+b6GIZuy2H05aq14tmqnxPVtHFaImHTZvGyPTpjSQ/n58f06c3knE/9JC3Cs6RW2f0ltzbnpD53xT4LWogsKr8TVfPKOjiw3VTDjeyN7wOfPrti9ZNjiAGnleseIhQqLjWa6FQMc13z2JLWWNmTjqVl/9wOaec1cknCZMUKyuEHw+/Lf9qdPHhOi2Hm5aqcX/4S9DmPz9f1bL6qx95+ciFb98t/f78Usp15rQDxvK3j995zW7K4aalmmwbg+iC1QytoM1/tHTMncXN+NehN/LaW3eRlmZUleNYWSH8ePht+eviw9VFjnjQ0fIMoi+8LlavBR3nvz42bRojedMa1LL6J0/OlPXrRvktWiDBBHztoYuy0EUOq+i6WOkqV7zEUuxB/JzF5Tvl7e+Ok8+nZ0teHpI3ubXlNE2T3nkgVpW/7d4+bpGTkyNz55oi4KChc2+g6uBnv35V7rOgFmHV119H5/mPRP7Gd1m683Wy08uYsakTy5/vyGej+nNQ00Yx31tUNJaCgr61gsVpaY047rjBtG7d202xtcZqbx+j/A2e4rdy0qExmR2SZQETEYYU3EJW+mJ2lDXiswWn0n7KUbz8yq1kZWdaOsbs2R0oLT2w+WMqNnOriZeN3QwGy/iZimglcO5G2wunjul3fx0n2V66jKz0xRTsbs2YYefx5GE3M3joXZYVP0Rv2maauVnDKH9DTJxUiH7VT1hVnG4sTk4dU5daArt8s/ljxq3siwj8uLEt7RZATo8TLb23qGgss2d3YNq0NKKpL9PMzRpG+Rti4rRCdCMVMdYCZVVxurE4OXXMoLXbrktxxQ5eWNCHgl3/plzSmLCgC80nHMWY9/tben+1j7/K1SNA5QFjTDO3OLASFfbj4Xe2j6E2Ou8vW/OYTmW5uFHcp0vBoB/krX9HXl98rgwryJW/zb5Szuz/D5n28dy4jlGV1XNgMVh+frrJ9qkBJtXT4DROKK8gVBa7uTgFJffeSfaUbZDhBbny/MLz5aL//l1uu+UVKdlfGvdxolUC5+crF6QOLlaVv3H7GCzhVJWx7pXFbgRVrR4zWfZYqElJ5R5GFt6PUvD1hmNIH9KU1wffEVdgtxqzYYuzGOVviImTCtFNv7UTC5Qbi5PVY0aKrVx2GWRk1B4XlAVhVtFEhhVezkENlrNo5+Gs+qItD/bvlfDxzIYtzmKUf0Dx0koMQpaJzmmQVhe8SIHhxx+Hp54KVpdOgDlbxrFs9zOUi2L8gi5seq0bM/7Zn26/OT3hY7Zu3ZvjjhtMVlZ7QJGV1T7lC7psYcU35MfD+PzrxwnfedBaR9SHU59Fh/YIdWMr8cQL7M6DE+0SyiqK5bXFl8nQgq7S+c3H5B9934z7GIbEwQR8kx+7QUQdFJ2O+BmcjXZuq8F2O99ptE1V4lkAvtsyWUbNPVEmf5UpefnIex8dLPPnvmj5/Qb7GOWfItjNwEnlLJT68CMtM5riHjgwvu8o0e80WirlrFntY753f8VueXHBzTJu3jEyNT/N1gJisIdV5W98/gHGiQBn0Hq/e4Ff+zhEiq088ECVHPHEMhL9ThNtl7B+33yGF15Ok8xFHLRrPWmEav09FCpmxYqHrAlh8Ayj/AOKUwFOXTas0QU/A8eRAsMVFfDxx/EF2xP9ThNNpczb+CqZ6eWMX9KFhqH9EceYfjv6YZR/QHEiA0fXDBk/8911y2yKNzXWzncabypleeV+Xlt8FyFZwPayRqydezgqdFjEsSYXXz9MS+cUxu/2ytGoqcB69jzwuSE6dr/ToqKxrFjxEKWla8jKakfHjk9GTKWcvz2fGZueoHGDEpbuas20T0+l/5Hn0e2SjabHvs+Yfv4JoqtCTDWC1rc+1a6bIQU9KQsJXxScQsP3OjDs1X4/b8BidQExuIPp558gfvabN/xC0ALRiVw3QWznUBEqZciSe8lMK6Vgd2uKRrdm1OA7a+281bp1b3JzV9GjR4jc3FVG8WuKUf518KvfvKE2QQtEJ3LdBM3QWLT9K15fcimZGV+zfE9Lvp16PH+/oAtZDbN+HlOz3/7s2R0oKhrro8SGerGSDxrrAVwMFADLgQER/n43sBiYD0wF2sc6pt95/jq0302mCtx4iLdQSad5ive6CUqdRVHxEhlekCuvLj5P/u/j2+Ty656WHVt21RrjRJGYwT54leevlEoHXgV+A5wI/FEpVXdbnnlAjoicCkwANL6x1cfqDJplCM64MuLNuLE6T267WRK5boLg3gpJBZ+sHYRSMHFZZ9JfbMvE0fdzcMumtcatWPFQrUAvmBx/rbGyQtT3AHKBL2o8fwB4oJ7xpwNfxzquX5a/bi0PgmIZVuPX/FmZJy/2Eoj32Lp/v8t2zpUXFlwkIwpz5R/zLpXTHviX5L3/bcSxdvrtJ9pTyIleRG4ezw/wsML3CGBtjefrwq9F4ybgs0h/UEr1VUrNVUrN3bJliwOixY9ued5BsAxr4lfMxMo82ZWtPn92IteNrnUWABWVJby5pD8ziu4iK2M/n604mdkDuzL5jn70/P2ZEd+TaJFY3e0ZS0tXU1DQN2a8INH3eXU83bGd6qmUugK4WERuDj+/HjhLRO6MMPY64E7gPBEpre+4Js+/iqCkPNZNdXz00SpFfP75MGWK++ePZ56qZXvkkSqla4VqxeBk/rqX6aHxpF8Wl29l+LI/0rjBPpbtacWULzpzV9Nz6H3Hr2OeI5E5mj27Q1jh1iYrqz25uascf59Xx/MLq6meGbEGWGA9cGSN523Dr9UV6ALgISwofkMVdYubevbUN/uo2u8+blzV85degoYNYe7cqs/hprzxzFNdv3z1+FjU589OVPlHUvBW5YmHukq52qIFIsr+v63v0rjBPr5ceyLrX2nHF0P+TrNDmsQ8T/Wx4s3xT7SnUKLv8+p4uuOE22cOcKxS6iilVCZwDTCp5gCl1OnAm8DlIrLZgXOmBLq5oOqjWrbf/Q5++1sQgU8+gQ8+cN+VYXWe7LhZgqwYrAZiQ1LB8IKHWblnLPsrG7CkoC29ju5oSfFXk0iOf6LuonjeZyUFNdW2ibSt/EWkgipXzhfAEmCciCxSSj2ulLo8POw/QGNgvFLqB6XUpCiHSznqy0Bxc8vDeOSwSs+eVXcAxcXw17/+YsW6vWBZnSc7i2mQFYOVhWvDvsW8svhS0tLyWFPcgtGfdeeWPafzt2fdL9BKdHtGq++z6stPtW0iTXsHn9Glj40TcgQlPpEIbvj8vcKKL/utwusIyWqmrDqR3aM7MOa52zjksOaetWpI9DxW3hePLz8ZWlOY3j4BQhelaUcOXRYxNwmqYqhv4Wp16DW889OzlMtHrNt/MO8OO493/vh/nHjWsYFe8GoybVoaEEnPKXr0CEV4PdiY3j4BQpd0TjtyuBGf0K33jS49a+Kdl2gbn5c0OpUXF15WpfiLD+bTr07nhmbtOfGsY4HkKdoKssvOTYzy1wBdKortyOFGfCKIFc52sKrUE5mXmgvXmWcv45Mdi5m68TYOarCXyatPIG9QN8Zdchf9/3nNz+8JcpC7Jqnmy7eMlUowPx5+9/bxCqerThPtc6NbZXNduXStgHWSeL4DO/MyatntMqIwVx77/reS+9xDMuS5SRIKhQ4YZ2dPX91Ihspdq2A2cA8GTjclS1SJ69QcrS46NNkT8WaO4lHqiczLxn3LZOjSrvL0govkpPueke+mzo861jRqCyZG+acwyWQt6/RZvLo7sqLU452XylCFjF72Lxm6tKsMXtpNrvjsVrn0sn9JeVl5ve/zymJOJcvcbZJG+etifQYNXaxlO+joinJ7MXKjQV1lqFxeXdxLRhTmyhM/XCJdX35QnnvsnYiuHj8wdxjOYlX5ax3wTfYAn1voEkC2i44Vzm5mZlmtQI53XtbunctBGZuZt70tE58/hwmX/5V7/nE1SinnhLdBsmQVBQ1t8/zbtMmR8vK5SZUn7gWpkG/vJ27WZDjd6E0kxHurXmRn2XsoYOz8s+j49mEM/eBeZwR2iFTLw3ebwOf5b9wYjBbGuqGjtWwV3fL66+J2C2Yn02U37CvghYW92Fcxnq1ljRk1vRs95p7G4PfucUZYBzF5+P5gLH+DNuh+1+JlC2Y7jFvxNHvKPyJEGl9vOIZVo45h5P030v74+rbZ8I9kqSTWhcBb/m3a6LW5hcF9/NgIJp67jfosc13uWnaVrWJ/5SSKSpsyYvJ5/OrrS8kf9qC2ih+iVyBHU/xmk3hn0Fb5Q7BcFgZn8LrVhVNVxDpUI2/dv4qRhVV7KH27viOtPmjAXx6+QpvAbn1YbZ2RarttuYnWyh/ca2EcjUQtOF0sv6DjdaaSU3cbfm1fWWUFtyd/mmLenOM5tLyQmRuPZut7h/HcCze6e3IfMJlBzqG98veaRC04HSy/oOPXnrZW7zZiLfBe37X8YgWvQQENQqW02LGSM+Y1ZsbghzjmlPbuCuADydJvSAeM8q9DohacX5ZfMuFXppLVu41YC7zXdy2RrODMtApOO/Mz0tKq/rW9uCP10gefSGaQiRFExij/CCRqwenSmjmoRAqozplz4N2Tk8ornruN+hZ4r+9aRISSCBuUQG0r2O07Uq998PF26DQxgugY5R+BRC24ZKms1YVnn4WMjNrK6/nn4bLLnFNe8d5tRFvg6zuO09b35v2rGLTg91SmZ0b8e00r2O07Uq998PFmBpkYQT1Y6QHhx8Ovxm6J9pPRsQ9N0Kmew4EDq35ef72IUlXP/ZYpnt4+Tl4bi3ZMlSFLu8qQpV3lifyu8sXkBpZ64rjV6yk/X0Vs+5yfr5w9UYLoLp8bkAy9ffwgUb9zkCtra+JV1pKV81TP4VNPwQknwOjRcN11cPfdzspilURdO05Z3yLCrKIhCIqRc7uxcfAVnHry8JhWsJt3pLpX5+oun69YWSH8eJiWzv7g1R1MPOe5/voqq7V7d3/vpuz287djfe8oWS8Df7xCRhTmyj9/uERO+/u/5Yfpi2K+z+3vU/eOnLrL5wYkS0tng3Wc2mzEqx76Vs4zcGCVq+f662u7gILmTrMzp3nrR8vgpd1kaME5ctuMayX39sflx1lLLb3Xiw1odO/Fr7t8TmOUfwripJXn1X4A9Z0nL0+kUaNffPw1YwD1KS/ddiWz+728VdBVnl94vpw77D4Z8Je3pKKiwj1hDYHHKP8UxQmrXRfLP1n2I070c+wp2yLPz+8tIwpz5davr5Xc3/5TykrL3BPUAVLNytYRo/zD6GYFeoEdq11Hn7+d4+uw/WO8hEIh+WzNMBn73XHy5VdZkpePvP9JM5k59Sm/RauXVPSv64hV5Z/02T66tl1wK6vGbmaHV1lLbp8nqAV320vWMWjBlezZ9m9a715Gg8pSFNC80S4qM57QujjJ5NQHC0eUv1LqYqVUgVJquVJqQIS/Zyml3g3//VulVAcnzmsFXdsuuLEoOVFl6uSGItHwoi9+EAvuREKMX3UjzbI30WjnBtKpvYuV24rUbhsE03cnWNhW/kqpdOBV4DfAicAflVIn1hl2E7BDRI4BBgHP2D1vPOhoBbqxKAWl1sDtuzG/GsTZZfqm8WSnF/PjjrY0kuKIY9xSpE60QfArp9501E0MJyz/M4HlIrJCRMqAd4Bedcb0AkaGf58AnK88bDKuqxXo9KLkhdXuBG7fjTm9CLqtXHaXbWbQgmtZvfdFdpQ1ZN6ioygtPiTiWLcUqRMum3j77jiFrq5d3XFC+R8BrK3xfF34tYhjRKQC2AVEvrodRmcrUNdFyQvcvBtzehF0U7lsL1nFOyuuoFnWauZuac87g89jYPtrOT3nxaiK1I0ulU64bOLtu+MUurp2dSfDbwFqopTqC/QFaNfOGQunPivQz4uj7v60PXum1kVbd+GrngMdqalc+vWrktep72lG0Qiy0it4b9nphF4+jJmTHiCjQQZwClBlkZeWriErq93PFnTN/W6r3TOALSWbldUu7PI58PV4aN26ty/77tY0Jh55RN9rSSecsPzXA0fWeN42/FrEMUqpDKAZsK3ugURksIjkiEhOq1atHBBNX1dIUPzzbqDz3Vg0nL5TKa7YyQsLbmRn2ZfsKG/ImkWH8ceLfhVW/FVE2trQrYwav1w2TmH3LjoVe/47ofznAMcqpY5SSmUC1wCT6oyZBPQJ/34FkBfOR9Uet/y9ui5KkXB6DoK48DnpoluyczqjlvWiWVYhP25vy39HnsegThfT+++XxnyvWxk1frlsnMCuMZGqPf9tK/+wD/9O4AtgCTBORBYppR5XSl0eHvYWcIhSajlwN3BAOqiumGCS83NgdeFza+GN97hO36nMKHoBFPz3x7PZ/vJZfPXPAfTsZW0y3cyosbqJum7YNSZStT7BkTx/EflURDqJyNEi8mT4tUdFZFL49xIRuVJEjhGRM0VkhRPn9QITTPJvDtxaeK0et3qRqKlcaj6P906ltHIvLy+8jay0IjaVNGVzfisee+AqsrIjb8oSiaC7Z9zA7l10qtYnJH2FrxPoWCfgNX7MgVuLjtXjVi8S1QVpNReJeLeX/GbzR7xVcBlNMuezaOfhfP7+GTzRrRtHnXRk5DdEIcjuGV1x824q3liCp7EHKz0g/Hjo1NgtyH1inMLPOXCrw6iV40b73FZ7E+0r3y6D5t8gIwpz5cVFPeWSCX+R3n0Gyr49xc5+mAQwTdiqcKsnUbzHdUoOTGM3Z9CtQ6Qf+DkHbi068Rw32iJh5RhDC66XYQXnyF9nXyVn9v+HTP94rjMfwCamCVtt3FgIq4534BaSs2a1d2R8NKwqf+P2iUEQM1Ocxq85cCslNJ7j1pflE8sV9sO2PDLUctaXNCPvnV8x6urenHvpGfaEd4hUDXJGw41gd7yxBK9jD0b5x8DvlEwd+pb4NQduLTpWjxtrkYi2MJRU7uHlhX35cfvD7K3IYmZhJ84oakCnM462J7iDBDXIGaR8/HhjCV73RlKiabp9Tk6OzJ07128xfKduJXDd5wb3qK/7aHUwuO738vY7pfzU5lKy04tZtOtwvpp0Co+ddCGXXNPVvw8SgdmzO0Sp6G1Pbu4qV85ZVDT2gIrleCzs6nz8mncsaWmNtA14xyuvU59PKfWdiOTEHGeUv/5UKxanWwsEHS9aQ8d77g+nLeRX1/Xl66KOLHruOKaMvZeGB2W7K0wCeK1InTifHwuWXeJd8OwukGBd+WvV28cQmWTsW+KE4o5mfY8b547MNakrY3nlfpYeOoDOvedQXNmARcvb0bPVobUUvxP/2E4cA37pA+TEsaxQX4yh5jnr+3x+uKrszne8vY687I1klH8AsNoEzU9LOF6cUNxuNlyLh++2Tmb25qdokllCwe7W5H16Kvccfi5XDjn/5zF1Ld9EGrI5cYyaeKlorCjuWJ/PqeZzVnF6vnXDBHw1J57MFL9bUcQTnHaqgMvPAjwRYejSO1i44zFQwvuLf8XyF7oy5S/3c+XN59ca60R2TZAzdKwEM2N9Pq+rm4M831Ywyl9z4sl48bsVRbyLjxOK2889EbaWLKZB+jyW7WnFyLd7ckf5dYwb3Z/GzRodMNYJl0VQM3TAmuKO9fm8rm4O8nxbwbh9NCeSu6a+3vd+xgfidcPY7env554Ii3fOJH/DozRuAHPWHcXi2dMuAAAeMUlEQVQx/0vnoifOjjreCZeF124PJ7ESY7Dy+bx0VQV5vq1gLP8kw+/dwaxa804UcPlRfFZeuZ/XFt/F/zbfR5qq5KNlp5E5/nBeG3Nnve9zwmUR9KZusQqpdPt8usnjOFbKgP146NLeIUjo0IrCatuEZ5458G95eVWv68yna5+REYW5cv+c38lZ/3xUJgybavm9TrQQSPZ+PLp9Pt3ksQIW2zuYPP8oBClzphq/Zfa6IM2ptEerxz/o8FuYXfIlIeDVTy/i3pIT6fPg72u9x+/vwC5Bl99gPc/fuH2i4HfmTCL43YqiphumqGgs2dkdGDcujbIy58vw3d59KdLxt65+lCbFReStPIGjvk3nyrsuPuB9QbxuahJ0+Q3WMZZ/PSRbZa1XVp0X1aNuV3tGO/6O0mbsKfwvN9z1m6jvDfp1E3T5Ux1j+TtA3eDlnDn+N1mzg1dWnRf50W6n4UU7TvOs3fUqfgj+5j9Bl99gDaP866Fu5kxGRrBvib2qA/AiP9rtDohZWZF32LJyfL8zruwSdPkN1jDKPwqRUhGfegoeeCDY+/l6YdV50ZrWzTS8ZTvnUtigKSFV+9/DyvHd2oPAK4IufzwEqT20GxjlH4VoOeQVFd7eEjt9gXph1XmRH+1Wted7q55m5ua/sL9xc/LKzmPXnoPjOr5btQdeKapU2bzI7YSBIGACvnHiZTDM6cCpl6mYbqdhukFIKhm9vDsbS5ry/hdnc1f2WVx/Z/3+fS9wI4AexO/HSeJNGAjSfJmAbxxYbUjm9S2x04FTL606N7bFc5O1+xbz4sLLACjY0ZrsKYpr+13ks1RVOH0dGKs3vriULvPl9N2fUf5Yz4Lx+pbY6cCp33UAOhKSCoYXPMzk9bdwUOYepqw+njVvHcebz7bif/87Wgt/sNPXQbJ3q7RCPHEpHebLjQXIKH+sZ8F4rTy93tPTCjrsKewUy3bN4eVFl5CWlsea4haM/OQ8zpnXi7HPtGfLznt9t/Sqcfo6SPZulVaIJy6lw3y5sQAZ5R9Gx9xmHRtLuV0r4NXiEpIKpm/qT1ZGKZ8uP4U5A7sy6dp76HdvL1au9N/Sq4nT14GORoXXxJMwoMN8ubEA2VL+SqkWSqnJSqll4Z/NI4zprJSarZRapJSar5S62s453ULH3Gav+5dbwe1aAS8K0UJSyTs/PU2DtHJ+2N6Wda8ewSejB3DIYVWXrw6WXk2cvg7sLCbJlB5pNS7l1OJrZ+7cWIDs9vMfAEwVkaeVUgPCz++vM6YYuEFEliml2gDfKaW+EJGdNs/tGH72hY+Fl/3LreLmngFub824es8C3l91Py2yd7K2+GDmzT6G23qcXmuMjn3cnbwOEt2/N9m3NYyGE/sd2527jh2fjJjxZccLYCvVUylVAPQQkY1KqcOBaSJyXIz3/AhcISLL6hvnZapnKnYytPOZY6W7OjGfjz76y+Ly+OPWP1d9bCiez+fr+lEeSmfG2k5sGt2Rkf+8iSOOal1rnBe9iYKI2/2Ukpl45i5aWqnVdFOrqZ52Lf/WIrIx/PsmoHV9g5VSZwKZwE82z+so8e6WlQwkuoG6lbsku5uz293hKxIiIaZueIN0JYxd3IVDh7Zk8icDUEodMNYJSy8ZcdodFqTcebtYnbtYdwhOzk9M5a+UmgIcFuFPtaJfIiJKqai3EeE7g9FAHxEJRRnTF+gL0K5d6gSf/CBR90p96a51X0vEdeOGC25T8XLe/uluDsneyrr9B7P1+0N48M89Iir+anR0t1nFLaXqpDvMDReSzouJ1bmrL6vH6c/iidtHKdUUmAb8W0QmWDm2rhW+yYYb7hU7x3baBTd787ss3vESIdKYteFoVow5lpH9+9DhhLbxHywAuOmycvLYTruQ/HDVxbPYWJVv2rQ0IJJOVvToEdFmPnCkRxW+k4A+4d/7ABMjCJIJfACMsqr4Dd6QSIZTPNXQiWRPOVlLISLM3z6czH07aLlmPjcwkqdvHkzDFtPjP1hAcLMgycmso6AXrsVbdGV17rxMK7Wr/J8GLlRKLQMuCD9HKZWjlBoaHnMVcC5wo1Lqh/Cjs83zGmySaKsKK6mYOnSG3FqyhkEL/kCLkp84fHchzRvsRqkq5eJGwZYuxW9up6k61bYj6IVriSw2VubOy9oeW8pfRLaJyPkicqyIXCAi28OvzxWRm8O/jxGRBiLSucbjByeENyROoq0qrOT5+9kZUiTE+BXP8+Hqa2mWtZlGOzeSQWWtMW5YhLpsf6hDQZIVgl645tZi42Vtj+nqaUgIN2MFiVJUvJy3f/obLbK3s2F/Uz7/pjP/OPYZIsd1rftQraLD9odBSlN1MkDr9efWOe3VdPU0uIaO1dAAE9c8SOPMXeSvPY4vBnZn1Hl3kZ3dPuJYNyxCHVqE6FgVHg0nO796/bl1bL0SL3bz/A0phpfV0FYtQxHh4zVvkpW+jlX7WvDDhOOZOKA37Y87guwi5ysjo+FGfUIiBDlN1Q5efu5kqAUxyt8QF1by/J3Aah741v2rGLXs7xzSsIjNpY3J++5kemUdSvtwKqdX/6Q6twgxuEPQF1nj8zdoiRWfamnlbkYt/y2KEN8WdWTx6E4Mu+M6juvcwVthSf4WIToXUBlq41V7B4PBFWJlU4gIeRuGk5lWwZT1x7P2qQ7M+OxB0tL8CWMlc4uQVGvolioLnQn4GrSkvtS9naUbGbTwajaXvMu2skYULDiSK3OP903xJzs67GTlFbps2egF5r/FoCXRsil2NOnJuJVX0SxrPd9u7sD4185j8Bl/5I5/XOWTpMmPbvsbuEkqLXRG+acAulSfxsOBqXvtWH3QyWzJKmBneSPGfN2NgydcwMyXHqTzOcf7LW5SE5TCMSdIpYXOKP8UQJfq03ipmQfe5uS3CTVN54ftbRn3UncG59zOvwf9mfT0dNvnCdLuVH7Imgw57VYJ8kJXfW106sQZVsYb5Z8CuL31opuICF+sG8kX6x+gUhTz1rTnV1uzOfaUyMVb8eKlj9eu4vbLHx2kwjG7BHWhKyy8nSVLro+YIRcNk+2TIri59aJb7KvYxuAl/WiRvY49FQfx+Y+nccrMo3n+3ZscO4dX/dOdyJjxstd7XYKe026VIBZvFRWNZcOGN4jcCjo6RvmnCLGqT3VMb/ty/QscnLWO2ZuP4scxxzP4xqs5pXenn//uhMxe+XidUNyp5I/2k3gXOr//d6qC0fHXaxm3TwoQq8VyIu4Et33PX218j+0leeyrzGLmjJO556QunHJWbcXvhAvEKx+vE4o7yP7oZEWH1NBEF3+j/FOAWC2W401vc/OC31u+lUELerNy70D2VGTz8YLTOaewMb36nl9rnFMpeV75eJ1Q3EH1RydKEALxOqSGJrr4G7dPChCr+jReq9QN33NVYHcEq/cNp1lWJXO3tmPOuyfywm9/x9nDTrUsm1UrqOatenp6CzIyGlJRsd212/aOHe03mAuiPzpRglJVrIMrLtK1ZQWj/A1xb8ztxgU/ZcPLFJW8w96Kg/hsXmdOnn0sM1/8MxkNIl+idjYTr6tYKiu3kZbWiBNOGO2aYnFKcadK4NXP4HY8OLmpfaLUvbYqKqTMyvuM28cQtzvBad/z/opdLN/9EcWVDRjxZQ+uXnEWL7zWN6riT0Tmmvh1q+5k/3odcNMto4NFbQVdXHE1r60VK1hg5T1G+RvizuN28oKfvnE8I5b1Ijt9Hz9ua0uTOcLvbv5/jstck6AoFp1xO9AZlOB2kGsgTEvnAONnG2En0ttGLPsLirnsKGvI5wtO48gpHXjl5VvJapjlktRV6LwFX1Bwew6DtB2lbpiWzilAdduG6kyemimdbmPX91xauRfFXFbta8HE/+byxq97cebQkx2UMDpOBF9THbfvnuzESPzOuw8KRvkHmJptG/zcNDxevtn8Md9tHUjjBrB4axtafl9Bl6dP8uz8qZQ14xZeBDoTMTCCkiWkA0b5B5wgtW0ortjB4CV/p3l2ISGyGb8gh2YTO/DhB/1QSnkqS6pkzbiFrndPumUJ6XwXYgK+PuJEq+W6bRvqHk8XZhW9x6hlv6NZViE/bGvLmOE96J95A2NH3k2jxg39Fs8XglDEFA1dA506BfN1qP6tD2P5+4hdn31QNg2vDJWydNfzlEo2H3x/Bh2mHsVXL99KVnam36L5htfuCTcsUB3vnnTIu69Gt7uQuhjL30fstlqO1bZBB0or9/L6kjtpUryZozd/y4DWg7jxzwPZuWu836L5ipe1BrpboE6iS9496HUXEglj+fuMHZ+97puGzyqayA/bXqB1+Tqa71xJBpWgTBAOvFUMulugTqJTMF+nu5BI2LL8lVItlFKTlVLLwj+b1zO2qVJqnVLqFTvnTDb88Nl7sa3j8t0zWLb7GSpQZG4rqlL8NUjWfVGt4lQRk5W4ge4WqNPoUkmt011IJOy6fQYAU0XkWGBq+Hk0ngC+snm+pCJWq2W3cHtbx/LKEqasfxWAd388i6ZpeyKOS1blYwUnFINVd05QqmWTDaeC4m4lBthV/r2AkeHfRwK/izRIKXUG0Br40ub5kgq/fPZubus4b+sU3iy4lIYZa1i86zD2z2pCumoTcWyQlY/df0gnFIPVuIHuFmgyY/cuxM14ja32DkqpnSJycPh3Beyofl5jTBqQB1wHXADkiMidUY7XF+gL0K5duzNWr7a+H6Uhfh599JdYw+OP2ztWSeUehiy5m6ZZi9hTkcWUZSeRMb4Dw1/uR3Hph0lVqq9L64Fp09KIvIOTokePUK1X/Mg31znHPSgk0kbDsfYOSqkpwGER/lTLvBARUUpFuhJvBz4VkXWxCnlEZDAwGKp6+8SSzZA4sbZ1jIdtJSsZt/LPNM4sZeHONnw18WT+eepFXDz6HACa4H4QzktFo0sANZ6AYt20zOo7Fze/D1Npax834zUxlb+IXBDtb0qpIqXU4SKyUSl1OLA5wrBcoLtS6nagMZCplNorIvXFBwwu4nR9wIzNw2iUUcr7y06n4rXDmT6uP9mNsmuNcTMn3GtFo0sANdEqWy/mS5cFMui4mTFk1+c/CegT/r0PMLHuABHpLSLtRKQD0B8YZRS/vyQaa6ibJVReuZ+/DX+Z/75yGLvKs1m54Aiuv/BXByh+t6i2Xpcsuc7T/vy6BFATjRt4UWPg5wIZ5MrpurgZr7Gr/J8GLlRKLaPKn/80gFIqRyk11K5wBntES+mEAy38nj1jt4GumSVUsGs2/Uc9xLB7riO94ybefq87j7Y8i2v+eolzH6AeagfCIuOWotEpgJpIQNELxezXAplsBW1uttGwVeQlItuA8yO8Phe4OcLrI4ARds5psI7TLZ9rZgmdeeVqvnrnUU69eyRlM5ow5bU7adysUeyDOEQk67UubikanQqJEsGL4iO/Gr8lo7vJLZepqfBNYtxo+dz9vFLOvnIGH79+Dd1ueZe9q/bzypN/9lTxQ2wr1W1Fo2NfG6t4oZj9WiB1iccEAaP8kxwnWz4v2j6TNyZ+yPR3HqL7Le8w++3L6P+nBRzW4dCI493MwIlmvVb9rX2gLHGv8Uox+7FA6t5SQSdMY7ckx6n2EcUVWxj58buMuOdhOt87gmbFq3lvtDD0nbMjHtNt32s0v/sJJ4xJis3R3UaXFghOo0s8JghBZ6P8A0CivXicah8RkgomrHya1QuPJ+fBt8ieuouPxtzP5b9vHDVLyO2MEl37yeuGFSUUBEVlFR2ui6AEnc0G7gGgbl5+3efRcGKD9592f89Hax6kedZuVu9rzsRJZ/HYkd25/E/1+4/iqT41uIOVSmRdqpWTCbc3t4+F1Qpfo/wDQrXC92qv3orKEoYVPkyD9NmUhtKZvuZ4dow5ilH/7suhbQ+J+X6//wFikQqtB6x8B7p/T0HEb8PHqvI3bp+AUDNw26+fu4q/PLSPwQW9yMqYxYq9LRn5YQ8uLfw/Ph/1gCXFD/r4XiMRlNvyusTrnrGS+RItaG6yYxJHlyLAWBjlHxC87Pu/dOdkDmqwh5mbjiHvyRw+v+Verrvz4riOoYPvNRpe7qLlFIksWLGUUNV7I/fb0k1RBQmdDZ+amFTPAODVXr0hqWBU4eOE1FQqJJ3FK9vSvWkLmh3SJKHj6ZoLH8Rc8ESKl2Ll81ctdpHdE7opqiARlCJAo/xt4ERA1Qr19eJxSvlvLVnB2z/1o1nmHtbua8GX0zpzbckZ3D4s4hYNgUaXXPB44g6JLFixlFD094p2iipo6Gr41MQofxs43T4hGl7s1Ttlw0s0brCXz1aexJZhHZj4fD9atmnh3Ak0wo0K13gDyPF21kx0wapPCdVXKDd7dgctrVWDcxifvw3c3BHLK0JSyX9/eprSyv+xq7whC2cfzb2Xdk9axQ/OxyMS8cfHG3dww48c6ZjV1PcZglIXEBQ5/cIof5t4mYUTi3iLwdbsXciLCy+jLDSJ9fsP5r3pXbi+sj2dL1yb9P80Tla4JhJAjteN40YAvfYxDyTSZ3AiU8oLpRzUjC4vMcrfJl5m4cTC6sbslVLOsKUPMWXDrRzUYC9TVh/P1IHdePfiv3LjHSEKC80/TTwk4o9PJB3QjZYM1ceMlvVT9zPYzZTySikHMaPLa4zyt4FT7ROcwqob6uM1T5Gens+afS0Y8cl5dJ/XiykjHuCIjoeZf5oESESR23XjOG09W/0MdjOlvLq+gpjR5TVG+dsg0R2x3CSWG2pT8U9s2p/H7oosJnzelXsadqPvvb2o3l85Wf5pvPT3JqLI7bhx3LCerX4GuwVMXl1fQSm08hOT7WMDL7Jw4iXaxuxVgd1nKKn8lDSlmLXuONp8V8nFw7rXer8uaZB28HpP30TzuhNNB3RjwxKrn8FuppRX15dfm8kECaP8k4hoxWDvvissa9Ob7PQ1bCw5mC9mduZ3W07j7vFX/WzxV+P2P40XPXX82M3Jy7xut6xnK5/BbgGTV0o5KIVWfmKUfxIRzQ01bdZOOl61hqV7WvPFG1346IGbOOKo1hGP4eY/jVcWeZBdV1YWR7/vzuwsdF4q5SAUWvmJUf5JRF03lEiI7R1e4sg271EhaSxc35aj1wltouy8VY1b/zReWeR+K8dEsbo4+u3SsHv3ZpSyHpiAb5KycV8hgxb2Ym/FOLaWHcSo6d3ImZHDux8/cICrxyu8ssiD0lirLtEWxyVL+tQK5vrZNM/kzycPxvJPQj5cPYhtJRNokplG/rpOrBxzLCP796HDLW19lcsrizyo/t7oi2DlAXcAflnPfsRTDO5glH+SUR4qZlfZeLaWNWHSVzlcuf10hg/9g2/Wfk28dFcE0bVQX68dXRRskOMphtoYt08SsbVkDa8suhaAhduOIGtiA/72yBVaKH7wzl0R1J4u9fXaAT0UrMmfTx6M5Z8EiIQYv3IQu8s/oGkmfL3xaH56uyPDnr7Wb9EOwG2L3Oscfyeplm/Jkj5A5QF/10HB+h1sNjiHUf5JwKKdn7K/8j22lDbls286c9GKk3jjjd6kpaXejV3QfdLVMuqqYIMaTzEciC3lr5RqAbwLdABWAVeJyI4I49oBQ4Ejqdo66BIRWWXn3IYqdpVuYtrG12icAR/My+G8aYfy8NvX+y2WbySDT1p3BRvEeIrhQOxa/gOAqSLytFJqQPj5/RHGjQKeFJHJSqnGgPtb2Cc5IsLHa95k4/6xNMoIMXtzRzLyG3LXP67wWzRfCWqOf12MgjW4jV2/QC9gZPj3kcABe/4ppU4EMkRkMoCI7BWR4rrjDNbZsn8Vgxb8H9vLRrGrohFjZnWl+fgezBg6gHad2jh6rqAFT/3M8Q/aXBlSG7uWf2sR2Rj+fRMQqWdAJ2CnUup94ChgCjBARA6MaBlism7fPD5fdxdNs2DGhmNYOroTb915Hcfd2MHxcwUxeOqXyySIc2VIbZSI1D9AqSnAYRH+9BAwUkQOrjF2h4g0r/P+K4C3gNOBNVTFCD4VkbcinKsv0BegXbt2Z6xeHTnnOVUREd5ecSeT3jyRgkbpnPzdTkZNvI+0tDRXNo6fPbtDFBdK+/AGIIZqzFwZdEEp9Z2I5MQaF9PtIyIXiMjJER4TgSKl1OHhEx4ObI5wiHXADyKyQkQqgA+BX0U512ARyRGRnFatWsUSLaXYWbqRQQuvpiI0j2bHrWbhf27g9NN/+7Pij7Rjl12SIXjqFWauDEHDrs9/EtAn/HsfYGKEMXOAg5VS1dr8/wGLbZ43pfh28/uMX3UVzbLW803RUSz8vglvPLOVp18/2dWN401Bj3XMXBmChl3l/zRwoVJqGXBB+DlKqRyl1FCAsG+/PzBVKbWAqs1Ch9g8b0rxw/Y3KAllMHp2Nw4aey4zXnqQm+9s5/rG8UFtkOYHZq4MQcNWwFdEtgHnR3h9LnBzjeeTgVPtnCsV2Vu+jSFL/0qL7L38tOtw9k1pxkPPXU16enrUHbucRPd8c50wc2UIGjEDvn6Rk5Mjc+fO9VsM35iyfjQ/7RlCg7RKftzWlm/Hn8TzF/finItOO2DHrrrPDamHFzukOU0QZQ4CVgO+pr2DhuwpX8/64tfZV3kQn313GifMPJaZL91ERoOqr6u+jeON8k89gphmGkSZkw1j+WvG/opdDCvoR+PMVUxccRrlzxzGR3mP+S2WQWPcTDN1yzo3qbHuYSz/ADJ94wQW73yFRg3KmL/9CNZ81obXHjygaNpgqIVbaaZuWucmNdZ/jPLXhLLKfRTufpnSUAaT5p9Fm8lH8fXLt5LVMMtv0Qya41Y/Izc7pCZLD6Ygo63bRym1BdCtxLclsNVvIeqgo0ygp1xJKVOzZrQ49FDaK/VL6rYIoc2bWb1rF9sTlatTJ9pH+2NhId8leFwgYZl1/P5AP7nai0jMKlltlb+OKKXmWvGleYmOMoGechmZrKOjXDrKBPrKFYvU2+3DYDAYDEb5GwwGQypilH98DPZbgAjoKBPoKZeRyTo6yqWjTKCvXPVifP4Gg8GQghjL32AwGFIQo/zrQSnVQik1WSm1LPyzeZRxlUqpH8KPSTrIFB7bVCm1Tin1ipsyWZVLKdVeKfV9eJ4WKaVu00Cmzkqp2WF55iulrvZbpvC4z5VSO5VSH7ssz8VKqQKl1PLwPtx1/56llHo3/PdvlVId3JTHokznhq+jivBmUZ5gQa67lVKLw9fRVKVU1FRZHTDKv36qN6g/Fpgafh6J/SLSOfy4XBOZAJ4AvnJZnmqsyLURyBWRzsBZwACllLObDscvUzFwg4icBFwMvKCUOjjCOC9lAvgPcL2LcqCUSgdeBX4DnAj8Mbzndk1uAnaIyDHAIOAZDWRaA9wIvO2mLAnINQ/IEZFTgQnAs17JlwhG+ddPzA3qfcCSTEqpM6jaU/lLXeQSkTIRKQ0/zcL968+KTIUisiz8+waqdqNzcxs5S9+fiEwF9rgoB8CZwPLwLntlwDth+WpSU94JwPlKKeWnTCKySkTmAyEX5UhErnwRqS6J/gZo66F8cWOUf/1Y2aAeIFspNVcp9Y1Syu0FIqZMSqk0YCBVm+h4haW5UkodqZSaD6wFngkrXF9lqiHbmUAm8JMuMrnMEVR9D9WsC78WcUx4G9ZdwCE+y+QH8cp1E/CZqxLZJOV7+8TYoP5nRESUUtFSo9qLyHqlVEcgTym1QEQSViAOyHQ78KmIrHPSSHNirkRkLXBq2N3zoVJqgogU+SlT+DiHA6OBPiJiy6J0SiZDMFFKXQfkAOf5LUt9pLzyF5ELov1NKVWklDpcRDbWs0E9IrI+/HOFUmoacDo2rEcHZMoFuiulbgcaA5lKqb0iUl98wAu5ah5rg1JqIdCdKneCbzIppZoCnwAPicg3icripEwesR44ssbztuHXIo1Zp5TKAJoB23yWyQ8syaWUuoCqRf68Gi5OLTFun/qJuUG9Uqq5Uior/HtLoCvublAfUyYR6S0i7USkA1Wun1F2Fb8Tciml2iqlGoZ/bw50Awp8likT+ICqOUp4EXJSJg+ZAxyrlDoqPA/XUCVfTWrKewWQJ+4WB1mRyQ9iyqWUOh14E7hcRPxc1K0hIuYR5UGVb3MqsAyYArQIv54DDA3/fg6wAPgx/PMmv2WqM/5G4BVN5upCYH54ruYDfTWQ6TqgHPihxqOz398fMAPYAuynyr/8a5fkuQQopOpO9aHwa49TpcAAsoHxwHLgf0BHD66lWDJ1Cc/JPqruQha5LZNFuaYARTWuo0leyJXow1T4GgwGQwpi3D4Gg8GQghjlbzAYDCmIUf4Gg8GQghjlbzAYDCmIUf4Gg8GQghjlbzAYDCmIUf4Gg8GQghjlbzAYDCnI/weWcrl358vGnwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "xp1 = np.array(np.linspace(np.min(X[:, 0]), np.max(X[:, 0]), 200))\n", "xp2 = np.array(np.linspace(np.min(X[:, 1]), np.max(X[:, 1]), 200))\n", "xp1_mesh, xp2_mesh = np.meshgrid(xp1, xp2)\n", "grid = np.array(list(zip(xp1_mesh.flatten(), xp2_mesh.flatten())))\n", "prediction_grid = clf.predict(grid).reshape((200, 200))\n", "plt.contour(xp1, xp2, prediction_grid)\n", "plt.plot(X[pos, 0], X[pos, 1], 'bx')\n", "plt.plot(X[neg, 0], X[neg, 1], 'yo')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Spam Classification\n", "---\n", "Many email services today provide spam filters that are able to classify emails into spam and non-spam email with high accuracy. In this part of the exercise, you will use SVMs to build your own spam filter.\n", "\n", "You will be training a classifier to classify whether a given email, x, is spam ($y = 1$) or non-spam ($y = 0$). In particular, you need to convert each email into a feature vector $x \\in \\mathbb{R}^n$. The following parts of the exercise will walk you through how such a feature vector can be constructed from an email.\n", "\n", "### Strategy\n", "Before starting on a machine learning task, it is usually insightful to take a look at examples from the dataset. Figure 8 shows a sample email that contains a URL, an email address (at the end), numbers, and dollar amounts. While many emails would contain similar types of entities (e.g., numbers, other URLs, or other email addresses), the specific entities (e.g., the specific URL or specific dollar amount) will be different in almost every email. Therefore, one method often employed in processing emails is to “normalize” these values, so that all URLs are treated the same, all numbers are treated the same, etc. For example, we could replace each URL in the email with the unique string “httpaddr” to indicate that a URL was present.\n", "\n", "This has the effect of letting the spam classifier make a classification decision based on whether any URL was present, rather than whether a specific URL was present. This typically improves the performance of a spam classifier, since spammers often randomize the URLs, and thus the odds of seeing any particular URL again in a new piece of spam is very small.\n", "\n", "### Preprocessing\n", "In `process_email.py`, I have implemented the following email preprocessing and normalization steps:\n", "- Lower-casing: The entire email is converted into lower case, so that captialization is ignored (e.g., IndIcaTE is treated the same as indicate).\n", "- Stripping HTML: All HTML tags are removed from the emails. Many emails often come with HTML formatting; we remove all the HTML tags, so that only the content remains.\n", "- Normalizing URLs: All URLs are replaced with the text “httpaddr”.\n", "- Normalizing Email Addresses: All email addresses are replaced\n", "with the text “emailaddr”.\n", "- Normalizing Numbers: All numbers are replaced with the text\n", "“number”.\n", "- Normalizing Dollars: All dollar signs ($) are replaced with the text\n", "“dollar”.\n", "- Word Stemming: Words are reduced to their stemmed form. For example, “discount”, “discounts”, “discounted” and “discounting” are all replaced with “discount”. Sometimes, the Stemmer actually strips off additional characters from the end, so “include”, “includes”, “included”, and “including” are all replaced with “includ”.\n", "- Removal of non-words: Non-words and punctuation have been removed. All white spaces (tabs, newlines, spaces) have all been trimmed to a single space character." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[85, 915, 793, 1076, 882, 369, 1698, 789, 1821, 1830, 882, 430, 1170, 793, 1001, 1892, 1363, 591, 1675, 237, 161, 88, 687, 944, 1662, 1119, 1061, 1698, 374, 1161, 478, 1892, 1509, 798, 1181, 1236, 809, 1894, 1439, 1546, 180, 1698, 1757, 1895, 687, 1675, 991, 960, 1476, 70, 529, 1698, 530]\n" ] } ], "source": [ "from process_email import *\n", "\n", "# Demo of the process_email function. \n", "# Note: the indexes are all one lower than the exercise, because Python is 0-indexed where Octave is 1-indexed.\n", "with open('emailSample1.txt', 'r') as email:\n", " email_contents = email.read()\n", " tokens = process_email(email_contents)\n", " print(tokens)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You will now implement the feature extraction that converts each email into a vector in $\\mathbb{R}^n$. For this exercise, you will be using $n = \\#$ words in vocabulary list. Specifically, the feature $x_i \\in \\{0, 1\\}$ for an email corresponds to whether the i-th word in the dictionary occurs in the email. That is, $x_i = 1$ if the i-th word is in the email and $x_i = 0$ if the i-th word is not present in the email.\n", "Thus, for a typical email, this feature would look like:\n", "$$\n", "x = \\begin{bmatrix}\n", "0 \\\\\n", "\\vdots \\\\\n", "1 \\\\\n", "0 \\\\\n", "\\vdots \\\\\n", "1 \\\\\n", "0 \\\\\n", "\\vdots \\\\\n", "0\n", "\\end{bmatrix} \\in \\mathbb{R}^n\n", "$$.\n", "\n", "**Exercise**: Implement `extract_features`." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "def extract_features(tokens):\n", " return np.zeros((1899, 1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following code should return 45." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features = extract_features(tokens)\n", "np.count_nonzero(features)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training an SVM for spam classification" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "training_data = sio.loadmat(\"spamTrain.mat\")\n", "X = training_data[\"X\"]\n", "y = training_data[\"y\"].flatten()" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "SVC(C=0.1, cache_size=200, class_weight=None, coef0=0.0,\n", " decision_function_shape='ovr', degree=3, gamma='auto_deprecated',\n", " kernel='linear', max_iter=-1, probability=False, random_state=None,\n", " shrinking=True, tol=0.001, verbose=False)" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "spam_predictor = SVC(C=0.1, kernel='linear')\n", "spam_predictor.fit(X, y)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Training accuracy: 99.825000%'" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "predictions = spam_predictor.predict(X)\n", "'Training accuracy: {:2f}%'.format(np.mean(predictions == y) * 100)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Test accuracy: 98.900000%'" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "testing_data = sio.loadmat(\"spamTest.mat\")\n", "X_test = testing_data[\"Xtest\"]\n", "y_test = testing_data[\"ytest\"].flatten()\n", "\n", "predictions = spam_predictor.predict(X_test)\n", "'Test accuracy: {:2f}%'.format(np.mean(predictions == y_test) * 100)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Top predictors\n", "\n", "To better understand how the spam classifier works, we can inspect the parameters to see which words the classifier thinks are the most predictive of spam." ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
vocabularyweights
1190our0.500614
297click0.465916
1397remov0.422869
738guarante0.383622
1795visit0.367710
\n", "
" ], "text/plain": [ " vocabulary weights\n", "1190 our 0.500614\n", "297 click 0.465916\n", "1397 remov 0.422869\n", "738 guarante 0.383622\n", "1795 visit 0.367710" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weights = spam_predictor.coef_[0]\n", "df = pd.DataFrame({'vocabulary': vocabulary, 'weights': weights})\n", "df.sort_values(by='weights', ascending = False).head()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: ex6/emailSample1.txt ================================================ > Anyone knows how much it costs to host a web portal ? > Well, it depends on how many visitors you're expecting. This can be anywhere from less than 10 bucks a month to a couple of $100. You should checkout http://www.rackspace.com/ or perhaps Amazon EC2 if youre running something big.. To unsubscribe yourself from this mailing list, send an email to: groupname-unsubscribe@egroups.com ================================================ FILE: ex6/process_email.py ================================================ import re from nltk.stem import PorterStemmer # Load vocabulary vocabulary = [] with open('vocab.txt', 'r') as f: lines = f.readlines() vocabulary = [line[:-1] for line in lines] def process_email(email_contents): """ preprocesses a the body of an email and returns a list of word_indices """ global vocabulary # ----- Preprocess email ----- # Lower case email_contents = email_contents.lower() # Remove all HTML html = re.compile(r'<[^<>]+>') email_contents = html.sub('', email_contents) # Handle numbers numbers = re.compile(r'[0-9]+') email_contents = numbers.sub('number', email_contents) # Handle URLs urls = re.compile(r'(http|https)://[^\s]*') email_contents = urls.sub('httpaddr', email_contents) # Email email_addresses = re.compile(r'[^\s]+@[^\s]+') email_contents = email_addresses.sub('emailaddr', email_contents) # Dollar sign dollar_sign = re.compile(r'[$]+') email_contents = dollar_sign.sub('dollar', email_contents) # ----- Tokenize email ----- tokens = [] words = re.split(r"\s|\@|\$|\/|\#|\.|\-|\:|\&|\*|\+|\=|\[|\]|\?|\!|\(|\)|\{|\}|\,|\'|\'|\"|\>|\_|\<|\;|\%", email_contents) stemmer = PorterStemmer() for word in words: # Remove nonalphanumeric characters alphanumeric = re.compile(r'[^a-zA-Z0-9]') word = alphanumeric.sub('', word) # Stem the word word = stemmer.stem(word) # Get index if it exists if word in vocabulary: tokens.append(vocabulary.index(word)) return tokens if __name__ == '__main__': # Run a test. # Note: the indexes are all 1 lower than the exercise, because Python is 0-indexed. with open('emailSample1.txt', 'r') as email: email_contents = email.read() tokens = process_email(email_contents) print(tokens) ================================================ FILE: ex6/vocab.txt ================================================ aa ab abil abl about abov absolut abus ac accept access accord account achiev acquir across act action activ actual ad adam add addit address administr adult advanc advantag advertis advic advis ae af affect affili afford africa after ag again against agenc agent ago agre agreement aid air al alb align all allow almost alon along alreadi alsa also altern although alwai am amaz america american among amount amp an analysi analyst and ani anim announc annual annuiti anoth answer anti anumb anybodi anymor anyon anyth anywai anywher aol ap apolog app appar appear appl appli applic appreci approach approv apt ar archiv area aren argument arial arm around arrai arriv art articl artist as ascii ask asset assist associ assum assur at atol attach attack attempt attent attornei attract audio aug august author auto autom automat avail averag avoid awai awar award ba babi back background backup bad balanc ban bank bar base basenumb basi basic bb bc bd be beat beberg becaus becom been befor begin behalf behavior behind believ below benefit best beta better between bf big bill billion bin binari bit black blank block blog blood blue bnumber board bodi boi bonu book boot border boss boston botan both bottl bottom boundari box brain brand break brian bring broadcast broker browser bug bui build built bulk burn bush busi but button by byte ca cabl cach calcul california call came camera campaign can canada cannot canon capabl capillari capit car card care career carri cartridg case cash cat catch categori caus cb cc cd ce cell cent center central centuri ceo certain certainli cf challeng chanc chang channel char charact charg charset chat cheap check cheer chief children china chip choic choos chri citi citizen civil claim class classifi clean clear clearli click client close clue cnet cnumber co code collect colleg color com combin come comfort command comment commentari commerci commiss commit common commun compani compar comparison compat compet competit compil complet comprehens comput concentr concept concern condit conf confer confid confidenti config configur confirm conflict confus congress connect consid consolid constitut construct consult consum contact contain content continu contract contribut control conveni convers convert cool cooper copi copyright core corpor correct correspond cost could couldn count countri coupl cours court cover coverag crash creat creativ credit critic cross cultur current custom cut cv da dagga dai daili dan danger dark data databas datapow date dave david dc de dead deal dear death debt decad decid decis declar declin decor default defend defens defin definit degre delai delet deliv deliveri dell demand democrat depart depend deposit describ descript deserv design desir desktop despit detail detect determin dev devel develop devic di dial did didn diet differ difficult digit direct directli director directori disabl discount discov discoveri discuss disk displai disposit distanc distribut dn dnumber do doc document doe doer doesn dollar dollarac dollarnumb domain don done dont doubl doubt down download dr draw dream drive driver drop drug due dure dvd dw dynam ea each earli earlier earn earth easi easier easili eat eb ebai ec echo econom economi ed edg edit editor educ eff effect effici effort either el electron elimin els email emailaddr emerg empir employ employe en enabl encod encourag end enemi enenkio energi engin english enhanc enjoi enough ensur enter enterpris entertain entir entri enumb environ equal equip equival error especi essenti establish estat estim et etc euro europ european even event eventu ever everi everyon everyth evid evil exactli exampl excel except exchang excit exclus execut exercis exist exmh expand expect expens experi expert expir explain explor express extend extens extra extract extrem ey fa face fact factor fail fair fall fals famili faq far fast faster fastest fat father favorit fax fb fd featur feder fee feed feedback feel femal few ffffff ffnumber field fight figur file fill film filter final financ financi find fine finish fire firewal firm first fit five fix flag flash flow fnumber focu folder folk follow font food for forc foreign forev forget fork form format former fortun forward found foundat four franc free freedom french freshrpm fri fridai friend from front ftoc ftp full fulli fun function fund further futur ga gain game gari garrigu gave gcc geek gener get gif gift girl give given global gnome gnu gnupg go goal god goe gold gone good googl got govern gpl grand grant graphic great greater ground group grow growth gt guarante guess gui guid ha hack had half ham hand handl happen happi hard hardwar hat hate have haven he head header headlin health hear heard heart heaven hei height held hello help helvetica her herba here hermio hettinga hi high higher highli highlight him histori hit hold home honor hope host hot hour hous how howev hp html http httpaddr huge human hundr ibm id idea ident identifi idnumb ie if ignor ii iii iiiiiiihnumberjnumberhnumberjnumberhnumb illeg im imag imagin immedi impact implement import impress improv in inc includ incom increas incred inde independ index india indian indic individu industri info inform initi inlin innov input insert insid instal instanc instant instead institut instruct insur int integr intel intellig intend interact interest interfac intern internet interview into intro introduc inumb invest investig investor invok involv ip ireland irish is island isn iso isp issu it item itself jabber jame java jim jnumberiiiiiiihepihepihf job joe john join journal judg judgment jul juli jump june just justin keep kei kept kernel kevin keyboard kid kill kind king kingdom knew know knowledg known la lack land languag laptop larg larger largest laser last late later latest launch law lawrenc le lead leader learn least leav left legal lender length less lesson let letter level lib librari licens life lifetim light like limit line link linux list listen littl live ll lo load loan local locat lock lockergnom log long longer look lose loss lost lot love low lower lowest lt ma mac machin made magazin mai mail mailer main maintain major make maker male man manag mani manual manufactur map march margin mark market marshal mass master match materi matter matthia mayb me mean measur mechan media medic meet member membership memori men mention menu merchant messag method mh michael microsoft middl might mike mile militari million mime mind mine mini minimum minut miss mistak mobil mode model modem modifi modul moment mon mondai monei monitor month monthli more morn mortgag most mostli mother motiv move movi mpnumber mr ms msg much multi multipart multipl murphi music must my myself name nation natur nbsp near nearli necessari need neg net netscap network never new newslett next nextpart nice nigeria night no nobodi non none nor normal north not note noth notic now nt null number numbera numberam numberanumb numberb numberbit numberc numbercb numbercbr numbercfont numbercli numbercnumb numbercp numberctd numberd numberdari numberdnumb numberenumb numberf numberfb numberff numberffont numberfp numberftd numberk numberm numbermb numberp numberpd numberpm numberpx numberst numberth numbertnumb numberx object oblig obtain obvious occur oct octob of off offer offic offici often oh ok old on onc onli onlin open oper opinion opportun opt optim option or order org organ origin os osdn other otherwis our out outlook output outsid over own owner oz pacif pack packag page pai paid pain palm panel paper paragraph parent part parti particip particular particularli partit partner pass password past patch patent path pattern paul payment pc peac peopl per percent percentag perfect perfectli perform perhap period perl perman permiss person pgp phone photo php phrase physic pick pictur piec piiiiiiii pipe pjnumber place plai plain plan planet plant planta platform player pleas plu plug pm pocket point polic polici polit poor pop popul popular port posit possibl post potenti pound powel power powershot practic pre predict prefer premium prepar present presid press pretti prevent previou previous price principl print printabl printer privaci privat prize pro probabl problem procedur process processor procmail produc product profession profil profit program programm progress project promis promot prompt properti propos proprietari prospect protect protocol prove proven provid proxi pub public publish pudg pull purchas purpos put python qnumber qualifi qualiti quarter question quick quickli quit quot radio ragga rais random rang rate rather ratio razor razornumb re reach read reader readi real realiz realli reason receiv recent recipi recommend record red redhat reduc refer refin reg regard region regist regul regular rel relat relationship releas relev reliabl remain rememb remot remov replac repli report repositori repres republ request requir research reserv resid resourc respect respond respons rest result retail return reveal revenu revers review revok rh rich right risk road robert rock role roll rom roman room root round rpm rss rule run sa safe sai said sale same sampl san saou sat satellit save saw scan schedul school scienc score screen script se search season second secret section secur see seed seek seem seen select self sell seminar send sender sendmail senior sens sensit sent sep separ septemb sequenc seri serif seriou serv server servic set setup seven seventh sever sex sexual sf shape share she shell ship shop short shot should show side sign signatur signific similar simpl simpli sinc sincer singl sit site situat six size skeptic skill skin skip sleep slow small smart smoke smtp snumber so social societi softwar sold solut solv some someon someth sometim son song soni soon sorri sort sound sourc south space spain spam spamassassin spamd spammer speak spec special specif specifi speech speed spend sponsor sport spot src ssh st stabl staff stai stand standard star start state statement statu step steve still stock stop storag store stori strategi stream street string strip strong structur studi stuff stupid style subject submit subscrib subscript substanti success such suffer suggest suit sum summari summer sun super suppli support suppos sure surpris suse suspect sweet switch system tab tabl tablet tag take taken talk tape target task tax teach team tech technic techniqu technolog tel telecom telephon tell temperatur templ ten term termin terror terrorist test texa text than thank that the thei their them themselv then theori there therefor these thi thing think thinkgeek third those though thought thousand thread threat three through thu thursdai ti ticket tim time tip tire titl tm to todai togeth token told toll tom toner toni too took tool top topic total touch toward track trade tradit traffic train transact transfer travel treat tree tri trial trick trip troubl true truli trust truth try tue tuesdai turn tv two type uk ultim un under understand unfortun uniqu unison unit univers unix unless unlik unlimit unseen unsolicit unsubscrib until up updat upgrad upon urgent url us usa usag usb usd usdollarnumb useless user usr usual util vacat valid valu valuabl var variabl varieti variou ve vendor ventur veri verifi version via video view virtual visa visit visual vnumber voic vote vs vulner wa wai wait wake walk wall want war warm warn warranti washington wasn wast watch water we wealth weapon web weblog websit wed wednesdai week weekli weight welcom well went were west what whatev when where whether which while white whitelist who whole whose why wi wide width wife will william win window wing winner wireless wish with within without wnumberp woman women won wonder word work worker world worldwid worri worst worth would wouldn write written wrong wrote www ximian xml xp yahoo ye yeah year yesterdai yet york you young your yourself zdnet zero zip ================================================ FILE: ex7/K-means Clustering and Principal Component Analysis (Exercises).ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# K-means Clustering and Principal Component Analysis\n", "\n", "Stanford CS229 - Machine Learning by Andrew Ng. Programming exercise 7.\n", "\n", "Please check out [the repository on GitHub](https://github.com/rickwierenga/CS229-Python/). If you spot any mistakes or inconcistencies, please create an issue. For questions you can find me on Twitter: [@rickwierenga](https://twitter.com/rickwierenga). Starring the project on GitHub means a ton to me!" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pylab as plt\n", "import scipy.io as sio\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## _K_-means Clustering\n", "---\n", "The K-means algorithm is a method to automatically cluster similar data examples together. Concretely, you are given a training set $\\{x^{(1)},...,x^{(m)}\\}$ (where $x^{(i)} \\in \\mathbb{R}^n$), and want to group the data into a few cohesive “clusters”. The intuition behind _K_-means is an iterative procedure that starts by guessing the initial centroids, and then refines this guess by repeatedly assigning examples to their closest centroids and then recomputing the centroids based on the assignments." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJztnXGMXNd13r+zw5E4S7ka2mYDayyarFFQML01N95YdFgEEd2abmSxCzqNqtABmj8qoGhTi1E3WBVESQJCyYJNZAMtArB2UgRUVcqSupBM17QBMgjMlrSX3mVoWmSRWBapkQOvI61siyNxuHv6x+wbvnlz73v3vXlv5r3Z7wcY5s7OvDnzVvPdc7977rmiqiCEEFIcRgYdACGEkHhQuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGA4CbeIVEXkORG5IiIvi8gnsw6MEEKImTWOz/sSgG+o6m+KyB0ARsOe/P73v183bdrUa2yEELJquHDhwk9VdYPLcyOFW0TuBvBrAP4FAKjqTQA3w16zadMmzM7Ourw/IYQQACLyqutzXaySzQAWAPypiMyJyJdFZF3i6AghhPSEi3CvAfDLAP5YVccBvA1gOvgkEXlURGZFZHZhYSHlMAkhhHi4CPdrAF5T1fMrPz+HlpB3oKrHVHVCVSc2bHCyaQghhCQgUrhV9W8AXBeRLSsPfQrADzKNihBCiBXXqpLfA/D0SkXJDwH8bnYhEUIICcNJuFV1HsBExrGQVcbMXB1HT13F64sN3FOtYGrXFkyO1wYdFiG5xzXjJiRVZubqeOKFS2g0lwAA9cUGnnjhEgBQvAmJIDfCzexrdXH01NW2aHs0mks4euoq/+6ERJAL4S5a9sVBpndeX2zEepwQcptcNJkKy77yhjfI1BcbUNweZGbm6oMOrVDcU63EepwQcptcCHeRsq8iDTJ5ZmrXFlTKpY7HKuUSpnZtsbyCEOKRC6vknmoFdYNI5zH7yvsgUxQbx4upCLESkjdykXEXKfvK8xS/KDbOzFwdO46cxr4T8wCAvds3AgD2nZjHjiOncxcvIXkjFxl3kbKvB+7bgOPnrhkfHwT+DHtEBEuqHb/PW6WGaSHafz/zvjBNSB7IhXADrS9pml/UrCyDM1fMDbRsj2dJUASDou2RpY0T9z6b1giC5G2wISRv5Ea40yTL8sI8edwuIghkZ+Mkuc+u9ykvawZpUpT1B5J/cuFxp02WlR958rhdxC3LtYIk99n1PuVhzSBNirL+QIrBUAp3lllxnhZSbeJWEoEAqFUr+NzHazh66io2T5/EjiOnsX/mEnYcOd3+uRfhSHKfTfcvSF4XpnuBZaQkTYZSuLPMiifHazi8Zwy1aqUtjof3jA1kymsbRP7wtz6GV448iKldW/D8hXpHlnf83LXUsr4k99l0/z6/fWMu7meW5MliI8VnKD3uqV1bOrxXIN0sLu2F1F7iAOzVOFkvBCa9z3m5f/2kSHsVSP4ZSuEuUnmhC2GLWmEiaBIKE0mzvmG7z1mSdTJBVhdDKdzA8GR1SStkZubqEADmAsFOesn6srrPw1aBwUGOpMlQCHcRvuRJY0za/vToqatOop3HrK9o3SJdGZZkggyewi9OFqHMqpcYky5qudgfeV0IZAUGIeEUPuOOykjjZLpZZe62GB9/9iIAexY5M1c3bmMHou0N22KYx/rRMs5O7wy9xqDuHSswCAmn8MId9iWPM+UexG7LJdXIeEyiHWVvzMzVcePmrdCY3rzRDP296X48dmIeh166jAMPbe2IN+zeAfF9XVZgEBJO4a2SsFriOFPuNKbnXte74OaWMMGJEw/Q2lwTZm94IholzFHY3v/NG03sOzGP/TOXQp/baC7h4IuXE1lEedrkREgeKbxwh33J40y5e52eh/nYUbsF48SzrNpzEycAqFbKob8Ps1kUwNPnrrUF2BbrYqOZaDDM0yYnQvJI4a2SsDKro6euGgVoRASbp092PLfX6XlYxu55yY8/ezHSr/a8YltFyN2VMnYcOW21HlwGmvKI4ODuraHPKVm8dQ8F2usIUX56EJcYWYFBiJ3CCzdg/5KbNj0At9uf+r3YXjdIRGXsXnxh7xH0ioOURwRv37yFxUazK37v+i4ievSffSxSFMNE26O+2MCOI6dRX2x01YxXyiWsLY8YLRt61YT0RuGtkjCCU+6SSNdz/BUovUzPXfp2RL1HmM1Rq1Zw19o1aC6ZD0rwiLJlatWKcSE06M3XHMRVcNtS0ZWfgdZ99j5HeaTznvfTq7atORBSdEQdMqu4TExM6OzsbOrX7ZXN0yeNFoQAeOXIgz1d25QtV8olo/jbSuei4nONf2aujkMvXe7Kdsslwbo71uCtRrP9voB5FvC5j9fw/IW6dSCx7coMPm56z35YIHH+HoTkARG5oKoTTs91EW4R+RGAnwNYAnAr6uJ5FW5vWh+kJII//K1o+8CPSXyB6NI3k6B44uZZIEFq1QrOTu+0xl+tlDF/4NOhMVZHy/jFO7fQXL799w6zM2or8ftfr4q2AMfxtL34+4ntXg0iFkJciCPccTzuB1T1pwljygVhnnecmm1T3fK+E/PYu31jpCgcfPFy1/s3l9Qq2n5rYWrXFkx99WKH+ALAz95pYmauHhr7zxq3jOdR2jLq+mIjdIHQJoy2awUXg7Mmqr4/7y0SCAljqD3uIJ7HHOZ1u2DyooMlciZm5upWgTYR9MAnx2sol7pjX9bWgBB8L395ostiox/TPfJj8tLDXuGVSE49dxHbDn0zc9/ZtuZQHS3nvkUCIVG4CrcC+KaIXBCRR7MMKGsmx2tY7vFQXdvzvBI5P/4FMm+LuwsC4Oz0zq5M8EZz2fj84IDgWs9tI0roTQute7dvjDzdxptdeKK578Q8NmUg4rb6flWwDwopPK5WyT9U1bqI/F0A3xKRK6r6F/4nrAj6owCwcePGlMNMl+pouacytTCP1y/qrqew296jF3rt6+FVlcTtBT7xofe2n+/yab3npN0B0Fbfv+/EvPH57INCioSTcKtqfeX/fyIi/wvAJwD8ReA5xwAcA1qLkynH2TOeANkEt1wS5zI1TwBMH9IvuEmzXgHwwH0bjL9bbxl0RNDhc8ddQPTj+epJ+rf4xTyODw70dhpPVCwetv8GWFtOikSkVSIi60TkPd6/AXwawPezDixN/H6vjXV3tMYwl7rfyfEa9m7f2OXpBmuUXU9h3/Hh93ZcSwE8f6FufP8DD201+tyq6PBqXQ7lNeH31Xvt35IkhqwzX/ZBIcOAi8f9SwC+LSIXAXwHwElV/Ua2YaWLS+a72GjGWrR6cnIMe7dvbC/ilUTwuY93Zngup7Af3jOGH/1tt63gtX0Nvv/keA0P/8q9xuv6RdXzoNePhvck8aiUS/jiw9s6fPVe+rd4M5xGc6l9j9aPlrs25ATJOvNlHxQyDERaJar6QwAf60MsmeEiNP7dfh5hU/eZuTqev1Bv+9ZLqnj+Qh0TH3pv+/m2bfRBobD5rrYyxTNXFqyfw/9Zvaw5qlNgzeddR1lKQLS4zszVO8oWl1RRHhEceGgrZl99A8fPXTO+ToC+ZL6roQ8KSx6Hm6HoVRJFlN9bKZesGblN9F2OFHM9Z9C2WGq6ZlhMQOuz+r+0YYsN5ZLg6G/e3ngU1SsFcLMVDr54uavWvLmsOPjiZay70/6fnKLYR5PlhWE9+o3cZuiF23aogLc1279DMM6ilauNEJXdzczV8Yt3wg89CF7TNhB5i5pR4utx151rsO/EPI6eutq+B2Gvi+oF7mGrVV9sNPFWSB27S3+UKJhpJj+nlBSHwgq3yxfUlkFWK2Uc3L216/lxugOmdUrL0VNXu7LTqGuaLBgBsHf7Rpy5shAp2uURAeT2KTheRhb1uqhe4C6EDTq92iTMNFvw6Lfhp5A7J10P37VlkOvuXNP1RY67aGWrTtj0vgo+/MTXsWn6JD78xNc7TooxEfVlMg0eplifengbnpwcC72e91xbl8Go3ZKug5JtQXS0PGLdcbl3+8aexZWHDLdw6VRJik0hM27XqWDczCPOopXJv970vgrO/vUb7ecsqbYX4p6cHDNeJ8x/r4VM9W2x2vzy9aNlzP2HViOqzdMnje+3pGr1++OUzB14aCt+/9l5BCcS3szi8J6xTOwMZpoteu0tT/JPITNu1y9ovzOP//PDN4yPP3P+uvU1cXt+RGHbnOl/3Pb5vVmG5zV7GXjckrnJ8RruNhyN1lzS9uB6dnonXjnyoHFbf1KYabZgyePwU8iM29VfzjLzMPmpNsK2uvsz9+BJMl4vj8dOzHdk3/tnLuGZ89expIqSCB65/952Rm9b/PMen5mr4+13uxdDvfvSS6mcSzVLltkvM83brIaSx9VMITNu191vWWYecbez+/334MksQKuhVK1a6RK8YC+Pvf/t/+L4uWsd9ePHz11re+lhWac32ASrPtaPlnu+L8F1BxtZZr/MNMlqoZAZt2t9tPfcLL64cTNHr7rB+7ep8iHqmo3mUoeH7ueZ89fx5ORYaNZpG2xG7+herI2Ly0DWj+yXmSZZDRRSuIHBf0Ftds1oecTYetVf3WBbWO2lMZSXgYcNall2xouqZlmtNdWEZEFhhXvQ2DLb/7hnzNo5MEzcXl9s4KmHtzlvngniL+WzDWpp1Z6bsF2bR4URkj6F9LjzQJifGuYzh/3Of02gu7rE6yRo4pH7zY2n/GTZGY9d9wjpH6vqlPd+EXbCOGDeoRnnNHhTVYn/AIMwWyLLLeHcbk5IclI/5T0uq124gXARS1vgTAOFtxvRtvGHEJIvKNyrDNtJMwLgqYe3MeslpADEEW563ENAnMOLCSHFh8I9BIRVhay2Ph2ErAYo3EPA1K4t1v4mq61PByGrAQr3EOB6eDEhZDigcA8JT06O4amHt7FPByGrAO6cHCIG3QaAENIfmHETQkjBoHATQkjBoFVCSI5g2wDiAoWbkJzAU+qJK7RKCMkJPKWeuMKMO+dw6rx64Cn1xBXnjFtESiIyJyJfyzIgcpvgOY7e1Nl/fiUZHnhKPXEljlXyBQAvZxUI6YZT59UFD6MgrjgJt4h8EMCDAL6cbTjED6fOqwueUk9ccfW4vwjgDwC8x/YEEXkUwKMAsHHjxt4jI5meEUnyCXe/EhciM24R+SyAn6jqhbDnqeoxVZ1Q1YkNGzakFuBqhlNnQogJl4x7B4DdIvIbANYC+DsiclxVP59taMTLvFhVQgjxE+voMhH5dQD/TlU/G/Y8Hl1GCCHx4NFlhBAyxMTagKOqfw7gzzOJhBBCiBPMuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGBQuAkhpGDwzElCyFAxM1fHoZcu480bTQBAtVLGwd1bh6qrJoWbEDI0zMzVMfXcRTSXbnc9XWw0MfXViwAwNOJN4SaEFIqZubq1R/3RU1c7RNujuaw4euoqhZsQQvrNzFwdT7xwqX2Idn2xgSdeuITZV9/AmSsLxqP+PIbprFYKNyEkFUyZMJDuCU5HT11ti7ZHo7mEp89dQ9SRMMN0ViuFm5AeCZu69zuG+mIDJREsqaIWEUuacZsy4amvXgQEbevCy46B5F6zLWuOEu3yiAzVWa0UbkJ6wDZ1B/q3EBaMYUmjhTKNuP3CP7IyWPhpLnfLaaO51JPXfE+1EmqHmGBVCSGkA9vUvZ8LYaYYTLFECa0tbpsF4q/eCF4rjF685qldWzoGHAAQmDPuWrWCs9M7E79XnqFwE9IDNhHq50JY1Hu9vtiwZuVR17Jl5iM+CyQuvXjN/uoRbyB54L4NeP5CvUPMK+XSUFkjQSjchPSAberez4WwKPvg7ko5NCsPXsuPbUbhQnlEOjxuoFNQk3rsk+O1rudNfOi9A19n6CcUbkJ6wDR1zzLbM4mdKQY/Iu4zgGDcSWYOAkRWlUSV9aUh5sMMhZuQHjBN3bPK9mxid3jPGA7vGcNjJ+aNr1u80XRe1AvGHXcxUAzXM90LWyZ//Ny19s/9XOjNQ2VQHERjLCq4MjExobOzs6lfl5DVzI4jp40i6i3Chf3+gfs2ONU6/+jIgx0/BweLOFTKJRzeM9a1MBp3MMh6kdH0Gf2x9wsRuaCqEy7PZcZNSEGwiV19sdEW7WCFRaVcwqb3VZxEu1opdz0WnFGYqlFseFUqALpmCrZKEBNZL/TmoTIoLhRuQnKIaepeChFNT9QVt8vj4mTa5RHBwd1bjb/z2x2bp0/G+hz1xQYef/ZiV9xx5vlZL/TmoTIoLhRuQkIYhPdp87JdM10FsH60lT37PWMbUTss/dhsDtugIohX4x2kH2V9eagMikvkQQoislZEviMiF0Xksogc6kdghAwaT0Driw0obgvozFw90/e1Td2DC39hvHmj6eQj+/3jHUdOY/P0Sew4ctr6Gad2bUGlXOp4rFIu4ZH77+16PI4d4lEaEVQrZchKbP3wmW2fKc914C4Z97sAdqrqL0SkDODbIvK/VfVcxrERMlD64X2aMvqk/TiSUF9sYNuhb+Ltm7eceoqEVdEEa6njbk0HgPfcuQbzBz4N4Pa92XdiPtPZTj8rg9IiVlWJiIwC+DaAf6Wq523PY1UJGQY2T580iqUAeCVQfZEEWzXD2vJI+/QWFyrlUqKqjyh6reawVbmE4d1b10qPopXxhRGnqsTpzEkRKYnIPICfAPhWmGgTMizYPM60vE9bRq+Krql7GIf3jKFWrbTthTiWShhetUpSayiJ1eDd27DZjsegrKw84CTcqrqkqtsAfBDAJ0Tko8HniMijIjIrIrMLCwtpx0lI38na+7RZIm81ms5iXBLB5HgNZ6d34pUjD+Ls9M5ULZVexHByvNZeJHXBf29dKj1cxH1YiVVVoqqLInIGwGcAfD/wu2MAjgEtqyS1CAnJGNt0Oyvv03s/25ekOlru2nG4f+aSsULkkfvv7XqsltBfthHl64fZFQce2uq0gackgs99vNb2tG314v7ZTi9lfEW3WCKFW0Q2AGiuiHYFwD8G8J8yj4yQPhDVlzrtHhguOxFNy05PTo4BAJ45fx1LqiiJ4JH7720/7mdq1xbsOzEfmnmPCGBol22lvtJhMG5fb//gZxtMyiXBw79yb0eHP5NoB2c7Scv48tBDvVdcrJIPADgjIn8J4LtoedxfyzYsQvqDbbp96KXLfXu/IG81uhcmZ+bqOHNloS3aS6o4c2XBaGFMjtewd/vG0PdIMiU2WSYudoVn5dQsgrrujjU4c2XBeF9KItbSQFcra2au3lHqeOily4W3WCIzblX9SwDjfYiFkL5jm1a/eaNpzDBN+Kfd1dEyVFvia5qCu0zjgxljkhNuvEzctgEnyZ6YRnMJv//sfLuZVbVSxqJhkAHMnzPM0zcNVgCwrGqt4HGxskzZtY0875QMwp2TZFUTVm/sUq8dFAZ/GZ937uKhly63O/TdHSJ2gDljdD3hJsiTk2P42sUfG98vbPt8GH57JexzmOyKKGsjie0RZWW59iF3ea884VRVQsiwElYh4pKBRQlDc1nx5o1mu1zt7Zu3WgcMWFhb7v5KupxwY+Pg7q3WnY5plQ0GCQ4+nlXhNZcyPbeXCp6gFeK3c1yz6LzvlAzCjJsMPWEVBJPjNRx88bIxe3TJwOJOr5tLivWjZYzesQavLzZwd6XcsWvxzRvNLvsjahdiWJxRdoJLAyoXSiJYVm0fJeZVhwQ/X7AJVtDaiFPpMTNXx6GXLnfNcvz3z3bvqpUy1t25prBVJezHTXJLGiVbLjvweunH3MvuwLDX+3cthlWi2OJ0vXfe89IoHxSgS6ht9LorM6o6x7t+Xnptu5D6zklC+k1au+Jcqx6CG15cv9imKX4UcWuR/fEBrewWIXHGuXdexUcYIc5OB4qW7+1yiHCvC4FRFpV3/V7+tnmGVgnJJWk1eHLdpJG0XjtoRfirSkzZZ9Ja5LD4gtn12+/eSrU5Vpx6b1d6XQiMEn7/9YfxPEoKN8klaTW370ev5TiiGrQsej1sOK1yN5FkJYJJSGMhMMz3L9pCYxIo3CSXpCW4/T6FPUhUttfrtvq0yt323r/R6dCFJJRHBHetXdMuiUxjIdB2sn21UsbB3VuHLsMOQuEmuSQtwS1Cr+VepvKuMxBBeOnjk5NjeGXhFzj71284X69cEtyM8LOzEtIi/F2zhMJNckmaX8xh9Dg9XA8sUET34Xj6X34SM3N14xmRQGfJ39SuLTj44mXcDNmEAwDv3lqOjC0pw/x3jYLCTXLLav5iumKzDILY+oQE8e63SwndvpWt72Hk/bT0osJyQEIKRHCXINB5kEK1Uka51Fm/F9dici2hc11vKFIPkKLADTiEFIS8Hefl0qIW6H2zzWohzgYcCjchBcFll2W/8Q8Strr1w3tanQqHeSExjcEyjnDT4yakIKRV254mwXUIk4ABKPzBBWEM4mAGCjchBaEfm4l6xbSgvOPI6VR3cuaNtHb5xoGLk4QUhKwPL86KPM4U0mQQn4/CTUhBKGrDJNuMIE8zhV4YxOejcBNSICbHa5jatQX3VCt4fbGBo6euxu6Y2G+KOlNwZRCfjx43IQUiDyeUx62gGPbt6YP4fCwHJCRHRInioEsCi3QwQdHgQQqEFBCXAxAGvdDncjAFyR4KNyE5wUUUB73QN+iBg7SgcBOSE1xEcdALfYMeOEgLCjchOcFFFAddEjjogYO0YFUJITnB9fCIQba7HfYKkaJA4SYkJxRFFNknffBECreI3AvgzwD8EloHaRxT1S9lHRghqxGKInHBJeO+BeBxVf2eiLwHwAUR+Zaq/iDj2AghhBiIXJxU1R+r6vdW/v1zAC8DYEpACCEDIlZViYhsAjAO4Lzhd4+KyKyIzC4sLKQTHSGEkC6chVtE7gLwPIDHVPVnwd+r6jFVnVDViQ0bNqQZIyGEEB9Owi0iZbRE+2lVfSHbkAghhIQRKdwiIgC+AuBlVf2j7EMihBAShkvGvQPA7wDYKSLzK//7jYzjIoQQYiGyHFBVvw1A+hALIYQQB7hzkgycuI35CVntULjJQMnDiS6EFA12ByQDhY35CYkPhZsMFDbmJyQ+FG4yUNiYn5D4ULjJQEnamH9mro4dR05j8/RJ7DhyuuNcRkKGHS5OkoGSpAc1FzTJaofCTTInqtwvbg/qsAVNCjdZDVC4SaZkkR1zQZOsduhxk0zJotyPC5pktUPhJpmSRXYcZ0GTi5hkGKFVQhLhuk39nmoFdYNI95Id2xY0AWDHkdPtxx64bwOev1DnIiYZOkRVU7/oxMSEzs7Opn5dkj5J+oQEfWuglfEe3jPW9VrTc8sjgrvWrsHijWZqvUlM7yNonW4dpFat4Oz0zp7ej5C0EZELqjrh8lxaJasYT+zqiw0obmekUXZCHN96cryGw3vGUKtWIACqlTIgwJs3mu333HdiHpt6tDJMMdlSEi5ikqJD4c4R/fZjky4cxvGtgxm9CNBc6pRU7yfXgSNOTCa4iEmKDj3unNBL2VzStqhJFw5dfWvTZ4oiaT22LaagXeKyK5OQvMOMOyckzX5d7A5bJp+0rM61qsP0mVxIYmXYYtq7fWPbpqlVK0YfnpCiQeHOCUmz3yjBDxP2pH1Cgr61TRCTeskKYP/MJevvTQORKabPfbyGM1cWeEADGTpoleSEpGVzUYIfJuxeZcWhly7jzRtNAMCda9zGcpdt6rbP5MLxc9cAAE9OjnU8HmUpeTG5Wk88fYcUEWbcOSFp9htld7hk8u80l9v/Xmw08cQLl7B/5lLPC6WmzxSHZ85f73rMNhA9/uzFjhhdrKekVTWEDBoKd04wlc2tLY9g34n5UOF84L4NoY9HCbtN4J4+d61nQfN/piQsGfYY2AaiJdWOGF0GLNtnP/jiZe62JLmGwp2ArMr2JsdrODu9E089vA3v3lruqHW2CeeZKwvGaz1z/jo2T5/EjZu3UB6Rjt/5M3mbwAUlM2l/kcnxGqZ2bYFEP9WI64JqMEbb80ZE2te02TiLjSazcJJrKNwx6cf0Ok6FSVgGqmhtdGku35bhaqXcsZAYp6Y56WLj0VNXrZthogTdu8ePnZjHtkPfxAP3bQi1X7wYbTMR777UFxvOgwnPwCR5g8Idk6wOt/Wy+E3TJ62ZoEk4424meffWcsfPU7u2oFxykzDvveLOOMIEP07DhcVGE8fPXQstMfRitM1Egu/tKt5JF1kJyQIKd0zS6HYXFL79M5faWXwYJpGOuwBoHGQC6jkCdIm5Z68kmXHEGVw8jz8JAkRaQEHU9561agXrR8vWa9MuIXmBwh2TXntBm4Tv6YgsErBXmAQXNUsSLXvBBTq/lQIAywDW3bHGWKedZMZhsy1M1FfK8pKgQGwLyGs49cqRB3F2eicOPLTVOHAoQLuE5IbIOm4R+RMAnwXwE1X9aPYh5ZupXVuMnfFct1HHaYbkx+9Lm2qPvZrsmbk69p2YD72mX9RsmelbjSbmD3y663GXGcf+mUv4H+evYTmOD+LDW1ANDihR+KtXTH+nIKa/2+R4DY+dmDc+v77YaG/2IWSQuGzA+e8A/guAP8s2lGKQ5HBbP0kW+GrVivPGksnxGmZffQNPn7tmFO+gWNk2ydxdKbd7W1dHy1BtifmIiLFMb0QEM3N1zL76RnvzTFLevNFEuSQYLY/gRnM5+gXo/ly2++D1LqmF/N1qIRuH2M+b5AGnftwisgnA11wzbvbjtmMrQ7P1jg72uba9Pthjemau3rEjEgDWj5Zx4KGtXTsHTf2ylwEsxcx4K+US3rm1hLRavNdWDkOwDUJRIux6r4KY7kmc1xOShDj9uLnlvc/YrBavr0Z9sYHSSlZrEqQ4i6PvBLLV4M+AeQaxeOMm3r4ZvzlUkoZSYby+2MCTk2OY+NB7cfTU1ch7Y3p9nMc9vGvaLBP28yaDJjXhFpFHATwKABs3bkzrskNHr1aLa0+TsEXE4HsF+45smj7pFEvWeJ/JpS+K7fVx+7/41w9KFluI/bzJoElNuFX1GIBjQMsqSeu6w0hSIQLcF0ezOKTXj03U7igJbi6Z//zVShnK8fOWAAAI10lEQVSLjabxd0HS6Jsddq9MC7wAOp5v+nzs503yAK2SguGascfNNv1CJoJIn3r731uP7117q0sU71wzgpsGca5Wyji4e6vRO14/WsaD/+ADoS1Yk3TxCztU2LTAu7Y8YrR7SiJYVmX3QJIbXMoBnwHw6wDeLyKvATigql/JOjBixyVjj1O22LUY5zBf+tHfNnB4z1iXKO6z+MJvNZqJbaJeTgfyqkueOX8d9cUGHn/2Iu5YI2gE/P5Gc8nq0S+r4pUjD4a+DyH9JFK4VfWRfgRC0sVUDjdi2ZtjO6kmLPN+fbHRNYDMzNWt5YK9+NVx/HovDm9waGXRt0V6SRWNZjwnj542yRu0SlIkT035Z+bqOPGd6x3J89s3lzD13EUAcKpUUW3ZGP6SQo+gmO2fueRcOx6XuIcT+7PzYGYdRrVSxru3lhNvriKkX3DLe0rkrSm/aSs70DphPbh1OyyjVEXkAQ8zc3WraJdEej7nsWrpH2KKO+k5l5VyCQd3b3U6ko2QQcOMOyXiTuezJqx6JPi7qV1brDXLbzWaeOrhbaEzibC2rcuqkXZG2OxkZq6OX7xzq+vxckmMmbBr1cyIAB+4u2J8fwo1yTsU7pTIuvwurg0Tdt5jMFOdHK/h4IuXjaV696xst0+y0SX4XvtnLuGZ89e7PPCwxUbbzGHdHWuMMbmec/nb92/sOs+SkKJAqyQleu0aGEYSG2Zq15auk2+AVqb6wH0buvppH9y9NdGZl4D9M/rbrO6fuYTj564ZFy6B+AdFvGWpBze1uR3B7YXZkgg+v52iTYoNM+6U6LVrYBhJbBjvcX8m7dVLP3+h3lVad3jPmLG8z8U2MH12AfCrH25tVY/qVuhhOygiTj163JLDLBaU87RITYYTCndK9LqVPYxeem6YGi/ZBoGz0zsTxWv67A/ct6FjgHDBdlBE3AHRteSwl/rwfl6TkCAU7hTpZSt7GEl6btjIyosPfnbTABFG2EERQDYDYhYLynlbpCbDCYW7AKRpw6Q5CIQRZyCI6vSX9oDoWRlxzvZ0tT+yXqQmBKBwF4I0s84svXg/UdUdJRE8cv+9fV8kjOq1DXQPYnHsj34NjGR1Q+EuCGllnWkMAi7Zp22AGPSGlqgNOqZBLI790a+BkaxuKNxDhouo9jIIuGafWXrTwXjivEeUZXHnmu4K2Tj2R78+N1ndULiHiH5UNPRyQENcokQ5yeeNsnAWG82uayQpSaRQkyzhBpwhwiaqjz97sWOzTS/0a/HNZdNR2CBiw7RBJ0jwGqbX0P4gg4QZ9xBhE09vt2IaGXi/Ft9cMvskg0jQyrBtDPJfY9D2Bzf0kCAU7iHCpU9HLzXFM3N13LjZ3fApi+zTRZSTDiJ+K8N2Erypn8sgxJIbeogJWiVDhIsNACSzNTwBCfbmrlbKPVWKzMzVu/qmAG69X9KwMPJugySxg8jww4x7iAhO6aNOo4mDrYxu3Z3mLn0uhGWTLmV1aVgYg7ZBouCGHmKCwj1k+Kf0ps0mSbPJMAFJ6sGGZZNnp3e2n5NVaWOa18gKbughJijcCSjKYlGa2aRNQKqj5cQebFQ2mWdB7Rfc0ENMULhjUrTForTEzyYgqkjcVCkv2WSeB+K8WzlkMFC4Y7Jau7/ZBGSf5cgzFw82D9lkEQZizjxIEAp3TFbzYpFJQGxd9lyy5jxkk6t1ICbFhsIdk7xM7/NCr1nzoLPJ1TwQk+LCOu6Y5L3ut99MjtdweM8YatUKBK3e2oPuABiHLM8KJSQrmHHHJA/T+7wx6Ky5F/LgsxMSFwp3AoosVKQTDsSkiDgJt4h8BsCXAJQAfFlVj2QaFSF9hAMxKRqRHreIlAD8VwD/BMBHADwiIh/JOjBCCCFmXBYnPwHgr1T1h6p6E8D/BPBPsw2LEEKIDRfhrgG47vv5tZXHOhCRR0VkVkRmFxYW0oqPEEJIgNTKAVX1mKpOqOrEhg0b0rosIYSQAC7CXQdwr+/nD648RgghZACIGvo1dzxBZA2A/wfgU2gJ9ncB/LaqXg55zQKAV1OMM8j7Afw0w+sXHd6fcHh/wuH9CSer+/MhVXWyKyLLAVX1loj8GwCn0CoH/JMw0V55TaZeiYjMqupElu9RZHh/wuH9CYf3J5w83B+nOm5V/TqAr2ccCyGEEAfYq4QQQgpGUYX72KADyDm8P+Hw/oTD+xPOwO9P5OIkIYSQfFHUjJsQQlYthRJuEfmMiFwVkb8SkelBx5MnROReETkjIj8Qkcsi8oVBx5RHRKQkInMi8rVBx5I3RKQqIs+JyBUReVlEPjnomPKEiOxb+W59X0SeEZG1g4qlMMLNZleR3ALwuKp+BMB2AP+a98fIFwC8POggcsqXAHxDVe8D8DHwPrURkRqAfwtgQlU/ilZp9D8fVDyFEW6w2VUoqvpjVf3eyr9/jtaXjr1KfYjIBwE8CODLg44lb4jI3QB+DcBXAEBVb6rq4mCjyh1rAFRWNiWOAnh9UIEUSbidml0RQEQ2ARgHcH6wkeSOLwL4AwDLgw4kh2wGsADgT1espC+LyLpBB5UXVLUO4D8DuAbgxwDeUtVvDiqeIgk3cUBE7gLwPIDHVPVng44nL4jIZwH8RFUvDDqWnLIGwC8D+GNVHQfwNgCuI60gIuvRmuFvBnAPgHUi8vlBxVMk4WazqwhEpIyWaD+tqi8MOp6csQPAbhH5EVo2204ROT7YkHLFawBeU1VvlvYcWkJOWvwjAK+o6oKqNgG8AOBXBxVMkYT7uwD+vohsFpE70FoYeHHAMeUGERG0/MmXVfWPBh1P3lDVJ1T1g6q6Ca3/dk6r6sAypryhqn8D4LqIeKckfwrADwYYUt64BmC7iIyufNc+hQEu3hbmsOAkza5WGTsA/A6ASyIyv/LYv1/pM0OIC78H4OmVxOiHAH53wPHkBlU9LyLPAfgeWhVccxjgDkrunCSEkIJRJKuEEEIIKNyEEFI4KNyEEFIwKNyEEFIwKNyEEFIwKNyEEFIwKNyEEFIwKNyEEFIw/j/h7eCdkxWSXAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "data = sio.loadmat(\"ex7data2.mat\")\n", "X = data[\"X\"]\n", "plt.plot(X[:, 0], X[:, 1], 'o')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Implementing _K_-Means\n", "\n", "The algorithm consists of two parts:\n", "1. Assign each data point to the closest centroid. $idx_i$ corresponds to $c^{(i)}$, the index of the centroid assigned to example $i$.\n", "2. Compute means based on centroids assignments.\n", "\n", "One runs the algorithm for a certain number of iterations. The algorithm will always converge, but not necessarily to the optimal value. Therefore, one should run the algorithm multiple times with different random initalizations and choose the one with the lowest cost function value (distortion). \n", "\n", "**Exercise**: Start by initializing _K_ random centroids for the data." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "def initialize_K_centroids(X, K):\n", " return None" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following should return three points from the dataset." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "initialize_K_centroids(X, 3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise**: Implement part 1 of the _K_-means algorithm (cluster assignment step)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def find_closest_centroids(X, centroids):\n", " m = len(X)\n", " c = np.zeros(m)\n", " return c" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "K = 3\n", "initial_centroids = np.array([[3, 3], [6, 2], [8, 5]])\n", "idx = find_closest_centroids(X, initial_centroids)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise**: Implement part 2 of the _K_-means algorithm (Move centroid step). Make sure it is vectorized." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def compute_means(X, idx, K):\n", " m, n = X.shape\n", " centroids = np.zeros((K, n))\n", " return centroids" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following code should return approximately:\n", "$$\n", "\\begin{bmatrix}\n", "2.428 && 3.158 \\\\\n", "5.814 && 2.634 \\\\\n", "7.119 && 3.617 \n", "\\end{bmatrix}\n", "$$" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0., 0.],\n", " [0., 0.],\n", " [0., 0.]])" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "centroids = compute_means(X, idx, K)\n", "centroids" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running _K_-means" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "def find_k_means(X, K, max_iters=10, plot=False):\n", " _, n = X.shape\n", " centroids = initialize_K_centroids(X, K)\n", " centroid_history = np.zeros((max_iters, K, n))\n", " for i in range(max_iters):\n", " idx = find_closest_centroids(X, centroids)\n", " centroids = compute_means(X, idx, K)\n", "\n", " if plot:\n", " centroid_history[i] = centroids\n", " \n", " if plot:\n", " for centroid in range(K):\n", " # Plot examples for this centroid\n", " examples = X[np.where(idx == centroid)]\n", " plt.plot(examples[:, 0], examples[:, 1], 'o')\n", " \n", " # Plot centroid history\n", " history = centroid_history[:, centroid, :]\n", " plt.plot(history[:, 0], history[:, 1], '-xk')\n", " \n", " return centroids, idx" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJztnX+MHOd537/PLYfkHu3qqPpa2GtRZIOChGlGvPhi02URhHRjupGtXmWnjCsHaP6ogKJNLUK94BwIEQWoFQu2kQ20CCDYSVuIdalfPUhmYNoAGRhmQcVHHxmaEgnElkRp5cBnS0dZupW4vHv6x94cZ2ffd+ad2Zndmd3vBxBE7s3OvDvH/b7PfN/neV5RVRBCCCkPI/0eACGEkGRQuAkhpGRQuAkhpGRQuAkhpGRQuAkhpGRQuAkhpGRQuAkhpGRQuAkhpGQ4CbeIjInIUyJyWUReFJFP5j0wQgghZtY5Hvc1AN9W1S+IyHoAo1EHf+ADH9CtW7d2OzZCCBkazp0793NVHXc5Nla4ReQWAL8B4F8BgKpeB3A96j1bt27F3Nycy/UJIYQAEJFXXI91sUq2AVgA8OciMi8iXxeRTalHRwghpCtchHsdgF8D8KeqOgHgHQAz4YNE5F4RmRORuYWFhYyHSQghxMdFuF8D8JqqPr/696fQEvI2VPUxVZ1U1cnxcSebhhBCSApihVtV/xbAqyKyffWlTwF4IddREUIIseKaVfIHAI6tZpT8BMDv5zckQgghUTgJt6qeBzCZ81jIkDE7X8fRk1fw+mIDHxqrYvrAdkxN1Po9LEIKj2vETUimzM7X8ZVnLqLRXAYA1Bcb+MozFwGA4k1IDIURbkZfw8XRk1fWRNun0VzG0ZNX+HsnJIZCCHfZoi9OMt3z+mIj0euEkJsUoslUVPRVNPxJpr7YgOLmJDM7X+/30ErFh8aqiV4nhNykEMJdpuirTJNMkZk+sB1Vr9L2WtWrYPrAdss7CCE+hbBKPjRWRd0g0kWMvoo+yZTFxvHHVIaxElI0CiHc0we2t3ncQHGjr6JNMkGhHhv18Pa7N9BcUQDFXSsITy737NmC05cXcOj4eRw9eYUCTkgMhRDuPKKvvCLPfTvG8fjZq8bXe014UffNpWbHMUXL1DAtRAfvZ1EnG0KKRCGEG2h9SbP6ouaZpXL6srmBlu31PDH57SbytHGSTpAuYy7aZENI0SiMcGdJnjnCRfK4Xa+Zl42TZoJ0HXNR1gyypCzrD6T4FCKrJGvyFNcipbG5XNMbESxdv4FtMyew98ipTNMW02TYuN6nIi5MdwPTSEmWDKRw5ymuRUpjM43FqwjGqh4EwFjVA6TlfftiMf3UBex+6DuZCHmaCdI05jBFXZjuBqaRkiwZSOHOU1ynJmp45O5dqI1VIQBqY1U8cveuvjzymsZy9At34PyDn8ZLR+7Epg3r0FzWtvc0lxWLjWYmUV+aCdI05i/t2VKI+5knRbLYSPkZSI877xzhLBdSuyVqLKa0xTDdeP9p0ziLdP96RdHSSEm5GUjhBgZLHNIsas3O1yEANPKoFmmjvjwnyEFbyCtTrQIpPgMr3INC2tTGoyevOIk20F3Ul8cEWbamYy6wUpRkyUAIdxmis7RjTJva6BpFFzHqG9SWr4P0FEj6S+mFOy46K4Kom8Z43/HzeOi5S3jwczut45mdr1t96jhhtnmqQWox9yPJvcvyPnMhj5BoSi/ccWlWSR658xJ5W7Xgm0tN63h8sbcRZW/MztexdP1G5Jg2j3o4M7M/8hyuk03U5Akktwe4kEdINKVPB4yKzpLkzmZRIDE7X8feI6c6cqSjIkXbeKJKw6PsDf9zmPqWBIn7edRkc+j4eTwwezHy2EZzGYefvZTqnhYpV56QIlJ64Y7KJU7yyN1tgUSU8MdFiqbxRIl9VJ6za/+SOKJsFgVw7OzV2IlpsdFMdU+LlCtPSBEpvXBHRWc2wRwRcY6KXX3VKOGPqxYMjtOP2m0ZIWNVD0dPXrFWPrqOd6zqRf68IhL5cwXWBDipheEyxqmJGs7M7MdLR+7EmZn9FG1CApTe445LswrnzgLAsnb2q+7WV40Sfn8sh5+9hMVGu0URtADCXnEYb0TwzvUba+cwefYui5LeiODwXTsjj/HvURT1xQb2HjmF+mKjI2e86lWw0RsxWjL0qgnpjtILN2BPswqL+ohIhyAFo+JuCiTihN8fY9QCaJTNURurYun6jQ4hDKfJmT5HkIoIjv7OHR2Li+Ex1RwmAMFNS0VX/66r12g0l7HRG4E3ImsbOwC99aqLkFFESB6U3iqJI/jIvWKJIv2ouBtf1XVBbWqitmbj+AuocXaNADgzsx+LlgXF4Pv8z7F5tNMK8SqC929ch0PHz6/ZLDZvft+O8Uh7x1SV6Yu3Pzm+udQEBGtNr3rpVbMbHxlknCJuEXkZwC8BLAO4oaqTeQ4qL2xR8YgIZufrXRVIuFbGmVLnpp+6gMPPXrL62n7Ubhv/LSG/2hTd+9uahW2Wjd6I0Zs/fXkBj9y9q+39qsC1RjPSjgl/huayYtOGdTj/4Kctny4fBrWIhxAgmVWyT1V/nttIeoDNRlhWTVxSPTtfb/OsN496ePBzOyNzo4GWzx2+vt+xz0Qwap8+sB3TT15osx4A4K13m2sTj423GjeMNpHNUqmvPoXYzul72y7UFxvYNnOip3ZF1JoDLRRSdgbeKgni2wimjImkqX/TT15oE9s3l5qYfupC5KP47HzdKtAmwtbC1EQNXqVz7CvamhDC1wpaBS6LjUHiskpM1lDUO/LoBx6FbQF0bNSjhUJKj6twK4DviMg5Ebk3zwHlzdRELdLrduHoySsdUS/QipzD4h8syrn/iQvO4/R97XAkuNRcMR4fnhC6zeeOE3rTmsA9e7bEbpIQ7gd+6Ph5bM1BxG1rDqrghgak9LhaJf9YVesi8vcAfFdELqvq94IHrAr6vQCwZcuWjIeZLWOjXldpalECH/xZ2M9OEvV2mzLXbV+P2ur1o2wFk5Uyefuta8e7fFr/mKw7ANrWHA4dP288nn1QSJlwEm5Vra/+/2ci8n8BfBzA90LHPAbgMQCYnJxM9lzeA3wBsvmyXkWwb8c49h45Fet9Ri3OBQU3bdQrAPbtGDf+bLNl0hFBm8/tks9tw/fV07RXDYp5Eh8cyH7x0DSx2P4NMLeclIlYq0RENonI+/0/A/g0gB/lPbAsCfq9NrwRwdPn6k7e5/SB7fBGOh3dyoi0pf+5RHFVr4K9v3Jrmz+sAJ4+Vzde+8HP7TT63KpoG6/L3o4mgr56N20AZufreOe96EZXJvKOfNkHhQwCLh733wfwfRG5AOCvAJxQ1W/nO6xscYl8l5orziI1NVHDwY/f1vF6+GbaoriKSFte88u/6LQVGs1l3P9E52Ln1EQNB3+989rh8Ublc5uoehV89eDuNl89bRsAf6IM++6b1leME16QvCNf9kEhg0CsVaKqPwFwRw/GkhvdRHG2956+vNDxWnNFY6sYq16lQyhsvqstTdF0bdN4/ag5rhNgsC93nKUExIurbaIcG12PfTvG8fjZq8b3CdCTyHcYNjRgyuNgMxAl73HE+b1Vr4IN60aMqXo2kXKJRl2LcmyLpYDZ942aiD40Vm370kYtNngVwdEv3Cx/j+uVArjZCrZ7XV9sRE46ivJuTVYkBnHrN9JOaYXbNaKwbSrgl2z70SbQ2ZAqSqRcm1LFRXez83W8/W60FxwWatu1/UXNOPH1ed+GVvm736slzlKqiDjZChVDTxj/9ahJp5aBTcJIk1Wjw0Aphds1orBFkGNVD4fvMm8Z5vqlz2rXbltOeJDwZGC6tgC4Z88WnL68ECva3ogAcnMzBf/+xb1vRdXpi29Le1xWtTavysImYaTZglu/DT6lFG7XiMIWQW7asM7aTdD1C26zQeZeeQP3P3EBy6qoiOCLn7gND0/tsp4n7stka1RluvbURA3bZk5YzyVoTQK2LoO2SNnHdeHQJs5jVS9y0ulWXBlptuDWb4NPKYXbNaLIO/IIC/0DsxfbFt6WVdf+bhPvKP+9IoLPf8zestb0us0v3zzqYf6PW42ebOK+rIqqVzFOdkmeJmw9Vd5ZtayCzauytDMYabbI6mmQFJdSCrdrRJF35BH2U1+/ZhaIbz7/qlW4o/pnL6vi6XN1TN5+a2z/bP/ntoDZf312vm7sSw7c9Pv9rBI/Ao/bDd40LlPjEr8lQF472jDSbOG6KE7KSymF2zWiyDPyMPmpNpZVrd37gl8y0zkazWXcF1hABNAWzdYXG5h+8sLaua5ZmlhdazTXxmwSbf++pE2V61hPsEwgeUa/jDRvMgwpj8NMKYXbNaLIM/JIWs4eXCQzRcxnZvZj28wJa/rezYU27bAgmiuKw89ewtRELTLqtI3ZNVskCtf7kWf0y0iTDAulFG7APaLIK/JIGjkGqxptmQ9x+eZRwujnoEdFnbZCH9dskShcy/vzjn4ZaZJhYKj6cWeJLXIc9ey31N+qLO1u8C5ElXTbxpxFFOxa3k9RJaR7Shtx9xtbZPufVjMmbHaFy27wUSXnI9LaOCFMsCeJLerM0wN2Le8nhHQPI+6UREW2UR3o4qJef3Pjrx7cbTzHv/zElo7ugF5FcOevfnBtwwbbpgR5Nlhi8yZCeodowi2tXJicnNS5ubnMz1smbCl7pmpOW2QadY7g6/t2jOPpc3VjUUtU8Q8hpDiIyDnXjdgp3H0g634atg0LBMCjB3cz6iWkBCQRbnrcfSDrzAebb67A0JV7EzIM0OMeAKKyQoat3JuQYYDCPQBMH9huqjAHMHzl3oQMAxTuAWBqooZ79mzpEO9hLfcmZNChcA8ID0/twqMHdzMdj5AhgIuTAwTLvQkZDhhxE0JIyaBwE0JIyaBVQkiB4GbHxAUKNyEFgZsdE1dolRBSEKJa/hIShBF3weGj8/DAzY6JK84Rt4hURGReRL6V54DITfxH5/piA4qbj86mlq2k/OS50QUZLJJYJV8G8GJeAyGd8NF5uIjq405IECfhFpEPA7gTwNfzHQ4Jwkfn4YKbURBXXD3urwL4QwDvtx0gIvcCuBcAtmzZ0v3ISOSO7WQwYfUrcSE24haRzwL4maqeizpOVR9T1UlVnRwfH89sgMMMH50JISZcIu69AO4Skd8GsBHA3xGRx1X1S/kOjQQ3D2ZWCSHEJ9HWZSLymwD+g6p+Nuo4bl1GCCHJSLJ1GQtwCCGkZCQqwFHVvwTwl7mMhBBCiBOMuAkhpGRQuAkhpGRQuAkhpGRQuAkhpGRQuAkhpGRQuAkhpGRQuAkhpGRQuAkhpGRQuAkhpGRw6zJCyEAxO1/HQ89dwptLTQDAWNXD4bt2DlRzNgo3IWRgmJ2vY/qpC2gu32yet9hoYvrJCwAwMOJN4SaElIqoDbSPnrzSJto+zRXF0ZNXKNyEENJr/A20/b1Y/Q205155A6cvLxh3jPIZpC3/KNyEkEwwRcJAthuB2DbQPnb2KuJ2FhikLf8o3IR0SdSje6/HUF9soCKCZVXUYsaS5bhNkfD0kxcAwZp14UfHQHqv2RY1x4m2NyIDteUfhZuQLrA9ugO9WwgLj2FZzUIZFOqxUQ9vv3sDzZV4UbUJ/AOzF/HN519du14Y/9xBGs3lrrxm2wbaUTCrhBDShu3RvZcLYaYxhMcCoE3c/VQ507HBcdsmpifnruLMj99INd5uvObpA9vbxgMAAnPEXRur4szM/tTXKjIswCGkC2wi1MuFsLhrvb7YiBT3qHPZJqa0og105zVPTdTwyN27UBurQtAS53v2bEHVq7QdV/UqA2WNhGHETUgX2B7de7kQFmcf3FL1nCeS8Li7mYC8EWnzuIF2QU3rsU9N1DqOm7z91r6vM/QSCjchXWB6dM8z2jOJnWkMQUTcveHwuNN4yrL6vqisEpMFc+j4eTw5dxUv/6KRiZgPMqKWhYVumJyc1Lm5uczPS0gR6VVWiakq0KsIjn7hDgDAfcfPG98nAB49uDtS3H1ePnJnxzVd3ucz6o2g0VzpuA/he/TOezew2Oj02cNUvQoeuXtX7qJchMwgETmnqpMuxzLiJqRLehXtPfTcpY6qwOay4o+e+Wts3rTB+j7f/tiwbsRZgH2CFYmvLzYwsppqaGJEgKXmCoD2LBUAHdG1K71Y6C1CZlBSKNyElARTJgjQEsslixhWvQq2/t0qDh0/H5vrPFb1jK8HJ6ZtMyes7w9n/wUzWpJOGEHyXugtQmZQUijchBQQWxViEmpjVezbMe5UVeiNCA7ftTP2nEk976T+uO2aeVKEzKCkMB2QkAhm5+vYe+QUts2cwN4jpzA7X+/JNb/yzEXUFxtQ3Hx0T/NlfdxBtGtjVRz9nTucosvpA9uNqXe2aF1cB2qhF2l9tomhyCXysf8WRGSjiPyViFwQkUsi8lAvBkZIv7EJaN7ibXt0X0lwDoFbtBssUnGZoEx51I/cvQuH79rZIei2wpgoqt5Ix7nztitsk1GR88Bjs0pERABsUtW3RcQD8H0AX1bVs7b3MKuEDAJ7j5wyil+WFXkmS8TFjw5S9SpOlYQ2xqoe3rl+oyPfOqlohj9LGptEALy0mtnikumRVTZI2bJKEqUDisgoWsL9b1T1edtxFG4yCGybOWEUwKC4dIMp1a7qVbDRG7EuRJr46sHdXQumjbhGVVHYJr64652Z2W+9N8HJxOWYMpFEuJ1sMxGpiMh5AD8D8F2TaIvIvSIyJyJzCwsLyUZMSAHJ2/u0WSKq6Hh0t1ERwdREDWdm9uOlI3fizMx+VKRbZ/km3dhDSa2GoD0Rlenh43LMoOIk3Kq6rKq7AXwYwMdF5KOGYx5T1UlVnRwfH896nITkhm0BMi/v07+eLRpdbDQ7fOS9v3Kr8dgvfuK2jtdsedZpiRND2/2bmqhh86h50TJMRQSf/1gNR09ewbaZE9Z7E8z0KGM2SFYkSgdU1UUROQ3gMwB+lM+QCOkdLsUXWXqfLpWIfhQdvk6wjWpFBF/8xG14eGpXx/trGdslQOu+zM7XjR5z1P178HM7Yz+vVxEc/PXb8PS5emy+d/Bpp5s+MUXwtLshVrhFZBxAc1W0qwB+C8B/zn1khPSAuOKLrKsiXbr02SLmh6d2GYU6TJoFThdM1YQu988/zjaZbFq/DqcvL8Tel/DTjmufmLBI79sx3jZJlKFSMoxLxP1BAP9TRCpoWStPqOq38h0WIb3B9lidJGJNEr25PMbXDBFjkh1upiZqmHvlDTx+9qrzZ3Ch0VzG/U9cwKHj59c+p4td4Qu4bbH3WqOJaxF9S4JNq4Kf1eWJyPREYCpIKnqlZJhY4VbVvwYw0YOxENJzbI/bAhitgTC2Lnf3HT+/Vrl4+vLCmrDcUvUimyvZIkaXHW6CPDy1C5O332ptPJWW4LWjzm2yK+KsjTSpl3FPRKYnAtuTSJm8cVZOkqFm+sB2Y3WfAk7ZCVHCUF9s4PGzV9sKeN65fqPVp9rCRq/zK+m6w02YqYmaMXoHWrnb2eWetBOefIKLseFr+sd2sxAcVd2aRIyLXCkZhr1KyMATZWVMTdSskaPLlz5plNZcVmwe9TC6fh1eX2zgllDxy5tLzY4o2mWHGxs2H/jwXTsx98obTn1MXKiIYEV1zUM+evIKDh0/3/H5FDcLhExWT5IFw9n5Oh567lJbznv4KSTqiSr4uYteKRmGwk0KSxYr/y5ZI7YsDJcILE3By+JSE/N//GkArSKVsHUS9lvjrmEaZ/De3VL1sNEbweJSs+0+Tk3U1naO6TYLZVkVAuCd927g+A9eXRNqky3ki3bYAkmyEByVnRO8f7aJ6/Mfq7VZWAOXVUJIP8iqR7JLy85udrGJ233GRFBoXRb3oq7h4okvNpqoehU8enC3cSFzaqKGrRHtWkeks2WrCYVZqE106yfHZef4588jpbMIULhJIcmqR7Jr1oN/zTRbZvnv9T3cKI0LC61LLnL4GuGsEqAVuQd3l8myv7SLaCelWz85TvjD96/sQh2Gwk0KSVZVca5FGt18uYPvNeUMRz2Su0b7tvGZnkxsRN07ESCHXQyNZOEnR9lHZfOr00DhJoUkq93Te72Zb9IJoNtHeZeCHp+oe3fPJ7Zknvft440I3rdxXYfH3g02+2is6uHwXTsHLsIOQ+EmhSQrwS2Dx9lNtO/6BCKIbvr08NQuvLTwNs78+A3n83kVwfXl6DA9LyEtw+81TyjcpJBk+cUcRI/TxzWrRRG/qHvsX38Ss/N13P/EBWPZfTDlb/rAdhx+9hKuxyxGvncjyfYPyRjk32scFG5SWIb5i+mKa1aLrRAnjH+/XfpcH3KoyixbKXlZYOUkISUiXCUIoK0F7FjVg1dpr09MajHZticLi6/rekOZSsnLQqIdcFzhDjiEZI/rji+9alnq0qIWyHart6KSxT3PbesyVyjchGRPL/bATEq4QtO2dyUwuAuJWW2hlkS46XETUhKKuONLeB3CFHkCyKQKtqhkVSyWBAo3ISUhq9z2PDEtKO89cqrnwtZL+jGhcnGSkJKQ1x6YeVPEJ4UsyXtTaRMUbkJKgp/tEdyAd8O64n+F+yFsvaQfE2rxf+uEkDbebd4sallstPp3BzcPyJuojQtMlPVJwRXX9MksocdNSInox0JYkDTtdoehPL3XxWIUbkIKRFw+cL/94rQTB6tgs4VWCSEFwY9mg3tUhm2QfvvF/Z44SAsKNyEFISqa9em3X9zviYO0oHATUhBcd+vp9UJYkH5PHKQFPW5CCkIvduvplmFYaCwDFG5CCkKvd+tJCxca+0+sVSIit4nIaRF5QUQuiciXezEwQoaNftsgpDy4RNw3ANyvqj8UkfcDOCci31XVF3IeGyFDB6NZ4kJsxK2qP1XVH67++ZcAXgTAf1mEENInEmWViMhWABMAns9jMIQQQuJxFm4ReR+ApwHcp6pvGX5+r4jMicjcwsJClmMkhBASwEm4RcRDS7SPqeozpmNU9TFVnVTVyfHx8SzHSAghJIBLVokA+AaAF1X1T/IfEiGEkChcIu69AH4PwH4ROb/632/nPC5CCCEWYtMBVfX7AKQHYyGEEOIAKydJ34lrZUoIaYfCTfpKmsb8hAw77A5I+opLK1NCSDsUbtJX2JifkOTQKiF9xbWVaRj64mSYYcRN+kqaxvwuW3wRMsgw4ia5ExUdp2nM3++dzgnpNxRukisuWSNJW5nSFyfDDq0Skit5ZI1ww1oy7FC4Sa7kER1zw1oy7NAqIbmSNmskCpsvDgB7j5zqeI3ZJ2TQEFXN/KSTk5M6NzeX+XlJ9qRNq3N9X9jjBlrRcdZ7KZqu440IIEBz+ea/8TyuTUgWiMg5VZ10OZZWyRCTNq0uyftMG+B+/mM1HD15BdtmTmDvkVOZpPGZvPTmiraJNsCqTDIYMOIeYvYeOWW0MWpjVZyZ2Z/5+wBzZCwAdPX9aa2MbTMn4PovWQC8dOTOxNcgJE+SRNz0uAtEr6sB0y4cJnlf8DPdUvXw1rtNrIQU1v9rNw2mbF667VhCygytkoLQTTXg7Hwde4+csloPtp+nTatzfV/4My02OkU7TForw5Rp4o0IvEp7K3lmn5BBgMJdENLmO8cJftTP06bVub7P9JlcqC828MDsRevPTRORyUs/+PHb8L4NNx8qx6oeFybJQECrpCCktS3iyr+jfu770Q89dwlvLjUBABvWxc/lrmXq3eRqP372KgDg4aldba/HVWL6YzB56e/dWOm4DptVkTJC4S4IafOd4wTfZUJ4t3lT0BYbTXzlmYuYe+UNnL68YBU0lzL1JL6ziW8+/2qHcNsmovufuLA2rqjjgv1MuIkDKSu0SgqCyX4QtMQkKmXulqoX+XqcH20TuGNnr3bdfc/0mZKwbMh4sk1Ey6ptY3SZsLiJAykrjLhTkMfjdTBSrC821lLkgOhIUCzbOL/1bhPbZk5gbNSDNyJorrQXofh+tE3gwpKZpvuef+zhZy9hsdF0fl+QbTMn2u5xVBQfHKPtuBGRtXPazuNPlrRPSFFhxJ2QPHtBT03UcGZmP2pjVatwhllcMgviirbE982lZptohxfokqTGpfGspyZq2LQhfXzg3+P7jp/H7oe+g307xiOjeH+M+3aMG3++rLp2Tsuct/akw17fpKhQuBOS1+O1nymxdeaENRI0CWfSnOTwAt30ge0dKXM2/GvFpR+Gyard6mKjicfPXo3MVPHHePryQuz5FOgQ7+CTjk+juYzDz15KNFZC8oTCnZAsut2Fhe+B2YtrUXwUJpFO6iMbJ5mQUo0A1vznNE8cSSYXP50vDV5FYi2gMH7Fpp9CaEszX2w0GXWTwkDhTki3vaBNwncsJooE7PnV4fzlis30DhBeoGuGqmJWAGxav65N0Hx7Jc0Th822MFFf9ZXTsGn9usQWkF+m/9KRO9dsKhtctCRFIdZ8FJE/A/BZAD9T1Y/mP6RiM31gu7HbnWs1nkn4XHpsBH1p0+Kon5M9O1/HoePnI88ZFDVbZHqt0cT5Bz/d8brLE8cDsxfxv5+/GlslaWPp+o2OBVUXrgUWQE2/pzCm39v0ge247/h54/H1xcZasQ8h/cRl1eh/APhvAP5XvkMpB2n2SAySxu+tjVWdc4+nJmqYe+UNHDt71SjeYbGyZVfcUvXWMivGRj2otoRxRMSYpjcigtn5OuZeeWOteCYtby414VUEo94IlpqdRTM2ghOS7T7ENbSamqi1FSSFYZ43KQKxVomqfg/AGz0YS2nwsz/8x+skX2LbI7zN4AgLrYtV8fDULjx6cDc2j7bneG8e7Sz5tvX4+OV7N9bsnDeXmlhsNKEw51YDN/Oojz3fnWj7NJcVmzdtwJf2bHHyvE3R8+nLCx2Tly/aUb+3Bz+307puwDxvUgTocfcYW5+Pe/ZsWfNXfZ866C37JFkcfTcUrYb/Dpj7Za9fN4LlFD5Ho7mMLLsEv77YWJuEwvdmrOph86jX4cGH3287bxT+PYkaFyH9JLMCHBG5F8C9ALBly5asTjtwdGu1uJbGu5R8B8cUfG3rzAmnseSN/5mS7gIffH/SNgLB9YOKxRZiW1jSbzITblV9DMBjQGsjhazOO4hDlYcWAAAJJ0lEQVSkFSLAfXE0abQZFCwXbKK2viK4vmz+9Y9VPecKyvBnSlOtGnWvTOcD0Ha86fOxLSwpAix5LxmuEXuSaNPUSS+OPf9gM3549VqHKG5YN4LrBnEeq3o4fNdO43U2j3q481c/aG1qlbYZlP+zcPfDuVfewNPn6h3n2+iNGO9BRQQrqix/J4XBJR3wmwB+E8AHROQ1AA+q6jfyHhix4xKxm6JNgTmnOk3f7Jd/0cAjd+9qm0D27Ri3ZpRcazRT20RJbB8Tb797Y+3PfvVlmEZz2XoPVlS51RkpFLHCrapf7MVASLZMTdTw5NxVnPnxzYQgBfD0uTomb7/VacEzitcXG8b+1za68au7tX269e3oaZOiwaySDEnawyPvsfy/H3dmcZrS2aJSFMMphbb3PPTcJWvE2q0vnKRaNVyZmkS0x6peqh2BCOk1FO6MyLNrYBqOnrxiFa1wpDp9YLsxV1oBqCJWzGbn69aCFQDGVL0kk9y+HeMd47MJatrt0qpeBYfv2tmRGsmtzkgR4eJkRnTrw8aRNKsiyv4IR6pTEzVrmfe1RhOPHtwdee2ogpRw1WfSfuOz83U8fa7eUf34+Y+ZLRdX22fT+grGRtcbPxOFmhQdCndGZNE10EaarApbVokAxki1FpGFEudLR31G/1qz83VMP3lhrf+I60YNtt4utratLtuleRXBf/znjKRJeaFVkhHddg2MIk1HPttWaPfsaRVHhW2KtDu+A/bPOFb12nbBiWsaZZoAkk6IthL+YJXl0S/cQdEmpYYRd0Z02zUwijTRvC31DoAxen/k7l0d6X2uOcu2z/7ZOz641qjKZZHQNAEkrX5MmnKYxzZ03Dme5A2FOyO6LWWPIu0O8CaLY++RU9boPWnDrOB1AHTkdAeLXOKwTXJpJkTXlMM8dnnnzvGkF1C4M6SbUvYosozm8/Liw5/dNEHYsLVY9c8L5DMhJllQdo2i816kJgSgcJeCLMUrbfSeFJeJwKuIk9+c9YQYzG4xER57kig6z0VqQnwo3CUhK/HKInp3iT5tE0S/+3649GXpptNiryZGMtxQuIeMbqN31+jTNkFkXdCSdCEwrkDH1M8lSRSd5yI1IT4U7gHDRci6id5do88s7J24z5JmITDOsjD1c0kSRefpyRPiI5rlliWrTE5O6tzcXObnJdGYbACvIti0fh2uNZqZiMi2mRPG1D4BMu2gZ/os4Yh975FTRkH1tyYzYXtP1DlcxkJIt4jIOVWddDmWBTgDhCkabi7r2n6RWfRPybPQKIhL0VGahUBTgU7cOUzbu/VStIvUvIwUA1olA4RL5kI3qWmz83UsXb/R8XoeHq6LKKdZCAxbGbZd6039XPoRXTMvnJhgxD1AuEa9aVLTfAEJdwEcq3buHJ/0vKZo0iWyT1umPzVRw5mZ/XjpyJ34r//ijkK3ck3T7oAMPhTuAcLVBkhja9iyMTZtWNd1laGpFa6LKGdhYfTbBomDeeHEBK2SASJsA4yNenj73RttzZ3yqLhM25sjKpr0FwbzzJDJ8hx5wbxwYoLCnYIiNxEKi1BWY7UJyNiol9qDjYsmiyyovYJ54cQEhTshZVssyrviUhWpe3MUJZos+kQMMC+ctEPhTsiwNhGyCcghy845Lh5sEaLJMkzEfPIgYSjcCRnmxSKTgNiaNblEzUWIJod1IiblhsKdkKI83heFbqPmfkeTwzwRk/LCdMCEdLPF1yBS9HS6OHpVCUpIljDiTkgRHu+LRr+j5m4ogs9OSFKchFtEPgPgawAqAL6uqkdyHVXBKbNQkXY4EZMyEivcIlIB8N8B/BaA1wD8QESeVdUX8h5caBwAgGA3Q9NrhCSFEzEpGy4e98cB/I2q/kRVrwP4PwD+Wb7DsuOLtf9/QggZNlyEuwbg1cDfX1t9raeYIu3w64QQMgxkllUiIveKyJyIzC0sLGR12jbCIk3RJoQMIy7CXQdwW+DvH159rQ1VfUxVJ1V1cnx8PPzjTAjbI7RLCCHDiItw/wDAPxSRbSKyHsDvAng232F1YrNHKN6EkGEjNqtEVW+IyL8DcBKtdMA/U9VLuY/MPp61/1O0CSHDiFMet6r+BYC/yHkscWNweo0QQgYdlrwTQkjJoHATQkjJoHATQkjJoHATQkjJoHATQkjJkDwyM0RkAcArmZ/4Jh8A8PMcz192eH+i4f2Jhvcnmrzuz+2q6lS9mItw542IzKnqZL/HUVR4f6Lh/YmG9yeaItwfWiWEEFIyKNyEEFIyyircj/V7AAWH9yca3p9oeH+i6fv9KaXHTQghw0xZI25CCBlaSiXcIvIZEbkiIn8jIjP9Hk+REJHbROS0iLwgIpdE5Mv9HlMREZGKiMyLyLf6PZaiISJjIvKUiFwWkRdF5JP9HlOREJFDq9+tH4nIN0VkY7/GUhrhDmxa/E8BfATAF0XkI/0dVaG4AeB+Vf0IgD0A/i3vj5EvA3ix34MoKF8D8G1V3QHgDvA+rSEiNQD/HsCkqn4UrRbXv9uv8ZRGuFGwTYuLhqr+VFV/uPrnX6L1pePW5QFE5MMA7gTw9X6PpWiIyC0AfgPANwBAVa+r6mJ/R1U41gGoisg6AKMAXu/XQMok3IXYtLgMiMhWABMAnu/vSArHVwH8IYCVfg+kgGwDsADgz1etpK+LyKZ+D6ooqGodwH8BcBXATwFcU9Xv9Gs8ZRJu4oCIvA/A0wDuU9W3+j2eoiAinwXwM1U91++xFJR1AH4NwJ+q6gSAdwBwHWkVEdmM1hP+NgAfArBJRL7Ur/GUSbidNi0eZkTEQ0u0j6nqM/0eT8HYC+AuEXkZLZttv4g83t8hFYrXALymqv5T2lNoCTlp8U8AvKSqC6raBPAMgH/Ur8GUSbgLsWlxUZHWBpzfAPCiqv5Jv8dTNFT1K6r6YVXdita/nVOq2reIqWio6t8CeFVEtq++9CkAL/RxSEXjKoA9IjK6+l37FPq4eOu052QRKNqmxQVkL4DfA3BRRM6vvvZHq/uFEuLCHwA4thoY/QTA7/d5PIVBVZ8XkacA/BCtDK559LGCkpWThBBSMspklRBCCAGFmxBCSgeFmxBCSgaFmxBCSgaFmxBCSgaFmxBCSgaFmxBCSgaFmxBCSsb/Bz3eehYFJbY9AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "centroids, idx = find_k_means(X, K, plot=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Image compression with _K_-means" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIAAAACACAIAAABMXPacAACBXUlEQVR4nDT9V7NuSZKeibmIiKU+vfXeR4uUpTJLZGt0YQA0QIDoATmYITE2N+QF/wEv+S9oRqPRhqTxAmZjwwFBAE1wCKCn0aKqu7qqurKyUmceLfY+W3xqiVDuvDiFtf5A2LIV7uEerz8vfvGv/3eNHScMWSMF4XIEkoA4J+/blktniyLFjggKgzlICNk1DZusfQqAwSCrY1VISYlVA4QB0ZliJ1tnrZHsSVNtJQr72BqrGk2XWjv0jm0CY1H8EFUhgdCQNnmgAiSVKStjjCJJLVFyKWpMymBqa6laD1tFRFNk04Q+xngpmSOPnROrqmYubBRaV9VV3TAScKF9q+AIBEUBkgxrUxfQt5K7eucW5aurJ8/mByej2bQazcvJ7HK7+vGPPv7pL66+ePzs8cu+bSmLDqqaY/S9IIQgGQUzigoqKiIggGQFQgDFrAoEJKqqaogAIQuACiEBgoKqgmnGs6BIUIokVxuSLsdBklGqxYXsO4OFLeYhd4GlnrhwsVWaJmKegKNQAfQXpzGxBSKrkros0VJB+Sr2voOCXENlA4lTCkKNFRkS+Ezsxh43OYTB2Ga0GLZrv1wPuW3J1YnadhtzZqPVyJHFEEzbd6RDY9jydLUJQy7URIMGHXaFq9xBt92ksM46yQwMa9UyawJSUxSqEbIOQQwENpRSxJyShibaCBFFqe3r0Uhc74d+FEeSe9HRpJp+962jpjTOyYuLx5l8VgLNOQcAIykSQxZEIAIRECZAQWQTcgYEAswIqsAECKQK+PolkqyGUFQE1IgYAzKEgJB8RGOnhCOAZRquGEm47MJgkdgWCCbSTAvEIkU1KMpqhaGaH5T9VepDzlCyU6eDHyLMoCggtQbWRcqbVG8TGBZTc5agnoK2peMcy9R1a419Nl1MJiZi7FSHFCALZN9J4ZwRNh2Os3dYNatXLeqAReWT8QlM3tQOk9kTVxh9lf1mqCalFpI0AKbBJIbaYFHagDbF5Mh5zaTG6CQOwXAzpBDbM1scjubXu21b50RxgHZbl+V4Nv7mG3S6bO3Pcu1429N8bFPm1ZUHdDln1aSghk0CyaqACpoRlRARmAAVhVQzgACQEoGoCBEKgAoCgel6IVFlyBlMKcIYfRIFdCUkgdyCGr/tTTk1BabUZlbgqkQXU5bgQ78RNSVOoc6WOUQfQ6fYQ/tKwEjRkHFAZPN5nUMz3jMKlCPgBlGQGq9D48rtZgPhKnQpihfErVc0Nslg2dgheX/BrmAGoerV1dahjyGnOFhHbUyjobYmZbsyhmOiBJ419VJmDcIGJQ8JW80NZbWHms+2MBbRhsRr6wKWbIbsTSpiADQlw3YYEjuD+TLAtJxMXVW9eae9cfLok6+3Nsvx9cNuWPXr85hCBhFQRIyaCNggJ4nMCICimFQIAJWiALACAIAAZARUABAhIiA1PmydKwwxokXjJNsEgy3nqQ+CGetC+rWqeL/OWLFCUVQpFMqqmAWbXLCENuacgjAjqDHgyBCSxmEjYYtmIcxgXSFDevVsK5hSEdE7ZwIJ8LRP7Tb6uozTrJtOEVpr7MsNeHW745h44KRDF5lKpW0/2Bayk9jntghlOaoM+m2XCEKuWKgYgjGYkzMxZB4dRp80bZvJpF/1OpkkkaC7pTNbHVKMDadCdUhkTQ5DV5S1tSOLOfncwVYMTqf71hVv3Dz4g9+9fnb1OUE5H4Nj+4BViSGJY0qiBIhAogmRQIGQiJUyZlVENYxZQUCyCgA55igJEQEBFU1lrTqTM4lRzlVMgQz5gCkmBCWYKFEMK5EW2ESNUE4ZETIHSCJBgUHHaIOj3LebzBVoabQvyEiRChk0rjIZH9lvImtKiEPsi8py0WSPPp0VmtostZlks3IVh2QldLNqE8z+ALX12eScBVtZEYNRaH2ptXepGHzIgj37pGC0LoIEQ4iub0MHjGZkw7DOWFF2yi1hOUTJnHEb65FPhoxkxSBR1EqEdbcZi1rEBA7S4CClMKTQFs3MjXZ/8J23/vyjF5980t+/ufjiSzVgGTIQgYBoJoScM6AiKChkFRBCRFJVAEVFRBJmUAVKIqoAgAoZBUwAqIKLBAZMHFTBC5ihGzCDgseQgUrBSlFCCEDGqGJOGQsQm2TQRJggY90O6+xTSf1m2yL2BmtXZWAUSZQGRwROlFxMXJjeisF+iL13o+n6bIXql1vWUDVVg2HDRCRS58tepz6oT0MWQlKJiQjRDiFjhqhitj0QDwxjbygH1mHrTckwQOwyj1NM06PbjK5LUtoibbzY8bB96fMiFvs7RpertoPEYg1EH8FBgJqMJMcOAJJPKfS2nJAp9/YOf/Dm7vry+cFi/rB5RTXXXKcc2s4TokECVQBWIs0ZUEFRUYAAVRFRFQQlKSBkEGAARMiAxGRQo4FBk4KI9JKAIvZA/vUGAdkmCGQKTVXSwCCCFBEyREJgZJ9CyDmt2XdryYNoyhiayWTwrh8GLBjU55RLayZ1HdW04EpqOPZ9VGK8PG8vfTYyLtA0NnuvXQbi0rkKVO22bYfUZkY2LAZRTJSQYhQ2rkI1aCn4Aill4GVil1vvnC2KzmtpoWl2UzdZRt+u1mG19UOY7Y8Ic0VGNlcyweWA44Ytjkq8Ap8TAAlY8IZrL1ACZJ+y70w1J2u/dfvIQlmOzaIa7026ZdsZ4ZyhH3pQQARD5EFQCUGVQBEgMyEoqIgighIQgoiqQlYl5qRidiYVaFQASlmsciDKJsNYMGNmoQioAkYJVAKyg+wUyqyRJKNEGoLfnmeREFoyGM1O5+Ypu0FSNtnVroDkhw1SV5kixqHCjUa7DeDR5i6gXi7K+lUbJfiSOWdxOAQkgWIYNJJ3FedAHnQzECPVBpJgHOKQfM2IhVHkMPgMV0us6qomGhFzOTtQny9W7uXjz188ezh0V9YVZbE493GxqGcG+v5Vl8ewHcazw2j6IRtAzyaTD7kofcwKaohCiIUEVTFu9+b9d2z1+PFL3/V47XAqp7GPdd8mQA8CAKCCCgCgSEqKmkABFBQUiFQBCFFfP4CGWRUMkFGtAiCoy0iomah1OcSkPcDrSMZKgJrIKDainFQKABCMMaAildxkG0VGxTT27XZ1MdR2nSpGzUqkaToy0cwmozoT9r1KyiwtMpKPQ447k0nq264kNOz7duWZxKJGlDZKTlk02woyit8GI0iilI1Rk3zyJAieQl3nEAplyzDYsUAxoyq29cMvvnj+4vP11XIy4lE1queNoWIEpFd+vXlpZlVfjuL5i4U0FnQwCzO8iJIyqkZKkkAhak4p5KyUPLumWVyfdZtnF0/QmvG0rLeQt8rEjJhRVTWqICiSIhAAIhkGVMgCIgIIAIoAiAQoQIQqkkVMwnGKKiFH9AgkapQEsXNAgS1BAWAEEAEomRg9BO/KRoQyVq4sSbOBAiUaQDQbHB5U26/HxUFPi9OUnbo64mUbJCGBthuskQouxzwkSHtTo8pR0sIVsShfnq9j2gjXQ8oTzpB9DiknToLAUJWhj9wLoRiAMaBPOSc/+OkuGKawoVxlqqRe9F3/9OnTJ4/PFPLxye7eYr9pxkVTOrajUclAIUnbbs0VDIdvXCmMpaHx3MWVj9sC6z6ZQiWlYIxVMClFGFbEVcZqvFhMm5eTIm2UQ48AqKyikEkYRYRBAIBVgUCFfl0DAyIiAgABiCoAEmHOoiqKavrQkYDIoAxgjGYCKjIVlIXBINVEVl6nkjjE5GPyXRcQDRsyLJQFDbYJu0xCLpV7yu2iantEKEYeFusu+GH7fJs2Awff35plBElOBbQIOUKfJBcstaojIEXm3PciyBhABgySgxIqlyUmMCjZZDPglklFOGnIgYuqbiVuttYVxOPpJ589ituNK+z1gzts3GQ23dvdN66om2npjCsMGg7d6sXTR9vLEI9noTwo0GSuOOY+A6aeTZMw90EnYlIyBKI5SEjGNQc7i7durf74F1dBnBIppCxKRASYAF4XXwKqIKA5qyoIKzKwooKKAjC+jlSIighgEAIbZ0xtSXLscwKAjCA5M6MBA2yJEACxtGVpbYhBEZRQDMXkGANwacEm8BHU1BPI4wFTznEmV5t+CFQwhunYBjTrLUbFs2XPO3YCKcR1YCTlnLDrO1eoc2Xvk1XfbxA0J5UMkjMpAIpzBXLGGLabDAW5nHNkqwG7cixO8vEumeblV0+eP3t+srM3nc4m871RMZrt741nI1JiV02aypbOukLni6Io+09/Dssge+X69BlbHVsWZtAgmmwxCr6NMcQQiEBIQSEN7e7B3nTy1Ts3qvMNrS6Y0ZTkkCTkiIQIgoigAogEr5/XZ9FfV74AiICqYpASEYAYxxEgIwIRAgE4JUGSZM2g6EE7zEbQIhoRw2hNyaCYKIq6wcegQXw2mgyDLUzmpJHWQ7luoYG+wos6u2Us+ghzN+iCqlE9E5w2rEMOA3spJOUS2rZtmW1ml3NfWw45JolMhpOSSYNaTBGJBmAvgGqHXhVbHychV5Nq7qqZhPLy6fLs668Pp9P9/RvTxa51OJkvRrM9W1jIqS4qW46aUVVXY+Y0ny+wpI8//qnCwVXIJdWjIiMQkMmSLDkiGYahbEgSg5BAkuTdePf2zfnq/NXvfGv+7/99S2CJAqiAAiMC8q9jjpKCAP467gsqgqoiIooCIAIgoCCAyWizKGRSZFJGTqoqmNiKEQWJGnsSMaDJWmtHGYwHS0pGgpWeA7Rh2EQNQScV1IacS1FoSFakTqA1hHJ7sYw1j6ZzzobHRVEnhgSsXDodrgZvSpEAWPTrrdecUUFEuhCtK8kIA0HU6FWzDGJFCInP4zApShky7dl+lT958Hh9udEcptPqzs230RTVeDwZjW01DjEyG+OabZ+T1WZartq0d33XNe7oYDa9c+PLp1+H1A0XLwQZQBIZoyyKWLgMEZKqGo2CBhkdUn3nxs0vfvX0d+8d/09/8bOcvKAoZDakygKQMYooE6gCKhIioCQBJGYBUWHkrCKkKCACpmBO7CFn0QQCEiGhKFrEIlNG6DmlrCmoBZ1EqCRvOG2IbNacIxIWNadJodsIOcM6Sa22bnDWgGjRR7bqBVK4aJcbk42ZjYb15tLp7stn3eFOYTUWTpU0EOFAXehichC8UK+AKXaudEYLJmgHwKyCKUOICdDYrVBT0tqbF1evfPLt6nR3Z+fk4M5kduRKw3YC2OSYbWHbIcaIdnRt2Y4uHuDu8fTi2ShWSMVBLHl2GFW2r7rzPhTWgjBIAp+pdIAW+5wNSU6R2CBxDj2a8uaNvb1mfPd289nXrxAIkBQkaQJEC5gABUAVAUAV5PXfL6oIqKQqTJQBlNQSmDi8gERYVsyYBYnQQMxCAmVgctwAgEpAvUJ4pbqTtQEVE4foCnIFx60aYUADZAgKBGcS2saxkAyFdZpLWoyG9ExW7Srp+nK4NeYXwwViX+RaMJdFSaRuFLvVwJgSmqXYmtCagZRysMiOGDwOMSYyicENwSdaQFFtAft2ZaCIVzLb2zk6ubc4vGkIDVZpCC3m8ezQNve6fvzkxbm+SCcn5WKxePSr83N8UBbXw7wpJ/L2/rwqRs65vudKgYA5D0599IDkxmUpOcUYStcQ++3yuavo+NqObEd/9/2DP/3TR8ukiiAeCFBAEJGIIIMAAKGCgioqogIgIBNoBgQUNcgZxERzgJysrTCtKayzoFJBGgHOR+gSlGKLDIby3IQtyrrgIWefUrnuJQqgqS1mxmghZXRZKOZsgogzzhFkCNmbopyMGk6d6VKI7dDSjE05sraKVhRlGsMGEtRsEAtrNBvOYKJWRkhBsnSaiwmXvUDWwIbUBJBtHB0Zdo0snn3+rDTrw4N39w9uUSJoxkvAerw4OvjWctP87MGlz8/eubdjsXp5dv7wF382mtfdoLFq8+Vi+v397aicmUPI94fTlxJfpGIn6qOsscvlgSmiXwubMqYUfVEZlSG2VNForfLdOzf+8IdP/2//5jkosiXOGsBkVdBMTAikKCqghCiYVSgjMQiyoBKSqiKwUSijBOgTsCUzx7Sl1BkuxVqI4JyATZGdetNvLxXFllNlzKkdG15j2Q6o2UdkMXVhnDCtg6fNclZbaGzIxGgry8bZtmiMDJz1au2LKlqymowjfvj469HC1clciWQmypGR206KBjoZfM+BKJPTLBECc8lkkjoop9XkjrZ69fnXU/WLk/s708N6tGtHu5frKsqsNqOHz93njz66fjjdxcWzz59VE4WAs50jcFdVbz7/8uV33rsuz9Tt31iMZ1h0S3qkkCykSTVCGQyxxhgyjqZ1JkLIMcX+qkMoxzsjMJum2vtnf/s7/+6nL85espeUAVSzAiKRKsnrLCsImAkAyQgIgIAAIwoIvM74mFbMlZiCzATBYzA4bIOyukOoBEMnYW3IG0SsIXSy2kbLrDoquZ+qzCZlq8XQ5qxuq5zamIaVCUPOwWfeJlCmee1sY2dxvEXbd4M6Dtp3g+tjEJDNtmtmNACK5C5ATIkLQLFDQGFEE8UPXhNQgWoj9CroZSLlKA/9q69eRLmY2Z1mcpzHCyg/+OKr9Gr54vrd0YMH8PLhX+8c8notUPT9Nnz54KEb0erpq/nO6M433v/W8e6nv/xxgqvPH5RvXKsn9lUKw829OzuzCZ1t2mW0JEVT2mJSllNkJrbDuhVjZ4dHlaty352edvN5/Q9//8b/6Z8/yqAODFDOkFNGC6JZgBANsLyORAIq+XVHAoEAVVQ1mwgjkJ5zF7ZaNDNy04gj8Bv1V2KsUGFgJ4RtZVHdqCDdB99nHmAaYp7Y8kqNusms8Odbl9bLyytfFZNe+jh092tOarJgDAySK0sxVi2L5DYOHnMoJovTs2eRAXLthy1AgTqgb2IWpMGrL9LIgEuIPsQOejRoUnEZtml0LSW3/egr7daLg5NRs5/cOw8e0suf/MXecXN0cPfBLz+N9GRytDtdvK2d+8XPfny6esRFEeLVvevX18vm//cv/98NhmI6gTF1Kz+pnSnk9rs/mDot/EUq5+OZra1Ho1TXVBjvM9qEGHduvmHZadrElGPe2Nns99+59q8XT68ueKtZVV+ffAQzI6qogiKiIoAygBCIAsckhhQAE6jJFlBHDFpP1v32hbYjHC+oqXRoc9zmtMl2bLCMooFKw1lJ67K3SexopGLNQD0oZdxz7VPKntD3MilHTcUiS8gCsdp2Mp5XoBRj5/0wGhVpWz29vOrWT7bJ7VRlH0IXMlk2zGtZaSiiZvJFoNyJ9sJqqxLzABLlMufC29HyeQftuqono4MjW3zw0Y8fmRm9ceNeF4rHXz2a19rFxealPH344xCe7SzKt9+4vre3qEeN19Hq5YPRuqzqvRvHb1bT47Q/M/bB9Wtpd/da2V+kp6eWuW5KtnUmrauK2FJV2mavnB6Hy1OeNQmrTdujFT8099/61t/64NEnv7z42ZNBAFUEVBQ0qyCyCmYVICIAJptzQhVDSAgAVCKaJLVhl0hyLsDMaXgVrp7i+MhVB5yzpE1KLSiAMGAV1aLmNcyMMZw6pjyqOITs2+BodTSrjYXn4UYF5428GgJJHLowsLFDG9CgNaDqCzY+0WTSrE5fJh58tKvY1bVkHQo3wXIlOYfWqkrvN8IO7URtwSZVopvAHqld9enyYjw7Orn+VpI3f/nnD2e7WOzd62g4PuEoe+MS3bixda0ZL571xYj29yaxxlT4Uem//Zv/WYh8sdx88tnnL3/2Vzs3d+7d8HryDaZFwFhUTQldYygXCysD2cKnWNcNGztsnhAkUNTM/WpYHEwuLx4eHhx/8P23w/lHn12e9oMEspAENBuiLKigzIQAoqKqCApASCBAIIgqJgsho++0tmyLBkZVo4p+SN0qYwFSsjBjSyoR0NYWcoEKq6EZ/LqZTtXq1MBpZ1eBS/TzbinWtH2IsrVGcgSXFTBj7OpRc7ENDPVwtTFyLsFo3dB2vaZUjazNo6RpVODINZc9aI69JqimdbXjBaQbWl8Mkr0AYQ3txu7UJzd+6+XT6qsH/9Eu8PrhH9x482B/sYvoqsVO3dhyQtCALaGPPTu9tSgbY64GXA+xO30xxYs7d+qTN7//pz+TsbM33v3mjWtHM1dwIe0VjlylriTLsTNDL1VdWy60X7btMJrOhOH0yWddjPWw1X4tOn775uz/w/LuW5O/+Mkr0IwIwBkUACFjft2iVkBQ/HVjTgERBFVVTQaKHovSUDHqvc+eiYSoZPJWlhHKTFUmNhixXw2pMuVO7WTrAZsjLohyti6f7OL5ugzqx3tUPfw4wbwZMwsmtAKpIFHUYTsUlOuGe6F+BSaEUgucYt7KdluJo4OxqZpi1e3H/jQE7wF8zpvuiiwl5RxCFI7laLnupns32+K3Pvqs680nP/gHHxwffTAZHW2jkLXjvXGxY7TQ5kgLzstXPQ7Sr/0vv3yyqEwJcT4pk7/68M//VXJNWly/cXB45/7dN3Z3K3BWCNEVpurytjaVxB6YIxalcaHvkHg0P4IUIboQe9G46aBG9tuuUr19/fh6Dz/GMwBARVT7uiPEaAQUELOKABASASiIEpkMmdlA5qBDGlzyLdmCxPi0yYZtchxaUwysW/Gc7Jh4TPmKcgU02hnpeiubIBYLjly5YTQiySl148XOrFg/HXzNZLbdurCIzClLHIKrzKgopV8Ll+quso+tz8ZGv9rw3uxwd3fTYg7LlKjPGcBpCNEGCSOgUqUL2mQ95J1rl/7+cPbVG9+4dXj0v2lGM02gNR3cWvBUD/YiMtQcHj9fn37+KFydjhY7prLt+vTlz39uz59Yzhe0W+3tdzUdHE9/67vf7a/WT3/5KVXNzvE1yLHEpnR+iGuHBsg1rmJIZBvjGvEbVOzigGZ3NIvqBzue9stsS/3u23c++fJiPqqXbSLQLAoIqiiMAK+TMTtQVVCAmIUJkyIqmG7YAFojcahL7AZDAxujihEcUxm7jgwzRvBnAaykoZTHCvvqpuMyeXWqNKiTlHs/lDSM60RpkvtTZumhTsA1qrOce8kxR9OHpZ6+GsZFMhaLvF23hSAUuJzi3npNm/Vp9H1CyMSSWFWxZ6yYXU5aRjo0+3+XVrNb+8vb779TlvdXIXPdTA53uhkW16OT8PTz84MGzx796tXp2dXyPKdh+8mLcP5VCN4bm9SpM9OT2WxvMcJ0fXrtw7/5KK4vynGaX/9mdsnyqKB3w7OPOHmFwNWInQGApBWiETOWiF89eLZ8fNqU2xQ2bPdSaGeuPphU672dw6PR6qtzUgIEIUYVyEJIAJoRkr5u0Ymi5pxILFsx6pcJGuXSRolh8BLKwpmSCS2YRYAiZzGSbXiJskQgYWb/CggFd1k6A680NW0eQLpl6g/HVu1QjOvYD6qJ2VUukSTDAiA22ucXy5DTkJFyAxydy8MAGfKrFw9qs5dilzw5jnXVhCCFrZWtkumCer1bjX8zhubNN/fnzfHF2bbvmno2rY+a6TcTrIYXP31ohnVF28+++sn6+S/S+syHDcpWVUxDewdTMzuEat5eEPptevYYjw+uXnx6eDS//t4PZuX+aYiFYckwDOt2uS6QSjepYvQCkt0wdJBl2/ZtnD1ZjodNulsQZlq3/aS0KaXK+ZOd4q3D+tGzwvugQqgAgGRQFVRe34cxohCgZYuQFTCImumk6FvfZZ8HRsSMOsRtCYWICaqha2O7hCEa05ZVLquJJ0KqWDeUYQ1zTP2IXmnGaKo89J3vi0hkHHOyyY8qsoVjYqup5p4ZA1oA7MK6Ig6ZyAQyHH1TNUXGFFJx0Q3WUEo+AUgwgtXSx7MwHS++eTA+nIxnFt35mdr5/b3dud2r9t+C9iotnzyGJ3999dlff/3sz2J6NG2ak+NrB9eOZ/Ny1BTlqGkmh9PF4Wz/jXJ89PjZ848/+fpyCzf3b+8c3CKYZKWZtVn0svXnL17p8sqWNDVuvQzJjNGMUkqbyyfLC/VhUAqbjlatjqEc+rTblAiJGesylTVPRkij6uxsyAjwutxVjACAzCqIkoReq0IBs0E2MZliVOXBr/ueBZA6C9q1g48K7VW3eqSlbXaPp7OTpnIqqYtJUzaUSmor5B6alZc0tDuLbrqofQhiMovDIkDsXWFMaSALKzhDgwIwGKMs3HkPSWvD0SUX1UtOMn528QKQuzYOoBGKIYR1f7mhnWL+raP9G6NxhYbarTfN3B0cjG6prXI8DyZs0sOfX/7ij0z64vq1vHPw1vWjnf3d3boujTWWhQiJVrbt+peneXnjeHL/1g9/r0vrjz959OTjs+Lkm+PD/aIuNl6/2m43/bDjjsqm7Pu4oj2lW2Mw+/tmLbuxfUZMm4uHaM02YMEIcegjVXURh8u+dbujxfFifbQ/+6OzzwgcIAEkQCJmRBTNCoAAjDnB6/tjMF4c5qKsLeCq71AQLq7a0xfb7bZv6jjZX9Tjw3oyL+rCMGZZaqjbFI2svSsrF5yJMRYtWE3Gum4AyuAYjEBh2KMIRFWDCmiZ152AZMweALYDTRxUtrhYD4yyavXSJiSRTG3WYUg+JUUqqutu+v354bccFylU0RZuvBPKmZ2aZT/sTXxxcfbkR//24rP/4figvXZtvLezWxZlgR33pzqIB40kbJwrLLlS+5yKx9p/Gbp7pjn+xhv7r662f/zjP1k/ee/4W99okwatpZisaW9dNWsf10vTvzyLm9X1o/lOWaE55BJtP5SzVKUHfn0BCbptLIs6p+CHfmSqW7vja8e7o/EX3eb14ZMJmF/3oxERySCExEwoWRTVZC0NGrJg0/jh6ctPvnz59OFZzd3tm6Nq/5jG12xRV46NyZIjCjCk1vuKIWTPOrF2mDYpcRVx0AyasybJyF4YqWSJg08QDREgZWtMASHEtI4bH0scFSJZs4nSSeTlRc+28CEkBSXjSqvN7Wrnd9zkflPtJrHoimp80qMrrjVSpAWeXf6HHz345b9t8PE33zS37t43Sfxyvdq0UQQ1O0XFBJIL0zeNjqZ1UQVbr6pm7Xyv/SXWu1Njbh6++ud/+tl589b+HktyKkdXg642uH4Zzn71V7D9+mBBbudOF+IQRzlIM5oujm7Veti/+HH/8kkXdewHU9iUfeXKO8fTsqS37hz89MMrQGBEAlUEFSRAEFUgYOD/VB8YW3FO9Pmnw4d//enffPZZDpujQ759b/7m29dctYg5GJCUqm1LyfcqUJeptiJJOEEfNJmq1L4wBWMMAwTvEX3KYYhi1GSVGBNAboos6mJSW1IM2F51UVNMc4AMmCAQSlx2GyE1hpgqV5pU3tb5b7vJm6q2T9yMF300vdhul3cXKb54+uin/y/97I8m86s337l5bWdPzvCij9twlId5tjYT1MyOgCSH2PerZd+HgvxkPurwsll/7WbrYnYATTNzXvuj2VbHc7q4jMsX/emLjTHVxec/s9sPP/jumyfzZn8yrpsam8n55eWPf/LLa7f/86jT6ZuLdvPf9WnrJTm0OSkzjGo3qqrv3rv31aNfrdsBMQsKg0EEpCQCAmAJswCiopJ59uXwP/34sy8/+9rZi/vf2r+7f7g/kunuPpg5iHFOEDUkn7JhNZtuaGOuGAsLjUkdKKU0IDIEyThs+iDecE+ZQu+DRpslxxA0F8g543ZIxBaBmYoSs3ghS0B2GELy6EOqaiLlhG6Dd2j8/qS5IVj0GZNP2/6Sjm7gLM933Plf/Nv+iz+2m58f7NPx0Xem5Z7vKw+TPJ5WrkZyVelc7Ua2sBZUBSVL3Ei71qEd2Nj8art6WaYhvnpkj+YcYVEJvHi+WhXLwaXcW0jb87N0+as3d9WcvXjyZNmN7d6NI9OU7TK+c7JP8Pzxg4v9kztv/t4/e/Ln/1dRs1xHMrDc9KbGu7PxNkz2F8/W6+fKCgpAyqT51xIVUAALpMAZs/m//Lf/ykzSe799/Ztvff/wYNY++zr5rpkfg2BIvJVR9oMjPyqktDSb2hBouR64wGJiRm4wCJvB1bYdfOzFA6gjCylLjF27tAikgM50Xe5iG1NCoSRSlIatqW3GSA5KlI21doRWAASKi3igs7cm9QkVUy/ldghe7Oz6/t69W37YLP/kX4eP/s9jbg+uHR4cvDsfXavme5NRVY8dGy0sWVsWBmxhrXNojQIrOSATU4ghxgCUhu78S//qMzZ89eK846W/epFmz2fVTWrKdVf326vh6cdjfNFddC+77vu//c6tG3eKyT7biU/4/Mnzl48+P9o7+OzjvynHP7z1W//F6Y//BQAQ2L5td0ZlVbLLYdrUCCDKjECAIECqAoAA8Fo8gchC5r3fqd5+6+2Ta3eawlw+u+jXV/X+lIqaQThr19E2FaRbRZNMXRXD2JmsvPXS5mzEZyBS1dhLQmO5JJf7VY4rkOh7YqdRxBhcS0YEkowqpUtEZUwSiTdDm3IkqbYpAdjg9TQ5mt2b1MdIzarXYbNUW1x/88bR3XvLR09f/uX/g179f4/HdHLrg/He9Z3J8fF8sndQTadY1YCU2DA5REQlIusAjUSvEkFZ8pDJ6u6+II2Pbyyf7F18+UWK1ba7ZNuNRq0P51YZvF+++Nz0p6MStf/6m2/Ob+/XExxhqAr2O5Xb+9571RcPTh9fTsz5j/7dH/2X/9v/Vd5/gOc/+fJ0IByu7U+Hru07b1gNE6IoYsoKCkyUspAiIAZQAiBE8xu/8V5JxTDEdt2vTx+lGJpir9umwlJl8WSynUL/8tXVhbczzIatY7M7RothSAWG5Cpw6ttOqrK2kFMcNptXDRtCcU7ZQt93oY0JZOxMCpiyWCea0nqL1dxIlpBoINzG5Aw+W1V08A2w80E0tuyhLUbFzXfebkbHj37yo2c/+W8Xm4/u3Llx6967i6OjcdMcTM3RXJt5sLXDwgAoEAFFAFGIoAEEMSZAQnRkkExQXClVSuV4dqO95l5++st2bX73N+5cv3PzL//iUYeNsO/WD/Dy6+Y4X9u/3EV4+fGXT6klIMphOqbr979//9p9BXj8+ReYLv7oX/742z/4+whnT3/09PrE3r6+WJ6vEP18PClK1w9tzpxfN/8ziYCCAAojJcWUxIDalH3oz6WPEM9GO3NW5/sNasFqi9LszEdNbR6d+udtFgtasCVomsgqw1CSOKAluV2iVfCiKbCtRJMjKVlXqwTW5OgladQcpRsS5YCbLnsw+47PWx0iIA2zovns1K1HN3fM9aa8L66Jybk6v/ved0EWH/3pvz775f9zzzx5551v3Lj/7vHR0e7C7dUwnqqtwFgGRAV5rUQGNQgGRFV6kAFDC6lDrAEbNAYRwVXgymLc76a99d7x2av43r3dYrppRqdnj9r9W/drFzoNI+sPTL8+69YV/c3Xn3/94PHNWXr/+sHy9CeHBwd3f+9//4tKR0gXlz/95Fdvx25/Uk9GVTuqqlTm/YkcD37SEMZqEyMSilIG4ddTeygi2SALk/nRX57bQm/sFLOm3T9eBLt3vmHnRrs7LAIhgqpx1eT2cftqGboBk8/TEYwcVbYflU6EAIsyQQ62dCFqlpwTgJOcBz/kNCqL7RKMI5WclMiw37ZhaIOloa+ebX0pjPXBi1fhpTSz4oibWXTJdJC0v3Hv9vDii9NHj9cPfnJttnnz5ndu371/+/DawU6xN1FbD1wwEoMyZAdBFQPmTlOn2WPOqBFANSeMW1ALbkqukuwpbXO54Ayj8cS4UjXx8LmZy/EhnJ1ePH9IREDFYDnawohrPnn68kcfPlleXHyMopzeR62ah6d/9n84fbq4fvd7k4I//ugXry4+/cG9gzePIw1D6WA6Lheb8nh8dK4bL69iEhBlZCHJKoQOMYMwUTJ//rNn2YTD0c6Na/SD78x2x9XI0OXWPd8Uh/M65ND1QdfBYZjwymkx5Dr6EEGcK4GCMVakiJxZkjC5JApiGFGJC9i30MW+hzxzlolcQkB9sU0ZeN7sn50rD6JN9fK0e7akcro7n+ylptr23Qjd/W+9B70+evzo6uyjQ3N54+DazQO+Pk1v3K6bMhhOahxpAa9vW4crjZcgK4wDSAtJUhg0DQCMaFEVLYI7x2JKtlGAnAeFRoqTGk6LtHJ6afKzG9f3kh7+xV8+SYM8fX56ewR5lFLwV+vVEAYQ6hM8XuU7BwwquHngzzs/u2uOrr148LOb3/ndW290sPzZEClHQDBs7e2D6Y3r0/Bl//TJEvHXbVEUVBElBA2QyARQk3UbhRBHFodtmE5kZ77a+HJ13oxtzj6fd54M7I6bSeF3dNOytaQ+9tYUUbcl7ORMpjDY+uBbDD6oH1XFdNTEFHRNY9MaFBFFW6pyO5zP9uab8+58uBwf7bbr8PBqM5TXTha3puW1F2tM5E6+8d5mmfvlx+3Fp7v+9I1be++8fe+dN/YX++PRZErpEkFRVCHAoNkvQQYdBogYU5IcNQZJWxm63HcqK2aL1RhKU5gxV3Ma75lq1F18GfzntRwTk3Fjl17ujPabdw+7QS9evtzfuYW09nBOpr81NhdH1cdDPy/h+lG92NFqOu1WGuPQhyvuKhVojKvf+s38448EbfYJ0E3q2fWDmBvz6Yurp7CUjEqvJdIAmkCLrCCSjZFQF/ze/dEPvk0LZzykF6tyYusSt47WmcZSjiZAOfXBp1CWdeEbVMOEIEPwBdvOr7Xg5HNMKJvBxB5UtchFUYDANmwJCH2fhdtl2w1bW4zaDUgJDR1sl3Kxbi/ydLbzRscH3eUplPNvfvf9sF6tnn06nL2cpY/feefkO2/ceffm/mxuXF1R7DQECAlUgVm85iQ+qm4TDH1OrSShTEkKSSKZlMdGgT1p6nJ+xXiO+IU6q8VOyqG9/CnmrkCC7mkxfofM5rvfvnb69PT5w1+eb3Kr7ng6zLrh/bv1t2/vWDOezrbX7hah8J89TGV9IxCR7zar9Vv/2T+MBDSZkRPqCakqm7ocqhdXZ6xdU5i2z4j0WpIuQCJCkJmM2awv3v3uzR/+3hsVsA+XljfXsf1qaX0bDyt/fT6AXW/MuDOYEw0+eERjIkgyjrNYn7oUiQOAat8tLQU2uV8nATGTDFlHXJkqdh1th2wYXGEfPnk2HR/GYrTZXsVsf/LU1fMj0cZv1sr8jQ9+aPzo7MnT8OKjuX72g3duvPfGwd2T0XjH8GQM5Vgvn2P0KFaBUQiyh1zKkH2Aocu6weC3mhJjNMRaVugQbCHWIYCaIfitdGvZvOT4IM2OY7sm3HCG9nJT7a4V4mS0/8Fv3/v8q9svvvji5w8mRq9u3tCj2CK5hsKk0IG6rx+nL7/euXYyb05u/PG//ffHb/6gaxbt2acH5ZzNFgxt2j5F4/t+28M33vgulJ/99MPPQdLr+TxVzZghZ6Rs3n8z/P2/966xVQYRLIJPpZ6/u1+eLuurlT7exIOFLww4qldgUaMXiVFLA5RVUsheFDesxavLjU/dQrHfxgQxhsG0Tey5KDUNEQRV86rdpqjzyvW+r8Y1LEb/5q82VE2gOA594CLevf9Dv64ul593L/98nz7+/fsn339zfO3WpDxcEBv0vfqX4INGFQAUzCQkY9QNEQPWsTIDubg1WYVCR6nlkIznAQbkLcYOUiYDzFboCPJSn56Sm1b1ddDD7bPpaPIEFzOq8tFi/u33dk6fflo2t/7Dx3H30dX7d+lgHDP6L9fm+VfmydNFPb159M53/92/+9PV+ebt/+IPHz54dqsYsBxF/0x5rBhTuwrbmAc42NndGS0L81UM2RjKURBRVdkYTWx+54P3nj7ctJ5iu07GAZsKx3OH2fillLuIIxedkwTbEqrSKEuIA1ANQ+yIXWZTmYIY4wqdpBhykAGNMTpaXQzWSlU6CzKtXOd7L/nZy66cmPGURiP78SfxxTbPdnf67bKe3jy68xs+wnZ4Uj/7m7v40W+8Pfv+O/bkzWv24DoAayZIDKEHkRxjCkF4xExilKodNz/kiKwYRLpVe7k87baXNg0FikIGjGl7EVKrnWe/dWU2qq6q/RBziuO9nXUceffu5dPPar+tjlPl0t/5vbd9l/76P37+vd/5xzFe/o8///H6/MXYmbKoJtXi8Ph+cXLnJ3/x8xePPpWdN1+Imy0vd799TFfj8Irb5RCHqyiTDjSpZ4CsYtkMpABACBnQAEgS5WT+w89GygnNBanxNtjos4KIEKCxcv+I92uosm5iAsh7I0EYJFI3eB+NKaCwpTUaxU8LHLocEhi0Iny5GZDySHXbbTIxW64s9da4AjM1rjz66ln4/Gk/raZ9tyqmo/nOPRMLcJHOP1z4f/sH39z57lswu33N7N4ErDSJ+AB9FmENGIYyAWpmplhmQRZCy7s7Do0Iu9ngTk40+bhZ577V7Ypiq67WddPK827Ylkux3CeotNzthv54792+jQfv/8HqYXH17CPFz6ujO6Nm9sO/+41XZ8//+ie/uvvGW7/1u/8589VmvTbFCLVZ9dUnv/jy4dNfnV6uvv39b8NyefO9eVidD50ngDZEFVEMKENqO0K8cf3Gx1980vulCggpC4ccLFNKalLVaMpMQaGIOWGSROwMmhIqW69j1XbnO/OrlNKQDURZbztBbTR1sXGimeNCJW4oDjFhHcKVZgHE0F6yLQTBokaBPnqL+XKD25ibgs4v+j/56FmmWRaoxvNRubB1o+VAy8d7p3/0D95333szj28em8VtyDYHHZZtDCbrJCirYCAwTGNLVYO2JBhN2Dkox+AcclEaI0hZUmw3mEVi7y+eYPC7/Wqzunz54sHys6/k8tGsfwW7423sZzffOv3yV/n4jWL/nbXPl09/dlC9IK4Pp7f+0R9+8D90f/zF5x9/9gVfO76+s3tjvcqPv/z6+ZNnq/7Ftg337nx/d/H25Iilh+PjXVmWWY0oaNSR0XkdunklhTta7Iym4xdnZ4YsgskIxtgswsSGYTBYXrUAYelcIVyZyhbGNsaYuqoKpdzl1E7HBYcQchadIbeb7VWGEJTUYbS6zlHUklkzpYza9cNo2qxWbSiZjdOMmHNCCb631KSeotsUZfmqg1ljPYIZTSBvVl/96vr6J7/7Tv2Nm77anZvpCVS7qXMXV/kV7HtX2QRooFBpGKY2VzVwUSAjaIRE2i8hN2AimBLZGFfY+S7kDIDN0XVFAJB5iDcpvnj4xWd/9C/Xn/+Za3sz3TPMpUvYPnFlw9M3Li6fyqMv53pRpPXNw7f+6T/7g//uv//zzz866wZ/8ejLl18/6LzmsAWsbrz71vd+9w90ho3t9mdu6PL62ZNrJRSWmunUmGkz3XVNTk25d3iyN999YB6zkCAiRRFgAGU1j097A31hiYgMqeFUInFRWcOG8aS4rHR1ednVTX8wcVuerEO5qGDYjJt0dbW1zcKcnvaefSFl7jkLi881UD8wMzGyKOWsQSSlLJAN18UY/UZ9tqKcTVmZ/So5efHhjf7z967Bt26sy+muaa5hcbPzi8vePhMJBiaFbYrc5NjY1DjgksFYeI2ByarSYQKIXm2DPKBhCA7Y6X86eQMyWkOmAB6d3P9g57+5/+P/YyvLj93h/e3Zg/lkz6J38xvjg1nW/upBzg++ml/7rAB749p3/vCfvP/P13/+k7/4Zde2TUkDkl0c37t/59qbbzZjTnLhyp31tp2xRp8yjJO/sOWU7SSKYwjbIcUu7x3slU0VtpFQFYwhEMgCaLS7ospBKqCoUK2nZAICdT3Yk/nFTrHGpMhTVNKwqel8boJEyj2BNYidzfPHV5uqtJ4Gy5KMjKpme9mJttMGG8uDJwDSVISYravZYlmNP//q6nIQhlywLXlaDY8O4uM7O9s3TlLt5lxfS/Ybr652z7wuUy8p7jo4NnY+MrZIQAwEAKyAv/66sUUVjQASkJdqCzAV1DUWNTIrEUgA75UQnINiqpJsTos3vnX6V49279zxp1/sv/23tSDtHxR4fHLtut9869WTTvlqYb4m4jeuvfnP/usPgsAvf3Eh0Y4PDsxhY+e6PHs8rMrpZGambl7N5OUppdhz0YWJlgtrSGlkhtXEGh383mIxK+urdpONoLDGICqExohPXUqV0wpwIxvy1iO4TX7ruLi7kwpDVDS1cyMMrKChnaKct1liGnLan9CLZ68uz9v5wja27BEZrZCaUiCIYUTLvt+qHRVl8fzVq6ZZxAEePs/P1gPDxBkuZFbEq/v60W4NN3e4JAPl3sq+9+LV3mW/JnM1HxV7lTuYutEE0RoFQFRVBEFUD2ELMUK/AUXIArFXIHAWyEJZQjFSV4J1YC0YBkUYeug62G7QFUf7O18pvrt/w/hTYIugZBqQq8n05PjWjavTJy+ebZSv9vkrVnvrcPRf/6+/9a+P9LOH2Lkk6C/7tRieZl37i+n5Ckd2b+atapvFRwzDS6r2aMQxQlKisrr/jff/5pe/bLdrEc0M/tctUTX99kxdjZBiiNYRSF+V6Z271Xdvu9JlQqduFikUtKKYxboSfYkUnWgGZ+328kJU2pVPxka0u2PrgdlYGSKpyWIhm3XXTacT74k1XUT388cvumCxUC4Kof5GfDQp03js1IDQ9Gm88fJR0fWfvHlc39rZ29mrXVODRWAFiZBVs6IklaQJISaIHQwZMEJIkDoQhE2ELMAG2IF1UJRgLTgL1oEySNb1eY4t9e2kLuXydLJ7lyJDbFVU5xOU7c7e+OYb9z77Rfvos0+lf3l0K7rJyb2TG//0H9/78Zfji56fXlyerZ4N7WqdT3dT3IFYdL5NaHrCYiPaxx7IzWWzTiLF2A1xuz+6c7hz+IA+E1VScWzAJ3TGpLgyLFmCxgAAztBsVFzbtdtOX6xsWTdEZPO2sK0tsCjs1OHeLJRBBj9q+35eU5uga/NAa6D5xSBVDQWqmiSgmBmS5BRcES2pKfKzJ1eXQyaeBsQ2wK5/PJ4nb2sxso7jVXuweqjHoy9+497o7tuHdncXbKVAAILZawgYEZTgP+GPUFU1g0RIPcSIGiGBhg46r6lTFQQGV4plNAZcpZJyHCD2Pq27q/OxGdKTX9Ktd9Pma6KIbcawp4d37fjw+Oa8a299+FfLD3/yK7+5uv7msraLW4fWkvnl2ezo+q1lvHN5dXrx4ld7X//VlMBucq9qaYAhJQXLk5zatm+R7LofRjv7zx4/7ruemaMQahRlJWVBYxTBb3ota85DjCBuszJ/9WGvOBAXVdVZtzUaPifnSm4KPRjFcWnGJoprwmYgiE2JYMbapz4Oz9avqr7aHTeVLbOB0GcFJIOWyyScYjbWiNoEGcUU0F/nXFBRGLnUar2aVNm9c7L97W9OTt48gslcLQO9HmsQSIrE6hDEQIwQMxIpAEBCHWDYaALVBFSBqyAnDarDGoEheiVVyRkANESNELsArW6jKatpZcL5M4resAUrNLyCeAVH3yxHo9s3Xbu5+5c/Xv70rz9M/uktpMYsDqa3Xm66V1hTcXCjnn55+stieFUSYXRoGYoMZDEyUZG8Us4xphwxV93VZjVzbto0frlNmgUJiRKIoYLUC8QkVhU5ZFh1ofWMjIVBCmtXmwpc0IJbw6z1ZawNzKwWjp0WnGxTDs5Z8h3BQCzttp/WVBUlpsL3Q6ZsjNOEXsxqA5ILNkPKoWSsJVxbFMZBpNmFHzvHH1yjD761t7h3HeYnWiyAHCSEHABRXQOSQBSQQQGVNEfQCPn1EHTCmEQTOQPNHCZznA+4vJDthfoIQRRS9lsNvcAgoY2SVkO18843mpMd/9nfCCTBEThVURO/VL/Gk7dHO0fvfKvuffz5X5lPPvl5VfzimGp3/PdvL05Onz/67Aspn/8NXP4qe5PjlclMVYVQ1KPGEwzDIBIKsI5yT3p++QzKpnZu7PiCRbIREAIjoMYYHIQEyQCmbBBNiilDz1jElAsPYNQxozXElFXa6NqEm2jVc8E7mMd1SBy9GWhUrtkUBi+sibWbpwAeSJgrW/fe+EjLXoeUEZBAcwz3HDWFrKVYpumoht+7J7/5zf3ZndswvaZ2BuAgKYgAgapq9ISAKWkkUAHIIAlT0JSTD6EP1K15WAMQVDNsdmA6x+O7FA/h6ixfnmrXq99m3wa/BMh+6Lb13WvNMc/3mD9M6jQlyqgRNEekl2od2Wa+f+O971cRqg//PH7xq5+6+hc7dl5MP6h60w87fujGvmOfokT1YZmh6HsOAxq04JRRk/M+VGWTqVgralH2SsLyWr2eMaGCIYUUUSEMkokNmDIBYw5M0biogllcZ0lKKbHADIkSFyaqsBbrLJWbCqGhBHH2qju3uC3MfJqWo7AdAkEzgpRsWa+X/XboNoPzQphJRGoz3B4XuTpcxlk5gu/tt9+9e316403YvwdmAmggC4QMKYJmSAHjAMivlw05AjMiQlNpyaruck1q6tpg4a94/dKtTvG8hvECRvtY7tB+BZszGV7GzTZst7poEu/WN27W148oR5ofpbOvuBiDRPBBnKOAtN3g2cdq3Pz41ne/X2zW/YMfnY8/fV6Mf1mBqYaj4vmjsT+Lm64fLl1ZEoc+dD0RiSwa60z2yRrM4kZJy5Eti6IsaHq8f3i52gzgrRqFrBTNtl93nSKoN1xUVaGJuQCFAClKyJCwBcaVOioyJ8yiyVBFFJFMtkXDVVOZotoDBCxmUUaBDj+82DxfvlyUq3HVNqYYYl73bR+0jyyMopGAbxSmXuxdwHyyA/cX8f07i73DOzA5UJ6CqZCdBgTyQAn8gD4jMFgBMqoRUgZJaEjLBqFJWXo/3saV2ZZTW02GV+K3cPmMTp9iMea6onqkzMXOvRS/jt2wzc4pXLv7hj24oaomJH35PMUrixaIAAvMDocM67XCTwDy3sHt975z6/Thmw8f9XvHl9R86sIF9nXu8eJsRTbj5XZcQI1CJl0NMUesy4KwbEWzQ6FA2o6LRTOb0jvfePzsmR8GEUSUCGCurnyKngiVIcTeW8O2NNZyJugwsWZNKitrbYs252DJZt1izloaBmoNlc4Rvax0h4xmI824cUW1oRsv+u7asN1zL0rXWiqcM92mBzPKkhjk3nySq/3KmpvT/M2j2eHejEcFFAQkgEm90WgAHEAZtfJQh9DLsmuKTTVWmE3QOq1GUI6gmBUJ9/ZaePH00qdXr9YHfX1UDFiSxIibM7Pt2RRqylRW2Y1ovECLCHa8M0e1YJCO37EvzvqXXxKcWlRUQSxULGoFMerlz9G4m3d277+79/HL3bMvV5OdyzJfFfYbL191PnfJB1K9GPyE3aQAg13qeOni2Hpj7NpjJ8TlqNM023Xz6dH1m7euNn1IQQg1s/GhNwSSUQVSzslH41JT187WwpCFjbXIgmpEQNQOoGDQEEtOiI6zDDkZ0ChLDN5rv9m62k7H01Fqqi9aXkO6C0tZrxybZRswd4pmv4Cd3RNvh4Mp31iMjye2moxxOod6B4pCiQA0Xfn1qttmTdb23nfL5d7RTjGptVauCgDRDBi86paLarI3aw73r33rve3qcvP5R5vPflSvNtSUmRQCQo4gMfWbJEEzA5XTd97ixTHYGkYTTKG88/ZyyfHV1WI6mLDVek71BHKP0WjMevkLt/ed97+xt3p0179o26vnXGOB51bwdFjPi9FydVHbfDF0fZIKq2ldAOUhxhFGA1qxHaKursJ6eeW7y1F3SegzSsrZIBkEVMGkAJoVMiNlHyMEKqgsLRjWooIchxQMQJmqwINDEgaLFjIJRsjAYKPzUWOMKWlMfYi+m8zHZWmfSOPGcDIOTZdS8JtOQsp3r09wPqs1TRueGbcztXS00IP7aGc65LSN3dnpoK1QMGXhrzbh/Pm1936Q8nZ79eXmy48X+9OqAug9YIPFdWiukZuQmxg3LSfT+W/8vv7gA1w+zF/8dfjsI//iRYxZZUVWxWd78o4dXk5++w+RC2iv1FgcTc31+8Vlf7ra2PVH01nLq8+B71FVgG8JSfVKr2Rstrdu2geXO2HzyvK20Mcvzlzyw9Zag5hFG0nS1WqBXXBkE2lIQBJNYW3yuYsAgx36sbS1DdvBAiJANoigKkQIQAaAGRNqjAEwIVcFsmxbBknRB5CkkZDCIIqEBWfua6iBIGtX+rlCy+JU0yB9PwS9jLyzKOx4FXRRIGFvwbU+HeyNrh8ecUqTinZ4cTLN5sYMTr7ZbsdnP/6z7XK9Hh4v9vJoVhjbqDE7t791+O7vheWj0XxR3PsAujeGj/9F2r6I7Qo35w76svmWOfw9nb+BvAFCwwZsA+UJ/+Ce+yDUjz/sf/JH4eFHsQvgpt3pV+Pf/wNTN5gZh6WuEVwF5Why7/rpk1fPH8/MctU0Ua8+g3wDmgoSwCCav6DKnUzpwrjYV9xcFNCvL/182lxs+l2eLvuzhWv6LhrTXIW4W1YJV+etNSqjFoQkGmvIFlZJR2NTrVBEACAbyhBEiRUBMiApkUAggSwcBmUnEVPKTGAZgEWSzRRiykjRAARqsxaiEviCMgeRBEYlk4qXPnhv6zpAROMUSsKWIL+1aCb7E+nipFhcn1fjvSTT7z75RZb234/fnO/yyO78kEdHVE7RNUA18kRllWbz7uXHaZvr+Q3+wX8jce2GbVg/8A9+sfzyw+L5fxjvfOAOflvrQ7JzTQq2xmYM9ZSOv9P8L96rvv5J95f/pqex/+ovj6aKaYAgkiIOzwB63HnDTfaO7t165mX56AkOj6sya1iR3KBc62YL5oz3j4xbzRam74MkLnAJnal2Dym9jI5Dj4NxZMJpNxwIeOigkfVwpX5kJpjjIBvKvOBRicnfnpSn28EnZCiMolpDopnJKoJkFQRJOSqWBRaGEmBKMSNAEoWQMWPGhOKGiGSCCZgScJHQKAcWMJpFARSCpC5vS6lqY9hREGemvGjyZGcMm2lpeDEaSVU/y/vt//hXhv98/533Rm/8z9zkWPJArgQARItQKiiwmsl0pO8M25dZ0biZcpXd1NYzMznik3urz3+2+uwXkyc/KYu5O/qe2Xmf6mNoT4EsNRO1jRntjL79ty9+/uPq/m176z0kgv4pDYOiwFa1fEXT48mda5dfPkqH3189Pcvd0yJ9ZGBt5EamqNCRu7S6ne1Pw0esDeQcAWd+tRybMqZQz0fnG39UNm2/OstFDt2Mea7VNodNB2hDUh02XOl2PraDSIW6JulUjBaUk7KYDFmTKBEKEAGzTRmz0YLL3MWcRRk4IxtCsI6p4JzFkcGKMLKxVkQAmYtsesqaQ1bpQ1yv+6835quntuMZ7FUHYF9Fu16nwvlH4habdvb8i3vXL++//7+UWRPbL/zqQ2bkep+sYTMiM0c3RzQKHY0nZTlmGn49ewVMINk43n1rMd6NJ290D79ePfqp+eRfjty/cKM3qv2/bfZvaVrj0Md4qTwysXWTKwzPobiubqzrUwgtdBsUBL+2R/dOvnvz2YewXn4Qz8NYHtVwCmpwNFcs49U5uVg3lNBDvwxtQLuACFjn4arbOzy4ak+9ShYHbcyqg5Gd2rItr7xHCapEZuu3eQWjxtK4wcseGdlQBiSTNIEIIQMCGeMsAoJggBCcGXsuPfSlqwRQna2yCiKQtSH3KVlgxT4MhpKqScDRqGZltWohch42NA15RgBsek3aQh1qitQEL6+055Z/sZnMHzw+XOzdvZFu3B5du39vVh7aaoq0VgiCPSgDEWYhVyAqiCh2STqNHfoLokbrfXd9wrvXijn3Zy8uH5/r6Wez9df16TU3fx/q6xhTtnB18Whk29n5X9GkxUFVWrx8gMQSzqD6PexDffebO5tHnz6bt/wbsZ0CfVXiY5M3UM6VxujE+s24WfSrqyTLDOC1sGSH+IK1mSYOGCrmuqBmOq0La20qm4SXw9Cj0QRFqqhBRe/n+7R6gtuQk4kZDKuzLoaekJCQUUBJlRkhhiRVhEKxB98PxMhZ2pzFJRUpMiPgGqzNkRpVZwhz0AzoiKExZKlaRgm8Lo2xwBLda0GXDQYL8s4IY40YxQ2tPPcvfvHoWfnv8874l2+/efTtD968/dZePZkb+58YYGmDZiaGCVSTqnqENVQVggOukRDtAb/xXZz8gkb7w/nt9dXz1dkj9+K/r0fHefz2MHr71av1ZNfkzXNLJaQxrjfatzAsod+i/jF2l3j9g9m7f29/+yefxSadk/ZmQR+XukKqwY4dnTB+wUUXTmnoiVxhbNEF7yxr36MltmCR94viZOfa3u0b9bTpfFs8fPzo6ScxbUIrmrZ9Hvb2jh+vgzrUXoymKMCMVNkm5qigApQTGKOaKAwSRmBJe0lAmtTkGAwBR9RsPIhKYGeTsRwRGRHIAxcQLdlCSqOlircxk9OsVilkLK2S9LLGUOWUoO5kyCUW2SU31EIt83l/+fVfr//jh5/ePZn95m+8/dZ7b+0c7FtXafJAA8FUUZgYhcXsCSlCDRqAIhPr+LCWjuUhjPbDwZvbp189+uJTc/pscvHHD07/6uQbby2+97dyfJ42n5g4R+9w+xzSJcgV5DPJGTTwrd/dvXbj0U8fnMMoX1rbI/PAdWeqAAGIjJrNkCUETVEaZ2PYCIQAocSoynvz3ZNbt2/dunVy6/7Ojes+DeNPP5Mf86Mnn3i9HAQV85OL843HEMSoM8gkKkMOJReAyITwGoScATBr0jh4QgsAGRRVUYjIqaSoGRWQCo4xWsFUGs3eaAWJXjumWNtnHyUAs0jEAGDqoMmkwIahUwEwkMkp5NQBF8EEwNfY6wi+y/zhF+e/+vrP7//x3/ztH37jO7/zmyVcMDmgClCVHGBDHAUQUo+YFVlyUACt9+1RbK5ewkVbLnj3xt7pS/zk8cX06PbdP/ivil1Ir67iZU+pw23GPEAGHCJogvOHKD2Azk6+d+f9W3/2r34a3PV5mSfhV9Z3OYsOXsSx36bEqacM6oskgUY0JhAu7bwoi9FkNN6d7p40ewdmvKir8t1y0rav1uvuooOUlkWSJXQloSFSjgZf81OUUEUlixITAKhIUuSomVOoCiIwKQckUYIIHiizEhEgEFECAUIYcrRKibU0KpaXvvUhCZHFSKxMHnLMdmRVY45O4mCo4spjKjAzYEdSqibHFlAUexREj8I/ezg8/L//6R+eDj/8+2/OSgegqCoaQb3IBiQTgqBFUKUMmoEULOEYub/g7dKMPa7qkx+88f4/+qfl/g0ZzgQfhLwDprNyqdpxjEQZsEGH0G718V+AhqMbb9x+88HD57mDmPMrGFY8dGAj+CRRdIjbqFNr/DCIHUzGEWJZF6PJrJnvXdvfnS/mo8mssAZR3Wx6+50fnF74dhhexHWMoUOQxKpZTDavkYsAqpAVlCCLECKoatZMqKzASoazqCRUQCmBowBoUiTBWAklYXHKzAhKjMYiRNmse2E0xiTkSOByJAUTMREmViQyuejZl6lEgmQ9O42ZKSoaQsiEGVRITU3o0X/+2Zff+f47051M6lURNAG0Ch0RKipJVBUlT6CiCR0bcMWok+FSeTzuxzd/82/Ve0c5hXD5dPPyc0xL1kKZiD2mDGxJVpoT5hLCGE6/qM34rXduq7yyQ5DuIHGgFC1w8jnkbAxEJUPRCWUqxwUUBdw8uVPV48nscDrbc1RJzHEQZ10Y2nI0ufHGneXFq+eXLxQEU24FyIAKmdcMG2YGIpAsr7lbiAqKmBEIcpbos+KveXTCghkyCtqcVUn6LBlRY0RLaFDYxEF7vwk5iWjK1nIhGg2TwwJMQmsLyzVyBHXsDLFViqisnDgHTKhIwKjM6BABCUblWMbzj375VTOWvZMbiK+5a2NDFlUybBS9SkYUAK+GiWYQN2icmx0MA11//3t7d94hM+mvPr347M9wYC4OU92wexV4K07cVYXDGnCJZoGugNWZ8i8m5viIlsEW3Bxp7iW9yP1TSdlmo9W2D+MoOi6sIu4UphkXx4fXRs3eaDqviiJr9P12yHlkdkFgu16NyurkxvXPvvhw6JcmA6JmJEEyKoJEqECIOWfDr6eYFJEAUQFFMaQcNBECKStkUUiSCFAo2cxRcuLkmAsCYg59uOqTEChkyKoGbFW5Ym4NWVNYrsllRGOQAzJCFpUhJUxGcshKlstsvOYBuOCCc5SQU/bxb37+1fMvHi4m/Wx3UZSNqgMkxAJUGAvQteKgwFmJCRVYcdO304sXRemmu3e/Z5tJ2JxvX/3czJzjm7lbbdenNDJu947p1klbuJrxsNTVBoFBBFxpjO5NaTtU0FvOE4Q+pj75qCDrIVozqtwoWioQOGxOpjcPdvYn4wXVs4xSVJUbj6EouWDriuD71eVyPJ4f3jj+en0WUyCVbJATG1REQlHIWRExy6/9TQAUBMkAgaakWQQMIAqgYWWxnkRQCJiCZqMmaQ6ZC9XY+ZgBCypdZWzhXFWXFZoSrJBgH4L4XrMhYYOYbC4iCg1gp2o8QJGBEghALjAdzas7hweS6eL81dmr1fIqvHjy3A9XtmCERoEBFAhRrUKNUIgEVa+QFYDsfL09efn04u3feqec7mnu0/rzchRx71jWXiNBtqr7ZhJsOcpuQApyWWH3krsNECgi2ouCZ14tmmzsFPI2BZ9jN2Rz/nynqha5KHtSsz7fnc0Xi3Ezblw9IltkImZj2CkzEkymO3feevvy5Wm7XN+5dferR59o8AwAwImSUUTNYphEsogAkqoQKgIQsCInjTlLylnUKCeDIgQCmhWARLIQKGrKQ/acwTlkQ5qJRDgLqo8+9P2vlVRKibBmZsMMBTJmMmQ4W2KKbKyYkMEZRcbCEmaVYlbcv3Y8qm7Erjs/XZ7s1a990fA1GhusaoDXnGYV0U4lAxJIBh8s8K1v/+DgjW8oDAjW1DU2J2garcBgC6kctovp7UK3D3i0k/QF9IPJexpXul6CKo7VFqOCh5gNFhUOBHKJKMNQLttGbBURWGiH0+5ssntw7Oq5oGEGZwpFEIkGHAli1vFi5+63v/Xz//hnY653Dxav1n0SQUwiahBEVRXSa9of8mvTDVAAQLFkokjMGQRQcgZATAAqIqqZmUVFM4LmQcQpBUJjWOKQvKiPrD0gkXWG0aIzRKwOSIQQUBDBZduRgqJCxFwZZmBUyAYtAb08H9Y/evhod/nmGzvvvnXyu+/cm+3MquYIgUUjahYNAK/nI18rJBgRFUDj2i/PmsXx9O732aEklbAiV6G5gUTqjM5e6qt49tWn0xvvVwffh/VDPhwDvZLHZ3KlkJV9q+MdLEYmdcH3yXeMoNgTQjvM02QBWKbcVjlMXDU7OpnNjthYZQYGxJSSYNebokoh8hCocvPF3sHdO4+/+Oj2zeOvHr/sY6+QgaxBBUHKQiJCr0doABEIQA0bQkohASIjIkiSDISUAQy4ZKPklERVGZCRoijnmBRiEkQBQCUEFBGfjWPQjMriRVGzQR6MmAIVVV8bMSoGlMJiUGMzSkkExBHk+dn67Ozq0w+f/Z2/9dYP//Hvs6lUPEBS9KoRJKn2AIIokFdABBKlPwMcz279blE1mjoU0LxE7REXZCegCcZlMW9wcvnVX3/xzu9/lyZvkJ7p4SJvBhlWHCagg0ZFrsyo0MurMKzRrTPYnKHtimKymJZ9vnRutZofzneP7xt2yI6JiE0iIQFIMfouEwtk7Xl9eWXZlLap3YGCJcT0miCtqKRgCHzKigAKxMRIhFgUVUZQVUYEBSIDiMhkCYNACCFBBn3NpjMMSCRs1bAJPWZRIibA/4RpRFYFyYAi4FBiRlWWBKEEJWUjjJhyBqPoiDNK1KBMyugKc7JT3pjZ4300PIBuATyBV0BSEU0AUVMPGkE2gAnShmFqr/+mLWvxKyUCTcSlpgwxoXOAY1NKdZT27ppHH4dnP/3pte9ew+oIhzPev8a+J6+ag0BGzcqFCIgQRSSsY/ZJ6PDoZHj5H2s7c6Pp7s13RuN9KivnqiwRiVEzEYl46gDrJiTr0JbOTUfVejoTDa6odNNmAFE0WcEypJSIfu1wIgqquSxLRQgh/JqySAAohITIQGxEcm1H6hIoxJSFFQWJxFhrK8Tu9SyUqCAgEiFJlGwsiAAHZRMjsRPTY3BQ4GuRG3QespoiB6qMOgsja0aOjhf4/jfm775/b+/WXdc41R4hgipABsoEkHIG8KoRyFN6pXmsk+9wtdB4qtqrFIgI3CAUCIIpKAphU0zfWbx16/zsk48//XQyerC434griVcwGiMFEEuxxe4KrU2ZV8vYTIFMEFW205s3ZtvymxdPnynv1NMjJudsCSSMzGRZLRgjnFGBoyZOWNH8YL+Y1inF2Lezven58oITZgoGFLIAgEJWRVUkQDFMxJSSIhBIEs0ARgkYFHMIAAaREJXJorNOmEqPAkNUSCkNoJkIGdEQCYGgOGJmJiACQ8SsAgKRooqiBLDoVX+9KTOh2TRlc/duc3e/urs3v/XGye6tW9XOCRoDeYMqgCMgAo2vPUsJRckBkehGecrNb2lxA9K5avfaOQoUABhtAYogHmEAJSCh2u7emT9+eu3jv/n0e+ZP3OwmktPRFCxB7KDLGHv26zCEbZw5KCmdqlAxqfonX157Z69tds5OQxcSEglkphKdAQUAAU0WWZ0BYy2Z/uoKYJZTZIVmPEVbKCtlZjSGCV8TzADgdegHAVtYBozZZxHNagjJZAFidAqKwhEzZWvARA1qq4JkZEuqp5iH1dVl1sRsLRMhomEytrSUIaM4o9ZwEiRVqZSCArI3Wg2cTQIDyi4UiSocvnly9Pt/+7vzG7ftaIJkAVAlomYVARbAQkFABIEJGFhBOgDG8vtY38e8Uu0QFLEAKECWGtdKJZk58EgRSLYgS8U4Ozk43D/9+utb186eXx95gAKqWtnRRsFWKF5j6yzX06nai+zXYI72bu79/E8euPF5efz3jpqIy7DabA92F64qTWliFyFmYBIigxYVU8opdqHFXvrLfr0N3a3d/SdffDHkqIYNAAFmVVJUQGVANBaQet+HmACZEBVYRRRTFmY2gBkAjapSzsrIkaRKGcHGYrQ/JhSNISsikWFryBIX4DwH0sTAigigCaIDqDEnMFbBZZ/YZAGOWSntTZt3vnX/4M41qKZgGhAA6TH3QK8XLACIWAIGzQGRAAJooPI3sXgT9AK0RURUoypIAFgDRIhbTQrmCKkWQdBkOOgYm71u1t94cOX2V1+V0ytNFmEk4xpMyu1zsRaNqape5JVo4ro8nE1OvvNb6+1pM6rLAjcvHo4m87JqinHTt9sUPRCys4QAWRPlRElS2KwSOo7bbnX26tbe0Y+QmYPN0SBlSK8Fx4oIxhhmE2LIGQiYEJCBCUCYkQwTSX4tV06YIYMlgxkG9daShYIhjJqFJbt5dYFWiRHIEWUlKLVhkqSZ2KrKnmivarEiSkiphgqzzmv4zTdP5gucHezceOs2To5efylARkBgq6rIRkEQMmgEBGTRvAUYw+i3gMeoLWRAKBC9coFoQBmwQgPwGlkSr5QHkRYhCrju6c/T+S/75z87L39wfrl37bhEnMJ2iSErtAAkMaRcra4ia3CFKXcmvWywsEmO6tE0eplPdwBjRok+AFolAdQ4RCbJ6sNGxCAQtJttMihIGdWm+sbx7idfPS+1NJBVCVAJCXIWQB6Cl5wJUQGSCAGLZERkSD4bVlbwRA5RLJFBiACsQAAak0ja1lRNx2PJy7NlhwlwMGQSByJTMizKwhjpM+rrBJIUyCjB9Wn5+9/d+f1/9Jv733wXgFEz1bsJFogGoAPwQIRqKL/2DXzt2JgQBtCMdF2rtxUDy1bTBjQDGuV9xYQQUIKKB8hAThURMsiWiH2f83BGVE9v3V9cfdw9/cWj8f3jvsOmxOmeXjyHBFkwia5W64df5HLU3P7Wyf53/ueXm9H6808O3nn75PabZw8e48n+/hs3RKyWBaw9EoqxTKh5kCGFPviYyGEcttEYjNpwnRGu7dz/8uFFLxuTBQR+Dbt3RZGyahIkVgVABSUCZFQBBmJCUEnALJAKrkExgUDKAi75aCxvtdWYFaeuqbG6hLbPAEIZkxK7iHiR0mxeO5ddNg6ptml3VPyTf/Sdv/WHv1Md3sRi9GvfRbSoDBIBWXGEYJUYNAJNQJICoXhAhJjU3QN7T2EFstFwBaCIBaJVBKAK1CgmAEXMSgTgIA+QArq5rpfgjDuYjevHd+23ytErkUknO6NSMG7AGSlYRUMfXqz6L7f1id785u57WcI2XnRe927cMWgnkwXyKKH168F2kRmpoJxi6oLGHFP20Q+xjW0cYoC6AWfNuAmr4fj4xHw4gHeGGEEYUNkYn6Km/NprQBEIiAmRUIGUXhdcBpCQmEBTHiA7w4iUxFRGXYoBgSqfTLqicj4qD4bu8a85dUCoiKxJhQTYWpPp1pj+4Q8/+N3/8u9Mbr8JPFLIAi1qIkSACsEqRMwekZQYlUFB0SBaVFEIED2YN8FdU9ig9pCDpo7dTGii5BAZUqcggA5xAGUEAm2VEHAECu7gTdGHSbdcJDOxzZxXT398ebrTzO9q8gAriRsJMXSw6pq+nhf37snkrtndOXv4p8HSZDGOrD7GWVX58y1bw2x7yZzEJIiiiSDKEP22933v/ZCgu1yhsdWobHk15+nR/NpX25eGGBkhiUpKkhO8LoMVEDRjRlAQNgYNMSMlEgJGQAAl5MIVosmQjXEYjJakkooO20GMAWTwCggiSEyMGTNarohMWb054b/322/9g//qn4zvfgewUVDQBLAFBEUGJQQRDKgRQBQdgigaBAANohH9JeL/v6g36bUtS87DvohYa+99+tu/Nt/LfNmxshrRVSRNUSYoA4Jhy4ZtWPBMkCYy4LF/kmcGPPHEAuGBGkMgJVC02RSrzazMfN3t7+l2t9aKCA9OFvwD7mTfs6L54msan34fvAB69x2X5GaAGEdIBQCWwRPyQqZulWlLJZkzCavfGarY/JDw1voduaneh4U0q2Ty/uGb28XFY833il0q/XYYtzhrzl5efP773/zV37758v+6H86Ozh4389WbL39zUk9C4OwsxF3uyWwo7uNYxnaXhq7t85CVcj/2u67vhrS/u6O6mbx41Mzmzz98+Zv3r0POxiA1cyImOYTtmSsxVxKEBcwiwYmcEKOwI0gIdTBlITgCeaZYVWxFhCnAtKDEPHTjYKoiTgJmdyZyXzT4Z3/y8T/55//12Wd/37A0BPYRlJ3IkAgENyd2JABECawE0CEoC0xekHYUHlP1GZGbt8CeyxaaIQ3JqcsKXAEKUlI99GrCBMGAztM2bS5BEZOLPN6DtoUGuNcVjXVsBXe3b7oYPVhTVaOEwfK2VDY5Pjp6dHq6ePT7//jP/vf/9er2/rOf/CHAPpRt2qGeV6FxMiTX7FRy1tR5ycM4rDe79caia/CSck5JZo32u/6rsnj16vnphYKDmxtMzYJUgBMMIGYmYTBxEBM3WCMTGBVyERcnHYoEzohO7lJVqBWJBZIzuRBs1KxZmayYuTiBBGiM/uf/4e//0//lX3hS0wlJDfSgQjBgFA+GAhSC/tZaUAEmH3HwerQCdao/g6wc6j64r9kSrIAqD3NAcNBQgvjgH+8EaeAgryEGtHk/5P1XzZMIwGMr2Yhqm8W4dK9Pt2++fsh37aZ98vI5y7Snycamq4sPfXr8i//4f978amit2aTq/IuP2649Plt5X+rpXJK2250VlxhLMJdQh0kuylWQWe1WoIUpwMdhSNRMi9O7r39z/uy8qSioKguTMEMNIBYmJwhTEGHiKEwBDEdmMINAasQhmiuRChEyeygCCUxUcwb5MPbd1oHYTGAm7g1jLPZf/NGH/+3/9D+WfULuceqAwrToWkuf+k1/fcUh0MQp1ACYwRRERNgYLLGR6gk1HwI1MCo68rVYbwSSpVshHSERZv//XEYKDG5MHr4rhjRZfPBDx4viAyEV3RJNCImpqlYyffzh3c1m9+3f7q+/7nfbs5efVLMLnj2yfnb+/PH+0n/9s38z2v7lFz9OqnVOEqZnrxbry51ZaSbTwiTgQFBN/dBJ4NlyUR8fJdXbu6t2s+FYTWJVhgTyPu+6h+Hl0xeBGQebbyMnIgI5QEJOakYMhzsCy0Ga6EHFhWEC0YpDDDCahASaoJGqcS2Tgs4HzYUFXjVVMwlFNRfm8Xt/+Gm77X797//vz/+r/zLdXrf72831u5t3X29u7q6+vvz6q9cc6jjhOJkhBA5eTerVdLpYhOXR/OLFT86eniyOriaTJtYO7smTceXU2CEbE4mtAO5UEYgApwhakmbkHjZ62ZKgeCS0qlcEY9SODhSCHDeYLuf3q2efjbt2GIb17YDw7vzjD+jo6Vgm716/2d3d7vM0TJsPfucHT588H/cPgoqE2MyyUlUzkEqfxrGk3PddyZkmk5KTgmaLVQ6y3Xa1DiS17M0i33Z352ePAzMfEE0H/5bmQACZE7ODFGBzGx0UhAlkbExWEAQVoGDXUgED83zQEPRhv2n7NhLAQcAxRK6mZRgbqf+Pf/nXf/Pnb7i2H9zu715v312/v/q2e79ZJwvF2GoItUKR4l68oUCIXhNPiUIdVtNvL07/1cuXpx9+cvLk5fmTZ+dnZxdhdg4QuR4Q7UMJ+s4bHr8NsSYijm576ODaOU8hElyMBM4o4ZD47tbVCzk9P25v5sRfdN39fQbv4yjLTof1u2+93w/d/ec/+uNX3/se8mBpMz17sf7N5bhpY2jSMBTTlJOOQ9ZchjFrUiQ+5GpzFesZh1G9ClRyneu06HO7ahqaL+YM90PzZQ5MzCQclUAOOTigCrMcgOoQPEKIhII3hRKFSGoTogQJkRjy8HCbxraKgSJVoZYwC/WsZuSh1VwKxsiaCpx1ujpuPI7DkCQLiYhHjywNMSJXFOBCkwiIu9XEYI4oPAm2XMbPXx39+Mff//xH33v28kU1WbkLoOSHcHcnJwIdjnCwHjp6eYA+AFZoDuuIs9INUnJ1UDRUxbwU7x/ah9u7/Sb37V13vY7TJ1t59Zuvt+3tugx9CP6P/sl/8+r7P0ZqjUQL1t/cBYdwGYZsRmYl913qh3Ecu3EgNUhVhCx4cqRc2r69en9dm6KWTddPQwhBGrURcBAJMRPD3c2YhEBEMMDMCWRc3NTZGTbRutDolkHuZCNRxZWxD90OehjkSQ2j5mA791HmU64kp6QC5+iBOEAwIfFeDWAE4xQKp0I8QVV8ZKiqABMCBdbg7J4t1HvIuNe7v1j/P3/1b7/36q/+4X/+wx/94d87f/rSeQ4HXAEnOJwAdU/kGT7ARwcbKMRlGQy8oLJzmEemQ8AIskBCHZrVkYdxcjSL9fLyq/5m+4AxVtKM4+7xB88te393FWeNetxe7bk4GGYARPNQ8qilFFPNSdtOAjezqgohFWPmEOpiVDe7vN3GKc/rKFQFDs5WF010KEVODiSUQE4QNzgcRLkIB9DhiYM7yxLcrIhGEgQSd+23fUqjiRJLcTNHOGjZU86pl1iDoaUkR8WxCPdpPxSzg7ydWAygSixBhCiYE0All1orDkyRpkHGnDJFGT0CbR7+w8/Lm/d//ge/fP0n/+g//fR3fy9WR67Knt0P45OTO9wBdxKzwTwxHhlFNsCZwpRgKMlLIXUjKCuzWB4f7m6LLRRV23YSF6OOk6p6/vKjxawp+/vJ4mMdNHd98IQcso5AYApGKXBIY1uGEcPYE+JMjqfNBjCA1YRpNq/3Jba74ejoSK0EU40hOEfVEc7qSkRulo0OvYDBfgikVya4wYzYYQHKoalAXSmgTIX6fl/UYx1iiDkXJmYEZ+TDv9St5HHsO2KM0piqEyejSRVATs5MkYjVNVuiw8mATDyrey1ShbhYTKt9uylDD08uKgAPl9vyp//mqzdvNv/4v2t/7z/7B3WzdBsPhchh+O09wM3hSZgdbHlHQUEwOCEoO0UxIR3WVHQyubCjWTdU+gAb1rndIuSUhmeffvTis1dV0KFUJ7Ojrr9BvzNiy6V4sjKYwTSdnE4D19v1ncHYQ783TWtlsrqqpBLLbiCX6Wqx3nWPH62CupFqjEHLIe+NiCDE6jAUcrgTgY0LnEgDsZkpCSx7CWyW1M25Jlc72Dg4HMZRCByZyF0JhhKcqiBJgmomjO5SggIwExFiJWcEKkB0M/MSBEQIIGQoQDFtu13wSGXXSExq5ohJMxxW/exXN/l/+3eB0k/++PcC1Q6GO/zAkCjQDjoA6lzDhKmQVMACemtwkugQba+hHsNSqQlNmCwEGKcXvtyU9V0XJ/Lk1YeTxazsTeJkd99DVaiQxLEf+zS6ltXxfFJVZ3NqmuXbr97ZmB1KUmmoC4PdvZSKeMrSByFA8+jmgRlgGIwpqCUiFHPhwzWdwG5uzEQHLJrYXDUdrsfCWRMTBVgpmpObHZqGuYkQQQhGDBZyJi2FgVBXfgA03VQROMDVrGZmICc4oSa3glR5VVlwNqVUcmx33A9dI1K716sw9oMU0RI9uPqQId+8uf/Tf/nn04l98fe+zzIHCbwA7J5hLfzBMTDOzEDCoEQ0Y1saWtNsJZJDqhO3YyuyOn2WytXl+//Ya6ln0wUwO7348NWryCGnnpGtvW8WzTCr+30qpsQcQRKrrOnt3b5XrpqJDjl1beq3zMbzeaAGbEYUmBeT+b7fTJqYUg6mGoKYG8GI2Ynou8wlMJgIIoellN1doTATEmbmIDBDtmEcyUDElVTKzkwCZhIKh+SywszMIaeULUcOzXyexzyUjg1gqJlwcYTigUwLDY0IG5VSCpEHN05jLhGVimsRYt52OqiJWk0eirjk0eFFfv7l1b//Vz999uzk5MmnsEN64GEfU8dIXEOOS7thSe6QeE58juEb1zsKp9S0JI9JT+aPXhRvNr/49fr69upNsrH67Hd/+PjpRyf1dLaYHTdHm7u7yGX/7j5Wk/X6wZi5UC6pf9jW8/nV9Z6cYlPjZMkNp1TczPp2MLMqWspj1xEwiw3NQu5zIJFcNLAcSo2bE0EdQmBxZ2IJMKhrMFcCE4kQCCmNVoqrqatQCFIZjJ3BzjGQM39HDqihpDYevieIq0kMgdJ2JJBZjggwFAOERDgQtAAGkCl3oQizmEdFCDAlHj2UwQkUnXIIQkWBRHDJe8Xbd8PV6+vTx5/aITGNorsTNSRHCItiUcevqB4RL5iOCwp4SjwxBfMxyZImx998/eWv/uovh9KtHn/ej7cwevri2cXFo8lkPjlauZGWoX3zQGLjriDlDBMlVd2lDSu47S0VBaFYrBqZVkacXVMay64rpeSu2/bt6YsP0vZ+MZ2GQKyHFmVO4sziMAYd+LkMsmLEITI5nECmZUhF3Rmu6kGIiUX4AOEBRsrIbsLmVCkF8kJWkmUttQQCXKj2WrwzclB1OAcBppo0oY6M0EgVglkMlQuLCSAZpsXVkpE3mdOEdmZEvdEkZPVqAvFVZFQpmdmh8bqbd+49eWaeOa+0f4BtXKfss3G419S5btQrmO/b/vXf/eu7h75d73bDbnF8Xs+nx8sjc6asi9lMg6ShFdDsuIk4zsXuvny3WC6176X2shlCDEms2+68HTUlqiPNJ3UzDVUMIBZeP2ymMgzoQ8pMfjRdxoYCi7ia6WHoYXVjYpiDiXFIqBc3z5bU1BzuTs7MZgAzMVVRHCAGJDZw5xizBDITkh65CkTCMGdzDgyQKlTJ3JhI3dQwuk8Ch8kkBI4c3FncFQJzNSU3cS1s7tpIkwgxasOhEyEMolKCi9pMwhffX/7g2dFSGdo5glvxcu9cXGaGoMOdjXcU1ClqP5T2Z5A0bEdujvfj/Ku/+7vLr96pTwonGJXWHh4ulxePVmdn7Xbc7XaTineX+9ztJXD94hEFfvTDVzdfvm5yt9u0Y9cXTfa+2G7IY6/BrB/R9VpCfbSS43lcTSawl6eLx0f2/rJ7f/3w7IuXb68uQ1YnZyIHoWgRZismgYmIiEy1lKxm5E4Ec48shzgOOJgjyIzEqQCB4WCBQ8idqZRE4kpsWVUHANkTWRDyvuwPc6KzV3XTcKyaqXsZXS0rGblWxORhNCFyV3DUikMgSEXklbdETUGKogFCFpBy7jc38sGffPLki5eqBcxkBhsAd3Z31nYDG7leucZh+8tgNypj1+9Tnty/Gx4eFLOV5ZK348P17XJxf3z0bLF4+fTxo8j87V/828m0mp8+I/Vx571e8nKBcq/EVUPDz16bm/U2lOJpD6XSWqZckHK/G9tjbo/j8cl0tqznIVTy0i+r1vuhHJ+chqaa9N2WiAhgwuH6DjJy5KzqhUjgABGxCL5bbMgpkKi7sMPhTO7ZEBhFUWsCB4/CBiqjlqRUwGxs0ahogiZT8vrgp1uMYEO3M/dIYsQuojzWHI0kEIMpENd1JAtwWGUKyh5j1Jo4qJRQFUMsVFJZ1MswXyIVt86sJ2uBxlXKmNxJ4iy3D/Ce+C77sNvf9MP51fXD7c3lOIBkIdolj5PT49mqCeO9vns9+eDT48fnef3o4ctvcleqZl4UTWqniuvb/vFHz/dX75//we/8/C9/au2+3+7cM3ml7EkPbWjjXCaMAuP46KarP3z+wf3Ynk26sZnvSh+sFEFQuLkGYiZWK5ZVKYJIWOBQOB1wOEQiMwdzINZIlVEO7lqomtRq7JzJC7m4sgd2N+2zagG7gGMkZhn2OZdETpkyDQ4xaOTATIGlQpQaBYEDBTjDyWEg9lI8mCEc/DJV4LVInhYxMmXNswn/4ItH58+ilRaaOW0IGw+NQQzsY6t4cCpWsvtdSnftftPvV/fb5vLq2/F6TfFxPOOcwVwo96vj04ujJ0+ffnT8ZIFQjp89mZ8/TZdvm6NJe7+9ekhX7dvrq5uwbOR0Mnabdr2tUXnSMQ3MxNM6QBU+f3Jx/PyimsxlutrmdPvu+nz2OFZhffdQ2/zs0UnIrocvy8QOL2aAhxiKmjtggFuUcMAX3dXdmRlmChIubjCxEGXUXHGdnYgsSCAhs5yHNOSRySOHKgYi5JI0G5E6kauHALPgTT0HFwZYXHOBGkniQhYJ7lYMFmsKQ4S0sLpx4tCIytgYCU0peJTjk/jZZ4s4g5Wecmt6CWG3xqkyJysbz9ejeUq5L3ft9rbdSrbj99/e7q7fZ55Ltd59sx2GW+Lh4tlFNUGr6eb676ZHZ5PwWaxiGYfNgPFqw5VNjybvvrm8//rNn/3m9edffP/xJ88+/tH3fv7v/jpWFZC6cStdxQyZLU+fnT1+9XhE2BbhFhyHX3/15gc/ePLxjHcb3/oYrBQhYirZjA64pwSmqqnh4FK0lEHNCAx3YSKKgBbmwOSaRSRI467CZFbEIYFJFZpGtZyVwMzEobIYzXTsx0hEcAIbixILcyw8mlIFyaWQZyYmIi9iZm4sFNSok8CjxyoZp8rcyFjRp0ldJcQp1Rpt1+eubyd4SzRCGDw1iKdc0n1J+1LK0A/b3ev9et211NmLm9dfjpt7G6Vecbfrze/3d++reppW54J63A+5Wd1evZleJx50t9nefLvmOTLh4duru877fuQmX+8evvnTX5w9f/77//0ft9u7hzeXi6NJf1u+/em3j14+PXp6ut70+1TptHLX1fnRb372y9fv4vMFkHfD2x2tjo6IjGB2GGoQjAuDOXDTzKfSjOO4S7uSBxAY8XCuBBkB5KyuXoVgJYRgKiIo5K6pDEqAQUlCjE0lrJZKVmdyIpjWoVYQi9ehUpdMeeqSgRAkeCRA6eCakI2DQ4LDSWOQRuqabazmTJVZnkZlhCLV0WT2/Cz80U8mf/CT48ePlyFEL6UkL2mVS932VyVv0u5u0z/cbOuSzu5er4fuxpOfPv10oHp7+fO+307rxfR8cVRPT8+OVxcrF8+5hF2c8kn3dt126HjIe3Ifb/dpDItufy+T2eL5Ed89hP32+NFy/sFz5KFanYy7cPzkuB36m7v9m7dX88XR0ZPlzbvbtO7qyn/0g0+m+e3165aOT44OeQLMDMAcgdigJMIc6mpCQcahH/ueyQjiRMTEcJAYKWBkDKYDsSuyDwoYVAuIJFZB3MEc6ohcrMA02WF8cjeZ1OwmHBgQoqTfHT2FxKMxIEFIqchBkwM0VMU6knmMsdQLsY5LYI4xkFMOsWlC/bsfxj/6B/PvfzGdcNPvMaamG7ohrYfSd4O1+7LdNPv9hFCqyTyM7qvl9vU3u/XPmuPl6cVTlunJfJ76m8XxRWCeTQ1JyhWnLx8eEscXR57y9q4b0myPTMsQq9ntr15DslSTV588d03NkM8/elo/e9Rvu8uH9eiR2vxwv90Pbd5sz87OwqmUnf34R0/z1WXwQzUgFUJxZnInhZOZuvM4jKFmYYiQK4PcrbDUZgr3SFJAzGRGEC2mgxrZYRcWEnGCqsKKp6LkTgjMAHkBsYqwqRFiKUXIS4CQEiJxoQMhHtK7sx/UIhHECq7cMkcrQbBTdjN1WKGaSi7ONpQ/+wVf3bXt+uiLz5f9xvbdprPdMKSxaL+TzX5o+yDTaiqMMnkom+7Ln+4fbszQnK2GjS1WvBv6cV/6/mq5nKxOP2mWpze//nXbdW78MB4vjpfpfY+0VYx6v+RH0/OPn+3ePVAVrVkwWVujff+Aq8vdza5g2jy6WF4cHUcpD7qIsnp6Lsvpf/iLf/3h46Ojk+Ng6h4yQYyoIioOIDCRsxUkZ4MHT+ru390qicyKO4Q8o3xH1QogFcCYmPig9faUssOZQCAlj3AzjFbIDRyMWGFMolYqRrJMRgDAZAhspgQhjRyMoIZAIUTxlPpBuAbD8kgx0KRapQJKLSInLeo8Myqjr+/C9Q3fX91u93eZLXlIXqUu7q86o1TnYlRuH345VnLz9p3et/X5cXO15otGtzfd9ft6tZitVg2q17/5eSgRW1NUmVKT9P66TU3lpDEvh3G70vOnZ8dHP/x4t9/Xi4UN5e1Dd7+n/s1D6YbqJPg4jOv7po4+Zl4u15t9M461yuZyf3p+GiQEZxJQcDpYerIT2JlYvAaLWUmWXUFsBiYymJMIOZmjuEZyUABbkJrcmYhApRQJZqpuDqLgcBLigwZTiDmQEB1YiFoKByF3YuboVpTL4fkwwKhdNJiZeTKIuZBr8qoidUvFsa7jySzOC3pwfrbgTz6Sl8/nEvOXv/qyG4dkMRGDl8MWeb/tUy9Zc/8wjutSpruuG65brqc89us902Jx2tQSKXeDTk8n0+f1NIzbXaH7cVg3J4/nz87xsN1vxo3PLKV6tQoTmX16XoXq4vhMUdpdt/nFr/q7XcoJqyoczep5/dVf/pxMY0XVSZYYnz97+fDR2cnpwssY4BSNlK2wwClwcDjDA7GSuzvM3UD4TgrHxEFITd0CC4EsmwKupA0TmIqrCNXTGh5yzlqSaoGrwsWZCM4gJriRM1MglEJceKypCoBDrbBEEEwAAwwOr82JyEVCzKRcJEME5LWRJ713TI6mi6Mp/c6P0gcXdd+m7bvbIEFFshZDtd913WZn406zgzvtZZ+wftiXwapmUkWxrYsg9+M4ncrqcYCgmnRFNZmlrGYl1MuXzwrBh5a5tLvbebMMi0Xk6vrN/bNn5+vb2+Pzi/c391VAaYek8Iyp6vbhfkht2u1qoXT1DdlsdbT4/D/57HjeSGqDw41cDhsPMaM4Cw7prFSncSyayQCQHGizTmps7nRIhAYTGbkzSVGFORErjLxIQJyEqFxSymkIB7YvE7EYcU0EkuwmZAHFlCkGt1KUiC0YFUFxh+bCJO4FCBY8V+puBS4jLBIVJpjEgKQosyZeb8WyUe4kNpnDOGpO2nftcNeVvpW64mhJjm+uh+1+P+wni9mYijOFWElX0v5+XdUyiXFICZvN/ubm+UcfzqXZtl1h3o2tlqGZN+Pb9URCSa10VZcIDW5v+O79zfZ6X4odnTwb1sjvbuZnq1DT65/+cnezCU3gZoEyFt99+ze/OHv0B/1EY1gEYmeOYAM7OyB0YGJ5EAdbMpgx4CA3d4YRGPpbiYQf9gM6SHXJGAyQGQ2WgmtNdS1VqMWBkjNA4YByuCqEyAlWFFHcnT2XTAYiIU9WQGxgqDnBRJjJNSf3wMpGhUQDyD2YVwqrDHFbZH53yyn08yOqEZFTHmW38W63WU0XZ48fzxaLOD26HMOb9c/2fTfhNGQEks5suVjMFpNEu3bfhZMnPLH9dkz3+yG9/eTTzwsmtGjUEwbxWNXny/m+23a9wky7pxcvd5dtvah9nyUyOV189GhILYCHu3U3dKGJdTV1QeYiqO6u327Xm+Ws2q7XAcRBorOxEA4fH0LsBD2AEwpRNoebaeDDFnUw7AFc3O0g7SMQDqXeDzQHIeMxlVJRVYWKp8RJUzL4gTNRoEoaVAA3Jzix8GhOBleTCHISEWYBmKIQO1QmHsBMJOwMUjF3lWK9svDeXieNTT1GDGMlouZeiSzm1auPf/zi088unj7ZDuny/cPN19eKsSseg5B5YXct316vH5Gfnc9KCePo1Zxjvdy2+/u27/YpdUO9DGNnsJ5SmD9dlVvZy1wq33bju6++vnj6LCWaPFkVL3fvbyaz6eOPXuw27e2vLqtqWc8la+r6zTCy5WE+b66+vXn25NEv/vqbEEMEO0uMJM7mTnSwZHPVnLOqEuDE9NsdAG5wATucyNwhzHaYZd0O2lcjMAwOcpQEgAOciROIQKaHR0Ks6i5OCmMRMxN2DkHAXtcVhToQiNUgBwkbmEyd1UfSQIpckjHY60hFZd8y9Wr7MTIvxEOoltPZZ58/+t0ff/L4xSMPMQ0pjyXpmC2XnIUphGJCVsQJxej9u7vUTc6OZ6zF/LQmXh0dqSbvttPFdHJ+ko3rSJNl9bBrr7q9xdXu9lIL7vPu6PxssThOZWzvN+O+H/bZJ7GkQYpBdexLUUeODXNcToYCz5ZLsTEHJjbY4SoOVTMlFmKyonkspkbk7m5wITYrB4JiIQ8ggjAZBQZcihRWIyaj75qnGpMwzEpy5lKSewEJjJgNxk7sKO5ODGcSuHDkKBrUCa5lVCM29RycjDwyK4kTKRDEFDEKBMFCacBGHkzc2EFbs2Pyi/PlZ9//5OTJhWboODp8RIFnMxWvBMQMcRVpnDO7CKphrw/YGI7n5/PTR6u233b3eb9fL4+eFURKu9xMttf97fVme7kPs22smpB7RLm5fktVTTmvtxst47AZW9h8MX/26ZP379+9e/OWaTpbTNypXtRp/bA8mkwms2HcBUCBAE+5eASIyZi8WBmzFXM4nJgE5OTOzkbERCByQAjGgYgkeKBgDmgmJiLm76A8JydxEpL8HVutmDOMmeBOIArMIRKYjcTFzYoPOpIxaWAxZzixkIgHRAkyQqZw41xLbWqVx2zklB2VObOzOEZyb3xxsahnjRUzM9PD2GAKZyrckHNMpUw4UFVgDQktKlgRLyj7Po87j2cx62zZdNtR7++Xp7GZzttde/PLq+w6m9Z326vF7IKZNdnd1+/azbg6PiulPLx/U8poYZGns8nLk1Mvo7pX1aQJ7Xbs+t2r7338weevLLfDehvMLBxU1gYKwci1KHI2LQdbHncQETHgFCQUwPwQjM7JRnBVeWB3rpgtuisZOVFBCQSAjVxctZAb4BCCkytcmMgMIIKQC0DqRuYEEQQXEnIiYiN1z3BWcjcDG6uRW+ISTaRORjWDqXIT9eRUhkLMPmtiNYlDTv0wBkbRbCaaXT2QOEQosmk1eDYvFU9mwUONgYtrGAUP67vF9ux4Ns2U2iF6HvZdq9UREB3x/n67WuXzFy9Lz2Z5eLhDL+3t2/XJer6oOQbPI4sN3fbuPnK9mMZ9X/o8MtV0tHg0Pzrt1Te33+TUhUPQcCGLISpQcjHNqg4iYoL5oejDmRlKIHYrJiJMpga4FS+VRSeOgeEFGdkzCUBEYBaIQAKxI3Ig95QKufvh0uzO39H4ERDgBSgAiTEO108nOMjcyQoIeqCUFCIuOjEpGiQqlyCRWaSBDsxFojdNxSGq5qH0jURzVXcvVnstYQKKg3kkL8pVrlApApFFCsbE0zCJVG+3N4vJEyLJ3i2OTqpqlXrmEp5/9lH3t7rZ7mgyhuWiPOxcI02jKFnX2SzOj858DKY87tqHr/vZxWPE0N1vc91YKdrQ5NMJtPvp//trVg0ONkdwcUZJqeT0nZs0CK7mMEAObO/Dz1c9Sk3kQGAhtxJY1IqBq+BVaIqNYtE9Mx9ObMGJCFVkUs5aDCSBUUDkBIK5kSuyVJQzOcgJXozMULEwyOBsBy8DJDIWOvwZ0KmLOO8okHkjUmgSOVYRlTSg6TDavhtiMIpqACjnDMsFWjmgLmOxxotzQ3ngug6MRmp4NZkvnz57Wmg0lygTw/a2a09L0NEm82OrabZq1Gzz7nY+WXIlo2htznXFXOWSl0eN1Zw2e+XUj7kUq2MNqUoZy6BUdg6psm2u70vXB4WLSzLVIbObuTF+K8wzYwbo4M8jcJgX4cpATAie4SAWVyI2hhWSGChiKsUUATByAzkBbspgdfZSlFyYI1FRZyGHRGiywnywXyIHm0JIDK6qxhCi4OwMV2YnZzcCG/sBvCIlUOeFqWVr3Gs00/W+e399Kc3IPNPYMGKElr6MmpL1RTt235kpcMyFImcrYTo7rVd77YqMJjxfnOSkEJpw02tJaeeb0ebHTHqymvdDC5t36/XR8XI6p2Ecue+lasY3W+QQZ9Pufu0+WE6ax3U1nB5PPVa7dj1kG3T97nI9T5pXq+DuyYvlBMDARN+hDobDIwB/9wQYMIGYuwBm2QJ7cSYuBIEYMhnHQCmiigSPvaoXtVQkRiN2zp6dKxJGGUGkTMVMonB2MEKBOyHagWzs31lIweFgjZkMMADF2F2J2bl2dyZXO7g1RikYqNcxjxgysX/dBk7z6izUOYBGlb5gGDvKfXQqamzoxKuiEXkhVR73ZTWbVCeah7bdVeLb/TZMARvVyDlUnNqbm+VHZ97I+fFJOh23gy0eHV/9zQ2lVq1ILWHBs6Pm5urBUKwj1dxP760NuphZ8hefvNrc3YuGb15vH+De7oOX5CZM7LDsRQ7KBmKGHmitxHLYbkU8H9LfvAgRe2QydnKCkwEkoGISIQeP0WCing3qxUg61WwOyUoS4GxUBBVQ1CRDKzctFpkVTkQQH78DR8zB2bMQshFxYChMiNkwKkEsFIriqSglx8QoQbVPpmzF+I3PJ6U6PRcPmXhMPAzjOLSFysFWhmy81/okNHd9yYxV4hnnxGHI/brbljFnTNgV2Yz7Upey+frhhqWazC9OcHkful63PTUROhGDeACbme+vr5qKqACZDcYVj8Y3t++M8MFHT+eL1cOb910/ouRgBhF3M3UXgBzZXcgFTMK/RRngTHbQO8NdQiUxkA5qRBwtDIKa1YsOlisBs7gnEresDiW4K6oY3ELKOSJmygeeC5HASyPRjYSRzUHEIuQMy8TuFvlgbGoUATLNQoEMHmDZwWYgcYWZ5uDiQcyKInqCI9mtvKks8LquqwBJOQ+99f1GLZlSgVY2NS7bsUgV1zvjN+sXjx85ti2zJkpl5+PQhPlU5ibThqva+82bd+GTz4btOKkrOWZY+ejVx+3Nu/X2rksaVLwdHjVeCu/Zemm82LOXz4uW1G2P5p/MT5b367v9Zj9Yx00TyMmVhAIIcDN4DOKluBAz3MFE5iwHXN9CDNErqyJZQhBiiAkqOvg3CLsLvChAjegIH8kpcGChQJysWLFECQenW1ECBdRDzrWw22EfNBRjDiBmADBzLuQBYq7kzg5mQXYJ4mzOmQs7S8XscGipvFLK5lKKm+Bui/n1/PQoRgqaJOvY5qSJjLQSN1bPxvCUPQZcbbeW7dXzsxmjHW5jCBVmbuaVbcZxP9ycSpDd/u7LN09ePR/acvrqkT90dDRfTF8u89OHt9d+dSWW5h88H3YPfvu+ifPq+MNU2s3t9UfPP5k+O7u6ufvmy5/5mI1iOzitTk8juflh1hOYsQi5g9gAh7GQE0eqAlMGBbZDmCkTi5iaCc2sJCqaSRU2FYlVNRQng0OTZ2gmsITa3XMqhuIQggk5EKJXmfRAu3Y+WKkQMxGZGQsRuWQShjM7QdxHkBBAUVVZiIparGMwMDETu1kIUS04eT0J0+nk2XmzXJgJClQT3Q9+e3073OaUSKFZtNLAzBTSEcQQlovqxYfH58uL6WKp2u0eutlySslsfzPLOrNw2T/QxQ+fnF+EulQZy6Olncw2v7qJNSoEOZ3fvL0OTkDqx9x2ozM4IM7Dzb7bvXlYb+7W+zQmT1z+PyVzBgRstXf0AAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from PIL import Image\n", "image = Image.open('bird_small.png')\n", "X = np.asarray(image) / 255 # range 0 - 1\n", "\n", "width, height, depth = X.shape\n", "X = X.reshape((width * height, depth))\n", "\n", "image" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise**: Try out different values for *K* and `max_iters`." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIAAAACACAIAAABMXPacAAAARElEQVR4nO3BAQEAAACAkP6v7ggKAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYwIAAAWMWdQAAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Run K-means\n", "K = 15\n", "colors, id2 = find_k_means(X, K, max_iters=10)\n", "idx = find_closest_centroids(X, colors)\n", "\n", "# Reconstruct image\n", "idx = np.array(idx, dtype=np.uint8)\n", "X_reconstructed = np.array(colors[idx, :] * 255, dtype=np.uint8).reshape((width, height, depth))\n", "compressed_image = Image.fromarray(X_reconstructed)\n", "compressed_image" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Principal Content Analysis\n", "---\n", "In this exercise, you will use principal component analysis (PCA) to perform dimensionality reduction. You will first experiment with an example 2D dataset to get intuition on how PCA works, and then use it on a bigger dataset of 5000 face image dataset." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Dimension reduction on example dataset\n", "\n", "Before using PCA it's important to use feature scaling to make sure every feature 'is of equal importance' when runnig PCA." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAEihJREFUeJzt3V2MXdV5xvHnYRjExE0yUplGeCg1V47UoODkCCmaCjUg4kRByKI3qZRKzY1vKkRUyZGpVLVc4QqpSq6qImhEFZoo5cNSSwtFGqQUJEDH2NThwxel0DA09UTFpSST4pq3Fz4D9nDOnL3P7I+19v7/JAvP+MyZNcf4OWu/611rOyIEAMjHJW0PAABQDsENAJkhuAEgMwQ3AGSG4AaAzBDcAJAZghsAMkNwA0BmCG4AyMyldTzpFVdcEXv27KnjqQGgk44dO/aziFgq8thagnvPnj0aDod1PDUAdJLtN4o+llIJAGSG4AaAzBDcAJAZghsAMkNwA0BmaukqAYBpjh5f0z1PnNJbZza0e3FBh/bv1YF9y20PKwsEN4DGHT2+pjsfOamNs+ckSWtnNnTnIyclifAugFIJgMbd88SpD0J708bZc7rniVMtjSgvBDeAxr11ZqPU53ExghtA43YvLpT6PC5GcANo3KH9e7UwP3fR5xbm53Ro/96WRpQXFicBNG5zAZKuktkQ3ABacWDfMkE9I0olAJAZghsAMkNwA0BmCG4AyAzBDQCZIbgBIDMENwBkhuAGgMxMDW7be22fuODXO7a/2cTgAAAfNXXnZEScknSdJNmek7Qm6dGaxwUAmKBsqeQmSf8aEW/UMRgAwHRlg/trkr4/7g9sH7Q9tD1cX1/f+cgAAGMVDm7bl0m6VdLfjvvziLg3IgYRMVhaWqpqfACALcrMuL8i6YWI+M+6BgMAmK5McP+uJpRJAADNKRTctndJulnSI/UOBwAwTaEbKUTEzyX9as1jAQAUwM5JAMgMwQ0AmSG4ASAzBDcAZIbgBoDMFOoqAYC+OHp8Tfc8cUpvndnQ7sUFHdq/Vwf2LVf+NTtBcAPAyNHja7rzkZPaOHtOkrR2ZkN3PnJSkiYG8Sxfs1OUSgBg5J4nTn0QwJs2zp7TPU+cqvRrdorgBoCRt85slPr8rF+zUwQ3AIzsXlwo9flZv2anCG4AGDm0f68W5ucu+tzC/JwO7d9b6dfsFIuTADCyuZhYpkNklq/ZKUdE5U86GAxiOBxW/rwA0FW2j0XEoMhjKZUAQGYolQA70PTGC0AiuIGZtbHxApAolQAza2PjBSAR3MDM2th4AUgENzCzNjZeABLBDcysjY0XgMTiJDCzNjZeABLBDezIgX3LBDUaR3AD6K1c+/AJbgC9lHMfPsENYFu5zkqn2a4PP/Wfj+AGMFHOs9Jpcu7Dpx0QwERd3h2acx8+wQ1gopxnpdPk3IdPcAOYKOdZ6TQH9i3r7tuu1fLigixpeXFBd992bRYlIGrcACY6tH/vRTVuKZ9ZaRG59uET3AAm6sPu0By7ZghuANvKdVZaRK5dMwQ3gFakMNPNtZeb4AbQuFRmurl2zdBVAnTA0eNrWjmyqmsOP6aVI6s6enyt7SFtK5X+8Fy7ZgoFt+1F2w/ZftX2K7a/UPfAABSzOXtdO7Oh0Iez15TDO5WZbq693EVn3N+R9HhEfFrSZyW9Ut+QAJSRyuy1jFRmurn2ck+tcdv+pKQbJP2+JEXEe5Leq3dYAIpKZfZaZrExpf7wHLtmisy4r5G0Lum7to/bvs/2rq0Psn3Q9tD2cH19vfKBAhgvhdlr2XJNrjPdVDgitn+APZD0rKSViHjO9nckvRMRfzzpawaDQQyHw2pHCmCsrR0a0vnZa5NBuHJkVWtjZvjLiwt65vCNjYwhd7aPRcSgyGOLtAO+KenNiHhu9PFDkg7POjhgp1Lo/01JCrsbUynX9MXU4I6In9r+ie29EXFK0k2SXq5/aMBHpdL/m5qyddqq3/x2Ly6MnXGn3laXq6IbcG6X9KDtyyS9Jukb9Q0JmCzXnW5t2hrSX/z0kh4+tlbpm19Ki419UCi4I+KEpEK1F6BOXJKXM+4K5cFn/11bV7Z2+uaXQrmmT9jyjqxwSV7OuCuUSe0IO33zy7GtLldseUdWct3p1pYyYcybXz4IbmSF/t9yJoWxt3zMm19eKJUgO1ySFzdp0fB3Pr+sp15dpx6dKYIb6DAWDbuJ4AY6rs9XKF3drEVwA5DUvZDr8mYtFicBZHmm9zQ5HndbFDNuoCe2m1F3cUdqlzdrMeMGemDajLqLIZfCcbd1IbiBHphWNuhiyHV5sxbBDfTAtBl1F0Ouy5u1qHEDPTDtjJeu9nvX3QrZVicOwQ30QJFjV/vc7z2LNtsNKZUAPdDlskFb2mw3ZMYNZKrsZToz6mq12YnDjBvIUBc3zOSmzU4cghuo2NHja1o5sqprDj+mlSOrtYRpl3cF5qLNThxKJUCFmlqwmvUyvWvnkbSpzU4cghuoUFNbx2e5hVuXD11qS1vrBpRKgAo1tWA1y2U65ZXuILiBCjW1YDVLe18XzyPpK0olQIWKbHSpStnL9FnKK0gTM26gQilvdOnieSR9xYwbqFiqG126eh5JHxHcQI+k+qaCciiVAEBmCG4AyAzBDQCZIbgBIDMENwBkhq4SYAcuPLRp8WPzipD+e+Nsba12HBLVnpRee4K7Ain9haI5Ww9tevsXZz/4szoOcOKQqPak9tpTKtkhDrTvr3GHNl2o6gOcOCSqPam99gT3DqX2F4rmFDmcqcoDnDgkqj2pvfaFgtv267ZP2j5he1j3oHKS2l8omlPkcKYqD3Bq81ZZfZfaa19mxv3FiLguIga1jSZDqf2FojnjDm26UNUHOHFIVHtSe+1ZnNyhJo/xRFq2HtpURVfJdgvdHBLVntRee0fE9AfZ/ybpbUkh6S8j4t4xjzko6aAkXX311Z9/4403Kh5quugqqVZfX8+tnQvS+UlAKsfCol62jxWtaBQN7uWIWLP9a5KelHR7RPxo0uMHg0EMh5TCUV6fw2vlyOrYGx0sLy7omcM3tjAiNKlMcBeqcUfE2ui/pyU9Kun62YcHTNbnLh0WulHU1OC2vcv2xzd/L+lLkn5c98DQT30OLxa6UVSRGfenJD1t+0VJz0t6LCIer3dY6Ks+h1dqnQtI19Sukoh4TdJnGxgLEtfEomGfu3RS61xAumgHRCFNndVQd3il3rHCrcVQBMGNQrZbNKw6aOoKr9QOCgJmxVklKKQLi4Z97lhBtxDcKKQLi4ZdePMBJIIbBXWh46ELbz6ARI0bBXWh4yHHjpXUF1PbwGtCcKOE3Dsecnvz6fti6riAltTr12RTobNKyuKsEmDn+nZ2ydb7d777y//T2fc/zKeF+TldPn/JRbeI29SF16TMWSXMuFEal6rN6NNi6nb379y0cfbcxFvFdfE12Q6LkyiFe2w2p0+LqdPu3zlNF1+T7RDcKIVe6OZ0oZOnqKIz5sWF+d68JtuhVIJSUrp873rJJrfF1J3Yvbgwtp5/oYX5Of3prb8pqR+vyXYIbpQy6R9Y05eqfem4yL2Tp6hxrZrzc9auyy4deyu4Prwm2yG4UUoqvdBNnp1SRtevAurSp6uLKhDcKCWVf2AplWw29eUqoC59ubqoAsGN0lL4B5ZKyeZCqV4FoHvoKkESjh5f08qRVV1z+DGtHFmd2l6YYsdFilcB6CZm3Gi9LjtLiSGVks2FUrwKQDcR3D2XQl121hJDCiWbC6WycIvuo1TScylsqOlKieHAvmXdfdu1Wl5ckHX+/Iy7b7s2qTcXdAMz7p5LITS7VGJI7SoA3cSMu+dSOA8jxYVGIGUEd8+lEJqUGIByKJX0XCrdGZQYgOIIbhCaQGYolQBAZghuAMgMwQ0AmaHGjdLa3iIP9B3BjVJS2CIP9B2lEpSSwhZ5oO8IbpSSwhZ5oO8IbpSSwhZ5oO8IbpSSwhZ5oO9YnEQpqWyRB/qscHDbnpM0lLQWEbfUNySkji3yQLvKzLjvkPSKpE/UNBZ0FH3fQLUK1bhtXyXpq5Luq3c46JrNvu+1MxsKfdj3Pe1mwAAmK7o4+W1J35L0/qQH2D5oe2h7uL6+XsngkD/6voHqTQ1u27dIOh0Rx7Z7XETcGxGDiBgsLS1VNkDkjb5voHpFZtwrkm61/bqkH0i60fb3ah0VOoO+b6B6U4M7Iu6MiKsiYo+kr0lajYiv1z4yVObo8TWtHFnVNYcf08qR1Ubry/R9A9Wjj7vj2j4UalLftyStHFml0wSYgSOi8icdDAYxHA4rf16Ut3JkVWtj6snLiwt65vCNLYzoo28m0vlZODcIRp/ZPhYRgyKPZct7x6W4OEinCbAzBHfHpbg4mOKbCZATgrvjql4crGKhM8U3EyAnBHfHHdi3rLtvu1bLiwuyzte2x9WSiwRyVbsg6TQBdobFSWy7WCh92BFyia1zY/5/mWWhk/NLgIuVWZykHRATFwvv+ruX9Muz73/wZ+NCW5qtNs0Jg8DsCG5MDN63f3G20NdTmwaaRY0bOwpeatNA8whuTFwsXFyYH/v4OXvbhc62tbnFH2gCpRJsuy09tx2ObW/xB5pAcEPS9ouFOXV/bLcrM+VxA2UQ3NhWbt0f7MpEH1DjRqewKxN9QHCjU9iViT6gVIJOmbTQmlO5B5iG4Ebn5FaXB8qiVAIAmSG4ASAzlEqQLU4YRF8R3MgSOyTRZ5RKkCXuW4k+I7iRJXZIos8olaByTdSedy8uaG1MSLNDEn3AjBuVquq+lNOwQxJ9RnCjUk3VnoveBBnoIkolBdB2VlyTtWd2SKKvmHFP0dSlf1dwOh9QP4J7CtrOyqH2DNSPUskUtJ2Vw+l8QP0I7iloOyuP2jNQL0olU3DpDyA1zLin4NIfQGoI7gK49AeQEkolAJAZghsAMjM1uG1fbvt52y/afsn2XU0MDAAwXpEa9/9KujEi3rU9L+lp2/8YEc/WPDYAwBhTgzsiQtK7ow/nR7+izkEBACYrVOO2PWf7hKTTkp6MiOfqHRYAYJJCwR0R5yLiOklXSbre9me2Psb2QdtD28P19fWqxwkAGCnVVRIRZyQ9JenLY/7s3ogYRMRgaWmpqvEBALaYWuO2vSTpbEScsb0g6WZJf1b1QDjzGgCKKdJVcqWkB2zP6fwM/YcR8fdVDmLzzOvN41M3z7yWRHgDwBZFukr+RdK+Ogex3ZnXBDcAXCyJnZOceQ0AxSUR3NzuCgCKSyK4OfMaAIpL4lhXzrwGgOKSCG6JM68BoKgkSiUAgOIIbgDIDMENAJkhuAEgMwQ3AGTG5++TUPGT2uuS3hh9eIWkn1X+TfLS99eAn7/fP7/Ea1Dk5/+NiCh0tGotwX3RN7CHETGo9Zskru+vAT9/v39+ideg6p+fUgkAZIbgBoDMNBHc9zbwPVLX99eAnx99fw0q/flrr3EDAKpFqQQAMlNbcNv+K9unbf+4ru+RMtu/bvsp2y/bfsn2HW2PqUm2L7f9vO0XRz//XW2PqS2252wft13pLf9yYPt12ydtn7A9bHs8TbO9aPsh26/afsX2Fyp53rpKJbZvkPSupL+OiM/U8k0SZvtKSVdGxAu2Py7pmKQDEfFyy0NrhG1L2hUR79qel/S0pDsi4tmWh9Y4238oaSDpExFxS9vjaZLt1yUNIqKXPdy2H5D0zxFxn+3LJH0sIs7s9Hlrm3FHxI8k/Vddz5+6iPiPiHhh9Pv/kfSKpN6cWxvnvTv6cH70q3cLKravkvRVSfe1PRY0y/YnJd0g6X5Jioj3qghtiRp3I2zv0fkbLj/X7kiaNSoRnJB0WtKTEdGrn3/k25K+Jen9tgfSkpD0T7aP2T7Y9mAado2kdUnfHZXK7rO9q4onJrhrZvtXJD0s6ZsR8U7b42lSRJyLiOskXSXpetu9KpnZvkXS6Yg41vZYWvRbEfE5SV+R9AejEmpfXCrpc5L+IiL2Sfq5pMNVPDHBXaNRbfdhSQ9GxCNtj6cto8vDpyR9ue2xNGxF0q2jOu8PJN1o+3vtDqlZEbE2+u9pSY9Kur7dETXqTUlvXnCl+ZDOB/mOEdw1GS3O3S/plYj487bH0zTbS7YXR79fkHSzpFfbHVWzIuLOiLgqIvZI+pqk1Yj4esvDaoztXaOFeY1KBF+S1Jsus4j4qaSf2N686/lNkippTqjtnpO2vy/ptyVdYftNSX8SEffX9f0StCLp9ySdHNV5JemPIuIfWhxTk66U9IDtOZ2fIPwwInrXDtdzn5L06Pk5jC6V9DcR8Xi7Q2rc7ZIeHHWUvCbpG1U8KTsnASAzlEoAIDMENwBkhuAGgMwQ3ACQGYIbADJDcANAZghuAMgMwQ0Amfl/lDB6+evK3WsAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "data = sio.loadmat(\"ex7data1.mat\")\n", "X = data[\"X\"]\n", "plt.plot(X[:, 0], X[:, 1], 'o')" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "def normalize_features(X):\n", " mu = np.mean(X, axis=0)\n", " sigma = np.std(X, axis=0)\n", " return (X-mu)/sigma, mu, sigma" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "X_norm, mu, sigma = normalize_features(X)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "PCA consists of two steps: first you compute the covariance matrix of the data given by:\n", "$$ \\Sigma = \\frac{1}{m}X^TX$$\n", "\n", ". Second you compute the [eigenvectors](https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors) of this matrix using numpy: `U, S, V = np.linalg.svd(Sigma)`." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "def pca(X):\n", " m = len(X)\n", " Sigma = (1/m) * X.T @ X\n", " U, S, _ = np.linalg.svd(Sigma)\n", " return U, S" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following code should return approximately $\\begin{bmatrix} -0.707 && -0.707\\end{bmatrix}$." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(-0.7071067811865474, -0.7071067811865474)" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "U, S = pca(X_norm)\n", "U[0, 0], U[1, 0]" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "def project_data(X, U, K):\n", " Ureduce = U[:, :K]\n", " z = X @ Ureduce\n", " return z" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following should return approximately $1.496$." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([1.49631261])" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Z = project_data(X_norm, U, K=1)\n", "Z[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise**: Implement `recover_data`." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "def recover_data(Z, U, K):\n", " return None" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "ename": "TypeError", "evalue": "'NoneType' object is not subscriptable", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mX_rec\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrecover_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mZ\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mU\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mK\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_rec\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX_rec\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'o'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtitle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Recoverd data\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mTypeError\u001b[0m: 'NoneType' object is not subscriptable" ] } ], "source": [ "X_rec = recover_data(Z, U, K=1)\n", "plt.plot(X_rec[0, :], X_rec[1, :], 'o')\n", "plt.title(\"Recoverd data\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### PCA on faces\n", "In this part of the exercise, you will run PCA on face images to see how it can be used in practice for dimension reduction." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = sio.loadmat(\"ex7faces.mat\")\n", "X = data[\"X\"]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def display_samples(X):\n", " rows = 4\n", " size = int(np.sqrt(len(X.T)))\n", " num_samples = rows ** 2\n", " samples = X[:num_samples]\n", " display_img = Image.new('RGB', (rows*32, rows*32))\n", "\n", " # loop over the images, turn them into a PIL image\n", " i = 0\n", " for col in range(rows):\n", " for row in range(rows):\n", " array = samples[i]\n", " array = ((array / max(array)) * 255).reshape((size, size)).transpose() # redistribute values\n", " img = Image.fromarray(array + 128)\n", " display_img.paste(img, (col*32, row*32))\n", " i += 1\n", "\n", " # present display_img\n", " plt.imshow(display_img, interpolation='nearest')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "display_samples(X)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "K = 100\n", "X_norm, mu, sigma = normalize_features(X)\n", "U, S = pca(X_norm)\n", "Z = project_data(X_norm, U, K)\n", "X_rec = recover_data(Z, U, K)\n", "display_samples(X_rec.T)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: ex7/ex7faces.mat ================================================ [File too large to display: 10.5 MB] ================================================ FILE: ex8/Anomaly Detection and Recommender Systems (Exercises).ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Anomaly Detection and Recommender Systems\n", "\n", "Stanford CS229 - Machine Learning by Andrew Ng. Programming exercise 8.\n", "\n", "Please check out [the repository on GitHub](https://github.com/rickwierenga/CS229-Python/). If you spot any mistakes or inconcistencies, please create an issue. For questions you can find me on Twitter: [@rickwierenga](https://twitter.com/rickwierenga). Starring the project on GitHub means a ton to me!" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pylab as plt\n", "import scipy.io as sio\n", "from scipy.optimize import minimize\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Anomaly detection\n", "\n", "$x \\in \\mathbb(R)$ is a distributed Gaussian with mean $\\mu$ and variance $\\sigma$.\n", "\n", "$x \\sim \\mathcal{N}(\\mu, \\sigma^2)$ ($\\sim$: distributed as)\n", "\n", "We fit the parameters\n", "\n", "$\\mu = \\frac{1}{m}\\displaystyle\\sum_{i = 1}^{m}x^{(i)}$, $\\sigma = \\frac{1}{m}\\displaystyle\\sum_{i = 1}^m(x^{(i)} - \\mu)$.\n", "\n", "Then we use the following function:\n", "\n", "$$p(x;\\mu, \\sigma^2) = \\displaystyle\\prod_{j = 1}^{n}p(x_j; \\mu_j, \\sigma^2)$$\n", "\n", "Anomaly if $p(x) < \\epsilon$.\n", "\n", "### Algorithm\n", "1. Choose features $x_j$.\n", "2. Fit parameters $\\mu_1, ..., \\sigma_n$, $\\mu_1, ..., \\sigma_n$.\n", "3. Given new example $x$, compute $p(x)$.\n", "\n", "### Anomaly Detection vs Supervised Learning\n", "Use *anomaly detection*:\n", "* Very small number of positive examples ($y = 1$).\n", "* Many different \"types\" of anomaly.\n", "* Future anomalies may look nothing like any of the anomalous examples we've seen so far.\n", "\n", "Use *supervised learning*:\n", "* Large number of positive and negative examples.\n", "* Enough positive examples for the algorithm to get a sense of what positive examples look like.\n", "* Future examples look like examples from the dataset.\n", "\n", "### Tips\n", "* Examine the examples the model got wrong by hand and try to come up with new features. \n", "* Have a look at the distribution. If it doesn't look like a bell curve, apply log or square root function to each traning example.\n", "\n", "### Multivariate Gaussian Distribution\n", "![mgd](images/mgd.png)\n", "\n", "Instead of modelling $p(x_1), p(x_2), ...$ seperately, we model $p(x)$ in one go. Parameters $\\mu \\in \\mathbb{R}^n$, $\\Sigma \\in \\mathbb{R}^{n \\times n}.$\n", "\n", "#### Original vs Multivariate\n", "With the original model you have to manually create featuers to capture anomalies where $x_1$, $x_2$ take unusual combinations of values where the multivariate model automatically captures the correlations. The original model is computationaly cheaper and works OK when $m$ is small (the multivariate model must have $m > n$." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3XmYXHWd7/H3N91JmjRJCKQNSSCGmQkKyEiSTogKXBoIQsgE9WrEkQZBp6Wvjgt6echggBv0agLBx2UmDCqLDIvhQhSVKFGai8sF02HfN1ESgYRFiIpgwvf+8Ts/6lR1VfXp7lp6+bye5zxVZ/91pXK+9dvN3REREenNqHonQEREhgYFDBERyUQBQ0REMlHAEBGRTBQwREQkEwUMERHJRAFDREQyUcAQEZFMFDBERCSTxmpd2Mz2Br4LTAEcuNjdv2Zm5wL/AmxLDv03d7+xyPnHAF8DGoBvu/tXervn5MmTfebMmZX5A0RERoBNmzY95+4tWY6tWsAAdgCfc/c7zGw8sMnMNiT7vuruF5Q60cwagH8HFgKbgY1mdoO7P1DuhjNnzqS7u7tCyRcRGf7M7HdZj61akZS7P+3udyTvtwMPAtMznj4feMzdn3D314BrgOOrk1IREcmiJnUYZjYTmA3cnmz6pJndY2aXmNmkIqdMB55KrW8me7AREZEqqHrAMLNdgeuAz7j7y8Aa4O+Bg4CngdUDvH6HmXWbWfe2bdt6P0FERPqlqgHDzEYTgsWV7n49gLs/6+473f114FuE4qdCW4C9U+t7Jdt6cPeL3b3V3VtbWjLV24iISD9ULWCYmQHfAR509wtT26emDnsvcF+R0zcCs8xsHzMbA5wA3FCttIqISO+qmcN4F9AOHGFmdyXLImCVmd1rZvcAbcBnAcxsmpndCODuO4BPAj8lVJavdff7K53AVaugqyt/W1dX2C4iIvmq1qzW3X8JWJFdPfpcJMf/AViUWr+x1LGVMm8eLF0Ka9dCW1sIFnFdRETyVbMfxqDX1haCw9Kl0NkJa9bkgoeIiOQb8UODtLWFYHHeeeFVwUJEpLgRHzC6ukLOYvny8FpYpyEiIsGIDhjpOosVK3LFUwoaIiI9jeiAsXFjfp1FrNPYuLG+6RIRGYzM3eudhoppbW11DT4oIpKdmW1y99Ysx47oHIaIiGSngCEiIpkoYIiISCYKGCIikokChoiIZKKAISIimShgiIhIJgoYIiKSiQKGiIhkooAhIiKZVHOK1r3NrMvMHjCz+83s08n2883sITO7x8zWmdluJc5/MpmZ7y4z03gfIiJ1Vs0cxg7gc+6+P7AA+ISZ7Q9sAN7m7v8IPAIsK3ONNnc/KOs4JyIiUj1VCxju/rS735G8306Ym3u6u9+UzNkNcBuwV7XSICIilVOTOgwzmwnMBm4v2HUqsL7EaQ7cZGabzKyjzLU7zKzbzLq3bdtWieSKiEgRVQ8YZrYrcB3wGXd/ObX9LEKx1ZUlTj3E3ecAxxKKsw4rdpC7X+zure7e2tLSUuHUi4hIVNWAYWajCcHiSne/PrX9I8Bi4MNeYkIOd9+SvG4F1gHzq5lWEREpr5qtpAz4DvCgu1+Y2n4McAawxN3/UuLcZjMbH98DRwP3VSutIiLSu2rmMN4FtANHJE1j7zKzRcA3gfHAhmTbRQBmNs3MbkzOnQL80szuBn4D/Njdf1LFtIqISC8aq3Vhd/8lYEV23VhkG+7+B2BR8v4J4O3VSpuIiPSdenqLiEgmChgiIpKJAoaIiGSigCEiIpkoYIiISCYKGCIikokChoiIZKKAISIimShgiIhIJgoYIiKSiQKGiIhkooAhIiKZKGCIiEgmChgiIpKJAoaIiGRSzRn39jazLjN7wMzuN7NPJ9t3N7MNZvZo8jqpxPknJ8c8amYnVyudIiKSTTVzGDuAz7n7/sAC4BNmtj9wJvBzd58F/DxZz2NmuwPnAAcT5vI+p1RgERGR2qhawHD3p939juT9duBBYDpwPHB5ctjlwHuKnP5uYIO7v+DuLwIbgGOqlVYREeldTeowzGwmMBu4HZji7k8nu54hzN9daDrwVGp9c7JNRETqpOoBw8x2Ba4DPuPuL6f3ubsDPsDrd5hZt5l1b9u2bSCXEhGRMqoaMMxsNCFYXOnu1yebnzWzqcn+qcDWIqduAfZOre+VbOvB3S9291Z3b21paalc4kVEJE81W0kZ8B3gQXe/MLXrBiC2ejoZ+EGR038KHG1mk5LK7qOTbSIiUifVzGG8C2gHjjCzu5JlEfAVYKGZPQoclaxjZq1m9m0Ad38BOA/YmCwrkm0iIlInFqoRhofW1lbv7u6udzJERIYMM9vk7q1ZjlVPbxERyUQBQ0REMlHAEBGRTBQwREQkEwUMERHJRAFDREQyaSy308yagMXAocA04BXgPuDH7n5/9ZMnIiKDRcmAYWb/ixAsbiEMGrgVaAL2Bb6SBJPPufs9NUiniIjUWbkcxm/c/ZwS+y40szcBM6qQJhERGYRKBgx3/3HhNjMbBezq7i+7+1aKDxwoIiLDUK+V3mZ2lZlNMLNmQv3FA2b2P6ufNBERGUyytJLaP5nH4j3AemAfwqCCIiIygmQJGKOTeS3eA9zg7n9jgJMeicjItGoVdHXlb+vqCttl8MsSMP4TeBJoBm41szcDL5c9Q0TeoIdkzrx5sHRp7vPo6grr8+bVN12STcmAYWbvMDNz96+7+3R3X5RMqfp7oK12SRQZ2vSQzGlrg7Vrw99/9tnhde3asF0Gv3I5jJOATWZ2jZl9xMz2hDAPt7vvqE3yRIY+PSTztbVBZyecd154Hamfw1BUrlltJ4CZvRU4FrjMzCYCXcBPgF+5+85S55vZJYSOf1vd/W3Jtu8Bb0kO2Q34o7sfVOTcJ4HtwE5gR9bJPUQGq/RDcvnykf2Q7OqCNWvC57BmTfgsRvLnMZT0Wofh7g+5+1fd/RjgCOCXwAcIvb/LuQw4puBaH3T3g5IgcR1wfZnz25JjFSxkyCt8SBbWaYwUsThu7VpYsSKX8xqpn8dQU3YsqcjM5gCHEFpH/crd/7W3c9z9VjObWeJ6BiwlBCCRYS39kIy/pkdqsdTGjfl/dyyu27hx5H0WQ1GWjntnA5cDewCTgUvN7AsDvO+hwLPu/miJ/Q7cZGabzKyjl/R1mFm3mXVv27ZtgMkSqZzYOir9kEyvb9xY7xTW3hln9AwMbW1huwx+WZrVfhiY5+7nJGNLLWDgHfc+BFxdZv8h7j6HUHfyCTM7rNSB7n6xu7e6e2tLS8sAkyVSObF11Lx5uWCRXs/6kFSzXBkssgSMPxBGqY3GAlv6e0MzawTeB3yv1DHuviV53QqsA+b3934i9VKp1lFqliuDRbl+GN8ws68DLwH3m9llZnYpYTypPw7gnkcBD7n75hL3bTaz8fE9cHRyT5GaG+iv+0o0IVWzXBksyuUwuoFNhF/4/0ZoTnsLcBbwg94ubGZXA/8PeIuZbTazjya7TqCgOMrMppnZjcnqFOCXZnY38BvCZE0/yfwXifRBbwGhv7/u43XTraO+9jU4+OD+BaBigUdFVVJz7j5slrlz57pIX9x8s/vkyeG12Hp62/LlPfeV0tHh3tzsPmFCOP7mm8P66NHuEyeWv1+hlSvdV6/OT8Pq1eEevaVdpDdAt2d8xvZ+QOh8dyfwAmEMqe3Ay1lvUMtFAUP6I0tAWL48/G9Zvjz7NZuaQpA48kj3ceNCoIgP+okTw/YsD/jVq93NwuvKle6dnbn1m2/u27VEClU6YDwG/CNgWS9ar0UBQ/qrXEDIElBWruy5ffXqkKOAsKxenbveLrvk36/w/JirWLky937iRPfZs0Ow6OzMnRPv0d6en+aOjnCMSDmVDhhdwKisF6znooAh/VEuIPRWZLVyZXgwxyKjWPx03HEhhzFmTPhfNmZMeOAvXx6KqWKuoKkpl1OIRU3z57sffHAuF+EeXhsbc4Ehpnfs2Nx94vGrV4ft6aKvYgHt5psVUKTyAWMeYeyoZcDpccl6g1ouChjSV1kCQnywpo+J67F4acKE8KCeMCE8rCH30F6+PLw2NITt48blAks8trMznA/uo0aF1wULQlra28P66NHhWs3N7gsX5t+jszM/N9PUFNIW07x6dbhvzLWovmPoq9SPgL4EjCz9ML4E/IXQF2N8ahEZ8soNVbFqVa6TXWwtdeGFYV9cP+EEWLcOzOALX4A//QlefRX22AMaG8O+tjaYOhV27gzHRXfeGdZHjw4tqZYtg7Fj4fXXw/5774X99oMrrgjr7353GH/ppJNgwwaYNSsc/8//HM7fa6/ctUeNgoYGWLw4pPnss+Goo+Dzn4f77gtpX7ZsZPY2Hy7q0j+nt4gC3Jc1+tR7UQ5DKqnwV3jMASxc2DNXcuSRuV/3jY3h1378RT9hQvjFD6GoaeLEkDOIRUix/iSeG3MdMUdi5m/UgcyfH3IYnZ3h+uncRzxnl11y95s1K5y/cGHuNV2spRzG0NafFnyFqHCR1Crg6KwXrOeigCGVVvgfMv3AjfvTAWHs2PBAHzcu9+CfNSvXQmrcOPd99gnb58wJ548blwsOMSAtWZJbj4Fm8uT8uo10oEkvc+fmrx94YO66Zu6HHppfPyJDW19b8BWqdMDYDrwOvIKa1coIFP9Dxl/l7e3hgdveHoLFmDHhF35sMtvcnKuH2HXX8LrHHmH/tGlhffz4cE5DQ7jWggW5fhqxHiMdRGLleGwxla7rAPeWlnBMYa5kwoRcEFEOY/gZdDmMobQoYEh/lKs87OgID90YJGKLptmzw/+e6dPdFy/OtUjq7Mw9yONDu3Ap3H/AASFYxBZO6aAxenQup3LccSFtxx2Xu9bMmeGc5ub8ABOvH5vgNjeHXEsMdLFFllpJDV1ZOp1mUZGAAcwseyIYsFfWG9ViUcCQvkr3oo6tiuIDOL7GFkfHHReKj2IP7lj0c/DBuZ7c8WEdm9PGeon0g7yhoefDfcmS/Id+DCbt7bliq6am/LqSAw8Mxy1ZEoJLsQA1d244f/HicI3OTvdjj1UrqeGgHq2kygWEawmz4p0EHAC8CZhBmPToPODXwMKsN6rFooAhpaT/c8X3sVns5MnhQVrYnyHmLmJAiRXMo0aFfYsX53IDsegp1lmkH+zFlrFjc8GhMKDEYqRYuR7TEnM1cWluzq+viDmSdA4jNuONfUUmTgzv3dUPQ4KKFUkB+xOa1d4CPEwYIuQq4ESgKetNarUoYEgp6V/UcTiNOM5THHojVg6PGZMrE465jhgAYh3E9OnlA8KYMT0rn9NLYZAoLK6aOzcEhHe+098otjILdSHpuovCezY05K6dznG0t+daZ82fX+9/DRlMVIchUkS6gjD2ti5s/XTooeEXeWx1EoPLqFHhQVysrqAvy5QpvR8Tm9zuu6+/kZOAEDwmT3bfb7+e58yYEV5j5XvMDaWDUGOjWkdJT30JGFk67okMC+khwj/9afjUp8L72bPhZz+D9na4555w7LhxYTjyO+8Mj90xY2DHDvjzn0MHvKzGjctf37q193NefRUmTYJHHoHx48M9x4+HX/86dAB88MGe5/z+96ED4E9/Gv6e9evD39PQkOsIuGMHnHYanH56WNfw6NJXChgy6FXqwfbxj4cgEOemOP98mDs39Jo+7TQ45RT429/gtdfg5JNDL+7ly+Gcc+ADH+h5vVFF/vc0Nuav/+Uv+ce59zwn3fsboKkJXngBmpth+/YQrLZvh912C72/0970ptz7mTPD69Kl8L73wUEHhWul73PppbnPUjP5SZ9lzYr0dQEuAbaS6ikOnEuY3vWuZFlU4txjCHUmjwFnZr2niqSGp0o0H4wd7GIHuokTc53tYlHP4sVh2zvfGYp2Yn3HggX+RmV2uhgonp9lKdXEtnCJldaxGGry5OL7Yz1HusgpVsbHdDc15Y7fc8/c+bHSO/1ZDqQdvwxtVLjj3s+zbCtyzGHAnCIB4/O9nNcAPA78HTAGuBvYP8sfo4AxfA30wZZuGRWH1Jg4MQSD2KEtNjuNfRfSdRv77de3ADGQJbaSinUlM2eWPrahITf4YFNTriLfPRfoYmur+LctXpz/2Qy0p7AMbX0JGAUZ6BwzawLGAZPNbBKh3wXABGB6hpzLrWY2s7fjipgPPObuTyTpuAY4HnigH9eSYSJd/7B8ed/nsz7jjNz7z30ud51ddw31Axs2wKGHwrXXhmlUv/WtUJyzYQMsXAhPPQV//SvsvXeoR3jiCXjuufx7TJoEL744sL9z3Dh4+eVQHwGhOOt3vwsDC27enNsGIWSsWgUPPwxXXQVf+lJ4HwcUfPHFkPYNG8LfumIF/MM/hPqaKD2F7Jo14XPVXOFSUqlIAnwa+C3wavIal7uBT2aJRsBMeuYwngTuIRRZTSpyzvuBb6fW24FvZrmfchjD10ByGLHHdsxhxD4VTU25HEZ6vKUJE3oOyzFqVK7oZ489wjm77165XMXcubl7xqKlOHBhYTPauB57fsfitnT/ith/o9xnVqmewjK0UeEiqX/NerEi5xYGjCmEIqdRhP4dlxQ5p08BA+gAuoHuGTNmVOPzlDrry4Ot1Mx3TU253tKdnfkjxsa+DjFoxCaq5eodpk0L9QzFjhlIs9sYEBYvzhUjpesvYhEZuL/1rT0/k/b2/F7q6Vn+CocD0aRK4l75gHFSsSXTxQsCRpZ9wDuAn6bWlwHLstxPOYzhKcuDLV1Hke6gF3tyx6CRrlTu6MiNChsfxLFDXlNT/sO51IO90vUase9FY2P+pEjpIUNGjw5LDCoxCMS+F+3t+UOeFE6ipByEpFU6YHwjtXwLeAL4P5ku3jOHMTX1/rPANUXOaUzusQ+5Su8DstxPAWNoG8gv3mI9uXfZJX+a0vRw4LE3d6zonjAh9OaOU57GCuNiS7qXd9bWT1mWOMZUDGLpoqh0C62xY3PBIFZkx6HLC0ehVSso6U1FA0aPE2A34CcZjrsaeBr4G7AZ+ChwBXBvUodxQwwgwDTgxtS5i4BHCK2lzsqaNgWMoW2gZerph+Muu4Rv9/LluV/bMYjE0V/jr/HY3DbdiqqvwWD06L7nNsxyOYdYHxJHpy0cIyrmOmLdSxxBd9y44nN0pD83tYKScqodMEYDD/f1vFosChhD30B/EceH47hxuWvEX+GxOCZdL9DQEHIb6bmuOzqy1UOkR6SFEHT6k7MonKsi3a8ifVx6EMEY1NrbQ4CZM6dnsE3P3a0chpRS6SKpHya5gRuAHyfFRV/JeoNaLgoYw0N/fxHH1kLjxuX6I8RhxxcsCA/0WM6/ZEl4yMYHf3Nzrk6gcLa7vgaPUoMDlrtmYT+PwtzNqFEh0HV0hL9j3LhcMVVzc8/6m/h5qBWU9KYvASPL0CAXAKuT5X8Dh7n7mRnOE+mzwn4BhUOClDtv6dIwnMePfgTf/35YB/jhD+G97w1jR11xReibcMst8Pjj4T4Q+lj84hdhHKYbboB9981dO/xuykkP9dHUFIYXee213PbXXy8+bEjazp35w3o8+GBIQ7F7Tp4M06fDRReFsaHcwz0efzzcv6EhHNfWBmvX5vphbNwY1mO/isL9In2WJaoAewJLgH8C9swajWq9KIcxtA3kF3FvFeaFI9XGSZBixXe6SCjOub1wYbbcQixW6mtRVCzCivUtxZapU8NrbA5c2LQ2jqirprDSX1S4SOpjwO+By4DLCR3vTs16g1ouChhDW7X6BRQGnvS0qxBaR8ViqNgXY+7c/HGbilVYp/c3NITAUyrAlNp+wAGl7xED15vfnKvoPvRQf6MITPUSUgmVDhgPA3uk1vdAld4yhJTqzBcrxmMOI+YS0rmFhoaeD/uYI2hpCYEmtrqaPbv4xEijRuXGh4qvxeop0vUacV9sfbV6dW7GvcbG/FZdChoyEH0JGFnqMJ4HtqfWtyfbRIaEM87IHx+pqwu+/OVQ17FiBZx4YhhCfNMmOPBAuOOOMGbUwQeHOoJRo0K9R/TKK2HbSy+FeSfMQj3CXXeFOScA5szJDS3uHsaH2nPP8BrrHNL22SfUa+y2W/72MWPCGFFdXeH6s2eHYc/PPjtXR6N6CamVLAHjMeB2MzvXzM4BbgMeMbPTzez06iZPpPIKK4NPOCE83GfPDvNNnHhiCAotLeHhvnIl/OpXsGRJCA7Tp4dK5507w/LXv4Z1Tyqqm5rgwx8O544aFbZPmgTPPBPWX389BKB4/JIlYT6Ozs4QhGLFdnt7uM6KFWEAwQsuCMFs3boQ8JYtC39LW1v+4Ioi1ZIlYDwOfB+I7TZ+QBiEcHyyiFRErWaAK8xxbNwYfrE/9VRoNbV+fXgY79wZWludfnoY3faHPwyjvT7/fHi479yZaxHlnpsIaffdw/XcYZddYNasMHLsLruE4+fMCSPGTpsWgsuee4Y0/cd/hFyNWS4dJ5wAH/xgmNApzpQXWzvt2KFAITWWtexqKCyqwxjaKtlvoC8V6Fnumx6raZddQp1Cuu9FrKBO10ssWJA7LtaDxGFFDj7Yff78/PGe4nhXhaPOqn5CqokKV3rvC1wM3ATcHJesN6jlooAx9FWqZ3J/R7iNQ6GnR3W9+ebQqa+5OaQrzoYXW0nFYJCuLE9XWsdmsWPH5gYVnDgx17R39epwzziibjqNajIr1VbpgHE30EmY2GhuXLLeoJaLAsbwUKmxj/oTfOLAhYU9xc1yQ4UvXpwLDAsX5gJIY2Nu7ox066c99wyBIE4Pmx6CJI5vle6dLlJLlQ4Ym7JerN6LAsbQV8mxj9LFSDH4ZPnFnh7tdty43OiwUUdHCBJHHpmbcyKObhsHM0yPaAvhGsX+lvQIuhocUOqhIgED2D1ZzgX+BzA1tW33rDeo5aKAMbRVeuyj1avzh/wunB+inFIP8lJp7OjIBafYEzvObRGXzs78YFUYmJTDkHqoVMD4LWGgwd8WWZ7IeoNaLgoYQ1sle3oXTlPa3p5frNTbuaUe5KXSGCdqikEjzuLX1BSuEyvI0zPgFRZ9pddFaqUvAcPC8cNDa2urd3d31zsZMgisWgXz5oUms/fdFwYdbG+Ht70tt71Yk9SurjBQoXtoUgv56+nmuOlzli7NdaBrbISzzgr7xoyB448P+xobQ7+K//zPkL7HHw/NZuM1u7rgmmvg7/9ezWWldsxsk7u3Zjm2McPF3ldk80vAve6+tcx5lwCLga3u/rZk2/mEAQxfI/TvOMXd/1jk3CcJPcp3Ajuy/jEiUXzg3nkn/Nd/hWCxfj0cdFDu4V7Mxo3wlreEvg/xQb5uHVx4IZx/fvGAke4IGPc//HB4nTIl9OFYvjzsiz2yiwWE9Pkig1JvWRDCHBgvANcly/OEJraPAu1lzjsMmEP+FK1HA43J+5XAyhLnPglMzppNiouKpCStv8VSlapL0eRFMhTQhyKpXnMYhFzIfu7+LICZTQG+CxwM3EqYdrVYILrVzGYWbLsptXob8P4M9xfpl/Qv/z/+MfzSb2/PjfdU7rxly0JOpLMzzMuRHoYji3QxVcw5pNdFhqIsQ4PsHYNFYmuy7QXCfN39dSqwvsQ+B24ys01m1jGAe8gIFocASU/KtH59qMMoZ968MFbTsceGIHPssWG9t/PSNHmRDEdZchi3mNmPgGuT9f+ebGsGetQ/ZGFmZwE7gCtLHHKIu28xszcBG8zsIXe/tcS1OoAOgBkzZvQnOTKM9eeXfltbyFF8/vNw6KGhDuSCC4ofHyvXC0fDjdcpvK5yFzKUZclhfIIwedJByfJd4BPu/md37/PX38w+QqgM/3BSftaDu29JXrcC6wi9zIty94vdvdXdW1taWvqaHBnm+vNLPw5/fuKJYdrWE08M68Wmi503LwSguC8GqL7kRkSGiqo2q03qMH7kuVZSxwAXAv/N3beVOKcZGOXu25P3G4AV7v6T3u6nZrVSCatWhSawX/5yfh1GqdFhY5CIx6qeQoaSSjer3U5uaPMxwGjgz+4+oZfzrgYOByab2WbgHGAZMJZQzARwm7ufZmbTgG+7+yJgCrAu2d8IXJUlWIhUSsw1FCvGKqatLQSLdPNZkeGo14Dh7m/MeWHhKX48sCDDeR8qsvk7JY79A7Aoef8E8Pberi9SLeWKsUp13IuV6mvWqK5Chq9+FUmZ2Z3uPrsK6RkQFUlJrRVWqheuiwx21ezpPQpoBf7az7SJDCt9zY2IDGW95jDM7NLU6g5CL+xveZlhQepFOQwRkb6paA7D3U8ZeJJERGSo67UfhpntZWbrzGxrslxnZnvVInEiIjJ4ZOm4dylwAzAtWX6YbBMRkREkS8BocfdL3X1HslwGqEu1iMgIkyVgPG9mJ5pZQ7KcSBjiXERERpAsAeNUYCnwDPA0YUhyVYSLiIwwZVtJmVkD8D53X1Kj9IiIyCBVNofh7juBYkN8iIjICJNlPoxfmdk3ge8Bf44b3f2OqqVKREQGnSwB46DkdUVqmwNHVD45IiJ9V2oiq40biw9JL/2Tpae3RsQRkUGtcEj69CCQUjlZBh8cS5iWdWb6eHdfUeocEZFaioM+aiKr6spSJPUD4CVgE/BqdZMjItI/msiq+rIEjL3c/Zj+XNzMLiHM3701NU3r7oQK9JmEkW+XuvuLRc49GfhCsvpFd7+8P2kQkZFBE1lVX5aOe782swP7ef3LgMJgcybwc3efBfw8Wc+TBJVzgIOB+cA5Zjapn2kQkWEuXWexYkWueKqrq94pG15KBgwzu8/M7gEOAe4ws4fN7B4zuzfZ3it3vxV4oWDz8UDMLVwOvKfIqe8GNrj7C0nuYwM9A4+ICFB+IiupnHJFUtPJNamtpCnu/nTy/hlgSol7P5Va35xsExHpoVjTWRVJVV65gPFbd/9dNW/u7m5mfZ9UPMXMOoAOgBkzZlQkXSIi0lO5gPEmMzu91E53v7Cf93zWzKa6+9NmNhUoNtXrFuDw1PpewC0l0nExcDGEKVr7mSYREelFuUrvBmBXYHyJpb9uAE5O3p9MaLZb6KfA0WY2KansPjrZJiIidVIuh/H0QDvnmdnVhJzCZDPbTGj59BVgrZl9FPgdYeh0zKwVOM3dP+buL5jZeUCsslrh7oWV5yJVpeEmRPKVCxjGrwUoAAALvUlEQVQ20Iu7e6mRbo8scmw38LHU+iXAJQNNg0h/abgJkXzlAkaPh7rISKLhJkTylazDUBGQSP5wE52dChYysmXp6S0yYhUON6GewzKSKWCIlKDhJkTyKWCIlKDhJkTymfvw6evW2trq3d3d9U6GiMiQYWab3L01y7HKYYiISCYKGCIikokChoiIZKKAISIimShgiIhIJgoYIiKSiQKGiIhkooAhIiKZKGCIiEgmNQ8YZvYWM7srtbxsZp8pOOZwM3spdczZtU6niIjkKzcfRlW4+8PAQQBm1kCYv3tdkUN/4e6La5k2EREprd5FUkcCj7v77+qcDhER6UW9A8YJwNUl9r3DzO42s/VmdkAtEyUiIj3VLWCY2RhgCXBtkd13AG9297cD3wC+X+Y6HWbWbWbd27Ztq05iRUSkrjmMY4E73P3Zwh3u/rK7/yl5fyMw2swmF7uIu1/s7q3u3trS0lLdFIuIjGD1DBgfokRxlJntaWaWvJ9PSOfzNUybiIgUqHkrKQAzawYWAh9PbTsNwN0vAt4PdJrZDuAV4AQfTjM9iYgMQXUJGO7+Z2CPgm0Xpd5/E/hmrdMlIiKl1buVlIiIDBEKGCIikokChoiIZKKAISIimShgiIhIJgoYIiKSiQKGiIhkooAhIiKZKGCIiEgmChgiIpKJAoaIiGSigCEiIpkoYIiISCYKGCIikokChoiIZKKAISIimdQtYJjZk2Z2r5ndZWbdRfabmX3dzB4zs3vMbE490ikiIkFdZtxLaXP350rsOxaYlSwHA2uSVxERqYPBXCR1PPBdD24DdjOzqfVOlIjISFXPgOHATWa2ycw6iuyfDjyVWt+cbMtjZh1m1m1m3du2batSUkVEpJ4B4xB3n0MoevqEmR3Wn4u4+8Xu3ururS0tLZVNoYiIvKFuAcPdtySvW4F1wPyCQ7YAe6fW90q2iYhIHdQlYJhZs5mNj++Bo4H7Cg67ATgpaS21AHjJ3Z+ucVJFRCRRr1ZSU4B1ZhbTcJW7/8TMTgNw94uAG4FFwGPAX4BT6pRWERGhTgHD3Z8A3l5k+0Wp9w58opbpEhGR0gZzs1oRkSFn1Sro6srf1tUVtg91ChgiIhU0bx4sXZoLGl1dYX3evPqmqxLq3dNbRGRYaWuDtWtDkOjshDVrwnpbW71TNnDKYYiIVFhbWwgW550XXodDsAAFDBGRiuvqCjmL5cvDa2GdxlClgCEiUkGxzmLtWlixIlc8NRyChgKGiEgFbdyYX2cR6zQ2bqxvuirBQneH4aG1tdW7u3tMrSEiIiWY2SZ3b81yrHIYIiKSiQKGiIhkooAhIiKZKGCIiEgmChgiIpLJsGolZWbbgN9V6HKTgecqdK1qURorQ2msjMGexsGePqhPGt/s7pmmKx1WAaOSzKw7a1OzelEaK0NprIzBnsbBnj4Y/GlUkZSIiGSigCEiIpkoYJR2cb0TkIHSWBlKY2UM9jQO9vTBIE+j6jBERCQT5TBERCSTER8wzOxJM7vXzO4ysx4jF1rwdTN7zMzuMbM5NU7fW5K0xeVlM/tMwTGHm9lLqWPOrkG6LjGzrWZ2X2rb7ma2wcweTV4nlTj35OSYR83s5Bqn8Xwzeyj5t1xnZruVOLfs96LKaTzXzLak/j0XlTj3GDN7OPlunlnD9H0vlbYnzeyuEufW6jPc28y6zOwBM7vfzD6dbB8038cyaRxU38deufuIXoAngcll9i8C1gMGLABur2NaG4BnCO2m09sPB35U47QcBswB7kttWwWcmbw/E1hZ5LzdgSeS10nJ+0k1TOPRQGPyfmWxNGb5XlQ5jecCn8/wXXgc+DtgDHA3sH8t0lewfzVwdp0/w6nAnOT9eOARYP/B9H0sk8ZB9X3sbRnxOYwMjge+68FtwG5mNrVOaTkSeNzdK9U5sd/c/VbghYLNxwOXJ+8vB95T5NR3Axvc/QV3fxHYABxTqzS6+03uviNZvQ3Yqxr3zqrE55jFfOAxd3/C3V8DriF8/hVVLn1mZsBS4OpK37cv3P1pd78jeb8deBCYziD6PpZK42D7PvZGAQMcuMnMNplZR5H904GnUuubk231cAKl/3O+w8zuNrP1ZnZALROVMsXdn07ePwNMKXLMYPo8TyXkHovp7XtRbZ9MiikuKVGUMhg+x0OBZ9390RL7a/4ZmtlMYDZwO4P0+1iQxrTB/H0EoLFeNx5EDnH3LWb2JmCDmT2U/KoaVMxsDLAEWFZk9x2EYqo/JeXd3wdm1TJ9hdzdzWzQNsEzs7OAHcCVJQ6p5/diDXAe4SFxHqHY59Qa3bsvPkT53EVNP0Mz2xW4DviMu78cMkDBYPk+FqYxtX0wfx/fMOJzGO6+JXndCqwjZPXTtgB7p9b3SrbV2rHAHe7+bOEOd3/Z3f+UvL8RGG1mk2udQODZWFyXvG4tckzdP08z+wiwGPiwJwXEhTJ8L6rG3Z91953u/jrwrRL3ruvnaGaNwPuA75U6ppafoZmNJjyIr3T365PNg+r7WCKNg/77mDaiA4aZNZvZ+PieUAF1X8FhNwAnWbAAeCmVza2lkr/mzGzPpDwZM5tP+Hd9voZpi24AYiuTk4EfFDnmp8DRZjYpKWo5OtlWE2Z2DHAGsMTd/1LimCzfi2qmMV1H9t4S994IzDKzfZLc5wmEz79WjgIecvfNxXbW8jNMvvvfAR509wtTuwbN97FUGofC9zFPvWvd67kQWpjcnSz3A2cl208DTkveG/DvhBYp9wKtdUhnMyEATExtS6fxk0n67yZUnL2zBmm6Gnga+Buh3PejwB7Az4FHgZ8BuyfHtgLfTp17KvBYspxS4zQ+RiizvitZLkqOnQbcWO57UcM0XpF81+4hPPSmFqYxWV9EaG3zeLXSWCx9yfbL4vcvdWy9PsNDCMV396T+XRcNpu9jmTQOqu9jb4t6eouISCYjukhKRESyU8AQEZFMFDBERCQTBQwREclEAUNERDJRwJARwcz+1IdjDzezd1YzPb3c/zNmdlIFrnONmdW1x78MLwoYIj0dDtQlYCQ9qE8FrqrA5dYQOoWJVIQChoxYZvZPZna7md1pZj8zsynJwHCnAZ9N5h441MxazOw6M9uYLO9Kzj83GRzwFjN7wsw+lbr2ScnggXeb2RVmNt7MfpsMD4GZTUivpxxBGAJmR3LcLWb2VTPrNrMHzWyemV1vYe6GLybHNJvZj5N73WdmH0yu9QvgqCQIiQyYvkgykv0SWODubmYfA85w98+Z2UXAn9z9AgAzuwr4qrv/0sxmEIaO2C+5xluBNsIcBw+b2RpgX+ALhB73z5nZ7u6+3cxuAY4jDA55AnC9u/+tIE3vAjYVbHvN3VstTLrzA2AuYcjxx83sq4Qc0R/c/bgkvRMB3P11M3sMeHuRa4r0mQKGjGR7Ad9Lxm4aA/y2xHFHAfunRj+dkIw6CvBjd38VeNXMthKG0D4CuNbdnwNw9zifxLcJRUTfB04B/qXIvaYS5kpIi2NE3Qvc78lYZmb2BGHgvHuB1Wa2kjCR1i9S524lDDOhgCEDpiIpGcm+AXzT3Q8EPg40lThuFCEnclCyTPdkdGDg1dRxOynzI8zdfwXMNLPDgQZ3LzaA3CtF0hHv8XrB/V4nzNb2CGFWvHuBL1r+FL1NyTVFBkwBQ0ayieSGsk7P5bydUMQU3QT8a1wxs4N6ue7NwAfMbI/k+N1T+75LqNC+tMS5DwL/0GvKU8xsGvAXd/8v4HxC8Ij2pV4jm8qwo4AhI8U4M9ucWk4nzJ19rZltAp5LHftD4L2x0hv4FNCaVGI/QKgUL8nd7we+BPxfM7sbSA+5fSVh7uhSEw+tJ8yj3RcHAr8xs7uAc4BYGT4FeMXdn+nj9USK0mi1IjVkZu8Hjnf39jLHrCNUwJea+jTrvT4LvOzu3xnIdUQiVXqL1IiZfYMwc+KiXg49k1D5PaCAAfyRMLeGSEUohyEiIpmoDkNERDJRwBARkUwUMEREJBMFDBERyUQBQ0REMlHAEBGRTP4/MsweUZ8mh+wAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "data = sio.loadmat('ex8data1.mat')\n", "X = data[\"X\"]\n", "plt.plot(X[:, 0], X[:, 1], 'bx')\n", "plt.xlabel('Latency (ms)')\n", "plt.ylabel('Throughput (mb/s)')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise**: Complete the `estimate_gaussian` function to return an n-dimensional vector $\\mu$ that holds the mean for every feature and a vector $\\sigma^2$ that holds the variance." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "def estimate_gaussian(X):\n", " return 0, 0" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "ename": "NameError", "evalue": "name 'X' is not defined", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mscipy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstats\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mmultivariate_normal\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mmu\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msigma2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mestimate_gaussian\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;31m# Create the grid for plotting\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mX1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmeshgrid\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m35\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0.5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m35\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0.5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mNameError\u001b[0m: name 'X' is not defined" ] } ], "source": [ "from scipy.stats import multivariate_normal\n", "mu, sigma2 = estimate_gaussian(X)\n", "\n", "# Create the grid for plotting\n", "X1, X2 = np.meshgrid(np.arange(0, 35, 0.5), np.arange(0, 35, 0.5))\n", "grid = np.vstack((X2.flatten(), X1.flatten())).T\n", "\n", "Z = multivariate_normal.pdf(x=grid, mean=mu, cov=sigma2)\n", "Z = Z.reshape(X1.shape)\n", "\n", "plt.contour(X1, X2, Z, np.array([10.]) ** np.arange(-21, 0, 3.))\n", "plt.plot(X[:, 0], X[:, 1], 'bx')\n", "plt.xlabel('Latency (ms)')\n", "plt.ylabel('Throughput (mb/s)')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The $F_1$ score and choosing $\\epsilon$\n", "Remember:\n", "$$prec = \\frac{tp}{tp + fp}$$\n", "$$rec = \\frac{tp}{tp+fn}$$\n", "\n", "* $tp$ is true positive\n", "* $tp$ is false positive\n", "* $tp$ is false negative\n", "* $precision$ is precision\n", "* $recall$ is false negative\n", "\n", "$$F_1 = \\frac{2 * prec * rec}{prec + rec}$$\n", "\n", "For this dataset we use the $F_1$ score to measure the accuracy because there are very few negative traning examples.\n", "\n", "**Exercise**: Write a method the compute the $F_1$ score and select the best $\\epsilon$." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def select_threshold(ycv, pcv):\n", " return 1, 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should find the best $F_1$ score for $\\epsilon \\approx 8.99\\cdot10^{-05}$ and $F_1 = 0.875$. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:15: RuntimeWarning: invalid value encountered in long_scalars\n", " from ipykernel import kernelapp as app\n" ] }, { "data": { "text/plain": [ "(8.990852779269493e-05, 0.8750000000000001)" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Xcv = data[\"Xval\"]\n", "ycv = data[\"yval\"].flatten()\n", "pcv = multivariate_normal.pdf(x=Xcv, mean=mu, cov=sigma2)\n", "select_threshold(ycv, pcv)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Multidimensional Outliers\n", "\n", "If you completed the exercises above correctly, the code should return $\\epsilon \\approx 1.38\\cdot10^{-18}$ and $F_1 \\approx 0.615385$." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/ipykernel_launcher.py:15: RuntimeWarning: invalid value encountered in long_scalars\n", " from ipykernel import kernelapp as app\n" ] }, { "data": { "text/plain": [ "(1.3772288907613604e-18, 0.6153846153846154)" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Load data\n", "data = sio.loadmat('ex8data2.mat')\n", "X = data[\"X\"]\n", "Xcv = data[\"Xval\"]\n", "ycv = data[\"yval\"].flatten()\n", "\n", "# Fit mu and sigma\n", "mu, sigma2 = estimate_gaussian(X)\n", "\n", "# Choose epsilon\n", "pcv = multivariate_normal.pdf(x=Xcv, mean=mu, cov=sigma2)\n", "select_threshold(ycv, pcv)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Recommender systems\n", "### Problem formulation\n", "$r(i,j) = 1$ if user $j$ has rated movie $i$\n", "\n", "$y^{(i,j)}$ = rating by user $j$ on movie $i$\n", "\n", "$\\theta(j)$ = parameter vector for user $j$\n", "\n", "$x^{(i)}$ = feature vector for movie $i$\n", "\n", "For user $j$, movie $i$, predict rating $(\\theta^{(j)})^T(x^{(i)})$\n", "\n", "$m^{(j)}$ = no. of movies rated by user $j$\n", "\n", "Learn $\\theta^{(j)}$:\n", "$$\\min_{\\theta^{(1)}, ..., \\theta^{(j)}}\n", "\\frac{1}{2}\\displaystyle\\sum_{j = 1}^{n_u}\\displaystyle\\sum_{i:r(i,j)=1}\\left((\\theta^{(j)})^Tx^{(i)} - y^{(i, j)}\\right)^2+\\frac{\\lambda}{2}\\displaystyle\\sum_{j = 1}^{n_u}\\displaystyle\\sum_{k = 1}^{n}(\\theta_k^{(j)})^2$$\n", "\n", "### Collaborative filtering\n", "Given $x^{(1)},...,x^{(n_m)}$ we can estimate $\\theta^{(1)},...,\\theta^{(n_m)}$. Given $\\theta^{(1)},...,\\theta^{(n_m)}$, we can estimate $x^{(1)},...,x^{(n_m)}$. \n", "\n", "Guess $\\theta \\rightarrow x \\rightarrow \\theta \\rightarrow x \\rightarrow\\ ...$\n", "\n", "#### Collaborative filtering algorithm\n", "\n", "We want to minimize $\\theta^{(1)},...,\\theta^{(n_m)}$ and $x^{(1)},...,x^{(n_m)}$ simultaniously:\n", "\n", "$$\\min_{\\theta^{(1)}, ..., \\theta^{(n_m)} \\\\ x^{(1)}, ..., x^{(n_u)}}\n", "\\frac{1}{2}\\displaystyle\\sum_{j = 1}^{n_u}\\displaystyle\\sum_{i:r(i,j)=1}\\left((\\theta^{(j)})^Tx^{(i)} - y^{(i, j)}\\right)^2+\\frac{\\lambda}{2}\\displaystyle\\sum_{i = 1}^{n_m}\\displaystyle\\sum_{k = 1}^{n}(x_k^{(j)})^2+\\frac{\\lambda}{2}\\displaystyle\\sum_{j = 1}^{n_u}\\displaystyle\\sum_{k = 1}^{n}(\\theta_k^{(j)})^2$$\n", "\n", "1. We start by randomly initializing $\\theta^{(1)}, ..., \\theta^{(n_m)}$, $x^{(1)}, ..., x^{(n_u)}$ with small random values (symmetry breaking).\n", "2. Minimize cost function.\n", "3. For a user with parameters $\\theta$ and a movie with learned featurs $x$, predict a star rating of $\\theta^Tx$.\n", "\n", "#### Low rank matrix factorization\n", "We can take $Y$ as the matrix of all ratings. We can get the predicted ratings in a vectorized fashion using $X$ and $\\Theta$ by $X\\Theta^T$.\n", "\n", "You can find related movies $j$ to movie $i$ by finding movies with a small value for $||x^{(i)} - x^{(j)}||$.\n", "\n", "#### Mean normalization\n", "We substract matrix $\\mu$ with the average ratings for each movie from $Y$. Then for user $j$ on movie $i$ predict $(\\theta^{(j)})^T(x^{(i)})+\\mu_i$" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "def normalize_ratings(Y, R):\n", " m, n = Y.shape\n", " Ymean = np.zeros(m)\n", " Ynorm = np.zeros_like(Y)\n", " print(1, Ymean.shape)\n", " Ymean = np.mean(Y, axis=0)\n", " print(2, Ymean.shape)\n", " \n", " return Ymean, Ynorm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this part of the exercise, you will implement the collaborative filtering learning algorithm and apply it to a dataset of movie ratings.2 This dataset consists of ratings on a scale of 1 to 5. The dataset has $n_u = 943$ users, and $n_m = 1682$ movies.\n", "\n", "The matrix $Y$ (a num movies $\\times$ num users matrix) stores the ratings $y(^{i,j)})$ (from 1 to 5). The matrix $R$ is an binary-valued indicator matrix, where $R(i, j) = 1$ if user $j$ gave a rating to movie $i$, and $R(i, j) = 0$ otherwise. The objective of collaborative filtering is to predict movie ratings for the movies that users have not yet rated, that is, the entries with $R(i, j) = 0$. This will allow us to recommend the movies with the highest predicted ratings to the user.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Loading the data" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((1682, 943), (1682, 943))" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = sio.loadmat('ex8_movies.mat')\n", "Y = data[\"Y\"]\n", "R = data[\"R\"]\n", "Y.shape, R.shape" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Average rating for movie 1: 4.52067868504772'" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'Average rating for movie 1: {}'.format(np.mean(Y[0, R[0, :]]))" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJztnX+w3lV959+fTSASqgIqlFxwk2ikk4Lcmlx+pGUnDa1B1inpjNuQ2Bosnewu2trirBdadu3u4tTsdrB0rFQWMLo0Max1McO4ZmyQLR0oJLFPBINIeqWQGzBWkLbGTUn87B/P99yc5zznnO855/v7eT6vmUye+/2e7/n983M+53OImSEIgiCMH/+i6QgIgiAIzSADgCAIwpgiA4AgCMKYIgOAIAjCmCIDgCAIwpgiA4AgCMKYUvsAQERXEdHTRHSQiG6qO3xBEAShD9V5DoCI5gH4NoBfBHAIwB4AG5j5QG2REARBEADUvwK4BMBBZp5h5n8G8HkA19QcB0EQBAHA/JrDmwDwvPb3IQCX6g6IaDOAzQAwD/NWLMTrAACnL2dMzP8RAGD2+GmYmP8jfPsbC5MicWzJaVjwnR/lvjt9OeOHB2jIzas/eTpOefGHzvc+3vb2o3PxTvne5PTl/RXcDw8Q3vb2o5g9fhpe+tHCgfT50mvzzxcnM/7/9PzpoH88OvCdHp76za9diH9+I2PBd3408F7FOSQfVBgqDqocXPHWn6nftrzwpdl0r/7Oy9NjS04DAKubvDzW06U/AzBX71S+h3wbUs9e/cnTccZZ/4QfHqAhP/LagUKvG3VRpA2pMiyrTdrSb/NPhava7ks/6n8TUy/zwv5HvPz3zPymvO/qFgG9B8BVzPwb2d+/BuBSZv6gzf3r6Cy+lK4EAOw63Jt7vnbR5Nwz9TsE5V7/X/fP9NPm/1TvBABgz+Q86/ehYee5C2Vm2ySWbuwNxMX0Y3Z6FSa2PBIUls8fX3yneiewZ3Le3HPzb92tmccqvDzM71Xa9Xcz2yaxfvm+oLCneidw69lPDIWvu5nZNolnVm+1xluvQxfefsNQHqs80P0w/beh0qW+33W4h1uOXDT3e9lD182lOxYzz1Ta9fJSdfzWs58Iiqfy65YjFzm/ia3XLvR4hvqpx9NGkbjpZQTE12OVZ+pbWx3Li7+Nv+Av7GPmlXnu6h4ALgfw+8y8Nvv7ZgBg5j+wudcHgDI4vmYF5j+4rzT/UgqmLnxptXXOoahv1aCS4pevEfsao2sgqxLXgGIyO70Kxy4+aq0PKYN6SJg+ipRxKCH1v8p4lN2eQymz3U/1TuD+7VfM1esig5H+begAUPcewB4Ay4hoCRGdCuBaADtDP951+ORMV0fNWFx/K1Rl0d+73AL9grb5PbOtn8kplSAvrmb6dh3uzYXn8k/5oft1+W2PD/hnC8eVdl/+qMa8bsPDA8/1OE71TljLST3bcWCF81tf5T928VFrnPRns9OrrHH35aP5XP9u/fJ9Q/FV/unfTmx5ZKg+zGzrz97UDNv81seeyXlDZeurq7bv89Drsu6/euardzpmXtjiYatTef6bdcpM/3enFgyEHxLH2HDUb/17Vc6m25B4mH7uOLAC6zY8PPfMrCu6+zxSBo5aVwAAQERXA/gjAPMA3MPMH3O5LXsFIAiCMA60dQUAZv4yM7+Nmd/i6/zbwMy2yYEZZZcInb0VQZ+ZhIQXOgMvgsuv42uGZ/FFCJ2VlZW22elVtZRpDG2LTyoxK6sqSOljysr7Tp0EtolkiuLzb/3yfUOijhj0TkdfatfB+uWD4i59eRpS4ULc6GIGlzgsZFn8zOqtuW7yUOl0xWP3vXcXDkNHbdzlUZas+MkPfarQxq8NV9m4xIYmC/aXq/FTd0es8qWsPYrU+Kfsa6n2XZRODQBmolNnlDGjp03+G1rQ+gZVVZvF+qCox0tVaqWFcuHtN8y9O3bx0cKNLVTeqcsl1y/fNzSIT/VO4JYjF839nbff4fpblVPKzMhXj2a2TVrTGipvVenN23fSw7CVp+u7EPR2o09K8tJgq/s6vj2ZqigrT4CTbbLJlYyqH7FxKGvQav0AoFdYs0K6OtW8zlZ/78rImW2T2HFghdWvlMwPrai2zkavHGZFWbqxNxef+7dfYfXTTEOIhsk5e45Zn+ubVT5ss8g9k/MG4qs2wfRydc0qbfnnS0NMx+CqL+uX78PSjXatjJABUGmLmPXWVJcFBvNTH7wVF95+w4A/LpGWLd36akWflJibnwoVl7x2pMrKttHrIlWkqqtf69jyMhSV/jJWoEUw+5mUASl1IKx9EzgG2QQeTZpQ5xSEcaK1m8ApKJVF1+zBpxZowzej1rFtvOkigZRRN290921W2tJvzuLM723qnnlx8OW1a9boygtdpVC50fdVQkVJwMm8Uf7oabWpFuq/fWXmioNN/KP8zIu3ynfbHowNPV42VVazHH1x9uFbOZh5lKdaa9aR0D261LjnqS2ninLMehKqJm6GHbu6USJGMxxb/tjy2haPWFo/AOinINdteHguI/RMsokCzE26PH16G09+6FNDz5Zu7M3JrPM2Ak257q7DPTyzequ3M/AdbLHNmk1xlvn92kWTQyIIteR1VRo9XWae6fLkW89+wro019On3K9fvs+aX8seum7gb99gbm7kqrMOwEnxoJ4felzViUpbHPQ9CD3+z6zeOhQ/3T9fnF3fudzr+acGyKneibl0qTaguPJXr3f678OsHyqteybnzaVJ5ZF+5sNWziqeqtP0iRb1NJv5rTBFMWbdU2cjzPjvOLBi7vCcK2zf+SE9XFfcXOjptfUXPpZu7PcHZppUHPTnpjKKmVehSgkmrR8A1i6aHMhk1QnaDkwozE1P5d4m0/Ttpq9dNFypVIVT7/PirtCPyMce2AiZbbpmiLZBR+WNq7GuXTQ5l895cmBV8Vxx1Cum6hT1cM38tW086iYQdDc2f1x1YtfhnlOeb+aDcnPLkYvwzOqtA7Nm28ah+b15UtTc6NbzxBZf216O2cBdEwVbefnaiWmSAzhZP/RO31ZXbIOTKyw9Ta7Oyux89bSo+q2bXVDfLN3Ys+61KPTBzRYnc7DW0+qr3+az2MFjqndirn/T424OwOZvYLgP0cOOWVW3fg/gTds/bK3URY+Bq8pUFy65dxny8Nij6WXJ4H15WCR/Xd/60pmXB740277Nq18hee4L0+V/3fVynLGVT0i5lmUKIs+fIuG00hZQLLIJLAjjQVN2fYDRHHRHahM4BdtSNHaTpsxTwKZfZZ9MLZvY+OVtUnWNvPSHiOWAduVDTFzqPpQ1/8F9jZ261zv/OssrT9Gijvzo1ACgN8q8Bmgb0WPFHhNbHimtQphhVznbKRrnXYd7wfGzafYA1R1809HrQKwBMx3bN670q0YZsv8z1TtRqtXIooRY7lSEdIpld5ZtUA2OPRNQpFz0g3Q2Y3N15EenBgC9UZZhWzyEsjuyOmZWRePs22A3sR1aCrHY6sI3sJt+6Jt3quGmmKTwaY+YKBPYIYRoxIS+r8O0sysMczPW9rxq6lqR6Hc8AHETzdgBUeWfuUm963CvttVQ6wcAPVOPr1kRbXI19p0Zdp5btSoJjdetZz/hrSgh2gY6up64rdKY+ugh6OqHrvDUb5t+/S1HLhqKs95QdFVec/bj06RQcfKlxWZSIQ9Xx+cKT1dHdJWNqhc+E8kuHr3xkqFnNvVHHV+HoX9rOxsB9AdB0488ExhmnGxxiDmj46vnt579hPOUvK7xFoPtTMfMtslgbR7TPH2eincM5uzf9LesAbH1A4COS4c5D/OgyI4DKwY6HvX/7PSqgYYdMptUuughKxLlt25P3KyEyx66zjvTNdE7E7WkNO297Jmch/u3XzHU8boq0Z7Jedal8FSvf4PW7PSqgfgCg6p992+/YqARmeGs2/Dw3OUVKv562bg6cX2mPtU7MWA2Qun5X3j7DdZ6oeehygfXASLzvgefCqTup94B7r737mh7+ipOys69+vv4mhVDOvBmfdM7DNuAZUP3Y8eBFUN67Hl1WlfJtHWmqr4ofMbXlKqsK8/WLpocis/s9KoB1V6zzun11Mb65fty+xCbTr5uvkO/XU797RvgzEN1IXtFtnemKm2qOG5ktYDKUNVyaQekaA0U0TRQaSlbW6Fo+nR3rvzOc1OlWQg9PJ+WiX6Tks+PFOpQuS3jW1s6q1RTrILYPFLx19Nh+lE0jSHtwnebXKy/ipFUA/U1VKE9FCknXyNua/nnxcv3vqqOvQ7VRtUJ1VUubRtwfKTmf1nlNjJqoPopV1227NuoMeXKLrsspgzPFbbtue6v73vTL93PUPm8z46NSqvur3ki2DSDkacxs+twb26/xQxTz0ObGM0WPzNO+rO8PNS/UctxXWRn4pJxm3lkPjefqTB1MZ3+LkQNVIkhbHFR6JZRbWkw80r/rZ++9Yl9bHlvlr1eR0wz2CH7Kup7V52ynVKP3Qw39ylUHPPakMukt16XXX2Cb69Hr7t6OwT8+zxm/dL9MU86q/jn7QHFbFzrJK8AiOh8AJ8DcA4ABnAnM99ORGcB2AFgMYBnAfwKM79MRATgdgBXAzgK4Dpm/rovDLUCMBNUdHap/6+baLC51y9sDvHbhT5LUr9jZ062MGa2TeKZ1VvnOkfzfawIQuW1aXvGRegsUDXcdRsedprZjsUWpl6men6p365v8kxLm/744qCe33LkIty//Qocu/gonlm9NbisbWIzPRxbfH1pCKkDtjTmzbj1vFbprXP1Nju9Cus2PJwbV6XU4HvvKt+YVUdsGpV7vfPX41AkzyoXARHRuQDOZeavE9FrAewDsA7AdQBeYuaPE9FNAM5k5unsLuDfRH8AuBTA7cx8qS8MNQCYHXcIMctul1uzYIqiV9iu42vweTL1PKNhdYuAYgaAlHiZ7/Pch9TzkMlLTD1LET2k7BukYsuzOkRPeRMG/XdKfMzB1jehiKkzlYuAmPkFNYNn5n8E8BSACQDXAPhs5uyz6A8KyJ5/jvv8NYAzskEkF31ZFLq8CVEnVEs2V4bHNIiQeE1secSpzubyJ89fc+kZ4meIyCUkParimtjyXmkmmZ1DiAhMuVMaUqGiszxRmJ4OG+qbvHqw7KHrrP66xCGphrsUuhaKi9jOXGnG5cXJJ0qZ6rkPvoWIKELqAXCyLinNr6InZ10mzlUYJnqHr+ezTf3ZhRmOyjdXmd5y5CJv/uhaQbWIgAY8IVoM4C8BXAjgOWY+I3tOAF5m5jOI6AEAH2fmv8re7QYwzcx7Db82A9gMAK/BwhU/R1cD6CfqwttvKEUcE0PTG0+mSqJJ3iolb8btCzdGbKHjmjWZ8UwtJ+V/nmgjVfulKGa4ebN1E5u4xyZC1CkzHaEzWTNMX13Li38eNi2uKjTI8sRFOmb6y1qRmPmY1wfYqG0TmIh+AsCfA/htZv4H/R33R5eoEYaZ72Tmlcy88hSc1IfWTRTn4ZvRhaI6LNuMtQghG2o6ytStC2Wq15U2/QrGGEI6f9eM2hdPwD9o+crInPnm1Qfb6WQzvj7roqGrEzPOZrp8nb9tQ1RfISi/9O/1wcFVT0NWji7U4GpDzz/zBLVvomGLv4kvzrvvvXvovTLVkrKKAk6mRc8LV+dva7emO995HR09PDOf1SpKz8e8PsD0M4ZCKwAiOgXAAwB2MfNt2bOnAaxm5hcyEc9DzHwBEX06+73ddOfyvw5roDPbJvHmu4YvUgGatVBYBWrGUseVjL7Zt23mWMVMKjQ+Ie9DKHMWXqZ8vSp1zSpNnKdSx75ADHVIEPQwVPorXwFk4p27ATylOv+MnQA2Zb83AfiS9vx91OcyAK/4On+FLt82VeHyvlOYpxT1b12dP9A/5WsLJzQOOrrKl6Jsex+22bgeRqiRKd8qRc9LXc1Rd2ezK6OrU+pH5ncdHp7d+DY2XehyYNvMLqbzt6lNmtj2asxBzKeS6sNs0Or/EENltjqgy5dDZsumqqhrladOnttm5iH+mxQ15Dezbfi0sM9f1+llm8qqCz2vbG59olvXKtOWPz436pAoEK8hWUQL6OcAPAzgCQA/zh7/LoDHANwH4M0A/g59NdCXsgHjkwCuQl8N9P2m/N+k6hVAnRonMauJMvcw6iImzkU1JxShM8268rOs2V6T+04xYZtuu1hvgXbFu0hc9LZUhxbQXzEzMfPbmXky+/dlZv4+M1/JzMuY+ReY+aXMPTPzB5j5Lcx8UV7nH0oRk7S+jE6ZhfoItWNU9ga2j6L7GToxcdbt6ChNjhTyOn+f7Zk8ysqblLSV2fmnWqhMcRtjGTNVbh9LyOy6LZ1/UWKvpARafhL4bW8/muum7plSSGUJqdyuASZEayWUvEM8qRo4Lv9S46WLNkLyTrnJE6G5zBiH+O2yhurDJqJp2nRBleH76mJeuFWd5zAxVZVTyjU17BDyFAiqptUDwLe/sRDH16zIvZ2pzJlsGf7ZtCjyOm5fZ6bLt5du7JWSHzsOrBgyWRwiY3c13CINK0RDBDjZWHQtIFfczYalLjoH/DcxKb/VZrlyq1uKdYXh057RceWzKo+88nXVJZtGiSuMFEztFb2TD6m/Nsq8Gc/VeerxfPNdJzXRbG00T0PH9twM28x3XxrLHAhDzKWbtN4Y3Af3rx6ZJdoo0vQ5CUEQhhkJY3DHlpw2oDseI/5ImcWbMwDbDn1IHHynQl0aRCHx9Rlvs+nk637GzrRc7s3Zj0+P3oyDiSpTX7xd/tr8MuuI74Snj+NrVkRraNn819Nm0yxy1RPbakPXbopBj0PZcnc9bXmaNnmEaF7lPTfzOi/smHyJEX266qAtjiFh2HDVndgybvUAcNZpJ/cA9kzOm9NlDiHUkJlOiJqdsh8OnFRLdR3rtvmtlnzmBR/mBQ82XJe0qIso1HvVGep5sPveu3P913FpLCmxS56qoIqLksHa1DNVmZr4yi51kzIvf/VyvPy2x+dUHFXcXegbzbo7pRaqDkvpaVK/dRv0imdWb8XElkcGZNdTvRNDl7WEsmD/wij1USUaMSdDOubEyDyxqpN3gt30Q52a9rk1wzLzU/2tX/hii9v65fsw1TvhvbxIEbNnp6fZPO+itwtVX1x5bRtYXYfQgH5Zh6r6KlotAlqwdIL/1bODN+iUoT5oEnNoqShKHbQsbZ8UA3N6mmLTp7vX0xCTntQ8NU0k6Kc2ixyccqkzxhxoKqueTPXCLdCmULXITs+z1IOUTYoVy2iXPj/yDs+Ftmc9DJufIyECMtEzRe8IbMRsxrmMmgH5M87Q0VaJVMwGEXLow8exi48ODIo+P9V71wwlBDWjVRUw5mCRK0ybH7Zn+oBvrghtF7ubnbqJKnPXpfDqAvi8g39qQ9S23A8tS70umo25iFJCqMhOhaPCchl909NkE7XocffV9bIVN1z++v5Wqz49rebKPA+zzH3GA2e2TTqv7ly34WEAg+3Zh+5GfWumJYTOrgBCaNux8DzUSL7rsN+2epexGfVLXUnUsepQhKwGylwF6OlKnUmnzGZd3+SZRfb5Y3NXZttMMT1R1ypDdfJl3SsSykisAC567fetz81ZjWkuQs1SbPbDlZsQbMf3zRmS6UZtxJjP81QIdx3uzY3kLvPUtpmXOaNx/Ta/9bm3hZ2Cbfa1dtHkXDrNMHYd7g2Y6vbNRoFBM+G2/NVnRD45uO6Ha8ao4qyXu14P1SrApjRg5rMrPa7nu++9O2pWquqhkqW7ZoZ6/io3rr2SPFv0Zjnondj926/I9c8sb9uK3JcHoatIlf96fbC1af23Hq5qc8qNrzwVt579BM7Zc8waJ5e5EDNu+h6a6c5cmcUoC7R6BaCbglCyUdvlya5RM280DZ3BzGwbNhgXegzefB4ywk/1TuDRGy/B/Af3Jc+UQi6aNv0uMitSla7orEqfzc1Or8KTH/pUUBx9cc9Ll9qcs+WzykegLyryzeRceRA7Q1XxtdWVkPqjvk+ZGaeuqsz9GT2f9HehdSxvHya1rtrahU3ubmt3qSuN9cv35e5XuUgtj5FYAejsmZzn7NBscjvfUlb/zsbs9Kq5fQGlzXH5bY8PuDHlxqY8XJ8J6TMk/cCUS4MGOCk/1RuOTcbs2utYt+Hh3JOwynytml3Ebgabv80B0ZyR5M1QpnonhhqY3ghdextTvRN4ZvXWgTDNsGyzJl2jQrdVrzOx5ZE5c7w7Dqywrl4UC/YvHKoXSu471TsxcPDON0tbsH8hgJN1R9Wf0M5foWadtnS5vgEG89lUpdXLUsUTGDZhrZejOoS36/BJrShTO8cMR6XTld+mBpWrnE1/lTaVXgZPfuhTc+3Xtyenpyl0lr10Y39VpcdXDc629mCGbx6w1A8nmt/qA0wonVgBFJnRCOVS575K18o7NL5dNDNetszc9K9rZR1DE9pkI7UCaMNJ0xAZbFHNhpSDPnVTpPMPyR99BhcrNqkD30w6NL7P/UZcPSnTXEIqtjboK0/XKsLlX12df4xmVoyfvnfrNjwcfLDQpZHmo0h6OrECiKGtmj9l7O53ceZYlKJpblKnPJW6yrlN9anuNNe54mgin0dqBRBDGzt/oBwrf1VXorpM9LqwzWSKptnV+VeV1jL8rauzKBpOmauumLj4tGVCw1Eb+3UQu+Krk9YPAKa9EfV3ir2WWDXQovZN6sBnd8SFEimY+bB20ckbilx+mY3PdiOY7W/fxq9NddBG6GalDZvlTte3IfnoqkvmprVtkzN0kKjqsJQN28Z02TfWlYXrPIIrv2zp0FV29fIo0rZd4fvMYbg2nvMoqw9qvQjoTds/3LklfNso63CJj9QldV1L8bKX4W0VNdbJKG/c1kGROpmX97WJgIhoHhH9DRE9kP29hIgeI6KDRLSDiE7Nni/I/j6YvV8c4r9+YMO0X18GoRtZdYRXxve22WVM5x/jv/5bX1LHpPHYxUedh9Ni4wa4V0S+hpa3iWcjpPO3Hb4zsaXXpx4c+m1enMqY7dcpRvGRl+a6Vuwx4cxsm8T8B/fNfROb/z515BgKrwCI6EYAKwG8jpnfTUT3AfgiM3+eiP4UwH5mvoOIbgDwdmb+d0R0LYBfZub1Pr9fR2fxD154w9jPtNqEa9ZSx2ZrFzd0BaEJalkBENF5AP41gLuyvwnAGgBfyJx8FsC67Pc12d/I3l+ZufcScpNTlbRBBa9NuGbS6qalKlGdv61M6pSXN01VM9q2yvxDKStfUv2J7SvasJdYVAT0RwA+AuDH2d9vAPADZj6e/X0IwET2ewLA8wCQvX8lcz8AEW0mor1EtPdVHBs4zaj/roPZ6VW5MrrQQmxDYbsoY5Az8yk2vTEduK1MTHv8MbS5bGwUNWrnouvy/FTTEDpFVpnKzHsoeZcp+ShrwpM8ABDRuwEcYeZSddaY+U5mXsnMK0/BglpUqFyNIs/mC9COQ2pF+e7UgmC3rsrpMjsc2rn69ilCByhlTz0W3aRBWbgMmsXi08hKQb9HWRhu4ykHsXTKULYI6VPKUuoosgL4WQC/RETPAvg8+qKf2wGcQUTzMzfnAZjNfs8COB8AsvevB2A396mhixbMjqosXW5XRx+i1hUahzoGitT8iJn56eWh549rU8rWoHyzF1sHrmb85nemNU7AnZYyTvCG+gcAV/7q9VbroED+7E1Pl7r3oKwZ3/E1K6x1sQ2DQkoa9W9CNt3z8N0Nkhe2y+qq67vZ6VVRNppcYRcheQBg5puZ+TxmXgzgWgAPMvN7AXwNwHsyZ5sAfCn7vTP7G9n7BzlgB1pfAZjX4uWZqE15pxMyysZuUFd52MoXl7L2MlRnbF58YVpSNJ+70m2WhU+zxCwPfdDJ68TLViTI80/lU8pMzZYHZc34XFeDpk5QQjW4XN/opKTRVgdT7kDQf4fmhRn2/duvCKpn6ruJLY84Tb8r8+IhYRehioNg0wBuJKKD6Mv4VY27G8Absuc3ArgpxDNdDVTfEM7Dl0Eh9+8CaTfsuFD+KAucdePby7AdqMqbBdkO47jeAYPp1i0cmuUUMjtVf9vK2GUp0UYZM8YU8hpvlatF3WpnGegHsELrdtlnUlLtdNkOIJp9Q8zKqIhqph7OLUcuSjI7nULrD4Kdd9N/nMuMOg40jSO2ja+yVS6rKDubSmrK4Zoq1EvL8rOKuI3jAa7QNJv1tKt5NTK2gHTNH9M2dtW4NhRTNhrbIGONIbbTceWJSnfRsrPN7G0dvUtpwJf/Vcy4F+xfOBfnIuK3KuLWxQ6tKKFpNicpbbRIWyatXwHEWgNNoY5j/baZadWzizpnL21enRWdRYemraw8MMttqndi4EYpQchjZFYACiWPr2ITtY6TxrbZalWds+0Go6rxdXxlGdtKpeg1lz5CbpiLxdwEVppAdRHaxkzb+rEmL9pCUzP3vD2oOuLVmQEA6G8ymdYW88QxNts15nV0rgveXZVWLelTBySl2x17ObjtvW6N03btoSt81zd5F12bFhRDLKyam4MpmiMmeuPQRSwu/fsUy7EL9i90pkvF23dlpcrf2elVXjGQeSXg+uX7hq5fDNGi0uulL0625/p7/RrSUNTqx3W9obI068PWDm3hmH67/FFqlnlWfWMH1zIuh9Lf33r2E1YbTeuX74sqg5S21BkRkK3BxZIi6gkVH4Qu//U4hMbH504v9Dw3oWnfdbiHW45chFvPfmLoGxUX08+8dOnufW5jxCi6W/238jNVJKOn0YyvLc6+6w1tcciLV0i8lZvYOq2bTtbL11VmvvdmPHcd7mHZQ9flthdXnPPSYoZ3fM0KXH7b43OaO75v89qQeSe0me6QfE4VJZvaSDPb+hfJ63U7TwRo1sFQEVBnBoAyaIOcehzMCI9KGl31pcqJhC/cNpAXt5C8KTt9bcmvlHhUFfeR2wNQFycUkSWmZHQbdvZjZLK2b2PzTOV1yJI9NH/0eJgioFA5skmonr/+u4zle0jnb4pvYsQMdWi7pZ590NuQTYQakjeh53BMbGVYdgeaekFLKvosX49DLCkXDgEdWwE0MbPsgj58Hinis1TRgvnbZHZ6FY5dfDQpT0N0tF3lVfWsvUyaCLdIvVTxDc3jtszYy0blQ0z6lLi7/cEpAAAgAElEQVRVd2/7Pq9OmN+M3AoghryNuxgW7F8Y9F3qZnBRf0O0o9YumoyS/wP904ghaVIzD31WZ5vBKnfHLj5ayOCW2hyd2TZpPXmpN5K8TcjQmV5s2ZqzMd+Gvy0Oefkz1Tth3VROMUBnOw1rMju9yhpPM023HLko6CrRssU/IYTa3DH9C92813GZRbE9U52/Hk5K/ujtL2YF05kVQFWzBp+/ZYdZ9rWEwOCmniuu5uZc6CajbRM4BNtMcHZ6FdZteHgorkVWdXmzoqIz6ZQZHWBfccWkM+/8RhXnO3Q/Q/3X8zcvr/T0p5RLFW0nhJiyL2vlFrPx7GLkVgA7DqyoRDZnFm5VMxWgb4irKn1oX1zXLhqsmCHpCu38Q2ecE1semQtXn634bAqZmDJg10xZvS/aGJdu7M+oY2Xy+opLqSrH2IDKu2rRPCRmI2Z/xPQzdHDR81+JgFx1Sy/nlHKpq/M388W2X+HKc9fq04dNdVvlVYzdptR+pTMrAKE8iqi0hcyIYtX5hG5Shw2pEKQ+DTNyK4BUYlYNVV6JZ4tHiKZNalg+2WXIzN7lxicuc31rprGKxpp3uXZM2eqHsGKoUnukLs2UmHDMGe9UL9yUcpl0vfN31bc6rugc+QEgpnLYNEpsHWlKwdiWknsm51VSefdMziu0ZPZ1fq5392+/wvq8rtmZutXLdW1ojMw8xPSCbclt09wo6x6GugwhppZVVXW5SmztuMhAW0S11lbf6jDlIiKgHJrafBLiUKIHlwiiKZXOtoQvFCOk/NrUV4yUCMhmJyNmtDU3D0MPHs1sm8Tltz0+9Lxuw1Z5G6O6KOn4msHNcp8Kos/P2FWOGUfz0JdrmWvGNyQM05YTcHJDUr/fV0+DT7XSl1Yzbj77UTrH16wYWAHoFxvlpddm+rqKmanZDnQbOq7yMutW0bjppB66zLvEyLUKs8U/ZINXR6/fl9/2eJRYV6ntqn/68zzKOqAqKwAPTVwG0YZZRMpstc6NOFu5qGdV5V9sXQg1NJaXb7Jy6A4pdcRUoy2LWmwBEdEZAO4CcCEABvDrAJ4GsAPAYgDPAvgVZn6ZiAj9S+OvBnAUwHXM/HWf/7YBoM6OpsuNr+6BJDU80eAIo8t1sa109bavEOoSAd0O4CvM/FMALgbwFPp3/e5m5mUAduPk3b/vArAs+7cZwB0Fw/YSok9uikdMOyDrl+8rTdxTZJkccorR9N+8GatqLRJTVGaLsy2uIQfSQt7ry+mQU6uhYeZpBKWKJV3o6dDD0EVIMWGZ+WH6a3MfKtox/YrVtHKFYXtmigr130rEY5ZVXvwntjySrPEV4r8NW/z0/LadVneVny0+sf1V8gqAiF4PoAdgKWueENHTAFYz8wtEdC6Ah5j5AiL6dPZ7u+nOFUbTIiChXcgsWBDCqGMFsATA9wB8hoj+hojuIqLTAZyjdeovAjgn+z0B4Hnt+0PZswGIaDMR7SWiva/ipEpfqIXKpoixitlmyoifyw9zA9Xm1he+2fm3tS4UpepLQMr4tghFZ85lk1qPbKd4u0aRAWA+gHcAuIOZfwbAD3FS3AMAyFYGUUsMZr6TmVcy88pTsGDuuTpc1FZ5cejMNOZ4d1nELA2VDZIiDc6VRqWJ4zuIFmP7JMU+TwxtMAWeRxHruFVY1q2qE6xy5Ze3Ce/CzL9UM9dlkFpXiwwAhwAcYubHsr+/gP6A8N1M9IPs/yPZ+1kA52vfn5c9C6aJzr/sCl13Gma2TWLP5Lyoxh5yq5MPVxptz1ONzZn4ZMqp4RTJg9npVcknOdsi5kqp+yH1O+Q0eQplDdi6PzFlUdbkzrU34yO1ziQPAMz8IoDnieiC7NGVAA4A2AlgU/ZsE4AvZb93Angf9bkMwCs++b+JMm9Q9zJLXbtXFinxj9mMM7FVjLzOcunGnrPzMsU3ZZtMCPFPdeZ6HFWnYruwJJa8jdIQ8gy6KdosEszrzMteIRWdHJUxcJr7TDHt1Rb/lDyypSNEnTiFomqgk+irgZ4KYAbA+9EfVO4D8GYAf4e+GuhLmRroJwFchb4a6PuZea/Pf3MTuMtqW13ZwKxSLTM1D8xyL7seVJHmNpf3OKrehpZHm8stBrkT2MKo3FVbJapzqKKTKOpn18pvVDoTF6npG8cBqG5GyhQEkGbSwEUdVvZM8kwHlEGqDXIdmyjFF0bMEle/BMaFLw1m56/7E2v/PvRdEUzdfVt4troYmpbUfCwL1fm7bgxzUfYeQBERaRPk3RJWJ50ZAJZujJv92TJUfR8qPrBpz6RW0tBLHlLs9ijMe0VdfoZsVrnsp5jX3fn2GFzo5WgO7D6rl6a/tktGXPJ7n4aGr0NypSWvnGa29S+FUR28Hhc9vNC6qH//6I2XAPBvbLuuJXR1hikH2lTa1GU/MYfTbFSlReNapdjUkk1iJg6hA42t3PRnIflohjUWF8K0wU6OIAhC2xk5ERBQ37Vwo0DekXHAb64h1s8Ud2Z8XHEqg6qX2HmzNt+9AKYYqIj4xjTJEDObjy03FU4RcwqKMkW8TZKaBttKsQ46uwLQf5dtA17fbCxrw6orG4J5GjZlXm6eR8yKzywnX7m5yqIM7aIqNNW6vmlqy++mVvNNhNtEmCOpBdRkQ6g77K4MGEJ76frAUQWqXbXVqnBZ7X7kREC2AitjuZTnh1qaqbCLalfkfa+WkEUrgXnYqyqtkJhN0pTNrdjvXdQpXrBtRMfEvayystVZPW62i2cAv7hKsevw8GHB0PaYurFu+y62D1DtyqUw4YtfHq64xLRlNTilhhVLZ1YAU70TePTGS2QfQBCEaMZtRT1yK4CiF50L3aCLFhV9xOrIA+3TWx8Fxqnzj6EzAwBQf8MI0Zkum1g7Oa4LUOrAd6ApRIygcOnIx6DCTznkV1V5zmybnNORjwm7qc4qth7FlHEbybtLuK7wm6QzIqAy6Jopga4ybsvtsgnNP199rqMMZqdX4djFR1tf1uNYH0dOBATEzVBss5OYzl9ZHi1rlK7T/ESVK4IQnf3YxlbGtYq2/A29Ss/3XROE5p+vPtfR4U1seaQVHWuZ5VWkvRfpn5paDYzVCiCVcZpBdGmVNKpqjl22eivEUVV7G8kVQFOMS+cPVHNLlIlvphSjrlll56/Ho+gMs+uychdtkGHrNGHkMQU933wGDuugUwNAlxtSmytnlflq6yR8HffaRZOFOpaUtJjfTPVODDRMW3xjGmqM9ppt9j+zLS5P6uqYzYlR0wOCb9XUdNx0fBNKdSVrXXRqALj8tsdL8SemMy6rcwxd0jcxyMV0UKENqciBtiIrrud+wz5b98Xb/KYs0x9A/MBvqydLN9qtrrpoasXa5pWyL26uA3GxxK4UXXWjThFsp/YARDYqCNUwjm2rCbMQdVHLHgAR/Q4RfZOIniSi7UT0GiJaQkSPEdFBItpBRKdmbhdkfx/M3i+ODe+cPceKRFcQBAfj1vkDdrMQ40byAEBEEwB+C8BKZr4QwDwA1wLYAuATzPxWAC8DuD775HoAL2fPP5G5i2L3vXfP/a5LVNIm2WGXaFqV0qTNezDCeNKGOll0D2A+gNOIaD6AhQBeALAGwBey958FsC77fU32N7L3V2YXxQej32RVl1mINss120zbZlXjOMMV2o2qk00qtyQPAMw8C+APATyHfsf/CoB9AH7AzMczZ4cATGS/JwA8n317PHP/BtNfItpMRHuJaO+rGBT5qKvwBCGWtq1I6mSc0x5C0zPxJm2cFREBnYn+rH4JgEUATgdwVdEIMfOdzLySmVeeggUD73QRkCDE0LYVSZ2Mc9pDGOfVYRER0C8A+A4zf4+ZXwXwRQA/C+CMTCQEAOcBmM1+zwI4HwCy968H8P2YAJc9dF2B6AqCIAg6RQaA5wBcRkQLM1n+lQAOAPgagPdkbjYB+FL2e2f2N7L3D3KkDuqb75KZjCAIQlkU2QN4DP3N3K8DeCLz604A0wBuJKKD6Mv4ldzmbgBvyJ7fCOCmAvEWBEEQClJIC4iZP8rMP8XMFzLzrzHzMWaeYeZLmPmtzPxvmPlY5vb/ZX+/NXs/Exue65RnHnmqnLb3s9OrnN/lbar53rveqef6MXDT1v9U78TQ9yFqquqbGPvnNs2EmHT5jrOnXiXoCle/dCWlzMx8Nv0octVh6Aas66rG2Hj4rk0MMTGg15G8uOedoLX5YbrT670vvJSN2jI3d812mYrtulDX+1DtoJRrRxWdOgl8fM0KuRVMEAQhh5G0Bpq6AhAEQRCG6dQA0GXK0MVu86lkc7mt0tukDnrecrsMmtYhb5JRSXsbzkk01bY7NQAs2L+w6SgkE6OL7aoMbT2VvOtwb0iXWqV3z+S8WjoKM89mp1cN5FdVeWemu02dYmxcfPteNvS0x3ZgTXV4tjyxtc2UfcOYMIHBgSelfs5sK246ulN7AG23WNil27RSGYc0CkLXGck9gGMXH7U+r3IJF2PXXe8YzdlUmTPDqmyHhMwmYjv/umZ6thVAE7jSe3zNitxyKzuv2iwy7AKxWjhlu00lJoxOrQBEC2g06LL9dVkBnaTtK3Ifo37P90iuAITRoKudP1DvbU1tp6udP9De/bS66dQAIMbguk2TIgkl3mqDxseo4Dt0ptOmjXFhkE4NAEVnX2VcttzFDiQlzvo3ZV1S3eSsS9WdMlYfeflRNL9iTh673NZxsfiOAyuGwreVcehKITbOXd3j8KXT9a6qfqdTA4CNmNlF7ADiUhlLqXh58fT5WbTw8zo922aX+mZm2+TQ5naZuDbaYtM8s23SuxHsy9/Y8ly7yK9+p+eX8jt0Q3GqdyJ4oNwzOc/ptuhkKSRPFuxfWNqgbtazEMywyxgQUlUrY+qaSqetLbnyoCqxaac2gZti1DeMQgjd8FMbvE3nmbnRbMZnHDZzu7RJW1d9UeGkhtd0vQ5lZDeBQ0fnspaHXSnwGGJm1yofVUdiqraasxjV6T6zeiuAtOVuEfcqfuaMySzDoqsa9Y3v26bFhV3p/IH6xIMqHF94vr7D9V1XxVGyAqiZvJlnysw0dpCqQg2zrIEy1Z8iaWrbIN+2+LjoSjzHkZFdAdRJFTO4vM49RSwR2whDOsqQ2bbuJnYAsv0u4k+RAa1tFw2Na6dax8Z120ntc1K/G8kBIC8zQpdrKZunPqZ6JwYqeVknelPuHwD8Dc61UaV/EzNY2eKx63CvUMedZ8PFlXYz312HC1Xai2pRdYG8NmFLT8xAVWbnXtVJeGD4zo2UeBdJq6s95PmpfxdT93JFQER0D4B3AzjCzBdmz84CsAPAYgDPAvgVZn45uxrydgBXAzgK4Dpm/nr2zSYAt2Te3srMn82LXNMioJBNtNBlcN3L5S5tAJqkiMGmeiew40C/Y4jN55S86pL4o6y4dvkEtw1lWaDOthJTFkUsH5QpAtoK4Crj2U0AdjPzMgC7cfJ6x3cBWJb92wzgDmBuwPgogEsBXALgo0R0ZkDYrUXNmEILs4rOwjcTCqnQsSqKsXFwhedCzXKWPXRddDyUSmRKPoc2fn015AqnjWKMsuqe3vmXeatbmeRt6OvvVeeql3/VK7eYsqjD7E3QJjARLQbwgLYCeBrAamZ+gYjOBfAQM19ARJ/Ofm/X3al/zPxvs+cD7lw0vQIQBGG0aNPKuMoVVdWbwOcw8wvZ7xcBnJP9ngDwvObuUPbM9TyZtsw4hHbQxvogJhDaR1s6fwBzIssmKbwJzP0lRGm6pES0mYj2EtHeV3HM6W7pxt7ccruNy+6qiFmi1rERabvMPhb925QzCkD+0toWv7IGDVec29TZVEGVp9djSK0zTWOrs7Hx0+t1ShsUEZBQG13aOBWELlO1CGgngE3Z700AvqQ9fx/1uQzAK5moaBeAdxLRmdnm7zuzZ63GtYRPWdpXpbrWlhlNSJ64On8RlbSPIjN4X3m63jV1cVCT2PKiSPxmp1dFl1uIGuh29GfwbwTwXfS1ee4HcB+ANwP4O/TVQF/K1EA/ib7W0FEA72fmvZk/vw7gdzNvP8bMn8mLnKwABEEIpU0bvE0TugIQUxA1MEqij3EwoiaMLuq8SJ3tMUafv6y+QkxBlEBZy8WlG3uVbYrVfeLU1fm3aWldlDLSsutwL+hUcgxtO13cxTuMbSa0qw43r/MPOV9SFbICqIk23Wdcx4qkTekFurMKG/UVlqsculI+bUedLZAVQMtoU2dYR0OLTW/Vqryp5n/rpuzLd1L9CPnO5ca3Uqlb9NI0et06vmZF5au42INlYzUAiLZJPk2dqUiZ9ZbVmNYvr25wzoujLb9VPS1jQzPVj5DvztljP6eTcrq1ioFBTULqEp3lGcyb/+A+Z940NViJCKgmZInbLE2IpEbNeNqoMcptUkRALaPuitb08jc1/KrEMU2I4FI7/7Zt9rro+op6VDv/GGQAGFHq7vBCbeznkdoo8waOJuT8qWF2ZdUgOvfdZ6wGgK7PWNpMaIdfVRnkDRxNzPZGfYbZps1zIY2xGgBkxtI8E1se6YyIQxBGnbEaAMqgbZ1Xk6uaVI2hrog4qqJtdSiVslc4XcyXonE2vy/iX8q3ogUkCIIwYogWkDDyiAy6WcbpHo6qMPOw7lXQWA4ARVUkqxS7jMpGta1zKFs1VRdBqMHA1YDKHiyq7PxsdaCN4pFRNFlR56RiqndiLg9V26hbPCoiIEEQhBFDREBCIbogXhlVEUTTh/hS6UKdEQaRFYAgCMKIISsAYawY1dWAoo17ADLj7z65AwAR3UNER4joSe3ZfyeibxHRN4jofxPRGdq7m4noIBE9TURrtedXZc8OEtFN5SdFGGfauCHpE+XEDlhtPDsxiiedx21QC1kBbEX/jl+drwK4kJnfDuDbAG4GACJaDuBaAD+dffMpIppHRPMA/AmAdwFYDmBD5laogKKVuIuaSEXS3ITMvW0D1uz0qrHr/Gz1fNzuKwjaAyCixQAeYOYLLe9+GcB7mPm9RHQzADDzH2TvdgH4/czp7zPz2uz5gDsXsgcgCGnUcbPYKJtT9tEFM9917gH8OoD/k/2eAPC89u5Q9sz1XBCECqhjhTGOnT/QTnFcKoUGACL6PQDHAfxZOdEBiGgzEe0lor2vwn7jUF2UufFW5ybezLbJViwvx5k2btqWzbiJjKqiyXxMHgCI6DoA7wbwXj4pR5oFcL7m7Lzsmev5EMx8JzOvZOaVp2BBavRax/3br6gtrKUbe626g3gcGaVZoosurgDauL/VZD4mDQBEdBWAjwD4JWY+qr3aCeBaIlpAREsALAPwOIA9AJYR0RIiOhX9jeKdxaJePWU2Ypsp6nGYJQqjSxdXAGISfpAQNdDtAB4FcAERHSKi6wF8EsBrAXyViHpE9KcAwMzfBHAfgAMAvgLgA8x8gpmPA/gggF0AngJwX+a29VQpShmHWaKirs4iJZw2zgq7QBdXAG1kZttkY4OpnAQWonBpfjRx6bpO3RopdWjZCN1idnpV8gqj7PYTqgUkA4DQOEUajiAIw4gpiI4yjto70vl3U57eRbqQz3WKJGUAaBm777276SiMJG23FdRFeXoXOlOTtuaznpd1TohkAGgZTcuVU2cfTa9c8jqjqvK16XQ3SVs7066g19mm8lIGAGGA1NlHyAZW6IwxZbbeVAPq0nkLUTtuF+uXl193YtuODABCbYR20imz9S6KI+qm7WrHTZWhPjDWKX8vszxU3sW2HdECEoQSEfVQoQ2IFpAgNIB0/kKXkAFAEARhTJEBoKW0VaZdpTrlqG5ShmoKjWr6hfYiA0BLaauKXZUijrZvUqYwO70qWFOoy+lv64SlCzRpi0oGgI4jDa/dNH3Kua5VRVsnLF2gyToiA0DHkYYn+OjyqkKoHhkAhLGl7eYhiiAmroUQZAAQBhinjchRVtlsWvRUJuNUJ31UYXZEBgBhABEZCG1D6mSfKsyOyAAgCEISooDQLlJWSiFXQt5DREeI6EnLuw8TERPRG7O/iYj+mIgOEtE3iOgdmttNRPRM9m9TdEwFK9IIBSGcUd4bSVkphawAtgK4ynxIROcDeCeA57TH70L/IvhlADYDuCNzexaAjwK4FMAlAD5KRGdGx1YYQrSA2s0oy6+7WPfWbXi46Si0itwBgJn/EsBLllefAPARALo1uWsAfI77/DWAM4joXABrAXyVmV9i5pcBfBWWQUVoFymzpaZnWG1bEYn8ul3cv/2KpqPQKpL2AIjoGgCzzLzfeDUB4Hnt70PZM9dzm9+biWgvEe19FcdSoidgWGMgpmNUs9Y8TRKbGqXvm6YHBxddm6W3bZDrEqOkHVUG0QMAES0E8LsA/lP50QGY+U5mXsnMK0/BgiqCGAtMjYGY5XrorDVWjbLKxqc68RSxRNdm6V0UvXSJrk0IipCyAngLgCUA9hPRswDOA/B1IvpJALMAztfcnpc9cz0XhFLoWicutJdxqkvRAwAzP8HMZzPzYmZejL445x3M/CKAnQDel2kDXQbgFWZ+AcAuAO8kojOzzd93Zs8EQRCEhghRA90O4FEAFxDRISK63uP8ywBmABwE8D8A3AAAzPwSgP8KYE/2779kzwRBEISGCNEC2sDM5zLzKcx8HjPfbbxfzMx/n/1mZv4AM7+FmS9i5r2au3uY+a3Zv8+UnxSh7cjmZTqjbLdoVGirkoMPuRNYEARhxJA7gQVBaCWyEmwPMgAIglArosbaHmQAEARBGFNkABCsVGF7vMmlfxc36JpCRDRpxB4ga0M+yyawIAjCiCGbwIIgCIIXGQAEYQSpw55NG0QYo0bddohkABBKoYo9AyGdOuzZiDZP+dRth0gGAKEUTOujbbOoKJvAgjCMDABCJfhmMmUNDjHmEcbNDryIZ4QQZAAQaqesZW7sfQTjRB3iGVlVdR8ZAISRwjXzbZtIqinK7LTHbVWVRxdXXXIOQBAEYcSQcwCC0DAiImkHsvpzIwOAIFSEiEjawThd8RiLDACCIAhjSsiVkPcQ0REietJ4/ptE9C0i+iYR/Tft+c1EdJCIniaitdrzq7JnB4nopnKTIQhC3XRx01MYZH6Am60APgngc+oBEf08gGsAXMzMx4jo7Oz5cgDXAvhpAIsA/AURvS377E8A/CL6l8jvIaKdzHygrIQIglAvchK4+4TcCfyXAMwL3P89gI8z87HMzZHs+TUAPs/Mx5j5O+hfDn9J9u8gM88w8z8D+HzmVhBqQe7UFYRhUvcA3gbgCiJ6jIj+LxFNZc8nADyvuTuUPXM9H4KINhPRXiLa+yqOJUZPEAaRQ2OCMEzqADAfwFkALgPwHwDcR0RURoSY+U5mXsnMK0/BgjK8FIRaEJl4+xGjhYOE7AHYOATgi9w/RfY4Ef0YwBsBzAI4X3N3XvYMnueCMBKITLz9mEYLx53UFcD9AH4eALJN3lMB/D2AnQCuJaIFRLQEwDIAjwPYA2AZES0holPR3yjeWTTygiC4kQNQQh4haqDbATwK4AIiOkRE1wO4B8DSTDX08wA2cZ9vArgPwAEAXwHwAWY+wczHAXwQwC4ATwG4L3MrCEJFVH0ASkRe3SdXBMTMGxyvftXh/mMAPmZ5/mUAX46KnSAIglAZchJYEIQkFuxf2HQUhILIACAIQhJi66j7yAAgCMLYIBZaB5EBQBCEsUGtWuQ8QB8ZAARBGDvqPA/QZm2pVt8IRkTfA/BD9M8YCMO8EZI3LiRv3EjeuBmVvPmXzPymPEetHgAAgIj2hlxtNo5I3riRvHEjeeNm3PJGRECCIAhjigwAgiAIY0oXBoA7m45Ai5G8cSN540byxs1Y5U3r9wAEQRCEaujCCkAQBEGoABkABEEQxpTWDgBEdBURPU1EB4nopqbjUzdEdD4RfY2IDhDRN4noQ9nzs4joq0T0TPb/mdlzIqI/zvLrG0T0jmZTUD1ENI+I/oaIHsj+XpJdU3qQiHZkd08gu59iR/b8MSJa3GS8q4aIziCiLxDRt4joKSK6XOpNHyL6naw9PUlE24noNeNcb1o5ABDRPAB/AuBdAJYD2EBEy5uNVe0cB/BhZl6O/tWbH8jy4CYAu5l5GYDd2d9AP6+WZf82A7ij/ijXzofQv19CsQXAJ5j5rQBeBnB99vx6AC9nzz+RuRtlbgfwFWb+KQAXo59HY19viGgCwG8BWMnMFwKYh/7lVONbb5i5df8AXA5gl/b3zQBubjpeDefJlwD8IoCnAZybPTsXwNPZ708D2KC5n3M3iv/Qv1Z0N4A1AB4AQOif4Jxv1iH0LyK6PPs9P3NHTaehonx5PYDvmOmTesMAMAHgefTvM5+f1Zu141xvWrkCwMmCUhzKno0l2dLzZwA8BuAcZn4he/UigHOy3+OWZ38E4CMAfpz9/QYAP+D+7XPAYPrn8iZ7/0rmfhRZAuB7AD6TicfuIqLTIfUGzDwL4A8BPAfgBfTrwT6Mcb1p6wAgZBDRTwD4cwC/zcz/oL/j/tRk7PR4iejdAI4ws9zwPcx8AO8AcAcz/wz6trQG9tDGuN6cCeAa9AfJRQBOB3BVo5FqmLYOALMAztf+Pi97NlYQ0Snod/5/xsxfzB5/l4jOzd6fC+BI9nyc8uxnAfwSET2L/p3Ua9CXe59BROqaUz39c3mTvX89gO/XGeEaOQTgEDM/lv39BfQHBKk3wC8A+A4zf4+ZXwXwRfTr0tjWm7YOAHsALMt2509Ff6NmZ8NxqhUiIgB3A3iKmW/TXu0EsCn7vQn9vQH1/H2ZVsdlAF7RlvwjBTPfzMznMfNi9OvGg8z8XgBfA/CezJmZNyrP3pO5H8kZMDO/COB5Iroge3QlgAOQegP0RT+XEdHCrH2pvBnfetP0JoRnw+ZqAN8G8LcAfq/p+DSQ/p9Df5n+DQC97N/V6MsgdwN4BsBfADgrc0/oa079LYAn0Nd0aDwdNeTTagAPZL+XAngcwEEA/wvAguz5a7K/D2bvlzYd7xzoVQkAAABfSURBVIrzZBLA3qzu3A/gTKk3c3nznwF8C8CTAP4ngAXjXG/EFIQgCMKY0lYRkCAIglAxMgAIgiCMKTIACIIgjCkyAAiCIIwpMgAIgiCMKTIACIIgjCkyAAiCIIwp/x+DwjcT8Oje3gAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.imshow(Y, aspect='auto')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Collaborative filtering learning algorithm\n", "\n", "**Exercise**: Implement collaborative filtering cost function." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "def cofi_cost(params, Y, R, num_users, num_movies, num_features, _lambda):\n", " \"\"\"\n", " Collaborative filtering cost function\n", " \"\"\"\n", " \n", " # unpack values\n", " X = params[:num_movies*num_features].reshape((num_movies, num_features))\n", " Theta = params[num_movies*num_features:].reshape((num_users, num_features))\n", " \n", " X_grad = np.zeros_like(X)\n", " \n", " return X_grad" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "def cofi_gradient(params, Y, R, num_users, num_movies, num_features, _lambda):\n", " Theta = params[num_movies*num_features:].reshape((num_users, num_features))\n", " Theta_grad = np.zeros_like(X)\n", " \n", " return Theta_grad" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll load pre-trained weights to examine the cost function." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "data = sio.loadmat('ex8_movieParams.mat')\n", "Theta = data[\"Theta\"]\n", "X = data[\"X\"]\n", "\n", "# Reduce the data set size so that this runs faster\n", "num_users = 4\n", "num_movies = 5\n", "num_features = 3\n", "X_ = X[:num_movies, :num_features]\n", "Theta_ = Theta[:num_users, :num_features]\n", "Y_ = Y[:num_movies, :num_users]\n", "R_ = R[:num_movies, :num_users]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following should return approximately $22.22$." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Cost at loaded parameters: 22.224603725685675'" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "J = cofi_cost(np.hstack((X_.flatten(), Theta_.flatten())), Y_, R_, num_users, num_movies, num_features, 0)\n", " \n", "'Cost at loaded parameters: {}'.format(J)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following should return approximately $31.34$." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Cost at loaded parameters: 31.34405624427422'" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "J = cofi_cost(np.hstack((X_.flatten(), Theta_.flatten())), Y_, R_, num_users, num_movies, num_features, 1.5)\n", " \n", "'Cost at loaded parameters: {}'.format(J)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training the model" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Toy Story (1995)\\n'" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from load_movie_list import load_movie_list\n", "movie_list = load_movie_list()\n", "movie_list[0]" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Rated 4 for Toy Story (1995)\n", "Rated 3 for Twelve Monkeys (1995)\n", "Rated 5 for Usual Suspects, The (1995)\n", "Rated 4 for Outbreak (1995)\n", "Rated 5 for Shawshank Redemption, The (1994)\n", "Rated 3 for While You Were Sleeping (1995)\n", "Rated 5 for Forrest Gump (1994)\n", "Rated 2 for Silence of the Lambs, The (1991)\n", "Rated 4 for Alien (1979)\n", "Rated 5 for Die Hard 2 (1990)\n", "Rated 5 for Sphere (1998)\n" ] } ], "source": [ "# Initialize new user ratings\n", "new_ratings = np.zeros((1682, 1))\n", "\n", "# Check the file movie_idx.txt for id of each movie in our dataset\n", "# For example, Toy Story (1995) has ID 1, so to rate it \"4\", you can set\n", "new_ratings[0] = 4\n", "\n", "# Or suppose did not enjoy Silence of the Lambs (1991), you can set\n", "new_ratings[97] = 2\n", "\n", "# We have selected a few movies we liked / did not like and the ratings we\n", "# gave are as follows:\n", "new_ratings[6] = 3\n", "new_ratings[11]= 5\n", "new_ratings[53] = 4\n", "new_ratings[63]= 5\n", "new_ratings[65]= 3\n", "new_ratings[68] = 5\n", "new_ratings[182] = 4\n", "new_ratings[225] = 5\n", "new_ratings[354]= 5\n", "\n", "for i in range(len(new_ratings)):\n", " rating = new_ratings[i]\n", " if rating > 0:\n", " print(\"Rated {} for {}\".format(int(rating), movie_list[i]), end='')" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "# Add new ratings to dataset\n", "Y = np.hstack((new_ratings, Y))\n", "R = np.hstack((new_ratings != 0, R))" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1 (1682,)\n", "2 (944,)\n", "num_users: 944 num_movies : 1682 num_features : 10\n", "Set initial parameters\n", "Train\n", "Warning: Desired error not necessarily achieved due to precision loss.\n", " Current function value: 152936.114754\n", " Iterations: 1\n", " Function evaluations: 112\n", " Gradient evaluations: 100\n", "Extract results\n", "[1.48354407 0.92772736 0.89064199 ... 0.34407187 0.99373252 0.53377299]\n", "(944,) (944,)\n", "Rated 11.15616688472356 for Toy Story (1995)\n", "Rated 11.108525219012083 for GoldenEye (1995)\n", "Rated 10.079762087907355 for Four Rooms (1995)\n", "Rated 9.851467527118134 for Get Shorty (1995)\n", "Rated 9.790186134596931 for Copycat (1995)\n", "Rated 9.643464976885834 for Shanghai Triad (Yao a yao yao dao waipo qiao) (1995)\n", "Rated 9.471699702534927 for Twelve Monkeys (1995)\n", "Rated 9.424885341163584 for Babe (1995)\n", "Rated 9.414902508479424 for Dead Man Walking (1995)\n", "Rated 9.05165410342046 for Richard III (1995)\n" ] } ], "source": [ "from scipy.optimize import fmin_cg\n", "\n", "# Perform normalization\n", "Ymean, Ynorm = normalize_ratings(Y, R)\n", "\n", "num_users = Y.shape[1]\n", "num_movies = Y.shape[0]\n", "num_features = 10\n", "print('num_users: ', num_users,' num_movies : ', num_movies,' num_features : ', num_features)\n", "\n", "# Set initial parameters\n", "print(\"Set initial parameters\")\n", "X = np.random.random((num_movies, num_features))\n", "Theta = np.random.random((num_users, num_features))\n", "initial_parameters = np.hstack((X.flatten(), Theta.flatten()))\n", "\n", "# Train\n", "print(\"Train\")\n", "_lambda = 10\n", "results = fmin_cg(cofi_cost, initial_parameters, args = (Y, R, num_users, num_movies, num_features, _lambda),\n", " fprime = cofi_gradient)\n", "\n", "# Extract results\n", "print(\"Extract results\")\n", "print(results)\n", "params = results\n", "X = params[:num_movies*num_features].reshape((num_movies, num_features))\n", "Theta = params[num_movies*num_features:].reshape((num_users, num_features))\n", "\n", "p = X @ Theta.T\n", "predictions = p[0, :] + Ymean\n", "print(predictions.shape, Ymean.shape)\n", "for i, prediction in enumerate(list(sorted(predictions, reverse=True))[:10]):\n", " if prediction > 0:\n", " print(\"Rated {} for {}\".format(prediction, movie_list[i]), end='')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: ex8/load_movie_list.py ================================================ def load_movie_list(): movies = [] with open('movie_ids.txt', 'r', encoding = "ISO-8859-1") as f: for line in f.readlines(): movie = ' '.join(line.split(' ')[1:]) movies.append(movie) return movies if __name__ == '__main__': movies = load_movie_list() print(movies[0]) ================================================ FILE: ex8/movie_ids.txt ================================================ 1 Toy Story (1995) 2 GoldenEye (1995) 3 Four Rooms (1995) 4 Get Shorty (1995) 5 Copycat (1995) 6 Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 7 Twelve Monkeys (1995) 8 Babe (1995) 9 Dead Man Walking (1995) 10 Richard III (1995) 11 Seven (Se7en) (1995) 12 Usual Suspects, The (1995) 13 Mighty Aphrodite (1995) 14 Postino, Il (1994) 15 Mr. Holland's Opus (1995) 16 French Twist (Gazon maudit) (1995) 17 From Dusk Till Dawn (1996) 18 White Balloon, The (1995) 19 Antonia's Line (1995) 20 Angels and Insects (1995) 21 Muppet Treasure Island (1996) 22 Braveheart (1995) 23 Taxi Driver (1976) 24 Rumble in the Bronx (1995) 25 Birdcage, The (1996) 26 Brothers McMullen, The (1995) 27 Bad Boys (1995) 28 Apollo 13 (1995) 29 Batman Forever (1995) 30 Belle de jour (1967) 31 Crimson Tide (1995) 32 Crumb (1994) 33 Desperado (1995) 34 Doom Generation, The (1995) 35 Free Willy 2: The Adventure Home (1995) 36 Mad Love (1995) 37 Nadja (1994) 38 Net, The (1995) 39 Strange Days (1995) 40 To Wong Foo, Thanks for Everything! Julie Newmar (1995) 41 Billy Madison (1995) 42 Clerks (1994) 43 Disclosure (1994) 44 Dolores Claiborne (1994) 45 Eat Drink Man Woman (1994) 46 Exotica (1994) 47 Ed Wood (1994) 48 Hoop Dreams (1994) 49 I.Q. (1994) 50 Star Wars (1977) 51 Legends of the Fall (1994) 52 Madness of King George, The (1994) 53 Natural Born Killers (1994) 54 Outbreak (1995) 55 Professional, The (1994) 56 Pulp Fiction (1994) 57 Priest (1994) 58 Quiz Show (1994) 59 Three Colors: Red (1994) 60 Three Colors: Blue (1993) 61 Three Colors: White (1994) 62 Stargate (1994) 63 Santa Clause, The (1994) 64 Shawshank Redemption, The (1994) 65 What's Eating Gilbert Grape (1993) 66 While You Were Sleeping (1995) 67 Ace Ventura: Pet Detective (1994) 68 Crow, The (1994) 69 Forrest Gump (1994) 70 Four Weddings and a Funeral (1994) 71 Lion King, The (1994) 72 Mask, The (1994) 73 Maverick (1994) 74 Faster Pussycat! Kill! Kill! (1965) 75 Brother Minister: The Assassination of Malcolm X (1994) 76 Carlito's Way (1993) 77 Firm, The (1993) 78 Free Willy (1993) 79 Fugitive, The (1993) 80 Hot Shots! Part Deux (1993) 81 Hudsucker Proxy, The (1994) 82 Jurassic Park (1993) 83 Much Ado About Nothing (1993) 84 Robert A. Heinlein's The Puppet Masters (1994) 85 Ref, The (1994) 86 Remains of the Day, The (1993) 87 Searching for Bobby Fischer (1993) 88 Sleepless in Seattle (1993) 89 Blade Runner (1982) 90 So I Married an Axe Murderer (1993) 91 Nightmare Before Christmas, The (1993) 92 True Romance (1993) 93 Welcome to the Dollhouse (1995) 94 Home Alone (1990) 95 Aladdin (1992) 96 Terminator 2: Judgment Day (1991) 97 Dances with Wolves (1990) 98 Silence of the Lambs, The (1991) 99 Snow White and the Seven Dwarfs (1937) 100 Fargo (1996) 101 Heavy Metal (1981) 102 Aristocats, The (1970) 103 All Dogs Go to Heaven 2 (1996) 104 Theodore Rex (1995) 105 Sgt. Bilko (1996) 106 Diabolique (1996) 107 Moll Flanders (1996) 108 Kids in the Hall: Brain Candy (1996) 109 Mystery Science Theater 3000: The Movie (1996) 110 Operation Dumbo Drop (1995) 111 Truth About Cats & Dogs, The (1996) 112 Flipper (1996) 113 Horseman on the Roof, The (Hussard sur le toit, Le) (1995) 114 Wallace & Gromit: The Best of Aardman Animation (1996) 115 Haunted World of Edward D. Wood Jr., The (1995) 116 Cold Comfort Farm (1995) 117 Rock, The (1996) 118 Twister (1996) 119 Maya Lin: A Strong Clear Vision (1994) 120 Striptease (1996) 121 Independence Day (ID4) (1996) 122 Cable Guy, The (1996) 123 Frighteners, The (1996) 124 Lone Star (1996) 125 Phenomenon (1996) 126 Spitfire Grill, The (1996) 127 Godfather, The (1972) 128 Supercop (1992) 129 Bound (1996) 130 Kansas City (1996) 131 Breakfast at Tiffany's (1961) 132 Wizard of Oz, The (1939) 133 Gone with the Wind (1939) 134 Citizen Kane (1941) 135 2001: A Space Odyssey (1968) 136 Mr. Smith Goes to Washington (1939) 137 Big Night (1996) 138 D3: The Mighty Ducks (1996) 139 Love Bug, The (1969) 140 Homeward Bound: The Incredible Journey (1993) 141 20,000 Leagues Under the Sea (1954) 142 Bedknobs and Broomsticks (1971) 143 Sound of Music, The (1965) 144 Die Hard (1988) 145 Lawnmower Man, The (1992) 146 Unhook the Stars (1996) 147 Long Kiss Goodnight, The (1996) 148 Ghost and the Darkness, The (1996) 149 Jude (1996) 150 Swingers (1996) 151 Willy Wonka and the Chocolate Factory (1971) 152 Sleeper (1973) 153 Fish Called Wanda, A (1988) 154 Monty Python's Life of Brian (1979) 155 Dirty Dancing (1987) 156 Reservoir Dogs (1992) 157 Platoon (1986) 158 Weekend at Bernie's (1989) 159 Basic Instinct (1992) 160 Glengarry Glen Ross (1992) 161 Top Gun (1986) 162 On Golden Pond (1981) 163 Return of the Pink Panther, The (1974) 164 Abyss, The (1989) 165 Jean de Florette (1986) 166 Manon of the Spring (Manon des sources) (1986) 167 Private Benjamin (1980) 168 Monty Python and the Holy Grail (1974) 169 Wrong Trousers, The (1993) 170 Cinema Paradiso (1988) 171 Delicatessen (1991) 172 Empire Strikes Back, The (1980) 173 Princess Bride, The (1987) 174 Raiders of the Lost Ark (1981) 175 Brazil (1985) 176 Aliens (1986) 177 Good, The Bad and The Ugly, The (1966) 178 12 Angry Men (1957) 179 Clockwork Orange, A (1971) 180 Apocalypse Now (1979) 181 Return of the Jedi (1983) 182 GoodFellas (1990) 183 Alien (1979) 184 Army of Darkness (1993) 185 Psycho (1960) 186 Blues Brothers, The (1980) 187 Godfather: Part II, The (1974) 188 Full Metal Jacket (1987) 189 Grand Day Out, A (1992) 190 Henry V (1989) 191 Amadeus (1984) 192 Raging Bull (1980) 193 Right Stuff, The (1983) 194 Sting, The (1973) 195 Terminator, The (1984) 196 Dead Poets Society (1989) 197 Graduate, The (1967) 198 Nikita (La Femme Nikita) (1990) 199 Bridge on the River Kwai, The (1957) 200 Shining, The (1980) 201 Evil Dead II (1987) 202 Groundhog Day (1993) 203 Unforgiven (1992) 204 Back to the Future (1985) 205 Patton (1970) 206 Akira (1988) 207 Cyrano de Bergerac (1990) 208 Young Frankenstein (1974) 209 This Is Spinal Tap (1984) 210 Indiana Jones and the Last Crusade (1989) 211 M*A*S*H (1970) 212 Unbearable Lightness of Being, The (1988) 213 Room with a View, A (1986) 214 Pink Floyd - The Wall (1982) 215 Field of Dreams (1989) 216 When Harry Met Sally... (1989) 217 Bram Stoker's Dracula (1992) 218 Cape Fear (1991) 219 Nightmare on Elm Street, A (1984) 220 Mirror Has Two Faces, The (1996) 221 Breaking the Waves (1996) 222 Star Trek: First Contact (1996) 223 Sling Blade (1996) 224 Ridicule (1996) 225 101 Dalmatians (1996) 226 Die Hard 2 (1990) 227 Star Trek VI: The Undiscovered Country (1991) 228 Star Trek: The Wrath of Khan (1982) 229 Star Trek III: The Search for Spock (1984) 230 Star Trek IV: The Voyage Home (1986) 231 Batman Returns (1992) 232 Young Guns (1988) 233 Under Siege (1992) 234 Jaws (1975) 235 Mars Attacks! (1996) 236 Citizen Ruth (1996) 237 Jerry Maguire (1996) 238 Raising Arizona (1987) 239 Sneakers (1992) 240 Beavis and Butt-head Do America (1996) 241 Last of the Mohicans, The (1992) 242 Kolya (1996) 243 Jungle2Jungle (1997) 244 Smilla's Sense of Snow (1997) 245 Devil's Own, The (1997) 246 Chasing Amy (1997) 247 Turbo: A Power Rangers Movie (1997) 248 Grosse Pointe Blank (1997) 249 Austin Powers: International Man of Mystery (1997) 250 Fifth Element, The (1997) 251 Shall We Dance? (1996) 252 Lost World: Jurassic Park, The (1997) 253 Pillow Book, The (1995) 254 Batman & Robin (1997) 255 My Best Friend's Wedding (1997) 256 When the Cats Away (Chacun cherche son chat) (1996) 257 Men in Black (1997) 258 Contact (1997) 259 George of the Jungle (1997) 260 Event Horizon (1997) 261 Air Bud (1997) 262 In the Company of Men (1997) 263 Steel (1997) 264 Mimic (1997) 265 Hunt for Red October, The (1990) 266 Kull the Conqueror (1997) 267 unknown 268 Chasing Amy (1997) 269 Full Monty, The (1997) 270 Gattaca (1997) 271 Starship Troopers (1997) 272 Good Will Hunting (1997) 273 Heat (1995) 274 Sabrina (1995) 275 Sense and Sensibility (1995) 276 Leaving Las Vegas (1995) 277 Restoration (1995) 278 Bed of Roses (1996) 279 Once Upon a Time... When We Were Colored (1995) 280 Up Close and Personal (1996) 281 River Wild, The (1994) 282 Time to Kill, A (1996) 283 Emma (1996) 284 Tin Cup (1996) 285 Secrets & Lies (1996) 286 English Patient, The (1996) 287 Marvin's Room (1996) 288 Scream (1996) 289 Evita (1996) 290 Fierce Creatures (1997) 291 Absolute Power (1997) 292 Rosewood (1997) 293 Donnie Brasco (1997) 294 Liar Liar (1997) 295 Breakdown (1997) 296 Promesse, La (1996) 297 Ulee's Gold (1997) 298 Face/Off (1997) 299 Hoodlum (1997) 300 Air Force One (1997) 301 In & Out (1997) 302 L.A. Confidential (1997) 303 Ulee's Gold (1997) 304 Fly Away Home (1996) 305 Ice Storm, The (1997) 306 Mrs. Brown (Her Majesty, Mrs. Brown) (1997) 307 Devil's Advocate, The (1997) 308 FairyTale: A True Story (1997) 309 Deceiver (1997) 310 Rainmaker, The (1997) 311 Wings of the Dove, The (1997) 312 Midnight in the Garden of Good and Evil (1997) 313 Titanic (1997) 314 3 Ninjas: High Noon At Mega Mountain (1998) 315 Apt Pupil (1998) 316 As Good As It Gets (1997) 317 In the Name of the Father (1993) 318 Schindler's List (1993) 319 Everyone Says I Love You (1996) 320 Paradise Lost: The Child Murders at Robin Hood Hills (1996) 321 Mother (1996) 322 Murder at 1600 (1997) 323 Dante's Peak (1997) 324 Lost Highway (1997) 325 Crash (1996) 326 G.I. Jane (1997) 327 Cop Land (1997) 328 Conspiracy Theory (1997) 329 Desperate Measures (1998) 330 187 (1997) 331 Edge, The (1997) 332 Kiss the Girls (1997) 333 Game, The (1997) 334 U Turn (1997) 335 How to Be a Player (1997) 336 Playing God (1997) 337 House of Yes, The (1997) 338 Bean (1997) 339 Mad City (1997) 340 Boogie Nights (1997) 341 Critical Care (1997) 342 Man Who Knew Too Little, The (1997) 343 Alien: Resurrection (1997) 344 Apostle, The (1997) 345 Deconstructing Harry (1997) 346 Jackie Brown (1997) 347 Wag the Dog (1997) 348 Desperate Measures (1998) 349 Hard Rain (1998) 350 Fallen (1998) 351 Prophecy II, The (1998) 352 Spice World (1997) 353 Deep Rising (1998) 354 Wedding Singer, The (1998) 355 Sphere (1998) 356 Client, The (1994) 357 One Flew Over the Cuckoo's Nest (1975) 358 Spawn (1997) 359 Assignment, The (1997) 360 Wonderland (1997) 361 Incognito (1997) 362 Blues Brothers 2000 (1998) 363 Sudden Death (1995) 364 Ace Ventura: When Nature Calls (1995) 365 Powder (1995) 366 Dangerous Minds (1995) 367 Clueless (1995) 368 Bio-Dome (1996) 369 Black Sheep (1996) 370 Mary Reilly (1996) 371 Bridges of Madison County, The (1995) 372 Jeffrey (1995) 373 Judge Dredd (1995) 374 Mighty Morphin Power Rangers: The Movie (1995) 375 Showgirls (1995) 376 Houseguest (1994) 377 Heavyweights (1994) 378 Miracle on 34th Street (1994) 379 Tales From the Crypt Presents: Demon Knight (1995) 380 Star Trek: Generations (1994) 381 Muriel's Wedding (1994) 382 Adventures of Priscilla, Queen of the Desert, The (1994) 383 Flintstones, The (1994) 384 Naked Gun 33 1/3: The Final Insult (1994) 385 True Lies (1994) 386 Addams Family Values (1993) 387 Age of Innocence, The (1993) 388 Beverly Hills Cop III (1994) 389 Black Beauty (1994) 390 Fear of a Black Hat (1993) 391 Last Action Hero (1993) 392 Man Without a Face, The (1993) 393 Mrs. Doubtfire (1993) 394 Radioland Murders (1994) 395 Robin Hood: Men in Tights (1993) 396 Serial Mom (1994) 397 Striking Distance (1993) 398 Super Mario Bros. (1993) 399 Three Musketeers, The (1993) 400 Little Rascals, The (1994) 401 Brady Bunch Movie, The (1995) 402 Ghost (1990) 403 Batman (1989) 404 Pinocchio (1940) 405 Mission: Impossible (1996) 406 Thinner (1996) 407 Spy Hard (1996) 408 Close Shave, A (1995) 409 Jack (1996) 410 Kingpin (1996) 411 Nutty Professor, The (1996) 412 Very Brady Sequel, A (1996) 413 Tales from the Crypt Presents: Bordello of Blood (1996) 414 My Favorite Year (1982) 415 Apple Dumpling Gang, The (1975) 416 Old Yeller (1957) 417 Parent Trap, The (1961) 418 Cinderella (1950) 419 Mary Poppins (1964) 420 Alice in Wonderland (1951) 421 William Shakespeare's Romeo and Juliet (1996) 422 Aladdin and the King of Thieves (1996) 423 E.T. the Extra-Terrestrial (1982) 424 Children of the Corn: The Gathering (1996) 425 Bob Roberts (1992) 426 Transformers: The Movie, The (1986) 427 To Kill a Mockingbird (1962) 428 Harold and Maude (1971) 429 Day the Earth Stood Still, The (1951) 430 Duck Soup (1933) 431 Highlander (1986) 432 Fantasia (1940) 433 Heathers (1989) 434 Forbidden Planet (1956) 435 Butch Cassidy and the Sundance Kid (1969) 436 American Werewolf in London, An (1981) 437 Amityville 1992: It's About Time (1992) 438 Amityville 3-D (1983) 439 Amityville: A New Generation (1993) 440 Amityville II: The Possession (1982) 441 Amityville Horror, The (1979) 442 Amityville Curse, The (1990) 443 Birds, The (1963) 444 Blob, The (1958) 445 Body Snatcher, The (1945) 446 Burnt Offerings (1976) 447 Carrie (1976) 448 Omen, The (1976) 449 Star Trek: The Motion Picture (1979) 450 Star Trek V: The Final Frontier (1989) 451 Grease (1978) 452 Jaws 2 (1978) 453 Jaws 3-D (1983) 454 Bastard Out of Carolina (1996) 455 Jackie Chan's First Strike (1996) 456 Beverly Hills Ninja (1997) 457 Free Willy 3: The Rescue (1997) 458 Nixon (1995) 459 Cry, the Beloved Country (1995) 460 Crossing Guard, The (1995) 461 Smoke (1995) 462 Like Water For Chocolate (Como agua para chocolate) (1992) 463 Secret of Roan Inish, The (1994) 464 Vanya on 42nd Street (1994) 465 Jungle Book, The (1994) 466 Red Rock West (1992) 467 Bronx Tale, A (1993) 468 Rudy (1993) 469 Short Cuts (1993) 470 Tombstone (1993) 471 Courage Under Fire (1996) 472 Dragonheart (1996) 473 James and the Giant Peach (1996) 474 Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1963) 475 Trainspotting (1996) 476 First Wives Club, The (1996) 477 Matilda (1996) 478 Philadelphia Story, The (1940) 479 Vertigo (1958) 480 North by Northwest (1959) 481 Apartment, The (1960) 482 Some Like It Hot (1959) 483 Casablanca (1942) 484 Maltese Falcon, The (1941) 485 My Fair Lady (1964) 486 Sabrina (1954) 487 Roman Holiday (1953) 488 Sunset Blvd. (1950) 489 Notorious (1946) 490 To Catch a Thief (1955) 491 Adventures of Robin Hood, The (1938) 492 East of Eden (1955) 493 Thin Man, The (1934) 494 His Girl Friday (1940) 495 Around the World in 80 Days (1956) 496 It's a Wonderful Life (1946) 497 Bringing Up Baby (1938) 498 African Queen, The (1951) 499 Cat on a Hot Tin Roof (1958) 500 Fly Away Home (1996) 501 Dumbo (1941) 502 Bananas (1971) 503 Candidate, The (1972) 504 Bonnie and Clyde (1967) 505 Dial M for Murder (1954) 506 Rebel Without a Cause (1955) 507 Streetcar Named Desire, A (1951) 508 People vs. Larry Flynt, The (1996) 509 My Left Foot (1989) 510 Magnificent Seven, The (1954) 511 Lawrence of Arabia (1962) 512 Wings of Desire (1987) 513 Third Man, The (1949) 514 Annie Hall (1977) 515 Boot, Das (1981) 516 Local Hero (1983) 517 Manhattan (1979) 518 Miller's Crossing (1990) 519 Treasure of the Sierra Madre, The (1948) 520 Great Escape, The (1963) 521 Deer Hunter, The (1978) 522 Down by Law (1986) 523 Cool Hand Luke (1967) 524 Great Dictator, The (1940) 525 Big Sleep, The (1946) 526 Ben-Hur (1959) 527 Gandhi (1982) 528 Killing Fields, The (1984) 529 My Life as a Dog (Mitt liv som hund) (1985) 530 Man Who Would Be King, The (1975) 531 Shine (1996) 532 Kama Sutra: A Tale of Love (1996) 533 Daytrippers, The (1996) 534 Traveller (1997) 535 Addicted to Love (1997) 536 Ponette (1996) 537 My Own Private Idaho (1991) 538 Anastasia (1997) 539 Mouse Hunt (1997) 540 Money Train (1995) 541 Mortal Kombat (1995) 542 Pocahontas (1995) 543 Misrables, Les (1995) 544 Things to Do in Denver when You're Dead (1995) 545 Vampire in Brooklyn (1995) 546 Broken Arrow (1996) 547 Young Poisoner's Handbook, The (1995) 548 NeverEnding Story III, The (1994) 549 Rob Roy (1995) 550 Die Hard: With a Vengeance (1995) 551 Lord of Illusions (1995) 552 Species (1995) 553 Walk in the Clouds, A (1995) 554 Waterworld (1995) 555 White Man's Burden (1995) 556 Wild Bill (1995) 557 Farinelli: il castrato (1994) 558 Heavenly Creatures (1994) 559 Interview with the Vampire (1994) 560 Kid in King Arthur's Court, A (1995) 561 Mary Shelley's Frankenstein (1994) 562 Quick and the Dead, The (1995) 563 Stephen King's The Langoliers (1995) 564 Tales from the Hood (1995) 565 Village of the Damned (1995) 566 Clear and Present Danger (1994) 567 Wes Craven's New Nightmare (1994) 568 Speed (1994) 569 Wolf (1994) 570 Wyatt Earp (1994) 571 Another Stakeout (1993) 572 Blown Away (1994) 573 Body Snatchers (1993) 574 Boxing Helena (1993) 575 City Slickers II: The Legend of Curly's Gold (1994) 576 Cliffhanger (1993) 577 Coneheads (1993) 578 Demolition Man (1993) 579 Fatal Instinct (1993) 580 Englishman Who Went Up a Hill, But Came Down a Mountain, The (1995) 581 Kalifornia (1993) 582 Piano, The (1993) 583 Romeo Is Bleeding (1993) 584 Secret Garden, The (1993) 585 Son in Law (1993) 586 Terminal Velocity (1994) 587 Hour of the Pig, The (1993) 588 Beauty and the Beast (1991) 589 Wild Bunch, The (1969) 590 Hellraiser: Bloodline (1996) 591 Primal Fear (1996) 592 True Crime (1995) 593 Stalingrad (1993) 594 Heavy (1995) 595 Fan, The (1996) 596 Hunchback of Notre Dame, The (1996) 597 Eraser (1996) 598 Big Squeeze, The (1996) 599 Police Story 4: Project S (Chao ji ji hua) (1993) 600 Daniel Defoe's Robinson Crusoe (1996) 601 For Whom the Bell Tolls (1943) 602 American in Paris, An (1951) 603 Rear Window (1954) 604 It Happened One Night (1934) 605 Meet Me in St. Louis (1944) 606 All About Eve (1950) 607 Rebecca (1940) 608 Spellbound (1945) 609 Father of the Bride (1950) 610 Gigi (1958) 611 Laura (1944) 612 Lost Horizon (1937) 613 My Man Godfrey (1936) 614 Giant (1956) 615 39 Steps, The (1935) 616 Night of the Living Dead (1968) 617 Blue Angel, The (Blaue Engel, Der) (1930) 618 Picnic (1955) 619 Extreme Measures (1996) 620 Chamber, The (1996) 621 Davy Crockett, King of the Wild Frontier (1955) 622 Swiss Family Robinson (1960) 623 Angels in the Outfield (1994) 624 Three Caballeros, The (1945) 625 Sword in the Stone, The (1963) 626 So Dear to My Heart (1949) 627 Robin Hood: Prince of Thieves (1991) 628 Sleepers (1996) 629 Victor/Victoria (1982) 630 Great Race, The (1965) 631 Crying Game, The (1992) 632 Sophie's Choice (1982) 633 Christmas Carol, A (1938) 634 Microcosmos: Le peuple de l'herbe (1996) 635 Fog, The (1980) 636 Escape from New York (1981) 637 Howling, The (1981) 638 Return of Martin Guerre, The (Retour de Martin Guerre, Le) (1982) 639 Tin Drum, The (Blechtrommel, Die) (1979) 640 Cook the Thief His Wife & Her Lover, The (1989) 641 Paths of Glory (1957) 642 Grifters, The (1990) 643 The Innocent (1994) 644 Thin Blue Line, The (1988) 645 Paris Is Burning (1990) 646 Once Upon a Time in the West (1969) 647 Ran (1985) 648 Quiet Man, The (1952) 649 Once Upon a Time in America (1984) 650 Seventh Seal, The (Sjunde inseglet, Det) (1957) 651 Glory (1989) 652 Rosencrantz and Guildenstern Are Dead (1990) 653 Touch of Evil (1958) 654 Chinatown (1974) 655 Stand by Me (1986) 656 M (1931) 657 Manchurian Candidate, The (1962) 658 Pump Up the Volume (1990) 659 Arsenic and Old Lace (1944) 660 Fried Green Tomatoes (1991) 661 High Noon (1952) 662 Somewhere in Time (1980) 663 Being There (1979) 664 Paris, Texas (1984) 665 Alien 3 (1992) 666 Blood For Dracula (Andy Warhol's Dracula) (1974) 667 Audrey Rose (1977) 668 Blood Beach (1981) 669 Body Parts (1991) 670 Body Snatchers (1993) 671 Bride of Frankenstein (1935) 672 Candyman (1992) 673 Cape Fear (1962) 674 Cat People (1982) 675 Nosferatu (Nosferatu, eine Symphonie des Grauens) (1922) 676 Crucible, The (1996) 677 Fire on the Mountain (1996) 678 Volcano (1997) 679 Conan the Barbarian (1981) 680 Kull the Conqueror (1997) 681 Wishmaster (1997) 682 I Know What You Did Last Summer (1997) 683 Rocket Man (1997) 684 In the Line of Fire (1993) 685 Executive Decision (1996) 686 Perfect World, A (1993) 687 McHale's Navy (1997) 688 Leave It to Beaver (1997) 689 Jackal, The (1997) 690 Seven Years in Tibet (1997) 691 Dark City (1998) 692 American President, The (1995) 693 Casino (1995) 694 Persuasion (1995) 695 Kicking and Screaming (1995) 696 City Hall (1996) 697 Basketball Diaries, The (1995) 698 Browning Version, The (1994) 699 Little Women (1994) 700 Miami Rhapsody (1995) 701 Wonderful, Horrible Life of Leni Riefenstahl, The (1993) 702 Barcelona (1994) 703 Widows' Peak (1994) 704 House of the Spirits, The (1993) 705 Singin' in the Rain (1952) 706 Bad Moon (1996) 707 Enchanted April (1991) 708 Sex, Lies, and Videotape (1989) 709 Strictly Ballroom (1992) 710 Better Off Dead... (1985) 711 Substance of Fire, The (1996) 712 Tin Men (1987) 713 Othello (1995) 714 Carrington (1995) 715 To Die For (1995) 716 Home for the Holidays (1995) 717 Juror, The (1996) 718 In the Bleak Midwinter (1995) 719 Canadian Bacon (1994) 720 First Knight (1995) 721 Mallrats (1995) 722 Nine Months (1995) 723 Boys on the Side (1995) 724 Circle of Friends (1995) 725 Exit to Eden (1994) 726 Fluke (1995) 727 Immortal Beloved (1994) 728 Junior (1994) 729 Nell (1994) 730 Queen Margot (Reine Margot, La) (1994) 731 Corrina, Corrina (1994) 732 Dave (1993) 733 Go Fish (1994) 734 Made in America (1993) 735 Philadelphia (1993) 736 Shadowlands (1993) 737 Sirens (1994) 738 Threesome (1994) 739 Pretty Woman (1990) 740 Jane Eyre (1996) 741 Last Supper, The (1995) 742 Ransom (1996) 743 Crow: City of Angels, The (1996) 744 Michael Collins (1996) 745 Ruling Class, The (1972) 746 Real Genius (1985) 747 Benny & Joon (1993) 748 Saint, The (1997) 749 MatchMaker, The (1997) 750 Amistad (1997) 751 Tomorrow Never Dies (1997) 752 Replacement Killers, The (1998) 753 Burnt By the Sun (1994) 754 Red Corner (1997) 755 Jumanji (1995) 756 Father of the Bride Part II (1995) 757 Across the Sea of Time (1995) 758 Lawnmower Man 2: Beyond Cyberspace (1996) 759 Fair Game (1995) 760 Screamers (1995) 761 Nick of Time (1995) 762 Beautiful Girls (1996) 763 Happy Gilmore (1996) 764 If Lucy Fell (1996) 765 Boomerang (1992) 766 Man of the Year (1995) 767 Addiction, The (1995) 768 Casper (1995) 769 Congo (1995) 770 Devil in a Blue Dress (1995) 771 Johnny Mnemonic (1995) 772 Kids (1995) 773 Mute Witness (1994) 774 Prophecy, The (1995) 775 Something to Talk About (1995) 776 Three Wishes (1995) 777 Castle Freak (1995) 778 Don Juan DeMarco (1995) 779 Drop Zone (1994) 780 Dumb & Dumber (1994) 781 French Kiss (1995) 782 Little Odessa (1994) 783 Milk Money (1994) 784 Beyond Bedlam (1993) 785 Only You (1994) 786 Perez Family, The (1995) 787 Roommates (1995) 788 Relative Fear (1994) 789 Swimming with Sharks (1995) 790 Tommy Boy (1995) 791 Baby-Sitters Club, The (1995) 792 Bullets Over Broadway (1994) 793 Crooklyn (1994) 794 It Could Happen to You (1994) 795 Richie Rich (1994) 796 Speechless (1994) 797 Timecop (1994) 798 Bad Company (1995) 799 Boys Life (1995) 800 In the Mouth of Madness (1995) 801 Air Up There, The (1994) 802 Hard Target (1993) 803 Heaven & Earth (1993) 804 Jimmy Hollywood (1994) 805 Manhattan Murder Mystery (1993) 806 Menace II Society (1993) 807 Poetic Justice (1993) 808 Program, The (1993) 809 Rising Sun (1993) 810 Shadow, The (1994) 811 Thirty-Two Short Films About Glenn Gould (1993) 812 Andre (1994) 813 Celluloid Closet, The (1995) 814 Great Day in Harlem, A (1994) 815 One Fine Day (1996) 816 Candyman: Farewell to the Flesh (1995) 817 Frisk (1995) 818 Girl 6 (1996) 819 Eddie (1996) 820 Space Jam (1996) 821 Mrs. Winterbourne (1996) 822 Faces (1968) 823 Mulholland Falls (1996) 824 Great White Hype, The (1996) 825 Arrival, The (1996) 826 Phantom, The (1996) 827 Daylight (1996) 828 Alaska (1996) 829 Fled (1996) 830 Power 98 (1995) 831 Escape from L.A. (1996) 832 Bogus (1996) 833 Bulletproof (1996) 834 Halloween: The Curse of Michael Myers (1995) 835 Gay Divorcee, The (1934) 836 Ninotchka (1939) 837 Meet John Doe (1941) 838 In the Line of Duty 2 (1987) 839 Loch Ness (1995) 840 Last Man Standing (1996) 841 Glimmer Man, The (1996) 842 Pollyanna (1960) 843 Shaggy Dog, The (1959) 844 Freeway (1996) 845 That Thing You Do! (1996) 846 To Gillian on Her 37th Birthday (1996) 847 Looking for Richard (1996) 848 Murder, My Sweet (1944) 849 Days of Thunder (1990) 850 Perfect Candidate, A (1996) 851 Two or Three Things I Know About Her (1966) 852 Bloody Child, The (1996) 853 Braindead (1992) 854 Bad Taste (1987) 855 Diva (1981) 856 Night on Earth (1991) 857 Paris Was a Woman (1995) 858 Amityville: Dollhouse (1996) 859 April Fool's Day (1986) 860 Believers, The (1987) 861 Nosferatu a Venezia (1986) 862 Jingle All the Way (1996) 863 Garden of Finzi-Contini, The (Giardino dei Finzi-Contini, Il) (1970) 864 My Fellow Americans (1996) 865 Ice Storm, The (1997) 866 Michael (1996) 867 Whole Wide World, The (1996) 868 Hearts and Minds (1996) 869 Fools Rush In (1997) 870 Touch (1997) 871 Vegas Vacation (1997) 872 Love Jones (1997) 873 Picture Perfect (1997) 874 Career Girls (1997) 875 She's So Lovely (1997) 876 Money Talks (1997) 877 Excess Baggage (1997) 878 That Darn Cat! (1997) 879 Peacemaker, The (1997) 880 Soul Food (1997) 881 Money Talks (1997) 882 Washington Square (1997) 883 Telling Lies in America (1997) 884 Year of the Horse (1997) 885 Phantoms (1998) 886 Life Less Ordinary, A (1997) 887 Eve's Bayou (1997) 888 One Night Stand (1997) 889 Tango Lesson, The (1997) 890 Mortal Kombat: Annihilation (1997) 891 Bent (1997) 892 Flubber (1997) 893 For Richer or Poorer (1997) 894 Home Alone 3 (1997) 895 Scream 2 (1997) 896 Sweet Hereafter, The (1997) 897 Time Tracers (1995) 898 Postman, The (1997) 899 Winter Guest, The (1997) 900 Kundun (1997) 901 Mr. Magoo (1997) 902 Big Lebowski, The (1998) 903 Afterglow (1997) 904 Ma vie en rose (My Life in Pink) (1997) 905 Great Expectations (1998) 906 Oscar & Lucinda (1997) 907 Vermin (1998) 908 Half Baked (1998) 909 Dangerous Beauty (1998) 910 Nil By Mouth (1997) 911 Twilight (1998) 912 U.S. Marshalls (1998) 913 Love and Death on Long Island (1997) 914 Wild Things (1998) 915 Primary Colors (1998) 916 Lost in Space (1998) 917 Mercury Rising (1998) 918 City of Angels (1998) 919 City of Lost Children, The (1995) 920 Two Bits (1995) 921 Farewell My Concubine (1993) 922 Dead Man (1995) 923 Raise the Red Lantern (1991) 924 White Squall (1996) 925 Unforgettable (1996) 926 Down Periscope (1996) 927 Flower of My Secret, The (Flor de mi secreto, La) (1995) 928 Craft, The (1996) 929 Harriet the Spy (1996) 930 Chain Reaction (1996) 931 Island of Dr. Moreau, The (1996) 932 First Kid (1996) 933 Funeral, The (1996) 934 Preacher's Wife, The (1996) 935 Paradise Road (1997) 936 Brassed Off (1996) 937 Thousand Acres, A (1997) 938 Smile Like Yours, A (1997) 939 Murder in the First (1995) 940 Airheads (1994) 941 With Honors (1994) 942 What's Love Got to Do with It (1993) 943 Killing Zoe (1994) 944 Renaissance Man (1994) 945 Charade (1963) 946 Fox and the Hound, The (1981) 947 Big Blue, The (Grand bleu, Le) (1988) 948 Booty Call (1997) 949 How to Make an American Quilt (1995) 950 Georgia (1995) 951 Indian in the Cupboard, The (1995) 952 Blue in the Face (1995) 953 Unstrung Heroes (1995) 954 Unzipped (1995) 955 Before Sunrise (1995) 956 Nobody's Fool (1994) 957 Pushing Hands (1992) 958 To Live (Huozhe) (1994) 959 Dazed and Confused (1993) 960 Naked (1993) 961 Orlando (1993) 962 Ruby in Paradise (1993) 963 Some Folks Call It a Sling Blade (1993) 964 Month by the Lake, A (1995) 965 Funny Face (1957) 966 Affair to Remember, An (1957) 967 Little Lord Fauntleroy (1936) 968 Inspector General, The (1949) 969 Winnie the Pooh and the Blustery Day (1968) 970 Hear My Song (1991) 971 Mediterraneo (1991) 972 Passion Fish (1992) 973 Grateful Dead (1995) 974 Eye for an Eye (1996) 975 Fear (1996) 976 Solo (1996) 977 Substitute, The (1996) 978 Heaven's Prisoners (1996) 979 Trigger Effect, The (1996) 980 Mother Night (1996) 981 Dangerous Ground (1997) 982 Maximum Risk (1996) 983 Rich Man's Wife, The (1996) 984 Shadow Conspiracy (1997) 985 Blood & Wine (1997) 986 Turbulence (1997) 987 Underworld (1997) 988 Beautician and the Beast, The (1997) 989 Cats Don't Dance (1997) 990 Anna Karenina (1997) 991 Keys to Tulsa (1997) 992 Head Above Water (1996) 993 Hercules (1997) 994 Last Time I Committed Suicide, The (1997) 995 Kiss Me, Guido (1997) 996 Big Green, The (1995) 997 Stuart Saves His Family (1995) 998 Cabin Boy (1994) 999 Clean Slate (1994) 1000 Lightning Jack (1994) 1001 Stupids, The (1996) 1002 Pest, The (1997) 1003 That Darn Cat! (1997) 1004 Geronimo: An American Legend (1993) 1005 Double vie de Vronique, La (Double Life of Veronique, The) (1991) 1006 Until the End of the World (Bis ans Ende der Welt) (1991) 1007 Waiting for Guffman (1996) 1008 I Shot Andy Warhol (1996) 1009 Stealing Beauty (1996) 1010 Basquiat (1996) 1011 2 Days in the Valley (1996) 1012 Private Parts (1997) 1013 Anaconda (1997) 1014 Romy and Michele's High School Reunion (1997) 1015 Shiloh (1997) 1016 Con Air (1997) 1017 Trees Lounge (1996) 1018 Tie Me Up! Tie Me Down! (1990) 1019 Die xue shuang xiong (Killer, The) (1989) 1020 Gaslight (1944) 1021 8 1/2 (1963) 1022 Fast, Cheap & Out of Control (1997) 1023 Fathers' Day (1997) 1024 Mrs. Dalloway (1997) 1025 Fire Down Below (1997) 1026 Lay of the Land, The (1997) 1027 Shooter, The (1995) 1028 Grumpier Old Men (1995) 1029 Jury Duty (1995) 1030 Beverly Hillbillies, The (1993) 1031 Lassie (1994) 1032 Little Big League (1994) 1033 Homeward Bound II: Lost in San Francisco (1996) 1034 Quest, The (1996) 1035 Cool Runnings (1993) 1036 Drop Dead Fred (1991) 1037 Grease 2 (1982) 1038 Switchback (1997) 1039 Hamlet (1996) 1040 Two if by Sea (1996) 1041 Forget Paris (1995) 1042 Just Cause (1995) 1043 Rent-a-Kid (1995) 1044 Paper, The (1994) 1045 Fearless (1993) 1046 Malice (1993) 1047 Multiplicity (1996) 1048 She's the One (1996) 1049 House Arrest (1996) 1050 Ghost and Mrs. Muir, The (1947) 1051 Associate, The (1996) 1052 Dracula: Dead and Loving It (1995) 1053 Now and Then (1995) 1054 Mr. Wrong (1996) 1055 Simple Twist of Fate, A (1994) 1056 Cronos (1992) 1057 Pallbearer, The (1996) 1058 War, The (1994) 1059 Don't Be a Menace to South Central While Drinking Your Juice in the Hood (1996) 1060 Adventures of Pinocchio, The (1996) 1061 Evening Star, The (1996) 1062 Four Days in September (1997) 1063 Little Princess, A (1995) 1064 Crossfire (1947) 1065 Koyaanisqatsi (1983) 1066 Balto (1995) 1067 Bottle Rocket (1996) 1068 Star Maker, The (Uomo delle stelle, L') (1995) 1069 Amateur (1994) 1070 Living in Oblivion (1995) 1071 Party Girl (1995) 1072 Pyromaniac's Love Story, A (1995) 1073 Shallow Grave (1994) 1074 Reality Bites (1994) 1075 Man of No Importance, A (1994) 1076 Pagemaster, The (1994) 1077 Love and a .45 (1994) 1078 Oliver & Company (1988) 1079 Joe's Apartment (1996) 1080 Celestial Clockwork (1994) 1081 Curdled (1996) 1082 Female Perversions (1996) 1083 Albino Alligator (1996) 1084 Anne Frank Remembered (1995) 1085 Carried Away (1996) 1086 It's My Party (1995) 1087 Bloodsport 2 (1995) 1088 Double Team (1997) 1089 Speed 2: Cruise Control (1997) 1090 Sliver (1993) 1091 Pete's Dragon (1977) 1092 Dear God (1996) 1093 Live Nude Girls (1995) 1094 Thin Line Between Love and Hate, A (1996) 1095 High School High (1996) 1096 Commandments (1997) 1097 Hate (Haine, La) (1995) 1098 Flirting With Disaster (1996) 1099 Red Firecracker, Green Firecracker (1994) 1100 What Happened Was... (1994) 1101 Six Degrees of Separation (1993) 1102 Two Much (1996) 1103 Trust (1990) 1104 C'est arriv prs de chez vous (1992) 1105 Firestorm (1998) 1106 Newton Boys, The (1998) 1107 Beyond Rangoon (1995) 1108 Feast of July (1995) 1109 Death and the Maiden (1994) 1110 Tank Girl (1995) 1111 Double Happiness (1994) 1112 Cobb (1994) 1113 Mrs. Parker and the Vicious Circle (1994) 1114 Faithful (1996) 1115 Twelfth Night (1996) 1116 Mark of Zorro, The (1940) 1117 Surviving Picasso (1996) 1118 Up in Smoke (1978) 1119 Some Kind of Wonderful (1987) 1120 I'm Not Rappaport (1996) 1121 Umbrellas of Cherbourg, The (Parapluies de Cherbourg, Les) (1964) 1122 They Made Me a Criminal (1939) 1123 Last Time I Saw Paris, The (1954) 1124 Farewell to Arms, A (1932) 1125 Innocents, The (1961) 1126 Old Man and the Sea, The (1958) 1127 Truman Show, The (1998) 1128 Heidi Fleiss: Hollywood Madam (1995) 1129 Chungking Express (1994) 1130 Jupiter's Wife (1994) 1131 Safe (1995) 1132 Feeling Minnesota (1996) 1133 Escape to Witch Mountain (1975) 1134 Get on the Bus (1996) 1135 Doors, The (1991) 1136 Ghosts of Mississippi (1996) 1137 Beautiful Thing (1996) 1138 Best Men (1997) 1139 Hackers (1995) 1140 Road to Wellville, The (1994) 1141 War Room, The (1993) 1142 When We Were Kings (1996) 1143 Hard Eight (1996) 1144 Quiet Room, The (1996) 1145 Blue Chips (1994) 1146 Calendar Girl (1993) 1147 My Family (1995) 1148 Tom & Viv (1994) 1149 Walkabout (1971) 1150 Last Dance (1996) 1151 Original Gangstas (1996) 1152 In Love and War (1996) 1153 Backbeat (1993) 1154 Alphaville (1965) 1155 Rendezvous in Paris (Rendez-vous de Paris, Les) (1995) 1156 Cyclo (1995) 1157 Relic, The (1997) 1158 Fille seule, La (A Single Girl) (1995) 1159 Stalker (1979) 1160 Love! Valour! Compassion! (1997) 1161 Palookaville (1996) 1162 Phat Beach (1996) 1163 Portrait of a Lady, The (1996) 1164 Zeus and Roxanne (1997) 1165 Big Bully (1996) 1166 Love & Human Remains (1993) 1167 Sum of Us, The (1994) 1168 Little Buddha (1993) 1169 Fresh (1994) 1170 Spanking the Monkey (1994) 1171 Wild Reeds (1994) 1172 Women, The (1939) 1173 Bliss (1997) 1174 Caught (1996) 1175 Hugo Pool (1997) 1176 Welcome To Sarajevo (1997) 1177 Dunston Checks In (1996) 1178 Major Payne (1994) 1179 Man of the House (1995) 1180 I Love Trouble (1994) 1181 Low Down Dirty Shame, A (1994) 1182 Cops and Robbersons (1994) 1183 Cowboy Way, The (1994) 1184 Endless Summer 2, The (1994) 1185 In the Army Now (1994) 1186 Inkwell, The (1994) 1187 Switchblade Sisters (1975) 1188 Young Guns II (1990) 1189 Prefontaine (1997) 1190 That Old Feeling (1997) 1191 Letter From Death Row, A (1998) 1192 Boys of St. Vincent, The (1993) 1193 Before the Rain (Pred dozhdot) (1994) 1194 Once Were Warriors (1994) 1195 Strawberry and Chocolate (Fresa y chocolate) (1993) 1196 Savage Nights (Nuits fauves, Les) (1992) 1197 Family Thing, A (1996) 1198 Purple Noon (1960) 1199 Cemetery Man (Dellamorte Dellamore) (1994) 1200 Kim (1950) 1201 Marlene Dietrich: Shadow and Light (1996) 1202 Maybe, Maybe Not (Bewegte Mann, Der) (1994) 1203 Top Hat (1935) 1204 To Be or Not to Be (1942) 1205 Secret Agent, The (1996) 1206 Amos & Andrew (1993) 1207 Jade (1995) 1208 Kiss of Death (1995) 1209 Mixed Nuts (1994) 1210 Virtuosity (1995) 1211 Blue Sky (1994) 1212 Flesh and Bone (1993) 1213 Guilty as Sin (1993) 1214 In the Realm of the Senses (Ai no corrida) (1976) 1215 Barb Wire (1996) 1216 Kissed (1996) 1217 Assassins (1995) 1218 Friday (1995) 1219 Goofy Movie, A (1995) 1220 Higher Learning (1995) 1221 When a Man Loves a Woman (1994) 1222 Judgment Night (1993) 1223 King of the Hill (1993) 1224 Scout, The (1994) 1225 Angus (1995) 1226 Night Falls on Manhattan (1997) 1227 Awfully Big Adventure, An (1995) 1228 Under Siege 2: Dark Territory (1995) 1229 Poison Ivy II (1995) 1230 Ready to Wear (Pret-A-Porter) (1994) 1231 Marked for Death (1990) 1232 Madonna: Truth or Dare (1991) 1233 Nnette et Boni (1996) 1234 Chairman of the Board (1998) 1235 Big Bang Theory, The (1994) 1236 Other Voices, Other Rooms (1997) 1237 Twisted (1996) 1238 Full Speed (1996) 1239 Cutthroat Island (1995) 1240 Ghost in the Shell (Kokaku kidotai) (1995) 1241 Van, The (1996) 1242 Old Lady Who Walked in the Sea, The (Vieille qui marchait dans la mer, La) (1991) 1243 Night Flier (1997) 1244 Metro (1997) 1245 Gridlock'd (1997) 1246 Bushwhacked (1995) 1247 Bad Girls (1994) 1248 Blink (1994) 1249 For Love or Money (1993) 1250 Best of the Best 3: No Turning Back (1995) 1251 A Chef in Love (1996) 1252 Contempt (Mpris, Le) (1963) 1253 Tie That Binds, The (1995) 1254 Gone Fishin' (1997) 1255 Broken English (1996) 1256 Designated Mourner, The (1997) 1257 Designated Mourner, The (1997) 1258 Trial and Error (1997) 1259 Pie in the Sky (1995) 1260 Total Eclipse (1995) 1261 Run of the Country, The (1995) 1262 Walking and Talking (1996) 1263 Foxfire (1996) 1264 Nothing to Lose (1994) 1265 Star Maps (1997) 1266 Bread and Chocolate (Pane e cioccolata) (1973) 1267 Clockers (1995) 1268 Bitter Moon (1992) 1269 Love in the Afternoon (1957) 1270 Life with Mikey (1993) 1271 North (1994) 1272 Talking About Sex (1994) 1273 Color of Night (1994) 1274 Robocop 3 (1993) 1275 Killer (Bulletproof Heart) (1994) 1276 Sunset Park (1996) 1277 Set It Off (1996) 1278 Selena (1997) 1279 Wild America (1997) 1280 Gang Related (1997) 1281 Manny & Lo (1996) 1282 Grass Harp, The (1995) 1283 Out to Sea (1997) 1284 Before and After (1996) 1285 Princess Caraboo (1994) 1286 Shall We Dance? (1937) 1287 Ed (1996) 1288 Denise Calls Up (1995) 1289 Jack and Sarah (1995) 1290 Country Life (1994) 1291 Celtic Pride (1996) 1292 Simple Wish, A (1997) 1293 Star Kid (1997) 1294 Ayn Rand: A Sense of Life (1997) 1295 Kicked in the Head (1997) 1296 Indian Summer (1996) 1297 Love Affair (1994) 1298 Band Wagon, The (1953) 1299 Penny Serenade (1941) 1300 'Til There Was You (1997) 1301 Stripes (1981) 1302 Late Bloomers (1996) 1303 Getaway, The (1994) 1304 New York Cop (1996) 1305 National Lampoon's Senior Trip (1995) 1306 Delta of Venus (1994) 1307 Carmen Miranda: Bananas Is My Business (1994) 1308 Babyfever (1994) 1309 Very Natural Thing, A (1974) 1310 Walk in the Sun, A (1945) 1311 Waiting to Exhale (1995) 1312 Pompatus of Love, The (1996) 1313 Palmetto (1998) 1314 Surviving the Game (1994) 1315 Inventing the Abbotts (1997) 1316 Horse Whisperer, The (1998) 1317 Journey of August King, The (1995) 1318 Catwalk (1995) 1319 Neon Bible, The (1995) 1320 Homage (1995) 1321 Open Season (1996) 1322 Metisse (Caf au Lait) (1993) 1323 Wooden Man's Bride, The (Wu Kui) (1994) 1324 Loaded (1994) 1325 August (1996) 1326 Boys (1996) 1327 Captives (1994) 1328 Of Love and Shadows (1994) 1329 Low Life, The (1994) 1330 An Unforgettable Summer (1994) 1331 Last Klezmer: Leopold Kozlowski, His Life and Music, The (1995) 1332 My Life and Times With Antonin Artaud (En compagnie d'Antonin Artaud) (1993) 1333 Midnight Dancers (Sibak) (1994) 1334 Somebody to Love (1994) 1335 American Buffalo (1996) 1336 Kazaam (1996) 1337 Larger Than Life (1996) 1338 Two Deaths (1995) 1339 Stefano Quantestorie (1993) 1340 Crude Oasis, The (1995) 1341 Hedd Wyn (1992) 1342 Convent, The (Convento, O) (1995) 1343 Lotto Land (1995) 1344 Story of Xinghua, The (1993) 1345 Day the Sun Turned Cold, The (Tianguo niezi) (1994) 1346 Dingo (1992) 1347 Ballad of Narayama, The (Narayama Bushiko) (1958) 1348 Every Other Weekend (1990) 1349 Mille bolle blu (1993) 1350 Crows and Sparrows (1949) 1351 Lover's Knot (1996) 1352 Shadow of Angels (Schatten der Engel) (1976) 1353 1-900 (1994) 1354 Venice/Venice (1992) 1355 Infinity (1996) 1356 Ed's Next Move (1996) 1357 For the Moment (1994) 1358 The Deadly Cure (1996) 1359 Boys in Venice (1996) 1360 Sexual Life of the Belgians, The (1994) 1361 Search for One-eye Jimmy, The (1996) 1362 American Strays (1996) 1363 Leopard Son, The (1996) 1364 Bird of Prey (1996) 1365 Johnny 100 Pesos (1993) 1366 JLG/JLG - autoportrait de dcembre (1994) 1367 Faust (1994) 1368 Mina Tannenbaum (1994) 1369 Forbidden Christ, The (Cristo proibito, Il) (1950) 1370 I Can't Sleep (J'ai pas sommeil) (1994) 1371 Machine, The (1994) 1372 Stranger, The (1994) 1373 Good Morning (1971) 1374 Falling in Love Again (1980) 1375 Cement Garden, The (1993) 1376 Meet Wally Sparks (1997) 1377 Hotel de Love (1996) 1378 Rhyme & Reason (1997) 1379 Love and Other Catastrophes (1996) 1380 Hollow Reed (1996) 1381 Losing Chase (1996) 1382 Bonheur, Le (1965) 1383 Second Jungle Book: Mowgli & Baloo, The (1997) 1384 Squeeze (1996) 1385 Roseanna's Grave (For Roseanna) (1997) 1386 Tetsuo II: Body Hammer (1992) 1387 Fall (1997) 1388 Gabbeh (1996) 1389 Mondo (1996) 1390 Innocent Sleep, The (1995) 1391 For Ever Mozart (1996) 1392 Locusts, The (1997) 1393 Stag (1997) 1394 Swept from the Sea (1997) 1395 Hurricane Streets (1998) 1396 Stonewall (1995) 1397 Of Human Bondage (1934) 1398 Anna (1996) 1399 Stranger in the House (1997) 1400 Picture Bride (1995) 1401 M. Butterfly (1993) 1402 Ciao, Professore! (1993) 1403 Caro Diario (Dear Diary) (1994) 1404 Withnail and I (1987) 1405 Boy's Life 2 (1997) 1406 When Night Is Falling (1995) 1407 Specialist, The (1994) 1408 Gordy (1995) 1409 Swan Princess, The (1994) 1410 Harlem (1993) 1411 Barbarella (1968) 1412 Land Before Time III: The Time of the Great Giving (1995) (V) 1413 Street Fighter (1994) 1414 Coldblooded (1995) 1415 Next Karate Kid, The (1994) 1416 No Escape (1994) 1417 Turning, The (1992) 1418 Joy Luck Club, The (1993) 1419 Highlander III: The Sorcerer (1994) 1420 Gilligan's Island: The Movie (1998) 1421 My Crazy Life (Mi vida loca) (1993) 1422 Suture (1993) 1423 Walking Dead, The (1995) 1424 I Like It Like That (1994) 1425 I'll Do Anything (1994) 1426 Grace of My Heart (1996) 1427 Drunks (1995) 1428 SubUrbia (1997) 1429 Sliding Doors (1998) 1430 Ill Gotten Gains (1997) 1431 Legal Deceit (1997) 1432 Mighty, The (1998) 1433 Men of Means (1998) 1434 Shooting Fish (1997) 1435 Steal Big, Steal Little (1995) 1436 Mr. Jones (1993) 1437 House Party 3 (1994) 1438 Panther (1995) 1439 Jason's Lyric (1994) 1440 Above the Rim (1994) 1441 Moonlight and Valentino (1995) 1442 Scarlet Letter, The (1995) 1443 8 Seconds (1994) 1444 That Darn Cat! (1965) 1445 Ladybird Ladybird (1994) 1446 Bye Bye, Love (1995) 1447 Century (1993) 1448 My Favorite Season (1993) 1449 Pather Panchali (1955) 1450 Golden Earrings (1947) 1451 Foreign Correspondent (1940) 1452 Lady of Burlesque (1943) 1453 Angel on My Shoulder (1946) 1454 Angel and the Badman (1947) 1455 Outlaw, The (1943) 1456 Beat the Devil (1954) 1457 Love Is All There Is (1996) 1458 Damsel in Distress, A (1937) 1459 Madame Butterfly (1995) 1460 Sleepover (1995) 1461 Here Comes Cookie (1935) 1462 Thieves (Voleurs, Les) (1996) 1463 Boys, Les (1997) 1464 Stars Fell on Henrietta, The (1995) 1465 Last Summer in the Hamptons (1995) 1466 Margaret's Museum (1995) 1467 Saint of Fort Washington, The (1993) 1468 Cure, The (1995) 1469 Tom and Huck (1995) 1470 Gumby: The Movie (1995) 1471 Hideaway (1995) 1472 Visitors, The (Visiteurs, Les) (1993) 1473 Little Princess, The (1939) 1474 Nina Takes a Lover (1994) 1475 Bhaji on the Beach (1993) 1476 Raw Deal (1948) 1477 Nightwatch (1997) 1478 Dead Presidents (1995) 1479 Reckless (1995) 1480 Herbie Rides Again (1974) 1481 S.F.W. (1994) 1482 Gate of Heavenly Peace, The (1995) 1483 Man in the Iron Mask, The (1998) 1484 Jerky Boys, The (1994) 1485 Colonel Chabert, Le (1994) 1486 Girl in the Cadillac (1995) 1487 Even Cowgirls Get the Blues (1993) 1488 Germinal (1993) 1489 Chasers (1994) 1490 Fausto (1993) 1491 Tough and Deadly (1995) 1492 Window to Paris (1994) 1493 Modern Affair, A (1995) 1494 Mostro, Il (1994) 1495 Flirt (1995) 1496 Carpool (1996) 1497 Line King: Al Hirschfeld, The (1996) 1498 Farmer & Chase (1995) 1499 Grosse Fatigue (1994) 1500 Santa with Muscles (1996) 1501 Prisoner of the Mountains (Kavkazsky Plennik) (1996) 1502 Naked in New York (1994) 1503 Gold Diggers: The Secret of Bear Mountain (1995) 1504 Bewegte Mann, Der (1994) 1505 Killer: A Journal of Murder (1995) 1506 Nelly & Monsieur Arnaud (1995) 1507 Three Lives and Only One Death (1996) 1508 Babysitter, The (1995) 1509 Getting Even with Dad (1994) 1510 Mad Dog Time (1996) 1511 Children of the Revolution (1996) 1512 World of Apu, The (Apur Sansar) (1959) 1513 Sprung (1997) 1514 Dream With the Fishes (1997) 1515 Wings of Courage (1995) 1516 Wedding Gift, The (1994) 1517 Race the Sun (1996) 1518 Losing Isaiah (1995) 1519 New Jersey Drive (1995) 1520 Fear, The (1995) 1521 Mr. Wonderful (1993) 1522 Trial by Jury (1994) 1523 Good Man in Africa, A (1994) 1524 Kaspar Hauser (1993) 1525 Object of My Affection, The (1998) 1526 Witness (1985) 1527 Senseless (1998) 1528 Nowhere (1997) 1529 Underground (1995) 1530 Jefferson in Paris (1995) 1531 Far From Home: The Adventures of Yellow Dog (1995) 1532 Foreign Student (1994) 1533 I Don't Want to Talk About It (De eso no se habla) (1993) 1534 Twin Town (1997) 1535 Enfer, L' (1994) 1536 Aiqing wansui (1994) 1537 Cosi (1996) 1538 All Over Me (1997) 1539 Being Human (1993) 1540 Amazing Panda Adventure, The (1995) 1541 Beans of Egypt, Maine, The (1994) 1542 Scarlet Letter, The (1926) 1543 Johns (1996) 1544 It Takes Two (1995) 1545 Frankie Starlight (1995) 1546 Shadows (Cienie) (1988) 1547 Show, The (1995) 1548 The Courtyard (1995) 1549 Dream Man (1995) 1550 Destiny Turns on the Radio (1995) 1551 Glass Shield, The (1994) 1552 Hunted, The (1995) 1553 Underneath, The (1995) 1554 Safe Passage (1994) 1555 Secret Adventures of Tom Thumb, The (1993) 1556 Condition Red (1995) 1557 Yankee Zulu (1994) 1558 Aparajito (1956) 1559 Hostile Intentions (1994) 1560 Clean Slate (Coup de Torchon) (1981) 1561 Tigrero: A Film That Was Never Made (1994) 1562 Eye of Vichy, The (Oeil de Vichy, L') (1993) 1563 Promise, The (Versprechen, Das) (1994) 1564 To Cross the Rubicon (1991) 1565 Daens (1992) 1566 Man from Down Under, The (1943) 1567 Careful (1992) 1568 Vermont Is For Lovers (1992) 1569 Vie est belle, La (Life is Rosey) (1987) 1570 Quartier Mozart (1992) 1571 Touki Bouki (Journey of the Hyena) (1973) 1572 Wend Kuuni (God's Gift) (1982) 1573 Spirits of the Dead (Tre passi nel delirio) (1968) 1574 Pharaoh's Army (1995) 1575 I, Worst of All (Yo, la peor de todas) (1990) 1576 Hungarian Fairy Tale, A (1987) 1577 Death in the Garden (Mort en ce jardin, La) (1956) 1578 Collectionneuse, La (1967) 1579 Baton Rouge (1988) 1580 Liebelei (1933) 1581 Woman in Question, The (1950) 1582 T-Men (1947) 1583 Invitation, The (Zaproszenie) (1986) 1584 Symphonie pastorale, La (1946) 1585 American Dream (1990) 1586 Lashou shentan (1992) 1587 Terror in a Texas Town (1958) 1588 Salut cousin! (1996) 1589 Schizopolis (1996) 1590 To Have, or Not (1995) 1591 Duoluo tianshi (1995) 1592 Magic Hour, The (1998) 1593 Death in Brunswick (1991) 1594 Everest (1998) 1595 Shopping (1994) 1596 Nemesis 2: Nebula (1995) 1597 Romper Stomper (1992) 1598 City of Industry (1997) 1599 Someone Else's America (1995) 1600 Guantanamera (1994) 1601 Office Killer (1997) 1602 Price Above Rubies, A (1998) 1603 Angela (1995) 1604 He Walked by Night (1948) 1605 Love Serenade (1996) 1606 Deceiver (1997) 1607 Hurricane Streets (1998) 1608 Buddy (1997) 1609 B*A*P*S (1997) 1610 Truth or Consequences, N.M. (1997) 1611 Intimate Relations (1996) 1612 Leading Man, The (1996) 1613 Tokyo Fist (1995) 1614 Reluctant Debutante, The (1958) 1615 Warriors of Virtue (1997) 1616 Desert Winds (1995) 1617 Hugo Pool (1997) 1618 King of New York (1990) 1619 All Things Fair (1996) 1620 Sixth Man, The (1997) 1621 Butterfly Kiss (1995) 1622 Paris, France (1993) 1623 Crmonie, La (1995) 1624 Hush (1998) 1625 Nightwatch (1997) 1626 Nobody Loves Me (Keiner liebt mich) (1994) 1627 Wife, The (1995) 1628 Lamerica (1994) 1629 Nico Icon (1995) 1630 Silence of the Palace, The (Saimt el Qusur) (1994) 1631 Slingshot, The (1993) 1632 Land and Freedom (Tierra y libertad) (1995) 1633 kldum klaka (Cold Fever) (1994) 1634 Etz Hadomim Tafus (Under the Domin Tree) (1994) 1635 Two Friends (1986) 1636 Brothers in Trouble (1995) 1637 Girls Town (1996) 1638 Normal Life (1996) 1639 Bitter Sugar (Azucar Amargo) (1996) 1640 Eighth Day, The (1996) 1641 Dadetown (1995) 1642 Some Mother's Son (1996) 1643 Angel Baby (1995) 1644 Sudden Manhattan (1996) 1645 Butcher Boy, The (1998) 1646 Men With Guns (1997) 1647 Hana-bi (1997) 1648 Niagara, Niagara (1997) 1649 Big One, The (1997) 1650 Butcher Boy, The (1998) 1651 Spanish Prisoner, The (1997) 1652 Temptress Moon (Feng Yue) (1996) 1653 Entertaining Angels: The Dorothy Day Story (1996) 1654 Chairman of the Board (1998) 1655 Favor, The (1994) 1656 Little City (1998) 1657 Target (1995) 1658 Substance of Fire, The (1996) 1659 Getting Away With Murder (1996) 1660 Small Faces (1995) 1661 New Age, The (1994) 1662 Rough Magic (1995) 1663 Nothing Personal (1995) 1664 8 Heads in a Duffel Bag (1997) 1665 Brother's Kiss, A (1997) 1666 Ripe (1996) 1667 Next Step, The (1995) 1668 Wedding Bell Blues (1996) 1669 MURDER and murder (1996) 1670 Tainted (1998) 1671 Further Gesture, A (1996) 1672 Kika (1993) 1673 Mirage (1995) 1674 Mamma Roma (1962) 1675 Sunchaser, The (1996) 1676 War at Home, The (1996) 1677 Sweet Nothing (1995) 1678 Mat' i syn (1997) 1679 B. Monkey (1998) 1680 Sliding Doors (1998) 1681 You So Crazy (1994) 1682 Scream of Stone (Schrei aus Stein) (1991) ================================================ FILE: requirements.txt ================================================ cycler==0.10.0 kiwisolver==1.1.0 matplotlib==3.1.1 nltk==3.4.5 numpy==1.16.4 pandas==0.24.2 Pillow==8.1.1 pyparsing==2.4.0 python-dateutil==2.8.0 pytz==2019.1 scipy==1.3.0 six==1.12.0 ================================================ FILE: week6/1-Evaluating-a-Learning-Algorithm.md ================================================ # Advice for Applying Machine Learning ## Evaluating a Learning Algorithm ### What to do next? If you trained your model and the model is performing worse than you had expected you may want to try the following things: * Get more training data (do not spend too much time as this does not always yield better results) * Try smaller sets of features * Try getting additional features * Try adding polynomial features * Try decreasing/increasing $\lambda$. Do not randomly pick an option from the list above. Use a **machine learning diagnostic**: a test that you can run to gain insight in what is isn't working with a learning algorithm, and gain guidance as to how best improve its performance. ### Evaluating your hypothesis A low training error does not necessarily mean a good hypothesis function. To prevent overfitting, split the data into a training set and test set. Usually 70% of the data is in the training set and 30% is in the test set. ### Model selection $d$: Degree of polynomial You may want to try out multiple values for $d$, $\lambda$, and so on. Then you choose the best values for $\Theta$ on a set that the model has not seen before. If you did this, however, on the test set you get a too optimistic accuracy. That's the reason we want to split our data into 3 sets: the training set (60%), the cross validation set (20%), the test set (20%). You use the training set for training, the cross validation set for model selection and the test set to obtain the accuracy. The errors: Training error: $J_{train}(\theta) = \frac{1}{2m}\displaystyle\sum_{i = 0}^{m}(h_\theta(x^{(i)})-y^{(i)})^2$ Cross validation error: $J_{cv}(\theta) = \frac{1}{2m_{cv}}\displaystyle\sum_{i = 0}^{m_{cv}}(h_\theta(x_{cv}^{(i)})-y_{cv}^{(i)})^2$ Test error: $J_{cv}(\theta) = \frac{1}{2m_{test}}\displaystyle\sum_{i = 0}^{m_{test}}(h_\theta(x_{test}^{(i)})-y_{test}^{(i)})^2$ You want to choose the values for $d$ from $1, 2, …, 10$ and $\lambda$ in $0.00, 0.02, 0.04, …, 10$. ## Bias vs. Variance High bias: underfitting. High variance: overfitting. ### Learning curves ![learning curves](/Users/rickwierenga/Desktop/cs229/programming/week6/learning_curves.png) ![high bias](high_bias.png) ![high variance](high_variance.png) ### Deciding what to do next - Get more training data (do not spend too much time as this does not always yield better results): fixes high variance - Try smaller sets of features: fixes high variance - Try getting additional features: fixes high bias - Try adding polynomial features: fixes high bias - Try decreasing $\lambda$: fixes high bias - increasing $\lambda$: fixes high variance Small neural networks are more prone to overfitting but computationally cheaper. Large neural networks are more prone to overfitting and computationally more expensive. Use $\lambda$ to prevent overfitting. ## Building a spam filter ### Error analysis - Start with a simple algorithm that you can implement quickly. Implement it and test it on your cross-validation data. - Plot learning curves to decide if more data, more features, etc. are likely to help - Error analysis: Manually examine the examples (in cross validation set) that your algorithm made errors on. See if you can spot any systematic trend in what type of examples it is making errors on. Use numerical evaluation. ### Handling skewed data | | 1 | 0 | | ---- | -------------- | -------------- | | 1 | True positive | False positive | | 0 | False negative | True negative | $P = precision = \frac{True\ positives}{True\ positives+False\ positives}$ $R = recall = \frac{True\ positives}{True\ positives+False\ negatives}$ Trading off precision and recall: predict 1 if $h_\theta(x) \geqslant threshold$ Single method for rating the accuracy: $F_1 = 2\frac{PR}{P+R}$ ## Using large datasets Useful test: Given the input $x$, can a human expert confidently predict $y$.