Repository: fengdu78/lihang-code Branch: master Commit: a0b8ab2366b6 Files: 25 Total size: 1.3 MB Directory structure: gitextract_a4phc2jx/ ├── readme.md ├── 第01章 统计学习方法概论/ │ └── 1.Introduction_to_statistical_learning_methods.ipynb ├── 第02章 感知机/ │ └── 2.Perceptron.ipynb ├── 第03章 k近邻法/ │ └── 3.KNearestNeighbors.ipynb ├── 第04章 朴素贝叶斯/ │ └── 4.NaiveBayes.ipynb ├── 第05章 决策树/ │ └── 5.DecisonTree.ipynb ├── 第06章 逻辑斯谛回归/ │ └── 6.LogisticRegression.ipynb ├── 第07章 支持向量机/ │ └── 7.support-vector-machine.ipynb ├── 第08章 提升方法/ │ └── 8.Boost.ipynb ├── 第09章 EM算法及其推广/ │ └── 9.Expectation_Maximization.ipynb ├── 第10章 隐马尔可夫模型/ │ └── 10.HMM.ipynb ├── 第11章 条件随机场/ │ └── 11.CRF.ipynb ├── 第12章 监督学习方法总结/ │ └── 12.Summary_of_Supervised_Learning_Methods.ipynb ├── 第13章 无监督学习概论/ │ └── 13.Introduction_to_Unsupervised_Learning.ipynb ├── 第14章 聚类方法/ │ └── 14.Clustering.ipynb ├── 第15章 奇异值分解/ │ └── 15.SVD.ipynb ├── 第16章 主成分分析/ │ ├── 16.PCA.ipynb │ └── data/ │ └── ex7data1.mat ├── 第17章 潜在语义分析/ │ └── 17.LSA.ipynb ├── 第18章 概率潜在语义分析/ │ └── 18.PLSA.ipynb ├── 第19章 马尔可夫链蒙特卡洛法/ │ └── 19.MCMC.ipynb ├── 第20章 潜在狄利克雷分配/ │ ├── 20.LDA.ipynb │ └── data/ │ └── LDA_test.txt ├── 第21章 PageRank算法/ │ └── 21.PageRank.ipynb └── 第22章 无监督学习方法总结/ └── 22.Summary_of_UnSupervised_Learning_Methods.ipynb ================================================ FILE CONTENTS ================================================ ================================================ FILE: readme.md ================================================ # 《统计学习方法》第二版的代码实现 李航老师编写的《统计学习方法》全面系统地介绍了统计学习的主要方法，特别是监督学习方法，包括感知机、k近邻法、朴素贝叶斯法、决策树、逻辑斯谛回归与支持向量机、提升方法、em算法、隐马尔可夫模型和条件随机场等。除第1章概论和最后一章总结外，每章介绍一种方法。叙述从具体问题或实例入手，由浅入深，阐明思路，给出必要的数学推导，便于读者掌握统计学习方法的实质，学会运用。 《统计学习方法》可以说是机器学习的入门宝典，许多机器学习培训班、互联网企业的面试、笔试题目，很多都参考这本书。 今天我们将李航老师的《统计学习方法》第二版的代码进行了整理，并提供下载。 非常感谢各位朋友贡献的自己的笔记、代码！ 2020年6月7日 ## 代码目录 第1章 统计学习方法概论 第2章 感知机 第3章 k近邻法 第4章 朴素贝叶斯 第5章 决策树 第6章 逻辑斯谛回归 第7章 支持向量机 第8章 提升方法 第9章 EM算法及其推广 第10章 隐马尔可夫模型 第11章 条件随机场 第12章 监督学习方法总结 第13章 无监督学习概论 第14章 聚类方法 第15章 奇异值分解 第16章 主成分分析 第17章 潜在语义分析 第18章 概率潜在语义分析 第19章 马尔可夫链蒙特卡洛法 第20章 潜在狄利克雷分配 第21章 PageRank算法 第22章 无监督学习方法总结 ## 参考 https://github.com/wzyonggege/statistical-learning-method https://github.com/WenDesi/lihang_book_algorithm https://blog.csdn.net/tudaodiaozhale https://github.com/hktxt/Learn-Statistical-Learning-Method 代码整理和修改：机器学习初学者（公众号） ![gongzhong](images/gongzhong.jpg) 知识星球：黄博的机器学习圈子![xingqiu](images/zhishixingqiu1.jpg) [知乎](https://www.zhihu.com/people/fengdu78) ================================================ FILE: 第01章 统计学习方法概论/1.Introduction_to_statistical_learning_methods.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第1章 统计学习方法概论" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1．统计学习是关于计算机基于数据构建概率统计模型并运用模型对数据进行分析与预测的一门学科。统计学习包括监督学习、非监督学习、半监督学习和强化学习。\n", "\n", "2．统计学习方法三要素——模型、策略、算法，对理解统计学习方法起到提纲挈领的作用。\n", "\n", "3．本书主要讨论监督学习，监督学习可以概括如下：从给定有限的训练数据出发， 假设数据是独立同分布的，而且假设模型属于某个假设空间，应用某一评价准则，从假设空间中选取一个最优的模型，使它对已给训练数据及未知测试数据在给定评价标准意义下有最准确的预测。\n", "\n", "4．统计学习中，进行模型选择或者说提高学习的泛化能力是一个重要问题。如果只考虑减少训练误差，就可能产生过拟合现象。模型选择的方法有正则化与交叉验证。学习方法泛化能力的分析是统计学习理论研究的重要课题。\n", "\n", "5．分类问题、标注问题和回归问题都是监督学习的重要问题。本书中介绍的统计学习方法包括感知机、$k$近邻法、朴素贝叶斯法、决策树、逻辑斯谛回归与最大熵模型、支持向量机、提升方法、EM算法、隐马尔可夫模型和条件随机场。这些方法是主要的分类、标注以及回归方法。它们又可以归类为生成方法与判别方法。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 使用最小二乘法拟和曲线\n", "\n", "高斯于1823年在误差$e_1,…,e_n$独立同分布的假定下,证明了最小二乘方法的一个最优性质: 在所有无偏的线性估计类中,最小二乘方法是其中方差最小的！\n", "对于数据$(x_i, y_i) (i=1, 2, 3...,m)$\n", "\n", "拟合出函数$h(x)$\n", "\n", "有误差，即残差：$r_i=h(x_i)-y_i$\n", "\n", "此时$L2$范数(残差平方和)最小时，$h(x)$ 和 $y$ 相似度最高，更拟合" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "一般的$H(x)$为$n$次的多项式，$H(x)=w_0+w_1x+w_2x^2+...w_nx^n$\n", "\n", "$w(w_0,w_1,w_2,...,w_n)$为参数\n", "\n", "最小二乘法就是要找到一组 $w(w_0,w_1,w_2,...,w_n)$ ，使得$\\sum_{i=1}^n(h(x_i)-y_i)^2$ (残差平方和) 最小\n", "\n", "即，求 $min\\sum_{i=1}^n(h(x_i)-y_i)^2$\n", "\n", "----" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "举例：我们用目标函数$y=sin2{\\pi}x$, 加上一个正态分布的噪音干扰，用多项式去拟合【例1.1 11页】" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import scipy as sp\n", "from scipy.optimize import leastsq\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* ps: numpy.poly1d([1,2,3]) 生成 $1x^2+2x^1+3x^0$*" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# 目标函数\n", "def real_func(x):\n", " return np.sin(2*np.pi*x)\n", "\n", "# 多项式\n", "def fit_func(p, x):\n", " f = np.poly1d(p)\n", " return f(x)\n", "\n", "# 残差\n", "def residuals_func(p, x, y):\n", " ret = fit_func(p, x) - y\n", " return ret" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# 十个点\n", "x = np.linspace(0, 1, 10)\n", "x_points = np.linspace(0, 1, 1000)\n", "# 加上正态分布噪音的目标函数的值\n", "y_ = real_func(x)\n", "y = [np.random.normal(0, 0.1) + y1 for y1 in y_]\n", "\n", "\n", "def fitting(M=0):\n", " \"\"\"\n", " M 为 多项式的次数\n", " \"\"\"\n", " # 随机初始化多项式参数\n", " p_init = np.random.rand(M + 1)\n", " # 最小二乘法\n", " p_lsq = leastsq(residuals_func, p_init, args=(x, y))\n", " print('Fitting Parameters:', p_lsq[0])\n", "\n", " # 可视化\n", " plt.plot(x_points, real_func(x_points), label='real')\n", " plt.plot(x_points, fit_func(p_lsq[0], x_points), label='fitted curve')\n", " plt.plot(x, y, 'bo', label='noise')\n", " plt.legend()\n", " return p_lsq" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### M=0" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting Parameters: [0.02515259]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XlcVPX+x/HXl0UQRBBwR8BdEBAV18qs3HNpszS6N62kblnd29XqlzfrVla23hbLi+VSUVp2Uyu3XCpzR8VdFJXNFVEQZBGY7++PQa96QVCGObN8no8Hj5k5c+ac9xngM985y/ertNYIIYRwLi5GBxBCCGF9UvyFEMIJSfEXQggnJMVfCCGckBR/IYRwQlL8hRDCCUnxF0IIJyTFXwghnJAUfyGEcEJuRgeoTGBgoA4NDTU6hhBC2JUtW7ac0lo3rGo+my3+oaGhJCYmGh1DCCHsilIqrTrzyW4fIYRwQlL8hRDCCUnxF0IIJ2Sz+/yFEPajpKSEzMxMioqKjI7iNDw9PQkKCsLd3f26Xi/FXwhRY5mZmfj4+BAaGopSyug4Dk9rTXZ2NpmZmbRs2fK6liG7fUStS0iA0FBwcTHfJiQYnUhYWlFREQEBAVL4rUQpRUBAQI2+aUnLX9SqhASIi4OCAvPjtDTzY4DYWONyCcuTwm9dNX2/peXvwGyhxT1p0n8L/wUFBebpQgjjSPF3UBda3GlpoPV/W9zW/gBIT7+26UIYZcyYMcyfP9/oGFYjxd9B2UqLOzj42qYLYQlaa0wmk9ExbJoUfwdlKy3uKVPAy+vyaV5e5ulCWFJqaiphYWE8/vjjdOnShS+//JJevXrRpUsXRo4cSX5+PgCvvPIK3bp1IyIigri4OLTWBic3hhzwdVDBweZdPRVNt6YLB3UnTTJ/8AQHmwu/HOx1XP/8cTd7jp616DLDm9XnpWEdq5wvOTmZWbNm8corr3DXXXexYsUKvL29mTp1Ku+99x6TJ09m/PjxTJ48GYA//elP/PTTTwwbNsyiee2BRVr+SqmZSqmTSqldlTyvlFIfKqVSlFI7lFJdLLFeUTlbanHHxkJqKphM5lsp/KK2hISE0LNnTzZs2MCePXu44YYbiI6OZs6cOaSVt4ZWr15Njx49iIyMZNWqVezevdvg1MawVMt/NvAx8EUlzw8G2pb/9AA+Lb8VtURa3MIo1Wmh1xZvb2/AvM+/f//+fPPNN5c9X1RUxOOPP05iYiItWrTg5Zdfdtqrki1S/LXWvyulQq8yywjgC23eubZBKeWnlGqqtT5mifWLisXGVlzsC8+XkXmmgMycQgqKyygqKUMpqO/pjq+XO8H+XjTy8ZDztoXd6tmzJ0888QQpKSm0adOGgoICMjMzadSoEQCBgYHk5+czf/587rnnHoPTGsNa+/ybAxmXPM4sn3ZZ8VdKxQFxAMFyOojFnMwrYtXek2xKPc229BwOnzpX5WvqebgR3rQ+PVr506tVAN1a+uPuKucHCPvQsGFDZs+ezejRoykuLgbgtddeo127dowbN47IyEhCQ0Pp1q2bwUmNoyx1pLu85f+T1jqigud+Bt7QWv9R/ngl8KzWektly4uJidEymMv1yy8uZWHSEb7fksm2jBy0hsB6HnQO9iOyuS8hAV4ENaiLj6c7Hm4uaA15RaWcLjhPWvY5Uk7msz0jh51HcjFpaODlzuDIptzdJYguwX7yrUBcZu/evYSFhRkdw+lU9L4rpbZorWOqeq21Wv6ZQItLHgcBR620bqdyLLeQ6b8eZP6WTM6dL6N9Yx/+1q8dAzs2oV3jetUs2v8dAS6vqIS1Kdn8vPMYP2w9wtcb0+kU5MtDN7ZkaFQzXF3kQ0AIe2St4r8IGK+Umov5QG+u7O+3rJNni/h4dQpzN2Vg0poR0c2J7RlM5xY1a6X7eLozKKIJgyKacK64lO+3ZjJ7bSpPz03i41Up/H2A+YNFvgkIYV8sUvyVUt8AfYFApVQm8BLgDqC1ng4sBoYAKUABMNYS6xVQWmZizvo03v9lP0UlZYyMCeLxvm1o4e9V9YuvkbeHG3/uFcoDPUJYuvs47yxP5rGvttI91J8pd0bQtrGPxdcphKgdljrbZ3QVz2vgCUusS/zXzsxcJs7fzr7jefRp15B/Du9Iy0DvWl+vi4tiSGRTBoQ35rstmUxduo/BH6whrk8rnrqtLZ7urrWeQQhRM3KFrx0qM2mm/3aQ93/ZT2A9D6Y/0JWBHRtbfdeLm6sLo7sHMyC8MW8s2ccnvx5k1b6TfDS6s3wLEMLGybl7dubk2SJGx2/g7WXJDIxowtK/3sSgCGP3uQfU8+CdkZ2YNaYbWXnFDP3oD77akOa0faYIYQ+k+NuRrelnGPrRH+w8kst793bi49Gd8fOqY3Ssi27p0Iglf72JHq0C+MeCXTz3/Q6KS8uMjiWcxIcffkhYWBixsbEsWrSIN998E4AFCxawZ8+ei/PNnj2bo0ev7WTD1NRUIiL+5yx2uya7fezEvM3pvLhgN419PZjzUG/CmtY3OlKFGvl4MntMN/618gAfrjxAysl8pj/QlUb1PY2OJhzcJ598wpIlSy6OaTt8+HDAXPyHDh1KeHg4YC7+ERERNGvWzLCsAKWlpbi5GVeCpeVv47TWvLs8mee+30mPVv78OP5Gmy38F7i4KJ7p345PYruw91ged36yjoNZ+UbHEg7sscce49ChQwwfPpz333+f2bNnM378eNatW8eiRYuYOHEi0dHRTJ06lcTERGJjY4mOjqawsJAtW7Zw880307VrVwYOHMixY+az0Lds2UKnTp3o1asX06ZNq3Tdb731FpGRkXTq1Innn38egL59+3LhItVTp04RGhoKmD94Ro4cybBhwxgwYAD33XcfixcvvrisMWPG8P3331NWVsbEiRPp1q0bUVFR/Pvf/7b4eyYtfxtWWmbiHwt2MXdzBvfFtGDKnRG42VEXC0MimxLs78WYWZsYOX09s8Z0o1MLP6Njidq25Hk4vtOyy2wSCYPfrPTp6dOns3TpUlavXk1gYCCzZ88GoHfv3gwfPpyhQ4de7MNnyZIlvPPOO8TExFBSUsKTTz7JwoULadiwIfPmzWPSpEnMnDmTsWPH8tFHH3HzzTczceLEijd1yRIWLFjAxo0b8fLy4vTp01Vuyvr169mxYwf+/v788MMPzJs3jyFDhnD+/HlWrlzJp59+yueff46vry+bN2+muLiYG264gQEDBlz8VmMJ9lNJnExxaRmPJ2xl7uYMxt/ShjfvjrSrwn9BRHNf5j/WG28PV0bP2MCaA1lGRxLiouTkZHbt2kX//v2Jjo7mtddeIzMzk9zcXHJycrj55psBc7//FVmxYgVjx47Fq7z/dH9//yrX2b9//4vzDR48mFWrVlFcXMySJUvo06cPdevWZfny5XzxxRdER0fTo0cPsrOzOXDggIW22kxa/jaouLSMx7/aysp9J3lpWDhjb7Dcp70RQgO9+f6x3vx55iYenpPIZ3+OoU+7hlW/UNinq7TQbY3Wmo4dO7J+/frLpufk5FTrDDqtdYXzubm5XRxG8souoy90Ow3g6elJ3759WbZsGfPmzWP06NEXl/vRRx8xcODAa96m6rK/pqSDO19q4omEbazcd5Ipd0bYfeG/oFF9T74Z15PWDesx7otE/jhwyuhIwkn4+PiQl5dX4eP27duTlZV1sfiXlJSwe/du/Pz88PX15Y8//gAgISGhwmUPGDCAmTNnUlA+YPaF3T6hoaFs2WLut7KqQeFHjRrFrFmzWLNmzcViP3DgQD799FNKSkoA2L9/P+fOVd0b77WQ4m9DSspMjP96Kyv2nuDVER2J7RFidCSLauBdh4RHetAy0JtHvtjMuoPyASBq36hRo3j77bfp3LkzBw8eZMyYMTz22GNER0dTVlbG/Pnzee655+jUqRPR0dGsW7cOgFmzZvHEE0/Qq1cv6tatW+GyBw0axPDhw4mJiSE6Opp33nkHgAkTJvDpp5/Su3dvTp26+t/5gAED+P333+nXrx916phP3X7kkUcIDw+nS5cuRERE8Oijj1JaWmrBd8WCXTpbmrN16ay1ZsJ3O/h+ayYvDwtnjIO0+CuSnV/M6BkbOHKmkLlxvYgM8jU6kqgh6dLZGDXp0lla/jZi6tJkvt+ayd/6tXPowg/mK4K/fLgHfl51GDt7E2nZlv06K4SomhR/GzDzj8NM/+0gsT2Ceeq2NkbHsYrG9T354uHulJk0f565iVP5xUZHEsKpSPE32OKdx3j15z0M7NiYV0ZEOFW/+K0b1uPzMd04cbaIh2ZvpuC8ZfdpCiEqJ8XfQDszc3nm2yS6BDfgg1GdnXJUrC7BDZh2fxd2Hcnl799ux2SyzWNQQjgaKf4GOXm2iHFfJBLgbe6S2Zn7wL8trDEvDAljya7jfLjKsheyCCEqJhd5GaCopIy4L7eQW1jC/L/0oqGPh9GRDPfwjS3ZdzyPf604QPvGPgyObGp0JCEcmrT8rUxrzQs/7CQpI4f37+tEx2ZymiOAUoopd0bQJdiPZ77dzu6juUZHEg5u8uTJrFixwugYhpHiXwsSEiA0FFxczLeXXhz49aZ0/rP1CH/t15ZBEdK6vZSHmyvT/9QVPy93/vLVVnILS4yOJGrJ1f5HrOWVV16hX79+1l+xjZDib2EJCRAXB2lpoLX5Ni7OPH1nZi7/XLSHPu0a8tStbY2OapMa+Xjy8f2dOZpTyLPzt8toYA7oav8jNZGamkpYWBjjxo2jY8eODBgwgMLCQpKSkujZsydRUVHceeednDlzBjB3n3yh64Xnn3+e8PBwoqKimDBhAgBZWVncfffddOvWjW7durF27dqaBbQ1Wmub/Onatau2RyEhWpv/pC//adHCpG+culL3fH2Fzs4vNjqmzZvx+0Ed8txPesbvB42OIqphz5491Z63sv+RkJCaZTh8+LB2dXXV27Zt01prPXLkSP3ll1/qyMhI/euvv2qttX7xxRf1008/rbXW+sEHH9Tfffedzs7O1u3atdMmk0lrrfWZM2e01lqPHj1ar1mzRmutdVpamu7QoUPNAtaCit53IFFXo8bKAV8LS0+veHpGBtTJKeLbx3rh7207Qy/aqodvbMmmw6d5c8k+Ogf70TWk6q5yhX2o7H+ksunXomXLlkRHRwPQtWtXDh48eFnXzA8++CAjR4687DX169fH09OTRx55hNtvv52hQ4cC5u6aLx3+8ezZs+Tl5eHj41PzoDZAdvtYWHBwxdNd6xfywpAwugQ3sG4gO6WU4u2RnWjmV5fxX2/jzLnzRkcSFlLZ/0hl06+Fh8d/z5xzdXUlJyenyte4ubmxadMm7r77bhYsWMCgQYMAMJlMrF+/nqSkJJKSkjhy5IjDFH6Q4m9xU6ZA+bgOFym3UvqMPs7YG0INyWSvfOu680lsF07lF/N//9kp+/8dREX/I15e5umW5uvrS4MGDVizZg0AX3755cVvARfk5+eTm5vLkCFD+Ne//kVSUhJg7m3z448/vjjfhemOQoq/hcXGQnw8hISAUhrPBkWE3rGX+e8GOVXXDZYS0dyXCQPas3T3cb5LzDQ6jrCAy/9HzLfx8ebptWHOnDlMnDiRqKgokpKSmDx58mXP5+XlMXToUKKiorj55pt5//33Afjwww9JTEwkKiqK8PBwpk+fXjsBDSJdOteilxbuYs76NL58uDs3tZWRq66XyaSJ/Wwj2zNzWPzUTYQGelf9ImFV0qWzMaRLZxu0et9J5qxP46EbWkrhryEXF8W793bCzUXx13lJlJSZjI4khN2T4l8LTuUXM3H+djo08eHZQe2NjuMQmvnV5fW7IknKyOGjVSlGxxHC7knxtzCtNZN+2MnZolI+GNXZqTtss7ShUc24q0tzPl51gG3pZ4yOI65gq7uQHVVN328p/hb2445jLNt9gmf6t6N9E8c5LcxW/HN4RxrX9+TZ+TsoLi0zOo4o5+npSXZ2tnwAWInWmuzsbDw9Pa97GXKRlwWdyi/mpYW76NTCj0dudOyhGI3i4+nOG3dFMmbWZj5ceYCJAzsYHUkAQUFBZGZmkpWVZXQUp+Hp6UlQUNB1v16KvwVNXriLc8VlvHNPFG6u8qWqtvRt34h7ugYx/bdDDOrYVAaAtwHu7u60bCkNHnsiFcpCft5xjMU7j/N0v7a0bSy7e2rbi7eHE+Bdh4nzt3O+VM7+EeJaSfG3gOz8YiYv3EVkc18e7dPK6DhOwdfLndfvjGTf8Tw+Xi1n/whxraT4W8A/f9zD2aIS3hnZSXb3WFG/8MbcEd2MT1anyOAvQlwjqVQ19Nv+LBZtP8oTt7SRs3sM8NKwjvh5ufPCf3ZSJoO/C1FtFin+SqlBSqlkpVSKUur5Cp4fo5TKUkollf88Yon1Gq3wfBn/WLCTVg29+Uvf1kbHcUoNvOvwj9vD2Z6ZS8LGNKPjCGE3alz8lVKuwDRgMBAOjFZKhVcw6zytdXT5z2c1Xa8t+GjVATJOFzLljkg83ORiLqOMiG7GjW0CeXtpMifOFhkdRwi7YImWf3cgRWt9SGt9HpgLjLDAcm1a8vE84n8/xD1dg+jVOsDoOE5NKcVrd0RQXGbilR/3VP0CIYRFin9zIOOSx5nl0650t1Jqh1JqvlKqhQXWaxiTydyFg4+nGy8MkZ4MbUFooDdP3tKGn3ceY3XySaPjCGHzLFH8K+qk/sojbz8CoVrrKGAFMKfCBSkVp5RKVEol2vKVgvMSM0hMO8MLQ8JkSEYbEndzK9o0qseLC3ZReF66fhDiaixR/DOBS1vyQcDRS2fQWmdrrYvLH84Aula0IK11vNY6Rmsd07ChbXaDfCq/mDcW76VHS3/u6Xr9l1YLy/Nwc2XKHRFkninkg5UHjI4jhE2zRPHfDLRVSrVUStUBRgGLLp1BKdX0kofDgb0WWK8hpi7ZR2FJGVPujJSRuWxQj1YBjOwaxGdrDpFyMs/oOELYrBoXf611KTAeWIa5qH+rtd6tlHpFKTW8fLanlFK7lVLbgaeAMTVdrxG2pp/huy2ZPHRjS9o0qmd0HFGJ5wd3wKuOKy8v2iO9TApRCRnGsZpMJs0dn6zleG4Rqyb0pZ6H9Ilny+asS+WlRbuZ/kAXBkU0rfoFQjgIGcbRwr5NzGBHZi6Tbg+Twm8HYnsE06GJD6/+tFcO/gpRASn+1ZBbUMJby5LpHurP8E7NjI4jqsHN1YWXh3fkSE4h0387aHQcIWyOFP9qeO+XZHIKzvPy8I5ykNeO9GwVwLBOzZj+20EyThcYHUcImyLFvwp7j53lyw1pxPYIIbxZfaPjiGv0wpAOuCjFqz/Jlb9CXEqK/1VorXlp0W5867rz9wHtjI4jrkNT37o8eVsblu85wYvvniU0FFxcIDQUEhKMTieEcaT4X8Wi7UfZdPg0Ewd2wM9LruS1Vw/f2BKv9Ja8/n/epKWB1pCWBnFx8gEgnJcU/0oUni/jzSX7iGhen/u62XVXRE7Pw82V7N/aYyq5vOfVggKYNMmgUEIYTIp/JWasOcSx3CImD+2Iq4sc5LV3Wccq/lNPT7dyECFshBT/Cpw4W8Snvx5kcEQTurf0NzqOsIDg4Io/wIODrRxECBshxb8C7y5PptRk4vnBHYyOIixkyhTw8rp8mpeXeboQzkiK/xV2H83luy2ZjOkdSkiAt9FxhIXExkJ8PAS10IDGO6CY+HjzdCGckRT/S2itmfLzXvzqujP+1rZGxxEWFhsLGemKT1YfIvCRFYT2OGV0JCEMI8X/Eiv3nmTdwWz+2q8dvnXdjY4jasnYG0IJalCXV3/aQ5nJNjs2FKK2SfEvV1Jm4vXFe2nd0Jv7e8hRQEfm6e7K/w0OY9/xPL5LzKj6BUI4ICn+5b7akMahU+eYdHsY7q7ytji6IZFNiAlpwDvL95NfXGp0HCGsTqockFNwnn+tOMCNbQK5pX0jo+MIK1BK8Y+h4ZzKL+bTX1OMjiOE1UnxBz5alcLZohIm3R4mvXY6kegWftwR3YwZaw5zNKfQ6DhCWJXTF//UU+f4Yn0q98W0IKyp9NrpbCYMbA8a3vtlv9FRhLAqpy/+7yxPxs3FhWf6S6+dziiogRdjbgjl+62Z7D121ug4QliNUxf/7Rk5/LTjGONuakmj+p5GxxEGeaJvG+p7ujN16T6jowhhNU5b/LXWvLlkH/7edRjXp5XRcYSBfL3ceeKW1vyanMW6FLnwSzgHpy3+v+3PYv2hbJ66tQ0+nnJBl7P7c69QmvvV5Y0l+zDJhV/CCThl8TeZzK3+YH8v7u8RYnQcYQM83V35+4B27DySy487jhodR4ha55TFf0HSEfYdz2PCwPbUcXPKt0BU4I7o5oQ1rc87y5MpLi0zOo4QtcrpKl9RSRnvLt9PZHNfhkY2NTqOsCEuLor/G9yBjNOFfLVBRnkRjs3piv9XG9I4klPI84M74CIjdIkr9GnXkJvaBvLxqgPkFpYYHUeIWuNUxT+3sISPV6dwU9tAbmgTaHQcYaOeG9SBMwUlTP/toNFRhKg1TlX8p/92kJyCEhmhS1xVRHNf7uzcnJl/HOZYrnT7IByT0xT/47lFzPzjMHdEN6NjM1+j4wgb90z/dmgN7y2Xbh+EY3Ka4v/+L/vRGv4+oL3RUYQdaOHvxYO9Q5i/NZN9x6XbB+F4nKL4HziRx3dbMnigZwgt/L2qfoEQwBO3tMHHw42pS6TbB+F4nKL4T12ajHcdN8bf2sboKMKO+HnV4fFb2rA6OYv1B7ONjiOERTl88d+cepoVe0/wWN/W+HvXMTqOsDNjeofS1NeTN5fsRWvp9kE4Docu/lpr3li8l0Y+Hoy9IdToOMIOebq78kz/dmzPzGXxzuNGxxHCYhyu+CckQGgouLhAk+Zl/L7Ei7/1b4dXHTejowk7dVeXINo39uHtZfsoKTMZHUcIi3Co4p+QAHFxkJYGWsPJY26cWRZFaXKQ0dGEHXN1UTw3uD2p2QXM3STdPgjHYJHir5QapJRKVkqlKKWer+B5D6XUvPLnNyqlQi2x3itNmgQFBZdPM5W48uKLDvUZJwxwS/tG9GjpzwcrD5BfXGp0HCFqrMb7QpRSrsA0oD+QCWxWSi3SWu+5ZLaHgTNa6zZKqVHAVOC+mq77SumVNMrS00wwa5ilVyeciAI+0yXsPn+WM9PqUq+BnDIsalGTSBj8Zq2uwhJN4u5Aitb6kNb6PDAXGHHFPCOAOeX35wO3KaUs3qtacHAl0wOyLL0q4YR8PNzx967Dsdwizsu+f1FLyrRGU/tnllniKGhzIOOSx5lAj8rm0VqXKqVygQDAomPmTZkC48ZpCgv/+7ni5QVTPmgMsT9bclXCSbmfOsf97/3G6MBgXr0jwug4wgE9/fVWsjPO87XW1EIb+SJLtPwrSnflx1Z15kEpFaeUSlRKJWZlXXtrPTYWPphWhm/D8yilCQmB+HjzdCEsoWWgN6O7B/PNpnQOnzpndBzhYHZk5vDTjmPEhDao1cIPlin+mUCLSx4HAVeOg3dxHqWUG+ALnL5yQVrreK11jNY6pmHDhtcVZtxYN3JO1sFkUqSmSuEXlvfUbW2p4+bC28uk2wdhOVqbh5dt4OVOXJ9Wtb4+SxT/zUBbpVRLpVQdYBSw6Ip5FgEPlt+/B1il5XJJYaca+ngw7qZWLN55nG3pZ4yOIxzEmgOnWL7Ig4xPbsHXy53QUPPp67WlxsVfa10KjAeWAXuBb7XWu5VSryilhpfP9jkQoJRKAZ4B/ud0UCHsybg+rQisV4c3l+yTbh9EjZlMmidfzebMsiiyT7ijtfl6pbi42vsAULb6hxsTE6MTExONjiFEpb5Yn8rkhbuZOSaGWzs0NjqOsGMLk45w980NKDv7v6cQh4RAamr1l6WU2qK1jqlqPrn6SYjrNLp7MKEBXkxdkkyZyTYbUcL2FZeW8fayZMrO1q3w+cquX6opKf5CXCd3VxcmDuxA8ok8/rM10+g4wk59vTGdzDOFNG5W8bUjlV2/VFNS/IWogSGRTegU5Mt7v+ynqKTM6DjCzuQVlfDRqhR6tw7g3bdc8Lpir4+Xl/n6pdogxV+IGlBK8fzgMI7lFjFnXarRcYSdmfH7IU6fO89zgzoQG6uIjzfv41eKWr9OSYq/EDXUq3UAt7RvyLTVKeQUnDc6jrATJ/OKmLHmMLdHNaVTCz/AXOhTU8FkotavU5LiL4QFPDe4A3nFpXzy60Gjowg78eHKA5SUmZgwoL0h65fiL4QFdGhSn7s6BzF7XSpHcgqNjiNs3OFT5/hmUwajuwfTMtDbkAxS/IWwkGcGtAPgveX7DU4ibN07y5PxcHPhydvaGJZBir8QFtLcry5je4fyn22Z7D121ug4wkZtz8jh5x3HeOSmVjTy8TQshxR/ISzoL31b4+PhxtSl0umb+F8XOm8L8K7DuJtaGppFir8QFuTnVYcnbmnDr8lZrDto0eEqhAP4/cAp1h/K5slb2+Dj6W5oFin+QljYg71DaebryVTp9E1cwmQyt/pb+Nfl/h4hRseR4i+EpXm6u/LMgPZsz8zl553HjI4jbMSCpCPsPXaWCQPaU8fN+NJrfAIhHNCdnZvToYkPby9LpkTG+3V6hefNnbdFBfkyLKqZ0XEAKf5C1ApXF8VzgzqQll3AN5tqqVtGYTdmrj3MsdwiXhgShotL7Q7PWF1S/IWoJX3bN6RnK38+WHGA/OJSo+MIg2TlFfPJ6hT6hzemZ6sAo+NcJMVfiFpyodO37HPnmfH7IaPjCIN8sHI/RaUmnh/cwegol5HiL0Qtim7hx+2RTZmx5hAn84qMjiOsLOVkHt9syiC2RzCtG9YzOs5lpPgLUcsmDmzP+VITH648YHQUYWVvLtlHXXdXnr6trdFR/ocUfyFqWWigN/f3COabTRkcyso3Oo6wknUHT7Fi70kev6U1AfU8jI7zP6T4C2EFT97aFk83F95Znmx0FGEFJpPm9cV7aebryUM3GNuNQ2Wk+AthBQ19PBjXpxWLdx4nMfW00XFELVu4/Qi7jpxl4qD2eLq7Gh2nQlL8hbCSuD6taFLfk1d+2oPJJN0+OKqikjLeXppMZHNfRnRqbnScSknxF8JKvOq48dzg9uzer1EjAAASfklEQVTIzOWHbUeMjiNqyed/HOaojV3QVREp/kJY0YhOzenUwo+pS/dxTi78cjgnzhYxrfyCrl6tbeeCropI8RfCilxcFC8NC+dkXjHTf5Pxfh3NW0uTKS3TTBoSZnSUKknxF8LKugQ3YER0M+J/P0TmmQKj4wgLScrI4futmTx0Y0tCDRqX91pI8RfCAM8N6oBS5ouAhP0zmTQvL9pNQx8Pxt9q3Li810KKvxAGaOZXl0f7tOanHcfYLKd+2r2F24+QlJHDswPbU8/Dzeg41SLFXwiDPHZza5r6evLKj3Lqpz07V1zKm0v2ERXky91dgoyOU21S/IUwSN06rjw3qAM7j+Ty/dZMo+OI6zT9t4OcOFvMS8M62vSpnVeS4i+ElSUkQGgouLjA03c0I/B4G6YuTeZsUYnR0cQ1yjhdwL9/P8SI6GZ0DWlgdJxrIsVfCCtKSIC4OEhLA60hPV2x+9t2pG0K4F+/SK+f9uaNJXtxVcrm+uqvDin+QljRpElQcMXZnUWFivMbOjJnfSr7jp81JJe4dn8cOMXincf5S9/WNPWta3ScaybFXwgrSq9kON+8U+7U93Rj8oLdaC0Hf21dcWkZkxfuIiTAi7g+rYyOc12k+AthRcHBlU037zrYlHqaBUnS74+t+2zNYQ6dOsc/h3e02V47q1Kj4q+U8ldK/aKUOlB+W+ERD6VUmVIqqfxnUU3WKYQ9mzIFvLwun+blZZ4+smsLolv4MeXnfXLw14ZlnC7go1UHGBzRhL7tGxkd57rVtOX/PLBSa90WWFn+uCKFWuvo8p/hNVynEHYrNhbi4yEkBJQy38bHm6e7uCheuyOC7HPFvP/LfqOjikr888c9uCjFi0PDjY5SIzUt/iOAOeX35wB31HB5Qji82FhITQWTyXwbG/vf5yKa+/JAjxDmrEtlz1E5+GtrVuw5wYq9J3j6trY087O/g7yXqmnxb6y1PgZQflvZdyBPpVSiUmqDUko+IIS4igkD2uPnVYfJC3fJlb82pPB8GS//uJu2jerx0I22OTTjtaiy+CulViildlXwM+Ia1hOstY4B7gf+pZRqXcm64so/JBKzsrKuYfFCOA5fL3eeH9yBxLQzzEvMMDqOKPfJrylknink1TsicHe1/3NlqtwCrXU/rXVEBT8LgRNKqaYA5bcnK1nG0fLbQ8CvQOdK5ovXWsdorWMaNmx4nZskhP0b2TWIXq0CeH3xXk6eLTI6jtPbfyKP6b8d5M7OzenZyrYHaamumn58LQIeLL//ILDwyhmUUg2UUh7l9wOBG4A9NVyvEA5NKcXrd0VSXGripUW7jY7j1MpMmue+30E9Dzf+cbvtD9JSXTUt/m8C/ZVSB4D+5Y9RSsUopT4rnycMSFRKbQdWA29qraX4C1GFloHePH1bW5bsOs7y3ceNjuO0vtqQxrb0HCYPCyegnofRcSxG2erVhDExMToxMdHoGEIYqqTMxLCP/iCnoIRfnumDj6e70ZGcypGcQga89xtdQ/2ZM7YbStl+r51KqS3lx1ivyv6PWgjhwNxdXXjz7ihO5BXx1tJko+M4Fa01//hhJxp4/c4Iuyj810KKvxA2LrqFH2N6h/LVxjQSZdQvq1m0/Sirk7OYMKA9QQ28qn6BnZHiL4QdmDCgPc186/Ls/B0Uni8zOo7Dy84v5p8/7iG6hR8P9g41Ok6tkOIvhB3w9nDj7XuiOHTqHG8tk0Hfa5PWmkk/7CK/qJSpd0fhakejc10LKf5C2InebQIZ0zuUWWtTWXfwlNFxHNaCpCMs3X2cZwa0o30TH6Pj1Bop/kLYkecGdaBloDcTv9tBfnGp0XEczrHcQiYv3E1MSAPG3WSf/fRXlxR/IexI3TquvDMyimO5hUz5WS6XsSStNc/O30FpmeadkZ0cdnfPBVL8hbAzXUP8ievTmm82ZbA6ucIeVcR1+GpjOmsOnOKF28MIDfQ2Ok6tk+IvhB36W/+2tGtcj2fn7yA7v9joOHbvUFY+r/+8l5vaBvJAj0qGW3MwUvyFsEMebq58MKozuYUl/P277dL1cw0Ul5bx5Dfb8HB34a17ohzuYq7KSPEXwk6FNa3PP24P49fkLGauPXzNr09IgNBQcHEx3yYkWDyiXZi6JJmNv9Tj2PTbaN6grtO8F1L8hbBjf+oZwoDwxkxduo+dmbnVfl1CAsTFQVoaaG2+jYtzjqJ3qVX7TvBhfDFnf+nEyWOuTvVeSPEXwo4ppXjrnigC63nw5Ddbq33656RJUFBw+bSCAvN0Z3E8t4gJ3+3g3NowSoovL4XO8F5I8RfCzvl51eGDUZ1JP13ApB92Up2eetPTr226oyktM/H03G0Uni+jOKfibpod/b2Q4i+EA+je0p+/9mvHwqSjzFmXWuX8wZWc0FLZdEczdek+Nh4+zZQ7IwgOrvgAr6O/F1L8hXAQ429pQ7+wRrz28142Hb56759TpoDXFR1VenmZpzu6H7cfZcaawzzYK4S7ugQ57XshxV8IB+Hionjvvmha+HvxeMJWjudWPvZvbCzEx0NICChlvo2PN093ZMnH83h2/g5iQhow6fZwwHnfCxnJSwgHs/9EHndMW0v7Jj7MjeuJh5ur0ZFsQm5hCXdMW0t+cSk/P3kjjep7Gh2pVshIXkI4qXaNfXj7nk5sS8/hxQW7qnUA2NGVlJkY//VWMk4X8GlsF4ct/NfCzegAQgjLuz2qKcnH2/DhqhRCArx54pY2RkcyjNaalxbtZs2BU7x1dxQxof5GR7IJUvyFcFB/69+OtNMFvL0smWB/L4Z1amZ0JEPMWHOIrzem83jf1tzbrYXRcWyGFH8hHNSFC8CO5hTy9++208zPk64hztXqXbrrGG8s2cftkU2ZMKC90XFsiuzzF8KBebi5Ev+nGJr71eXhOYkkH88zOpLVrDt4iqfmJhHdwo937+2Ei4P3z3+tpPgL4eAaeNdhztjueLi58MDnGzl86pzRkWpdUkYO4+YkEuLvxcwHu+HpLmc8XUmKvxBOIDjAi68e7kGZSfPAZxs5mlNodKRak3w8jzGzNhFQz4OvHulBA+86RkeySVL8hXASbRv78MVD3TlbWMIDn23kxNnKLwKzV/tP5BH72UY83FxIeKQHjeWUzkpJ8RfCiUQ092XW2G6cOFvEvf9eT+aZgqpfZCd2H81lVPwGXBQkPNKTFv5eVb/IiUnxF8LJxIT689UjPThz7jz3Tl/vEMcAtmfkcP+MjXi6ufDto71o06ie0ZFsnhR/IZxQ5+AGzI3rRXGpiZHT17PrSPUHgrE1vyaf5P4ZG6hf1415j/ZyisHXLUGKvxBOKrxZfeY92hMPNxdGTl/P8t3HjY50zb7emM7DcxIJCfDmu0d7y66eayDFXwgn1qaRDz880Zt2jevx6FdbmPH7IbvoC6jMpHljyV5e+GEnfdoG8u1jvWjiKwd3r4UUfyGcXCMfT+bG9WJQxyZMWbyXp+cmVXs4SCOcyi/mT59v5N+/HeKBnsHM+HMM9Tyks4JrJe+YEIK6dVyZdn8Xpq1O4f0V+9l1JJdpsV0Ia1rf6GiX2Xgom6fmbiOnoIS37oni3hjpq+d6SctfCAGYB4N58ra2fD2uJ/nFpYyYtpZpq1MoKTMZHY3C82W88uMeRs3YQF13V354/AYp/DUkxV8IcZmerQJY/PRN9AtrxNvLkrlj2lpDzwZam3KKIR+uYebaw/ypZwg/P3UT4c1s6xuJPZKRvIQQlVq66xj/WLCb7HPF3NU5iAkD29HUt65V1n0oK5/XF+9lxd6TtPCvy9S7o+jdOtAq67Zn1R3JS/b5CyEqNSiiKb1aB/LJ6hRmrU3lpx1Hie0RwkM3hhLUoHZOq0w5mcenvx5iYdIRPN1deW5QB8beECqds1lYjVr+SqmRwMtAGNBda11hU10pNQj4AHAFPtNav1nVsqXlL4RtyThdwPsr9rMo6SgaGBzRhHu6BnFjm0DcXGu2B7m4tIyVe0/yXWIGv+7PwsPNhdHdg3m8bxsa+nhYZgOcRHVb/jUt/mGACfg3MKGi4q+UcgX2A/2BTGAzMFprvedqy5biL4RtOppTyKy1h/k2MZPcwhIC63lwa4eG3NS2IT1a+dPIp+rz7bXWZJ4pZHPqaVYnZ/Fb8knOFpXSpL4n93ZrwYO9QgioJ0X/elhlt4/Wem/5yq42W3cgRWt9qHzeucAI4KrFXwhhm5r51WXS7eFMGNie1fuyWLT9CEt3HefbxEwAArzr0K6xD43rexBQzwNPdxdMGs6XmjiZV8yJ3CKST+SRW1gCQGA9D/qHN2F4dDNubBOIqwy6YhXW2OffHMi45HEm0KOiGZVScUAcQHBwcO0nE0JcNw83VwZFNGFQRBNKy0zsOJJLUnoO+46f5cDJfLakn+FU3nlKyky4KIWbq6KRjweNfDwZEtmUiOb16RTkR3jT+jLKlgGqLP5KqRVAkwqemqS1XliNdVT0W61wX5PWOh6IB/Nun2osWwhhA9xcXegS3IAuwQ2MjiKqqcrir7XuV8N1ZAKXXo0RBByt4TKFEELUgDUu8toMtFVKtVRK1QFGAYussF4hhBCVqFHxV0rdqZTKBHoBPyullpVPb6aUWgygtS4FxgPLgL3At1rr3TWLLYQQoiZqerbPD8APFUw/Cgy55PFiYHFN1iWEEMJypG8fIYRwQlL8hRCGSUiA0FBwcTHfJiQYnch5SN8+QghDJCRAXBwUFJgfp6WZHwPExhqXy1lIy18IYYhJk/5b+C8oKDBPF7VPir8QwhDp6dc2XViWFH8hhCEq68FFenaxDin+QghDTJkCXlcMCeDlZZ4uap8UfyGEIWJjIT4eQkJAKfNtfLwc7LUWOdtHCGGY2Fgp9kaRlr8QQjghKf5CCOGEpPgLIYQTkuIvhBBOSIq/EEI4IaW1bY6WqJTKAtJqsIhA4JSF4tgLZ9tmZ9tekG12FjXZ5hCtdcOqZrLZ4l9TSqlErXWM0Tmsydm22dm2F2SbnYU1tll2+wghhBOS4i+EEE7IkYt/vNEBDOBs2+xs2wuyzc6i1rfZYff5CyGEqJwjt/yFEEJUwq6Lv1JqkFIqWSmVopR6voLnPZRS88qf36iUCrV+SsuqxjY/o5Tao5TaoZRaqZQKMSKnJVW1zZfMd49SSiul7P7MkOpss1Lq3vLf9W6l1NfWzmhp1fjbDlZKrVZKbSv/+x5iRE5LUUrNVEqdVErtquR5pZT6sPz92KGU6mLRAFpru/wBXIGDQCugDrAdCL9inseB6eX3RwHzjM5thW2+BfAqv/8XZ9jm8vl8gN+BDUCM0bmt8HtuC2wDGpQ/bmR0bitsczzwl/L74UCq0blruM19gC7ArkqeHwIsARTQE9hoyfXbc8u/O5CitT6ktT4PzAVGXDHPCGBO+f35wG1KKWXFjJZW5TZrrVdrrS+MjLoBCLJyRkurzu8Z4FXgLaDImuFqSXW2eRwwTWt9BkBrfdLKGS2tOtusgfrl932Bo1bMZ3Fa69+B01eZZQTwhTbbAPgppZpaav32XPybAxmXPM4sn1bhPFrrUiAXCLBKutpRnW2+1MOYWw72rMptVkp1BlporX+yZrBaVJ3fczugnVJqrVJqg1JqkNXS1Y7qbPPLwANKqUxgMfCkdaIZ5lr/36+JPQ/mUlEL/spTl6ozjz2p9vYopR4AYoCbazVR7bvqNiulXID3gTHWCmQF1fk9u2He9dMX87e7NUqpCK11Ti1nqy3V2ebRwGyt9btKqV7Al+XbbKr9eIao1fplzy3/TKDFJY+D+N+vgRfnUUq5Yf6qeLWvWbauOtuMUqofMAkYrrUutlK22lLVNvsAEcCvSqlUzPtGF9n5Qd/q/m0v1FqXaK0PA8mYPwzsVXW2+WHgWwCt9XrAE3MfOI6qWv/v18uei/9moK1SqqVSqg7mA7qLrphnEfBg+f17gFW6/EiKnapym8t3gfwbc+G39/3AUMU2a61ztdaBWutQrXUo5uMcw7XWicbEtYjq/G0vwHxwH6VUIObdQIesmtKyqrPN6cBtAEqpMMzFP8uqKa1rEfDn8rN+egK5Wutjllq43e720VqXKqXGA8swnykwU2u9Wyn1CpCotV4EfI75q2EK5hb/KOMS11w1t/ltoB7wXfmx7XSt9XDDQtdQNbfZoVRzm5cBA5RSe4AyYKLWOtu41DVTzW3+OzBDKfU3zLs/xthzY04p9Q3m3XaB5ccxXgLcAbTW0zEf1xgCpAAFwFiLrt+O3zshhBDXyZ53+wghhLhOUvyFEMIJSfEXQggnJMVfCCGckBR/IYRwQlL8hRDCCUnxF0IIJyTFXwghnND/A/9YS9YpzkrcAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# M=0\n", "p_lsq_0 = fitting(M=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### M=1" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting Parameters: [-1.50626624 0.77828571]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XlYVGX7wPHvwyIIIoq4K+CuCIqKuW/lvmZpaVRqqZmalq9Wb74tr7+sbDN303JLcknf1NxzSzOXUHGXXFHcNxAVkOX5/XGo1EBQZuYMzP25Li6Yw5lz7jPKPWee5X6U1hohhBCOxcnsAIQQQtieJH8hhHBAkvyFEMIBSfIXQggHJMlfCCEckCR/IYRwQJL8hRDCAUnyF0IIByTJXwghHJCL2QFkxtfXVwcEBJgdhhBC5Cq7du26orUumtV+dpv8AwICiIiIMDsMIYTIVZRS0dnZT5p9hBDCAUnyF0IIByTJXwghHJDdtvkLIXKP5ORkYmJiSExMNDsUh+Hu7k6ZMmVwdXV9pOdL8hdC5FhMTAxeXl4EBASglDI7nDxPa83Vq1eJiYmhXLlyj3QMafYRVhceDgEB4ORkfA8PNzsiYWmJiYkUKVJEEr+NKKUoUqRIjj5pyZ2/sKrwcOjfH27fNh5HRxuPAcLCzItLWJ4kftvK6estd/55mD3ccY8c+Xfi/9Pt28Z2IYR5JPnnUX/ecUdHg9Z/33Hb+g3g9OmH2y6EWXr37s2iRYvMDsNmJPnnUfZyx+3n93DbhbAErTVpaWlmh2HXJPnnUfZyxz16NHh43LvNw8PYLoQlnTp1imrVqjFw4EBq167Nd999R4MGDahduzbdu3fn5s2bAIwaNYq6desSFBRE//790VqbHLk5pMM3j/LzM5p6MtpuS3926o4cabzx+PkZiV86e/Ou//50kEPnblj0mIGlCvJ+p+pZ7hcVFcXMmTMZNWoUTz31FOvWrcPT05MxY8bw5Zdf8t577zF48GDee+89AF544QWWL19Op06dLBpvbmCRO3+l1Ayl1CWl1IFMfq+UUuOVUseUUvuUUrUtcV6ROXu64w4Lg1OnIC3N+C6JX1iLv78/9evXZ/v27Rw6dIhGjRoREhLC7NmziU6/G9q4cSP16tUjODiYDRs2cPDgQZOjNoel7vxnAROBOZn8vh1QKf2rHjAl/buwErnjFmbJzh26tXh6egJGm3+rVq2YN2/ePb9PTExk4MCBREREULZsWT744AOHnZVskeSvtd6slAp4wC5dgDnaaFzbrpQqpJQqqbU+b4nzi4yFhWWc7BPupBJz/TYxsQncTkolMTkVpaCguyveHq74+XhQzMtNxm2LXKt+/foMGjSIY8eOUbFiRW7fvk1MTAzFihUDwNfXl5s3b7Jo0SK6detmcrTmsFWbf2ngzF2PY9K33ZP8lVL9gf4AfjIcxGIuxSey4fAldp66xp7TsZy8civL5xRwcyGwZEHqlfehQfki1C3ng6uzjA8QuUPRokWZNWsWPXv2JCkpCYAPP/yQypUr069fP4KDgwkICKBu3bomR2oeZame7vQ7/+Va66AMfrcC+Fhr/Wv64/XAm1rrXZkdLzQ0VMtiLo/uZlIKSyPPsnhXDHvOxKI1+BZwo5ZfIYJLe+NfxIMyhfPj5e6Km4sTWkN8YgrXbt8h+uotjl26yd4zsew/G0eahsIerrQLLsnTtctQ26+QfCoQ9zh8+DDVqlUzOwyHk9HrrpTapbUOzeq5trrzjwHK3vW4DHDORud2KOfjEpi66TiLdsVw604qVYp78UbLyrSpXoLKxQtkM2n/vQJcfGIyW49dZcX+8/y4+yzf7zhNzTLevNS4HB1rlMLZSd4EhMiNbJX8lwGDlVLzMTp646S937Iu3Uhk4sZjzN95hjSt6RJSmrD6ftQqm7O7dC93V9oGlaBtUAluJaWweHcMs7aeYuj8SCZuOMa/WhtvLPJJQIjcxSLJXyk1D2gO+CqlYoD3AVcArfVUYCXQHjgG3Ab6WOK8AlJS05i9LZqxP/9BYnIq3UPLMLB5Rcr6eGT95Ifk6ebCiw0CeL6eP6sPXuDztVEMmLubxwJ8GN01iErFvSx+TiGEdVhqtE/PLH6vgUGWOJf42/6YOEYs2suRC/E0rVyU/3auTjlfT6uf18lJ0T64JK0Di/PDrhjGrD5Cu3Fb6N+0PEOeqIS7q7PVYxBC5IzM8M2FUtM0U385ztif/8C3gBtTn69Dm+rFbd704uLsRM/H/GgdWJyPVx1h8qbjbDhyiQk9a8mnACHsnIzdy2Uu3Uik57TtfLYmijZBJVj9ehPaBpnb5l6kgBufd6/JzN51uRyfRMcJvzJ3e7TD1kwRIjeQ5J+L7D59nY4TfmX/2Ti+fKYmE3vWopBHPrPD+kuLqsVY9XoT6pUvwn+WHOCtxftISkk1OyzhIMaPH0+1atUICwtj2bJlfPLJJwAsWbKEQ4cO/bXfrFmzOHfu4QYbnjp1iqCgf4xiz9Wk2SeXWPD7ad5dcpDi3m7Mfqkh1UoWNDukDBXzcmdW77p8tf4o49cf5dilm0x9vg7FCrqbHZrI4yZPnsyqVav+WtO2c+fOgJH8O3bsSGBgIGAk/6CgIEqVKmVarAApKSm4uJiXguXO385prflibRRvLd5PvfI+/DS4sd0m/j85OSmGtarM5LDaHD4fT9fJv3H88k2zwxJ52IABAzhx4gSdO3dm7NixzJo1i8GDB/Pbb7+xbNkyRowYQUhICGPGjCEiIoKwsDBCQkJISEhg165dNGvWjDp16tCmTRvOnzdGoe/atYuaNWvSoEEDJk2alOm5P/30U4KDg6lZsyZvv/02AM2bN+fPSapXrlwhICAAMN54unfvTqdOnWjdujXPPvssK1eu/OtYvXv3ZvHixaSmpjJixAjq1q1LjRo1+Prrry3+msmdvx1LSU3jP0sOMP/3MzwbWpbRXYNwyUUlFtoHl8TPx4PeM3fSfeo2ZvauS82yhcwOS1jbqrfhwn7LHrNEMLT7JNNfT506ldWrV7Nx40Z8fX2ZNWsWAA0bNqRz58507Njxrxo+q1at4vPPPyc0NJTk5GRee+01li5dStGiRVmwYAEjR45kxowZ9OnThwkTJtCsWTNGjBiR8aWuWsWSJUvYsWMHHh4eXLt2LctL2bZtG/v27cPHx4cff/yRBQsW0L59e+7cucP69euZMmUK3377Ld7e3vz+++8kJSXRqFEjWrdu/denGkvIPZnEwSSlpDIwfDfzfz/D4BYV+eTp4FyV+P8UVNqbRQMa4unmTM/p29ly9LLZIQnxl6ioKA4cOECrVq0ICQnhww8/JCYmhri4OGJjY2nWrBlg1P3PyLp16+jTpw8e6fXTfXx8sjxnq1at/tqvXbt2bNiwgaSkJFatWkXTpk3Jnz8/a9euZc6cOYSEhFCvXj2uXr3K0aNHLXTVBrnzt0NJKakMnLub9Ucu8X6nQPo0sty7vRkCfD1ZPKAhL87YycuzI/jmxVCaVi6a9RNF7vSAO3R7o7WmevXqbNu27Z7tsbGx2RpBp7XOcD8XF5e/lpG8v2T0n2WnAdzd3WnevDlr1qxhwYIF9OzZ86/jTpgwgTZt2jz0NWVX7ruVzOPupKQxKHwP649cYnTXoFyf+P9UrKA78/rVp0LRAvSbE8GvR6+YHZJwEF5eXsTHx2f4uEqVKly+fPmv5J+cnMzBgwcpVKgQ3t7e/PrrrwCEh4dneOzWrVszY8YMbqcvmP1ns09AQAC7dhl1K7NaFL5Hjx7MnDmTLVu2/JXs27Rpw5QpU0hOTgbgjz/+4NatrKvxPgxJ/nYkOTWNwd/vZt3hi/xfl+qE1fM3OySLKuyZj/C+9Sjn60nfOb/z23F5AxDW16NHDz777DNq1arF8ePH6d27NwMGDCAkJITU1FQWLVrEW2+9Rc2aNQkJCeG3334DYObMmQwaNIgGDRqQP3/+DI/dtm1bOnfuTGhoKCEhIXz++ecADB8+nClTptCwYUOuXHnw//PWrVuzefNmWrZsSb58xtDtvn37EhgYSO3atQkKCuKVV14hJSXFgq+KBUs6W5qjlXTWWjP8h30s3h3DB50C6Z1H7vgzcvVmEj2nb+fs9QTm929AcBlvs0MSOSQlnc2Rk5LOcudvJ8asjmLx7hjeaFk5Tyd+MGYEf/dyPQp55KPPrJ1EX7Xsx1khRNYk+duBGb+eZOovxwmr58eQJyqaHY5NFC/ozpyXHyM1TfPijJ1cuZlkdkhCOJS8mfwjv4fbWY+3tQcr95/n/1Ycok314ozqEuRQdfErFC3At73rcvFGIi/N+p3bdyzbpimEyFzeS/5Xj8OSV+HLQFg+DK4cMzuiTO2PiWPYwkhq+xVmXI9aDrkqVm2/wkx6rjYHzsbxr4V7SUuzzz4oIfKavJf8i1SAgdshuBvs+Q4mhsK8nnDqV7Cjzu1LNxLpNyeCIp5GSWZHroH/RLXivNO+GqsOXGD8BstOZBFCZCzvJX+AYtWgy0R44yA0exPO7IBZHWBac9i/CFKTTQ0vMTmV/t/tIi4hmWkv1qGol5up8diDlxuXo1udMny17iir9ssKn0JYW95M/n8qUAxavGO8CXT8Cu7cgsUvw7gQ2DoeEuNsHpLWmnd+3E/kmVjGPluT6qVkmCOAUorRXYOo7VeIYQv3cvCc7f9thGN57733WLdundlhmCZvJ/8/ueaH0D4waCc8txB8ysHP7xr9Aqv/DdejLXq68HAICAAnJ+P73ZMDv995mv/tPsvrLSvRNqikRc+b27m5ODP1hToU8nDl1bm7iUsw9xOasJ4H/Y3YyqhRo2jZsqXtT2wnHCP5/8nJCSq3gd7Lof8vUKU97JwG40NgYS+IyfmksvBw6N8foqONLoboaONxeLjRwfvfZYdoWrkoQx6vZIELynuKebkz8blanItN4M1Fe2U1sDzoQX8jOXHq1CmqVatGv379qF69Oq1btyYhIYHIyEjq169PjRo16Nq1K9evXweM8sl/ll54++23CQwMpEaNGgwfPhyAy5cv8/TTT1O3bl3q1q3L1q1bcxagvdFa2+VXnTp1tE3Exmi99l2tPyqr9fsFtf6mtdYHl2qdmvJIh/P319r4L33vV9myabrxmPW6/kfr9NWbSZa9hjxo+ubj2v+t5Xr65uNmhyKy4dChQ9neN7O/EX//nMVw8uRJ7ezsrPfs2aO11rp79+76u+++08HBwXrTpk1aa63fffddPXToUK211r169dI//PCDvnr1qq5cubJOS0vTWmt9/fp1rbXWPXv21Fu2bNFaax0dHa2rVq2aswCtIKPXHYjQ2cixUtXTuzS0GgVN34Q9c2H7ZFj4AhQOgPoDISQM3Apk+3CnT2e8/cwZyBebyMIBDfDxtJ+lF+3Vy43LsfPkNT5ZdYRafoWo4591qVyRO2T2N5LZ9odRrlw5QkJCAKhTpw7Hjx+/pzRzr1696N69+z3PKViwIO7u7vTt25cOHTrQsWNHwCjXfPfyjzdu3CA+Ph4vL6+cB2oHHKvZ50HcCkD9ATBkDzwzBzyLwao3YWwgrPsAbmRvzU8/v4y3OxdM4J321ajtV9hyMedhSik+616TUoXyM/j7PVy/dcfskISFZPY3ktn2h+Hm9vfIOWdnZ2JjY7N8jouLCzt37uTpp59myZIltG3bFoC0tDS2bdtGZGQkkZGRnD17Ns8kfpDk/09OzhDYBfr+DC//DOWbw9Zx8FUw/O8VOL/vgU8fPRrS13X4i3JJoWnPC/RpFGCtqPMk7/yuTA6rzZWbSfz7f/ul/T+PyOhvxMPD2G5p3t7eFC5cmC1btgDw3Xff/fUp4E83b94kLi6O9u3b89VXXxEZGQkY1TYnTpz4135/bs8rpNnnQco+BmXnwPVTsH2qMWls33wo1xQaDIaKrYxO5LuEhRnfR46E06c1boWSKPnEURZ9UcWhSjdYSlBpb4a3rsLHq47wQ0QMz9Qta3ZIIofu/Rsx7vhHj/57u6XNnj2bAQMGcPv2bcqXL8/MmTPv+X18fDxdunQhMTERrTVjx44FYPz48QwaNIgaNWqQkpJC06ZNmTp1qnWCNIGUdH4YCbGwezbs+BpunAXfytBgENR41hhOep/3lx5g9rZovnv5MZpUkpWrHlVamibsmx3sjYll5ZAmBPh6Zv0kYVNS0tkcUtLZVvIXgkZDYeheeOobI+H/NBTGBsHGj+Hm3+vTbjxyidnbonmpUTlJ/Dnk5KT44pmauDgpXl8QSXJqmtkhCZHrSfJ/FM6uUKO7MVeg9wooUxd++QTGVodlr3H91D5GLNpL1RJevNm2itnR5gmlCuXno6eCiTwTy4QN9lusT4jcQtr8c0IpCGhsfF05CtsnoyO/p/DuOXyRFkL5xm/j7iLvr5bSsUYpNhy5xMQNR2lRpSi1ZOSUXdGZLGYurCOnTfaSmSzFtxJ0HMua1hv4PLk7dd3PUHZ5D5jaBCLnQYoMVbSE/3auTvGC7ry5aB9JKalmhyPSubu7c/XqVRmRZSNaa65evYq7u/sjH0M6fC3oys0kWn35C35FPFnctxYuh/4H2ybBpUNQoATU6w91+oCHTFjKiU1Rl+g983cGtajAiDZVzQ5HAMnJycTExJCYmGh2KA7D3d2dMmXK4Orqes/27Hb4SvK3oIHhu1h36BIrhjSmUvH0ySBaw/H1xpvA8Q3g6mHMGq7/qrH2gHgkw3/Yy497zrJkYCNZAF6Iu8hoHxtbse88K/dfYGjLSn8nfjD6BSq2hBd+hFd/g+pdYdcsmFAH5odB9Da7WmQmt3i3QyBFPPMxYtFe7qTI6B8hHpYkfwu4ejOJ95YeILi0N680LZ/5jsWrw5OT4Y0D0ORfEL0VZraFb56AA4shVdawzS5vD1c+6hrMkQvxTNwoo3+EeFiS/C3gvz8d4kZiMp93r4mLczZeUq8S8MS78MYh6PCFMXls0UtGaenfJkLiDesHnQe0DCzOkyGlmLzxmCz+IsRDkuSfQ7/8cZlle88xqEVFqpR4yKJP+Tygbl8YHAE95kEhP1g70pgvsGYkxJ6xTtB5yPudqlPIw5V3/refVFn8XYhss0jyV0q1VUpFKaWOKaXezuD3vZVSl5VSkelffS1xXrMl3EnlP0v2U76oJ682z0HnrZMTVG0PfVZCv41QqTVsnwLjahqfCM7uslzQeUxhz3z8p0Mge2PiCN9h2RXZhMjLcpz8lVLOwCSgHRAI9FRKBWaw6wKtdUj61zc5Pa89mLDhKGeuJTD6yWDcXJwtc9DStaHbt/D6PmgwEI7+DNMfhxnt4PBySJOx7ffrElKKxhV9+Wx1FBdvyFBDIbLDEnf+jwHHtNYntNZ3gPlAFwsc165FXYhn2uYTdKtThgYVilj+BN5loPWHxuLzbT6GuBhYEAYTQ2HndGMxegEYtf8/fDKIpNQ0Rv10KOsnCCEskvxLA3c3Tsekb7vf00qpfUqpRUqpXF2XNy1NM/LH/Xi5u/BOeytXMnQvaHwCGLIHus+C/D6wcrjRL7B+FMRfsO75c4kAX09ea1GRFfvPszHqktnhCGH3LJH8MyrmcX/P209AgNa6BrAOmJ3hgZTqr5SKUEpFXL58OaNd7MKCiDNERF/nnfbVbLcko7OLMUeg7zp4aa1RT2jLl0ZF0R9fhQsHbBOHHevfrDwVixXg3SUHSLgjzWNCPIglkn8McPedfBngnjUPtdZXtdZJ6Q+nA3UyOpDWeprWOlRrHVq0qH2WQb5yM4mPVx6mXjkfutUpY/sAlAK/evDsXBiyG0JfgkNLYWojmNMFjq5z2Eljbi7OjH4yiJjrCYxbf9TscISwa5ZI/r8DlZRS5ZRS+YAewLK7d1BKlbzrYWfgsAXOa4oxq46QkJzK6K7B5lcw9CkP7T+FYQeh5QdwOQrCn4bJ9WH3HEh2vM7PeuWL0L1OGb7ZcoJjl+LNDkcIu5Xj5K+1TgEGA2swkvpCrfVBpdQopVTn9N2GKKUOKqX2AkOA3jk9rxl2n77OD7tieKlxOSoWK2B2OH/LXxgavwFD90HXacZ6A8teg6+CYNMYuHXF7Aht6u12VfHI58wHyw5JlUkhMiGF3bIpLU3z5OStXIhLZMPw5hRws+OlELSGk5th20Q4uhZc3KFmD6g/CIpWNjs6m5j92yneX3aQqc/Xpm1QyayfIEQeIYXdLGxhxBn2xcQxskM1+078YPQLlG8GYT/AoJ3GGsOR82BSXfj+WeONwU7f9C0lrJ4fVUt48X/LD0vnrxAZkOSfDXG3k/l0TRSPBfjQuWYps8N5OEWrQOfxxnyB5v+GmAiY3Qm+bgp7F0BqstkRWoWLsxMfdK7O2dgEpv5y3OxwhLA7kvyz4cufo4i9fYcPOlc3v5P3URUoCs3fNt4EOo2HlCT4sT98VQN+/QoSrpsdocXVL1+ETjVLMfWX45y5dtvscISwK5L8s3D4/A2+2x5NWD1/AksVNDucnHN1hzq9YOB2CFtkLD+57n34sjqseguunTQ7Qot6p31VnJTi/5bLzF8h7ibJ/wG01ry/7CDe+V35V+s81lHq5ASVWkGvZTDgVwjsDL9/CxNqw4IX4PQOsyO0iJLe+XntiYqsPXSRd7+4QUCAcekBARAebnZ0QphHkv8DLNt7jp0nrzGiTVUKedhoJq8ZSgRD16nw+n5o9LrRITyjNXzTEg4uyfWLzLzcuBwep8vx0b89iY42+rqjo6F/f3kDEI5LhnpmIuFOKo9/sYkiBfKxdFBjnJ1yaVv/o7hzCyK/N9Ydvn7SWGeg/kCo9Ty4PeSaBXaieOlULp37Z+VVf384dcr28QhhLTLUM4embznB+bhE3utY3bESP0A+T3isH7y2C54Nh4KlYfXbRr/A2neNCqO5zOXzGf9XP33axoEIYSck+Wfg4o1Epmw6TrugEjxWzsfscMzj5AzVOsJLq6HvBqj4hPFpYFxNWNwXzkWaHWG2+fll/Abu52fjQISwE5L8M/DF2ihS0tJ4u11Vs0OxH2XqQPeZMDQS6g2AqNUwrRnM7ABRqyAtzewIH2j0aPDwuHebh4exXQhHJMn/PgfPxfHDrhh6NwzAv4in2eHYn0J+0Ga0UUyu9WiIjYZ5PYzZw79/C3fsczx9WBhMmwZlympA41kkiWnTjO1COCJJ/nfRWjN6xWEK5Xdl8OOVzA7Hvrl7Q8PBMCQSnv7W6AheMcxYZGbDhxB/0ewI/yEsDM6cVkzeeALfvusIqOdYBe+EuJsk/7usP3yJ345f5fWWlfHO72p2OLmDswsEdzMWnu+zCvwawObPjYqiSwfBRfubXNWnUQBlCufn/5YfIjXNPke7CWFtkvzTJaem8dHKw1Qo6slz9aQX8KEpBf4Noef3xiih2i/C/sUwpQF89xQcW283xeTcXZ35d7tqHLkQzw8RZ7J+ghB5kCT/dHO3R3Piyi1GdqiGq7O8LDlSpAJ0+AKGHYLH34WLB2DuUzClIeyZa9QVMln74BKE+hfm87V/cDMpd09iE+JRSJYDYm/f4at1R2lc0ZcWVYqZHU7e4eEDTYcbM4efnAIooynoq2DY/BncvmZaaEop/tMxkCs3k5iy6ZhpcQhhFkn+wIQNx7iRmMzIDtVyb9VOe+biBiHPwatb4YUlRjmJDR/Cl4GwfBhcMSf5hpQtxJMhpZi+5STnYhNMiUEIszh88j915RZztp3i2dCyVCuZB6p22jOloEILeH6xUVU0uBvs+Q4mhsK8nnDqV5v3CwxvUwU0fPnzHzY9rxBmc/jk//naKFycnBjWKo9V7bR3xapBl4nG+gLN3oQzO2BWB5jWHPYvstkiM2UKe9C7UQCLd8dw+PwNm5xTCHvg0Ml/75lYlu87T78m5ShW0N3scBxTgWLQ4h3jTaDjV0ZRucUvw7gQ2DoeEuOsHsKg5hUp6O7KmNVHrH4uIeyFwyZ/rTWfrDqCj2c++jUtb3Y4wjU/hPYx1hx+biH4lIOf3zX6BVb/G65HW+3U3h6uDGpRgU1Rl/ntmEz8Eo7BYZP/L39cZtuJqwx5vCJe7jKhy244OUHlNtB7OfT/Bap2gJ3TYHwILOxlrEFsBS82CKB0ofx8vOoIaTLxSzgAh0z+aWnGXb+fjwfP1fM3OxyRmVIh8NQ0GLoPGg6BExvhmyfg29ZwaBmkpVrsVO6uzvyrdWX2n43jp33nLHZcIeyVQyb/JZFnOXIhnuFtqpDPxSFfgtzFuzS0+i+8cQjafQrxF2DhC8aSkzu+hqSbFjnNkyGlqVayIJ+vjSIpxXJvLELYI4fLfInJqXyx9g+CS3vTMbik2eGIh+FWAOq9AkP2wDNzwLMYrHoTxgbCug/gRs7u2J2cFP9uV5Uz1xKYu11WeRF5m8Ml/7nbozkbm8Db7ari5GgrdOUVTs4Q2AX6/gwv/wzlm8PWccbM4f+9Auf3PfKhm1YuSpNKvkzccJS4BNsMNxXCDA6V/OMSkpm48RhNKvnSqKKv2eEISyj7mPEpYMgeqNsPjiyHr5vA7E7wx5pHWmTmrbZVuX47mam/HLdCwELYB4dK/lN/OU7s7WRZoSsvKhwA7T4x5gu0GgVXj8P3z8DkerBrFiRnv3xDUGlvutYqzYxfT3I+Tso+iLzJYZL/hbhEZvx6kidDSlG9lLfZ4QhryV8IGg2FoXvhqW+M+QM/DYWxQbDxY7h5OVuHGdaqMlrDl2ul7IPImxwm+Y/9+Q+0hn+1rmJ2KMIWnF2hRndjrkDvFVCmLvzyibHS2LLX4NKDZ/OW9fGgV0N/Fu2O4cgFKfsg8h6HSP5HL8bzw64zPF/fn7I+Hlk/QeQdSkFAY3huPgyOgFphsG+h0Rw0txuc2JRpMblBLSri5ebCmFVS9kHkPQ6R/MesjsIznwuDH69odijCTL6VoONYY75Ai//A+b0wpwtMbQKR8yDlzj27F/LIx8AWFdkYdZltx6+aFLQQ1pHnk//vp66x7vBFBjSvgI9nPrPDEfbAswg0G2EsMtNlEuhUWDLAGCq65Yt7Fpnp3TCAkt7ufLLqMNpOlqEUwhLydPLXWvPxysMU83KjT6MAs8MR9sbVHWo9D6/+ZqwxUDwQ1o8y+gVWDIerx3F3dWZYq8rsjYlj5f4LZkcshMUuVewsAAAfyElEQVTkueQfHg4BAUZ9sBKlU9m8yoM3WlXGI5+L2aEJe6UUVGwJL/xovBFUfwp2z4YJdWB+GE/5nqFKsQJ8tuYIyakPP29ACHuUp5J/eDj07w/R0UYf3qXzLlxfU4OUqDJmhyZyi+LV4clJ8PoBY/3h6K04z2rHDy4jCbq+ngXbT5gdoRAWoSzRjqmUaguMA5yBb7TWn9z3ezdgDlAHuAo8q7U+9aBjhoaG6oiIhyvfGxBgJP77+fvDqQeeTYhM3LkNe79Hb5uMunac8/ji88RQ3Or2BndZ9lPYH6XULq11aFb75fjOXynlDEwC2gGBQE+lVOB9u70MXNdaVwTGAmNyet6MnM6kFldm24XIUj4PqNsXNTiC4y2nczrNF7f17xr9AmtGQuwZsyMU4pFYotnnMeCY1vqE1voOMB/oct8+XYDZ6T8vAp5QSlm8qpqf38NtFyLbnJyo0PgZZleZTPe0j0gs1xK2T4FxNWHRS3B2l9kRijziZlKKTUaWWSL5lwbuvv2JSd+W4T5a6xQgDihigXPfY/RoyJ//3hfNw8PYLoQljGhTlT0p5Rjt/i94fR80GAhHf4bpj8OMdnB4uUUXmRGO5+3F+3hu+g6rvwFYIvlndAd/f9TZ2QelVH+lVIRSKuLy5ezVYLlbWBiMm5SKd9E7KKXx94dp04ztQlhCOV9Pej7mx7ydpzmZXBhaf2gUk2vzMcTFwIIwmBgKO6cbi9EL8RD2xcSyfN95QgMKY4XGkXtYIvnHAGXvelwGuH9Vjb/2UUq5AN7Atfv2QWs9TWsdqrUOLVq06CMF06+PC7GX8pGWpjh1ShK/sLwhT1Qin4sTn61JL/vgXtD4BDBkD3SfBfl9YOVwo19g/Shj5TEhsqC1sbxsYQ9X+jctb/XzWSL5/w5UUkqVU0rlA3oAy+7bZxnQK/3nbsAGLdMlRS5V1MuNfk3Ks3L/Bfacvv73L5xdoHpX6LsOXlpr1BTa8qVRUfTHV+HCAfOCFnZvy9ErrF3mxpnJLfD2cCUgwBi+bi05Tv7pbfiDgTXAYWCh1vqgUmqUUqpz+m7fAkWUUseAYcDbOT2vEGbq17Q8vgXy8cmqI/9sm1UK/OrBs3NhyG4IfQkOLYWpjYxaQkfXZVpMTjimtDTNa/93letranD1oitaG8PW+/e33huARcb5W8OjjPMXwpbmbDvFe0sPMqN3KI9XLf7gnROuG4vK7Pga4s9D0arQYBAEP2OUmRAObWnkWZ5uVpjUG/+sOvyw85RsNs5fCEfV8zE/Aop4MGZVFKlpWdxE5S8Mjd+Aofug6zRjvYFlr8FXQbBpDNy6Ypughd1JSknlszVRpN7In+HvrTVPSZK/EI/I1dmJEW2qEnUxnv/tjsnek1zyQc1n4ZUt8OIyKFUbNn1kdA7/NBQuy8phjub7HaeJuZ5A8VIZ142y1jwlSf5C5ED74BLULOPNlz//QWLyQ4zvVwrKN4OwhTBoJ9TsAXvnw6S6EP4MnNws/QIOID4xmQkbjtGwQhG++NQJj/tafaw5T0mSvxA5oJTi7XbVOB+XyOzfTj3aQYpWgU7jjPkCzd8xZgvP7gRfN4W9CyA12aIxC/sxffMJrt26w1ttqxIWppg2zWjjVwqrz1OSDl8hLKDPzJ3sir7O5jdbUMgjh4sGJSfCvgWwbRJciQKvUlDvFajTy+g7EHnCpfhEmn26icerFWPSc7Utdlzp8BXCht5qV5X4pBQmbzqe84O5uhuJfuB2CFtkLD+57n34sjqseguuncz5OYTpxq8/SnJqGsNbVzHl/JL8hbCAqiUK8lStMsz67RRnYxMsc1AnJ6jUCnotgwG/QmBn+P1bmFAbFrwAp3dY5jzC5k5eucW8nWfo+Zgf5Xw9TYlBkr8QFjKsdWUAvlxrhRE7JYKh61Rj3eFGrxsdwjNawzct4eASSE2x/DmF1Xy+Ngo3Fydee6KiaTFI8hfCQkoXyk+fhgH8b08Mh8/fsM5JCpaElu/DsEPQ/nNjfsAPvWBCLaPEdFK8dc4rLGbvmVhW7DtP3yblKeZl3gQ/Sf5CWNCrzSvg5ebCmNVHrHuifJ7wWD94bRc8Gw4FS8Pqt41+gbXvGhVGhd35s3hbEc989GtSztRYJPkLYUGFPPIxqEVFNkVd5rfjNpi16+QM1TrCS6uh7wao+IQxSmhcTVjcF85FWj8GkW2bj15h24mrvPZ4RbzcXU2NRZK/EBbWq2EApbzdGZNR0TdrKlMHus+EoZFQbwBErYZpzWBmB4haBWkZzyAVtpGWZtz1l/XJz3P1/M0OR5K/EJbm7urMsNZV2BsTx4r9520fQCE/aDMahh2E1qMhNhrm9TBmD//+rbEovbC5JZFnOXz+BsNbVyGfi/mp1/wIhMiDutYqTdUSXny2JorkVJPuuN29oeFgGBIJT38Lbl6wYphRR2jDhxB/0Zy4HFDCHaN4W40y3nSqUcrscABJ/kJYhbOT4q22VYm+ept5O61UljHbwbhAcDfotxH6rAL/hrD5c6Oi6JJBcPGQufE5gBlbT3I+LpF32lfDycm6yzNmlyR/IaykeZWi1C/vw7h1R7mZZAfj8JUyEn+PcGOUUO1ecPB/MKUBfNcVjq2XYnJWcDk+ickbj9EqsDj1yxcxO5y/SPIXwkr+LPp29dYdpm8+YXY49ypSATp8bhSTe/xduHgQ5j4FUxrCnrmQkmR2hHnGuPV/kJiSxtvtqpodyj0k+QthRSFlC9EhuCTTt5zgUnyi2eH8k4cPNB1uzBx+cgqgYOkg+CoYNn8Gt6+ZHWGuduxSPPN2niGsnh8VihYwO5x7SPIXwspGtKnCnZQ0xq8/anYomXNxg5Dn4NWt8MISo5zEhg/hy0BYPgyuHDM7wlzpk1VHyO/qzNAnKpkdyj9I8hfCygJ8PXmunh/zdp7hxOWbZofzYEpBhRbw/GKjqmhwN9jzHUwMhXk94dSv0i+QTb8dv8K6w5cY2KICRQq4mR3OP0jyF8IGXnu8Eu4uTny+NsrsULKvWDXoMtHoF2j2JpzZAbM6wLTmsH+RLDLzAGlpmo9WHqaUtzsvNTK3jENmJPkLYQNFvdzo17Q8K/dfIOJULmtHL1AMWrxjvAl0/Aru3ILFL8O4ENg6HhLjzI7Q7izde5YDZ28wom0V3F2dzQ4nQ5L8hbCR/k3LU6KgO6OWHyItLRc2nbjmh9A+xprDzy0En3Lw87tGv8Dqf8P1aLMjtAuJyal8tjqK4NLedKlZ2uxwMiXJXwgb8cjnwlvtqrAvJo4f95w1O5xH5+QEldtA7+XQ/xeo2gF2ToPxIbCwF8Q49vKr3/56knN2NqErI5L8hbChLjVLU7NsIcasPsIte5j4lVOlQuCpaTB0HzQcAic2wjdPwLet4dAySEs1O0KbungjkUnpE7oaVLCfCV0ZkeQvhA05OSne7xTIpfgkpv5igfV+7YV3aWj1X3jjELT7FOIvwMIXjCUnd3wNSXY+yslCPl0dRUqqZmT7amaHkiVJ/kLYWG2/wnQJKcW0zSeIuZ7HKmy6FYB6r8CQPfDMHChQHFa9CWMD4ef34cY5syO0msgzsSzeHcNLjcsRYNK6vA9Dkr8QJnirbVWUMiYB5UlOzhDYBV5eCy+vg/It4Lfxxszh//WH8/vMjtCi0tI0Hyw7SFEvNwY/bt66vA9Dkr8QJihVKD+vNK3A8n3n+T23Df18WGXrwjOzjU8DdfvBkRXwdROY3Qn+WJMnFplZuvcskWdiebNNFQq4uZgdTrZI8hfCJAOaVaCktzujfsqlQz8fVuEAaPeJMV+g1Si4ehy+fwYm14NdsyA5wewIH8mtpBQ+WXWEGmW8ebp2GbPDyTZJ/kKYJH8+Z95qW5X9Z+NYvNuBFlzPXwgaDYWhe+Gpb4z5Az8NhbFBsPFjuHnZ7AgfytRfjnPxRhLvd6pu10M77yfJXwgbCw+HgABjuPzQJ0vhe6EiY1ZHcSPRwcolOLtCje7GXIHeK6BMXfjlE2OlsWWvwSX77w85c+02X28+QZeQUtTxL2x2OA9Fkr8QNhQeDv37Q3S0UR/t9GnFwYWVid5ZhK9+tuOqn9akFAQ0hufmw+AIqBUG+xYazUFzu8GJTXZbTO7jVYdxVsruavVnhyR/IWxo5Ei4fd/ozsQExZ3t1Zm97RRHLtwwJS674VsJOo415gu0+A+c3wtzusDUJhA5D1LumB3hX349eoWV+y/wavMKlPTOb3Y4D02SvxA2dDqT5Xzjr7hS0N2F95YcRNvpXa5NeRaBZiOMRWa6TAKdCksGGENFt3xh+iIzSSmpvLf0AP5FPOjftLypsTwqSf5C2JCfX2bbjaaDnaeusSQyF9f9sTRXd6j1PLz6m7HGQPFAWD/K6BdYMdwYMWSCb7ac5MSVW/y3c3W7rdqZlRwlf6WUj1LqZ6XU0fTvGfZ4KKVSlVKR6V/LcnJOIXKz0aPBw+PebR4exvbudcoSUrYQo1cccbzO36woBRVbwgs/Gm8E1Z+C3bNhQh2YHwbR22zWL3Dm2m0mbDhKu6ASNK9SzCbntIac3vm/DazXWlcC1qc/zkiC1jok/atzDs8pRK4VFgbTpoG/v5HP/P2Nx2FhRt2fD58M4uqtJMb+/IfZodqv4tXhyUnw+gFj/eHorTCzLUx/HA4shlTrFsz770+HcFKKdzsGWvU81qZy0r6olIoCmmutzyulSgKbtNZVMtjvptb6oVYvDg0N1RERjl0aVjimd5ccIHxHNMtfa0JgqYJmh2P/7tyGvd/Dtslw7Th4l4V6A6D2i+Bu2ddv3aGL9J0Twb/bVeWVZhUsemxLUUrt0lqHZrVfTu/8i2utzwOkf8/sM5C7UipCKbVdKfVkDs8pRJ42vHUVCnnk472lBxxj5m9O5fOAun2NYaI95kEhf1g70lhkZs1IiD1jkdMk3Enlg58OUqlYAV5qbJ9LMz6MLJO/UmqdUupABl9dHuI8funvRM8BXymlMnzLVEr1T3+TiLh8OXfN8hPCUrw9XHm7XVUioq+zIMIyicshODlB1fbQZwX02whV2sL2KTCuJvzQB87uytHhJ286Rsz1BP7vySBcnXP/WBmbNPvc95xZwHKt9aIH7SfNPsKRaa15bvoODpyLY/2wZhQr6G52SLlTXAzsmAq7ZkPSDfBrCA0GQZV2RuXRbPrjYjwdxm+hY41SjH02xIoB55ytmn2WAb3Sf+4FLM0gkMJKKbf0n32BRsChHJ5XiDxNKcVHTwWTlJLG+8sOmh1O7uVdBlp/aBSTa/Ox8WawIAwmhsLO6cZi9FlITdO8tXgfBdxc+E8H+1+kJbtymvw/AVoppY4CrdIfo5QKVUp9k75PNSBCKbUX2Ah8orWW5C9EFsr5ejL0iUqsOnCBtQcvmB1O7uZeEBoMNMpKd58F+X1g5XBjvsD6UcbKY5mYuz2aPadjea9TIEUKuNkuZivLUbOPNUmzjxCQnJpGpwm/Ens7mZ+HNcXL3dXskPIGreHMTtg2AQ4vBycXCO5uNAmVCPprt7OxCbT+8hfqBPgwu09dlLL/qp22avYRQliRq7MTnzxdg4vxiXy6OsrscPIOpcCvHjw7F4bshtCX4NBSmNrIqCV0dB06LY3//LgfDXzUNShXJP6HIclfCDsXUrYQvRsGMHdHNBF5fdUvM/iUh/afwrCD0PIDuBwF4U8TPzaUoscW8tYTAZQp7JHVUXIdafYRIhe4lZRC67GbcXNxYsWQJuTPlzvryeQKKXeI372Qsys/oyqn0J5FUXX7Qd2XwdPX7OiyJM0+QuQhnm4ufNatBieu3OLTNfa/yElupp1dGRFVjc7JH3Om0wJUqdqw6SOjc/inoXA5b5TekOQvRC7RsKIvvRsGMHPrKX47fsXscPKsJZFnWX3wAsNaV6FsnbYQthAG7YSaPWDvfJhUF8KfgZOb7XaRmeyQ5C9ELvJW26qU8/VkxA/7uJlk3QJmjuh8XALvLT1IqH9h+jW5q05/0SrQaZwxX6D5O8Zs4dmd4OumsHeBXS0yk12S/IXIRfLnc+bz7jU4H5fA6BUyXcaStNa8uWgfKamaz7vXxDmjxdg9faH5W8abQKfxkJIEP/Y3Skj8OhYSrts+8EckyV+IXKaOvw/9m1Zg3s4zbIy6ZHY4ecbcHafZcvQK73SoRoCv54N3dnWHOr1g4HYIWwRFK8O6D+DL6rDyTbh20iYx54QkfyFyoTdaVaJy8QK8uWgfV28mmR1Ornfi8k0+WnGYJpV8eb5eJsutZcTJCSq1gheXwoBfIbAzRMyACbVhwQtweof1gs4hSf5C5EJuLs6M61GLuIRk/vXDXin9nANJKam8Nm8Pbq5OfNqtxqNP5ioRDF2nGusON3rd6BCe0Rq+aQkHl1h9kZmHJclfiFyqWsmC/KdDNTZFXWbG1odvZggPh4AA4+Y1IMB47IjGrIpix88FOD/1CUoXzp/z16JgSWj5Pgw7BO0/h1tX4IdeMKGWUWI6Kd5SoeeITPISIhfTWvPKd7vYGHWJ/73aiOAy3tl6Xng49O8Pt2//vc3D4+8lJR3FhiMXeWbEOW78XJPkpL/vhS36WqSlQtQq2DYRTm8DN2+jv6DeK0bVUQvL7iQvSf5C5HKxt+/QbtwW3FycWD6kCQXcXLJ8TkAAREf/c7u/P5w6ZfEQ7dKFuETaj9/C4bFNSLz+z/USrPJaxOwy3gQOLTXqC1XvCg0GQynLrREgM3yFcBCFPPIxrkctTl+7zcgf95OdG7rTpx9ue16TkprG0Pl7SLiTSlJsxmWarfJalKkD3WfC0EhjneGo1TCtGczsYHw6SEuzwkkzJslfiDzgsXI+vN6yMksjzzH7t1NZ7u+XyYCWzLbnNWNWH2HHyWuM7hqEn1/GHbxWfS0K+UGb0UYxudajITYa5vUwZg///q2xKL2VSfIXIo8Y3KIiLasV48MVh9l58sHVP0ePNtq17+bhYWzP637ae47pW07Sq4E/T9UuY+5r4e4NDQfDkEh4+ltw84IVw4wRQlZukpfkL0Qe4eSk+PLZEMr6eDAwfDcX4hIz3TcszOjQ9Pc3mp79/R2jszfqQjxvLtpHqH9hRnYIBOzktXB2geBuxsLzfVZB87eNYKxIOnyFyGP+uBjPk5O2UqWEF/P718fNRco/A8QlJPPkpK3cTEphxWuNKVbwn528eYF0+ArhoCoX9+KzbjXZczqWd5ccyFYHcF6XnJrG4O93c+babaaE1c6zif9hZD0mTAiR63SoUZKoCxUZv+EY/kU8GdSiotkhmUZrzfvLDrLl6BU+fboGoQE+ZodkFyT5C5FHvdGqMtHXbvPZmij8fDzoVLOU2SGZYvqWE3y/4zQDm1fgmbplzQ7HbkjyFyKPUkrxabcanItN4F8/7KVUIXfq+DvWXe/qA+f5eNUROgSXZHjrKmaHY1ekzV+IPMzNxZlpL4RSulB+Xp4dQdQF+6grYwu/Hb/CkPmRhJQtxBfP1MQpo/r8DkySvxB5XGHPfMzu8xhuLk48/+0OTl65ZXZIVhd5JpZ+syPw9/FgRq+6uLvKiKf7SfIXwgH4FfFg7sv1SE3TPP/NDs7FJpgdktVEXYin98ydFCngxty+9Sjsmc/skOySJH8hHESl4l7MeekxbiQk8/w3O7h4I/NJYLnVHxfjCftmB24uToT3rUdxGdKZKUn+QjiQoNLezOxTl4s3Ennm623EXLd+DRlbOXgujh7TtuOkILxvfcr6eGT9JAcmyV8IBxMa4MPcvvW4fusOz0zdlif6APaeieW56Ttwd3Fi4SsNqFisgNkh2T1J/kI4oFp+hZnfvwFJKWl0n7qNA2fjzA7pkW2KusRz07dTML8LC15pkPXi6wKQ5C+EwwosVZAFr9THzcWJ7lO3sfbgBbNDemjf7zjNy7Mj8C/iyQ+vNJSmnocgyV8IB1axmBc/DmpI5eIFeGXuLqZvPpEragGlpmk+XnWYd37cT9NKviwc0IAS3tK5+zAk+Qvh4Ip5uTO/fwPaVi/B6JWHGTo/kptJKWaHlakrN5N44dsdfP3LCZ6v78f0F0OztXSluJe8YkII8udzZtJztZm08Rhj1/3BgbNxTAqrTbWSBc0O7R47TlxlyPw9xN5O5tNuNXgmVGr1PCq58xdCAMZiMK89UYnv+9XnZlIKXSZtZdLGYySn2m5d2cwk3Ell1E+H6DF9O/ldnflxYCNJ/DkkyV8IcY/65YuwcmgTWlYrxmdronhy0lZTRwNtPXaF9uO3MGPrSV6o78+KIU0ILGVfn0hyI1nJSwiRqdUHzvOfJQe5eiuJp2qVYXibypT0zm+Tc5+4fJOPVh5m3eFLlPXJz5ina9Cwgq9Nzp2bZXclL2nzF0Jkqm1QSRpU8GXyxmPM3HqK5fvOEVbPn5caB1CmsHWGVR67FM+UTSdYGnkWd1dn3mpblT6NAqQ4m4Xl6M5fKdUd+ACoBjymtc7wVl0p1RYYBzgD32itP8nq2HLnL4R9OXPtNmPX/cGyyHNooF1QCbrVKUPjir64OOesBTkpJZX1hy/xQ8QZNv1xGTcXJ3o+5sfA5hUp6uVmmQtwENm9889p8q8GpAFfA8MzSv5KKWfgD6AVEAP8DvTUWh960LEl+Qthn87FJjBz60kWRsQQl5CMbwE3Hq9alCaVilKvvA/FvLIeb6+1JuZ6Ar+fusbGqMv8EnWJG4kplCjozjN1y9KrgT9FCkjSfxQ2afbRWh9OP9mDdnsMOKa1PpG+73ygC/DA5C+EsE+lCuVnZIdAhrepwsYjl1m29yyrD1xgYUQMAEU881G5uBfFC7pRpIAb7q5OpGm4k5LGpfgkLsYlEnUxnriEZAB8C7jRKrAEnUNK0biiL86y6IpN2KLNvzRw5q7HMUC9jHZUSvUH+gP4+flZPzIhxCNzc3GmbVAJ2gaVICU1jX1n44g8HcuRCzc4eukmu05f50r8HZJT03BSChdnRTEvN4p5udM+uCRBpQtSs0whAksWlFW2TJBl8ldKrQNKZPCrkVrrpdk4R0b/qhm2NWmtpwHTwGj2ycaxhRB2wMXZidp+hantV9jsUEQ2ZZn8tdYtc3iOGODu2RhlgHM5PKYQQogcsMUkr9+BSkqpckqpfEAPYJkNziuEECITOUr+SqmuSqkYoAGwQim1Jn17KaXUSgCtdQowGFgDHAYWaq0P5ixsIYQQOZHT0T4/Aj9msP0c0P6uxyuBlTk5lxBCCMuR2j5CCOGAJPkLIUwTHg4BAeDkZHwPDzc7IschtX2EEKYID4f+/eH2beNxdLTxGCAszLy4HIXc+QshTDFy5N+J/0+3bxvbhfVJ8hdCmOL06YfbLixLkr8QwhSZVXCRyi62IclfCGGK0aPB474lATw8jO3C+iT5CyFMERYG06aBvz8oZXyfNk06e21FRvsIIUwTFibJ3ixy5y+EEA5Ikr8QQjggSf5CCOGAJPkLIYQDkuQvhBAOSGltn6slKqUuA9E5OIQvcMVC4eQWjnbNjna9INfsKHJyzf5a66JZ7WS3yT+nlFIRWutQs+OwJUe7Zke7XpBrdhS2uGZp9hFCCAckyV8IIRxQXk7+08wOwASOds2Odr0g1+worH7NebbNXwghROby8p2/EEKITOTq5K+UaquUilJKHVNKvZ3B792UUgvSf79DKRVg+ygtKxvXPEwpdUgptU8ptV4p5W9GnJaU1TXftV83pZRWSuX6kSHZuWal1DPp/9YHlVLf2zpGS8vG/20/pdRGpdSe9P/f7c2I01KUUjOUUpeUUgcy+b1SSo1Pfz32KaVqWzQArXWu/AKcgeNAeSAfsBcIvG+fgcDU9J97AAvMjtsG19wC8Ej/+VVHuOb0/byAzcB2INTsuG3w71wJ2AMUTn9czOy4bXDN04BX038OBE6ZHXcOr7kpUBs4kMnv2wOrAAXUB3ZY8vy5+c7/MeCY1vqE1voOMB/oct8+XYDZ6T8vAp5QSikbxmhpWV6z1nqj1vrPlVG3A2VsHKOlZeffGeD/gE+BRFsGZyXZueZ+wCSt9XUArfUlG8doadm5Zg0UTP/ZGzhnw/gsTmu9Gbj2gF26AHO0YTtQSClV0lLnz83JvzRw5q7HMenbMtxHa50CxAFFbBKddWTnmu/2MsadQ26W5TUrpWoBZbXWy20ZmBVl59+5MlBZKbVVKbVdKdXWZtFZR3au+QPgeaVUDLASeM02oZnmYf/eH0puXswlozv4+4cuZWef3CTb16OUeh4IBZpZNSLre+A1K6WcgLFAb1sFZAPZ+Xd2wWj6aY7x6W6LUipIax1r5disJTvX3BOYpbX+QinVAPgu/ZrTrB+eKayav3LznX8MUPaux2X458fAv/ZRSrlgfFR80Mcse5eda0Yp1RIYCXTWWifZKDZryeqavYAgYJNS6hRG2+iyXN7pm93/20u11sla65NAFMabQW6VnWt+GVgIoLXeBrhj1MDJq7L19/6ocnPy/x2opJQqp5TKh9Ghu+y+fZYBvdJ/7gZs0Ok9KblUltec3gTyNUbiz+3twJDFNWut47TWvlrrAK11AEY/R2etdYQ54VpEdv5vL8Ho3Ecp5YvRDHTCplFaVnau+TTwBIBSqhpG8r9s0yhtaxnwYvqon/pAnNb6vKUOnmubfbTWKUqpwcAajJECM7TWB5VSo4AIrfUy4FuMj4bHMO74e5gXcc5l85o/AwoAP6T3bZ/WWnc2LegcyuY15ynZvOY1QGul1CEgFRihtb5qXtQ5k81r/hcwXSn1BkbzR+/cfDOnlJqH0Wznm96P8T7gCqC1norRr9EeOAbcBvpY9Py5+LUTQgjxiHJzs48QQohHJMlfCCEckCR/IYRwQJL8hRDCAUnyF0IIByTJXwghHJAkfyGEcECS/IUQwgH9P2LOGTBEfrbhAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# M=1\n", "p_lsq_1 = fitting(M=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### M=3 " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting Parameters: [ 2.21147559e+01 -3.34560175e+01 1.13639167e+01 -2.82318048e-02]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XlcVFUbwPHfYRMQRBF3BNxFxV1zKzX3vVxKo9I226y3zbQsWy3N3reyzLRMLakstTRzS1s0d3DfRQXEXTbZ1/P+cdFEQVFm5gLzfD+f+cice+be5w74zJ1zz6K01gghhLAvDmYHIIQQwvYk+QshhB2S5C+EEHZIkr8QQtghSf5CCGGHJPkLIYQdkuQvhBB2SJK/EELYIUn+Qghhh5zMDqAgPj4+OiAgwOwwhBCiRAkLC7ugta50o3rFNvkHBAQQGhpqdhhCCFGiKKUiC1NPmn2EEMIOSfIXQgg7JMlfCCHsULFt8xdClByZmZlER0eTlpZmdih2w9XVFV9fX5ydnW/p9ZL8hRBFFh0djaenJwEBASilzA6n1NNaExMTQ3R0NLVq1bqlfUizjxCiyNLS0qhYsaIkfhtRSlGxYsUifdOS5C+sLiQEAgLAwcH4NyTE7IiENUjit62ivt/S7FNaZKTAuQNwMRqSzkFafO4GBa5e4FEZvHyhUkNwKWuzsEJCYPRoSEkxnkdGGs8BgoNtFoYQ4iqS/EuqjGQ49jeEr4GI9RATDjonT5WQPUOZsPYNohJ88fOKZlK3twgOWgTetcG/PdTqDHXuhLI+VgtzwoR/E/8lKSlGuSR/UZyMGjWK/v37M3ToULNDsQlJ/iWJ1hC1CXbMh32/QGYyOJeFgE7QeDBUbQIVAsCjCiE/ezN6qiMpKcZXw8gEP0YvnwmBgwiu9AMc+NXYj3KE2l0gaCgEDoQyHhYNOSrq5sqFsAStNVprHBykZbsgkvxLAq3h8EpYNxVOhoGLBwQNgSZDwK8DOLlc85IJr+dzxZ3mxIT5AwmOGAg52XBmt/EhsGch/PIkrBgPLR+AtqOhgr9FQvfzM5p68isXwpIiIiLo06cPXbt2ZdOmTTz33HN88cUXpKenU6dOHebMmYOHhwdvv/02v/76K6mpqXTo0IGZM2fa5f0KpbU2O4Z8tW7dWsvcPkB0GKwYayT98n7Q6Xloeu8N2+0dHIzPjKspBTk5VxVqDSe2wtaZxjcKNDS/DzqPM45ZBFe3+QO4u8OsWdLsU5ocOHCAwMBAAN76dR/7T1206P4bVS/HGwMaX7dOREQEtWvXZuPGjdStW5fBgwezYsUKypYty5QpU0hPT2fixInExsbi7e0NwAMPPMA999zDgAEDSmSzz5Xv+yVKqTCtdesbvVau/Iur1DhY/ZrRNONRFQZNN5K+Y+EGdNzUFbdS4Heb8ejxNmyaDttmw+4fofXD0OUVcCt/S6dxKcFPmGA09fj5waRJkviFdfj7+9OuXTuWLVvG/v376dixIwAZGRm0b98egD///JMPPviAlJQUYmNjady4MQMGDDAzbFNYJPkrpb4G+gPntNZN8tmugE+AvkAKMEprvd0Sxy6Vjv0FvzwFSWehw7PQ+WUo43lTu5g0Kf8r7kmTbvBCL1/o/T60fxr+ngJbZ8HeRdDzXePD5xa+HgcHS7K3Jze6QremsmWNb8Raa3r06MH333+fZ3taWhpPPfUUoaGh1KxZkzfffNNuRyVb6m7IXKD3dbb3AerlPkYDMyx03NIlJxvWvAnfDAJnd3jkd+j5zk0nfjCS7axZ4O9v5Gt//5tsavHyhYGfwmN/Gk0/Pz8Oc/tDXMRNxyKErbVr144NGzYQHh4OQEpKCocPH76c6H18fEhKSmLhwoVmhmkqi1z5a63XKaUCrlNlEPCNNm4wbFZKlVdKVdNan7bE8UuF1DhY+AgcXQutRkGv98HFvUi7zO+KOy0zm0NnEomOS+V0QioXUzPJzNHkaE05V2cquLtQo4Ib9at4ULWcK6p6c3hkDWyfB79PhBkdoc8UaB58S98ChLCFSpUqMXfuXEaMGEF6ejoA7777LvXr1+exxx4jKCiIgIAA2rRpY3Kk5rHYDd/c5L+sgGafZcBkrfU/uc/XAuO01qFX1RuN8c0APz+/VpH5NVqXRrHHYf5giD8B/T40kr+FZGbnsPV4LGsPnGPzsRgOn00kKyfv79zZUaFQZGTnvRPs5ebMbbW86VjXh26BlfFVF+DnJyHyH2jY37gPcYv3AkTpkt+NR2F9JeGGb36XiNd86mitZwGzwOjtY+2gioUze43En50Bo5aBXzuL7PbA6Yv8sDWKX3aeIiE1ExcnB9oEVODxzrUJqlEe/4ruVPdyo5yb0+VubmmZ2cSlZBAZk8KRs4nsOZnAxqMxrN5/ljeW7qNNQAUGN/+MoXV+wfmvd2FWF7h3vjG+QAhRotgq+UcDNa947gucstGxi68TW2H+UKPb5kMroXLDIu1Oa82mozFM++MIm4/F4uLoQK8mVRnQtBqd6vng7nL9X7ersyPVvNyo5uVGu9oVL5dHXEjmtz2n+WXHSV75ZT9T3BszNvBzRkS8jsNX3Y17A02HFSl2IYRt2Sr5LwXGKKV+AG4DEuy+vf/kdpg/xJha4cElRe5PHxYZx3vLDxAWGUdlzzJM6BvI0Fa+VCh77QCwmxXgU5anu9blqS51CI2MY/b647y2PZNZLu/wndcMaix+1Bgw1v0tY4CBEKLYs1RXz++BLoCPUioaeANwBtBafwEsx+jmGY7R1fMhSxy3xDqzF76922gvH/mr0bPmFp1LTGPyioMs3n6SKuXK8M6gxgxrXRNXZ0cLBmxQStEmwJs2Ad6En0viw1WH6LzveSa7hzB04zR0wgnUXV+As6vFjy2EsCxL9fYZcYPtGnjaEscq8WKPwbd3GV05H1x6y4lfa83PO07yxtJ9pGVm82SXOozpWpeyZWzzZa5uZQ++eKAV26Nq8+YSbw6f8ebVfd+THneSMvcvAHdvm8QhhLg18h3dllLjIOQeyMmCkUvB+9ZW4IlNzuDJ+dt54cddNKzqyarn7mBc74Y2S/xXaulXgcVPdaRy75d5Puc5OLmDpBnd4KLc0hGiOJPkbytZGbDgAYiPhOHfgU+9W9rN9qg4+n6ynj8OnuOVPg35YXR7aley7EycN8vJ0YFHb6/N8/8Zx1vl34WLp4j5rBvp54+bGpewL9OmTSMwMJDg4GCWLl3K5MmTAfjll1/Yv3//5Xpz587l1KmbuziJiIigSZPS1atNkr8taA2/PW/Muz/wM/DvcAu70IRsieTemZtwdlL8/HQHHu9cB0eH4jPQyq+iO289O5qFTabjlB7PxRndORex1+ywhJ34/PPPWb58OSEhIQwcOJDx48cDlkn+1pCVlWXq8SX520LYXGOCtjvGQrN7b/rlWdk5vPrzXib8vJf2dXz4dUwnGlf3snycFuDs6MCoYUM52Ot7HLIzcJzbj4O7t5odlijlnnjiCY4dO8bAgQP56KOPmDt3LmPGjGHjxo0sXbqUsWPH0rx5c6ZMmUJoaCjBwcE0b96c1NRUwsLC6Ny5M61ataJXr16cPm10RAwLC6NZs2a0b9+e6dOnF3jsDz74gKCgIJo1a3b5A6dLly5cmpX4woULBAQEAMYHz7BhwxgwYAA9e/bk3nvvZfny5Zf3NWrUKBYtWkR2djZjx46lTZs2NG3alJkzZ1r8PZNZPa3t1A5Y8TLU6QZdXr3pl6dkZDHmux38cfAcT3apw0s9GxSrq/2C3NahC8crLMVzwd1UXDSUdck/ckd7ywxgE8XcivFwZo9l91k1CPpMLnDzF198wcqVK/nzzz/x8fFh7ty5AHTo0IGBAwfmmap5xYoVfPjhh7Ru3ZrMzEyeeeYZlixZQqVKlViwYAETJkzg66+/5qGHHuLTTz+lc+fOjB07Nv9TXbGCX375hS1btuDu7k5sbOwNT2XTpk3s3r0bb29vfv75ZxYsWEDfvn3JyMhg7dq1zJgxg9mzZ+Pl5cW2bdtIT0+nY8eO9OzZk1q1bu0+YX4k+VtTahz8OBLKVoLBX950H/iYpHQenhfKnuh43r2rCfe3s8wCK7ZSK7AF8Q/9iprbj7or72NZ5nf0v0M+AETxcejQIfbu3UuPHj0AyM7Oplq1aiQkJBAfH0/nzp0BY97/FStWXPP6NWvW8NBDD+HubszDdWmdgOvp0aPH5Xp9+vTh2WefJT09nZUrV3LHHXfg5ubG6tWr2b179+WJ5xISEjhy5Igk/xJBa1gyBi6eNEbvlq1449dc4dzFNEZ8uZnouFRm3N+KXo2rWilQ6yrvH0Tqw0tx+rofQWse4Lv0b7mvh3wAlGrXuUIvbrTWNG7cmE2bNuUpj4+PL9TqXlrrfOs5OTmRk7tq0tVTRl+adhrA1dWVLl26sGrVKhYsWMCIESMu7/fTTz+lV69eN31OhSVt/tay8zs4uAzufB1q3tzMgWcvpjF81mbOJKTx7SO3ldjEf4lbzeY4j/qZyk5JtF3/EF+u3GZ2SMKOeHp6kpiYmO/zBg0acP78+cvJPzMzk3379lG+fHm8vLz4559/AAgJCcl33z179uTrr78mJXfhjEvNPgEBAYSFhQHccNro4cOHM2fOHNavX3852ffq1YsZM2aQmZkJwOHDh0lOTr6l8y+IJH9riIuEFePAvyN0eOamXnr2YhojZm3m7MU05j3clra1SsdgKRf/trg88BP+jhdoufFJZv8hvYCEbQwfPpypU6fSokULjh49yqhRo3jiiSdo3rw52dnZLFy4kHHjxtGsWTOaN2/Oxo0bAZgzZw5PP/007du3x83NLd999+7dm4EDB9K6dWuaN2/Ohx9+CMBLL73EjBkz6NChAxcuXLhufD179mTdunV0794dFxdjOpZHH32URo0a0bJlS5o0acLjjz9u8d5BsoavpeVkw7wBcHo3PLnhphZCj0lKZ9jMTZxNMBJ/64DSkfivlL13CWrhSNZmt+Rkz5mMuv3WxjuI4kWmdDZHUaZ0lit/S9s6CyI3GO2eN5H4k9KzeGjuNk7GpTK3lCZ+AMcmg9C9J9PDMQznVeP4cWuU2SEJYZck+VtS/AlY+w7U7WGsdFVI6VnZPPFtGPtOXeTz4Ja0KaWJ/xLHdk+Q3eE5gp3WEr30bf48eM7skISwO5L8LUVrWP4SoKHffwu9xGFOjubFH3fxT/gFpgxpSrfAKtaNs5hw7PEmmU3u4QWnn/j1u8/YdSLe7JCEsCuS/C1l/xI4vBK6vnpTzT1TVx9i2e7TvNKnIUNb3frUziWOUjjfNZ2MGu1432EGH875nsgYy/ZmEEIUTJK/JaRdNHr3VG0Ktz1Z6Jct3h7NjL+OEnybH6PvqG3FAIspJxdc7gvBwbMy/82ZwtjZK0lIzTQ7KiHsgiR/S1g3FZLOwoCPwbFw4+bCImMZv2gPHepU5M2BjQs1oKRUKuuD8/0/UtEpg9eS3uWl7zaTnVM8e6AJUZpI8i+qmKOweYZxg7dGq0K9JDouhce/DaN6eVc+D26Js6Od/xqqNMZx6FcEORxnQMQkpq48aHZEwg5MnDiRNWvWmB2Gaew861jA6tfAqQx0m1io6mmZ2Yz+Joz0rBy+GtmG8u5FX2O3VGjYF9X9DQY6boINH7Nk50mzIxJWFBICAQHGdFcBAcZzW3v77bfp3r277Q9cTEjyL4qjf8Ch5XD7i+D5by+d6/1hT1yyl/2nLzJteAvqVjZ3EZZip+NzZDe6m5edf2Txou/Yf+qi2REJKwgJgdGjITLS6CQXGWk8L+oHQEREBIGBgTz22GM0btyYnj17kpqays6dO2nXrh1Nmzbl7rvvJi4uDjCmT7409cL48eNp1KgRTZs25aWXXgLg/PnzDBkyhDZt2tCmTRs2bNhQtACLG611sXy0atVKF2tZmVp/dpvWHzfVOiP1cvH8+Vq7u2tt/FkbD3d3o3zB1ijtP26Z/nDVQRMDL+bSEnXmJ611zBu+euiUn/TF1AyzIxKFsH///kLX9ffP+//j0sPfv2gxHD9+XDs6OuodO3ZorbUeNmyY/vbbb3VQUJD+66+/tNZav/766/o///mP1lrrkSNH6p9++knHxMTo+vXr65ycHK211nFxcVprrUeMGKHXr1+vtdY6MjJSN2zYsGgBWkF+7zsQqguRY2VWz1u1MwTOH4B7vgVn18vFEyZA7hxPl6WkwNhxOXiO2kuHOhV5rnt9GwdbgpTxwGlECF4zu/BK0vu8tqgmH9/X1n5viJdCUQUM6i6o/GbUqlWL5s2bA9CqVSuOHj2aZ2rmkSNHMmzYsDyvKVeuHK6urjz66KP069eP/v37A8Z0zVeuAHbx4kUSExPx9PQseqDFgDT73IrMNPh7CtRoDYED8mwq6A/49ElFeXdnpo1oUSIWYzFVpfo43v05LR3CaXlgKvO3yBQQpYmf382V34wyZcpc/tnR0ZH4+BsPHnRycmLr1q0MGTKEX375hd69ewOQk5PDpk2b2LlzJzt37uTkyZOlJvGDJP9bEzrbmKe/28RrRvIW9AfsVC6Vz+5riY9HmfwriLwa34VuN4aRTr+za9lM9p5MMDsiYSGTJkHu2ieXubsb5Zbm5eVFhQoVWL9+PQDffvvt5W8BlyQlJZGQkEDfvn35+OOP2blzJ2DMtvnZZ59drnepvLSQ5H+z0hNh/X+hdheo3fmazfn9YSunLILHXCz1c/ZYmurxFpm+7XnH6Ss+mL+U5HRzF7wWlhEcDLNmgb+/ce3k7288Dy78dFg3Zd68eYwdO5amTZuyc+dOJk7M2zMvMTGR/v3707RpUzp37sxHH30EwLRp0wgNDaVp06Y0atSIL774wjoBmkSmdL5Zf02Bv96DR/8A3/z79YeEGG3/UVEax3JptBx8go1f1ZPmnltx8TSZ0zsQnurBd02/5p2hN7cwjrANmdLZHDKls62kxMLGT6Fh/wITPxhXMEeO5jDw0400fG4di/9XUxL/rSpXDechMwl0iKLuzims3nfG7IiEKBUk+d+MzTMgIxHufO2GVT/9I5ydJ+J57+4gapTPfxUgUUj1e5Kd2/6/atFXnEtMu/FrhBDXJcm/sNISYMtMo3dP5et/vQ2NiOWzP44wuGUNBjSrbqMASzfH7m+QVrkZE7M/Z8oPv1NcmyvtmfxObKuo77ck/8La9hWkJ8DtL123WmpGNi/9tIvq5d14a2BjGwVnB5xccB0+DzdnxX0n3mL+xnCzIxJXcHV1JSYmRj4AbERrTUxMDK6urjeuXAAZ5FUYGcmwabqxQlf15tetOnXVISJiUvjusdvwdHW2UYB2wrsWzoOm0WrRw2xbNYmohp/jV9H9xq8TVufr60t0dDTnz583OxS74erqiq/vra8BIsm/MMLmQUoM3DH2utW2RcQyZ+NxHmjnT4c6PjYKzr6ooCEkH1zDY3u/553vOjHx6UdwkJvppnN2dqZWrVpmhyFugjT73EhWOmycBgG3g99tBVZLzcjm5YW7qVHejfF9GtowQPtTduAHpJStwSPnJ/PTxv03foEQ4hqS/G9k1w+QeNqYufM6Plx9iOMXkvlgSFPKlpEvVFZVxhOP4bOp4RCDy++vcjI+1eyIhChxJPlfj9ZGW3/VIGNEbwFCI2L5esNx7m/nR4e60txjC8qvHYltnuFu9ReL5n8uNxqFuEmS/K8nfC1cOATtx1wzh88l6VnZvLxoN9W93BjfR0Y42pJX79e5UK4R95//H79u2GF2OEKUKJL8r2fTZ+BZDRoPLrDKjL+Ocux8Mu8NDsJDmntsy9EZ7+A5eKgMKqx5gXMXpflHiMKySPJXSvVWSh1SSoUrpcbns32UUuq8Umpn7uNRSxzXqs7ug2N/QtvR4JT/UotHzyfx+Z9HGdisOp3rV7JxgALAoUpDLt4+kdvZwV/zJ5sdjhAlRpGTv1LKEZgO9AEaASOUUo3yqbpAa9089/FVUY9rdZumg7M7tBqV72atNRN+3oOrswOv9ZfmHjP53DmGiPLtGHD2c7aFbjU7HCFKBEtc+bcFwrXWx7TWGcAPwCAL7Nc8iWdhz0/QPBjc85+GeWFYNJuPxTK+TyCVPW99lJ2wAKWo+uBsspQzrsufIS09w+yIhCj2LJH8awAnrngenVt2tSFKqd1KqYVKqZr57UgpNVopFaqUCjV1pGDYXMjOgHZP5rs5NjmD95YfoLV/BYa3yfdUhI25evtypsNbBOUcZPP375odjhDFniWSf37dYK7ud/crEKC1bgqsAebltyOt9SytdWutdetKlUxqQ8/OMpJ/nW5QsU6+Vd79bT+JaVm8NzhIRpcWI/V6PMo+j/a0O/45kYd3mR2OEMWaJZJ/NHDl5a8vcOrKClrrGK11eu7TL4GCJ8M32+EVkHgK2jyS7+ZNR2NYvP0kj3euTf0qpWc9z1JBKardP5MM5ULaT0+gs2XlLyEKYonkvw2op5SqpZRyAYYDS6+soJSqdsXTgcABCxzXOrbNhnI1oF6vazZlZufwxtK9+FZw45k765kQnLgR76r+7Gs2gQaZ+9m9aIrZ4QhRbBU5+Wuts4AxwCqMpP6j1nqfUuptpdTA3GrPKqX2KaV2Ac8Co4p6XKuIOWp072z1EDhe22f/m02RHD6bxMT+jXB1djQhQFEYtw18km0ubWmw/2OSThXf6wwhzGSRfv5a6+Va6/pa6zpa60m5ZRO11ktzf35Fa91Ya91Ma91Va33QEse1uNCvwcEJWj54zaZziWl8/PthOtevRI9GVUwIThSWg6MDnkM/I107ERfyGORkmx2SEMWOjPC9JDMVdsw3VuryvDa5T1lxiLSsbN4Y0AhVwFQPovhoWL8Bq/xeoGbyHs6t+cTscIQodiT5X7J3MaTFQ+trb/SGRcaxaHs0j95em9qVPEwITtyK7vc+y9+0ovzG99EXjpgdjhDFiiT/S8Lmgk99COiUpzg7R/PG0r1ULefKmK51zYlN3BJvjzKc6zyZNO1E7A9PQk6O2SEJUWxI8gc4fxiit0KLB66ZvfOHbVHsPXmRCf0CZZ7+EujuO1rztfvDVLywjYxtc80OR4hiQ5I/wM75oByh2fA8xXHJGUxddYh2tb3p37RaAS8WxZmTowPthz7PpuxGfDc1jAC/LBwcICAAQkLMjk4I88ilbHaWsVpX/V7gUTnPpv/+fojEtCzeGthEbvKWYLfV8WHouc9YsTSQlEzjTz4yEkaPNrYHB5sYnBAmkSv/o2sh6awxidsVDp65yHdbonignT8NqspI3pJuy4oWpGS65ylLSYEJE0wKSAiTSfLf8S2UrWRc+efSWjPptwN4ujrzXHcZyVsanIzO/089KsrGgQhRTNh38k+OgUMroem94Oh8ufjPQ+dYf+QC/+lWj/Lu+S/kIkoWP7+bKxeitLPv5L/nR8jJzNPkk5mdw6TfDlDbpywPtPc3MThhSZMmgXveVh/c3bKZNMmceIQwm30n/x3zoXpLqPLvwmPfbYni6PlkXukbiLOjfb89pUlwMMyaBX5+GtDU8DrJzCGvE3yPrPsr7JP9Zrcze+HsXmh+3+WihJRMPlpzmA51KtI9sPJ1XixKouBgiIxUhEbE0/mpb7m/znT4+wOzwxLCFPab/Pf8ZEzi1njw5aJpfxwhITWT1/rJ/D2lWSv/CngH9WJxTmf0hk/gzB6zQxLC5uwz+efkwJ6FxmpdZSsCcPxCMt9siuDe1jVpVL2cufEJqxvXuwHv5zxAkoMn/PqczPwp7I59Jv8Tm+FiNAQNu1z03vIDuDg68ELP+iYGJmzFt4I7wzoF8XrqfXAyFMLmmB2SEDZln8l/94/g7A4N+gCw8egFft9/lqe61qWyp6vJwQlbebJLHf5x68pul+boNW9B4hmzQxLCZuwv+WdlwP5foGE/KONBdo7m3WUHqFHejUc61TI7OmFDnq7OvNirIf9JfICczDRY+YrZIQlhM/aX/I+uhdS4y00+P+84yf7TF3m5dwNZmtEO3dO6JmWq1Geu42DYtxjC15gdkhA2YX/Jf89P4OYNde4kLTOb/60+RFNfLwY0rW52ZMIEjg6KV/oGMiWxD/HuAbDsBchIMTssIazOvpJ/eiIcXA6N7wJHZ+ZtjOBUQhrj+zTEwUG6dtqrO+r50LZuNV5KHQXxkbBuqtkhCWF19pX8D62ArFQIGkZ8SgbT/wynS4NKdKjjY3ZkwkRKKcb3acia1Prs8ekHG6fBuQNmhyWEVdlX8t/3C3hWh5rtmP5nOInpWYzr3dDsqEQx0KSGF3c1r85jZwaR4+IJy56XZR9FqWY/yT890biZ12gg0QlpzNsYyeAWvgRWkwFdwvBizwbE6nIsrPg4RG0yVngTopSyn+R/eBVkp0OjQfxv9WFQ8KIM6BJXqOntzoPt/Rl/LIiUarfB6tch6bzZYQlhFfaT/PcvAY8q7HcM5OedJ3moYwDVy7uZHZUoZsbcWRePMs68qx6DjGRY/ZrZIQlhFfaR/DOS4cjvEDiQyauPUM7Vmac61zU7KlEMlXd34emudfnumDsnGo+G3T/Asb/NDksIi7OP5H9kNWSlsserC+sOn2dM17p4uTvf+HXCLo3sEEB1L1eeP9UNXaGWcfM3M83ssISwKPtI/vuXoN19mLDdgxrl3WSFLnFdrs6OvNizAaEn09jcaALEHoV/PjI7LCEsqvQn/4wUOLyaiMp3svtUMi/2rC/TOIgbuqtFDRpW9WTcDh+yGw+Ff/4HF8LNDksIiyn9yf/oWshM5pPTjQmsVo67mtcwOyJRAlya9iEqNoWfKj4BTm7w2/OgtdmhCWERpT/5719CmnN5fk2oLdM4iJtyRz0fOtX1Yco/caR2eQ2OrzOmAxeiFCjdyT8rA314FSszW9C+bhXuqCfTOIjCuzTtQ1xKJp/Gd4IarWHVq8assEKUcKU7+UduQKVf5LeMFozv01DW5RU37dK0D7M3RHKh6wdG4l/zptlhCVFkpS75h4RAQAA4OIBf+2bM2T0cryY9aVLDy+zQRAn1Ys8GaA1TdzpDuychbC5EbTE7LCGKpFQl/5AQGD0aIiON+3Inznvz5LJP8EtqaXZoogSr6e3OA+39+SnsBOGNn4FyNYyOAe/SAAAgAElEQVS+/9mZZocmxC0rVcl/wgRIuWodjvRMV/73XhlzAhKlxpiudSlbxonJa09Anw/g3D7YPMPssIS4ZRZJ/kqp3kqpQ0qpcKXU+Hy2l1FKLcjdvkUpFWCJ414tKurmyoUorAplXXiySx3WHDjHVtcO0KAv/PU+xMsflyiZipz8lVKOwHSgD9AIGKGUanRVtUeAOK11XeAjYEpRj5sfP7+bKxfiZjzcsRZVy7ny3vID6N6TjcIV48wNSohbZIkr/7ZAuNb6mNY6A/gBGHRVnUHAvNyfFwLdlBW63kyaBO7ueQfhuLsb5UIUlauzIy/0qM/OE/GsiHaBLq/AoeVwYJnZoYlS5LetB1iyIxpt5QGFlkj+NYATVzyPzi3Lt47WOgtIACpevSOl1GilVKhSKvT8+ZufRz04GCb9Nx0vr1gUOfj7ZjJrllEuhCUMaeVL/SoeTF11iMw2j0PlxrDiZUhPMjs0UQokpWfhtuJZGi0fYvVjWSL553cFf/VHVmHqoLWepbVurbVuXalSpVsK5rknXDn/8UNkT2tDxAlnSfzCohwdFON6N+T4hWR+CDsNAz6Gi6eM9n8himjuX/ton7OT8nXbWn1ckiWSfzRQ84rnvsCpguoopZwALyDWAse+VloCzlEbUA36WmX3QtzZsDJta3nzydojJFVuCa1GGT1/Tu82OzRRgp1PTGfZF+EEfhJGlRHTCAgwuq9biyWS/zagnlKqllLKBRgOLL2qzlJgZO7PQ4E/tLUatLKzoMMz0GSwVXYvhFKKV/o05EJSBl+uOwbd3wB379xF37PNDk+UUI+/cY6/futHZIIfWisiI41xS9b6AChy8s9twx8DrAIOAD9qrfcppd5WSg3MrTYbqKiUCgdeAK7pDmoxZSsa/xmrt7DaIYRo4VeBvkFV+XL9Mc5luUGv9+BkKITNMTs0UQJFxiTz6+yKpGe65ilPSTHGL1mDRfr5a62Xa63ra63raK0n5ZZN1Fovzf05TWs9TGtdV2vdVmt9zBLHFcJMY3s1JCMrh2lrj0DQMKjVGda8DYlnzQ5NlDAfrj5M1sX81xS31jilUjXCVwhbquVTlhFt/fh+6wmOXUiGfv+DrFRj5k8hCmlPdAK/7jpFtQox+W631jglSf5CFMGz3erh6uTA1FWHwKcudHoB9i6Eo3+YHZooAbTWTFl5EB83Bz7o+S7uLul5tltznJIkfyGKoJJnGUbfUYcVe8+wPSoOOj0P3nXgtxdl0XdxQ+uOXOCf8Au83SKR+xvOYdY7e/D3B6XA3x+rjlOS5C9EET16ey18PMoweflBtFMZ6PdfiD1mrPsrRAGyczTvLz9ATW83eqnN4ORG8HOBRERATg5ERFh3gKokfyGKqGwZJ57rXo+tEbGsPXAO6nSFoHvgn4/gwhGzwxPF1C87TnLwTCIv96iL44El0KA3uJS12fEl+QthAfe2qUltn7JMWXmQrOwc6DUJnN2Mvv+y6Lu4SlpmNv9dfYimvl708wiHlAvQxPpTOlxJkr8QFuDs6MDLvRtw5FwSi7ZHg0dl6P4mRKyH3QvMDk8UM3M3RnAqIY3xfRrisH8RuHhC3R42jUGSvxAW0qtxVVr6led/vx8mNSMbWo4C3zawagKkWGc2E1HyxCVnMP3PcLo2qEQHf0848CsE9gdn1xu/2IIk+QthIUopXukbyNmL6Xy94bixkHT/j2TRd5HH9D/DSU7PYnyfQKNLcFqCzZt8QJK/EBbVJsCb7oFV+OKvo8QmZ0DVIGPR9+3zZNF3wYnYFL7ZFMnQVr40qOoJexeDWwWo3cXmsUjyF8LCxvVuQHJGFp/9EW4UdHkFyvnCsudk0Xc79+HqQzg4wPM96kNGirEYUOBAcHS2eSyS/IWwsHpVPLmndU2+3RzBidgUKOMBfT+Ac/th03SzwxMm2XsygSU7T/Fwx1pU83KDI6shI8mUJh+Q5C+EVTzfoz6ODooPVx8yChr2gwb94K/JEBdpbnDC5rTWvLf8ABXcnXmiSx2jcO8iKFsZAjqZEpMkfyGsoEo5Vx7pVIslO0+x92SCUdhnCigHY9lH6ftvV/4+fJ6NR2N4tls9yrk6G50ADq8y1h1xcDQlJkn+QljJ453rUMHdmfdXHDAW4y5fE7q+AodXwkFZ9N1eZOdoJq84iJ+3O8G3+RuF+36G7HRoNty0uCT5C2El5VydeebOemwIj2HdkQtG4W1PQJUmsPxlSE80N0BhEz/nTuMwtlcDXJxyU+6uH6BSIFRrblpckvyFsKLgdn7U9HZj8oqD5ORoo1dH/48h8TT8KYu+l3YpGVlMXXWQZr5e9AuqZhTGHIUTW4yrfisv0n49kvyFsKIyTo681LMBB05f5JedJ43Cmm2MRd+3zIBTO0yNT1jXrHXHOHsxndf7N8LBITfR7/rBuPfT9B5TY5PkL4SVDWhanaa+Xnyw8hApGVlGYfc3jZ4eS56Rvv+l1JmENGb+fYx+QdVoHeBtFObkGMm/dhcoV93M8CT5C2FtDg6K1/s34szFNGaty12+2q28Me//2T2w4RNzAxRW8cGqg2TnaMb3afhvYdRGSIiCZiPMCyyXJH8hbKBNgDf9gqox8+9jnE5INQoD+0Oju+DvKXD+sLkBCovaHR3P4u0neahTADW93f/dsOt7cPGAhv3NCy6XJH8hbGR8n4Zka83UlYf+Lew7FZzdYekzRpOAKPG01ry77AAVy7owpmvdfzdkpMC+JcYHvot7wTuwEUn+QthITW93HulUi8U7TrLzRLxR6FEZer8PJzZD6GxzAxQWsWrfGbZGxPJCz/p4ul4xZ8/+XyAjEZqb3+QDkvyFsKmnutTBx6MM7yzbbwz8AqP9t86dxrTP8SdMjU8UTXpWNu8tP0j9Kh7c27pm3o1h86BiXfDvaE5wV5HkL4QNebo6M7ZXfcIi41i2+7RRqJTR919rWfaxhJu3MYKo2BRe69cIJ8cr0uu5A8a3u5YjTe3bfyVJ/kLY2NBWNWlUrRyTVxwkLTPbKKzgD90mQvjvsPtHcwMUtyQmKZ1P14bTpUEl7qhfKe/GsHng4AzN7zMnuHxI8hfCxhwdFK/1D+RkfCqz/zn+74a2j4FvW1g5DpLOmxeguCUfrzlCSmY2E/oG5t2QmWb08gkcAGV9zAkuH5L8hTBBhzo+9GxUhel/hnPuYppR6OAIAz+FjGTjA0CUGPtPXSRkSyT33+ZHvSqeeTceWApp8dBqpDnBFUCSvxAmebVvIJnZOUy5sutn5YZwx1hjrveDv5kXnCg0rTVvLt1HeXcXXujR4NoKYfOgQi0IuMP2wV2HJH8hTBLgU5aHO9Vi0fZowiLj/t3Q8TmoEgS/PgcpseYFKApl6a5TbI2IZWyvBni5X7Uc4/nDEPmPcdXvULzSbfGKRgg7EBICAQFGLpjxREOcjvkzcclesnNye/k4ucDdM4wFP5a/ZGqs4vqS0rN4b/kBgmp4cc/VXTsBts4CRxdofr/tg7sBSf5C2FBICIweDZGRRo/OE1GK6F8bseV3D77bGvVvxapB0Hmc0fyz72fzAhbX9ekfRzh7MZ23BjXG0eGqLpxpF40bvU2GgEel/HdgIkn+QtjQhAmQkpK3LD3NgdSNjfhw1SFikzP+3dDpeajeApa9AEnnbBuouKGj55P4+p/jDG3lS0u/CtdW2PmdsUB729G2D64QJPkLYUNRUfmXp8a5kJxuLPxxmaMT3PWF0ftHBn8VK1pr3v51P65Ojozr3fDaCjk5sHWm0XW3RkvbB1gIkvyFsCE/v4LKFaM6BPDDthPsujTvDxi9f+6cYKz5K4O/io01B87x9+Hz/Kd7PSp5lrm2wtG1EHsMbnvc9sEVkiR/IWxo0iRwv2pCR3d3o/w/3evh41GGiUv2Gks+XtJ+DNS8DVaMhYunbBuwuEZqRjZvL9tHvcoejOwQkH+lLV+ARxUIHGjT2G5GkZK/UspbKfW7UupI7r/5NHyBUipbKbUz97G0KMcUoiQLDoZZs8Df35jixd/feB4cbMz7M6FvILuiE1gQesUEbw6OcNcMyMqApc9K84/JPv3jCCdiU3l7UBOcHfNJoRfCIXwNtH7E6LlVTBX1yn88sFZrXQ9Ym/s8P6la6+a5j+L7USiEDQQHQ0SE0SwcEWE8v2RQ8+q0reXNlJUHiUlK/3dDxTrG0o/hv8P2ebYNWFx2+Gwis9YdY0hLX9rXqZh/pU2fgmMZaP2QbYO7SUVN/oOAS3+J84C7irg/IeyaUopJdzUhOT2Ld387kHdj29FQ6w5Y+YpxdSlsKidH8+riPXi4OjGhX2D+lRLPws7vjQncPCrbNsCbVNTkX0VrfRog99+CztZVKRWqlNqslCrwA0IpNTq3Xuj58zKxlbBP9ap48mTnOvy84yTrj1zx/8DBwej94+gCix+Vhd9t7MfQE4RGxvFq30C8yxbQnLPlC8jOgA7P2Da4W3DD5K+UWqOU2pvPY9BNHMdPa90auA/4WClVJ79KWutZWuvWWuvWlSoVv0ERQtjKU13rUtunLBN+3ktqRva/G7xqwMBpcGoH/PW+eQHamQtJ6by/4iBta3kzrJVv/pXSE2HbbGg00GimK+ZumPy11t211k3yeSwBziqlqgHk/pvvSBSt9ancf48BfwEtLHYGQpRCrs6OvHt3E6JiU5j2x5G8GxsNghb3w/r/QcQGcwK0M+/9doCUjCzeu7sJqqDFWMLmQnoCdPyPTWO7VUVt9lkKXJqndCSw5OoKSqkKSqkyuT/7AB2B/UU8rhClXoc6Pgxr5cuX645x8MzFvBt7TwHvWrB4NKTG578DYREbwy+weMdJnuhch7qVPfOvlJUBmz6HgNuhRivbBniLipr8JwM9lFJHgB65z1FKtVZKfZVbJxAIVUrtAv4EJmutJfkLUQiv9g2knJsz4xft+XfiN4AyHjD4K0g8LaN/rSg5PYtxi3cTUNGdp7vWLbji7gWQeKrEXPVDEZO/1jpGa91Na10v99/Y3PJQrfWjuT9v1FoHaa2b5f472xKBC2EPKpR14fX+gew8Ec83myLybvRtBV1fgX2LjeQjLG7qqkNEx6XywdBmuDo75l8pOxPWTTXmYarb3bYBFoGM8BWimLureQ26NKjEBysPEXEhOe/GTi+AX3v47SWIOWpOgKXU1uOxzN0Ywcj2AbSt5V1wxV3fQ3wkdHml2CzOXhiS/IUo5pRSTB7cFCdHxdiFu/JO/eDgCIO/NCaB+2mksV6sKLLUjGxeXriLmt5uvNw7n9W5Lrnyqr9eT9sFaAGS/IUoAap6ufLGgMZsi4hjzsaIvBvL1zT6/5/ZA6teMSW+0uZ/vx8iIiaFKUOa4u7iVHDFnd9BfFSJu+oHSf5ClBhDWtbgzoaVmbrqIMfOJ+Xd2KA3dHgWQr+GPQvNCbCU2B4Vx+x/jhN8mx8d6vgUXDErA9Z/CNVblrirfpDkL0SJoZTi/cFBuDg6MHbh7ry9fwC6TTRm//z1PzL9wy1KycjipR93Uc3LjVf6FjCFwyU75+de9Y8vcVf9IMlfiBKlSjlX3hzYmLDIOGb/cyzvRkdnGPq18e9PoyAz1ZQYS7JJvx3geEwyU4c1xaPMdZp70pPgr8lQs12JvOoHSf5ClDh3t6hBj0ZV+HDVYfadSsi70csX7p4JZ/fAinHmBFhCrT1wlpAtUYy+vfb1m3sANk2HpLPQ850SedUPkvyFKHGUUkwZ0pTy7s48+/2OvHP/ANTvBR2fM6Z+DpPpnwvjfGI6Ly/cTWC1crzQs/71Kyedh43TIHAA1GxrmwCtQJK/ECWQd1kXPrq3OccuJPPOb/kMmO82EWp3heUvwYlttg+wBNFaM37RbhLTs/hkeHPKOBUwmOuSv6cYTWrd3rRJfNYiyV+IEqpjXR9G316b77ZEsXLvmbwbHRyN9n/PavDjA8Y881cJCYGAAGOm6IAA47k9CtkSxZJFTsR+1Z2G1Tyv/16cPwxhc6DVKPC5znQPJYAkfyFKsBd7NqBJjXKMX7ybMwlXDfBy94bhIcbEbz8+aHRNzBUSAqNHQ2SkMS1QZKTx3N4+APadSuClyXHEr25KzFnn678XWhvrKLuUha6vmhKvJUnyF6IEc3FyYNrwFqRn5vDs9zvIzM7JW6FqEAz6DE5shpX/rrI6YQKkpOStmpJilNuLi2mZPB2ynbi/G5CdkbepJ9/34sCvcOwv6DoByt7ghnAJIMlfiBKudiUP3h8cxNaIWKauOnRthaChuQPAZkPoHACiovLfV0Hlpc2ldv4TcalkJrjmWyfPe5GRAqtehcqNjYXZSwFJ/kKUAne1qMED7fyZte4YK/eevrZC9zehbg/47UU4+gd+fvnvp6Dy0mbexgiW7znDy70a4OeXf1fNPO/Fho8h4QT0/cCYR6kUkOQvRCnxWv9AmtUsz0s/7b52+odLN4ArNYQfRzJp3Enc3fNWcXeHSZNsF69ZdkTFMWn5AboHVuax22szaRLXfy/OH4J/PoImQyGgk83jtRZJ/kKUEmWcHPk8uCXOjoon528nJSMrbwXXcnDfAnB2IzitN7M+TcDf3xij5O8Ps2ZBcLA5sdvK2YtpPP5tGFW9XPlwWDMcHBTBwca55/te5GTDkjHGTd7ek80O36Ik+QtRitQo78Ynw1tw5FwiLyy4avpnMGYAHfEDJJ8nOPtuIg6nkJMDERGlP/GnZWYz+tswktOz+OrBNpR3d7m8LTjYeA+ueS+2fQXRW41lMz0qmRG21UjyF6KUuaN+JV7tG8jKfWf47+/53ACu0RKGfAUnt8Pix4yr21JOa82ri/ew60Q8H93bnAZVC1iL90rxUbDmLeNeSdN7rB+kjUnyF6IUeqRTLUa0rcn0P4+yeHv0tRUC+xvNGAeXGbOAlvI1gGetO8biHSd5oUd9ejaueuMX5OTAkqeNdqD+H5XY+Xuup3TcthZC5KGU4u1BTTh+IZnxi/bg5+1O64CrliJs9wSkXDBWonL3hh5vmxOslS3ddYr3VxykX9NqPHNnIUflbvoUjq+DQdONprJSSK78hSilnB0d+OL+VtSo4MYj80I5fDbx2kpdJxj91jd8Av98bPsgrWxj+AVe/HEnbWt5899hzVCFuYI/tRPWvgONBkHz0nsjRJK/EKVYeXcXvnm4LS5ODjw4eyvRcVcN61UK+k6FxoNhzRsQNteUOK3hwOmLPP5tGLV8yvLlA61xdb7BhG0AGcmw6FEoWwn6f1wqm3sukeQvRClX09udbx5uS3JGFg/O3kpMUnreCg6OxhoAdXvAr8/B9m/NCdSCImOSGTVnK2XLODH3obZ4uTvf+EVaG/c/YsLh7i+MprBSTJK/EHYgsFo5vh7VhpPxqYyas42E1My8FZxc4N75UOdOWDoGtn9jTqAWcCI2hfu+3EJGVg7zHm5L9fJuhXvhli9gz09w5wSo3dm6QRYDkvyFsBNtAryZcX9LDp65yAOzt5CQctUHgLMrDP8O6naHpc+UyA+AU/GpjPhyM4lpmXz7yG2F69IJEPEPrJoADfpBpxetG2QxIclfCDtyZ8MqzHygFQdPJxI8ezPxKRl5Kzi7wr0h/34AbJllTqC3IDouhRFfbiYhxUj8TWp4Fe6FcZHGmsfeteDuGcYCB3bAPs5SCHHZnQ2rMPPBVhw+m8SIL7dwLvGqdQAufQA06GfMX//Hu8V+HMCRs4kMnbGJuOQM5j3SlmY1yxfuhSmxMH8IZGcY33pcC/mBUQpI8hfCDnVtUJkvH2xNxIVkBn++kaNXTwTn7Ar3fAMt7jfGASx7rtiOBN55Ip5hMzeRrTULHm9PS78KhXthZhr8cB/ER8Lw76FSA+sGWsxI8hfCTnWuX4kfRrcjNSObITM2EhYZl7eCoxMM/Axuf9HoAvrjg0ZXyGJk9b4z3PflZsq5OrPwifYEVitXuBdmZ8LiRyFqk9HTKaCjdQMthiT5C2HHmtUsz+KnOlDezZn7vtzMorCrpoJQylgMvvcUOLQcvu4F8SfMCfYKWmum/xnO4/PDqFvZg4VPtMe/YtnCvTg7y5jT6MCvxhQXTQZbN9hiSpK/EHbOv2JZFj3ZgRZ+5Xnxp11MXLKXjKyrloNs9wTc96Nxc3RWF4jcZEqsAEnpWTz7w06mrjpE/6bV+fHx9lQul/9qXNfIzoKfR8O+n6Hnu9DuSesGW4xJ8hdCUNGjDPMfuY3Hbq/FN5siuWfmJiIuXNXEU68HPLrWuCk6bwBs/sLmN4J3R8fTb9p6ftt9irG9GjBtePPCjdwFo8nqh/tg7yLo/hZ0eMa6wRZzkvyFEAA4OTowoV8jpt/XkmPnk+jzyXpCtkSir0zwlerDY2uhbjdYOQ6+HwHJMVaPLTM7h+l/hjNkxkYysnL4YXR7nu5at3Bz9YAR47yBEP67MUtnp+esG3AJoHQx7cLVunVrHRoaanYYQtil0wmpjP1pN/+EX6BDnYq8PagxdStfMWBKa9gyE35/Hdx9jOkQrDQqdkdUHK8s3sPBM4n0DarKe3cH5VmI5YZO74IFD0DSWRgy25jOuhRTSoVprVvfsJ4kfyFEfnJyNCFbo5i68iApGdk81DGAp7vWzZt4T++ChQ8b8+G0fNCYFtqtkF0tbyA6LoWP1xxh0fZoqni68tagxvQqzFz8V9r5HSx7HtwrGl1XfW+YE0s8Sf5CCIuISUpn6qpDLAg9QVkXJ0Z28OeRTrXxLpv7IZCRAn9Pho2fGUm29/vQZMgtz4h5Mj6V2euPM39zJCgY2d6fZ7vVw9O1EJOzXZIcA7+9APt/gYDbYeicUrcMY0FskvyVUsOAN4FAoK3WOt9srZTqDXwCOAJfaa1vuBKyJH8hipdDZxKZ9scRlu85TRknB/o3rc6Itn609CtvtL2f3mVMCXF6F9RoZfSm8e9QqH1nZeew6VgMP2w9wcp9Z9BaM6SlL8/1qE+Nwk7MBkZz1N5FsHI8pMZDl/HQ8TljzIKdsFXyDwRygJnAS/klf6WUI3AY6AFEA9uAEVrr/dfbtyR/IYqnw2cTmbMhgqU7T5KckU2N8m70aFSFzg0q0aKGJ+WPLDamhEg8BfV6Qqfnwa/9Nd8EYpLS2Xwslg1HL7B63xkuJGVQztWJEW39eKC9P74V3G8usBPbYNWrxoLr1ZrDXZ9DlcYWPPOSwabNPkqpvyg4+bcH3tRa98p9/gqA1vr96+1Tkr8QxVtSehbLd59m9f4zrD9ygfTcsQEBFd2p7+3IkMxl3H5hAe5Z8ZzxDGJr5aGEunXkeEIO4eeSOJ1gzCnk7uJIlwaVGNisOl0aVC58100wrvSP/gEbPjaWXfSoAne+Ds3vM9YpsEOFTf62+C5UA7hySGA0cFt+FZVSo4HRAH5+ftaPTAhxyzzKOHFPm5rc06YmKRlZ7IyKZ8eJePaeTOBEXArj47qTmtKOoY7reDRhOQMT3+BO3NhYphNRlbqi29xBq/q+BNXwwtnxJnqdaw0XDsPexbB7AcQdB89q0OMdaP0QlCnkNM527obJXym1BsjvFvsErfWSQhwjv7s++X7d0FrPAmaBceVfiH0LIYoBdxcnOtT1oUNdnzzlWmuycu4iO3sq+uRmPHZ9T8/9SyD6dzjlDCfaGk001ZpChVpQrppx09jB2bhyT0swZt6Mj4TzB+H0buMKP/EUoKDWHUa7fuO7wamMOSdfQt0w+WutuxfxGNFAzSue+wKnirhPIUQJoJTC2VEZV/a1bjce/T82JlQLXwORGyB0NmSl3XhnAGUrG5Ow1eps3E/wqmHdEyjFbNHssw2op5SqBZwEhgP32eC4QojiyMnFGBB2aVBYdpYxTiDhBFw8BamxkJMFOTngWg7cvKFcdagcCGV9rr9vUWhFSv5KqbuBT4FKwG9KqZ1a615KqeoYXTr7aq2zlFJjgFUYXT2/1lrvK3LkQojSwdEJKjc0HsJmipT8tdY/Az/nU34K6HvF8+XA8qIcSwghhOXIxG5CCGGHJPkLIUwTEgIBAcaa6QEBxnNhG/Yz5lkIUayEhMDo0ZCSYjyPjDSeAwQHmxeXvZArfyGEKSZM+DfxX5KSYpQL65PkL4QwRVTUzZULy5LkL4QwRUEzuMjMLrYhyV8IYYpJk8D9qok73d2NcmF9kvyFEKYIDoZZs8Df35jt2d/feC43e21DevsIIUwTHCzJ3ixy5S+EEHZIkr8QQtghSf5CCGGHJPkLIYQdkuQvhBB2yCILuFuDUuo8EFmEXfgAFywUTklhb+dsb+cLcs72oijn7K+1rnSjSsU2+ReVUiq0MCvYlyb2ds72dr4g52wvbHHO0uwjhBB2SJK/EELYodKc/GeZHYAJ7O2c7e18Qc7ZXlj9nEttm78QQoiCleYrfyGEEAUo0clfKdVbKXVIKRWulBqfz/YySqkFudu3KKUCbB+lZRXinF9QSu1XSu1WSq1VSvmbEacl3eicr6g3VCmllVIlvmdIYc5ZKXVP7u96n1LqO1vHaGmF+Nv2U0r9qZTakfv33deMOC1FKfW1UuqcUmpvAduVUmpa7vuxWynV0qIBaK1L5ANwBI4CtQEXYBfQ6Ko6TwFf5P48HFhgdtw2OOeugHvuz0/awznn1vME1gGbgdZmx22D33M9YAdQIfd5ZbPjtsE5zwKezP25ERBhdtxFPOc7gJbA3gK29wVWAApoB2yx5PFL8pV/WyBca31Ma50B/AAMuqrOIGBe7s8LgW5KKWXDGC3thuestf5Ta31pZdTNgK+NY7S0wvyeAd4BPgDSbBmclRTmnB8Dpmut4wC01udsHKOlFeacNVAu92cv4JQN47M4rfU6IPY6VQYB32jDZqC8UqqapY5fkpN/DeDEFc+jc8vyraO1zgISgIo2ic46CnPOV3oE48qhJLvhOSulWgA1tdbLbBmYFRXm91wfqK+U2qCU2qyU6m2z6KyjMOf8JnC/UioaWA48Y5vQTHOz/99vSklezHxHQnsAAAIPSURBVCW/K/iruy4Vpk5JUujzUUrdD7QGOls1Iuu77jkrpRyAj4BRtgrIBgrze3bCaPrpgvHtbr1SqonWOt7KsVlLYc55BDBXa/1fpVR74Nvcc86xfnimsGr+KslX/tFAzSue+3Lt18DLdZRSThhfFa/3Nau4K8w5o5TqDkwABmqt020Um7Xc6Jw9gSbAX0qpCIy20aUl/KZvYf+2l2itM7XWx4FDGB8GJVVhzvkR4EcArfUmwBVjDpzSqlD/329VSU7+24B6SqlaSikXjBu6S6+qsxQYmfvzUOAPnXsnpYS64TnnNoHMxEj8Jb0dGG5wzlrrBK21j9Y6QGsdgHGfY6DWOtSccC2iMH/bv2Dc3Ecp5YPRDHTMplFaVmHOOQroBqCUCsRI/udtGqVtLQUezO310w5I0FqfttTOS2yzj9Y6Syk1BliF0VPga631PqXU20Co1nopMBvjq2E4xhX/cPMiLrpCnvNUwAP4KffedpTWeqBpQRdRIc+5VCnkOa8Ceiql9gPZ/2/P3m0QBoIAiM4G7oCqqAKXgCzHtGGJAmiKFpySroOjAMsgWaed18B9gtFpD5gzcz1v17/ZeeYJeEbEnTb+uPX8mIuIF21sd/n+YzyAASAzF9q/xhV4Ax9g/Ov6Hd+dJOmgnsc+kqSDjL8kFWT8Jakg4y9JBRl/SSrI+EtSQcZfkgoy/pJU0AZ5HP0S9tjunAAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# M=3\n", "p_lsq_3 = fitting(M=3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### M=9" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting Parameters: [-1.70872086e+04 7.01364939e+04 -1.18382087e+05 1.06032494e+05\n", " -5.43222991e+04 1.60701108e+04 -2.65984526e+03 2.12318870e+02\n", " -7.15931412e-02 3.53804263e-02]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XdcVfX/wPHXhw2CKMOJDDcOcOA2NbfmyNTSaGlmtve0bHzTsmXDsmxZSWpZOXLmym2CW3ALiAsQRZENn98fB/2pgazLvVx4Px8PHvdyzrnnvM9F3/dzP+dz3h+ltUYIIUTlYmPpAIQQQpifJH8hhKiEJPkLIUQlJMlfCCEqIUn+QghRCUnyF0KISkiSvxBCVEKS/IUQohKS5C+EEJWQnaUDKIiXl5f29/e3dBhCCGFVIiIiErXW3oVtV26Tv7+/P+Hh4ZYOQwghrIpSKqYo20m3jxBCVEKS/IUQohKS5C+EEJWQJH8hhKiEJPkLIUQlJMlfCCEqIUn+QghRCZXbcf6inEhJgMSDkHQc0i9A5mWwcwJHV6jmD95NwN0HlLJ0pEKIYpDkL66XmQqHV8DRNXDsH7hQhPtFqtaFhr2g2VCo3xNs5AulEOWdJH8BWsOJfyH8ezjwF2SmgJM7+N8C7cdDjUDwbADOHuBQBbIzID0Zko5BfCREb4D9C2HHT1DdHzpMgJCxYOdo6TMTQhRAaa0tHUO+QkJCtJR3KGO5ObD/T9jyBZzaAY7u0GwIBN0Jfl3Axrbo+8rOhKhFsP07iN0M7r7QaxK0HCFdQkKYkVIqQmsdUth28v28MtIaohbDjM7w+4NGK37gh/BcFAydDgHdipf4AewcjEQ/dhncuwBcqsMf42DOaMK+u4C/v9Eb5O8PYWFlcVJCiOKQbp/K5sR2WP4SnIwAz0Yw4gdodrtp++kb3Gp8gGz7irCP9zJ+oQOpWcaqmBgYP954HhpqukMKIYpHWv4VWFgY/9/i9s0h7Nkf4LvecPE0DP0CHt0KLe4omwu0NrbQ6TEmbv2c1CyX61alpsLEiaY/pBCi6KTlX0GFhRkt7NRU4/eYE7aMn34XPOtK6CsDjKGaZhB70j7/5bFmObwQogDS8q+gJk78/8R/RWqWCxPnjjRb4gfw9S1geb3yOdBAiMpCkn9FpDWxsfknV3O3uCdPBpfre31wsU9l8sgw48KzEMIiTJL8lVLfK6XilVL7ClivlFKfKaWOKKX2KKXamOK4Ih8XT8HsO/CteiLf1QW1xMtKaCjMnAl+fsaITz8/mPnsEkJdH4O1U8wbjBDiKlO1/GcB/W+yfgDQKO9nPDDDRMcV1zq4DGZ0gditTH7uCC4u17esXVyMlri5hYZCdDTk5hqPoe+OgNb3wvr3jfsChBBmZ5Lkr7VeDyTdZJOhwE/asBWoppSqbYpjC4w7bpe9BHNGGXV2Ht5A6Os9mTlTXd/inllOhlcqBYM+gUZ9YdmLEL3R0hEJUemYa7RPXeDafoi4vGWnzXT8iivxMMwfA2f2QodHoM9bV8sqhIb+f7K/lJ5FeMx5vlh7kSPxKRxPvEzS5UwupGaSlaOxs1E42tvg5epIjapO1PeqQrPaVWnp406Tmm7Y2Jj4Ll1bOxj+LXzTC369D8avg2pm7pMSohIzV/LPL3P852qfUmo8RrcQvubunLZGu+fCX88ayX70XGgy4LrVsedSWbL3NMv3n2Fv3AVy897xOu5O1Pd2xc/ThWrO9jjY2ZCTC+nZOSRcyuDsxXR+DU8iNTMHAC9XB7o29KJ/i9rc2tQbR7ti3v1bECd3GD0HvukJc0Nh3CqpBySEmZgr+ccB9a753Qc4deNGWuuZwEwwavuYJzQrdKWbJ+IHowbPHd+Ae10AMrNzWbbvNLM2R7Mz9gIAwfWq8fitDelQ35PgetVwdSz8z56bq4lJSiUi5jwbDyew4XAiC3adwt3ZnsHBtXmgcwANa5hgyKhXIxj2NcwdDavehP7vln6fQohCmSv5LwIeV0rNBToAyVpr6fIpieQ4o5vkZAR0eQp6TgJbO9Iyc/hxSzTfbTxOwqUM6ntV4ZUBTRnYsjb1PFwK3e2NbGwUAV5VCPCqwoi2PmTn5LLxSCJ/7jzJb+FxzN4aS+/AGjzSowFt/TxKd05NBxrVQ7d+CQ16QqM+pdufEKJQJqnqqZSaA/QAvICzwBuAPYDW+iullAKmY4wISgXGaK1vWrJTqnrm49g/Rv9+dgbc/iU0G0pWTi5zt5/g89WHib+UwS2NvBjbNYDujbxN30+fJzElg5+2xPDzlmjOp2bRt1lNXhrQlAbepfgmkJVudP+knIVHNoNbTZPFK0RlUtSqnlLS2RpoDZs+hdVvGcXY7poN3o0Jj05i4p/7OHj2Eu38q/NCv6a0DyhlK7wYUjOz+X7jcWasO0p6di73dfLj+b5NqFKEbqV8xUfB192hcV/jHIUQxSbJv6LISoOFj8G+343qm0Onc0k7MWVpFHP+PUEddyfeGNKcvs1qoixUNz/hUgYf/32IOf/GUreaM1PuaEn3xt4l29mGj40PuTt/MmYGE0IUiyT/iuDSGZgzGk7tNCZG6foMEbHneXreLk6eT2PcLfV5qlejkre0TSw8OomXft/D0YTL3N3Bl0mDmuFkX8yRQTlZRvfPpTPw2DZwMd83GSEqApnMxdqd3g0zb4WEA3DXbHK6PMMnqw9z59dbAfhtQideHRhYbhI/QIi/B0ufuoWHu9fnl22xDJ2+icNnLxVvJ7b2RrnptCRYIXWfhSgrkvzLo8hF8H1/UDYwdgXJfv0YO2s7n6w6zJDgOix98pbSj7ApI452trwyIJAfx7YnMSWDwdM3smDnyeLtpHYQdHkadv8CxzeUTaBCVHKS/MubbV8bQzlrNIOH1nDYJoChX2xk89FEpgxrybS7WuHmlH+N/PKke2Nvlj11C0E+1Xh63i4+WHGA3NxidDF2e96YB3jZS5CTXXaBClFJSfIvL7SGVW8ZtW6aDIQH/mLDGRuGfbmZlIwcfnmoI3d3sK67nmtUdWL2gx0Y1a4eX6w9yiNhEaRmFjGR2ztD/ykQvx/CpfibEKYmyb88yMmGhY/Dxo+h7QNw508s3J/E2Fnb8anuzKLHu9DOv3x28xTGwc6Gd+9oyaRBzfg78ix3f7ON85czi/bipoOg/q2wZjKkJJRtoEJUMpL8LS0rHebdA7tmQ/eXYdAnfLs5lqfm7qKNb3XmPdyJOtWcLR1lqSilGNs1gK/uaUvk6Yvc+fUWziSnF+WFMOB9yLpsDP8UQpiMJH9LykozatocWga3fQS3vsJna47wzpIoBrSoxY9j2+PuXP7794uqb/Na/DimPaeT0xk+YzPRiZcLf5F3Y+gwAXaFwdnIsg9SiEpCkr+lZKbCL3fB0bXG0MZ24/hk1SE+/vsQw9v4MP3uNsUfI28FOjXwZM5DHUnLymH0N1uJPZda+ItueQ4c3YzCb0IIk5DkbwmZl+GXOyF6Awz7ClrfwyerDvHJqsOMaOvD+yOCsC2jujzlQUsfd8LGdbj6AXAiqZAPABcP6PosHF4hE78IYSKS/M0tO8Po44/ZZJRiDh7F9DWHryb+qcMrduK/IrB2VWY/2IFL6Vnc/e1WTl5Iu/kLOjwMVevC35Nk4nchTECSvznl5sAf4+HoGhj8GbQcQdi2GD5ceYhhretWmsR/RYu67swe14ELqVmEfrOVxJSMgje2d4ZbJxqlrCMXmC9IISooSf7mojX89YyRuPpOhjb3smzvaV5fsI+eTWtU+K6eggT5VGPWmPacuZjO2FnbuZxxk/sAgkdBjeaw+n9y45cQpSTJ31z+eR92/Ai3PA+dH2fz0USemruL1r7V+eLuNtjbVt4/RVs/4z3Yf+oij4TtICsnN/8NbWyh50RIOgp7fzVvkEJUMJU345jTvt9h3RQIvht6vkbU6YuM/ykCfy8Xvrs/BGeHijeqp7h6BdZkyrAWrD+UwEvz9xRcCqLJQKgVZHyYSutfiBKT5F8GwsLA3x9sbMDfJ4Owt/4G384w+BMSUjIZ92M4ro52/Di2PdVcHCwdbrlxVztfnuvTmD92nuSzNYfz30gp6PEKnD8Oe+aaN0AhKhBJ/iYWFgbjx0NMjNHNH3PSkfELpxHGr6RrO8b/HE7S5Uy+vT+E2u7WfeduWXi8Z0OGt/Hhk1WHWbKngGmemwyA2q3yWv9Z5g1QiApCkr+JTZwIqTcMW0/NcubVt1x56fc97Iy9wLS7gmlR190yAZZzSimm3NGCNr7VeO63Xew7mZzfRkbr/0IM7J5j/iCFKAmtIf4ApCZZOhJAkr/JxcYWsPwELNx1ihf6NaF/i9rmDcrKONrZ8vW9IXhWcWTcj+HEX8ynDlDjflCnNaz/ALKLWChOCEvJzoS5ofBlB5jWwriz38Ik+ZuYbwFVl23d0ri9VR0e7dHAvAFZKW83R765L4TktCzG/xxBRnbO9Rtcbf3Hwt7fLBOkEEW17l04uMQoVVLN17jfJz2fb7VmJMnfxCa/moCL/fX9Pso+h+ZDYnn3jiCLTbJujZrVqcrHdwaz68QFJi+J+u8GjfpCzRaw6RPILWB4qBCWdvE0bP4cgkcbc3Hf/gVcjocdP1k0LEn+ppSbQ6jdvcy8/UX8fLJRSuNUPZ26g/ex6JN6MqSzBAa0rM24rgH8tCWGhbtumA5SKej6DCQeMlpVQpRH/86E3Gzo/pLxe922xui/iB8tWqpEkr8pbf4cYrcQ+moXok/Y8cKve6k5fjWz3qmFn2cVS0dntV4a0JR2/tV5+fe9HLpxQvhmt0N1f9jwsdT8EeVPThZEzIKmt4FHwP8vbzkczh2GhAMWC02Sv6mc2Qtr3oHAwRA8innbY5kXfoLHb21Ir8Calo7Oqtnb2jD97jZUcbRjwuwIUq4tAWFrB52fhFM74Ph6ywUpRH6OrYO0JGh9z/XLm9xmPB7+2+whXSHJ3xRysuDPCeBcHQZ9ysGzKUxauJ+uDb14pk9jS0dXIdSs6sTno1sTnXiZl3/fg762ld8qFFxrGtNgClGe7J0PTu7QoOf1y6vWBo8GELvFMnEhyd80tkyHs/tg0MekO1TjiTk7cHOyZ9pdrSplsbay0qmBJ8/1bcJfe07zW3jc/6+wd4KOjxqtrJM7LBafENfJzYFDy41Wvp3jf9f7dTKSv4UGK0jyL62k47BuqjHZeOBgJi+J4tDZFD66Mxhvt3z+4KJUHunegM4NPHlj0X6OJqT8/4qQseDoDhunWS44Ia51cgekX4BGvfNf79sZ0s5D4kHzxpVHkn9paA1LngUbOxjwPiv2n+HnrTE8dEsA3Rt7Wzq6CsnGRvHxna1wsrfhqbk7yczOazU5VYV2YyFqMSQds2yQQgAcWQXKBurfmv/6eu2NRwt9W5XkXxr7fjcmZuk1iVPagxfn76FlXXde6NfU0pFVaLXcnZg6PIh9Jy/y0cprWk3tHzY+iLd+ZbnghLjiyCpjWKeLR/7rPeqDnTOc3W/euPJI8i+pzMuw8nWo05qctmN5Zt4usnJy+Wx0axzs5G0ta32b1+Kejr58vf4YGw4nGAur1oaWI2DnbOPrtBCWknbBGIF244Xea9nYQo2mEC/J37ps/AQunYL+U/l2Uwzbjifx1pDmBHjJeH5zmTiwGY1quPLsr7tJupxX36fjo5B12biBRghLidsOOhf8utx8u5rNpeVvVS6cgM2fQYvhHHJsxkcrD9G3WU1GtPWxdGSVirODLZ+Nbs2F1ExeX7jPWFg7CAK6wbavpdyzsJzYraBswSfk5tvVaA6XEyAl3jxxXcMkyV8p1V8pdVApdUQp9XI+6x9QSiUopXbl/YwzxXEtZtUbAGT1fINnf92Fq5MdU+5oKXV7LCCwdlWe7t2YJXtOs3j3KWNhp8eNb2X7ZaJ3YSGxW42GiEMhPQE1Ao1HC9zpW+rkr5SyBb4ABgDNgNFKqWb5bDpPa90q7+fb0h7XYmK3Ghd6Oz/JFzsy2HfyIlOGtcDLVYZ1WsrD3eoTXK8ary/cR/yldGjYBzwbwZbPpeSDML/sTDgZAb6dCt/Wo77xmHS8bGPKhyla/u2BI1rrY1rrTGAuMNQE+y1/tDYu8rrVZl/AWKavOcLtrepIfX4Ls7O14aORwaRl5vDqH/vQSkGnx+D0bojZZOnwRGVzZg9kp0G9DoVv6+4DNvYWGZ5siuRfFzhxze9xectuNFwptUcpNV8pVS+/HSmlxiulwpVS4QkJCSYIzcQOrYC4f8nq+gLPLjiEp6sDbw1pYemoBNCwhisv9GvCqqiz/L7jJASPAmcP2PKFpUMTlU3sVuPRt2Ph29rYQnU/Y05qMzNF8s+vo/vG79qLAX+tdRCwCsh3KIbWeqbWOkRrHeLtXc5uksrNhTX/A4/6fJzYnkNnU3hveBDuLvaWjkzkGdMlgPb+Hry1eD+nU4F24+DgMkg8YunQRGVyMhzcfcGtVtG296hvtS3/OODalrwPcOraDbTW57TWGXm/fgO0NcFxzWv/H3B2H9FBT/P1xlhGtavHrU1qWDoqcQ1bG8UHI4PIztG8/PtedLsHwdYetslNX8KMTu2Euq2Lvr1HfaPP38zXp0yR/LcDjZRSAUopB2AUsOjaDZRS13aKDwHymZapHMvJgrWTya3RjEd2+lHDzYlXbwu0dFQiH36eVXipfxP+OZTAgiPZ0GK4Mcm7hafME5VE2nk4Hw21WxX9NdUDIDMFLieWWVj5KXXy11pnA48DKzCS+q9a6/1KqbeVUkPyNntSKbVfKbUbeBJ4oLTHNatdv0DSMZZ4jSPq7GX+d3sLqjpJd095dW8nf1r7VuPtxZFcaDnW+I+16xdLhyUqg9O7jcc6xWj5V8ub+Ds51vTx3IRJxvlrrZdqrRtrrRtorSfnLZuktV6U9/wVrXVzrXWw1vpWrbXlpq8pruwM+Gcq6TXb8Nzu2twWVJs+zWRylvLM1kYxdXgQKRnZvBFub4y62Pa1zPMryt6pncZj7eCiv6ZqHeMx+eTNtzMxucO3MLt+gYsn+SDzDpwd7HhzcHNLRySKoHFNNx7t0ZCFu06xv95oYzTFEcvNmiQqiVO7oJpfwcXc8uOeVxng4qmbb2dikvxvJicLNn5MonsLvjsdwGu3BUqNfivy6K0NaFjDlQnhdch1rWW0/oUoS6d3Fa/LB8DFE+yc4GJc4duakCT/m9k7Hy7E8saFgXRt6C21e6yMo50tU4cHEXcpm3Vug+Hoakg4ZOmwREWVmmRc7K1TjIu9AEoZXT/S7VNO5OagN3xErEMDVue2Zsowqd1jjdr6Vee+jn68GN2G2XtH4d/cCxsb8PeHsDBLRycqlJJc7L2ial3p9ik3Iheizh3mvZTbeK5PU3w9XSwdkSihF/o3JfloEOMWf0pMggdaQ0wMjB8vHwDChM7sNR5rBRX/tVXrwkVp+Vtebi45/3zAceoSV6sXY7r4WzoiUQqujnZc3BhIRpbTdctTU2HiRAsFJSqe+Chwq128i71XuOe1/HNzTB9XAST55+fQcmwTIvk8awj/G9YKO1t5m6xdwhnbfJfHmndotajI4iP/v0RzcbnVBp0DqedMG9NNSFa7kdakrn6PWO2Na8hdBNerZumIhAn4+hZvuRDFkpsDCQfBu4TJ3zWvVEzKWdPFVAhJ/jfIPbIGl4Td/GR7B8/1k4qdFcXkyeByw2UbFxdjuRCldj7aKONc0pa/a96No5L8LSdh2RROaw9aDJggFTsrkNBQmDkTfH01oPFzj2X61BOEhlo6MlEhXJmJq0Z+81gVwdWWv/mmc5Tkf43kg+upmRTOCvc7GRoSYOlwhImFhkJMjCIi8ggHn2pDkM0blg5JVBTxkcajd5OSvb6KdPtY1MnFU0jSbnS981kZ01+BtQlsxO7qfWh69i+OxJr3rkpRQcVHGWUdHF1L9npHV3BwhRTzTWIlyT/P3oiNNEvZwr56d9PQRwq3VXSNBz9HFZXBpt8+Rcs8v6K04qNK3uVzhWsNafmbW1ZOLonLp3IZZ0JGvmjpcIQZVGsQQnz1Ntya/Ce/R8h4T1EK2ZmQeAhqNC3dflxrSvI3t/kr/6Fb5gbim9yDi7uXpcMRZuLV6yl8bRLYvDSMi+lZlg5HWKuko5CbbaKWv1zwNZszyenYbf2MXGVHwKAXLB2OMCObwEFkVqnDiOzFfPL3YUuHI6zVlYu9JR3meUUV6fYxqxmL1jOUf0hvGQpu0tdfqdja4dDpYTrbRLJ16z8cPHPJ0hEJaxR/AJQteDYq3X6qeEH6BcjJNk1chajUyX97dBL1Dn6PnQK3ns9aOhxhCW3uQ9s5M85+JW8u2i8Xf0XxxUeCZwOwdyp825tx8TQe086XPqYiqLTJPydX89Gfmwm1W0NuyxFQ3c/SIQlLcPFABY9iiM0mDhw7zpK9py0dkbA28VGl7/IBcK5uPKYllX5fRVBpk//c7bH0ODcHJ7Kw6/a8pcMRltRhAna5GTxVbTOTl0SRmmmer92iAshKg6RjJa/pc60rLX8zFXerlMn/Qmom3y/fxhi7vyFoBHg3tnRIwpJqNIUGPbnbZgUJySl8sfaIpSMS1iLhIKBN0/K/Ugpakn/Zmfb3Ie7O+gMHlY3q/rKlwxHlQYdHcEg9y6T6h/lm/XGiEy9bOiJhDeKjjMfSDvOEa1r+0u1TJqJOX2TF1l3cZ78aFTzauFAjRMPe4NmQ0blLsLdVvP1XpKUjEtYgPhJsHcCjfun35Swt/zKjtebNRft51nEhdioXpK9fXGFjA+0fxv7MDia3S2PNgXhWR5lvzLWwUgkHwKsJ2NqVfl8OLmDnLBd8y8KSvadJit7DCFahQsaCh1TuFNdoNRocqzI4bRENvKvw9l+RpGeZb1o9YYVMNdLnChdP6fYxtdTMbKYsiWJKlXkoJzfo8YqlQxLljaMbtL4X2wOLmNLbk5hzqXy38biloxLlVfpFSD5R+po+13KpLsnf1GasO0qTS1tolx2B6v5SySZZFhVfh/Ggc+mQ+Cf9m9di+pojnLqQZumoRHlU2glc8uPiKX3+phR7LpXZ6/fxscss8G4K7R6ydEiivKruD00GQvgPvNbPn1ytmbI0ytJRifLIVDV9ruXsIcnflN5ZEslLtnOolp0IQ6aDnYOlQxLlWYcJkJaET9wSJnRvwF97TrP1mHn+QworEh8F9lXA3dd0+3TxlAu+prL+UAJ2BxYySv2N6vQY1Gtn6ZBEeeffFWq2gK0zeKR7fepWc+bNRfvJzsm1dGSiPImPNPr7bUyYRl08IO0C5Jb9QIMKl/zDwsDf3/h7+Plp3p60lw8dZpJbNwR6yZytogiUgo6PQHwkTrH/8NptgRw4c4lf/pVJX8Q14g+YtssH8m700sYHQBmrUMk/LAzGj4eYGNAaYmMVm//oyLyD92Fz18/S3SOKruVIcK0Fmz+jf4tadGnoyUcrD5F0OdPSkYny4HIiXI43TU2fa5nxRq8KlfwnToTU1OuXpWW58ObGd6FqHcsEJayTnSN0nADH1qFO7+aNwc1Jycjmw5UHLR2ZKA+ulnUwdfLPq+yZbiUtf6VUf6XUQaXUEaXUf4rlKKUclVLz8tZvU0r5m+K4N4ot4Fv5iZMmuPtOVD5tx4CDG2z+nMY13bi/kz9z/o1l38lkS0cmLM2UNX2u5VzNeLSGbh+llC3wBTAAaAaMVkrd+I48CJzXWjcEpgFTS3vc/PgWcNG9oOVC3JRzNWh7P+z/E87H8FTvRni4OPCGTPoi4iPBqRq41TLtfp3cjUcrafm3B45orY9prTOBucDQG7YZCvyY93w+0EsppUxw7OtMngwuLtf/p3RxMZYLUSIdHzUuAG/9Endne17s34SImPMs2HXS0pEJS4qPMlr9pk5jTkbLP/zA8TJvYJgi+dcFTlzze1zesny30VpnA8mA5407UkqNV0qFK6XCExISih1IaChM/igDe/c0lNL4+cHMmcZyIUrEva5x8XfHT5CaxMi29QjycefdpQdIyZBJXyolrfOSvwnLOuRJynUG4Hhc2TcuTJH88/vou/EjqyjboLWeqbUO0VqHeHt7lyiYpyc4kXjGjuxsiI6WxC9MoPMTkJUK27/Dxkbx1pDmxF/KYPoamfSlUrp0GjKSTd/fD3y0+jip2pHeAY6UQefIdUyR/OOAetf87gOcKmgbpZQd4A6U2W1sVZ3ssbEp2zdOVCI1m0PDPvDv15CVRmvf6oxo68N3G49xLCHF0tEJcyuLsg5A5KmLzPwhm6af7sRz+Pv4+xvD18uKKZL/dqCRUipAKeUAjAIW3bDNIuD+vOcjgDVarpgJa9LlSbicALvnAPBi/yY42tnyP5n0pfK5MtLHhGP8tdaMee0M55YHcSK5LlorYmKM+5bK6gOg1Mk/rw//cWAFEAX8qrXer5R6Wyk1JG+z7wBPpdQR4FlA5k4U1sX/FqjTBjZ+AjlZ1HBz4unejVh7MEEmfals4qPAtSZU+c9lyxJbsvc0Eb/7oLNsr1uemmrcv1QWTDLOX2u9VGvdWGvdQGs9OW/ZJK31orzn6VrrkVrrhlrr9lrrY6Y4rhBmoxR0ewEuxMDe3wC4r5P/1UlfMrJl0pdKIz7SpF0+V+YaybnonO/6gu5fKq0KdYevEGWqyQCo2RI2fAS5OTjY2fDG4ObEnEvl2w0y6UulkJtr1PQxYZfPV+uOcio5nVp18i8cWFb3KUnyF6KolILuL8C5I8aNX0C3xt70bVaT6WuOcDpZJn2p8C5EQ3aayVr+sedS+Wr9MYYE1+HD921xcbl+fVnepyTJX4jiaDrYaPWt/8BoBQKv3daMHK15b9kBCwcnyly8aWfvemdJJHY2ilcHBhIaatyX5OdntDPK+j4lSf5CFIeNDXR73pjCL8oY1Obr6cKEbvVZuOsU/x43z0QcwkKuDPP0blLqXa0/lMDKyLM8dmtDark7AUaij4422hVlfZ+SJH8hiqv5MPBseF3r/5EeDanj7sQbi/aTkyujmCus+Chj5i6nqqXaTWZ2Lm8u3o+fpwvjbgkwUXDFI8lfiOKysYVbnoez++DAXwA4O9jy6m2BRJ2+KJO+VGTxUSbp7/9pSzTHEi4zaVAzHO1sC92+LEgHweOZAAAgAElEQVTyF6IkWo4Ez0awdvLVKfdua1mbjvU9+GjlQc7LpC8VT04WJB4qdU2f+EvpfLLqMLc28aZXYE0TBVd8kvyFKAlbO+g50ej73/MrAEop3hzSnEvp2Xz0t0z6UuGcOwq5WaW+2Pv+8oNkZOfw+iDT1wYqDkn+QpRU4FCoHQzrpkC20dJvWqsq93b045dtsew/JZO+VCgmqOmzI/Y88yPiGNs1gPreriYKrGQk+QtRUjY20HMSXIiFHT9eXfxM78ZUc3HgrUWRMulLRZJwAJQNeDUu0ctzczVvLtpPDTdHnujZyMTBFZ8kfyFKo2Ev8OsC/7wPmZcBcHex54V+Tfg3OolFu28scCus1tn9UD0A7PMvw1CY+RFx7IlL5pWBTXF1tPzUspL8hSgNpaDXJLgcD9u+urr4zpB6tKhblXeXHuCyTPpSMcRHQs2S9dMnp2UxdfkB2vpV5/ZWN851ZRmS/IUoLd+O0GQgbJgGKfEA2OZN+nLmYjpfrJVJX6xeZiokHYcazUv08k9XHSYpNZO3hjQv80laikqSvxCm0Od/Rs2XNe9cXdTWz4M7Wtfl2w3HiU68bMHgRKklHAB0iVr+B85c5Mct0Yxq50uLuu4mD62kJPkLYQpeDaH9w8Zcv2f2Xl388oCm2NsqmfTF2l0d6VO8lr/WmtcX7KOqkx0v9it9SQhTkuQvhKl0fwGcq8PyV4xJvoEaVZ14slcjVh+IZ+2BeAsHKErsbCTYOYFH8Uox/L7jJNujz/NS/6ZUr+JQRsGVjCR/IUzFuTrc+ipEb4ADS64uHtMlgPpexqQvmdn512wX5Vx8pFHMzabopRiSU7N4d2kUrX2rcWdIvcJfYGaS/IUwpbZjwLsprHjFuEgIONjZMGlwM44nXub7TTLpi1WKjyx2l88HKw9wPjWTd25vgY1N+bjIey1J/kKYkq0d3PaxcePXP1OvLu7RpAa9A2vw+erDnL2YbsEARbFdPgcpZ4t1sXdP3AXCtsVyXyd/mtcpPxd5ryXJXwhT8+8Cre+BLdONG4PyvD6oGVm5Wi7+Wpv4vL9hEWv65OQaF3m9XB15tm/J7gY2B0n+QpSFPv8DJ3dY/PTVmv9+nlV4rEdD/tpzmnUH5eKv1YiPMh6LmPzn/BvL7rhkXrstkKpO9mUYWOlI8heiLLh4QN/JEPcv7Jh1dfGEHvWp712F1xfuIy0zx3LxiaI7u9+4mO9Wq9BNE1My+GDFQTrV92RIcB0zBFdykvyFKCvBoyCgG6ycBOdjAHC0s+Wd21twIimNz9cctnCAokiuXOwtwp257y0zynm8PbT83MlbEEn+QpQVpWDIdOP5wseudv90buDFHW3qMnP9MQ6dvWTBAEWhcnONbp8iXOzdeuwc8yPiePCWABrVdDNDcKUjyV+IslTdD/q/a4z9v6bw28SBgbg62THxz73kypy/5VfyCchMKbS/Pz0rh1f/3Es9D2ee6mX5cs1FIclfiLLW+h5oPABWvQnxBwDwdHXk1QGBbI8+z28RJywbnyjY1bION0/+X647yrGEy0y+vSUuDpYv11wUkvyFKGtKwZDPwNEN5o+5evPXyBAf2vt7MGXpARJTMiwcpMjXlaG6N5m96/DZS8xYd4RhrevSrbG3mQIrPUn+QpiDaw24YybERxH20lz8/cHWVrFpcgfO7PBmypIoS0co8hMfCe6+4FQ139W5uZqX/9hLFUc7Xrut5NM7WoJ1fD8RoiJo2IuwrFmM/6IvqVnGolNxNtgnBPNT7m6GtUnglkbW03KsFM7shVotClz9y7+xRMSc58ORwXi6OpoxsNKTlr8QZjRx9lBSs1yuW5aVYUPKxkBe/n2vzPpVnmRehsTDUCso39VnL6YzddkBujT0ZHib8jE7V3FI8hfCjGJP5D/2OzPZkVPJaXyw4qCZIxIFOhsJaKjVMt/Vby7aT2ZOLpNvb1nux/TnR5K/EGbk61vQcsX9nfyZtTma7dFJ5g1K5O/MHuOx9n9b/sv3nWHZvjM83bsx/l5VzByYaZQq+SulPJRSfyulDuc9Vi9guxyl1K68n0WlOaYQ1mzyZHC5vtcHF4d0Jv8vmxf6NcGnujMvzd9DepaUfrC4M3vAqRq4X1+LP+lyJq8t2EuLulUZd0vxJncpT0rb8n8ZWK21bgSszvs9P2la61Z5P0NKeUwhrFZoKMycCX5+xghQv1qXmDnoMUKdH6aKveK9O4I4lniZT1ZJ6QeLO7PX6PK5oUvnzUX7SU7L4sORwdjbWm/nSWkjHwr8mPf8R+D2Uu5PiAovNBSio43KAdGn3Qh9pgXsmw+Ln6RrAw9GtavHzPVH2RN3wdKhVl452cYY/xsu9i7fd4ZFu0/xZM9GNK2V//BPa1Ha5F9Ta30aIO+xRgHbOSmlwpVSW5VS8gEhxLW6PgPdX4Kds2HJs7w6oDE13Jx4cf4emfbRUs4dgez06/r7r+3umdCjgQWDM41Cx/krpVYB+dUynViM4/hqrU8ppeoDa5RSe7XWR/M51nhgPIBvQVfGhKiIerwCOVmw8WOqpiUxZchkxs7ew2erD/N8vyaWjq7yuXKx95qRPm/kdff8/GAHq+7uuaLQ5K+17l3QOqXUWaVUba31aaVUbSDfGSq01qfyHo8ppdYBrYH/JH+t9UxgJkBISIhUuxKVh1LQ+w1w8YSVE+mZdp57Wr3Cl+uOcGvTGrT1y3cshSgrZ/aArSN4GTNxLd93msW7T/Fsn8YE1rbu7p4rSvvxtQi4P+/5/cDCGzdQSlVXSjnmPfcCugAyj50Q+en8OAz7GmI281bCk3Sseo7nft0lN3+Z2+k9Rj0fW3sSLmUw8c99NK9TlUcqQHfPFaVN/u8BfZRSh4E+eb+jlApRSn2bt00gEK6U2g2sBd7TWkvyF6IgwaPg3gXYpp3n59xXaHzhHyYvldo/ZqP11ZE+WmtenL+blIxspt3VqkJ091xRqjPRWp/TWvfSWjfKe0zKWx6utR6X93yz1rql1jo47/E7UwQuRIUWcAs8/A+23o2YaT+NxhFvs25ftKWjqhwunoS0JKgdzOytMaw9mMArA5rS2AomaCmOivMxJkRF4+4DY5aT3X4CD9itJGD+AJIPb7Z0VBXfyR0AnHBuzDtLoujW2Jv7O/tbNqYyIMlfiPLM3gm7gVOJGTQXe52BW9hA9OJnIFVKQJSZUzvQNnY8uTabKo52fDgiyCpr9xRGkr8QVsAvZAAru//JrOx+6B0/wudtYft3xvBQYVond3DWqQE7T6Xz7h0tqVHVydIRlQlJ/kJYift6BLG+wXMMyXqXy9UawZJnYXo72D0PcqUWkEnk5pIdt4PVl3wY1a4e/Zrnd4tTxSDJXwgrYWOj+GhkMAkuDRh88RXSR/wCDq7w53j4shPsmgPZmZYO06qdOxGFXdYlzlQJZNLgm8/ba+0k+QthRTxdHfl0VGuik1J5ZX9d9MP/wMhZoGxgwQT4NBg2fQrpyZYO1epk5+Tyy58LABgxZIjVTMReUpL8hbAyHet78lSvxvy58yTzd5yC5sPg0S0QOh+8GsLfk+Dj5rBiIiTHWTpcq/Hp6sNUSdxDto0Tfk3aWjqcMifJXwgr9HjPhnSq78mkhfs5eOaSUR6iUR+4fzGMXweN+8LWGfBJEMx/EE7ttHTI5dr6QwlMX3uEnm5x2NVtBbYVu9UPkvyFsEq2NopPR7XC1cmO8T+Hk5x6zaifOq1hxPfw1C7o+AgcWgEze8CsQXBwuVFLGggLA39/sLExHsPCLHEmlnfqQhr3vXqGM1/3pP7LG/B/dW6leC+U1uWzflpISIgODw+3dBhClGvh0UmM/mYrXRp68d397bC1yWc8enoyRPwI274y7l71akzYhQ8Z/243UlP/f3sXF2OimdBQM56AhaVn5dDpwcPsntsInWV7dbk1vxdKqQitdUhh20nLXwgrFuLvwRuDm7PuYALT/j6U/0ZO7tDlSXhqN9zxLdg7M/HDgOsSP0BqKkwsTqF2K6e15pU/9rJ3oe91iR8qx3shyV8IKxfawZe7Quoxfe0Rlu87XfCGtvYQNBLG/0PsxXr5bhIbW0ZBlkPfbTzOnztPknPJOd/1Ff29sKqrGllZWcTFxZGenm7pUCoNJycnfHx8sLe3t3QoogBKKd4a2pwDZy/x3K+78fOscvOa80rh6wsxMf9d5Vs3E3Aos1jLi01HEpmyNIp+zWuyvF7+ib6izydlVck/Li4ONzc3/P39K2StjfJGa825c+eIi4sjICDA0uGIm3Cyt+Xre9oy9IuNjJ21nQWPdaHmTcoSTJ4M48cb3RtXuNinMrn9U7DcC3q+Bg5VzBC5+R0+e4lHZkfQsIYrH93Zim42ivEP5ZKa9v8dIS4uxntUkVlVt096ejqenp6S+M1EKYWnp6d807IStdyd+P6BdlxMy2LsrO03nQAmNNS4oOnnZ4wS9fODmV/nEjrG1Rgi+lVXOLHdjNGbR/yldB74YTuO9rZ8/0A7XB3tjPfijQj83GNRShvvhZVe7C0Oq0r+gCR+M5P327o0r+PO9LvbEHX6Ik/M2Ul2TsETwIeGQnS0MfIzOhoj8Q+aBg/8ZRSM+74vrJ1SYeoGXc7IZuys7SRdzuT7+9vhU93l6rrQ4D+Ifr4duZlZxntRwRM/WGHyt3YPPPAA8+fPt3QYogK7tWkN3hragjUH4nlz8X6KPZzbvys8sgmC7oJ/pkLYCKsvIZ2dk8sTc3YSeeoi0+9uTUsf9+s3iN0KdduCXcW/3nGFJP9S0FqTm1twy0oIS7m3ox8Pd6/P7K2xfLSygCGgN+PkDsO+gsGfQvRGmNndmNfWCuXmal7+Yy9rDsTz9tAW9Aqsef0GmanGhO31OlgmQAuR5F9M0dHRBAYG8uijj9KmTRt+/vlnOnXqRJs2bRg5ciQpKSkAvP3227Rr144WLVowfvz44re+hCill/s3ZVQ7Ywjo1/8cLdlO2j4AY5YbXT8/DIAjq00aY1nTWvPm4v3Mj4jjqV6NuKej3383OhkBudng29H8AVqQVY32udZbi/cTeeqiSffZrE5V3hjcvNDtDh48yA8//MDbb7/NHXfcwapVq6hSpQpTp07l448/ZtKkSTz++ONMmjQJgHvvvZe//vqLwYMHmzReIW5GKcXkYS1Jycjm3WUHcHWyI7RDPsmvMD5tYdxqCBsJv9wJQ6ZDq9GmD9jEtNZMXX6Qn7bEML5bfZ7u3Sj/DWM2AwrqtTdrfJZmtcnfkvz8/OjYsSN//fUXkZGRdOnSBYDMzEw6deoEwNq1a3n//fdJTU0lKSmJ5s2bS/IXZmdro/j4zlZczsjmtQX7sFGK0e1LMIC9am0YswTm3WuUjk5Lgk6PmT5gE/ps9RG++ucooR18eWVA04IHLxxfD7WDwbm6eQO0MKtN/kVpoZeVKlWM8c9aa/r06cOcOXOuW5+ens6jjz5KeHg49erV480335ThksJiHOxsmHFPWx7+OYJX/thLZnZuySYkd3I3ykb/MQ5WvAo6Fzo/YfJ4S0trzQcrDvLluqMMb+PD/4a2KDjxZ6ZC3L/QYYJ5gywHpM+/FDp27MimTZs4cuQIAKmpqRw6dOhqovfy8iIlJUVG9wiLc7K3ZeZ9benTrCZvLNrPzPUlvAZg5wDDvzPmEFj5mjFxTDmiteatxZF8ue4od3fw5YMRQdjkV+zuihPbICcTArqbL8hywmpb/uWBt7c3s2bNYvTo0WRkZADwzjvv0LhxYx566CFatmyJv78/7dq1s3CkQoCjnS1fhrbh6Xm7mLL0ABfTsnmub+Pi38tha28UiEMZE8fYu0D7h8ok5uLIysll4p97+TU8jge7BvDabYGFn9vx9WBjV+ku9oKVlXSOiooiMDDQQhFVXvK+VyzZOblM/HMf88JPMKx1XaYOD8LBrgSdADnZ8Ou9cHAZjPgOWgw3fbBFdCk9i0fDdrDhcCJP9mrEM70bFe1D7ZteRvJ/cEXZB2kmUtJZCJEvO1sb3hvekuf6GFNB3vf9tusngykqWztj0hjfTvDHw3B0jemDLYJTF9IY+dUWthw9x/vDg3i2TxG/zaRfhFM7IKBb2QdZDknyF6ISUkrxRK9GTLsrmIiY8wz5YmPJhk7bO8PoOeDdBObeA6d2mT7Ym9hy9BxDpm/i5Pk0Zo1pz53t8i9Vna/j/xgXretXvv5+kOQvRKU2rLUPcx7qSHpWDsO+3MT8iBJM+O5cDe753RgqOWc0XLzJnAImorVmxrqjhH67larOdvzxaGe6NvIq3k4OrQBH90p3Z+8VkvyFqORC/D3464lbaO1bjed/283zv+3mYnoxu4HcasHdc40pI+eONoZQlpGESxk89FM4U5cfYEDL2ix6vCuNaroVbydaw+G/ocGtxgXsSkiSvxACbzdHZj/YgcdvbcgfO+LoP209Gw8nFm8ntVrC8G+Nrp8Fj1ydKN6Uluw5Td9p/7D+cCJvDG7G9NGtcXUswaDF07sh5Qw07mfyGK2FJH8hBGBcCH6+XxN+f6QzTg623PPdNl6av4fElIyi76TpQOjzFkQugPXvmyy2E0mpjP8pnMd+2YGvhwtLn+zKmC4BJS85fvhv47Fhb5PFaG0k+RfTZ599RmBgIKGhoSxatIj33nsPgAULFhAZGXl1u1mzZnHq1Kli7Ts6OpoWLVqYNF4hiqu1b3WWPnkL47vV5/cdcdz64Tq+3XCMzOwituQ7PwnBo2Hdu0a/eimkZGTz8cqD9Pr4HzYcTuTF/saHU8MaxezmudHhFVCnDbjWKN1+rJjc5FVMX375JcuWLbs6reGQIUMAI/kPGjSIZs2aAUbyb9GiBXXq1LFYrADZ2dnY2cmfWRSPk70trw4M5M4QH97+K4p3lkTx3cbjPNKjAXeG1MPJ3rbgFytlTApzdh/88RA8tBY8GxTr+JfSs/hpSwzfbDjGhdQshraqw8sDmlLbPf/J1ou387MQFw49Xi79vqxYqVr+SqmRSqn9SqlcpVSBNxUopforpQ4qpY4opaz2HZ8wYQLHjh1jyJAhTJs2jVmzZvH444+zefNmFi1axAsvvECrVq2YOnUq4eHhhIaG0qpVK9LS0oiIiKB79+60bduWfv36cfq0MSIiIiKC4OBgOnXqxBdffFHgsd9//31atmxJcHAwL79svIU9evTgyo1wiYmJ+Pv7A8YHz8iRIxk8eDB9+/blrrvuYunSpVf39cADD/D777+Tk5PDCy+8QLt27QgKCuLrr78uo3dOWKuGNdz4cUw7fhrbnrrVnJm0cD9dp67lgxUHOJF0k4u69s5w12xAGcXgMi8X6XgHz1zizUX76fzeGj5YcZA2vtVZ8FgXPh3V2jSJHyBqEaAhcIhp9melStsk3AfcARSYNZRStsAXQB8gDtiulFqktY4s6DVFsuxlOLO3VLv4j1otYcB7Ba7+6quvWL58OWvXrsXLy4tZs2YB0LlzZ4YMGcKgQYMYMWKEEd6yZXz44YeEhISQlZXFE088wcKFC/H29mbevHlMnDiR77//njFjxvD555/TvXt3XnjhhfxPddkyFixYwLZt23BxcSEpqfBZlbZs2cKePXvw8PDgzz//ZN68eQwcOJDMzExWr17NjBkz+O6773B3d2f79u1kZGTQpUsX+vbtK5O1i+sopejW2JtbGnmx5dg5vttwnBnrjvLF2qN0CPCgT7Oa9AqsSYDXDRO+V/c37vydPQIWPwV3fGN8K7iG1prD8Sn8HXmWlfvPsDsuGQdbG/q3qMVDt9T/74xbphC5ELwaQ43Kfdd6qZK/1joKCp3ntT1wRGt9LG/bucBQoHTJ34ocPHiQffv20adPHwBycnKoXbs2ycnJXLhwge7djZtM7r33XpYtW/af169atYoxY8bg4mLMOerh4VHoMfv06XN1uwEDBvDkk0+SkZHB8uXL6datG87OzqxcuZI9e/ZcLTyXnJzM4cOHJfmLfCml6NzAi84NvDidnMZv4XEs2XOad5YY3UK1qjoR5ONOi7ru1PNwpm41F6q6tqdmhxeovu19Trs1J7rBfcSdTyU2KZX9py6yM/Y85/PuLg72cWfiwECGt/XBo0oZTaeYkgAxm+CW5//zQVTZmKMzuC5w4prf44DS31VxkxZ6eaO1pnnz5mzZsuW65RcuXCjSaAWtdb7b2dnZXZ1G8saS0VfKTgM4OTnRo0cPVqxYwbx58xg9evTV/X7++ef061d5h7uJkqnt7syTvRrxZK9GnEhKZe3BeCJizrMnLpmVkWev21YRxEz7NvTY9D+eXJvLdt0UGwUBXlXo06wmbf2q06NJDWpWdSr7wKMWGXf1Nr+97I9VzhWa/JVSq4Ba+ayaqLVeWIRj5Jfd8q0mp5QaD4wH8PUtwYQTFuTm5salS5fy/b1JkyYkJCSwZcsWOnXqRFZWFocOHaJ58+a4u7uzceNGunbtSlhYWL777tu3L2+//TZ333331W4fDw8P/P39iYiIoH379oWWjR41ahTffvst4eHhV7ur+vXrx4wZM+jZsyf29vYcOnSIunXrXvfBIURh6nm4cF8nf+7r5A9AelYOJy+kcfJ8GikZ2aRn5XA580syN44iLPtLzoxaSS2fgJIVkyut3XPBqwnUaGb+Y5czhb77WuveWusW+fwUJfGD0dK/tuCGD5DvGEit9UytdYjWOsTb27uIuy8fRo0axQcffEDr1q05evQoDzzwABMmTKBVq1bk5OQwf/58XnrpJYKDg2nVqhWbN28G4IcffuCxxx6jU6dOODvnf0Grf//+DBkyhJCQEFq1asWHH34IwPPPP8+MGTPo3LkziYk3vyGnb9++rF+/nt69e+PgYHylHjduHM2aNaNNmza0aNGChx9+mOzsbBO+K6IycrK3pYG3K90aezOwZW3uaOPD7R0DqXLvHBxyUvFd/QgOWODfWcIhY+KW1qGVvssHTFTSWSm1Dnheax2ezzo74BDQCzgJbAfu1lrvv9k+paRz+SHvuzCZfb/D/LHQfjwM/MC8x/77Ddj8OTwbBW41zXtsMzJLSWel1DClVBzQCViilFqRt7yOUmopgNY6G3gcWAFEAb8WlviFEBVUi+HQ6XH4d6bRBWMuOdmwZx406lOhE39xlHa0z5/An/ksPwUMvOb3pcDSG7cTQlRCvd8y6v8sfsroe68dVPbHPLAYLp2G2z4q+2NZCSnvIIQwL1s7GPkDOHvAvHsgtfD7Vkpt6wzjvoPG/cv+WFZCkr8Qwvxca8CdPxG2oSP+vtnY2Gj8/aGAAW+lExdhTNTe4RGwuUlZikpGir4IISwibH07xi9tRWq6UU8/JgbGjzfWhYaa8ECbpoFjVWOUj7hKWv5CCIuYOJGrif+K1FRjucmc3AFRi6HTY+BYykqgFYwk/zI2adIkVq1aZekwhCh3YmOLt7xE1rxjXFvo+KgJd1oxVOjkHxYG/v5gY0PZ9ScW4u2336Z378o7YYQQBSnoJn7fupmmOcDhv+Hoauj6DDhVNc0+K5AKm/zDwoz+w5gYY7rOK/2Jpf0AiI6OJjAwkIceeojmzZvTt29f0tLS2LVrFx07diQoKIhhw4Zx/vx5wCiffKX0wssvv0yzZs0ICgri+eefByAhIYHhw4fTrl072rVrx6ZNm0oXoBBWYvJkyKtVeJWLfRqTu74E8QdKt/PMVFjyrFG9s8PDpdtXRaW1Lpc/bdu21TeKjIz8z7KC+PlpbaT963/8/Iq8i3wdP35c29ra6p07d2qttR45cqT++eefdcuWLfW6deu01lq//vrr+qmnntJaa33//ffr3377TZ87d043btxY5+bmaq21Pn/+vNZa69GjR+sNGzZorbWOiYnRTZs2LV2AZaA477sQxTF7tvF/UinjcfaMU1p/0Ejr9xtqHX+w5Dte/LTWb1TV+vgGU4VqNYBwXYQcW2FH+5Rlf2JAQACtWrUCoG3bthw9evS60sz3338/I0eOvO41VatWxcnJiXHjxnHbbbcxaNAgwCjXfO30jxcvXuTSpUu4ucnFKVHxhYbeOLKnNiQshlmD4If+MHou1GtfvJ3u+Q3Cv4cuT4F/V1OGW6FU2G6fAvsTTVAs1NHR8epzW1tbLly4UOhr7Ozs+Pfffxk+fDgLFiygf3/jZpPc3Fy2bNnCrl272LVrFydPnpTELyo37yYwdjk4uRsfArvmGF/ci+LIaljwCPh1gZ6vl22cVq7CJv98+xNdjOWm5u7uTvXq1dmwYQMAP//889VvAVekpKSQnJzMwIED+eSTT9i1axdgVNucPn361e2uLBeiUvNsAA+uAp8QWDAB5o8xJmIpiNawMwx+uQu8m8KoX8DWvuDtRcXt9rnyVXLiRKOrx9fXSPwmvXnkGj/++CMTJkwgNTWV+vXr88MPP1y3/tKlSwwdOpT09HS01kybNg2Azz77jMcee4ygoCCys7Pp1q0bX331VdkEKYQ1qeIJ9y+GjdNg3btwaCW0fwiCRxkJXinIzYHYrcY2R/6GgG5w50/gXM3S0Zd7JinpXBakpHP5Ie+7sLjEw8aY/SszcTm5g3N149tA1mVwdIfuL0KHCUbtoEqsqCWdK/e7JISwDl6N4M4fISUeDvwFZ/ZBZorxAVA3BJreBg4uhe9HXCXJXwhhPVxrQMhYS0dRIVTYC75CCCEKZnXJv7xeo6io5P0WomKyquTv5OTEuXPnJCGZidaac+fO4eTkZOlQhBAmZlV9/j4+PsTFxZGQcJPxvsKknJyc8PHxsXQYQggTs6rkb29vT0BAgKXDEEIIq2dV3T5CCCFMQ5K/EEJUQpL8hRCiEiq35R2UUglATCl24QUkmigca1HZzrmynS/IOVcWpTlnP621d2EbldvkX1pKqfCi1LeoSCrbOVe28wU558rCHOcs3T5CCFEJSfIXQohKqCIn/5mWDsACKts5V7bzBTnnyqLMz7nC9vkLIYQoWEVu+QshhCiAVRRtUxoAAAPTSURBVCd/pVR/pdRBpdQRpdTL+ax3VErNy1u/TSnlb/4oTasI5/ysUipSKbVHKbVaKeVniThNqbBzvma7EUoprZSy+pEhRTlnpdSdeX/r/UqpX8wdo6kV4d+2r1JqrVJqZ96/74GWiNNUlFLfK6XilVL7ClivlFKf5b0fe5RSbUwagNbaKn8AW+AoUB9wAHYDzW7Y5lHgq7zno4B5lo7bDOd8K+CS9/yRynDOedu5AeuBrUCIpeM2w9+5EbATqJ73ew1Lx22Gc54JPJL3vBkQbem4S3nO3YA2wL4C1g8ElgEK6AhsM+Xxrbnl3x44orU+prXOBOYCQ2/YZijwY97z+UAvpZQyY4ymVug5a63Xaq1T837dClh7Sc6i/J0B/ge8D6SbM7gyUpRzfgj4Qmt9HkBrHW/mGE2tKOesgap5z92BU2aMz+S01uuBpJtsMhT4SRu2AtWUUrVNdXxrTv51gRPX/B6XtyzfbbTW2UAy4GmW6MpGUc75Wg9itBysWaHnrJRqDdTTWv9lzsDKUFH+zo2BxkqpTUqprUqp/maLrmwU5ZzfBO5RSsUBS4EnzBOaxRT3/3uxWFVJ5xvk14K/cehSUbaxJkU+H6XUPUAI0L1MIyp7Nz1npZQNMA34v/bu3sWJKArj8O+FVSy0S6mwFhbC/gHaCYqFRSoLbXTF1kbEykKwFXtFFMFC0EbTbSOChYXbKiwsKotgJbiNIH68FncQUTCXfDqZ96kSGJJzksnJvecMyeqsApqBmvd5idL6OULZ3T2XtGL705Rjm5aanE8D92zfkHQYuN/k/GP64c3FVOtXm1f+74F9v93fy9/bwF/HSFqibBX/tc3639XkjKRjwBWgb/vLjGKblmE57wFWgGeS3lF6o4OWD31rz+0ntr/afgtsUL4M2qom5/PAQwDbL4BdlN/AWVRVn/dRtbn4vwQOSNovaSdloDv445gBcLa5fRJ46maS0lJDc25aILcohb/tfWAYkrPtbds928u2lylzjr7t9fmEOxE15/ZjynAfST1KG+jNTKOcrJqct4CjAJIOUor/Iv+t3wA401z1cwjYtv1hUg/e2raP7W+SLgBrlCsF7tp+JekasG57ANyhbA03KSv+U/OLeHyVOV8HdgOPmtn2lu3+3IIeU2XOC6Uy5zXguKTXwHfgsu2P84t6PJU5XwJuS7pIaX+stnkxJ+kBpW3Xa+YYV4EdALZvUuYaJ4BN4DNwbqLP3+LXLiIiRtTmtk9ERIwoxT8iooNS/CMiOijFPyKig1L8IyI6KMU/IqKDUvwjIjooxT8iooN+Aq9eY5x0J4CWAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# M=9\n", "p_lsq_9 = fitting(M=9)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "当M=9时，多项式曲线通过了每个数据点，但是造成了过拟合" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 正则化" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "结果显示过拟合， 引入正则化项(regularizer)，降低过拟合\n", "\n", "$Q(x)=\\sum_{i=1}^n(h(x_i)-y_i)^2+\\lambda||w||^2$。\n", "\n", "回归问题中，损失函数是平方损失，正则化可以是参数向量的L2范数,也可以是L1范数。\n", "\n", "- L1: regularization\\*abs(p)\n", "\n", "- L2: 0.5 \\* regularization \\* np.square(p)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "regularization = 0.0001\n", "\n", "\n", "def residuals_func_regularization(p, x, y):\n", " ret = fit_func(p, x) - y\n", " ret = np.append(ret,\n", " np.sqrt(0.5 * regularization * np.square(p))) # L2范数作为正则化项\n", " return ret" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# 最小二乘法,加正则化项\n", "p_init = np.random.rand(9 + 1)\n", "p_lsq_regularization = leastsq(\n", " residuals_func_regularization, p_init, args=(x, y))" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XdcVfUfx/HXuewlMgVlOREHOEA0t+ZeDW1IQ82faWpDM0flqFyVM620NC3JLM2RO/dEBQcKyFARkS3KkM09vz8OmRamyIXD+D4fDx8Hzj333Dfw8HO/93u+5/uVZFlGEARBqF40agcQBEEQyp8o/oIgCNWQKP6CIAjVkCj+giAI1ZAo/oIgCNWQKP6CIAjVkCj+giAI1ZAo/oIgCNWQKP6CIAjVkL7aAR7G1tZWdnNzUzuGIAhCpRIUFJQiy7Ldo46rsMXfzc2NwMBAtWMIgiBUKpIkXX+c40S3jyAIQjUkir8gCEI1JIq/IAhCNSSKvyAIQjUkir8gCEI1JIq/IAhCNSSKvyAIQjVUYcf5CxVEZjKkhEPqNci5A3l3Qd8YjMyhphvYuYOlE0iS2kkFQSgBUfyFB+VlQeQeuHIArh6GO49xv0iNOtCgOzQZBPW6gUZ8oBSEik4UfwFkGW6chsDVcHk75GWCsSW4dYQ2o8DeA2zqg4k1GJpBQS7kpEHqVUgKheijELIVzv4IVm7gOxq8R4C+kdo/mSAIDyHJsqx2hmJ5e3vLYnqHMqYthJDNcHI5xJ0FI0toMhA8XwDX9qDRe/xzFeRB2DY4swpiToClC3SfDs0Hiy4hQShHkiQFybLs/ajjxOfz6kiWIewP+OYp2PSG0orv+yVMDINBy6Bup5IVfgB9Q6XQj9gFr24BUyv4fSSsfxn/VXdwc1N6g9zcwN+/LH4oQRBKQnT7VDc3zsDuyXAzCGwawuAfoMkzuu2nr99VeQM59S3+Cy8yaqshWfnKQ9evw6hRytd+frp7SUEQSka0/Kswf3/+bnG7FOI/4QdY9TSkx8Og5fBWADR7rmwu0Gr0oN1YPgz4iqx80wceysqCDz/U/UsKgvD4RJ9/FeXvr7Sws7L+3mdokMUzY76h/nMJSHqGaCQNRnpG2JrYYmdih5OFE+5W7tQ0rqmzHBqN0sv0T5IEWq3OXkYQhCKP2+cvun2qmNs5tzl68yhjJ3YkK8vqgcfy8k3Z8vNQunQchiRJFMqF5BTkcCf3zgPH1TKtRUv7ljxV+yna1W6Hg5nDE+dxcVG6ev6131kGxIVgQVCLKP5VQE5BDnuv72XntZ0ExAVQKBeSlnih2GPzbzmyZ/CeB/cV5pOcncz19OuEp4YTlhrGmYQz7I7eDYCnnSf96vajd93eWBtblyjb7Nn//gRiapDF7CG/g+wnRgIJgkp00u0jSdJqoD+QJMtys2Iel4AlQF8gCxgmy/LZ/zqn6PZ5tIS7Cfxy+Rc2Rm4kLTeNOuZ16F27Az0ij9P37RXEpLn86zmurhAd/ehzy7JM1J0ojsQeYee1nUTcjkBfo08ftz680uQVmtg0eeyc/v5KH39MjPJJYPZLv+FnPBI6fQDdROe/IOjS43b76Kr4dwIygR8fUvz7AuNRir8vsESWZd//Oqco/g+Xkp3C9xe/59fwXymUC+nq3BU/Dz+87yQjbR0LBTn4a/0ZNa8rWVl/t6xNTWHlyicbZRN5O5KNERvZErWFrIIs2ji0YXzL8bSwb1Hyk8kybBsP536CfgvB542Sn0MQhGKVa/EvekE3YPtDiv8K4JAsy+uLvg8HusiyHP+w84ni/2+5hbmsvriaH0J+IK8wj2caPMPI5iNxMrGDP6fDqW/BwVMZvmnb4N8t7tmlH16ZkZfB75G/s/rSalJzUuni1IXxrcbTyKpRyU5UWAC/vKxMI/HaVnDrULpggiAAFa/4bwfmybJ8rOj7/cBkWZYfWt1F8X/QiZsnmH1qNjEZMfR07cn4luNxs3SDlEjYOBwSLoLvGOgxq9hpFTJy8gm8fpvQuHSikjK5lnKX1Lt53MnKI79QRl8jYWSgwdbcCPsaxtSzNaOJYw2aO1niXssCjebBvvms/Cx+vvwzqy+tJis/i6EeQxnbYixmBmaP/0PlpMF33SE7FUYdgpr/7qYSBKFkKlrx3wHM/Ufx/0CW5aB/HDcKGAXg4uLS+npxw0Sqmbv5d5l/ej6bozbjWsOVab7TeKr2U8qDF36B7ROUYv/M1+De54HnxtzKYsfFeHaHJHAx9g7aoj91bUtj6tmZY2NuSE0TAwz1NRRqIaegkOSMXBLTc4hKyiQrrxAAW3NDOjSwpXczR7o2tsNI/++7f9Ny01hydgkbIzZiZ2LHFN8p9HDt8fg/YEokfNdNmRNo5D4xH5AglFJFK/6i2+cJXEi+wNSjU7mZeZMRzUYw2ms0RnpGysRquyZD0A/KHDzPfQeWdQDIK9Cy61I8a05Ecy5GGcLp5VyTzg1t8a1ng5dzTcyNHj3IS6uVuZ6aRdD12xyLTOZoZAq37uZhaWLAAC9Hhj1Vlwb25veOD04O5rOAzwhLDaN/vf5M9Z1KDcMaj/eDXt6pdAG1fQt6zy35L0oQhHsqWvHvB4zj7wu+S2VZbvNf56vOxV+WZfzD/Pky8EtqmdZiTsc5tK7VWnkwLRZ+fU2ZnqH9O9BtOujpk51XyNqT0aw6do3kjFzq2Zrxoo8zfZs74mxt+p+v9zgKCrUci0ph87mb7L6UQG6Blqc97BnTpT6tXZXhn/nafL4P/p4VwSuwM7VjToc5+Dj4PN4L7JwEp1eC30ZoWIJPDoIgPKC8R/usB7oAtkAiMAMwAJBl+duioZ7LgN4oQz2H/1d/P1Tf4p9TkMOnAZ+y7co2ujp3ZXaH2VgYWigPXj2s9O8X5CrdPE0GkV+o5ZczN/hqfyRJGbl0bGjLiA516dzQ7l/99LqSkpnLjyev89PJaG5n5dOzSS0m92lMfTvlk0BwcjDTjk3jRsYNxrcczxvN3kB61Hj+/Byl+yczEcacAItaZZJdEKq6cm/561p1LP7JWcmMPzCekFshvNXiLd70fBONVDQ/wvElsH+WMhnbi+vArhGB0al8uPkS4YkZ+LhZMalXY9rULdlNWKWRlVfA6mPX+ObQFXIKtLzWzpX3e7pjZqRPVn4WM07MYHf0bro5d+OzDp/9/Sb2MElhsKIzNOqp/IyCIJSYKP6VTHRaNKP3jSY1J5X5HefT1aWr8kB+NmwdC5c2KbNvDlpGhmzMnJ1hrD99g9qWxswY2JSeTWo9unVdRpIzcln4ZwTrT8dQp6YJc55rTudGdsiyzLqwdSwIXICzhTNfd/8a5xrO/32yowuVN7kXflRWBhMEoURE8a9ELqVc4q19bwHw9dNf08y26LJJRgKsfxnizikLo3R4j6CY27y74Tw3b2czsmM93uneELPHuIBbHgKjU5m8KZgryXcZ6uvC9P5NMDbQIzAhkHcPvYsGDUu7Lf3vG8MK85Xun4wEGHsKTMvvk4wgVAViMZdK4kzCGUbsGYGpgSk/9f3p78IffwFWdoXky/DiOgrbv8fi/ZG8sCIAgN9Gt2NaX48KU/gBvN2s2flOR97sXI+fT8UwaNlxIhMz8Hbwxr+vP+aG5ryx5w32Ru99+En0DJTpprNTYY+Y+kEQyooo/io6k3CGsfvH4mjmyE99fsK1hqvyQOg2WN0bJA2M2EOaay9GrDnD4n2RDPSqzc63O94bYVPRGOnrMbWPB2tHtCElM5cBy46x5dxNXGu4sq7vOjxsPJh4eCLrL69/+EkcPaH9u3DhZ7h2tPzCC0I1Ioq/SgITAu8V/lW9VmFnaqc8cGqFMpTTvgn87wCRmroMWn6ME1dSmPNscxa92AILYwN1wz+Gzo3s2PVORzydavLuhvN8secyNQ2t+L7n93Rx7sKcU3NYfWn1w0/Q6X1lHeBdk5WpIARB0ClR/FVwIfkCb+1/CwczB1b1WoWtia0yomffLNj1Abj3hWHbOZqg4dmvT5CZW8jP/2vLUN/KNf2BfQ1j1r3hy0s+ziw/eIUx/kFotfos7LKQPm59WBS0iGXnllHsdScDE+g9B5JCIHBV+YcXhCpOFP9ydvXOVcbuH4utiS2rehYV/sIC2DoOji2E1sPghR/ZGpLKiDVncLIyYdu49vi4Vcxunkcx1Ncw97nmTO/fhD9DExn63Skys2XmdpzLsw2eZUXwChYFLSr+DaBxf6jXFQ7Mhszk8g8vCFWYKP7lKPFuIm/uexM9SY8VTyt3wZKfAxtegfProPMU6L+Y70/E8M4v52nlYsWGN9tRu6aJ2tFLRZIkRnSoy7evtCY0Pp0XVpwkOSOfmU/N5EX3F/kh5Ae+ufBNcU+EPp9D/l1l+KcgCDojin85Sc9LZ/S+0WTkZfDt098q493zs5U5bSJ2Qb8F0HUqSw9E8dmOMPo0c2DtiDZYmlT8/v3H1bOpA2uHtyE+LYfnvzlBzK1spvlO49kGz/LNhW+KvwZg1wh8R8N5f0gMLf/QglBFieJfDgq0BUw6PIno9GgWd12Mh40H5GXBzy/ClYPK0EafkSzeF8HCPyN4vpUTy4a2wthA79Enr2Ta1bdh/f/akp1fyMvfBRCbmsOMdjPoU1e5BvBz2M//flLHiWBkAftmlnteQaiqRPEvBwuDFnIi7gQf+X5EW8e2kHcXfn4Boo/Cs99Cy1dYvC+CxfsiGdzaic8He6JXRvPyVATNnSzxH+l77w0g7k4uszvMpptzN+aenssfV/548Amm1tBhAkTugehj6oQWhCpGFP8ytjlyMz+F/oSfhx/PN3pemZRtwytw/bgyFbPXSyw7EHmv8M9/vmoX/r94ONZg3Ru+ZOTkM/T7AJLSC/ii8xf4Ovoy/fh0TsSdePAJvm9CjTrKimUV9K50QahMRPEvQ+eSzvFJwCe0c2zH+97vg7YQfh+lLF04YCk0H4z/qet8uTeCZ1vWqTaF/y/N6liybqQvd7Ly8fsugPRsmUVdFlGvZj0mHJrA5dTLfx9sYAJdP1Smsg7dol5oQagiRPEvI7eybzHx0ERqm9Xmi85foC/pwfb3lMLVcza0epVdF+P5eMslujW2r/JdPQ/j6VSTNcPbkJCew4g1Z9DIJnzd/WssDC0Ys28McZlxfx/s9RLYN4X9n4obvwShlETxLwOF2kKmHJ1Cel46C7ssxNLIEg5/DmfXQsf34alxnLiSwju/nKelixXLh7bCQK/6/ilauyq/g5C4dMb4n8Xa2I5vun9DbmEuY/aNISMvQzlQowfdPoTUK3DxV3VDC0IlV30rThlaeXElAfEBTPOdhru1uzId86E54DUUun1EWHw6o34Mws3WlFWve2NiWPVG9ZRUd49azHm2GUcikpm8MZh6lvVZ0nUJMekxTDoyiUKtsp4w7n3BwVN5MxWtf0F4YqL461hAfABzvo4mZvIRBrs/i5tTLv6z/gSXp2DAYpIz8xi5NhBzI33WjmhDTVNDtSNXGC/6uDCxRyN+P3eTpQci8XHwYVrbaRy/eZxFQYuUgyQJukyF29cg+Bd1AwtCJVZx5gOuAm5l32L4nD+JXzuLwlxjAK7fNGLU1kXQp4DnZX1G/RRA6t08fhvdDkfLyn3nblkY160B0beyWLwvkob2FgzxHELk7UjWhq6loVVDBjUYBO59wLGF0vr3fFGZBloQhBIRLX8dkWWZmSdncmX9G/cK/1+y8k2YNsucyZuCORdzh0UvetGsjqVKSSs2SZKY81wzWrnUZOJv57l0M40PfD7A19GXWSdncT7p/N+t/zvX4cJ/TA0tCBWJLEPSZchKVTsJIIq/zmyO2syhG4fIT3Us9vGYG7D1fByTernTu1nxxwgKI309VrzqjY2ZESPXBpKaWcCCzgtwMHPg3YPvkpyVDI16Qe2WcOQLKMhTO7Ig/LeCPPjFD772hUXNlDv7VSaKvw7cSL/BvNPzaOPQBpeHLFGrZ5HNMy1q81aX+uUbrpKyszDiu9e8ScvOZ9RPQRjrmbO061KyCrKYdGQSBXJhUes/Bi7+pnZcQfhvh+ZC+A5lqpKaLsr9PjlpqkYSxb+UCrWFTDs2DX1Jn8/af8acD1MwNch64BjJoJCmA2OY+5ynaousV0ZNatdg4QtenL9xh9k7wmhg1YDp7aYTlBjE0rNLoWFPqNUMji8GrVbtuIJQvPR4OPEVeL2srMX9zHK4mwRnf1Q1lij+pfRj6I+cTz7PtLbTcDS1x0//VVY+8wGuTgVIkoyxVQ51Blxi22JnMaTzCfRp7sjIDnX58eR1tp6/Sf96/e9NA73/xgHo8B6kRCitKkGoiE6vBG0BdJ6sfF+ntTL6L2itqlOViOJfCjHpMSw/v5xuzt3oV7ef8u4ecxK/ae2JvqHPpF8vUmvUftZ85oCrjZnacSutyX0a4+NmxZRNF4lIzOADnw9oZtOMj459RIxzK7Byg6MLxZw/QsVTmA9Ba6BxP7Cu+/f+5s/DrUhIvvzQp5Y1UfyfkCzLzDo5C0ONIR+2/RAp8RIc+Aw8BoDXS2w4E8OGwBuM69qA7h611I5bqRnoaVg2tBVmRvqMXhdEXoGGBV0WoKfR470jk8hp+xbEnYVrR9SOKggPunoIslOh5SsP7nfvp2wj/yz3SH8Rxf8J/R75O6cTTjPBewL2RlaweTSYWEH/JYQnZjJ9awgdGtjyXo9GaketEmrVMOarl1sSnXKXKZuCcTRzZG6HuUTcjuDLwgQwr6UsgykIFcnFjWBsCfW7Pbi/hiNY14eYk+rkQhT/J5KUlcSCwAX4OPjwfMPn4eQySLwE/ReSY1iT8evPYmFswKIXW1TLydrKSrv6Nkzs6c724Hh+C4ylo1NHXmvyGhsiN3KgeT+llXXzrNoxBUGhLYSI3UorX9/o34+7tlOKv0qDFUTxfwJzTs0hT5vHzHYzkW5Hw6H5ymLjHgOYvSOMiMRMFrzghZ1FMX9woVTGdK7PU/VtmLEthCvJmbzT6h08rD2YnnqaRJOacGyR2hEFQXHzLOTcgYZPF/+4y1OQfRtSwss3VxFR/Evo8I3D7I/ZzxivMbhYOMOOCaDRhz6fsyckgZ8CrvO/jnXp3MhO7ahVkkYjsfCFFhgbaHjnl3Mg6zO/03zytPlMc6lPYdgfkHpV7ZiCAFH7QNJAva7FP+7cRtmq9GlVFP8SyC3MZd7pedSzrMdrTV9TZuu8cgC6TydOtuaDjcE0r2PJpF6N1Y5apTlYGjP/eU8u3Uxnwd5w6lrWZWqbqZzOS+aHmpYQ8K3aEQVBKf51WivLkBbHuh7om0BiSPnmKiKKfwmsvrSa2MxYpvlOw6AgD/Z+DLVbUth6BO9tOE9+oZalL7fEUF/8Wstaz6YOvNLWhRVHrnI0MplnGjxDL7deLLey5OKlX5SP04Kgluw7ygi0f17ovZ9GD+wbQ5Io/hVabEYsqy6uordbb3wdfeHYYsiIg97z+f74dU5dS2XWwKbUtRXj+cvLh32b0NDenAm/XuB2Vj7T203H3tiGydamZJ35Xu14QnUWewZkLbi2/+/jajUVLf+Kbv6Z+WgkDRO9J8KdG3BiKTR7ngijJizYG0HPJrUY3NpJ7ZjViomhHktfbsmdrDw+3nqJGoY1+Kzz58QaGLAodI1yg40gqCEmACQ9cPL+7+Psm8LdZMhMKp9c99FJ8ZckqbckSeGSJEVJkjSlmMeHSZKULEnS+aJ/I3XxuuXlSOwRDt04xGiv0TiYOcC+GQDkd5vBhF/PY26sz5znmot5e1Tg4ViDd59uxI7geP64EIePgw+vOHbiFxMNJ0+Kcf+CSmICwNETDB/RE2DvoWxVuNO31MVfkiQ9YDnQB2gCvCxJUpNiDt0gy3KLon+V5jN5fmE+n5/5HLcabrzq8aryR720CZ56m+Vnc7l0M505zzbD1lwM61TLm53q4eVck4+3XiIpI4e3u3xB3UKJj6P8ychNVzueUN0U5MHNIHBp9+hjresp29RrZZupGLpo+bcBomRZvirLch7wCzBIB+etEH6N+JXr6deZ5DMJA42+cpHXwpFLdUew7EAUz7SoLebnV5m+noYFQ7zIzitk2u+XMDIwYXaDl0hBy7yDE9WOJ1Q3CcFQkA3Ovo8+1tIJNAaqDE/WRfGvA9y47/vYon3/9LwkScGSJG2UJKnYWe8lSRolSVKgJEmBycnJOohWOul56Xx74Vt8HXzpWKcjROyB2NPkd5jEhC0R2JgbMmtgM7VjCkADe3Mm9XJnX1gim87epHm793jjbj7bEgM4EHNA7XhCdRIToGxd2j76WI0eWLkqa1KXM10U/+I6uv85veIfgJssy57APmBtcSeSZXmlLMvesix729mpf5PU98Hfk5abxvs+7yPJMhz4FKzrsTClDRGJmcx73hNLU7F+bEUxvH1d2rhZM+uPEOKzYHRjPxrn5jHr+HRu54ihn0I5uRkIli5g4fB4x1vXq7Qt/1jg/pa8ExB3/wGyLN+SZTm36NvvgNY6eN0yFZsRy7qwdQysP5DG1o0h5HdIvES057usOBbDSz7OdHW3VzumcB89jcQXQzwpKJSZsuki+m1GMTs1nfS8dGafmq12PKG6iDsHdVo+/vHW9ZQ+/3KeklwXxf8M0FCSpLqSJBkCLwHb7j9AkqT7O8UHAmE6eN0yteTsEvQkPca3HK8MGTw4G619E8acc8Xewphp/TzUjigUw9XGjMm93TkckcyWqAIauQ9iTNpd9kTvYX/MfrXjCVVd9m24HQ2OLR7/OVZ1IS8T7qaUWazilLr4y7JcAIwD9qAU9V9lWQ6RJOkTSZIGFh32tiRJIZIkXQDeBoaV9nXL0oXkC+yO3s3rTV+nllktOP8zpF5lh+1IwhLv8ukzzahhLLp7KqpX27nR0qUmn/wRyp3mIxiemkIjIxtmB8wmPU+M/hHKUPwFZVu7BC3/mi7KNi1G93n+g07G+cuyvFOW5UayLNeXZXl20b7psixvK/p6qizLTWVZ9pJluassy+otX/MIsiyzKGgRNsY2jGg2Agpy4fB8cmq1YuIFR/p5OtKjiVicpSLT00jMf96TzNwCZgQaYODsyycpt7mVc4uFgWLsv1CG4s4pW0evx39OjdrKNu2m7vP8B3GH7z+cjDtJUGIQb3q9iamBqdLqT7/JF3nPYWKoz8wBTdWOKDyGRrUseKtLA7aejyPE+WWapkTzumMnNkVu4lT8KbXjCVVV3Hmo6frwydyKY1k0M0B63H8fp2Oi+N9HlmWWnltKbbPaDG44WOnrP7aQFMtmrIqvy0f9PMQc/ZXIW13r08DenNGBtdGaO/BW/A1cLFyYeWIm2QXZascTqqL48yXr8gEwtQF9Y0iPLZtMDyGK/30OxBwg5FYIo71GY6BnoCzBdieGGXf60qGBnZi7p5Ix0tdj/vOexGYUcMhiAMZXDzKzyRvEZsay/NxyteMJVU1WqnKxt3YJLvYCSJLS9SO6fdRRqC1k2flluNVwY0D9AaAtRD66gBjD+uzXtmTOs2LunsqotasVr7V15YPoVqy7+BJDOnXh0vBgpvUZyrxvy/cCm1DFPcnF3r/UqCO6fdSy89pOou5EMbblWPQ1+hC6FelWJPMy+zGxR2NcbEzVjig8oUm9G5N2xZORfyzherI1yBL5t2rz0bv2/LiuQO14QlWRcFHZOniW/Lk16kC6aPmXu3xtPl+f/xp3K3d6uvYErZbCw19wjTrEOnRneHs3tSMKpWBupE/6MQ9y840f2F+Ya8yED3JUSiVUOUlhYOFYsou9f7EsavlrC3Wf6yFE8Qe2RG0hNjOW8S3Ho5E0ELEbveRQvsofyKfPtkBfT/yaKrvkBL1i99+KNyUus3w/bgtVVFLo31M0l5SFI8iFkHVLt5n+Q7WvavnafL4P/h5PW086OXUCWSZr/zxiZDvMvV/Ey7mm2hEFHXBxKX6/oU0ic0/NRS7nW+uFKkZbCMnhYPeExd+8aKqYzETdZXqEal/8t1/ZTtzdON70ehNJktBGHcA0+QI/6j3HxF5ixs6qYvZsMP3HZRtTUxg2KYJDsYc4cEPM/CmUwu1oZRrnJ235mxfdOCqKf/ko0Bbw3cXvaGLTRJmyGUjeNYd42ZpmfUaLGTurED8/WLkSXFxkQMbVMoZl82+wbNJTNLJqxNxTc7mbf1ftmEJl9ddKXPbFrWP1GO61/MtvOcdqXfx3XdvFjYwbjPIchSRJpIUfoVZqIHssX2CQd1214wk65ucH169LBIVGEf5OKzw1MzDQGDC93XSSspJYfl6M/ReeUFKosrVzf7Lnm4lun3JTqC3ku4vf0dCqIV2duwJw8485pMoWdHhhghjTX4W18mjIBaseNE7cTlRMLF52XgxuNBj/MH/CblX4CWeFiigpTJnWwcj8yZ5vZA6G5pBZfotYVdvi/2fMn1xLu8Yoz1FoJA0Xg47RJPMkl5yH0sBJTNxW1TUaMBEzKZfjvy1BlmXeafUONY1q8mnApxSW43A7oYpICnvyLp+/mNuLln9Z08paVgavpK5lXXq49CC/UEvK7vncxQTvIR+oHU8oBzXre5Nk1YquaZvZFBSDpZElk3wmcTHlIhsjNqodT6hMCvIgJQLsG5fuPOa1RPEvawdvHCTydiT/a/4/9DR6bNx7mE55R0lyfwVTS1u14wnlxLb7O7hokjmx05/0nHz61e1HW8e2LDm7hJTs8l1YQ6jEUq+AtkBHLX9xwbfMyLLM6ourcTJ3ok/dPiSk5aAfsBStpE/d/pPUjieUI41Hf/LMajO44A8W/xmJJElM851GdmE2S84uUTueUFn8dbH3SYd5/sVMdPuUqbNJZwlOCWZY02Hoa/T5ZtsRBnGYnOZ+YCH6+qsVPX0M273JU5pQAgIOE56QQV3LurzW5DW2RG3hfNJ5tRMKlUHSZZD0wKZh6c5jZgs5d6CwfOabqnbF/4dLP2BlZMWgBoM4E52Kc/hq9CWw6DZB7WiCGlq9hqxvwkiDvczcFoIsy7zp+Sb2pvbMOTVHXPwVHi0pFGzqg4Hxo4/9L6Y2yjb7dukzPYZqVfyv3LnC4djDvOzxMgYaIxZsPoGf/gG0zQeDlava8QQ1mFojeb3EQM1xLl+9xo6L8ZgamDLJexLNBAJnAAAgAElEQVRhqWFsitykdkKhoksKK32XD4CJlbLNTi39uR5DtSr+a0LWYKJvwsvuL/PLmRi63FqPMfnod3pf7WiCmnxHo6/N5Z2aJ5i9I4ysvAJ6ufWijUMblpxdwu2c8mmJCZVQfjakXn3yOX3u91fLv5wmd6s2xT/xbiLbr27nmQbPgNaU1btPMVz/T/AcDHaN1I4nqMm+MdTvxlDNHpLTMll+MApJkpjaZipZ+VksPbdU7YRCRZUcDsi6afn/NRW0KP665X/ZH62s5bUmr7HozwiG5v+OoVSA1HmK2tGEisB3DIZZiUyvF8l3R64RnXKXBlYNGOoxlE0RmwhJCVE7oVARJRXdEV7aYZ5wX8tfdPvoTGZeJr+F/0ZP155kZNZgT8B5XjPYj+T1snKhRhAaPA02DXhZuwMDPYlPtivD98Z4jcHGxIbZp2ajlbUqhxQqnKRQ0DME63qlP5eJaPnr3MaIjWTmZzKs6TBmbgthgtFW9CUtiL5+4S8aDbR5E4OEs8z2yebA5ST2hyVibmjOhNYTuJhykS1RW9ROKVQ0yZfB1h309Et/LkNT0DcRF3x1JV+bz7qwdfg6+HItzorU6GAGsw/JewRYi5k7hfu0eBmMajAgexv17cz4ZHsoOfmF9K/Xn1b2rVgctJi03DS1UwoVia5G+vzF1EZ0++jK/pj9JGYl8kKjoczZEcYcsw1IxhbQZara0YSKxsgCWr6K3uVtzHnahuu3slh17Nq9O3/T8tJYdm6Z2imFiiInHdJulH5On/uZWoniryv+of44WzgTHOGIe8ZJfAqCkDpPfrJFloWqz3cUyFp8UzbTu6kDyw5EEXcnG3drd15yf4lfI37lcupltVMKFUFpF3ApjqmN6PPXhZCUEM4nn6eP82D8j4ay0HQN2DUGn/+pHU2oqKzcwL0vBP7AR73c0Moyc3YqIzrGthyLpaEl807PE2v+Crqb0+d+Jtai+OvCurB1mBmYcS60IZP11lOzIAUGLgN9Q7WjCRWZ72jITsUpdgejO9dne3A8AVdvUcOwBuNbjScoMYg91/eonVJQW1IYGJiBpYvuzmlqIy74llZyVjK7o3fjY9ML4/C9vCT9idRuLDj7qB1NqOjcOkCtZhDwDWM616NOTRNmbguhoFDLcw2eo7F1YxYELiC7IFvtpIKakkKV/n6NDsuoqTVk34FymFOqyhV/f39wcwN7c1tC3tvBhbXefGm4Em0db+g+Q+14QmUgSdB2DCSFYhxzmI/6eXA5IYOfT8egp9FjSpspJNxN4IdLP6idVFBT0mXddvlA0Y1esvIGUMaqVPH394dRo+D6dUCWyL9Vm8CN3dkQ/hqaF38S3T3C42s+BMwd4MRSejdzoH0DGxbsjSD1bh6ta7Wmt1tvVl9aTVxmnNpJBTXcTYG7SbqZ0+d+5XijV5Uq/h9+CFlZD+7Lzjdl5rG5UKO2OqGEyknfCNqOhquHkOIvMGNAUzJzC/hybzgAE70nIiGxMGihykEFVdyb1kHXxb9oZs+cStLylySptyRJ4ZIkRUmS9K/JciRJMpIkaUPR46ckSXLTxev+U0xM8ftv3NTB3XdC9dN6OBhawImvaFTLgtfbubH+dAyXbqbhYObAiOYj2BO9hzMJZ9ROKpQ3Xc7pcz+Tmsq2MnT7SJKkBywH+gBNgJclSfrnb+QN4LYsyw2ARcD80r5ucVwectH9YfsF4T+Z1ITWr0PIZrh9nXeeboi1qSEzihZ9Gd50OI5mjsw/PV8s+lLdJIWCcU2wcNDteY0tlW0lafm3AaJkWb4qy3Ie8Asw6B/HDALWFn29EeguSZKkg9d+wOzZYGr64PhrU1NlvyA8kbZvKReAA77G0sSAD3q7E3T9NlvO38RY35iJ3hMJvx0uFn2pbpLClFa/rsuYsdLyD7x8rczvJdFF8a8D3Ljv+9iifcUeI8tyAZAG2PzzRJIkjZIkKVCSpMDk5OQSB/Hzg9kLcjGwzEaSZFxdYeVKZb8gPBHLOsrF37M/QlYqQ1o74+lkydydl8nMLaCna0+8a3nz1bmvxLw/1YUsFxV/HU7rUCRVawLAtdibOj/3P+mi+Bf31vfPt6zHOQZZllfKsuwty7K3nZ3dE4V5d7QxKQn6FBRAdLQo/IIOPDUe8rPgzCo0GolZA5uSlJHLsgPKoi9T2kwhPS+dby58o3ZSoTxkxENumu77+4EF+6+RJRvxdF0jyqBz5AG6KP6xgPN93zsB/xz/du8YSZL0AUugzG5jq2FsgEZTtr84oRqp1RQa9IDTKyA/m5YuVgxu7cSqY1e5mpyJu7U7QxoN4ZfLvxB1O0rttEJZK4tpHYDQuHRW/lBA4yXnsHn+c9zclOHrZUUXxf8M0FCSpLqSJBkCLwHb/nHMNuD1oq8HAwdkMTmKUJm0fxvuJsOF9QB80NsdI309Pi1a9GVci3GYGZgx/8x8Me9PVffXSB8djvGXZZnhHyVwa7cnN9LqIMsS168r9y2V1RtAqYt/UR/+OGAPEAb8KstyiCRJn0iSNLDosFWAjSRJUcAEQKydKFQubh2hdis4thgK87G3MObdpxtyMDyZ/WGJ1DSuydgWYwmID+DAjQNqpxXKUlIYmNcCs39dtnxiOy7GE7TJCTlf74H9WVnK/UtlQSfj/GVZ3inLciNZluvLsjy7aN90WZa3FX2dI8vyEFmWG8iy3EaW5au6eF1BKDeSBJ0mwZ3rcPE3AF5r53Zv0ZfcgkJecH+BBjUb8MWZL8gtzFU5sFBmkkJ12uWTlVfAnB1hFKabFPv4w+5fKq0qdYevIJQp9z5QqzkcXQDaQgz1NcwY0JTrt7L4/ug19DX6TG4zmZuZN/kp9Ce10wplQatV5vTRYZfPt4euEJeWg61jXrGPl9V9SqL4C8LjkiToPAluRSk3fgGdGtnRs0ktlh2IIj4tm7aObenu0p2VwStJvJuocmBB5+5EQ0G2zlr+Mbey+PbIVfp4WuA4ZBEaw5wHHi/L+5RE8ReEkmg8QGn1HflCaQUCH/VrQqEsM2+XsrLT+97vU6gtZPHZxWomFcpCkm5X7/psRyj6GpmsmmvR9/6Vz5Yk4eqqtDPK+j4lUfwFoSQ0Guj0vrKEX5gyqM3FxpTRneqx9Xwcp6+l4mThxOtNX2f71e2cTzqvcmBBp/4a5mnnXupTHYlIZm9oIm1bneds8mk+bvsxU0e7EB2ttCvK+j4lUfwFoaSaPgs2DR5o/Y/p0oDalsbM2BZCoVZmZPOR2JvaM+/0PLSyVuXAgs4khSkrdxnXKNVp8gq0zPwjhNoOCZxN/5V+9frxbMNndRTy8YjiLwglpdGDju9D4iW4vB0AE0M9pvXzICw+nZ9Px2BqYMp7rd8j5FYIW6O2qhxY0JmkMJ309/94Mpqrt1LQr/UzjmaOfOT7UemzlZAo/oLwJJoPAZuGcHD2vSX3+jV3pG09axbsDef23Tz61e1HS/uWLD67mIy8DJUDC6VWmA8pEaWe0ycpI4fF+yJwabSTjPxU5neaj7mhuY5CPj5R/AXhSejpQ7cPlb7/4F8BkCSJmQObkpFTwII/w+/N+3M757aY96cquHUFtPmlvtj7+e5w8k3Ocls6w9iWY/G089RRwJIRxV8QnpTHIHD0gkNzoEAZo93YoQavtnXl51MxhMSl0cSmCYMbDWZ92Hqu3LmicmChVHQwp8/ZmNtsOn8Zi9rb8bT1ZHjT4ToKV3Ki+AvCk9JooNt0uBMDZ9fe2/3e042oaWrIrG2hyLLM+JbjMTEwYe7puWLen8os+TJIGrBt9ERP12plZm4LwdJpO1oph0/af4KeRu/RTywjovgLQmk06A6u7eHw55B3FwBLUwMm9XLndHQq2y7EYWVsxbgW4zgVf4oDMWLen0orMQSs6oJB8dMwPMrGoFhC045TaHqeMV5jqF+zvo4Dlowo/oJQGpIE3afD3SQ49e293S94O9OsTg3m7rzM3dwCXnB/gYZWDfki8AtyCnL+44RChZUUCrWerL8/LTufeXuDMK+zjcbWjRnWbJhusz0BUfwFobRc2oJ7Xzi6CDKTANArWvQlIT2H5Qej0NfoM7XNVG5m3uSHkB9UDiyUWF4WpF4D+6ZP9PQl+yLJstgMmrt82v5TDDQGOg5YcqL4C4Iu9PhUmfPlwGf3drV2tea5lnX4/ug1olPu4uPgQy+3Xqy6uIq4zH+udyRUaMmXAfmJWv6XE9L56fxBDCzPMrzZcBpb6375xychir8g6IJtA2jzprLWb8LFe7un9GmMgZ50b9GX973fR0Liy8Av1UoqPIl7I31K1vKXZZmPtlzA2GEr9ia1+F/z/5VBuCcjir8g6ErnSWBiBbunKot8A/Y1jHm7e0P2X07i4OUkHMwcGNl8JH9e/5OA+ACVAwuPLTEU9I3Bum6Jnrbp7E0upO0Bw3gmt/kAUwPTMgpYcqL4C4KumFhB12kQfRQu77i3e3j7utSzVRZ9ySvQMqzZMOqY12H+6fnka/NVDCw8tqRQZTK3EgzNTMvKZ87uM5jW+hNfB196uPYow4AlJ4q/IOhS6+Fg1xj2TFUuEgKG+hqmD2jCtZS7rD5+DSM9Iz7w+YCoO1H8Gv6ryoGFx5IUWuIuny/2XibL/A8kTS5TfaciSVIZhXsyovgLgi7p6UO/hcqNX4fn39vdxd2epz3s+Wp/JInpOXR17spTtZ9i+bnl3Mq+pWJg4ZHu3oLMxBJd7A2OvcP6C8cxqBnIKx6vqD6mvzii+AuCrrm1h5avwMllyo1BRT7u34R8rcyn20ORJInJbSaTXZDNV+e+UjGs8EhJRX/Dx5zTp1Ar89GWYMxq/4GNsQ2jvUaXYbgnJ4q/IJSFHp+CsSX88e69Of9dbcwY26UB24PjORSeRD3Levh5+PF75O+EpIQ84oSCapLClO1jFv/1p2MIyziAbBjDRO8JqszY+ThE8ReEsmBqDT1nQ+xpOLvm3u7RXepRz86Mj7deIjuvkNFeo7E2tmbO6Tli0ZeKKjFEuZhv4fDIQ1Myc/n8z3OYOe6hpX1L+tfrXw4Bn4wo/oJQVrxegrqdYO90uH0dACN9PT57phk3UrP56kAk5obmvNf6PYKTg9l2ZZvKgYVi/XWx9zEu2M7bdZl8i13IUhbTfKdVuIu89xPFXxDKiiTBwGXK11vH3uv+eaq+Lc+1qsPKI1eJSMxgQP0BeNl5sShoEWm5aSoGFv5Fq1W6fR7jYm/A1Vv8fuk0BlYnecH9hQpzJ+/DiOIvCGXJyhV6z1XG/t838duHfT0wN9bnw80XQZb4uO3H3Mm9w9KzS1UMK/xL2g3Iy3xkf39OfiFTNwdTw2k7lkaWjGs5rpwCPjlR/AWhrLV8BRr1gX0zIekyADbmRkzr48GZ6Nv8FnQDd2t3hjYeym8Rv3Ex+eJ/n08oP/emdfjv4v/1oSvcyD1OoeFV3m31LpZGluUQrnRE8ReEsiZJMHApGFnAxuH3bv4a4u1EGzdr5uy8TEpmLmNbjMXOxI5PAz6lsGhdYEFlfw3V/Y/VuyITM/jmyCUs6+ymmU0znm34bDmFKx1R/AWhPJjbw3MrISkM/8m/4OYGenoSx2f7knDWjjk7wjA3NGdSm0mEpYaxIXyD2okFUFr+li5gXKPYh7VamSm/X8TE7iB5pDHNdxoaqXKU1cqRUhCqggbd8c9fw6jlL3H9ujL3W1yshtt7vPhxnczRyGR6ufainWM7vjr3FSnZKWonFhIugkOzhz788+kYzsWHI1ke5bmGz9Hcrnk5hisdUfwFoRx9uG4QWfkPzuyYn6sh85gHUzZdJCuvkGm+08gtzOWLM1+olFIAlGU5UyLBwbPYhxPTc5i/K4xadXdhZmjKO63eKeeApSOKvyCUo5gbxY/7zkszIi4tmy/2hONm6caIZiPYeW0np+JPlXNC4Z7EUEAGh+Jb8zO3hZBvEkymJozxLcdjbWxdvvlKSRR/QShHLi4P2y/xejs31pyI5kx0KiObj8TJ3InPAj4jv1BM+6yKhGBl6/jvlv/uSwnsConBymkX7lbuDGk0pJzDlV6pir8kSdaSJP0pSVJk0dbqIccVSpJ0vuifuI1RqLZmzwbTf6znYWqYw+xPC5jUyx0nKxMmbwwG2YBpvtOITo9mTcgaVbJWewnBYFwTLJ0f2J16N4+PtlzEqe5JMgtTmOY7DX2Nvkohn1xpW/5TgP2yLDcE9hd9X5xsWZZbFP0bWMrXFIRKy88PVq4EV1dlBKirQwYr+4/Fz+RNzAwk5j3nydWUuyzeF0lHp4487fI0K4JXEJMeo3b06ifhotLl848pGmZuCyG9IJ67xvvoX68/rWq1Uilg6ZS2+A8C1hZ9vRZ4ppTnE4Qqz88PoqOVmQOi4y3we68ZXNoIf7xNh/rWvOTjzMojVwiOvcNU36kYaAyYdXIWctHSkEI5KCxQxvj/42Lv7ksJbLsQR0OPgxjqGTCh9QSVApZeaYt/LVmW4wGKtvYPOc5YkqRASZICJEkSbxCCcL8O70HnyXBuHeyYwLQ+jbC3MOaDjcHUNLRlgvcETiecZkvUFrWTVh+3oqAg54H+/r+6e+q5XicmJ5C3WryFnamdiiFL55HFX5KkfZIkXSrm36ASvI6LLMvewFBgsSRJxS5rI0nSqKI3icDk5OQSnF4QKrkuU6HDBAj6gRrbRzFnYCMuJ2SwdH8kzzd8nta1WvNF4Bdi7H95+eti730jfWZsCyEtJwuNzTbqWtZlaOOhKoXTjUcWf1mWn5ZluVkx/7YCiZIkOQIUbZMeco64ou1V4BDQ8iHHrZRl2VuWZW87u8r7jioIJSZJ8PQMZQ2A0K10C3qLV1pY8fWhKM7FpDGj3QxyC3KZe2qu2kmrh4Rg0DMC20YA7L4Uzx8X4mjf+hKJ2TeZ5jsNAz0DlUOWTmkvUW8DXgfmFW23/vOAohFAWbIs50qSZAu0Bz5/khfLz88nNjaWnJycUkQWypKxsTFOTk4YGFTu/xiqeWocmNnC1rHMsnqbqzXeZuKv59nxdkdGe41m6bmlHIg5QDeXbmonrdrig5X5fPQMSM7I5cPNl3Cvk8elu5vp7dabto5t1U5YaqUt/vOAXyVJegOIAYYASJLkDYyWZXkk4AGskCRJi/JJY54sy6FP8mKxsbFYWFjg5uZWoRdJqK5kWebWrVvExsZSt25dteNUXl4vQY066P02jJ+0Uxlz53/M3mnLrEHD2BW9i9kBs/Fx8MHC0ELtpFWTLCsjfRr3Q5ZlPth4gczcArwa7OV2qoaJ3hPVTqgTpbrgK8vyLVmWu8uy3LBom1q0P7Co8CPL8glZlpvLsuxVtF31pK+Xk5ODjY2NKPwVlCRJ2NjYiE9mulC3I7x5GD27hqw0WESjoE84HnqTWe1mkZKTwpKzS9ROWHWl34TsVHD0Yl3AdQ6GJ/NCpwzOJB1jjNcYHMwevZxjZVDp7vAVhb9iE38fHbJ0guG7KWgzmmH6e6m7sQ8udzLw8/BjQ/gGAhMC1U5YNd08C8ANk0Z8tiOMDo0sOZOxmnqW9XjF4xWVw+lOpSv+ld2wYcPYuHGj2jGEysLAGP2+87ne/xcM5Fws/PsyNj4WJ7PafHz8Y7Lys9ROWPXEnUXW6PP2wQLMjPRp0fQcNzOrxkXe+4niXwqyLKMtWpdVEMqSq3cf9nbezJqCXpicW8dnMVHczIxlYeCXakerem6eJdG4Puficni/ny3rI9bQx60Pvo6+aifTKVH8Syg6OhoPDw/eeustWrVqxU8//US7du1o1aoVQ4YMITMzE4BPPvkEHx8fmjVrxqhRo8TdmUKpvdbFkyP1JzIwfy6Nzerzalo6GyJ+4+SxuSBW/tINrZaC2LPsz3DiRW8nTtxehb5Gv8pc5L1f5ZuNqMisP0IIjUvX6Tmb1K7BjAFNH3lceHg4P/zwA5988gnPPfcc+/btw8zMjPnz57Nw4UKmT5/OuHHjmD59OgCvvvoq27dvZ8CAATrNK1QvGo3EgiFe9F2azoD0qfzeK42jQTOYHv4jv5//HYsOE6DZ86BvqHbUSuvWjTBs8jNIMPOgY8t4ph47zPve71PLrJba0XROtPyfgKurK23btiUgIIDQ0FDat29PixYtWLt2LdevXwfg4MGD+Pr60rx5cw4cOEBISIjKqYWqwMbciCUvtSQ6NYtZYW581m8tSfoGfGFcAFtGwxIvOL4EctLUjlrpFBRq+XmzMoVG775PszDoczysPfDz8FM5WdmotC3/x2mhlxUzMzNA6fPv0aMH69evf+DxnJwc3nrrLQIDA3F2dmbmzJli+KOgM23r2fBO90Ys2hfBU/U9eaP5SL67+B3d+8yi8+X98Od0OPwFtH4d2o5RRg0Jj7RkfyRWKcEUGBqzKfMwt3Ju8VX3ryrldM2PQ7T8S6Ft27YcP36cqKgoALKysoiIiLhX6G1tbcnMzBSjewSdG9etAe3q2TB9awjdHPxoZNWImde3kvriWhh1CBr1hIBvYLEnbHwD4s6pHblCOxKRzLKDUXSziCW4jge/RW7iFY9XaGqjXiOzrIniXwp2dnasWbOGl19+GU9PT9q2bcvly5epWbMm//vf/2jevDnPPPMMPj4+akcVqhg9jcSSl1pgbqzPWP9gpnl/QnpuOh8f/xjZsQUMXg3vnFda/hF7YGUXWNMfwncrc0kD/v7g5gYajbL191fzJ1JP3J1sXpuWQMKKbtSbcpQOk9fDWT/GthirdrQyJVXUUSje3t5yYOCDN7GEhYXh4eGhUiLhcYm/U/kJjE7l5e8CaN/Alqd9I5l/Zh5T20xlqMd9M07mpEHQWjj1rXL3qm0j/O98yai5ncjK+vumPFNTZaEZv6rZxV2snPxC2r0RyYVfGiLn693bb2RSyKrv9Crl70KSpKCiWZT/k2j5C0Il5u1mzYwBTTkUnkx8TGs6OXViQeACwlPD/z7I2BLavw3vXIDnvgcDEz78su4DhR8gKws+/LCcfwAVybLM1N8vcnGrywOFHyA3W6/K/y5E8ReESs7P14UXvZ1ZfugKXa3HUcOoBpOPTCan4B+DDPQMwHMIjDpMTLpzseeKqUarRa46do3N525SmGFS7ONV/Xchir8gVHKSJDFrUFO8nGsyY/N13vSYxpW0K3z5sLt/JQkXl+LnYHKpk1eGSSuO41EpzNkZRq+mtXAp/n0QF5fyzVTeRPEXhCrA2ECPFa+0xtxYnyXbNQxpoEz+tjd6b7HHz56t9PHfz9Qgi9ltxsLuqZB3txxSqyMyMYMx64JoYG/OghdaMG5aHJJh9gPHmJoqv6OqTBR/QagiHCyNWT3Mh/TsfAKCfGlm05yPj3/MtbRr/zrWz0+5uOvqqiwi5uoKK1do8RturgwR/bYD3Dijwk9RtpIychj2wxmMDPRYPcwHQ30tJx3H03jkPJwtY5AkWfldVIML36L4C0IV0rS2JcuGtuJyfBaalNcx1DPivYPvFTv7p58fREcrIz+jo1EKf/9FMGw7FObD6p5wcE6VmTfobm4BI9acIfVuHqtf98HJypRvLnxD5O1Ivu15g5j3fdDm5Su/iype+EEU/xJbunQpHh4e+Pn5sW3bNubNmwfAli1bCA39e4GyNWvWEBcXV6JzR0dH06xZM53mFaqfro3tmTWoGccvF+ChP5qraVeZeXLm408u6NYBxhwHzxfh8HzwHwxZqWUbuowVFGoZv/4coXHpLBvakuZOlpxJOMPqS6t5tsGzdEq8BnVaV6t5kUTxL6Gvv/6anTt34u/vz8CBA5kyZQqgm+JfFgoKCtSOIKjg1bauvNm5HnsDLWlh/hK7ru3il/BfHv8Expbw7LcwYAlEH4OVnZV1bSshrVZmyu8XOXA5iU8GNaO7Ry1u59xmypEpuFi4MKXF28qC7c5Va8rmRxHFvwRGjx7N1atXGThwIIsWLWLNmjWMGzeOEydOsG3bNiZNmkSLFi2YP38+gYGB+Pn50aJFC7KzswkKCqJz5860bt2aXr16ER8fD0BQUBBeXl60a9eO5cuXP/S1P//8c5o3b46Xl9e9N5wuXbrw141wKSkpuLm5Acobz5AhQxgwYAA9e/bkxRdfZOfOnffONWzYMDZt2kRhYSGTJk3Cx8cHT09PVqxYUUa/OUENU3o35iUfZ46caY6riTefn/mcoMSgkp2k9TAYvlvp+vmhD0TtL5OsZUWWZWb+EcLGoFje6d6QV9q6Issy049P53bubT7v9DmmSWGgLQCXyr8oe0lU3hmLdk1RFlnWJYfm0GfeQx/+9ttv2b17NwcPHsTW1pY1a9YA8NRTTzFw4ED69+/P4MGDlXi7dvHll1/i7e1Nfn4+48ePZ+vWrdjZ2bFhwwY+/PBDVq9ezfDhw/nqq6/o3LkzkyZNKv5H3bWLLVu2cOrUKUxNTUlNffRH8JMnTxIcHIy1tTWbN29mw4YN9O3bl7y8PPbv388333zDqlWrsLS05MyZM+Tm5tK+fXt69uwpFl+vIiRJYvazzcnMLWD7+b64No/nvYPv8XO/n3GyKMFkb06tYeR+8B8CP78AA5dBi5fLLriOyLLM/N3h/HjyOqM61ePdpxsC8PPlnzkUe4jJPpPxsPGAi/MBCZzbqBu4nImWfzkIDw/n0qVL9OjRgxYtWvDZZ58RGxtLWload+7coXPnzoAy739x9u3bx/DhwzEtGptnbW39yNfs0aPHveP69OnDgQMHyM3NZdeuXXTq1AkTExP27t3Ljz/+SIsWLfD19eXWrVtERkbq6KcWKgI9jcTCF1rQtaErN8JeIjs/j/EHxpOZl1myE9VwhOE7wLW9MnX0yYd/Sq0olu6P4tvDV/Dz/X979x5e45UvcPy7ciEhkoa4R+wEEXKRRJCUKuKSajEER8qpdNzp0cvRcpiW6eWcUVpGDQaDNrS02kFTRmmjCWLcxzRBXZIQl1QkSAS5rfPHG7tEIjuyk5299/o8T57snXbK1hUAABfSSURBVL3e9f5WdvLb77ve9a7lwf8854MQgqTrSXx0+COedX/2t6maU+KheSdwdDVtwDXMfI/8H3OEXttIKfH19SUxMfGhn9+4ccOgBc+llGWWs7Oz0y8jWXrK6PvTTgM4ODjQq1cvdu7cyaZNm4iKitLX+8knnzBgwIBKt0kxH3XsbFg+pjOTYmBvShT3Wq9hZsJMlvRegq2NbcUV3OfgAqM3wzfjYedskMXw9H9VX+BPSErJgp2nWbbnHJHB7rw3xA8hBFl3s3gt7jUaOTbive7vaf9T+XmQfhC6TTZ12DVOHfkbSYMGDcjJySnzefv27bl27Zo++RcUFJCUlMRTTz2Fi4sLe/fuBWBDOdMq9u/fnzVr1pCXpw3Xu9/to9PpOHJE68OtaNroUaNGsXbtWhISEvTJfsCAASxfvpyCggIAfvnlF27fttybe6yZg70tK1/qTB9dd+5cGUx8ejwfHfmo8hXZ1YHIv4HvUPj+D9rCMbWIlJI/fpvMsj3neLGbBwuGB2BjIygsLuTNn94k604Wi3svxtWh5Cj/4j+hKB88nzVt4Cagkr+RjBo1igULFhAUFMS5c+eIjo5m8uTJBAYGUlRUxObNm5k5cyadOnUiMDCQ/fv3A7B27VqmTZtGWFgYjo5lzzESERHB4MGDCQkJITAwkIULtdv2Z8yYwfLly3n66afJzMx8bHz9+/cnPj6evn37UqeONpxt/PjxdOzYkeDgYPz8/Jg0aZIaHWTB6trZsmx0MP09hpKf1Z2Y5BjW/HtN5SuytdcmiPMdpi0cc3CV8YN9AgVFxcz8+gTr9qcyrocnH/zODxsb7Yx50ZFFHLx6kHfC3nl4jv6UeLCxs7qLvaCmdFaqgXqfarfComJmf3OCbVcWYO9ygnlh7xHp/bvKV1RUCF/+J5zeAcP/pq0fbCI5dwuYuuEoCWcymR7ejtf7ttN3lW49u5U/7PsDUT5RzO42++ENV4VryX/cThNEXT3UlM6KopTJztaG+cM7MbnjHApz2zJv/1y2P8kQTls7bdEYjzD4ZhKc+9H4wRrg8o07jFiRSOK563wYGcAb/bz1if/AlQPM2z+Pbs268WZIqdF0d2/B5aPg2dMEUZueSv6KYoWEELzWtyNzu82n+F4LZu59ky9/jqt8RfaOEPUFNG4PG8fA5ePGD/YxEs9dZ/DSfVzKvsO6l7sysstvU3SeyT7D63Gvo3PR8XHvj7G3tX9445SftIvWXtbX3w8q+SuKVRsV4s0nvf+CKHTj3UP/zYc/fVv5ShyfgjFfa0Mlv4iCW1eMH2gpUkqW7znH6NUHcHa045upT9OjnZv+9YzbGUz9YSqOdo4sC1+Gcx3nRyv5ZSfUdbG6O3vvU8lfUaxcH28vvv7dZzjQmM/Ov8PLGzdw625B5Spp0Axe3KgtGbkxShtCWU2u5dxjwmeHmf+PUzzn35xtr/SgXdMG+tcz72Qy/vvx3Lp3i7+E/4XmTs0frURKOLML2vTWLmBbIZX8FUWhnVszto/cQMM6LTh0ZyF9l/2VvWceP4LsEc38IXK11vWzZYp+oXhj+u7EFfov+on4M5nMHdSRpVFBONX97Xal7LvZTPh+Ahl5GSzru0y7g7csV/4FuVfB23rvcVHJX1EUAJrUd2NLZAytnXXcabiK6K+WM3PzCTJz7xleic9A6PdHSN4C8R8aLbaLWXlM/Oww0z4/ikfDemyf3oOXu3s+dPNj9t1sJu2axIVbF1jSZwmdm3Yuv8Izu7TvbfsaLUZzo5J/LeLk5FTpbQYOHMiNGzcqvd3ixYv1N41VpR7FsjR0aMjGQZ8R0iwEx5ZfsjV1Pb0XxrE64Tz5hQYeyT89HTpFwZ7/0/rVqyD3XiEff3+a8I9/IuFMJm9FtOfrKU/TtkmDh8pdvX2Vsf8Yy/mb51ncezGhzSsYt39mJ7QIBqcmVYrPnKlx/lUgpURKiY2NcT5DnZycyM01bM6Vqu5bp9Nx+PBh3NzcKi5cSbXtfVIqr6CogDn75rAjZQeNinuS+kt/mjs7MaVXG0aGtMLBvoJpIQruwN/6wY0LMCEOGrWp1P5z7hbwWWIaqxLOcyOvgCGBLZj1nA/NXR69ETLlZgoTd00kNz+XT/p8QkizCoa452TAR+2h1yzty8LUyDh/IcQIIUSSEKJYCFHuzoQQEUKI00KIs0IIs/5tp6am0qFDB6ZOnUpwcDAxMTGEhYURHBzMiBEj9Ml7+/bt+Pj40KNHD6ZPn84LL7wAwLx58/R36AL4+fmRmpr60D5yc3MJDw8nODgYf39/tm7dWua+L168iE6nIzMzkxUrVhAYGEhgYCCenp707t0bgClTphASEoKvry9z584FtAVpLl++TO/evfXl7tcD8PHHH+Pn54efnx+LFy9+aN8TJkzA19eX/v37c+fOw+ueKpbD3taePz3zJ8b7j+e6TTwBIZ/TxPUu72xNosf8OBbsPMXFrMdc1LV3hP9YDwjY9J8Grwl8+moO87Yl8fSffmTBztMEe7iyZVp3/jwqqMzEn3g5kTHbx5BflM+aAWsqTvwAJ7cBEjoMNigmS1WlI38hRAegGPgrMENKebiMMrbAL0A/IB04BERJKZNLl31QRUf+8w/O51TWqSeOvSw+DX2Y2XXmY8ukpqbi5eXF/v37adu2LcOGDWPHjh3Ur1+f+fPnc+/ePd566y3atWtHfHw8np6eREVFkZOTQ2xsLPPmzcPJyYkZM2YAWvKPjY1Fp9Ppj/wLCwvJy8vD2dmZzMxMQkNDOXPmDGlpafp9h4Zqp7Wlj+ALCgro06cPb731FoMGDSIrK4uGDRtSVFREeHg4S5YsISAg4JHt7j9PS0sjOjqaAwcOIKWkW7durF+/HldXV9q2bcvhw4cJDAxk5MiRDB48mDFjxjzyO1JH/pZlV9ou5uydQz27eoxpM4vEnxsRd/pXiiV082xIv45NCe/QFE+3+o9ufHY3rB8O/sNh2CptweAHSCk582suu5Iz+D7pKv9Kv0kdWxsi/Jox4Rkv/N1dyoxJSsn6k+tZeHghXi5eLOmzhFYNWpVZ9hHrXoDcDJh28JF4LIGhR/5VmtVTSnmyZGePK9YVOCulPF9SdiMwBHhs8q/NWrduTWhoKLGxsSQnJ9O9e3cA8vPzCQsL49SpU3h5eennxY+KimLlypUG1y+lZPbs2cTHx2NjY8OlS5fIyMh4aN/lefXVV+nTpw+DBg0C4Msvv2TlypUUFhZy5coVkpOTCQgIKHf7vXv3MnToUP2soMOGDSMhIYHBgwfj6elJYGAgAJ07d37kjEWxTP1a98PLxYs39rzBn5Pe5MUOL/KHQZP59ngm3524wvvfneT9707SzNmBAHcX/Fq60KqhIy2fqoezU1eadnsT139+yJUGvqS2eYn07DwuZOWRdPkWxy5kk52nDSvt5O7CnIEdiOzsTsP65S+neOPuDeYlzuOHCz/Qp1Uf/veZ/6W+fRkfPGXJvQZp++CZGRaZ+CujJqZ0bglcfOB5OlDluyoqOkKvTvcTo5SSfv368cUXXzz0+rFjx8rd9sFpmOHRqZhBm93z2rVrHDlyBHt7e3Q6nb7cg1M1l7Zu3TrS0tJYunQpACkpKSxcuJBDhw7h6upKdHR0mft70OPOBOvWrat/bGtrq7p9rEibp9qw6YVN/Pnon1l/cj37L+9ndrfZTA/vycWsPOJO/8qRtGxOpN/k++SMh7YVBLDSPphe+95jelwxh6QPNgI83erTr2NTOrd2pVf7JjR1dqgwjr2X9jJ331yy7mXxRuc3GOs7FhtRid7rk9u0u3p9n2AuIwtTYfIXQuwGmpXx0hwp5VYD9lHWx2uZGUYIMRGYCODh4WFA1aYVGhrKtGnTOHv2LG3btiUvL4/09HR8fHw4f/48qamp6HQ6Nm3apN9Gp9MRGxsLwNGjR0lJSXmk3ps3b9KkSRPs7e2Ji4sjLS2twliOHDnCwoULSUhI0F8EvnXrFvXr18fFxYWMjAx27NhBr169gN+mnC59wbdnz55ER0cza9YspJT8/e9/JyYm5kl/RYoFcbBzYGbXmfR078m7ie8ycddE+rXux4yQGbwUpuOlMB0AdwuKuHTjDpey75B7r5C7BUXczl9G/t5RbChcxtVR39PM3ZM6doYn7au3r/LhoQ/ZlbYLLxcvloYvLX8M/+P8ayO4tYcmHSu/rYWpMPlLKas6EDYdeLAzzh0oc2VzKeVKYCVoff5V3G+1a9y4MevWrSMqKop797Sx0O+//z7e3t4sW7aMiIgI3Nzc6Nr1t+XhIiMj9atndenSBW9v70fqHT16NIMGDdJP4ezj41NhLEuXLiUrK0t/ATckJITVq1cTFBSEr68vXl5e+u4pgIkTJ/Lcc8/RvHlz4uJ+m9MlODiY6Ohofczjx48nKChIdfEoemEtwtjyuy2s+3kdq/+9mriLcQxrO4xx/uNo4dQCB3tb2jR2ok3jUkOXPb+A1eF4/DAFxsYC5Xft3Jd5J5N1P69j02ntAGp60HTG+o6ljm3F2z7i2i/awi393rX6Lh8w0lBPIcQeyr/ga4d2wTccuIR2wfdFKWXS4+o0h6Gej5Obm4uTkxNSSqZNm0a7du14/fXXTR1WjTCn90mpmqu3r7LqxCq+OfsNSOjVqhfDvYcT1iKs7O6Yn7+Gzb+HrhNh4IIy65RS8u/Mf/PNmW+IPR9LQXEBz3s+zytBr9DCqcWTB7trLuz/BN44CQ2aPnk9tVyNXPAVQgwFPgEaA98JIY5LKQcIIVoAq6WUA6WUhUKIV4CdgC2wpqLEbwlWrVrFp59+Sn5+PkFBQUyaNMnUISmK0TWr34y3w95mQsAE1ievZ9u5bey+sJtGDo14xv0ZerTsgb+bP83rN9cGhvhFwqWjkLgUWnaGTqMAuJV/i6TMJPZf3s9P6T+RcjMFRztHnvd6nnF+4/BwrmI3cFEhnNgE7fpZdOKvDHWTl2J06n2yXvlF+fx48Ud+vPAjey/tJSdfW8rUpa4L7k7uuDm64WzvhDwfR/6dLDJbdCKjMJdLuZcAsLOxo3PTzgzQDeA53XM41an8Xe9lSvo7fBUNoz4Hn+eNU2ctVSNH/oqiKA+qY1uHCF0EEboICooLOHX9FMnXkzmZdZKreVfJyMvgbP5ZqO+CfdFtGl07TYBXOMO9h9OhYQeCmgRRz76e8QM7sBxcdeAdYfy6zZRK/oqiVAt7G3v8G/vj39i/7AIXD7HhtVXM+Wk2F7Ia4+Eh+OADGD3ayIGkH9EWao+YDzYVTEthRVTyVxTFJDbEd2Hi9kDy7mrz6aelwcSJ2mtG/QDYtwjqOkOQsT9VzJua1VNRFJOYMwd94r8vL0/7udFcOgonv4WwaVC3QcXlrYhK/tXsnXfeYffu3aYOQ1FqnQsXKvfzJ/Lj++DYEEKnGrFSy2DRyX/DBtDpwMZG+75hQ83H8O6779K3r/UuGKEo5SnvJn6PlvnG2cGZXXDuB+jxOjiUsYavlbPY5L9hg9Z/mJamLdd5vz+xqh8A5U1tfPz4cUJDQwkICGDo0KFkZ2cDEB0dzebNmwGYNWsWHTt2JCAgQD+r57Vr14iMjKRLly506dKFffv2VS1ARTETH3wA9UoN7Klnf4cPesyEX6s4Y29+Hnz3Brh5Qzd1j02Z7i8KUtu+OnfuLEtLTk5+5Gflad1aSi3tP/zVurXBVZQpJSVF2traymPHjkkppRwxYoSMiYmR/v7+cs+ePVJKKd9++2356quvSimlHDt2rPzqq6/k9evXpbe3tywuLpZSSpmdnS2llDIqKkomJCRIKaVMS0uTPj4+VQuwFqjM+6RYt/Xrtf9JIbTv65dflnJBOyk/bCvlr6efvOJvX5NyrrOUKQnGCtVsAIelATnWYkf7VGd/Yumpjc+dO8eNGzd49tlnARg7diwjRox4aBtnZ2ccHBwYP348zz//vH5xl927d5Oc/Nvs1rdu3SInJ4cGDdTFKcXyjR5demRPc7j2rTbn/toIiNoIrbqWt3nZTnwFh9dA91dB18OY4VoUi+32Kbc/0QiThZae2tiQtW/t7Ow4ePAgkZGRbNmyhYgI7WaT4uJiEhMTOX78OMePH+fSpUsq8SvWrXF7+P0/wMFF+xA4/oV24m6Isz/AlinQujv0ebt64zRzFpv8y+xPrKf93NhcXFxwdXUlISEBgJiYGP1ZwH25ubncvHmTgQMHsnjxYo4fPw5A//799fPvA/qfK4pVa9QGxu0G9xDYMhk2v6wtxFIeKeHYBvj8P6CxjzaNg619+eUVy+32uX8qOWeO1tXj4UH13D1Y4tNPP2Xy5Mnk5eXh5eXF2rVrH3o9JyeHIUOGcPfuXaSULFq0CNDW0502bRoBAQEUFhbSs2dPVqxYUT1BKoo5qd8Ixn4LexfBnv+DX76HrhO0yeAa+2jTMhcXwYUDWpmzu8CzJ4z8DByfMnX0tZ6a2E0xOvU+KUaXeUYbs39/JS4HF3B01c4GCm5DXRd49i3oNhlsLfaY1iBqYjdFUSyHWzsY+Snk/gqnYuHqz5Cfq30AtAzRZuqsUw0TwlkwlfwVRTEfTk0g5PemjsIiWOwFX0VRFKV8Zpf8a+s1CkWj3h9FMQ9mlfwdHBy4fv26SjC1lJSS69ev4+DgYOpQFEWpgFn1+bu7u5Oens61a48Z76uYlIODA+7u7qYOQ1GUCphV8re3t8fT09PUYSiKopg9s+r2URRFUYxDJX9FURQrpJK/oiiKFaq10zsIIa4BaVWowg3INFI45sLa2mxt7QXVZmtRlTa3llI2rqhQrU3+VSWEOGzI/BaWxNrabG3tBdVma1ETbVbdPoqiKFZIJX9FURQrZMnJf6WpAzABa2uztbUXVJutRbW32WL7/BVFUZTyWfKRv6IoilIOs07+QogIIcRpIcRZIcSsMl6vK4TYVPL6P4UQupqP0rgMaPMbQohkIcQJIcQPQojWpojTmCpq8wPlhgshpBDC7EeGGNJmIcTIkvc6SQjxeU3HaGwG/G17CCHihBDHSv6+B5oiTmMRQqwRQvwqhPi5nNeFEGJJye/jhBAi2KgBSCnN8guwBc4BXkAd4F9Ax1JlpgIrSh6PAjaZOu4aaHNvoF7J4ynW0OaScg2AeOAAEGLquGvgfW4HHANcS543MXXcNdDmlcCUkscdgVRTx13FNvcEgoGfy3l9ILADEEAo8E9j7t+cj/y7AmellOellPnARmBIqTJDgE9LHm8GwoUQogZjNLYK2yyljJNS5pU8PQCY+xSbhrzPAO8BHwJ3azK4amJImycAf5FSZgNIKX+t4RiNzZA2S8C55LELcLkG4zM6KWU8kPWYIkOAz6TmAPCUEKK5sfZvzsm/JXDxgefpJT8rs4yUshC4CTSqkeiqhyFtftA4tCMHc1Zhm4UQQUArKWVsTQZWjQx5n70BbyHEPiHEASFERI1FVz0MafM8YIwQIh3YDvxXzYRmMpX9f68Us5rSuZSyjuBLD10ypIw5Mbg9QogxQAjwbLVGVP0e22YhhA2wCIiuqYBqgCHvsx1a108vtLO7BCGEn5TyRjXHVl0MaXMUsE5K+ZEQIgyIKWlzcfWHZxLVmr/M+cg/HWj1wHN3Hj0N1JcRQtihnSo+7jSrtjOkzQgh+gJzgMFSyns1FFt1qajNDQA/YI8QIhWtb3SbmV/0NfRve6uUskBKmQKcRvswMFeGtHkc8CWAlDIRcECbA8dSGfT//qTMOfkfAtoJITyFEHXQLuhuK1VmGzC25PFw4EdZciXFTFXY5pIukL+iJX5z7weGCtospbwppXSTUuqklDq06xyDpZSHTROuURjyt70F7eI+Qgg3tG6g8zUapXEZ0uYLQDiAEKIDWvK35GX9tgEvlYz6CQVuSimvGKtys+32kVIWCiFeAXaijRRYI6VMEkK8CxyWUm4D/oZ2angW7Yh/lOkirjoD27wAcAK+Krm2fUFKOdhkQVeRgW22KAa2eSfQXwiRDBQBb0opr5su6qoxsM3/DawSQryO1v0Rbc4Hc0KIL9C67dxKrmPMBewBpJQr0K5rDATOAnnAy0bdvxn/7hRFUZQnZM7dPoqiKMoTUslfURTFCqnkryiKYoVU8lcURbFCKvkriqJYIZX8FUVRrJBK/oqiKFZIJX9FURQr9P8MQ7xzNSkcegAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.plot(x_points, real_func(x_points), label='real')\n", "plt.plot(x_points, fit_func(p_lsq_9[0], x_points), label='fitted curve')\n", "plt.plot(\n", " x_points,\n", " fit_func(p_lsq_regularization[0], x_points),\n", " label='regularization')\n", "plt.plot(x, y, 'bo', label='noise')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第1章统计学习方法概论-习题\n", "**撰写人：**胡锐锋-天国之影-Relph\n", "\n", "**github地址：**https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "### 习题1.1\n", "  说明伯努利模型的极大似然估计以及贝叶斯估计中的统计学习方法三要素。伯努利模型是定义在取值为0与1的随机变量上的概率分布。假设观测到伯努利模型$n$次独立的数据生成结果，其中$k$次的结果为1，这时可以用极大似然估计或贝叶斯估计来估计结果为1的概率。\n", "\n", "**解答：**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "伯努利模型的极大似然估计以及贝叶斯估计中的**统计学习方法三要素**如下： \n", "1. **极大似然估计** \n", "**模型：** $\\mathcal{F}=\\{f|f_p(x)=p^x(1-p)^{(1-x)}\\}$ \n", "**策略：** 最大化似然函数 \n", "**算法：** $\\displaystyle \\mathop{\\arg\\min}_{p} L(p)= \\mathop{\\arg\\min}_{p} \\binom{n}{k}p^k(1-p)^{(n-k)}$\n", "2. **贝叶斯估计** \n", "**模型：** $\\mathcal{F}=\\{f|f_p(x)=p^x(1-p)^{(1-x)}\\}$ \n", "**策略：** 求参数期望 \n", "**算法：**\n", "$$\\begin{aligned} E_\\pi\\big[p \\big| y_1,\\cdots,y_n\\big]\n", "& = {\\int_0^1}p\\pi (p|y_1,\\cdots,y_n) dp \\\\\n", "& = {\\int_0^1} p\\frac{f_D(y_1,\\cdots,y_n|p)\\pi(p)}{\\int_{\\Omega}f_D(y_1,\\cdots,y_n|p)\\pi(p)dp}dp \\\\\n", "& = {\\int_0^1}\\frac{p^{k+1}(1-p)^{(n-k)}}{\\int_0^1 p^k(1-p)^{(n-k)}dp}dp\n", "\\end{aligned}$$\n", "\n", "**伯努利模型的极大似然估计：** \n", "定义$P(Y=1)$概率为$p$，可得似然函数为：$$L(p)=f_D(y_1,y_2,\\cdots,y_n|\\theta)=\\binom{n}{k}p^k(1-p)^{(n-k)}$$方程两边同时对$p$求导，则：$$\\begin{aligned}\n", "0 & = \\binom{n}{k}[kp^{k-1}(1-p)^{(n-k)}-(n-k)p^k(1-p)^{(n-k-1)}]\\\\\n", "& = \\binom{n}{k}[p^{(k-1)}(1-p)^{(n-k-1)}(m-kp)]\n", "\\end{aligned}$$可解出$p$的值为$p=0,p=1,p=k/n$，显然$\\displaystyle P(Y=1)=p=\\frac{k}{n}$ \n", "\n", "**伯努利模型的贝叶斯估计：** \n", "定义$P(Y=1)$概率为$p$，$p$在$[0,1]$之间的取值是等概率的，因此先验概率密度函数$\\pi(p) = 1$，可得似然函数为： $$L(p)=f_D(y_1,y_2,\\cdots,y_n|\\theta)=\\binom{n}{k}p^k(1-p)^{(n-k)}$$ \n", "根据似然函数和先验概率密度函数，可以求解$p$的条件概率密度函数：$$\\begin{aligned}\\pi(p|y_1,\\cdots,y_n)&=\\frac{f_D(y_1,\\cdots,y_n|p)\\pi(p)}{\\int_{\\Omega}f_D(y_1,\\cdots,y_n|p)\\pi(p)dp}\\\\\n", "&=\\frac{p^k(1-p)^{(n-k)}}{\\int_0^1p^k(1-p)^{(n-k)}dp}\\\\\n", "&=\\frac{p^k(1-p)^{(n-k)}}{B(k+1,n-k+1)}\n", "\\end{aligned}$$所以$p$的期望为：$$\\begin{aligned}\n", "E_\\pi[p|y_1,\\cdots,y_n]&={\\int}p\\pi(p|y_1,\\cdots,y_n)dp \\\\\n", "& = {\\int_0^1}\\frac{p^{(k+1)}(1-p)^{(n-k)}}{B(k+1,n-k+1)}dp \\\\\n", "& = \\frac{B(k+2,n-k+1)}{B(k+1,n-k+1)}\\\\\n", "& = \\frac{k+1}{n+2}\n", "\\end{aligned}$$\n", "$\\therefore \\displaystyle P(Y=1)=\\frac{k+1}{n+2}$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题1.2\n", "  通过经验风险最小化推导极大似然估计。证明模型是条件概率分布，当损失函数是对数损失函数时，经验风险最小化等价于极大似然估计。\n", "\n", "**解答：**\n", "\n", "假设模型的条件概率分布是$P_{\\theta}(Y|X)$，现推导当损失函数是对数损失函数时，极大似然估计等价于经验风险最小化。\n", "极大似然估计的似然函数为：$$L(\\theta)=\\prod_D P_{\\theta}(Y|X)$$两边取对数：$$\\ln L(\\theta) = \\sum_D \\ln P_{\\theta}(Y|X) \\\\ \n", "\\mathop{\\arg \\max}_{\\theta} \\sum_D \\ln P_{\\theta}(Y|X) = \\mathop{\\arg \\min}_{\\theta} \\sum_D (- \\ln P_{\\theta}(Y|X))$$ \n", "反之，经验风险最小化等价于极大似然估计，亦可通过经验风险最小化推导极大似然估计。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "参考代码：https://github.com/wzyonggege/statistical-learning-method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第02章 感知机/2.Perceptron.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第2章 感知机" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1．感知机是根据输入实例的特征向量$x$对其进行二类分类的线性分类模型：\n", "\n", "$$\n", "f(x)=\\operatorname{sign}(w \\cdot x+b)\n", "$$\n", "\n", "感知机模型对应于输入空间（特征空间）中的分离超平面$w \\cdot x+b=0$。\n", "\n", "2．感知机学习的策略是极小化损失函数：\n", "\n", "$$\n", "\\min _{w, b} L(w, b)=-\\sum_{x_{i} \\in M} y_{i}\\left(w \\cdot x_{i}+b\\right)\n", "$$\n", "\n", "损失函数对应于误分类点到分离超平面的总距离。\n", "\n", "3．感知机学习算法是基于随机梯度下降法的对损失函数的最优化算法，有原始形式和对偶形式。算法简单且易于实现。原始形式中，首先任意选取一个超平面，然后用梯度下降法不断极小化目标函数。在这个过程中一次随机选取一个误分类点使其梯度下降。\n", " \n", "4．当训练数据集线性可分时，感知机学习算法是收敛的。感知机算法在训练数据集上的误分类次数$k$满足不等式：\n", "\n", "$$\n", "k \\leqslant\\left(\\frac{R}{\\gamma}\\right)^{2}\n", "$$\n", "\n", "当训练数据集线性可分时，感知机学习算法存在无穷多个解，其解由于不同的初值或不同的迭代顺序而可能有所不同。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 二分类模型\n", "$f(x) = sign(w\\cdot x + b)$\n", "\n", "$\\operatorname{sign}(x)=\\left\\{\\begin{array}{ll}{+1,} & {x \\geqslant 0} \\\\ {-1,} & {x<0}\\end{array}\\right.$\n", "\n", "给定训练集：\n", "\n", "$T=\\left\\{\\left(x_{1}, y_{1}\\right),\\left(x_{2}, y_{2}\\right), \\cdots,\\left(x_{N}, y_{N}\\right)\\right\\}$\n", "\n", "定义感知机的损失函数 \n", "\n", "$L(w, b)=-\\sum_{x_{i} \\in M} y_{i}\\left(w \\cdot x_{i}+b\\right)$\n", "\n", "---\n", "#### 算法\n", "\n", "随即梯度下降法 Stochastic Gradient Descent\n", "\n", "随机抽取一个误分类点使其梯度下降。\n", "\n", "$w = w + \\eta y_{i}x_{i}$\n", "\n", "$b = b + \\eta y_{i}$\n", "\n", "当实例点被误分类，即位于分离超平面的错误侧，则调整$w$, $b$的值，使分离超平面向该无分类点的一侧移动，直至误分类点被正确分类" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "拿出iris数据集中两个分类的数据和[sepal length，sepal width]作为特征" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "from sklearn.datasets import load_iris\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# load data\n", "iris = load_iris()\n", "df = pd.DataFrame(iris.data, columns=iris.feature_names)\n", "df['label'] = iris.target" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "2 50\n", "1 50\n", "0 50\n", "Name: label, dtype: int64" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.columns = [\n", " 'sepal length', 'sepal width', 'petal length', 'petal width', 'label'\n", "]\n", "df.label.value_counts()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEHCAYAAACjh0HiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5wcdZ3n8df7hqyJCuSAuGsy0aB4eQghSyCCGBd/4C6K2RBREW7Z3Sgnq4sLHis+jOeioiconnjInS6gK4objFmMgAIqGAURuAnBRBLxJ25m4I4YNgHWoCF+7o+qSSZDz0zXdH+7q6rfz8djHtNdXV3zqWroT6rq8/l+FRGYmVnv+g/dDsDMzLrLicDMrMc5EZiZ9TgnAjOzHudEYGbW45wIzMx63D6p/4CkPmAAGIqIxaNeWwZcDAzliy6LiCvH295BBx0Uc+bMSRCpmVl9rV279tcRMaPRa8kTAXAOsAnYb4zXvxwR72h2Y3PmzGFgYKAtgZmZ9QpJvxrrtaSXhiT1A68Fxv1XvpmZdU/qewSfBN4N/H6cdV4vab2kVZJmN1pB0pmSBiQNbNmyJUmgZma9KlkikLQYeDgi1o6z2vXAnIiYD3wbuKrRShFxeUQsjIiFM2Y0vMRlZmaTlPIewSJgiaQTganAfpKujojTh1eIiK0j1r8C+GjCeMzMWrJz504GBwd54oknuh3KmKZOnUp/fz9Tpkxp+j3JEkFELAeWA0h6OfCukUkgX/7siHgof7qE7KaymVkpDQ4Osu+++zJnzhwkdTucp4gItm7dyuDgIAcffHDT7+t4H4GkCyQtyZ+eLek+ST8EzgaWdToeM7NmPfHEExx44IGlTAIAkjjwwAMLn7F0onyUiFgDrMkfnz9i+e6zBrO6Wb1uiItvvp8Ht+1g5vRpnHfCXJYumNXtsKxFZU0CwyYTX0cSgVmvWb1uiOXXbmDHzl0ADG3bwfJrNwA4GVjpeIgJswQuvvn+3Ulg2I6du7j45vu7FJHVxU033cTcuXM55JBDuOiii9qyTScCswQe3Laj0HKzZuzatYuzzjqLG2+8kY0bN7JixQo2btzY8nZ9acgsgZnTpzHU4Et/5vRpXYjGuqXd94nuvvtuDjnkEJ73vOcBcOqpp/K1r32NQw89tKU4fUZglsB5J8xl2pS+vZZNm9LHeSfM7VJE1mnD94mGtu0g2HOfaPW6oQnfO5ahoSFmz94zAEN/fz9DQ5Pf3jAnArMEli6YxYUnH86s6dMQMGv6NC48+XDfKO4hKe4TRcRTlrWjismXhswSWbpglr/4e1iK+0T9/f1s3rx59/PBwUFmzpw56e0N8xmBmVkCY90PauU+0Yte9CJ++tOf8stf/pLf/e53XHPNNSxZsmTiN07AicDMLIEU94n22WcfLrvsMk444QRe+MIXcsopp3DYYYe1GqovDZmZpTB8WbDd3eUnnngiJ554YjtC3M2JwMwskarcJ/KlITOzHudEYGbW45wIzMx6nBOBmVmPcyIwM+txTgTW81avG2LRRbdy8Hu+zqKLbm1pLBiz1N7ylrfwrGc9i3nz5rVtm04E1tNSDAxmltKyZcu46aab2rpNJwLraZ5AxpJavxIumQcfmJ79Xr+y5U0ed9xxHHDAAW0Ibg83lFlP8wQylsz6lXD92bAz/29p++bsOcD8U7oXVwM+I7CelmJgMDMAbrlgTxIYtnNHtrxknAisp3kCGUtm+2Cx5V3kS0PW01INDGbG/v3Z5aBGy0vGicB6XlUGBrOKOf78ve8RAEyZli1vwWmnncaaNWv49a9/TX9/Px/84Ac544wzWtqmE4F1Tbsn9jYrleEbwrdckF0O2r8/SwIt3ihesWJFG4LbmxOBdcVw/f5w6eZw/T7gZGD1Mf+U0lUINeKbxdYVrt83Kw8nAusK1+9bVUVEt0MY12TicyKwrnD9vlXR1KlT2bp1a2mTQUSwdetWpk6dWuh9vkdgXXHeCXP3ukcArt+38uvv72dwcJAtW7Z0O5QxTZ06lf7+YiWqTgTWFa7ftyqaMmUKBx98cLfDaDsnAusa1++blUPyRCCpDxgAhiJi8ajXngZ8ATgK2Aq8KSIeSB2TWdm4p8K6qRM3i88BNo3x2hnAv0XEIcAlwEc7EI9ZqXhOBOu2pIlAUj/wWuDKMVY5Cbgqf7wKOF6SUsZkVjbuqbBuS31G8Eng3cDvx3h9FrAZICKeBLYDB45eSdKZkgYkDZT5br3ZZLinwrotWSKQtBh4OCLWjrdag2VPKdCNiMsjYmFELJwxY0bbYjQrA/dUWLelPCNYBCyR9ABwDfBKSVePWmcQmA0gaR9gf+CRhDGZlY7nRLBuS5YIImJ5RPRHxBzgVODWiDh91GrXAX+dP35Dvk45W/bMElm6YBYXnnw4s6ZPQ8Cs6dO48OTDXTVkHdPxPgJJFwADEXEd8Fngi5J+RnYmcGqn4zErA/dUWDd1JBFExBpgTf74/BHLnwDe2IkYrHe8b/UGVty1mV0R9EmcdsxsPrz08G6HZVZa7iy2Wnnf6g1cfee/7n6+K2L3cycDs8Y8+qjVyoq7GswRO85yM3MisJrZNUatwVjLzcyJwGqmb4zG9LGWm5kTgdXMacfMLrTczHyz2Gpm+Iawq4bMmqeq9W8tXLgwBgYGuh2GmVmlSFobEQsbveYzAmurv7jiB3z/53tGCVn0/AP40luP7WJE3eM5BqwqfI/A2mZ0EgD4/s8f4S+u+EGXIuoezzFgVeJEYG0zOglMtLzOPMeAVYkTgVkCnmPAqsSJwCwBzzFgVeJEYG2z6PkHFFpeZ55jwKrEicDa5ktvPfYpX/q9WjXkOQasStxHYGbWA9xHYB2Tqna+yHZdv29WjBOBtc1w7fxw2eRw7TzQ0hdxke2misGsznyPwNomVe18ke26ft+sOCcCa5tUtfNFtuv6fbPinAisbVLVzhfZruv3zYpzIrC2SVU7X2S7rt83K843i61thm/Gtrtip8h2U8VgVmfuIzAz6wHuIyiRMtS4F42hDDGbWTpOBB1Uhhr3ojGUIWYzS8s3izuoDDXuRWMoQ8xmlpYTQQeVoca9aAxliNnM0nIi6KAy1LgXjaEMMZtZWk4EHVSGGveiMZQhZjNLyzeLO6gMNe5FYyhDzGaWlvsIzMx6QFf6CCRNBb4HPC3/O6si4v2j1lkGXAwM5Ysui4grU8Vkk/O+1RtYcddmdkXQJ3HaMbP58NLDW163LP0JZYnDrFtSXhr6LfDKiHhc0hTgdkk3RsSdo9b7ckS8I2Ec1oL3rd7A1Xf+6+7nuyJ2Px/9BV9k3bL0J5QlDrNumvBmsaSnSfrPkt4r6fzhn4neF5nH86dT8p9qXYcyVty1uenlRdYtS39CWeIw66Zmqoa+BpwEPAn8+4ifCUnqk3Qv8DDwrYi4q8Fqr5e0XtIqSbPH2M6ZkgYkDWzZsqWZP21tsmuMe0iNlhdZtyz9CWWJw6ybmrk01B8Rr57MxiNiF3CEpOnAVyXNi4gfjVjlemBFRPxW0tuAq4BXNtjO5cDlkN0snkwsNjl9UsMv8j6ppXVnTp/GUIMv2073J5QlDrNuauaM4A5Jje/2NSkitgFrgFePWr41In6bP70COKqVv2Ptd9oxDU/SGi4vsm5Z+hPKEodZN415RiBpA9k1/X2AN0v6BdkNYJHdApg/3oYlzQB2RsQ2SdOAVwEfHbXOsyPiofzpEmDTpPfEkhi+ydtMJVCRdcvSn1CWOMy6acw+AknPHe+NEfGrcTcszSe71NNHduaxMiIukHQBMBAR10m6kCwBPAk8Arw9In483nbdR2BmVtx4fQQTNpRJ+mJE/OVEyzql6okgVc16kfr9lNsusn9VPBaVs34l3HIBbB+E/fvh+PNh/indjsq6oNWGssNGbawPX8uflFQ160Xq91Nuu8j+VfFYVM76lXD92bAzvxm+fXP2HJwMbC9j3iyWtFzSY8B8SY/mP4+RlYJ+rWMR1kiqmvUi9fspt11k/6p4LCrnlgv2JIFhO3dky81GGDMRRMSFEbEvcHFE7Jf/7BsRB0bE8g7GWBupataL1O+n3HaR/avisaic7YPFllvPGu+M4EhJRwJfGX488qeDMdZGqrH9G9Xpj7c81baL7F8Vj0Xl7N9fbLn1rPH6CP5H/vO/gLvIGrquyB9fmj60+klVs16kfj/ltovsXxWPReUcfz5MGZVYp0zLlpuNMObN4oh4BYCka4AzI2JD/nwe8K7OhFcvqWrWi9Tvp9x2kf2r4rGonOEbwq4asgk0Uz56b0QcMdGyTql6+aiZWTe0Wj66SdKVwNVkncan4w7gnlKG3gCrOPczlFozieDNwNuBc/Ln3wM+nSwiK5Uy9AZYxbmfofQmHHQuIp6IiEsi4nX5zyUR8UQngrPuK0NvgFWc+xlKb7xB51ZGxCkjBp/by0SDzlk9lKE3wCrO/QylN96loeFLQYs7EYiVU5Hx+j22vzW0f392OajRciuF8TqLh4eHPh74g4j41cifzoRn3VaG3gCrOPczlF4zN4vnAKfnw1KvBW4DbouIe1MGZuVQht4Aqzj3M5TehH0Eu1fMJpd5K1kz2ayI6JvgLUm4j8DMrLiW+ggkvQ9YBDwTWEeWCG5ra4QllKoevsh2yzKuvnsDSqbuNfl1378iOnQsmrk0dDLZDGJfB74L3Fn38tFU9fBFtluWcfXdG1Ayda/Jr/v+FdHBY9FMH8GRZDeM7wb+FNgg6fa2RlEyqerhi2y3LOPquzegZOpek1/3/Suig8eimUtD84A/AV4GLAQ2U/NLQ6nq4Ytstyzj6rs3oGTqXpNf9/0rooPHYsIzAuCjwL5kQ0+/MCJeERG1rvtKNVZ+ke2WZVz9VMfCJqnucwzUff+K6OCxaObS0Gsj4mMRcUdE7Gx7BCWUqh6+yHbLMq6+ewNKpu41+XXfvyI6eCyauVncc1LVwxfZblnG1XdvQMnUvSa/7vtXRAePRdN9BGXhPgIzs+JanY/A2sj9CWYVccO5sPbzELtAfXDUMlj8ida3W8I+ifFGH72eBqOODouIJUkiqjH3J5hVxA3nwsBn9zyPXXuet5IMStonMealIUkvG++NEfHdJBFNoMqXhhZddGvD0TlnTZ/G99/zyo5s9/nLv9GwBLVP4ucXnjjpGMxq5YMHZF/+o6kP3v/I5Ld7ybwxRmKdDf/1R5PfbhMmdWmoW1/0deb+BLOKaJQExlverJL2SUxYPirpBZJWSdoo6RfDP50Irm7cn2BWERpjTM2xljerpH0SzTSU/RPZHMVPAq8AvgB8MWVQdeX+BLOKOGpZseXNKmmfRDOJYFpE3EJ2P+FXEfEBYPIXtHvY0gWzuPDkw5k1fRoiu4Z/4cmHt6U/odntfnjp4Zz+4ufsPgPokzj9xc/xjWKzkRZ/AhaesecMQH3Z81arhuafAn9+aXZPAGW///zSrlcNTdhHIOn7ZGMNrQJuBYaAiyKiK62lVb5ZbGbWLa32EbwTeDpwNvAhsrOBv27ij04Fvgc8Lf87qyLi/aPWeRrZpaajgK3AmyLigSZiKqxo/X7VxuAv0htQ92ORtE67SG15qjhS7l8Ja9zbpui+1flYjFJkhrL9gIiIx5pcX8AzIuJxSVOA24FzIuLOEev8LTA/It4m6VTgdRHxpvG2O5kzgtF19pBdQx/r8knR9bttdG/AsEaXfOp+LJ5Spw3ZNdh2nH6Pri0f1uiSQao4Uu5fym13W9F9q+GxGO+MoJmqoYWSNgDryeYi+KGkoyZ6X2Qez59OyX9GZ52TgKvyx6uA4/ME0lZFx9Sv2hj8ReYuqPuxSDqG+9rPN788VRwp96/OcwEU3bc6H4sGmrlZ/DngbyNiTkTMAc4iqySakKQ+SfcCDwPfioi7Rq0yi2x+AyLiSWA7cGCD7ZwpaUDSwJYtW5r503spWr9ftTH4i/QG1P1YJK3TLlJbniqOlPtX0hr3tii6b3U+Fg00kwgei4jdE9FExO1AU5eHImJXRBwB9ANH55PcjNToX/9P+faKiMsjYmFELJwxY0Yzf3ovRev3qzYGf5HegLofi6R12kVqy1PFkXL/Slrj3hZF963Ox6KBZhLB3ZL+UdLLJb1M0v8G1kg6UtKRzfyRiNgGrAFePeqlQWA2gKR9gP2BFvq3Gytav1+1MfiL9AbU/VgkrdMuUlueKo6U+1fSGve2KLpvdT4WDTRTNXRE/vv9o5a/hOxf7w17CiTNAHZGxDZJ04BXkc12NtJ1ZBVIPwDeANwaCcbFLjqmftXG4C8yd0Hdj0XSMdyHbwg3UzWUKo6U+1fnuQCK7ludj0UDyeYjkDSf7EZwH9mZx8qIuEDSBcBARFyXl5h+EVhAdiZwakSMO3yF+wjMzIprqY9A0h8CHwFmRsRrJB0KHBsRDero9oiI9WRf8KOXnz/i8RPAGyeKwczM0mnmHsHngZuBmfnzn5A1mdXa6nVDLLroVg5+z9dZdNGtrF431O2QrAzWr8yGEv7A9Oz3+pXtWTeVojGUYf+qtt0aaOYewUERsVLScsjKPCW1OBZruaWaQMYqrsikImWYgKRoDGXYv6pttyaaOSP4d0kHkpd1SnoxWb1/bVWuico6o0iTURkaklI2UVWtYa4Mn0eJNXNGcC5Zdc/z8wHoZpBV+NRW5ZqorDOKNBmVoSEpZRNV1RrmyvB5lNiEZwQRcQ/wMrJy0b8BDstvBNdW5ZqorDOKNBmVoSEpZRNV1RrmyvB5lFgzYw29kWxOgvuApcCXm20kq6rKNVFZZxRpMipDQ1LKJqqqNcyV4fMosWbuEfxDRDwm6aXACWS9AZ9OG1Z3pZpAxiquyKQiZZiApGgMZdi/qm23JpqZmGZdRCyQdCGwISL+eXhZZ0LcmxvKzMyKa3VimiFJ/0g+REQ+mUwzZxJmva3IJDZlUbWYyzJ5TFnimKRmEsEpZIPFfTwfN+jZwHlpwzKruNGT2MSuPc/L+sVatZjL0htQljha0EzV0G8i4tqI+Gn+/KGI+Gb60MwqrMgkNmVRtZjL0htQljha4Es8ZikUmcSmLKoWc1l6A8oSRwucCMxSKDKJTVlULeay9AaUJY4WOBGYpVBkEpuyqFrMZekNKEscLXAiMEth8Sdg4Rl7/jWtvux5GW+6DqtazGXpDShLHC1INjFNKu4jMDMrrtU+ArM0qlh7nSrmVPX7VTzG1nFOBNYdVay9ThVzqvr9Kh5j6wrfI7DuqGLtdaqYU9XvV/EYW1c4EVh3VLH2OlXMqer3q3iMrSucCKw7qlh7nSrmVPX7VTzG1hVOBNYdVay9ThVzqvr9Kh5j6wonAuuOKtZep4o5Vf1+FY+xdYX7CMzMesB4fQQ+IzBbvxIumQcfmJ79Xr+y89tNFYNZE9xHYL0tVa19ke263t+6zGcE1ttS1doX2a7r/a3LnAist6WqtS+yXdf7W5c5EVhvS1VrX2S7rve3LnMisN6Wqta+yHZd729d5kRgvS1VrX2R7bre37rMfQRmZj2gK30EkmZL+o6kTZLuk3ROg3VeLmm7pHvzH58LV10V6+Fd75+ej1uppewjeBL4+4i4R9K+wFpJ34qIjaPWuy0iFieMwzqlivXwrvdPz8et9JKdEUTEQxFxT/74MWATMCvV37MSqGI9vOv90/NxK72O3CyWNAdYANzV4OVjJf1Q0o2SDhvj/WdKGpA0sGXLloSRWkuqWA/vev/0fNxKL3kikPRM4F+Ad0bEo6Nevgd4bkT8MfApYHWjbUTE5RGxMCIWzpgxI23ANnlVrId3vX96Pm6llzQRSJpClgS+FBHXjn49Ih6NiMfzx98Apkg6KGVMllAV6+Fd75+ej1vppawaEvBZYFNENBxYXdIf5esh6eg8nq2pYrLEqlgP73r/9HzcSi9ZH4GklwK3ARuA3+eL3ws8ByAiPiPpHcDbySqMdgDnRsQd423XfQRmZsWN10eQrHw0Im4HNME6lwGXpYrBxrB+ZVaxsX0wu057/Pm9/a+zG86FtZ/PJotXXzZFZKuzg5lViOcj6DWu6d7bDefCwGf3PI9de547GViP8FhDvcY13Xtb+/liy81qyImg17ime2+xq9hysxpyIug1runem/qKLTerISeCXuOa7r0dtazYcrMaciLoNa7p3tviT8DCM/acAagve+4bxdZDPB+BmVkP6EofQS9ZvW6Ii2++nwe37WDm9Gmcd8Jcli6o0UCrde87qPv+lYGPcak5EbRo9bohll+7gR07syqToW07WH7tBoB6JIO69x3Uff/KwMe49HyPoEUX33z/7iQwbMfOXVx88/1diqjN6t53UPf9KwMf49JzImjRg9t2FFpeOXXvO6j7/pWBj3HpORG0aOb0aYWWV07d+w7qvn9l4GNcek4ELTrvhLlMm7J389G0KX2cd8LcLkXUZnXvO6j7/pWBj3Hp+WZxi4ZvCNe2amj4Zl5dKz7qvn9l4GNceu4jMDPrAeP1EfjSkFmdrV8Jl8yDD0zPfq9fWY1tW0f50pBZXaWs33dvQK34jMCsrlLW77s3oFacCMzqKmX9vnsDasWJwKyuUtbvuzegVpwIzOoqZf2+ewNqxYnArK5Szj3heS1qxX0EZmY9wH0EZmY2JicCM7Me50RgZtbjnAjMzHqcE4GZWY9zIjAz63FOBGZmPc6JwMysxyVLBJJmS/qOpE2S7pN0ToN1JOlSST+TtF7SkanisRZ43HmzWks5H8GTwN9HxD2S9gXWSvpWRGwcsc5rgBfkP8cAn85/W1l43Hmz2kt2RhARD0XEPfnjx4BNwOiJfE8CvhCZO4Hpkp6dKiabBI87b1Z7HblHIGkOsAC4a9RLs4DNI54P8tRkgaQzJQ1IGtiyZUuqMK0RjztvVnvJE4GkZwL/ArwzIh4d/XKDtzxlFLyIuDwiFkbEwhkzZqQI08bicefNai9pIpA0hSwJfCkirm2wyiAwe8TzfuDBlDFZQR533qz2UlYNCfgssCkiPjHGatcBf5VXD70Y2B4RD6WKySbB486b1V7KqqFFwF8CGyTdmy97L/AcgIj4DPAN4ETgZ8BvgDcnjMcma/4p/uI3q7FkiSAibqfxPYCR6wRwVqoYzMxsYu4sNjPrcU4EZmY9zonAzKzHORGYmfU4JwIzsx7nRGBm1uOcCMzMepyyUv7qkLQF+FW34xjDQcCvux1EQt6/6qrzvoH3rxnPjYiGg7VVLhGUmaSBiFjY7ThS8f5VV533Dbx/rfKlITOzHudEYGbW45wI2uvybgeQmPevuuq8b+D9a4nvEZiZ9TifEZiZ9TgnAjOzHudEMAmS+iStk3RDg9eWSdoi6d785790I8ZWSHpA0oY8/oEGr0vSpZJ+Jmm9pCO7EedkNLFvL5e0fcTnV6k5OSVNl7RK0o8lbZJ07KjXK/vZQVP7V9nPT9LcEXHfK+lRSe8ctU6Szy/lDGV1dg6wCdhvjNe/HBHv6GA8KbwiIsZqYHkN8IL85xjg0/nvqhhv3wBui4jFHYumvf4ncFNEvEHSHwBPH/V61T+7ifYPKvr5RcT9wBGQ/WMTGAK+Omq1JJ+fzwgKktQPvBa4stuxdNFJwBcicycwXdKzux1Ur5O0H3Ac2VzhRMTvImLbqNUq+9k1uX91cTzw84gYPYpCks/PiaC4TwLvBn4/zjqvz0/bVkma3aG42imAb0paK+nMBq/PAjaPeD6YL6uCifYN4FhJP5R0o6TDOhlci54HbAH+Kb90eaWkZ4xap8qfXTP7B9X9/EY6FVjRYHmSz8+JoABJi4GHI2LtOKtdD8yJiPnAt4GrOhJcey2KiCPJTkPPknTcqNcbzUVdlTrkifbtHrIxWf4Y+BSwutMBtmAf4Ejg0xGxAPh34D2j1qnyZ9fM/lX58wMgv+S1BPhKo5cbLGv583MiKGYRsETSA8A1wCslXT1yhYjYGhG/zZ9eARzV2RBbFxEP5r8fJrtGefSoVQaBkWc6/cCDnYmuNRPtW0Q8GhGP54+/AUyRdFDHA52cQWAwIu7Kn68i++IcvU4lPzua2L+Kf37DXgPcExH/r8FrST4/J4ICImJ5RPRHxByyU7dbI+L0keuMul63hOymcmVIeoakfYcfA38G/GjUatcBf5VXMLwY2B4RD3U41MKa2TdJfyRJ+eOjyf4f2drpWCcjIv4vsFnS3HzR8cDGUatV8rOD5vavyp/fCKfR+LIQJPr8XDXUBpIuAAYi4jrgbElLgCeBR4Bl3YxtEv4Q+Gr+/9I+wD9HxE2S3gYQEZ8BvgGcCPwM+A3w5i7FWlQz+/YG4O2SngR2AKdGtdrv/w74Un554RfAm2vy2Q2baP8q/flJejrwp8DfjFiW/PPzEBNmZj3Ol4bMzHqcE4GZWY9zIjAz63FOBGZmPc6JwMysxzkRmBWUj3DZaOTZhsvb8PeWSjp0xPM1kmo7Ubt1nhOBWfktBQ6dcC2zSXIisNrJO4i/ng889iNJb8qXHyXpu/mAczcPd4Hn/8L+pKQ78vWPzpcfnS9bl/+eO97fbRDD5yT9n/z9J+XLl0m6VtJNkn4q6WMj3nOGpJ/k8Vwh6TJJLyHrUL9Y2Rj1z89Xf6Oku/P1/6RNh856lDuLrY5eDTwYEa8FkLS/pClkg5CdFBFb8uTw34G35O95RkS8JB+E7nPAPODHwHER8aSkVwEfAV7fZAz/jWwIkrdImg7cLenb+WtHAAuA3wL3S/oUsAv4B7Kxcx4DbgV+GBF3SLoOuCEiVuX7A7BPRBwt6UTg/cCrJnOgzMCJwOppA/BxSR8l+wK9TdI8si/3b+VfpH3AyDFaVgBExPck7Zd/ee8LXCXpBWQjPE4pEMOfkQ1Q+K78+VTgOfnjWyJiO4CkjcBzgYOA70bEI/nyrwD/aZztX5v/XgvMKRCX2VM4EVjtRMRPJB1FNibLhZK+STbS6H0RcexYb2vw/EPAdyLidZLmAGsKhCHg9fmsU3sWSseQnQkM20X2/2Gj4YXHM7yN4febTZrvEVjtSJoJ/CYirgY+Tna55X5ghvI5biVN0d6TlgzfR3gp2YiO24H9yaYLhOKDB94M/N2IkTAXTLD+3cDLJP1HSfuw9yWox8jOTsyS8L8krI4OJ7u5+ntgJ/D2iPidpDcAlz3SjnEAAACjSURBVEran+y//U8C9+Xv+TdJd5DNQz183+BjZJeGziW7Zl/Eh/Ltr8+TwQPAmPPoRsSQpI8Ad5GNL78R2J6/fA1whaSzyUbXNGsrjz5qPU/SGuBdETHQ5TieGRGP52cEXwU+FxGjJy83aztfGjIrjw9IupdsspxfUsFpFq2afEZgZtbjfEZgZtbjnAjMzHqcE4GZWY9zIjAz63FOBGZmPe7/A5LawLRyzHcuAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.scatter(df[:50]['sepal length'], df[:50]['sepal width'], label='0')\n", "plt.scatter(df[50:100]['sepal length'], df[50:100]['sepal width'], label='1')\n", "plt.xlabel('sepal length')\n", "plt.ylabel('sepal width')\n", "plt.legend()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "data = np.array(df.iloc[:100, [0, 1, -1]])" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "X, y = data[:,:-1], data[:,-1]" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "y = np.array([1 if i == 1 else -1 for i in y])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Perceptron" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# 数据线性可分，二分类数据\n", "# 此处为一元一次线性方程\n", "class Model:\n", " def __init__(self):\n", " self.w = np.ones(len(data[0]) - 1, dtype=np.float32)\n", " self.b = 0\n", " self.l_rate = 0.1\n", " # self.data = data\n", "\n", " def sign(self, x, w, b):\n", " y = np.dot(x, w) + b\n", " return y\n", "\n", " # 随机梯度下降法\n", " def fit(self, X_train, y_train):\n", " is_wrong = False\n", " while not is_wrong:\n", " wrong_count = 0\n", " for d in range(len(X_train)):\n", " X = X_train[d]\n", " y = y_train[d]\n", " if y * self.sign(X, self.w, self.b) <= 0:\n", " self.w = self.w + self.l_rate * np.dot(y, X)\n", " self.b = self.b + self.l_rate * y\n", " wrong_count += 1\n", " if wrong_count == 0:\n", " is_wrong = True\n", " return 'Perceptron Model!'\n", "\n", " def score(self):\n", " pass" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Perceptron Model!'" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "perceptron = Model()\n", "perceptron.fit(X, y)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEICAYAAABS0fM3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU5fX48c8hAUJYZRMEQthEIIAs4oKKCm6IQKtWrLaitbSgdamt358VsIK4tNW6VS3WWhfcais7KKIoVAXBBcKOrGGRfQ2BLOf3x51IiAm5Se6dO3fmvF+veSVz586d88zAnNz7nOd5RFUxxhiTuKoEHYAxxphgWSIwxpgEZ4nAGGMSnCUCY4xJcJYIjDEmwVkiMMaYBOd7IhCRJBH5SkSmlvDYUBHZISJfR263+B2PMcaY4yVH4TXuAJYDdUp5/C1Vvc3twRo2bKjp6elexGWMMQlj0aJFO1W1UUmP+ZoIRKQ5cAUwDvitF8dMT09n4cKFXhzKGGMShohsKO0xvy8NPQHcAxScYJ+rRGSxiLwjIi1K2kFEhonIQhFZuGPHDl8CNcaYROVbIhCRAcB2VV10gt2mAOmq2gX4AHi5pJ1Udbyq9lTVno0alXhmY4wxpoL8PCPoDQwUkfXAm8BFIvJa0R1UdZeqHoncfQHo4WM8xhhjSuBbH4Gq3gvcCyAiFwC/U9Ubiu4jIk1VdWvk7kCcTuVyy83NJSsri5ycnEpE7L+UlBSaN29O1apVgw7FGGO+F42qoeOIyBhgoapOBm4XkYFAHrAbGFqRY2ZlZVG7dm3S09MREe+C9ZCqsmvXLrKysmjVqlXQ4RhjzPeiMqBMVeeo6oDI76MjSQBVvVdVO6lqV1W9UFVXVOT4OTk5NGjQIGaTAICI0KBBg5g/azGxYcIESE+HKlWcnxMmBB2RiWdRPyPwSywngUJhiNEEb8IEGDYMsrOd+xs2OPcBrr8+uLhM/LIpJoyJMffddywJFMrOdrYb4wdLBB6aOXMm7du3p23btjzyyCNBh2NCauPG8m03prISMhH4cf01Pz+fW2+9lRkzZrBs2TLeeOMNli1bVvkDm4STlla+7cZUVsIlgsLrrxs2gOqx66+VTQYLFiygbdu2tG7dmmrVqjFkyBAmTZrkTdAmoYwbB6mpx29LTXW2G+OHhEsEfl1/3bx5My1aHJsho3nz5mzevLlyBzUJ6frrYfx4aNkSRJyf48dbR7HxT9xUDbnl1/VXVf3BNqsSMhV1/fX2xW+iJ+HOCPy6/tq8eXM2bdr0/f2srCxOOeWUyh3UGGOiIOESgV/XX8844wxWr17NunXrOHr0KG+++SYDBw6s3EGNMSYKEi4R+HX9NTk5mWeeeYZLL72UDh068JOf/IROnTp5E7Qxxvgo4foIwL/rr/3796d///7eH9gYY3yUcGcExhhjjmeJwBhjEpwlAmOMiXEHcnIZM2UZ81bv9OX4CdlHYIwxYaCqTFuylbFTl7H9wBHq16zKue0aev46dkZgjAds/QDjtbU7DvLzfy7gtte/olHt6rw7oje3XdTOl9eyMwJjKsnWDzBeysnN59mP1vD8x2upnlyFBwZ24oazWpJUxb+ZCuyMwCM333wzjRs3JiMjI+hQTJTZ+gHGKx+t3M4lf/2Epz5cw+WdmzD7d3248Zx0X5MAJGoiWDcBJqbD61Wcn+sqfx4/dOhQZs6cWenjmPCx9QNMZW3Ze5hfv7qIm176guQk4fVbzuTJId1oXDslKq+feJeG1k2ABcMgP/InXPYG5z5Aq4qfx59//vmsX7++8vGZ0ElLcy4HlbTdmBPJzS/gpf+t44kPVlOgyu8vbc8vz2tNteTo/o2eeGcE39x3LAkUys92thtTAbZ+gKmIBet2c8VTc3lo+grOadOAWXf14dYL20Y9CUAinhFkl3K+Xtp2Y8pQ2CF8333O5aC0NCcJWEexKcnOg0d4ePoK/vNlFs3q1eCFn/fk4o4nBxpT4iWC1DTnclBJ242pIFs/wJSloEB544uN/GnmSrKP5jHigjbcdlFbUqsF/zWceJeGuo6DpGLn8UmpznYTalbLb2JV5uZ9/Oi5T7nv3Uw6NK3NjDvO457LTouJJACJmAhaXQ+9xkNqS0Ccn73GV6qjGOC6667j7LPPZuXKlTRv3pwXX3zRm3iNK36tRW1MZezPyeWPk5cy8Jl5bN6TzRPXns4bvzyLto1rBx3acaSkJRZjWc+ePXXhwoXHbVu+fDkdOnQIKKLyCVOsYZKeXnLlTsuWYMVcJtpUlcnfbGHs1OXsOnSEn53VkrsvaU/dGlUDi0lEFqlqz5Iei43zEmMqyWr5TaxYs/0goydl8um3u+jSvC4vDT2Dzs3rBh3WCVkiMHHBavlN0A4fzeeZj1Yz/pO11KiaxIODM7iuV5rvo4K9EDeJQFURie03PGyX4cJk3Ljj5/sBq+U30TN7+XfcP3kpWXsO8+Puzbj38g40ql096LBci4tEkJKSwq5du2jQoEHMJgNVZdeuXaSkRGfIeKKxWn4ThKw92TwwZRmzln1Hu8a1eHPYWZzVukHQYZVbXHQW5+bmkpWVRU5OTkBRuZOSkkLz5s2pWjW4DiNjTOUdzSvgH/PW8tTs1QjCHf3a8YtzW1E1KXYLMQPtLBaRJGAhsFlVBxR7rDrwCtAD2AVcq6rry/saVatWpVWrVh5Ea0y4TJhgZ0HR9tm3uxg1KZM12w9yaaeTGX1lJ5rVqxF0WJUSjUtDdwDLgTolPPYLYI+qthWRIcCjwLVRiMmY0LN1EKJrx4EjPDR9Oe9+tZkW9Wvwz6E9uei0YKeG8Iqv5zEi0hy4AvhHKbsMAl6O/P4O0Fdi9SK/MTHG1kGIjvwC5ZXP1nPRY3OYungLv7moLe/f2SdukgD4f0bwBHAPUNowumbAJgBVzRORfUAD4LgVmkVkGDAMIM3qAY0BbOxENHyzaS8jJ2ayZPM+erdtwJhBGbRpVCvosDznWyIQkQHAdlVdJCIXlLZbCdt+0HutquOB8eB0FnsWpDEhZmMn/LMvO5c/v7+CCfM30rBWdZ66rhtXdmkas1WJleXnGUFvYKCI9AdSgDoi8pqq3lBknyygBZAlIslAXWC3jzEZEzds7IT3VJX/frmZh6YvZ0/2UYaek85dF59KnZT4rvTzLRGo6r3AvQCRM4LfFUsCAJOBG4HPgKuBDzVs9azGBMTGTnhr1XcHGDkxkwXrdtMtrR6v/KIXnU6J7akhvBL1AWUiMgZYqKqTgReBV0VkDc6ZwJBox2NMmNk6CJWXfTSPp2av4R9z11KzejIP/7gz1/ZsQZUQTA3hlagkAlWdA8yJ/D66yPYc4JpoxGBMeY0YAePHQ34+JCU5l2GefTboqIxXVJX3l33HA5OXsmVfDj/p2Zz/u+w0GtQKz9QQXomLKSaM8dqIEfDcc8fu5+cfu2/JIPw27srmj1OW8uGK7bQ/uTb/vq4bZ6TXDzqswMTFFBPGeC052fnyLy4pCfLyoh+P8caRvHzGf7yWZz5aQ3IV4a6LT+XGc9JjemoIr9h6BMaUU0lJ4ETbTeybt3onoydlsnbnIfp3bsKoAR1pWjfcU0N4xRKBMSVISir9jMCEy/b9OYydtpwp32yhZYNU/nXTGVzQvnHQYcUUSwTGlGDYsOP7CIpuN+GQl1/Aq59v4LH3V3E0v4A7+7Xj133akFLVsnlxlgiMKUFhh7BVDYXTlxv3MPLdTJZt3c/5pzbigYGdaNWwZtBhxSzrLDbGxI09h47yp/dW8MaCTTSpk8LoKztyeUaTuJ0aojxO1Fkc/13lJvT69QORY7d+/YKOyH8TJkB6OlSp4vycMCHoiGJbQYHy9sJN9H38Y95emMUvz2vFB3f3oX/n+J0fyEt2acjEtH79YPbs47fNnu1s/+CDYGLym60zUD7Lt+5n1MRMFm7YQ4+WJ/Hg4Aw6NC1p+RNTGrs0ZGLaif6YC9k/XdfS00ueVbRlS1i/PtrRxK6DR/J4YtYqXvp0PXVSkrm3fweu7t48oaaGKA8bR2BMiNg6AyemqszI3MaYKcvYtj+H63q14J5LT+OkmtWCDi20LBEYE2NsnYHSrd95iPsnL+XjVTvo2LQOz97Qne5pJwUdVuhZZ7GJaX37lm97PBg3zllXoKhEX2cgJzefJz5YxSVPfMKiDXu4/8qOTL6ttyUBj9gZgYlpH3zwww7jvn3jt6MYbJ2B4j5etYPRkzLZsCubK7uewsgrOnBynZSgw4or1llsjIlJ2/blMGbqUqYv2UbrhjUZMyiDc9s1DDqs0LJxBCbUvK6pd3s8q+UPRm5+Af+Yu5a+j81h9vLt3H3xqcy48zxLAj6yS0MmpnldU+/2eFbLH4yF63czcmImK7Yd4ML2jXhgYAZpDVLLfqKpFLs0ZGKa1zX1bo9ntfzRtfvQUR6ZsZy3F2ZxSt0U7h/YiUs6nmyjgj1k4whMaHldU+/2eFbLHx0FBcpbCzfx6MwVHMzJ41d9WnP7Re2oWd2+mqLJ3m0T07yuqXd7PKvl99/SLfsYOTGTrzbupVer+jw4OINTT64ddFgJyTqLTUzzuqbe7fGslt8/B3JyeWDKUq58eh4bd2Xz2DVdeWvYWZYEAmRnBCameV1T7/Z4VsvvPVVl6uKtjJ26jB0Hj3D9mWn8/pLTqJtaNejQEp51FhtjfLd2x0FGT1rKvDU76dysLg8OzqBri3pBh5VQbBxBAgqyBt7q9E2hnNx8Hnt/JZc9MZdvsvYyZlAnJt7a25JAjLFLQ3EoyBp4q9M3hT5asZ3RkzPZtPswP+rWjHv7n0bj2jY1RCyyS0NxKMgaeKvTN5v3HmbMlKW8t/Q72jSqydjBGZzTxkYFB83GESSYIGvgrU4/ceXmF/DPeet44oPVKMo9l7XnlnNbUy3ZrkDHOksEcSjIGnir009M89fuYtSkTFZ9d5B+HU7m/is70qK+TQ0RFpaq41CQNfBWp59Ydh48wm/f/pprx3/OoSP5vPDznvzjxp6WBELGzgjiUJA18FannxjyC5Q3FmzkTzNXcDg3nxEXtOE3F7WjRrWkoEMzFWCdxcaYclmStY+RE5fwTdY+zm7dgLGDO9G2sY0KjnWBjCMQkRQRWSAi34jIUhF5oIR9horIDhH5OnK7xa94TOwZMQKSk0HE+TliROX2C8PYiTDbdziX+ydlMuhv89i8N4cnh5zO678805JAPFBVX26AALUiv1cF5gNnFdtnKPBMeY7bo0cPNeE3fLgq/PA2fHjF9nvtNdXU1OP3SU11tvstyNeOhoKCAn33yyztMXaWtvp/U3X0xCW6N/to0GGZcgIWainfq1G5NCQiqcA8YLiqzi+yfSjQU1Vvc3ssuzQUH5KTIT//h9uTkiAvr/z7hWHsRBit2X6AUROX8tnaXXRtXpcHB3emc/O6QYdlKiCwcQQikgQsAtoCfyuaBIq4SkTOB1YBd6nqphKOMwwYBpBm9YVxoaQv95K2u90vDGMnwuTw0Xye/nA1L8xdS42qSTw4OIPreqWRVMUWiolHZSYCEakOXAWkF91fVceU9VxVzQdOF5F6wLsikqGqmUV2mQK8oapHROTXwMvARSUcZzwwHpwzgrJe18S+pKTS/9KvyH5hGDsRFh8s+477Jy9l897DXNW9Off2P42GtaoHHZbxkZvO4knAICAPOFTk5pqq7gXmAJcV275LVY9E7r4A9CjPcU14Fc4rVNZ2t/uFYexErNu0O5tbXl7ILa8sJLVaEm8NO4vHftLVkkAiKK3zoPAGZJa1TynPawTUi/xeA5gLDCi2T9Miv/8I+Lys41pncfwYPlw1KcnpXE1K+mEHcHn3e+011ZYtVUWcn9HsrA3ytSvrSG6+PvPham0/crqeNnKGPj9njR7Nyw86LOMxKtNZLCLjgadVdUl5EoyIdMG51JOEc+bxtqqOEZExkYAmi8jDwECcs43dOJ3JK050XOssNsY7n367k1ETM/l2xyEu69SE0Vd25JR6NYIOy/igQuMIRGSJiCwGzgW+FJGVIrK4yPYTUtXFqtpNVbuoaoZG+hRUdbSqTo78fq+qdlLVrqp6YVlJwLjnR12723p+r48X1PoGXrc3lmw/kMOdb37FT1+YT26+8tLQM3j+Zz04Zc9/YWI6vF7F+bkuDgdEmB8q7VQBaHmiW2nP8/tml4bK5kddu9t6fq+P57YtXrfZ6/bGirz8Av3X/9ZpxuiZ2u4P0/Wx91bo4aN5zoNrX1N9M1V1Asdub6Y6203oUclLQ6+q6s/K2hYtdmmobH7Utbut5/f6eEGtb+B1e2PB15v2MnLiEjI37+fctg0ZM6gTrRvVOrbDxHTILuFNTG0Jg9dHK0zjk8qOI+hU7GBJWHVPTPOjrt1tPb/XxwtqfQOv2xukfdm5/Om9Fby+YCONalXn6eu6MaBLU0SKjQnILuXNKm27iRulJgIRuRf4A1BDRPYXbgaOEqnpN7HJj7p2t/X8Xh8vqPUNvG5vEFSV/365mYemL2dP9lFuOqcVd13cjtopVUt+QmpaKWcEIR0QYVwrtbNYVR9W1drAn1W1TuRWW1UbqOq9UYzRlJMfde1u6/m9Pl5Q6xt43d5oW/XdAa79++fc/e9vaNkglSm/OZfRV3YsPQkAdB0HScXexKRUZ7uJb6V1HgDdT3Qr7Xl+36yz2B0/6trd1vN7fTy3bfG6zV63NxoO5uTqQ9OWaZt7p2nXB97TN+Zv0Pz8AvcHWPua6rstVSeI89M6iuMGFeksFpGPIr+mAD2Bb3AuDXUB5qvquT7mp1JZZ7ExP6SqvLf0O8ZMWcqWfTlc27MF/3f5adSvWS3o0EyMqNA4AnXq+i8ENuCcAfRU1R5AN2CNP6GaRBLU+IB4s3FXNjf/6wt+/doi6tSoyju/PptHr+4SviSwboKNYQiIm6qh07TIqGJVzRSR032MySSACROc6+3Z2c79DRuOXX8vulyl2/0S0ZG8fP7+8Vr+9tEakqsII6/owNBz0klOCuFS5OsmwIJhkB/5oLM3OPcBWiX4Bx0FbsYRvIEzydxrgAI34Cw4c53/4f2QXRqKD0GND4gXc1fvYPSkpazbeYgrOjdl1ICONKmbEnRYFWdjGHxX2XEENwHDgTsi9z8BnvMoNpOgghofEHbf7c9h7NRlTF28lfQGqbx8cy/6nNoo6LAqz8YwBKrMRKCqOcBfIzdjPBHU+ICwyssv4JXPNvD4rFUczS/grn6n8qs+rUmpGqKBDSdiYxgCdaJJ596O/FwSmWzuuFv0QjTxKKjxAWG0aMMernzmf4yZuoweLU9i1l3nc0e/dvGTBMDGMATsRGcEhZeCBkQjEJNYCjt677vPucyTluZ8uRfvAHa7Xzzac+goj85cwZtfbKJJnRSeu747l2U0+eHUEPGgsEP4m/ucy0GpaU4SsI7iqHDTWXwzMFdVV0cnpBOzzmIT7woKlHcWZfHwjOXsz8nj5t7p3NHvVGpV93WJcRPnKjSOoIh04O8i8q2IvC0iv7HyUe95XSvv9nhBzrlv4wN+aPnW/Vzz98+45z+LadOoFtNuP5f7ruhY+SQQTzX68dQWt/xuc2lDjovfcJabvB3YCOS7fZ7Xt3icYsLrufTdHi/IOff9WDMhzA7k5OrYKUu19b3TtNuY9/XtLzaWb2qIE4mndQbiqS1uedRmKrkewUigN1AL+AqYh3OpaKu3KcmdeLw05HWtvNvjBTnnvo0PcKgq05dsY8zUpWw/cIQhZ7TgnktP4yQvRwXHU41+PLXFLY/aXNlxBD/GWVN4GvAxzgLzOa5f3ZTJ61p5t8cLcs59Gx8A63YeYvSkTOau3knHpnV47oYedE87yfsXiqca/Xhqi1tRaHOZfQSq2h3oCywALgaWiMg8zyIwpdbEV7RW3u3xSptbPxpz7nvd5jDJyc3nr7NWcekTn/D1xr388cqOTL6ttz9JAEqvxQ9jjX48tcWtKLS5zEQgIhk400rcCFwLZAEfehaB8bxW3u3xgpxzP1HHB8xZuZ1Ln/iEJ2ev5rJOTZh9dx+G9m7l7/xA8VSjH09tcSsabS6t86DwhnNJ6B7gHKBqWfv7fYvHzmJV7+fSd3u8IOfc92PNhFi1ZW+2Dn9tobb8v6l64Z8/0nmrd0Q3gHhaZyCe2uKWB23mBJ3FgX6pV+QWr4nAa358yYYhucSao3n5Ov7jb7XDqBl66n3T9enZqzQnNy/osExFzB+u+nqSU7XzepJzv7KimNROlAhshEoc8mPqZrfHHDECnisyJWF+/rH7zz5bsdcOqy/W72bUxExWbDvARac15oGBnWhRP7XsJ5rYs2AErCnyD1vzj93vVcF/2DE09XaZ5aOxJh7LR73mR2lmGEpSY8Wug0d4ZMYK/r0oi2b1anD/lR25uOPJ8Tk1RKJ4I9n58i9OkuC6Cv7DjnIpbGXLR03I+FGaGYaS1KAVFChvfrGJR2eu4NCRPH7dpw23921LajX7bxZ6JSWBE213I4ZKYUv9FyoiU3AWoimRqg70JSJTaX5M3ez2mElJpZ8RxLPMzfsYOTGTrzft5cxW9XlwcAbtTq4ddFjGK5JU+hlBRcXQ1Nsn+lPlL1GLwnhq3Ljjr+dD5Usz3R5z2LDj+wiKbo9HB3Jyeez9Vbzy2Xrq16zG4z/pyo+6NbPLQPGmzbDj+wiKbq+oruOO7yOAwEphS00EqvpxNAMx3vFj6ma3xyzsEB4/3jkzSEpykkC8dRSrKlMWb+XBqcvYcfAIN5zZkt9d0p66qVWDDs34obBD+NvxzpmBJDlJoKIdxRBTU2+7mWuoHfAw0BH4flFUVW3tb2gls85iE7Rvdxxk9KRM/rdmF52b1eXBwRl0bVEv6LCMOaHKTkP9Es4axXnAhcArwKsuXjRFRBaIyDcislREHihhn+oi8paIrBGR+SKS7iKeUHE71XIYpmR2O2V13LS52NS/R9e8yl/eW8llT3zC4qx9jB3UiYm39i5fElgwwqlAeV2cnwtKeRO9nnY41o8XJLdtiac2F1faAIPCG7Ao8nNJkW1zXTxPgFqR36sC84Gziu0zAng+8vsQ4K2yjhumAWVup1oOw5TMbqesjps2lzD1b/ar1fU3D9+td735lW7fn1P+Y84ffvxUwoW34gOTvJ5qOdaPFyS3bYmDNlPJaaj/B5wHvIMzx9Bm4BFVbe822YhIKs701cNVdX6R7e8Bf1TVz0QkGdgGNNITBBWmS0Nua+/DMCWz2/EBcdPmUmq8c6o1J+XqTRU7pttadK/ry2P9eEFy25Y4aHNlLw3dCaTiLErTA/gZzgR0bl44SUS+BrYDs4omgYhmwCYAVc0D9gENSjjOMBFZKCILd+zY4ealY4Lb2vswTMnsdnxAvLRZS6nlTjm6uRIHdVmL7nV9eawfL0hu2xJPbS6Bm2mov1DVg8B+4HZV/bGqfu7m4Kqar6qnA82BXpGZTIsqqcbuB2cDqjpeVXuqas9GjRq5eemY4Haq5TBMyex2yup4aPPna3exPa9xyQ9Wpsa7tJrz4tu9nnY41o8XJLdtiac2l8DNNNQ9RWQJsBhnLYJvRKRHeV5EVfcCc4DLij2UBbSIvE4yUBfYXZ5jxzK3Uy2HYUpmt1NWh7nNOw4c4bdvf82Q8Z/z3L6byZcax+9Q2Rrv0mrOi2/3etrhWD9ekNy2JZ7aXJLSOg8KbzgJ4Lwi988FFrt4XiOgXuT3GsBcYECxfW7l+M7it8s6bpg6i1Xdz9gZhimZ3c4qGrY25+UX6CufrdfO98/Utn+Ypn+euUKzj+T5MzOk2xksvX7tWD9ekNy2JeRtprKdxarau6xtJTyvC/AykIRz5vG2qo4RkTGRgCaLSApOKWo3nDOBIaq69kTHDVNnsYl9i7P2MnJiJouz9nFOmwaMGZRB28a1gg7LGM9VtrN4gYj8XUQuEJE+IvIsMEdEuotI99KepKqLVbWbqnZR1QxVHRPZPlpVJ0d+z1HVa1S1rar2KisJxLOYr6mPM/sO5zJqYiaD/vY/tu7L4ckhpzPhljP9TwKxXrPuR3yxPoYhnscHuORmWsTTIz/vL7b9HJyO3Ys8jSgB+bF+gCmZqjLx682Mm7ac3YeOcuPZ6fz2klOpkxKFqSHczj8f1Dz1fsTndVti/XghZesRxICYr6mPE6u/O8DIiZnMX7ebri3qMW5wBhnN6kYvgFivWfcjvlgfwxAH4wPcqtR6BCJyMvAQcIqqXi4iHYGzVfVFj+NMWLFeUx922UfzePrDNbzwyVpqVk/moR91ZsgZLahSJcozhMZ6zbof8cX6GIY4Hx/glps+gn8B7wGnRO6vwhlkZjwSyzX1Yff+0m1c/PgnPDfnWwZ3a8bsu/vw0zPTop8EIPZr1v2IL9bHMMT5+AC33CSChqr6NlAA348AToD1pqInFmvqw27T7mxuefkLhr26iFrVk3n7V2fzl2u60rBW9eCCivWadT/ii/UxDPE+PsAlN53Fh0SkAZERvyJyFs5UEMYjfqwfkKiO5OXzj7nrePrD1VQR4Q/9T+Om3q2omuTmbx6fuZ1/Pqh56v2Iz+u2xPrxQsrNOILuwNNABpCJM1DsalVd7H94PxSPncXGG5+u2cnISZms3XGIyzOaMGpAR06pV6PsJxqTACrVWayqX4pIH6A9ztxAK1U11+MYjamw7ftzGDd9OZO+3kJa/VReuukMLmxfylxBYbFghLerYXkt1uMDpzQ0iL/0g3rdSnBTNXQNMFNVl4rISKC7iDyoql/6H54xpcsvUF79bD2Pvb+KI3kF3N63HSMuaENK1UosKB4LFow4fn1czT92Pxa+bGM9Poj9sRgxxs2locWq2kVEzsVZsvIvwB9U9cxoBFicXRoyAF9t3MPIiZks3bKf89o1ZMygDFo1rBl0WN5wu25BUGI9Poj9sRgBqNSlIY5VCF0BPKeqk0Tkj14FZ0x57M0+yp/eW8kbCzbSuHZ1/vbT7vTv3ASRAMpB/eJ23YKgxHp8EPtjMWKMm0SwWUT+DvQDHhWR6rgrOzXGM6rKO4uyeHjGCvYdzuXm3q24s187akdjaohok6TS/+KOBbEeHzjX5kv8yzwKYzGCeN1KcvOF/hOcAWWXqbOuQH3g975GZU2furkAABPwSURBVEwRK7cd4Cd//4zfv7OY9AapTLntXEYN6BifSQDcr1sQlFiPD2J/LEaMcVM1lA38t8j9rcBWP4MyBuDQkTyenL2aF+eto3ZKMo9e1ZlregQwNUS0FXa4xmpVTqzHB7E/FiPG2KRzJuaoKjMztzFm6jK27sthyBktuOey06hfs1rQoRkTWpVdj8CYqNmw6xA3/esLhk/4knqp1fjP8HN45Kou7pJAGOaV9zrGBSOcKp7Xxfm5YERsxWdCwU1nsTG+y8nNZ/wna/nbR2tIriKMGtCRG89uSbLbqSHCUL/tdYxe1/OH4T00vrBLQyZwc1fvYPSkpazbeYgBXZoy8oqONKmbUr6DxHD99ve8jtHrev4wvIemwio7jsAYX2zbl8PYacuYtngr6Q1SeeXmXpx/aqOKHSwM9dtex+h1PX8Y3kPjC0sEJury8gv416fr+eusVeQWKHf1O5Vf9WlduakhwlC/7XWMXtfzh+E9NL6wzmITVYs27GbA0/N4cNpyzmhVn1l3nc8d/dpVfn6gMNRvex2j1/X8YXgPjS/sjMBExZ5DR3lkxgreWriJpnVTeP6G7lzaycOpIcJQv+11jF7X84fhPTS+sM5i46uCAuXfizbxyIwVHMjJ4xfntuL2vu2oWd3+BjEmmmwcgQnEsi37ufr5T/m//yyhbeNaTLv9PO7t3yE+k4DX9fduj2d1/8YDcfg/0gTt4JE8/jprFf/6dD11a1TlL9d05aruzeJrhtCivK6/d3s8q/s3HrFLQ8Yzqsq0JVsZO3UZ2w8c4bpeadxzaXvqpcb51BBe19+7PZ7V/ZtysHEExnfrdh5i9KRM5q7eSadT6vD8DT3olnZS0GFFh9f1926PZ3X/xiOWCEyl5OTm8+ycb3l+zrdUT67CAwM7ccNZLUmK9xlCi/K6/t7t8azu33jEOotNhX20cjuX/PUTnpq9mss7N2H23X248Zz0xEoC4H39vdvjWd2/8YidEZhy27L3MGOnLmNG5jZaN6rJ67ecyTltGwYdVnC8rr93ezyr+zcesc5i41pufgEv/W8dT3ywmvwC5fa+7bjlvFZUT46hJQqNMSUKZByBiLQQkY9EZLmILBWRO0rY5wIR2SciX0duo/2Kx1TOF+t3M+CpeTw0fQVnt27AB7/tw60Xto1OEghDrbzV/VeevTeB8fPSUB5wt6p+KSK1gUUiMktVlxXbb66qDvAxDlMJuw4e4eEZK3hnURbN6tVg/M96cEmnJtELIAy18lb3X3n23gTKtzMCVd2qql9Gfj8ALAea+fV6xlsFBcqE+Ru46LGPmfjVZoZf0IZZvz0/ukkAnOvfhV8OhfKzne2xwm2MYWhLUOy9CVRUOotFJB3oBswv4eGzReQbYAvwO1VdWsLzhwHDANLSrDTOb5mb93HfxEy+2bSXs1rXZ+ygDNqdXDuYYMJQK291/5Vn702gfE8EIlIL+A9wp6ruL/bwl0BLVT0oIv2BiUC74sdQ1fHAeHA6i30OOWHtz8nl8fdX8cpn66lfsxp/vbYrg08PeGqIMNTKW91/5dl7EyhfxxGISFWcJDBBVf9b/HFV3a+qByO/TweqikgC1yEGQ1WZ9PVm+j72MS9/tp4bzmrJ7Lsv4Efdmgc/P1AYauWt7r/y7L0JlG9nBOJ8g7wILFfVx0vZpwnwnaqqiPTCSUy7/IrJ/NCa7QcZPSmTT7/dRZfmdXnxxp50aV4v6LCOCUOtvNX9V569N4HybRyBiJwLzAWWAAWRzX8A0gBU9XkRuQ0YjlNhdBj4rap+eqLj2jgCbxw+ms8zH61m/CdrSamaxD2XncZPe6Ul3qhgYxJEIJPOqeo84ITfKqr6DPCMXzGYks1e/h33T15K1p7D/LhbM+7t34FGtat79wLrJiTeX3YLRni3UpgxUWZTTCSQrD3ZPDBlGbOWfUe7xrV445dncXabBt6+SCLWgy8YAWueO3Zf84/dt2RgQsCmmEgAR/MKeHHeOp6avRqAO/q14+beraiW7EOtQCLOkf9GsvPlX5wkwXV50Y/HmBLYegQJ7LNvdzFqUiZrth/kko4nM/rKjjQ/KbXsJ1ZUItaDl5QETrTdmBhjiSBO7ThwhIemL+fdrzbT/KQavHhjT/p2ONn/F07EenBJKv2MwJgQsEQQZ/ILlNfnb+BP760kJzef2y5sy60XtqVGtSh9KXUdd3wfAcR/PXibYcf3ERTdbkwIWCKII4uz9nLfu5ks2byP3m0bMGZQBm0a1YpuEIlYD17YIWxVQyakrLM4DuzLzuUv76/ktfkbaFirOqMGdOTKLk2DHxVsjIkZgaxHYPwxYQKkp0OVKtCypXLnuF30fXwOE+Zv4Maz05l9dx8Gdj0lHEkgnuafj6e2BMXew8DYpaEQmTABhg2D7Mjl940bhaf+WJdu16cxeVwTMprVDTbA8oin8Qbx1Jag2HsYKDsjCJH77juWBAppXjI7Pzo1XEkA4mv++XhqS1DsPQyUJYKQUFU2biy5P2fTphBcBiounsYbxFNbgmLvYaAsEYTApt3Z/OLlhVSpfbjEx0O5Vk9p4wrCON4gntoSFHsPA2WJIIYdycvnmQ9X0+/xj/l87S6G3n6A1NTjzwpSU2FcGEv042n++XhqS1DsPQyUdRbHqP+t2cmoiZms3XmI/p2bMGpAR5rWrcGFpzl9BRs3OmcC48bB9WHsS4un8Qbx1Jag2HsYKBtHEGO278/hwWnLmfzNFtLqp/LAoE5c2L5x0GEZY0LOJp0Lgbz8Al79fAOPvb+Ko3kF3NG3HcMvaENKVZuvxkSRH2tJJOL6FCFjiSAGfLVxDyMnZrJ0y37Oa9eQMYMyaNWwZtBhmUTjRy2/jQ8IBUsEAdqbfZRHZ67kzS820rh2df720+7079wkHKOCTfw5US1/Rb+0/Tim8ZwlggAUFCjvfJnFIzNWsO9wLr/o3Yo7Lz6VWtXt4zAB8qOW38YHhIJ980TZim37GTUxky/W76F7Wj0eHNyZjqfUCTosY/xZSyIR16cIIRtHECUHj+QxbtoyrnhqHqu3H+TRqzrzzq/PsSRgYocftfw2PiAU7IzAZ6rKzMxtPDBlGdv25zDkjBb832WncVLNakGHZszx/Kjlt/EBoWDjCHy0fuch7p+8lI9X7aBD0zo8ODiDHi1PCjosY0wCsnEEUZaTm8/zH3/Ls3O+pWoVYdSAjtx4dkuSk+xKnDEm9lgi8Ngnq3YwelIm63dlM6BLU0Ze0ZEmdVOCDssYY0plicAj2/blMHbqMqYt2UqrhjV59Re9OK9do6DDMsaYMlkiqKS8/AL+9el6/jprFXkFyt0Xn8qwPq2pnmxTQxhjwsESQSUsXL+bkRMzWbHtABe0b8SYgRmkNUgt+4nGGBNDLBFUwO5DR3lkxnLeXphF07opPH9DDy7tdLJNDWGMCSVLBOVQUKC8vXATj8xcwcGcPH51fmtu79uOmjY1hDEmxOwbzKWlW/YxcmImX23cS6/0+owdnEH7JrWDDssYYyrNt0QgIi2AV4AmQAEwXlWfLLaPAE8C/YFsYKiqfulXTBVxICeXx2et4uVP13NSajUeu6YrP+7ezC4DecHmqTcmJvh5RpAH3K2qX4pIbWCRiMxS1WVF9rkcaBe5nQk8F/kZOFVl6uKtjJ26jB0Hj/DTXmn8/tL21Eu1qSE8YfPUGxMzfEsEqroV2Br5/YCILAeaAUUTwSDgFXXmufhcROqJSNPIcwOzdsdB7p+8lLmrd5LRrA7jf96T01vUCzKk+GPz1BsTM6LSRyAi6UA3YH6xh5oBm4rcz4psOy4RiMgwYBhAWpp/09fm5Obz7EdreP7jtVRPrsKYQZ24/syWJFWxy0Ces3nqjYkZvicCEakF/Ae4U1X3F3+4hKf8YBY8VR0PjAdn0jnPgwQ+WrGd0ZMz2bT7MINPP4U/XNGBxrVtagjf2Dz1xsQMXxOBiFTFSQITVPW/JeySBbQocr85sMXPmIrbsvcwD0xZyntLv6NNo5q8/sszOadNw2iGkJi6jju+jwBsnnpjAuJn1ZAALwLLVfXxUnabDNwmIm/idBLvi1b/QG5+Af+ct44nZ6+mQJXfX9qeX57XmmrJNkNoVNg89cbEDD/PCHoDPwOWiMjXkW1/ANIAVPV5YDpO6eganPLRm3yM53sL1u1m5MQlrPruIP06NOb+KzvRor5NDRF1ra63L35jYoCfVUPzKLkPoOg+CtzqVwzF7Tx4hIenr+A/X2bRrF4NXvh5Ty7ueHK0Xt4YY2JSwows/mjFdu5862uyj+Yx4oI23HZRW1KrJUzzjTGmVAnzTdiqYU1Ob1GPUQM60LaxTQ1hjDGFEiYRpDesycs39wo6DGOMiTlWImOMMQnOEoExxiQ4SwTGGJPgLBEYY0yCs0RgjDEJzhKBMcYkOEsExhiT4CwRGGNMghNnup/wEJEdQAkT2bvSENjpYThBsrbEpnhpS7y0A6wthVqqaqOSHghdIqgMEVmoqj2DjsML1pbYFC9tiZd2gLXFDbs0ZIwxCc4SgTHGJLhESwTjgw7AQ9aW2BQvbYmXdoC1pUwJ1UdgjDHmhxLtjMAYY0wxlgiMMSbBxW0iEJEkEflKRKaW8Fh1EXlLRNaIyHwRSY9+hO6V0ZahIrJDRL6O3G4JIkY3RGS9iCyJxLmwhMdFRJ6KfC6LRaR7EHGWxUU7LhCRfUU+k9FBxOmGiNQTkXdEZIWILBeRs4s9HorPBFy1JRSfi4i0LxLj1yKyX0TuLLaPp59LPK9QdgewHKhTwmO/APaoalsRGQI8ClwbzeDK6URtAXhLVW+LYjyVcaGqljYg5nKgXeR2JvBc5GcsOlE7AOaq6oCoRVNxTwIzVfVqEakGpBZ7PEyfSVltgRB8Lqq6EjgdnD8Cgc3Au8V28/RzicszAhFpDlwB/KOUXQYBL0d+fwfoKyISjdjKy0Vb4skg4BV1fA7UE5GmQQcVr0SkDnA+8CKAqh5V1b3FdgvFZ+KyLWHUF/hWVYvPpuDp5xKXiQB4ArgHKCjl8WbAJgBVzQP2AQ2iE1q5ldUWgKsip4fviEiLKMVVEQq8LyKLRGRYCY9//7lEZEW2xZqy2gFwtoh8IyIzRKRTNIMrh9bADuClyKXHf4hIzWL7hOUzcdMWCMfnUtQQ4I0Stnv6ucRdIhCRAcB2VV10ot1K2BZzdbQu2zIFSFfVLsAHHDvTiUW9VbU7zmntrSJyfrHHQ/G5UHY7vsSZ16Ur8DQwMdoBupQMdAeeU9VuwCHg/xXbJyyfiZu2hOVzASByeWsg8O+SHi5hW4U/l7hLBEBvYKCIrAfeBC4SkdeK7ZMFtAAQkWSgLrA7mkG6VGZbVHWXqh6J3H0B6BHdEN1T1S2Rn9txrnn2KrbL959LRHNgS3Sic6+sdqjqflU9GPl9OlBVRBpGPdCyZQFZqjo/cv8dnC/T4vvE/GeCi7aE6HMpdDnwpap+V8Jjnn4ucZcIVPVeVW2uquk4p1UfquoNxXabDNwY+f3qyD4x91eOm7YUuy44EKdTOeaISE0RqV34O3AJkFlst8nAzyMVEWcB+1R1a5RDPSE37RCRJoV9TiLSC+f/2a5ox1oWVd0GbBKR9pFNfYFlxXaL+c8E3LUlLJ9LEddR8mUh8PhzieeqoeOIyBhgoapOxulQelVE1uCcCQwJNLhyKtaW20VkIJCH05ahQcZ2AicD70b+HyYDr6vqTBH5NYCqPg9MB/oDa4Bs4KaAYj0RN+24GhguInnAYWBILP6hEfEbYELkMsRa4KYQfiaFympLaD4XEUkFLgZ+VWSbb5+LTTFhjDEJLu4uDRljjCkfSwTGGJPgLBEYY0yCs0RgjDEJzhKBMcYkOEsExpRTZBbLkmaCLXG7B683WEQ6Frk/R0TiYjF2ExssERgT+wYDHcvcy5gKskRg4k5k9O+0yORimSJybWR7DxH5ODJZ3HuFo7Ijf2E/ISKfRvbvFdneK7Ltq8jP9id63RJi+KeIfBF5/qDI9qEi8l8RmSkiq0XkT0We8wsRWRWJ5wUReUZEzsEZMf5nceambxPZ/RoRWRDZ/zyP3jqToBJmZLFJKJcBW1T1CgARqSsiVXEmGhukqjsiyWEccHPkOTVV9ZzIBHL/BDKAFcD5qponIv2Ah4CrXMZwH86UIDeLSD1ggYh8EHnsdKAbcARYKSJPA/nAKJz5cQ4AHwLfqOqnIjIZmKqq70TaA5Csqr1EpD9wP9CvIm+UMWCJwMSnJcBfRORRnC/QuSKSgfPlPivyRZoEFJ2b5Q0AVf1EROpEvrxrAy+LSDucmR2rliOGS3AmDPxd5H4KkBb5fbaq7gMQkWVAS6Ah8LGq7o5s/zdw6gmO/9/Iz0VAejniMuYHLBGYuKOqq0SkB85cLA+LyPs4s4QuVdWzS3taCffHAh+p6o/EWc50TjnCEOCqyGpTxzaKnIlzJlAoH+f/YXkXRio8RuHzjakw6yMwcUdETgGyVfU14C84l1tWAo0kso6tiFSV4xcmKexHOBdnJsd9ONOTb448PrScYbwH/KbIbJfdyth/AdBHRE4SZ2r0opegDuCcnRjjC/tLwsSjzjidqwVALjBcVY+KyNXAUyJSF+ff/hPA0shz9ojIpzjrQhf2G/wJ59LQb3Gu2ZfH2MjxF0eSwXqg1LVyVXWziDwEzMeZV34Zzsp54KxF8YKI3I4zg6YxnrLZR03CE5E5wO9UdWHAcdRS1YORM4J3gX+qavFFy43xnF0aMiZ2/FFEvsZZ6GYdMb6UookfdkZgjDEJzs4IjDEmwVkiMMaYBGeJwBhjEpwlAmOMSXCWCIwxJsH9fzHpR270gQJMAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "x_points = np.linspace(4, 7, 10)\n", "y_ = -(perceptron.w[0] * x_points + perceptron.b) / perceptron.w[1]\n", "plt.plot(x_points, y_)\n", "\n", "plt.plot(data[:50, 0], data[:50, 1], 'bo', color='blue', label='0')\n", "plt.plot(data[50:100, 0], data[50:100, 1], 'bo', color='orange', label='1')\n", "plt.xlabel('sepal length')\n", "plt.ylabel('sepal width')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### scikit-learn实例" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "import sklearn\n", "from sklearn.linear_model import Perceptron" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'0.23.1'" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sklearn.__version__" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Perceptron()" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf = Perceptron(fit_intercept=True, \n", " max_iter=1000, \n", " shuffle=True)\n", "clf.fit(X, y)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[ 23.2 -38.7]]\n" ] } ], "source": [ "# Weights assigned to the features.\n", "print(clf.coef_)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[-5.]\n" ] } ], "source": [ "# 截距 Constants in decision function.\n", "print(clf.intercept_)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAJYCAYAAACzeOibAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXhU5fnG8e+bBIRhFcQNTAZZlR0jIiDiWlzqgqBgqqK1UbAg2roVf9aqKW2tG1jU1K3VAQQXXFGLioCKGEhYBUFIwo7syxCyvb8/zkRCnIEEZp/7c125Ts47J2eeOZMMN+e884yx1iIiIiIioZcU6QJEREREEoWCl4iIiEiYKHiJiIiIhImCl4iIiEiYKHiJiIiIhImCl4hENWPMCUHcV1NjTEqw9iciUlMKXiJSY8aYBsaYm3zfdzPGPG+MaRSkfd9kjPFUGlp9iG37G2NOr7Q+0BhzrjEm0Gvbp0CrKvv4P2PM/X72/XtjTB1jzOfGmE7GmHuMMQ2NMS8YY/oGqOc6Y0zLQz7Ao2CMudoYM+wQt48OZlAVkeDT//xE5EjsAzKNMeXAe4AXuM4Y0wBoCzwL9AIeAdYE2EdLYLC19n++oJRirS0GioG9lbYrqfjGGFMXKLHWlvp+5gnghkrb9gbygQuNMbustY9Xuc/9vv1jjMkCvvGNlfqpLwV40HdbPV+tjxtjzgf+HuAxXeZ7vKuNMTuAxZVuSwJKrbU/hzZjzASgY0VNlSQDBdbaq6qM3wi8a4yZDnQGlgGLrLV3+G7vD7wSoDYRiQIKXiJSLcaYs4BBQB2cMLIUuB8nhHwMNPaNfYATfrrihI1ArzOm0vfdgOeNMfuB44BGxpjZvtvqVfq+NvB7YC7wG+BtoMgYc4G19jOcwFYE/A24xhjTw1o71xhjfD9bDlxmjHkdOAOYCrQHyiu2sdbuN8bUAdYDFjgW6AfM9J3pqm+tXeU7Jsf4tq/v2+4YoIkxphmw2lrbp9Lxq+87Nj+z1l4f8IBXPVjGpALnAUOAyUBeRYgzxswBbgXKfI9RRKKUgpeIVNdiYIW1dhuAMSYTeB7o5PsqAGYAj+IEAwu8Zq29u2IHvkDSxrf9cmvtDABr7TzgTN821wHdrLX3G2OOA76rHGB827QBhgGXAi8CC40xTXHOsjUHrgW2AS6ckJbq2w6cM0xDccLec8AJvlpvwDlbdRVOSOsL9ADSge99+xsGJBljcnz73G2M6erb1+992zYG2gHNfYHR4Jwx8wK7anLAq/gzsMdaW2SM6Q6UGmN+A6wAdnPwWUIRiVIKXiJSLdbavcBeY8w/OBBIluIECnDOhrUCtgPfAa0Ba4x5HzgJJ9x0BF4DVgENqt6H76zTlcBK39C9wF99tw3AOcuzCueS2knALGATTtC7CVgOzLLW/tl3KfIMXyBrjxOeuvvu/1Ygx1p7mTHmjziXAJ+uVMo+X43Ncc6g7fA9rl7As9baR40xk4AnrLW7gZk4Z8ReBV7CCVhDrLV9jDHHAh9aa3tVepwNcAJhse+4BJICDAaaARf6agG42neMewIdcM506fPfRGKAgpeI1FR74I84l9aKrbVfAhhjlgFjrO8DYI0xLpxgcQNOyGoK/A+YjhOazjHGzLLW7vBtXxvnrFTbSpfgioESY8xgnAA2GCcQjbfWjjPGvIFzJqgQ5zJeBjDAGNMbqO8bfxA4HpiNE7zW4cxLO9TZpxbAFpzg94jvPq4CauFcogTnjNePFT/ge7fkcTjB60Ggje/MWFKl7w0wzlr7qjGmG7C/0vEaCPSx1o7yrRvfY9iHc2k3E3jGGNMc+C1O6LoaZw5ct0M8FhGJIgpeIlJTFWdWNgGPGmPygX8DaytChE8DYA8wGufy4macOVCpwAacs1NeAN87ASfgnEHbU+X+fo8Tks621m6qqMH3TsR1OJPjO+CEsqXAWcAVOOFknrX2B+AH3/2MBLDWvmOMGWOMeQInEFrfuzQXWWtvBJoAdwF/wQlO7+MEnF7ADN9ZrAaVLrv2wpnUXgdnAvx8INda269ibpfv+59fc32XDJsYY94GflUx7tv3P4HbfGfTwJlEv8zJYvQBXgDewQlymb7HKyIxQO0kRKSmUnD+sR8A5AI348wzmm2MedgYc7Nvu5Nx5h69jXMmq2K+U2ucMzWjcSbBg3Mp7zFgDDhnv4wxQ3EuH34HXFgpdIEzL+tOnDNYjwLJ1trfW2vHAXVx5mIN4xDznqy1D1hruwLPAE9aa7v5QlfFnLM+wE6cd2n+A2dO2j7f45kKfF5pd8twws8XOEHwFGCdMeZDYBrQ1RgzA8i21lZ+B+UDOJc891eqazvOWa5xAep+w1r7F9+xORmYhxP4kgM9VhGJHjrjJSI1dTvOZPGqDM7Zpktxzv60xznT9DhOMCvACSUTcSaxb8EJGBVh40NjTGvfvu4H0nAuTX5rrS0HMMYcb63dbK3dAJxkjKmH8+7JtsAc37yuApyzVd2Bn47icZ4M5OC8Y7M98JBv/G2cwPhoxYa+M1/bfGekwLkMOA+43lqb/vMBci43VnzfC2c+m7/LhHcBXxpjbrPWvhCgvt8CL1prre+NDmtr/AhFJOwUvESkWowxLYApOJcCyyrfhBPE/uO7bbUx5hicuVDXWGufr7SPERVzwnzr7xpj/m6t/brSvoy19hHf7Y/hzJuqmAP2oTHmHpzXrv8AG3HOpH1mjOkEjMUJey1wwk83Y8waa+3Lvv2nAMnGmFqArXL2qeI+Sn1BbyUwCWeO12igs2+C/4M4c9yyjTE3WWtnVdpFEs58rP6+7Q7VLuJpYJjvTQvgvJMSnMJKjDFDgPHGmH9XBE8gxTf3axTO5ckzfePrcfqiNeOXPcFEJIooeIlItVhr1wJnB7rdGPMozmXHoTiX3Rb5Ls1VttwYswTnTFfF68+CSrcf4/uqMBP4i29yvcWZwzULJ6Cdaa1d77vvK3Em14/yzd9y45ypOpmDzyjVwgk4twC3GGMqN2cd6Lvv240x3wMfAnnAAGvtRl/rhuE4jVR/MMbMAx4yxlxlrd1rjBmOc7lvr3O47A/GmIqWEpUfX4VzKi4xGmNeAAYCt1XcaK0twGnIWlkDnDOBvwZ+VSm0JeGcHfyyYt6ZiEQnc/BcWBGRI2OMORHnXXrbfevNrbXrwnj/tX2d739eBxpaa7eEqwY/NdX1Ez79bdcE2FedbUUktil4iYiIiISJ3tUoIiIiEiYKXiIiIiJhEhOT64877jjrdrsjXYaIiIjIYc2bN2+LtbaZv9tiIni53W5ycnIOv6GIiIhIhBljCgLdpkuNIiIiImESkuBljDnBGJMb4LYUY0yhMWaG76tTKGoQERERiTahutT4T5zPS/OnMzDRWntfiO5bREREJCoFPXgZY87H6dy8McAmPYHLjTHnAYuA26p+bIeIiEgiKSkpYe3atRQVFUW6FKmBOnXq0KJFC2rVqlXtnwlq8PJ1iv4/4GpgaoDNvgMutNZuMMb8F+cDdd/zs69MIBMgNTU1mGWKiIhElbVr19KgQQPcbjeVPmxdopi1lq1bt7J27VpatmxZ7Z8L9hyv+4Hx1todh9hmobV2g+/7HKCNv42stdnW2nRrbXqzZn7fkSkiIhIXioqKaNq0qUJXDDHG0LRp0xqfpQx28LoQuMMYMwPoaox50c82rxljuhhjkoGrOPgDckVERBKSQlfsOZLnLKiXGq21fSsVMwN40hjzmLX2wUqbPQJMAAzwnrV2ejBrEBEREYlWIevjZa3tZ61dWiV0Ya1dbK3tbK3tZK0dHar7FxERkZpp3bp1tbYbNWpUSOvIy8sjLy8vpPcRKWqgKiIiEmM8HnC7ISnJWXo84b3/p59+OqT7j+fgFRMfGSQiIiIOjwcyM8HrddYLCpx1gIyM4N5Xv379OPPMM1m4cCGffPLJQeMzZswAYN++fQwaNIhdu3bRtGlTpkyZQkrKL+OFv+2Ki4u58cYb2bx5M506deJf//oXDzzwAO+88w4Ar732Gp999hn79+9n6NChrF+/nhYtWvDKK69QVlb2i/0VFRUxcOBA9u7dS+vWrXnllVeCe0CCQGe8REREYsjo0QdCVwWv1xkPtjlz5nD22WcfFLqqWrp0KUlJScycOZObb76ZPXv2cNttt9GvX7+fvx555BG/22VnZ9OxY0dmzpzJhg0bWLhwIWPGjOH+++/n/vvv57PPPgPg3//+Nx07duTLL7+kTZs2vPzyy373t2HDBkaMGMH06dPJz89n06ZNwT8oR0lnvERERGJIYWHNxo9Gx44dGTBgwCG36d69Ox07duTiiy+mTZs29O/fnxdeeOEX21lrf7Hd8uXL+frrr5kxYwY7duxg3bp1dO7c+Rc/u3Tp0p/r6NmzJ9OmTeO22277xf5q1arFiy++yCuvvMK2bdvYt29fcA5EEOmMl4iISAwJ1FM8FL3G69evf9htFixYQO/evfn000/Zvn07s2bNqvZ27dq1Y9SoUcyYMYPHHnvs54bpdevWxes7rWetpUOHDsyZMwdwzsJ16NDB7/5eeuklBg4cyMSJE6lXr16QjkJwKXiJiIjEkKwscLkOHnO5nPFIcLvdjB07ll69erFx40bS09Orvd3vfvc7pk2bRt++fXn++ec55ZRTALjooot4++236d27N7NmzeLWW29lyZIl9O3blxUrVjB06FC/+7vooosYM2YM559/PgDr1q0L23GoLmOtjXQNh5Wenm5zcnIiXYaIiEhIfP/995x22mnV3t7jceZ0FRY6Z7qysoI/sV6qx99zZ4yZZ631m0A1x0tERCTGZGQoaMUqXWoUERERCRMFLxEREZEwUfASERERCRMFLxEREZEwUfASEYmASH/Wnog/0fIh2TVxNLX069cveIVUk4KXiEiYVXzWXkEBWHvgs/YUvqTaVntgqhsmJDnL1eH95Qn1h2TXRDTVUh0KXiIiYRbOz9qTOLTaA3MzwVsAWGc5NzMk4atfv37cc889/OpXv/rFeIV9+/Zx+eWX07dvX66++mpKS0v97isrK4upU6cCMGbMGKZMmYLX62XgwIH07duXO+64I+D9Huo+KtdSVFTE4MGD6dOnD5dffjler5f9+/czZMgQzj33XDIyMiguLvZbX6DtAh2DI6XgJSISZuH8rD2JQwtGQ1mV5F7mdcaDLJgfkj1o0CCmTZsGwMyZM7n00kv9fki2v/v1dx/+ZGdn06VLF2bPns0111zD4sWL/X7Atj+BtqvOMagJNVAVEQmz1FTn8qK/cZHD8gZI6IHGj0IwPyQbYO3atezatYvGjRtTr169gB+SXfV+/d2HP8uWLeOaa64BYOjQoQC8+uqrv/iAbX/8fRB3dY9BTeiMl4hImEXbZ+1JjHEFSOiBxo9CMD8kG6BHjx48/fTTXHHFFQABPyS76v1W9z7at2/Pd999B8Bf//pXXnzxRb8fsO1PoO2qcwxqQsFLRCTMMjIgOxvS0sAYZ5mdrY+AkWrqkgXJVZJ7sssZj4Dqfkg2wKBBg3j66ae5/PLLAQJ+SPaR3sfvfvc75s+fT79+/Zg/fz433HCD3w/Y9qe62x0tfUi2iIhIhNX0Q7JZ7XHmdHkLnTNdXbKgpZJ7JOhDskVEROJdywwFrRilS40iIiJRIBauQMnBjuQ5U/ASERGJsDp16rB161aFrxhirWXr1q3UqVOnRj+nS40iIiIR1qJFC9auXctPP/0U6VKkBurUqUOLFi1q9DMKXiIiIhFWq1YtWrZsGekyJAx0qVFEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEREQkTBS8RERERMJEwUtEEo7HA243JCU5S48n0hWJSKJIiXQBIiLh5PFAZiZ4vc56QYGzDpCREbm6RCQx6IyXiCSU0aMPhK4KXq8zLiISagpeIpJQCgtrNi4iEkwKXiKSUFJTazYuIhJMCl4iklCyssDlOnjM5XLGRURCTcFLRBJKRgZkZ0NaGhjjLLOzNbFeRMJD72oUkYSTkaGgJSKRoTNeIiIiImGi4CUiIiISJgpeIiIiImGi4CUiIiISJgpeIiIiImGi4CUiIiISJgpeIiIiImGi4CUiIiISJgpeIiIiImGi4CUiIiISJgpeIiIiImGi4CUiIiISJgpeIlItHg+43ZCU5Cw9nkhXJCISe1IiXYCIRD+PBzIzwet11gsKnHWAjIzI1SUiEmt0xktEDmv06AOhq4LX64yLiEj1KXiJyGEVFtZsXERE/FPwEpHDSk2t2biIiPin4CUih5WVBS7XwWMulzMuIiLVp+AlIoeVkQHZ2ZCWBsY4y+xsTawXEakpvatRRKolI0NBS0TkaOmMl4iIiEiYKHiJiIhIXMtbs4OBz33Nlj37I12KgpeIiIjEp/Jyy/Nf/sjA575mw84iNu+KfPDSHC8RERGJO5t3FXH35AXMXrmFSzqeyN8GdKaRq1aky1LwEhERkfjyxbLN/HHKAvYWlzJmQCcGn3kKxphIlwUoeImIiEic2F9axj8+Xs5Ls1fT/sQGTBrSkzYnNIh0WQdR8BIREZGYt+qnPYyYmMuS9bu46ew0Hrj0NOrUSo50Wb+g4CUiIiIxy1rLm/PW8uf3llA7JYnsG87g4g4nRrqsgBS8REREJCbtKirhwXcW896C9fQ8tQlPX9eNExvViXRZh6TgJSIiIjEnt3A7Iyflsn5HEX+8uC3D+rUmOSk6JtAfioKXiIiIxIzycsvzM3/kyU9/4ISGdZh8W0/OSGsS6bKqTcFLREREYsKmXUXcPTmPr1Zu5bJOJ/HXAZ1oVDfyvblqQsFLREREot7nyzbxxykL8RaX8vdrOnFtevT05qqJkHxkkDHmBGNM7iFuf8kY840x5sFQ3L+ISCLweMDthqQkZ+nxRLoikeDbX1rGX95fwi2v5nBCwzp8MKIP152ZGpOhC0J3xuufQF1/NxhjBgDJ1tqzjTEvG2PaWGtXhKgOEZG45PFAZiZ4vc56QYGzDpCREbm6RIJp5eY9jJyYy9INuxjay839l7SPyt5cNRH0M17GmPOBvcDGAJv0Ayb7vv8U6BPsGkRE4t3o0QdCVwWv1xkXiXXWWiZ/t4Zfj5vNhp37ePHGdB6+okPMhy4I8hkvY0xt4P+Aq4GpATarB6zzfb8N6B5gX5lAJkBqamowyxQRiXmFhTUbF4kVu4pK+NPbi/hg4QbOPrUpT13XNep7c9VEsC813g+Mt9buOMS11z0cuAxZnwBn3ay12UA2QHp6ug1ynSIiMS011bm86G9cJFbNK9jOnZNy2bCziHt+1Y7bz20VE725aiLYlxovBO4wxswAuhpjXvSzzTwOXF7sAuQHuQYRkbiXlQUu18FjLpczLhJrysot//piJde+8A0AU24/mzvOi42GqDUV1DNe1tq+Fd/7wteTxpjHrLWV3704FZhljDkZuAToGcwaREQSQcUE+tGjncuLqalO6NLEeok1G3cWcdcbeXyzaiuXd3Z6czWsE1u9uWrCWBv+q3jGmGOBi4CZ1tpAk/B/lp6ebnNyckJfmIiIiITN9KWbuOfNBRSVlPOXKzowKL1FzLaJqMwYM89am+7vtog0ULXWbufAOxtFREQkgRSVlPG3act49et8Tj+pIeOu70arZvUjXVZYqHO9iIiIhM3Kzbv5/YRclm3czc29nd5cx6TEfpuI6lLwEhERkZCz1jI5Zw0Pv7eUurWTeXloOue3PyHSZYWdgpeIiIiE1M59Tm+uDxdtoHfrpjx1bVeObxg/vblqQsFLREREQmZewTZGTsxj464i7u3fjtv7tiIpDttEVJeCl4iIiARdWbnluRkreWr6Ck5uXIc3bz+bbqnHRrqsiFPwEhERkaDasHMfd72Rx5xV27iiy8k8dnXHuO7NVRMKXiIiIhI0ny7ZyL1vLaS4tJzHB3Zm4Bnx0ZsrWBS8RERE5KgVlZQx5qPv+c83BXQ4uSHjhnTj1ATpzVUTCl4iIiJyVFZs2s2IiU5vrlv7tOSe/u0SqjdXTSh4iYiIyBGx1jJx7hoe+WAJ9Wqn8MrQMzmv/fGRLiuqKXiJiIhIje30lvDAOwv5aNFG+rQ+jiev7ZKwvblqIinSBYiI1MTw4ZCSAsY4y+HDI12RSOLJyd/GpWNn8emSTTxwSXv+e0sPha5q0hkvEYkZw4fDc88dWC8rO7A+fnxkahJJJGXlln99sZKnp/9Ai2NdvDmsF11PaRzpsmKKsdZGuobDSk9Ptzk5OZEuQ0QiLCXFCVtVJSdDaWn46xFJJBt27uPOSXnMXb2NK7uezGNXdaSBenP5ZYyZZ61N93ebzniJSMzwF7oONS4iwfHJko3c5+vN9cSgLgzo3ly9uY6QgpeIxIzk5MBnvEQk+IpKysj68Htem1NAx+YNGTekOy2PqxfpsmKaJteLSMzIzKzZuIgcuR827ebKZ7/itTkF/O6clrw9rLdCVxDojJeIxIyKCfTZ2c6Zr+RkJ3RpYr1I8FhrmTC3kEfeX0qDOim8evOZ9Gun3lzBouAlIjFl/HgFLZFQ2eEt5v63FvHxko2c0+Y4nri2C8c3UJuIYFLwEhEREeau3saoSbls3r2fP13anlv7nEpSkibQB5uCl4iISAIrLSvn2S9WMvazFZzSxMVbw3rRRb25QkbBS0REJEGt27GPuyblMTd/GwO6NeeRqzpS/xhFg1DS0RUREUlAHy/ewH1vLaK0rJwnr+3CgO4tIl1SQlDwEhERSSBFJWU8+sFSPN8W0rlFI8YO7oZbbSLCRsFLREQkQSzfuJsRE+fzw6Y93Nb3VP5wcTtqp6ilZzgpeImIiMQ5ay2vf1vIYx8spUGdWvz3lh70bdss0mUlJAUvERGROLbDW8y9by7k06WbOLdtM/45qAvNGhwT6bISloKXiIhInPp21VZGvZHHlj37efCy07ild0v15oowBS8REZE4U1pWztjPV/Ls5ytIbeLi7WG96dSiUaTLEhS8RERE4sra7V5GTcojp2A713RvwV+u7KDeXFFEz4SIiEicmLZoA/e9tZByC09f15WrujWPdElShYKXiIhIjNtXXMYjHyxl4txCurRoxNgh3Uhrqt5c0UjNO0QSwIUXgjEHvi68MNIViccDbjckJTlLjyfSFUms+n7DLn797Gwmzi3k9nNbMeX2XgpdUUxnvETi3IUXwmefHTz22WfO+PTpkakp0Xk8kJkJXq+zXlDgrANkZESuLokt1lpem1PAYx9+T6O6tXjttz04p416c0U7Y62NdA2HlZ6ebnNyciJdhkhMMod453gM/PnHJbfbCVtVpaVBfn64q5FYtH1vMfe+tZD/Ld1Ev3ZOb67j6qs3V7Qwxsyz1qb7u01nvEREwqywsGbjIpV98+NW7nojj6179/N/l5/Ozb3c6s0VQxS8RETCLDXV/xmv1NTw1yKxo7SsnGc+W8GzX6ykZdN6vHhTbzo2V2+uWKPJ9SJx7oILajYuoZeVBS7XwWMulzMu4s/a7V6uy57DuM9XMrB7C94f0UehK0YpeInEuenTfxmyLrhAE+sjKSMDsrOdOV3GOMvsbE2sF/8+XLiBS56ZxfKNu3lmcFceH9SFemqIGrM0uV5ERCQKeYtLeeT9pUz6bg1dT2nM2MHdSG3qOvwPSsRpcr2IiEgMWbp+FyMmzmfVlr0M79eKuy5qS61kXaSKBwpeIiIiUcJay3++zuev05bRuG4tXv/tWfRufVyky5IgUvASERGJAtv2FnPvmwuY/v1mzm9/PI8P7ExT9eaKOwpeIiIiEfb1j1u46408tu8t4c+/Pp2hvdyYQ3U/lpil4CUiIhIhJWXlPDN9Bf+asZKWx9Xj5aFn0uFktYmIZwpeIiIiEbBmm5c7J+Uyv3AH16Wfwp+vOB1Xbf2zHO/0DIuIiITZ+wvW86e3FwEwbkg3ft3l5AhXJOGi4CUiIhIm3uJS/vLeUt7IWUO3VKc31ylN1JsrkSh4iYiIhMGS9TsZMTGX1Vv2csd5rRh1oXpzJSIFLxERkRCy1vLq1/mM+WgZx9arhee3Z9FLvbkSloKXiIhIiGzds5973lzI58s2c+Fpx/OPgV1oUq92pMuSCFLwEhERCYGvVjq9uXZ4S3j416dzk3pzCaCLyyIJwOMBtxuSkpylxxO7dUTLYxEJpKSsnH98vIzfvPQtDeqkMPWO3gzt3VKhSwCd8RKJex4PZGaC1+usFxQ46wAZGbFVR7Q8FpFA1mzzMmJiLnlrdjD4zFN46NfqzSUHM9baSNdwWOnp6TYnJyfSZYjEJLfbCShVpaVBfn5s1REtj0XEn3fz1vHgO4vBwJgBnbi8s3pzJSpjzDxrbbq/2xTDReJcYWHNxqO5jmh5LCKV7d1fysPvLWHKvLV0T23MM+rNJYegOV4icS41tWbj0VxHtDwWkQqL1+3k1+Nm8+b8tYw4vzWTbztboUsOScFLJM5lZYGryr8DLpczHmt1RMtjEbHW8tLs1QwY/zV7i0uZcGtP/nBxO1LUEFUOQ78hInEuIwOys515UMY4y+zs8E9GD0Yd0fJYJLFt2bOfW179jkc/WErfts2Ydmdfzm7VNNJlSYzQ5HoREZFqmr1iC3dNzmPnvhIevOw0buiZpjYR8guaXC8iInIUSsrKeeLTH3hh5o+0alaf/97Sg9NOahjpsiQGKXiJiIgcQsHWvYyclMeCNTsY0iOVhy4/nbq1kyNdlsQoBS8REZEA3s1bx+h3FpNkYHxGdy7tdFKkS5IYp+AlIiJSxd79pTz07hLemr+W9LRjeXpwV1ocqzYRcvQUvERERCpZtHYnIyflOpcYL2jDyPNbq02EBI2Cl4iICFBebnn5q9X8/eNlHFf/GCb8ric9T1WbCAkuBS8REUl4P+3ezx+nLODLH37i4tNP4O/XdOgFLkAAACAASURBVObYerUjXZbEIQUvERFJaLNW/MRdbyxgV1EJj17Vkd+clareXBIyCl4iIpKQikvLeeLT5bwwcxVtT6jP67f2oP2J6s0loaXgJSIiCSd/y15GTspl4dqdZJyVyoOXqTeXhIeCl4iIJJSpuesY/c4ikpMMz/+mO/07qjeXhI+Cl4iIJIQ9+0t5aOpi3s5dRw93E54a3JXmjetGuixJMApeIiIS9xau3cHIibkUbvMy6sI2/P489eaSyNBvnUQljwfcbkhKcpYeT6QrOjLBeBzxcixEIqG83PLvmau45rmvKS4tZ1Lm2Yy6sK1Cl0SMznhJ1PF4IDMTvF5nvaDAWQfIyIhcXTUVjMcRL8dCJBI27y7iD5MXMGvFFvp3OJG/XdOJxi715pLIMtbaSNdwWOnp6TYnJyfSZUiYuN1OwKgqLQ3y88NdzZELxuOIl2MhEm5f/vATf5icx+6iUh769elc30O9uSR8jDHzrLXp/m7TGS+JOoWFNRuPVsF4HPFyLETCpbi0nH9+upzsmatod0IDJvyuJ21PaBDpskR+povcEnVSU2s2Hq2C8Tji5ViIhMPqLXu55rmvyZ65iht6pvHu73srdEnUUfCSqJOVBS7XwWMulzMeS4LxOOLlWIiE2tvz13L52FkUbvPywg1n8OhVHalTSw1RJfooeEnUyciA7GxnHpMxzjI7O/YmkwfjccTLsRAJld1FJdz1Rh53T15Ah+aNmHbnOfyqw4mRLkskIE2uFxGRmLRgzQ5GTsplzTYvoy5syx3ntSY5SRPoJfI0uV5EROJGebnl37NW8fgnyzmhYR0m33Y26e4mkS5LpFoUvEREJGZU7s11SccT+duAzjRy1Yp0WSLVpuAlIiIx4Yvlm/nj5AXsLS5lzIBODD7zFPXmkpij4CUiIlFtf2kZj3+8nBdnr6b9iQ2YNKQnbdQmQmKUgpeIiEStVT/tYeSkXBav28VNZ6fxwKWnqU2ExDQFLxERiTrWWt6av46H3l1M7ZQksm84g4vVJkLigIKXiIhEld1FJTw4dTHv5q2n56lNeOq6rpzUqG6kyxIJCgUvERGJGrmF2xk5KZf1O4r448VtGdZPvbkkvih4iYhIxJWXW16YuYonPq3ozdWTM9LUm0vij4KXiIhE1OZdRdw9eQGzV27hsk4n8dcBnWhUV725JD4peImISMR8sWwzf5iyAG9xKX+/phPXpqs3l8S3kAQvY0wT4Awg11q7JRT3ISIisWt/aRl/n7acl79yenM9e31PWh+v3lwS/5KCvUNjzLHAB0AP4AtjTDM/26QYYwqNMTN8X52CXYdIvBg+HFJSwBhnOXx4ZPbh8YDbDUlJztLjqfk+okU8PZZY9ONPe7j6X1/z8lerGdrLzdQ7eit0ScIIxRmvzsDd1to5vhDWHfjEzzYTrbX3heD+ReLG8OHw3HMH1svKDqyPHx++fXg8kJkJXq+zXlDgrANkZFRvH9Einh5LrLHWMmXeWv787hLq1ErixRvTufD0EyJdlkhYGWttaHZsTF/gMeBya+2uKrcNB+4A9gKLgNustaWB9pWenm5zcnJCUqdINEtJcYJSVcnJUBrwLyb4+3C7nYBSVVoa5OdXbx/RIp4eSyzZVVTC6HcW8/6C9Zx9alOeuq4rJzaqE+myRELCGDPPWpvu77ZQzfEywHXAdqDEzybfARdaazcYY/4LXAq8V2UfmUAmQGpqaijKFIl6/gLTocZDtY/CwpqNR7N4eiyxYn7hdkZOzGXDziLu+VU7bj+3lXpzScIK+hwvAOu4A1gIXOFnk4XW2g2+73OANn72kW2tTbfWpjdr9otpYiIJITnAR9IFGg/VPgL93ycW/08UT48l2pWVW/71xUoGPf8NAFNuP5s7zlNDVElsoZhcf58x5kbfamNgh5/NXjPGdDHGJANXAQuCXYdIPKiYe1Td8VDtIysLXK6Dx1wuZzzWxNNjiWabdhVxw0vf8vgny7mk44l8dOc5dE89NtJliURcKM54ZQM3GGNmAsnAWmPMY1W2eQR4DcgDvrHWTg9BHSIxb/x4GDbswNmp5GRnvbqT4oO1j4wMyM525kEZ4yyzs2NzMno8PZZo9dn3m+j/9ExyC3fwj2s6M25INxrWUUNUEQjh5Ppg0uR6EZHot7+0jDEfLePVr/M5/aSGjLu+G62a1Y90WSJhF/bJ9SIiklhWbt7DyIm5LN2wi5t7u7n/kvYck1KDiYQiCULBS0REjpi1lsk5a3j4vaXUrZ3MSzelc8Fp6s0lEoiCl4iIHJGd+0oY/c4iPli4gd6tm/LktV05oaF6c4kcioKXiIjU2LyCbYycmMfGXUXc278dt/dtRZLaRIgcloKXiIhUW1m55bkZK3lq+gpOblyHKbefrTYRIjWg4CUiItWycWcRd72RxzertnJFl5N57OqOahMhUkMKXiIicljTl27injcXsL+0nMcHdmbgGS1wPh1ORGpCwUtERAIqKiljzEff859vCuhwckPGDlFvLpGjoeAlIiJ+rdy8m99PyGXZxt38tk9L7u3fTr25RI6SgpeIiBzEWsuk79bwl/eXUK92Cq8MPZPz2h8f6bJE4oKCl4iI/Gynt4QH3lnIR4s20qf1cTx5bReOV28ukaBR8BIREeBAb65Nu4q4/5L2ZJ5zqnpziQRZUqQLEPHH4wG3G5KSnKXHE5k6hg+HlBQwxlkOHx6bNQTjeOo5iV9l5Zaxn63g2hfmkJxkeHNYL24/twYNUVd7YKobJiQ5y9UR+uUQiQXW2qj/OuOMM6wkjtdft9blshYOfLlczng4DRt2cA0VX8OGxVYNwTieek7i1/odXnvt81/btPs+sCMnzre79hXXbAerXrd2kstaDwe+JrmccZEEBeTYAJnGOLdHt/T0dJuTkxPpMiRM3G4oKPjleFoa5OeHr46UFCgr++V4cjKUlsZODcE4nnpO4tOnSzZy71sLKS4t59ErOzKge/Oa9+aa6gavn18OVxpclR+MMkVijjFmnrU23d9tmuMlUaewsGbjoeLvH/hDjUdrDcE4nnpO4ktRSRlZH37Pa3MK6Ni8IWMHd+PUI+3N5Q3wSxBoXCTBaY6XRJ3U1JqNh0pygHZFgcajtYZgHE89J/Hjh027ufLZr3htTgG/O6clbw/rfeShC8AV4Jcg0LhIglPwkqiTlQUu18FjLpczHk6ZmTUbj9YagnE89ZzEPmstE74t5IpnZ7N1735evflMRl92OrVTjvKfgS5ZkFzllyPZ5YyLyC8FmvwVTV+aXJ94Xn/d2rQ0a41xluGexF1h2DBrk5OdCdzJyZGZxB2MGoJxPPWcxK4de4vt7a/l2LT7PrC/eXGO3bRrX3DvYNXr1r6TZq3HOEtNrJcEhybXi4gkpu/yt3HnxFw2797Pvf3bcWsf9eYSCTVNrhcRSTBl5ZZnP1/JM5/9wClNXLw1rBddTmkc6bJEEp6Cl4hInFm/Yx+j3shj7uptDOjWnEeu6kj9Y/RyLxIN9JcoIhJHPl68kfveWkhpWTlPXtuFAd1bRLokEalEwUtEJA4UlZTx2IdLeX1OIZ1bNGLs4G64j6sX6bJEpAoFLxGRGLd8425GTJzPD5v2cFvfU/nDxe2Ovk2EiISEgpeISIyy1uL5tpBHP1hKgzq1+O8tPejbtlmkyxKRQ1DwEhGJQTu8xdz31kI+WbKJvm2b8cSgLjRrcEykyxKRw1DwEhGJMd+u2sqoN/LYsmc/oy89jd/2aaneXCIxQsFLRCRGlJaVM+7zlYz7fAWpTVy8Paw3nVo0inRZIlIDCl4iIjFg3Y59jJqUy3f527mmewv+cmUH9eYSiUH6qxURiXLTFm3gvrcWUlZuefq6rlzVrXmkSxKRI3TY4GWMOR24AqhdMWatfSSURYmICOwrLuPRD5cy4dtCurRoxNgh3Uhrqt5cIrGsOme8JgN/A9aEuBYREfFZtnEXIybksmLzHm4/txV3X9RWvblE4kB1/oo3AROttV9WfIW6KBE5wOMBtxuSkpylxxOZfUh4WGt57Zt8rnj2K7Z7S3jttz24/5L2Cl3hsNoDU90wIclZrtYfigRfwDNexpgbfd8uAL4wxkwE9gJYa/8bhtpEEp7HA5mZ4PU66wUFzjpARkb49iHhsX1vMfe+tZD/Ld1Ev3bN+OegLhxXX725wmK1B+ZmQpnvD8Vb4KwDtNQfigSPsdb6v8GYmwL8jA138EpPT7c5OTnhvEuRqOB2O0GpqrQ0yM8P3z4k9Oas2sqoSXls3buf+y85jZt7udWbK5ymup2wVZUrDa7KD3c1EuOMMfOsten+bgt4xsta+x/fDze11m6ttLNrg1+iiPhTWFiz8VDtQ0KntKycsZ+tYNwXK3E3rcc7N/WmY3P15go7b4A/iEDjIkeoOpMGplRZvyMUhYjIL6Wm1mw8VPuQ0Fi73ct12XMY+/lKBnZvwQcj+ih0RYorwB9EoHGRIxQweBljzjXG/BlwG2Me8n39HdgevvJEEltWFrhcB4+5XM54OPchwffhwg1c8swslm/czTODu/L4oC7UU0PUyOmSBclV/lCSXc64SBAd6q88H5gBXAVUvJNxH5Ab2pJEpELF5PfRo51Lg6mpTmCqyaT4YOxDgmdfcRmPfLCEiXPX0OWUxowb3I3Upq7D/6CEVsUE+gWjncuLrlQndGlivQRZwMn1P29gzEhr7dgw1eOXJteLSDxYun4XIyfl8uNPB3pz1UpWmwiReHNEk+srtZPYUel7QO0kRERqwlrLf78pIOuj72lUtxav3XIWfdocF+myRCQCDnWpseJ9zEOAQiAH6Aa0ARS8RESqYdveYu59cwHTv9/Meb7eXE3Vm0skYVWnncSN1trMinFjzOfhKExEJNZ9/eMW7nojj+17S3jo8tO5ubcbY9SbSySRVectNDuNMU/idLDvAOwObUkiIrGtpKycZ6av4F8zVtLyuHq8dNOZahMhIkD1gte1wACgJc47Gh8MaUUiIjFszTYvd07KZX7hDq5Nb8HDV3TAVVttIkTEcdhXA2ttKTA5DLWIiMS0Dxau54G3F4GFsUO6cUWXkyNdkohEGf03TETkKHmLS/nLe0t5I2cN3VIbM3ZwN05pot5cIvJLh2on8aS19m5jzBdARbMvg/Mh2eeHpToRkSi3ZP1ORkzMZfWWvdxxXitGXajeXCIS2KHe1Xi3b3le+MoREYkN1lpe/TqfMR8to7GrFp7fnkWv1urNJSKHpkuNIiI1tHXPfu59cyGfLdvMBe2P5/FBXWhSr3akyxKRGHDY4GWM+Q74HpgHzAdyrbV7Ql2YiEg0+nrlFka9kccObwkP//p0buql3lwiUn3VOePVH7gMGASMAQqA00JZlIhItCkpK+ep//3Ac1/+yKnH1ePVm3tw+skNI12WiMSY6swAXQ5cA7wPnGatVeiKYx4PuN2QlOQsPZ7YrWP4cEhJAWOc5fDhwa4yPKLlOUlka7Z5GfT8N4yf8SPXpZ/C+yP6xE/oWu2BqW6YkOQsV+sX7KjoeEafKHtOqnPGqw3OGa9rgWeNMVustWpOE4c8HsjMBK/XWS8ocNYBMjJiq47hw+G55w6sl5UdWB8/Pni1hlq0PCeJ7L0F6xn99iIw8Oz13bi8cxy9/K32wNxMKPP9gnkLnHWAlvoFqzEdz+gThc+JsdYeegNjPsU3twtnftcP4SissvT0dJuTkxPuu004brfzD3tVaWmQnx9bdaSkOGGrquRkKC09murCK1qek0TkLS7lz+8uYcq8tXRPbcwz8diba6rb+YeoKlcaXJUf7mpin45n9InQc2KMmWetTfd3W3U6118c/JIkGhUW1mw8muvwF7oONR6touU5STSL1+1k5MRcVm/dy4jzW3PnBW1IicfeXN4Av0iBxuXQdDyjTxQ+J3H4SiJHKjW1ZuPRXEdycs3Go1W0PCeJwlrLS7NXM2D81+wtLsVz61n84eJ28Rm6AFwBfpECjcuh6XhGnyh8TuL01USORFYWuKpcSXG5nPFYq6NiHlR1x6NVtDwniWDrnv3c8up3PPrBUvq2bca0O/vSq1WcN0TtkgXJVX7Bkl3OuNScjmf0icLnRMFLfpaRAdnZzvwhY5xldnb4J3EHo47x42HYsANnuJKTnfVYmlgP0fOcxLvZK7bQ/5lZfPXjVh65sgP/vvGMxGiI2jIDemQ7810wzrJHtiaCHykdz+gThc/JYSfXRwNNrheRUCgpK+eJT3/ghZk/0qpZfcYN6cZpJ8VJmwgRiZijmlwvIhKPCrd6GTEplwVrdjCkRyoPXX46dWvH2CRAEYk5Cl4iknDezVvH6HcWk2RgfEZ3Lu10UqRLEpEEETB4GWO+AKpehzSAtdaeH9KqRERCYO/+Uh56dwlvzV9LetqxPD24Ky2OjbPeXCIS1QIGL2vteeEsREQklBav28mIibkUbN3LyAvaMPL81vHbJkJEopYuNYpIXCsvt7z81Wr+/vEymtY7hgm/60nPU5tGuiwRSVCHDV7GmGSgO1DXN9TcWjsxpFWJiATBlj37+eOUBcxY/hMXnX4C/7imM8cmQpsIEYla1Tnj9SawG2gJrAeOBRS8RCSqzVrxE3e9sYBdRSU8emUHftMzDWNMpMsSkQRXneB1HDAQmGytvc4YMyvENYmIHLHi0nKe+N9yXvhyFW2Or8/rt/ag/YnqzSUi0aE6wasQuBbYb4x5ANArmIhEpYKtexk5MZcFa3dy/Vmp/N9l6s0lItGlOsHrBqApMA0YgBPCRESiytTcdYx+ZxHJSYbnf9Od/h3Vm0tEok9139XYB3ADK4EfQlaNiEgN7dlfykPvLubt+es4030sTw/uRvPGdQ//gyIiEVCd4DUJ2AwsBC4FhgDXh7IoEZHqWLh2ByMn5lK4zcudF7RhhHpziUiUq07wOt5a+/PlRV9HexGRiCkvt7w0ezX/+GQZzeofw6TMs+nRskmkyxIROazq/NfQa4y53xhzkTFmNLDTGNM31IVJYvN4wO2GpCRn6fHEZh3Dh0NKChjjLIcPD0WVieWn3fsZ+up3ZH30Pee3P56P7jxHoUskFOYOh4kpMME4y7kRegFb7YGpbpiQ5CxXR+gfhCCpzhmvb4FjgF6+9VygHzAzRDVJgvN4IDMTvF5nvaDAWQfIyIidOoYPh+eeO7BeVnZgffz44NaaKL784Sf+MDmP3UWlPHZVRzLOSlVvLpFQmDscVlZ6AbNlB9Z7hPEFbLUH5mZCme+F2FvgrAO0DOM/CEFkrK36Odh+NjKmI9Acp7XEGmvtnlAXVll6errNyckJ511KBLndTsipKi0N8vNjp46UFCdsVZWcDKWlR1tdYikuLeefny4ne+Yq2p5Qn3FDutPuxAaRLkskfk1MccJWVSYZhoTxBWyq2wlbVbnS4Kr88NVRQ8aYedbadH+3Vecjg8YBJ+N0rv8/4O/AFUGtUKSSwsKajUdrHf5C16HGxb/VW5zeXIvW7eQ3PVN58LLTqVNLvblEQspf6DrUeKh4A7zgBhqPAdWZ49XJWnsNsMNa+yHQKMQ1SYJLTa3ZeLTWkRwgGwQal196e/5aLh87i8JtXp7/zRk8dlUnhS6RcDAB/s4CjYeKK8ALbqDxGFCd4PWTMeYh4FhjzE3AxhDXJAkuKwtcroPHXC5nPJbqqJgPVt1xOWB3UQl3vZHH3ZMX0OHkRky78xz6dzwx0mWJJI5WAV6oAo2HSpcsSK7yQpzscsZjVHWC143ATuAbnLNdN4e0Ikl4GRmQne3MpTLGWWZnh3difTDqGD8ehg07cIYrOdlZ18T6Q1uwZgeXj5vNu3nruOvCtkzM7MnJaogqEl49xkPrYQfOcJlkZz2cE+vBmUDfI9uZ04Vxlj2yY3ZiPVRjcr0xJgmoD+zD6WCfY63dHYbafqbJ9SLxr7zc8u9Zq3j8k+Uc3+AYnhnSjTPdahMhIrHnqCbXA1OAV4BfAU2A0cCFwStPRBLd5t1F/GHyAmat2MIlHU/kbwM608hVK9JliYgEXXWCV1Nr7QfGmN9ba/sbY74KeVUikjC+WL6ZP05ewN7iUv56dSeG9DhFvblEJG5VJ3jtNsZMBeYZYy4FwnqZUUTi0/7SMh7/eDkvzl5N+xMbMGlIT9qcoN5cIhLfqhO8BgGnW2vnG2O6ANeFuCYRiXOrftrDyEm5LF63ixvPTuNPl56mNhEikhAOG7ystUXAfN/3C0JekYjELWstb81fx0PvLqZ2ShLZN5zBxR3UJkJEEkd1zniJiBy13UUlPDh1Me/mreeslk14enBXTmqkNhEiklgUvEQk5HILt3PnpDzW7djHHy5qy/DzWpOcpAn0IpJ4FLxEJGTKyy0vzFzFE58u54SGdXgjsyfp6s0lIglMwUtEQmLzriLunryA2Su3cFmnk/jr1Z3Um0tEEp6Cl4gE3RfLNvOHKQvwFpfytwGduO5M9eYSEQEFLxEJov2lZfx92nJe/srpzfXs9T1pfbx6c4mIVFDwEpGg+PGnPYyYkMvSDbsY2svN/Ze0V28uEZEqkkKxU2NME2PMRcaY40Kxf/HP4wG3G5KSnKXHE5l9xJPhwyElBYxxlsOH1+znE+E5sdYyOWcNl4+dzYad+3jxxnQevqJD6ELXag9MdcOEJGe5OgIHZO5wmJgCE4yznFvDXwyIjsehOuJXMI6nnpOQCPoZL2PMscAHwIfAk8aY8621P/nZ7iXgdOBDa+1jwa4j0Xg8kJkJXq+zXlDgrANkZIRvH/Fk+HB47rkD62VlB9bHjz/8zyfCc7KrqITR7yzm/QXr6XlqE56+rhsnNqoTujtc7YG5mVDmOyDeAmcdoGWYDsjc4bCy0i+GLTuw3qMavxgQHY9DdcSvYBxPPSchY6y1wd2hMecC+621c4wx/wT+Z639pMo2A4ArrLVDjTEvA2OstSsC7TM9Pd3m5OQEtc5443Y7/yhXlZYG+fnh20c8SUlxwlZVyclQWnr4n4/352R+4XZGTsxlw84i7r6oLbef2yr0vbmmup1/AKpypcFV+aG97woTU5ywVZVJhiHV+MWA6HgcqiN+BeN46jk5KsaYedbadH+3Bf2Ml7X2S9+d9gV6AI/42awfMNn3/adAH+Cg4GWMyQQyAVJTU4NdZtwpLKzZeKj2EU/8ha5DjVcVr89JWbnl+S9/5Mn//cCJDesw+bazOSPt2PDcuTfAAw80Hgr+Qtehxv2JhsehOuJXMI6nnpOQCdUcL4PzYdrbgRI/m9QD1vm+3wacUHUDa222tTbdWpverFmzUJQZVwJl05pk1mDsI54kB5iiFGi8qnh8TjbtKuKGl77l8U+W07/jiXx05znhC10ArgAPPNB4KJgAvwCBxv2JhsehOuJXMI6nnpOQCUnwso47gIXAFX422QNUfEhb/VDVkUiyssDlOnjM5XLGw7mPeFIxl6q641XF23Py2feb6P/0THILd/CPazrz7JBuNKob5oaoXbIgucoBSXY54+HSKsAvQKBxf6LhcaiO+BWM46nnJGSCHniMMfcZY270rTYGdvjZbB7O5UWALkB+sOtINBkZkJ3tzP0xxllmZ9dsAnYw9hFPxo+HYcMOnOFKTnbWqzOxHuLnOSkqKePh95bw2//kcGKjurw/og/XRqohassM6JHtzDPBOMse2eGd7NtjPLQeduAMl0l21qs7sR6i43GojvgVjOOp5yRkQjG5/lic+VvHAIuBfwFDrLUPVtqmITAL+Ay4BOhprd0ZaJ+aXC8SGSs372HExFy+37CLm3u7ua+/enOJiBxOuCfXbwcuqjL8YJVtdhlj+vm2+8ehQpeIhF9Fb66H31tKnVpJvHRTOhec9oupmCIiUkMR61zvC2iTD7uhiITVzn0l/OmdRXy4cAO9WjXlqeu6ckLDEPbmEhFJIPrIIBH52byCbYycmMfGXUXc278dt/UNQ28uEZEEouAlIpSVW56bsZKnpq/gpEZ1mHL72XRPDWObCBGRBKHgJZLgNu4sYtQbucxZtY1fdzmZrKs70rBOmNtEiIgkCAUvkQT2v6WbuOfNBewvKefxgZ0ZeEaLyLSJEBFJEApeIgmoqKSMMR99z3++KeD0kxoy7vputGpWP9JliYjEPQUvkQSzcvNufj8hl2Ubd3NL75bcd0k7jklRby4RkXBQ8BJJENZaJn23hr+8v4R6tVN4ZeiZnNf++EiXJSKSUBS8RBLATm8JD7yzkI8WbaR366Y8dW1XjldvLhGRsFPwEolzFb25Nu0q4r7+7bmt76kkqTeXiEhEKHiJxKmycsu/vljJM5+toHnjurw5rBddT2kc6bJERBKagpdIHNqwcx+jJuXx7eptXOHrzdVAvblERCIuKdIFSHTxeMDthqQkZ+nxRLoiqalPl2zkkmdmsWjdTv45qAvPDO6q0BVsqz0w1Q0Tkpzl6iP4QwnGPqJFtByPaDim0VBDNNUhv6AzXvIzjwcyM8HrddYLCpx1gIyMyNUl1VNUUkbWh9/z2pwCOjZvyNjB3ThVvbmCb7UH5mZCme8PxVvgrAO0rOYfSjD2ES2i5XhEwzGNhhqiqQ7xy1hrI13DYaWnp9ucnJxIlxH33G4nbFWVlgb5+eGuRmrih027GTEhl+WbdnNrn5bc01+9uUJmqtv5h6wqVxpclR++fUSLaDke0XBMo6GGaKojgRlj5llr0/3dpjNe8rPCwpqNS+RZa5k4dw2PfODrzXXzmZzXTr25Qsob4A8i0Hio9hEtouV4RMMxjYYaoqkO8UtzvORnqak1G5fI2uEtZtjr8/nTO4s4092EaaPOUegKB1eAP4hA46HaR7SIluMRDcc0GmqIpjrELwUv+VlWFrhcB4+5XM64RJe5q7dx6TOzmP79Jh64pD3/ubkHxzdQQ9Sw6JIFyVX+UJJdzng49xEtouV4RMMxjYYaoqkO8UvBS36WkQHZ2c6cLmOcZXa2JtZHk7JyyzPTzZm9NAAAHupJREFUVzA4+xtqpSTx1rBe3HZuKzVEDaeWGdAj25kvg3GWPbJrNmk5GPuIFtFyPKLhmEZDDdFUh/ilyfUiMWL9jn2MeiOPuau3cXW35jxyZQe1iRARiUKaXC8S4z5evJH73lpIaVk5T17bhQHdW0S6JBEROQIKXiJRrKikjMc+XMrrcwrp1LwRY4d0o+Vx9SJdloiIHCEFL5EotXzjbkZMnM8Pm/aQ2fdU/nhxO2qnaFqmiEgsU/ASiTLWWl7/tpDHPlhKgzop/OeWHpzbtlmkyxIRkSBQ8BKJIju8xdz31kI+WbKJvm2b8cSgLjRrcEykyxIRkSBR8BKJEt+u2sqoN/LYsmc/oy89jd/2aak2ESIicUbBSyTCSsvKGff5SsZ9voLUJi7eGtaLzi0aR7osEREJAQUvkQhat2Mfoybl8l3+dgZ0b84jV3ak/jH6sxQRiVd6hReJkGmLNnDfWwspK7c8dV0Xru6m3lwiIvFOwUskzPYVl/Hoh0v/v707j4+qvPc4/v0lgDgIyK6CIQERikgAYwQVpV43XIp1V9pbt6aKF9RarV7kemvltu6iVmy8tlZNiOKCa91aUVxjIAmboEBIFNl3GNmS5/5xhhugCdkm58zyeb9evCbnmZPkN08OM9885+Q3yv+iQpk92mvSpYOVTm8uAEgKBC/ARwtWbNLY/GJ9s2qLfnVyL918Gr25ACCZELwAHzjn9Nzn5fr9m1+pXeuWeuaqbJ1Eby4ASDr8qg00s/Vbdyjn2Zma8Oo8DevVSW/fOJzQBalwjDSlhZRv3m3hmKArChbzUa0sT5qWLuWneLdleUFX1HiJ9FiihBUvoBl9vmStbiwo0dqt23XH2T/SVSfQmwvyQsWiydXbrrJ6O/vxYGoKEvNRrSxPKsyRKsPedrjc25akjNHB1dUYifRYoogVL6AZ7Kqs0oPvLtRlT36uA1ul6pUxJ+ia4b0IXfAszm3YeKJjPqqVjq8OKrtVhr3xeJNIjyWKWPECouy79WHdUFCimeXrdcGQHrpr1FFqQ28u7MlVNmw80TEf1cIVDRuPZYn0WKKIVwMgit6cvVy3vTxbzkmTLh2kUYO6B10SYpGl1hwqLNX/WmIB81EtlOadkqtpPN4k0mOJIk41AlHww45K3f7ybF2fP0u9uhykt8YNJ3Shdr1zGjae6JiPapkTpdTQ3mOpIW883iTSY4kiVryAJvpq+SaNnVKsRau26NqTe+vm049Uy1R+p8F+7L5gfHGut9JjqV7ISLYLyXdjPqrtvui8dLx3Si6U5gWVeLwYPZEeSxSZcy7oGuqUlZXlioqKgi4D2ItzTs98Vq6Jb32l9ge21EMXD9KJfToHXRYAIGBmNtM5l1XTfax4AY2wbusO3fpiqd7/apV+3LeL7r8oU50OOiDosgAAMY7gBTTQp4vX6KbnS7Ru6w5NOKe/rjohXWa0iQAA1I3gBdTTzsoqTXr/G/1p+iJldGqjp35xrAZ0bx90WQCAOELwAurh23Vh3VBQrFkVG3RxVg/deS69uQAADccrB1CHN2Z/r9tfniPRmwsA0EQEL6AW4R27dNfr81Xw5bcadPjBeuTSwUrrFKr7EwEAqAXBC6jB/O83aeyUWVqyZqvGjOitm06jNxcAoOkIXsAenHN6+tOl+sNbC3RwqKWeu/o4nXAEvbkAANFB8AIi1m3doVumluofC1bplH5ddd+FA+nNBQCIKoIXIOnTRWt04/Ml2hDeqTvP7a8rjqc3FwAg+gheSGo7K6v00Htfa/KHi5XRuY3+euWxOuowenMBAJoHwQtJ69t1YY0rKFZxxQZdknW47vxJf4Va8V8CANB8+DMtJKXXSr/XWZNmaNHKLXr0ssG658KBhK66lOVJ09Kl/BTvtiwv6IqCEytzUThGmtJCyjfvtnBMMHXEynwAcYBXGiSV8I5duvPVeZo68zsNTvN6cx3ekd5cdSrLkwpzpMqwtx0u97YlKWN0cHUFIVbmonCMtGhy9barrN7Ofty/OmJlPoA4Yc65oGuoU1ZWlisqKgq6DMS5ucs2alxBscrWbNX1I47QDaf2oTdXfU1L915Q9xXqKZ231O9qghUrczGlhRe29mWp0mW7/KsjVuYDiCFmNtM5l1XTfax4IeE55/SXT5bqnr8vUIc2LZV3zXE6vje9uRokXNGw8UQWK3NRU+ja33hziZX5AOIEwQsJbe2W7frN1FJ9sHC1Tv1RV917YaY6tmkVdFnxJ5RWy6pGmv+1BC1W5sJSa1/x8lOszAcQJzjPgoT1yaI1OnPSDH2yeK1+95Oj9OS/ZxG6GitzopS6z7VwqSFvPNnEylz0zmnYeHOJlfkA4gQrXkg4Oyur9OB7X+uJDxerV+c2+tuV2ep/WLugy4pvuy+SLh3vnUIKpXkvrMl48XSszMXuC+gX53orX5bqhS4/L6yXYmc+gDjBxfVIKBVrwxpbUKzSbzfosuzDNeEcenMBAPzFxfVICq+WLNP4V+bKTPrT5UN09sBDgy4JAIC9ELwQ97Zu36U7X5unF2d+p2N6dtCkSwepRwd6cwEAYg/BC3Ft7rKNGjulWEvXbtW4U47QuH/roxb05gIAxCiCF+JSVZXTXz4p0z1vL1CnNgco/5qhGta7U9BlAQCwXwQvxJ01kd5c0xeu1mn9u+neCwaqA20iAABxgOCFuDLjm9W66flSbdq2U78fdZR+NrSnzCzosgAAqBeCF+LCjl1VeuC9hfrzh0vUp+tBeu6abPU7hN5cAID4QvBCzCtfu1XjphSr9LuNuvy4NE04u78ObOXz26IAABAFBC/EtGnFy3THtLlKMenx0UN01tH05gIAxC+CF2LSlu279F+vztXLs5bp2PQOevjSwep+8IFBlwUAQJMQvBBz5ny3UWOnzFLFurBu+Lc+GnvKEfTmAgAkBIIXYkZVldNTH5fp3ncWqPNBByj/l0M1tBe9uQAAiYPghZiwevN23Ty1VB99vVpnHNVN91wwUAeH6M0FAEgsnL9B4D78erVGTvpIXyxZq7vPG6AnfnYMoQvNqyxPmpYu5ad4t2V58VlDLDwOAA3CihcCs2NXle5/d6FyP1qiI7sdpLxrhqrvIW2DLguJrixPKsyRKsPedrjc25akjNHxU0MsPA4ADcaKFwJRtmarLnziU+V+tESjj0vTq9efSOiCP0rHV4eV3SrD3ng81RALjwNAg7HiBd+9POs7TZg2Vy1SU/TEz47RmQMOCbokJJNwRcPGY7WGWHgcABqM4AXfbNm+SxOmzdUrxcuUnd5RD106iN5c8F8ozTstV9N4PNUQC48DQINxqhG+mP3dBp39yAy9WrJMN57aR/m/PI7QhWBkTpRSQ3uPpYa88XiqIRYeB4AGY8ULzaqqyunJGUt03zsL1bXtASrIGabsjI5Bl4VktvvC89Lx3mm5UJoXVvy8ID0aNcTC4wDQYOacC7qGOmVlZbmioqKgy0ADrdq8TTe/UKoZ36zRmUcdoj9ecDRtIgAACc/MZjrnsmq6jxUvNIvpC1fpN1NLtXnbLk386QBdnp0mMwu6LAAAAkXwQlRt31Wp+95eqP/9uEx9u7VV/i+H6shutIkAAEAieCGKlqzeonEFxZq7bJN+PrSnxp/9I7VumRp0WQAAxAyCF5rMOaeXZy3ThFfnqmVqiv7882N0xlH05gIAYF8ELzTJ5m07dce0uXq15HtlZ3TUw5cM0mG0iQAAoEYELzRaybcbNG5Ksb5bH9avTztS1//4CKWmcAE9AAC1IXihwaqqnHJnLNH97yxUt3at9cKvhikrnd5cAADUheCFBlm1aZt+/UKpPl60RmcdfYj+8NOBah9qGXRZAADEBYIX6u2DBat089RShXfs0h/OP1qXHns4vbkAAGgAghfqtH1Xpe59e6Ge+rhM/Q5pq0cvG6o+9OYCAKDBCF7YryWrt2jslGLN+36TfjGsp24/i95cAAA0FsELNXLO6cWZ3+nO1+apVYsU5f78GJ1Oby4AAJokJdpf0Mzam9nfzexdM3vFzP7lXZHNrIWZVZjZ9Mi/o6NdBxpv07aduqGgRLe8OFsDe7TX2zecROiKd2V50rR0KT/Fuy3LC7qiYDV1PpjPxMTPFT5ojhWv0ZIedM69Z2aTJZ0p6bV99hkoaYpz7rfN8P3RBMUV6zWuoFjfb9im35x+pK4bQW+uuFeWJxXmSJVhbztc7m1LUsbo4OoKSlPng/lMTPxc4ZOor3g55x53zr0X2ewiaVUNuw2VdI6ZFZrZU2bGKc+AVVU5PT59kS564jNVVUkv/Gqo/uOUPoSuRFA6vvrFZLfKsDeejJo6H8xnYuLnCp80W+Axs2GSOjjnPq/h7i8lneqcW25mz0g6S/usiplZjqQcSUpLS2uuMiFp5aZt+vULJfpk0VqdffSh+p/zj1b7A+nNlTDCFQ0bT3RNnQ/mMzHxc4VPor7iJUlm1lHSo5KuqmWX2c655ZGPiyT12XcH51yucy7LOZfVpUuX5igTkv65YKVGTpqhmeXrdc8FR+uxywcTuhJNqJZfXGobT3RNnQ/mMzHxc4VPmuPi+laSpkq63TlXXstuz5pZppmlSjpPUmm068D+bd9Vqd+9Pk9XPV2kbu1a642xJ+qSY9NoiJqIMidKqaG9x1JD3ngyaup8MJ+JiZ8rfNIcpxqvljRE0ngzGy/pA0ktnXN37LHPXZLyJZmk15xz7zdDHajFolVbNG5KseYv36Qrjk/XbSP70Zsrke2+MLh0vHfaJJTmvZgk6wXDTZ0P5jMx8XOFT8w5F3QNdcrKynJFRUVBlxH3nHOaWuT15mrdMkX3XZipU/t3C7osAAASipnNdM5l1XQff02YJDZt26n/fHmO3pi9XMN6ddJDlwzSIe1bB10WAABJheCVBGaWr9cNBcVavnGbbjmjr649uTdtIgAACADBK4FVVjk98eFiPfje1zq0fWtNvXaYhqR1CLosAACSFsErQa3YuE03PV+iz5as1TkDvd5c7VrTJgIAgCARvBLQ+/NX6pYXS7VtZ5XuvWCgLsrqQZsIAABiAMErgWzbWak//n2Bnv50qfof2k6PXj5YvbscFHRZAAAgguCVIBat2qz/yC/WghWbdeUJXm+uA1rQmwsAgFhC8Ipzzjk9/+W3+u/X5ynUqoX+ckWWTulHby4AAGIRwSuObfzB68315pzlOuGITnro4kHq2o7eXAAAxCqCV5yaWb5O46aUaMWmbbr1zL669qTeSqE3FwAAMY3gFWcqq5wmT1+kh97/RocdTG8uAADiCcErjizf+INuer5Eny9Zp59kHqa7fzqA3lwAAMQRgleceHfeCt360mzt2FWl+y4cqAuPoTcXAADxhuAV47btrNQf3vpKf/usXEcd1k6PXEZvrrhSlieVjpfCFVIoTcqcKGWMDroqFI6RFudKrlKyVKl3jpT9eNBVAUgCBK8Y9s3KzRo7xevNdfWJGbr1zL705oonZXlSYY5UGfa2w+XetkT4ClLhGGnR5OptV1m9TfgC0MxSgi4A/8o5p/wvKnTuYx9r9ebt+usVx2rCOf0JXfGmdHx16NqtMuyNIziLcxs2DgBRxIpXjNkY3qnbX5mtt+as0IlHdNaDF2fSmytehSsaNg5/uMqGjQNAFBG8YkjR0nW6oaBEKzdt020j+ylneC96c8WzUJp3erGmcQTHUmsOWcaKMoDmx6nGGFBZ5TTp/W908Z8/U2qK6cXrjte1J9MQNe5lTpRSQ3uPpYa8cQSnd07DxgEgiljxCtjyjT/ohoISFZat06hBh+nu8waoLb25EsPuC+j5q8bYsvsCev6qEUAAzDkXdA11ysrKckVFRUGXEXXvzFuh30Z6c/1+1ACdP6Q7vbkAAIhzZjbTOZdV032seAVg285K3f3mfD33eYUGdG+nRy8boozObYIuCwAANDOCl8++XrlZY/OLtXDlZv1yeIZuOaOfWrXgUjsAAJIBwcsnzjnlF1bortfnq23rFnr6ymM1om/XoMsCAAA+Inj5YEN4h257aY7enrdCw/t01gMXZ6prW3pzAQCQbAhezaywbJ1uLCjWqs3b9Z9n9dM1J9KbCwCAZEXwaia7Kqv02AeL9Mg/vtHhHUN66brjlXn4wUGXBQAAAkTwagbLNvygmwpKVLh0nc4f3F13nTdABx3AVAMAkOxIA1H29tzl+u1Lc7SrskoPXpyp84f0CLokAAAQIwheUbJtZ6V+/8Z85X1RoYE92uuRSwcrnd5cAABgDwSvKFi4YrPGTpmlr1du0a9O6qWbT+9Lby4AAPAvCF5N4JzTc19U6O435qtt65Z65qpsnXRkl6DLAgAAMYrg1Ugbwjt064uz9e78lTrpyC564KJMdWl7QNBlAQCAGMb5sEb4YslajZw0Qx8sXKXxZ/1IT19xLKFrD3l5Unq6lJLi3eblBV1RnCvLk6alS/kp3m0ZE9pkzClqwnEBH7Di1QC7Kqv0yD8X6bF/fqO0jiG9fN0JOrpH+6DLiil5eVJOjhQOe9vl5d62JI0eHVxdcassTyrMkSojExou97YlKYMJbRTmFDXhuIBPzDkXdA11ysrKckVFRYHW8N36sG4sKFFR+XpdMKSHfjfqKHpz1SA93Qtb++rZU1q61O9qEsC0dO8FYF+hntJ5S/2uJjEwp6gJxwWiyMxmOueyarqP5FAPf5+zXL99abYqq5wevmSQzhvcPeiSYlZFRcPGUYdwLRNX2zjqxpyiJhwX8AnXeO3HDzsqdfvLc3Rd3ixldG6jt24YTuiqQ1paw8ZRh1AtE1fbOOrGnKImHBfwCcGrFl8t36RzH/tYUwordO3JvTX12uPVsxMNUesycaIUCu09Fgp542iEzIlS6j4TmhryxtE4zClqwnEBnxC89uGc0zOfLdWoP32iDeGdevbqbN02sh8NUetp9GgpN9e7psvMu83N5cL6RssYLWXneteZyLzb7Fwu9m0K5hQ14biAT7i4fg/rt+7QLS/O1vtfrdSIvl10/0WZ6nwQbSIAAED9cXF9PXy2eK1uer5Ea7du14Rz+uvK49OVkmJBlwUAABIIwUvS0jVb9bOnvlBax5Be+cUJGtCd3lwAACD6CF6S0ju30cOXDNIp/bqqDb25AABAMyFlRJybeVjQJQAAgATHn+oBAAD4hOAFAADgE4IXAACATwheAAAAPiF4AQAA+ITgBQAA4BOCFwAAgE8IXgAAAD4heAEAAPiE4AUAaJqyPGlaupSf4t2W5SV3HcB+8JZBAIDGK8uTCnOkyrC3HS73tiUpY3Ty1QHUgRUvAEDjlY6vDju7VYa98WSsA6gDwQsA0HjhioaNJ3odQB0IXgCAxgulNWw80esA6kDwAgA0XuZEKTW091hqyBtPxjqAOhC8AACNlzFays6VQj0lmXebnev/Be2xUgdQB3POBV1DnbKyslxRUVHQZQAAANTJzGY657Jquo8VLwAAAJ8QvAAAAHxC8AIAAPAJwQsAAMAnBC8AAACfELwAAAB8QvACAADwCcELAADAJwQvAAAAnxC8AAAAfELwAgAA8AnBCwAAwCcELwAAAJ8QvAAAAHxC8AIAAPAJwQsAAMAnBC8AAACfELwAAAB8QvACAADwCcELAADAJwQvAAAAnxC8AAAAfELwAgAA8AnBCwAAwCcELwAAAJ8QvAAAAHxC8AIAAPAJwQsAAMAnBC8AAACfELwAAAB8QvACAADwCcELAADAJwQvINaV5UnT0qX8FO+2LC/oigAAjdQi2l/QzNpLKpCUKmmrpEuccztq2O8pSf0lvemcuzvadQAJoSxPKsyRKsPedrjc25akjNHB1QUAaJTmWPEaLelB59zpklZIOnPfHczsfEmpzrlhknqZWZ9mqAOIf6Xjq0PXbpVhbxwAEHeivuLlnHt8j80uklbVsNsISS9EPn5X0omSvtlzBzPLkZQjSWlpadEuE4gP4YqGjQMAYlqzXeNlZsMkdXDOfV7D3W0kLYt8vE5St313cM7lOueynHNZXbp0aa4ygdgWquWXjtrGAQAxrVmCl5l1lPSopKtq2WWLpAMjHx/UXHUAcS9zopQa2nssNeSNAwDiTtQDj5m1kjRV0u3OufJadpsp7/SiJGVKWhrtOoCEkDFays6VQj0lmXebncuF9QAQp6J+jZekqyUNkTTezMZL+kBSS+fcHXvsM03SDDM7TNJISUOboQ4gMWSMJmgBQIJojovrJ0uaXMc+m8xshKTTJN3rnNsY7ToAAABiTXOseNWLc269qv+yEQAAIOFxUTsAAIBPCF4AAAA+IXgBAAD4hOAFAADgE4IXAACATwheAAAAPiF4AQAA+ITgBQAA4BOCFwAAgE8IXgAAAD4heAEAAPiE4AUAAOATghcAAIBPCF4AAAA+IXgBAAD4hOAFAADgE4IXAACATwheAAAAPiF4AQAA+ITgBQAA4BOCFwAAgE/MORd0DXUys9WSyn34Vp0lrfHh+yQL5jO6mM/oY06ji/mMLuYzuvycz57OuS413REXwcsvZlbknMsKuo5EwXxGF/MZfcxpdDGf0cV8RleszCenGgEAAHxC8AIAAPAJwWtvuUEXkGCYz+hiPqOPOY0u5jO6mM/oion55BovAAAAn7DiBQAA4BOCF6LCzLqZWXEt97Uwswozmx75d7Tf9SE51ffYM7OSPfY5ze86ATN73MzOreU+nkMTSFIGr/2FhMj9T5nZZ2Z2h591xbn7JR1Yy30DJU1xzo2I/JvjY11xpQFB4Xdm9qWZ/cnvGuNMnceemXWStGCPfd7zv8z4s7+gELmf59F6MrPhkg5xzr1eyy48h9aTmV23x/NniZn9uZb9Ajs+kzJ4aT8hwczOl5TqnBsmqZeZ9fG1sjhkZqdI2ippRS27DJV0jpkVRg72Fv5VF3fqExSOkXSipGxJq8zsVL+LjCP1OfaOk5RtZp+a2TQza+tzjXGnrqDA82j9mVlLSU9KWmpmo2rZjefQenLOTd79/Clphry53UvQx2fSBa96hIQRkl6IfPyuvBc41MLMWkmaIOm2/ez2paRTnXPZklpKOsuP2uJUfZ5gT5b0kvP+MuYdScN9rTC+1OfYWyLpDOfc8ZJmS7rSx/riTj2DwgjxPFpf/y5pvqR75f0CMLaGfXgObSAz6y6pm3OuqIa7RyjA4zOpglc9Q0IbScsiH6+T1K2564pzt0l63Dm3YT/7zHbOLY98XCSJ335rV58nWI7R+qvPsbdE0qI69kG1+gQFjtH6Gywp1zm3QtJzkn5cwz48hzbc9ZIm13JfoMdnUgUv1S8kbFH1aciDlHxz1FCnSrrezKZLGmRm/1vDPs+aWaaZpUo6T1KpnwXGmfo8wXKM1l99jr2JknZfq3RhLfugWn2CAsdo/S2S1CvycZZqfl9inkMbwMxS5B2X02vZJdDjM9n+M9QnJMxU9bJjpqSl/pQWn5xzJ+1xPr1E0oNmdvc+u90l6dnI/Z855973ucx4Up8nWI7R+tvr2JM0q4b/9w9KGm9mcyVtl/Q3f0uMO/UJChyj9feUpB+b2UeSxkh6kefQJhsu6QtXe6PSQI/PpG2gGglfYyRd7py7Y4/xdvIuyPuHpJGShjrnNgZSJJKOmQ2QlC/JJL0m6QFJ9zrnrtljnxR5x2iRpDMlnemcKwugXCShyB8f/EXe6ZmWkn4jaSTPo4gVZvY/koqccy+bWX/F2Ot80gav/TGzDpJOk/RRZDkdiClmdqCksyXNcs4tCboeYF88jyKWBXl8ErwAAAB8kmzXeAEAAASG4AUAAOATgheAhBX5I5qo7VfPr5VuZiP22L7CzK6I1tcHEN8IXgAQXenyOmMDwL/g/Z4ABC7yV5pTJbWTtFbSRZJaSXpGUldJc5xz15vZf8t7b8WQpNWSLpXk5PU46ilpjaQLnXM7m1CLScqVdGTke1wi6efy+v1kSjpE0sWS5slrINpbXhfs8si/KyUdHFn1uijyZTPN7J+7P9c5N7ex9QGIb6x4AYgF/SVVOedOkvRXed2kcyTNjYwdamYDI/vOcM6dLGmlpFGSOkl6U957WG6SNKSJtYyS1DLyPSrkte2QpGMlnSHpj5J+IqmDpK7OuaGSMpxzv3bOTZJ0o6SnI42FV9fyuQCSFMELQCyYJWmumb0rL6CEJfWV9NPI9Ve9JHWP7Dszcjtb3mm9nZLOkbdi1kvVbwXSWH0lDYt835NU/T5uUyIraRXyVuPCkg4wsy8k5dXxNff9XABJiuAFIBZkSvrEOXe6vJWk4ZIWSno48nZUd8gLLZKUHbkdLO/ta86XNDdyu0xNt1BSQeT73ijvDaElaes++2VLesU5d5xz7oE9xn+Qdyp092nLmj4XQJIieAGIBUsljTOzT+VdB1Uk6UlJIyPvYXetpG8j+x4bWY06WNIbkj6Rdx3Wx5I6qnplrLFek3SYmX0o6W7V/F6EkrRA0q/N7AMze8nMhkfGiyX1NbMZkboA4P/RuR5A3IhcXD/dOTc94FJkZmdLulXSDnmrXPnOuYJgqwIQ6wheAAAAPuFUIwAAgE8IXgAAAD4heAEAAPiE4AUAAOATghcAAIBPCF4AAAA++T+bd1dMFHCtAQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# 画布大小\n", "plt.figure(figsize=(10,10))\n", "\n", "# 中文标题\n", "plt.rcParams['font.sans-serif']=['SimHei']\n", "plt.rcParams['axes.unicode_minus'] = False\n", "plt.title('鸢尾花线性数据示例')\n", "\n", "plt.scatter(data[:50, 0], data[:50, 1], c='b', label='Iris-setosa',)\n", "plt.scatter(data[50:100, 0], data[50:100, 1], c='orange', label='Iris-versicolor')\n", "\n", "# 画感知机的线\n", "x_ponits = np.arange(4, 8)\n", "y_ = -(clf.coef_[0][0]*x_ponits + clf.intercept_)/clf.coef_[0][1]\n", "plt.plot(x_ponits, y_)\n", "\n", "# 其他部分\n", "plt.legend() # 显示图例\n", "plt.grid(False) # 不显示网格\n", "plt.xlabel('sepal length')\n", "plt.ylabel('sepal width')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**注意 !**\n", "\n", "在上图中，有一个位于左下角的蓝点没有被正确分类，这是因为 SKlearn 的 Perceptron 实例中有一个`tol`参数。\n", "\n", "`tol` 参数规定了如果本次迭代的损失和上次迭代的损失之差小于一个特定值时，停止迭代。所以我们需要设置 `tol=None` 使之可以继续迭代：" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAJYCAYAAACzeOibAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXxU1f3/8dfJJBCGfRMRTIKCGzsEAmqRWm2ttXUDt4CAQJBoF/v91mq19udCbW2/rbUtSsoqGajQulSt1rrgCoSEXXYhiez7GgIkOb8/7kQgzmSBmbkzk/fz8cjj5p65c+dz750k75x75l5jrUVEREREwi/B7QJERERE6gsFLxEREZEIUfASERERiRAFLxEREZEIUfASERERiRAFLxGJasaYdiFcV2tjTGKo1iciUlcKXiJSZ8aYpsaYEf7vextjXjDGNA/RukcYY3ynNG2qZtnrjDGXnTI/xBhzlTEm2O+2d4ALq6zjl8aYhwKs+35jTLIx5n1jTHdjzM+MMc2MMZOMMYOC1HO7MaZTtRt4FowxNxtjxlfz+COhDKoiEnr6z09EzsRRIMsYUwH8CygBbjfGNAUuAv4CXA48AXwZZB2dgDustf/1B6VEa+1x4Dhw5JTlTlR+Y4xpBJyw1pb5n/N/wPBTlr0CKASuMcYctNb+rsprHvOvH2PMBGC+v60sQH2JwKP+xxr7a/2dMeZq4LdBtul7/u3dZIzZD6w85bEEoMxa+1VoM8bMArpV1nQKD1Bkrb2pSvvdwGvGmHeBHsAaYIW19j7/49cB04LUJiJRQMFLRGrFGJMBDAWSccLIKuAhnBDyNtDC3/YGTvjphRM2gv2eMad83xt4wRhzDGgDNDfGfOJ/rPEp3zcA7gfygGHAy0CpMeZb1tr3cAJbKfAb4FZjTH9rbZ4xxvifWwF8zxiTC/QFXgUuASoql7HWHjPGJANbAQu0BAYDH/l7uppYazf690lD//JN/Ms1BFoZY9oCm6y1V56y/5r4981XrLV3Bd3hVXeWMSnAN4E7gTnA0soQZ4xZAIwByv3bKCJRSsFLRGprJbDeWrsXwBiTBbwAdPd/FQHzgCdxgoEFZlprf1q5An8g6eJffq21dh6AtbYA6Odf5nagt7X2IWNMG2DRqQHGv0wXYDxwPTAZWG6MaY3Ty9YBuA3YC3hxQlqKfzlwephG4oS954F2/lqH4/RW3YQT0gYB/YF0YLV/feOBBGNMvn+dh4wxvfzrut+/bAvgYqCDPzAanB6zEuBgXXZ4Fb8CDltrS40xfYAyY8wwYD1wiNN7CUUkSil4iUitWGuPAEeMMc9wMpCswgkU4PSGXQjsAxYBnQFrjHkdaI8TbroBM4GNQNOqr+HvdboR2OBvehD4tf+xW3B6eTbinFJrD3wM7MAJeiOAtcDH1tpf+U9F9vUHsktwwlMf/+uPAfKttd8zxvwvzinAZ08p5ai/xg44PWj7/dt1OfAXa+2Txpi/A/9nrT0EfITTIzYdmIITsO601l5pjGkJvGmtvfyU7WyKEwiP+/dLMInAHUBb4Bp/LQA3+/fxAKArTk+X7v8mEgMUvESkri4B/hfn1Npxa+2HAMaYNcDT1n8DWGOMFydYDMcJWa2B/wLv4oSmbxhjPrbW7vcv3wCnV+qiU07BHQdOGGPuwAlgd+AEoonW2j8bY17C6QkqxjmNlwncYoy5Amjib38UOAf4BCd4bcEZl1Zd71NHYDdO8HvC/xo3AUk4pyjB6fH6ovIJ/k9LtsEJXo8CXfw9YwmnfG+AP1trpxtjegPHTtlfQ4ArrbU/8c8b/zYcxTm1mwX8yRjTARiNE7puxhkD17uabRGRKKLgJSJ1VdmzsgN40hhTCPwN2FwZIvyaAoeBR3BOL+7EGQOVAmzD6Z0qAfB/EnAWTg/a4Sqvdz9OSBpord1RWYP/k4hbcAbHd8UJZauADOAHOOGkwFq7Dljnf50fAVhrXzHGPG2M+T+cQGj9n9JcYa29G2gFPAA8jhOcXscJOJcD8/y9WE1POe16Oc6g9mScAfCLgSXW2sGVY7v833/1O9d/yrCVMeZl4DuV7f51/x4Y5+9NA2cQ/Roni3ElMAl4BSfIZfm3V0RigC4nISJ1lYjzx/4WYAkwCmec0SfGmP9njBnlX+48nLFHL+P0ZFWOd+qM01PzCM4geHBO5T0FPA1O75cxZiTO6cNFwDWnhC5wxmX9GKcH60nAY62931r7Z6ARzlis8VQz7sla+7C1thfwJ+AP1tre/tBVOebsSuAAzqc0n8EZk3bUvz2vAu+fsro1OOHnA5wgeD6wxRjzJvAW0MsYMw/Isdae+gnKh3FOeR47pa59OL1cfw5S90vW2sf9++Y8oAAn8HmCbauIRA/1eIlIXd2LM1i8KoPT23Q9Tu/PJTg9Tb/DCWZFOKFkNs4g9t04AaMybLxpjOnsX9dDQCrOqcmF1toKAGPMOdbandbabUB7Y0xjnE9PXgQs8I/rKsLpreoD7DqL7TwPyMf5xOYlwGP+9pdxAuOTlQv6e772+nukwDkNWADcZa1N/2oHOacbK7+/HGc8W6DThA8AHxpjxllrJwWpbzQw2Vpr/R902FznLRSRiFPwEpFaMcZ0BObinAosP/UhnCA2w//YJmNMQ5yxULdaa184ZR0/rBwT5p9/zRjzW2vtZ6esy1hrn/A//hTOuKnKMWBvGmN+hvO7awawHacn7T1jTHfgOZyw1xEn/PQ2xnxprZ3qX38i4DHGJAG2Su9T5WuU+YPeBuDvOGO8HgF6+Af4P4ozxi3HGDPCWvvxKatIwBmPdZ1/ueouF/EsMN7/oQVwPkkJTmEnjDF3AhONMX+rDJ5Aon/s109wTk/287dvxbkuWlu+fk0wEYkiCl4iUivW2s3AwGCPG2OexDntOBLntNsK/6m5U601xnyO09NV+ftn2SmPN/R/VfoIeNw/uN7ijOH6GCeg9bPWbvW/9o04g+t/4h+/lYbTU3Uep/coJeEEnHuAe4wxp16cdYj/te81xqwG3gSWArdYa7f7L92QjXMh1XXGmALgMWPMTdbaI8aYbJzTfUec3WXXGWMqLylx6vZV+kblKUZjzCRgCDCu8kFrbRHOBVlP1RSnJ/D7wHdOCW0JOL2DH1aOOxOR6GROHwsrInJmjDHn4nxKb59/voO1dksEX7+B/8r3X80Dzay1uyNVQ4CaGgUIn4GWawUcrc2yIhLbFLxEREREIkSfahQRERGJEAUvERERkQiJicH1bdq0sWlpaW6XISIiIlKjgoKC3dbatoEei4nglZaWRn5+fs0LioiIiLjMGFMU7DGdahQRERGJkLAEL2NMO2PMkiCPJRpjio0x8/xf3cNRg4iIiEi0Cdepxt/j3C8tkB7AbGvtz8P02iIiIiJRKeTByxhzNc6Vm7cHWWQAcIMx5pvACmBc1dt2iIiI1CcnTpxg8+bNlJaWul2K1EFycjIdO3YkKSmp1s8JafDyXyn6l8DNwKtBFlsEXGOt3WaMeRHnhrr/CrCuLCALICUlJZRlioiIRJXNmzfTtGlT0tLSOOVm6xLFrLXs2bOHzZs306lTp1o/L9RjvB4CJlpr91ezzHJr7Tb/9/lAl0ALWWtzrLXp1tr0tm0DfiJTREQkLpSWltK6dWuFrhhijKF169Z17qUMdfC6BrjPGDMP6GWMmRxgmZnGmJ7GGA9wE6ffIFdERKReUuiKPWdyzEJ6qtFaO+iUYuYBfzDGPGWtffSUxZ4AZgEG+Je19t1Q1iAiIiISrcJ2HS9r7WBr7aoqoQtr7UprbQ9rbXdr7SPhen0RERGpm86dO9dquZ/85CdhrWPp0qUsXbo0rK/hFl1AVUREJMb4fJCWBgkJztTni+zrP/vss2FdfzwHr5i4ZZCIiIg4fD7IyoKSEme+qMiZB8jMDO1rDR48mH79+rF8+XL+85//nNY+b948AI4ePcrQoUM5ePAgrVu3Zu7cuSQmfj1eBFru+PHj3H333ezcuZPu3bvz17/+lYcffphXXnkFgJkzZ/Lee+9x7NgxRo4cydatW+nYsSPTpk2jvLz8a+srLS1lyJAhHDlyhM6dOzNt2rTQ7pAQUI+XiIhIDHnkkZOhq1JJidMeagsWLGDgwIGnha6qVq1aRUJCAh999BGjRo3i8OHDjBs3jsGDB3/19cQTTwRcLicnh27duvHRRx+xbds2li9fztNPP81DDz3EQw89xHvvvQfA3/72N7p168aHH35Ily5dmDp1asD1bdu2jR/+8Ie8++67FBYWsmPHjtDvlLOkHi8REZEYUlxct/az0a1bN2655ZZql+nTpw/dunXj29/+Nl26dOG6665j0qRJX1vOWvu15dauXctnn33GvHnz2L9/P1u2bKFHjx5fe+6qVau+qmPAgAG89dZbjBs37mvrS0pKYvLkyUybNo29e/dy9OjR0OyIEFKPl4iISAwJdk3xcFxrvEmTJjUus2zZMq644greeecd9u3bx8cff1zr5S6++GJ+8pOfMG/ePJ566qmvLpjeqFEjSvzdetZaunbtyoIFCwCnF65r164B1zdlyhSGDBnC7Nmzady4cYj2QmgpeImIiMSQCRPA6z29zet12t2QlpbGc889x+WXX8727dtJT0+v9XJjx47lrbfeYtCgQbzwwgucf/75AFx77bW8/PLLXHHFFXz88ceMGTOGzz//nEGDBrF+/XpGjhwZcH3XXnstTz/9NFdffTUAW7Zsidh+qC1jrXW7hhqlp6fb/Px8t8sQEREJi9WrV3PppZfWenmfzxnTVVzs9HRNmBD6gfVSO4GOnTGmwFobMIFqjJeIiEiMycxU0IpVOtUoIiIiEiEKXiIiIiIRouAlIiIiEiEKXiIiIiIRouAlIuICt++1JxJItNwkuy7OppbBgweHrpBaUvASEYmwynvtFRWBtSfvtafwJbW2yQevpsGsBGe6KbJvnnDfJLsuoqmW2lDwEhGJsEjea0/i0CYf5GVBSRFgnWleVljC1+DBg/nZz37Gd77zna+1Vzp69Cg33HADgwYN4uabb6asrCzguiZMmMCrr74KwNNPP83cuXMpKSlhyJAhDBo0iPvuuy/o61b3GqfWUlpayh133MGVV17JDTfcQElJCceOHePOO+/kqquuIjMzk+PHjwesL9hywfbBmVLwEhGJsEjea0/i0LJHoLxKci8vcdpDLJQ3yR46dChvvfUWAB999BHXX399wJtkB3rdQK8RSE5ODj179uSTTz7h1ltvZeXKlQFvsB1IsOVqsw/qQhdQFRGJsJQU5/RioHaRGpUESejB2s9CKG+SDbB582YOHjxIixYtaNy4cdCbZFd93UCvEciaNWu49dZbARg5ciQA06dP/9oNtgMJdCPu2u6DulCPl4hIhEXbvfYkxniDJPRg7WchlDfJBujfvz/PPvssP/jBDwCC3iS76uvW9jUuueQSFi1aBMCvf/1rJk+eHPAG24EEW642+6AuFLxERCIsMxNyciA1FYxxpjk5ugWM1FLPCeCpktw9XqfdBbW9STbA0KFDefbZZ7nhhhsAgt4k+0xfY+zYsSxevJjBgwezePFihg8fHvAG24HUdrmzpZtki4iIuKyuN8lmk88Z01VS7PR09ZwAnZTc3aCbZIuIiMS7TpkKWjFKpxpFRESiQCycgZLTnckxU/ASERFxWXJyMnv27FH4iiHWWvbs2UNycnKdnqdTjSIiIi7r2LEjmzdvZteuXW6XInWQnJxMx44d6/QcBS8RERGXJSUl0alTJ7fLkAjQqUYRERGRCFHwEhEREYkQBS8RERGRCFHwEhEREYkQBS8RERGRCFHwEhEREYkQBS8RERGRCFHwEhEREYkQBS8RERGJe7sOHXO7BEDBS0REROLcnEVf8o1n3qegaK/bpSh4iYiISPya/PFGHvzncvqlteLS9s3cLkf3ahQREZH4Y63lj++u57n31vPdbufy7B29aJjocbssBS8RERGJLxUVlifeWMX0zwoZ2rcjT9/SnURPdJzkU/ASERGRuFFWXsGD/1zOy4u3cM8VnXj0e5eSkGDcLusrCl4iIiISF0pPlPOj2Ut4Z9UOHrjmIn70rc4YEz2hCxS8REREJA4cOVZG1sx8Pt2wh199/zJGXdHJ7ZICUvASERGRmLa/5Dijpi9i2Zf7+f3Qngzp29HtkoJS8BIREZGYtfNgKcOn5LFp9xEmZvblum7nul1StRS8REREJCZ9ubeEYVMWsuvQMaaO7MeVXdq4XVKNFLxEREQk5qzfcYhhUxZy9Hg5uWMy6JPS0u2SakXBS0RERGLK8s37GTE1D09CAi+NGxgVV6SvLQUvERERiRkLNu5hzIx8mjdKwjcmg7Q2jd0uqU4UvERERCQmvL9mB+NzF9OxZSNyx2TQvnkjt0uqMwUvERERiXqvLd3C/8xZxqXtmzF9VD9aN2nodklnRMFLREREolrugiJ++dpK+qW1YsqIdJomJ7ld0hlT8BIREZGoNXHeBp55ey1XX3IOEzP7kJzkcbuksxIdt+oWEYkgnw/S0iAhwZn6fG5XJCJVWWv5zVtreObttXy/53lMGt435kMXqMdLROoZnw+ysqCkxJkvKnLmATIz3atLRE4qr7A89tpKfAuLuSsjhSdv7IYnIbpudn2m1OMlIvXKI4+cDF2VSkqcdhFx34nyCh54aSm+hcXce9WFTLgpfkIXqMdLROqZ4uK6tYtI5JSeKCfbt5j31+zkwesuJntwZ7dLCjn1eIlIvZKSUrd2EYmMQ6UnuHtqHh+s3clTN3WLy9AFCl4iUs9MmABe7+ltXq/TLiLu2HvkOHf9bSGLi/bx7O29GDYg1e2SwkbBS0TqlcxMyMmB1FQwxpnm5GhgvYhbth04ym2T5rNuxyFy7u7Ljb06uF1SWGmMl4jUO5mZCloi0aBw9xEyJy/kwNETzLinPwMuaO12SWGn4CUiIiIRt3rbQYZPyaO8ooJZYzPo0bGF2yVFhIKXiIiIRNTi4n2MnJpHowYeZo8dSJd2Td0uKWIUvERERCRiPlm/m6yZ+bRt2pDc0Rmc38pb85PiiIKXiIiIRMTbK7fzo9lL6NSmMTNH9+ecZslulxRxCl4iIiISdv8o2MyD/1hGj44tmD6qHy28DdwuyRUKXiIiIhJW0z7dxOOvr+KKzq3JGZ5O44b1N37U3y0XERGRsLLW8tx7G/jju+v49mXteO7O3iQnedwuy1UKXiIiIhJy1lqeenM1Uz7ZxC19OvDMrT1I9Oi67QpeIiIiElJl5RU8/PIK5hZsZuTlaTx2w2UkJBi3y4oKCl4iIiISMsfKyvnx7KW8/fl2fvStLjxwTReMUeiqpOAlIiIiIVFyvIxxMwv4eP1uHv3epYz5xgVulxR1FLxERETkrB0oOcE9MxaxpHgfz9zag9v6ne92SVFJo9xEpFZ8PkhLg4QEZ+rzuV2RiESLXYeOcXvOfJZv3s9f7+qj0FUN9XiJSI18PsjKgpISZ76oyJkHyMx0ry4Rcd/mfSUMn5LH9gOlTB7Rj6suaut2SVFNPV4iUqNHHjkZuiqVlDjtIlJ/bdh5mKEvzGf34WPkjumv0FUL6vESkRoVF9etXUTi38otB7h7ah4JBv6eNYCu5zV3u6SYoB4vEalRSkrd2kUkvuVt2sudOQtolORhzriBCl11oOAlIjWaMAG83tPbvF6nXUTqlw/W7uTuqQtp26whc+8dyAVtm7hdUkxR8BKRGmVmQk4OpKaCMc40J0cD60XqmzeWb2XsjHwubNuEOeMGcl6LRm6XFHM0xktEaiUzU0FLpD6bnVfML15ZQXpqS6aM7Eez5CS3S4pJCl4iIiJSrZyPvuDX/17DVRe15YVhfWnUwON2STFLwUtEREQCstby+3fW8tcPvuB7Pdrzx9t60SBRo5TOhoKXiIiIfE1FheVX//qcmQuKuKPf+Uy4uTueBN3s+mwpeImIiMhpTpRX8LO5y3h16VayBl3Aw9+9BGMUukJBwUtERES+UnqinPtnLebd1Tv52XcuJnvwhQpdIaTgJSIiIgAcPlbG2Bn5zN+4hydv7MrwgWlulxR3FLxERESEfUeOM3JaHiu3HuSPt/fk5t4d3S4pLil4iYiI1HM7DpYybPJCivaW8MKwvlx7WTu3S4pbCl4iIiL1WPGeEjKnLGDv4eNMH9WPyy9s43ZJcU3BS0REpJ5au/0Qw6cs5Hh5Bb6xA+h1fgu3S4p7Cl4iIiL10NIv9zNyWh4NPAnMGTeQi9o1dbukekGXnxURiVE+H6SlQUKCM/X53K5IYsVnX+wm828LaJqcyD/uvVyhK4LCEryMMe2MMUuqeXyKMWa+MebRcLy+iEi88/kgKwuKisBaZ5qVpfAlNfvvqh2MnLaIDi0b8Y97LyeltdftkuqVcPV4/R5oFOgBY8wtgMdaOxC4wBjTJUw1iIjErUcegZKS09tKSpx2kWBeWbKZe3MLuPTcpryUNZB2zZLdLqneCXnwMsZcDRwBtgdZZDAwx//9O8CVQdaTZYzJN8bk79q1K9RliojEtOLiurWLvDi/kAdeWkb/tFb4xg6gZeMGbpdUL4U0eBljGgC/BB6qZrHGwBb/93uBgBcLsdbmWGvTrbXpbdu2DWWZIiIxLyWlbu1Sf1lr+esHG3jstc+55tJ2TBvVjyYN9dk6t4S6x+shYKK1dn81yxzm5GnIJmGoQUQk7k2YAN4qQ3O8XqddpJK1lqffWsPv/rOWm3t34PlhfUhO8rhdVr0W6tBzDXCfMWYe0MsYMznAMgWcPL3YEygMcQ0iInEvMxNyciA1FYxxpjk5TrsIQHmF5eGXV5Dz0UbuHpjK/w3tSZJHfR1uM9ba8KzYCV/ZwF3W2kdPaW8GfAy8B3wXGGCtPVDdutLT021+fn5Y6hQREYk3x8sqeOClpby5Yhv3f7Mz//PtizDGuF1WvWGMKbDWpgd6LGwnea21g/3fPlql/aAxZjBwLfBMTaFLREREau/o8XLuzS3gw3W7+MX1l5A16EK3S5JTuDK6zlq7j5OfbBQREZEQOFh6gtHTF5FftI/f3NKdO/rr0xbRRh9rEBERiQO7Dx9jxNQ81u04xJ/v7M0NPc5zuyQJQMFLREQkxm3df5Rhkxey9cBR/nZ3OoMvPsftkiQIBS8REZEYtnHXYYZPyePg0RPMHJ1Bv7RWbpck1VDwEhERiVGfbz3AiKl5WAuzswbQrUNzt0uSGih4iYiIxKD8wr2Mmr6Ipg0TmTkmgwvbNnG7JKkFBS8REZEY89G6XYybWcC5zZPJHZNBhxaNan6SRAUFLxERkRjy7xXb+PHfl9D5nKa8eE9/2jZt6HZJUgcKXiIiIjFizqIveejl5fROacnUkf1o3ijJ7ZKkjhS8REREYsDkjzfy1Jur+UaXNkwa3hdvA/0Jj0U6aiIiIlHMWssf/7uO597fwHe7ncuzd/SiYaLH7bLkDOk25SISU7KzITERjHGm2dluVyQSPhUVlsdfX8Vz72/gtvSO/PnO3gpdMU49XiISM7Kz4fnnT86Xl5+cnzjRnZpEwqWsvIIH/7mclxdvYfSVnXj0e5dijHG7LDlLxlrrdg01Sk9Pt/n5+W6XISIuS0x0wlZVHg+UlUW+HpFwKT1Rzo9mL+GdVTv46bUX8cOrOyt0xRBjTIG1Nj3QY+rxEpGYESh0VdcuEouOHCsja2Y+n27Yw//7/mWMvKKT2yVJCCl4iUjM8HiC93iJxIP9JccZOW0RK7Yc4P+G9uTWvh3dLklCTIPrRSRmZGXVrV0kluw8WMrtkxawautBJmb2UeiKU+rxEpGYUTmAPifH6fnyeJzQpYH1Euu+3FvCsCkL2XXoGNNG9eOKzm3cLknCRMFLRGLKxIkKWhJf1u84xLApCyk9UUHumAz6pLR0uyQJIwUvERERlyzfvJ8RU/NI9CTw0rgBXHJuM7dLkjBT8BIREXHBgo17GDMjnxbeJHJHZ5DWprHbJUkEKHiJiIhE2PtrdjA+dzHnt/KSOzqDc5snu12SRIiCl4iISAS9tnQL/zNnGZe2b8aMe/rTqnEDt0uSCFLwEhERiZDcBUX88rWV9E9rxeQR6TRNTnK7JIkwBS8REZEImDhvA8+8vZarLzmHiZl9SE7SlX/rIwUvERGRMLLW8tu31/LCh19wY6/z+P3QniR5dP3y+krBS0REJEzKKyy/fG0lsxYWk5mRwpM3diMhQTe7rs8UvERERMLgRHkFP52zjNeXbSV78IX87DsXY4xCV32n4CUiIhJipSfKyfYt5v01O/n5dZcwfvCFbpckUULBS0REJIQOlZ5g9Ix8FhXuZcLN3cjMSHW7JIkiCl4iIiIhsvfIcUZMzWP1toP86Y7e/KDneW6XJFFGH6sQqQeuuQaMOfl1zTVuVyQ+H6SlQUKCM/X53K5Izta2A0e5bdJ81u04RM7dfRW6JCAFL5E4d8018N57p7e9957Cl5t8PsjKgqIisNaZZmUpfMWywt1HGPL8fLYfKOXFe/pz9SXt3C5JopSx1rpdQ43S09Ntfn6+22WIxKTqPkQVAz/+cSktzQlbVaWmQmFhpKuRs7V620GGT8mjvKKCF+/JoHvH5m6XJC4zxhRYa9MDPaYxXiIiEVZcXLd2iV6Li/cxcmoe3gaJ/D1rIJ3Paep2SRLldKpRRCTCUlLq1i7R6ZP1uxk2eSGtGjdg7r0KXVI7Cl4ice5b36pbu4TfhAng9Z7e5vU67RIb3l65nXumLyKllZc59w7k/Fbemp8kgoKXSNx7992vh6xvfctpF3dkZkJOjjOmyxhnmpPjtEv0+0fBZrJ9BXTt0IyXsgZyTtNkt0uSGKLB9SIiIrU07dNNPP76Kq7s3IZJw/vSuKGGSsvXaXC9iIjIWbDW8tx7G/jju+v4Ttd2PHdnbxometwuS2KQgpeIiEg1KiosT725mqmfbuLWPh357a3dSfRopI6cGQUvERGRIMrKK3j45RXMLdjMyMvTeOyGy0hIqObieCI1UPASEREJ4FhZOT+evZS3P9/Oj7/VhZ9c0wVT3RWJRWpBwUtERKSKkuNljJtZwMfrd/PLGy5j9JWd3C5J4oSCl4iIyCkOlJzgnhmLWFK8j2eG9OC29PPdLl0mY9MAACAASURBVEniiIKXiIiI365Dxxg+ZSEbdx1hYmYfruvW3u2SJM4oeImIiACb95UwfEoe2w+UMmVkOt/o0tbtkiQOKXiJiEi9t2HnYYZPWciRY2XkjulP39RWbpckcUrBS0RE6rWVWw5w99Q8Egz8PWsgl53XzO2SJI4peImISL2Vt2kvo6cvolmjJHLHZNCpTWO3S5I4p0vvitQDPh+kpUFCgjP1+WK3jmjZFol9H6zdyfApC2nbrCFz7x2o0CURoR4vkTjn80FWFpSUOPNFRc48QGZmbNURLdsise/1ZVt54KWlXNK+KTNG9ad1k4ZulyT1hLHWul1DjdLT021+fr7bZYjEpLQ0J6BUlZoKhYWxVUe0bIvEttl5xfzilRX0S23F5JHpNEtOcrskiTPGmAJrbXqgx9TjJRLniovr1h7NdUTLtkjsmvThFzz91hoGX9yW5zP70qiBx+2SpJ7RGC+ROJeSUrf2aK4jWrZFYo+1lt/9Zw1Pv7WGG3q0J2d4ukKXuELBSyTOTZgAXu/pbV6v0x5rdUTLtkhsqaiwPPba5/z1gy+4s38Kf7qjNw0S9edP3KF3nkicy8yEnBxnHJQxzjQnJ/KD0UNRR7Rsi8SOE+UV/HTOUmYuKGLcVRfw65u74Ukwbpcl9ZgG14uISFwqPVHO/bMW8+7qnTx43cVkD+7sdklST2hwvYiI1CuHj5UxdkY+Czbt4cmbujF8QKrbJYkACl4iIhJn9h05zshpeazcepA/3taLm3p3cLskka8oeImISNzYcbCUYZMXUrS3hEnD+nLNZe3cLknkNApeIiISF4r3lJA5ZQF7Dx9nxqj+DLywtdsliXyNgpeIiMS8tdsPMXzKQo6XVzBr7AB6nt/C7ZJEAlLwEhGRmLb0y/2MmJpHw8QE5owbyEXtmrpdkkhQCl4iIhKzPtuwm7Ev5tO6SUNyR2eQ0tpb85NEXKTgJSIiMemdz7dz/+wldGrdmJmj+3NOs2S3SxKpkYKXiIjEnFeWbOZ/5y6nW4fmzBjVjxbeBm6XJFIrumWQiIjElBfnF/LAS8vI6NQK35gMhS6JKerxEhGRmGCt5a8fbOD376zj2sva8ec7e5Oc5HG7LJE6UfASEZGoZ63l6bfWkPPRRm7p3YFnhvQg0aOTNhJ79K6VqOTzQVoaJCQ4U5/P7YrOTCi2I172hciZKq+wPPzyCnI+2siIgan8fmhPhS6JWerxkqjj80FWFpSUOPNFRc48QGame3XVVSi2I172hciZOl5WwQMvLeXNFdv44dWd+em1F2GMcbsskTNmrLVu11Cj9PR0m5+f73YZEiFpaU7AqCo1FQoLI13NmQvFdsTLvhA5E0ePl3NvbgEfrtvFI9dfythBF7hdkkitGGMKrLXpgR5Tj5dEneLiurVHq1BsR7zsC5G6Olh6gtHTF1FQtI/f3tqd2/uluF2SSEjoJLlEnZQgv1+DtUerUGxHvOwLkbrYffgYd+YsYOmX+/nLXX0UuiSuKHhJ1JkwAbxV7vrh9TrtsSQU2xEv+0KktrbuP8ptL8zni12HmTyiH9d3b+92SSIhpeAlUSczE3JynHFMxjjTnJzYG0weiu2Il30hUhsbdx1m6Avz2XX4GDNHZ3DVRW3dLkkk5DS4XkREXPf51gOMmJqHtfDi6P50Pa+52yWJnDENrhcRkaiVX7iXUdMX0bRhIjPHZHBh2yZulyQSNgpeIiLimg/X7WLczHzOa96ImWMy6NCikdsliYSVgpeIiLji3yu28eO/L6HLOU15cXR/2jRp6HZJImGn4CUiIhE3Z9GXPPTycvqktGTKyH40b5TkdkkiEaHgJSIiETX544089eZqBl3UlheG9cHbQH+KpP7Qu11ERCLCWssf/7uO597fwPXdz+XZ23vTIFFXNZL6RcFLRETCrqLC8sQbq5j+WSG3p5/Pr2/pjidBN7uW+kfBS0REwqqsvIIH/7mclxdvYew3OvGL6y/FGIUuqZ8UvEREJGxKT5Tzo9lLeGfVDv732xdx3zc7K3RJvabgJSIiYXHkWBlZM/P5dMMeHv9BV0ZcnuZ2SSKuU/ASEZGQ219ynJHTFrFiywH+cFtPbunT0e2SRKJCWIKXMaYV0BdYYq3dHY7XEBGR6LTzYCnDp+SxafcRns/sw7e7nut2SSJRI+Sf4zXGtATeAPoDHxhjvnZ7eWNMojGm2Bgzz//VPdR1iMSL7GxITARjnGl2tjvr8PkgLQ0SEpypz1f3dUSLeNqWaPPl3hKGTprPl/tKmDaqn0KXSBXh6PHqAfzUWrvAH8L6AP8JsMxsa+3Pw/D6InEjOxuef/7kfHn5yfmJEyO3Dp8PsrKgpMSZLypy5gEyM2u3jmgRT9sSbdbvOMSwKQspPVGBb0wGvVNaul2SSNQx1trwrNiYQcBTwA3W2oNVHssG7gOOACuAcdbasmDrSk9Pt/n5+WGpUySaJSY6QakqjwfKgv7EhH4daWlOQKkqNRUKC2u3jmgRT9sSTZZv3s+IqXkkehLIHZ3Bxec2dbskEdcYYwqstemBHgvLJYON81nh24F9wIkAiywCrrHW9geSgOsDrCPLGJNvjMnftWtXOMoUiXqBAlN17eFaR3Fx3dqjWTxtS7RYsHEPd/1tIU2SE/nHvQMVukSqEZbgZR33AcuBHwRYZLm1dpv/+3ygS4B15Fhr06216W3bfm2YmEi94PHUrT1c60hJqVt7NIunbYkG763ewYipebRvnszccZeT2rqx2yWJRLVwDK7/uTHmbv9sC2B/gMVmGmN6GmM8wE3AslDXIRIPKsce1bY9XOuYMAG83tPbvF6nPdbE07a47bWlWxg3s4CLz23KnHEDObd5stsliUS9cPR45QDDjTEfAR5gszHmqSrLPAHMBJYC862174ahDpGYN3EijB9/snfK43HmazsoPlTryMyEnBxnHJQxzjQnJzYHo8fTtrgpd0ERP3lpKelpLfGNyaBl4wZulyQSE8I2uD6UNLheRCR6TJy3gWfeXss1l57DX+7qQ3JSHc5bi9QD1Q2u15XrRUSkVqy1/Pbttbzw4Rfc2Os8fj+0J0mesAwVFolbCl4iIlKj8grLL19byayFxQwbkMITP+hGQoJudi1SVwpeIiJSrRPlFfx0zjJeX7aV7MEX8rPvXIxz1SARqSsFLxERCero8XKyfQV8sHYXD333Eu696kK3SxKJaQpeIiIS0KHSE4yekc+iwr38+ubu3JWhi52JnC0FLxER+Zo9h48xYloea7Yd4rk7evP9nue5XZJIXFDwEhGR02w7cJRhkxeyed9R/nZ3Ot+85By3SxKJGwpeIiLylcLdR8icvJCDR08wc3QG/Tu1crskkbii4CUiIgCs3naQ4VPyqLCW2VkD6NahudslicQdXflOREQoKNrH7ZPmk+QxzBk3UKFLJEzU4yUiUs99sn43Y1/Mp12zhuSOyaBjS2/NTxKRM6IeL4lKPh+kpUFCgjP1+dypIzsbEhOdmyknJjrzsVhDKPanjkl8envldu6ZvojU1l7m3DvwzELXJh+8mgazEpzpJpfeHCKxwFob9V99+/a1Un/k5lrr9VoLJ7+8Xqc9ksaPP72Gyq/x42OrhlDsTx2T+DQ3/0vb6aE37M1//cTuP3L8zFayMdfav3ut9XHy6+9ep12kngLybZBMY5zHo1t6errNz893uwyJkLQ0KCr6entqKhQWRq6OxEQoL/96u8cDZWWxU0Mo9qeOSfyZ9ukmHn99Fd/o0oZJw/vibXCGI09eTYOSAG8ObyrcVHg2JYrELGNMgbU2PdBjGuMlUae4uG7t4RLoD3x17dFaQyj2p45J/LDW8qf31vPsu+u5ruu5/OnOXjRM9Jz5CkuCvAmCtYvUcxrjJVEnJchdSYK1h4snyN+iYO3RWkMo9qeOSXyoqLA8+cZqnn13PUP6duQvd/U+u9AF4A3yJgjWLlLPKXhJ1JkwAbxVxvd6vU57JGVl1a09WmsIxf7UMYl9ZeUVPPjP5Uz9dBOjrkjjmVt7kOgJwZ+AnhPAU+XN4fE67SLydcEGf0XTlwbX1z+5udamplprjDON9CDuSuPHW+vxOAO4PR53BnGHooZQ7E8dk9hVeqLMjnsx36b+/A37x/+utRUVFaF9gY251r6Saq3POFMNrJd6Dg2uFxGpn0qOlzFuZgEfr9/NYzdcxj1XdnK7JJG4p8H1IiL10IGSE4yansfSL/fzuyE9GJp+vtslidR7Cl4iInFo16FjDJ+ykI27jjAxsw/XdWvvdkkigoKXiEjc2byvhGGTF7Lj4DGmjuzHlV3auF2SiPgpeImIxJENOw8zfMpCjhwrI3dMBn1TW7pdkoicQsFLRCROrNxygLun5pFgDC+NG8il7Zu5XZKIVKHgJSISB/I27WX09EU0a5RE7pgMOrVp7HZJIhKAgpeISIz7YO1O7p1ZQMeWjcgdk0H75o3cLklEglDwEhGJYa8v28oDLy3lkvZNmTGqP62bNHS7JBGphoKXiEiMmp1XzC9eWUG/1FZMHplOs+Qkt0sSkRooeImIxKBJH37B02+t4ZsXt2ViZl8aNdCdwkVigYKXiEgMsdbyu/+sZeK8L7ihR3v+cFsvGiSG4GbXIhIRCl4iIjGiosLy2L9WkrugmDv7p/DUTd3wJBi3yxKROtC/SSJRzueDtDRISHCmPp876xB3nSiv4KdzlpK7oJh7r7qQX9+s0BVym3zwahrMSnCmm/SDIqGnHi+RKObzQVYWlJQ480VFzjxAZmbk1iHuKj1Rzv2zFvPu6p08eN3FZA/u7HZJ8WeTD/KyoNz/g1JS5MwDdNIPioSOsda6XUON0tPTbX5+vttliERcWpoTlKpKTYXCwsitQ9xz+FgZY2YsYuGmvTxxYzeGD0h1u6T49GqaE7aq8qbCTYWRrkZinDGmwFqbHugx9XiJRLHi4rq1h2sd4o59R44zcloeK7ce5Nnbe3Fjrw5ulxS/SoL8QARrFzlDGuMlEsVSUurWHq51SORtP1DKbZPms2b7IXKG91XoCjdvkB+IYO0iZ0jBSySKTZgAXu/pbV6v0x7JdUhkFe8pYeikz9i6/yjTR/XnW5e2c7uk+NdzAniq/KB4vE67SAgpeIlEscxMyMlxxmMZ40xzcuo2KD4U65DIWbv9EENe+IzDpWXMzhrAwAtbu11S/dApE/rnOGO6MM60f44G1kvI1Ti43hhzGfADoEFlm7X2iTDXdRoNrheR+mBJ8T5GTltEclICuaMz6NKuqdslicgZONvB9XOA3wBfhrQqERH5ymcbdjPmxXzaNGmIb0wG57fy1vwkEYk5tQleO4DZ1trycBcjIlIfvfP5du6fvYROrRszc3R/zmmW7HZJIhImQYOXMeZu/7fLgA+MMbOBIwDW2hcjUJuISNx7efFmfvaP5XTv0Jzpo/rRwtug5ieJSMyqrser8l4Uy/xflW3Rf8VVEZEYMOOzQn71r8+5onNrcoan07ihLq0oEu+C/pRba2cAGGNaW2v3VLYbY26LRGEiIvHKWstfP9jA799Zx7WXtePPd/YmOcnjdlkiEgG1uZzE3Crz94WjEBGR+sBay6//vZrfv7OOW/p04PnMPgpdIvVIdWO8rgIGA2nGmMf8zY2BfRGoS0Qk7pRXWH7x8gpeyv+SkZen8dgNl5GQYGp+oojEjeoGFBQC84CbgA/9bUeBJeEtSUQk/hwvq+CBl5by5opt/Ojqzjxw7UUYo9AlUt9UN8arCCgyxkyz1n4YbDkREane0ePl3JtbwIfrdvHo9y5lzDcucLskEXFJbS4nsf+U7wFdTkJEpLYOHD3B6OmLWFy8j2du7cFt/c53uyQRcVFtLidxJ1AM5AO9gS6AgpeISA12Hz7G3VPyWL/zEH+5qw/Xd2/vdkki4rKgn2q01s7wX1IiyVqbZa3NsdaOr+45Evt8PkhLg4QEZ+rzxW4d2dmQmOjcGDox0ZmPRdFyTKRutuw/ym0vzGfj7sNMHtEvekPXJh+8mgazEpzpJr3Bzor2Z/SJsmNSm6v1HTDG/AHnIqpdgUPhLUnc4vNBVhaUlDjzRUXOPEBmZmzVkZ0Nzz9/cr68/OT8xImhqzXcouWYSN1s3HWYYZMXcuhYGbmjM0hPa+V2SYFt8kFeFpT732AlRc48QCe9wepM+zP6ROExMdZWfyF6Y0wicAvQCeeU4z+ttccjUNtX0tPTbX5+fiRfsl5KS3P+sFeVmgqFhbFVR2KiE7aq8nigrOxsqousaDkmUnufbz3A3VPyAHhxdH+6ntfc5Yqq8Wqa84eoKm8q3FQY6Wpin/Zn9HHpmBhjCqy16YEeq7HHy1pbBswJeVUSdYqL69YezXUECl3VtUeraDkmUjv5hXsZNX0RTRsmkjsmgwvaNnG7pOqVBHkjBWuX6ml/Rp8oPCYaryVfSUmpW3s01+EJciHwYO3RKlqOidTsw3W7GDZlIW2bNGTu+MujP3QBeIO8kYK1S/W0P6NPFB6ToMHLP64LY8wHxpj3/V8fGGPej1x5EkkTJoDXe3qb1+u0x1odleOgatseraLlmEj1/r1iG2NmLOKCNk2Yc+9AOrRo5HZJtdNzAniqvME8Xqdd6k77M/pE4TGp7lONP/VPv2mtvdr/9U1r7dWRK08iKTMTcnKc8UPGONOcnMgP4g5FHRMnwvjxJ3u4PB5nPpYG1kP0HBMJbs6iL7l/1mJ6dmzB7KwBtGnS0O2Saq9TJvTPcca7YJxp/xwNBD9T2p/RJwqPSY2D66OBBteLSDSa/PFGnnpzNVdd1JYXhvWlUYMYO5ctImFxVoPrjTGLgNVAAbAYWGKtPRzaEkVEYoe1lj/+dx3Pvb+B73Vvzx9v70WDRA2ZFZGa1eY3xXXAu8A1wH+ARWGtSEQkilVUWB5/fRXPvb+B29PP57k7eyt0iUit1eYCqmuBT4HXgfv9N88WEal3ysorePAfy3l5yRbGfqMTv7j+UowxNT9RRMSvNv+mdQHmAjcA640xW8NbkohI9Ck9Uc5432JeXrKF//32RQpdInJGahO8XgK6AT6gm7X2vPCWJCISXQ4fK+Oe6Yv476odPHFjV+6/uotCl4ickdpcuf7bkShERCQa7S85zshpi1ix5QB/uK0nt/Tp6HZJIhLDajPGS0SkXtp5sJThU/LYtOcIz2f24dtdz3W7JBGJcQpeIiIBfLm3hMzJC9l9+BjTR/bj8s5t3C5JROKAgpeISBXrdxxi2JSFHCurYNbYAfQ6v4XbJYlInFDwEhE5xfLN+xkxNY8kTwIvZQ3k4nObul2SiMQRBS8REb/5X+xh7Iv5tGycRO7oDFJbN3a7JBGJMwpeIiLAe6t3MN63mNRWXnLHZNCuWbLbJYlIHAp6HS9jzAfGmPerfH1gjHk/kgVK/eTzQVoaJCQ4U58vNuvIzobERDDGmWZnh6NKOVuvLd3CuJkFXHpuU+aMG6jQJQKQlw2zE2GWcaZ5Lv0C2+SDV9NgVoIz3eTSH4QQCdrjZa39ZiQLEank80FWFpSUOPNFRc48QGZm7NSRnQ3PP39yvrz85PzEiaGtVc7czAVFPPbaSjI6tWLyiH40aagTASLkZcOGU36B2fKT8/0j+Atskw/ysqDc/4u4pMiZB+gUwT8IIWSstW7XUKP09HSbn5/vdhkSIWlpTsipKjUVCgtjp47ERCdsVeXxQFnZ2VYnofDXDzbwu/+s5ZpLz+Evd/UhOcnjdkki0WF2ohO2qjIeuDOCv8BeTXPCVlXeVLipMHJ11JExpsBamx7osRr/tTPGeIA+QCN/Uwdr7ewQ1idymuLiurVHax2BQld17RI51lp+8/YaJn24kZt6ncfvhvYkyVObO6iJ1BOBQld17eFSEuQXbrD2GFCbPvV/AIeATsBWoCWg4CVhk5ISuKcpJSW26vB4gvd4iXvKKyyPvrqS2XnFDB+QyuM/6EpCgu67KHIa4wne4xVJ3pQgPV4R/oMQQrX5F68NMArYaa29nZM9XyJhMWECeL2nt3m9Tnss1VE5Hqy27RJ+x8sq+PHflzA7r5j7vnkhT9yo0CUS0IVBflEFaw+XnhPAU+UXscfrtMeo2gSvYuA24Jgx5mGgWXhLkvouMxNycpyxVMY405ycyA6sD0UdEyfC+PEne7g8HmdeA+vdcfR4OeNm5vPG8m08/N1L+Nl3LsEYhS6RgPpPhM7jT/ZwGY8zH8mB9eAMoO+f44zpwjjT/jkxO7AeajG43hiTALQGTgC3AJ9aa9dGoLavaHC9iJyNg6UnGDM9n0VFe/n1zd25s3/snqYQkeh3VoPr/a4E0oANwLoQ1SUiEnZ7Dh9jxLQ81mw7xHN39Ob7Pc9zuyQRqcdqc6rx78C3gCPA9UBsX7lMROqNbQeOctuk+WzYeZi/jUhX6BIR19Wmx+sca+1tlTPGmA/CWI+ISEhs2n2EYZMXcvDoCV68J4P+nVq5XZKISK2CV4kx5iGgAOgPHDDGDLLWfhTe0kREzszqbQcZPiWPCmuZnTWAbh2au12SiAhQu1ONC4GGwOU4QW0JMDiMNYmInLGCon3cPmk+SR7DnHEDFbpEJKrU2ONlrX3cGNMN6IBzaYkvrbWHw16ZiEgdfbx+F1kvFtCuWUNyx2TQsaW35ieJiERQjT1expg/A48DTwMXALPCXZSISF29vXIbo6fnk9ray9x7L1foEpGoVJtTjd2ttbcC+621bwLqtxeRqDI3/0uyfYvp1qEZL2UNpG3Thm6XJCISUG0G1+8yxjwGtDTGjAC2h7kmEZFam/rJJp54YxXf6NKGScP74m1Q28sTiohEXm16vO4GDgDzcXq7RoW1IhGRWrDW8uy763jijVVc1/VcJo9IV+gSkahXm99Sx4BpwFGcK9hH+NbkIiKnq6iwPPnmKqZ9WsjQvh15+pbuJHpq83+kiIi7avObai4wCPgDMAZ4paYnGGNaGWOuNca0Ocv6REROU1ZewYP/XM60Twu554pO/PbWHgpdIhIzavPbqrW19g2gi7U2E2hU3cLGmJbAGzgXW/3AGNM2yHJTjDHzjTGP1rVoCczng7Q0SEhwpr4zuLlTKNYRT7KzITERjHGm2dl1e76OSWgdKyvn/smv8Y+CzTzQzscvy64nociFD1rnZcPsRJhlnGleHd8YAJt88GoazEpwpptcOrCqIz6FYn/qmIRFbU41HjLGvAoUGGOuBw7VsHwP4KfW2gX+ENYH+M+pCxhjbgE81tqBxpipxpgu1tr1Z7IB4vD5ICsLSkqc+aIiZx4gMzNy64gn2dnw/PMn58vLT85PnFjz83VMQqvkeBnjct7g480N+NV5kxjV5nVnAESef4d0itAOycuGDae8MWz5yfn+tXhjgPMHLC8Lyv0HtqQo8tuhOuJXKPanjknYGGtt9QsYkwxcZq1dbIzpCRRaaw/UuGJjBgFPATdYaw9Weew54G1r7b+NMXcAjay104KtKz093ebn59dic+qvtDTnj3JVqalQWBi5dcSTxEQnbFXl8UBZWc3P1zEJnQMlJxg1PY+lxXt5puOzDGn1/ukLeFPhpsLIFDM70QlbVRkP3FmLNwY4vQclAQ5sJLdDdcSvUOxPHZOzYowpsNamB3qsNleuLwUW+79fVssXNMDtwD7gRIBFGgNb/N/vxekVq7qOLCALICUlpTYvW68VF9etPVzriCeBQld17VXpmITGzkOl3D0lj427jjAx9Wmuaz7/6wuVRHCHBApd1bUHEqzeSG6H6ohfodifOiZhE5YRqdZxH7Ac+EGARQ5zcqxYk0B1WGtzrLXp1tr0tm0DDhOTUwTLpnXJrKFYRzzxBPn8brD2qnRMzt7mfSXc9sJ8iveWMHVkP65rvzXwgt4I7hAT5A0QrD2QYPVGcjtUR/wKxf7UMQmbkAcvY8zPjTF3+2dbAPsDLFaAc2kKgJ5AYajrqG8mTABvlTukeL1OeyTXEU8qx1LVtr0qHZOzs2HnIYY8P5+9R44zc3QGV3ZpAz0ngKfKDvF4nfZIuTDIGyBYeyDRsB2qI36FYn/qmISPtTakX0BL4L/AR8BEoCvwVJVlmgHLcC5RsRpoXt06+/bta6VmubnWpqZaa4wzzc11Zx3xZPx4az0ea8GZjh9ft+frmJyZFZv3295PvGP7Pvlfu2rrgdMf3Jhr7Sup1vqMM93owg5ZON7aWR5rfTjThXV8Y1gbHduhOuJXKPanjskZA/JtkExT4+D6cPF/4vFa4CNrbbW3IdLgepH6Y+HGPYyZkU+zRkn4xmSQ1qax2yWJiNTJWQ2uDxdr7T5gjluvLyLR54M1O7k3t4COLRuROyaD9s2rvWygiEjM0Y3NRCQqvL5sKw+8tJRL2zdj+qh+tG7S0O2SRERCTsFLRFw3a2Exj7y6gn5prZgyIp2myUlulyQiEhYKXiLiqhc+/ILfvLWGqy85h4mZfUhOqsNlGUREYoyCl4i4wlrL7/6zlonzvuD7Pc/jD7f1JEk3uxaROKfgJSIRV1FheexfK8ldUMxdGSk8eWM3PAnG7bJERMJOwUtEIupEeQX/O3cZry3dyr1XXcjPr7sY5y5jIiLxT8FLRCKm9EQ5989azLurd/LgdReTPbiz2yWJiESUgpeIRMSh0hOMmZFPXuFenrqpG8MGpLpdkohIxCl4iUjY7T1ynJHT8li19SDP3t6LG3t1cLskERFXKHiJSFhtP1DK8CkLKd5bwqThffnWpe3cLklExDX67LacxueDtDRISHCmPp/bFUksK9pzhCEvfMa2A6XMuKd//ISuTT54NQ1mJTjTTWfwgxKKdUSLaNkf0bBPo6GGaKpDvkY9XvIVnw+ysqCkxJkvKnLmATIz3atLYtPa7YcYNmUhZeUVzBqbQY+OLdwuKTQ2+SAvC8r9PyglRc48QKda/qCEYh3RIlr2RzTs02ioIZrqkICMtdbtGmqUnp5u8/Pz3S4j7qWlOWGrqtRUz0A+PAAAIABJREFUKCyMdDUSy5YU72PktEUkJyWQOzqDLu2aul1S6Lya5vwhq8qbCjcVRm4d0SJa9kc07NNoqCGa6qjHjDEF1tr0QI+px0u+Ulxct3aRQD7dsJuxL+bTtmlDckdncH4rr9slhVZJkB+IYO3hWke0iJb9EQ37NBpqiKY6JCCN8ZKvpKTUrV2kqnc+386oaYs4v6WXueMGxl/oAvAG+YEI1h6udUSLaNkf0bBPo6GGaKpDAlLwkq9MmADeKn8nvV6nXaQmLy/ezHjfYi47rxkvjRvAOc2S3S4pPHpOAE+VHxSP12mP5DqiRbTsj2jYp9FQQzTVIQEpeMlXMjMhJ8cZ02WMM83J0cB6qdmMzwr56ZxlDLigFb4xGbTw/v/27jw+qvJQ4/jzJoQl7PsmSwBBUQhLmIBXLXq17nXH2rAHo9LW1vbWLtjNK22vt9re1osaCbIkIOBCtXbRtqJWMcMECCAKAoEga9iXISGZvPePCTdKE5LA5Jw5M7/v58Nnct45YZ55c5g8nDlzTlO3IzWelAzJlx0+XkYmfOvLbthBy5H4O6JFtMxHNMxpNGSIphyoEQfXAzhn1lo9/Y/NevKtTfry4K763b3D1Twp0e1YAOAqDq4HEHHWWs1842PN/meR7hjRU0/cOVRNEtmJDgBnQ/EC0GChSqsfvbJOiwM7NPmyvvrJzYOVkGDcjgUAUY/iBaBByipCenjxGv1p3R499O8X6uFrLpQxlC4AqA+KF4B6C56q0AO5q/TuphI9etPFmnZFP7cjAYCnULwA1MuRk+XKnLtSq4oP6Yk7h2rcqF5uRwIAz6F4AajT/uNlmpjj16f7junpr43QjUO6ux0JADyJ4gXgrHYePqkJs/O1+0ipZk8apS8N7Ox2JADwLIoXgFptKTmuCbPzdaysQrnTfBrZp4PbkQDA0yheAGq0fucRTZrjlzHSi1mjdUmPtm5HAgDPo3gB+BeBbQc1Ze5KtW7WRLnT0tWvcyu3IwFATKB4AfiC5Rv36YHcAvVo10K5menq0a6F25EAIGZQvAD8vzfW7ta3F6/WhV1aa36mT51aNXM7EgDEFC6sBkCStHhlsb65aJWG9WqnRVmjKV2NzT9dWtREWmjCt/7pbidyF/NRrShPWtZXWpgQvi3KczvRuYul5xIh7PECoOff3aqZf/pYXxrYWc+OH6kWTRPdjhTb/NOlzc9UL9tQ9bJvljuZ3MR8VCvKk/xZUigYXg5uDy9LUkqGe7nORSw9lwhijxcQx6y1evLNjZr5p49105Duen5iGqXLCVuyGzYe65iPaoUzqovKaaFgeNxrYum5RBB7vIA4VVlp9fPXP9K8Fdv11VG9NPP2IUpM4GLXjrChho3HOuajWrC4YePRLJaeSwSxxwuIQxWhSn13aaHmrdiurCv76Zd3ULocZWrZq1jbeKxjPqol927YeDSLpecSQRQvIM6Ulof0YN4qvbp6p7533SD98IaLZAyly1H9sxo2HuuYj2qpM6XE5C+OJSaHx70mlp5LBPFWIxBHjpdVKGt+QB9sOaD/vPUSTRjT1+1I8en0AeNbssNvp5nEcMmItwPJT2M+qp0+6LxwRvgtueTe4aLixYPRY+m5RJCx1rqdoU5paWk2EAi4HQPwtMPBU5r0wkqt33lEv757qG4ffoHbkQAgJhljCqy1aTXdxx4vIA7sO1qqCTl+FR04oWfHj9S1g7u6HQkA4hLFC4hxOw4GlTE7XweOl2nulFG6rH8ntyMBQNyieAEx7NO9xzQ+J19lFZXKu2+0hvVq53YkAIhrFC8gRhXuOKzJL/iVlJigxVljNKhba7cjAUDco3gBMWjFlgOaNm+lOrRqqrzM0erdMbnubwIANDqKFxBj/rZhr6YvXKU+HZKVOy1dXds0dzsSAKAKxQuIIctW79R3lxbq0h5tNHeKT+1bNnU7EgDgcyheQIxYsGKbfvLaRxqd0lHPT0pTq2b88waAaMMrM+Bx1lrNWr5F//3Xjbrm4q56+mvD1TwpDq9xBwAeQPECPMxaq1/95RM9985W3Tash/777lQlJXIJVgCIVhQvwKNClVaPLluvRf5iTRzTRz+75RIlJHCxawCIZvzXGPCgUxWV+taLq7XIX6xvXDVAP/+KA6WrKE9a1ldamBC+Lcpr3MeLZtEyF/7p0qIm0kITvvVPdydHtMwH4AHs8QI85uSpkB7MK9DyjSX60Y0XKevK/o3/oEV5kj9LCgXDy8Ht4WVJSslo/MePJtEyF/7p0uZnqpdtqHrZN8u5HNEyH4BHsMcL8JCjpeWaNMevdzaV6Jd3DHGmdElS4YzqX6ynhYLh8XgTLXOxJbth440lWuYD8Aj2eAEeceB4mSa94NfGPcf0+3uH6+ahPZx78GBxw8ZjWbTMhQ01bLyxRMt8AB7BHi/AA3YfOalxz63Q5n3HlT0xzdnSJUnJvRs2HsuiZS5MLacMqW28sUTLfAAeQfEColzR/hO665kV2ne0TPOnpuuqQV2cD5E6U0o843qPicnh8XgTLXPRP6th440lWuYD8AiKFxDFNuw6qrufXaGT5SEtyhotX0oHd4KkZEi+bCm5jyQTvvVlx+fB09EyF75Z0oAHq/dwmcTwspMH1kvRMx+ARxhrrdsZ6pSWlmYDgYDbMQBHFWw/qCkvrFTLZk20IDNdA7q0cjsSAKAejDEF1tq0mu7j4HogCr33aYmy5heoW9vmWpDp0wXtk+v+JgBA1KN4AVHmL+t366FFa9S/SyvNn+pT59bN3I4EAIgQihcQRZYGduj7L6/V8N7tNWfyKLVtkeR2JABABFG8gCgx559FeuyPG3TFhZ303ISRSm7KP08AiDW8sgMus9bqt3/7VP/z9091w6Xd9NuvDlOzJg6fiwkA4AiKF+Ciykqrx/64QXM/2Ka7R16gX94xRE0SOcsLAMQqihfgkopQpb7/8jq9vOozZV6eohk3XqyEBON2LABAI6J4AS4oqwjpoUWr9deP9uo71w7UN68eIGMoXQAQ6yhegMNOlFXo/gUF+ufm/frpLYM15d9S3I4EAHAIxQtw0JFguSbP9atwx2E9eXeq7hx5gduRAAAOongBDtl3rFQTc/zaWnJCszJG6vpLu7kdCQDgMIoX4IAdB4OakJOvfcfKNGfyKF1+YSe3IwEAXMDn1oFGtnnfMd397AodPHFKudPSKV3RoChPWtZXWpgQvi3K82aGaHgeABqEPV5AI1r32RFNesGvxASjJQ+M0UXd2rgdCUV5kj9LCgXDy8Ht4WVJSsnwToZoeB4AGow9XkAjyd96QPc+/6FaJCVq6f2UrqhROKO6rJwWCobHvZQhGp4HgAZjjxfQCN7+ZJ8eyC1Qrw7Jys1MV7e2zd2OhNOCxQ0bj9YM0fA8ADQYe7yACHutcJfumx/QwK6tteT+MZSuaJPcu2Hj0ZohGp4HgAajeAERlJe/Xd96cbVG9Gmvhfelq0PLpm5HwplSZ0qJyV8cS0wOj3spQzQ8DwANRvECIuSZ5Vs049X1umpQF82f6lPr5kluR0JNUjIkX7aU3EeSCd/6sp09ID0SGaLheQBoMGOtdTtDndLS0mwgEHA7BlAja62e+OtGPbN8i76S2kNPjktVUiL/pwGAeGWMKbDWptV0HwfXA+ehstLqx39Yr7z8YmWk99Zjt16qxAQudg0AqBnFCzhH5aFK/cfSQv1hzS49OLa/HrlukIyhdAEAakfxAs5BaXlIX89bpb9/sk+PXD9I08cOcDsSAMADKF5AAx0rLde0eQH5tx3U47ddqvGj+7gdCQDgERQvoAEOnjilyS/4tWHXUf32nmG6dVhPtyMBADyE4gXU054jpRqfk68dB4PKnjhSV1/U1e1IAACPoXgB9bBt/wmNz8nX4WC55k31aXS/jm5HAgB4EMULqMMne45qQo5fFaFKLbpvtIZc0NbtSAAAj+Isj8BZrCo+pHue+1CJxmjJ/WMoXQCA88IeL6AW72/er/vmB9S5dTPlZqarV4fkur8JAICzoHgBNfjrR3v0zYWr1a9zS82f6lOXNs3djgQAiAEUL+AMLxd8pkdeXquhF7TVC5NHqV1yU7cjAQBiRMSP8TLGtDXG/NkY86Yx5lVjzL/81jLGNDHGFBtjllf9GRLpHMC5mPt+kb67tFCj+3VQbmZ67JSuojxpWV9pYUL4tijP7UTuOt/5YD5jEz9XOKAx9nhlSHrKWvuWMeYZSddLeu2MdYZKWmSt/X4jPD7QYNZa/f4fm/XUW5v05cFd9bt7h6t5UqLbsSKjKE/yZ0mhYHg5uD28LEkpGe7lcsv5zgfzGZv4ucIhEd/jZa2dZa19q2qxs6R9Naw2WtLNxhi/MSbHGMNbnnCNtVYz3/hYT721SXeM6KlZGSNip3RJUuGM6l8mp4WC4fF4dL7zwXzGJn6ucEijnU7CGDNGUntr7Yc13L1S0jXWWp+kJEk31vD9WcaYgDEmUFJS0lgxEedClVbff3mtZv+zSJMv66tf35WqJokxdpaVYHHDxmPd+c4H8xmb+LnCIY3yG8YY00HS7yVNrWWVtdba3VVfByRdeOYK1tpsa22atTatc+fOjRETca6sIqRvLFylJYHP9K1/v1A/vWWwEhKM27EiL7l3w8Zj3fnOB/MZm/i5wiGNcXB9U0lLJf3QWru9ltUWGGNSjTGJkm6TVBjpHMDZBE9VaNq8gP68fo9+fPNgPXztQBkTg6VLklJnSolnnIMsMTk8Ho/Odz6Yz9jEzxUOaYw9XpmSRkiaUfWJxZ8aYx4/Y53HJC2QtEbSCmvt3xohB1CjIyfLNSHHr/c379cTdw1V5uUpbkdqXCkZki9bSu4jyYRvfdnxe8Dw+c4H8xmb+LnCIcZa63aGOqWlpdlAIOB2DMSAkmNlmjjHr837jul3Xx2uG4Z0dzsSACDGGGMKrLVpNd3HpwkRN3YePqnxs/O150ipciaN0pUDOXYQAOAsihfiwpaS45owO1/HyiqUO82nkX06uB0JABCHKF6Ieet3HtGkOX4ZIy3OGqPBPdq4HQkAEKcoXohpK7cd1NQXVqpNiyQtyPSpX+dWbkcCAMQxihdi1vKN+/RAboF6tGuh3Mx09WjXwu1IAIA4R/FCTPrj2l16ePEaDezaWvOm+tSpVTO3IwEAQPFC7HnRX6wfvrpOaX3aK2fyKLVpnuR2JAAAJFG8EGOy392iX/zpE40d1FnPZIxUi6YxdLFrAIDnUbwQE6y1evLNTXr67c26aWh3/WbcMDVtEmMXuwYAeB7FC55XWWn1s9c/0vwV23Wvr5cev22IEmPxYtcAAM+jeMHTykOVeuSltXp19U7df2U//eCGi2L3YtcAAM/jvRh4Vml5SA/mrtKrq3fqe9cNis7SVZQnLesrLUwI3xbluZ0IkuSfLi1qIi004Vv/dLcTAYgT7PGCJx0vq9B98wJasfWA/vPWSzRhTF+3I/2rojzJnyWFguHl4PbwsiSlZLiXK975p0ubn6letqHqZd8sdzIBiBvs8YLnHDpxShmz8+XfdlC/uSc1OkuXJBXOqC5dp4WC4XG4Z0t2w8YBIILY4wVP2Xu0VBNy8rXtQFDPjh+pawd3dTtS7YLFDRuHM2yoYeMAEEHs8YJnFB8I6u5nV2jnoZOaO2VUdJcuSUru3bBxOMPUcm632sYBIIIoXvCETXuP6a5nP9DR0nLl3Tdal/Xv5HakuqXOlBKTvziWmBweh3v6ZzVsHAAiiOKFqLdmx2GNe26FJGnJ/WM0rFc7lxPVU0qG5MuWkvtIMuFbXzYH1rvNN0sa8GD1Hi6TGF7mwHoADjDWWrcz1CktLc0GAgG3Y8AFH2zZr/vmBdSxVTPlZqard8fkur8JAAAXGWMKrLVpNd3HwfWIWm9t2KuvL1ylvh2TtSAzXV3bNHc7EgAA54Xihaj06urP9B9L1+rSnm01d/IotW/Z1O1IAACcN4oXos6CFdv04z98pDH9Our5SWlq1YzNFAAQG/iNhqhhrdWs5Vv033/dqGsu7qqnvzZczZP4iD8AIHZQvBAVrLX61Z8/0XPvbtXtw3vqibuGKimRD90CAGILxQuuC1VaPbpsnRb5d2jimD762S2XKCEhyi52DQBABFC84KpTFZV6eMkavbF2t75x1QB998sDZQylCwAQmyhecM3JUyE9kFugdzaV6Ec3XqSsK/u7HQkAgEZF8YIrjpaWK3PuSgW2H9Kv7hiir/q4fiEAIPZRvOC4/cfLNGmOX5v2HtPT947QTUO7ux0JAABHULzgqF2HT2p8Tr52HT6p5yemaeygLm5HAgDAMRQvOGZryXFNyPHr6MlyLchM16i+HdyOBACAozhREiIuL0/q21dKSAjf5uVJG3Yd1bjnVqi0PKRFWaMpXQ1RlCct6ystTAjfFuW5ncj7mFPUhO0CDmCPFyIqL0/KypKCwfDy9u3S9J8cVPd7VqpD6yZaMC1d/Tu3cjeklxTlSf4sKVQ1ocHt4WVJSslwL5eXMaeoCdsFHMIeL0TUjBnVpUuSmvctUdtb/So93ExLH7yM0tVQhTOqfxGcFgqGx3FumFPUhO0CDqF4IaKKi6u/Th64W13uWqmKQy312fwx6tmuhXvBvCpY3LBx1I05RU3YLuAQihciqnfV6bhaDtmhTreuUtnudtqzaLQu6NTM3WBelVzL+c1qG0fdmFPUhO0CDqF4IaJmzpQ6XrZVnW5cq9LtnbRviU8tEpM0c6bbyTwqdaaUmPzFscTk8DjODXOKmrBdwCEUL0SMtVZ7u2xSqys+lnZ0U8kraerdo4mys6UMjk09NykZki9bSu4jyYRvfdkc7Hs+mFPUhO0CDjHWWrcz1CktLc0GAgG3Y+AsKiutHvvjBs39YJvGpV2gX9w+RE0S6fUAgPhjjCmw1qbVdB+nk8B5qwhV6pGX1+qVVTuVeXmKHr3pYhlj3I4FAEDUoXjhvJSWh/TQotV6c8Nefefagfrm1QMoXQAA1ILihXN2oqxCWQsCen/zAf3slsGa/G8pbkcCACCqUbxwTg4HT2nK3JVa+9kRPXl3qu4ceYHbkQAAiHoULzTYvmOlmpjj19aSE5qVMULXXdLN7UgAAHgCxQsNsuNgUONz8lVyrEwvTBmlfxvQye1IAAB4BsUL9fbp3mMan5Ov0vJK5U5L14je7d2OBACAp1C8UC9rPzusSXP8apKYoMX3j9ZF3dq4HQkAAM+heKFOH249oGnzAmqXnKTczHT17dTS7UgAAHgSxQtn9Y9P9urB3FXq1SFZuZnp6ta2uduRAADwLIoXavWHNTv13SWFurh7G82b6lOHlk3djgQAgKdRvFCj3A+368d/WC9f3w6aPSlNrZsnuR0JAADP4yrG+Bezlm/Wo8vW66pBXTRvqo/SBeDsivKkZX2lhQnh26K8+M4BnAV7vPD/rLX6r79s1LPvbNGtw3ro13enKimRbg7gLIryJH+WFAqGl4Pbw8uSlJIRfzmAOvBbFZKkUKXVo8vW69l3tigjvbd+M24YpQtA3QpnVJed00LB8Hg85gDqwB4vqDxUqe8sKdTrhbs0fWx/fe+6QTLGuB0LgBcEixs2Hus5gDqwSyPOlZaHdP+CAr1euEvfv/4iPXL9RZQuAPWX3Lth47GeA6gDxSuOHSst18Q5fr29cZ9m3n6pHhzb3+1IALwmdaaUmPzFscTk8Hg85gDqQPGKUwdPnNLXns/Xqu2H9D9fHa6M9D5uRwLgRSkZki9bSu4jyYRvfdnOH9AeLTmAOhhrrdsZ6pSWlmYDgYDbMWLG7iMnNSHHrx0Hg3pm/AhdfVFXtyMBABAzjDEF1tq0mu7j4Po4s23/CWXMzteRk+WaP9Wn9H4d3Y4EAEDcoHjFkY93H9WEHL9ClZVadN9oDbmgrduRAACIKxSvOLGq+JAmz/EruWkTvZg1RgO6tHY7EgAAcYfiFQf++el+ZS0IqHPrZsrNTFevDsl1fxMAAIg4ileM+8v6PXpo0Wr169xS8zN96tK6uduRAACIWxSvGPZSwWd65KVCpfZqp7mTfWqbzMWuAQBwE8UrRr3wfpF+/voGXT6gk56bMFItm/GjBgDAbfw2jjHWWv3u75v1m79t0nWXdNXv7h2uZk0S3Y4FAABE8YoplZVWj7/xsea8X6Q7R1yg/7pziJokcnECAACiBcUrRlSEKvXDV9ZpacFnmnxZX/3k5sFKSOBi1wAARBOKVwwoqwjpW4vW6C8f7dG3/v1CffuaC2UMpQsAgGhD8fK44KkK3b+gQO99ul8/vnmwMi9PcTsSAACoBcXLw44EyzV13kqtLj6kJ+4aqnFpvdyOBAAAzoLi5VElx8o0ISdfW0qOa1bGCF1/aXe3IwEAgDpQvDzos0NBTcjxa8+RUuVMGqUrB3Z2OxIAAKgHipfHbN53XBNy8nWirEK503wa2aeD25EAAEA9Ubw8ZP3OI5o4x68EI72YNUaDe7RxOxIAAGgAipdH+IsOKnPuSrVpkaTcaelK6dTS7UgAAKCBKF4e8PbGfXpgQYF6tm+h3Mx09WjXwu1IAADgHFC8otzrhbv08OI1uqh7a82b4lPHVs3cjgQAAM4RxSuKLfIX60evrtOoPh00e3Ka2jRPcjsSAAA4DxSvKPXcO1v0yz9/orGDOuuZjJFq0TTR7UgAAOA8UbyijLVWv35zo/737S26eWh3PTVumJo2SXA7FgAAiACKVxSprLT66WsfacGH23Wvr5cev22IEhO42DUAALGC4hUlykOV+t7SQi1bs0v3X9lPP7jhIhlD6QIAIJZQvKJAaXlI31i4Sn/7eJ++d90gTR/bn9IFAEAMoni57HhZhabNW6n8ooP6z9su1YTRfdyOBAAAGgnFy0WHTpzS5Bf8Wr/rqH4zbphuG97T7UgAAKARUbxcsudIqSbk5Gv7waCeGz9S1wzu6nYkAADQyCheLig+EFRGzoc6ePyU5k3xaUz/jm5HAgAADqB4OWzjnmOakJOvU6FKLbxvtFJ7tXM7EgAAcEjEz8xpjGlrjPmzMeZNY8yrxpimtayXY4xZYYx5NNIZotWaHYc17rkVkqQl94+hdKF+ivKkZX2lhQnh26I8txMBAM5RY5wSPUPSU9baL0vaI+n6M1cwxtwhKdFaO0ZSP2PMhY2QI6p8sHm/Mp7/UG1bJOmlBy7TwK6t3Y4ELyjKk/xZUnC7JBu+9WdRvgDAoyJevKy1s6y1b1Utdpa0r4bVxkpaUvX1m5Iuj3SOaPLmR3s0ee5K9WzfQi89MEa9Oya7HQleUThDCgW/OBYKhscBAJ7TaBcBNMaMkdTeWvthDXe3lLSz6uuDkv7lI33GmCxjTMAYEygpKWmsmI3u1dWf6cG8Vbq4exstzhqjLm2aux0JXhIsbtg4ACCqNUrxMsZ0kPR7SVNrWeW4pBZVX7eqKYe1Nttam2atTevcuXNjxGx08z7YpocXFyo9pYPypqWrfcsaD3cDapfcu2HjAICo1hgH1zeVtFTSD62122tZrUDVby+mStoW6Rxustbq6X98qp++9pGuHdxVcyaPUqtmfIAU5yB1ppR4xlvTicnhcQCA5zRGG8iUNELSDGPMDElvS0qy1n7+04vLJL1njOkh6QZJoxshhyustfrFnz7W8+8V6fbhPfXEXUOVlNho7+gi1qVkhG8LZ4TfXkzuHS5dp8cBAJ5irLXuPLAx7SVdK+lda+2es62blpZmA4GAM8HOQ6jS6kevrNPiwA5NHNNHP7vlEiUkcLFrAADiiTGmwFqbVtN9rr3/Za09pOpPNnreqYpKPbx4jd5Yt1vfvHqAvnPtQBlD6QIAANU48CgCTp4K6YHcAr2zqUQzbrxY913Zz+1IAAAgClG8ztORk+XKnLtSq4oP6b/uHKJ7RvFpMwAAUDOK13nYf7xME3P8+nTfMf3+3hG6aWh3tyMBAIAoRvE6RzsPn9SE2fnadeSknp+YprGDurgdCQAARDmK1znYWnJc42fn61hphRZkpmtU3w5uRwIAAB5A8Wqgj3Yd0aQ5flkrLcoarUt7tnU7EgAA8AiKVwMEth3UlLkr1bpZEy2Ylq7+nVu5HQkAAHgIxaue3tlUovsXBNSjbQstmJaunu1a1P1NAAAAn0Pxqoc/rdutb724WgO6tNb8qT51bt3M7UgAAMCDKF51WLJyh37wylqN6N1eOZNHqW2LJLcjAQAAj6J4ncXs97bq8Tc+1pUDO+vZ8SOU3JTpAgAA544mUQNrrX7z1ib97h+bdeOQbvrtPcPVtEmC27EAAIDHUbzOUFlp9dgfN2juB9t0T1ov/eKOIUpM4GLXAADg/FG8PqciVKlHXlqrV1bv1LTLUzTjpotlDKULAABEBsWrSml5SA8tWq03N+zVd68dqG9cPYDSBQAAIoriJel4WYWy5gf0wZYD+vlXLtGky/q6HQkAAMQgipek4KkK7TlSqifvTtWdIy9wOw4AAIhRFC9JXVo311++fSWfXAQAAI2KplGF0gUAABobbQMAAMAhFC8AAACHULwAAAAcQvECAABwCMULAADAIRQvAAAAh1C8AAAAHELxAgAAcAjFCwAAwCEULwAAAIdQvAAAABxC8QIAAHAIxQsAAMAhFC8AAACHULwAAAAcQvECAABwCMULAADAIRQvAAAAh1C8AAAAHELxAgAAcAjFCwAAwCEULwAAAIcYa63bGepkjCmRtN2Bh+okab8DjxMvmM/IYj4jjzmNLOYzspjPyHJyPvtYazvXdIcnipdTjDEBa22a2zliBfMZWcxn5DGnkcV8RhbzGVnRMp+81QgAAOAQihcAAIBDKF5flO12gBjDfEYW8xl5zGlkMZ+RxXxGVlTMJ8d4AQAAOIQ9XgAAAA6heCEijDFdjTGra7mviTGm2BizvOrPEKfzIT7Vd9szxqz53DrXOp0TMMbMMsbcUst9vIbGkLgsXmcrCVX35xhjVhhjHnUyl8f9WlJ2CqPhAAAFiUlEQVSLWu4bKmmRtXZs1Z91DubylAYUhZ8bY1YaY/7X6YweU+e2Z4zpKOmTz63zlvMxvedsRaHqfl5H68kYc4Wkbtba12tZhdfQejLGPPi51881xpjnalnPte0zLouXzlISjDF3SEq01o6R1M8Yc6GjyTzIGHO1pBOS9tSyymhJNxtj/FUbexPn0nlOfYrCSEmXS/JJ2meMucbpkB5Sn20vXZLPGPOBMWaZMaa1wxk9p66iwOto/RljkiQ9L2mbMebWWlbjNbSerLXPnH79lPSewnP7BW5vn3FXvOpREsZKWlL19ZsK/4JDLYwxTSX9WNIPzrLaSknXWGt9kpIk3ehENo+qzwvslyS9bMOfjPmrpCscTegt9dn2tkq6zlp7maS1kqY4mM9z6lkUxorX0fqaKGmDpCcU/g/AN2tYh9fQBjLG9JTU1VobqOHusXJx+4yr4lXPktBS0s6qrw9K6trYuTzuB5JmWWsPn2Wdtdba3VVfByTxv9/a1ecFlm20/uqz7W2VtLmOdVCtPkWBbbT+hkvKttbukZQr6aoa1uE1tOG+LumZWu5zdfuMq+Kl+pWE46p+G7KV4m+OGuoaSV83xiyXNMwYM7uGdRYYY1KNMYmSbpNU6GRAj6nPCyzbaP3VZ9ubKen0sUp31bIOqtWnKLCN1t9mSf2qvk5Tzdcl5jW0AYwxCQpvl8trWcXV7TPe/jHUpyQUqHq3Y6qkbc5E8yZr7ZWfez99jaSnjDGPn7HaY5IWVN2/wlr7N4djekl9XmDZRuvvC9uepFU1/Lt/StIMY8x6SWWS5jkb0XPqUxTYRusvR9JVxph3JU2X9BKvoeftCkn5tvYTlbq6fcbtCVSrytd0SV+z1j76ufE2Ch+Q93dJN0gaba094kpIxB1jzKWSFkoykl6T9KSkJ6y10z63ToLC22hA0vWSrrfWFrkQF3Go6sMHcxR+eyZJ0n9IuoHXUUQLY8wvJAWsta8YYwYryn7Px23xOhtjTHtJ10p6t2p3OhBVjDEtJN0kaZW1dqvbeYAz8TqKaObm9knxAgAAcEi8HeMFAADgGooXAACAQyheAGJW1YdoIrZePf+uvsaYsZ9bnmyMmRypvx+At1G8ACCy+ip8ZmwA+Bdc7wmA66o+pblUUhtJByTdLamppPmSukhaZ639ujHmZwpfWzFZUomkr0qyCp/jqI+k/ZLustaWn0cWIylb0sCqx7hH0gSFz/eTKqmbpHGSPlL4BKL9FT4L9vaqP1Mktava63V31V+baoz5x+nvtdauP9d8ALyNPV4AosFgSZXW2islvaDw2aSzJK2vGutujBlate571tovSdor6VZJHSW9ofA1LI9KGnGeWW6VlFT1GMUKn7ZDkkZJuk7SryR9RVJ7SV2staMlpVhrv2Ot/R9J35Y0t+rEwiW1fC+AOEXxAhANVklab4x5U+GCEpQ0SNLtVcdf9ZPUs2rdgqrbtQq/rVcu6WaF95j1U/WlQM7VIEljqh73SlVfx21R1Z60YoX3xgUlNTPG5EvKq+PvPPN7AcQpiheAaJAq6X1r7ZcV3pN0haSNkn5bdTmqRxUuLZLkq7odrvDla+6QtL7qdqfO30ZJL1Y97rcVviC0JJ04Yz2fpFettenW2ic/N35S4bdCT79tWdP3AohTFC8A0WCbpIeMMR8ofBxUQNLzkm6ouobdA5J2VK07qmpvVDtJf5T0vsLHYf1TUgdV7xk7V69J6mGMeUfS46r5WoSS9Imk7xhj3jbGvGyMuaJqfLWkQcaY96pyAcD/48z1ADyj6uD65dba5S5HkTHmJkmPSDql8F6uhdbaF91NBSDaUbwAAAAcwluNAAAADqF4AQAAOITiBQAA4BCKFwAAgEMoXgAAAA6heAEAADjk/wDUHmK+zOsfCQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "clf = Perceptron(fit_intercept=True, \n", " max_iter=1000,\n", " tol=None,\n", " shuffle=True)\n", "clf.fit(X, y)\n", "\n", "# 画布大小\n", "plt.figure(figsize=(10,10))\n", "\n", "# 中文标题\n", "plt.rcParams['font.sans-serif']=['SimHei']\n", "plt.rcParams['axes.unicode_minus'] = False\n", "plt.title('鸢尾花线性数据示例')\n", "\n", "plt.scatter(data[:50, 0], data[:50, 1], c='b', label='Iris-setosa',)\n", "plt.scatter(data[50:100, 0], data[50:100, 1], c='orange', label='Iris-versicolor')\n", "\n", "# 画感知机的线\n", "x_ponits = np.arange(4, 8)\n", "y_ = -(clf.coef_[0][0]*x_ponits + clf.intercept_)/clf.coef_[0][1]\n", "plt.plot(x_ponits, y_)\n", "\n", "# 其他部分\n", "plt.legend() # 显示图例\n", "plt.grid(False) # 不显示网格\n", "plt.xlabel('sepal length')\n", "plt.ylabel('sepal width')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在可以看到，所有的两种鸢尾花都被正确分类了。\n", "\n", "----" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第2章感知机-习题\n", "\n", "### 习题2.1\n", "  Minsky 与 Papert 指出：感知机因为是线性模型，所以不能表示复杂的函数，如异或 (XOR)。验证感知机为什么不能表示异或。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "\n", "对于异或函数XOR，全部的输入与对应的输出如下： \n", "\n", "|
$x^{(1)}$
|
$x^{(2)}$
|$y$|\n", "|:-: | :-: | :-: | \n", "|  1 |  1 |-1 | \n", "|  1 | -1 |  1 | \n", "|-1 |  1 |  1 | \n", "|-1 | -1 |-1 | " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "参考代码：https://github.com/wzyonggege/statistical-learning-method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第03章 k近邻法/3.KNearestNeighbors.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第3章 k近邻法" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1．$k$近邻法是基本且简单的分类与回归方法。$k$近邻法的基本做法是：对给定的训练实例点和输入实例点，首先确定输入实例点的$k$个最近邻训练实例点，然后利用这$k$个训练实例点的类的多数来预测输入实例点的类。\n", "\n", "2．$k$近邻模型对应于基于训练数据集对特征空间的一个划分。$k$近邻法中，当训练集、距离度量、$k$值及分类决策规则确定后，其结果唯一确定。\n", "\n", "3．$k$近邻法三要素：距离度量、$k$值的选择和分类决策规则。常用的距离度量是欧氏距离及更一般的**pL**距离。$k$值小时，$k$近邻模型更复杂；$k$值大时，$k$近邻模型更简单。$k$值的选择反映了对近似误差与估计误差之间的权衡，通常由交叉验证选择最优的$k$。\n", "\n", "常用的分类决策规则是多数表决，对应于经验风险最小化。\n", "\n", "4．$k$近邻法的实现需要考虑如何快速搜索k个最近邻点。**kd**树是一种便于对k维空间中的数据进行快速检索的数据结构。kd树是二叉树，表示对$k$维空间的一个划分，其每个结点对应于$k$维空间划分中的一个超矩形区域。利用**kd**树可以省去对大部分数据点的搜索， 从而减少搜索的计算量。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 距离度量" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "设特征空间$x$是$n$维实数向量空间 ，$x_{i}, x_{j} \\in \\mathcal{X}$,$x_{i}=\\left(x_{i}^{(1)}, x_{i}^{(2)}, \\cdots, x_{i}^{(n)}\\right)^{\\mathrm{T}}$,$x_{j}=\\left(x_{j}^{(1)}, x_{j}^{(2)}, \\cdots, x_{j}^{(n)}\\right)^{\\mathrm{T}}$\n", "，则：$x_i$,$x_j$的$L_p$距离定义为:\n", "\n", "\n", "$L_{p}\\left(x_{i}, x_{j}\\right)=\\left(\\sum_{i=1}^{n}\\left|x_{i}^{(i)}-x_{j}^{(l)}\\right|^{p}\\right)^{\\frac{1}{p}}$\n", "\n", "- $p= 1$ 曼哈顿距离\n", "- $p= 2$ 欧氏距离\n", "- $p= \\infty$ 切比雪夫距离" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import math\n", "from itertools import combinations" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "def L(x, y, p=2):\n", " # x1 = [1, 1], x2 = [5,1]\n", " if len(x) == len(y) and len(x) > 1:\n", " sum = 0\n", " for i in range(len(x)):\n", " sum += math.pow(abs(x[i] - y[i]), p)\n", " return math.pow(sum, 1 / p)\n", " else:\n", " return 0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 课本例3.1" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "x1 = [1, 1]\n", "x2 = [5, 1]\n", "x3 = [4, 4]" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(4.0, '1-[5, 1]')\n", "(4.0, '1-[5, 1]')\n", "(3.7797631496846193, '1-[4, 4]')\n", "(3.5676213450081633, '1-[4, 4]')\n" ] } ], "source": [ "# x1, x2\n", "for i in range(1, 5):\n", " r = {'1-{}'.format(c): L(x1, c, p=i) for c in [x2, x3]}\n", " print(min(zip(r.values(), r.keys())))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "python实现，遍历所有数据点，找出$n$个距离最近的点的分类情况，少数服从多数" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "from sklearn.datasets import load_iris\n", "from sklearn.model_selection import train_test_split\n", "from collections import Counter" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# data\n", "iris = load_iris()\n", "df = pd.DataFrame(iris.data, columns=iris.feature_names)\n", "df['label'] = iris.target\n", "df.columns = ['sepal length', 'sepal width', 'petal length', 'petal width', 'label']\n", "# data = np.array(df.iloc[:100, [0, 1, -1]])" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sepal lengthsepal widthpetal lengthpetal widthlabel
05.13.51.40.20
14.93.01.40.20
24.73.21.30.20
34.63.11.50.20
45.03.61.40.20
..................
1456.73.05.22.32
1466.32.55.01.92
1476.53.05.22.02
1486.23.45.42.32
1495.93.05.11.82
\n", "

150 rows × 5 columns

\n", "
" ], "text/plain": [ " sepal length sepal width petal length petal width label\n", "0 5.1 3.5 1.4 0.2 0\n", "1 4.9 3.0 1.4 0.2 0\n", "2 4.7 3.2 1.3 0.2 0\n", "3 4.6 3.1 1.5 0.2 0\n", "4 5.0 3.6 1.4 0.2 0\n", ".. ... ... ... ... ...\n", "145 6.7 3.0 5.2 2.3 2\n", "146 6.3 2.5 5.0 1.9 2\n", "147 6.5 3.0 5.2 2.0 2\n", "148 6.2 3.4 5.4 2.3 2\n", "149 5.9 3.0 5.1 1.8 2\n", "\n", "[150 rows x 5 columns]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEHCAYAAACjh0HiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5wcdZ3n8df7hqyJCuSAuGsy0aB4eQghSyCCGBd/4C6K2RBREW7Z3Sgnq4sLHis+jOeioiconnjInS6gK4objFmMgAIqGAURuAnBRBLxJ25m4I4YNgHWoCF+7o+qSSZDz0zXdH+7q6rfz8djHtNdXV3zqWroT6rq8/l+FRGYmVnv+g/dDsDMzLrLicDMrMc5EZiZ9TgnAjOzHudEYGbW45wIzMx63D6p/4CkPmAAGIqIxaNeWwZcDAzliy6LiCvH295BBx0Uc+bMSRCpmVl9rV279tcRMaPRa8kTAXAOsAnYb4zXvxwR72h2Y3PmzGFgYKAtgZmZ9QpJvxrrtaSXhiT1A68Fxv1XvpmZdU/qewSfBN4N/H6cdV4vab2kVZJmN1pB0pmSBiQNbNmyJUmgZma9KlkikLQYeDgi1o6z2vXAnIiYD3wbuKrRShFxeUQsjIiFM2Y0vMRlZmaTlPIewSJgiaQTganAfpKujojTh1eIiK0j1r8C+GjCeMzMWrJz504GBwd54oknuh3KmKZOnUp/fz9Tpkxp+j3JEkFELAeWA0h6OfCukUkgX/7siHgof7qE7KaymVkpDQ4Osu+++zJnzhwkdTucp4gItm7dyuDgIAcffHDT7+t4H4GkCyQtyZ+eLek+ST8EzgaWdToeM7NmPfHEExx44IGlTAIAkjjwwAMLn7F0onyUiFgDrMkfnz9i+e6zBrO6Wb1uiItvvp8Ht+1g5vRpnHfCXJYumNXtsKxFZU0CwyYTX0cSgVmvWb1uiOXXbmDHzl0ADG3bwfJrNwA4GVjpeIgJswQuvvn+3Ulg2I6du7j45vu7FJHVxU033cTcuXM55JBDuOiii9qyTScCswQe3Laj0HKzZuzatYuzzjqLG2+8kY0bN7JixQo2btzY8nZ9acgsgZnTpzHU4Et/5vRpXYjGuqXd94nuvvtuDjnkEJ73vOcBcOqpp/K1r32NQw89tKU4fUZglsB5J8xl2pS+vZZNm9LHeSfM7VJE1mnD94mGtu0g2HOfaPW6oQnfO5ahoSFmz94zAEN/fz9DQ5Pf3jAnArMEli6YxYUnH86s6dMQMGv6NC48+XDfKO4hKe4TRcRTlrWjismXhswSWbpglr/4e1iK+0T9/f1s3rx59/PBwUFmzpw56e0N8xmBmVkCY90PauU+0Yte9CJ++tOf8stf/pLf/e53XHPNNSxZsmTiN07AicDMLIEU94n22WcfLrvsMk444QRe+MIXcsopp3DYYYe1GqovDZmZpTB8WbDd3eUnnngiJ554YjtC3M2JwMwskarcJ/KlITOzHudEYGbW45wIzMx6nBOBmVmPcyIwM+txTgTW81avG2LRRbdy8Hu+zqKLbm1pLBiz1N7ylrfwrGc9i3nz5rVtm04E1tNSDAxmltKyZcu46aab2rpNJwLraZ5AxpJavxIumQcfmJ79Xr+y5U0ed9xxHHDAAW0Ibg83lFlP8wQylsz6lXD92bAz/29p++bsOcD8U7oXVwM+I7CelmJgMDMAbrlgTxIYtnNHtrxknAisp3kCGUtm+2Cx5V3kS0PW01INDGbG/v3Z5aBGy0vGicB6XlUGBrOKOf78ve8RAEyZli1vwWmnncaaNWv49a9/TX9/Px/84Ac544wzWtqmE4F1Tbsn9jYrleEbwrdckF0O2r8/SwIt3ihesWJFG4LbmxOBdcVw/f5w6eZw/T7gZGD1Mf+U0lUINeKbxdYVrt83Kw8nAusK1+9bVUVEt0MY12TicyKwrnD9vlXR1KlT2bp1a2mTQUSwdetWpk6dWuh9vkdgXXHeCXP3ukcArt+38uvv72dwcJAtW7Z0O5QxTZ06lf7+YiWqTgTWFa7ftyqaMmUKBx98cLfDaDsnAusa1++blUPyRCCpDxgAhiJi8ajXngZ8ATgK2Aq8KSIeSB2TWdm4p8K6qRM3i88BNo3x2hnAv0XEIcAlwEc7EI9ZqXhOBOu2pIlAUj/wWuDKMVY5Cbgqf7wKOF6SUsZkVjbuqbBuS31G8Eng3cDvx3h9FrAZICKeBLYDB45eSdKZkgYkDZT5br3ZZLinwrotWSKQtBh4OCLWjrdag2VPKdCNiMsjYmFELJwxY0bbYjQrA/dUWLelPCNYBCyR9ABwDfBKSVePWmcQmA0gaR9gf+CRhDGZlY7nRLBuS5YIImJ5RPRHxBzgVODWiDh91GrXAX+dP35Dvk45W/bMElm6YBYXnnw4s6ZPQ8Cs6dO48OTDXTVkHdPxPgJJFwADEXEd8Fngi5J+RnYmcGqn4zErA/dUWDd1JBFExBpgTf74/BHLnwDe2IkYrHe8b/UGVty1mV0R9EmcdsxsPrz08G6HZVZa7iy2Wnnf6g1cfee/7n6+K2L3cycDs8Y8+qjVyoq7GswRO85yM3MisJrZNUatwVjLzcyJwGqmb4zG9LGWm5kTgdXMacfMLrTczHyz2Gpm+Iawq4bMmqeq9W8tXLgwBgYGuh2GmVmlSFobEQsbveYzAmurv7jiB3z/53tGCVn0/AP40luP7WJE3eM5BqwqfI/A2mZ0EgD4/s8f4S+u+EGXIuoezzFgVeJEYG0zOglMtLzOPMeAVYkTgVkCnmPAqsSJwCwBzzFgVeJEYG2z6PkHFFpeZ55jwKrEicDa5ktvPfYpX/q9WjXkOQasStxHYGbWA9xHYB2Tqna+yHZdv29WjBOBtc1w7fxw2eRw7TzQ0hdxke2misGsznyPwNomVe18ke26ft+sOCcCa5tUtfNFtuv6fbPinAisbVLVzhfZruv3zYpzIrC2SVU7X2S7rt83K843i61thm/Gtrtip8h2U8VgVmfuIzAz6wHuIyiRMtS4F42hDDGbWTpOBB1Uhhr3ojGUIWYzS8s3izuoDDXuRWMoQ8xmlpYTQQeVoca9aAxliNnM0nIi6KAy1LgXjaEMMZtZWk4EHVSGGveiMZQhZjNLyzeLO6gMNe5FYyhDzGaWlvsIzMx6QFf6CCRNBb4HPC3/O6si4v2j1lkGXAwM5Ysui4grU8Vkk/O+1RtYcddmdkXQJ3HaMbP58NLDW163LP0JZYnDrFtSXhr6LfDKiHhc0hTgdkk3RsSdo9b7ckS8I2Ec1oL3rd7A1Xf+6+7nuyJ2Px/9BV9k3bL0J5QlDrNumvBmsaSnSfrPkt4r6fzhn4neF5nH86dT8p9qXYcyVty1uenlRdYtS39CWeIw66Zmqoa+BpwEPAn8+4ifCUnqk3Qv8DDwrYi4q8Fqr5e0XtIqSbPH2M6ZkgYkDWzZsqWZP21tsmuMe0iNlhdZtyz9CWWJw6ybmrk01B8Rr57MxiNiF3CEpOnAVyXNi4gfjVjlemBFRPxW0tuAq4BXNtjO5cDlkN0snkwsNjl9UsMv8j6ppXVnTp/GUIMv2073J5QlDrNuauaM4A5Jje/2NSkitgFrgFePWr41In6bP70COKqVv2Ptd9oxDU/SGi4vsm5Z+hPKEodZN415RiBpA9k1/X2AN0v6BdkNYJHdApg/3oYlzQB2RsQ2SdOAVwEfHbXOsyPiofzpEmDTpPfEkhi+ydtMJVCRdcvSn1CWOMy6acw+AknPHe+NEfGrcTcszSe71NNHduaxMiIukHQBMBAR10m6kCwBPAk8Arw9In483nbdR2BmVtx4fQQTNpRJ+mJE/OVEyzql6okgVc16kfr9lNsusn9VPBaVs34l3HIBbB+E/fvh+PNh/indjsq6oNWGssNGbawPX8uflFQ160Xq91Nuu8j+VfFYVM76lXD92bAzvxm+fXP2HJwMbC9j3iyWtFzSY8B8SY/mP4+RlYJ+rWMR1kiqmvUi9fspt11k/6p4LCrnlgv2JIFhO3dky81GGDMRRMSFEbEvcHFE7Jf/7BsRB0bE8g7GWBupataL1O+n3HaR/avisaic7YPFllvPGu+M4EhJRwJfGX488qeDMdZGqrH9G9Xpj7c81baL7F8Vj0Xl7N9fbLn1rPH6CP5H/vO/gLvIGrquyB9fmj60+klVs16kfj/ltovsXxWPReUcfz5MGZVYp0zLlpuNMObN4oh4BYCka4AzI2JD/nwe8K7OhFcvqWrWi9Tvp9x2kf2r4rGonOEbwq4asgk0Uz56b0QcMdGyTql6+aiZWTe0Wj66SdKVwNVkncan4w7gnlKG3gCrOPczlFozieDNwNuBc/Ln3wM+nSwiK5Uy9AZYxbmfofQmHHQuIp6IiEsi4nX5zyUR8UQngrPuK0NvgFWc+xlKb7xB51ZGxCkjBp/by0SDzlk9lKE3wCrO/QylN96loeFLQYs7EYiVU5Hx+j22vzW0f392OajRciuF8TqLh4eHPh74g4j41cifzoRn3VaG3gCrOPczlF4zN4vnAKfnw1KvBW4DbouIe1MGZuVQht4Aqzj3M5TehH0Eu1fMJpd5K1kz2ayI6JvgLUm4j8DMrLiW+ggkvQ9YBDwTWEeWCG5ra4QllKoevsh2yzKuvnsDSqbuNfl1378iOnQsmrk0dDLZDGJfB74L3Fn38tFU9fBFtluWcfXdG1Ayda/Jr/v+FdHBY9FMH8GRZDeM7wb+FNgg6fa2RlEyqerhi2y3LOPquzegZOpek1/3/Suig8eimUtD84A/AV4GLAQ2U/NLQ6nq4Ytstyzj6rs3oGTqXpNf9/0rooPHYsIzAuCjwL5kQ0+/MCJeERG1rvtKNVZ+ke2WZVz9VMfCJqnucwzUff+K6OCxaObS0Gsj4mMRcUdE7Gx7BCWUqh6+yHbLMq6+ewNKpu41+XXfvyI6eCyauVncc1LVwxfZblnG1XdvQMnUvSa/7vtXRAePRdN9BGXhPgIzs+JanY/A2sj9CWYVccO5sPbzELtAfXDUMlj8ida3W8I+ifFGH72eBqOODouIJUkiqjH3J5hVxA3nwsBn9zyPXXuet5IMStonMealIUkvG++NEfHdJBFNoMqXhhZddGvD0TlnTZ/G99/zyo5s9/nLv9GwBLVP4ucXnjjpGMxq5YMHZF/+o6kP3v/I5Ld7ybwxRmKdDf/1R5PfbhMmdWmoW1/0deb+BLOKaJQExlverJL2SUxYPirpBZJWSdoo6RfDP50Irm7cn2BWERpjTM2xljerpH0SzTSU/RPZHMVPAq8AvgB8MWVQdeX+BLOKOGpZseXNKmmfRDOJYFpE3EJ2P+FXEfEBYPIXtHvY0gWzuPDkw5k1fRoiu4Z/4cmHt6U/odntfnjp4Zz+4ufsPgPokzj9xc/xjWKzkRZ/AhaesecMQH3Z81arhuafAn9+aXZPAGW///zSrlcNTdhHIOn7ZGMNrQJuBYaAiyKiK62lVb5ZbGbWLa32EbwTeDpwNvAhsrOBv27ij04Fvgc8Lf87qyLi/aPWeRrZpaajgK3AmyLigSZiKqxo/X7VxuAv0htQ92ORtE67SG15qjhS7l8Ja9zbpui+1flYjFJkhrL9gIiIx5pcX8AzIuJxSVOA24FzIuLOEev8LTA/It4m6VTgdRHxpvG2O5kzgtF19pBdQx/r8knR9bttdG/AsEaXfOp+LJ5Spw3ZNdh2nH6Pri0f1uiSQao4Uu5fym13W9F9q+GxGO+MoJmqoYWSNgDryeYi+KGkoyZ6X2Qez59OyX9GZ52TgKvyx6uA4/ME0lZFx9Sv2hj8ReYuqPuxSDqG+9rPN788VRwp96/OcwEU3bc6H4sGmrlZ/DngbyNiTkTMAc4iqySakKQ+SfcCDwPfioi7Rq0yi2x+AyLiSWA7cGCD7ZwpaUDSwJYtW5r503spWr9ftTH4i/QG1P1YJK3TLlJbniqOlPtX0hr3tii6b3U+Fg00kwgei4jdE9FExO1AU5eHImJXRBwB9ANH55PcjNToX/9P+faKiMsjYmFELJwxY0Yzf3ovRev3qzYGf5HegLofi6R12kVqy1PFkXL/Slrj3hZF963Ox6KBZhLB3ZL+UdLLJb1M0v8G1kg6UtKRzfyRiNgGrAFePeqlQWA2gKR9gP2BFvq3Gytav1+1MfiL9AbU/VgkrdMuUlueKo6U+1fSGve2KLpvdT4WDTRTNXRE/vv9o5a/hOxf7w17CiTNAHZGxDZJ04BXkc12NtJ1ZBVIPwDeANwaCcbFLjqmftXG4C8yd0Hdj0XSMdyHbwg3UzWUKo6U+1fnuQCK7ludj0UDyeYjkDSf7EZwH9mZx8qIuEDSBcBARFyXl5h+EVhAdiZwakSMO3yF+wjMzIprqY9A0h8CHwFmRsRrJB0KHBsRDero9oiI9WRf8KOXnz/i8RPAGyeKwczM0mnmHsHngZuBmfnzn5A1mdXa6nVDLLroVg5+z9dZdNGtrF431O2QrAzWr8yGEv7A9Oz3+pXtWTeVojGUYf+qtt0aaOYewUERsVLScsjKPCW1OBZruaWaQMYqrsikImWYgKRoDGXYv6pttyaaOSP4d0kHkpd1SnoxWb1/bVWuico6o0iTURkaklI2UVWtYa4Mn0eJNXNGcC5Zdc/z8wHoZpBV+NRW5ZqorDOKNBmVoSEpZRNV1RrmyvB5lNiEZwQRcQ/wMrJy0b8BDstvBNdW5ZqorDOKNBmVoSEpZRNV1RrmyvB5lFgzYw29kWxOgvuApcCXm20kq6rKNVFZZxRpMipDQ1LKJqqqNcyV4fMosWbuEfxDRDwm6aXACWS9AZ9OG1Z3pZpAxiquyKQiZZiApGgMZdi/qm23JpqZmGZdRCyQdCGwISL+eXhZZ0LcmxvKzMyKa3VimiFJ/0g+REQ+mUwzZxJmva3IJDZlUbWYyzJ5TFnimKRmEsEpZIPFfTwfN+jZwHlpwzKruNGT2MSuPc/L+sVatZjL0htQljha0EzV0G8i4tqI+Gn+/KGI+Gb60MwqrMgkNmVRtZjL0htQljha4Es8ZikUmcSmLKoWc1l6A8oSRwucCMxSKDKJTVlULeay9AaUJY4WOBGYpVBkEpuyqFrMZekNKEscLXAiMEth8Sdg4Rl7/jWtvux5GW+6DqtazGXpDShLHC1INjFNKu4jMDMrrtU+ArM0qlh7nSrmVPX7VTzG1nFOBNYdVay9ThVzqvr9Kh5j6wrfI7DuqGLtdaqYU9XvV/EYW1c4EVh3VLH2OlXMqer3q3iMrSucCKw7qlh7nSrmVPX7VTzG1hVOBNYdVay9ThVzqvr9Kh5j6wonAuuOKtZep4o5Vf1+FY+xdYX7CMzMesB4fQQ+IzBbvxIumQcfmJ79Xr+y89tNFYNZE9xHYL0tVa19ke263t+6zGcE1ttS1doX2a7r/a3LnAist6WqtS+yXdf7W5c5EVhvS1VrX2S7rve3LnMisN6Wqta+yHZd729d5kRgvS1VrX2R7bre37rMfQRmZj2gK30EkmZL+o6kTZLuk3ROg3VeLmm7pHvzH58LV10V6+Fd75+ej1uppewjeBL4+4i4R9K+wFpJ34qIjaPWuy0iFieMwzqlivXwrvdPz8et9JKdEUTEQxFxT/74MWATMCvV37MSqGI9vOv90/NxK72O3CyWNAdYANzV4OVjJf1Q0o2SDhvj/WdKGpA0sGXLloSRWkuqWA/vev/0fNxKL3kikPRM4F+Ad0bEo6Nevgd4bkT8MfApYHWjbUTE5RGxMCIWzpgxI23ANnlVrId3vX96Pm6llzQRSJpClgS+FBHXjn49Ih6NiMfzx98Apkg6KGVMllAV6+Fd75+ej1vppawaEvBZYFNENBxYXdIf5esh6eg8nq2pYrLEqlgP73r/9HzcSi9ZH4GklwK3ARuA3+eL3ws8ByAiPiPpHcDbySqMdgDnRsQd423XfQRmZsWN10eQrHw0Im4HNME6lwGXpYrBxrB+ZVaxsX0wu057/Pm9/a+zG86FtZ/PJotXXzZFZKuzg5lViOcj6DWu6d7bDefCwGf3PI9de547GViP8FhDvcY13Xtb+/liy81qyImg17ime2+xq9hysxpyIug1runem/qKLTerISeCXuOa7r0dtazYcrMaciLoNa7p3tviT8DCM/acAagve+4bxdZDPB+BmVkP6EofQS9ZvW6Ii2++nwe37WDm9Gmcd8Jcli6o0UCrde87qPv+lYGPcak5EbRo9bohll+7gR07syqToW07WH7tBoB6JIO69x3Uff/KwMe49HyPoEUX33z/7iQwbMfOXVx88/1diqjN6t53UPf9KwMf49JzImjRg9t2FFpeOXXvO6j7/pWBj3HpORG0aOb0aYWWV07d+w7qvn9l4GNcek4ELTrvhLlMm7J389G0KX2cd8LcLkXUZnXvO6j7/pWBj3Hp+WZxi4ZvCNe2amj4Zl5dKz7qvn9l4GNceu4jMDPrAeP1EfjSkFmdrV8Jl8yDD0zPfq9fWY1tW0f50pBZXaWs33dvQK34jMCsrlLW77s3oFacCMzqKmX9vnsDasWJwKyuUtbvuzegVpwIzOoqZf2+ewNqxYnArK5Szj3heS1qxX0EZmY9wH0EZmY2JicCM7Me50RgZtbjnAjMzHqcE4GZWY9zIjAz63FOBGZmPc6JwMysxyVLBJJmS/qOpE2S7pN0ToN1JOlSST+TtF7SkanisRZ43HmzWks5H8GTwN9HxD2S9gXWSvpWRGwcsc5rgBfkP8cAn85/W1l43Hmz2kt2RhARD0XEPfnjx4BNwOiJfE8CvhCZO4Hpkp6dKiabBI87b1Z7HblHIGkOsAC4a9RLs4DNI54P8tRkgaQzJQ1IGtiyZUuqMK0RjztvVnvJE4GkZwL/ArwzIh4d/XKDtzxlFLyIuDwiFkbEwhkzZqQI08bicefNai9pIpA0hSwJfCkirm2wyiAwe8TzfuDBlDFZQR533qz2UlYNCfgssCkiPjHGatcBf5VXD70Y2B4RD6WKySbB486b1V7KqqFFwF8CGyTdmy97L/AcgIj4DPAN4ETgZ8BvgDcnjMcma/4p/uI3q7FkiSAibqfxPYCR6wRwVqoYzMxsYu4sNjPrcU4EZmY9zonAzKzHORGYmfU4JwIzsx7nRGBm1uOcCMzMepyyUv7qkLQF+FW34xjDQcCvux1EQt6/6qrzvoH3rxnPjYiGg7VVLhGUmaSBiFjY7ThS8f5VV533Dbx/rfKlITOzHudEYGbW45wI2uvybgeQmPevuuq8b+D9a4nvEZiZ9TifEZiZ9TgnAjOzHudEMAmS+iStk3RDg9eWSdoi6d785790I8ZWSHpA0oY8/oEGr0vSpZJ+Jmm9pCO7EedkNLFvL5e0fcTnV6k5OSVNl7RK0o8lbZJ07KjXK/vZQVP7V9nPT9LcEXHfK+lRSe8ctU6Szy/lDGV1dg6wCdhvjNe/HBHv6GA8KbwiIsZqYHkN8IL85xjg0/nvqhhv3wBui4jFHYumvf4ncFNEvEHSHwBPH/V61T+7ifYPKvr5RcT9wBGQ/WMTGAK+Omq1JJ+fzwgKktQPvBa4stuxdNFJwBcicycwXdKzux1Ur5O0H3Ac2VzhRMTvImLbqNUq+9k1uX91cTzw84gYPYpCks/PiaC4TwLvBn4/zjqvz0/bVkma3aG42imAb0paK+nMBq/PAjaPeD6YL6uCifYN4FhJP5R0o6TDOhlci54HbAH+Kb90eaWkZ4xap8qfXTP7B9X9/EY6FVjRYHmSz8+JoABJi4GHI2LtOKtdD8yJiPnAt4GrOhJcey2KiCPJTkPPknTcqNcbzUVdlTrkifbtHrIxWf4Y+BSwutMBtmAf4Ejg0xGxAPh34D2j1qnyZ9fM/lX58wMgv+S1BPhKo5cbLGv583MiKGYRsETSA8A1wCslXT1yhYjYGhG/zZ9eARzV2RBbFxEP5r8fJrtGefSoVQaBkWc6/cCDnYmuNRPtW0Q8GhGP54+/AUyRdFDHA52cQWAwIu7Kn68i++IcvU4lPzua2L+Kf37DXgPcExH/r8FrST4/J4ICImJ5RPRHxByyU7dbI+L0keuMul63hOymcmVIeoakfYcfA38G/GjUatcBf5VXMLwY2B4RD3U41MKa2TdJfyRJ+eOjyf4f2drpWCcjIv4vsFnS3HzR8cDGUatV8rOD5vavyp/fCKfR+LIQJPr8XDXUBpIuAAYi4jrgbElLgCeBR4Bl3YxtEv4Q+Gr+/9I+wD9HxE2S3gYQEZ8BvgGcCPwM+A3w5i7FWlQz+/YG4O2SngR2AKdGtdrv/w74Un554RfAm2vy2Q2baP8q/flJejrwp8DfjFiW/PPzEBNmZj3Ol4bMzHqcE4GZWY9zIjAz63FOBGZmPc6JwMysxzkRmBWUj3DZaOTZhsvb8PeWSjp0xPM1kmo7Ubt1nhOBWfktBQ6dcC2zSXIisNrJO4i/ng889iNJb8qXHyXpu/mAczcPd4Hn/8L+pKQ78vWPzpcfnS9bl/+eO97fbRDD5yT9n/z9J+XLl0m6VtJNkn4q6WMj3nOGpJ/k8Vwh6TJJLyHrUL9Y2Rj1z89Xf6Oku/P1/6RNh856lDuLrY5eDTwYEa8FkLS/pClkg5CdFBFb8uTw34G35O95RkS8JB+E7nPAPODHwHER8aSkVwEfAV7fZAz/jWwIkrdImg7cLenb+WtHAAuA3wL3S/oUsAv4B7Kxcx4DbgV+GBF3SLoOuCEiVuX7A7BPRBwt6UTg/cCrJnOgzMCJwOppA/BxSR8l+wK9TdI8si/3b+VfpH3AyDFaVgBExPck7Zd/ee8LXCXpBWQjPE4pEMOfkQ1Q+K78+VTgOfnjWyJiO4CkjcBzgYOA70bEI/nyrwD/aZztX5v/XgvMKRCX2VM4EVjtRMRPJB1FNibLhZK+STbS6H0RcexYb2vw/EPAdyLidZLmAGsKhCHg9fmsU3sWSseQnQkM20X2/2Gj4YXHM7yN4febTZrvEVjtSJoJ/CYirgY+Tna55X5ghvI5biVN0d6TlgzfR3gp2YiO24H9yaYLhOKDB94M/N2IkTAXTLD+3cDLJP1HSfuw9yWox8jOTsyS8L8krI4OJ7u5+ntgJ/D2iPidpDcAlz3SjnEAAACjSURBVEran+y//U8C9+Xv+TdJd5DNQz183+BjZJeGziW7Zl/Eh/Ltr8+TwQPAmPPoRsSQpI8Ad5GNL78R2J6/fA1whaSzyUbXNGsrjz5qPU/SGuBdETHQ5TieGRGP52cEXwU+FxGjJy83aztfGjIrjw9IupdsspxfUsFpFq2afEZgZtbjfEZgZtbjnAjMzHqcE4GZWY9zIjAz63FOBGZmPe7/A5LawLRyzHcuAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.scatter(df[:50]['sepal length'], df[:50]['sepal width'], label='0')\n", "plt.scatter(df[50:100]['sepal length'], df[50:100]['sepal width'], label='1')\n", "plt.xlabel('sepal length')\n", "plt.ylabel('sepal width')\n", "plt.legend()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "data = np.array(df.iloc[:100, [0, 1, -1]])\n", "X, y = data[:,:-1], data[:,-1]\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "class KNN:\n", " def __init__(self, X_train, y_train, n_neighbors=3, p=2):\n", " \"\"\"\n", " parameter: n_neighbors 临近点个数\n", " parameter: p 距离度量\n", " \"\"\"\n", " self.n = n_neighbors\n", " self.p = p\n", " self.X_train = X_train\n", " self.y_train = y_train\n", "\n", " def predict(self, X):\n", " # 取出n个点\n", " knn_list = []\n", " for i in range(self.n):\n", " dist = np.linalg.norm(X - self.X_train[i], ord=self.p)\n", " knn_list.append((dist, self.y_train[i]))\n", "\n", " for i in range(self.n, len(self.X_train)):\n", " max_index = knn_list.index(max(knn_list, key=lambda x: x[0]))\n", " dist = np.linalg.norm(X - self.X_train[i], ord=self.p)\n", " if knn_list[max_index][0] > dist:\n", " knn_list[max_index] = (dist, self.y_train[i])\n", "\n", " # 统计\n", " knn = [k[-1] for k in knn_list]\n", " count_pairs = Counter(knn)\n", "# max_count = sorted(count_pairs, key=lambda x: x)[-1]\n", " max_count = sorted(count_pairs.items(), key=lambda x: x[1])[-1][0]\n", " return max_count\n", "\n", " def score(self, X_test, y_test):\n", " right_count = 0\n", " n = 10\n", " for X, y in zip(X_test, y_test):\n", " label = self.predict(X)\n", " if label == y:\n", " right_count += 1\n", " return right_count / len(X_test)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "clf = KNN(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1.0" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.score(X_test, y_test)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test Point: 1.0\n" ] } ], "source": [ "test_point = [6.0, 3.0]\n", "print('Test Point: {}'.format(clf.predict(test_point)))" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEHCAYAAACjh0HiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5xVdb3v8dfHcY6AIhTSOcKg44/yYQLJD3+FZqYd/EH+IDVNTlHeOHrsaJmUZMcflKnhTa95s4NmmSJKHMIflVoqqZl4BxhBJRNLZQbvlShQBD2An/vHWgPDds/MXrP3d++11n4/H4957Nlrr73ms9bW/WGt9fl8v+buiIhI/dqh1gGIiEhtKRGIiNQ5JQIRkTqnRCAiUueUCERE6pwSgYhIndsx9B8wswagBWh39wkFr00GZgDt8aIb3f2W7ra32267eXNzc4BIRUTya9GiRX9198HFXgueCIALgOXArl28fre7f7nUjTU3N9PS0lKRwERE6oWZvdLVa0EvDZlZE3AC0O2/8kVEpHZC3yO4Hvg68G4363zazJaa2VwzG1ZsBTObYmYtZtayevXqIIGKiNSrYInAzCYAr7v7om5Wuw9odveRwG+B24qt5O4z3X2su48dPLjoJS4REemlkPcIxgEnmtnxQB9gVzO7w90ndazg7ms6rX8zcE3AeEQkhTZt2kRbWxtvv/12rUPJhT59+tDU1ERjY2PJ7wmWCNx9GjANwMw+DlzUOQnEy3d399fipycS3VQWkTrS1tZG//79aW5uxsxqHU6muTtr1qyhra2Nvfbaq+T3Vb2PwMymm9mJ8dPzzew5M3sGOB+YXO14RKS23n77bQYNGqQkUAFmxqBBgxKfXVWjfBR3XwAsiH+/tNPyrWcNInkzf0k7Mx58gVVrNzJkYF+mjt+Pk0cNrXVYqaQkUDm9OZZVSQQi9Wb+knamzVvGxk1bAGhfu5Fp85YBKBlI6miICZEAZjz4wtYk0GHjpi3MePCFGkUk0jUlApEAVq3dmGi5lG7WLGhuhh12iB5nzSpve2vXruWHP/xhr957/fXXs2HDhvICKHDppZfy29/+ttt1FixYwJNPPlmxv6lEIBLAkIF9Ey2X0syaBVOmwCuvgHv0OGVKeckgbYlg+vTpHHPMMd2uo0QgkgFTx+9H38aG7Zb1bWxg6vj9ahRRPlxyCRR+727YEC3vrYsvvpiXXnqJAw88kKlTpzJjxgwOOuggRo4cyWWXXQbAW2+9xQknnMBHPvIRhg8fzt13380NN9zAqlWrOOqoozjqqKO63P4uu+zC1772NUaPHs3RRx9Nx+gIra2tHHrooYwcOZJTTjmFv//97wBMnjyZuXPnAtHYapdddhmjR49mxIgR/PGPf+Tll1/mRz/6Eddddx0HHnggjz/+eO93PqZEIBLAyaOGctXEEQwd2BcDhg7sy1UTR+hGcZlefTXZ8lJcffXV7LPPPrS2tvLJT36SF198kaeffprW1lYWLVrEY489xgMPPMCQIUN45plnePbZZzn22GM5//zzGTJkCI8++iiPPvpol9t/6623GD16NIsXL+bII4/kiiuuAOBzn/sc11xzDUuXLmXEiBFblxfabbfdWLx4Meeeey7XXnstzc3NnHPOOXz1q1+ltbWVI444ovc7H1PVkEggJ48aqi/+Cttjj+hyULHllfDQQw/x0EMPMWrUKADWr1/Piy++yBFHHMFFF13EN77xDSZMmJDoy3eHHXbgM5/5DACTJk1i4sSJrFu3jrVr13LkkUcC8PnPf57TTjut6PsnTpwIwJgxY5g3b145u9d1jEG2KiISwJVXQr9+2y/r1y9aXgnuzrRp02htbaW1tZUVK1Zw9tln86EPfYhFixYxYsQIpk2bxvTp03v9N5LW+e+0004ANDQ0sHnz5l7/3e4oEYhIZpx1FsycCXvuCWbR48yZ0fLe6t+/P2+++SYA48eP59Zbb2X9+vUAtLe38/rrr7Nq1Sr69evHpEmTuOiii1i8ePF73tuVd999d+s1/zvvvJPDDz+cAQMG8L73vW/r9f3bb79969lB0pgrQZeGRCRTzjqrvC/+QoMGDWLcuHEMHz6c4447js9+9rMcdthhQHSj94477mDFihVMnTqVHXbYgcbGRm666SYApkyZwnHHHcfuu+/e5X2CnXfemeeee44xY8YwYMAA7r77bgBuu+02zjnnHDZs2MDee+/NT37yk5Jj/tSnPsWpp57KPffcww9+8IOy7xOYu5e1gWobO3asa4YykfxYvnw5+++/f63DCGaXXXbZeoZRLcWOqZktcvexxdbXpSERkTqnS0MiIhVwyCGH8M4772y37Pbbb6/62UBvKBGIiFTAwoULax1Cr+nSkIhInVMiEBGpc7o0JHVPE8hIvdMZgdS1jglk2tduxNk2gcz8Je21Dk2q6IEHHmC//fZj33335eqrr651OFWnRCB1TRPIyJYtWzjvvPP49a9/zfPPP8/s2bN5/vnnax1WVenSkNQ1TSCTPZW+lPf000+z7777svfeewNwxhlncM899/DhD3+4UiGnns4IpK5pAplsCXEpr729nWHDhm193tTURHt7fV0aVCKQuqYJZLIlxKW8YsPsJB0hNOt0aUjqWsclBVUNZUOIS3lNTU2sXLly6/O2tjaGDBnS6+1lkRKB1D1NIJMdQwb2pb3Il345l/IOOuggXnzxRf7yl78wdOhQ7rrrLu68885ywswcXRqSmpm/pJ1xVz/CXhf/knFXP6KSTelRiEt5O+64IzfeeCPjx49n//335/TTT+eAAw4oN9RM0RmB1ETHTb+O670dN/0A/etcuhTqUt7xxx/P8ccfX4kQM0mJQGqiu5t+SgTSHV3KqzxdGpKaUP2+SHooEUhNqH5fJD2UCKQmVL8vkh66RyA1ofp9kfRQIpCa0U0/kXQIfmnIzBrMbImZ3V/ktZ3M7G4zW2FmC82sOXQ8Immknora+uIXv8gHPvABhg8fXutQaqIa9wguAJZ38drZwN/dfV/gOuCaKsQjkiqaE6H2Jk+ezAMPPFDrMGomaCIwsybgBOCWLlY5Cbgt/n0ucLTV22hPUvc0J0JCS+fAdcPh8oHR49I5ZW/yYx/7GO9///srEFw2hT4juB74OvBuF68PBVYCuPtmYB0wqHAlM5tiZi1m1rJ69epQsYrUhHoqElg6B+47H9atBDx6vO/8iiSDehYsEZjZBOB1d1/U3WpFlr1nTFh3n+nuY9197ODBgysWo0gaqKcigYenw6aCBLlpY7Rcei3kGcE44EQzexm4C/iEmd1RsE4bMAzAzHYEBgB/CxiTSOqopyKBdW3JlktJgiUCd5/m7k3u3gycATzi7pMKVrsX+Hz8+6nxOu+dJUIkx04eNZSrJo5g6MC+GDB0YF+umjhCpbXFDGhKtlxKUvU+AjObDrS4+73Aj4HbzWwF0ZnAGdWORyQN1FNRoqMvje4JdL481Ng3Wl6GM888kwULFvDXv/6VpqYmrrjiCs4+++wyg82OqiQCd18ALIh/v7TT8reB06oRg9SPb81fxuyFK9niToMZZx4yjO+cPKLWYUkljDw9enx4enQ5aEBTlAQ6lvfS7NmzKxBcdqmzWHLlW/OXccdTr259vsV963Mlg5wYeXrZX/yyPQ06J7kye+HKRMtFRIlAcmZLF7UGXS2XdFCNSOX05lgqEUiuNHTRmN7Vcqm9Pn36sGbNGiWDCnB31qxZQ58+fRK9T/cIJFfOPGTYdvcIOi+XdGpqaqKtrQ2NGlAZffr0oakpWTmtEoHkSscNYVUNZUdjYyN77bVXrcOoa5a107GxY8d6S0tLrcMQEckUM1vk7mOLvaYzAqmos27+A79/adsoIeP2eT+zvnRYDSOqnflL2jUDm2SCbhZLxRQmAYDfv/Q3zrr5DzWKqHY0x4BkiRKBVExhEuhpeZ5pjgHJEiUCkQA0x4BkiRKBSACaY0CyRIlAKmbcPsWn+utqeZ5pjgHJEiUCqZhZXzrsPV/69Vo1pDkGJEvURyAiUgfURyBVE6p2Psl2Vb8vkowSgVRMR+18R9lkR+08UNYXcZLthopBJM90j0AqJlTtfJLtqn5fJDklAqmYULXzSbar+n2R5JQIpGJC1c4n2a7q90WSUyKQiglVO59ku6rfF0lON4ulYjpuxla6YifJdkPFIJJn6iMQEakD6iNIkTTUuCeNIQ0xi0g4SgRVlIYa96QxpCFmEQlLN4urKA017kljSEPMIhKWEkEVpaHGPWkMaYhZRMJSIqiiNNS4J40hDTGLSFhKBFWUhhr3pDGkIWYRCUs3i6soDTXuSWNIQ8wiEpb6CERE6kBN+gjMrA/wGLBT/HfmuvtlBetMBmYA7fGiG939llAxSe98a/4yZi9cyRZ3Gsw485BhfOfkEWWvm5b+hLTEIVIrIS8NvQN8wt3Xm1kj8ISZ/drdnypY7253/3LAOKQM35q/jDueenXr8y3uW58XfsEnWTct/QlpiUOklnq8WWxmO5nZZ83sm2Z2acdPT+/zyPr4aWP8k63rUMLshStLXp5k3bT0J6QlDpFaKqVq6B7gJGAz8Fannx6ZWYOZtQKvA79x94VFVvu0mS01s7lmNqyL7UwxsxYza1m9enUpf1oqZEsX95CKLU+yblr6E9ISh0gtlXJpqMndj+3Nxt19C3CgmQ0EfmFmw9392U6r3AfMdvd3zOwc4DbgE0W2MxOYCdHN4t7EIr3TYFb0i7zBrKx1hwzsS3uRL9tq9yekJQ6RWirljOBJMyt+t69E7r4WWAAcW7B8jbu/Ez+9GRhTzt+RyjvzkKInaUWXJ1k3Lf0JaYlDpJa6PCMws2VE1/R3BL5gZn8mugFsRLcARna3YTMbDGxy97Vm1hc4BrimYJ3d3f21+OmJwPJe74kE0XGTt5RKoCTrpqU/IS1xiNRSl30EZrZnd29091e63bDZSKJLPQ1EZx5z3H26mU0HWtz9XjO7iigBbAb+Bpzr7n/sbrvqIxARSa67PoIeG8rM7HZ3/5eellVL1hNBqJr1JPX7IbedZP+yeCwyZ+kceHg6rGuDAU1w9KUw8vRaRyU1UG5D2QEFG2tA1/J7JVTNepL6/ZDbTrJ/WTwWmbN0Dtx3PmyKb4avWxk9ByUD2U6XN4vNbJqZvQmMNLM34p83iUpB76lahDkSqmY9Sf1+yG0n2b8sHovMeXj6tiTQYdPGaLlIJ10mAne/yt37AzPcfdf4p7+7D3L3aVWMMTdC1awnqd8Pue0k+5fFY5E569qSLZe61d0ZwWgzGw38vOP3zj9VjDE3Qo3tX6xOv7vlobadZP+yeCwyZ0BTsuVSt7rrI/if8c//BhYSNXTdHP9+Q/jQ8idUzXqS+v2Q206yf1k8Fplz9KXQWJBYG/tGy0U66fJmsbsfBWBmdwFT3H1Z/Hw4cFF1wsuXUDXrSer3Q247yf5l8VhkTscNYVUNSQ9KKR9tdfcDe1pWLVkvHxURqYVyy0eXm9ktwB1EncaTUAdwXUlDb4BknPoZUq2URPAF4Fzggvj5Y8BNwSKSVElDb4BknPoZUq/HQefc/W13v87dT4l/rnP3t6sRnNReGnoDJOPUz5B63Q06N8fdT+80+Nx2ehp0TvIhDb0BknHqZ0i97i4NdVwKmlCNQCSdkozXr7H9pagBTdHloGLLJRW66yzuGB76aOAf3P2Vzj/VCU9qLQ29AZJx6mdIvVJuFjcDk+JhqRcBjwOPu3tryMAkHdLQGyAZp36G1Ouxj2DritHkMl8iaiYb6u4NPbwlCPURiIgkV1YfgZl9CxgH7AIsIUoEj1c0whQKVQ+fZLtpGVdfvQEpk/ea/LzvXxJVOhalXBqaSDSD2C+B3wFP5b18NFQ9fJLtpmVcffUGpEzea/Lzvn9JVPFYlNJHMJrohvHTwCeBZWb2REWjSJlQ9fBJtpuWcfXVG5Ayea/Jz/v+JVHFY1HKpaHhwBHAkcBYYCU5vzQUqh4+yXbTMq6+egNSJu81+XnfvySqeCx6PCMArgH6Ew09vb+7H+Xuua77CjVWfpLtpmVc/VDHQnop73MM5H3/kqjisSjl0tAJ7v49d3/S3TdVPIIUClUPn2S7aRlXX70BKZP3mvy8718SVTwWpdwsrjuh6uGTbDct4+qrNyBl8l6Tn/f9S6KKx6LkPoK0UB+BiEhy5c5HIBWk/gSRjLj/Qlj0U/AtYA0wZjJM+H75201hn0R3o4/eR5FRRzu4+4lBIsox9SeIZMT9F0LLj7c99y3bnpeTDFLaJ9HlpSEzO7K7N7r774JE1IMsXxoad/UjRUfnHDqwL7+/+BNV2e4+035VtAS1wYyXrjq+1zGI5MoV74++/AtZA1z2t95v97rhXYzEOgy++mzvt1uCXl0aqtUXfZ6pP0EkI4olge6WlyqlfRI9lo+a2QfNbK6ZPW9mf+74qUZweaP+BJGMsC7G1OxqealS2idRSkPZT4jmKN4MHAX8DLg9ZFB5pf4EkYwYMznZ8lKltE+ilETQ190fJrqf8Iq7Xw70/oJ2HTt51FCumjiCoQP7YkTX8K+aOKIi/Qmlbvc7J49g0qF7bD0DaDBj0qF76EaxSGcTvg9jz952BmAN0fNyq4ZGng6fuiG6J4BFj5+6oeZVQz32EZjZ74nGGpoLPAK0A1e7e01aS7N8s1hEpFbK7SP4CtAPOB/4NtHZwOdL+KN9gMeAneK/M9fdLytYZyeiS01jgDXAZ9z95RJiSixp/X7WxuBP0huQ92MRtE47SW15qDgSbHfWLLjkEnj1VdhjD7jySjjrrMpsO3OS7luej0WBJDOU7Qq4u79Z4voG7Ozu682sEXgCuMDdn+q0zr8BI939HDM7AzjF3T/T3XZ7c0ZQWGcP0TX0ri6fJF2/1gp7AzoUu+ST92PxnjptiK7BVuL0u7C2vEOxSwah4kiw3VmzYMoU2LBh27J+/WDmzC6SQchjV2tJ9y2Hx6K7M4JSqobGmtkyYCnRXATPmNmYnt7nkfXx08b4pzDrnATcFv8+Fzg6TiAVlXRM/ayNwZ9k7oK8H4ugY7gv+mnpy0PFkWC7l1yyfRKA6Pkll5S/7cxJum95PhZFlHKz+Fbg39y92d2bgfOIKol6ZGYNZtYKvA78xt0XFqwylGh+A9x9M7AOGFRkO1PMrMXMWlavXl3Kn95O0vr9rI3Bn6Q3IO/HImiddpLa8lBxJNjuq+89Sex2eVpr3Csi6b7l+VgUUUoieNPdt05E4+5PACVdHnL3Le5+INAEHBxPctNZsX/9v+fby91nuvtYdx87ePDgUv70dpLW72dtDP4kvQF5PxZB67ST1JaHiiPBdvfYo/iqXS1Pa417RSTdtzwfiyJKSQRPm9l/mtnHzexIM/shsMDMRpvZ6FL+iLuvBRYAxxa81AYMAzCzHYEBQBn928Ulrd/P2hj8SXoD8n4sgtZpJ6ktDxVHgu1eeWV0T6Czfv2i5eVuO3OS7luej0URpVQNHRg/Xlaw/KNE/3ov2lNgZoOBTe6+1sz6AscQzXbW2b1EFUh/AE4FHvEA42InHVM/a2PwJ5m7IO/HIugY7h03hEupGgoVR4LtdtwQLrlqKM9zASTdtzwfiyKCzUdgZiOJbgQ3EJ15zHH36WY2HWhx93vjEtPbgVFEZwJnuHu3w1eoj0BEJLmy+gjM7B+B7wJD3P04M/swcJi7F6mj28bdlxJ9wRcuv7TT728Dp/UUg4iIhFPKPYKfAg8CQ+LnfyJqMsu1+UvaGXf1I+x18S8Zd/UjzF/SXuuQJA2WzomGEr58YPS4dE5l1g0laQxp2L+sbTcHSrlHsJu7zzGzaRCVeZpZmWOxpluoCWQk45JMKpKGCUiSxpCG/cvadnOilDOCt8xsEHFZp5kdSlTvn1uZa6KS6kjSZJSGhqSQTVQpaJhLxXZzopQzgguJqnv2iQegG0xU4ZNbmWuikupI0mSUhoakkE1UKWiYS8V2c6LHMwJ3XwwcSVQu+q/AAfGN4NzKXBOVVEeSJqM0NCSFbKJKQcNcKrabE6WMNXQa0ZwEzwEnA3eX2kiWVZlropLqSNJklIaGpJBNVClomEvFdnOilHsE/+Hub5rZ4cB4ot6Am8KGVVuhJpCRjEsyqUgaJiBJGkMa9i9r282JUiamWeLuo8zsKmCZu9/Zsaw6IW5PDWUiIsmVOzFNu5n9J/EQEfFkMqWcSYjUtyST2KRF1mJOy+QxaYmjl0pJBKcTDRZ3bTxu0O7A1LBhiWRc4SQ2vmXb87R+sWYt5rT0BqQljjKUUjW0wd3nufuL8fPX3P2h8KGJZFiSSWzSImsxp6U3IC1xlEGXeERCSDKJTVpkLea09AakJY4yKBGIhJBkEpu0yFrMaekNSEscZVAiEAkhySQ2aZG1mNPSG5CWOMqgRCASwoTvw9izt/1r2hqi52m86dohazGnpTcgLXGUIdjENKGoj0BEJLly+whEwshi7XWomEPV72fxGEvVKRFIbWSx9jpUzKHq97N4jKUmdI9AaiOLtdehYg5Vv5/FYyw1oUQgtZHF2utQMYeq38/iMZaaUCKQ2shi7XWomEPV72fxGEtNKBFIbWSx9jpUzKHq97N4jKUmlAikNrJYex0q5lD1+1k8xlIT6iMQEakD3fUR6IxAZOkcuG44XD4welw6p/rbDRWDSAnURyD1LVStfZLtqt5fakxnBFLfQtXaJ9mu6v2lxpQIpL6FqrVPsl3V+0uNKRFIfQtVa59ku6r3lxpTIpD6FqrWPsl2Ve8vNaZEIPUtVK19ku2q3l9qTH0EIiJ1oCZ9BGY2zMweNbPlZvacmV1QZJ2Pm9k6M2uNf3QunHVZrIdXvX94Om6pFrKPYDPwNXdfbGb9gUVm9ht3f75gvcfdfULAOKRaslgPr3r/8HTcUi/YGYG7v+bui+Pf3wSWA0ND/T1JgSzWw6vePzwdt9Srys1iM2sGRgELi7x8mJk9Y2a/NrMDunj/FDNrMbOW1atXB4xUypLFenjV+4en45Z6wROBme0C/BfwFXd/o+DlxcCe7v4R4AfA/GLbcPeZ7j7W3ccOHjw4bMDSe1msh1e9f3g6bqkXNBGYWSNREpjl7vMKX3f3N9x9ffz7r4BGM9stZEwSUBbr4VXvH56OW+qFrBoy4MfAcncvOrC6mf1TvB5mdnAcz5pQMUlgWayHV71/eDpuqResj8DMDgceB5YB78aLvwnsAeDuPzKzLwPnElUYbQQudPcnu9uu+ghERJLrro8gWPmouz8BWA/r3AjcGCoG6cLSOVHFxrq26Drt0ZfW97/O7r8QFv00mizeGqIpIsudHUwkQzQfQb1RTff27r8QWn687blv2fZcyUDqhMYaqjeq6d7eop8mWy6SQ0oE9UY13dvzLcmWi+SQEkG9UU339qwh2XKRHFIiqDeq6d7emMnJlovkkBJBvVFN9/YmfB/Gnr3tDMAaoue6USx1RPMRiIjUgZr0EdST+UvamfHgC6xau5EhA/sydfx+nDwqRwOt5r3vIO/7lwY6xqmmRFCm+UvamTZvGRs3RVUm7Ws3Mm3eMoB8JIO89x3kff/SQMc49XSPoEwzHnxhaxLosHHTFmY8+EKNIqqwvPcd5H3/0kDHOPWUCMq0au3GRMszJ+99B3nfvzTQMU49JYIyDRnYN9HyzMl730He9y8NdIxTT4mgTFPH70ffxu2bj/o2NjB1/H41iqjC8t53kPf9SwMd49TTzeIyddwQzm3VUMfNvLxWfOR9/9JAxzj11EcgIlIHuusj0KUhkTxbOgeuGw6XD4wel87JxralqnRpSCSvQtbvqzcgV3RGIJJXIev31RuQK0oEInkVsn5fvQG5okQgklch6/fVG5ArSgQieRWyfl+9AbmiRCCSVyHnntC8FrmiPgIRkTqgPgIREemSEoGISJ1TIhARqXNKBCIidU6JQESkzikRiIjUOSUCEZE6p0QgIlLngiUCMxtmZo+a2XIze87MLiiyjpnZDWa2wsyWmtnoUPFIGTTuvEiuhZyPYDPwNXdfbGb9gUVm9ht3f77TOscBH4x/DgFuih8lLTTuvEjuBTsjcPfX3H1x/PubwHKgcCLfk4CfeeQpYKCZ7R4qJukFjTsvkntVuUdgZs3AKGBhwUtDgZWdnrfx3mSBmU0xsxYza1m9enWoMKUYjTsvknvBE4GZ7QL8F/AVd3+j8OUib3nPKHjuPtPdx7r72MGDB4cIU7qicedFci9oIjCzRqIkMMvd5xVZpQ0Y1ul5E7AqZEySkMadF8m9kFVDBvwYWO7u3+9itXuBz8XVQ4cC69z9tVAxSS9o3HmR3AtZNTQO+BdgmZm1xsu+CewB4O4/An4FHA+sADYAXwgYj/TWyNP1xS+SY8ESgbs/QfF7AJ3XceC8UDGIiEjP1FksIlLnlAhEROqcEoGISJ1TIhARqXNKBCIidU6JQESkzikRiIjUOYtK+bPDzFYDr9Q6ji7sBvy11kEEpP3LrjzvG2j/SrGnuxcdrC1ziSDNzKzF3cfWOo5QtH/Zled9A+1fuXRpSESkzikRiIjUOSWCyppZ6wAC0/5lV573DbR/ZdE9AhGROqczAhGROqdEICJS55QIesHMGsxsiZndX+S1yWa22sxa45//UYsYy2FmL5vZsjj+liKvm5ndYGYrzGypmY2uRZy9UcK+fdzM1nX6/DI1J6eZDTSzuWb2RzNbbmaHFbye2c8OStq/zH5+ZrZfp7hbzewNM/tKwTpBPr+QM5Tl2QXAcmDXLl6/292/XMV4QjjK3btqYDkO+GD8cwhwU/yYFd3tG8Dj7j6hatFU1v8CHnD3U83sH4B+Ba9n/bPraf8go5+fu78AHAjRPzaBduAXBasF+fx0RpCQmTUBJwC31DqWGjoJ+JlHngIGmtnutQ6q3pnZrsDHiOYKx93/293XFqyW2c+uxP3Li6OBl9y9cBSFIJ+fEkFy1wNfB97tZp1Px6dtc81sWJXiqiQHHjKzRWY2pcjrQ4GVnZ63xcuyoKd9AzjMzJ4xs1+b2QHVDK5MewOrgZ/Ely5vMbOdC9bJ8mdXyv5Bdj+/zs4AZhdZHuTzUyJIwMwmAK+7+6JuVrsPaHb3kcBvgduqElxljXP30USnoRWHANkAAASASURBVOeZ2ccKXi82F3VW6pB72rfFRGOyfAT4ATC/2gGWYUdgNHCTu48C3gIuLlgny59dKfuX5c8PgPiS14nAz4u9XGRZ2Z+fEkEy44ATzexl4C7gE2Z2R+cV3H2Nu78TP70ZGFPdEMvn7qvix9eJrlEeXLBKG9D5TKcJWFWd6MrT0765+xvuvj7+/VdAo5ntVvVAe6cNaHP3hfHzuURfnIXrZPKzo4T9y/jn1+E4YLG7/78irwX5/JQIEnD3ae7e5O7NRKduj7j7pM7rFFyvO5HopnJmmNnOZta/43fgn4FnC1a7F/hcXMFwKLDO3V+rcqiJlbJvZvZPZmbx7wcT/T+yptqx9oa7/19gpZntFy86Gni+YLVMfnZQ2v5l+fPr5EyKXxaCQJ+fqoYqwMymAy3ufi9wvpmdCGwG/gZMrmVsvfCPwC/i/5d2BO509wfM7BwAd/8R8CvgeGAFsAH4Qo1iTaqUfTsVONfMNgMbgTM8W+33/w7Mii8v/Bn4Qk4+uw497V+mPz8z6wd8EvjXTsuCf34aYkJEpM7p0pCISJ1TIhARqXNKBCIidU6JQESkzikRiIjUOSUCkYTiES6LjTxbdHkF/t7JZvbhTs8XmFluJ2qX6lMiEEm/k4EP97iWSC8pEUjuxB3Ev4wHHnvWzD4TLx9jZr+LB5x7sKMLPP4X9vVm9mS8/sHx8oPjZUvix/26+7tFYrjVzP5P/P6T4uWTzWyemT1gZi+a2fc6vedsM/tTHM/NZnajmX2UqEN9hkVj1O8Tr36amT0dr39EhQ6d1Cl1FkseHQuscvcTAMxsgJk1Eg1CdpK7r46Tw5XAF+P37OzuH40HobsVGA78EfiYu282s2OA7wKfLjGGS4iGIPmimQ0Enjaz38avHQiMAt4BXjCzHwBbgP8gGjvnTeAR4Bl3f9LM7gXud/e58f4A7OjuB5vZ8cBlwDG9OVAioEQg+bQMuNbMriH6An3czIYTfbn/Jv4ibQA6j9EyG8DdHzOzXeMv7/7AbWb2QaIRHhsTxPDPRAMUXhQ/7wPsEf/+sLuvAzCz54E9gd2A37n73+LlPwc+1M3258WPi4DmBHGJvIcSgeSOu//JzMYQjclylZk9RDTS6HPuflhXbyvy/NvAo+5+ipk1AwsShGHAp+NZp7YtNDuE6Eygwxai/w+LDS/cnY5tdLxfpNd0j0Byx8yGABvc/Q7gWqLLLS8Agy2e49bMGm37SUs67iMcTjSi4zpgANF0gZB88MAHgX/vNBLmqB7Wfxo40szeZ2Y7sv0lqDeJzk5EgtC/JCSPRhDdXH0X2ASc6+7/bWanAjeY2QCi//avB56L3/N3M3uSaB7qjvsG3yO6NHQh0TX7JL4db39pnAxeBrqcR9fd283su8BCovHlnwfWxS/fBdxsZucTja4pUlEafVTqnpktAC5y95Yax7GLu6+Pzwh+Adzq7oWTl4tUnC4NiaTH5WbWSjRZzl/I4DSLkk06IxARqXM6IxARqXNKBCIidU6JQESkzikRiIjUOSUCEZE69/8B8WomZYwoj+IAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.scatter(df[:50]['sepal length'], df[:50]['sepal width'], label='0')\n", "plt.scatter(df[50:100]['sepal length'], df[50:100]['sepal width'], label='1')\n", "plt.plot(test_point[0], test_point[1], 'bo', label='test_point')\n", "plt.xlabel('sepal length')\n", "plt.ylabel('sepal width')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### scikit-learn实例" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "from sklearn.neighbors import KNeighborsClassifier" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "KNeighborsClassifier()" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf_sk = KNeighborsClassifier()\n", "clf_sk.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1.0" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf_sk.score(X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### sklearn.neighbors.KNeighborsClassifier\n", "\n", "- n_neighbors: 临近点个数\n", "- p: 距离度量\n", "- algorithm: 近邻算法，可选{'auto', 'ball_tree', 'kd_tree', 'brute'}\n", "- weights: 确定近邻的权重" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### kd树" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**kd**树是一种对k维空间中的实例点进行存储以便对其进行快速检索的树形数据结构。\n", "\n", "**kd**树是二叉树，表示对$k$维空间的一个划分（partition）。构造**kd**树相当于不断地用垂直于坐标轴的超平面将$k$维空间切分，构成一系列的k维超矩形区域。kd树的每个结点对应于一个$k$维超矩形区域。\n", "\n", "构造**kd**树的方法如下：\n", "\n", "构造根结点，使根结点对应于$k$维空间中包含所有实例点的超矩形区域；通过下面的递归方法，不断地对$k$维空间进行切分，生成子结点。在超矩形区域（结点）上选择一个坐标轴和在此坐标轴上的一个切分点，确定一个超平面，这个超平面通过选定的切分点并垂直于选定的坐标轴，将当前超矩形区域切分为左右两个子区域\n", "（子结点）；这时，实例被分到两个子区域。这个过程直到子区域内没有实例时终止（终止时的结点为叶结点）。在此过程中，将实例保存在相应的结点上。\n", "\n", "通常，依次选择坐标轴对空间切分，选择训练实例点在选定坐标轴上的中位数\n", "（median）为切分点，这样得到的**kd**树是平衡的。注意，平衡的**kd**树搜索时的效率未必是最优的。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 构造平衡kd树算法\n", "输入：$k$维空间数据集$T＝\\{x_1，x_2,…,x_N\\}$，\n", "\n", "其中$x_{i}=\\left(x_{i}^{(1)}, x_{i}^{(2)}, \\cdots, x_{i}^{(k)}\\right)^{\\mathrm{T}}$ ，$i＝1,2,…,N$；\n", "\n", "输出：**kd**树。\n", "\n", "（1）开始：构造根结点，根结点对应于包含$T$的$k$维空间的超矩形区域。\n", "\n", "选择$x^{(1)}$为坐标轴，以T中所有实例的$x^{(1)}$坐标的中位数为切分点，将根结点对应的超矩形区域切分为两个子区域。切分由通过切分点并与坐标轴$x^{(1)}$垂直的超平面实现。\n", "\n", "由根结点生成深度为1的左、右子结点：左子结点对应坐标$x^{(1)}$小于切分点的子区域， 右子结点对应于坐标$x^{(1)}$大于切分点的子区域。\n", "\n", "将落在切分超平面上的实例点保存在根结点。\n", "\n", "（2）重复：对深度为$j$的结点，选择$x^{(1)}$为切分的坐标轴，$l＝j(modk)+1$，以该结点的区域中所有实例的$x^{(1)}$坐标的中位数为切分点，将该结点对应的超矩形区域切分为两个子区域。切分由通过切分点并与坐标轴$x^{(1)}$垂直的超平面实现。\n", "\n", "由该结点生成深度为$j+1$的左、右子结点：左子结点对应坐标$x^{(1)}$小于切分点的子区域，右子结点对应坐标$x^{(1)}$大于切分点的子区域。\n", "\n", "将落在切分超平面上的实例点保存在该结点。\n", "\n", "（3）直到两个子区域没有实例存在时停止。从而形成**kd**树的区域划分。" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "# kd-tree每个结点中主要包含的数据结构如下\n", "class KdNode(object):\n", " def __init__(self, dom_elt, split, left, right):\n", " self.dom_elt = dom_elt # k维向量节点(k维空间中的一个样本点)\n", " self.split = split # 整数（进行分割维度的序号）\n", " self.left = left # 该结点分割超平面左子空间构成的kd-tree\n", " self.right = right # 该结点分割超平面右子空间构成的kd-tree\n", "\n", "\n", "class KdTree(object):\n", " def __init__(self, data):\n", " k = len(data[0]) # 数据维度\n", "\n", " def CreateNode(split, data_set): # 按第split维划分数据集exset创建KdNode\n", " if not data_set: # 数据集为空\n", " return None\n", " # key参数的值为一个函数，此函数只有一个参数且返回一个值用来进行比较\n", " # operator模块提供的itemgetter函数用于获取对象的哪些维的数据，参数为需要获取的数据在对象中的序号\n", " #data_set.sort(key=itemgetter(split)) # 按要进行分割的那一维数据排序\n", " data_set.sort(key=lambda x: x[split])\n", " split_pos = len(data_set) // 2 # //为Python中的整数除法\n", " median = data_set[split_pos] # 中位数分割点\n", " split_next = (split + 1) % k # cycle coordinates\n", "\n", " # 递归的创建kd树\n", " return KdNode(\n", " median,\n", " split,\n", " CreateNode(split_next, data_set[:split_pos]), # 创建左子树\n", " CreateNode(split_next, data_set[split_pos + 1:])) # 创建右子树\n", "\n", " self.root = CreateNode(0, data) # 从第0维分量开始构建kd树,返回根节点\n", "\n", "\n", "# KDTree的前序遍历\n", "def preorder(root):\n", " print(root.dom_elt)\n", " if root.left: # 节点不为空\n", " preorder(root.left)\n", " if root.right:\n", " preorder(root.right)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "# 对构建好的kd树进行搜索，寻找与目标点最近的样本点：\n", "from math import sqrt\n", "from collections import namedtuple\n", "\n", "# 定义一个namedtuple,分别存放最近坐标点、最近距离和访问过的节点数\n", "result = namedtuple(\"Result_tuple\",\n", " \"nearest_point nearest_dist nodes_visited\")\n", "\n", "\n", "def find_nearest(tree, point):\n", " k = len(point) # 数据维度\n", "\n", " def travel(kd_node, target, max_dist):\n", " if kd_node is None:\n", " return result([0] * k, float(\"inf\"),\n", " 0) # python中用float(\"inf\")和float(\"-inf\")表示正负无穷\n", "\n", " nodes_visited = 1\n", "\n", " s = kd_node.split # 进行分割的维度\n", " pivot = kd_node.dom_elt # 进行分割的“轴”\n", "\n", " if target[s] <= pivot[s]: # 如果目标点第s维小于分割轴的对应值(目标离左子树更近)\n", " nearer_node = kd_node.left # 下一个访问节点为左子树根节点\n", " further_node = kd_node.right # 同时记录下右子树\n", " else: # 目标离右子树更近\n", " nearer_node = kd_node.right # 下一个访问节点为右子树根节点\n", " further_node = kd_node.left\n", "\n", " temp1 = travel(nearer_node, target, max_dist) # 进行遍历找到包含目标点的区域\n", "\n", " nearest = temp1.nearest_point # 以此叶结点作为“当前最近点”\n", " dist = temp1.nearest_dist # 更新最近距离\n", "\n", " nodes_visited += temp1.nodes_visited\n", "\n", " if dist < max_dist:\n", " max_dist = dist # 最近点将在以目标点为球心，max_dist为半径的超球体内\n", "\n", " temp_dist = abs(pivot[s] - target[s]) # 第s维上目标点与分割超平面的距离\n", " if max_dist < temp_dist: # 判断超球体是否与超平面相交\n", " return result(nearest, dist, nodes_visited) # 不相交则可以直接返回，不用继续判断\n", "\n", " #----------------------------------------------------------------------\n", " # 计算目标点与分割点的欧氏距离\n", " temp_dist = sqrt(sum((p1 - p2)**2 for p1, p2 in zip(pivot, target)))\n", "\n", " if temp_dist < dist: # 如果“更近”\n", " nearest = pivot # 更新最近点\n", " dist = temp_dist # 更新最近距离\n", " max_dist = dist # 更新超球体半径\n", "\n", " # 检查另一个子结点对应的区域是否有更近的点\n", " temp2 = travel(further_node, target, max_dist)\n", "\n", " nodes_visited += temp2.nodes_visited\n", " if temp2.nearest_dist < dist: # 如果另一个子结点内存在更近距离\n", " nearest = temp2.nearest_point # 更新最近点\n", " dist = temp2.nearest_dist # 更新最近距离\n", "\n", " return result(nearest, dist, nodes_visited)\n", "\n", " return travel(tree.root, point, float(\"inf\")) # 从根节点开始递归" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 例3.2" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[7, 2]\n", "[5, 4]\n", "[2, 3]\n", "[4, 7]\n", "[9, 6]\n", "[8, 1]\n" ] } ], "source": [ "data = [[2,3],[5,4],[9,6],[4,7],[8,1],[7,2]]\n", "kd = KdTree(data)\n", "preorder(kd.root)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "from time import clock\n", "from random import random\n", "\n", "# 产生一个k维随机向量，每维分量值在0~1之间\n", "def random_point(k):\n", " return [random() for _ in range(k)]\n", " \n", "# 产生n个k维随机向量 \n", "def random_points(k, n):\n", " return [random_point(k) for _ in range(n)] " ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Result_tuple(nearest_point=[2, 3], nearest_dist=1.8027756377319946, nodes_visited=4)\n" ] } ], "source": [ "ret = find_nearest(kd, [3,4.5])\n", "print (ret)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:2: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "time: 5.170202400000001 s\n", "Result_tuple(nearest_point=[0.09916902877403755, 0.5005978535517558, 0.7997848590100571], nearest_dist=0.0010460533893058112, nodes_visited=38)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:5: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead\n", " \"\"\"\n" ] } ], "source": [ "N = 400000\n", "t0 = clock()\n", "kd2 = KdTree(random_points(3, N)) # 构建包含四十万个3维空间样本点的kd树\n", "ret2 = find_nearest(kd2, [0.1,0.5,0.8]) # 四十万个样本点中寻找离目标最近的点\n", "t1 = clock()\n", "print (\"time: \",t1-t0, \"s\")\n", "print (ret2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第3章 k近邻法-习题\n", "\n", "### 习题3.1\n", "  参照图3.1，在二维空间中给出实例点，画出$k$为1和2时的$k$近邻法构成的空间划分，并对其进行比较，体会$k$值选择与模型复杂度及预测准确率的关系。\n", "\n", "**解答：**" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import numpy as np\n", "from sklearn.neighbors import KNeighborsClassifier\n", "import matplotlib.pyplot as plt\n", "from matplotlib.colors import ListedColormap\n", "\n", "data = np.array([[5, 12, 1], [6, 21, 0], [14, 5, 0], [16, 10, 0], [13, 19, 0],\n", " [13, 32, 1], [17, 27, 1], [18, 24, 1], [20, 20,\n", " 0], [23, 14, 1],\n", " [23, 25, 1], [23, 31, 1], [26, 8, 0], [30, 17, 1],\n", " [30, 26, 1], [34, 8, 0], [34, 19, 1], [37, 28, 1]])\n", "X_train = data[:, 0:2]\n", "y_train = data[:, 2]\n", "\n", "models = (KNeighborsClassifier(n_neighbors=1, n_jobs=-1),\n", " KNeighborsClassifier(n_neighbors=2, n_jobs=-1))\n", "models = (clf.fit(X_train, y_train) for clf in models)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2cAAAE/CAYAAADCCbvWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzde3SbV3rf++8GQRIkQRLUlZIoCbJ1oWTZA9myTc/IM8jMJOPcmqRx0s6k6aBN4/S0OeewzVmrbv4Jpj1t1HNyQdZJmmTSZGHSZJJmOUnTpMmkk8trR7bpsWzDtmxTliy9omgJkigJpCAKJEHs8wdAiDdJJEXyBcDfZy2sAV7cHtAabDzv3vt5jLUWERERERER8ZbP6wBEREREREREyZmIiIiIiEhFUHImIiIiIiJSAZSciYiIiIiIVAAlZyIiIiIiIhVAyZmIiIiIiEgFUHImFccYkzXGPLDAx1pjzO473Bczxhxb3uiWjzHmR4wx/+su90eNMYOLeD3HGPPPlic6ERGRmTQ+l+/X+CwrRsmZ3JMxxjXGfH7a7X9ojLlujPnMPI+Nlr6Qf2XW8WPGmNhC3s9aG7TWnrnvwCuctfZ3rbXfMXX7bgPZajHG/LAx5hVjzKgxxvEyFhERuTuNzyujQsfnnzPGnDLG3DDG9Btj/rGX8cjKUXImi2KM+TLwK8B3W2tfvMPDbgL/2BgTXq24Vpoxxu91DKvkGpAAjnodiIiILJzG55p3E/heoB34MvBLxphPehuSrAQlZ7JgxpjngJ8HvmCtfeUuD80ASeBn7vJa/9QY80HpDN9fGmN2TruvfIbKGLPeGPOnxpgRY8zrxpj/e56lEJ8vnU26boz5FWOMmflW5v8zxgyXzjR9btodW40x/8MYc80Yc9oY8+PT7osbY14wxvyOMWYEiBljnjDGHC/FcskY8wt3+GwvGmN+sHT9SOnzfFfp9ueNManS9fKyDmPMS6Wnv11aNvIPpr3eTxljLhtjLhpj/skd/+ozY9hijHnHGPN/LeTxU6y1f2Wt/QPgwmKeJyIi3tH4vCbG55+x1vZbawvW2teAvwOeWsxrSHVQciYL9b8B/x74nLX2+AIe/x+AHzTG7Jt9hzHm+4GfBv4+sJHiF8zv3eF1foXi2aJOimeKvjzPY74HeBz4BPDDwBem3fckcAbYQHEw+iNjzLrSfb8HDAJbgWeB/zh9cAC+D3gBCAG/C/wS8EvW2jbgQeAP7hDzi0C0dP3Tpff/zLTbc85oWms/Xbr6idKykf9Wut1J8SzZNuDHgF8xxnTc4X0BKJ0RfRH4ZWvtz5WO/WdjTOYOl3fu9noiIlLRND6vsfHZGNNE8e/63t3eT6qTkjNZqG8H+oB3F/Jga20a+DXg381z908AP2ut/cBamwf+IxCZfnYOwBhTB/wg8DPW2lFr7fvA1+Z5vaPW2oy1dgD4WyAy7b7LQMJaO1H6Qj0JfLcxZjtwBPg31tqctTYF/BfgR6c991Vr7X8vnaW6BUwAu40xG6y1WWtt3x0+/ovM/LL/2Wm3P8M8X/53MQH8u1L8fw5kgTkD6jQHAIfi3+yrUwettf/CWhu6w+WRRcQjIiKVRePz2huffw14G/jLRcQrVULJmSzUPwf2Av9l1rKEu/lPwBeMMZ+YdXwnxbXSGWNMhuI+J0Px7NN0GwE/cH7asfPMlZ52fRQITrv9sbXWTrt9juKZuK3ANWvtjVn3TY9h9nv9GMW/QX9pCcf3zBMLwKvAXmPMZooD0W8D240xG4AngJfu8Lz5XC0NkFNmf77ZfgT4mOIZRRERqX0an9fQ+GyM+X+Bg8APz/r7SY1QciYLdRn4HPA08J8X8gRr7VWKxSX+/ay7zgM/MevsUNM86+SvAHmga9qx7YuMe9uswWoHxf1UF4B1xpjWWfd9PP0jzPo8p6y1XwQ2URzYXjDGtMx+Q2vtKPAG8H8CJ6y148ArwL8GPrLWDi3yMyxGHBgCvl46swmAMebXSmvl57toWYSISPXS+LxGxmdjzFeA7wS+w1o7soKxioeUnMmCWWsvAJ8FnjHG/OICn/YLwCeB/dOO/Rrwb40xDwEYY9qNMT80z/tNAn8ExI0xzcaYbmCxpWM3Af+HMaa+9B77gT+31p6n+IX8s8aYgDHmEYpn3n73Ti9kjPlHxpiN1toCxU3VAJN3ePiLwE9ye4mEM+v2fC4BC+ofcxcTwA8BLcB/Ncb4AKy1/7y0Vn6+y0NTTzbG1BljAhTPiPpKf5v6+4xJRERWkMbnNTE+/1vgS8C3l5JrqVFKzmRRSl+anwWeNcb87AIePwL8P8C6acf+mOKZrd83xUpLJyieCZrPT1LccJsG/ivFTcJjiwj5NWAPxbNV/wF4dtqX2heBMMWzdH9McR34N+/yWs8A7xljshQ3H/9Da23uDo99EWjl9hKJ2bfnEwe+VlpO8sP3+Fx3VDoT+PcpDny/NTUALNCPAreAX6V4FvYW8BtLjUVERFaHxueaH5//I8UZxFPTZtZ+eqmxSOUyWq4q1cQY85+ATmvtfFWhRERExAMan0WWh2bOpKIZY7qNMY+YoicoLm34Y6/jEhERWcs0PousjLXSVV2qVyvFpRJbKW56/nngTzyNSERERDQ+i6wALWsUERERERGpAFrWKCIiIiIiUgGUnImIiIiIiFSAVd1z1tzebEOdodV8SxGRNevihxeHrLUbvY5DKt+G5mYbDq3S+Hz1Kql141BXx6aWTavzniIiFeRu4/OqJmehzhDP/fpzq/mWIiJr1le+7SvnvI5BqkM4FOL4c6s4PieThJ8dJLNhlN6e3tV7XxGRCnC38VnLGkVERGR1xWK4L3RB7k59gkVE1iYlZzXIWsvwpWEG3h3g4w8+ZvzWuNchiYiIzMtxHa9DWFW3btzi/HvnGXh3gOy1rNfhiEiFUZ+zGpMfz/PGN9/g4tBF7HqLyRt8r/iI9ETY8dAOr8MTEREpCocJZQdJuX0ARMNRb+NZYdZaTr1+ig/e/QC70YIBjsOu8C4e/vTD+Op0vlxElJzVnPdffp8LExcIfTqEMQaA/GieN7/1Jq0drXRs7fA4QhERESAaxXUg2t1HitpP0C5+eJETH56g/el26hrqALCTlo+Of0RLqoXdj+32OEIRqQQ6TVNDxm+N455xaX+ovZyYAfib/dSF6zhz4oyH0YmIiMwSjeJ8o5MQAa8jWXGn3j1F096mcmIGYOoMrQ+1curEKWzBehidiFQKJWc1JJfNQQB89XP/szaua2T4+rAHUYmIiNxDLoebcb2OYkWNZEZoXNc453h9sJ7x/DgTYxMeRCUilUbJWQ1pbGnE5iyFfGHOfePD4wTbgh5EJSIichexGOEMZIYGSaaSXkezYlpaWxgfnlugKz+ax+/z42/QThMRUXJWUxqbG+na3sXIhyNYe3t5xOT4JBNnJ3jgoQc8jE5ERGR+Tufz9J4Iksmka7Z6456Dexj9cBQ7eXt8tgXLSP8ID3Y/qIIgIgKoIEjNefjph7n5P29yte8qvg0+7ISFS3Dw4EHWb1/vdXgiIiLziqdCJCMZr8NYMV37u7h2+Rpnj52FzWB8BnvJsm3dNvYc3uN1eCJSIZSc1ZiGpgaO/MARrp6/ytWLV6lvqKfzqU5aOlq8Dk1EROTusllS6VRNVm00PsMj0UcIXwlz2b2MtZYND22gY2vHjCJeIrK2aQ69BvnqfGwMb6T7qW4efOxBJWYVqDBZoDA5d2+giMiaFYsROx2EkRESfQmvo1kRxhjaN7Wz54k97H1yL+u2rVNiVmFswWp8Fk9p5kxkFY1cGeHk8ZNcOH8BgK07ttJ9uJvWDa0eRyYi4r14pJd4MkkoVtx7VoszaFKZxm+N8+HrH+J+6JKfzNOxvoPuR7vZ/MBmr0OTNUYzZyKrZOTKCC/92Uukm9K0fraV1s+2kg6keelPX+LG1RtehyciUjHWQt8zqRz58Twv/4+XOX3jNE1PN9H+He2M7hjllRdfYbB/0OvwZI1RciaySk6+cRK7y9IabsXn9+Hz+2gNt1IIF/jwjQ+9Dk9EpHJks6Tcvpqt3CiV5eOTH5OpzxB6KERdYx3GGJo2NdHyWAvvvvauljnKqlJyJrIKCpMFLgxcoKVr7v6/lu0tDJ4dnNH+QERkzYrFcF/oIpTNK0GTVTF4dpBA19zZ2oa2Bsbrxxm5MuJBVLJWKTkTWQXa8C0isghTCZqWN4rXdN5UVpmSM5FVYHyGbTu2cfP8zTn3ZQeydO3qUgInIjJbLud1BLIGbN+1ndz5uf/WxofHaZxspG1jmwdRyVql5Exklex7fB8+18fImREK+QKFfIGRMyPUnatj3+F9XocnIlJZwmHCGUidPkYylfQ6GqlhW/dtpWOyg8y7GfK38tiCZTQ9ys03b/LIk4/gq9PPZVk9+tcmskpa17fyme/7DNvy28j+TZbs32TpmuziM9/3GYLrgl6HJyJSWaJRnP4eIkN+MkODStBkxfgb/Hzyez/JvnX7GHt1jOFvDtN2sY2nP/c0W/dt9To8WWPU50xkFQXXBXnsOx7jUfsooL1oIiJ3FY3iECWeSpA4WEzQYpGY11FJDWpoamD/p/bT/clusMXtCCJe0MyZiAeMMUrMREQWKB7ppfdEUDNosuKMMUrMxFP3TM6MMQFjzLeMMW8bY94zxnyldHyXMeY1Y8wpY8x/M8Y0rHy4IiIiAmtvfI5nIkSyWgIuIrVtITNnY8BnrbWfACLAM8aYHuA/Ab9ord0DXAd+bOXCFBERkVnW3vicy5HJZbyOQkRkxdwzObNF2dLN+tLFAp8FXigd/xrw/SsSoYiIiMyx5sbnaBTnG50wMkKiL+F1NCIiK2JBe86MMXXGmBRwGfgm8BGQsdbmSw8ZBLbd4bnPGWOOG2OOX75ymWQqqfXiIiIiy2C5xucro6OrE/D9isXIfH0HoUxOvyVEpCYtKDmz1k5aayNAF/AEsH++h93huV+11h621h5uqoewm9GGXhERkWWwXOPzxubmlQxzeYXDhHMBr6MQEVkRi6rWaK3NAA7QA4SMMVOl+LuAC/d6/r7RJpxUZEbFJc2kiYiI3J/7HZ+rTi5HJpPGcR2vIxERWVYLqda40RgTKl1vAj4PfAD8LfBs6WFfBv7knu/W2grRaLkkbrg/Tbg/TebygNaPi4iILMKyjs/VpNScOpTNk3L7lKCJSE1ZyMzZFuBvjTHvAK8D37TW/hnwb4B/bYw5DawHfnMxbxyP9OJ0Po/T+Ty977dpg6+IiMjirMj4XBWiUdzjR4hkArgZ1+toRESWjf9eD7DWvgMcmuf4GYrr2+9bPNJLPJkk9KXiDFpvT+9yvKyIiEjNWo3xWUREVtc9k7NVE4uRSSYJPztIwjkKgeJm30hnhGg46m1sIiIiUnlyufIe9lgk5nU0IiL3bVEFQVZcLIb7Qhe9qQC9fRAaymo9uYiIiMwVjeJ0Pk9kyK8q0CJSMypn5mxKLEa8dDXuOIQPHyPl9gFoBk1ERERmcPp7iEZSuBu8jkRE5P5V1szZbFMbftOQOn1MM2giIiIyVy7ndQQiIsuispMzKJfMjQz5SZ0+RsI5SsI5quULIiIiAtEo0XRAbXlEpCZUfnIG5XXlmd/pIpPspDdV/BJWgiYiIiLxSG+5LY9+G4hINauO5GxKLFbck1b6EtZZMhEREYFigha51uB1GCIi96W6krNppp8lS/QlcFxHe9JERETWuEwu43UIIiJLVrXJGRQTtMzXdxAayuL29xX3pGkmTUREZE1yvtFJaChb7JcqIlKFKq+U/mLFYriOU7zuuuUm1qFQJ+FQGFAJfhGpHSNXRjjff54bIzdo72hnx/4dtHS0eB2WSGWIxXCTyfJvgUi4R78BRGRV5LI5BvsHuXr5KoGmANv3bqdjawfGmEW9TvUnZwDRaPmqm0wSfSYNmQxuoI9MoPQQfTmLSJUbeG+AN7/1Jr4uH/71fi5lLnHqj0/xZPRJNj+w2evwRCpD6aSt+qSKyGoZvjTMsT8/Rn5Tnvr19eRH85z9q7N07+1mX8++RSVotZGcTReL4UzNpAHR7j5S+WPF6/pyFpEqdWvkFm+99hbBniD+5tJXdyeMbxnnded1nul6Bn9D7X2liyxJNIrrlH4DBFIa/0VkxdiC5fhfH8fsN7R3tpePF7YX6H+ln807N9OxtWPBr1fVe87uKBotX6b3SFN5XRGpVhc/uojdZG8nZiUN7Q3k2/Ncca94FJlIhSr1PxMRWUnDl4e5YW/Q3Nk847iv3odvu4+BkwOLer3aTM6mm5agZYYGSaaS5YuISLUYuzWGr+kOX9kBmBibWN2ARKpFNqsxX0RWzERuAhOYf9miv9lP7lZuUa9X+8kZlJtY954IEu5PE+5Pq4m1iFSVjk0dFK4V5hy31sI1aF3f6kFUIpUtHumdcXJWRGS5BdcHsRlLIT93jB6/Ms76zesX9XprIzkriUd6cTqfLyZqamItIlVkY3gjrROt3Dhzo5iQUVznPnJyhA0tGwhtCXkcoUhlmjo5q/5nIrISmlqbCIfDDJ8YLido1lpG06M0XG2ga1/Xol5vTSVn081uYi0iUsnq/HU89d1PEcqEGH5pmOE3hhl+cZjN+c08/szjiy7VKyIiIsvj4U8/zK7WXYy8OMLIGyOMvDxC49lGPvVdnyIQXNze1zVd2ise6SWeTBL60swZtN6eXg+jEhGZX3N7M0d+4AjZq1ly2RzN7c3qcSayUKWTsRrjRWS51dXXEflchH039pG9lqW+sZ72ze1LOnG6ZmfOymIxMl/fQW8f9PZBaChLwjmK4zpeR1bzCpMFMukM1z6+Rn4873U4IlXBGEPrhlY2hjcqMRNZIK2WWRxrLTeGbnB18CpjN8e8DkekajS1NrFx50ZCnaElr2hZ0zNnZbEY8dLVeDJJ+NlBNa5cYemP0qReTjFWNwZ14L/l58ChA4Q/EdbyLBERWXbxSC+kEiR6vI6ksmWvZXnjb97g+s3rmCYDNyD8QJiDRw5SV1/ndXgiNU/J2WyxGK7jED58TAnaCrl+4Tp9L/XRdKiJ9lCxWV9+NM9bb7yFv8HP9gPbPY5QRERk7Rm/Nc7Lf/YyE7smaO8qLskq5AucefcMhRcLHPr8Ia9DFKl5WtY4n2gU9/gRImlInT5GwjlKwjmqMrzL5FTqFP4H/TSGGsvH/M1+Wh5u4YM3P8AWrIfRiYhIrYqnQtq+cBcXPrzArbZbBLcHy6tYfH4foUdCnDt3jtHhUY8jFKl9Ss7upNS8OvM7XWSSnUTSlPukOK6jL/X7cCV9habNTXOON4YaGR0bZTw37kFUIiJS82Ix3Be6CGXzpNIpr6OpOEOXhqjfWD/nuKkzmA7Djas3PIhKZG3Rssa7iUbLVx0gnkqQ3D2Im8mQIUcqnVLVpyVoDDQymZukrnHm2vXCRAFfwUedX2vaRURkhcRixFIJEhu8DqTyTI3P8xoDf4N+NoqsNM2cLUI80ov7QhduMkRvKqCqT0u0a98ubp6+WW6kO+XG2Rt0hbv05S8iIisvl9MqmFm69nRRGCxQmCjMOJ4byhGYCNCxpcOjyETWDiVnixWLFas7qizvku18eCeb6zaT+VaG0Yuj3Lp0i0wqQ3AoyIGnDngdnoiI1Lh4JlLeV6795LeFOkPs27OP4VeHuXHuBrmhHMP9w0y8O8Hjn3scX51+NoqsNE1R3IfZTaxDgRAAsUjM28AqnL/Bz5Pf8yTpj9KcP32eQqHA1ge2sm3vNuoDc9e6i4iILKtoFMeBaHcfKX/a62gqhjGG7qe62bR9E+dOnuPWx7cIbwqz46kdNLc3ex2eyJqg5Ox+xWJkkkmiz6QhkMEN5Eg4R4mEe1SC/y7q/HVs27eNbfu2eR2KiIisRdEoTtIlFFNyNp0xhvXb17N++3qvQxFZk5ScLYdYDMdxyjfVI01kaQqTBT7u/xj3Q5fxsXE2b93Mrod30dLR4nVoIlKr8nmSqaRWvYjchbWWoYEhzpw4Q/ZGlrZQGw8cfID1XUril5sWDy+XaLR8md4jTZuNRRamMFng9b94nddPvE52W5bJA5N8lPuIv/3vf8v1i9e9Dk9EalEsRu+JIJnLA9o/LnIXp4+f5tiLxxhqH2LywCSXmi/x0l+9xNm3z3odWs1RcrYSSj3SIkP+8mbjqYuIzO/iqYt8fPNjOp7ooGlTEw1tDbTvbce330fqpdSc6p4iIstBBb5E7u5m5ibvv/s+bT1ttHS10NDWQHBHkNYnW3n3+LuM3RzzOsSaouRspUxL0ML9acL96XITaxGZ69ypcwR2BjDGzDjetLmJ4dwwN6/f9CgyEal1StBE7uzS2UvYzZa6hpl9aP1NfgobClw5d8WjyGqTkrOVFI3idD5fvkwtnVCCJjLXxPjEnC9+KG5ONw2GyYk7NEYVEVkG8Ugvma/vUP8zkVnyE3lMvZn3PltvNT4vMyVnq2jqzJzWtovM1dnVSe5ibs7x/Ggef86voiAisipCBLwOQaSirOtch71i52wvsNZihgztm9s9iqw2KTlbZVo6ITK/nQ/tpOFKA9mBLLZQHAAmbk4w8tYI3Z/oxt+g4rIisgqyWVJuH4m+BIm+hGbRZM1b37WeDc0bGH5vmEK+AEBhokDmnQxb1m9RcrbMlJx5oLx0QgmaSFkgGODI9x4hdDXE8IvDDB8bZuL1CSIPRXjg0ANehycia0EshvtCF72pAL19EBoqJmpK0GQtMz7Dk9/5JDubdpJ9Mcvwy8NkX8ryYPuDPPbtj83ZKy73R6eivRKL0ZtKkOjxOhCRytG6vpUjP3CEWyO3yE/kaW5vps4/dx+aiMiKicWIl67GHYdodx+p/O3+pVNCoU71RpM1oz5Qz6HPHeKhWw8xdnOMQDBAfaDe67BqkpIzr+Xm7rGR2jZ2c4xrH18DYN22dTS2NHocUeVpamvyOgQRkWJhLwdw3ZmHn0mTyg+qeXWNmZyY5OrgVfLjedo2thFcF/Q6pIrT0NRAQ1OD12HUtHsmZ8aY7cBvA51AAfiqtfaXjDFx4MeBqfqZP22t/fOVCrQWxVMhkrsHSThHiYR7iIajXockK8hay6nXT9H/bj92ncVi8f2dj/2P7Gf34d1aFiAii6LxeZVEo3MOOUA8lSBxYEAJWo24dOYSx188zkRwAhqAV6BrSxeRz0a051lW1UL2nOWBn7LW7gd6gH9pjDlQuu8XrbWR0kVf/IsVi+EeP0Iom5+zXEJqz8cffMyJD0/Q8nQL7YfaCR0K0XKkhXdPvsuFkxe8Dk9Eqo/GZw+pAnPtyF7L8tqLr1H/aD2hx0OEPhGi/TPtDOYGOXHshNfhyRpzz+TMWnvRWvtm6foN4ANg20oHtmZEo7gvdHkdhawway0n3z5Jy4GWGb286hrraN7fzMm3T3oYnYhUI43P3lMF5tpw7r1z2C5LQ/vt5XrGZ2g72Ma5s+cYuznmYXSy1iyqWqMxJgwcAl4rHfpJY8w7xpjfMsZ0LHNsa0s+j+M65YvUFluw3Bi5QUPH3HXajesaGbk+Ui4fLyKyWBqfvaMKzNXv2tA1GtfN3f/t8/sgCKPDox5EJWvVgpMzY0wQ+EOg11o7Avwq8CAQAS4CP3+H5z1njDlujDl+ZVT/uOcVDhMZ8uP29+H295E6fYxkKul1VLKMjM/Q0NhAfjQ/5778zTyBpgDGpz1nIrJ4Gp8rQCxG5us7CA1lSThHvY5GFqkl2MJEdmLOcVuw2FFLQ7MKYMjqWVByZoypp/jF/7vW2j8CsNZestZOWmsLwG8AT8z3XGvtV621h621hzc2Ny9X3LUlGsXpfB63rwe3r4feE0Eyl4ubjKcuUt2MMew+sJsbJ29g7e0ZMluwZE9mefDAgx5GJyLVSuNzBSn1SAtl8ySco1oFU0V2du8k7+YpTBRmHM+6WTZ2bKQl1OJRZLIWLaRaowF+E/jAWvsL045vsdZeLN38AUA7Ju9XqSJUnCikEjjX0hAIkApmSfQl6O3p9TQ8uT8PHnqQa5eucfHVi9R1FvedFS4W2BraqibLIrJoGp8rUCyG6ziED9/ui6ZKzJVvXdc6Htr/EO8fex+z1eAL+JgcmiSYC3Loew55HZ6sMQupDfop4EeBd40xqdKxnwa+aIyJABZwgZ9YkQjXqHikl2JzFYiHUiQOFNeyhwIhAJXtrUJ19XU8+d1PcnXwKhfdi/iMj81Pb2Z913otaRSRpdD4XImiUVyHcvNqUIJW6Ywx7H1iL527Orlw+gLjY+OsO7COzgc771pGX20UZCWY6UusVtrhrVvt8eeeW7X3qynJJNFn0gC4wTyZoJ/e6PMeByUilewr3/aVN6y1h72OQyqfxucV4DjFBG1DntCGLv2IrzGJvgRks+D3q1etLNrdxmd11asWsRjO1PVkkvCzxebVoVAnAOFQuCa/GAqTBQZODHD6vdOMZkdpX9fO3k/spXN3p5o2i4hI5YpGcZzSDBqDNTnLcvnsZT5Mfci1K9cINAfYfWA3Ox/ZSZ2/7t5PrjKO6+BmXAAymTShbB73hS7Czw5qCassq0WV0pcKUWpeHUlDuD8Ng8UvhlrbfGyt5a2/eos3+99k8sAkwc8GGd0xyquvvsqZN894HZ6IiMjdlQp+RYb8ZIYGa6rA17l3z/HySy8zsmWE4GeD8DCkzqQ4/pfHKUwW7v0CVcRxnWICNjhIuD9NJE2xR23p91gom6/J32HiDc2cVatoFIdo8frU0on8MVLp4raDUCBU9Wforl+4zvnL5+n4VEd5T1bTpiYa2hs48Xcn6NrfRWPz3L4kIiIilcTpfJ54KkHiwEBNzKBNjE3wzrfeofWpVvzNxZ+SDe0NdDzawYW+C1w9f5WN4Y0eR3l/kqkkmVymeCObJTLkx+k/Ui7eRqz0wKk9hpFUeWZN5H5o5qwWRKM4/cUS/L19EHFzNXGGLu2mMVvMnGIZdY11sB6unr/qUWQiIiKLE4/00vt+G5nLA1XfrPr6hesU2grlxGyK8Rn8W/1cOHvBo8iWRzKVJDM0SMTN0dsHvSeCOP09txMzkRWkmbNaEY0WS/CX1MIZOmvtHasYWrN6hWxERESWQzzSC6lEuQLzlGprlWOtvePpfR+L/mcAACAASURBVOMzc/qFVYN5Z8o6n4fOBb5A7vaJ8Wr93SWVQTNnNaoWztBt2r6JyfQksyuKFiYKmKuGjq0dHkUmIiKyNPFIL5mv7yjOyPRBaChLwjnqdViL0rGlA5MxTI5NzjhurSV/Mc+WnVs8imxpEn0JMpcHyv9Nek8Ei4nZQtXw3kJZfZo5q2Gzz9BV25m5Dds30BnsJP12mtZ9rfib/IyPjJN9L8u+7n00tTZ5HaKIiMjixWLES1fj0yowT6n00uwNTQ10P9zNieMnaHmohcZQI5O5SUZOjbDBv4FNuzbNeHyiLwG53D1fdzU/94yYxsfpfb+t+LvpPtTa3kLxhpKzGlfNCZrxGR5/5nFOHT/FmVfPkC1kaWps4tFHHmXnwzu9Dk9EROT+xWK4jgOuC1A1pdn3PL6HQEuA/rf6yeQy+I2fvfv2svfxvfjqiguzpqocTpWdv5vV/NyJvgSMjJD5+o7bB2OxZXnt8u8ulKDJ0ig5WwPikd7bZ+aqLEHzN/jZ/8n97HtyH5P5Sfz1/jvuQxMREalK0wpNuNMrMJeSlVCos+J+5Btj2PHQDrYf2E5+PE+dvw5fne922fmSUDaPe/wIxKJ3fT3XcQgfPrYiCdq8Mb2wY9kSstnikV6c9FHcTfd+rMhsSs7WilgMd9rSiUpfMjGbr85XPhMnIiJSs0rNq6dm0qLPpEnli/uYwqFw8VgFjd/GGF6++HL5dur0sWIxjW+UKmmEwwurcjhVkr6UmM64awmfd3rPsXln71YoMZuuXGBEZBGUnK0l0xK0algyISIisiZNS2YcihWYk7sHcTMZMuRwM27FzKQlU0kymTQhApDLEcn4i2Xn7zFTNq9SYhrt7sPN90EgsKTPOzcmij3KlhLTEjnf6KzKFUviPTO7Et5KOrx1qz3+3HOr9n5yB6WlA5mgv+pm0ERk4b7ybV95w1p72Os4pPJpfK4CySQA8UiGxIERaGsj0hkBVu9E6/TZKAA34xarHL7fRjwVKh5crhmpWZ83tGnHXWcOp2JzM26xR9lSZu+WW+mEeGZDUAmazHC38VkzZ2vRPEsHlKCJiIhUsFLSEwdIJUjuzuJm+sj486TSqRX/8Z/oS0A2Syg/7adjPk/v6VKVw8gyv+Hszzs+iJtOz/t5p5pGT8UWyZR6lMWWOabFisWIpRIkNngch1QVJWdr1bSlAymOVdQSiUriuI4SVxERqSjxSC9xxynecN3yfvJQqDhTFA6Fl2XsmurXlcllblc3DIdnPmgVZqSmf954KFWuQB0KhMqx9b7fRjwTWbWYFiWX0+8JWTAlZ2tZNIpDlGj6KCnU1X66cmWn/OqckRQREVmU6RUek0miz6Qhk8EN5Ehl0sWHLDEZKJfAz0E4V9y35Xxj5aobLkjp88aJEp/6vIEM5HJE0/ffo2ylxDMRnHQfKbTXXxZGyZmoaWKJ4zq4GReguJE4myd2OjjjDB2wZv8+IiJSoWIxnKmZNCiXpJ8a0xYykzbfGOi+0HV7pmwVi2nc06zPW3EzZdNNrVSiuJVEK5XkXpScCTCzaeJarCw0tV49MnT7/xJO/5HiF34qgdOZA9Kk1o2vyb+PiIhUuBm90orbFkgXZ9BSGwbvmhQ4rlMugV8+tsrVDRetkhOy2WZsJdFKJbk7JWdSVk7QSjNFtZ6ATK2lB25Xdurvuf2FXyryNGOpxLReccu9tl9ERGRZlLYtlG9O274wn7uNgbJMpm8lCar/mdyZuvrKDPFIb3HD70gxQatVyVSSzOUBwv1pwv1pek8Ei5Wd7nUmLhbDfaGLSBrC/WkYLPaMm11eWEREpFI4nc/TeyJYHvNmXxY8Bsp9i6YDXocgFU4zZzJXLEZm2gxRb/R5ryNaFjOSzanKTlOzYos5QxiL4Uxddxy1JBARkYp314IZmiVbXSNrY4WSLI1mzmR+pRmiUDZPwjla9TNDib5EMSHro3h5f5kqO0WjOP09RIb8pE4fu+OSEREREZF4pJfe99tqfoWSLJ1mzuTOYjHc0gxayq2+ErCzZ8oyX1+hMsCzNvpOvW8oENKGXxEREZmhvMe/x+tIpBIpOZO7i8Vwp5buVUmPjnJ/llIpfIB4aoX7s5Q2+sZTtxPCtd6aQEREREQWR8mZ3FsV9OgoN40uCWXzuMeP3N7cHFmdOGYslZxqTeAcLd4OBLS+XERERIinQiR319beflke2nMmCzNtb1Vm6M7leL2QTCWL/VnSkEl2kkl2zkzMPDJV+TKT7KQ3FdD6chERESmqsb39snw0cyYLN71Hh8dNFJOpJJlMmhAByOWIZEr9WSqtYWZpKWUciE9VwFSFJhEREanyvf2yMjRzJos21S8lc3lgVWfQHNfBcZ1yj7LeVAA3GcL9/c7q6M8ydZZsKFs+SzZ1ERERkTUoFsM9foRImhnbM2Tt0syZLEm50hADqzITlOhLQC5XnCnLZuk9XSqFv0p7yZbNtLNkrpsCIEMO0NkyERGRNSkaxUm6hGJpryORClCVydnI2BhvfvwxV4aHaWtu5lBXF5taWrwOa80pJ2gHVqaZ4tSMUiqdut00OhUCQitbeXGllSpg4roARJ9Jl5tYT1GiJiLVaHxyknfSadzLl2loaOChLVt4oKMDY4zXoYmIVIWqS84+unaNP3z1VQ5OTNDt93NlcpKv9ffzbY89xuGuLq/DW3PikV7iySShLy3vDFrCOUqoOKFEKJ/HfaFUCr/aZsruZNoSTKfUqsDNF5czZPz5iqyIKSJyN8O5HMlXXmHzyAj7/X5uFQp849QptjzwAN//8MP4lKCJiNxTVSVnE5OT/NG3vsU/qKtj57SZskMTE/zGG2/wwPr1rGtq8jDCNSoWIzNV7GKJJWEd18HNuADFQh/ZPO4LXRAOl94jumzhVpxSoRUcp3iz1MxaPdJEpJr82bvv8uiNGzwdCpWPPVYo8NunT/P2xo0c2rLFw+hEKpjjEH1GSxqlqKoKgpy8epUtuRw7A4EZxzvq64lYS+rCBY8ik/spCTvVoyzsZgi7GSJpiqXwY7HiDFOlF/pYLqXP6lXBFRGRpboxNsbghQv0tLbOOF7v8/F0IMBbZ896FJlIhSutnkltyBMKdXodjVSAqpo5y46Ps/4O962vq+PjW7dWNR6ZpbSXKnz42D1Lwk5POjJDg0SG/Dj909YsrpWE7A5mF1wJBYpnojWTJiKV6ObEBG0Uk7HZ1tfXkx0dXf2gRKqB65KKQiR8RPvNBaiy5GxzSwtvAdbaOZuL3clJutrbvQlMbotGcZ3S0rxSkYvZXzbJVLKckBWVepSt8YRstqkEzenMAWlS68bVI01EKlJHIMBwXR3ZyUmCdXUz7nNzOTZv2+ZRZCJVIBBQYiZlVZWchUMhzPr1HLt2jSNtbeUE7YObNznb1MR3az17ZYhGcZypvVPHynvJpmQuDxQrL0amJRmayZ/XjL/RtH19U0sfwqGwvtBFxHONfj+R3bv50w8+4Nn29vIM2tDEBC8WCvzgAw94HKGISHWoquTMGMMXH3+cP3jjDVJXrrDdGK5Yy83WVr70xBME/FX1cWpbKUGLh1I4nbc3uUbTAWBWYiYLU+qRFo9kcDrTuME8qUzxb6sETUS89vm9e/nT8XF+8cwZdhvDLWDQ7+cLTzzBzmlFQkRE5M6qLptpDwT4Z5/8JIMjIwyNjnKwsZEHOjpUorcSRaPEic48phmy+xOLEZ+6vsD9fSIiq8Hv8/EDjzzCtT17GBgept7n49l162jUiVOR+ZVWxEDQ60ikglTlN6Yxhu3t7WzXHjNZooK1XMpmsRT3MtbNs4m94i1gf5+IyGpb19SktjZyX66OjnIrn2dDc3Ptrooq9YilrU17yWWGe/6LN8ZsB36b4pxHAfiqtfaXjDHrgP8GhAEX+GFr7fWVC1Vkebx/+TL/6+238d+8iQ+41dTEZx9+mENbt3od2uLN2t+XSqcACAVCquwoUuM0PkutuZTN8qdvv83wlSu0GcNVn49De/fy+T17qvMk6l3EIxklZjKvhfxLzwM/Za3dD/QA/9IYcwB4Hvhra+0e4K9Lt0Uq2pnr1/mLV17hBycn+clQiH8RCvEjxvDia6/x3qVLXoe3NNN6o/X2QW8f6pEmsjZofJaakR0f57++/DKPXrvGv2pv58fb2/nJ5mauvP8+3+jv9zo8kVVzz+TMWnvRWvtm6foN4ANgG/B9wNdKD/sa8P0rFaTIcnnp5Em+UF/P9mmNzDsbGvh7gQAv9fdjrfUwuvsTj/SWL73vt5G5XOyRJiK1SeOz1JLjg4N037rFo62t5ToCwbo6nm1v593Tp8mOj3sc4fKKp0KEhrIknKNehyIVZlFzxMaYMHAIeA3YbK29CMUBAti03MGJLLfzly+zb569ELsCAa5dv8745KQHUS2/qQSNkRHNoImsARqfpdqdv3yZfQ0Nc44HfD62W8vHIyMeRLWCYjHcF7oIZfMknKM4ruN1RFIhFpycGWOCwB8CvdbaBf8/xBjznDHmuDHm+JXR0aXEKLJsGvx+RguFOcfHrAWfr3bWtDtOsXm1z0c4FPY6GhFZQRqfpRY0NDTMOz4D3AQaZjU3rwnTErSU26cETYAFJmfGmHqKX/y/a639o9LhS8aYLaX7twCX53uutfar1trD1trDG5ublyNmkSU7uGsXr2Szc46/duMG+3bswF8jyVm0u49UJ0R2H1EFR5EapvFZasXBri5em5hgctb2gnO5HNmmptrtlReL4R4/QiQNqdPHcFynfJG16Z6/RI0xBvhN4ANr7S9Mu+t/AF8uXf8y8CfLH57I8oo++CCn29v54+vXOZfLcT6X439mMrzZ0sLnu7u9Dm9ZhUKdSsxEapjGZ6kl+zduJBQO87VMhpOjo6THxzk2PMwfTEzwvY89Vtv9bKNRnP4eIkN+3P4+XDdFyu3TtoQ1aiHNIz4F/CjwrjEmVTr208BR4A+MMT8GDAA/tDIhiiyfloYG/tmRI7w+OMhfnj+PtZY9u3fz4zt2EJxnrbuISAXT+Cw1w2cMz0YivLNlC6+ePcutsTG2bt3KPw6H2RxcA02aS61xcN3izWfSpEjjuI5OtK4x90zOrLXHgDudrvjc8oYjsvKa6uv59K5dfHrXLq9DERFZMo3PUmt8xhDp7CTS2el1KN6IRstXnWSScCzjXSzimRptu762DedyvDYwwMClSzTU13Nw504+sXlz7RS7EPFYJp3Bfc9lZGSEtvY2wgfChDprdD+EiCyb8clJ3rxwgQ/On6dQKLB72zYe7+qiub7e69CkUjgOUGxSnSHnbSxVaHR4lIH3B7iSvkJDYwPhfWE27dqE8VXPslglZzXm4o0b/M6xY3xibIxvDwS4VSjQd/Ei73V18cXHHquZghciXjn37jneOv4Wvp0+Gnc2kslkcP/C5dEnHmXHQzu8Dk9EKlQunyf56qt0DA3xdCBAHfDu22/z6x99xD85coTQtP6bsgY5DtHuPtye4s2MP08o1KUljYuQSWd4+S9eJt+Zp3FHIyO5ES68doGwGybybZGqSdCUnNWYP3v7bb5jcpJPTKtqtLepid8ZHOStLVt4vKvLw+hkNcRTCVIH82geZ/nlsjne/tbbBD8ZxN9U/PpsXNdIvjNP6tUUm8ObaWxp9DhKEalEf3f2LFuGhvh7oRCmVNxiV1MTLw4P883+fn4oEvE4Qll1jjNzj9mGPJHdR8p3KzFbOGstbzpvYvYb2jvby8ebtzTjvuqy7dw2Nu2qjpaPSs5qyNXRUUaGhni4vX3GcZ8xfDIQ4EXXVXJW4+KpBIkDIxBsIxaJeR1Ozbl05hKFjYVyYjbF3+ynsKHA5bOX2X5wu0fRiUgle+fMGb4cDJYTsylPtrbyCwMDTDz8MPW12MtL5pdMEn52EKLFYicZIBJW+5ulujF0g5GJEdo3z/wNbHyG+h31DJwcUHImqy+XzxM0Zt5ys8G6OsYmJjyISlaN4+B056Ctjd6eXq+jqUkT4xNwh4kx22CL94uIzCM3Pk5rS8uc443G4CsUmCgUlJzVumSyfDX87CCZoJ9I+PaMqRKzpcuP5zENZs7JD4C6QB1j18Y8iGpplJzVkI0tLWT8fobzedr9M//Tnrp1i65t2zyKTFZNIEAooAWNKyW0KQQfFpdPTB8ArLWYa4b2h9vv8mwRWcu2b9rEh1eu8PCssvCDY2M0t7bS5NdPspqWTBL60gCU2/b46Y0+72lItSS4LojJGibHJ6lrmHmSY+zyGJu2VMesGSg5qykNdXU8sW8ff/jOO/xwezvBujqstXyUy/FqXR2xcNjrEGUFxFOJ4pUQpIJZQtpttmLWd61nXcM6rvdfp21vG6bOYCctwx8Os6FpA+u2rfM6RBGpUEf27uWPL15k/dgYWxuLU/BXJyb4k1u3ePqRR+Y94y+1Ix7JaGXLCmpoamD3/t30v9VPe6Sdusbib+DRC6M0XGlg+2eqZ8uBkrMa85kHH6RgLb988iSbJicZtZZCays/9OijbJpnOYVUsVJlp9TBPJTOxIYCXdprtoKMz/Dkdz3JOy++w4UXL0ALcBO6tnbxyHfqx5WI3NkDHR184amn+P2336Z5eBg/cK2+nk8/9hiHtm71OjyRqtf9ZDc+4+PUsVMUWgrYMUuoOcSj3/0ogWD1VENVclZjfMbwuT17+NSuXVy8cYNGv58t82xAlirlOMRDqeLV7ly5spPWqa+exuZGHv/Ox7l14xa5GzkCrQGaWpu8DktEqsDBzZvZ//nPc+HGDQrWsrW1VfvM1oBysS7avA6lpvnqfHQ/1c2Dhx4kez2Lv8FfXO5YZb+BlZzVqIDfz66ODq/DkOVUquyUCfohEAACRDojSsw80tTapKRMRBatzudje7v2p64VU4lZaNMOrWxZJfWBejq2VO9vYCVnIpUsmSyuUweSz2ZLlZ16lJCJiIhUiza1t5GFU3ImUqmmKju1TS2DCGojsYiIiEgNU3ImUkHiqQROZw6A1JfGVdlJREREZA1RcibiNccBIB5KFTcMt7URChQL4msZhIiISJVyHJzuHFA9lQLFe0rORDwUTR+FSPFLOxXMEtqgDcMiIiJVb6rdTSdEOiNeRyNVRMmZyGoqzZIBxS/tDXlCG4pNo0OElJiJiIjUgHJipiJeskhKzkRWS+ksmlvMxcj41aNMRESkVoVCnRrjZdGUnImspKmZMtct9yiLhHvKd+tLW0RERESmKDkTWSmlptH4/dADmYB6lImIiIjInSk5E1lOyWT56lSPsqmNwErK7i0/nmfgvQHOnjzLxPgEm7ZsYndkN20b2+79ZBEREVkR1lounrrIR+99RHYkS1uojd0P72bTrk0YY7wOr6YoORNZJvFUguSzWQgGyZCDgHqULcbkxCR9/7OPK/YKzfua8Qf8DF4cZPBPB/nUt3+K9dvXex2iiIjImtT/Sj8fnP2Apj1N1HfXM5wZ5pVjr3Dw2kH2HN7jdXg1RcmZyP0ozZTFI5lyjzLNlC3NhQ8vcCV/hdDhUPksXNsDbdxqvcVbf/cWn/vi53R2TkREKl48lSB1YJyQ14Esk+y1LCc/PEnoSAhfvQ8Af5OfxnWNvH/sfbr2ddHU2uRxlLVDyZnIEkXTR0n9o3xxTxkQCqlH2f0YOD1AYGdgTgIW2BBg+INhsleztG5o9Sg6ERGRe4unEuWTtbXym+DS2UvQSTkxm1LXWIfdaBk6N8T2g9s9iq72KDkTWSjHIR5KFa925ko9yrpq5svXa5OTk5i6uTNjxhhMnaEwWfAgKhERkYVzOnOENtXWydrJyUmou8OddWh8XmZKzkQWwnEIHz5GJuiHQAAIEOmMaOniMtqyfQvvXXiPwPrAjOMT2QnqJ+oJrg96FJmIiMgCOA5EAvd8WLXZsG0D9qTF7rEzVrfYgoUr0PFEh4fR1Z5VTc4ujF4mnkrMOBaPqGCCVKhkkngkU7x6OFvuUaaEbGXsOLCDM/1nGPlohNZwK6bOMD48TvadLI8+9ih1/judthMREZGV0rG1gy2hLVx8+yJt+9uoa6wjfyvPjfdvsGPLDm05WGarmpxdDkKiZ9qBkRFIJZSgSeUp9Si7PVMW1EzZCmtsaeTpv/c0J14+wcW/vQh+aPI38fhjj9O1v8vr8ERERNYkYwyHv3CYk986yZljZyj4C9QV6tjfvZ89j+9Rsa5ltqrJ2aaWTTzX81z5djKVJOEbxEkfJZq+v2ngeCoEsdh9Rihr2fRZ3cSXipt5VQp/dTW3N/PEdz3BRG6C/ESeQEsA49OXvoiIVDjHIdrdRyqYJ1QzdRpv8zf4eejIQ3Q/2c34rXEamhqoq9eKlpXg6Z6zWCRGMpUkFcyQCt/fayUODJBJJpWgyZJMr65UpMTMS/WBeuoD9V6HISIicm9TiVknRMJHanqVTV19HU31Kpu/kjwvCLJc1WwSfQlCXxogkj5636/l9PdANHr/QUlFi6cSOJ05gGI/khqrriQiIiKrJBAgFArVdGImq8Pz5Gy59Pb0kkwlcTfd3+tkMmnCwWO4zrSDStRqg+OUr0a7+0gdLJbCBwixfCcKRERERESWomaSM1ieH9eO65By+wj39BHOBSCXw0m6Wi5Z7UpLDorFPSiuCVePMhERERGpIDWVnC2HqeloN+PiUppJe3YQV/vZqs+0mbKpHmWhUHGTbiSkyosiIiKyDFwXtyfndRRSI5SczWP2j/ap/WwqOFJFSqXw8Rf/iWcC6lEmIiIiyyyZJPSlgWKFZ63GkWXg8zqAatDb0wttbYS+NDCnibZUEMcpXqb1KAt39xDu7lFiJiIiIsvLcYg+k1brHVlWmjlboN6eXhJ9pXLrapxdceKpBMnDWfD7yUSBQFBflCIiIrKyAgFCgdrraybeUXK2CFMVIRMMKEGrBMkkAPFIptyjLNIZAeYuTRURERERqXRKzhZpqnF2wjeIkz6K0/m81yGtSfFUguSzWQgGyZAjFFKPMhERERGpbvfcc2aM+S1jzGVjzIlpx+LGmI+NManS5btWNszKEovECG3oIrUhTzR9dEZVQFkhpb1kJJNE00dJHMxCVxfhcIRIuEeJmYisSRqjRURqy0JmzpLALwO/Pev4L1prf27ZI6oSsUis2BONY0S7+4r5mZpVr4xSj7LUkXy5+mIopB5lIiJojBYRqSn3TM6stS8ZY8IrH0r1mdrXlPL3EQ4ew3VQgrbcHKfcoywSPqK9ZCIi02iMFvHI1InjYJ4QKggiC5fou3vl9/sppf+Txph3SksqOu7jdapaNBwlEu4plm0/fKxcpEKWQTI5LTFTKXwRkUXQGC2yUqYSsw15Qhu0kkcWLtGXgJGRuz5mqQVBfhX494At/e/PA/90vgcaY54DngNo39y+xLerbOUZNLeP8LODuGpWvTzCYWDQ6yhERKrNgsbo6ePzjvbaHJ9FVkwgQGhDSImZzMtxHVLp1JzjoaEs7gs7MJy743OXNHNmrb1krZ201haA3wCeuMtjv2qtPWytPdzc3ryUt6sK0XCU3ujzZDYECT87qBm05RCN4h4/QiQNqdPHcFzH64hERCreQsfo6ePzxubaHZ9FRFaT4zqk3D5CQ1l6+5hxcV/ouucEzpJmzowxW6y1F0s3fwA4cbfHryWRzghubm6mLEsUjeI4FJcPcAw34+oslYjIXWiMFhFZXVMJGQD5PJEhP07/kbm1KCL3fq17JmfGmN8DosAGY8wg8DNA1BgTobhkwgV+YuHh174MOeKRDHGvA6kVMxK0QRzX0f4zERE0Rot4wnVxe3JeRyEem1rR5WZcMkODxYTsG53FO8PhJRcJXEi1xi/Oc/g3l/Rua0A0HCWVTpE4MAKpBPFIr9ch1YZSghbu6fM6EhGRiqExWmSVJZOEnx0kE/DTq5U8a1YylSSTSRMiALkckYwfp/N5iN3/ay+1IIjcRW9PL8lUkgQDStCWWz6Pm3G9jmLJrLVcv3CdywOXAdi0YxMdWzswxngcmYiIyNp2KZvlrcsXyebH2d22noc2bqS+ru72AxyH6DNpMhuCRDojWsmzxkzNlKXSKRgZoff9NuKpUhuFZSwEqORshcQisWKC5hvESR8tZtNyf6JRYqkUiYYBEn0JenuqK+mdzE/y5jff5OOhjzGdxWSs/2/62bZhG499x2P46u6ns4WIiIgs1V+ePc3vXzyB2WKoazH82eVTdLlt/NQnnqKjqan4oGiUaCqFG8ziZvrI+POk0qmq+z0ii5foS0A2SyjvJ5TPEzvdVpx8WcAessVScraCphK0FINE00dx+nvUpPo+xSO9kEqQOJglmUpWVXGQs6mznL95no5PdWB8xeTMPmg5/8Z51qXW8eBjD3ocoYiIyNrz0bVr/N7lE2x7vJX6+tJM2Tb4+PwIyf4U/+rQU+XHxiO9xB2neD1U3MaS6EsQ6Zz5K10zatVpvsrgUzNlma/vKLV5YkV/zys5W2EzErTuPhwHJWj3KZ6J4GRTuBu8jmThrLWcfu80rY+1lhMzAOMzBPcFOZ06reRMRETEAy9ePEfD9rrbiVnJ1q5W3jl/iaHRUTZMbzdR+h0XJ0o8mST6TBo3M3NPfMLtIxLuUZJWRcol8GfVeolkwPnGjlXrYazkbBXEIrHif3COKUFbLrkcmVzG6ygWzBYsuVyOUDA057761nqGR4exBTsjcRMREZGVlx7L0tJSP+e4MYa6ZsPI2NjM5Gy6WAynNJNW5rqEnx0k5faV98mHQ2ElahXIcZ3yf6NMJk0om8c9fmTuA2PRVYtJydkqmfo/ZMrfRzh4DNdBCdpSRaM4yeIXX8I5Sm+08vfzGZ+hpbWFsetjNHY0zrhv/Po4re2tSsxEREQ8sKs5xJmR67S3B2Ycn5wsULgJ66f2nN3JPL/nXMch2t0HmeKJ5FTwWPGhStAqRjKVLJbAzwaLB3LM35tslSk5W0XlBM3tI3z4GG7SXbUp0poTiq9DOAAAHhhJREFUi+Emk4T/Yboq9p4ZY9j3yD6Ov3Oc+sfr8dUXi38UJgrc/OAmjx963OMIRURE1qbPbAvzzXfPMLp+gubm4gyatZbzZ0Z4un0H7YHAPV5hHqUWQOWb3X2kODaj4nSl/3aZbmqGqZpink8ylSxfL/cm65+2X7ACJk6UnK2yaDhKNBwl4Rwl/OwgbjKpBG2pwmHCuQyu13Es0PaHtnNz5CYnXzoJpf1yZshw4MABuvZ3eRuciIjIGrW1tZX//YEn+PU33+BKx01oAHsNDjV28qWHHl76C0/7oe8QJZo+CkNpAFLrxqvi5DJM24uVzZPIVMeKpfkkU0kylweIXGsAIJoOFgvNdXoc2CzGWrtqb7Z131b73K8/t2rvV+kSfYnb1V+UoC1eaclAqpOq2nR7a+QWVwevYoxhXdc6mlrvsVxCKprjOsVKThVo+N8Ov2GtPex1HFL5Dm/dao8/p/FZ1rZbExP0Dw2Ry+fpamujq61txfqQ/v/t3W1sXNd95/HvIYfi04gaihJFPdG0ZNmyYytj+SGyrCjjuMmmybbpInG6CVJk2kXdAl0g3L6pdt+UKVBAWHR3580iRbLNTnYRtwmUJummhZEg9oWtpHQs2zeyJTMSbY0pShpJjDSihuKIHPLsi5mh+CiNyCHvvcPfBzBI3qHJ/8EV58xvzlOPmyDx0DC0tExd89OW/DP6tlyOaBqcFzsKB3CHQ7CY0USvlc4m88H5w+ZrX1uwf9bImYe693WT6E0Q+dIA3Tqs+u4VpwzE6MXNB2cud2NLI9se0khZNZj+bmK8P+x1OXN8zesCREQCpLGujkc3b16R31U6GqiktCW/HwLa1FqsoRCxdAPQQE8mCvEYqWSSnmhwNmSbyR/B7E4UzjxWCmiySKWANm0udxCmCEjwJN3k3B1Cc7lbOzv5YJ76bF/7keKZiIhfTQ8KPckkkS8NzPuaMNoRXfY3n2f0cdlsYS1Wx6G5U/7icXqWtRJROJPgmxHQBgMzh1uCY+67iCXFdxN9GMxERCRA4nEy84xKJe/L4uYKZ6gtV0ArLbPpPlmaYhkOxAhTtVI4k+pQCmjRYB1OLf5V9ruIIiIilTDPqFSP49D1+FHcVO/UGrBKjKTNGKHz0VosUTiTapPL3fl7RO5g9ruIPe42bdojIiIrLxYj5UBPpBDMnI4cbnZpyzjm9nHamM5PFM6kesRixFyXxPoB3yyqleBI9CZuhfuxsZnvIkYX/v9ERESWVSxGD7FbX6YP4zJIwjkMQCTScdugVtq8qiSSzZM6Mi2QqY/zFYUzqSql3Y8SDw1r7ZnckZNyAKZ2XEwdmbaLpt5FFBERH3I6DkEyCUBPNEPioQGSbpKuSNe8368+LlgUzjyWcA4TyeYL06b0zkVF9ES7cdKHSbV7XYn41dQW+BQ295jqtNRZiYhIEBT7qx4AN0FybJBUZp4t7nM5ohlw+g5APLZy9cmiKZx5KNGb0IvCZTRn23NZstJI0+347ay5+Wq+9S5ipHglor9BEREJpJ5oNz3FkbR5dXVpV+EAUTjzWLw/rBeFy6B0in3COUx37JDX5VSFRG8Cslki+YWfNjKhPG7a9c16v9IW+LNr1ruIIiJSVfRasmr4JpxZa7ly7gpnf32Wmzdv0rapje27t1PfXO91aRJE8TipZLIQ0LQ5yKI5KYdUJlUYhRweJvNCZ+EduIWkUlOhOBLpoCvSteIjaaWagamzyZy+fXO/Ue8iipTtYjbLz8+f5ezoNbY2ruXpzZ1sXrvW67JERKqOL8KZtZaTR09y6swpQp0hatfXcuH8BU6/fZqnP/M0LRtb7vxDRGaLx4m7CRI69+yuTa3JykFXrgFyOZwXy9tqN5VMEvtUGjIZUg29JFK9RLv2rUhIK43uRbNhyOWIpYsHaepsMpFFezN9gf/53i+xWyG8YQ3Hr1/kX97u50+6HuMjW7bd+QeIiEjZfBHOfnP2N5xKnWLd/nXUhGoAaOpoYuTcCG+8/Aax52IYYzyuUgIrl8NJOb5bC+U3M0acMulb6yFLI2XlTgGMx3EcZ+rL0uGZpZ9d6dG0pJsEmBrd6z7ZQk+muLuORsdElmRkbIy/7T9G294mmprqAFi/vpHR9nH+1xtv8mDbRlrqNcNFRKRSfBHOBn49QKgzNBXMSpq2NHHtvWtcH7qu0TNZlJ5MFCfdi5s/Cvhvswq/cFIObv/RwogTQG6Ja7KmhaKUA7HdvVDcRcoNL+3wzOlmj5SVO7onIuV559IlbrZNTAWzksbGOsY2Zjl+Mc2Bzns8qk5EpPr4IpzlcjlCkbmlGGMwDYbxm+MeVCVVIRbDcQrhwKVyoaAalEacYPrarGkjTpWaChiL4RCD4mha4V4Mzvj95d6Tio3uiUhZbuTHMQ123sdqGgzZcfXPIiKV5Itw1tbexuXLl2nc2Djj+uT4JAxDuDXsUWVSFYrhIJY+PBUKVntAS7pJMpcGiF5ZU7wSKhxquZxrs4qjaQ4xetwETkcaAHf9WFn3ZGodXDZPV7bw1OW8qGMoRJbTlvBaGDBw79zH7FXYtkWzWkREKskX4azzwU76v9/P6IZRGtsLAW0yP8m149fYed9O7dgoFeF0HKLHTZDYtzrPP0v0Jm59UVqbFfVmF8vpv7fHTZBgYGZ988lmi6N7B25Nm4wvW4kiAuxqa2PH+6188EGGbZ0tGGOw1nJ+8Drbb67jwQ3acUlEpJJ8Ec4aWxrZ/6n9vPHSG2ROZzD1Bq7Bjh07+NCBD3ldnkjgJXoTU4GswLtgNltPtBvcOwQzAMKFjT60yYfIiqkxhq/u+Qh/9+5bvH3hIjVhw2TW8mDdRv54z15qa2ru/ENERKRsvghnAK2bW3n2i89y7dI1xm+Os7ZtLQ3hBq/LWjalF8ugKSErbnh4VZx9NnukLPOCfzfL8EtQFJG51jU08OePPkU6m+XK6Cit9zTQEQ5rF2URkWXgm3AGYGoMkY6I12Usu+mjGHpRurJKozSJh6o3oE1fmxXvL6zX7HH9G8xEJBg6wmE6wloDLiKynHwVzlYTBTPvTAW0fV5XUjlOysFNu4UvcrnCLobHpq3NinpWmoiIiIiUSeFMJOCmzigbCuG8WNxusatLa7NEREREAkbhTFalHjdC8r5BEs5humOHvC7nriXdJJlMYSt68vniLob7dM6XiIiISIApnMnqFI+TSibp+vxgYNaeOSkHgFQmRebSQGFqrFtcoxmPL+8ZZSIiIiKy7BTOZPWKx4m7CRIBOKYn0ZsorCWjAbJZuvuLaxa1lkxERESkaiicieRyOCmHWFfM60pmKI2UuWn31u6ebgSIaOdFERERkSqkcCarWk8mipPuxc0fBfBNQEs4h4nkCp9H8nlSR4pb4WukTERERKRqKZytMCflQDYL6KwYX4jFcByI7e7F5SipTIp4NL7iZTgph1QmBUAmky5shX9kW2HXRdBGHyIiIiKrgMLZCkq6STJDg0SHQvRkNATiG7EYDjFi6cO4DJJ0kysa0KYOjc5BV64BcuD0HVAgExEREVll7hjOjDHfAv4tcMla+3Dx2nrgu0AXkAK+YK29unxlVodMLnNry/MqPYPqYjbLv545w+ClS9SvWcOeri4e27qVUE2N16XdkdNxiB43QSKaXvY1aEk3OfV5KbA7fdNOxa7Sfx8iUlnqo6VcI2Nj9A4McGpwEGst92/bxkc6O1lbX+91aSIyTTmvmJPAp2ZdOwT8zFq7C/hZ8WspQyzdULUvvN+/epVvv/wyG95/ny9MTPBsNsvpY8f4zuuvk5+c9Lq8svS4kcKOiMuoNILalcrQlZoV2Ev/iYiUJ4n6aLmD4Zs3+earrzL6zjv87s2b/N7YGOMnTvDNV14hk8t5XZ6ITHPHkTNr7SvGmK5Zlz8LxIqffxtwgL+oYF0SMNZafvzmm3wuFGJnYyMA7cC9DQ383/Pn+VU6zWNbtnhbZLmyWdy0W9GRsxkjZaUzykpTW2MxnVEmIouiPlrK4fT388j16zwbiUxd21JfT9O1a/zs17/mcx/+sIfVich0i51rtslaewGg+LG9ciVJEF3IZqnNZtnRMHPUyRjDk/X1nBgY8KiyuxSPE+8Pw/Bw4WyxCkj0JshcGqCrL01XX7oQzKLdGiUTkeWiPlqmWGt558wZPrJ27ZzHngyHefeDD5i01oPKRGQ+y74hiDHmeeB5gHWb1i33rxOPjE1M0EghjM3WVFPD2Pj4yhe1SD3RbnqSSSLxxa89mxHsSmeURbsLX2uUTER8YHr/3LlO/XM1G8/naZxn7Xd9TQ12cpJJa6mZp/8WkZW32JGzi8aYzQDFj5cW+kZr7TestY9bax9vWte0yF8nfrc5HOZSbS3X8/k5j72by3FPR/ASyWLXniWcw0SGsnT3QncvZF7ovBXMRESWX1l99PT+eWOT+udqZYyhc9Mm+m7cmPPY6dFROjZsCMSmXSKrxWJHzv4J+ApwuPjxRxWrSAKpPhTiyd27+d7x43yupYVIKMSktRwfGeF4fT3Pb9/udYl3L5vFTfUCtz+c2kk5uGm38EUud+uMsni8cE2nJojIylIfLTMcfOABfvjKK4RzOTrr6zHGcDaX45/HxvjM7t1elyci05Szlf7fU1hYvMEYMwj8JYUn/O8ZY/4DMAA8t5xFSjA8s3MntTU1fKOvj7UjI4wAkQ0b+IMPf5h1Dcu7A2LFxeOkkkm6Pj9424A2dUZZNl9Yq0ZDYaMPnVEmIitAfbSUY+f69Xx6/35+9PbbMDyMASaamvjkU0/xwIYNXpcnItOUs1vjFxd46NkK11LVEs5hItk8Pe62qh1JMcbwsR072H/PPQzduEFDKERrcefGQCoFtHhmxuWkmySTSRe+yOeLW+Ef0OYeIrLi1EdLuR5sb2f3xz/O5Rs3sNaysblZ68xEfGjZNwSRwuYQc6a6VbG62lo2z7MrVDUonVEWHQrhvFhcR9fVpWAmIiK+Z4yhvbnZ6zJE5DYUzlZCLrdqglnV6eqiK92Lmz9amN44NnZr58W418WJiIiISDVROFtmTsrxugRZilgMxwFSqVvXFLJFREREZBkonC2j0mYR0TSFqW8STJqyKCIiIiIrQOFsmcxYm9S3Ty/wRURERETktnTq4DJQMBMRERERkbulkbMKmxHMOg5Bh9cViYiIiIhIEGjkrIKSbpLMpQG63wkXgpmIiIiIiEiZFM4qxEk5ZDLpW9usi4iIiIiI3AWFswqK0ECPG/G6DBERERERCSCtOasQN+0SyWYBhTPxp3Q2yyunT9N/7hzGGHZv387Hdu1ifWOj16UFyqS1vD44yOunT3M1m6U1HObJ++/nia1bMcZ4XZ6IiATMyNgYr7z/Pm+fOcPY+Didmzbx0fvv597WVq9LC5zTv/kNR0+d4tzlyzSsWcOeHTs4uGMHDaHgRJ7gVOpjid4EkaEsqSPbdECx+NK54WG+8+qrfGxigt9pbmYCePPMGb514QJ/dPCgAtpd+H8nTnDl1Ck+29TE5rVrOX/zJj/95S+5+MAD/M6HPuR1eSIiEiCj4+N86xe/YOfVq/xxOExzfT19ly/z/XSazzz1FA+2t3tdYmC4Fy7w8muv8W/q6rgvHOb6xARHT5wgefEif7R/P2tqa70usSya1rgETsoh4RxWMBPf+9m77/KJyUk+0tJCY20t4dpaDq5bxxO5HK/093tdXmBcuH6d9/r7+XIkwvaGBkLG0NnQwB9EIpw6fZpLIyNelygiIgHy+uAg265e5dOtrbTW1bGmpoY94TDP1dfzk+PHmbTW6xIDIT85yU9/9Su+1NTEQ83NrKmpoa2ujt+NRFg3NIR74YLXJZZN4WyRnJSDm+olks0rmImvjU1McDad5pFweM5je5ub6Tt71oOqgunXQ0PssZa6mplPnWtqanjYWvouX/aoMhERCaK+s2fZO8/slc76empGRvSmX5kGh4dpvXmTTWvWzLhujGFvfT19g4MeVXb3NK1xEWYEs2MHIB7zuiSRBVlrwdp534mpNYbJyckVrymoJq2lZoF1ZbXFx0VERMplraV2nn7FGEOttepXyjS5wOscCN5rHY2c3SUn5eD2HyWaphDMYjGvSxK5rfpQiI6NG+m7cWPOY8dHRti1dasHVQXTfW1tnAAmZnWWE9Zywhh2tbV5U5iIiATSfVu3cnx0dM719NgYo42NbGpu9qCq4NnW0sLlUIir4+NzHjueywXqtY7C2V1IuslCMBsK4fTtUzCTwHjmoYf453yekyMjTFpL3lrevH6dV2pr+eiuXV6XFxjbW1rYsH07RzKZqQ7gyvg437t6lY577mFrS4vHFYqISJA8uX07fc3NvHrtGjcnJ7HWcmZ0lO+OjPDMI49QW6OX6uVYU1vLwUce4YXr1zmby2GtZXRigpeuXePsunXs3bLF6xLLpmmNZUq6STJDg4Vg1nEIOryuSKR8O1pb+dzBg7x88iQ/uHwZjKFz82a+/OCDdMyzFk3mZ4zhuUcfxWlp4Zv9/UyOjFBTX89je/bwsR07vC5PREQCZm19PX944AA/7evjb86epWZyknWRCM/u3cvDmzZ5XV6gPNXZSUMoxA/7+sheu4atrWX3vffyhw88QGNdndfllU3h7C5Es2GcvqiCmQTSjtZWdjz9NDfzeYwxgdlS1m9CNTX81q5dfPy++8jl8zSEQguuQxMREbmT1sZGvvDoo4zv2UN+cpKGUEjnZi7So1u2EN28mVw+T11tLaEAjjwqnN2NXM7rCkSWrD5ABzH6WY0xNAXonTgREfG3utpa6vTG6ZIZYwI1UjabXqWVIdGbgOFhYukWrTMTEREREZFlEbyxvhVWCmbdJ1voiXZ7XY6IiIiIiFQpjZzdRimYZV7o1CHTIiIiIiKyrBTOFpBwDhcOmT6iYCYiIiIiIstP4WwWJ+XgpnqLwWybgpmIiIiIiKwIhbNZUplUIZgdOwDxmNfliIiIiIjIKqENQebRlQ1pV0YREREREVlRCmfTOCmHzNCg12WIiIiIiMgqpGmNRUk3SWZokOhQCKdvH3R4XZGIiIiIiKwmCmfMCmYdhxTMRERERERkxa36aY1JN0nm0gDd74QLwUxERERERMQDq3rkrHTIdPfJFnqi3V6XIyIiIiIiq9iqHTlTMBMRERERET9ZlSNnpWCWeaFTh0yLiIiIiIgvrLqRs0RvgshQVsFMRERERER8ZdWFM4B4f1jBTEREREREfGVVhTMn5UAu53UZIiIiIiIic6yacOakHNxUL5Fsnp5M1OtyREREREREZlgVG4I4KQe3/2jhkOm+AxCLeV2SL1hr6b9yhZPnz5OfmKCrvZ09mzZRV1vrdWkiIiKr2sVslrfOneP6jRu0RyLs3bKFtfX1XpclIstsSeHMGJMCrgMTQN5a+3gliqqkpJskMzRYDGb7FMyKJq3liOvymw8+YG9NDfU1NZxMpfhFJEL8qafUAYiIBFwQ+miZ378ODPDzt97icWBrbS0DAwN8/d13+cL+/XRFIl6XJyLLqBIjZ89Ya4cq8HMqTsFsYW+cO8eNM2d4vrWVWmMAiAIvXbvGv5w4we/v3ettgSIiUgm+7aNlfpdGRvj5W2/xfHMzLaHCy7RHgAdHRzny2mv8p098gtqaVbMqRWTVqdq/7lIw634njNNxSMFslrfef5+DjY1Twazk6bVrOTM4yI3xcY8qExERWb3cc+fYC1PBrGRHYyNto6OcvnLFm8JEZEUsNZxZ4CfGmDeMMc/P9w3GmOeNMceMMcduXLuxxF9XHiflkMmk6X4nTE+0e0V+Z9BkR0dpq6ubc72+poYmaxlVOBMRCbrb9tHT++fLN1amf5Y7y46O0rbA2u82YGRsbGULEpEVtdRw9rS1di/w28CfGWMOzv4Ga+03rLWPW2sfb1rXtMRfV74IDfS4mpe9kPbWVlLzHCuQyecZDYVo0ZozEZGgu20fPb1/3ti0cv2z3F77unWk8vk51621fAC0NzevfFEismKWFM6steeLHy8BPwCerERRS+WmXchmvS7D15667z5eGh/nyrQRsrHJSX58/TqP3X+/dmwUEQk4v/bRcnuPbtnCqfp6Tk0bzbTW4gwP09jezraWFg+rE5HltugNQYwxzUCNtfZ68fNPAn9VscoWKdGbgOFhUkc6IR73uhzf2rl+PR994gm+4bp0joxQD7xnDLvvv59ndu70ujwREVkCv/bRcmfNa9bw7/fv58jrr/NqJkObMQxYS3N7O7//2GOYWWvFRaS6LGW3xk3AD4pPEiHgBWvtixWpapESzmEi2byCWZke37aNRzo6OH3lCvnJSZ6NRIg0NHhdloiILJ3v+mgp3/Z16/jqs8/y3pUrZMfGeKK5mS1r1yqYiawCiw5n1tr3gQ9XsJZFc1IObqq3GMy2KZjdhfpQiIfb270uQ0REKshPfbQsTo0x7Gpr87oMEVlhgd9KX8FMRERERESqQSUOofbMjGB27ADEY16XJCIiIiIisiiBHTlzUg5u/1GiaQrBTIdMi4iIiIhIgAUynJVGzKJDIZy+fQpmIiIiIiISeMZau3K/zJjLwAdlfOsGYGiZy1lJ1dSeamoLqD1+Vk1tAW/ac4+1duMK/04JoLvon6G6/jarqS1QXe2pprZAdbWnmtoCPuufVzSclcsYc8xa+7jXdVRKNbWnmtoCao+fVVNboPraI6tXNf1brqa2QHW1p5raAtXVnmpqC/ivPYGc1igiIiIiIlJtFM5ERERERER8wK/h7BteF1Bh1dSeamoLqD1+Vk1tgeprj6xe1fRvuZraAtXVnmpqC1RXe6qpLeCz9vhyzZmIiIiIiMhq49eRMxERERERkVXFd+HMGJMyxrxtjHGNMce8ruduGWO+ZYy5ZIx5Z9q19caYnxpjThc/tnpZY7kWaEuPMeZc8f64xphPe1ljuYwx240xLxtj3jXGnDDGfLV4Paj3ZqH2BPX+NBhjfmmM+VWxPV8rXr/XGPNa8f581xizxuta7+Q2bUkaY85MuzdRr2sVuRvqn/2jmvpnqK4+Wv2zfwWlf/bdtEZjTAp43FobyPMTjDEHgSzwf6y1Dxev/VfgirX2sDHmENBqrf0LL+ssxwJt6QGy1tq/8bK2u2WM2Qxstta+aYxZC7wB/B4QJ5j3ZqH2fIFg3h8DNFtrs8aYOuAo8FXgz4F/tNb+gzHmb4FfWWu/7mWtd3Kbtvwp8GNr7RFPCxRZJPXP/lFN/TNUVx+t/tm/gtI/+27kLOista8AV2Zd/izw7eLn36bwR+p7C7QlkKy1F6y1bxY/vw68C2wluPdmofYEki3IFr+sK/5ngY8DpSfLQNyf27RFRDyk/tm/qqmPVv/sX0Hpn/0YzizwE2PMG8aY570upkI2WWsvQOGPFmj3uJ6l+o/GmOPFaRW+n2IwmzGmC3gUeI0quDez2gMBvT/GmFpjjAtcAn4KvAdkrLX54rcMEpAObnZbrLWle/PXxXvzP4wx9R6WKLIY6p/9L5DP/9NVUx+t/tl/gtA/+zGcPW2t3Qv8NvBnxaF78Y+vAzuBKHAB+G/elnN3jDFh4PtAt7V22Ot6lmqe9gT2/lhrJ6y1UWAb8CTw4HzftrJVLc7sthhjHgb+M7AbeAJYD/h6ao7IPNQ/+1tgn/9LqqmPVv/sT0Hon30Xzqy154sfLwE/oPCPIOguFucgl+YiX/K4nkWz1l4s/sOeBL5JgO5PcX7x94HvWGv/sXg5sPdmvvYE+f6UWGszgAPsAyLGmFDxoW3Aea/qWoxpbflUcaqLtdbeBP43Abw3srqpf/a3oD//V1Mfrf7Z//zcP/sqnBljmouLJzHGNAOfBN65/f8VCP8EfKX4+VeAH3lYy5KUniSL/h0BuT/FRaB/B7xrrf3v0x4K5L1ZqD0Bvj8bjTGR4ueNwG9RmKf/MvD54rcF4v4s0Ja+aS8wDIW5+YG4NyKg/jkIgvr8D9XVR6t/9q+g9M++2q3RGLODwrtxACHgBWvtX3tY0l0zxvw9EAM2ABeBvwR+CHwP6AQGgOestb5fyLtAW2IUhuQtkAL+pDQf3M+MMQeAV4G3gcni5f9CYR54EO/NQu35IsG8P3soLCiupfCm0festX9VfE74BwrTDN4Cvlx8Z8u3btOWl4CNgAFc4E+nLUwW8TX1z/5STf0zVFcfrf7Zv4LSP/sqnImIiIiIiKxWvprWKCIiIiIislopnImIiIiIiPiAwpmIiIiIiIgPKJyJiIiIiIj4gMKZiIiIiIiIDyiciYiIiIiI+IDCmYiIiIiIiA8onImIiIiIiPjA/weyvjipu20WGQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "titles = ('K Neighbors with k=1', 'K Neighbors with k=2')\n", "\n", "fig = plt.figure(figsize=(15, 5))\n", "plt.subplots_adjust(wspace=0.4, hspace=0.4)\n", "\n", "X0, X1 = X_train[:, 0], X_train[:, 1]\n", "\n", "x_min, x_max = X0.min() - 1, X0.max() + 1\n", "y_min, y_max = X1.min() - 1, X1.max() + 1\n", "xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.2),\n", " np.arange(y_min, y_max, 0.2))\n", "\n", "for clf, title, ax in zip(models, titles, fig.subplots(1, 2).flatten()):\n", " Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z = Z.reshape(xx.shape)\n", " colors = ('red', 'green', 'lightgreen', 'gray', 'cyan')\n", " cmap = ListedColormap(colors[:len(np.unique(Z))])\n", " ax.contourf(xx, yy, Z, cmap=cmap, alpha=0.5)\n", " ax.scatter(X0, X1, c=y_train, s=50, edgecolors='k', cmap=cmap, alpha=0.5)\n", " ax.set_title(title)\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题3.2\n", "  利用例题3.2构造的$kd$树求点$x=(3,4.5)^T$的最近邻点。\n", "\n", "**解答：**" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "x点的最近邻点是(2, 3)\n" ] } ], "source": [ "import numpy as np\n", "from sklearn.neighbors import KDTree\n", "\n", "train_data = np.array([(2, 3), (5, 4), (9, 6), (4, 7), (8, 1), (7, 2)])\n", "tree = KDTree(train_data, leaf_size=2)\n", "dist, ind = tree.query(np.array([(3, 4.5)]), k=1)\n", "x1 = train_data[ind[0]][0][0]\n", "x2 = train_data[ind[0]][0][1]\n", "\n", "print(\"x点的最近邻点是({0}, {1})\".format(x1, x2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题3.3\n", "  参照算法3.3，写出输出为$x$的$k$近邻的算法。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "**算法：用kd树的$k$近邻搜索** \n", "输入：已构造的kd树；目标点$x$； \n", "输出：$x$的最近邻 \n", "1. 在$kd$树中找出包含目标点$x$的叶结点：从根结点出发，递归地向下访问树。若目标点$x$当前维的坐标小于切分点的坐标，则移动到左子结点，否则移动到右子结点，直到子结点为叶结点为止； \n", "2. 如果“当前$k$近邻点集”元素数量小于$k$或者叶节点距离小于“当前$k$近邻点集”中最远点距离，那么将叶节点插入“当前k近邻点集”； \n", "3. 递归地向上回退，在每个结点进行以下操作： \n", "(a)如果“当前$k$近邻点集”元素数量小于$k$或者当前节点距离小于“当前$k$近邻点集”中最远点距离，那么将该节点插入“当前$k$近邻点集”。 \n", "(b)检查另一子结点对应的区域是否与以目标点为球心、以目标点与于“当前$k$近邻点集”中最远点间的距离为半径的超球体相交。如果相交，可能在另一个子结点对应的区域内存在距目标点更近的点，移动到另一个子结点，接着，递归地进行最近邻搜索；如果不相交，向上回退；\n", "4. 当回退到根结点时，搜索结束，最后的“当前$k$近邻点集”即为$x$的最近邻点。" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "# 构建kd树，搜索待预测点所属区域\n", "from collections import namedtuple\n", "import numpy as np\n", "\n", "\n", "# 建立节点类\n", "class Node(namedtuple(\"Node\", \"location left_child right_child\")):\n", " def __repr__(self):\n", " return str(tuple(self))\n", "\n", "\n", "# kd tree类\n", "class KdTree():\n", " def __init__(self, k=1):\n", " self.k = k\n", " self.kdtree = None\n", "\n", " # 构建kd tree\n", " def _fit(self, X, depth=0):\n", " try:\n", " k = self.k\n", " except IndexError as e:\n", " return None\n", " # 这里可以展开，通过方差选择axis\n", " axis = depth % k\n", " X = X[X[:, axis].argsort()]\n", " median = X.shape[0] // 2\n", " try:\n", " X[median]\n", " except IndexError:\n", " return None\n", " return Node(location=X[median],\n", " left_child=self._fit(X[:median], depth + 1),\n", " right_child=self._fit(X[median + 1:], depth + 1))\n", "\n", " def _search(self, point, tree=None, depth=0, best=None):\n", " if tree is None:\n", " return best\n", " k = self.k\n", " # 更新 branch\n", " if point[0][depth % k] < tree.location[depth % k]:\n", " next_branch = tree.left_child\n", " else:\n", " next_branch = tree.right_child\n", " if not next_branch is None:\n", " best = next_branch.location\n", " return self._search(point,\n", " tree=next_branch,\n", " depth=depth + 1,\n", " best=best)\n", "\n", " def fit(self, X):\n", " self.kdtree = self._fit(X)\n", " return self.kdtree\n", "\n", " def predict(self, X):\n", " res = self._search(X, self.kdtree)\n", " return res" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "x点的最近邻点是(2, 3)\n" ] } ], "source": [ "KNN = KdTree()\n", "X_train = np.array([[2, 3], [5, 4], [9, 6], [4, 7], [8, 1], [7, 2]])\n", "KNN.fit(X_train)\n", "X_new = np.array([[3, 4.5]])\n", "res = KNN.predict(X_new)\n", "\n", "x1 = res[0]\n", "x2 = res[1]\n", "\n", "print(\"x点的最近邻点是({0}, {1})\".format(x1, x2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "参考代码：https://github.com/wzyonggege/statistical-learning-method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第04章 朴素贝叶斯/4.NaiveBayes.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第4章 朴素贝叶斯" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1．朴素贝叶斯法是典型的生成学习方法。生成方法由训练数据学习联合概率分布\n", "$P(X,Y)$，然后求得后验概率分布$P(Y|X)$。具体来说，利用训练数据学习$P(X|Y)$和$P(Y)$的估计，得到联合概率分布：\n", "\n", "$$P(X,Y)＝P(Y)P(X|Y)$$\n", "\n", "概率估计方法可以是极大似然估计或贝叶斯估计。\n", "\n", "2．朴素贝叶斯法的基本假设是条件独立性，\n", "\n", "$$\\begin{aligned} P(X&=x | Y=c_{k} )=P\\left(X^{(1)}=x^{(1)}, \\cdots, X^{(n)}=x^{(n)} | Y=c_{k}\\right) \\\\ &=\\prod_{j=1}^{n} P\\left(X^{(j)}=x^{(j)} | Y=c_{k}\\right) \\end{aligned}$$\n", "\n", "\n", "这是一个较强的假设。由于这一假设，模型包含的条件概率的数量大为减少，朴素贝叶斯法的学习与预测大为简化。因而朴素贝叶斯法高效，且易于实现。其缺点是分类的性能不一定很高。\n", "\n", "3．朴素贝叶斯法利用贝叶斯定理与学到的联合概率模型进行分类预测。\n", "\n", "$$P(Y | X)=\\frac{P(X, Y)}{P(X)}=\\frac{P(Y) P(X | Y)}{\\sum_{Y} P(Y) P(X | Y)}$$\n", " \n", "将输入$x$分到后验概率最大的类$y$。\n", "\n", "$$y=\\arg \\max _{c_{k}} P\\left(Y=c_{k}\\right) \\prod_{j=1}^{n} P\\left(X_{j}=x^{(j)} | Y=c_{k}\\right)$$\n", "\n", "后验概率最大等价于0-1损失函数时的期望风险最小化。\n", "\n", "\n", "模型：\n", "\n", "- 高斯模型\n", "- 多项式模型\n", "- 伯努利模型" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "from sklearn.datasets import load_iris\n", "from sklearn.model_selection import train_test_split\n", "\n", "from collections import Counter\n", "import math" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# data\n", "def create_data():\n", " iris = load_iris()\n", " df = pd.DataFrame(iris.data, columns=iris.feature_names)\n", " df['label'] = iris.target\n", " df.columns = [\n", " 'sepal length', 'sepal width', 'petal length', 'petal width', 'label'\n", " ]\n", " data = np.array(df.iloc[:100, :])\n", " # print(data)\n", " return data[:, :-1], data[:, -1]" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "X, y = create_data()\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(array([5.7, 2.6, 3.5, 1. ]), 1.0)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_test[0], y_test[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "参考：https://machinelearningmastery.com/naive-bayes-classifier-scratch-python/\n", "\n", "## GaussianNB 高斯朴素贝叶斯\n", "\n", "特征的可能性被假设为高斯\n", "\n", "概率密度函数：\n", "$$P(x_i | y_k)=\\frac{1}{\\sqrt{2\\pi\\sigma^2_{yk}}}exp(-\\frac{(x_i-\\mu_{yk})^2}{2\\sigma^2_{yk}})$$\n", "\n", "数学期望(mean)：$\\mu$\n", "\n", "方差：$\\sigma^2=\\frac{\\sum(X-\\mu)^2}{N}$" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "class NaiveBayes:\n", " def __init__(self):\n", " self.model = None\n", "\n", " # 数学期望\n", " @staticmethod\n", " def mean(X):\n", " return sum(X) / float(len(X))\n", "\n", " # 标准差（方差）\n", " def stdev(self, X):\n", " avg = self.mean(X)\n", " return math.sqrt(sum([pow(x - avg, 2) for x in X]) / float(len(X)))\n", "\n", " # 概率密度函数\n", " def gaussian_probability(self, x, mean, stdev):\n", " exponent = math.exp(-(math.pow(x - mean, 2) /\n", " (2 * math.pow(stdev, 2))))\n", " return (1 / (math.sqrt(2 * math.pi) * stdev)) * exponent\n", "\n", " # 处理X_train\n", " def summarize(self, train_data):\n", " summaries = [(self.mean(i), self.stdev(i)) for i in zip(*train_data)]\n", " return summaries\n", "\n", " # 分类别求出数学期望和标准差\n", " def fit(self, X, y):\n", " labels = list(set(y))\n", " data = {label: [] for label in labels}\n", " for f, label in zip(X, y):\n", " data[label].append(f)\n", " self.model = {\n", " label: self.summarize(value)\n", " for label, value in data.items()\n", " }\n", " return 'gaussianNB train done!'\n", "\n", " # 计算概率\n", " def calculate_probabilities(self, input_data):\n", " # summaries:{0.0: [(5.0, 0.37),(3.42, 0.40)], 1.0: [(5.8, 0.449),(2.7, 0.27)]}\n", " # input_data:[1.1, 2.2]\n", " probabilities = {}\n", " for label, value in self.model.items():\n", " probabilities[label] = 1\n", " for i in range(len(value)):\n", " mean, stdev = value[i]\n", " probabilities[label] *= self.gaussian_probability(\n", " input_data[i], mean, stdev)\n", " return probabilities\n", "\n", " # 类别\n", " def predict(self, X_test):\n", " # {0.0: 2.9680340789325763e-27, 1.0: 3.5749783019849535e-26}\n", " label = sorted(\n", " self.calculate_probabilities(X_test).items(),\n", " key=lambda x: x[-1])[-1][0]\n", " return label\n", "\n", " def score(self, X_test, y_test):\n", " right = 0\n", " for X, y in zip(X_test, y_test):\n", " label = self.predict(X)\n", " if label == y:\n", " right += 1\n", "\n", " return right / float(len(X_test))" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "model = NaiveBayes()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'gaussianNB train done!'" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.0\n" ] } ], "source": [ "print(model.predict([4.4, 3.2, 1.3, 0.2]))" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1.0" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.score(X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### scikit-learn实例" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from sklearn.naive_bayes import GaussianNB" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "GaussianNB()" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf = GaussianNB()\n", "clf.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1.0" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.score(X_test, y_test)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([0.])" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.predict([[4.4, 3.2, 1.3, 0.2]])" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "from sklearn.naive_bayes import BernoulliNB, MultinomialNB # 伯努利模型和多项式模型" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第4章朴素贝叶斯法-习题\n", "\n", "### 习题4.1\n", "  用极大似然估计法推出朴素贝叶斯法中的概率估计公式(4.8)及公式 (4.9)。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "**第1步：**证明公式(4.8)：$\\displaystyle P(Y=c_k) = \\frac{\\displaystyle \\sum_{i=1}^N I(y_i=c_k)}{N}$ \n", "由于朴素贝叶斯法假设$Y$是定义在输出空间$\\mathcal{Y}$上的随机变量，因此可以定义$P(Y=c_k)$概率为$p$。 \n", "令$\\displaystyle m=\\sum_{i=1}^NI(y_i=c_k)$，得出似然函数：$$L(p)=f_D(y_1,y_2,\\cdots,y_n|\\theta)=\\binom{N}{m}p^m(1-p)^{(N-m)}$$使用微分求极值，两边同时对$p$求微分：$$\\begin{aligned}\n", "0 &= \\binom{N}{m}\\left[mp^{(m-1)}(1-p)^{(N-m)}-(N-m)p^m(1-p)^{(N-m-1)}\\right] \\\\\n", "& = \\binom{N}{m}\\left[p^{(m-1)}(1-p)^{(N-m-1)}(m-Np)\\right]\n", "\\end{aligned}$$可求解得到$\\displaystyle p=0,p=1,p=\\frac{m}{N}$ \n", "显然$\\displaystyle P(Y=c_k)=p=\\frac{m}{N}=\\frac{\\displaystyle \\sum_{i=1}^N I(y_i=c_k)}{N}$，公式(4.8)得证。\n", "\n", "----\n", "\n", "**第2步：**证明公式(4.9)：$\\displaystyle P(X^{(j)}=a_{jl}|Y=c_k) = \\frac{\\displaystyle \\sum_{i=1}^N I(x_i^{(j)}=a_{jl},y_i=c_k)}{\\displaystyle \\sum_{i=1}^N I(y_i=c_k)}$ \n", "令$P(X^{(j)}=a_{jl}|Y=c_k)=p$，令$\\displaystyle m=\\sum_{i=1}^N I(y_i=c_k), q=\\sum_{i=1}^N I(x_i^{(j)}=a_{jl},y_i=c_k)$，得出似然函数：$$L(p)=\\binom{m}{q}p^q(i-p)^{m-q}$$使用微分求极值，两边同时对$p$求微分：$$\\begin{aligned}\n", "0 &= \\binom{m}{q}\\left[qp^{(q-1)}(1-p)^{(m-q)}-(m-q)p^q(1-p)^{(m-q-1)}\\right] \\\\\n", "& = \\binom{m}{q}\\left[p^{(q-1)}(1-p)^{(m-q-1)}(q-mp)\\right]\n", "\\end{aligned}$$可求解得到$\\displaystyle p=0,p=1,p=\\frac{q}{m}$ \n", "显然$\\displaystyle P(X^{(j)}=a_{jl}|Y=c_k)=p=\\frac{q}{m}=\\frac{\\displaystyle \\sum_{i=1}^N I(x_i^{(j)}=a_{jl},y_i=c_k)}{\\displaystyle \\sum_{i=1}^N I(y_i=c_k)}$，公式(4.9)得证。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题4.2\n", "  用贝叶斯估计法推出朴素贝叶斯法中的慨率估计公式(4.10)及公式(4.11)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "**第1步：**证明公式(4.11)：$\\displaystyle P(Y=c_k) = \\frac{\\displaystyle \\sum_{i=1}^N I(y_i=c_k) + \\lambda}{N+K \\lambda}$ \n", "加入先验概率，在没有任何信息的情况下，可以假设先验概率为均匀概率（即每个事件的概率是相同的）。 \n", "可得$\\displaystyle p=\\frac{1}{K} \\Leftrightarrow pK-1=0\\quad(1)$ \n", "根据习题4.1得出先验概率的极大似然估计是$\\displaystyle pN - \\sum_{i=1}^N I(y_i=c_k) = 0\\quad(2)$ \n", "存在参数$\\lambda$使得$(1) \\cdot \\lambda + (2) = 0$ \n", "所以有$$\\lambda(pK-1) + pN - \\sum_{i=1}^N I(y_i=c_k) = 0$$可得$\\displaystyle P(Y=c_k) = \\frac{\\displaystyle \\sum_{i=1}^N I(y_i=c_k) + \\lambda}{N+K \\lambda}$，公式(4.11)得证。 \n", "\n", "----\n", "\n", "**第2步：**证明公式(4.10)：$\\displaystyle P_{\\lambda}(X^{(j)}=a_{jl} | Y = c_k) = \\frac{\\displaystyle \\sum_{i=1}^N I(x_i^{(j)}=a_{jl},y_i=c_k) + \\lambda}{\\displaystyle \\sum_{i=1}^N I(y_i=c_k) + S_j \\lambda}$ \n", "根据第1步，可同理得到$$\n", "P(Y=c_k, x^{(j)}=a_{j l})=\\frac{\\displaystyle \\sum_{i=1}^N I(y_i=c_k, x_i^{(j)}=a_{jl})+\\lambda}{N+K S_j \\lambda}$$ \n", "$$\\begin{aligned} \n", "P(x^{(j)}=a_{jl} | Y=c_k)\n", "&= \\frac{P(Y=c_k, x^{(j)}=a_{j l})}{P(y_i=c_k)} \\\\\n", "&= \\frac{\\displaystyle \\frac{\\displaystyle \\sum_{i=1}^N I(y_i=c_k, x_i^{(j)}=a_{jl})+\\lambda}{N+K S_j \\lambda}}{\\displaystyle \\frac{\\displaystyle \\sum_{i=1}^N I(y_i=c_k) + \\lambda}{N+K \\lambda}} \\\\\n", "&= (\\lambda可以任意取值，于是取\\lambda = S_j \\lambda) \\\\\n", "&= \\frac{\\displaystyle \\frac{\\displaystyle \\sum_{i=1}^N I(y_i=c_k, x_i^{(j)}=a_{jl})+\\lambda}{N+K S_j \\lambda}}{\\displaystyle \\frac{\\displaystyle \\sum_{i=1}^N I(y_i=c_k) + \\lambda}{N+K S_j \\lambda}} \\\\ \n", "&= \\frac{\\displaystyle \\sum_{i=1}^N I(y_i=c_k, x_i^{(j)}=a_{jl})+\\lambda}{\\displaystyle \\sum_{i=1}^N I(y_i=c_k) + \\lambda} (其中\\lambda = S_j \\lambda)\\\\\n", "&= \\frac{\\displaystyle \\sum_{i=1}^N I(x_i^{(j)}=a_{jl},y_i=c_k) + \\lambda}{\\displaystyle \\sum_{i=1}^N I(y_i=c_k) + S_j \\lambda}\n", "\\end{aligned} $$ \n", "公式(4.11)得证。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "----\n", "参考代码：https://github.com/wzyonggege/statistical-learning-method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第05章 决策树/5.DecisonTree.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第5章 决策树" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1．分类决策树模型是表示基于特征对实例进行分类的树形结构。决策树可以转换成一个**if-then**规则的集合，也可以看作是定义在特征空间划分上的类的条件概率分布。\n", "\n", "2．决策树学习旨在构建一个与训练数据拟合很好，并且复杂度小的决策树。因为从可能的决策树中直接选取最优决策树是NP完全问题。现实中采用启发式方法学习次优的决策树。\n", "\n", "决策树学习算法包括3部分：特征选择、树的生成和树的剪枝。常用的算法有ID3、\n", "C4.5和CART。\n", "\n", "3．特征选择的目的在于选取对训练数据能够分类的特征。特征选择的关键是其准则。常用的准则如下：\n", "\n", "（1）样本集合$D$对特征$A$的信息增益（ID3）\n", "\n", "\n", "$$g(D, A)=H(D)-H(D|A)$$\n", "\n", "$$H(D)=-\\sum_{k=1}^{K} \\frac{\\left|C_{k}\\right|}{|D|} \\log _{2} \\frac{\\left|C_{k}\\right|}{|D|}$$\n", "\n", "$$H(D | A)=\\sum_{i=1}^{n} \\frac{\\left|D_{i}\\right|}{|D|} H\\left(D_{i}\\right)$$\n", "\n", "其中，$H(D)$是数据集$D$的熵，$H(D_i)$是数据集$D_i$的熵，$H(D|A)$是数据集$D$对特征$A$的条件熵。\t$D_i$是$D$中特征$A$取第$i$个值的样本子集，$C_k$是$D$中属于第$k$类的样本子集。$n$是特征$A$取 值的个数，$K$是类的个数。\n", "\n", "（2）样本集合$D$对特征$A$的信息增益比（C4.5）\n", "\n", "\n", "$$g_{R}(D, A)=\\frac{g(D, A)}{H(D)}$$\n", "\n", "\n", "其中，$g(D,A)$是信息增益，$H(D)$是数据集$D$的熵。\n", "\n", "（3）样本集合$D$的基尼指数（CART）\n", "\n", "$$\\operatorname{Gini}(D)=1-\\sum_{k=1}^{K}\\left(\\frac{\\left|C_{k}\\right|}{|D|}\\right)^{2}$$\n", "\n", "特征$A$条件下集合$D$的基尼指数：\n", "\n", " $$\\operatorname{Gini}(D, A)=\\frac{\\left|D_{1}\\right|}{|D|} \\operatorname{Gini}\\left(D_{1}\\right)+\\frac{\\left|D_{2}\\right|}{|D|} \\operatorname{Gini}\\left(D_{2}\\right)$$\n", " \n", "4．决策树的生成。通常使用信息增益最大、信息增益比最大或基尼指数最小作为特征选择的准则。决策树的生成往往通过计算信息增益或其他指标，从根结点开始，递归地产生决策树。这相当于用信息增益或其他准则不断地选取局部最优的特征，或将训练集分割为能够基本正确分类的子集。\n", "\n", "5．决策树的剪枝。由于生成的决策树存在过拟合问题，需要对它进行剪枝，以简化学到的决策树。决策树的剪枝，往往从已生成的树上剪掉一些叶结点或叶结点以上的子树，并将其父结点或根结点作为新的叶结点，从而简化生成的决策树。\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "from sklearn.datasets import load_iris\n", "from sklearn.model_selection import train_test_split\n", "from collections import Counter\n", "import math\n", "from math import log\n", "import pprint" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 书上题目5.1" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# 书上题目5.1\n", "def create_data():\n", " datasets = [['青年', '否', '否', '一般', '否'],\n", " ['青年', '否', '否', '好', '否'],\n", " ['青年', '是', '否', '好', '是'],\n", " ['青年', '是', '是', '一般', '是'],\n", " ['青年', '否', '否', '一般', '否'],\n", " ['中年', '否', '否', '一般', '否'],\n", " ['中年', '否', '否', '好', '否'],\n", " ['中年', '是', '是', '好', '是'],\n", " ['中年', '否', '是', '非常好', '是'],\n", " ['中年', '否', '是', '非常好', '是'],\n", " ['老年', '否', '是', '非常好', '是'],\n", " ['老年', '否', '是', '好', '是'],\n", " ['老年', '是', '否', '好', '是'],\n", " ['老年', '是', '否', '非常好', '是'],\n", " ['老年', '否', '否', '一般', '否'],\n", " ]\n", " labels = [u'年龄', u'有工作', u'有自己的房子', u'信贷情况', u'类别']\n", " # 返回数据集和每个维度的名称\n", " return datasets, labels" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "datasets, labels = create_data()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "train_data = pd.DataFrame(datasets, columns=labels)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
年龄有工作有自己的房子信贷情况类别
0青年否否一般否
1青年否否好否
2青年是否好是
3青年是是一般是
4青年否否一般否
5中年否否一般否
6中年否否好否
7中年是是好是
8中年否是非常好是
9中年否是非常好是
10老年否是非常好是
11老年否是好是
12老年是否好是
13老年是否非常好是
14老年否否一般否
\n", "
" ], "text/plain": [ " 年龄 有工作 有自己的房子 信贷情况 类别\n", "0 青年 否 否 一般 否\n", "1 青年 否 否 好 否\n", "2 青年 是 否 好 是\n", "3 青年 是 是 一般 是\n", "4 青年 否 否 一般 否\n", "5 中年 否 否 一般 否\n", "6 中年 否 否 好 否\n", "7 中年 是 是 好 是\n", "8 中年 否 是 非常好 是\n", "9 中年 否 是 非常好 是\n", "10 老年 否 是 非常好 是\n", "11 老年 否 是 好 是\n", "12 老年 是 否 好 是\n", "13 老年 是 否 非常好 是\n", "14 老年 否 否 一般 否" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_data" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# 熵\n", "def calc_ent(datasets):\n", " data_length = len(datasets)\n", " label_count = {}\n", " for i in range(data_length):\n", " label = datasets[i][-1]\n", " if label not in label_count:\n", " label_count[label] = 0\n", " label_count[label] += 1\n", " ent = -sum([(p / data_length) * log(p / data_length, 2)\n", " for p in label_count.values()])\n", " return ent\n", "# def entropy(y):\n", "# \"\"\"\n", "# Entropy of a label sequence\n", "# \"\"\"\n", "# hist = np.bincount(y)\n", "# ps = hist / np.sum(hist)\n", "# return -np.sum([p * np.log2(p) for p in ps if p > 0])\n", "\n", "\n", "# 经验条件熵\n", "def cond_ent(datasets, axis=0):\n", " data_length = len(datasets)\n", " feature_sets = {}\n", " for i in range(data_length):\n", " feature = datasets[i][axis]\n", " if feature not in feature_sets:\n", " feature_sets[feature] = []\n", " feature_sets[feature].append(datasets[i])\n", " cond_ent = sum(\n", " [(len(p) / data_length) * calc_ent(p) for p in feature_sets.values()])\n", " return cond_ent\n", "\n", "\n", "# 信息增益\n", "def info_gain(ent, cond_ent):\n", " return ent - cond_ent\n", "\n", "\n", "def info_gain_train(datasets):\n", " count = len(datasets[0]) - 1\n", " ent = calc_ent(datasets)\n", "# ent = entropy(datasets)\n", " best_feature = []\n", " for c in range(count):\n", " c_info_gain = info_gain(ent, cond_ent(datasets, axis=c))\n", " best_feature.append((c, c_info_gain))\n", " print('特征({}) - info_gain - {:.3f}'.format(labels[c], c_info_gain))\n", " # 比较大小\n", " best_ = max(best_feature, key=lambda x: x[-1])\n", " return '特征({})的信息增益最大，选择为根节点特征'.format(labels[best_[0]])" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "特征(年龄) - info_gain - 0.083\n", "特征(有工作) - info_gain - 0.324\n", "特征(有自己的房子) - info_gain - 0.420\n", "特征(信贷情况) - info_gain - 0.363\n" ] }, { "data": { "text/plain": [ "'特征(有自己的房子)的信息增益最大，选择为根节点特征'" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "info_gain_train(np.array(datasets))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "---\n", "\n", "利用ID3算法生成决策树，例5.3" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# 定义节点类 二叉树\n", "class Node:\n", " def __init__(self, root=True, label=None, feature_name=None, feature=None):\n", " self.root = root\n", " self.label = label\n", " self.feature_name = feature_name\n", " self.feature = feature\n", " self.tree = {}\n", " self.result = {\n", " 'label:': self.label,\n", " 'feature': self.feature,\n", " 'tree': self.tree\n", " }\n", "\n", " def __repr__(self):\n", " return '{}'.format(self.result)\n", "\n", " def add_node(self, val, node):\n", " self.tree[val] = node\n", "\n", " def predict(self, features):\n", " if self.root is True:\n", " return self.label\n", " return self.tree[features[self.feature]].predict(features)\n", "\n", "\n", "class DTree:\n", " def __init__(self, epsilon=0.1):\n", " self.epsilon = epsilon\n", " self._tree = {}\n", "\n", " # 熵\n", " @staticmethod\n", " def calc_ent(datasets):\n", " data_length = len(datasets)\n", " label_count = {}\n", " for i in range(data_length):\n", " label = datasets[i][-1]\n", " if label not in label_count:\n", " label_count[label] = 0\n", " label_count[label] += 1\n", " ent = -sum([(p / data_length) * log(p / data_length, 2)\n", " for p in label_count.values()])\n", " return ent\n", "\n", " # 经验条件熵\n", " def cond_ent(self, datasets, axis=0):\n", " data_length = len(datasets)\n", " feature_sets = {}\n", " for i in range(data_length):\n", " feature = datasets[i][axis]\n", " if feature not in feature_sets:\n", " feature_sets[feature] = []\n", " feature_sets[feature].append(datasets[i])\n", " cond_ent = sum([(len(p) / data_length) * self.calc_ent(p)\n", " for p in feature_sets.values()])\n", " return cond_ent\n", "\n", " # 信息增益\n", " @staticmethod\n", " def info_gain(ent, cond_ent):\n", " return ent - cond_ent\n", "\n", " def info_gain_train(self, datasets):\n", " count = len(datasets[0]) - 1\n", " ent = self.calc_ent(datasets)\n", " best_feature = []\n", " for c in range(count):\n", " c_info_gain = self.info_gain(ent, self.cond_ent(datasets, axis=c))\n", " best_feature.append((c, c_info_gain))\n", " # 比较大小\n", " best_ = max(best_feature, key=lambda x: x[-1])\n", " return best_\n", "\n", " def train(self, train_data):\n", " \"\"\"\n", " input:数据集D(DataFrame格式)，特征集A，阈值eta\n", " output:决策树T\n", " \"\"\"\n", " _, y_train, features = train_data.iloc[:, :\n", " -1], train_data.iloc[:,\n", " -1], train_data.columns[:\n", " -1]\n", " # 1,若D中实例属于同一类Ck，则T为单节点树，并将类Ck作为结点的类标记，返回T\n", " if len(y_train.value_counts()) == 1:\n", " return Node(root=True, label=y_train.iloc[0])\n", "\n", " # 2, 若A为空，则T为单节点树，将D中实例树最大的类Ck作为该节点的类标记，返回T\n", " if len(features) == 0:\n", " return Node(\n", " root=True,\n", " label=y_train.value_counts().sort_values(\n", " ascending=False).index[0])\n", "\n", " # 3,计算最大信息增益 同5.1,Ag为信息增益最大的特征\n", " max_feature, max_info_gain = self.info_gain_train(np.array(train_data))\n", " max_feature_name = features[max_feature]\n", "\n", " # 4,Ag的信息增益小于阈值eta,则置T为单节点树，并将D中是实例数最大的类Ck作为该节点的类标记，返回T\n", " if max_info_gain < self.epsilon:\n", " return Node(\n", " root=True,\n", " label=y_train.value_counts().sort_values(\n", " ascending=False).index[0])\n", "\n", " # 5,构建Ag子集\n", " node_tree = Node(\n", " root=False, feature_name=max_feature_name, feature=max_feature)\n", "\n", " feature_list = train_data[max_feature_name].value_counts().index\n", " for f in feature_list:\n", " sub_train_df = train_data.loc[train_data[max_feature_name] ==\n", " f].drop([max_feature_name], axis=1)\n", "\n", " # 6, 递归生成树\n", " sub_tree = self.train(sub_train_df)\n", " node_tree.add_node(f, sub_tree)\n", "\n", " # pprint.pprint(node_tree.tree)\n", " return node_tree\n", "\n", " def fit(self, train_data):\n", " self._tree = self.train(train_data)\n", " return self._tree\n", "\n", " def predict(self, X_test):\n", " return self._tree.predict(X_test)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "datasets, labels = create_data()\n", "data_df = pd.DataFrame(datasets, columns=labels)\n", "dt = DTree()\n", "tree = dt.fit(data_df)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "{'label:': None, 'feature': 2, 'tree': {'否': {'label:': None, 'feature': 1, 'tree': {'否': {'label:': '否', 'feature': None, 'tree': {}}, '是': {'label:': '是', 'feature': None, 'tree': {}}}}, '是': {'label:': '是', 'feature': None, 'tree': {}}}}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tree" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'否'" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dt.predict(['老年', '否', '否', '一般'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### scikit-learn实例" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# data\n", "def create_data():\n", " iris = load_iris()\n", " df = pd.DataFrame(iris.data, columns=iris.feature_names)\n", " df['label'] = iris.target\n", " df.columns = [\n", " 'sepal length', 'sepal width', 'petal length', 'petal width', 'label'\n", " ]\n", " data = np.array(df.iloc[:100, [0, 1, -1]])\n", " # print(data)\n", " return data[:, :2], data[:, -1]\n", "\n", "\n", "X, y = create_data()\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "from sklearn.tree import DecisionTreeClassifier\n", "from sklearn.tree import export_graphviz\n", "import graphviz" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "DecisionTreeClassifier()" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf = DecisionTreeClassifier()\n", "clf.fit(X_train, y_train,)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9666666666666667" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.score(X_test, y_test)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "tree_pic = export_graphviz(clf, out_file=\"mytree.pdf\")\n", "with open('mytree.pdf') as f:\n", " dot_graph = f.read()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "Tree\r\n", "\r\n", "\r\n", "0\r\n", "\r\n", "X[0] <= 5.45\r\n", "gini = 0.5\r\n", "samples = 70\r\n", "value = [35, 35]\r\n", "\r\n", "\r\n", "1\r\n", "\r\n", "X[1] <= 2.8\r\n", "gini = 0.153\r\n", "samples = 36\r\n", "value = [33, 3]\r\n", "\r\n", "\r\n", "0->1\r\n", "\r\n", "\r\n", "True\r\n", "\r\n", "\r\n", "6\r\n", "\r\n", "X[1] <= 3.45\r\n", "gini = 0.111\r\n", "samples = 34\r\n", "value = [2, 32]\r\n", "\r\n", "\r\n", "0->6\r\n", "\r\n", "\r\n", "False\r\n", "\r\n", "\r\n", "2\r\n", "\r\n", "X[0] <= 4.7\r\n", "gini = 0.375\r\n", "samples = 4\r\n", "value = [1, 3]\r\n", "\r\n", "\r\n", "1->2\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "5\r\n", "\r\n", "gini = 0.0\r\n", "samples = 32\r\n", "value = [32, 0]\r\n", "\r\n", "\r\n", "1->5\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "3\r\n", "\r\n", "gini = 0.0\r\n", "samples = 1\r\n", "value = [1, 0]\r\n", "\r\n", "\r\n", "2->3\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "4\r\n", "\r\n", "gini = 0.0\r\n", "samples = 3\r\n", "value = [0, 3]\r\n", "\r\n", "\r\n", "2->4\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "7\r\n", "\r\n", "gini = 0.0\r\n", "samples = 32\r\n", "value = [0, 32]\r\n", "\r\n", "\r\n", "6->7\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "8\r\n", "\r\n", "gini = 0.0\r\n", "samples = 2\r\n", "value = [2, 0]\r\n", "\r\n", "\r\n", "6->8\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "\r\n" ], "text/plain": [ "" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "graphviz.Source(dot_graph)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第5章决策树-习题\n", "\n", "### 习题5.1\n", "根据表5.1所给的训练数据集，利用信息增益比（C4.5算法）生成决策树。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "\n", "表5.1 贷款申请样本数据表 \n", "\n", "ID | 年龄 | 有工作 | 有自己的房子 | 信贷情况 | 类别\n", ":-: | :-: | :-: | :-: | :-: | :-: \n", "1 | 青年 | 否 | 否 | 一般 | 否\n", "2 | 青年 | 否 | 否 | 好 | 否\n", "3 | 青年 | 是 | 否 | 好 | 是\n", "4 | 青年 | 是 | 是 | 一般 | 是\n", "5 | 青年 | 否 | 否 | 一般 | 否\n", "6 | 中年 | 否 | 否 | 一般 | 否\n", "7 | 中年 | 否 | 否 | 好 | 否\n", "8 | 中年 | 是 | 是 | 好 | 是\n", "9 | 中年 | 否 | 是 | 非常好 | 是\n", "10 | 中年 | 否 | 是 | 非常好 | 是\n", "11 | 老年 | 否 | 是 | 非常好 | 是\n", "12 | 老年 | 否 | 是 | 好 | 是\n", "13 | 老年 | 是 | 否 | 好 | 是\n", "14 | 老年 | 是 | 否 | 非常好 | 是\n", "15 | 老年 | 否 | 否 | 一般 | 否" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ "\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "Tree\r\n", "\r\n", "\r\n", "0\r\n", "\r\n", "有自己的房子 ≤ 3.0\r\n", "gini = 0.48\r\n", "samples = 15\r\n", "value = [6, 9]\r\n", "class = 1\r\n", "\r\n", "\r\n", "1\r\n", "\r\n", "有工作 ≤ 3.0\r\n", "gini = 0.444\r\n", "samples = 9\r\n", "value = [6, 3]\r\n", "class = 0\r\n", "\r\n", "\r\n", "0->1\r\n", "\r\n", "\r\n", "True\r\n", "\r\n", "\r\n", "4\r\n", "\r\n", "gini = 0.0\r\n", "samples = 6\r\n", "value = [0, 6]\r\n", "class = 1\r\n", "\r\n", "\r\n", "0->4\r\n", "\r\n", "\r\n", "False\r\n", "\r\n", "\r\n", "2\r\n", "\r\n", "gini = 0.0\r\n", "samples = 6\r\n", "value = [6, 0]\r\n", "class = 0\r\n", "\r\n", "\r\n", "1->2\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "3\r\n", "\r\n", "gini = 0.0\r\n", "samples = 3\r\n", "value = [0, 3]\r\n", "class = 1\r\n", "\r\n", "\r\n", "1->3\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "\r\n" ], "text/plain": [ "" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.tree import DecisionTreeClassifier\n", "from sklearn import preprocessing\n", "import numpy as np\n", "import pandas as pd\n", "\n", "from sklearn import tree\n", "import graphviz\n", "\n", "features = [\"年龄\", \"有工作\", \"有自己的房子\", \"信贷情况\"]\n", "X_train = pd.DataFrame([\n", " [\"青年\", \"否\", \"否\", \"一般\"],\n", " [\"青年\", \"否\", \"否\", \"好\"],\n", " [\"青年\", \"是\", \"否\", \"好\"],\n", " [\"青年\", \"是\", \"是\", \"一般\"],\n", " [\"青年\", \"否\", \"否\", \"一般\"],\n", " [\"中年\", \"否\", \"否\", \"一般\"],\n", " [\"中年\", \"否\", \"否\", \"好\"],\n", " [\"中年\", \"是\", \"是\", \"好\"],\n", " [\"中年\", \"否\", \"是\", \"非常好\"],\n", " [\"中年\", \"否\", \"是\", \"非常好\"],\n", " [\"老年\", \"否\", \"是\", \"非常好\"],\n", " [\"老年\", \"否\", \"是\", \"好\"],\n", " [\"老年\", \"是\", \"否\", \"好\"],\n", " [\"老年\", \"是\", \"否\", \"非常好\"],\n", " [\"老年\", \"否\", \"否\", \"一般\"]\n", "])\n", "y_train = pd.DataFrame([\"否\", \"否\", \"是\", \"是\", \"否\", \n", " \"否\", \"否\", \"是\", \"是\", \"是\", \n", " \"是\", \"是\", \"是\", \"是\", \"否\"])\n", "# 数据预处理\n", "le_x = preprocessing.LabelEncoder()\n", "le_x.fit(np.unique(X_train))\n", "X_train = X_train.apply(le_x.transform)\n", "le_y = preprocessing.LabelEncoder()\n", "le_y.fit(np.unique(y_train))\n", "y_train = y_train.apply(le_y.transform)\n", "# 调用sklearn.DT建立训练模型\n", "model_tree = DecisionTreeClassifier()\n", "model_tree.fit(X_train, y_train)\n", "\n", "# 可视化\n", "dot_data = tree.export_graphviz(model_tree, out_file=None,\n", " feature_names=features,\n", " class_names=[str(k) for k in np.unique(y_train)],\n", " filled=True, rounded=True,\n", " special_characters=True)\n", "graph = graphviz.Source(dot_data)\n", "graph" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题5.2\n", "  已知如表5.2所示的训练数据，试用平方误差损失准则生成一个二叉回归树。 \n", "表5.2 训练数据表 \n", "\n", "| $x_i$ | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | \n", "| - | - | - | - | - | - | - | - | - | - | - | \n", "| $y_i$ | 4.50 | 4.75 | 4.91 | 5.34 | 5.80 | 7.05 | 7.90 | 8.23 | 8.70 | 9.00" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "  决策树的生成就是递归地构建二叉决策树的过程，对回归树用平方误差最小化准则，对分类树用基尼指数（Gini index）最小化准则，进行特征选择，生成二叉树。 \n", "> 算法5.5（最小二乘回归树生成算法） \n", "输入：训练数据集$D$ \n", "输出：回归树$f(x)$ \n", "在训练数据集所在的输入空间中，递归地将每个区域划分为两个子区域并决定每个子区域上的输出值，构建二叉决策树； \n", "(1)选择最优切分变量$j$与切分点$s$，求解$$\\min_{j,s} \\left[ \\min_{c_1} \\sum_{x_i \\in R_1(j,s)} (y_i - c_1)^2 + \\min_{c_2} \\sum_{x_i \\in R_2(j,s)} (y_i - c_2)^2\\right]$$遍历变量$j$，对固定的切分变量$j$扫描切分点$s$，选择使得上式达到最小值的对$(j,s)$ \n", "(2)用选定的对$(j,s)$划分区域并决定相应的输出值：$$R_1(j,s)=\\{x|x^{(j)}\\leqslant s\\}, R_2(j,s)=\\{x|x^{(j)} > s\\} \\\\ \n", "\\hat{c_m} = \\frac{1}{N_m} \\sum_{x_i \\in R_m(j,s)} y_i, x \\in R_m, m=1,2 $$\n", "(3)继续对两个子区域调用步骤(1),(2)，直至满足停止条件 \n", "(4)将输入空间划分为$M$个区域$R_1,R_2,\\cdots,R_M$，生成决策树：$$f(x)=\\sum_{m=1}^M \\hat{c_m} I(x \\in R_m)$$" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "\n", "class LeastSqRTree:\n", " def __init__(self, train_X, y, epsilon):\n", " # 训练集特征值\n", " self.x = train_X\n", " # 类别\n", " self.y = y\n", " # 特征总数\n", " self.feature_count = train_X.shape[1]\n", " # 损失阈值\n", " self.epsilon = epsilon\n", " # 回归树\n", " self.tree = None\n", "\n", " def _fit(self, x, y, feature_count, epsilon):\n", " # 选择最优切分点变量j与切分点s\n", " (j, s, minval, c1, c2) = self._divide(x, y, feature_count)\n", " # 初始化树\n", " tree = {\"feature\": j, \"value\": x[s, j], \"left\": None, \"right\": None}\n", " if minval < self.epsilon or len(y[np.where(x[:, j] <= x[s, j])]) <= 1:\n", " tree[\"left\"] = c1\n", " else:\n", " tree[\"left\"] = self._fit(x[np.where(x[:, j] <= x[s, j])],\n", " y[np.where(x[:, j] <= x[s, j])],\n", " self.feature_count, self.epsilon)\n", " if minval < self.epsilon or len(y[np.where(x[:, j] > s)]) <= 1:\n", " tree[\"right\"] = c2\n", " else:\n", " tree[\"right\"] = self._fit(x[np.where(x[:, j] > x[s, j])],\n", " y[np.where(x[:, j] > x[s, j])],\n", " self.feature_count, self.epsilon)\n", " return tree\n", "\n", " def fit(self):\n", " self.tree = self._fit(self.x, self.y, self.feature_count, self.epsilon)\n", "\n", " @staticmethod\n", " def _divide(x, y, feature_count):\n", " # 初始化损失误差\n", " cost = np.zeros((feature_count, len(x)))\n", " # 公式5.21\n", " for i in range(feature_count):\n", " for k in range(len(x)):\n", " # k行i列的特征值\n", " value = x[k, i]\n", " y1 = y[np.where(x[:, i] <= value)]\n", " c1 = np.mean(y1)\n", " y2 = y[np.where(x[:, i] > value)]\n", " c2 = np.mean(y2)\n", " y1[:] = y1[:] - c1\n", " y2[:] = y2[:] - c2\n", " cost[i, k] = np.sum(y1 * y1) + np.sum(y2 * y2)\n", " # 选取最优损失误差点\n", " cost_index = np.where(cost == np.min(cost))\n", " # 选取第几个特征值\n", " j = cost_index[0][0]\n", " # 选取特征值的切分点\n", " s = cost_index[1][0]\n", " # 求两个区域的均值c1,c2\n", " c1 = np.mean(y[np.where(x[:, j] <= x[s, j])])\n", " c2 = np.mean(y[np.where(x[:, j] > x[s, j])])\n", " return j, s, cost[cost_index], c1, c2" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\numpy\\core\\fromnumeric.py:3335: RuntimeWarning: Mean of empty slice.\n", " out=out, **kwargs)\n", "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\numpy\\core\\_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars\n", " ret = ret.dtype.type(ret / rcount)\n" ] }, { "data": { "text/plain": [ "{'feature': 0,\n", " 'value': 5,\n", " 'left': {'feature': 0, 'value': 3, 'left': 4.72, 'right': 5.57},\n", " 'right': {'feature': 0,\n", " 'value': 7,\n", " 'left': {'feature': 0, 'value': 6, 'left': 7.05, 'right': 7.9},\n", " 'right': {'feature': 0, 'value': 8, 'left': 8.23, 'right': 8.85}}}" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_X = np.array([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]]).T\n", "y = np.array([4.50, 4.75, 4.91, 5.34, 5.80, 7.05, 7.90, 8.23, 8.70, 9.00])\n", "\n", "model_tree = LeastSqRTree(train_X, y, .2)\n", "model_tree.fit()\n", "model_tree.tree" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "根据上面程序的输出，可得到用平方误差损失准则生成一个二叉回归树：$$f(x)=\\begin{cases}\n", "4.72 & x \\le 3\\\\\n", "5.57 & 3 < x \\le 5\\\\\n", "7.05 & 5 < x \\le 6\\\\\n", "7.9 & 6 < x \\le 7 \\\\\n", "8.23 & 7 < x \\le 8\\\\\n", "8.85 & x > 8\\\\\n", "\\end{cases}$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题5.3\n", "\n", "  证明 CART 剪枝算法中，当$\\alpha$确定时，存在唯一的最小子树$T_{\\alpha}$使损失函数$C_{\\alpha}(T)$最小。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "**第1步：**内部节点是否剪枝只与以该节点为根节点的子树有关。 \n", "剪枝过程： \n", "计算子树的损失函数：$$C_{\\alpha}(T)=C(T)+\\alpha$$其中，$\\displaystyle C(T) = \\sum_{t=1}^{|T|}N_t (1 - \\sum_{k=1}^K (\\frac{N_{tk}}{N_t})^2)$，$|T|$是叶结点个数，$K$是类别个数。 \n", "有剪枝前子树$T_0$，剪枝后子树$T_1$，满足$C_{\\alpha}(T_1) \\leqslant C_{\\alpha}(T_0)$则进行剪枝。 \n", "\n", "----\n", "\n", "**第2步（反证法）：**假设当$\\alpha$确定时，存在两颗子树$T_1,T_2$都使得损失函数$C_{\\alpha}$最小。 \n", "第1种情况：假设被剪枝的子树在同一边，易知其中一个子树会由另一个子树剪枝而得到，故不可能存在两个最优子树，原结论得证。 \n", "第2种情况：假设被剪枝的子树不在同一边，易知被剪枝掉的子树都可以使损失函数$C_{\\alpha}$最小，故两颗子树都可以继续剪枝，故不可能存在两个最优子树，原结论得证。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题5.4\n", "\n", "  证明 CART 剪枝算法中求出的子树序列$\\{T_0,T_1,\\cdots,T_n\\}$分别是区间$\\alpha \\in [\\alpha_i,\\alpha_{i+1})$的最优子树$T_{\\alpha}$，这里$i=0,1,\\cdots,n,0=\\alpha_0 < \\alpha_1 < \\cdots, \\alpha_n < +\\infty$。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "原结论可以表述为：将$\\alpha$从小增大，$0=\\alpha_0<\\alpha_1<\\cdots<\\alpha_n < +\\infty$，在每个区间$[\\alpha_i,\\alpha_{i+1})$中，子树$T_i$是这个区间里最优的。 \n", "**第1步：**易证，当$\\alpha=0$时，整棵树$T_0$是最优的，当$\\alpha \\rightarrow +\\infty$时，根结点组成的单结点树（即$T_n$）是最优的。\n", "\n", "----\n", "\n", "**第2步：** \n", "  由于每次剪枝剪的都是某个内部结点的子结点，也就是将某个内部结点的所有子结点回退到这个内部结点里，并将这个内部结点作为叶子结点。因此在计算整体的损失函数时，这个内部结点以外的值都没变，只有这个内部结点的局部损失函数改变了，因此本来需要计算全局的损失函数，但现在只需要计算内部结点剪枝前和剪枝后的损失函数。 \n", "从整体树$T_0$开始剪枝，对$T_0$的任意内部结点$t$ \n", "剪枝前的状态：有$|T_t|$个叶子结点，预测误差是$C(T_t)$ \n", "剪枝后的状态：只有本身一个叶子结点，预测误差是$C(t)$\n", "因此剪枝前的以$t$结点为根结点的子树的损失函数是$$C_{\\alpha}(T_t) = C(T_t) + \\alpha|T_t|$$剪枝后的损失函数是$$C_{\\alpha}(t) = C(t) + \\alpha$$易得，一定存在一个$\\alpha$使得$C_{\\alpha}(T_t) = C_{\\alpha}(t)$，这个值为$$\\alpha=\\frac{C(t)-C(T_t)}{|T_t|-1}$$可知，找到$\\alpha$即找到了子结点$t$，即完成了剪枝，得到最优子树$T_1$ \n", "根据书中第73页，采用以下公式计算剪枝后整体损失函数减少的程度：$$g(t)=\\frac{C(t)-C(T_t)}{|T_t|-1}$$在$T_0$中剪去$g(t)$最小的$T_t$，将得到的子树作为$T_1$，同时将最小的$g(t)$设为$\\alpha_1$，$T_1$为区间$[\\alpha_1,\\alpha_2)$的最优子树。 \n", "依次类推，子树$T_i$是区间$[\\alpha_i,\\alpha_{i+1})$里最优的，原结论得证。\n", "\n", "----\n", "\n", "**参考文献：** \n", "1. MrTriste：https://blog.csdn.net/wjc1182511338/article/details/76793164\n", "2. http://www.pianshen.com/article/1752163397/\n", "\n", "----\n", "\n", "**讨论：**为什么$\\alpha$要取最小的$g(t)$呢？ \n", "
\n", "
图5.1 最小的$g(t)$
\n", "  以图中两个点为例，结点1和结点2，$g(t)_2$大于$g(t)_1$，假设在所有结点中$g(t)_1$最小，$g(t)_2$最大，两种选择方法：当选择最大值$g(t)_2$，即结点2进行剪枝，但此时结点1的剪枝前的误差大于剪枝后的误差，即如果不剪枝，误差变大，依次类推，对其它所有的结点的$g(t)$都是如此，从而造成整体的累计误差更大。反之，如果选择最小值$g(t)_1$，即结点1进行剪枝，则其余结点不剪的误差要小于剪枝后的误差，不剪枝为好，且整体的误差最小。从而以最小$g(t)$剪枝获得的子树是该$\\alpha$值下的最优子树。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "参考代码：https://github.com/wzyonggege/statistical-learning-method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第06章 逻辑斯谛回归/6.LogisticRegression.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第6章 逻辑斯谛回归" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "逻辑斯谛回归(LR)是经典的分类方法\n", "\n", "1．逻辑斯谛回归模型是由以下条件概率分布表示的分类模型。逻辑斯谛回归模型可以用于二类或多类分类。\n", "\n", "$$P(Y=k | x)=\\frac{\\exp \\left(w_{k} \\cdot x\\right)}{1+\\sum_{k=1}^{K-1} \\exp \\left(w_{k} \\cdot x\\right)}, \\quad k=1,2, \\cdots, K-1$$\n", "\n", "$$P(Y=K | x)=\\frac{1}{1+\\sum_{k=1}^{K-1} \\exp \\left(w_{k} \\cdot x\\right)}$$\n", "这里，$x$为输入特征，$w$为特征的权值。\n", "\n", "逻辑斯谛回归模型源自逻辑斯谛分布，其分布函数$F(x)$是$S$形函数。逻辑斯谛回归模型是由输入的线性函数表示的输出的对数几率模型。\n", "\n", "2．最大熵模型是由以下条件概率分布表示的分类模型。最大熵模型也可以用于二类或多类分类。\n", "\n", "$$P_{w}(y | x)=\\frac{1}{Z_{w}(x)} \\exp \\left(\\sum_{i=1}^{n} w_{i} f_{i}(x, y)\\right)$$\n", "$$Z_{w}(x)=\\sum_{y} \\exp \\left(\\sum_{i=1}^{n} w_{i} f_{i}(x, y)\\right)$$\n", "\n", "其中，$Z_w(x)$是规范化因子，$f_i$为特征函数，$w_i$为特征的权值。\n", "\n", "3．最大熵模型可以由最大熵原理推导得出。最大熵原理是概率模型学习或估计的一个准则。最大熵原理认为在所有可能的概率模型（分布）的集合中，熵最大的模型是最好的模型。\n", "\n", "最大熵原理应用到分类模型的学习中，有以下约束最优化问题：\n", "\n", "$$\\min -H(P)=\\sum_{x, y} \\tilde{P}(x) P(y | x) \\log P(y | x)$$\n", "\n", "$$s.t. \\quad P\\left(f_{i}\\right)-\\tilde{P}\\left(f_{i}\\right)=0, \\quad i=1,2, \\cdots, n$$\n", " \n", " $$\\sum_{y} P(y | x)=1$$\n", " \n", "求解此最优化问题的对偶问题得到最大熵模型。\n", "\n", "4．逻辑斯谛回归模型与最大熵模型都属于对数线性模型。\n", "\n", "5．逻辑斯谛回归模型及最大熵模型学习一般采用极大似然估计，或正则化的极大似然估计。逻辑斯谛回归模型及最大熵模型学习可以形式化为无约束最优化问题。求解该最优化问题的算法有改进的迭代尺度法、梯度下降法、拟牛顿法。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "回归模型：$f(x) = \\frac{1}{1+e^{-wx}}$\n", "\n", "其中wx线性函数：$wx =w_0\\cdot x_0 + w_1\\cdot x_1 + w_2\\cdot x_2 +...+w_n\\cdot x_n,(x_0=1)$\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from math import exp\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "from sklearn.datasets import load_iris\n", "from sklearn.model_selection import train_test_split" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# data\n", "def create_data():\n", " iris = load_iris()\n", " df = pd.DataFrame(iris.data, columns=iris.feature_names)\n", " df['label'] = iris.target\n", " df.columns = ['sepal length', 'sepal width', 'petal length', 'petal width', 'label']\n", " data = np.array(df.iloc[:100, [0,1,-1]])\n", " # print(data)\n", " return data[:,:2], data[:,-1]" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "X, y = create_data()\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "class LogisticReressionClassifier:\n", " def __init__(self, max_iter=200, learning_rate=0.01):\n", " self.max_iter = max_iter\n", " self.learning_rate = learning_rate\n", "\n", " def sigmoid(self, x):\n", " return 1 / (1 + exp(-x))\n", "\n", " def data_matrix(self, X):\n", " data_mat = []\n", " for d in X:\n", " data_mat.append([1.0, *d])\n", " return data_mat\n", "\n", " def fit(self, X, y):\n", " # label = np.mat(y)\n", " data_mat = self.data_matrix(X) # m*n\n", " self.weights = np.zeros((len(data_mat[0]), 1), dtype=np.float32)\n", "\n", " for iter_ in range(self.max_iter):\n", " for i in range(len(X)):\n", " result = self.sigmoid(np.dot(data_mat[i], self.weights))\n", " error = y[i] - result\n", " self.weights += self.learning_rate * error * np.transpose(\n", " [data_mat[i]])\n", " print('LogisticRegression Model(learning_rate={},max_iter={})'.format(\n", " self.learning_rate, self.max_iter))\n", "\n", " # def f(self, x):\n", " # return -(self.weights[0] + self.weights[1] * x) / self.weights[2]\n", "\n", " def score(self, X_test, y_test):\n", " right = 0\n", " X_test = self.data_matrix(X_test)\n", " for x, y in zip(X_test, y_test):\n", " result = np.dot(x, self.weights)\n", " if (result > 0 and y == 1) or (result < 0 and y == 0):\n", " right += 1\n", " return right / len(X_test)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "LogisticRegression Model(learning_rate=0.01,max_iter=200)\n" ] } ], "source": [ "lr_clf = LogisticReressionClassifier()\n", "lr_clf.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9666666666666667" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lr_clf.score(X_test, y_test)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU9b3/8dfXJJCwRnbIQiBRlE02RUTZtKXixlWqUGuL1XJFuLa11169t4+21/Z31fr72VsNqCjVuuFClWrd2pqAiIAmoqAgMglLErYQSNiSkEw+vz8SFMcsJ8mZOd8583k+HjySOXP4zuc7Ix/PnPOe7xgRQSmlVPQ7zesClFJKuUMbulJK+YQ2dKWU8glt6Eop5RPa0JVSyifivXrgXr16SUZGhlcPr5RSUSk/P/+AiPRu7D7HDd0YEwfkASUicnnIfXOB+4GShk3ZIvJ4c+NlZGSQl5fn9OGVUkoBxpidTd3XmiP0nwBbgG5N3P+CiCxsTWFKKaXc4+gcujEmFbgMaPaoWymllHecXhT9X+AXQF0z+1xjjNlojFlujElrbAdjzDxjTJ4xJq+0tLS1tSqllGpGi6dcjDGXA/tFJN8YM6WJ3V4DlolItTHmFuDPwLTQnURkCbAEYNy4cd9Yc6Cmpobi4mKqqqpaMYXIS0xMJDU1lYSEBK9LUUqpLzk5hz4RuNIYMwNIBLoZY54Rke+f3EFEyk7Z/zHgvrYUU1xcTNeuXcnIyMAY05Yhwk5EKCsro7i4mEGDBnldjlJKfanFUy4icpeIpIpIBjAbyDm1mQMYY/qfcvNK6i+etlpVVRU9e/a0tpkDGGPo2bOn9e8ilFKxp805dGPM3UCeiLwK3GaMuRKoBQ4Cc9sxblv/asREQ41KqdjTqoYuIiuBlQ2//+qU7XcBd7lZmFLRbMWGEu5/eyu7yysZkJzEHdOHMHN0itdlKZ/Tj/434q233mLIkCFkZWVx7733el2OijIrNpRw18ubKCmvRICS8kruenkTKzaUtPh3lWoPbeghgsEgCxYs4M0332Tz5s0sW7aMzZs3e12WiiL3v72Vyprg17ZV1gS5/+2tHlWkYoVna7m4IRxvaz/44AOysrIYPHgwALNnz+avf/0rQ4cOdaNkFQN2l1e2artSbonaI/Rwva0tKSkhLe2rz0WlpqZSUqJvlZVzA5KTWrVdKbdEbUMP19vaxr5jVVMtqjXumD6EpIS4r21LSojjjulDPKpIxYqoPeUSrre1qampFBUVfXm7uLiYAQMGtGtMFVtOnvbTlIuKtKht6AOSkyhppHm3923tueeey7Zt29i+fTspKSk8//zzPPfcc+0aU8WemaNTtIGriIvaUy7helsbHx9PdnY206dP5+yzz+baa69l2LBh7RpTKaUiIWqP0MP5tnbGjBnMmDGj3eMopVQkRW1DB31bq5RSp4raUy5KKaW+Thu6Ukr5hDZ0pZTyCW3oSinlE9rQlVLKJ7Shh/jRj35Enz59GD58uNelKKVUq2hDDzF37lzeeustr8tQEbZiQwkT781h0J2vM/HeHF27XEWl6G7oG1+EPwyH3yTX/9z4YruHnDRpEj169HChOBUt9AsplF9Eb0Pf+CK8dhtUFAFS//O121xp6iq26BdSKL+I3ob+zt1QE7I4V01l/XalWkG/kEL5RfQ29Iri1m1Xqgn6hRTKL6K3oXdPbd12pZqgX0ih/CJ6G/rFv4KEkCOohKT67e0wZ84cJkyYwNatW0lNTWXp0qXtGk/Zb+boFO65egQpyUkYICU5iXuuHqELv6moE72rLY68tv7nO3fXn2bpnlrfzE9ub6Nly5a5UJyKNrpypwqX6togf8kv4Yy+XTg3I7wJuuht6FDfvNvZwJVdVmwo0a9uU75QeSLIsg92seTdQvYeruIHEwZqQ1ex42Qe/GSE8GQeHNCmrqLGkaoanlm3i8dXF1J27ATjB/Xg/u+O5MKsXmF/bOsauohgjPG6jGaJiNcl+FJzeXBt6Mp25cdP8MSaHTz5/g4qKmuYdGZvFk7N4rxBkfugolUNPTExkbKyMnr27GltUxcRysrKSExM9LoU39E8uIpGpUeqWfredp5eu4NjJ4J8e2hfFkzN4py05IjXYlVDT01Npbi4mNLSUq9LaVZiYiKpqRqPdNuA5CRKGmnemgdXNtpTUcmjqwpZ9sEuaoJ1XDZyAAumZnJWv26e1WRVQ09ISGDQoEFel6E8csf0IV87hw6aB1f22VV2nIdXFbA8vwgR+JfRKcyfksng3l28Ls2uhq5i28nz5JpyUTYK7D/C4twC/vrJbuKM4bpz0/jXSZmk9ejkdWlfctzQjTFxQB5QIiKXh9zXEXgKGAuUAdeJyA4X61QxQvPgyjaf7a5gUW6ANz/dS2J8HDdekMGPJw2mbzf7rqO15gj9J8AWoLETRDcBh0QkyxgzG7gPuM6F+pTyNc3d2+ujXYdYlBPgnc/307VjPLdOyeRHEwfRs0tHr0trkqOGboxJBS4D/g9weyO7XAX8puH35UC2McaI5vuUapLm7u0jIqwrPEh27jbWBMpI7pTAz791Jj+4IIPuSQlel9cip0fo/wv8AujaxP0pQBGAiNQaYyqAnsCBdleolE9p7t4eIsLKL0pZlBMgb+chenXpyH/OOIvrxw+kc8foudTYYqXGmMuB/SKSb4yZ0tRujWz7xtG5MWYeMA8gPT29FWUq5T+au/deXZ3w9837yM7dxqclhxnQPZG7rxrGtePSSAxZgTMaOPlfz0TgSmPMDCAR6GaMeUZEvn/KPsVAGlBsjIkHugMHQwcSkSXAEoBx48bp6RgV0zR3753aYB2vb9rDotwAX+w7SkbPTvz+mpHMHJ1Ch/joXYS2xYYuIncBdwE0HKH/e0gzB3gV+CGwFpgF5Oj5c6Wap7n7yDtRW8eKDSUsXhlgR9lxzuzbhT/OHsVlI/oTHxe9jfykNp8cMsbcDeSJyKvAUuBpY0yA+iPz2S7Vp5Rvae4+cqpqgryYV8SjqwopKa9keEo3Hvn+WL49tC+nnWbnMiNtYbw6kB43bpzk5eV58thKqdhwrLqW59bvYsnqQkqPVDN24OksnJbFlDN7W7teVEuMMfkiMq6x+6Ln8q1SbfDLFZtYtr6IoAhxxjBnfBq/mznC67JUmFVU1vDU+zv405rtHDpew8Ssnjw4ezTnD+4RtY3cCW3oyrd+uWITz6zb9eXtoMiXt7Wp+9PBYyf403vb+fP7OzhSXcvFZ/VhwbQsxqSf7nVpEaENXfnWsvVFTW7Xhu4v+w9XseTdQp5dv4uq2iCXDu/HrVOyGJ7S3evSIkobuvKtYBPXh5rarqJP8aHjPLqqkBfyigjWCVedM4Bbp2aS1aepz0D6mzZ05VtxxjTavON8fA41Vmw/cIzFuQFe2VCCMTBrbBrzJ2eS3tOelQ+9oA1d+dac8WlfO4d+6nYVnbbuPcKi3AB/27ibhLjT+P75A5k3abB+GKuBNnTlWyfPk2vKJfptLC4nOyfA3zfvo3OHOOZNyuSmCwfRu6u9Kx96QXPoSilrfbjjINk5AVZ9UUq3xHhunDiIGydmkNypg9eleUZz6Mpz1z+2ljUFXy3vMzGzB8/+eIKHFYWXrnPediLCmkAZD+VsY/32g/Ts3IH/+M5ZfP/8dLom2r+ErZe0oauwC23mAGsKDnL9Y2t92dR1nfO2ERHe2bKfh3IDfFJUTr9uifzq8qHMOS+dpA7Rt/KhF7Shq7ALbeYtbY92us556wTrhLc+3Ut2boAtew6T1iOJ//mXEVwzNoWO8drIW0MbulIu03XOnakJ1vHqx7tZtDJAYekxBvfuzAPXnsOV5wzwxcqHXtCGrpTLdJ3z5lXXBlmeX8wjqwooOljJWf26suh7Y/jO8H7E+WjlQy9oQ1dhNzGzR6OnVyZm9vCgmvDTdc4bV3kiyLIPdrHk3UL2Hq5iVFoyv7liGNPO6uPrBbMiSRu6CrtnfzwhplIuus751x2pquGZdbt4fHUhZcdOMH5QD/7vd89hYlZPbeQu0xy6Uiosyo+f4Ik1O3hizXYOV9Uy+czeLJyWxbkZ/nxnFimaQ1eeczuX7XQ8zYNHXumRah5/r5Bn1u7k2Ikg3x7al4XTshiZmux1ab6nDV2Fndu5bKfjaR48svZUVPLoqkKWfbCLmmAdl48cwIKpWQzpF5srH3pBG7oKO7dz2U7H0zx4ZOwqO87DqwIszy9GBK4ek8L8KVkM6tXZ69JijjZ0FXZu57Kdjqd58PAK7D/C4twC/vrJbuJOM8w+N51/nTyY1NNjewlbL2lDV2Hndi7b6XiaBw+Pz3ZXsCg3wJuf7iUxPo4fTczgxxcNpk+3RK9Li3n6cSwVdndMH0JSwtc/wt2eXLbT8dx+3Fj30a5D3PTkh1z24Hus/uIAC6ZksebOafzXZUO1mVtCj9BV2Lmdy3Y6nubB209EWFd4kOzcbawJlHF6pwT+/dtncsOEDLon6cqHttEculLqG0SElV+UsignQN7OQ/Tu2pF5Fw3me+PT6dxRjwO9pDn0KOZVjlpz3rGprk74++Z9ZOdu49OSw6QkJ/Hbq4bx3XFpJCboyoe204ZuMa9y1Jrzjj21wTpe37SHRbkBvth3lIyenfj9rJHMHJVCh3i91BYt9JWyWHM5ahse16v6lHtO1Nbx4odFXPLAKn7y/McA/HH2KP55+2SuHZemzTzK6BG6xbzKUWvO2/+qaoK8mFfEIysL2F1RxYiU7jx6w1i+dXZfTtMlbKOWNnSLeZWj1py3fx2rruXZ9Tt5bPV2So9UM27g6fzP1SOYfGZvXfnQB/T9lMW8ylFrztt/KipreOidbUy8L4f/eeNzhvTtyvPzzuelWyYwZYiuR+4XeoRuMa9y1Jrz9o+yo9X8ac12nnp/J0eqa7nk7D4smJrF6PTTvS5NhYHm0JXyoX2Hq3js3UKeXb+LqtogM4b359apmQwb0N3r0lQ7tSuHboxJBN4FOjbsv1xEfh2yz1zgfqCkYVO2iDzenqJVdPjlik0sW19EUIQ4Y5gzPo3fzRzR5v1sz93brvjQcR5dVcgLeUUE64SrRg3g1ilZZPXp4nVpKgKcnHKpBqaJyFFjTALwnjHmTRFZF7LfCyKy0P0Sla1+uWITz6zb9eXtoMiXt09t1k73sz13b7PC0qM8vLKAVzaUYAzMGpvG/MmZpPfUlQ9jSYsXRaXe0YabCQ1/vDlPo6yybH2Ro+1O97M9d2+jz/ce5t+WbeCSB1bx2sbd3DBhIO/+Yir3XD1Cm3kMcnRR1BgTB+QDWcAiEVnfyG7XGGMmAV8APxORb/wrNsbMA+YBpKent7loZYdgE9dfQrc73c/23L1NNhaXk50T4O+b99G5QxzzJmVy04WD6N21o9elKQ85augiEgRGGWOSgVeMMcNF5NNTdnkNWCYi1caYW4A/A9MaGWcJsATqL4q2u3rlqThjGm3WcSEROKf72Z67t8GHOw7yUE6Ad78opXtSAj+95AzmXpBBcqcOXpemLNCqHLqIlAMrge+EbC8TkeqGm48BY12pTlltzvg0R9ud7md77t4rIsLqbaVc9+havvvIWjbvruDOS89izZ3T+OklZ2ozV19yknLpDdSISLkxJgm4BLgvZJ/+IrKn4eaVwBbXK1XWOXlBs6X0itP9bM/dR5qI8M6W/TyUG+CTonL6dUvk11cMZfa56SR10JUP1Te1mEM3xoyk/hRKHPVH9C+KyN3GmLuBPBF51RhzD/WNvBY4CMwXkc+bG1dz6Eo1LlgnvPnpHrJzAny+9whpPZK4dUoWV49JoWO8NvJY11wOXT9YZDm389FO8+Buj+fV+upuzzecaoJ1vPrxbhatDFBYeozM3p1ZMDWLK88ZQHycw7OjG1+Ed+6GimLongoX/wpGXhvewlVE6RdcRCm389FO8+Buj+fV+upuzzdcqmuDLM8v5uGVBRQfquTs/t1YfP0Ypg/rR1xrVj7c+CK8dhvUNFzgrSiqvw3a1GOELs5lMbfz0U7z4G6P59X66m7P122VJ4IsfW87k36fy3+98im9unRk6Q/H8cZtFzJjRP/WNXOoPzKvCUnr1FTWb1cxQY/QLeZ2PtppHtzt8bxaX93t+brlSFUNT6/bydLV2yk7doLzB/fggWtHcUFmz/atelhR3Lrtyne0oVvM7Xy00zy42+N5tb662/Ntr/LjJ/jTmh08uWY7h6tqmTKkNwunZjEuo4c7D9A9tf40S2PbVUzQUy4Wczsf7TQP7vZ4Xq2v7vZ826r0SDX3vLmFiffm8OA725iQ2ZPXFl7Ikzee514zh/oLoAkh//NLSKrfrmKCHqFbzO18tNM8uNvjebW+utvzba3d5ZUsebeQZR/soiZYxxXn1K98OKRf1/A84MkLn5pyiVkaW1TKZTvLjvHIqgKW5xcjAlePSWH+lCwG9ersdWnKBzS2qMLGq3y5jbbtO8LilQX89eMS4uNOY8556cybNJjU02Ng1UPNv1tBG7pqM6/y5bb5tKSCRbkB3vpsL0kJcdx04SB+fNFg+nRL9Lq0yND8uzW0oas2ay43fmqjdrpftMnfeYhFuQFyPt9P147xLJyaxY0TB9Gjc4wtltVc/l0bekRpQ1dt5lW+3EsiwtrCMrJzArxfUMbpnRK4Y/oQbpgwkG6JCV6X5w3Nv1tDG7pqM6/y5V4QEVZuLSU7N0D+zkP07tqRX152Nt8bn06nDjH+z0jz79bQHLpqM6/y5ZFUVye89ekeLn/oPW588kP2VlTx25nDWf2Lqdx80WBt5qD5d4vof42qzbzKl0dCbbCOv23cw6LcANv2H2VQr878ftZI/mV0CglOVz6MFZp/t4bm0JU6xYnaOl7ZUMzilQXsLDvOkL5dWTAti8vasliWUmGgOfQIcjtv7XQ8r9b99ku+vKomyAsfFvHoqgJ2V1QxIqU7j94wlm+d3ZfTbGjkfsp5+2kuTkRwvtrQXeR23trpeF6t++2HfPnR6lqeXbeTx1Zv58DRas7NOJ17rhnJpDN6tW/lQzf5Keftp7k4EeH56slAF7m9nrfT8bxa99vt+UZSRWUND76zjQvvy+GeNz/n7P5deWHe+bx0ywVMPrO3Pc0c/LXOuZ/m4kSE56tH6C5yO2/tdDyv1v2Oxnx52dFqlr63nafX7uRIdS2XnN2HBVOzGJ1+utelNc1POW8/zcWJCM9XG7qL3M5bOx3Pq3W/oylfvu9wFUveLeS59buoqg0yY0R/FkzJYuiAbl6X1jI/5bz9NBcnIjxfPeXiIrfz1k7H82rd72jIlxcdPM5/vbKJi+7L5cn3d3DpiH7842eTWfS9MdHRzMFfOW8/zcWJCM9Xj9Bd5Hbe2ul4Xq37bXO+vLD0KItXFvDKhhLijGHWuFRumZRJes8oXPnQTzlvP83FiQjPV3Poyle27DnMotwAr2/aQ8f4r5aw7d/dvtNASrWF5tCjWKzl2tvqk6JysnMD/GPzPrp0jOeWyZncdOEgenXp6HVpyhZ/ux3ynwQJgomDsXPh8gfaPp6FeXpt6BaLtVx7W3yw/SAP5Wxj9bYDdE9K4GeXnMncCzLo3ilGVz5Ujfvb7ZC39KvbEvzqdluauqV5er0oarFYy7U7JSK8+0Up1z66lmsfXcuWPYe589KzWHPnNH5yyRnazNU35T/Zuu0tsTRPr0foFou1XHtL6uqEdz7fT3bONj4prqB/90R+c8VQrjs3naQOcS0PoGKXBFu3vSWW5um1oVss1nLtTQnWCW9sql/58PO9R0jrkcQ9V4/g6jEpdIzXRq4cMHGNN2/Txv9+LM3T6ykXi8Varj1UTbCO5fnFfOuBVfzbsg3UBOv4w3XnkPvzKcw5L12buXJu7NzWbW+JpXl6PUK3WKzl2k+qqgmyPL+YR1YVUHyokqH9u7H4+jF8Z1g/O1Y+VNHn5IVPt1IulubpNYeurHH8RC3Prd/FY6sL2Xe4mlFpydx2cRZTh/Sxa7EspTzUrhy6MSYReBfo2LD/chH5dcg+HYGngLFAGXCdiOxoZ91WcZrftn19cKf58kjO93BVDU+v3cnS97Zz8NgJJgzuyQPXjuKCzJ7tb+RuZ4WdZpndflzbx/OS07n4ac5NcHLKpRqYJiJHjTEJwHvGmDdFZN0p+9wEHBKRLGPMbOA+4Low1OsJp/lt29cHd5ovj9R8Dx07wRNrtvPE+zs4UlXLlCG9WTg1i3EZPdo50wZuZ4WdZpndflzbx/OS07n4ac7NaPGiqNQ72nAzoeFP6Hmaq4A/N/y+HLjY+Og9stP8tu3rgzvNl4d7vvuPVHHPG1uYeF8OD+YEmJjZi9cWXsiTN57nXjMH97PCTrPMbj+u7eN5yelc/DTnZji6KGqMiQPygSxgkYisD9klBSgCEJFaY0wF0BM4EDLOPGAeQHp6evsqjyCn+W3b1wd3mi8P13xLyitZsqqA5z8soiZYxxXnDODWKVkM6de1pdLbxu2ssNMss9uPa/t4XnI6Fz/NuRmOYosiEhSRUUAqcJ4xZnjILo0djX+je4jIEhEZJyLjevfu3fpqPdJU7jt0u9P9vNJUjjx0u9vz3XHgGP+xfCNT7s/l2fW7mDkqhZyfT+GPs0eHr5lD05ngtmaFm8osh253+3FtH89LTufipzk3o1U5dBEpB1YC3wm5qxhIAzDGxAPdgYMu1GcFp/lt29cHd5ovd2u+2/Yd4afPb2Da/1vJKx+XMOe8dFb9Yir3zRpJRq/O7Z1Oy9zOCjvNMrv9uLaP5yWnc/HTnJvhJOXSG6gRkXJjTBJwCfUXPU/1KvBDYC0wC8gRr/KQYeA0v23z+uDgPF/e3vlm9enCLU/n89Zne+nUIY6bLxrMzRcOok+3xAjM8hRuZ4WdZpndflzbx/OS07n4ac7NaDGHbowZSf0Fzzjqj+hfFJG7jTF3A3ki8mpDtPFpYDT1R+azRaSwuXE1h+4/+TsPkp0TIHdrKV0T45l7QQY3ThxEj84dvC5NKd9oVw5dRDZS36hDt//qlN+rgO+2p0i/sD2H7jYRYW1BGQ/lBFhbWMbpnRK4Y/oQbpgwkG6JPlz10PbMs9v1hWMemqkPG/3ov4tsz6G7SUTI3bqf7JwAH+0qp0/XjvzysrP53vh0OnXw6X9Wtmee3a4vHPPQTH1Y6Uf/XTTx3pxGVzNMSU5izZ3TPKjIfXV1wtuf7SU7N8Bnuw+TkpzELVMy+e7YVBITfL5Y1h+GN7HCXhr87NPW72d7feGYh9tjevVce0i/gi5CbM+ht0dtsI7XNu5mUW4Bgf1HGdSrM/fPGsnM0SkkxMXIop22Z57dri8c89BMfVhpQ3eR2+uX2+BEbR0vf1TM4pUF7Dp4nCF9u/LgnNFcNqI/cbG28qHTNbC9Wivb7frCMQ+3x7R0XXKvxMihVWTYnkNvjaqaIE+u2c7k+3O58+VNJHdKYMkNY3nzJxdx5TkDYq+Zg/2ZZ7frC8c8NFMfVnqE7iLbc+hOHK2u5dl1O3ls9XYOHK3m3IzTufeakUw6o5cuYWt75tnt+sIxD83Uh5VeFFUAVByv4cn3d/DE+9spP17DRWf0YuHULMYP7ul1aUqpU+hFUdWksqPVLH1vO0+t3cnR6louObsvC6dlMSot2evSopvTddO9Ynt9YH+W30La0GPU3ooqlrxbyHMf7KS6to4ZI/qzYEoWQwd087q06Od03XSv2F4f2J/lt5SecokxRQeP88iqAl7KKyYowsxRKcyfkklWny5el+Yf/92j6W+Y/7UFa9bZXh/Yn+X3kJ5yURSUHmVxbgErPi4hzhhmjUtl/uRM0np08ro0/3G6brpXbK8P7M/yW0obus9t2XOYRbkBXt+0h47xp/HDCRnMmzSYft0jvPJhLDFxTR8B28D2+sD+LL+lNIfuUx8XlXPzn/O49I+rWbm1lPmTM3nvP6bxqyuGajMPN6frpnvF9vrA/iy/pfQI3WfWF5aRnRtg9bYDdE9K4GeXnMncCzLo3smHKx/ayum66V6xvT6wP8tvKb0o6gMiwuptB8jOCfDBjoP06tKBmy8azPfPH0iXjvr/bKX8RC+K+lRdnfDPLfvIzg2wsbiC/t0T+c0VQ5l9Xnr0rnxoewbY7frczoPb/vypsNKGHoWCdcLrm/awODfA53uPkN6jE/dePYKrx6TSIT6KL4vYngF2uz638+C2P38q7PSUSxSpCdaxYkMJD68soPDAMbL6dGHB1EyuGDmAeD8sYWt7Btjt+tzOg9v+/ClX6CmXKFdVE+Sl/GIeWVlASXklQ/t34+HrxzB9WD9O89Oqh7ZngN2uz+08uO3Pnwo7begWO36ilufW72LJu4XsP1LN6PRkfjtzGFOH9PHnyoe2Z4Ddrs/tPLjtz58KOx+8T/efw1U1LMoNcOF9ufzu9S1k9u7CczeP5+X5FzDtrL7+bOZgfwbY7frczoPb/vypsNMjdIscOnaCJ9Zs54n3d3CkqpapQ3qzcFoWYwf28Lq0yLA9A+x2fW7nwW1//lTY6UVRC+w/UsXjq7fzzLqdHD8R5DvD+rFwWhbDU7p7XZpSyjJ6UdRSJeWVLFlVwLIPi6gN1nHlOQO4dWoWZ/bt6nVpscft/LbT8TQ3rlykDd0DOw4c4+GVBby8oT59cPXoVOZPySSjV2ePK4tRbue3nY6nuXHlMj3lEkHb9h1hUW6AVz/ZTXzcacw5N415kzNJSU5q+S+r8HE7v+10PM2NqzbQUy4e+7SkguycAG99tpdOHeL48UWDuemiQfTpqqseWsHt/LbT8TQ3rlymDT2M8nceJDsnQO7WUromxnPbtCxunDiI0zt38Lo0dSq389tOx9PcuHKZ5tBdJiK8HzjAnCXruObhtXxSXMEd04ew5s5p3P7tIdrMbeR2ftvpeJobVy7TI3SXiAi5W/eTnRPgo13l9OnakV9edjbfG59Opw76NFvN7fy20/E0N65cphdF26muTnj7s71k5wb4bPdhUpKTmD8lk1ljU6N3CVullLXadVHUGJMGPAX0A+qAJSLyx5B9pgB/BbY3bHpZRO5uT9G2qw3W8drG3SzKLSCw/yiDe3Xm/lkjmTk6hQQ/rHzoNtvz1pobbx99Xqzg5FxALXSjDBwAAAstSURBVPBzEfnIGNMVyDfG/ENENofst1pELne/RLucqK3j5Y+KWbyygF0Hj3NWv648NGc0M0b0J85PKx+6yfa8tebG20efF2u0eCgpIntE5KOG348AW4CUcBdmm6qaIE+u2c7k+3O58+VNJHdK4LEfjOON2y7iinMGaDNvzjt3f/WP/aSayvrtNnBan+3z8Io+L9Zo1dU6Y0wGMBpY38jdE4wxnwC7gX8Xkc8a+fvzgHkA6enpra3VE0era3l23U4eW72dA0erOS+jB/ddM5KLzujl31UP3WZ73lpz4+2jz4s1HDd0Y0wX4C/AT0XkcMjdHwEDReSoMWYGsAI4I3QMEVkCLIH6i6JtrjoCKo7X8OT7O3ji/e2UH6/hojN6sXDqaMYP7ul1adHH9ry15sbbR58Xazi6emeMSaC+mT8rIi+H3i8ih0XkaMPvbwAJxpherlYaIWVHq/n9W58z8b4c/vDPLxg3sAcrFkzk6ZvGazNvK9vz1pobbx99XqzhJOVigKXAFhFpdKFmY0w/YJ+IiDHmPOr/R1HmaqVhtreiiiXvFvLcBzuprq3jshH9WTA1i7P7d/O6tOhne95ac+Pto8+LNVrMoRtjLgRWA5uojy0C/CeQDiAijxhjFgLzqU/EVAK3i8j7zY1rSw696OBxHllVwEt5xQRFmDkqhVunZpLZu4vXpSml1De0K4cuIu8BzV79E5FsILtt5XmjoPQoi3MLWPFxCXHG8N1xqdwyOZO0Hp28Ls0OsZYr/tvt7n1zkFIeibnPpG/Zc5hFuQFe37SHjvGn8cMJGcybNJh+3XXlwy/FWq74b7dD3tKvbkvwq9va1FUUiZmG/nFROdk5Af65ZR9dOsYzf3ImP7pwEL26dPS6NPs0lyv2Y0PPf7Lp7drQVRTxfUNfX1hGdm6A1dsOkNwpgdu/dSY/nJBB904JXpdmr1jLFUuwdduVspQvG7qIsHrbAbJzAnyw4yC9unTgrkvP4vrzB9Kloy+n7K5YyxWbuMabt9HF1VR08VV3q6sT/rllH9m5ATYWV9C/eyL/feUwrjs3TVc+bI2Lf/X1c+jg71zx2LlfP4d+6naloogvGnqwTnh90x4W5wb4fO8RBvbsxL1Xj+DqMal0iNeVD1st1nLFJ8+Ta8pFRbmoXg+9JljHig0lPLyygMIDx8jq04WFU7O4fGR/4nUJW6WUD/nuS6KraoK8lF/MIysLKCmvZNiAbjx8/RimD+vHaVGy6uGKDSXc//ZWdpdXMiA5iTumD2Hm6ChcxNIveXW/zMMr+vxZIeoa+qovSrnjpU/Yf6SaMenJ/G7mcKYM6R1VKx+u2FDCXS9vorKm/kJcSXkld728CSC6mrpf8up+mYdX9PmzRtSdl0hJTuSMvl147sfj+cv8C5h6Vp+oauYA97+99ctmflJlTZD7397qUUVt5Jd1sP0yD6/o82eNqDtCz+rTlWdvPt/rMtpld3llq7Zbyy95db/Mwyv6/Fkj6o7Q/WBAclKrtlurqVx6tOXV/TIPr+jzZw1t6B64Y/oQkkJy8UkJcdwxfYhHFbWRX9bB9ss8vKLPnzWi7pSLH5y88Bn1KRe/5NX9Mg+v6PNnjajOoSulVKxpLoeup1yU8pONL8IfhsNvkut/bnzRrvFUWOkpF6X8wu08uObLo44eoSvlF27nwTVfHnW0oSvlF27nwTVfHnW0oSvlF27nwTVfHnW0oSvlF27nwTVfHnW0oSvlFyOvhSsehO5pgKn/ecWDbb+A6fZ4Kuw0h66UUlFEc+hKKRUDtKErpZRPaENXSimf0IaulFI+oQ1dKaV8Qhu6Ukr5hDZ0pZTyCW3oSinlEy02dGNMmjEm1xizxRjzmTHmJ43sY4wxDxpjAsaYjcaYMeEpV1lH18tWyhpO1kOvBX4uIh8ZY7oC+caYf4jI5lP2uRQ4o+HPeODhhp/Kz3S9bKWs0uIRuojsEZGPGn4/AmwBQr/88irgKam3Dkg2xvR3vVplF10vWymrtOocujEmAxgNrA+5KwUoOuV2Md9s+hhj5hlj8owxeaWlpa2rVNlH18tWyiqOG7oxpgvwF+CnInI49O5G/so3Vv0SkSUiMk5ExvXu3bt1lSr76HrZSlnFUUM3xiRQ38yfFZGXG9mlGEg75XYqsLv95Smr6XrZSlnFScrFAEuBLSLyQBO7vQr8oCHtcj5QISJ7XKxT2UjXy1bKKk5SLhOBG4BNxpiPG7b9J5AOICKPAG8AM4AAcBy40f1SlZVGXqsNXClLtNjQReQ9Gj9Hfuo+AixwqyillFKtp58UVUopn9CGrpRSPqENXSmlfEIbulJK+YQ2dKWU8glt6Eop5RPa0JVSyidMfYTcgwc2phTY2ca/3gs44GI5XvLLXHQedvHLPMA/c3FrHgNFpNHFsDxr6O1hjMkTkXFe1+EGv8xF52EXv8wD/DOXSMxDT7kopZRPaENXSimfiNaGvsTrAlzkl7noPOzil3mAf+YS9nlE5Tl0pZRS3xStR+hKKaVCaENXSimfsL6hG2PijDEbjDF/a+S+jsaYF4wxAWPM+oYvsbZSC/OYa4wpNcZ83PDnZi9qdMIYs8MYs6mhzrxG7jfGmAcbXpONxpgxXtTZEgfzmGKMqTjlNbHye/WMMcnGmOXGmM+NMVuMMRNC7o+W16OleUTL6zHklBo/NsYcNsb8NGSfsL0mTr6xyGs/AbYA3Rq57ybgkIhkGWNmA/cB10WyuFZobh4AL4jIwgjW0x5TRaSpD0hcCpzR8Gc88HDDTxs1Nw+A1SJyecSqaZs/Am+JyCxjTAegU8j90fJ6tDQPiILXQ0S2AqOg/iAOKAFeCdktbK+J1UfoxphU4DLg8SZ2uQr4c8Pvy4GLG74D1SoO5uEnVwFPSb11QLIxpr/XRfmRMaYbMIn67/xFRE6ISHnIbta/Hg7nEY0uBgpEJPQT8WF7Taxu6MD/Ar8A6pq4PwUoAhCRWqAC6BmZ0lqlpXkAXNPw9mu5MSYtQnW1hQB/N8bkG2PmNXL/l69Jg+KGbbZpaR4AE4wxnxhj3jTGDItkcQ4NBkqBJxpO5z1ujOkcsk80vB5O5gH2vx6hZgPLGtkettfE2oZujLkc2C8i+c3t1sg2q3KYDufxGpAhIiOBf/LVuw4bTRSRMdS/bVxgjJkUcr/1r0mDlubxEfVrZpwDPASsiHSBDsQDY4CHRWQ0cAy4M2SfaHg9nMwjGl6PLzWcNroSeKmxuxvZ5sprYm1DByYCVxpjdgDPA9OMMc+E7FMMpAEYY+KB7sDBSBbpQIvzEJEyEaluuPkYMDayJTonIrsbfu6n/tzgeSG7fPmaNEgFdkemOudamoeIHBaRow2/vwEkGGN6RbzQ5hUDxSKyvuH2cuobY+g+tr8eLc4jSl6PU10KfCQi+xq5L2yvibUNXUTuEpFUEcmg/q1Ljoh8P2S3V4EfNvw+q2Efq44+nMwj5PzZldRfPLWOMaazMabryd+BbwOfhuz2KvCDhiv55wMVIrInwqU2y8k8jDH9Tl6PMcacR/2/lbJI19ocEdkLFBljhjRsuhjYHLKb9a+Hk3lEw+sRYg6Nn26BML4m0ZBy+RpjzN1Anoi8Sv1FlKeNMQHqj8xne1pcK4TM4zZjzJVALfXzmOtlbc3oC7zS8O8qHnhORN4yxtwCICKPAG8AM4AAcBy40aNam+NkHrOA+caYWqASmG3bwUKDfwOebXiLXwjcGIWvB7Q8j2h5PTDGdAK+BfzrKdsi8proR/+VUsonrD3lopRSqnW0oSullE9oQ1dKKZ/Qhq6UUj6hDV0ppXxCG7pSSvmENnSllPKJ/w90OHLreeEEDAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "x_ponits = np.arange(4, 8)\n", "y_ = -(lr_clf.weights[1]*x_ponits + lr_clf.weights[0])/lr_clf.weights[2]\n", "plt.plot(x_ponits, y_)\n", "\n", "#lr_clf.show_graph()\n", "plt.scatter(X[:50,0],X[:50,1], label='0')\n", "plt.scatter(X[50:,0],X[50:,1], label='1')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### scikit-learn实例\n", "\n", "#### sklearn.linear_model.LogisticRegression\n", "\n", "solver参数决定了我们对逻辑回归损失函数的优化方法，有四种算法可以选择，分别是：\n", "- a) liblinear：使用了开源的liblinear库实现，内部使用了坐标轴下降法来迭代优化损失函数。\n", "- b) lbfgs：拟牛顿法的一种，利用损失函数二阶导数矩阵即海森矩阵来迭代优化损失函数。\n", "- c) newton-cg：也是牛顿法家族的一种，利用损失函数二阶导数矩阵即海森矩阵来迭代优化损失函数。\n", "- d) sag：即随机平均梯度下降，是梯度下降法的变种，和普通梯度下降法的区别是每次迭代仅仅用一部分的样本来计算梯度，适合于样本数据多的时候。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "from sklearn.linear_model import LogisticRegression" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "clf = LogisticRegression(max_iter=200)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "LogisticRegression(max_iter=200)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9666666666666667" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.score(X_test, y_test)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[ 2.59546005 -2.81261232]] [-5.08164524]\n" ] } ], "source": [ "print(clf.coef_, clf.intercept_)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXwU9f348debBIFwqRBUCCHIKbeAoILgLR4FEa1WtFVsqWKrrV+LKN6KZ2u1RbGo7U8LXuXy4lBUFFFBQEm47yPch5wh5Nj374/ZaIw5JmR2Z3b3/Xw89rG7s7Oz789usu+dmfe8R1QVY4wxiaua3wEYY4zxlyUCY4xJcJYIjDEmwVkiMMaYBGeJwBhjElyy3wFUVsOGDTUjI8PvMIwxJqYsWLBgl6qmlvZYzCWCjIwM5s+f73cYxhgTU0RkQ1mP2aYhY4xJcJYIjDEmwVkiMMaYBBdz+whKk5+fT3Z2Nrm5uX6HUq6aNWuSlpZG9erV/Q7FGGN+EBeJIDs7m7p165KRkYGI+B1OqVSV3bt3k52dTfPmzf0OxxhjfhAXm4Zyc3Np0KBBYJMAgIjQoEGDwK+1GGMST1wkAiDQSaBILMRojEk8cZMIjDEmXuUXhnhh1moWbdobkeVbIvDQ9OnTadOmDS1btuSJJ57wOxxjTBxYvHkflz8/h6emr2Da4m0ReY2ETATjx0NGBlSr5lyPH1/1ZRYWFnLrrbcybdo0li5dyhtvvMHSpUurvmBjTELKzS/k6RnLGfD8HLbvP8KYwV0ZcXHbiLxWXFQNVcb48TB0KOTkOPc3bHDuAwwefPTLnTdvHi1btuTkk08G4JprruGdd96hXbt2VYzYGJNo5q/fw/CJmazdeYiruqVx76XtqJ8SubLzhFsjGDnyxyRQJCfHmV4VmzdvpmnTpj/cT0tLY/PmzVVbqDEmoRw8UsAD7yzmqn99xZH8EK8N6cHTV3WOaBKABFwj2LixctPdKu3cz1YlZIxx67OVO7lnUhZb9h3mN2dk8JeL2lC7RnS+ohMuEaSnO5uDSpteFWlpaWzatOmH+9nZ2TRu3LhqCzXGxL29OXk88v4yJi7MpkVqbf73+zPonnF8VGNIuE1Do0ZBSspPp6WkONOr4rTTTmPVqlWsW7eOvLw83nzzTfr371+1hRpj4tq0rK2c/8znvPPdZv5wTks+uO2sqCcBSMA1gqIdwiNHOpuD0tOdJFCVHcUAycnJjB49mosuuojCwkKGDBlC+/btqx6wMSbu7Nify/3vLGH6km10aFKPV4ecRvvG9X2LJ+KJQESSgPnAZlW9rMRjNwBPA0V7VUer6suRjmnw4Kp/8Zfmkksu4ZJLLvF+wcaYuKCq/G9BNo++v5TcghB39WvL785qTnKSvxtnorFGcDuwDKhXxuNvqeofohCHMcb4ZtOeHO6ZnMXsVbvokXE8TwzqyMmpdfwOC4hwIhCRNOBSYBRwRyRfyxhjgqgwpLz21XqenrECAR4Z0J7BPZtRrVpwqgojvUbwLDAcqFvOPINEpA+wEvizqm4qOYOIDAWGAqRXtbzHGGOiZPWOA9w1MYsFG77n7DapjBrYkSbH1vI7rJ+J2IYpEbkM2KGqC8qZ7T0gQ1U7ATOBV0ubSVXHqmp3Ve2empoagWiNMcY7+YUhRn+yikue+4I1Ow/y96s7858bTgtkEoDIrhH0AvqLyCVATaCeiIxT1euKZlDV3cXmfwl4MoLxGGNMxGVl7+MvExaxfNsBLu10Eg/1b0/DOjX8DqtcEUsEqno3cDeAiJwN3Fk8CYSnn6SqW8N3++PsVDbGmJiTm1/IszNX8dLstTSofQz/ur4bF7U/0e+wXIl6zZKIPCwiRUda3SYiS0RkEXAbcEO04/HKkCFDaNSoER06dPA7FBMHItEh10TO3LW7ufi52bz42Rqu7JrGR3f0jZkkAFFKBKo6q+gYAlW9X1XfDd++W1Xbq2pnVT1HVZdHIx7WjYcpGfB6Ned6XdX/y2644QamT59e5eUYU9Qhd8MGUP2xQ64lg+A5kJvPfVMWc/XYrykIhRj/2548eWUn6teKbJM4ryXckcWsGw/zhkJhuAVpzgbnPkDzoz/KrE+fPqxfv77q8ZmEV16H3EgcCGmOzqfLdzBychZb9+dyU+/m/N+FrUk5Jja/UmMz6qpYNPLHJFCkMMeZXoVEYIxXItUh13hjz6E8Hnl/KZO/3UyrRnWYeMuZdE0/zu+wqiTxEkFOGf9NZU03Jsoi1SHXVI2q8kHWVh54Zwn7Dudz23mtuPWcFtRITvI7tCpLuO6jpJTx31TWdGOiLFIdcs3R274/l6H/XcAfXv+WJsfV4r0/9uaOC1rHRRKARFwj6Dzqp/sIAJJSnOnGBECkOuSaylNV3p6/iUc/WEZeQYh7LmnLkF7+N4nzWuIlgqL9AItGOpuDUtKdJFDF/QO/+tWvmDVrFrt27SItLY2HHnqIm266yYOATSKKVIdc497G3TmMmJTJl2t207P58Tw5qBMZDWv7HVZEJF4iAOdL3+Mdw2+88YanyzPG+KMwpPxnzjr++uEKkqtV47GBHbnmtKaBahLntcRMBMYYU4qV2w8wfEIm323ay7ltGzFqYAdOqh/M/kBeskRgjEl4eQUhxsxaw+hPV1G3ZnWeu6YL/Ts3RiR+1wKKi5tEoKqB/9BU1e8QjDElLNq0l7smZrJ82wEGdGnM/Ze1o0HAm8R5LS4SQc2aNdm9ezcNGjQIbDJQVXbv3k3NmjX9DsUYAxzOK+TvM1fy8uy1NKpbk5d/3Z3z253gd1i+iItEkJaWRnZ2Njt37vQ7lHLVrFmTtLQ0v8MwJuF9tWY3IyZlsmF3Dtf2TGfExW2pVzO2+gN5KS4SQfXq1WnevLnfYRhjAm5/bj6PT13OG/M20qxBCq//ridntmjod1i+i6+jIozxibWNDr6Pl23nwmc+561vNjK0z8lMv72PJYGwuFgjMMZPRW2jizqGFrWNBjsoLAh2HzzCQ+8t5d1FW2h7Yl3+dX03Ojc91u+wAkVirZKle/fuOn/+fL/DMOYHGRmlN4lr1gysM7l/VJV3F23hofeWciA3nz+c04pbzm7BMcmJuSFERBaoavfSHrM1AmOqyNpGB8/WfYe5d/JiPl6+gy5Nj+WpKzvR+oS6focVWJYIjKkiaxsdHKGQ8sY3G3l86nIKQiHuvfQUbuzVnKQ4bg/hBUsExlTRqFE/3UcA1jbaD+t3HWLEpEy+XruHM1s04IkrOpHeIKXiJxpLBMZUlbWN9ldBYYh/z1nH3z5cyTHJ1XhyUEd+2b1pYA8uDaLE3Gti4pKfJZyDBzs7hkMh59qSQHQs37afQWO+5LGpy+nTOpWZd/Tl6tPSLQlUkq0RmLhgJZyJ5UhBIc9/uoYXPl1N/VrVGX3tqVza8SRLAEfJykdNXLASzsSxcOP33DUhk1U7DjLw1Cbcf1k7jqt9jN9hBZ6Vj5q4ZyWc8S8nr4C/fbiSf89Zx4n1avKfG07jnLaN/A4rLlgiMHHBSjjj25zVuxgxKZNNew5z/enNGN6vDXUTuEmc12xnsYkLo0Y5JZvFWQln7Nt3OJ8REzMZ/PJckqtV462hp/PI5R0sCXjM1ghMXLASzvjz4ZJt3DtlMbsP5XFz3xb86fxW1Kye5HdYcSniawQikiQi34rI+6U8VkNE3hKR1SIyV0QyIh2PiV+JWMIZj11Pdx44wq2vL2TofxfQoE4NpgzrxYiL21oSiKBorBHcDiwD6pXy2E3A96raUkSuAZ4Ero5CTMbEvHgrmVVVpny3mYfeW0rOkULuvLA1v+/bgupJtgU70iL6DotIGnAp8HIZswwAXg3fngCcJ1YIbIwrI0f+tK0FOPdHjvQnnqrYvPcwN/6/b/jzW4s4uWFtpt7emz+c28qSQJREeo3gWWA4UFbbvybAJgBVLRCRfUADYFfxmURkKDAUIN3KQIwB4qNkNhRSxs/dwBPTlhNSeOAX7fj1GRnWJC7KIpYIROQyYIeqLhCRs8uarZRpPzvCTVXHAmPBOaDMsyCNiWGxXjK7dudBRkzMYt76PZzVqiGPDexI0+OtSZwfIrne1QvoLyLrgTeBc0VkXIl5soGmACKSDNQH9kQwJmPiRqyWzBYUhhgzaw39npvN8m37efrKTrw2pIclAR9FbI1AVe8G7gYIrxHcqarXlZjtXeA3wFfAlcAnGms9L4zxSSyWzC7dsp/hExexePN+Lmp/Ao8M6ECjejX9DivhRX1PjIg8LCL9w3dfARqIyGrgDmBEtOMxpizDhkFyMog418OG+R3Rz8VKyWxufiF/nbGC/qO/YNu+I4wZ3JV/Xd/dkkBAROWAMlWdBcwK376/2PRc4KpoxGBMZQwbBmPG/Hi/sPDH+y+84E9MsWrBhj0Mn5DJmp2HGNQ1jfsuO4VjU6xJXJBY91FjSpGc7Hz5l5SUBAUF0Y8nFh06UsDTM1bw6lfraVy/Fo9d0ZG+rVP9DithWfdRYyqptCRQ3nTzU5+v3Mndk7LYsu8wvz69GX/p15Y6NezrJqjskzGmFElJZa8RmLLty8nnkQ+WMmFBNien1ubt35/BaRnH+x2WqYAlAmNKMXToT/cRFJ9uSjd98Vbue2cJew7lMezsFtx2njWJixWWCIwpRdEO4bFjnTWDpCQnCdiO4p/bcSCXB95ZwrTF22jfuB7/ueE0OjSp73dYphKskYcJvPPPd0o4iy7nnx+d133hBWfHsKpzHc0kEAtdRVWVCQuyueCZz/l4+Q6G92vDlFt7WRKIQbZGYALt/PPh449/Ou3jj53pM2f6E1OkxUJX0U17crhnchazV+3itIzjeGJQJ1qk1vE7LHOUrHzUBFp5vWhj7E/XtYyM0nsINWvmHDTmp1BIee2r9Tw1YwUC3HVxW67r2Yxq1iQu8Kx81JgYEtSuoqt3HGTExEzmb/iePq1TeWxgB9KOs/5A8cASgTEBE7SuovmFIcZ+vpbnZq4ipUYSf7uqM1d0bYKdOiR+2M5iE2jnnVe56fEgSF1FF2/ex4DRc3h6xgouaHcCH/25L4O6pVkSiDO2RmACbebMn+8wPu+8+N1RDMHoKpqbX8hzH69i7OdrOb72Mbx4XTf6dTgxegGYqLI1AhN4N97o7CgVca5vvLFqy3NbmulnCaefXUW/Wb+HS56bzZhZaxjUtQkz/9zXkkCcszUCE2hel1K6XV4slHB67eCRAp6avpzXvtpA2nG1GHdTT3q3auh3WCYKrHzUBJrXpZRulxfkEs5ImLViByMnL2bLvsPceGZz/u/C1tS2JnFxxcpHTczyupTS7fKCWsLpte8P5fHIB0uZtHAzLRvVYcLNZ9Kt2XF+h2WizBKBCTSvSyndLi9oJZxeU1WmZm3jgXcXszcnn9vObcmt57akRrI1iUtEtrPYBJrXpZRulxekEk6v7dify+//u4BbX1/ISfVr8e4fenPHhW0sCSQwWyMwgeZ1KaXb5QWhhNNrqsr/5mfzyAdLySsIcffFbbmpd3OSk+z3YMJT1Zi6dOvWTU3Fxo1TbdZMVcS5HjcueK/tZ4yJZuPuQzr4pa+12V3v61UvfqlrdhzwOyQTZcB8LeN7tcI1AhGpAQwCMii2BqGqD0csO5kq8bP00cozg6UwpLz65XqenrGCpGrCo5d34Noe6dYkzvxEheWjIjId2AcsAH44eZ+q/i2yoZXOykcr5mfpo5VnBseq7QcYPjGTbzfu5Zw2qYwa2JHGx9byOyzjk6qWj6apaj+PYzIR5Gfpo5Vn+i+vIMSLn61h9CerqV0jiWev7sKALo2tP5Apk5tE8KWIdFTVrIhHYzzhZ+mjlWf6KzN7L8MnZLJ82wF+0bkxD/yiHQ3r1PA7LBNwZZYLiEiWiGQCvYGFIrJCRDKLTTcB5Wfpo5Vn+iM3v5DHpy7j8ufn8H1OHi/9ujv//NWplgSMK+WtEVwWtSiMp/wsfUzk8ky/fL12NyMmZrJ+dw6/6tGUERefQv1a1f0Oy8QQNzuL/6uq11c0rZTn1QQ+B2rgJJwJqvpAiXluAJ4GNocnjVbVl8tbru0sNsZxIDefJ6YtZ/zcjaQfn8ITV3TkzJbWJM6UrrydxW6OJGlfYmFJQDcXzzsCnKuqnYEuQD8ROb2U+d5S1S7hS7lJwMSXYcMgOdlpL52c7Nyvynx+to2O9mt/snw7F/79c96Yt5Hf9m7O9D+dZUnAHLUyNw2JyN3APUAtEdlfNBnIA8ZWtODwAQwHw3erhy+x1erURMywYTBmzI/3Cwt/vP/CC5WfLxaOnfDCnkN5PPzeEqZ8t4XWJ9ThhcFncmq6NYkzVeNm09Djqnr3US3cWXtYALQEnlfVu0o8fgPwOLATWAn8WVU3lbdM2zQUH5KTnS/1kpKSoKCg8vPFwrETVaGqvJe5lQffXcKB3HyGnd2SW89pyTHJ1h7CuFPepqEyE4GIdC1voaq6sBIBHAtMBv6oqouLTW8AHFTVIyJyM/BLVT23lOcPBYYCpKend9tQ2n+diSnllbQX/5N0O1+1aj+9X/z5oVDl46uMSL/2tn253DtlMTOXbadzWn2evLITbU+sV/UFm4RytAeUFR05XBPoDizC2TTUCZiLU1bqiqruFZFZQD9gcbHpu4vN9hLwZBnPH0t4c1T37t1t81IcSEoq+5f+0cwXC8dOVJaq8uY3m3jsg2Xkh0KMvOQUhvRuTpK1hzAeK3O9UlXPUdVzgA1AV1XtrqrdgFOB1RUtWERSw2sCiEgt4HxgeYl5Tip2tz+wrPJDMLGoaBt6RdPdzhcLx05Uxobdh7j2pbncPSmL9k3qMf32Pvyuz8mWBExklNWNrugCfOdmWinzdAK+BTJx1gLuD09/GOgfvv04sARnbeNToG1Fy7Xuo/HjlltUk5JUwbm+5ZaqzRcLHVcrUlAY0pc+X6Nt7p2qHe6fruO/3qCFhSEvQzUJinK6j7rZ07RMRF4WkbNFpK+IvISLX+6qmqmqp6pqJ1XtoOFupap6v6q+G759t6q2V9XO6qyBLC9/qcatSJQzui3j9FqvXpCW5rxuWppzPxoqM97Bg50dw6GQc3001UIrth3gijFf8ugHy+jVoiEf3tGHa3tGuVPouvEwJQNer+Zcr4tiDa7xT1kZQn/8ZV8T+DPOzt7J4ds1K3pepC62RlCxceNUU1KcX9BFl5SUqv1CvuWWny6v6FLWr3Ovlud2LF6P2evxludIfqH+/aMV2vKeD/TUhz/Ud77brKGQD2sBa8epvpmiOp4fL2+mONNNzKOcNYIKy0eDxspHKxaJcka3ZZxeL8+vttZej7cs323ay10TMlmx/QADujTmgV+05/jax3j3ApUxJQNySnkTU5rB5eujHY3x2FFVDYnI26r6SxHJopQDwVS1k4cxGg9FosVzaV+K5U33anl+tbX2erwlHc4r5JmPVvDKF+toVLcmr/ymO+edcoI3Cz9aOWW8WWVNN3GjvPLR28PX1nwuxkSinNFtGafXy/OrrbXX4y3uyzW7GDExi417cri2ZzojLm5LvZoBaBKXkl7GGoH1Bo935ZWPbg3fPA84RlU3FL9EJzxzNCJRzui2jNPr5fnV1trr8QLsz83n7klZXPvSXETgjd+dzmMDOwYjCQB0HgVJJd7EpBRnuolvZe08KLrglHt+AqwB3gb+CHSp6HmRutjOYnciUUrptozT6+W5HYvXY/ZyvB8t2aY9Rn2kzUe8r6M+WKo5RwqqFlykrB2nOrmZ6nhxrm1HcdzAi53F4YPCfgfcCTRRVQ9WkivPdhbHj/Hj3Z2PwO18QbT74BEefG8p7y3aQtsT6/LkoE50bnqs32EF07rxsGiks08iJd1ZE2keIx90DKjSOYtF5F6gF1AH5wCxO4HZnkZoEo7bjp1+dhWtClXl3UVbePDdJRw8UsAdF7Tm5r4trElcWdaNh3lDoTD8QedscO6DJYMocNN9dCFQAHwAfAZ8raq5UYitVLZGEB/8KguNhi17D3PvlMV8snwHXZoey1NXdqL1CXX9DivYrHQ14qq0RqCqXUWkLk6TuQuAl0Rku6q6bjpnTEl+lYVGUiikvPHNRh6fupzCkHLfZe244cwM6w/khpWu+srNpqEOwFlAX5wupJuwTUOmivwqC42UdbsOMWJiJnPX7aFXywY8PrAT6Q1SKn6icVjpqq/cbLB8EqgL/AM4RZ2eQPdHNiwT7/wqC/VaQWGIf322hn7Pfs7Srft5clBHxt3U05JAZVnpqq/cbBq6NBqBmMRStKO3omogt/P5YdnW/dw1MZPM7H1c0O4EHr28AyfUq+l3WLGpaIewVQ35wkoYAsLrbqFul+dXR9HK8KKzp5eOFBTyzIcr+MU/v2Dz94cZfe2pjL2+W7CTQCx0FW0+2NkxfG3IuS4rCcTCWLwW4TFXuEZgIs/rEkm3y3N7YvhIiNWy0IUbv+euCZms2nGQK05twn2XteM4v5rEuRVPpZnxNBa3ojBm6z4aAF6XSLpdXrQ6bJYm1spCc/IK+OuMlfzny3WcVK8mo67oyDltGvkdljvxVJoZT2Nxy6MxH2330fcopetoEVXt7zoCUy6vSyTdLi/SHTbLE0tloV+s2sXdkzPZtOcw15/ejOH92lA3KP2B3Iin0sx4GotbURhzeZuG/urZq5hyeV0i6XZ5keywWZFYKAvddzifUR8s5e352TRvWJu3hp5Oz5Mb+B1W5cVTaWY8jcWtKIy5vO6jn5V38SwC43mJpNvlRaLDpltBLwudsWQbFzzzGRMXbubmvi2YdvtZsZkEIL5KM+NpLG5FY8xldaMrugCtgAnAUmBt0aWi50XqEq/dR73unOl2eV53FI1EjNG0Y3+uDhu3QJvd9b72e/Zzzdy01++QvBFPXUXjaSxueTBmyuk+6iYRfIFzToJMoBnwIPBQRc+L1CVeE4HXIvElGwvJ5WiFQiGduGCTdn5ohra6Z6r+8+OVmldQ6HdYJkjm3qL6epJzLufXk5z7VRXFpFZeInBTPlpLVT8WEVHnhDQPishs4AHv1kuMlyJRmhkLJalHa/Pew4ycnMWsFTvpmu40iWvZyJrEmWLmDYPVxf6wtfDH+z2O8g87QKWwbrqPzsHpNTQB5wQ1m4EnVLVN5MP7uXgsH/VaJEozY6EktbJCIWX83A08MW05Cgy/qA3Xn2FN4kwp3kh2vvxLkiT41VH+YUe5FLZK3UeBPwEpwG3AI8C5wG+8C894LRKlmbFQkloZa3YeZMTETL5Z/z1ntWrIYwM70vR46w9kylBaEihvuhsBKoV102voGwARqQbcpqoHIh6VqZJIlGbGQkmqGwWFIcbOXsuzM1dRM7kaT1/ZiSu7pSFiawGmHJJU9hrB0QpQKWyFvYZEpLuIZOHsLM4SkUUi0i3yoZmjFYnSzFgoSa3Iki37uPyFOTw1fQXntEll5h19uap7U0sCpmItyvgDLmu6G0EqhS1rL3LRBScBnFXsfm8gs6LnRepiVUPuWNXQjw7nFehT05fpyXd/oN0e+UinZm7xNyATm+K4ashN99EDqvrDiWhU9Qugws1DIlJTROaF1yCWiMhDpcxTQ0TeEpHVIjJXRDJcZa8Y4rYLqNfdRyPRsXPOHMjOBlXnes6c0ufr1QvS0pyOpmlpzv3SeD3m0izYsIdL/zGb5z9dw+VdmjDzjj5c3PEkd0+ORMfHecOcHY+vi3M9r4x2r16/dtCX5ye3Y0ntBbXSAHGuU8v4w64Mtx1XI8zNzuJ5IvIv4A2c3kNXA7NEpCuAqi4s43lHgHNV9aCIVAe+EJFpqvp1sXluAr5X1ZYicg3OSXCuPtrBBE08naDdbVloUMZ86EgBT89Ywatfradx/Vq8OqQHfVunul9AJEr73JYgev3aQV+en9yOJZ7GXAo35aOflvOwquq5Fb6ISArOgWm3qOrcYtNnAA+q6lcikgxsA1K1nKBiqXw0nk7Q7rYsNAhj/nzlTu6elMWWfYf59enN+Eu/ttSpUcmO65Eo7XNbguj1awd9eX5yO5Y4GHNVT15/ThVeOAlYALQEni+eBMKa4JwDGVUtEJF9QANgV4nlDAWGAqQHqStZBeLpBO1uy0L9HPPenDwe/WAZExZkc3Jqbd7+/RmclnH80S0sEqV9bksQvX7toC/PT27HEk9jLoWbqqETROQVEZkWvt9ORG5ys3BVLVTVLkAa0ENEOpRcfGlPK2U5Y1W1u6p2T02txOq9z8rKWaWdoL0yz/dDWeWfJaf7NeZpWVs5/5nPmfztZoad3YKpt5119EkAyi7hq0ppX1mlhiWne/3aQV+en9yOJZ7GXAo3O4v/HzADaBy+vxLnIDPXVHUvMAvoV+KhbKApQHjTUH1gT2WWHWTxcoJ2cF8WGu0x7ziQyy3jFnDL+IU0qluDd27txfB+balZvYoHLkSitM9tCaLXrx305fnJ7VjiacylKaucqOgCfBO+/rbYtO9cPC8VODZ8uxYwG7isxDy3Ai+Gb18DvF3RcmOtfNRtyWUQO3GW5LYsNBpjDoVC+vY3G7XTgzO01cipOvqTVd43iYtEaZ/bEkSvXzvoy/OT27HE+JipYtO5QyLSgPAmGxE5Hdjn4nknAa+G9xNUC3/Jvy8iD4cDehd4BfiviKzGWRO4xsVyY8rgwcGp/KmqF15w1zjO7ZiP9r3ZtCeHeyZnMXvVLro3O44nBnWiZaM6lV9QRZoP9r4iJLUXbJnqbFv2qgTRDbdjWTceFo104ktJd37xVvU98HqZkYjRjUj8PQSEm6qhrsA/gQ7AYpxf+leqambkw/u5WKoacqtkKSU4m0nGjo2fJOKFUEh57av1PDVjBQLcdXFbruvZjGqx0iSuZAkiOJsXeowtv1SxrPliIT6vxxL05QVYeVVDrk5eH95+3wZn5+4KVc33NkT34jERxEL5qN9W7zjAXROzWLDhe/q0TuWxgR1IOy7GmsQFvVQxEvEFvXQ1DspC3apS+aiIXAVMV9UlInIv0FVEHtWyDyQzlRQL5aN+yS8MMfbztTw3cxW1jknib1d15oquTWKzP1DQSxUjEV/QS1fjvCzULTdVQ/ep6gER6Q1cBLwKjKngOaYSYqF81A+LN+9jwOg5PD1jBee3a8TMOw2Pyu0AABUBSURBVPoyKJY7hQa9VDES8QW9dDXOy0LdcpMIio52uRQYo6rvAMdELqTEEwvlo9GUm1/Ik9OXM+D5Oew8eIQXr+vKC4O7kVq3ht+hVU3QSxUjEV/QS1fjvSzUJTeJYHO419AvgakiUsPl84xLgwc7O4abNXMatTVrlrg7iuet28Mlz81mzKw1XHFqE2b+uS/9OrhsEhd0zQc7OyFTmgHiXJe2U9LtfLEQn9djCfryYpSbqqEUnAPBslR1lYicBHRU1Q+jEWBJ8biz2MDBIwU8OW05//16A2nH1eLxKzpyVqvYOYrcmKCraq+hHGBSsftbga3ehWcS3acrdjByUhZb9+dyY68M7rywDbUr2yQu3swbBmvGOn2IJMk5+vhoT5IeCUGPD/w73sCv162CBP9vM376/lAej7y/lEnfbqZlozpMuPlMujU7zu+w/Oe2XbVfgh4f+Nc2OkbbVbs6jiBIbNNQ7FNVpmZt44F3F7M3J5+b+7bgj+e1pEZyQE5s7De37ar9EvT4IPjHYvigSpuGjPHS9v253DdlMR8u3U7HJvV5bUhP2jWu53dYweK2XbVfgh4fBP9YjICxRGCiQlV5e/4mHv1gGXkFIUZc3Jbf9m5OcpIVoP2MJJX9izsIgh4fONvmS/1lHoVjMfx43Sqy/0ITcRt353DdK3O5a2IWp5xUj2m3n8XNfVtYEiiL23bVfgl6fBD8YzECxtYITMQUhpT/9+V6/jpjBUnVhEcv78C1PdJjp0mcX4p2uAa1Kifo8cGPO2ajXb3j1+tWke0sNhGxavsBhk/M5NuNezm7TSqPDexI42NrRfZFY6Fsz+sYvS7jjIX30BwV21lsoiavIMSLn61h9CerqV0jiWev7sKALo0j3x8oFsr2vI7R6zLOWHgPTUTYGoHxTGb2XoZPyGT5tgNc1ukkHuzfnoZ1otQfKMBlez/wOkavyzhj4T00R83WCExEHc4r5NmZK3lp9loa1qnB2Ou7cWH7E6MbRCyU7Xkdo9dlnLHwHpqIsERgquTrtbsZMTGT9btz+FWPpoy4+BTq16oe/UBioWzP6xi9LuOMhffQRITV75mjciA3n5GTs7hm7NeEFF7/bU8ev6KTP0kAYqNsz+sYvS7jjIX30ESErRGYSvtk+XZGTl7M9v25/LZ3c+64sDUpx/j8pxQLZXtex+h1GWcsvIcmImxnsXFtz6E8Hn5vCVO+20KrRnV46spOnJpuTeIA78su3S7Pyj2NS7az2FSJqvJe5lYefHcJ+w/nc/t5rRh2TgtrElfE67JLt8uzck/jEdtHYMq1bV8uv3ttPre98S1Nj6vF+7f15s8XtLYkUNyikT9+GRcpzHGmR3J5Xr+uSVi2RmBKpaq8+c0mHvtgGfmhECMvOYUhvZuTZO0hfs7rsku3y7NyT+MRSwTmZzbsPsSIiVl8tXY3p598PE9c0YmMhrX9Diu4vC67dLs8K/c0HrFNQ+YHhSHl5dlruejZz1m8eR+PDezI67893ZJARbwuu3S7PCv3NB6xNQIDwIptTpO4RZv2cl7bRjw6sAMn1Y9wk7h44XXZpdvlWbmn8UjEEoGINAVeA04EQsBYVX2uxDxnA+8A68KTJqnqw5GKyfxcXkGI5z9dzQuzVlO3ZnWeu6YL/TtHoUlcZcRTiaTbsTQf7G6MbueLBfH0OceYSK4RFAD/p6oLRaQusEBEPlLVpSXmm62ql0UwDlOG7zbtZfiERazcfpABXRpz/2XtaBCtJnFuxUKJpJV7Vp29N76K2D4CVd2qqgvDtw8Ay4AmkXo9497hvEIefX8pV7wwh/2HC3jlN9157ppTg5cEIDZKJK3cs+rsvfFVVPYRiEgGcCowt5SHzxCRRcAW4E5VXVLK84cCQwHS060ioiq+XLOLEROz2Lgnh2t7pjPi4rbUq+lTfyA3YqFE0so9q87eG19FvGpIROoAE4E/qer+Eg8vBJqpamfgn8CU0pahqmNVtbuqdk9NTY1swHFqf24+d0/K5NqX5iICb/zudB4b2DHYSQDKLoUMUomk2xhjYSx+sffGVxFNBCJSHScJjFfVSSUfV9X9qnowfHsqUF1EGkYypkQ0c+l2LnjmM976ZhND+5zM9Nv7cEaLBn6H5U4slEhauWfV2Xvjq0hWDQnwCrBMVZ8pY54Tge2qqiLSAycx7Y5UTIlm18EjPPTeUt5btIW2J9Zl7PXd6dz0WL/DqpxYKJG0cs+qs/fGVxHrPioivYHZQBZO+SjAPUA6gKq+KCJ/AG7BqTA6DNyhql+Wt1zrPloxVeWd77bw0HtLOHikgD+e24qb+7bgmOQoHT+YiGWAXp9E3hiP+dJ9VFW/AMotRlfV0cDoSMWQiLbsPcy9UxbzyfIddGl6LE9d2YnWJ9SNXgCJWAbo9UnkjYkyazERJ0IhZdzXG7jw75/z1Zrd3HdZOybecmZ0kwAkZhngmrGVm25MwFiLiTiwbtchRkzMZO66PfRq2YDHB3YivUFKxU+MhEQsA/T6JPLGRJklghhWUBjilS/W8cxHKzkmuRpPDurIL7s39bc9RCJ2xPT6JPLGRJltGopRS7fsZ+ALX/L4tOX0aZ3KzDv6cvVp6f73CErEMkCvTyJvTJTZGkGMOVJQyOhPVjNm1hqOTanO89d25ZKOJ/qfAIokYhmg1yeRNybKLBHEkAUbvufX92xlxfsZFB5oTVoa7G0uSCe/IyvBbUfMeCozTe0FW6Y6Y6mV5tw3lRNPfw8xxhJBDMjJK+DpGSsY/VIee6Z3IpTvbHvO3gRDw1sfBsfa/0s8lZnG01j8Yu+hr2wfQcB9sWoXF/79c/4zZz35X7f/IQkUycmBkbFYmRlPZabxNBa/2HvoK0sEAbUvJ5/hExZx3StzqZ5UjbeGns7+XceUOu/GWKzMjKcy03gai1/sPfSVbRoKoOmLt3HfO4vZcyiPW85uwe3ntaJm9STS02FDKZWZMdmZO57KTONpLH6x99BXtkYQIDsPHOHW8Qu5edwCGtapwZRhvbirX1tqVnc2B40aBSklKjNTUpzpMSeeykzjaSx+sffQV7ZGEACqyqSFm3n4/aUczivkLxe1YWifk6me9NM8XbRDeORIZ3NQerqTBGJuRzHEV5lpPI3FL/Ye+ipi3UcjJd66j27ee5h7JmXx2cqddE13msS1bBTl/kDGFIlECaeVhQaCL91HTflCIWXc3A08OW05Cjz4i3Zcf0YGSdUCcmCYSTyRKOG0stCYYInAB2t2HmTExEy+Wf89Z7VqyGMDO9L0eJ+axBlTpLwSzqP90o7EMo3nLBFEUX5hiJdmr+XZmauomVyNp6/sxJXd0oLTHsIktkiUcFpZaEywRBAlizfv466JmSzZsp9+7U/k4cvb06huTb/DMuZHkSjhtLLQmGDloxGWm1/I0zOWM+D5OWzff4Qxg7vy4vXdLAmY4IlECaeVhcYEWyOIoPnr9zB8YiZrdx5iUNc07rvsFI5NKf3oYGN8F4kSTisLjQlWPhoBh444TeJe/Wo9jevX4rErOtK3darfYRljEpiVj0bRZyt3cs+kLLbsO8xvzsjgLxe1oXYNe5uNMcFl31Ae2ZuTxyPvL2PiwmxOTq3N/35/Bt0zjvc7LGOMqZAlAg9My9rKfe8s4fucPG49pwV/PLfVD/2BjDEm6CwRVMGO/bnc/84Spi/ZRvvG9Xh1yGm0b1zf77CMMaZSLBEcBVVlwoJsHnl/KbkFIYb3a8Pvzvp5kzhjjIkFlggqadOeHO6ZnMXsVbs4LeM4nhjUiRapdfwOyxhjjpolApcKQ8prX63n6RkrEOCRAe0Z3LMZ1axJnDEmxkVsW4aINBWRT0VkmYgsEZHbS5lHROQfIrJaRDJFpGuk4qmK1TsO8Mt/fcVD7y3ltIzjmfHnPlx/RoYlgapaNx6mZMDr1ZzrdeP9jsiYhBTJNYIC4P9UdaGI1AUWiMhHqrq02DwXA63Cl57AmPB1IOQXhvjXZ2v4x8erSamRxDO/7MzAU5tYkzgvWHtiYwIjYolAVbcCW8O3D4jIMqAJUDwRDABeU+fw5q9F5FgROSn8XF8t3ryPv0zIZNnW/Vza8SQe7N+e1Lo1/A4rflh7YmMCIyr7CEQkAzgVmFvioSbApmL3s8PTfpIIRGQoMBQgPcJnas/NL+TZmat4afZajq99DC9e141+HU6M6GsmJGtPbExgRDwRiEgdYCLwJ1XdX/LhUp7ys+ZHqjoWGAtOryHPgwybt24PIyZmsnbXIa7u3pR7LjmF+inVI/Vyic3aExsTGBFNBCJSHScJjFfVSaXMkg00LXY/DdgSyZhKcyA3n6emr+C/X28g7bhajLupJ71bNYx2GIml86if7iMAa09sjE8ilgjE2aP6CrBMVZ8pY7Z3gT+IyJs4O4n3RXv/wKcrdjByUhZb9+cypFdz7ryoNSnHWFVtxFl7YmMCI5LfeL2A64EsEfkuPO0eIB1AVV8EpgKXAKuBHODGCMbzE98fyuOR95cy6dvNtGxUhwk3n0m3ZsdF6+UNOF/69sVvjO8iWTX0BaXvAyg+jwK3RiqGMl6TD7K28sA7S9h3OJ/bzm3Jree2pEayNYkzxiSmhNoGsn1/LvdNWcyHS7fTsUl9xv22J6ecVM/vsIwxxlcJkwg+Xb6D2978lryCEHdf3Jabejcn2ZrEGWNM4iSC5g1r0zX9OB7s357mDWv7HY4xxgRGwiSCjIa1eXVID7/DMMaYwLFtI8YYk+AsERhjTIKzRGCMMQnOEoExxiQ4SwTGGJPgLBEYY0yCs0RgjDEJzhKBMcYkOHH6vsUOEdkJlHJGE1caArs8DMdPNpZgipexxMs4wMZSpJmqppb2QMwlgqoQkfmq2t3vOLxgYwmmeBlLvIwDbCxu2KYhY4xJcJYIjDEmwSVaIhjrdwAesrEEU7yMJV7GATaWCiXUPgJjjDE/l2hrBMYYY0qwRGCMMQkubhOBiCSJyLci8n4pj9UQkbdEZLWIzBWRjOhH6F4FY7lBRHaKyHfhy2/9iNENEVkvIlnhOOeX8riIyD/Cn0umiHT1I86KuBjH2SKyr9hncr8fcbohIseKyAQRWS4iy0TkjBKPx8RnAq7GEhOfi4i0KRbjdyKyX0T+VGIeTz+XeD5D2e3AMqC0s9PfBHyvqi1F5BrgSeDqaAZXSeWNBeAtVf1DFOOpinNUtawDYi4GWoUvPYEx4esgKm8cALNV9bKoRXP0ngOmq+qVInIMkFLi8Vj6TCoaC8TA56KqK4Au4PwIBDYDk0vM5unnEpdrBCKSBlwKvFzGLAOAV8O3JwDniYhEI7bKcjGWeDIAeE0dXwPHishJfgcVr0SkHtAHeAVAVfNUdW+J2WLiM3E5llh0HrBGVUt2U/D0c4nLRAA8CwwHQmU83gTYBKCqBcA+oEF0Qqu0isYCMCi8ejhBRJpGKa6jocCHIrJARIaW8vgPn0tYdnha0FQ0DoAzRGSRiEwTkfbRDK4STgZ2Av8Jb3p8WURql5gnVj4TN2OB2PhcirsGeKOU6Z5+LnGXCETkMmCHqi4ob7ZSpgWujtblWN4DMlS1EzCTH9d0gqiXqnbFWa29VUT6lHg8Jj4XKh7HQpy+Lp2BfwJToh2gS8lAV2CMqp4KHAJGlJgnVj4TN2OJlc8FgPDmrf7A/0p7uJRpR/25xF0iAHoB/UVkPfAmcK6IjCsxTzbQFEBEkoH6wJ5oBulShWNR1d2qeiR89yWgW3RDdE9Vt4Svd+Bs8+xRYpYfPpewNGBLdKJzr6JxqOp+VT0Yvj0VqC4iDaMeaMWygWxVnRu+PwHny7TkPIH/THAxlhj6XIpcDCxU1e2lPObp5xJ3iUBV71bVNFXNwFmt+kRVrysx27vAb8K3rwzPE7hfOW7GUmK7YH+cncqBIyK1RaRu0W3gQmBxidneBX4drog4HdinqlujHGq53IxDRE4s2uckIj1w/s92RzvWiqjqNmCTiLQJTzoPWFpitsB/JuBuLLHyuRTzK0rfLAQefy7xXDX0EyLyMDBfVd/F2aH0XxFZjbMmcI2vwVVSibHcJiL9gQKcsdzgZ2zlOAGYHP4/TAZeV9XpInIzgKq+CEwFLgFWAznAjT7FWh4347gSuEVECoDDwDVB/KER9kdgfHgzxFrgxhj8TIpUNJaY+VxEJAW4APh9sWkR+1ysxYQxxiS4uNs0ZIwxpnIsERhjTIKzRGCMMQnOEoExxiQ4SwTGGJPgLBEYU0nhLpaldYItdboHr3e5iLQrdn+WiMTFydhNMFgiMCb4LgfaVTiXMUfJEoGJO+Gjfz8INxdbLCJXh6d3E5HPws3iZhQdlR3+hf2siHwZnr9HeHqP8LRvw9dtynvdUmL4t4h8E37+gPD0G0RkkohMF5FVIvJUsefcJCIrw/G8JCKjReRMnCPGnxanN32L8OxXici88PxnefTWmQSVMEcWm4TSD9iiqpcCiEh9EamO02hsgKruDCeHUcCQ8HNqq+qZ4QZy/wY6AMuBPqpaICLnA48Bg1zGMBKnJcgQETkWmCciM8OPdQFOBY4AK0Tkn0AhcB9Of5wDwCfAIlX9UkTeBd5X1Qnh8QAkq2oPEbkEeAA4/2jeKGPAEoGJT1nAX0XkSZwv0Nki0gHny/2j8BdpElC8N8sbAKr6uYjUC3951wVeFZFWOJ0dq1cihgtxGgbeGb5fE0gP3/5YVfcBiMhSoBnQEPhMVfeEp/8PaF3O8ieFrxcAGZWIy5ifsURg4o6qrhSRbji9WB4XkQ9xuoQuUdUzynpaKfcfAT5V1YHinM50ViXCEGBQ+GxTP04U6YmzJlCkEOf/sLInRipaRtHzjTlqto/AxB0RaQzkqOo44K84m1tWAKkSPo+tiFSXn56YpGg/Qm+cTo77cNqTbw4/fkMlw5gB/LFYt8tTK5h/HtBXRI4TpzV68U1QB3DWToyJCPslYeJRR5ydqyEgH7hFVfNE5ErgHyJSH+dv/1lgSfg534vIlzjnhS7ab/AUzqahO3C22VfGI+HlZ4aTwXqgzHPlqupmEXkMmIvTV34pzpnzwDkXxUsichtOB01jPGXdR03CE5FZwJ2qOt/nOOqo6sHwGsFk4N+qWvKk5cZ4zjYNGRMcD4rIdzgnullHwE+laOKHrREYY0yCszUCY4xJcJYIjDEmwVkiMMaYBGeJwBhjEpwlAmOMSXD/Hyeq4RAZabQNAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "x_ponits = np.arange(4, 8)\n", "y_ = -(clf.coef_[0][0]*x_ponits + clf.intercept_)/clf.coef_[0][1]\n", "plt.plot(x_ponits, y_)\n", "\n", "plt.plot(X[:50, 0], X[:50, 1], 'bo', color='blue', label='0')\n", "plt.plot(X[50:, 0], X[50:, 1], 'bo', color='orange', label='1')\n", "plt.xlabel('sepal length')\n", "plt.ylabel('sepal width')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 最大熵模型" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "import math\n", "from copy import deepcopy" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "class MaxEntropy:\n", " def __init__(self, EPS=0.005):\n", " self._samples = []\n", " self._Y = set() # 标签集合，相当去去重后的y\n", " self._numXY = {} # key为(x,y)，value为出现次数\n", " self._N = 0 # 样本数\n", " self._Ep_ = [] # 样本分布的特征期望值\n", " self._xyID = {} # key记录(x,y),value记录id号\n", " self._n = 0 # 特征键值(x,y)的个数\n", " self._C = 0 # 最大特征数\n", " self._IDxy = {} # key为(x,y)，value为对应的id号\n", " self._w = []\n", " self._EPS = EPS # 收敛条件\n", " self._lastw = [] # 上一次w参数值\n", "\n", " def loadData(self, dataset):\n", " self._samples = deepcopy(dataset)\n", " for items in self._samples:\n", " y = items[0]\n", " X = items[1:]\n", " self._Y.add(y) # 集合中y若已存在则会自动忽略\n", " for x in X:\n", " if (x, y) in self._numXY:\n", " self._numXY[(x, y)] += 1\n", " else:\n", " self._numXY[(x, y)] = 1\n", "\n", " self._N = len(self._samples)\n", " self._n = len(self._numXY)\n", " self._C = max([len(sample) - 1 for sample in self._samples])\n", " self._w = [0] * self._n\n", " self._lastw = self._w[:]\n", "\n", " self._Ep_ = [0] * self._n\n", " for i, xy in enumerate(self._numXY): # 计算特征函数fi关于经验分布的期望\n", " self._Ep_[i] = self._numXY[xy] / self._N\n", " self._xyID[xy] = i\n", " self._IDxy[i] = xy\n", "\n", " def _Zx(self, X): # 计算每个Z(x)值\n", " zx = 0\n", " for y in self._Y:\n", " ss = 0\n", " for x in X:\n", " if (x, y) in self._numXY:\n", " ss += self._w[self._xyID[(x, y)]]\n", " zx += math.exp(ss)\n", " return zx\n", "\n", " def _model_pyx(self, y, X): # 计算每个P(y|x)\n", " zx = self._Zx(X)\n", " ss = 0\n", " for x in X:\n", " if (x, y) in self._numXY:\n", " ss += self._w[self._xyID[(x, y)]]\n", " pyx = math.exp(ss) / zx\n", " return pyx\n", "\n", " def _model_ep(self, index): # 计算特征函数fi关于模型的期望\n", " x, y = self._IDxy[index]\n", " ep = 0\n", " for sample in self._samples:\n", " if x not in sample:\n", " continue\n", " pyx = self._model_pyx(y, sample)\n", " ep += pyx / self._N\n", " return ep\n", "\n", " def _convergence(self): # 判断是否全部收敛\n", " for last, now in zip(self._lastw, self._w):\n", " if abs(last - now) >= self._EPS:\n", " return False\n", " return True\n", "\n", " def predict(self, X): # 计算预测概率\n", " Z = self._Zx(X)\n", " result = {}\n", " for y in self._Y:\n", " ss = 0\n", " for x in X:\n", " if (x, y) in self._numXY:\n", " ss += self._w[self._xyID[(x, y)]]\n", " pyx = math.exp(ss) / Z\n", " result[y] = pyx\n", " return result\n", "\n", " def train(self, maxiter=1000): # 训练数据\n", " for loop in range(maxiter): # 最大训练次数\n", " print(\"iter:%d\" % loop)\n", " self._lastw = self._w[:]\n", " for i in range(self._n):\n", " ep = self._model_ep(i) # 计算第i个特征的模型期望\n", " self._w[i] += math.log(self._Ep_[i] / ep) / self._C # 更新参数\n", " print(\"w:\", self._w)\n", " if self._convergence(): # 判断是否收敛\n", " break" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "dataset = [['no', 'sunny', 'hot', 'high', 'FALSE'],\n", " ['no', 'sunny', 'hot', 'high', 'TRUE'],\n", " ['yes', 'overcast', 'hot', 'high', 'FALSE'],\n", " ['yes', 'rainy', 'mild', 'high', 'FALSE'],\n", " ['yes', 'rainy', 'cool', 'normal', 'FALSE'],\n", " ['no', 'rainy', 'cool', 'normal', 'TRUE'],\n", " ['yes', 'overcast', 'cool', 'normal', 'TRUE'],\n", " ['no', 'sunny', 'mild', 'high', 'FALSE'],\n", " ['yes', 'sunny', 'cool', 'normal', 'FALSE'],\n", " ['yes', 'rainy', 'mild', 'normal', 'FALSE'],\n", " ['yes', 'sunny', 'mild', 'normal', 'TRUE'],\n", " ['yes', 'overcast', 'mild', 'high', 'TRUE'],\n", " ['yes', 'overcast', 'hot', 'normal', 'FALSE'],\n", " ['no', 'rainy', 'mild', 'high', 'TRUE']]" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "maxent = MaxEntropy()\n", "x = ['overcast', 'mild', 'high', 'FALSE']" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "iter:0\n", "w: [0.0455803891984887, -0.002832177999673058, 0.031103560672370825, -0.1772024616282862, -0.0037548445453157455, 0.16394435955437575, -0.02051493923938058, -0.049675901430111545, 0.08288783767234777, 0.030474400362443962, 0.05913652210443954, 0.08028783103573349, 0.1047516055195683, -0.017733409097415182, -0.12279936099838235, -0.2525211841208849, -0.033080678592754015, -0.06511302013721994, -0.08720030253991244]\n", "iter:1\n", "w: [0.11525071899801315, 0.019484939219927316, 0.07502777039579785, -0.29094979172869884, 0.023544184009850026, 0.2833018051925922, -0.04928887087664562, -0.101950931659509, 0.12655289130431963, 0.016078718904129236, 0.09710585487843026, 0.10327329399123442, 0.16183727320804359, 0.013224083490515591, -0.17018583153306513, -0.44038644519804815, -0.07026660158873668, -0.11606564516054546, -0.1711390483931799]\n", "iter:2\n", "w: [0.18178907332733973, 0.04233703122822168, 0.11301330241050131, -0.37456674484068975, 0.05599764270990431, 0.38356978711239126, -0.07488546168160945, -0.14671211613144097, 0.15633348706002106, -0.011836411721359321, 0.12895826039781944, 0.10572969681821211, 0.19953102749655352, 0.06399991656546679, -0.17475388854415905, -0.5893308194447993, -0.10405912653008922, -0.16350962040062977, -0.24701967386590512]\n", "iter:3\n", "w: [0.2402117261976856, 0.06087651054892573, 0.14300856884173724, -0.44265412294427664, 0.08623192206158618, 0.47264512563925376, -0.09600090083002198, -0.18353847640798293, 0.17967535014110475, -0.04398112111909075, 0.15854994616895085, 0.09937760679990165, 0.22754399461146121, 0.12138068016302067, -0.15616500410638443, -0.7136213594089919, -0.13342640817803014, -0.2097936229338585, -0.3153356710047331]\n", "iter:4\n", "w: [0.2914313208012359, 0.07547654306538813, 0.16668283431764536, -0.5013655616789854, 0.1129176109082406, 0.553725824276617, -0.11340104779016447, -0.214026170852028, 0.19932565541497924, -0.07698174342694904, 0.18676347888513212, 0.08897527479225055, 0.250034281885875, 0.17966909953761648, -0.12561912266368833, -0.8214131440732644, -0.15887039192864807, -0.255021849396353, -0.3775163032854775]\n", "iter:5\n", "w: [0.3371038340609469, 0.08719816942080917, 0.1858885244336221, -0.5536101929687616, 0.13629778855333752, 0.6284587190599515, -0.12800357294309486, -0.23983404211792342, 0.21652676634149073, -0.10944257416223822, 0.2137132093479417, 0.07676820672685672, 0.2690824414813502, 0.2363909590693551, -0.0894456215757756, -0.9176374337279947, -0.18113135827470755, -0.298867529863144, -0.43486330681003177]\n", "iter:6\n", "w: [0.3785824456804424, 0.09688384478129128, 0.2020182323803342, -0.6009874705178111, 0.15692184161636785, 0.6978719259357552, -0.14051015547758733, -0.26225964542470354, 0.231946295562788, -0.14075188805495795, 0.23936253047337575, 0.06390380813674021, 0.2858409112894462, 0.290497131241793, -0.05118245076440544, -1.0054122371529666, -0.20087035680546067, -0.34104258966535955, -0.4883751534969831]\n", "iter:7\n", "w: [0.41683868296814264, 0.10509553912347498, 0.2160232475329699, -0.644505058740366, 0.1753178575881968, 0.7626984543065831, -0.1514056957963993, -0.282232009175799, 0.24599431757478898, -0.17065092475910307, 0.26368621835608597, 0.0509851487070869, 0.3009923148407335, 0.3416039921608126, -0.012757343850577675, -1.0867762087449189, -0.218615635191165, -0.3813835600318599, -0.5387838195368846]\n", "iter:8\n", "w: [0.45255283727071915, 0.11219377184169538, 0.2285350985725094, -0.6848643375624139, 0.19192374296509346, 0.823505441985143, -0.1610195381343653, -0.3003952716953118, 0.2589475960707626, -0.19905447303514948, 0.28670003432375235, 0.03832451391813461, 0.3149608328705212, 0.38965093728157896, 0.024829827991967576, -1.1631007696620375, -0.234775223274346, -0.4198375433999549, -0.5866232900677824]\n", "iter:9\n", "w: [0.4862047341699185, 0.11841188281430236, 0.23997121190141055, -0.7225858897027357, 0.20708669598089463, 0.8807531715193149, -0.16957936712401173, -0.3171924798474509, 0.27100498873946605, -0.2259668872025463, 0.30845455112675946, 0.02607293739294267, 0.3280204662447941, 0.43473488104744773, 0.06110854499605007, -1.235334453720919, -0.2496614393236233, -0.4564297086590982, -0.632287633902211]\n", "iter:10\n", "w: [0.5181394259405306, 0.12390610358940803, 0.25060941880919724, -0.7580715708198793, 0.22107810865652675, 0.9348258037844982, -0.17724821171219282, -0.3329298610812439, 0.28231538356745023, -0.25143909184677066, 0.3290224934515896, 0.01429111840050851, 0.34035457030022254, 0.4770259468712468, 0.09590445328801561, -1.3041525627362545, -0.2635134591021718, -0.4912340995712618, -0.676073174743828]\n", "iter:11\n", "w: [0.5486110820346487, 0.12878621492610728, 0.260637166739537, -0.7916392134506605, 0.23410952494004347, 0.9860494937682006, -0.18414786231688304, -0.34782183358804564, 0.29299341588094513, -0.2755452075588528, 0.3484877282959149, 0.0029897746989391953, 0.3520901674994048, 0.5167232266096159, 0.12920353885993147, -1.3700506081491441, -0.2765153028028459, -0.5243519897715138, -0.7182076215567675]\n", "iter:12\n", "w: [0.5778117345209275, 0.13313363017604996, 0.27018325256816517, -0.8235442751999533, 0.24634619366524757, 1.0347042729774998, -0.19037323865834888, -0.36202132448933444, 0.3031291458576307, -0.2983694137224091, 0.36693744589848354, -0.007847001125305455, 0.36331836466580314, 0.5540314435560996, 0.16107375873385849, -1.43340343709795, -0.2888095132477581, -0.5558971523336302, -0.7588699023545793]\n", "iter:13\n", "w: [0.6058901351026187, 0.13701191167326482, 0.27933812605788444, -0.8539945297938123, 0.2579178021404349, 1.0810325152985145, -0.19600113559558124, -0.3756398474403068, 0.3127944422531689, -0.31999848717048135, 0.38445717638507615, -0.01824819811068835, 0.374106791262247, 0.5891489137038122, 0.19162049156992134, -1.494503092195292, -0.3005073854549682, -0.5859863925416857, -0.7982036819948133]\n", "iter:14\n", "w: [0.6329642597528577, 0.14047287909653053, 0.288166877587425, -0.8831606263070614, 0.2689267548911857, 1.1252453301017262, -0.2010955813068598, -0.38876072186638005, 0.32204740845586133, -0.3405176107915853, 0.40112790161689965, -0.02824811730597898, 0.38450732666115006, 0.6222617230970029, 0.22096178876573108, -1.553583366887057, -0.3116965772848507, -0.614733784830676, -0.8363266559598186]\n", "iter:15\n", "w: [0.6591297272306862, 0.14356019142928794, 0.29671758022981304, -0.9111839584847241, 0.27945449028395125, 1.1675275789036936, -0.2057111806049296, -0.40144778446237106, 0.3309355607379767, -0.3600081062970341, 0.417024586229216, -0.0378820228099265, 0.3945609896455332, 0.6535413341943247, 0.2492151298378952, -1.6108359714954945, -0.322446770130228, -0.6422474013311557, -0.8733370228190042]\n", "iter:16\n", "w: [0.6844655652033881, 0.14631149268101387, 0.3050266859363578, -0.9381826602314054, 0.28956628018821073, 1.208041909775652, -0.20989526326523208, -0.4137511617306136, 0.3394981675496673, -0.37854630883233975, 0.43221561587913865, -0.047183815928172015, 0.40430109093285005, 0.6831440602226162, 0.27649078947951705, -1.6664213645385508, -0.3328138884351746, -0.6686276609175331, -0.9093180606867393]\n", "iter:17\n", "w: [0.7090382322303439, 0.14875974316242174, 0.3131225496097004, -0.9642562360118443, 0.29931487416739566, 1.2469320534764214, -0.2136893199603249, -0.42571112990484933, 0.34776800227139515, -0.39620311776233963, 0.4467627857364783, -0.05618489532677288, 0.41375530711155634, 0.7112115242872562, 0.30288894318648024, -1.7204761577827474, -0.3428432562746063, -0.6939666992818952, -0.9443414163459253]\n", "iter:18\n", "w: [0.7329044755919177, 0.15093408736171834, 0.3210277580307024, -0.9894891689757307, 0.30874326974092275, 1.2843255450191382, -0.2171300078491035, -0.43736072740088694, 0.35577267425006043, -0.41304394260476845, 0.4607216021549229, -0.06491370074276534, 0.422947075247345, 0.7378716028031354, 0.328498833440739, -1.7731182942183688, -0.3525719694978748, -0.7183483589746492, -0.9784695091163789]\n", "iter:19\n", "w: [0.7561133995570936, 0.15286045515640212, 0.328760690901024, -1.0139537508210092, 0.31788682371577265, 1.3203359878921699, -0.22024989032144238, -0.44872754933949704, 0.3635356504637764, -0.4291288733368194, 0.47414174564472794, -0.07339563949692657, 0.4318965531986031, 0.7632395717459113, 0.35339902389681255, -1.8244507576736042, -0.36203068772373537, -0.7418485369835647, -1.0117573172957148]\n", "iter:20\n", "w: [0.7787079892539899, 0.15456200602459644, 0.33633658450314596, -1.0377123132497756, 0.3267748670740448, 1.3550649489120608, -0.2230780070832732, -0.4598350016169348, 0.3710770468508463, -0.44451297113313276, 0.487067601013812, -0.08165321702581137, 0.44062129776518627, 0.7874193002477285, 0.37765818433851467, -1.8745642981534274, -0.3712449958974367, -0.7645357216530283, -1.0442537275202144]\n", "iter:21\n", "w: [0.8007262529140036, 0.1560594772059649, 0.34376827002154436, -1.060818996362751, 0.3354319455237139, 1.3886035518622473, -0.22564033126798433, -0.4707031942919707, 0.3784142461967869, -0.4592466165100184, 0.4995387984981119, -0.08970626642269312, 0.44913675622087285, 0.8105044087079029, 0.4013360928412452, -1.9235394849020984, -0.3802364452138948, -0.7864716135654638, -1.0760025700060472]\n", "iter:22\n", "w: [0.8222020921833231, 0.15737146972957614, 0.3510666965898703, -1.0833211570708576, 0.34387877721624305, 1.4210338238029174, -0.2279601474838501, -0.4813495913406772, 0.38556238422571915, -0.4733758770600258, 0.5115907340536995, -0.09757221648482109, 0.45745663199363035, 0.83257935031115, 0.424484683802042, -1.971448290683485, -0.38902335410334277, -0.8077117662399449, -1.1070434246350234]\n", "iter:23\n", "w: [0.8431659755582256, 0.15851469135106674, 0.3582413099761185, -1.1052604965753057, 0.35213299614000754, 1.4524298372162394, -0.2300583715773359, -0.4917894935106267, 0.3925347347639661, -0.48694287254911917, 0.5232550522617134, -0.10526636447823189, 0.46559316365873105, 0.8537203985625873, 0.4471490528467217, -2.0183553418422466, -0.3976214287564327, -0.8283062091823951, -1.1374122574144727]\n", "iter:24\n", "w: [0.8636454670864389, 0.15950416732792658, 0.3653003329720429, -1.1266739678034468, 0.360209732509703, 1.482858682861421, -0.2319538253007387, -0.5020364049989126, 0.399343017076967, -0.49998612479593435, 0.5345600846495382, -0.11280213572953302, 0.47355734286275886, 0.8739965366602681, 0.46936837668843867, -2.064318924528685, -0.406044247183912, -0.8483000328258562, -1.1671419296079542]\n", "iter:25\n", "w: [0.8836656471038739, 0.1603534255734902, 0.3722509777005583, -1.1475945098423088, 0.36812206865863517, 1.5123813016864462, -0.23366347464932266, -0.5121023178222437, 0.4059976428045398, -0.5125408857080916, 0.5455312417781026, -0.12019132149050894, 0.48135908820857093, 0.8934702516155534, 0.491176732355382, -2.109391808862334, -0.41430363948916343, -0.8677339255435279, -1.1962626100073512]\n", "iter:26\n", "w: [0.9032494513148862, 0.16107466037129672, 0.3790996098764867, -1.1680516458111565, 0.375881399398912, 1.5410531989440968, -0.23520263801690178, -0.521997936755754, 0.412507915726925, -0.5246394405006506, 0.5561913605853828, -0.12744429189123818, 0.48900738662403054, 0.9121982395725802, 0.5126038144468337, -2.1536219340679157, -0.42240998873736496, -0.8866446594233551, -1.2248021125938482]\n", "iter:27\n", "w: [0.922417947304039, 0.1616788775625432, 0.38585187847365326, -1.1880719724770439, 0.38349771870651894, 1.5689250594879982, -0.2365851687319244, -0.5317328604974513, 0.41888219447126124, -0.5363113853272858, 0.5665610101206564, -0.13457018378364388, 0.49651041016127917, 0.9302320302856513, 0.5336755563163902, -2.197052985176947, -0.43037247070539114, -0.9050655253342557, -1.2527861760416592]\n", "iter:28\n", "w: [0.9411905625141772, 0.16217602342568374, 0.39251281995973497, -1.2076795636593056, 0.3909798492859816, 1.5960432798599258, -0.2378236155362006, -0.5413157299328164, 0.4251280259261702, -0.5475838798535487, 0.5766587595378964, -0.14157706484455113, 0.5038756138194018, 0.9476185390802628, 0.5544146643308008, -2.2397248835085883, -0.43819924629378326, -0.9230267199596871, -1.2802386973829956]\n", "iter:29\n", "w: [0.9595852741409165, 0.1625750990706656, 0.3990869434363314, -1.2268963046481758, 0.39833562760535696, 1.6224504300557419, -0.23892936390516475, -0.550754351175396, 0.43125225536528544, -0.5584818760317878, 0.5865014124117355, -0.14847207604172713, 0.5111098184106876, 0.9644005543745499, 0.5748410753847142, -2.281674207406961, -0.4458976170440247, -0.9405556885568458, -1.3071819291635138]\n", "iter:30\n", "w: [0.9776187688173722, 0.16288426192335587, 0.40557830114418625, -1.2457421711424088, 0.4055720540203952, 1.6481856556449563, -0.2399127606410839, -0.5600557988963532, 0.43726111794043027, -0.5690283247125373, 0.5961042113291977, -0.1552615548230238, 0.5182192814229061, 0.9806171682798118, 0.5949723477012533, -2.322934555715983, -0.45347415171844135, -0.9576774276294637, -1.3336366472304257]\n", "iter:31\n", "w: [0.9953065780883319, 0.16311091570219172, 0.4119905475305536, -1.2642354633336523, 0.41269541537110693, 1.6732850291165284, -0.24078322381995113, -0.5692265039754737, 0.4431603151835325, -0.5792443618829073, 0.6054810164393999, -0.161951141381158, 0.5252097580923409, 0.9963041571084452, 0.6148239942141389, -2.3635368636365333, -0.4609347900399002, -0.9744147517426334, -1.3596222946768448]\n", "iter:32\n", "w: [1.0126631942929332, 0.1632617901510506, 0.41832698921506856, -1.2823930035348632, 0.4197113857459025, 1.6977818578508115, -0.2415493398948393, -0.578272328478973, 0.44895507937375845, -0.5891494763448589, 0.614644461299508, -0.1685458702019946, 0.5320865543832194, 1.011494317894958, 0.634409766848547, -2.403509678558329, -0.4682849282896805, -0.9907885285272439, -1.3851571062550208]\n", "iter:33\n", "w: [1.0297021704475329, 0.16334301167193746, 0.4245906275954937, -1.3002303040160663, 0.42662510982864776, 1.7217069549105917, -0.2422189495285712, -0.5871986302462173, 0.4546502280239552, -0.5987616605900034, 0.6236060889903036, -0.17505024889696633, 0.5388545731968212, 1.026217766334997, 0.6537418989819439, -2.4428794019441593, -0.4755294904046779, -1.0068178856250027, -1.4102582166559912]\n", "iter:34\n", "w: [1.046436206947735, 0.16336016589063823, 0.4307841954168563, -1.31776171035854, 0.4334412722694842, 1.7450888778519553, -0.24279922353275318, -0.5960103188500857, 0.46025021027460095, -0.6080975465274945, 0.6323764711182516, -0.18146832609308042, 0.5455183548611607, 1.0405022008958331, 0.6728313123705808, -2.481670502204123, -0.4826729874119255, -1.0225203929778097, -1.4349417553550248]\n", "iter:35\n", "w: [1.0628772273216864, 0.1633183530852298, 0.43691018832902084, -1.3350005245804004, 0.4401641557749125, 1.767954139937502, -0.24329673012045083, -0.6047119043193363, 0.46575914662142326, -0.61717252759963, 0.6409653119885281, -0.1878037499302339, 0.552082112748962, 1.0543731372612555, 0.6916877939185442, -2.519905702623322, -0.48971956742505873, -1.0379122235066314, -1.4592229301848803]\n", "iter:36\n", "w: [1.0790364448169805, 0.1632222373106267, 0.4429708922407209, -1.3519591114555574, 0.44679769003991354, 1.790327397457484, -0.24371749453115982, -0.6133075397327731, 0.4711808631201613, -0.6260008686895925, 0.6493815399322238, -0.1940598185072494, 0.5585497647171871, 1.0678541167471884, 0.7103201468616923, -2.5576061477240772, -0.49667305795648603, -1.0530082948776114, -1.4831161013818943]\n", "iter:37\n", "w: [1.0949244212558502, 0.16307608996234713, 0.4489684071173446, -1.368648990792826, 0.45334549320271567, 1.8122316163026093, -0.24406705195602538, -0.6218010585852058, 0.47651892099016024, -0.6345958050957102, 0.657633387503034, -0.2002395234304271, 0.5649249609418182, 1.0809668918596087, 0.7287363202401675, -2.5947915509024004, -0.503537001935657, -1.0678223947291068, -1.5066348475237694]\n", "iter:38\n", "w: [1.1105511193224098, 0.16288382843968888, 0.4549046677485249, -1.3850809179233177, 0.45981090715981504, 1.8336882204643785, -0.2443504945761886, -0.630196007664431, 0.4817766423636429, -0.642969631721608, 0.6657284620235139, -0.20634558745295647, 0.5712111086274904, 1.09373159175908, 0.7469435199341006, -2.631480325742677, -0.5103146885419312, -1.0823672914383413, -1.5297920245209762]\n", "iter:39\n", "w: [1.1259259492329807, 0.1626490504926992, 0.4607814619184564, -1.4012649542319064, 0.46619702781148514, 1.8547172247479498, -0.24457251342586653, -0.6384956760532191, 0.48695713278899466, -0.651133783514366, 0.6736738077562269, -0.21238049704962586, 0.5774113939963701, 1.1061668700409768, 0.7649483040277117, -2.66768970305921, -0.5170091797407274, -1.0966548322421525, -1.5525998186198335]\n", "iter:40\n", "w: [1.1410578105722688, 0.16237506476868643, 0.46660044633909475, -1.4172105292393644, 0.47250673109858604, 1.8753373536548952, -0.24473743570277745, -0.6467031207711921, 0.49206330098614925, -0.6590989080746382, 0.6814759607979815, -0.21834653064726997, 0.5835288018997601, 1.1182900369315123, 0.7827566648361761, -2.7034358354202697, -0.5236233332393198, -1.1106960302923272, -1.575069794212521]\n", "iter:41\n", "w: [1.1559551299441309, 0.16206491801235573, 0.47236316064862144, -1.432926495477744, 0.47874269552664483, 1.8955661481177095, -0.2448492580704435, -0.6548211894919176, 0.49709787626398166, -0.6668749312640252, 0.68914099764274, -0.22424578312520774, 0.5895661333459766, 1.130117177731465, 0.8003740995664982, -2.7387338906663863, -0.5301598224428798, -1.1245011420204416, -1.5972129371184205]\n", "iter:42\n", "w: [1.1706258949781938, 0.16172141931861386, 0.4780710397310232, -1.4484211771889233, 0.4849074217428049, 1.9154200615355261, -0.24491167642852157, -0.6628525407058644, 0.5020634239387433, -0.6744711165457573, 0.6966745782276667, -0.23008018710936537, 0.595526021196693, 1.1416632591068792, 0.8178056712782106, -2.773598135729871, -0.5366211538838394, -1.1380797360075168, -1.6190396938953464]\n", "iter:43\n", "w: [1.185077685146832, 0.1613471617882285, 0.4837254245746624, -1.4637024137041252, 0.4910032496276282, 1.9349145463638193, -0.24492811256751953, -0.6707996616466448, 0.506962359035349, -0.6818961187145012, 0.7040819841643154, -0.23585153150675092, 0.601410944249386, 1.1529422246243355, 0.835056061553053, -2.8080420118894813, -0.543009682512066, -1.1514407543974905, -1.6405600076531104]\n", "iter:44\n", "w: [1.1993176997765405, 0.16094454189335364, 0.4893275718563842, -1.4787775982223486, 0.4970323732809923, 1.954064132343929, -0.2449017380720393, -0.6786648842537609, 0.5117969585068955, -0.6891580325992614, 0.7113681527605952, -0.24156147766152572, 0.6072232398944304, 1.163967080753521, 0.8521296160671726, -2.842078202446987, -0.5493276251648829, -1.1645925677578381, -1.661783350771384]\n", "iter:45\n", "w: [1.2133527835801412, 0.16051577682191626, 0.49487866241160106, -1.4936537125914016, 0.5029968542149327, 1.9728824973161108, -0.2448354957911386, -0.6864503994080099, 0.5165693721699401, -0.6962644372592381, 0.7185377073566026, -0.24721157345839398, 0.6129651155107275, 1.174749974408159, 0.8690303840781799, -2.875718693686637, -0.5555770724809243, -1.1775430241731384, -1.6827187548658014]\n", "iter:46\n", "w: [1.227189449988899, 0.16006292003648345, 0.5003798087288612, -1.5083373586015465, 0.5088986330127642, 1.9913825314384455, -0.24473211915470333, -0.6941582696444135, 0.52128163252208, -0.7032224361354144, 0.725594984426699, -0.2528032656519123, 0.6186386587427478, 1.1853022629646983, 0.8857621526870053, -2.9089748298705667, -0.5617599994773589, -1.190299493255645, -1.7033748382982068]\n", "iter:47\n", "w: [1.2408339025244393, 0.1595878752539907, 0.5058320615888524, -1.5228347862245477, 0.5147395396704653, 2.0095763955300066, -0.2445941495801336, -0.7017904405209363, 0.5259356635828739, -0.7100386935699113, 0.7325440578395995, -0.2583379106603215, 0.6242458467838733, 1.1956345775838009, 0.9023284766062019, -2.941857362930209, -0.5678782749739237, -1.2028689056689232, -1.7237598314869575]\n", "iter:48\n", "w: [1.2542920544177703, 0.15909240902714533, 0.511236415952003, -1.5371519191670293, 0.5205213028010427, 2.027475574167438, -0.2444239521835253, -0.7093487507985308, 0.5305332888780785, -0.7167194680600958, 0.7393887606162666, -0.2638167840285218, 0.629788554775462, 1.2057568805602186, 0.918732704058188, -2.97437649743339, -0.573933670017777, -1.215257788684493, -1.7438816002397186]\n", "iter:49\n", "w: [1.2675695466554302, 0.15857816208598835, 0.5165938161853348, -1.5512943780539223, 0.526245557853767, 2.0450909240872455, -0.2442237299832945, -0.7168349415685342, 0.5350762386696887, -0.7232706425744582, 0.7461327044807721, -0.2692410887359195, 0.6352685634177097, 1.2156785173400755, 0.9349779993356275, -3.0065419313371287, -0.5799278654390619, -1.2274722982255624, -1.7637476673030004]\n", "iter:50\n", "w: [1.2806717646098413, 0.15804665957860914, 0.5219051607076248, -1.565267501513449, 0.531913854476506, 2.0624327183797524, -0.2439955367613434, -0.7242506644466232, 0.5395661565196902, -0.7296977522218349, 0.7527792974609447, -0.27461196250027875, 0.6406875658768324, 1.2254082637691288, 0.951067362478955, -3.0383628929756363, -0.5858624586472239, -1.2395182477949291, -1.7833652322988558]\n", "iter:51\n", "w: [1.2936038533915348, 0.15749932033301228, 0.5271713061219352, -1.5790763653978963, 0.5375276631298644, 2.079510686903264, -0.2437412887269575, -0.7315974889379809, 0.5440046052632351, -0.7360060095341238, 0.7593317597626166, -0.27993048420774796, 0.6460471740630701, 1.2349543690697453, 0.9670036464606849, -3.0698481746802933, -0.5917389697617063, -1.2514011346347527, -1.802741190198874]\n", "iter:52\n", "w: [1.3063707320444093, 0.1569374652473128, 0.5323930708959317, -1.5927258003431402, 0.543088381045685, 2.0963340532971406, -0.2434627751112681, -0.7388769090657148, 0.5483930724566909, -0.7422003275947768, 0.7657931381128499, -0.2851976795813397, 0.6513489243452854, 1.2443245949868127, 0.9827895722106955, -3.101006163382383, -0.5975588471570251, -1.2631261634230824, -1.8218821484682721]\n", "iter:53\n", "w: [1.3189771066910794, 0.1563623249025127, 0.537571238642919, -1.6062204078434033, 0.5485973376090173, 2.1129115689291647, -0.2431616678049766, -0.7460903493435844, 0.5527329753573567, -0.7482853412206436, 0.7721663187429895, -0.29041452618485153, 0.6565942827603267, 1.2535262514924854, 0.9984277417695611, -3.131844868509152, -0.6033234724908482, -1.2746982677748813, -1.8407944429980059]\n", "iter:54\n", "w: [1.3314274827232555, 0.155775046480836, 0.5427065610500206, -1.619564574995279, 0.5540557992314453, 2.1292515440748003, -0.24283953013882234, -0.7532391701645315, 0.5570256654843081, -0.7542654263828161, 0.7784540391611849, -0.2955819578461266, 0.6617846497686763, 1.26256622939559, 1.0139206498169098, -3.1623719474485377, -0.6090341652741876, -1.2861221297830006, -1.859484152929961]\n", "iter:55\n", "w: [1.3437261761216424, 0.15517670006276393, 0.5477997604942677, -1.6327624880459013, 0.5594649737742484, 2.1453618765929385, -0.24249782489472843, -0.7603246726681443, 0.5612724328035876, -0.7601447180326721, 0.7846588988456781, -0.30070086857233336, 0.6669213646020844, 1.2714510301628497, 1.0292706937875882, -3.1925947288270264, -0.6146921870348035, -1.297402197806485, -1.8779571144691578]\n", "iter:56\n", "w: [1.3558773239807893, 0.15456828436732212, 0.5528515323824308, -1.6458181448637803, 0.564826014571951, 2.1612500783338215, -0.24213792162546655, -0.7673481031428816, 0.5654745095756346, -0.7659271264820341, 0.7907833689742486, -0.3057721160203117, 0.6720057092437857, 1.280186793225137, 1.044480182759226, -3.22252023381794, -0.6202987451181533, -1.308542702689136, -1.8962189337671929]\n", "iter:57\n", "w: [1.3678848943064308, 0.153950731992666, 0.5578625472461193, -1.6587353664367566, 0.5701400241001218, 2.1769232994894803, -0.24176110335183842, -0.7743106570124639, 0.5696330738982798, -0.7716163524710106, 0.7968298012914021, -0.3107965245767515, 0.6770389120774131, 1.2887793210120908, 1.0595513452698089, -3.252155195673595, -0.6258549961644928, -1.3195476725699353, -1.9142749989526673]\n", "iter:58\n", "w: [1.3797526951459715, 0.15332491420744937, 0.5628334526199114, -1.6715178074894281, 0.5754080573256017, 2.1923883510747104, -0.24136857269862871, -0.7812134824502208, 0.5737492529747072, -0.7772159010434373, 0.8028004362028806, -0.31577488809583903, 0.6820221512367748, 1.2972341019321394, 1.074486336202476, -3.2815060776537246, -0.6313620492958532, -1.330420946428303, -1.9321304913768982]\n", "iter:59\n", "w: [1.3914843831067198, 0.15269164533771676, 0.5677648747269814, -1.6841689663019288, 0.5806311247725101, 2.207651725706976, -0.24096145752378464, -0.7880576836602355, 0.5778241261323951, -0.7827290943376856, 0.8086974101766186, -0.3207079723358566, 0.6869565576851888, 1.305556331491776, 1.0892872428563953, -3.310579089504004, -0.6368209690424527, -1.34116618649089, -1.9497903961366556]\n", "iter:60\n", "w: [1.4030834713111435, 0.15205168678902659, 0.5726574199938064, -1.6966921938028556, 0.5858101953332414, 2.2227196168362813, -0.2405408160893083, -0.7948443238597765, 0.5818587276161165, -0.7881590833897919, 0.8145227625201509, -0.3255965171309119, 0.6918432180499976, 1.3137509317274503, 1.10395609030679, -3.339380202622116, -0.6422327780345327, -1.3517868896123357, -1.9672595119298593]\n", "iter:61\n", "w: [1.4145533368337266, 0.15140575073909854, 0.5775116764130007, -1.709090702001243, 0.5909461988501297, 2.2375979365607077, -0.2401076418170941, -0.8015744279936724, 0.5858540491755332, -0.7935088590362205, 0.820278441596505, -0.3304412383293807, 0.6966831772351781, 1.3218225691053342, 1.1184948461435995, -3.367915164034308, -0.6475984594825316, -1.3622863977299242, -1.9845424602950263]\n", "iter:62\n", "w: [1.4258972276598232, 0.15075450353240116, 0.5823282147711089, -1.7213675718155756, 0.5960400284904042, 2.252292332149733, -0.23966286766831954, -0.8082489852079048, 0.5898110424656942, -0.7987812619949675, 0.8259663105336432, -0.3352428295266686, 0.7014774408325634, 1.3297756710281825, 1.1329054246665653, -3.3961895092925354, -0.6529189594658632, -1.3726679074810857, -2.001643694280673]\n", "iter:63\n", "w: [1.4371182692032016, 0.150098568804689, 0.587107589756231, -1.733525760351798, 0.6010925429344355, 2.2668082013853774, -0.2392073701809167, -0.814868951106707, 0.5937306212768135, -0.8039789921960404, 0.831588152476405, -0.34000196361645124, 0.7062269773500657, 1.337614441074297, 1.1471896906044103, -3.424208574390924, -0.6581951890482663, -1.3829344790630396, -2.018567506586789]\n", "iter:64\n", "w: [1.4482194704156641, 0.14943853036249458, 0.591850340958642, -1.7455681076769627, 0.6061045683949907, 2.281150706820507, -0.23874197319604976, -0.8214352498148175, 0.5976136636079988, -0.8091046174254813, 0.8371456754245366, -0.34471929418155445, 0.7109327202734093, 1.3453428730809853, 1.161349462417062, -3.451977506790156, -0.6634280262357015, -1.3930890444053639, -2.0353180372168347]\n", "iter:65\n", "w: [1.4592037295191649, 0.14877493483993992, 0.5965569937760512, -1.7574973431304983, 0.6110769004832393, 2.2953247890440585, -0.2382674513013388, -0.8279487758642015, 0.6014610135981118, -0.8141605813409631, 0.8426405166956777, -0.34939545674303574, 0.7155955699762049, 1.3529647641736873, 1.1753865152323002, -3.4795012756294295, -0.6686183177910302, -1.4031344147187828, -2.051899280675428]\n", "iter:66\n", "w: [1.4700738393882025, 0.1481082941528937, 0.6012280602338225, -1.769316091210941, 0.6160103059355293, 2.309335179034454, -0.237784533015748, -0.8344103959224848, 0.6052734833256204, -0.8191492109114938, 0.8480742470480246, -0.3540310698837582, 0.7202163954917106, 1.3604837268319612, 1.1893025834616688, -3.5067846811976326, -0.6737668809181947, -1.4130732874768348, -2.0683150927439318]\n", "iter:67\n", "w: [1.4808324926078806, 0.14743908776843906, 0.6058640397292991, -1.7810268770723283, 0.6209055242134487, 2.3231864096748325, -0.23729390373855447, -0.8408209503785138, 0.6090518544881425, -0.8240727233288344, 0.8534483744937194, -0.35862673626077535, 0.7247960361582974, 1.3679032000746276, 1.203099363134805, -3.5338323637282767, -0.6788745048272872, -1.4229082528812287, -2.0845691968634803]\n", "iter:68\n", "w: [1.491482286230882, 0.1467677648057893, 0.6104654197083458, -1.7926321316612122, 0.6257632689883657, 2.3368828264969475, -0.23679620848259408, -0.847181254798824, 0.6127968799713495, -0.8289332324338196, 0.858764347830789, -0.36318304351911984, 0.7293353031494596, 1.3752264598384307, 1.2167785139864096, -3.5606488115763746, -0.6839419521907244, -1.4326417998565024, -2.1006651901525792]\n", "iter:69\n", "w: [1.5020257262546708, 0.14609474598317004, 0.6150326762813042, -1.8041341965223543, 0.6305842295204873, 2.350428597714446, -0.2362920544100032, -0.8534921012673611, 0.6165092853159811, -0.8337327546967984, 0.8640235599185886, -0.3677005651180834, 0.7338349808981526, 1.3824566286174864, 1.2303416613257876, -3.5872383688298006, -0.688969960499717, -1.4422763216150019, -2.1166065490842296]\n", "iter:70\n", "w: [1.5124652318384837, 0.1454204254237493, 0.619566274784756, -1.8155353282986113, 0.6353690719414534, 2.363827723600793, -0.2357820131869165, -0.8597542596195086, 0.620189770090912, -0.8384732147878481, 0.869227350719173, -0.3721798610797578, 0.7382958284243017, 1.3895966844244456, 1.2437903977151608, -3.6136052424026, -0.6939592433293102, -1.4518141208291073, -2.1323966348455468]\n", "iter:71\n", "w: [1.5228031392780919, 0.1447451723324196, 0.6240666702947715, -1.8268377029482041, 0.6401184404485828, 2.377084045262167, -0.23526662317201252, -0.8659684785803399, 0.6238390091794805, -0.8431564507691997, 0.8743770101247693, -0.3766214786684533, 0.7427185805734831, 1.3966494691286202, 1.2571262844796918, -3.6397535086531967, -0.6989104915194684, -1.461257414443998, -2.148038698401048]\n", "iter:72\n", "w: [1.533041705754871, 0.14406933255409884, 0.6285343080966932, -1.8380434197005184, 0.6448329584180933, 2.39020125285123, -0.23474639145239742, -0.8721354868159921, 0.6274576539856439, -0.8477842189394275, 0.8794737805895262, -0.3810259530086016, 0.7471039491740237, 1.4036176962212261, 1.270350853069335, -3.665687119566391, -0.7038243742789589, -1.4706083381610138, -2.1635358852791455]\n", "iter:73\n", "w: [1.5431831128744198, 0.14339323002320292, 0.632969624115947, -1.8491545047697309, 0.649513229443909, 2.4031828932636885, -0.23422179573906363, -0.8782559939061554, 0.631046333565941, -0.8523581983563444, 0.8845188595819379, -0.38539380764786074, 0.7514526241190913, 1.4105039580533278, 1.2834656062901408, -3.691409908534411, -0.7087015402181582, -1.4798689506187788, -2.178891240099888]\n", "iter:74\n", "w: [1.5532294700087699, 0.14271716811303836, 0.6373730453138768, -1.8601729148439006, 0.6541598383080401, 2.4160323773559496, -0.2336932861330394, -0.8843306912448633, 0.6346056556927214, -0.8568799950631929, 0.8895134018727575, -0.3897255550713635, 0.7557652743797414, 1.4173107325879564, 1.2964720194204868, -3.7169255957690157, -0.7135426183163298, -1.4890412372966801, -2.1941077108606395]\n", "iter:75\n", "w: [1.563182817455156, 0.14204143089305352, 0.6417449900521649, -1.8711005403656644, 0.6587733518879622, 2.4287529867189357, -0.2331612867723375, -0.8903602528760489, 0.6381362078536329, -0.8613511460405979, 0.8944585216717991, -0.39402169717236496, 0.7600425489543394, 1.4240403897041798, 1.309371541225813, -3.7422377933737283, -0.7183482188284249, -1.4981271141629855, -2.209188152995126]\n", "iter:76\n", "w: [1.5730451294233219, 0.14136628430116238, 0.6460858684290095, -1.8819392086193327, 0.6633543200059202, 2.4413478800401713, -0.2326261973689084, -0.896345336269688, 0.6416385581919325, -0.8657731229048233, 0.8993552946257692, -0.39828272568394196, 0.7642850777592924, 1.43069519708757, 1.3221655948837847, -3.7673500101026383, -0.723118934135996, -1.5071284310878221, -2.2241353332201617]\n", "iter:77\n", "w: [1.582818316862435, 0.14069197723770646, 0.6503960825898881, -1.892690686637955, 0.6679032762246379, 2.4538200990835937, -0.23208839464399222, -0.902286583043776, 0.6451132563918097, -0.8701473353711368, 0.9042047596881371, -0.4025091225758692, 0.7684934724655786, 1.4372773257385087, 1.3348555788303609, -3.7922656558298464, -0.7278553395464122, -1.5160469750393888, -2.238951933183301]\n", "iter:78\n", "w: [1.5925042301378645, 0.1400187425870353, 0.6546760270154312, -1.9033566839428313, 0.6724207385935108, 2.4661725743141227, -0.23154823366952865, -0.9081846196368683, 0.6485608345125621, -0.8744751344995096, 0.90900792087104, -0.40670136042032595, 0.7726683272851692, 1.443788855127061, 1.3474428675359686, -3.816988045751489, -0.7325579940441964, -1.5248844730801112, -2.253640552923723]\n", "iter:79\n", "w: [1.6021046615673122, 0.13934679817216336, 0.6589260887886565, -1.9139388551269338, 0.6769072103490039, 2.4784081301918306, -0.23100604912262895, -0.914040057935458, 0.651981807775153, -0.8787578157384499, 0.913765748888316, -0.41085990272967904, 0.7768102197110788, 1.4502317780206964, 1.359928812219883, -3.841520404340355, -0.7372274409979777, -1.5336425951779613, -2.2682037141577704]\n", "iter:80\n", "w: [1.6116213478250998, 0.1386763476474871, 0.6631466478435779, -1.9244388022928023, 0.6813631805726481, 2.4905294901585626, -0.23046215645951415, -0.919853495860046, 0.6553766753043987, -0.882996621781468, 0.9184791826979398, -0.41498520426922014, 0.780919711214456, 1.456608005008915, 1.372314741509923, -3.8658658690713725, -0.7418642088262551, -1.542322956846814, -2.282643863399753]\n", "iter:81\n", "w: [1.621055972222777, 0.13800758133412222, 0.6673380771969835, -1.9348580773546482, 0.6857891248097426, 2.502539281338045, -0.22991685301478973, -0.9256255179133946, 0.6587459208297729, -0.8871927452494962, 0.9231491309514042, -0.41907771134741717, 0.7849973479018354, 1.46291936874683, 1.3846019620537273, -3.8900274939346686, -0.7464688116249024, -1.5509271216285099, -2.2969633749278877]\n", "iter:82\n", "w: [1.6304101668736355, 0.1373406770020346, 0.671500743164992, -1.9451981842136488, 0.6901855056516047, 2.514440038968862, -0.22937041903143612, -0.9313566956941186, 0.6620900133475862, -0.8913473312115113, 0.92777647335693, -0.4231378620859531, 0.7890436611354049, 1.469167627937934, 1.3967917590871122, -3.9140082527514877, -0.7510417497590982, -1.559456603428199, -2.3111645536045695]\n", "iter:83\n", "w: [1.6396855147481715, 0.13667580060279272, 0.6756350055658242, -1.955460580814729, 0.694552773283972, 2.5262342105881874, -0.22882311862645613, -0.9370475883784746, 0.6654094077470831, -0.8954614795546308, 0.9323620619627957, -0.4271660866715755, 0.793059168118909, 1.4753544710746227, 1.4088853969643615, -3.9378110423069677, -0.7555835104221513, -1.5679128687135526, -2.3252496375595166]\n", "iter:84\n", "w: [1.6488835516270506, 0.13601310695645316, 0.6797412179100766, -1.9656466810924997, 0.6988913660039465, 2.5379241599827727, -0.22827520069671742, -0.9426987431729384, 0.6687045454028054, -0.8995362472140659, 0.9369067223665352, -0.4311628075915572, 0.7970443724515833, 1.4814815199535496, 1.4208841196547088, -3.9614386853126096, -0.7600945681634891, -1.5762973385875465, -2.3392208007437767]\n", "iter:85\n", "w: [1.658005767957672, 0.1353527403958069, 0.6838197275796475, -1.9757578568134533, 0.7032017107076768, 2.5495121709224424, -0.22772689976916635, -0.9483106957399208, 0.6719758547353956, -0.9035726502725023, 0.9414112548552738, -0.43512843985437066, 0.8009997646523278, 1.4875503329815116, 1.432789151208766, -3.9848939332102185, -0.7645753853878962, -1.5846113907437038, -2.3530801553620253]\n", "iter:86\n", "w: [1.6670536106200102, 0.13469483537095364, 0.6878708759963521, -1.9857954393209873, 0.7074842233507956, 2.5610004506902064, -0.22717843679925692, -0.9538839705987542, 0.6752237517428481, -0.9075716659377386, 0.945876435482029, -0.4390633911970065, 0.8049258226561405, 1.493562408286328, 1.4446016961982018, -4.008179468828133, -0.7690264128279284, -1.5928563613119633, -2.3668297541900993]\n", "iter:87\n", "w: [1.6760284846070248, 0.1340395170169388, 0.6918949987811487, -1.995760721189352, 0.7117393093834715, 2.5723911334220344, -0.22663001992113965, -0.9594190815038913, 0.6784486405040742, -0.9115342344067353, 0.9503030170824075, -0.44296806228020735, 0.8088230122846748, 1.4995191866460356, 1.456322940131576, -4.031297908899693, -0.7734480899912758, -1.6010335466026684, -2.3804715927842506]\n", "iter:88\n", "w: [1.6849317546245755, 0.13338690168697068, 0.6958924259048084, -2.0056549577921765, 0.7159673641617855, 2.583686283268393, -0.22608184515287708, -0.9649165318020756, 0.681650913656503, -0.9154612606236093, 0.9546917302357686, -0.4468428468727553, 0.812691787692635, 1.5054220542486918, 1.4679540498488908, -4.054251806453096, -0.7778408455847124, -1.6091442047555782, -2.394007611588176]\n", "iter:89\n", "w: [1.6937647466154455, 0.13273709745354184, 0.6998634818307734, -2.0154793687908232, 0.7201687733370109, 2.594887897388747, -0.2255340970597071, -0.9703768147700894, 0.6848309528493245, -0.9193536159385417, 0.9590432841745975, -0.45068813202582597, 0.8165325917915895, 1.5112723452941292, 1.4794961738971086, -4.077043653081084, -0.7822050979161465, -1.617189557300247, -2.407439697943488]\n", "iter:90\n", "w: [1.7025287492117747, 0.13209020457960172, 0.7038084856508744, -2.0252351395474473, 0.7243439132242543, 2.6059979087894303, -0.22498694937814087, -0.9758004139345412, 0.6879891291738596, -0.923212139674047, 0.9633583676455356, -0.454504298238316, 0.82034585665266, 1.5170713444481432, 1.4909504428886218, -4.099675881098218, -0.7865412552761702, -1.6251707906336268, -2.42076968800893]\n", "iter:91\n", "w: [1.7112250151199173, 0.1314463159617635, 0.7077277512145077, -2.0349234224673016, 0.7284931511518027, 2.6170181890145394, -0.2244405656034775, -0.9811878033750231, 0.691125803572442, -0.9270376406045782, 0.967637649725242, -0.45829171961395193, 0.8241320038894295, 1.522820289158805, 1.5023179698444171, -4.122150865592939, -0.7908497163004068, -1.6330890574202876, -2.433999368593298]\n", "iter:92\n", "w: [1.7198547624414877, 0.13080551754738068, 0.7116215872518145, -2.044545338274508, 0.7326168457924217, 2.6279505506988188, -0.22389509954312317, -0.9865394480118553, 0.694241327227097, -0.9308308983550021, 0.9718817805940168, -0.4620507640109056, 0.8278914450223166, 1.5285203718438667, 1.513599850523471, -4.144470926381023, -0.7951308703138513, -1.640945477920246, -2.447130478906716]\n", "iter:93\n", "w: [1.7284191759341012, 0.13016788872719282, 0.7154902974913491, -2.0541019772252245, 0.7367153474777568, 2.6387967499908878, -0.22335069583792802, -0.9918558038795264, 0.6973360419292148, -0.9345926647230833, 0.9760913922698902, -0.46578179318456286, 0.8316245818255615, 1.5341727419575644, 1.5247971637397262, -4.166638329866568, -0.7993850976583218, -1.6487411412490007, -2.460164712234627]\n", "iter:94\n", "w: [1.7369194082151063, 0.1295335027051154, 0.7193341807726755, -2.0635944002618825, 0.7407889984969038, 2.6495584888545642, -0.22280749045359283, -0.9971373183868448, 0.7004102804313344, -0.9383236649307405, 0.9802670993056793, -0.46948516292402487, 0.8353318066578921, 1.5397785079445154, 1.5359109716678394, -4.188655290816187, -0.8036127700040516, -1.6564771065740445, -2.473103717538587]\n", "iter:95\n", "w: [1.7453565809113873, 0.12890242684663494, 0.7231535311542866, -2.073023640111901, 0.7448381333801387, 2.6602374172555194, -0.22226561114405038, -1.0023844305647285, 0.703464366782076, -0.9420245988085058, 0.9844094994513231, -0.4731612231828629, 0.8390135027778535, 1.5453387390878488, 1.546942320138744, -4.210523974051646, -0.8078142506463825, -1.6641544042517915, -2.485949100987696]\n", "iter:96\n", "w: [1.7537317857581278, 0.12827472300716491, 0.7269486380172036, -2.082390702334074, 0.7488630791687224, 2.670835135239997, -0.2217251778885885, -1.0075975713024834, 0.7064986166451948, -0.9456961419173014, 0.9885191742836384, -0.4768103182045898, 0.8426700446447178, 1.5508544672581945, 1.5578922389259482, -4.2322464960658275, -0.8119898947884494, -1.6717740369085816, -2.498702427424275]\n", "iter:97\n", "w: [1.762046085649243, 0.12765044784162247, 0.7307197861645772, -2.091696566315615, 0.7528641556716336, 2.681353194911879, -0.22118630330435934, -1.012777163573349, 0.709513337603658, -0.9493389466113634, 0.992596689805483, -0.4804327866432672, 0.8463017982058226, 1.5563266885696851, 1.5687617420233768, -4.253824926566538, -0.8161400498106874, -1.6793369804691514, -2.5113652217671625]\n", "iter:98\n", "w: [1.7703005156420246, 0.12702965309639755, 0.7344672559175824, -2.10094218622263, 0.7568416757100199, 2.69179310231396, -0.2206490930358032, -1.017923622650024, 0.712508829449594, -0.9529536430458747, 0.9966425970161648, -0.48402896167962356, 0.8499091211711258, 1.5617563649486925, 1.5795518279154688, -4.27526128995234, -0.8202650555279302, -1.686844185135719, -2.523938970355827]\n", "iter:99\n", "w: [1.7784960839183837, 0.12641238588480344, 0.7381913232078672, -2.110128491906634, 0.7607959453501036, 2.702156319218899, -0.22011364612240858, -1.02303735631083, 0.7154853844609019, -0.956540840132627, 1.000657432454809, -0.4875991711330179, 0.853492363275712, 1.5671444256206155, 1.5902634798401545, -4.296557566724327, -0.8243652444348203, -1.6942965763206126, -2.536425122238277]\n", "iter:100\n", "w: [1.786633772704935, 0.12579868894702345, 0.7418922596667961, -2.1192563897695424, 0.7647272641252251, 2.712444264834966, -0.21958005534613342, -1.0281187650371133, 0.718443287665262, -0.9601011264468057, 1.0046417187182706, -0.49114373756955376, 0.8570518665309313, 1.5724917685196753, 1.6008976660452618, -4.317715694837451, -0.828440941940202, -1.7016950555351582, -2.5488250904055914]\n", "iter:101\n", "w: [1.7947145391540316, 0.12518860089549927, 0.7455703327117021, -2.128326763589415, 0.7686359252476627, 2.7226583174313634, -0.21904840755972207, -1.033168242202439, 0.721382817092239, -0.9636350710877798, 1.0085959649550713, -0.4946629784066153, 0.8605879654648076, 1.5777992616263266, 1.611455340038837, -4.3387375709947955, -0.832492466591119, -1.709040501237362, -2.5611402529757172]\n", "iter:102\n", "w: [1.802739316187727, 0.12458215644664146, 0.7492258056293434, -2.1373404753090925, 0.7725222158108211, 2.7327998158875895, -0.2185187839970679, -1.0381861742540894, 0.7243042440141259, -0.9671432244965917, 1.0125206673367364, -0.49815720601406965, 0.8641009873523064, 1.5830677442365861, 1.6219374408338003, -4.359625051887922, -0.8365201302870023, -1.7163337696407435, -2.5733719543290423]\n", "iter:103\n", "w: [1.8107090133065384, 0.12397938663968455, 0.7528589376567398, -2.146298365789724, 0.7763864169823435, 2.7428700611710344, -0.21799126056669701, -1.0431729408873327, 0.7272078331761352, -0.9706261192326578, 1.0164163095078107, -0.5016267278123585, 0.8675912524360173, 1.5882980281672843, 1.632344893187315, -4.380379955386241, -0.8405242384845928, -1.7235756954865202, -2.5855215061980865]\n", "iter:104\n", "w: [1.8186245173647586, 0.12338031904345412, 0.7564699840595539, -2.1552012555310673, 0.7802288041886603, 2.7528703177467286, -0.21746590812937416, -1.0481289152128996, 0.7300938430165078, -0.9740842707120285, 1.020283363015753, -0.5050718463676765, 0.8710590741377654, 1.5934908989009826, 1.6426786078351936, -4.401004061678137, -0.8445050903941062, -1.7307670927811907, -2.5975901887135415]\n", "iter:105\n", "w: [1.8264866933139772, 0.12278497795176203, 0.7600591962081598, -2.164049945360319, 0.7840496472914606, 2.7628018149229145, -0.21694279276076778, -1.0530544639180641, 0.7329625258770727, -0.9775181779094031, 1.0241222877218201, -0.5084928594844182, 0.8745047592616341, 1.5986471166740481, 1.6529394817216319, -4.421499114366403, -0.848462979167118, -1.7379087555014336, -2.6095792514087512]\n", "iter:106\n", "w: [1.8342963849163647, 0.12219338456810008, 0.7636268216515334, -2.172845217091129, 0.7878492107565331, 2.7726657481358936, -0.2164219760000502, -1.0579499474217005, 0.7358141282047583, -0.9809283240259514, 1.027933532193984, -0.5118900602950556, 0.8779286081888504, 1.6037674175111483, 1.6631283982245253, -4.441866821520387, -0.8523981920766139, -1.7450014582681102, -2.621489914184609]\n", "iter:107\n", "w: [1.8420544154291958, 0.12160555718025866, 0.767173104189088, -2.1815878341543593, 0.7916277538153983, 2.7824632801773728, -0.21590351508525024, -1.062815720023656, 0.7386488907445286, -0.9843151771248649, 1.0317175340828546, -0.5152637373475938, 0.881330915064954, 1.6088525142092176, 1.67324622737659, -4.462108856687071, -0.8563110106896212, -1.7520459569910425, -2.6333233682367445]\n", "iter:108\n", "w: [1.849761588261991, 0.1210215113254552, 0.770698283940563, -2.1902785422020346, 0.7953855306201256, 2.7921955423673523, -0.21538746317612556, -1.0676521300487585, 0.7414670487241827, -0.987679190736435, 1.0354747204815185, -0.5186141746907393, 0.8847119679796442, 1.613903097273748, 1.6832938260824861, -4.482226859863191, -0.8602017110328135, -1.759042989486133, -2.6450807769467635]\n", "iter:109\n", "w: [1.8574186876075938, 0.1204412599465225, 0.7742025974140685, -2.1989180696858703, 0.799122790391703, 2.8018636356754016, -0.2148738695652706, -1.0724595199857485, 0.744268832031437, -0.9910208044343437, 1.0392055082701463, -0.5219416519569015, 0.8880720491396779, 1.6189198358100754, 1.6932720383321167, -4.502222438430354, -0.864070563751455, -1.765993276066289, -2.656763276739206]\n", "iter:110\n", "w: [1.8650264790484106, 0.1198648135396698, 0.7776862775723798, -2.207507128411663, 0.8028397775623041, 2.811468631793003, -0.2143627798781329, -1.077238226621411, 0.7470544653836793, -0.994340444384749, 1.0429103044461656, -0.5252464444431372, 0.8914114350351613, 1.6239033783721606, 1.7031816954102574, -4.522097168054988, -0.8679178342620273, -1.772897520107526, -2.6683719779058093]\n", "iter:111\n", "w: [1.8725857101389898, 0.11929218029429828, 0.7811495538975638, -2.2160464140707594, 0.8065367319117751, 2.821011574159481, -0.21385423626256592, -1.0819885811701577, 0.749824168490763, -0.9976385238696432, 1.0465895064407482, -0.528528823190141, 0.8947303965995661, 1.628854353771211, 1.7130236161026484, -4.541852593554864, -0.871743782898862, -1.7797564085915365, -2.6799079653985594]\n", "iter:112\n", "w: [1.8800971109660383, 0.11872336622532252, 0.7845926524540179, -2.224536606749747, 0.810213888698645, 2.8304934789438896, -0.21334827756850974, -1.0867109093992933, 0.7525781562111901, -1.0009154437858778, 1.0502435024223131, -0.5317890550593704, 0.8980291993637719, 1.6337733718463368, 1.7227986068986738, -4.561490229733797, -0.8755486650550816, -1.786570612625932, -2.691372299592955]\n", "iter:113\n", "w: [1.88756139468692, 0.11815837529842106, 0.788015795949992, -2.2329783714194527, 0.8138714787859417, 2.839915335985084, -0.21284493951835246, -1.091405531750187, 0.7553166387020053, -1.0041715931211572, 1.0538726715877051, -0.53502740280839, 0.9013081036044255, 1.6386610241993032, 1.7325074621907315, -4.58101156218606, -0.8793327313181333, -1.7933407879432934, -2.7027660170228156]\n", "iter:114\n", "w: [1.8949792580476266, 0.11759720954861379, 0.7914192037976604, -2.241372358404256, 0.8175097287620822, 2.849278109692081, -0.212344254868492, -1.0960727634555496, 0.7580398215627143, -1.0074073494082314, 1.0574773844416652, -0.5382441251645137, 0.9045673644868859, 1.6435178848953136, 1.7421509644703908, -4.600418048071938, -0.8830962276001839, -1.8000675753800912, -2.714090131087905]\n", "iter:115\n", "w: [1.902351381881151, 0.11703986919253917, 0.7948030921718023, -2.2497192038326834, 0.821128861057085, 2.858582739906684, -0.21184625356258588, -1.1007129146530081, 0.7607479059735126, -1.0106230791584345, 1.0610580030651764, -0.5414394768968114, 0.9078072322030112, 1.6483445111316362, 1.7517298845214209, -4.61971111686578, -0.8868393952636269, -1.806751601336482, -2.725345632735575]\n", "iter:116\n", "w: [1.9096784315871551, 0.11648635273478092, 0.7981676740671512, -2.258019530070182, 0.8247290940543412, 2.8678301427302406, -0.21135096287694757, -1.105326290495151, 0.7634410888280996, -1.0138191382756574, 1.0646148813732315, -0.5446137088865474, 0.9110279521040254, 1.6531414438757823, 1.761244981609768, -4.638892171077802, -0.8905624712419385, -1.8133934782179186, -2.7365334911175667]\n", "iter:117\n", "w: [1.9169610575937623, 0.11593665706857452, 0.8015131593544604, -2.266273946134933, 0.8283106421981637, 2.8770212113162907, -0.2108584075585195, -1.1099131912562135, 0.7661195628613376, -1.0169958724517711, 1.0681483653625372, -0.5477670681961089, 0.9142297648286936, 1.6579092084748375, 1.770697003670548, -4.657962586950859, -0.8942656881561073, -1.819993804859465, -2.747654654223054]\n", "iter:118\n", "w: [1.924199895802277, 0.11539077757120227, 0.8048397548353375, -2.2744830480975082, 0.8318737160973244, 2.8861568166307667, -0.21036860995582737, -1.1144739124355538, 0.7687835167719989, -1.0201536175444592, 1.0716587933496384, -0.55089979813648, 0.917412906427016, 1.6626483152374532, 1.7800866874921188, -4.6769237151333, -0.8979492744268456, -1.826553166933646, -2.7587100494889563]\n", "iter:119\n", "w: [1.9313955680155797, 0.11484870819436836, 0.808147664295889, -2.282647419465128, 0.8354185226247721, 2.895237808181311, -0.20988159014329696, -1.1190087448580677, 0.7714331353408305, -1.023292699938364, 1.075146496199922, -0.554012138333311, 0.9205776084796453, 1.6673592599899172, 1.7894147588972862, -4.695776881328975, -0.9016134543827822, -1.8330721373426244, -2.7697005843884925]\n", "iter:120\n", "w: [1.9385486823509155, 0.11431044154982781, 0.8114370885592195, -2.290767631551247, 0.8389452650137179, 2.9042650147171916, -0.20939736603929093, -1.1235179747716733, 0.7740685995441572, -1.0264134368903954, 1.0786117975479281, -0.5571043247916299, 0.9237240982132139, 1.6720425246076376, 1.798681932921697, -4.714523386925405, -0.9052584483648229, -1.839551276595442, -2.780627146998908]\n", "iter:121\n", "w: [1.945659833637753, 0.11377596899052617, 0.8147082255368226, -2.2988442438311467, 0.8424541429502588, 2.913239244901216, -0.20891595351820344, -1.1280018839419959, 0.7766900866632259, -1.0295161368600054, 1.082055014009377, -0.5601765899592414, 0.9268525986117537, 1.6766985775232954, 1.807888913989466, -4.733164509601062, -0.9088844728268561, -1.8459911331710306, -2.791490606549251]\n", "iter:122\n", "w: [1.9527296038013586, 0.11324528068749423, 0.8179612702789014, -2.306877804284174, 0.8459453526627048, 2.922161287954962, -0.2084373665169288, -1.1324607497443713, 0.779297770389489, -1.0326010998251811, 1.0854764553852945, -0.5632291627888523, 0.9299633285243771, 1.6813278742128528, 1.81703639608608, -4.751701503912655, -0.9124917404329679, -1.8523922438676486, -2.802291813949036]\n", "iter:123\n", "w: [1.9597585622326956, 0.11271836570272467, 0.8211964150236511, -2.3148688497232426, 0.849419087007763, 2.9310319142785777, -0.20796161713600453, -1.1368948452532779, 0.7818918209260086, -1.035668617584872, 1.0888764248585983, -0.5662622687989612, 0.9330565027693788, 1.6859308576605285, 1.8261250629286216, -4.770135601863281, -0.9160804601513266, -1.858755134139371, -2.8130316022985937]\n", "iter:124\n", "w: [1.966747266145233, 0.1121952120582469, 0.824413849245537, -2.3228179061121703, 0.8528755355537236, 2.9398518760463332, -0.20748871573570946, -1.1413044393293053, 0.784472405085159, -1.0387189740485219, 1.0922552191834878, -0.5692761301335502, 0.9361323322349124, 1.6905079588037972, 1.83515558813335, -4.788468013452236, -0.9196508373448845, -1.865080318420223, -2.823710787381857]\n", "iter:125\n", "w: [1.9736962609192132, 0.11167580680160274, 0.827613759702597, -2.330725488871399, 0.8563148846607851, 2.9486219077790423, -0.20701867102738283, -1.1456897967037567, 0.7870396863827909, -1.0417524455133405, 1.0956131288679611, -0.5722709656206081, 0.939191023976385, 1.6950595969594013, 1.8441286353806743, -4.8066999272072595, -0.9232030738590361, -1.871368300436511, -2.8343301681423143]\n", "iter:126\n", "w: [1.9806060804339063, 0.11116013606791464, 0.8307963304827983, -2.3385921031726173, 0.8597373175586484, 2.9573427268944164, -0.206551490160215, -1.1500511780609795, 0.7895938251290147, -1.0447693009299135, 1.098950438349766, -0.5752469908295209, 0.9422327813107063, 1.699586180231315, 1.8530448585775574, -4.824832510699931, -0.9267373681063684, -1.8776195735078858, -2.8448905271428053]\n", "iter:127\n", "w: [1.987477247388347, 0.11064818513872601, 0.833961743049473, -2.3464182442227743, 0.863143014421503, 2.9660150342363494, -0.20608717880374539, -1.1543888401185112, 0.7921349785157524, -1.0477698021567192, 1.1022674261660736, -0.5782044181273556, 0.9452578039075222, 1.7040881059015414, 1.8619049020173788, -4.842866911044898, -0.9302539151486275, -1.8838346208376322, -2.855392631009824]\n", "iter:128\n", "w: [1.994310273611035, 0.11013993849778508, 0.8371101762858589, -2.354204397537952, 0.866532152440519, 2.9746395145840925, -0.20562574122629104, -1.1587030357051258, 0.7946633007011986, -1.0507542042040847, 1.105564365117153, -0.5811434567340666, 0.9482662878775553, 1.7085657608045812, 1.8707094005372915, -4.860804255383603, -0.9337529067760209, -1.89001391579266, -2.8658372308629425]\n", "iter:129\n", "w: [2.0011056603590456, 0.109635379883932, 0.8402418065387707, -2.361951039207538, 0.8699049058939561, 2.9832168371422143, -0.20516718036951576, -1.1629940138368566, 0.7971789428913225, -1.053722755468088, 1.1088415224243007, -0.5840643127766525, 0.9512584258581694, 1.7130195216863608, 1.8794589796731036, -4.878645651353108, -0.9372345315839691, -1.8961579221736458, -2.8762250627299535]\n", "iter:130\n", "w: [2.007863898606987, 0.1091344923412416, 0.8433568076614216, -2.3696586361491176, 0.8732614462149926, 2.991747656012206, -0.20471149791933907, -1.1672620197910728, 0.7996820534185417, -1.0566756979548857, 1.1120991598822796, -0.5869671893422849, 0.9542344070962694, 1.717449755548364, 1.8881542558117126, -4.896392187540621, -0.940698975047414, -1.9022670944757492, -2.886556847948298]\n", "iter:131\n", "w: [2.014585469326211, 0.1086372582665641, 0.8464553510554201, -2.377327646354482, 0.8766019420573721, 3.0002326106465396, -0.2042586943733716, -1.1715072951786776, 0.8021727778176851, -1.0596132674959182, 1.1153375340064955, -0.5898522865304375, 0.9571944175286404, 1.7218568199776763, 1.8967958363411246, -4.914044933924273, -0.9441464195927846, -1.9083418781403092, -2.896833293553315]\n", "iter:132\n", "w: [2.021270843754669, 0.10814365945460036, 0.8495376057119591, -2.3849585191271334, 0.8799265593589611, 3.0086723262859523, -0.20380876910505508, -1.1757300780144964, 0.8046512588993608, -1.06253569395442, 1.1185568961751353, -0.5927198015040352, 0.9601386398598284, 1.7262410634636047, 1.9053843197980855, -4.931604942300678, -0.9475770446677168, -1.9143827097979, -2.9070550926538337]\n", "iter:133\n", "w: [2.0279204836577884, 0.10765367714063842, 0.8526037382522202, -2.3925516953116426, 0.8832354614033057, 3.017067414380689, -0.20336172042467415, -1.1799306027859184, 0.8071176368208377, -1.065443201423642, 1.12175749276648, -0.5955699285396461, 0.9630672536376558, 1.7306028257015098, 1.9139202960133523, -4.949073246699788, -0.9509910268086196, -1.920390017503113, -2.917222924795593]\n", "iter:134\n", "w: [2.034534841580722, 0.10716729204107224, 0.8556539129670104, -2.4001076075152055, 0.8865288088792691, 3.0254184729963916, -0.2029175456373978, -1.1841091005198512, 0.8095720491545442, -1.0683360084171711, 1.124939565291589, -0.5984028590767374, 0.9659804353264642, 1.7349424378844414, 1.9224043462546347, -4.966450863787514, -0.9543885397061737, -1.9263642209614067, -2.9273374563129635]\n", "iter:135\n", "w: [2.041114361092312, 0.10668448439181753, 0.858688291855649, -2.4076266803217186, 0.8898067599388326, 3.0337260872053067, -0.2024762410985011, -1.1882657988480445, 0.8120146309542821, -1.071214328051709, 1.1281033505225504, -0.6012187817660161, 0.9688783583781702, 1.7392602229831504, 1.930837043367233, -4.983738793256583, -0.9577697542688456, -1.9323057317483565, -2.9373993406694145]\n", "iter:136\n", "w: [2.047659477021093, 0.1062052339847322, 0.8617070346641226, -2.4151093304986837, 0.8930694702531309, 3.0419908294634235, -0.20203780226590837, -1.192400922070841, 0.8144455148192501, -1.0740783682226558, 1.1312490806164741, -0.6040178825168718, 0.9717611933012167, 1.743556496015009, 1.9392189519124008, -5.000938018206052, -0.9611348386844955, -1.9382149535216122, -2.9474092187871643]\n", "iter:137\n", "w: [2.0541706156836415, 0.10572952020214312, 0.8647102989225254, -2.422555967197237, 0.896317093066796, 3.0502132599741536, -0.20160222375019154, -1.1965146912194016, 0.816864830955965, -1.0769283317728284, 1.134376983235404, -0.606800344543943, 0.9746291077274986, 1.7478315643023485, 1.94755062830346, -5.018049505509922, -0.9644839584801539, -1.9440922822258622, -2.9573677193664087]\n", "iter:138\n", "w: [2.0606481951055624, 0.10525732204957695, 0.8676982399817986, -2.429966992145581, 0.8995497792506764, 3.05839392703911, -0.20116949936215092, -1.2006073241164583, 0.8192727072381659, -1.0797644166546208, 1.1374872816623103, -0.6095663484128219, 0.9774822664773366, 1.7520857277206983, 1.9558326209396966, -5.035074206175222, -0.9678172765800377, -1.949938106291086, -2.967275459194534]\n", "iter:139\n", "w: [2.067092625235407, 0.10478861818678561, 0.8706710110497877, -2.4373427998360873, 0.9027676773529951, 3.0665333673965303, -0.20073962215809885, -1.2046790354356391, 0.8216692692647827, -1.0825868160859018, 1.1405801949133205, -0.6123160720849178, 0.9803208316225731, 1.7563192789373807, 1.964065470338062, -5.052013055689954, -0.9711349533618737, -1.9557528068243633, -2.9771330434456753]\n", "iter:140\n", "w: [2.0735043081517803, 0.10432338695715282, 0.8736287632266289, -2.444683777706319, 0.9059709336490099, 3.074632106547855, -0.20031258448295858, -1.208730036759412, 0.8240546404160443, -1.0853957186999323, 1.1436559378463353, -0.6150496909614951, 0.9831449625478559, 1.7605325036408972, 1.9722497092627111, -5.068866974361274, -0.9744371467115934, -1.961536757795495, -2.986941065970991]\n", "iter:141\n", "w: [2.0798836382638966, 0.10386160641556476, 0.8765716455394811, -2.4519903063142223, 0.9091596921892307, 3.082690659072951, -0.1998883780112875, -1.2127605366356882, 0.8264289419078017, -1.0881913086895634, 1.1467147212661732, -0.6177673779269014, 0.9859548160101761, 1.7647256807615168, 1.9803858628524016, -5.085636867644234, -0.9777240120764615, -1.967290326216682, -2.996700109579984]\n", "iter:142\n", "w: [2.0862310025058357, 0.10340325435482112, 0.8794998049766181, -2.4592627595077117, 0.9123340948462522, 3.09070952893445, -0.19946699378632618, -1.2167707406331294, 0.8287922928441351, -1.0909737659459706, 1.1497567520263776, -0.6204693033910026, 0.988750546196723, 1.768899082683458, 1.9884744487457826, -5.102323626461425, -0.9809957025166954, -1.9730138723164916, -3.006410746313205]\n", "iter:143\n", "w: [2.0925467805247306, 0.10294830833066107, 0.8824133865208921, -2.4665015045888694, 0.9154942813602539, 3.098689209771639, -0.19904842225716946, -1.2207608513951955, 0.8311448102683109, -1.0937432661921633, 1.152782233127816, -0.6231556353308401, 0.9915323047811145, 1.7730529754490332, 1.996515977204601, -5.11892812751385, -0.9842523687556318, -1.9787077497083334, -3.01607353770665]\n", "iter:144\n", "w: [2.098831344863109, 0.1024967456854729, 0.8853125331825833, -2.473706902472971, 0.9186403893832162, 3.1066301851843305, -0.19863265331415206, -1.2247310686929709, 0.8334866092121533, -1.0964999811114962, 1.1557913638141932, -0.6258265393315259, 0.9943002409780608, 1.7771876189551115, 2.004510951234853, -5.135451233583305, -0.9874941592284945, -1.9843723055536573, -3.0256890350481522]\n", "iter:145\n", "w: [2.105085061135612, 0.10204854357075449, 0.8881973860316477, -2.480879307842535, 0.921772554521902, 3.114532929007117, -0.19821967632253504, -1.2286815894768062, 0.8358178027438915, -1.0992440784713993, 1.1587843396645985, -0.6284821786263886, 0.9970545015965154, 1.7813032671422322, 2.012459866705904, -5.151893793826589, -0.9907212201298127, -1.9900078807200743, -3.035257779626055]\n", "iter:146\n", "w: [2.1113082882002887, 0.1016036789683859, 0.8910680842293752, -2.4880190692965884, 0.9248909103796482, 3.12239790557439, -0.19780948015457564, -1.232612607926809, 0.8381385020145388, -1.101975722242534, 1.1617613526831938, -0.6311227141363865, 0.9997952310913665, 1.7854001681766891, 2.0203632124676134, -5.168256644061784, -0.99393369545954, -1.9956148099345923, -3.044780302970445]\n", "iter:147\n", "w: [2.1175013783246683, 0.10116212871077446, 0.8939247650594712, -2.4951265294953315, 0.9279955885970125, 3.1302255699764947, -0.19740205322005855, -1.2365243155022179, 0.8404488163028607, -1.1046950727135683, 1.164722591386155, -0.6337483045088009, 1.0025225716137167, 1.7894785646258868, 2.0282214704654806, -5.184540607046911, -0.9971317270679183, -2.0011934219321508, -3.054257127087206]\n", "iter:148\n", "w: [2.1236646773467984, 0.10072386949992748, 0.8967675639585706, -2.50220202530037, 0.9310867188913136, 3.1380163683073685, -0.19699738349536317, -1.240416900989691, 0.8427488530589845, -1.1074022866017608, 1.1676682408859662, -0.6363591061552225, 1.0052366630598026, 1.7935386936272584, 2.0360351158538483, -5.200746492751181, -1.0003154546991326, -2.0067440395996305, -3.06368876468515]\n", "iter:149\n", "w: [2.129798524831428, 0.10028887792550718, 0.8995966145461991, -2.5092458879106894, 0.9341644290951083, 3.145770737903998, -0.19659545855113664, -1.2442905505505384, 0.8450387179466993, -1.1100975171595275, 1.1705984829731642, -0.638955273288846, 1.0079376431185953, 1.7975807870510179, 2.0438046171071855, -5.216875098619114, -1.0034850160337985, -2.012266980115505, -3.0730757193964684]\n", "iter:150\n", "w: [2.135903254221513, 0.09985713048191737, 0.9024120486541897, -2.516258442994528, 0.9372288451936405, 3.153489107578015, -0.1961962655786398, -1.2481454477669298, 0.8473185148844975, -1.112780914277163, 1.1735134961956277, -0.6415369579610845, 1.0106256473181305, 1.8016050716570093, 2.0515304361294793, -5.232927209827743, -1.0066405467303219, -2.0177625550852953, -3.0824184859907358]\n", "iter:151\n", "w: [2.1419791929852066, 0.09942860358447025, 0.9052139963555667, -2.5232400108172914, 0.9402800913612981, 3.1611718978397323, -0.19579979141482962, -1.2519817736871057, 0.849588346085402, -1.1154526245818752, 1.1764134559355004, -0.6441043100975167, 1.0133008090706075, 1.8056117692459002, 2.0592130283617625, -5.248903599537131, -1.0097821804651717, -2.02323107067298, -3.091717550582688]\n", "iter:152\n", "w: [2.1480266627584976, 0.09900327358467903, 0.9080025859929077, -2.5301909063656707, 0.9433182899971154, 3.1688195211149175, -0.1954060225662376, -1.2557997068696165, 0.8518483120956232, -1.1181127915332882, 1.1792985344838323, -0.6466574775331808, 1.0159632597162975, 1.8096010968049532, 2.066852842887805, -5.264805029134429, -1.0129100489721, -2.0286728277285087, -3.1009733908339934]\n", "iter:153\n", "w: [2.154045979483647, 0.09858111678471951, 0.9107779442061938, -2.5371114394680854, 0.9463435617593497, 3.1764323819545766, -0.1950149452317025, -1.2595994234266195, 0.8540985118320896, -1.120761555515558, 1.1821689011130232, -0.6491966060472246, 1.0186131285663034, 1.8135732666486073, 2.074450322537993, -5.280632248471658, -1.0160242820803471, -2.0340881219115574, -3.110186476149222]\n", "iter:154\n", "w: [2.160037453543571, 0.09816210945110121, 0.9135401959601569, -2.5440019149115987, 0.9493560255991669, 3.1840108772380162, -0.19462654532401033, -1.2633810970662567, 0.8563390426188914, -1.1233990539262402, 1.1850247221471453, -0.6517218393969267, 1.0212505429442027, 1.817528486554078, 2.0820059039914267, -5.296385996097446, -1.0191250077518645, -2.0394772438116635, -3.119357267866208]\n", "iter:155\n", "w: [2.1660013898923105, 0.09774622782758789, 0.916289464571135, -2.550862632555423, 0.9523557987934649, 3.1915553963694396, -0.19424080849049435, -1.2671448991341376, 0.8585700002226749, -1.1260254212620457, 1.1878661610302192, -0.6542333193510991, 1.0238756282266155, 1.8214669598921824, 2.089520017876261, -5.312066999482883, -1.022212352117588, -2.044840479064865, -3.1284862194409984]\n", "iter:156\n", "w: [2.171938088181723, 0.09733344814740343, 0.9190258717334454, -2.557693887441141, 0.9553429969768638, 3.1990663214683184, -0.19385772013264388, -1.2708909986539525, 0.860791478887028, -1.1286407892016048, 1.1906933783925182, -0.6567311857228819, 1.0264885078827268, 1.825388885753586, 2.0969930888683166, -5.327675975241691, -1.0252864395127939, -2.050178108466969, -3.137573776627572]\n", "iter:157\n", "w: [2.1778478428845265, 0.09692374664475899, 0.9217495375452835, -2.564495969899759, 0.9583177341728902, 3.2065440275537727, -0.19347726542476773, -1.2746195623672367, 0.8630035713658878, -1.1312452866853664, 1.1935065321149652, -0.6592155764019442, 1.0290893035128006, 1.8292944590706552, 2.1044255357879873, -5.343213629344893, -1.028347392511565, -2.0554904080835703, -3.146620377652498]\n", "iter:158\n", "w: [2.183730943413819, 0.09651709956573444, 0.9244605805341582, -2.5712691656557056, 0.961280122824384, 3.2139888827231844, -0.19309942933175805, -1.2783307547723088, 0.8652063689560087, -1.1338390399927447, 1.1963057773916932, -0.6616866273860992, 1.0316781348857136, 1.833183870735094, 2.111817771695472, -5.358680657330132, -1.031395331960399, -2.0607776493569268, -3.1556264533847105]\n", "iter:159\n", "w: [2.1895876742391938, 0.09611348317854583, 0.927159117681871, -2.578013755927877, 0.9642302738231497, 3.221401248325254, -0.1927241966259952, -1.282024738162404, 0.8673999615285206, -1.1364221728166233, 1.1990912667908293, -0.6641444728123462, 1.0342551199755416, 1.8370573077115346, 2.119170203984354, -5.374077744505823, -1.0344303770109842, -2.0660400992098085, -3.1645924275005557]\n", "iter:160\n", "w: [2.195418314999569, 0.09571287378322885, 0.9298452644490491, -2.584730017527841, 0.9671682965388798, 3.2287814791277074, -0.19235155190343498, -1.2857016726630224, 0.8695844375596101, -1.1389948063353221, 1.2018631503135646, -0.6665892449873503, 1.0368203749972287, 1.8409149531472428, 2.126483234473559, -5.38940556615028, -1.037452645152168, -2.071278020146416, -3.1735187166442698]\n", "iter:161\n", "w: [2.2012231406128344, 0.09531524772076762, 0.9325191347992404, -2.591418222955288, 0.9700942988473716, 3.236129923479849, -0.19198147959891548, -1.2893617162685131, 0.8717598841603543, -1.1415570592821287, 1.2046215754515677, -0.6690210744173684, 1.039374014441364, 1.8447569864780922, 2.1337572594977123, -5.404664787705989, -1.040462252241147, -2.0764916703504754, -3.182405730584038]\n", "iter:162\n", "w: [2.207002421382426, 0.09492058138169503, 0.9351808412225783, -2.5980786404908325, 0.9730083871580617, 3.243446923470142, -0.19161396400072056, -1.293005024877912, 0.8739263871057357, -1.1441090480124871, 1.2073666872427968, -0.6714400898376348, 1.0419161511080968, 1.8485835835309559, 2.1409926699959216, -5.419856064969143, -1.043459312533901, -2.081681303780599, -3.1912538723637773]\n", "iter:163\n", "w: [2.212756423100935, 0.09452885121419066, 0.9378304947590252, -2.6047115342862477, 0.9759106664408965, 3.250732815079007, -0.19124898926443357, -1.2966317523300515, 0.8760840308628657, -1.1466508865689382, 1.2100986283257678, -0.6738464182412142, 1.0444468961402154, 1.8523949166226563, 2.1481898515990157, -5.434980044274619, -1.0464439387148923, -2.0868471702630105, -3.2000635384507836]\n", "iter:164\n", "w: [2.218485407150838, 0.09414003373170216, 0.940468205021203, -2.6113171644522275, 0.9788012402525631, 3.2579879283269975, -0.19088653942611533, -1.3002420504379633, 0.8782328986184447, -1.1491826867438975, 1.2128175389923248, -0.676240184907332, 1.0469663590554135, 1.856191154655606, 2.1553491847152593, -5.450037362676495, -1.049416241926058, -2.091989515581717, -3.2088351188793753]\n", "iter:165\n", "w: [2.2241896306024564, 0.0937541055201115, 0.9430940802168165, -2.6178957871437523, 0.9816802107620979, 3.265212587418529, -0.19052659841483668, -1.3038360690225863, 0.880373072305482, -1.151704558140355, 1.215523557238967, -0.678621513429194, 1.0494746477777703, 1.8599724632102694, 2.16247104461457, -5.465028648124264, -1.052376331795112, -2.0971085815662174, -3.2175689973906634]\n", "iter:166\n", "w: [2.2298693463092216, 0.09337104324446972, 0.9457082271706808, -2.6244476546431446, 0.9845476787758927, 3.272407110881311, -0.19016915006459556, -1.3074139559458016, 0.8825046326293019, -1.1542166082305767, 1.2182168188167772, -0.6809905257413016, 1.0519718686684667, 1.8637390046345688, 2.169555801511263, -5.47995451963486, -1.0553243164631796, -2.102204606176823, -3.22626555156857]\n", "iter:167\n", "w: [2.235524803000346, 0.09299082365532071, 0.9483107513463569, -2.6309730154408864, 0.9874037437621168, 3.2795718117016404, -0.18981417812564752, -1.3109758571428067, 0.884627659092858, -1.156718942412883, 1.220897457279998, -0.6833473421462758, 1.05445812655576, 1.8674909381303506, 2.1766038206453504, -5.494815587460617, -1.0582603026117843, -2.1072778235876766, -3.2349251529722167]\n", "iter:168\n", "w: [2.2411562453709757, 0.09261342359463465, 0.9509017568674045, -2.637472114314277, 0.9902485038745724, 3.2867069974556955, -0.189461666275276, -1.3145219166538482, 0.8867422300213796, -1.1592116640665804, 1.2235656040333007, -0.6856920813411954, 1.0569335247642413, 1.8712284198370257, 2.1836154623624116, -5.509612453253285, -1.061184395489208, -2.1123284642675375, -3.243548167264797]\n", "iter:169\n", "w: [2.24676391416991, 0.0922388200013696, 0.9534813465382596, -2.6439451924039976, 0.9930820559760001, 3.2938129704369756, -0.18911159812802839, -1.318052276655326, 0.888848422586371, -1.1616948746051123, 1.2262213883777893, -0.6880248604434613, 1.0593981651433926, 1.8749516029124902, 2.190591082192069, -5.524345710224209, -1.0640966989362386, -2.1173567550584096, -3.252134954339042]\n", "iter:170\n", "w: [2.2523480462849625, 0.09186698991667971, 0.9560496218647427, -2.6503924872886553, 0.9959044956608512, 3.3008900277800173, -0.18876395724544268, -1.321567077490284, 0.8909463128289846, -1.1641686735274996, 1.2288649375557774, -0.6903457950161949, 1.0618521480954677, 1.8786606376114303, 2.1975310309250857, -5.5390159433007815, -1.0669973154113284, -2.122362919252081, -3.2606858684393853]\n", "iter:171\n", "w: [2.257908874826042, 0.09149791048878793, 0.9586066830742065, -2.6568142330573665, 0.9987159172775426, 3.307938461580516, -0.18841872714528707, -1.3250664576983027, 0.8930359756827888, -1.1666331584681326, 1.2314963767943818, -0.6926549990931783, 1.0642955726027137, 1.8823556713611096, 2.2044356546891093, -5.553623729279286, -1.0698863460151757, -2.12734717666464, -3.269201258280938]\n", "iter:172\n", "w: [2.263446629206023, 0.0911315589775387, 0.9611526291353268, -2.6632106603804497, 1.0015164139502089, 3.3149585590119783, -0.18807589131033595, -1.3285505540448084, 0.8951174849959483, -1.1690884252449771, 1.2341158293479668, -0.6949525852033472, 1.0667285362539547, 1.886036848834732, 2.2113052950230885, -5.5681696369742095, -1.0727638905147503, -2.1323097437090373, -3.2776814671653596]\n", "iter:173\n", "w: [2.268961535219477, 0.09076791275864675, 0.9636875577775479, -2.669581996578282, 1.004306077599965, 3.32195060243902, -0.18773543319670302, -1.3320195015498095, 0.8971909135528371, -1.1715345679062534, 1.2367234165394774, -0.6972386643948425, 1.069151135270553, 1.8897043120224735, 2.218140288950383, -5.5826542273641495, -1.0756300473667748, -2.1372508334657483, -3.2861268330937294]\n", "iter:174\n", "w: [2.2744538151193314, 0.09040694932765772, 0.9662115655101835, -2.675928465688382, 1.0070849989656976, 3.328914869527431, -0.1873973362417512, -1.3354734335160767, 0.899256333095101, -1.1739716787756433, 1.239319257800695, -0.6995133462586302, 1.0715634645317673, 1.8933582003002638, 2.2249409690505892, -5.59707805373439, -1.078484913740681, -2.1421706557516034, -3.294537688876511]\n", "iter:175\n", "w: [2.279923687691518, 0.09004864630363332, 0.968724747641185, -2.6822502885307715, 1.0098532676243928, 3.3358516333511052, -0.1870615838715992, -1.3389124815567768, 0.9013138143421869, -1.1763998484960811, 1.2419034707114516, -0.701776738951698, 1.0739656175995258, 1.8969986504964083, 2.2317076635301034, -5.611441661816247, -1.081328585541053, -2.147069417186839, -3.302914362240693]\n", "iter:176\n", "w: [2.2853713683276795, 0.08969298143257594, 0.9712271982955794, -2.6885476827716754, 1.0126109720110181, 3.342761162495953, -0.18672815950824345, -1.3423367756225755, 0.9033634270113566, -1.1788191660721787, 1.244476171037831, -0.7040289492198357, 1.076357686742627, 1.900625796956121, 2.2384406962914465, -5.625745589923275, -1.0841611574295746, -2.151947321260425, -3.311257175934195]\n", "iter:177\n", "w: [2.290797069095987, 0.0893399325906046, 0.973719010433585, -2.6948208629856043, 1.0153581994379697, 3.349643721160889, -0.18639704657631181, -1.3457464440282194, 0.9054052398371986, -1.1812297189113343, 1.247037472769393, -0.7062700824200078, 1.0787397629603892, 1.9042397716040522, 2.2451403870013724, -5.6399903690844155, -1.0869827228464914, -2.156804568393724, -3.319566447827624]\n", "iter:178\n", "w: [2.2962009988101317, 0.08898947778689532, 0.9762002758684112, -2.701070040715877, 1.0180950361140946, 3.3564995692559956, -0.18606822850946678, -1.3491416134786092, 0.9074393205906555, -1.1836315928635746, 1.2495874881554465, -0.708500242542325, 1.0811119360057588, 1.9078407040048784, 2.251807051157778, -5.65417652317417, -1.0897933740316035, -2.1616413560025296, -3.327842491013454]\n", "iter:179\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "w: [2.301583363096544, 0.08864159516639707, 0.9786710852837464, -2.707295424533628, 1.0208215671633034, 3.3633289624979548, -0.18574168875647365, -1.3525224090943762, 0.9094657360975817, -1.1860248722601725, 1.2521263277404038, -0.710719532231625, 1.0834742944078968, 1.9114287214220274, 2.2584410001554382, -5.6683045690398846, -1.0925932020448004, -2.1664578785575417, -3.336085613902714]\n", "iter:180\n", "w: [2.3069443644598944, 0.08829626301233429, 0.9811315282509422, -2.7134972200953444, 1.0235378766427827, 3.3701321525028396, -0.1854174107869489, -1.3558889544369725, 0.911484552256844, -1.1884096399510877, 1.2546541003982408, -0.7129280528086654, 1.0858269254942545, 1.9150039488746042, 2.265042541350589, -5.6823750166262155, -1.095382296786152, -2.1712543276433145, -3.344296120319254]\n", "iter:181\n", "w: [2.3122842023469303, 0.0879534597485074, 0.983581693245898, -2.7196756301989815, 1.0262440475608179, 3.376909386876351, -0.18509537809680351, -1.3592413715332874, 0.91349583405798, -1.190785977341269, 1.257170913366093, -0.715125904290941, 1.088169915412155, 1.918566509192583, 2.2716119781243735, -5.696388369096859, -1.0981607470155652, -2.1760308920157345, -3.3524743095916594]\n", "iter:182\n", "w: [2.3176030732086947, 0.08761316394140065, 0.9860216676656527, -2.725830854838697, 1.0289401618942373, 3.3836609093015824, -0.1847755742133945, -1.3625797808997966, 0.9154996455984277, -1.193153964425861, 1.259676872277013, -0.7173131854131286, 1.090503349149891, 1.9221165230703245, 2.2781496099451783, -5.710345122953609, -1.100928640372021, -2.180787757658065, -3.360620476642888]\n", "iter:183\n", "w: [2.3229011705611793, 0.08727535430210683, 0.9884515378446878, -2.7319630912582458, 1.0316263006054887, 3.390386959624395, -0.18445798270039876, -1.3659043015662606, 0.9174960501003391, -1.1955136798243526, 1.2621720811919115, -0.7194899936471687, 1.0928273105573554, 1.9256541091184827, 2.2846557324298726, -5.724245768152828, -1.1036860633924006, -2.185525107835606, -3.3687349120776955]\n", "iter:184\n", "w: [2.3281786850444557, 0.0869400096880792, 0.9908713890709485, -2.7380725340030776, 1.0343025436593556, 3.3970877739364758, -0.18414258716242082, -1.3692150510989765, 0.9194851099269886, -1.1978652008137036, 1.2646566426307102, -0.7216564252219902, 1.095141882366214, 1.9291793839143516, 2.2911306374039757, -5.7380907882193775, -1.1064331015299127, -2.1902431231490027, -3.3768179022679066]\n", "iter:185\n", "w: [2.3334358044803336, 0.08660710910471733, 0.9932813056015876, -2.744159374971172, 1.036968970039326, 3.4037635846561596, -0.1838293712493476, -1.3725121456235962, 0.9214668865987905, -1.2002086033604882, 1.2671306576027277, -0.7238125751428849, 1.0974471462096347, 1.9326924620507118, 2.29757461296077, -5.751880660358098, -1.109169839172132, -2.1949419815862554, -3.3848697294356085]\n", "iter:186\n", "w: [2.338672713928585, 0.08627663170679739, 0.9956813706784368, -2.7502238034626547, 1.0396256577636203, 3.4104146206070807, -0.18351831866046026, -1.3757956998475196, 0.9234414408089353, -1.2025439621520857, 1.2695942256363209, -0.7259585372105379, 1.099743182641583, 1.9361934561832257, 2.3039879435193784, -5.765615855562888, -1.1118963596586606, -2.1996218585734533, -3.3928906717343126]\n", "iter:187\n", "w: [2.34388959574178, 0.08594855679975326, 0.9980716665432118, -2.7562660062282207, 1.0422726839008876, 3.4170411070947253, -0.18320941314831607, -1.3790658270818719, 0.9254088324386565, -1.2048713506269537, 1.2720474448078045, -0.7280944040397223, 1.1020300711556994, 1.939682477076435, 2.3103709098818235, -5.779296838723444, -1.114612745298419, -2.204282927024283, -3.4008810033281534]\n", "iter:188\n", "w: [2.3490866296187725, 0.08562286384081647, 1.0004522744524553, -2.7622861675164128, 1.0449101245855807, 3.423643265980953, -0.18290263852240848, -1.3823226392630743, 0.9273691205721386, -1.2071908410040157, 1.2744904117696696, -0.7302202670776613, 1.1043078902037626, 1.943159633648404, 2.3167237892890897, -5.792924068729733, -1.1173190773865795, -2.208925357388341, -3.408840994469175]\n", "iter:189\n", "w: [2.35426399265688, 0.08529953244002265, 1.0028232746922225, -2.768284469119776, 1.0475380550330153, 3.430221315756547, -0.18259797865261743, -1.3855662469740173, 0.9293223635110764, -1.2095025043111904, 1.2769232217781206, -0.7323362166220683, 1.1065767172137575, 1.9466250330140615, 2.3230468554762065, -5.806497998574241, -1.1200154362211499, -2.2135493176982823, -3.416770911572763]\n", "iter:190\n", "w: [2.359421859402786, 0.0849785423610905, 1.005184746592515, -2.774261090419928, 1.0501565495541239, 3.4367754716118593, -0.18229541747245817, -1.3887967594648432, 0.9312686187888959, -1.2118064104130957, 1.2793459687199518, -0.7344423418388664, 1.1088366286075515, 1.9500787805272792, 2.3293403787263665, -5.820019075452066, -1.122701901119215, -2.2181549736158486, -3.424671017291272]\n", "iter:191\n", "w: [2.364560401902213, 0.08465987352218053, 1.0075367685414671, -2.780216208431575, 1.0527656815699107, 3.4433059455056068, -0.1819949389821381, -1.392014284673349, 0.933207943184646, -1.2141026280379537, 1.281758745138781, -0.7365387307795971, 1.1110876998181936, 1.9535209798217346, 2.3356046259241023, -5.833487740858906, -1.1253785504328482, -2.222742488476797, -3.4325415705859017]\n", "iter:192\n", "w: [2.369679789748392, 0.08434350599653943, 1.0098794179992887, -2.7861499978455018, 1.0553655236256163, 3.449812946231878, -0.18169652725143043, -1.395218929245016, 0.9351403927365708, -1.2163912248037223, 1.2841616422606605, -0.7386254703985228, 1.1133300053068438, 1.9569517328505965, 2.3418398606075335, -5.84690443068699, -1.128045461564697, -2.227312023334767, -3.440382826796873]\n", "iter:193\n", "w: [2.3747801901293712, 0.08402942001303597, 1.0122127715119704, -2.7920626310705674, 1.0579561474045975, 3.4562966794854035, -0.18140016642237294, -1.3984107985526735, 0.9370660227553713, -1.2186722672434827, 1.2865547500190797, -0.740702646569428, 1.1155636185793434, 1.960371139925076, 2.3480463430197025, -5.86026957531902, -1.130702710983255, -2.231863737004115, -3.4481950377119444]\n", "iter:194\n", "w: [2.3798617678741882, 0.08371759595659456, 1.0145369047247548, -2.797954278274733, 1.060537623741932, 3.462757347925147, -0.18110584071179958, -1.4015899967158072, 0.9389848878371646, -1.2209458208301036, 1.288938157079381, -0.7427703441021263, 1.1177886122024345, 1.9637792997518764, 2.354224330159019, -5.873583599720146, -1.1333503742378241, -2.236397786101744, -3.4559784516333254]\n", "iter:195\n", "w: [2.384924685497946, 0.08340801436853121, 1.0168518923953775, -2.803825107425148, 1.0631100226377517, 3.4691951512362613, -0.18081353441371323, -1.404756626619516, 0.9408970418761504, -1.2232119500002099, 1.2913119508626019, -0.7448286467586781, 1.1200050578196379, 1.9671763094695824, 2.3603740758288274, -5.88684692352806, -1.1359885259731783, -2.24091432508796, -3.4637333134430204]\n", "iter:196\n", "w: [2.389969103245819, 0.08310065594679741, 1.019157808407084, -2.809675284327327, 1.0656734132703138, 3.475610286190469, -0.18052323190150532, -1.4079107899331265, 0.942802538076991, -1.2254707181774764, 1.293676217568761, -0.7468776372693248, 1.1222130261667989, 1.9705622646840237, 2.3664958306861092, -5.9000599611412134, -1.1386172399439343, -2.2454135063063774, -3.4714598646666577]\n", "iter:197\n", "w: [2.39499517913602, 0.08279550154613705, 1.0214547257814246, -2.815504972663432, 1.0682278640088159, 3.482002946704907, -0.18023491763003127, -1.411052587128474, 0.9447014289669148, -1.22772218779527, 1.2960310421996035, -0.748917397348144, 1.1244125870873078, 1.9739372595026448, 2.3725898422893446, -5.913223121805247, -1.1412365890286351, -2.2498954800229023, -3.4791583435358358]\n", "iter:198\n", "w: [2.4000030690017593, 0.08249253217816048, 1.0237427166908326, -2.8213143340296964, 1.0707734424259583, 3.488373323899483, -0.17994857613754686, -1.4141821174978542, 0.946593766407549, -1.229966420318661, 1.29837650858082, -0.7509480077084326, 1.1266038095470052, 1.9773013865679145, 2.378656355145541, -5.926336809697642, -1.143846645243558, -2.2543603944638235, -3.486828985049034]\n", "iter:199\n", "w: [2.404992926532219, 0.08219172901134053, 1.0260218524709899, -2.827103527973007, 1.0733102153102645, 3.4947216061527953, -0.17966419204751224, -1.4172994791716527, 0.9484796016064899, -1.2322034762658258, 1.300712699383755, -0.7529695480778215, 1.1287867616487772, 1.9806547370898093, 2.3846956107564488, -5.939401424010657, -1.146447479756248, -2.2588083958530274, -3.4944720210311235]\n", "iter:200\n", "w: [2.4099649033125767, 0.0818930733709348, 1.0282922036329833, -2.8328727120266666, 1.0758382486781621, 3.5010479791566467, -0.17938175007027046, -1.4204047691356618, 0.950358985128618, -1.2344334152288567, 1.3030396961466164, -0.754982097213127, 1.1309615106468514, 1.983997400877395, 2.3907078476639785, -5.952417359032584, -1.1490391628987873, -2.263239628448367, -3.5020876801915186]\n", "iter:201\n", "w: [2.4149191488631, 0.08159654673883807, 1.0305538398752558, -2.838622041745367, 1.07835760778583, 3.507352625969206, -0.17910123500460456, -1.4234980832480872, 0.9522319669071642, -1.2366562958940042, 1.3053575792952032, -0.756985732914946, 1.1331281229607983, 1.9873294663695427, 2.39669330149483, -5.9653850042273655, -1.151621764180805, -2.2676542345772077, -3.5096761881810026]\n", "iter:202\n", "w: [2.419855810677336, 0.08130213075336873, 1.0328068300953568, -2.8443516707393854, 1.080868357140819, 3.513635727066849, -0.17882263173918095, -1.426579516256254, 0.9540985962545333, -1.2388721760613637, 1.3076664281631616, -0.7589805320419959, 1.1352866641892478, 1.9906510206648005, 2.4026522050043555, -5.9783047443126165, -1.1541953523022361, -2.272052354671168, -3.5172377676472677]\n", "iter:203\n", "w: [2.424775034259428, 0.08100980720899299, 1.0350512424014946, -2.8500617507080284, 1.083370560513452, 3.5198974603947226, -0.17854592525388235, -1.429649161813017, 0.9559589218728939, -1.2410811126640295, 1.3099663210117831, -0.7609665705252088, 1.137437199123325, 1.9939621495504547, 2.4085847881196636, -5.991176959336081, -1.156759995165831, -2.276434127300082, -3.5247726382892015]\n", "iter:204\n", "w: [2.4296769631605746, 0.08071955805599045, 1.037287144123895, -2.8557524314723453, 1.0858642809480037, 3.526138001416068, -0.1782711006210368, -1.4327071124928807, 0.9578129918645367, -1.2432831617867295, 1.3122573350493574, -0.7629439233815802, 1.139579791759815, 1.9972629375308002, 2.4144912779819836, -6.004002024750569, -1.1593157598894257, -2.280799689205203, -3.5322810169099506]\n", "iter:205\n", "w: [2.4345617390146583, 0.0804313654000638, 1.0395146018259702, -2.8614238610071294, 1.0883495807736727, 3.5323575231603375, -0.17799814300654668, -1.4357534598078368, 0.9596608537420115, -1.2454783786839585, 1.3145395464500922, -0.7649126647277804, 1.1417145053140614, 2.000553467854649, 2.4203718989883, -6.016780311487407, -1.161862712817977, -2.2851491753316653, -3.539763117468801]\n", "iter:206\n", "w: [2.43942950157307, 0.080145211501897, 1.041733681315299, -2.8670761854722246, 1.0908265216153439, 3.5385561962701404, -0.17772703767092293, -1.438788294222922, 0.9615025544380454, -1.2476668177976273, 1.3168130303726107, -0.7668728677935303, 1.1438414022326056, 2.0038338225421, 2.426226872832273, -6.029512186028435, -1.1644009195353673, -2.289482718860229, -3.5472191511319004]\n", "iter:207\n", "w: [2.4442803887387403, 0.07986107877666358, 1.043944447654427, -2.872709549243156, 1.0932951644041502, 3.544734189047057, -0.1774577699702284, -1.4418117051715063, 0.9633381403152509, -1.2498485327742404, 1.3190778609780376, -0.7688246049347478, 1.145960544205573, 2.0071040824105952, 2.4320564185444584, -6.042198010476586, -1.1669304448759845, -2.293800451238324, -3.5546493263218557]\n", "iter:208\n", "w: [2.4491145365994123, 0.07957894979348903, 1.0461469651714834, -2.8783240949411035, 1.0957555693878367, 3.550891667496345, -0.1771903253569351, -1.4448237810703113, 0.9651676571756257, -1.2520235764816177, 1.321334111447687, -0.7707679476464694, 1.1480719921788118, 2.010364327100279, 2.4378607525318374, -6.054838142625085, -1.1694513529360844, -2.298102502210412, -3.5620538487662334]\n", "iter:209\n", "w: [2.4539320794601642, 0.07929880727486976, 1.0483412974706234, -2.8839199634622386, 1.098207796140932, 3.557028795370575, -0.17692468938069913, -1.4478246093341696, 0.9669911502698536, -1.2541920010251768, 1.3235818540003599, -0.7727029665755486, 1.1501758063657916, 2.0136146350986905, 2.4436400886166725, -6.067432936025296, -1.171963707084937, -2.3023889998476847, -3.569432921544996]\n", "iter:210\n", "w: [2.4587331498752105, 0.07902063409605052, 1.0505275074422966, -2.8894972940064307, 1.1006519035747309, 3.563145734212227, -0.17666084768905665, -1.4508142763905274, 0.9688086643064089, -1.2563538577637854, 1.325821159909262, -0.774629731533139, 1.1522720462592657, 2.016855083764803, 2.4493946380746983, -6.079982740053242, -1.174467569975764, -2.306660070577115, -3.576786745136896]\n", "iter:211\n", "w: [2.463517878678994, 0.07874441328436417, 1.052705657273345, -2.895056224105353, 1.1030879499470925, 3.5692426433952713, -0.17639878602804612, -1.4537928676936955, 0.9706202434604698, -1.2585091973252007, 1.328052099518554, -0.776548311506963, 1.1543607706427033, 2.0200857493524347, 2.455124609672663, -6.0924878999748495, -1.176963003556474, -2.310915839209881, -3.584115517464862]\n", "iter:212\n", "w: [2.4682863950165927, 0.07847012801853503, 1.0548758084569339, -2.9005968896499907, 1.1055159928720593, 3.5753196801657685, -0.17613849024275785, -1.4567604677388548, 0.9724259313826469, -1.260658069621107, 1.3302747422595411, -0.7784587746733727, 1.156442037601499, 2.0233067070330466, 2.4608302097052315, -6.10494875700992, -1.1794500690801937, -2.3151564289691726, -3.591419433940392]\n", "iter:213\n", "w: [2.473038826373457, 0.0781977616279485, 1.0570380218023194, -2.9061194249175775, 1.1079360893292993, 3.581376999681515, -0.17587994627781728, -1.4597171600758214, 0.9742257712075297, -1.2628005238617634, 1.3324891566665127, -0.7803611884092038, 1.158515904533961, 2.0265180309179525, 2.466511642031258, -6.117365648394879, -1.1819288271156074, -2.319381961517405, -3.5986986875069955]\n", "iter:214\n", "w: [2.477775298604494, 0.07792729759188924, 1.0591923574444544, -2.911623962597966, 1.1103482956733768, 3.5874147550507565, -0.1756231401778027, -1.462663027322574, 0.9760198055620571, -1.264936608570275, 1.3346954103922404, -0.7822556193034302, 1.1605824281620871, 2.02971979407995, 2.4721691081094486, -6.129738907444319, -1.1843993375571038, -2.323592556982848, -3.605953468682689]\n", "iter:215\n", "w: [2.4824959359625174, 0.07765871953874957, 1.0613388748534367, -2.9171106338194535, 1.1127526676428532, 3.593433097370002, -0.17536805808760147, -1.465598151178551, 0.9778080765737159, -1.2670663715964963, 1.3368935702231446, -0.7841421331686202, 1.1626416645421298, 2.0329120685744, 2.477802807033416, -6.142068863611373, -1.1868616596347366, -2.327788333985687, -3.61318396560159]\n", "iter:216\n", "w: [2.4872008611260803, 0.07739201124521072, 1.063477632843802, -2.9225795681740725, 1.1151492603692257, 3.5994321757609575, -0.1751146862527084, -1.4685226124377198, 0.9795906258785708, -1.2691898601305822, 1.339083702094136, -0.7860207950521982, 1.1646936690749592, 2.0360949254597633, 2.4834129355661436, -6.15435584254693, -1.189315851924002, -2.3319694096635346, -3.6203903640546145]\n", "iter:217\n", "w: [2.4918901952267025, 0.07712715663539801, 1.0656086895836645, -2.9280308937423607, 1.1175381283857033, 3.605412137406608, -0.1748630110194676, -1.4714364910014242, 0.9813674946291316, -1.2713071207161923, 1.341265871103142, -0.7878916692475176, 1.1667384965162242, 2.0392684348176155, 2.4889996881738647, -6.166600166157743, -1.1917619723554387, -2.336135899696401, -3.627572847529312]\n", "iter:218\n", "w: [2.4965640578755135, 0.07686413978001258, 1.067732102603708, -2.933464737117625, 1.1199193256358255, 3.611373127586468, -0.17461301883526206, -1.4743398658910127, 0.9831387235020607, -1.273418199263363, 1.343440141525325, -0.7897548193047459, 1.1687762009863207, 2.0424326657721545, 2.4945632570593754, -6.178802152663428, -1.1942000782240523, -2.3402879183311405, -3.634731597248855]\n", "iter:219\n", "w: [2.501222567189322, 0.07660294489544033, 1.0698479288060312, -2.9388812234297097, 1.1222929054819277, 3.6173152897110246, -0.1743646962486524, -1.4772328152602547, 0.984904352705724, -1.2755231410610546, 1.3456065768270018, -0.7916103080415675, 1.1708068359801684, 2.0455876865092146, 2.500103832194783, -6.190962116652394, -1.1966302261985695, -2.3444255784053842, -3.641866792210208]\n", "iter:220\n", "w: [2.5058658398161295, 0.07634355634284089, 1.071956224472846, -2.9442804763682817, 1.1246589207134552, 3.6232387653553975, -0.17411802990946795, -1.4801154164075445, 0.9866644219875916, -1.2776219907893842, 1.347765239679269, -0.7934581975537061, 1.1728304543768013, 2.048733564294805, 2.505621601353706, -6.203080369136732, -1.1990524723305254, -2.348548991370978, -3.648978609221488]\n", "iter:221\n", "w: [2.5104939909600983, 0.07608595862721743, 1.0740570452750382, -2.9496626182056467, 1.1270174235551296, 3.6291436942922317, -0.17387300656885205, -1.482987745787902, 0.9884189706414895, -1.2797147925315548, 1.3499161919713443, -0.7952985492252712, 1.1748471084487768, 2.051870365493183, 2.511116750142935, -6.215157217606067, -1.2014668720631867, -2.3526582673169285, -3.6560672229385505]\n", "iter:222\n", "w: [2.5151071344059925, 0.07583013639646849, 1.0761504462805862, -2.955027769819105, 1.1293684656749716, 3.6350302145238507, -0.17362961307926456, -1.4858498790247703, 0.9901680375147062, -1.281801589785486, 1.3520594948236297, -0.7971314237389315, 1.176856849871407, 2.0549981555844776, 2.51658946203356, -6.2271929660804215, -1.2038734802403164, -2.35675351499188, -3.663132805900806]\n", "iter:223\n", "w: [2.5197053825431004, 0.07557607444042416, 1.078236481962845, -2.960376050712862, 1.1317120981921829, 3.640898462313682, -0.17338783639444272, -1.488701890921616, 0.9919116610149595, -1.2838824254751595, 1.3541952086005045, -0.7989568810859182, 1.1788597297318153, 2.058116999181878, 2.5220399183915836, -6.23918791516209, -1.2062723511147804, -2.360834841826124, -3.670175528566296]\n", "iter:224\n", "w: [2.5242888463886555, 0.07532375768986722, 1.0803152062086947, -2.9657075790394973, 1.1340483716848877, 3.646748572216983, -0.1731476635693235, -1.4915438554733365, 0.9936498791172251, -1.2859573419616843, 1.3563233929228513, -0.8007749805758616, 1.1808557985378247, 2.061226960048397, 2.5274682985080172, -6.251142362086561, -1.2086635383570026, -2.36490235395316, -3.6771955593460413]\n", "iter:225\n", "w: [2.528857635610766, 0.07507317121554063, 1.08238667232656, -2.971022471621013, 1.1363773361977418, 3.6525806771108784, -0.1729090817599287, -1.4943758458774754, 0.9953827293704306, -1.2880263810540922, 1.3584441066803277, -0.8025857808464643, 1.1828451062266783, 2.0643281011132215, 2.5328747796284805, -6.263056600772497, -1.2110470950632692, -2.3689561562308166, -3.6841930646376846]\n", "iter:226\n", "w: [2.533411858550867, 0.07482430022714286, 1.0844509330542977, -2.9763208439694617, 1.1386990412494076, 3.6583949082237335, -0.17267207822321484, -1.4971979345452533, 0.9971102489040188, -1.2900895840198685, 1.360557408043383, -0.8043893398730136, 1.1848277021735987, 2.06742048448767, 2.538259536982307, -6.274930921870802, -1.2134230737638876, -2.372996352261941, -3.6911682088584414]\n", "iter:227\n", "w: [2.5379516222457026, 0.07457713007231175, 1.0865080405669594, -2.9816028103071712, 1.1410135358398987, 3.6641913951638756, -0.17243664031689038, -1.5000101931124143, 0.9988324744343832, -1.2921469915952295, 1.3626633544750326, -0.806185714977737, 1.186803635200188, 2.070504171480753, 2.543622743811167, -6.2867656128127924, -1.2157915264311998, -2.3770230444146705, -3.698121154477376]\n", "iter:228\n", "w: [2.542477032448858, 0.07433164623559846, 1.0885580464844287, -2.9868684835865715, 1.1433208684577998, 3.669970265947685, -0.17220275549920047, -1.5028126924498937, 1.000549442271178, -1.2941986439951496, 1.3647620027423903, -0.8079749628390015, 1.188772953582673, 2.0735792226143634, 2.5489645713972164, -6.2985609578574895, -1.218152504487458, -2.381036333842295, -3.7050520620470224]\n", "iter:229\n", "w: [2.546988193651844, 0.0740878343374325, 1.0906010018789365, -2.992117975509635, 1.1456210870873607, 3.6757316470270704, -0.17197041132868246, -1.5056055026743094, 1.0022611883235062, -1.2962445809231498, 1.3668534089279691, -0.809757139500363, 1.1907357050599976, 2.0766456976380994, 2.554285189090781, -6.3103172381380555, -1.22050605881256, -2.3850363205027216, -3.711961090234362]\n", "iter:230\n", "w: [2.5514852091047495, 0.07384568013307773, 1.0926369572824561, -2.9973513965469403, 1.1479142392154684, 3.681475663316345, -0.17173959546389322, -1.5083886931582815, 1.0039677481059888, -1.2982848415808494, 1.3689376284407526, -0.8115323003794648, 1.1926919368417674, 2.07970365554373, 2.5595847643375826, -6.322034731707387, -1.2228522397516528, -2.3890231031775477, -3.718848395851176]\n", "iter:231\n", "w: [2.555968180836472, 0.07360516951158243, 1.0946659626939812, -3.002568855956365, 1.1502003718385014, 3.687202438218516, -0.1715102956631093, -1.5111623325405819, 1.0056691567447176, -1.3003194646772933, 1.3710147160270447, -0.8133005002767913, 1.1946416956160493, 2.0827531545793168, 2.564863462705518, -6.333713713582892, -1.225191097122602, -2.392996779490755, -3.725714133883789]\n", "iter:232\n", "w: [2.560437209674536, 0.0733662884947223, 1.0966880675866857, -3.007770461801422, 1.152479531469065, 3.6929120936510103, -0.1712824997840021, -1.513926488736117, 1.0073654489830939, -1.3023484884380547, 1.3730847257811016, -0.8150617933842778, 1.1965850275570278, 2.085794252263001, 2.570121447910997, -6.345354455790458, -1.2275226802233337, -2.3969574459270335, -3.7325584575222104]\n", "iter:233\n", "w: [2.564892395264507, 0.07312902323593817, 1.0987033209149728, -3.0129563209692383, 1.1547517641426144, 3.6986047500708428, -0.17105619578328893, -1.5166812289457494, 1.009056659187557, -1.304371950614126, 1.3751477111555521, -0.8168162332937781, 1.1985219783325238, 2.088827005396462, 2.575358881844849, -6.35695722740764, -1.2298470378390485, -2.400905197849741, -3.7393815181887007]\n", "iter:234\n", "w: [2.5693338360890174, 0.07289336001926909, 1.100711771121411, -3.018126539188198, 1.1570171154239646, 3.7042805264992453, -0.17083137171636179, -1.5194266196659574, 1.0107428213532024, -1.3063898884905993, 1.3772037249716107, -0.8185638730053956, 1.2004525931113759, 2.091851470078061, 2.580575924597805, -6.368522294606068, -1.2321642182493135, -2.4048401295185107, -3.746183465565765]\n", "iter:235\n", "w: [2.5737616294864054, 0.07265928525828172, 1.102713466143559, -3.0232812210452424, 1.1592756304136906, 3.70993954054577, -0.17060801573689474, -1.5221627266983393, 1.0124239691092969, -1.3084023388951438, 1.379252819429089, -0.8203047649356774, 1.2023769165706897, 2.094867701715674, 2.585772734485561, -6.380049920693114, -1.2344742692350326, -2.408762334106513, -3.7529644476235924]\n", "iter:236\n", "w: [2.578175871668981, 0.07242678549499615, 1.1047084534206857, -3.0284204700028465, 1.1615273537544206, 3.7155819084318806, -0.17038611609643112, -1.5248896151589633, 1.0141001357246873, -1.310409338206287, 1.3812950461162095, -0.8220389609256754, 1.2042949929029563, 2.0978757550392286, 2.590949468073438, -6.391540366152812, -1.2367772380852962, -2.4126719037173796, -3.759724610646956]\n", "iter:237\n", "w: [2.5825766577409284, 0.07219584739880969, 1.1066967799003824, -3.0335443884156748, 1.1637723296370222, 3.721207745014044, -0.17016566114395204, -1.5276073494875655, 1.0157713541131095, -1.3124109223615024, 1.383330456019227, -0.8237665122488765, 1.206206865823043, 2.1008756841129475, 2.5961062802006327, -6.402993888686057, -1.2390731716041175, -2.416568929401799, -3.7664640992615803]\n", "iter:238\n", "w: [2.5869640817158506, 0.07196645776541853, 1.1086784920450714, -3.0386530775469223, 1.1660106018066874, 3.7268171638063343, -0.16994663932542695, -1.5303159934566009, 1.0174376568383983, -1.314407126865112, 1.3853590995318612, -0.8254874696190044, 1.2081125785750602, 2.1038675423473086, 2.6012433240040824, -6.414410743250101, -1.2413621161170525, -2.420453501173789, -3.7731830564599953]\n", "iter:239\n", "w: [2.591338236533967, 0.07173860351573827, 1.1106536358384125, -3.043746637584352, 1.1682422135689157, 3.7324102770025633, -0.16972903918334778, -1.5330156101801475, 1.0190990761196004, -1.3163979867960065, 1.387381026464546, -0.8272018831976947, 1.2100121739391057, 2.1068513825107305, 2.606360750941936, -6.4257911820973534, -1.2436441174777098, -2.424325708026657, -3.779881623626882]\n", "iter:240\n", "w: [2.595699214078971, 0.07151227169482414, 1.1126222567916098, -3.048825167656033, 1.1704672077953964, 3.7379871954979467, -0.16951284935624705, -1.5357062621226694, 1.0207556438359926, -1.3183835368151886, 1.3893962860534999, -0.8289098026020468, 1.2119056942378887, 2.1098272567409895, 2.61145871081665, -6.437135454813505, -1.2459192210741492, -2.428185637948651, -3.7865599405639307]\n", "iter:241\n", "w: [2.600047105194556, 0.07128744947079038, 1.1145843999496192, -3.0538887658457847, 1.172685626929796, 3.743548028910322, -0.16929805857820174, -1.538388011107637, 1.0224073915320082, -1.3203638111731457, 1.391404926969619, -0.830611276912054, 1.2137931813432385, 2.112795216556377, 2.616537351797706, -6.448443808354986, -1.2481874718351758, -2.4320333779383123, -3.793218145514211]\n", "iter:242\n", "w: [2.604381999700617, 0.07106412413373088, 1.1165401098972594, -3.058937529208337, 1.1748975129934494, 3.749092885600923, -0.16908465567832281, -1.5410609183260129, 1.0240543504220727, -1.3223388437170545, 1.393406997327201, -0.8323063546779139, 1.215674676682497, 2.1157553128666025, 2.6215968204439695, -6.459716487085778, -1.250448914236526, -2.4358690140195356, -3.799856375186077]\n", "iter:243\n", "w: [2.6087039864091364, 0.07084228309464055, 1.1184894307652278, -3.063971553784212, 1.177102907590956, 3.7546218726947287, -0.16887262958023258, -1.5437250443446004, 1.0256965513953513, -1.3243086678978235, 1.395402544692501, -0.8339950839272218, 1.2175502212448002, 2.1187075959834503, 2.626637261725681, -6.470953732813587, -1.2527035923069532, -2.439692631256339, -3.8064747647766115]\n", "iter:244\n", "w: [2.613013153139762, 0.07062191388433763, 1.1204324062360236, -3.0689909346143307, 1.1793018519156875, 3.760135096100391, -0.16866196930152919, -1.5463804491142603, 1.0273340250204102, -1.3262733167769771, 1.3973916160921227, -0.8356775121720487, 1.2194198555872489, 2.1216521156311954, 2.631658819046099, -6.482155784825391, -1.25495154963421, -2.4435043137673604, -3.813073447994618]\n", "iter:245\n", "w: [2.6173095867350797, 0.07040300415238857, 1.1223690795497778, -3.073995765754356, 1.1814943867552024, 3.765632660529757, -0.1684526639532409, -1.549027191977997, 1.0289668015497921, -1.3282328230333862, 1.3993742580212511, -0.8373536864159069, 1.221283619840973, 2.1245889209567883, 2.6366616342627958, -6.4933228799223786, -1.2571928293709342, -2.4473041447400736, -3.819652557083178]\n", "iter:246\n", "w: [2.6215933730755965, 0.07018554166603459, 1.1242994935099941, -3.07898614028877, 1.183680552496575, 3.7711146695169897, -0.168244702739269, -1.5516653316789184, 1.0305949109245114, -1.3301872189698478, 1.4013505164517284, -0.8390236531606038, 1.2231415537170893, 2.1275180605398085, 2.641645847708608, -6.504455252454291, -1.2594274742404354, -2.4510922064447427, -3.8262122228417765]\n", "iter:247\n", "w: [2.625864597094429, 0.06996951430912134, 1.1262236904892011, -3.0839621503447012, 1.1858603891316368, 3.7765812254373032, -0.16803807495582176, -1.554294926368068, 1.0322183827784661, -1.3321365365195197, 1.4033204368399765, -0.8406874584129871, 1.2249936965125579, 2.130439582402202, 2.6466115982122607, -6.515553134353176, -1.2616555265423874, -2.4548685802481116, -3.832752574648012]\n", "iter:248\n", "w: [2.630123342791717, 0.06975491008103157, 1.1281417124345172, -3.0889238871054996, 1.1880339362621337, 3.78203242952532, -0.1678327699908391, -1.5569160336121366, 1.0338372464427728, -1.3340808072522135, 1.4052840641347724, -0.8423451476915828, 1.2268400871159348, 2.133353534017797, 2.6515590231186548, -6.526616755166568, -1.2638770281584284, -2.45863334662684, -3.839273740478891]\n", "iter:249\n", "w: [2.634369693248756, 0.06954171709562074, 1.1300536008731303, -3.0938714408240666, 1.1902012331047989, 3.7874683818930537, -0.167628777323409, -1.5595287104010502, 1.0354515309500223, -1.3360200623805476, 1.407241442784875, -0.843996766033128, 1.2286807640130286, 2.136259962321617, 2.6564882583088374, -6.5376463420901025, -1.2660920205576676, -2.462386585180688, -3.845775846931728]\n", "iter:250\n", "w: [2.638603730641866, 0.06932992358015697, 1.1319593969176929, -3.098804900835949, 1.1923623184963466, 3.7928891815475367, -0.1674260865231763, -1.5621330131554418, 1.0370612650384619, -1.3379543327659675, 1.4091926167465112, -0.8456423579990011, 1.2305157652924583, 2.139158913718983, 2.6613994382196533, -6.548642119999585, -1.2683005448021045, -2.4661283746454576, -3.85225901924465]\n", "iter:251\n", "w: [2.6428255362559936, 0.06911951787426457, 1.1338591412716355, -3.1037243555721963, 1.1945172308983836, 3.7982949264080914, -0.16722468724974468, -1.564728997734005, 1.0386664771561007, -1.339883648924634, 1.411137629490722, -0.8472819676815497, 1.232345128651117, 2.1420504340944215, 2.6662926958630857, -6.559604311482514, -1.2705026415519587, -2.4698587929056957, -3.8587233813167203]\n", "iter:252\n", "w: [2.64703519049806, 0.06891048842887235, 1.135752874234398, -3.1086298925719924, 1.1966660084022436, 3.8036857133232584, -0.1670245692520718, -1.5673167194407343, 1.0402671954647453, -1.3418080410331834, 1.4130765240105716, -0.8489156387103192, 1.2341688913995421, 2.144934568820378, 2.6711681628452912, -6.57053313686908, -1.2726983510709136, -2.4735779170071623, -3.865169055727687]\n", "iter:253\n", "w: [2.6512327729100567, 0.06870282380516603, 1.1376406357065825, -3.113521598495061, 1.1988086887337444, 3.809061638087393, -0.166825722367859, -1.5698962330320514, 1.0418634478439623, -1.3437275389343644, 1.4150093428282235, -0.8505434142581837, 1.235987090467195, 2.147811362765739, 2.6760259693853334, -6.581428814262639, -1.2748877132312768, -2.477285823169071, -3.871596163757366]\n", "iter:254\n", "w: [2.655418362181896, 0.0684965126735458, 1.1395224651950262, -3.118399559133854, 1.2009453092578695, 3.8144227954569274, -0.16662813652293557, -1.5724675927238234, 1.0434552618949728, -1.345642172142552, 1.4169361280018875, -0.852165337047381, 1.237799762407651, 2.1506808603041754, 2.6808662443336253, -6.592291559569684, -1.2770707675190562, -2.4809825867961055, -3.8780048254046653]\n", "iter:255\n", "w: [2.6595920361640215, 0.06829154381258926, 1.141398401817799, -3.123263859425524, 1.2030759069833763, 3.81976927916632, -0.16643180173063812, -1.5750308521982703, 1.0450426649444786, -1.3475519698491454, 1.4188569211326365, -0.8537814493554529, 1.2396069434037023, 2.1535431053223006, 2.685689115190078, -6.603121586529313, -1.2792475530389558, -2.48466828249022, -3.8843951594062602]\n", "iter:256\n", "w: [2.663753871879784, 0.06808790610801899, 1.143268484309124, -3.12811458346369, 1.2052005185673302, 3.8251011819436873, -0.16623670809118582, -1.5775860646107678, 1.0466256840484214, -1.3494569609278495, 1.4207717633711008, -0.8553917930210927, 1.2414086692723731, 2.1563981412276605, 2.690494708121968, -6.613919106742209, -1.2814181085192902, -2.4883429840622218, -3.8907672832549274]\n", "iter:257\n", "w: [2.667903945537589, 0.06788558855167663, 1.1451327510242257, -3.132951814509993, 1.2073191803195693, 3.830418595526134, -0.16604284579105164, -1.5801332825965446, 1.0482043459956771, -1.3513571739398447, 1.4226806954240394, -0.8569964094499024, 1.243204975469855, 2.159246010956549, 2.6952831479815225, -6.624684329699145, -1.283582472316823, -2.4920067645431474, -3.89712131321754]\n", "iter:258\n", "w: [2.672042332542817, 0.06768458024050172, 1.1469912399441018, -3.137775635005457, 1.2094319282070976, 3.835721610674783, -0.1658502051023303, -1.5826725582772772, 1.049778677311687, -1.3532526371388471, 1.4245837575607925, -0.8585953396200593, 1.2449958970963553, 2.162086756981661, 2.7000545583232336, -6.635417462809013, -1.2857406824215265, -2.4956596961954336, -3.9034573643527404]\n", "iter:259\n", "w: [2.676169107509529, 0.0674848703755173, 1.1488439886802269, -3.142586126581649, 1.2115387978584102, 3.8410103171895176, -0.16565877638210333, -1.5852039432675837, 1.0513487042620258, -1.3551433784760631, 1.4264809896196158, -0.8601886240878955, 1.2467814689008672, 2.1649204213195805, 2.7048090614208995, -6.646118711426399, -1.287892776461267, -2.499301850523889, -3.909775550528288]\n", "iter:260\n", "w: [2.6802843442719535, 0.06728644826082109, 1.1506910344791828, -3.1473833700716463, 1.2136398245677518, 3.8462848039234347, -0.16546855007180142, -1.5877274886814197, 1.0529144528559078, -1.3570294256050388, 1.4283724310139023, -0.8617763029933916, 1.2485617252858612, 2.167747045538116, 2.709546778284403, -6.6567882788787, -1.2900387917064167, -2.502933298286467, -3.9160759844380966]\n", "iter:261\n", "w: [2.6843881158957648, 0.0670893033025831, 1.152532414227221, -3.1521674455208184, 1.2157350432993073, 3.851545158797027, -0.16527951669656482, -1.5902432451383743, 1.0544759488496358, -1.3589108058864094, 1.4302581207382914, -0.8633584160655836, 1.2503367003118968, 2.1705666707634768, 2.71426782867623, -6.667426366492804, -1.2921787650743906, -2.5065541095048505, -3.9223587776189635]\n", "iter:262\n", "w: [2.688480494689157, 0.06689342500804998, 1.1543681644547572, -3.1569384321974217, 1.2178244886913294, 3.8567914688120872, -0.16509166686460156, -1.5927512627698737, 1.056033217749988, -1.36078754639255, 1.4321380973746678, -0.8649350026278892, 1.2521064277021605, 2.1733793376873067, 2.718972331127729, -6.678033173621331, -1.2943127331341158, -2.5101643534748455, -3.928624040466998]\n", "iter:263\n", "w: [2.6925615522137156, 0.06669880298455542, 1.1561983213407963, -3.161696408603014, 1.2199081950602015, 3.862023820065353, -0.16490499126654493, -1.5952515912252876, 1.0575862848175501, -1.362659673912129, 1.434012399098055, -0.8665061016033502, 1.2538709408469277, 2.176185086573566, 2.723660402955127, -6.688608897668452, -1.2964407321104279, -2.513764098776594, -3.9348718822537583]\n", "iter:264\n", "w: [2.6966313592950946, 0.06650542693853753, 1.1580229207172952, -3.166441452482694, 1.2219861964044383, 3.8672422977618903, -0.16471948067480896, -1.5977442796779466, 1.0591351750699896, -1.3645272149545684, 1.4358810636824024, -0.8680717515197949, 1.2556302728079516, 2.1789839572652734, 2.7283321602752872, -6.699153734115289, -1.2985627978884007, -2.517353413284606, -3.9411024111420994]\n", "iter:265\n", "w: [2.700689986033499, 0.066313286674563, 1.1598419980734582, -3.1711736408351623, 1.2240585264086243, 3.8724469862282227, -0.16453512594294392, -1.6002293768310658, 1.0606799132852758, -1.3663901957544131, 1.4377441285062713, -0.8696319905149221, 1.2573844563227814, 2.1817759891911104, 2.732987718021237, -6.709667876544904, -1.3006789660176072, -2.5209323641776136, -3.947315734201738]\n", "iter:266\n", "w: [2.7047375018139803, 0.06612237209435813, 1.1616555885599695, -3.1758930499226197, 1.2261252184472937, 3.877637968925217, -0.1643519180049909, -1.6027069309235817, 1.0622205240048461, -1.3682486422756104, 1.4396016305584187, -0.8711868563413072, 1.2591335238090096, 2.1845612213718897, 2.737627189957449, -6.720151516666887, -1.3027892717163163, -2.524501017948256, -3.9535119574245465]\n", "iter:267\n", "w: [2.7087739753165487, 0.06593267319584718, 1.1634637269931634, -3.1805997552804923, 1.228186305588749, 3.882815328460724, -0.16416984787483627, -1.6051769897359003, 1.0637570315367175, -1.370102580215702, 1.4414536064432841, -0.8727363863713316, 1.2608775073684495, 2.187339692426891, 2.7422506886948903, -6.730604844341551, -1.3048937498756226, -2.5280594404125933, -3.959691185739572]\n", "iter:268\n", "w: [2.7127994745261037, 0.06574418007219703, 1.1652664478591321, -3.185293831726999, 1.2302418205988233, 3.8879791466019844, -0.16398890664556628, -1.607639600595559, 1.065289459958549, -1.3719520350099328, 1.44330009238638, -0.8742806176020373, 1.2626164387912469, 2.1901114405800692, 2.7468583257058428, -6.741028047603741, -1.3069924350635143, -2.5316076967194565, -3.965853523027796]\n", "iter:269\n", "w: [2.7168140667421894, 0.06555688291087014, 1.1670637853177748, -3.189975353372555, 1.232291795944584, 3.893129504287805, -0.16380908548882198, -1.610094810382806, 1.0668178331206506, -1.3737970318352741, 1.4451411242395868, -0.8758195866599086, 1.2643503495599242, 2.192876503666137, 2.751450211338498, -6.751421312686262, -1.3090853615288771, -2.535145851359635, -3.9719990721366325]\n", "iter:270\n", "w: [2.7208178185885767, 0.06537077199268373, 1.168855773206785, -3.1946443936290216, 1.2343362637979824, 3.8982664816405066, -0.1636303756541549, -1.6125426655360935, 1.0683421746489445, -1.3756375956143692, 1.4469767374863578, -0.8773533298055799, 1.2660792708533601, 2.1956349191365234, 2.7560264548313276, -6.761784824042933, -1.3111725632054387, -2.538673968174906, -3.9781279348941756]\n", "iter:271\n", "w: [2.7248107960226773, 0.0651858376908771, 1.170642445045581, -3.1993010252188028, 1.2363752560394459, 3.9033901579776527, -0.16345276846838316, -1.6149832120574912, 1.0698625079478767, -1.377473751019397, 1.4488069672468313, -0.8788818829384741, 1.2678032335507046, 2.198386724065217, 2.760587164327238, -6.772118764371281, -1.313254073715651, -2.542192110366908, -3.9842402121231975]\n", "iter:272\n", "w: [2.7287930643447895, 0.06500207047018584, 1.1724238340391788, -3.203945320183789, 1.2384088042614179, 3.9085006118235626, -0.16327625533494913, -1.6174164955180204, 1.0713788562032807, -1.3793055224758617, 1.4506318482828562, -0.8804052816013697, 1.2695222682352323, 2.2011319551544863, 2.765132446887509, -6.782423314634872, -1.3153299263745144, -2.545700340505863, -3.9903360036549045]\n", "iter:273\n", "w: [2.7327646882071877, 0.06481946088592369, 1.1741999730820072, -3.2085773498941554, 1.2404369397718438, 3.913597920920614, -0.16310082773327794, -1.6198425610629066, 1.0728912423851942, -1.3811329341663043, 1.452451415002929, -0.8819235609848995, 1.2712364051981344, 2.2038706487404935, 2.769662408505522, -6.792698654085292, -1.317400154193343, -2.549198720539151, -3.996415408342456]\n", "iter:274\n", "w: [2.7367257316230464, 0.06463799958307176, 1.1759708947616687, -3.2131971850570187, 1.2424596935976056, 3.918682162240338, -0.16292647721813766, -1.6222614534167576, 1.0743996892506302, -1.3829560100339433, 1.4542657014670486, -0.8834367559319829, 1.2729456744422498, 2.206602840798791, 2.7741771541202827, -6.802944960283789, -1.319464789883475, -2.552687311799736, -4.0024785240742515]\n", "iter:275\n", "w: [2.7406762579752155, 0.06445767729537556, 1.1777366313626456, -3.217804895724945, 1.244477096487904, 3.9237534119943116, -0.16275319541900116, -1.624673216888663, 1.0759042193463035, -1.3847747737862424, 1.4560747413914845, -0.88494490094219, 1.2746501056857376, 2.2093285669497127, 2.778676787629737, -6.813162409122562, -1.321523865859924, -2.5561661750144506, -4.008525447786986]\n", "iter:276\n", "w: [2.7446163300248383, 0.06427848484444948, 1.1794972148699512, -3.2224005513043266, 1.2464891789175914, 3.928811745644856, -0.16258097403940977, -1.62707789537722, 1.077404855011312, -1.3865892488984086, 1.4578785681534672, -0.8864480301760417, 1.2763497283656913, 2.212047862463659, 2.783161411903891, -6.823351174845733, -1.3235774142449774, -2.559635370312142, -4.014556275478488]\n", "iter:277\n", "w: [2.7485460099198242, 0.06410041313888835, 1.1812526769727298, -3.226984220563619, 1.2484959710904553, 3.9338572379155394, -0.16240980485633896, -1.6294755323754857, 1.0789016183797755, -1.3883994586168227, 1.4596772147957953, -0.8879461774592453, 1.2780445716416964, 2.2147607622662786, 2.787631128797732, -6.833511430069987, -1.3256254668717407, -2.563094957231677, -4.020571102220333]\n", "iter:278\n", "w: [2.7524653592031747, 0.0639234531733877, 1.1830030490678034, -3.2315559716414457, 1.250497502942452, 3.93888996280149, -0.16223967971956613, -1.6318661709758566, 1.0803945313834313, -1.390205425962402, 1.4614707140313676, -0.8894393762868675, 1.2797346643993321, 2.2174673009435493, 2.7920860391639533, -6.843643345804886, -1.3276680552876274, -2.566544994729818, -4.026570022170248]\n", "iter:279\n", "w: [2.756374438821165, 0.06374759602787061, 1.1847483622631676, -3.236115872054571, 1.2524938041448945, 3.943909993579527, -0.16207059055104084, -1.6342498538748773, 1.0818836157541896, -1.392007173733899, 1.4632590982476363, -0.8909276598274446, 1.281420035253618, 2.2201675127467597, 2.796526242865497, -6.85374709147288, -1.3297052107577982, -2.569985541188962, -4.032553128584304]\n", "iter:280\n", "w: [2.760273309131388, 0.06357283286662278, 1.1864886473814382, -3.2406639887057476, 1.2544849041075914, 3.9489174028181098, -0.1619025293442575, -1.6366266233779787, 1.083368893026646, -1.3938047245111367, 1.4650423995109865, -0.8924110609270323, 1.283100712552407, 2.2228614315973956, 2.800951838787903, -6.863822834929, -1.3317369642685486, -2.5734166544247548, -4.0385205138289]\n", "iter:281\n", "w: [2.76416202991066, 0.06339915493743521, 1.188223934963247, -3.245200387891435, 1.2564708319819418, 3.9539122623871075, -0.16173548816363045, -1.6389965214041489, 1.0848503845405562, -1.3955981006581801, 1.466820649571041, -0.8938896121131957, 1.284776724379726, 2.225549091091932, 2.805362924851476, -6.873870742480253, -1.333763346530645, -2.576838391693575, -4.044472269392552]\n", "iter:282\n", "w: [2.7680406603627925, 0.06322655357075442, 1.1899542552705897, -3.249725135309397, 1.258451616663983, 3.9588946434674006, -0.1615694591438723, -1.6413595894905346, 1.0863281114432692, -1.3973873243264494, 1.4685938798648939, -0.8953633455989407, 1.286448098559064, 2.2282305245065337, 2.809759598023274, -6.883890978904722, -1.3357843879826141, -2.580250809699897, -4.050408485897476]\n", "iter:283\n", "w: [2.771909259126233, 0.06305502017884035, 1.1916796382901256, -3.254238296066179, 1.2604272867973951, 3.963864616560308, -0.16140443448937442, -1.643715868796978, 1.0878020946921243, -1.3991724174577727, 1.470362121521274, -0.8968322932865866, 1.2881148626566108, 2.2309057648016655, 2.8141419543289192, -6.893883707470365, -1.3378001187939803, -2.5836539646035317, -4.05632925311099]\n", "iter:284\n", "w: [2.775767884281575, 0.06288454625493214, 1.193400113736431, -3.258739934684464, 1.262397870776462, 3.968822251496853, -0.1612404064735911, -1.6460654001104862, 1.089272355056808, -1.4009534017873806, 1.4721254053646375, -0.8982964867715827, 1.289777043984443, 2.2335748446266153, 2.8185100888642314, -6.903849089953536, -1.3398105688684585, -2.5870479120267484, -4.06223465995671]\n", "iter:285\n", "w: [2.7796165933589463, 0.06271512337242127, 1.1951157110552055, -3.2632301151103134, 1.2643633967489876, 3.9737676174468612, -0.16107736743842635, -1.648408223849639, 1.090738913121676, -1.4027302988468455, 1.473883761919194, -0.8997559573462686, 1.2914346696036638, 2.236237796323933, 2.8228640958066946, -6.913787286657226, -1.3418157678470983, -2.5904327070612774, -4.068124794525576]\n", "iter:286\n", "w: [2.7834554433452667, 0.06254674318403247, 1.1968264594264337, -3.2677089007202973, 1.2663238926191724, 3.978700782927906, -0.1609153097936239, -1.6507443800689308, 1.0922017892880371, -1.4045031299669644, 1.4756372214128657, -0.9012107360035795, 1.2930877663274942, 2.238894651933783, 2.8272040684267514, -6.923698456429015, -1.3438157451113824, -2.5938084042752014, -4.073999744086688]\n", "iter:287\n", "w: [2.7872844906913916, 0.062379397421012665, 1.198532387767501, -3.272176354328508, 1.2682793860504462, 3.9836218158140935, -0.16075422601616074, -1.6530739084630528, 1.0936610037764023, -1.4062719162805883, 1.4773858137811813, -0.9026608534406977, 1.2947363607243152, 2.2415454431982176, 2.831530099098934, -6.93358275667877, -1.34581052978628, -2.5971750577197334, -4.079859595097961]\n", "iter:288\n", "w: [2.7911037913191326, 0.06221307789232672, 1.2002335247362654, -3.276632538193465, 1.2702299044682617, 3.9885307833446983, -0.16059410864964385, -1.6553968483711126, 1.0951165766286992, -1.4080366787253993, 1.479129568671106, -0.9041063400626506, 1.2963804791206655, 2.244190201565369, 2.835842279312829, -6.94344034339606, -1.3478001507432558, -2.6005327209358846, -4.0857044332166135]\n", "iter:289\n", "w: [2.7949134006281615, 0.06204777648386156, 1.2019298987340883, -3.2810775140249158, 1.2721754750628471, 3.9934277521326482, -0.16043495030371055, -1.657713238780794, 1.0965685277104507, -1.409797438046635, 1.4808685154448087, -0.9055472259858564, 1.298020147604193, 2.2468289581935643, 2.8401406996838876, -6.9532713711673235, -1.3497846366032353, -2.6038814469610236, -4.091534343309474]\n", "iter:290\n", "w: [2.7987133735028036, 0.061883485157637516, 1.203621537908819, -3.2855113429905254, 1.2741161247919202, 3.9983127881728637, -0.16027674365343192, -1.6600231183324585, 1.098016876712922, -1.4115542147997635, 1.4826026831833685, -0.9069835410416189, 1.2996553920265614, 2.2494617439553637, 2.8444254499640738, -6.963075993192768, -1.3517640157395265, -2.607221288335332, -4.097349409463131]\n", "iter:291\n", "w: [2.8025037643187147, 0.06172019595102734, 1.205308470157741, -3.2899340857224675, 1.2760518803833618, 4.003185956850455, -0.1601194814387205, -1.6623265253231876, 1.0994616431552322, -1.4133070293531098, 1.4843321006904195, -0.9084153147795714, 1.3012862380063137, 2.252088589441524, 2.848696619052357, -6.972854361303023, -1.3537383162807004, -2.6105522971081534, -4.103149714993907]\n", "iter:292\n", "w: [2.8062846269494526, 0.06155790097598276, 1.2069907231304733, -3.2943458023239107, 1.2779827683378537, 4.008047322948779, -0.159963156463741, -1.6646234977107688, 1.1009028463864359, -1.4150559018904316, 1.4860567964957383, -0.9098425764710703, 1.302912710931691, 2.2547095249648903, 2.8529542950050573, -6.982606625975544, -1.3557075661134292, -2.6138745248442423, -4.108935342457678]\n", "iter:293\n", "w: [2.810056014772938, 0.06139659241826828, 1.2086683242318348, -3.298746552375406, 1.2799088149314772, 4.012896950657358, -0.1598077615963253, -1.666914073117625, 1.1023405055875708, -1.4168008524134503, 1.4877767988587718, -0.911265355112541, 1.3045348359634106, 2.2573245805642146, 2.857198565046037, -6.992332936350771, -1.3576717928852842, -2.617188022629913, -4.114706373659533]\n", "iter:294\n", "w: [2.8138179806778116, 0.06123626253670298, 1.2103413006246657, -3.303136394941176, 1.2818300462182755, 4.0177349035796635, -0.15965328976739077, -1.6691982888346892, 1.1037746397736754, -1.4185419007443354, 1.4894921357721098, -0.9126836794287753, 1.3061526380374018, 2.259933786007907, 2.8614295155767446, -7.002033440248051, -1.3596310240074934, -2.6204928410790918, -4.1204628896632745]\n", "iter:295\n", "w: [2.8175705770696875, 0.061076903662409344, 1.2120096792326123, -3.3075153885753092, 1.2837464880327796, 4.022561244740775, -0.15949973397036293, -1.6714761818252237, 1.1052052677957753, -1.4202790665281433, 1.4912028349649005, -0.9140975778761804, 1.307766141867501, 2.262537170797717, 2.8656472321861166, -7.01170828418132, -1.3615852866576594, -2.623789030339269, -4.1262049708007735]\n", "iter:296\n", "w: [2.8213138558773054, 0.060918508198069724, 1.2136734867428702, -3.3118835913278577, 1.2856581659924997, 4.027376036594899, -0.15934708726060143, -1.673747788728587, 1.1066324083428396, -1.4220123692352125, 1.492908923906212, -0.9155070786459835, 1.3093753719481065, 2.2651347641723496, 2.8698517996603354, -7.021357613374558, -1.3635346077824395, -2.627076640097362, -4.131932696681167]\n", "iter:297\n", "w: [2.825047868558583, 0.06076106861718993, 1.2153327496088924, -3.316241060750844, 1.28756510550038, 4.032179341032772, -0.15919534275483035, -1.6760131458639471, 1.108056079943708, -1.4237418281635164, 1.494610429808341, -0.9169122096673887, 1.3109803525567933, 2.267726595111015, 2.8740433019924474, -7.030981571777022, -1.3654790141001867, -2.630355719585482, -4.137646146199913]\n", "iter:298\n", "w: [2.828772666106571, 0.06060457746337038, 1.216987494053058, -3.320587853904177, 1.2894673317472203, 4.036971219388932, -0.15904449363057258, -1.6782722892339443, 1.1094763009689883, -1.4254674624409727, 1.4963073796300657, -0.9183129986106902, 1.3125811077568907, 2.270312692336915, 2.878221822391841, -7.040580302078241, -1.3674185321035537, -2.633626317586613, -4.143345397547703]\n", "iter:299\n", "w: [2.832488299055315, 0.06044902734958445, 1.2186377460693048, -3.3249240273614764, 1.2913648697140625, 4.041751732448876, -0.15889453312558782, -1.6805252545283018, 1.1108930896329257, -1.4271892910277122, 1.4979998000798525, -0.9197094728903402, 1.3141776614000211, 2.272893084320669, 2.882387443293595, -7.050153945722808, -1.3693531880620597, -2.6368884824401984, -4.149030528219232]\n", "iter:300\n", "w: [2.836194817485618, 0.06029441095746411, 1.2202835314257254, -3.3292496372158134, 1.2932577441745454, 4.046520940456093, -0.15874545453731484, -1.6827720771273869, 1.1123064639952438, -1.428907332718308, 1.4996877176190064, -0.9211016596679743, 1.3157700371286003, 2.2754677992836747, 2.8865402463676872, -7.059702642924948, -1.3712830080246223, -2.6401422620476436, -4.154701615021825]\n", "iter:301\n", "w: [2.839892271030718, 0.06014072103659293, 1.2219248756671284, -3.3335647390853613, 1.295145979697226, 4.051278903118981, -0.15859725122231785, -1.6850127921057223, 1.1137164419629595, -1.4306216061439638, 1.501371158464776, -0.9224895858553934, 1.3173582583783032, 2.278036865201414, 2.8906803125280742, -7.069226532682873, -1.3732080178220516, -2.6433877038777305, -4.160358734083938]\n", "iter:302\n", "w: [2.84358070888187, 0.059987950403805855, 1.223561804117563, -3.3378693881189645, 1.2970296006478677, 4.056025679617657, -0.15844991659573707, -1.6872474342354502, 1.1151230412921684, -1.4323321297746656, 1.503050148593407, -0.923873278117504, 1.3189423483804918, 2.2806003098066974, 2.8948077219416457, -7.07872575279294, -1.3751282430695113, -2.6466248549719498, -4.166001960863518]\n", "iter:303\n", "w: [2.8472601797938415, 0.05983609194249642, 1.2251943418828108, -3.342163639001623, 1.2989086311916984, 4.060761328610645, -0.15830344413074324, -1.6894760379897473, 1.1165262795898059, -1.4340389219212935, 1.504724713743149, -0.9252527628752162, 1.3205223301646085, 2.2831581605928526, 2.898922554037046, -7.088200439863588, -1.3770437091689447, -2.6498537619497498, -4.171631370156232]\n", "iter:304\n", "w: [2.8509307320903226, 0.059685138601931115, 1.226822513852842, -3.3464475459598964, 1.3007830952956365, 4.065485908241462, -0.15815782735799616, -1.6916986375461938, 1.1179261743153808, -1.4357420007376978, 1.5063948794172164, -0.9266280663083019, 1.3220982265605339, 2.2857104448168575, 2.903024887513374, -7.097650729329086, -1.378954441311466, -2.6530744710137033, -4.177247036103574]\n", "iter:305\n", "w: [2.854592413669249, 0.05953508339657031, 1.228446344704239, -3.3507211627672264, 1.3026530167304864, 4.0701994761450955, -0.158013059865108, -1.6939152667900954, 1.1193227427826848, -1.4374413842227394, 1.5080606708867006, -0.9279992143582122, 1.3236700602009113, 2.288257189502417, 2.9071148003487632, -7.107076755463083, -1.3808604644797198, -2.656287027954597, -4.182849032200837]\n", "iter:306\n", "w: [2.8582452720080416, 0.05938591940539673, 1.2300658589025877, -3.3549845427491833, 1.3045184190731036, 4.07490208945437, -0.15786913529611002, -1.6961259593177604, 1.1207160021614755, -1.439137090222294, 1.5097221131934397, -0.9293662327308561, 1.325237853523437, 2.2907984214429886, 2.911192369808833, -7.116478651391962, -1.3827618034502065, -2.6594914781564425, -4.188437431304971]\n", "iter:307\n", "w: [2.8618893541687664, 0.059237639771250544, 1.2316810807048346, -3.3592377387886327, 1.3063793257085312, 4.07959380480622, -0.15772604735092424, -1.6983307484397328, 1.1221059694791347, -1.4408291364312213, 1.5113792311528425, -0.9307291468993393, 1.3268016287731177, 2.2933341672047534, 2.9152576724550303, -7.125856549107994, -1.3846584827955768, -2.6626878666014075, -4.194012305642313]\n", "iter:308\n", "w: [2.8655247068032104, 0.059090237700171376, 1.2332920341616136, -3.3634808033308268, 1.308235759832106, 4.084274678347854, -0.1575837897848386, -1.7005296671839787, 1.1234926616223042, -1.442517540395302, 1.5130320493566687, -0.9320879821066659, 1.3283614080044945, 2.2958644531295374, 2.9193107841528474, -7.135210579482325, -1.386550526886893, -2.665876237874677, -4.199573726816201]\n", "iter:309\n", "w: [2.8691513761578817, 0.05894370646074729, 1.2348987431195406, -3.3677137883884236, 1.3100877444515362, 4.0889447657428235, -0.157442356407987, -1.7027227482990333, 1.1248760953384962, -1.4442023195131393, 1.514680592175767, -0.9334427633684017, 1.3299172130838368, 2.298389305337682, 2.9233517800799285, -7.144540872277757, -1.3884379598958605, -2.669056636169238, -4.20512176581447]\n", "iter:310\n", "w: [2.8727694080789314, 0.05879803938347023, 1.2365012312234775, -3.37193674554643, 1.3119353023889513, 4.09360412217699, -0.15730174108483325, -1.704910024257102, 1.1262562872376798, -1.4458834910380305, 1.5163248837627707, -0.9347935154753011, 1.3314690656913035, 2.300908749730866, 2.9273807347340615, -7.153847556161362, -1.390320805797028, -2.672229105290589, -4.210656493016833]\n", "iter:311\n", "w: [2.8763788480169983, 0.0586532298600986, 1.2380995219187654, -3.376149725967075, 1.3137784562829238, 4.098252802364405, -0.1571619377336591, -1.7070915272571208, 1.1276332537938467, -1.4475610720798042, 1.5179649480547541, -0.936140262995897, 1.3330169873230733, 2.3034228119948783, 2.9313977219410594, -7.163130758716914, -1.3921990883699578, -2.675393688661384, -4.216177978202152]\n", "iter:312\n", "w: [2.8799797410319803, 0.05850927134302607, 1.2396936384534283, -3.3803527803946114, 1.3156172285904637, 4.102890860553087, -0.15702294032605693, -1.7092672892277758, 1.1290070113465525, -1.4492350796066287, 1.5196008087758446, -0.9374830302790557, 1.334560999293445, 2.3059315176023447, 2.935402814862531, -7.1723906064571405, -1.3940728312013655, -2.678550429326003, -4.2216862905555965]\n", "iter:313\n", "w: [2.883572131797732, 0.058366157344657554, 1.2412836038803472, -3.384545959160047, 1.3174516415889863, 4.107518350530715, -0.15688474288642631, -1.7114373418304814, 1.130377576102436, -1.4509055304467882, 1.5212324894397982, -0.9388218414564956, 1.3361011227369088, 2.3084348918154083, 2.9393960860035464, -7.181627224835812, -1.3959420576872326, -2.681699369955054, -4.22718149867569]\n", "iter:314\n", "w: [2.8871560646066907, 0.05822388143679114, 1.2428694410594046, -3.3887293121858115, 1.3192817173782527, 4.112135325630227, -0.15674733949147504, -1.7136017164623183, 1.1317449641367177, -1.45257244129043, 1.5228600133525356, -0.9401567204452724, 1.337637378610186, 2.3109329596883645, 2.9433776072201914, -7.19084073825965, -1.397806791034888, -2.684840552849809, -4.232663670581259]\n", "iter:315\n", "w: [2.890731583374433, 0.05808243725000716, 1.244451172659601, -3.3929028889903523, 1.3211074778822853, 4.116741838735331, -0.15661072426972428, -1.7157604442589336, 1.1331091913946756, -1.4542358286912824, 1.524483403614639, -0.941487690950229, 1.3391697876942426, 2.3134257460702523, 2.947347449727024, -7.200031270100081, -1.3996670542650624, -2.6879740199465716, -4.238132873718268]\n", "iter:316\n", "w: [2.8942987316441586, 0.057941818473063185, 1.2460288211611434, -3.397066738692666, 1.3229289448512573, 4.121337942285931, -0.15647489140101783, -1.7179135560973995, 1.1344702736931012, -1.4558957090683438, 1.526102683123812, -0.9428147764664122, 1.3406983705962723, 2.315913275607401, 2.951305684104422, -7.2091989427048135, -1.4015228702139158, -2.6910998128209833, -4.243589174966559]\n", "iter:317\n", "w: [2.897857552591111, 0.05780201885229553, 1.2476024088575055, -3.4012209100167654, 1.3247461398633564, 4.125923688283466, -0.15633983511603622, -1.7200610825990368, 1.1358282267217334, -1.4575520987075432, 1.5277178745773026, -0.9441380002814571, 1.3422231477516533, 2.3183955727459367, 2.9552523803058355, -7.218343877409268, -1.4033742615350353, -2.6942179726922624, -4.249032640646484]\n", "iter:318\n", "w: [2.9014080890269276, 0.05766303219102762, 1.2491719578574607, -3.4053654512960843, 1.3265590843266246, 4.130499128296171, -0.15620554969581382, -1.7222030541321989, 1.1371830660446736, -1.459205013763375, 1.5293290004742894, -0.945457385477938, 1.3437441394258767, 2.3208726617342457, 2.9591876076649335, -7.227466194547834, -1.4052212507014088, -2.697328540427381, -4.254463336525445]\n", "iter:319\n", "w: [2.904950383403926, 0.05752485234898391, 1.250737490087089, -3.409500410477817, 1.3283677994807728, 4.13506431346425, -0.15607202947126173, -1.724339500815019, 1.1385348071017793, -1.4608544702605042, 1.530936083118233, -0.9467729549356875, 1.3452613657164485, 2.3233445666253973, 2.9631114349026584, -7.236566013464983, -1.4070638600073695, -2.700431556545182, -4.259881327824331]\n", "iter:320\n", "w: [2.908484477819325, 0.057387473241710535, 1.2522990272917558, -3.4136258351271977, 1.3301723063989717, 4.139619294504969, -0.15593926882269343, -1.7264704525181218, 1.1398834652100396, -1.462500484095348, 1.5325391446191918, -0.9480847313340842, 1.3467748465547642, 2.325811311279527, 2.9670239301341805, -7.245643452526224, -1.4089021115705165, -2.7035270612204347, -4.2652866792238635]\n", "iter:321\n", "w: [2.912010414019405, 0.05725088884000197, 1.2538565910380672, -3.4177417724317207, 1.3319726259896185, 4.144164121717675, -0.1558072621793555, -1.7285959388672985, 1.1412290555649294, -1.4641430710376295, 1.5341382068961036, -0.9493927371543099, 1.3482846017079586, 2.32827291936618, 2.970925160875759, -7.254698629128905, -1.4107360273336083, -2.70661509428783, -4.2706794548708435]\n", "iter:322\n", "w: [2.915528233403602, 0.0571150931693338, 1.2554102027157967, -3.4218482692053, 1.3337687789980819, 4.148698844988738, -0.15567600401896234, -1.730715989246147, 1.1425715932417468, -1.4657822467319073, 1.535733291679034, -0.9506969946815753, 1.3497906507807294, 2.33072941436662, 2.9748151940515104, -7.26373165971287, -1.412565629066432, -2.7096956952459186, -4.276059718384314]\n", "iter:323\n", "w: [2.919037977028545, 0.05698008030930155, 1.256959883539789, -3.4259453718923703, 1.3355607860084213, 4.153223513796411, -0.15554548886723465, -1.7328306327986769, 1.1439110931969307, -1.4674180266990793, 1.5373244205113905, -0.9519975260073161, 1.351293013217135, 2.333180819576093, 2.9786940960000816, -7.2727426597709695, -1.414390938367647, -2.71276890326099, -4.281427532861624]\n", "iter:324\n", "w: [2.922539685612031, 0.05684584439306591, 1.2585056545518376, -3.4300331265719315, 1.3373486674450865, 4.157738177215621, -0.15541571129744267, -1.734939898431881, 1.1452475702693592, -1.4690504263378632, 1.5389116147521054, -0.9532943530313597, 1.3527917083023686, 2.335627158106062, 2.982561932481238, -7.281731743859428, -1.4162119766666048, -2.715834757170899, -4.286782960884408]\n", "iter:325\n", "w: [2.9260333995369416, 0.05671237960680308, 1.260047536622538, -3.4341115789615366, 1.339132443574593, 4.162242883922684, -0.15528666592995316, -1.7370438148182716, 1.1465810391816318, -1.4706794609262521, 1.5404948955777855, -0.9545874974640622, 1.3542867551645072, 2.3380684528864037, 2.9864187686823573, -7.290699025608068, -1.4180287652251444, -2.7188932954888307, -4.29212606452448]\n", "iter:326\n", "w: [2.9295191588551024, 0.056579680189162034, 1.2615855504531182, -3.4381807744212254, 1.3409121345071762, 4.1667376821999556, -0.15515834743178047, -1.7391424103983866, 1.1479115145413328, -1.4723051456229481, 1.5420742839848314, -0.9558769808284181, 1.3557781727762368, 2.340504726667565, 2.99026466922484, -7.299644617730396, -1.419841325139364, -2.721944556407018, -4.297456905349632]\n", "iter:327\n", "w: [2.9329970032910855, 0.05644774043072677, 1.2631197165772425, -3.442240757957401, 1.3426877601984246, 4.171222619940397, -0.15503075051614207, -1.741235713383259, 1.149239010842277, -1.4739274954687718, 1.5436498007915251, -0.9571628244621393, 1.3572659799565536, 2.3429360020226917, 2.9940996981704293, -7.308568632033557, -1.421649677341369, -2.7249885778003997, -4.302775544429364]\n", "iter:328\n", "w: [2.9364669722459564, 0.05631655467348496, 1.2646500553627957, -3.446291574226654, 1.34445934045089, 4.175697744652089, -0.15490386994201769, -1.7433237517568576, 1.1505635424657388, -1.4755465253880495, 1.5452214666400879, -0.9584450495197091, 1.3587501953724432, 2.345362301349719, 2.997923919027451, -7.317471179428143, -1.4234538426009984, -2.7280253972302297, -4.308082042340517]\n", "iter:329\n", "w: [2.939929104800967, 0.056186117310302035, 1.2661765870136403, -3.4503332675395373, 1.3462268949156793, 4.180163103462665, -0.15477770051371284, -1.7454065532784941, 1.1518851236816643, -1.4771622501899793, 1.5467893019987076, -0.959723676974407, 1.3602308375405359, 2.3477836468734314, 3.0017373947569657, -7.326352369937879, -1.4252538415275267, -2.7310550519476338, -4.3133764591728285]\n", "iter:330\n", "w: [2.943383439721191, 0.05605642278440104, 1.2676993315713545, -3.4543658818642813, 1.3479904430940233, 4.184618743123687, -0.15465223708042625, -1.7474841454851995, 1.153203768649867, -1.478774684569974, 1.5483533271635377, -0.9609987276203068, 1.361707924828739, 2.350200060647487, 3.0055401877788386, -7.335212312709181, -1.4270496945713453, -2.7340775788971134, -4.318658854534417]\n", "iter:331\n", "w: [2.946830015459113, 0.05592746558884779, 1.269218308916945, -3.458389460830467, 1.3497500043388269, 4.189064710014947, -0.15452747453582139, -1.7495565556940702, 1.1545194914212067, -1.4803838431109853, 1.5499135622606668, -0.9622702220742488, 1.3631814754578484, 2.3526115645564127, 3.009332359977733, -7.34405111602057, -1.4288414220256211, -2.737093014720004, -4.3239292875571715]\n", "iter:332\n", "w: [2.95026887015816, 0.05579924026604234, 1.27073353877254, -3.462404047732644, 1.3515055978561985, 4.193501050148712, -0.1544034078176019, -1.7516238110045836, 1.155832305938752, -1.4819897402848043, 1.5514700272480602, -0.9635381807777856, 1.3646515075031382, 2.3550181803175665, 3.0131139727090193, -7.352868887291984, -1.430629044027933, -2.740101395757879, -4.329187816902079]\n", "iter:333\n", "w: [2.9537000416561847, 0.05567174140721537, 1.2722450407030588, -3.4664096855338986, 1.35325724270696, 4.197927809173905, -0.15428003190709108, -1.7536859383008834, 1.1571422260389288, -1.4835923904533446, 1.553022741917473, -0.964802623999102, 1.3661180388959282, 2.3574199294830693, 3.0168850868046064, -7.3616657330939415, -1.4324125805618886, -2.7431027580559104, -4.334434500764469]\n", "iter:334\n", "w: [2.9571235674888974, 0.05554496365192996, 1.273752834117861, -3.4704064168693773, 1.355004957808138, 4.202345032380219, -0.15415734182881524, -1.7557429642540356, 1.158449265452651, -1.4851918078699027, 1.5545717258963365, -0.9660635718349093, 1.3675810874251302, 2.359816833441707, 3.0206457625786967, -7.370441759156599, -1.4341920514587188, -2.746097137366178, -4.339669396879181]\n", "iter:335\n", "w: [2.9605394848932503, 0.05541890168758943, 1.2752569382723742, -3.47439428404976, 1.356748761934435, 4.206752764702178, -0.15403533265009112, -1.7577949153242571, 1.1597534378064376, -1.486788006680401, 1.5561169986496186, -0.9673210442123159, 1.3690406707387748, 2.362208913420803, 3.024396059833465, -7.3791970703786856, -1.4359674763988521, -2.7490845691509365, -4.344892562525665]\n", "iter:336\n", "w: [2.9639478308107714, 0.05529355024894968, 1.2767573722697017, -3.4783733290646848, 1.3584886737196817, 4.211151050723132, -0.15391399948061696, -1.7598418177631137, 1.1610547566235139, -1.4883810009246103, 1.557658579481655, -0.9685750608906716, 1.370496806345517, 2.3645961904880606, 3.0281360378646625, -7.387931770836311, -1.4377388749134694, -2.75206508858583, -4.3501040545330065]\n", "iter:337\n", "w: [2.967348641890853, 0.05516890411763754, 1.2782541550622082, -3.4823435935861315, 1.360224711658273, 4.215539934679194, -0.15379333747206797, -1.761883697615691, 1.1623532353248982, -1.4899708045373548, 1.5591964875379583, -0.9698256414633891, 1.371949511616121, 2.3669786855533794, 3.0318657554671438, -7.396645963791671, -1.4395062663860374, -2.755038730563071, -4.355303929284882]\n", "iter:338\n", "w: [2.9707419544939913, 0.05504495812167396, 1.279747305453087, -3.4863051189717544, 1.3619568941065823, 4.21991946046313, -0.15367334181769488, -1.7639205807227385, 1.163648887230474, -1.4915574313496964, 1.5607307418069973, -0.9710728053597413, 1.3733988037849272, 2.3693564193706407, 3.035585270940327, -7.405339751701624, -1.4412696700538237, -2.758005529694564, -4.360492242724441]\n", "iter:339\n", "w: [2.9741278046949824, 0.054921707135002426, 1.2812368420979066, -3.490257946268175, 1.3636852392843606, 4.224289671628172, -0.153554007751927, -1.7659524927227843, 1.1649417255600478, -1.4931408950901037, 1.5622613611219556, -0.9723165718466343, 1.3748446999512975, 2.371729412539468, 3.0392946420935756, -7.414013236226166, -1.4430291050093915, -2.760965520314996, -4.365669050359127]\n", "iter:340\n", "w: [2.97750622828607, 0.054799146077022805, 1.2827227835061374, -3.494202116214227, 1.3654097652761172, 4.228650611391798, -0.1534353305499788, -1.7679794590542257, 1.1662317634343926, -1.4947212093856008, 1.5637883641624615, -0.973556960030359, 1.376287217081044, 2.37409768550696, 3.0429939262515164, -7.422666518236789, -1.4447845902020762, -2.7639187364848774, -4.370834407265422]\n", "iter:341\n", "w: [2.980877260780053, 0.05467726991212993, 1.2842051480426606, -3.4981376692441613, 1.3671304900324832, 4.2330023226394395, -0.15331730552746023, -1.7700015049573907, 1.1675190138762785, -1.4962983877629004, 1.565311769456297, -0.9747939888583179, 1.377726372007836, 2.3764612585693965, 3.0466831802592833, -7.431299697824729, -1.4465361444394422, -2.7668652119935424, -4.375988368093534]\n", "iter:342\n", "w: [2.984240937413345, 0.05455607364925724, 1.2856839539292557, -3.5020646454908055, 1.3688474313715564, 4.23734484792815, -0.15319992803999088, -1.772018655476576, 1.168803489811488, -1.4978724436495185, 1.5668315953810803, -0.9760276771207318, 1.3791621814345913, 2.37882015187392, 3.050362460487693, -7.439912874309112, -1.4482837863887223, -2.76980498036211, -4.381130987072012]\n", "iter:343\n", "w: [2.9875972931489936, 0.054435552341425796, 1.287159219246072, -3.5059830847886846, 1.3705606069802314, 4.24167822949021, -0.153083193482818, -1.7740309354620567, 1.1700852040698195, -1.4994433903748734, 1.5683478601659269, -0.9772580434523218, 1.3805946619348466, 2.381174385420189, 3.0540318228383563, -7.448506146244988, -1.4500275345782379, -2.772738074846401, -4.386262318012303]\n", "iter:344\n", "w: [2.990946362679658, 0.05431570108529762, 1.288630961933079, -3.5098930266770987, 1.3722700344155119, 4.246002509236684, -0.1529670972904377, -1.7760383695720738, 1.1713641693860768, -1.5010112411713676, 1.5698605818930857, -0.9784851063339721, 1.382023829954112, 2.3835239790620126, 3.05769132274872, -7.457079611431266, -1.4517674073988016, -2.775664528439819, -4.391382414313241]\n", "iter:345\n", "w: [2.994288180430543, 0.05419651502073433, 1.2900991997915012, -3.5137945104031596, 1.3739757311058067, 4.250317728760929, -0.15285163493622073, -1.7780409822747938, 1.1726403984010454, -1.5025760091754536, 1.5713697784995553, -0.9797088840943695, 1.3834497018112077, 2.385868952508951, 3.0613410151970446, -7.465633366918541, -1.4535034231051023, -2.7785843738761873, -4.396491328965476]\n", "iter:346\n", "w: [2.9976227805622937, 0.05407798933036064, 1.2915639504852339, -3.51768757492479, 1.3756777143522099, 4.254623929342051, -0.15273680193204087, -1.7800387978502457, 1.173913903662457, -1.5041377074286837, 1.572875467778674, -0.9809293949116227, 1.3848722936995812, 2.388209325327902, 3.0649809547073192, -7.474167509016826, -1.4552355998170732, -2.7814976436325516, -4.401589114555846]\n", "iter:347\n", "w: [3.0009501969738492, 0.053960119239132236, 1.29302523154224, -3.5215722589136806, 1.377376001329765, 4.258921151948314, -0.15262259382790772, -1.782031840392233, 1.1751846976259406, -1.5056963488787451, 1.5743776673816916, -0.9821466568148607, 1.38629162168861, 2.3905451169446583, 3.068611195354109, -7.482682133303175, -1.456963955521242, -2.7844043699319414, -4.406675823271684]\n", "iter:348\n", "w: [3.0042704633052586, 0.05384290001390866, 1.2944830603559314, -3.5254486007582115, 1.3790706090887128, 4.263209437240498, -0.1525090062116027, -1.7840201338102228, 1.1764527926559607, -1.5072519463804783, 1.575876394819317, -0.9833606876858106, 1.387707701724886, 2.3928763466454415, 3.0722317907673444, -7.491177334629224, -1.4586885080720635, -2.7873045847460944, -4.411751506905065]\n", "iter:349\n", "w: [3.0075836129404565, 0.053726326963031146, 1.2959374541865332, -3.529316638566333, 1.3807615545557257, 4.26748882557521, -0.15239603470831875, -1.7860037018312096, 1.177718201026745, -1.5088045126968819, 1.5773716674632474, -0.9845715052603554, 1.3891205496334833, 2.3952030335784174, 3.0758427941370456, -7.499653207128618, -1.4604092751932383, -2.790198319798147, -4.416816216857004]\n", "iter:350\n", "w: [3.0108896790100013, 0.0536103954359042, 1.2973884301624294, -3.5331764101684096, 1.382448854535124, 4.271759357008155, -0.15228367498030349, -1.7879825680015582, 1.1789809349231986, -1.5103540605001013, 1.5788635025476725, -0.9857791271300715, 1.3905301811192103, 2.3975251967551854, 3.07944425821799, -7.50810984422435, -1.4621262744790116, -2.7930856065652825, -4.421870004141589]\n", "iter:351\n", "w: [3.0141886943937752, 0.05349510082258279, 1.2988360052814938, -3.5370279531200293, 1.384132525710079, 4.276021071297351, -0.15217192272650587, -1.789956755688822, 1.1802410064418067, -1.511900602372405, 1.5803519171707625, -0.9869835707437473, 1.3919366117678451, 2.3998428550522473, 3.0830362353343155, -7.5165473386360135, -1.4638395233954584, -2.795966476281351, -4.426912919390056]\n", "iter:352\n", "w: [3.017480691723648, 0.05338043855336321, 1.3002801964124029, -3.540871304704775, 1.385812584643802, 4.280274007906308, -0.15206077368222645, -1.7919262880835405, 1.1814984275915268, -1.513444150807144, 1.5818369282961335, -0.9881848534088817, 1.3933398570473554, 2.402156027212455, 3.086618777384068, -7.524965782386951, -1.4655490392817523, -2.7988409599394495, -4.431945012854822]\n", "iter:353\n", "w: [3.020765703386106, 0.053266404098378864, 1.3017210202959342, -3.544706501936958, 1.3874890477807174, 4.2845182060071565, -0.151950223618771, -1.7938911882010122, 1.1827532102946672, -1.5149847182096992, 1.5833185527542928, -0.9893829922931647, 1.3947399323091028, 2.404464731846437, 3.09019193584369, -7.533365266811317, -1.467254839351417, -2.8017090882944657, -4.436966334413446]\n", "iter:354\n", "w: [3.024043761524842, 0.05315299296720067, 1.3031584935462468, -3.5485335815643215, 1.3891619314476205, 4.288753704483738, -0.15184026834310757, -1.795851478883049, 1.1840053663877566, -1.5165223168984148, 1.5847968072440664, -0.9905780044259379, 1.396136852789031, 2.406768987434004, 3.0937557617724534, -7.541745882561057, -1.4689569406935652, -2.8045708918655947, -4.441976933572555]\n", "iter:355\n", "w: [3.027314898043313, 0.05304020070844109, 1.3045926326521484, -3.552352580070706, 1.390831251854824, 4.2929805419346465, -0.15173090369752681, -1.7978071827997064, 1.185254907622401, -1.5180569591055177, 1.5862717083340057, -0.9917699066996377, 1.3975306336088404, 2.409068812325533, 3.0973103058168334, -7.550107719612786, -1.4706553602741188, -2.807426400938815, -4.4469768594716985]\n", "iter:356\n", "w: [3.030579144607262, 0.05292802290936329, 1.306023453978345, -3.55616353367868, 1.392497025097289, 4.297198756676236, -0.15162212555930604, -1.7997583224509932, 1.1865018456661307, -1.5195886569780244, 1.587743272463776, -0.9929587158712198, 1.3989212897771464, 2.411364224743336, 3.1008556182148292, -7.558450867274593, -1.4723501149370175, -2.810275645569336, -4.451966160887172]\n", "iter:357\n", "w: [3.0338365326472068, 0.05281645519549398, 1.3074509737666762, -3.5599664783521403, 1.3941592671557415, 4.301408386745578, -0.15151392984037598, -1.801704920168561, 1.1877461921032357, -1.5211174225786341, 1.589211515945526, -0.9941444485635657, 1.4003088361906229, 2.4136552427830016, 3.104391748800229, -7.566775414192751, -1.4740412214054106, -2.8131186555840118, -4.45694488623577]\n", "iter:358\n", "w: [3.0370870933608947, 0.05270549323024087, 1.3088752081373352, -3.5637614497988763, 1.3958179938977766, 4.305609469903386, -0.15140631248699168, -1.8036469981173728, 1.1889879584355911, -1.5226432678866102, 1.5906764549652401, -0.9953271212668721, 1.4016932876351331, 2.4159418844147265, 3.107918747006822, -7.575081448358348, -1.4757286962828344, -2.815955460583723, -4.461913083578505]\n", "iter:359\n", "w: [3.040330857715725, 0.05259513271451426, 1.3102961730900744, -3.567548483473105, 1.397473221078949, 4.309802043636895, -0.1512992694794058, -1.8055845782973505, 1.1902271560834716, -1.5241662047986477, 1.5921381055840707, -0.9965067503400222, 1.4030746587868437, 2.4182241674846194, 3.1114366618725557, -7.583369057113832, -1.4774125560543758, -2.8187860899457267, -4.4668708006242674]\n", "iter:360\n", "w: [3.043567856451139, 0.052485369386352704, 1.3117138845053962, -3.5713276145779704, 1.3991249643438501, 4.313986145162707, -0.151192796831546, -1.807517682545004, 1.1914637963863564, -1.5256862451297306, 1.5935964837396541, -0.9976833520119402, 1.404452964213327, 2.4205021097159927, 3.114945542043644, -7.591638327159478, -1.4790928170878217, -2.8216105728259775, -4.471818084733441]\n", "iter:361\n", "w: [3.046798120080976, 0.05237619902055238, 1.3131283581457285, -3.5750988780680135, 1.4007732392271732, 4.318161811429587, -0.1510868905906948, -1.809446332535038, 1.192697890603723, -1.5272034006139745, 1.5950516052474097, -0.9988569423829291, 1.4058282183746484, 2.4227757287106324, 3.11844543577862, -7.599889344559769, -1.480769495634794, -2.824428938161416, -4.476754982921468]\n", "iter:362\n", "w: [3.050021678895804, 0.05226761742830139, 1.3145396096565878, -3.5788623086516114, 1.4024180611547639, 4.322329079121241, -0.15098154683717327, -1.8113705497819437, 1.193929449915833, -1.5287176829054596, 1.59650348580182, -1.0000275374259913, 1.4072004356244399, 2.425045041950052, 3.1219363909523423, -7.608122194749706, -1.4824426078318713, -2.827241214672227, -4.481681541862377]\n", "iter:363\n", "w: [3.0532385629652117, 0.05215962045681732, 1.3159476545677267, -3.5826179407933854, 1.4040594454446593, 4.326487984659032, -0.15087676168402736, -1.813290355641566, 1.1951584854245045, -1.5302291035790514, 1.5979521409776953, -1.0011951529881336, 1.4085696302109607, 2.4273100667967284, 3.1254184550599464, -7.616336962541029, -1.4841121697016961, -2.830047430864073, -4.48659780789225]\n", "iter:364\n", "w: [3.0564488021400766, 0.05205220398898889, 1.3173525082942683, -3.586365808716579, 1.4056974073081154, 4.33063856420468, -0.15077253127671772, -1.8152057713126557, 1.196385008153879, -1.531737674131209, 1.599397586231421, -1.0023598047916553, 1.4099358162781448, 2.429570820495319, 3.1288916752207507, -7.62453373212837, -1.485778197154071, -2.832847615030291, -4.491503827012652]\n", "iter:365\n", "w: [3.0596524260548006, 0.051945363943021856, 1.3187541861378267, -3.5901059464054055, 1.407331961850621, 4.334780853662915, -0.15066885179281242, -1.8171168178384016, 1.1976090290511756, -1.5332434059807833, 1.6008398369021886, -1.0035215084354205, 1.411299007866635, 2.431827320173864, 3.1323560981821137, -7.632712587095328, -1.4874407059870383, -2.835641795254068, -4.49639964489402]\n", "iter:366\n", "w: [3.062849464129515, 0.05183909627208839, 1.3201527032876146, -3.5938383876073687, 1.4089631240728995, 4.338914888684097, -0.1505657194416831, -1.8190235161079444, 1.1988305589874384, -1.5347463104698045, 1.6022789082132116, -1.0046802793961152, 1.4126592189148044, 2.434079582844973, 3.13581177032324, -7.640873610420465, -1.4890997118879499, -2.8384299994105837, -4.501285306879002]\n", "iter:367\n", "w: [3.066039945572262, 0.05173339696398058, 1.3215480748215371, -3.5975631658355525, 1.4105909088719, 4.343040704666802, -0.15046313046420348, -1.8209258868578728, 1.2000496087582717, -1.5362463988642565, 1.6037148152729241, -1.0058361330294878, 1.4140164632597652, 2.4363276254069888, 3.139258737658944, -7.649016884483239, -1.4907552304345224, -2.8412122551691286, -4.506160857985752]\n", "iter:368\n", "w: [3.0692238993811385, 0.05162826204076755, 1.3229403157072726, -3.6012803143708827, 1.4122153310417753, 4.347158336760372, -0.15036108113245175, -1.8228239506737012, 1.2012661890845693, -1.5377436823548436, 1.605147573076164, -1.0069890845715765, 1.4153707546383651, 2.438571464645145, 3.142697045843362, -7.657142491069847, -1.4924072770958803, -2.843988589995192, -4.511026342911187]\n", "iter:369\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "w: [3.072401354346423, 0.05152368755845623, 1.324329440803341, -3.604989866264362, 1.41383640527485, 4.351267819867435, -0.15025956774941518, -1.82471772799133, 1.2024803106132318, -1.5392381720577446, 1.6065771965053417, -1.0081391491399183, 1.4167221066881721, 2.440811117232699, 3.1461267401736226, -7.665250511379017, -1.4940558672335864, -2.8467590311525273, -4.515881806034199]\n", "iter:370\n", "w: [3.0755723390526657, 0.051419669606656095, 1.3257154648601595, -3.6086918543392765, 1.4154541461625751, 4.355369188646386, -0.1501585866486983, -1.8266072390984893, 1.203691983917878, -1.5407298790153576, 1.6080037003315935, -1.0092863417347469, 1.418070532948445, 2.443046599732054, 3.1495478655934694, -7.673341026027711, -1.4957010161026607, -2.8495236057051847, -4.520727291418829]\n", "iter:371\n", "w: [3.0787368818807574, 0.051316204308247404, 1.3270984025210846, -3.612386311193376, 1.417068568196473, 4.35946247751384, -0.1500581341942337, -1.8284925041361642, 1.204901219499545, -1.542218814197034, 1.6094270992159192, -1.010430677240173, 1.419416046861095, 2.445277928595865, 3.152960466696842, -7.681414115056775, -1.4973427388525864, -2.852282340519525, -4.525562842817394]\n", "iter:372\n", "w: [3.0818950110099688, 0.05121328781905305, 1.3284782683234437, -3.6160732692010256, 1.4186796857690709, 4.363547720647049, -0.14995820677999572, -1.8303735431000037, 1.2061080277873817, -1.5437049884998038, 1.6108474077103059, -1.011572170425352, 1.4207586617716332, 2.4475051201681284, 3.156364587731408, -7.689469857936506, -1.498981050528304, -2.855035262266202, -4.530388503673584]\n", "iter:373\n", "w: [3.085046754419964, 0.05111091632751405, 1.3298550766995527, -3.6197527605153343, 1.4202875131748232, 4.367624951986294, -0.14985880082971728, -1.8322503758417137, 1.2073124191393327, -1.5451884127490885, 1.6122646402588372, -1.0127108359456363, 1.4220983909301081, 2.44972819068526, 3.1597602726020586, -7.697508333572161, -1.500615966071195, -2.8577823974221244, -4.535204317125508]\n", "iter:374\n", "w: [3.088192139892789, 0.051009086054368546, 1.331228841977723, -3.623424817070255, 1.4218920646110234, 4.371694205237238, -0.14975991279660936, -1.834123022070433, 1.2085144038428144, -1.546669097699408, 1.6136788111987879, -1.0138466883437152, 1.4234352474920304, 2.451947156277154, 3.1631475648743548, -7.705529620309399, -1.5022475003200522, -2.860523772272388, -4.540010326008712]\n", "iter:375\n", "w: [3.091331195014836, 0.050907793252334174, 1.3325995783832556, -3.62708947058266, 1.4234933541787045, 4.37575551387325, -0.1496615391630831, -1.8359915013540924, 1.2097139921153823, -1.548147054035075, 1.6150899347617047, -1.0149797420507387, 1.4247692445192879, 2.454162032968232, 3.1665265077779368, -7.71353379593965, -1.503875668012039, -2.863259412912188, -4.544806572859151]\n", "iter:376\n", "w: [3.0944639471787774, 0.05080703420579421, 1.3339673000394245, -3.6307467525543924, 1.4250913958835305, 4.379808911137706, -0.14956367644047544, -1.8378558331207597, 1.2109111941053916, -1.5496222923708818, 1.616498025074474, -1.01611001138743, 1.4261003949810478, 2.4563728366784745, 3.1698971442098887, -7.72152093770543, -1.5055004837836394, -2.865989345248706, -4.549593099916125]\n", "iter:377\n", "w: [3.0975904235854816, 0.05070680523048668, 1.335332020968448, -3.63439669427429, 1.4266862036366763, 4.383854430046253, -0.14946632116877664, -1.8397160366599667, 1.212106019892648, -1.551094823252777, 1.6179030961603749, -1.0172375105651825, 1.42742871175465, 2.4585795832244415, 3.1732595167380633, -7.729491122305583, -1.5071219621715937, -2.868713595002969, -4.554369949125186]\n", "iter:378\n", "w: [3.100710651245901, 0.05060710267319735, 1.3366937550924485, -3.6380393268201865, 1.4282777912556974, 4.387892103389043, -0.14936946991636124, -1.8415721311240227, 1.2132984794890525, -1.5525646571585343, 1.619305161940119, -1.0183622536871457, 1.4287542076264879, 2.4607822883202783, 3.176613667604368, -7.737444425900469, -1.5087401176138258, -2.871432187711694, -4.5591371621409955]\n", "iter:379\n", "w: [3.1038246569829364, 0.0505079229114557, 1.3380525162344012, -3.6416746810608913, 1.4298661724653903, 4.391921963732943, -0.1492731192797213, -1.843424135529311, 1.2144885828392362, -1.554031804498411, 1.6207042362328787, -1.0194842547492964, 1.4300768952928793, 2.4629809675787064, 3.179959638728009, -7.745380924117081, -1.5103549644503584, -2.8741451487290997, -4.563894780330156]\n", "iter:380\n", "w: [3.1069324674332783, 0.05040926235323422, 1.3394083181190728, -3.6453027876581396, 1.4314513608986423, 4.395944043423715, -0.14917726588320268, -1.845272068757572, 1.2156763398211907, -1.555496275615799, 1.6221003327572998, -1.0206035276414973, 1.431396787360927, 2.465175636512005, 3.183297471708697, -7.7533006920541085, -1.5119665169242187, -2.8768525032287022, -4.568642844774003]\n", "iter:381\n", "w: [3.1100341090492254, 0.05031111743665118, 1.3407611743739474, -3.648923677068525, 1.4330333700972724, 4.399958374588169, -0.14908190637874355, -1.84711594955717, 1.2168617602468874, -1.5569580807878671, 1.6234934651325035, -1.021720086148544, 1.4327138963493693, 2.4673663105329764, 3.186627207829815, -7.761203804286938, -1.513574789182332, -2.8795542762050865, -4.573381396271363]\n", "iter:382\n", "w: [3.113129608100479, 0.05021348462967661, 1.3421110985301432, -3.6525373795454055, 1.4346122135128616, 4.403964989136285, -0.148987037445616, -1.8489557965443457, 1.2180448538628925, -1.5584172302261943, 1.624883646879075, -1.0228339439511978, 1.4340282346894195, 2.4695530049558982, 3.189948888061546, -7.769090334872592, -1.5151797952764061, -2.8822504924756562, -4.578110475341275]\n", "iter:383\n", "w: [3.1162189906759172, 0.050116360429841, 1.3434581040233182, -3.6561439251407872, 1.4361879045075743, 4.407963918763311, -0.14889265579017014, -1.8507916282044548, 1.2192256303509725, -1.5598737340773952, 1.6262708914200399, -1.0239451146272078, 1.4353398147255965, 2.471735734997468, 3.1932625530639664, -7.776960357354616, -1.5167815491638057, -2.8849411766823603, -4.582830122225689]\n", "iter:384\n", "w: [3.1193022826853447, 0.0500197413639479, 1.3448022041945655, -3.6597433437071887, 1.4377604563549695, 4.411955194951837, -0.14879875814558058, -1.8526234628931912, 1.220404099328694, -1.5613276024237375, 1.6276552120818284, -1.0250536116523197, 1.4366486487165442, 2.4739145157777296, 3.1965682431901006, -7.7848139447679, -1.5183800647084158, -2.8876263532934003, -4.587540376892112]\n", "iter:385\n", "w: [3.122379509861221, 0.049923623987789184, 1.3461434122912972, -3.663335664899479, 1.439329882240804, 4.415938848973834, -0.14870534127159576, -1.8544513188377978, 1.2215802703500154, -1.5627788452837512, 1.6290366220952275, -1.026159448401273, 1.4379547488358426, 2.4760893623209914, 3.1998659984889377, -7.792651169643451, -1.5199753556814957, -2.8903060466049153, -4.592241279036239]\n", "iter:386\n", "w: [3.1254506977603684, 0.049828004885863246, 1.3474817414681197, -3.666920918176697, 1.440896195263826, 4.419914911892681, -0.14861240195428962, -1.8562752141382612, 1.222754152905872, -1.5642274726128298, 1.6304151345963205, -1.0272626381487868, 1.4392581271728087, 2.4782602895567307, 3.203155858708415, -7.8004721040130995, -1.5215674357625233, -2.8929802807426475, -4.596932868084543]\n", "iter:387\n", "w: [3.1285158717656594, 0.04973288067109663, 1.3488172047876987, -3.6704991328038488, 1.4424594084365585, 4.423883414565151, -0.14851993700581578, -1.8580951667684948, 1.223925756424755, -1.5656734943038242, 1.6317907626274164, -1.028363194070533, 1.4405587957332875, 2.4804273123204883, 3.2064378632983623, -7.808276819414166, -1.5231563185400296, -2.895649079663586, -4.601615183196835]\n", "iter:388\n", "w: [3.13157505708768, 0.04963824798456862, 1.3501498152216134, -3.6740703378536814, 1.444019534686076, 4.427844387643385, -0.14842794326416434, -1.8599111945775066, 1.2250950902732816, -1.5671169201876285, 1.6331635191379672, -1.0294611292440987, 1.4418567664404343, 2.4825904453547487, 3.2097120514134176, -7.8160653868940555, -1.5247420175124236, -2.8983124671575915, -4.6062882632687945]\n", "iter:389\n", "w: [3.1346282787663764, 0.04954410349523859, 1.3514795856512012, -3.6776345622084414, 1.4455765868547705, 4.431797861576833, -0.14833641759292132, -1.861723315290555, 1.2262621637567612, -1.568557760033759, 1.6345334169854728, -1.0305564566499377, 1.4431520511354878, 2.484749703309812, 3.212978461915901, -7.823837877014813, -1.5263245460888075, -2.9009704668490004, -4.610952146934463]\n", "iter:390\n", "w: [3.1376755616726784, 0.04945044389967642, 1.3528065288683937, -3.681191834561606, 1.44713057770111, 4.435743866614177, -0.14824535688102994, -1.8635315465102908, 1.2274269861197529, -1.5699960235509247, 1.6359004689363768, -1.0316491891723087, 1.4444446615785338, 2.486905100744654, 3.216237133378659, -7.831594359857617, -1.527903917589784, -2.90362310219821, -4.615606872568718]\n", "iter:391\n", "w: [3.1407169305101053, 0.04935726592179569, 1.3541306575765417, -3.6847421834196, 1.4486815199003882, 4.439682432805226, -0.14815475804255496, -1.8653359057178862, 1.228589566546617, -1.571431720387592, 1.6372646876669492, -1.032739339600206, 1.4457346094492598, 2.4890566521277737, 3.219488104087873, -7.8393349050272265, -1.529480145248252, -2.9062703965032446, -4.620252478289705]\n", "iter:392\n", "w: [3.14375240981635, 0.04926456631259023, 1.3554519843912327, -3.6882856371034887, 1.4502294260454651, 4.443613590002785, -0.14806461801644893, -1.8671364102741508, 1.2297499141620614, -1.572864860132541, 1.63862608576416, -1.0338269206282753, 1.4470219063477023, 2.491204371838031, 3.2227314120458384, -7.847059581656369, -1.5310532422101966, -2.9089123729013027, -4.62488900196125]\n", "iter:393\n", "w: [3.146782023964845, 0.04917234184987268, 1.356770521841097, -3.6918222237506537, 1.4517743086475008, 4.447537367864514, -0.14797493376632126, -1.868933077420636, 1.2309080380316786, -1.5742954523154153, 1.6399846757265408, -1.0349119448577218, 1.4483065637949828, 2.4933482741654744, 3.2259670949737047, -7.854768458410089, -1.5326232215354667, -2.911549054370284, -4.629516481195236]\n", "iter:394\n", "w: [3.149805797166312, 0.04908058933801668, 1.3580862823686062, -3.6953519713164478, 1.4533161801366792, 4.451453795854744, -0.1478857022802092, -1.8707259242807257, 1.2320639471624808, -1.5757235064072652, 1.6413404699650371, -1.0359944247972066, 1.4495885932340375, 2.49548837331216, 3.229195190314191, -7.862461603490036, -1.5341900961985462, -2.9141804637303013, -4.6341349533539535]\n", "iter:395\n", "w: [3.152823753470284, 0.04898930560770152, 1.3593992783308622, -3.698874907575831, 1.4548550528629256, 4.4553629032462885, -0.14779692057035132, -1.8725149678607145, 1.2332176505034247, -1.5771490318210832, 1.6426934808038494, -1.0370743728637326, 1.4508680060303383, 2.497624683392953, 3.2324157352342624, -7.8701390846387085, -1.5357538790893157, -2.9168066236451695, -4.638744455552425]\n", "iter:396\n", "w: [3.155835916766622, 0.048898487515659284, 1.3607095220003773, -3.7023910601249894, 1.4563909390966145, 4.459264719122225, -0.1477085856729634, -1.8743002250508753, 1.234369156945933, -1.5785720379123342, 1.644043720481264, -1.03815180138352, 1.4521448134726034, 2.4997572184363297, 3.235628766627781, -7.877800969143648, -1.537314583013806, -2.919427556623882, -4.6433450246606975]\n", "iter:397\n", "w: [3.1588423107870005, 0.04880813194442506, 1.3620170255658457, -3.705900456382933, 1.4579238510292711, 4.463159272377655, -0.14762069464801653, -1.8760817126265141, 1.2355184753244086, -1.5799925339794774, 1.6453912011504745, -1.039226722592872, 1.4534190267735028, 2.501885992385161, 3.2388343211181225, -7.885447323841585, -1.5388722206949426, -2.922043285022065, -4.64793669730611]\n", "iter:398\n", "w: [3.161842959106385, 0.04871823580208984, 1.363321801132906, -3.7094031235930767, 1.4594538007742637, 4.467046591721441, -0.14753324457901734, -1.8778594472490118, 1.2366656144167436, -1.5814105292644833, 1.6467359348803934, -1.0402991486390303, 1.4546906570703533, 2.5040110190974896, 3.2420324350607634, -7.89307821512254, -1.540426804773282, -2.9246538310434187, -4.652519509875535]\n", "iter:399\n", "w: [3.164837885144485, 0.04862879602205571, 1.3646238607248955, -3.712899088824803, 1.4609808003674896, 4.470926705677927, -0.14744623257279046, -1.8796334454668566, 1.2378105829448218, -1.5828260329533432, 1.6480779336564526, -1.0413690915810208, 1.4559597154258066, 2.506132312347298, 3.2452231445458377, -7.900693708933874, -1.5419783478077411, -2.9272592167411364, -4.657093498517591]\n", "iter:400\n", "w: [3.1678271121671946, 0.04853980956279417, 1.3659232162835957, -3.7163883789750063, 1.4625048617680523, 4.474799642588639, -0.14735965575926332, -1.8814037237166628, 1.2389533895750162, -1.584239054176573, 1.649417209381396, -1.0424365633904893, 1.4572262128285292, 2.508249885825265, 3.2484064854006656, -7.908293870784291, -1.543526862276316, -2.9298594640193123, -4.661658699144829]\n", "iter:401\n", "w: [3.170810663288012, 0.04845127340760654, 1.3672198796699697, -3.7198710207696215, 1.4640259968589324, 4.4786654306139555, -0.14727351129125318, -1.8831702983241798, 1.2400940429186806, -1.5856496020097106, 1.6507537738760607, -1.0435015759525277, 1.4584901601938742, 2.510363753139516, 3.2515824931922506, -7.915878765747807, -1.5450723605767953, -2.9324545946343274, -4.666215147435895]\n", "iter:402\n", "w: [3.173788561469444, 0.04836318456438727, 1.3685138626648918, -3.723347040765132, 1.4655442174476503, 4.482524097734769, -0.14718779634425597, -1.8849331855052884, 1.241232551532635, -1.5870576854738077, 1.6520876388801518, -1.0445641410664912, 1.4597515683645472, 2.51247392781636, 3.2547512032297488, -7.923448458467656, -1.5466148550274634, -2.9350446301962227, -4.670762878837665]\n", "iter:403\n", "w: [3.1767608295243948, 0.04827554006538982, 1.3698051769698683, -3.7268164653500646, 1.4670595352669218, 4.486375671754121, -0.14710250811623782, -1.8866924013669877, 1.242368923919647, -1.5884633135359143, 1.6534188160530043, -1.0456242704468053, 1.4610104481112616, 2.514580423301022, 3.2579126505669103, -7.93100301316017, -1.5481543578677988, -2.937629592170055, -4.675301928567358]\n", "iter:404\n", "w: [3.179727490117536, 0.04818833696699468, 1.37109383420775, -3.730279320746464, 1.468571961975307, 4.490220180298821, -0.147017643827428, -1.8884479619083694, 1.2435031685289057, -1.589866495109559, 1.6547473169743392, -1.0466819757237635, 1.4622668101333893, 2.516683252958362, 3.2610668700044916, -7.938542493618603, -1.5496908812591608, -2.9402095018772365, -4.679832331614621]\n", "iter:405\n", "w: [3.1826885657666626, 0.04810157234948024, 1.372379845923439, -3.7337356330113525, 1.4700815091578516, 4.494057650821042, -0.1469332007201146, -1.8901998830215827, 1.2446352937564926, -1.591267239055222, 1.6560731531450084, -1.0477372684443174, 1.4635206650596024, 2.518782430073589, 3.2642138960926412, -7.946066963216913, -1.5512244372854724, -2.94278438049686, -4.684354122743589]\n", "iter:406\n", "w: [3.1856440788440303, 0.04801524331679598, 1.3736632235845845, -3.737185428038174, 1.471588188326721, 4.497888110599902, -0.14684917605844208, -1.8919481804927876, 1.245765307945845, -1.592665554180804, 1.6573963359877317, -1.0487901600728555, 1.464772023448508, 2.5208779678529654, 3.267353763133256, -7.953576484913512, -1.5527550379538937, -2.9453542490670053, -4.688867336494925]\n", "iter:407\n", "w: [3.188594051577681, 0.04792934699633795, 1.3749439785822726, -3.74062873155822, 1.4730920109218284, 4.501711586743024, -0.14676556712821046, -1.8936928700030982, 1.246893219388216, -1.5940614492420873, 1.6587168768478258, -1.0498406619919756, 1.4660208957892764, 2.522969879424501, 3.2704865051823138, -7.9610711212549585, -1.5542826951954882, -2.9479191284860353, -4.693372007187834]\n", "iter:408\n", "w: [3.1915385060527486, 0.04784388053872685, 1.3762221222317097, -3.74406556914204, 1.4745929883114555, 4.505528106188076, -0.14668237123667727, -1.8954339671295155, 1.2480190363231283, -1.5954549329431935, 1.6600347869939223, -1.0508887855032458, 1.4672672925022618, 2.525058177838641, 3.273612156052176, -7.968550934379623, -1.5558074208658819, -2.9504790395138736, -4.697868168922051]\n", "iter:409\n", "w: [3.1944774642127527, 0.04775884111758838, 1.3774976657728957, -3.7474959662008387, 1.476091131792866, 4.509337695704298, -0.14659958571236079, -1.89717148734585, 1.249142766938823, -1.5968460139370344, 1.6613500776186807, -1.0519345418279593, 1.4685112239396156, 2.5271428760689445, 3.2767307493138658, -7.976015986021303, -1.5573292267459156, -2.9530340027732667, -4.702355855579811]\n", "iter:410\n", "w: [3.197410947860875, 0.04767422592933592, 1.3787706203712922, -3.750919947987853, 1.4775864525929137, 4.513140381894004, -0.1465172079048459, -1.8989054460236345, 1.2502644193727042, -1.5982347008257578, 1.66266275983949, -1.0529779421078793, 1.4697527003858937, 2.529223987012755, 3.2798423182993206, -7.983466337512802, -1.5588481245422896, -2.9555840387510357, -4.706835100827795]\n", "iter:411\n", "w: [3.200338978661224, 0.04759003219295546, 1.3800409971184822, -3.7543375395997196, 1.479078961868643, 4.516936191194074, -0.14643523518459134, -1.9006358584330254, 1.2513840017117774, -1.5996210021611885, 1.6639728446991642, -1.0540189974059755, 1.4709917320586563, 2.531301523491864, 3.282946896103618, -7.990902049789468, -1.5603641258882015, -2.9581291677993087, -4.711305938119047]\n", "iter:412\n", "w: [3.2032615781400797, 0.04750625714979305, 1.3813088070328219, -3.75774876597782, 1.4805686707078838, 4.520725149877421, -0.14635366494273935, -1.9023627397436966, 1.2525015219930848, -1.6010049264452635, 1.665280343166628, -1.0550577187071528, 1.4722283291090614, 2.5333754982531653, 3.2860445155871765, -7.998323183392693, -1.5618772423439775, -2.9606694101367417, -4.71576840069488]\n", "iter:413\n", "w: [3.2061787676871294, 0.04742289806334414, 1.3825740610600878, -3.76115365190962, 1.4820555901298387, 4.524507284054448, -0.1462724945909269, -1.9040861050257223, 1.253616988204134, -1.6023864821304623, 1.6665852661375962, -1.0560941169189717, 1.4734625016224518, 2.5354459239693012, 3.289135209377933, -8.005729798473373, -1.5633874853976961, -2.9632047858497255, -4.720222521586752]\n", "iter:414\n", "w: [3.2090905685566864, 0.047339952219045495, 1.3838367700741145, -3.7645522220299843, 1.4835397310856655, 4.528282619674484, -0.1461917215610991, -1.9058059692504492, 1.2547304082833226, -1.603765677620232, 1.667887624435243, -1.05712820287236, 1.4746942596189354, 2.537512813239304, 3.2922190098734934, -8.013121954795333, -1.564894866465806, -2.9657353148935774, -4.7246683336181245]\n", "iter:415\n", "w: [3.211997001868895, 0.047257416924069165, 1.3850969448774262, -3.767944500822486, 1.4850211044590522, 4.532051182527201, -0.14611134330532433, -1.907522347291362, 1.2558417901203596, -1.6051425212694073, 1.6691874288108666, -1.058159987322318, 1.4759236130539586, 2.539576178589226, 3.29529594924326, -8.020499711738708, -1.5663993968937364, -2.9682610170937216, -4.729105869406296]\n", "iter:416\n", "w: [3.214898088610922, 0.04717528950711858, 1.3863545962018615, -3.7713305126206955, 1.4864997210667872, 4.5358129982440225, -0.14603135729561129, -1.9092352539249369, 1.256951141556679, -1.6065170213846263, 1.6704846899445436, -1.0591894809486135, 1.477150571818874, 2.541636032472766, 3.2983660594305357, -8.027863128303293, -1.5679010879565018, -2.9707819121468524, -4.73353516136422]\n", "iter:417\n", "w: [3.2177938496381335, 0.04709356731822691, 1.3876097347091922, -3.7747102816094564, 1.4879755916593214, 4.539568092299506, -0.14595176102372778, -1.9109447038314868, 1.2580584703858522, -1.6078891862247406, 1.6717794184457764, -1.0602166943564713, 1.4783751457415024, 2.5436923872718844, 3.3014293721546037, -8.035212263111854, -1.5693999508592993, -2.9732980196220886, -4.737956241702295]\n", "iter:418\n", "w: [3.220684305675261, 0.04701224772855743, 1.3888623709917338, -3.7780838318261503, 1.4894487269213266, 4.543316490012718, -0.1458725520010215, -1.9126507115959983, 1.2591637843539938, -1.6092590240012214, 1.6730716248541337, -1.0612416380772536, 1.479597344586687, 2.5457452552974154, 3.304485918912784, -8.042547174413402, -1.5708959967380995, -2.975809358962111, -4.742369142430145]\n", "iter:419\n", "w: [3.2235694773175503, 0.046931328130206236, 1.3901125155729506, -3.7814511871619447, 1.4909191374722466, 4.54705821654859, -0.1457937277582426, -1.914353291708958, 1.260267091160164, -1.6106265428785598, 1.6743613196398839, -1.062264322569134, 1.4808171780568429, 2.5477946487896688, 3.307535730982467, -8.049867920086433, -1.572389236660233, -2.9783159494842892, -4.746773895358367]\n", "iter:420\n", "w: [3.226449385031901, 0.046850805936006795, 1.3913601789080539, -3.7848123713630293, 1.4923868338668427, 4.550793296919256, -0.14571528584536772, -1.9160524585671717, 1.2613683984567665, -1.6119917509746635, 1.67564851320462, -1.063284758217762, 1.4820346557924997, 2.549840579919026, 3.3105788394231244, -8.057174557642131, -1.5738796816249674, -2.9808178103817955, -4.751170532100271]\n", "iter:421\n", "w: [3.229324049157991, 0.04677067857933647, 1.3926053713845943, -3.7881674080318377, 1.4938518265957337, 4.554521755985378, -0.1456372238314264, -1.917748226474573, 1.2624677138499405, -1.6133546563612489, 1.6769332158818784, -1.0643029553369232, 1.4832497873728387, 2.551883060786529, 3.313615275078297, -8.06446714422754, -1.5753673425640815, -2.9833149607247047, -4.75555908407359]\n", "iter:422\n", "w: [3.2321934899093883, 0.04669094351392567, 1.3938481033230465, -3.7915163206282556, 1.4953141260859302, 4.558243618457457, -0.14555953930432866, -1.9194406096430252, 1.2635650448999516, -1.6147152670642277, 1.6782154379377503, -1.0653189241691887, 1.4844625823162247, 2.5539221034244637, 3.316645068577563, -8.071745736628696, -1.5768522303424308, -2.9858074194610835, -4.75993958250218]\n", "iter:423\n", "w: [3.2350577273746506, 0.046611598213668254, 1.3950883849773894, -3.7948591324708185, 1.4967737427013628, 4.56195890889713, -0.14548222987069462, -1.9211296221931127, 1.264660399121576, -1.6160735910640904, 1.6794951895714867, -1.0663326748865605, 1.4856730500807307, 2.555957719796936, 3.319668250338481, -8.079010391273725, -1.5783343557585088, -2.9882952054180674, -4.764312058417693]\n", "iter:424\n", "w: [3.2379167815184133, 0.04653264017243422, 1.3963262265356784, -3.7981958667378937, 1.4982306867434054, 4.565667651718445, -0.14540529315568565, -1.9228152781549255, 1.2657537839844808, -1.6174296362962843, 1.680772480916095, -1.0673442175911074, 1.4868812000646574, 2.5579899218004387, 3.322684850568514, -8.086261164235918, -1.579813729545001, -2.990778337302923, -4.768676542661237]\n", "iter:425\n", "w: [3.2407706721824647, 0.046454066903884664, 1.397561638120613, -3.80152654646885, 1.4996849684513927, 4.569369871189132, -0.1453287268028374, -1.9244975914688345, 1.2668452069136014, -1.6187834106515884, 1.6820473220389314, -1.0683535623155955, 1.4880870416070482, 2.560018721264416, 3.3256948992669297, -8.093498111236759, -1.5812903623693342, -2.993256833704103, -4.7730330658850155]\n", "iter:426\n", "w: [3.243619419086808, 0.046375875941288294, 1.3987946297900975, -3.8048511945652175, 1.5011365980031321, 4.5730655914318525, -0.14525252847389447, -1.926176575986259, 1.267934675289514, -1.6201349219764822, 1.6833197229422832, -1.0693607190241106, 1.4892905839881974, 2.5620441299518193, 3.328698426226683, -8.100721287648929, -1.5827642648342195, -2.9957307130922834, -4.7773816585539475]\n", "iter:427\n", "w: [3.2464630418307143, 0.046298064837339815, 1.4000252115377956, -3.808169833791831, 1.5025855855154115, 4.576754836425436, -0.1451766958486465, -1.9278522454704263, 1.2690221964488044, -1.6214841780735127, 1.6845896935639484, -1.0703656976126747, 1.4904918364301525, 2.564066159559657, 3.3316954610362757, -8.107930748499278, -1.5842354474781895, -2.998199993821397, -4.781722350947272]\n", "iter:428\n", "w: [3.2493015598937594, 0.046220631163980855, 1.4012533932936801, -3.8114824867779644, 1.5040319410444993, 4.580437630006105, -0.14510122662476627, -1.9295246135971218, 1.2701077776844323, -1.6228311867016556, 1.6858572437778054, -1.0713685079098558, 1.4916908080972129, 2.566084821719539, 3.3346860330815953, -8.115126548471759, -1.5857039207761299, -3.0006646941296466, -4.786055173160135]\n", "iter:429\n", "w: [3.252134992636854, 0.04614357251222175, 1.4024791849245766, -3.8147891760184516, 1.5054756745866418, 4.584113995868686, -0.14502611851764924, -1.9311936939554333, 1.2711914262460926, -1.6241759555766722, 1.687122383394379, -1.0723691596773697, 1.4928875080964215, 2.5681001279982127, 3.3376701715477353, -8.122308741910334, -1.5871696951398058, -3.0031248321405157, -4.790380155105154]\n", "iter:430\n", "w: [3.2549633593032596, 0.04606688649196621, 1.4037025962346996, -3.8180899238747967, 1.506916796078554, 4.587783957567804, -0.14495136926025456, -1.932859500048486, 1.2722731493405726, -1.625518492371464, 1.6883851221613972, -1.0733676626106765, 1.4940819454780518, 2.5701120898980965, 3.340647905420795, -8.129477382821856, -1.5886327809183827, -3.0055804258637595, -4.794697326513973]\n", "iter:431\n", "w: [3.2577866790195933, 0.04599057073183707, 1.404923636966186, -3.82138475257627, 1.5083553153979063, 4.591447538519068, -0.1448769766029479, -1.9345220452941705, 1.2733529541321047, -1.626858804716422, 1.6896454697643453, -1.0743640263395708, 1.49527412923609, 2.5721207188578044, 3.343619263489658, -8.136632524878907, -1.5900931883989406, -3.008031493196393, -4.799006716938794]\n", "iter:432\n", "w: [3.2606049707968228, 0.04591462287900418, 1.4061423167996208, -3.8246736842209947, 1.509791242363805, 4.59510476200024, -0.1448029383133457, -1.9361813430258625, 1.2744308477427164, -1.6281969001997723, 1.6909034358270105, -1.0753582604287646, 1.4964640683087116, 2.574126026252666, 3.3465842743477543, -8.143774221422618, -1.591550927806986, -3.010478051923662, -4.803308355753894]\n", "iter:433\n", "w: [3.263418253531248, 0.04583904059901398, 1.407358645354558, -3.8279567407770223, 1.5112245867372678, 4.59875565115239, -0.1447292521761611, -1.9378374064931359, 1.2755068372525766, -1.6295327863679179, 1.6921590299120226, -1.076350374378464, 1.4976517715787525, 2.5761280233952406, 3.3495429663948, -8.150902525465447, -1.5930060093069538, -3.012920119720006, -4.807602272157128]\n", "iter:434\n", "w: [3.266226546005476, 0.045763821575621276, 1.4085726321900354, -3.8312339440833947, 1.512655358221697, 4.602400228981047, -0.14465591599305117, -1.939490248862467, 1.2765809297003368, -1.6308664707257774, 1.693412261521388, -1.0773403776249402, 1.4988372478741763, 2.578126721535826, 3.3524953678385225, -8.158017489693945, -1.5944584430027091, -3.015357714150014, -4.811888495171411]\n", "iter:435\n", "w: [3.26902986688938, 0.045688963510621836, 1.4097842868050856, -3.8345053158511972, 1.5140835664633445, 4.606038518357322, -0.14458292758246624, -1.9411398832179334, 1.27765313208347, -1.6321979607371189, 1.6946631400970176, -1.0783282795410942, 1.500020505968535, 2.5801221318629595, 3.35544150669636, -8.165119166471474, -1.59590823893804, -3.0177908526693615, -4.816167053646188]\n", "iter:436\n", "w: [3.271828234741053, 0.04561446412368779, 1.4109936186392402, -3.8377708776645987, 1.5155092210517735, 4.609670542019036, -0.14451028477949987, -1.9427863225619038, 1.2787234513586072, -1.6335272638248908, 1.695911675021249, -1.0793140894370155, 1.5012015545814268, 2.5821142655039178, 3.3583814107971497, -8.172207607840907, -1.5973554070971472, -3.020219552625745, -4.8204379762588845]\n", "iter:437\n", "w: [3.2746216680077467, 0.04554032115220393, 1.4122006370730291, -3.841030650981883, 1.516932331520316, 4.613296322571822, -0.14443798543574093, -1.9444295798157212, 1.279791894441867, -1.6348543873715493, 1.6971578756173635, -1.0802978165605333, 1.502380402378947, 2.584103133525207, 3.361315107782795, -8.179282865527298, -1.5987999574051281, -3.022643831259803, -4.824701291516344]\n", "iter:438\n", "w: [3.2774101850268043, 0.04546653235110614, 1.413405351428475, -3.8442846571364675, 1.5183529073465245, 4.616915882490221, -0.14436602741912716, -1.9460696678203795, 1.2808584682091848, -1.6361793387193817, 1.6984017511500966, -1.0812794700977646, 1.5035570579741362, 2.586088746933049, 3.3642426251099122, -8.186344990940526, -1.6002418997284562, -3.0250637057060277, -4.82895702775625]\n", "iter:439\n", "w: [3.2801938040265783, 0.04539309549272145, 1.4146077709695821, -3.847532917337912, 1.5197709579526204, 4.620529244118767, -0.14429440861379977, -1.947706599337193, 1.2819231794966377, -1.6375021251708262, 1.6996433108261433, -1.0822590591736543, 1.5047315299274213, 2.5880711166738624, 3.367163990051464, -8.193394035177906, -1.6016812438754546, -3.0274791929936664, -4.83320521314853]\n", "iter:440\n", "w: [3.2829725431273444, 0.04532000836660943, 1.4158079049028205, -3.8507754526729188, 1.5211864927059366, 4.624136429673058, -0.1442231269199599, -1.9493403870484578, 1.2829860351007656, -1.6388227539887885, 1.7008825637946576, -1.0832365928525096, 1.505903826747055, 2.59005025363474, 3.370079229698372, -8.200430049026776, -1.6031179995967662, -3.0298903100476133, -4.8374458756967496]\n", "iter:441\n", "w: [3.285746420342199, 0.0452472687794058, 1.4170057623776051, -3.8540122841063185, 1.5225995209193566, 4.627737461240814, -0.14415218025372636, -1.9509710435581085, 1.284047041778889, -1.6401412323969553, 1.7021195191477476, -1.0842120801385302, 1.507073956889548, 2.5920261686439177, 3.3729883709611137, -8.20745308296706, -1.6045521765858177, -3.0322970736892927, -4.841679043239483]\n", "iter:442\n", "w: [3.288515453577952, 0.04517487455466696, 1.4182013524867698, -3.8572434324820493, 1.5240100518517483, 4.631332360782928, -0.14408156654699464, -1.952598581392366, 1.285106206249424, -1.641457567580104, 1.7033541859209638, -1.0851855299763307, 1.5082419287600972, 2.59399887247124, 3.3758914405713023, -8.214463187173795, -1.6059837844792801, -3.034699500637531, -4.84590474345168]\n", "iter:443\n", "w: [3.2912796606360084, 0.04510282353271668, 1.4193946842670366, -3.8604689185241248, 1.525418094708394, 4.634921150134502, -0.14401128374729744, -1.9542230130003801, 1.2861635351921947, -1.6427717666844084, 1.7045865730937826, -1.0861569512514595, 1.5094077507130115, 2.5959683758286225, 3.3787884650832476, -8.221460411519644, -1.6074128328575235, -3.03709760750942, -4.850123003846008]\n", "iter:444\n", "w: [3.29403905921324, 0.04503111357049421, 1.4205857666994801, -3.863688762837592, 1.5268236586414161, 4.638503851005871, -0.14394132981766675, -1.9558443507548646, 1.2872190352487396, -1.6440838368177428, 1.705816689590085, -1.087126352790911, 1.51057143105213, 2.5979346893705046, 3.381679470875503, -8.228444805577375, -1.6088393312450673, -3.039491410821172, -4.854333851774188]\n", "iter:445\n", "w: [3.2967936669028473, 0.04495974254140381, 1.4217746087099872, -3.866902985909479, 1.5282267527501985, 4.642080484983624, -0.143871702736497, -1.957462606952727, 1.2882727130226186, -1.6453937850499811, 1.70704454427863, -1.0880937433636324, 1.5117329780312387, 2.5998978236943033, 3.3845644841523956, -8.235416418622314, -1.6102632891110273, -3.0418809269889655, -4.858537314428309]\n", "iter:446\n", "w: [3.2995435011952146, 0.04488870833516592, 1.4229612191697119, -3.870111608109734, 1.5296273860818028, 4.645651073531601, -0.14380240049740975, -1.9590777938156905, 1.2893245750797142, -1.6467016184132943, 1.708270145973522, -1.0890591316810256, 1.5128923998544808, 2.601857789340857, 3.3874435309455366, -8.242375299634782, -1.6116847158695564, -3.0442661723297793, -4.862733418842137]\n", "iter:447\n", "w: [3.302288579478753, 0.04481800885766996, 1.4241456068955256, -3.8733146496921536, 1.5310255676313824, 4.649215637991892, -0.1437334211091197, -1.9606899234909103, 1.2903746279485304, -1.648007343902443, 1.7094935034346743, -1.090022526397444, 1.5140497046767636, 2.6038145967948676, 3.3903166371153204, -8.249321497302489, -1.6131036208802818, -3.0466471630622216, -4.866922191892403]\n", "iter:448\n", "w: [3.305028919040736, 0.044747642030828476, 1.4253277806504627, -3.8765121307953017, 1.5324213063425902, 4.652774199585818, -0.14366476259530192, -1.9622990080515843, 1.2914228781204893, -1.6493109684750684, 1.7107146253682672, -1.0909839361106826, 1.515204900604162, 2.6057682564853364, 3.393183828352406, -8.256255060022927, -1.6145200134487379, -3.049023915307348, -4.871103660300079]\n", "iter:449\n", "w: [3.3077645370681275, 0.04467760579243291, 1.4265077491441625, -3.879704071443421, 1.5338146111079838, 4.656326779414902, -0.14359642299446043, -1.9639050594975558, 1.2924693320502232, -1.6506124990519802, 1.7119335204272015, -1.0919433693624652, 1.516357995694316, 2.6077187787859963, 3.3960451301791807, -8.263176035905717, -1.6159339028267945, -3.051396445089471, -4.875277850631644]\n", "iter:450\n", "w: [3.3104954506483972, 0.04460789809601073, 1.4276855210333044, -3.8828904915473332, 1.5352054907694261, 4.659873398461834, -0.14352840035979827, -1.9655080897559118, 1.2935139961558648, -1.6519119425174396, 1.713150197211546, -1.0929008346389242, 1.5175089979568261, 2.60966617401574, 3.398900567951212, -8.270084472774945, -1.6173452982130812, -3.0537647683369613, -4.879444789300331]\n", "iter:451\n", "w: [3.313221676770332, 0.04453851691068423, 1.4288611049220419, -3.8860714109053296, 1.5365939541184819, 4.663414077591423, -0.14346069275908857, -1.9671081106815749, 1.2945568768193347, -1.6532093057194417, 1.7143646642689816, -1.0938563403710775, 1.518657915353643, 2.6116104524390407, 3.401750166858682, -8.276980418171469, -1.6187542087534068, -3.0561289008830417, -4.883604502567369]\n", "iter:452\n", "w: [3.3159432323248366, 0.04446946022103019, 1.4300345093624292, -3.889246849204056, 1.5379800098968104, 4.666948837551534, -0.14339329827454694, -1.9687051340578883, 1.2955979803866249, -1.6545045954699935, 1.7155769300952404, -1.0948098949352985, 1.5198047557994543, 2.613551624266372, 3.404593951927805, -8.283863919355204, -1.620160643541175, -3.0584888584665704, -4.887757016543204]\n", "iter:453\n", "w: [3.3186601341057234, 0.04440072602694214, 1.4312057428548453, -3.892416826019384, 1.5393636667965547, 4.670477698974027, -0.14332621500270545, -1.9702991715971958, 1.2966373131680804, -1.6557978185453897, 1.7167870031345391, -1.0957615066537822, 1.5209495271620668, 2.6154896996546224, 3.4074319480222353, -8.290735023307384, -1.6215646116177964, -3.0608446567328174, -4.89190235718871]\n", "iter:454\n", "w: [3.3213723988104995, 0.04433231234349269, 1.432374813848413, -3.8955813608172782, 1.5407449334607264, 4.674000682375677, -0.14325944105428737, -1.9718902349414162, 1.2976748814386778, -1.6570889816864844, 1.7179948917800092, -1.0967111837950072, 1.5220922372627852, 2.617424688707503, 3.410264179844456, -8.297593776732802, -1.622966121973095, -3.0631963112342344, -4.89604055031639]\n", "iter:455\n", "w: [3.3240800430411412, 0.044264217200798095, 1.4335417307414142, -3.898740472954652, 1.5421238184835866, 4.677517808159085, -0.14319297455408336, -1.9734783356626109, 1.2987106914383, -1.6583780915989623, 1.7192006043741217, -1.0976589345741905, 1.523232893876787, 2.619356601475955, 3.4130906719371534, -8.30444022606202, -1.6243651835457127, -3.0655438374312123, -4.9001716215915625]\n", "iter:456\n", "w: [3.3267830833048615, 0.04419643864388376, 1.4347065018817013, -3.9018941816802144, 1.5435003304110237, 4.6810290966135835, -0.14312681364082913, -1.9750634852635462, 1.2997447493720105, -1.6596651549536046, 1.7204041492091082, -1.0986047671537402, 1.524371504733493, 2.6212854479585492, 3.41591144868458, -8.311274417453573, -1.6257618052235077, -3.067887250692835, -4.904295596533536]\n", "iter:457\n", "w: [3.3294815360148697, 0.04412897473255175, 1.435869135567103, -3.9050425061353105, 1.5448744777409276, 4.6845345679161285, -0.14306095646708367, -1.976645695178251, 1.3007770614103222, -1.6609501783865532, 1.7216055345273762, -1.0995486896437021, 1.5255080775169343, 2.6232112381018844, 3.4187265343139, -8.318096396796129, -1.6271559958439503, -3.070226566297625, -4.90841250051677]\n", "iter:458\n", "w: [3.3321754174911247, 0.04406182354124874, 1.4370296400458273, -3.908185465354752, 1.5462462689235588, 4.688034242132186, -0.1429954011991092, -1.9782249767725673, 1.3018076336894662, -1.662233168499573, 1.7228047685219214, -1.1004907101022017, 1.526642619866117, 2.6251339818009796, 3.421535952896524, -8.32490620971065, -1.628547764194515, -3.0725617994342773, -4.912522358772025]\n", "iter:459\n", "w: [3.3348647439610772, 0.04399498315893605, 1.438188023516861, -3.9113230782676403, 1.5476157123619159, 4.691528139216608, -0.14293014601675244, -1.9798013413446964, 1.3028364723116552, -1.6635141318603104, 1.7240018593367352, -1.1014308365358818, 1.5277751393753798, 2.627053688899663, 3.4243397283494255, -8.331703901552508, -1.6299371190130671, -3.07489296520239, -4.916625196387503]\n", "iter:460\n", "w: [3.3375495315604082, 0.043928451688960826, 1.4393442941303627, -3.9144553636981794, 1.548982816412098, 4.695016279014495, -0.14286518911332632, -1.9813748001257392, 1.3038635833453467, -1.6647930750025488, 1.7251968150672072, -1.102369076900336, 1.5289056435947523, 2.628970369190958, 3.4271378844364495, -8.338489517413604, -1.631324068988248, -3.0772200786131876, -4.920721038309976]\n", "iter:461\n", "w: [3.3402297963337553, 0.043862227248928, 1.4404984599880557, -3.9175823403664856, 1.5503475893836647, 4.698498681262058, -0.1428005286954935, -1.982945364280231, 1.3048889728255022, -1.666070004426463, 1.726389643760525, -1.1033054391005377, 1.5300341400303061, 2.630884032417462, 3.4299304447696017, -8.345263102124447, -1.6327086227598537, -3.079543154590231, -4.924809909345894]\n", "iter:462\n", "w: [3.3429055542354362, 0.04379630797057443, 1.4416505291436132, -3.9207040268893834, 1.551710039539991, 4.701975365587466, -0.14273616298315084, -1.9845130449066717, 1.3059126467538444, -1.6673449265988698, 1.7275803534160687, -1.1042399309912634, 1.5311606361445047, 2.6327946882717255, 3.432717432810329, -8.352024700256216, -1.6340907889192127, -3.081862207970128, -4.928891834162497]\n", "iter:463\n", "w: [3.3455768211301615, 0.043730691999643544, 1.442800509603042, -3.923820441781197, 1.5530701750986207, 4.705446351511685, -0.14267209020931473, -1.9860778530380496, 1.3069346110991116, -1.668617847953477, 1.7287689519858014, -1.1051725603775135, 1.5322851393565486, 2.634702346396624, 3.435498871870785, -8.358774356122813, -1.6354705760095576, -3.0841772535032317, -4.932966837288904]\n", "iter:464\n", "w: [3.348243612793742, 0.04366537749576182, 1.4439484093250625, -3.9269316034545336, 1.5544280042316152, 4.708911658449309, -0.14260830862000815, -1.9876397996423614, 1.3079548717973095, -1.6698887748911284, 1.7299554473746557, -1.1061033350149279, 1.5334076570427175, 2.6366070163857294, 3.43827478511508, -8.36551211378288, -1.6368479925263952, -3.086488305854332, -4.937034943117195]\n", "iter:465\n", "w: [3.3509059449137877, 0.04360036263231582, 1.4450942362214827, -3.9300375302210577, 1.5557835350658993, 4.712371305709385, -0.14254481647414818, -1.9891988956231255, 1.308973434751961, -1.671157713780048, 1.7311398474409174, -1.1070322626101965, 1.5345281965367088, 2.6385087077836733, 3.4410451955605232, -8.372238017041811, -1.6382230469178716, -3.0887953796033423, -4.94109617590348]\n", "iter:466\n", "w: [3.3535638330904, 0.04353564559633136, 1.446237998157571, -3.93313824029226, 1.557136775683603, 4.715825312496228, -0.14248161204343493, -1.9907551518198923, 1.3099903058343532, -1.6724246709560802, 1.732322159996603, -1.1079593508214678, 1.5356467651299732, 2.640407430086511, 3.443810126078849, -8.378952109453722, -1.6395957475851337, -3.0910984892459763, -4.945150559768965]\n", "iter:467\n", "w: [3.3562172928368574, 0.043471224588353126, 1.4473797029524245, -3.936233751780216, 1.5584877341224013, 4.719273697910223, -0.1424186936122415, -1.992308579008747, 1.3110054908837818, -1.6736896527229292, 1.7335023928078352, -1.1088846072587506, 1.5367633700720467, 2.6423031927420797, 3.4465695993974306, -8.385654434323433, -1.6409661028826885, -3.093397649194421, -4.949198118700995]\n", "iter:468\n", "w: [3.358866339580293, 0.04340709782232594, 1.448519358379333, -3.9393240826983407, 1.5598364183758497, 4.722716480948631, -0.14235605947750518, -1.9938591879028098, 1.312018995707794, -1.6749526653523952, 1.7346805535952137, -1.1098080394843142, 1.537878018570879, 2.6441960051503517, 3.449323638100482, -8.3923450347084, -1.6423341211187577, -3.0956928737780003, -4.953238876554093]\n", "iter:469\n", "w: [3.3615109886623675, 0.04334326352547703, 1.44965697216614, -3.942409250962132, 1.561182836393717, 4.726153680506371, -0.14229370794861915, -1.99540698915273, 1.3130308260824273, -1.6762137150846073, 1.7358566500341819, -1.110729655013082, 1.5389907177931588, 2.646085876663788, 3.4520722646302486, -8.399023953420649, -1.6436998105556293, -3.0979841772438332, -4.957272857050993]\n", "iter:470\n", "w: [3.364151255339932, 0.04327971993819947, 1.4507925519956013, -3.945489274389911, 1.5625269960823145, 4.7295853153768075, -0.1422316373473262, -1.9969519933471749, 1.3140409877524486, -1.6774728081282557, 1.737030689755391, -1.111649461313024, 1.5401014748646364, 2.647972816587685, 3.4548155012881834, -8.405691233028676, -1.6450631794100061, -3.100271573757485, -4.961300083783655]\n", "iter:471\n", "w: [3.3667871547856874, 0.04321646531393679, 1.4519261055057366, -3.9485641707035506, 1.5638689053048223, 4.7330114042525215, -0.14216984600761234, -1.998494211013315, 1.3150494864315883, -1.6787299506608213, 1.7382026803450599, -1.1125674658055416, 1.5412102968704433, 2.6498568341805178, 3.4575533702361136, -8.412346915859347, -1.6464242358533503, -3.1025550774036117, -4.965320580214271]\n", "iter:472\n", "w: [3.3694187020888355, 0.04315349791906899, 1.4530576402901823, -3.9516339575292014, 1.5652085718816122, 4.736431965726081, -0.14210833227560246, -2.0000336526173044, 1.3160563278027737, -1.6799851488288013, 1.7393726293453295, -1.113483675865853, 1.5423171908554065, 2.6517379386542834, 3.460285893497394, -8.418991043999755, -1.6477829880122252, -3.104834702186599, -4.9693343696762655]\n", "iter:473\n", "w: [3.3720459122557234, 0.04309081603279944, 1.4541871638985373, -3.954698652398008, 1.5665460035905692, 4.7398470182907975, -0.14204709450945605, -2.0015703285647546, 1.31706151751836, -1.6812384087479355, 1.7405405442546176, -1.1143980988233706, 1.543422163824363, 2.6536161391748365, 3.463013092958049, -8.425623659299081, -1.6491394439686333, -3.1071104620311933, -4.9733414753752845]\n", "iter:474\n", "w: [3.3746688002104817, 0.04302841794704274, 1.4553146838367073, -3.9577582727468195, 1.5678812081674067, 4.743256580341478, -0.14198613107926494, -2.003104249201207, 1.3180650612003584, -1.6824897365034277, 1.7417064325279654, -1.1153107419620771, 1.5445252227424684, 2.655491444862225, 3.4657349903679027, -8.432244803370423, -1.6504936117603515, -3.1093823707831274, -4.977341920390169]\n", "iter:475\n", "w: [3.377287380795658, 0.04296630196631402, 1.4564402075672445, -3.9608128359188917, 1.5692141933059816, 4.746660670175166, -0.1419254403669513, -2.0046354248125993, 1.3190669644406625, -1.6837391381501663, 1.742870301577384, -1.1162216125208975, 1.5456263745355043, 2.657363864791019, 3.4684516073416995, -8.438854517592615, -1.651845499381263, -3.111650442209739, -4.98133572767393]\n", "iter:476\n", "w: [3.379901668772841, 0.042904466407619475, 1.4575637425096855, -3.9638623591645845, 1.570544966658605, 4.750059305991885, -0.1418650207661666, -2.006163865625724, 1.320067232801273, -1.6849866197129424, 1.7440321587721965, -1.117130717694066, 1.5467256260901816, 2.6592334079906412, 3.47116296536021, -8.445452843112026, -1.6531951147816861, -3.1139146900005854, -4.985322920054698]\n", "iter:477\n", "w: [3.3825116788232807, 0.042842909600347506, 1.4586852960408845, -3.9669068596420525, 1.5718735358363516, 4.753452505895362, -0.14180487068219225, -2.007689581808689, 1.3210658718145194, -1.6862321871866661, 1.7451920114393753, -1.1180380646314907, 1.5478229842544418, 2.6611000834456893, 3.4738690857713297, -8.452039820844334, -1.6545424658687, -3.1161751277680487, -4.9893035202366836]\n", "iter:478\n", "w: [3.3851174255485024, 0.042781629886161146, 1.459804875495344, -3.9699463544179268, 1.5731999084093629, 4.75684028789375, -0.14174498853184028, -2.0092125834713688, 1.3220628869832796, -1.6874758465365802, 1.746349866863878, -1.1189436604391148, 1.5489184558377542, 2.662963900096258, 3.4765699897911624, -8.458615491476301, -1.6558875605064667, -3.1184317690479375, -4.993277550801108]\n", "iter:479\n", "w: [3.3877189234709126, 0.042720625618891755, 1.460922488165543, -3.9729808604679926, 1.5745240919071513, 4.760222669900345, -0.14168537274335577, -2.0107328806658544, 1.323058283781198, -1.6887176036984721, 1.747505732288978, -1.119847512179272, 1.5500120476114114, 2.6648248668382575, 3.479265698505098, -8.46517989546751, -1.6572304065165506, -3.1206846273000814, -4.997245034207139]\n", "iter:480\n", "w: [3.3903161870344016, 0.04265989516443371, 1.4620381413022594, -3.9760103946778598, 1.5758460938188987, 4.763599669734293, -0.14162602175631966, -2.0122504833868957, 1.3240520676529015, -1.6899574645788833, 1.7486596149165938, -1.1207496268710415, 1.5511037663088212, 2.6666829925237283, 3.481956232868874, -8.471733073052102, -1.6585710116782357, -3.1229337159089208, -5.001205992792812]\n", "iter:481\n", "w: [3.3929092306049387, 0.04259943690063963, 1.463151842114893, -3.979034973843626, 1.5771659215937541, 4.766971305121292, -0.1415669340215528, -2.013765401572342, 1.325044244014213, -1.6911954350553178, 1.7498115219076134, -1.121650011490597, 1.5521936186257959, 2.6685382859611533, 3.484641613709629, -8.47827506424048, -1.6599093837288383, -3.1251790481840893, -5.005160448775942]\n", "iter:482\n", "w: [3.3954980684711606, 0.04253924921721732, 1.464263597771783, -3.9820546146725357, 1.5784835826411268, 4.7703375936942845, -0.1415081080010208, -2.015277645103578, 1.3260348182523627, -1.6924315209764473, 1.750961460382216, -1.1225486729715533, 1.5532816112208387, 2.6703907559157667, 3.487321861726949, -8.484805908821018, -1.661245530364018, -3.127420637360992, -5.009108424255026]\n", "iter:483\n", "w: [3.3980827148449575, 0.04247933051562767, 1.4653734154005225, -3.9850693337836303, 1.579799084330978, 4.773698552994146, -0.1414495421677399, -2.0167872238059554, 1.3270237957261988, -1.6936657281623153, 1.7521094374201904, -1.1234456182053094, 1.5543677507154279, 2.672240411109861, 3.489996997493895, -8.491325646361735, -1.6625794592380845, -3.1296584966013756, -5.013049941210143]\n", "iter:484\n", "w: [3.4006631838620494, 0.042419679208982916, 1.466481302088271, -3.9880791477083943, 1.5811124339941098, 4.777054200470365, -0.1413912350056834, -2.0182941474492204, 1.3280111817663938, -1.6948980624045389, 1.7532554600612513, -1.1243408540413882, 1.5554520436942973, 2.6740872602230903, 3.4926670414580303, -8.497834316211959, -1.6639111779643037, -3.131892638993897, -5.016985021503835]\n", "iter:485\n", "w: [3.403239489582558, 0.04236029372194651, 1.4675872648820636, -3.9910840728913954, 1.5824236389224502, 4.780404553481718, -0.14133318500968978, -2.0197984257479367, 1.3289969816756513, -1.6961285294665083, 1.7543995353053505, -1.1252343872877724, 1.556534496705715, 2.675931311892768, 3.4953320139424284, -8.504331957503977, -1.665240694115198, -3.1341230775546824, -5.020913686881988]\n", "iter:486\n", "w: [3.405811645991576, 0.0423011724906339, 1.4686913107891169, -3.994084125690916, 1.583732706369337, 4.783749629296937, -0.14127539068537068, -2.021300068361905, 1.3299812007289098, -1.6973571350835854, 1.755541670112986, -1.1261262247112376, 1.557615116261759, 2.677772574714167, 3.497991935146678, -8.510818609154668, -1.6665680152228473, -3.136349825227882, -5.024835958974698]\n", "iter:487\n", "w: [3.4083796669997257, 0.04224231396251408, 1.4697934467771316, -3.997079322379582, 1.5850396435497978, 4.787089445095368, -0.1412178505490204, -2.0227990848965787, 1.330963844173544, -1.6985838849633002, 1.756681871405509, -1.1270163730376823, 1.55869390883859, 2.6796110572408134, 3.5006468251478746, -8.517294309867124, -1.6678931487791833, -3.1385728948862215, -5.028751859297138]\n", "iter:488\n", "w: [3.4109435664437173, 0.04218371659631231, 1.4708936797745942, -4.000069679144984, 1.5863444576408288, 4.79042401796763, -0.14116056312752612, -2.024295484903475, 1.3319449172295652, -1.6998087847855436, 1.7578201460654264, -1.127904838952454, 1.559770880876722, 2.6814467679847787, 3.5032967039016025, -8.523759098132244, -1.6692161022362841, -3.140792299331545, -5.032661409250403]\n", "iter:489\n", "w: [3.413503358086898, 0.042125378861913476, 1.4719920166710732, -4.003055212090293, 1.5876471557816698, 4.793753364916261, -0.14110352695827905, -2.025789277880583, 1.3329244250898196, -1.7010318402027618, 1.7589565009367005, -1.1287916291006723, 1.5608460387812904, 2.683279715416967, 3.5059415912429084, -8.53021301223033, -1.6705368830066651, -3.143008051295355, -5.0365646301223626]\n", "iter:490\n", "w: [3.416059055619798, 0.04206729924026686, 1.4730884643175142, -4.006035937234869, 1.5889477450740768, 4.797077502856363, -0.14104674058908628, -2.0272804732727665, 1.3339023729201842, -1.702253056840145, 1.760090942825047, -1.129676750087549, 1.5619193889223166, 2.685109907967404, 3.5085815068872637, -8.536656090232656, -1.6718554984635667, -3.1452201634393453, -5.040461543088489]\n", "iter:491\n", "w: [3.418610672660669, 0.04200947622329161, 1.4741830295265321, -4.009011870514869, 1.5902462325825932, 4.80039644861623, -0.14099020257808353, -2.0287690804721654, 1.3348787658597618, -1.7034724402958183, 1.7612234784982288, -1.1305602084787059, 1.5629909376349722, 2.686937354025519, 3.511216470431519, -8.543088370003023, -1.6731719559412397, -3.1474286483559304, -5.044352169212694]\n", "iter:492\n", "w: [3.4211582227560218, 0.04195190831378293, 1.4752757190727, -4.011983027783843, 1.591542625334817, 4.803710218937987, -0.14093391149364884, -2.030255108818591, 1.3358536090210733, -1.7046899961410267, 1.762354114686347, -1.131442010800487, 1.5640606912198376, 2.6887620619404253, 3.513846501354849, -8.549509889199301, -1.6744862627352286, -3.1496335185687694, -5.04823652944814]\n", "iter:493\n", "w: [3.423701719381153, 0.041894594025319504, 1.4763665396928354, -4.014949424813325, 1.592836930321667, 4.807018830478213, -0.14087786591431672, -2.031738567599918, 1.3368269074902488, -1.705905729920322, 1.7634828580821305, -1.1323221635402712, 1.5651286559431612, 2.6905840400212004, 3.516471619019687, -8.555920685274966, -1.6757984261026517, -3.1518347865332847, -5.0521146446380625]\n", "iter:494\n", "w: [3.426241175940672, 0.041837531882171534, 1.4774554980862846, -4.017911077293426, 1.5941291544976448, 4.810322299808552, -0.14082206442869374, -2.0332194660524747, 1.3377986663272163, -1.7071196471517454, 1.76460971534122, -1.1332006731467785, 1.5661948380371142, 2.6924032965371603, 3.5190918426726516, -8.5623207954806, -1.6771084532624785, -3.154032464637174, -5.05598653551657]\n", "iter:495\n", "w: [3.428776605769018, 0.041780720419210104, 1.4785426009152032, -4.0208680008334134, 1.5954193047810965, 4.813620643416334, -0.14076650563537424, -2.0346978133614275, 1.33876889056589, -1.7083317533270097, 1.765734693082453, -1.1340775460303758, 1.567259243700044, 2.6942198397181323, 3.5217071914454623, -8.568710256865405, -1.6784163513958046, -3.1562265652009205, -5.05985222270944]\n", "iter:496\n", "w: [3.431308022130978, 0.04172415818181709, 1.4796278548048343, -4.023820210962288, 1.5967073880544707, 4.816913877705175, -0.14071118814285727, -2.036173618661162, 1.3397375852143554, -1.7095420539116801, 1.7668577978881421, -1.1349527885633781, 1.5683218790967246, 2.696033677754728, 3.52431768435585, -8.575089106278675, -1.6797221276461252, -3.1584171004782964, -5.063711726734912]\n", "iter:497\n", "w: [3.4338354382221934, 0.041667843725795885, 1.4807112663437854, -4.026767723129356, 1.5979934111645744, 4.820202018995581, -0.14065611056946437, -2.0376468910356618, 1.3407047552550526, -1.7107505543453518, 1.767979036304354, -1.1358264070803479, 1.569382750358605, 2.697844818798608, 3.5269233403084552, -8.581457380371281, -1.681025789119604, -3.160604082656862, -5.067565068004469]\n", "iter:498\n", "w: [3.4363588671696674, 0.041611775617283187, 1.4817928420843007, -4.0297105527047945, 1.5992773809228262, 4.823485083525543, -0.14060127154325755, -2.039117639518883, 1.3416704056449604, -1.7119572600418274, 1.7690984148411826, -1.136698407878391, 1.5704418635840554, 2.69965327096275, 3.5295241780957185, -8.587815115597122, -1.6823273428853422, -3.162787523858459, -5.071412266823611]\n", "iter:499\n", "w: [3.438878322032263, 0.04155595243266162, 1.4828725885425327, -4.0326487149802155, 1.6005593041055084, 4.826763087451123, -0.1405466697019589, -2.040585873095125, 1.342634541315775, -1.7131621763892917, 1.7702159399730233, -1.13756879721745, 1.5714992248386104, 2.7014590423217117, 3.532120216398763, -8.594162348214565, -1.6836267959756424, -3.1649674361397016, -5.0752533433926255]\n", "iter:500\n", "w: [3.4413938158012, 0.04150037275847308, 1.4839505121988095, -4.035582225169218, 1.6018391874540157, 4.830036046847044, -0.14049230369287025, -2.042051600699399, 1.34359716717409, -1.7143653087504862, 1.7713316181388405, -1.1384375813205938, 1.572554840155211, 2.7032621409118907, 3.534711473788267, -8.60049911428789, -1.6849241553862717, -3.1671438314924596, -5.079088317807345]\n", "iter:501\n", "w: [3.4439053614005455, 0.041445035191333156, 1.4850266194979014, -4.038511098407944, 1.6031170376751018, 4.833303977707262, -0.14043817217279383, -2.04351483121779, 1.344558288101574, -1.7155666624628805, 1.772445455742436, -1.1393047663743066, 1.5736087155344438, 2.7050625747317834, 3.5372979687253308, -8.606825449688694, -1.686219428076723, -3.16931672184434, -5.082917210059905]\n", "iter:502\n", "w: [3.446412971687699, 0.041389938337846044, 1.4861009168492838, -4.04143534975562, 1.6043928614411247, 4.836566895945543, -0.14038427380795396, -2.0449755734878208, 1.3455179089551454, -1.7167662428388426, 1.7735574591527123, -1.1401703585287715, 1.5746608569447778, 2.706860351742241, 3.5398797195623315, -8.613141390097304, -1.6875126209704723, -3.1714861190591623, -5.086740040039484]\n", "iter:503\n", "w: [3.448916659453875, 0.041335080814520565, 1.487173410627398, -4.044354994195101, 1.6056666653902882, 4.839824817396031, -0.1403306072739188, -2.0464338362988075, 1.3464760345671472, -1.7179640551658089, 1.7746676347039354, -1.1410343638981528, 1.5757112703227987, 2.7086554798667226, 3.5424567445437734, -8.61944697100417, -1.688803740955234, -3.1736520349374295, -5.09055682753305]\n", "iter:504\n", "w: [3.451416437424581, 0.04128046124768688, 1.4882441071719097, -4.047270046633404, 1.606938456126883, 4.843077757813808, -0.14027717125552375, -2.0478896283922143, 1.3474326697455192, -1.71916010470645, 1.7757759886959934, -1.1418967885608766, 1.5767599615734422, 2.710447966991548, 3.5450290618071265, -8.625742227711235, -1.690092794883215, -3.1758144812167957, -5.094367592226087]\n", "iter:505\n", "w: [3.4539123182600875, 0.041226078273414006, 1.4893130127879655, -4.050180521902244, 1.6082082402215239, 4.846325732875453, -0.1402239644467946, -2.0493429584620046, 1.3483878192739698, -1.7203543966988382, 1.776882527394654, -1.1427576385599056, 1.5778069365702245, 2.7122378209661453, 3.5475966893836612, -8.632027195333315, -1.6913797895713663, -3.177973469572527, -5.098172353703324]\n", "iter:506\n", "w: [3.4564043145558982, 0.04117193053742808, 1.490380133746446, -4.053086434758557, 1.6094760242113868, 4.8495687581795925, -0.14017098555087212, -2.0507938351549893, 1.349341487912144, -1.7215469363566107, 1.7779872570318185, -1.1436169199030148, 1.5788522011554702, 2.7140250496032983, 3.5501596451992703, -8.63830190879944, -1.6926647318016304, -3.1801290116179612, -5.101971131449456]\n", "iter:507\n", "w: [3.458892438843214, 0.04111801669503165, 1.4914454762842175, -4.055987799885021, 1.610741814600442, 4.85280684924745, -0.14011823327993714, -2.0522422670711706, 1.3502936803957928, -1.7227377288691335, 1.7790901838057738, -1.1444746385630624, 1.5798957611405389, 2.7158096606793913, 3.5527179470752865, -8.644566402854203, -1.6939476283211894, -3.182281118904961, -5.105763944849849]\n", "iter:508\n", "w: [3.4613767035893908, 0.04106433541102333, 1.4925090466043816, -4.058884631890579, 1.6120056178596858, 4.856040021523387, -0.14006570635513615, -2.053688262764084, 1.351244401436939, -1.7239267794016615, 1.7801913138814427, -1.1453308004782585, 1.5809376223060492, 2.717591661934651, 3.555271612729292, -8.65082071205909, -1.6952284858427085, -3.1844298029243636, -5.109550813191256]\n", "iter:509\n", "w: [3.463857121198397, 0.04101088535961857, 1.493570850876521, -4.0617769453109425, 1.6132674404273704, 4.859268290375436, -0.14001340350650784, -2.055131830741137, 1.3521936557240422, -1.7251140930955002, 1.7812906533906303, -1.1461854115524321, 1.5819777904021008, 2.719371061073388, 3.5578206597759183, -8.657064870793796, -1.696507311044579, -3.186575075106425, -5.11333175566251]\n", "iter:510\n", "w: [3.4663337040112627, 0.04095766522437105, 1.4946308952369451, -4.064664754609108, 1.614527288709232, 4.86249167109584, -0.13996132347291024, -2.0565729794639434, 1.353141447922162, -1.726299675068163, 1.78238820843227, -1.1470384776552944, 1.583016271148495, 2.7211478657642325, 3.56036510572764, -8.663298913257528, -1.6977841105711573, -3.188716946821263, -5.1171067913552175]\n", "iter:511\n", "w: [3.468806464306528, 0.04090467369809492, 1.4956891857889314, -4.067548074175857, 1.6157851690787157, 4.8657101789015735, -0.13990946500194845, -2.058011717348657, 1.3540877826731206, -1.7274835304135283, 1.783483985072665, -1.1478900046227005, 1.584053070234952, 2.7229220836403734, 3.562904967995561, -8.669522873470298, -1.6990588910330044, -3.1908554293792935, -5.120875939264447]\n", "iter:512\n", "w: [3.4712754143006856, 0.04085190948278754, 1.4967457286029666, -4.070426918330254, 1.6170410878772004, 4.868923828934869, -0.13985782684990317, -2.0594480527662995, 1.355032664595663, -1.7286656642019966, 1.7845779893457292, -1.1487399982569082, 1.5850881933213279, 2.724693722299789, 3.565440263890193, -8.675736785274212, -1.7003316590071205, -3.192990534031663, -5.124639218289409]\n", "iter:513\n", "w: [3.4737405661486185, 0.040799371289553306, 1.4978005297169839, -4.073301301320142, 1.6182950514142187, 4.87213263626373, -0.13980640778165992, -2.0608819940430885, 1.355976098285616, -1.7298460814806431, 1.7856702272532259, -1.1495884643268346, 1.5861216460378282, 2.726462789305481, 3.567971010622226, -8.681940682334734, -1.701602421037179, -3.1951222719706784, -5.128396647234127]\n", "iter:514\n", "w: [3.476201931944036, 0.04074705783852791, 1.4988535951365987, -4.076171237322633, 1.6195470659676787, 4.87533661588245, -0.13975520657063883, -2.0623135494607587, 1.356918088316046, -1.7310247872733722, 1.786760704765003, -1.15043540856831, 1.5871534339852205, 2.7282292921857034, 3.5704972253032943, -8.688134598141943, -1.7028711836337587, -3.1972506543302326, -5.132148244808112]\n", "iter:515\n", "w: [3.4786595237199034, 0.040694967858803575, 1.4999049308353434, -4.079036740444595, 1.6207971377840804, 4.878535782712114, -0.13970422199872504, -2.063742727256884, 1.3578586392374161, -1.7322017865810682, 1.7878494278192265, -1.1512808366843295, 1.5881835627350442, 2.72999323843419, 3.573018924946729, -8.694318566011788, -1.7041379532745728, -3.1993756921862238, -5.135894029627019]\n", "iter:516\n", "w: [3.4811133534488703, 0.040643100088354596, 1.5009545427548976, -4.08189782472313, 1.6220452730787323, 4.881730151601108, -0.13965345285619982, -2.0651695356251945, 1.3587977555777397, -1.7333770843817458, 1.7889364023226129, -1.1521247543453028, 1.5892120378298193, 2.7317546355103803, 3.5755361264683114, -8.700492619087315, -1.7054027364046973, -3.201497396556974, -5.139634020213304]\n", "iter:517\n", "w: [3.4835634330436918, 0.040591453273964184, 1.5020024368053184, -4.084754504126056, 1.6232914780359649, 4.884919737325615, -0.13960289794167235, -2.0665939827158915, 1.3597354418427354, -1.7345506856306994, 1.7900216341506576, -1.1529671671893007, 1.5902388647832533, 2.7335134908396435, 3.578048846687015, -8.706656790339903, -1.7066655394367958, -3.203615778403644, -5.143368234996876]\n", "iter:518\n", "w: [3.4860097743576484, 0.040540026171151274, 1.5030486188652679, -4.087606792552375, 1.6245357588093436, 4.888104554590111, -0.13955255606201206, -2.0680160766359585, 1.360671702515978, -1.7357225952606505, 1.791105129147863, -1.1538080808223004, 1.5912640490804446, 2.7352698118134997, 3.580557102325739, -8.712811112570465, -1.7079263687513433, -3.205730848630639, -5.14709669231574]\n", "iter:519\n", "w: [3.488452389184962, 0.04048881754409878, 1.5040930947822375, -4.090454703832747, 1.6257781215218785, 4.891284618027854, -0.13950242603228183, -2.0694358254494714, 1.3616065420590504, -1.7368928181818946, 1.7921868931279628, -1.154647500818427, 1.5922875961780867, 2.7370236057898403, 3.583060910012039, -8.718955618410664, -1.7091852306968485, -3.2078426180860196, -5.150819410416636]\n", "iter:520\n", "w: [3.4908912892612065, 0.04043782616558212, 1.5051358703727729, -4.0932982517299505, 1.6270185722662331, 4.8944599422013715, -0.13945250667567133, -2.070853237177904, 1.3625399649116927, -1.7380613592824452, 1.7932669318741457, -1.1554854327201942, 1.593309511504668, 2.7387748800931444, 3.5855602862788483, -8.725090340324098, -1.7104421315900729, -3.2099510975618997, -5.154536407455669]\n", "iter:521\n", "w: [3.4933264862637157, 0.040387050816898334, 1.506176951422694, -4.096137449939348, 1.628257117104931, 4.897630541602938, -0.1394027968234314, -2.072268319800433, 1.3634719754919506, -1.7392282234281793, 1.7943452511392763, -1.1563218820387429, 1.5943298004606723, 2.7405236420146974, 3.5880552475651917, -8.731215310607482, -1.7116970777162486, -3.2120562977948475, -5.158247701498941]\n", "iter:522\n", "w: [3.495757991811988, 0.040336490287796165, 1.5072163436873154, -4.0989723120893355, 1.6294937620705603, 4.900796430655059, -0.13935329531480906, -2.0736810812542363, 1.3644025781963225, -1.7403934154629783, 1.7954218566461138, -1.1571568542540758, 1.5953484684187749, 2.7422698988128023, 3.590545810216897, -8.737330561391824, -1.7129500753292946, -3.2141582294662796, -5.161953310523168]\n", "iter:523\n", "w: [3.498185817468086, 0.04028614337640644, 1.508254052891663, -4.101802851741804, 1.6307285131659779, 4.903957623710936, -0.1393040009969827, -2.0750915294347942, 1.3653317773999047, -1.7415569402088704, 1.7964967540875292, -1.1579903548152912, 1.5963655207240388, 2.7440136577129937, 3.5930319904872934, -8.743436124643582, -1.7142011306520306, -3.2162569032028525, -5.165653252416296]\n", "iter:524\n", "w: [3.5006099747370345, 0.04023600888917363, 1.50929008473069, -4.1046290823925835, 1.6319613763645098, 4.907114135054943, -0.1392549127249982, -2.076499672196183, 1.3662595774565365, -1.7427188024661708, 1.7975699491267199, -1.1588223891408154, 1.5973809626941091, 2.7457549259082477, 3.5955138045379105, -8.749532032165815, -1.7154502498763895, -3.218352329576851, -5.169347544978115]\n", "iter:525\n", "w: [3.503030475067213, 0.040186085640787274, 1.5103244448694904, -4.107451017471887, 1.6331923576101508, 4.910265978903085, -0.1392060293617057, -2.0779055173513683, 1.3671859826989425, -1.7438790070136199, 1.798641447397423, -1.1596529626186303, 1.5983947996194043, 2.7474937105591906, 3.5979912684391637, -8.755618315599326, -1.716697439163627, -3.220444519106571, -5.173036205920862]\n", "iter:526\n", "w: [3.505447329850746, 0.04013637245411514, 1.51135713894351, -4.110268670344754, 1.6344214628177631, 4.91341316940346, -0.13915734977769662, -2.079309072672496, 1.3681109974388759, -1.7450375586085218, 1.7997112545041265, -1.160482080606502, 1.5994070367633073, 2.7492300187943046, 3.6004643981710402, -8.761695006423793, -1.7179427046445308, -3.2225334822567024, -5.176719252869819]\n", "iter:527\n", "w: [3.507860550423889, 0.04008686816013577, 1.5123881725587578, -4.113082054311488, 1.635648697873272, 4.916555720636719, -0.1391088728512416, -2.0807103458911795, 1.3690346259672574, -1.7461944619868808, 1.800779376022278, -1.1613097484322052, 1.6004176793623544, 2.750963857710135, 3.6029332096237727, -8.76776213595888, -1.7191860524196272, -3.2246192294387046, -5.180396703363908]\n", "iter:528\n", "w: [3.5102701480674128, 0.040037571597872704, 1.5134175512920118, -4.115891182608087, 1.6368740686338603, 4.919693646616512, -0.1390605974682289, -2.082109344698786, 1.3699568725543165, -1.747349721863536, 1.8018458174984924, -1.1621359713937465, 1.6014267326264215, 2.752695234371491, 3.6053977185985104, -8.773819735365365, -1.7204274885593855, -3.2267017710111814, -5.184068574856281]\n", "iter:529\n", "w: [3.51267613400698, 0.03998848161432867, 1.5144452806910274, -4.118696068406678, 1.638097580928162, 4.922826961289942, -0.1390125225221032, -2.0835060767467173, 1.3708777414497284, -1.7485033429322967, 1.8029105844507565, -1.1629607547595846, 1.602434201738911, 2.75442415581165, 3.607857940807983, -8.779867835646224, -1.721667019104422, -3.228781117280252, -5.187734884714904]\n", "iter:530\n", "w: [3.515078519413525, 0.039939597064420684, 1.5154713662747397, -4.1214967248159375, 1.639319240556452, 4.925955678538006, -0.13896464691380503, -2.0849005496466924, 1.3717972368827518, -1.7496553298660744, 1.803973682368633, -1.1637841037688503, 1.6034400918569343, 2.756150629032554, 3.610313891877159, -8.785906467647733, -1.722904650065701, -3.2308572784999177, -5.191395650223134]\n", "iter:531\n", "w: [3.5174773154036236, 0.03989091681091574, 1.5164958135334672, -4.12429316488152, 1.6405390532908375, 4.929079812176039, -0.13891696955171118, -2.086292770971023, 1.3727153630623645, -1.7508056873170157, 1.8050351167134622, -1.164606023631563, 1.6044444081114948, 2.7578746610050104, 3.612765587343896, -8.791935662060546, -1.724140387424734, -3.2329302648724263, -5.195050888580294]\n", "iter:532\n", "w: [3.519872533039864, 0.039842439724366956, 1.5175186279291124, -4.127085401586471, 1.641757024875445, 4.932199375954146, -0.13886948935157478, -2.087682748252891, 1.3736321241773988, -1.7519544199166328, 1.8060948929185618, -1.1654265195288451, 1.6054471556076682, 2.759596258668888, 3.615213042659589, -8.797955449420765, -1.725374237133779, -3.2350000865486335, -5.198700616902241]\n", "iter:533\n", "w: [3.522264183331214, 0.039794164683050366, 1.5185398148953602, -4.129873447851647, 1.6429731610266063, 4.935314383557637, -0.13882220523646688, -2.08907048898662, 1.3745475243966738, -1.753101532275933, 1.8071530163894247, -1.1662455966131364, 1.6064483394247817, 2.76131542893331, 3.617656273189807, -8.803965860111008, -1.7266062051160351, -3.2370667536283584, -5.2023448522219295]\n", "iter:534\n", "w: [3.524652277233382, 0.03974609057290251, 1.5195593798378757, -4.132657316536125, 1.644187467433044, 4.93842484860746, -0.13877511613671786, -2.090456000627947, 1.3754615678691295, -1.7542470289855472, 1.808209492503915, -1.1670632600084034, 1.6074479646165916, 2.7630321786768497, 3.62009529421493, -8.80996692436146, -1.7278362972658383, -3.239130276160741, -5.2059836114899705]\n", "iter:535\n", "w: [3.527036825649178, 0.039698216287458454, 1.5205773281344992, -4.135437020437611, 1.6453999497560547, 4.94153078466062, -0.13872822098985982, -2.09183929059429, 1.376374258723957, -1.7553909146158586, 1.809264326612463, -1.1678795148103498, 1.608446036211459, 2.76474651474772, 3.622530120930776, -8.815958672250915, -1.7290645194488543, -3.2411906641445904, -5.209616911575186]\n", "iter:536\n", "w: [3.529417839428871, 0.03965054072779003, 1.5215936651354405, -4.138212572292844, 1.6466106136296907, 4.944632205210605, -0.13868151874056905, -2.093220366265014, 1.3772856010707286, -1.756533193717128, 1.810317524038258, -1.1686943660866234, 1.609442559212524, 2.7664584439639643, 3.6249607684492235, -8.821941133707817, -1.7302908775022694, -3.243247927528736, -5.213244769265153]\n", "iter:537\n", "w: [3.5317953293705404, 0.039603062802445424, 1.5226083961634709, -4.140983984777996, 1.6478194646609396, 4.947729123687803, -0.13863500834060952, -2.094599234981694, 1.3781955989995283, -1.7576738708196198, 1.8113690900774384, -1.169507818877021, 1.6104375385978795, 2.768167973113644, 3.6273872517988295, -8.827914338511283, -1.7315153772349805, -3.2453020762123694, -5.2168672012667585]\n", "iter:538\n", "w: [3.5341693062204294, 0.03955578142738853, 1.5236215265141133, -4.143751270509078, 1.6490265084299034, 4.950821553459915, -0.1385886887487766, -2.095975904048378, 1.3791042565810783, -1.758812950433726, 1.812419029999282, -1.1703198781936928, 1.6114309793207413, 2.7698751089550258, 3.629809585925439, -8.83387831629212, -1.7327380244277832, -3.2473531200453896, -5.220484224206726]\n", "iter:539\n", "w: [3.5365397806732903, 0.03950869552593954, 1.5246330614558312, -4.146514442042323, 1.6502317504899757, 4.953909507832368, -0.13854255893084125, -2.0973503807318443, 1.3800115778668665, -1.75995043705009, 1.8134673490463928, -1.1711305490213433, 1.6124228863096193, 2.7715798582167666, 3.6322277856927925, -8.839833096533836, -1.7339588248335582, -3.24940106882874, -5.22409585463216]\n", "iter:540\n", "w: [3.53890676337273, 0.03946180402871554, 1.5256430062302158, -4.149273511874583, 1.6514351963680172, 4.956993000048722, -0.13849661785949488, -2.0987226722618595, 1.3809175668892724, -1.761086335139729, 1.8145140524348877, -1.1719398363174316, 1.6134132644684847, 2.7732822275980977, 3.6346418658831254, -8.845778708573638, -1.735177784177456, -3.251445932314745, -5.227702109011071]\n", "iter:541\n", "w: [3.541270264911551, 0.039415105873571915, 1.5266513660521717, -4.15202849244372, 1.6526368515645293, 4.960072043291073, -0.1384508645142945, -2.1000927858314324, 1.381822227661692, -1.7622206491541546, 1.8155591453545803, -1.1727477450123696, 1.614402118676938, 2.7749822237690056, 3.6370518411977635, -8.85171518160343, -1.7363949081570809, -3.2534877202074433, -5.231303003732905]\n", "iter:542\n", "w: [3.5436302958320898, 0.03936860000554444, 1.5276581461101004, -4.154779396128985, 1.6538367215538285, 4.963146650680455, -0.13840529788160874, -2.101460728597067, 1.3827255641786618, -1.7633533835254938, 1.8166026329691636, -1.173554280009718, 1.615389453790374, 2.776679853370414, 3.6394577262577124, -8.857642544670783, -1.7376102024426718, -3.2555264421629184, -5.234898555109061]\n", "iter:543\n", "w: [3.5459868666265537, 0.03932228537679152, 1.5286633515660832, -4.157526235251403, 1.655034811784216, 4.966216835277235, -0.13835991695456396, -2.102826507679012, 1.3836275804159803, -1.764484542666608, 1.8176445204163914, -1.174359446186381, 1.6163752746401459, 2.7783751230143623, 3.6418595356042416, -8.863560826679926, -1.7388236726772837, -3.257562107789625, -5.23848877937341]\n", "iter:544\n", "w: [3.548339987737352, 0.03927616094653725, 1.5296669875560616, -4.160269022074148, 1.6562311276781478, 4.969282610081508, -0.138314720732991, -2.10419013016151, 1.3845282803308316, -1.765614130971212, 1.8186848128082567, -1.1751632483927994, 1.6173595860337284, 2.7800680392841826, 3.6442572836994636, -8.869470056392698, -1.7400353244769653, -3.259594726648714, -5.242073692682804]\n", "iter:545\n", "w: [3.5506896695574257, 0.03923022568101504, 1.5306690591900165, -4.163007768802923, 1.6574256746324043, 4.972343988033486, -0.1382697082233724, -2.1055516030930423, 1.3854276678619044, -1.7667421528139897, 1.8197235152311713, -1.175965691453141, 1.6183423927548788, 2.781758608734677, 3.6466509849269078, -8.875370262429515, -1.7412451634309376, -3.261624308254355, -5.24565331111759]\n", "iter:546\n", "w: [3.5530359224305768, 0.03918447855341151, 1.531669571552146, -4.165742487586328, 1.658618458018256, 4.975400982013887, -0.1382248784387902, -2.1069109334865734, 1.3863257469295127, -1.7678686125507126, 1.82076063274614, -1.1767667801654904, 1.6193236995637965, 2.7834468378922907, 3.649040653592089, -8.881261473270316, -1.742453195101768, -3.263650862074053, -5.249227650682108]\n", "iter:547\n", "w: [3.55537875665179, 0.039138918543810956, 1.5326685297010416, -4.168473190516228, 1.6598094831816295, 4.978453604844318, -0.13818023039887398, -2.108268128319793, 1.387222521435714, -1.7689935145183537, 1.8217961703889376, -1.1775665193020373, 1.6203035111972823, 2.7851327332552875, 3.6514263039230723, -8.887143717255505, -1.7436594250255466, -3.2656743975289677, -5.252796727305191]\n", "iter:548\n", "w: [3.5577181824675543, 0.039093544639140526, 1.5336659386698628, -4.171199889628125, 1.6609987554432726, 4.981501869287654, -0.13813576312974962, -2.109623194535355, 1.388117995264427, -1.7701168630352029, 1.82283013317028, -1.1783649136092613, 1.6212818323688953, 2.7868163012939196, 3.6538079500710294, -8.89301702258689, -1.7448638587120573, -3.267694923994223, -5.256360556840664]\n", "iter:549\n", "w: [3.5600542100761845, 0.03904835583311589, 1.53466180346651, -4.173922596901512, 1.6621862800989162, 4.984545788048417, -0.13809147566398847, -2.1109761390411164, 1.3890121722815483, -1.7712386624009797, 1.8238625260759975, -1.1791619678081173, 1.6222586677691087, 2.7884975484505987, 3.6561856061107942, -8.8988814173286, -1.7460665016449506, -3.269712450799221, -5.25991915506783]\n", "iter:550\n", "w: [3.5623868496281332, 0.039003351126187066, 1.5356561290737967, -4.176641324260243, 1.6633720624194366, 4.98758537377315, -0.13804736704055667, -2.1123269687103714, 1.3899050563350688, -1.7723589168969471, 1.8248933540672039, -1.179957686594218, 1.623234022065464, 2.790176481140066, 3.6585592860414105, -8.90473692940801, -1.7472673592819126, -3.2717269872279475, -5.263472537691957]\n", "iter:551\n", "w: [3.564716111226307, 0.03895852952548504, 1.5366489204496192, -4.17935608357288, 1.6645561076510151, 4.990620639050787, -0.13800343630476541, -2.1136756903820864, 1.3907966512551877, -1.7734776307860218, 1.825922622080466, -1.180752074638015, 1.6242078999027252, 2.791853105749558, 3.6609290037866757, -8.91058358661665, -1.7484664370548342, -3.2737385425192804, -5.267020720344765]\n", "iter:552\n", "w: [3.567042004926376, 0.03891389004476881, 1.5376401825271258, -4.182066886653052, 1.6657384210152977, 4.993651596413022, -0.13795968250822108, -2.1150223108611312, 1.3916869608544264, -1.7745948083128866, 1.8269503350279703, -1.1815451365849778, 1.62518030590303, 2.7935274286389755, 3.663294773195681, -8.916421416611106, -1.7496637403699784, -3.2757471258672886, -5.2705637185849]\n", "iter:553\n", "w: [3.569364540737082, 0.03886943170437302, 1.5386299202148834, -4.184773745259807, 1.666919007709552, 4.996678258334671, -0.13791610470877674, -2.1163668369185085, 1.392575988927742, -1.7757104537040993, 1.8279764977976893, -1.1823368770557732, 1.6261512446660409, 2.795199456141045, 3.6656566080433444, -8.922250446913916, -1.7508592746081457, -3.277752746421535, -5.274101547898411]\n", "iter:554\n", "w: [3.5716837286205436, 0.03882515353115579, 1.5396181383970446, -4.187476671097955, 1.668097872906824, 4.999700637234039, -0.13787270197048282, -2.1177092752915825, 1.3934637392526386, -1.7768245711682018, 1.8290011152535453, -1.18312730064644, 1.6271207207690939, 2.7968691945614865, 3.6680145220309415, -8.928070704914457, -1.752053045124839, -3.279755413287372, -5.277634223699218]\n", "iter:555\n", "w: [3.57399957849256, 0.03878105455844717, 1.5406048419335114, -4.190175675818421, 1.6692750217560937, 5.002718745473272, -0.13782947336353935, -2.1190496326843036, 1.3943502155892786, -1.7779371648958286, 1.8300241922355736, -1.1839164119285652, 1.6280887387673475, 2.7985366501791717, 3.670368528786629, -8.933882217869828, -1.7532450572504268, -3.281755135526239, -5.281161761329582]\n", "iter:556\n", "w: [3.5763121002229092, 0.0387371338259982, 1.5415900356600984, -4.192870771018579, 1.670450459382427, 5.005732595358721, -0.1377864179642479, -2.120387915767433, 1.3952354216805933, -1.7790482390598144, 1.8310457335600843, -1.1847042154494565, 1.6290553031939288, 2.8002018292462885, 3.672718641865966, -8.939685012905718, -1.7544353162903041, -3.2837519221559526, -5.284684176060563]\n", "iter:557\n", "w: [3.5786213036356487, 0.038693390379930304, 1.5425737243886946, -4.195561968242597, 1.6716241908871305, 5.0087421991412855, -0.1377435348549643, -2.1217241311787656, 1.3961193612523914, -1.7801577978152998, 1.832065744019822, -1.1854907157323145, 1.6300204185600797, 2.8018647379884984, 3.675064874752428, -8.945479117017276, -1.755623827525054, -3.285745782150997, -5.288201483092484]\n", "iter:558\n", "w: [3.5809271985094084, 0.03864982327268512, 1.5435559129074239, -4.1982492789817725, 1.6727962213479017, 5.011747569016771, -0.13770082312405188, -2.1230582855233493, 1.397002038013468, -1.7812658452998376, 1.8330842283841249, -1.1862759172764028, 1.6309840893553014, 2.8035253826050965, 3.6774072408579217, -8.951264557069967, -1.756810596210606, -3.287736724442813, -5.291713697555382]\n", "iter:559\n", "w: [3.583229794577684, 0.03860643156297465, 1.5445366059808043, -4.200932714674867, 1.6739665558189791, 5.014748717126232, -0.1376582818658345, -2.1243903853737027, 1.3978834556557116, -1.7823723856334968, 1.8341011913990821, -1.187059824557217, 1.631946320047497, 2.8051837692691675, 3.679745753523287, -8.95704135980042, -1.7579956275783946, -3.289724757920081, -5.295220834509461]\n", "iter:560\n", "w: [3.5855291015291275, 0.03856321431573234, 1.5455158083499054, -4.2036122867084345, 1.675135199331291, 5.017745655556317, -0.1376159101805511, -2.125720437270034, 1.3987636178542109, -1.7834774229189674, 1.8351166377876895, -1.187842442026652, 1.6329071150831145, 2.8068399041277425, 3.6820804260188025, -8.962809551817282, -1.7591789268355154, -3.2917098914290053, -5.298722908945539]\n", "iter:561\n", "w: [3.587825129007836, 0.03852017060206409, 1.5464935247325047, -4.2062880064171555, 1.6763021568926038, 5.0207383963396115, -0.13757370717430925, -2.1270484477204525, 1.3996425282673606, -1.784580961241663, 1.8361305722500045, -1.1886237741131676, 1.6338664788872863, 2.808493793301954, 3.6844112715446826, -8.968569159602042, -1.76036049916488, -3.293692133773594, -5.302219935785491]\n", "iter:562\n", "w: [3.590117886613635, 0.0384772994991999, 1.5474697598232428, -4.208959885084159, 1.6774674334876676, 5.0237269514549725, -0.13753167195904029, -2.1283744232011856, 1.4005201905369664, -1.785683004669823, 1.8371429994632997, -1.1894038252219534, 1.6348244158639702, 2.8101454428871877, 3.68673830323157, -8.974320209509873, -1.7615403497253712, -3.295671493715936, -5.30571192988269]\n", "iter:563\n", "w: [3.5924073839023642, 0.038434600090446004, 1.5484445182937772, -4.2116279339413465, 1.6786310340783621, 5.026711332827868, -0.13748980365245447, -2.1296983701567864, 1.4013966082883487, -1.7867835572546147, 1.838153924082215, -1.1901825997350917, 1.6357809303960873, 2.8117948589532364, 3.689061534141027, -8.980062727770445, -1.7627184836519945, -3.2976479799764786, -5.309198906022443]\n", "iter:564\n", "w: [3.5946936303861565, 0.03839207146513737, 1.549417804792935, -4.214292164169715, 1.6797929636038407, 5.029691552330709, -0.13744810137799612, -2.131020295000345, 1.402271785130446, -1.787882623030233, 1.8391633507389078, -1.1909601020117186, 1.6367360268456592, 2.8134420475444513, 3.6913809772660184, -8.98579674048875, -1.7638949060560305, -3.2996216012342985, -5.312680878922425]\n", "iter:565\n", "w: [3.5969766355337165, 0.03834971271859071, 1.5503896239468637, -4.21695258689967, 1.6809532269806728, 5.032667621783179, -0.13740656426480002, -2.132340204113696, 1.4031457246559174, -1.788980206014001, 1.8401712840432032, -1.1917363363881854, 1.6376897095539449, 2.8150870146798903, 3.693696645531393, -8.991522273645908, -1.7650696220251847, -3.3015923661273736, -5.316157863233106]\n", "iter:566\n", "w: [3.599256408770598, 0.03830752295205776, 1.5513599803591818, -4.219609213211349, 1.6821118291029868, 5.035639552952562, -0.13736519144764717, -2.1336581038476234, 1.4040184304412435, -1.790076310206469, 1.8411777285827418, -1.1925113071782159, 1.6386419828415748, 2.816729766353468, 3.6960085517943604, -8.997239353099966, -1.7662426366237372, -3.303560283252853, -5.319629873538178]\n", "iter:567\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "w: [3.6015329594794765, 0.038265501272679074, 1.5523288786111267, -4.222262054134928, 1.6832687748426098, 5.038607357554067, -0.1373239820669217, -2.1349740005220674, 1.4048899060468283, -1.791170939591513, 1.842182688923127, -1.193285018673065, 1.6395928510086861, 2.818370308534102, 3.6983167088449638, -9.002948004586703, -1.7674139548926897, -3.305525361167321, -5.323096924354977]\n", "iter:568\n", "w: [3.603806297000422, 0.03822364679343828, 1.553296323261703, -4.224911120650935, 1.6844240690492072, 5.041571047251153, -0.13728293526856802, -2.136287900426324, 1.4057601550170977, -1.7922640981364317, 1.8431861696080705, -1.1940574751416748, 1.6405423183350545, 2.820008647165862, 3.700621129406547, -9.008648253720411, -1.7685835818499134, -3.3074876083870652, -5.326559030134903]\n", "iter:569\n", "w: [3.6060764306311675, 0.038181958633116665, 1.5542623188478286, -4.227556423690558, 1.6855777165504218, 5.044530633655842, -0.1372420502040481, -2.1375998098192484, 1.4066291808806, -1.7933557897920438, 1.8441881751595368, -1.1948286808308284, 1.6414903890802264, 2.8216447881681113, 3.702921826136221, -9.014340125994687, -1.7697515224902933, -3.309447033388333, -5.330016205263834]\n", "iter:570\n", "w: [3.6083433696273763, 0.03814043591624819, 1.5552268698844796, -4.230197974135949, 1.6867297221520106, 5.047486128329047, -0.1372013260302992, -2.1389097349294524, 1.407496987150103, -1.794446018492784, 1.845188710077887, -1.195598639965304, 1.64243706748365, 2.823278737435654, 3.7052188116253215, -9.020023646783207, -1.7709177817858737, -3.311403644607597, -5.3334684640625385]\n", "iter:571\n", "w: [3.610607123202908, 0.038099077773074905, 1.5561899808648343, -4.232835782820531, 1.6878800906379812, 5.050437542780874, -0.1371607619096922, -2.140217681955504, 1.4083635773226928, -1.7955347881567982, 1.8461877788420202, -1.1963673567480277, 1.6433823577648043, 2.8249105008388766, 3.707512098399868, -9.025698841340489, -1.7720823646860002, -3.3133574504418086, -5.336915820787083]\n", "iter:572\n", "w: [3.612867700530081, 0.03805788333950263, 1.5571516562604157, -4.235469860529294, 1.689028826770727, 5.053384888470943, -0.13712035700999012, -2.1415236570661214, 1.409228954879869, -1.796622102686038, 1.8471853859095149, -1.1971348353602231, 1.6443262641233285, 2.82654008422389, 3.709801698921015, -9.031365734802671, -1.7732452761174633, -3.3153084592486564, -5.34035828962924]\n", "iter:573\n", "w: [3.615125110739932, 0.038016851757057525, 1.5581119005212336, -4.238100217999098, 1.6901759352911612, 5.056328176808695, -0.13708011050430685, -2.142827666400369, 1.4100931232876426, -1.7977079659663548, 1.8481815357167684, -1.1979010799615615, 1.6452687907391483, 2.8281674934126704, 3.7120876255855, -9.03702435218826, -1.7744065209846376, -3.3172566793468183, -5.3437958847168865]\n", "iter:574\n", "w: [3.6173793629224753, 0.037975982172842405, 1.5590707180759242, -4.240726865918965, 1.6913214209188498, 5.059267419153697, -0.1370400215710666, -2.1441297160678503, 1.4109560859966293, -1.7987923818675926, 1.8491762326791354, -1.1986660946903096, 1.6462099417726037, 2.829792734203199, 3.7143698907260903, -9.042674718398883, -1.7755661041696231, -3.3192021190162126, -5.347228620114407]\n", "iter:575\n", "w: [3.619630466126961, 0.037935273739493756, 1.56002811333189, -4.243349814930377, 1.692465288352144, 5.062202626815949, -0.13700008939396344, -2.145429812148897, 1.4118178464421456, -1.7998753542436803, 1.850169481191066, -1.1994298836634778, 1.6471497213645736, 2.831415812369601, 3.7166485066120214, -9.048316858220042, -1.7767240305323833, -3.3211447864982486, -5.350656509823085]\n", "iter:576\n", "w: [3.621878429362127, 0.03789472561513925, 1.5609840906754375, -4.245969075627562, 1.6936075422683106, 5.065133811056185, -0.13696031316192125, -2.146727960694762, 1.4126784080443016, -1.8009568869327244, 1.8511612856262412, -1.2001924509769644, 1.6480881336366007, 2.833036733662283, 3.718923485449436, -9.053950796321844, -1.7778803049108824, -3.323084689996073, -5.354079567781498]\n", "iter:577\n", "w: [3.624123261596453, 0.037854336963355224, 1.5619386544719143, -4.248584658557789, 1.694748187323662, 5.068060983086174, -0.13692069206905408, -2.148024167727804, 1.4135377742080941, -1.8020369837570984, 1.8521516503377082, -1.200953800705701, 1.6490251826910145, 2.834655503808068, 3.721194839381817, -9.059576557259742, -1.7790349321212227, -3.325021837674814, -5.357497807865906]\n", "iter:578\n", "w: [3.6263649717584094, 0.03781410695312496, 1.562891809065845, -4.251196574221653, 1.6958872281536856, 5.070984154069016, -0.13688122531462676, -2.1493184392416778, 1.4143959483234998, -1.8031156485235347, 1.853140579658015, -1.201713936903796, 1.6499608726110544, 2.8362721285103336, 3.7234625804904176, -9.065194165475258, -1.7801879169577792, -3.3269562376618276, -5.360911243890637]\n", "iter:579\n", "w: [3.628603568736707, 0.03777403475879738, 1.563843558781066, -4.253804833073356, 1.6970246693731712, 5.073903335119438, -0.1368419121030159, -2.1506107812015167, 1.4152529337655653, -1.804192885023213, 1.854128077899342, -1.2024728636046758, 1.6508952074609904, 2.8378866134491454, 3.7257267207946865, -9.07080364529671, -1.7813392641933348, -3.3288878980469354, -5.364319889608472]\n", "iter:580\n", "w: [3.630839061380543, 0.03773411956004561, 1.5647939079208584, -4.256409445520993, 1.698160515576338, 5.076818537304083, -0.1368027516436713, -2.1519011995441173, 1.416108733894499, -1.8052686970318497, 1.8551141493536356, -1.2032305848212264, 1.6518281912862443, 2.839498964281389, 3.727987272252691, -9.076405020939925, -1.7824889785792126, -3.3308168268826663, -5.367723758711024]\n", "iter:581\n", "w: [3.633071458499844, 0.037694360541826226, 1.565742860768082, -4.259010421926834, 1.6992947713369602, 5.079729771641809, -0.1367637431510776, -2.153189700178122, 1.4169633520557612, -1.8063430883097862, 1.856098798292737, -1.2039871045459327, 1.6527598281135085, 2.841109186640904, 3.7302442467615378, -9.08199831650895, -1.7836370648454092, -3.332743032184492, -5.371122864829111]\n", "iter:582\n", "w: [3.635300768865511, 0.03765475689433911, 1.5666904215853055, -4.261607772607595, 1.7004274412084925, 5.082637049103965, -0.1367248858447162, -2.1544762889841977, 1.4178167915801527, -1.8074160626020765, 1.8570820289685133, -1.2047424267510176, 1.653690121950865, 2.842717286138615, 3.7324976561577836, -9.087583555996758, -1.7847835277007251, -3.334666521931065, -5.374517221533138]\n", "iter:583\n", "w: [3.6375270012096568, 0.03761530781298696, 1.5676365946149375, -4.264201507834723, 1.7015585297241937, 5.08554038061469, -0.1366861789490278, -2.1557609718152184, 1.4186690557839046, -1.808487623638574, 1.8580638456129852, -1.2054965553885788, 1.654619076787903, 2.844323268362662, 3.734747512217853, -9.093160763285947, -1.7859283718328955, -3.336587304064448, -5.37790684233346]\n", "iter:584\n", "w: [3.639750164225848, 0.03757601249833608, 1.5685813840793559, -4.26679163783466, 1.7026880413972498, 5.088439777051184, -0.13664762169337485, -2.15704375449644, 1.4195201479687656, -1.8095577751340175, 1.8590442524384554, -1.206249494390725, 1.655546696595835, 2.8459271388785297, 3.736993826658442, -9.098729962149434, -1.7870716019087192, -3.33850538649035, -5.381291740680757]\n", "iter:585\n", "w: [3.641970266569338, 0.03753687015607655, 1.569524794181036, -4.269378172789124, 1.7038159807208957, 5.091335249244, -0.1366092133120047, -2.1583246428256775, 1.4203700714220897, -1.8106265207881176, 1.860023253637634, -1.2070012476697112, 1.6564729853276132, 2.847528903229175, 3.739236611136927, -9.104291176251143, -1.7882132225741867, -3.3404207770783527, -5.384671929966392]\n", "iter:586\n", "w: [3.6441873168573045, 0.03749787999698354, 1.570466829102678, -4.27196112283537, 1.7049423521685363, 5.0942268079773125, -0.1365709530440128, -2.159603642573481, 1.4212188294169223, -1.8116938642856406, 1.8610008533837645, -1.2077518191180727, 1.6573979469180442, 2.849128566935157, 3.7414758772517622, -9.109844429146685, -1.7893532384546074, -3.3423334836621397, -5.388047423522779]\n", "iter:587\n", "w: [3.646401323669081, 0.037459041236878585, 1.571407493007333, -4.274540498066465, 1.7060671601938673, 5.0971144639892065, -0.13653284013330638, -2.160880759483308, 1.4220664252120871, -1.812759809296494, 1.8619770558307482, -1.2085012126087578, 1.6583215852839024, 2.850726135494761, 3.7437116365428817, -9.11538974428404, -1.7904916541547358, -3.3442435140397224, -5.391418234623741]\n", "iter:588\n", "w: [3.648612295546388, 0.03742035309659109, 1.5723467900385288, -4.277116308531547, 1.7071904092309933, 5.099998227971942, -0.13649487382856842, -2.162155999271696, 1.4229128620522709, -1.8138243594758094, 1.8629518651132677, -1.2092494319952594, 1.659243904324043, 2.8523216143841257, 3.7459439004920925, -9.120927145004226, -1.7916284742588968, -3.346150875973664, -5.394784376484863]\n", "iter:589\n", "w: [3.6508202409935637, 0.037381814801920606, 1.5732847243203938, -4.279688564236091, 1.7083121036945474, 5.102878110572231, -0.13645705338322187, -2.1634293676284337, 1.423758143168108, -1.8148875184640256, 1.863925285346909, -1.2099964811117456, 1.6601649079195149, 2.8539150090573666, 3.748172680523466, -9.126456654541967, -1.79276370333111, -3.348055577191302, -5.398145862263853]\n", "iter:590\n", "w: [3.6530251684777886, 0.037343425583598835, 1.5742212999577798, -4.28225727514217, 1.7094322479798074, 5.1057541223915095, -0.13641937805539409, -2.1647008702167296, 1.4246022717762645, -1.8159492898869716, 1.864897320628283, -1.2107423637731896, 1.6610845999336707, 2.8555063249467003, 3.7503979880037273, -9.131978296026354, -1.7938973459152125, -3.3499576253849686, -5.401502705060884]\n", "iter:591\n", "w: [3.6552270864293135, 0.03730518467725254, 1.5751565210363854, -4.284822451168716, 1.7105508464628127, 5.108626273986202, -0.13638184710788195, -2.1659705126733813, 1.4254452510795224, -1.8170096773559476, 1.8658679750351461, -1.2114870837754976, 1.662002984212278, 2.8570955674625673, 3.7526198342426396, -9.137492092481502, -1.7950294065349819, -3.3518570282122098, -5.404854917918949]\n", "iter:592\n", "w: [3.657426003241682, 0.03726709132336659, 1.5760903916228763, -4.287384102191774, 1.7116679035004803, 5.111494575867991, -0.13634445980811685, -2.16723830060894, 1.4262870842668607, -1.8180686844678073, 1.8668372526265191, -1.2122306448956368, 1.6629200645836284, 2.858682741993754, 3.7548382304933865, -9.142998066827197, -1.7961598896942568, -3.3537537932960024, -5.408202513824205]\n", "iter:593\n", "w: [3.659621927271954, 0.03722914476724712, 1.5770229157650064, -4.289942238044758, 1.7127834234307187, 5.11435903850408, -0.13630721542813004, -2.1685042396078797, 1.427127774513539, -1.8191263148050376, 1.8678051574428063, -1.212973050891761, 1.6638358448586459, 2.8602678539075135, 3.7570531879529505, -9.148496241879545, -1.7972887998770581, -3.3556479282249696, -5.411545505706312]\n", "iter:594\n", "w: [3.6618148668409245, 0.03719134425898512, 1.5779540974917368, -4.292496868518707, 1.7138974105725422, 5.117219672317456, -0.13627011324451874, -2.169768335228757, 1.4279673249811782, -1.820182571935839, 1.8687716935059129, -1.2137143055033368, 1.6647503288309946, 2.8618509085496853, 3.7592647177624885, -9.153986640351615, -1.7984161415477085, -3.357539440553594, -5.414883906438779]\n", "iter:595\n", "w: [3.664004830233343, 0.03715368905342038, 1.578883940813354, -4.295048003362533, 1.7150098692261837, 5.12007648768715, -0.13623315253841184, -2.171030593004377, 1.4288057388178417, -1.8212374594142056, 1.869736864819361, -1.2144544124512668, 1.6656635202771861, 2.8634319112448154, 3.761472831007704, -9.15946928485407, -1.7995419191509496, -3.3594283378024303, -5.418217728839297]\n", "iter:596\n", "w: [3.6661918256981303, 0.03711617841010616, 1.5798124497215877, -4.297595652283272, 1.716120803673207, 5.122929494948494, -0.13619633259543648, -2.1722910184419537, 1.4296430191581153, -1.8222909807800027, 1.8707006753684063, -1.2151933754380138, 1.6665754229566851, 2.8650108672962737, 3.7636775387192176, -9.164944197895803, -1.8006661371120616, -3.3613146274583148, -5.421546985670076]\n", "iter:597\n", "w: [3.668375861448594, 0.0370788115932733, 1.5807396281897281, -4.300139824946333, 1.7172302181766177, 5.125778704393379, -0.13615965270568456, -2.17354961702327, 1.4304791691231873, -1.8233431395590463, 1.8716631291201515, -1.2159311981477223, 1.6674860406120147, 2.866587781986372, 3.7658788518729325, -9.170411401884556, -1.8017887998369788, -3.3631983169745743, -5.424871689638179]\n", "iter:598\n", "w: [3.6705569456626415, 0.037041587871795276, 1.58166548017274, -4.30268053097574, 1.718338116980974, 5.128624126270507, -0.13612311216367956, -2.1748063942048383, 1.431314191820927, -1.82439393926318, 1.872624230023661, -1.2166678842463403, 1.6683953769688604, 2.8681626605764805, 3.7680767813903984, -9.175870919127545, -1.8029099117124057, -3.3650794137712334, -5.4281918533958455]\n", "iter:599\n", "w: [3.6727350864829926, 0.037004506519153275, 1.5825900096073786, -4.305217779954379, 1.7194445043124953, 5.131465770785648, -0.1360867102683441, -2.1760613554180557, 1.432148090345963, -1.825443383390353, 1.873583982010074, -1.217403437381739, 1.6693034357361745, 2.8697355083071425, 3.77027133813917, -9.181322771832082, -1.8040294771059324, -3.3669579252352198, -5.431507489540825]\n", "iter:600\n", "w: [3.6749102920173886, 0.03696756681340192, 1.5835132204123037, -4.307751581424239, 1.7205493843791724, 5.134303648101884, -0.13605044632296692, -2.177314506069363, 1.432980867779762, -1.8264914754246957, 1.8745423889927149, -1.2181378611838327, 1.6702102206062774, 2.871306330398191, 3.772462532933168, -9.186766982106175, -1.8051475003661486, -3.368833858720567, -5.4348186106166985]\n", "iter:601\n", "w: [3.6770825703388024, 0.036930768037134735, 1.5844351164881914, -4.310281944886651, 1.7216527613708743, 5.137137768339863, -0.13601431963517105, -2.1785658515403976, 1.433812527190705, -1.827538218836596, 1.8754994548672066, -1.218871159264696, 1.6711157352549615, 2.8728751320488617, 3.774650376533029, -9.192203571959139, -1.806263985822756, -3.370707221548617, -5.438125229113202]\n", "iter:602\n", "w: [3.6792519294856447, 0.036894109477450726, 1.585355701717847, -4.312808879802529, 1.7227546394594555, 5.139968141578041, -0.13597832951688163, -2.17981539718815, 1.4346430716341645, -1.8285836170827758, 1.876455183511579, -1.2196033352186817, 1.6720199833415905, 2.874441918437906, 3.7768348796464615, -9.197632563302202, -1.8073789377866822, -3.372578021008222, -5.441427357466544]\n", "iter:603\n", "w: [3.681418377461969, 0.03685759042592019, 1.5862749799663163, -4.315332395592604, 1.7238550227988623, 5.142794777852932, -0.1359424752842942, -2.1810631483451157, 1.4354725041525802, -1.8296276736063641, 1.877409578786379, -1.2203343926225378, 1.6729229685092013, 2.876006694723705, 3.7790160529285917, -9.203053977949097, -1.8084923605501908, -3.374446264355941, -5.444725008059727]\n", "iter:604\n", "w: [3.6835819222376767, 0.0368212101785516, 1.5871929550809956, -4.31785250163766, 1.7249539155252382, 5.145617687159345, -0.13590675625784335, -2.182309110319446, 1.4363008277755351, -1.8306703918369729, 1.8783626445347785, -1.221064335035522, 1.673824694384602, 2.877569466044378, 3.7811939069823093, -9.20846783761666, -1.8096042583869931, -3.376311958816239, -5.4480181932228575]\n", "iter:605\n", "w: [3.685742571748718, 0.03678496803575878, 1.5881096308917408, -4.320369207278766, 1.726051321757029, 5.148436879450632, -0.13587117176217142, -2.1835532883950997, 1.4371280455198296, -1.8317117751907703, 1.8793143845826834, -1.221793165999516, 1.6747251645784722, 2.8791302375178964, 3.7833684523586126, -9.21387416392541, -1.8107146355523573, -3.378175111581682, -5.45130692523346]\n", "iter:606\n", "w: [3.6879003338972938, 0.036748863302327676, 1.5890250112109772, -4.322882521817509, 1.727147245595086, 5.15125236463892, -0.13583572112609782, -2.1847956878319916, 1.4379541603895567, -1.832751827070554, 1.880264802738838, -1.2225208890391401, 1.6756243826854602, 2.880689014242192, 3.7855396995569466, -9.219272978400143, -1.811823496283218, -3.380035729813131, -5.454591216316789]\n", "iter:607\n", "w: [3.6900552165520537, 0.03671289528738444, 1.5899390998338054, -4.325392454516223, 1.7282416911227698, 5.154064152595355, -0.1358004036825881, -2.186036313866142, 1.4387791753761754, -1.8337905508658243, 1.881213902794933, -1.2232475076618647, 1.6765223522842805, 2.882245801295267, 3.7877076590255414, -9.2246643024705, -1.8129308447982841, -3.3818938206399367, -5.457871078646133]\n", "iter:608\n", "w: [3.6922072275482947, 0.03667706330436264, 1.5908519005381103, -4.327899014598214, 1.7293346624060522, 5.156872253150331, -0.1357652187687239, -2.1872751717098224, 1.4396030934585837, -1.834827949952856, 1.8821616885257098, -1.2239730253581225, 1.6774190769378103, 2.8838006037353012, 3.7898723411617468, -9.230048157471552, -1.8140366852981464, -3.3837493911601273, -5.461146524343124]\n", "iter:609\n", "w: [3.6943563746881574, 0.036641366670971745, 1.5917634170846657, -4.330402211247989, 1.7304261634936176, 5.1596766760937305, -0.1357301657256725, -2.1885122665517014, 1.440425917603192, -1.8358640276947698, 1.883108163689064, -1.2246974456014192, 1.6783145601931855, 2.8853534266007603, 3.7920337563123643, -9.235424564644362, -1.8151410219653838, -3.385602448440602, -5.464417565478037]\n", "iter:610\n", "w: [3.6965026657408195, 0.036605804709165427, 1.5926736532172403, -4.332902053611482, 1.7315161984169645, 5.1624774311751525, -0.13569524389865714, -2.1897476035569907, 1.4412476507639944, -1.8368987874416047, 1.8840533320261503, -1.2254207718484438, 1.6792088055818954, 2.8869042749105036, 3.794191914773976, -9.240793545136553, -1.8162438589646699, -3.387452999517317, -5.4676842140700925]\n", "iter:611\n", "w: [3.69864610844269, 0.03657037674510996, 1.5935826126627015, -4.335398550796273, 1.7326047711905053, 5.165274528104144, -0.1356604526369273, -2.190981187867587, 1.4420682958826414, -1.8379322325303877, 1.884997197261484, -1.2261430075391766, 1.6801018166198776, 2.888453153663888, 3.7963468267932727, -9.246155120002875, -1.817345200442877, -3.389301051395473, -5.470946482087754]\n", "iter:612\n", "w: [3.7007867104976007, 0.036535082109153155, 1.5944902991311198, -4.337891711871811, 1.7336918858116652, 5.168067976550429, -0.1356257912937294, -2.192213024602217, 1.4428878558885103, -1.8389643662852044, 1.8859397631030432, -1.2268641560969986, 1.6809935968076106, 2.8900000678408757, 3.798498502567375, -9.251509310205755, -1.8184450505291807, -3.3911466110497, -5.4742043814490255]\n", "iter:613\n", "w: [3.702924479576996, 0.03649992013579375, 1.5953967163158713, -4.340381545869633, 1.7347775462609811, 5.170857786144136, -0.1355912592262776, -2.1934431188565764, 1.4437063336987759, -1.8399951920172686, 1.8868810332423698, -1.2275842209287977, 1.6818841496302084, 2.891545022402137, 3.8006469522441586, -9.256856136615852, -1.8195434133351638, -3.3929896854242436, -5.477457924021743]\n", "iter:614\n", "w: [3.7050594233201233, 0.03646489016365058, 1.5963018678937393, -4.342868061783582, 1.7358617565021996, 5.173643966476022, -0.13555685579572496, -2.1946714757034713, 1.4445237322184803, -1.8410247130249915, 1.8878210113546696, -1.2283032054250758, 1.6827734785575106, 2.893088022289154, 3.802792185922569, -9.262195620012607, -1.8206402929549184, -3.3948302814331446, -5.480707121623869]\n", "iter:615\n", "w: [3.707191549334218, 0.036429991535432284, 1.5972057575250167, -4.345351268570023, 1.7369445204823741, 5.176426527097697, -0.1355225803671348, -2.1958981001929576, 1.4453400543406034, -1.8420529325940502, 1.8887597010989121, -1.2290211129600541, 1.6836615870441756, 2.894629072424326, 3.8049342136529405, -9.26752778108479, -1.8217356934651479, -3.3966684059604217, -5.48395198602378]\n", "iter:616\n", "w: [3.7093208651946914, 0.036395223597907235, 1.5981083888536054, -4.34783117514806, 1.738025842131961, 5.179205477521846, -0.13548843230945232, -2.1971229973524795, 1.4461553029461307, -1.8430798539974569, 1.8896971061179286, -1.229737946891778, 1.6845484785297704, 2.8961681777110693, 3.807073045437309, -9.272852640431033, -1.822829618925268, -3.3985040658602514, -5.487192528940556]\n", "iter:617\n", "w: [3.7114473784453144, 0.03636058570187384, 1.5990097655071167, -4.350307790399744, 1.7391057253649154, 5.181980827222447, -0.13545441099547637, -2.198346172187006, 1.4469694809041225, -1.8441054804956254, 1.890633230038511, -1.2304537105622204, 1.6854341564388615, 2.8977053430339197, 3.809208691229723, -9.278170218560376, -1.8239220733775077, -3.4003372679571453, -5.490428762044262]\n", "iter:618\n", "w: [3.7135710965984012, 0.03632607720213074, 1.5999098910969698, -4.35278112317029, 1.7401841740787864, 5.184752585634996, -0.13542051580183156, -2.199567629679167, 1.447782591071782, -1.8451298153364393, 1.8915680764715077, -1.2311684072973854, 1.6863186241811035, 2.8992405732586333, 3.811341160936554, -9.283480535892794, -1.8250130608470083, -3.402168019046129, -5.493660696956235]\n", "iter:619\n", "w: [3.715692027134991, 0.0362916974574479, 1.60080876921849, -4.355251182268283, 1.7412611921548113, 5.187520762156717, -0.13538674610894055, -2.2007873747893894, 1.4485946362945226, -1.8461528617553182, 1.8925016490119213, -1.2318820404074105, 1.6872018851513282, 2.9007738732322874, 3.8134704644167994, -9.288783612759723, -1.8261025853419228, -3.403996325892916, -5.496888345249364]\n", "iter:620\n", "w: [3.7178101775050276, 0.03625744583053696, 1.601706403451007, -4.3577179764658895, 1.7423367834580088, 5.1902853661467825, -0.1353531013009966, -2.20200541245603, 1.4494056194060347, -1.8471746229752846, 1.8934339512390033, -1.2325946131866672, 1.688083942729633, 2.9023052477833793, 3.8155966114823894, -9.294079469404588, -1.8271906508535136, -3.405822195234083, -5.500111718448367]\n", "iter:621\n", "w: [3.7199255551275403, 0.03622332168802295, 1.6026027973579502, -4.360181514499061, 1.7434109518372727, 5.1930464069265225, -0.13531958076593636, -2.203221747595509, 1.450215543228353, -1.8481951022070289, 1.8943649867163495, -1.233306128913863, 1.6889648002814677, 2.903834701721925, 3.817719611898487, -9.299368125983326, -1.8282772613562501, -3.4076456337772423, -5.5033308280300695]\n", "iter:622\n", "w: [3.7220381673908203, 0.036189324400415515, 1.6034979544869463, -4.362641805067742, 1.7444837011254635, 5.195803893779644, -0.13528618389541286, -2.204436385102443, 1.4510244105719217, -1.849214302648976, 1.895294758991994, -1.2340165908521399, 1.6898444611577217, 2.9053622398395573, 3.819839475383787, -9.304649602564895, -1.8293624208079056, -3.409466648201214, -5.506545685423679]\n", "iter:623\n", "w: [3.7241480216525975, 0.03615545334208041, 1.604391878369913, -4.365098856836075, 1.7455550351395, 5.198557835952434, -0.13525291008476856, -2.205649329849774, 1.4518322242356603, -1.8502322274873493, 1.8962232715985028, -1.2347260022491742, 1.6907229286948098, 2.9068878669096248, 3.821956211610813, -9.309923919131794, -1.8304461331496533, -3.411285245156197, -5.5097563020110565]\n", "iter:624\n", "w: [3.7262551252402156, 0.03612170789121178, 1.6052845725231544, -4.367552678432597, 1.746624957680451, 5.201308242653974, -0.13521975873300893, -2.206860586688901, 1.4526389870070293, -1.8512488798962357, 1.8971505280530665, -1.2354343663372744, 1.691600206214758, 2.9084115876872856, 3.8240698302062115, -9.315191095580566, -1.8315284023061618, -3.4131014312639363, -5.51296268912699]\n", "iter:625\n", "w: [3.728359485450807, 0.03608808742980385, 1.606176040447455, -4.370003278450448, 1.7476934725336242, 5.204055123056348, -0.13518672924277622, -2.2080701604498083, 1.4534447016620937, -1.8522642630376491, 1.8980765318575925, -1.2361416863334789, 1.692476297025288, 2.909933406909606, 3.8261803407510437, -9.320451151722308, -1.8326092321856893, -3.414915213117893, -5.516164858059461]\n", "iter:626\n", "w: [3.730461109551463, 0.03605459134362371, 1.6070662856281719, -4.372450665447566, 1.7487605834686573, 5.2067984862948435, -0.1351538210203232, -2.209278055941195, 1.4542493709655884, -1.853278380061595, 1.8990012864987957, -1.2368479654396525, 1.6933512044199015, 2.911453329295655, 3.8282877527810752, -9.325704107283169, -1.8336886266801775, -3.4167265972834087, -5.519362820049911]\n", "iter:627\n", "w: [3.732560004779408, 0.036021219022183616, 1.607955311535328, -4.374894847946886, 1.7498262942396055, 5.209538341468163, -0.13512103347548715, -2.2104842779506004, 1.455052997670981, -1.8542912341061326, 1.89992479544829, -1.2375532068425823, 1.6942249316779638, 2.9129713595465985, 3.830392075787063, -9.33094998190485, -1.8347665896653456, -3.4185355902978727, -5.522556586293508]\n", "iter:628\n", "w: [3.734656178342167, 0.035987969858714004, 1.6088431216237036, -4.377335834436536, 1.7508906085850306, 5.212274697638623, -0.13508836602166438, -2.2116888312445306, 1.4558555845205352, -1.8553028282974382, 1.900847062162678, -1.2382574137140734, 1.6950974820647868, 2.914487502345794, 3.8324933192150414, -9.336188795145095, -1.835843125000782, -3.420342198670885, -5.525746167939408]\n", "iter:629\n", "w: [3.7367496374177365, 0.03595484325013655, 1.6097297193329267, -4.379773633370035, 1.751953530228089, 5.215007563832356, -0.13505581807578457, -2.2128917205685843, 1.456657134245374, -1.8563131657498673, 1.9017680900836396, -1.2389605892110427, 1.695968858831711, 2.9160017623588845, 3.8345914924666036, -9.341420566478181, -1.836918236530037, -3.4221464288844192, -5.528931576091016]\n", "iter:630\n", "w: [3.738840389154751, 0.03592183859703735, 1.6106151080875648, -4.382208253166481, 1.753015062876618, 5.217736949039512, -0.13502338905828556, -2.214092950647577, 1.4574576495655414, -1.8573222495660162, 1.9026878826380214, -1.2396627364756128, 1.6968390652161873, 2.91751414423389, 3.836686604899184, -9.34664531529541, -1.8379919280807129, -3.423948287392985, -5.532112821806245]\n", "iter:631\n", "w: [3.7409284406726484, 0.03588895530364045, 1.6114992912972126, -4.384639702210751, 1.754075210223222, 5.220462862214459, -0.13499107839308847, -2.215292526185664, 1.4582571331900647, -1.8583300828367841, 1.9036064432379243, -1.2403638586352053, 1.6977081044418583, 2.9190246526013, 3.8387786658263345, -9.351863060905584, -1.8390642034645557, -3.4257477806237877, -5.535289916097775]\n", "iter:632\n", "w: [3.743013799061837, 0.035856192777781716, 1.6123822723565824, -4.387067988853685, 1.7551339759453592, 5.223185312275977, -0.13495888550757248, -2.216490451866463, 1.4590555878170157, -1.8593366686414334, 1.9045237752807906, -1.2410639588026324, 1.6985759797186384, 2.920533292074167, 3.840867684518003, -9.357073822535492, -1.8401350664775447, -3.4275449149768873, -5.538462869933305]\n", "iter:633\n", "w: [3.7450964713838584, 0.03582355043088253, 1.613264054645592, -4.389493121412281, 1.7561913637054265, 5.225904308107455, -0.1349268098325505, -2.217686732353176, 1.459853016133572, -1.8603420100476509, 1.9054398821494913, -1.241763040076189, 1.6994426942427945, 2.9220400672481937, 3.8429536702008056, -9.36227761933038, -1.8412045208999814, -3.429339696825357, -5.541631694235807]\n", "iter:634\n", "w: [3.7471764646715493, 0.035791027677924324, 1.6141446415294525, -4.391915108169879, 1.7572473771508432, 5.228619858557088, -0.13489485080224456, -2.2188813722887084, 1.4606494208160772, -1.8613461101116073, 1.906354767212411, -1.2424611055397434, 1.7003082511970242, 2.9235449827018267, 3.845036632058299, -9.367474470354423, -1.8422725704965786, -3.4311321325154402, -5.544796399883781]\n", "iter:635\n", "w: [3.7492537859292057, 0.035758623937422546, 1.615024036358755, -4.394333957376353, 1.758302019914136, 5.231331972438068, -0.13486300785426145, -2.2200743762957913, 1.461444804530101, -1.8623489718780177, 1.907268433823534, -1.2431581582628273, 1.7011726537505354, 2.925048042996344, 3.8471165792312507, -9.372664394591204, -1.8433392190165476, -3.432922228366706, -5.547956997711501]\n", "iter:636\n", "w: [3.7513284421327397, 0.03572633863140158, 1.615902242469557, -4.39674967724829, 1.7593552956130216, 5.234040658528776, -0.134831280429569, -2.2212657489770984, 1.462239169930499, -1.8633505983802006, 1.9081808853225282, -1.2438542013007248, 1.7020359050591238, 2.926549252675945, 3.849193520817907, -9.377847410944165, -1.844404470193687, -3.434709990672203, -5.551113498509264]\n", "iter:637\n", "w: [3.753400440229842, 0.0356941711853695, 1.6167792631834685, -4.399162275969181, 1.76040720785049, 5.236745925572973, -0.13479966797247195, -2.2224554949153648, 1.4630325196614726, -1.8643509926401367, 1.90909212503483, -1.2445492376945622, 1.702898008265251, 2.928048616267839, 3.851267465874258, -9.383023538237076, -1.8454683277464679, -3.436495425698614, -5.554265913023636]\n", "iter:638\n", "w: [3.7554697871401372, 0.03566212102829286, 1.6176551018077365, -4.401571761689599, 1.7614577602148862, 5.23944778227999, -0.1347681699305884, -2.223643618673503, 1.4638248563566265, -1.865350157668528, 1.9100021562717273, -1.2452432704713947, 1.7037589664981212, 2.929546138282332, 3.8533384234143, -9.388192795214495, -1.8465307953781203, -3.438278539686407, -5.557414251957698]\n", "iter:639\n", "w: [3.7575364897553425, 0.03563018759257218, 1.6185297616353298, -4.403978142527383, 1.7625069562799922, 5.242146237324915, -0.13473678575482628, -2.2248301247947215, 1.4646161826390285, -1.8663480964648558, 1.910910982330442, -1.2459363026442951, 1.7046187828737576, 2.931041823212915, 3.8554064024103, -9.393355200542224, -1.8475918767767194, -3.4400593388499874, -5.560558525971285]\n", "iter:640\n", "w: [3.7596005549394227, 0.03559837031401738, 1.6194032459450227, -4.406381426567819, 1.7635547996051075, 5.244841299348778, -0.1347055148993601, -2.226015017802637, 1.4654065011212665, -1.8673448120174376, 1.9118186064942124, -1.2466283372124394, 1.7054774604950782, 2.9325356755363505, 3.8574714117930524, -9.398510772807755, -1.8486515756152702, -3.441837829377847, -5.563698745681229]\n", "iter:641\n", "w: [3.761661989528745, 0.03556666863182311, 1.6202755580014783, -4.408781621863817, 1.7646012937351299, 5.247532976958738, -0.134674356821608, -2.227198302201391, 1.4661958144055065, -1.8683403073034859, 1.9127250320323748, -1.247319377161193, 1.7063350024519717, 2.934027699712758, 3.8595334604521376, -9.403659530520724, -1.8497098955517906, -3.4436140174327132, -5.566834921661601]\n", "iter:642\n", "w: [3.763720800332232, 0.035535081988544895, 1.6211467010553309, -4.41117873643609, 1.7656464422006353, 5.250221278728269, -0.13464331098220864, -2.2283799824757637, 1.4669841250835498, -1.869334585289164, 1.913630262200444, -1.2480094254621967, 1.7071914118213716, 2.9355179001856997, 3.861592557236178, -9.408801492113353, -1.8507668402293969, -3.4453879091516977, -5.569967064443943]\n", "iter:643\n", "w: [3.7657769941315156, 0.03550360983007511, 1.6220166783432681, -4.413572778273333, 1.7666902485179565, 5.2529062131973365, -0.13461237684499858, -2.2295600630912853, 1.46777143573689, -1.8703276489296436, 1.9145343002401944, -1.2486984850734506, 1.7080466916673305, 2.937006281382266, 3.86364871095309, -9.413936675940896, -1.8518224132763854, -3.447159510646442, -5.573095184517505]\n", "iter:644\n", "w: [3.767830577681085, 0.03547225160561903, 1.6228854930881123, -4.415963755332393, 1.7677327161892629, 5.255587788872584, -0.13458155387698978, -2.230738548494349, 1.4685577489367692, -1.8713195011691601, 1.9154371493797389, -1.2493865589393984, 1.708900845041094, 2.938492847713159, 3.8657019303703377, -9.419065100282072, -1.8528766183063157, -3.4489288280032637, -5.576219292329482]\n", "iter:645\n", "w: [3.769881557708442, 0.03544100676767151, 1.623753148498901, -4.418351675538446, 1.7687738487026377, 5.2582660142275115, -0.1345508415483473, -2.2319154431123227, 1.4693430672442334, -1.8723101449410693, 1.9163388128336085, -1.2500736499910112, 1.7097538749811738, 2.939977603572776, 3.86775222421518, -9.42418678333951, -1.8539294589180921, -3.4506958672833004, -5.579339398285246]\n", "iter:646\n", "w: [3.771929940914245, 0.035409874771993324, 1.6246196477709676, -4.420736546785173, 1.769813649532156, 5.260940897702653, -0.13452023933236687, -2.2330907513536578, 1.4701273932101893, -1.873299583167902, 1.9172392938028309, -1.2507597611458696, 1.7106057845134204, 2.9414605533392946, 3.86979960117492, -9.429301743240169, -1.8549809386960452, -3.4524606345226543, -5.582455512748571]\n", "iter:647\n", "w: [3.7739757339724602, 0.035378855077588284, 1.6254849940860203, -4.4231183769349265, 1.7708521221379614, 5.263612447705756, -0.13448974670545336, -2.2342644776080003, 1.4709107293754586, -1.87428781876142, 1.918138595475008, -1.2514448953082462, 1.7114565766510947, 2.9429417013747523, 3.871844069897151, -9.43440999803578, -1.8560310612100126, -3.454223135732534, -5.585567646041869]\n", "iter:648\n", "w: [3.776018943530508, 0.03534794714668008, 1.626349190612222, -4.425497173818904, 1.7718892699663435, 5.266280672611956, -0.13445936314709858, -2.2354366262462992, 1.471693078270833, -1.8752748546226703, 1.9190367210243942, -1.2521290553691873, 1.7123062543949403, 2.944421052025132, 3.8738856389899996, -9.439511565703269, -1.8570798300154192, -3.455983376899396, -5.588675808446414]\n", "iter:649\n", "w: [3.7780595762094085, 0.03531715044468935, 1.627212240504268, -4.427872945237317, 1.7729250964498129, 5.268945580763953, -0.1344290881398599, -2.2366072016209135, 1.472474442417129, -1.8762606936420398, 1.919933673611972, -1.252812244206593, 1.713154820733254, 2.9458986096204405, 3.875924317022368, -9.444606464145169, -1.8581272486533562, -3.4577413639850847, -5.591780010202568]\n", "iter:650\n", "w: [3.7800976386039267, 0.035286464440211295, 1.6280741469034636, -4.430245698959556, 1.7739596050071769, 5.271607180472183, -0.1343989211693386, -2.237776208065721, 1.4732548243252415, -1.8772453386993089, 1.9208294563855295, -1.2534944646852983, 1.7140022786419566, 2.947374378474792, 3.877960112524174, -9.449694711190057, -1.8591733206506609, -3.459497102926973, -5.5948802615100055]\n", "iter:651\n", "w: [3.7821331372827163, 0.035255888604993096, 1.6289349129378028, -4.432615442724365, 1.774992799043615, 5.274265480014995, -0.13436886172415882, -2.2389436498962225, 1.4740342264961976, -1.8782287926637051, 1.9217240724797349, -1.2541757196571526, 1.7148486310846627, 2.948848362886486, 3.87999303398659, -9.454776324592963, -1.8602180495199947, -3.4612505996380993, -5.597976572527937]\n", "iter:652\n", "w: [3.784166078788463, 0.03522542241391181, 1.6297945417220432, -4.434982184239998, 1.7760246819507524, 5.276920487638815, -0.13433890929594627, -2.240109531409651, 1.4748126514212103, -1.8792110583939572, 1.9226175250162125, -1.254856011961098, 1.7156938810127504, 2.95032056713809, 3.8820230898622787, -9.459851322035783, -1.8612614387599213, -3.4630018600073056, -5.601068953375329]\n", "iter:653\n", "w: [3.7861964696380257, 0.03519506534495193, 1.6306530363577834, -4.4373459311843915, 1.7770552571067344, 5.2795722115583255, -0.13430906337930718, -2.2412738568850723, 1.475590101581731, -1.8801921387383473, 1.9235098171036173, -1.255535344423249, 1.7165380313654304, 2.9517909954965162, 3.8840502885656294, -9.464919721127695, -1.862303491854984, -3.4647508898993737, -5.604157414131121]\n", "iter:654\n", "w: [3.788224316322578, 0.03516481687918403, 1.6315103999335383, -4.4397066912053225, 1.7780845278763, 5.282220659956626, -0.13427932347180802, -2.2424366305834926, 1.4763665794495022, -1.881172036534764, 1.9244009518377085, -1.2562137198569692, 1.7173810850698144, 2.953259652213103, 3.886074638472989, -9.46998153940556, -1.8633442122757817, -3.466497695155161, -5.607241964834452]\n", "iter:655\n", "w: [3.7902496253077476, 0.03513467650074252, 1.632366635524814, -4.442064471920573, 1.7791124976108543, 5.284865840985406, -0.13424968907395426, -2.24359785674796, 1.47714208748661, -1.8821507546107539, 1.9252909323014236, -1.256891141062949, 1.7182230450409828, 2.954726541523691, 3.888096147922895, -9.475036794334338, -1.8643836034790475, -3.468242281591734, -5.610322615484865]\n", "iter:656\n", "w: [3.792272403033756, 0.03510464369680462, 1.6332217461941831, -4.444419280918093, 1.7801391696485411, 5.28750776276511, -0.13422015968917048, -2.2447575396036674, 1.4779166281455351, -1.8831282957835744, 1.9261797615649507, -1.2575676108292828, 1.7190639141820532, 2.956191667648703, 3.890114825216304, -9.48008550330748, -1.8654216689077219, -3.4699846550025026, -5.613399376042532]\n", "iter:657\n", "w: [3.7942926559155548, 0.0350747179575686, 1.6340757349913573, -4.4467711257561575, 1.7811645473143147, 5.290146433385105, -0.1341907348237797, -2.245915683358055, 1.4786902038692056, -1.884104662860245, 1.9270674426858019, -1.2582431319315444, 1.7199036953842466, 2.957655034793219, 3.892130678616819, -9.485127683647336, -1.8664584119910295, -3.471724821157353, -5.616472256428461]\n", "iter:658\n", "w: [3.7963103903429647, 0.03504489877623256, 1.6349286049532616, -4.449120013963527, 1.7821886339200113, 5.292781860903839, -0.13416141398698345, -2.247072292200911, 1.4794628170910469, -1.8850798586375979, 1.927953978708884, -1.2589177071328623, 1.7207423915269542, 2.9591166471470545, 3.894143716350917, -9.490163352605547, -1.8674938361445532, -3.4734627858027776, -5.6195412665247115]\n", "iter:659\n", "w: [3.7983256126808103, 0.03501518564897388, 1.6357803591041078, -4.451465953039606, 1.7832114327644195, 5.2954140533490115, -0.13413219669084198, -2.2482273703044724, 1.4802344702350332, -1.8860538859023297, 1.928839372666571, -1.259591339183996, 1.7215800054778039, 2.960576508884837, 3.896153946608172, -9.49519252736344, -1.8685279447703087, -3.475198554662007, -5.622606416174604]\n", "iter:660\n", "w: [3.800338329269055, 0.034985578074927866, 1.636631000455466, -4.4538089504546, 1.7842329471333513, 5.298043018717732, -0.13410308245025404, -2.2493809218235237, 1.4810051657157377, -1.8870267474310518, 1.9297236275787744, -1.2602640308234088, 1.7224165400927252, 2.962034624166081, 3.898161377541478, -9.500215225032418, -1.869560741256818, -3.476932133435139, -5.625667715182929]\n", "iter:661\n", "w: [3.802348546422936, 0.03495607555616735, 1.637480532006337, -4.456149013649668, 1.7852531802997116, 5.300668764976682, -0.1340740707829377, -2.250532950895497, 1.481774905938383, -1.8879984459903405, 1.9306067464530139, -1.2609357847773428, 1.7232519982160153, 2.9634909971352643, 3.9001660172672703, -9.505231462654356, -1.8705922289791828, -3.4786635277992684, -5.628725173316152]\n", "iter:662\n", "w: [3.8043562704330967, 0.0349266775976822, 1.6383289567432233, -4.458486150037083, 1.7862721355235676, 5.3032913000622735, -0.13404516120941062, -2.2516834616405705, 1.4825436932988902, -1.8889689843367865, 1.9314887322844871, -1.2616066037598919, 1.724086382680403, 2.964945631921901, 3.902167873865744, -9.510241257201976, -1.8716224112991564, -3.480392743408613, -5.631778800302624]\n", "iter:663\n", "w: [3.806361507565719, 0.0348973837073587, 1.6391762776402004, -4.46082036700038, 1.7872898160522181, 5.305910631880809, -0.13401635325297073, -2.2528324581617647, 1.4833115301839292, -1.8899383652170454, 1.9323695880561387, -1.2622764904730739, 1.7249196963071136, 2.966398532640618, 3.904166955381073, -9.515244625579237, -1.8726512915652174, -3.4821197858946427, -5.634828605832783]\n", "iter:664\n", "w: [3.8083642640626554, 0.03486819339595951, 1.6400224976589866, -4.463151671894514, 1.7883062251202617, 5.308526768308639, -0.13398764643967714, -2.2539799445450406, 1.4840784189709668, -1.890906591367886, 1.933249316738729, -1.2629454476069037, 1.7257519419059324, 2.967849703391228, 3.9061632698216244, -9.520241584621713, -1.8736788731126397, -3.483844660866203, -5.637874599559359]\n" ] } ], "source": [ "maxent.loadData(dataset)\n", "maxent.train()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "predict: {'yes': 0.9999971802186581, 'no': 2.819781341881656e-06}\n" ] } ], "source": [ "print('predict:', maxent.predict(x))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第6章Logistic回归与最大熵模型-习题\n", "\n", "### 习题6.1\n", "  确认Logistic分布属于指数分布族。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "**第1步：** \n", "首先给出指数分布族的定义： \n", "对于随机变量$x$，在给定参数$\\eta$下，其概率分别满足如下形式：$$p(x|\\eta)=h(x)g(\\eta)\\exp(\\eta^Tu(x))$$我们称之为**指数分布族**。 \n", "其中： \n", "$x$：可以是标量或者向量，可以是离散值也可以是连续值 \n", "$\\eta$：自然参数 \n", "$g(\\eta)$：归一化系数 \n", "$h(x),u(x)$：$x$的某个函数 \n", "\n", "----\n", "\n", "**第2步：**证明伯努利分布属于指数分布族 \n", "伯努利分布：$\\varphi$是$y=1$的概率，即$P(Y=1)=\\varphi$ \n", "$\\begin{aligned}\n", "P(y|\\varphi) \n", "&= \\varphi^y (1-\\varphi)^{(1-y)} \\\\\n", "&= (1-\\varphi) \\varphi^y (1-\\varphi)^{(-y)} \\\\\n", "&= (1-\\varphi) (\\frac{\\varphi}{1-\\varphi})^y \\\\\n", "&= (1-\\varphi) \\exp\\left(y \\ln \\frac{\\varphi}{1-\\varphi} \\right) \\\\\n", "&= \\frac{1}{1+e^\\eta} \\exp (\\eta y)\n", "\\end{aligned}$ \n", "其中，$\\displaystyle \\eta=\\ln \\frac{\\varphi}{1-\\varphi} \\Leftrightarrow \\varphi = \\frac{1}{1+e^{-\\eta}}$ \n", "将$y$替换成$x$，可得$\\displaystyle P(x|\\eta) = \\frac{1}{1+e^\\eta} \\exp (\\eta x)$\n", "对比可知，伯努利分布属于指数分布族，其中$\\displaystyle h(x) = 1, g(\\eta)= \\frac{1}{1+e^\\eta}, u(x)=x$ \n", "\n", "----\n", "\n", "**第3步：** \n", "广义线性模型（GLM）必须满足三个假设：\n", "1. $y | x;\\theta \\sim ExponentialFamily(\\eta)$，即假设预测变量$y$在给定$x$，以$\\theta$为参数的条件概率下，属于以$\\eta$作为自然参数的指数分布族； \n", "2. 给定$x$，求解出以$x$为条件的$T(y)$的期望$E[T(y)|x]$，即算法输出为$h(x)=E[T(y)|x]$ \n", "3. 满足$\\eta=\\theta^T x$，即自然参数和输入特征向量$x$之间线性相关，关系由$\\theta$决定，仅当$\\eta$是实数时才有意义，若$\\eta$是一个向量，则$\\eta_i=\\theta_i^T x$\n", "\n", "----\n", "\n", "**第4步：**推导伯努利分布的GLM \n", "已知伯努利分布属于指数分布族，对给定的$x,\\eta$，求解期望：$$\\begin{aligned}\n", "h_{\\theta}(x) \n", "&= E[y|x;\\theta] \\\\\n", "&= 1 \\cdot p(y=1)+ 0 \\cdot p(y=0) \\\\\n", "&= \\varphi \\\\\n", "&= \\frac{1}{1+e^{-\\eta}} \\\\\n", "&= \\frac{1}{1+e^{-\\theta^T x}}\n", "\\end{aligned}$$可得到Logistic回归算法，故Logistic分布属于指数分布族，得证。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题6.2\n", "  写出Logistic回归模型学习的梯度下降算法。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "对于Logistic模型：$$P(Y=1 | x)=\\frac{\\exp (w \\cdot x+b)}{1+\\exp (w \\cdot x+b)} \\\\ P(Y=0 | x)=\\frac{1}{1+\\exp (w \\cdot x+b)}\n", "$$对数似然函数为：$\\displaystyle L(w)=\\sum_{i=1}^N \\left[y_i (w \\cdot x_i)-\\log \\left(1+\\exp (w \\cdot x_i)\\right)\\right]$ \n", "似然函数求偏导，可得$\\displaystyle \\frac{\\partial L(w)}{\\partial w^{(j)}}=\\sum_{i=1}^N\\left[x_i^{(j)} \\cdot y_i-\\frac{\\exp (w \\cdot x_i) \\cdot x_i^{(j)}}{1+\\exp (w \\cdot x_i)}\\right]$ \n", "梯度函数为：$\\displaystyle \\nabla L(w)=\\left[\\frac{\\partial L(w)}{\\partial w^{(0)}}, \\cdots, \\frac{\\partial L(w)}{\\partial w^{(m)}}\\right]$ \n", "Logistic回归模型学习的梯度下降算法： \n", "(1) 取初始值$x^{(0)} \\in R$，置$k=0$ \n", "(2) 计算$f(x^{(k)})$ \n", "(3) 计算梯度$g_k=g(x^{(k)})$，当$\\|g_k\\| < \\varepsilon$时，停止迭代，令$x^* = x^{(k)}$；否则，求$\\lambda_k$，使得$\\displaystyle f(x^{(k)}+\\lambda_k g_k) = \\max_{\\lambda \\geqslant 0}f(x^{(k)}+\\lambda g_k)$ \n", "(4) 置$x^{(k+1)}=x^{(k)}+\\lambda_k g_k$，计算$f(x^{(k+1)})$，当$\\|f(x^{(k+1)}) - f(x^{(k)})\\| < \\varepsilon$或 $\\|x^{(k+1)} - x^{(k)}\\| < \\varepsilon$时，停止迭代，令$x^* = x^{(k+1)}$ \n", "(5) 否则，置$k=k+1$，转(3)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import numpy as np\n", "import time\n", "import matplotlib.pyplot as plt\n", "from mpl_toolkits.mplot3d import Axes3D\n", "from pylab import mpl\n", "\n", "# 图像显示中文\n", "mpl.rcParams['font.sans-serif'] = ['Microsoft YaHei']\n", "\n", "\n", "class LogisticRegression:\n", " def __init__(self, learn_rate=0.1, max_iter=10000, tol=1e-2):\n", " self.learn_rate = learn_rate # 学习率\n", " self.max_iter = max_iter # 迭代次数\n", " self.tol = tol # 迭代停止阈值\n", " self.w = None # 权重\n", "\n", " def preprocessing(self, X):\n", " \"\"\"将原始X末尾加上一列，该列数值全部为1\"\"\"\n", " row = X.shape[0]\n", " y = np.ones(row).reshape(row, 1)\n", " X_prepro = np.hstack((X, y))\n", " return X_prepro\n", "\n", " def sigmod(self, x):\n", " return 1 / (1 + np.exp(-x))\n", "\n", " def fit(self, X_train, y_train):\n", " X = self.preprocessing(X_train)\n", " y = y_train.T\n", " # 初始化权重w\n", " self.w = np.array([[0] * X.shape[1]], dtype=np.float)\n", " k = 0\n", " for loop in range(self.max_iter):\n", " # 计算梯度\n", " z = np.dot(X, self.w.T)\n", " grad = X * (y - self.sigmod(z))\n", " grad = grad.sum(axis=0)\n", " # 利用梯度的绝对值作为迭代中止的条件\n", " if (np.abs(grad) <= self.tol).all():\n", " break\n", " else:\n", " # 更新权重w 梯度上升——求极大值\n", " self.w += self.learn_rate * grad\n", " k += 1\n", " print(\"迭代次数：{}次\".format(k))\n", " print(\"最终梯度：{}\".format(grad))\n", " print(\"最终权重：{}\".format(self.w[0]))\n", "\n", " def predict(self, x):\n", " p = self.sigmod(np.dot(self.preprocessing(x), self.w.T))\n", " print(\"Y=1的概率被估计为：{:.2%}\".format(p[0][0])) # 调用score时，注释掉\n", " p[np.where(p > 0.5)] = 1\n", " p[np.where(p < 0.5)] = 0\n", " return p\n", "\n", " def score(self, X, y):\n", " y_c = self.predict(X)\n", " error_rate = np.sum(np.abs(y_c - y.T)) / y_c.shape[0]\n", " return 1 - error_rate\n", "\n", " def draw(self, X, y):\n", " # 分离正负实例点\n", " y = y[0]\n", " X_po = X[np.where(y == 1)]\n", " X_ne = X[np.where(y == 0)]\n", " # 绘制数据集散点图\n", " ax = plt.axes(projection='3d')\n", " x_1 = X_po[0, :]\n", " y_1 = X_po[1, :]\n", " z_1 = X_po[2, :]\n", " x_2 = X_ne[0, :]\n", " y_2 = X_ne[1, :]\n", " z_2 = X_ne[2, :]\n", " ax.scatter(x_1, y_1, z_1, c=\"r\", label=\"正实例\")\n", " ax.scatter(x_2, y_2, z_2, c=\"b\", label=\"负实例\")\n", " ax.legend(loc='best')\n", " # 绘制p=0.5的区分平面\n", " x = np.linspace(-3, 3, 3)\n", " y = np.linspace(-3, 3, 3)\n", " x_3, y_3 = np.meshgrid(x, y)\n", " a, b, c, d = self.w[0]\n", " z_3 = -(a * x_3 + b * y_3 + d) / c\n", " ax.plot_surface(x_3, y_3, z_3, alpha=0.5) # 调节透明度\n", " plt.show()" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "迭代次数：3232次\n", "最终梯度：[ 0.00144779 0.00046133 0.00490279 -0.00999848]\n", "最终权重：[ 2.96908597 1.60115396 5.04477438 -13.43744079]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADnCAYAAAC9roUQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOy9aXAc53ktfLqnZ5/BvgMEiY0ECZIiJa6Sd1uxPseS7DjRlePkWrEqFTsuO4oX2RXdVPTDN+XITuLEspJUIkfeYluO7SipxJKs3MiKFFIiZVLiAoLYgcGO2beeXr8fjbfRM9M90z0LCEhzqlgkgellgJ7TT5/3POehZFlGDTXUUEMNWwP6Rp9ADTXUUMObCTXSraGGGmrYQtRIt4YaaqhhC1Ej3RpqqKGGLUSNdGuooYYathBMke/XrA011FBDDdZBGX2jVunWUEMNNWwhaqRbQw011LCFqJFuDTXUUMMWoka6NdRQQw1biGILaTXUUEMNuuB5HoFAACzL3uhTuWFwuVzo6emB3W43vQ1VJHuh5l6ooYYadDE9PQ2/34/m5mZQlOFi/RsWsiwjGAwiHo+jr68v99s190INNdRQWbAs+6YlXACgKArNzc2WK/0a6dZQQw0l481KuASlvP8a6dZQQw01bCFqpFtDDTW8abCysoJQKGT4fUmSqn4ONfdCDTXUsCPx5JNP4pFHHin4mk996lP46Ec/qv7/+eefx3PPPYe///u/z3vt+fPn8e1vfxu//OUvceXKFYyMjODo0aP4+te/jh/84AdgWRb33Xdf2eddI90aaqhhR+Kee+7BPffck/W14eFhXLt2Tf3/o48+ipGREfj9fvVrPM/j1KlTAIBkMomPfOQj+OIXv4gf//jH+K3f+i088MAD+L//9//i8ccfRzKZxFe+8hXs2rWrYuddI90aaqhh67C2BszMAHv2AK2tZe8uEokgHA7rWbYAAE6nE1/96lextLSEoaEhvPWtb8Wzzz6Lzs5OHDp0CL/4xS9w4cIFCIKAH/zgB3jooYfwz//8zxgfH8fnPvc53H333bhy5UqNdGvYHpBlGZIkIZPJQBAEMAwDmqZhs9lA0zRomn7Tr27XoMH3vw/cfz/gcAAcBzz+OPDhD5e1y7Nnz+Lpp5/G1772Nd3v/+7v/i54nkcoFMIDDzyAixcvYm1tDceOHcOhQ4fw9re/HW9/+9vx+OOPIxgMAgD+5V/+BV/72tfwxBNPFNR/S0VtIa0Gy5BlGYIgIJPJgOM4SJIESZLAcRzS6TSSySTi8Tii0ShisRiSySRYlgXP8xBFEbVhqG9CrK0phJtOA9Go8vf99ytfryCWlpZw7Ngx9c/S0hJ+9KMf4b777sMDDzyAT3ziE4jH41lyA6BovXfddRcmJiYgCAJ8Ph/GxsZw9OjRip4fUKt0a7AAWZYhiiIEQYAsy6AoCjRNQ5Ik9d+5r5dlGTzPg+M4zM7Ooq2tDR6PR62IyZ9aZfwGx8yMUuGm05tfs9uVr1dAZiDo7OzE+fPns772m7/5mzhy5AjOnj2LkydPYnp6Gr29vVmveeyxx/CpT30Kzz77LL74xS/iM5/5DDweD+rq6ip2bgQ10q2hKEhlGwgEUFdXB6/Xm0eweqAoKotEOY4DAHVbQRDA83zWNjUyfoNizx5FUtCC55WvVxEPP/wwnn76afX/X/3qV7G4uIiPfOQj6tduvvlmPPbYYwCA3//934fP58OePXtw9913V+WcaqRbgyEI2QqCAACIRqNwuVzw+Xwl7Y+iKLVCJv/PPR6wSca5r7XZbKpuTMi5RsY7BK2tioZ7//1Khcvzyv8rWOXq4eGHH8bDDz8MALh69Sruv/9++P1+vP/978fHP/5xtLS05G3z+OOPg2VZ3H777QCAJ554At/97ndNFRpmUCPdGvIgSZIqIwCbFSuREnJRKeIrRsa50oYsy1nkyzCMWh3nVtk1bAN8+MPAe95TMffCXXfdhcXFRQDAsWPHAAA+n0/9NwDs2rULd911F37yk58gk8ngsccew+HDh/Hkk0/ife97H44cOYLPfvaz2LdvHwAlxOfpp5/G9773PQBAKBTC3XffjUwmg8cff7ys8yWopYzVoEKSJAiCAFEUAeTLA9evX0djYyNacz4soiiC5/milcDY2Bg6OjpQX19fkfMl164syxgdHcWuXbvg9XrVc6dpOouIa2RcWYyOjmL//v03+jQK4nvf+x7m5+fxgQ98AMPDw1nfk2UZTz31FFwuF+64446Sj2HwczC8yGqV7psc2sUuUsUaERNN02U5D0h1WiloK2NCsjabTT0GsbPlbkNep5UqamT8xoRWu80FRVH4wAc+sIVno6BGum9SEI+tIAhFyZaAoqiyetMrTbqFjqP9WwvyvkVRVBf2CLQLeDWZooZqoUa6bzJoyXZ6ehpOpxOdnZ2miGWrSLOaMCJRLRlrF/AART6hKEp1bdQcFTWUgxrpvkmg57EllatZ8thu8kIlUYiMV1dXkclk0NPTk/Wamr2thlJQI903OIwaGgDlcdqKXGAkL2QyGaRSKfh8voKLaduZdI1ACJRowAS5jR81Mq7BLGqk+wYF8dhqH5dzCdGqRptLmqlUCtPT04hGo3C73Uin05AkCW63G16vV/1DOtDIeb0RUKgyBmqNHzUYo0a6bzDkNjTokS0BTdN5xFAIxKebTCYxNTWFZDKJ/v5+DA8Pq5YxWZbBsiySySQSiQTW19dVMpYkCel0GqIo5pHxdkauxlsIZhs/tAiHw2hubobD4ag1fpQIs7+jlZUV2O12NDU16X5fkqSqX5M10n2DQJIkxONxUBQFu91uatXdqNnBCCzLYmVlBcvLy+jv71eHEmr3QVEU3G433G53VrePLMsYHx+HzWZDOp3G+vo6UqkUZFkuWBlvB1ghXSMUIuOZmRk0NDSAZVndxg+9yrhGyJt44YUX8Kd/+qdZ7b5GqIWY11A2tA0N09PTaGtrQ3Nzs6ltzZJuNBrF5OQk0uk0GhoaMDIyYvk8KYqCw+GAx+NBW1ub+nVZltVksmQymVUZbxcyrgTpGoEQbG5lm9uFl7tNrfFDwfnz5/E7v/M76OzsxFve8hYAwMWLF/H9738fd955Jx599FH8zd/8TS3EvIbyYNTQUMrCWCGNNRQKYWpqCjRNY2BgAOl0GqlUquzzzz0Hj8cDj8eT1emmR8bk2C6XK4+Mq41qk1nu/ovJFDu18aOSGeY//OEP8dBDD+Gpp57CwYMHAQDf+MY3cPz4cdx5550AaiHmNZSJYg0NVuUCvdfLsoxgMIipqSk4HA7s27dPrRJYlt2y5girZJxOp5HJZFBfX1/xyriala5VlNL4sbKygq6urhve+FHpDHOfz4fbb78d9957LxoaGrK+d9ttt8Fut+P555/fdiHmNdLdAcglW/Jhyf3AlEO6xI86PT0Nr9eLkZERNceAoFzLVyUsY0ZkfOXKFbS3t0OWZVOVsRUy3imOCyMSDQQC6OjoyHKykL/1NONqOCq0GeYkUvf++5X8m1Ir3l/91V/FuXPn8NWvfhV33HEHXn75ZVy5cgUf+9jHwLKsmqfwox/9CN/5znfw8MMP45ZbbsHnP/950yHmn/vc5/DTn/60nLeehxrpbmMYNTQYfSBKJd2lpSXMzMygvr4eN910E9xut+7rtwPpFto3iZ0sR6YwIuPtUumWCr33VagLr9L2tq3IMA+HwwgEAnlfr4WY11AUhGwXFxfR2tpa0PalhRXSlSQJq6urWFlZAcMwOHr0KFwuV8n7366kZFWm0LopfD4fPB6Ppa69nYRCXmMzjR/kdWRfhbAVGeaBQEBNyCOohZjXUBC5DQ2Tk5Nob283vT1N03kXXS5EUcTCwgLm5+fR0NCApqamvMg7I2znStcqzJLx2toaIpEIKIrC6uqqSsZerxdut3tbWdv0UMrP22zjh/ZJjGGYrO1y/65WhvkHPvABtLa2YmFhAd/4xjfQ39+P97///XjooYfwpS99CW95y1tqIeY15MNKQ0Mh2Gw2w2YHQRAwPz+PhYUFdHZ24sSJE+B5HmNjY6b3Xyh7wcxi03YiXSPokfH09DQ8Hg/8fn8WGROZwu12w+PxbEsyJn7fSkBvEY+iKAiCAJvNph7PaJv/9b+Ad7+bwswM0NdHlU24y8vLGB8fx5//+Z/j2rVr+Ku/+iu87W1vw+uvv47Pf/7zqK+vR2dnJ1544YVtFWJeI90bCKMJDaVCr9LleR5zc3NYWlpCd3c3Tp06pWYIWJ3MW6ht2Ox5b3fSNQJN07qVsSRJageeHhlrNeMbQcZb0WGlXW8w+j75vTc1KX8AQBTzK+PcfxfCmTNncOnSJXziE5/Arbfeqn798OHDeOaZZ/DjH/8Y3/3ud+F0OvGVr3wl64nuwx/+MO6991489dRTmJ6exr59+/DEE08AUBbeCJqamvDf//3fps7HLGqkewNQbEIDASE5sx8arebKcRxmZmawtraGXbt24fTp02o1krt/s9CrVDmOw/T0NEKhUBbB+Hw+tTOu0PY7AYWqeDNknEgkDMmYaMbVwlaRbiGYIePc15sh4w9+8IP44Ac/aHjcD33oQ/jQhz5U8LxqIeZvYFiZ0EBASNQK6XIch2vXriEUCqG3txenT58umL1g1e1APiSZTEYl297eXnR3d4NlWSQSCQSDQczNzYHjODAMoxJMJpPJI/6dgFJ8uoXImDSZJBIJrK6uIp1O49y5c1WpjEVR3JKfeSlPaGbJmGjHdrtd3a6Uyni7oEa6VQax5ayvr8Nut8PtdpuWEWw2G0RRzIoUNEI6ncbs7CwikQj279+Pffv2VTx7gaIo8DyP0dFRhMNh7NmzR9XCOI6Dw+HIs9jwPK8+esdiMbAsi9XVVdjtdlUDJX/MvM+dDpqm1ffb2toKURRx8eJFHD16NGsBj5AxUJ5MUe1KtxpPLrnXLXkyJNdHId2YfG+rbu6lvP83/lV+g5Db0LC+vq6SjFmYIUVt4ld7ezvsdjs6OztN79/sRcOyLCYnJxEOh9HV1YXh4WH1Yi+0D7vdjoaGBjQ0NMBms4HjOPT29oLjOCQSCSSTSSwtLSGZTEIURTidziyJwuPx3PDquJodacSOpiXj3O+XQ8bVJF2Hw4FQKASn01mV/WtRTDMmf8diMaRSKfT09FT9nEj3ZjGrZS5qpFthGDU0MAxT1M6VC1Lp6iEej2NychIcx6mJX8lkEvF43PT+zWis6XQaU1NTiMVi6OnpAc/zpkm90PEcDgeampqyIvZkWUYmk1EJZn5+HqlUCpIkqY0PNyL8ppqkW8xdUC4ZV/PcW1pasL6+jng8XlXiJbGgZp6Eksmkmri3FXC5XJYJvka6FUKhCQ1AYQI1gt42JPFLkiT09/dnkZYZn65ZpFIpTE1NIR6Po7+/HwcOHADHcVhZWSl5n8VInnSVuVyurKQ0vYxe0sjg8Xjg9XrBsizS6fS2i4UshlItXWbJOBwOqzfjSmvGDMOgvb0d8/PzOH78eMn7KYaVlRWwLJvXRaaHv/3bv0VdXR1+7/d+r2rnUy5qpFsmzExoABQCzU2FKgatvJCb+FVfX1/w9aUilUphcnJSDSgfGRnR1cxKQanbU5R+Rq+WYNbW1rCwsICZmRkAUMmYVMcul6vkim8r5IVKIZeM/X4/YrEY9uzZUxXNeCsW6rR6bjFEo1Hs3r27qudTLmqkWyKsNjSUUunSNI1QKITx8fG8xC+j15dKuslkUs3M7e/vR0tLi26gznayfGkJZn19Hb29vfD5fJAkSXUHRKNRLC4ugmXZrNcTQnY4HEVJr9ryQjVX34mmWy3NeKtI16x8EY1G8xLHthtqpGsRkiQhkUiA4zh4vV7LTgQzkGUZa2trWF5ehsfjwYEDB+Dz+YpuVwrpiqKI1157DSzLYmBgQJ0GoQervl697beCtGmahs/ny/uZiaKY9dgdCARUG5uWiL1eLxwOh7pdtSvdasohxfZfDhmTQaTVlnPMOngAIBaL1Uj3jQJtQ0MoFEIqlcLAwIDp7c2QrizLWF5exszMDOrq6tDZ2Ym6ujpThAtYI12yEMeyLA4cOICmpqaqt/He6OYIm82Gurq6PFubIAhZHWUzMzPgeR4Mw8Dn82VposQrWilsVaVrFWbIOJFIIBKJIJFI4Ny5c6ovWSvpVIKQrcgLsVhMV3rbTqiRbgEYNTTY7fa8ESrFUIh0Sbzi7Owsmpqa1MSvmZkZS5KEmcf/WCyGyclJCIKAgYEBpFIp0+N9KkGa20meIGAYBvX19XkfVp7nkUgkEIvFEA6HsbKyAkEQ4HA48mxtpXqMq13pVvrxP5eMI5EIVldXMTg4mEXGKysramVM9HWtTGHlRmNV021sbLT+xrYQNdLVQbEJDZVyImgTv1pbW3Hs2LGsx1qr43cKQet6GBgYKOnCLLci22mdQ3a7HY2NjfB4PKpeLMsyOI5TK+OFhQWkUinVY5xraytGeFtR6Va6OteCkLqZyjgej5dExtpAnWKoabo7DMT2JYpiwQkN5XpuBUFAIBDAwsICOjo6cOLECd0PRimOh1xEo1FMTEwAAAYGBnQvyHI/+MlkEouLi3C73fD5fIar3zdaXigV2nOmKApOpxNOp9PQY5xIJFQJKnfAZu7PZys03WoudImiaFoz1g4kJYudqVQK8Xgcy8vLYFkWQD4ZW6l0te3C2xU10oX1CQ02m60keYHneUxOTqqJXydPnix4MZXjRgiHw5iamgJFURgcHDTUucgxSvlgah0P7e3tSKfTWFtby6tktJ14O5F0geJVeiGPsdHkCkK+oigilUpZfuw2g62QL0qRVrSLnXpkTCrj5eVlxGIxXLx4sWhlvFOurTc16RLTPXlEMptja7XSJYlfsVgMXV1duolfeihFxhAEAefOnQPDMBgaGio6bqQUG1gqlcLExATS6TQGBgbQ1NQEnuezPgB6ti3S6itJkqFTYDuinCcBo7B08ti9tLSUNeKeoqgsYinXY7wVpFvJ/es5T86dO4dbbrklj4y1lfH6+joCgUBZRcRW4U1JulqP7ezsLOx2u6VWPrOVbm4Sl9frNdVVQ2ClwywUCqltwUePHi3o59XCig2MTN29dOlSlr1Mj5T0PjyEfDs7O/OcAtoAHPL3dvngVKOCIo/ddXV1YBgGezbm1pCqN5lM6nqMtZqxGY/xVsgLW5G9YGQDJDf3SCSC8+fPY3V1FcePHwfDMPjDP/xDfLicccMARkZGVInvbW97G775zW+WtT/gTUa6eg0Ndru9pKaFQkSVTqcxPT2NSCSiJnFRFIW5uTlLxym2kCbLskq2TqcTw8PDuHTpkmmLmZn3Amx2qaVSKdjtdpw4caLkKD+KonSdAtoAHO3iFBkeScjmRrX5blVHms1mg9/vz7tpCoKgkrFRdKY2x5ig0pVoLrYqOtIIhIxvu+029PX1IRAI4OmnnwbHcarMVQ4ymYy6JlIpvClIt9CEBoZhLC9WGX0AtYlffX192L9/f1kfViN5gaQbTU5Owu12ZzVPGFWeRihEuqlUCpeuXsPiahDNre04dtMRXPzlq2U9ahtVjUYBOHqZC0C+XlxNPe9GBt4QMAyj6zHmeV6VcXKfHLxer6olu1yuqkRnVpt0rbRJa50LDoejIrJVNX7vb2jSNTOhgWEYy4tiuYjH45iamgLLsoYttKUglxBlWcb6+jomJyfh9Xpx8ODBPIsOqY7NVjdEXoinWKxH4ghGE1haC+HaxDRWw1H46+rh8XgwGV7Aa9MrsPMJHLmZg8eVfUFXg5gKZS7kPoJHIhEkk0nU1dVVXC/ezm3Adrvd8MmBdN4RvTM3OpP8KYc0qy1fWPXoVrIxIplMYmVlBf39/di9ezceeeSRigT7vOFI1+qEhnJINxqNYmpqCqIo5iV+FTo/sx8yUumStuCpqSn4fD4cPnwYHo9HdxuiAxtdqFpyDcYSuHh5HM9dCgC0DYLAIxwOI5Ph0NjYiPaOTgCb55rhBUzMr+Ef/vV53DTUi+P7++BxmdfzKmUZ0+p7ZFry1atX0d3dDYqi1KrYSC8up5mh0qjWQhep9JxOJ4aGhuBwOFSPsVbGIVGIRMbR/ozMnJcVD20psLL/SCRSUY+u1+tFLBYDoMxN++AHP4hAIFD2frfHlVcBFGtoMEIppBsOh9UVfCPvqx6srqxSFIV0Oo2zZ8+irq4ON910E9xut6lj5JLreiSBUCyBDJf9XoOxFLxeCqlUEplMBo2NTRur7MY/N44Xce7qNF4bn8PhwV4cG94Du624rFFtn67NZoPP58t7BCdVXyKRyHJRmNWLt3OlWwxaTVfrMTaKziSaMYnOJN5ro9CbrVios1LpVqsx4jd+4zfwiU98oiLEvuNJV9vQcPHiRRw+fNh0CA1gnnSJjjo1NQWHwwGPx4PDhw9bMmKTytVMl9LKygqmpqbAcRxOnz5tSLa55Hrh0gSeH12GVIA0CQRBAMuyYNk0mpubi5Kt5gwBUOB4EedHFfI9sKcTt+zbDZ+ncIr+jfBSkqpP24VnRS+udPyiFjc68AYwlnG0HmPtPDdg82eUSqXA83xZtrZCsJq7UEnSjUajEAQBzc3N+NnPfoampqaK7H/Hkq5eQ0MmkzGdME9QjHS1j/baxK9XX33VcvdLMd8tCbyZnp5GY2Mjjh49igsXLsDtdiOeYhGMJhSCjSUUko3qV66NNgccDuPHfkEQNmQEFnY7g4aGBrjd+nJFLigKkGXlbwJeEPHLsVm8NjGPQwM9OL5/D3zufPLdTm3AxTJ6SebC4uIi4vE4XnvttSwiJrGQ5aLalW6pIelAYY8x0dQFQcDMzIxqa9MG3ni9XjidzrLen1VNt9SpJnoIhUJ4z3veAwDo6OjIGs1eDnYc6Raa0GC329V0KLMo5BDQJn7l6qilyBJGx5IkST2W0+NDS/duxNMcXnh9Aq9cnMKrC8k8cjVCoUd4Ldk2NDSitbUFwWAQVorPQu4IURRx8focLk8GcLC/G8f298GvqXx3QhuwXobAL3/5S9WvSaYdz87OZrkEtERj5frbihHplYZWU5+bm8OhQ4dAUZTqMU4kEobRmVpbmxkytkq6lax0+/r6MDk5WbH9Eew40hVFUe1+yr1YSyHC3F+8NvGLVJt6g+fKDb2Jp1ishWO4NjmN8ak5iDQD2u6GIEUBLMBmo+FzORBKpOGuN/+eaDqf2ARBQCQSBstuki2REawTIQVFXtiELMtIpVKgaRoOhwMCgIvj87g0GcBIfzeO7+9Dnde9I0hXD7Isg2EYOJ3OgnqxdsCm2YWpale6WwFtEJSRx1irF+t5jMlNK/fJ0YolbSeE3QA7kHQLhSaTSrcUSJKEQCBgmPiVC7MET2SBYDSBi9fmcWZ8GWlewnowjGg0Ao/Hq07KlaRNQhJFCdGk0uZY73UhwXIQxeKdY4oFTN7Yh1LZptMsGhsbNx6jqbzXWyFCIi8okBGPxxGJROByuSBJMniegyQpJOVwOHAmnsCFsRncNLQbNw10mT7OdkIhYjSjF5OFKQBZC1M+nw+iKFaVdLcDoReKztSGpCcSCQiCkOU2SSQSpm1gNdK9ASil0hVFEZlMBmfOnCmY+JWL3EpXS67r0biu5hoMrkOSZLBsGl6vF11d3UXv4pIkI5pkYbPRqPe6kWQzEAqQr/KYJ2B9fW2DbBt0yVb7+tzKtRAUkpaQSKQQDkfg8bjR1dW98V0ZpBIWBAEcx6mV4DPLK/j5SxS6m3ywOVzobGupmC663WBGL47H42ruQjgcVsPqK6kXb3fY7XY0NDTkEaXW1haLxRCNRjE/P5/lMSZPD9rPTywW2/ZZusCbmHQFQcDc3ByWlpZAURSOHz9u6kIn5Hptfg2J8UXIzKTugpYWsiwhFosjFlOmD3R3d4OmrdlslMo3DRtNoc7rQorl8shXFAWkUmmIYhzNzc0FyXYTVipdRU9fWlqC2+1GV1cnbDZm49haqYUCw9jBMHZ4PJvaqCDwCKyu4acvXUZPsx/9bX64GHpb5y4AlZMA9PTisbExtLa2gmGYiuvF1ZZyqrV/bXdiKpVCZ2cn/H6/Gp2ZTCYxPz+vRmemUin827/9G2KxGBYWFtDU1LStb1o7jnQLXfxm5AWO4zA7O4vV1VX09PTg1KlTuHDhQt7rEikW69FEXvVKyDUajYKigLo640cfWZYQjcYQj8fg8/nQ0FAPhmEsEa7yOL/5oRclGbEkm0W+GY5DOBxBOp2Gw2FHfX0dfD7zgTfFPzyKZhsKhSGKItrb2+FyFfYL60GRhih4vT6EWRkXAnHs6+3A0d5dsFNSnmE/91G8GtGHZlFNy5jdboff7zfUi5PJ5LbUi7diEZA0RxSKzgwGg1hZWcGLL76Iv/zLv8TY2Bje+ta34i/+4i/KPv773vc+dHV14R/+4R/K3hfBjiPdQiiUo5DJZDAzM4NgMIje3l6cPn0aNE0jkWKxFk3h3NUpJDNCHrkaoVACmEK2UcRicfj9fnR394CmacRi0Szd1gyM3AKiJCMcSyIWjUAWeHh9frS0NCMajZbgRjCSKxSfZigUgt1uR3t7O0KhoO5NI1vrNQdJkjE6s4Rrs0vY19uBEwf6sWvXLuXIGx5R8phJJg7kpm3t9EfxUvRi4qLQNjIA+XqxcoPf2WE3xdwLFEWhpaUF99xzDx599FF85zvfqdiC7TPPPIOLFy+iq6uyaxE7jnStVrq5iV9DQ0OgaRrxFItnX76E6cV1rK6uor4+ZCmijqYp8Hw2WUmShFhMIdu6Oj96enqyLnqKoiFJVt0V9AYpbu5HFEVEImGkUmk0NDTA19gExkbD63YhlUyCF8y7KigK0Mu7IWTLMAza2tpgtxNis6YB5xxNl5hlGbg2u4yxuWXs3aWQb0uDT/WIakEm+uY+inMcBwBobGzU1fu2K6xWi9qKT08v1ubNplIpZDIZjI6OVuUmtR1INxeEH8qt8JPJJP7kT/4EDz74IC5fvlzWvnKx40gXMH4k1mq6yWQS09PTSCQSuolffo8LH3rncUwtrOInP/9vpAVrExqU8G9lG0lSKttEIg6/vw67dvWAovI/SDRNg+etERZNK24Em42QbQSpVAoNDVDD94IAACAASURBVPVoamrOkR3SkGXA67LDzthMkW/uz5JlWYRCIdA0jdbW1rwPqPJ6S29Bsy1QiLBlGRibW8b1+WUM9rTj5Eg/WhuyZRKjib6XLl1CfX09eJ7P0vu03WU+n69qnVOlohp6MZnEkE6nMT4+ju7ubl27Vq6ObjWPYitI1+xNiWXZiub6/sEf/AE+85nPqE8RlcSOJF0jMAyDdDqN1157TU38GhkZKXhR93e34e7bDiPCiri+EMJSMGrqWBSlyAvhcAiJRAJ1dXXo6dEnWwLFQ2uN3IkbIRaLIZVKor6+YeM4+u9JhrLY53B7Ued1IZ3hC5IvId1MJoNQKAiAQnNzs+EFTDTmUmFmU1kGxudXMBFYwUB3G06O9KOtsfAEDJvNhvr6+qws4dzqb2lpCSzLqmZ9LencqLla1dRFSS6CUSQkkSi0erF2uGax4JutytI1c1OqZMLYt771LVAUhXvuuQdPPPFERfapxRuGdMkAxlgshqNHj6KxsdFS/kJ3sxfHDw1jamEVZy5NFCRfSRIRj8eQSCTR3NyMnp5dpo6l9dCagSSJ4DgOKyuraGxsNHUcQqLSxoIbTVOo87qRznC65CsIIhKJOFiWRXNzE5zOwtkJ5coLViDLwERgFZMLq+jrasXpkQG0NRUmXy2MhiISsz7JoJ2enlZHq5OKmGxX7YWiai52FSJ0Mum4kF4cCoWQTCYB5OvFLpfrhgeYa1FJj+7Xv/51RCIRDA8PIxqNIp1OQ5KkikyNAHYo6WofickARkBp2xsbGzMVsaiFVpbo725Df3ebLvmKoohoVMlt9fn8cLtdlu6uWkmiECRJVPNhadqGlpZGC9kI2XKBtCE7bJJvBrwggec5hEIhcBwHp9OJ9vYOk/u3vmCm3bYUwpZlYGphDVMLa+jvasXJkX50NJde1eiZ9bWxh2SaLyGcdDqNmZkZlZDLzRPQotqVbrX04nQ6rZLu/Py8pRFCZmHliSoWi1Ws0j1//rz67yeeeAIvvvhizb2gTfyy2+1ZAxhLefQlsoQWhHwnF1bx4sUxXJucQSqV2ni837VxIVrTe5SFNOPz05JtfX09enp2IRQqLRshf98K+YoCDzaZQCKVRl1DAwAKyWSi7P2b3LrE7TYxtbiGqcU17OlsxqmRAXS2VKa6MYo9lCQJr7zyCtxuN6LRKBYWFtQ8AW3lR9wCVlHNSrdSo3qMnhgWFxeRSCTAMEzF9WLA2k0jEolUNMC8mtiRpDsxMYFkMon9+/dbmgdmBKOmikwmAz4ewlCjDUNvOYLJlRhWQkqocSnj0Y3sWQrZRpFMJlSy3VyFNVcdE+hlLwDICihvampER1cjfB4XItEYEmVmLyjWMhaADIfDuSUBLjNLQcwsBbG7oxknR/qrdhyapmGz2dDe3q4GpgOKRKG1s01OTlrWRAFr42isotpZt7Isw+v15iV7afXi5eVlJBKJkn42NzLshuC+++7DfffdV9F97kjSHRoaskx4hZBLuizLYnp6GuFwGHs0gyVvBTC5sIqzG7KD1YJPIWrto7+0UdkmUFdXr6vZWs9GyNaNtcli2oByWQbiSRaSJMPjdMDB2MCZcjtkywvEWmaz2UDTFMLhsCZ7wQ6HwwmHw76xUFV5cpldDmJ2OQibkMYdnjrsrcBN2AwYhslrYdVqormZCyTykPzRPobv1KxeURR17WeF9GKipYdCoaygdG1VTJpgrJBupadGVBM7knQLVZmkacHKHZ6QLsuymJqaQiQSQV9fH4aHh/M+EAPdbRjYkB2++y/rFs9bqXQVi1lkw/WgT7babawsvhGSVpwVYaTTaTQ2ZieLaSEDSKQz8EsS6rwusBm+IPluuh1YBIMh0DSltrEqFbk2e4FXu6pIMpwgCIjFoqrx32o7tBGWwwn89IWL6J9awqmDA9jVZk3XrwQKaaJ6kYcMw4BlWSwuLpb1GG6ErSBdK1NQ9DrKjPRimlbawwVBQCgUKqoXx2IxDA4OVuR9VRs7knQLgRCoFdIl1eCFCxdMT/Ed6G7D7UcH0L57EGcuTWDZhNVMkhQyWlgIbFjMzLsRzEKSlIpicXERDQ0NaGlpRqEKM9ftQFEU/B4XMjwPjs8nX6U5IQybzZbldsi+CWqzFzYXAGVZwsLCIiiKRjKpBOYoofM2OBwO2O2Ojb/NZa3qIbAaxj//v/Pobm3EqZF+9HY0F9+oytDmz2rB8zzOnz8PWZbz2nxz259LIc/tRLpGMNKLScbH+vo6QqFQUb14pySMAW9A0iVdaWaM0qlUClNTU4jFYrDZbDh16pTlDzupfCcCKzh7eVKXfLXNEwBV1M+rhdkuNiJVJBIJUBRV0Mubvf9sUpdlGfGUQr51XhdYTiFfnucRCoXAsmn4/X40NeWTWe5ECb33QtNUTt6qUpWTRLJoNKV2FRISJn+sfMAX1sL48fOvoru1ASdH+rG7o6X4RlsMu90OhmHQ09Ojfo3EQhJNdG1tTV3k1Wt/LvQ7rralq5r7t9mUG3FDQwP27Nmjfp3EQSYSCSwvLyOZTOLTn/40BEHA6uoqIpEIDh8+jEOHDlXlvCqBHUm6hS40M0ljqVQKk5OTSCaT6O/vx4EDB3D27FnLhKvop0o1MdjTjsGe9izyze1U6+npQSCwYJpwAdJuXMjxsHmMurp6dHd3YXl5xfR7MYp2lGWl8hVFAZlUErFEAnX1DXA6K71QRsFmY+B2M1m2OGWis0LE6XQK0WgEoiipH0ZFL1aq40LvdWEtgp88/0t0Ntfj5Eg/+rpaDV+7HaCNhdSOyMmdyjA/P59V+Wn9xYQIJUmqai5FtUldT9PVi4N84YUXcN999+Etb3kLlpaW8Nprr+GRRx4p+biSJOG9730vZmdnQVEU/vqv/xrvfe97S95fLnYk6RYCwzCGSWPJZBKTk5NIp9Po7+9HS0tLWYsYhOC1F/ZgTzv2dDTjxfOv4fnzV0BRdkuVbS6M5AVtghkhdMXpIFueBKH3em2+Q2NjIzo7u+HzurCyulYwz7dSoChqYxEu+4kluyqOgee5ja9LsNlouN1uTVW8+btdCkbxLy9cQMcG+fZvc/LNhdFUBu3kityENp7nUVdXB7/fX5WEtq0gXTOdgjRNI5lM4s4770R3d3fR1xcDRVH49re/jc7OTjz99NN46KGHaqRbLPQmt9JNJBKYmppCOp3GwMAAmpubK3IB5gaZi6KIubk5LCwsYE9PD/74Ex/B9NI6zlyaUK1mCkjYd3HkWsZkWUYsFkMsFoXPt5lgtvl6a17l/GaKTUdFQ0PDxqIHpbQXJ1mIkgyvywGn3YaMjuZbbdhsNrUSJJBlGaurK6BpG9JpFtFobIMQaLUadjoVrXg5GMVTL1xAW6MfJ0cGMNjTVuBo1Ue5aVhGSWTpdBqTk5PIZDJqoVHphLatkC+MpmDnIhqNVizAnKIo1QY3OzuLm266qSL7JdiRpFsIWnkhHo9jcnISHMdhYGAATU1NFb3bk2MRsl1cXERXVxdOnz6tXoxa2eHMpQksLChuBJo2//hPqtd4PI5oNAKfz5dHtpotLL2Hzf1vVs7FFvniqQwYpxt+rwscx98Q8tVCmZdng8fjySIRUhXzPIdYLAaO4yHLMux2OyKRCGYWVtDd1oTbbtqLwZ72bRWEUw7IFF/ioiBkZJTQRkLkte3PZsj0RsgLRshkMqYJ2gweeeQR/Nmf/RlaW1vxzDPPVGy/wBuQdMkH6sKFCxAEQSXbYtDqs2ZBURQCgQCCwSC6u7tx8uRJw4uEkK+fYhHmGQRj5rrZKIpCJsMhEAjA6/WUNHWiMGQIgohAIACfz29CClE0YOLzpSjA73WBZTmkM1wFz6t8GFXFgqBY2TIZFtcmZ3B5bBJNdR4cG96Nkf5d8Pv9Wza9oppEn0uKRuE32vZnvRD5XP8sQbXdEWZJtxoTLB588EE8+OCD+MlPfoL3vve9GB0drdjvakeSbiGv3uzsLDKZDA4fPmzpcYNhGNNtk6IoYn5+Hmtra2hra8OpU6dM35F3tzfhV/btw0IwjrOXc2UHLeQNE3kYsiyjp6enwiQgIx5PIBJR9m+WzHOjHQn5Qpbg97jAi6LpcfE3AhRFwW5XJAfN1BxIkohXJ1dwLRDEUGcjWrx21bifyWSwtrZW8WjIao/TMUuK2vE4BNoQeW1Cm1aiEATBtO5aCqxaP6txA/u1X/s1fPrTn0YwGMzyXpeDHUm6QLYWSRLGAKC7uxvxeNyyvkOkgkIXECHbhYUFdHZ2oqenB/X19ZYM7UQHHtrVjqFd7RifX8khXxnJZBLhcBgulwvt7W1qx1dloN2/MlRycXHRdPVspBmTSEmaVny+HC8gw29f8s0FTdvgcrkhAri2nEBzvRfHhvvQ0+zH5cuX8qIhcx0DpWYLbNdxOkSi8Hg8ugltyWQSgiDg0qVLWQltWs243CpYFEVTP1cikVQKU1NT8Hg86OjowJkzZ/KaXcrFjiVdQGn9m5ycBEVRGBwcRH19vdpiaBU2m83QaiaKyuN3IBBAZ2enKiPMzMwYjuwpdBztNkO72jHY04aJ+RU8e/YixqcDcDod6OjoBMMwkCTR8ogfI6TTKQSDITgcm/u3Dv3shWQyteGPdqgf9p1IvgTBaBLPvHwZDT43/BSLU6f2qCRCiId0UJFsAdLUQP4UcwzIslzVx/NqPP5rE9oWFhZw8803ZyW0kaGRuZGQhIitPCmYlRei0WieZFIOIpEI7rjjDoiiiLa2Nvzwhz+s2L6BHUy6ly9fBsdxWQljQGlj2I22kyQJgUAA8/Pz6OjoyNNsCxG1EXJJFwBCoRBCS7N463AX3nZ0GBfGA1gNxwGUm+qlQJkGEdwIbtGO3rEO/eyFIOx2JV8hHk+A4zh1wcrpcKCx3geKZiBW+XG6GgjHkxhdXkYc/4MTB/qwf3enYTQkaWpIJBJZM920pKMNTK/24MhKpYzpQXtNFkpoIxJFNBrF4uJiXog8+bnoVapmb0qV7ka7+eabcf369YrtLxc7lnT379+v+wsxMxFYD1rSLUa22m1YlrV0HC3phsNhTExMwOFw4ODBg+po7oODuzERWMWZS+NYDccsky6RAHieQzCoVP0tLS15ntdSoJ00EQwGQdM02traNjRx7eOyrM4uC0Zi4DgODhsNWRIRDAbhcBAbV+XyV6sB8qOPxFN49uUrePnKFE7s78P+PV2w2bRWPeOmBkLE2sB0p9OpemmTyWTJrb6FUM2UMTM3DG2Lb25CG3FRrKysqFJFbgqZ2eu+klMjtgI7lnTtdrvuo30pkYuAQqAcx2Fubg7z8/Nob2/HiRMnCmpFpVa68Xgcc3NzsNlsGB4ezjO8UxS1KTsEVvDdp6wF68gysLKyDEmS0NTUDJer2DQI8xAExXbEcVxW9kJ+/KT+glVqYQFtzQ1IpNJZzQ12u13T8uvcNhMJFGySSzSRxs/PXcUro9MblW82+eaCjBHKrYqV8UghhEIhTE9PI51Og6KorAqw3DFC1XQXlGMXy3tSkCTIkoSMJi6TJLSdO3dOTWgjP5vcEPmdlDAG7GDSNUIpVZMkSSoR9vT0FCVbAqukG4vFsLi4CIqicOjQoaI6lEK+HfiVowNo6x3AmUsTquygB0HgEQqFwfO8OhW3UhBFJe0pnWbhcDjyMlTNgqIAETZ4vD60NbeAFwVljttGVaw0N0RzWn7LD8IpHTIoCrAzNJwOO2iKgiBKiKdY/PyVqzh7ZQrHh/twsL+7IPlqQRK3GhoaEAqFcPDgQQDZPtr19XVMT0+rOSJaIi6WQ5t7rGqgUh5d+soV0K+8AkqWYdu3D66TJ9HS0gJZlnH+/HnccsstavuzNkSeYRh4vV5cvXoVExMTFfXoVhtvONK1AkmSsLCwgLm5ObjdbvT09FiKhyM2s2KIx+OYmJhQhXmn02lJ+Cfkq22y0JKvKCopaSyrZOYKglBCdavfJSdJIsJhZQJxY2Mj/P46xGJ6NjcrH24Zskwhnt7w+XpcEEQHWC5baxZFwTAIx+ncJOPK+paVvAuXww4bTYEXRDA2mzLiSMjkvTaeZPH/Xh3FudEpHNuvkC9jkoxyH9H1fLS5Y4S0Gb25VXE1cxZyUQnSpQIB2F56CVJnJ2SaBn3lCuDxQDp6VF1EK5TQlkwm8dJLL+Gll17C9PQ0/v3f/x29vb341re+VXJ3GsdxeOCBB/Dcc89BlmV8+ctfxoc+9KGy3mcudizpFrqDF2t0kCQJi4uLmJ2dRVtbG06cOIFwOIxo1NwkYAK9RTEtkskkJiYmwHEcBgcH0djYiKWlpbzRQGaRS74vXhzD+Mw8kskUGhsbNmwtFOJxazowWRzT/kiVDrUo4vF41gTiTCaD0gdT5kOWlTxfhXyd4AUJLKeQq34QjqTJ6d2Mh5QkUdVVlaqYgZkbAUUBTjsDh52BLAMcL4ATBKRYkukgmsqaiKcy+K9Xr+Hc1WncMrwHhwd7ipKvGctYoUWqYt1loihWTWKoCOmurkJ2OoGN9RK5qQnU/DywQbqF9k+Cbz72sY8hHo/jox/9KH77t38bc3NzZTkZQqEQ3vWud+Gxxx7D9evXceLECdx1110VtaTtWNItBL0gGiCfbI8fP66+phTXg9E2JMUslUphcHAw68NSjKiNoK2KRFGEjU9hqNGGgfaDmF6LYy2yOeeMBN+YBVkcI3/H4zFEo1H4/fkdauUMptzYg24EpCwrxLVJviJYnSYLiqJVEtJsjdXVVTgcdvA8h2QyAZ4XNkJzsuMhXQ4HHA4bKFAQJYXgWU7QPRbZt5Un9EQ6g19cGMP50Wncsn8PDg/sgp3RJ49yLGM0TRsG4JCqmOM4vPrqqwAKT64oBZUgXdnnAzKbTw9UMglpYxHSrEcXgBrnSFEUdu/eXdY5dXR04Nd//dcBAHv37gXDMBuzESu3ULdjSddMvCMhVC3Ztra2ZpFt7jZWkCsvsCyLyclJxONxDAwM6KaYlUK62vyF+fl5zM/Po6enR814eKssYyKwgv+5NIG1cBxGs9gK719CIpFCOByGx2PcblyuhY1SJwLr//605OtzOyGI+uSbs1dQFA2Xy531e6UoGbQsQ+B5pDMs1qMR8IIIu53JImLlw218PqWMGUqyHF64cB2vjs7gluE9ODTQA4c9++NWDZ+utrtseXkZx48fLzi5wigWshgqQrr9/ZCnpkAHAsr/6+og3XwzAGu5C7FYrCoLaf/4j/+Iw4cPV9wZsWNJtxCIbUySJCwtLWF2dhYtLS26ZEtQCumShbRMJqOO+SH5vEY3hVJIl6ZpzM/PIxAIoKOjI6/tWCs7jM+v4Cc//28IFohRFCUsLi7B5XKhq6sTNluhy0I/f9c89CvdXGhlB5/HCUGUwGYKWAEpwOVg4PO4IEMGJ4jgeAEiKFB2Jzx2Jzy+eihWNmEje4FDPJ6AIAigaRIlScjYvlHhW6t0c5FkObxw8TrOX5vBzft246bBXSr5VrMjTXtjNNJFjWIhc6tivZHzFVlIYxiIt98OaX1dcTA0NQEbTzA3eijll7/8Zfzwhz/Ef/zHf1R0v8AblHRtNhuWl5dx5coVNDc349ixY0UXGUohXZ7nwbIsXn31VcOZanrnZpZ0ZVnGysoK4vE4/H5/wZsGoJDv3t4O3H3rQcR5CqOBdawVcDsoc86CEEUB7e0dplaAc7MXqg1ZBhI6la/DboOTYQAKECUZNACWEyChmG+aUhs5vBovmyRt5vTG45uJZAyj3FhTqRSczvycXrNIsRxefG0crxLyHeqtOukWq6L1YiELNTRoiZjn+crY+mw2yBoPL4GV3IVKxjoCwCc/+Ul1kU47bqpS2LGkq3exkllTy8vLqK+vN0W2BFZIl+d5TE9PY319HTRN4/Tp06Y/PGZIV5ZlrK+vY2JiAvX19erIErPvxWazoa+lEccP7cP4/ArOXJ7IIl/SNCHLEpqbWxAOh8EY6I65sJrXq7e91UrZZqPgsttB0xQY2OBzMxBEAfH0ph5Ybqc0yV5wubITyVg2jXA4gkyGRTwegyBs5vRuWtnM66PpDI+XXp/Aq9dm0d9ej90t1ZleXOoCmlFDg3ZMztLSEkKhEGRZRjgcziLjSgUCWdF0Y7FYxUj37NmzGBsbw3PPPVeR/elhx5KuFoRsZ2Zm0NTUhN7eXjidTksWGpouvvgkCAJmZmawsrKC3bt349SpU5bH/BQj3VAohPHxcXg8Htx0003weDx4/fXXLUkSpEGEVL4kWOeFC6O4PjUHnufQ1NSsVrbWqtd8eUGWleBz4uEldi59rbTwsSgALqcDdhsNGQAnCOB4AUk2PzbS53ZClCSkC8kOZYCiKDAMA7udQWPjZgKX0fQKpcFjU6IoVKmxHI/zY7M4PyrhXbwNR/f2wuWo3Ap5pbNuc8fkzMzMwO12w+/35yWRkapYa2mzmvNhxfaYSqUq5tO9ePEizp8/n2UdffTRR3HHHXdUZP/ADiZdsqCzvLyM6elpNDU14ZZbboHT6cTCwkJJrcBGEAQBc3NzWFpawq5du3D69OmsKsJKD70R6UajUYyPj4NhGIyMjGTpb1a77Gw2W9breZ6HnI5if4sDB3pvwfXFMNaz3A7mq1ctQec6HZqamtS5ZnpaqdOZfxN0Ohg4GBtAURAFESzPI53hYMZUl9iodH1uJ2L2rbuUjXJ6C890U94/w2w2eMiyDEEUcfbyJC5cn8WRoV7cvHc3XM7yyXcrJgEzDFMwiYy0+U5OTloOBLKapVup9/rxj38cH//4xyuyLyPsWNKVZRnnzp2D3+9XyZbAbrerBvJyoI1y7O7uxqlTp/KqB0KIZquKXNJNJBIYHx+HJEnYu3evrsfQ6uIbRVEQRRGiKGJ2dhZLS0vYs2cP9u7dC4qicPqIjOtzyzhzeQLrkYQlR4JC0IpHNBQKqU4HiqIhiiLsdjs8Hn2tNJGIg5IFxMLrsDscoGgb0jY7nM7yGhwS6QwyvIAml/KoX8kwdbPuBf2ZbpuTjjMZDqlUCoLAA6DgcNhV3VWSRGQ44OUrU7hwfQ5Hhnbh5n274da5SZnFjRy/bjUQSK/12Wqlvp3zO3KxY0mXoigcO3ZM98IqNWkMgGrNIoE32ihHPRDbmNkLhFwcqVQKExMTYFkWQ0NDBTUpmqYtywvr6+uYnZ3VvVlQFIV9uzuxt7cD1+eW8dPnXgRvknRZNqOuend2bsZDKtW+5hwoKF1djAuy5AEviMjwAlZWVuCrb9rotMognU4hElEaHBjGlvV4brbBgSDFZuCQAK/bAVmSkaqI7FCOe8Fo0rHS4EFGCK2srG68f8XK9l+RKF65Molb9vfh2HAfPC7r5HsjSVcPhQKBSFWsDQTieR6iKCKTyahVsd77sRp0vh2wY0kXUCpAvQqt1KQxmqYxNzeHQCCA9vb2gmSrPQe9RgwjsCyLdDqN119/3dDLq3cMM/ICkVumpqbg9XqLnj8h3w/cdgjRjIzRwHqW7KAFx3EIhYKQZeVGo32cBGS4HAxoigFF0xAEESzHK6SXQ3wUpfyx2x05PzNlbBDHEVJXGhwUecKhIWNi5cp9L5v/TqaVStfrdkKWpLLItxpODdLg4XI5ATjh99dBef+bbc+rawn86+ISnn7hZRzs68aJkQG0NjeaDkyvGumKIrDx5FUJsjNqfb548SLq6+uRTqextramdnFqq2K3241UKlXRLN2twI4mXaPHYquVLlmIIwlHZgNvyLHMVKEcx2F6ehrBYBAMw+DkyZMVczzIsoxgMIjx8XE0NDSgv7/fks+Rpmn0dzXixOHhLNkBUCqJcDi0kSrWDJfLjcWFALxuJ2wbFXhmI6hcOcdSy0Jl0UrRCbXyhLRBRBnVygXIeYtWekhuaL5elwOyDKRKlh2qZ+vavIFQYBg7GCZXnpEwF0pi+r9+id2tddjT6oNjY9R8IddApRfSqKkpuO+5B/T4OOTGRvj/z/+BbWioYvvPOtbG55pEhhKIoqg2eQSDQfz4xz/GP/3TP0EQBHzhC1/A4cOHcfvtt+cUBNsPO5p0jaA3hl0P2oW45uZmNDc3o7e311KfdbGkMa3jgeiqZ86csaRBFVpIi0QiGB8fh9PpVN0OKysr4DjzBKN1OxDZ4er0Av79F69geXUdnR1taKyrV5oOeBGiJKuEZh3WfL40TcPlcmWtZCuLVnzWolUmw4HnebhcLpWMiTxBnA+K7GCVfMtrjii4ZxlFp0LTNK3EZzqBtZSEcCCBg/3d6OvqgCzyanJdbodZqfKaLiQJnrvuAjU/D0qWQYVC2P9Hf4T4e94D7NlTueNooHfTsNlsWa3PX/ziF3H77bfjsccewzvf+U68/vrr6tzCUpDJZPDNb34Tzz77LH7605+W/R6MsKNJt1DXV6GLTpaVXv2pqSk0NDSoC3FXrlypWP6Cdix7NRwP2gW43Exeq24H7eslScLc3BwiSwu4993HkBRtePnqlKHsYBWl+HTz90FpKlzF5aEMjvQqoTVcRkeecCCTccLhsMPndkEG1FCbQii1DdgMSpkcIYgSLo7P4/LUAg72d+PY/j4MeJQbEs/z6kLV+vr6xlSPkNph5vf7S8pdoNbXQa2sgNLcLWWahuO11yBWiXQBc4tj8XgcnZ2duOOOO8q2de3btw9Hjx5FPG7cUFQJ7GjSNYLRL4s0HUxOTsLv9+Po0aNZFVSprcBaQtROnejq6tJ1PJBtrDz+E406nU5jYmIC6XTacAHO6sIbSWVbWlrC1NRU3rSM4T1dGJtdwpnLkwhGyyXf6nS0URQ27FnOrC6iTXkiu9PM4bCjzueF0+mEKFMFHsWr2X5XehVNyPfSZAAj/d04vr8PdV43Ghsb0djYCLvdDo7j0NPTk5W7MD8/D47jstLI/H5/wYxeua4uX9yWJECnk2yrUckW4IsXL+LixYv40pe+VJH9GWFHk66Vu3UwGMTExAQ8Hg8OHz6s295XYRJtGgAAIABJREFUTv4C0YWnp6fVuEgjmcIq6dpsNnAch2vXriEcDmNgYACtra0FK30rXWPpdBpzc3N5yWsEFEVheE8X9u3uxNjsEr73r6um932jUUyeSEZioGURoiRCkDZbY0loOlA8J6JUVKKKFiUZr08EcGVqAQf6unH8QB/qvW51Plqh3AVSFc/OzhbO6HW5kPnTP4Xzj/9Y2ZimETx2DJ6TJ8s690qgkqS7VdMndjTpFoMsy4hEIrpzyPRQKulGIhHMz8+jsbGxaD4C2cZsJSoIis1qfX0dw8PD2Ldvn6nZVGb2H4vFcP36dXAch+7u7qIB7oR877hlEM3dfSVVvpWQF8pFtjxBIMNht4FjM4jEk+p0Y5KIFo/H1G303BOloJKDKUVJxqXJAK5OL2D/ni501zvQ4De+1rVpZAS5Gb0zMzOqTu5797vR3NeHuvFx2Pv7caWpCceNzl2WgWRSycktYVSUlUyKSCSC7u5uy8e4kXjDki4Z98EwjO4cMj0wDGPaakYcAzMzM2AYJk+qKAQzpCtJkpos1tTUhI6ODnR1dZnafzFNl3iEM5kM9u7di0QiYclil1v5nrk8ifVIHIIgbmQ4FPrAbG1gjnlQ4HgJsNnR1aEsxKRYxY+cSiWVyMmcScfalufCyWxGqPwinSjJuDy1gP+JRbF/dyd+xd+ABr+50Ba9jF4yzy2RSCAyNIRAZydSqRRSqRRGR0dVnVht9U2nwTz5JOixMcU++O53Q3znOy29B6uxjiMjI5b2f6Oxo0lX724Yi8WyCEVrxC4GhmFMTXUgU3ydTif6+vqQSqUsjccpRLpamYJ4hZPJJObn503v34h0OY7D1NQUwuEwBgcHVY9wMpksKcSGkG97vRs/f/EVXJmNYSWe3lhxd2hcBJutrzuhcYgssHlcDkASwHGZPB+pIPDIZMhMt5i62q4lYkWeMH7D1RzBLkkSxhfWsPSzl7CvtwMnR/rRWKDyNQKZ5+ZyuTYmkyj7fvXVV9Hd3Y14PJ7V6tt19iyax8ZA9fXBSdNw/uxnkLu7Ie3da/qYVkl3J00CBnY46WpBVvNFUcTg4CDm5uYszwkrJi/EYjGMj48rZLNRPYfDYcurnXqkK8sy1tbWMDk5mSdTlONGAJDVDtzX15cnUZQ6QTmVSuH69esQRRH/3ztuxV12O67Pr+B/Xh/H8noYHJfRzDZTHumVfIKMYZPDdkKK5ZBiObgcdnhdDtV6RlGbk461IDPdMhlOlSe0UgYZOU8WrLbCGSFJMkZnlnBtdgn7ejtw4kA/muvLSzYjNxi9pgbqF78A19EBNpNBPJMBHQoh8otfQJDlLK24kIfYise4Glm61caOJl1SpZHKVruav7S0ZLkrzYh0k8kkxsfHwfM8hoaGsn7JpY5h15JuOBzG+Pg43G43jhw5kpeYZDV7gZCoLMtYWFjA7OysoZNC+3qzkCQJo6OjiEajGBoaQnNzszpAcXh3J/b1dmBsbhmvXJ1CMJoEQFpfOYRCYaRSacTjCWw2OWxWxduvpVNGZiPlzONygAJ0E8+AQjPdOHXhiuM4SJKsdk0qaWy2knN6Dc86p4qWZeDa7DLG5paxd5dCvi0NpZEvaVnOBUVRYHp64Lh2Dd7OTkCWQbEsmo4fR7S9XY2FTCQSBcPSrQaYVzJL9x3veAfe8Y53VGx/etjRpMtxHK5evYr+/v6sOWRAeU4EgnQ6jcnJSSSTybxZZ9ptrE6CIF1s8Xgc169fB03TOHDggOHI9FIqXZZlcebMGTQ3NxftsDO7f+I9TqVSBUPbKYrSJV+n0wWn0wGPxwOXy63b5CCKmxkEhSMitx5a2UGSgaujPNbWKHg8wOCgBL10QaXl16U0OKhQ3vf6+jo4LoNUKrVR3dFZiWzaRDKrkCRZ90lCloGxuWVcn1/GYE87To70o7Wh+HqHFkp6nP5TinDHHbAvLSkDJmUZ0i23gDp0CA02W1axIsuy4Wh1m80GiqIQj8fh8XgK3ogrTbpbgR1Nug6HA8ePH9f9Xjkzz7Tjd4rZs0o5jiAImJ2dhc1my6uc9WCF2CORiOpIuO2220xJLMVSxrQ6c2dnJ3w+Hzo7O4sSQi75vnxlCqFQMOv7uU0O2gyC7HE6NDbHrjuzdGIjhELAf/wHg2iMQneXjDvuECwvputJACmWw9WrNOZnXfB6KSxGMgiFbLj1VhHmmhkVecJms6G+vkG9ISqJZEr2RCSilWWyW57NZCoU04tlGRifX8FEYAUD3W04OdKPtkZzGQYFU/UaG8F/8pOgVlcBhoHc0aEr5FMUZRiWTixs8/PzSCaVJyW9qhhQZEWjYmW7YkeTbqGLqpTQGyUfNl618TuEzFdXV9Hc3IyRkRFTlYyZY2g71Pbv349Lly6Z1rQLVbok06G+vl7VmdfW1nQDVYw+jFry/fmLDCZWosgY3qeMMghEZDIkODySpRM7HI6NceObN450GvjOd+xIp5Wf73iSQixmx//+31bDb/IdBrIMzM1RqKvnQFFAs9OBcIRCNJpGS4v5BUllVtzmzpWcXo/ByHmlyy4cJvIEk1UV58oTZhfpZBmYCKxicmEVfV2tOD0ygLamwuRbNNnL6YS8a1fRY+vBbrfD5XLB4/Gobp3cwZpzc3MYHx/H3/3d3yGTyeD73/8+jhw5gn379lV0VHq1sKNJFygcepPJmMsHIJXn0tISAODUqVOmE5rMEKIgCJiensba2hr6+vrQ0NCAdDpt+tGxUCVKJhAnEgkMDQ1l+S7NQo90E4kExsbGYLPZ8ppJcs9HlmVVQ9be6GiaBkVR6s+SoigMdLXg5v0DWE/ySuUbS5o8R73g8E29lOcFBIPrIDPQFhf9EMXND6AoAisrFNJp6MoARjBa7KIopSnLZgN4iYMgAW6nAz4XkGDN5lIUt4wZjZzfHK6pN0bIaVnykmVgamENUwtr6O9qxcmRfnQ067sCrORHlwJBELKuN70Gj6NHj+LkyZO49957sbi4iJ/97Ge49dZb8clPfrLk4z755JP4whe+AJvNhj/6oz/Cxz72sbLehxF2POkawUylS7yw2pHmL7/8sqVIvELESXIMFhYWsGvXLpXM19bWLLfp5kJL5MUmEBeDlnQzmQzGx8eRTCaxd+9eXb1MOxJekiR1W7vdnkXA5N/kvZKvybKMvbvasa+3A9dml/DK1WnT5Jt9Hpt6aSaj2LrsdgcEgUcwKOt2rrJsEnZ7eToxRQF790oYHaVhtwOCADQ0yHB5MkiwgNtph42m1ckWRijdMqY/XFM7Rojneayurm5IONbkianFNUwtrqGvswUnR/rR2ZItf1U6wSwXZhfSenp6YLfb8eCDD5Z9zHg8js9+9rM4e/YsbDYbjhw5gjvvvNOS5dQsdjzplhLvKEkSFhcXMTs7qzvSvFzIsozFxUXMzMzk5RgACsmVmgKlbZrQEnk5IJMmJiYmsLq6iv7+/oLSBzl/MleOoqi8x2Tt+QLKB2lhYQHBYBCtra0qEQ92t2KwuxXX51dwbnQG4Xh5Ez+InWvfPuDMWSAUVEjRbpexbx8HiuIQDFrRifWr0b4+GR6PhFBIqZx7emSQt01mtrmddtA0bZjIVmnLmHaMUCqVREdHBwBsyBOcjjyx6RrRa2qZXlrH9NI6dnc049TBAXRtkO92IV0yJbsSeOaZZ/D2t79d7W5717vehf/8z//EvffeW5H9a7HjSdcIevGO2ijHlpYWw1X9UisQ4rWdmJhAc3OzYUuw2Qze3H1rsx3MBKyb3e/KygqCwSCampoKkji5uTEMg9HRUdTX16Ourg5+v9+w9ZmiKDVkqLW1FSdOnADDMGo1TP7eu6s9j3yVXwFV0u+CpoHf/i0e58/bEAoBvb0yDh2iAGjHjRvpxEpl6HQ6NnRi/eO3t8sFM1+KkW81myM2901p5AlCUFp5ImMw5XjzJjS7HMTschC97U04OTIAbAHpmtl/JBKpmEd3fn4eu3fvVv/f09Ojyo2Vxo4n3UKuAiIvaEeaa6Mc9WB15hkBCdTxer24+eabCy5imZ0Eod13KpVCOBwueO5WQH4mJPi8vr4+66LTez2REvbv368ubKyvr2N6eho8z6vTYQkRcxyn5l4cOXIk62dCiD23Kj402Iv9e7pwbXYJL1+ZQiSR0sgElCUittuB06eNb27GOrGycJVIJJBOp9X5Xtqq2Mr1oSVfm41GPMlunH91s3qNq+ji8kQ0GlWnHBMivs6mMbO0Dq8dODnSD5Nd6ZZhNgyqko0RHMdlFRs0TVftxrLjSdcIRF7QjjTXazww2s7sDzwWiyGVSmF2djZviq8RzDoe4vE4xsbGwDAM3G439u/fb6kyMqqkotEorl+/DqfTiaNHj8Jms+H111833IdWtyULY6RHv7OzU30dy7KIxWIIBoMYHR2FIAjwer3w+/2IRCJqhGAh2QJQfj437d2DQ4O9GJ1ZxNnLkxuar7xRbcuQpE0nEplQXIlMh9yFq3g8BlkG3G6Xpu03uuEntqkkrPhqC+vEKvm6HGBoeiP358b7jwn0pxxL4Hle7bILhyPgeQ6T88s4c2kCpw8NYbivJ29yRTkwKy9Eo9GKtQB3dnbi+eefV/8fCARwskopam9Y0o3H40gkEpibmzNNhsAm6RarJlOpFMbHx8FxHHw+Hw4cOGAp8KaQpptOpzE+Pq7mR9TX1+Ps2bOWKnC9il2733379qktnDzP51XeemRb6ENFPLeplDJocnh4GK2t/3973x0dV3mm/9zpGo3qSJpRH6uMihu2ZCfgJMvBAS+wYdlsSDjeXZYaIEDMwaY7HJxDMfYBDmUJNqGFbDawCXDCSYidLD/TLctFxmD13tv03u7vD+m7/mY05U5V8Tzn+IDl0b2fRnOf7/3e93mftxAulwsmk4nr0bfZbBCJRFw0nJWVhczMzKApDYFAgNVVZX6Rr958zidi7ujPwm63w+l0gGGy4fN5OSnWOb+H2MmAXGth2y+Z6ebiomKSJ/b3X1hoGO6YJ1+JWAhFhhQWuwOJJt9ERdAMQxo2zj0Ps7OzkEql0FudeO/jk8g5dga1xbkoK8z1M8AJ9XuNBGJLGQmJjHS3bduGBx98EFNTc0NCv/jiCxw4cCAh1w7EsifdwA+02WxGT08PfD4fZDIZLrjggqiuF6nZwel0ore3FyaTiTONaWtriypHGyrSDWVIA0Sf9qBf73a70dfXB51Ot+C69GsBf/lXsCJZMJBc+cDAAEpKSrB582buoZFKpSgsLPSrArvdbpjNZphMJgwMDMBqtXKyIELGdH++QCBA46pS1FcW+5Gvz+fB7KwODAOo1cUQi0WUQmIuSpsjTXLn6NIT8z8dghMiPdONNk0/N3LdaDTNH9GDKwicbi8sdidkUglEQgEstsSTbzJAXNakUimnpe7Ve2BmHVgtz4HU7fZrbCCnHXrEeiTw+f0k0uxGpVLh8ccfx4UXXggAePrpp8PawMaDZU+6BMFGmn/xxRdRXycU6brdbvT392NmZgZVVVV+R/1ou9LCGdJoNJqgnrmEqPmKv0k0PTo6itHRUVRWVkKr1Qb9MNNeDV6vlyNbPtEGmdGWlZWFpqYmXlORxWLxAi9X0hZtNpsxMjICi8UCdt4khUTEWVlZaFxViprSQnz05Qkc/boXOTnZfsfhwJ/vHAHPRcXn/st9R0QijiZqFAiEkMkyIJMFzxPTCgKPxwODwcA5smXKZcuCfFnWF3S225TBgtmvrNigrcSWdfXc5yrYiHWZTOY3QiiW9ITBYEiopOv666/H9ddfn7DrhcKyJ12Xy4X29naYzWbOH4GODqOVtwQSKD3rrKKiImh1P1r/BbI+voY00d6DmM+cOHEiqGQtGFwuF8bHx5Gdnc0Zj4QDSa+wLIvGxsa4owLhfG8+fVwkD6zJZMLU1BR6e3vhcDjg8XigzsvDz67ZhnGDFSe7hqE3B9f5zkXqQGC3FiFi8icUESdC1hWqwWFkZHTeTnQuTzwxnydWZMqRmZEB97wxTvT3T65hcaCvQ0GOAuWqfFSqlSgtzINE7C+PDJX/t1gsMJvNGB8fh8Ph4AZrejwemM3miOkJctpcblj2pOv1elFQUBC0OYDIxmIhXVrLG4kQo5WAkc4tvoY0AH9TGr1ej66uLrjdbqxZsyZsh1qgIsFgMGBsbGy+Ui/1UyJkZGSAYRgu4jcYDKipqYmpA44v6AfWYDDAZDKhqKgIarV6Pnesh8dshjZfiEmhGL2TJjg8PkgkUohEopARaiQiJhExy/rgdDogl8vh83k5CRa5RnxgIBAwAbWGc3lig9kC+LxgfT443B6uWBcqT0wjmZaRACATC9G4qgSrSopQoc6HIiM6QwuGYbiCXWDayWg0YmZmZkF6gp7nRp6V5Wh2A6wA0pXL5SEJi8jGopFYCYVC6HQ6DA4OhtXyBn4P3/QCMaTxeDzYvHlzRDUFfY9wxG61WtHV1QWWZbF69Wr09/eHjG6DFckKCgo4k2oA8/rNubzrxMQEbDYbfL65SnZRUVFCols+cDgcnE/y6tWruXvm5OQsiJyMRhNOdw3g2Nk+TJisEAqF3NFdKiW60+D3oYmYZQGbzQqdTo+sLAXk8kwwDCEzkvs+V2SLtWC3UG0RPE8sFjHwejzQG81Unhh+mlq60yzR+l+JSIjSwjxUqJWoUCkx0NOBjRvXxd2UEwixWMwVVhsbGwGEHiH01FNPwel0orKyEhqNBqtWrUr4epKFZU+64T5c0eZa6fE7kbS2NPgc/Qkp0oY0fAkXCD8NoqenByaTCVqtlos8g70+miIZOQ4rlUq/5obc3FzYbDb09fXBZrNBKBT6RcSxVqwD4fV6MTAwgJmZmZC2mgR05KRWq3DpdzahfWAcn3/VhalZPZxOF2w267ypuMCPiOfM1M+9By6XG7OzsxAKBSguLp7v1PJ/D8l/z6UozhHx/IriauwIhNvDAhBCVVgAsUgIs80xn0IK1mkmni8o+uD1emIaIyQQMFDlZaNCrUS5Kh8lylwIhed+p/0smzSCC5SLBRsh5PP5sG/fPuzcuRPj4+O49957kZ2djTfeeCMha2hpacEdd9yB999/H2VlZQm5Jo1lT7rhwNdpzGg0oru7GyKRCBqNJurxO+FmqzmdTvT09MBsNvuRIhl7HquxTuA0iEANbyDpxlIkM5vN6O7uDtrcQEDybyaTCYODg7BYLGGVCJFAOuT6+/tRWlqKTZs2Rf2Az0nNStGgKUb7wDhavumFwTLXXuz1+uByOefVBQa4XOfUBR6PF16vB0plAeTy4BtiqKg2XMHuXEQMnCPi6BsjHC4PHC4PpBIxJCIRLAJHkE4zN+x2O3w+H6anZ+YbDYR+EbFYvFBPnJslR6VKiQp1PsqK8iGTLI5bFx+NrkAgQE1NDbxeL3bv3h12Q44W9913H9ra2jA4OJiwawZi2ZNuPJEumQjh8Xig1WqRnZ0d8/gdh8Ph97VIhjSERKMlXdrXoaSkBBdeeGFIjSvdastXbwuc2yjsdjv3voSCSCRCXl6eX24tmBIBAJeTI2QcSMRk81MoFLyVEOEQinzpBgCWndtcDAY9pFIZxGIR9HoddDoWYvGc3nYuIpb6RXuBiLZgd06i50O0EbHT5YHT5YFUIoJEJITF7uTyuERL7HQ6UVhYBICd94h2ceoJt9uDDKkYleoCVJWpUKcpQ3GRckkcz6OpwZjN5oTPR3vooYeQm5sLjUaT0OvSWPakC0RvekPbIQYeXWM1Pyffw9eQJloJmEAggNFoRH9/P69R7wKBAG63m1sXH7Il0TMxvQln3h4OoZQIFosFJpMJ4+PjXKolMzMTcrkcRqMRPp8P9fX1CTelDkW+TqcLs7MzkEgkKC0t8yPVc8d3J6xWG3Q6PXw+37w+9Vx6IjD9QCMYEft8LGw2K/R6PbKzc7g8Mfn4sizm5ViR88ShyNc/p8tAKBQhO0uC4oIyVKqVqFDlI0+RweVKjbppjA0PADinqSUnlEQaQfEB3xZgIDkWk6mYt7YiSDcUxGKx33Rf0iQwOzuL6urqoIqHeMb8RGNIE40EzGw2Y3R0FAKBABdccIFfkSUQJLLNzs5Gb28v+vv7/Y76wSJMurmhtLTUr7khURAIBAsGGXo8HvT09GBsbAwKhQIsy+Lrr7+GXC73i4jjjXjpNayuKkVNaQH+9lkrjncOQqksgFS68PoMw8xHuRKQdCLLYn4KsHO+5dnIjZ0n0fAcEQdXTsyN6CH54hKOsOkOO4ZhOSIm94xUsOPIVyyCRCzCjNMBASNAYV4WKlT5qFDNSbnEARuERCJZcEKx2WwLJvwST41AT9tkgG8LcCzTq2nceuutOHHihN/XXn31Vaxfvz6u6/LBiiDdSJEunf8M1yRAf080sFqtmJiYAMMwvA1p+JCuw+FAT08PbDYbioqK5juAghNuYJFMqVSioKAAPp+PO+qPjY1xQwEJEQNzQzxzc3PR3NycEud94sbW19cHtVqNLVu2+FXebTYb5+FAm+kQEiZa4ljuOzo6iuHhYTQ1VuPyiy9E++A4js0b60QCw4AziiHcw7JzU4CdzjnHLovFDLfbw807I8U6q9UKm80OpVKJjACJFfksCoXB8sT8C3YSsRDlqnysX6WCTOjD2saGqN4fUhSli1YsO9dmbTabYTQaOSlXW1sbR8S0pDBeeDweXpssed5jvWeyWnz5YEWQbigIhULo9XocPXoUpaWlYbW2BNGQLhksCQDZ2dmczIXv2kKRLp0Prq6uRlFRESYmJvyidhrhimQCgQA5OTl+uS+fz4eZmRn09fXB7XZDJBJBr9ejs7PTL8JMxtGSvGcZGRnYsGHDAvKkZ2fRkjDy4BsMBgwPD8PpdEImk/mtN1xXE5HqkdQM+dnWVJWhUVOCswNjvMnXf73gZF6Zmec2RJJHJU5sAsGcHMxiscDtds2TcXi9bbCUEE3CErEQJQW5KC/Kn0sZZMkhEAig0+mirkuEW4NcLodcLodKpYLT6UR7ezsaGho4f5OpqSnY7XaOtAkRx6Jk8Xg8vFQ9FoslYV66qcaKJF1yXO7p6QHDMFGZlPPZOQMNaTIyMnD69Omo1hiMdH0+H0ZGRjA0NLSg+y3Y62MpktHNDbSagu7+mpyc5OatkRwfIbZYI2GXy8VNVo5UnAtE4INPfnan08mZ6YyOjsLhcEAikfhFxAKBAD09PXC73X46XxoCgcCPfFu+6YMxSvINhM/HwmQygWGA8vIyzkOYFLSMRtO8cgKUQc5cA0QoohIwDNQFuahUF6BCrUSxMmdBwZQMdFSr1WFHJ8UKknMlkkJa202ULBaLJaT3QqTNPBpbx2g+Q0sJK4J06bZaepDi+vXrOSlYIuB2u9Hb27vAkMbr9UZtSk6TKG1+XlBQEHSTCGZKEw3ZEkInPgy1tbULJGbB9JDkqE/acL1eL+RyuR+xhSNiUlgkvhJ8hn3yAcMwkMlkkMlkKCoq4r5OmjqMRiP6+vpgt9uRkZEBpVIJk8kElmWRmZkZ0oMiXvJlWRYGgxFWqwX5+Uo/6ZlAIEBGhswvvUBatudSExbodE4/QxmVMhc1FSWoKVOjTJUPqXjhZ5l8NoaHhzE9Pc05yEUanURGnZNr8EG4tvpgShaymZvNZi6lFJgnJnl7hmEWxdYxGAYGBpJ27RVBusA54xWpVIr169dDLpfD6/VGPRE4GGj/hWCGNHxbdGkQ0jUajejs7ERGRkbYhgzy+rmpt/zJlpiV05MbonEqI8UTMpmVZVkuIiYpCjJIkBTKyENEjOOLioqwadOmpE4bIJBKpZy5ilqtRmVlJXw+HxcRT09Pw2azcYU9snHQR+FYyddut2N2dhaZmQqUlpby2lzmCnbnfBnkUgnKVPkoyslEnlwMn3tuE5ka7oV5Zszv1EE8MnQ6Hbq6uqBWqxfomoONTiIbNiFiABwZ09FwMCKO1ssk2GYemCceGRmBy+WCRCKB3W7ndN3h8sSJnBqRaqwI0h0ZGcHY2Bjq6+v9frmxkCFwrnGBYRhehjSxRG6kuCcWi9HQ0BA2P0VytXq9HmNjYxxJRLovyZ8Si8toGj5CgWGYBVVsuvhFiNZqtUIsFkOlUiE7Ozvpc7WAOdLr6uoCwzB+P69QKIRSqfSTBpKjsNlsDtvUwYd8PR4PZmdnwbKASqWebz7gB5FwrsW2Uq1EhVqJwtysoL/XYAUtYv4jEAhQUVGxwLIzEKHIlCZhUh8A/ImY/EnE7zFYugiYO6WcPn2aS0WRPDHtRqZQKDj5ZJp0FxFlZWVc0YVGrMdYoVCIiYkJDA4OIj8/n5f/Al+QFMXk5CQKCgqwevXqsK8nD4FcLkdtbS1MJhN6e3ths9m4XnUSYZKpDKS5gdhcJjv3RYpfEokEJpMJAoEAGzZsgEQigdls5rwsXC5XQlQIgaBbhvmOoQ/V1EG0xIFNHdlZWfiXLWsxNGPEic4hGC02sCwLo9EEi8WM/Pz8sFI+AoZhUJSXjQrVHMmWFuZCxIPEaKIqKirC2NgYhoaGUFVVBalUynUPRmMSTyAQCIISMZ2eIMGLyWQCMLfRECJOlLxQKpVCIBD4+Sh4PB7OjWx0dBQWiwW//vWvMTo6ipycHHz88cdYv379siJgJoLeLbkecQmC1+sNqTj44osvcNFFF/G+lsFgwMmTJ5GXl4f6+nre/giR7kOPY9doNBAIBLDZbKiurg76ej55W5fLxbXgkrFBRCJXXFyMsrKysONxEgWWZTEyMoKRkRFUVlaiuLg4YrRG1uxyuSCTyfxSE3wjcpZlMTU1hb6+PpSWlqKsrCzh+mK6qYNExh6vFyOzFrT1jMDHCJGfnx82D5mdmYEKlRKVxQWoUOUjI4gumC8sFgs6OjqgUChQU1MT9L60SbzZbI5oEs8HDocDHR0dXAsuKQwGIt6C3bFjx7B58+aIa9m3bx+mpqaQk5ODM2fO4NChQ3GS/9yHAAAgAElEQVTrud977z088sgjMJvN2Lp1Kw4ePBhPVB/yoVsRkW4k2Q0fjwPSEuz1epGbm4vq6uqoDGlC3YcoKYgmlaQopqengxbfoimSSSQSKJVK5OfnY3x8HIODgygrK4NCoYDFYuE0vmKx2I/UEknEZAYdmX4cjnxCqRDIbDVybA6UgwXz+LVYLNyct0S0DIdCYFOHy+VCZ2cnVolEaG6swdn+UbR2DMBsc0AsnpsOka3IRHW5GlVlKlSqlcjLit+Nzev1or+/HzqdDnV1dWGLSLGYxIeSCRJt88jICFc8phEYESeqYBcOMpkMUqkUV155JX784x/HfT0CvV6PlpYWiMViXH755Xj77bexffv2hF2fYEWQbjgQ3W2oh5Iev1NbWwulUskNVYwGwbwUSIEjOzsbzc3NfkfpQAlYLIoEYO6DQtQadHMD7VNKR8RkThlNxNnZ2VGL24mJOQCsW7cuqg2KBu0QFkwOZjKZODmYVCqFQqGA1WqFy+VCXV1dyo6VdDRfXV3NtUhXVa3C5RdfhG/6R6EzmJCXKYGU8cFqtcA+M4pBqw66IMWvaEAUOSUlJWhubo6JuCKZxE9OTqKnp4dLZRF54NjYGPfZCrahhprqTP4brmAXT3oiGTndG2+8kfv/9evXY3p6OqHXJ1gRpMvH9CaQdD0eDwYGBjifgXjG75DvIV4KJApjGAZr164Nqg2lDWyinUkG+JNeKP0pAYmI6UJSsIGREonEL7oMRsSkcYPI5pJhYh5MDkYkUUNDQ1AoFJBIJGhvb+d0uYFm64kEUZjk5eUFVX8IhQKsqylf8H2honipVOqX1w7V1OF0OtHZ2QmWZRNWCKURSllgsVjQ19cHg8EAmUwGnU4Hm83mlycO14gSrmAXLE9MBx98T6YmkylpG+7Y2BjeffddHD58OCnXXxGkGw6B9o5Erzo8PBzSkCZW/wWbzYbe3l6YzWbU1dWFdbWnJWDR2C0S/wij0cjNgosFEolkgXE5IWI6IqaJ2OFwcGOLampqkp4rJjCZTOjs7ER2dja+/e1v+xU16QaJ8fFx2O12XpsHHxCvYrvdHnFjC4ZwUTw5edBrpkltdnYWo6OjqKmpSegcsEggG0xRURHWrl0LgUDAbR5kzXQjCp2aiJS2ChUVk+uTExutew8VEcczNSKc78Lx48exfft27N+/H7W1tTFdPxJWRCENmHv4gqGjowOFhYXIz8/H5OQk+vr6UFhYiFWrVoXMPw4NDQEAKioqeN3b6/WipaUFHo8HtbW1UKvVYT985MFrbW3lop6cnJyw88kCmxtCFasSDTI7bXBwkKtyB0ZqyYguyb0J6dXV1fE2W6GjeFJgJOkUsuZwBEF7NKxatQoqlSrp7zVZ88zMDOfjEdiEIpfLk2oeTqR+DQ0NvJQY9PtsNpv9TO2D6Z+DgfZPJi3voSJiGldeeSX+9Kc/JXRD+uSTT3DnnXfirbfeSoTxTcgPzIon3d7eXq5BIDs7G9XV1RFlSmNjY3A6nVi1alXY19GDJUUiEWeHGO71gXlbOro0mUxwOBwLqvkmkwl9fX0oKipCZWVlSpoMgHOjcojfMIn0AtdMR2qJOOb7fD6ueFNVVYWioqK4SS9Q6UE0oPSaMzMzYTab0dnZiZycHFRVVaXM2tDj8XAnGKI3D6VCoKNLoluNB6QbsqKiAiUlJXG912TN5I/FYgHDMH4TnYnTHfFxEIvF0Gq1ETsbgbnP5DPPPIPf/OY3OHv2bELlkI2Njfjwww9RWVmZiMutfNJ1uVwLnMbMZjNOnz4NgUCA9evX8z4eTk1Nccf3UCAf1Pz8fFRVVWFgYAA5OTl+LakE0RTJ6CLS9PQ0pqenwbIssrOzkZuby0XFyarWA+d0r9PT00Er1sEQiYjD5S5pkMJgfn4+Vq1aldQNxu12c4RmMBhgMBjAsiwKCgqgVCqTHl0SkBbr8vLyiJ1sdFOHyWTi/A3o6JKvHIzkjAGgrq4uIZrpYCD6Z7Jmi8UCp9MJj8cDlUoFtVrNy9ejra0NO3bswFVXXYUHHnggoY54drsdWVlZ0FDm5ZdccgkOHjwY6yXPL9K12+3csZRoKOk3MxJ0Oh0mJyfR0LDQGs9kMqGrqwtisRi1tbXcMay/vx9SqZRrlwXOWfORYxLfIhkxWXc4HNBqtVAoFFxBhta3Bmu9jQeBo3Li1b3S+dZIREwPoNRqtbyOt4kAy7Jc+qSyshKFhYVBo8tQLcPxgGhfhUIhtFptzKRHN3WQ6JJ4TNB5YhK1k+kjQ0NDKc8Z2+12tLe3QyaToaSkhPPvNZlMXDs5HRHLZDI4nU7s27cPR44cwYEDB7Bu3bqUrTcOrHzSdbvnzKWJSTn5ME1PT0eMWgNB5n2tXbuW+xohBZJfDNRJDg0NgWEYlJfPVbED7Rb5kC0dYVZXV4dt6ySNBjQRu91u7kGL1hWMbCYKhQJVVVVJi6RpKZjZbIbdbucKJ6WlpSgtLeUVEScCdKNBdXV1yPeKngNHSI2u/JPoki8R0yZARKaYaJCmDnrdPp8PUqkUVqsVCoUCdXV1MUv9ogWR3I2Ojvq52wW+hnymzWYzWlpa8OSTT8LlckGr1eK2227DxRdf7Nc6vISx8km3r68PAwMDqKys9DuihYtaQ8Fms6GzsxMbNmzgcm1kKm2oETYkD6zRaKLW29LRVjwRJu2BQP54vd4FREznKUnLMLGpTPZkABrT09Po7e2FUqlETk4ORxBEk0srEBJJxHT+lDhyRQvSbEDe52DeDVlZWQt+j0QdoFQqodFoUpaf9/l8GBgYwMTEBIqKiuD1ev2iSzqST/SGa7PZ0N7eznXR8U197N27F59++in27NkDh8OBkydPYsuWLfj+97+f0PUlCSufdHU6HWQy2YJfqNlsRn9/f1RHEpfLhba2NqhUKgwPD3NEHo4IJycnodPpUF1d7Sdz4bPunp4ernCT6MkNtCsY+UN8cj0eD9eKnIoKPQHZ1EQiEWpra4PqT+mImCZiPvrWUKDTJ3zyp9GC7voiRAyAk1MZjUa43W40NDRELT+LB4ToCwoKuBZ0ArJR0+smHhnBHM2iAcuyGBoawvj4OOrr63nrak+cOIG7774b11xzDXbt2pXyOW0JwsonXeI5EAiSQ9q4cSOv65AH86uvvsKqVavCSsvI64nvbGdnpx85hGphBc61HTMM45cbTjaIX0FPTw8UCgXXzOHz+fwiy2h78/mAbmWNRWMcjogjTY+wWq3o7OyETCZDTU1NUguRNDweD4aGhjAyMoKMjAzuBBRpbl0i4PV60dPTA7PZHBXR000dhIijaeoA5lI37e3tyMvL410QdTgcePLJJ/Hll1/iwIEDEc2gljjOX9L1eDw4efJkRBMN4NxIF7lcDoPBgO985zshXxuuSOZ0OmE0GjlyIF4COTk5kMvl0Ol0sFgscTU3xALa6rGmpsavcEMbu5CHDcACIo417UEizLKyMpSVlSUswqRF+8GIODMzE2NjY9Dr9RE9CxINshFLJBLU1tZyRB/4XgfOrSPvdTwRHrHYLCsrS0hEH9jUQfLxgQ0SMpkMQ0NDmJ6eRn19Pe/UTWtrK+655x785Cc/wT333LNco1saK590QzmNsSyLL7/8MqwDmM1mQ1dXF1c5z8rKCusaFm2RjBzhSJGMuOTL5XKuKSKeUTiREOuoHLoqTh40WifKp5JvsVg4k/ZURZgkSpuYmMDMzAxEIpEfoYVrQkkEfD4fBgcHMTk5GbEzkf6eYIWvSIY0gSCGPGScfbJkYPT9yJp1Oh0MBgMkEgkKCwv9LEdDfUbsdjueeOIJtLa24sCBA1HVXpY4VrbLWDiEe7AIGZF5YZGqyLGY0tCTb4uKivDd734XQqHQr+hFj8KhySHeYyfdxbZq1aqoR+UIhcIFQy3pAhIx/yZdSDQRezwev5boVM6zIs0VYrEYW7ZsgUQiCWmgk2gi1uv16OrqQlFRUVSj7Gk3s9LSUu7nCGZIE2xuHV2MJZ1dqQAZ467X6+Hz+fCtb32L8/c1mUzo7+/3a+ogTR/FxcU4ffo0du3ahe3bt+Ojjz5aCdEtL6yYnzKah4UevxOOjEgkG6sDmMlkQnd3N2Qy2YLJt8T4m556Sz9kY2Nj3BGfEHFOTg7vIz49KieaET2REMytyuPxcIRGVAEulwu5ubkoLS3lNplkF+poM3OtVusXYQYa6ARzMrPbHQAykJeXBaUyKyoidrvd6OrqgtPpxNq1axOSo+c7t87tdnNj6qurq1OasjIajejo6IBKpUJTUxP32Qy0liRm5CaTCW+99Rbeeecd6HQ6XHbZZZDL5bBarSlN/SwmVgzpRgJJo4yPj6O/vx/FxcVhR7KLRCK43W4IhcKoydbhcPjJsPiOiqYfMhLt0Ef84eFhvyM+SU3Qo3usVivXvJEMZ6pgEIlEyM/P53yCVSoVysvLOXII5uubaM8GsskUFxcvmBMWDIFOZnY78Pe/MxgZ8cLhcKCubhZq9dmI3r50hJkKnwZ6bp1arebUAWTQqF6vx9DQEDfKPJHNMzS8Xi9nibpmzZqIRTqRSITc3Fy0t7fj8OHDuOuuu3DLLbfg7NmzOHnyZMLWtRywYnK6ZKpqMBw7dgwajQZ9fX3IyclBdXV12A8gy7I4efIkKioqkJOTw7nhR0I0zQ3xgBbrG41GbtAiMY9OtQTM5XL5jaQPpfVNVKswDXoumlarjXmT+X//T4D+fgZqNeDxAOPjDP75nz1QqdgFqglCxDKZDAaDAVlZWairq0taTj4YTCYTOjo6oFQq/cbbENDpK/JZcbvdXMcXeb9jIWK9Xo/Ozk5OU87XivSXv/wlvvrqKxw8eBBarTbq+y4zrPxCWijStVgsOHbsGLKzs3nJZkiRbGZmBuPj49zMKRJVBiMGOtoh1eJk9+vT9x4dHcXQ0BAKCgogEok4xyea0HJychJePCI547GxMc7sJ9rrR5KBhTri08WqRHR1/e53QsjlAOGgsTHge9/zQatd+AiQfPXMzAxyc3PhdDo5Ig6UryUaXq+X87ptaGiIqpklsOOL1uTSUrBQxTfiRGaz2dDQ0MCrm40Usu+9917ccMMNuOOOO1LWELLIOP9I1+l0oru7G1arFUKhEHV1dREn7obK25IIjcjAaCcwgUCAyclJzqAlldEOPSpHo9EsKEQEIzSaGOIxziH3JoL7RD1IgblWOrIk6/b5fOjv74dKpUJlZWVCNrhDhwSYmGBQWAj4fMDoKPBP/+RDaan/I0CmOBQXF6O8vJy7d6DnbLB1hyM0PiD3jibCjARak0vIONgGQtJW0TiRWa1W7NmzB2fPnsXBgwdRU1MT93qXEVY+6QLgnIvoiRAqlQodHR1Qq9VBCwyxKhKIG5bH44FYLPZrt83JyeEl74kV9NQIrVbLu38+8AELZpyTnZ0dduOw2+3o7u4Gy7JR3TsekHXPzs5icHAQbrcbYrHYr7053vZVkwn461+FMJkAn4/B2rVefPvbLMhHgUixvF4vb8+CUE0G0RIxSd+43W7U19cnPU9PbyB6vR6Tk5PweDx+Tnfhiowsy+Kzzz7D/fffj5tvvhm33377+RLd0lj5pMuyLPr6+rgjPh2FEEd6WkYTqyLB5XKhr68PJpPJr7khVLstObJFozwIhWSMyuHj10DkXoODg5iamkqaSUso0AYxxGoylOFPNBtIINxuwGgExGKAFNJpQ/NESLH4RJaE0MhQ04GBgYT5CkcDYl+q0WigUqkWmMPTqSC5XA6z2Yzi4mLs2bMHXV1dOHjwYMhp18nGFVdcgZKSEvz6179elPvjfCBdYM6wXKVSLXjQBgYGIJFIUFJSErPdInnwx8bGoNFoIk6HIN9jsVi4tARRHtAPF608CAU6Z5wMz4Bg66Y3EJ1OB4fDAYVCgeLiYi6ST0Xemuhe+aQxQm0ggUTM9wRiNpvR0dGB7Ow58/tknVyCnUAcDgc8Hg+kUik0Gg3y8vKS3uhAQKJ6lmUj+uySVNDw8DB27tyJvr4+FBQU4KqrrsLll1+Oiy++OCVrpnHo0CHccMMNuOKKK9Kkm2y43e6goz1GRkbg9XpRUVERtd0i3dygUqlQUVER11GJKA8IEUcq1JHW5Nzc3JTnjGn5WXV1tZ/6gJ4IwLc7LRq4XC50dXXB7Xajrq4uZt1r4AnEbDZHbEKhi1VkikOqwLIshoeHMTo6ypnT0KkgvkWvWEFGWpHUHB+YzWY88sgj6OvrwyuvvAKZTIaTJ09CKBRi27ZtCV1fJFitVmzduhXXXnstvv766zTpJhuhSHdiYoJzACPyL77NDcSLgc+Yn1gRrFAnkUjgcrkgEAhSOmoc8E9jaLXakPcm3Wlk3aRoSUeVkYYVBiLUqPNEIjCSJy23WVlZYBgGOp0O5eXlqKioSOlxnkTWoUxiQuXkaT1urLltp9PJGarzlb+xLIuPP/4YDz74IG6//Xb89Kc/TZlqJxRuvvlmXHbZZbDZbPjss8/SpJtsBJIuSSOQSQwkOqOr98FIgTQ3uFwu1NbWpjTSIU5cU1NTKCoqAsuyC/KVySrU0TnEWNMYZAQO+RMoXQunxaVHnVdVVaW0+GKz2XD27Fl4PB5kZmbCZrMBwIKIOBmkQn7ner0+6sia5LZp1UQ0REz/zqOZImE2m7F7924MDQ3h4MGDiZorFhfefPNNfPbZZ3jllVfwxhtvpEk3FSBOY+GKZMS8mY7OCCkoFAoYjUYYDIakNjcEQ6RROcku1JGBjJGmKMQCl8vl57pGCjAkpZKRkYGhoSHY7XbU19en1GuWHOfHxsYWzIMLNPwhmzbdlh1vSkWn06GrqwslJSUoLy9PmAyMDxH7fD60t7dDKpWitraWd3R75MgRPPTQQ7jzzjtx0003LXp0S9Dc3AyDwQCRSASj0Qi73Y4f/vCHeO211xZjOecH6brdbng8nqiLZE6nE/39/ZiYmPBzACMPVjTFl1hA0hjRjspJRKHO7XZzo7cjaZkTBaLFNRqNGB0dhcFggFgs9mttjlZ5EAtIV1c0QzBDTYyItjjqdru5Lr76+vqkS+8C1R7T09NwOBzIyspCQUEBr9SEyWTC7t27MTo6igMHDqCioiKpa44HSznSXVHeCw899BAUCgWam5vR1NTEi0DI5Ia8vDxs2bKFc2yy2+0wGo3cSBlSfKGtGOPd4elROfX19VGPyqEfdgK6UNfX1xfU84DoPIkUSqPRRO1AFg8YhoHb7cbQ0BCys7Oxbt06CIVCjhRmZmbQ19fHHfXp9zwRmx/tgNbY2BjV+x7K8CfQVStUbpuYyPf19fFWwSQCJJBgGAajo6MoKCjwK47qdDoMDg766bbJ+52Tk4OPPvoIDz/8MHbs2IEbbrgh5dGtz+fDtm3bMDg4CIZh8Pzzz6e8SJcorKhIt7OzE0ePHkVLSwtOnjwJl8uFNWvWoKmpCZs2bcLq1au56Emv12NgYABCoRC1tbURIw3aeNpoNPrlhwkp8C0a0S2ssbbPRoPAQp3VaoXb7YZCoUBFRQXy8vJSNknB7XZz+XU+XYLBUir05hfNhAua8KLprIoFbrfbLyIm/hgulwsymQy1tbXIzs5OafqKpFHCjc4JjIjvu+8+tLe3w+124z//8z+xdetWXHrppSktMJJ1TUxMoLi4GH/961+xe/duHD9+PKVriBLnR3ohEA6HA21tbTh69ChaW1vxzTffQCwWQywWQyqVYv/+/aivr49516YtDYnxDBGLE1KgFQ+0/EytVqOioiKlEQNpjXa5XNBoNFzRi8zuSmZHHV2wqaysRHFxcUwPbrDND4BfbjtYntVut6OjowNisRharTZlmwzgr8hQq9UAEHWRMR5YrVa0t7dzc/j4bFIsy+Jvf/sbHnnkEdx9993YsmULTp06hfb2duzZsyeh64sWBw4cwLFjx/Dqq68u6joi4Pwk3UD88Y9/xKOPPoorrrgCMpkMx48f5zrYNm3ahKamJjQ3NyMvLy/mDz49qof4ysrlcshkMuj1emRmZkKr1aZM6A7MEdXQ0BAmJiZCFgiTWagj0yMyMzMTXqQDgudZibE68Q2YnZ1FXV1dQrr4ogGZFZabmxuU8II5ryXKXJ383icnJ1FfX8/br9ZgMODBBx+ETqfDyy+/zNmMLjb27duHp556CoWFhTh06NCSUEyEQZp0gbkcZn5+vl8qgYymbmlpQUtLC44fP84N8iMkvH79+pj73Z1OJzo7O2E2m5GVlQWHw8GRGX1ETlbES0xSioqKUFlZGZUMK7BQFyi5i1Q0SsSo81jh8XgwNjaGgYEBiEQiMAzD5bbJ+56MqJKAmPLMzs5GNSsMCD73LVq/BrPZjPb29pDWj8HAsiwOHTqERx99FLt27cK///u/LxllAo13330XDz30ENrb21Oe5ogCadKNBm63G2fOnOGI+KuvvoJIJMLGjRuxceNGNDc3o7a2NiyBBY7Kof1tfT6fnx8uicxo/XC8Bt+J8pkNBJ+OOqlUiqmpqaSNOo8EogyYMyOv4yRokaJKYn8ZLwwGA2eylIgUUijntWBaXJrsGxoaeKtR9Ho9HnjgAZhMJvzqV79CSUlJXGtONsrKytDW1uYn8VtiSJNuPGBZFmazGcePH0dLSwuOHTvGjcIh0XBzczNHrJ2dndDr9VFFl6TwQpMZrWXla8NIj6ypra1NyXGaLtTp9XqYTCaIxWKoVCrk5+fH7QDGF3TemK8yILDDKxSZ8QGR39ntdt5+s7EilF+Dy+VCTk4OysvLkZubGzGVw7IsPvzwQ+zZswf3338/tm/fviSj276+PsjlcqjVanz55Zf4j//4D/T09Cz2ssIhTbqJBsuyGBsb46LhY8eOYXR0FCzLoqamBjt27EBTU1PUbbA0yENFiNjlcvlJqLKzszlCJ5X5/v5+lJSULGiuSDZIV5VOp+Ny1nSRMdmFOpvNho6ODk4ZEGveOJR7WaDrWuDaybyyVMrACIhXhF6vR1VVlV9XIJHd0V115L3R6XS4//77Ybfb8V//9V/crL5UweVy4e6778bf//53sCyLvXv34l//9V+DvvbkyZO49tpr4fV6UVRUhBdffBFNTU0pXW+USJNusvHmm2/iwIEDuP322+FyudDa2oqTJ0/C6/Vi3bp1XDTc0NAQM9nQxS6j0Qiz2QyWZSGTyWC1WiGXy9HQ0JDSIh2tyAhnrp2sQh3JyU9PTyfNoyKcdC0jIwN6vR5SqRR1dXUpVUUA51IZoTraghn+7N+/HybT3Oy6W265Bbt27UppqzvBxMQEPvvsM/zoRz9CV1cXNm/ejOnp6ZSaOiURadJNNgwGA7Kzsxe07trtdpw4cQLHjh1DS0sLOjo6kJOTw2mHm5ubYx7vQ8an6PV65Ofnw+12c/lhOi2RrIKRzWZDZ2cnxGIxamtroyZ7OrcdrKMulDcGAWmhXQz5HYnsx8bGoFAo4Ha7AcDPdS2ZBVKv14uenh5YLBY0NDTwdmGbnZ3Frl274HK5cPHFF6O7uxunTp3C4cOHU9p+HQwFBQXo7e1dKVOB06S7VMCyLGZmZhakJTQaDRcNb9y4ETk5OSHJhvbXDSbyp/W3pGBEqt+EjOOJyMKNOo8XfAp1AoHAb5JCKqZX0CC616ysLNTU1HApnkS1CEcC2WzIPD6+FqV/+tOf8MQTT+Dhhx/GT37ykyVV+X/99dfx1ltv4aOPPlrspSQKadJdyvD5fOjp6eFI+MSJE7DZbFi9ejVHxGvWrIFUKsU333wDm82G7OxsVFVV8TYpCdQP07k+kmPlU/CjR53T0zmSCbpQNzU1BZvNhszMTBQWFiZkE+ELksqYmZlBXV0dr4iMbqChNxGaiPkqVTweD7q6uuB0OtHQ0MBbkTIzM4OdO3eCYRi8+OKLcU+/SDT27t2Lt99+G3/5y19SnldOItKku9zgcrnQ1tbGEXFbWxvMZjPkcjl27tyJTZs2oaqqKmbSI7k+2iwHWNjZRcggWRI0vrBYLOjo6OBc0AK7AZNdqDMYDOjs7OQUKfFsNi6Xyy+a59MQMTMzg+7u7qi6+ViWxfvvv4+9e/fiF7/4Ba655polFd0CwB133AGr1YqXXnopZqP6JYo06S5ndHZ24sc//jF27NiBkpIStLa2orW1lStebdy4keuoi8eOMpQpOTCnpKipqVmUyjxRRYRrMghW7GJZ1s+GMZYcK8mbW63WqHKn0YI+iRDpmkwm4+xGhUIhGhsbeefNp6amsHPnTojFYrzwwgu8fXJTiaNHj2L37t34+9//vthLSQbSpLuc4fP5YLfbFxQ6yNy2o0eP4tixY2htbYXBYEBdXR1XqFu/fn3MjRY6nQ6dnZ3IysqCTCaD2WyGw+HgdKzkaJ+sajNJZcTqNRtvoY4MZky2OU4wkIGY/f39yMrKgsfj4Yzsade1wPeeZVm8++672LdvHx599FH88Ic/THl063Q68dprr+Hw4cN47733Qr7u5ZdfxgMPPODX4PDiiy/iH//xH1OxzGQjTbrnCzweD7755hvO5KetrQ0Mw+CCCy7gGjnq6urC5m8dDge6urrg8/kWjBsnonw6P0zPHEvE0ErSOk0GIyYylcGnUMcwDLq6ugAg4mDGZMDlcqGjowMMw/jJ0OjBm0QySN77I0eOoKSkBO+88w6ysrLw3HPPLVq3lkajwYYNG2A2m1dqFMsHadI9X8GyLCwWC06cOIGWlha0traiq6sLSqUSTU1NaGpqwubNm6FWq+F2u3H69Gl4vV5uPhkfkJljdH6Yz1ikYGslblzRjI6JF8EKdQqFAgUFBSkt1MUyOodE87/85S/x6aefch1pzc3NeOmllxYlh2swGNDW1obHHnssTbpBsKJMzNNYCDKx9+KLL+bGYZOH+9ixYzh69CheffVV9Pb2wuPx4JJLLsG1114LmUzGTd+IBIFAwDl6EdBjkXp7e/3GItGGMwRkKGNubi42bywkuaUAAAy4SURBVN6c0vloEokEcrkcg4ODyMvLQ3NzM1eoI77LyS7UORwOznqyubmZd8pmamoK99xzD7KysvDxxx9DqVRyM/4Wq2iWyiGqyxErNtJ94YUX8PLLL8NoNOKaa67BM888s+Qqt0sFe/fuxSeffIKf//znGB8fx7Fjx3Dq1Cm4XC6sXbuWyw83NjbGlb+lZ6UZjUauWETGLDU0NCRU88sHxFB+amoqbEdbsjrqSDv50NAQtFotlEol73W/8847ePbZZ/HYY4/hqquuWlKf7yNHjqQj3VD/sFJJd2xsDCUlJbDb7WhsbMQHH3yANWvWLPayliSMRmPQKQYOhwOnTp3yM4FXKBR+Jj/xdIJNTk6ip6cHubm5EAqFfjnKRI5FCgUyfbigoAAajSamFuR4CnV2ux3t7e2Qy+WoqanhHTlPTExgx44dyM/Px7PPPptyj2A+SJPueZheINZ0U1NTyMjIQFlZ2SKvaOkilMhfJpPhwgsvxIUXXghgLiqbnZ1Fa2srjh49it///vcYGhpCRUUFN5euqakpogk8OUqLRCI0Nzf7FaroyRAjIyN+RBbtWKRQIC20ZrMZq1evjrn9VSAQICcnx+/9o/XDJK0iFosXWF+Ojo5idHQUdXV1vKN7n8+H3//+93j++efxxBNP4Morr1xS0W0a/LBiSbe7uxuXXnoppqam8NJLL6XzTAkAwzAoKCjA5ZdfjssvvxzAObPulpYWfPTRR9i3bx8sFgsaGxu5iHjdunWQyWRwuVzo7u6G2WxGbW1t0KM0HSmSjZImsp6eHm7MDSE8PqbeBESGVlZWBq1Wm3DSEolEyM/P94s+6ULd8PAwTCYTJBIJ1Go1vF4vXC5XxELd+Pg4duzYgcLCQnz88ccpT8OkkTgs+/TCrbfeihMnTvh97dVXX8X69esBAMPDw7jyyivx0ksv4Tvf+U5C7vnee+/hkUcegdlsxtatW3Hw4MGUFn6WOlwul58J/JkzZ+B0OmGz2XD11VfjuuuuQ21tbdyyMtpfwul0+mlYA+0XXS4XOjs74fP5UF9fn3IZGMuy3DBSIkOjO+o8Hg83hZf8DEKhED6fD7/73e/w4osv4sknn8QVV1yxaNHtO++8g/vvvx9CoRAPPfQQbrzxxkVZxzLB+ZfTpbF7927k5uZi165dCbnea6+9hmuvvRZisRiXX345brzxRmzfvj0h116J2LNnD44cOYLrrruOK9T19vZCpVL55YeLiopiJpRADStd6GJZFgaDAbW1tVCpVAn+6SKDzEnLy8sL2bodWKhraWnBs88+C5FIhLy8PDz++OO46KKLUm4dSUDG1R89ehRCoRAXXHABzpw5syQ73ZYIzj/S/fzzz7FlyxZYLBZs3boVTz31FCeZSiR27tyJiooK7NixI+HXXikYHh5e4LNLOq5aWlq4Qt3s7Cy0Wi2XH96wYUNc+Vur1YpvvvkGDMNAKpXCZrMlfCxSONAGOdGMzvH5fPjtb3+Ll19+GTfccAPEYjGOHz+O+vp63HfffUlZayT84Q9/wPvvv4/f/va3AIDt27fjqquuwrXXXrso61kGOP8KaU888QTOnDmDjIwM3HrrrUkh3LGxMbz77rs4fPhwwq+9klBeXr7gawzDoKysDGVlZdy0AK/Xi7Nnz6KlpQV/+MMf8PDDD4NlWT8T+Pr6+ohVfnoKbqAMjB6LRBohYhmLFAkmkwkdHR0oLCxEc3Mz71TKyMgIfv7zn6O8vBwff/zxkvGWHR4e9pu+W1ZWhvHx8UVc0fLFiiXdP//5z3FfI1y++Pjx49i+fTv279+P2trauO+VBiAUCrF27VqsXbsWN998M5cyICbw+/fvR2dnJ/Ly8jilxKZNm/w8ZXU6HXp6eqBUKrFp06YFZCcWixcUushYJIPBgKGhobBjkSLB5/Nxo3MaGxuhUCh4f99vfvMbHDhwAPv378ell166pJQJLpfL770UCATpOkaMWLGkmwgcOHAg6Nc/+eQT3Hnnnfjf//1frmCXRuLBMAwyMzPxve99D9/73vcAnBsPRIp0b775JsbGxlBRUQGXy4WMjAw899xzftOXI0Emk0Emk3E+s4TsjUYjJiYm0N3dDZZlF9heBhI6GZ1TXFyM5uZm3vcfHh7GXXfdhaqqKnz66acpHVXPF8XFxThy5Aj395GREXzrW99avAUtY6zYnG4y0djYiA8//NDvuJVotLS04I477sD777+f1hhHwOeff45bbrkFmzZtgkKhwMmTJ+FwOBaYwMc7LcNisXBFOnoskkKhgF6v5xpx+No/+nw+vPHGG3jllVfw9NNPY+vWrUsquqUxOTmJjRs34tSpU/D5fLjoootw5syZRR/xs4Rx/uV0kwVi5r1161bua5dccgkOHjyYsHvcd999aGtrw+DgYMKuuZJRUFCAw4cP+21OTqeTM4E/cOAAvv76a8hkMmzcuJEj4lWrVvHOtRKCpXOsbrcbo6Oj6O7u5gi9s7OT11ikoaEh3HnnndBqtfj88895pyGSiXAbvUqlwuOPP841yjz99NNpwo0R6Uh3CcJgMCA3NxcajQafffZZOtJNAIhsjAwIbW1tRX9/P0pLSzkSbmpqglKp5D06p7u7G3a7HQ0NDcjIyIg4FmlychL19fV4++238frrr+Ppp5/GJZdcsiSiW7LRnzp1CqdOnUp/5uLH+ScZWwlIk25yQVQOtAm80WhEfX39AhN4GtGMzqHHIv3iF7/Al19+CYfDgR/84AfYsmUL/u3f/m3RtLc00ht9wpFOLyxFROqmSyO5EAgE0Gg00Gg0nN7U7XZzJvD//d//jXvvvRcCgQAbNmxAfX09/va3v+G6667Dtm3beJmrMwyDjIwM/O53v0NHRwfefPNNbNq0CadPn8bx48cTag8ZD9Jt8qlDOtJdwkhk1JFu4YwNxAT++eefx4svvoh169ZhdHQURUVFXDfdpk2bQqol+vv7cdddd2Ht2rV44oknFj0PGmmjT0e6CUM60j2fYTabsXPnTr8Wzh/84AfpFk4eYBiG88n96quvUFhYCJZluXbmo0eP4uDBg5iamkJNTQ1HxOvXr8f//M//4K233sJzzz2H7373u0sidxtKBplG6pAm3fMAhw4dwj/8wz+gtLQUwJza4v/+7//SLZw8wTAMHnzwQb+/l5SU4Oqrr8bVV18NYE5S1tnZiZaWFrz//vu47bbbsHnzZnz++ecrbbR4GnEiTbpLGAMDAwm5TrqFM/kgI9IbGxtxww038B51lMb5h+RY8qexpJBu4Uw9Fptw33vvPaxduxYajQY33XQTvF4vr+8bGBhI53OTjDTpngcoLi7G6Ogo9/eRkZGgJjTxwOl04le/+hX+5V/+JaHXTSM26PV6tLS0oLu7G4ODg3j77bcXe0lpzCNNuucBtm3bhkOHDmFqagoTExP44osvcNlllyX0HnV1dTh8+DDMZnNCr5tGbLjxxhshl8shFouxfv16TE9PL/aS0phHOqd7HiAVLZxtbW1oa2vDY489ltDrphEf0vajSw9p0j1PcP311+P6669P2vXT4vrFQdp+dPkhTbpppLGMkbYfXX5I53TTWPJwuVz42c9+Bq1Wi9raWvzxj39c7CUtedx222344IMP0oS7BJEm3TSWPHQ6HS655BJ0dXXhz3/+M2666Sa43e7FXtaSBW0/WlNTg5qaGvz0pz9d7GWlMY90eiGNJQ+1Wo0f/ehHAACtVguRSASbzbZk5oelCi+88AJefvllGI1GXHPNNXjmmWeC6oEzMjLg8XgWYYVp8EHa8CaNZYXXX38db731Fj766KPFXkrKMTY2hpKSEm5CxQcffIA1a9Ys9rLSCI604U0ayx979+7F22+/jb/85S+LvZRFQUlJCQBgamoKGRkZ6c6xZYp0TjeNZYE77rgDHR0d+Pzzz1FcXJzw6/t8Plx66aXQarWoq6vDoUOHEn6PeNHd3Q2NRoOGhgbcd999aZneMkWk9EIaaSw6GIb5NoDHWJb9fhLvwQBQsyw7zjDMP87frzlZ94uwlgMAmgK+fBPLsqfn/70cwJ8B/Ixl2c9Svb404kM6vZDGcsAFAJoZhumhvnYny7J/TdQN2Lnog1ivVQI4nahrx7CWWyP8+zDDMH8C8G0AadJdZkhHummkMQ+GYe4DcD+AaQDbWJZdUuOYGYbZwrLs5wzDKAD8H4D7WZY9ssjLSiNKpEk3jTQCwDDMDwE8AaCBXUIPCMMwfwawFoAdwAGWZZ9Z5CWlEQPSpJtGGkHAMMwIgAtYlp1Z7LWksbKQVi+kkQYAhmGqGIZRz///hQAcacJNIxlIF9LSSGMOuQD+yjCMEMAUgJ8s8nrSWKH4/7aWao2wHcqvAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# 训练数据集\n", "X_train = np.array([[3, 3, 3], [4, 3, 2], [2, 1, 2], [1, 1, 1], [-1, 0, 1],\n", " [2, -2, 1]])\n", "y_train = np.array([[1, 1, 1, 0, 0, 0]])\n", "# 构建实例，进行训练\n", "clf = LogisticRegression()\n", "clf.fit(X_train, y_train)\n", "clf.draw(X_train, y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题6.3\n", "  写出最大熵模型学习的DFP算法。（关于一般的DFP算法参见附录B）" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "**第1步：** \n", "最大熵模型为：$$\n", "\\begin{array}{cl}\n", "{\\max } & {H(p)=-\\sum_{x, y} P(x) P(y | x) \\log P(y | x)} \\\\ \n", "{\\text {st.}} &\n", "{E_p(f_i)-E_{\\hat{p}}(f_i)=0, \\quad i=1,2, \\cdots,n} \\\\ \n", "& {\\sum_y P(y | x)=1}\n", "\\end{array}$$引入拉格朗日乘子，定义拉格朗日函数：$$\n", "L(P, w)=\\sum_{xy} P(x) P(y | x) \\log P(y | x)+w_0 \\left(1-\\sum_y P(y | x)\\right) \\\\\n", "+\\sum_{i=1} w_i\\left(\\sum_{xy} P(x, y) f_i(x, y)-\\sum_{xy} P(x, y) P(y | x) f_i(x, y)\\right)$$\n", "最优化原始问题为：$$\\min_{P \\in C} \\max_{w} L(P,w)$$对偶问题为：$$\\max_{w} \\min_{P \\in C} L(P,w)$$令$$\\Psi(w) = \\min_{P \\in C} L(P,w) = L(P_w, w)$$$\\Psi(w)$称为对偶函数，同时，其解记作$$P_w = \\mathop{\\arg \\min}_{P \\in C} L(P,w) = P_w(y|x)$$求$L(P,w)$对$P(y|x)$的偏导数，并令偏导数等于0，解得：$$P_w(y | x)=\\frac{1}{Z_w(x)} \\exp \\left(\\sum_{i=1}^n w_i f_i (x, y)\\right)$$其中：$$Z_w(x)=\\sum_y \\exp \\left(\\sum_{i=1}^n w_i f_i(x, y)\\right)$$则最大熵模型目标函数表示为$$\\varphi(w)=\\min_{w \\in R_n} \\Psi(w) = \\sum_{x} P(x) \\log \\sum_{y} \\exp \\left(\\sum_{i=1}^{n} w_{i} f_{i}(x, y)\\right)-\\sum_{x, y} P(x, y) \\sum_{i=1}^{n} w_{i} f_{i}(x, y)$$ " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**第2步：** \n", "DFP的$G_{k+1}$的迭代公式为：$$G_{k+1}=G_k+\\frac{\\delta_k \\delta_k^T}{\\delta_k^T y_k}-\\frac{G_k y_k y_k^T G_k}{y_k^T G_k y_k}$$ \n", "**最大熵模型的DFP算法：** \n", "输入：目标函数$\\varphi(w)$，梯度$g(w) = \\nabla g(w)$，精度要求$\\varepsilon$； \n", "输出：$\\varphi(w)$的极小值点$w^*$ \n", "(1)选定初始点$w^{(0)}$，取$G_0$为正定对称矩阵，置$k=0$ \n", "(2)计算$g_k=g(w^{(k)})$，若$\\|g_k\\| < \\varepsilon$，则停止计算，得近似解$w^*=w^{(k)}$，否则转(3) \n", "(3)置$p_k=-G_kg_k$ \n", "(4)一维搜索：求$\\lambda_k$使得$$\\varphi\\left(w^{(k)}+\\lambda_k P_k\\right)=\\min _{\\lambda \\geqslant 0} \\varphi\\left(w^{(k)}+\\lambda P_{k}\\right)$$(5)置$w^{(k+1)}=w^{(k)}+\\lambda_k p_k$ \n", "(6)计算$g_{k+1}=g(w^{(k+1)})$，若$\\|g_{k+1}\\| < \\varepsilon$，则停止计算，得近似解$w^*=w^{(k+1)}$；否则，按照迭代式算出$G_{k+1}$ \n", "(7)置$k=k+1$，转(3) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "参考代码：https://github.com/wzyonggege/statistical-learning-method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第07章 支持向量机/7.support-vector-machine.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第7章 支持向量机" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1．支持向量机最简单的情况是线性可分支持向量机，或硬间隔支持向量机。构建它的条件是训练数据线性可分。其学习策略是最大间隔法。可以表示为凸二次规划问题，其原始最优化问题为\n", "\n", "$$\\min _{w, b} \\frac{1}{2}\\|w\\|^{2}$$\n", "\n", "$$s.t. \\quad y_{i}\\left(w \\cdot x_{i}+b\\right)-1 \\geqslant 0, \\quad i=1,2, \\cdots, N$$\n", "\n", "求得最优化问题的解为$w^*$，$b^*$，得到线性可分支持向量机，分离超平面是\n", "\n", "$$w^{*} \\cdot x+b^{*}=0$$\n", "\n", "分类决策函数是\n", "\n", "$$f(x)=\\operatorname{sign}\\left(w^{*} \\cdot x+b^{*}\\right)$$\n", "\n", "最大间隔法中，函数间隔与几何间隔是重要的概念。\n", "\n", "线性可分支持向量机的最优解存在且唯一。位于间隔边界上的实例点为支持向量。最优分离超平面由支持向量完全决定。\n", "二次规划问题的对偶问题是\n", "$$\\min \\frac{1}{2} \\sum_{i=1}^{N} \\sum_{j=1}^{N} \\alpha_{i} \\alpha_{j} y_{i} y_{j}\\left(x_{i} \\cdot x_{j}\\right)-\\sum_{i=1}^{N} \\alpha_{i}$$\n", "\n", "$$s.t. \\quad \\sum_{i=1}^{N} \\alpha_{i} y_{i}=0$$\n", "\n", "$$\\alpha_{i} \\geqslant 0, \\quad i=1,2, \\cdots, N$$\n", "\n", "通常，通过求解对偶问题学习线性可分支持向量机，即首先求解对偶问题的最优值\n", " \n", "$a^*$，然后求最优值$w^*$和$b^*$，得出分离超平面和分类决策函数。\n", "\n", "2．现实中训练数据是线性可分的情形较少，训练数据往往是近似线性可分的，这时使用线性支持向量机，或软间隔支持向量机。线性支持向量机是最基本的支持向量机。\n", "\n", "对于噪声或例外，通过引入松弛变量$\\xi_{\\mathrm{i}}$，使其“可分”，得到线性支持向量机学习的凸二次规划问题，其原始最优化问题是\n", "\n", "$$\\min _{w, b, \\xi} \\frac{1}{2}\\|w\\|^{2}+C \\sum_{i=1}^{N} \\xi_{i}$$\n", "\n", "$$s.t. \\quad y_{i}\\left(w \\cdot x_{i}+b\\right) \\geqslant 1-\\xi_{i}, \\quad i=1,2, \\cdots, N$$\n", "\n", "$$\\xi_{i} \\geqslant 0, \\quad i=1,2, \\cdots, N$$\n", "\n", "求解原始最优化问题的解$w^*$和$b^*$，得到线性支持向量机，其分离超平面为\n", "\n", "$$w^{*} \\cdot x+b^{*}=0$$\n", "\n", "分类决策函数为\n", "\n", "$$f(x)=\\operatorname{sign}\\left(w^{*} \\cdot x+b^{*}\\right)$$\n", "\n", "线性可分支持向量机的解$w^*$唯一但$b^*$不唯一。对偶问题是\n", "\n", "$$\\min _{\\alpha} \\frac{1}{2} \\sum_{i=1}^{N} \\sum_{j=1}^{N} \\alpha_{i} \\alpha_{j} y_{i} y_{j}\\left(x_{i} \\cdot x_{j}\\right)-\\sum_{i=1}^{N} \\alpha_{i}$$\n", "\n", "$$s.t. \\quad \\sum_{i=1}^{N} \\alpha_{i} y_{i}=0$$\n", "\n", "$$0 \\leqslant \\alpha_{i} \\leqslant C, \\quad i=1,2, \\cdots, N$$\n", "\n", "线性支持向量机的对偶学习算法，首先求解对偶问题得到最优解$\\alpha^*$，然后求原始问题最优解$w^*$和$b^*$，得出分离超平面和分类决策函数。\n", "\n", "对偶问题的解$\\alpha^*$中满$\\alpha_{i}^{*}>0$的实例点$x_i$称为支持向量。支持向量可在间隔边界上，也可在间隔边界与分离超平面之间，或者在分离超平面误分一侧。最优分离超平面由支持向量完全决定。\n", "\n", "线性支持向量机学习等价于最小化二阶范数正则化的合页函数\n", "\n", "$$\\sum_{i=1}^{N}\\left[1-y_{i}\\left(w \\cdot x_{i}+b\\right)\\right]_{+}+\\lambda\\|w\\|^{2}$$\n", "\n", "3．非线性支持向量机\n", "\n", "对于输入空间中的非线性分类问题，可以通过非线性变换将它转化为某个高维特征空间中的线性分类问题，在高维特征空间中学习线性支持向量机。由于在线性支持向量机学习的对偶问题里，目标函数和分类决策函数都只涉及实例与实例之间的内积，所以不需要显式地指定非线性变换，而是用核函数来替换当中的内积。核函数表示，通过一个非线性转换后的两个实例间的内积。具体地，$K(x,z)$是一个核函数，或正定核，意味着存在一个从输入空间x到特征空间的映射$\\mathcal{X} \\rightarrow \\mathcal{H}$，对任意$\\mathcal{X}$，有\n", "\n", "$$K(x, z)=\\phi(x) \\cdot \\phi(z)$$\n", "\n", "对称函数$K(x,z)$为正定核的充要条件如下：对任意$$\\mathrm{x}_{\\mathrm{i}} \\in \\mathcal{X}, \\quad \\mathrm{i}=1,2, \\ldots, \\mathrm{m}$$，任意正整数$m$，对称函数$K(x,z)$对应的Gram矩阵是半正定的。\n", "\n", "所以，在线性支持向量机学习的对偶问题中，用核函数$K(x,z)$替代内积，求解得到的就是非线性支持向量机\n", "\n", "$$f(x)=\\operatorname{sign}\\left(\\sum_{i=1}^{N} \\alpha_{i}^{*} y_{i} K\\left(x, x_{i}\\right)+b^{*}\\right)$$\n", "\n", "4．SMO算法\n", "\n", "SMO算法是支持向量机学习的一种快速算法，其特点是不断地将原二次规划问题分解为只有两个变量的二次规划子问题，并对子问题进行解析求解，直到所有变量满足KKT条件为止。这样通过启发式的方法得到原二次规划问题的最优解。因为子问题有解析解，所以每次计算子问题都很快，虽然计算子问题次数很多，但在总体上还是高效的。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "分离超平面：$w^Tx+b=0$\n", "\n", "点到直线距离：$r=\\frac{|w^Tx+b|}{||w||_2}$\n", "\n", "$||w||_2$为2-范数：$||w||_2=\\sqrt[2]{\\sum^m_{i=1}w_i^2}$\n", "\n", "直线为超平面，样本可表示为：\n", "\n", "$w^Tx+b\\ \\geq+1$\n", "\n", "$w^Tx+b\\ \\leq+1$\n", "\n", "#### margin：\n", "\n", "**函数间隔**：$label(w^Tx+b)\\ or\\ y_i(w^Tx+b)$\n", "\n", "**几何间隔**：$r=\\frac{label(w^Tx+b)}{||w||_2}$，当数据被正确分类时，几何间隔就是点到超平面的距离\n", "\n", "为了求几何间隔最大，SVM基本问题可以转化为求解:($\\frac{r^*}{||w||}$为几何间隔，(${r^*}$为函数间隔)\n", "\n", "$$\\max\\ \\frac{r^*}{||w||}$$\n", "\n", "$$(subject\\ to)\\ y_i({w^T}x_i+{b})\\geq {r^*},\\ i=1,2,..,m$$\n", "\n", "分类点几何间隔最大，同时被正确分类。但这个方程并非凸函数求解，所以要先①将方程转化为凸函数，②用拉格朗日乘子法和KKT条件求解对偶问题。\n", "\n", "①转化为凸函数：\n", "\n", "先令${r^*}=1$，方便计算（参照衡量，不影响评价结果）\n", "\n", "$$\\max\\ \\frac{1}{||w||}$$\n", "\n", "$$s.t.\\ y_i({w^T}x_i+{b})\\geq {1},\\ i=1,2,..,m$$\n", "\n", "再将$\\max\\ \\frac{1}{||w||}$转化成$\\min\\ \\frac{1}{2}||w||^2$求解凸函数，1/2是为了求导之后方便计算。\n", "\n", "$$\\min\\ \\frac{1}{2}||w||^2$$\n", "\n", "$$s.t.\\ y_i(w^Tx_i+b)\\geq 1,\\ i=1,2,..,m$$\n", "\n", "②用拉格朗日乘子法和KKT条件求解最优值：\n", "\n", "$$\\min\\ \\frac{1}{2}||w||^2$$\n", "\n", "$$s.t.\\ -y_i(w^Tx_i+b)+1\\leq 0,\\ i=1,2,..,m$$\n", "\n", "整合成：\n", "\n", "$$L(w, b, \\alpha) = \\frac{1}{2}||w||^2+\\sum^m_{i=1}\\alpha_i(-y_i(w^Tx_i+b)+1)$$\n", "\n", "推导：$\\min\\ f(x)=\\min \\max\\ L(w, b, \\alpha)\\geq \\max \\min\\ L(w, b, \\alpha)$\n", "\n", "根据KKT条件：\n", "\n", "$$\\frac{\\partial }{\\partial w}L(w, b, \\alpha)=w-\\sum\\alpha_iy_ix_i=0,\\ w=\\sum\\alpha_iy_ix_i$$\n", "\n", "$$\\frac{\\partial }{\\partial b}L(w, b, \\alpha)=\\sum\\alpha_iy_i=0$$\n", "\n", "代入$ L(w, b, \\alpha)$\n", "\n", "$\\min\\ L(w, b, \\alpha)=\\frac{1}{2}||w||^2+\\sum^m_{i=1}\\alpha_i(-y_i(w^Tx_i+b)+1)$\n", "\n", "$\\qquad\\qquad\\qquad=\\frac{1}{2}w^Tw-\\sum^m_{i=1}\\alpha_iy_iw^Tx_i-b\\sum^m_{i=1}\\alpha_iy_i+\\sum^m_{i=1}\\alpha_i$\n", "\n", "$\\qquad\\qquad\\qquad=\\frac{1}{2}w^T\\sum\\alpha_iy_ix_i-\\sum^m_{i=1}\\alpha_iy_iw^Tx_i+\\sum^m_{i=1}\\alpha_i$\n", "\n", "$\\qquad\\qquad\\qquad=\\sum^m_{i=1}\\alpha_i-\\frac{1}{2}\\sum^m_{i=1}\\alpha_iy_iw^Tx_i$\n", "\n", "$\\qquad\\qquad\\qquad=\\sum^m_{i=1}\\alpha_i-\\frac{1}{2}\\sum^m_{i,j=1}\\alpha_i\\alpha_jy_iy_j(x_ix_j)$\n", "\n", "再把max问题转成min问题：\n", "\n", "$\\max\\ \\sum^m_{i=1}\\alpha_i-\\frac{1}{2}\\sum^m_{i,j=1}\\alpha_i\\alpha_jy_iy_j(x_ix_j)=\\min \\frac{1}{2}\\sum^m_{i,j=1}\\alpha_i\\alpha_jy_iy_j(x_ix_j)-\\sum^m_{i=1}\\alpha_i$\n", "\n", "$s.t.\\ \\sum^m_{i=1}\\alpha_iy_i=0,$\n", "\n", "$ \\alpha_i \\geq 0,i=1,2,...,m$\n", "\n", "以上为SVM对偶问题的对偶形式\n", "\n", "-----\n", "#### kernel\n", "\n", "在低维空间计算获得高维空间的计算结果，也就是说计算结果满足高维（满足高维，才能说明高维下线性可分）。\n", "\n", "#### soft margin & slack variable\n", "\n", "引入松弛变量$\\xi\\geq0$，对应数据点允许偏离的functional margin 的量。\n", "\n", "目标函数：\n", "\n", "$$\\min\\ \\frac{1}{2}||w||^2+C\\sum\\xi_i\\qquad s.t.\\ y_i(w^Tx_i+b)\\geq1-\\xi_i$$ \n", "\n", "对偶问题：\n", "\n", "$$\\max\\ \\sum^m_{i=1}\\alpha_i-\\frac{1}{2}\\sum^m_{i,j=1}\\alpha_i\\alpha_jy_iy_j(x_ix_j)=\\min \\frac{1}{2}\\sum^m_{i,j=1}\\alpha_i\\alpha_jy_iy_j(x_ix_j)-\\sum^m_{i=1}\\alpha_i$$\n", "\n", "$$s.t.\\ C\\geq\\alpha_i \\geq 0,i=1,2,...,m\\quad \\sum^m_{i=1}\\alpha_iy_i=0,$$\n", "\n", "-----\n", "\n", "#### Sequential Minimal Optimization\n", "\n", "首先定义特征到结果的输出函数：$u=w^Tx+b$.\n", "\n", "因为$w=\\sum\\alpha_iy_ix_i$\n", "\n", "有$u=\\sum y_i\\alpha_iK(x_i, x)-b$\n", "\n", "\n", "----\n", "\n", "$$\\max \\sum^m_{i=1}\\alpha_i-\\frac{1}{2}\\sum^m_{i=1}\\sum^m_{j=1}\\alpha_i\\alpha_jy_iy_j<\\phi(x_i)^T,\\phi(x_j)>$$\n", "\n", "$$s.t.\\ \\sum^m_{i=1}\\alpha_iy_i=0,$$\n", "\n", "$$ \\alpha_i \\geq 0,i=1,2,...,m$$\n", "\n", "-----\n", "参考资料：\n", "\n", "[1] :[Lagrange Multiplier and KKT](http://blog.csdn.net/xianlingmao/article/details/7919597)\n", "\n", "[2] :[推导SVM](https://my.oschina.net/dfsj66011/blog/517766)\n", "\n", "[3] :[机器学习算法实践-支持向量机(SVM)算法原理](http://pytlab.org/2017/08/15/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E7%AE%97%E6%B3%95%E5%AE%9E%E8%B7%B5-%E6%94%AF%E6%8C%81%E5%90%91%E9%87%8F%E6%9C%BA-SVM-%E7%AE%97%E6%B3%95%E5%8E%9F%E7%90%86/)\n", "\n", "[4] :[Python实现SVM](http://blog.csdn.net/wds2006sdo/article/details/53156589)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "from sklearn.datasets import load_iris\n", "from sklearn.model_selection import train_test_split\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# data\n", "def create_data():\n", " iris = load_iris()\n", " df = pd.DataFrame(iris.data, columns=iris.feature_names)\n", " df['label'] = iris.target\n", " df.columns = [\n", " 'sepal length', 'sepal width', 'petal length', 'petal width', 'label'\n", " ]\n", " data = np.array(df.iloc[:100, [0, 1, -1]])\n", " for i in range(len(data)):\n", " if data[i, -1] == 0:\n", " data[i, -1] = -1\n", " # print(data)\n", " return data[:, :2], data[:, -1]" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "X, y = create_data()\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD5CAYAAAA3Os7hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZ80lEQVR4nO3df4xdZZ3H8fd3h1k7KjIBxlVm6hbFNAqtVkaQdENccbdaa20QsURWq6zsGlwwuBgxBrUxKYZEXSTR8CMrClvsVqzA8mNVbPyxUjMFbNdWIiraGdhlKLbIWrSM3/3j3mmnd+7M3Oeee+55nmc+r2Qyc8995vT7nINfz5zzOeeauyMiIun7s6oLEBGRzlBDFxHJhBq6iEgm1NBFRDKhhi4ikgk1dBGRTBzV6kAz6wFGgDF3X9Xw3jrgKmCsvugad79+tvUdf/zxvmjRoqBiRUTmu+3btz/h7gPN3mu5oQOXALuBF8zw/tfc/YOtrmzRokWMjIwE/PMiImJmv57pvZZOuZjZEPAWYNajbhERqU6r59A/D3wE+NMsY95uZjvMbLOZLWw2wMwuNLMRMxsZHx8PrVVERGYxZ0M3s1XA4+6+fZZhtwOL3H0p8G3gxmaD3P1adx929+GBgaangEREpE2tnENfDqw2s5XAAuAFZnaTu58/OcDd904Zfx3wmc6WKSLSOQcPHmR0dJRnnnmm6lJmtGDBAoaGhujt7W35d+Zs6O5+OXA5gJm9Hvjnqc28vvzF7v5Y/eVqahdPRUSiNDo6ytFHH82iRYsws6rLmcbd2bt3L6Ojo5x44okt/17bOXQzW29mq+svLzazn5rZT4CLgXXtrldEpGzPPPMMxx13XJTNHMDMOO6444L/ggiJLeLuW4Gt9Z+vmLL80FG8SG62PDDGVfc8xKP7DnBCfx+XrVjMmmWDVZclBcXazCe1U19QQxeZb7Y8MMblt+7kwMEJAMb2HeDyW3cCqKlLdHTrv8gsrrrnoUPNfNKBgxNcdc9DFVUkubj77rtZvHgxJ510EldeeWVH1qmGLjKLR/cdCFou0oqJiQkuuugi7rrrLnbt2sXGjRvZtWtX4fXqlIvILE7o72OsSfM+ob+vgmqkKp2+jvLjH/+Yk046iZe+9KUArF27lm9+85u88pWvLFSnjtBFZnHZisX09fYcsayvt4fLViyuqCLptsnrKGP7DuAcvo6y5YGxOX93JmNjYyxcePiG+qGhIcbG2l/fJDV0kVmsWTbIhrOXMNjfhwGD/X1sOHuJLojOI2VcR3H3acs6kbrRKReROaxZNqgGPo+VcR1laGiIPXv2HHo9OjrKCSec0Pb6JukIXURkFjNdLylyHeW1r30tP//5z/nVr37FH//4R2655RZWr1499y/OQQ1dRGQWZVxHOeqoo7jmmmtYsWIFr3jFKzj33HM5+eSTi5aqUy4iIrOZPN3W6buFV65cycqVKztR4iFq6CIic0jlOopOuYiIZEINXUQkE2roIiKZUEMXEcmEGrqISCbU0CUbWx4YY/mV93LiR/+D5VfeW+hZGyJle9/73scLX/hCTjnllI6tUw1dslDGA5REyrRu3Truvvvujq5TDV2yoA+ikFLt2ASfOwU+2V/7vmNT4VWeeeaZHHvssR0o7jDdWCRZ0AdRSGl2bILbL4aD9f+W9u+pvQZYem51dTWhI3TJQhkPUBIB4DvrDzfzSQcP1JZHRg1dsqAPopDS7B8NW14hnXKRLJT1ACURjhmqnWZptjwyauiSjVQeoCSJOeuKI8+hA/T21ZYXcN5557F161aeeOIJhoaG+NSnPsUFF1xQaJ1q6FJYpz9AVyQqkxc+v7O+dprlmKFaMy94QXTjxo0dKO5IauhSyGT+ezIyOJn/BtTUJR9Lz40u0dKMLopKIcp/i8RDDV0KUf5bUuXuVZcwq3bqU0OXQpT/lhQtWLCAvXv3RtvU3Z29e/eyYMGCoN/TOXQp5LIVi484hw7Kf0v8hoaGGB0dZXx8vOpSZrRgwQKGhsKikWroUojy35Ki3t5eTjzxxKrL6Dg1dClM+W+ROLTc0M2sBxgBxtx9VcN7zwG+ApwK7AXe6e6PdLBOkSQoky9VCrkoegmwe4b3LgB+6+4nAZ8DPlO0MJHU6JnsUrWWGrqZDQFvAa6fYcjbgBvrP28GzjIzK16eSDqUyZeqtXqE/nngI8CfZnh/ENgD4O7PAvuB4xoHmdmFZjZiZiMxX10WaYcy+VK1ORu6ma0CHnf37bMNa7JsWsDT3a9192F3Hx4YGAgoUyR+yuRL1Vo5Ql8OrDazR4BbgDeY2U0NY0aBhQBmdhRwDPBkB+sUiZ6eyS5Vm7Ohu/vl7j7k7ouAtcC97n5+w7DbgPfUfz6nPibOW7BESrJm2SAbzl7CYH8fBgz297Hh7CVKuUjXtJ1DN7P1wIi73wbcAHzVzB6mdmS+tkP1iSRFmXypUlBDd/etwNb6z1dMWf4M8I5OFiby8S072bhtDxPu9Jhx3ukL+fSaJVWXJRIt3SkqUfr4lp3cdN9vDr2ecD/0Wk1dpDk9bVGitHFbk89wnGW5iKihS6QmZrimPtNyEVFDl0j1zHCj8UzLRUQNXSJ13ukLg5aLiC6KSqQmL3wq5SLSOqvq/p/h4WEfGRmp5N8WEUmVmW139+Fm7+kIXZp613U/4oe/OPz0huUvO5ab339GhRVVR884l1ToHLpM09jMAX74iyd513U/qqii6ugZ55ISNXSZprGZz7U8Z3rGuaREDV1kFnrGuaREDV1kFnrGuaREDV2mWf6yY4OW50zPOJeUqKHLNDe//4xpzXu+plz0jHNJiXLoIiIJUQ5dgpWVvQ5Zr/LfImHU0GWayez1ZFxvMnsNFGqoIestqwaRnOkcukxTVvY6ZL3Kf4uEU0OXacrKXoesV/lvkXBq6DJNWdnrkPUq/y0STg1dpikrex2yXuW/RcLpoqhMM3nRsdMJk5D1llWDSM6UQxcRSYhy6CWIISMdWkMMNYtIedTQ2xBDRjq0hhhqFpFy6aJoG2LISIfWEEPNIlIuNfQ2xJCRDq0hhppFpFxq6G2IISMdWkMMNYtIudTQ2xBDRjq0hhhqFpFy6aJoG2LISIfWEEPNIlIu5dBFRBJSKIduZguA7wHPqY/f7O6faBizDrgKGKsvusbdry9StHTex7fsZOO2PUy402PGeacv5NNrlhQeG0u+PZY6RKrSyimXPwBvcPenzawX+IGZ3eXu9zWM+5q7f7DzJUonfHzLTm667zeHXk+4H3rd2KhDxsaSb4+lDpEqzXlR1Guerr/srX9Vc55G2rZx256Wl4eMjSXfHksdIlVqKeViZj1m9iDwOPAtd9/WZNjbzWyHmW02s4UzrOdCMxsxs5Hx8fECZUuoiRmulTRbHjI2lnx7LHWIVKmlhu7uE+7+amAIOM3MTmkYcjuwyN2XAt8GbpxhPde6+7C7Dw8MDBSpWwL1mLW8PGRsLPn2WOoQqVJQDt3d9wFbgTc1LN/r7n+ov7wOOLUj1UnHnHd60z+ami4PGRtLvj2WOkSqNGdDN7MBM+uv/9wHvBH4WcOYF095uRrY3ckipbhPr1nC+a97yaGj7B4zzn/dS5omV0LGrlk2yIazlzDY34cBg/19bDh7SdcvRMZSh0iV5syhm9lSaqdQeqj9H8Amd19vZuuBEXe/zcw2UGvkzwJPAh9w95/NuFKUQxcRacdsOXTdWNSmsjLPIfnvMtcdMr8Ut0VydmyC76yH/aNwzBCcdQUsPbfqqqQC+oCLDisr8xyS/y5z3SHzS3FbJGfHJrj9YjhYT+zs31N7DWrqcgQ9nKsNZWWeQ/LfZa47ZH4pbovkfGf94WY+6eCB2nKRKdTQ21BW5jkk/13mukPml+K2SM7+0bDlMm+pobehrMxzSP67zHWHzC/FbZGcY4bClsu8pYbehrIyzyH57zLXHTK/FLdFcs66Anob/g+yt6+2XGQKXRRtQ1nPFp+82FdGsiNk3SHzS3FbJGfywqdSLjIHxRZFRBKi2KIAcWTLJXHKw0dNDX2eiCFbLolTHj56uig6T8SQLZfEKQ8fPTX0eSKGbLkkTnn46KmhzxMxZMslccrDR08NfZ6IIVsuiVMePnq6KDpPxJAtl8QpDx895dBFRBIyr3PoZeWpQ9Yby3O9lS2PTO6Z7tznF6JL2yLrhl5WnjpkvbE811vZ8sjknunOfX4hurgtsr4oWlaeOmS9sTzXW9nyyOSe6c59fiG6uC2ybuhl5alD1hvLc72VLY9M7pnu3OcXoovbIuuGXlaeOmS9sTzXW9nyyOSe6c59fiG6uC2ybuhl5alD1hvLc72VLY9M7pnu3OcXoovbIuuLomXlqUPWG8tzvZUtj0zume7c5xeii9tCOXQRkYTM6xx6WZRvF0nEHZfC9i+DT4D1wKnrYNVni683wpy9GnoblG8XScQdl8LIDYdf+8Th10WaeqQ5+6wvipZF+XaRRGz/ctjyVkWas1dDb4Py7SKJ8Imw5a2KNGevht4G5dtFEmE9YctbFWnOXg29Dcq3iyTi1HVhy1sVac5eF0XboHy7SCImL3x2OuUSac5eOXQRkYQUyqGb2QLge8Bz6uM3u/snGsY8B/gKcCqwF3inuz9SsO6mQvPfqT0DPCRbnvu2KDXnG5JNLquOMucXYUa6Y0LnlvO2aNDKKZc/AG9w96fNrBf4gZnd5e73TRlzAfBbdz/JzNYCnwHe2eliQ/PfqT0DPCRbnvu2KDXnG5JNLquOMucXaUa6I0LnlvO2aGLOi6Je83T9ZW/9q/E8zduAG+s/bwbOMut83CI0/53aM8BDsuW5b4tSc74h2eSy6ihzfpFmpDsidG45b4smWkq5mFmPmT0IPA58y923NQwZBPYAuPuzwH7guCbrudDMRsxsZHx8PLjY0Px3as8AD8mW574tSs35hmSTy6qjzPlFmpHuiNC55bwtmmipobv7hLu/GhgCTjOzUxqGNDsan9aF3P1adx929+GBgYHgYkPz36k9AzwkW577tig15xuSTS6rjjLnF2lGuiNC55bztmgiKIfu7vuArcCbGt4aBRYCmNlRwDHAkx2o7wih+e/UngEeki3PfVuUmvMNySaXVUeZ84s0I90RoXPLeVs00UrKZQA46O77zKwPeCO1i55T3Qa8B/gRcA5wr5eQhwzNf6f2DPCQbHnu26LUnG9INrmsOsqcX6QZ6Y4InVvO26KJOXPoZraU2gXPHmpH9Jvcfb2ZrQdG3P22erTxq8Ayakfma939l7OtVzl0EZFwhXLo7r6DWqNuXH7FlJ+fAd5RpEgRESkm+1v/k7uZRroj5GaTGG5MKfNmmtRunIphf0Qq64ae3M000h0hN5vEcGNKmTfTpHbjVAz7I2JZP20xuZtppDtCbjaJ4caUMm+mSe3GqRj2R8SybujJ3Uwj3RFys0kMN6aUeTNNajdOxbA/IpZ1Q0/uZhrpjpCbTWK4MaXMm2lSu3Eqhv0RsawbenI300h3hNxsEsONKWXeTJPajVMx7I+IZd3Q1ywbZMPZSxjs78OAwf4+Npy9RBdE57ul58Jbr4ZjFgJW+/7Wq5tfVAsZG0O9oePLml9q682EPuBCRCQhhW4sEpn3Qj4MIxap1RxLtjyWOtqkhi4ym5APw4hFajXHki2PpY4Csj6HLlJYyIdhxCK1mmPJlsdSRwFq6CKzCfkwjFikVnMs2fJY6ihADV1kNiEfhhGL1GqOJVseSx0FqKGLzCbkwzBikVrNsWTLY6mjADV0kdms+iwMX3D46NZ6aq9jvLg4KbWaY8mWx1JHAcqhi4gkRDl0KVeK2d2yai4r/53iNpauU0OXYlLM7pZVc1n57xS3sVRC59ClmBSzu2XVXFb+O8VtLJVQQ5diUszullVzWfnvFLexVEINXYpJMbtbVs1l5b9T3MZSCTV0KSbF7G5ZNZeV/05xG0sl1NClmBSzu2XVXFb+O8VtLJVQDl1EJCGz5dB1hC752LEJPncKfLK/9n3Hpu6vt6waRFqgHLrkoaysdsh6lReXiukIXfJQVlY7ZL3Ki0vF1NAlD2VltUPWq7y4VEwNXfJQVlY7ZL3Ki0vF1NAlD2VltUPWq7y4VEwNXfJQVlY7ZL3Ki0vFlEMXEUlIoRy6mS00s++a2W4z+6mZXdJkzOvNbL+ZPVj/0t+YqUsxT628ePm03aLWSg79WeDD7n6/mR0NbDezb7n7roZx33f3VZ0vUbouxTy18uLl03aL3pxH6O7+mLvfX//5d8BuYLDswqRCKeaplRcvn7Zb9IIuiprZImAZsK3J22eY2U/M7C4zO3mG37/QzEbMbGR8fDy4WOmSFPPUyouXT9stei03dDN7PvB14EPu/lTD2/cDf+nurwK+AGxptg53v9bdh919eGBgoN2apWwp5qmVFy+ftlv0WmroZtZLrZnf7O63Nr7v7k+5+9P1n+8Ees3s+I5WKt2TYp5aefHyabtFr5WUiwE3ALvdvemDnc3sRfVxmNlp9fXu7WSh0kUp5qmVFy+ftlv05syhm9lfAd8HdgJ/qi/+GPASAHf/kpl9EPgAtUTMAeBSd/+v2darHLqISLjZcuhzxhbd/QeAzTHmGuCa9sqTtu3YVEsY7B+tncc864r5fbR0x6Ww/cu1D2W2ntpHvxX9tCCRhOh56KlSJvhId1wKIzccfu0Th1+rqcs8oWe5pEqZ4CNt/3LYcpEMqaGnSpngI/lE2HKRDKmhp0qZ4CNZT9hykQypoadKmeAjnboubLlIhtTQU6VM8JFWfRaGLzh8RG49tde6ICrziJ6HLiKSkEI59PlkywNjXHXPQzy67wAn9Pdx2YrFrFmW0YMlc8+t5z6/GGgbR00NvW7LA2NcfutODhyspSLG9h3g8lt3AuTR1HPPrec+vxhoG0dP59DrrrrnoUPNfNKBgxNcdc9DFVXUYbnn1nOfXwy0jaOnhl736L4DQcuTk3tuPff5xUDbOHpq6HUn9PcFLU9O7rn13OcXA23j6Kmh1122YjF9vUfehNLX28NlKxZXVFGH5Z5bz31+MdA2jp4uitZNXvjMNuUyedEq14RC7vOLgbZx9JRDFxFJyGw5dJ1yEUnBjk3wuVPgk/217zs2pbFu6SqdchGJXZn5b2XLs6IjdJHYlZn/VrY8K2roIrErM/+tbHlW1NBFYldm/lvZ8qyooYvErsz8t7LlWVFDF4ldmc++13P1s6IcuohIQpRDFxGZB9TQRUQyoYYuIpIJNXQRkUyooYuIZEINXUQkE2roIiKZUEMXEcnEnA3dzBaa2XfNbLeZ/dTMLmkyxszsajN72Mx2mNlryilXCtFzr0Wy1srz0J8FPuzu95vZ0cB2M/uWu++aMubNwMvrX6cDX6x/l1joudci2ZvzCN3dH3P3++s//w7YDTR+0ObbgK94zX1Av5m9uOPVSvv03GuR7AWdQzezRcAyYFvDW4PAnimvR5ne9DGzC81sxMxGxsfHwyqVYvTca5HstdzQzez5wNeBD7n7U41vN/mVaU/9cvdr3X3Y3YcHBgbCKpVi9Nxrkey11NDNrJdaM7/Z3W9tMmQUWDjl9RDwaPHypGP03GuR7LWScjHgBmC3u392hmG3Ae+up11eB+x398c6WKcUpedei2SvlZTLcuDvgJ1m9mB92ceAlwC4+5eAO4GVwMPA74H3dr5UKWzpuWrgIhmbs6G7+w9ofo586hgHLupUUSIiEk53ioqIZEINXUQkE2roIiKZUEMXEcmEGrqISCbU0EVEMqGGLiKSCatFyCv4h83GgV9X8o/P7XjgiaqLKJHml66c5waaXyv+0t2bPgyrsoYeMzMbcffhqusoi+aXrpznBppfUTrlIiKSCTV0EZFMqKE3d23VBZRM80tXznMDza8QnUMXEcmEjtBFRDKhhi4ikol53dDNrMfMHjCzO5q8t87Mxs3swfrX31dRYxFm9oiZ7azXP9LkfTOzq83sYTPbYWavqaLOdrQwt9eb2f4p+y+pz9ozs34z22xmPzOz3WZ2RsP7ye47aGl+ye4/M1s8pe4HzewpM/tQw5hS9l8rn1iUs0uA3cALZnj/a+7+wS7WU4a/dveZbmR4M/Dy+tfpwBfr31Mx29wAvu/uq7pWTWf9C3C3u59jZn8OPLfh/dT33Vzzg0T3n7s/BLwaageNwBjwjYZhpey/eXuEbmZDwFuA66uupUJvA77iNfcB/Wb24qqLmu/M7AXAmdQ+yxd3/6O772sYluy+a3F+uTgL+IW7N94VX8r+m7cNHfg88BHgT7OMeXv9z6HNZrawS3V1kgP/aWbbzezCJu8PAnumvB6tL0vBXHMDOMPMfmJmd5nZyd0srqCXAuPAv9ZPCV5vZs9rGJPyvmtlfpDu/ptqLbCxyfJS9t+8bOhmtgp43N23zzLsdmCRuy8Fvg3c2JXiOmu5u7+G2p93F5nZmQ3vN/us2FRyrHPN7X5qz7x4FfAFYEu3CyzgKOA1wBfdfRnwf8BHG8akvO9amV/K+w+A+qmk1cC/N3u7ybLC+29eNnRgObDazB4BbgHeYGY3TR3g7nvd/Q/1l9cBp3a3xOLc/dH698epncM7rWHIKDD1L48h4NHuVFfMXHNz96fc/en6z3cCvWZ2fNcLbc8oMOru2+qvN1NrgI1jktx3tDC/xPffpDcD97v7/zZ5r5T9Ny8burtf7u5D7r6I2p9E97r7+VPHNJzPWk3t4mkyzOx5Znb05M/A3wL/3TDsNuDd9SvurwP2u/tjXS41WCtzM7MXmZnVfz6N2n/re7tdazvc/X+APWa2uL7oLGBXw7Ak9x20Nr+U998U59H8dAuUtP/me8rlCGa2Hhhx99uAi81sNfAs8CSwrsra2vAXwDfq/5s4Cvg3d7/bzP4RwN2/BNwJrAQeBn4PvLeiWkO1MrdzgA+Y2bPAAWCtp3Vb9D8BN9f/bP8l8N5M9t2kueaX9P4zs+cCfwP8w5Rlpe8/3fovIpKJeXnKRUQkR2roIiKZUEMXEcmEGrqISCbU0EVEMqGGLiKSCTV0EZFM/D/9eJgL33QQQgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.scatter(X[:50,0],X[:50,1], label='0')\n", "plt.scatter(X[50:,0],X[50:,1], label='1')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "class SVM:\n", " def __init__(self, max_iter=100, kernel='linear'):\n", " self.max_iter = max_iter\n", " self._kernel = kernel\n", "\n", " def init_args(self, features, labels):\n", " self.m, self.n = features.shape\n", " self.X = features\n", " self.Y = labels\n", " self.b = 0.0\n", "\n", " # 将Ei保存在一个列表里\n", " self.alpha = np.ones(self.m)\n", " self.E = [self._E(i) for i in range(self.m)]\n", " # 松弛变量\n", " self.C = 1.0\n", "\n", " def _KKT(self, i):\n", " y_g = self._g(i) * self.Y[i]\n", " if self.alpha[i] == 0:\n", " return y_g >= 1\n", " elif 0 < self.alpha[i] < self.C:\n", " return y_g == 1\n", " else:\n", " return y_g <= 1\n", "\n", " # g(x)预测值，输入xi（X[i]）\n", " def _g(self, i):\n", " r = self.b\n", " for j in range(self.m):\n", " r += self.alpha[j] * self.Y[j] * self.kernel(self.X[i], self.X[j])\n", " return r\n", "\n", " # 核函数\n", " def kernel(self, x1, x2):\n", " if self._kernel == 'linear':\n", " return sum([x1[k] * x2[k] for k in range(self.n)])\n", " elif self._kernel == 'poly':\n", " return (sum([x1[k] * x2[k] for k in range(self.n)]) + 1)**2\n", "\n", " return 0\n", "\n", " # E（x）为g(x)对输入x的预测值和y的差\n", " def _E(self, i):\n", " return self._g(i) - self.Y[i]\n", "\n", " def _init_alpha(self):\n", " # 外层循环首先遍历所有满足0= 0:\n", " j = min(range(self.m), key=lambda x: self.E[x])\n", " else:\n", " j = max(range(self.m), key=lambda x: self.E[x])\n", " return i, j\n", "\n", " def _compare(self, _alpha, L, H):\n", " if _alpha > H:\n", " return H\n", " elif _alpha < L:\n", " return L\n", " else:\n", " return _alpha\n", "\n", " def fit(self, features, labels):\n", " self.init_args(features, labels)\n", "\n", " for t in range(self.max_iter):\n", " # train\n", " i1, i2 = self._init_alpha()\n", "\n", " # 边界\n", " if self.Y[i1] == self.Y[i2]:\n", " L = max(0, self.alpha[i1] + self.alpha[i2] - self.C)\n", " H = min(self.C, self.alpha[i1] + self.alpha[i2])\n", " else:\n", " L = max(0, self.alpha[i2] - self.alpha[i1])\n", " H = min(self.C, self.C + self.alpha[i2] - self.alpha[i1])\n", "\n", " E1 = self.E[i1]\n", " E2 = self.E[i2]\n", " # eta=K11+K22-2K12\n", " eta = self.kernel(self.X[i1], self.X[i1]) + self.kernel(\n", " self.X[i2],\n", " self.X[i2]) - 2 * self.kernel(self.X[i1], self.X[i2])\n", " if eta <= 0:\n", " # print('eta <= 0')\n", " continue\n", "\n", " alpha2_new_unc = self.alpha[i2] + self.Y[i2] * (\n", " E1 - E2) / eta #此处有修改，根据书上应该是E1 - E2，书上130-131页\n", " alpha2_new = self._compare(alpha2_new_unc, L, H)\n", "\n", " alpha1_new = self.alpha[i1] + self.Y[i1] * self.Y[i2] * (\n", " self.alpha[i2] - alpha2_new)\n", "\n", " b1_new = -E1 - self.Y[i1] * self.kernel(self.X[i1], self.X[i1]) * (\n", " alpha1_new - self.alpha[i1]) - self.Y[i2] * self.kernel(\n", " self.X[i2],\n", " self.X[i1]) * (alpha2_new - self.alpha[i2]) + self.b\n", " b2_new = -E2 - self.Y[i1] * self.kernel(self.X[i1], self.X[i2]) * (\n", " alpha1_new - self.alpha[i1]) - self.Y[i2] * self.kernel(\n", " self.X[i2],\n", " self.X[i2]) * (alpha2_new - self.alpha[i2]) + self.b\n", "\n", " if 0 < alpha1_new < self.C:\n", " b_new = b1_new\n", " elif 0 < alpha2_new < self.C:\n", " b_new = b2_new\n", " else:\n", " # 选择中点\n", " b_new = (b1_new + b2_new) / 2\n", "\n", " # 更新参数\n", " self.alpha[i1] = alpha1_new\n", " self.alpha[i2] = alpha2_new\n", " self.b = b_new\n", "\n", " self.E[i1] = self._E(i1)\n", " self.E[i2] = self._E(i2)\n", " return 'train done!'\n", "\n", " def predict(self, data):\n", " r = self.b\n", " for i in range(self.m):\n", " r += self.alpha[i] * self.Y[i] * self.kernel(data, self.X[i])\n", "\n", " return 1 if r > 0 else -1\n", "\n", " def score(self, X_test, y_test):\n", " right_count = 0\n", " for i in range(len(X_test)):\n", " result = self.predict(X_test[i])\n", " if result == y_test[i]:\n", " right_count += 1\n", " return right_count / len(X_test)\n", "\n", " def _weight(self):\n", " # linear model\n", " yx = self.Y.reshape(-1, 1) * self.X\n", " self.w = np.dot(yx.T, self.alpha)\n", " return self.w" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "svm = SVM(max_iter=200)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'train done!'" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "svm.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.64" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "svm.score(X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### scikit-learn实例" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "SVC()" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.svm import SVC\n", "clf = SVC()\n", "clf.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.96" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.score(X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### sklearn.svm.SVC\n", "\n", "*(C=1.0, kernel='rbf', degree=3, gamma='auto', coef0=0.0, shrinking=True, probability=False,tol=0.001, cache_size=200, class_weight=None, verbose=False, max_iter=-1, decision_function_shape=None,random_state=None)*\n", "\n", "参数：\n", "\n", "- C：C-SVC的惩罚参数C?默认值是1.0\n", "\n", "C越大，相当于惩罚松弛变量，希望松弛变量接近0，即对误分类的惩罚增大，趋向于对训练集全分对的情况，这样对训练集测试时准确率很高，但泛化能力弱。C值小，对误分类的惩罚减小，允许容错，将他们当成噪声点，泛化能力较强。\n", "\n", "- kernel ：核函数，默认是rbf，可以是‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’ \n", " \n", " – 线性：u'v\n", " \n", " – 多项式：(gamma*u'*v + coef0)^degree\n", "\n", " – RBF函数：exp(-gamma|u-v|^2)\n", "\n", " – sigmoid：tanh(gamma*u'*v + coef0)\n", "\n", "\n", "- degree ：多项式poly函数的维度，默认是3，选择其他核函数时会被忽略。\n", "\n", "\n", "- gamma ： ‘rbf’,‘poly’ 和‘sigmoid’的核函数参数。默认是’auto’，则会选择1/n_features\n", "\n", "\n", "- coef0 ：核函数的常数项。对于‘poly’和 ‘sigmoid’有用。\n", "\n", "\n", "- probability ：是否采用概率估计？.默认为False\n", "\n", "\n", "- shrinking ：是否采用shrinking heuristic方法，默认为true\n", "\n", "\n", "- tol ：停止训练的误差值大小，默认为1e-3\n", "\n", "\n", "- cache_size ：核函数cache缓存大小，默认为200\n", "\n", "\n", "- class_weight ：类别的权重，字典形式传递。设置第几类的参数C为weight*C(C-SVC中的C)\n", "\n", "\n", "- verbose ：允许冗余输出？\n", "\n", "\n", "- max_iter ：最大迭代次数。-1为无限制。\n", "\n", "\n", "- decision_function_shape ：‘ovo’, ‘ovr’ or None, default=None3\n", "\n", "\n", "- random_state ：数据洗牌时的种子值，int值\n", "\n", "\n", "主要调节的参数有：C、kernel、degree、gamma、coef0。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第7章支持向量机-习题\n", "\n", "### 习题7.1\n", "  比较感知机的对偶形式与线性可分支持向景机的对偶形式。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "**感知机算法的原始形式：** \n", "给定一个训练数据集$$T=\\{(x_1,y_1),(x_2,y_2),\\cdots,(x_N,y_N)\\}$$其中，$x_i \\in \\mathcal{X} = R^n, y_i \\in \\mathcal{Y}=\\{-1,1\\}, i=1,2,\\cdots,N$，求参数$w,b$，使其为以下损失函数极小化问题的解：$$\\min_{w,b} L(w,b)=-\\sum_{x_i \\in M} y_i(w \\cdot x_i + b)$$其中M为误分类点的集合。 \n", "上式等价于：$$\\min_{w,b} L(w,b)=\\sum_{i=1}^N (-y_i(w \\cdot x_i + b))_+$$\n", "\n", "----\n", "\n", "**补充：** 合页损失函数$$L(y(w \\cdot x + b)) = [1-y(w \\cdot x + b)]_+$$下标“+”表示以下取正数的函数。$$[z]_+ = \\left\\{\\begin{array}{ll} z, & z>0 \\\\\n", "0, & z \\leqslant 0 \n", "\\end{array} \\right.$$当样本点$(x_i,y_i)$被正确分类且函数间隔（确信度）$y_i(w \\cdot x_i + b)$大于1时，损失是0，否则损失是$1-y_i(w \\cdot x_i + b)$。\n", "\n", "----\n", "\n", "**感知机算法的对偶形式：** \n", "$w,b$表示为$\\langle x_i,y_i \\rangle$的线性组合的形式，求其系数（线性组合的系数）$\\displaystyle w=\\sum_{i=1}^N \\alpha_i y_i x_i, b=\\sum_{i=1}^N \\alpha_i y_i$，满足：$$\n", "\\min_{w,b} L(w,b) = \\min_{\\alpha_i} L(\\alpha_i) = \\sum_{i=1}^N (-y_i (\\sum_{j=1}^N \\alpha_j y_j x_j \\cdot x_i + \\sum_{j=1}^N \\alpha_j y_j))_+$$ \n", "\n", "**线性可分支持向量机的原始问题：** \n", "$$\\begin{array}{cl} \n", "\\displaystyle \\min_{w,b} & \\displaystyle \\frac{1}{2} \\|w\\|^2 \\\\\n", "\\text{s.t.} & y_i(w \\cdot x_i + b) -1 \\geqslant 0, i=1,2,\\cdots,N\n", "\\end{array}$$ \n", "\n", "**线性可分支持向量机的对偶问题：** \n", "$$\\begin{array}{cl} \n", "\\displaystyle \\max_{\\alpha} & \\displaystyle -\\frac{1}{2} \\sum_{i=1}^N \\sum_{j=1}^N \\alpha_i \\alpha_j y_i y_j (x_i \\cdot x_j) + \\sum_{i=1}^N \n", "alpha_i \\\\\n", "\\text{s.t.} & \\displaystyle \\sum_{i=1}^N \\alpha_i y+i = 0 \\\\\n", "& \\alpha \\geqslant 0, i=1,2,\\cdots,N\n", "\\end{array}$$根据书上**定理7.2**，可得$\\displaystyle w^*=\\sum_{i=1}^N \\alpha_i^* y_j x_i, b^*=y_i-\\sum_{i=1}^N \\alpha^* y_i (x_i \\cdot x_j)$，可以看出$w,b$实质上也是将其表示为$\\langle x_i, x_j\\rangle$的线性组合形式。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题7.2\n", "\n", "  已知正例点$x_1=(1,2)^T,x_2=(2,3)^T,x_3=(3,3)^T$，负例点$x_4=(2,1)^T,x_5=(3,2)^T$，试求最大间隔分离平面和分类决策函数，并在图中挂出分离超平面、间隔边界及支持向量。 \n", "\n", "**解答：**" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "w = [[-1. 2.]]\n", "b = [-2.]\n", "support vectors = [[3. 2.]\n", " [1. 2.]\n", " [3. 3.]]\n" ] } ], "source": [ "%matplotlib inline\n", "from sklearn.svm import SVC\n", "\n", "# 加载数据\n", "X = [[1, 2], [2, 3], [3, 3], [2, 1], [3, 2]]\n", "y = [1, 1, 1, -1, -1]\n", "\n", "# 训练SVM模型\n", "clf = SVC(kernel='linear', C=10000)\n", "clf.fit(X, y)\n", "\n", "print(\"w =\", clf.coef_)\n", "print(\"b =\", clf.intercept_)\n", "print(\"support vectors =\", clf.support_vectors_)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**最大间隔分离超平面：**$-x^{(1)}+2x^{(2)}-2=0$ \n", "**分类决策函数：**$f(x)=\\text{sign}(-x^{(1)}+2x^{(2)}-2)$ \n", "**支持向量：**$x_1=(3,2)^T,x_2=(1,2)^T, x_3=(3,3)^T$ " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEKCAYAAAAW8vJGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3gU5fr/8feTkISE3nsTkCJNiiACUg3SBARBiigtbDjHctSjXxVU9AfoOR4OqLsh9EgTARFERDAiCkiVSChSBEnoNYGQkLL374+EHIKUQJKd3eR+Xddc1yYzO/kwxr0zM8/cjxERlFJKqWu8rA6glFLKvWhhUEoplYEWBqWUUhloYVBKKZWBFgallFIZaGFQSimVQT6rA2RVyZIlpWrVqlbHUEopj7J9+/azIlLqZus8vjBUrVqVbdu2WR1DKaU8ijHmz1ut00tJSimlMtDCoJRSKgMtDEoppTLQwqCUUioDLQxKKaUy0MKglFIqAy0MSinlYRITEzl//nyO7V8Lg1JKeYioqCjGjBlDlSpVeO2113Ls53j8A25KKZWbiQjGGABef/11FixYQNeuXXnqqady7GfqGYNSSrmh8+fP85///IdatWqxa9cuAMaNG8cff/zBihUr6NSpU479bD1jUEopN7J161bsdjsLFy4kISGBli1bcuXKFQCqV6/ukgxaGJRSymLXLhddvnyZdu3aATBkyBBsNhsNGzZ0eR4tDEopZZEDBw4QEhLCzp07Wbt2LQULFuTrr7+mcePGFC5c2LJceo9BKaVcKDk5mWXLlhEYGMj999/PlClTKFmyZPrlorZt21paFEDPGJRSyqUWLVrEwIEDqVixIu+99x7Dhg2jXLlyVsfKQAuDUkrlEBFh/fr12O12HnnkEZ5//nl69erFl19+Sbdu3ciXzz0/gvVSklJKZbPY2Fg+/fRT6tWrR9u2bVmzZg2JiYkA+Pv707NnT7ctCqBnDEople0GDBjAypUradq0KTNnzqRfv34EBARYHSvT9IxBKZXnXb58mcjISCIiIjh79uxdvffq1avMnz+ftm3bcuzYMQDefvttNm/ezNatW3nuuec8qiiAnjEopfKw3bt3M2XKFBYtWkT58uXx9vYmKiqKdu3a8fe//z39mYKbOXLkCKGhoUyfPp0zZ85Qo0YNjh49SoUKFWjWrJkL/xXZT88YlFJ50sKFC2nXrh2VKlVi79697N69m99++42oqCgCAwMZOnQob731FiLyl/eeP3+e+++/nw8++ICWLVuyevVqfv/9dx5++GEL/iXZT88YlPIkV65AUhIUKWJ1Eo8WHh7Oiy++yPfff0/9+vVTv5l2bAsWKUJQUBC9e/emY8eOlChRgsGDBzNr1iz279/PtGnTKF68OLNmzaJNmzZUqlTJ2n9MThARlyxAfmALEAHsBt69yTZ+wOfAQWAzUPVO+23SpIkoleudPSvyxBMivr4iPj4i9euLbN1qdSqP1aJFC1myZEnqF6dOiXTpknpcfXxEHnxQZOdOcTqdsnjxYvH19RU/Pz8BpG3btpKQkGBt+GwCbJNbfK668lLSVaC9iDQEGgGdjTEtbthmGHBBRGoAk4APXJhPKfckAh07wjffQGJi6hnDrl3Qvj2k3exUmbdjxw6OHz/OE088AU4ntGkD332XelyTkuDXX6FNG2ZNmUKfPn1wOp08/PDDREZG8sMPP+Dn52f1PyHHuawwpBWpy2lf+qQtN168ewKYk/Z6MdDBXGtErlRetWULHDiQ+qF1vcRECAmxJpMHW7t2Lb1798bb2xt+/DG1uCYnsw94gdQPHpKS6HHuHHa7HYfDQalSpXjggQesDe5CLr35bIzxNsbsBE4Da0Rk8w2bVACiAEQkGYgBStxkPyONMduMMdvOnDmT07GVstYff4DXTf5XvXoV9u51fR4Pd/nyZYoWLQpA0v79LElMpANQB3AA+wDi4ykZFYXNZqNs2bLExcVZF9gCLi0MIpIiIo2AisBDxph6N2xys7ODvwwJEJFQEWkqIk1LlSqVE1GVch+NGkFy8l+/7+8Pjzzi+jwerlixYpw4cQKAbrNn0ycxkUPABCAaeAugQAFIG2F04sQJihUrZlFaa1gyXFVELgLrgM43rIoGKgEYY/IBRYCcm/FaKU9Qpw489lhqIbgmX77UkUlDh1qXy8OICOHh4axbt44vvviC+Ph4XhgzhhUNG3Iof35eB0pD6rEtXhwGDgQgLCyM3r17Wxnd5VxWGIwxpYwxRdNe+wMdSTtru85yYEja6z5AeNrdc6Xyti++gDfegAoVoFgxePpp2L5dh61mwsWLF5kyZQp169alQ4cObNiwgVq1ajFjxgy6dOlCt82b8X71VShXLrUgPPMMbN0KBQqwadMmDh8+TI8ePaz+Z7jWrYYrZfcCNAB+BX4DIoGxad8fB/SQ/w1p/YLU4apbgPvutF8drqqUupVjx45JQECAANKiRQsJCwuT+Ph4iYyMlFKlSsmKFStu+d5du3ZJ+fLlZenSpS5M7DrcZriqEQ//g7xp06aybds2q2MopdxAQkICixYtIioqijfffBOACRMm0LlzZx588MEM227ZsoWePXvSunVrgoODadWqFcYY9uzZQ0hICAsWLGDKlCkMTLuklNsYY7aLSNObrtPCoJTydIcOHWLq1KnMnDmTc+fO0bBhQ7Zt23bH1taxsbF89tlnOBwO9u7dizGGcuXKMXToUEaMGEHFihVd9C9wPS0MSqlca9q0aQQFBeHl5UXPnj0JDg6mXbt23O0jUE6nE6fT6dbzJGSn2xWGvHEElFK5xunTp5kxYwatWrWidevWtG/fnrFjxzJixAgqVKhwz/v18vLC62bPi+RBehSUUm5PRPj555/T50p+4403WLNmDQDVq1fnnXfeyVJRUBnpGYNSyu117dqVVatWUbhwYWw2Gzabjdq1a1sdK9fSMwallNvZvXs3r732Gklp/aH69OnDtGnTOH78OJMnT9aikMP0jEEp5RYSExNZtmwZdrudH3/8ET8/P/r06UOzZs0Yqk94u5SeMSilLHf48GEqV65Mv379OHr0KB9++CHR0dEeP0Wmp9IzBqWUyzmdTtauXcuJEycYMmQIVapUoUePHvTq1YvAwEAdHWQxfY5BKeUy58+fZ/bs2TgcDg4ePEitWrXSHyxTrnW75xi0LCulXGL69OlUqFCBl19+mTJlyjBv3jwiIiK0KLghLQxKqRxx5coVZs2axZ49ewBo0KABzz77LBEREfz8888MGDAgT0yTmRPi4uKYNm0aFy5cyJH9a2FQSmWr/fv3849//IMKFSowdOhQFixYAMBDDz2Ew+GgQYMGFif0XHv37uX555+nfPnyjBw5ki+//DJHfo7efFZKZZs+ffqwZMkS8uXLx5NPPonNZqNNmzZWx/JoSUlJ6cN4161bh6+vL3379iU4OJiH02aZy25aGJRS9+zEiRMsWbKE0aNHY4yhYcOGNGrUiOHDh1O2bFmr43m06Ohopk2bxrRp0zhx4gRVqlRh4sSJDB06lJye0lgLg1LqrogI69evx263s3TpUpKTk2nTpg0NGjRgzJgxVsfzaE6nk/DwcOx2O8uXL8fpdNK5c2emTZtG586d8fb2dkkOLQxKqUzbv38/vXr1Ys+ePRQrVoznn3+eUaNGUbNmTaujebQLFy4wZ84cHA4H+/fvp0SJErzyyisEBQVRrVo1l+fRwqCUuq2IiAhOnjxJYGAglStXpmLFirz66qv069cPf39/q+N5tO3bt2O321mwYAHx8fE8/PDDfPbZZ/Tp04f8+fNblksLg1LqL65evcrixYux2+1s3LiRunXrEhkZSf78+Vm9erXV8TxafHw8ixYtwm63s2XLFgICAhg0aBA2m+0v049aRYerKqUymDNnDhUrVmTQoEGcPn2ajz76iJ9++kkfRMuigwcP8sorr1CxYkWeffZZYmNjmTJlCsePHyc0NNRtigLoGYNSeZ7T6WT16tU0aNCAChUqUKJECVq1akVwcDAdOnTQvkVZkJKSwsqVK7Hb7axevZp8+fLRs2dPRo8ezaOPPuq2xVZ7JSmVR509e5aZM2cSEhLC4cOHee+993jrrbesjpUrnDp1ihkzZjB16lSOHj1K+fLlCQoKYvjw4ZQvX97qeIDO+ayUuo6IMGzYMObPn8/Vq1d59NFHmThxIj179rQ6mke7Nv2ow+Fg8eLFJCUl0aFDByZNmkT37t3x8fGxOmKmaWFQKg+Ii4vj+++/p0ePHhhj8PLyYvjw4YwaNYp69epZHc+jXbp0iblz52K324mMjKRIkSIEBwczatQoj51pTguDUrnYvn37cDgczJkzh5iYGPbv30/NmjWZPn261dE8XmRkJA6Hg7CwMC5fvsyDDz7I9OnT6d+/PwUKFLA6XpZoYVAqF9q/fz82m43w8HB8fHzSe+vUqFHD6mgeLTExkaVLl2K32/npp5/w8/OjX79+BAcH89BDD7ntzeS7pYVBqVwiOjqa06dP07hxY0qWLMmJEyeYMGECQ4cOpXTp0lbH82hHjx4lNDSUadOmcfr0ae677z7+9a9/8dxzz1GiRAmr42U7lxUGY0wlIAwoCziBUBGZfMM2bYGvgMNp31oqIuNclVEpTyMi6b11vvrqK5o0acLmzZspXrw4u3fvzjV/wVrB6XSyZs0a7HY7X3/9NQDdunUjODiYTp065ephvK48Y0gGXhaRHcaYQsB2Y8waEdlzw3Y/iUg3F+ZSyiMtWrSIsWPH8vvvv1OiRAlefvllgoKC0tdrUbg3586dY9asWYSEhHDo0CFKly7Na6+9RlBQEFWqVLE6nku4rDCIyAngRNrrS8aYvUAF4MbCoJS6hR07dlC9enWKFClCbGwsxYoVIywsjL59+1raW8fTiQhbt27FbrezcOFCrl69SuvWrXn//ffp3bs3vr6+Vkd0LRFx+QJUBY4ChW/4flvgHBABrAIeuMX7RwLbgG2VK1cWpXKz+Ph4mTNnjjRv3lwAmTJlioiIOJ1Oi5N5vri4OJk+fbo0btxYAClYsKDYbDb57bffrI6W44BtcqvP6FutyKkFKAhsB3rfZF1hoGDa6y7AgTvtr0mTJjlwyJSyXnJysrz66qtSokQJAaR27doyefJkuXDhgtXRPN6+ffvkhRdekKJFiwog9erVE7vdLrGxsVZHc5nbFQaXjkoyxvgAS4B5IrL0xvUiEnvd62+MMXZjTEkROevKnEpZJSUlhZ07d9KkSRO8vb3Zvn077dq1Izg4mLZt2+p9gyxITk5m+fLl2O12vv/+e3x8fOjTpw82m41WrVrpsb2OK0clGWAGsFdE/nOLbcoCp0REjDEPkdr99ZyrMipllet765w4cYKoqCjKlCnDd99957JZu3Kr48ePM336dEJDQzl27BiVKlXi/fffZ/jw4ZQpU8bqeG7JlWcMjwCDgV3GmJ1p33sDqAwgIiFAH8BmjEkG4oH+aac8SuVKBw8eZOzYsX/prVO8eHEALQr3SERYt24ddrudZcuWkZycTGBgIHa7na5du+pxvQNXjkr6GbjtuZqIfAJ84ppESlnj0qVLnD9/nipVquDt7c23337r8b113MXFixcJCwsjJCSEvXv3Urx4cV544QVGjRqlT33fBX3yWSkXub63Ttu2bVmxYgXVqlXj5MmTeW84ZDbbuXMndrudefPmceXKFR566CFmz57NU089pdOP3gMtDErlsJUrV/Lhhx+yfv16/Pz86N+/PzabLX29FoV7k5CQkD796KZNm/D392fAgAHYbDaaNGlidTyPpoVBqRxw9OhRypYti6+vLxEREURHR/Phhx8ydOjQXNlbx5UOHz5MSEgIM2bM4Ny5c9SsWZNJkyYxZMgQihUrZnW8XEFncFMqm9zYW2fBggU89dRTJCQk4Ovrm6t76+S0lJQUvv32W+x2O6tWrcLLy4snnngCm81G+/bt9djeA53BTakclJiYyMcff4zD4eDQoUOUKlWK1157jYcffhhAW1VkwZkzZ9KH8R45coSyZcsyZswYRowYQcWKFa2Ol2tpYVDqHogIx44do0KFCvj4+DB9+nTKlSvHe++9R+/evfHz87M6oscSETZt2oTdbueLL74gMTGRtm3b8sEHH9CrVy+PmiLTU2lhUOouXLlyhYULF2K32zl48CDHjh2jQIECbN68mcKFC1sdz6NdvnyZefPm4XA4iIiIoHDhwgQFBTFq1Cjq1q1rdbw8RQuDUplw9OhRJk2axOzZs7l48SIPPPAA48ePT7+2rUXh3u3Zsyd9+tFLly7RsGFDpk6dyoABAyhYsKDV8fIkLQxK3UJycjKxsbEUL16c06dP88knn6T31mndurX21smCpKQkli1bht1uZ926dfj6+vLUU08RHBxMixYt9NhaTAuDUjc4ceIE06ZNIzQ0lMDAQGbMmEHTpk05ceIEJUuWtDqeR4uOjk6fIvPkyZNUrVqViRMnMnToUEqVKmV1PJVGC4NSaX7++WemTJnCl19+md5b58knn0xfr0Xh3jidzvTpR5cvX47T6aRLly4EBwcTGBiofYvckBYGlafFxsZSqFAhjDF88cUXfP/997z44osEBQVpb50sunDhAnPmzMHhcLB//35KlizJK6+8QlBQENWqVbM6nkc7d+4cly9fzrmpRm81UYOnLDpRj7oXO3fulJEjR0qBAgXkhx9+EBGRc+fOyZUrV6wNlgts27ZNhg4dKv7+/gJIy5YtZe7cuZKQkGB1NI+3ZcsWeeaZZ8TPz0/69++fpX3hLhP1KGWlpKQkFi1ahN1uZ+PGjeTPn58BAwZQrlw5gPRW1+ruxcfH8/nnn+NwONiyZQsBAQEMHjwYm81Go0aNrI7n0a5cuYK/vz/GGKZPn87SpUsZNmxYhn5b2U1bYqhc78qVKwQEBJCYmEjlypUpXLgwNpuNIUOGaDHIooMHDxISEsLMmTO5cOECtWvXTj+2RYoUsTqeR9u3bx8hISHMnj2bb7/9lhYtWnD69Gn8/f0pVKhQlvevLTFUnnOtt47D4WD37t0cPHgQX19fNm7cSNWqVbW3ThYkJyezcuVKHA4Hq1evJl++fPTq1Yvg4GAeffRRHWqaBcnJyXz11VfY7XbCw8Px8fHhySefTH9OpnTp0i7JoYVB5Spnz55lxowZhISEpPfWGTFiBAkJCRQoUID77rvP6oge69SpU0yfPp2pU6cSFRVF+fLleffddxk+fDjly5e3Op5HS0xMxNfXl6SkJEaOHEnBggX5f//v/zFs2DBLph/VwqA8noiQlJSEr68vmzdv5vXXX6dt27Z8+OGH9OzZU3vrZIGI8NNPP2G321myZAnJycl07NiRyZMn0717d/Ll04+QeyUi/PDDD9jtdvbs2UNkZCT+/v78/PPP3H///ZYO49X/qspjXb58mfnz5+NwOOjcuTMTJkzg8ccfZ8+ePdSpU8fqeB4tNjaWuXPnYrfb2b17N0WLFuVvf/sbo0aNolatWlbH82jXph91OBzs27eP4sWLM2zYMBISEggICHCL310tDMrj7N27N723TmxsLA0aNKB+/foAeHl5ucX/WJ7qt99+w+FwMHfuXC5fvkzjxo2ZMWMG/fv3JyAgwOp4Hi0lJQVvb2/Cw8N54YUXaNGiBWFhYfTt29f9WrPfahyrpyz6HEPekJycnP66f//+4uvrKwMHDpQNGzaI0+m0MJnnS0hIkPnz50urVq0EkPz588uQIUNk8+bNemyzKD4+XubMmSMtWrSQd955R0REkpKSZPv27RYnu/1zDJZ/sGd10cKQu0VFRcnYsWOlXLlyEhkZKSIihw8fltOnT1uczPMdOXJE3njjDSldurQAUr16dfn3v/8tZ8+etTqaxzt06JC8+uqrUqJECQHk/vvvl9mzZ1sdK4PbFQa9lKTcjojw/fffZ+it8/jjj5OSkgJA1apVrQ3owZxOJ9999x12u52VK1cC0L17d2w2G506ddJhvFkgIulDdV9++WVWrFhBz54906cf9aRhvPqAm8pW0dHR/PbbbyQlJVGhQgWaNGmS6f8hnE4nXl5exMTEUL58efz9/Rk+fDgjR47UYaZZdO7cOWbNmkVISAiHDh2idOnSjBgxgpEjR1K5cmWr41nu8OHD7Nmzh5SUFKpWrUr9+vUz/Xt7+vRpZs6cSWhoKKtXr6ZmzZrs37+fAgUKUKFChRxOfu9u94Cb5ZeCsrropST3sGHDBunZs6cUK1ZMHnvsMenWrZvUrFlT6tatK59++qkkJSXd8r3btm2TYcOGSatWrdKvaW/atEni4+NdFT9Xcjqd8ssvv8iQIUPEz89PAGnTpo0sWLBArl69anU8t7BmzRoJDAyUkiVLSufOnaVbt25SpUoVady4scyaNeuW91icTqf8/PPPMnDgQPH19RVA2rdvL7/++quL/wX3Dr3HoHLSrFmzpGzZshISEiKXLl1K/77T6ZTw8HBp06aNdO/ePUMTtStXrsjs2bPloYceEkACAgJk5MiRWgyyQVxcnEyfPl0aN24sgBQsWFCCg4Nl165dVkdzKx999JFUqlRJ5syZk6F5YkpKinzzzTfSpEkTGTx4cIaBD9cKxblz58TPz08KFy4sf//732XPnj0uz59VWhhUjlm7dq2UK1dOfv/99/998/BhkYgIkcREERFJTEyUnj17yvDhw9P/x5o9e7YAUrt2bfn444/l4sWLFqTPXfbt2ycvvPCCFC1aVACpX7++2O12iY2NtTqa21m0aJFUrVpVoqKi0r934IDIrl0i1+pAXFyctG3bVl5//XWJjIyU0aNHS+fOndO3Dw8Pz/CHkKdxi8IAVAJ+APYCu4EXbrKNAaYAB4HfgMZ32q8WBms9+uijsnDhwtQvoqJEmjYV8fcXKVhQpFgxkSVLJCkpSRYsWCA+Pj4yduxYEUk9YwgPD9fhkFmUlJQkS5YskQ4dOgggPj4+8vTTT8tPP/2kx/YWnE6nPPDAA7J27VoREdm/X6RuXZGAgNRf29KlRdasEbl69apMnTpVvL29BRBfX18ZNGhQrmkf7i6Fody1D3qgELAfqHvDNl2AVWkFogWw+U771cJgncjISClXrpwkJiaKOJ0itWuLeHun/lqBnAB538dHKpUtK4AUKFAgw19c6t4dO3ZM3nnnHalQoYIAUrlyZRk/frycPHnS6mhu78cff5TatWuL0+mUpCSRcuVEjEn/tRUQKVBAZOLEaemX4rp27ZrrhkjfrjC4bLiqiJwATqS9vmSM2QtUAPZct9kTQFha6F+MMUWNMeXS3qvczMaNG+ncuXNqL6ItWyA6GtKGlAI8A6xJSqKjtzeTly6lYMGCvP/++9YF9nAiwrp167Db7Xz55ZekpKQQGBiI3W6na9euOkVmJm3YsIHu3btjjOG77yAuDkScwPeAHXicpKSRXLzYj5UryxMbG8sXX3yRp+aktmTQsjGmKvAgsPmGVRWAqOu+jk773o3vH2mM2WaM2XbmzJmciqnuID4+ngIFCgAQ+8cf2JOTaUxa9Qc+BPYBa+rXp1evXhQqVIiEhASL0nquixcvMmXKFOrWrUv79u0JDw/npZde4sCBA3z77bf06NFDi8JduNaTCODQoQskJEwCagOPAT8DySQmwqlThejSpQsFCxbMc7+3Ln/AzRhTEFgCvCgisTeuvslb/vKghYiEAqGQ+hxDtodUmVKmTBkWL15McHAwn4WFcTkhgcbASVKvGzYCCAiAxx8H4MiRIy7rJ58b/PrrrzgcDubNm8eVK1do3rw5c+bMoW/fvvj7+1sdz2OVLl2arVu3AjBv3pMkJv4APAK8AzwJ+FGwIHTsmLp9nvy9vdU1ppxYAB9gNfCPW6yfCjx93de/A+Vut0+9x2CdgwcPCiB+fn6pvXUGDhRnQMD/LtT6+YlUqyaSNiqmY8eOMn/+fItTu7f4+HgJCwuTFi1aCCD+/v4ybNgw2bZtm9XRPF5cXJzMnDlTmjVrJkWKFJGLFy/Kpk2bpFu3nXL9r23+/CING4pcvZp6o7pRo0by3XffWR0/25EdN5+BUkB1wCez77nh/QYIA/57m226kvHm85Y77VcLg+tc663zzDPPpH+vc+fOMmzYsNQvnE6Rzz8XadlS5IEHRN56S+T8eRER2bhxo5QpUybXjOjIbocOHZJ//vOfGXrrTJo0Sc6nHT917/bv3y//+Mc/pFixYgJInTp1pFOnTulN7VJSRGbNEnnoIZH69UXGjxe5fDn1vStWrJCaNWtKSkqKdf+AHJKlwgCMBD4j9dLNh8A8Uu/QVLvTe2/YTytSLwv9BuxMW7oAo4BR8r/i8SlwCNgFNL3TfrUw5KyUlBRZtWqVdO/eXby8vMTLy0t69uyZOhJJRM6ePSs1a9aU999//5bDI7ds2SJly5aVFStWuDK620tOTpYVK1bI448/LsYY8fb2lt69e8vatWt1qGk2OX78uHh5eUm+fPnkqaeeknXr1onT6ZSjR49KxYoVxeFw3PK94eHhUqpUKVm/fr0LE7tOVgvDX54lAAoANe/0XlcsWhhy1scffyyAlC5dWt588035888//7LNsWPHpEmTJtKwYUMJCQmRQ4cOSVRUlKxZs0b69OkjxYoVk2XLllmQ3j2dOnVKJkyYIFWqVBFAypUrJ2+//bZER0dbHc3jnThxQt577z0JDg5O/95nn30mx48f/8u2Bw8elNq1a0vz5s1lzpw5cvjwYTl69Kh8/fXX0q1bNylVqpSEh4e7Mr5LZbUwdAKmAQ3Tvh55p/e4ctHCkH2cTqds3rxZhgwZIosWLRIRkTNnzmSqt05KSop899130rNnT6lSpYqUK1dOmjVrJp988onExMS4Ir5bczqdsmHDhgy9ddq1aydffPFF+tmXujdOp1PWrVsnTz31lOTLl08A6dy58237c12TnJwsy5cvly5dukjlypWlQoUK0rJlS5k+fbrExcW5IL11sloYvgSKAv8G2gP2O73HlYsWhqy71lunSZMm6Q/0TJo0yepYucKlS5ckJCREGjZsKIBH99ZxV5MnTxZAihYtKi+99FLG9izqlm5XGDIzXPWMiFwEXjHGTASaZeI9yoMEBkCNB3MAACAASURBVAby888/U69ePex2O4MGDaJQoUJWx/Jou3fvxuFwEBYWxqVLl2jYsCFTp05lwIABFCxY0Op4Hu3a9KNdunShe/fu9O3bl0KFCtGvXz+dfjSbZKYwrLz2QkReN8b8PQfzqByWnJzM8uXLmTVrFvPnz6dQoUKMHTuW/Pnz06pVK4+aTMTdJCYmsmzZMux2Oz/++CO+vr489dRTBAcH06JFCz22WXD16lWWLFmC3W5nw4YN5M+fn5o1awJQrlw5nnvuOYsT5jK3OpW4tgD57+b7rl70UlLmHDt2TN5999303jqVKlWSHTt2WB0rV4iKipIxY8ZI2bSeUNWqVZMPPvgg1/XWsdIjjzwigNSoUUM++ugjOXfunNWRPB5ZvJQ00RjjJLWn0UWgWtoyD9iQA7VKZbM///yTGjVqkJycTGBgIJ9++ildu3YlXz6d2fVeOZ3ODNOPighdu3bFZrMRGBioLSqywOl0snr1aubMmcOsWbPw9/fntddew9fXV6cfdZFMTe1pjClGaoeDYsABEdmV08EyS6f2/KuYmBjCwsI4e/Ys7777LgD//e9/6datGzVq1LA4nWe7cOECs2fPxuFwcODAAUqWLJk+/Wi1atWsjufRzp49y8yZMwkJCeHw4cOUKVOGVatW8eCDD1odLVfKtqk9gf+SVkzcZdFLSf/z66+/yogRIyQgIEAAefTRR3PlE5tW2Lp1qzz33HOSP39+AeThhx+WuXPn6pPc2eTQoUPp04+2bt1aFi5cqNOP5jCyaz4G4D1gBVAg7evHgA13s4/sXrQwpJo0aZL21slmV65cSe+tQ9p8EkFBQbJz506ro3m8y5cvS2hoqEycOFFEUp9FeP/993X6URfKtsKQui8GAFtJ7U+7Gmh9t/vIziWvFoY//vhD/vnPf8qPP/4oIiIHDhzQ3jrZ5Ga9dXT60eyxd+9eef7556VIkSLpZ17a/sMa2XnG0IHU6TnXkdr5tNbdvD8nlrxUGJKTk+Xrr7+WLl26iDFGvLy85IMPPrA6Vq6QlJQkX375pTz22GMC/KW3jsq6a2e1Pj4+MmDAAJ1+1GLZWRjCgVZpr+uT2giv/d3sI7uXvFQYrg3ZK1u2rIwdOzbDRObq3lzrrVOxYkUBpGLFijJu3Lib9tZRdyc6Olrefvtt+eWXX0REZM+ePTJhwgQ5deqUxcmUSDYWhr+8OXU+lo1Z2UdWl9xaGK711gkODk7v+RIWFiaLFi3S3jpZ5HQ65ccff5R+/fql99bp2LGjfPnll5nqr6Nuzel0yvfffy9PPvmkeHt7izEm/T6Cci85VhhS941/VveRlSW3FYab9daJiIiwOlauEBMTI59++qk88MAD6b11/vGPf8j+/futjpZrtG7dWgApXry4vPrqq3Lw4EGrI6lbuF1hyPITTiISn9V9qFS///47zZo1S++tExoaytNPP629dbLoWm+duXPncvnyZZo0acLMmTO1t0422LFjBwsXLmTixIl4eXkxaNAgRowYQd++fcmfP7/V8dS9ulXF8JTFk88YEhMTZdGiRRIaGioiqa2rX3rpJdm4caPelMuihIQEmT9/vrRq1UoAyZ8/vzz77LOyZcsWq6N5vPj4eJkzZ440b95cAAkICNBhph6InLyUZPXiiYXhxt46TZs2tTpSrnHkyBH5v//7PylVqpQAUr16dfn3v/8tZ8+etTparrB79+706Udr1aolkydPlgsXLlgdS92D2xUGbZbjYpMnT+bll1/G6XTSpUsXgoODCQwMtDqWR7vWW8dut7Ny5UqMMXTv3p3g4GA6duyovXWyICUlhW+++YbY2FgGDhzI/fffT+/evenfvz/t2rXTjrG5VKZ6Jbkzd++VdK23zmOPPcYDDzzA1q1bWbJkCUFBQdpbJ4vOnj3LrFmzCAkJ4Y8//qBMmTLpfYsqV65sdTyPdvr0aWbMmMHUqVP5888/ady4Mdu2bdNCkItkW68kd1zc9VLStm3bZOjQoeLv7y+ADtnLJk6nUzZt2iSDBw9O763Tpk0b7a2TjaZMmSI+Pj4CSPv27WXx4sU6RDoXQi8luY6I0KlTJ77//nsCAgIYNGgQwcHBNGrUyOpoHi0uLo4FCxZgt9v59ddfKVSoEMOGDcNms1GvXj2r43m0S5cuMW/ePDp06EDNmjVp1KgRNpsNm81G7dq1rY6nrHCriuEpizucMRw4cEAmTpyYPpJowoQJ2lsnm9zYW6d+/fricDgkNjbW6mgeLzIyUoKDg6VQoUICyIQJE6yOpFwIHZWU/ZKSkmTZsmUZeuvs27fPkiy5TWJioixevFjat2+vvXVygNPplMDAQAHEz89PnnnmGfnll1/02OYxtysMeinpHkRGRtKlSxeioqKoUKEC48aNY/jw4ZQrV87qaB7t2LFjTJs2jdDQUE6cOEHlypUZP348Q4cOpUyZMlbH82hRUVEsX76c0aNHY4yhWbNmdOjQgeeee46SJUtaHU+5m1tVDE9ZXHHGcK23zooVK0Qk9QGf3r17y9KlS7W3ThbdrLfO448/LitWrJDk5GSr43m0lJQUWb16tTzxxBPi5eUlxhht/6HSoZeS7k1MTIx88skn6b113OF+Rm5x4cIFmTx5stSqVUt76+SA3377TWrUqCGAlCpVSv7v//5PDh8+bHUs5UZuVxj0UtItfPrpp7z22mvExcXRpEkTZsyYQf/+/a2O5fF+/fVXHA4H8+bN48qVK7Ro0YKwsDDtrZMNtm7dSkxMDB07duS+++6jevXqjBs3jt69e+Pn52d1POVJblUxsnsBZgKngchbrG8LxJA6x8NOYGxm9ptdf8Vf660THR0tIiJff/219tbJJvHx8RIWFiYtWrRIn350+PDhsn37dqujeby4uDiZOXOmNG3aVABp1qyZ1ZGUh8AdLiUBbYDGdygMX9/tfrNaGI4cOSJvvPGGlC5dWgD517/+laX9qf85dOiQ/POf/0zvrXP//ffLpEmTtLdONgkJCUmffrRu3bry8ccfS0xMjNWxlIe4XWFw2aUkEVlvjKnqqp93J06nk759+7Js2TKADL111L1LSUlh1apV2O12vv32W7y8vOjZsyc2m4327dtrS4UsSE5O5uuvv6Zly5aULl2aEiVK0KlTJ0aPHk3r1q312Krsc6uKkRMLUJXbnzGcAyKAVcADt9nPSGAbsK1y5cr3XDFHjRolb775pvz555/3vA+V6tSpUzJ+/HipUqWKAFKuXDl5++230y/NqXt3/PhxGTduXPr0ox999JHVkVQugDtcSpI7F4bCQMG0112AA5nZp44Uso7T6ZSffvpJBgwYoL11ckBycrL0798/ffrRxx57TJYtW6ZDpFW2uF1hcJtRSSISe93rb4wxdmNMSRE5a2Uu9VfXeuvY7XZ27dpF4cKFsdlsjBo1ijp16lgdz6PFxMTw448/0qNHD7y9vcmXLx/PP/88o0aNombNmlbHU3mE2xQGY0xZ4JSIiDHmIcCL1EtLyk3s3r0bh8NBWFgYly5dolGjRoSGhjJgwAAKFChgdTyPFhERgd1uTx/Ge/ToUSpWrMhnn31mdTSVB7msMBhjFpB6H6GkMSYaeBvwARCREKAPYDPGJAPxQP+00x1locTERL788kvsdjvr16/H19eXfv36ERwcTPPmzfWGZxbt2rWLUaNGsXHjRvLnz8/TTz+NzWajYsWKVkdTeZgrRyU9fYf1nwCfuCiOuoOjR48SGhrK9OnTOXXqFNWqVePDDz/U3jrZ4MiRI8TExNCwYUNKlSpFTEwM//nPfxgyZAjFixe3Op5S7nMpSVnP6XSydu1a7HY7K1asQETo2rVr+vSjOkXmvUtJSUmffvSbb76hbdu2hIeHU7ZsWXbt2qVnXsqtaGFQnD9/ntmzZ+NwODh48CClSpXitddeY+TIkVStWtXqeB5vzpw5vPvuuxw+fJiyZcvy1ltvMWLEiPT1WhSUu9HCkIdt3boVu93OwoULSUhI4JFHHuHdd9/lySef1N46WSAi/PLLLzRo0IACBQoQExNDpUqVmDhxIj179sTX19fqiErdlvH0+7tNmzaVbdu2WR3DY1y5coXPP/8cu93Otm3bKFCgAIMHD8Zms9GgQQOr43m0uLg45s+fj91uZ+fOnUyfPp1hw4YhInpWoNyOMWa7iDS92To9Y8gjDhw4QEhICLNmzeLChQvUrVuXTz75hMGDB1O4cGGr43m0xMREXnnlFebMmUNsbCwNGjRg6tSp9OvXD9BLRcrzaGHIxa711rHb7axZs4Z8+fLx5JNPYrPZaNOmjX5gZUFSUhIRERE0bdoUX19fduzYQffu3bHZbLRs2VKPrfJoWhhyoZMnTzJ9+nSmTp1KdHQ0FStW5L333mP48OGULVvW6nge7frpRy9cuMDx48cpVqwYP/74I97e3lbHUypbaGHIJUSE9evXY7fbWbp0KcnJyXTq1ImPP/6Ybt26kS+f/qfOij179jBmzBi++uornE4ngYGBjB49Ov0ynBYFlZvop4WHi42N5bPPPsNut7Nnzx6KFSumvXWyycWLF4mNjaVy5coYY1i/fj0vv/wyQUFB3HfffVbHUyrHaGHwUBERETgcDubOnUtcXBzNmjVj5syZ9OvXj4CAAKvjebQdO3Zgt9uZP38+3bp1Y9GiRdSpU4fjx4/j4+NjdTylcpwWBg9y9epVlixZgt1uZ8OGDRl66zRr1szqeB5v6dKlfPjhh2zevJmAgAAGDRqEzWZLX69FQeUVWhg8wJEjR5g6dSozZszgzJkz1KhRg48++ohnn31We+tk0R9//EGVKlXw9vZm+/btxMTEMHnyZJ555hmKFi1qdTylLKEPuLmpa711HA4HK1euxBhDjx49CA4OpkOHDtq3KAtSUlL45ptv0qcfXb58Od27dychIQE/Pz8daqryBH3AzYOcPXuWmTNnEhISwuHDhylTpgxvvvkmI0eOpFKlSlbH82gJCQn85z//YerUqRw9epTy5cvz7rvvpl+Gy58/v8UJlXIPWhjcwLXeOna7nUWLFpGYmMijjz6qvXWygYgQHR1NpUqV8PHxYfr06dSsWZNJkybRvXt3vW+g1E1oYbDQjb11ChUqxIgRI7DZbDzwwANWx/Noly5dYu7cudjtdk6dOkVUVBR+fn5ERERQqFAhq+Mp5da0MFhg3759OBwO5syZQ0xMDA0aNCAkJISBAwdSsGBBq+N5tD/++IOPPvqIsLAwLl++zIMPPsj48ePT12tRUOrOtDC4SFJSEl999RUOh4Pw8HB8fX3p27cvwcHBPPzww3rDMwsSExOJi4ujWLFiREdHM2PGDPr374/NZuOhhx7SY6vUXdLCkMOu761z4sQJqlSpwoQJExg6dCilS5e2Op5Huzb96LRp0+jbty+ffPIJrVu35vjx4zqMV6ks0MKQA0SE8PBwHA4Hy5Ytw+l00rlzZ0JDQ3n88ce1r04WhYeHM2XKlPTpR7t160avXr2A1BbXWhSUyhotDNno4sWLzJkzB4fDwe+//07x4sV56aWXGDVqFNWrV7c6nke7ePEiRYoUwRjDwoUL2bhxI//85z8JCgrS6UeVymb6gFs22LFjBw6Hg3nz5hEfH0+LFi0IDg6mb9++OjY+C0SErVu34nA4WLhwIevWraN58+acO3eOggUL6vSjSmWBPuCWAxISEli0aBF2u53Nmzfj7+/PwIEDsdlsNG7c2Op4Hu3q1avMmzcPu93O9u3bKViwIM899xylSpUCoESJEhYnVCp308Jwlw4dOkRISAgzZ87k/Pnz1KpVS3vrZJO4uDgKFChASkoKr7zyCuXLl+fTTz9l0KBBOv2oUi6khSETbuyt4+3tTa9evbDZbLRr106HQ2ZBcnIyK1aswG63ExUVxZ49ewgICGDHjh1UqVJFj61SFtDCcBunTp1ixowZGXrrvPPOOwwfPpwKFSpYHc+jnTx5kmnTpjF16lSOHTtGpUqVCAoKIikpCT8/P72hrJSFtDDcQETYsGEDdrudxYsXk5SURIcOHbS3TjYQERITE/Hz8+Pnn39m7NixBAYG8umnn9K1a1edfvQOkpPhp58gLg5at4YiRaxOpHItEXHJAswETgORt1hvgCnAQeA3oHFm9tukSRPJDrGxsWK326VevXoCSJEiReTFF1+Uffv2Zcv+87KLFy/KlClTpE6dOjJu3DgREUlMTJT9+/dbnMxz7NghUrq0SKFCIoULi/j7i0ydanUq5cmAbXKLz1VXNvWfDXS+zfrHgZppy0jA4YJMREZGMnr0aMqXL09wcDA+Pj5MmzaNY8eOMWnSJGrVquWKGLlSREQEQUFBlC9fnueff55ChQqlNwf08fHROakzKSkJHnsMTp+GS5cgNhbi4+Gll2DnTqvTqdzIZefuIrLeGFP1Nps8AYSlVbJfjDFFjTHlRORETuT55ptv+OCDD1i/fj1+fn7069eP4OBg7a2TRcnJyemXhMaOHct3333HgAEDsNlsNG160yHT6g6+/x4SE//6/atXITQU7HbXZ1K5mztd1K0ARF33dXTa9/5SGIwxI0k9q6By5cr39MMiIyOJjo7mww8/5LnnnqNkyZL3tB+V6vDhw0ydOpVZs2axceNGqlevzn//+1+KFi1KsWLFrI7n0WJj4WbPoaakwPnzrs+jcj93Kgw3+zP9po9li0goEAqpTz7fyw974YUXeOWVV3SKzCxISUnh22+/xW63s2rVKowxPPHEEyQlJQFQrVo1ixPmDm3b3vyMoUAB6N3b5XFUHuBOn4rRwPVzV1YEjufUD/Pz89OicI+cTicA58+fp3fv3uzYsYO33nqLI0eOsHTpUmrXrm1xwtyldGl45x0ICIBrVzkLFIAmTbQwqJzhTmcMy4G/GWMWAs2BmJy6v6DunoiwadMm7HY7J0+eZO3atZQqVYr169fTuHFjHcabw15/HR55JPWeQkwMPPUU9OsHOsJX5QSX/VoZYxYAbYGSxpho4G3AB0BEQoBvgC6kDle9Ajznqmzq1i5fvpw+/WhERASFCxfmmWeeISkpCR8fH5o3b251xDyjdevURamc5spRSU/fYb0Ao10UR92BiGCMISwsjNGjR9OgQQOmTp3KgAEDdPpRpXI5vciu0iUlJfHFF1/Qrl07QkNDARg0aBAbNmxg586djBw5UouCUnmAXqFUREdHp08/evLkSapWrYq/vz8AhQsXpmXLlhYnVEq5khYGRf/+/dm4cSOPP/44wcHBdO7cWacfVSoP08KQx1y4cIHZs2cze/ZswsPDKVGiBJMnT6Z48eL63IFSCtDCkGds374du93OggULiI+P5+GHH+bkyZOUKFGCJk2aWB1PKeVGtDDkAX/++SdNmzYlICCAwYMHY7PZaNSokdWxlFJuSgtDLnRt+tGLFy8ybdo0qlSpwpIlS+jQoQNFtIm/UuoOdLhqLpGSksLy5cvp3LkzNWrU4L///S9xcXHp7St69+6tRUEplSlaGHKJf//73zzxxBNERkYybtw4jh49yvz587UflFLqrumlJA8kIvz00084HA6efvppevToweDBg7n//vvp3r27TpGplMoS/XPSg8TGxmK326lfvz6PPvooq1at4uTJkwCUL1+eXr16aVFQSmWZfop4kLZt2/Lrr7/SuHFjZsyYQf/+/QkICLA6llIql9HC4KYSExNZsmQJ8+fPZ9GiRfj7+zN+/HiKFy9Os2bNdPpRpVSO0cLgZv78809CQ0OZPn06p0+f5r777uPw4cPUrVuXzp07Wx1PKZUHaGFwI3v37qVevXoAdOvWjeDgYDp16qQji5RSLqWFwULnzp1j1qxZxMfHM2bMGGrXrs1HH31E7969qVy5stXxlFJ5lBYGFxMRtmzZgsPhYOHChVy9epUuXbqkT4zz4osvWh1RKZXH6TUKFxs/fjwtWrRgyZIlDBs2jF27drFy5Uq9mayUcht6xpDDfv/9d0JCQujfvz/Nmzend+/eFC9enEGDBlGoUCGr4yml1F9oYcgBycnJLF++HIfDwdq1a/Hx8aFGjRo0b96cOnXqUKdOHasjKqXULWlhyGYiQtOmTYmIiKBy5cq8//77DB8+nDJlylgdTSmlMkULQxaJCOvWrWPx4sV8/PHHeHl58cILL1CyZEm6dOmiU2QqpTyOERGrM2RJ06ZNZdu2bS7/uTExMYSFheFwONi7dy/Fixdn06ZN3H///S7PopRSd8sYs11Emt5snY5Kugc7d+6kfPnyPP/88xQqVIjZs2cTHR2tRUEplSvopaRMSEhIYPHixSQnJ/Pss89Sr149Ro4cycCBA2na9KYFVymlPJZeSrqNP/74g6lTpzJjxgzOnTtH27Zt+eGHH3LkZymllCvppaR7MH78eGrUqMFHH33Eo48+ytq1awkPD7c6llJK5TiXFgZjTGdjzO/GmIPGmNdvsv5ZY8wZY8zOtGW4q7KdOXOGDz74gAMHDgDQpk0bxowZw5EjR1iyZAkdOnTQp5OVUnmCy+4xGGO8gU+BTkA0sNUYs1xE9tyw6eci8jdXZBIRNm3ahN1u54svviAxMZECBQpQs2ZNWrVqRatWrVwRQyml3Iorbz4/BBwUkT8AjDELgSeAGwuDSzidTlq2bMnmzZspXLgwQUFBjBo1irp161oRRyml3IYrC0MFIOq6r6OB5jfZ7kljTBtgP/CSiETduIExZiQwErjn9tReXl507dqVYcOG8fTTT1OwYMF72o9SSuU2riwMN7tAf+OQqBXAAhG5aowZBcwB2v/lTSKhQCikjkq610Bjxoy517cqpVSu5cqbz9FApeu+rggcv34DETknIlfTvpwGNHFRNqWUUmlcWRi2AjWNMdWMMb5Af2D59RsYY8pd92UPYK8L8ymllMKFl5JEJNkY8zdgNeANzBSR3caYccA2EVkOPG+M6QEkA+eBZ12VTymlVCp98lkppfIgffJZKaVUpmlhUEoplYEWBqWUUhloYVBKKZWBx998NsacAf68x7eXBM5mY5yc5kl5PSkreFZeT8oKnpXXk7JC1vJWEZFSN1vh8YUhK4wx2251V94deVJeT8oKnpXXk7KCZ+X1pKyQc3n1UpJSSqkMtDAopZTKIK8XhlCrA9wlT8rrSVnBs/J6UlbwrLyelBVyKG+evseglFLqr/L6GYNSSqkbaGFQSimVQZ4oDMaYzsaY340xB40xr99kvZ8x5vO09ZuNMVVdnzI9y52yPmuMOWOM2Zm2DLciZ1qWmcaY08aYyFusN8aYKWn/lt+MMY1dnfGGPHfK29YYE3PdsR3r6ozXZalkjPnBGLPXGLPbGPPCTbZxi+ObyazudGzzG2O2GGMi0vK+e5Nt3OIzIZNZs/8zQURy9UJqi+9DwH2ALxAB1L1hm2AgJO11f+BzN876LPCJ1cc1LUsboDEQeYv1XYBVpM7e1wLY7OZ52wJfW31c07KUAxqnvS5E6lS3N/4uuMXxzWRWdzq2BiiY9toH2Ay0uGEbd/lMyEzWbP9MyAtnDA8BB0XkDxFJBBYCT9ywzROkTiMKsBjoYIy52VSkOS0zWd2GiKwndd6MW3kCCJNUvwBFb5iMyaUykddtiMgJEdmR9voSqZNWVbhhM7c4vpnM6jbSjtfltC990pYbR+G4xWdCJrNmu7xQGCoAUdd9Hc1ff2nTtxGRZCAGKOGSdLfIkeZmWQGeTLt0sNgYU+km691FZv897uThtNP2VcaYB6wOA5B2GeNBUv9avJ7bHd/bZAU3OrbGGG9jzE7gNLBGRG55bC3+TMhMVsjmz4S8UBhuVuVvrLiZ2cYVMpNjBVBVRBoAa/nfXzXuyF2Oa2btILV/TEPgY2CZxXkwxhQElgAvikjsjatv8hbLju8dsrrVsRWRFBFpROrc8w8ZY+rdsInbHNtMZM32z4S8UBiigesraEXg+K22McbkA4pgzSWHO2YVkXMicjXty2lAExdluxeZOfZuQ0Rir522i8g3gI8xpqRVeYwxPqR+0M4TkaU32cRtju+dsrrbsb1GRC4C64DON6xyl8+EdLfKmhOfCXmhMGwFahpjqhljfEm9kbT8hm2WA0PSXvcBwiXtro6L3THrDdeQe5B6PdddLQeeSRs90wKIEZETVoe6FWNM2WvXkY0xD5H6/8c5i7IYYAawV0T+c4vN3OL4Ziarmx3bUsaYommv/YGOwL4bNnOLz4TMZM2Jz4R8Wd2BuxORZGPM34DVpI76mSkiu40x44BtIrKc1F/qz4wxB0n9q6C/G2d93hjTA0hOy/qsFVkBjDELSB1tUtIYEw28TerNMUQkBPiG1JEzB4ErwHPWJE2Vibx9AJsxJhmIB/pb9AcCwCPAYGBX2vVlgDeAyuB2xzczWd3p2JYD5hhjvEktUItE5Gt3/EzIZNZs/0zQlhhKKaUyyAuXkpRSSt0FLQxKKaUy0MKglFIqAy0MSimlMtDCoJRSKgMtDEoppTLQwqBUDjDGTDbGBBhj7jPGzDDGLE77fhNjTJDV+ZS6HS0MSmUzY0xxUhtjXknrlDvs2joR2Q60ti6dUnemhUGp7NcI2HOb9QnGmDKuCqPU3dLCoFQWmNSZyzqlvX7fGDMFKA5cvM3bLgCFXZFPqXuhhUGprHkbeNMYM5DUeQheInUGs6oAxpgSxpgQ4EFjzP+lvacCcNSCrEplSq5voqdUThKR9WldQ/8BtBWRFGPMLlKnhkREzgGjrm2fNmdB7HVtkpVyO3rGoFQWGGPqk9oB82ratJakdQ2dZ4wJuMlbygP/cmFEpe6aFgal7lFaH/x5pM4PHGeMCby2TkR+EpErN75HRPaLyCEXxlTqrmlhUOoepJ0NLAVeFpG9wHvAO5aGUiqb6HwMSimlMtAzBqWUUhloYVBKKZWBFgallFIZaGFQSimVgRYGpZRSGWhhUEoplYEWBqWUUhloYVBKKZWBFgallFIZ/H99IMl6NgAAAANJREFUBWTwQy3Q9gAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "# 绘制数据点\n", "color_seq = ['red' if v == 1 else 'blue' for v in y]\n", "plt.scatter([i[0] for i in X], [i[1] for i in X], c=color_seq)\n", "# 得到x轴的所有点\n", "xaxis = np.linspace(0, 3.5)\n", "w = clf.coef_[0]\n", "# 计算斜率\n", "a = -w[0] / w[1]\n", "# 得到分离超平面\n", "y_sep = a * xaxis - (clf.intercept_[0]) / w[1]\n", "# 下边界超平面\n", "b = clf.support_vectors_[0]\n", "yy_down = a * xaxis + (b[1] - a * b[0])\n", "# 上边界超平面\n", "b = clf.support_vectors_[-1]\n", "yy_up = a * xaxis + (b[1] - a * b[0])\n", "# 绘制超平面\n", "plt.plot(xaxis, y_sep, 'k-')\n", "plt.plot(xaxis, yy_down, 'k--')\n", "plt.plot(xaxis, yy_up, 'k--')\n", "# 绘制支持向量\n", "plt.xlabel('$x^{(1)}$')\n", "plt.ylabel('$x^{(2)}$')\n", "plt.scatter(clf.support_vectors_[:, 0],\n", " clf.support_vectors_[:, 1],\n", " s=150,\n", " facecolors='none',\n", " edgecolors='k')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题7.3\n", "\n", "  线性支持向量机还可以定义为以下形式：$$\\begin{array}{cl} \n", "\\displaystyle \\min_{w,b,\\xi} & \\displaystyle \\frac{1}{2} \\|w\\|^2 + C \\sum_{i=1}^N \\xi_i^2 \\\\\n", "\\text{s.t.} & y_i(w \\cdot x_i + b) \\geqslant 1 - \\xi_i, i=1,2,\\cdots, N \\\\\n", "& \\xi_i \\geqslant 0, i=1,2,\\cdots, N\n", "\\end{array}$$试求其对偶形式。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "根据支持向量机的对偶算法，得到对偶形式，由于不能消去变量$\\xi_i$的部分，所以拉格朗日因子也包含$\\beta_i$。 \n", "拉格朗日函数为：$$L(w,b,\\xi, \\alpha, \\beta) = \\frac{1}{2} \\|w\\|^2 + C \\sum_{i=1}^N \\xi_i^2 + \\sum_{i=1}^N \\alpha_i - \\sum_{i=1}^N \\alpha_i \\xi_i - \\sum_{i=1}^N \\alpha_i y_i (w \\cdot x_i + b) - \\sum_{i=1}^N \\beta_i \\xi_i$$ \n", "分别求$w,b,\\xi$的偏导数：$$\\left \\{ \\begin{array}{l}\n", "\\displaystyle \\nabla_w L = w - \\sum_{i=1}^N \\alpha_i y_i x_i = 0 \\\\ \n", "\\displaystyle \\nabla_b L = -\\sum_{i=1}^N \\alpha_i y_i = 0 \\\\\n", "\\nabla_{\\xi} L = 2C \\xi_i - \\alpha_i - \\beta_i = 0 \n", "\\end{array} \\right.$$化简可得：$$\\left \\{ \\begin{array}{l}\n", "\\displaystyle w = \\sum_{i=1}^N \\alpha_i y_i x_i = 0 \\\\ \n", "\\displaystyle \\sum_{i=1}^N \\alpha_i y_i = 0 \\\\\n", "2C \\xi_i - \\alpha_i - \\beta_i = 0 \n", "\\end{array} \\right.$$ \n", "可解得：$$\n", "L=-\\frac{1}{2} \\sum_{i=1}^N \\sum_{j=1}^N \\alpha_i \\alpha_j y_i y_j (x_i \\cdot x_{j})+\\sum_{i=1}^N \\alpha_i-\\frac{1}{4C}\\sum_{i=1}^N(\\alpha_i+\\beta_i)^2$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题7.4\n", "\n", "  证明内积的正整数幂函数：$$K(x,z)=(x\\cdot z)^p$$是正定核函数，这里$p$是正整数，$ x,z\\in R^n$。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "根据书中第121页定理7.5可知，如果需要证明$K(x,z)$是正定核函数，即证明$K(x,z)$对应的Gram矩阵$K=\\left[ K(x_i,x_j) \\right]_{m \\times m}$是半正定矩阵。 \n", "对任意$c_1,c_2,\\cdots,c_m \\in \\mathbf{R}$，有$$\\begin{aligned} \n", "\\sum_{i,j=1}^m c_i c_j K(x_i,x_j) \n", "&= \\sum_{i,j=1}^m c_i c_j (x_i \\cdot x_j)^p \\\\\n", "&= \\left(\\sum_{i=1}^m c_i x_i \\right)\\left(\\sum_{j=1}^m c_i x_j \\right)(x_i \\cdot x_j)^{p-1} \\\\\n", "&= \\Bigg\\|\\left( \\sum_{i=1}^m c_i x_i \\right)\\Bigg\\|^2 (x_i \\cdot x_j)^{p-1}\n", "\\end{aligned}$$\n", "$\\because p$是正整数，$p \\geqslant 1$ \n", "$\\therefore p-1 \\geqslant 0 \\Rightarrow (x_i \\cdot x_j)^{p-1} \\geqslant 0$ \n", "故$\\displaystyle \\sum_{i,j=1}^m c_i c_j K(x_i,x_j) \\geqslant 0$，即Gram矩阵是半正定矩阵。 \n", "根据定理7.5，可得$K(x,z)$是正定核函数，得证。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "参考代码：https://github.com/wzyonggege/statistical-learning-method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第08章 提升方法/8.Boost.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第8章 提升方法" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1．提升方法是将弱学习算法提升为强学习算法的统计学习方法。在分类学习中，提升方法通过反复修改训练数据的权值分布，构建一系列基本分类器（弱分类器），并将这些基本分类器线性组合，构成一个强分类器。代表性的提升方法是AdaBoost算法。\n", "\n", "AdaBoost模型是弱分类器的线性组合：\n", "\n", "$$f(x)=\\sum_{m=1}^{M} \\alpha_{m} G_{m}(x)$$\n", "\n", "2．AdaBoost算法的特点是通过迭代每次学习一个基本分类器。每次迭代中，提高那些被前一轮分类器错误分类数据的权值，而降低那些被正确分类的数据的权值。最后，AdaBoost将基本分类器的线性组合作为强分类器，其中给分类误差率小的基本分类器以大的权值，给分类误差率大的基本分类器以小的权值。\n", "\n", "3．AdaBoost的训练误差分析表明，AdaBoost的每次迭代可以减少它在训练数据集上的分类误差率，这说明了它作为提升方法的有效性。\n", "\n", "4．AdaBoost算法的一个解释是该算法实际是前向分步算法的一个实现。在这个方法里，模型是加法模型，损失函数是指数损失，算法是前向分步算法。\n", "每一步中极小化损失函数\n", "\n", "$$\\left(\\beta_{m}, \\gamma_{m}\\right)=\\arg \\min _{\\beta, \\gamma} \\sum_{i=1}^{N} L\\left(y_{i}, f_{m-1}\\left(x_{i}\\right)+\\beta b\\left(x_{i} ; \\gamma\\right)\\right)$$\n", "\n", "得 到 参 数$\\beta_{m}, \\gamma_{m}$。\n", "\n", "5．提升树是以分类树或回归树为基本分类器的提升方法。提升树被认为是统计学习中最有效的方法之一。\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# Boost\n", "\n", "“装袋”（bagging）和“提升”（boost）是构建组合模型的两种最主要的方法，所谓的组合模型是由多个基本模型构成的模型，组合模型的预测效果往往比任意一个基本模型的效果都要好。\n", "\n", "- 装袋：每个基本模型由从总体样本中随机抽样得到的不同数据集进行训练得到，通过重抽样得到不同训练数据集的过程称为装袋。\n", "\n", "- 提升：每个基本模型训练时的数据集采用不同权重，针对上一个基本模型分类错误的样本增加权重，使得新的模型重点关注误分类样本\n", "\n", "### AdaBoost\n", "\n", "AdaBoost是AdaptiveBoost的缩写，表明该算法是具有适应性的提升算法。\n", "\n", "算法的步骤如下：\n", "\n", "1）给每个训练样本（$x_{1},x_{2},….,x_{N}$）分配权重，初始权重$w_{1}$均为1/N。\n", "\n", "2）针对带有权值的样本进行训练，得到模型$G_m$（初始模型为G1）。\n", "\n", "3）计算模型$G_m$的误分率$e_m=\\sum_{i=1}^Nw_iI(y_i\\not= G_m(x_i))$\n", "\n", "4）计算模型$G_m$的系数$\\alpha_m=0.5\\log[(1-e_m)/e_m]$\n", "\n", "5）根据误分率e和当前权重向量$w_m$更新权重向量$w_{m+1}$。\n", "\n", "6）计算组合模型$f(x)=\\sum_{m=1}^M\\alpha_mG_m(x_i)$的误分率。\n", "\n", "7）当组合模型的误分率或迭代次数低于一定阈值，停止迭代；否则，回到步骤2）" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "from sklearn.datasets import load_iris\n", "from sklearn.model_selection import train_test_split\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# data\n", "def create_data():\n", " iris = load_iris()\n", " df = pd.DataFrame(iris.data, columns=iris.feature_names)\n", " df['label'] = iris.target\n", " df.columns = ['sepal length', 'sepal width', 'petal length', 'petal width', 'label']\n", " data = np.array(df.iloc[:100, [0, 1, -1]])\n", " for i in range(len(data)):\n", " if data[i,-1] == 0:\n", " data[i,-1] = -1\n", " # print(data)\n", " return data[:,:2], data[:,-1]" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "X, y = create_data()\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD5CAYAAAA3Os7hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZ80lEQVR4nO3df4xdZZ3H8fd3h1k7KjIBxlVm6hbFNAqtVkaQdENccbdaa20QsURWq6zsGlwwuBgxBrUxKYZEXSTR8CMrClvsVqzA8mNVbPyxUjMFbNdWIiraGdhlKLbIWrSM3/3j3mmnd+7M3Oeee+55nmc+r2Qyc8995vT7nINfz5zzOeeauyMiIun7s6oLEBGRzlBDFxHJhBq6iEgm1NBFRDKhhi4ikgk1dBGRTBzV6kAz6wFGgDF3X9Xw3jrgKmCsvugad79+tvUdf/zxvmjRoqBiRUTmu+3btz/h7gPN3mu5oQOXALuBF8zw/tfc/YOtrmzRokWMjIwE/PMiImJmv57pvZZOuZjZEPAWYNajbhERqU6r59A/D3wE+NMsY95uZjvMbLOZLWw2wMwuNLMRMxsZHx8PrVVERGYxZ0M3s1XA4+6+fZZhtwOL3H0p8G3gxmaD3P1adx929+GBgaangEREpE2tnENfDqw2s5XAAuAFZnaTu58/OcDd904Zfx3wmc6WKSLSOQcPHmR0dJRnnnmm6lJmtGDBAoaGhujt7W35d+Zs6O5+OXA5gJm9Hvjnqc28vvzF7v5Y/eVqahdPRUSiNDo6ytFHH82iRYsws6rLmcbd2bt3L6Ojo5x44okt/17bOXQzW29mq+svLzazn5rZT4CLgXXtrldEpGzPPPMMxx13XJTNHMDMOO6444L/ggiJLeLuW4Gt9Z+vmLL80FG8SG62PDDGVfc8xKP7DnBCfx+XrVjMmmWDVZclBcXazCe1U19QQxeZb7Y8MMblt+7kwMEJAMb2HeDyW3cCqKlLdHTrv8gsrrrnoUPNfNKBgxNcdc9DFVUkubj77rtZvHgxJ510EldeeWVH1qmGLjKLR/cdCFou0oqJiQkuuugi7rrrLnbt2sXGjRvZtWtX4fXqlIvILE7o72OsSfM+ob+vgmqkKp2+jvLjH/+Yk046iZe+9KUArF27lm9+85u88pWvLFSnjtBFZnHZisX09fYcsayvt4fLViyuqCLptsnrKGP7DuAcvo6y5YGxOX93JmNjYyxcePiG+qGhIcbG2l/fJDV0kVmsWTbIhrOXMNjfhwGD/X1sOHuJLojOI2VcR3H3acs6kbrRKReROaxZNqgGPo+VcR1laGiIPXv2HHo9OjrKCSec0Pb6JukIXURkFjNdLylyHeW1r30tP//5z/nVr37FH//4R2655RZWr1499y/OQQ1dRGQWZVxHOeqoo7jmmmtYsWIFr3jFKzj33HM5+eSTi5aqUy4iIrOZPN3W6buFV65cycqVKztR4iFq6CIic0jlOopOuYiIZEINXUQkE2roIiKZUEMXEcmEGrqISCbU0CUbWx4YY/mV93LiR/+D5VfeW+hZGyJle9/73scLX/hCTjnllI6tUw1dslDGA5REyrRu3Truvvvujq5TDV2yoA+ikFLt2ASfOwU+2V/7vmNT4VWeeeaZHHvssR0o7jDdWCRZ0AdRSGl2bILbL4aD9f+W9u+pvQZYem51dTWhI3TJQhkPUBIB4DvrDzfzSQcP1JZHRg1dsqAPopDS7B8NW14hnXKRLJT1ACURjhmqnWZptjwyauiSjVQeoCSJOeuKI8+hA/T21ZYXcN5557F161aeeOIJhoaG+NSnPsUFF1xQaJ1q6FJYpz9AVyQqkxc+v7O+dprlmKFaMy94QXTjxo0dKO5IauhSyGT+ezIyOJn/BtTUJR9Lz40u0dKMLopKIcp/i8RDDV0KUf5bUuXuVZcwq3bqU0OXQpT/lhQtWLCAvXv3RtvU3Z29e/eyYMGCoN/TOXQp5LIVi484hw7Kf0v8hoaGGB0dZXx8vOpSZrRgwQKGhsKikWroUojy35Ki3t5eTjzxxKrL6Dg1dClM+W+ROLTc0M2sBxgBxtx9VcN7zwG+ApwK7AXe6e6PdLBOkSQoky9VCrkoegmwe4b3LgB+6+4nAZ8DPlO0MJHU6JnsUrWWGrqZDQFvAa6fYcjbgBvrP28GzjIzK16eSDqUyZeqtXqE/nngI8CfZnh/ENgD4O7PAvuB4xoHmdmFZjZiZiMxX10WaYcy+VK1ORu6ma0CHnf37bMNa7JsWsDT3a9192F3Hx4YGAgoUyR+yuRL1Vo5Ql8OrDazR4BbgDeY2U0NY0aBhQBmdhRwDPBkB+sUiZ6eyS5Vm7Ohu/vl7j7k7ouAtcC97n5+w7DbgPfUfz6nPibOW7BESrJm2SAbzl7CYH8fBgz297Hh7CVKuUjXtJ1DN7P1wIi73wbcAHzVzB6mdmS+tkP1iSRFmXypUlBDd/etwNb6z1dMWf4M8I5OFiby8S072bhtDxPu9Jhx3ukL+fSaJVWXJRIt3SkqUfr4lp3cdN9vDr2ecD/0Wk1dpDk9bVGitHFbk89wnGW5iKihS6QmZrimPtNyEVFDl0j1zHCj8UzLRUQNXSJ13ukLg5aLiC6KSqQmL3wq5SLSOqvq/p/h4WEfGRmp5N8WEUmVmW139+Fm7+kIXZp613U/4oe/OPz0huUvO5ab339GhRVVR884l1ToHLpM09jMAX74iyd513U/qqii6ugZ55ISNXSZprGZz7U8Z3rGuaREDV1kFnrGuaREDV1kFnrGuaREDV2mWf6yY4OW50zPOJeUqKHLNDe//4xpzXu+plz0jHNJiXLoIiIJUQ5dgpWVvQ5Zr/LfImHU0GWayez1ZFxvMnsNFGqoIestqwaRnOkcukxTVvY6ZL3Kf4uEU0OXacrKXoesV/lvkXBq6DJNWdnrkPUq/y0STg1dpikrex2yXuW/RcLpoqhMM3nRsdMJk5D1llWDSM6UQxcRSYhy6CWIISMdWkMMNYtIedTQ2xBDRjq0hhhqFpFy6aJoG2LISIfWEEPNIlIuNfQ2xJCRDq0hhppFpFxq6G2IISMdWkMMNYtIudTQ2xBDRjq0hhhqFpFy6aJoG2LISIfWEEPNIlIu5dBFRBJSKIduZguA7wHPqY/f7O6faBizDrgKGKsvusbdry9StHTex7fsZOO2PUy402PGeacv5NNrlhQeG0u+PZY6RKrSyimXPwBvcPenzawX+IGZ3eXu9zWM+5q7f7DzJUonfHzLTm667zeHXk+4H3rd2KhDxsaSb4+lDpEqzXlR1Guerr/srX9Vc55G2rZx256Wl4eMjSXfHksdIlVqKeViZj1m9iDwOPAtd9/WZNjbzWyHmW02s4UzrOdCMxsxs5Hx8fECZUuoiRmulTRbHjI2lnx7LHWIVKmlhu7uE+7+amAIOM3MTmkYcjuwyN2XAt8GbpxhPde6+7C7Dw8MDBSpWwL1mLW8PGRsLPn2WOoQqVJQDt3d9wFbgTc1LN/r7n+ov7wOOLUj1UnHnHd60z+ami4PGRtLvj2WOkSqNGdDN7MBM+uv/9wHvBH4WcOYF095uRrY3ckipbhPr1nC+a97yaGj7B4zzn/dS5omV0LGrlk2yIazlzDY34cBg/19bDh7SdcvRMZSh0iV5syhm9lSaqdQeqj9H8Amd19vZuuBEXe/zcw2UGvkzwJPAh9w95/NuFKUQxcRacdsOXTdWNSmsjLPIfnvMtcdMr8Ut0VydmyC76yH/aNwzBCcdQUsPbfqqqQC+oCLDisr8xyS/y5z3SHzS3FbJGfHJrj9YjhYT+zs31N7DWrqcgQ9nKsNZWWeQ/LfZa47ZH4pbovkfGf94WY+6eCB2nKRKdTQ21BW5jkk/13mukPml+K2SM7+0bDlMm+pobehrMxzSP67zHWHzC/FbZGcY4bClsu8pYbehrIyzyH57zLXHTK/FLdFcs66Anob/g+yt6+2XGQKXRRtQ1nPFp+82FdGsiNk3SHzS3FbJGfywqdSLjIHxRZFRBKi2KIAcWTLJXHKw0dNDX2eiCFbLolTHj56uig6T8SQLZfEKQ8fPTX0eSKGbLkkTnn46KmhzxMxZMslccrDR08NfZ6IIVsuiVMePnq6KDpPxJAtl8QpDx895dBFRBIyr3PoZeWpQ9Yby3O9lS2PTO6Z7tznF6JL2yLrhl5WnjpkvbE811vZ8sjknunOfX4hurgtsr4oWlaeOmS9sTzXW9nyyOSe6c59fiG6uC2ybuhl5alD1hvLc72VLY9M7pnu3OcXoovbIuuGXlaeOmS9sTzXW9nyyOSe6c59fiG6uC2ybuhl5alD1hvLc72VLY9M7pnu3OcXoovbIuuLomXlqUPWG8tzvZUtj0zume7c5xeii9tCOXQRkYTM6xx6WZRvF0nEHZfC9i+DT4D1wKnrYNVni683wpy9GnoblG8XScQdl8LIDYdf+8Th10WaeqQ5+6wvipZF+XaRRGz/ctjyVkWas1dDb4Py7SKJ8Imw5a2KNGevht4G5dtFEmE9YctbFWnOXg29Dcq3iyTi1HVhy1sVac5eF0XboHy7SCImL3x2OuUSac5eOXQRkYQUyqGb2QLge8Bz6uM3u/snGsY8B/gKcCqwF3inuz9SsO6mQvPfqT0DPCRbnvu2KDXnG5JNLquOMucXYUa6Y0LnlvO2aNDKKZc/AG9w96fNrBf4gZnd5e73TRlzAfBbdz/JzNYCnwHe2eliQ/PfqT0DPCRbnvu2KDXnG5JNLquOMucXaUa6I0LnlvO2aGLOi6Je83T9ZW/9q/E8zduAG+s/bwbOMut83CI0/53aM8BDsuW5b4tSc74h2eSy6ihzfpFmpDsidG45b4smWkq5mFmPmT0IPA58y923NQwZBPYAuPuzwH7guCbrudDMRsxsZHx8PLjY0Px3as8AD8mW574tSs35hmSTy6qjzPlFmpHuiNC55bwtmmipobv7hLu/GhgCTjOzUxqGNDsan9aF3P1adx929+GBgYHgYkPz36k9AzwkW577tig15xuSTS6rjjLnF2lGuiNC55bztmgiKIfu7vuArcCbGt4aBRYCmNlRwDHAkx2o7wih+e/UngEeki3PfVuUmvMNySaXVUeZ84s0I90RoXPLeVs00UrKZQA46O77zKwPeCO1i55T3Qa8B/gRcA5wr5eQhwzNf6f2DPCQbHnu26LUnG9INrmsOsqcX6QZ6Y4InVvO26KJOXPoZraU2gXPHmpH9Jvcfb2ZrQdG3P22erTxq8Ayakfma939l7OtVzl0EZFwhXLo7r6DWqNuXH7FlJ+fAd5RpEgRESkm+1v/k7uZRroj5GaTGG5MKfNmmtRunIphf0Qq64ae3M000h0hN5vEcGNKmTfTpHbjVAz7I2JZP20xuZtppDtCbjaJ4caUMm+mSe3GqRj2R8SybujJ3Uwj3RFys0kMN6aUeTNNajdOxbA/IpZ1Q0/uZhrpjpCbTWK4MaXMm2lSu3Eqhv0RsawbenI300h3hNxsEsONKWXeTJPajVMx7I+IZd3Q1ywbZMPZSxjs78OAwf4+Npy9RBdE57ul58Jbr4ZjFgJW+/7Wq5tfVAsZG0O9oePLml9q682EPuBCRCQhhW4sEpn3Qj4MIxap1RxLtjyWOtqkhi4ym5APw4hFajXHki2PpY4Csj6HLlJYyIdhxCK1mmPJlsdSRwFq6CKzCfkwjFikVnMs2fJY6ihADV1kNiEfhhGL1GqOJVseSx0FqKGLzCbkwzBikVrNsWTLY6mjADV0kdms+iwMX3D46NZ6aq9jvLg4KbWaY8mWx1JHAcqhi4gkRDl0KVeK2d2yai4r/53iNpauU0OXYlLM7pZVc1n57xS3sVRC59ClmBSzu2XVXFb+O8VtLJVQQ5diUszullVzWfnvFLexVEINXYpJMbtbVs1l5b9T3MZSCTV0KSbF7G5ZNZeV/05xG0sl1NClmBSzu2XVXFb+O8VtLJVQDl1EJCGz5dB1hC752LEJPncKfLK/9n3Hpu6vt6waRFqgHLrkoaysdsh6lReXiukIXfJQVlY7ZL3Ki0vF1NAlD2VltUPWq7y4VEwNXfJQVlY7ZL3Ki0vF1NAlD2VltUPWq7y4VEwNXfJQVlY7ZL3Ki0vFlEMXEUlIoRy6mS00s++a2W4z+6mZXdJkzOvNbL+ZPVj/0t+YqUsxT628ePm03aLWSg79WeDD7n6/mR0NbDezb7n7roZx33f3VZ0vUbouxTy18uLl03aL3pxH6O7+mLvfX//5d8BuYLDswqRCKeaplRcvn7Zb9IIuiprZImAZsK3J22eY2U/M7C4zO3mG37/QzEbMbGR8fDy4WOmSFPPUyouXT9stei03dDN7PvB14EPu/lTD2/cDf+nurwK+AGxptg53v9bdh919eGBgoN2apWwp5qmVFy+ftlv0WmroZtZLrZnf7O63Nr7v7k+5+9P1n+8Ees3s+I5WKt2TYp5aefHyabtFr5WUiwE3ALvdvemDnc3sRfVxmNlp9fXu7WSh0kUp5qmVFy+ftlv05syhm9lfAd8HdgJ/qi/+GPASAHf/kpl9EPgAtUTMAeBSd/+v2darHLqISLjZcuhzxhbd/QeAzTHmGuCa9sqTtu3YVEsY7B+tncc864r5fbR0x6Ww/cu1D2W2ntpHvxX9tCCRhOh56KlSJvhId1wKIzccfu0Th1+rqcs8oWe5pEqZ4CNt/3LYcpEMqaGnSpngI/lE2HKRDKmhp0qZ4CNZT9hykQypoadKmeAjnboubLlIhtTQU6VM8JFWfRaGLzh8RG49tde6ICrziJ6HLiKSkEI59PlkywNjXHXPQzy67wAn9Pdx2YrFrFmW0YMlc8+t5z6/GGgbR00NvW7LA2NcfutODhyspSLG9h3g8lt3AuTR1HPPrec+vxhoG0dP59DrrrrnoUPNfNKBgxNcdc9DFVXUYbnn1nOfXwy0jaOnhl736L4DQcuTk3tuPff5xUDbOHpq6HUn9PcFLU9O7rn13OcXA23j6Kmh1122YjF9vUfehNLX28NlKxZXVFGH5Z5bz31+MdA2jp4uitZNXvjMNuUyedEq14RC7vOLgbZx9JRDFxFJyGw5dJ1yEUnBjk3wuVPgk/217zs2pbFu6SqdchGJXZn5b2XLs6IjdJHYlZn/VrY8K2roIrErM/+tbHlW1NBFYldm/lvZ8qyooYvErsz8t7LlWVFDF4ldmc++13P1s6IcuohIQpRDFxGZB9TQRUQyoYYuIpIJNXQRkUyooYuIZEINXUQkE2roIiKZUEMXEcnEnA3dzBaa2XfNbLeZ/dTMLmkyxszsajN72Mx2mNlryilXCtFzr0Wy1srz0J8FPuzu95vZ0cB2M/uWu++aMubNwMvrX6cDX6x/l1joudci2ZvzCN3dH3P3++s//w7YDTR+0ObbgK94zX1Av5m9uOPVSvv03GuR7AWdQzezRcAyYFvDW4PAnimvR5ne9DGzC81sxMxGxsfHwyqVYvTca5HstdzQzez5wNeBD7n7U41vN/mVaU/9cvdr3X3Y3YcHBgbCKpVi9Nxrkey11NDNrJdaM7/Z3W9tMmQUWDjl9RDwaPHypGP03GuR7LWScjHgBmC3u392hmG3Ae+up11eB+x398c6WKcUpedei2SvlZTLcuDvgJ1m9mB92ceAlwC4+5eAO4GVwMPA74H3dr5UKWzpuWrgIhmbs6G7+w9ofo586hgHLupUUSIiEk53ioqIZEINXUQkE2roIiKZUEMXEcmEGrqISCbU0EVEMqGGLiKSCatFyCv4h83GgV9X8o/P7XjgiaqLKJHml66c5waaXyv+0t2bPgyrsoYeMzMbcffhqusoi+aXrpznBppfUTrlIiKSCTV0EZFMqKE3d23VBZRM80tXznMDza8QnUMXEcmEjtBFRDKhhi4ikol53dDNrMfMHjCzO5q8t87Mxs3swfrX31dRYxFm9oiZ7azXP9LkfTOzq83sYTPbYWavqaLOdrQwt9eb2f4p+y+pz9ozs34z22xmPzOz3WZ2RsP7ye47aGl+ye4/M1s8pe4HzewpM/tQw5hS9l8rn1iUs0uA3cALZnj/a+7+wS7WU4a/dveZbmR4M/Dy+tfpwBfr31Mx29wAvu/uq7pWTWf9C3C3u59jZn8OPLfh/dT33Vzzg0T3n7s/BLwaageNwBjwjYZhpey/eXuEbmZDwFuA66uupUJvA77iNfcB/Wb24qqLmu/M7AXAmdQ+yxd3/6O772sYluy+a3F+uTgL+IW7N94VX8r+m7cNHfg88BHgT7OMeXv9z6HNZrawS3V1kgP/aWbbzezCJu8PAnumvB6tL0vBXHMDOMPMfmJmd5nZyd0srqCXAuPAv9ZPCV5vZs9rGJPyvmtlfpDu/ptqLbCxyfJS9t+8bOhmtgp43N23zzLsdmCRuy8Fvg3c2JXiOmu5u7+G2p93F5nZmQ3vN/us2FRyrHPN7X5qz7x4FfAFYEu3CyzgKOA1wBfdfRnwf8BHG8akvO9amV/K+w+A+qmk1cC/N3u7ybLC+29eNnRgObDazB4BbgHeYGY3TR3g7nvd/Q/1l9cBp3a3xOLc/dH698epncM7rWHIKDD1L48h4NHuVFfMXHNz96fc/en6z3cCvWZ2fNcLbc8oMOru2+qvN1NrgI1jktx3tDC/xPffpDcD97v7/zZ5r5T9Ny8burtf7u5D7r6I2p9E97r7+VPHNJzPWk3t4mkyzOx5Znb05M/A3wL/3TDsNuDd9SvurwP2u/tjXS41WCtzM7MXmZnVfz6N2n/re7tdazvc/X+APWa2uL7oLGBXw7Ak9x20Nr+U998U59H8dAuUtP/me8rlCGa2Hhhx99uAi81sNfAs8CSwrsra2vAXwDfq/5s4Cvg3d7/bzP4RwN2/BNwJrAQeBn4PvLeiWkO1MrdzgA+Y2bPAAWCtp3Vb9D8BN9f/bP8l8N5M9t2kueaX9P4zs+cCfwP8w5Rlpe8/3fovIpKJeXnKRUQkR2roIiKZUEMXEcmEGrqISCbU0EVEMqGGLiKSCTV0EZFM/D/9eJgL33QQQgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.scatter(X[:50,0],X[:50,1], label='0')\n", "plt.scatter(X[50:,0],X[50:,1], label='1')\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "\n", "### AdaBoost in Python" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "class AdaBoost:\n", " def __init__(self, n_estimators=50, learning_rate=1.0):\n", " self.clf_num = n_estimators\n", " self.learning_rate = learning_rate\n", "\n", " def init_args(self, datasets, labels):\n", "\n", " self.X = datasets\n", " self.Y = labels\n", " self.M, self.N = datasets.shape\n", "\n", " # 弱分类器数目和集合\n", " self.clf_sets = []\n", "\n", " # 初始化weights\n", " self.weights = [1.0 / self.M] * self.M\n", "\n", " # G(x)系数 alpha\n", " self.alpha = []\n", "\n", " def _G(self, features, labels, weights):\n", " m = len(features)\n", " error = 100000.0 # 无穷大\n", " best_v = 0.0\n", " # 单维features\n", " features_min = min(features)\n", " features_max = max(features)\n", " n_step = (features_max - features_min +\n", " self.learning_rate) // self.learning_rate\n", " # print('n_step:{}'.format(n_step))\n", " direct, compare_array = None, None\n", " for i in range(1, int(n_step)):\n", " v = features_min + self.learning_rate * i\n", "\n", " if v not in features:\n", " # 误分类计算\n", " compare_array_positive = np.array(\n", " [1 if features[k] > v else -1 for k in range(m)])\n", " weight_error_positive = sum([\n", " weights[k] for k in range(m)\n", " if compare_array_positive[k] != labels[k]\n", " ])\n", "\n", " compare_array_nagetive = np.array(\n", " [-1 if features[k] > v else 1 for k in range(m)])\n", " weight_error_nagetive = sum([\n", " weights[k] for k in range(m)\n", " if compare_array_nagetive[k] != labels[k]\n", " ])\n", "\n", " if weight_error_positive < weight_error_nagetive:\n", " weight_error = weight_error_positive\n", " _compare_array = compare_array_positive\n", " direct = 'positive'\n", " else:\n", " weight_error = weight_error_nagetive\n", " _compare_array = compare_array_nagetive\n", " direct = 'nagetive'\n", "\n", " # print('v:{} error:{}'.format(v, weight_error))\n", " if weight_error < error:\n", " error = weight_error\n", " compare_array = _compare_array\n", " best_v = v\n", " return best_v, direct, error, compare_array\n", "\n", " # 计算alpha\n", " def _alpha(self, error):\n", " return 0.5 * np.log((1 - error) / error)\n", "\n", " # 规范化因子\n", " def _Z(self, weights, a, clf):\n", " return sum([\n", " weights[i] * np.exp(-1 * a * self.Y[i] * clf[i])\n", " for i in range(self.M)\n", " ])\n", "\n", " # 权值更新\n", " def _w(self, a, clf, Z):\n", " for i in range(self.M):\n", " self.weights[i] = self.weights[i] * np.exp(\n", " -1 * a * self.Y[i] * clf[i]) / Z\n", "\n", " # G(x)的线性组合\n", " def _f(self, alpha, clf_sets):\n", " pass\n", "\n", " def G(self, x, v, direct):\n", " if direct == 'positive':\n", " return 1 if x > v else -1\n", " else:\n", " return -1 if x > v else 1\n", "\n", " def fit(self, X, y):\n", " self.init_args(X, y)\n", "\n", " for epoch in range(self.clf_num):\n", " best_clf_error, best_v, clf_result = 100000, None, None\n", " # 根据特征维度, 选择误差最小的\n", " for j in range(self.N):\n", " features = self.X[:, j]\n", " # 分类阈值，分类误差，分类结果\n", " v, direct, error, compare_array = self._G(\n", " features, self.Y, self.weights)\n", "\n", " if error < best_clf_error:\n", " best_clf_error = error\n", " best_v = v\n", " final_direct = direct\n", " clf_result = compare_array\n", " axis = j\n", "\n", " # print('epoch:{}/{} feature:{} error:{} v:{}'.format(epoch, self.clf_num, j, error, best_v))\n", " if best_clf_error == 0:\n", " break\n", "\n", " # 计算G(x)系数a\n", " a = self._alpha(best_clf_error)\n", " self.alpha.append(a)\n", " # 记录分类器\n", " self.clf_sets.append((axis, best_v, final_direct))\n", " # 规范化因子\n", " Z = self._Z(self.weights, a, clf_result)\n", " # 权值更新\n", " self._w(a, clf_result, Z)\n", "\n", "\n", "# print('classifier:{}/{} error:{:.3f} v:{} direct:{} a:{:.5f}'.format(epoch+1, self.clf_num, error, best_v, final_direct, a))\n", "# print('weight:{}'.format(self.weights))\n", "# print('\\n')\n", "\n", " def predict(self, feature):\n", " result = 0.0\n", " for i in range(len(self.clf_sets)):\n", " axis, clf_v, direct = self.clf_sets[i]\n", " f_input = feature[axis]\n", " result += self.alpha[i] * self.G(f_input, clf_v, direct)\n", " # sign\n", " return 1 if result > 0 else -1\n", "\n", " def score(self, X_test, y_test):\n", " right_count = 0\n", " for i in range(len(X_test)):\n", " feature = X_test[i]\n", " if self.predict(feature) == y_test[i]:\n", " right_count += 1\n", "\n", " return right_count / len(X_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 例8.1" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "X = np.arange(10).reshape(10, 1)\n", "y = np.array([1, 1, 1, -1, -1, -1, 1, 1, 1, -1])" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "clf = AdaBoost(n_estimators=3, learning_rate=0.5)\n", "clf.fit(X, y)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "X, y = create_data()\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.5151515151515151" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf = AdaBoost(n_estimators=10, learning_rate=0.2)\n", "clf.fit(X_train, y_train)\n", "clf.score(X_test, y_test)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "average score:63.394%\n" ] } ], "source": [ "# 100次结果\n", "result = []\n", "for i in range(1, 101):\n", " X, y = create_data()\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)\n", " clf = AdaBoost(n_estimators=100, learning_rate=0.2)\n", " clf.fit(X_train, y_train)\n", " r = clf.score(X_test, y_test)\n", " # print('{}/100 score：{}'.format(i, r))\n", " result.append(r)\n", "\n", "print('average score:{:.3f}%'.format(sum(result)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### scikit-learn实例" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-----\n", "#### sklearn.ensemble.AdaBoostClassifier\n", "\n", "- algorithm：这个参数只有AdaBoostClassifier有。主要原因是scikit-learn实现了两种Adaboost分类算法，SAMME和SAMME.R。两者的主要区别是弱学习器权重的度量，SAMME使用了和我们的原理篇里二元分类Adaboost算法的扩展，即用对样本集分类效果作为弱学习器权重，而SAMME.R使用了对样本集分类的预测概率大小来作为弱学习器权重。由于SAMME.R使用了概率度量的连续值，迭代一般比SAMME快，因此AdaBoostClassifier的默认算法algorithm的值也是SAMME.R。我们一般使用默认的SAMME.R就够了，但是要注意的是使用了SAMME.R， 则弱分类学习器参数base_estimator必须限制使用支持概率预测的分类器。SAMME算法则没有这个限制。\n", "\n", "\n", "- n_estimators： AdaBoostClassifier和AdaBoostRegressor都有，就是我们的弱学习器的最大迭代次数，或者说最大的弱学习器的个数。一般来说n_estimators太小，容易欠拟合，n_estimators太大，又容易过拟合，一般选择一个适中的数值。默认是50。在实际调参的过程中，我们常常将n_estimators和下面介绍的参数learning_rate一起考虑。\n", "\n", "\n", "- learning_rate: AdaBoostClassifier和AdaBoostRegressor都有，即每个弱学习器的权重缩减系数ν\n", "\n", "\n", "- base_estimator：AdaBoostClassifier和AdaBoostRegressor都有，即我们的弱分类学习器或者弱回归学习器。理论上可以选择任何一个分类或者回归学习器，不过需要支持样本权重。我们常用的一般是CART决策树或者神经网络MLP。" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "AdaBoostClassifier(learning_rate=0.5, n_estimators=100)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.ensemble import AdaBoostClassifier\n", "clf = AdaBoostClassifier(n_estimators=100, learning_rate=0.5)\n", "clf.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9090909090909091" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.score(X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第8章提升方法-习题\n", "\n", "### 习题8.1\n", "  某公司招聘职员考查身体、业务能力、发展潜力这3项。身体分为合格1、不合格0两级，业务能力和发展潜力分为上1、中2、下3三级分类为合格1 、不合格-1两类。已知10个人的数据，如下表所示。假设弱分类器为决策树桩。试用AdaBoost算法学习一个强分类器。 \n", "\n", "应聘人员情况数据表\n", "\n", "  |1|2|3|4|5|6|7|8|9|10\n", "-|-|-|-|-|-|-|-|-|-|-\n", "身体|0|0|1|1|1|0|1|1|1|0\n", "业务|1|3|2|1|2|1|1|1|3|2\n", "潜力|3|1|2|3|3|2|2|1|1|1\n", "分类|-1|-1|-1|-1|-1|-1|1|1|-1|-1" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "# 加载训练数据\n", "X = np.array([[0, 1, 3], [0, 3, 1], [1, 2, 2], [1, 1, 3], [1, 2, 3], [0, 1, 2],\n", " [1, 1, 2], [1, 1, 1], [1, 3, 1], [0, 2, 1]])\n", "y = np.array([-1, -1, -1, -1, -1, -1, 1, 1, -1, -1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**AdaBoostClassifier分类器实现：**\n", "\n", "采用sklearn的AdaBoostClassifier分类器直接求解，由于AdaBoostClassifier分类器默认采用CART决策树弱分类器，故不需要设置base_estimator参数。" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "原始输出: [-1 -1 -1 -1 -1 -1 1 1 -1 -1]\n", "预测输出: [-1 -1 -1 -1 -1 -1 1 1 -1 -1]\n", "预测正确率：100.00%\n" ] } ], "source": [ "from sklearn.ensemble import AdaBoostClassifier\n", "\n", "clf = AdaBoostClassifier()\n", "clf.fit(X, y)\n", "y_predict = clf.predict(X)\n", "score = clf.score(X, y)\n", "print(\"原始输出:\", y)\n", "print(\"预测输出:\", y_predict)\n", "print(\"预测正确率：{:.2%}\".format(score))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**自编程实现：**" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "# 自编程求解例8.1\n", "import numpy as np\n", "\n", "\n", "class AdaBoost:\n", " def __init__(self, X, y, tol=0.05, max_iter=10):\n", " # 训练数据 实例\n", " self.X = X\n", " # 训练数据 标签\n", " self.y = y\n", " # 训练中止条件 right_rate>self.tol\n", " self.tol = tol\n", " # 最大迭代次数\n", " self.max_iter = max_iter\n", " # 初始化样本权重w\n", " self.w = np.full((X.shape[0]), 1 / X.shape[0])\n", " self.G = [] # 弱分类器\n", "\n", " def build_stump(self):\n", " \"\"\"\n", " 以带权重的分类误差最小为目标，选择最佳分类阈值\n", " best_stump['dim'] 合适的特征所在维度\n", " best_stump['thresh'] 合适特征的阈值\n", " best_stump['ineq'] 树桩分类的标识lt,rt\n", " \"\"\"\n", " m, n = np.shape(self.X)\n", " # 分类误差\n", " e_min = np.inf\n", " # 小于分类阈值的样本属于的标签类别\n", " sign = None\n", " # 最优分类树桩\n", " best_stump = {}\n", " for i in range(n):\n", " range_min = self.X[:, i].min() # 求每一种特征的最大最小值\n", " range_max = self.X[:, i].max()\n", " step_size = (range_max - range_min) / n\n", " for j in range(-1, int(n) + 1):\n", " thresh_val = range_min + j * step_size\n", " # 计算左子树和右子树的误差\n", " for inequal in ['lt', 'rt']:\n", " predict_vals = self.base_estimator(self.X, i, thresh_val,\n", " inequal)\n", " err_arr = np.array(np.ones(m))\n", " err_arr[predict_vals.T == self.y.T] = 0\n", " weighted_error = np.dot(self.w, err_arr)\n", " if weighted_error < e_min:\n", " e_min = weighted_error\n", " sign = predict_vals\n", " best_stump['dim'] = i\n", " best_stump['thresh'] = thresh_val\n", " best_stump['ineq'] = inequal\n", " return best_stump, sign, e_min\n", "\n", " def updata_w(self, alpha, predict):\n", " \"\"\"\n", " 更新样本权重w\n", " \"\"\"\n", " # 以下2行根据公式8.4 8.5 更新样本权重\n", " P = self.w * np.exp(-alpha * self.y * predict)\n", " self.w = P / P.sum()\n", "\n", " @staticmethod\n", " def base_estimator(X, dimen, threshVal, threshIneq):\n", " \"\"\"\n", " 计算单个弱分类器（决策树桩）预测输出\n", " \"\"\"\n", " ret_array = np.ones(np.shape(X)[0]) # 预测矩阵\n", " # 左叶子 ，整个矩阵的样本进行比较赋值\n", " if threshIneq == 'lt':\n", " ret_array[X[:, dimen] <= threshVal] = -1.0\n", " else:\n", " ret_array[X[:, dimen] > threshVal] = -1.0\n", " return ret_array\n", "\n", " def fit(self):\n", " \"\"\"\n", " 对训练数据进行学习\n", " \"\"\"\n", " G = 0\n", " for i in range(self.max_iter):\n", " best_stump, sign, error = self.build_stump() # 获取当前迭代最佳分类阈值\n", " alpha = 1 / 2 * np.log((1 - error) / error) # 计算本轮弱分类器的系数\n", " # 弱分类器权重\n", " best_stump['alpha'] = alpha\n", " # 保存弱分类器\n", " self.G.append(best_stump)\n", " # 以下3行计算当前总分类器（之前所有弱分类器加权和）分类效率\n", " G += alpha * sign\n", " y_predict = np.sign(G)\n", " error_rate = np.sum(\n", " np.abs(y_predict - self.y)) / 2 / self.y.shape[0]\n", " if error_rate < self.tol: # 满足中止条件 则跳出循环\n", " print(\"迭代次数:\", i + 1)\n", " break\n", " else:\n", " self.updata_w(alpha, y_predict) # 若不满足，更新权重，继续迭代\n", "\n", " def predict(self, X):\n", " \"\"\"\n", " 对新数据进行预测\n", " \"\"\"\n", " m = np.shape(X)[0]\n", " G = np.zeros(m)\n", " for i in range(len(self.G)):\n", " stump = self.G[i]\n", " # 遍历每一个弱分类器，进行加权\n", " _G = self.base_estimator(X, stump['dim'], stump['thresh'],\n", " stump['ineq'])\n", " alpha = stump['alpha']\n", " G += alpha * _G\n", " y_predict = np.sign(G)\n", " return y_predict.astype(int)\n", "\n", " def score(self, X, y):\n", " \"\"\"对训练效果进行评价\"\"\"\n", " y_predict = self.predict(X)\n", " error_rate = np.sum(np.abs(y_predict - y)) / 2 / y.shape[0]\n", " return 1 - error_rate" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "迭代次数: 8\n", "原始输出: [-1 -1 -1 -1 -1 -1 1 1 -1 -1]\n", "预测输出: [-1 -1 -1 -1 -1 -1 1 1 -1 -1]\n", "预测正确率：100.00%\n" ] } ], "source": [ "clf = AdaBoost(X, y)\n", "clf.fit()\n", "y_predict = clf.predict(X)\n", "score = clf.score(X, y)\n", "print(\"原始输出:\", y)\n", "print(\"预测输出:\", y_predict)\n", "print(\"预测正确率：{:.2%}\".format(score))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题8.2\n", "  比较支持向量机、 AdaBoost 、Logistic回归模型的学习策略与算法" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "- **支持向量机** \n", "学习策略：极小化正则化合页损失，软间隔最大化； \n", "学习算法：序列最小最优化算法（SMO） \n", "- **AdaBoost** \n", "学习策略：极小化加法模型指数损失； \n", "学习算法：前向分步加法算法 \n", "- **Logistic回归** \n", "学习策略：极大似然估计，正则化的极大似然估计； \n", "学习算法：改进的迭代尺度算法，梯度下降，拟牛顿法" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "参考代码：https://github.com/wzyonggege/statistical-learning-method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第09章 EM算法及其推广/9.Expectation_Maximization.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第9章 EM算法及其推广" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Expectation Maximization algorithm\n", "\n", "### Maximum likehood function\n", "\n", "[likehood & maximum likehood](http://fangs.in/post/thinkstats/likelihood/)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1．EM算法是含有隐变量的概率模型极大似然估计或极大后验概率估计的迭代算法。含有隐变量的概率模型的数据表示为$\\theta$ )。这里，$Y$是观测变量的数据，$Z$是隐变量的数据，$\\theta$ 是模型参数。EM算法通过迭代求解观测数据的对数似然函数${L}(\\theta)=\\log {P}(\\mathrm{Y} | \\theta)$的极大化，实现极大似然估计。每次迭代包括两步：\n", "\n", "$E$步，求期望，即求$logP\\left(Z | Y, \\theta\\right)$ )关于$ P\\left(Z | Y, \\theta^{(i)}\\right)$)的期望：\n", "\n", "$$Q\\left(\\theta, \\theta^{(i)}\\right)=\\sum_{Z} \\log P(Y, Z | \\theta) P\\left(Z | Y, \\theta^{(i)}\\right)$$\n", "称为$Q$函数，这里$\\theta^{(i)}$是参数的现估计值；\n", "\n", "$M$步，求极大，即极大化$Q$函数得到参数的新估计值：\n", "\n", "$$\\theta^{(i+1)}=\\arg \\max _{\\theta} Q\\left(\\theta, \\theta^{(i)}\\right)$$\n", " \n", "在构建具体的EM算法时，重要的是定义$Q$函数。每次迭代中，EM算法通过极大化$Q$函数来增大对数似然函数${L}(\\theta)$。\n", "\n", "2．EM算法在每次迭代后均提高观测数据的似然函数值，即\n", "\n", "$$P\\left(Y | \\theta^{(i+1)}\\right) \\geqslant P\\left(Y | \\theta^{(i)}\\right)$$\n", "\n", "在一般条件下EM算法是收敛的，但不能保证收敛到全局最优。\n", "\n", "3．EM算法应用极其广泛，主要应用于含有隐变量的概率模型的学习。高斯混合模型的参数估计是EM算法的一个重要应用，下一章将要介绍的隐马尔可夫模型的非监督学习也是EM算法的一个重要应用。\n", "\n", "4．EM算法还可以解释为$F$函数的极大-极大算法。EM算法有许多变形，如GEM算法。GEM算法的特点是每次迭代增加$F$函数值（并不一定是极大化$F$函数），从而增加似然函数值。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 在统计学中，似然函数（likelihood function，通常简写为likelihood，似然）是一个非常重要的内容，在非正式场合似然和概率（Probability）几乎是一对同义词，但是在统计学中似然和概率却是两个不同的概念。概率是在特定环境下某件事情发生的可能性，也就是结果没有产生之前依据环境所对应的参数来预测某件事情发生的可能性，比如抛硬币，抛之前我们不知道最后是哪一面朝上，但是根据硬币的性质我们可以推测任何一面朝上的可能性均为50%，这个概率只有在抛硬币之前才是有意义的，抛完硬币后的结果便是确定的；而似然刚好相反，是在确定的结果下去推测产生这个结果的可能环境（参数），还是抛硬币的例子，假设我们随机抛掷一枚硬币1,000次，结果500次人头朝上，500次数字朝上（实际情况一般不会这么理想，这里只是举个例子），我们很容易判断这是一枚标准的硬币，两面朝上的概率均为50%，这个过程就是我们运用出现的结果来判断这个事情本身的性质（参数），也就是似然。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$$P(Y|\\theta) = \\prod[\\pi p^{y_i}(1-p)^{1-y_i}+(1-\\pi) q^{y_i}(1-q)^{1-y_i}]$$\n", "\n", "### E step:\n", "\n", "$$\\mu^{i+1}=\\frac{\\pi (p^i)^{y_i}(1-(p^i))^{1-y_i}}{\\pi (p^i)^{y_i}(1-(p^i))^{1-y_i}+(1-\\pi) (q^i)^{y_i}(1-(q^i))^{1-y_i}}$$" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import math" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "pro_A, pro_B, por_C = 0.5, 0.5, 0.5\n", "\n", "\n", "def pmf(i, pro_A, pro_B, por_C):\n", " pro_1 = pro_A * math.pow(pro_B, data[i]) * math.pow(\n", " (1 - pro_B), 1 - data[i])\n", " pro_2 = pro_A * math.pow(pro_C, data[i]) * math.pow(\n", " (1 - pro_C), 1 - data[i])\n", " return pro_1 / (pro_1 + pro_2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### M step:\n", "\n", "$$\\pi^{i+1}=\\frac{1}{n}\\sum_{j=1}^n\\mu^{i+1}_j$$\n", "\n", "$$p^{i+1}=\\frac{\\sum_{j=1}^n\\mu^{i+1}_jy_i}{\\sum_{j=1}^n\\mu^{i+1}_j}$$\n", "\n", "$$q^{i+1}=\\frac{\\sum_{j=1}^n(1-\\mu^{i+1}_jy_i)}{\\sum_{j=1}^n(1-\\mu^{i+1}_j)}$$" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "class EM:\n", " def __init__(self, prob):\n", " self.pro_A, self.pro_B, self.pro_C = prob\n", "\n", " # e_step\n", " def pmf(self, i):\n", " pro_1 = self.pro_A * math.pow(self.pro_B, data[i]) * math.pow(\n", " (1 - self.pro_B), 1 - data[i])\n", " pro_2 = (1 - self.pro_A) * math.pow(self.pro_C, data[i]) * math.pow(\n", " (1 - self.pro_C), 1 - data[i])\n", " return pro_1 / (pro_1 + pro_2)\n", "\n", " # m_step\n", " def fit(self, data):\n", " count = len(data)\n", " print('init prob:{}, {}, {}'.format(self.pro_A, self.pro_B,\n", " self.pro_C))\n", " for d in range(count):\n", " _ = yield\n", " _pmf = [self.pmf(k) for k in range(count)]\n", " pro_A = 1 / count * sum(_pmf)\n", " pro_B = sum([_pmf[k] * data[k] for k in range(count)]) / sum(\n", " [_pmf[k] for k in range(count)])\n", " pro_C = sum([(1 - _pmf[k]) * data[k]\n", " for k in range(count)]) / sum([(1 - _pmf[k])\n", " for k in range(count)])\n", " print('{}/{} pro_a:{:.3f}, pro_b:{:.3f}, pro_c:{:.3f}'.format(\n", " d + 1, count, pro_A, pro_B, pro_C))\n", " self.pro_A = pro_A\n", " self.pro_B = pro_B\n", " self.pro_C = pro_C" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "data=[1,1,0,1,0,0,1,0,1,1]" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "init prob:0.5, 0.5, 0.5\n" ] } ], "source": [ "em = EM(prob=[0.5, 0.5, 0.5])\n", "f = em.fit(data)\n", "next(f)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1/10 pro_a:0.500, pro_b:0.600, pro_c:0.600\n" ] } ], "source": [ "# 第一次迭代\n", "f.send(1)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2/10 pro_a:0.500, pro_b:0.600, pro_c:0.600\n" ] } ], "source": [ "# 第二次\n", "f.send(2)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "init prob:0.4, 0.6, 0.7\n" ] } ], "source": [ "em = EM(prob=[0.4, 0.6, 0.7])\n", "f2 = em.fit(data)\n", "next(f2)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1/10 pro_a:0.406, pro_b:0.537, pro_c:0.643\n" ] } ], "source": [ "f2.send(1)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2/10 pro_a:0.406, pro_b:0.537, pro_c:0.643\n" ] } ], "source": [ "f2.send(2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第9章EM算法及其推广-习题\n", "### 习题9.1\n", "\n", "  如例9.1的三硬币模型，假设观测数据不变，试选择不同的处置，例如，$\\pi^{(0)}=0.46,p^{(0)}=0.55,q^{(0)}=0.67$，求模型参数为$\\theta=(\\pi,p,q)$的极大似然估计。 \n", "\n", "**解答：**" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import numpy as np\n", "import math\n", "\n", "class EM:\n", " def __init__(self, prob):\n", " self.pro_A, self.pro_B, self.pro_C = prob\n", "\n", " def pmf(self, i):\n", " pro_1 = self.pro_A * math.pow(self.pro_B, data[i]) * math.pow(\n", " (1 - self.pro_B), 1 - data[i])\n", " pro_2 = (1 - self.pro_A) * math.pow(self.pro_C, data[i]) * math.pow(\n", " (1 - self.pro_C), 1 - data[i])\n", " return pro_1 / (pro_1 + pro_2)\n", "\n", " def fit(self, data):\n", " print('init prob:{}, {}, {}'.format(self.pro_A, self.pro_B,\n", " self.pro_C))\n", " count = len(data)\n", " theta = 1\n", " d = 0\n", " while (theta > 0.00001):\n", " # 迭代阻塞\n", " _pmf = [self.pmf(k) for k in range(count)]\n", " pro_A = 1 / count * sum(_pmf)\n", " pro_B = sum([_pmf[k] * data[k] for k in range(count)]) / sum(\n", " [_pmf[k] for k in range(count)])\n", " pro_C = sum([(1 - _pmf[k]) * data[k]\n", " for k in range(count)]) / sum([(1 - _pmf[k])\n", " for k in range(count)])\n", " d += 1\n", " print('{} pro_a:{:.4f}, pro_b:{:.4f}, pro_c:{:.4f}'.format(\n", " d, pro_A, pro_B, pro_C))\n", " theta = abs(self.pro_A - pro_A) + abs(self.pro_B -\n", " pro_B) + abs(self.pro_C -\n", " pro_C)\n", " self.pro_A = pro_A\n", " self.pro_B = pro_B\n", " self.pro_C = pro_C" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "init prob:0.46, 0.55, 0.67\n", "1 pro_a:0.4619, pro_b:0.5346, pro_c:0.6561\n", "2 pro_a:0.4619, pro_b:0.5346, pro_c:0.6561\n" ] } ], "source": [ "# 加载数据\n", "data = [1, 1, 0, 1, 0, 0, 1, 0, 1, 1]\n", "\n", "em = EM(prob=[0.46, 0.55, 0.67])\n", "f = em.fit(data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "可见通过两次迭代，参数已经收敛，三个硬币的概率分别为0.4619，0.5346，0.6561" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题9.2\n", "证明引理9.2。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **引理9.2：**若$\\tilde{P}_{\\theta}(Z)=P(Z | Y, \\theta)$，则$$F(\\tilde{P}, \\theta)=\\log P(Y|\\theta)$$\n", "\n", "**证明：** \n", "由$F$函数的定义（**定义9.3**）可得：$$F(\\tilde{P}, \\theta)=E_{\\tilde{P}}[\\log P(Y,Z|\\theta)] + H(\\tilde{P})$$其中，$H(\\tilde{P})=-E_{\\tilde{P}} \\log \\tilde{P}(Z)$ \n", "$\\begin{aligned}\n", "\\therefore F(\\tilde{P}, \\theta) \n", "&= E_{\\tilde{P}}[\\log P(Y,Z|\\theta)] -E_{\\tilde{P}} \\log \\tilde{P}(Z) \\\\\n", "&= \\sum_Z \\log P(Y,Z|\\theta) \\tilde{P}_{\\theta}(Z) - \\sum_Z \\log \\tilde{P}(Z) \\cdot \\tilde{P}(Z) \\\\\n", "&= \\sum_Z \\log P(Y,Z|\\theta) P(Z|Y,\\theta) - \\sum_Z \\log P(Z|Y,\\theta) \\cdot P(Z|Y,\\theta) \\\\\n", "&= \\sum_Z P(Z|Y,\\theta) \\left[ \\log P(Y,Z|\\theta) - \\log P(Z|Y,\\theta) \\right] \\\\\n", "&= \\sum_Z P(Z|Y,\\theta) \\log \\frac{P(Y,Z|\\theta)}{P(Z|Y,\\theta)} \\\\\n", "&= \\sum_Z P(Z|Y,\\theta) \\log P(Y|\\theta) \\\\\n", "&= \\log P(Y|\\theta) \\sum_Z P(Z|Y,\\theta)\n", "\\end{aligned}$ \n", "$\\displaystyle \\because \\sum_Z \\tilde{P}_{\\theta}(Z) = P(Z|Y, \\theta) = 1$ \n", "$\\therefore F(\\tilde{P}, \\theta) = \\log P(Y|\\theta)$，引理9.2得证。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题9.3\n", "已知观测数据 \n", "-67，-48，6，8，14，16，23，24，28，29，41，49，56，60，75 \n", "试估计两个分量的高斯混合模型的5个参数。\n", "\n", "**解答：**" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "labels = [1 1 0 0 0 0 0 0 0 0 0 0 0 0 0]\n" ] } ], "source": [ "from sklearn.mixture import GaussianMixture\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "# 初始化观测数据\n", "data = np.array([-67, -48, 6, 8, 14, 16, 23, 24, 28, 29, 41, 49, 56, 60,\n", " 75]).reshape(-1, 1)\n", "\n", "# 聚类\n", "gmmModel = GaussianMixture(n_components=2)\n", "gmmModel.fit(data)\n", "labels = gmmModel.predict(data)\n", "print(\"labels =\", labels)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEWCAYAAACNJFuYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZ40lEQVR4nO3de5RlZX3m8e8jpSLYCKSb5tJoo2IiIaQlJQM6YRKRiIrgmkknGC9EnZAYb0nMiMRZSXTFeEmWxkxudkTFSDQ03tqMJiJqmEREG20vgEpHIrRAU0aB9gaW/OaPvQsPvU91FXSf2nU4389avc7Zl3P271RVn2e/77svqSokSRp0n74LkCQtP4aDJKnDcJAkdRgOkqQOw0GS1GE4SJI6DAfdKyX5UJIze9z+g5N8O8lefdWw1JL8XJJti1z3D5O8Y9Q16Z4zHLRHJDkjyWVJvpPkpvb5byZJH/VU1ROr6rw9/b5JfjVJJXn9TvOf2s5/W7v9a6vqgVX1w0W859uS/NGernWBbVaS7UmmBuZNtb87T36S4aDdl+QlwBuBPwEOBlYDvwE8Frhfj6WNyr8Dvzz4xQo8C/hKH8XsVMfdcTPwxIHpJwHf2v2KdG9gOGi3JHkQ8ErgN6vqwqraUY3PVtXTq+q2dr0nJ/lskluTXJfkDwfeo9MdkeQ/kjy+fX5cks3ta7fP7bUn2TvJO5L8Z5Kbk3w6yep22ceT/M/2+cOSfLRd7xtJzk+y/07b+t0kn09yS5J/SLL3Lj72jcAXgCe0rz8QeAywaeA917Z751NJDkyyLclT2mUPTLI1ybOSnAU8HXhp2w31gXadSvLwgfe7s3Ux9/NKcnaSG4G3tvNPTbKl/Vl8IskxC/z6/o4m1OY8C3j7Tr+HQ5NsSvLNtuZfG1j2gLaubyW5Enj0kNe+O8lMkmuSvGiBerSMGA7aXScA9wfev8B636H58tkfeDLwvCRPXeQ23gi8sar2Ax4GXNDOPxN4EHA48GM0rZXvDXl9gFcDhwKPbNf/w53W+SXgFOAI4BjgVxeo6e386Iv1DJrPf9uwFavqm8BzgL9NchDwBmBLVb29qjYA5wOva7uhnrLAduccDBwIPAQ4K8mxwFuAX6f5WbwJ2JTk/rt4j/cBJybZvw3Ln6X7e3wnsI3mZ/eLwB8nOald9gc0v4+H0QTlnWM8Se4DfAD4HHAYcBLwW0mesMjPp54ZDtpdK4FvVNXs3Ix2r/XmJN9LciJAVX28qr5QVXdU1edpvnT+2yK38QPg4UlWVtW3q+qTA/N/DHh4Vf2wqi6vqlt3fnFVba2qi6rqtqqaAV4/ZNt/XlXXt1/kHwDWLVDTe4Gfa1tOnT3uITV8GNgIXEwTjr++wPsv5A7gD9rP9D3g14A3VdVl7c/iPJqwOn4X7/F9ms/6yzQBt6mdB0CSw4H/CpxdVd+vqi3Am4Fntqv8EvCqqvpmVV0H/PnAez8aWFVVr6yq26vqq8DfttvRGDActLv+E1g52O9dVY+pqv3bZfcBSPJfknys7WK4hWYvf+Uit/Fc4BHAl9quo1Pb+X8H/DPwriTXJ3ldkvvu/OIkByV5V5KvJ7kVeMeQbd848Py7wAN3VVD7hfx/gf8NrKyqf1vE59gAHA28tar+cxHr78pMVX1/YPohwEvaUL45yc00LaRDF3ifuRbQsIA7FPhmVe0YmPc1mpbA3PLrdlo2WM+hO9XzezTjURoDhoN216U0e6inL7De39PsmR5eVQ8C/oamuweaLqd95lZMc/jnqrnpqrq6qp4GHAS8Frgwyb5V9YOqekVVHUXT538qd+1Dn/NqoIBj2q6pZwxse3e8HXgJTUjtUvuZ3tS+5nmD4wltbTv7LgM/E5pupEE7v+Y6mr34/Qf+7VNV71ygtP8HHELzpf2vOy27HjgwyYqBeQ8Gvt4+v4EmgAaXDdZzzU71rKiqJy1Qj5YJw0G7papuBl4B/FWSX2wHW++TZB2w78CqK2j2Qr+f5DjgVwaWfQXYux20vi/N3vidfeVJnpFkVVXdQXOEDcAPk/x8kp9qv3hvpelmGnbo6Arg28DNSQ4D/tce+fDwL8DJwP9ZxLq/1z4+B/hT4O350TkQ24GH7rT+FuBXkuyV5BQW7oL7W+A32hZakuzb/jxX7OpF1Vyz/ynAabXT9fvbrqJPAK9uB/+PoWnFnd+ucgFwTpIDkqwBXjjw8k8Bt7aD5g9oP8fRSe4yaK3ly3DQbquq1wG/A7wUuInmy+5NwNk0Xy4Avwm8MskO4Pf50aAyVXVLu/zNNHul36EZBJ1zCnBFkm/TDE6f0XapHAxcSBMMV9F8WQ87seoVwLHALTRdQe/Z7Q/d1F1VdXE7TjGvJD9D8/N5Vnvew2tp9vxf1q5yLnBU2/3yvnbei2m+tG+mOZrpfexCVW2mGXf4C5rDUbey8KD63GuvqKor5ln8NGAtTSvivTTjHBe1y15B05V0DfBhBlpQ7ed8Cs3YzTXAN2h+vw9aTE3qX7zZjyRpZ7YcJEkdhoMkqcNwkCR1GA6SpI57esGuZWXlypW1du3avsuQpLFy+eWXf6OqVg1bdq8Ih7Vr17J58+a+y5CksZLka/Mt67VbKclvJ7kiyReTvLM90eaINPcCuDrN1THvjZd8lqRlrbdwaM9UfREwXVVHA3vRXJTrtcAbqupImpN5nttXjZI0qfoekJ4CHtBetG0fmmu1PI7mrFeA84DFXtZZkrSH9BYOVfV1mmvMXEsTCrcAlwM3D1z+eRs/ugLkXSQ5K80NYDbPzMwsRcmSNDH67FY6gOZKnkfQXPp3X+56y8I5Q6/vUVUbqmq6qqZXrRo62C5Juof67FZ6PM0lfWeq6gc0F0N7DLD/wL0B1tBc8EuStIT6DIdrgeOT7JMkNLcRvBL4GM3tCKG57eBCt5+UJO1hfY45XEYz8PwZmpu134fmTllnA7+TZCvNLSDP7atGSZpUvR6tVFV/UFU/UVVHV9Uz2/vhfrWqjquqh1fV+qoaetN2SZpI27fDiSfCfvs1j9u3j2QzfR/KKkm6O9avh0svhR07msf160eyGcNBksbJli0w2x7tPzvbTI+A4SBJ42TdOphqD+icmmqmR8BwkKRxsnEjnHACrFjRPG7cOJLN3CuuyipJE2P1arjkkpFvxpaDJKnDcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR2GgySpw3CQJHUYDpKkDsNBktRhOEiSOgwHSVJHr+GQZP8kFyb5UpKrkpyQ5MAkFyW5un08oM8aJWkS9d1yeCPwT1X1E8BPA1cBLwMurqojgYvbaUkaL9u3w4knwn77NY/bt/dd0d3SWzgk2Q84ETgXoKpur6qbgdOB89rVzgOe2k+FkrQb1q+HSy+FHTuax/Xr+67obumz5fBQYAZ4a5LPJnlzkn2B1VV1A0D7eNCwFyc5K8nmJJtnZmaWrmpJWowtW2B2tnk+O9tMj5E+w2EKOBb466p6FPAd7kYXUlVtqKrpqppetWrVqGqUpHtm3TqYau/EPDXVTI+RPsNhG7Ctqi5rpy+kCYvtSQ4BaB9v6qk+SbrnNm6EE06AFSuax40b+67obuktHKrqRuC6JD/ezjoJuBLYBJzZzjsTeH8P5UmaJKMYPF69Gi65BG69tXlcvXr333MJTfW8/RcC5ye5H/BV4Nk0gXVBkucC1wLjNYojafzMDR7Pzv5o8PiSS/quqle9hkNVbQGmhyw6aalrkTTBxnzweBT6Ps9Bkvo35oPHo2A4SNKYDx6PQt9jDpLUv7nBY93JloMkqcNwkCR1GA6SpA7DQZLUYThIGh9jfhnscWI4SBofY34Z7HFiOEgaH57JvGQMB0njwzOZl4zhIGl8eCbzkvEMaUnjwzOZl4wtB0lSh+EgSeowHCRJHYaDJKnDcJAkdfQeDkn2SvLZJP/YTh+R5LIkVyf5h/b+0pLGjZe6GGu9hwPwYuCqgenXAm+oqiOBbwHP7aUqSbvHS12MtV7DIcka4MnAm9vpAI8DLmxXOQ94aj/VSdotXupirPXdcvgz4KXAHe30jwE3V1X7F8U24LBhL0xyVpLNSTbPzMyMvlJJd4+XuhhrvYVDklOBm6rq8sHZQ1atYa+vqg1VNV1V06tWrRpJjZJ2g5e6GGt9Xj7jscBpSZ4E7A3sR9OS2D/JVNt6WANc32ONku4pL3Ux1nprOVTVOVW1pqrWAmcAH62qpwMfA36xXe1M4P09lShJE6vvMYdhzgZ+J8lWmjGIc3uuR5ImzrK4KmtVfRz4ePv8q8BxfdYjSZNuObYcJEk9MxwkSR2GgySpw3CQJHUYDpKkDsNBktRhOEiSOgwHadJ53wUNYThIk877LmgIw0EaF6Paw/e+CxrCcJDGxaj28L3vgoYwHKRxMao9fO+7oCGWxYX3JC3CunVNi2F2ds/u4XvfBQ1hy0EaF+7hawkZDtIojGLweG4P/9Zbm8fVq3f/PaV5GA7SKHh4qMac4aDJ5uGh0lCGgyabh4dKQxkOmmweHioN1Vs4JDk8yceSXJXkiiQvbucfmOSiJFe3jwf0VaMmwKj28B081pjrs+UwC7ykqh4JHA88P8lRwMuAi6vqSODidloaDffwpaF6Owmuqm4Abmif70hyFXAYcDrwc+1q5wEfB87uoURNAk8Ak4ZaFmMOSdYCjwIuA1a3wTEXIAfN85qzkmxOsnlmZmapSpWkidB7OCR5IPBu4Leq6tbFvq6qNlTVdFVNr1q1anQFStIE6jUcktyXJhjOr6r3tLO3JzmkXX4IcFNf9UnSpOrzaKUA5wJXVdXrBxZtAs5sn58JvH+pa9My5R3LpCXTZ8vhscAzgccl2dL+exLwGuDkJFcDJ7fTkpekkJZQn0cr/SuQeRaftJS1aEx4SQppyfQ+IK17oVF1/3hJCmnJGA7a80bV/eMJa9KS8U5w2vNG1f3jCWvSkrHloD3P7h9p7BkO2vPs/pHGnt1K2vPs/pHGni0HSVKH4SBJ6jAcJpmXo5A0D8Nhknk5CknzMBwmmZejkDQPw2FcjKILyPMRJM3DcBgXo+gC8nwESfPwPIdxMYouIM9HkDSPBVsOSV6Q5IClKEa7YBeQpCW0mG6lg4FPJ7kgySntHdy01OwCkrSEUlULr9QEwi8AzwamgQuAc6vq30db3uJMT0/X5s2b+y5DksZKksuranrYskUNSFeTIDe2/2aBA4ALk7xuj1UpSVo2FjPm8KIklwOvA/4N+Kmqeh7wM8D/GFVhbRfWl5NsTfKyUW1HktS1mKOVVgL/vaq+Njizqu5IcuooikqyF/CXwMnANpoxj01VdeUotidJuqsFWw5V9fs7B8PAsqv2fEkAHAdsraqvVtXtwLuA00e0LUnSTpbrSXCHAdcNTG9r590pyVlJNifZPDMzs6TFSdK93XINh2GHy97lsKqq2lBV01U1vWrVqiUqS5Imw3INh23A4QPTa4Dre6pFkibOcg2HTwNHJjkiyf2AM4BNPdckSRNjWV5bqapmk7wA+GdgL+AtVXVFz2VJ0sRYluEAUFUfBD7Ydx2SNImWa7eSJKlHhoMkqcNwkCR1GA6SpA7DQZLUYThIkjoMB0lSh+EgSeowHCRJHYaDJKnDcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR2GgySpw3CQJHX0Eg5J/iTJl5J8Psl7k+w/sOycJFuTfDnJE/qoT5ImXV8th4uAo6vqGOArwDkASY4CzgB+EjgF+Kske/VUoyRNrF7Coao+XFWz7eQngTXt89OBd1XVbVV1DbAVOK6PGiVpki2HMYfnAB9qnx8GXDewbFs7ryPJWUk2J9k8MzMz4hIlabJMjeqNk3wEOHjIopdX1fvbdV4OzALnz71syPo17P2ragOwAWB6enroOpKke2Zk4VBVj9/V8iRnAqcCJ1XV3Jf7NuDwgdXWANePpkJJ0nz6OlrpFOBs4LSq+u7Aok3AGUnun+QI4EjgU33UKEmTbGQthwX8BXB/4KIkAJ+sqt+oqiuSXABcSdPd9Pyq+mFPNUrSxOolHKrq4btY9irgVUtYjiRpJ8vhaCVJ0jJjOEiSOgwHSVKH4SBJ6jAcJEkdhoMkqcNwkCR1GA6SpA7DQZLUYThIkjoMB0lSh+EgSeowHCRJHYaDJKnDcJAkdRgOkqQOw0GS1GE4SJI6eg2HJL+bpJKsbKeT5M+TbE3y+STH9lmfJE2q3sIhyeHAycC1A7OfCBzZ/jsL+OseSpOkiddny+ENwEuBGph3OvD2anwS2D/JIb1UJ0kTrJdwSHIa8PWq+txOiw4DrhuY3tbOkyQtoalRvXGSjwAHD1n0cuD3gF8Y9rIh82rIPJKcRdP1xIMf/OB7WKUkaZiRhUNVPX7Y/CQ/BRwBfC4JwBrgM0mOo2kpHD6w+hrg+nnefwOwAWB6enpogEiS7pkl71aqqi9U1UFVtbaq1tIEwrFVdSOwCXhWe9TS8cAtVXXDUtcoSZNuZC2He+iDwJOArcB3gWf3W44kTabew6FtPcw9L+D5/VUjSQLPkJYkDWE4SJI6DAdJUofhIEnqMBz2sO3b4cQTYb/9msft2/uuSJLuPsNhD1u/Hi69FHbsaB7Xr++7Ikm6+wyHPWzLFpidbZ7PzjbTkjRuDIc9bN06mGrPHpmaaqYladwYDnvYxo1wwgmwYkXzuHFj3xVJ0t3X+xnS9zarV8Mll/RdhSTtHlsOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR2GgySpw3CQJHUYDpKkjt7CIckLk3w5yRVJXjcw/5wkW9tlT+irPkmaZL1cPiPJzwOnA8dU1W1JDmrnHwWcAfwkcCjwkSSPqKof9lGnJE2qvloOzwNeU1W3AVTVTe3804F3VdVtVXUNsBU4rqcaJWli9RUOjwB+NsllSf4lyaPb+YcB1w2st62d15HkrCSbk2yemZkZcbmSNFlG1q2U5CPAwUMWvbzd7gHA8cCjgQuSPBTIkPVr2PtX1QZgA8D09PTQdSRJ98zIwqGqHj/fsiTPA95TVQV8KskdwEqalsLhA6uuAa4fVY2SpOH66lZ6H/A4gCSPAO4HfAPYBJyR5P5JjgCOBD7VU42SNLH6utnPW4C3JPkicDtwZtuKuCLJBcCVwCzwfI9UkqSl10s4VNXtwDPmWfYq4FVLW5EkaZBnSEuSOgwHSVLHRIfD9u1w4omw337N4/btfVckScvDRIfD+vVw6aWwY0fzuH593xVJ0vIw0eGwZQvMzjbPZ2ebaUnShIfDunUw1R6vNTXVTEuSJjwcNm6EE06AFSuax40b+65IkpaHvk6CWxZWr4ZLLum7Cklafia65SBJGs5wkCR1GA6SpA7DQZLUYThIkjoMB0lSR5rbKIy3JDPA13bjLVbS3GxoHFjraFjr6IxTvZNW60OqatWwBfeKcNhdSTZX1XTfdSyGtY6GtY7OONVrrT9it5IkqcNwkCR1GA6NDX0XcDdY62hY6+iMU73W2nLMQZLUYctBktRhOEiSOiY6HJKckuTLSbYmeVnf9cwnyeFJPpbkqiRXJHlx3zUtJMleST6b5B/7rmUhSfZPcmGSL7U/4xP6rmk+SX67/Rv4YpJ3Jtm775rmJHlLkpuSfHFg3oFJLkpydft4QJ81Dpqn3j9p/w4+n+S9Sfbvs8Y5w2odWPa7SSrJyj25zYkNhyR7AX8JPBE4CnhakqP6rWpes8BLquqRwPHA85dxrXNeDFzVdxGL9Ebgn6rqJ4CfZpnWneQw4EXAdFUdDewFnNFvVXfxNuCUnea9DLi4qo4ELm6nl4u30a33IuDoqjoG+ApwzlIXNY+30a2VJIcDJwPX7ukNTmw4AMcBW6vqq1V1O/Au4PSeaxqqqm6oqs+0z3fQfHkd1m9V80uyBngy8Oa+a1lIkv2AE4FzAarq9qq6ud+qdmkKeECSKWAf4Pqe67lTVV0CfHOn2acD57XPzwOeuqRF7cKweqvqw1XV3lmeTwJrlrywIeb52QK8AXgpsMePLJrkcDgMuG5gehvL+At3TpK1wKOAy/qtZJf+jOYP9o6+C1mEhwIzwFvbbrA3J9m376KGqaqvA39Ks5d4A3BLVX2436oWtLqqboBmJwc4qOd67o7nAB/qu4j5JDkN+HpVfW4U7z/J4ZAh85b1cb1JHgi8G/itqrq173qGSXIqcFNVXd53LYs0BRwL/HVVPQr4Dsur6+NObX/96cARwKHAvkme0W9V905JXk7TnXt+37UMk2Qf4OXA749qG5McDtuAwwem17CMmug7S3JfmmA4v6re03c9u/BY4LQk/0HTVfe4JO/ot6Rd2gZsq6q5ltiFNGGxHD0euKaqZqrqB8B7gMf0XNNCtic5BKB9vKnnehaU5EzgVODptXxPBHsYzU7C59r/a2uAzyQ5eE9tYJLD4dPAkUmOSHI/moG9TT3XNFSS0PSJX1VVr++7nl2pqnOqak1VraX5mX60qpbt3m1V3Qhcl+TH21knAVf2WNKuXAscn2Sf9m/iJJbp4PmATcCZ7fMzgff3WMuCkpwCnA2cVlXf7bue+VTVF6rqoKpa2/5f2wYc2/497xETGw7toNMLgH+m+Q92QVVd0W9V83os8EyavfAt7b8n9V3UvcgLgfOTfB5YB/xxz/UM1bZuLgQ+A3yB5v/vsrncQ5J3ApcCP55kW5LnAq8BTk5yNc1RNa/ps8ZB89T7F8AK4KL2/9nf9Fpka55aR7vN5dtqkiT1ZWJbDpKk+RkOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR2GgzQCSR7d3hNg7yT7tvdgOLrvuqTF8iQ4aUSS/BGwN/AAmus3vbrnkqRFMxykEWmv2fVp4PvAY6rqhz2XJC2a3UrS6BwIPJDmWj3L5nae0mLYcpBGJMkmmsuWHwEcUlUv6LkkadGm+i5AujdK8ixgtqr+vr1f+SeSPK6qPtp3bdJi2HKQJHU45iBJ6jAcJEkdhoMkqcNwkCR1GA6SpA7DQZLUYThIkjr+Pw5PPY71Yn0iAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "means = [[ 32.98489643 -57.51107027]]\n", "covariances = [[429.45764867 90.24987882]]\n", "weights = [[0.86682762 0.13317238]]\n" ] } ], "source": [ "for i in range(0, len(labels)):\n", " if labels[i] == 0:\n", " plt.scatter(i, data.take(i), s=15, c='red')\n", " elif labels[i] == 1:\n", " plt.scatter(i, data.take(i), s=15, c='blue')\n", "plt.title('Gaussian Mixture Model')\n", "plt.xlabel('x')\n", "plt.ylabel('y')\n", "plt.show()\n", "print(\"means =\", gmmModel.means_.reshape(1, -1))\n", "print(\"covariances =\", gmmModel.covariances_.reshape(1, -1))\n", "print(\"weights = \", gmmModel.weights_.reshape(1, -1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题9.4\n", "  EM算法可以用到朴素贝叶斯法的非监督学习，试写出其算法。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "> **EM算法的一般化：** \n", "**E步骤：**根据参数初始化或上一次迭代的模型参数来计算出隐变量的后验概率，其实就是隐变量的期望。作为隐变量的现估计值：$$w_j^{(i)}=Q_{i}(z^{(i)}=j) := p(z^{(i)}=j | x^{(i)} ; \\theta)$$\n", "**M步骤：**将似然函数最大化以获得新的参数值：$$\n", "\\theta :=\\arg \\max_{\\theta} \\sum_i \\sum_{z^{(i)}} Q_i (z^{(i)}) \\log \\frac{p(x^{(i)}, z^{(i)} ; \\theta)}{Q_i (z^{(i)})}\n", "$$ \n", "\n", "使用NBMM（朴素贝叶斯的混合模型）中的$\\phi_z,\\phi_{j|z^{(i)}=1},\\phi_{j|z^{(i)}=0}$参数替换一般化的EM算法中的$\\theta$参数，然后依次求解$w_j^{(i)}$与$\\phi_z,\\phi_{j|z^{(i)}=1},\\phi_{j|z^{(i)}=0}$参数的更新问题。 \n", "**NBMM的EM算法：** \n", "**E步骤：** \n", "$$w_j^{(i)}:=P\\left(z^{(i)}=1 | x^{(i)} ; \\phi_z, \\phi_{j | z^{(i)}=1}, \\phi_{j | z^{(i)}=0}\\right)$$**M步骤：**$$\n", "\\phi_{j | z^{(i)}=1} :=\\frac{\\displaystyle \\sum_{i=1}^{m} w^{(i)} I(x_{j}^{(i)}=1)}{\\displaystyle \\sum_{i=1}^{m} w^{(i)}} \\\\ \n", "\\phi_{j | z^{(i)}=0}:= \\frac{\\displaystyle \\sum_{i=1}^{m}\\left(1-w^{(i)}\\right) I(x_{j}^{(i)}=1)}{ \\displaystyle \\sum_{i=1}^{m}\\left(1-w^{(i)}\\right)} \\\\ \n", "\\phi_{z^{(i)}} :=\\frac{\\displaystyle \\sum_{i=1}^{m} w^{(i)}}{m} \n", "$$ " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "参考代码：https://github.com/wzyonggege/statistical-learning-method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第10章 隐马尔可夫模型/10.HMM.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第10章 隐马尔可夫模型" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1．隐马尔可夫模型是关于时序的概率模型，描述由一个隐藏的马尔可夫链随机生成不可观测的状态的序列，再由各个状态随机生成一个观测而产生观测的序列的过程。\n", "\n", "隐马尔可夫模型由初始状态概率向$\\pi$、状态转移概率矩阵$A$和观测概率矩阵$B$决定。因此，隐马尔可夫模型可以写成$\\lambda=(A, B, \\pi)$。\n", "\n", "隐马尔可夫模型是一个生成模型，表示状态序列和观测序列的联合分布，但是状态序列是隐藏的，不可观测的。\n", "\n", "隐马尔可夫模型可以用于标注，这时状态对应着标记。标注问题是给定观测序列预测其对应的标记序列。\n", "\n", "2．概率计算问题。给定模型$\\lambda=(A, B, \\pi)$和观测序列$O＝(o_1，o_2,…,o_T)$，计算在模型$\\lambda$下观测序列$O$出现的概率$P(O|\\lambda)$。前向-后向算法是通过递推地计算前向-后向概率可以高效地进行隐马尔可夫模型的概率计算。\n", " \n", "3．学习问题。已知观测序列$O＝(o_1，o_2,…,o_T)$，估计模型$\\lambda=(A, B, \\pi)$参数，使得在该模型下观测序列概率$P(O|\\lambda)$最大。即用极大似然估计的方法估计参数。Baum-Welch算法，也就是EM算法可以高效地对隐马尔可夫模型进行训练。它是一种非监督学习算法。\n", "\n", "4．预测问题。已知模型$\\lambda=(A, B, \\pi)$和观测序列$O＝(o_1，o_2,…,o_T)$，求对给定观测序列条件概率$P(I|O)$最大的状态序列$I＝(i_1，i_2,…,i_T)$。维特比算法应用动态规划高效地求解最优路径，即概率最大的状态序列。\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "class HiddenMarkov:\n", " def __init__(self):\n", " self.alphas = None\n", " self.forward_P = None\n", " self.betas = None\n", " self.backward_P = None\n", "\n", " # 前向算法\n", " def forward(self, Q, V, A, B, O, PI):\n", " # 状态序列的大小\n", " N = len(Q)\n", " # 观测序列的大小\n", " M = len(O)\n", " # 初始化前向概率alpha值\n", " alphas = np.zeros((N, M))\n", " # 时刻数=观测序列数\n", " T = M\n", " # 遍历每一个时刻，计算前向概率alpha值\n", " for t in range(T):\n", " # 得到序列对应的索引\n", " indexOfO = V.index(O[t])\n", " # 遍历状态序列\n", " for i in range(N):\n", " # 初始化alpha初值\n", " if t == 0:\n", " # P176 公式(10.15)\n", " alphas[i][t] = PI[t][i] * B[i][indexOfO]\n", " print('alpha1(%d) = p%db%db(o1) = %f' %\n", " (i + 1, i, i, alphas[i][t]))\n", " else:\n", " # P176 公式(10.16)\n", " alphas[i][t] = np.dot([alpha[t - 1] for alpha in alphas],\n", " [a[i] for a in A]) * B[i][indexOfO]\n", " print('alpha%d(%d) = [sigma alpha%d(i)ai%d]b%d(o%d) = %f' %\n", " (t + 1, i + 1, t - 1, i, i, t, alphas[i][t]))\n", " # P176 公式(10.17)\n", " self.forward_P = np.sum([alpha[M - 1] for alpha in alphas])\n", " self.alphas = alphas\n", "\n", " # 后向算法\n", " def backward(self, Q, V, A, B, O, PI):\n", " # 状态序列的大小\n", " N = len(Q)\n", " # 观测序列的大小\n", " M = len(O)\n", " # 初始化后向概率beta值，P178 公式(10.19)\n", " betas = np.ones((N, M))\n", " #\n", " for i in range(N):\n", " print('beta%d(%d) = 1' % (M, i + 1))\n", " # 对观测序列逆向遍历\n", " for t in range(M - 2, -1, -1):\n", " # 得到序列对应的索引\n", " indexOfO = V.index(O[t + 1])\n", " # 遍历状态序列\n", " for i in range(N):\n", " # P178 公式(10.20)\n", " betas[i][t] = np.dot(\n", " np.multiply(A[i], [b[indexOfO] for b in B]),\n", " [beta[t + 1] for beta in betas])\n", " realT = t + 1\n", " realI = i + 1\n", " print('beta%d(%d) = sigma[a%djbj(o%d)beta%d(j)] = (' %\n", " (realT, realI, realI, realT + 1, realT + 1),\n", " end='')\n", " for j in range(N):\n", " print(\"%.2f * %.2f * %.2f + \" %\n", " (A[i][j], B[j][indexOfO], betas[j][t + 1]),\n", " end='')\n", " print(\"0) = %.3f\" % betas[i][t])\n", " # 取出第一个值\n", " indexOfO = V.index(O[0])\n", " self.betas = betas\n", " # P178 公式(10.21)\n", " P = np.dot(np.multiply(PI, [b[indexOfO] for b in B]),\n", " [beta[0] for beta in betas])\n", " self.backward_P = P\n", " print(\"P(O|lambda) = \", end=\"\")\n", " for i in range(N):\n", " print(\"%.1f * %.1f * %.5f + \" %\n", " (PI[0][i], B[i][indexOfO], betas[i][0]),\n", " end=\"\")\n", " print(\"0 = %f\" % P)\n", "\n", " # 维特比算法\n", " def viterbi(self, Q, V, A, B, O, PI):\n", " # 状态序列的大小\n", " N = len(Q)\n", " # 观测序列的大小\n", " M = len(O)\n", " # 初始化daltas\n", " deltas = np.zeros((N, M))\n", " # 初始化psis\n", " psis = np.zeros((N, M))\n", " # 初始化最优路径矩阵，该矩阵维度与观测序列维度相同\n", " I = np.zeros((1, M))\n", " # 遍历观测序列\n", " for t in range(M):\n", " # 递推从t=2开始\n", " realT = t + 1\n", " # 得到序列对应的索引\n", " indexOfO = V.index(O[t])\n", " for i in range(N):\n", " realI = i + 1\n", " if t == 0:\n", " # P185 算法10.5 步骤(1)\n", " deltas[i][t] = PI[0][i] * B[i][indexOfO]\n", " psis[i][t] = 0\n", " print('delta1(%d) = pi%d * b%d(o1) = %.2f * %.2f = %.2f' %\n", " (realI, realI, realI, PI[0][i], B[i][indexOfO],\n", " deltas[i][t]))\n", " print('psis1(%d) = 0' % (realI))\n", " else:\n", " # # P185 算法10.5 步骤(2)\n", " deltas[i][t] = np.max(\n", " np.multiply([delta[t - 1] for delta in deltas],\n", " [a[i] for a in A])) * B[i][indexOfO]\n", " print(\n", " 'delta%d(%d) = max[delta%d(j)aj%d]b%d(o%d) = %.2f * %.2f = %.5f'\n", " % (realT, realI, realT - 1, realI, realI, realT,\n", " np.max(\n", " np.multiply([delta[t - 1] for delta in deltas],\n", " [a[i] for a in A])), B[i][indexOfO],\n", " deltas[i][t]))\n", " psis[i][t] = np.argmax(\n", " np.multiply([delta[t - 1] for delta in deltas],\n", " [a[i] for a in A]))\n", " print('psis%d(%d) = argmax[delta%d(j)aj%d] = %d' %\n", " (realT, realI, realT - 1, realI, psis[i][t]))\n", " #print(deltas)\n", " #print(psis)\n", " # 得到最优路径的终结点\n", " I[0][M - 1] = np.argmax([delta[M - 1] for delta in deltas])\n", " print('i%d = argmax[deltaT(i)] = %d' % (M, I[0][M - 1] + 1))\n", " # 递归由后向前得到其他结点\n", " for t in range(M - 2, -1, -1):\n", " I[0][t] = psis[int(I[0][t + 1])][t + 1]\n", " print('i%d = psis%d(i%d) = %d' %\n", " (t + 1, t + 2, t + 2, I[0][t] + 1))\n", " # 输出最优路径\n", " print('最优路径是：', \"->\".join([str(int(i + 1)) for i in I[0]]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题10.1\n", "  给定盒子和球组成的隐马尔可夫模型$\\lambda=(A,B,\\pi)$，其中，$$A=\\left[\\begin{array}{ccc}0.5&0.2&0.3\\\\0.3&0.5&0.2\\\\0.2&0.3&0.5\\end{array}\\right], \\quad B=\\left[\\begin{array}{cc}0.5&0.5\\\\0.4&0.6\\\\0.7&0.3\\end{array}\\right], \\quad \\pi=(0.2,0.4,0.4)^T$$设$T=4,O=(红,白,红,白)$，试用后向算法计算$P(O|\\lambda)$。\n", "\n", "**解答：**" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "#习题10.1\n", "Q = [1, 2, 3]\n", "V = ['红', '白']\n", "A = [[0.5, 0.2, 0.3], [0.3, 0.5, 0.2], [0.2, 0.3, 0.5]]\n", "B = [[0.5, 0.5], [0.4, 0.6], [0.7, 0.3]]\n", "# O = ['红', '白', '红', '红', '白', '红', '白', '白']\n", "O = ['红', '白', '红', '白'] #习题10.1的例子\n", "PI = [[0.2, 0.4, 0.4]]" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "delta1(1) = pi1 * b1(o1) = 0.20 * 0.50 = 0.10\n", "psis1(1) = 0\n", "delta1(2) = pi2 * b2(o1) = 0.40 * 0.40 = 0.16\n", "psis1(2) = 0\n", "delta1(3) = pi3 * b3(o1) = 0.40 * 0.70 = 0.28\n", "psis1(3) = 0\n", "delta2(1) = max[delta1(j)aj1]b1(o2) = 0.06 * 0.50 = 0.02800\n", "psis2(1) = argmax[delta1(j)aj1] = 2\n", "delta2(2) = max[delta1(j)aj2]b2(o2) = 0.08 * 0.60 = 0.05040\n", "psis2(2) = argmax[delta1(j)aj2] = 2\n", "delta2(3) = max[delta1(j)aj3]b3(o2) = 0.14 * 0.30 = 0.04200\n", "psis2(3) = argmax[delta1(j)aj3] = 2\n", "delta3(1) = max[delta2(j)aj1]b1(o3) = 0.02 * 0.50 = 0.00756\n", "psis3(1) = argmax[delta2(j)aj1] = 1\n", "delta3(2) = max[delta2(j)aj2]b2(o3) = 0.03 * 0.40 = 0.01008\n", "psis3(2) = argmax[delta2(j)aj2] = 1\n", "delta3(3) = max[delta2(j)aj3]b3(o3) = 0.02 * 0.70 = 0.01470\n", "psis3(3) = argmax[delta2(j)aj3] = 2\n", "delta4(1) = max[delta3(j)aj1]b1(o4) = 0.00 * 0.50 = 0.00189\n", "psis4(1) = argmax[delta3(j)aj1] = 0\n", "delta4(2) = max[delta3(j)aj2]b2(o4) = 0.01 * 0.60 = 0.00302\n", "psis4(2) = argmax[delta3(j)aj2] = 1\n", "delta4(3) = max[delta3(j)aj3]b3(o4) = 0.01 * 0.30 = 0.00220\n", "psis4(3) = argmax[delta3(j)aj3] = 2\n", "i4 = argmax[deltaT(i)] = 2\n", "i3 = psis4(i4) = 2\n", "i2 = psis3(i3) = 2\n", "i1 = psis2(i2) = 3\n", "最优路径是： 3->2->2->2\n" ] } ], "source": [ "HMM = HiddenMarkov()\n", "# HMM.forward(Q, V, A, B, O, PI)\n", "# HMM.backward(Q, V, A, B, O, PI)\n", "HMM.viterbi(Q, V, A, B, O, PI)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题10.2\n", "\n", "  给定盒子和球组成的隐马尔可夫模型$\\lambda=(A,B,\\pi)$，其中，$$A=\\left[\\begin{array}{ccc}0.5&0.1&0.4\\\\0.3&0.5&0.2\\\\0.2&0.2&0.6\\end{array}\\right], \\quad B=\\left[\\begin{array}{cc}0.5&0.5\\\\0.4&0.6\\\\0.7&0.3\\end{array}\\right], \\quad \\pi=(0.2,0.3,0.5)^T$$设$T=8,O=(红,白,红,红,白,红,白,白)$，试用前向后向概率计算$P(i_4=q_3|O,\\lambda)$\n", "\n", "**解答：**" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "Q = [1, 2, 3]\n", "V = ['红', '白']\n", "A = [[0.5, 0.2, 0.3], [0.3, 0.5, 0.2], [0.2, 0.3, 0.5]]\n", "B = [[0.5, 0.5], [0.4, 0.6], [0.7, 0.3]]\n", "O = ['红', '白', '红', '红', '白', '红', '白', '白']\n", "PI = [[0.2, 0.3, 0.5]]" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "alpha1(1) = p0b0b(o1) = 0.100000\n", "alpha1(2) = p1b1b(o1) = 0.120000\n", "alpha1(3) = p2b2b(o1) = 0.350000\n", "alpha2(1) = [sigma alpha0(i)ai0]b0(o1) = 0.078000\n", "alpha2(2) = [sigma alpha0(i)ai1]b1(o1) = 0.111000\n", "alpha2(3) = [sigma alpha0(i)ai2]b2(o1) = 0.068700\n", "alpha3(1) = [sigma alpha1(i)ai0]b0(o2) = 0.043020\n", "alpha3(2) = [sigma alpha1(i)ai1]b1(o2) = 0.036684\n", "alpha3(3) = [sigma alpha1(i)ai2]b2(o2) = 0.055965\n", "alpha4(1) = [sigma alpha2(i)ai0]b0(o3) = 0.021854\n", "alpha4(2) = [sigma alpha2(i)ai1]b1(o3) = 0.017494\n", "alpha4(3) = [sigma alpha2(i)ai2]b2(o3) = 0.033758\n", "alpha5(1) = [sigma alpha3(i)ai0]b0(o4) = 0.011463\n", "alpha5(2) = [sigma alpha3(i)ai1]b1(o4) = 0.013947\n", "alpha5(3) = [sigma alpha3(i)ai2]b2(o4) = 0.008080\n", "alpha6(1) = [sigma alpha4(i)ai0]b0(o5) = 0.005766\n", "alpha6(2) = [sigma alpha4(i)ai1]b1(o5) = 0.004676\n", "alpha6(3) = [sigma alpha4(i)ai2]b2(o5) = 0.007188\n", "alpha7(1) = [sigma alpha5(i)ai0]b0(o6) = 0.002862\n", "alpha7(2) = [sigma alpha5(i)ai1]b1(o6) = 0.003389\n", "alpha7(3) = [sigma alpha5(i)ai2]b2(o6) = 0.001878\n", "alpha8(1) = [sigma alpha6(i)ai0]b0(o7) = 0.001411\n", "alpha8(2) = [sigma alpha6(i)ai1]b1(o7) = 0.001698\n", "alpha8(3) = [sigma alpha6(i)ai2]b2(o7) = 0.000743\n", "beta8(1) = 1\n", "beta8(2) = 1\n", "beta8(3) = 1\n", "beta7(1) = sigma[a1jbj(o8)beta8(j)] = (0.50 * 0.50 * 1.00 + 0.20 * 0.60 * 1.00 + 0.30 * 0.30 * 1.00 + 0) = 0.460\n", "beta7(2) = sigma[a2jbj(o8)beta8(j)] = (0.30 * 0.50 * 1.00 + 0.50 * 0.60 * 1.00 + 0.20 * 0.30 * 1.00 + 0) = 0.510\n", "beta7(3) = sigma[a3jbj(o8)beta8(j)] = (0.20 * 0.50 * 1.00 + 0.30 * 0.60 * 1.00 + 0.50 * 0.30 * 1.00 + 0) = 0.430\n", "beta6(1) = sigma[a1jbj(o7)beta7(j)] = (0.50 * 0.50 * 0.46 + 0.20 * 0.60 * 0.51 + 0.30 * 0.30 * 0.43 + 0) = 0.215\n", "beta6(2) = sigma[a2jbj(o7)beta7(j)] = (0.30 * 0.50 * 0.46 + 0.50 * 0.60 * 0.51 + 0.20 * 0.30 * 0.43 + 0) = 0.248\n", "beta6(3) = sigma[a3jbj(o7)beta7(j)] = (0.20 * 0.50 * 0.46 + 0.30 * 0.60 * 0.51 + 0.50 * 0.30 * 0.43 + 0) = 0.202\n", "beta5(1) = sigma[a1jbj(o6)beta6(j)] = (0.50 * 0.50 * 0.21 + 0.20 * 0.40 * 0.25 + 0.30 * 0.70 * 0.20 + 0) = 0.116\n", "beta5(2) = sigma[a2jbj(o6)beta6(j)] = (0.30 * 0.50 * 0.21 + 0.50 * 0.40 * 0.25 + 0.20 * 0.70 * 0.20 + 0) = 0.110\n", "beta5(3) = sigma[a3jbj(o6)beta6(j)] = (0.20 * 0.50 * 0.21 + 0.30 * 0.40 * 0.25 + 0.50 * 0.70 * 0.20 + 0) = 0.122\n", "beta4(1) = sigma[a1jbj(o5)beta5(j)] = (0.50 * 0.50 * 0.12 + 0.20 * 0.60 * 0.11 + 0.30 * 0.30 * 0.12 + 0) = 0.053\n", "beta4(2) = sigma[a2jbj(o5)beta5(j)] = (0.30 * 0.50 * 0.12 + 0.50 * 0.60 * 0.11 + 0.20 * 0.30 * 0.12 + 0) = 0.058\n", "beta4(3) = sigma[a3jbj(o5)beta5(j)] = (0.20 * 0.50 * 0.12 + 0.30 * 0.60 * 0.11 + 0.50 * 0.30 * 0.12 + 0) = 0.050\n", "beta3(1) = sigma[a1jbj(o4)beta4(j)] = (0.50 * 0.50 * 0.05 + 0.20 * 0.40 * 0.06 + 0.30 * 0.70 * 0.05 + 0) = 0.028\n", "beta3(2) = sigma[a2jbj(o4)beta4(j)] = (0.30 * 0.50 * 0.05 + 0.50 * 0.40 * 0.06 + 0.20 * 0.70 * 0.05 + 0) = 0.026\n", "beta3(3) = sigma[a3jbj(o4)beta4(j)] = (0.20 * 0.50 * 0.05 + 0.30 * 0.40 * 0.06 + 0.50 * 0.70 * 0.05 + 0) = 0.030\n", "beta2(1) = sigma[a1jbj(o3)beta3(j)] = (0.50 * 0.50 * 0.03 + 0.20 * 0.40 * 0.03 + 0.30 * 0.70 * 0.03 + 0) = 0.015\n", "beta2(2) = sigma[a2jbj(o3)beta3(j)] = (0.30 * 0.50 * 0.03 + 0.50 * 0.40 * 0.03 + 0.20 * 0.70 * 0.03 + 0) = 0.014\n", "beta2(3) = sigma[a3jbj(o3)beta3(j)] = (0.20 * 0.50 * 0.03 + 0.30 * 0.40 * 0.03 + 0.50 * 0.70 * 0.03 + 0) = 0.016\n", "beta1(1) = sigma[a1jbj(o2)beta2(j)] = (0.50 * 0.50 * 0.02 + 0.20 * 0.60 * 0.01 + 0.30 * 0.30 * 0.02 + 0) = 0.007\n", "beta1(2) = sigma[a2jbj(o2)beta2(j)] = (0.30 * 0.50 * 0.02 + 0.50 * 0.60 * 0.01 + 0.20 * 0.30 * 0.02 + 0) = 0.007\n", "beta1(3) = sigma[a3jbj(o2)beta2(j)] = (0.20 * 0.50 * 0.02 + 0.30 * 0.60 * 0.01 + 0.50 * 0.30 * 0.02 + 0) = 0.006\n", "P(O|lambda) = 0.2 * 0.5 * 0.00698 + 0.3 * 0.4 * 0.00741 + 0.5 * 0.7 * 0.00647 + 0 = 0.003852\n" ] } ], "source": [ "HMM.forward(Q, V, A, B, O, PI)\n", "HMM.backward(Q, V, A, B, O, PI)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "可知，$\\displaystyle P(i_4=q_3|O,\\lambda)=\\frac{P(i_4=q_3,O|\\lambda)}{P(O|\\lambda)}=\\frac{\\alpha_4(3)\\beta_4(3)}{P(O|\\lambda)}$" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "alpha4(3)= 0.033757709999999996\n", "beta4(3)= 0.049728909999999994\n", "P(O|lambda)= 0.0038519735794910986\n", "P(i4=q3|O,lambda) = 0.4358114321796269\n" ] } ], "source": [ "print(\"alpha4(3)=\", HMM.alphas[3 - 1][4 - 1])\n", "print(\"beta4(3)=\", HMM.betas[3 - 1][4 - 1])\n", "print(\"P(O|lambda)=\", HMM.backward_P[0])\n", "result = (HMM.alphas[3 - 1][4 - 1] *\n", " HMM.betas[3 - 1][4 - 1]) / HMM.backward_P[0]\n", "print(\"P(i4=q3|O,lambda) =\", result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题10.3\n", "\n", "  在习题10.1中，试用维特比算法求最优路径$I^*=(i_1^*,i_2^*,i_3^*,i_4^*)$。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "delta1(1) = pi1 * b1(o1) = 0.20 * 0.50 = 0.10\n", "psis1(1) = 0\n", "delta1(2) = pi2 * b2(o1) = 0.40 * 0.40 = 0.16\n", "psis1(2) = 0\n", "delta1(3) = pi3 * b3(o1) = 0.40 * 0.70 = 0.28\n", "psis1(3) = 0\n", "delta2(1) = max[delta1(j)aj1]b1(o2) = 0.06 * 0.50 = 0.02800\n", "psis2(1) = argmax[delta1(j)aj1] = 2\n", "delta2(2) = max[delta1(j)aj2]b2(o2) = 0.08 * 0.60 = 0.05040\n", "psis2(2) = argmax[delta1(j)aj2] = 2\n", "delta2(3) = max[delta1(j)aj3]b3(o2) = 0.14 * 0.30 = 0.04200\n", "psis2(3) = argmax[delta1(j)aj3] = 2\n", "delta3(1) = max[delta2(j)aj1]b1(o3) = 0.02 * 0.50 = 0.00756\n", "psis3(1) = argmax[delta2(j)aj1] = 1\n", "delta3(2) = max[delta2(j)aj2]b2(o3) = 0.03 * 0.40 = 0.01008\n", "psis3(2) = argmax[delta2(j)aj2] = 1\n", "delta3(3) = max[delta2(j)aj3]b3(o3) = 0.02 * 0.70 = 0.01470\n", "psis3(3) = argmax[delta2(j)aj3] = 2\n", "delta4(1) = max[delta3(j)aj1]b1(o4) = 0.00 * 0.50 = 0.00189\n", "psis4(1) = argmax[delta3(j)aj1] = 0\n", "delta4(2) = max[delta3(j)aj2]b2(o4) = 0.01 * 0.60 = 0.00302\n", "psis4(2) = argmax[delta3(j)aj2] = 1\n", "delta4(3) = max[delta3(j)aj3]b3(o4) = 0.01 * 0.30 = 0.00220\n", "psis4(3) = argmax[delta3(j)aj3] = 2\n", "i4 = argmax[deltaT(i)] = 2\n", "i3 = psis4(i4) = 2\n", "i2 = psis3(i3) = 2\n", "i1 = psis2(i2) = 3\n", "最优路径是： 3->2->2->2\n" ] } ], "source": [ "Q = [1, 2, 3]\n", "V = ['红', '白']\n", "A = [[0.5, 0.2, 0.3], [0.3, 0.5, 0.2], [0.2, 0.3, 0.5]]\n", "B = [[0.5, 0.5], [0.4, 0.6], [0.7, 0.3]]\n", "O = ['红', '白', '红', '白']\n", "PI = [[0.2, 0.4, 0.4]]\n", "\n", "HMM = HiddenMarkov()\n", "HMM.viterbi(Q, V, A, B, O, PI)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题10.4\n", "  试用前向概率和后向概率推导$$P(O|\\lambda)=\\sum_{i=1}^N\\sum_{j=1}^N\\alpha_t(i)a_{ij}b_j(o_{t+1})\\beta_{t+1}(j),\\quad t=1,2,\\cdots,T-1$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "$$\\begin{aligned}\n", "P(O|\\lambda)\n", "&= P(o_1,o_2,...,o_T|\\lambda) \\\\\n", "&= \\sum_{i=1}^N P(o_1,..,o_t,i_t=q_i|\\lambda) P(o_{t+1},..,o_T|i_t=q_i,\\lambda) \\\\\n", "&= \\sum_{i=1}^N \\sum_{j=1}^N P(o_1,..,o_t,i_t=q_i|\\lambda) P(o_{t+1},i_{t+1}=q_j|i_t=q_i,\\lambda)P(o_{t+2},..,o_T|i_{t+1}=q_j,\\lambda) \\\\\n", "&= \\sum_{i=1}^N \\sum_{j=1}^N [P(o_1,..,o_t,i_t=q_i|\\lambda) P(o_{t+1}|i_{t+1}=q_j,\\lambda) P(i_{t+1}=q_j|i_t=q_i,\\lambda) \\\\\n", "& \\quad \\quad \\quad \\quad P(o_{t+2},..,o_T|i_{t+1}=q_j,\\lambda)] \\\\\n", "&= \\sum_{i=1}^N \\sum_{j=1}^N \\alpha_t(i) a_{ij} b_j(o_{t+1}) \\beta_{t+1}(j),{\\quad}t=1,2,...,T-1\n", "\\end{aligned}$$\n", "命题得证。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题10.5\n", "\n", "  比较维特比算法中变量$\\delta$的计算和前向算法中变量$\\alpha$的计算的主要区别。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "**前向算法：** \n", "1. 初值$$\\alpha_1(i)=\\pi_ib_i(o_i),i=1,2,\\cdots,N$$\n", "2. 递推，对$t=1,2,\\cdots,T-1$：$$\\alpha_{t+1}(i)=\\left[\\sum_{j=1}^N \\alpha_t(j) a_{ji} \\right]b_i(o_{t+1})，i=1,2,\\cdots,N$$\n", "\n", "**维特比算法：** \n", "1. 初始化$$\\delta_1(i)=\\pi_ib_i(o_1),i=1,2,\\cdots,N$$\n", "2. 递推，对$t=2,3,\\cdots,T$$$\\delta_t(i)=\\max_{1 \\leqslant j \\leqslant N} [\\delta_{t-1}(j)a_{ji}]b_i(o_t), i=1,2,\\cdots,N$$ \n", "\n", "  由上面算法可知，计算变量$\\alpha$的时候直接对上个的结果进行数值计算，而计算变量$\\delta$需要在上个结果计算的基础上选择最大值" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "参考代码：https://blog.csdn.net/tudaodiaozhale\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第11章 条件随机场/11.CRF.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第11章 条件随机场\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1．概率无向图模型是由无向图表示的联合概率分布。无向图上的结点之间的连接关系表示了联合分布的随机变量集合之间的条件独立性，即马尔可夫性。因此，概率无向图模型也称为马尔可夫随机场。\n", "\n", "概率无向图模型或马尔可夫随机场的联合概率分布可以分解为无向图最大团上的正值函数的乘积的形式。\n", "\n", "2．条件随机场是给定输入随机变量$X$条件下，输出随机变量$Y$的条件概率分布模型， 其形式为参数化的对数线性模型。条件随机场的最大特点是假设输出变量之间的联合概率分布构成概率无向图模型，即马尔可夫随机场。条件随机场是判别模型。\n", "\n", "3．线性链条件随机场是定义在观测序列与标记序列上的条件随机场。线性链条件随机场一般表示为给定观测序列条件下的标记序列的条件概率分布，由参数化的对数线性模型表示。模型包含特征及相应的权值，特征是定义在线性链的边与结点上的。线性链条件随机场的数学表达式是\n", "$$\n", "P(y | x)=\\frac{1}{Z(x)} \\exp \\left(\\sum_{i, k} \\lambda_{k} t_{k}\\left(y_{i-1}, y_{i}, x, i\\right)+\\sum_{i, l} \\mu_{l} s_{l}\\left(y_{i}, x, i\\right)\\right)\n", "$$\n", "\n", "其中，\n", " $$\n", "Z(x)=\\sum_{y} \\exp \\left(\\sum_{i, k} \\lambda_{k} t_{k}\\left(y_{i-1}, y_{i}, x, i\\right)+\\sum_{i, l} \\mu_{l} s_{l}\\left(y_{i}, x, i\\right)\\right)\n", "$$\n", "\n", "4．线性链条件随机场的概率计算通常利用前向-后向算法。\n", "\n", "5．条件随机场的学习方法通常是极大似然估计方法或正则化的极大似然估计，即在给定训练数据下，通过极大化训练数据的对数似然函数以估计模型参数。具体的算法有改进的迭代尺度算法、梯度下降法、拟牛顿法等。\n", "\n", "6．线性链条件随机场的一个重要应用是标注。维特比算法是给定观测序列求条件概率最大的标记序列的方法。\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 例11.1" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from numpy import *" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "24.532530197109345\n", "24.532530197109352\n" ] } ], "source": [ "#这里定义T为转移矩阵列代表前一个y(ij)代表由状态i转到状态j的概率,Tx矩阵x对应于时间序列\n", "#这里将书上的转移特征转换为如下以时间轴为区别的三个多维列表，维度为输出的维度\n", "T1 = [[0.6, 1], [1, 0]]\n", "T2 = [[0, 1], [1, 0.2]]\n", "#将书上的状态特征同样转换成列表,第一个是为y1的未规划概率，第二个为y2的未规划概率\n", "S0 = [1, 0.5]\n", "S1 = [0.8, 0.5]\n", "S2 = [0.8, 0.5]\n", "Y = [1, 2, 2] #即书上例一需要计算的非规划条件概率的标记序列\n", "Y = array(Y) - 1 #这里为了将数与索引相对应即从零开始\n", "P = exp(S0[Y[0]])\n", "for i in range(1, len(Y)):\n", " P *= exp((eval('S%d' % i)[Y[i]]) + eval('T%d' % i)[Y[i - 1]][Y[i]])\n", "print(P)\n", "print(exp(3.2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 例11.2" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "非规范化概率 24.532530197109345\n" ] } ], "source": [ "#这里根据例11.2的启发整合为一个矩阵\n", "F0 = S0\n", "F1 = T1 + array(S1 * len(T1)).reshape(shape(T1))\n", "F2 = T2 + array(S2 * len(T2)).reshape(shape(T2))\n", "Y = [1, 2, 2] #即书上例一需要计算的非规划条件概率的标记序列\n", "Y = array(Y) - 1\n", "\n", "P = exp(F0[Y[0]])\n", "Sum = P\n", "for i in range(1, len(Y)):\n", " PIter = exp((eval('F%d' % i)[Y[i - 1]][Y[i]]))\n", " P *= PIter\n", " Sum += PIter\n", "print('非规范化概率', P)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 第11章条件随机场-习题\n", "\n", "### 习题11.1\n", "  写出图11.3中无向图描述的概率图模型的因子分解式。\n", "
\n", "
图11.3 无向图的团和最大团
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "  图11.3表示由4个结点组成的无向图。图中由2个结点组成的团有5个：$\\{Y_1,Y_2\\},\\{Y_2,Y_3\\},\\{Y_3,Y_4\\},\\{Y_4,Y_2\\}$和$\\{Y_1,Y_3\\}$，有2个最大团：$\\{Y_1,Y_2,Y_3\\}$和$\\{Y_2,Y_3,Y_4\\}$，而$\\{Y_1,Y_2,Y_3,Y_4\\}$不是一个团，因为$Y_1$和$Y_4$没有边连接。 \n", "  根据概率图模型的因子分解定义：将概率无向图模型的联合概率分布表示为其最大团上的随机变量的函数的乘积形式的操作。公式在书中(11.5)，(11.6)。 \n", "$$P(Y)=\\frac{\\Psi_{(1,2,3)}(Y_{(1,2,3)})\\cdot\\Psi_{(2,3,4)}(Y_{(2,3,4)})}{\\displaystyle \\sum_Y \\left[ \\Psi_{(1,2,3)}(Y_{(1,2,3)})\\cdot\\Psi_{(2,3,4)}(Y_{(2,3,4)})\\right]}$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题11.2\n", "\n", "  证明$Z(x)=a_n^T(x) \\cdot \\boldsymbol{1} = \\boldsymbol{1}^T\\cdot\\beta_1(x)$，其中$\\boldsymbol{1}$是元素均为1的$m$维列向量。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "**第1步：**证明$Z(x)=a_n^T(x) \\cdot \\boldsymbol{1}$ \n", "根据条件随机场的矩阵形式：$$(M_{n+1}(x))_{i,j}=\\begin{cases}\n", "1,&j=\\text{stop}\\\\\n", "0,&\\text{otherwise}\n", "\\end{cases}$$根据前向向量的定义：$$\\alpha_0(y|x)=\\begin{cases}\n", "1,&y=\\text{start} \\\\\n", "0,&\\text{otherwise}\n", "\\end{cases}$$ \n", "$\\begin{aligned}\n", "\\therefore Z_n(x) \n", "&= \\left(M_1(x)M_2(x){\\cdots}M_{n+1}(x)\\right)_{(\\text{start},\\text{stop})} \\\\\n", "&= \\alpha_0(x)^T M_1(x)M_2(x){\\cdots}M_n(x) \\cdot 1\\\\\n", "&=\\alpha_n(x)^T\\cdot \\boldsymbol{1}\n", "\\end{aligned}$ \n", "\n", "-----\n", "\n", "**第2步：**证明$Z(x)=\\boldsymbol{1}^T \\cdot \\beta_1(x)$ \n", "根据条件随机场的矩阵形式：$$(M_{n+1}(x))_{i,j}=\\begin{cases}\n", "1,&j=\\text{stop}\\\\\n", "0,&\\text{otherwise}\n", "\\end{cases}$$根据后向向量定义：$$\\beta_{n+1}(y_{n+1}|x)=\n", "\\begin{cases}\n", "1,& y_{n+1}=\\text{stop} \\\\\n", "0,& \\text{otherwise}\n", "\\end{cases}$$ \n", "$\\begin{aligned}\n", "\\therefore Z_n(x)\n", "&= (M_1(x)M_2(x) \\cdots M_{n+1}(x))_{(\\text{start},\\text{stop})} \\\\\n", "&= (M_1(x)M_2(x) \\cdots M_n(x) \\beta_{n+1}(x))_{\\text{start}} \\\\\n", "&=(\\beta_1(x))_{\\text{start}} \\\\\n", "&=\\boldsymbol{1}^T \\cdot \\beta_1(x)\n", "\\end{aligned}$ \n", "综上所述：$Z(x)=a_n^T(x) \\cdot \\boldsymbol{1} = \\boldsymbol{1}^T \\cdot \\beta_1(x)$，命题得证。 " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题11.3\n", "  写出条件随机场模型学习的梯度下降法。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解答：** \n", "条件随机场的对数极大似然函数为：$$L(w)=\\sum^N_{j=1} \\sum^K_{k=1} w_k f_k(y_j,x_j)-\\sum^N_{j=1} \\log{Z_w(x_j)}$$梯度下降算法的目标函数是$f(w)=-L(w)$ \n", "目标函数的梯度为：$$g(w)=\\nabla{f(w^{(k)})}=\\left(\\frac{\\partial{f(w)}}{\\partial{w_1}},\\frac{\\partial{f(w)}}{\\partial{w_2}},\\cdots,\\frac{\\partial{f(w)}}{\\partial{w_k}}\\right)$$其中$$\\begin{aligned}\n", "\\frac{\\partial{f(w)}}{\\partial{w_i}}\n", "&= -\\sum^N_{j=1} w_i f_i(y_j,x_j) + \\sum^N_{j=1} \\frac{1}{Z_w(x_j)} \\cdot \\frac{\\partial{Z_w(x_j)}}{\\partial{w_i}}\\\\\n", "&= -\\sum^N_{j=1}w_if_i(y_j,x_j)+\\sum^N_{j=1}\\frac{1}{Z_w(x_j)}\\sum_y(\\exp{\\sum^K_{k=1}w_kf_k(y,x_j))}w_if_i(y,x_j)\n", "\\end{aligned}$$ \n", "根据梯度下降算法： \n", "1. 取初始值$w^{(0)} \\in \\mathbf{R}^n$，置$k=0$ \n", "2. 计算$f(w^{(k)})$ \n", "3. 计算梯度$g_k=g(w^{(k)})$，当$\\|g_k\\|<\\varepsilon$时，停止迭代，令$w^*=w^{(k)}$；否则令$p_k=-g(w^{(k)})$，求$\\lambda_k$，使$$\n", "f(w^{(k)}+\\lambda_k p_k)=\\min_{\\lambda \\geqslant 0}{f(w^{(k)}+\\lambda p_k)}$$ \n", "4. 置$w^{(k+1)}=w^{(k)}+\\lambda_k p_k$，计算$f(w^{(k+1)})$ \n", "当$\\|f(w^{(k+1)})-f(w^{(k)})\\| < \\epsilon$或$\\|w^{(k+1)}-w^{(k)}\\| < \\epsilon$时，停止迭代，令$w^*=w^{(k+1)}$ \n", "5. 否则，置$k=k+1$，转(3)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 习题11.4\n", "\n", "参考图11.6的状态路径图，假设随机矩阵$M_1(x),M_2(x),M_3(x),M_4(x)$分别是\n", "$$M_1(x)=\\begin{bmatrix}0&0\\\\0.5&0.5\\end{bmatrix} ,\n", "M_2(x)=\\begin{bmatrix}0.3&0.7\\\\0.7&0.3\\end{bmatrix}$$\n", "$$\n", "M_3(x)=\\begin{bmatrix}0.5&0.5\\\\0.6&0.4\\end{bmatrix},\n", "M_4(x)=\\begin{bmatrix}0&1\\\\0&1\\end{bmatrix}$$\n", "求以$start=2$为起点$stop=2$为终点的所有路径的状态序列$y$的概率及概率最大的状态序列。\n", "\n", "**解答：**" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[[0, 0], [0.5, 0.5]], [[0.3, 0.7], [0.7, 0.3]], [[0.5, 0.5], [0.6, 0.4]], [[0, 1], [0, 1]]]\n" ] } ], "source": [ "import numpy as np\n", "\n", "# 创建随机矩阵\n", "M1 = [[0, 0], [0.5, 0.5]]\n", "M2 = [[0.3, 0.7], [0.7, 0.3]]\n", "M3 = [[0.5, 0.5], [0.6, 0.4]]\n", "M4 = [[0, 1], [0, 1]]\n", "M = [M1, M2, M3, M4]\n", "print(M)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[array([2, 1, 1, 1, 2]), array([2, 1, 1, 2, 2]), array([2, 1, 2, 1, 2]), array([2, 1, 2, 2, 2]), array([2, 2, 1, 1, 2]), array([2, 2, 1, 2, 2]), array([2, 2, 2, 1, 2]), array([2, 2, 2, 2, 2])]\n" ] } ], "source": [ "# 生成路径\n", "path = [2]\n", "for i in range(1, 4):\n", " paths = []\n", " for _, r in enumerate(path):\n", " temp = np.transpose(r)\n", " paths.append(np.append(temp, 1))\n", " paths.append(np.append(temp, 2))\n", " path = paths.copy()\n", "\n", "path = [np.append(r, 2) for _, r in enumerate(path)]\n", "print(path)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[([2, 1, 2, 1, 2], 0.21), ([2, 2, 1, 1, 2], 0.175), ([2, 2, 1, 2, 2], 0.175), ([2, 1, 2, 2, 2], 0.13999999999999999), ([2, 2, 2, 1, 2], 0.09), ([2, 1, 1, 1, 2], 0.075), ([2, 1, 1, 2, 2], 0.075), ([2, 2, 2, 2, 2], 0.06)]\n" ] } ], "source": [ "# 计算概率\n", "\n", "pr = []\n", "for _, row in enumerate(path):\n", " p = 1\n", " for i in range(len(row) - 1):\n", " a = row[i]\n", " b = row[i + 1]\n", " p *= M[i][a - 1][b - 1]\n", " pr.append((row.tolist(), p))\n", "pr = sorted(pr, key=lambda x: x[1], reverse=True)\n", "print(pr)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "以start=2为起点stop=2为终点的所有路径的状态序列y的概率为：\n", " 路径为：2->1->2->1->2 概率为：0.21\n", " 路径为：2->2->1->1->2 概率为：0.175\n", " 路径为：2->2->1->2->2 概率为：0.175\n", " 路径为：2->1->2->2->2 概率为：0.13999999999999999\n", " 路径为：2->2->2->1->2 概率为：0.09\n", " 路径为：2->1->1->1->2 概率为：0.075\n", " 路径为：2->1->1->2->2 概率为：0.075\n", " 路径为：2->2->2->2->2 概率为：0.06\n", "概率[0.21]最大的状态序列为: 2->1->2->1->2\n" ] } ], "source": [ "# 打印结果\n", "print(\"以start=2为起点stop=2为终点的所有路径的状态序列y的概率为：\")\n", "for path, p in pr:\n", " print(\" 路径为：\" + \"->\".join([str(x) for x in path]), end=\" \")\n", " print(\"概率为：\" + str(p))\n", "print(\"概率[\" + str(pr[0][1]) + \"]最大的状态序列为:\",\n", " \"->\".join([str(x) for x in pr[0][0]]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "参考代码：https://blog.csdn.net/GrinAndBearIt/article/details/79229803\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "习题解答：https://github.com/datawhalechina/statistical-learning-method-solutions-manual\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第12章 监督学习方法总结/12.Summary_of_Supervised_Learning_Methods.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第12章 监督学习方法总结\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1 适用问题\n", "\n", "监督学习可以认为是学习一个模型，使它能对给定的输入预测相应的输出。监督学习包括分类、标注、回归。本篇主要考虑前两者的学习方法。\n", "\n", "分类问题是从实例的特征向量到类标记的预测问题；标注问题是从观测序列到标记序列(或状态序列)的预测问题。可以认为分类问题是标注问题的特殊情况。\n", "分类问题中可能的预测结果是二类或多类；而标注问题中可能的预测结果是所有的标记序列，其数目是指数级的。\n", " \n", "感知机、$k$近邻法、朴素贝叶斯法、决策树是简单的分类方法，具有模型直观、方法简单、实现容易等特点；\n", "\n", "逻辑斯谛回归与最大熵模型、支持向量机、提升方法是更复杂但更有效的分类方法，往往分类准确率更高；\n", "\n", "隐马尔可夫模型、条件随机场是主要的标注方法。通常条件随机场的标注准确率更事高。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2 模型\n", "\n", "分类问题与标注问题的预测模型都可以认为是表示从输入空间到输出空间的映射.它们可以写成条件概率分布$P(Y|X)$或决策函数$Y=f(X)$的形式。前者表示给定输入条件下输出的概率模型，后者表示输入到输出的非概率模型。\n", "\n", "朴素贝叶斯法、隐马尔可夫模型是概率模型；感知机、$k$近邻法、支持向量机、提升方法是非概率模型；而决策树、逻辑斯谛回归与最大熵模型、条件随机场既可以看作是概率模型，又可以看作是非概率模型。\n", "\n", "直接学习条件概率分布$P(Y|X)$或决策函数$Y=f(X)$的方法为判别方法，对应的模型是判别模型：感知机、$k$近邻法、决策树、逻辑斯谛回归与最大熵模型、支持向量机、提升方法、条件随机场是判别方法。\n", "\n", "首先学习联合概率分布$P(X,Y)$，从而求得条件概率分布$P(Y|X)$的方法是生成方法，对应的模型是生成模型：朴素贝叶斯法、隐马尔可夫模型是生成方法。\n", "\n", "决策树是定义在一般的特征空间上的，可以含有连续变量或离散变量。感知机、支持向量机、k近邻法的特征空间是欧氏空间(更一般地，是希尔伯特空间)。提升方法的模型是弱分类器的线性组合，弱分类器的特征空间就是提升方法模型的特征空间。\n", "\n", "感知机模型是线性模型；而逻辑斯谛回归与最大熵模型、条件随机场是对数线性模型；$k$近邻法、决策树、支持向量机(包含核函数)、提升方法使用的是非线性模型。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3  学习策略\n", "\n", "在二类分类的监督学习中，支持向量机、逻辑斯谛回归与最大熵模型、提升方法各自使用合页损失函数、逻辑斯谛损失函数、指数损失函数，分别写为：\n", "\n", "$$\n", "[1-y f(x)]_{+}\n", "$$\n", "\n", "$$\n", "\\log[1+\\exp (-y f(x))]\n", "$$\n", "\n", "$$\n", "\\exp (-y f(x))\n", "$$\n", "\n", "这3种损失函数都是0-1损失函数的上界，具有相似的形状。(见下图，由代码生成）" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAIDCAYAAADG5lBbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsnXlcVNX7xz+HYd8REFAUZHE3SrEUXEj9KtU3NVMxlzQ32tVyya9bmkvuVqa5UFpqKpalEkkqaPbDBRQ3FBdkUxEXthkZmGGe3x8XbjMwbHoFHM779bovueeee+4zn8GZh+c85zmMiMDhcDgcDofDeXKM6toADofD4XA4HEOBO1YcDofD4XA4EsEdKw6Hw+FwOByJ4I4Vh8PhcDgcjkRwx4rD4XA4HA5HIrhjxeFwOBwOhyMR3LHicDiPDWOsPWPsXcaY41N8RivGmNvTGp/D4XCkxLiuDeBwOM803QCsB3AEwIPSRsaYAoBlJfcNJKLfq/mM3QAcAbjXxDDG2CgADgAKARTX4NYfiEinP2PsFQDJRJTEGJsFYD4AOwAaAFMBfEdE9xhja0tsnUhE+TWxl8PhGAbcseJwOE/Co5J/i8q0KwFsB7CqTLs9gNiS6+VgjA0H8CKApUR0R2us249h28cA2gFQQ3CAKsNO6+dt0HLEGGMMgvPkwxjrVHJNVmLXKACzAGwt6d4dgCl3qjichgufCuRwOE9CRQ5LIYCHRHRF+wBwveR6WUeslCEAhgNQaLUVA8irqWFE1JmILInIlojsyx4QolnDASQCIAB/AOhPRMoy4xCAASWv9a0S26nk/DMAYUSUxhizBtAGwC81tZXD4RgOPGLF4XBqBGNMBuA4BOencUnzNiGwgw+I6BwAMwCNGGOty9xuX/KvqZ5xGwN4DcA8ItJ2pAhC1Em7Lyt5hhkR5dbQfhcIDtVEAL4AfoYwdXexgv5+AHpCmPIsABBQYtM4AHsBqBhjgyFEsEwARJa53wSArKzDxuFwDBPuWHE4nJpCAO5AmAY0K2m7ByGCUxqJMgMwoeTQh7metskQHJPFjLHFZS8yxvRtbFpYwVja9wUDOEj/boy6AkAIgI0AXiWim4yxyj4L2wH4sORZzgBcStrfgxD1NwbggX+jasdLnExtvip5fRwOx8BhfBNmDofzuDDGRgL4CUALIkp5gnGaArgKIdqzs8zlJQByAXypfQsEh8qYiH6oZNzWAC4COABgOBE9YoytBvAGEXmW9OkPYCmAXlp5XfrGag7gJIRomx2AFwC4ENGhkijeLQBHAcwrueU5ALsgRMf+IaK0qnTgcDjPPjxixeFwJIUx1gyAVTW6JhERlUzrbYEQ8fqQiDLLjPcFhBV5e2pgw6cAjIhoOWPsVQh5T0cZY/2gld/FGHsXwDcAzgKwYYxNBCAnopVlxvMH8BuEHLE/ACwGMAjANMZYHwj5Wi4AMkpyycAYa1Vyexx3qjichgNPXudwOJLAGLNljJkB2AzgcjWO0vmy1gBaAVhU1qkqwQG6yezV4U0AgwGAiKIAvFHSroAwlQnGmDOEacEdALoT0dWS+wZpvSZWMi0ZC+A8gFfwb77XlxDKTKyEsAIRAJ7XsqFJybMyamg7h8N5huERKw6H8ySUOkdrISR4vwAh9+ocET2v9wbGvgQwmYhKVxSmQnBICip4hj3+LetQdiwTCFOChUSkvdJQBa3IVMl03d9EVFia/1RSd6orEV3Quk8OIZeqtA8xxn4vsW0xgGYAvAH8D0KEbSSEEgsHACQA8GeMyUrqYDWFEMGq6HVxOBwDhEesOBxOjWCM2TPGJjPGDgDYVNLcGEIydyrKFONkjH3LGNvDGNMuGKq9ym8ZhOKijxhjVPaAkAj/fgXXiiAkjQ8vY6a+gqCvaVeIZ4y1AxBakh9VShHKl5C4CGA1hM/LNyEk5B8mIlXJ6sUcAAcBfADAFoB/yX3PAzinxw4Oh2PA8IgVh8OpKYUQojePAJyCELEZWpq8rmdFXEcA7kSkN+oEYSourmRcVZlrrwN4u+TntRCSw7UxhRCx+r8y7eKqnJKo1hcApkOoR1XKyxCcIRfG2IgyES9tNgAYUabtpNbrLIDgWD4CkAlgCGMsDkAXAMsrGJPD4Rgo3LHicDg1gogKGGOjAeyHkMfUvUwXWZlzCwj5SdqI0XIi+j+Ud4xK62XNhbBaUA2gN4DPiKg6+ValeVS9AXwNYQpvNBHtZ4wFljx3bUlO2AoAFoyxQRWM9SmAmQB6QKjKHgTgUslriAcQS0Tykuf9DMERPAlha5voatjK4XAMCO5YcTicGkNE4YDe6BQgOFIouc4AtECZopnQUyBUD3MAdAAQDCHpPAbAbsbY4GrkLZUa1rnk505EdK2kzQYljhcRrSypmP45hO1pykFEd0tex0cQImsJRJTLGAuBsH+hdlRqDYSaV99DyDM7VY3XyeFwDAjuWHE4HKk5AsC65Oc+EPKO/o8xxkqKdB4DYMwYM9JKYNeBMfYhhGjVOiI6WNL2AYTq57GMsbFEdKYSG0odq9UQNkjOKdlIORjCtJ4YQSOi+YyxFAA/ogLnCkIphesQomYPGWPnISSn/wMhalU6VhpjbD+ElYUV1tficDiGCy8QyuFwHpvKCoQyxuwgTImZA2gL4C8IU34LK9qGpiS5fA2E1XZ7AIRoO1+MsREQEubNIdSVWkRE8XrGiQEAIgrSansTQDiEqNMHRHS6gvvURNSnktccBGEVoAZC9CsZwFtEdIoxNgyCg0YAHgLoQ0SXKhqLw+EYHnxVIIfDeRJKV9npRJ4YYy8COAFhq5dREKYHLwKYBOA6Y+wD7W1kGGONGGMfQahvNRJC0c63yka0iGg7hIrmpbWp4hhjV0uiWdoYAbBkjLUuPQBcg5An9TaAfO1rWn0sUQGMMYuSAqI/Q8j7agthr8H9AO4yxjaWXPsZQCcIzl8MY+w/VYnI4XAMBx6x4nA4NYYxNgZCUrcPhMRyBwhO1AAA/4Ww4i4LwDAiita6rxWA7yAkgCdAmCp8DcK+fWYQnJ+PSqf/qrBhIIR6Up0B9CWiv7SunSppfxyOEVFPrbECALwLYYViMYTyEKuJSFVyfTqEFYemAGYS0bcl7f4A9gFwAzCbiBY9pj0cDucZgjtWHA6nxjDGfCAkaCcA2EpE8SV76Z2GkN+0HsAqfVN+jDEjCE5ZWyJ6p6Qcwi4ItaC+L3VYamDLc0RUdtXhY1EyFWhFRJ212mwhRKEOAPipdAWg1vV+AIYCmENEt8tccwIwC4LDpZTCRg6HU7/hjhWHw5EMxpgXhGrjFdWE4nA4HIOGO1YcDofD4XA4EsGT1zkcDofD4XAkok7qWDk5OZGnp+dTf05RURFMTatTh5BTHbie0sM1lRaup/RwTaWF6yk9taFpfHz8fSJyrk7fOnGsPD09ERcX99SfExMTg6CgoKf+nIYC11N6uKbSwvWUHq6ptHA9pac2NGWMpVa3L58K5HA4HA6Hw5GIOkle9/f3p9qIWGk0GhgZcd9RKrie0sM1lRaup/RwTaWF6yk9taEpYyyeiPyr09eg391Ll/hOElLC9ZQerqm0cD2lh2sqLVxP6alvmhq0Y/XgwYO6NsGg4HpKD9dUWrie0sM1lRaup/TUN00N2rHicDgcDofDqU0MOscqOzsbDg4OT/05DQWup/RwTaWlVM+8vDxkZWVBparR7jgcPfCcIGnhekrPk2pqYmKCxo0bw9bWtsI+NcmxqpNyC7VFfn4+/9KSEK6n9HBNpSU/Px8ymQx3795F06ZNYWFhAcZYXZv1TFNYWAgzM7O6NsNg4HpKz5NoSkQoKCjArVu3AKBS56q6GLTbnJycXNcmGBRcT+nhmkpLcnIysrKy0LRpU1haWnKnSgKKivi2j1LC9ZSeJ9GUMQZLS0s0bdoUWVlZkthj0I4Vh8NpeKhUKlhYWNS1GRwO5xnCwsJCstQBg3asamPbnIYE11N6uKbSUqonj1RJB99+RVq4ntIjhaZSfmYYtGPVqFGjujbBoOB6Sg/XVFq4ntJjbGzQqbi1DtdTeuqbpgbtWJ05c6auTTAouJ7SwzWVFq6n9Dx69KiuTTAouJ7SU980NWjHisPhcDj1l3v37iE/P7+uzeBwJMWgHSu+jF1auJ7SwzWVloamp0KhwNChQ5Genv5E41RWz1Amk+mcd+7cGZs2bRLPBw4cCEdHR3h6eoqHvb09pk2bpne8wsJC8edPPvkEH330UZX2zZ49G/v376+ynxQ87dqOZfXkPDn1TVODdqz8/Pzq2gSDguspPVxTaWlIehYVFSE4OBju7u5wd3d/7HH+/vtvrFmzpsLrlpaW5c61V11aWFhg0aJFSElJEY/JkyfD3Ny83FiXLl2Ch4cHHj58CEAozOji4lKljcOGDcOkSZNqxbkaMWKEjvMnNWX15Dw59U1Tg3asjh49WtcmGBRcT+nhmkpLQ9Jz+vTp8PPzw6pVq3RWNGVnZ2Pw4MGwtbVFp06dcP78+QrHICJ8+eWXGDdunE57aGgoPv/8cwAQp+q2bduGlStX4tatW4iMjMTy5csBCFWvKxq7LLt378Z//vMfcZGBkZERTExMdO4pLCwsN2b79u0RFRWFDz/8EKmpqRW+Hn3s2rUL3t7ecHR0xGeffVahvQBw4MABdOjQQafY5OXLl2scCQ0KCsLUqVP1XuNTn9JT3zQ1aMeqLrbrMWS4ntLDNZWWhqLn1atXERkZiZUrV5a79tZbbyE5ORmxsbH45JNPMGDAAMjlcr3j7NixA6+99ppYbVqj0WDq1KnYuHFjub7Xr19HfHw88vPzkZycLC4UMDY2xmeffQZ7e3vxWLNmTbnl6wqFAhs2bMC2bdvAGANjDGFhYVi0aJF4bmRkBHNzc72LEHx8fDB37lzMmjWr2jr9888/GDFiBKZMmYKTJ08iNjYWa9eu1dtXrVZj7dq1mDJlith24cIFBAcHIycnp9rP5HDq1xpFiSAi5P6TC+wEEFTX1hgOvDaQ9HBNpaUiPdn8utWZ5knr8G3evBmhoaHltvG4dOkSDh48iNjYWLRr1w7t2rXD9u3b8dtvv2HkyJE6fZVKJX788UdERESIbV988QWSkpIwaNCgcs8sjWAFBQVh/Pjx4njbt2+vls1LlixB+/btkZmZKbaNHDkSLVu2xNy5cwEIjl1BQYHeacTS/nPnzkVOTg7s7e2rfOaaNWvQp08ffPjhhwCA5cuXY8SIEfj444/L9d24cSNGjhwpPluhUCA4OBjTp0/H5MmTq/UaORzAQCNW6odqnOtzDtgIPLpev5ZhPsv07Nmzrk0wOLim0tJQ9ExISEBAQEC59jNnzsDe3h4vvfSS2Na1a1ecPHmyXN/Vq1fjww8/1KkBNGLECOzfvx82NjZim/bP2mRnZyMqKgqMMRgbG5c7GGPIzc0FAKSnp2PFihVYvHixzhh5eXmwsrISz42MjGBlZVVhMrKZmRnatWuHK1eu6L1eljNnziA4OFg879SpE1JTU/HgwYNydhw4cAAjRowQ20xMTBAbG4sBAwZU61nVxcbGBjdu3MArr7wCOzs7+Pn5ITIyUryuUqnw3nvvwdnZGba2thgwYICOvTExMejYsSMsLCzg5eWFbdu2SWrfs0hFv6N1hUFGrEwcTeAywgWZ32fi1je34PuVb12bZBCcO3euQSUH1wZcU2k5d+6c3irMUkeM6pqKNu/Ozc2Fl5eXTuTO3t6+3NRaVlYWTp48iZkzZ+q0+/j4lBvz0aNHYnLwvXv3kJ2djS+++AIrV67E119/DQ8PD6SkpKCgoACAkMyemZkJNzc38b1wcHDAyZMn4eXlpTMtmZ2dDZlMptNGRFAqlbCzs9P7XjZq1Ah5eXlValSqh7e3t3guk8lgY2ODjIwMODo6iu2LFy/GjBkzdHQzNTVF8+bNkZKSUq1nVZf79++jT58+6NWrF86fP4/Dhw9jwIAB+L//+z/4+/vjhx9+wO7du/HXX3/B1tYWEydOxIIFC/DVV18BAIYPH44hQ4YgIiICx44dw4gRIxAcHAwnJydJ7XyW0P4drQ8YZMQKANwnCatkMr/PhDpXXcfWGAbZ2dl1bYLBwTWVloaiZ+PGjZGWllau3djYuNw0mqWlZbkcq/nz52PevHnVelZxcTEAoayCm5sbLl++jFdeeQX//POPjiMSGRkJOzs70cEC/p2a/fbbb+Hv7w9XV1ed4/jx4/jkk0/Ktbu5uSE2NlavPenp6dVaSQhUT4/U1FSkpqbWWrQzIiIC2dnZWLduHTw8PDB27Fj0798f33zzDQDAysoKRASVSgUPDw8cOnQIq1evFu+3srKCWq2GTCZDSEgIlEplg3aqgH9/R+sLButYWT9nDbwAFMuLcef7O3VtDofD4UhGQECATm5UKU5OTjo5TIAQtdHOxUpMTIRSqcQLL7xQo2e++eabOH/+PAICAuDv7w9LS0udFXZKpRKtWrXSKcVQen3GjBlQqVRQKBSQy+WQy+U4cOAALC0tYWlpiYMHD4rtCoUCarVar6OTlZWF5ORktG7dulo269MjLy9PR4+5c+di/vz5NdLiSUhPT4e7u7uODT4+PmJkbNiwYZgyZQrGjx+PRo0aYcCAAbh+/brYNzw8HOnp6WjVqhV8fX0rLZXBqRsM1rECAK/pXgCAW9/cAhUb1lRAXdCxY8e6NsHg4JpKS0PRc+TIkdi+fTsyMjJ02rt06YLU1FTcunVLbIuLi0OzZs3E83nz5tXIkSidYhk1ahTatm2rc007OnX16lU899xzOteLior0jvno0SNMmjQJs2bNwvz58/H222/jxo0bVdqybNkyvPnmm+WS9iuia9euOH78uI6NeXl5Yt2vU6dOwd7eHi1btqzWeFLg6+uL9PR0nVpZ165dQ4sWLQAIjm9ISAguXLiAtLQ0mJiYYMKECQCEhPr79+/j119/xcOHD7FhwwbMnDkT0dHRtWZ/faQ+TQMCBu5YFXcuhrmXOZQ3lbi//35dm/PMU1rUjyMdXFNpaSh6Nm3aFFOnTsWwYcOgVCrFdnd3d3Tr1g0zZ85EcXExzp07h19//RX//e9/AQCHDh1C69ata1RQVK2uOJWiS5cu2L9/P/Lz87FlyxYEBwdj7969MDMzw+XLl/UmFWdnZ6N///5wcXHBtGnTMGXKFPTq1QudO3fGtm3bKqwztW/fPuzevRsLFiyotu0jRozA9u3bcfbsWWg0GsyfP1+ckgSEVZClKxKlRi6XIyMjQ+coLCxEv3794OjoiPfffx+pqan4/vvvsX//fnHlYnh4OIYMGYL4+Hg8evQIjDEdTUqnDbWd6spqczUEKvsdrROIqNaPTp06UW0QHR1NtzffpuszrlNBekGtPNOQiY6OrmsTDA6uqbRER0dTYmJiXZtRKxQXF9OoUaOoa9eulJ2dLbYnJiZS8+bNydHRkUxMTGjUqFFi/759+1J+fn6VY48ePZrmzZtHRER5eXk61/z9/Wn79u3i+c2bNykwMJB69OhBubm51LdvX3J2dqa1a9eSSqXSsXfXrl3k6upKr7zySjk7VqxYQRYWFtSyZUtasmQJ3bx5U7y2bds2cnV1pZMnT1Zbn1KmTp1KxsbG5OzsTI0aNaJTp04REVF4eDgtX768yvtv3rxJwldl9enZsycBKHdERkZSXl4eXb9+nfr160c2NjbUoUMHioiIEO9VKBQ0duxYaty4MVlYWFBgYCCdP39evB4REUEvvPACWVhYkKurK82ZM6dGthkiZX9HH5fKPjsAxFE1fRyDd6w40sH1lB6uqbQ0JMeqlLCwMB0Hhkj4cv7jjz/oxIkTYptaraaLFy/WePzSL61jx46Rm5sbmZiY0Pnz5+mvv/6i/v37k4mJCQ0dOlTnyy0yMpLc3d2pb9++RET0999/U7NmzcjW1pZWrFhBGo1G77OSk5Np5MiRZGpqSv/884/YfuTIEUpKSqqx7aVcvnyZfvvtN7p3757YduXKFVIqlY895uMilRPA+Zf65lgxqoNKxf7+/hQXF/fUn5OWlobmzZs/9ec0FLie0sM1lZa0tDQoFAq0adOmrk0xGAoLC2FmZobCwkIcPnwYL774IpycnPDw4UPMnTsXY8eO1Zvblp2djeTkZHTq1AkajQZfffUVRo8eLW5nUxnZ2dkGu6F2qZ4c6ZBK08uXL1f42cEYiyci/+qMY9COlfZ/ztsbbuPWulvwi/KDqUv52iicqjHkD7u6gmsqLdnZ2cjMzOSOlYSo1WqdIqKcJ4PrKT1SaSqVY2XQyevnzp0Tf37wxwMozitw+7vbdWjRs422nhxp4JpKC9dTerRX/nGeHK6n9NQ3TQ3asdLGfbKwCubW+lvQFDbsFRQcDofD4XCeDgbtWGlvWWAfZA+r56yguqtC1q6sOrTq2UVbT440cE2lhespPRXt28d5PLie0lPfNDVox6pdu3biz4wxMWqVsSYDdZFb9qyjrSdHGrim0sL1lB7tSuqcJ4frKT31TVODdqyOHTumc974rcYwcTaB/KwcuX/n1pFVzy5l9eQ8OVxTaeF6Sk/ZfQY5TwbXU3rqm6YG7ViVRWYuQ5P3mgAAMn/IrKI3h8PhGAb37t3D1q1bq6zQXVBQUO0NbdVqNR49eiSFeXpRqVS4c+dOg68qznn2MGjHSt/yy6YfNEXrn1qj5Yba2xvKUOBLhKWHayotDU1PhUKBoUOHIj09vdJ+c+bMwTvvvIOkpKRK+73yyiswNjYGY0w8bG1tdc5LDxMTk3L1qyIiItC0aVPxPCEhATKZDJ6enjoHYwwKhaLc87W350lLS0OTJk2QlpZWqc3p6ekYOnToU3XyOJyaYNCfQt26dSvXZtrYFK4jXevAmmcffXpyngyuqbR069YNly9frmszaoWioiIEBwejc+fOle79t3//fuzcuRMLFizA4MGD8ffff1dYpPOXX36BRqOBiYmJ2FZQUIAmTZrg5s2bsLe3F9uLi4vLRZMsLS118l0sLCzg7OyMlJQUnX6MMZibm5d7/qhRo+Dn54fZs2eLNri4uFQsAoT9Ed3d3REcHIxDhw7B1LR+1ynUt38i58mob5oadMTqzJkzlV4vul8EVbaqlqx59qlKT07N4ZpKS0PSc/r06fDz88OqVavAGNPbJy4uDm+//TY2bdqE2bNn47XXXkPfvn1x/77+TekdHR3h7OwMe3t78cjNzYVMJoOHh4dOe2lfQIguLV26FHv27EFubi6WLVuGEydOVDqNV3YBUX5+PqKiohASEgIAMDISvp60nTy1Wo2ioiKd+xhjWLVqFdq3b48ZM2ZUodq/bNmyBU5OTjptw4YNw5gxY3TaPv/883JtT4K+SF11SUlJAWMMFy9elMweQ+BJNH0aGLRjlZeXV+G125tu40TzE8hYnVFhH44ulenJeTy4ptLSUPS8evUqIiMjsXLlygr7HDt2DH379sXs2bMxZMgQAMDSpUvRsWNHdOrUCYcOHarWs+7du4dGjRpV6LwBwhdbfHw8kpKSoFQqcfr0aWRmZkKtVkOpVOo4ZPb29rCzs0NhYaHOGBs3bkReXh5atmwJxhiaNWsGQHCstKcfBw0apNeG1atXIyIiAteuXavW66oun332GdatWyfZeDxnTHrqm6YG7VhVhmUbS2gKNLj17S0UP6pesiaHw+HUBzZv3ozQ0FC9+6MplUp8/vnn6Nu3LyZPnoxJkyaJ1xhjWL9+PUaMGIHg4GC89tpr2LdvH9Rqtc79jDHY2NjA3t4er7/+OoqKinQcI8aYzgrMNm3aYPfu3Zg9ezZcXFwQHh6OgQMHokOHDsjJydF7WFlZifdnZWVh8eLFOHbsmLiR7bVr12BkZKSzuW1hYSHCw8P1amJmZobQ0FBs3rxZColFzM3NYWlpKemYHMPGoB0rf/+Kt/WxC7SDzUs2UD9UI3MLXyFYHSrTk/N4cE2lpSI9GavbQ2oSEhIQEBBQrv3u3bto27YtVq5cia1bt2LBggU6ER/GGIyNjbF7927ExsZCLpcjNDQU+fn54hjm5uZgjCEyMhI5OTl48OCBjkNUOo2oL0eqFI1Gg7y8PAQEBEAmk8HY2FjnMDIywpQpU8T+8+bNQ5cuXdC9e3exLS8vr5xDY2pqWmnNoq5du0q+rZG+qcCYmBh4enri0KFDaNOmDaysrPDGG2/oJN9v3boVTZs2hZubG6ZPn462bdti4cKFsLS0xO3btzFkyBA4OjrC29sbO3bskMzeHTt2oE2bNnBwcMDgwYNx584d8dqFCxfQvXt3WFlZoWnTpuUinvPmzUOTJk1gZWWFXr16ITk5WTK7nib1zfE1aMfq7t27FV5jjKHZVCHUnL4qHVTMC4ZWRWV6ch4Prqm0NBQ98/Pz9W7e7eLigo8//hhnzpxBSEgIcnNzUVxcrBP1UavVSEhIQOfOnXH06FGcOHGi3Fil+U0AEB0dDW9vb9y+fbvCPoDgTF27dg2ZmZnw9vZGVFQULCwsEBYWBrVajaysLKhUKqjVakycOFEnyfx///sf1qxZA7lcLh53796FhYWFTptcLkd2djYePnyoV5dGjRrVaDr44cOHOpG4X3/9tdr3PnjwAFOmTMGmTZtw7Ngx/P3339i5cycA4MaNG/j444/xyy+/YO/evVi3bh3CwsLw3nvvQalU4vXXX4dcLkdcXBwWL16MUaNGIT4+vtrProjIyEiMGzcOCxYsQHx8PIgIr732mlhCIzQ0FK6urrh69Sq2b9+OmTNn4uzZswCAqKgoLF68GDt27EBiYiLc3Nx0nN/6jEpVv3KlDXpVYHp6Ory9vSu87vyGM8xbmEN5Q4n7v9+H8yDnWrTu2aMqPTk1h2sqLenp6XpXkRnaRguNGzdGWloaWrVqVe7a5MmTxZ+1p9tKkclksLa2Fs89PDwqfdaDBw+Qm5uLJk2aVNgnOjoaQ4YMQX5+PmxtbfHbb7/Bz88P3333ndgnKCgIAwYMwBdffAEAOjlbrVu3RnFxsU65jKKiIqhUKjg5Oem0q1QqBAYG4siRI+XsqOj9rwh7e3udBQ8fffRRte+Vy+XYuHEjunbtCgD4z3/+g9TUVADCIoo2bdqgS5cuAID27dsjNTUVXbt2xeHDh3G+AUZKAAAgAElEQVTmzBlcu3YNLVq0QIsWLbBq1Srs2bMHnTp1qvbz9fHNN99g9OjRYk5dWFgYnJyccOzYMbz88suwsrISV3MGBQWhoKBAdJCtrKxARCgqKoKLiwu2bdtW73KXKkKlUlUaQa1tDDpiVRVMxuD+ibBMOX1F5XVgOBwOp74QEBCAiIiICq+rVCqxBlXZxHF7e3tYWlqiffv2Fd6vvWIvOTkZL774Yrk+2l+6rVq1wpIlS/D777/Dzs4Ofn5+5foolUo8//zzeu9XKBRQKpU6kanAwEA4Ojri5Zdf1mkvLCzU61QBwB9//CE6OtXByMhIp76WPke0Iuzs7HSeZWZmJurm7e2NK1eu4Pbt27h9+zauXr2KDh06AAAyMoQFU/7+/uL7cfbs2XIlKR6H1NRU+Pj4iOf29vZwcnISx964cSPMzc3h7++PZs2aYdasWWI0KzAwEF9//TVmzZoFR0dH9OrVS5IoWkPEoB0rX1/fKvu4veMG40bGkFnJoM5XV9m/IVMdPTk1g2sqLQ1Fz5EjR2L79u3il3RZSvOqzp8/rzdxfN26dXoT3wEhUqTt9Pz1118ICgoq1087n6hJkyaYMGFCuahBQUGB2DctLQ3PPfecznMqYvv27UhMTERcXBwuXryI+fPnV9i3lFu3bmHbtm0YOXJklX2lQLumV1lcXFzg4OAAT09PeHl5Ydq0aeI+lt7e3jA2NkZ8fDwSEhKQkJCAc+fOYfHixU9sk6enp86qyJycHNy7dw8tWrSARqPBjRs3sGHDBty9exf79u3Dpk2b8NNPPwEArl+/jsDAQJw+fRr37t1Du3btxMhXfaei3+W6wqAdq+qILbOS4cUrL8LvLz8Y2xj0zOgTU99+eQ0Brqm0NBQ9mzZtiqlTp2LYsGE6Do42ZXOgylJR+QSZTIbw8HC0adMGu3btQlxcHEaNGoVVq1Zh+PDhSE9Px507d8Rprsr4/vvv8frrr2Pjxo1wd3eHiYkJYmJiMH/+fMycOVPvPeHh4Xjvvffw008/wdPTE3v37sW3336LQYMGVRjVUSqVGDZsGKZNm1bplGVtsWHDBvTr1w/nz59HamqqTn0tf39/PPfcc/juu+8gk8lw9+5dvPHGG9iwYUO1x7979y4yMjLEozRB/aOPPsKPP/6I8PBw3Lx5E+PGjcPzzz+P7t27w8jICGPHjsWCBQtw8+ZNaDQaEJHoRB87dgyvvvoqjh8/jtzcXDDGnpmpwMpKgdQFBu1YVbeImqlz/a7UW1/gRemkh2sqLQ1Jz2nTpsHLywu9evVCTk5OuetEhHbt2sHa2rrc8e6775Yr0FmKTCZD7969sXz5cowdOxbr16+Hm5sbBg8eDHNzc7Rt2xZff/213j0FFQqFjkPXqlUrhIeHY+bMmVi7di1OnjyJwYMHY8KECeVsvn37NoYPH45x48bhxx9/RN++fQEIjkhsbCwyMzPRsmVLjBw5Er/99pv4pZ+Tk4PevXvD29sbU6dOfWw9paR///7YunUrunTpAjc3N1haWmLYsGFi3tj+/fuRkpICPz8/9O/fH/369cOCBQuqPX6fPn3QrFkz8XjhhRcAAMHBwQgLC8OcOXPEtgMHDkAmkwEQnNbY2Fh06NAB/fr1Q0hICN5++20AwJgxYzB8+HCEhISgRYsWOH78uBjNqu9U9MdFnaG9WqS2jk6dOlFtEB0dXaP+uSdz6fbm20/HGAOgpnpyqoZrKi3R0dGUmJhY12bUKmFhYaRSqXTaVCoVAaCbN2/qveeHH36gdu3alWv/5ptv6LXXXiMzMzPy9/enEydOUF5enk6fU6dOUdu2bcnX15eSkpLE9oEDB5KVlRWFhISQUqmkTz/9lLy9vcnFxYX27dsn9nv48CGNHz+eTE1NKSYmhjQaDY0ePZpkMhn16NGDrl69qtdmjUZDYWFh5OPjQ3379tV5rWFhYVXqVJt069aNvv76a8rMzKR79+7RwYMHyczMjC5evFhOT86TI5WmlX12AIijavo4jKpYLsMYswOwE4AMgAJACBEVleljDCC55ACAj4joQkVj+vv7U1xc3ON7g9UkMTERbdu2rVbfgpsFOOl1EkbmRuiS0gWmLjyKVZaa6MmpHlxTaUlMTARjDG3atKlrU55Jdu7ciaNHj2LEiBHiPpYFBQXlakc9evQIy5cvx2effSZOv5Yu2/fz84ORkRE2bdoElUqFMWPG6K0zdOTIEfTs2RMymQwxMTEoLCxEv379qrRRo9FALpfD1tb2SV/uU2Pnzp1YvHgxbty4AbVaDU9PT7zzzjuYMWMGlEplpbW4ODVH3+/o43D58uUKPzsYY/FEVK3Cg9VxrN4HcI2I/mKMrQcQSUT7yvTpCMHhqtZGTbXlWKnV6hrtdn9hwAU82PcAzWc2h9dir6do2bNJTfXkVA3XVFrUajWuXbvGHSsJIaJ6l8PyLMP1lB6pNJXKsaryE52ItDdJcgaQpadbFwD/ZYy9DOACgFAi0llixxibCGAiIKwgiYmJAQB4eXnBxsZGrJbr6OiIdu3aidslGBsbo1u3bjhz5oxY+M3f3x93795FerpQIsHX1xdmZmZifkXjxo3RsmVL/Pnnn7C2toaZmRm6du2KuLg4yOVyAMBLL72EjIwM3Lp1C4CQC2ATaoMH+x4g7es0KN9Qwvs5b8TGxgIQdml/6aWXcPLkSXGlS9euXXHz5k1kZgqV29u2bYvi4mIkJSUBEBJM3d3dcfLkSQCAtbW1mC9Quk9Wt27dcPXqVWRlCbK2b98ehYWF4sqOZs2awcXFBaWOqK2tLTp27Ijjx4+L21D06NEDly5dwoMHDwAIfzHm5+eLVXM9PT3RqFEjsV6Lg4MD/Pz8cPToUfEXsmfPnjh37hyys7MBAB07dsTDhw/FZFEvLy9cvXpVdAKkfJ+OHz8OANV+n2QyGRITEwEArq6uaNGixTP7Pp0+fVoslCjV+/S0/j89C+8TEcHR0VGsJG5ubi5uhQIIq+VMTEzw6NEjAEKCt5WVlU7lcWtraxQUFIg5RBYWFiguLhZXsZmamsLY2FgcQyaTwdLSUmcMGxsbPHr0SBzD0tJSZwNhU1NTyGQy8bXLZDKxGKb2GAqFQswlsrS0hEqlEoshmpmZgTEm5pcYGxvD3NxcHIMxBmtra50xrKysxDyfisYwMzMTN7VljIm/e6V/hFtZWaGwsFD8vdansampqThGqcZyuVwcw9raGkqlstIxDPV90mg0sLW1lfx9sra21tG4Ib1PGo0GVlZWT/w+FRcXi75J2c+9mlBlxErsyFhXAAuJqLeea50BZBDRHcbYjwD2lI1qaVNbEauYmBi9y4Qr42zQWeQezYXXl15oPqP50zHsGeVx9ORUDtdUWmJiYuDi4sIjVhKSn58PGxubujbDYOB6So9UmkoVsarWqkDGWCMA3wAYW0GX80RUuiFRHIB6UUzmcZZee8wUqhCnr05HcQHfnFmbhrKUvTbhmkoL11N6+LSVtHA9pae+aVqlY8UYMwUQDmAmEaVW0O0nxpgfY0wGYCAAaXfBfExqUoG3FIe+DrB+wRqquyq+OXMZHkdPTuVwTaWF6yk92tvfcJ4crqf01DdNqxOxGgegI4BZjLEYxtg8xtjCMn0WAPgJQAKAWCI6JLGdj8XjTDcyxuAx1wPuk93h+F/Hp2DVs0ttTN82NLim0sL1lJ7SHByONHA9pae+aVqd5PX1ANZX0ecigOcq61MXaCeu1QTngc5wHsg3ZC7L4+rJqRiuqbTI5fIa7ffGqZpnpfr2swLXU3rqm6YGXXldCkhDFVYo5nA4HM7TQaVS4c6dO/XuS5PDqQqDdqxeeumlJ7r/3q/3cLrdaTyIeCCRRc82T6onpzxcU2lpaHoqFAoMHTpULJWh3c4Yw40bNwAAmzZtKqeNQqHA3bt3ddouXLiAsLAwAELpjj///BPx8fFi24ABA/Dzzz9XaE9ERASaNm0qnickJEAmk8HT01PnYIzpnb7R3pokLS0NTZo0QVpaWqUapKenY+jQoeJy/foOj6hKT33T1KAdq4p2fq8uyhQlHl15hLQlaTxqhSfXk1Merqm0NCQ9i4qKEBwcDHd3d7i7u+tcK610XrpK0szMDObm5sjOzsbBgwcxefJkuLu7Y+LEiTr3OTg4YMGCBYiJiYGpqSlMTEywbNkyGBkZQaVS4ciRI2jevOIyNJaWljoVsC0sLODs7IyUlBSdAxBqIpVl1KhRWLhQSOE1MTEBALi4uFSqQ+nrDw4OFusa1WeeBRufNeqbpgbtWJUWK3xc3Ca4wdjBGHn/l4fc47kSWfXs8qR6csrDNZWWhqTn9OnT4efnh1WrVpVbbl723MjICLGxsQgICMCOHTvQpk0b/PPPP/jtt990+qWkpODjjz9Gbm4uioqKcO7cOZiamsLOzg5Hjx6FiYmJzsrL0mm6tLQ0LF26FHv27EFubi6WLVuGEydOVDqNV/aP1fz8fERFRSEkJES0GfjXwQKgUxRS+7WuWrUK7du3x4wZ1dr8Q4fc3FyYmppi7dq1lfaLiYmBp6dntcb09PQUC02WpbQgZX0iJiYGjLFnNuezvmlq0I7Vk2JsY4ymHwlh7bQllYejORwOp7a4evUqIiMjsXLlynLXnJ2dYWdnB0CogD9gwAAAQEBAAC5fvoytW7ciNDQUbdu2hUKhwPr160UHaNeuXYiJicGECRNw8+ZN7Nu3DxqNBmFhYdi9ezfkcjkaNWoEOzs7mJubw9dXKFmoUCgQHx+PpKQkKJVKnD59GpmZmVCr1VAqlbC3t9c57OzsxOrdpWzcuBF5eXlo2bIlGGNo1qwZAMGxYoyBMQYTExMMGjRIryarV69GRESEuBtCdYmKioJKpUJERESN7quvfP755xgzZkxdm9GgMWjHqlWrVk88RtOPmsLI0ggPIx8iPyG/6hsMGCn05OjCNZWWhqLn5s2bERoaqrcgampqqrjl0aVLl7Br1y4AQE5OjrglSCk7d+7E1KlTxfykr7/+Gt27d4eLiws8PDzQpUsX+Pr64tdff8XevXvx9ddfIycnB3/88QdatGgh5nC1adMGu3fvxuzZs+Hi4oLw8HAMHDgQHTp0QE5Ojt5DOy8mKysLixcvxrFjx0AkLBi6du0ajIyMxPPSrVTCw8P1amJmZobQ0FBs3ry5RlpGRESgd+/eiImJqZU8LV7EVnrqm6YG7VjJZLInHsPUyRRuE9wAAGmLG3bUSgo9ObpwTaWlMj1jWEyFx+2Nt8V+tzferrSvNnGd4irslzQx6Wm9TCQkJCAgIEDvNUtLS3EarXT/wHbt2uHixYuwtLSEtbU1rK2tYWFhgQkTJuDdd98V95kLCgrCmTNncPz4cVhZWaFHjx5IS0vDhAkT8ODBAzGHLSMjo1xelzYajQZ5eXkICAiATCaDsbGxzmFkZIQpU6aI/efNm4cuXbqge/fuYlteXp6YK1aKqampTg5XWbp27Sruk1kdiAiRkZF4//33YWVlhcOHD4vXlEolPvjgAzg6OqJFixY61wDgr7/+gp+fHywsLNCqVSvs26e7i9vhw4fRokULNGrUCKGhoaLTxhjDmTNn0K1bN9ja2qJr1644deqUeJ9cLse7776Lxo0bw9PTE8uXL9eZNl23bh08PT1hYWGBF198EfHx8QCAhQsXgjGG+fPnY+vWrWKUT3tBwONCRFizZg28vLzg5OSE8ePHi3uNAsJUYseOHWFhYQEvLy9s27ZNvKZSqfDee+/B2dkZtra2GDBggLhfqlQ8c5XXn2VKN4B9UppPaw5mypAXmwd1vrrqGwwUqfTk/AvXVFoaip75+flwcHCost/atWuxaNEivPDCC7h9+zbOnTuHuLg4xMXF4ezZs0hLSxOnE62trfHjjz/ip59+wsqVK9GrVy+8+uqr+P7772FsbIyZM2eKm3nfvHkTrVu31nmWRqPBtWvXkJmZCW9vb0RFRcHCwgJhYWFQq9XIysqCSqWCWq3GxIkTxc3HAeB///sf1qxZA7lcLh53794VN9LVPrKzs/Hw4UO9r7dRo0Y6X/hVcerUKWRlZaFHjx7o1q2bznTgwoULceDAAURGRmL37t3YsmWLeE0ul2PgwIF48803kZKSghkzZmDUqFE6EcEtW7Zg165d+Ouvv3Do0CHMmTMHgBBR7NWrF4KCgnDhwgW8/vrr6N27t7j6cezYsThz5gyio6OxZcsWrF69GqtXrwYAJCUl4YMPPsCSJUtw7do19OjRA2PHCjvNTZs2DdnZ2ZgxYwbeeustZGdnIzs7W+8igZqyfv16LFq0CBs2bMDx48eRlJSEkSNHiteHDx+O7t27Izk5GUuWLMGYMWNw//59AMAPP/yA3bt34+DBgzhz5gzy8/OxYMGCJ7ZJGymcR0nRDrPW1tGpUyeqDaKjoyUbK/toNqkL1JKN9ywipZ4cAa6ptERHR1NiYmJdm/HU6d+/P0VFRZVrLygooD///JMmTpxIAKhfv350/Phx2rhxI8lkMrKzs9M5ANDly5fF+83Nzcna2poAiH1kMhktXryYioqKyMnJie7cuUOvvvoq/fzzz+J9R44cIUdHRzI1NSUnJydKSEggIqLevXvTDz/8QEREHTp0oNmzZxMRUWhoKM2YMUO839LSkszMzMjKyko8TExMCEC5dlNTU3r55Zf16hIVFUUDBw6sto5z5syhNm3aEBHRihUrqFmzZuI1b29vWrlypXj+zTffkIeHBxERqVQqunPnDimVSjpz5gwtXLiQAFBaWhoREXl4eNA333wj3rt+/XpydXUlIqIFCxZQ69atdezo1KkTzZ07l+7evUsAKDY2Vue5zZs3JyKi1NRUMjIyoi1btlB+fj5pNBpSq3W/l+bNm0ejR4+utgZEwv8bAJSfn6/3eps2bWjJkiXi+YULFwgAJScnExGRj48Pvf/++3T37l1Rn1K2bdtGDg4OdOLECSoqKqLi4mIqLi6ukX1VkZeXJ8k4lX12AIijavo4Bh2xcnV1lWws+x72kJk37GkbKfXkCHBNpaWh6BkQEKA32fratWv47LPP0KlTJwBCLlZgYCDMzc3RrVu3cnlOMplMJz+loKAAW7ZsQWBgIHJycpCWlgYbGxuEhITAxMQEQ4cOxfTp03H06FH06tVLvK9Vq1ZYsmQJfv/9d9jZ2cHPzw+AbkVspVKJ559/XjzXvqZQKKBUKnUiU4GBgXB0dMTLL7+s015YWIgjR47o1eWPP/6o0X6Rpcnu9vb2mDt3LtLT03HhwgUAwgpT7VWAPj4+OvcuXLgQrq6uePvtt8U6YtoRq9LEfgBo0aKFmMyfkZEBb29vnbF8fHyQkpKC1NTUcs/y8fFBRkYGiouL0bx5c+zcuRObNm2Ci4sLXnrpJURFRVX79T4uqamp5WwCIJbOCA8PR3p6Olq1agVfX1+sWbNG7Dts2DBMmTIF48ePR6NGjTBgwABcv35dUvuMjavcRKZWMWjHqkWLFpKPqcpW4d7ee5KP+yzwNPRs6HBNpaWh6Dly5Ehs3769XN2uDh064OzZs+XqUzHGEBsbK9Z8Kj2Ki4vL5afs2LEDd+7cweTJkzFx4kT897//hZeXFwDgk08+wfbt2zF48GA0btxYvKdJkyaYMGFCuWmngoICAIJTlZaWhuee+3fns8pqD23fvh2JiYmIi4vDxYsXMX/+/Co1uXXrFrZt26YzRVUZd+7cwdmzZxEWFoaEhARcunQJrq6uosPq6uqqo2+pEwEIGu3atQuXL1/GhQsX9K7O1O6fnp6Oxo0bw9jYGN7e3uUci2vXrqFFixaiI6e9svHatWto1qwZZDIZMjIy0Lx5cxw/fhzZ2dkYNGgQ3nzzTZ2k+9KEfynx9PQsZxMg/H9TKBS4f/8+fv31Vzx8+BAbNmzAzJkzER0dDUCYng8JCcGFCxeQlpYGExMTTJgwQVL76lvyOp8KrAGqPBX97fA3Rcui6dGNR5KO/SzAp62kh2sqLQ1lKpCI6Msvv6TAwEAqKCjQex0ApaenExHRTz/9RD179izXRyaT0c2bN8u137lzh1555RVydnYmBwcH+vzzz0mj0dC8efPI3t6ePD09xWkgbaKjo8nb21s8T0xMpPv379NXX31F3t7edPPmTYqOjqbMzEzKzMzUa/fu3bvJxsaGDh48SEREp0+fJmdnZ3rjjTf02kokTIF269aNli5dqve6PjZv3kzm5uakUCjEtrfffpu6detGRETTp08nLy8vio+Pp/j4ePL09BSnAteuXUuOjo505coVOnv2LPXt21dnaszDw4M8PT0pLi6Ozp49Sy1btqQPP/yQiIiuX79OdnZ2NGvWLEpNTaVFixaRtbW1OI0YEhJCnTt3posXL1JMTAy5ubnRqlWriIjo8OHDZGVlRZGRkXT79m368ssvydTUlB49+vf7aOfOneTj40Opqal0+fJlOnv2bJValE4FJiUlUXp6unhkZWURkTCV6ejoSFFRUXT58mXq1q0b9e/fn4iI5HI5WVhY0KpVqygtLY0OHz5MxsbGdOjQISISplufe+45iouLo4yMDBo0aJCosVTUt6lA7ljVkMTRiRSNaLoy/orkY9d3uBMgPVxTaWlIjlVxcTGNGjWKunbtStnZ2WJ7TEwMbd68mQDQvXv3iIjoxx9/rDDHSttBUiqVtH//furTpw/17NmTMjIyKCUlhbZv3069e/emLl26UFZWFo0bN47c3Nzol19+IY1GI95/4MAB8vX11bFx/fr1ZGlpSZGRkbRz505ydHSk119/na5c0f0MvXXrFr311ltkY2NDe/fu1bl2/fp16tq1K5mYmNCIESNo7969Yp5OdnY2BQQE0OjRo2uUu/PGG29Qv379dNp+/vlnkslk9PDhQ1IoFDRx4kRycHAgX19f+uSTT0THKj8/nwYOHEhWVlbk5eVF3377LdnZ2VF4eDgRCY7VJ598Qr6+vmRvb0/vvPOOmL+Ul5dH8fHxFBgYSNbW1tSlSxc6ceKEaEN+fj6FhoaSk5MTNW/enJYuXarzupYtW0YtWrQgMzMzatWqlfhMbc0nTpxINjY2ZGNjQ5s3b65Si1LHquzx0ksvERGRRqOhVatWkaenJzk6OtLYsWMpNzdXvD8iIoJeeOEFsrCwIFdXV5ozZ454TaFQ0NixY6lx48ZkYWFBgYGBdP78+eq8RdWGO1a16Fhp/7JKhSJJQdFG0RRjEkMFKfr/UjRUnoaeDR2uqbScOHGiwThWpYSFhekkC3/66afk6upK7777rti2adOmCiNW2snrY8aMoaCgIPr999/FtvT0dGrbti198cUXVFhYSEREarWa5s2bR1OmTBH7lToaISEhpFQq6dNPPyVvb29ycXGhffv2if0ePnxI48ePJ1NTU4qJiSGNRkOjR48mmUxGPXr0oKtXr+p9nRqNhsLCwsjHx4f69u0rtqtUKgoLC6uBYnVLRQninMdHKk2lcqwYSTwXWx38/f0pLi6u1p8rFYkjEpG1IwtN3muCluta1rU5HA5Hi8uXL6NNmzZ1bUaD4+zZswAAPz8/GBkZYdOmTVCpVBgzZky5elQAcOTIEfTs2RMymQwxMTEoLCxEv379qnyORqOBXC6Hra2t5K+B07Cp7LODMRZPRP7VGcegHauTJ08+ld3uFYkKnG5/GsyEoUtyF5g1rWeJc0+Jp6VnQ4ZrKi0nT56Era0td6wkRC6Xw9rauq7NMBi4ntIjlaZSOVYGvSqwdEWK1Fi1tYLzYGdQESFtWcOpxv609GzIcE2lhespPXXxx7chw/WUnvqmaf0q/vAM4THbA0WZRXDq71TXpnA4HA6Hw6knGPRUYGFhYf2rb/EMw/WUHq6ptBQWFiI5OZlPBUqIRqMR9x7kPDlcT+mRSlM+FVgNbt68WWvPqm+hyKdBberZUOCaSgvXU3oKCwvr2gSDguspPfVNU4N2rDIzM5/6MwpuFOBSyCUkT09+6s+qa2pDz4YG11RauJ7So1Y33I3nnwZcT+mpb5oatGNVG6jz1Li3+x5urb2Fwjv1y2vmcDgcDodTuxi0Y9W2bdun/gybF2zg9IYTNEoN0pYY9grB2tCzocE1lRaup7AvX1JS0hON8fHHH4t75pXd/6+6aDQa5OTkPJEdAHDx4kUsX778icepLiqVCnfu3NHZJFpKHldPTsXUN00N2rHS3mn8aeI53xMAcHvDbSgzlLXyzLqgtvRsSHBNpaWh6alQKDB06FCkp6eLbXv27EFQUBA0Gg2uXbsGe3v7So/PP/9cZ0wiwrZt28RNlokI48ePh7u7O9q3by8eMpkMWVlZOvd26tQJe/bsAQAkJyfDwcEBSqUSCoUC586dw7Fjx7B3714sWrQIkydPFu9LTk6Gn5+fXkcsNTUVc+bMgUqlqlCHiIgING3aVDxPSEiATCaDp6enzsEYg0KhKHe/Uvnv53ZaWhqaNGmCtLTK/1BOT0/H0KFDdTZArg4NIR+3tqlvmhp0uYWkpCS4ubk99edYd7CG81Bn3Nt9D2mL0tByvWFWY68tPRsSXFNpSUpKgouLS12bUSsUFRUhODgYnTt3hru7u9j++uuvY+zYsYiNjUWTJk2Qm5sLIhIdqDFjxiAoKAgpKSkYP368+KV04cIFzJkzBxqNBrm5uVixYgWKi4thb28PCwsLLFy4EGPGjBGfY29vD1NTUwBCdEqj0cDMzAxWVlYAIF4DgOPHj+PVV1+Fm5sbnJyc4OrqCi8vL+Tk5MDe3h4eHh7Izc3F6tWrMX36dLi4uMDOzg4mJibQaDQwNzeHj48PGGNQqVTIzc3FrVu3YGdnBwCwtLSEhYWF+DwLCws4OzsjJSVFRzPGmN7oxqhRo+Dn54fZs2fDxMQEAKr8PXJ3d4e7uzuCg4Nx6NAhnddbGYWFhdXuy6ke9U1Tg45Y1Saen3sCDLgTdgcFKbxIIYfDebpMnz4dfn5+WLVqFWHNmRoAACAASURBVBhjYrudnR2mTZsGc3PzGi1BJyKo1WqkpaXB3d0dhYWFKCgogEqlqjASWOqUnThxAra2tjh16hSGDBkCe3t7dOjQAQDQsWNHmJqaolmzZsjIyEBCQgL+/PNPrFu3Dvb29gAAmUyGSZMmIT8/H1ZWVpDL5Th79iyGDh2KtLQ05OTk4MyZM5g0aRIyMjIgl8thZ2eHtLQ0LF26FHv27EFubi6WLVuGEydOVDqNVza6kZ+fj6ioKISEhACAqFmpgwUIydFFRUU69zHGsGrVKrRv3x4zZsyots5///03GGOQy+XVvqemjBkzplwkUgq2bNmCoKAgycaLiYl56lrUCdXdVFDKo7Y2Ya5oM8+nxaURlyhaFk13ttyp1efWFrWtZ0OAayotV69ebRCbMCclJVHLli1JqVRW2u/y5csEgDw8PMjOzo7s7OyoadOmJJPJyMPDg6ytrWnGjBk697zzzjv07bffiucFBQU0Z84cateuXbnj/v37OvcGBgZSZGQkEQmbNwtfMUQxMTHk4eFRqa1qtVrnPDs7m0xNTcVNosPCwsjX15cKCgrEPomJiTRkyBDq3bs3WVtb0+DBg2nv3r10/vx58fWWPeRyuc5zVqxYQQCqPF577TW9diuVSvL19a32/+WDBw8SgKe6GbNCodDRqbr88MMPejfrLqWwsFBSu6OjoyXR4nFeqz6k2oTZoKcCtcPjtYHXYi94fu4JS5/yG44aArWtZ0OAayot7u7u5aZ/DJHNmzcjNDS0XHHZI0eOYMuWLTAxMUFoaChefPFFMUIzZswYdOnSBe+++26F46rVakRGRmLo0KEYO3Ysvv/+exQXF2Pu3LlYsGBBuf4KhaJcccbRo0fDwsJCJ8qlL3JWWFgIIyMjHD9+HKdPn4aRkREGDRoELy8vAMJU48yZM8VxNm7ciNWrV+tM5bVp0wa7d+9GTEwMxo8fj/DwcPFadRLns7KysHjxYhw7dgzdu3cHAFy/fh2tWrXSsb+oqKjCqJ2ZmRlCQ0OxefNmLF26tMpnGhs//a9dfZteS4GpqWm9mnIrpb7ZZNBTgSdPnqzV55k3NzdYpwqofT0bAlxTaalQT8bq9pCYhIQEBAQElGvXaDQoKipCREQEUlNTxfYjR47gxx9/rLKQ4q+//opWrVrBwcEBZ8+eBQAMGDAA9vb2cHJyEpPAmzVrBmNjY/j6+uLWrVs6Y2zduhUpKSmIjY0V21QqFdLT02Fvbw8rKysYGxvDzs4OO3bswL1793D16lUsW7YMycnJOHv2LExMTNC+fXvs2bMHISEh4nTRjBkzxMT5xMREva8/Ly8PAQEBkMlkMDY21jmMjIwwZcoUsf+8efPQpUsX0akCgLy8vHKOiampqU4OV1m6du2Kc+fOVaptKdqJ8vogIqxZswZeXl5wcnLC+PHjkZeXJ16/ceMGAgMDYWtri8GDB2Pw4MHlVsNWNBW4Z88etG7dGhYWFmjXrh0OHjwIANi2bRsYY3jnnXdw9OhRMMbAGMOVK1d07q9oKvD69et45ZVXYGtrCy8vL3z33XfV0qIqqtIiJiYGHTt2hJWVFby8vLBt2zbxmkqlwnvvvQdnZ2fY2tpiwIABePDggSR2VYVBO1Z1BRHh/u/38ehazVaLcDgcTnXIz8+Hg4NDufY+ffpg586daN26tZgjdP36dQwfPhzm5uaYN2+euBrQzMwMjo6OYmQnNzcXkyZNgo2NDX755Rc8ePAAa9euxaxZszB48GCsWLECKSkpotPk7u6O27dvo1mzZjo2TJgwAa1bt9b5AlYqlfDx8UFOTg62b9+OoKAgKJVKjB49GkOHDsXmzZvRvHlz0flxcXHBxYsXxQOAzrmNjY0YrStd/ZiZmQlvb29ERUXBwsICYWFhUKvVyMrKgkqlglqtxsSJE3WiG//73/+wZs0ayOVy8bh79y4sLCx02uRyObKzs/Hw4UO970ejRo10vvCfhPXr12PRokXYsGEDjh8/jqSkJIwcOVJH35dffhmXLl1CXl4emjRpgiNHjlQ5rlwux/DhwzFu3DjcuHED48aNw1tvvQWNRoOQkBBkZ2fj22+/RWBgILKzs5GdnY2WLateiKVQKNCnTx80b94cFy9exLp16zBp0iTExMQ8iQwAqtZi+PDh6N69O86fP48lS5ZgzJgxuH//PgDghx9+wO7du3Hw4EGcOXMG+fn/z96dx1VVrX8c/+zDPCqgiPOI5ogahuZUNtjkra6ZNlqW3mYr66ZZNnfrVv60rG7zaHmzbtbNrFs5l1KKMzmLCiUqoKAys35/LMAR4cCCs8/meb9evQI5Z591vmdTj3uv9aycU151rQ2OvhUYGhrqkdfd+fROUqakEH1NNF0+cU5fHU/l6WSSqVkV5mmz5dg1FR0dza5du+jUqdNpH7du3TqGDh1Kt27deOSRR2jZsiXt27cnPT2dnj178tRTT5VPIG/QoAHXX389DRs2ZM+ePZSUlHDo0KHyrtaPPPIIL774InByp+vCwsLyCeNvvfUWF110EampqbRs2ZL8/Hz27dtX/jqVadasGddccw1t2rQ57s+P/f7uu+8mIiKCBQsWMGLECHJycggPD2fOnDnExcUdd8XknHPO4fLLL+epp54COG6i/xlnnEFxcfFxt+cKCgooLCykUaNGx/15YWEh/fv3P2URs3v37iqvRrUquYI5Y8YMJkyYwAUXXADo4qJ79+7s2LGDtm3bkpiYyGuvvUbLli3561//ypw5c4iJian0dX19fQkICKCwsJDg4GDuv/9+7rnnHlwuFy6Xi4YNGxIcHIyvr2+VPyuAb775hoMHD/LKK6/g7+9Pq1atmDNnjpHVzpVlERISQlFREb6+vowcOZLhw4eXf2YhISEopSgsLKR169b8+OOPNR5PVTn6ilV8fJX2SzQu5sYYLD+LvbP2cnjDyT1TvJWn8nQyydSs+pLn2WefXd7A83Tat2/PxIkTmTdvHkuWLKFXr168/PLLDBkyhKFDh3Lrrbce9/gXXniByZMnM3r0aGJiYpg4cWL5Lcenn366/IrRif+T+uyzzwgPD2fdunWMGjWKhg0b0q1bN8LDwwkPD2f16tW0atWqSu+tcePG3HDDDRQVFZVfIQOO+3r48OFERkbSqVMn/vGPf/DVV1/RoEED4uLiAI5bFZiXl0fPnj3Lvz/2Z4cPHyYvL++4K1P9+/cnKiqKc88997g/z8/Pr/DK0Lfffku/fv2q9P5Od0sRdN+uDh06lH9f9nXZe2/fvj3z58+npKSEhQsXlr/nygQGBvLVV1+xYMECWrZsSffu3fn000+r9NzT2bVrF82bNz/uSuDFF19cadFfFZVlMXv2bHbv3k3v3r2JjY1l2rRp5Y8dNWoU9913H7feeiuRkZFcfvnlbN26tcZjqgpHF1bH3uOvS4GtA2k6tiko2PGYczaF9VSeTiaZmlVf8rz++uuZOXMmqampp31ccHAw99xzD35+ftxyyy0MHTqUCRMmMHz4cN59992THq+UYtu2bSxYsICUlBT69u3LuHHjAH3Fqqw56Pnnn3/c86677jry8/OZOnUqq1ev5sCBA0yfPp1NmzaRn5/Pxo0bSUhIqPR9paamMn369OMmu6vTXG1s1qwZY8eOPak3VW6ubnmTl5fHrl276NGjR/nPTmybcKyZM2eSnJzMihUrWL9+PU888USlY05LS+Pjjz8+7hbV6ZSNrSJt2rRhy5Yt5d+Xfd22bVsAunbtysMPP0xgYCBpaWlMnDixSq+bkZGBy+Xihx9+4MCBA0ycOJHRo0ezffvRfW5dLpfbzTZbt25NamrqcbmOHz++/AphTZwui8OHD7N//37+85//sGvXLt544w0mTZrEggULAEhOTmbkyJGsW7eOXbt24efnx9ixY2s8pqpwdGHlyR2vW09ujSvQxf4v9pP9m5l7755mtx3EnUAyNau+5Nm8eXMeeOABRo0adcrJ0CUlJfz+++8UFhYycuRImjdvTseOHfHz82PVqlU8+eSTJ63US01NpXnz5owdO5a1a9cSFRXF119/XX5b7dFHH2Xu3Lm8/fbbPPjgg2RnZzNx4sTySeQLFizg9ttvZ/HixQAsWrSI8847j99++42ffvqJq6++utL3dccddzB//vzjVuCVXWFSSrFgwQIyMjLw8fE57XHeffddhg0bxptvvkmLFi3w8/Nj4cKFPPHEE0yaNOmUz5k9eza33347H330EW3atOHLL7/k1Vdf5a9//WuFK03z8vIYNWoUDz74IM2aNav0/ZW9D4A//viD1NTU8n/27dsH6NucL730Ej/88AMbN27kjjvu4C9/+Qtt2rRh+/bt/PzzzyxdupTk5GR++umnU861O5WsrCzOO+88Zs2aRXp6evlYji2kOnXqxJo1a9i0aRPbtm1j+fLllR730ksvpWHDhtx1113s2rWL77//nnfeeYdevXpVaVzVzQLgL3/5C6+88spxf8EoO19mz57NiBEjWLlyJUeOHMGyrFrbpugkVe3LYPKfuupjtWDBgjp5nYps/ftWtYAFavX5qz06DlM8nacTSaZmLViwoF70sVJKqeLiYnXDDTeofv36qaysLKWUUgUFBer+++9X7dq1UxdeeKHq1auXGjp0qBo/frxatGiR2rBhg0pOTlbr1q1TK1euVEuWLFHffPONSklJUUod7SW1bNky1a1bN6WUUmPHjlXR0dHqrLPOUiNGjFD333+/evjhh1Xjxo3VypUrVW5urlq3bp2KiIhQ77//fvn4CgoK1JQpU1SfPn3UiBEjyv/8888/V+ecc85x7yUjI0OFh4erUaNGqaKiIvX777+rq666SimlVGFhoQJUUVGRmjJlihozZsxJ/bu++eYbFRsbe1w2r7/+ugoODlbz5s1Ts2bNUlFRUWrYsGFq48aNxz03LS1NXXPNNSosLEx9+eWXx/1s69atql+/fsrPz09dd9116ssvv1TFxcVKKd1n6+yzz1ajR48u/7OqmDt37in7ZCUkJCillCopKVFTp05Vbdq0UVFRUWrMmDHq4MGD5e+rV69eKjIyUvn5+SlAde7cWSUlJR33GqNHj1aPPfbYSa/9/vvvqzPOOEMFBASo1q1bqxkzZpz0mClTpqjIyEgVFBSkHn/88eN+VlGfq82bN6uhQ4eqsLAw1b59e/Xmm29WKYuyPlbVyaIsy169eqmgoCAVExOjHn300fKfHT58WI0ZM0ZFR0eroKAg1b9/f7V27drTjsdUHytHF1aFhYV18joVKcgoUIsbLFYLWKAyf8z06FhM8HSeTiSZmlVYWFhvCqsy77zzTvl5tGHDBpWQkKDS0tKUUkqtXr1aTZgwQQ0ZMkR16NBBRUVFqaCgIOXr66tcLpcClI+Pj8rIyDjumIsXL1a9evVSSqlTNiHduHGjatSoUfn3BQUF6vvvvz/pcVOmTFGtWrVSe/bsKf+zjz/+WPXt2/e4xxUXF6tp06adstHj4cOHFVBhM9QrrrhChYSEqJEjR6q8vDw1YcIE1b59e9WkSRP19ddflz8uMzNT3Xrrrcrf318tXLhQlZSUqNGjRysfHx81aNCgCht8lpSUqHfeeUd16NBBXXjhheV/XlhYqN55551TPud0SkpK3H5OmbffflsNGTJEpaSkqIyMDLVp0yY1cOBA9cADD1T7mE5Qk0yPZaqwspQHVsvEx8erFStW1PrrJCcne3y3+9RXUik+XEyLe1rgE3z6y9d2Z4c8nUYyNSs5ORnLsujcubOnh+IYubm5lU64rkhJSQnp6em1uh9mWb+tuLg4XC4Xb731FoWFhdx0002nbJQ5f/58Bg8ejI+PDwsXLiQ/P5+hQ4dW+jplqyTDw8NrNN6a5Llz507uuOMOli9fTnZ2NhEREQwZMoSpU6dW+VakE9Uk02P9/vvvFf63w7KslUqpKq2OcXS7hb1793r8f1ot7nZOZ2075Ok0kqlZe/furTebMNeVE1sruMPlctX6JuMnzuWpbILykCFDyr92Z987l8tV46IKapZn69atq7QatL6pSaa1wdGT1+2mKLuIkqI6mjwnhBBCiDrn6MKqW7dunh5CuT0f7mF5u+Wkf5ju6aFUm53ydArJ1KyyPD0xxcGpTmxlIGpG8jTPRKYm/5vh6MLKTkuvLR+LoowiUh5PoTjv1Jt52p2d8nQKydSs/Px8/Pz8Ku0VJKpOilSzJE/zTGSam5tbvg1UTTm6sDq2sZinRV8TTUiPEPJ35/PH6394ejjVYqc8nUIyNWvLli1ER0eTlpbGkSNH5H9iBkjxb5bkaV5NMlVKceTIEdLS0oiOjjYyHkdPXrcTy2XR9pm2rB+2np3P7KTpLU3xDZf4hTCtbILxH3/8QWFhoYdH4/3y8vLk9pVBkqd5Nc3Uz8+PJk2aGFmcAA4vrE7cdd3Toi6NIrx/ONk/Z7N76m7aPt7W00Nyi93ydALJ1KyyPMv2qBM1t23bNtq3b+/pYTiG5Gme3TJ19K1Auy27tiyLds+1AyD1pVQK9la8Z5Ud2S1PJ5BMzZI8zZNMzZI8zbNbpo4urOqiCam7Gg5oSOSlkQR1CKJwn3fdprBjnt5OMjVL8jRPMjVL8jTPbpk6+lagXXX+sDO+DX2xXJanhyKEEEIIgxxdWNl1joVfpJklnXXNrnl6M8nULMnTPMnULMnTPLtl6ui9Au0ub1ceOx7dQdOxTWk4oKGnhyOEEEKIU3Bnr0BHz7FaunSpp4dwWn++8yfpH6az/cHtXtFvx+55eiPJ1CzJ0zzJ1CzJ0zy7ZerowspuGzOeqOUDLfFr4kf28mz2fb7P08OplN3z9EaSqVmSp3mSqVmSp3l2y9TRhZXd+Yb50vYJ3ctq+6TtlBTIBs1CCCGEN3P0HKuSkhJcLnvXjiVFJazosYIjvx+hw7QOtBjfwtNDqpA35OltJFOzJE/zJFOzJE/z6iJTmWNVasOGDZ4eQqVcvi7aPa+bhqY8mULhAfv2tvKGPL2NZGqW5GmeZGqW5Gme3TJ1dGGVkZHh6SFUSdRlUTQY3ICirCKy/pfl6eFUyFvy9CaSqVmSp3mSqVmSp3l2y9TRfay8hWVZdHy1I6pYEdoj1NPDEUIIIUQ1ObqwiouL8/QQqiyka4inh1Apb8rTW0imZkme5kmmZkme5tktU0ffCszJyfH0ENymlCLj2wxykuw3dm/M0+4kU7MkT/MkU7MkT/PslqmjC6vt27d7eghu2/PuHtZduo4td26xXdNQb8zT7iRTsyRP8yRTsyRP8+yWqaMLK2/U+OrG+Mf4k708m72f7PX0cIQQQgjhBkcXVm3atPH0ENzmG+ZL22d109BtD22j+HCxh0d0lDfmaXeSqVmSp3mSqVmSp3l2y9TRhVVkZKSnh1AtMaNjCO0dSkFaAbte2OXp4ZTz1jztTDI1S/I0TzI1S/I0z26ZOrqwSkpK8vQQqsVyWXSY1gGA3f/cTd7uPA+PSPPWPO1MMjVL8jRPMjVL8jTPbpk6urDyZg0HNqTx1Y0pyS1h17P2uWolhBBCiIo5uo9VRESEp4dQI+3/2Z6gDkG0mtjK00MBvD9PO5JMzZI8zZNMzZI8zbNbppVuwmxZVgNgFuADHAZGKqUKTvG4d4AuwFyl1NOnO2ZdbMKcmQk2u+0qhBBCCC9kehPm64CpSqkLgT3ARad4wb8CPkqpfkA7y7Ji3RmwaZs3Q8eOMG7cdmzWCqraig4VkbPKs03QFi1a5NHXdyLJ1CzJ0zzJ1CzJ0zy7ZVrprUCl1GvHfNsYOFVzpXOAz0q//h8wANhy7AMsyxoHjANo1qwZCxcuBKBdu3aEhYWxZs0aAKKioujatSuLFy/WA/T1ZcCAASQlJZGdnQ1AfHw86enp7N69G4DY2FgCAgJYv349AEuWdCQjoxlvvdWOtLQ/mThxJwMH9mXFihUcOnQIgISEBFJTU0lLSwOgU6dO+Pj4kJycDEBMTAxt27Zl2bJlAAQFBZGQkEBiYiK5ubkA9OvXjx07drBnzx4AunTpQnFxMZs2bQKgefPmtGjRgsTERABCQ0OJj49n2bJl5OfnAzBgwAA2b97M3r061m7dupGfn8+WLTq+li1b0jC3IevOXQdA2JdhnDngTJYuXUpRUREAgwYNYsOGDeUbUcbFxZGTk1PeNK1NmzZERkaWT/CLiIggLi6ORYsWoZTCsiwGDx7MmjVryMrSm0D37t2bzMxMUlJSyj+nwsLC8s/NxOcUHR1Nx44dWbp0KQABAQH069fPaz+nJk2aUHYlNjw8nN69e1f6OeXn55dnaupzMv375E2fk1KK5ORk459Tbfw+ecvnpJTymt8nb/ic8vLyKCoq8orfJ2/5nAoKCsjKyqrV3yd3VHorsPyBltUPeFopdd4pfvYO8LJSao1lWRcCvZVSz1V0rLq4FfjFF3DttcUUFPhw3nnw+efQsGGtvmStUSWKpL5J5PyWQ6tJrWj3bDuPjGPRokUMHjzYI6/tVJKpWZKneZKpWZKneXWRqTu3AqtUWFmWFYm+EjVcKbXzFD+fDnyqlFpeelvwDKXUsxUdry4KK4DERPjLX2DvXujSBebOBZv1Eauyg8sPsqrfKix/iz7r+xAcG+zpIQkhhBD1gtE5VpZl+QOzgUmnKqpKrUTf/gOIA1Kq8uK1LTBwDYmJuqhKToa+feHXXz09qupp0LcBMTfHoAoUW+/Z6pF9BMsuswpzJFOzJE/zJFOzJE/z7JZpVSav3wL0BiZblrXQsqzHLMs6cdXfHOAGy7KmAlcDcw2Ps1qysrJo0wZ+/hnOOw/S0+Gcc+A///H0yKqn3XPt8GngQ+Z3mWR8nVHnr192f1uYI5maJXmaJ5maJXmaZ7dMKy2slFKvK6UilFLnlP7zhFLqkRMek42ewL4cOFcpdbB2hls9DRvCt9/CmDGQmwtXXQUvvojXrRj0j/an7VN6H8Gt926lJL/EwyMSQgghxLGqPHndpLqaY5WdnU14eHj590rBc8/Bww/r7//2N5gxA3y9qE1qSVEJySOSibklhkaXNarT1z4xT1FzkqlZkqd5kqlZkqd5dZGp6T5WXiszM/O47y0LJk2CWbMgIADeeAMuuwxKV196BZevi25fdqvzogpOzlPUnGRqluRpnmRqluRpnt0ydXRhVdbj4kQjR8L8+dCoEXz/PQwYAKWtLLxO7vbcOnutivIU1SeZmiV5mieZmiV5mme3TB1dWJ3O2WfD8uXQqROsWwcJCbBypadH5Z7tk7aTGJtIxry6n8guhBBCiJM5urBq1+70jTTbt4dffoHBg+HPP2HQIPj66zoanAF+jf2gBLbeUzcT2SvLU7hPMjVL8jRPMjVL8jTPbpk6urAKCwur9DGRkfC//8GNN8KRI3DFFTB9eh0MzoDmdzcnuHMwuVtz2f1i7d/LrEqewj2SqVmSp3mSqVmSp3l2y9TRhVVVm4b5+8P778OTT+qVg/feC3ffDaVbHdmWy89F7Ay93/XOZ3aSm1K7863s1oTNCSRTsyRP8yRTsyRP8+yWqaMLK3dYFjz6KHz8sS60ZszQV69K98S0rYghEUSPiqYkt4Qtd23xSEd2IYQQQmjOLKzy8mDkSJqnp7v91Ouugx9+0LcI586FgQOhdINx22o/tT0+4T5kzs1k/5f7a+11oqKiau3Y9ZVkapbkaZ5kapbkaZ7dMnVmg9Bnn4XJk1GhoVhffgnnn+/2ITZvhksvha1boXlz+OYb6NmzFsZqSNpraRxYeIAO0zoQ0CygVl6jpKQEl8uZtbinSKZmSZ7mSaZmSZ7m1UWm0iD0gQfgmmuwDh2CSy6BTz5x+xAdO8KyZbrHVVqa/ve339bCWA1pdnszun7WtdaKKoDFixfX2rHrK8nULMnTPMnULMnTPLtl6szCyt8fPv6Y3VdfDYWF+v7eSy+5fZhGjeDHH+Haa+HwYRg2DF57rRbGa4BlWeVflxSVkP9nvgdHI4QQQtRPziysAFwudt59t95tGfRVrAkToMS9fk8BAXpC+6OP6qfeeSfcfz8UF9fCmA3I3ZZL0llJrL98ParY7G1eX2/aVNFLSKZmSZ7mSaZmSZ7m2S1TZ86xOtEnn8BNN+mrV9dcA++9pysmN33wAYwdqw9z+eUwcyaEhJgfbk0U5RTxa+dfKUgrIHZGLM3vbO7pIQkhhBBeTeZYlUpKStJfXHutniAVGgqffqpnpVdj5+XRo/Xegg0bwldfHe3Ybie+Yb7Evqx7W21/eLvRW4LleQpjJFOzJE/zJFOzJE/z7Japowur7GOLp/PPh8WLoUkT+OknXRXt2eP2Mc89V09qb9tW7y2YkKD3GrSTRlc2IuqyKIqzi9l631Zjx82uRjEqTk8yNUvyNE8yNUvyNM9umTq6sDpJr156c8DYWFi9Gvr1030V3HTGGZCYCH37wu7d0L+/vpJlF5ZlETsjFlewi33/3kfGd7JJsxBCCFEXHF1Yxcef4nZou3bw88/Qpw+kpOiqKDHR7WM3bgzz58OIEZCTo+8uvvlmzcdsSmDrQNo83gaALbdvofhwzWfbnzJPUSOSqVmSp3mSqVmSp3l2y9TRhVV6RZ3XGzeGBQt0j6v9+2HIEN1m3U1BQTBrFkyapFcJ/u1v8Pe/u73wsNa0uLcFDQY1oOXfW+IKqvlHXWGeotokU7MkT/MkU7MkT/PslqmjC6vdu3dX/MOQEJgzR68WPHJEL/N77z23X8Pl0o3e334bfH3hhRfg6qsht3b3Q67a2Pxc9FzYk+a3N8dyWZU/oRKnzVNUi2RqluRpnmRqluRpnt0ydXRhVSk/P3j3XZg8WV9yGjMGnnkGqtGC4pZbYN48CA+HL77Qk9ztUEQf2zg0b1ceJQU2uZwmhBBCOJCjC6vY2NjKH2RZ8PTT8Oqr+utHHoG77qpWB9Dzz9dz41u3Pjq5PTm5GgOvBekz0/m1y6/sen5XtY9RpTyFLhJS6QAAIABJREFUWyRTsyRP8yRTsyRP8+yWqaMLqwB3moDecQfMnq0bh772Glx5pd7Hxk1du8Ly5Ufnxp99tu7u4Gn+zfwpOVzCzqd3cvh3998XuJmnqBLJ1CzJ0zzJ1CzJ0zy7Zerowmr9+vXuPWH4cL05YGQk/Pe/1b6fFxMDCxfq2uzgQbjoIn3H0ZMizo2g6a1NUQWKTbduQpW4f7vT7TxFpSRTsyRP8yRTsyRP8+yWqaMLq2oZMEDfz2vbFn77Tfe62rTJ7cMEB8Pnn+stCouK9BysyZM9u2Kw3Qvt8G/qT/Yv2aS9lua5gQghhBAO5ejCKjo6unpP7NRJt1fv0wd27ND385YudfswLpdeJfj66+Djo1cPXnst5OVVb1g15dfQj9hX9b3oHZN2kLfLvYFUO09RIcnULMnTPMnULMnTPLtl6uhNmIuKimq26/Xhw3rT5v/+V8+9+vBD3UuhGr77Tj81J0fXaXPm6HZanrD+qvXs/2I/UX+JovtX3av8vBrnKU4imZoleZonmZoleZpXF5nKJsylllbjKtNxQkLgyy/1xPb8fBg5El58sVrtGC66SF/0atFC32ns27dadxiNiJ0RS8SFEbR9uq1bz6txnuIkkqlZkqd5kqlZkqd5dsvU0YWVET4+MGOGvqcH8OCDcPfd1WrH0KOHbsPQuzds366nby1aZHi8VRAQE0Dc93GEdg+t+xcXQgghHMzRhZWxJZiWpWehz5oF/v6659Vf/1qtdgzNmsHixfCXv0BWFlxwAXz0kZlhVodSiqz5WVTllrDdlrQ6gWRqluRpnmRqluRpnt0ydfQcq1qxZIne/iYrS09u/+9/oUkTtw9TXAwTJsD06fr7KVPg8cd1DVeXNt6ykT3v7qHTe51oelPTun1xIYQQwgvIHKtStVK8DRyoJ0m1aVOjdgw+PjBtGrz8sl49+OSTcMMNeipXXWo4uCEAW8dvJS/19KsEvbYYtjHJ1CzJ0zzJ1CzJ0zy7ZerowurQoUO1c+AzztDt1ePja9SOAfR0ra++0vPkZ87UtwYzMgyP9zSa3NCEqGFRFGcXs3ns5tPeEqy1POsxydQsydM8ydQsydM8u2Xq6MKqVjVpoturX3YZZGbqjQJnz67WoS67TN9hbNZM/7tfP9i61exwK2JZFh3f6IhvhC+Z32Wy5709dfPCQgghhAM5eo5Vbm4uQUFBtfsiRUVwzz26CyjAc8/B3/9erclSqam6yFqzBqKidK+rAQMMj7cCez7ew8YbNuIT7kOf9X0IbBl40mPqJM96RjI1S/I0TzI1S/I0ry4ylTlWpVJTU2v/RXx99SrBf/5TF1MTJ8LYsVBQ4PahWrTQV6wuuUTfDjzvPPj001oY8yk0ua4JUZfrW4I7n9p5ysfUSZ71jGRqluRpnmRqluRpnt0ydXRhlZZWR/vhWZbub/X55xAUBO+8AxdfrFcOuiksTM+5uuMOXZtdey08/XS1epK6xbIsOv6rI60mtaLD9A6nfEyd5VmPSKZmSZ7mSaZmSZ7m2S1TRxdWde6vf9UdP2NiYP58Pal9+3a3D+Prq3uS/t//6Zrt0Ufh5purdRHMLQExAbR7th0+QT61+0JCCCGEQzm6sOrUqVPdv2ifPrq9evfusHEjJCTo9gxusiy49169o05wMHzwgd4WpxoXwaql6GARO5/diSo+eqnMI3k6nGRqluRpnmRqluRpnt0ydXRh5ePjoSsvrVrp9gsXXQT798OQIdWeLHX55Ucvgi1YUO2LYG5RSrH2krXsmLyDXS/sKv9zj+XpYJKpWZKneZKpWZKneXbL1NGFVXJysudePDxcd2Uv28D52mvhqaeqNVkqPl63zerWTV8E69sXli2rhTGXsiyLNlPaAJAyJYWcVTmAh/N0KMnULMnTPMnULMnTPLtl6ujCyuPKJktNm6bv7U2ZAqNHV6u9euvW+iLYhRfCvn1w7rnVbptVJZFDI2l+V3NUoeL363+nONf9TaeFEEKI+sbRhVVMTIynh6ALqvHjj7ZX/+ijardXb9AAvvkGxo3TtdnVV8Pzz9feisF2z7cjqFMQR5KPsOPhHfbI02EkU7MkT/MkU7MkT/PslqmjC6u2bdt6eghHDRt2fHv1vn1h82a3D+PnB//6F7zwgv5+4kRdaBUWGh4v4BPsQ+ePO2P5WqROSyVyR6T5F6nnbHWOOoDkaZ5kapbkaZ7dMnV0YbWsNiciVUevXvDrr/rfW7fqvWsWLXL7MJYFDzyg22YFBsLbb+umogcPmh9yeHw4rR9rDUDyTHvdx3YC252jXk7yNE8yNUvyNM9umTq6sLKl5s1h8WJ9BSszU98W/OCDah1q+HC9XWF0NPz4o14xmJJidLQAtJrYirif4uBW88cWQgghnMTRhZVt92MKDdUNqu67T9/Du+kmeOghKHZ/gnhCgl4x2LkzJCfrO4y//WZ2uC5fFxFDIsrz9MT+kk5l23PUS0me5kmmZkme5tktU0dvwuwV/vUvuOsuXVQNGwYzZ+p9bdx04IC+gjV/vt5VZ+ZMuPJK88PNXpHN5r9tpuvsrgS1s9fJLIQQQtQG2YS5VGJioqeHULnbboP//Q8iInTfq/79q3U/r2FDmDdPb32Tm6uLrJdeMrtiMDExkdSXUjmUdIjka5MpKSwxd/B6yivOUS8ieZonmZoleZpnt0wdXVjl5uZ6eghVM2SI3ganUydYtw7OOks3rXKTv7/e//mZZ3RB9cADuj9pUZGZYebm5hL7WiwBLQPIScwh5bEUMweux7zmHPUSkqd5kqlZkqd5dsvU0YWVV4mN1ZOlyjqADhkC77/v9mEsCx5+GGbNgoAAfadx2DDIzjYzTL8IPzrP7Awu2PXcLrLm19HmhUIIIYQXcPQcq/z8fAICAmr9dYwqKoIJE+Dll/X3DzwAzz0H1dgL6Zdf9F6D+/frPaHnzoWWLas/tGPz3PH4DnY+sRP/pv7Er43Hv5F/9Q9cj3nlOWpjkqd5kqlZkqd5dZGpzLEqtWPHDk8PwX2+vjB9Orzxhv76xRfhiiuqdcnp7LP1RbCyO4wJCZCUVP2hHZtn60daE94/nII/C9g0ZpOsFKwmrzxHbUzyNE8yNUvyNM9umTq6sNqzZ4+nh1B948bpSe2RkXofm/79oRonT/v2+srV4MHw558wcKCeI18dx+bp8nXRZWYXAtsE0uT6JliWVb2D1nNefY7akORpnmRqluRpnt0ydXRh5fXOPVdPaj/jDFi/Xk9qX7LE7cNERsL338MNN8CRI/r2YNmdxpoIbB3IWZvPIvrq6JofTAghhHAARxdWXbp08fQQaq5DB30/b+hQPVnqvPPgvffcPkxAgG7w/sQTesXg+PFwzz3u9SQ9VZ4uv6OnUM6qHIoPu9/ktD5zxDlqI5KneZKpWZKneXbL1NGFVXE1OpnbUoMG+nbg+PG6U/uYMXqCu5t9FCwLpkyBjz7SrRleeUVP3zp0qGrPP12e6Z+mk9Q3ic13ur+xdH3mmHPUJiRP8yRTsyRP8+yWqaMLq02bNnl6COb4+sK0afDmm/rrqVPh0kshy/12B9dfDz/8cHT61qBBkJZW+fNOl2dI9xAsH4v0D9L5870/3R5TfeWoc9QGJE/zJFOzJE/z7JapowsrRxo7Fn76CRo10pPb+/SBDRvcPsygQbBsmb7TuGqVXjG4Zk31hxXaLZTY12IB2HLnFg6tr+JlMCGEEMJBHF1YNW/e3NNDqB2DBsGKFdCzJ2zbpnde/uortw/TsaMurvr311esBgyAb7+t+PGV5dn0pqbE3BRDSW4JySOSKTpkqOW7gzn2HPUQydM8ydQsydM8u2Xq6MKqRYsWnh5C7WndGn7+GUaO1JOkrrgCnnwSStzbv69RI/jxR7jmGn2YYcPgtddO/diq5Bk7I5bgLsEc2XiELbdvkf5WlXD0OeoBkqd5kqlZkqd5dsvU0YWV3TZmNC44GD79VHdmtyx47DEYMaLqs9FLBQbCzJnw6KO6LrvzTj03/sT5gFXJ0yfEh66zu+IKdrH/6/3k7cxzayz1jePP0TomeZonmZoleZpnt0wdXVjVC5YFDz2kZ6E3aAD/+Q/06wfbt7t9mCef1J0cyubGDx8Ohw+7P6SQLiF0+XcXzlx5JkFtgtw/gBBCCOGlHF1YhYaGenoIdeeSS3Qz0U6ddDPRPn30JHc33XSTbibasKGetlXWsR3cy7PRZY0I7hBc/r3cEjy1enWO1gHJ0zzJ1CzJ0zy7ZVqlTZgty2oCfK6UGljBz5sDicDW0j8aoZTaV9Hx6moT5nrp4EG47jq947KPj95rcPx4fUnKDb//rrs57NgBrVrpw3Xr5v5wlFKkzUgjZ0UOZ7x/hmx9I4QQwusY3YTZsqwI4AMg5DQPSwCeUUqdU/pPhUVVXVq2bJmnh1D3GjTQl5oeflhPkrrvPrj5Zshzb65T58664XvfvrBrl145OG1astvDyU/LZ8fDO0j/MJ20V6rQLKueqZfnaC2SPM2TTM2SPM2zW6ZVuRVYDIwEsk/zmL7ArZZlJVmW9ayRkRmQn5/v6SF4ho8PPPMM/PvfeoL7Bx/oe3pV6QJ6jOhomD9fz4fPzoYJEzrz1lvuDSWwRSCd3u0EwLYJ2ziw5IB7B3C4enuO1hLJ0zzJ1CzJ0zy7Zepb2QOUUtlAZbdw5gFPAUeAHy3L6qGUWnvsAyzLGgeMA2jWrBkLFy4EoF27doSFhbGmtDtlVFQUXbt2ZfHixXqAvr4MGDCApKQksrN1bRcfH096ejq7d+8GIDY2loCAANavXw9AdHQ0HTt25NChQyxcuJCAgAD69evHihUrOFS6Yi4hIYHU1FTSSouNTp064ePjQ3KyvioTExND27ZtyyvhoKAgEhISSExMJDc3F4B+/fqxY8eO8p21u3TpQnFxcXkX2ObNm9OiRYvyFQuhoaHEx8ezbNmy8hNhwIABbN68mb179wLQrVs38vPz2bJlCwAtW7akSZMmlN06DQ8Pp3fv3ixdupSi0i1tBg0axIYNG8jIyAAgLi6OnJwctkdHEzp9Oj0ffxzfX38lv0cPNjz+OD6DBhEXF8eiRYtQSmFZFoMHD2bNmjVklXZy7927N5mZmaSkpHDbbdC4cS9ee60B48bB/Pm7mDjxIN27V/Fzarwbrgb1mWL98PUUvVYEjY5+TkuXLgWol59TQUFB+e9CmzZtiIyMJCkpCYCIiAi3Pqfa/n3yhs9JKUVycnLt/D6VLgipb58T4DW/T97wOeXl5VFUVOQVv0/e8jkVFBSQlZVVq79P7qjSHCsAy7IWKqXOqeBnAUqp/NKvpwI/K6W+qOhYdTXHqqioCF/fSmtH59u3D66+GhYuPLo1zh13uD3v6o03irnzTh+Ki/WKwY8+gqAqLvorKSph7QVrObDwAOFnh9NzQU9c/o5eO1Elco6aJXmaJ5maJXmaVxeZGp1jVUXfW5bV1LKsYOBCYL2h49bI5s2yITAAjRvr7W/uv19v3HzXXXr5X+nfbKpq4MBNzJsH4eHwxRdw7rmQnl6157p8XXT5dxf8m/uT/Us2OybvcP99OJCco2ZJnuZJpmZJnubZLVO3CyvLsoZYlnXXCX/8BLAAWA78Syllix0Ryy5fCsDPD156STcUDQ6GDz+Es8/Wy/6qaO/evVxwAfzyi278npioJ7cnV3FOu3+0P92+6EZYQhjNbm9WzTfiLHKOmiV5mieZmiV5mme3TKtcWJXdBlRKzVdKzTjhZwuUUmcopXqc+DNhM6NG6eV+7dvD6tVw5pm6cZUbunbVh+jTB1JSdH1W1ZZZ4Qnh9F7Wm6B20jhUCCGE8zh6kku36jReqg+6d9ebOF96KWRlwcUXw7PPVrrP4LF5xsToKVtXXqlbZ110Ebz7btVevmwhhFKKP974g8LMwuq+E68n56hZkqd5kqlZkqd5dsvU0YWV3ZZg2krDhvD11/D446AUTJ6sZ6RnV9xV48Q8g4Nh9my9r2BREdxyiz5MVfeBTnkihc23beb3635HFdfPzuxyjpoleZonmZoleZpnt0wdXViVLQkVFXC59MbN//2vbiw6Zw6cdVaFk6ZOlWdZc/fXX9dfP/usbvxelX6kTW9uim+UL5nfZbLjkfo5mV3OUbMkT/MkU7MkT/PslqmjCytRRZddpm8NdusGmzZBQoJe9ueG227T+0CHhsKsWXDeebrLw+kEtg6k62ddwQd2PbeL9E+quMRQCCGEsClHF1YtW7b09BC8R4cOekb6qFFw6BBcdRU89JC+x1eqsjwvugiWLoUWLfTKwX79dJ12OhFDIoidHgvApls2kf3b6Rr8O4+co2ZJnuZJpmZJnubZLVNHF1ZNmjTx9BC8S0gIfPIJTJ2q7+v985+6Wiq99FSVPOPidBuGXr1g2zZdXC1adPrnNLujGU3HNaUkr4T1V6wn/0973S+vTXKOmiV5mieZmiV5mme3TB1dWNVFd3fHsSy9cfOPP+rNAn/6SVdJv/xS5TybNYPFi2HYML3o8IILdJf2il/SIvaVWBoMbIB/U3+o4uR3J5Bz1CzJ0zzJ1CzJ0zy7ZerowkrUwDnnwMqVuklVWhoMHkyLzz/XKwirIDQUvvwSxo+HwkK48cajCxBPxeXvotuX3ei1pBcBzQOMvQ0hhBCiLjm6sAoPD/f0ELxbixa6WdV990FRER1efVXvOXialgzH8vHR2xK+/LJegPjEE7rAqmhlrF+UHz5BPgCoEsXBZQcNvRH7knPULMnTPMnULMnTPLtlWuVNmE2qq02YhUFffAE33ww5ORAbq7/v3r3KT//mGz0v/vBhGDhQX82Kijr1Y1WJIvmaZPZ9vo/u33Qn6uIKHiiEEELUAU9swmxLS5cu9fQQnGP4cFb861+6mNqyRbdk+OCDKj/9sstgyRI9/2rJEj2pfevWUz/WclkEdw6GEkgelczh3w8behP2I+eoWZKneZKpWZKneXbL1NGFVdExrQJEzR1q1ky3ZBg9GnJz4aabYOzYqnUDRc+BT0zUKwe3bNEbOP/886kf22ZKGxoNb0RxdjHrLl1Hwb4Cc2/ERuQcNUvyNE8yNUvyNM9umTq6sBK1IDgY3nsP3n4bAgL0v88+W/dWqIIWLfQVq4svhowMGDIEPv305MdZLovOH3Qm9MxQ8nbksf6K9RTnFRt+M0IIIYRZjp5jVVJSgssltaMpJ+W5apVuJLp9u94S54MP4PLLq3SsoiK45x69FQ7A00/Dww/rbg/Hyv8zn6SEJPJ35xM9KprOMztjuayTD+il5Bw1S/I0TzI1S/I0ry4ylTlWpTZs2ODpITjKSXn26qVbMlx+ORw8CFdcAX//+3Hd2ivi6wuvvqp7kVoWPPIIjBkDBSfc8QtoGkD3b7rjE+ZD3q48ig8766qVnKNmSZ7mSaZmSZ7m2S1TRxdWGRkZnh6Co5wyz4YN9RK/F17Q/RVeeEHf30tNrfR4Zb1I//MfCAqC99/Xjd6zso5/XGiPUHou6EncT3H4hvmaeTM2IeeoWZKneZKpWZKneXbL1NGFlagjlgUPPAALFkDTpnoSVc+eMG9elZ5+xRW6U3tMjD7E2Wfru4vHCjszDJ/A0h5XxYojm46YfhdCCCFEjTm6sIqLi/P0EByl0jwHDoTVq+HCC/XM9Esu0Rs5FxZWeuz4eL3gsGtX2LhRrxhcvvzkxxUfKWb9FetJ6pfkiOJKzlGzJE/zJFOzJE/z7JapowurnJwcTw/BUaqUZ3S0vlL17LNHN3IePBh27ar0qa1b6/YLF1yg930+91yYPfv4x7gCXOCCoqwi1l66loL93t2GQc5RsyRP8yRTsyRP8+yWqaMLq+0n3k8SNVLlPF0umDRJb4fTvDksW6ZvDX79daVPbdAA5s492h7r6qvh+eeP7jFo+Vh0ntmZ0F6h5G3LY/2w9RQf8d4J7XKOmiV5mieZmiV5mme3TB1dWAkPGzBA3xq89FI9I/3yy/Vs9ROX/p3Azw/eeENf7AKYOBH+9rejdxR9Q33p/k13AloFkL08m+RrkikpKqnlNyOEEEJUztGFVZs2bTw9BEepVp6NGukrVS++qHssTJsG/fufPDv9BJYFDz6obwUGBsJbb+n67GDpvswBzQLo8V0PfCN8yfg6gy13bsETPdlqSs5RsyRP8yRTsyRP8+yWqaMLq8jISE8PwVGqnafLBRMmwNKleiLVihW6B9bnn1f61Kuu0isFGzeGH37QNdnOnfpnIZ1D6P7f7rgCXeRuy6Uk3/uuWsk5apbkaZ5kapbkaZ7dMnV0YZWUlOTpIThKjfNMSNDd2q+8ErKzYcQIuPPOSvca7NtX7zF4xhmwYYM+zG+/6Z816N+Anot60uPbHuXtGLyJnKNmSZ7mSaZmSZ7m2S1TRxdWwoYiIuCLL+CVV8DfH157Dfr1g82bT/u0tm3hl19079H0dL3Q8Msv9c/CzwrH5a9P5ZKCEnKS7LVCRAghRP3h6MIqIiLC00NwFGN5WhbcdZeulNq31xPce/fWrddPM08qIkJ3crj5ZsjNheHD9ZY4ZU8pzi1m3bB1rBq4iuxfs82MtZbJOWqW5GmeZGqW5Gme3TJ19CbMwgtkZ+slf7Nm6e9HjdI7MzdsWOFTlIJ//AMmT9bf3347vPwy+PgoNt68kfQP0vFr5Eevn3sR3DG4Dt6EEEIIJ5NNmEstWrTI00NwlFrJMzwcPvlEX60KCdEFVs+e+mpWBSwLHn4YPv0UAgJ0HTZsGOTkWHR6qxORF0VSuL+QNReuIS/19PO3PE3OUbMkT/MkU7MkT/PslqmjCytvXH5vZ7WWp2XB6NF6YvuZZ+plf4MGwVNPQXHFzT9HjYKffoKoKPjuO72jTtoeF11mdyEsIYz8nfmsvdDe3dnlHDVL8jRPMjVL8jTPbpk6urCyLMvTQ3CUWs8zNlZfqXrwQV1QTZmi97U5zXY4/fvrPQU7doS1a/UKwrWbfenxbQ9CuoVw5PcjrLtkHUU5RbU79mqSc9QsydM8ydQsydM8u2Uqc6yEPf3wA9x4I+zZo+dbvf22nq1egcxM3cVh8WJ9R/HTT+HCM/NZNXAVIV1C6PJZF3yCvK8dgxBCCM+TOVal1qxZ4+khOEqd5nnBBfoS1KWXwoEDulPouHFw+PApHx4ZCf/7H9xwg37IFVfAG58H0GtJL7r+p6ttiyo5R82SPM2TTM2SPM2zW6aOLqyysrI8PQRHqfM8GzeG//5XL/kLCND72sTH6/YMpxAQAB98AI8/DiUlMH48PPhcAMqlT/PivGJSZ6SiSuxzP17OUbMkT/MkU7MkT/PslqmjCyvhAJYFd98Nv/4KnTvDxo269fq0abp6OsXDH3sMPvpIb+b8yiv66tWhQ5A8Ipmtd29l6/1bbTfZUQghhDM4urDq3bu3p4fgKB7Ns0cPvcfgbbdBQQHcdx9cfDH88ccpH3799fDjj/oW4Tff6EWGgdc1x/KzSJueRsoTKXU7/grIOWqW5GmeZGqW5Gme3TJ1dGGVmZnp6SE4isfzDA7WTav+85+jk6q6d69wM+dBg2DZMt3cfdUqGPJgJMHPdgYX7HxiJ7uer3i1YV3xeKYOI3maJ5maJXmaZ7dMHV1YpaSkeHoIjmKbPK+8Etatg6FD9XLAESN0H6yDB096aMeOuh1D//6QmgpDnoim4L4zwILtE7eze9puD7yBo2yTqUNInuZJpmZJnubZLVNHF1bCwZo10xsHvvIKBAbChx9CXBwsWXLSQxs10rcFR43Sc60u/r8Y/ry2IwDb7tvG/m/21/XohRBCOJSjC6t27dp5egiOYrs8yzZzPrZj++DBMHGinod1jMBAmDkTHnlEz3m/dmYz1g2JJerKRkReGOmhN2DDTL2c5GmeZGqW5Gme3TJ1dGEVFhbm6SE4im3zPOMM3bF98mRdbD3/vF45uGHDcQ9zufQuOe+9B76+cM/85kwp6Upuof418MRKQdtm6qUkT/MkU7MkT/PslqmjCyu7NQ3zdrbO098fnn5at15v21b3ujrzTJg+/aS2DDfdBN9/Dw0awJyvLM45B/7YXsy6S9axd/beOh22rTP1QpKneZKpWZKneXbL1NGFlaiH+veHNWtgzBjIz4d774WLLoK0tOMeNmSIXjHYpo3u4vD3PnvJ/C6T36/9nf1fyZwrIYQQ1ePowioqKsrTQ3AUr8kzLAzeeUe3ZYiK0vsOdu8O//73cQ/r3BkSE/XGzTMzY/jCvyWqSLFhxAb2zdlXJ0P1mky9hORpnmRqluRpnt0ydfQmzCUlJbhcjq4d65RX5vnnn3DLLXoFIejWDK+9ppcKlsrN1d0aZs9W3G5t52q1G8vXosusLjQe3rhWh+eVmdqY5GmeZGqW5GleXWQqmzCXWrx4saeH4ChemWfTpjB3LvzrXxASArNnQ9euMGdO+UOCgmDWLHjoIYvXVTs+ofTK1cgNtT7nyisztTHJ0zzJ1CzJ0zy7ZerowkoIQK8U/NvfYO1a3Y5h717dZPTGG6F0806XC557Dt580+JdVzs+phUUw55/75N9BYUQQlSZowsrX19fTw/BUbw+z3btYP58vYFzYKDeqbl7d71EsNTYsTDvO4vZYW15is7ctqsz+/ZZtTYkr8/UZiRP8yRTsyRP8+yWqaPnWAlRoU2bdN+F5cv19+PGwYsv6onvwPr1cOmlsGuXXjk49/Nimuw9QNTF9pokKYQQovbJHKtSSUlJnh6Cozgqz06dYOlSff/P3x/efBN69ICFCwHo1k2vGIyPh90pJfyYsI61l67jz/f/NDoMR2VqA5KneZKpWZKneXbL1NGFVXZ2tqeH4CiOy9PHBx56CFauhF69ICUFzj0Xxo+HI0eIiYFFi2DYFS5+LY7AUrDp5k2kvZpW6aGrynGZepjkaZ5kapbkaZ7dMnV0YSVElZRdnnr8cb3XzcsvQ8+e8MsvBAfD559DzITWvEZ7ALbctYVpPN3MAAAgAElEQVSUf+z07JiFEELYkqPnWB06dIjQ0NBaf536ol7kmZSkVwtu2KBXE44fr7fKCQnh9ddh3p1/cK/ajAto9mArYp9vi2VVf3J7vci0Dkme5kmmZkme5tVFpjLHqlR6erqnh+Ao9SLP3r31rcFJk3QPhmnT9Nyr+fO5/Xa4fW4zpgZ0phj444VdbH29ZpnUi0zrkORpnmRqluRpnt0ydXRhtXv3bk8PwVHqTZ4BAfDss/Drr7qo2r4dzjsPbruNi/tn80xiE6ZHduMnorn0pWg2b67+S9WbTOuI5GmeZGqW5Gme3TJ1dGElRI307g2//QZPPgl+fvDGG9C1K3F/zOP1tY2Y26sLW7a76NsXFn1XRElBiadHLIQQwsMcXVjFxsZ6egiOUi/z9PeHRx/Vc6/69IHUVLjkEpo/PJrFczK57DI4klXM2kvW8X2f9RQfKXbr8PUy01okeZonmZoleZpnt0wdXVgFBAR4egiOUq/z7NYNfvkFXnhBd23/8ENCz+rCnJu+ZML1ebRQRwham8k3ndZQkFFY5cPW60xrgeRpnmRqluRpnt0ydXRhtX79ek8PwVHqfZ6+vvDAA3rPwYEDIT0dn6v+yjP5N5M9vhnpBNAgNZu57VeRvTWvSoes95kaJnmaJ5maJXmaZ7dMHV1YCVErYmN1h/YZMyAkBGbPZvTHfeg4diMpBBFx8AgLuq7ij18Oe3qkQggh6pijC6vo6GhPD8FRJM9juFxw5516U8ELLoCMDHq9NYYr+0whzfcQDQryWTVwFRt/OnLaw0imZkme5kmmZkme5tktU0c3CC0qKrLdrtfeTPKsgFLw/vswYQJkZVESGMQv6nZ+zL+RVyN7MOdri/79T/1UydQsydM8ydQsydO8ushUGoSWWrp0qaeH4CiSZwUsC26+GTZuhGuvxZWXy4D8qdwQdhOtM1dy3nkw6+NTt2KQTM2SPM2TTM2SPM2zW6aOLqyEqFPR0TBzJsybB23a0D5nNb9aCbyQP56cG37mgwt2UlJS91eIhRBC1B1HF1Z2W4Lp7STPKrroIj33asIELAvu5mVuYAS9fvyEtztvJu/w0atXkqlZkqd5kqlZkqd5dsvU0XOshPC4pCQYO1b/G9jLOfzQ8BGGrh5Mo9Yyz0IIIbyB8TlWlmU1sSxryWl+7mdZ1n8ty/rZsqwxVR1obZPizSzJsxp694bERHjpJYr9g4lmIVcfuIKtHR5jy5Ijkqlhkqd5kqlZkqd5dsu00r8yW5YVAXwAhJzmYXcDK5VSj1uW9a1lWbOVUjmmBumuHVk7+H7b92zevJkV2CtwbyZ51sCgYMJmP0y/ybNpt34NfYueZd+guSy/637i46v0lyBRBYcOHfL0EBxHMjVL8jTPbplW5V5EMTAS+Oo0jzkHmFj69WIgHlhQo5HVwJr0Ndw+93b9zRZPjcKhJM+aGQ43tYjg1e8UjVnDqBm3sGnXejrNnAKhoZ4enRBCiBqqtLBSSmUDWJZ1uoeFAGmlX2cCTU58gGVZ44BxAM2aNWPhwoUAtGvXjrCwMNasWQNAVFQUXbt2ZfHixXqAvr4MGDCApKQksrOzAYiPjyc9PZ3du3cDegPGgICA8rb2xf7FjO01lj179gDg4+NDkyZN2LdvH4WFeh+36OhoDh8+zOHDujt2w4YNsSyLrKwsAIKDgwkLCyM9Pb18HNHR0ezdu5eioiIAmjRpQk5ODkeO6CaQERERKKU4cOCADiUkhJCQEPbu3QuAn58fjRs3Jj09neJivVlvTEwMBw8eJDc3F4DIyEiKi4s5ePAgAKGhoQQFBbFv3z4A/P39adSoEXv27KGkRE+Cbtq0KVlZWeTl5ZVnWFhYWJ5XWFgYAQEB7N+/H9AT/aKiovjjjz/KP59mzZqRkZFBfn4+AI0aNSI/P5+cHH3hMTw8HJfLVf7eAgMDiYiI4M8//wTA5XIRExPD/v37KSgoAKBx48bk5uaW/22iQYMG+Pj4kJmZCUBQUBANGjSod59TRrM8Lj4rlWtfDWdsxmI6ff0CmTEfU/jiYwRdM4qkVavKxxkXF8eiRYtQSmFZFoMHD2bNmjXl7793795kZmaSkpIC1M7vU3R0NB07dixf0hwQEEC/fv1YsWJF+WebkJBAamoqaWn6PwOdOnXCx8eH5OTk8vzatm3LsmXLyj/7hIQEEhMTyzPt168fO3bsKD8funTpQnFxMZs2bQKgefPmtGjRgsTExPLM4+PjWbZsWfl5e+aZZ5KcnFz+WXbr1o38/Hy2bNF/I2jZsiVNmjQpv3UQHh5O7969Wbp0afn5MmjQIDZs2EBGRgYAcXFx5OTksH37dgDatGlDZGQkSaXz5uRzcv9zGjBgAJs3b66Xn1NkZCRFRUXyORn8nFq0aEFWVlat/j65o8qT1y3LWqiUOqeCn30F/E0ptceyrPuBPUqpTyo6Vl1NXt+yZYvtdr32ZpKnOe8kvcOtX9/KqJ8f4+8/fk0vdDFVctEluF59Bdq18/AIvZOco+ZJpmZJnubVRaaeaBC6EhhQ+nUckGLouDVSVu0LMyRPcyzLAgsODE9k16Tv+Z3xFBGC67tvUV26wtNPQ+nfGkXVyTlqnmRqluRpnt0ydbuwsixriGVZd53wxx8AT1iWNR3oAiSaGJwQTmVx9Nb65c82Juu1x/mJT0jnfKz8PHj0UejRA3780YOjFEII4a4qF1ZltwGVUvOVUjNO+NlO4ALgZ+B8pVSxyUFWV6dOnTw9BEeRPM0pm7MY3iAcgLNvb0jbhRfwY8CzrGYqObSGzZv1Bs/XXAOlc9nE6ck5ap5kapbkaZ7dMjXWeV0p9YdS6jOl1EFTx6wpHx8fTw/BUSTP2tVxcBBDN/cisdH5JPEWW7mVIr8gmDULOnWCl1+G0gmh4tTkHDVPMjVL8jTPbpk6ekubshUUwgzJ05yyW4FZB7OO+/NGrXy5aXc31nRvz6s8SGzhBrZ1GQY5OTB+vG44WrqiVpxMzlHzJFOzJE/z7JapowsrIeyqvH3JKRblBgRa3L2mHY2fjiWFtnRI/pr3B39GScvWsG4dnHsujBwJpcuEhRBC2IejC6uYmBhPD8FRJE9zyq5YBQYFnvrnFjw82eKTTyDSrxDXolbM3fs+OWMmQ1AQfPYZnHEGPPMMlPYvE3KO1gbJ1CzJ0zy7Zerowqpt27aeHoKjSJ7mlF2xCg4JPu3jrrkG5nxWQq6vH2H5sPy9C9hy3yIYMQKOHIFHHoGuXeHrr8EDG6rbjZyj5kmmZkme5tktU0cXVmVdaYUZkqc5ZVes9u3fV+ljB14RwJDVPVnYoBl+SpH27GF+UU9T8t0Puqjavh0uvxwuuQRKuyrXV3KOmieZmiV5mme3TB1dWAlhV6ebY3UqsV1d3L2tI3M6dqIAi4LP/2DBvZHkz10O06dDgwbw3XfQvTv8/e96srsQQog65+jCKigoyNNDcBTJ05yyK1buLBOOioLn1zZl3sW9SCcAn43Z/HvKYbjnHt3z6tZbdTuGF17Q7Rk++ghK9ymsL+QcNU8yNUvyNM9umTq6sEpISPD0EBxF8jSn7IpVZFSkW88LCICpc8PZ+sCZ/B+xjP6wCePHQ3FUNLz1FiQmQkKCbih6443Qty/8/HNtvAVbknPUPMnULMnTPLtl6ujCqmzXbmGG5Gne/oz9bj/HsmDyC/5c9WFz/Px039CxQw6xdmQyRR17wS+/wPvvQ9Om8NtvMGCAbs9QuhO8k8k5ap5kapbkaZ7dMnV0YZWbm+vpITiK5GlO2a3AouLqd1K/4Qb44QeIaKg4a/EmMj/bS2LcCnJWH4bRo/XtwUcfhcDAo+0ZJk2C7GxTb8N25Bw1TzI1S/I0z26ZOrqwEsKu3J28XpHBg2HZcosPW3VmC6EU7sxjZd8k0l5LQ4WEwJNP6gLruusgPx+eew46doS334ZiW2zpKYQQjuLowqpfv36eHoKjSJ7mlF2xcneO1al06gRfrwzm4769+IpmUKjYcucWkq9Jpii7CFq2hI8/huXL9Zyr9HQYOxbOPBPmz6/x69uJnKPmSaZmSZ7m2S1TRxdWO3bs8PQQHEXyNKfsitWhw4eMHK9RI/h+gQ9/juzIU3TmCD7s+/c+Vp+7GlVSelksIUHPv/r0U11srVkD552ne2Bt2WJkHJ4m56h5kqlZkqd5dsvU0YXVnj17PD0ER5E8zSm7YpVncDuawED45BPoN7kJt3Em2whhcUxLVOlr6Re2YNQo3Uj06achJER3be/SBe69F/a7P5neTuQcNU8yNUvyNM9umTq6sBLCrsquWCnD29C4XLpeeuLdYO70OZMJ3zbhqqv07jcZ8zIozCzUDwwKgsmT9ZWqMWP0fKvp06F9ez0Py2aTQYUQwls4urDq0qWLp4fgKJKnOWVXrELDQmvl+DffDN/+z0WDBvDllzDqrEOsv3I9K+JWcGDRgaMPbNoU3nkHVq2CoUP1isFJk/QE9/fe87oJ7nKOmieZmiV5mme3TB1dWBV72f8U7E7yNKe2rlgda8gQWLYM2rSB3zb4sEWFkp+az+pzV7Pj0R2UFB7TlT0uTm+J87//Qc+ekJqqr2T17Anz5nnNBs9yjponmZoleZpnt0wdXVhtqucb0pomeZpTdsUq51Dt7unXubNeDNg6IYjbCnrxmX8rFLDz6Z2sHrSa3O0n3PK74AJYuVKvImzdGtav15s7n3++/nObk3PUPMnULMnTPLtl6ujCSggBTZrAggVw5VUuXi9oxwSrJ4URAWQvz2ZFzxVkzMs4/gkul+57tXEjvPgiRETotgzx8frPbbYCRwgh7MTRhVXz5s09PQRHkTzNKbsVGBAQUCevFxQE//43PPQQrCppyPCseNI7NkKVKII6VLCBaWAgTJgA27bBgw/qjQo/+UR3cL//fluuIJRz1DzJ1CzJ0zy7ZerowqpFixaeHoKjSJ7mlN0KDAism8IK9IWo556DN9+EIz5+jNrclY8GxmO1CAb0fK/sFafY7iYiAv75T92i4YYboLAQ/u//oF07eOIJW22RI+eoeZKpWZKneXbL1NGFld02ZvR2kqc5ZVessg5k1flrjx0L334LYWEWb38XzJAhsHcv/PnmnyT1SWLL3VsoPnKKyaCtW8OHH0JSElx8MeTkwOOP6wJr6lRbtGiQc9Q8ydQsydM8u2Xq6MJKCLuyjm3a6QEXXqibsLdqdXSnmz+2F2P5WqTNSGNFrxUcXH7w1E/u2VNXZosXw4ABkJGhbxnGxsJbb+krWkIIUU85urAKDa2dHkH1leRpTtkVK5eP534Fu3XTRVV8vJ6PfsGbLSl5tTch3ULI3ZzLqv6r2P7IdkoKSk59gIEDdXH17be62EpLg3HjdBf3Tz+FkgqeV4vkHDVPMjVL8jTPbpk6urCKj4/39BAcRfI0p+yKVXh4uEfH0bQpLFwIV1wBBw7AhXeGkXxPb1o+2BIU7HpmF0kJSeSlVrD1jmXp24IrV8KsWfqq1datcO210KsXfPNNnfbAknPUPMnULMnTPLtl6ujCatmyZZ4egqNInuaUXbE6cOBAJY+sfSEh8PnneqFfURHcNM6H9wPaE7ewJ4HtAlFK4R/tf/qDuFwwciQkJ+vbgS1awNq1MGyYvl24cGGdvBc5R82TTM2SPM2zW6aOLqzy8/M9PQRHkTzNKbtiVVxij47BPj7w0kvw6qtH9xu8818N6Z4YT7c53XD56/9UFOwrIGfVaZqa+vrCrbfqPQinToVGjfRkrnPP1a3glyyp1fch56h5kqlZkqd5dsvU0YWVEOL/2zvv8Liqow+/Z1e9ujdZ7r3gJtvI2MZAMDWEGiChhRAgAUKoIQRCbyEQWkLKR4AQILRAAiRgmouwcS+4Cxfce5NkSZa05/tjdtUsWyv7aHd1Ne/z3EfS7tl7R789ezWaM2emYfzsZ/D++5CWJmlSp5wdR1FaVZ2r/BvymTtyLqvvXE1FyWGcwqQkuOkmWL0a7r8fMjOlSun48VLF/csvI/DbKIqiRB5PO1Zjx46NtgmeQvV0R2gpMLNFZpQtOZjTT4e8PMjKEv8nNxdWrgRbYUnsmAgBWPfIOuYOm8ve6YfYORgiPR3uvhvWroV77oGMDPjsM1kenDhRmhk6ROeoe1RTt6ie7ok1TT3tWK1cuTLaJngK1dMdoaXAoqKiKFtSN0OGwMyZkn/+zTfiXE370tDr970YljeMlH4p7F++n/lj55N/Yz4VRfUsabZoITWv1q4VRys9HT75BMaMgVNPlYs5QOeoe1RTt6ie7ok1TT3tWG3bti3aJngK1dMdoYhV6YHYyg2oTlaWVFM44wzYtUtW8P7xD8gck8mI+SPocmcX8MHGZzYye8jsuouK1qZlS1kaXLsWfv1rWXP8+GMppHX66TB79lHZrHPUPaqpW1RP98Sapp52rBQlVol2gdBwSUuDf/8bbrhB6n5eeql0sfEl+unxUA9GzB5B6pBUWp/WGn+KP/wTt2olGfJr18KvfiVbE//3Pxg1Cs4886gdLEVRlGjhacdq0KBB0TbBU6ie7ghFrFJSU6JsSf34/fDMM/D007Jj8N574fLLobQU0oelM2L2CHo82qNy/K5PdrHx+Y3YQBj1q1q3hocflgqlt98OKSnw4YfiYJ1ySoN3EeocdY9q6hbV0z2xpqmnHatY24LZ1FE93VFZbqEiNsothMPPfw7vvSe+zyuviN+zaxf44n34UyVaVVFSwcprVpL/s3zmjZlH4cLC8E7eti089phEsG6/XUJlkybJLsLjj5d8rDAKjeocdY9q6hbV0z2xpqmnHav8/Pxom+ApVE93hCJWxTHQuLghfPe7EkTq2BGmTJGk9lWrqp73Jfro+XhPEjolUDCzgDkj5vDNrd9QXlge3gVCDta338ouwhYtJNFr4kTJw3r//cM6WDpH3aOaukX1dE+saeppx0pRYpVQxMoSuXYvrhg+XDbxDR4sZRiOPbaqLJUxhrbntWXUslFk3ZAFAdjwxAZmD5zNjv/swIbb3qZVK1lz/PZbeOQRKTQ6axacdZZsVXzrLWhC0T5FUZoPnnassrOzo22Cp1A93ZOYmBhtE46I7GypdXXqqbBjB5x0ErzxRtXzcRlx9H6mN8NnDSdteBql60pZ+bOVBEoa2Jg5IwPuuEOWCH//ewmVLVwI3/++dJF+5RXpw1Npl85R16imblE93RNrmnrasWrfvn20TfAUqqc7QkuBcfFxUbbkyMnIkJW5a6+VRPaLLpI89OpBqYycDIbPHE6vp3vR+5ne+JODuVhFFeEvD4LsGvzFL6SS+/PPQ9eusHw5XHYZ9OkDf/wjFBfrHG0EVFO3qJ7uiTVNPe1YzZkzJ9omeArV0x2hpcCCgsP03WsCxMWJT/PEE2CMlKb68Y/hwIGqMb44H51/3pm257atfGzt/WuZ1W8WW1/fGv7yIEirnGuvlV6EL74IvXvLjsLrroOuXdn+859LRr3iDP3cu0X1dE+saeppx0pRYpVQxKop5ljVxhi4+WZ45x1IThZ/57TTYM+eusfbCsu+6fs4sPEAy36wjAUTFlC4KMzdgyHi4+GKK2DZMnjzTRgxArZvp/uLL0KXLtKncN26o/7dFEVRGoqnHauMjIxom+ApVE93hCJWfn8DimrGOOecIzsF27eHzz+XbjVr1hw8zvgNQ6cMpe//9SW+bTx7p+5lzrA55N+QT9musoZd1O+HCy6QgqKffcbe3FwoKoKnnoKePWWp8Ouv3fyCzRT93LtF9XRPrGnqacdq+PDh0TbBU6ie7mhKBUIbwsiRsmNw4EAJJo0eXXcbQOMzdPxxR0atHEXWz7PAwMbnNjKz10xKNx1BTRpj4MQTyZw+HebPhx/8QJK9XnkFjjlG+vJMnRpWLSylJvq5d4vq6Z5Y09TTjlVeXl60TfAUqqc7QhGrffv2RdkS93TtKuUXTj4Ztm+HCRNkmbAu4lvE0/vp3uTMz6HFiS3IyM0gsdOR75TMy8uDoUPh1VclD+v662V98r//lUKjubnw7rtaqqEB6OfeLaqne2JNU087VuXlDdh1pNSL6umOUMQqYBtYfqCJkJkpnWmuugpKSuD88+Hxxw8dMEobnMaQT4cw4I0BlY8VzCtgwYkLKJgXfoJ/jTnavTs8+6zkWt1zj7TPmTkTzj0X+vaV5wobmNvVDNHPvVtUT/fEmqaedqwUJVZpKk2Yj4b4ePjLX+DRR+Xn22+XDX1lh0ijMsYQl1ZVfuLbB75lzxd7mJszl2VXLKN04xG2rWjTpqrY6LPPisO1apX06OncWQxbv/7Izq0oilIL06Ctzo7IycmxkdgeGQgE8PnUd3SF6umOWRtnMfr/RjOy00hm/WRWtM1pdN56Cy69VOpdTZwoG/kyMw//mrI9Zax7aB0bnt6ALbP4kn1k35ZN9q3ZxKXXXf8rrDlaUQH//rcUHA0tIfj9Ela7+WZpAK1Uop97t6ie7omEpsaYudbanHDGevrdXbJkSbRN8BSqp3v2798fbRMiwgUXwBdfSCvASZNg7Nj6qyHEt4in5+M9GbVsFG3Pb0ugOMC393/LzJ4z2f357jpfE9Yc9ftlOXDaNGmTc/HF8vgbb0i2/XHHwdtv16jo3pzRz71bVE/3xJqmnnasdu7cGW0TPIXq6Y7QUmBZeQPLCzRhcnPhq6+gXz9YvFh8mHAC18k9kxn41kCGThtKxpgMKooqSOlf927KBs/RkSPhtdekLsTtt0vT5+nTxRPs1QuefBL27m3YOT2Gfu7donq6J9Y09bRjpSixipcKhDaEHj3EbznhBNiyBcaPh/feC++1Lca2YFjeMHLm5ZDYUXYOBsoDLD5vMTveb0CD57rIzobHHpNcq+eeE6fq22/hllvkuRtvhBUrjvz8iqI0GzztWA0ZMiTaJngK1dMdoYhVcnJylC2JPC1bwkcfSeH04mJZlfv978MrMWWMIaVvVbRq22vb2PGvHSw+azHzx82nZ3HPozMuLU3a46xYIXlYEyZAQQE884yE2k45RRokNqNyDfq5d4vq6Z5Y09TTjlVT78MWa6ie7ghFrMormmceT0IC/O1v8OCD4lDdfLOUnGpoWlO7C9vR66lexLeJZ9+X+1h1+ioWnbmIgvlHOVd9PjjrLEkMW7BA6kYkJ0uC2FlnSUTr8cchxpYgGgP93LtF9XRPrGnqacdq9erV0TbBU6ie7ghFrEpLj7CEgAcINW1+7TVxtP74R/je9yRAFC6+RB+db+zM6FWj6XpPV0iGXR/uYu7wuay8fqUbQ4cMgb/+FTZsgN/9Tso1rF0rOVmdO4vTNX++m2vFIPq5d4vq6Z5Y09TTjpWixCrNNceqLi6+GD77TOp3/ve/MG6c+DANIS4jju73dodXofMtnfEl+UgbnFb5vJOyMq1aSc5Vfj588AGceqpUP33hBRg+XLY6/vOfcODA0V9LUZQmi6cdq27dukXbBE+herojFLFKSEiIsiWxwdixMGMG9O4NCxfKjsEjCQJ1G9aNXr/rxeg1o+nwow6Vj6+5ew1Lvr+EoiVFR2+s3y+9B//3P8nFuvFGyMiQPj4XXyw9fX7zm/rrSTQR9HPvFtXTPbGmqacdq1atWkXbBE+herojFLHy+/1RtiR26N1bnKtx42DTJvn6wQcNO0dojiZ2SMSXILe3QGmAzX/ezPa3tjN78GyWXLSEomUOHCyAPn3gqadg40Z4/nnpPr1lCzzwgCwZfve78ks04WR3/dy7RfV0T6xp6mnHat68edE2wVOonu7ZX9w8CoSGS+vW8Mkn8MMfQlGR5Fw991z4r69rjvoSfeQsyKHTdZ0w8Ybtb2xn9sDZLLlwCQULHCW9pqVJv56vv5aE94suksjWBx+Ic9W9O9x/vzhgTQz93LtF9XRPrGnqacdKUWKV0FJgNFpKxTqJifDKK9I3ORCAG26AX/zi6II+iVmJ9HmuD6O/GU2nn3bCxBm2v7mducPmUvi1w0bMxkiJhtdfFyfqt7+VHYTr18sv1LUrnH22LCM24SiWoiiHxtOOVcuWLaNtgqdQPd0RWgr0+T39ETxijJG+yS+/LM2cn34azjkHCuvxgeqbo0nZSfT5Yx9Grx5N5190puUpLWskuRfMK3Dn7LZtC7fdJnlYn34q1dyNkfpYp58OPXvCQw/B5s1urtdI6OfeLaqne2JNU083YVaUWGXJtiUMen4Q/dv0Z+l1S6NtTkwzZYo4Vbt3y+a799+HTp3cnNtaW+nkFi4qZM6QOaSPTKfLnV1oc1YbjM+4uVCIrVvhxRfhL3+RNjoAcXGyXPjjH0sB0ri6G0wrihI9tAlzkClTpkTbBE+heroj9Me8aL+jJGoPc/zxktTeowfMmyc7BhctqntsQ+do6H0AKFlbQnzbeApmF7DknCXMPmY2W/6+hcCBwNGYX5P27eGOO+Cbb+Djj6XsvLXw7rtw5pnQpQv86lew0lENLgfo594tqqd7Yk1TTztWmr/iFtXTHZpj1TD69pUGzmPGSI2rsWOlLU5tjkbPNme14di1x9LrmV4kdk5k/5L9LL98OV91/4oNTzewsFZ9+HwwcSK8847kXz36qOww3LxZvu/bV7ZFvvhi/eufjYzOUbeonu6JNU097VhV/29UOXpUT3eolg2nbVspJHrhhVKd/cwz4U9/qjnmaHX1p/jpfINUcu/7Yl9SBqZwYNMBd+UZ6qJjR/jlL2H5csjLgyuvhNTUqu87dJBlwi+/DK+homN0rrpF9XRPrGmqOVaKEgVW7FhBvz/0o0/rPqy4fkW0zWlSBAJw993w8MPy8y23yOY7XyP8m2itZfek3ST3Tia5hzTM3vbmNrb/azvZt2aTkZPh/qIgUaq33pKGinl5VY/36SPO1mWXiUOmKEpE0ByrIAsXLoy2CZ5C9XRH6D+skpKSKFvS9PD5ZDPdCy9InvcTT8D558P+/e7nqDGGVqe0qnSqAP6/d/UAACAASURBVNY/uZ7tb2xn3sh5zJ8wnx3v78AGHP+DmpYGP/oRTJsmkaxf/lIiVytXSo5W587SUufVV6XgVyOin3u3qJ7uiTVNw3KsjDEvGGNmGGPuOsTzccaYdcaYycFjsFszj4zdu3dH2wRPoXq6pyKgtYyOlCuvlDyrzEzJ/Z4wAVatavzNAAPfGkj2rdn40/3snbKXxWctZmafmax/aj3le8vdX7BvX8m7Wr9etkSec44UH/34Y7jkEnG4Lr9c1kkboTaWfu7donq6J9Y0rdexMsacC/ittblAD2NM7zqGHQO8bq2dEDy+dm2oongJTV53w0knwfTp0K0bzJ4NP/vZcJYsadxrJmUn0fPxnuSuz6Xn73qS2DWRklUlrLppFVtf39p4F46Lk8Syf/1Lktz/8Ac49lhZNvz73+E735ECpL/8JSxe3Hh2KIpyWOrNsTLGPAN8ZK39rzHmIiDZWvtirTE/A64DioCvgWusteW1xlwNXA3QqVOnEa+++ioAPXr0ID09vTKU17p1awYOHMjUqVMBiIuLY+zYscybN499+/YBkJOTw9atW1m/fj0AvXv3JjExkcXBm0m7du3o06cPU6dOxefzkZiYSG5uLnPmzKEwuMNm9OjRbNiwgY3BFhN9+/bF7/ezdKnUFOrQoQPdu3dnxowZACQnJzN69GhmzpxJcXExALm5uaxZs4YtW7YAMGDAACoqKlixQnJmsrKy6Ny5MzNnzgQgLS2NnJwcZsyYQWlpKQBjx45l5cqVbNu2DYBBgwZRWlpKfn4+ANnZ2bRv355QTlpGRgbDhw8nLy+P8nKRePz48SxZsoSdO3cCMGTIEAoKCli9ejUgDSpbtWpVWfa/ZcuWDBkyhClTplTW8Tn++ONZuHBhpec/fPhwdu3axdq1ayvfJ5/PxzfffOP8fcoL5pA0p/dpY/FGLpl1Cdmp2fw95+9O36fG+jzF8vu0eXOAX/96MMuXZ5CWVsG99y5mxIjdEfk8tchoQda6LJY8vgTuApMi79Ose2exP2E/jIbhIxvvfVr10Ue0nzSJTl98QcKGqt2LRb17k3rttSwYMIA9SUlH/D7179+fZcuWxfTnKZz3KVY+T5mZmQwePDimP09N7X3q2LEj7dq1a9T73gknnBB2jlU4jtULwDPW2oXGmInAcGvto7XGjAQ2WGs3G2P+Drxtrf3Poc4ZqeT1tWvXxlzX66aM6umOVbtW0evZXmSnZbPulnXRNscT7N8P551XxEcfpRIXJz2Rr7oqOrZUFFUwo/MMyveUk9Q9iazrsuhwZQfiW8Y33kWtlZ2Dr7wCb74Je/bI46HSDj/8oTRfTE9v0Gn1c+8W1dM9kdDUdfJ6IRDK3Ew7xGsWWWtDfRnmAHUtF0ackDeruEH1dEcoeb2svCzKlniHlBS47bbZ3H47lJfDT34itTYDDut7NoSud3UlqXsSJWtKWHXrKmZ0msGyK5axd8bexlkCNkYKfP35z7JU+NZbcNZZ4lh99BFceim0ayetdf71LwhGNupDP/duUT3dE2uahuNYzQXGBr8fAqytY8wrxpghxhg/cDYQWyn6ihJjaI5V4+DzwWOPiW/h90vO90UXhe1DOMOf6if7lmxG549m0H8G0fLklgRKAmx9eSvzx8yncGEjF/1MSpKtkv/+tzhZzz0nTldJCbz9Npx3nlSBv+wyaQhdpg6+orgiHMfqPeBSY8yTwPeBJcaYB2uNuR94BVgAzLDWfurWzCOjR48e0TbBU6ie7ghFrOK0L5xTQnP06qvhv/+VVa+33oITT4RgmkhEMX5Dm++2YcikIYzKH0X27dm0nNiStCFVjZ/XPb6OfXP2NZ4RbdrAdddJ6YZ16+Dxx2HECKmy+sor0hC6Qwe45hqYPPmgnYX6uXeL6umeWNM0rAKhxpiWwMnAVGvtlqO9aKRyrHbv3h1zXa+bMqqnO77d8y3dnu5GVloWG25x3C6lGVN7jn79NZxxhlQq6N4dPvwQ+vePooG12L9iP7P6zQIgbXgana7pRLuL2xGXHgGHOz8f/vlPeP11WLas6vGOHeH734eLL4ZRo9i9Z49+7h2i91H3REJT5wVCrbW7rbVvunCqIkmsFQ1r6qie7qjMsdIlGKfUnqODB8PMmRKgWbNGeg1+8UWUjKsDf5qfzjd3Jq5VHIXzCll5zUqmd5zO8h8tZ8+0PY27VNy7t5SwX7JEulrfead0ut68GZ5+Wko5dOtG4dVXS1J8tJLVPIbeR90Ta5p6uvK6osQqlTlWaI5VY9OxI0yZIhvi9uyRDXIvvxxtq4TErER6PdGL3I259P9HfzLHZRIoCrDlpS0sOmURFQURKCBrjHigDz0E33wjnuhNN0FWFqxbR/bbb0t+VufOcP31dS4XKopShacdq9atW0fbBE+heron1pqHNnUONUdTU+Gdd8RfKC+HK66A3/wmKj2N68Sf5Kf9D9szbOowRq0YRZdfdSHruiziMmRJMHAgwJILlrDtjW1UlDSiU2MMjBoFTz4p+VjTp7P9ssuk8GioKOkJJ0CnTpKT9cknmvjeQPQ+6p5Y09TTTZgDgQC+xujM2kxRPd2xYd8Gsn+fTaf0Tmy8eWO0zfEM4czRP/wBfv5zWdn6wQ+k52CwfmbMsv2d7Sw5X0rKx7WMo93F7ejwow6kj0hvdOc8EAjgMwbmzhXv9O23JbIVolUrCQeef76Uwk9MbFR7mjp6H3VPJDTVJsxBQlVXFTeonu4ILQWGKhwrbghnjl53nbTcS0uD116Dk0+GHTsiYNxR0OKEFvR6thdpw9Mo313Opj9uYt7IeczqP4u1D6wlUNZ4+U9Tp06VSFZODjzyiDSCXrhQ8rP694ddu+DFF2WXQLt2Ut/i9derCpQqNdD7qHtiTVNPO1aKEquEogyaYxUdTj9dqg9kZUFeHuTmyia5WCW+VTydr+9MztwcchbkkHVjFvFt4yleUcy2f27DxFVFrRqlEXR1jIFjjoH774elSyX5/f775bF9++CNNyQU2LateK3PPSfLiorSTPC0Y6U1gtyierojFLFS3NKQOTp0qORpDx0qK1vHHivOVqyTNiSN3k/1JndTLoP/N5gej/SodNSLVxXzZdsv+fq7X7P1n1up2H/0+Vj1ajpggESvFi6EVaskP+v442Wt9dNP4YYbJEdr+HC47z5YsCB2ktuigN5H3RNrmno6x0pRYpUthVvo+ERH2qW2Y+utW6NtTrOmsFBWrz78EBIS4G9/k7Z6TZGtr25l2eXLIOhP+dP8tDmnDe1+0I6WJ7XEFx/B/6V37hRR//1v+PhjKCqqeq5LF2m3873viRMW34g9FBXFAZpjFSTULVtxg+rpjlDESutYueVI5mhaGrz3nlQSOHAALrlEVraaYlCl/Q/bM2bTGHo904v00elUFFaw9ZWtfH3a18zsMRNb0fBf6og/961bS8ucd96RJLYPPpAGjh06yNLgc8/JUmHbtnDhhVIDIxrl8SOM3kfdE2uaetqx2revEdtENENUT3dU5lg1xb/eMcyRztG4OHj2WamLaQzccw9cfjk0xb0FCe0S6HxDZ0Z8NYJR+aPodl83UgamkDEmA+OXeRcoDbDyupXs+mQXgfLDJ747+dwnJUly+1/+Ahs3wowZcMcdkvy+dy+8+abUwOjQQco93HcfzJ7tyaKkeh91T6xp6mnHSlFiFS0QGpv8/OcSvUpJkTZ6p5wim96aKim9Uuj2m26MWjyKfi/2q3x81ye72PTHTSyauIgZHWew4poV7P5sd71OlhN8Pkloe+QRSX7/5ht45hk49VRZi509G+69Vxysjh3F4XrrLd1lqDQZPJ1jVVhYSFpaWv0DlbBQPd2xvWg77X7XjlZJrdj5y53RNsczuJqjc+fCd78rNTH79pVUoZ49HRgYIxSvKWbLS1vY9sY2ilcUVz4e3zae1me1ps8f+uBLlP+7I/q5LyqSnkMffijH+vVVz/n9UgH+9NPlGDhQwotNDL2PuicSmmqOVZCtWzUp2CWqpztCS4EBDy51RBNXc3TECNkxOHgwrFghAZbp052cOiZI7p5M9/u6M2rZKHIW5dD1rq4k90mmbHsZ+6bvq3SqANa+tpayPRHKBUxNhTPPhOefh2+/lS7ajz0G48fL81OmwC9/KW9MdjZceaU0ko71QmTV0Puoe2JNU087Vuur/7ejHDWqpztCS4EVAe255hKXczQ7W2pcnXKK/N0+8UQp0eQljDGkDU6j+wPdGbVcnKzez/WufL54TTE7rtnB9LbTWThxIRuf30jppgglnhkDgwbB7beLQ7Vjh+RiXX65FCLduFEKk158sfyckyONpCdPll0IMYreR90Ta5p62rFSlFhFC4Q2DTIyZDPbNddIIvtFF8HDDzfNHYP1EXKyWp7YsvKx8j3lMBRswLL7k93k/yyfGVkzmHvsXL599FvK9zVyMdLqtGgBF1wAL70ka7QLFsBvfwvf+Y7kZs2dK3lbJ5wgbXbOPFNyt5Yv9+YbpsQsns6x2rhxI1lZWY1+neaC6umO3cW7afXbVmQkZLD3V3ujbY5naKw5aq3UvbztNvn+yivhT39qHuWXNm7cSNvEtuz8YCc73tvB7o93EygJ4EvycdzO4/Cn+AEomF9ASv8U/En+yBu5f79Ud500SY7Fi2s+n50NEydKeYcTTpAIV5TQ+6h7IqFpQ3KsYqtcqWMStRmoU1RPd2jEqnForDlqDNxyC3TvLnWu/vY3SQF6+20JpHiZxMREEtok0PGKjnS8oiMVRRXsmrSLkjUllU5VoCzAggkLsGWWlie1pPWZrWl1eiuSsiPU3TolRdZsTzlFft64Uaq+T5oEn3wiSfAvvCAHSI7WiSeKk3X88RF9E/U+6p5Y09TTS4GLa//XohwVqqc7KnOsKjTHyiWNPUfPPVdSeNq1g88+gzFjYM2aRr1k1KmtqT/VT9tz2pJ9c3blYwc2HSC5ZzKB4gA7P9jJymtX8lWXr5g9ZDar71xNybqSyBqdlSW5WK++Clu2wLx58OijcNJJUlPr66+laNnZZ0sh05EjJSn+o4+kFH8jovdR98Sapp52rBQlVjFNcJu4IowaJTsGBwyAZctkx+DMmdG2KrokdU0iZ14OuRty6fOXPrT+Xmt8qT6KFhWx7pF1VBRU/QNRMLeAkvURdLR8Phg2TBynTz+VeliTJ8NvfgPjxkkZhzlzJF/rtNOgZUsp63D33VL6oSTCTqHS5PH0UmC7KK6jexHV0x2VTZjVv3JKpOZot27w5ZeSS/3ppzBhAvzjH3DeeRG5fERpiKaJWYl0+kknOv2kE4HSAHum7GHvtL2kDEipHLPimhUUzi0kpX8KLSe2pNUprWgxvgX+1AjlZiUmyvLf8cdLhfeiInkzv/gCPv9cnKwvv5TjwQdlfG6ulHwYN06+T0094svrfdQ9saapp5PXy8vLY67rdVNG9XRHQWkBGY9mkBqfSuGdjbv00JyI9BwtK4Of/lRSd4yRkku33tok61YeEpeaBsoDLL1gKbs/3U1FYVUUyyQYMsdmkn1bNq1Pbe3kWkfM3r2SCP/553IsXFjzeb9fCp2NGyfO1tixsgsxTPQ+6p5IaKoFQoPk5eVF2wRPoXq6I7QUqDlWbon0HI2Ph7/+VXb5Wysll669Vhwur+BSU1+cj0HvDuK4nccxdMpQuvy6C+kj07Fllj2f76mxZLhn2h42/d8milcXR7anZmamlGp48kkp6bB9O7z7Ltx0k9TKshZmzYInnoDvfU9ytAYPhp/9TIqVbtx42NPrfdQ9saapus2KEgW0V6B3MEb6CffoAZddJn2G166VWpaZmdG2LjbxJfhoMb4FLca3gAehbGcZuz/bTcuTq2pobX5hM1tfloraiV0SaXliS1qc0IIWJ7SI3G5DgDZtJMn97LPl54ICaSI9dapEtmbOlPIOixdLxXiQyTBunBxjxkhfJJ+n4xhKNTztWMXaFsymjurpDi230DhEc45+//tSLumss2SX/9ix0u6uS5eomeSESGga3zqedt+vmSfT6uRWVBRUsGfyHkrXlbLlpS1seWkLAO0uaseA1wc0ul11kp4uNbEmTpSfS0ulcfS0aeJsffklrF4tx8svy5gWLWSXw5gxtMvMlKXE9PTo2O9BYu1vk6dzrBQlVikuKybl4RQS/YmU3KW7jrzE6tVwxhlS8LtDB6ncPmJEtK1qutiApXBRIXu+2MOez/ewZ+oeOv+iM93v6w5AwbwCll2yjBYntCBzXCYtxrUgMSuKf2grKiQva+pUaTA5ffrBy4M+n7TrGTNGkuHHjJEu315KzvMYDcmx8rRjNWfOHHJywtJBCQPV0x0l5SUkP5RMvC+eA3fHbl+zpkaszNHdu6tqXqWkwGuvSTpOUyRWNA0RKA8QKAkQlyYLLuufWM+qW1fVGJPULYnMcZlkjs2kwxUd8CVEeRlu/XpxsGbMoOiTT0hduRLKa7UDatOmysnKzRVvPC0tOvY2MSIxR7XyepDCRi701txQPd1RmWOlPcycEitztGVL+PhjuPpqWQ065xzJhb7xxqYXlIgVTUP44nz40qocpawbskgfnc7eqXvZO20ve6fvpWRtCSVrS9j54U46XtWxcuzWV7eS3CeZtKFp+OIj6GxlZ8OFF8KFFzJ78mQmjB4tZR1mzKh0uNi2Dd5/Xw6QqNaAAVK8dNQo+Tp4sPRFVGoQa3PU046VosQqmmPlfRIS4MUXoVcvqTV5003wzTfw1FOgu+3d4Uvw0WJsC1qMlbY0tkKWDvfm7SVQHMD4gjtwiypYfsVybLnFl+oj49gMMo/LJOPYDDJGZxDfKoKNH5OTq5LbQXYarl5d5Wh99ZVUhw8lxb/4ooxLTIShQ8XRCjlbvXtrYnyM4emlwOLiYpKTkxv9Os0F1dMdZRVlJDyYQJwvjrK7PbQ3P8rE6hx97TX40Y/gwAE4/XTZld9UcpdjVdOGUrqllDW/XsPevL0Uryw+6PlB7w+izZltACjbU4Y/zY8vzr3DEraexcVS7mH2bCnvMHs2rFx58LjMTCkDEXK0Ro6Ulj5NLTR6FERijmqOVZD8/Hx69+7d6NdpLqie7igPlBP/QDx+46f8N+X1v0AJi1ieo3l5smN/504YMkSS2jt3jrZV9RPLmh4pB7YeYG/eXvZ9tY+9M/ZSOLeQUStGkdRFyjis+MkKtr6+lYxRGRLRCh4J7Y5+Ge6o9Ny9G+bOrXK0Zs2CTZsOHte2LQwfLq18hg+Xo0cPzzpbkZijmmMVZOPGjZ67IUQT1dM9mmPlllieo2PHykrPGWfIprHRo8W5GjYs2pYdnljW9EhJaJ9A2/Pa0va8tgAEDgQw8VVOR+mGUgJFAdmJ+MWeyseTeiTR8ccd6Xpn1yO+9lHp2bIlfOc7clSdUJyskKM1d64UNf34YzlCZGTUdLSGDZP6Wh5Yl461Odr0FVWUJogWCG2e9O4tztXZZ0sEa9w4WRY888xoW9a8qb1r8Jj/HcOBrQfYN3Mf+77ax74Z+9g3ex8lq0so31sVYS6YW8CyS5eRnpNeeaQNTcOfEqG+hyDLfllZVQVMrYVvv4X582HePDnmz4fNm2HKFDlCJCdL+DTkcA0bBgMHQlIEC7B6EE8vBW7evJmOHTvWP1AJC9XTHQEbwH+/3HztPepcuaKpzNHSUvjxj+HVVyXv+Omn4frro21V3TQVTRubQHmAosVFxGXEkdxD8nk2/mkj+T/NrznQB6kDU0nPSafX072IS68Zv4ianps3i4NV3eFau/bgcX4/9OkDxxxT88jOjtmlxEhoqkuBQfz+CP7X0AxQPd0RilgpbmkqczQxEV55RXYM3ncf3HADrFoFv/ud/F2LJZqKpo2NL85H+tCaOw46XNGB9BHpFMwpqDyKlhRR9HURJetK6Pt/fSvHLv3BUkyCgR6QNC6JtCFpkd2J2LGjHKefXvXYrl2SIF89srVyJSxbJscbb1SNzcys6WgNHixFTmNgF0aszVFPO1ZLly6lXbt29Q9UwkL1dIep9p+ftbbGz8qR05TmqDFw772SU3zVVVKGYfVq2UGYmhpt66poSppGGn+Sn4yRGWSMzKh8rGJ/BYULCzmw6UBlqYdAWYDt72zHHpDo9FaqeiCmDU0j67osWk1sFflfoFUrOPFEOUIUF4tTtWhR1bFwIezYIW17pk2reY4ePWo6WwMGyJp3fOScxlibo552rBSlKWCxGsFqxlx2mfQTPOcc+M9/YPx4qRHZqVO0LVOOBH+Kn8zcmt23jc8wdMpQChcUkv/ffNK3pVO0qIjSdaWUriul3YVVTsHWV7ey6c+bSBuaRtrQNFKPSSV1QGrk8raSk6sS3ENYC1u31nS2Fi2CpUur+iK+917V+Lg4WU4cOFAcrQED5PvevZtFgVNPO1YdOnSItgmeQvV0i8FgsbIzUP0qJzTVOTphQtWOwXnzpF/vBx9IECDaNFVNYwnjN2Qem0nmsZkUTCigX79+2ArL/vz9FC4opMX4FpVj904PVpCftrfaCSC5ZzIZx2XQ/6X+lQ8HygONUmvr4F/ASOPLDh2qmk8DlJXBihVVUa0lS8TZWrNGvi5dWvM8cXHiXIUcrZDT1aePrI8fIbE2Rz2dvF5aWhpzXa+bMqqnW/z3+wnYAOV3l+P3xVaOQFOlqc/R7dulp+CMGZK68tZbcMop0bWpqWsaa9Sn54FtByicX0jhgkIK5hdQtLiI4hXF2HJL5thMhk2T+hyBsgB5LfNI6pZE6qDUGkdy92SMP4r/rRUVSRfypUurnK0lS8Thqsvn8Psl4XDAACkB0a+ffO3bV0pM1EMk5qgWCA0yefJkJkyY0OjXaS6onm6Juz+OCltB2d1lxPk8HTyOGF6YoyUlcMUVkjfs98Mf/gDXXBM9e7ygaSxxJHoGDgTYv3I/gZIAGTmSz7U/fz+z+s6irootvmQfA98ZSOvTWgNQvKqYQGmA5F7J0W1IvX+/RLiqO1tLl8rOjUP5Im3b1nS0Qo5X9+6VNbgiMUd1V6CiNCG0SKhSnaQkSWDv2RMefhiuvVZ6DD72mLaEa674EnykDUqr8VhK7xTGFYyjaFkRRYtrHgc2HiCxc1UEZ/2T69n0x03gh+QeyaT0SyGlfwop/VJIOyaN9BER2tmXkiK1smpXxS0uFodr2TL5uny5fF25UsK427cfnDQfHy8fkr596ZGSIuvnMVJ/y9OOlRf6W8USqqdbjDFgtUioS7wyR30+eOgh+btxzTVShmH1ainRkJISWVu8omms4FJPf6qfjJyMyihWiLLdZfjTq9IL4lrEkdQziZI1JRTnF1OcX8zO93cCkDk+k2FTgsuLpQHyb8wXx6tvCsm9kknqloQvvpE9+uRkaS49dGjNxwMBqSwfcrSqH+vWyePLl5OVnCxF4WIETy8FKkosk/BAAmWBMkp+XUJinOawKHXz2Wdw3nmwd6/02f3Pf6B9+2hbpTRFKkoqKM4vZv/y/XIs20/q4FS6/kpa9BQtKWL2oNk1X+SHpG5JpPROocdjPUg7RiJn5fvK8aX4IpM8XxdFRZCfL07Wnj2Nvl7ekKVATweWZ86cGW0TPIXq6ZZQ7SqNWLnDi3P0pJNg+nTo1k1awY0effBmq8bEi5pGk2jq6U/ykzY4jXYXtKPb3d0Y8NqASqcKIL5NPL2e6kWnazvR4qQWJHZJhACUrCph10e7angMq25bxbTkaczsM5NFpy8i/8Z8Njy7gZ0f7aTk25LG/2VSUyXCdeGFzKwd6Yoynl4KLC4ujrYJnkL1dEtlv0DNsXKGV+fogAHw1Vdw1lniXI0ZA2+/XbMXb2PhVU2jRSzrmdA+gc43dq7xWEVJBSWrZQkxpXfVOnT5nnJsua1cWuR/Va9pcVILhn4qzk55YTnf/OIbknskk9Q9iaTuSST3SCa+bbyzwsixpqmnHStFiWU0YqU0hPbt4Ysv4NJL4V//gtNOgz/9SXoOKkpj4U/ykzpAipRWZ+AbA6l4SZyu/fn7Kf6muNLJql4gtWRNCVte2HLQeX0pPpK6J9HvpX5VOx2DOx+Tuicd1GOxKeHpHCutv+IW1dMtKQ+lUFxeTOGvCklNiKEeJk2Y5jBHAwG44w54/HH5+Y47JNG9sXYMNgdNI0lz0/PAtgNsf2c7JWtKJHl+dTEla0oo310OwMhlI0ntJ/e/5VcuZ8uL4oTFtY4jqVsSSV3lSBueRodLqgqBVm8FFmt1rJquSxgGa9asoV+/ftE2wzOonm7RiJV7msMc9fngt7+VHYPXXQePPio7Bl96STZXuaY5aBpJmpueCe0SyPpp1kGPl+8tp3hNMck9qyZtQvsEUvqlULK2hPKd5RTuLKRwbiEArU5tVelYle0pY0bnGSR1SSKxSyIJjyXQf0j/g64RLTydvL5ly8HhR+XIUT3dov0B3dOc5ug118CHH0qF9jfflCT37dvdX6c5aRoJVE8hLjOO9KHpNUo59HikB6OWjWJc0ThyN+YybPow+r/enx6P9aD9ZVVbYUvXlRIoCrB/2X72Td/H1l1bo/ErHBJPR6wUpSmgyevKkXLKKfDll9JjcMYMqZH44YdSmFpRmirGZ0jslEhip8SDGloDpB2TxnG7j6N0XSllO8tYaBZGwcpD4+mI1YABA6JtgqdQPd2iS4HuaY5zdPBgmDkTRoyQJcHcXJg82d35m6OmjYnq6Yb4FvGkHZNGyxNaxpymnnasKioqom2Cp1A93aLlFtzTXOdox44wZYo0cN6zByZOhJdfdnPu5qppY6F6uifWNPW0Y7VixYpom+ApVE+3aMTKPc15jqamwjvvwE03QVmZNHK+555D97YNl+asaWOgeron1jT1tGOlKLGMRqwU1/j98OST8Nxzsnvw/vvhkkugtDTalilK88HTjlVW1sFbPJUjR/V0i0as3KNzVLjuOukpmJoKr70mFdp37Diyc6mmjxRiiAAAFhhJREFUblE93RNrmnrasercuXP9g5SwUT3dohEr9+gcreKMMyAvDzp1kq+5udKztqGopm5RPd0Ta5p62rHS5qFuUT3dohEr9+gcrcnQobJjcOhQ+OYbKccwbVrDzqGaukX1dE+saeppx0pRYhktEKpEgs6dYepUiWDt2iXLgq+9Fm2rFMW7eNqxSktLi7YJnkL1dEtlxEqXAp2hc7Ru0tPhvfck9+rAAfjhD+GBB8LbMaiaukX1dE+saerpJsyKEsu0fbwtO/bvYOutW2mX2i7a5ijNAGvhmWekJIO1cNll8Ne/QkJCtC1TlNimIU2YPR2xmjFjRrRN8BSqp1s0ed09OkcPjzFw443w7ruQkgJ//7u0xdm169CvUU3donq6J9Y09bRjVarFW5yierpFk9fdo3M0PL73Pcm76tBB2t+MGQOrVtU9VjV1i+rpnljT1NOOlaLEMhqxUqLJiBGyY3DwYFixQnYMTp8ebasUpenj6Ryr8vJy4uLiGv06zQXV0y0dn+jIlsItbLx5I53SO0XbHE+gc7Th7NsHF1wAkyZBYqL0GLzwwqrnVVO3qJ7uiYSmmmMVZOXKldE2wVOonm7RiJV7dI42nIwM+OADuOYaaX1z0UXwyCNVOwZVU7eonu6JNU097Vht27Yt2iZ4CtXTLZpj5R6do0dGfDw8/zw8/rgkuN95J/zkJ9LMWTV1i+rpnljTNCzHyhjzgjFmhjHmrqMZoyhKFVogVIkljIFbb4W334bkZHjhBTjtNCgs1GUrRWkI9X5ijDHnAn5rba4x5m/GmN7W2vyGjokGgwYNirYJnkL1dEsoYtXzmZ7qZLkkL9oGNH0Cl+XAP97hs8/a89kXx4KvJNomeYhjAdXTKXFDscXRNqKKcP4VmQC8Gfx+EjAWqO001TvGGHM1cDVAp06dmDx5MgA9evQgPT2dhQsXAtC6dWsGDhzI1KlTxcC4OMaOHcu8efPYt28fADk5OWzdupX169cD0Lt3bxITE1m8eDEA7dq1o0+fPsyfP5/4+HgSExPJzc1lzpw5FBYWAjB69Gg2bNjAxo0bAejbty9+v5+lS5cC0KFDB7p3715ZHyM5OZnRo0czc+ZMiovlHczNzWXNmjVs2bIFgAEDBlBRUcGKFSsA6bjduXPnyj5GaWlp5OTkMGPGjMrtoWPHjmXlypWVocxBgwZRWlpKfrBbanZ2Nu3btyeU7J+RkcHw4cPJy8ujvLwcgPHjx7NkyRJ27twJwJAhQygoKGD16tUAdOvWjVatWjFv3jwAWrZsyZAhQ5gyZQrWWowxHH/88SxcuJDdu3cDMHz4cHbt2sXatWsr36fi4uJKjV2+T3l58pewub1P47LG8fq+1zlQcQBFiSk6fglXjYY334HNIyCgUSslhjEBdu/e3Sh+ROjvU4PMqS9x1hjzAvCMtXahMWYiMNxa+2hDx1QnUrsCJ0+ezIQJExr9Os0F1dM9kz6fxPjx46NthmeYOnWq6ukQa+Gzz/IYO3ZstE3xDHl5qqdr8vLyOPPU7zTqNRqyKzCcf0MKgeTg92nUnZcVzhhFUWqR4EsgKS4p2mZ4BtXTPWlJcbRIU01doXq6Jy0ptiKq4ThAc5GlPYAhwNojHBNxsrOzo22Cp1A93aOaukX1dI9q6hbV0z2xpmk4bt57wDRjTCfgNOAiY8yD1tq7DjPmWPemNpz27dtH2wRPoXq6RzV1i+rpHtXULaqne2JN03ojVtbafUhy+lfACdbahbWcqrrG7HVvasOJRB5Xc0L1dI9q6hbV0z2qqVtUT/fEmqZhLUxaa3dTtevviMcoiqIoiqJ4GU8nmWdkZETbBE+herpHNXWL6uke1dQtqqd7Yk1TTzdhVhRFURRFOVq0CXOQIynspRwa1dM9qqlbVE/3qKZuUT3dE2uaetqxClW8VtygerpHNXWL6uke1dQtqqd7Yk1TTztWiqIoiqIokcTTOVaBQACfT31HV6ie7lFN3aJ6ukc1dYvq6Z5IaKo5VkGWLFkSbRM8herpHtXULaqne1RTt6ie7ok1TT3tWO3cuTPaJngK1dM9qqlbVE/3qKZuUT3dE2uaetqxUhRFURRFiSSedqyGDBkSbRM8herpHtXULaqne1RTt6ie7ok1TT3tWBUUFETbBE+herpHNXWL6uke1dQtqqd7Yk1TTztWq1evjrYJnkL1dI9q6hbV0z2qqVtUT/fEmqaedqwURVEURVEiSVTqWBljtgPfRuBSbYAdEbhOc0H1dI9q6hbV0z2qqVtUT/dEQtOu1tq24QyMimMVKYwxc8It6KXUj+rpHtXULaqne1RTt6ie7ok1TXUpUFEURVEUxRHqWCmKoiiKojjC647VX6JtgMdQPd2jmrpF9XSPauoW1dM9MaWpp3OsFEVRFEVRIonXI1aKoiiKoigRQx0rRVEURWkEjDHtjTHTDvN8ljFmgzFmcvAIazu/Ett4xrGqbwIHx8QbY943xnxpjLkyUrY1RYwxLxhjZhhj7jrMmDhjzLpqN4XBkbSxKRGmnvWOUYT6tNK52XDCcAL0/tkAjDEtgZeB1MMMGw08ZK2dEDy2R8a6pocxJtMY8z9jzCRjzLvGmIRDjIv6fdQTjlWYExjgBmCutfY44HxjTHqjG9cEMcacC/ittblAD2NM70MMPQZ4vdpN4evIWdl0CEfPBmje7AlTK52bDSDMe6jePxtGBXAhsO8wY44FrjLGzDPGPBwZs5osPwSetNZOBLYAp9YeECv3UU84VoQ3gQEmAG8Gv58KxExBsRhjAlU6TQLGHmLcscCZxphZwf8S4iJhXBNkAvXrGc4YRZhA/Vrp3GwY4dxDJ6D3z7Cx1u6z1u6tZ9j/EF1HArnGmGMa3bAmirX2j9baT4I/tgW21TFsAjFwH22SjpUx5s/VQvyTgV+EMYFB/hvbGPx+F9C+sWxsStSh5w2Ep9Ns4DvW2lFAPHB6oxvbNAln3uncDJ9wtNK52QDCdAJ0jrpnurW2wFpbAcwHNFJdD8aYXKCltfarOp6OiTnaJP+Ls9Zec4QvLQSSgb1AWvDnZk9tPY0xTyM6geh0KAd8kbW2NPj9HPSmcChC8w4OrWc4YxQhHK10brpH75/u+dgYczGi6UTgz1G2J6YxxrQCngXOO8SQmLiPNreb91yqQoNDgLXRMyWmCVenV4wxQ4wxfuBsYGEEbGuKhKOnzs3wCUcrnZvu0Tl6FBhjTjTGXF/r4fuAL4CvgD9Za1dE3rKmQTBZ/S3gV9babw8xLCbmqKcKhBpjJltrJwS/PxEYYK19rtrzXYH/Ap8CY4BjgyFYpRrGmAxgGvAZcBqSr5IF/MBae1e1cYOA1wAD/Mda++somBvz1KHnRcAFtbQ8SPMwl7ebHWHqqXPzCAjdQ/X+qcQaxpifAg9T9U/SF0B8LN5HPeVYhYMxphPi0X6sf7gOTXCX0MnAVGvtlmjb09QJR0/VPHxUq+ig908l1omFe0Ozc6wURVEURVEai+aWY6UoiqIoitJoqGOlKIqiKIriCHWsFEVRFEVRHKGOlaIoiqIoiiPUsVIUJWIYY/zGmEu1xYyiKF5Fb26K4iGMMZcCLYFSpP9buLxYuyaRMeY0YLW1doUx5tdIMcNMIADcihQ03G6MeQ5oDVxtrS2o5zpXA38E+gN3Bq8zOmhzGXAAONRW5TnW2pLga7Kpu2HwOmAQUAyUH+ZccciuaG3OrCiKU7TcgqJ4CGPMbGAg4lQE6hmeWe375JDTEjyPQQpB9gJGAFcBjyAOyaXAn4A+1tp1xpiFQIK1tn8Y9sUjffwGAydZaycbYz4CTgnj1+tqrV0XPM8kpFZNdfYD2cDOMM4FsMJa2y/MsUdFsHDhg8BZQCekf+Ed1trHjDF/B04Fultri47w/COQ1j1XWWtfcGS2oihHgEasFMVDWGtHHu75oMN0GnAXUlH/f0jkqaT6OGutNcZ8D/gauJiqSFIAuAN4IehUpSHRp9+GaV+ZMeZmpDLyT4DJSBQrEYlYlXJwlOlW4JagDSGKgYXW2qHB32sBsNxau8sY0wtxsio4tHPpR6qyR4pXgTORyuX/QBzf/xhjcoBLgFuP1KkCsNbONca8BzxojHnDWqt9/BQlSqhjpSjNAGNMe+AHiBPTG3gdWbpbfIjxQ4DjgecRJ2YM4vD8GHgXKDPGnA+UINGX/9V6fTzgr7Z0l4k4YBustZ8HX/suQCgKdRjbQ05CabWHK6o93xXpC/Zgtec3He6cwHXW2j/WM8YJxph+iFP1sbX2jFrPTQL2ITofLY8AM4GfI60/FEWJAupYKYoHMcacivwhD0V/fgdcCPwFON1au6aeBPKBwPWIM9MWaB98/KfIppc4oCviFADkSTCsBk8Dvwh+3w+YAfwGeMBa+44xJq2O14SwdURwqkeyqkeiLgra+XHw59Dr/gA8R02ykCXO/Ye6cCNwYvDrO9UfNMb0Ab4D/J+1tvhoL2KtnWWMWQ5cY4x5TPv4KUp00F2BiuIxghGSD4B3jTEpwYd3AJustdcHnaqzgK+NMR3rOoe19jVrbR/gDMSh2YVEia4AbrPWDgauBC4A3kSiUf0R5w0kOvZktVPuCX6t7kDsAgoOcaxuwK98KfB5tcT50H0tgCy51T7gCJcBjTG3GGOsMeaWQzzf1xhTaoyZaow5zxhjEQcP4C/B19rge3Rl0I43DnGuScGx59Z63BhjXgo+92itl/0T6II4bIqiRAF1rBTFIwT/6N9mrV0OnA6cBEwxxrSiWn6SMeZaJHpSAKQbY+6py1EI5v9MB75BIl5+4FzgPWPMscBEJJK1wVq7PHjd0HLdnFpLfKHrl1d7bD/wApJwXv14i5oO2OF+50ygJ1WOG0By8OsNQH6tY3KtMQ0lL/j12EM8/yyi0/XAZmQn5Xbk974veNwbtOU7iLP61SHOdRviHD5ojPFXe/x3wOXAX621d9R6zZfBr7UT+xVFiRDqWCmKdzgPOB/AWjsJOCf4eBHBZTRjTFvkD/NrwDhr7crg6yqjIsGIyMPI0t0iJNk95BA9CnwOPIHk8gAMrWZDp+C1NtSyra7tx+VAobV2Q/UDcbjK6xh/ENbavYijcqExZmjwsVVI3tfhjjpzmowx3YKRoJcOccl5iNM3uo7XXoA4NH+w1i6y1k4HHgDSgGXW2nuDx31AEqLbskMlrVtrFwKvIJHAS4PXuBO4GYkSXlvHy2YHv44/hP2KojQymmOlKN4hVAcKAGvtp8aYadba0lAuU7DuVG6t+k2FVEsMD+4I/DfiQDyMRJF6InWnAsgutnHIcuMCIMcY4w/m9GQhEaxwc4ZaBpfFqpNZ58hD8wQSwbkrWMerazgvMsaUWmvX1Hp4O+LErKrrNcFdjbOB8caYTtbaTcFzpSJLn9uQPLIQA5Do2Lxap8pCIlub6zHzLmR59d7gDsyHkFyyS621B+14tNbuNcaUIMuBiqJEAXWsFMU7VHBwFPoMY8yU0A/GmIFIcvON1ZKbD3BwWYLFwJLg+c5DSiPkWmvLkB2Be5A/8Pcjy085yI60ocDCBth8WfCoTZ2OTV1Ya8uNMX9DdsWdiDh84bAAGFbrXEVIOYTD8SUSEToW+Ffwsd8AnYEfBaNoIYYHv9Z2rFoHv+4+3IWstRuMMU8hJS6eRZZmz7XWHjjMy3ZRtdlAUZQIo0uBiuIdKpfbjDHxwcTmt5FSCSFOAK4D/mmMSTjMuf6M5GAdQJYOfcDMUPI18Amy7PgVsAW4IJgHdCxVeUjh8LS11lQ/gJcb8PoQS5F/FHsHfz4HqeZ+J1K81AD3IEU5DeIUltR1ojAI5TGNhsrNAjchS6e1bQ85VvNrPR6K6CWFcb3t1b7/sbW2vh2NyYSZo6YoinvUsVIU7xDKozoJicb8DLjcWvt+5QBrn0MKbp4P/OswztUtyHLSJcGfJ1BVdmED8IG1tjC4HPU6EnU6F4nEfOH21wqLUJ5X9ZY6vZDaVjcGf74E6F7t+SMtRzAd0TqUwP4csqx3XbXyFiGGB8cuqPX4tuDX1hwGY8zFiGO7JfjQjYcZjjHGB7Sodn5FUSKMOlaK4h1CJQRGBr8fYa19JfhYOkHHy1r7BJLwfQbBpOjaWGu3Ig7UDUirlAXW2h1IxKsz8Hi14U8hf8z/hlRDnxWmvXEEc6yqH0iOVVhpCsaY04wxq5Dcoy+pSt4GyTn7e2gcEs36wBjTLUz76sRauxtYhuSW/QDZfflna22NqFSwyv0QIL+OHoqbkUhU30NdxxhzOhIBWwIcAywHrqojJ606fZH3vrYjpyhKhFDHSlG8Q8ix+j0wxlqbH3Q8ngZ+CKwPDQzuTLsCcYYORXuk1EJnYJcxZj6S5/MlMLfaudYB7yO7315sgL0pSKRrWa3j7OBz4fA5sAaJsJ1MzVyxD5Hf8XGklQxBO2uXKDgS8oI2/hmpEXZXHWP6IA5t7WVAgpGtqUCbYAueGhhjxiLLuBuAidba7cDdiMNZu3ZVdUJRtGhEDRVFQR0rRfESPgBrbam1NlTXKQWJOq1EnI9KrLUv17F0Vf35LdbaS6y1HZGoTG8kJ+g4IN8YMwrAGHMR8F0kH+uOYIJ8vVhrE2rnV1U72tXzcn+13/U71tongzsRqxf+7I/0IHwFWB3MrUpGyhXUiTEm1RhziTEmt57rh/Ks0oBfWWt31THmUInrIUKV2Gs0oA62E/oA2AucbK3dDGCtfRuJHn7PGDPuEOeciCxx/rse+xVFaSTUsVIU7+ADUmotq+UjO9guAwpqL7sFxxwyOmSMSTbGXI3kUa1Eygf0RiI/W40xfwk+9zowAnG8JhtjaheoDF3jSHYidwh+rR6Nij/E2MpCmtbaguDuOX/ousHehcVAG+qurdUWccSuqcemUJmG2UiR07oIx7HaSrVdkcHo1cdB204J1uSqzq+CXx+v9XioWOrZSP7b+trPK4oSGbTcgqJ4hyQkv2rZEbx2avUfjDFjkAKU30UiIL8Ffh8st4AxZhOyHJgAXG+t/UPw8ZOB/wCTjDF3WWsfCp4yLfj1cDsRa2CMuQuJ5uQiRUOrF9Ks17GqxlaC5RuMMQ8hhU3TkPYvNbDWriW8djehquh1JayHznVbcFydWGsPBJdpHzbGDLPWzrfWfkOVI1nXaz49jH2XIXPgiTDsVxSlkdCIlaJ4BGvtqMMsrR3yAKZwcNRqMbJj7U6gm7X2tyGnKshCZLmpX8ipCtowB0m0foqaf+Ab7FgB64CxSJL37bZmU+FDOVYBYAXVdgdaa2+21oYaIX+BtL95gyPMtQomrH8XeN5aO7u+8fXwe+T3vP9oTmKMSUaiWe9Ya6cdpU2KohwF5jApFoqiKFHDGNMSaGOtzY8BW7ogjaV7IpGhfGBUGDWlwjn3eGS35e8O1d4mjHOEGmC/FIy6KYoSJdSxUhRFqYdgntmfkWjXJ8AvQu1sFEVRqqOOlaIoiqIoiiM0x0pRFEVRFMUR6lgpiqIoiqI4Qh0rRVEURVEUR6hjpSiKoiiK4gh1rBRFURRFURyhjpWiKIqiKIoj1LFSFEVRFEVxhDpWiqIoiqIojvh/szpIJEGb3kYAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import math\n", "import matplotlib.pyplot as plt\n", "plt.rcParams['font.sans-serif'] = ['SimHei']\n", "plt.rcParams['axes.unicode_minus'] = False\n", "plt.figure(figsize=(10,8))\n", "x = np.linspace(start=-1, stop=2, num=1001, dtype=np.float)\n", "logi = np.log(1 + np.exp(-x)) / math.log(2)\n", "boost = np.exp(-x)\n", "y_01 = x < 0\n", "y_hinge = 1.0 - x\n", "y_hinge[y_hinge < 0] = 0\n", "\n", "plt.plot(x, y_01, 'g-', mec='k', label='（0/1损失）0/1 Loss', lw=2)\n", "plt.plot(x, y_hinge, 'b-', mec='k', label='（合页损失）Hinge Loss', lw=2)\n", "plt.plot(x, boost, 'm--', mec='k', label='（指数损失）Adaboost Loss', lw=2)\n", "plt.plot(x, logi, 'r-', mec='k', label='（逻辑斯谛损失）Logistic Loss', lw=2)\n", "plt.grid(True, ls='--')\n", "plt.legend(loc='upper right',fontsize=15)\n", "plt.xlabel('函数间隔:$yf(x)$',fontsize=20)\n", "plt.title('损失函数',fontsize=20)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " \n", "可以认为支持向量机、逻辑斯谛回归与最大熵模型、提升方法使用不同的代理损失函数(surrogateloas Punotion)表示分类的损失，定义经验风险或结构风险函数，实现二类分类学习任务。学习的策略是优化以下结构风险函数，\n", "\n", "$$\n", "\\min _{f \\in H} \\frac{1}{N} \\sum_{i=1}^{N} L\\left(y_{i}, f\\left(x_{i}\\right)\\right)+\\lambda J(f)\n", "$$\n", "\n", "第1项为经验风险(经验损失)，第2项为正则化项，$L(y,f(x))$为损失函数，$J(f)$为模型的复杂度，$\\lambda \\geq 0$为系数。\n", "\n", "支持向量机用$L_2$范数表示模型的复杂度。原始的逻辑斯谛回归与最大熵模型没有正则化项，可以给它们加上$L_2$范数正则化项。提升方法没有显式的正则化项，通常通过早停止(early stopping)的方法达到正则化的效果。\n", " \n", "概率模型的学习可以形式化为极大似然估计或贝叶斯估计的极大后验概率估计。学习的策略是极小化对数似然损失或极小化正则化的对数似然损失。对数似然损失可以写成：\n", "\n", "$$-logP(y|x)$$\n", "\n", "极大后验概率估计时，正则化项是先验概率的负对数。\n", "\n", "决策树学习的策略是正则化的极大似然估计，损失函数是对数似然损失，正则化项是决策树的复杂度。\n", "\n", "逻辑斯谛回归与最大熵模型、条件随机场的学习策略既可以看成是极大似然估计(或正则化的极大似然估计)，又可以看成是极小化逻辑斯谛损失(或正则化的逻辑斯谛损失)。\n", "\n", "朴素贝叶斯模型、隐马尔可夫模型的非监督学习也是极大似然估计或极大后验概率估计，但这时模型含有隐变量。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4 学习算法\n", "\n", "统计学习的问题有了具体的形式以后，就变成了最优化问题。\n", "\n", "朴素贝叶斯法与隐马尔可夫模型的监督学习，最优解即极大似然估计值，可以由概率计算公式直接计算。\n", "\n", "感知机、逻辑斯谛回归与最大熵模型、条件随机场的学习利用梯度下降法、拟牛顿法等一般的无约束最优化问题的解法。\n", "支持向量机学习，可以解凸二次规划的对偶问题。有序列最小最优化算法等方法。\n", "\n", "决策树学习是基于启发式算法的典型例子。可以认为特征选择、生成、剪枝是启发式地进行正则化的极大似然估计。\n", "\n", "提升方法利用学习的模型是加法模型、损失函数是指数损失函数的特点，启发式地从前向后逐步学习模型，以达到逼近优化目标函数的目的。\n", "\n", "EM算法是一种迭代的求解含隐变量概率模型参数的方法，它的收敛性可以保证，但是不能保证收敛到全局最优。\n", "\n", "支持向量机学习、逻辑斯谛回归与最大熵模型学习、条件随机场学习是凸优化问题，全局最优解保证存在。而其他学习问题则不是凸优化问题。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第13章 无监督学习概论/13.Introduction_to_Unsupervised_Learning.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第13章 无监督学习概论" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1.机器学习或统计学习一般包括监督学习、无监督学习、强化学习。\n", "\n", "无监督学习是指从无标注数据中学习模型的机器学习问题。无标注数据是自然得到的数据，模型表示数据的类别、转换或概率无监督学习的本质是学习数据中的统计规律或潜在结构，主要包括聚类、降维、概率估计。\n", "\n", "2.无监督学习可以用于对已有数据的分析，也可以用于对未来数据的预测。学习得到的模型有函数$z＝g(x)$，条件概率分布$P(z|x)$，或条件概率分布$P(x|z)$。\n", "\n", "无监督学习的基本想法是对给定数据（矩阵数据）进行某种“压缩”，从而找到数据的潜在结构，假定损失最小的压缩得到的结果就是最本质的结构。可以考虑发掘数据的纵向结构，对应聚类。也可以考虑发掘数据的横向结构，对应降维。还可以同时考虑发掘数据的纵向与横向结构，对应概率模型估计。\n", "\n", "3.聚类是将样本集合中相似的样本（实例）分配到相同的类，不相似的样本分配到不同的类。聚类分硬聚类和软聚类。聚类方法有层次聚类和$k$均值聚类。\n", "\n", "4.降维是将样本集合中的样本（实例）从高维空间转换到低维空间。假设样本原本存在于低维空间，或近似地存在于低维空间，通过降维则可以更好地表示样本数据的结构，即更好地表示样本之间的关系。降维有线性降维和非线性降维，降维方法有主成分分析。\n", "\n", "5.概率模型估计假设训练数据由一个概率模型生成，同时利用训练数据学习概率模型的结构和参数。概率模型包括混合模型、率图模型等。概率图模型又包括有向图模型和无向图模型。\n", "\n", "6.话题分析是文本分析的一种技术。给定一个文本集合，话题分析旨在发现文本集合中每个文本的话题，而话题由单词的集合表示。话题分析方法有潜在语义分析、概率潜在语义分析和潜在狄利克雷分配。\n", "\n", "7.图分析的目的是发掘隐藏在图中的统计规律或潜在结构。链接分析是图分析的一种，主要是发现有向图中的重要结点，包括 **PageRank**算法。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第14章 聚类方法/14.Clustering.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "_lhcu-eFro6f" }, "source": [ "# 第14章聚类方法" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1.聚类是针对给定的样本，依据它们属性的相似度或距离，将其归并到若干个“类”或“簇”的数据分析问题。一个类是样本的一个子集。直观上，相似的样本聚集在同类，不相似的样本分散在不同类。\n", "\n", "2.距离或相似度度量在聚类中起着重要作用。\n", "\n", "常用的距离度量有闵可夫斯基距离，包括欧氏距离曼哈顿距离、切比雪夫距离、、以及马哈拉诺比斯距离。常用的相似度度量有相关系数、夹角余弦。\n", "用距离度量相似度时，距离越小表示样本越相似；用相关系数时，相关系数越大表示样本越相似。\n", "\n", "3.类是样本的子集，比如有如下基本定义：\n", "用$G$表示类或簇，用$x_i$,$x_j$；等表示类中的样本，用$d_{ij}$表示样本$x_i$与样本$x_j$之间的距离。如果对任意的$x _ { i } , x _ { j } \\in G$，有$$d _ { i j } \\leq T$$\n", "则称$G$为一个类或簇。\n", "\n", "描述类的特征的指标有中心、直径、散布矩阵、协方差矩阵。\n", "\n", "4.聚类过程中用到类与类之间的距离也称为连接类与类之间的距离包括最短距离、最长距离、中心距离、平均距离。\n", "\n", "5.层次聚类假设类别之间存在层次结构，将样本聚到层次化的类中层次聚类又有聚合或自下而上、分裂或自上而下两种方法。\n", "\n", "聚合聚类开始将每个样本各自分到一个类；之后将相距最近的两类合并，建立一个新的类，重复此操作直到满足停止条件；得到层次化的类别。分裂聚类开始将所有样本分到一个类；之后将已有类中相距最远的样本分到两个新的类，重复此操作直到满足停止条件；得到层次化的类别。\n", "\n", "聚合聚类需要预先确定下面三个要素：\n", "\n", "（1）距离或相似度；\n", "（2）合并规则；\n", "（3）停止条件。\n", "\n", "根据这些概念的不同组合，就可以得到不同的聚类方法。\n", "\n", "6.$k$均值聚类是常用的聚类算法，有以下特点。基于划分的聚类方法；类别数k事先指定；以欧氏距离平方表示样本之间的距离或相似度，以中心或样本的均值表示类别；以样本和其所属类的中心之间的距离的总和为优化的目标函数；得到的类别是平坦的、非层次化的；算法是迭代算法，不能保证得到全局最优。\n", "\n", "$k$均值聚类算法，首先选择k个类的中心，将样本分到与中心最近的类中，得到一个聚类结果；然后计算每个类的样本的均值，作为类的新的中心；重复以上步骤，直到收敛为止。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "hGPg5M7wsOQY" }, "source": [ "### 层次聚类 \n", "\n", "1. **聚合**（自下而上）：聚合法开始将每个样本各自分裂到一个类，之后将相距最近的两类合并，建立一个新的类，重复次操作知道满足停止条件，得到层次化的类别。\n", "\n", "2. **分裂**（自上而下）： 分裂法开始将所有样本分到一个类，之后将已有类中相距最远的样本分到两个新的类，重复此操作直到满足停止条件，得到层次化的类别。\n", "\n", "\n", "### k均值聚类\n", "\n", "k均值聚类是基于中心的聚类方法，通过迭代，将样本分到k个类中，使得每个样本与其所属类的中心或均值最近，得到k个平坦的，非层次化的类别，构成对空间的划分。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": {}, "colab_type": "code", "id": "qAlQYJ2Srd2_" }, "outputs": [], "source": [ "import math\n", "import random\n", "import numpy as np\n", "from sklearn import datasets,cluster\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": {}, "colab_type": "code", "id": "M_XOaWU5xpjI" }, "outputs": [], "source": [ "iris = datasets.load_iris()" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 143 }, "colab_type": "code", "id": "_swSYxCr0RzU", "outputId": "88d09d3b-7700-4af5-e3b6-4d7735a3dd75" }, "outputs": [ { "data": { "text/plain": [ "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n", " 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n", " 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "gt = iris['target'];gt" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "UoIRpftd9Uh2" }, "source": [ "3类" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "pI6cS2sjy3Sz", "outputId": "3d9d01de-31b4-4eea-989c-b0a986a77af5" }, "outputs": [ { "data": { "text/plain": [ "(150, 2)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "iris['data'][:,:2].shape" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": {}, "colab_type": "code", "id": "YwIVX5j81348" }, "outputs": [], "source": [ "data = iris['data'][:,:2]" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": {}, "colab_type": "code", "id": "h-wEZbDR03E_" }, "outputs": [], "source": [ "x = data[:,0]\n", "y = data[:,1]" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 269 }, "colab_type": "code", "id": "bW_lxjVdy4rW", "outputId": "ded2fc31-a69a-4e40-b0d4-350b995488ce" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAdAElEQVR4nO3df4wc9Znn8ffHMyYEkphTmNsAxjanZDciCwQYEXxerdiMb0UCC3+E03rliB9KNLc2uZC7XUXHWiKxJe8p0mnDZRFmJ8ne4sAm5NgkZ34kF3A2CuEurMaEHyHkJC7Bxgt7TCAZjnOOw+a5P7pndqbdM1U99e3qqq7PSxrRXVVT/dS3i2fa/XzrKUUEZmY2/FYNOgAzMyuHE76ZWUM44ZuZNYQTvplZQzjhm5k1hBO+mVlD5Er4kp6V9KSkxyRNd1kvSZ+T9IykJyRdkD5UMzMrYrSHbX8nIn6+xLoPAO9q/7wP2NP+r5mZVUSqr3SuBPZGyw+AUySdlmjfZmaWQN5P+AF8W1IAfxERUx3rzwCeW/D8cHvZCws3kjQJTAKcfPLJF7773e9eUdBmZk114MCBn0fE2Ep+N2/C3xQRz0v6p8ADkn4SEd9bsF5dfue4ng3tPxRTAOPj4zE9fVw5wMzMliHp4Ep/N9dXOhHxfPu/LwJfBy7q2OQwcOaC52uB51calJmZpZeZ8CWdLOmtc4+B3wV+1LHZPuDq9mydi4HZiHgBMzOrjDxf6fwa8HVJc9v/dUR8S9IfAkTEbcD9wAeBZ4AjwHX9CdfMzFYqM+FHxE+B87osv23B4wCuTxuamZml5CttzcwawgnfzKwhnPDNzBrCCd/MrCGc8M3MGsIJ38ysIZzwzcwawgnfzKwhnPDNzBrCCd/MrCGc8M3MGsIJ38ysIZzwzcwawgnfzKwhnPDNzBrCCd/MrCGc8M3MGsIJ38ysIXInfEkjkn4o6d4u666VNCPpsfbPR9OGaWZmReW5ifmcG4Cngbctsf6uiPhY8ZDMzKwfcn3Cl7QWuAz4Qn/DMTOzfsn7lc7NwCeBN5bZ5kOSnpB0t6Qzi4dmZmYpZSZ8SZcDL0bEgWU2uwfYEBHnAg8Cty+xr0lJ05KmZ2ZmVhSwmZmtTJ5P+JuAKyQ9C3wFeL+kOxZuEBEvRcRr7aefBy7stqOImIqI8YgYHxsbKxC2mZn1KjPhR8SNEbE2IjYAW4DvRMSHF24j6bQFT6+gVdw1S+bOJ+9kw80bWLVzFRtu3sCdT9456JDMaqeXWTqLSNoFTEfEPuDjkq4AjgIvA9emCc+slewn75nkyOtHADg4e5DJeyYB2HrO1kGGZlYrioiBvPD4+HhMT08P5LWtXjbcvIGDswePW75+zXqe/cSz5QdkNkCSDkTE+Ep+11faWuUdmj3U03Iz684J3ypv3Zp1PS03s+6c8K3ydk/s5qTVJy1adtLqk9g9sXtAEZnVkxO+Vd7Wc7Yy9XtTrF+zHiHWr1nP1O9NuWBr1iMXbc3MasRFWzMzy+SEb2bWEE74ZmYN4YRvpXBrBLPBW3FrBbO83BrBrBr8Cd/6bsf+HfPJfs6R14+wY/+OAUVk1kxO+NZ3bo1gVg1O+NZ3bo1gVg1O+Nb3gqpbI5hVgxN+w80VVA/OHiSI+YJqyqTv1ghm1eDWCg3nXvNm9eLWCrZiLqiaNYcTfsO5oGrWHE74DeeCqllz5E74kkYk/VDSvV3WvUnSXZKekfSIpA0pg7T+GaaCqts3mC2vl9YKNwBPA2/rsu4jwC8i4p2StgCfAX4/QXxWgq3nbK1lgl/I7RvMsuX6hC9pLXAZ8IUlNrkSuL39+G5gQpKKh2eWj9s3mGXL+5XOzcAngTeWWH8G8BxARBwFZoG3d24kaVLStKTpmZmZFYRr1p1nG5lly0z4ki4HXoyIA8tt1mXZcRP8I2IqIsYjYnxsbKyHMM2W59lGZtnyfMLfBFwh6VngK8D7Jd3Rsc1h4EwASaPAGuDlhHFahW2/bzuju0bRTjG6a5Tt920vPQbPNjLLlpnwI+LGiFgbERuALcB3IuLDHZvtA65pP76qvc1gLuG1Um2/bzt7pvdwLI4BcCyOsWd6T+lJf5hmG5n1S0+tFSRdAvxxRFwuaRcwHRH7JJ0IfAk4n9Yn+y0R8dPl9uXWCsNhdNfofLJfaEQjHL3p6AAiMhtuRVor9HTHq4j4LvDd9uObFiz/v8C/XEkAVm/dkv1yy81scHylrRUyopGelpvZ4DjhWyGTF072tNzMBscJf8ht3rsZ7dT8z+a9m5Pu/9bLbmXb+Lb5T/QjGmHb+DZuvezWpK/jtglWZSnOzzLOcffDH2Kb925m/8/2H7d84qwJHrz6wQFEtDKdbROgNeXSs3CsClKcn73so0jR1gl/iGnn0t0t4lP1mTXrm7RYlaU4P3vZh2+AYkPNbROsylKcn2Wd4074Vnlum2BVluL8LOscd8IfYhNnTfS0fClZxaR+F5vcNsGqLMX5WdY57oQ/xK47/zrU0ddOiOvOvy73PuaKSQdnDxLEfJ/5uaSetT4Ft02wKktxfpZ1jrtoO8TKKCa5oGpWLhdtrasyikkuqJrVhxP+ECujmOSCqll9OOGvUFlXfhYpmJZRTHJB1frJV1in1VO3TGsp64bZWa+TtX4ulh37d3Bo9hDr1qxj98TunotJy+0jxWuYdeMb06fnou0KlFWodMHUmsznd3cu2pasrEKlC6bWZD6/03PCX4GyCpUumFqT+fxOzwl/BcoqVLpgak3m8zu9zIQv6URJfyfpcUlPSdrZZZtrJc1Ieqz989H+hFsNZV0Vl/U6W8/Zysa1Gxf9zsa1GxfFsf2+7YzuGkU7xeiu0a43Fy9jJoRnW1ivfIV1eplFW0kCTo6IVyWtBr4P3BARP1iwzbXAeER8LO8L17loWxXb79vOnuk9xy2fuwFJ1noop9e8+9mbpdPXom20vNp+urr9U59m6kNs6sDUssuz1kNrOuXCRAxw5PUj7Ni/I1GU5byGmWXL9R2+pBFJjwEvAg9ExCNdNvuQpCck3S3pzCX2MylpWtL0zMxMgbAN4FgcW3Z51nooZyaEZ1uYVUOuhB8RxyLivcBa4CJJv9mxyT3Ahog4F3gQuH2J/UxFxHhEjI+NjRWJ22D+PrJLLc9aD+XMhPBsC7Nq6GmWTkT8EvgucGnH8pci4rX2088DFyaJrsJSFCHzFFSXM3nh5LLLs9ZDaybE6lWrF61fvWp1TzMhso6jTrMtir6veX7fBWwblDyzdMYkndJ+/GZgM/CTjm1OW/D0CuDplEFWTYoe8HMF1YVfv+yZ3tNz0l/OpnWbGF21uHvG6KpRNq3btGhZqy6/9PPl5DmOusy2KPq+5vn9Mu4fYLaUPLN0zqX1Fc0IrT8QX42IXZJ2AdMRsU/Sv6eV6I8CLwPbIuInS+6Ues/SSXHJ9+iu0a7fsY9ohKM3HU2yjzxxFj2WFMdRFUXHoozxNisySyezeVpEPAGc32X5TQse3wjcuJIA6ihFETJPQbXoPvLEWfRYUhxHVRQdizLG26wIX2m7AimKkHkKqkX3kSfOoseS4jiqouhYlDHeZkU44a9AiiJknoJq0X3kibPosaQ4jqooOhZljLdZEU74K5CiCHnrZbeybXzboimUC6+AzbuPibMmFi2bOGtifh9bz9nKNedds+g1rjnvmkVxFj2WFMdRFUXHIs/v52mHUYaiM8SsntwPv8ayWha4pUH15Gl30YQYbOWKFG2d8GvMN0ipnyrMaqpCDLZyvgFKQ/kGKfVThVlNVYjBBsMJv8Z8g5T6qcKspirEYIPR2IRfxiX0/S6M7Z7YzQkjJyxadsLICb5ByhKq0NIgz6ymrPOm6HHknVmV9TpF1+dRhfdsmGReeDWMOouZc5e3A7mKmXl+v7MwNtdyAEhaGOuswSx8PhfLjv07ODR7iHVr1rF7YncjC7ZF3/NU5t77qQNTHItjjGiEyQsn55dnnTcpjiMrBsger6Lr86jKezZMGlm0LeMS+jIKYy7K5leXsUrRLiOFohMCUsRZl/esbC7a9qiMS+jLKIy5KJtfXcYqRbuMFIpOCEgRZ13eszppZMIv4xL6MgpjLsrmV5exStEuI4WiEwJSxFmX96xOGpnwy7iEvoyWAy7K5leXsUrRLiOFrNcpuj5FDLYCETGQnwsvvDAG6Y4n7oj1n10f+rRi/WfXxx1P3NHT72+7d1uM7BwJPk2M7ByJbfdu63mbPDGk2Ie11GWsJm6fCD7N/M/E7ROL1qc4jqzXyPM6Z99y9qJ9nH3L2cnjrMt7ViZabelXlHcbWbQtKkXLgjz78CXwzVNGO4zNezez/2f7j1s+cdYED179YGn7sJVxa4WSlTUDwZfAN08ZM1O0c+k7msWn8uWDFPuwlfEsnZKVNQPBl8A3j2emWD854a9AWTMQfAl883hmivVTnpuYnyjp7yQ9LukpSTu7bPMmSXdJekbSI5I29CPYvPJcjl3kku1UMxA6E/eIRnqe6VOFS89TxNDvdgJ5XiPP65TRLiPr3Nq8dzPaqfmfzXs393QcnfdQWGp5in1kKaM9Qwp1iTNLnk/4rwHvj4jzgPcCl0q6uGObjwC/iIh3Ap8FPpM2zPzmil4HZw8SxPzl2AvfgDzbLCfFDVAePvTwcV/NHItjPHzo4fnnm9ZtYlXHW7SKVWxatynJcaSQIoa54vTceMy1E5hLpmW8Rp7XybOPorLOrW7F0v0/278o6Wcdx6+//de7vvbC5Vn7ePDqB7vefKeXgm3Wa1Th/K5TnHn0VLSVdBLwfWBbRDyyYPl/BT4dEf9d0ijwD8BYLLPzfhVt8xS9qnDJdp6CbB363aeIoYx2AinGuwpF9DzF0hTHUca5VYfzG6oXZ9+LtpJGJD0GvAg8sDDZt50BPAcQEUeBWeDtXfYzKWla0vTMzMxK4s2Up+hVhcJYnoJsHfrdp4ihjHYCKca7LkX0FMdRxrlVh/M7TxxViTOPXAk/Io5FxHuBtcBFkn6zY5NuHzuO+3QfEVMRMR4R42NjY71Hm0OeolcVCmN5CrJ16HefIoYy2gmkGO+6FNFTHEcZ51Ydzu88cVQlzjx6mqUTEb8Evgtc2rHqMHAmQPsrnTXAywni61meolcVLtnOU5At4/L1olLEUEY7gRTjXUa7jCx5iqUpjqOMc6sO53eeOKoSZy5Zl+ICY8Ap7cdvBh4CLu/Y5nrgtvbjLcBXs/bbz9YKKVoWlCFPDFnHUoVLz8u41D/F+5WiHUaKtgdF39M8bRGyjiPFuZdC0bEo6/yvShwRfW6tIOlc4HZghNa/CL4aEbsk7Wq/8D5JJwJfAs6n9cl+S0T8dLn9DvJK2zIuX7f8st6Pst6vonHkibOMY23K+d2U4+zk1go9qkr131qqMguijJt++MYh6TTlODu5tUKP6lRVb4KqzIIo46YfvnFIOk05zpQamfDrVFVvgqrMgijjph++cUg6TTnOlBqZ8GtVVW+A3RO7Wb1q9aJlq1etXjQLIqsNBRS/vD3PbIxuVz4vXL/cceQ91uVeI89xpjq/q94uIO9xlnEcVR+rOY1M+ClaI1hakpZ8nqcNRYrL27POi4cPPcwbvLHod97gjUVxLHcceY91udfIc5wpzu86tAvIc5xlHEcdxmpOI4u2Vi11aQWQogVE0WOtSgG7LqrQIiI1F22t1urSCiBFC4iix1qVAnZdVKFFRJU44dvA1aUVQIoWEEWPtSoF7LqoQouIKnHCt4EXnHZP7OaEkRMWLTth5ISeWwEste9eLDcWKVpAFD3WsiYcDMvEhiq0iKgSJ/yGq0rBqbOWtPD5rZfdyrbxbfOfckc0ctyN3P/0e3/adb9LLe8mayyy4shbLF3uWLPugVDWhINhmdhQxnHUaaxctG24KhTnUsSQ4qbaVSjwVeH9sGpz0dZWrAoFpyrEUFYcVbmq2JrJCb/hqlBwqkIMZcVRlauKrZmc8BuuCgWnFDGcferZPS3vVxxFX6MK74cNLyf8hiur4LTc7Je8V0wuN5Poqeuf4vS3nL5o2elvOZ2nrn8qd4wp4ij6GnUqANo/GvRMt7xctLW+K9q3PEWf+Sochw2nss8L98O3Sis68yRFy4IUPIPGunFrBbMFis48SdGyIAXPoLFu6nReOOFb3xWdeZKiZUEKnkFj3dTpvMhM+JLOlPS3kp6W9JSkG7psc4mkWUmPtX9u6k+4zVO0GFSFYlJWOwFYPs68feYHPcMmr+33bWd01yjaKUZ3jbL9vu3JYsyrCufFsKjTzKrRHNscBf4oIh6V9FbggKQHIuLHHds9FBGXpw+xuTqLQXOX+gMrKnb2+vspLddOIE+cWX3m57bbsX8Hh2YPsW7NOnZP7E5+CX3R19h+33b2TO+Zf34sjs0/X9gqop+qdF4MgzLOvVR6LtpK+i/ALRHxwIJllwB/3EvCd9E2WxnFzjKUcXPwusjT27/fhmk8m6i0oq2kDcD5wCNdVm+U9Likb0p6zxK/PylpWtL0zMxMz8E2TRnFzjKUcXPwusjT27/fhmk8rTe5E76ktwB/A3wiIl7pWP0osD4izgP+HPhGt31ExFREjEfE+NjY2Epjbowyip1lKOPm4HWRp7d/vw3TeFpvciV8SatpJfs7I+Jrnesj4pWIeLX9+H5gtaRTk0baQEWLQVUpJhVtJ1CV40ghT2//fhum8bTe5JmlI+CLwNMR8WdLbPOO9nZIuqi935dSBtpEW8/ZyjXnXbOo//o1512TuxhUlcv0t56zlY1rNy5atnHtxtztBKpyHCnk6e3fb8M0ntabzKKtpN8CHgKeBN5oL/4TYB1ARNwm6WPANlozen4F/NuI+G/L7ddF22zDcil/58yUOWUnOrNh4NYKQ2pYZlNUYWaK2bBwa4UhNSyzKaowM8XMnPArbVhmU1RhZoqZOeEvqQqXnueZTVGFOLNUYWZKldThPbPh5ITfxVyx9ODsQYKYv/S87P8xs2ZTVCXOLJvWbWJVx6m2ilVsWrdpQBENTl3eMxtOLtp2UZdiqeOsH4+FFeWibWJ1KZY6zvrxWNggOeF3UZdiqeOsH4+FDZITfhd1ufTccdaPx8IGyQm/i7pceu440yt6c5KsGTh1GgsbPi7amrUVbQExLK0wrNpctDVLYOrAVE/LO+3Yv2NRsgc48voRduzfUTg2sxSc8M3airaA8AwcqzonfLO2oi0gPAPHqs4J36ytaAsIz8CxqnPCN2srenMSz8CxqvMsHTOzGvEsHTMzy+SEb2bWEHluYn6mpL+V9LSkpyTd0GUbSfqcpGckPSHpgv6Ea53cW93M8hrNsc1R4I8i4lFJbwUOSHogIn68YJsPAO9q/7wP2NP+r/VR55Wdc73VARcKzew4mZ/wI+KFiHi0/fh/A08DZ3RsdiWwN1p+AJwi6bTk0doivrLTzHrR03f4kjYA5wOPdKw6A3huwfPDHP9HAUmTkqYlTc/MzPQWqR3HV3aaWS9yJ3xJbwH+BvhERLzSubrLrxw33zMipiJiPCLGx8bGeovUjuMrO82sF7kSvqTVtJL9nRHxtS6bHAbOXPB8LfB88fBsOb6y08x6kWeWjoAvAk9HxJ8tsdk+4Or2bJ2LgdmIeCFhnNaFr+w0s15kXmkr6beAh4AngTfai/8EWAcQEbe1/yjcAlwKHAGui4hlL6P1lbZmZr0rcqVt5rTMiPg+3b+jX7hNANevJAAzMyuHr7Q1M2sIJ3wzs4ZwwjczawgnfDOzhnDCNzNrCCd8M7OGcMI3M2sIJ3wzs4ZwwjczawgnfDOzhnDCNzNrCCd8M7OGcMI3M2sIJ3wzs4ZwwjczawgnfDOzhnDCNzNrCCd8M7OGyHMT87+U9KKkHy2x/hJJs5Iea//clD5MMzMrKvOetsBf0bpB+d5ltnkoIi5PEpGZmfVF5if8iPge8HIJsZiZWR+l+g5/o6THJX1T0nsS7dPMzBLK85VOlkeB9RHxqqQPAt8A3tVtQ0mTwCTAunXrEry0mZnlVfgTfkS8EhGvth/fD6yWdOoS205FxHhEjI+NjRV9aTMz60HhhC/pHZLUfnxRe58vFd2vmZmllfmVjqQvA5cAp0o6DHwKWA0QEbcBVwHbJB0FfgVsiYjoW8RmZrYimQk/Iv4gY/0ttKZtmplZhflKWzOzhnDCNzNrCCd8M7OGcMI3M2sIJ3wzs4ZwwjczawgnfDOzhnDCNzNrCCd8M7OGcMI3M2sIJ3wzs4ZwwjczawgnfDOzhnDCNzNrCCd8M7OGcMI3M2sIJ3wzs4ZwwjczawgnfDOzhshM+JL+UtKLkn60xHpJ+pykZyQ9IemC9GGamVlReT7h/xVw6TLrPwC8q/0zCewpHpaZmaWWmfAj4nvAy8tsciWwN1p+AJwi6bRUAZqZWRqjCfZxBvDcgueH28te6NxQ0iStfwUAvLbU10QVcyrw80EHkYPjTKsOcdYhRnCcqf3GSn8xRcJXl2XRbcOImAKmACRNR8R4gtfvK8eZluNMpw4xguNMTdL0Sn83xSydw8CZC56vBZ5PsF8zM0soRcLfB1zdnq1zMTAbEcd9nWNmZoOV+ZWOpC8DlwCnSjoMfApYDRARtwH3Ax8EngGOANflfO2pFcQ7CI4zLceZTh1iBMeZ2orjVETXr9vNzGzI+EpbM7OGcMI3M2uIUhK+pBFJP5R0b5d1b5J0V7s1wyOSNpQRUzcZcV4raUbSY+2fjw4oxmclPdmO4bjpWVVpdZEjzkskzS4Yz5sGEOMpku6W9BNJT0va2LG+KmOZFWcVxvI3Frz+Y5JekfSJjm0GPp454xz4eLbj+DeSnpL0I0lflnRix/qec2eKefh53AA8Dbyty7qPAL+IiHdK2gJ8Bvj9kuLqtFycAHdFxMdKjGcpvxMRS10gsrDVxftotbp4X1mBdVguToCHIuLy0qI53n8EvhURV0k6ATipY31VxjIrThjwWEbE/wDeC60PTsDfA1/v2Gzg45kzThjweEo6A/g4cHZE/ErSV4EttFrdzOk5d/b9E76ktcBlwBeW2ORK4Pb247uBCUndLubqqxxx1oVbXeQg6W3AbwNfBIiI/xcRv+zYbOBjmTPOqpkA/mdEHOxYPvDx7LBUnFUxCrxZ0iitP/Kd1zf1nDvL+ErnZuCTwBtLrJ9vzRARR4FZ4O0lxNUpK06AD7X/KXq3pDOX2a6fAvi2pANqtarotFSri7JlxQmwUdLjkr4p6T1lBgf8M2AG+E/tr/G+IOnkjm2qMJZ54oTBjmWnLcCXuyyvwngutFScMODxjIi/B/4DcIhWm5rZiPh2x2Y9586+JnxJlwMvRsSB5TbrsqzUuaI547wH2BAR5wIP8o9/Wcu2KSIuoPXP4+sl/XbH+oGPZ1tWnI8C6yPiPODPgW+UHN8ocAGwJyLOB/4P8O86tqnCWOaJc9BjOa/9ldMVwH/utrrLsoHMC8+Ic+DjKemf0PoEfxZwOnCypA93btblV5cdz35/wt8EXCHpWeArwPsl3dGxzXxrhvY/XdawfHfOfsiMMyJeiojX2k8/D1xYbojzcTzf/u+LtL57vKhjk0q0usiKMyJeiYhX24/vB1ZLOrXEEA8DhyPikfbzu2kl1s5tBj2WmXFWYCwX+gDwaET8ry7rqjCec5aMsyLjuRn4WUTMRMTrwNeAf96xTc+5s68JPyJujIi1EbGB1j+fvhMRnX+l9gHXtB9f1d6m1L/6eeLs+K7xClrF3VJJOlnSW+ceA78LdHYcHXirizxxSnrH3PeNki6idS6+VFaMEfEPwHOS5joPTgA/7ths4GOZJ85Bj2WHP2Dpr0kGPp4LLBlnRcbzEHCxpJPasUxwfM7pOXeWNUtnEUm7gOmI2EerGPUlSc/Q+uu0ZRAxddMR58clXQEcpRXntQMI6deAr7fPxVHgryPiW5L+EAq3uig7zquAbZKOAr8CtpT9hx7418Cd7X/e/xS4roJjmSfOKowlkk4C/gXwrxYsq9x45ohz4OMZEY9IupvW10tHgR8CU0Vzp1srmJk1hK+0NTNrCCd8M7OGcMI3M2sIJ3wzs4ZwwjczawgnfDOzhnDCNzNriP8Pq8UAjgO63JUAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.scatter(x, y, color='green')\n", "plt.xlim(4, 8)\n", "plt.ylim(1, 5)\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": {}, "colab_type": "code", "id": "WkcMa9cs2onL" }, "outputs": [], "source": [ "# 定义聚类数的节点\n", "\n", "class ClusterNode:\n", " def __init__(self, vec, left=None, right=None, distance=-1, id=None, count=1):\n", " \"\"\"\n", " :param vec: 保存两个数据聚类后形成新的中心\n", " :param left: 左节点\n", " :param right: 右节点\n", " :param distance: 两个节点的距离\n", " :param id: 用来标记哪些节点是计算过的\n", " :param count: 这个节点的叶子节点个数\n", " \"\"\"\n", " self.vec = vec\n", " self.left = left\n", " self.right = right\n", " self.distance = distance\n", " self.id = id\n", " self.count = count" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": {}, "colab_type": "code", "id": "_M1vGW8s5ycx" }, "outputs": [], "source": [ "def euler_distance(point1: np.ndarray, point2: list) -> float:\n", " \"\"\"\n", " 计算两点之间的欧拉距离，支持多维\n", " \"\"\"\n", " distance = 0.0\n", " for a, b in zip(point1, point2):\n", " distance += math.pow(a - b, 2)\n", " return math.sqrt(distance)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": {}, "colab_type": "code", "id": "udgrrhsn19X1" }, "outputs": [], "source": [ "# 层次聚类（聚合法）\n", "\n", "class Hierarchical:\n", " def __init__(self, k):\n", " self.k = k\n", " self.labels = None\n", " \n", " def fit(self, x):\n", " nodes = [ClusterNode(vec=v, id=i) for i, v in enumerate(x)]\n", " distances = {}\n", " point_num, feature_num = x.shape\n", " self.labels = [-1] * point_num\n", " currentclustid = -1\n", " while(len(nodes)) > self.k:\n", " min_dist = math.inf\n", " nodes_len = len(nodes)\n", " closest_part = None\n", " for i in range(nodes_len - 1):\n", " for j in range(i+1, nodes_len):\n", " d_key = (nodes[i].id, nodes[j].id)\n", " if d_key not in distances:\n", " distances[d_key] = euler_distance(nodes[i].vec, nodes[j].vec)\n", " d = distances[d_key]\n", " if d < min_dist:\n", " min_dist = d\n", " closest_part = (i, j)\n", " \n", " part1, part2 = closest_part\n", " node1, node2 = nodes[part1], nodes[part2]\n", " new_vec = [ (node1.vec[i] * node1.count + node2.vec[i] * node2.count ) / (node1.count + node2.count)\n", " for i in range(feature_num)]\n", " new_node = ClusterNode(vec=new_vec,\n", " left=node1,\n", " right=node2,\n", " distance=min_dist,\n", " id=currentclustid,\n", " count=node1.count + node2.count)\n", " currentclustid -= 1\n", " del nodes[part2], nodes[part1]\n", " nodes.append(new_node)\n", " \n", " self.nodes = nodes\n", " self.calc_label()\n", " \n", " def calc_label(self):\n", " \"\"\"\n", " 调取聚类的结果\n", " \"\"\"\n", " for i, node in enumerate(self.nodes):\n", " # 将节点的所有叶子节点都分类\n", " self.leaf_traversal(node, i)\n", "\n", " def leaf_traversal(self, node: ClusterNode, label):\n", " \"\"\"\n", " 递归遍历叶子节点\n", " \"\"\"\n", " if node.left == None and node.right == None:\n", " self.labels[node.id] = label\n", " if node.left:\n", " self.leaf_traversal(node.left, label)\n", " if node.right:\n", " self.leaf_traversal(node.right, label)\n", " \n", "# https://zhuanlan.zhihu.com/p/32438294" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 107 }, "colab_type": "code", "id": "LwD9Iots6871", "outputId": "be527c5e-3be7-40ee-c361-37a0ab57440b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n", " 2 2 2 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 2 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 2 0 0 0 1 0 0 1 2 1 0 1 0\n", " 0 0 0 0 0 0 1 1 0 0 0 1 0 0 1 0 0 0 1 1 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0\n", " 0 0]\n" ] } ], "source": [ "my = Hierarchical(3)\n", "my.fit(data)\n", "labels = np.array(my.labels)\n", "print(labels)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 281 }, "colab_type": "code", "id": "yJN0NPWn8F6K", "outputId": "f8238840-dd5e-45b3-e5b2-b4fa3836f7bb" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5wkdX3n8dd7ZxZxACGRSRD2x8TThwYTVJhDyOaUZDeJ/AjkLiZusp5iJBtnNeqdOS9m74GwyZrkHl4kCceS8UeE7EYxRM2K6PkTEYzrLYggYu5BlFlWMYwgi7iGuMvn/qia3Z7enq7qqerq6qn38/Hox3RXVVd9qrrnMz31+danFRGYmdnSt2zQAZiZWTWc8M3MGsIJ38ysIZzwzcwawgnfzKwhnPDNzBrCCX+ISbpb0jk1iGNCUkgaXWD+H0h6Zz+30eV5F0u6pci2yzLo1yvrdej1WEm6T9K6cqKzKjjh11SnX6b2X8iIeE5E3FR5cD2KiLdGxCWDjqOI9I/NM4qsY9CvV+vrsNg/oGWTtF3SA5IelfT/JA31+6TunPAbaDG/5JJG+hFLEww6qdbcHwMTEfEU4ELgjySdMeCYliwn/CHW+l+ApGWSfl/SP0t6SNL7Jf1oOm/u09yrJO0BPp1O/ztJ35a0T9LNkp7Tsu73SNom6UZJ3wd+TtKTJf0vSTPpc26R9OSWkDZI2iPpO5I2t6zrMknbWx7/rKTPS3pE0v2SLk6nny/pS+mnvfslXdbDsVgp6QOSZtP9v7LDMkd8qpV009ynSknPkPTZdN++I+m6dPrN6eJflvSYpJem0y+QdEe6H5+XdFrba/PfJd0JfF/SaNvrdVn6Gl0r6Xvp6Z7Jluefnh6L76Wv03WS/miBfZ+ZS5KSXpbu46np40skfajD6zC3T4+k+3R2y/reJum7kr4h6dycx//Z6fLr8yw/JyLujojH5x6mt3/XyzosPyf8peN1wK8ALwJOBr4L/O+2ZV4E/CTwS+njjwLPBH4MuB3Y0bb8bwJbgeOAW4C3AWcAPwP8KPAm4ImW5X8WeBawFrhU0k+2BylpVbrdvwTGgecBd6Szvw+8HDgBOB+YkvQrWTue/vdxAzADTACnAO/Lel4Hfwh8HPgRYEUaIxHxwnT+cyPi2Ii4TtLpwLuB3wGeCvwVsFPSk1rW9xvpfpwQEQc6bO/CNM4TgJ3Alen+HAV8EHgPyXF+L/Afu8T9WeCc9P4Lga+TvNZzjz/b4Tlz+3RCuk//mD5+AfBPwInA/wTeJUldtk16LD4O/G5EvC+ddkP6h7DT7Ya2518laT/wNeAB4MZu27MCIsK3Gt6A+4DHgEdabvuBW9qWWZfevwdY2zLvacAPgVGSJBjA07ts74R0mePTx+8Brm2Zvwz4AUnSa3/u3PpXtEz7IrA+vX8ZsD29/2bggzmPwRXA29u2MdphubOB2QXmXTx3zDqtA7gJuCS9fy0w3bofLcsF8IyWx9uAP2xb5p+AF7W8Nr/V4TWde70uAz7ZMu9U4Afp/RcC3wTUMv8W4I8WOE6vAna2vA8uAd6XPp4BTu/wOnQ6FhcD97Y8HkuXOanLe/RyYC/wcwXf7yMkHxj+B7B8kL97S/nmT/j19isRccLcDdjUZdnVwAfnPkWR/OIfBH68ZZn75+5IGpH0J+kpoEdJfnkh+WR3xPLp9KOBf+4Sw7db7u8Hju2wzMqF1iHpBZI+k56W2Qe8ui2ehawEZqLzp+hevAkQ8MX0FMtvdVl2NfDG1k+uaRwntyxzf+enHtJ+vI5OTzedDHwz0kyYY12fBf6DpJNIEud1wBpJE8DxHP4PKo9DMUXE/vRup9dxzquBz0fEZ3rYxhEi4mBE3ELyn9VUkXXZwpzwl477gXNb/0BExNER8c2WZVoTyG8CFwHrSJLCRDpdCyz/HeBfKX5+9f4u6/hbklMbKyPieODqtni6rXOVsouj309/jrVMO2nuTkR8OyJ+OyJOJjlVc5UWHplzP7C17XiPRcR7W5ZZbCvaB4BT2k6lrFxo4Yi4l+QPxuuAmyPieySJeyPJfzdPdHraImNr92qSY//21omSPprWBjrdPtplfaP4HH7fOOEvHVcDWyWtBpA0LumiLssfBzwOPESSAN/abeVp0ng38GeSTk7/Qzi77Zx1HjuAdZJ+PS1kPlXS81piejgi/lXSmSR/lPL4IkmS/BNJx0g6WtKaDvswS3Kq5GVp/L9FS3KR9GuSVqQPv0uSFA+mj/8FeHrL6t4BvDr9r0Tpds+XdFzOmLv5x3S7r02P0UXAmRnP+SzwWg6fr7+p7XG7WZL6y9MXmJ/X94AXAy+U9CdzEyPi3EhqA51u5wJI+jFJ6yUdm74ev0RS9/h0wZhsAU74S8efk3w6/rik7wFfICnALeRakvO73wS+mi6f5feAu4D/CzwM/Ck9vociYg9wHvDGdB13AM9NZ28CtqTxXwq8P+c6DwK/DDwD2ENyTvmlCyz+28B/I/lD9xzg8y3z/j2wS9JjJMfy9RHxjXTeZcA16embX4+I3em6riT543AvyTnwwiLi34D/RHJu/hHgZSRF6ce7PO2zJH8wb17gcfs29pMU5G9N9+msAvE+AvwCcK6kP+zlqSSnb/aSHMO3AW+IiH9YbCzWneafJjSzOpK0C7g6Iv560LHY8PInfLMakvQiSSelp3ReAZwGfGzQcdlwy3UFoKT7SM7VHQQORMRk23yRnFI4j6R4dHFE3F5uqGaN8iySU1rHkoxqeklEPDDYkGzY5Tqlkyb8yYj4zgLzzwN+lyThvwD484jodv7YzMwqVtYpnYtILtKJiPgCcIKkp5W0bjMzK0Hepk5BMvojgL+KiOm2+acw/8KQvem0ef+CStpIMjaYY4455oxnP/vZiwrazKypbrvttu9ExPhinps34a+JiG9J+jHgE5K+FhGtw706XRxzxLmi9A/FNMDk5GTs3r2754DNzJpM0sxin5vrlE5EfCv9+SBJU6f2i0D2Mv9KwBXAtxYblJmZlS8z4adXEB43dx/4ReArbYvtBF6eXnF4FrDPIwrMzOolzymdHydpyjW3/N9GxMckvRogIq4maWd6HsnVhvuBV/YnXDMzW6zMhB8RX+fwpe+t069uuR/Aa8oNzczMyuQrbc3MGsIJ38ysIZzwzcwawgnfzKwhnPDNzBrCCd/MrCGc8M3MGsIJ38ysIZzwzcwawgnfzKwhnPDNzBrCCd/MrCGc8M3MGsIJ38ysIZzwzcwawgnfzKwhnPDNzBrCCd/MrCFyJ3xJI5K+JOmGDvMuljQr6Y70dkm5YZqZWVF5vsR8zuuBe4CnLDD/uoh4bfGQzMysH3J9wpe0AjgfeGd/wzEzs37Je0rnCuBNwBNdlvlVSXdKul7SyuKhmZlZmTITvqQLgAcj4rYui30YmIiI04BPAtcssK6NknZL2j07O7uogM3MbHHyfMJfA1wo6T7gfcDPS9reukBEPBQRj6cP3wGc0WlFETEdEZMRMTk+Pl4gbDMz61Vmwo+IN0fEioiYANYDn46Il7UuI+lpLQ8vJCnumpVmxw6YmIBly5KfO3YMOiKz4dPLKJ15JG0BdkfETuB1ki4EDgAPAxeXE55Zktw3boT9+5PHMzPJY4ANGwYXl9mwUUQMZMOTk5Oxe/fugWzbhsvERJLk261eDffdV3U0ZoMl6baImFzMc32lrdXenj29TTezzpzwrfZWreptupl15oRvtbd1K4yNzZ82NpZMN7P8nPCt9jZsgOnp5Jy9lPycnnbB1qxXix6lY1alDRuc4M2K8id8M7OGcMI3M2sIJ3wzs4ZwwrdKuDWC2eC5aGt959YIZvXgT/jWd5s3H072c/bvT6abWXWc8K3v3BrBrB6c8K3v3BrBrB6c8K3vBVW3RjCrByf8hpsrqM7MQMThgmqZSd+tEczqwf3wG8695s2Gi/vh26K5oGrWHE74DeeCqllzOOE3nAuqZs2RO+FLGpH0JUk3dJj3JEnXSbpX0i5JE2UGaf2zlAqqbt9g1l0vrRVeD9wDPKXDvFcB342IZ0haD/wp8NIS4rMKLIVe827fYJYt1yd8SSuA84F3LrDIRcA16f3rgbWSVDw8s3zcvsEsW95TOlcAbwKeWGD+KcD9ABFxANgHPLV9IUkbJe2WtHt2dnYR4Zp15tFGZtkyE76kC4AHI+K2bot1mHbEAP+ImI6IyYiYHB8f7yFMs+482sgsW55P+GuACyXdB7wP+HlJ29uW2QusBJA0ChwPPFxinFZjmzbB6GhS9B0dTR5XzaONzLJlJvyIeHNErIiICWA98OmIeFnbYjuBV6T3X5IuM5hLeK1SmzbBtm1w8GDy+ODB5HHVSX8pjTYy65eeWitIOgf4vYi4QNIWYHdE7JR0NPA3wPNJPtmvj4ivd1uXWyssDaOjh5N9q5EROHCg+njMlroirRV6+sariLgJuCm9f2nL9H8Ffm0xAdhw65Tsu003s8HxlbZWyMhIb9PNbHCc8K2QuYub8k43s8Fxwl/i1q1Liphzt3Xryl3/VVfB1NThT/QjI8njq64qdztum2C1VsYbtIo3eUQM5HbGGWeE9dfatRHJ15rMv61dO+jIerN9e8TY2Px9GBtLppsNXBlv0B7WQTJYZlF511+AsoR1a24xTINm/SUtVmtlvEF7WIe/AMWWNLdNsFor4w1a0ZvcCd9qz20TrNbKeINW9CZ3wl/C1q7tbfpCsmpJ/a41uW2C1VoZb9Cq3uSLPflf9Oaibf9t3x4hza8DSeXWkqoqqG7fHrF6dRL/6tUu2FrNlPEGzbkOXLS1TqqoJbmgalYtF22toypqSS6omg0PJ/wlrIpakguqZsPDCX+Rqrrys0jBtIpakguq1le+xLpciz35X/Q2zEXbKguVRQumVdSSXFC1vvAl1h3hom21qipUumBqjeY3eEdFirZO+IuwbFnn1gQSPLHQ17z3YTtVxWE2EH6Dd+RROhWrqlDpgqk1mt/gpXPCX4SqCpUumFqj+Q1evqyT/MDRwBeBLwN3A5d3WOZiYBa4I71dkrXeYS7aRlRXqMzaTnsL5PbWx1NTESMjybyRkeTxIPbFhV1bFL9xjkCBom2ehC/g2PT+cmAXcFbbMhcDV/ay4WFP+HUwNTU/2c/d5pJ61vyIagZCeLCFWXmKJPyeiraSxoBbgKmI2NUy/WJgMiJem3ddw1y0rYvR0c5fFj4yAgcOZM+HagZCeLCFWXn6XrSVNCLpDuBB4BOtyb7Fr0q6U9L1klYusJ6NknZL2j07O7uYeK1Fp2TeOj1rPlTTGsHtF8zqIVfCj4iDEfE8YAVwpqSfalvkw8BERJwGfBK4ZoH1TEfEZERMjo+PF4nbOPw9sgtNz5oP1QyE8GALs3roaZRORDwC3AS8uG36QxHxePrwHcAZpURXY2Vc8b1pU3LaRUp+btrU2/M3buw+PWs+JAMeli+fP3/58t4GQmTtxzANtthx1w4mrphg2eXLmLhigh139fbC5nl+0W2YLVZmwpc0LumE9P6TgXXA19qWeVrLwwuBe8oMsm527EiS5sxMUoKcmUke95L0N22Cbdvmn37Ztq33pN/NmjVJAm41OppMb9X+3bfdvgu3XZ792LABpqeTc/ZS8nN6OpleJzvu2sHGD29kZt8MQTCzb4aNH96YOyHneX7RbZgVkVm0lXQaySmaEZI/EO+PiC2StpBUi3dK+mOSRH8AeJikqPu1BVfKcBdtyyhC5imoFl1HnjiL7ksZ+1EXE1dMMLPvyIOx+vjV3PeG+0p5ftFtmLm1QsXKuOK726fovC9J1jryxFl0X8rYj7pYdvkygiODFuKJt2QfjDzPL7oNM7dWqFgZRcg8BdWi68gTZ9F9KWM/6mLV8Z13eqHpi3l+0W2YFeGEvwhlFCHzFFSLriNPnEX3pYz9qIuta7cytnz+wRhbPsbWtfkORp7nF92GWSGLvWKr6G3Yr7Qt44rvPG0PstShtUIZ+1EX2+/cHqvfvjp0mWL121fH9jt7Oxh5nr/2mrXBZRy6rb1mbYc19dfUDVMxcvlIcBkxcvlITN0wxC9aw+B++M00N1po//7D08bGDo+AyZpv1dv0kU1s273tiOlTk1Ncdf5VjYnBFs9F24byF6QMn9EtoxyMI4c1jWiEA5dWM6ypDjHY4rlo21BZLQvc0qB+OiXabtOXagw2GE74Q8xfkDJ8RtR5+NJC05dqDDYYjU34RVsj5Hl+0dYJWbZuhaOOmj/tqKP8BSkLqUNLg41ndB6+1Dp900c2MbplFF0uRreMsukj8984RfcjTwx5tlN0fi5l9DCxwxZb7S16G+QonaL92fM8P08v+jL2Y/ny+etfvnx+HP7+iMT2O7fH2NaxeaNjxraO9TwKpwzdRshM3TA1L8a529wyZe1H1iidrO0UnZ+Lv0ihIzxKpzdFi5l5nl9FywEXZfMblpYGWQXVqvYjaztF5+cLYsJv8A5ctO1R0WJmnufn6UVflIuy+e3Z1/mgLDR9ULIKqlXtR9Z2is7PF4Tf4GVrZMIvWszM8/wqWg64KJvfsLQ0yCqoVrUfWdspOj9fEH6Dl62RCb9oMTPP86toOeCibH7D0tIgq6Ba1X5kbafo/HxB+A1eusWe/C96G3RrhSraCWQtkyeGMtZhiaJtE6qS1XqhjP3I094hazunXnnqvHWceuWppcfpN/iRcNG2WmW0LMizjrkvF2k3NQVX+Qr4JWnuC1L2//DwG2Ns+RjTvzzNhp8upx/GumvX8alvfOqI6Wt/Yi2ffPknK1uHLY5bK1SsjMEDdRnpY/VSxSgcXb7wlxjEW/LlgzLWYYvjUToVK2PwQF1G+li9DMtoIhtOTviLUMbggbqM9LF6GZbRRDac8nyJ+dGSvijpy5LulnR5h2WeJOk6SfdK2iVpoh/B5pXnauwiV2yXMXhg69YjE/fISO8jfepw5XkZl9D3u51Anm3k2U6edRSRZ3TLumvXoct16Lbu2nU97cfan1jbcdvt08tYR6asN3Ad3uB54qhLnFmyqrqAgGPT+8uBXcBZbctsAq5O768Hrstab79G6eS5GruMK7bLGOWT1Xph+/aIZcvmz1+27PC26nDleRmX0FfRTiBrG3m2k2cdZeg2uqV9dE2nUTZl7EeeY174i1yy3sB1eIPXME6qGqUjaQy4BZiKiF0t0/8PcFlE/KOkUeDbwHh0WXm/irZ5iqF1uGI7T0F2GPrdl1FkrKKdQJ4e8FnbqUMf+TzF0jL2o5IWDsPwBofaxdn3oq2kEUl3AA8Cn2hN9qlTgPsBIuIAsA94aof1bJS0W9Lu2dnZxcSbKU8xtA5XbOcpyA5Dv/syioxVtBPI0wM+azvD0ke+jP2opHg8DG/wPHHUJc4cciX8iDgYEc8DVgBnSvqptkU6few44tN9RExHxGRETI6Pj/cebQ55iqF1uGI7T0F2GPrdl1FkrKKdQJ4e8FnbGZY+8mXsRyXF42F4g+eJoy5x5tDTKJ2IeAS4CXhx26y9wEqA9JTO8cDDJcTXszwF1TpcsZ2nIJsVZx32o4xL6KtoJ5CnB3zWdvL2ke+nPMXSMvajkhYOw/AGzxNHXeLMI+skPzAOnJDefzLwOeCCtmVew/yi7fuz1tvP1gpltCyoQp4YsvalDleeV3Gpf1b/9jzyrCNrmTLaHmQtkzU/T7E0az/yHItKWlEUfYNX9QtQlziiz0VbSacB1wAjJP8RvD8itkjakm54p6Sjgb8Bnk/yyX59RHy923oHeaVtGa0RrDxZ7QSqaDdQRhx54qxiX6s6XgPX0F9kt1boUV2K/5ao5Ms0KogjT5xV7OuwfNlLYQ39RXZrhR4NUVG9ESr5Mo0K4sgTZxX72pj2DP5F7lkjE/4QFdUboZIv06ggjjxxVrGvjWnP4F/knjUy4Q9TUb0Jtq7dyvJly+dNW75s+bwv02gfSjiikSNGjBRtv5DnSz2Wtf3KLGPZvPnd9iPvvnbbRp79LGuETRntLPoq7y9yFW0PhqS1QiMT/oYNSV1n9WqQkp9LvM5Te5IWfHzrnluPuFjoYBzk1j23Hno8V6ic2TdDEMzsm2Hjhzf2lKQ2/PQGpn95mtXHr0aI1cevnlfovHXPrTzBE/Oe8wRPzIuj237k3ddu28izn1n7kUcZx7Pv8vwizxV2Z2aSpgczM8njMhNyFdsoSSOLtlYvw9IKoIwWEEX3tS4F7KFRRWF3qbVWMOunYWkFUEYLiKL7WpcC9tCoorA7RMVjJ3wbuGFpBVBGC4ii+1qXAvbQqKKwO0TFYyd8G3i9aevarRw1ctS8aUeNHNVzK4CF1t2LboXKMlpAFN3XSloeVLidvqtihMYQjQJxwm+4utSb2mtJrY+vOv8qpianDn3KHdEIU5NTXHX+4W9yf+vNb+243oWmd5JVqMyKI2+xtNu+rlm1puMonTWr1vS0jaKq2k7fVTFCY4hGgbho23B1uFixjAJhGV+qXUWhsi5XFdvwctHWFq0O9aa6FAiriKMuVxVbMznhN1wd6k11KRBWEUddriq2ZnLCb7g61JvKKBCeeuKpPU3vVxxFt7FkiqVWS074DVdVvanb6Jc8BcKsy/zvfs3dnHzsyfOmnXzsydz9mrtzx1hGHEW3sWSKpU0z6KFuObloa31XtD97GX3m67AftkRV3Jff/fCt1oqOPCmjZUEZPILGOnJrBbPDio48KaNlQRk8gsY6qsNQt5yc8K3vio48KaNlQRk8gsY6qsNQt5wyE76klZI+I+keSXdLen2HZc6RtE/SHent0v6E2zxFa0F1qCVltROA7sXQvH3mBz3CJq9NH9nE6JZRdLkY3TLKpo9sKi3GvGrf636Y1GGoW06jOZY5ALwxIm6XdBxwm6RPRMRX25b7XERcUH6IzdVeC5prewD5akFFn1+mbu0E2ouhcy0NgEPF0Kw+83PLbf7UZvbs28Oq41exde3WUoupZWxj00c2sW33tkOPD8bBQ49bW0X0U57jbT2Y+2XavDk5jbNqVZLsl0JrBUn/AFwZEZ9omXYO8Hu9JHwXbbMVrQXVoW0CVPPl4MMiT2//fltKx7OJKivaSpoAng/s6jD7bElflvRRSc9Z4PkbJe2WtHt2drbnYJumaC2oLrWkKr4cfFjk6e3fb0vpeFpvcid8SccCfw+8ISIebZt9O7A6Ip4L/CXwoU7riIjpiJiMiMnx8fHFxtwYRWtBdaklVfHl4MMiT2//fltKx9N6kyvhS1pOkux3RMQH2udHxKMR8Vh6/0ZguaQTS420gYrWgupSSyraTmAptRvI09u/35bS8bTe5BmlI+BdwD0R8WcLLHNSuhySzkzX+1CZgTbRhg3wilfASPrhb2QkeZy3FlSXNt0bfnoDZ684e960s1ecnbudwFJqN5Cnt3+/LaXjab3JLNpK+lngc8BdwBPp5D8AVgFExNWSXgtMkYzo+QHwXyPi893W66Jttoqv2O6b9pEpc6pOdGZLgVsrLFF1GWVTVB1GppgtFW6tsETVZZRNUXUYmWJmTvi1VpdRNkXVYWSKmTnhL6gWLQlyjLKpQ5xZ6jAypU7c1sAGxQm/g7li6cwMRBxuSVB1Ms0aZVOXOLOsWbWGZW1vtWUsY82qNQOKaHDm2hrM7JshiENtDZz0rQou2nYwLMXSoYnTl/If4mNhRbloW7JhKZYOTZy+lP8QHwsbJCf8DoalWDo0cfpS/kN8LGyQnPA7qEtLgixDE6cv5T/Ex8IGyQm/g7q0JMgyNHEO0aX8Rb+cJGsEzjAdC1t6XLQ1SxVtAdH+xSKQfHp3QrcyuWhrVoLp26Z7mt5u86c2z0v2APt/uJ/Nn9pcODazMjjhm6WKtoDwCByrOyd8s1TRFhAegWN154RvliraAsIjcKzunPDNUkW/nMQjcKzuPErHzGyIeJSOmZllcsI3M2uIPF9ivlLSZyTdI+luSa/vsIwk/YWkeyXdKen0/oRr7YahH76Z1cNojmUOAG+MiNslHQfcJukTEfHVlmXOBZ6Z3l4AbEt/Wh+1f8n5XD98qF97BTMbvMxP+BHxQETcnt7/HnAPcErbYhcB10biC8AJkp5WerQ2z+bNh5P9nP37k+lmZu16OocvaQJ4PrCrbdYpwP0tj/dy5B8FJG2UtFvS7tnZ2d4itSMMSz98M6uH3Alf0rHA3wNviIhH22d3eMoR4z0jYjoiJiNicnx8vLdI7QjD0g/fzOohV8KXtJwk2e+IiA90WGQvsLLl8QrgW8XDs26GpR++mdVDnlE6At4F3BMRf7bAYjuBl6ejdc4C9kXEAyXGaR0MSz98M6uHPKN01gD/GbhL0h3ptD8AVgFExNXAjcB5wL3AfuCV5YdqnWzY4ARvZvlkJvyIuIXO5+hblwngNWUFZWZm5fOVtmZmDeGEb2bWEE74ZmYN4YRvZtYQTvhmZg3hhG9m1hBO+GZmDeGEb2bWEE74ZmYN4YRvZtYQTvhmZg3hhG9m1hBO+GZmDeGEb2bWEE74ZmYN4YRvZtYQTvhmZg3hhG9m1hB5vsT83ZIelPSVBeafI2mfpDvS26Xlh2lmZkXl+RLz9wBXAtd2WeZzEXFBKRGZmVlfZH7Cj4ibgYcriMXMzPqorHP4Z0v6sqSPSnpOSes0M7MS5Tmlk+V2YHVEPCbpPOBDwDM7LShpI7ARYNWqVSVs2szM8ir8CT8iHo2Ix9L7NwLLJZ24wLLTETEZEZPj4+NFN21mZj0onPAlnSRJ6f0z03U+VHS9ZmZWrsxTOpLeC5wDnChpL/AWYDlARFwNvASYknQA+AGwPiKibxGbmdmiZCb8iPiNjPlXkgzbNDOzGvOVtmZmDeGEb2bWEE74ZmYN4YRvZtYQTvhmZg3hhG9m1hBO+GZmDeGEb2bWEE74ZmYN4YRvZtYQTvhmZg3hhG9m1hBO+GZmDeGEb2bWEE74ZmYN4YRvZtYQTvhmZg3hhG9m1hBO+GZmDZGZ8CW9W9KDkr6ywHxJ+gtJ90q6U9Lp5YdpZmZF5fmE/x7gxV3mnws8M71tBLYVD8vMzMqWmfAj4mbg4S6LXARcG4kvACdIelpZAZqZWTlGS1jHKcD9LY/3ptMeaF9Q0kaS/wIAHl/oNFHNnAh8Z9BB5Mw+lGoAAATHSURBVOA4yzUMcQ5DjOA4y/asxT6xjISvDtOi04IRMQ1MA0jaHRGTJWy/rxxnuRxneYYhRnCcZZO0e7HPLWOUzl5gZcvjFcC3SlivmZmVqIyEvxN4eTpa5yxgX0QccTrHzMwGK/OUjqT3AucAJ0raC7wFWA4QEVcDNwLnAfcC+4FX5tz29CLiHQTHWS7HWZ5hiBEcZ9kWHaciOp5uNzOzJcZX2pqZNYQTvplZQ1SS8CWNSPqSpBs6zHuSpOvS1gy7JE1UEVMnGXFeLGlW0h3p7ZIBxXifpLvSGI4YnlWXVhc54jxH0r6W43npAGI8QdL1kr4m6R5JZ7fNr8uxzIqzDsfyWS3bv0PSo5Le0LbMwI9nzjgHfjzTOP6LpLslfUXSeyUd3Ta/59xZxjj8PF4P3AM8pcO8VwHfjYhnSFoP/Cnw0oriatctToDrIuK1FcazkJ+LiIUuEGltdfECklYXL6gqsDbd4gT4XERcUFk0R/pz4GMR8RJJRwFjbfPrciyz4oQBH8uI+CfgeZB8cAK+CXywbbGBH8+cccKAj6ekU4DXAadGxA8kvR9YT9LqZk7PubPvn/AlrQDOB965wCIXAdek968H1krqdDFXX+WIc1i41UUOkp4CvBB4F0BE/FtEPNK22MCPZc4462Yt8M8RMdM2feDHs81CcdbFKPBkSaMkf+Tbr2/qOXdWcUrnCuBNwBMLzD/UmiEiDgD7gKdWEFe7rDgBfjX9V/R6SSu7LNdPAXxc0m1KWlW0W6jVRdWy4gQ4W9KXJX1U0nOqDA54OjAL/HV6Gu+dko5pW6YOxzJPnDDYY9luPfDeDtPrcDxbLRQnDPh4RsQ3gbcBe0ja1OyLiI+3LdZz7uxrwpd0AfBgRNzWbbEO0yodK5ozzg8DExFxGvBJDv9lrdqaiDid5N/j10h6Ydv8gR/PVFactwOrI+K5wF8CH6o4vlHgdGBbRDwf+D7w+23L1OFY5olz0MfykPSU04XA33Wa3WHaQMaFZ8Q58OMp6UdIPsH/BHAycIykl7Uv1uGpXY9nvz/hrwEulHQf8D7g5yVtb1vmUGuG9F+X4+nenbMfMuOMiIci4vH04TuAM6oN8VAc30p/Pkhy7vHMtkVq0eoiK86IeDQiHkvv3wgsl3RihSHuBfZGxK708fUkibV9mUEfy8w4a3AsW50L3B4R/9JhXh2O55wF46zJ8VwHfCMiZiPih8AHgJ9pW6bn3NnXhB8Rb46IFRExQfLv06cjov2v1E7gFen9l6TLVPpXP0+cbecaLyQp7lZK0jGSjpu7D/wi0N5xdOCtLvLEKemkufONks4keS8+VFWMEfFt4H5Jc50H1wJfbVts4McyT5yDPpZtfoOFT5MM/Hi2WDDOmhzPPcBZksbSWNZyZM7pOXdWNUpnHklbgN0RsZOkGPU3ku4l+eu0fhAxddIW5+skXQgcIInz4gGE9OPAB9P34ijwtxHxMUmvhsKtLqqO8yXAlKQDwA+A9VX/oQd+F9iR/nv/deCVNTyWeeKsw7FE0hjwC8DvtEyr3fHMEefAj2dE7JJ0PcnppQPAl4DpornTrRXMzBrCV9qamTWEE76ZWUM44ZuZNYQTvplZQzjhm5k1hBO+mVlDOOGbmTXE/wedMVUnY8SBMwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# visualize result\n", "\n", "cat1 = data[np.where(labels==0)]\n", "cat2 = data[np.where(labels==1)]\n", "cat3 = data[np.where(labels==2)]\n", "\n", "plt.scatter(cat1[:,0], cat1[:,1], color='green')\n", "plt.scatter(cat2[:,0], cat2[:,1], color='red')\n", "plt.scatter(cat3[:,0], cat3[:,1], color='blue')\n", "plt.title('Hierarchical clustering with k=3')\n", "plt.xlim(4, 8)\n", "plt.ylim(1, 5)\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 107 }, "colab_type": "code", "id": "7xilKyap8ghX", "outputId": "0a58379a-679a-4d0b-fe13-241585f60d81" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n", " 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 2 0 2 0 1 0 1 1 0 2 0 2 0 2 2 2 2 0 0 2 0\n", " 0 0 0 0 0 2 2 2 2 0 2 0 0 2 2 2 2 0 2 1 2 2 2 0 1 2 0 2 0 0 0 0 1 0 0 0 0\n", " 0 0 2 2 0 0 0 0 2 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 2 0\n", " 0 0]\n" ] } ], "source": [ "sk = cluster.AgglomerativeClustering(3)\n", "sk.fit(data)\n", "labels_ = sk.labels_\n", "print(labels_)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 281 }, "colab_type": "code", "id": "hlFZeDdW9Bzr", "outputId": "7a837637-70e6-4c40-b6f2-c6a651fc4cae" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5wkdX3n8dd7fqw4gJDIJPzY3Vk9fWgwosIcQjanJLNJ5IeQu5i4yXoGI9k4q1HvkvNiuAcCyZofDy9iwu2S1Rghs1EMUYMEPUVFRCPegggSTB5EGVjBsIIs4hri7n7uj6phe3p6pqqnqqurp97Px6Mf211VXfWp6t5P99TnW59WRGBmZivfUL8DMDOzajjhm5k1hBO+mVlDOOGbmTWEE76ZWUM44ZuZNYQT/gCTdJekM2oQxzpJIWlkkfm/K+k9vdzGEs87X9LNRbZdln6/XlmvQ7fHStK9kjaUE51VwQm/pjr9Z2r/DxkRz4uIGysPrksR8faIuKDfcRSRftg8q8g6+v16tb4Oy/0ALZukGUkPSnpM0j9LGuj3Sd054TfQcv6TSxruRSxN0O+kWnN/AKyLiKcB5wK/L+mUPse0YjnhD7DWvwIkDUn6HUn/IulhSR+U9MPpvLlvc6+VdB/w6XT630j6lqS9km6S9LyWdb9P0nZJ10v6HvBTkp4q6X9Lmk2fc7Okp7aEtEnSfZK+LenClnVdLGmm5fFPSvqCpEcl3S/p/HT62ZK+nH7bu1/SxV0cizWSPiRpT7r/l3dYZsG3Wkk3zn2rlPQsSZ9N9+3bkq5Op9+ULv4VSY9LemU6/RxJt6f78QVJJ7W9Nv9T0h3A9ySNtL1eF6ev0VWSvpue7plsef7J6bH4bvo6XS3p9xfZ99m5JCnpVek+npg+vkDSRzq8DnP79Gi6T6e3rO8dkr4j6RuSzsx5/J+bLr8xz/JzIuKuiHhi7mF6+w/drMPyc8JfOd4I/DzwUuB44DvA/2lb5qXAjwE/lz7+GPBs4EeA24Cdbcv/CrAVOBK4GXgHcArwE8APA28BDrYs/5PAc4Ap4CJJP9YepKS16Xb/DBgHXgjcns7+HvBq4GjgbGBa0s9n7Xj618d1wCywDjgB+EDW8zr4PeATwA8Bq9MYiYiXpPNfEBFHRMTVkk4G3gv8BvB04M+BayU9pWV9v5zux9ERsb/D9s5N4zwauBa4PN2fVcCHgfeRHOf3A/95ibg/C5yR3n8J8HWS13ru8Wc7PGdun45O9+kf0scvBv4JOAb4Y+AvJGmJbZMei08AvxkRH0inXZd+EHa6Xdf2/G2S9gFfAx4Erl9qe1ZARPhWwxtwL/A48GjLbR9wc9syG9L7dwNTLfOOA34AjJAkwQCeucT2jk6XOSp9/D7gqpb5Q8D3SZJe+3Pn1r+6ZdqXgI3p/YuBmfT+W4EP5zwGlwHvbNvGSIflTgf2LDLv/Llj1mkdwI3ABen9q4AdrfvRslwAz2p5vB34vbZl/gl4actr82sdXtO51+ti4IaWeScC30/vvwT4JqCW+TcDv7/IcXotcG3L++AC4APp41ng5A6vQ6djcT5wT8vjsXSZY5d4j14C7AZ+quD7fZjkC8P/Akb7+X9vJd/8Db/efj4ijp67AVuWWHYC+PDctyiS//gHgB9tWeb+uTuShiX9YXoK6DGS/7yQfLNbsHw6/TDgX5aI4Vst9/cBR3RYZs1i65D0YkmfSU/L7AVe1xbPYtYAs9H5W3Q33gII+FJ6iuXXllh2Avit1m+uaRzHtyxzf+enPqn9eB2Wnm46HvhmpJkwx7o+C/wnSceSJM6rgfWS1gFHcegvqDyejCki9qV3O72Oc14HfCEiPtPFNhaIiAMRcTPJX1bTRdZli3PCXznuB85s/YCIiMMi4psty7QmkF8BzgM2kCSFdel0LbL8t4F/o/j51fuXWMdfk5zaWBMRRwFXtMWz1DrXKrs4+r3037GWacfO3YmIb0XEr0fE8SSnarZp8ZE59wNb2473WES8v2WZ5baifRA4oe1UyprFFo6Ie0g+MN4I3BQR3yVJ3JtJ/ro52Olpy4yt3etIjv07WydK+lhaG+h0+9gS6xvB5/B7xgl/5bgC2CppAkDSuKTzllj+SOAJ4GGSBPj2pVaeJo33An8i6fj0L4TT285Z57ET2CDpl9JC5tMlvbAlpkci4t8knUryoZTHl0iS5B9KOlzSYZLWd9iHPSSnSl6Vxv9rtCQXSb8oaXX68DskSfFA+vhfgWe2rO7dwOvSv0qUbvdsSUfmjHkp/5Bu9w3pMToPODXjOZ8F3sCh8/U3tj1ut4ek/vLMRebn9V3gZcBLJP3h3MSIODOS2kCn25kAkn5E0kZJR6Svx8+R1D0+XTAmW4QT/srxLpJvx5+Q9F3giyQFuMVcRXJ+95vAP6bLZ/lt4E7g/wGPAH9El++hiLgPOAv4rXQdtwMvSGdvAS5N478I+GDOdR4AXg48C7iP5JzyKxdZ/NeB/0HyQfc84Ast8/4jcIukx0mO5Zsi4hvpvIuBK9PTN78UEbvSdV1O8uFwD8k58MIi4t+B/0Jybv5R4FUkReknlnjaZ0k+MG9a5HH7NvaRFOQ/n+7TaQXifRT4GeBMSb/XzVNJTt/sJjmG7wDeHBF/t9xYbGmaf5rQzOpI0i3AFRHxl/2OxQaXv+Gb1ZCkl0o6Nj2l86vAScDH+x2XDbZcVwBKupfkXN0BYH9ETLbNF8kphbNIikfnR8Rt5YZq1ijPITmldQTJqKZXRMSD/Q3JBl2uUzppwp+MiG8vMv8s4DdJEv6LgXdFxFLnj83MrGJlndI5j+QinYiILwJHSzqupHWbmVkJ8jZ1CpLRHwH8eUTsaJt/AvMvDNmdTpv3J6ikzSRjgzn88MNPee5zn7usoM3MmurWW2/9dkSML+e5eRP++oh4QNKPAJ+U9LWIaB3u1enimAXnitIPih0Ak5OTsWvXrq4DNjNrMkmzy31urlM6EfFA+u9DJE2d2i8C2c38KwFXAw8sNygzMytfZsJPryA8cu4+8LPAV9sWuxZ4dXrF4WnAXo8oMDOrlzyndH6UpCnX3PJ/HREfl/Q6gIi4gqSd6VkkVxvuA17Tm3DNzGy5MhN+RHydQ5e+t06/ouV+AK8vNzQzMyuTr7Q1M2sIJ3wzs4ZwwjczawgnfDOzhnDCNzNrCCd8M7OGcMI3M2sIJ3wzs4ZwwjczawgnfDOzhnDCNzNrCCd8M7OGcMI3M2sIJ3wzs4ZwwjczawgnfDOzhnDCNzNrCCd8M7OGyJ3wJQ1L+rKk6zrMO1/SHkm3p7cLyg3TzMyKyvMj5nPeBNwNPG2R+VdHxBuKh2RmZr2Q6xu+pNXA2cB7ehuOmZn1St5TOpcBbwEOLrHML0i6Q9I1ktYUD83MzMqUmfAlnQM8FBG3LrHYR4F1EXEScANw5SLr2ixpl6Rde/bsWVbAZma2PHm+4a8HzpV0L/AB4KclzbQuEBEPR8QT6cN3A6d0WlFE7IiIyYiYHB8fLxC2mZl1KzPhR8RbI2J1RKwDNgKfjohXtS4j6biWh+eSFHfNyrNzJ6xbB0NDyb87d/Y7IrOB080onXkkXQrsiohrgTdKOhfYDzwCnF9OeGYkyX3zZti3L3k8O5s8Bti0qX9xmQ0YRURfNjw5ORm7du3qy7ZtwKxblyT5dhMTcO+9VUdj1leSbo2IyeU811faWv3dd193082sIyd8q7+1a7ubbmYdOeFb/W3dCmNj86eNjSXTzSw3J3yrv02bYMeO5Jy9lPy7Y4cLtmZdWvYoHbNKbdrkBG9WkL/hm5k1hBO+mVlDOOGbmTWEE75Vw60RzPrORVvrPbdGMKsFf8O33rvwwkPJfs6+fcl0M6uME771nlsjmNWCE771nlsjmNWCE771vqDq1ghmteCE33RzBdXZWYg4VFAtM+m7NYJZLbgfftO517zZQHE/fFs+F1TNGsMJv+lcUDVrDCf8pnNB1awxcid8ScOSvizpug7zniLpakn3SLpF0royg7QeWkkFVbdvMFtSN60V3gTcDTytw7zXAt+JiGdJ2gj8EfDKEuKzKqyEXvNu32CWKdc3fEmrgbOB9yyyyHnAlen9a4ApSSoenllObt9glinvKZ3LgLcABxeZfwJwP0BE7Af2Ak9vX0jSZkm7JO3as2fPMsI1W4RHG5llykz4ks4BHoqIW5darMO0BQP8I2JHRExGxOT4+HgXYZpl8Ggjs0x5vuGvB86VdC/wAeCnJc20LbMbWAMgaQQ4CnikxDitzrZsgZGRpOg7MpI8rppHG5llykz4EfHWiFgdEeuAjcCnI+JVbYtdC/xqev8V6TL9uYTXqrVlC2zfDgcOJI8PHEgeV530V9JoI7Me6aq1gqQzgN+OiHMkXQrsiohrJR0G/BXwIpJv9hsj4utLrcutFVaIkZFDyb7V8DDs3199PGYrXJHWCl394lVE3AjcmN6/qGX6vwG/uJwAbMB1SvZLTTezvvGVtlbM8HB3082sb5zwrZi5i5vyTjezvnHCX+k2bEiKmHO3DRvKXf+2bTA9fegb/fBw8njbtnK347YJVmM779zJusvWMXTJEOsuW8fOO7t/f5axjizuh7+SbdgAn/rUwulTU3DDDdXHs1ztbRMgGXLpUThWAzvv3Mnmj25m3w8OvT/HRsfY8fIdbHp+vvdnN+soUrR1wl/JlupuMUijZv0jLVZj6y5bx+zehe/PiaMmuPfN95a+Dv8Aiq1sbptgNXbf3s7vw8Wm92odeTjhW/25bYLV2NqjOr8PF5veq3Xk4YS/kk1NdTd9MVkF014XVN02wWps69RWxkbnvz/HRsfYOpX//VnGOnKJiL7cTjnllLAem5mJkCKSM/bJTUqmd7OOsbH56xgbO7SOrPll7svERBL/xET56zcrYOaOmZh450ToYsXEOydi5o7u359510HS4WBZeddF25WsjGJn1jpcUDWrlIu21lkZxc6sdbigajYwnPBXsjKKnVnrcEHVbGA44S9XVVd+FimYllHszFqHC6rWQ1Vcfdooyz35X/Q20EXbKguVRQumZRQ7s9bhgqr1wMwdMzG2dSy4mCdvY1vHllUQXUlw0bZiVRUqXTC1BivjCtaVyEXbqlVVqHTB1BqsqqtPm8QJfzmqKlS6YGoNVtXVp03ihL8cVRUqXTC1Bqvs6tMGyUz4kg6T9CVJX5F0l6RLOixzvqQ9km5Pbxf0JtyaqOoHs7O2s2kTnH76/Oecfvr8OLZsSX53Vkr+7fTj4lWMOHI/e+vSpudvYsfLdzBx1ARCTBw10VXLYesgq6oLCDgivT8K3AKc1rbM+cDl3VSLB3qUTl1MT88foTN3m57ONz+imhFHVY1qMmsAqhqlI2kMuBmYjohbWqafD0xGxBvyrmugR+nUxchI5x8LHx6G/fuz50M1I308msisND0fpSNpWNLtwEPAJ1uTfYtfkHSHpGskrVlkPZsl7ZK0a8+ePcuJ11p1Suat07PmQzUjfTyayKwWciX8iDgQES8EVgOnSvrxtkU+CqyLiJOAG4ArF1nPjoiYjIjJ8fHxInEbHPod2cWmZ82Hakb6eDSRWS10NUonIh4FbgRe1jb94Yh4In34buCUUqKrszKKkHkKqkvZvHnp6VnzIRnRMzo6f/7oaHcjfbL2Y4BGExW9lD/P890uwPolzyidcUlHp/efCmwAvta2zHEtD88F7i4zyNqZ+1Ht2dmkBDk7mzzuJulv2QLbt88//bJ9e/dJfynr1ycJuNXISDK9Vftv3y71W7jt8uxHVaOaCpr7IenZvbMEwezeWTZ/dHPuhJzn+UW3YVZEZtFW0kkkp2iGST4gPhgRl0q6lKRafK2kPyBJ9PuBR0iKul9bdKUMeNG2jCJknoJq0XXkibPovpSxHzVR9FL+PM93uwArqkjR1r10lmNoKPlm306CgwfzrWOpb9F5X5OsdeSJs+i+lLEfNTF0yRDBwpiFOPi27GOR5/lFt2HmXjpVK6MImaegWnQdeeIsui9l7EdNFL2UP8/z3S7A+skJfznKKELmKagWXUeeOIvuSxn7URNFL+XP83y3C7C+Wu4VW0VvA3+lbRk94KenI4aHkytPh4fnXwGb19TU/CtYp6a630bRfSljP2qi6I9R53n+1JVT83q8T1051WFNvTV93XQMXzIcXEwMXzIc09cN7mvWNLgffkPNjRbat+/QtLGxQyNgsuZb5bb8/Ra279q+YPr05DTbzt7WmBhs+Vy0bSr/QMrAGbl0hAOxcFTTsIbZf1E1o5rqEIMtn4u2TeUfSBk4nRLtUtNXagzWH074g8w/kDJwhtV59NJi01dqDNYfzU34RVsj5Hl+0dYJWbZuhVWr5k9btco/kLKIOrQ02HxK59FLrdO3/P0WRi4dQZeIkUtH2PL38983RfcjTwx5tlN0fh51eM1WlOVWe4ve+jpKp2h/9jzPz9OLvoz9GB2dv/7R0flxlDGaaAWYuWMmxraOzRsdM7Z1rOtROGVYaoTM9HXT82Kcu80tU9Z+ZI3SydpO0fl51Ok1qxM8SqdLRYuZeZ5fRcsBF2VzG5SWBlkF1ar2I2s7ReeXEUNTuWjbraLFzDzPz9OLvigXZXO7b2/nY7LY9H7JKqhWtR9Z2yk6v4wYrHvNTPhFi5l5nl9FywEXZXMblJYGWQXVqvYjaztF55cRg3WvmQm/aDEzz/OraDngomxug9LSIKugWtV+ZG2n6PwyYrBlWO7J/6K3vrdWqKKdQNYyeWIoYx0WEcXbJlQlq/VCGfuRp71D1nZOvPzEees48fITS49zUF6zKuGibcXKaFmQZx1zPy7SbnoatvkS+JVo7gdS9v3g0PtibHSMHS/fwabnl9MOY8NVG/jUNz61YPrUM6a44dU3VLYOWx63VqhaGaNj6jLSx2qlipEpumTx3zCIt+XLB2Wsw5bHo3SqVsbomLqM9LFa8cgU6yUn/OUoY3RMXUb6WK14ZIr1Up4fMT9M0pckfUXSXZIu6bDMUyRdLekeSbdIWteLYHPL0/agSGuFMkbHbN26MHEPD3c/0qdoi4gSlBFCVheKUraR0bIAsi/lz7OOIvKMTNlw1QZ0iZ68bbhqQ1f7MfWMqY7bbp9exjqyVNGeoQyDEmemrKouIOCI9P4ocAtwWtsyW4Ar0vsbgauz1tuzUTp52h4Uba0wt46io3yyWi/MzEQMDc2fPzR0aFtl7EdBZYSQdShK2UZGy4KI7Ev586yjDEuNTGkfXdNplE0Z+5GnrUHRH3Kpoj1DGeoWJ1WN0pE0BtwMTEfELS3T/y9wcUT8g6QR4FvAeCyx8p4VbfMUQ+vQkiBPQXYA+t2XEULWoShlGzl6wGcVTOvQRz5PsbSM/aiieFxFe4Yy1C3OnhdtJQ1Luh14CPhka7JPnQDcDxAR+4G9wNM7rGezpF2Sdu3Zs2c58WbLUwytQ0uCPAXZAeh3X0YIWYeilG3k6AGfVTAdlD7yZexHFcXjKtozlGFQ4swjV8KPiAMR8UJgNXCqpB9vW6TT144F3+4jYkdETEbE5Pj4ePfR5pGnGFqHlgR5CrID0O++jBCyDkUp28jRAz6rYDoofeTL2I8qisdVtGcow6DEmUdXo3Qi4lHgRuBlbbN2A2sA0lM6RwGPlBBf9/IUVOvQkiBPQTYrzhrsRxkhZB2KUraRowd8VsE0bx/5XspTLC1jP6poa1BFe4YyDEqcuWSd5AfGgaPT+08FPgec07bM65lftP1g1np72lqhjJYFVcgTQ9a+1KC1QhkhTE3NL8pOtdX/yni5snrA51mmjLYHWctkzc9TLM3ajzzHooq2BkWPRVWtF+oSR0SPi7aSTgKuBIZJ/iL4YERcKunSdMPXSjoM+CvgRSTf7DdGxNeXWm9fr7QtozWClSbr5ajq5cpqa1B0fhnbKGM/Voqm7Gc7t1boVg1Gt9ghdRmMVMWPfviHQ8rTlP1s59YK3arB6BY7pC6Dkar40Q//cEh5mrKfZWpmwq/B6BY7pC6Dkar40Q//cEh5mrKfZWpmwq/B6BY7ZOtWGB2dP210dP5gpKwuFFC8/UKe0RhDbf9lhhiaN390aP6OjA6NzhutkbVM1jYg+zL+skaN1L1dQN79rGI/6n6s5jQz4W/alFT8JiaS5i0TEy7Y9pm0+OPPf37hxVkHDiTT58wVdmdnk3E+s7PJ426S/qbnb2LHy3cwcdQEQkwcNTGvAPj5+z7PQQ7Oe85BDvL5+w4ForYdaX+ctUzWNuYKlbN7ZwmC2b2zbP7o5nkJJms/8siznX7Ls59V7McgHKs5zSzaWq1kFWXL6EJRhqyWBGUUbcvYRhlWSkG0Di0iyuairQ20rKJsGV0oypDVkqCMom0Z2yjDSimI1qFFRJ044VvfZRVly+hCUYaslgRlFG3L2EYZVkpBtA4tIurECd/63lN/61ZYtWr+tFWrDhVl83ahWGzd3Viq+JbVkiBPEXHr1FZWDc/f2VXDq3K3PajqMv6BahewhDq0iKgTJ/ymK6PaWYL2UlLr423bkt9tn/tGPzy88Hfc3/72zutdbHonWcW3bWdvY3py+slv28MaZnpymm1nJ4HkLZa2181aH69fu77jKJ31a9d3tY2iqtpOr1WxH4N0rFy0bboaXHVcRggdBsM8Ke9bvA4FvpVSLLXecdHWlq8GVx3XIIRkezUo8A1SAdAGjxN+09XgquMahJBsrwYFvkEqANrgccJvuhpcdVxGCCee2N30jnHUoMA3SAVAG0DL7atc9NbTfvjWnQp66hdt658nxOOPn99T//jjlxFnBX3P69Rb3cqxYvrh94qLts1RtJ99nudX0TO/qf3XbWlVvy/cD99qregonDzPr2KwkUfQWCdurWDWougonDzPr2Kkj0fQWCeD9L5wwreeKzoKJ8/zqxjp4xE01skgvS8yE76kNZI+I+luSXdJelOHZc6QtFfS7entot6E20BF2x70uW0CZLdOgKXDzOqXP7dMrwcblTWCZsuWpAOolPy7ZUt5MeY1KP3bB8FAjazKquoCxwEnp/ePBP4ZOLFtmTOA67qpFnuUTg4zMxFjY/OHnoyN5R9FU/T5JZmZiRgdnR/G6OihMLLCnJmJWLVq/vxVq5Y3kqfwvhQcjTE9PX8/5m7T0+XHupiZO2ZibOtYcDFP3sa2jnk0UAErdpSOpL8DLo+IT7ZMOwP47Yg4J+96XLTNoYpqZwWK/kh5TXajFHl6+/eai8+DrbJROpLWATcBPx4Rj7VMPwP4W2A38ABJ8r+rw/M3A5sB1q5de8psp//FdsjQUOdGMBIcPLhwetnPL0lWGEXnD5Iyev4UNXTJEMHCjQlx8G0DdkAbqJJROpKOIEnqb25N9qnbgImIeAHwZ8BHOq0jInZExGRETI6Pjy8n3mapotpZgaI/Ul6T3ShFnt7+vTZIRUYrV66EL2mUJNnvjIgPtc+PiMci4vH0/vXAqKRjSo20iYpWImvQNiFPGEXnD5I8vf17baCKjFaurJP8gICrgMuWWOZYDp0eOhW4b+7xYjcXbXOano4YHk4qe8PD3Vf3qqhk5jA1Nb9IOTU1f34ZrRUGRdGXtAxu3zC46GXRVtJPAp8D7gTmTvD9LrA2/cC4QtIbgGlgP/B94L9HxBeWWq+LtjlU0S+gAlu2wPbtC6e3/4iJmWVza4WVaoUMT6nDyBSzlcKtFVaquvwySEGdkv1S082sN5zw62yFDE+pw8gUM3PCX1wNWhLkGp5Shzgz1GFkSp0MwEtmK9Vyq71Fb7UepVOTlgRPxrLY8JQ6xbmEmZmIoaH5YQ4N1S7MSgzIS2Y1hn8ApWSDUiwdkDgHJMxK+FhYUR6lU7ZBuZZ/QOIckDAr4WNhRXmUTtkGpVg6IHEOSJiV8LGwfnLC72RQruUfkDgHJMxK+FhYXy335H/RW62LthGDcy3/gMQ5IGFW0sliUI6F1RMu2poVV7QFxArphGE156KtWQmKtoDwCByrgou2ZiUo2gJihXTCsBXMCd8sVbQFhEfgWN054ZuliraA8AgcqzsnfLPUtm1JgXbuG/3wcHc9+zdtSgq0ExPJhVQTEy7YWr24aGtmNkBctDUzs0xO+GZmDZGZ8CWtkfQZSXdLukvSmzosI0l/KukeSXdIOrk34doCbq5uZjmN5FhmP/BbEXGbpCOBWyV9MiL+sWWZM4Fnp7cXA9vTf62X2i/tnJ09NKTElUIza5P5DT8iHoyI29L73wXuBk5oW+w84Kq01cMXgaMlHVd6tDbfhRfOv44fkscXXtifeMys1ro6hy9pHfAi4Ja2WScA97c83s3CDwUkbZa0S9KuPXv2dBepLeRLO82sC7kTvqQjgL8F3hwRj7XP7vCUBeM9I2JHRExGxOT4+Hh3kdpCvrTTzLqQK+FLGiVJ9jsj4kMdFtkNrGl5vBp4oHh4tiRf2mlmXcgzSkfAXwB3R8SfLLLYtcCr09E6pwF7I+LBEuO0Tnxpp5l1Ic8onfXAfwXulHR7Ou13gbUAEXEFcD1wFnAPsA94TfmhWkebNjnBm1kumQk/Im6m8zn61mUCeH1ZQZmZWfl8pa2ZWUM44ZuZNYQTvplZQzjhm5k1hBO+mVlDOOGbmTWEE76ZWUM44ZuZNYQTvplZQzjhm5k1hBO+mVlDOOGbmTWEE76ZWUM44ZuZNYQTvplZQzjhm5k1hBO+mVlDOOGbmTVEnh8xf6+khyR9dZH5Z0jaK+n29HZR+WGamVlReX7E/H3A5cBVSyzzuYg4p5SIzMysJzK/4UfETcAjFcRiZmY9VNY5/NMlfUXSxyQ9r6R1mplZifKc0slyGzAREY9LOgv4CPDsTgtK2gxsBli7dm0JmzYzs7wKf8OPiMci4vH0/vXAqKRjFll2R0RMRsTk+Ph40U2bmVkXCid8ScdKUnr/1HSdDxddr5mZlSvzlI6k9wNnAMdI2g28DRgFiIgrgFcA05L2A98HNkZE9CxiMzNblsyEHxG/nDH/cpJhm2ZmVmO+0tbMrCGc8M3MGsIJ38ysIZzwzcwawgnfzKwhnPDNzBrCCd/MrCGc8M3MGsIJ38ysIZzwzcwawgnfzKwhnPDNzBrCCd/MrCGc8M3MGsIJ38ysIZzwzcwawgnfzHxbp4oAAAVHSURBVKwhnPDNzBrCCd/MrCEyE76k90p6SNJXF5kvSX8q6R5Jd0g6ufwwzcysqDzf8N8HvGyJ+WcCz05vm4HtxcMyM7OyZSb8iLgJeGSJRc4DrorEF4GjJR1XVoBmZlaOkRLWcQJwf8vj3em0B9sXlLSZ5K8AgCcWO01UM8cA3+53EDk4znINQpyDECM4zrI9Z7lPLCPhq8O06LRgROwAdgBI2hURkyVsv6ccZ7kcZ3kGIUZwnGWTtGu5zy1jlM5uYE3L49XAAyWs18zMSlRGwr8WeHU6Wuc0YG9ELDidY2Zm/ZV5SkfS+4EzgGMk7QbeBowCRMQVwPXAWcA9wD7gNTm3vWMZ8faD4yyX4yzPIMQIjrNsy45TER1Pt5uZ2QrjK23NzBrCCd/MrCEqSfiShiV9WdJ1HeY9RdLVaWuGWyStqyKmTjLiPF/SHkm3p7cL+hTjvZLuTGNYMDyrLq0ucsR5hqS9Lcfzoj7EeLSkayR9TdLdkk5vm1+XY5kVZx2O5XNatn+7pMckvbltmb4fz5xx9v14pnH8N0l3SfqqpPdLOqxtfte5s4xx+Hm8CbgbeFqHea8FvhMRz5K0Efgj4JUVxdVuqTgBro6IN1QYz2J+KiIWu0CktdXFi0laXby4qsDaLBUnwOci4pzKolnoXcDHI+IVklYBY23z63Iss+KEPh/LiPgn4IWQfHECvgl8uG2xvh/PnHFCn4+npBOANwInRsT3JX0Q2EjS6mZO17mz59/wJa0Gzgbes8gi5wFXpvevAaYkdbqYq6dyxDko3OoiB0lPA14C/AVARPx7RDzatljfj2XOOOtmCviXiJhtm97349lmsTjrYgR4qqQRkg/59uubus6dVZzSuQx4C3BwkflPtmaIiP3AXuDpFcTVLitOgF9I/xS9RtKaJZbrpQA+IelWJa0q2i3W6qJqWXECnC7pK5I+Jul5VQYHPBPYA/xlehrvPZIOb1umDscyT5zQ32PZbiPw/g7T63A8Wy0WJ/T5eEbEN4F3APeRtKnZGxGfaFus69zZ04Qv6RzgoYi4danFOkyrdKxozjg/CqyLiJOAGzj0yVq19RFxMsmfx6+X9JK2+X0/nqmsOG8DJiLiBcCfAR+pOL4R4GRge0S8CPge8Dtty9ThWOaJs9/H8knpKadzgb/pNLvDtL6MC8+Is+/HU9IPkXyDfwZwPHC4pFe1L9bhqUsez15/w18PnCvpXuADwE9Lmmlb5snWDOmfLkexdHfOXsiMMyIejogn0ofvBk6pNsQn43gg/fchknOPp7YtUotWF1lxRsRjEfF4ev96YFTSMRWGuBvYHRG3pI+vIUms7cv0+1hmxlmDY9nqTOC2iPjXDvPqcDznLBpnTY7nBuAbEbEnIn4AfAj4ibZlus6dPU34EfHWiFgdEetI/nz6dES0f0pdC/xqev8V6TKVfurnibPtXOO5JMXdSkk6XNKRc/eBnwXaO472vdVFnjglHTt3vlHSqSTvxYerijEivgXcL2mu8+AU8I9ti/X9WOaJs9/Hss0vs/hpkr4fzxaLxlmT43kfcJqksTSWKRbmnK5zZ1WjdOaRdCmwKyKuJSlG/ZWke0g+nTb2I6ZO2uJ8o6Rzgf0kcZ7fh5B+FPhw+l4cAf46Ij4u6XVQuNVF1XG+ApiWtB/4PrCx6g964DeBnemf918HXlPDY5knzjocSySNAT8D/EbLtNodzxxx9v14RsQtkq4hOb20H/gysKNo7nRrBTOzhvCVtmZmDeGEb2bWEE74ZmYN4YRvZtYQTvhmZg3hhG9m1hBO+GZmDfH/ATYspSon588iAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# visualize result of sklearn\n", "\n", "cat1_ = data[np.where(labels_==0)]\n", "cat2_ = data[np.where(labels_==1)]\n", "cat3_ = data[np.where(labels_==2)]\n", "\n", "plt.scatter(cat1_[:,0], cat1_[:,1], color='green')\n", "plt.scatter(cat2_[:,0], cat2_[:,1], color='red')\n", "plt.scatter(cat3_[:,0], cat3_[:,1], color='blue')\n", "plt.title('Hierarchical clustering with k=3')\n", "plt.xlim(4, 8)\n", "plt.ylim(1, 5)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "La_XZDI5_Bng" }, "source": [ "---------------------------------------------------------------------------------------------------------------------------------" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": {}, "colab_type": "code", "id": "fNl9AY6vAFJG" }, "outputs": [], "source": [ "# kmeans\n", "\n", "class MyKmeans:\n", " def __init__(self, k, n=20):\n", " self.k = k\n", " self.n = n\n", " \n", " def fit(self, x, centers=None):\n", " # 第一步，随机选择 K 个点, 或者指定\n", " if centers is None:\n", " idx = np.random.randint(low=0, high=len(x), size=self.k)\n", " centers = x[idx]\n", " #print(centers)\n", " \n", " inters = 0\n", " while inters < self.n:\n", " #print(inters)\n", " #print(centers)\n", " points_set = {key: [] for key in range(self.k)}\n", "\n", " # 第二步，遍历所有点 P，将 P 放入最近的聚类中心的集合中\n", " for p in x:\n", " nearest_index = np.argmin(np.sum((centers - p) ** 2, axis=1) ** 0.5)\n", " points_set[nearest_index].append(p)\n", "\n", " # 第三步，遍历每一个点集，计算新的聚类中心\n", " for i_k in range(self.k):\n", " centers[i_k] = sum(points_set[i_k])/len(points_set[i_k])\n", " \n", " inters += 1\n", "\n", " \n", " \n", " return points_set, centers\n", " " ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "colab": {}, "colab_type": "code", "id": "SyLthTXfBfnV" }, "outputs": [], "source": [ "m = MyKmeans(3)\n", "points_set, centers = m.fit(data)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 71 }, "colab_type": "code", "id": "VE2ryNB-O_Zt", "outputId": "55a70e58-fccd-4001-c67c-964ede3a8550" }, "outputs": [ { "data": { "text/plain": [ "array([[5.006 , 3.428 ],\n", " [6.81276596, 3.07446809],\n", " [5.77358491, 2.69245283]])" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "centers" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 281 }, "colab_type": "code", "id": "M26gflVYDzY4", "outputId": "abb1b7b1-df4f-4fa1-9a24-1dc722112c2c" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5xcdX3v8dd7fwRYIIlilB9JNloUigoKuSDGq2jSll9Ce7UVG4tYaerGX1h7aa33we+g9mELVkxotAqYVLFULVK0igqKVmz4IUrRXqpsCCyXyI8ABtEkn/vHOZOdnZ3Zc2bn7MyZnffz8ZjHzpzznXM+58zsZ8+e7/d8jiICMzOb/fo6HYCZmbWHE76ZWY9wwjcz6xFO+GZmPcIJ38ysRzjhm5n1CCf8LibpLknHlSCOJZJC0kCD+X8l6RMzuY4p3neGpJtbWXdROv15ZX0Oze4rSfdKWlFMdNYOTvglVe+XqfYXMiJeGBE3tj24JkXExRFxZqfjaEX6x+bgVpbR6c+r+nOY7h/QoknaIGlM0uOS/ktSV39Pys4JvwdN55dcUv9MxNILOp1US+4DwJKImAucAlwk6agOxzRrOeF3ser/AiT1SfpLSf8t6WFJn5P0zHRe5WjurZI2A99Ip/+TpAclbZP0LUkvrFr2FZLWSbpe0i+AV0vaS9LfSBpN33OzpL2qQlopabOkn0t6f9WyzpO0oer1KyR9V9Jjku6TdEY6/SRJt6dHe/dJOq+JfbFI0uclbU23/7I6bSYd1Uq6sXJUKelgSTel2/ZzSVen07+VNv+BpCclvSGdfrKkO9Lt+K6kw2s+m7+QdCfwC0kDNZ/XeelndJWkJ9LTPUur3n9kui+eSD+nqyVd1GDbRytJUtKb0m08LH19pqQv1vkcKtv0WLpNx1Yt78OSHpX0M0kn5Nz/h6btT8vTviIi7oqIpysv08dvNLMMy88Jf/Z4F/C7wKuAA4FHgY/VtHkV8JvA76Svvww8H3g2cBuwsab9HwJrgH2Bm4EPA0cBLweeCZwN7Kpq/wrgEGA5cI6k36wNUtLidL0fBRYALwHuSGf/AjgdmA+cBIxI+t2sDU//+7gOGAWWAAcBn816Xx0XAl8FngEsTGMkIl6Zzj8iIvaJiKslHQl8EvhTYD/g74FrJe1Rtbw3ptsxPyJ21FnfKWmc84FrgcvS7ZkDfAG4gmQ/fwb4vSnivgk4Ln3+SuCnJJ915fVNdd5T2ab56Tb9e/r6GOAnwLOAvwb+QZKmWDfpvvgq8M6I+Gw67br0D2G9x3U1718raTvwY2AMuH6q9VkLIsKPEj6Ae4EngceqHtuBm2varEif3w0sr5p3APBrYIAkCQbwvCnWNz9tMy99fQVwVdX8PuApkqRX+97K8hdWTfs+cFr6/DxgQ/r8fcAXcu6DS4FLatYxUKfdscDWBvPOqOyzessAbgTOTJ9fBayv3o6qdgEcXPV6HXBhTZufAK+q+mz+uM5nWvm8zgNuqJp3GPBU+vyVwP2AqubfDFzUYD+9Fbi26ntwJvDZ9PUocGSdz6HevjgDuKfq9VDaZv8pvqPnA1uAV7f4fe8nOWD4P8BgJ3/3ZvPDR/jl9rsRMb/yAFZP0XYY+ELlKIrkF38n8JyqNvdVnkjql/TB9BTQ4yS/vJAc2U1qn07fE/jvKWJ4sOr5dmCfOm0WNVqGpGMkfTM9LbMNeFtNPI0sAkaj/lF0M84GBHw/PcXyx1O0HQbeW33kmsZxYFWb++q/dbfa/bVnerrpQOD+SDNhjmXdBPxPSfuTJM6rgWWSlgDzGP8PKo/dMUXE9vRpvc+x4m3AdyPim02sY5KI2BkRN5P8ZzXSyrKsMSf82eM+4ITqPxARsWdE3F/VpjqB/CFwKrCCJCksSaerQfufA7+k9fOr902xjH8kObWxKCLmAZfXxDPVMhcru3P0F+nPoapp+1eeRMSDEfEnEXEgyamatWo8Muc+YE3N/h6KiM9UtZluKdox4KCaUymLGjWOiHtI/mC8C/hWRDxBkrhXkfx3s6ve26YZW623kez7S6onSvpy2jdQ7/HlKZY3gM/hzxgn/NnjcmCNpGEASQsknTpF+32Bp4GHSRLgxVMtPE0anwT+VtKB6X8Ix9acs85jI7BC0h+kHZn7SXpJVUyPRMQvJR1N8kcpj++TJMkPStpb0p6SltXZhq0kp0relMb/x1QlF0m/L2lh+vJRkqS4M339/4DnVS3u48Db0v9KlK73JEn75ox5Kv+ervcd6T46FTg64z03Ae9g/Hz9jTWva20l6X95XoP5eT0BHA+8UtIHKxMj4oRI+gbqPU4AkPRsSadJ2if9PH6HpN/jGy3GZA044c8eHyE5Ov6qpCeA75F0wDVyFcn53fuB/0zbZ/lz4IfAfwCPAB+iye9QRGwGTgTemy7jDuCIdPZq4II0/nOAz+Vc5k7gtcDBwGaSc8pvaND8T4D/TfKH7oXAd6vm/Q/gFklPkuzLd0fEz9J55wFXpqdv/iAiNqXLuozkj8M9JOfAWxYRvwL+F8m5+ceAN5F0Sj89xdtuIvmD+a0Gr2vXsZ2kQ/476Ta9rIV4HwN+CzhB0oXNvJXk9M0Wkn34YeCsiPiX6cZiU9PE04RmVkaSbgEuj4hPdToW614+wjcrIUmvkrR/ekrnzcDhwFc6HZd1t1xXAEq6l+Rc3U5gR0QsrZkvklMKJ5J0Hp0REbcVG6pZTzmE5JTWPiSjml4fEWOdDcm6Xa5TOmnCXxoRP28w/0TgnSQJ/xjgIxEx1fljMzNrs6JO6ZxKcpFORMT3gPmSDiho2WZmVoC8RZ2CZPRHAH8fEetr5h/ExAtDtqTTJvwLKmkVydhg9t5776MOPfTQaQVtZtarbr311p9HxILpvDdvwl8WEQ9IejbwNUk/jojq4V71Lo6ZdK4o/UOxHmDp0qWxadOmpgM2M+tlkkan+95cp3Qi4oH050MkRZ1qLwLZwsQrARcCD0w3KDMzK15mwk+vINy38hz4beBHNc2uBU5Przh8GbDNIwrMzMolzymd55AU5aq0/8eI+IqktwFExOUk5UxPJLnacDvwlpkJ18zMpisz4UfETxm/9L16+uVVzwN4e7GhmZlZkXylrZlZj3DCNzPrEU74ZmY9wgnfzKxHOOGbmfUIJ3wzsx7hhG9m1iOc8M3MeoQTvplZj3DCNzPrEU74ZmY9wgnfzKxHOOGbmfUIJ3wzsx7hhG9m1iOc8M3MeoQTvplZj3DCNzPrEbkTvqR+SbdLuq7OvDMkbZV0R/o4s9gwzcysVXluYl7xbuBuYG6D+VdHxDtaD8nMzGZCriN8SQuBk4BPzGw4ZmY2U/Ke0rkUOBvYNUWb10m6U9I1kha1HpqZmRUpM+FLOhl4KCJunaLZl4AlEXE4cANwZYNlrZK0SdKmrVu3TitgMzObnjxH+MuAUyTdC3wWeI2kDdUNIuLhiHg6fflx4Kh6C4qI9RGxNCKWLliwoIWwzcysWZkJPyLeFxELI2IJcBrwjYh4U3UbSQdUvTyFpHPXrDgbN8KSJdDXl/zcuLHTEZl1nWZG6Uwg6QJgU0RcC7xL0inADuAR4IxiwjMjSe6rVsH27cnr0dHkNcDKlZ2Ly6zLKCI6suKlS5fGpk2bOrJu6zJLliRJvtbwMNx7b7ujMesoSbdGxNLpvNdX2lr5bd7c3HQzq8sJ38pv8eLmpptZXU74Vn5r1sDQ0MRpQ0PJdDPLzQnfym/lSli/PjlnLyU/1693h61Zk6Y9SsesrVaudII3a5GP8M3MeoQTvplZj3DCNzPrEU741h4ujWDWce60tZnn0ghmpeAjfJt573//eLKv2L49mW5mbeOEbzPPpRHMSsEJ32aeSyOYlYITvs18h6pLI5iVghN+r6t0qI6OQsR4h2qRSd+lEcxKwfXwe51rzZt1FdfDt+lzh6pZz3DC73XuUDXrGU74vc4dqmY9I3fCl9Qv6XZJ19WZt4ekqyXdI+kWSUuKDNJm0GzqUHX5BrMpNVNa4d3A3cDcOvPeCjwaEQdLOg34EPCGAuKzdpgNteZdvsEsU64jfEkLgZOATzRocipwZfr8GmC5JLUenllOLt9glinvKZ1LgbOBXQ3mHwTcBxARO4BtwH61jSStkrRJ0qatW7dOI1yzBjzayCxTZsKXdDLwUETcOlWzOtMmDfCPiPURsTQili5YsKCJMM0yeLSRWaY8R/jLgFMk3Qt8FniNpA01bbYAiwAkDQDzgEcKjNPKbPVqGBhIOn0HBpLX7ebRRmaZMhN+RLwvIhZGxBLgNOAbEfGmmmbXAm9On78+bdOZS3itvVavhnXrYOfO5PXOncnrdif92TTayGyGNFVaQdJxwJ9HxMmSLgA2RcS1kvYEPg28lOTI/rSI+OlUy3JphVliYGA82Vfr74cdO9ofj9ks10pphabueBURNwI3ps/PqZr+S+D3pxOAdbl6yX6q6WbWMb7S1lrT39/cdDPrGCd8a03l4qa8082sY5zwZ7sVK5JOzMpjxYpil792LYyMjB/R9/cnr9euLXY9LptgJbbxhxtZcukS+s7vY8mlS9j4w+a/n0UsI4vr4c9mK1bA178+efry5XDDDe2PZ7pqyyZAMuTSo3CsBDb+cCOrvrSK7b8e/34ODQ6x/rXrWfnifN/PZpbRSqetE/5sNlV1i24aNeubtFiJLbl0CaPbJn8/h+cNc+9Z9xa+DN8AxWY3l02wEtu8rf73sNH0mVpGHk74Vn4um2Altnhe/e9ho+kztYw8nPBns+XLm5veSFaH6Ux3qLpsgpXYmuVrGBqc+P0cGhxizfL8388ilpFLRHTkcdRRR4XNsA0bIqSI5Ix98pCS6c0sY2ho4jKGhsaXkTW/yG0ZHk7iHx4ufvlmLdhw54YYvmQ4dJ5i+JLh2HBn89/PvMsgqXAwrbzrTtvZrIjOzqxluEPVrK3caWv1FdHZmbUMd6iadQ0n/NmsiM7OrGW4Q9WsazjhT1e7rvxspcO0iM7OrGW4Q9VmUDuuPu0p0z353+qjqztt29lR2WqHaRGdnVnLcIeqzYANd26IoTVDwXnsfgytGZpWh+hsgjtt26xdHZXuMLUeVsQVrLORO23brV0dle4wtR7WrqtPe4kT/nS0q6PSHabWw9p19WkvccKfjnZ1VLrD1HpY264+7SGZCV/SnpK+L+kHku6SdH6dNmdI2irpjvRx5syEWxLtumF21npWroRjjyVg94Njj50Yx+rVyX1npeRnvZuLt2PEkevZW5NWvngl61+7nuF5wwgxPG+4qZLDVkdWry4gYJ/0+SBwC/CymjZnAJc101vc1aN0ymJkJHZVj9CB5PXIyO75UTM/qudHtGfEUbtGNZn1ANo1SkfSEHAzMBIRt1RNPwNYGhHvyLusrh6lUxYDA/VvFt7fDzt2ZM+H9oz08Wgis8LM+CgdSf2S7gAeAr5WneyrvE7SnZKukbSowXJWSdokadPWrVunE69ViXrJvHp6g/kTprdjpI9HE5mVQq6EHxE7I+IlwELgaEkvqmnyJWBJRBwO3ABc2WA56yNiaUQsXbBgQStxG7CrwR2tdk+v3Ge2VvX0doz08Wgis1JoapRORDwG3AgcXzP94Yh4On35ceCoQqIrsyI6IfN0qDYwNjbGx/sGqD0hF8An+gZ48MEHk/vA1lM9fc0aGBycOH9wsLmRPlnb0UWjiVq9lD/P+10uwDolzyidBZLmp8/3AlYAP65pc0DVy1OAu4sMsnQqN9UeHU26IEdHk9fNJP3Vq2HduvHTKzt3Jq9zJv0LL/4gA/strDuvf7+FXHjxB2HZsiQBVxsYSKZXq/1PYap74dbKsx3tGtXUosqNpEe3jRIEo9tGWfWlVbkTcp73t7oOs1ZkdtpKOpzkFE0/yR+Iz0XEBZIuIOktvlbSB0gS/Q7gEZJO3R83XChd3mlbRCdkng7VBsbGxviNQ36Tx598goHYNWn+DvUxb5+5bJu3DwNbtkwdZ6vb0sJ2lE2rl/Lneb/LBVirWum0HchqEBF3Ai+tM/2cqufvA943nQC6UhGdkHk6VBu48OIPMnTYa+i/5Qt15/fHLvY67NUN50+Is9VtaWE7yqbVS/nzvN/lAqyTfKXtdBTRCZmnQ7WOsbExrrjySvZc+nvsVP2Pb6f62HPp73Ffo1Mz1XG2ui3T3I4yavVS/jzvd7kA6yQn/OkoohMyT4dqHZWj+4F9nsnGI46v22m78YjjGdjnmVz0vKN4uvYcfm2crW7LNLejjFq9lD/P+10uwDpquldstfro+itti6gBPzIS0d+fXHna3z/xCtg6Hnjggdhr33lx0NuviuG/uC6G/+K6uGnxEbErvcJ2F8RNi4/YPe+gt18Vl/cPxq6sdbS6LU1uR5m1ejPqPO9ffuXyCTXel1+5vKjwcxu5biT6z+8PziP6z++Pkeu69zPrNbgefm9Y/c5387n/uI99jnsrAKfc9U0++JXLGNrx9O422wf24C+PfwfXvvDVyfzrL2VoV9X59KGhUo6Q6RWr/3U16zatmzR9ZOkIa09a2zMx2PS10mnrhN8lKiNznnn6RxnY55kA3LzuLSx8fPIVy1vmLuAVI59qON8lDTpn4IIBdsbkDu1+9bPjnPaMaipDDDZ9vgFKD6g+d19x4OM/r9u2Mr3RfJc06Jx6iXaq6bM1BuuMzGGZVg5fuvZfeHjzKA9XDbXcDCyp03YzweiHTm443yUNOqdf/Q2PrnspBuuM3j3Cb7U0Qp73t1A6odZ9o/dO6oBZsmEDzJkzseGcOSzZsGF8fpeUNGiHMpQ0WHVU/dFL1dNX/+tqBi4YQOeLgQsGWP2vE783rW5HnhjyrKfV+XmU4TObVabb29vqo6OjdFqtz57n/Xlq0RexHYODE5c/ODgxjiJGE80CG+7cEENrhiaMjhlaM9T0KJwiTDVCZuS6kQkxVh6VNkVtR9Yonaz1tDo/jzJ9ZmWCR+k0qdVyAnne346SA64zn1u3lDTI6lBt13ZkrafV+UXE0KvcadusVssJ5Hl/O0oOuM58bt1S0iCrQ7Vd25G1nlbnFxGDNa83E36r5QTyvL8dJQdcZz63bilp0KjjtDK9XdvRaHkH7XNQrjiKiLNbPrNu0psJv9VyAnne346SA11UZ77TuqWkQVaHaru2o956BnbCi8YOzRVHEXF2y2fWVaZ78r/VR8dLK7SjnEBWmzwxFLEMi4jWyya0S1bphSK2I095h8tuuiz63qPQucTwuYrLzt4jnjl3rxgbG4uIiMMuO2zCMg677LDC4+yWz6ydcKdtm1VugLJ9+/i0ZksW5FlG5eYitUZGYK0vgZ+NKjdI2f7r8e/F0OAQ61+7npUvLqYcxoqrVvD1n3190vTlz13ODaffsPv1e965Gm7/NJesGD8R8J4bdqEjT+fOpT/JtQwrnksrtFsRo2PKMtLHSqUdI1N0fuM7msW5ST4YGxvjhYf8Bnf9ST8H7Due8Mee2MWLPrGTR/7sqcxl2MzwKJ12K2J0TFlG+liplGVkyl9ffCFvPnxisgc4YN8+Tn9xP5PqcltXcMKfjiJGx5RlpI+VShlGpoyNjXHllVdw9jH15zeabuWX5ybme0r6vqQfSLpL0vl12uwh6WpJ90i6RdKSmQg2tzxlD1oprVDE6Jg1ayYn7v7+5kf6tFoiogBFhJBVhaKQdWSULIDsS/nzLKMVeUamrLhqBTpfux8rrlrR1HYsf+7yuuuuTK8c3X9j3x0s4Qn6eJwlPMFGfgUkR/mLHlbdo/xGy26kHeUZitAtcWbJc4T/NPCaiDgCeAlwvKSX1bR5K/BoRBwMXAJ8qNgwm1DpDB0dTYoNjI4mr6szRJ42U1m5MulcHR5OMtTwcPM15r/zncmnZnbuTKZXLFuWZLhqfX3J9CK2owBFhFDpm67sjp07k9eVpF/IOtIa8JULmHbGTtZtWjchYVc6TEe3jRIEo9tGWfWlVbt/efMso1UrX7yS9a9dz/C8YYQYnjc8ocO2Xofr13/29QlJP2s7XrDfC+qu+wX7vWD30f3By37NKn7JqIIQjCpYxS93J/1b5uxNf01XQ7MdtllxZs1vl26JM4+mOm0lDQE3AyMRcUvV9H8DzouIf5c0ADwILIgpFj5jnbZ5OkPLUJIgT4dsVpwl2I4iQsjaFYWsI0cN+KwO0zLUkc/T4drKdrxz6yq4/dN8YcUvGNXkX9/hEPeyLzA+Yudv/+5j09qWdpRnKELZ4pzxTltJ/ZLuAB4Cvlad7FMHAfcBRMQOYBuwX53lrJK0SdKmrVvr3JijCHk6Q8tQkiBPh2xWnCXYjiJCyNoVhawjRw34rA7Tbqkj38p2VM7db27QK1s9/exj4MorP8WDDz44I3GWpQO7W+LMI1fCj4idEfESYCFwtKQX1TSpd9gx6RsTEesjYmlELF2wYEHz0eaRpzO0DCUJ8nTIZsVZgu0oIoSsXVHIOjJKFkB2h2meZZTBdLdDod0jcxbX/ZVmwvTKiJ2/vvjCGYmzDB3YeeIoS5x5NDVKJyIeA24Ejq+ZtQVYBJCe0pkHPFJAfM3L06FahpIEeTpks+IswXYUEULWrihkHTlqwGd1mOatIz+TsjpcYfrbMXCHdo/AWcMeDNUcsg1FMr1aK0f57SjPUIRuiTOXrEtxgQXA/PT5XsC3gZNr2rwduDx9fhrwuazlzmhphSJKFrRDnhiytqUEpRWKCGH58oll/ZfXXOlfxMeVVQM+T5siyh5ktcman6csQtZ21M4//OzD4qxl+0ScO3f3Y8O5e8bwueOlFTacu+eE+ZXHWcv2ife8c/XUO7+BVvdFu0ovlCWOiNZKK+RJ+IcDtwN3Aj8CzkmnXwCckj7fE/gn4B7g+8Dzspbb1TdAsUJlfRzt+rjacdOPst445LmLDwqS07DTejx38UHT2eUt6dUbpLSS8HuztEIJRrfYuLIMRmrHTT9845Di9Mp21nJphWaVYHSLjSvLYKR23PTDNw4pTq9sZ5F6M+GXYHSLjSvLYKRWR2PkGa3hG4cUp1e2s0i9mfBLMLrFxq1ZA4ODE6cNDk4cjJRVhQJaL7+QZzRGX82vTB99E+YP9k3ckMG+wQmjNbLaZK0Dsi/jL2rUSNnLBeTdznZsR9n3VUVvJvwiSiNYoaTGr/NUofjYxx7lj/5oe0vlF7LKGnxn83fYxa4J79nFLr6zeTwQ1WxI7eusNlnryHMZf9Z25NEN5QLybGc7tqMb9lVFb3baWqlkdcrmqUKx79yHefKJSRd3F9qxm1VaoYhO2yLWUYTZ0iHaju2YdaUVzGZSVqdsVumFsbExnnziGU0tezqySisU0WlbxDqKMFs6RNuxHd20r5zwreOyOmWzSi9c9IGL6Nvz/qaWPR1ZpRWK6LQtYh1FmC0dou3Yjm7aV0741vGa+mvWwJw5E6fNmTPeKTtV6YWxsTGuuPIKnv36S6Hm3Hdl2c2YqvMtq7RCnk7ENcvXMKd/4sbO6Z+Tu3xDuy7j76pyAVNox3Z0075ywu91JaipD8mqG71euza5b3vliL6/f/w+7hd94CLmvXweD3/jT6lXw+/ii/PHkNX5tvaktYwsHdl9tN2vfkaWjrD2pOSG8nk7S2v7zapfL1u8rO4onWWLlzW1jla1az0zrR3b0U37yp22va4EVx1PN4SxsTEOPvRgFl2wiJ+cdTf1i7ZO/mPSMI4SdPDNls5SmznutLXpK8FVx9MNoXJ0Pzh/cOqGeeMoQQdfN3UAWvdxwu91JbjqeDohVM7dzz1+bsbS8/8HW4YOvm7qALTu44Tf60pw1fF0Qqg9uh888P8yObkHz3hG/dE7deMoQQdfN3UAWvdxwu91bbrqeKqBQHlCqH7/woU7+PjHt084uj/k4tfRN/9Bqqv2au4D/GrXoblvzpH3ys1WLqHPWkc3dQDauG4preBOW5txlYFA27ePTxsayv93pd776dvOwjPPZ/7Lrwfgse+eyP1XnEf8aq/dTTTnKeYe+l7edFwfl33ksta3Ix3Fs/3X44EMDQ45Ife4dn8vWum0dcK3GdfqQKBG7x/c7wEO+ZvfAeAn7/03fv3wgZPaDDzjfvbYdQj3/Pge9t9//6binhSHR9BYHS6tYFal1YFAjdr9+uH96z6vtuPRA5h77Fwu+sBF+VY2VRweQWN1dNP3wgnfZlyrA4EatRvc78G6z2vbzD1+LldcecW0brQ9IQ6PoLE6uul7kZnwJS2S9E1Jd0u6S9K767Q5TtI2SXekj3NmJtwe1GrZgw6XTYDs0gkwdZj16uXT/yue87qP7H75nNd9BM15akITzXmK57zuIwzOHyzkKL+oETSrVycVQKXk5+rVLYU1Ld3SydgNumpkVdZNb4EDgCPT5/sC/wUcVtPmOOC6Zm6m29GbmHeLVu/eXZKbtW/YEDE4ODGMwcH8NynfsCFizpyJ8+GXAW+suZn2GwN+FrAz/Tlx/sLhha1vy50bYviS4dB5iuFLhpu+YfbISO12JI+RkZZDy61Xb/49k1r9XjSDdt7EXNK/AJdFxNeqph0H/HlEnJx3Oe60zWGmejvbfLP2Vm9SXpLNKESe2v4zzZ3P3a1to3QkLQG+BbwoIh6vmn4c8M/AFuABkuR/V533rwJWASxevPio0Xq/xTaur69+IRgJdk2uDFn4+wuSFUar87tJnRtg7dauAXN95/cRda5AFmLXuV22Q3tQW0bpSNqHJKmfVZ3sU7cBwxFxBPBR4Iv1lhER6yNiaUQsXbBgwXTi7S0z1dvZ5pu1t3qT8pJsRiGyavu3Qzd1MlqxciV8SYMkyX5jRHy+dn5EPB4RT6bPrwcGJT2r0Eh7UatlD0pQNiFPGK3O7yZT1fZvl67qZLRiZZ3kJ6k5exVw6RRt9mf89NDRwObK60YPd9rmNDIS0d+f9Oz19zffu7dhQ8TwcISU/Gxzh23F8uUTOymXL584PyvMkmxGIVr9SIvQzk5GKxYz2Wkr6RXAt4EfMn5Lob8CFqd/MC6X9A5gBNgBPAX8WUR8d6rlutM2h1ZrEpTE6tWwbt3k6ZWbmJhZfi6tMFvNkuEpZRiZYjZbuLTCbFWCm5MUoV6yn2q6mc0MJ/wymyXDU8owMsXMnPAbK0FJglzDU+MCh+0AAAlWSURBVMoQZ4YyjEwpky74yGy2mm5vb6uPUo/SKUlJgt2xNBqeUqY4p7BhQ0Rf38Qw+/pKF2ZbdMlHZiVGO0srFKXUnbbd0lnaJXF2SZht4X1hrfIonaJ1y7X8XRJnl4TZFt4X1iqP0ilat3SWdkmcXRJmW3hfWCc54dfTLdfyd0mcXRJmW3hfWEdN9+R/q49Sd9pGdM+1/F0SZ5eE2ZZKFt2yL6yccKetWetaLQExSyphWMm509asAK2WgPAIHGsHd9qaFaDVEhCzpBKGzWJO+GapVktAeASOlZ0Tvlmq1RIQHoFjZeeEb5ZauzbpoK0c0ff3N1ezf+XKpIN2eDi5kGp42B22Vi7utDUz6yLutDUzs0xO+GZmPSIz4UtaJOmbku6WdJekd9dpI0l/J+keSXdKOnJmwrVJXFzdzHIayNFmB/DeiLhN0r7ArZK+FhH/WdXmBOD56eMYYF3602ZS7aWdo6PjQ0rcU2hmNTKP8CNiLCJuS58/AdwNHFTT7FTgqrTUw/eA+ZIOKDxam+j97594HT8kr9///s7EY2al1tQ5fElLgJcCt9TMOgi4r+r1Fib/UUDSKkmbJG3aunVrc5HaZL6008yakDvhS9oH+GfgrIh4vHZ2nbdMGu8ZEesjYmlELF2wYEFzkdpkvrTTzJqQK+FLGiRJ9hsj4vN1mmwBFlW9Xgg80Hp4NiVf2mlmTcgzSkfAPwB3R8TfNmh2LXB6OlrnZcC2iBgrME6rx5d2mlkT8ozSWQb8EfBDSXek0/4KWAwQEZcD1wMnAvcA24G3FB+q1bVypRO8meWSmfAj4mbqn6OvbhPA24sKyszMiucrbc3MeoQTvplZj3DCNzPrEU74ZmY9wgnfzKxHOOGbmfUIJ3wzsx7hhG9m1iOc8M3MeoQTvplZj3DCNzPrEU74ZmY9wgnfzKxHOOGbmfUIJ3wzsx7hhG9m1iOc8M3MeoQTvplZj8hzE/NPSnpI0o8azD9O0jZJd6SPc4oP08zMWpXnJuZXAJcBV03R5tsRcXIhEZmZ2YzIPMKPiG8Bj7QhFjMzm0FFncM/VtIPJH1Z0gsLWqaZmRUozymdLLcBwxHxpKQTgS8Cz6/XUNIqYBXA4sWLC1i1mZnl1fIRfkQ8HhFPps+vBwYlPatB2/URsTQili5YsKDVVZuZWRNaTviS9pek9PnR6TIfbnW5ZmZWrMxTOpI+AxwHPEvSFuBcYBAgIi4HXg+MSNoBPAWcFhExYxGbmdm0ZCb8iHhjxvzLSIZtmplZiflKWzOzHuGEb2bWI5zwzcx6hBO+mVmPcMI3M+sRTvhmZj3CCd/MrEc44ZuZ9QgnfDOzHuGEb2bWI5zwzcx6hBO+mVmPcMI3M+sRTvhmZj3CCd/MrEc44ZuZ9QgnfDOzHuGEb2bWI5zwzcx6RGbCl/RJSQ9J+lGD+ZL0d5LukXSnpCOLD9PMzFqV5wj/CuD4KeafADw/fawC1rUelpmZFS0z4UfEt4BHpmhyKnBVJL4HzJd0QFEBmplZMQYKWMZBwH1Vr7ek08ZqG0paRfJfAMDTjU4TlcyzgJ93OogcHGexuiHObogRHGfRDpnuG4tI+KozLeo1jIj1wHoASZsiYmkB659RjrNYjrM43RAjOM6iSdo03fcWMUpnC7Co6vVC4IEClmtmZgUqIuFfC5yejtZ5GbAtIiadzjEzs87KPKUj6TPAccCzJG0BzgUGASLicuB64ETgHmA78Jac614/jXg7wXEWy3EWpxtiBMdZtGnHqYi6p9vNzGyW8ZW2ZmY9wgnfzKxHtCXhS+qXdLuk6+rM20PS1WlphlskLWlHTPVkxHmGpK2S7kgfZ3Yoxnsl/TCNYdLwrLKUusgR53GStlXtz3M6EON8SddI+rGkuyUdWzO/LPsyK84y7MtDqtZ/h6THJZ1V06bj+zNnnB3fn2kc75F0l6QfSfqMpD1r5jedO4sYh5/Hu4G7gbl15r0VeDQiDpZ0GvAh4A1tiqvWVHECXB0R72hjPI28OiIaXSBSXeriGJJSF8e0K7AaU8UJ8O2IOLlt0Uz2EeArEfF6SXOAoZr5ZdmXWXFCh/dlRPwEeAkkB07A/cAXapp1fH/mjBM6vD8lHQS8CzgsIp6S9DngNJJSNxVN584ZP8KXtBA4CfhEgyanAlemz68BlkuqdzHXjMoRZ7dwqYscJM0FXgn8A0BE/CoiHqtp1vF9mTPOslkO/HdEjNZM7/j+rNEozrIYAPaSNEDyR772+qamc2c7TulcCpwN7Gowf3dphojYAWwD9mtDXLWy4gR4Xfqv6DWSFk3RbiYF8FVJtyopVVGrUamLdsuKE+BYST+Q9GVJL2xncMDzgK3Ap9LTeJ+QtHdNmzLsyzxxQmf3Za3TgM/UmV6G/VmtUZzQ4f0ZEfcDHwY2k5Sp2RYRX61p1nTunNGEL+lk4KGIuHWqZnWmtXWsaM44vwQsiYjDgRsY/8vabssi4kiSf4/fLumVNfM7vj9TWXHeBgxHxBHAR4Evtjm+AeBIYF1EvBT4BfCXNW3KsC/zxNnpfblbesrpFOCf6s2uM60j48Iz4uz4/pT0DJIj+OcCBwJ7S3pTbbM6b51yf870Ef4y4BRJ9wKfBV4jaUNNm92lGdJ/XeYxdXXOmZAZZ0Q8HBFPpy8/DhzV3hB3x/FA+vMhknOPR9c0KUWpi6w4I+LxiHgyfX49MCjpWW0McQuwJSJuSV9fQ5JYa9t0el9mxlmCfVntBOC2iPh/deaVYX9WNIyzJPtzBfCziNgaEb8GPg+8vKZN07lzRhN+RLwvIhZGxBKSf5++ERG1f6WuBd6cPn992qatf/XzxFlzrvEUks7dtpK0t6R9K8+B3wZqK452vNRFnjgl7V853yjpaJLv4sPtijEiHgTuk1SpPLgc+M+aZh3fl3ni7PS+rPFGGp8m6fj+rNIwzpLsz83AyyQNpbEsZ3LOaTp3tmuUzgSSLgA2RcS1JJ1Rn5Z0D8lfp9M6EVM9NXG+S9IpwA6SOM/oQEjPAb6QfhcHgH+MiK9Iehu0XOqi3XG+HhiRtAN4Cjit3X/ogXcCG9N/738KvKWE+zJPnGXYl0gaAn4L+NOqaaXbnzni7Pj+jIhbJF1DcnppB3A7sL7V3OnSCmZmPcJX2pqZ9QgnfDOzHuGEb2bWI5zwzcx6hBO+mVmPcMI3M+sRTvhmZj3i/wOdLNk3yalUlAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# visualize result\n", "\n", "cat1 = np.asarray(points_set[0])\n", "cat2 = np.asarray(points_set[1])\n", "cat3 = np.asarray(points_set[2])\n", "\n", "for ix, p in enumerate(centers):\n", " plt.scatter(p[0], p[1], color='C{}'.format(ix), marker='^', edgecolor='black', s=256)\n", " \n", "plt.scatter(cat1_[:,0], cat1_[:,1], color='green')\n", "plt.scatter(cat2_[:,0], cat2_[:,1], color='red')\n", "plt.scatter(cat3_[:,0], cat3_[:,1], color='blue')\n", "plt.title('Hierarchical clustering with k=3')\n", "plt.xlim(4, 8)\n", "plt.ylim(1, 5)\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "colab": {}, "colab_type": "code", "id": "HiFFoBdWN4fW" }, "outputs": [], "source": [ "# using sklearn\n", "from sklearn.cluster import KMeans\n", "kmeans = KMeans(n_clusters=3, max_iter=100).fit(data)\n", "gt_labels__ = kmeans.labels_\n", "centers__ = kmeans.cluster_centers_" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 143 }, "colab_type": "code", "id": "LgKRAwmxObYS", "outputId": "032bcef5-ba68-4707-fbb0-f7a225ee968f" }, "outputs": [ { "data": { "text/plain": [ "array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 0, 0, 0, 2, 0, 2, 0, 2, 0, 2, 2, 2, 2, 2, 2, 0,\n", " 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 0, 2,\n", " 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 2, 0, 0, 0, 0, 2, 0, 0, 0,\n", " 0, 0, 0, 2, 2, 0, 0, 0, 0, 2, 0, 2, 0, 2, 0, 0, 2, 2, 0, 0, 0, 0,\n", " 0, 2, 2, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 2])" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "gt_labels__" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 71 }, "colab_type": "code", "id": "A0iu18HPOcrH", "outputId": "5794f33c-4ebd-47dc-d351-05f44c17343d" }, "outputs": [ { "data": { "text/plain": [ "array([[6.81276596, 3.07446809],\n", " [5.006 , 3.428 ],\n", " [5.77358491, 2.69245283]])" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "centers__" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 281 }, "colab_type": "code", "id": "wEfO5JVjOC5p", "outputId": "a6386044-2c7b-420d-f429-0b4960900f83" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de5gcdZ3v8fdnLgEGSIIQAQmZuKJw8C5zuGx23Wjiioh4fHCPuFGER0/WiRdc2dWjnsPV4OXZVVwROEFXwOBtcXURWVdQUcEVN1wEEd1FzUBkkMglgEEgyff8UTVJT6d7qnq6prt66vN6nn7SXVVd/a2azndq6vurbykiMDOz2a+v2wGYmVlnOOGbmVWEE76ZWUU44ZuZVYQTvplZRTjhm5lVhBN+iUlaL2l5t+PoJkkrJH2rw595hqS1rc4rK0m3S1o6xfxrJb0l57qWStpQWHDWUU74VmoRcVlE/Hm34+hlEfHsiLgWyvELS9KhktZJejB9XCPp0G7GVBVO+GYlIGmg2zF00D3Aa4GnAPsAVwBf7GpEFeGE3yMkHSLp15JOSF+vl/S3km6V9HtJn5G0r6R/lfRIetS0V837j5T0Q0kPSfpJ7Z/4kk6WdEf6vl9J+quaeUslbZB0qqT7JI1LOrlm/jGSfpa+9zeS/qZJ/JOOLCUtlhQTiU7SSelnP5Ju54qa6dfVvC8kvVXSf6VHh5+SpHRev6S/l/S7dB1vr/2MBjG9N435EUm/kLSswTKDkr4g6SuS5jSY3+5+fa+ke4HPZu3rus99iaTbal5fI+nHNa+vk/Q/0ufrJS2XdDTwfuB1kh6V9JOaVQ5Luj6N9VuS9mn0uQ3ieGf681+YZ3mAiHgoItZHcpm/gK3AQXnfb22ICD9K+gDWA8uBFwF3AcfWzfsRsC9wAHAfcBPwQmAX4DvA6emyBwD3A8eQ/JJ/Wfp6QTr/lcAzSP7z/RmwGXhROm8psAU4CxhM17EZ2CudPw78afp8r4n3NdiWM4C1Na8XAwEMALsDDwMHp/P2B56dPj8JuK7mfQFcCcwHFgEbgaPTeW8FfgYsTGO5ZuIzGsRzMHA38LSaeJ5RGyuwG/AN4GKgv347CtqvH0l/Xrtl7eu6+HcFHiM5Qh4A7iU5ct4zXddjwN6136NGP4d02rXAL4Fnpe+9Fvhwk5/jUmBD+vz/knznJrZ3EfDQFI+/rFvXQ+n2bgP+T7f/v1Xh4SP88vtTkj953xQRV9bN+2RE/DYifgP8ALghIm6OiMeBr5Ikf4A3AFdFxFURsS0irgbWkSQUIuIbEfHLSHwP+Fb6uROeBM6KiCcj4irgUZKEOTHvUElzI+LBiLhpmtu5DXiOpN0iYjwibp9i2Q9HcpR4F/Bd4AXp9P8JfCIiNkTEg8CHp1jHVpJEe6ikwUiOOH9ZM38u8E2SRHhyRGxtsI529+s2kl/Kj0fEY+m0qfb1dhHxh/SzXgyMALcC1wFLgCOB/4qI+6fY/nqfjYj/TOP4Mjv2aSOS9DHg5cBLImJjGtNdETF/isfn67ZhPjAPeDtwcwux2jQ54ZffW4EfRsR3G8z7bc3zxxq83iN9Pgz8RXra4SFJDwF/QnIkjaRXSPqRpAfSeceQHDlOuD8ittS83lyz7uPT5cckfU/SUa1uYET8Hnhduq3jkr4h6ZAp3nJvk1ieRnLUPqH2ef1n3gm8i+SI9z5JX5T0tJpFjgSeR/LLpVmHwXb368Y0cdeaal/X+x7JEfeL0+fXkvwl8Wfp61Y026eNzAdWAh+KiE0tfs4k6c/+QuBSSU9tZ12WzQm//N4KLJL08TbWcTfwubqjrd0j4sOSdgG+AvwdsG961HUVyWmITBHxHxHxauCpwNdIjg4b+T0wVPN6v7r1/FtEvIwkWf4cuKiF7ZswTnI6Z8KBUy0cEZ+PiD8hSdxBcnplwreADwHflrRvk1W0u1/bbVVbn/C/R3bCL6I97oPAsSR1hyUTEyUtSmsDzR4rmqyvj+S7cUABsdkUnPDL7xHgaODFkqY6RTGVtcCrJL08LWzumhYIFwJzSE5tbAS2SHoFkGsYpKQ5SsbJz4uIJ0nOwzc69QFwS7oNiyTNA95Xs559JR0naXfgcZLTGM3WM5UvA6dIOkDSfOC9U8R+sKSXpon5DyR/EU36zIj4KPB5kqTfqIg5I/u1BT8kOd1zOPDj9DTYMHAE8P0m7/ktsFhSW//3IxnmuQL4qqQj0ml3RcQeUzwuA5D0MkkvTPfZXOBjJL9E7mgnJsvmhN8DIuIhkoLgKySdPY333w28mmSExkaSI9O/Bfoi4hHgnSTJ8kHgL0lqBnm9EVgv6WGSv0be0CSGq4EvkZxrvpGk8DqhDziVpOj4AMkR6qoWYphwEcmR+a0k54SvIikKNvrlsQvJOf7fkZzOeCrJ/qmP+2ySv1yukfSUunkzuV8zpadDbgJuj4gn0sn/DoxFxH1N3vZP6b/3S5puvWXi868GTgaukHRYC2+dD3wB2ERSIzmIpPBef3rLCqbmpyfNelt6VH1hRAx3OxazMvARvs0aknZTcl3AgKQDgNNJRiuZGTmP8CWtJzmXvBXYEhEjdfMFfIId44ZPamN4ntm0SBoiKVYeQnJO/hvAKRHxcFcDMyuJVi7nfklE/K7JvFcAz0wfRwAXpP+adUxEbAb+e7fjMCurok7pvBq4NL3A5EfAfEn7F7RuMzMrQN4j/AC+JSmA/xcRa+rmH8Dki1w2pNPGaxeStJLkgg123333ww45ZKpra8zMrN6NN974u4hYMJ335k34SyLinvRKuKsl/Twiasf5NrpIZ6fiQPqLYg3AyMhIrFu3ruWAzcyqTNLYdN+b65RORNyT/nsfyaiHw+sW2cDkqxoXkoypNjOzkshM+JJ2l7TnxHOSqwV/WrfYFcCJShwJbIqIcczMrDTynNLZl+Ty6YnlPx8R35T0VoCIuJDkisZjgDtJhmU27OFtZmbdk5nwI+JXwPMbTL+w5nkAbys2NDMzK5KvtDUzqwgnfDOzinDCNzOrCCd8M7OKcMI3M6sIJ3wzs4pwwjczqwgnfDOzinDCNzOrCCd8M7OKcMI3M6sIJ3wzs4pwwjczqwgnfDOzinDCNzOrCCd8M7OKcMI3M6sIJ3wzs4rInfAl9Uu6WdKVDeadJGmjpFvSx1uKDdPMzNqV5ybmE04B7gDmNpn/pYh4e/shmZnZTMh1hC9pIfBK4NMzG46Zmc2UvKd0zgXeA2ybYpnjJd0q6XJJB7YfmpmZFSkz4Us6FrgvIm6cYrGvA4sj4nnANcAlTda1UtI6Ses2btw4rYDNzGx68hzhLwGOk7Qe+CLwUklraxeIiPsj4vH05UXAYY1WFBFrImIkIkYWLFjQRthmZtaqzIQfEe+LiIURsRg4AfhORLyhdhlJ+9e8PI6kuGtWnMsug8WLoa8v+feyy7odkVnPaWWUziSSzgLWRcQVwDslHQdsAR4ATiomPDOS5L5yJWzenLweG0teA6xY0b24zHqMIqIrHzwyMhLr1q3rymdbj1m8OEny9YaHYf36Tkdj1lWSboyIkem811faWvnddVdr082sISd8K79Fi1qbbmYNOeFb+a1eDUNDk6cNDSXTzSw3J3wrvxUrYM2a5Jy9lPy7Zo0LtmYtmvYoHbOOWrHCCd6sTT7CNzOrCCd8M7OKcMI3M6sIJ3zrDLdGMOs6F21t5rk1glkp+AjfZt4HPrAj2U/YvDmZbmYd44RvM8+tEcxKwQnfZp5bI5iVghO+zXxB1a0RzErBCb/qJgqqY2MQsaOgWmTSd2sEs1JwP/yqc695s57ifvg2fS6omlWGE37VuaBqVhlO+FXngqpZZeRO+JL6Jd0s6coG83aR9CVJd0q6QdLiIoO0GTSbCqpu32A2pVZaK5wC3AHMbTDvzcCDEXGQpBOAjwCvKyA+64TZ0Gve7RvMMuU6wpe0EHgl8Okmi7wauCR9fjmwTJLaD88sJ7dvMMuU95TOucB7gG1N5h8A3A0QEVuATcDe9QtJWilpnaR1GzdunEa4Zk14tJFZpsyEL+lY4L6IuHGqxRpM22mAf0SsiYiRiBhZsGBBC2GaZfBoI7NMeY7wlwDHSVoPfBF4qaS1dctsAA4EkDQAzAMeKDBOK7NVq2BgICn6DgwkrzvNo43MMmUm/Ih4X0QsjIjFwAnAdyLiDXWLXQG8KX3+2nSZ7lzCa521ahVccAFs3Zq83ro1ed3ppD+bRhuZzZCWWitIWgr8TUQcK+ksYF1EXCFpV+BzwAtJjuxPiIhfTbUut1aYJQYGdiT7Wv39sGVL5+Mxm+Xaaa3Q0h2vIuJa4Nr0+Wk10/8A/MV0ArAe1yjZTzXdzLrGV9pae/r7W5tuZl3jhG/tmbi4Ke90M+saJ/zZbvnypIg58Vi+vNj1n38+jI7uOKLv709en39+sZ/jtglWYpfddhmLz11M35l9LD53MZfd1vr3s4h1ZHE//Nls+XL49rd3nr5sGVxzTefjma76tgmQDLn0KBwrgctuu4yVX1/J5id3fD+HBodY86o1rHhuvu9nK+top2jrhD+bTdXdopdGzfomLVZii89dzNimnb+fw/OGWf+u9YWvwzdAsdnNbROsxO7a1Ph72Gz6TK0jDyd8Kz+3TbASWzSv8few2fSZWkceTviz2bJlrU1vJqtgOtMFVbdNsBJbvWw1Q4OTv59Dg0OsXpb/+1nEOnKJiK48DjvssLAZtnZthBSRnLFPHlIyvZV1DA1NXsfQ0I51ZM0vcluGh5P4h4eLX79ZG9beujaGPz4cOkMx/PHhWHtr69/PvOsg6XAwrbzrou1sVkSxM2sdLqiadZSLttZYEcXOrHW4oGrWM5zwZ7Miip1Z63BB1axnOOFPV6eu/GynYFpEsTNrHS6o2gzqxNWnlTLdk//tPnq6aNvJQmW7BdMiip1Z63BB1WbA2lvXxtDqoeAMtj+GVg9NqyA6m+CibYd1qlDpgqlVWBFXsM5GLtp2WqcKlS6YWoV16urTKnHCn45OFSpdMLUK69TVp1XihD8dnSpUumBqFdaxq08rJDPhS9pV0o8l/UTS7ZLObLDMSZI2SrolfbxlZsItiU7dMDvrc1asgKOOImD7g6OOmhzHqlXJfWel5N9GNxfvxIgj97O3Fq147grWvGoNw/OGEWJ43nBLLYetgayqLiBgj/T5IHADcGTdMicB57VSLe7pUTplMToa22pH6EDyenR0+/yomx+18yM6M+KoU6OazCqATo3SkTQEXAeMRsQNNdNPAkYi4u1519XTo3TKYmCg8c3C+/thy5bs+dCZkT4eTWRWmBkfpSOpX9ItwH3A1bXJvsbxkm6VdLmkA5usZ6WkdZLWbdy4cTrxWo1olMxrpzeZP2l6J0b6eDSRWSnkSvgRsTUiXgAsBA6X9Jy6Rb4OLI6I5wHXAJc0Wc+aiBiJiJEFCxa0E7cB25rc0Wr79In7zNarnd6JkT4eTWRWCi2N0omIh4BrgaPrpt8fEY+nLy8CDiskujIrogiZp6DaxPj4OP/YJ+pPyAXw2X5x7733JveBbaR2+urVMDg4ef7gYGsjfbK2o4dGE7V7KX+e97tdgHVLnlE6CyTNT5/vBiwHfl63zP41L48D7igyyNKZuKn22FhSghwbS163kvRXrYILLthxemXr1uR1zqT/0XPO5pB9RP0xvoCD9xYfPedsWLIkScC1BgaS6ZPepKlfTyXPdnRqVFObJm4kPbZpjCAY2zTGyq+vzJ2Q87y/3c8wa0dm0VbS80hO0fST/IL4ckScJekskmrxFZI+RJLotwAPkBR1f950pfR40baIImSegmoT4+PjPPvgZ3D/o4+hBj++EOyz5278du7eDGzYMHWc7W5LG9tRNu1eyp/n/W4XYO1qp2g7kLVARNwKvLDB9NNqnr8PeN90AuhJRRQh8xRUm/joOWfzpuf1o+sbz1fAic/tp//6BskeJsfZ7ra0sR1l0+6l/Hne73YB1k2+0nY6iihC5imoNjA+Ps4ll1zMe46Anc7nTBC85wjY0OzUTG2c7W7LNLejjNq9lD/P+90uwLrJCX86iihC5imoNjBxdL//nn1wWJM/0A4bYP89+7j2oDk8UX8Ovz7OdrdlmttRRu1eyp/n/W4XYF013Su22n30/JW2RfSAHx2N6O9Prjzt7598BWwD99xzT+y1525xz7v3iDh9bvJ4et1Nyp+u7fPuefcecdFAX2zL+ox2t6XF7Sizdm9Gnef9yy5ZNqnH+7JLlhUVfm6jV45G/5n9wRlE/5n9MXpl7/7Mqgb3w6+Gv37HKrj5c3x8efqH2W1PwNf/AE/WLDQIvGpXeO4cuO0JnvjaH5izrWb+0FApR8hUxapvrOKCdRfsNH10ZJTzX3l+ZWKw6WunaOuE3yMmRubc/r/S0zkA5z4Cmxr8/OYJ3rVn8/luadA1A2cNsDV2Lmj3q58tp3VmVFMZYrDp8w1QKmDSufsJjZJ57fRm893SoGsaJdqpps/WGKw7ModlWjn8yxVX8Ou7HuXcmqGYvwYWN1h2PfD0Mx9uOt8tDbqnX/1Nj66rFIN1R3WP8NttjZDn/W20Tqj3q7ENOxVgFq9dC3PmTF5wzhwWr127Y36PtDTohDK0NFh5WOPRS7XTV31jFQNnDaAzxcBZA6z6xuTvTbvbkSeGPJ/T7vw8yvAzm1WmW+1t99HVUTrt9mfP8/48veiL2I7BwcnrHxycHEcRo4lmgbW3ro2h1UOTRscMrR5qeRROEaYaITN65eikGCceE8sUtR1Zo3SyPqfd+XmU6WdWJniUTovabSeQ5/2daDngPvO59UpLg6yCaqe2I+tz2p1fRAxV5aJtq9ptJ5Dn/Z1oOeA+87n1SkuDrIJqp7Yj63PanV9EDNa6aib8dtsJ5Hl/J1oOuM98br3S0qBZ4XRieqe2o9n6DtjjgFxxFBFnr/zMekk1E3677QTyvL8TLQd6qM98t/VKS4OsgmqntqPR57Cln0PGn50rjiLi7JWfWU+Z7sn/dh9db63QiXYCWcvkiaGIdVhEtN82oVOyWi8UsR152juc973zQn+t4HSi/7QFMf/db42hPefH+Ph4REQcet6hk9Zx6HmHFh5nr/zMOgkXbTts4gYomzfvmNZqy4I865i4uUi90VE435fAz0YTN0jZ/OSO78XQ4BBrXrWGFc8tph3G8kuX8+1ff3un6cuevoxrTrxm++tV7ziFL//H3eyx9M3bpz167Wd43eGL+MXIT3Otw4rn1gqdVsTomLKM9LFS6cTIFJ3Z/I5mcXqSD8bHx3nGwf+Np5z4SQb2eMr2+VsefYAHL30nm099KHMdNjM8SqfTihgdU5aRPlYqZRmZcvY5H2bo0JdOSvYAA3s8hd0OfQk73UzZeoIT/nQUMTqmLCN9rFTKMDJlfHyciy+5hF1HXtNwfrPpVn55bmK+q6QfS/qJpNslndlgmV0kfUnSnZJukLR4JoLNLU/bg3ZaKxQxOmb16p0Td39/6yN92m0RUYAiQsjqQlHIZ2S0LIDsS/nzrKMdeUamLL90OTpT2x/LL13e0nYse/qyhp89MX3i6P4P837Chl1OZmzXV7Fhl5N5tP+7QHKU3/+7fRoe5TdbdzOdaM9QhF6JM0ueI/zHgZdGxPOBFwBHSzqybpk3Aw9GxEHAx4GPFBtmCyaKoWNjSbOBsbHkdW2GyLPMVFasSIqrw8NJhhoebr3H/PXX73xqZuvWZPqEJUuSDFerry+ZXsR2FKCIECZq0xO7Y+vW5PVE0i/kM9Ie8BMXMG2NrVyw7oJJCXuiYDq2aYwgGNs0xsqvr9z+nzfPOtq14rkrWPOqNQzPG0aI4XnDkwq2jQqu3/71tycl/azteNbez2r42c/a+1nbj+63LtmXBwbPY2vfRlCwtW8jDwyetz3p78fHYGzyd7PVgm1WnFnzO6VX4syjpaKtpCHgOmA0Im6omf5vwBkR8e+SBoB7gQUxxcpnrGibpxhahpYEeQqyWXGWYDuKCCFrVxTyGTl6wGcVTMvQRz5PwbWd7Vi58W18+T/u5qGXX5ck+/plti1g4eOfBXaM2PnUP5w7rW3pRHuGIpQtzhkv2krql3QLcB9wdW2yTx0A3A0QEVuATcDeDdazUtI6Ses2btz5y1SIPMXQMrQkyFOQzYqzBNtRRAhZu6KQz8jRAz6rYNorfeTb2Y6Jc/db9bvGy9RM33XkNVx88SXce++9MxJnWQrYvRJnHrkSfkRsjYgXAAuBwyU9p26RRocdOx3dR8SaiBiJiJEFCxa0Hm0eeYqhZWhJkKcgmxVnCbajiBCydkUhn5HRsgCyC6Z51lEG090OhbaPzOmPfRouUzt9YsTO2ed8eEbiLEMBO08cZYkzj5ZG6UTEQ8C1wNF1szYABwKkp3TmAQ8UEF/r8hRUy9CSIE9BNivOEmxHESFk7YpCPiNHD/isgmnePvIzKavgCtPfDt0ysH0EzvwtJ6LYZfL82IX5W06cNK2do/xOtGcoQq/EmUvWpbjAAmB++nw34AfAsXXLvA24MH1+AvDlrPXOaGuFIloWdEKeGLK2pQStFYoIYdmyyW39l9Vd6V/EjyurB3yeZYpoe5C1TNb8PG0Rsrajfv6h73lu7H3Ea2L4vVduf+z9/lOj/7QFwemK/tMWxN7vP3XS/O3LHfGaWPWOU6be+U20uy861XqhLHFEtNdaIU/Cfx5wM3Ar8FPgtHT6WcBx6fNdgX8C7gR+DPxR1np7+gYoVqisH0enflyduOlHWW8csnDRcJCchp3WY+Gi4Wns8fZU9QYp7ST8arZWKMHoFtuhLIOROnHTD984pDhV2c56bq3QqhKMbrEdyjIYqRM3/fCNQ4pTle0sUjUTfglGt9gOZRmM1O5ojDyjNXzjkOJUZTuLVM2EX4LRLbbD6tUwODh52uDg5MFIWV0ooP32C3lGY/TV/Zfpo2/S/MG+yRsy2Dc4abRG1jJZnwHZl/EXNWqk7O0C8m5nJ7aj7PtqQjUTfhGtEaxQUvPXebpQfOpTD/LGN25uq/1CVluD6++6nm1sm/SebWzj+rt2BKK6Dal/nbVM1mfkuYw/azvy6IV2AXm2sxPb0Qv7akI1i7ZWKllF2TxdKPacez+PPrLTxd2FFnazWisUUbQt4jOKMFsKop3YjlnXWsFsJmUVZbNaL4yPj/PoI3u1tO7pyGqtUETRtojPKMJsKYh2Yjt6aV854VvXZRVls1ovfPBDH6Rv19+0tO7pyGqtUETRtojPKMJsKYh2Yjt6aV854VvXe+qvXg1z5kyeNmfOjqLsVK0Xkna+F/PU154Ldee+J9bdiqmKb1mtFfIUEVcvW82c/skbO6d/Tu72DZ26jL+n2gVMoRPb0Uv7ygm/6krQUx+Sj272+vzzk/u2TxzR9/fvuI/7Bz/0Qeb98Tzu/85f0aiH3znn5I8hq/h2/ivPZ3RkdPvRdr/6GR0Z5fxXJjeUz1ssra+b1b5esmhJw1E6SxYtaekz2tWpz5lpndiOXtpXLtpWXQmuOp5uCOPj4xx0yEEceNaB/OJdd9C4aevOv0yaxlGCAt9sKZbazHHR1qavBFcdTzeEiaP7wfmDUy+YN44SFPh6qQBovccJv+pKcNXxdEKYOHc/9+i5GWvP/xdsGQp8vVQAtN7jhF91JbjqeDoh1B/dDz7tv9g5uQd77dV49E7DOEpQ4OulAqD1Hif8quvQVcdTDQTKE0Lt+xcu3MJFF22edHR/8DnH0zf/Xmq79mruPTyx7ZDcN+fIe+VmO5fQZ31GLxUAbYdeaa3goq3NuImBQJs375g2NJT/90qj99O3mYVvOZP5f3wVAA/98Bh+c/EZxBO7bV9Ecx5j7iGn8oalfZz3ifPa3450FM/mJ3cEMjQ45IRccZ3+XrRTtHXCtxnX7kCgZu8f3PseDv77lwPwi1P/jSfvf9pOywzs9Rt22XYwd/78Tvbbb7+W4t4pDo+gsQbcWsGsRrsDgZot9+T9+zV8XmvLg/sz96i5fPBDH8z3YVPF4RE01kAvfS+c8G3GtTsQqNlyg3vf2/B5/TJzj57LxZdcPK0bbU+KwyNorIFe+l5kJnxJB0r6rqQ7JN0u6ZQGyyyVtEnSLenjtJkJt4LabXvQ5bYJkN06AaYOs1G/fPqfYN/jP7H95b7HfwLNeWzSIprzGPse/wkG5w8WcpRf1AiaVauSDqBS8u+qVW2FNS29UmTsBT01sirrprfA/sCL0ud7Av8JHFq3zFLgylZuptvVm5j3inbv3l2Sm7WvXRsxODg5jMHB/DcpX7s2Ys6cyfPhDwGvr7uZ9usDfh2wNf138vyFwwvb35Zb18bwx4dDZyiGPz7c8g2zR0frtyN5jI62HVpuVb3590xq93vRCjp5E3NJ/wKcFxFX10xbCvxNRBybdz0u2uYwU9XODt+svd2blJdkMwqRp7f/THPxubd1bJSOpMXA94HnRMTDNdOXAl8BNgD3kCT/2xu8fyWwEmDRokWHjTX6X2w79PU1bgQjwbadO0MW/v6CZIXR7vxe0uAGWNt1asBc35l9RIMrkIXYdnqP7dAK6sgoHUl7kCT1d9Um+9RNwHBEPB/4JPC1RuuIiDURMRIRIwsWLJhOvNUyU9XODt+svd2blJdkMwqR1du/E3qpyGjFypXwJQ2SJPvLIuKf6+dHxMMR8Wj6/CpgUNI+hUZaRe22PShB24Q8YbQ7v5dM1du/U3qqyGjFyjrJT9Jz9lLg3CmW2Y8dp4cOB+6aeN3s4aJtTqOjEf39SWWvv7/16t7atRHDwxFS8m+HC7YTli2bXKRctmzy/KwwS7IZhWj3R1qEThYZrVjMZNFW0p8APwBuY8cthd4PLEp/YVwo6e3AKLAFeAx4d0T8cKr1umibQ7s9CUpi1Sq44IKdp0/cxMTM8nNrhdlqlgxPKcPIFLPZwq0VZqsS3JykCI2S/VTTzWxmOOGX2SwZnlKGkSlm5oTfXAlaEuQanlKGODOUYWRKmfTAj8xmq+lWe9t9lHqUTklaEmyPpdnwlDLFOYW1ayP6+iaH2ddXujA7okd+ZFZidLK1QlFKXbTtlWJpj8TZI2F2hPeFtcujdIrWK9fy90icPRJmR3hfWLs8SqdovVIs7ftc2qIAAAjsSURBVJE4eyTMjvC+sG5ywm+kV67l75E4eyTMjvC+sK6a7sn/dh+lLtpG9M61/D0SZ4+E2ZFOFr2yL6yccNHWrH3ttoCYJZ0wrORctDUrQLstIDwCxzrBRVuzArTbAmKWdMKwWcwJ3yzVbgsIj8CxsnPCN0u12wLCI3Cs7JzwzVLnn58UaCeO6Pv7W+vZv2JFUqAdHk4upBoedsHWysVFWzOzHuKirZmZZXLCNzOriMyEL+lASd+VdIek2yWd0mAZSfoHSXdKulXSi2YmXNuJm6ubWU4DOZbZApwaETdJ2hO4UdLVEfGzmmVeATwzfRwBXJD+azOp/tLOsbEdQ0pcKTSzOplH+BExHhE3pc8fAe4ADqhb7NXApWmrhx8B8yXtX3i0NtkHPjD5On5IXn/gA92Jx8xKraVz+JIWAy8EbqibdQBwd83rDez8SwFJKyWtk7Ru48aNrUVqO/OlnWbWgtwJX9IewFeAd0XEw/WzG7xlp/GeEbEmIkYiYmTBggWtRWo786WdZtaCXAlf0iBJsr8sIv65wSIbgANrXi8E7mk/PJuSL+00sxbkGaUj4DPAHRHxsSaLXQGcmI7WORLYFBHjBcZpjfjSTjNrQZ5ROkuANwK3SbolnfZ+YBFARFwIXAUcA9wJbAZOLj5Ua2jFCid4M8slM+FHxHU0Pkdfu0wAbysqKDMzK56vtDUzqwgnfDOzinDCNzOrCCd8M7OKcMI3M6sIJ3wzs4pwwjczqwgnfDOzinDCNzOrCCd8M7OKcMI3M6sIJ3wzs4pwwjczqwgnfDOzinDCNzOrCCd8M7OKcMI3M6sIJ3wzs4rIcxPzf5R0n6SfNpm/VNImSbekj9OKD9PMzNqV5ybmFwPnAZdOscwPIuLYQiIyM7MZkXmEHxHfBx7oQCxmZjaDijqHf5Skn0j6V0nPLmidZmZWoDyndLLcBAxHxKOSjgG+Bjyz0YKSVgIrARYtWlTAR5uZWV5tH+FHxMMR8Wj6/CpgUNI+TZZdExEjETGyYMGCdj/azMxa0HbCl7SfJKXPD0/XeX+76zUzs2JlntKR9AVgKbCPpA3A6cAgQERcCLwWGJW0BXgMOCEiYsYiNjOzaclM+BHx+oz555EM2zQzsxLzlbZmZhXhhG9mVhFO+GZmFeGEb2ZWEU74ZmYV4YRvZlYRTvhmZhXhhG9mVhFO+GZmFeGEb2ZWEU74ZmYV4YRvZlYRTvhmZhXhhG9mVhFO+GZmFeGEb2ZWEU74ZmYV4YRvZlYRTvhmZhWRmfAl/aOk+yT9tMl8SfoHSXdKulXSi4oP08zM2pXnCP9i4Ogp5r8CeGb6WAlc0H5YZmZWtMyEHxHfBx6YYpFXA5dG4kfAfEn7FxWgmZkVY6CAdRwA3F3zekM6bbx+QUkrSf4KAHi82WmiktkH+F23g8jBcRarF+LshRjBcRbt4Om+sYiErwbTotGCEbEGWAMgaV1EjBTw+TPKcRbLcRanF2IEx1k0Seum+94iRulsAA6seb0QuKeA9ZqZWYGKSPhXACemo3WOBDZFxE6nc8zMrLsyT+lI+gKwFNhH0gbgdGAQICIuBK4CjgHuBDYDJ+f87DXTiLcbHGexHGdxeiFGcJxFm3acimh4ut3MzGYZX2lrZlYRTvhmZhXRkYQvqV/SzZKubDBvF0lfSlsz3CBpcSdiaiQjzpMkbZR0S/p4S5diXC/ptjSGnYZnlaXVRY44l0raVLM/T+tCjPMlXS7p55LukHRU3fyy7MusOMuwLw+u+fxbJD0s6V11y3R9f+aMs+v7M43jryXdLumnkr4gade6+S3nziLG4edxCnAHMLfBvDcDD0bEQZJOAD4CvK5DcdWbKk6AL0XE2zsYTzMviYhmF4jUtro4gqTVxRGdCqzOVHEC/CAiju1YNDv7BPDNiHitpDnAUN38suzLrDihy/syIn4BvACSAyfgN8BX6xbr+v7MGSd0eX9KOgB4J3BoRDwm6cvACSStbia0nDtn/Ahf0kLglcCnmyzyauCS9PnlwDJJjS7mmlE54uwVbnWRg6S5wIuBzwBExBMR8VDdYl3flznjLJtlwC8jYqxuetf3Z51mcZbFALCbpAGSX/L11ze1nDs7cUrnXOA9wLYm87e3ZoiILcAmYO8OxFUvK06A49M/RS+XdOAUy82kAL4l6UYlrSrqNWt10WlZcQIcJeknkv5V0rM7GRzwR8BG4LPpabxPS9q9bpky7Ms8cUJ392W9E4AvNJhehv1Zq1mc0OX9GRG/Af4OuIukTc2miPhW3WIt584ZTfiSjgXui4gbp1qswbSOjhXNGefXgcUR8TzgGnb8Zu20JRHxIpI/j98m6cV187u+P1NZcd4EDEfE84FPAl/rcHwDwIuACyLihcDvgf9dt0wZ9mWeOLu9L7dLTzkdB/xTo9kNpnVlXHhGnF3fn5L2IjmCfzrwNGB3SW+oX6zBW6fcnzN9hL8EOE7SeuCLwEslra1bZntrhvRPl3lM3Z1zJmTGGRH3R8Tj6cuLgMM6G+L2OO5J/72P5Nzj4XWLlKLVRVacEfFwRDyaPr8KGJS0TwdD3ABsiIgb0teXkyTW+mW6vS8z4yzBvqz1CuCmiPhtg3ll2J8TmsZZkv25HPh1RGyMiCeBfwb+uG6ZlnPnjCb8iHhfRCyMiMUkfz59JyLqf0tdAbwpff7adJmO/tbPE2fducbjSIq7HSVpd0l7TjwH/hyo7zja9VYXeeKUtN/E+UZJh5N8F+/vVIwRcS9wt6SJzoPLgJ/VLdb1fZknzm7vyzqvp/lpkq7vzxpN4yzJ/rwLOFLSUBrLMnbOOS3nzk6N0plE0lnAuoi4gqQY9TlJd5L8djqhGzE1UhfnOyUdB2whifOkLoS0L/DV9Ls4AHw+Ir4p6a3QdquLTsf5WmBU0hbgMeCETv+iB94BXJb+ef8r4OQS7ss8cZZhXyJpCHgZ8Fc100q3P3PE2fX9GRE3SLqc5PTSFuBmYE27udOtFczMKsJX2pqZVYQTvplZRTjhm5lVhBO+mVlFOOGbmVWEE76ZWUU44ZuZVcT/B2KUAh0mKiXYAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# visualize result\n", "\n", "cat1 = data[gt_labels__ == 0]\n", "cat2 = data[gt_labels__ == 1]\n", "cat3 = data[gt_labels__ == 2]\n", "\n", "for ix, p in enumerate(centers__):\n", " plt.scatter(p[0], p[1], color='C{}'.format(ix), marker='^', edgecolor='black', s=256)\n", " \n", "plt.scatter(cat1_[:,0], cat1_[:,1], color='green')\n", "plt.scatter(cat2_[:,0], cat2_[:,1], color='red')\n", "plt.scatter(cat3_[:,0], cat3_[:,1], color='blue')\n", "plt.title('kmeans using sklearn with k=3')\n", "plt.xlim(4, 8)\n", "plt.ylim(1, 5)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "u5XVOLGBKC4A" }, "source": [ "#### 寻找 K 值" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 281 }, "colab_type": "code", "id": "uCe9-EHaJrFz", "outputId": "4c2fe667-ee92-4909-c582-6605bd862ecc" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3Rc5Xnv8e8jaXS1LEuyLsaSjA0GbJmLsTABGgcCtkzJAdJwTxqSE0pIoU2bnp6TnOQ0DV1N09DTNm1oEm45SUlCgDQpaQmCEMBJuVnmasnGGGMs2dbFlixblmSNpef8MdvOWJbtkSV5j2Z+n7Vmafbsd8888pJ/e8+73/1uc3dERCR1ZYRdgIiITC4FvYhIilPQi4ikOAW9iEiKU9CLiKQ4Bb2ISIpT0IsAZvZRM3vyKOsvNrPWMbzfs2Z2y8RUJzI+CnqZcsxss5ldFrd8g5l1m9kHjvc93f0H7r4i7j3dzE4db60iyUBBL1Oamd0M3A1c4e7PhV2PSDJS0MuUZWa3Av8XqHf354/Q5jkz+0jw/HeCI/XfDZYvM7PXguefMLPfBM9XBZu/bma9ZnZ93Pv9mZl1mNl2M/tkgnVmmNmXzOy9YNvvm1lRsC7XzB40s51mtsvMVptZRVxNm8xsj5m9a2YfPa5/KEl7CnqZqj4D/BVwqbs3HqXdc8DFwfNlwCbgA3HLh30LcPdlwdOz3X2au/84WK4EioDZwKeAu82sOIFaPxE8LgHmAdOAbwbrbg7esxooBW4D+s2sAPgn4HJ3LwQuBF5L4LNEDqOgl6lqOfAi8OYx2j3HocH+N3HLH2CUoD+KKHCnu0fd/XGgFzg9ge0+Cvy9u29y917gC8ANZpYVvGcpcKq7D7n7GnffHWw3DCwyszx33+7uTWOoVeQgBb1MVbcBpwH3mZkdpd0LwGlBd8g5wPeBajObCSwFVh1l25F2uvv+uOU+Ykfnx3IS8F7c8ntAFlAB/CvQADxkZtvM7OtmFnH3vcD1xH7P7Wb2n2Z2xhhqFTlIQS9TVQdwKfB+4F+O1Mjd+4A1wGeBte4+CDwPfA54x913nIBatwFz4pZrgP1Ae/Dt4CvuvpBY98yHgI8HtTe4+3JgFrAeuPcE1CopSEEvU5a7bwM+CKw0s384StPngDv4bTfNsyOWR9NOrD99IvwI+FMzm2tm04CvAj929/1mdomZnWlmmcBuYl05Q2ZWYWZXBn31+4h1Ew1NUD2SZhT0MqW5ewuxsL/GzP7mCM2eAwr5bTfNyOXR/CXwvWAkzHXjLPMBYl00q4B3gQHgj4J1lcCjxEJ+XVDbg8T+b/4ZsW8DXcTOJ/zhOOuQNGW68YiISGrTEb2ISIpT0IuIpDgFvYhIilPQi4ikuKxEGpnZSuAbQCZwn7t/bcT624DbiQ3/6gVudffmYN0XiF0uPgT8sbs3HO2zZs6c6SeffPIYfw0RkfS2Zs2aHe5eNtq6Y466Ccb3biB2yXkrsBq48UCQB22mH7hs28yuBP7Q3Vea2UJiY4iXErs68JfAae5+xPHAdXV13th4tKlLRERkJDNb4+51o61LpOtmKbAxmKdjEHgIuCq+QdzcHAAFwIG9x1XAQ+6+z93fBTYG7yciIidIIl03s4GWuOVW4PyRjczsdmKXlWcTu4DlwLYvjth29ijb3grcClBTU5NI3SIikqBEjuhHmzDqsP4ed7/b3U8B/hfwpTFue4+717l7XVnZqF1MIiJynBIJ+lZic2UfUEXssuwjeQi4+ji3FRGRCZZI0K8G5gcTMmUDNwCPxTcws/lxi1cAbwfPHyM273aOmc0F5gMvj79sERFJ1DH76IMZ9u4gNmd2JvCAuzeZ2Z1Ao7s/BtwR3Kw5CnQTu2sOQbuHgWZi07LefrQRNyIiMvGSblIzDa8UERm78Q6vnBJ29Q3yjV++TfO23cduLCKSRhK6MnYqMDP++Vdv0xfdz8KTpoddjohI0kiZI/qivAgXnFJKw9o2kq07SkQkTCkT9AArF1WyeWcfG9p7wy5FRCRppFTQL19YgRk8sbYt7FJERJJGSgV9eWEuS2qKaWhS0IuIHJBSQQ9QX1tJ8/bdtHT1hV2KiEhSSMmgB3RULyISSLmgrynNZ8Gs6eqnFxEJpFzQA9TXVrBmSzcdewbCLkVEJHQpGfQrF1XiDk81t4ddiohI6FIy6E+vKGROaT4NTQp6EZGUDHozY2VtJS+8s4Oe/mjY5YiIhColgx5gRW0l0SHnmfUdYZciIhKqlA36xdUzKC/M0TBLEUl7KRv0GRnGitoKnn2rk4Go7nUiIukrZYMeYGXtLPqjQ6za0Bl2KSIioUnpoD9/XglFeRGeUPeNiKSxlA76SGYGly4o5+l1HUSHhsMuR0QkFCkd9BCb+6anP8pLm7rCLkVEJBQpH/TL5peRG8nQ6BsRSVspH/R52ZlcfFo5DU1tDA/rFoMikn5SPugB6hdV0LFnH6+17gq7FBGREy4tgv6DZ1SQlWHqvhGRtJQWQV+UF+GCU0ppWNuGu7pvRCS9pEXQQ2zq4s07+9jQ3ht2KSIiJ1TaBP3yhRWYoTtPiUjaSZugLy/MZUlNsfrpRSTtJBT0ZrbSzN4ys41m9vlR1n/OzJrN7A0ze9rM5sStGzKz14LHYxNZ/FjV11bSvH03LV19YZYhInJCHTPozSwTuBu4HFgI3GhmC0c0exWoc/ezgEeBr8et63f3c4LHlRNU93Gpr60E0FG9iKSVRI7olwIb3X2Tuw8CDwFXxTdw92fc/cBh8otA1cSWOTFqSvNZMGu6+ulFJK0kEvSzgZa45dbgtSP5FPCLuOVcM2s0sxfN7OrRNjCzW4M2jZ2dkzul8MraStZs6aZjz8Ckfo6ISLJIJOhtlNdGHYxuZh8D6oC74l6ucfc64CbgH83slMPezP0ed69z97qysrIESjp+9YsqcIenmnXjcBFJD4kEfStQHbdcBWwb2cjMLgO+CFzp7vsOvO7u24Kfm4BngcXjqHfcTq8oZE5pPg1NCnoRSQ+JBP1qYL6ZzTWzbOAG4JDRM2a2GPgOsZDviHu92MxyguczgYuA5okq/niYGStrK3nhnR309EfDLEVE5IQ4ZtC7+37gDqABWAc87O5NZnanmR0YRXMXMA14ZMQwygVAo5m9DjwDfM3dQw16gBW1lUSHnGfWdxy7sYjIFJeVSCN3fxx4fMRrfxH3/LIjbPc8cOZ4CpwMi6tnUF6YQ0NTG1cvPtp5ZRGRqS9troyNl5FhrKit4Nm3OhmIDoVdjojIpErLoAdYWTuL/ugQqzZM7nBOEZGwpW3Qnz+vhKK8CE/oKlkRSXFpG/SRzAwuXVDO0+s6iA4Nh12OiMikSdugh9jcNz39UV7a1BV2KSIikyatg37Z/DLyIpma5ExEUlpaB31ediYfOK2MhqY2hod1i0ERSU1pHfQQu8Vgx559vNa6K+xSREQmRdoH/SVnlJOVYTRo6mIRSVFpH/RFeREuOKWUhqY23NV9IyKpJ+2DHmLdN5t39rGhvTfsUkREJpyCHli+sAIzdOcpEUlJCnqgvDCXJTXFGmYpIilJQR+or62keftuWrr6jt1YRGQKUdAH6msrAXRULyIpR0EfqCnNZ8Gs6eqnF5GUo6CPs7K2kjVbuunYMxB2KSIiE0ZBH6d+UQXu8FSzbhwuIqlDQR/n9IpCTi7Np6FJQS8iqUNBH8fMqK+t5PmNO+jpj4ZdjojIhFDQj1C/qJL9w84z6zvCLkVEZEIo6Ec4p2oG5YU5GmYpIilDQT9CRoaxoraCZ9/qZCA6FHY5IiLjpqAfxcraWfRHh1i1oTPsUkRExk1BP4rz55VQlBfhCXXfiEgKUNCPIpKZwaULynl6XQfRoeGwyxERGRcF/RHU11bS0x/lpU1dYZciIjIuCvojWDa/jLxIpkbfiMiUl1DQm9lKM3vLzDaa2edHWf85M2s2szfM7GkzmxO37mYzezt43DyRxU+mvOxMPnBaGQ1NbQwP6xaDIjJ1HTPozSwTuBu4HFgI3GhmC0c0exWoc/ezgEeBrwfblgBfBs4HlgJfNrPiiSt/cq1cVEnHnn281ror7FJERI5bIkf0S4GN7r7J3QeBh4Cr4hu4+zPufuCOHS8CVcHzeuApd+9y927gKWDlxJQ++S45o5ysDKNBUxeLyBSWSNDPBlrilluD147kU8AvxrKtmd1qZo1m1tjZmTxj14vyIlx46kwamtpwV/eNiExNiQS9jfLaqKlnZh8D6oC7xrKtu9/j7nXuXldWVpZASSdOfW0Fm3f2saG9N+xSRESOSyJB3wpUxy1XAdtGNjKzy4AvAle6+76xbJvMli+swAzdeUpEpqxEgn41MN/M5ppZNnAD8Fh8AzNbDHyHWMjHT/vYAKwws+LgJOyK4LUpo7wwlyU1xRpmKSJT1jGD3t33A3cQC+h1wMPu3mRmd5rZlUGzu4BpwCNm9pqZPRZs2wX8FbGdxWrgzuC1KaW+tpLm7btp6eo7dmMRkSRjyXaSsa6uzhsbG8Mu4xBbdvax7K5n+NIVC7jl/fPCLkdE5DBmtsbd60ZbpytjE1BTms+CWdPVTy8iU5KCPkEraytZs6Wbjj0DYZciIjImCvoE1S+qwB2eataNw0VkalHQJ+j0ikJOLs2noUlBLyJTi4I+QWZGfW0lz2/cQU9/NOxyREQSpqAfg/pFlewfdp5Z33HsxiIiSUJBPwbnVM2gvDBHF0+JyJSioB+DjIxY982zb3UyEB0KuxwRkYQo6MeovraS/ugQqzYkzyybIiJHo6Afo/PnlVCUF+EJdd+IyBShoB+jSGYGly4o5+l1HUSHhsMuR0TkmBT0x6G+tpKe/igvbZpy87OJSBpS0B+HZfPLyItkavSNiEwJCvrjkJedyQdOK6OhqY3h4eSa/VNEZCQF/XFauaiSjj37eK11V9iliIgclYL+OF1yRjlZGUaDpi4WkSSnoD9ORXkRLjx1Jg1NbSTbzVtEROIp6MehvraCzTv72NDeG3YpIiJHpKAfh+ULKzBDd54SkaSmoB+H8sJcltQUa5iliCQ1Bf04rVxUSfP23bR09YVdiojIqBT041RfWwmgo3oRSVoK+nGqLsln4azp6qcXkaSloJ8A9bWVrNnSTceegbBLERE5jIJ+AtQvqsAdnmrWjcNFJPko6CfA6RWFnFyaT0OTgl5Eko+CfgKYxW4x+PzGHfT0R8MuR0TkEAr6CVK/qJL9w84z6zvCLkVE5BAJBb2ZrTSzt8xso5l9fpT1y8zsFTPbb2bXjFg3ZGavBY/HJqrwZHNO1QzKC3M0zFJEkk7WsRqYWSZwN7AcaAVWm9lj7t4c12wL8Angf4zyFv3ufs4E1JrUMjJi3TePrmllIDpEbiQz7JJERIDEjuiXAhvdfZO7DwIPAVfFN3D3ze7+BpDWN1Gtr62kPzrEqg2dYZciInJQIkE/G2iJW24NXktUrpk1mtmLZnb1aA3M7NagTWNn59QNyfPnlVCUF+EJdd+ISBJJJOhtlNfGMgF7jbvXATcB/2hmpxz2Zu73uHudu9eVlZWN4a2TSyQzg0sXlPP0ug6iQ2n95UZEkkgiQd8KVMctVwHbEv0Ad98W/NwEPAssHkN9U87K2kp6+qO8tKkr7FJERIDEgn41MN/M5ppZNnADkNDoGTMrNrOc4PlM4CKg+ehbTW3LTisjL5Kp0TcikjSOGfTuvh+4A2gA1gEPu3uTmd1pZlcCmNl5ZtYKXAt8x8yags0XAI1m9jrwDPC1EaN1Uk5uJJOLTy+joamN4WHdYlBEwnfM4ZUA7v448PiI1/4i7vlqYl06I7d7HjhznDVOOfW1lfxibRuvte7i3JrisMsRkTSnK2MnwSVnlJOVYTRo6mIRSQIK+klQlBfhwlNn0tDUhru6b0QkXAr6SVJfW8HmnX1saO8NuxQRSXMK+kmyfGEFZujOUyISOgX9JCkvzGVJTbGGWYpI6BT0k2jlokqat++mpasv7FJEJI0p6CdRfW0lgI7qRSRUCvpJVF2Sz8JZ09VPLyKhUtBPsvraStZs6aZjz0DYpYhImlLQT7KViypxh3tXbdKYehEJhYJ+kp1WMY1rl1Rx76/f5Ss/b9b8NyJywiU0140cPzPjbz9yFtPzItz/m3fZ1TfIXdeeTSRT+1gROTEU9CdARobxpSsWUFKQzV0Nb7GrP8q3PrqEvGzdV1ZEJp8OK08QM+P2S07lqx8+k1UbOvnY/S/R0xcNuywRSQMK+hPspvNruPumc3mztYfrvvMC7bs1GkdEJpeCPgSXnzmL737yPFq7+/jIt55n8469YZckIilMQR+Si06dyQ//4H3s3befa779PGu39oRdkoikKAV9iM6unsEjt11IdmYGN97zIi9t2hl2SSKSghT0ITu1fBqPfuZCyqfn8PEHXuap5vawSxKRFKOgTwInzcjjkdsu5IzKQm57cA2PNLaEXZKIpBAFfZIoKcjmB3/wPi6YV8qfP/oG967aFHZJIpIiFPRJZFpOFvd/oo4rzpzFXz++jq/9Yr3mxxGRcdOVsUkmJyuTf7pxMUX5Eb793Dt07x3krz+8iCxNmSAix0lBn4QyM4y/vnoRpQXZ/POvNrKrf5Bv3LCY3IimTBCRsdNhYpIyM/5sxen8xYcW0tDUzie/u5o9A5oyQUTGTkGf5P7778zlH64/m5c3d3HTvS+xo3df2CWJyBSjoJ8CPry4ins/voQN7Xu47tsv0Nqtm42LSOIU9FPEB8+o4MFbzmdH7z6u+dYLvN2+J+ySRGSKSCjozWylmb1lZhvN7POjrF9mZq+Y2X4zu2bEupvN7O3gcfNEFZ6Ozju5hB9/+gKG3Ln2Oy/wypbusEsSkSngmEFvZpnA3cDlwELgRjNbOKLZFuATwA9HbFsCfBk4H1gKfNnMisdfdvpaMGs6P7ntQqbnRvjovS/x3IbOsEsSkSSXyBH9UmCju29y90HgIeCq+Abuvtnd3wCGR2xbDzzl7l3u3g08BaycgLrTWk1pPo9+5gJOnlnALd9bzc9f3xZ2SSKSxBIJ+tlA/OQrrcFriRjPtnIU5YW5PHTr+1hcXcwfP/Qq//rC5rBLEpEklUjQ2yivJXpdfkLbmtmtZtZoZo2dneqKSFRRXoTvf2opl55Rzv/59ya+8cu3NWWCiBwmkaBvBarjlquARPsKEtrW3e9x9zp3rysrK0vwrQUgN5LJtz62hN87dzb/8MsNfOXnzQwPK+xF5LcSmQJhNTDfzOYCW4EbgJsSfP8G4KtxJ2BXAF8Yc5VyVJHMDP7umrMpyc/mvt+8S3ffIH937dlEND+OiJBA0Lv7fjO7g1hoZwIPuHuTmd0JNLr7Y2Z2HvBToBj4b2b2FXevdfcuM/srYjsLgDvdvWuSfpe0lpFhfPGKBZRMy+brT7xFT3+Uf/noueRnazojkXRnydanW1dX542NjWGXMaX96OUtfPGnb3JO9Qwe+MR5zMjPDrskEZlkZrbG3etGW6fv9inoxqU13H3Tuazdupvrv/Mi7bsHwi5JREKkoE9Rl585i+9+8jxau/v4yLee590de8MuSURCoqBPYRedOpMf3fo++gaHuPbbz7N2a0/YJYlICBT0Ke6sqhk8/OkLyM7M4MZ7XuTFTTvDLklETjAFfRo4tXwaj37mQiqKcvn4Ay/zVHN72CWJyAmkoE8TJ83I45FPX8CCWdO57cE1PNLYcuyNRCQlKOjTSHFBNj+85XwuPKWUP3/0De5Z9U7YJYnICaCgTzMFOVncd3MdV5w5i68+vp7fv/8l/vON7ezbPxR2aSIySXTZZBrKycrkn25czMKTpvPgi+9x+w9foTg/wtWLZ3NdXTULZk0Pu0QRmUC6MjbNDQ07v367k0caW3myuY3okHNWVRHX1lVz5dknUZQXCbtEEUnA0a6MVdDLQV17B/nZq1t5uLGF9W17yMnK4PJFlVxXV8375pWSkTHarNMikgwU9DIm7s7arbv5ceMW/v21bewZ2E91SR7XLqnmI0uqmD0jL+wSRWQEBb0ct4HoEA1NbTzc2MJ/bdyJGbx/fhnX1VWxfGEFOVmZYZcoIijoZYK0dPXxyJpWHm1sYVvPADPyI1x9TuwE7sKTdAJXJEwKeplQQ8POf23cwcONLTzZ1M7g0DBnzi7iuroqrjx7NkX5OoErcqIp6GXS7OqLncD9cWMr67bvJjsrg5W1lVx/XjUX6ASuyAmjoJcTYu3WHh5ubOFnr25l98B+qopjJ3CvqdMJXJHJpqCXE2ogOsSTze08vLqF/3pnBwC/c+pMrqurZvnCCnIjOoErMtEU9BKalq4+fvJKK480trJ1Vz9FeRE+vHg219ZVUXtSUdjliaQMBb2EbnjYef6dnTzc2MITTW0M7h+m9qTpXFdXzdXn6ASuyHgp6CWp7Oob5LHXt/FwYwtrt8ZO4NbXVnJ9XTUXnqITuCLHQ0EvSatpWw+PNLby01e30tMfZfaMPK5ZUsXVi2czd2ZB2OWJTBkKekl6A9EhfrmunR+vbuE3G3fgDvPLp1FfW8mK2grOnF2EmY70RY5EQS9TyvaefhrWtvFkczsvvdvF0LAzqyiXFQsrqK+t5Ly5JUQydSsFkXgKepmyuvcO8vT6Dp5samPV250MRIcpyotw6YJy6msrWTa/jLxsDdcUUdBLSugfHGLV2500NLXx9LoOevqj5EYyWDa/jBW1lVx6RjnFBdlhlykSiqMFve4wJVNGXnYm9bWV1NdWEh0aZvW7XTQ0xbp4nmxuJzPDWHpyCfW1FSyvrdTVuCIBHdHLlOfuvLm1hyeb2mloauPtjl4AzpxdFOvXX1TJ/PJpOpkrKU1dN5JWNnX2xo7ym9p4ZcsuAObOLGDFwgpW1FayuHqGxupLyhl30JvZSuAbQCZwn7t/bcT6HOD7wBJgJ3C9u282s5OBdcBbQdMX3f22o32Wgl4mUsfugYNdOy+8s4PokFNWmMPyhRWsWFjBhafMJDtLI3hk6htX0JtZJrABWA60AquBG929Oa7NHwJnufttZnYD8GF3vz4I+v9w90WJFqugl8nS0x/l2bc6eLKpnWff6mDv4BCFOVlcfEY59bUVXHx6OdNydNpKpqbxnoxdCmx0903Bmz0EXAU0x7W5CvjL4PmjwDdNHaKSZIryIlx1zmyuOmc2A9Ehnn9nBw1r2/nlunZ+/vo2sjMzuOjUUuprK7l0QQVlhTlhlywyIRIJ+tlAS9xyK3D+kdq4+34z6wFKg3VzzexVYDfwJXf/9cgPMLNbgVsBampqxvQLiByP3EgmHzyjgg+eUcHQsLPmvW6ebGqjobmNZ/7tTczepG5OMSsWxkb51JTmh12yyHFLpOvmWqDe3W8Jln8fWOrufxTXpilo0xosv0Psm0AvMM3dd5rZEuBnQK277z7S56nrRsLk7qxv20NDUxsNTe2s2x77Uz2jspAVtZUsmVPMnJJ8Zhfn6epcSSrj7bppBarjlquAbUdo02pmWUAR0OWxvcg+AHdfE+wATgOU5JKUzIwFs6azYNZ0/uSy02jp6js4Vv+bv3qb4eC4KDPDOGlGLnNKCqgpzWdOST5zSvOpKSlgTmk+BerrlySSyF/jamC+mc0FtgI3ADeNaPMYcDPwAnAN8Ct3dzMrIxb4Q2Y2D5gPbJqw6kUmWXVJPre8fx63vH8e3XsHebujl/d27mVLVx/v7ezjva4+fvHmdrr7oodsN3NaNjUl+cwpLQh+/nZHMHNatsb0ywl1zKAP+tzvABqIDa98wN2bzOxOoNHdHwPuB/7VzDYCXcR2BgDLgDvNbD8wBNzm7l2T8YuITLbigmyWzi1h6dySw9btHoiyZeeB8N978PnL73bxs9e2Et9DWpCdSfXB8I/bEZQUcNKMXLLUJSQTTBdMiUyyffuHaO3uD8J/L+919cWed/WxpauPwf3DB9tmZRizi/MOCf+ag98G8snPVpeQjE5z3YiEKCcrk1PKpnFK2bTD1g0PO+17Bnhv54Hw3xt73tXHz1/fTk//oV1CZYU5zCnJD84LxM4H1AQ7gdICdQnJ6BT0IiHKyDBmFeUxqyiP980rPWx9T1/0kPB/b2fs+Qvv7OSnrx7aJZQXyaS6JI/q4nyqS/KpKs475Of0XN2XN10p6EWSWFF+hLPyZ3BW1YzD1g1Eh2jtjp0LaOnqo6W7/+DPl9/tYs++/Ye+V17kkB1BdXEeVSX5VBfHdga5Ec3rn6oU9CJTVG4kk1PLCzm1vPCwde5OT3+Ulq5+WroP7Aj6aOnq5632PTy9vuOQcwMA5YU5B3cA1Qd2AMGOYVaRThJPZQp6kRRkZszIz2ZGfjZnVhUdtn542Ons3XfIDuDA89Wbu3ns9W0HrxmA2EniWTNyY98GivNj3wxK8qkKnpdNy9H5gSSmoBdJQxkZRsX0XCqm51J38uHDRaNDw2zfNXDYt4GW7j6eXt/Bjt59h7TPjWTEQj/u20B1SR6VRXnMnJbNzGk56hoKkYJeRA4TycyIjeY5whw//YOx8wMjvw20dPXT+F43ewb2H7bNtJysg6E/c1oOMwuzKS3IYWZhDmVxr5dOy2ZaTpa+IUwgBb2IjFledibzKwqZX3H4+QGIjRZq6e6jrWeAnXv3saN3kM49+9jRG3u809nLS+/uO+yK4gNysjKCnUFsJxDbIcTtJKblHNxpzMiPaKdwDAp6EZlwRfkRivKLWDT78PMD8aJDw3Ttje0Edu4dZEfczmBH7yA7evexddcAr7f20LV3kKHhwy/wzMowSoPQLw12AGUjvzUEz0vys9PypLKCXkRCE8nMOHiu4FiGh53uvsGDO4TOuJ3BzrjnG9v3sKN3kMGh4cPewwxK8rMpKTj8UZyfTem02M/411Ph3IKCXkSmhIwMozQ4aj/tCF1GB7g7e/btD74hDB72LaGrd5CuYJK6rr2DdPcNcqTZYPKzM4+4Eziwg4hfLsqLkJlk9yRW0ItIyjEzpudGmJ4bYV7ZsdsPDTu7+6PsDEJ/Z2/sZ9fe2KN77+DBde90xnYOfYNDo75XhsGM/GyK8yOUFuRQXBA55reGyZ7DSEEvImkvM8MoLsimuCA74W0GokO/3REcYaews3eQd3fsZc17u+juG/0cAwfmc/sAAAS5SURBVMSGp5bkZ3PunGK+edO5E/VrHaSgFxE5DrmRTE6akcdJM/ISaj887OwZ2M/OvfuCHUOUrr376NobPbhTqCyanPsUK+hFRE6AjAwLRiOd+Mnl0m+ckYhImlHQi4ikOAW9iEiKU9CLiKQ4Bb2ISIpT0IuIpDgFvYhIilPQi4ikOPMjzeQTEjPrBN4bx1vMBHZMUDkTSXWNjeoaG9U1NqlY1xx3H3Vmn6QL+vEys0Z3rwu7jpFU19iorrFRXWOTbnWp60ZEJMUp6EVEUlwqBv09YRdwBKprbFTX2KiusUmrulKuj15ERA6Vikf0IiISR0EvIpLiUibozewBM+sws7Vh13KAmVWb2TNmts7Mmszss2HXBGBmuWb2spm9HtT1lbBrimdmmWb2qpn9R9i1HGBmm83sTTN7zcwaw67nADObYWaPmtn64O/sgrBrAjCz04N/qwOP3Wb2J0lQ158Gf/NrzexHZpYbdk0AZvbZoKamyfh3Spk+ejNbBvQC33f3RWHXA2Bms4BZ7v6KmRUCa4Cr3b055LoMKHD3XjOLAL8BPuvuL4ZZ1wFm9jmgDpju7h8Kux6IBT1Q5+5JdZGNmX0P+LW732dm2UC+u+8Ku654ZpYJbAXOd/fxXAw53jpmE/tbX+ju/Wb2MPC4u/+/sGoK6loEPAQsBQaBJ4DPuPvbE/UZKXNE7+6rgK6w64jn7tvd/ZXg+R5gHTA73KrAY3qDxUjwSIo9vplVAVcA94VdS7Izs+nAMuB+AHcfTLaQD1wKvBNmyMfJAvLMLAvIB7aFXA/AAuBFd+9z9/3Ac8CHJ/IDUibok52ZnQwsBl4Kt5KYoHvkNaADeMrdk6Iu4B+B/wkMh13ICA48aWZrzOzWsIsJzAM6ge8GXV33mVlB2EWN4gbgR2EX4e5bgb8DtgDbgR53fzLcqgBYCywzs1Izywd+F6ieyA9Q0J8AZjYN+AnwJ+6+O+x6ANx9yN3PAaqApcHXx1CZ2YeADndfE3Yto7jI3c8FLgduD7oKw5YFnAt8y90XA3uBz4db0qGC7qQrgUeSoJZi4CpgLnASUGBmHwu3KnD3dcDfAk8R67Z5Hdg/kZ+hoJ9kQR/4T4AfuPu/hV3PSMFX/WeBlSGXAnARcGXQH/4Q8EEzezDckmLcfVvwswP4KbH+1LC1Aq1x38YeJRb8yeRy4BV3bw+7EOAy4F1373T3KPBvwIUh1wSAu9/v7ue6+zJiXdAT1j8PCvpJFZz0vB9Y5+5/H3Y9B5hZmZnNCJ7nEfsPsD7cqsDdv+DuVe5+MrGv+79y99CPuMysIDiZTtA1soLY1+1QuXsb0GJmpwcvXQqEeqJ/FDeSBN02gS3A+8wsP/i/eSmx82ahM7Py4GcN8HtM8L9Z1kS+WZjM7EfAxcBMM2sFvuzu94dbFRcBvw+8GfSHA/xvd388xJoAZgHfC0ZDZAAPu3vSDGVMQhXAT2PZQBbwQ3d/ItySDvoj4AdBF8km4JMh13NQ0N+8HPh02LUAuPtLZvYo8AqxrpFXSZ6pEH5iZqVAFLjd3bsn8s1TZniliIiMTl03IiIpTkEvIpLiFPQiIilOQS8ikuIU9CIiKU5BLyKS4hT0IiIp7v8DGgNl2ymEqEoAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from sklearn.cluster import KMeans\n", "\n", "loss = []\n", "\n", "for i in range(1, 10):\n", " kmeans = KMeans(n_clusters=i, max_iter=100).fit(data)\n", " loss.append(kmeans.inertia_ / len(data) / 3)\n", "\n", "plt.title('K with loss')\n", "plt.plot(range(1, 10), loss)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "oaa4ModGKN4c" }, "source": [ "##### 例 14.2" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "colab": {}, "colab_type": "code", "id": "87a-GPzLKK62" }, "outputs": [], "source": [ "X = [[0, 2], [0, 0], [1, 0], [5, 0], [5, 2]]" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 107 }, "colab_type": "code", "id": "NoacPJOONbqE", "outputId": "70147d21-d14d-416c-ab26-80b3f23793a2" }, "outputs": [ { "data": { "text/plain": [ "array([[0, 2],\n", " [0, 0],\n", " [1, 0],\n", " [5, 0],\n", " [5, 2]])" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.asarray(X)" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "colab": {}, "colab_type": "code", "id": "w5CrPk68I9B9" }, "outputs": [], "source": [ "m = MyKmeans(2, 100)\n", "points_set, centers = m.fit(np.asarray(X))" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 53 }, "colab_type": "code", "id": "THUOLIKiKkLc", "outputId": "693794ef-67b6-4bd4-e1bb-3d05facd10f1" }, "outputs": [ { "data": { "text/plain": [ "{0: [array([0, 2]), array([0, 0]), array([1, 0])],\n", " 1: [array([5, 0]), array([5, 2])]}" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "points_set" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 53 }, "colab_type": "code", "id": "1TqaAPnnKrkn", "outputId": "7bfe9b28-4cda-4a20-ca99-112373a4313c" }, "outputs": [ { "data": { "text/plain": [ "array([[0, 0],\n", " [5, 1]])" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "centers" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "colab": {}, "colab_type": "code", "id": "7f_Jv2EFLmms" }, "outputs": [], "source": [ "kmeans = KMeans(n_clusters=2, max_iter=100).fit(np.asarray(X))" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "S5oVNBQdL-sl", "outputId": "d04fd98e-8f47-4d8b-cf79-da6a19157843" }, "outputs": [ { "data": { "text/plain": [ "array([0, 0, 0, 1, 1])" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "kmeans.labels_" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 53 }, "colab_type": "code", "id": "MG_sSMnHMU_r", "outputId": "9c4393a2-3f9e-492e-bd10-98d48ee6495c" }, "outputs": [ { "data": { "text/plain": [ "array([[0.33333333, 0.66666667],\n", " [5. , 1. ]])" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "kmeans.cluster_centers_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "本章代码来源：https://github.com/hktxt/Learn-Statistical-Learning-Method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "Clustering.ipynb", "provenance": [], "version": "0.3.2" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 1 } ================================================ FILE: 第15章 奇异值分解/15.SVD.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Ipoq5dkUZEI1" }, "source": [ "# 第15章 奇异值分解" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1.矩阵的奇异值分解是指将$m \\times n$实矩阵$A$表示为以下三个实矩阵乘积形式的运算\n", "$$A = U \\Sigma V ^ { T }$$\n", "\n", "其中$U$是$m$阶正交矩阵，$V$是$n$阶正交矩阵，$\\Sigma$是$m \\times n$矩形对角矩阵\n", "$$\\Sigma = \\operatorname { diag } ( \\sigma _ { 1 } , \\sigma _ { 2 } , \\cdots , \\sigma _ { p } ) , \\quad p = \\operatorname { min } \\{ m , n \\}$$\n", "其对角线元素非负，且满足$\\sigma _ { 1 } \\geq \\sigma _ { 2 } \\geq \\cdots \\geq \\sigma _ { p } \\geq 0$\n", "\n", "2.任意给定一个实矩阵，其奇异值分解一定存在，但并不唯一。\n", "\n", "3.奇异值分解包括紧奇异值分解和截断奇异值分解。紧奇异值分解是与原始矩阵等秩的奇异值分解，截断奇异值分解是比原始矩阵低秩的奇异值分解。\n", "\n", "4.奇异值分解有明确的几何解释。奇异值分解对应三个连续的线性变换：一个旋转变换，一个缩放变换和另一个旋转变换第一个和第三个旋转变换分别基于空间的标准正交基进行。\n", "\n", "5.设矩阵$A$的奇异值分解为$A = U \\Sigma V ^ { T }$，则有$$\\left. \\begin{array} { l } { A ^ { T } A = V ( \\Sigma ^ { T } \\Sigma ) V ^ { T } } \\\\ { A A ^ { T } = U ( \\Sigma \\Sigma ^ { T } ) U ^ { T } } \\end{array} \\right.$$\n", "\n", "即对称矩阵$A^TA$和$AA^T$的特征分解可以由矩阵$A$的奇异值分解矩阵表示。\n", "\n", "6.矩阵$A$的奇异值分解可以通过求矩阵$A^TA$的特征值和特征向量得到：$A^TA$的特征向量构成正交矩阵$V$的列；从$A^TA$的特征值$\\lambda _ { j }$的平方根得到奇异值$\\sigma _ { i } $,即$$\\sigma _ { j } = \\sqrt { \\lambda _ { j } } , \\quad j = 1,2 , \\cdots , n$$\n", "\n", "对其由大到小排列，作为对角线元素，构成对角矩阵$\\Sigma$;求正奇异值对应的左奇异向量，再求扩充的$A^T$的标准正交基，构成正交矩阵$U$的列。\n", "\n", "7.矩阵$A = [ a _ { i j } ] _ { m \\times n }$的弗罗贝尼乌斯范数定义为$$\\| A \\| _ { F } = ( \\sum _ { i = 1 } ^ { m } \\sum _ { j = 1 } ^ { n } ( a _ { i j } ) ^ { 2 } ) ^ { \\frac { 1 } { 2 } }$$在秩不超过$k$的$m \\times n$矩阵的集合中，存在矩阵$A$的弗罗贝尼乌斯范数意义下的最优近似矩阵$X$。秩为$k$的截断奇异值分解得到的矩阵$A_k$能够达到这个最优值。奇异值分解是弗罗贝尼乌斯范数意义下，也就是平方损失意义下的矩阵最优近似。\n", "\n", "8.任意一个实矩阵$A$可以由其外积展开式表示$$A = \\sigma _ { 1 } u _ { 1 } v _ { 1 } ^ { T } + \\sigma _ { 2 } u _ { 2 } v _ { 2 } ^ { T } + \\cdots + \\sigma _ { n } u _ { n } v _ { n } ^ { T }$$\n", "其中$u _ { k } v _ { k } ^ { T }$为$m \\times n$矩阵，是列向量$u _ { k }$和行向量$v _ { k } ^ { T }$的外积，$\\sigma _ { k }$为奇异值，$u _ { k } , v _ { k } ^ { T } , \\sigma _ { k }$通过矩阵$A$的奇异值分解得到。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "149N2eYfZLed" }, "source": [ "---\n", "任意一个$m$ x $n$ 矩阵，都可以表示为三个矩阵的乘积（因子分解）形式，分别是$m$阶**正交矩阵**，由**降序**排列的**非负**的对角线元素组成的$m$ x $n$ 矩形对角矩阵，和$n$阶**正交矩阵**，称为该矩阵的奇异值分解。矩阵的奇异值分解一定存在，但不唯一。 \n", "\n", "奇异值分解可以看作是矩阵数据压缩的一种方法，即用因子分解的方式近似地表示原始矩阵，这种近似是在平方损失意义下的最优近似。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "1i4xNylpaWVA" }, "source": [ "矩阵的奇异值分解是指，将一个非零的$m$ x $n$ **实矩阵**$A, A\\in R^{m\\times n}$表示为一下三个实矩阵乘积形式的运算： \n", "$A = U\\Sigma V^{T}$, \n", "其中 $U$ 是 $m$ 阶正交矩阵， $V$ 是 $n$ 阶正交矩阵，$\\Sigma$ 是由降序排列的非负的对角线元素组成的$m$ x $n$矩形对角矩阵。称为$A$ 的奇异值分解。 $U$的列向量称为左奇异向量， $V$的列向量称为右奇异向量。 \n", "\n", "奇异值分解不要求矩阵$A$ 是方阵，事实上矩阵的奇异值分解可以看作方阵的对角化的推广。 \n", "\n", "**紧奇奇异值分解**是与原始矩阵等秩的奇异值分解， **截断奇异值分解**是比原始矩阵低秩的奇异值分解。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "uAEFyqLD1Rbp" }, "source": [ "---------------------------------------------------------------------------------------------------------------------------------" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": {}, "colab_type": "code", "id": "RaH0xqPcZB94" }, "outputs": [], "source": [ "# 实现奇异值分解， 输入一个numpy矩阵，输出 U, sigma, V\n", "# https://zhuanlan.zhihu.com/p/54693391\n", "\n", "import numpy as np\n", "\n", "\n", "#基于矩阵分解的结果，复原矩阵\n", "def rebuildMatrix(U, sigma, V):\n", " a = np.dot(U, sigma)\n", " a = np.dot(a, np.transpose(V))\n", " return a\n", "\n", "\n", "#基于特征值的大小，对特征值以及特征向量进行排序。倒序排列\n", "def sortByEigenValue(Eigenvalues, EigenVectors):\n", " index = np.argsort(-1 * Eigenvalues)\n", " Eigenvalues = Eigenvalues[index]\n", " EigenVectors = EigenVectors[:, index]\n", " return Eigenvalues, EigenVectors\n", "\n", "\n", "#对一个矩阵进行奇异值分解\n", "def SVD(matrixA, NumOfLeft=None):\n", " #NumOfLeft是要保留的奇异值的个数，也就是中间那个方阵的宽度\n", " #首先求transpose(A)*A\n", " matrixAT_matrixA = np.dot(np.transpose(matrixA), matrixA)\n", " #然后求右奇异向量\n", " lambda_V, X_V = np.linalg.eig(matrixAT_matrixA)\n", " lambda_V, X_V = sortByEigenValue(lambda_V, X_V)\n", " #求奇异值\n", " sigmas = lambda_V\n", " sigmas = list(map(lambda x: np.sqrt(x)\n", " if x > 0 else 0, sigmas)) #python里很小的数有时候是负数\n", " sigmas = np.array(sigmas)\n", " sigmasMatrix = np.diag(sigmas)\n", " if NumOfLeft == None:\n", " rankOfSigmasMatrix = len(list(filter(lambda x: x > 0,\n", " sigmas))) #大于0的特征值的个数\n", " else:\n", " rankOfSigmasMatrix = NumOfLeft\n", " sigmasMatrix = sigmasMatrix[0:rankOfSigmasMatrix, :] #特征值为0的奇异值就不要了\n", "\n", " #计算右奇异向量\n", " X_U = np.zeros(\n", " (matrixA.shape[0], rankOfSigmasMatrix)) #初始化一个右奇异向量矩阵，这里直接进行裁剪\n", " for i in range(rankOfSigmasMatrix):\n", " X_U[:, i] = np.transpose(np.dot(matrixA, X_V[:, i]) / sigmas[i])\n", "\n", " #对右奇异向量和奇异值矩阵进行裁剪\n", " X_V = X_V[:, 0:NumOfLeft]\n", " sigmasMatrix = sigmasMatrix[0:rankOfSigmasMatrix, 0:rankOfSigmasMatrix]\n", " #print(rebuildMatrix(X_U, sigmasMatrix, X_V))\n", "\n", " return X_U, sigmasMatrix, X_V" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 143 }, "colab_type": "code", "id": "Hf9KqmH110KX", "outputId": "56ed2cdc-3f83-45eb-8c16-63881afee5a9" }, "outputs": [ { "data": { "text/plain": [ "array([[1, 1, 1, 2, 2],\n", " [0, 0, 0, 3, 3],\n", " [0, 0, 0, 1, 1],\n", " [1, 1, 1, 0, 0],\n", " [2, 2, 2, 0, 0],\n", " [5, 5, 5, 0, 0],\n", " [1, 1, 1, 0, 0]])" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "A = np.array([[1, 1, 1, 2, 2], [0, 0, 0, 3, 3], [0, 0, 0, 1, 1], [1, 1, 1, 0, 0],\n", " [2, 2, 2, 0, 0], [5, 5, 5, 0, 0], [1, 1, 1, 0, 0]])\n", "\n", "A" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": {}, "colab_type": "code", "id": "Tmecvggl15Gn" }, "outputs": [], "source": [ "X_U, sigmasMatrix, X_V = SVD(A, NumOfLeft=3)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 143 }, "colab_type": "code", "id": "r9TbEba32HcQ", "outputId": "abfa62a2-3ea8-419b-eb08-209afaa5cea1" }, "outputs": [ { "data": { "text/plain": [ "array([[ 1.96602638e-01, -5.12980706e-01, -6.20066911e-09],\n", " [ 3.08997616e-02, -8.04794293e-01, 1.69140901e-09],\n", " [ 1.02999205e-02, -2.68264764e-01, 5.63803005e-10],\n", " [ 1.76002797e-01, 2.35488225e-02, -7.63159275e-09],\n", " [ 3.52005594e-01, 4.70976451e-02, -1.52631855e-08],\n", " [ 8.80013984e-01, 1.17744113e-01, -3.81579637e-08],\n", " [ 1.76002797e-01, 2.35488225e-02, -7.63159275e-09]])" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_U" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 71 }, "colab_type": "code", "id": "IoVH0RA32MxA", "outputId": "4b1ca501-7ce9-4cff-b929-150084bcd2fa" }, "outputs": [ { "data": { "text/plain": [ "array([[9.81586105e+00, 0.00000000e+00, 0.00000000e+00],\n", " [0.00000000e+00, 5.25821946e+00, 0.00000000e+00],\n", " [0.00000000e+00, 0.00000000e+00, 1.16381789e-07]])" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sigmasMatrix" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 107 }, "colab_type": "code", "id": "42ag3hPE2OBa", "outputId": "8fa214d1-3ec3-456e-8698-72c7e5ea5d8c" }, "outputs": [ { "data": { "text/plain": [ "array([[ 5.75872999e-01, 4.12749590e-02, 8.16496581e-01],\n", " [ 5.75872999e-01, 4.12749590e-02, -4.08248290e-01],\n", " [ 5.75872999e-01, 4.12749590e-02, -4.08248290e-01],\n", " [ 5.05512944e-02, -7.05297502e-01, 3.28082013e-17],\n", " [ 5.05512944e-02, -7.05297502e-01, 3.28082013e-17]])" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_V" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 143 }, "colab_type": "code", "id": "1RHUFh0w2O0K", "outputId": "e6947501-4932-4c53-dee3-25dd38f542de" }, "outputs": [ { "data": { "text/plain": [ "array([[ 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n", " 2.00000000e+00, 2.00000000e+00],\n", " [ 5.39915464e-17, 7.72260438e-16, -7.54662738e-16,\n", " 3.00000000e+00, 3.00000000e+00],\n", " [ 9.57429619e-18, 2.48997260e-16, -2.59977132e-16,\n", " 1.00000000e+00, 1.00000000e+00],\n", " [ 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n", " 1.25546281e-17, 1.25546281e-17],\n", " [ 2.00000000e+00, 2.00000000e+00, 2.00000000e+00,\n", " 2.51092563e-17, 2.51092563e-17],\n", " [ 5.00000000e+00, 5.00000000e+00, 5.00000000e+00,\n", " 9.74347659e-18, 9.74347659e-18],\n", " [ 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n", " 1.38777878e-17, 1.38777878e-17]])" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# rebuild from U, sigma, V\n", "\n", "rebuildMatrix(X_U, sigmasMatrix, X_V)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "c7FtRwkh2WlI" }, "source": [ "same as A." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 321 }, "colab_type": "code", "id": "r_5WIyV33P1H", "outputId": "ee629ad1-caca-4f8b-8b8b-4ddb7df88824" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAATAAAAEwCAIAAAC/mbqkAAB9j0lEQVR4nOz9d7Bs6XUfiv3W+r4dOp18zs15EiZhAAwSiUAADBBA0RIpPUtPwZYsyfYzZbns57L/cNnlVy5XvVe2n0uusvxkvWdRpPRIigkMIEgEAiBAxAEm55mb80ndp8MO37eW//h29+lz7p25M8ML4szc/auuPn067N579/fbK69FqooaNWrsDfCPewdq1KixjZqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQakLWqLGHUBOyRo09hJqQNWrsIdSErFFjD6EmZI0aewg1IWvU2EOoCVmjxh5CTcgaNfYQ7I97B2q8Ofg3+X6jNzxFt2dPavwoUBPybYa/rEpTs3Fvoybk2wwEeZOfmKJwzcY9j5qQ73TUJHxboSbk2w032oSvj5qQbyvUhHy7oSbYOxo1Id92qCNV72TUhHyb4c1rrLucQDWf9zRqQu5piIiqEhERAVBVT8wEBUTAvK3AegHt1GaJQADDQ3n8DKHiZ03LPQpSfbPX3Bp/FZj8LrSTZ5mHNTveKYqicGlyk2urAhHc1DZJVcOWjYlu+z7X+MujJuQehXOOmZkrURZ+JiKaZOp48SIS3kOgwXDAzMaY6U8BiLY/QeNNEQCinbSuXqudRj9m1ITcoxCRiabqvd9WXKkMf0RFVQMbb/y4QifCMICIiCZv5qn77c8ANSF/zKgJuaehus0rZiYiuHUwYyIDnRPnRMTGMUQqGlsLUwlARayqIhMZGySqIaKakHsQtVNn70JVA8EqKgYYD1/oMHPOee+dc0FOrm5tiQgAa22apkmSWGuJiDsHiYiMAmZy8Q3ydueX/ZUeWo3XQk3IvYsgGyeKa1mWZVk2dTPb2hoMBt5vF34wc0Tk1XvvvcOo6I8A7733PtlvoyiK4ziO48gmAAAC1V7WPYpaZd27EJGJ28Y5NxgMhsOhu/o955wxptVqxXFcFMVgMCiKoixLAMaYOI6TJDHGiIj3/kI202g0ZmdnZ2ZmWq1OHMXYNiOnaDlZBbXK+mNFTcgfN4aABYx4I0N4D8OwMaCZa1gLP0D/EvzV0ebL19dfURrO80PMdmOz/+Krr545f/Xa+ub11fXV9S4RlWWZRPHi/Ny+pbn9y8t3nTpx8vgJiS+TFsPNa74oDx+8p3HiQ2icAhaQYENFTUkoDcommlYSZEBzeucEO/IKxhyeXjI1gW8rakL+uJEB7EHiI5MDHqzwEdAAYdiD5ti8cOHsU9b2o6a/fOXs2rn4qSef/t4Pfnjpyips3J5daM/MN5pta21ZlkWe58NBv9t1xbDTai8vzn/gI/c9eN9dhw7Mjrqbo1Exv3TqwPH30tJd3nZckjqIwBuIASIXM2inEVMT8q8aNSF/3CgdghM0ThwggGqRwjNK5P2N089dOf9KmkKQf++H3/3an3/1ye9e6HTah48eu++Bh++67/4Dh44t7dvfnplL0wa8lGUx6m9du3Lp+eeeefKHPzjz6unTr7508MD8p37qgx/76Hvazag7GDZn9y0eum/5xMOIl0qk/VHRaiQGgOSGGRpNcUym7rEjWFKruD8a1IT8ccOPwBFgC4UXtYYiKoGB37h47ewLa9cvzHZam5ub/+E3fuer3/jW7Nziux989NEPfOBDP/GRA4eP2UYLaRMUoXRghipEYQ1IoTK8fOn8+fNf/+LXv/LlP37l5affde/BT3zyvQ89cjJqmo1udve7PrSy/922c7eUkQAmBeBK6cc0B2AnJ29GSNRhkh8JakL+mFG6fmRThXUODBgW+C3XPXfu5R8Me1fareTihcv/9ld/67EnXn33+z768U/+tU/99MeWllcas3NQAgySJsBFWcZRHDZYjIYQH6cxiAGBsxefe+43f+Pf//7nft3rtZ/9uUc++tH7Z9rxxlqxsvzQg+/5H6B9Coy8VNMUhyzVFoA3JCRr/AhQE/LHjAyFhSUwiTAJJCuuvHzxzJPkNkX63/rWt37tv//DtS38zf/kH/2tv/PP9h04NXOgAQ1xCwaMgktRYjMaFXFiDTHEMcgYgnpflJxl1JpBQX/xtS//yr/7l9/59h89+sjMP/77n2kYO9qyR4/+xOFHPovGCtTmEZSR3kTuTXNymoo1LW8/akL+mJFVKeBq4eAGxeqZa+ee27x6em4m+tKX/vTXf/NLlPA/+Cf/+U///N9b2ncPN1JX9lSJjDE2caKFaGS3rT6vIEFkKq7kRZ5GmZaRapMtXnr6hX/5X//vnvjO5x65r/0//jufTcgWeXN+30PH3/9ZzBzJEHmD1k3SWeWGBzUtf1Soz+aPGR7wHgYKDKR//uqFpwYbZ1uJfvlPvvD7v/uldrvxz//F/+Hv/v1/unL8bkpTAWzciNKGjdLSi1dKoogJWeELp05gCNZAgVK0FE3ipPBDiqkkWV0v737w3v/i//Kvf/Zn/8H3vtX/+he/l7JJ7eD8me+MVp8BbUmRq7y+TThZLTJ1q3E7URPyxwwCxIFQIl+7dP7pC2efWFqIv/H1r/zqv/tKszn3T//Z//4zf+ufUGu5GJUwcIxJ1WNsTWINAQQ0Y5NaihkMhGcMU8QEILKtEmJibs1Hwwzz+1d++Zf/i1/47N/6zV9/9Tt/8UIjac7PxV/9s99+6QdfbjYk7w2mU9LHyUA8lRVUL5gfLerz+2OGVbQSDHtrq5df6a2f3b+v9e3vfO3zf/KVlf3tX/zb/+xnPvOfonFAywQ2EpSMHAoooAKAICaYklM3Ht8IQhBB0ysLSmNd5rsqmDt6/H/1v/mvfuInf+o3fus75y/lWa6RlWef+jqKS4tzpaIULYL0I9KxGKx11L8i1Cf3x40cRmF0eP7si1m2udVb/5Vf/f3eAJ/8a7/06b/x93jlLpSW4jRKGh65YLT9QRUoAKHXuAEARGG9s4IC6Ldn4GkEQuPYif/sf/1fOnvs3/361ws/02h0WIbPPfbH0POhM4GXQuFDSck4E523EwN2pd3VuH2oT+uPGZEOkK+uXnzFDbuNKP79z33h9Fl89JOf+uwv/cPm4XtB6JfQCAKUZQ6UOz8tlcCcfmaXgaeIowgQAx+hsJGHV+Q4+ZEP/ONf/j9++8mr33vykjGdfXMzZ5//TnHu+4rSsBKpSAmooiqDnvqKmpY/QtQn9McAnQIl/Wzj9AtPfnu+1X7p2TN/+icvf+BDd33i03/7wP3vAyVCaMyiVOTlKIlstEMnneLDNid3vcohvJkgVVUCK0RtiRkgwmf/3j/82V/8B7/5+1976ZVr+aBcSuNzT39vMOgBnkkUXl9vkgjXhPxRoD6hPx5sVx7T9Wcf/+rSbDLY3Pq1X/mt2dn0Qx/+a+/70E/DtkaqZYSSUCiMtaZKM91NucqNs2v741sUgQGotZgDZuAiYVKLjSxPVuw/+hf/vODGH//JN31GS43Z048/de78mVLy0E9LQ7ePGn+FqAn5V43pwK+qFucf31x7dWW+87U/++qrr4w+9clfeuTRT9r5o6LWRVQAHmATKvw5GxQ32yJDt/mpYAXL+AaMRHItCN6ijIgaSjzEKJrBiNyph+/5H/1P/9mZM1fOvnx5tJb7rnv++ac3N9cBURWQACCe0oknRK/xo0FNyB8DAifD/ZlXnj5xdN+LLzz7lS/92cMPnXzXfY889BM/jQLDQoyFAh5QQCAKarRmd2yIpm5hywCmgoMKCLYMgy3ggRxQEBtBKcj76MLo3/n7f29x5cBXv/zN7urWweWD586d6/U2ccOFY3vr099U43ajJuTtRuCQA3xgRq5Yd7i+ml/OgX4O54iFuegbPtO98HtyXTq8+OUvf/dKX+77+KeW3/sIGiZPXKMhDbgZlDMoWyhjMCEGkhtJOI0qCDm+WSDSFSBRA03g23AWCjQw00J7tkiiYri0b+4f/y//Zz9c7/7Oy2uj+3/uY83/4J78D+iT+sUttAaAkJInFIDA02BIgyHBUTjM8iY7UeMvgZqQPwLchC2cJCkAIlgGFIgN8uz6lStRlDzzzDNPP/PMfffdd+zYiQcffBBg72VqW/jLllQo6CYCjQHYuIFSPvCBD73rXQ889+zzF69cjaPOxsbG6tmXkxiqUCiBpwo7mOs186NEfXJvN2jXPYeOADFSoOplI06Bsr929drViyaKv/6Nb1+/PnrgoUceeviRuNEWxVTcb7Ktt1pmsTNZfLJfopLEKcAqeuju+z7z87/w6tkLzz3/UhLNbXV7L7/8OAEpwcAyCKpgjDMELE+2Vft8bjdqQt5mKGQHjyoV0hIsAaQgArOHzy5fPMPwm93BDx9/6tCRleMnTt1zz33qPYgjm+yM9f0lip6ocsOQ7uS3MsDZKKekCfBnPvsLi8sHHn/imaJoRlG0sXamv3ExAiIwwYIFVII8wIRoe8s1I283akLedniFTEkPrky58F/4Y32+frm3cXVpYeaJJ59ZXZf7H3jPqVP32Jn50bAkGIIZf3YMfau/lG7fTzgJgBkKFlGA/cgtHj31s3/tF55+7uUL5wfNZpux+fwz3zBwVgEHsAOPQAXDbB8ZSc3H246akLcZAhGIbvtdGIiASPxY2xMHGq2vX1YpSctvfe/xmdn46PFT99z3INQ2O7MAb6uZIaQxYeNb8G3eUN84+clV0WzNAoZtCpi//gt/Uzl54vGL4jVKhqdf/G6+eR0OmgfxGPKEmLd11ZqQtx81IW8zFAIo4Ka4wwS2JgIQWYBylFsbqxdj1leef/H02Usn7rr/2Mm72wePwgtAIrihqXi16bcEmQ4eBu8OA6pV3YgvQXEKMfc9+J4P/uTHnnjy3Np6L+I861+8+PJzEJAw4IAcENpWfQXQmpC3HTUhbz8UIgiq3jYY8IUADijz3vWt7joTnn36mayQo8fuOnrsbpAVp0Uh7rWJ91YTZ3ZULQZOOgcAzkNAUFbipNn5+E996tUL5fpGn8mlNl+9cnY8fme79JE1XC+kjkT+KFAT8raDw3oNQxxVMM54gbUs5RCUra1dVimKYf797z65uO/QwvL+oyfvhgMnDWZrTcWW24abhU6iCAhBDGYQFNzvDz/9mV9YPnDwW999otlst9P43IvPYDhApFA/2QQpjCIkHfh6/dxu1Cf0tiOc0nG9BYEUUKgD1HPEYLe2eiWO44sXr66tYmnlwKHDx227A2NAHPRIuo2q4M3YuCMsSSG8yI1We2Zu/qEP/OSLp6+cP3elGacrC43188+DCoCASKd6tlKls9a4zagJeZtBMAyufB7QsPSpstkUKNxw8/r1y8z83LMvj3pY3nfo6KlTSFoQBjFAqtuzrUBTbTLeGkt3RQvH3qaKp1MvGUPtdvPRj3z06jqefupMI25n/c0Xn/8uaASwoKFIpjYkANcNPG47akLeZjAmMxil4iQAILJgVsBfuXoRpIN+/uzTryQptebmlvcdBLF4B2UYeN0lISUEGN6qOJoqktolJwECmOB9aNUMAMfuv6e9MPPsM5eLES5efOX8hSe1dxlsVZuCeLyRSqWuJeRtR03I2wxSQG1wZN7QA0qA4vr1y7NzcxcvXDl75srC/KH2zFxnbh4gAWP37FUJrTr+Uu2kXiP3dWpkOpgh4rx3IrL/+KGlAyfOnu9dvtjrtOLR6PKVa68CRjQef6IEtk3KGrcXNSFvN3YUKElotEHB8apOUTpfENGzzz7f6/tOa2VpeSXtdECwcaRT26g+HuIMFbflLZDST+2RTjUTUPWqftI7xxgKY2AXDu2bXTjQ7dJLz547cfxwnORbw+uA9ToeeE4F1I1LvWrcZtSEvN3YXqQhUleRiChIodJaPnPmzONPPJPEjThuLy8vI21CBMYoVACz+zf5y1pq0w0bJ1RnBkGwzUliIgA2ifYdPKrSeuHFc1v9bqvDohmUApsBgDzI19rqjwg1IW83dtXvjvsOEwGkZVky8wsvvHD69JVWsxNFabPdAoCyBCAit2pb/abJqdPydtcD2m6TreJEHNSPJDt85Lg1rWtXN55+8olG04C8E6kiHQDgbtbFvMbtQT1B+Za4DhiEWsSQkjpe4JUJpbvU1OBUTYBku17YAx4mMw1q06X81W88uZBjsaknZ5v7jr/LjdQ2Z+DYKsEAHiI5xTxhjoLkrf5SVib7yuMQ/zaRiG11QAwIvGDGrBw6cUD2l6/2+9+6IIfx7uvX3H3miq3OgAVikABbBqWB5DgwORU8Tpcdp51PbOBx9p+iytKt8RqoL3K3RCj0NZOweIVJfK9yVo5vVRuqEpQrOzUQhoZPJwrOX7n0yvm1C30go2z24EJ7ZsbE0XalVUhts3YqK+f2/ka32lpZHl3ZR0WRMl+9dPnyuUvttIPMQ2NoAm0ATaAJtIA20Jk0gzXj07B9nQo28K7TVeN1UUvIWyIec7Kq091eVFSF/iv5E4QAlYBTlAJRWCBVY0EQXyQpXFZEc8nf+Z/8vSe/9+Ta5a2FY/vmFxcojiohZSa54AbkKrEFYPrRW8CtyBBYo1Sx6P4Txx+9/8HzZ1+IRNtxI+EU1ICkoJaAZLvIMyHAoty+IG1frWQ7cSB0W1au2fhGUBPylkiqBTddALW94ATVuvNCDMBU3TscoACH8J4yiqJIIvyH//hb/7f/8v9674lDTdNaPHIkNyZptgGCkk6FLxkI2a+YUoTfoqB8wzSohFmezywvP/rww4utqNWOL5y93Om8dP8jH0crBkgIolBgsrcGRbV3yiAeC/ZdIR8GSfDK1qx8fdSEvCWinf9OU7Fio1SBOVHAaARiAgzEwzCMAipope0fPP7N/8f//f81P7v0/e9dXL+Kv/HXZo6feFfYqBNv2AZhq75atvoaVR+3HzTF+zgB7OZW/8tf++oD99/7vcee/YmPXfr03/xP09aO92/HNdWDeNxBiMcVZ9MI0ZHJSILaSno91GfnFqi8OJO4IgSq4zb+1VMeCJ0XPRjewiWQBJoYJFbYOkQOZQ9nXzxXbrn//J//b+86svjQ3Z2f+vCn9s8fEhWAiUh1h3tId/40la16e4INfMODKWPYWBj83C/90vWBP3zv3Z/+hU9+47vf/vJXvwxyIDVABMSKGOEmgIHStsTc8S2sMOMaLR33/6rxeqgJeSvodBRDoJPQYtUKINwURmAUphKfaqAG3lZ5b4I0RUOoSdwivOv4yRP79j90911H3/1uEQGEDPsgRwjKk8jhbfh1All0d+bdTklV9fhQUskgW3l593veQx1rZ9r3vvfh1R4GxSC4qYg8kRAcYQT0gB6QAAkoMNQCVmEUUbivbO8Q9NmeOFLjNVGrrG8YtG00QhlqQg4407YErdz9FNpbKMhB7SQk/+m/9Znrl/7R+rWzP/no/dcvr99zfAXlJtFK2FwpKsQwmJ6jIaCxq0T+8lfPm5lwMt7v8LrmirTZGQ3zX/y7f/eu++/7zjf//OS9iz/z6Z8DhRiGBwjkgBLIoaKYE9re/o4TNjaGCTxOf60JeQvUE5RvAdVJbYQDfOUODWPgqlQylAAxMifWMhcgFrYlwTMIiCAWArg+4gFo8wu/+q98Nmhwc3nm0EOf+aXMzsZxkzkGrAKDkTQak5Q0IUj4Gtr24r7J/b/hmckGRIQZ3jlmptBEAOiLxkal2NravPoHv/MbcWQ+9vFPHT1xD0wTmghZAAwPlEAJuFIXlJDlIJY44umv4EnxZBX/LEEOaN2wRzW2Uaust8DYchuLx9DZibkqcRRAECmsom05BSSGt1yAM+gIyKG+ihFEUA9XHLv7VC8bbgy2evkQltI0Lkdb6nKoz7Oi1eAbMlb/Uo7JGwcdB4qqaij6N9ZSyIRXUe/aRLFzqY0vvfhy2e3/J3/9F4+efBc0hUtVIwE5UAE7ksbQNQdF6kow0EyQREyAQJxiOPLhm6pLi2I7MaDG66Im5BvAzh4W41w4kAEEWgAlMCow6rmta7miAEqwR+RhFJGrGuwk4GZZ4OCxe3Kfdke0toXy/GbZ30haCVkFxNA4mfxHgF00p+0SL4H4kGJDhrC5jmEfw+GV51946MSpeG4J1zagEWBJIQ6lR6FQBqw1cbPBoMyhzCXv5sPrBllCrt0wjHFW/HbKfL3Ybo3ahrwVtq3D0CynshvDI6r8/AJTgEc2LlMKttN2yZMUcAV6g42FlTRqrmBjY3bu0NmXLg67m7/3h19tHOCf/9t/F+J8mUWNOe+ciexttLSm7cbdk5CZxTtmUhEKVwLv0O8+9s2vb/VWV8+fmTl5fO0HP1h85AMYDZE2AJjgVfYAg0MSnPaMKWHERi6tgh9xNjJpOqPTxmllUNacvAVqQt4KY0IqIGA/fs4T4nFiNrTfP/PDJBqN+qu+9V4bSZxQ0rBIWjAzoARxlMZN+EF25cznfvM3pJTe2sZ7Hr7/2JF7v/ns57/+R7/3sU/8nEhkGt4VZRK1GZDdmuZbX8q3DMerelIC8YUL577wr/+bu08dm51rnzs9Go02/uhzv/7355t87D4MLyPqmLhpKBR8FCiHcDmGl7G1iZTEZZk605gxyXw6cxTqaLK6aDu9tcbroybkraDjzDh4PzaLFGALCnE1v4nNV7//rd/32ZWN9Yup7FPjjNUosWmjkzaWm8mSjdoLy4tR4krXW4h6H/roh189uvLehx62xx9Ilrp/8oUvf+yzfz2iFC5PkhiQsig4joLCV8XxVEkrVfmtH8QUytJH1kABUmYGtBj2n3766bvf+9DHf/rjmxdfffalxz7+Mz/57W987bvf/pMPHZ4HM4ZX/NXexsba5sbl3ublbLBelMNWb9Ad9iim1cFGBmrOrew//vBHf+Z/iGYEnam+nTD2xHKdW/76qAl5K1SGEAtxyMUJYssL4H1kHJIC0agcXoppox13D5TXvBTFKMu2ytybvm86aammJbTw/eZMtNlbP9vpZVu48KI7NLw025pRKXpnT88cPIXSIWGo9d5bmCn5ePs1veBd996zMapKhr33Fy9eeN977yst5h645+8d+icw5eLB9gvPPPn93/1vNtc3fJGX2aDMe0a2Ys7SyEXsIz3QkSI2qW2MChtvji5fPY3B6gdbR5eABhApTTKZWOFrQr4+akK+CShYxtLGMFgBKaA5ZOjdVqNdNqJoaf00p0pWYdhrknuX5UXhUSq2Rn2K1LYHrzzzFabZcy89E9v5jdn7IP6LX/ziQ4/0Dh2/R0ZlZzmNkx/V71KZkQprLQB13sOrems5jmPn3Go+euXqxdNfe3z1wgu9q2dmYq9Z3w2vtdOoYXhpHp0obUZICCx9FlcMzOag9D5vGb+40I6G1HNFK0YIioAijNMSag/rG0Edh7wFCoFhGAwAhaSAVXYlSgc2SKyD4ev585/74Rf/9SINk8wPeFVVRQQAkamadwhFUSQCdV5ERcR79d577wvXKByKAqqNtLUwu3R4Yf/R5vzy/vseRmcBs0tI5kAtoIGq6dsqiGCiUIOiMB7sQQ7ewAoE8AJH8AoRSCpzRsFSFV1sM0MUVCDyoAyj1fXTzzz31PfPnH6xcf53rbVpGjebaaOZJkkUxcRMxhhVL9uoZKztD8qocMYWxUyqs+2GWS2zRz/zPzcP/sPczOVAjCIFw1kwwDl2tK6rsRu1hLwFdkzVoEkx7zjApgBQulygIhJYpqrbLaQUqiQezjnVMB2ciIhDlg9z1NDUUV4iy0aD0aXuudXTF54Vk2Zf+Z255QMHjt2zcuSuuX3H2vMHbHsR6QyoDWXkCqcCImONhWXEsEIQcFkyUxRFldHouNpPCwWXVYBeSnA5vPDyS8/+4Pzpp7fWL/jsuhY9dfnRw0eYYSxZa4whYiVV9VK4AlVPAxURKIdjJCKqesmSiop4eJdlWYtu4iquqz1uiZqQt8aYkzKOpPFY76swzDIiEhWv6qUipCoRKUQDIb33ALOC2TDzOKFcs/K6MTZtpEkjbkvknC81cyg98aC7/vR3Hh9+gyiZWVo5cvDwiaXlA0sHPzMzM5csLSOd4VB+XwIl1MNEMIyIAFspjC5D3uknlmKrwAj99fVLZy6ffmH96vlXXnxKyx4V3dQMF1LMz0knQUTiqYOqX6uIhlZ0XuEnY9jH4823FasJIRnw3otIv99ridZxx7eAmpC3wPiK7iY8FAAwBBateDkcDmEgAs+iXoMYUQ0dBljEqwTTQEVUFUGhVSVVhYGK89onjVjjmEzMRpkKV3bayaG5uFQaZL3hxpNnrzz+itchf3N2fm5l/+H9B44u7zu8tHLYLh1Aa45DGrfz8AqxMAYktoFZu7p25dxTzz5+/uVnNy+fLQdrXG6x5AeXZ+Nm0bC+aSjF0PoBjfpwyKJjICEiCjouqVZzBEISr05VPILGXXlEhJlVPUiJ0R/09okfZx3uKI6p8fqoCXkLMAEobxJUB1SqRPBRNiAD74UNoQzLdLrEEBPxAlVAAlcDbyML8RCBL0oRTwJmZqZOI0E5FNe3Sg213E6ihVYUxWeLMyLYuvLY6jmUYpK00+wsNdsLKyuHZ2YXl5aWW7PzsIkOh1euXLl+/foTz37bFSMtBjGyxYRmOiZNbGSMNSXUwRdUZs6PCJJwGqfazUaBjWyIGQSqpj4HPgIKUZkcnTrxCvVeYkMAWGGZsuEQ4qAwN1Zl1Xhd1IS8FQhUlf9je6JFWGdVzoCMRkMYKImSEBnVqrGiTk2rEpGxDbmt4wHIhmzIEmxElpiYjGWw0cHGprEaG7IGBKNlocNBSWZ2oc/WoG29mKJEVl4rB2f6fbt2TpUigoVaY4y1sYiUZXlXc4ZTHzV8bCg2nk2h6r3X4agktmSsalLC+LLMfUkKk2jQV0lJRUCVSNQpqa4yxTJlACIORoIstRFn+RAyGRgUaiYn5neN10NNyFuAgKlq952VVg6wgBRFNgTEq7ux3//Y6AqeDyUwKgutssIMEtIIMOJYnKj3pEJwSZQSHKmQV8CRDztQRgWo8Mo+MmialNOUTYNNKrCGY1AkAiI2xpCy9z7tdq1BRELIXDnKRsO8LApQ0ppxooUT8aSqVhPDiSUW6Y0LPoOjuFJNAYyvL6GRlRAZAMysjMpiViWoJcqyEdRTPT/yzaMm5K2gOwr1J7KBQ24AAHGuHMXQEOCg6sXtXv3hsQnx98qYDEQVEQETVFRC7TOIDYiZjXABQnjaEJiFGUzWjIyIKOAp8myJVagUSBQ1jAWR904BMFtDrKpS9pg8k4PP1I9AGqeUxo1BseXEqFgIszIp4OABb3IAREpEbMDEzEREznkCwqz14FllMgAMGeFSCNsuLDbeZZP5HwyZdNMTUJ0Y8PqoCXlLTHJKt3kZ4h42AXyBQZd8no22ji8vXjp9oQEmUmYO0TqVyuLy3mNcsDvxVRKRsdrr9lWRxK3MiTURoIPu1tLyTF6MjNVGwk7VOSUVQ0WcLxjAsJGQ300MMLNhilVIlQhEEMB5QFV7VpM4MgovDtDIwEBdPiwdiGDZJEm6dmUw3zFWsdX1MhtbywCJOggbwwCcc8aY0LoECN2cTWil4AFXuEajId5bG6sfiXfiHbzzrjAmBhREMPC+bsp6a9SEfCOoAo9ClZ5JAJOD8xhuwqrLB51mcvnipSSOUWQ7Pqo8cUiiCmSKBveOkqqud/Olxdn+Vr7RG8zOLDZas9bao3fdc/rMy1uZNhuUFzTKpNOJW63W+vrGCpiIIDQ2x5RYNFRphnGU1XVDwutl0sxdxurSpJE0ktINs9KLgBmGo2trJfuBZCDvU8ahQzNne0NVQ1QZh2Hy+aRWi3jKgCQhGPUKU2nmpAoSUlKU8DlDAKcIriww356mJO9s1IS8FZSrjjCT2TIEhgNykCBSrF2lMksbdpCPZjtLeV4geFMrc/PmpVRaFQhrmrRWVwfOR8vLp/YfOLq1lfcGfd3iR97/qW5v7YUXn+5tdVf2HciybPVSf2XlkO/2CKRERCBmmNAiQ9VWrVuVhFSoEpYqxXzETIiL0aDwBcMYE4OizUGxtp61mumpk3cPu1tGddDtfeeZjaUDzKzMhoMurRokMAAiTCUDhCuTEPHYwgyXLAExiUdZGBPmR5fQCGPLucbroybkraAALODD1T1IJkKJYgtk4fvnnvkh+9wNthZmOuqCI4SCk4OIMH483toOw1JVve/kuT116oE0nXvmqdPPPv9SFKdZWUTxE5/+7M8cO/q+x5/4/rWrbm5uqZnMDHoUkw+ZPmAQCxkDViJVVq3KEZVJQErMAJL+sNNsqtBg5NWbtNEqy2hzUGY6874PPTq3ePD8+atPnnni0MGDy4fnMT/wvT/zSgbMxCCScCniSs1WAEyhp0jVpI4MDAeGEqkxbKBQybNhYm0OAALWsSFZk/IWqAl5KyiglakmmPTXKREruhuXHv/epTMvGpQuH3UayUavb8cMnBrAuP3MlDCp9ECRtNlsDYd4+aVXhiO9775HZ+cXX375xSzvf+VPv3HfvSfuOfHAq6+8uHph9cCBfeK9UiKk4yHNhsBgaFUTZoGqGZcCogRgMWKfDdT5WZvapFF42+0V3a4++OgHr13T/+7f/8es1FMn737m1UH/iSs/+7M/K4O/ICJVRjV5NriIjaobHwtCJINYAWVmZsukYToWG7CBqu9vdZNp7aBy6tTJc7dATcg3AsZ2f9GAHMh7l04/9q2v7p91aWxjxHmWGXk9vwWNBSWP80tV0R9u3Xvv/f1e+dSzLyRRQ3Fp//6D+1cW7rnnvaNss93g0XCzZR1ZiWXQ7XUbcwlAIAIZYjPuhxoS16QSy2HEjaiqRqnLugNVakZJfzhcXe3Dzh47duKHjz3+/adeWevhl//Ff/ahn/25r/3B73/ud3/nlReeuWcxVVVRZRAbE3gtUCUGBU1biQikGmxLH64sVWoSg0i9kl9bu74IUJCMqIKRNW6JmpC3QmjQRNONTAVwbv3qc0//sN9bi+bjhmGUmmdZI25lviTCRJ5gWn0FoEqkAE2UwNkFHuXXz1y6tLqO+9/FrlQv60x86cJTBw7Mz7TSgyudQ0snn3ryh+vXunedWNrIhIiUmYxXBjETcRWIIBtiFKqqHgIRlS4GZWpY0UeZoYg7yfzS8qEjh9SYB999//ziSpKa3/1X/+cXnnu66YvVl7/28MGDRVF4Kav2jcFwVJ02IKfOgxGoqooGp7JT9UERv3b9ykmXw0ZadXX/q/7d3qaoCfmGMGmeRlVgTZ568vEXnntq//xMv3u1Meuyrb5RtdbAlwAADrzAVMInqtW8Q2ejOPfcb7SkcDh3YdBpY3aGL158vtUyo0G81rbLC61Txw9+/NMf3jhz+vnnLlC7oyE4yIaMEnMw4ZRtMCxDP0eRkKPne/A2NVr6iHDg1JF9S4fUx5vdXrtNZXH96R88Pexvba6tzhkwwXUliuKyLKGsleNocvQ3RVWLpagYqaphLu2w1yuKgixXIUidGJ41Xg81IW8BF8GE9hneORuPBFTIYmLT1TP32pfibN3GZM0CRQs2Gg4HVxmpqnIVBiClaZdrKFmiiqdgIkTlIrL28sLhUfHtzRzRzMFXr0WLs8elyMuNnEx07tmz51f7hw4v7DuwcNdPdl56oRSXJamJuMyzzTilOE76WZamM0IRGauwZVbmZR6Bm2mjwIZRf/jwysHlpdHm5rkXv762VmY5toZotOzmhhv0Qdq5dHlr/TqW5mNnBkgKE6thKEPEGSJm430Z2kZyUIyDi4ZgTER5M9GyTNZzm1/lBXIr80yrT/9KdJyiD/0vVI9uNeGa2sEgggLtH/dPuqdRE/IWyD1CXydVMUDMiNMYzhNpZKw6NcY655TJl37s83hNTLt2QAplQ6WXbGZu6ec+ffK5Fy7lxZZ6Tqyfa0XNdrK+vhoZTuLWiy+eu3jpwvs/8J677p69fOniaNBLG7Y9My/iVU27Pec5cl5drgyNorgTRXClK/Pl+WOtZjMGv/T01UvnzvtSrEmd08jE16+7RvuAczLKOG1ajjeOnLwfuMzMEK0ydHZ6oVR1pzFINxnqAQAYjvq9Xm9RHFXtnUlfT9LWqFAT8haIxquPmXMPx5Ayj/rdbDAUEV+WTU7yfGRAos6QhuHfY1QujV1a65QZBtZ8lGfdLu/fP/P5L7y6NcxOnaR9+w43LF584ZlRD+97X+f6lY1RhmYjeey7zx85ebwzExtujPoDIgJHo75kLhsUwzht2jhi8s1YOs0oii2obCUHs83RhcvXLp9ZG/Wo1Vxyni9dXb+0uvXg+x7s9vTJZ19yHhvrxcxM+u6f+AjRbxli5UBIBXj6GnOjyq3qVT0mKbsKhmFQNhiuXb+2OBhgBpjY315r187roybkLeDH8+iMQQR4QSuOzFxns7tu86EWGZkZ54o0psJXHZx2dUWZYmOVkD1NyLm5RDcHpevd/8C9v/iLG+cvbXTXB499/5n3P3Lg0IG57766+erLW/fc1V6Yb/U3s15/MCyfOXbsUJw0BgPKRygFamfa8ys/85FPIm30N68/99wPL555ttct9i235juN1UurF05f3FoftKLZpfkjo4Fcvbhx9kKJpPnn33hhIOXC/uXnn7leFPjnf/+T6VKCLlUzSxhEzFWdiq+y49XjZh5TVakSkIjHh4y1tesYbqHzo/pp3pGoCXkLiAIkWuQUWQjFxhr4rXOvjoa9hitiY4wxLAyoqhdXKuJdGwjNAab/DY/Ck0W+YYyurm+cO/f0Pfccuu9dd8dR49K5S+vXru9f2Le19jgE62v9ubk5G3cigjWj3uaVRmtxZm5+GNnL14uF+aPvef9P4fiDWFxoU/H++aXI4Prl51dXV9evFWU/7g8GrVZ7aXZ22NONjY1BNurMzvokXlleOH/9wvrg+tH78PN/42N/429+/Ctf+dJBUmO21dRQdixBgwWCeAyKKwBTSciQKl+VgzCRgTZS29/qohhNBCoBoHq93QL1CboFKnOPBWVmOQGYZPTis09bSGSokTSY2Vrr/IAV3nuy2/1mUMmKmw+uCu/Z6g0OHNzXnOls9jaFMBrp4vxKs+lnjy/lW4PjR2fyUXH5cqbSO3XyXvFxZ+ZCmiRRZBTsyZi0PfLm5bOrB3HeNq4N+2vnTj+1dm0jMslcZzmN5OqFc7MEK3lv6+y1q8P+CI12I03S6731lcNLD330/UuHZu9/z6nN3uXPf+n/3ZlpUI8qQ1cV6jHVMgc70xsAeFVGaPARcnNBwkTESu1mIxv1pcw4hEVZ62LIN4KakLeAiBJ7SAlrjORlWRgdNWJi8gzEsXXOkeUyKxPLWgR3qsc4ORvbnNwWkkH2BEJ2Wjzo90YORe7jRtpppoS+LwYAtdtpHMvGanby2OxGd/jqq6cPHz2pgiRO47TZ3SqGBTrzy7lGP3j66evDgqKYZOSysjWz2OQU5fpgq9uatV3nNtfK/kbZ7yNK0OwUiPnEA0eTOXr3+/bnuvXM019QDFvN9fnZObOlRBAVJw4ATw/HCxlwu84PxmnyWhVks4KhaWy7g776IhQnTyzxH9kP9Q5BTchbIGbyPtNB17YbF1858/SLLyw0o9HF01acuMxQnJUFm7j0vhkRLIqJUyfMk9TKpgrrNryiqhi3jUpsc2uU2ThdmG+PipHnkqi01pOX9bXVA/tnDPlBf5QXZRSZ69evntyfbg3KhpRsm2mj0Vxs2+Y+O+ujVgQiQtJozCVquTBZNhwMuxTZ3sANMniCiWEYzaZvzXFn1jfn9OLpH2bSdxgmKeat6V+52KB908niYX8BIeLQT+eGEmyhSV2aMiuHqhbD6spMpQDEEFfNWeu+V7dCTcgKu9JQJqnUIG8YL77w9MbV8y+fOX/p2rX7Th2xW9dm2cexFXFRmpg4brbS3uq12QSh4f+2X4dkUq8UnJYTlW9sQ2psGs5Tnpcggrgy77OCvaYpinIriUlbtjEq1rvZgZmWaCtJmkWJqBURkbK7cPnV9YF/6MiHLl25nsZRqxG/9OTpI0udTntl49qmYKk/vDDKUQ6RGnRaiCNemG9HkSLvk6JhiSjFyDF0jubLMg/7TwTV0CwPRCTjKXnTHikAWg3rU1XSkIFPYGhkyJBfX7u2os5SLKHhgN40RFJjGzUhbwEphlu9y9/4yhcWO8nPf+oTTz77wovP/KBZXhv2Tx+dx+zsfLfINrvdRkqNRlLmw5soZSRTkkF2hQ2gsaqCiLwGbQ8qgELKMHWD2NvIxCmbgTjN4+TQ1euri8tLnZmFjSvXnn7sO5tD9/TLm57l/R/4cHej93u/9zvF5qbcc+ruIweOHbnviZfPlb7IstUkkqX5Vis1hgspRpw2WJRKwwIiYyRiBRHlpjclygHckkSiNMVVZVICCUOZvPgh4IAYkB/RmL13GGpC3gKp0ce++xdFf/Wjn/j07D1HP3p0376W//of/fuFlrEorCFjTDHcSmIbR1ExwrgX8nahVehtsZOTmFhTIqqEMBVSFSSqpFAn6qwSiNiINVGjGZleXvi8u+UPHDjV7DQvXLy62evGlk+dPLrW637pT7/251/7GqtJbfThdz+ydf3ac8++cveJo/e860Nr1y8///j3iq3rDmxiM9fpNFtGNCMPA8NqSY3xxGBWzprVfut4zE+Q5rtOi1QDiEL6eMibNwBYQ/W2RAYMn2cDqNs+2lplvRVqQt4CG9cvrF89/+df+nz37NO//Mu/3Dp11z0f++A9n3zw+m/+f66efXIw2IqarTRNS5exc9aGTOqA7SKsGziJ6fhH1UBg3CySJkwItfqGI46ilDjKBT5uLTZmFi9fudQfFI109tzFVwpvH333+9a7oyvXNpbmlxY6s340crlrzM/2+0OH1WYr7SzM9PwmR5o0OGmSky02AlZwBJRQW41ehyHxUz3lxjtMk3as23OsJodhxoFJVq76X4raiBk66m9By/EHTO3UuSVqQt4CZX9z2F0r+4NLrzz3jS/+4cmz99597104ubgw21xl+DI33CZrfC4iEhmIVDbkdBnka2xbAAicEPxYlR0bb2TIEghsmYhMRAwbIU4aHKWFEJnG/Q/cBXA+onOXrloaHj1yMtI2hLJ+fvX8+QfvPnHs4P61a5evbF46dezw8VMHz2NDZcBxDOvLLIsTQ0aV1SsRnKI0bBQmRC8AaJhQPomacqW8Trp5hD86blBCxESVR4vUR4ZBfjDsQj2p/1HNhX7HoSbkLbCxdgXFoB0jYYx6q9//1pUnvvu19z94FL0LRl2jMTN0rixLy6wciq6AHdk5U5wkuYnOxjlARNYHXY9IQVAwWxE1ZNlGxDGgSdKcX1yZWVh87LHHzrx8tt1sbGyMlhYXvTfn11efe/LSzNw8kXqXv/rC1Stnr3YaaDdBS4udph7et7g13+mv9q2NiRE12sRe2EvVKNYZEoVjmKoHpHLwRhHtCKLuEo86PtjJwZByCGJaS0w6Gg6g40Sf0HyrxuuiJuQt8Kef/4O/+MJvt4BTR/ddOvsKbLS8Mv+1L3/xoROLsy0Tx1E3L1WVI6OFyk6XzQ6V9TXkJJEnYqAkMgAh1E7CGLbee6LIWuMRKbTRNPv3H94cdL//g7MRYWG22Uro0rm1K5fggaTFw55ubnb7fRw6gONHZ4f97vkLOPfc2sWza7/08x9ZmF+2pWs2Z/Oix6YJI0JO4RQCeMdiAMMCH/bT062yTsfHI0Iw49BO9ZKqMQZAlg2hfuJlfsu/wp2DmpC3wPzszM9/5tPZpRfQv2oIwyK7evHCTOwbaRxHVTVgHMdMRelc/LrcA4KQ3Bk2MKIqBFv1m+QxJ4lVRYgNJyIR4NKkMTe/+PQLTy0s48j+o3k/P37s1NJc6YuzonF/kF291G21eWVJjhxebLWiA/vbc52Ldg1GkCSp8XZTrXe2KOzsXCfzQ4gTLgVlKMFWqJBGU7vKymHowY22384jFCKLcQpr1TqAmQjjIIqneqW9MdxxpykPrZkAU2mUAlQ9MbYG/WarDRVX+CROSOnFJ5/673/jDz98tHVo+NL8sFiZw5URzg/g9rcHrplaRTmKZUBKToRMs5DCqsd2HJIn/ORJ1QRXEc4QnyQ0CMpCsVSp6URCpE4LahgHePWlt9qWNGmOfJdW2wvoNP1aszE4edxfv9w9f7q0PJpvp0QtG3lfZgvR2qnFhWLQ9YqldstGevnyhRMnluK5sojXoigv0E8MM6BeVU3VJocEvuqyjnBStu/DwwkN/eTpJI8L+JHNIlErpZj+MLE9as5QMptuoPcMrnw1PfnXe2gIkMBzXe7xurgTvV5842ETAKStJgAmtpZdPoK4L335T4db61IOF+dnZ2bQ60OBOEWcNEKXKSWoUhhuRTcEzV8L04kBN75ERGAyxoQ3mACEtm4kvmTI0tJCmkBcSaxpDGNgTeim45kRGRCUWFspbGTEuSg2ZVlOmomEB9MGnY4xvVdCN+zY5N/XVAKqrTKzZSMiyPIqRR9gW2utt8AdR8ibkQBQLr0wbFE6gJmISa9fuvCnf/L5lY6dbdDyfGduxuQlECFzGBV5CIDruBwQABA68t9qB6aX9cRlGfoHBFU1ILKhVssYY61lZpASq2XxZbG0MJ9ERkpnQM0UTIiYCE7EGUYUsaoa4k7HWGtFXLvd9qWrwijGMPPk66b3bbsH8tQlY/ptr8HJsSd2kt6E0BLSiEje70MkzAMB33Hr7c3iDj9BoeE3g8BkAZRlUDg9R9F3vv3n1y6dv//Y4sH5RkQZyCct2CQd5IAxIKNMApbQZwfgKingDYG2EVY5ExGxJbZkmJmNiSbvmLSuEudnZ1q+LEnFMsQ7wxRFgIANkXj1ygZxZBiiKq1mk1Uiy7PtFqDGGDaw1gaGM7NONs4kNAlgEKqwx25hPr1cNLR6rv7ZLnDRKr9BrWX1bqvf3eau1m7WW+COIyRPhOSUyhZ6xSgQx7GKMBPg/uTzn5vrxAdnMJ/ko95qf4RGBwUTpzh8/IQQFKzhHgid8s1ru3OmPT1TbKQwpmaae8wMtkpgZjJMRMwwhlR9no9WVpaSlAeDLVUl4sjGcVxlFYwnwSKKjBKJKBvrXTbTahnLXkrLUFUwAiED85UIhidAEPQ8vVM75CRNNfXAWNet7OOQSa8gUfE+SMhBrwciU/WKdajxurjjCEm7ODluSREu95E1YU2++uQPnnzsWwdWOovxsEUj+KIsUQDXu6OlAweOnTopUIUXgjJJECCihLey4EKz8UAMGAbb0CTSRFEUVXIyiiIDykfD2fmZpZWVLMvIRKKUNJpp0yrBew9mIYBh04aNEy8ovVrGyr6lfDQs8yyUZYQOqhXlpqhoTJhht1sw7tJsJxb4OP9+ewmNJ58rABEXW6Pitra2ppyydXrALXDHEXLqgKtpcqHKyGvoQwqIRzH8w8/9Jvv+vvl0X+pt2YsIMOgW6Ja4+6FHZpeWqoAdEYHDkDWGsLwhlYzo5vInyEgwkbECjuM4juPwbBxZNsjzXAiHjx5Ra9narBAbNxvNGQeUqmpZDIRh09TEDQeblTo70zp6aP+gv1nmI2OMtbxNtjACwDAZZkvBebRLEhKF8pWb6K58owarTFUGoBqQZeN9Oej3IJ7Han2N18edeIaCOAMwfcEOCaRF5lEUOux+66tfOLDYWGrzbCzFcMsLHGOkRM3WPQ8+nKTNIGaAyvqrWvu/AT7u8qNge60bhJbHqIoKTRRHUUSshmAtM9S7Yphn8ytLNkqEomFeKsUcNwsPB9LIioEacBxz3HAUjzzPd9r7V5aLbOR9aQ1FUUW6ykvMRscjIMFEhkON1PTtpvu8EztWUdA52JCxBC9FPgqaKlNtQ94adxwhp3yDCohAPFQAawCPxDAi/vzv/tbmtXPH93cWW1L21sjDCXKg63jl2N2HHv3Q1bXVNI1UCpVCVUrvvFNDzNOFlDsR1MIdXsqgMxoT/KgmssYYGKtMCoridJRlRemjMYfixKZpfP7SxW5v69CxY82ZuWEu671h2p6JG7Ea60CO0OjEnvjS6lpWSmdu+f533XP2zMviy0aaGMNlWXrvKewJmbATE5aoahRF1tpdopLG0VS9ARiXejBo2vD0ZRFbkyZRr7dZbm0w4Iq8bmB+S9xxiQE3gwCGAMOQLOei+8Nvf32hgVmbL6RlPNSRRz/HADRE65GHP4DFleFwaEkZ/o1kZ+4i4dQiJgDVqAym8ZC50PRNyDBNqbIMYQVDnLj17gbbpNFoHTh8hLR55fJoc6uwCYpRkeVotgsHnVtY0vbC3Nzy2vXLRT6wLIaUFKQSeKOqYIKvyjNCCupNajl3ykYKFuLkhlChvH0JYgAQHktCIkC8uNIAgGK6GKbGzXBHErIawSTjcmFDgJQwDGZ++rFvv/j09+cbWO5wE1tlJl5RKEamZWf3PfCBj4CTXq+/z4jRAuJp7CR6vYy5cVoZpj0l4yKrQDslYrIgE/pBMbOy4XEmLDMTeUNot5L+8PrmRn95uXPk4CHLswvzZmX55LX1XqGZUn9xMTl8YCkhKoeOwVevXEzTuNmyHNqQM2DYT5LcmRisFC4LXFErjB+Z2vPqwQ2tjisZubP+Yzx7Tkh9bNmXRTYatCDEWkvIW+LOI+T0WqsIKYCp2lCxfuNLf5JosZjockv94MpWH87C22iE1qG73j1/7F0Y5lmWRR214h1KRkRkPVRucCHuFi+79NWwH4ahxMwaDMgqKqga2vAwsRIzG9bIMilZKrQcdYddKTbzzmyS5u3WgbtOHXvg4RmY0klvOLzW27h2db1XbOXqTDYcpAlZExO8whsTmClkDECspOFLwFBHQJV1NO6Ts9ttM6W1YjzCloSqsMskryCMjlUfR6Z0+WAwaIekA60l5C1w5xFyChMCscIYANh49dVnn/jBXMMsdqKWyTVbd4rC282M80bnXY9+HLP7hmefZIYlMSi8dyGf5kZsC5ZtHu4mJBEJCdE4QM8GxDBMkMnehTcykzGASMNKmdKwdMNulm10Ra9BLyp3hNkmQlHm3CZc3ooai52V2c7iy6dXk8gwg4gMVXas9y5YiUpgtQSnYcqkbueyVmrtFAlpSgtgneLnZBCrKMw2mUkRRZEf5IOtHkOJFK+rRNTAHUrIbZU1iDVFKMD1+PM/+5IWQz/cWjgYWz9QdYiT4RCbI1o4ceKehx4Fp93eMLLWEhn1rCU4pqoBx2tKSJoqAQlPMldBlvBAiYkNsa18rawgp9s+SSFihiqjFcOxR6TKMODIJlt9f+HSaRMnwnkh3loszkFENrNLRa/fSGNrjSEl0vFEvNCYAACYiIlADCEFEQhM0/l/Ctw4roTHuquqUpgpLdu0pOo9oiSxNWVZDoZbCk+kkFpC3gJ3JCFvBCmEhqvr3/7WX+xfWrx07Wk3ymOK8gKjArlwc3bpgx/56XTfUVA0ygoRiQisAvHEIVMMfkpl3aWdju+3+blDQmI7WwfMZBgQYgYIqiIIoqka91ZsjXprEGkkSKyZm1tYnEnLzG2NMsRkHObmcNfxZZS0caWHIqNGFWoNPHO+IJiQABS+VcN8SWYRmqjS27luO2ewTx7c9DEqJ7ZMXjLGii+LUaZhVlYtIG+FOy7scTMIAJe5yxcvXL54aWFuDopeVxqRyQbo9nKlaP+Bww/85McQN1H6rHRlWRIriaexUaQAlIV2b3pXPB03lk3sTgyo/jUmmoRJxmwUEUkMIoOZFi3PNxuJjVkTaxPDs610vtNsxkgMkohJCl9mEYmUzvlCVUP2j3NOREL18PT37t4xvkk2OSqReBNW7dIOJp9iZhEpy5LGIvUN/Bx3NO44CakMB4hwQg3KRzZuZIz1LXcwvv7Sn/6fHk6+sdJ8ZNRqXY/n1zdNc/1snrZeKpY//Kl/MTr4oWwk89H5fcXz6+7ljNP19txI4UsX58Nm6ayzcG2JS5rUN5EPUghsVCGh9NjwNgsI4NbYSmQYIXZEBkSFKGCAiAxC6MAZV0rpMNAkXR9g+ejdPl+T5jBNsqS7nm9itr3P5b35mUajaUb5MFk0W9lQHUcci/PFUOJmHJmYoKRMECXxQdMkghrAgEAsEIVKqBFliJAnkaoDkHqBgxdRCW3M2ahQJEgcbKQU+zzJ1zPTzuPOVZqxlvalF9tn/oCHHyjt3f1k/9xO1b3GLtyJElIVlgERRBEIpcdix25du/jk49/fv7JQjgbNCNevXkjT1KS4tjEQ4vsfflgE1jLYXLtyNUkShKid6I0VHtOejwlulEU34o3s/HhKsYjI5IPWUmhqrIpJpkEcx1FkwTu2P23WMjORITIhl/wmX0Y3iVLwuBf76+/wpPYSQFmW8KUxppaPt8SdSEgWNQpxCraOMBi4GPLY17+4funCvsX5hMojy20d4PTZM6tbWM9w4l33zx0/7gk2AiDDfj/iiJWr0JpqcIoKQejmi3iyOneBdzzA+EF4cxgpHuIHCpGJV5OIQho3c5UAZCMmgogwk7WWTVVFaYyZFB5X9iEQ5DKzpXFxyTS1KnfoG4ESTXWyxM6ICCrOA9B8OECWjedn1Xg93HEqqwBMCihZUwIl0Gpayte+/+XPnTwwi3KYQI/MRH4fzp3LD80gWZr/1C/8IsgaBhTobSaRzUsHz0xMXo1ULhAllaAJ0o0DFIXITDixg5PQ0A8Z2NEFisYjenQ7dqcM9YAxBihFtnNujDHMPshMay0q3rKIOOe8VxNoosGlSuNiSA4NxZXCsQX7cNsRWoUX9ebhiinHD1V7V4En1wBDbI0ZjgYY9u/Ii/+bxp14kpgE+YgYA48CSCxefuzrW5eev+vwohtscN5L3ebJfe3FZdZ2cuTd75//wE8Os1wAUodsCFcaAQlByCgbBamARAlCY4GmYUibkoRyQWB8rqnqzjYlRhSAEEJPxhDu8+Pe+0ISxsJJyC41lqMoYmYRH8oXRYSZvUdRFN4Lm8AiCVz1vvIGYVtNDZeFsDvbNRusDExe3Rb11W6HQ6jCjxzGs+5yt46L26rj8r4ESRSZfDR0W5t1ms4bwR1HSFEAXv0IQAEUQAQ88eefX4yyqOimxreNS10/9cPlAwfXpXX/hz4Ok1KzIaFLYmw21leTKIWAHZMoeTUigAiJ33U6aTsAMB0MGMsWBRQkIOGgmsKTelWBClQVnlTDRkyVQAdmDkWSOu5/EwRjnmM4HJbl+CuIbBTUUehYJDJzKEamMAxPJnvCvD2lC6hSeDDZ4Wk3b3hcTaDbZiMB2yQNm3LOkWgaR2WR9XtdwN9x+tibxx1HSFWFCjErwVgQcPaVZ19+/BuHZ5nKXkS+YaVDsCLdYdHcf+qu930EjhWmEETwGGyOtrqNJAEswCzEQbhBFPC07dEZp6uMxYIoIJPS3krLVYX4sWboIV5VSasyX5LwZCCzhCbERMRckWFi/hljRFCWCGN7MOYkAJvEcWxtHHFk2ZpJWlFQKkWgWnUeqexWVtD2YBwOWRNj3TUk1o2ZR6H886aBEGb2zqn6RhJByqzfhbq64dwtcUcSEgLLZakEWODCi09sXX5p30w0305GA6d5ERNaTXutO3zoo5+2h07lnHigdDmQX7t0OjLKXPlwWKsZGcGps0tChmW9HatU1YpsmBZQQUENWuCEk6QeUKiEl4gDFben2Y21UAXEGDImpKmHIeQVY7wvo8jYOIoiY22VQBf2qboEjLkFVIP0SHdVqG2ft6B+Q2iHsrqthzLAIRgbdtL7EipxZKHeFRmkvONW25vHHXeKYsPIc5AxhooSCfAHv/5v7jm8kFLeW+/PzbEXEEHYpvP7H/qJn4ZPTDQDoBEZII80lzIPmpuGYY8hTUxV4B3IMgxpWOjbOp66iWBhSDAvx54PIa5EnyEFCcSrehFRcSKi3nnv1IuqV++C+7TVaooIkeZ5HljKjE4nmZ1NmOGcAyS0tIkSG0UmimwUmyg2xpJyiJ1U/WNDqdSEhJMQSXChTnZ751lkgFkRJthNeYtJCSokgsFg0Gq1smxI4o3Ixvp1lFlUm5G3wh1HSIJD0gizuhuEiy8+g97l+SaLL9ii1xMbY+jQ93z8vve2D54UbuZCuUiDFBhtXD/nyr6QeAqtdJhVGEoQmcoMC2WBk6GIrNhBQqlMRwAiPjheVJyIExFRp15IZWx5ho0oBW/o9pqu/DTVNzLYVMTSqcCDMcxMwiLwGqQuJhvZsSlVIaiIF/Eigp1GY+WdmhygYmcTjxs63RIgjgBrKInZ50PwW+w5dEfhjiPkOLXcKnTWlN/83K/Mcq8Vw4sWAg8UijJGTxoPfvin0F4kahKMBRIU0NHm2qU0YWZ4hgQdr6o2EuzqLFypppWOKiIKP5E2we5S+LDu1XsRryLwTsSJOlGnQXFVrVy6pAa7E/Em/1qLquHqdn6pBNcORRQ0W4GIuKCpbnOShII/CVBVH64O3o8fSPVgCjfvdqk0TdGQnRccUbE1ve461PlydFt/yncg7jy/l7q88Gyb5LzrnnvpO3/80EKkrufY9nM3P283Nxy1F6L42LH73wdKoIgZFkw6wmgj27reaseORMB+XD8BCIfEcd7htqjsPfHKhgRCwfgEQgGyqBBCUrowkTeopg+TqoNaqIrS2KhjVggqL8r2d5AEuWUtmBm6zZzqdavMVTBGxBGrwikMVQPWoSAVIYXoxNOrMtmOiKpQGCsrCB6x4OgZC9ubF3CES4OBkgq07G6sIRvZdp2rcwvceRJSvY0aBbO1/IOv/lHUPbOvIfC+NLFjwCZ9h64kdz/6CV446DzgIQViAOQx2MgG65BM2TnynqGAhPghSZWbploVCWJsg+1sRRPmaQQ9EH7ir1QNKqVoyBH1vgwGZJBOEKfwCo9p5xBNiCfjxgPBxTp+GzwxQKrsFV7UOQ2KsQuisjJWNQQrHcTRdIBRtnd+EvYYS34lIp4KPE6Bxp8SIlJxhjQfDbW3/hqlozW2cecRcjSAMZ4ByDf++HeOtZ0drRpLJSyl0Xp/ILEdoPXAh38atkWWCXBZUPdK2bgqbuh8bmPjCdtdAki2IwdUdT/FDjZ6jKPqQVJth/XGxBiv+mBG+snbdPxm7736HarjBCKikzgKgB2XgEpBVS1VvYgLVqKqiHhRr+K8lF5K9aX3fry17S1jnFowEbyV1j0l7W6scwm7GrzEURQZxoXz5+DLH+VP+07AnaeyEjxQAo99//urF15876EERRe2lXmYKMkHJaLG/lMPpIfvRtIOzEotoNI798rpHz5m4eKEBRCokNk1jmbnPNaptTvVK0DHJmUlUUOWHJGoEI1HclFw4qiqkjIgrNDK3pzYiF6EFGP+hJSd8fdiLOlEnFdnlARqIIH0xFrtatVJFRAlIYw9wGGbpBCRIByDa2lbFZ4cdaUKhP3mcKxV80iFqjI4ttY6euaZp+zBd+8/tnS7fsl3JO48CZlEo0yc4s///OuzzXihFaUM732pXAiljaYnvvehdwNGKcoLoABHgHfPPPH4d7/zTZAkSZzlo2BJvZZJNOnENskE2PWGsZa4DZKJWuhERKeS06Ze2hEy2Skkd9cNA+MRA5VjaZyevmMLXrcD/SqvjbBNnjrAG7BjLVUtQsJJYFbV0y+/cu7s2bfyk91JeAdIyBIgBXvwxL1ggOAHHXfmZXjxIAUbe3UWC+nLevlLX3/oLj3jV1c6aK4lrXJWW6PNuBvNHjn50F8HH88EZYy1GLOAHeRf/7Vf/cC93EbPWvLcNKVlFxkYZwuvKh5cthsyLybTifqoOkmnIycAGGPBOHYHqalGnRKp2V3WxKHpvypEACFWFdfwTiP2rVR8udZoWl8qS0sLTxSVMowS57XPzK6IDBaaCbMnhomshRcDjSgiUXF+HO0Ym3xgAJEvpmiopAqhakAyBaFKAGjsvsqjAupYDIvxkgo1ScVKocWqJ1Cjva6NYdoou/7uOdn4418b/eRPJ7YBJXFsLaufOMLc2D8UuvjZ8cD0H8GS2cN4BxAygDF2/I1LjEIJgkABJtgo9FcUiQzKb/z5HzXTjJ1LGJbgkDsdOue6pTz8oQ9gZh683XHCC+xoqOpVt7sGa8gyVblxydBUBx1lQzdrJKPjhJspmSbTWW9jEUdQBEJWaWvTVU47+2uMnwtycuz/nZRNClXRDtKQdo5tQRq+1wE8bUDuOr2vgZszxhhT+kJVx/lDcL5YX+tmWdZoN/z48Cdtfupy5YB3ACFpwsbJqjEY91tSUgmXc/ahNJkW3dqlr3/t3y7NbiUqicDCFJxJYrYEhZm//9GfQ2MGk1SV4NDJR3FiCYWpvJY3aGyhenFnJsqYVzz1PIfWVRPKBcKoBpJsz0tVJcCPs4CgSuIDf7Z1y+rSINXE2Iniur1TquIcqWFScRiHUKSKhE57gCprWIJfZ5vbN3YluQWqNHpjjLqgtBNIjIW4/PrqpTLLta0AESuNM9jHhGTc8ak87wBC2skCrPqdAQSI92wIxhKTwnjAC0qPNma//63f6W18+74T7Y5wlHVK5J68NvLhsLF87FEceS84gRbq48jAAYZBWs7PdZjXiMw4hwbj1cNQhCYdxOOyQFUar3hlIql4qKE1MTDpKKcg3e7zvZ1hE3LyoGPPjowfjBPyRIRZQ/HhJFgoAlWAeCLeKYjXkKLDwewVYhUfcgPGcx1VQxtIPzYXVceOH/B2uZbewE+dJAOITjmvwlWjSjxQiQzYekjmXAGIYXOTTdV4BxByOluNJk+RiAibCMoKFFUTZBgGevyVP/7NI/sGTdOdoTmfpU5yStFTV6St+9//WaRHIKxxCVGLBEDEoBj7VhYS6rJClRkG6hUyrh6eiGcBh/HmpMoTWhKRKo/b0lQ+yLCzUmmqgQbVcwAH7RVjzbTSWlVUxXsf5Fh1uEreiypV3lAJ2e7b3bGCAA7CNdR8hfhK9f2q45Q4USEaD/BSIUBfo+vstAY7PTZaQsqckhTObwdFyYE0iXVxuU1VEsV4KIFWfuY3/8u/M/G297JOcsDCAkYoJhRYNlAGoXTqJTgswMDL3/72+RcfO7Kikc8Sn3OhqqoWGyVaB04ee+RTQBs2Bqllb0NiDARwc/Mta5RBJKRSfTNBoAy1UENEmLT+D5wLgZDtbnRcCbQpLycwEX2TKgoKyueNrk5Vdc5570W0ugooVYl3IuKr4UETaCgf8aLiVJx6Hx5ANNzUy47H6gUsYK8hW4+rAZjgsbDDrlvYh0qW0uTXgPfeWquh4sSA4OLIHzywMHYmKYVeruP47VTB9BTuvMSed4KE3BaMKttqJBlUBg1ZCwZKxXCYfe2Pf/3AvO1wHjmQHxhKyfIAkAadeOij2He3EgrAwllKKvnHHtlWGkPFs7WEijqkUpk/asYr0m9rYkw6GS+gLNCqwCoModwuWZ4s5WA3Vk+qCjQouqyCsauFxswExutYxLlSKjtTxhX947Ws3gMIKqtSSHBVHavHIX8VKmGCuaqCTbDoKoNWJ5rHtu93+vyT7EjWURrXcQHWRKVzohynFjqAFkuLC0U2gJRgBhlWhDxaUQke3Klw7VtZDO8AvBMkpE7/pxrig+p9uABbC0MwQNHf3Lh87tkf/tHDp/Zj4GJAy4JsH6kfCNK5uw/f+xMwnDFG3uvYUWQBSJHnXaMl1BkQw6oQh1lOJFBSWCI73h1gWwcbD7SqXqvMrbG2yarEldysqplYOTyGB0TgBV7GD4CpfhxA1YknlDtBubpEVHteqZrqZXxDSJVTj5C8Gvw7pKjGJockBGWCmdyH73ktNmLHszqlr4DIGBOFZIY4Moa896NGg4p85MsyJFZgHEkRHZ+uXdu98zTZtz0hAwio2DhG6TUoU+LUQAjlbKz//r/9l0dXBrOJyIAiAUfISahp1gY49cAnV068zzMGDnGcACbPCwgMAazD/obIsCyyNG6oU8uRiFOUAIgMkYVaZVJ2FOivPMna0aqvFaPyxRoiU9GGSELLjsqQ47EHNai74RlSCZJQgmoaGB7HKQCCMSYajfKy9N6JKgaDgbU28DbPc++D44ecc2VZBhEqImXpnRPvNXhVAwMnvl+iMGSAmatBkcyW2W63AiEKRxquINWvMI5oOoH3ymxVKTK2LIvIkrih0XJzfdUkEUJCn/eBcca8BvPuPDn5DiEkgB0ec+I4ibYGhSrE55ACOvjhN7944fnv33VUelfPLbQOxHZmJMgj5BFFM4fvfuBn0DjoGCFlHLCWLDg4H0T8SHwGKUhVxYiAWEHBHGJoLJXyrxPjarJkAYRG4EQ8MbPGVtO4BaNOi6DtEqqJ6TjxrIqI9z64e4JUFIF4HX8qhC6r3AMiEh96C1Tl/ACFZIPtLL/tx2F/eKc8nNx2YqrH3PiU72QPGRVDsAAbCqPuHFGh3gNS1YvskIg3fsXr/9zvTLwTCFmJR+i4LolDOmXajEFIU4N8E8PVL/3ev1tKRgvNLc577NLSN0YC38DFrezEgx+Zu/cnwSRAFIcEA2PIAM4QAFeWA/FDqAtKmXolFR7H3KCm6uPIPkg/IjP2eSAIzKnZ4ETEUknLqRrCqTCAqqqE+kVUrprARg/vvXMuNFwlMqpwpXivqNxC45Q95tDyw6tWaq9S6I0XiqOVUOnINL5V7Tt4cguG5s7b+NWpgefbl5XwDUxEBmrEE1EEtaRkidWXkFLLAipj634SdA3ndepacLP2tncC3gmEBLBLPALojybZOg6JvvAXf3L91cffc2IBg+GBpbnBYDBy8I2kaJoN13z/J/4mojkvECAFDGDUgiNoaQlQccXAuxEgzMw69oSFqn9lIgPdtiHHRuO4L3hovggT/EzjDJUgLYkUDGKYyhsyzqWBBusuzOkKjtAq3dw5H0WRCo3HdXjnQtgD3m17WQMnXYmqIcHEztw2NXnsI6Wp8mIei0pzs8e7EQ5wYj2OM26IyIhnpojEQNkQiyvVu7LMx3u5PVBzOs9iesrkHYi3PSEZYEyuuAgj1TwhakABUck2V+EHX/vj37r7QKstG9ahFZONtCCVdGbDzZx4+BPN+x6FwAmi0OHYh0geqRsQAJ+XxdBLwRBDzGQNDDEAT4QxG0NARLBtUBFRaNJvJi1nMPV4+8kAHcfoK8/nlOtyKkoiIt4jjmMAzCwezjnnqlQe7+G9F8HkQuAcvFPvdKzN8jjlwNyUh5M9nz6K1zn/k6ZYk8IzAAomRCLEFKsSKRuyofOAdwW8A0kIF4f3T9KB7lCxOIW3PSHH1+aqRU24rAYhUSogLp1tfuf3fuPa2edP7WtF+VrHYuP6equDwhQ9iQe68sFP/B2gDaNxUi1JLSoDhowwAFe4YiQuI1VLNnR0Yp22mqqm+tXSDQ4SVPKwUuGq99FY4EzW+pTk2VG9hbE0o2D4BT1WVZ1DFEVBZRWRPC+LPBDVOwfnxjVfYa6GRynqFE5CgTPtDDOyEELsMdzeoGDchWnSEgXVgMWDKYIagI0xvnQqTl1QwSfpQdg+bzsO/pbf+c7E256QNBGPYyhBAQcwgZldd+2LX/iDpZkk1eFs5CNtuBKl9jQqtxwdv/cnFk59EIVR44GRgbBHHJJqSBAxKeCKPB+VZR4K9seW0+QbiWAmZ3I6ISz4LccwY0MrqKxm4tQZK7ETg0p1qtXidAlUuJUljDEhWcc5VxRFnofgh3gP76uBPGFr3iMkwe7KQ6h8OZVH5yas2ykYX5OW1cHp9r/jB0YEzDZoxcxWxg34IC6UYE5U1ZuL4DuSk297QoacDwAThsg4W2BrKEz47d/8DbhiZWGWUZKMpOSVxbg/yExKZNL3f+CnYeaQNp0UHoXzBWM8HLEoK3nrnPOF+HKyyseiTFCVU41PY4jvv7aKt1OS7Frx4y2HsVpT2TnVfeXdQQj1h1Qe77UsK6k4yWXVKZusqqVWCv4bUUy7ZEIru+pBcOS83m6//mqRafU7VI0BUzszbl0HACHlQqf3tAbwDsjUgVNwAsGoRNSAYFRgLUbDqm2luv7Ct174xn97/4Hy+Iw13QJZ3I83NUky2ndxtLz/wZ9bfM9nwbN5joQbEAVyUA/GgQQWUAwIiQq6V+3WpZVFZMWFHEugWZNby6marVE0cElGLk5cnGZpaTIA45nfoJCxDSixhn5tU10wAKhZF0TqY5Gm00jEKjzICw8Nl8bnKmo81CNx8B5FiZkG1F8zkfF+1svc2kZRigM3gbI/Ki+dx7GDy0xeBakO+kOoJkBMlpnVRsZYwHjyqqQwAJMw+SozAKnenHXjHd4hzIHQtAdB1OvkJSqJJDLsym6cxF4KKOUFRhk3iwyDHhZSFCOks36cakUAhyuhTtrhhVfeATLjTeBtf7QhvdsL0hgA+qOSkBgYyYdA/hu/8m8OLi+201Z3YzNKQLYogX6Bwtu5pYPv/4mPixMPeEIZJCNFoASUgFJQ6qnBwDhZFNAwb3xSq1G1GH6D49sCdvlIlKnKVCEhUoInVYKDOFXvBLmDJ9iIC8F6D6MSonAeo9z3R8MsHxZF1u+74WjLkO+0EcfIXV6WpYjzULJQdgpHLEpSSp6XoeRie6dDrAM3yZS59f7fEtNCUHWSI1e3u7o53vYS0lorZW7iRBX9QTnTmjHAoLva7tjvf/7X1y8+/+6TszbvznU6G+sX59uJzxuFNEY+ue++dx+4/z2liwWgGE5CopwBG1DkQB7kQ7anF+dEPMAGbMSoF6fkzHamWHVde/3LGxFtW01TRczg8I8wPHEoR/QwEO+JEKWA56wkZ01jRhaW20vLsysH54ej7tr61uZaHlvMzcI7OF9kGbzH2tpqs5k0k6YaLRXFyDs/bLNJ4wikxnIcR1ol0zKUg1nLQkK3IOWNKvfrvJsmNWCTN08VfN5aGNyRYY+3PSEBKYrNNG6XQgxLCj9Eu7mAjRf+9Lf/zYNH52cod0XZmE1HFhLl5A6KmM7M/ofe/QGAojju5RonIVAOqYqSeZKXaQHv1ZXwAoEhtuIhVFIg7GR81HTuy2uDxlOrJvdgCg4kYiEtQ+NFYm8II6fEiGzUH5S9Pjrt1pGTxw6eOtofrm1lW3HaPHnXvvyQTeOrZ09f31jdGgxBBp15rBxaXllaaKTtpd5ofvXq5uZmPvIciYlVxImSVOXUYIUSSIPtywyAbt5ndZdgfINycvK2EBT1IUk4CMnx+NZJecAdScDdePsTsijSdgPIyEStVrPsY7RWzO/jP/q1f91yazNGG15M2li7dnFxf7M3GOZl6jV+5N0fjA4eR+kQu0jVIlaFBQjV+KjtciBBGPSoAkXsidWQc96yA1xokUzCAPh19b0dInHqyTDZgxREXkgNqUJBouqZ4QXDQoYOppnOHTi6fPTUc2fOOT8ADVqpXr22Xgw1Teff/fAD3//O94ncvgPYf3j20JGDAEZeCqLZhZlB1u/2vc8UlEcGIPLq7FRIgxQMhrIAr9X4eHqf3+AvEySkTs02rxJxp91gAKZLdnZFIpXvNJq+/QkZx0C+Neo2G0uiiCOk++IX/uyPnv7WVz7y0MG4WLXOpUmU53Zza1Nj9F1nZt/hIx/4JGwLJoX4Rhp5FUZoEKkAQcmqYaJx0YdyqBKE8WQ9QY2olAoHCBDMyonf9TURVNYd4hFQWEJV4G/IgyhcAIrSRTZWoVGmUbMzv3RkYd/hMmnN7j+xsXFx9dr6KC+b1qLkJPJF1k8TE8fu6NGDSRvX19d6WwNRKzAr8y02FpwXDrGHtWzYoordAwCH7uoKAOZWNuRNvMSv/ZFwZdOpN4sI2IDCpK6QKDdNuOmzt6sc607B25+QMM4hTecUZnNzfXlmAaPiD3/7/3tsfzrfEAPnJPNl2Z6dubC52Yxt3j51zwd+FvvuhybglqhVWAfPrLsMKFIwYNSxSgRijgErCo8wMspDPUFIx0HIqfLc18K0nKSqi05cJeaQMqAQUgiRMZGHJYrjNNrK6cyF66evDJK0Nb98cG5u/4HEkN90/a4ryjhOUZgyy9MGtVutkeuOspFn22h1wLGxcWt2ed6bfDQgqCiXHuzVpIzgjiIAYMjr7/pbU1mBHVkKIqIUVNbd6v04jDv5gjtRPOIdQEjxVJYmts3M5UvzMzpc++1/+183kvV3nVzqrz5j8/W52ZlefzQcmgOHHzh/7fJo8eF9P/mLiNpQzilWpkwQswnamqHxTFWM/+iQKIsIBqQwDkYIylLNOQZIiWBCO+M3mBK9Q32VJMxLhnrAU5gXopTE6cbWyFtrkvaV85e+/p18rYvlFXTmnzpyeDY1g+UFPrg4M9eYkVLOvPQygxpJs9VqjbpbpVOTxlvDYmNz44XVHhuoFxVNLNot207TyISkubD/Ouakx+s6aab3/w0f5o6PqKonroQzvYbJTTputPxGvuSdhrc9IdkYIy1WNIx1+erzj3/huWf+8P3H57ubZw4vzLitvLu13piZ92heujJic/Rjf+uf9spWs9nwQA54gBgOYBQMMVVaARM8wIYUPEDiDXIpSldKDObI5EVuwzBjYVbLoWoxTCO/FSYqa/VYoqpgi0BMrAxmUgLiVqvZL9DvuySd+YkPA3Hanpn/s68+eeZ091OfuEeyKxEnVy5d7V4Z+KzqIH7l8iolVBbaG/YLbwdDt3rNP/4YllfwwUdbhlHkomk0zEatRjzOsFEOljMrAP8GImE7jWG6yUtEzrk0beaZi5NGluXtVqqqURQVHFOrAzDYegURSqexfc1cnTuNlW97QualxpZcjijC1ur53/2N/+d9J8xsp5cOqSzLUiRqpH03unS9ZxunHr7/E8v7luIkVsAADYQYZKYui6wBCpQZ8kxHeVkUZVmKyKB7oZNfa8doNxtpkghxXrpSygaHGEEIG1SLWMi92VNKsFXaCnFVIKgMMlkmnKbMKIviyJHjcWumn5Wra+sLcx1XbCXGzu07uDzX3LrWLQq0k6TIyv7WoH96qwSSGRw8cozi9tVrG83DS/OzvdVrG6dPD/Yv88L8TJ5JEqdQAonAM6lSuEyEZOC3EpreZVuGDIFJZ4NQlRLAJqoMyJCxs+OzAtI3JKPfuXjbEzKOiTysA1z/87/5rw90hnNpzt6L10FGUNucSRqgmZIOH3zXkfd/DNEGRqNyq7vVXR8MNvJBbzjYLAebG2tXtMx9PnTZwOe5K0txXtWvER+w+bFkkG1tFfMN22FmbaczGK0CgDKpITUhvQZ404Q0MKF7IsErgdWqQjTi2Ga539gYXrm+kXbmh6PrsGZhNk1IZ2caS7OzMy2nvsxHI1dArbIxRVFaRmsOy/tWFhcXi5LXqV+KmW3PLS8ubm5cv3qla6g3d3R/WQxsZMy421yISQoJAEL0pvb/tWxL5u2ey5PHIhKlaWi8AzKqBAK/hvvmziTm256QBCkHoyhNvvSr/yq/9sLDJ+dl+LKORsbMcJoWBbrDgogM4u7G1evf+upzX/ytPBvmg0Ex7KobSZm5vO/z0XynCXUspVEfQUJONEh6zSNu0JufXxhGkNIxyDs1cahgsqzMYolsiGLqawTxXm//OQxUDotTvWoojGo1Z3SkfnXr+tVy38FNYm7MNJMo6a/1D983P9eOhv3V/sY1V/pmK7QOsr4oZ2eTE/cc6czPZk56w0FqueuGpci+fQeXFlq+eMqLOJcBQmSCwurHnWRp0j3yLf8WU+QMJARIVY0xzrnwhrTZRihEJgrNAMfxSNlBQeVQN3CnZfS8LQk5bcOU/bW4lbz6zc+df+4r9x7kdLCBsmwkLedJ2KraPPPqVLJybe3ltTNn87gP9UbdLEuDERmJWjAtjLINDqk4VWyuyonLlghrw1Yy24zhytzaWMuyKMoUzEqkNozkAAUvq3uzF3aCBwvIq4qQCzVd6rG+tdnp7EvTOBsiMXrk6IrX8qWXX2jHWGiRG230N64Oe71WAyZtlSOo50JQihuNhmQlc6qOUktHDi8YY4aDdRV/4uTBrd56b2tj38oCsQgkNMeZVBW/5V/khnAIBZWVOVJVY7jISwDW2ka7BTZQxc0CszXeloTEFCfjlrny3De//qf/v3cdpXR0vU1ZzA0pxXmfI1dupI1FPyo99xvpKKZhkah6F6m36tgVWioBxEgZoG1hIQIoRDDcPIfNYa/hrWI4GETGxnHsoFPlSzbYXUJi4N8sIRlO1Qk5ZQcKDR8JLHnpGy5LG9GBA7CUFdmaSpby6EPv2deeMaOta5aydpvgjBQET0yRiVOh7OqVS42tqNWZUzG9jf7ygQPGCDMNh5kT12iRaTWUSyZQaFVFTCAZT8J7sxbka6mswSg11oQGQmVZMnOSJM1GG0S7KmJucs7uNGfOGG8/Qu5OVu5d+9xv/XcH2v0W+ssNV27mSwuL569etLNtUhWCdzYf5jTMEtO1XAxo1qN0IgmJCYqiggCxkSdWIk8hfUwUnhStpIhnkBqfzifdy7mIEBnyIWjJpFXbtVDzLm++E4yqAzkglBCLgpUIFM3Ot4kwM9N++KHDkZHB5rU40VNHF3257GQrK7qNWFXJ5fAgljiOU1HjyZR+ULrSlQNxUZH3N3uXjDHLSwvtVuvSpQuGaWZ2LsuHxKZiI5EyEzCuvXJv+afZ7dqprEcAcM4xc5qmabMBNlAAPBX78QgurTteZL79CIkxJ8P9f/i1X1leaLR90Upz9LfmkrR7eX2us7DpMxNbElPkXsUuzHZm7KgcFLmJnBOWXNnExgJCSmpiIVsql0ReRUiIPLEQqZSD2EB83owaw2Hu8sLGEZsUw+FYnEw5Cd98DwqCE/VaJRWIQAUW8EWeJYYBajQi1qLdbHg/yAabRR6lTVgWgvNebWQbSUsasYopSmdNnLDP8mzYH6aN1v6VhW7Zm2l1tvrrlvjw4ZVut7uxuba4OF/mBZiJKtNx2ld8G8FcZRx475nZWhtFyY1vU70DAxw3x97T48uQYulAw9CFo0QiaEpoc+gQE4wbcNxHfvWLv/3vzIX/2Gm3m0kTuSDECZ1IUUSEohwWed9pRlyCXRgpMNuFMpirzC0hdsqixpNVkCerxF5ZQj9/8IvNfVmvv5Q0ji8sbpw7e2hxZmE2yX0PkSswKrQUEZLYiGEXsSC3HiG4rdsOHlKlqSQg0u2cIJVdbTt0+sGN95ZcaBfgvCpxkiSllyzL4jguyxLioygyBPElAGZ245Zc08FPIjJmh7tkItzKW/mldolBZ8CCSNSqGIGQCKMwKKyJOjODvIhsIzXp2edPW28++dGPR3/7dyY9FN7QerjDsPck5NhtPl3TpAADwwFmmtBc2Qok+8FXPr9+9fzDyytFluf9viU2QOkEolHEmfeeNOfIexaRkKsp0HxlznufF64oysJJ4eGVRFmNBVllIxSBmWBC8cPiyileLA/PzeRXL/WyYbMvh46c5MIPyp6oVSVAq+6JJDJe5KF2ZMJBna5Ewo5ElF3tBXTXg5BeN/U4y0tmzzYqfWkiO8idMYZMpGTIQNkqofTO+9DHCjBVigyqrG5UzR130eGNsePGXNZxLSUpQgMiE2ajJ1GaZ7khy0pEkfOm3Zq3jfnpTiWT9J1dW76TsScJGdwsoOD0pnE162wT5MDWI9t8/Ku/c+6lJ5Yaqv0sNSZqNIhQlFnmcw9fwGbOgazEM16iokDh2HlWoad7lq2J4zRpp0mj1Wh2GmnTRPHKvoM2ThvNTntmrj0zG3Vm0WggstAcRR/WbX71DzbWLmz2rwyK/lZ/08aRahLmUTGJGAE8ARhLPKLKJAq0DEbmdEXIG9dMJql2RJS0Oqra6swOh1mSpmubm3HSsIkpyxLkKCx0jtk4y7DWlmVor77ta8Hrrv43W1cVjk7J6GSCNSlBYrbZaJA0EkNRPpJCk8b8YbtwZFqnn1YHakIG7DlCegIoBKps8HcyQuc3tZY0z+A3Lz777bPPfW+xUy7ORp11Y4wpfNEfbuXecRxxkuSOBo5L38h9XLomqNOcXV6Y29/pzN//0HuiOG61Ws3OTLPZRpJiuxAplAWFKASciiqlo3XNiTo+mtvP7XYxxFCzpN10pQBxWI1CJVGp5AlOXRLWGeskfRwYS6BJv8MJM3cZnVN67diaCA+2B7wapzKbNo1w2p7R3oDiljGmwMg7o74kBZGyiQpVV8Iw77BKpoTkNN4UG6bJYwHAANaRIRiCEAnDkZcEnIBs3L58ec2hGc0ewb57b9zanjOafqzYe4Qcm1QME0p3Q4fFyBJGJSXm0hPPvPj09+8+uqLZxWLrakp5f5BlruTIJml7JNQdyqCM+rlpzh1eWjm1vHLP0tLJxaWjtHAQppGxZ2ZjDLH1qqVX58R5NXEcSvV4XBVUKXWNA2TnYNZbdz/cefzQ6tor569duefYsdLn5C0hpGirolSjUMdIqsqGcBihif50RxoAU8x8/fKl6cUa/i1EiSxHaZyYpNVJWzOt9iwzt9qzeZ7n+UhFImaQuKLM8xHU3xiZuGlx5hvBjSorwwuRKglHIEsqTGKgPis6aQywZbu61rOdE9HcMSzfM13s8hZ24B2PPUfI4G5TYKp1b5hdA/Elgw4ePjC4uq/cejmJOe/nzvdckcGmcWO2V5pr3QKtg8tH7vqpRz7emD0ULx5FNANNoQkKlcKns00g9DtjAlJLsNukCNzabuOvWBtBS9tiaTTnm4sHJWqs9bomSXlUQkIJSOWx8aoKnSSeTRa9hDi5VEeHneLotWzIyVamxSOIDMdM1jt1qlmeM9u89KouiiIBM1uwGGtjY0pTgsmN8t2nOFxubnwSAKpJsbfEhE5GBWDHJMwebNgwHCmh9G3bHjnd2twqnZ1bPto+cBda+yfDgiYbqZk5jT1HyJ0wAKAVY7jRxGgTC4t3P/r+Z75+FpyatB1pHBGPMl5dG/Yx11p66NRDHz1y/4fRXIE2SonLISmxMZKmETciSBnGEVdKG4fqiuA5gtkOjCsYUM0SE8WRuhm4rcMnHrn0yjOrZx8/f/HKUqdD7EK4QtR7Elb2tH0+lYBJ5W9oDjLx8aAiwOv0r5m2G3dkJuUFs1tdXS3LshtFXpBlmXMuZKgxM9QbwFqr4pxzoY5il+PkL6kl7pCTcARWUk9QMkIk8CQUGTakLLh47kpn7sDigeNLR+8F2jdKyJqQ09hzhAz+m3El+VSjGoI45UYL+RCziw986BOP/8Xne2Uk3WGUzg4pLuLZI8ff/+AHPoP97yqzRNyc2phjWJ4EvAuFj8QEElab1SATJ9FwDwkzxEO2DhwaUPiS8kG5dPD+fQcfWL949tVXL608ci+oAJcQDwWURJkQv2Vt8EbcWM0MICIxTGUxIiWjPDszy8yF96PRqJkkxhhX5kWWS5mRehaB+dH6Tgie2IOckhcSGluncRyrg3i+cnV138PvaS3s76wcydQE2TypQavZuAt7rg1kcK2anc6OqgNVRB4GtgFuYuHwwXseHZn5oTm0US5e6bcXDr/vwQ9+VpuHtjbJRwvext7CUdXVXFESvIVXEyvZcANZsIWJYGOYGCZWisMzwlwS5arOIcyrAVqba7K0fN/B/Q8MBrq52QWKZiKJyRIqm1GUcOqGwOSSHybkjG02mp5jPh6ks+MYpzCpk5g8nvRNthDyZQyJjRh1xaBXjPrs8lbEJCXKjKWMjVhxWuZSZGVZeu+995MthMe3vGroTkzIE/4Nx8XMaRqLuLLMQc5E4lF6X4IpSVtr3cHLpy/MLB2cXdp/6MTdI8XGqMjzPJRlhYOafNdtWTzvAOxBCTlueaoI14vAKM8QgIm9NlCMysxsaQeto9cvd/tZcfTUA3c98inMHi8lEU29IW/AVX9DYBJCAU0sqsmVeRymAAFCUDIkUGJP5OGNDiHOlYXL1fqmiVZm5k8eOHzp5ZefuffUTHuplQ+zbOjj1KbxHDdbpWxoaB4+RtXydOxrHcsEHX/7WAzq9tMEotAeChiPggwRT2VfCFSFPBRgR0a5yuQmCs1qRNWTqKpjUbVJ1bhr1z2AKYE5rYNWf19DdtFOrK1vduYX2TbXBkOfFbOz8ylHZX+rRHTx2ubmiI88eN/hU+8aFG5zdS1eSCen5XW+4k7GniPkeOLi1BNUDSf0BCWUxMNBOegW3Szl5gFqsdPe4bs/0dl/f3eQI2XbihA7qANHBA5UZGWESr+pIAR2fEll2rEEPomIUxWDrnfiirIclTYjaKcxc3jlwH0vX3v10qVN4/OlxdkkNmsbuRsMmq2FwEAd30LSDunUpUGmmAeoysSwm9xPvLI6eXK8iFVKAkOVFaIiUgjxRIiJChExKVej+MIYuu1oB40bbVVfNGHFJIlHbi6sxpeR6rPMlbhcWNy/sdXPUc7OzquJh4Oe52hpdunxJ55VNOcOHk1mD/VLQyWRkkopFAf5XMWEprp+vdEF8o7GHiTkuGHUDW1Ogw7ovA5yv9ErRyP22k4W5lcWsO/4I0MfXe91983Nk1WnQyJiFAaxgTHj4cEQxNa95i9fWY+q4r336h28tzJQgXdlURSjYSkZCt8Uu3Dg2EPdS4+fPjuCRgvzK0kSuYHko6GwG/dOHjtQwwr326NHINsK6zTxbno/kSfhnksHoOr/HVIodLK+BYCKE6gSGWgos8ANYg07fSrTthzzzQXXrgBMEMhENBwhTjtW1eWFjbTTaAwG2fMvnVnb0sPHT8ztv6cvnX63nF807Sjyeeaj1Hsf/E81CW/EniPkuD0hg3aLSlEPUfIOXshBSoaLCmq0Z1olxT73M7MLrWZnc9A1hkwcGTVEYsZRk7E2WO7abPXvRKiJkqrxTlyp3kNKOFVfOMkzX46yvBipZHEjPTSzWHSvvvjc873FheLAoYOdueZgmIvzAIjIj3N2AqPMeO1VvJrQDNvpY2+EkEXhlAnwoXebTJrzQzXMiFRV9UwIGqwPmjpRMPkm3GOuRkGOoyDjRDba4fvZ5ZidmJET+TbKOE1tFFvSosyLIs+ur/WvrOV33/fBkc5cH6aNleVWe39RYLDZS2fUUdsYE/g8fYx/6YXzDsGeI6Qf58sQqj5uYTS4ARg5QQ35Vmy12Y5KFxWyrkWB4dlLr7RnZxYWF0eDvBi4peVl5xzDmOC5CQihjZv2F6/Mx6BlKlRE1Yt3XqR0rhDvyxJljmLky6FXccmwn7Z5f3tGh3Jho9vLysvt2YZNbFJ5EUEM1UqtVA2tiIHxynaTdlh6Ex5iapnu4iQRB4ePqKiqEKTK4AnXLx9MVsMgUoGKVKX6RIQpQpKpmBpuNM509dgm3vT9hJC7KNpoHRkNuqPRILa61e9eW83V2IMHT458o1c2ELUsL2Ta4KHnOOt0OhP3UrhA1MrqLuw5QgoCESe9GwRQhgASs1jDBDKNpDM/W8ZxL0pS2sizclR2bU69XhSNyjhtk0vYJxxSYk0wlXzVAoB2lv9MYoNBUChU4dU7cgWxI2fFqi/Ksiz8KJcs1zJXFU00T8tR3KSFmfl4OLi4unlhtT+amcOcTYjIGDPR/VSVRENPN1Wlyh6cuJENXldCYic5jTGiKgKn3gsw7jwZ0ucniqzxIAITvJfAteDiIqpkox9LyInwDP+6/397V7fltm2EZwZ/JKXdtdc9Tn9y25u+/3v0IXqOT9okrb1eSSQBzPQCJARRonc3blJtjO9CgigQBEB8GGAwGNDxVPaSfgklGxP6h922df6w+/DPH0OAN/fa3fx5B+0P/9qZ+780zfvdoPqHcPeWjNLoPWvOOt4qGM9xdYSUmZMyqV4EgBFChGiTftBQaw2QBtd1ZHQbA2MMiqAdh56w6+zNx5/6u9vkAyCvaSanqSEsnDjN7S0JyKRAiqg8qBEwgLKkRYYQvffjEIaBw8gCQgitj+2nx08P8bFx+Pb+7cD/+fwIwzAoBVqDMaAUmJmWSmPBRoBZiQPgc/j0+hSelUPzJ0QRCAIxOXNN/jdo2nICs7EBzuayeJSCmZOcbPvy9VlGIiJ6jAs2lpzM4bww8/PHj//YfVQc7m/h3VsDaB8f93vo7t//dYfbvceWTWM3xjgJPO53yr0rZf5czMrMCVdHyFP5lTdhuXT8J6d1OefYsDQCt3e34U95bQ3SDkDau1sS5+OJIiPx0K0VmIEBIQ0FQaJAAApKRe+9CqFl4Mhq7GncQXg44OPBHuiN8d0f+ke3+/xp+LgH+V4p9Tf9wfvQ95E5AAuSIEcEMRZJOPmNREScjjdlr+5hvUVKgfQTZ4m6EJ684nk8SeZyArkQiYtPiwdIG6hS/SOmOSoLMouPcQwxhBCjCChEbN/8Ud1+L0A/CPhHZLTUdGi7xnfUsFUPhhUNHsZ7I981qk2SOW1WTo6wUvg5beNbwKupiEINSKVSQWudBnJ5kvOV6rtFqzVNC0hhXllMOhNtDQDw2AcWMSPZln3o9zu/7//+6UcAUCjGGGt144wxrSI4fPoM02HPQtMqgiCAjw/nZYQzY7eMGC8PaxcbjgvwolAJuf8q9T0AoKSfMoCAoAQhHTgEyXEjIZEF7ZRTVjul1IefH4gUoBKl0Ti36Tq3tdtb0Q61I9uSc2is0g0QcTGJXfQpdSaZ8GoImZBeZ9kQFxOSxSgrt7/np58ax7QjBJGaTivTkCJrddO6zba7ufP9/t13++GwH3YP49CHsT/sHh8eHg67z7B/H8I4juMwjvswhjByHyKH1r1HiSJRmJmnAADYM1k3F2Q+7eJ0uqWVu6jsSX4WzyETIY+1MaWjNQAkj3NJm5P+dXRUeqUt2jKZZyCzeA7DeBjHEAKIACN0m3eubW9uunZz13Rb026h3Xi9UbbFZkNuo5qtarboGtYukG60Pu8xKyEzrs+FxwsRQjh/l/kFP5+QZePOVmYiEsPIHCREligxRD+EcYjBE3Acej/0wpGAQxgPu/3hcPACPgzDMPh+GIaDH0fvB/Zh9ANwjDHGMIYQYvQSIjPj47/zE7ON2yIn6Xr+CXM3VFrYrZWLufAkUtRDShDOOgLiQ4rJMAnxJCfHMZC21lrjrHONc842jdZa0abrNrd3d9vbO9feoDFAVsi4zUbbzm1u2s1Nt7nrttt2uzG2abY3WmtjjNZ6IbSf/c5/z3hlEvIcC91D2TS/RjzirJSPaYOkjYgIHAWiREZhDh44SAyAyT/UxOG96ExmEUFgZoYY+r4HFpbIIfHLA7OI3OKQqFVanIpIOsggXQ8h5L9KZuafX+Dk2VbkCSXtyxRMOnYoLw6RAgABbNsWFE3GeTDVDyi6624RERWhMlob7RptG6UtaaeMVa6xrjFNa01jnFXKEB0nkJWK53hlEnK92S1nX6WQfFHKi0f0QIiAMtlDAycfVhEFADnREEUAOLUwr2wmFfLEHIncdg7nJcdknjbls9+Vec6BkmknDKeTi/mvtSHr2tzSGFMO7HOgNXP8ZKAxebjEECMREWlUhKhoBvse5mEtqrQhUwOhIi2kkDQRIWlEFEIA0nCqa6tsPMUrI2RGzvbaG72oFPnyXGUhXVNgmFdGkqnPfMAaa0yiQwSSJbfMypKQU1A0ORsVkTB6yEsRRU8R0S6e+OU26v1wcQ65RrxiSnih24LzIYYfi8iTv1kQCjwRUo73EiISjESktJ4Tpxijj8EYIyIznyFn2Ck6f3RFxmslZMJFsVa+419GyDLOmBcyZyOCOTxZfBPOhtwADGxhAJg2XgGqNBGUyGTttITDp6uQdMFPafF/LkgOLa+k7xWbcDi1LV/em5+SA3ouaVlPaXNo4rbA8TPVTGRgjrlLIjpRRylAJMkdk5pFPHzxTX2zeN2E/A0QL/s+Xtrf5UrUsLsQR6ho+1TeI9q+KD9rr+ul7Xk1nfTvefyz9C8mkGPRyZWzs8orVvDqlTq/Nsqt0mtdFxcNca7SRbPD493ZQTjmjxdhZRb94pRW0pntewBACpaX5SnvjHCMX8Y85WHpUKgS8kuoEvJJ+CJ8sdGXzlaBVyRAGae8gV56+oCseBZfFZErBHjpe8fLlJfT9LMlIuRAfs6k733Z+ZPfGqqEfBJPSh7GolE+54DIWCTpnnEK+ml2/kdzrdV01vJz+SScbMBQEI+LK3RMU37hkOCbQiXkk/jaIdYrG4GUhHlO1i9MN89rrA5Tn4tKyCeQh2TP7NnzgOy5gu/Fc6rfsHGvlrkonPATWcJX1yf9P1HnkE+gHIKutbuTdssnX8cFkWPskwqXK5tTnfcjK6WePbPziY4q75rLOKpnpzXYr83h7xtVQr4Aa8r7kmEohRIIC5XGQuWYf+B1EfK8e14R9VMFnJgjnN0sCCDzuQl4Fr/iDFVCVlRcEepsu6LiilAJWVFxRaiErKi4IlRCVlRcESohKyquCJWQFRVXhErIioorQiVkRcUVoRKyouKKUAlZUXFFqISsqLgiVEJWVFwRKiErKq4IlZAVFVeESsiKiivCfwGU/jXqV4DbsAAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from PIL import Image\n", "import requests\n", "from io import BytesIO\n", "\n", "url = 'https://images.mulberry.com/i/mulberrygroup/RL5792_000N651_L/small-hampstead-deep-amber-small-classic-grain-ayers/small-hampstead-deep-amber-small-classic-grain-ayers?v=3&w=304'\n", "response = requests.get(url)\n", "img = Image.open(BytesIO(response.content))\n", "img" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "本章代码来源：https://github.com/hktxt/Learn-Statistical-Learning-Method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "SVD.ipynb", "provenance": [], "version": "0.3.2" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 1 } ================================================ FILE: 第16章 主成分分析/16.PCA.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第16章 主成分分析" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1.假设$x$为$m$ 维随机变量，其均值为$\\mu$，协方差矩阵为$\\Sigma$。\n", "\n", "考虑由$m$维随机变量$x$到$m$维随机变量$y$的线性变换\n", "$$y _ { i } = \\alpha _ { i } ^ { T } x = \\sum _ { k = 1 } ^ { m } \\alpha _ { k i } x _ { k } , \\quad i = 1,2 , \\cdots , m$$\n", "\n", "其中$\\alpha _ { i } ^ { T } = ( \\alpha _ { 1 i } , \\alpha _ { 2 i } , \\cdots , \\alpha _ { m i } )$。\n", "\n", "如果该线性变换满足以下条件，则称之为总体主成分：\n", "\n", "（1）$\\alpha _ { i } ^ { T } \\alpha _ { i } = 1 , i = 1,2 , \\cdots , m$；\n", "\n", "（2）$\\operatorname { cov } ( y _ { i } , y _ { j } ) = 0 ( i \\neq j )$;\n", "\n", "（3）变量$y_1$是$x$的所有线性变换中方差最大的；$y_2$是与$y_1$不相关的$x$的所有线性变换中方差最大的；一般地，$y_i$是与$y _ { 1 } , y _ { 2 } , \\cdots , y _ { i - 1 } , ( i = 1,2 , \\cdots , m )$都不相关的$x$的所有线性变换中方差最大的；这时分别称$y _ { 1 } , y _ { 2 } , \\cdots , y _ { m }$为$x$的第一主成分、第二主成分、…、第$m$主成分。\n", "\n", "2.假设$x$是$m$维随机变量，其协方差矩阵是$\\Sigma$，$\\Sigma$的特征值分别是$\\lambda _ { 1 } \\geq\\lambda _ { 2 } \\geq \\cdots \\geq \\lambda _ { m } \\geq 0$，特征值对应的单位特征向量分别是$\\alpha _ { 1 } , \\alpha _ { 2 } , \\cdots , \\alpha _ { m }$，则$x$的第2主成分可以写作\n", "\n", "$$y _ { i } = \\alpha _ { i } ^ { T } x = \\sum _ { k = 1 } ^ { m } \\alpha _ { k i } x _ { k } , \\quad i = 1,2 , \\cdots , m$$\n", "并且，$x$的第$i$主成分的方差是协方差矩阵$\\Sigma$的第$i$个特征值，即$$\\operatorname { var } ( y _ { i } ) = \\alpha _ { i } ^ { T } \\Sigma \\alpha _ { i } = \\lambda _ { i }$$\n", "\n", "3.主成分有以下性质：\n", "\n", "主成分$y$的协方差矩阵是对角矩阵$$\\operatorname { cov } ( y ) = \\Lambda = \\operatorname { diag } ( \\lambda _ { 1 } , \\lambda _ { 2 } , \\cdots , \\lambda _ { m } )$$\n", "\n", "主成分$y$的方差之和等于随机变量$x$的方差之和\n", "$$\\sum _ { i = 1 } ^ { m } \\lambda _ { i } = \\sum _ { i = 1 } ^ { m } \\sigma _ { i i }$$\n", "其中$\\sigma _ { i i }$是$x_2$的方差，即协方差矩阵$\\Sigma$的对角线元素。\n", "\n", "主成分$y_k$与变量$x_2$的相关系数$\\rho ( y _ { k } , x _ { i } )$称为因子负荷量（factor loading），它表示第$k$个主成分$y_k$与变量$x$的相关关系，即$y_k$对$x$的贡献程度。\n", "$$\\rho ( y _ { k } , x _ { i } ) = \\frac { \\sqrt { \\lambda _ { k } } \\alpha _ { i k } } { \\sqrt { \\sigma _ { i i } } } , \\quad k , i = 1,2 , \\cdots , m$$\n", "\n", "4.样本主成分分析就是基于样本协方差矩阵的主成分分析。\n", "\n", "给定样本矩阵\n", "$$X = \\left[ \\begin{array} { l l l l } { x _ { 1 } } & { x _ { 2 } } & { \\cdots } & { x _ { n } } \\end{array} \\right] = \\left[ \\begin{array} { c c c c } { x _ { 11 } } & { x _ { 12 } } & { \\cdots } & { x _ { 1 n } } \\\\ { x _ { 21 } } & { x _ { 22 } } & { \\cdots } & { x _ { 2 n } } \\\\ { \\vdots } & { \\vdots } & { } & { \\vdots } \\\\ { x _ { m 1 } } & { x _ { m 2 } } & { \\cdots } & { x _ { m n } } \\end{array} \\right]$$\n", "\n", "其中$x _ { j } = ( x _ { 1 j } , x _ { 2 j } , \\cdots , x _ { m j } ) ^ { T }$是$x$的第$j$个独立观测样本，$j=1,2，…,n$。 \n", "\n", "$X$的样本协方差矩阵\n", "$$\\left. \\begin{array} { c } { S = [ s _ { i j } ] _ { m \\times m } , \\quad s _ { i j } = \\frac { 1 } { n - 1 } \\sum _ { k = 1 } ^ { n } ( x _ { i k } - \\overline { x } _ { i } ) ( x _ { j k } - \\overline { x } _ { j } ) } \\\\ { i = 1,2 , \\cdots , m , \\quad j = 1,2 , \\cdots , m } \\end{array} \\right.$$\n", "\n", "给定样本数据矩阵$X$，考虑向量$x$到$y$的线性变换$$y = A ^ { T } x$$\n", "这里\n", "$$A = \\left[ \\begin{array} { l l l l } { a _ { 1 } } & { a _ { 2 } } & { \\cdots } & { a _ { m } } \\end{array} \\right] = \\left[ \\begin{array} { c c c c } { a _ { 11 } } & { a _ { 12 } } & { \\cdots } & { a _ { 1 m } } \\\\ { a _ { 21 } } & { a _ { 22 } } & { \\cdots } & { a _ { 2 m } } \\\\ { \\vdots } & { \\vdots } & { } & { \\vdots } \\\\ { a _ { m 1 } } & { a _ { m 2 } } & { \\cdots } & { a _ { m m } } \\end{array} \\right]$$\n", "\n", "如果该线性变换满足以下条件，则称之为样本主成分。样本第一主成分$y _ { 1 } = a _ { 1 } ^ { T } x$是在$a _ { 1 } ^ { T } a _ { 1 } = 1$条件下，使得$a _ { 1 } ^ { T } x _ { j } ( j = 1,2 , \\cdots , n )$的样本方差$a _ { 1 } ^ { T } S a _ { 1 }$最大的$x$的线性变换；\n", "\n", "样本第二主成分$y _ { 2 } = a _ { 2 } ^ { T } x$是在$a _ { 2 } ^ { T } a _ { 2 } = 1$和$a _ { 2 } ^ { T } x _ { j }$与$a _ { 1 } ^ { T } x _ { j } ( j = 1,2 , \\cdots , n )$的样本协方差$a _ { 1 } ^ { T } S a _ { 2 } = 0$条件下，使得$a _ { 2 } ^ { T } x _ { j } ( j = 1,2 , \\cdots , n )$的样本方差$a _ { 2 } ^ { T } S a _ { 2 }$最大的$x$的线性变换；\n", "\n", "一般地，样本第$i$主成分$y _ { i } = a _ { i } ^ { T } x$是在$a _ { i } ^ { T } a _ { i } = 1$和$a _ { i } ^ { T } x _ { j }$与$a _ { k } ^ { T } x _ { j } ( k < i , j = 1,2 , \\cdots , n )$的样本协方差$a _ { k } ^ { T } S a _ { i } = 0$条件下，使得$a _ { i } ^ { T } x _ { j } ( j = 1,2 , \\cdots , n )$的样本方差$a _ { k } ^ { T } S a _ { i }$最大的$x$的线性变换。\n", "\n", "5.主成分分析方法主要有两种，可以通过相关矩阵的特征值分解或样本矩阵的奇异值分解进行。\n", "\n", "（1）相关矩阵的特征值分解算法。针对$m \\times n$样本矩阵$X$，求样本相关矩阵\n", "$$R = \\frac { 1 } { n - 1 } X X ^ { T }$$\n", "再求样本相关矩阵的$k$个特征值和对应的单位特征向量，构造正交矩阵\n", "$$V = ( v _ { 1 } , v _ { 2 } , \\cdots , v _ { k } )$$\n", "\n", "$V$的每一列对应一个主成分，得到$k \\times n$样本主成分矩阵\n", "$$Y = V ^ { T } X$$\n", "\n", "（2）矩阵$X$的奇异值分解算法。针对$m \\times n$样本矩阵$X$ \n", "$$X ^ { \\prime } = \\frac { 1 } { \\sqrt { n - 1 } } X ^ { T }$$\n", "对矩阵$X ^ { \\prime }$进行截断奇异值分解，保留$k$个奇异值、奇异向量，得到\n", "$$X ^ { \\prime } = U S V ^ { T }$$\n", "$V$的每一列对应一个主成分，得到$k \\times n$样本主成分矩阵$Y$\n", "$$Y = V ^ { T } X$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "本章代码直接使用Coursera机器学习课程的第六个编程练习。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "PCA（principal components analysis）即主成分分析技术旨在利用降维的思想，把多指标转化为少数几个综合指标。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import seaborn as sb\n", "from scipy.io import loadmat" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "data = loadmat('data/ex7data1.mat')\n", "# data" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAArkAAAHSCAYAAADohdOwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAbf0lEQVR4nO3dUWil6Xkf8P8bjUyO3RjRehosbZzd3OjGQy0zuJQB09pplBBjhqUXDqTQ9mJvSnAoKOyUQkkvOgOCkl4FlnWDS1yH1BnrwttaMUyNa2hsZq111MTWRYONfabpjikicXqoh+nbi5VmdsYa6RzpnPOd857fD4ad/STrPGsdOP/v/Z7nfUutNQAA0JKf6LoAAAAYNyEXAIDmCLkAADRHyAUAoDlCLgAAzRFyAQBozqVJ/ND3vOc99fnnn5/EjwYAgCTJ66+//oNa6+WTvjaRkPv888/n7t27k/jRAACQJCmlfPdZX9OuAABAc4RcAACaI+QCANAcIRcAgOYIuQAANEfIBQCgOUIuAADNEXIBAGiOkAsAQHOEXAAAmiPkAgDQHCEXAIDmCLkAADRHyAUAoDlCLgAAzbnUdQEAAM+ys9fP9u5B7h0OsrrSy9bmeq5vrHVdFnNAyAUAZtLOXj83bu9n8OBhkqR/OMiN2/tJIuhyJu0KAMBM2t49eBRwjw0ePMz27kFHFTFPhFwAYCbdOxyMdB3eTsgFAGbS6kpvpOvwdkIuADCTtjbX01teeuJab3kpW5vrHVXEPDF4BgDMpOPhMrsrcB5CLgAws65vrAm1nIt2BQAAmiPkAgDQHCEXAIDmCLkAADRHyAUAoDlCLgAAzRFyAQBojpALAEBzhFwAAJoj5AIA0BwhFwCA5gi5AAA0R8gFAKA5Qi4AAM0RcgEAaI6QCwBAc4RcAACaI+QCANAcIRcAgOYIuQAANEfIBQCgOUIuAADNEXIBAGjOmSG3lLJeSnnjbX/+opTy69MoDgAAzuPSWd9Qaz1I8oEkKaUsJekn+fyE6wIAgHMbtV3ho0n+R631u5MoBgAAxmHUkPuJJJ+dRCEAADAuQ4fcUso7knw8yX98xtdfKqXcLaXcvX///rjqAwCAkY2ykvtLSb5Ra/1fJ32x1vpKrfVqrfXq5cuXx1MdAACcwygh91eiVQEAgDkwVMgtpbwzyd9Pcnuy5QAAwMWduYVYktRa/0+SvzHhWgAAYCyceAYAQHOEXAAAmiPkAgDQHCEXAIDmCLkAADRHyAUAoDlCLgAAzRFyAQBojpALAEBzhFwAAJoj5AIA0BwhFwCA5gi5AAA0R8gFAKA5Qi4AAM0RcgEAaI6QCwBAc4RcAACaI+QCANAcIRcAgOZc6roAAADGZ2evn+3dg9w7HGR1pZetzfVc31ib29c5LyEXAKARO3v93Li9n8GDh0mS/uEgN27vJ8lYA+i0XucitCsAADRie/fgUfA8NnjwMNu7B3P5Ohch5AIANOLe4WCk67P+Ohch5AIANGJ1pTfS9Vl/nYsQcgEAGrG1uZ7e8tIT13rLS9naXJ/L17kIg2cAAI04Hvqa9K4H03qdiyi11rH/0KtXr9a7d++O/ecCAMCxUsrrtdarJ31NuwIAAM0RcgEAaI6QCwBAc4RcAACaI+QCANAcIRcAgOYIuQAANEfIBQCgOUIuAADNcawvACPZ2evP9FGeAImQC8AIdvb6uXF7P4MHD5Mk/cNBbtzeTxJBF5gp2hUAGNr27sGjgHts8OBhtncPOqoI4GRCLgBDu3c4GOk6QFeEXACGtrrSG+k6QFeEXACGtrW5nt7y0hPXestL2dpc76gigJMZPANgaMfDZXZXAGadkAvASK5vrAm1wMzTrgAAQHOEXAAAmiPkAgDQHCEXAIDmCLkAADRHyAUAoDlCLgAAzRFyAQBojsMgAADmxM5e34mDQxJyAQDmwM5ePzdu72fw4GGSpH84yI3b+0ki6J5AuwIAwBzY3j14FHCPDR48zPbuQUcVzTYhFwBgDtw7HIx0fdEJuQAAc2B1pTfS9UUn5ALAgtnZ6+farTt54eXXcu3Wnezs9bsuiSFsba6nt7z0xLXe8lK2Ntc7qmi2GTwDgAVieGl+Hf9+7K4wHCEXABbIacNLwtLsu76x5vc0JO0KALBADC+xKIRcAFgghpdYFEIuACwQw0ssCj25ALBADC+xKIRcAFgwhpdYBNoVAABojpALAEBzhFwAAJoj5AIA0ByDZwAAc2Jnr29njCEJuQAAc2Bnr58bt/cfHcvcPxzkxu39JBF0T6BdAQBgDmzvHjwKuMcGDx5me/ego4pmm5VcAGCmeUT/lnuHg5GuLzoruQDAzDp+RN8/HKTm8SP6nb1+16VN3epKb6Tri07IBQBmlkf0j21trqe3vPTEtd7yUrY21zuqaLZpVwAAZpZH9I8dt2ho3RiOkAsAzKzVlV76JwTaRX1Ef31jTagdknYFALiAnb1+rt26kxdefi3Xbt1ZyF7RSfKInvMaaiW3lLKS5NUk709Sk/yTWut/m2RhADDr7Fs6eR7Rc17Dtiv82yRfrLX+g1LKO5K8c4I1AcBcOG0oSggbH4/oOY8zQ24p5d1JPpzkHyVJrfVHSX402bIAYPYZioLZNUxP7s8luZ/kd0ope6WUV0sp75pwXQAw8+xbCrNrmJB7KckHk/x2rXUjyV8lefnpbyqlvFRKuVtKuXv//v0xlwkAs8dQFMyuYXpyv5/k+7XWrx39++dyQsittb6S5JUkuXr1ah1bhQAwowxFvcWxu8yiM0NurfXPSynfK6Ws11oPknw0yZ9OvjQAmH2LPhRlhwlm1bD75P5aks+UUv44yQeS/OvJlQQAzAvH7jKrhtpCrNb6RpKrE64FAJgzdphgVjnxDAA4NztMMKuEXADg3Owwwawa9sQzgIViWhyGY4cJZpWQC/AU0+LMknm44Vr0HSaYTdoVAJ5iWpxZcXzD1T8cpObxDdfOXr/r0mDmWckFeIppcaZhmBXa0264rJzC6azkAjzFtDiTNuwKrRsuOD8hF+AppsWZtGFbYtxwwfkJuQBPub6xlpsvXsnaSi8lydpKLzdfvOLxMGMz7AqtGy44Pz25ACcwLc4kra700j8h6D69Qmt7Ljg/IRcApmxrc/2JbeqSZ6/QuuGC8xFyAWDKrNDC5Am5ANABK7QwWQbPAABojpALAEBzhFwAAJoj5AIA0BwhFwCA5gi5AAA0R8gFAKA59skFAJiwnb2+wz+mTMgFAJignb3+E8c49w8HuXF7P0kE3QkScgGgAVYKZ9f27sGjgHts8OBhtncP/I4mSMgFgDlnpXC23TscjHSd8TB4BgBz7rSVQrq3utIb6TrjIeQCwAzY2evn2q07eeHl13Lt1p3s7PWH/t9aKZxtW5vr6S0vPXGtt7yUrc31jipaDNoVAKBjF203WF3ppX9CoLVSOBuOf4d6pqdLyAWAjl10MGlrc/2JkJxYKZw11zfWhNopE3IBoGMXbTewUgg/TsgFgI6No93ASiE8yeAZAHTMYBKMn5VcAOiYdgMYPyEXAGaAdgMYL+0KAAA0x0ouAABn2tnrz1VLjZALAMCpLnpgSRe0KwAAcKrTDiyZVUIuAACnuuiBJV0QcgEAONWzDiYZ5cCSaRNyAeAMO3v9XLt1Jy+8/Fqu3bqTnb1+1yXBVM3jgSUGzwDgFPM4cAPjNo8Hlgi5AHCK0wZuZvkDHsZt3g4s0a4AAKeYx4EbQMgFgFPN48ANIOQCMCGtDGvN48ANoCcXgAloaVhrmgM383ZsKswyIReAsWttWGsaAzct3RjALNCuAMDYGdYa3TwemwqzTMgFYOwMa43OjQGMl5ALwNgZ1hqdGwMYLyEXgLG7vrGWmy9eydpKLyXJ2kovN1+8orf0FG4MYLwMngEwEfN2OlLX5vHYVJhlQi4AzAg3BjA+2hUAAGiOkAsAQHOEXAAAmiPkAgDQHCEXAIDmCLkAADRHyAUAoDlCLgAAzRFyAQBojpALAEBzhFwAAJoj5AIA0BwhFwCA5gi5AAA0R8gFAKA5Qi4AAM0RcgEAaM6lrgsAgFmxs9fP9u5B7h0OsrrSy9bmeq5vrHVdFkxMy+95IReAoT3rA7GFD8qdvX5u3N7P4MHDJEn/cJAbt/eTZO7+W2AYrb/ntSsAMJTjD8T+4SA1jz8Q/8XO/onXd/b6XZc8ku3dg0cf9scGDx5me/ego4pgslp/zwu5AAzlWR+In/3a95r4oLx3OBjpOsy71t/z2hW4kBYeUQLDedYH38NaR/r+WbW60kv/hJpXV3odVAOT1/p73kou5/asR5fz9ogSGM6zPviWShnp+2fV1uZ6estLT1zrLS9la3O9o4pgslp/zwu5nFvrvTzAk571gfgrf/tnmvigvL6xlpsvXsnaSi8lydpKLzdfvOLpFM1q/T2vXYFza72XB3jS8QffSS1KV3/2rzfRunR9Y20u64bzavk9L+Rybq338gA/7lkfiC1/UALzaah2hVLKd0op+6WUN0opdyddFPOh9V4eAGB+jbKS+/dqrT+YWCXMndMeXQIAdEm7AhfiESUAMIuG3V2hJvnDUsrrpZSXJlkQAABc1LAruddqrfdKKX8zyZdKKd+utX7l7d9wFH5fSpL3ve99Yy4TAACGN1TIrbXeO/rnm6WUzyf5UJKvPPU9ryR5JUmuXr168vE3ADAhTmAE3u7MkFtKeVeSn6i1/uXR338hyb+aeGUAFyT0LI7jExiPD6g5PoExid85LKhhenJ/OslXSynfTPL1JK/VWr842bIALsax04vFCYzA085cya21/lmSvzWFWgDG5rTQY2WvPU5gBJ427O4KAHNF6Fkszzpp0QmMsLiEXKBJQs9icQIj8DQhF2iS0LNYrm+s5eaLV7K20ktJsrbSy80Xr2hNgQXmxDOgSY6dXjxOYATeTsgFmiX0ACwu7QoAADRHyAUAoDlCLgAAzdGTCzziGFwAWiHkAkkeH4N7fErY8TG4SQTdKXOzAXBx2hWAJKcfg8v0HN9s9A8HqXl8s7Gz1++6NIC5IuQCSRyDOyvcbACMh5ALJHEM7qxwswEwHkIukMQxuLPCzQbAeAi5QJK3hstuvnglayu9lCRrK73cfPGKgacpc7MBMB52VwAecQxu947//7e7AsDFCLkAM8bNBsDFaVcAAKA5Qi4AAM3RrgDAQnGiHCwGIReAheH4aobhRqgN2hUAWBhOlOMsjtZuh5VcABaGE+UW0ygrs6fdCFnNnS9CLgALY3Wll/4JgdaJcvPvWUF21BYVN0Lt0K4AwMJwolybTmsxGLVFxdHa7RBygWfa2evn2q07eeHl13Lt1h09acw9x1e36bQgO+rKrBuhdmhXAE5kCp1WOVGuPacF2VFbVByt3Q4hFziR4QtgXpwWZLc215+4YU/OXpl1I9QG7QrAiQxfAPPitBYDLSqLy0oucCJT6MC8OKvFwMrsYhJygROd5xFfy5yABLNNkOVpQi5wIsMXjxnCA5g/Qi7wTFZG3mIID2D+GDwDOIMhPID5I+QCnMEJSADzR8gFOIMTkEbjpDxgFujJBTiDIbzhGdIDZoWQCzAEQ3jDMaQHzArtCgCMjSE9YFYIuQCMjSE9YFYIuQCMjSE9YFboyQXmniN3Z4chPWBWCLnAXDPNP3sM6QGzQLsCMNdOm+YHYHFZyYU55RH9W0zzA3ASK7kwh44f0fcPB6l5/Ih+EU+WMs0PwEmEXJhDHtE/ZpofgJNoV4A55BH9Y6b5ATiJkAtzaHWll/4JgXZRH9Gb5gfgadoVYA55RA8Ap7OSC3PII3oAOJ2QC3PKI3oAeDbtCgAANEfIBQCgOUIuAADNEXIBAGiOkAsAQHOEXAAAmiPkAgDQHCEXAIDmCLkAADRHyAUAoDlCLgAAzRFyAQBozqWuCwBm185eP9u7B7l3OMjqSi9bm+u5vrHWdVkAcCYhFzjRzl4/N27vZ/DgYZKkfzjIjdv7SSLoAjDztCsAJ9rePXgUcI8NHjzM9u5BRxUBwPCEXOBE9w4HI10HgFki5AInWl3pjXQdAGaJkAucaGtzPb3lpSeu9ZaXsrW53lFFADA8g2fAiY6Hy+yuAMA8EnKBZ7q+sSbUAjCXtCsAANAcIRcAgOYIuQAANEfIBQCgOUIuAADNEXIBAGiOkAsAQHOGDrmllKVSyl4p5QuTLAgAAC5qlJXcTyb51qQKAQCAcRkq5JZSnkvyy0lenWw5AABwccMe6/tbSX4jyU9NsBZgwezs9bO9e5B7h4OsrvSytbnuGGEAxuLMldxSyseSvFlrff2M73uplHK3lHL3/v37YysQaNPOXj83bu+nfzhITdI/HOTG7f3s7PW7Lg2ABgzTrnAtycdLKd9J8ntJPlJK+d2nv6nW+kqt9Wqt9erly5fHXCbQmu3dgwwePHzi2uDBw2zvHnRUEQAtOTPk1lpv1Fqfq7U+n+QTSe7UWn914pUBTbt3OBjpOgCMwj65QCdWV3ojXQeAUYwUcmutX661fmxSxQCLY2tzPb3lpSeu9ZaXsrW53lFFALRk2N0VAMbqeBcFuysAMAlCLtCZ6xtrQi0AEyHkQofsEwsAkyHkQkeO94k93kbreJ/YJIIuAFyQkAsdOW2fWCH3MavdAJyHkAsdsU/s2ax2A3Be9smFjtgn9mxORQPgvIRc6Ih9Ys9mtRuA8xJyoSPXN9Zy88UrWVvppSRZW+nl5otXPIZ/G6vdAJyXnlzokH1iT7e1uf5ET25itRuA4Qi5wMxyKhoA5yXkAjPNajcA5yHkAj/G3rQAzDshF3iCvWkBaIHdFYAn2JsWgBYIucAT7E0LQAu0K0ADxtlDu7rSS/+EQGtvWgDmiZVcmHPHPbT9w0FqHvfQ7uz1n/n9127dyQsvv5Zrt+782Pc5iQ2AFgi5MOdG6aEdJhA7iQ2AFmhXgDk3Sg/taYH47SHW3rQAzDsruTDnntUre9J1Q2UALAohF+bcKD20owRiAJhnQi7MuVF6aA2VAbAo9ORCA4btoT3+Hkf2AtA6IRcWjKGy6Rnn/sUAjEbIBZiA4+3ajnezON6uLYmgCzAFenIBJmCU/YsBGD8hF2ACbNcG0C0hF2ACbNcG0C0hF2ACbNcG0C2DZwATYLs2gG4JuQATYrs2gO5oVwAAoDlCLgAAzRFyAQBojpALAEBzhFwAAJoj5AIA0BwhFwCA5gi5AAA0R8gFAKA5Qi4AAM0RcgEAaI6QCwBAc4RcAACaI+QCANCcS10XADAvdvb62d49yL3DQVZXetnaXM/1jbWuywLgBEIuwBB29vq5cXs/gwcPkyT9w0Fu3N5PEkEXYAZpVwAYwvbuwaOAe2zw4GG2dw86qgiA0wi5AEO4dzgY6ToA3RJyAYawutIb6ToA3RJyAYawtbme3vLSE9d6y0vZ2lzvqCIATmPwDGAIx8NldlcAmA9CLjARLW63dX1jbe7/GwAWhZALjJ3ttgDomp5cYOxstwVA14RcYOxstwVA14RcYOxstwVA14RcsrPXz7Vbd/LCy6/l2q072dnrd10Sc852WwB0zeDZgjMgxCTYbguArgm5C+60ASGBhIuw3RYAXdKusOAMCAEALRJyF5wBIQCgRULugjMgBAC0SE/ugjMgBAC0SMjFgBAA0BztCgAANEfIBQCgOUIuAADNEXIBAGiOkAsAQHOEXAAAmiPkAgDQHCEXAIDmCLkAADRHyAUAoDlCLgAAzRFyAQBojpALAEBzzgy5pZSfLKV8vZTyzVLKn5RSfnMahQEAwHldGuJ7/m+Sj9Raf1hKWU7y1VLKf661/tGEawMAgHM5M+TWWmuSHx796/LRnzrJogAA4CKG6sktpSyVUt5I8maSL9VavzbZsgAA4PyGCrm11oe11g8keS7Jh0op73/6e0opL5VS7pZS7t6/f3/cdQIAwNBG2l2h1nqY5MtJfvGEr71Sa71aa716+fLlMZUHAACjG2Z3hcullJWjv/eS/HySb0+6MAAAOK9hdld4b5JPl1KW8lYo/v1a6xcmWxYAAJzfMLsr/HGSjSnUAgAAY+HEMwAAmiPkAgDQnGF6cufCzl4/27sHuXc4yOpKL1ub67m+sdZ1WQAAdKCJkLuz18+N2/sZPHiYJOkfDnLj9n6SCLoAAAuoiXaF7d2DRwH32ODBw2zvHnRUEQAAXWoi5N47HIx0HQCAtjURcldXeiNdBwCgbU2E3K3N9fSWl5641lteytbmekcVAQDQpSYGz46Hy+yuAABA0kjITd4KukItAABJI+0KAADwdkIuAADNEXIBAGiOkAsAQHOEXAAAmiPkAgDQHCEXAIDmCLkAADRHyAUAoDlCLgAAzRFyAQBojpALAEBzhFwAAJoj5AIA0BwhFwCA5gi5AAA0p9Rax/9DS7mf5LvP+PJ7kvxg7C/KPPEeWGx+/3gPLDa/f8b5HvjZWuvlk74wkZB7mlLK3Vrr1am+KDPFe2Cx+f3jPbDY/P6Z1ntAuwIAAM0RcgEAaE4XIfeVDl6T2eI9sNj8/vEeWGx+/0zlPTD1nlwAAJg07QoAADRnaiG3lPLvSilvllL++7Rek9lRSvmZUsp/KaV8q5TyJ6WUT3ZdE9NVSvnJUsrXSynfPHoP/GbXNTF9pZSlUspeKeULXdfC9JVSvlNK2S+lvFFKudt1PUxXKWWllPK5Usq3j/LA35no602rXaGU8uEkP0zy72ut75/KizIzSinvTfLeWus3Sik/leT1JNdrrX/acWlMSSmlJHlXrfWHpZTlJF9N8sla6x91XBpTVEr5Z0muJnl3rfVjXdfDdJVSvpPkaq3VPrkLqJTy6ST/tdb6ainlHUneWWs9nNTrTW0lt9b6lST/e1qvx2yptf7PWus3jv7+l0m+lWSt26qYpvqWHx796/LRH0MBC6SU8lySX07yate1ANNVSnl3kg8n+VSS1Fp/NMmAm+jJpQOllOeTbCT5WreVMG1Hj6rfSPJmki/VWr0HFstvJfmNJP+v60LoTE3yh6WU10spL3VdDFP1c0nuJ/mdo5alV0sp75rkCwq5TFUp5a8l+YMkv15r/Yuu62G6aq0Pa60fSPJckg+VUrQuLYhSyseSvFlrfb3rWujUtVrrB5P8UpJ/etTKyGK4lOSDSX671rqR5K+SvDzJFxRymZqjPsw/SPKZWuvtruuhO0ePqL6c5Bc7LoXpuZbk40c9mb+X5COllN/ttiSmrdZ67+ifbyb5fJIPdVsRU/T9JN9/2xO8z+Wt0DsxQi5TcTR09Kkk36q1/puu62H6SimXSykrR3/vJfn5JN/utiqmpdZ6o9b6XK31+SSfSHKn1vqrHZfFFJVS3nU0eJyjx9S/kMSOSwui1vrnSb5XSlk/uvTRJBMdPr80yR/+dqWUzyb5u0neU0r5fpJ/WWv91LRen85dS/IPk+wf9WQmyT+vtf6nDmtiut6b5NOllKW8dYP9+7VW20jB4vjpJJ9/a80jl5L8h1rrF7stiSn7tSSfOdpZ4c+S/ONJvpgTzwAAaI52BQAAmiPkAgDQHCEXAIDmCLkAADRHyAUAoDlCLgAAzRFyAQBojpALAEBz/j9McNcMFNHIXgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "X = data['X']\n", "\n", "fig, ax = plt.subplots(figsize=(12,8))\n", "ax.scatter(X[:, 0], X[:, 1])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "PCA的算法相当简单。 在确保数据被归一化之后，输出仅仅是原始数据的协方差矩阵的奇异值分解。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def pca(X):\n", " # normalize the features\n", " X = (X - X.mean()) / X.std()\n", " \n", " # compute the covariance matrix\n", " X = np.matrix(X)\n", " cov = (X.T * X) / X.shape[0]\n", " \n", " # perform SVD\n", " U, S, V = np.linalg.svd(cov)\n", " \n", " return U, S, V" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(matrix([[-0.79241747, -0.60997914],\n", " [-0.60997914, 0.79241747]]),\n", " array([1.43584536, 0.56415464]),\n", " matrix([[-0.79241747, -0.60997914],\n", " [-0.60997914, 0.79241747]]))" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "U, S, V = pca(X)\n", "U, S, V" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在我们有主成分（矩阵U），我们可以用这些来将原始数据投影到一个较低维的空间中。 对于这个任务，我们将实现一个计算投影并且仅选择顶部K个分量的函数，有效地减少了维数。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def project_data(X, U, k):\n", " U_reduced = U[:,:k]\n", " return np.dot(X, U_reduced)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "matrix([[-4.74689738],\n", " [-7.15889408],\n", " [-4.79563345],\n", " [-4.45754509],\n", " [-4.80263579],\n", " [-7.04081342],\n", " [-4.97025076],\n", " [-8.75934561],\n", " [-6.2232703 ],\n", " [-7.04497331],\n", " [-6.91702866],\n", " [-6.79543508],\n", " [-6.3438312 ],\n", " [-6.99891495],\n", " [-4.54558119],\n", " [-8.31574426],\n", " [-7.16920841],\n", " [-5.08083842],\n", " [-8.54077427],\n", " [-6.94102769],\n", " [-8.5978815 ],\n", " [-5.76620067],\n", " [-8.2020797 ],\n", " [-6.23890078],\n", " [-4.37943868],\n", " [-5.56947441],\n", " [-7.53865023],\n", " [-7.70645413],\n", " [-5.17158343],\n", " [-6.19268884],\n", " [-6.24385246],\n", " [-8.02715303],\n", " [-4.81235176],\n", " [-7.07993347],\n", " [-5.45953289],\n", " [-7.60014707],\n", " [-4.39612191],\n", " [-7.82288033],\n", " [-3.40498213],\n", " [-6.54290343],\n", " [-7.17879573],\n", " [-5.22572421],\n", " [-4.83081168],\n", " [-7.23907851],\n", " [-4.36164051],\n", " [-6.44590096],\n", " [-2.69118076],\n", " [-4.61386195],\n", " [-5.88236227],\n", " [-7.76732508]])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Z = project_data(X, U, 1)\n", "Z" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们也可以通过反向转换步骤来恢复原始数据。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "def recover_data(Z, U, k):\n", " U_reduced = U[:,:k]\n", " return np.dot(Z, U_reduced.T)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "matrix([[3.76152442, 2.89550838],\n", " [5.67283275, 4.36677606],\n", " [3.80014373, 2.92523637],\n", " [3.53223661, 2.71900952],\n", " [3.80569251, 2.92950765],\n", " [5.57926356, 4.29474931],\n", " [3.93851354, 3.03174929],\n", " [6.94105849, 5.3430181 ],\n", " [4.93142811, 3.79606507],\n", " [5.58255993, 4.29728676],\n", " [5.48117436, 4.21924319],\n", " [5.38482148, 4.14507365],\n", " [5.02696267, 3.8696047 ],\n", " [5.54606249, 4.26919213],\n", " [3.60199795, 2.77270971],\n", " [6.58954104, 5.07243054],\n", " [5.681006 , 4.37306758],\n", " [4.02614513, 3.09920545],\n", " [6.76785875, 5.20969415],\n", " [5.50019161, 4.2338821 ],\n", " [6.81311151, 5.24452836],\n", " [4.56923815, 3.51726213],\n", " [6.49947125, 5.00309752],\n", " [4.94381398, 3.80559934],\n", " [3.47034372, 2.67136624],\n", " [4.41334883, 3.39726321],\n", " [5.97375815, 4.59841938],\n", " [6.10672889, 4.70077626],\n", " [4.09805306, 3.15455801],\n", " [4.90719483, 3.77741101],\n", " [4.94773778, 3.80861976],\n", " [6.36085631, 4.8963959 ],\n", " [3.81339161, 2.93543419],\n", " [5.61026298, 4.31861173],\n", " [4.32622924, 3.33020118],\n", " [6.02248932, 4.63593118],\n", " [3.48356381, 2.68154267],\n", " [6.19898705, 4.77179382],\n", " [2.69816733, 2.07696807],\n", " [5.18471099, 3.99103461],\n", " [5.68860316, 4.37891565],\n", " [4.14095516, 3.18758276],\n", " [3.82801958, 2.94669436],\n", " [5.73637229, 4.41568689],\n", " [3.45624014, 2.66050973],\n", " [5.10784454, 3.93186513],\n", " [2.13253865, 1.64156413],\n", " [3.65610482, 2.81435955],\n", " [4.66128664, 3.58811828],\n", " [6.1549641 , 4.73790627]])" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_recovered = recover_data(Z, U, 1)\n", "X_recovered" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAHSCAYAAADmLK3fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dfYyl51kf4N/NeCgnITCo2YJ3bONWoK0U3HhhlARZqsBQNoTUrEwkjAptUCu3qB+hrRZlEeIjqmTQSDRtIxG5SdtAwmeymZo0sKRKEBA1RrMeh41xRkrblHg2rZeESeJmRMebp3/szLI7ntn52PP9Xpc02nPe8845t3SU5Jdn7ud+qrUWAADomi8bdQEAADAKgjAAAJ0kCAMA0EmCMAAAnSQIAwDQSYIwAACddNtBbqqqTyb5QpIrSZ5vrS3seP3bkvznJP9z69K51tqbbvaeL33pS9vdd999yHIBAOBwLly48KettWM7rx8oCG/59tban97k9d9vrb32oG929913Z3l5+RAfDwAAh1dV/2u361ojAADopIMG4Zbkd6rqQlU9vMc931pVH62q36qql/WpPgAAGIiDtkbc11q7VFV/JckHqurjrbXfu+71J5J8fWvtuap6TZKlJN+48022QvTDSXLXXXfdYukAAHB0B1oRbq1d2vr32STvTfKKHa9/vrX23Nbj9yeZraqX7vI+j7bWFlprC8eOvaBfGQAAhmbfIFxVL66ql2w/TvJdST62456vq6raevyKrff9TP/LBQCA/jhIa8TXJnnvVs69Lckvt9Z+u6r+UZK01t6a5HVJfqSqnk+ykeSh1lobUM0AAHDL9g3CrbX/keTlu1x/63WP35LkLf0tDQAABsf4NAAAOkkQBgCgkwRhAAA6SRAGAKCTBGEAADpJEAYAoJMEYQAAOkkQBgCgkw5yshwAABzJ0spaFs+v5tL6Ro7P9XLm1ImcPjk/6rKSCMIAAAzI0spazp67mI3NK0mStfWNnD13MUnGIgxrjQAAYCAWz69eC8HbNjavZPH86ogqupEgDADAQFxa3zjU9WEThAEAGIjjc71DXR82QRgAgIE4c+pEerMzN1zrzc7kzKkTI6roRjbLAQAwENsb4kyNAACgc06fnB+b4LuT1ggAADpJEAYAoJMEYQAAOkmPMAAAexrnI5JvlSAMAMCuxv2I5FulNQIAgF2N+xHJt0oQBgBgV+N+RPKt0hoBAMCuvcDH53pZ2yX0jssRybfKijAAQMdt9wKvrW+k5S96gb/9rx8b6yOSb5UgDADQcXv1An/o45fzyIP3ZH6ul0oyP9fLIw/eMxUb5RKtEQAAnXezXuBxPiL5VlkRBgDouL16fqelF3gvgjAAQMedOXViqnuB96I1AgCg47ZbH6b1BLm9CMIAAEx1L/BetEYAANBJgjAAAJ0kCAMA0EmCMAAAnSQIAwDQSYIwAACdZHwaAMCEWFpZ69ys30EShAEAJsDSylrOnruYjc0rSZK19Y2cPXcxSYThI9IaAQAwARbPr14Lwds2Nq9k8fzqiCqafIIwAMAEuLS+cajr7E8QBgCYAMfneoe6zv4EYQCACXDm1In0ZmduuNabncmZUydGVNHks1kOAGACbG+IMzWifw4UhKvqk0m+kORKkudbaws7Xq8k/ybJa5J8McnrW2tP9LdUAIBuO31yXvDto8OsCH97a+1P93jtu5N849bPK5P8wta/AADchNnAo9Ov1ojvTfKLrbWW5CNVNVdVt7fWPt2n9wcAmDpmA4/WQTfLtSS/U1UXqurhXV6fT/Kp654/s3UNAIA9mA08WgddEb6vtXapqv5Kkg9U1cdba7933eu1y++0nRe2QvTDSXLXXXcdulgAgGliNvBoHWhFuLV2aevfZ5O8N8krdtzyTJI7r3t+R5JLu7zPo621hdbawrFjx45WMQDAlDAbeLT2DcJV9eKqesn24yTfleRjO257LMnfrateleRz+oMBAG7ObODROkhrxNcmee/VCWm5Lckvt9Z+u6r+UZK01t6a5P25OjrtE7k6Pu2HB1MuAMD0MBt4tOrqoIfhW1hYaMvLyyP5bAAAuqOqLuw8ByNxxDIAAB0lCAMA0EmCMAAAndSvk+UAADrF0ciTTxAGADgkRyNPB60RAACH5Gjk6SAIAwAckqORp4MgDABwSI5Gng6CMADAITkaeTrYLAcAcEiORp4OgjAAwBGcPjkv+E44rREAAHSSIAwAQCcJwgAAdJIgDABAJwnCAAB0kqkRAEAnLK2sGXfGDQRhAGDqLa2s5ey5i9nYvJIkWVvfyNlzF5NEGO4wrREAwNRbPL96LQRv29i8ksXzqyOqiHEgCAMAU+/S+sahrtMNWiMAgKmyWy/w8ble1nYJvcfneiOokHFhRRgAmBrbvcBr6xtp+Yte4G//68fSm5254d7e7EzOnDoxmkIZC4IwADA19uoF/tDHL+eRB+/J/FwvlWR+rpdHHrzHRrmO0xoBAEyNm/UCnz45L/hyA0EYAJhIeoG5VVojAICJoxeYfhCEAYCJoxeYftAaAQBMHL3A9IMVYQBg4uzV86sXmMMQhAGAiXPm1Am9wNwyrREAwMTZbn3YOTVCSwSHIQgDAGNlt7FouwVcvcDcKkEYABgb22PRtidCbI9FSyL00nd6hAGAsbHXWLTF86sjqohpJggDAGPjZmPRoN+0RgAAI+GIZEbNijAAMHSOSGYcCMIAwNA5IplxoDUCABg6RyQzDqwIAwBD54hkxoEgDAAMnSOSGQdaIwCAoXNEMuNAEAYARkIvMKOmNQIAgE4ShAEA6KQDB+Gqmqmqlap63y6vvb6qLlfVk1s//6C/ZQIAQH8dpkf4DUmeTvJVe7z+a621f3LrJQEAwOAdaEW4qu5I8j1J3jbYcgAAYDgO2hrx5iQ/luRLN7nn+6rqj6rq3VV15243VNXDVbVcVcuXL18+bK0AANA3+wbhqnptkmdbaxducttvJrm7tfY3kvzXJO/Y7abW2qOttYXW2sKxY8eOVDAAAPTDQVaE70vyQFV9MsmvJrm/qt55/Q2ttc+01v586+m/T/Itfa0SAAD6bN8g3Fo721q7o7V2d5KHknywtfaD199TVbdf9/SBXN1UBwAAY+vIJ8tV1ZuSLLfWHkvyz6rqgSTPJ/lsktf3pzwAABiMaq2N5IMXFhba8vLySD4bAIDuqKoLrbWFndePvCIMAEyHpZW1LJ5fzaX1jRyf6+XMqRM5fXJ+1GXBwAnCANBhSytrOXvuYjY2ryRJ1tY3cvbcxSQRhpl6Bz5iGQCYPovnV6+F4G0bm1eyeH51RBXB8AjCANBhl9Y3DnUdponWCADoiN16gY/P9bK2S+g9PtcbQYUwXIIwAEy5q33Af5SNzS9du7bdC/x93zKf91xYu6E9ojc7kzOnToyiVBgqrREAMMWWVtZy5jc+ekMI3raxeSUf+vjlPPLgPZmf66WSzM/18siD99goRydYEQaAKbZ4fjWbX9r7zIBL6xs5fXJe8KWTrAgDwBTbb9ObXmC6TBAGgCl2s6BbiV5gOk0QBoAJt7Sylvt+9oP5q2/8L7nvZz+YpZW1a6+dOXUis19Wu/7e33nVXVoi6DQ9wgAwwfY7GW476P70Y09lfWMzSfI1L5rNT/3tlwnBdJ4gDAAT7GYnw20HXZvhYHeCMABMmJ9YuphfefxTudJuPg0CuDlBGAAmyE8sXcw7P/In+95nGgTsz2Y5AJggv/L4p/a9x8lwcDBWhAFgzC2trGXx/GourW9k72aIq+PQjs/1cubUCT3BcACCMACMsZ1TIfYyU5X//shrhlQVTAetEQAwxnabCrGbH3jlnUOoBqaLFWEAGBPXt0BstzjsN/1hpio/8Mo7869O3zOkKmF6CMIAMAb2Ohjjq3uz1w7CuN78XC8ffuP9wy4TpoogDAAjtL0KvLbLyu/G5pV8xeyXpTc7c0N7hKkQ0B96hAFgRLZXgXcLwdvWv7iZRx68J/NzvVSurgQ/8uA9pkJAH1gRBoAROchGuONzPUckw4BYEQaAEdlvI5wWCBgsQRgARuRmxyBrgYDBE4QBYETOnDqR3uzMDdd6szN58/ffmw+/8X4hGAZMjzAAjMh20N05O1gAhuEQhAFghGyEg9ERhAGgj3Y7HU7QhfEkCANAn+x1OlwSYRjGkM1yANAnu80F3ti8ksXzqyOqCLgZQRgA+mSvucD7zQsGRkMQBoA+2Wsu8M3mBQOjIwgDQJ/sNRfY6XAwnmyWA4A+MRcYJosgDAB9ZC4wTA6tEQAAdJIgDABAJwnCAAB0kiAMAEAnCcIAAHSSIAwAQCcJwgAAdNKBg3BVzVTVSlW9b5fX/lJV/VpVfaKqHq+qu/tZJAAA9NthDtR4Q5Knk3zVLq/9/SR/1lr7hqp6KMnPJfn+PtQHAH2ztLLm1DfgmgOtCFfVHUm+J8nb9rjle5O8Y+vxu5N8R1XVrZcHAP2xtLKWs+cuZm19Iy3J2vpGzp67mKWVtVGXBozIQVsj3pzkx5J8aY/X55N8Kklaa88n+VySv7zzpqp6uKqWq2r58uXLRygXAI5m8fxqNjav3HBtY/NKFs+vjqgiYNT2DcJV9dokz7bWLtzstl2utRdcaO3R1tpCa23h2LFjhygTAG7NpfWNQ10Hpt9BVoTvS/JAVX0yya8mub+q3rnjnmeS3JkkVXVbkq9O8tk+1gkAt+T4XO9Q14Hpt28Qbq2dba3d0Vq7O8lDST7YWvvBHbc9luTvbT1+3dY9L1gRBoBROXPqRHqzMzdc683O5MypEyOqCBi1w0yNuEFVvSnJcmvtsSRvT/JLVfWJXF0JfqhP9QFAX2xPhzA1AthWo1q4XVhYaMvLyyP5bACmi7FowM1U1YXW2sLO60deEQaAcbA9Fm17IsT2WLQkwjBwU45YBmCiGYsGHJUgDMBEMxYNOCpBGICJZiwacFSCMAATzVg04KhslgNgbB1kGoSxaMBRCcIAjKXDTIM4fXJe8AUOTWsEAGPJNAhg0ARhAMaSaRDAoAnCAIwl0yCAQROEARhLpkEAg2azHABjyTQIYNAEYQCGamllLT/92FNZ39hMknzNi2bzU3/7ZbsGXNMggEEShAEYmp9Yuph3fuRPbrj2Z1/czJl3fzTJC8eiAQySHmEAhmJpZS3v2hGCt21eacaiAUNnRRiAgdo+HW5tn7FnxqIBwyYIAzAwO0+Huxlj0YBh0xoBwMDsdjrcbmZnylg0YOgEYQAG5iDtDi/+8pksvu7lNsoBQ6c1AoCBOT7X27M3eN5cYGDErAgDMDB7nQ735u+/Nx9+4/1CMDBSVoQBGBinwwHjTBAGYKCcDgeMK60RAAB0kiAMAEAnCcIAAHSSIAwAQCcJwgAAdJIgDABAJwnCAAB0kiAMAEAnCcIAAHSSIAwAQCcJwgAAdJIgDABAJ9026gIAGLyllbUsnl/NpfWNHJ/r5cypEzl9cn7UZQGMlCAMMOWWVtZy9tzFbGxeSZKsrW/k7LmLSSIMA52mNQJgyi2eX70WgrdtbF7J4vnVEVUEMB4EYYApd2l941DXAbpCEAaYcsfneoe6DtAVgjDAlDtz6kR6szM3XOvNzuTMqRMjqghgPNgsBzDltjfEmRoBcCNBGKADTp+cF3wBdhCEASaQucAAt27fHuGq+oqq+sOq+mhVPVVVP7PLPa+vqstV9eTWzz8YTLkAbM8FXlvfSMtfzAVeWlkbdWkAE+Ugm+X+PMn9rbWXJ7k3yaur6lW73PdrrbV7t37e1tcqAbjGXGCA/ti3NaK11pI8t/V0duunDbIoAPZmLjBAfxxofFpVzVTVk0meTfKB1trju9z2fVX1R1X17qq6s69VAnCNucAA/XGgINxau9JauzfJHUleUVXftOOW30xyd2vtbyT5r0nesdv7VNXDVbVcVcuXL1++lboBOstcYID+ONSBGq219SS/m+TVO65/prX251tP/32Sb9nj9x9trS201haOHTt2hHIBOH1yPo88eE/m53qpJPNzvTzy4D2mRgAc0r49wlV1LMlma229qnpJvjPJz+245/bW2qe3nj6Q5Om+VwrANeYCA9y6g8wRvj3JO6pqJldXkH+9tfa+qnpTkuXW2mNJ/llVPZDk+SSfTfL6QRUMAAD9UFeHQgzfwsJCW15eHslnAwDQHVV1obW2sPP6oXqEAQBgWgjCAAB00kF6hAG4RUsra1k8v5pL6xs5PtfLmVMnbHYDGDFBGGDAllbWcvbcxWvHIq+tb+TsuYtJIgwDjJDWCIABWzy/ei0Eb9vYvJLF86sjqgiARBAGGLhL6xuHug7AcAjCAAN2fK53qOsADIcgDDBgZ06dSG925oZrvdmZnDl1YkQVAZDYLAcwcNsb4kyNABgvgjDAEJw+OS/4AowZQRjgiMwGBphsgjDAEZgNDDD5bJYDOAKzgQEmnyAMcARmAwNMPkEY4AjMBgaYfIIwwBGYDQww+WyWAzgCs4EBJp8gDHBEZgMDTDatEQAAdJIgDABAJwnCAAB0kiAMAEAnCcIAAHSSIAwAQCcJwgAAdJI5wsBUW1pZc+gFALsShIGptbSylrPnLmZj80qSZG19I2fPXUwSYRgArRHA9Fo8v3otBG/b2LySxfOrI6oIgHFiRRiYGjvbINbWN3a979Ie1wHoFkEYmAq7tUFUkrbLvcfnekOtDYDxpDUCmAq7tUG0JLXjvt7sTM6cOjG0ugAYX4IwMBX2andoSebneqmtfx958B4b5QBIojUCmBJ79QTPz/Xy4TfeP4KKABh3VoSBqXDm1In0ZmduuKYNAoCbsSIMjL2DHIqx/dzhGQAclCAMjLXDHIpx+uS84AvAgWmNAMaaQzEAGBRBGBhre02DcCgGALdKEAbG2l6HXzgUA4BbJQgDY800CAAGxWY5YKyZBgHAoAjCwEgcZCTaNtMgABgEQRgYusOMRAOAQdEjDAydkWgAjIN9g3BVfUVV/WFVfbSqnqqqn9nlnr9UVb9WVZ+oqser6u5BFAtMByPRABgHB1kR/vMk97fWXp7k3iSvrqpX7bjn7yf5s9baNyT510l+rr9lAtPESDQAxsG+Qbhd9dzW09mtn7bjtu9N8o6tx+9O8h1VVX2rEpgqRqIBMA4O1CNcVTNV9WSSZ5N8oLX2+I5b5pN8Kklaa88n+VySv9zPQoHpcfrkfB558J7Mz/VSSebnennkwXtslANgqA40NaK1diXJvVU1l+S9VfVNrbWPXXfLbqu/O1eNU1UPJ3k4Se66664jlAtMCyPRABi1Q02NaK2tJ/ndJK/e8dIzSe5Mkqq6LclXJ/nsLr//aGttobW2cOzYsSMVDAAA/bDvinBVHUuy2Vpbr6peku/MCzfDPZbk7yX5b0lel+SDrbUXrAgD0+cwB2MAwDg5SGvE7UneUVUzubqC/OuttfdV1ZuSLLfWHkvy9iS/VFWfyNWV4IcGVjEwNhyMAcAkq1Et3C4sLLTl5eWRfDZw65ZW1vLPf/3J7PZfIfNzvXz4jfcPvygA2EVVXWitLey87ohl4NB+Yuli3vmRP9nzdQdjADAJHLEMHMrSylredZMQnDgYA4DJIAgDh7J4fvWFsxF3cDAGAJNAawRwUzunQqzt0/Yw15u1UQ6AiSAIA3vabSpEZZfTcrbMzlR++oGXDa0+ALgVWiOAPS2eX70Wgre17H6U5Iu/fCaLr3u51WAAJoYVYWBPe01/aLk6Is0hGgBMMkEY2NNePcHmBAMwDbRGAHs6c+pEerMzN1zrzc6YCgHAVLAiDOxpu93h+qkR2iAAmBaCMHBTp0/OC74ATCVBGDpk50xgq7sAdJkgDB2x20zgs+cuJokwDEAn2SwHHbHbTOCNzStZPL86oooAYLQEYeiIvWYC73UdAKadIAwdcXyud6jrADDtBGGYMksra7nvZz+Yv/rG/5L7fvaDWVpZS2ImMADsZLMcTJGDbIgzNQIArhKEYYrcbEPc9jxgwRcArtIaAVPEhjgAODhBGKaIDXEAcHCCMEwRG+IA4OD0CMME2e+IZBviAODgBGGYEAc9ItmGOAA4GK0RMCEckQwA/SUIw4QwEQIA+ktrBIyh3XqBj8/1srZL6DURAgCORhCGMbK0spaffuyprG9sXru23Qv8fd8yn/dcWLuhPcJECAA4OkEYxsDSylp+5jefyp99cXPX1zc2r+RDH7+cRx68x0QIAOgTQRhGbOc0iL1cWt8wEQIA+shmORix3aZB7EYvMAD0lyAMI3aQqQ96gQGg/wRhGLH9Vnq/5kWzeeTBe7REAECfCcIwYmdOnUhvduYF1+d6s3nz99+blZ/8LiEYAAbAZjkYse2QaxoEAAyXIAxjwDQIABg+rREAAHSSIAwAQCcJwgAAdJIgDABAJwnCAAB0kiAMAEAnCcIAAHSSIAwAQCftG4Sr6s6q+lBVPV1VT1XVG3a559uq6nNV9eTWz08OplwAAOiPg5ws93ySf9lae6KqXpLkQlV9oLX2xzvu+/3W2mv7XyIAAPTfvivCrbVPt9ae2Hr8hSRPJ3EWLAAAE+1QPcJVdXeSk0ke3+Xlb62qj1bVb1XVy/pQGwAADMxBWiOSJFX1lUnek+RHW2uf3/HyE0m+vrX2XFW9JslSkm/c5T0eTvJwktx1111HLhoAAG7VgVaEq2o2V0Pwu1pr53a+3lr7fGvtua3H708yW1Uv3eW+R1trC621hWPHjt1i6QAAcHQHmRpRSd6e5OnW2s/vcc/Xbd2XqnrF1vt+pp+FAgBAPx2kNeK+JD+U5GJVPbl17ceT3JUkrbW3Jnldkh+pqueTbCR5qLXWBlAvAAD0xb5BuLX2B0lqn3vekuQt/SoKAAAGzclyAAB0kiAMAEAnCcIAAHSSIAwAQCcJwgAAdJIgDABAJwnCAAB0kiAMAEAnCcIAAHSSIAwAQCcJwgAAdJIgDABAJwnCAAB0kiAMAEAnCcIAAHSSIAwAQCcJwgAAdJIgDABAJwnCAAB0kiAMAEAnCcIAAHSSIAwAQCcJwgAAdJIgDABAJwnCAAB0kiAMAEAnCcIAAHSSIAwAQCcJwgAAdJIgDABAJwnCAAB0kiAMAEAnCcIAAHSSIAwAQCcJwgAAdJIgDABAJwnCAAB00m2jLoDxsbSylsXzq7m0vpHjc72cOXUip0/Oj7osAICBEIRJcjUEnz13MRubV5Ika+sbOXvuYpIIwwDAVNIaQZJk8fzqtRC8bWPzShbPr46oIgCAwRKESZJcWt841HUAgEknCJMkOT7XO9R1AIBJJwiTJDlz6kR6szM3XOvNzuTMqRMjqggAYLBsliPJX2yIMzUCAOgKQZhrTp+cF3wBgM7YtzWiqu6sqg9V1dNV9VRVvWGXe6qq/m1VfaKq/qiqvnkw5QIAQH8cZEX4+ST/srX2RFW9JMmFqvpAa+2Pr7vnu5N849bPK5P8wta/AAAwlvZdEW6tfbq19sTW4y8keTrJzr+ff2+SX2xXfSTJXFXd3vdqAQCgTw41NaKq7k5yMsnjO16aT/Kp654/kxeG5VTVw1W1XFXLly9fPlylAADQRwcOwlX1lUnek+RHW2uf3/nyLr/SXnChtUdbawuttYVjx44drlIAAOijAwXhqprN1RD8rtbauV1ueSbJndc9vyPJpVsvDwAABuMgUyMqyduTPN1a+/k9bnssyd/dmh7xqiSfa619uo91AgBAXx1kasR9SX4oycWqenLr2o8nuStJWmtvTfL+JK9J8okkX0zyw/0vFQAA+mffINxa+4Ps3gN8/T0tyT/uV1EAADBoh5oaAQAA00IQBgCgkwRhAAA6SRAGAKCTBGEAADpJEAYAoJMEYQAAOkkQBgCgkwRhAAA6SRAGAKCTBGEAADpJEAYAoJMEYQAAOkkQBgCgkwRhAAA6SRAGAKCTBGEAADpJEAYAoJMEYQAAOkkQBgCgkwRhAAA6SRAGAKCTBGEAADpJEAYAoJMEYQAAOkkQBgCgkwRhAAA6SRAGAKCTBGEAADrptlEXMExLK2tZPL+aS+sbOT7Xy5lTJ3L65PyoywIAYAQ6E4SXVtZy9tzFbGxeSZKsrW/k7LmLSSIMAwB0UGdaIxbPr14Lwds2Nq9k8fzqiCoCAGCUOhOEL61vHOo6AADTrTNB+Phc71DXAQCYbp0JwmdOnUhvduaGa73ZmZw5dWJEFQEAMEqd2Sy3vSHO1AgAAJIOBeHkahgWfAEASDrUGgEAANcThAEA6CRBGACAThKEAQDoJEEYAIBOEoQBAOgkQRgAgE7aNwhX1X+oqmer6mN7vP5tVfW5qnpy6+cn+18mAAD010EO1PhPSd6S5Bdvcs/vt9Ze25eKAABgCPZdEW6t/V6Szw6hFgAAGJp+9Qh/a1V9tKp+q6pettdNVfVwVS1X1fLly5f79NEAAHB4/QjCTyT5+tbay5P8uyRLe93YWnu0tbbQWls4duxYHz4aAACO5paDcGvt862157Yevz/JbFW99JYrAwCAAbrlIFxVX1dVtfX4FVvv+ZlbfV8AABikfadGVNWvJPm2JC+tqmeS/FSS2SRprb01yeuS/EhVPZ9kI8lDrbU2sIoBAKAP9g3CrbUf2Of1t+TqeDUAAJgYNarF26q6nOR/HeFXX5rkT/tcDpPBd99Nvvfu8t13k++9mwb9vX99a+0FkxpGFoSPqqqWW2sLo66D4fPdd5Pvvbt8993ke++mUX3v/ZojDAAAE0UQBgCgkyYxCD866gIYGd99N/neu8t3302+924ayfc+cT3CAADQD5O4IgwAALdsYoJwVd1ZVR+qqqer6qmqesOoa2LwquorquoPq+qjW9/7z4y6JoanqmaqaqWq3jfqWhieqvpkVV2sqierannU9TAcVTVXVe+uqo9v/W/9t466Jgavqk5s/Wd9++fzVfWjQ/v8SWmNqKrbk9zeWnuiql6S5EKS0621Px5xaQzQ1vHdL26tPVdVs0n+IMkbWmsfGXFpDEFV/YskC0m+qrX22lHXw3BU1SeTLLTWzJLtkKp6R5Lfb629raq+PMmLWmvro66L4amqmSRrSV7ZWjvKWROHNjErwq21T7fWnth6/IUkTyeZH21VDFq76rmtp7NbP5Px/964JVV1R5LvSa/ubbcAAAIdSURBVPK2UdcCDFZVfVWSv5nk7UnSWvt/QnAnfUeS/z6sEJxMUBC+XlXdneRkksdHWwnDsPXn8SeTPJvkA60133s3vDnJjyX50qgLYehakt+pqgtV9fCoi2Eo/lqSy0n+41Y71Nuq6sWjLoqheyjJrwzzAycuCFfVVyZ5T5Ifba19ftT1MHittSuttXuT3JHkFVX1TaOuicGqqtcmeba1dmHUtTAS97XWvjnJdyf5x1X1N0ddEAN3W5JvTvILrbWTSf5vkjeOtiSGaasd5oEkvzHMz52oILzVI/qeJO9qrZ0bdT0M19afyX43yatHXAqDd1+SB7Z6RX81yf1V9c7RlsSwtNYubf37bJL3JnnFaCtiCJ5J8sx1f/F7d64GY7rju5M80Vr7P8P80IkJwlubpt6e5OnW2s+Puh6Go6qOVdXc1uNeku9M8vHRVsWgtdbOttbuaK3dnat/Kvtga+0HR1wWQ1BVL97aEJ2tP41/V5KPjbYqBq219r+TfKqqTmxd+o4kNsN3yw9kyG0RydU/RUyK+5L8UJKLW/2iSfLjrbX3j7AmBu/2JO/Y2kn6ZUl+vbVmlBZMr69N8t6rax+5Lckvt9Z+e7QlMST/NMm7tv5E/j+S/PCI62FIqupFSf5Wkn849M+elPFpAADQTxPTGgEAAP0kCAMA0EmCMAAAnSQIAwDQSYIwAACdJAgDANBJgjAAAJ0kCAMA0En/HzEfuWmKqvfDAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "fig, ax = plt.subplots(figsize=(12,8))\n", "ax.scatter(list(X_recovered[:, 0]), list(X_recovered[:, 1]))\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "请注意，第一主成分的投影轴基本上是数据集中的对角线。 当我们将数据减少到一个维度时，我们失去了该对角线周围的变化，所以在我们的再现中，一切都沿着该对角线。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "----\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 1 } ================================================ FILE: 第17章 潜在语义分析/17.LSA.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "LOWANK49Pi27" }, "source": [ "# 第十七章 潜在语义分析" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1.单词向量空间模型通过单词的向量表示文本的语义内容。以单词-文本矩阵$X$为输入，其中每一行对应一个单词，每一列对应一个文本，每一个元素表示单词在文本中的频数或权值（如TF-IDF）\n", "$$X = \\left[ \\begin{array} { c c c c } { x _ { 11 } } & { x _ { 12 } } & { \\cdots } & { x _ { 1 n } } \\\\ { x _ { 21 } } & { x _ { 22 } } & { \\cdots } & { x _ { 2 n } } \\\\ { \\vdots } & { \\vdots } & { } & { \\vdots } \\\\ { x _ { m 1 } } & { x _ { m 2 } } & { \\cdots } & { x _ { m n } } \\end{array} \\right]$$\n", "单词向量空间模型认为，这个矩阵的每一列向量是单词向量，表示一个文本，两个单词向量的内积或标准化内积表示文本之间的语义相似度。\n", "\n", "2.话题向量空间模型通过话题的向量表示文本的语义内容。假设有话题文本矩阵$$Y = \\left[ \\begin{array} { c c c c } { y _ { 11 } } & { y _ { 12 } } & { \\cdots } & { y _ { 1 n } } \\\\ { y _ { 21 } } & { y _ { 22 } } & { \\cdots } & { y _ { 2 n } } \\\\ { \\vdots } & { \\vdots } & { } & { \\vdots } \\\\ { y _ { k 1 } } & { y _ { k 2 } } & { \\cdots } & { y _ { k n } } \\end{array} \\right]$$\n", "其中每一行对应一个话题，每一列对应一个文本每一个元素表示话题在文本中的权值。话题向量空间模型认为，这个矩阵的每一列向量是话题向量，表示一个文本，两个话题向量的内积或标准化内积表示文本之间的语义相似度。假设有单词话题矩阵$T$\n", "$$T = \\left[ \\begin{array} { c c c c } { t _ { 11 } } & { t _ { 12 } } & { \\cdots } & { t _ { 1 k } } \\\\ { t _ { 21 } } & { t _ { 22 } } & { \\cdots } & { t _ { 2 k } } \\\\ { \\vdots } & { \\vdots } & { } & { \\vdots } \\\\ { t _ { m 1 } } & { t _ { m 2 } } & { \\cdots } & { t _ { m k } } \\end{array} \\right]$$ \n", "其中每一行对应一个单词，每一列对应一个话题，每一个元素表示单词在话题中的权值。\n", "\n", "给定一个单词文本矩阵$X$\n", "$$X = \\left[ \\begin{array} { c c c c } { x _ { 11 } } & { x _ { 12 } } & { \\cdots } & { x _ { 1 n } } \\\\ { x _ { 21 } } & { x _ { 22 } } & { \\cdots } & { x _ { 2 n } } \\\\ { \\vdots } & { \\vdots } & { } & { \\vdots } \\\\ { x _ { m 1 } } & { x _ { m 2 } } & { \\cdots } & { x _ { m n } } \\end{array} \\right]$$\n", "\n", "潜在语义分析的目标是，找到合适的单词-话题矩阵$T$与话题文本矩阵$Y$，将单词文本矩阵$X$近似的表示为$T$与$Y$的乘积形式。\n", "$$X \\approx T Y$$\n", "\n", "等价地，潜在语义分析将文本在单词向量空间的表示X通过线性变换$T$转换为话题向量空间中的表示$Y$。\n", "\n", "潜在语义分析的关键是对单词-文本矩阵进行以上的矩阵因子分解（话题分析）\n", "\n", "3.潜在语义分析的算法是奇异值分解。通过对单词文本矩阵进行截断奇异值分解，得到\n", "$$X \\approx U _ { k } \\Sigma _ { k } V _ { k } ^ { T } = U _ { k } ( \\Sigma _ { k } V _ { k } ^ { T } )$$\n", "\n", "矩阵$U_k$表示话题空间，矩阵$( \\Sigma _ { k } V _ { k } ^ { T } )$是文本在话题空间的表示。\n", "\n", "4.非负矩阵分解也可以用于话题分析。非负矩阵分解将非负的单词文本矩阵近似分解成两个非负矩阵$W$和$H$的乘积，得到\n", "$$X \\approx W H$$\n", "\n", "矩阵$W$表示话题空间，矩阵$H$是文本在话题空间的表示。\n", "\n", "非负矩阵分解可以表为以下的最优化问题：\n", "$$\\left. \\begin{array} { l } { \\operatorname { min } _ { W , H } \\| X - W H \\| ^ { 2 } } \\\\ { \\text { s.t. } W , H \\geq 0 } \\end{array} \\right.$$\n", "非负矩阵分解的算法是迭代算法。乘法更新规则的迭代算法，交替地对$W$和$H$进行更新。本质是梯度下降法，通过定义特殊的步长和非负的初始值，保证迭代过程及结果的矩阵$W$和$H$均为非负。\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "0WD7XVWRPkX1" }, "source": [ "-----\n", "**LSA** 是一种无监督学习方法，主要用于文本的话题分析，其特点是通过矩阵分解发现文本与单词之间的基于话题的语义关系。也称为潜在语义索引（Latent semantic indexing, LSI）。\n", "\n", "LSA 使用的是非概率的话题分析模型。将文本集合表示为**单词-文本矩阵**，对单词-文本矩阵进行**奇异值分解**，从而得到话题向量空间，以及文本在话题向量空间的表示。\n", "\n", "**非负矩阵分解**（non-negative matrix factorization, NMF）是另一种矩阵的因子分解方法，其特点是分解的矩阵非负。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "P1sWKgTGQ7r-" }, "source": [ "## 单词向量空间 \n", "word vector space model" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "CqXj1777RM8y" }, "source": [ "给定一个文本，用一个向量表示该文本的”语义“， 向量的**每一维对应一个单词**，其数值为该单词在该文本中出现的频数或权值；基本假设是文本中所有单词的出现情况表示了文本的语义内容，文本集合中的每个文本都表示为一个向量，存在于一个向量空间；向量空间的度量，如内积或标准化**内积**表示文本之间的**相似度**。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "3HVCXf6CSmTT" }, "source": [ "给定一个含有$n$个文本的集合$D=({d_{1}, d_{2},...,d_{n}})$，以及在所有文本中出现的$m$个单词的集合$W=({w_{1},w_{2},...,w_{m}})$. 将单词在文本的出现的数据用一个单词-文本矩阵（word-document matrix）表示，记作$X$:\n", "\n", "$\n", "X = \\begin{bmatrix}\n", "x_{11} & x_{12}& x_{1n}& \\\\ \n", "x_{21}& x_{22}& x_{2n}& \\\\ \n", "\\vdots & \\vdots & \\vdots & \\\\ \n", "x_{m1}& x_{m2}& x_{mn}& \n", "\\end{bmatrix}\n", "$\n", "\n", "这是一个$m*n$矩阵，元素$x_{ij}$表示单词$w_{i}$在文本$d_{j}$中出现的频数或权值。由于单词的种类很多，而每个文本中出现单词的种类通常较少，所有单词-文本矩阵是一个稀疏矩阵。\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "K2ncB3cde1Ab" }, "source": [ "权值通常用单词**频率-逆文本率**（term frequency-inverse document frequency, TF-IDF）表示：\n", "\n", "$TF-IDF(t, d ) = TF(t, d) * IDF(t)$, \n", "\n", "其中，$TF(t,d)$为单词$t$在文本$d$中出现的概率，$IDF(t)$是逆文本率，用来衡量单词$t$对表示语义所起的重要性， \n", "\n", "$IDF(t) = log(\\frac{len(D)}{len(t \\in D) + 1})$." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "bpu7MycIgu65" }, "source": [ "单词向量空间模型的优点是**模型简单，计算效率高**。因为单词向量通常是稀疏的，单词向量空间模型也有一定的局限性，体现在内积相似度未必能够准确表达两个文本的语义相似度上。因为自然语言的单词具有一词多义性（polysemy）及多词一义性（synonymy）。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "ns5wncZohn-z" }, "source": [ "## 话题向量空间" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "mmZpPHIdhrAy" }, "source": [ "**1. 话题向量空间**：\n", "\n", "给定一个含有$n$个文本的集合$D=({d_{1}, d_{2},...,d_{n}})$，以及在所有文本中出现的$m$个单词的集合$W=({w_{1},w_{2},...,w_{m}})$. 可以获得其单词-文本矩阵$X$: \n", "\n", "$\n", "X = \\begin{bmatrix}\n", "x_{11} & x_{12}& x_{1n}& \\\\ \n", "x_{21}& x_{22}& x_{2n}& \\\\ \n", "\\vdots & \\vdots & \\vdots & \\\\ \n", "x_{m1}& x_{m2}& x_{mn}& \n", "\\end{bmatrix}\n", "$\n", "\n", "\n", "假设所有文本共含有$k$个话题。假设每个话题由一个定义在单词集合$W$上的$m$维向量表示，称为话题向量，即： \n", "$t_{l} = \\begin{bmatrix}\n", "t_{1l}\\\\ \n", "t_{2l}\\\\ \n", "\\vdots \\\\ \n", "t_{ml}\\end{bmatrix}, l=1,2,...,k$\n", "\n", "其中$t_{il}$单词$w_{i}$在话题$t_{l}$的权值，$i=1,2,...,m$, 权值越大，该单词在该话题中的重要程度就越高。这$k$个话题向量 $t_{1},t_{2},...,t_{k}$张成一个话题向量空间（topic vector space）， 维数为$k$.**话题向量空间是单词向量空间的一个子空间**。\n", "\n", "话题向量空间$T$： \n", "\n", "\n", "$\n", "T = \\begin{bmatrix}\n", "t_{11} & t_{12}& t_{1k}& \\\\ \n", "t_{21}& t_{22}& t_{2k}& \\\\ \n", "\\vdots & \\vdots & \\vdots & \\\\ \n", "t_{m1}& t_{m2}& t_{mk}& \n", "\\end{bmatrix}\n", "$ \n", "\n", "矩阵$T$,称为**单词-话题矩阵**。 $T = [t_{1}, t_{2}, ..., t_{k}]$" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Oc1c3JcKlTBD" }, "source": [ "**2. 文本在话题向量空间中的表示** ：\n", "\n", "考虑文本集合$D$的文本$d_{j}$, 在单词向量空间中由一个向量$x_{j}$表示，将$x_{j}$投影到话题向量空间$T$中，得到话题向量空间的一个向量$y_{j}$, $y_{j}$是一个$k$维向量： \n", "\n", "$y_{j} = \\begin{bmatrix}\n", "y_{1j}\\\\ \n", "y_{2j}\\\\ \n", "\\vdots \\\\ \n", "y_{kj}\\end{bmatrix}, j=1,2,...,n$ \n", "\n", "其中，$y_{lj}$是文本$d_{j}$在话题$t_{l}$中的权值， $l = 1,2,..., k$, 权值越大，该话题在该文本中的重要程度就越高。 \n", "\n", "矩阵$Y$ 表示话题在文本中出现的情况，称为话题-文本矩阵（topic-document matrix）,记作： \n", "\n", "$\n", "Y = \\begin{bmatrix}\n", "y_{11} & y_{12}& y_{1n}& \\\\ \n", "y_{21}& y_{22}& y_{2n}& \\\\ \n", "\\vdots & \\vdots & \\vdots & \\\\ \n", "y_{k1}& y_{k2}& y_{kn}& \n", "\\end{bmatrix}\n", "$ \n", "\n", "也可写成： $Y = [y_{1}, y_{2} ..., y_{n}]$" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "YcU3xwYindTo" }, "source": [ "**3. 从单词向量空间到话题向量空间的线性变换**： \n", "\n", "如此，单词向量空间的文本向量$x_{j}$可以通过他在话题空间中的向量$y_{j}$近似表示，具体地由$k$个话题向量以$y_{j}$为系数的线性组合近似表示： \n", "\n", "$x_{j} = y_{1j}t_{1} + y_{2j}t_{2} + ... + y_{yj}t_{k}, j = 1,2,..., n$ \n", "\n", "所以，单词-文本矩阵$X$可以近似的表示为单词-话题矩阵$T$与话题-文本矩阵$Y$的乘积形式。\n", "\n", "$X \\approx TY$ \n", "\n", "直观上，潜在语义分析是将单词向量空间的表示通过线性变换转换为在话题向量空间中的表示。这个线性变换由矩阵因子分解式的形式体现。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Cu4JekfXFMqs" }, "source": [ "### 潜在语义分析算法 " ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "0awNXCy1Gw0K" }, "source": [ "潜在语义分析利用矩阵奇异值分解，具体地，对单词-文本矩阵进行奇异值分解，将其左矩阵作为话题向量空间，将其对角矩阵与右矩阵的乘积作为文本在话题向量空间的表示。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "otq3HMu5HVoK" }, "source": [ "给定一个含有$n$个文本的集合$D=({d_{1}, d_{2},...,d_{n}})$，以及在所有文本中出现的$m$个单词的集合$W=({w_{1},w_{2},...,w_{m}})$. 可以获得其单词-文本矩阵$X$: \n", "$\n", "X = \\begin{bmatrix}\n", "x_{11} & x_{12}& x_{1n}& \\\\ \n", "x_{21}& x_{22}& x_{2n}& \\\\ \n", "\\vdots & \\vdots & \\vdots & \\\\ \n", "x_{m1}& x_{m2}& x_{mn}& \n", "\\end{bmatrix}\n", "$\n", "\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "mwNGRDgrHmmV" }, "source": [ "**截断奇异值分解**：\n", "\n", "潜在语义分析根据确定的话题数$k$对单词-文本矩阵$X$进行截断奇异值分解： \n", "\n", "$\n", "X \\approx U_{k}\\Sigma _{k}V_{k}^{T} = \\begin{bmatrix}\n", "\\mu _{1} & \\mu _{2}& \\cdots & \\mu _{k}\n", "\\end{bmatrix}\\begin{bmatrix}\n", "\\sigma_{1} & 0& 0& 0\\\\ \n", " 0& \\sigma_{2}& 0& 0\\\\ \n", " 0& 0& \\ddots & 0\\\\ \n", " 0& 0& 0& \\sigma_{k}\n", "\\end{bmatrix}\\begin{bmatrix}\n", "v_{1}^{T}\\\\ \n", "v_{2}^{T}\\\\ \n", "\\vdots \\\\ \n", "v_{k}^{T}\\end{bmatrix}\n", "$\n", "\n", "矩阵$U_{k}$的每一个列向量 $u_{1}, u_{2},..., u_{k}$ 表示一个话题，称为**话题向量**。由这 $k$ 个话题向量张成一个子空间： \n", "\n", "$\n", "U_{k} = \\begin{bmatrix}\n", "u_{1} & u_{2}& \\cdots & u_{k}\n", "\\end{bmatrix}\n", "$\n", "\n", "称为**话题向量空间**。 \n", "\n", "综上， 可以通过对单词-文本矩阵的奇异值分解进行潜在语义分析： \n", "\n", "$ X \\approx U_{k} \\Sigma_{k} V_{k}^{T} = U_{k}(\\Sigma_{k}V_{k}^{T})$ \n", "\n", "得到话题空间 $U_{k}$ ， 以及文本在话题空间的表示($\\Sigma_{k}V_{k}^{T}$). " ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "UTNHyq8mK8l5" }, "source": [ "### 非负矩阵分解算法" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "RQvqaMDYK_jf" }, "source": [ "非负矩阵分解也可以用于话题分析。对单词-文本矩阵进行非负矩阵分解，将**其左矩阵作为话题向量空间**，将其**右矩阵作为文本在话题向量空间的表示**。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "ApM8tE3MLqpP" }, "source": [ "#### 非负矩阵分解 " ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "glMwmkiwLyIn" }, "source": [ "若一个矩阵的索引元素非负，则该矩阵为非负矩阵。若$X$是非负矩阵，则： $X >= 0$. \n", "\n", "给定一个非负矩阵$X$, 找到两个非负矩阵$W >= 0$ 和 $H>= 0$, 使得： \n", "\n", "$ X \\approx WH$\n", "\n", "即非负矩阵$X$分解为两个非负矩阵$W$和$H$的乘积形式，成为非负矩阵分解。因为$WH$与$X$完全相等很难实现，所以只要求近似相等。 \n", "\n", "假设非负矩阵$X$是$m\\times n$矩阵，非负矩阵$W$和$H$分别为 $m\\times k$ 矩阵和 $k\\times n$ 矩阵。假设 $k < min(m, n)$ 即$W$ 和 $H$ 小于原矩阵 $X$, 所以非负矩阵分解是对原数据的压缩。\n", "\n", "称 $W$ 为基矩阵， $H$ 为系数矩阵。非负矩阵分解旨在用较少的基向量，系数向量来表示为较大的数据矩阵。\n", "\n", "令 $W = \\begin{bmatrix}\n", "w_{1} & w_{2}& \\cdots& w_{k} \n", "\\end{bmatrix}$\n", "为话题向量空间， $w_{1}, w_{2}, ..., w_{k}$ 表示文本集合的 $k$ 个话题， 令 $H = \\begin{bmatrix}\n", "h_{1} & h_{2}& \\cdots& h_{n} \n", "\\end{bmatrix}$\n", "为文本在话题向量空间的表示， $h_{1}, h_{2},..., h_{n}$ 表示文本集合的 $n$ 个文本。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "1DcvVSR0N_CF" }, "source": [ "##### 算法" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "hvZyHT85O5qt" }, "source": [ "非负矩阵分解可以形式化为最优化问题求解。可以利用平方损失或散度来作为损失函数。\n", "\n", "目标函数 $|| X - WH ||^{2}$ 关于 $W$ 和 $H$ 的最小化，满足约束条件 $W, H >= 0$, 即： \n", "\n", "$\\underset{W,H}{min} || X - WH ||^{2}$ \n", "\n", "\n", "$s.t. W, H >= 0$" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "-zIGS1AEQdWp" }, "source": [ "乘法更新规则： \n", "\n", "\n", "$W_{il} \\leftarrow W_{il}\\frac{(XH^{T})_{il}}{(WHH^{T})_{il}}$ （17.33）\n", "\n", "\n", "$H_{lj} \\leftarrow H_{lj}\\frac{(W^{T}X)_{lj}}{(W^{T}WH)_{lj}}$ （17.34）\n", "\n", "\n", "选择初始矩阵 $W$ 和 $H$ 为非负矩阵，可以保证迭代过程及结果的矩阵 $W$ 和 $H$ 非负。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "MeiA0REkRpRi" }, "source": [ "**算法 17.1 （非负矩阵分解的迭代算法）**\n", "\n", "输入： 单词-文本矩阵 $X >= 0$, 文本集合的话题个数 $k$, 最大迭代次数 $t$； \n", "输出： 话题矩阵 $W$, 文本表示矩阵 $H$。 \n", "\n", "**1)**. 初始化\n", "\n", "$W>=0$, 并对 $W$ 的每一列数据归一化； \n", "$H>=0$；\n", "\n", "**2)**. 迭代 \n", "\n", "对迭代次数由1到$t$执行下列步骤： \n", "a. 更新$W$的元素，对 $l$ 从1到 $k,i$从1到$m$按(17.33)更新 $W_{il}$; \n", "a. 更新$H$的元素，对 $l$ 从1到 $k,j$从1到$m$按(17.34)更新 $H_{lj}$; " ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "rIw6a0HITg08" }, "source": [ "### 图例 17.1" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": {}, "colab_type": "code", "id": "0hPH9VEMPVGu" }, "outputs": [], "source": [ "import numpy as np\n", "from sklearn.decomposition import TruncatedSVD" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 125 }, "colab_type": "code", "id": "kjHirYzQWItl", "outputId": "0e6f2615-6a0b-4c4f-e74c-559727519eab" }, "outputs": [ { "data": { "text/plain": [ "array([[2, 0, 0, 0],\n", " [0, 2, 0, 0],\n", " [0, 0, 1, 0],\n", " [0, 0, 2, 3],\n", " [0, 0, 0, 1],\n", " [1, 2, 2, 1]])" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X = [[2, 0, 0, 0], [0, 2, 0, 0], [0, 0, 1, 0], [0, 0, 2, 3], [0, 0, 0, 1], [1, 2, 2, 1]]\n", "X = np.asarray(X);X" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": {}, "colab_type": "code", "id": "I2yFnNJKWcPP" }, "outputs": [], "source": [ "# 奇异值分解\n", "U,sigma,VT=np.linalg.svd(X)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 233 }, "colab_type": "code", "id": "ollDH_QNXAdY", "outputId": "8d599d82-4bae-4047-941c-8ca524142349" }, "outputs": [ { "data": { "text/plain": [ "array([[-7.84368672e-02, -2.84423033e-01, 8.94427191e-01,\n", " -2.15138396e-01, -6.45002451e-02, -2.50012770e-01],\n", " [-1.56873734e-01, -5.68846066e-01, -4.47213595e-01,\n", " -4.30276793e-01, -1.29000490e-01, -5.00025540e-01],\n", " [-1.42622354e-01, 1.37930417e-02, -1.25029761e-16,\n", " 6.53519444e-01, 3.88575115e-01, -6.33553733e-01],\n", " [-7.28804669e-01, 5.53499910e-01, -2.24565656e-16,\n", " -1.56161345e-01, -3.23288048e-01, -1.83248673e-01],\n", " [-1.47853320e-01, 1.75304609e-01, 8.49795536e-18,\n", " -4.87733411e-01, 8.40863653e-01, 4.97204799e-02],\n", " [-6.29190197e-01, -5.08166890e-01, -1.60733896e-16,\n", " 2.81459486e-01, 1.29000490e-01, 5.00025540e-01]])" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "U" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "lmxB_JViXFAF", "outputId": "9cdcba5c-74f8-42e4-9a4b-fad1c06b83cd" }, "outputs": [ { "data": { "text/plain": [ "array([4.47696617, 2.7519661 , 2. , 1.17620428])" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sigma" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 161 }, "colab_type": "code", "id": "AiXURUScXMsj", "outputId": "d6fc0576-8bf1-491e-caed-02035079f0d3" }, "outputs": [ { "data": { "text/plain": [ "array([[-1.75579600e-01, -3.51159201e-01, -6.38515454e-01,\n", " -6.61934313e-01],\n", " [-3.91361272e-01, -7.82722545e-01, 3.79579831e-02,\n", " 4.82432341e-01],\n", " [ 8.94427191e-01, -4.47213595e-01, -2.23998094e-16,\n", " 5.45344065e-17],\n", " [-1.26523351e-01, -2.53046702e-01, 7.68672366e-01,\n", " -5.73674125e-01]])" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "VT" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 53 }, "colab_type": "code", "id": "DKOxld5lXRCK", "outputId": "0832796a-9952-4f39-e1ef-79347c495d16" }, "outputs": [ { "data": { "text/plain": [ "TruncatedSVD(algorithm='randomized', n_components=3, n_iter=7, random_state=42,\n", " tol=0.0)" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 截断奇异值分解\n", "\n", "svd = TruncatedSVD(n_components=3, n_iter=7, random_state=42)\n", "svd.fit(X) " ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "btnGrF0LXzZI", "outputId": "ba85127a-4fa6-4092-828c-9036c47f82f6" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0.39945801 0.34585056 0.18861789]\n" ] } ], "source": [ "print(svd.explained_variance_ratio_)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "F1hSe5NxX1zw", "outputId": "b0d0b87d-195b-4653-a857-48ff1eca887a" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.933926460028446\n" ] } ], "source": [ "print(svd.explained_variance_ratio_.sum())" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "cV4L2i9WX30R", "outputId": "6c313215-d095-41b2-a384-3dc1575729a2" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[4.47696617 2.7519661 2. ]\n" ] } ], "source": [ "print(svd.singular_values_)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "4CbG9kJXictK" }, "source": [ "#### 非负矩阵分解" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": {}, "colab_type": "code", "id": "KcA2Rd4Df_DE" }, "outputs": [], "source": [ "def inverse_transform(W, H):\n", " # 重构\n", " return W.dot(H)\n", "\n", "def loss(X, X_):\n", " #计算重构误差\n", " return ((X - X_) * (X - X_)).sum()" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": {}, "colab_type": "code", "id": "yRXZt6CfYPJq" }, "outputs": [], "source": [ "# 算法 17.1\n", "\n", "class MyNMF:\n", " def fit(self, X, k, t):\n", " m, n = X.shape\n", " \n", " W = np.random.rand(m, k)\n", " W = W/W.sum(axis=0)\n", " \n", " H = np.random.rand(k, n)\n", " \n", " i = 1\n", " while i < t:\n", " \n", " W = W * X.dot(H.T) / W.dot(H).dot(H.T)\n", " \n", " H = H * (W.T).dot(X) / (W.T).dot(W).dot(H)\n", " \n", " i += 1\n", " \n", " return W, H" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": {}, "colab_type": "code", "id": "zc1IFBBIajXk" }, "outputs": [], "source": [ "model = MyNMF()\n", "W, H = model.fit(X, 3, 200)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 125 }, "colab_type": "code", "id": "EYo7JyXXbNpJ", "outputId": "716dbb03-5229-4c93-a75b-81d07ff4be85" }, "outputs": [ { "data": { "text/plain": [ "array([[0.00000000e+00, 4.27327705e-01, 6.30117924e-27],\n", " [5.11680721e-97, 8.57828102e-01, 0.00000000e+00],\n", " [2.97520805e-88, 2.39454414e-18, 4.36332453e-01],\n", " [2.15653741e+00, 3.38756557e-21, 8.38350315e-01],\n", " [7.36106818e-01, 1.00339294e-54, 8.72892573e-38],\n", " [6.78344810e-01, 1.07009504e+00, 8.57259947e-01]])" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "W" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 71 }, "colab_type": "code", "id": "1cEFDsgXbnXZ", "outputId": "e3c8eaf0-bcd8-48f5-edee-57f643b07fed" }, "outputs": [ { "data": { "text/plain": [ "array([[7.14647214e-10, 1.01233436e-03, 3.76097657e-02, 1.35755597e+00],\n", " [9.30509415e-01, 1.86788842e+00, 1.16682319e-02, 4.54479182e-03],\n", " [4.95440453e-03, 6.18432747e-04, 2.28890170e+00, 8.61836630e-02]])" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "H" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 125 }, "colab_type": "code", "id": "JFqGat0JdVlL", "outputId": "567c82ff-997f-49e7-b829-f9d4c828f9c9" }, "outputs": [ { "data": { "text/plain": [ "array([[3.97632453e-01, 7.98200474e-01, 4.98615876e-03, 1.94211546e-03],\n", " [7.98217125e-01, 1.60232718e+00, 1.00093372e-02, 3.89865014e-03],\n", " [2.16176748e-03, 2.69842277e-04, 9.98722093e-01, 3.76047290e-02],\n", " [4.15352814e-03, 2.70160021e-03, 2.00000833e+00, 2.99987233e+00],\n", " [5.26056687e-10, 7.45186228e-04, 2.76848049e-02, 9.99306203e-01],\n", " [9.99980721e-01, 2.00003500e+00, 2.00018226e+00, 9.99636205e-01]])" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 重构\n", "X_ = inverse_transform(W, H);X_" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "FmXCjjnyfcfY", "outputId": "819c8029-12d3-4344-e5af-b352b51507a3" }, "outputs": [ { "data": { "text/plain": [ "4.002356735601103" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 重构误差\n", "\n", "loss(X, X_)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "dGu-tGDxhEcf" }, "source": [ "### 使用 sklearn 计算" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "colab": {}, "colab_type": "code", "id": "sLN4FLmvb6tt" }, "outputs": [], "source": [ "from sklearn.decomposition import NMF\n", "model = NMF(n_components=3, init='random', max_iter=200, random_state=0)\n", "W = model.fit_transform(X)\n", "H = model.components_" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 125 }, "colab_type": "code", "id": "Fm5W6xQ0b_jl", "outputId": "d2ae79f6-0fea-47ba-a2ba-1742b78f71b1" }, "outputs": [ { "data": { "text/plain": [ "array([[0. , 0.53849498, 0. ],\n", " [0. , 1.07698996, 0. ],\n", " [0.69891361, 0. , 0. ],\n", " [1.39782972, 0. , 1.97173859],\n", " [0. , 0. , 0.65783848],\n", " [1.39783002, 1.34623756, 0.65573258]])" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "W" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 71 }, "colab_type": "code", "id": "wheFi8ZwcCY9", "outputId": "9dd57216-889a-4cc4-ccb8-82e790595d59" }, "outputs": [ { "data": { "text/plain": [ "array([[0.00000000e+00, 0.00000000e+00, 1.43078959e+00, 1.71761682e-03],\n", " [7.42810976e-01, 1.48562195e+00, 0.00000000e+00, 3.30264644e-04],\n", " [0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.52030365e+00]])" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "H" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 125 }, "colab_type": "code", "id": "9hfj3bRXgHRb", "outputId": "3be1a4d8-a160-4200-d4e7-f5c92f2aa1af" }, "outputs": [ { "data": { "text/plain": [ "array([[3.99999983e-01, 7.99999966e-01, 0.00000000e+00, 1.77845853e-04],\n", " [7.99999966e-01, 1.59999993e+00, 0.00000000e+00, 3.55691707e-04],\n", " [0.00000000e+00, 0.00000000e+00, 9.99998311e-01, 1.20046577e-03],\n", " [0.00000000e+00, 0.00000000e+00, 2.00000021e+00, 3.00004230e+00],\n", " [0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00011424e+00],\n", " [1.00000003e+00, 2.00000007e+00, 2.00000064e+00, 9.99758185e-01]])" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X__ = inverse_transform(W, H);X__" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "-iALOTKfgKzP", "outputId": "80c73327-dd36-403f-aa20-d4e2d3f796a7" }, "outputs": [ { "data": { "text/plain": [ "4.000001672582457" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "loss(X, X__)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "本章代码来源：https://github.com/hktxt/Learn-Statistical-Learning-Method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "LSA.ipynb", "provenance": [], "version": "0.3.2" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 1 } ================================================ FILE: 第18章 概率潜在语义分析/18.PLSA.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "0I-Es-jovJzm" }, "source": [ "# 第18章 概率潜在语义分析" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1.概率潜在语义分析是利用概率生成模型对文本集合进行话题分析的方法。概率潜在语义分析受潜在语义分析的启发提出两者可以通过矩阵分解关联起来。\n", "\n", "给定一个文本集合，通过概率潜在语义分析，可以得到各个文本生成话题的条件概率分布，以及各个话题生成单词的条件概率分布。\n", "\n", "概率潜在语义分析的模型有生成模型，以及等价的共现模型。其学习策略是观测数据的极大似然估计，其学习算法是EM算法。\n", "\n", "2.生成模型表示文本生成话题，话题生成单词从而得到单词文本共现数据的过程；假设每个文本由一个话题分布决定，每个话题由一个单词分布决定。单词变量$w$与文本变量$d$是观测变量话题变量$z$是隐变量。生成模型的定义如下：\n", "$$P ( T ) = \\prod _ { ( w , d ) } P ( w , d ) ^ { n ( w , d ) }$$\n", "\n", "$$P ( w , d ) = P ( d ) P ( w | d ) = P ( d ) \\sum _ { \\alpha } P ( z | d ) P ( w | z )$$\n", "3.共现模型描述文本单词共现数据拥有的模式。共现模型的定义如下：\n", "$$P ( T ) = \\prod _ { ( w , d ) } P ( w , d ) ^ { n ( w , d ) }$$\n", "\n", "$$P ( w , d ) = \\sum _ { z \\in Z } P ( z ) P ( w | z ) P ( d | z )$$\n", "\n", "4.概率潜在语义分析的模型的参数个数是$O ( M \\cdot K + N \\cdot K )$。现实中$K \\ll M$，所以概率潜在语义分析通过话题对数据进行了更简洁地表示，实现了数据压缩。\n", "\n", "5.模型中的概率分布$P ( w | d )$可以由参数空间中的单纯形表示。$M$维参数空间中，单词单纯形表示所有可能的文本的分布，在其中的话题单纯形表示在$K$个话题定义下的所有可能的文本的分布。话题单纯形是单词单纯形的子集，表示潜在语义空间。\n", "\n", "6.概率潜在语义分析的学习通常采用EM算法通过迭代学习模型的参数，$P ( w | z )$\n", "和$P ( z| d )$，而$P（d）$可直接统计得出。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "EeHek0KrvNbO" }, "source": [ "----\n", "概率潜在语义分析（probabilistic latent semantic analysis, PLSA）,也称概率潜在语义索引（probabilistic latent semantic indexing, PLSI）,是一种利用概率生成模型对文本集合进行话题分析的无监督学习方法。\n", "\n", "模型最大特点是用隐变量表示话题，整个模型表示文本生成话题，话题生成单词，从而得到单词-文本共现数据的过程；假设每个文本由一个话题分布决定，每个话题由一个单词分布决定。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "ZpnNY-eRwjq3" }, "source": [ "### **18.1.2 生成模型**\n", "\n", "假设有单词集合 $W = $ {$w_{1}, w_{2}, ..., w_{M}$}， 其中M是单词个数；文本（指标）集合$D = $ {$d_{1}, d_{2}, ..., d_{N}$}, 其中N是文本个数；话题集合$Z = $ {$z_{1}, z_{2}, ..., z_{K}$}，其中$K$是预先设定的话题个数。随机变量 $w$ 取值于单词集合；随机变量 $d$ 取值于文本集合，随机变量 $z$ 取值于话题集合。概率分布 $P(d)$、条件概率分布 $P(z|d)$、条件概率分布 $P(w|z)$ 皆属于多项分布，其中 $P(d)$ 表示生成文本 $d$ 的概率，$P(z|d)$ 表示文本 $d$ 生成话题 $z$ 的概率，$P(w|z)$ 表示话题 $z$ 生成单词 $w$ 的概率。\n", "\n", " 每个文本 $d$ 拥有自己的话题概率分布 $P(z|d)$，每个话题 $z$ 拥有自己的单词概率分布 $P(w|z)$；也就是说**一个文本的内容由其相关话题决定，一个话题的内容由其相关单词决定**。\n", " \n", " 生成模型通过以下步骤生成文本·单词共现数据： \n", " （1）依据概率分布 $P(d)$，从文本（指标）集合中随机选取一个文本 $d$ , 共生成 $N$ 个文本；针对每个文本，执行以下操作； \n", " （2）在文本$d$ 给定条件下，依据条件概率分布 $P(z|d)$, 从话题集合随机选取一个话题 $z$, 共生成 $L$ 个话题，这里 $L$ 是文本长度； \n", " （3）在话题 $z$ 给定条件下，依据条件概率分布 $P(w|z)$ , 从单词集合中随机选取一个单词 $w$. \n", " \n", " 注意这里为叙述方便，假设文本都是等长的，现实中不需要这个假设。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "_YwFFCuCgugI" }, "source": [ "生成模型中， 单词变量 $w$ 与文本变量 $d$ 是观测变量， 话题变量 $z$ 是隐变量， 也就是说模型生成的是单词-话题-文本三元组合 ($w, z ,d$)的集合， 但观测到的单词-文本二元组 （$w, d$）的集合， 观测数据表示为单词-文本矩阵 $T$的形式，矩阵 $T$ 的行表示单词，列表示文本， 元素表示单词-文本对（$w, d$）的出现次数。 \n", "\n", "从数据的生成过程可以推出，文本-单词共现数据$T$的生成概率为所有单词-文本对($w,d$)的生成概率的乘积： \n", "\n", "$P(T) = \\prod_{w,d}P(w,d)^{n(w,d)}$ \n", "\n", "这里 $n(w,d)$ 表示 ($w,d$)的出现次数，单词-文本对出现的总次数是 $N*L$。 每个单词-文本对（$w,d$）的生成概率由一下公式决定： \n", "\n", "$P(w,d) = P(d)P(w|d)$ \n", "\n", "$= P(d)\\sum_{z}P(w,z|d)$ \n", "\n", "$=P(d)\\sum_{z}P(z|d)P(w|z)$" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "rIUH6dILnmQs" }, "source": [ "### **18.1.3 共现模型**\n", "\n", "$P(w,d) = \\sum_{z\\in Z}P(z)P(w|z)P(d|z)$" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "JSt5kq4LoFJT" }, "source": [ "虽然生成模型与共现模型在概率公式意义上是等价的，但是拥有不同的性质。生成模型刻画文本-单词共现数据生成的过程，共现模型描述文本-单词共现数据拥有的模式。 \n", "\n", "如果直接定义单词与文本的共现概率 $P(w,d)$, 模型参数的个数是 $O(M*N)$, 其中 $M$ 是单词数， $N$ 是文本数。 概率潜在语义分析的生成模型和共现模型的参数个数是 $O(M*K + N*K)$, 其中 $K$ 是话题数。 现实中 $K<=1$ 的随机变量 $X_{t}$与前\n", "\n", "一个时刻的随机变量 $X_{t-1}$ 之间有条件分布 $P(X_{t}|X_{t-1})$ 如果 $X_{t}$ 只依赖于 $X_{t-1}$, 而不依赖于过去的随机变量 \n", "\n", "${X_{0}，X_{1},...，X_{t-2}}$ 这一性质称为马尔可夫性，即 \n", "\n", "$P(X_{t}|X_{0},X_{1},...,X_{t-1}) = P(X_{t}|X_{t-1}), t=1,2,...$\n", "\n", "具有马尔可夫性的随机序列$X = {X_{0}, X_{1},..., X(t),...}$称为马尔可夫链， 或马尔可夫过程（Markov process)。 条件概率分布 \n", "\n", "$P(X_{t}|X_{t-1})$ 称为**马尔可夫链的转移概率分布**。 **转移概率分布决定了马尔可夫裢的特性**。\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "BL8oLbgBttgs" }, "source": [ "#### 平稳分布 \n", "\n", "设有马尔可夫链$X = {X_{0}, X_{1},..., X(t),...}$，其状态空间为 $S$,转移概率矩阵为 $P=(p_{ij})$， 如果存在状态空间 $S$ 上的一个分布 \n", "\n", "$\\pi = \\begin{bmatrix}\n", "\\pi_{1}\\\\ \n", "\\pi_{2}\\\\ \n", "\\vdots \\end{bmatrix}$\n", "\n", "使得 \n", "\n", "$\\pi = P\\pi$\n", "\n", "则称丌为马尔可夫裢$X = {X_{0}, X_{1},..., X(t),...}$的平稳分布。\n", "\n", "\n", "直观上，如果马尔可夫链的平稳分布存在，那么以该平稳分布作为初始分布，面向未来进行随机状态转移，之后任何一个时刻的状态分布都是该平稳分布。\n", "\n", "**引理19．1**\n", "\n", "给定一个马尔可夫链$X = {X_{0}, X_{1},..., X(t),...}$, 状态空间为$S$, 移概率矩阵为$P=(p_{ij})$， 则分布 $\\pi=(\\pi_{1}, \\pi_{2},...)^{T}$ 为 $X$ 的平稳分布的充要条件是$\\pi=(\\pi_{1}, \\pi_{2},...)^{T}$是下列方程组的解：\n", "\n", "$x_{i} = \\sum_{j}p_{ij}x_{j}, i=1,2,...$ \n", "\n", "$x_{i} >= 0, i = 1,2,...$ \n", "\n", "$\\sum_{i}x_{i} = 1$ \n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "HqSHX7PSwOjP" }, "source": [ "### 吉布斯采样\n", "\n", "输入： 目标概率分布的密度函数$p(x)$, 函数$f(x)$;\n", "\n", "输出： $p(x)$的随机样本 $x_{m+1}, x_{m+2}, ..., x_{n}$，函数样本均值 $f_{mn}$;\n", "\n", "参数： 收敛步数$m$, 迭代步数 $n$.\n", "\n", "\n", "1. 初始化。给出初始样本 $x^{0} = $($x^{0}_{1}, x^{0}_{2},..., x^{0}_{k}$)$^{T}$.\n", "\n", "2. 对$i$循环执行 \n", " 设第$i-1$次迭代结束前的样本为$x^{i-1} = $($x^{i-1}_{1}, x^{i-1}_{2},..., x^{i-1}_{k}$)$^{T}$，则第$i$次迭代进行如下几步操作： \n", "\n", " + (1)由满条件分布 $p(x_{1}|x^{i-1}_{2},...,x^{i-1}_{k})$ 抽取 $x^{i}_{1}$ \n", " \n", " + ...\n", " \n", " + (j)由满条件分布 $p(x_{j}|x^{i}_{1},...,x^{i}_{j-1}, x^{i-1}_{j+1},..., x^{i-1}_{k})$ 抽取 $x^{i}_{j}$ \n", " \n", " + (k)由满条件分布 $p(x_{k}|x^{i}_{1},...,x^{i}_{k})$ 抽取 $x^{i}_{k}$ \n", " \n", "得到第 $i$ 次迭代值 $x^{(i)} = (x^{(i)}_{1}, x^{(i)}_{2},..., x^{(i)}_{k})^{T}$.\n", "\n", "\n", " 3. 得到样本集合\n", " \n", " {$x^{(m+1)}, x^{(m+2)},..., x^{(n)}$}\n", " \n", " 4. 计算\n", " \n", " $f_{mn} = \\frac{1}{n-m}\\sum_{i=m+1}^{n}f(x^{(i)})$" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "5ZeiXcVWBQZb" }, "source": [ "--------------------------------------------------------------------------------------------------------------------------------" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "PIcMXLgQBSio" }, "source": [ "#### 网络资源：\n", "\n", "LDA-math-MCMC 和 Gibbs Sampling： https://cosx.org/2013/01/lda-math-mcmc-and-gibbs-sampling \n", "\n", "MCMC蒙特卡罗方法： https://www.cnblogs.com/pinard/p/6625739.html" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "kIIlKmr0I8d_", "outputId": "265030d7-7e28-443c-eed0-e63ee8b21a73" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[0.23076935 0.30769244 0.46153864]]\n" ] } ], "source": [ "import random\n", "import math\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "import numpy as np\n", "\n", "transfer_matrix = np.array([[0.6, 0.2, 0.2], [0.3, 0.4, 0.3], [0, 0.3, 0.7]],\n", " dtype='float32')\n", "start_matrix = np.array([[0.5, 0.3, 0.2]], dtype='float32')\n", "\n", "value1 = []\n", "value2 = []\n", "value3 = []\n", "for i in range(30):\n", " start_matrix = np.dot(start_matrix, transfer_matrix)\n", " value1.append(start_matrix[0][0])\n", " value2.append(start_matrix[0][1])\n", " value3.append(start_matrix[0][2])\n", "print(start_matrix)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 269 }, "colab_type": "code", "id": "A2oybaKXJGqd", "outputId": "3b2abcf8-71f1-4953-8536-667de8a1f36e" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD4CAYAAAANbUbJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU9b3/8ddnJpM9ELIiEPYACQHBBlGxIqK4At5W/bmWan9X/VWrrT8XvFq1tP706m3rbS9X6+21thc3ipWiaFUQBGkVEsWFDIEQQcKShIQEsicz398fMwkh6yRMcmb5PB/M4+xnPod58J7D95zzHTHGoJRSKrTZrC5AKaXUwNOwV0qpMKBhr5RSYUDDXimlwoCGvVJKhYEIqwvoKCUlxYwdO9bqMpRSKqjk5+cfMcakdrc84MJ+7Nix5OXlWV2GUkoFFRHZ19NybcZRSqkwoGGvlFJhQMNeKaXCgIa9UkqFAQ17pZQKAxr2SikVBjTslVIqDATcffYq8LmNm2Z3M42uRppcTTS6Gml2NeMyLlrcLbSYFlxu14lpdwsu48LldtFiWnAbN27jxhjjGccz7jKuTvPcxo3BYIzpcQicGLab16q1K+/267Qt67C9d6KTjvvsbX6g0e7MA196XDpXT7p6QPatYR9G6prrKKsro6KhgpqmGo43H6emqYaa5pq24fGm4ydNtw/0tmB3N1t9KKqfBLG6BNWDaanTNOxVz2qbazlUc4jSulLPq9YzPFx3uG38eNPxbrePkAjiI+OJd8STEJlAfGQ8I+JHEGOPIdIeSZQ9ikh75EnjUfYoouxROGwOHHYHEbYIIiQCu9iJsEVgt9k90zY7drHjsDmwiQ2b2LCLHZvYEBHPPDzjdrG3zRMEEWkbtq4DnDTdPsDalnu3aR1vPzwxaLcdctL2HZd33L9SwUbDPgjVNddReLSQr458xVdHvmJHxQ72Hev8pHRydDLpcelkJGSQm55Lelw66bHpJMckk+DwBHpCZALxjnii7FEaZEqFMA37ANfsambX0V3sqNjhCfeKr9hTtQe3cQOQFptGTnIOiyYsIiMhg/TYdNLj0kmLScNhd1hcvVIqUGjYB6BmVzMflnzI67tf55NDn7S1kSdGJTI1ZSoXZFzA1OSp5KTkkBrbbSd3SinVRsM+gBRXF/PG7jdYs2cNlQ2VpMWmce2Ua5meOp2c5BxGxo/UphalVL9o2FusrrmOd/e+y192/4Xt5duJkAjmZszlO5nfYc6IOdhtdqtLVEqFAJ/CXkQuAf4dsAO/N8Y82c16VwF/BmYZY/JEZCzgBAq9q3xsjLn9VIsOdsYYvjryFa/vfp13vn6HupY6xg4Zyz3fuoeFExaSEpNidYlKqRDTa9iLiB1YDlwElADbRGSNMaagw3oJwF3AJx12sccYM8NP9QY9Z4WTR//+KM5KJzERMSwYs4DvZH6HmWkztYlGKTVgfDmzPxMoMsYUA4jIq8BioKDDej8HngLu9WuFIcIYw8s7X+aXeb9kWPQwfnrWT7ls3GXER8ZbXZpSKgz40jfOSGB/u+kS77w2IjITyDDGvNXF9uNE5DMR+VBEvt3VG4jIrSKSJyJ55eXlvtYeNKoaqrhrw108ufVJ5oyYw+sLX+eayddo0CulBo0vZ/ZdtS20dbIhIjbg18D3u1jvEDDaGFMhIt8CVovIVGPMsZN2ZszzwPMAubm5IdWBR97hPJZuXkplQyVLz1zK9VOu1+YapdSg8yXsS4CMdtOjgIPtphOAHGCjN8SGA2tEZJExJg9oBDDG5IvIHmASEPK/KO5yu3j+y+d57vPnyEjIYMVlK8hOzra6LKVUmPIl7LcBmSIyDjgAXAtc37rQGFMNtN0+IiIbgXu9d+OkApXGGJeIjAcygWI/1h+QSmtLWbp5KXmleSwcv5CHznqIOEec1WUppcJYr2FvjGkRkTuBd/HcevmCMWaHiCwD8owxa3rY/DxgmYi0AC7gdmNMpT8KD1Qf7v+Qh7c8TKOrkcfPfZxFExZZXZJSSiGB1sd1bm6uycsLvlaeJlcTv87/NSucK5iSNIWnz3uasUPHWl2WUipMiEi+MSa3u+X6BK0fGGNYunkp7+97n+unXM89ufcQZY+yuiyllGqjYe8Hq3av4v1973PPt+7h5pybrS5HKaU60d+gPUXFVcU8tfUpzhlxDkumLrG6HKWU6pKG/SlodDVy/6b7iXXE8vi5j2MT/etUSgUmbcY5Bc/kP0Ph0UKWz1+unZcppQKanor20+aSzaxwruCGrBs4b9R5VpejlFI90rDvhyP1R3h4y8NMGjaJn3zrJ1aXo5RSvdJmnD5yGzcPf/Qwtc21vHDxC3qLpVIqKOiZfR+tKFjBloNbuH/W/UxInGB1OUop5RMN+z5wVjj59ae/5oKMC7h60tVWl6OUUj7TsPdRXXMd92+6n6SoJH52zs+0m2KlVFDRNnsfPbXtKfYd28d/LfgvEqMTrS5HKaX6RM/sffDe3vd4fffr3JJzC7NPm211OUop1Wca9r04XHuYx/7xGDnJOdwx8w6ry1FKqX7RsO+B27hZunkpLreLp857CofNYXVJSinVLxr2Pdh6eCv5pfncO+teMoZk9L6BUkoFKA37HqwuWk2CI4GF4xdaXYpSSp0SDftuHG86zvp967l03KVER0RbXY5SSp0SDftuvLv3XRpcDVw58UqrS1FKqVOmYd+N1UWrmTB0AjkpOVaXopRSp0zDvgvF1cV8Xv45iycu1idllVIhQcO+C38t+it2sbNwgl6YVUqFBg37DlxuF2/teYtzR56rvz6llAoZGvYd/P3g3ymrL9MLs0qpkKJh38HqotUkRiUyd9Rcq0tRSim/0bBvp7qxmg37N3D5+Mtx2LVrBKVU6NCwb2dt8Vqa3c3ahKOUCjka9u2sLlrNlKQpTEmaYnUpSinlVxr2XoWVhTgrnXpWr5QKSRr2Xn/d81cibBFcNu4yq0tRSim/07AHmt3NrC1ey7yMeQyLHmZ1OUop5Xca9sCmkk1UNlSyeMJiq0tRSqkBoWGP58JsSkwKc0bOsboUpZQaEGEf9kfqj7C5ZDMLxy8kwhZhdTlKKTUgfAp7EblERApFpEhElvaw3lUiYkQkt928B73bFYrIxf4o2p/WFq/FZVx6F45SKqT1eiorInZgOXARUAJsE5E1xpiCDuslAHcBn7Sblw1cC0wFRgDrRGSSMcblv0PoP2MMq4tWMz1lOuMTx1tdjlJKDRhfzuzPBIqMMcXGmCbgVaCrK5k/B54CGtrNWwy8aoxpNMZ8DRR59xcQCioKKKoqYvFEvTCrlAptvoT9SGB/u+kS77w2IjITyDDGvNXXbb3b3yoieSKSV15e7lPh/vBG0RtE2aO4ZNwlg/aeSillBV/CvqufajJtC0VswK+B/9vXbdtmGPO8MSbXGJObmprqQ0mnrtHVyDtfv8MFoy9gSOSQQXlPpZSyii+3n5QAGe2mRwEH200nADnARu9P+A0H1ojIIh+2tcyG/Rs41nRML8wqpcKCL2f224BMERknIpF4LriuaV1ojKk2xqQYY8YaY8YCHwOLjDF53vWuFZEoERkHZAJb/X4U/bC6aDXD44Yze/hsq0tRSqkB12vYG2NagDuBdwEnsNIYs0NElnnP3nvadgewEigA/gbcEQh34pTWlvKPg/9g4fiF2G12q8tRSqkB59NTRMaYt4G3O8x7pJt1z+8w/TjweD/rGxBvFb+F27i1CUcpFTbC8gnavNI8JiZOZPSQ0VaXopRSgyJkwr6pxc32/VWUHmvocT1jDAUVBWQnZw9SZUopZb2QCfuquiauXL6Fv311uMf1yuvLqWyo1LBXSoWVkAn71IQoEmMdFJYe73G9nZU7AchKyhqMspRSKiCETNiLCJPSEth1uOewL6goQBAmJ00epMqUUsp6IRP2AJOGx7Or9DjGdHpIt42zwsmYIWOIc8QNYmVKKWWt0Ar79ASONbRQeqyx23WclU5twlFKhZ2QC3uAXd2021c1VHGo9hBZyRr2SqnwElZh76x0AmjYK6XCTkiFfVJcJCnxUb2HvTbjKKXCTEiFPcDk4fEUltZ0ucxZ4WRE3AiGRg0d5KqUUspaIRf2mWkJ7C49jtvd+Y4cZ6VTm3CUUmEp5MJ+8vAE6ppcHKiqP2l+TVMN+47t0yYcpVRYCrmwn5QeD3S+SFt4tBDQi7NKqfAUcmGf2XZHzsnt9s4KvTirlApfIRf2Q6IdnDY0utOZvbPSSUpMCqmxg/Mbt0opFUh8+vGSYDMpPaHLsNezeqUCT3NzMyUlJTQ09Nw9ufKIjo5m1KhROByOPm0XomEfzz+KK3C5DXab0NDSQHFVMfMy5lldmlKqg5KSEhISEhg7diwiYnU5Ac0YQ0VFBSUlJYwbN65P24ZcMw54zuybWtzsq6gFYPfR3biMS8/slQpADQ0NJCcna9D7QERITk7u1/+CQjLsJw8/+SKtdpOgVGDToPddf/+uQjLsJ6adfPtlQUUBQyKHMCJuhJVlKaWUZUIy7GMjIxidFNsW9jsrd5KVnKVnD0opn33/+99n1apVA7LvzZs3M3XqVGbMmEF9fX23651//vnk5eX55T1DMuzBc5F2V+lxmt3N7Dq6i+wk/c1ZpZT1XC4XL730Evfeey/bt28nJiZmUN43hMM+geLyWgorimh2NzMlaYrVJSmlAtif/vQnpk+fzumnn85NN90EwKZNmzjnnHMYP378SWf5Tz/9NLNmzWL69Ok8+uijbfNXrFjBmWeeyYwZM7jttttwuVwAxMfH88gjjzB79myeeOIJVq5cybJly7jhhhvYuHEjV1xxRds+7rzzTl588UW/H19I3noJnrBvcRs++mY7oBdnlQoGP3tzBwUHj/l1n9kjhvDowqk9rrNjxw4ef/xxtmzZQkpKCpWVldxzzz0cOnSIjz76iJ07d7Jo0SKuuuoq3nvvPXbv3s3WrVsxxrBo0SI2bdpEamoqr732Glu2bMHhcPDDH/6Ql156ie9973vU1taSk5PDsmXLACgqKuKKK67gqquuYuPGjX493u6EdNgDfFq6g9iIWMYMGWNxRUqpQPXBBx9w1VVXkZKSAkBSUhIAV155JTabjezsbEpLSwF47733eO+995g5cyYANTU17N69my+++IL8/HxmzZoFQH19PWlpaQDY7Xa++93vDvZhnSRkw358ahx2m7CnqpApSVOwSci2WCkVMno7Ax8oxpgub+CIioo6aZ3W4YMPPshtt9120rq//e1vWbJkCU888USn/URHR2O327t874iICNxud9v0QD1JHLIJGO2wMzo5miPNX2t7vVKqR/Pnz2flypVUVFQAUFlZ2e26F198MS+88AI1NZ7neA4cOEBZWRnz589n1apVlJWVte1j3759vb73mDFjKCgooLGxkerqatavX++HI+osZM/sAUan1XLE3ajt9UqpHk2dOpWHHnqIuXPnYrfb25pourJgwQKcTidnn3024Ln4umLFCrKzs/nFL37BggULcLvdOBwOli9fzpgxPTchZ2RkcM011zB9+nQyMzN7fO9TIa3/NQkUubm5xl/3ld7x1/9mU9UzvHzpSqalaeArFYicTidZWfrvsy+6+jsTkXxjTG5324RsMw4AkSUYdwTupjSrK1FKKUuFdNgfbfkad+Nwisu7f0JNKaXCQciGvTGGvTW7MI0jOv1qlVJKhZuQDfsDNQc43nScFMd4dh0+3vsGSikVwnwKexG5REQKRaRIRJZ2sfx2EflSRLaLyEciku2dP1ZE6r3zt4vIc/4+gO7srNwJwPihUygs1bBXSoW3XsNeROzAcuBSIBu4rjXM23nZGDPNGDMDeAr4Vbtle4wxM7yv2/1VeG8KKgqwi52Zw6dQcrSe2saWwXprpZQKOL6c2Z8JFBljio0xTcCrwOL2Kxhj2ndmEQdYfj+ns9LJhMQJZA1PBmB3mbbbK6XCly9hPxLY3266xDvvJCJyh4jswXNmf1e7ReNE5DMR+VBEvt3VG4jIrSKSJyJ55eXlfSi/a8YYCioKmJI0hcnprb9apU05Sqnw5UvYd/WLH53O3I0xy40xE4AHgIe9sw8Bo40xM4F7gJdFZEgX2z5vjMk1xuSmpqb6Xn03yuvLqWyoJDs5m4ykWKIibHqRVinVrdraWi6//HJOP/10cnJyeO2111i/fj0zZ85k2rRp3HLLLTQ2Nnba7tChQ5x33nnMmDGDnJwcNm/eDMArr7zCtGnTyMnJ4YEHHhjsw+mSL90llAAZ7aZHAQd7WP9V4FkAY0wj0Ogdz/ee+U8C/POIbDdaL85mJWVhtwmZ6fHs0mYcpQLfO0vh8Jf+3efwaXDpkz2u8re//Y0RI0awdu1aAKqrq8nJyWH9+vVMmjSJ733vezz77LP8+Mc/Pmm7l19+mYsvvpiHHnoIl8tFXV0dBw8e5IEHHiA/P59hw4axYMECVq9ezZVXXunf4+ojX87stwGZIjJORCKBa4E17VcQkcx2k5cDu73zU70XeBGR8UAmUOyPwntSUFGAIExOmgx4ujvWM3ulVHemTZvGunXreOCBB9i8eTN79+5l3LhxTJo0CYAlS5awadOmTtvNmjWLP/zhDzz22GN8+eWXJCQksG3bNs4//3xSU1OJiIjghhtu6HLbwdbrmb0xpkVE7gTeBezAC8aYHSKyDMgzxqwB7hSRC4Fm4CiwxLv5ecAyEWkBXMDtxpjuu5PzE2eFkzFDxhDniAM8Yf+XTw9QXd/M0BjHQL+9Uqq/ejkDHyiTJk0iPz+ft99+mwcffJAFCxZ0ud4nn3zS1rXxsmXL2n64ZO3atdx0003cd999DBnSqaU6IPjU66Ux5m3g7Q7zHmk3fnc3270OvH4qBfaHs9LJjNQZbdOtF2l3lx4nd2zSYJejlApwBw8eJCkpiRtvvJH4+Hiee+459u7dS1FRERMnTuR//ud/mDt3LrNnz2b79u1t2+3bt4+RI0fyz//8z9TW1vLpp5/ywAMPcPfdd3PkyBGGDRvGK6+8wo9+9CMLj84j5Lo4rmqo4lDtIa6bcl3bvMz0eAAKNeyVUl348ssvue+++7DZbDgcDp599lmqq6u5+uqraWlpYdasWdx+e+fHhDZu3MjTTz+Nw+EgPj6eP/3pT5x22mk88cQTzJs3D2MMl112GYsXL+7iXQdXyIW9s9IJnPybsyMTY4iLtLNb+8hRSnXh4osv5uKLL+40/7PPPutxuyVLlrBkyZJO86+//nquv/56v9XnDyHXN05b2CedCHsRITM9gUK9SKuUClOhF/YVTkbEjWBo1NCT5k9OT2B3mYa9Uio8hV7YVzq7/BnCzPR4jtQ0UVHT+cEIpZQKdSEV9jVNNew7tu+kJpxWk4e3dpug7fZKqfATUmFfeLQQoMsze+0jRykVzkIq7J0VnS/OtkpNiGJojEP7tldKhaXQCvtKJykxKaTGdu5MTUQ8F2k17JVSfrJ3715ycnKsLsMnIRf2XZ3Vt8pMj6fw8HGMsby7faWUGlQh81BVQ0sDxVXFzMuY1+06k4cn8NInLZQdbyR9SPQgVqeUCmS1tbVcc801lJSU4HK5+OlPf0phYSFvvvkm9fX1nHPOOfzud79DRMjPz+eWW24hNjaWc8891+rSfRYyYV/TXMOCsQvITc/tdp3MNM9F2sLDxzXslQpA/7r1X9u6KPeXKUlTeODMnvuU76qL44suuohHHvF0AXbTTTfx1ltvsXDhQm6++WZ++9vfMnfuXO677z6/1jqQQqYZJyUmhafOe4qzR5zd7TqTvH3k6B05Sqn2OnZxPHToUDZs2MDs2bOZNm0aH3zwATt27KC6upqqqirmzp0LeL4EgkXInNn7Ijk+ipT4KA17pQJUb2fgA6WrLo6XL19OXl4eGRkZPPbYYzQ0NGCMQaSrH+8LfCFzZu+rSenxFOqDVUqpdg4ePEhsbCw33ngj9957L59++ikAKSkp1NTUsGrVKgASExMZOnQoH330EQAvvfSSZTX3VVid2YPnh0z+nLcft9tgswXnN7RSyr+66uJ49erVTJs2jbFjxzJr1qy2df/whz+0XaDtqqfMQCWBdhtibm6uycsbuJ+offmTb/iXN75k8/3zyEiKHbD3UUr5xul0kpXV/S3TqrOu/s5EJN8Y0+0dKmHXjDN5uOcirfaAqZQKJ2EX9hPbbr/UdnulVPgIu7AfGuPgtKHR2m2CUgEk0JqTA1l//67CLuzBc5FWO0RTKjBER0dTUVGhge8DYwwVFRVER/f9odCwuxsHIGfkEH73YTHHGpoZEu2wuhylwtqoUaMoKSmhvLzc6lKCQnR0NKNGjerzdmEZ9vMmp7F8wx4+LCxn4ekjrC5HqbDmcDgYN26c1WWEvLBsxpk5ehhJcZGsc5ZaXYpSSg2KsAx7u02YNzmNjYXltLjcVpejlFIDLizDHuCi7DSq65vJ23fU6lKUUmrAhW3YfzszlUi7jXUF2pSjlAp9YRv2cVERnD0hmXXOUr3lSykV8sI27AEuzEpjb0Ude8prrS5FKaUGVFjeetlqflY6P/3rDtY7S5mYFm91OafO1QIt9dDSCM3eYVfTribPuu4WcDd7ht1NGxcYN7hdYIxnvHVe23z3iWUYH8Y5Md2qbdycPN5pOT4s67D8pNm+/i9O/7enLJAyCS791wHZdViH/YjEGLJPG8I6Zym3zZ1gdTknNDfAsQNQvR9qyqChGhqqvMMeXu4W/9YhNhA72OwnxsUGIu3mtc4XzzjiHRfvuK3rcehhnA7z2wrqUF8PyzotP2mBj8evXWCrQdY0cK0MYR324GnK+Y8NRRytbWJYXOTgvGntETi6zxPm1SUngr26xPOq7eZJQkcsRA898YpLheSJnvGoIZ7ljmiIiIaIKIiI8Qwd3mGEd5k9EuwOsEV4Xu3HW6fFDrawbuVTKqRo2Gen85sPithQWMZ3zuj7I8i9ammEQ19AyTbP60AeVH1z8jqOOEjMgKGj4LTTPcOh3un44RCT6AnziEH6MlJKhRyfwl5ELgH+HbADvzfGPNlh+e3AHYALqAFuNcYUeJc9CPzAu+wuY8y7/iv/1OWMGEpaQhTrnKWnHvbGQNU+KMnzvrbB4S88beTgCfCR34Izb/WckQ8d5XlFJ2qTgVJqQPUa9iJiB5YDFwElwDYRWdMa5l4vG2Oe866/CPgVcImIZAPXAlOBEcA6EZlkjHH5+Tj6zWYT5mel8+bnB2lscREVYe/bDozxBPunL8Kud080wUTEwMgz4Kz/A6NmwchcGHKa3+tXSilf+HJmfyZQZIwpBhCRV4HFQFvYG2OOtVs/jhO3MiwGXjXGNAJfi0iRd3//8EPtfnNhVhqvbP2GT4orOW9Sqm8b1R+FL1ZC/h+hbIenKWbK5TB6tifc06aCPexbyZRSAcKXNBoJ7G83XQLM7riSiNwB3ANEAhe02/bjDtuO7FelA2jOxBSiHTbWO0t7Dntj4JuPIf9FKFgNLQ0wYiZc8QxMuwqiEgatZqWU6gtfwr6rxuRONyEbY5YDy0XkeuBhYImv24rIrcCtAKNHj/ahJP+Kdtg5d2Iq65xlPLbIIB3bz+sq4fNXPGfxRwohMgFm3ADfWuK5oKqUUgHOl7AvATLaTY8CDvaw/qvAs33Z1hjzPPA8QG5uriVPs1yUncY6Zyk7Dx8n67QhnpnHDsK6n8GOv3guso6aBYuXw9R/gsg4K8pUSql+8SXstwGZIjIOOIDnguv17VcQkUxjzG7v5OVA6/ga4GUR+RWeC7SZwFZ/FO5v86akAbCuoJSs4Qmw/SX42794Qv5bN3vO4tOnWlylUkr1T69hb4xpEZE7gXfx3Hr5gjFmh4gsA/KMMWuAO0XkQqAZOIqnCQfveivxXMxtAe4IpDtx2ktLiGZGRiLbd+yAQ0uhaB2MPgcW/wckB9DTtUop1Q8SaD0+5ubmmry8vMF/Y2P44OWnmbXrV8Q7QC5aBrP+tz5FqpQKCiKSb4zJ7W653hsInq4L3ryLC4o38nd3NhXn/hsLZ8+xuiqllPKb8A57txvyX4D3HwXAXP4r7l83hin7olhocWlKKeVP4Rv2lV/Dmh/B3s0wfh4s+g2SOJr5B7/itbz9NDS7iHb08WlapZQKUOHXIG0MbPtvePYcOPQ5LPwN3PQGJHru778wO52GZjdbio5YXKhSSvlPeIV9/VFY+T1Yew+MPgt++A/PLZXtHqKaPS6Z+KgI1jn1t2mVUqEjfJpx9m+FVT+A4wfhop/D2Xd2eadNZISNuZNSWe8sw+022GzaG6VSKviF/pm92w2bfwkvXOI5g7/lPZhzV4+3VM7PSqPseCNfHawexEKVUmrghPaZ/fFSeONWKN4IU78DC5/x/KpTL+ZNTsMmnqdpp49KHPg6lVJqgIXumX3RenhuDnzzieci7FUv+BT0AMPiIskdk8Q6Z9kAF6mUUoMj9MLe1ey5b37FdyA2BW7d0OkirC8uzE6j4NAxDlTVD1ChSik1eEIr7I/u9bTNb3nG03nZrRsgLatfu5qflQ7AB3pXjlIqBIRO2B8pgufOgyO74eoXPe3zjph+725CajzjUuJ4X5tylFIhIHTCPmk85N4Mt2/y9DfvBxdmpfHxngpqGlv8sj+llLJK6IS9zQYX/QyGjfXbLudnpdPkcrNem3KUUkEudMJ+AOSOGcb41Dj+ff1uml1uq8tRSql+07DvQYTdxr9cmkVxeS2vbP3G6nKUUqrfNOx7MT8rjbPHJ/Pr93dRXd9sdTlKKdUvGva9EBEeujyLqvpm/nNDkdXlKKVUv2jY+yBn5FC+e8Yo/rBlL/sr66wuRyml+kzD3kf3LpiM3SY8+bedVpeilFJ9pmHvo+FDo7n1vPGs/eIQ+fsqrS5HKaX6RMO+D26bO560hCh+sdaJMcbqcpRSymca9n0QGxnBvRdP5rNvqnjri0NWl6OUUj7TsO+j754xiqzThvDkOztpaHZZXY5SSvlEw76P7Dbh4cuzOFBVz4t/32t1OUop5RMN+36YMzGF+VPSWP5BERU1jVaXo5RSvdKw76cHL8uirtnFM+t2W12KUkr1SsO+nyamxXPD7NG8vPUbisqOW12OUuXqZecAAAn6SURBVEr1SMP+FNw9P5NYh53/97Y+aKWUCmwa9qcgOT6KOy+YyAc7y/ho9xGry1FKqW5p2J+iJeeMZdSwGH6xtgCXWx+0UkoFJg37UxTtsLP00insPHycVfn7rS5HKaW6pGHvB5dPO40zRifyxDs7+epAtdXlKKVUJxr2fiAiPPO/ZhIXGcF1z3+sHaUppQKOT2EvIpeISKGIFInI0i6W3yMiBSLyhYisF5Ex7Za5RGS797XGn8UHktHJsfz59rNJSYjixt9vZUuRXrBVSgWOXsNeROzAcuBSIBu4TkSyO6z2GZBrjJkOrAKeares3hgzw/ta5Ke6A9KIxBheu+0sRifFcvOL21hXUGp1SUopBfh2Zn8mUGSMKTbGNAGvAovbr2CM2WCMaf0Jp4+BUf4tM3ikJUTz6q1nMWV4ArevyOfNzw9aXZJSSvkU9iOB9reZlHjndecHwDvtpqNFJE9EPhaRK7vaQERu9a6TV15e7kNJgW1YXCQv/e/ZnDF6GHe/+hkr8/QuHaWUtXwJe+liXpc3lIvIjUAu8HS72aONMbnA9cAzIjKh086Med4Yk2uMyU1NTfWhpMCXEO3gj7ecyZyJKdy/6gte3PK11SUppcKYL2FfAmS0mx4FdGqbEJELgYeARcaYtq4gjTEHvcNiYCMw8xTqDSoxkXZ+vySXBdnpPPZmAcs3FFldklIqTPkS9tuATBEZJyKRwLXASXfViMhM4Hd4gr6s3fxhIhLlHU8B5gAF/io+GERF2Fl+wxksnjGCp98t5Ol3d+pPGiqlBl1EbysYY1pE5E7gXcAOvGCM2SEiy4A8Y8waPM028cCfRQTgG++dN1nA70TEjeeL5UljTFiFPYDDbuNX18wgNtLO8g17qG108cgV2dhsXbWQKaWU/0mgnWXm5uaavLw8q8sYEMYYfv6Wkxe2fM20kUP58YWZXDAlDe8XpFJK9ZuI5Huvj3ZJn6AdRCLCT6/I4pdXn05VfRM/+GMeVy7fwobCMm3aUUoNKD2zt0izy81fPi3hN+uLOFBVz8zRifzkwkl8OzNFz/SVUn3W25m9hr3FmlrcrMov4T8+2M3B6gZyxwzjJxdN4pwJyRr6SimfadgHicYWFyvzSlj+QRGHjzVw5rgkfnLhJM6ekGx1aUqpIKBhH2Qaml28uvUb/nPjHsqON3J6RiLzJqfy7cwUpo9KxGHXyyxKqc407INUQ7OLlz/5htXbD/DlgWqMgfioCM4an8y3M1M4NzOF8Slx2tSjlAI07ENCVV0Tf99TwebdR/ioqJz9lfUAjBgazZyJnuCfMzGFlPgoiytVSllFwz4EfVNRx+aicj7afYS/76mgur4ZgNSEKMYlxzEmOZaxKXGMbTceH9Xr83NKqSCmYR/iXG7DVweq+UdxBXvKathXUcfXFbWUH288ab2U+CjGpcQyJjmOUcNiGBYbSWKsg6S4yJPGYxx2bRpSKgj1FvZ6uhfk7Dbh9IxETs9IPGl+bWML+yrq2FtR63kdqWVvRR2bdpVT1uGLoL3ICBvDYh0Mi/V8CcRF2Yl22Ilx2ImJ9L7aT3uHURF2HHbBYbcRYRMcETYcNhuOCCHCZiPSbiPCLp6XzYZNwGYT7CLYbYKtbYh+2Sg1ADTsQ1RcVATZI4aQPWJIp2UtLjdV9c1U1TVxtK6ZytqmtvGjdU0crfWMV9U1cai6mfpmF/VNrrZhY4t7QGsXAbsINpsggE0EEe/Qu7zjMvAMpd0+pN289l8graNtQ+9WJ6Zbl3f+0un2a8jH76e+fI0Fy5decFQZHKacNoTfXjcwHQNr2IehCLuNlPiofl/QdbkNDc0nwr+h2UVDs5tmt5vmFjctbkOTy02Ly9DicreNN7vcNLsNbrfB5Ta4jWfoMq3zwG1Ono/nD2638QyNwRhPP0NuAwbv0EDrzywY77TBuy6tyz3zOLFq2w8ztDZnnpjufNzdNXj62hTapwbTwGpd7ZYJlkKDRMawmAHbt4a96jO7TYiLiiBOL/oqFTT0CR2llAoDGvZKKRUGNOyVUioMaNgrpVQY0LBXSqkwoGGvlFJhQMNeKaXCgIa9UkqFgYDrCE1EyoF9p7CLFOCIn8oJBKF2PBB6xxRqxwOhd0yhdjzQ+ZjGGGNSu1s54ML+VIlIXk89vwWbUDseCL1jCrXjgdA7plA7Huj7MWkzjlJKhQENe6WUCgOhGPbPW12An4Xa8UDoHVOoHQ+E3jGF2vFAH48p5NrslVJKdRaKZ/ZKKaU60LBXSqkwEDJhLyKXiEihiBSJyFKr6/EHEdkrIl+KyHYRCbpfYReRF0SkTES+ajcvSUTeF5Hd3uEwK2vsq26O6TEROeD9nLaLyGVW1tgXIpIhIhtExCkiO0Tkbu/8oPycejieYP6MokVkq4h87j2mn3nnjxORT7yf0WsiEtnjfkKhzV5E7MAu4CKgBNgGXGeMKbC0sFMkInuBXGNMUD4MIiLnATXAn4wxOd55TwGVxpgnvV/Kw4wxD1hZZ190c0yPATXGmH+zsrb+EJHTgNOMMZ+KSAKQD1wJfJ8g/Jx6OJ5rCN7PSIA4Y0yNiDiAj4C7gXuAvxhjXhWR54DPjTHPdrefUDmzPxMoMsYUG2OagFeBxRbXFPaMMZuAyg6zFwN/9I7/Ec8/xKDRzTEFLWPMIWPMp97x44ATGEmQfk49HE/QMh413kmH92WAC4BV3vm9fkahEvYjgf3tpksI8g/YywDviUi+iNxqdTF+km6MOQSef5hAmsX1+MudIvKFt5knKJo8OhKRscBM4BNC4HPqcDwQxJ+RiNhFZDtQBrwP7AGqjDEt3lV6zbxQCXvpYl7wt0/BHGPMGcClwB3eJgQVeJ4FJgAzgEPAL60tp+9EJB54HfixMeaY1fWcqi6OJ6g/I2OMyxgzAxiFpyUjq6vVetpHqIR9CZDRbnoUcNCiWvzGGHPQOywD3sDzIQe7Um+7amv7apnF9ZwyY0yp9x+jG/gvguxz8rYDvw68ZIz5i3d20H5OXR1PsH9GrYwxVcBG4CwgUUQivIt6zbxQCfttQKb36nQkcC2wxuKaTomIxHkvMCEiccAC4KuetwoKa4Al3vElwF8trMUvWkPR658Ios/Je/HvvwGnMeZX7RYF5efU3fEE+WeUKiKJ3vEY4EI81yI2AFd5V+v1MwqJu3EAvLdSPQPYgReMMY9bXNIpEZHxeM7mASKAl4PtmETkFeB8PF2xlgKPAquBlcBo4BvgamNM0Fzw7OaYzsfTPGCAvcBtre3dgU5EzgU2A18Cbu/sf8HTzh10n1MPx3MdwfsZTcdzAdaO5wR9pTFmmTcjXgWSgM+AG40xjd3uJ1TCXimlVPdCpRlHKaVUDzTslVIqDGjYK6VUGNCwV0qpMKBhr5RSYUDDXimlwoCGvVJKhYH/D4LfMAdoXwWTAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "#进行可视化\n", "x = np.arange(30)\n", "plt.plot(x,value1,label='cheerful')\n", "plt.plot(x,value2,label='so-so')\n", "plt.plot(x,value3,label='sad')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "z2OC3tyoJNeN" }, "source": [ "可以发现，从10轮左右开始，我们的状态概率分布就不变了，一直保持在 \n", "[0.23076934,0.30769244,0.4615386]\n", "\n", "### https://zhuanlan.zhihu.com/p/37121528" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "3Taf1Z8nJU12" }, "source": [ "#### M-H采样python实现 \n", "https://zhuanlan.zhihu.com/p/37121528\n", "\n", "假设目标平稳分布是一个均值3，标准差2的正态分布，而选择的马尔可夫链状态转移矩阵 $Q(i,j)$ 的条件转移概率是以 $i$ 为均值,方差1的正态分布在位置 $j$ 的值。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 323 }, "colab_type": "code", "id": "eUoFG0tYJx9e", "outputId": "b404de1a-535b-4772-f90d-91f1f17ebe1e" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD4CAYAAAAEhuazAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXiU5b3/8fc3YQmi7CCyGZRdgoFGFlEWxUqrCLVa0FK1Vam16rG2tLFyiXKwh1P6U9ujtqXUA62oKLUpVqsFoVgVKghRNlGwCAEFBEF2SLh/f8wkJ8skc0+S2T+v68rlzDP38+QbnPk+99yrOecQEZH0khHvAEREJPaU/EVE0pCSv4hIGlLyFxFJQ0r+IiJpqEG8A6isTZs2Ljs7O95hiIgklXfeeecz51xb3/IJl/yzs7NZtWpVvMMQEUkqZvZxJOXV7CMikoaU/EVE0pCSv4hIGkq4Nv9QTp48SVFREceOHYt3KJIEsrKy6NSpEw0bNox3KCIJKymSf1FREWeccQbZ2dmYWbzDkQTmnGPv3r0UFRXRtWvXeIcjkrCSotnn2LFjtG7dWolfwjIzWrdurW+JImEkRfIHlPjFm94rIuElTfIXEZH6kxRt/lWMGVO/13vxxbBFHnroIZ5++mkyMzPJyMjgt7/9LYMGDarfOMoZMWIEv/jFL8jLy6v1NebMmcPkyZPp1KkThw4d4pxzzmHq1KlceOGFANx///0MGzaMUaNGhTy/oKCAHj160KdPn5Cv/+Y3v+G0007jhhtuiDje/fv38/TTT3P77bcDsHPnTu666y4WLFhQi79URCKVnMk/xpYvX85f//pXVq9eTePGjfnss884ceJEvMPyMn78eB577DEAli5dytVXX83SpUvp3bs306ZNq/HcgoICrrzyypDJv7i4mNtuu63Wce3fv58nnniiLPl36NBBiV8khtTs4+GTTz6hTZs2NG7cGIA2bdrQoUMHAKZNm8YFF1xA3759mTRpEqU7o40YMYIf/OAHDBs2jN69e7Ny5UquvvpqunfvzpQpUwDYunUrvXr14sYbb6Rfv35cc801HDlypMrv//vf/86QIUMYMGAA1157LYcOHQIgPz+fPn360K9fP370ox+F/TtGjhzJpEmTmDVrFgA33XRTWcKtfK233nqLhQsXMnnyZHJzc9myZQsjRozgpz/9KcOHD+eXv/wlDzzwAL/4xS/Krv/UU09x4YUX0rdvX95++22AKmX69u3L1q1byc/PZ8uWLeTm5jJ58mS2bt1K3759gUAH/7e//W1ycnLo378/S5cuBQLfZK6++mpGjx5N9+7d+fGPf+z7v1BEKlHy9/DlL3+Z7du306NHD26//XaWLVtW9todd9zBypUrWbduHUePHuWvf/1r2WuNGjXi9ddf57bbbmPs2LE8/vjjrFu3jjlz5rB3714ANm3axKRJk3jvvfdo1qwZTzzxRIXf/dlnnzF9+nQWL17M6tWrycvL4+GHH2bfvn38+c9/Zv369bz33ntlN5RwBgwYwPvvv1/hWKhrXXjhhVx11VXMnDmTwsJCzj33XCBQY1+2bBk//OEPq1z78OHDvPXWWzzxxBN85zvfqTGOGTNmcO6551JYWMjMmTMrvPb4448DsHbtWp555hluvPHGstE7hYWFzJ8/n7Vr1zJ//ny2b9/u9XeLSEVK/h5OP/103nnnHWbNmkXbtm0ZP348c+bMAQJNKYMGDSInJ4clS5awfv36svOuuuoqAHJycjjvvPM466yzaNy4Meecc05Z0urcuTNDhw4FYOLEibzxxhsVfveKFSvYsGEDQ4cOJTc3l7lz5/Lxxx/TrFkzsrKyuOWWW3jhhRc47bTTvP6WUHs2R3Kt8ePHV/vaddddB8CwYcP44osv2L9/v1dMlb3xxht861vfAqBXr16cffbZfPDBBwBceumlNG/enKysLPr06cPHH0e0lpWIBKnN31NmZiYjRoxgxIgR5OTkMHfuXCZMmMDtt9/OqlWr6Ny5Mw888ECF8eWlzUQZGRllj0ufFxcXA1WHJVZ+7pzjsssu45lnnqkS09tvv81rr73Gs88+y2OPPcaSJUvC/h1r1qyhd+/eFY41aNDA+1pNmzat9tqh/pYGDRpw6tSpsmM+4+9D3aBKlf93zMzMLPt3FJHIqObvYdOmTXz44YdlzwsLCzn77LPLElmbNm04dOhQrTost23bxvLlywF45plnuOiiiyq8PnjwYN588002b94MwJEjR/jggw84dOgQBw4c4Ktf/SqPPvoohYWFYX/XsmXLmDVrFrfeemuF49Vd64wzzuDgwYPef8v8+fOBQM29efPmNG/enOzsbFavXg3A6tWr+fe//x322sOGDWPevHkAfPDBB2zbto2ePXt6xxEXY8aE/hFJUMlZ8/cYmlmfDh06xJ133sn+/ftp0KAB3bp1Y9asWbRo0YJbb72VnJwcsrOzueCCCyK+du/evZk7dy7f/e536d69O9/73vcqvN62bVvmzJnDddddx/HjxwGYPn06Z5xxBmPHjuXYsWM453jkkUdCXn/+/Pm88cYbHDlyhK5du/KnP/2pSs3/4MGDIa81YcIEbr31Vn71q1953dhatmzJhRdeyBdffMGTTz4JwNe//nX+8Ic/kJubywUXXECPHj0AaN26NUOHDqVv37585Stf4fvf/37ZdW6//XZuu+02cnJyaNCgAXPmzKlQ4xeRurOavmLHQ15enqu8mcvGjRurJKxUsHXrVq688krWrVsX71BSTszfM9XV8mNcUZH0ZWbvOOe8JwZ51fzNbDTwSyATmO2cm1Hp9XuAW4BiYA/wHefcx8HXbgRKh6JMd87N9Q1OJJFk579U7WuzN+4CYFTvM2MVjkidhE3+ZpYJPA5cBhQBK81soXNuQ7lia4A859wRM/se8HNgvJm1AqYCeYAD3gme+3l9/yHJKDs7W7X+BFdTwg9lcfAmUCr03GmR+POp+Q8ENjvnPgIws2eBsUBZ8nfOLS1XfgUwMfj4cmCRc25f8NxFwGig6tAVkQQSadIPd52tM66ol+uJ1Bef0T4dgfIzaYqCx6pzM/C3SM41s0lmtsrMVu3Zs8cjJJHoKFizo94Sf3nRuKZIXfgk/1Dr44bsJTaziQSaeEqnbHqd65yb5ZzLc87ltW3b1iMkkfqXnf8Sd88PP2S2LtcvWLMjatcXiYRP8i8COpd73gnYWbmQmY0C7gOucs4dj+RckXiLVc387vmF+hYgCcEn+a8EuptZVzNrBEwAFpYvYGb9gd8SSPy7y730KvBlM2tpZi2BLwePJZW9e/eSm5tLbm4u7du3p2PHjmXPo7W65+rVq3nllVdCvrZ48WKaN29O//796dGjB8OHD+fll18ue/3xxx8vmyQVypIlS1ixYkW1r//5z38uW29n4sSJFBQUeMd96tQpZsz4v8FgJSUlXHzxxd7nx0M8krFuABJvYTt8nXPFZnYHgaSdCTzpnFtvZtOAVc65hQSaeU4Hng9O8d/mnLvKObfPzP6TwA0EYFpp528yad26ddms1wceeIDTTz/daxXNUiUlJWRmZkb0O1evXs26desYPXp0yNdHjhxZlpRXr17N1772Nf7whz8wfPjwChOmQlmyZAlt2rRh8ODBVV4rLi7ma1/7WkSxllea/PPz84HAEgz//Oc/a329aItnEs7Of0kdwRI3Xss7OOdeds71cM6d65x7KHjs/mDixzk3yjl3pnMuN/hzVblzn3TOdQv+/G90/oyKCtbsYOiMJXTNf4mhM5ZEtZ11zJgxfOlLX+K8885j9uzZQCCBtmjRgilTpjBw4EDefvttFi5cSM+ePbn44ou58847GTduHBCYPXzTTTcxcOBA+vfvz4svvsjRo0eZNm0a8+bNIzc3N+zs2gEDBnDfffeVrds/ZcoUHn30UQAeeeQR+vTpw/nnn8/EiRPZsmULs2fPZubMmeTm5vLWW28xceJEfvjDHzJy5Eh++tOfMnv2bO6+++6y67/66qtcfPHF9OjRg7/9LdCXX7nM6NGjeeONN8jPz+fgwYPk5uZyww03lP1bQODGcM8999C3b19ycnLK/q7Fixdz6aWXcvXVV9OzZ09uuOGG+vhfE1ZtEn8DC4zcqfxT2/H9+gYg8ZKcyzvUoGDNDu59YS1HT5YAsGP/Ue59YS0A4/rXNEipdubOnUurVq04cuQIeXl5fP3rX+eMM87gwIEDDBgwgOnTp3PkyBF69OjBm2++SZcuXfjGN75Rdv60adMYPXo0c+bM4fPPP2fQoEG899573H///axbt64siYczYMAA/ud//qfK8Z///Od8/PHHNGrUiP3799OiRQtuueUW2rRpU5a8n3jiCbZs2cJrr71GRkZG2U2s1Pbt21m2bBkffvgho0aNKltnKJQZM2Ywe/bssm9K5Rdee/7559mwYQPvvvsue/bs4YILLmDYsGFA4NvLhg0baNeuHYMHD2bFihUhv5nUl9ok3XC19NIbQOWx/j6xlF1bM4UlRlJuYbeZr24qS/yljp4sYearm6Ly+x555BHOP/98hgwZQlFREVu2bAECa/mXNp9s2LCBnj17cvbZZ2NmZUsfQ2Cjloceeojc3FxGjhzJsWPH2LZtW8RxVLdMx3nnncfEiROZN28eDRs2rPb8a6+9loyM0G+Hb3zjG2RkZNCzZ086d+5cYZG7SLzxxhtcf/31ZGZm0r59ey666CJKl/IYPHgwZ511FpmZmeTm5rJ169Za/Q4fkSb+0tq9r9p8C9AoIIm1lKv579x/NKLjdbF48WJef/11VqxYQZMmTbjooovKVvps0qRJ2RLHNa2f5JyjoKCgbLOUUq+//npEsYRaqhkCTTbLli3jL3/5C9OnT692RnG6LNVcOfHPXvBgyHK3XDMVqP3krEhvAHfPL4zKN1OR6qRczb9DiyYRHa+LAwcO0KpVK5o0acL69etZuXJlyHLnnXcemzZtYvv27TjnypY+Brj88sv51a9+VfZ8zZo1QGTLKRcWFvKzn/2sSkdvSUkJRUVFXHLJJcycOZM9e/Zw5MiRiJdqfv7553HO8cEHH7B9+3a6d+9OdnY2a9aswTnH1q1beeedd4DA3gBAyOQ9bNgwnn32WUpKSti1axdvvvlmnTaoj1Sv+14OX6ic+uiMjeQaav+XWEq55D/58p40aVhxZE2ThplMvrz+14O/4oorOHLkCOeffz7Tpk1j0KBBIcuddtppPPbYY4waNYqLL76YDh060Lx5cwCmTp3KkSNHynb7euCBBwC45JJLePfdd+nfv3/IDt+lS5fSv39/evbsyV133cUTTzzB8OHDK5QpLi7m+uuvp1+/fgwYMICf/OQnZUtBP/fcc/Tv35+33nor7N/ZrVs3hg0bxpgxY5g1axaNGjVi+PDhdOzYkZycHPLz88nNzS0rf/PNN9OvX78qHbfXXHMNvXr14vzzz2fUqFE8/PDDtGvXLuzvry/HSvxXsK3PUTiRXCvS/gKR2krJJZ0L1uxg5qub2Ln/KB1aNGHy5T3j/pX60KFDnH766Tjn+O53v0tOTg533nlnXGNKZZXfM9XVqkM1+4zqfWbkHaweHbU+NfvZCx7EgEsrNxupw1fCiMqSzslmXP+OcU/2lf36179m3rx5HD9+nLy8vCq7aUn0RNKcEs0lmbfOuKIslur6GqCatVNE6lnKNfskqsmTJ1NYWMjGjRv54x//SFZWVrxDSguDHlrkXTYWa/EPPbeVVzk1/0i0JU3N3zlXZdSJSCjlmzJ3HQwsv1FTTRtitwnLvFuH0NXzm8hrG3dVbf4RqSdJUfPPyspi7969NQ4VFIFA4t+7dy9ZWVnezT19OzSPclQV/duzA1jvdommpKj5d+rUiaKiIrTWv/jIysrioX986l2+ffMoNsFV0xE8qveZXk07izfu0taQEhVJkfwbNmxI165d4x2GJJHFv90QvhDx3XO3YYZx8lT4+v2Sjbu4JAbxSHpJiuQvUkF1wyoBXnyRbvf6NffEu0Y9vGc7r9r/qbAlRCKn5C8pp9ijsbxp4zBv/RgtsObb/KPln6W+KflLSlncbSCzwxdjyDmtox6Lr1ZNG7HvcHQ2BRKpTlKM9hHxsWzT7vCFiH9zT2UDurT0Kqe1f6Q+KflLyvDpPE1UvjekSBenE6mOmn0kJfjOiK1zrb+mzuYYiGRxOpGaqOYvaSPWk7ki5XtjUvOP1Aclf0l6vrX+qE7mqieJ1h8hqUvJX9JCqiVV1f6lrpT8Jaml4uqXqXajksSk5C8pLxmTadhJaKj2L3Wj5C9JKxVr/aUSaRKapCYN9ZSUloy1/lCq248gm/rdb1jSh2r+kpR8av2dWjaJQSTRkyo3LklMqvlLyurVvlm8Q6izxg0yOF5c/bqesxc8yOIFD1a9UWjDdwlDNX9JOj61/lZNG8Ugkui7uHvbeIcgKUrJX1KS72JpycCn+SqVO78lOpT8Jan4DG9MlVp/qVRovpLEo+QvKSeVav2lfNYlUu1fIqHkL0nDp9afqiNkkmFdIkkuSv4iSUK1f6lPSv6SFNK51l+qffMsfWCl3ni9l8xstJltMrPNZpYf4vVhZrbazIrN7JpKr5WYWWHwZ2F9BS6Sji7xuMGp9i8+wk7yMrNM4HHgMqAIWGlmC51zG8oV2wbcBPwoxCWOOudy6yFWSVM+tf5E36hFJNH41PwHApudcx85504AzwJjyxdwzm11zr0HVD8VUSSK0qlD1Gfcv1b8lHB8kn9HYHu550XBY76yzGyVma0ws3ERRSdpb9BDi8KWSbdav8b9S33wSf4W4lgku0h3cc7lAdcDj5rZuVV+gdmk4A1i1Z49eyK4tKS6XQdPhC2TTrX+Uj7r/XdV7V9q4JP8i4DO5Z53Anb6/gLn3M7gfz8C/gH0D1FmlnMuzzmX17at1jKRgG73pt9sXl8+6/1HUkOT9OOT/FcC3c2sq5k1AiYAXqN2zKylmTUOPm4DDAU21HyWSECxR/ZKxdm8vnw+vN/83fKoxyHJKez7xzlXDNwBvApsBJ5zzq03s2lmdhWAmV1gZkXAtcBvzWx98PTewCozexdYCsyoNEpIJKSCNTvClsm0UC2S6cNn2OebW/bFIBJJRl7r+TvnXgZernTs/nKPVxJoDqp83ltATh1jlDR09/zCsGVG9moXg0gSW6YZJa7mr0hTCtYyfZw+hlKRJgxKwvGp9euNG+BzA3xqxbYYRCLJRp8hSTg+tX6fJg8RqZ6SvyQdvWkr8lnTyGe+hKQXfY4kofgkKdX6I+czX0LSi5K/JBQlqdrxqf37zJuQ9KHkLwmj130vhy2zdcYVMYgkNfnMm5D0oeQvCeNYibJTXfiscdRv6isxiESSgZK/JASftv5Hx2tl8Jr4rHH0xfGSGEQiyUDJXxKCT1v/uP6RLCabnhpmhJ/17DOPQlKfkr/EnU8yatY4MwaRJL/hPcNP+vKZRyGpT8lf4s4nGb334OgYRJIa9KEWH15r+4hE0+wFD4Y8fss1UwFokN7rt0XMZx5Edv5LGjmV5pT8Ja6y819idjWvld4URvU+E8b8JnZBiaQBJX+RFLR1/W9YvHFXleOl36YgMOxTzWnpS82DEjc+k7p8Zq5K7WjYZ3pT8pe40aSu6PKZ9DWlYG0MIpFEpOQvceEzqcsneUn1fCZ9aa3/9KXkL3HhM6nLJ3lJzTq1bBK2jJZ7Tk9K/hJzPk0NPklLwuvVvlnYMlpJNT0p+UvM+TQ1+CQtEak9JX9JOJrTVb98Rkxl52ut/3Sj5C8x5ZNkLtXwznqXabqlSkWa5CWSBkb2asfijbuqXUoDoGB8rlZOTSNK/hI7Y8YwO8Ss0/I0qSt+7p5fqOSfRtTsIzETarkBiR2fEVTf/N3yGEQiiUDJX2LCZyy5hndGl88Iqje37ItBJJIIlPwlJnzGkmt4Z/T5dPxqp6/0oOQvCaFxA70VY2FkL+30JQH6xEnU+czovbh72xhEIqAbrQToXSBRF25Gr8agx5bPjVaTvlKfkr9EVbd7wycRn6YIEalfSv4SVcVhluxvmKFafzz4zKfQsM/UpuQvUeMzamR4T9X6E5WGfaY2JX+JGo0aSWw+8yo07DN1KflLVPg0GWjUSXz5zKvQDTx1ea3tY2ajgV8CmcBs59yMSq8PAx4F+gETnHMLyr12IzAl+HS6c25ufQQuCWDMmNDHX3zRq8lAwzvjr3GDDI4XnwKoftG3GVfEMCKJlbBVLzPLBB4HvgL0Aa4zsz6Vim0DbgKernRuK2AqMAgYCEw1s5Z1D1uSnWr9iUHDPtOXT81/ILDZOfcRgJk9C4wFNpQWcM5tDb52qtK5lwOLnHP7gq8vAkYDz9Q5cklYi7sNZHaYMqr1Jw4DwgzKkhTkU/3qCGwv97woeMyH17lmNsnMVpnZqj179nheWkTqg8/mOar9px6f5B9qILZvRcHrXOfcLOdcnnMur21b1QiT2T8/DH/z1pr9IvHnk/yLgM7lnncCdnpevy7nShIq7TysjqZ0JSafG/JlD/8j+oFIzPgk/5VAdzPramaNgAnAQs/rvwp82cxaBjt6vxw8Jilo2abdYctof97k9eHuw/EOQepR2OTvnCsG7iCQtDcCzznn1pvZNDO7CsDMLjCzIuBa4Ldmtj547j7gPwncQFYC00o7fyX1nDylbsNkpua49OI1zt859zLwcqVj95d7vJJAk06oc58EnqxDjJIEfLZo7NuheQwikWjKzn+JrRr3nxI02Fpipn3zrHiHIGG0atoobBkt+JYalPylzt7/9IuwZbQ/b3IY0CX8HEwt+JYalPylzoo+Pxq2jPbnTR4+s6+14FvyU/KXOvGp9Tdt7NW1JAnCZ/a1FnxLfkr+Uic+tf4h57SOQSRSn3w22fHZm1kSl5K/1JpPrV9vsOTks8lOuL2ZJbHpsym15lPrv0Rjx5OWav+pTclfokZt/clNtf/UpuQvtdLVY5VHtfUnP59x/5KclPylVsIt5KDNWlKDz7h/LfecnPQJlYj5fNi1WUvqyMrUWqypSMlf6l2mKVmkkvcf+mrYMur4TT5K/hIRn1r/yF7hOwoluYS7navjN/ko+YtIWP/2WMmz39RXYhCJ1Bclf/E26KFFYctoTfjU1SBM9f+L4yWxCUTqhZK/eNt18ESNr+vNlNo2/1f42r/PEGBJDJqFI158av2azZuixowpezi73KY9t1wztUpR7eWWPFRZEy+q9Qtoo/dUos+shKVav0RCG70nByV/qVHBmh1ha/2azZtefPZiVu0/8anNX2rks2mHZvOml/bNs9h/tOYKgWr/iU9VNqmWz0bdegOlJ59tObXmT2LTZ1eq5bNRt9r609fQc1uFLaO9fhOXkr+E5NNmq/X609u8W4eELaO9fhOXPr3yf8qN5/5JmPHcoPX6Bbq3axq2ff+bv1vudaOQ2FLylyqWf7S3wvPZCx6sUqZxgwxQR2/aW3TPiLBt+z7NhxJ7avaRKg4fLw5bRiN8pFSzxplhy2jRt8Sj5C8VVK71h+Kzsbekj/ceHB22jBZ9SzxK/lKBT63fZ2NvSS+Pjs8NW0ZDPxOLkr+UWVyuk7c6WrJZQhnXv6NXOQ39TBxK/gL4De3U9oxSk4mDu4Qto6GfiUPJXwC/6fjanlFqMn1cTtjtHkHr/iQKJX/x2oCjVdNGMYhEkp3Pdo9a9ycxaJx/mptSsNZrA44BXVpGPRZJMuUmBZY38eafhd3Qvd/UV7xGCUn0qOaf5sJ9SEFvEonM9HE5Ycto6Gf8eX2uzWy0mW0ys81mlh/i9cZmNj/4+r/MLDt4PNvMjppZYfDnN/UbvtSFb9urFm+TSPks+qahn/EVttnHzDKBx4HLgCJgpZktdM5tKFfsZuBz51w3M5sA/DcwPvjaFudc+EHAEjvBr+s/8RjaqbZ+qY15tw5h0EOLwm4EdNnD/2DRPSNiE5RU4FPzHwhsds595Jw7ATwLjK1UZiwwN/h4AXCpmcYFJrIlHonfUFu/1N6/7rssbJkPdx/W2P848enw7QhsL/e8CBhUXRnnXLGZHQBKl3zsamZrgC+AKc65f1b+BWY2CZgE0KVL+LHCUjefHjjGKY9yl6q5R2qjXEfw3E27OXkqMKSgutVh755f6D1JTOqPT80/VA2+8gCR6sp8AnRxzvUH7gGeNrMqWwA552Y55/Kcc3lt22rBsGhbt/NA2DKaySv1wXcpkEEPLYpyJFKZT/IvAjqXe94J2FldGTNrADQH9jnnjjvn9gI4594BtgA96hq01J5PJ5tm8kp96tSySdgyuw6e8No2VOqPT7PPSqC7mXUFdgATgOsrlVkI3AgsB64BljjnnJm1JXATKDGzc4DuwEf1Fr3UrNI47KXv72a2Cz+qXzN5pT71at+MPQePhy2ndf9jK2zN3zlXDNwBvApsBJ5zzq03s2lmdlWw2O+B1ma2mUDzTulw0GHAe2b2LoGO4Nucc/o/HCclHolfdX6JBt/9H7T0Q+x4zfB1zr0MvFzp2P3lHh8Drg1x3p+AP9UxRqkHr3mM7gF18kr0bJ1xRdhmRy39EDuavJkGlr6/22sJB3XySrRp16/EobV9Utzyj/Z6Nff4dMqJ1MmYMbxH1X0jKg8B/eJ4CV3zX/JaJE5qTzX/FPbpgWNeO3NBoFNOJBZ8Zo07NPwz2pT8U5jPeH5Qc4/E1oAuLb2GE+86eEKzf6NIyT9F+Y6Z7tuheZQjEalqZK92NG4QPv1o56/oUfJPQb3ue9lrzHSnlk1o3zwrBhGJVOU7/FMdwNGh5J9iet33MsdKwnfwZpqpnV/izuebZ2kHsNQvJf8U45P4QbN4JTG0b57lNbHQ4bfdqPhT8k8hvptjNG2sEb6SOHyHdDo0A7g+KfmnCN/E3zDDGHJO6/AFRWJoq+cN4MPdh7UAXD1R8k8BkWyH57vErkis+Wz9CIEF4DQEtO70/T+ZVFqlE4I7clWzSUZlGs8vCSn4vp5Hxdm/1W3+MnvBg7AA3m/ZpOKghRdfjGaUKUfJP4m9tnEXjuCHoQYNM0w1fkkKo3qfWfa+Dqfo86McOVGirUZrSc0+SWqx5wcE1NQjyeXS3md6LQEBsO/wCT49cCzKEaUmJUIMlEsAAAsUSURBVP8kVHlhrJpowTZJRgO6tKR7u6ZeZdftPMDqbZ9HOaLUo2afJPLpgWPe6/UANG6QoYlckrQWLf1/LCu3AXxN9h0+waCHFvGv+y6LQWSpQTX/JNFv6isRJf5WTRt5T58XSVTDe7bznpey6+AJzQOIgJJ/EsjOf4kvjpd4l+/bobk6wSRlDDmntXfz5Ye7D+sG4EnJP8FFOqW9VdNGWqxNUk4kzZcf7j5Mdv5LmgsQhpJ/Arvs4X94j+iBQOeuavySqiKdp3L3/EJtCFMDJf8ENKVgLdn5L3lvZp1B4IOhzl1JdZHeAHYdPEG3e7UgXChK/gmma/5LPLVim3d5Ay7RzF1JI6N6nxlR4ip2WhAuFCX/BHHZw/8gO/+liJp5jMCEGJF0c0nvM2mY4bMYdMCHuw9zjvoBKtA4/zgrWLOjylZ14ZZrgMAYfg3llHQ2vGe7iOa+nCLQD3D3/EImDu7C9HE50Q0wwSn5x9FlD//Du12/vL4dmmtEjwiBzWDaN89i6fu7IzrvqRXbeGrFNoae24p5tw6JUnSJTck/xr75u+Ve++tWR4lfpKqRvdrxx4L/5HjxqSqvVbc6KASWh87OfyktvwmYc5G0MkdfXl6eW7VqVbzDqHcFa3Zwz/xCqr41qwrV7KNmHpHw3v/0C4o+P+pVNtRN4dHxuYzr37H6k0Isqw4kxHLSZvaOcy7Pt7xq/lFW15o+BMbvaxinSHi92jejV/tmLP9oL4ePF0d8fmmfQMcWTZh8ec+abwRJTsk/SgrW7GDy84X8+rkH+XYtr9GqaSNN2hKphSHntK71DQBgx/6jZTeCVO0XUPKvJ6XJ/qRPu04YTRs30D67InVU+hn69MAx1u88ENEw6vJK+wUAZm/clTKfTyX/WipYs4MHX1zP50dO1ts1U+VNJZJISkcErd72OfsOn6jz9Q4fL66wp0anlk3oVeerxp6Sv4cpBWuZt2JbWc3BoNa1iFAyzOhzVjON4hGJotIm1NKbQHXzaWoaHRRK0edH+UpwgmYy9RUo+VdSsGYHM1/dxM79R+nQognZrZtU6LCtrzdMKbXri8RW6eetuo1ifCZZVlZ6lR37j/KD+YWs+ngfeWe3qpBLEu2mkLbJP1SSX/7RPsq/F3bsP8qO/X7DxiJ5wxjQUSN4ROKq8t7Wnx44xoZPvuBUHYe/OwKTyOav3M7JksC1duw/yr0vrC0rU/mmsOrjfTzzr+2UOEemGdcN6hz1eQdeyd/MRgO/BDKB2c65GZVebwz8AfgSsBcY75zbGnztXuBmoAS4yzn3ar1FX07lZD758p5A1X/kcf07UrBmB/e+sJajJwMbpESS5Gsjq2Em3dqermYdkQRW2jdQl1FC5ZUm/lJHT5bw4IvrOXbyVIXcU3n+T4lzZYs7RvMGEDb5m1km8DhwGVAErDSzhc65DeWK3Qx87pzrZmYTgP8GxptZH2ACcB7QAVhsZj2cc/7bUnkIlcwnL3gXHGVf68rfeWe+uqmsbH0345SnJh2R5DPknNZ8euAYm/cc4vjJknrt3ws1QKS6AYLP/Gt7fJM/MBDY7Jz7CMDMngXGAuWT/1jggeDjBcBjZmbB4886544D/zazzcHrLa+f8AN8k/nRkyVl3wTCqU27XylNyhJJbqXfAkqF20g+0kqkT/mSKK++4JP8OwLbyz0vAgZVV8Y5V2xmB4DWweMrKp1b7z0ePsm8fNkOLZp4N/NkZhjNmzTk88MnKoz2gUDbnmF0bJmlZC+Swir3DwBl3w6OnazakJFhgSxRPslnZhgZZpws8ZsMlGn+S1bXhk/yDxVB5VtSdWV8zsXMJgGTALp06eIRUkXlk3m45prStv/SZqKayjdpmMF/Xd2PkQnUQy8iiaF98Af8+xyBCk3UENhUJdTt4LpBnaMav0/yLwLKR9EJ2FlNmSIzawA0B/Z5notzbhYwCwILu/kGX6p8Mi/VMNMqtPkDNGmYWWG4VeXRPis++jymve0ikhrG9e8YchhndUM7E2G0T9hVPYPJ/APgUmAHsBK43jm3vlyZ7wM5zrnbgh2+VzvnvmFm5wFPE2jn7wC8BnSvqcO3tqt6RjLaR0Qk1dT7qp7BNvw7gFcJDPV80jm33symAauccwuB3wN/DHbo7iMwwodguecIdA4XA9+v75E+pSK984qIpDOt5y8ikgIirflrA3cRkTSk5C8ikoaU/EVE0pCSv4hIGlLyFxFJQ0r+IiJpSMlfRCQNKfmLiKQhJX8RkTSk5C8ikoaU/EVE0pCSv4hIGlLyFxFJQ0r+IiJpSMlfRCQNKfmLiKShhNvMxcz2AB+HKdYG+CwG4dS3ZI0bkjd2xR17yRp7ssYNgdibOufa+p6QcMnfh5mtimTHmkSRrHFD8sauuGMvWWNP1rihdrGr2UdEJA0p+YuIpKFkTf6z4h1ALSVr3JC8sSvu2EvW2JM1bqhF7EnZ5i8iInWTrDV/ERGpAyV/EZE0lNTJ38x+ZGbOzNrEOxZfZjbTzN43s/fM7M9m1iLeMdXEzEab2SYz22xm+fGOx5eZdTazpWa20czWm9l/xDumSJhZppmtMbO/xjuWSJhZCzNbEHyPbzSzIfGOyYeZ/SD4PllnZs+YWVa8Y6qOmT1pZrvNbF25Y63MbJGZfRj8b8tw10na5G9mnYHLgG3xjiVCi4C+zrl+wAfAvXGOp1pmlgk8DnwF6ANcZ2Z94huVt2Lgh8653sBg4PtJFDvAfwAb4x1ELfwSeMU51ws4nyT4G8ysI3AXkOec6wtkAhPiG1WN5gCjKx3LB15zznUHXgs+r1HSJn/gEeDHQFL1WDvn/u6cKw4+XQF0imc8YQwENjvnPnLOnQCeBcbGOSYvzrlPnHOrg48PEkhCHeMblR8z6wRcAcyOdyyRMLNmwDDg9wDOuRPOuf3xjcpbA6CJmTUATgN2xjmeajnnXgf2VTo8FpgbfDwXGBfuOkmZ/M3sKmCHc+7deMdSR98B/hbvIGrQEdhe7nkRSZJAyzOzbKA/8K/4RuLtUQIVm1PxDiRC5wB7gP8NNlnNNrOm8Q4qHOfcDuAXBFoRPgEOOOf+Ht+oInamc+4TCFR8gHbhTkjY5G9mi4Ptb5V/xgL3AffHO8bqhIm9tMx9BJom5sUv0rAsxLGk+qZlZqcDfwLuds59Ee94wjGzK4Hdzrl34h1LLTQABgC/ds71Bw7j0fwQb8H28bFAV6AD0NTMJsY3quhrEO8AquOcGxXquJnlEPif9K6ZQaDZZLWZDXTOfRrDEKtVXeylzOxG4ErgUpfYEy2KgM7lnncigb8OV2ZmDQkk/nnOuRfiHY+nocBVZvZVIAtoZmZPOeeSIRkVAUXOudJvWAtIguQPjAL+7ZzbA2BmLwAXAk/FNarI7DKzs5xzn5jZWcDucCckbM2/Os65tc65ds65bOdcNoE33IBESfzhmNlo4CfAVc65I/GOJ4yVQHcz62pmjQh0gi2Mc0xeLFAz+D2w0Tn3cLzj8eWcu9c51yn43p4ALEmSxE/wM7jdzHoGD10KbIhjSL62AYPN7LTg++ZSkqCjupKFwI3BxzcCfwl3QsLW/FPYY0BjYFHwm8sK59xt8Q0pNOdcsZndAbxKYATEk8659XEOy9dQ4FvAWjMrDB77qXPu5TjGlA7uBOYFKwsfAd+OczxhOef+ZWYLgNUEmmLXkMBLPZjZM8AIoI2ZFQFTgRnAc2Z2M4Gb2bVhr5PYrQ4iIhINSdfsIyIidafkLyKShpT8RUTSkJK/iEgaUvIXEUlDSv4iImlIyV9EJA39f6eBBVAiurkzAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from scipy.stats import norm\n", "\n", "\n", "def norm_dist_prob(theta):\n", " y = norm.pdf(theta, loc=3, scale=2)\n", " return y\n", "\n", "\n", "T = 5000\n", "pi = [0 for i in range(T)]\n", "sigma = 1\n", "t = 0\n", "while t < T - 1:\n", " t = t + 1\n", " pi_star = norm.rvs(loc=pi[t - 1], scale=sigma, size=1,\n", " random_state=None) #状态转移进行随机抽样\n", " alpha = min(\n", " 1, (norm_dist_prob(pi_star[0]) / norm_dist_prob(pi[t - 1]))) #alpha值\n", "\n", " u = random.uniform(0, 1)\n", " if u < alpha:\n", " pi[t] = pi_star[0]\n", " else:\n", " pi[t] = pi[t - 1]\n", "\n", "plt.scatter(pi, norm.pdf(pi, loc=3, scale=2), label='Target Distribution')\n", "num_bins = 50\n", "plt.hist(pi,\n", " num_bins,\n", " density=1,\n", " facecolor='red',\n", " alpha=0.7,\n", " label='Samples Distribution')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "RRUK6UCjKa4Y" }, "source": [ "#### 二维Gibbs采样实例python实现 \n", "\n", "假设我们要采样的是一个二维正态分布 $N(\\mu, \\Sigma)$ ，其中： $\\mu=(\\mu_{1}, \\mu_{2})= (5, -1)$ , $\\Sigma = \\begin{pmatrix}\n", "\\sigma^{2}_{1} & \\rho \\sigma_{1}\\sigma_{2}b\\rho \\sigma_{2}& \n", "\\sigma^{2}_{2}\\end{pmatrix} = \\begin{pmatrix}\n", " 1& 1b1 & \n", "4\\end{pmatrix}$;\n", "\n", "而采样过程中的需要的状态转移条件分布为：\n", "\n", "$P(x_{1}|x_{2}) = N(\\mu_{1}+ \\rho \\sigma_{1}/\\sigma_{2}(x_{2} - \\mu_{2}), (1 - \\rho^{2})\\sigma^{2}_{1})$\n", "\n", "$P(x_{2}|x_{1}) = N(\\mu_{2}+ \\rho \\sigma_{2}/\\sigma_{1}(x_{1} - \\mu_{1}), (1 - \\rho^{2})\\sigma^{2}_{2})$" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 335 }, "colab_type": "code", "id": "3ZSOvaDcMHfv", "outputId": "0c3d6e18-ba4e-4e06-b61f-2585cc78715d" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEICAYAAAC+iFRkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAcOUlEQVR4nO3df5BV5Z3n8fcnrciOGn8AMYYGabWdCRpLMi0mZscQf2JmAjplNvijxJUUZSKjM6614mpIB8upaGbdSUoSZSaUmo1BYzbZ3hSGmCgzmx0x3cafoCwtorT4g4BKiID8+O4f57R7uNzbfbr79u3bfT6vqluc85znOfd7T1++97nPOfc5igjMzGxk+9BQB2BmZoPPyd7MrACc7M3MCsDJ3sysAJzszcwKwMnezKwAnOxtRJK0StK0oY7DrF442duwJGm9pLNLyq6Q9BuAiDgxIlb0so9JkkLSAYMYqlldcLI3GyT+ELF64mRvI1K25y9pqqQOSVslvSnpjrTav6b/viNpm6RPS/qQpJslvSLpLUn3SToss9/L022bJX2t5HlaJT0k6b9L2gpckT7345LekfS6pDsljcrsLyR9VdJaSX+QdIuk49I2WyU9mK1v1l9O9lYE3wa+HREfBo4DHkzLz0j/PTwiDomIx4Er0sfngGOBQ4A7ASRNBr4LXAocDRwGjC95rpnAQ8DhwA+BPcDfAWOBTwNnAV8taTMd+HPgU8B/BhanzzEBOAm4eACv3Qxwsrfh7Wdpj/kdSe+QJOJydgHHSxobEdsiYmUP+7wUuCMi1kXENuBGYFY6JHMR8L8i4jcR8T6wACidXOrxiPhZROyNiO0R8WRErIyI3RGxHrgb+GxJm9siYmtErAKeB36ZPv+7wMPAlPyHxKw8J3sbzi6IiMO7H+zfY+42BzgBeFFSu6S/6mGfHwNeyay/AhwAHJVu29C9ISLeAzaXtN+QXZF0gqSfS3ojHdr5e5JeftabmeXtZdYP6SFes1yc7G3Ei4i1EXEx8BHgNuAhSQezf68cYCNwTGZ9IrCbJAG/DjR2b5D074AxpU9Xsv494EWgOR1G+i+A+v9qzPrHyd5GPEmXSRoXEXuBd9LiPcAmYC/J2Hy3HwF/J6lJ0iEkPfEHImI3yVj8FySdnp40/Qa9J+5Dga3ANkl/Bnylai/MrA+c7K0IpgOrJG0jOVk7KyJ2pMMwtwL/Jx33/xSwBPgByZU6LwM7gL8BSMfU/wZYStLL/wPwFrCzh+e+HrgkrftPwAPVf3lmvZNvXmLWP2nP/x2SIZqXhzoes564Z2/WB5K+IOlP0jH/fwCeA9YPbVRmvXOyN+ubmSQncTcCzSRDQv56bHXPwzhmZgXgnr2ZWQHU3URNY8eOjUmTJg11GGZmw8qTTz75+4gYV2l73SX7SZMm0dHRMdRhmJkNK5Je6Wm7h3HMzArAyd7MrACc7M3MCqDuxuzNzIbKrl276OrqYseOHUMdSkWjR4+msbGRAw88sE/tnOzNzFJdXV0ceuihTJo0Can+JieNCDZv3kxXVxdNTU19authHDOz1I4dOxgzZkxdJnoASYwZM6Zf3zxyJXtJ0yWtkdQpaX4P9S5K76nZkim7MW23RtJ5fY7QzKyG6jXRd+tvfL0O40hqABYB5wBdQLuktohYXVLvUOAa4IlM2WRgFnAiyV1+fiXphIjY069ozcysX/KM2U8FOiNiHYCkpSSTQa0uqXcLcDvJ/N3dZgJLI2In8LKkznR/jw80cDOzwda6orW6+5tW3f31RZ5kP55976vZBZyWrSBpCjAhIn4u6fqStitL2o4vfQJJc4G5ABMnTswXuZkNC9mEOZTJrujyjNmXGyD6YKpMSR8C/hvwn/ra9oOCiMUR0RIRLePGVZzawcyGudYVrVXvLY807e3tnHzyyezYsYM//vGPnHjiiTz//PMD3m+enn0XMCGz3kgyl3e3Q4GTgBXpiYOPAm2SZuRoa2ZmGaeeeiozZszg5ptvZvv27Vx22WWcdNJJA95vnmTfDjRLagJeIznhekn3xoh4FxjbvS5pBXB9RHRI2g7cL+kOkhO0zcBvBxy1mdkItmDBAk499VRGjx7Nd77znarss9dkHxG7Jc0DlgMNwJKIWCVpIdAREW09tF0l6UGSk7m7gat9JY6ZWc+2bNnCtm3b2LVrFzt27ODggw8e8D5z/YI2IpYBy0rKFlSoO61k/Vbg1n7GZ2ZWOHPnzuWWW27h5Zdf5oYbbuDOO+8c8D49XYKZVd1IOQk7FFcP3XfffRxwwAFccskl7Nmzh9NPP51HH32UM888c0D7dbI3M6sjl19+OZdffjkADQ0NPPHEE720yMdz45iZFYCTvZlZATjZm5kVgJO9mVkBONmbmRWAr8Yxs6oYKZdbjlRO9mZmlbS21vf++sDDOGZmBeCevZnVnOe4r+xrX/saY8eO5dprrwXgpptu4qijjuKaa64Z0H7dszczqyNz5szh3nvvBWDv3r0sXbqUSy+9dMD7dc/ezKyOTJo0iTFjxvDUU0/x5ptvMmXKFMaMGTPg/TrZm5nVmS9/+cvcc889vPHGG1x55ZVV2aeHcczM6syFF17IL37xC9rb2znvvPOqsk/37M3MKhmiSyVHjRrF5z73OQ4//HAaGhqqss9cPXtJ0yWtkdQpaX6Z7VdJek7S05J+I2lyWj5J0va0/GlJd1UlajOzEWzv3r2sXLmSOXPmVG2fvSZ7SQ3AIuB8YDJwcXcyz7g/Ij4REacAtwN3ZLa9FBGnpI+rqhW4mdlItHr1ao4//njOOussmpubq7bfPMM4U4HOiFgHIGkpMJPkvrIARMTWTP2DgahahGZmBTJ58mTWrVtX9f3mGcYZD2zIrHelZfuQdLWkl0h69tmr/5skPSXpXyT9xYCiNTMbZBH13Vftb3x5kr3KPV+ZABZFxHHADcDNafHrwMSImAJcB9wv6cP7PYE0V1KHpI5Nmzblj97MrIpGjx7N5s2b6zbhRwSbN29m9OjRfW6bZxinC5iQWW8ENvZQfynwvTSwncDOdPnJtOd/AtCRbRARi4HFAC0tLfV5lM1sxGtsbKSrq4t67nSOHj2axsbGPrfLk+zbgWZJTcBrwCzgkmwFSc0RsTZd/UtgbVo+DtgSEXskHQs0A9UfjDKzITHSpjU+8MADaWpqGuowBkWvyT4idkuaBywHGoAlEbFK0kKgIyLagHmSzgZ2AW8Ds9PmZwALJe0G9gBXRcSWwXghZmZWmeptbKqlpSU6Ojp6r2hmQ67aPXvPgNl/kp6MiJZK2z1dgplZATjZm5kVgJO9mVkBONmbmRWAk72ZWQE42ZuZFYCTvZlZATjZm5kVgJO9mVkBONmbmRWAk72ZWQE42ZuZFYCTvZlZATjZm5kVgJO9mVkBONmbmRVAntsSmpntY6TdjrAIcvXsJU2XtEZSp6T5ZbZfJek5SU9L+o2kyZltN6bt1kg6r5rBm9nI0rqi9YOHVVevyV5SA7AIOB+YDFycTeap+yPiExFxCnA7cEfadjLJDcpPBKYD3033Z2ZmNZSnZz8V6IyIdRHxPrAUmJmtEBFbM6sHA903tp0JLI2InRHxMtCZ7s/MzGooz5j9eGBDZr0LOK20kqSrgeuAUcCZmbYrS9qO71ekZmbWb3l69ipTFvsVRCyKiOOAG4Cb+9JW0lxJHZI6Nm3alCMkMzPrizzJvguYkFlvBDb2UH8pcEFf2kbE4ohoiYiWcePG5QjJzMz6Ik+ybweaJTVJGkVywrUtW0FSc2b1L4G16XIbMEvSQZKagGbgtwMP28zM+qLXMfuI2C1pHrAcaACWRMQqSQuBjohoA+ZJOhvYBbwNzE7brpL0ILAa2A1cHRF7Bum1mJlZBbl+VBURy4BlJWULMsvX9tD2VuDW/gZoZmYD5+kSzMwKwMnezKwAnOzNzArAyd7MrACc7M3MCsDJ3sysAJzszcwKwMnezKwAfKcqM6tL2RuYtE5rrVjP8nHP3sysAJzszcwKwMnezKwAnOzNzArAyd7MrAB8NY6Z5ZK9OsaGH/fszcwKwMnezKwAciV7SdMlrZHUKWl+me3XSVot6VlJv5Z0TGbbHklPp4+20rZmZjb4eh2zl9QALALOAbqAdkltEbE6U+0poCUi3pP0FeB24Evptu0RcUqV4zYzsz7I07OfCnRGxLqIeB9YCszMVoiIxyLivXR1JdBY3TDNzGwg8iT78cCGzHpXWlbJHODhzPpoSR2SVkq6oFwDSXPTOh2bNm3KEZKZmfVFnksvVaYsylaULgNagM9miidGxEZJxwKPSnouIl7aZ2cRi4HFAC0tLWX3bWZm/ZenZ98FTMisNwIbSytJOhu4CZgRETu7yyNiY/rvOmAFMGUA8ZqZWT/kSfbtQLOkJkmjgFnAPlfVSJoC3E2S6N/KlB8h6aB0eSzwGSB7YtfMzGqg12GciNgtaR6wHGgAlkTEKkkLgY6IaAO+BRwC/FgSwKsRMQP4OHC3pL0kHyzfLLmKx8zMaiDXdAkRsQxYVlK2ILN8doV2/wZ8YiABmpnZwPkXtGZmBeBkb2ZWAE72ZmYF4GRvZlYATvZmZgXgZG9mVgBO9mZmBeBkb2Z1r3VFq2+LOEBO9mZmBeAbjptZRe5Njxzu2ZuZFYCTvZlZATjZm5kVgJO9mVkBONmbmRWAk72ZWQE42ZuZFUCuZC9puqQ1kjolzS+z/TpJqyU9K+nXko7JbJstaW36mF3N4M3MLJ9ek72kBmARcD4wGbhY0uSSak8BLRFxMvAQcHva9kjg68BpwFTg65KOqF74ZmaWR56e/VSgMyLWRcT7wFJgZrZCRDwWEe+lqyuBxnT5POCRiNgSEW8DjwDTqxO6mZnllWe6hPHAhsx6F0lPvZI5wMM9tB1f2kDSXGAuwMSJE3OEZNZHra19KzcbYfIke5Upi7IVpcuAFuCzfWkbEYuBxQAtLS1l922WmxO42X7yDON0ARMy643AxtJKks4GbgJmRMTOvrQ1M7PBladn3w40S2oCXgNmAZdkK0iaAtwNTI+ItzKblgN/nzkpey5w44CjNquWct8C/M3ARqBek31E7JY0jyRxNwBLImKVpIVAR0S0Ad8CDgF+LAng1YiYERFbJN1C8oEBsDAitgzKKzEzs4pyzWcfEcuAZSVlCzLLZ/fQdgmwpL8BmlXkHrhZbr55iZnto55vWJKNrXVaa8V6tj9Pl2BmVgDu2ZuV8klbG4HcszczKwAnezOzAvAwjg0PHkYxGxD37M3MCsDJ3sysAJzszcwKwGP2Znl4imQb5tyzNzMrACd7M7MCcLI3MysAJ3szswJwsjczKwBfjWP1x1e4mFWde/ZmZgWQq2cvaTrwbZLbEv5zRHyzZPsZwD8CJwOzIuKhzLY9wHPp6qsRMaMagZvVBU+HbMNEr8leUgOwCDgH6ALaJbVFxOpMtVeBK4Dry+xie0ScUoVYzcw+4LtW9U2env1UoDMi1gFIWgrMBD5I9hGxPt22dxBiNDOzAcozZj8e2JBZ70rL8hotqUPSSkkXlKsgaW5ap2PTpk192LWZmeWRJ9mrTFn04TkmRkQLcAnwj5KO229nEYsjoiUiWsaNG9eHXZuZWR55kn0XMCGz3ghszPsEEbEx/XcdsAKY0of4zMysCvKM2bcDzZKagNeAWSS99F5JOgJ4LyJ2ShoLfAa4vb/BmtngyJ7stJGp1559ROwG5gHLgReAByNilaSFkmYASDpVUhfwReBuSavS5h8HOiQ9AzwGfLPkKh4zM6uBXNfZR8QyYFlJ2YLMcjvJ8E5pu38DPjHAGG2k8vXoZjXjX9CamRWAk72ZWQF4IjSzavMtDK0OuWdvZlYATvZmZgXgZG9mVgBO9mZmBeBkb2ZWAE72ZmYF4GRvZlYATvZmZgXgH1VZbfgHRWZDysnerFZ8c3IbQk72ZgU2Uuax983He+cxezOzAnCyNzMrACd7M7MCyJXsJU2XtEZSp6T5ZbafIel3knZLuqhk22xJa9PH7GoFbmZm+fWa7CU1AIuA84HJwMWSJpdUexW4Ari/pO2RwNeB04CpwNfTm5CbmVkN5bkaZyrQGRHrACQtBWYCH9w4PCLWp9v2lrQ9D3gkIrak2x8BpgM/GnDkVr98OaFZ3ckzjDMe2JBZ70rL8sjVVtJcSR2SOjZt2pRz12ZmlleeZK8yZZFz/7naRsTiiGiJiJZx48bl3LWZmeWVJ9l3ARMy643Axpz7H0hbMzOrkjzJvh1oltQkaRQwC2jLuf/lwLmSjkhPzJ6blpmZWQ31eoI2InZLmkeSpBuAJRGxStJCoCMi2iSdCvwUOAL4gqRvRMSJEbFF0i0kHxgAC7tP1poZni/HaibX3DgRsQxYVlK2ILPcTjJEU67tEmDJAGI0M7MB8i9ozcwKwMnezEaU1hWtI2Y2z2ryFMdmBeNEWEzu2ZuZFYB79tZ/vmrEbNhwz97MrACc7M3MCsDJ3sysADxmb1ZvKp0L8TkSGwD37M3MCsDJ3sysAJzszcwKwMnezKwAfILW8vHJQRtmstNCtE5rrVivKJzszQrA8+GYk73ZcOEbndgAeMzezKwAciV7SdMlrZHUKWl+me0HSXog3f6EpElp+SRJ2yU9nT7uqm74ZmaWR6/DOJIagEXAOUAX0C6pLSJWZ6rNAd6OiOMlzQJuA76UbnspIk6pctxmZtYHeXr2U4HOiFgXEe8DS4GZJXVmAvemyw8BZ0lS9cI0M7OByHOCdjywIbPeBZxWqU5E7Jb0LjAm3dYk6SlgK3BzRPzv0ieQNBeYCzBx4sQ+vQCrMp/wMxuR8vTsy/XQI2ed14GJETEFuA64X9KH96sYsTgiWiKiZdy4cTlCMjOzvsjTs+8CJmTWG4GNFep0SToAOAzYEhEB7ASIiCclvQScAHQMNHAzs7z8A6t8Pft2oFlSk6RRwCygraROGzA7Xb4IeDQiQtK49AQvko4FmoF11QndzMzy6rVnn47BzwOWAw3AkohYJWkh0BERbcD3gR9I6gS2kHwgAJwBLJS0G9gDXBURWwbjhZgVUg8/tPKvZi0r1y9oI2IZsKykbEFmeQfwxTLtfgL8ZIAxmpnZAHm6hCLzlTdmheHpEszMCsDJ3sysADyMYzbSpMNz09av2Kd4xRXTah6K1Q/37M3MCsA9e7MRZkVJj972VdQfWDnZF4WvvDErNA/jmJkVgHv2ZgUx7Z4V+5X5pG1xONmbWWEVafzeyd5sBPBJWeuNk/1I4xOxZlaGk71ZgZUbxweP5Y9EvhrHzKwAlNxMqn60tLRER4dvZJWLh2wKrdbj9EXp7Q/XE7WSnoyIlkrb3bM3MysA9+yHC/fijfq76mak9/aHUy+/t559rhO0kqYD3ya5LeE/R8Q3S7YfBNwH/DmwGfhSRKxPt90IzCG5LeE1EbG8H6+jOJzUzerGSLoOv9eefXrD8P8LnAN0kdyA/OKIWJ2p81Xg5Ii4StIs4MKI+JKkycCPgKnAx4BfASdExJ5Kz1eonr0Tu/Wg3nrxAzWSvgXUY+KvRs9+KtAZEevSHS4FZgKrM3VmAq3p8kPAnZKUli+NiJ3Ay+kNyacCj/f1hdQlJ2urgpGW1CupdJlnOfX+wTAce/x5kv14YENmvQs4rVKdiNgt6V1gTFq+sqTt+NInkDQXmJuubpO0JkdcY4Hf56g3FOo5Nqjv+Bxb/9VzfH2L7d5/GbxIyuv3sfsG36hyKPvJG9sxPW3Mk+xVpqx07KdSnTxtiYjFwOIcsfz/J5Q6evrKMpTqOTao7/gcW//Vc3z1HBvUd3zVii3PpZddwITMeiOwsVIdSQcAhwFbcrY1M7NBlifZtwPNkpokjQJmAW0lddqA2enyRcCjkZz5bQNmSTpIUhPQDPy2OqGbmVlevQ7jpGPw84DlJJdeLomIVZIWAh0R0QZ8H/hBegJ2C8kHAmm9B0lO5u4Gru7pSpw+6tOwT43Vc2xQ3/E5tv6r5/jqOTao7/iqElvd/ajKzMyqz9MlmJkVgJO9mVkB1HWyl/RFSask7ZXUUrLtRkmdktZIOq9C+yZJT0haK+mB9ATzYMT5gKSn08d6SU9XqLde0nNpvZr9TFhSq6TXMjF+vkK96enx7JQ0v0axfUvSi5KelfRTSYdXqFezY9fbcUgvOHgg3f6EpEmDGU/Jc0+Q9JikF9L/G9eWqTNN0ruZv/eCGsbX499Jie+kx+5ZSZ+sUVx/mjkeT0vaKulvS+rU9LhJWiLpLUnPZ8qOlPRImrMekXREhbaz0zprJc0uV2c/EVG3D+DjwJ8CK4CWTPlk4BngIKAJeAloKNP+QWBWunwX8JUaxPxfgQUVtq0Hxg7BcWwFru+lTkN6HI8FRqXHd3INYjsXOCBdvg24bSiPXZ7jAHwVuCtdngU8UMO/5dHAJ9PlQ0mmMimNbxrw81q/z/L8nYDPAw+T/AbnU8ATQxBjA/AGcMxQHjfgDOCTwPOZstuB+eny/HL/H4AjgXXpv0eky0f09nx13bOPiBciotyvaT+YhiEiXga6p2H4QDpdw5kk0zcA3AtcMJjxps/5H0jmAxpuPpgWIyLeB7qnxRhUEfHLiNidrq4k+S3GUMpzHGaSvJ8geX+dlf7tB11EvB4Rv0uX/wC8QJlfpdexmcB9kVgJHC7p6BrHcBbwUkS8UuPn3UdE/CvJ1YtZ2fdWpZx1HvBIRGyJiLeBR4DpvT1fXSf7HpSbwqH0DT8GeCeTSMpO1VBlfwG8GRFrK2wP4JeSnkyniKileenX5iUVvhrmOaaD7UqSXl85tTp2eY7DPtODAN3Tg9RUOnw0BXiizOZPS3pG0sOSTqxhWL39nerhfTaLyh2yoTpu3Y6KiNch+WAHPlKmTr+O4ZDfg1bSr4CPltl0U0T8z0rNypTlncKhX3LGeTE99+o/ExEbJX0EeETSi+mn+4D1FB/wPeAWktd/C8lQ05WluyjTtirX5eY5dpJuIvktxg8r7GbQjl1puGXKBvW91R+SDgF+AvxtRGwt2fw7kiGKben5mZ+R/KCxFnr7Ow3psUvP280AbiyzeSiPW1/06xgOebKPiLP70SzPNAy/J/mKeEDa+xrQVA29xalkmoi/JpnTv9I+Nqb/viXppyRDBlVJWHmPo6R/An5eZtOgTW2R49jNBv4KOCvSQcky+xi0Y1eiL9ODdGnf6UFqQtKBJIn+hxHxP0q3Z5N/RCyT9F1JYyNi0CdJy/F3GuopVM4HfhcRb5ZuGMrjlvGmpKMj4vV0eOutMnW6SM4vdGskOa/Zo+E6jNPrNAxp0niMZPoGSKZzqPRNoRrOBl6MiK5yGyUdLOnQ7mWSE5PPl6tbbSVjohdWeN4802IMRmzTgRuAGRHxXoU6tTx2A5keZNCl5wa+D7wQEXdUqPPR7nMIkqaS/D/fXIPY8vyd2oDL06tyPgW82z1sUSMVv30P1XErkX1vVcpZy4FzJR2RDsmem5b1rFZnnvt5tvpCkk+xncCbwPLMtptIrppYA5yfKV8GfCxdPpbkQ6AT+DFw0CDGeg9wVUnZx4BlmVieSR+rSIYwanUcfwA8BzybvpmOLo0vXf88ydUdL9UqvvRvswF4On3cVRpbrY9dueMALCT5QAIYnb6fOtP317E1/Fv+e5Kv7M9mjtnngau633/AvPQ4PUNy0vv0GsVW9u9UEpuARemxfY7MVXY1iO9PSJL3YZmyITtuJB86rwO70jw3h+Tcz6+Btem/R6Z1W0juEtjd9sr0/dcJ/Mc8z+fpEszMCmC4DuOYmVkfONmbmRWAk72ZWQE42ZuZFYCTvZlZATjZm5kVgJO9mVkB/D/VaN1LAduNJAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from mpl_toolkits.mplot3d import Axes3D\n", "from scipy.stats import multivariate_normal\n", "\n", "samplesource = multivariate_normal(mean=[5,-1], cov=[[1,0.5],[0.5,2]])\n", "\n", "def p_ygivenx(x, m1, m2, s1, s2):\n", " return (random.normalvariate(m2 + rho * s2 / s1 * (x - m1), math.sqrt(1 - rho ** 2) * s2))\n", "\n", "def p_xgiveny(y, m1, m2, s1, s2):\n", " return (random.normalvariate(m1 + rho * s1 / s2 * (y - m2), math.sqrt(1 - rho ** 2) * s1))\n", "\n", "N = 5000\n", "K = 20\n", "x_res = []\n", "y_res = []\n", "z_res = []\n", "m1 = 5\n", "m2 = -1\n", "s1 = 1\n", "s2 = 2\n", "\n", "rho = 0.5\n", "y = m2\n", "\n", "for i in range(N):\n", " for j in range(K):\n", " x = p_xgiveny(y, m1, m2, s1, s2) #y给定得到x的采样\n", " y = p_ygivenx(x, m1, m2, s1, s2) #x给定得到y的采样\n", " z = samplesource.pdf([x,y])\n", " x_res.append(x)\n", " y_res.append(y)\n", " z_res.append(z)\n", "\n", "num_bins = 50\n", "plt.hist(x_res, num_bins,density=1, facecolor='green', alpha=0.5,label='x')\n", "plt.hist(y_res, num_bins, density=1, facecolor='red', alpha=0.5,label='y')\n", "plt.title('Histogram')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab": {}, "colab_type": "code", "id": "OfldxGWecECT" }, "source": [ "----\n", "本章代码来源：https://github.com/hktxt/Learn-Statistical-Learning-Method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "MCMC.ipynb", "provenance": [], "version": "0.3.2" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 1 } ================================================ FILE: 第20章 潜在狄利克雷分配/20.LDA.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第20章 潜在狄利克雷分配" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1.狄利克雷分布的概率密度函数为$$p ( \\theta | \\alpha ) = \\frac { \\Gamma ( \\sum _ { i = 1 } ^ { k } \\alpha _ { i } ) } { \\prod _ { i = 1 } ^ { k } \\Gamma ( \\alpha _ { i } ) } \\prod _ { i = 1 } ^ { k } \\theta _ { i } ^ { \\alpha _ { i } - 1 }$$\n", "其中$\\sum _ { i = 1 } ^ { k } \\theta _ { i } = 1 , \\theta _ { i } \\geq 0 , \\alpha = ( \\alpha _ { 1 } , \\alpha _ { 2 } , \\cdots , \\alpha _ { k } ) , \\alpha _ { i } > 0 , i = 1,2 , \\cdots , $狄利克雷分布是多项分布的共轭先验。\n", "\n", "2.潜在狄利克雷分配2.潜在狄利克雷分配（LDA）是文本集合的生成概率模型。模型假设话题由单词的多项分布表示，文本由话题的多项分布表示，单词分布和话题分布的先验分布都是狄利克雷分布。LDA模型属于概率图模型可以由板块表示法表示LDA模型中，每个话题的单词分布、每个文本的话题分布、文本的每个位置的话题是隐变量，文本的每个位置的单词是观测变量。\n", "\n", "3.LDA生成文本集合的生成过程如下：\n", "\n", "（1）话题的单词分布：随机生成所有话题的单词分布，话题的单词分布是多项分布，其先验分布是狄利克雷分布。\n", "\n", "（2）文本的话题分布：随机生成所有文本的话题分布，文本的话题分布是多项分布，其先验分布是狄利克雷分布。\n", "\n", "（3）文本的内容：随机生成所有文本的内容。在每个文本的每个位置，按照文本的话题分布随机生成一个话题，再按照该话题的单词分布随机生成一个单词。\n", "\n", "4.LDA模型的学习与推理不能直接求解。通常采用的方法是吉布斯抽样算法和变分EM算法，前者是蒙特卡罗法而后者是近似算法。\n", "\n", "5.LDA的收缩的吉布斯抽样算法的基本想法如下。目标是对联合概率分布$p ( w , z , \\theta , \\varphi | \\alpha , \\beta )$进行估计。通过积分求和将隐变量$\\theta$和$\\varphi$消掉，得到边缘概率分布$p ( w , z | \\alpha , \\beta )$；对概率分布$p ( w | z , \\alpha , \\beta )$进行吉布斯抽样，得到分布$p ( w | z , \\alpha , \\beta )$的随机样本；再利用样本对变量$z$，$\\theta$和$\\varphi$的概率进行估计，最终得到LDA模型$p ( w , z , \\theta , \\varphi | \\alpha , \\beta )$的参数估计。具体算法如下对给定的文本单词序列，每个位置上随机指派一个话题，整体构成话题系列。然后循环执行以下操作。对整个文本序列进行扫描，在每一个位置上计算在该位置上的话题的满条件概率分布，然后进行随机抽样，得到该位置的新的话题，指派给这个位置。\n", "\n", "6.变分推理的基本想法如下。假设模型是联合概率分布$p ( x , z )$，其中$x$是观测变量（数据），$z$是隐变量。目标是学习模型的后验概率分布$p ( z | x )$。考虑用变分分布$q ( z )$近似条件概率分布$p ( z | x )$，用KL散度计算两者的相似性找到与$p ( z | x )$在KL散度意义下最近的$q ^ { * } ( z )$，用这个分布近似$p ( z | x )$。假设$q ( z )$中的$z$的所有分量都是互相独立的。利用Jensen不等式，得到KL散度的最小化可以通过证据下界的最大化实现。因此，变分推理变成求解以下证据下界最大化问题：\n", "$$L ( q , \\theta ) = E _ { q } [ \\operatorname { log } p ( x , z | \\theta ) ] - E _ { q } [ \\operatorname { log } q ( z ) ]$$\n", "\n", "7.LDA的变分EM算法如下。针对LDA模型定义变分分布，应用变分EM算法。目标是对证据下界$L ( \\gamma , \\eta , \\alpha , \\varphi )$进行最大化，其中$\\alpha$和$\\varphi$是模型参数，$\\gamma$和$\\eta$是变分参数。交替迭代E步和M步，直到收敛。\n", "\n", "- （1）E步：固定模型参数$\\alpha$，$\\varphi$，通过关于变分参数$\\gamma$，$\\eta$的证据下界的最大化，估计变分参数$\\gamma$，$\\eta$。\n", "- （2）M步：固定变分参数$\\gamma$，$\\eta$，通过关于模型参数$\\alpha$，$\\varphi$的证据下界的最大化，估计模型参数$\\alpha$，$\\varphi$。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "潜在狄利克雷分配（latent Dirichlet allocation,LDA），作为基于贝叶斯学习的话题模型，是潜在语义分析、概率潜在语义分析的扩展，于2002年由Blei等提出dA在文本数据挖掘、图像处理、生物信息处理等领域被广泛使用。\n", "\n", "LDA模型是文本集合的生成概率模型假设每个文本由话题的一个多项分布表示，每个话题由单词的一个多项分布表示，特别假设文本的话题分布的先验分布是狄利克雷分布，话题的单词分布的先验分布也是狄利克雷分布。先验分布的导入使LDA能够更好地应对话题模型学习中的过拟合现象。\n", "\n", "LDA的文本集合的生成过程如下：首先随机生成一个文本的话题分布，之后在该文本的每个位置，依据该文本的话题分布随机生成一个话题，然后在该位置依据该话题的单词分布随机生成一个单词，直至文本的最后一个位置，生成整个文本。重复以上过程生成所有文本。\n", "\n", "LDA模型是含有隐变量的概率图模型。模型中，每个话题的单词分布，每个文本的话题分布，文本的每个位置的话题是隐变量；文本的每个位置的单词是观测变量。LDA模型的学习与推理无法直接求解通常使用吉布斯抽样（ Gibbs sampling）和变分EM算法（variational EM algorithm），前者是蒙特卡罗法，而后者是近似算法。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from gensim import corpora, models, similarities\n", "from pprint import pprint\n", "import warnings" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "After\n" ] } ], "source": [ "f = open('data/LDA_test.txt')\n", "stop_list = set('for a of the and to in'.split())\n", "# texts = [line.strip().split() for line in f]\n", "# print 'Before'\n", "# pprint(texts)\n", "print('After')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Text = \n", "[['human', 'machine', 'interface', 'lab', 'abc', 'computer', 'applications'],\n", " ['survey', 'user', 'opinion', 'computer', 'system', 'response', 'time'],\n", " ['eps', 'user', 'interface', 'management', 'system'],\n", " ['system', 'human', 'system', 'engineering', 'testing', 'eps'],\n", " ['relation', 'user', 'perceived', 'response', 'time', 'error', 'measurement'],\n", " ['generation', 'random', 'binary', 'unordered', 'trees'],\n", " ['intersection', 'graph', 'paths', 'trees'],\n", " ['graph', 'minors', 'iv', 'widths', 'trees', 'well', 'quasi', 'ordering'],\n", " ['graph', 'minors', 'survey']]\n" ] } ], "source": [ "texts = [[\n", " word for word in line.strip().lower().split() if word not in stop_list\n", "] for line in f]\n", "print('Text = ')\n", "pprint(texts)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dictionary(35 unique tokens: ['abc', 'applications', 'computer', 'human', 'interface']...)\n" ] } ], "source": [ "dictionary = corpora.Dictionary(texts)\n", "print(dictionary)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TF-IDF:\n", "[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1)]\n", "[(2, 1), (7, 1), (8, 1), (9, 1), (10, 1), (11, 1), (12, 1)]\n", "[(4, 1), (10, 1), (12, 1), (13, 1), (14, 1)]\n", "[(3, 1), (10, 2), (13, 1), (15, 1), (16, 1)]\n", "[(8, 1), (11, 1), (12, 1), (17, 1), (18, 1), (19, 1), (20, 1)]\n", "[(21, 1), (22, 1), (23, 1), (24, 1), (25, 1)]\n", "[(24, 1), (26, 1), (27, 1), (28, 1)]\n", "[(24, 1), (26, 1), (29, 1), (30, 1), (31, 1), (32, 1), (33, 1), (34, 1)]\n", "[(9, 1), (26, 1), (30, 1)]\n" ] } ], "source": [ "V = len(dictionary)\n", "corpus = [dictionary.doc2bow(text) for text in texts]\n", "corpus_tfidf = models.TfidfModel(corpus)[corpus]\n", "corpus_tfidf = corpus\n", "\n", "print('TF-IDF:')\n", "for c in corpus_tfidf:\n", " print(c)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "LSI Model:\n", "[[(0, 0.9334981916792652), (1, 0.10508952614086528)],\n", " [(0, 2.031992374687025), (1, -0.047145314121742235)],\n", " [(0, 1.5351342836582078), (1, 0.13488784052204628)],\n", " [(0, 1.9540077194594532), (1, 0.21780498576075008)],\n", " [(0, 1.2902472956004092), (1, -0.0022521437499372337)],\n", " [(0, 0.022783081905505403), (1, -0.7778052604326754)],\n", " [(0, 0.05671567576920905), (1, -1.1827703446704851)],\n", " [(0, 0.12360003320647955), (1, -2.6343068608236835)],\n", " [(0, 0.23560627195889133), (1, -0.9407936203668315)]]\n" ] } ], "source": [ "print('\\nLSI Model:')\n", "lsi = models.LsiModel(corpus_tfidf, num_topics=2, id2word=dictionary)\n", "topic_result = [a for a in lsi[corpus_tfidf]]\n", "pprint(topic_result)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "LSI Topics:\n", "[(0,\n", " '0.579*\"system\" + 0.376*\"user\" + 0.270*\"eps\" + 0.257*\"time\" + '\n", " '0.257*\"response\"'),\n", " (1,\n", " '-0.480*\"graph\" + -0.464*\"trees\" + -0.361*\"minors\" + -0.266*\"widths\" + '\n", " '-0.266*\"ordering\"')]\n" ] } ], "source": [ "print('LSI Topics:')\n", "pprint(lsi.print_topics(num_topics=2, num_words=5))" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Similarity:\n", "[array([ 1. , 0.9908607 , 0.9997008 , 0.9999994 , 0.9935261 ,\n", " -0.08272626, -0.06414512, -0.06517283, 0.13288835], dtype=float32),\n", " array([0.9908607 , 0.99999994, 0.9938636 , 0.99100804, 0.99976987,\n", " 0.0524564 , 0.07105229, 0.070025 , 0.2653665 ], dtype=float32),\n", " array([ 0.9997008 , 0.9938636 , 0.99999994, 0.999727 , 0.99600756,\n", " -0.05832579, -0.03971674, -0.04074576, 0.15709123], dtype=float32),\n", " array([ 0.9999994 , 0.99100804, 0.999727 , 1. , 0.9936501 ,\n", " -0.08163348, -0.06305084, -0.06407862, 0.13397504], dtype=float32),\n", " array([0.9935261 , 0.99976987, 0.99600756, 0.9936501 , 0.99999994,\n", " 0.03102366, 0.04963995, 0.04861134, 0.24462426], dtype=float32),\n", " array([-0.08272626, 0.0524564 , -0.05832579, -0.08163348, 0.03102366,\n", " 0.99999994, 0.99982643, 0.9998451 , 0.97674036], dtype=float32),\n", " array([-0.06414512, 0.07105229, -0.03971674, -0.06305084, 0.04963995,\n", " 0.99982643, 1. , 0.9999995 , 0.9805657 ], dtype=float32),\n", " array([-0.06517283, 0.070025 , -0.04074576, -0.06407862, 0.04861134,\n", " 0.9998451 , 0.9999995 , 1. , 0.9803632 ], dtype=float32),\n", " array([0.13288835, 0.2653665 , 0.15709123, 0.13397504, 0.24462426,\n", " 0.97674036, 0.9805657 , 0.9803632 , 1. ], dtype=float32)]\n" ] } ], "source": [ "similarity = similarities.MatrixSimilarity(lsi[corpus_tfidf]) # similarities.Similarity()\n", "print('Similarity:')\n", "pprint(list(similarity))" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "LDA Model:\n", "Document-Topic:\n", "\n", "[[(0, 0.02668742), (1, 0.97331256)],\n", " [(0, 0.9784582), (1, 0.021541778)],\n", " [(0, 0.9704323), (1, 0.02956772)],\n", " [(0, 0.97509205), (1, 0.024907947)],\n", " [(0, 0.9785106), (1, 0.021489413)],\n", " [(0, 0.9703556), (1, 0.029644381)],\n", " [(0, 0.04481229), (1, 0.9551877)],\n", " [(0, 0.023327617), (1, 0.97667235)],\n", " [(0, 0.058409944), (1, 0.9415901)]]\n" ] } ], "source": [ "print('\\nLDA Model:')\n", "num_topics = 2\n", "lda = models.LdaModel(\n", " corpus_tfidf,\n", " num_topics=num_topics,\n", " id2word=dictionary,\n", " alpha='auto',\n", " eta='auto',\n", " minimum_probability=0.001,\n", " passes=10)\n", "doc_topic = [doc_t for doc_t in lda[corpus_tfidf]]\n", "print('Document-Topic:\\n')\n", "pprint(doc_topic)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[(0, 0.026687337), (1, 0.9733126)]\n", "[(0, 0.9784589), (1, 0.021541081)]\n", "[(0, 0.97043234), (1, 0.029567692)]\n", "[(0, 0.9750935), (1, 0.024906479)]\n", "[(0, 0.9785101), (1, 0.021489937)]\n", "[(0, 0.9703557), (1, 0.029644353)]\n", "[(0, 0.044812497), (1, 0.9551875)]\n", "[(0, 0.02332762), (1, 0.97667235)]\n", "[(0, 0.058404233), (1, 0.9415958)]\n" ] } ], "source": [ "for doc_topic in lda.get_document_topics(corpus_tfidf):\n", " print(doc_topic)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Topic 0\n", "[('system', 0.094599016),\n", " ('user', 0.073440075),\n", " ('eps', 0.052545987),\n", " ('response', 0.052496374),\n", " ('time', 0.052453455),\n", " ('survey', 0.031701956),\n", " ('trees', 0.03162545),\n", " ('human', 0.03161709),\n", " ('computer', 0.031570844),\n", " ('testing', 0.031543963)]\n", "Topic 1\n", "[('graph', 0.0883405),\n", " ('trees', 0.06323685),\n", " ('minors', 0.06296622),\n", " ('interface', 0.03810195),\n", " ('computer', 0.03798469),\n", " ('human', 0.03792907),\n", " ('applications', 0.03792245),\n", " ('abc', 0.037920628),\n", " ('machine', 0.037917122),\n", " ('lab', 0.037909806)]\n", "Similarity:\n", "[array([1. , 0.04940351, 0.05783966, 0.05292428, 0.04934979,\n", " 0.05791992, 0.99981046, 0.99999374, 0.99940336], dtype=float32),\n", " array([0.04940351, 1. , 0.99996436, 0.9999938 , 1. ,\n", " 0.99996364, 0.06883725, 0.04587576, 0.08387101], dtype=float32),\n", " array([0.05783966, 0.99996436, 1.0000001 , 0.99998796, 0.99996394,\n", " 1. , 0.07726298, 0.05431345, 0.09228647], dtype=float32),\n", " array([0.05292428, 0.9999938 , 0.99998796, 1. , 0.9999936 ,\n", " 0.9999875 , 0.07235384, 0.04939714, 0.08738345], dtype=float32),\n", " array([0.04934979, 1. , 0.99996394, 0.9999936 , 1. ,\n", " 0.99996316, 0.06878359, 0.04582203, 0.08381741], dtype=float32),\n", " array([0.05791992, 0.99996364, 1. , 0.9999875 , 0.99996316,\n", " 0.99999994, 0.07734313, 0.05439373, 0.09236652], dtype=float32),\n", " array([0.99981046, 0.06883725, 0.07726298, 0.07235384, 0.06878359,\n", " 0.07734313, 0.99999994, 0.9997355 , 0.9998863 ], dtype=float32),\n", " array([0.99999374, 0.04587576, 0.05431345, 0.04939714, 0.04582203,\n", " 0.05439373, 0.9997355 , 0.99999994, 0.9992751 ], dtype=float32),\n", " array([0.99940336, 0.08387101, 0.09228647, 0.08738345, 0.08381741,\n", " 0.09236652, 0.9998863 , 0.9992751 , 1. ], dtype=float32)]\n", "\n", "\n", "USE WITH CARE--\n", "HDA Model:\n", "[[(0, 0.18174982193320122),\n", " (1, 0.02455260642448283),\n", " (2, 0.741340573910992),\n", " (3, 0.013544078061059922),\n", " (4, 0.010094377639823477)],\n", " [(0, 0.39419292675663636),\n", " (1, 0.2921969355337328),\n", " (2, 0.26125786014858376),\n", " (3, 0.013539627392486701),\n", " (4, 0.01009410883245766)],\n", " [(0, 0.5182077872999125),\n", " (1, 0.3880947736463974),\n", " (2, 0.023895609845034207),\n", " (3, 0.01805202212531745),\n", " (4, 0.013458421673222807)],\n", " [(0, 0.03621384798236036),\n", " (1, 0.5504573172680752),\n", " (2, 0.020442846194997377),\n", " (3, 0.348529241707211),\n", " (4, 0.011535562414627153)],\n", " [(0, 0.9049762450848856),\n", " (1, 0.024748801100993395),\n", " (2, 0.017919024335434904),\n", " (3, 0.013543460312481508),\n", " (4, 0.010093932388992328)],\n", " [(0, 0.04681359723231631),\n", " (1, 0.03233799461088905),\n", " (2, 0.8510430252219996),\n", " (3, 0.01805587061936895),\n", " (4, 0.013458128836093802)],\n", " [(0, 0.42478083784052273),\n", " (1, 0.03858547281122597),\n", " (2, 0.4528531768644199),\n", " (3, 0.021680841796584305),\n", " (4, 0.016150009359845837),\n", " (5, 0.011953757612369628)],\n", " [(0, 0.2466808290730598),\n", " (1, 0.6908552821243853),\n", " (2, 0.015924569811569197),\n", " (3, 0.012039668311419834)],\n", " [(0, 0.500366457263008),\n", " (1, 0.048221177670061226),\n", " (2, 0.34671234963274666),\n", " (3, 0.02707530995137571),\n", " (4, 0.02018763747377598),\n", " (5, 0.014942188361070167),\n", " (6, 0.010992923111633942)]]\n", "HDA Topics:\n", "[(0, '0.122*graph + 0.115*minors + 0.098*management + 0.075*random + 0.063*error'), (1, '0.114*human + 0.106*system + 0.086*user + 0.064*iv + 0.063*measurement')]\n" ] } ], "source": [ "for topic_id in range(num_topics):\n", " print('Topic', topic_id)\n", " # pprint(lda.get_topic_terms(topicid=topic_id))\n", " pprint(lda.show_topic(topic_id))\n", "similarity = similarities.MatrixSimilarity(lda[corpus_tfidf])\n", "print('Similarity:')\n", "pprint(list(similarity))\n", "\n", "hda = models.HdpModel(corpus_tfidf, id2word=dictionary)\n", "topic_result = [a for a in hda[corpus_tfidf]]\n", "print('\\n\\nUSE WITH CARE--\\nHDA Model:')\n", "pprint(topic_result)\n", "print('HDA Topics:')\n", "print(hda.print_topics(num_topics=2, num_words=5))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "----\n", "代码参考：邹博\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: 第20章 潜在狄利克雷分配/data/LDA_test.txt ================================================ Human machine interface for lab abc computer applications A survey of user opinion of computer system response time The EPS user interface management system System and human system engineering testing of EPS Relation of user perceived response time to error measurement The generation of random binary unordered trees The intersection graph of paths in trees Graph minors IV Widths of trees and well quasi ordering Graph minors A survey ================================================ FILE: 第21章 PageRank算法/21.PageRank.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第21章 PageRank算法" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. PageRank是互联网网页重要度的计算方法，可以定义推广到任意有向图结点的重要度计算上。其基本思想是在有向图上定义随机游走模型，即一阶马尔可夫链，描述游走者沿着有向图随机访问各个结点的行为，在一定条件下，极限情况访问每个结点的概率收敛到平稳分布，这时各个结点的概率值就是其 PageRank值，表示结点相对重要度。\n", "\n", "2. 有向图上可以定义随机游走模型，即一阶马尔可夫链，其中结点表示状态，有向边表示状态之间的转移，假设一个结点到连接出的所有结点的转移概率相等。转移概率由转移矩阵$M$表示\n", "$$M = [ m _ { i j } ] _ { n \\times n }$$\n", "第$i$行第$j$列的元素$m _ { i j }$表示从结点$j$跳转到结点$i$的概率。\n", "\n", "3. 当含有$n$个结点的有向图是强连通且非周期性的有向图时，在其基础上定义的随机游走模型，即一阶马尔可夫链具有平稳分布，平稳分布向量$R$称为这个有向图的 PageRank。若矩阵$M$是马尔可夫链的转移矩阵，则向量R满足$$MR=R$$向量$R$的各个分量称 PageRank为各个结点的值。\n", "$$R = \\left[ \\begin{array} { c } { P R ( v _ { 1 } ) } \\\\ { P R ( v _ { 2 } ) } \\\\ { \\vdots } \\\\ { P R ( v _ { n } ) } \\end{array} \\right]$$\n", "其中$P R ( v _ { i } ) , i = 1,2 , \\cdots , n$，表示结点$v_i$的 PageRank值。这是 PageRank的基本定义。\n", "\n", "4. PageRank基本定义的条件现实中往往不能满足，对其进行扩展得到 PageRank的一般定义。任意含有$n$个结点的有向图上，可以定义一个随机游走模型，即一阶马尔可夫链，转移矩阵由两部分的线性组合组成，其中一部分按照转移矩阵$M$，从一个结点到连接出的所有结点的转移概率相等，另一部分按照完全随机转移矩阵，从任一结点到任一结点的转移概率都是$1/n$。这个马尔可夫链存在平稳分布，平稳分布向量R称为这个有 PageRank向图的一般，满足\n", "$$R = d M R + \\frac { 1 - d } { n } 1$$\n", "\n", "其中$d ( 0 \\leq d \\leq 1 )$是阻尼因子，1是所有分量为1的$n$维向量。\n", "\n", "5. PageRank的计算方法包括迭代算法、幂法、代数算法。\n", "\n", "幂法将 PageRank的等价式写成$$R = ( d M + \\frac { 1 - d } { n } E ) R = A R$$\n", "其中$d$是阻尼因子，$E$是所有元素为1的$n$阶方阵。\n", "\n", "PageRank算法可以看出$R$是一般转移矩阵$A$的主特征向量，即最大的特征值对应的特征向量。\n", "幂法就是一个计算矩阵的主特征值和主特征向量的方法。\n", "\n", "步骤是：选择初始向量$x_0$；计算一般转移矩阵$A$；进行迭代并规范化向量\n", "$$y _ { t + 1 } = A x _ { t }$$\n", "$$x _ { t + 1 } = \\frac { y _ { t + 1 } } { \\| y _ { t + 1 } \\| }$$\n", "直至收敛。" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "iWOZV94kYsbM" }, "source": [ "---\n", "在实际应用中许多数据都以图(graph)的形式存在，比如，互联网、社交网络都可以看作是一个图。图数据上的机器学习具有理论与应用上的重要意义。pageRank算法是图的链接分析 (link analysis)的代表性算法，属于图数据上的无监督学习方法。 \n", "\n", "pageRank算法最初作为互联网网页重要度的计算方法，1996年由page和Brin提出，并用于谷歌搜索引擎的网页排序。事实上，pageRank可以定义在任意有向图上，后来被应用到社会影响力分析、文本摘要等多个问题。 \n", "\n", "pageRank算法的基本想法是在有向图上定义一个随机游走模型，即一阶马尔可夫链，描述随机游走者沿着有向图随机访问各个结点的行为。在一定条件下，极限情况访问每个结点的概率收敛到平稳分布， 这时各个结点的平稳概率值就是其 pageRank值，表示结点的重要度。 pageRank是递归定义的，pageRank的计算可以通过迭代算法进行。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": {}, "colab_type": "code", "id": "fAN4q0cqYn-f" }, "outputs": [], "source": [ "#https://gist.github.com/diogojc/1338222/84d767a68da711a154778fb1d00e772d65322187\n", "\n", "import numpy as np\n", "from scipy.sparse import csc_matrix\n", "\n", "\n", "def pageRank(G, s=.85, maxerr=.0001):\n", " \"\"\"\n", " Computes the pagerank for each of the n states\n", " Parameters\n", " ----------\n", " G: matrix representing state transitions\n", " Gij is a binary value representing a transition from state i to j.\n", " s: probability of following a transition. 1-s probability of teleporting\n", " to another state.\n", " maxerr: if the sum of pageranks between iterations is bellow this we will\n", " have converged.\n", " \"\"\"\n", " n = G.shape[0]\n", "\n", " # transform G into markov matrix A\n", " A = csc_matrix(G, dtype=np.float)\n", " rsums = np.array(A.sum(1))[:, 0]\n", " ri, ci = A.nonzero()\n", " A.data /= rsums[ri]\n", "\n", " # bool array of sink states\n", " sink = rsums == 0\n", "\n", " # Compute pagerank r until we converge\n", " ro, r = np.zeros(n), np.ones(n)\n", " while np.sum(np.abs(r - ro)) > maxerr:\n", " ro = r.copy()\n", " # calculate each pagerank at a time\n", " for i in range(0, n):\n", " # inlinks of state i\n", " Ai = np.array(A[:, i].todense())[:, 0]\n", " # account for sink states\n", " Di = sink / float(n)\n", " # account for teleportation to state i\n", " Ei = np.ones(n) / float(n)\n", "\n", " r[i] = ro.dot(Ai * s + Di * s + Ei * (1 - s))\n", "\n", " # return normalized pagerank\n", " return r / float(sum(r))" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 53 }, "colab_type": "code", "id": "Ds-wQEFFZ1F7", "outputId": "b2860902-8712-4583-ab47-bec602c6791b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0.12727557 0.03616954 0.12221594 0.22608452 0.28934412 0.03616954\n", " 0.16274076]\n" ] } ], "source": [ "# Example extracted from 'Introduction to Information Retrieval'\n", "G = np.array([[0,0,1,0,0,0,0],\n", " [0,1,1,0,0,0,0],\n", " [1,0,1,1,0,0,0],\n", " [0,0,0,1,1,0,0],\n", " [0,0,0,0,0,0,1],\n", " [0,0,0,0,0,1,1],\n", " [0,0,0,1,1,0,1]])\n", "print(pageRank(G,s=.86))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "本章代码来源：https://github.com/hktxt/Learn-Statistical-Learning-Method\n", "\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "PageRank.ipynb", "provenance": [], "version": "0.3.2" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 1 } ================================================ FILE: 第22章 无监督学习方法总结/22.Summary_of_UnSupervised_Learning_Methods.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 第22章 无监督学习方法总结\n", "## 无监督学习方法的关系和特点\n", "\n", "第2篇详细介绍了八种常用的统计机器学习方法，即聚类方法（包括层次聚类与k均值聚类）、奇异值分解（SVD）、主成分分析（PCA）、无监督学习方法总结\n", "22.1无监潜在语义分析（LSA）、概率潜在语义分析（PLSA）、马尔可夫链蒙特卡罗法（CMC，包括 Metropolis-Hastings-算法和吉布斯抽样）、潜在狄利克雷分配（LDA）、 PageRank算法。此外，还简单介绍了另外三种常用的统计机器学习方法，即非负矩阵分解（NMF）变分推理、幂法。这些方法通常用于无监督学习的聚类、降维、话题分析以及图分析。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 表 无监督学习方法的特点\n", "| | 方法 | 模型 | 策略 | 算法 |\n", "| -------- | ------------ | -------------------- | -------------------- | -------------------- |\n", "| 聚类 | 层次聚类 | 聚类树 | 类内样本距离最小 | 启发式算法 |\n", "| | k均值聚类 | k中心聚类 | 样本与类中心距离最小 | 迭代算法 |\n", "| | 高斯混合模型 | 高斯混合模型 | 似然函数最大 | EM算法 |\n", "| 降维 | PCA | 低维正交空间 | 方差最大 | SVD |\n", "| 话题分析 | LSA | 矩阵分解模型 | 平方损失最小 | SVD |\n", "| | NMF | 矩阵分解模型 | 平方损失最小 | 非负矩阵分解 |\n", "| | PLSA | PLSA模型 | 似然函数最大 | EM算法 |\n", "| | LDA | LDA模型 | 后验概率估计 | 吉布斯抽样，变分推理 |\n", "| 图分析 | PageRank | 有向图上的马尔可夫链 | 平稳分布求解 | 幂法 |\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 表 含有隐变量概率模型的学习方法的特点\n", "| 算法 | 基本原理 | 收敛性 | 收敛速度 | 实现难易度 | 适合问题 |\n", "|------------|----------------------------|----------------------|----------|------------|----------|\n", "| EM算法 | 迭代计算、后验概率估计 | 收敛于局部最优 | 较快 | 容易 | 简单模型 |\n", "| 变分推理 | 迭代计算、后验概率近似估计 | 收敛于局部最优 | 较慢 | 较复杂 | 复杂模型 |\n", "| 吉布斯抽样 | 随机抽样、后验概率估计 | 依概率收敛于全局最优 | 较慢 | 容易 | 复杂模型 |\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 表 矩阵分解的角度看话题模型\n", "| 方法 | 一般损失函数 $B ( D \\| U V )$ | 矩阵$U$的约束条件 | 矩阵$V$的约束条件 |\n", "| ---- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |\n", "| LSA | $$\\| D - U V \\| _ { F } ^ { 2 }$$ | $$U^TU=I$$ | $$V V ^ { T } = \\Lambda ^ { 2 }$$ |\n", "| NMF | $$\\| D - U V \\| _ { F } ^ { 2 }$$ | $u _ { m k } \\geq 0$ | $$v _ { k n } \\geq 0$$ |\n", "| PLSA | $$\\sum _ { m n } d _ { m n } \\operatorname { log } \\frac { d _ { m n } } { ( U V ) _ { m n } }$$ | $$\\left. \\begin{array} { l } { U ^ { T } 1 = 1 } \\\\ { u _ { m k } \\geq 0 } \\end{array} \\right.$$ | $$\\left. \\begin{array} { l } { V ^ { T } 1 = 1 } \\\\ { v _ { k n } \\geq 0 } \\end{array} \\right.$$ |\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "本文代码更新地址：https://github.com/fengdu78/lihang-code\n", "\n", "中文注释制作：机器学习初学者公众号：ID:ai-start-com\n", "\n", "配置环境：python 3.5+\n", "\n", "代码全部测试通过。\n", "![gongzhong](../gongzhong.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }