[
{
"path": ".gitattributes",
"content": "*.txt linguist-language=python\n"
},
{
"path": "LICENSE",
"content": "MIT License\n\nCopyright (c) 2020 Mingchao Zhu\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "README.md",
"content": "# Deep Learning\n\n《**深度学习**》是深度学习领域唯一的综合性图书，全称也叫做**深度学习 AI圣经(Deep Learning)**，由三位全球知名专家IanGoodfellow、YoshuaBengio、AaronCourville编著，全书囊括了数学及相关概念的背景知识，包括线性代数、概率论、信息论、数值优化以及机器学习中的相关内容。同时，它还介绍了工业界中实践者用到的深度学习技术，包括深度前馈网络、正则化、优化算法、卷积网络、序列建模和实践方法等，并且调研了诸如自然语言处理、语音识别、计算机视觉、在线推荐系统、生物信息学以及视频游戏方面的应用。最后，深度学习全书还提供了一些研究方向，涵盖的理论主题包括线性因子模型、自编码器、表示学习、结构化概率模型、蒙特卡罗方法、配分函数、近似推断以及深度生成模型，适用于相关专业的大学生或研究生使用。\n\n![\"深度学习封面\"](\"https://github.com/MingchaoZhu/DeepLearning/blob/master/docs/cover.jpg\")
\n\n可以下载《深度学习》的中文版 [pdf](https://github.com/MingchaoZhu/DeepLearning/releases/download/v0.0.1/DL_cn.pdf) 和英文版 [pdf](https://github.com/MingchaoZhu/DeepLearning/releases/download/v0.0.0/DL_en.pdf) 直接阅读。\n\n对于本项目的工作，你可以直接下载 [深度学习_原理与代码实现.pdf](https://github.com/MingchaoZhu/DeepLearning/releases/download/v1.1.1/default.pdf) (后面会对该书不断更新)\n\n---\n\n《深度学习》可以说是深度学习与人工智能的入门宝典，许多算法爱好者、机器学习培训班、互联网企业的面试，很多都参考这本书。但本书晦涩，加上官方没有提供代码实现，因此某些地方较难理解。本项目**基于数学推导和产生原理重新描述了书中的概念**，并用**Python** (numpy 库为主) 复现了书本内容 ( **源码级代码实现。推导过程和代码实现均放在了下载区的 pdf 文件中**，重要部分的实现代码也放入 **code 文件夹**中 )。\n\n然而我水平有限，但我真诚地希望这项工作可以帮助到更多人学习深度学习算法。我需要大家的建议和帮助。如果你在阅读中遇到有误或解释不清的地方，希望可以汇总你的建议，在 Issues 提出。如果你也想加入这项工作书写中或有其他问题，可以联系我的邮箱。如果你在你的工作或博客中用到了本书，还请可以注明引用链接。\n\n写的过程中参考了较多网上优秀的工作，所有参考资源保存在了`reference.txt`文件中。\n\n# 留言\n\n这份工作就是在写这一本 [深度学习_原理与代码实现.pdf](https://github.com/MingchaoZhu/DeepLearning/releases/download/v1.1.1/default.pdf)。正如你在 pdf 文件中所见到的，《深度学习》涉及到的每一个概念，都会去给它详细的描述、原理层面的推导，以及用代码的实现。代码实现不会调用 Tensorflow、PyTorch、MXNet 等任何深度学习框架，甚至包括 sklearn (pdf 里用到 sklearn 的部分都是用来验证代码无误)，一切代码都是从原理层面实现 (Python 的基础库 NumPy)，并有详细注释，与代码区上方的原理描述区一致，你可以结合原理和代码一起理解。\n\n这份工作的起因是我自身的热爱，但为完成这份工作我需要投入大量的时间精力，一般会写到凌晨两三点。推导、代码、作图都是慢慢打磨的，我会保证这份工作的质量。这份工作会一直更新完，已经上传的章节也会继续补充内容。如果你在阅读过程中遇到有想要描述的概念点或者错误点，请发邮件告知我。\n\n真的很感谢你的认可与推广。最后，请等待下一次更新。\n\n我是 朱明超，我的邮箱是：deityrayleigh@gmail.com\n\n# 更新说明\n\n2020/3/：\n\n```python\n1. 修改第五章决策树部分，补充 ID3 和 CART 的原理，代码实现以 CART 为主。\n2. 第七章添加 L1 和 L2 正则化最优解的推导 (即 L1稀疏解的原理)。\n3. 第七章添加集成学习方法的推导与代码实现，包括 Bagging (随机森林)、Boosting (Adaboost、GBDT、XGBoost)。\n4. 第八章添加牛顿法与拟牛顿法 (DFP、BFGS、L-BFGS) 的推导。\n5. 第十一章节添加贝叶斯线性回归、高斯过程回归 (GPR) 与贝叶斯优化的推导与代码实现。\n```\n后面每次的更新内容会统一放在 `update.txt` 文件中。\n\n# 章节目录与文件下载\n\n除了《深度学习》书中的概念点，**本项目也在各章节添加一些补充知识，例如第七章集成学习部分的 随机森林、Adaboost、GBDT、XGBoost 的原理剖析和代码实现等，又或者第十二章对当前一些主流方法的描述**。大的章节目录和 pdf 文件下载链接可以详见下表，而具体 pdf 文件中的实际目录请参考 `contents.txt`。你可以在下面的 pdf 链接中下载对应章节，也可以在 [releases](https://github.com/MingchaoZhu/DeepLearning/releases) 界面直接下载所有文件。\n\n| 中文章节 | 英文章节 | 下载
(含推导与代码实现) |\n| ------------ | ------------ | ------------ |\n| 第一章 前言 | 1 Introduction | |\n| 第二章 线性代数 | 2 Linear Algebra | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/2%20%E7%BA%BF%E6%80%A7%E4%BB%A3%E6%95%B0.pdf) |\n| 第三章 概率与信息论 | 3 Probability and Information Theory | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/3%20%E6%A6%82%E7%8E%87%E4%B8%8E%E4%BF%A1%E6%81%AF%E8%AE%BA.pdf) |\n| 第四章 数值计算 | 4 Numerical Computation | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/4%20%E6%95%B0%E5%80%BC%E8%AE%A1%E7%AE%97.pdf) |\n| 第五章 机器学习基础 | 5 Machine Learning Basics | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/5%20%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E5%9F%BA%E7%A1%80.pdf) |\n| 第六章 深度前馈网络 | 6 Deep Feedforward Networks | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/6%20%E6%B7%B1%E5%BA%A6%E5%89%8D%E9%A6%88%E7%BD%91%E7%BB%9C.pdf) |\n| 第七章 深度学习中的正则化 | 7 Regularization for Deep Learning | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/7%20%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E4%B8%AD%E7%9A%84%E6%AD%A3%E5%88%99%E5%8C%96.pdf) |\n| 第八章 深度模型中的优化 | 8 Optimization for Training Deep Models | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/8%20%E6%B7%B1%E5%BA%A6%E6%A8%A1%E5%9E%8B%E4%B8%AD%E7%9A%84%E4%BC%98%E5%8C%96.pdf) |\n| 第九章 卷积网络 | 9 Convolutional Networks | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/9%20%E5%8D%B7%E7%A7%AF%E7%BD%91%E7%BB%9C.pdf) |\n| 第十章 序列建模：循环和递归网络 | 10 Sequence Modeling: Recurrent and Recursive Nets | |\n| 第十一章 实践方法论 | 11 Practical Methodology | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/11%20%E5%AE%9E%E8%B7%B5%E6%96%B9%E6%B3%95%E8%AE%BA.pdf) |\n| 第十二章 应用 | 12 Applications | |\n| 第十三章 线性因子模型 | 13 Linear Factor Models | |\n| 第十四章 自编码器 | 14 Autoencoders | |\n| 第十五章 表示学习 | 15 Representation Learning | |\n| 第十六章 深度学习中的结构化概率模型 | 16 Structured Probabilistic Models for Deep Learning | |\n| 第十七章 蒙特卡罗方法 | 17 Monte Carlo Methods | |\n| 第十八章 直面配分函数 | 18 Confronting the Partition Function | |\n| 第十九章 近似推断 | 19 Approximate Inference | |\n| 第二十章 深度生成模型 | 20 Deep Generative Models | |\n\n尚未上传的章节会在后续陆续上传。\n\n# 致谢\n\n感谢对本项目的认可和推广。\n\n+ 专知：https://mp.weixin.qq.com/s/dVD-vKJsMGqnBz2v4O-Q3Q\n+ GitHubDaily：https://m.weibo.cn/5722964389/4504392843690487\n+ 程序员遇见GitHub：https://mp.weixin.qq.com/s/EzFOnwpkv7mr2TSjPtVG9A\n+ 爱可可：https://m.weibo.cn/1402400261/4503389646699745\n\n# 赞助\n\n本项目书写耗费时间精力。如果本项目对你有帮助，可以请作者吃份冰淇淋：\n\n![\"支付\"](\"./docs/pay.jpg\")
\n"
},
{
"path": "code/chapter 11.py",
"content": "import pandas as pd\nimport numpy as np\nimport itertools\nimport time\nimport re\nfrom scipy.stats import norm\nimport matplotlib.pyplot as plt\n\n\ndef cal_conf_matrix(labels, preds):\n \"\"\"\n 计算混淆矩阵。\n \n 参数说明：\n labels：样本标签 (真实结果)\n preds：预测结果\n \"\"\"\n n_sample = len(labels)\n result = pd.DataFrame(index=range(0,n_sample),columns=('probability','label'))\n result['label'] = np.array(labels)\n result['probability'] = np.array(preds)\n cm = np.arange(4).reshape(2,2)\n cm[0,0] = len(result[result['label']==1][result['probability']>=0.5]) # TP，注意这里是以 0.5 为阈值\n cm[0,1] = len(result[result['label']==1][result['probability']<0.5]) # FN\n cm[1,0] = len(result[result['label']==0][result['probability']>=0.5]) # FP\n cm[1,1] = len(result[result['label']==0][result['probability']<0.5]) # TN \n return cm\n\n\ndef cal_PRF1(labels, preds):\n \"\"\"\n 计算查准率P，查全率R，F1值。\n \"\"\"\n cm = cal_conf_matrix(labels, preds)\n P = cm[0,0]/(cm[0,0]+cm[1,0])\n R = cm[0,0]/(cm[0,0]+cm[0,1])\n F1 = 2*P*R/(P+R)\n return P, R, F1\n\n\ndef cal_PRcurve(labels, preds):\n \"\"\"\n 计算PR曲线上的值。\n \"\"\"\n n_sample = len(labels)\n result = pd.DataFrame(index=range(0,n_sample),columns=('probability','label'))\n y_pred[y_pred>=0.5] = 1\n y_pred[y_pred<0.5] = 0\n result['label'] = np.array(labels)\n result['probability'] = np.array(preds)\n result.sort_values('probability',inplace=True,ascending=False)\n PandR = pd.DataFrame(index=range(len(labels)),columns=('P','R'))\n for j in range(len(result)):\n # 以每一个概率为分类的阈值，统计此时正例和反例的数量\n result_j = result.head(n=j+1)\n P = len(result_j[result_j['label']==1])/float(len(result_j)) # 当前实际为正的数量/当前预测为正的数量\n R = len(result_j[result_j['label']==1])/float(len(result[result['label']==1])) # 当前真正例的数量/实际为正的数量\n PandR.iloc[j] = [P,R]\n return PandR\n\n\ndef cal_ROCcurve(labels, preds):\n \"\"\"\n 计算ROC曲线上的值。\n \"\"\"\n n_sample = len(labels)\n result = pd.DataFrame(index=range(0,n_sample),columns=('probability','label'))\n y_pred[y_pred>=0.5] = 1\n y_pred[y_pred<0.5] = 0\n result['label'] = np.array(labels)\n result['probability'] = np.array(preds)\n # 计算 TPR,FPR\n result.sort_values('probability',inplace=True,ascending=False)\n TPRandFPR=pd.DataFrame(index=range(len(result)),columns=('TPR','FPR'))\n for j in range(len(result)):\n # 以每一个概率为分类的阈值，统计此时正例和反例的数量\n result_j=result.head(n=j+1)\n TPR=len(result_j[result_j['label']==1])/float(len(result[result['label']==1])) # 当前真正例的数量/实际为正的数量\n FPR=len(result_j[result_j['label']==0])/float(len(result[result['label']==0])) # 当前假正例的数量/实际为负的数量\n TPRandFPR.iloc[j]=[TPR,FPR]\n return TPRandFPR\n\n\ndef timeit(func):\n \"\"\"\n 装饰器，计算函数执行时间\n \"\"\"\n def wrapper(*args, **kwargs):\n time_start = time.time()\n result = func(*args, **kwargs)\n time_end = time.time()\n exec_time = time_end - time_start\n print(\"{function} exec time: {time}s\".format(function=func.__name__,time=exec_time))\n return result\n return wrapper\n\n@timeit\ndef area_auc(labels, preds):\n \"\"\"\n AUC值的梯度法计算\n \"\"\"\n TPRandFPR = cal_ROCcurve(labels, preds)\n # 计算AUC，计算小矩形的面积之和\n auc = 0.\n prev_x = 0\n for x, y in zip(TPRandFPR.FPR,TPRandFPR.TPR):\n if x != prev_x:\n auc += (x - prev_x) * y\n prev_x = x\n return auc\n\n@timeit\ndef naive_auc(labels, preds):\n \"\"\"\n AUC值的概率法计算\n \"\"\"\n n_pos = sum(labels)\n n_neg = len(labels) - n_pos\n total_pair = n_pos * n_neg # 总的正负样本对的数目\n labels_preds = zip(labels, preds)\n labels_preds = sorted(labels_preds,key=lambda x:x[1]) # 对预测概率升序排序\n count_neg = 0 # 统计负样本出现的个数\n satisfied_pair = 0 # 统计满足条件的样本对的个数\n for i in range(len(labels_preds)):\n if labels_preds[i][0] == 1:\n satisfied_pair += count_neg # 表明在这个正样本下，有哪些负样本满足条件\n else:\n count_neg += 1\n return satisfied_pair / float(total_pair)\n\n\n#####----Bayesian Hyperparameter Optimization----####\nclass KernelBase(ABC):\n \n def __init__(self):\n super().__init__()\n self.params = {}\n self.hyperparams = {}\n\n @abstractmethod\n def _kernel(self, X, Y):\n raise NotImplementedError\n\n def __call__(self, X, Y=None):\n return self._kernel(X, Y)\n\n def __str__(self):\n P, H = self.params, self.hyperparams\n p_str = \", \".join([\"{}={}\".format(k, v) for k, v in P.items()])\n return \"{}({})\".format(H[\"op\"], p_str)\n\n def summary(self):\n return {\n \"op\": self.hyperparams[\"op\"],\n \"params\": self.params,\n \"hyperparams\": self.hyperparams,\n }\n\n\nclass RBFKernel(KernelBase):\n \n def __init__(self, sigma=None):\n \"\"\"\n RBF 核。\n \"\"\"\n super().__init__()\n self.hyperparams = {\"op\": \"RBFKernel\"}\n self.params = {\"sigma\": sigma} # 如果 sigma 未赋值则默认为 np.sqrt(n_features/2)，n_features 为特征数。\n\n def _kernel(self, X, Y=None):\n \"\"\"\n 对 X 和 Y 的行的每一对计算 RBF 核。如果 Y 为空，则 Y=X。\n\n 参数说明：\n X：输入数组，为 (n_samples, n_features)\n Y：输入数组，为 (m_samples, n_features)\n \"\"\"\n X = X.reshape(-1, 1) if X.ndim == 1 else X\n Y = X if Y is None else Y\n Y = Y.reshape(-1, 1) if Y.ndim == 1 else Y\n assert X.ndim == 2 and Y.ndim == 2, \"X and Y must have 2 dimensions\"\n sigma = np.sqrt(X.shape[1] / 2) if self.params[\"sigma\"] is None else self.params[\"sigma\"]\n X, Y = X / sigma, Y / sigma\n D = -2 * X @ Y.T + np.sum(Y**2, axis=1) + np.sum(X**2, axis=1)[:, np.newaxis]\n D[D < 0] = 0\n return np.exp(-0.5 * D)\n \n\nclass KernelInitializer(object):\n \n def __init__(self, param=None):\n self.param = param\n\n def __call__(self):\n r = r\"([a-zA-Z0-9]*)=([^,)]*)\"\n kr_str = self.param.lower()\n kwargs = dict([(i, eval(j)) for (i, j) in re.findall(r, self.param)])\n if \"rbf\" in kr_str:\n kernel = RBFKernel(**kwargs)\n else:\n raise NotImplementedError(\"{}\".format(kr_str))\n return kernel\n\n\nclass GPRegression:\n \"\"\"\n 高斯过程回归\n \"\"\"\n def __init__(self, kernel=\"RBFKernel\", sigma=1e-10):\n self.kernel = KernelInitializer(kernel)()\n self.params = {\"GP_mean\": None, \"GP_cov\": None, \"X\": None}\n self.hyperparams = {\"kernel\": str(self.kernel), \"sigma\": sigma}\n\n def fit(self, X, y):\n \"\"\"\n 用已有的样本集合得到 GP 先验。\n\n 参数说明：\n X：输入数组，为 (n_samples, n_features)\n y：输入数组 X 的目标值，为 (n_samples)\n \"\"\"\n mu = np.zeros(X.shape[0])\n Cov = self.kernel(X, X)\n self.params[\"X\"] = X\n self.params[\"y\"] = y\n self.params[\"GP_cov\"] = Cov\n self.params[\"GP_mean\"] = mu\n\n def predict(self, X_star, conf_interval=0.95):\n \"\"\"\n 对新的样本 X 进行预测。\n\n 参数说明：\n X_star：输入数组，为 (n_samples, n_features)\n conf_interval：置信区间，浮点型 (0, 1)，default=0.95\n \"\"\"\n X = self.params[\"X\"]\n y = self.params[\"y\"]\n K = self.params[\"GP_cov\"]\n sigma = self.hyperparams[\"sigma\"]\n K_star = self.kernel(X_star, X)\n K_star_star = self.kernel(X_star, X_star)\n sig = np.eye(K.shape[0]) * sigma\n K_y_inv = np.linalg.pinv(K + sig)\n mean = K_star @ K_y_inv @ y\n cov = K_star_star - K_star @ K_y_inv @ K_star.T\n percentile = norm.ppf(conf_interval)\n conf = percentile * np.sqrt(np.diag(cov))\n return mean, conf, cov\n\n\nclass BayesianOptimization:\n \n def __init__(self):\n self.model = GPRegression()\n \n def acquisition_function(self, Xsamples):\n mu, _, cov = self.model.predict(Xsamples)\n mu = mu if mu.ndim==1 else (mu.T)[0]\n ysample = np.random.multivariate_normal(mu, cov) \n return ysample\n \n def opt_acquisition(self, X, n_samples=20):\n # 样本搜索策略，一般方法有随机搜索、基于网格的搜索，或局部搜索\n # 我们这里就用简单的随机搜索，这里也可以定义样本的范围\n Xsamples = np.random.randint(low=1,high=50,size=n_samples*X.shape[1])\n Xsamples = Xsamples.reshape(n_samples, X.shape[1])\n # 计算采集函数的值并取最大的值\n scores = self.acquisition_function(Xsamples)\n ix = np.argmax(scores)\n return Xsamples[ix, 0]\n \n def fit(self, f, X, y):\n # 拟合 GPR 模型\n self.model.fit(X, y)\n # 优化过程\n for i in range(15):\n x_star = self.opt_acquisition(X) # 下一个采样点\n y_star = f(x_star)\n mean, conf, cov = self.model.predict(np.array([[x_star]]))\n # 添加当前数据到数据集合\n X = np.vstack((X, [[x_star]]))\n y = np.vstack((y, [[y_star]]))\n # 更新 GPR 模型\n self.model.fit(X, y)\n ix = np.argmax(y)\n print('Best Result: x=%.3f, y=%.3f' % (X[ix], y[ix]))\n return X[ix], y[ix] \n\n"
},
{
"path": "code/chapter5.py",
"content": "import numpy as np\nimport cvxopt\nimport math\n\n\n########-----NaiveBayes------#########\nclass NaiveBayes():\n \n def __init__(self):\n self.parameters = [] # 保存每个特征针对每个类的均值和方差\n self.y = None\n self.classes = None\n\n def fit(self, X, y):\n self.y = y\n self.classes = np.unique(y) # 类别 \n # 计算每个特征针对每个类的均值和方差\n for i, c in enumerate(self.classes):\n # 选择类别为c的X\n X_where_c = X[np.where(self.y == c)]\n self.parameters.append([])\n # 添加均值与方差\n for col in X_where_c.T:\n parameters = {\"mean\": col.mean(), \"var\": col.var()}\n self.parameters[i].append(parameters)\n \n def _calculate_prior(self, c):\n \"\"\"\n 先验函数。\n \"\"\"\n frequency = np.mean(self.y == c)\n return frequency\n\n def _calculate_likelihood(self, mean, var, X):\n \"\"\"\n 似然函数。\n \"\"\"\n # 高斯概率\n eps = 1e-4 # 防止除数为0\n coeff = 1.0 / math.sqrt(2.0 * math.pi * var + eps)\n exponent = math.exp(-(math.pow(X - mean, 2) / (2 * var + eps)))\n return coeff * exponent\n \n def _calculate_probabilities(self, X):\n posteriors = []\n for i, c in enumerate(self.classes):\n posterior = self._calculate_prior(c)\n for feature_value, params in zip(X, self.parameters[i]):\n # 独立性假设\n # P(x1,x2|Y) = P(x1|Y)*P(x2|Y)\n likelihood = self._calculate_likelihood(params[\"mean\"], params[\"var\"], feature_value)\n posterior *= likelihood\n posteriors.append(posterior)\n # 返回具有最大后验概率的类别\n return self.classes[np.argmax(posteriors)]\n \n def predict(self, X):\n y_pred = [self._calculate_probabilities(sample) for sample in X]\n return y_pred\n \n def score(self, X, y):\n y_pred = self.predict(X)\n accuracy = np.sum(y == y_pred, axis=0) / len(y)\n return accuracy\n\n\n########-----LogisticRegression------#########\ndef Sigmoid(x):\n return 1/(1 + np.exp(-x))\n\nclass LogisticRegression():\n\n def __init__(self, learning_rate=.1):\n self.param = None\n self.learning_rate = learning_rate\n self.sigmoid = Sigmoid\n\n def _initialize_parameters(self, X):\n n_features = np.shape(X)[1]\n # 初始化参数theta， [-1/sqrt(N), 1/sqrt(N)]\n limit = 1 / math.sqrt(n_features)\n self.param = np.random.uniform(-limit, limit, (n_features,))\n\n def fit(self, X, y, n_iterations=4000):\n self._initialize_parameters(X)\n # 参数theta的迭代更新\n for i in range(n_iterations):\n # 求预测\n y_pred = self.sigmoid(X.dot(self.param))\n # 最小化损失函数，参数更新公式\n self.param -= self.learning_rate * -(y - y_pred).dot(X)\n\n def predict(self, X):\n y_pred = self.sigmoid(X.dot(self.param))\n return y_pred\n\n def score(self, X, y):\n y_pred = self.predict(X)\n accuracy = np.sum(y == y_pred, axis=0) / len(y)\n return accuracy\n \n\n########-----SupportVectorMachine------#########\n# 隐藏cvxopt输出\ncvxopt.solvers.options['show_progress'] = False\n\ndef linear_kernel(**kwargs):\n \"\"\"\n 线性核\n \"\"\"\n def f(x1, x2):\n return np.inner(x1, x2)\n return f\n\ndef polynomial_kernel(power, coef, **kwargs):\n \"\"\"\n 多项式核\n \"\"\"\n def f(x1, x2):\n return (np.inner(x1, x2) + coef)**power\n return f\n\ndef rbf_kernel(gamma, **kwargs):\n \"\"\"\n 高斯核\n \"\"\"\n def f(x1, x2):\n distance = np.linalg.norm(x1 - x2) ** 2\n return np.exp(-gamma * distance)\n return f\n\nclass SupportVectorMachine():\n\n def __init__(self, kernel=linear_kernel, power=4, gamma=None, coef=4):\n self.kernel = kernel\n self.power = power\n self.gamma = gamma\n self.coef = coef\n self.lagr_multipliers = None\n self.support_vectors = None\n self.support_vector_labels = None\n self.intercept = None\n\n def fit(self, X, y):\n\n n_samples, n_features = np.shape(X)\n\n # gamma默认设置为1 / n_features\n if not self.gamma:\n self.gamma = 1 / n_features\n \n # 定义核函数\n self.kernel = self.kernel(\n power=self.power,\n gamma=self.gamma,\n coef=self.coef)\n\n # 计算Gram矩阵\n kernel_matrix = np.zeros((n_samples, n_samples))\n for i in range(n_samples):\n for j in range(n_samples):\n kernel_matrix[i, j] = self.kernel(X[i], X[j])\n \n # 构造二次规划问题\n # 形式为 min (1/2)x.T*P*x+q.T*x, s.t. G*x<=h, A*x=b\n P = cvxopt.matrix(np.outer(y, y) * kernel_matrix, tc='d')\n q = cvxopt.matrix(np.ones(n_samples) * -1)\n A = cvxopt.matrix(y, (1, n_samples), tc='d')\n b = cvxopt.matrix(0, tc='d')\n\n G = cvxopt.matrix(np.identity(n_samples) * -1)\n h = cvxopt.matrix(np.zeros(n_samples))\n\n # 用cvxopt求解二次规划问题\n minimization = cvxopt.solvers.qp(P, q, G, h, A, b)\n lagr_mult = np.ravel(minimization['x'])\n # 非0的alpha值\n idx = lagr_mult > 1e-7\n # alpha值\n self.lagr_multipliers = lagr_mult[idx]\n # 支持向量\n self.support_vectors = X[idx]\n # 支持向量的标签\n self.support_vector_labels = y[idx]\n\n # 通过第一个支持向量计算b\n self.intercept = self.support_vector_labels[0]\n for i in range(len(self.lagr_multipliers)):\n self.intercept -= self.lagr_multipliers[i] * self.support_vector_labels[\n i] * self.kernel(self.support_vectors[i], self.support_vectors[0])\n\n def predict(self, X):\n y_pred = []\n for sample in X:\n # 对于输入的x, 计算f(x)\n prediction = 0\n for i in range(len(self.lagr_multipliers)):\n prediction += self.lagr_multipliers[i] * self.support_vector_labels[\n i] * self.kernel(self.support_vectors[i], sample)\n prediction += self.intercept\n y_pred.append(np.sign(prediction))\n return np.array(y_pred)\n \n def score(self, X, y):\n y_pred = self.predict(X)\n accuracy = np.sum(y == y_pred, axis=0) / len(y)\n return accuracy\n\n \n########-----KNN------#########\nclass KNN():\n \n def __init__(self, k=10):\n self._k = k\n\n def fit(self, X, y):\n self._unique_labels = np.unique(y)\n self._class_num = len(self._unique_labels)\n self._datas = X\n self._labels = y.astype(np.int32)\n\n def predict(self, X):\n # 欧式距离计算\n dist = np.sum(np.square(X), axis=1, keepdims=True) - 2 * np.dot(X, self._datas.T)\n dist = dist + np.sum(np.square(self._datas), axis=1, keepdims=True).T\n dist = np.argsort(dist)[:,:self._k]\n return np.array([np.argmax(np.bincount(self._labels[dist][i])) for i in range(len(X))])\n idx = lagr_mult > 1e-7\n # alpha值\n self.lagr_multipliers = lagr_mult[idx]\n # 支持向量\n self.support_vectors = X[idx]\n # 支持向量的标签\n self.support_vector_labels = y[idx]\n\n # 通过第一个支持向量计算b\n self.intercept = self.support_vector_labels[0]\n for i in range(len(self.lagr_multipliers)):\n self.intercept -= self.lagr_multipliers[i] * self.support_vector_labels[\n i] * self.kernel(self.support_vectors[i], self.support_vectors[0])\n\n def predict(self, X):\n y_pred = []\n for sample in X:\n # 对于输入的x, 计算f(x)\n prediction = 0\n for i in range(len(self.lagr_multipliers)):\n prediction += self.lagr_multipliers[i] * self.support_vector_labels[\n i] * self.kernel(self.support_vectors[i], sample)\n prediction += self.intercept\n y_pred.append(np.sign(prediction))\n return np.array(y_pred)\n\n def score(self, X, y):\n y_pred = self.predict(X)\n accuracy = np.sum(y == y_pred, axis=0) / len(y)\n return accuracy\n\n \n########-----DecisionTree------#########\nclass DecisionNode():\n\n def __init__(self, feature_i=None, threshold=None,\n value=None, true_branch=None, false_branch=None):\n self.feature_i = feature_i # 当前结点测试的特征的索引\n self.threshold = threshold # 当前结点测试的特征的阈值\n self.value = value # 结点值（如果结点为叶子结点）\n self.true_branch = true_branch # 左子树（满足阈值， 将特征值大于等于切分点值的数据划分为左子树）\n self.false_branch = false_branch # 右子树（未满足阈值， 将特征值小于切分点值的数据划分为右子树）\n\n \ndef divide_on_feature(X, feature_i, threshold):\n \"\"\"\n 依据切分变量和切分点，将数据集分为两个子区域\n \"\"\"\n split_func = None\n if isinstance(threshold, int) or isinstance(threshold, float):\n split_func = lambda sample: sample[feature_i] >= threshold\n else:\n split_func = lambda sample: sample[feature_i] == threshold\n\n X_1 = np.array([sample for sample in X if split_func(sample)])\n X_2 = np.array([sample for sample in X if not split_func(sample)])\n\n return np.array([X_1, X_2])\n\n\nclass DecisionTree(object):\n\n def __init__(self, min_samples_split=2, min_impurity=1e-7,\n max_depth=float(\"inf\"), loss=None):\n self.root = None # 根结点\n self.min_samples_split = min_samples_split # 满足切分的最少样本数\n self.min_impurity = min_impurity # 满足切分的最小纯度\n self.max_depth = max_depth # 树的最大深度\n self._impurity_calculation = None # 计算纯度的函数，如对于分类树采用信息增益\n self._leaf_value_calculation = None # 计算y在叶子结点值的函数\n self.one_dim = None # y是否为one-hot编码\n\n def fit(self, X, y):\n self.one_dim = len(np.shape(y)) == 1\n self.root = self._build_tree(X, y)\n\n def _build_tree(self, X, y, current_depth=0):\n \"\"\"\n 递归方法建立决策树\n \"\"\"\n largest_impurity = 0\n best_criteria = None # 当前最优分类的特征索引和阈值\n best_sets = None # 数据子集\n\n if len(np.shape(y)) == 1:\n y = np.expand_dims(y, axis=1)\n\n Xy = np.concatenate((X, y), axis=1)\n\n n_samples, n_features = np.shape(X)\n\n if n_samples >= self.min_samples_split and current_depth <= self.max_depth:\n # 对每个特征计算纯度\n for feature_i in range(n_features):\n feature_values = np.expand_dims(X[:, feature_i], axis=1)\n unique_values = np.unique(feature_values)\n\n # 遍历特征i所有的可能值找到最优纯度\n for threshold in unique_values:\n # 基于X在特征i处是否满足阈值来划分X和y， Xy1为满足阈值的子集\n Xy1, Xy2 = divide_on_feature(Xy, feature_i, threshold)\n\n if len(Xy1) > 0 and len(Xy2) > 0:\n # 取出Xy中y的集合\n y1 = Xy1[:, n_features:]\n y2 = Xy2[:, n_features:]\n\n # 计算纯度\n impurity = self._impurity_calculation(y, y1, y2)\n\n # 如果纯度更高，则更新\n if impurity > largest_impurity:\n largest_impurity = impurity\n best_criteria = {\"feature_i\": feature_i, \"threshold\": threshold}\n best_sets = {\n \"leftX\": Xy1[:, :n_features], # X的左子树\n \"lefty\": Xy1[:, n_features:], # y的左子树\n \"rightX\": Xy2[:, :n_features], # X的右子树\n \"righty\": Xy2[:, n_features:] # y的右子树\n }\n\n if largest_impurity > self.min_impurity:\n # 建立左子树和右子树\n true_branch = self._build_tree(best_sets[\"leftX\"], best_sets[\"lefty\"], current_depth + 1)\n false_branch = self._build_tree(best_sets[\"rightX\"], best_sets[\"righty\"], current_depth + 1)\n return DecisionNode(feature_i=best_criteria[\"feature_i\"], threshold=best_criteria[\n \"threshold\"], true_branch=true_branch, false_branch=false_branch)\n\n # 如果是叶结点则计算值\n leaf_value = self._leaf_value_calculation(y)\n\n return DecisionNode(value=leaf_value)\n\n\n def predict_value(self, x, tree=None):\n \"\"\"\n 预测样本，沿着树递归搜索\n \"\"\"\n # 根结点\n if tree is None:\n tree = self.root\n\n # 递归出口\n if tree.value is not None:\n return tree.value\n\n # 选择当前结点的特征\n feature_value = x[tree.feature_i]\n\n branch = tree.false_branch\n if isinstance(feature_value, int) or isinstance(feature_value, float):\n if feature_value >= tree.threshold:\n branch = tree.true_branch\n elif feature_value == tree.threshold:\n branch = tree.true_branch\n\n return self.predict_value(x, branch)\n\n def predict(self, X):\n y_pred = [self.predict_value(sample) for sample in X]\n return y_pred\n\n def score(self, X, y):\n y_pred = self.predict(X)\n accuracy = np.sum(y == y_pred, axis=0) / len(y)\n return accuracy\n \n def print_tree(self, tree=None, indent=\" \"):\n \"\"\"\n 输出树\n \"\"\"\n if not tree:\n tree = self.root\n\n if tree.value is not None:\n print(tree.value)\n else:\n print(\"feature|threshold -> %s | %s\" % (tree.feature_i, tree.threshold))\n print(\"%sT->\" % (indent), end=\"\")\n self.print_tree(tree.true_branch, indent + indent)\n print(\"%sF->\" % (indent), end=\"\")\n self.print_tree(tree.false_branch, indent + indent)\n\n\ndef calculate_entropy(y):\n log2 = lambda x: math.log(x) / math.log(2)\n unique_labels = np.unique(y)\n entropy = 0\n for label in unique_labels:\n count = len(y[y == label])\n p = count / len(y)\n entropy += -p * log2(p)\n return entropy\n\n\ndef calculate_gini(y):\n unique_labels = np.unique(y)\n var = 0\n for label in unique_labels:\n count = len(y[y == label])\n p = count / len(y)\n var += p ** 2\n return 1 - var\n\n\nclass ClassificationTree(DecisionTree):\n \"\"\"\n 分类树，在决策书节点选择计算信息增益/基尼指数，在叶子节点选择多数表决。\n \"\"\"\n def _calculate_gini_index(self, y, y1, y2):\n \"\"\"\n 计算基尼指数\n \"\"\"\n p = len(y1) / len(y)\n gini = calculate_gini(y)\n gini_index = gini - p * \\\n calculate_gini(y1) - (1 - p) * \\\n calculate_gini(y2)\n return gini_index\n \n \n def _calculate_information_gain(self, y, y1, y2):\n \"\"\"\n 计算信息增益\n \"\"\"\n p = len(y1) / len(y)\n entropy = calculate_entropy(y)\n info_gain = entropy - p * \\\n calculate_entropy(y1) - (1 - p) * \\\n calculate_entropy(y2)\n return info_gain\n\n def _majority_vote(self, y):\n \"\"\"\n 多数表决\n \"\"\"\n most_common = None\n max_count = 0\n for label in np.unique(y):\n count = len(y[y == label])\n if count > max_count:\n most_common = label\n max_count = count\n return most_common\n\n def fit(self, X, y):\n self._impurity_calculation = self._calculate_gini_index\n self._leaf_value_calculation = self._majority_vote\n super(ClassificationTree, self).fit(X, y)\n\n\ndef calculate_mse(y):\n return np.mean((y - np.mean(y)) ** 2)\n\n\ndef calculate_variance(y):\n n_samples = np.shape(y)[0]\n variance = (1 / n_samples) * np.diag((y - np.mean(y)).T.dot(y - np.mean(y)))\n return variance\n\n\nclass RegressionTree(DecisionTree):\n \"\"\"\n 回归树，在决策书节点选择计算MSE/方差降低，在叶子节点选择均值。\n \"\"\"\n def _calculate_mse(self, y, y1, y2):\n \"\"\"\n 计算MSE降低\n \"\"\"\n mse_tot = calculate_mse(y)\n mse_1 = calculate_mse(y1)\n mse_2 = calculate_mse(y2)\n frac_1 = len(y1) / len(y)\n frac_2 = len(y2) / len(y)\n mse_reduction = mse_tot - (frac_1 * mse_1 + frac_2 * mse_2)\n return mse_reduction\n \n def _calculate_variance_reduction(self, y, y1, y2):\n \"\"\"\n 计算方差降低\n \"\"\"\n var_tot = calculate_variance(y)\n var_1 = calculate_variance(y1)\n var_2 = calculate_variance(y2)\n frac_1 = len(y1) / len(y)\n frac_2 = len(y2) / len(y)\n variance_reduction = var_tot - (frac_1 * var_1 + frac_2 * var_2)\n return sum(variance_reduction)\n\n def _mean_of_y(self, y):\n \"\"\"\n 计算均值\n \"\"\"\n value = np.mean(y, axis=0)\n return value if len(value) > 1 else value[0]\n\n def fit(self, X, y):\n self._impurity_calculation = self._calculate_mse\n self._leaf_value_calculation = self._mean_of_y\n super(RegressionTree, self).fit(X, y)\n\n\n########-----PCA------#########\nclass PCA():\n \n def __init__(self):\n pass\n \n def fit(self, X, n_components):\n n_samples = np.shape(X)[0]\n covariance_matrix = (1 / (n_samples-1)) * (X - X.mean(axis=0)).T.dot(X - X.mean(axis=0))\n\n # 对协方差矩阵进行特征值分解\n eigenvalues, eigenvectors = np.linalg.eig(covariance_matrix)\n\n # 对特征值（特征向量）从大到小排序\n idx = eigenvalues.argsort()[::-1]\n eigenvalues = eigenvalues[idx][:n_components]\n eigenvectors = np.atleast_1d(eigenvectors[:, idx])[:, :n_components]\n\n # 得到低维表示\n X_transformed = X.dot(eigenvectors)\n\n\n########-----KMeans------#########\ndef distEclud(x,y):\n \"\"\"\n 计算欧氏距离\n \"\"\"\n return np.sqrt(np.sum((x-y)**2)) \n\ndef randomCent(dataSet,k):\n \"\"\"\n 为数据集构建一个包含 K 个随机质心的集合\n \"\"\"\n m,n = dataSet.shape\n centroids = np.zeros((k,n))\n for i in range(k):\n index = int(np.random.uniform(0,m))\n centroids[i,:] = dataSet[index,:]\n return centroids\n\nclass KMeans():\n \n def __init__(self):\n self.dataSet = None\n self.k = None\n \n def fit(self, dataSet, k):\n self.dataSet = dataSet\n self.k = k\n m = np.shape(dataSet)[0]\n # 第一列存样本属于哪一簇\n # 第二列存样本的到簇的中心点的误差\n clusterAssment = np.mat(np.zeros((m,2)))\n clusterChange = True\n centroids = randomCent(self.dataSet,k)\n while clusterChange:\n clusterChange = False\n for i in range(m):\n minDist = 1e6\n minIndex = -1\n # 遍历所有的质心, 找出最近的质心\n for j in range(k):\n distance = distEclud(centroids[j,:], self.dataSet[i,:])\n if distance < minDist:\n minDist = distance\n minIndex = j\n # 更新每一行样本所属的簇\n if clusterAssment[i,0] != minIndex:\n clusterChange = True\n clusterAssment[i,:] = minIndex, minDist**2\n # 更新质心\n for j in range(k):\n pointsInCluster = dataSet[np.nonzero(clusterAssment[:,0].A == j)[0]] # 获取簇类所有的点\n centroids[j,:] = np.mean(pointsInCluster,axis=0) # 对矩阵的行求均值\n\n return centroids,clusterAssment\n\n return X_transformed\n"
},
{
"path": "code/chapter6.py",
"content": "from abc import ABC, abstractmethod\nimport numpy as np\nimport time\nimport re\nimport inspect\nfrom collections import OrderedDict\n\nimport sys\nsys.path.append('../')\nfrom method.optimizer import OptimizerInitializer\nfrom method.weight import WeightInitializer\nfrom method.activation import ActivationInitializer\n\n\ndef sigmoid(x):\n return 1 / (1 + np.exp(-x))\n\n\ndef softmax(x):\n e_x = np.exp(x - np.max(x, axis=-1, keepdims=True))\n return e_x / e_x.sum(axis=-1, keepdims=True)\n \n \nclass LayerBase(ABC):\n \n def __init__(self, optimizer=\"sgd\"):\n self.X = [] # 网络层输入\n self.gradients = {} # 网络层待梯度更新变量\n self.params = {} # 网络层参数变量\n self.acti_fn = None # 网络层激活函数\n self.optimizer = OptimizerInitializer(optimizer)() # 网络层优化方法\n\n @abstractmethod\n def _init_params(self, **kwargs):\n \"\"\"\n 函数作用：初始化参数\n \"\"\"\n raise NotImplementedError\n \n @abstractmethod\n def forward(self, X, **kwargs): \n \"\"\"\n 函数作用：前向传播\n \"\"\"\n raise NotImplementedError\n\n @abstractmethod\n def backward(self, out, **kwargs):\n \"\"\"\n 函数作用：反向传播\n \"\"\"\n raise NotImplementedError\n \n def flush_gradients(self):\n \"\"\"\n 函数作用：重置更新参数列表\n \"\"\"\n self.X = []\n for k, v in self.gradients.items():\n self.gradients[k] = np.zeros_like(v)\n \n for k, v in self.derived_variables.items():\n self.derived_variables[k] = []\n\n def update(self):\n \"\"\"\n 函数作用：更新参数\n \"\"\"\n for k, v in self.gradients.items():\n if k in self.params:\n self.params[k] = self.optimizer(self.params[k], v, k)\n \n \nclass FullyConnected(LayerBase):\n \"\"\"\n 定义全连接层，实现 a=g(x*W+b)，前向传播输入x，返回a；反向传播输入\n \"\"\"\n \n def __init__(self, n_out, acti_fn, init_w, optimizer=None):\n \"\"\"\n 参数说明：\n acti_fn：激活函数， str型\n init_w：权重初始化方法， str型\n n_out：隐藏层输出维数\n optimizer：优化方法\n \"\"\"\n super().__init__(optimizer)\n \n self.n_in = None # 隐藏层输入维数， int型\n self.n_out = n_out # 隐藏层输出维数， int型\n self.acti_fn = ActivationInitializer(acti_fn)()\n self.init_w = init_w\n self.init_weights = WeightInitializer(mode=init_w)\n self.is_initialized = False # 是否初始化， bool型变量\n \n def _init_params(self):\n b = np.zeros((1, self.n_out))\n W = self.init_weights((self.n_in, self.n_out))\n self.params = {\"W\": W, \"b\": b}\n self.gradients = {\"W\": np.zeros_like(W), \"b\": np.zeros_like(b)}\n self.derived_variables = {\"Z\": []}\n self.is_initialized = True\n \n def forward(self, X, retain_derived=True):\n \"\"\"\n 全连接网络的前向传播，原理见上文 反向传播算法 部分。\n \n 参数说明：\n X：输入数组，为（n_samples, n_in），float型\n retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型\n \"\"\"\n if not self.is_initialized: # 如果参数未初始化，先初始化参数\n self.n_in = X.shape[1]\n self._init_params()\n \n W = self.params[\"W\"]\n b = self.params[\"b\"]\n z = X @ W + b\n a = self.acti_fn.forward(z)\n \n if retain_derived:\n self.X.append(X)\n \n return a\n \n def backward(self, dLda, retain_grads=True):\n \"\"\"\n 全连接网络的反向传播，原理见上文 反向传播算法 部分。\n \n 参数说明：\n dLda：关于损失的梯度，为（n_samples, n_out），float型\n retain_grads：是否计算中间变量的参数梯度，bool型\n \"\"\"\n if not isinstance(dLda, list):\n dLda = [dLda]\n \n dX = []\n X = self.X\n for da, x in zip(dLda, X):\n dx, dw, db = self._bwd(da, x)\n dX.append(dx)\n\n if retain_grads:\n self.gradients[\"W\"] += dw\n self.gradients[\"b\"] += db\n \n return dX[0] if len(X) == 1 else dX\n\n def _bwd(self, dLda, X):\n W = self.params[\"W\"]\n b = self.params[\"b\"]\n\n Z = X @ W + b\n dZ = dLda * self.acti_fn.grad(Z)\n\n dX = dZ @ W.T\n dW = X.T @ dZ\n db = dZ.sum(axis=0, keepdims=True)\n return dX, dW, db\n \n @property\n def hyperparams(self):\n return {\n \"layer\": \"FullyConnected\",\n \"init_w\": self.init_w,\n \"n_in\": self.n_in,\n \"n_out\": self.n_out,\n \"acti_fn\": str(self.acti_fn),\n \"optimizer\": {\n \"hyperparams\": self.optimizer.hyperparams,\n },\n \"components\": {\n k: v for k, v in self.params.items()\n }\n }\n \n \nclass ObjectiveBase(ABC):\n \n def __init__(self):\n super().__init__()\n\n @abstractmethod\n def loss(self, y_true, y_pred):\n \"\"\"\n 函数作用：计算损失\n \"\"\"\n raise NotImplementedError\n\n @abstractmethod\n def grad(self, y_true, y_pred, **kwargs):\n \"\"\"\n 函数作用：计算代价函数的梯度\n \"\"\"\n raise NotImplementedError\n\n\nclass SquaredError(ObjectiveBase):\n \"\"\"\n 二次代价函数。\n \"\"\"\n def __init__(self):\n super().__init__()\n\n def __call__(self, y_true, y_pred):\n return self.loss(y_true, y_pred)\n\n def __str__(self):\n return \"SquaredError\"\n\n @staticmethod\n def loss(y_true, y_pred):\n \"\"\"\n 参数说明：\n y_true：训练的 n 个样本的真实值， 形状为(n,m)数组；\n y_pred：训练的 n 个样本的预测值， 形状为(n,m)数组；\n \"\"\"\n (n, _) = y_true.shape\n return 0.5 * np.linalg.norm(y_pred - y_true) ** 2 / n\n\n @staticmethod\n def grad(y_true, y_pred, z, acti_fn):\n (n, _) = y_true.shape\n return (y_pred - y_true) * acti_fn.grad(z) / n\n\n\nclass CrossEntropy(ObjectiveBase):\n \"\"\"\n 交叉熵代价函数。\n \"\"\"\n def __init__(self):\n super().__init__()\n\n def __call__(self, y_true, y_pred):\n return self.loss(y_true, y_pred)\n\n def __str__(self):\n return \"CrossEntropy\"\n\n @staticmethod\n def loss(y_true, y_pred):\n \"\"\"\n 参数说明：\n y_true：训练的 n 个样本的真实值， 要求形状为(n,m)二进制（每个样本均为 one-hot 编码）；\n y_pred：训练的 n 个样本的预测值， 形状为(n,m)；\n \"\"\"\n (n, _) = y_true.shape\n eps = np.finfo(float).eps # 防止 np.log(0)\n cross_entropy = -np.sum(y_true * np.log(y_pred + eps)) / n \n return cross_entropy\n \n @staticmethod\n def grad(y_true, y_pred):\n (n, _) = y_true.shape\n grad = (y_pred - y_true) / n\n return grad\n \n \ndef minibatch(X, batchsize=256, shuffle=True):\n \"\"\"\n 函数作用：将数据集分割成 batch， 基于 mini batch 训练。\n \"\"\"\n N = X.shape[0]\n idx = np.arange(N)\n n_batches = int(np.ceil(N / batchsize))\n\n if shuffle:\n np.random.shuffle(idx)\n\n def mb_generator():\n for i in range(n_batches):\n yield idx[i * batchsize : (i + 1) * batchsize]\n\n return mb_generator(), n_batches\n\n\nclass DFN(object):\n \n def __init__(\n self,\n hidden_dims_1=None,\n hidden_dims_2=None,\n optimizer=\"sgd(lr=0.01)\",\n init_w=\"std_normal\",\n loss=CrossEntropy()\n ):\n self.optimizer = optimizer\n self.init_w = init_w\n self.loss = loss\n self.hidden_dims_1 = hidden_dims_1\n self.hidden_dims_2 = hidden_dims_2\n self.is_initialized = False\n \n def _set_params(self):\n \"\"\"\n 函数作用：模型初始化\n FC1 -> Sigmoid -> FC2 -> Softmax\n \"\"\"\n self.layers = OrderedDict()\n self.layers[\"FC1\"] = FullyConnected(\n n_out=self.hidden_dims_1,\n acti_fn=\"sigmoid\", \n init_w=self.init_w,\n optimizer=self.optimizer\n )\n self.layers[\"FC2\"] = FullyConnected(\n n_out=self.hidden_dims_2,\n acti_fn=\"affine(slope=1, intercept=0)\",\n init_w=self.init_w,\n optimizer=self.optimizer\n )\n self.is_initialized = True\n \n def forward(self, X_train):\n Xs = {}\n out = X_train\n for k, v in self.layers.items():\n Xs[k] = out\n out = v.forward(out)\n return out, Xs\n \n def backward(self, grad):\n dXs = {}\n out = grad\n for k, v in reversed(list(self.layers.items())):\n dXs[k] = out\n out = v.backward(out)\n return out, dXs\n \n def update(self):\n \"\"\"\n 函数作用：梯度更新\n \"\"\"\n for k, v in reversed(list(self.layers.items())):\n v.update()\n self.flush_gradients()\n \n def flush_gradients(self, curr_loss=None):\n \"\"\"\n 函数作用：更新后重置梯度\n \"\"\"\n for k, v in self.layers.items():\n v.flush_gradients()\n \n def fit(self, X_train, y_train, n_epochs=20, batch_size=64, verbose=False, epo_verbose=True):\n \"\"\"\n 参数说明：\n X_train：训练数据\n y_train：训练数据标签\n n_epochs：epoch 次数\n batch_size：每次 epoch 的 batch size\n verbose：是否每个 batch 输出损失\n epo_verbose：是否每个 epoch 输出损失\n \"\"\"\n self.verbose = verbose\n self.n_epochs = n_epochs\n self.batch_size = batch_size\n \n if not self.is_initialized:\n self.n_features = X_train.shape[1]\n self._set_params()\n \n prev_loss = np.inf\n for i in range(n_epochs):\n loss, epoch_start = 0.0, time.time()\n batch_generator, n_batch = minibatch(X_train, self.batch_size, shuffle=True)\n\n for j, batch_idx in enumerate(batch_generator):\n batch_len, batch_start = len(batch_idx), time.time()\n X_batch, y_batch = X_train[batch_idx], y_train[batch_idx]\n out, _ = self.forward(X_batch)\n y_pred_batch = softmax(out)\n batch_loss = self.loss(y_batch, y_pred_batch)\n grad = self.loss.grad(y_batch, y_pred_batch)\n _, _ = self.backward(grad)\n self.update()\n loss += batch_loss\n\n if self.verbose:\n fstr = \"\\t[Batch {}/{}] Train loss: {:.3f} ({:.1f}s/batch)\"\n print(fstr.format(j + 1, n_batch, batch_loss, time.time() - batch_start))\n\n loss /= n_batch\n if epo_verbose:\n fstr = \"[Epoch {}] Avg. loss: {:.3f} Delta: {:.3f} ({:.2f}m/epoch)\"\n print(fstr.format(i + 1, loss, prev_loss - loss, (time.time() - epoch_start) / 60.0))\n prev_loss = loss\n \n def evaluate(self, X_test, y_test, batch_size=128):\n acc = 0.0\n batch_generator, n_batch = minibatch(X_test, batch_size, shuffle=True)\n for j, batch_idx in enumerate(batch_generator):\n batch_len, batch_start = len(batch_idx), time.time()\n X_batch, y_batch = X_test[batch_idx], y_test[batch_idx]\n y_pred_batch, _ = self.forward(X_batch)\n y_pred_batch = np.argmax(y_pred_batch, axis=1)\n y_batch = np.argmax(y_batch, axis=1)\n acc += np.sum(y_pred_batch == y_batch)\n return acc / X_test.shape[0]\n \n @property\n def hyperparams(self):\n return {\n \"init_w\": self.init_w,\n \"loss\": str(self.loss),\n \"optimizer\": self.optimizer,\n \"hidden_dims_1\": self.hidden_dims_1,\n \"hidden_dims_2\": self.hidden_dims_2,\n \"components\": {k: v.params for k, v in self.layers.items()}\n }\n "
},
{
"path": "code/chapter7.py",
"content": "from abc import ABC, abstractmethod\nimport numpy as np\nimport math\nimport re\nimport progressbar\nfrom chapter5 import RegressionTree, DecisionTree, ClassificationTree\n\n#########---Regularizer---######\nclass RegularizerBase(ABC):\n \n def __init__(self, **kwargs):\n super().__init__()\n \n @abstractmethod\n def loss(self, **kwargs):\n raise NotImplementedError\n \n @abstractmethod\n def grad(self, **kwargs):\n raise NotImplementedError\n\nclass L1Regularizer(RegularizerBase):\n \n def __init__(self, lambd=0.001):\n super().__init__()\n self.lambd = lambd\n \n def loss(self, params):\n loss = 0\n pattern = re.compile(r'^W\\d+')\n for key, val in params.items():\n if pattern.match(key):\n loss += 0.5 * np.sum(np.abs(val)) * self.lambd\n return loss\n \n def grad(self, params):\n for key, val in params.items():\n grad = self.lambd * np.sign(val)\n return grad\n \nclass L2Regularizer(RegularizerBase):\n \n def __init__(self, lambd=0.001):\n super().__init__()\n self.lambd = lambd\n \n def loss(self, params):\n loss = 0\n for key, val in params.items():\n loss += 0.5 * np.sum(np.square(val)) * self.lambd\n return loss\n \n def grad(self, params):\n for key, val in params.items():\n grad = self.lambd * val\n return grad\n \nclass RegularizerInitializer(object):\n \n def __init__(self, regular_name=\"l2\"):\n self.regular_name = regular_name\n \n def __call__(self):\n r = r\"([a-zA-Z]*)=([^,)]*)\"\n regular_str = self.regular_name.lower()\n kwargs = dict([(i, eval(j)) for (i, j) in re.findall(r, regular_str)])\n if \"l1\" in regular_str.lower():\n regular = L1Regularizer(**kwargs)\n elif \"l2\" in regular_str.lower():\n regular = L2Regularizer(**kwargs)\n else:\n raise ValueError(\"Unrecognized regular: {}\".format(regular_str))\n return regular\n \n\n#######----Dataset Augmentation----####\nclass Image(object):\n \n def __init__(self, image):\n self._set_params(image)\n \n def _set_params(self, image):\n self.img = image \n self.row = image.shape[0] # 图像高度\n self.col = image.shape[1] # 图像宽度\n self.transform = None\n\n def Translation(self, delta_x, delta_y):\n \"\"\"\n 平移。\n \n 参数说明：\n delta_x：控制左右平移，若大于0左移，小于0右移\n delta_y：控制上下平移，若大于0上移，小于0下移\n \"\"\"\n self.transform = np.array([[1, 0, delta_x], \n [0, 1, delta_y], \n [0, 0, 1]])\n\n def Resize(self, alpha):\n \"\"\"\n 缩放。\n \n 参数说明：\n alpha：缩放因子，不进行缩放设置为1\n \"\"\"\n self.transform = np.array([[alpha, 0, 0], \n [0, alpha, 0], \n [0, 0, 1]])\n\n def HorMirror(self): \n \"\"\"\n 水平镜像。\n \"\"\"\n self.transform = np.array([[1, 0, 0], \n [0, -1, self.col-1], \n [0, 0, 1]])\n\n def VerMirror(self): \n \"\"\"\n 垂直镜像。\n \"\"\"\n self.transform = np.array([[-1, 0, self.row-1], \n [0, 1, 0], \n [0, 0, 1]])\n\n def Rotate(self, angle): \n \"\"\"\n 旋转。\n \n 参数说明：\n angle：旋转角度\n \"\"\"\n self.transform = np.array([[math.cos(angle),-math.sin(angle),0],\n [math.sin(angle), math.cos(angle),0],\n [ 0, 0, 1]]) \n\n def operate(self):\n temp = np.zeros(self.img.shape, dtype=self.img.dtype)\n for i in range(self.row):\n for j in range(self.col):\n temp_pos = np.array([i, j, 1])\n [x,y,z] = np.dot(self.transform, temp_pos)\n x = int(x)\n y = int(y)\n\n if x>=self.row or y>=self.col or x<0 or y<0:\n temp[i,j,:] = 0\n else:\n temp[i,j,:] = self.img[x,y]\n return temp\n \n def __call__(self, act):\n r = r\"([a-zA-Z]*)=([^,)]*)\"\n act_str = act.lower()\n kwargs = dict([(i, eval(j)) for (i, j) in re.findall(r, act_str)])\n if \"translation\" in act_str:\n self.Translation(**kwargs)\n elif \"resize\" in act_str:\n self.Resize(**kwargs)\n elif \"hormirror\" in act_str:\n self.HorMirror(**kwargs)\n elif \"vermirror\" in act_str:\n self.VerMirror(**kwargs)\n elif \"rotate\" in act_str:\n self.Rotate(**kwargs)\n return self.operate()\n\n \n#######----Early Stopping----####\ndef early_stopping(valid):\n \"\"\"\n 参数说明：\n valid：验证集正确率列表\n \"\"\"\n if len(valid) > 5:\n if valid[-1] < valid[-5] and valid[-2] < valid[-5] and valid[-3] < valid[-5] and valid[-4] < valid[-5]:\n return True\n return False\n\n\n#####---Bagging--#####\ndef bootstrap_sample(X, Y):\n N, M = X.shape\n idxs = np.random.choice(N, N, replace=True)\n return X[idxs], Y[idxs]\n\nclass BaggingModel(object):\n\n def __init__(self, n_models):\n \"\"\"\n 参数说明：\n n_models：网络模型数目\n \"\"\"\n self.models = []\n self.n_models = n_models\n\n def fit(self, X, Y):\n self.models = []\n for i in range(self.n_models):\n print(\"training {} base model:\".format(i))\n X_samp, Y_samp = bootstrap_sample(X, Y)\n model = DFN(hidden_dims_1=200, hidden_dims_2=10)\n model.fit(X_samp, Y_samp)\n self.models.append(model)\n\n def predict(self, X):\n model_preds = np.array([[np.argmax(t.forward(x)[0]) for x in X] for t in self.models])\n return self._vote(model_preds)\n\n def _vote(self, predictions):\n out = [np.bincount(x).argmax() for x in predictions.T]\n return np.array(out)\n \n def evaluate(self, X_test, y_test):\n acc = 0.0\n y_pred = self.predict(X_test)\n y_true = np.argmax(y_test, axis=1)\n acc += np.sum(y_pred == y_true)\n return acc / X_test.shape[0]\n \n\n#####----Dropout----#######\nclass Dropout(ABC):\n \n def __init__(self, wrapped_layer, p):\n \"\"\"\n 参数说明：\n wrapped_layer：被 dropout 的层\n p：神经元保留率\n \"\"\"\n super().__init__()\n self._base_layer = wrapped_layer\n self.p = p\n self._init_wrapper_params()\n \n def _init_wrapper_params(self):\n self._wrapper_derived_variables = {\"dropout_mask\": None}\n self._wrapper_hyperparams = {\"wrapper\": \"Dropout\", \"p\": self.p}\n \n def flush_gradients(self):\n \"\"\"\n 函数作用：调用 base layer 重置更新参数列表\n \"\"\"\n self._base_layer.flush_gradients()\n \n def update(self):\n \"\"\"\n 函数作用：调用 base layer 更新参数\n \"\"\"\n self._base_layer.update()\n \n def forward(self, X, is_train=True):\n \"\"\"\n 参数说明：\n X：输入数组；\n is_train：是否为训练阶段，bool型；\n \"\"\"\n mask = np.ones(X.shape).astype(bool)\n if is_train:\n mask = (np.random.rand(*X.shape) < self.p) / self.p\n X = mask * X\n self._wrapper_derived_variables[\"dropout_mask\"] = mask\n return self._base_layer.forward(X)\n \n def backward(self, dLda):\n return self._base_layer.backward(dLda)\n \n @property\n def hyperparams(self):\n hp = self._base_layer.hyperparams\n hpw = self._wrapper_hyperparams\n if \"wrappers\" in hp:\n hp[\"wrappers\"].append(hpw)\n else:\n hp[\"wrappers\"] = [hpw]\n return hp\n\n\n#####----Bagging----#######\n# 进度条\nbar_widgets = [\n 'Training: ', progressbar.Percentage(), ' ', progressbar.Bar(marker=\"-\", left=\"[\", right=\"]\"),\n ' ', progressbar.ETA()\n]\n\ndef get_random_subsets(X, y, n_subsets, replacements=True):\n \"\"\"从训练数据中抽取数据子集 (默认可重复抽样)\"\"\"\n n_samples = np.shape(X)[0]\n # 将 X 和 y 拼接，并将元素随机排序\n Xy = np.concatenate((X, y.reshape((1, len(y))).T), axis=1)\n np.random.shuffle(Xy)\n subsets = []\n # 如果抽样时不重复抽样，可以只使用 50% 的训练数据；如果抽样时可重复抽样，使用全部的训练数据，默认可重复抽样\n subsample_size = int(n_samples // 2)\n if replacements:\n subsample_size = n_samples \n for _ in range(n_subsets):\n idx = np.random.choice(\n range(n_samples),\n size=np.shape(range(subsample_size)),\n replace=replacements)\n X = Xy[idx][:, :-1]\n y = Xy[idx][:, -1]\n subsets.append([X, y])\n return subsets\n\n\nclass Bagging():\n \"\"\"\n Bagging分类器。使用一组分类树，这些分类树使用特征训练数据的随机子集。\n \"\"\"\n def __init__(self, n_estimators=100, max_features=None, min_samples_split=2,\n min_gain=0, max_depth=float(\"inf\")):\n self.n_estimators = n_estimators # 树的数目\n self.min_samples_split = min_samples_split # 分割所需的最小样本数\n self.min_gain = min_gain # 分割所需的最小纯度 (最小信息增益)\n self.max_depth = max_depth # 树的最大深度\n self.progressbar = progressbar.ProgressBar(widgets=bar_widgets)\n\n # 初始化决策树\n self.trees = []\n for _ in range(n_estimators):\n self.trees.append(\n ClassificationTree(\n min_samples_split=self.min_samples_split,\n min_impurity=min_gain,\n max_depth=self.max_depth))\n\n def fit(self, X, y):\n # 对每棵树选择数据集的随机子集\n subsets = get_random_subsets(X, y, self.n_estimators)\n for i in self.progressbar(range(self.n_estimators)):\n X_subset, y_subset = subsets[i]\n # 用特征子集和真实值训练一棵子模型 (这里的数据也是训练数据集的随机子集)\n self.trees[i].fit(X_subset, y_subset)\n\n def predict(self, X):\n y_preds = np.empty((X.shape[0], len(self.trees)))\n # 每棵决策树都在数据上预测\n for i, tree in enumerate(self.trees):\n # 基于特征做出预测\n prediction = tree.predict(X)\n y_preds[:, i] = prediction\n \n y_pred = []\n # 对每个样本，选择最常见的类别作为预测\n for sample_predictions in y_preds:\n y_pred.append(np.bincount(sample_predictions.astype('int')).argmax())\n return y_pred\n \n def score(self, X, y):\n y_pred = self.predict(X)\n accuracy = np.sum(y == y_pred, axis=0) / len(y)\n return accuracy\n\n \n#####----RandomForest----#######\nclass RandomForest():\n \"\"\"\n 随机森林分类器。使用一组分类树，这些分类树使用特征的随机子集训练数据的随机子集。\n \"\"\"\n def __init__(self, n_estimators=100, max_features=None, min_samples_split=2,\n min_gain=0, max_depth=float(\"inf\")):\n self.n_estimators = n_estimators # 树的数目\n self.max_features = max_features # 每棵树的最大使用特征数\n self.min_samples_split = min_samples_split # 分割所需的最小样本数\n self.min_gain = min_gain # 分割所需的最小纯度 (最小信息增益)\n self.max_depth = max_depth # 树的最大深度\n self.progressbar = progressbar.ProgressBar(widgets=bar_widgets)\n\n # 初始化决策树\n self.trees = []\n for _ in range(n_estimators):\n self.trees.append(\n ClassificationTree(\n min_samples_split=self.min_samples_split,\n min_impurity=min_gain,\n max_depth=self.max_depth))\n\n def fit(self, X, y):\n n_features = np.shape(X)[1]\n # 如果 max_features 没有定义，取默认值 sqrt(n_features)\n if not self.max_features:\n self.max_features = int(math.sqrt(n_features))\n\n # 对每棵树选择数据集的随机子集\n subsets = get_random_subsets(X, y, self.n_estimators)\n\n for i in self.progressbar(range(self.n_estimators)):\n X_subset, y_subset = subsets[i]\n # 选择特征的随机子集\n idx = np.random.choice(range(n_features), size=self.max_features, replace=True)\n # 保存特征的索引用于预测\n self.trees[i].feature_indices = idx\n # 选择索引对应的特征\n X_subset = X_subset[:, idx]\n # 用特征子集和真实值训练一棵子模型 (这里的数据也是训练数据集的随机子集)\n self.trees[i].fit(X_subset, y_subset)\n\n def predict(self, X):\n y_preds = np.empty((X.shape[0], len(self.trees)))\n # 每棵决策树都在数据上预测\n for i, tree in enumerate(self.trees):\n # 使用该决策树训练使用的特征\n idx = tree.feature_indices\n # 基于特征做出预测\n prediction = tree.predict(X[:, idx])\n y_preds[:, i] = prediction\n \n y_pred = []\n # 对每个样本，选择最常见的类别作为预测\n for sample_predictions in y_preds:\n y_pred.append(np.bincount(sample_predictions.astype('int')).argmax())\n return y_pred\n \n def score(self, X, y):\n y_pred = self.predict(X)\n accuracy = np.sum(y == y_pred, axis=0) / len(y)\n return accuracy\n\n \n#####----Adaboost----#######\n# 决策树桩，作为 Adaboost 算法的弱分类器 (基分类器)\nclass DecisionStump():\n \n def __init__(self):\n self.polarity = 1 # 表示决策树桩默认输出的类别为 1 或是 -1\n self.feature_index = None # 用于分类的特征索引\n self.threshold = None # 特征的阈值\n self.alpha = None # 表示分类器准确性的值\n\nclass Adaboost():\n \"\"\"\n Adaboost 算法。\n \"\"\"\n def __init__(self, n_estimators=5):\n self.n_estimators = n_estimators # 将使用的弱分类器的数量\n self.progressbar = progressbar.ProgressBar(widgets=bar_widgets)\n\n def fit(self, X, y):\n n_samples, n_features = np.shape(X)\n # 初始化权重 (上文中的 D)，均为 1/N\n w = np.full(n_samples, (1 / n_samples))\n self.trees = []\n # 迭代过程\n for _ in self.progressbar(range(self.n_estimators)):\n tree = DecisionStump()\n min_error = float('inf') # 使用某一特征值的阈值预测样本的最小误差\n # 迭代遍历每个 (不重复的) 特征值，查找预测 y 的最佳阈值\n for feature_i in range(n_features):\n feature_values = np.expand_dims(X[:, feature_i], axis=1)\n unique_values = np.unique(feature_values)\n # 将该特征的每个特征值作为阈值\n for threshold in unique_values:\n p = 1\n # 将所有样本预测默认值可以设置为 1\n prediction = np.ones(np.shape(y))\n # 低于特征值阈值的预测改为 -1\n prediction[X[:, feature_i] < threshold] = -1\n # 计算错误率\n error = sum(w[y != prediction])\n # 如果错误率超过 50%，我们反转决策树桩默认输出的类别\n # 比如 error = 0.8 => (1 - error) = 0.2，\n # 原来计算的是输出到类别 1 的概率，类别 1 作为默认类别。反转后类别 0 作为默认类别\n if error > 0.5:\n error = 1 - error\n p = -1\n # 如果这个阈值导致最小的错误率，则保存\n if error < min_error:\n tree.polarity = p\n tree.threshold = threshold\n tree.feature_index = feature_i\n min_error = error\n \n # 计算用于更新样本权值的 alpha 值，也是作为基分类器的系数。\n tree.alpha = 0.5 * math.log((1.0 - min_error) / (min_error + 1e-10))\n # 将所有样本预测默认值设置为 1\n predictions = np.ones(np.shape(y))\n # 如果特征值低于阈值，则修改预测结果，这里还需要考虑弱分类器的默认输出类别\n negative_idx = (tree.polarity * X[:, tree.feature_index] < tree.polarity * tree.threshold)\n predictions[negative_idx] = -1\n # 计算新权值，未正确分类样本的权值增大，正确分类样本的权值减小\n w *= np.exp(-tree.alpha * y * predictions)\n w /= np.sum(w)\n # 保存分类器\n self.trees.append(tree)\n\n def predict(self, X):\n n_samples = np.shape(X)[0]\n y_pred = np.zeros((n_samples, 1))\n # 用每一个基分类器预测样本\n for tree in self.trees:\n # 将所有样本预测默认值设置为 1\n predictions = np.ones(np.shape(y_pred))\n negative_idx = (tree.polarity * X[:, tree.feature_index] < tree.polarity * tree.threshold)\n predictions[negative_idx] = -1\n # 对基分类器加权求和，权重 alpha\n y_pred += tree.alpha * predictions\n # 返回预测结果 1 或 -1\n y_pred = np.sign(y_pred).flatten()\n return y_pred\n \n def score(self, X, y):\n y_pred = self.predict(X)\n accuracy = np.sum(y == y_pred, axis=0) / len(y)\n return accuracy\n\n \n#####----GBDT----#######\nclass Loss(ABC):\n\n def __init__(self):\n super().__init__()\n\n @abstractmethod \n def loss(self, y_true, y_pred):\n return NotImplementedError()\n\n @abstractmethod \n def grad(self, y, y_pred):\n raise NotImplementedError()\n\nclass SquareLoss(Loss):\n \n def __init__(self): \n pass\n\n def loss(self, y, y_pred):\n pass\n\n def grad(self, y, y_pred):\n return -(y - y_pred)\n \n def hess(self, y, y_pred):\n return 1\n\nclass CrossEntropyLoss(Loss):\n \n def __init__(self): \n pass\n\n def loss(self, y, y_pred):\n pass\n\n def grad(self, y, y_pred):\n return - (y - y_pred) \n \n def hess(self, y, y_pred):\n return y_pred * (1-y_pred)\n\n\ndef softmax(x):\n e_x = np.exp(x - np.max(x, axis=-1, keepdims=True))\n return e_x / e_x.sum(axis=-1, keepdims=True)\n\n\ndef line_search(self, y, y_pred, h_pred):\n Lp = 2 * np.sum((y - y_pred) * h_pred)\n Lpp = np.sum(h_pred * h_pred)\n return 1 if np.sum(Lpp) == 0 else Lp / Lpp\n\n\ndef to_categorical(x, n_classes=None):\n \"\"\"\n One-hot编码\n \"\"\"\n if not n_classes:\n n_classes = np.amax(x) + 1\n one_hot = np.zeros((x.shape[0], n_classes))\n one_hot[np.arange(x.shape[0]), x] = 1\n return one_hot\n\n\nclass GradientBoostingDecisionTree(object):\n \"\"\"\n GBDT 算法。用一组基学习器 (回归树) 学习损失函数的梯度。\n \"\"\"\n def __init__(self, n_estimators, learning_rate=1, min_samples_split=2,\n min_impurity=1e-7, max_depth=float(\"inf\"), is_regression=False, line_search=False):\n self.n_estimators = n_estimators # 迭代的次数\n self.learning_rate = learning_rate # 训练过程中沿着负梯度走的步长，也就是学习率\n self.min_samples_split = min_samples_split # 分割所需的最小样本数\n self.min_impurity = min_impurity # 分割所需的最小纯度\n self.max_depth = max_depth # 树的最大深度\n self.is_regression = is_regression # 分类问题或回归问题\n self.line_search = line_search # 是否使用 line search\n self.progressbar = progressbar.ProgressBar(widgets=bar_widgets) \n # 回归问题采用基础的平方损失，分类问题采用交叉熵损失\n self.loss = SquareLoss()\n if not self.is_regression:\n self.loss = CrossEntropyLoss()\n\n def fit(self, X, Y):\n # 分类问题将 Y 转化为 one-hot 编码\n if not self.is_regression:\n Y = to_categorical(Y.flatten())\n else:\n Y = Y.reshape(-1, 1) if len(Y.shape) == 1 else Y\n self.out_dims = Y.shape[1]\n self.trees = np.empty((self.n_estimators, self.out_dims), dtype=object)\n Y_pred = np.full(np.shape(Y), np.mean(Y, axis=0))\n self.weights = np.ones((self.n_estimators, self.out_dims))\n self.weights[1:, :] *= self.learning_rate\n # 迭代过程\n for i in self.progressbar(range(self.n_estimators)):\n for c in range(self.out_dims):\n tree = RegressionTree(\n min_samples_split=self.min_samples_split,\n min_impurity=self.min_impurity,\n max_depth=self.max_depth)\n # 计算损失的梯度，并用梯度进行训练\n if not self.is_regression: \n Y_hat = softmax(Y_pred)\n y, y_pred = Y[:, c], Y_hat[:, c]\n else:\n y, y_pred = Y[:, c], Y_pred[:, c]\n neg_grad = -1 * self.loss.grad(y, y_pred)\n tree.fit(X, neg_grad)\n # 用新的基学习器进行预测\n h_pred = tree.predict(X)\n # line search\n if self.line_search == True:\n self.weights[i, c] *= line_search(y, y_pred, h_pred)\n # 加法模型中添加基学习器的预测，得到最新迭代下的加法模型预测\n Y_pred[:, c] += np.multiply(self.weights[i, c], h_pred)\n self.trees[i, c] = tree\n \n def predict(self, X):\n Y_pred = np.zeros((X.shape[0], self.out_dims))\n # 生成预测\n for c in range(self.out_dims):\n y_pred = np.array([])\n for i in range(self.n_estimators):\n update = np.multiply(self.weights[i, c], self.trees[i, c].predict(X))\n y_pred = update if not y_pred.any() else y_pred + update\n Y_pred[:, c] = y_pred\n if not self.is_regression: \n # 分类问题输出最可能类别\n Y_pred = Y_pred.argmax(axis=1)\n return Y_pred\n \n def score(self, X, y):\n y_pred = self.predict(X)\n accuracy = np.sum(y == y_pred, axis=0) / len(y)\n return accuracy\n\n\nclass GradientBoostingRegressor(GradientBoostingDecisionTree):\n \n def __init__(self, n_estimators=200, learning_rate=1, min_samples_split=2,\n min_impurity=1e-7, max_depth=float(\"inf\"), is_regression=True, line_search=False):\n super(GradientBoostingRegressor, self).__init__(n_estimators=n_estimators, \n learning_rate=learning_rate, \n min_samples_split=min_samples_split, \n min_impurity=min_impurity,\n max_depth=max_depth,\n is_regression=is_regression,\n line_search=line_search)\n\n\nclass GradientBoostingClassifier(GradientBoostingDecisionTree):\n \n def __init__(self, n_estimators=200, learning_rate=1, min_samples_split=2,\n min_impurity=1e-7, max_depth=float(\"inf\"), is_regression=False, line_search=False):\n super(GradientBoostingClassifier, self).__init__(n_estimators=n_estimators, \n learning_rate=learning_rate, \n min_samples_split=min_samples_split, \n min_impurity=min_impurity,\n max_depth=max_depth,\n is_regression=is_regression,\n line_search=line_search)\n\n \n#####----XGBoost----#######\nclass XGBoostRegressionTree(DecisionTree):\n \"\"\"\n XGBoost 回归树。此处基于第五章介绍的决策树，故采用贪心算法找到特征上分裂点 (枚举特征上所有可能的分裂点)。\n \"\"\"\n def __init__(self, min_samples_split=2, min_impurity=1e-7,\n max_depth=float(\"inf\"), loss=None, gamma=0., lambd=0.):\n super(XGBoostRegressionTree, self).__init__(min_impurity=min_impurity, \n min_samples_split=min_samples_split, \n max_depth=max_depth)\n self.gamma = gamma # 叶子节点的数目的惩罚系数\n self.lambd = lambd # 叶子节点的权重的惩罚系数\n self.loss = loss # 损失函数\n \n def _split(self, y):\n # y 包含 y_true 在左半列，y_pred 在右半列\n col = int(np.shape(y)[1]/2)\n y, y_pred = y[:, :col], y[:, col:]\n return y, y_pred\n\n def _gain(self, y, y_pred):\n # 计算信息\n nominator = np.power((y * self.loss.grad(y, y_pred)).sum(), 2)\n denominator = self.loss.hess(y, y_pred).sum()\n return nominator / (denominator + self.lambd)\n\n def _gain_by_taylor(self, y, y1, y2):\n # 分割为左子树和右子树\n y, y_pred = self._split(y)\n y1, y1_pred = self._split(y1)\n y2, y2_pred = self._split(y2)\n true_gain = self._gain(y1, y1_pred)\n false_gain = self._gain(y2, y2_pred)\n gain = self._gain(y, y_pred)\n # 计算信息增益\n return 0.5 * (true_gain + false_gain - gain) - self.gamma\n\n def _approximate_update(self, y):\n y, y_pred = self._split(y)\n # 计算叶节点权重\n gradient = self.loss.grad(y, y_pred).sum()\n hessian = self.loss.hess(y, y_pred).sum()\n leaf_approximation = -gradient / (hessian + self.lambd)\n return leaf_approximation\n\n def fit(self, X, y):\n self._impurity_calculation = self._gain_by_taylor\n self._leaf_value_calculation = self._approximate_update\n super(XGBoostRegressionTree, self).fit(X, y)\n\n\nclass XGBoost(object):\n \"\"\"\n XGBoost学习器。\n \"\"\"\n def __init__(self, n_estimators=200, learning_rate=0.001, min_samples_split=2,\n min_impurity=1e-7, max_depth=2, is_regression=False, gamma=0., lambd=0.):\n self.n_estimators = n_estimators # 树的数目\n self.learning_rate = learning_rate # 训练过程中沿着负梯度走的步长，也就是学习率\n self.min_samples_split = min_samples_split # 分割所需的最小样本数\n self.min_impurity = min_impurity # 分割所需的最小纯度\n self.max_depth = max_depth # 树的最大深度\n self.gamma = gamma # 叶子节点的数目的惩罚系数\n self.lambd = lambd # 叶子节点的权重的惩罚系数\n self.is_regression = is_regression # 分类或回归问题\n self.progressbar = progressbar.ProgressBar(widgets=bar_widgets)\n # 回归问题采用基础的平方损失，分类问题采用交叉熵损失\n self.loss = SquareLoss()\n if not self.is_regression:\n self.loss = CrossEntropyLoss()\n\n def fit(self, X, Y):\n # 分类问题将 Y 转化为 one-hot 编码\n if not self.is_regression:\n Y = to_categorical(Y.flatten())\n else:\n Y = Y.reshape(-1, 1) if len(Y.shape) == 1 else Y\n self.out_dims = Y.shape[1]\n self.trees = np.empty((self.n_estimators, self.out_dims), dtype=object)\n Y_pred = np.zeros(np.shape(Y))\n self.weights = np.ones((self.n_estimators, self.out_dims))\n self.weights[1:, :] *= self.learning_rate\n # 迭代过程\n for i in self.progressbar(range(self.n_estimators)):\n for c in range(self.out_dims):\n tree = XGBoostRegressionTree(\n min_samples_split=self.min_samples_split,\n min_impurity=self.min_impurity,\n max_depth=self.max_depth,\n loss=self.loss,\n gamma=self.gamma,\n lambd=self.lambd)\n # 计算损失的梯度，并用梯度进行训练\n if not self.is_regression: \n Y_hat = softmax(Y_pred)\n y, y_pred = Y[:, c], Y_hat[:, c]\n else:\n y, y_pred = Y[:, c], Y_pred[:, c]\n\n y, y_pred = y.reshape(-1, 1), y_pred.reshape(-1, 1)\n y_and_ypred = np.concatenate((y, y_pred), axis=1)\n tree.fit(X, y_and_ypred)\n # 用新的基学习器进行预测\n h_pred = tree.predict(X)\n # 加法模型中添加基学习器的预测，得到最新迭代下的加法模型预测\n Y_pred[:, c] += np.multiply(self.weights[i, c], h_pred)\n self.trees[i, c] = tree\n\n def predict(self, X):\n Y_pred = np.zeros((X.shape[0], self.out_dims))\n # 生成预测\n for c in range(self.out_dims):\n y_pred = np.array([])\n for i in range(self.n_estimators):\n update = np.multiply(self.weights[i, c], self.trees[i, c].predict(X))\n y_pred = update if not y_pred.any() else y_pred + update\n Y_pred[:, c] = y_pred\n if not self.is_regression: \n # 分类问题输出最可能类别\n Y_pred = Y_pred.argmax(axis=1)\n return Y_pred\n \n def score(self, X, y):\n y_pred = self.predict(X)\n accuracy = np.sum(y == y_pred, axis=0) / len(y)\n return accuracy\n \n \nclass XGBRegressor(XGBoost):\n \n def __init__(self, n_estimators=200, learning_rate=1, min_samples_split=2,\n min_impurity=1e-7, max_depth=float(\"inf\"), is_regression=True,\n gamma=0., lambd=0.):\n super(XGBRegressor, self).__init__(n_estimators=n_estimators, \n learning_rate=learning_rate, \n min_samples_split=min_samples_split, \n min_impurity=min_impurity,\n max_depth=max_depth,\n is_regression=is_regression,\n gamma=gamma,\n lambd=lambd)\n\n\nclass XGBClassifier(XGBoost):\n \n def __init__(self, n_estimators=200, learning_rate=1, min_samples_split=2,\n min_impurity=1e-7, max_depth=float(\"inf\"), is_regression=False,\n gamma=0., lambd=0.):\n super(XGBClassifier, self).__init__(n_estimators=n_estimators, \n learning_rate=learning_rate, \n min_samples_split=min_samples_split, \n min_impurity=min_impurity,\n max_depth=max_depth,\n is_regression=is_regression,\n gamma=gamma,\n lambd=lambd) \n"
},
{
"path": "code/chapter8.py",
"content": "from chapter import LayerBase\nimport numpy as np\n\n######### 优化方法(Optimizer)见 method/optimizer #######\n\n\n######## 参数初始化(Parameter Initialization) 见method/weight #####\n\n\n######## BatchNorm1D #####\nclass BatchNorm1D(LayerBase):\n\n def __init__(self, momentum=0.9, epsilon=1e-5, optimizer=None):\n \"\"\"\n 参数说明：\n momentum：动量项，越趋于 1 表示对当前 Batch 的依赖程度越小，running_mean和running_var的计算越平滑\n float型 (default: 0.9)\n\n epsilon：避免除数为0，float型 (default : 1e-5)\n optimizer：优化器\n \"\"\"\n super().__init__(optimizer)\n\n self.n_in = None\n self.n_out = None\n self.epsilon = epsilon\n self.momentum = momentum\n self.params = {\n \"scaler\": None,\n \"intercept\": None,\n \"running_var\": None,\n \"running_mean\": None,\n }\n self.is_initialized = False\n\n def _init_params(self):\n scaler = np.random.rand(self.n_in)\n intercept = np.zeros(self.n_in)\n running_mean = np.zeros(self.n_in)\n running_var = np.ones(self.n_in)\n \n self.params = {\n \"scaler\": scaler,\n \"intercept\": intercept,\n \"running_mean\": running_mean,\n \"running_var\": running_var,\n }\n self.gradients = {\n \"scaler\": np.zeros_like(scaler),\n \"intercept\": np.zeros_like(intercept),\n }\n self.is_initialized = True\n\n def reset_running_stats(self):\n self.params[\"running_mean\"] = np.zeros(self.n_in)\n self.params[\"running_var\"] = np.ones(self.n_in)\n\n def forward(self, X, is_train=True, retain_derived=True):\n \"\"\"\n Batch 训练时 BN 的前向传播，原理见上文。\n\n [train]: Y = scaler * norm(X) + intercept，其中 norm(X) = (X - mean(X)) / sqrt(var(X) + epsilon)\n\n [test]: Y = scaler * running_norm(X) + intercept，\n 其中 running_norm(X) = (X - running_mean) / sqrt(running_var + epsilon)\n \n 参数说明：\n X：输入数组，为（n_samples, n_in），float型\n is_train：是否为训练阶段，bool型\n retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型\n \"\"\"\n if not self.is_initialized:\n self.n_in = self.n_out = X.shape[1]\n self._init_params()\n\n epsi, momentum = self.hyperparams[\"epsilon\"], self.hyperparams[\"momentum\"]\n rm, rv = self.params[\"running_mean\"], self.params[\"running_var\"]\n\n scaler, intercept = self.params[\"scaler\"], self.params[\"intercept\"]\n X_mean, X_var = self.params[\"running_mean\"], self.params[\"running_var\"]\n\n if is_train and retain_derived:\n X_mean, X_var = X.mean(axis=0), X.var(axis=0) \n self.params[\"running_mean\"] = momentum * rm + (1.0 - momentum) * X_mean\n self.params[\"running_var\"] = momentum * rv + (1.0 - momentum) * X_var\n\n if retain_derived:\n self.X.append(X)\n\n X_hat = (X - X_mean) / np.sqrt(X_var + epsi)\n y = scaler * X_hat + intercept\n return y\n\n def backward(self, dLda, retain_grads=True):\n \"\"\"\n BN 的反向传播，原理见上文。\n \n 参数说明：\n dLda：关于损失的梯度，为（n_samples, n_out），float型\n retain_grads：是否计算中间变量的参数梯度，bool型\n \"\"\"\n if not isinstance(dLda, list):\n dLda = [dLda]\n\n dX = []\n X = self.X\n for da, x in zip(dLda, X):\n dx, dScaler, dIntercept = self._bwd(da, x)\n dX.append(dx)\n\n if retain_grads:\n self.gradients[\"scaler\"] += dScaler\n self.gradients[\"intercept\"] += dIntercept\n\n return dX[0] if len(X) == 1 else dX\n\n def _bwd(self, dLda, X):\n scaler = self.params[\"scaler\"]\n epsi = self.hyperparams[\"epsilon\"]\n\n n_ex, n_in = X.shape\n X_mean, X_var = X.mean(axis=0), X.var(axis=0)\n X_hat = (X - X_mean) / np.sqrt(X_var + epsi)\n \n dIntercept = dLda.sum(axis=0)\n dScaler = np.sum(dLda * X_hat, axis=0)\n dX_hat = dLda * scaler\n \n dX = (n_ex * dX_hat - dX_hat.sum(axis=0) - X_hat * (dX_hat * X_hat).sum(axis=0)) / (\n n_ex * np.sqrt(X_var + epsi)\n )\n\n return dX, dScaler, dIntercept\n \n @property\n def hyperparams(self):\n return {\n \"layer\": \"BatchNorm1D\",\n \"acti_fn\": None,\n \"n_in\": self.n_in,\n \"n_out\": self.n_out,\n \"epsilon\": self.epsilon,\n \"momentum\": self.momentum,\n \"optimizer\": {\n \"cache\": self.optimizer.cache,\n \"hyperparams\": self.optimizer.hyperparams,\n },\n }\n"
},
{
"path": "code/chapter9.py",
"content": "from abc import ABC, abstractmethod\nimport numpy as np\nfrom chapter6 import LayerBase, CrossEntropy, FullyConnected, minibatch, softmax\nfrom collections import OrderedDict\n\n\n########## Padding ################\ndef calc_pad_dims_sameconv_2D(X_shape, out_dim, kernel_shape, stride, dilation=1):\n \"\"\"\n 当填充方式为相同卷积时，计算 padding 的数目，保证输入输出的大小相同。这里在卷积过程中考虑填充(Padding)，\n 卷积步幅(Stride)，扩张率(Dilation rate)。根据扩张卷积的输出公式可以得到 padding 的数目。\n \n 参数说明：\n X_shape：输入数组，为 (n_samples, in_rows, in_cols, in_ch)\n out_dim：输出数组维数，为 (out_rows, out_cols)\n kernel_shape：卷积核形状，为 (fr, fc)\n stride：卷积步幅，int 型\n dilation：扩张率，int 型，default=1\n \"\"\"\n d = dilation\n fr, fc = kernel_shape\n out_rows, out_cols = out_dim\n n_ex, in_rows, in_cols, in_ch = X_shape\n\n # 考虑扩张率\n _fr, _fc = fr + (fr-1) * (d-1), fc + (fc-1) * (d-1)\n\n # 计算 padding 维数\n pr = int((stride * (out_rows-1) + _fr - in_rows) / 2)\n pc = int((stride * (out_cols-1) + _fc - in_cols) / 2)\n\n # 校验，如不等 (right/bottom处) 添加不对称0填充\n out_rows1 = int(1 + (in_rows + 2 * pr - _fr) / stride)\n out_cols1 = int(1 + (in_cols + 2 * pc - _fc) / stride)\n \n pr1, pr2 = pr, pr\n if out_rows1 == out_rows - 1:\n pr1, pr2 = pr, pr + 1\n elif out_rows1 != out_rows:\n raise AssertionError\n\n pc1, pc2 = pc, pc\n if out_cols1 == out_cols - 1:\n pc1, pc2 = pc, pc + 1\n elif out_cols1 != out_cols:\n raise AssertionError\n \n # 返回对 X 的 Padding 维数 (left, right, up, down)\n return (pr1, pr2, pc1, pc2)\n\n\ndef pad2D(X, pad, kernel_shape=None, stride=None, dilation=1):\n \"\"\"\n 二维填充\n \n 参数说明：\n X：输入数组，为 (n_samples, in_rows, in_cols, in_ch)，\n 其中 padding 操作是应用到 in_rows 和 in_cols\n pad：padding 数目，4-tuple, int, 或 'same'，'valid'\n 在图片的左、右、上、下 (left, right, up, down) 0填充\n 若为int，表示在左、右、上、下均填充数目为 pad 的 0，\n 若为same，表示填充后为相同 (same) 卷积，\n 若为valid，表示填充后为有效 (valid) 卷积\n kernel_shape：卷积核形状，为 (fr, fc)\n stride：卷积步幅，int 型\n dilation：扩张率，int 型，default=1\n \"\"\"\n p = pad\n if isinstance(p, int):\n p = (p, p, p, p)\n\n if isinstance(p, tuple):\n X_pad = np.pad(\n X,\n pad_width=((0, 0), (p[0], p[1]), (p[2], p[3]), (0, 0)),\n mode=\"constant\",\n constant_values=0,\n )\n\n # 'same'卷积，首先计算 padding 维数\n if p == \"same\" and kernel_shape and stride is not None:\n p = calc_pad_dims_sameconv_2D(\n X.shape, X.shape[1:3], kernel_shape, stride, dilation=dilation\n )\n X_pad, p = pad2D(X, p)\n \n if p == \"valid\":\n p = (0, 0, 0, 0)\n X_pad, p = pad2D(X, p)\n \n return X_pad, p\n\n\n####### conv2D ##################\ndef conv2D(X, W, stride, pad, dilation=1):\n \"\"\"\n 二维卷积实现过程。\n\n 参数说明：\n X：输入数组，为 (n_samples, in_rows, in_cols, in_ch)\n W：卷积层的卷积核参数，为 (kernel_rows, kernel_cols, in_ch, out_ch)\n stride：卷积核的卷积步幅，int型\n pad：padding 数目，4-tuple, int, 或 'same'，'valid'型\n 在图片的左、右、上、下 (left, right, up, down) 0填充\n 若为int，表示在左、右、上、下均填充数目为 pad 的 0，\n 若为same，表示填充后为相同 (same) 卷积，\n 若为valid，表示填充后为有效 (valid) 卷积\n dilation：扩张率，int 型，default=1\n\n 输出说明：\n Z：卷积结果，为 (n_samples, out_rows, out_cols, out_ch)\n \"\"\"\n s, d = stride, dilation\n X_pad, p = pad2D(X, pad, W.shape[:2], stride=s, dilation=d)\n\n pr1, pr2, pc1, pc2 = p\n fr, fc, in_ch, out_ch = W.shape\n n_samp, in_rows, in_cols, in_ch = X.shape\n\n # 考虑扩张率\n _fr, _fc = fr + (fr-1) * (d-1), fc + (fc-1) * (d-1)\n\n out_rows = int((in_rows + pr1 + pr2 - _fr) / s + 1)\n out_cols = int((in_cols + pc1 + pc2 - _fc) / s + 1)\n\n Z = np.zeros((n_samp, out_rows, out_cols, out_ch))\n for m in range(n_samp):\n for c in range(out_ch):\n for i in range(out_rows):\n for j in range(out_cols):\n i0, i1 = i * s, (i * s) + fr + (fr-1) * (d-1)\n j0, j1 = j * s, (j * s) + fc + (fc-1) * (d-1)\n\n window = X_pad[m, i0 : i1 : d, j0 : j1 : d, :]\n Z[m, i, j, c] = np.sum(window * W[:, :, :, c])\n return Z\n\n\n####### conv2D GEMM ############\n\"\"\"\nconv2D 的 GEMM 实现过程，将 X 和 W 转化为 2D 矩阵，\n这里我们将 X 转化为 (kernel_rows*kernel_cols*in_ch, n_samples*out_rows*out_cols)\nW 转化为 (out_ch, kernel_rows*kernel_cols*in_ch)\n\"\"\"\ndef _im2col_indices(X_shape, fr, fc, p, s, d=1):\n \"\"\"\n 生成输入矩阵的 (c,h_in,w_in) 三个维度的索引\n \n 输出说明：\n i：输入矩阵的i值，(kernel_rows*kernel_cols*in_ch, out_rows*out_cols)，图示中第二维坐标\n j：输入矩阵的j值，(kernel_rows*kernel_cols*in_ch, out_rows*out_cols)，图示中第三维坐标\n k：输入矩阵的c值，(kernel_rows*kernel_cols*in_ch, 1)，图示中第一维坐标\n \"\"\"\n pr1, pr2, pc1, pc2 = p\n n_ex, n_in, in_rows, in_cols = X_shape\n\n # 考虑扩张率\n _fr, _fc = fr + (fr-1) * (d-1), fc + (fc-1) * (d-1)\n\n out_rows = int((in_rows + pr1 + pr2 - _fr) / s + 1)\n out_cols = int((in_cols + pc1 + pc2 - _fc) / s + 1)\n\n # i0/i1/j0/j1：用于得到i，j，k。i0/j0过程见图示，i1/j1由滑动过程得出\n i0 = np.repeat(np.arange(fr), fc)\n i0 = np.tile(i0, n_in) * d\n i1 = s * np.repeat(np.arange(out_rows), out_cols)\n j0 = np.tile(np.arange(fc), fr * n_in) * d\n j1 = s * np.tile(np.arange(out_cols), out_rows)\n\n i = i0.reshape(-1, 1) + i1.reshape(1, -1)\n j = j0.reshape(-1, 1) + j1.reshape(1, -1)\n k = np.repeat(np.arange(n_in), fr * fc).reshape(-1, 1)\n return k, i, j\n\n\n\ndef im2col(X, W_shape, pad, stride, dilation=1):\n \"\"\"\n im2col 实现\n\n 参数说明：\n X：输入数组，为 (n_samples, in_rows, in_cols, in_ch)，此时还未 0 填充(padding)\n W_shape：卷积层的卷积核的形状，为 (kernel_rows, kernel_cols, in_ch, out_ch)\n pad：padding 数目，4-tuple, int, 或 'same'，'valid'型\n 在图片的左、右、上、下 (left, right, up, down) 0填充\n 若为int，表示在左、右、上、下均填充数目为 pad 的 0，\n 若为same，表示填充后为相同 (same) 卷积，\n 若为valid，表示填充后为有效 (valid) 卷积\n stride：卷积核的卷积步幅，int型\n dilation：扩张率，int 型，default=1\n\n 输出说明：\n X_col：输出结果，形状为 (kernel_rows*kernel_cols*n_in, n_samples*out_rows*out_cols)\n p：填充数，4-tuple\n \"\"\"\n fr, fc, n_in, n_out = W_shape\n s, p, d = stride, pad, dilation\n n_samp, in_rows, in_cols, n_in = X.shape\n\n X_pad, p = pad2D(X, p, W_shape[:2], stride=s, dilation=d)\n pr1, pr2, pc1, pc2 = p\n\n # 将输入的通道维数移至第二位\n X_pad = X_pad.transpose(0, 3, 1, 2)\n\n k, i, j = _im2col_indices((n_samp, n_in, in_rows, in_cols), fr, fc, p, s, d)\n\n # X_col.shape = (n_samples, kernel_rows*kernel_cols*n_in, out_rows*out_cols)\n X_col = X_pad[:, k, i, j]\n X_col = X_col.transpose(1, 2, 0).reshape(fr * fc * n_in, -1)\n return X_col, p\n\n\ndef conv2D_gemm(X, W, stride=0, pad='same', dilation=1):\n \"\"\"\n 二维卷积实现过程，依靠“im2col”函数将卷积作为单个矩阵乘法执行。\n\n 参数说明：\n X：输入数组，为 (n_samples, in_rows, in_cols, in_ch)\n W：卷积层的卷积核参数，为 (kernel_rows, kernel_cols, in_ch, out_ch)\n stride：卷积核的卷积步幅，int型\n pad：padding 数目，4-tuple, int, 或 'same'，'valid'型\n 在图片的左、右、上、下 (left, right, up, down) 0填充\n 若为int，表示在左、右、上、下均填充数目为 pad 的 0，\n 若为same，表示填充后为相同 (same) 卷积，\n 若为valid，表示填充后为有效 (valid) 卷积\n dilation：扩张率，int 型，default=1\n\n 输出说明：\n Z：卷积结果，为 (n_samples, out_rows, out_cols, out_ch)\n \"\"\"\n s, d = stride, dilation\n _, p = pad2D(X, pad, W.shape[:2], s, dilation=dilation)\n\n pr1, pr2, pc1, pc2 = p\n fr, fc, in_ch, out_ch = W.shape\n n_samp, in_rows, in_cols, in_ch = X.shape\n \n # 考虑扩张率\n _fr, _fc = fr + (fr-1) * (d-1), fc + (fc-1) * (d-1)\n\n # 输出维数，根据上面公式可得\n out_rows = int((in_rows + pr1 + pr2 - _fr) / s + 1)\n out_cols = int((in_cols + pc1 + pc2 - _fc) / s + 1)\n\n # 将 X 和 W 转化为 2D 矩阵并乘积\n X_col, _ = im2col(X, W.shape, p, s, d)\n W_col = W.transpose(3, 2, 0, 1).reshape(out_ch, -1)\n\n Z = (W_col @ X_col).reshape(out_ch, out_rows, out_cols, n_samp).transpose(3, 1, 2, 0)\n\n return Z\n\n\n########### Conv2D ##################\nclass Conv2D(LayerBase):\n \n def __init__(\n self,\n out_ch,\n kernel_shape,\n pad=0,\n stride=1,\n dilation=1,\n acti_fn=None,\n optimizer=None,\n init_w=\"glorot_uniform\",\n ):\n \"\"\"\n 二维卷积\n\n 参数说明：\n out_ch：卷积核组的数目，int 型\n kernel_shape：单个卷积核形状，2-tuple\n acti_fn：激活函数，str 型\n pad：padding 数目，4-tuple, int, 或 'same'，'valid'型\n 在图片的左、右、上、下 (left, right, up, down) 0填充\n 若为int，表示在左、右、上、下均填充数目为 pad 的 0，\n 若为same，表示填充后为相同 (same) 卷积，\n 若为valid，表示填充后为有效 (valid) 卷积\n stride：卷积核的卷积步幅，int型\n dilation：扩张率，int 型，default=1\n init_w：权重初始化方法，str型\n optimizer：优化方法，str型\n \"\"\"\n super().__init__(optimizer)\n\n self.pad = pad\n self.in_ch = None\n self.out_ch = out_ch\n self.stride = stride\n self.dilation = dilation\n self.kernel_shape = kernel_shape\n self.init_w = init_w\n self.init_weights = WeightInitializer(mode=init_w)\n self.acti_fn = ActivationInitializer(acti_fn)()\n self.parameters = {\"W\": None, \"b\": None}\n self.is_initialized = False\n\n def _init_params(self):\n fr, fc = self.kernel_shape\n W = self.init_weights((fr, fc, self.in_ch, self.out_ch))\n b = np.zeros((1, 1, 1, self.out_ch))\n\n self.params = {\"W\": W, \"b\": b}\n self.gradients = {\"W\": np.zeros_like(W), \"b\": np.zeros_like(b)}\n self.derived_variables = {\"Y\": []}\n self.is_initialized = True\n\n def forward(self, X, retain_derived=True):\n \"\"\"\n 卷积层的前向传播，原理见上文。\n\n 参数说明：\n X：输入数组，形状为 (n_samples, in_rows, in_cols, in_ch)\n retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型\n\n 输出说明：\n a：卷积层输出，形状为 (n_samples, out_rows, out_cols, out_ch)\n \"\"\"\n if not self.is_initialized:\n self.in_ch = X.shape[3]\n self._init_params()\n\n W = self.params[\"W\"]\n b = self.params[\"b\"]\n\n n_samp, in_rows, in_cols, in_ch = X.shape\n s, p, d = self.stride, self.pad, self.dilation\n\n # 卷积操作\n Y = conv2D(X, W, s, p, d) + b\n a = self.acti_fn(Y)\n\n if retain_derived:\n self.X.append(X)\n self.derived_variables[\"Y\"].append(Y)\n\n return a\n\n def backward(self, dLda, retain_grads=True):\n \"\"\"\n 卷积层的反向传播，原理见上文。\n\n 参数说明：\n dLda：关于损失的梯度，为 (n_samples, out_rows, out_cols, out_ch) \n retain_grads：是否计算中间变量的参数梯度，bool型\n\n 输出说明：\n dXs：即dX，当前卷积层对输入关于损失的梯度，为 (n_samples, in_rows, in_cols, in_ch)\n \"\"\"\n if not isinstance(dLda, list):\n dLda = [dLda]\n\n W = self.params[\"W\"]\n b = self.params[\"b\"]\n Ys = self.derived_variables[\"Y\"]\n Xs, d = self.X, self.dilation\n (fr, fc), s, p = self.kernel_shape, self.stride, self.pad\n dXs = []\n \n for X, Y, da in zip(Xs, Ys, dLda):\n n_samp, out_rows, out_cols, out_ch = da.shape\n X_pad, (pr1, pr2, pc1, pc2) = pad2D(X, p, self.kernel_shape, s, d)\n\n dY = da * self.acti_fn.grad(Y)\n\n dX = np.zeros_like(X_pad)\n dW, db = np.zeros_like(W), np.zeros_like(b)\n for m in range(n_samp):\n for i in range(out_rows):\n for j in range(out_cols):\n for c in range(out_ch):\n i0, i1 = i * s, (i * s) + fr + (fr-1) * (d-1)\n j0, j1 = j * s, (j * s) + fc + (fc-1) * (d-1)\n\n wc = W[:, :, :, c]\n kernel = dY[m, i, j, c]\n window = X_pad[m, i0:i1:d, j0:j1:d, :]\n\n db[:, :, :, c] += kernel\n dW[:, :, :, c] += window * kernel\n dX[m, i0:i1:d, j0:j1:d, :] += (\n wc * kernel\n )\n\n if retain_grads:\n self.gradients[\"W\"] += dW\n self.gradients[\"b\"] += db\n\n pr2 = None if pr2 == 0 else -pr2\n pc2 = None if pc2 == 0 else -pc2\n dXs.append(dX[:, pr1:pr2, pc1:pc2, :])\n \n return dXs[0] if len(Xs) == 1 else dXs\n \n @property\n def hyperparams(self):\n return {\n \"layer\": \"Conv2D\",\n \"pad\": self.pad,\n \"init_w\": self.init_w,\n \"in_ch\": self.in_ch,\n \"out_ch\": self.out_ch,\n \"stride\": self.stride,\n \"dilation\": self.dilation,\n \"acti_fn\": str(self.acti_fn),\n \"kernel_shape\": self.kernel_shape,\n \"optimizer\": {\n \"cache\": self.optimizer.cache,\n \"hyperparams\": self.optimizer.hyperparams,\n },\n }\n\n\n######### Conv2D GEMM #############\ndef col2im(X_col, X_shape, W_shape, pad, stride, dilation=0):\n \"\"\"\n col2im 实现，“col2im”函数将 2D 矩阵变为 4D 图像\n\n 参数说明：\n X_col：X 经过 im2col 后 (列) 的矩阵，形状为 (Q, Z)，具体形状见上文\n X_shape：原始的输入数组形状，为 (n_samples, in_rows, in_cols, in_ch)，\n 此时还未 0 填充(padding)\n W_shape：卷积核组形状，4-tuple 为 (kernel_rows, kernel_cols, in_ch, out_ch)\n pad：padding 数目，4-tuple\n 在图片的左、右、上、下 (left, right, up, down) 0填充\n stride：卷积核的卷积步幅，int型\n dilation：扩张率，int 型，default=1\n\n 输出说明：\n img：输出结果，形状为 (n_samples, in_rows, in_cols, in_ch)\n \"\"\"\n s, d = stride, dilation\n pr1, pr2, pc1, pc2 = pad\n fr, fc, n_in, n_out = W_shape\n n_samp, in_rows, in_cols, n_in = X_shape\n\n X_pad = np.zeros((n_samp, n_in, in_rows + pr1 + pr2, in_cols + pc1 + pc2))\n k, i, j = _im2col_indices((n_samp, n_in, in_rows, in_cols), fr, fc, pad, s, d)\n\n X_col_reshaped = X_col.reshape(n_in * fr * fc, -1, n_samp)\n X_col_reshaped = X_col_reshaped.transpose(2, 0, 1)\n\n np.add.at(X_pad, (slice(None), k, i, j), X_col_reshaped)\n\n pr2 = None if pr2 == 0 else -pr2\n pc2 = None if pc2 == 0 else -pc2\n return X_pad[:, :, pr1:pr2, pc1:pc2]\n\n\nclass Conv2D_gemm(LayerBase):\n \n def __init__(\n self,\n out_ch,\n kernel_shape,\n pad=0,\n stride=1,\n dilation=1,\n acti_fn=None,\n optimizer=None,\n init_w=\"glorot_uniform\",\n ):\n \"\"\"\n 二维卷积\n\n 参数说明：\n out_ch：卷积核组的数目，int 型\n kernel_shape：单个卷积核形状，2-tuple\n acti_fn：激活函数，str 型\n pad：padding 数目，4-tuple, int, 或 'same'，'valid'型\n 在图片的左、右、上、下 (left, right, up, down) 0填充\n 若为int，表示在左、右、上、下均填充数目为 pad 的 0，\n 若为same，表示填充后为相同 (same) 卷积，\n 若为valid，表示填充后为有效 (valid) 卷积\n stride：卷积核的卷积步幅，int型\n dilation：扩张率，int 型，default=1\n init_w：权重初始化方法，str型\n optimizer：优化方法，str型\n \"\"\"\n super().__init__(optimizer)\n\n self.pad = pad\n self.in_ch = None\n self.out_ch = out_ch\n self.stride = stride\n self.dilation = dilation\n self.kernel_shape = kernel_shape\n self.init_w = init_w\n self.init_weights = WeightInitializer(mode=init_w)\n self.acti_fn = ActivationInitializer(acti_fn)()\n self.parameters = {\"W\": None, \"b\": None}\n self.is_initialized = False\n\n def _init_params(self):\n fr, fc = self.kernel_shape\n W = self.init_weights((fr, fc, self.in_ch, self.out_ch))\n b = np.zeros((1, 1, 1, self.out_ch))\n\n self.params = {\"W\": W, \"b\": b}\n self.gradients = {\"W\": np.zeros_like(W), \"b\": np.zeros_like(b)}\n self.derived_variables = {\"Y\": []}\n self.is_initialized = True\n\n def forward(self, X, retain_derived=True):\n \"\"\"\n 卷积层的前向传播，原理见上文。\n\n 参数说明：\n X：输入数组，形状为 (n_samples, in_rows, in_cols, in_ch)\n retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型\n\n 输出说明：\n a：卷积层输出，形状为 (n_samples, out_rows, out_cols, out_ch)\n \"\"\"\n if not self.is_initialized:\n self.in_ch = X.shape[3]\n self._init_params()\n\n W = self.params[\"W\"]\n b = self.params[\"b\"]\n\n n_samp, in_rows, in_cols, in_ch = X.shape\n s, p, d = self.stride, self.pad, self.dilation\n\n # 卷积操作\n Y = conv2D_gemm(X, W, s, p, d) + b\n a = self.acti_fn(Y)\n\n if retain_derived:\n self.X.append(X)\n self.derived_variables[\"Y\"].append(Y)\n\n return a\n\n def backward(self, dLda, retain_grads=True):\n \"\"\"\n 卷积层的反向传播，原理见上文。\n\n 参数说明：\n dLda：关于损失的梯度，为 (n_samples, out_rows, out_cols, out_ch) \n retain_grads：是否计算中间变量的参数梯度，bool型\n\n 输出说明：\n dX：当前卷积层对输入关于损失的梯度，为 (n_samples, in_rows, in_cols, in_ch)\n \"\"\"\n if not isinstance(dLda, list):\n dLda = [dLda]\n\n dX = []\n X = self.X\n Y = self.derived_variables[\"Y\"]\n\n for da, x, y in zip(dLda, X, Y):\n dx, dw, db = self._bwd(da, x, y)\n dX.append(dx)\n\n if retain_grads:\n self.gradients[\"W\"] += dw\n self.gradients[\"b\"] += db\n\n return dX[0] if len(X) == 1 else dX\n\n def _bwd(self, dLda, X, Y):\n W = self.params[\"W\"]\n d = self.dilation\n fr, fc, in_ch, out_ch = W.shape\n n_samp, out_rows, out_cols, out_ch = dLda.shape\n (fr, fc), s, p = self.kernel_shape, self.stride, self.pad\n \n dLdy = dLda * self.acti_fn.grad(Y)\n dLdy_col = dLdy.transpose(3, 1, 2, 0).reshape(out_ch, -1)\n W_col = W.transpose(3, 2, 0, 1).reshape(out_ch, -1).T\n X_col, p = im2col(X, W.shape, p, s, d)\n\n dW = (dLdy_col @ X_col.T).reshape(out_ch, in_ch, fr, fc).transpose(2, 3, 1, 0)\n db = dLdy_col.sum(axis=1).reshape(1, 1, 1, -1)\n\n dX_col = W_col @ dLdy_col\n dX = col2im(dX_col, X.shape, W.shape, p, s, d).transpose(0, 2, 3, 1)\n\n return dX, dW, db\n \n @property\n def hyperparams(self):\n return {\n \"layer\": \"Conv2D\",\n \"pad\": self.pad,\n \"init_w\": self.init_w,\n \"in_ch\": self.in_ch,\n \"out_ch\": self.out_ch,\n \"stride\": self.stride,\n \"dilation\": self.dilation,\n \"acti_fn\": str(self.acti_fn),\n \"kernel_shape\": self.kernel_shape,\n \"optimizer\": {\n \"cache\": self.optimizer.cache,\n \"hyperparams\": self.optimizer.hyperparams,\n },\n }\n\n\n######## Pool2D ################\nclass Pool2D(LayerBase):\n \n def __init__(self, kernel_shape, stride=1, pad=0, mode=\"max\", optimizer=None):\n \"\"\"\n 二维池化\n\n 参数说明：\n kernel_shape：池化窗口的大小，2-tuple\n stride：和卷积类似，窗口在每一个维度上滑动的步长，int型\n pad：padding 数目，4-tuple, int, 或 str('same','valid')型 (default: 0)\n 和卷积类似\n mode：池化函数，str型 (default: 'max')，可选{\"max\",\"average\"}\n optimizer：优化方法，str型\n \"\"\"\n super().__init__(optimizer)\n\n self.pad = pad\n self.mode = mode\n self.in_ch = None\n self.out_ch = None\n self.stride = stride\n self.kernel_shape = kernel_shape\n self.is_initialized = False\n\n def _init_params(self):\n self.derived_variables = {\"out_rows\": [], \"out_cols\": []}\n self.is_initialized = True\n\n def forward(self, X, retain_derived=True):\n \"\"\"\n 池化层前向传播\n\n 参数说明：\n X：输入数组，形状为 (n_samp, in_rows, in_cols, in_ch)\n retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型\n \n 输出说明：\n Y：输出结果，形状为 (n_samp, out_rows, out_cols, out_ch)\n \"\"\"\n if not self.is_initialized:\n self.in_ch = self.out_ch = X.shape[3]\n self._init_params()\n\n n_samp, in_rows, in_cols, nc_in = X.shape\n (fr, fc), s, p = self.kernel_shape, self.stride, self.pad\n X_pad, (pr1, pr2, pc1, pc2) = pad2D(X, p, self.kernel_shape, s)\n\n out_rows = int((in_rows + pr1 + pr2 - fr) / s + 1)\n out_cols = int((in_cols + pc1 + pc2 - fc) / s + 1)\n\n if self.mode == \"max\":\n pool_fn = np.max\n elif self.mode == \"average\":\n pool_fn = np.mean\n\n Y = np.zeros((n_samp, out_rows, out_cols, self.out_ch))\n for m in range(n_samp):\n for i in range(out_rows):\n for j in range(out_cols):\n for c in range(self.out_ch):\n i0, i1 = i * s, (i * s) + fr\n j0, j1 = j * s, (j * s) + fc\n\n xi = X_pad[m, i0:i1, j0:j1, c]\n Y[m, i, j, c] = pool_fn(xi)\n\n if retain_derived:\n self.X.append(X)\n self.derived_variables[\"out_rows\"].append(out_rows)\n self.derived_variables[\"out_cols\"].append(out_cols)\n\n return Y\n\n def backward(self, dLdy, retain_grads=True):\n \"\"\"\n 池化层的反向传播，原理见上文。\n\n 参数说明：\n dLdy：关于损失的梯度，为 (n_samples, out_rows, out_cols, out_ch) \n retain_grads：是否计算中间变量的参数梯度，bool型\n\n 输出说明：\n dXs：即dX，当前卷积层对输入关于损失的梯度，为 (n_samples, in_rows, in_cols, in_ch)\n \"\"\"\n if not isinstance(dLdy, list):\n dLdy = [dLdy]\n\n Xs = self.X\n out_rows = self.derived_variables[\"out_rows\"]\n out_cols = self.derived_variables[\"out_cols\"]\n\n (fr, fc), s, p = self.kernel_shape, self.stride, self.pad\n\n dXs = []\n for X, dy, out_row, out_col in zip(Xs, dLdy, out_rows, out_cols):\n n_samp, in_rows, in_cols, nc_in = X.shape\n X_pad, (pr1, pr2, pc1, pc2) = pad2D(X, p, self.kernel_shape, s)\n\n dX = np.zeros_like(X_pad)\n for m in range(n_samp):\n for i in range(out_row):\n for j in range(out_col):\n for c in range(self.out_ch):\n i0, i1 = i * s, (i * s) + fr\n j0, j1 = j * s, (j * s) + fc\n\n if self.mode == \"max\":\n xi = X[m, i0:i1, j0:j1, c]\n mask = np.zeros_like(xi).astype(bool)\n x, y = np.argwhere(xi == np.max(xi))[0]\n mask[x, y] = True\n dX[m, i0:i1, j0:j1, c] += mask * dy[m, i, j, c]\n \n elif self.mode == \"average\":\n frame = np.ones((fr, fc)) * dy[m, i, j, c]\n dX[m, i0:i1, j0:j1, c] += frame / np.prod((fr, fc))\n\n pr2 = None if pr2 == 0 else -pr2\n pc2 = None if pc2 == 0 else -pc2\n dXs.append(dX[:, pr1:pr2, pc1:pc2, :])\n \n return dXs[0] if len(Xs) == 1 else dXs\n\n @property\n def hyperparams(self):\n return {\n \"layer\": \"Pool2D\",\n \"acti_fn\": None,\n \"pad\": self.pad,\n \"mode\": self.mode,\n \"in_ch\": self.in_ch,\n \"out_ch\": self.out_ch,\n \"stride\": self.stride,\n \"kernel_shape\": self.kernel_shape,\n \"optimizer\": {\n \"cache\": self.optimizer.cache,\n \"hyperparams\": self.optimizer.hyperparams,\n },\n }\n\n\n############### Flatten ##################\nclass Flatten(LayerBase):\n \n def __init__(self, keep_dim=\"first\", optimizer=None):\n \"\"\"\n 将多维输入展开\n\n 参数说明：\n keep_dim：展开形状，str (default : 'first')\n 对于输入 X，keep_dim可选 'first'->将 X 重构为(X.shape[0], -1)，\n 'last'->将 X 重构为(-1, X.shape[0])，'none'->将 X 重构为(1,-1)\n optimizer：优化方法\n \"\"\"\n super().__init__(optimizer)\n\n self.keep_dim = keep_dim\n self._init_params()\n\n def _init_params(self):\n self.X = []\n self.gradients = {}\n self.params = {}\n self.derived_variables = {\"in_dims\": []}\n\n def forward(self, X, retain_derived=True):\n \"\"\"\n 前向传播\n\n 参数说明：\n X：输入数组\n retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型\n \"\"\"\n if retain_derived:\n self.derived_variables[\"in_dims\"].append(X.shape)\n if self.keep_dim == \"none\":\n return X.flatten().reshape(1, -1)\n rs = (X.shape[0], -1) if self.keep_dim == \"first\" else (-1, X.shape[-1])\n return X.reshape(*rs)\n\n def backward(self, dLdy, retain_grads=True):\n \"\"\"\n 反向传播\n\n 参数说明：\n dLdy：关于损失的梯度\n retain_grads：是否计算中间变量的参数梯度，bool型\n\n 输出说明：\n dX：将对输入的梯度进行重构为原始输入的形状\n \"\"\"\n if not isinstance(dLdy, list):\n dLdy = [dLdy]\n in_dims = self.derived_variables[\"in_dims\"]\n dX = [dy.reshape(*dims) for dy, dims in zip(dLdy, in_dims)]\n return dX[0] if len(dLdy) == 1 else dX\n\n @property\n def hyperparams(self):\n return {\n \"layer\": \"Flatten\",\n \"keep_dim\": self.keep_dim,\n \"optimizer\": {\n \"cache\": self.optimizer.cache,\n \"hyperparams\": self.optimizer.hyperparams,\n },\n }\n\n\n########### LeNet ################\nclass LeNet(object):\n \n def __init__(\n self,\n fc3_out=128,\n fc4_out=84,\n fc5_out=10,\n conv1_pad=0,\n conv2_pad=0,\n conv1_out_ch=6,\n conv2_out_ch=16,\n conv1_stride=1,\n pool1_stride=2,\n conv2_stride=1,\n pool2_stride=2,\n conv1_kernel_shape=(5, 5),\n pool1_kernel_shape=(2, 2),\n conv2_kernel_shape=(5, 5),\n pool2_kernel_shape=(2, 2),\n optimizer=\"adam\",\n init_w=\"glorot_normal\",\n loss=CrossEntropy()\n ):\n self.optimizer = optimizer\n self.init_w = init_w\n self.loss = loss\n self.fc3_out = fc3_out\n self.fc4_out = fc4_out\n self.fc5_out = fc5_out\n self.conv1_pad = conv1_pad\n self.conv2_pad = conv2_pad\n self.conv1_stride = conv1_stride\n self.conv1_out_ch = conv1_out_ch\n self.pool1_stride = pool1_stride\n self.conv2_out_ch = conv2_out_ch\n self.conv2_stride = conv2_stride\n self.pool2_stride = pool2_stride\n self.conv2_kernel_shape = conv2_kernel_shape\n self.pool2_kernel_shape = pool2_kernel_shape\n self.conv1_kernel_shape = conv1_kernel_shape\n self.pool1_kernel_shape = pool1_kernel_shape\n \n self.is_initialized = False\n \n def _set_params(self):\n \"\"\"\n 函数作用：模型初始化\n Conv1 -> Pool1 -> Conv2 -> Pool2 -> Flatten -> FC3 -> FC4 -> FC5 -> Softmax\n \"\"\"\n self.layers = OrderedDict()\n self.layers[\"Conv1\"] = Conv2D(\n out_ch=self.conv1_out_ch,\n kernel_shape=self.conv1_kernel_shape,\n pad=self.conv1_pad,\n stride=self.conv1_stride,\n acti_fn=\"sigmoid\",\n optimizer=self.optimizer,\n init_w=self.init_w,\n )\n self.layers[\"Pool1\"] = Pool2D(\n mode=\"max\",\n optimizer=self.optimizer,\n stride=self.pool1_stride,\n kernel_shape=self.pool1_kernel_shape,\n )\n self.layers[\"Conv2\"] = Conv2D(\n out_ch=self.conv1_out_ch,\n kernel_shape=self.conv1_kernel_shape,\n pad=self.conv1_pad,\n stride=self.conv1_stride,\n acti_fn=\"sigmoid\",\n optimizer=self.optimizer,\n init_w=self.init_w,\n )\n self.layers[\"Pool2\"] = Pool2D(\n mode=\"max\",\n optimizer=self.optimizer,\n stride=self.pool2_stride,\n kernel_shape=self.pool2_kernel_shape,\n )\n self.layers[\"Flatten\"] = Flatten(optimizer=self.optimizer)\n self.layers[\"FC3\"] = FullyConnected(\n n_out=self.fc3_out,\n acti_fn=\"sigmoid\",\n init_w=self.init_w,\n optimizer=self.optimizer\n )\n self.layers[\"FC4\"] = FullyConnected(\n n_out=self.fc4_out,\n acti_fn=\"sigmoid\",\n init_w=self.init_w,\n optimizer=self.optimizer\n )\n self.layers[\"FC5\"] = FullyConnected(\n n_out=self.fc5_out,\n acti_fn=\"affine(slope=1, intercept=0)\",\n init_w=self.init_w,\n optimizer=self.optimizer\n )\n self.is_initialized = True\n \n def forward(self, X_train):\n Xs = {}\n out = X_train\n for k, v in self.layers.items():\n Xs[k] = out\n out = v.forward(out)\n return out, Xs\n \n def backward(self, grad):\n dXs = {}\n out = grad\n for k, v in reversed(list(self.layers.items())):\n dXs[k] = out\n out = v.backward(out)\n return out, dXs\n \n def update(self):\n \"\"\"\n 函数作用：梯度更新\n \"\"\"\n for k, v in reversed(list(self.layers.items())):\n v.update()\n self.flush_gradients()\n \n def flush_gradients(self, curr_loss=None):\n \"\"\"\n 函数作用：更新后重置梯度\n \"\"\"\n for k, v in self.layers.items():\n v.flush_gradients()\n \n def fit(self, X_train, y_train, n_epochs=20, batch_size=64, verbose=False, epo_verbose=True):\n \"\"\"\n 参数说明：\n X_train：训练数据\n y_train：训练数据标签\n n_epochs：epoch 次数\n batch_size：每次 epoch 的 batch size\n verbose：是否每个 batch 输出损失\n epo_verbose：是否每个 epoch 输出损失\n \"\"\"\n self.verbose = verbose\n self.n_epochs = n_epochs\n self.batch_size = batch_size\n \n if not self.is_initialized:\n self.n_features = X_train.shape[1]\n self._set_params()\n \n prev_loss = np.inf\n for i in range(n_epochs):\n loss, epoch_start = 0.0, time.time()\n batch_generator, n_batch = minibatch(X_train, self.batch_size, shuffle=True)\n\n for j, batch_idx in enumerate(batch_generator):\n batch_len, batch_start = len(batch_idx), time.time()\n X_batch, y_batch = X_train[batch_idx], y_train[batch_idx]\n out, _ = self.forward(X_batch)\n y_pred_batch = softmax(out)\n batch_loss = self.loss(y_batch, y_pred_batch)\n grad = self.loss.grad(y_batch, y_pred_batch)\n _, _ = self.backward(grad)\n self.update()\n loss += batch_loss\n\n if self.verbose:\n fstr = \"\\t[Batch {}/{}] Train loss: {:.3f} ({:.1f}s/batch)\"\n print(fstr.format(j + 1, n_batch, batch_loss, time.time() - batch_start))\n\n loss /= n_batch\n if epo_verbose:\n fstr = \"[Epoch {}] Avg. loss: {:.3f} Delta: {:.3f} ({:.2f}m/epoch)\"\n print(fstr.format(i + 1, loss, prev_loss - loss, (time.time() - epoch_start) / 60.0))\n prev_loss = loss\n \n def evaluate(self, X_test, y_test, batch_size=128):\n acc = 0.0\n batch_generator, n_batch = minibatch(X_test, batch_size, shuffle=True)\n for j, batch_idx in enumerate(batch_generator):\n batch_len, batch_start = len(batch_idx), time.time()\n X_batch, y_batch = X_test[batch_idx], y_test[batch_idx]\n y_pred_batch, _ = self.forward(X_batch)\n y_pred_batch = np.argmax(y_pred_batch, axis=1)\n y_batch = np.argmax(y_batch, axis=1)\n acc += np.sum(y_pred_batch == y_batch)\n return acc / X_test.shape[0]\n \n @property\n def hyperparams(self):\n return {\n \"init_w\": self.init_w,\n \"loss\": str(self.loss),\n \"optimizer\": self.optimizer,\n \"fc3_out\": self.fc3_out, \n \"fc4_out\": self.fc4_out,\n \"fc5_out\": self.fc5_out,\n \"conv1_pad\": self.conv1_pad, \n \"conv2_pad\": self.conv2_pad, \n \"conv1_stride\": self.conv1_stride,\n \"conv1_out_ch\": self.conv1_out_ch,\n \"pool1_stride\": self.pool1_stride,\n \"conv2_out_ch\": self.conv2_out_ch,\n \"conv2_stride\": self.conv2_stride, \n \"pool2_stride\": self.pool2_stride,\n \"conv2_kernel_shape\": self.conv2_kernel_shape,\n \"pool2_kernel_shape\": self.pool2_kernel_shape,\n \"conv1_kernel_shape\": self.conv1_kernel_shape,\n \"pool1_kernel_shape\": self.pool1_kernel_shape,\n \"components\": {k: v.params for k, v in self.layers.items()}\n }\n\n\n############# LeNet GEMM ################\nclass LeNet_gemm(object):\n \n def __init__(\n self,\n fc3_out=128,\n fc4_out=84,\n fc5_out=10,\n conv1_pad=0,\n conv2_pad=0,\n conv1_out_ch=6,\n conv2_out_ch=16,\n conv1_stride=1,\n pool1_stride=2,\n conv2_stride=1,\n pool2_stride=2,\n conv1_kernel_shape=(5, 5),\n pool1_kernel_shape=(2, 2),\n conv2_kernel_shape=(5, 5),\n pool2_kernel_shape=(2, 2),\n optimizer=\"adam\",\n init_w=\"glorot_normal\",\n loss=CrossEntropy()\n ):\n self.optimizer = optimizer\n self.init_w = init_w\n self.loss = loss\n self.fc3_out = fc3_out\n self.fc4_out = fc4_out\n self.fc5_out = fc5_out\n self.conv1_pad = conv1_pad\n self.conv2_pad = conv2_pad\n self.conv1_stride = conv1_stride\n self.conv1_out_ch = conv1_out_ch\n self.pool1_stride = pool1_stride\n self.conv2_out_ch = conv2_out_ch\n self.conv2_stride = conv2_stride\n self.pool2_stride = pool2_stride\n self.conv2_kernel_shape = conv2_kernel_shape\n self.pool2_kernel_shape = pool2_kernel_shape\n self.conv1_kernel_shape = conv1_kernel_shape\n self.pool1_kernel_shape = pool1_kernel_shape\n \n self.is_initialized = False\n \n def _set_params(self):\n \"\"\"\n 函数作用：模型初始化\n Conv1 -> Pool1 -> Conv2 -> Pool2 -> Flatten -> FC3 -> FC4 -> FC5 -> Softmax\n \"\"\"\n self.layers = OrderedDict()\n self.layers[\"Conv1\"] = Conv2D_gemm(\n out_ch=self.conv1_out_ch,\n kernel_shape=self.conv1_kernel_shape,\n pad=self.conv1_pad,\n stride=self.conv1_stride,\n acti_fn=\"sigmoid\",\n optimizer=self.optimizer,\n init_w=self.init_w,\n )\n self.layers[\"Pool1\"] = Pool2D(\n mode=\"max\",\n optimizer=self.optimizer,\n stride=self.pool1_stride,\n kernel_shape=self.pool1_kernel_shape,\n )\n self.layers[\"Conv2\"] = Conv2D_gemm(\n out_ch=self.conv1_out_ch,\n kernel_shape=self.conv1_kernel_shape,\n pad=self.conv1_pad,\n stride=self.conv1_stride,\n acti_fn=\"sigmoid\",\n optimizer=self.optimizer,\n init_w=self.init_w,\n )\n self.layers[\"Pool2\"] = Pool2D(\n mode=\"max\",\n optimizer=self.optimizer,\n stride=self.pool2_stride,\n kernel_shape=self.pool2_kernel_shape,\n )\n self.layers[\"Flatten\"] = Flatten(optimizer=self.optimizer)\n self.layers[\"FC3\"] = FullyConnected(\n n_out=self.fc3_out,\n acti_fn=\"sigmoid\",\n init_w=self.init_w,\n optimizer=self.optimizer\n )\n self.layers[\"FC4\"] = FullyConnected(\n n_out=self.fc4_out,\n acti_fn=\"sigmoid\",\n init_w=self.init_w,\n optimizer=self.optimizer\n )\n self.layers[\"FC5\"] = FullyConnected(\n n_out=self.fc5_out,\n acti_fn=\"affine(slope=1, intercept=0)\",\n init_w=self.init_w,\n optimizer=self.optimizer\n )\n self.is_initialized = True\n \n def forward(self, X_train):\n Xs = {}\n out = X_train\n for k, v in self.layers.items():\n Xs[k] = out\n out = v.forward(out)\n return out, Xs\n \n def backward(self, grad):\n dXs = {}\n out = grad\n for k, v in reversed(list(self.layers.items())):\n dXs[k] = out\n out = v.backward(out)\n return out, dXs\n \n def update(self):\n \"\"\"\n 函数作用：梯度更新\n \"\"\"\n for k, v in reversed(list(self.layers.items())):\n v.update()\n self.flush_gradients()\n \n def flush_gradients(self, curr_loss=None):\n \"\"\"\n 函数作用：更新后重置梯度\n \"\"\"\n for k, v in self.layers.items():\n v.flush_gradients()\n \n def fit(self, X_train, y_train, n_epochs=20, batch_size=64, verbose=False, epo_verbose=True):\n \"\"\"\n 参数说明：\n X_train：训练数据\n y_train：训练数据标签\n n_epochs：epoch 次数\n batch_size：每次 epoch 的 batch size\n verbose：是否每个 batch 输出损失\n epo_verbose：是否每个 epoch 输出损失\n \"\"\"\n self.verbose = verbose\n self.n_epochs = n_epochs\n self.batch_size = batch_size\n \n if not self.is_initialized:\n self.n_features = X_train.shape[1]\n self._set_params()\n \n prev_loss = np.inf\n for i in range(n_epochs):\n loss, epoch_start = 0.0, time.time()\n batch_generator, n_batch = minibatch(X_train, self.batch_size, shuffle=True)\n\n for j, batch_idx in enumerate(batch_generator):\n batch_len, batch_start = len(batch_idx), time.time()\n X_batch, y_batch = X_train[batch_idx], y_train[batch_idx]\n out, _ = self.forward(X_batch)\n y_pred_batch = softmax(out)\n batch_loss = self.loss(y_batch, y_pred_batch)\n grad = self.loss.grad(y_batch, y_pred_batch)\n _, _ = self.backward(grad)\n self.update()\n loss += batch_loss\n\n if self.verbose:\n fstr = \"\\t[Batch {}/{}] Train loss: {:.3f} ({:.1f}s/batch)\"\n print(fstr.format(j + 1, n_batch, batch_loss, time.time() - batch_start))\n\n loss /= n_batch\n if epo_verbose:\n fstr = \"[Epoch {}] Avg. loss: {:.3f} Delta: {:.3f} ({:.2f}m/epoch)\"\n print(fstr.format(i + 1, loss, prev_loss - loss, (time.time() - epoch_start) / 60.0))\n prev_loss = loss\n \n def evaluate(self, X_test, y_test, batch_size=128):\n acc = 0.0\n batch_generator, n_batch = minibatch(X_test, batch_size, shuffle=True)\n for j, batch_idx in enumerate(batch_generator):\n batch_len, batch_start = len(batch_idx), time.time()\n X_batch, y_batch = X_test[batch_idx], y_test[batch_idx]\n y_pred_batch, _ = self.forward(X_batch)\n y_pred_batch = np.argmax(y_pred_batch, axis=1)\n y_batch = np.argmax(y_batch, axis=1)\n acc += np.sum(y_pred_batch == y_batch)\n return acc / X_test.shape[0]\n \n @property\n def hyperparams(self):\n return {\n \"init_w\": self.init_w,\n \"loss\": str(self.loss),\n \"optimizer\": self.optimizer,\n \"fc3_out\": self.fc3_out, \n \"fc4_out\": self.fc4_out,\n \"fc5_out\": self.fc5_out,\n \"conv1_pad\": self.conv1_pad, \n \"conv2_pad\": self.conv2_pad, \n \"conv1_stride\": self.conv1_stride,\n \"conv1_out_ch\": self.conv1_out_ch,\n \"pool1_stride\": self.pool1_stride,\n \"conv2_out_ch\": self.conv2_out_ch,\n \"conv2_stride\": self.conv2_stride, \n \"pool2_stride\": self.pool2_stride,\n \"conv2_kernel_shape\": self.conv2_kernel_shape,\n \"pool2_kernel_shape\": self.pool2_kernel_shape,\n \"conv1_kernel_shape\": self.conv1_kernel_shape,\n \"pool1_kernel_shape\": self.pool1_kernel_shape,\n \"components\": {k: v.params for k, v in self.layers.items()}\n }\n"
},
{
"path": "code/method/__init__.py",
"content": "from . import optimizer\nfrom . import activation\n\n"
},
{
"path": "code/method/activation/activation.py",
"content": "from abc import ABC, abstractmethod\nimport numpy as np\nimport re\n\n\nclass ActivationBase(ABC):\n \n def __init__(self, **kwargs):\n super().__init__()\n\n def __call__(self, z):\n if z.ndim == 1:\n z = z.reshape(1, -1)\n return self.forward(z)\n\n @abstractmethod\n def forward(self, z):\n raise NotImplementedError\n\n @abstractmethod\n def grad(self, x, **kwargs):\n raise NotImplementedError\n\n\nclass Sigmoid(ActivationBase):\n \"\"\"\n Sigmoid(x) = 1 / (1 + e^(-x))\n \"\"\"\n\n def __init__(self):\n super().__init__()\n\n def __str__(self):\n return \"Sigmoid\"\n\n def forward(self, z):\n return 1 / (1 + np.exp(-z))\n\n def grad(self, x):\n return self.forward(x) * (1 - self.forward(x))\n\n\nclass Tanh(ActivationBase):\n \"\"\"\n Tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x))\n \"\"\"\n\n def __init__(self):\n super().__init__()\n\n def __str__(self):\n return \"Tanh\"\n\n def forward(self, z):\n return np.tanh(z)\n\n def grad(self, x):\n return 1 - np.tanh(x) ** 2\n \n \nclass ReLU(ActivationBase):\n \"\"\"\n ReLU(x) =\n x if x > 0\n 0 otherwise\n \"\"\"\n\n def __init__(self):\n super().__init__()\n\n def __str__(self):\n return \"ReLU\"\n\n def forward(self, z):\n return np.clip(z, 0, np.inf)\n\n def grad(self, x):\n return (x > 0).astype(int)\n\n\nclass LeakyReLU(ActivationBase):\n \"\"\"\n LeakyReLU(x) =\n alpha * x if x < 0\n x otherwise\n \"\"\"\n\n def __init__(self, alpha=0.3):\n self.alpha = alpha\n super().__init__()\n\n def __str__(self):\n return \"Leaky ReLU(alpha={})\".format(self.alpha)\n\n def forward(self, z):\n _z = z.copy()\n _z[z < 0] = _z[z < 0] * self.alpha\n return _z\n\n def grad(self, x):\n out = np.ones_like(x)\n out[x < 0] *= self.alpha\n return out\n\n\nclass Affine(ActivationBase):\n \"\"\"\n Affine(x) = slope * x + intercept\n \"\"\"\n\n def __init__(self, slope=1, intercept=0):\n self.slope = slope\n self.intercept = intercept\n super().__init__()\n\n def __str__(self):\n return \"Affine(slope={}, intercept={})\".format(self.slope, self.intercept)\n\n def forward(self, z):\n return self.slope * z + self.intercept\n\n def grad(self, x):\n return self.slope * np.ones_like(x)\n\n\nclass SoftPlus(ActivationBase):\n \"\"\"\n SoftPlus(x) = log(1 + e^x)\n \"\"\"\n\n def __init__(self):\n super().__init__()\n\n def __str__(self):\n return \"SoftPlus\"\n\n def forward(self, z):\n return np.log(np.exp(z) + 1)\n\n def grad(self, x):\n return np.exp(x) / (np.exp(x) + 1)\n \n \nclass ELU(ActivationBase):\n \"\"\"\n ELU(x) =\n x if x >= 0\n alpha * (e^x - 1) otherwise\n \"\"\"\n\n def __init__(self, alpha=1.0):\n self.alpha = alpha\n super().__init__()\n\n def __str__(self):\n return \"ELU(alpha={})\".format(self.alpha)\n\n def forward(self, z):\n return np.where(z > 0, z, self.alpha * (np.exp(z) - 1))\n\n def grad(self, x):\n return np.where(x >= 0, np.ones_like(x), self.alpha * np.exp(x))\n\n\nclass Exponential(ActivationBase):\n \"\"\"\n Exponential(x) = e^x\n \"\"\"\n\n def __init__(self):\n super().__init__()\n\n def __str__(self):\n return \"Exponential\"\n\n def forward(self, z):\n return np.exp(z)\n\n def grad(self, x):\n return np.exp(x)\n\n\nclass SELU(ActivationBase):\n \"\"\"\n SELU(x) = scale * ELU(x, alpha)\n = scale * x if x >= 0\n scale * [alpha * (e^x - 1)] otherwise\n \"\"\"\n\n def __init__(self):\n self.alpha = 1.6732632423543772848170429916717\n self.scale = 1.0507009873554804934193349852946\n self.elu = ELU(alpha=self.alpha)\n super().__init__()\n\n def __str__(self):\n return \"SELU\"\n\n def forward(self, z):\n return self.scale * self.elu.forward(z)\n\n def grad(self, x):\n return np.where(\n x >= 0, np.ones_like(x) * self.scale, np.exp(x) * self.alpha * self.scale\n )\n\n\nclass HardSigmoid(ActivationBase):\n \"\"\"\n HardSigmoid(x) =\n 0 if x < -2.5\n 0.2 * x + 0.5 if -2.5 <= x <= 2.5.\n 1 if x > 2.5\n \"\"\"\n\n def __init__(self):\n super().__init__()\n\n def __str__(self):\n return \"Hard Sigmoid\"\n\n def forward(self, z):\n return np.clip((0.2 * z) + 0.5, 0.0, 1.0)\n\n def grad(self, x):\n return np.where((x >= -2.5) & (x <= 2.5), 0.2, 0)\n\n \nclass ActivationInitializer(object):\n \n def __init__(self, acti_name=\"affine(slope=1, intercept=0)\"):\n self.acti_name = acti_name\n\n def __call__(self):\n acti_str = self.acti_name.lower()\n if acti_str == \"relu\":\n acti_fn = ReLU()\n elif acti_str == \"tanh\":\n acti_fn = Tanh()\n elif acti_str == \"sigmoid\":\n acti_fn = Sigmoid()\n elif \"affine\" in acti_str:\n r = r\"affine\\(slope=(.*), intercept=(.*)\\)\"\n slope, intercept = re.match(r, acti_str).groups()\n acti_fn = Affine(float(slope), float(intercept))\n elif \"leaky relu\" in acti_str:\n r = r\"leaky relu\\(alpha=(.*)\\)\"\n alpha = re.match(r, acti_str).groups()[0]\n acti_fn = LeakyReLU(float(alpha))\n else:\n raise ValueError(\"Unknown activation: {}\".format(acti_str))\n return acti_fn\n"
},
{
"path": "code/method/optimizer/optimizer.py",
"content": "from abc import ABC, abstractmethod\nimport numpy as np\nimport re\n\n\nclass OptimizerBase(ABC):\n \n def __init__(self):\n pass\n \n def __call__(self, params, params_grad, params_name):\n \"\"\"\n 参数说明：\n params：待更新参数， 如权重矩阵 W；\n params_grad：待更新参数的梯度；\n params_name：待更新参数名；\n \"\"\"\n return self.update(params, params_grad, params_name)\n \n @abstractmethod\n def update(self, params, params_grad, params_name):\n raise NotImplementedError\n\n \nclass SGD(OptimizerBase):\n \"\"\"\n sgd 优化方法\n \"\"\"\n \n def __init__(self, lr=0.01):\n super().__init__()\n self.lr = lr \n self.cache = {}\n \n def __str__(self):\n return \"SGD(lr={})\".format(self.hyperparams[\"lr\"])\n \n def update(self, params, params_grad, params_name):\n update_value = self.lr * params_grad\n return params - update_value\n \n @property\n def hyperparams(self):\n return {\n \"op\": \"SGD\",\n \"lr\": self.lr\n }\n\nclass Momentum(OptimizerBase):\n \n def __init__(\n self, lr=0.001, momentum=0.0, **kwargs\n ):\n \"\"\"\n 参数说明：\n lr： 学习率，float (default: 0.001)\n momentum：考虑 Momentum 时的 alpha，决定了之前的梯度贡献衰减得有多快，取值范围[0, 1]，默认0\n \"\"\"\n super().__init__()\n self.lr = lr \n self.momentum = momentum\n self.cache = {}\n\n def __str__(self):\n return \"Momentum(lr={}, momentum={})\".format(self.lr, self.momentum)\n\n def update(self, param, param_grad, param_name):\n C = self.cache\n lr, momentum = self.lr, self.momentum\n\n if param_name not in C: # save v\n C[param_name] = np.zeros_like(param_grad)\n\n update = momentum * C[param_name] - lr * param_grad\n self.cache[param_name] = update\n return param + update\n \n @property\n def hyperparams(self):\n return {\n \"op\": \"Momentum\",\n \"lr\": self.lr,\n \"momentum\": self.momentum\n }\n \n\nclass AdaGrad(OptimizerBase):\n\n def __init__(self, lr=0.001, eps=1e-7, **kwargs):\n \"\"\"\n 参数说明：\n lr： 学习率，float (default: 0.001)\n eps：delta 项，防止分母为0\n \"\"\"\n super().__init__()\n self.lr = lr\n self.eps = eps\n self.cache = {}\n\n def __str__(self):\n return \"AdaGrad(lr={}, eps={})\".format(self.lr, self.eps)\n\n def update(self, param, param_grad, param_name):\n C = self.cache\n lr, eps = self.hyperparams[\"lr\"], self.hyperparams[\"eps\"]\n\n if param_name not in C: # save r\n C[param_name] = np.zeros_like(param_grad)\n\n C[param_name] += param_grad ** 2\n update = lr * param_grad / (np.sqrt(C[param_name]) + eps)\n self.cache = C\n return param - update\n\n @property\n def hyperparams(self):\n return {\n \"op\": \"AdaGrad\",\n \"lr\": self.lr,\n \"eps\": self.eps\n }\n \n \nclass RMSProp(OptimizerBase):\n \n def __init__(\n self, lr=0.001, decay=0.9, eps=1e-7, **kwargs\n ):\n \"\"\"\n 参数说明：\n lr： 学习率，float (default: 0.001)\n eps：delta 项，防止分母为0\n decay：衰减速率\n \"\"\"\n super().__init__()\n self.lr = lr\n self.eps = eps\n self.decay = decay\n self.cache = {}\n\n def __str__(self):\n return \"RMSProp(lr={}, eps={}, decay={})\".format(\n self.lr, self.eps, self.decay\n )\n\n def update(self, param, param_grad, param_name):\n C = self.cache\n lr, eps = self.hyperparams[\"lr\"], self.hyperparams[\"eps\"]\n decay = self.hyperparams[\"decay\"]\n\n if param_name not in C: # save r\n C[param_name] = np.zeros_like(param_grad)\n\n C[param_name] = decay * C[param_name] + (1 - decay) * param_grad ** 2\n update = lr * param_grad / (np.sqrt(C[param_name]) + eps)\n self.cache = C\n return param - update\n \n @property\n def hyperparams(self):\n return {\n \"op\": \"RMSProp\",\n \"lr\": self.lr,\n \"eps\": self.eps,\n \"decay\": self.decay\n } \n \n \nclass AdaDelta(OptimizerBase):\n \n def __init__(\n self, lr=0.001, decay=0.95, eps=1e-7, **kwargs\n ):\n \"\"\"\n 参数说明：\n lr： 学习率，float (default: 0.001)\n eps：delta 项，防止分母为0\n decay：衰减速率\n \"\"\"\n super().__init__()\n self.lr = lr\n self.eps = eps\n self.decay = decay\n self.cache = {}\n\n def __str__(self):\n return \"AdaDelta(eps={}, decay={})\".format(self.eps, self.decay)\n\n def update(self, param, param_grad, param_name):\n C = self.cache\n eps = self.hyperparams[\"eps\"]\n decay = self.hyperparams[\"decay\"]\n\n if param_name not in C: # save r, delta_theta\n C[param_name] = {\n \"r\": np.zeros_like(param_grad),\n \"d\": np.zeros_like(param_grad)\n }\n\n C[param_name][\"r\"] = decay * C[param_name][\"r\"] + (1 - decay) * param_grad ** 2\n update = (np.sqrt(C[param_name][\"d\"] + eps)) * param_grad / (np.sqrt(C[param_name][\"r\"]) + eps)\n C[param_name][\"d\"] = decay * C[param_name][\"d\"] + (1 - decay) * update ** 2\n self.cache = C\n return param - update\n \n @property\n def hyperparams(self):\n return {\n \"op\": \"AdaDelta\",\n \"eps\": self.eps,\n \"decay\": self.decay\n }\n \n \nclass Adam(OptimizerBase):\n \n def __init__(\n self,\n lr=0.001,\n decay1=0.9,\n decay2=0.999,\n eps=1e-7,\n **kwargs\n ):\n \"\"\"\n 参数说明：\n lr： 学习率，float (default: 0.01)\n eps：delta 项，防止分母为0\n decay1：历史梯度的指数衰减速率，可以理解为考虑梯度均值 (default: 0.9)\n decay2：历史梯度平方的指数衰减速率，可以理解为考虑梯度方差 (default: 0.999)\n \"\"\"\n super().__init__()\n self.lr = lr\n self.decay1 = decay1\n self.decay2 = decay2\n self.eps = eps\n self.cache = {}\n\n def __str__(self):\n return \"Adam(lr={}, decay1={}, decay2={}, eps={})\".format(\n self.lr, self.decay1, self.decay2, self.eps\n )\n\n def update(self, param, param_grad, param_name, cur_loss=None):\n C = self.cache\n d1, d2 = self.hyperparams[\"decay1\"], self.hyperparams[\"decay2\"]\n lr, eps= self.hyperparams[\"lr\"], self.hyperparams[\"eps\"]\n\n if param_name not in C:\n C[param_name] = {\n \"t\": 0,\n \"mean\": np.zeros_like(param_grad),\n \"var\": np.zeros_like(param_grad),\n }\n\n t = C[param_name][\"t\"] + 1\n mean = C[param_name][\"mean\"]\n var = C[param_name][\"var\"]\n\n C[param_name][\"t\"] = t\n C[param_name][\"mean\"] = d1 * mean + (1 - d1) * param_grad\n C[param_name][\"var\"] = d2 * var + (1 - d2) * param_grad ** 2\n self.cache = C\n\n m_hat = C[param_name][\"mean\"] / (1 - d1 ** t)\n v_hat = C[param_name][\"var\"] / (1 - d2 ** t)\n update = lr * m_hat / (np.sqrt(v_hat) + eps)\n return param - update\n\n @property\n def hyperparams(self):\n return {\n \"op\": \"Adam\",\n \"lr\": self.lr,\n \"eps\": self.eps,\n \"decay1\": self.decay1,\n \"decay2\": self.decay2\n } \n \n \nclass OptimizerInitializer(ABC):\n \n def __init__(self, opti_name=\"sgd\"):\n self.opti_name = opti_name\n \n def __call__(self):\n r = r\"([a-zA-Z]*)=([^,)]*)\"\n opti_str = self.opti_name.lower()\n kwargs = dict([(i, eval(j)) for (i, j) in re.findall(r, opti_str)])\n if \"sgd\" in opti_str:\n optimizer = SGD(**kwargs)\n elif \"momentum\" in opti_str:\n optimizer = Momentum(**kwargs) \n elif \"adagrad\" in opti_str:\n optimizer = AdaGrad(**kwargs)\n elif \"rmsprop\" in opti_str:\n optimizer = RMSProp(**kwargs)\n elif \"adadelta\" in opti_str:\n optimizer = AdaDelta(**kwargs)\n elif \"adam\" in opti_str:\n optimizer = Adam(**kwargs)\n else:\n raise NotImplementedError(\"{}\".format(opt_str))\n return optimizer\n \n"
},
{
"path": "code/method/weight/weight.py",
"content": "from abc import ABC, abstractmethod\nimport numpy as np\nimport re\n\n\ndef calc_fan(weight_shape):\n \"\"\"\n 对权重矩阵计算 fan-in 和 fan-out\n\n 参数说明： \n weight_shape：权重形状\n \"\"\"\n if len(weight_shape) == 2: \n fan_in, fan_out = weight_shape\n elif len(weight_shape) in [3, 4]:\n in_ch, out_ch = weight_shape[-2:]\n kernel_size = np.prod(weight_shape[:-2])\n fan_in, fan_out = in_ch * kernel_size, out_ch * kernel_size\n else:\n raise ValueError(\"Unrecognized weight dimension: {}\".format(weight_shape))\n return fan_in, fan_out\n\n\nclass random_uniform:\n \"\"\"\n 初始化网络权重 W--- 基于 Uniform(-b, b)\n\n 参数说明：\n weight_shape：权重形状\n \"\"\"\n def __init__(self, b=1.0):\n self.b = b\n \n def __call__(self, weight_shape):\n return np.random.uniform(-b, b, size=weight_shape)\n\n\nclass random_normal:\n \"\"\"\n 初始化网络权重 W--- 基于 TruncatedNormal(0, std)\n\n 参数说明： \n weight_shape：权重形状\n std：权重标准差\n \"\"\"\n def __init__(self, std=0.01):\n self.std = std\n \n def __call__(self, weight_shape):\n return truncated_normal(0, std, weight_shape)\n\n \n# def random_uniform(weight_shape, b=1.0):\n# \"\"\"\n# 初始化网络权重 W--- 基于 Uniform(-b, b)\n\n# 参数说明：\n# weight_shape：权重形状\n# \"\"\"\n# return np.random.uniform(-b, b, size=weight_shape)\n\n\n# def random_normal(weight_shape, std=1.0):\n# \"\"\"\n# 初始化网络权重 W--- 基于 TruncatedNormal(0, std)\n\n# 参数说明： \n# weight_shape：权重形状\n# std：权重标准差\n# \"\"\"\n# return truncated_normal(0, std, weight_shape)\n \n\nclass he_uniform:\n \"\"\"\n 初始化网络权重 W--- 基于 Uniform(-b, b)，其中 b=sqrt(6/fan_in)，常用于 ReLU 激活层\n\n 参数说明：\n weight_shape：权重形状\n \"\"\"\n def __init__(self):\n pass\n \n def __call__(self, weight_shape):\n fan_in, fan_out = calc_fan(weight_shape)\n b = np.sqrt(6 / fan_in)\n return np.random.uniform(-b, b, size=weight_shape)\n \n \nclass he_normal:\n \"\"\"\n 初始化网络权重 W--- 基于 TruncatedNormal(0, std)，其中 std=2/fan_in，常用于 ReLU 激活层\n\n 参数说明： \n weight_shape：权重形状\n \"\"\"\n def __init__(self):\n pass\n \n def __call__(self, weight_shape):\n fan_in, fan_out = calc_fan(weight_shape)\n std = np.sqrt(2 / fan_in)\n return truncated_normal(0, std, weight_shape)\n \n \n \n# def he_uniform(weight_shape):\n# \"\"\"\n# 初始化网络权重 W--- 基于 Uniform(-b, b)，其中 b=sqrt(6/fan_in)，常用于 ReLU 激活层\n\n# 参数说明：\n# weight_shape：权重形状\n# \"\"\"\n# fan_in, fan_out = calc_fan(weight_shape)\n# b = np.sqrt(6 / fan_in)\n# return np.random.uniform(-b, b, size=weight_shape)\n \n \n# def he_normal(weight_shape):\n# \"\"\"\n# 初始化网络权重 W--- 基于 TruncatedNormal(0, std)，其中 std=2/fan_in，常用于 ReLU 激活层\n\n# 参数说明： \n# weight_shape：权重形状\n# \"\"\"\n# fan_in, fan_out = calc_fan(weight_shape)\n# std = np.sqrt(2 / fan_in)\n# return truncated_normal(0, std, weight_shape)\n \n\nclass glorot_uniform:\n \"\"\"\n 初始化网络权重 W--- 基于 Uniform(-b, b)，其中 b=gain*sqrt(6/(fan_in+fan_out))，\n 常用于 tanh 和 sigmoid 激活层\n\n 参数说明：\n weight_shape：权重形状\n \"\"\"\n def __init__(self, gain=1.0):\n self.gain = gain\n \n def __call__(self, weight_shape):\n fan_in, fan_out = calc_fan(weight_shape)\n b = self.gain * np.sqrt(6 / (fan_in + fan_out))\n return np.random.uniform(-b, b, size=weight_shape)\n \n\nclass glorot_normal:\n \"\"\"\n 初始化网络权重 W--- 基于 TruncatedNormal(0, std)，其中 std=gain^2*2/(fan_in+fan_out)，\n 常用于 tanh 和 sigmoid 激活层\n\n 参数说明：\n weight_shape：权重形状\n \"\"\"\n def __init__(self, gain=1.0):\n self.gain = gain\n \n def __call__(self, weight_shape):\n fan_in, fan_out = calc_fan(weight_shape)\n std = self.gain * np.sqrt(2 / (fan_in + fan_out))\n return truncated_normal(0, std, weight_shape)\n \n \n# def glorot_uniform(weight_shape, gain=1.0):\n# \"\"\"\n# 初始化网络权重 W--- 基于 Uniform(-b, b)，其中 b=gain*sqrt(6/(fan_in+fan_out))，\n# 常用于 tanh 和 sigmoid 激活层\n\n# 参数说明：\n# weight_shape：权重形状\n# \"\"\"\n# fan_in, fan_out = calc_fan(weight_shape)\n# b = gain * np.sqrt(6 / (fan_in + fan_out))\n# return np.random.uniform(-b, b, size=weight_shape)\n \n \n# def glorot_normal(weight_shape, gain=1.0):\n# \"\"\"\n# 初始化网络权重 W--- 基于 TruncatedNormal(0, std)，其中 std=gain^2*2/(fan_in+fan_out)，\n# 常用于 tanh 和 sigmoid 激活层\n\n# 参数说明：\n# weight_shape：权重形状\n# \"\"\"\n# fan_in, fan_out = calc_fan(weight_shape)\n# std = gain * np.sqrt(2 / (fan_in + fan_out))\n# return truncated_normal(0, std, weight_shape)\n\n\ndef truncated_normal(mean, std, out_shape):\n \"\"\"\n 通过拒绝采样生成截断正态分布\n\n 参数说明：\n mean：正态分布均值\n std：正态分布标准差\n out_shape：矩阵形状\n \"\"\"\n samples = np.random.normal(loc=mean, scale=std, size=out_shape)\n reject = np.logical_or(samples >= mean + 2 * std, samples <= mean - 2 * std)\n while any(reject.flatten()):\n resamples = np.random.normal(loc=mean, scale=std, size=reject.sum())\n samples[reject] = resamples\n reject = np.logical_or(samples >= mean + 2 * std, samples <= mean - 2 * std)\n return samples\n\n \nclass WeightInitializer(object):\n\n def __init__(self, mode=\"he_normal\"):\n \"\"\"\n mode：权重初始化策略 str型 (default: 'he_normal')\n \"\"\"\n self.mode = mode\n r = r\"([a-zA-Z]*)=([^,)]*)\"\n mode_str = self.mode.lower()\n kwargs = dict([(i, eval(j)) for (i, j) in re.findall(r, mode_str)])\n \n if \"random_uniform\" in mode_str:\n self.init_fn = random_uniform(**kwargs)\n elif \"random_normal\" in mode_str:\n self.init_fn = random_normal(**kwargs)\n elif \"he_uniform\" in mode_str:\n self.init_fn = he_uniform(**kwargs)\n elif \"he_normal\" in mode_str:\n self.init_fn = he_normal(**kwargs)\n elif \"glorot_uniform\" in mode_str:\n self.init_fn = glorot_uniform(**kwargs)\n elif \"glorot_normal\" in mode_str:\n self.init_fn = glorot_normal(**kwargs)\n else:\n raise ValueError(\"Unrecognize initialization mode: {}\".format(mode_str))\n \n def __call__(self, weight_shape):\n W = self.init_fn(weight_shape)\n return W\n"
},
{
"path": "contents.txt",
"content": "注：目录是基于《深度学习》的目录起的。基于本项目的内容，目录其实可以分的更细致，这里就分到目录的第三级为止。\n\n**目录**:\n\n- 第二章 线性代数\n - 1 标量, 向量, 矩阵, 张量\n - 2 矩阵转置\n - 3 矩阵加法\n - 4 矩阵乘法\n - 5 单位矩阵\n - 6 矩阵的逆\n - 7 范数\n - 8 特征值分解\n - 9 奇异值分解\n - 10 PCA (主成分分析)\n\n\n- 第三章 概率与信息论\n - 1 概率\n - 1.1 概率与随机变量\n - 1.2 概率分布\n - 1.2.1 概率质量函数\n - 1.2.2 概率密度函数\n - 1.2.3 累积分布函数\n - 1.3 条件概率与条件独立\n - 1.4 随机变量的度量\n - 1.5 常用概率分布\n - 1.5.1 伯努利分布 (两点分布)\n - 1.5.2 范畴分布 (分类分布)\n - 1.5.3 高斯分布 (正态分布)\n - 1.5.4 多元高斯分布 (多元正态分布)\n - 1.5.5 指数分布\n - 1.5.6 拉普拉斯分布\n - 1.5.7 Dirac 分布\n - 1.6 常用函数的有用性质\n - 1.6.1 logistic sigmoid 函数\n - 1.6.2 softplus 函数\n - 2 信息论\n - 3 图模型\n - 3.1 有向图模型\n - 3.1.1 贝叶斯网的独立性\n - 3.2 无向图模型\n - 3.1.2 马尔可夫网的条件独立性\n\n\n- 第四章 数值计算\n - 1 上溢和下溢\n - 2 优化方法\n - 2.1 梯度下降法\n - 2.2 牛顿法\n - 2.3 约束优化\n\n\n- 第五章 机器学习基础\n - 1 学习算法\n - 1.1 举例:线性回归 \n - 2 容量、过拟合、欠拟合\n - 2.1 泛化问题\n - 2.2 容量\n - 3 超参数与验证集\n - 4 偏差和方差\n - 4.1 偏差\n - 4.2 方差\n - 4.3 误差与偏差和方差的关系\n - 5 最大似然估计\n - 6 贝叶斯统计\n - 7 最大后验估计\n - 7.1 举例:线性回归\n - 8 监督学习方法\n - 8.1 概率监督学习\n - 8.2 支持向量机\n - 8.2.1 核技巧\n - 8.3 k-近邻\n - 8.4 决策树\n - 8.4.1 特征选择\n - 8.4.2 决策树生成\n - 8.4.3 决策树正则化\n - 9 无监督学习方法\n - 9.1 主成分分析法\n - 9.2 k-均值聚类\n\n\n- 第六章 深度前馈网络\n - 1 深度前馈网络\n - 2 DFN 相关设计\n - 2.1 隐藏单元\n - 2.2 输出单元\n - 2.3 代价函数\n - 2.4 架构设计\n - 3 反向传播算法\n - 3.1 单个神经元的训练\n - 3.2 多层神经网络的训练\n - 3.2.1 定义权重初始化方法\n - 3.2.2 定义激活函数\n - 3.2.3 定义优化方法\n - 3.2.4 定义网络层的框架\n - 3.2.5 定义代价函数\n - 3.2.6 定义深度前馈网络\n - 4 神经网络的万能近似定理\n - 5 实例:学习 XOR\n\n\n- 第七章 深度学习中的正则化\n - 1 参数范数惩罚\n - 1.1 L2 正则化\n - 1.2 L1 正则化\n - 1.3 总结 (L2 正则化与L1 正则化的解)\n - 1.4 作为约束的范数惩罚\n - 1.5 欠约束问题\n - 2 数据增强\n - 2.1 数据集增强\n - 2.2 噪声鲁棒性\n - 3 训练方案\n - 3.1 半监督学习\n - 3.2 多任务学习\n - 3.3 提前终止\n - 4 模型表示\n - 4.1 参数绑定与共享\n - 4.2 稀疏表示\n - 4.3 Bagging 及其他集成方法\n - 4.3.1 Bagging 方法\n - 4.3.2 随机森林\n - 4.3.3 方法解决过拟合\n - 4.4 Dropout\n - 5 样本测试\n - 6 补充材料\n - 6.1 Boosting\n - 6.1.1 前向分步加法模型\n - 6.1.2 AdaBoost 算法\n - 6.1.3 Boosting Tree 算法与 GBDT 算法\n - 6.1.4 XGBoost 算法\n\n\n- 第八章 深度模型中的优化\n - 1 基本优化算法\n - 1.1 梯度\n - 1.1.1 梯度下降\n - 1.1.2 随机梯度下降\n - 1.2 动量\n - 1.2.1 Momentum 算法\n - 1.2.2 NAG 算法\n - 1.3 自适应学习率\n - 1.3.1 AdaGrad 算法\n - 1.3.2 RMSProp 算法\n - 1.3.3 AdaDelta 算法\n - 1.3.4 Adam 算法\n - 1.4 二阶近似方法\n - 1.4.1 牛顿法\n - 1.4.2 拟牛顿法\n - 2 优化策略\n - 2.1 参数初始化\n - 3 批标准化\n - 4 坐标下降\n - 5 Polyak 平均\n - 6 监督预训练\n - 7 设计有助于优化的模型\n\n\n- 第九章 卷积网络\n - 1 卷积运算\n - 2 池化\n - 3 深度学习框架下的卷积\n - 3.1 多个并行卷积\n - 3.2 输入值与核\n - 3.3 填充 (Padding)\n - 3.4 卷积步幅 (Stride)\n - 4 更多的卷积策略\n - 4.1 深度可分离卷积 (Depthwise Separable Convolution)\n - 4.2 分组卷积 (Group Convolution)\n - 4.3 扩张卷积 (Dilated Convolution)\n - 5 GEMM 转换\n - 6 卷积网络的训练\n - 6.1 卷积网络示意图\n - 6.2 单层卷积层/池化层\n - 6.2.1 卷积函数的导数及反向传播\n - 6.2.2 池化函数的导数及后向传播\n - 6.3 多层卷积层/池化层\n - 6.4 Flatten 层 & 全连接层\n - 7 平移等变\n - 8 代表性的卷积神经网络\n - 8.1 卷积神经网络 (LeNet)\n\n\n- 第十一章 实践方法论\n - 1 实践方法论\n - 2 性能度量指标\n - 2.1 错误率与准确性\n - 2.2 查准率、查全率与 F1 值\n - 2.2.1 混淆矩阵\n - 2.2.2 查准率和查全率的定义与关联\n - 2.2.3 F1 值\n - 2.3 PR 曲线\n - 2.4 ROC 曲线与 AUC 值\n - 2.4.1 ROC 曲线\n - 2.4.2 AUC 值的计算方法\n - 2.5 覆盖\n - 2.6 指标性能的瓶颈\n - 3 默认基准模型\n - 4 确定是否收集更多数据\n - 5 选择超参数\n - 5.1 手动超参数调整\n - 5.2 自动超参数优化算法\n - 5.2.1 网格搜索 (Grid Search)\n - 5.2.2 随机搜索 (Random Search)\n - 5.2.3 基于模型的超参数优化 (Model-based Hyperparameter Optimization)\n"
},
{
"path": "reference.txt",
"content": "**参考文献**:\n\n- 全局参考\n - https://github.com/exacity/deeplearningbook-chinese/\n\n- 线性代数\n - https://github.com/npetrakis/YearOfML\n\n- 概率与信息论\n - https://github.com/akhilvasvani/Probability-and-Information-Theory\n - https://docs.scipy.org/doc/scipy/reference/stats.html\n - https://www.joinquant.com/view/community/detail/be3d8bc42275ea491897ac13fbf5838f\n - https://my.oschina.net/dillan/blog/134011\n \n- 数值计算\n\n- 机器学习基础\n - 《统计学习方法》\n - 《机器学习》\n - https://github.com/paruby/ml-basics\n - https://github.com/akhilvasvani/machine-learning-basics\n - https://blog.csdn.net/ajianyingxiaoqinghan/article/details/72897399\n\n- 深度前馈网络\n - https://peterroelants.github.io/posts/cross-entropy-logistic/\n - https://blog.csdn.net/weixin_36586536/article/details/80468426\n - https://github.com/yg19930918/deep-learning-from-scratch-master\n - https://www.cnblogs.com/34fj/p/9036369.html\n - https://github.com/peterroelants/peterroelants.github.io\n\n- 深度学习中的正则化\n - https://zhuanlan.zhihu.com/p/35893078\n - https://www.jiqizhixin.com/articles/2017-06-23-5\n - https://www.zybuluo.com/songying/note/1400484\n - https://zhuanlan.zhihu.com/p/37120298\n - https://kevinzakka.github.io/2016/09/14/batch_normalization/\n - http://gitlinux.net/2018-10-29-xgboost/\n - https://medium.com/swlh/boosting-and-bagging-explained-with-examples-5353a36eb78d\n - http://www.ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/slides/gradient_boosting.pdf\n - https://blog.csdn.net/liangjun_feng/article/details/79603705\n - https://blog.csdn.net/sinat_22594309/article/details/60957594\n - https://www.zybuluo.com/yxd/note/611571\n - http://freemind.pluskid.org/machine-learning/sparsity-and-some-basics-of-l1-regularization/#ed61992b37932e208ae114be75e42a3e6dc34cb3\n\n- 深度模型中的优化\n - https://zhuanlan.zhihu.com/p/32626442\n - https://github.com/exacity/deeplearningbook-chinese\n - http://cthorey.github.io./backpropagation/\n - http://www.ludoart.cn/2019/02/22/Optimization-Methods/\n - https://blog.csdn.net/itplus/article/details/21897715\n \n- 卷积神经网络\n - https://www.slideshare.net/kuwajima/cnnbp\n - https://github.com/exacity/simplified-deeplearning\n - https://github.com/yg19930918/deep-learning-from-scratch-master\n - https://adventuresinmachinelearning.com/keras-tutorial-cnn-11-lines/\n - https://zhangting2020.github.io/2018/05/30/Transform-Invariance/\n - https://zh.gluon.ai/chapter_convolutional-neural-networks/lenet.html\n - https://zhuanlan.zhihu.com/p/32702031\n - https://blog.csdn.net/marsjhao/article/details/73088850\n\n- 实践方法论\n - https://github.com/masakazu-ishihata/BayesianOptimization\n - https://github.com/bjzhao143/MLwithPython\n - https://medium.com/inveterate-learner/deep-learning-book-chapter-11-c6ad1d3c3c08\n - https://www.alexejgossmann.com/auc/\n - https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-imbalanced-classification/\n - https://www.yuque.com/books/share/f4031f65-70c1-4909-ba01-c47c31398466/kqbfug\n - http://bridg.land/posts/gaussian-processes-1\n - https://zhuanlan.zhihu.com/p/76269142\n \n"
},
{
"path": "update.txt",
"content": "**更新记录**:\n\n2020/3/：\n\n \t1. 修改第五章决策树部分，补充 ID3 和 CART 的原理，代码实现以 CART 为主。\n \t2. 第七章添加 L1 和 L2 正则化最优解的推导 (即 L1稀疏解的原理)。\n \t3. 第七章添加集成学习方法的推导与代码实现，包括 Bagging (随机森林)、Boosting (Adaboost、GBDT、XGBoost)\n \t4. 第八章添加牛顿法与拟牛顿法 (DFP、BFGS、L-BFGS) 的推导。\n \t5. 第十一章节添加高斯过程回归 (GPR) 与贝叶斯优化的推导与代码实现。\n\n"
}
]