Repository: MingchaoZhu/DeepLearning Branch: master Commit: c433d6c4acbb Files: 16 Total size: 140.3 KB Directory structure: gitextract_8qsunllr/ ├── .gitattributes ├── LICENSE ├── README.md ├── code/ │ ├── chapter 11.py │ ├── chapter5.py │ ├── chapter6.py │ ├── chapter7.py │ ├── chapter8.py │ ├── chapter9.py │ └── method/ │ ├── __init__.py │ ├── activation/ │ │ └── activation.py │ ├── optimizer/ │ │ └── optimizer.py │ └── weight/ │ └── weight.py ├── contents.txt ├── reference.txt └── update.txt ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitattributes ================================================ *.txt linguist-language=python ================================================ FILE: LICENSE ================================================ MIT License Copyright (c) 2020 Mingchao Zhu Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ # Deep Learning 《**深度学习**》是深度学习领域唯一的综合性图书，全称也叫做**深度学习 AI圣经(Deep Learning)**，由三位全球知名专家IanGoodfellow、YoshuaBengio、AaronCourville编著，全书囊括了数学及相关概念的背景知识，包括线性代数、概率论、信息论、数值优化以及机器学习中的相关内容。同时，它还介绍了工业界中实践者用到的深度学习技术，包括深度前馈网络、正则化、优化算法、卷积网络、序列建模和实践方法等，并且调研了诸如自然语言处理、语音识别、计算机视觉、在线推荐系统、生物信息学以及视频游戏方面的应用。最后，深度学习全书还提供了一些研究方向，涵盖的理论主题包括线性因子模型、自编码器、表示学习、结构化概率模型、蒙特卡罗方法、配分函数、近似推断以及深度生成模型，适用于相关专业的大学生或研究生使用。深度学习封面

可以下载《深度学习》的中文版 [pdf](https://github.com/MingchaoZhu/DeepLearning/releases/download/v0.0.1/DL_cn.pdf) 和英文版 [pdf](https://github.com/MingchaoZhu/DeepLearning/releases/download/v0.0.0/DL_en.pdf) 直接阅读。对于本项目的工作，你可以直接下载 [深度学习_原理与代码实现.pdf](https://github.com/MingchaoZhu/DeepLearning/releases/download/v1.1.1/default.pdf) (后面会对该书不断更新) --- 《深度学习》可以说是深度学习与人工智能的入门宝典，许多算法爱好者、机器学习培训班、互联网企业的面试，很多都参考这本书。但本书晦涩，加上官方没有提供代码实现，因此某些地方较难理解。本项目**基于数学推导和产生原理重新描述了书中的概念**，并用**Python** (numpy 库为主) 复现了书本内容 ( **源码级代码实现。推导过程和代码实现均放在了下载区的 pdf 文件中**，重要部分的实现代码也放入 **code 文件夹**中 )。然而我水平有限，但我真诚地希望这项工作可以帮助到更多人学习深度学习算法。我需要大家的建议和帮助。如果你在阅读中遇到有误或解释不清的地方，希望可以汇总你的建议，在 Issues 提出。如果你也想加入这项工作书写中或有其他问题，可以联系我的邮箱。如果你在你的工作或博客中用到了本书，还请可以注明引用链接。写的过程中参考了较多网上优秀的工作，所有参考资源保存在了`reference.txt`文件中。 # 留言这份工作就是在写这一本 [深度学习_原理与代码实现.pdf](https://github.com/MingchaoZhu/DeepLearning/releases/download/v1.1.1/default.pdf)。正如你在 pdf 文件中所见到的，《深度学习》涉及到的每一个概念，都会去给它详细的描述、原理层面的推导，以及用代码的实现。代码实现不会调用 Tensorflow、PyTorch、MXNet 等任何深度学习框架，甚至包括 sklearn (pdf 里用到 sklearn 的部分都是用来验证代码无误)，一切代码都是从原理层面实现 (Python 的基础库 NumPy)，并有详细注释，与代码区上方的原理描述区一致，你可以结合原理和代码一起理解。这份工作的起因是我自身的热爱，但为完成这份工作我需要投入大量的时间精力，一般会写到凌晨两三点。推导、代码、作图都是慢慢打磨的，我会保证这份工作的质量。这份工作会一直更新完，已经上传的章节也会继续补充内容。如果你在阅读过程中遇到有想要描述的概念点或者错误点，请发邮件告知我。真的很感谢你的认可与推广。最后，请等待下一次更新。我是朱明超，我的邮箱是：deityrayleigh@gmail.com # 更新说明 2020/3/： ```python 1. 修改第五章决策树部分，补充 ID3 和 CART 的原理，代码实现以 CART 为主。 2. 第七章添加 L1 和 L2 正则化最优解的推导 (即 L1稀疏解的原理)。 3. 第七章添加集成学习方法的推导与代码实现，包括 Bagging (随机森林)、Boosting (Adaboost、GBDT、XGBoost)。 4. 第八章添加牛顿法与拟牛顿法 (DFP、BFGS、L-BFGS) 的推导。 5. 第十一章节添加贝叶斯线性回归、高斯过程回归 (GPR) 与贝叶斯优化的推导与代码实现。 ``` 后面每次的更新内容会统一放在 `update.txt` 文件中。 # 章节目录与文件下载除了《深度学习》书中的概念点，**本项目也在各章节添加一些补充知识，例如第七章集成学习部分的随机森林、Adaboost、GBDT、XGBoost 的原理剖析和代码实现等，又或者第十二章对当前一些主流方法的描述**。大的章节目录和 pdf 文件下载链接可以详见下表，而具体 pdf 文件中的实际目录请参考 `contents.txt`。你可以在下面的 pdf 链接中下载对应章节，也可以在 [releases](https://github.com/MingchaoZhu/DeepLearning/releases) 界面直接下载所有文件。 | 中文章节 | 英文章节 | 下载
(含推导与代码实现) | | ------------ | ------------ | ------------ | | 第一章前言 | 1 Introduction | | | 第二章线性代数 | 2 Linear Algebra | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/2%20%E7%BA%BF%E6%80%A7%E4%BB%A3%E6%95%B0.pdf) | | 第三章概率与信息论 | 3 Probability and Information Theory | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/3%20%E6%A6%82%E7%8E%87%E4%B8%8E%E4%BF%A1%E6%81%AF%E8%AE%BA.pdf) | | 第四章数值计算 | 4 Numerical Computation | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/4%20%E6%95%B0%E5%80%BC%E8%AE%A1%E7%AE%97.pdf) | | 第五章机器学习基础 | 5 Machine Learning Basics | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/5%20%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E5%9F%BA%E7%A1%80.pdf) | | 第六章深度前馈网络 | 6 Deep Feedforward Networks | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/6%20%E6%B7%B1%E5%BA%A6%E5%89%8D%E9%A6%88%E7%BD%91%E7%BB%9C.pdf) | | 第七章深度学习中的正则化 | 7 Regularization for Deep Learning | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/7%20%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E4%B8%AD%E7%9A%84%E6%AD%A3%E5%88%99%E5%8C%96.pdf) | | 第八章深度模型中的优化 | 8 Optimization for Training Deep Models | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/8%20%E6%B7%B1%E5%BA%A6%E6%A8%A1%E5%9E%8B%E4%B8%AD%E7%9A%84%E4%BC%98%E5%8C%96.pdf) | | 第九章卷积网络 | 9 Convolutional Networks | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/9%20%E5%8D%B7%E7%A7%AF%E7%BD%91%E7%BB%9C.pdf) | | 第十章序列建模：循环和递归网络 | 10 Sequence Modeling: Recurrent and Recursive Nets | | | 第十一章实践方法论 | 11 Practical Methodology | [pdf](https://github.com/MingchaoZhu/DeepLearning/raw/master/11%20%E5%AE%9E%E8%B7%B5%E6%96%B9%E6%B3%95%E8%AE%BA.pdf) | | 第十二章应用 | 12 Applications | | | 第十三章线性因子模型 | 13 Linear Factor Models | | | 第十四章自编码器 | 14 Autoencoders | | | 第十五章表示学习 | 15 Representation Learning | | | 第十六章深度学习中的结构化概率模型 | 16 Structured Probabilistic Models for Deep Learning | | | 第十七章蒙特卡罗方法 | 17 Monte Carlo Methods | | | 第十八章直面配分函数 | 18 Confronting the Partition Function | | | 第十九章近似推断 | 19 Approximate Inference | | | 第二十章深度生成模型 | 20 Deep Generative Models | | 尚未上传的章节会在后续陆续上传。 # 致谢感谢对本项目的认可和推广。 + 专知：https://mp.weixin.qq.com/s/dVD-vKJsMGqnBz2v4O-Q3Q + GitHubDaily：https://m.weibo.cn/5722964389/4504392843690487 + 程序员遇见GitHub：https://mp.weixin.qq.com/s/EzFOnwpkv7mr2TSjPtVG9A + 爱可可：https://m.weibo.cn/1402400261/4503389646699745 # 赞助本项目书写耗费时间精力。如果本项目对你有帮助，可以请作者吃份冰淇淋：

================================================ FILE: code/chapter 11.py ================================================ import pandas as pd import numpy as np import itertools import time import re from scipy.stats import norm import matplotlib.pyplot as plt def cal_conf_matrix(labels, preds): """ 计算混淆矩阵。参数说明： labels：样本标签 (真实结果) preds：预测结果 """ n_sample = len(labels) result = pd.DataFrame(index=range(0,n_sample),columns=('probability','label')) result['label'] = np.array(labels) result['probability'] = np.array(preds) cm = np.arange(4).reshape(2,2) cm[0,0] = len(result[result['label']==1][result['probability']>=0.5]) # TP，注意这里是以 0.5 为阈值 cm[0,1] = len(result[result['label']==1][result['probability']<0.5]) # FN cm[1,0] = len(result[result['label']==0][result['probability']>=0.5]) # FP cm[1,1] = len(result[result['label']==0][result['probability']<0.5]) # TN return cm def cal_PRF1(labels, preds): """ 计算查准率P，查全率R，F1值。 """ cm = cal_conf_matrix(labels, preds) P = cm[0,0]/(cm[0,0]+cm[1,0]) R = cm[0,0]/(cm[0,0]+cm[0,1]) F1 = 2*P*R/(P+R) return P, R, F1 def cal_PRcurve(labels, preds): """ 计算PR曲线上的值。 """ n_sample = len(labels) result = pd.DataFrame(index=range(0,n_sample),columns=('probability','label')) y_pred[y_pred>=0.5] = 1 y_pred[y_pred<0.5] = 0 result['label'] = np.array(labels) result['probability'] = np.array(preds) result.sort_values('probability',inplace=True,ascending=False) PandR = pd.DataFrame(index=range(len(labels)),columns=('P','R')) for j in range(len(result)): # 以每一个概率为分类的阈值，统计此时正例和反例的数量 result_j = result.head(n=j+1) P = len(result_j[result_j['label']==1])/float(len(result_j)) # 当前实际为正的数量/当前预测为正的数量 R = len(result_j[result_j['label']==1])/float(len(result[result['label']==1])) # 当前真正例的数量/实际为正的数量 PandR.iloc[j] = [P,R] return PandR def cal_ROCcurve(labels, preds): """ 计算ROC曲线上的值。 """ n_sample = len(labels) result = pd.DataFrame(index=range(0,n_sample),columns=('probability','label')) y_pred[y_pred>=0.5] = 1 y_pred[y_pred<0.5] = 0 result['label'] = np.array(labels) result['probability'] = np.array(preds) # 计算 TPR,FPR result.sort_values('probability',inplace=True,ascending=False) TPRandFPR=pd.DataFrame(index=range(len(result)),columns=('TPR','FPR')) for j in range(len(result)): # 以每一个概率为分类的阈值，统计此时正例和反例的数量 result_j=result.head(n=j+1) TPR=len(result_j[result_j['label']==1])/float(len(result[result['label']==1])) # 当前真正例的数量/实际为正的数量 FPR=len(result_j[result_j['label']==0])/float(len(result[result['label']==0])) # 当前假正例的数量/实际为负的数量 TPRandFPR.iloc[j]=[TPR,FPR] return TPRandFPR def timeit(func): """ 装饰器，计算函数执行时间 """ def wrapper(*args, **kwargs): time_start = time.time() result = func(*args, **kwargs) time_end = time.time() exec_time = time_end - time_start print("{function} exec time: {time}s".format(function=func.__name__,time=exec_time)) return result return wrapper @timeit def area_auc(labels, preds): """ AUC值的梯度法计算 """ TPRandFPR = cal_ROCcurve(labels, preds) # 计算AUC，计算小矩形的面积之和 auc = 0. prev_x = 0 for x, y in zip(TPRandFPR.FPR,TPRandFPR.TPR): if x != prev_x: auc += (x - prev_x) * y prev_x = x return auc @timeit def naive_auc(labels, preds): """ AUC值的概率法计算 """ n_pos = sum(labels) n_neg = len(labels) - n_pos total_pair = n_pos * n_neg # 总的正负样本对的数目 labels_preds = zip(labels, preds) labels_preds = sorted(labels_preds,key=lambda x:x[1]) # 对预测概率升序排序 count_neg = 0 # 统计负样本出现的个数 satisfied_pair = 0 # 统计满足条件的样本对的个数 for i in range(len(labels_preds)): if labels_preds[i][0] == 1: satisfied_pair += count_neg # 表明在这个正样本下，有哪些负样本满足条件 else: count_neg += 1 return satisfied_pair / float(total_pair) #####----Bayesian Hyperparameter Optimization----#### class KernelBase(ABC): def __init__(self): super().__init__() self.params = {} self.hyperparams = {} @abstractmethod def _kernel(self, X, Y): raise NotImplementedError def __call__(self, X, Y=None): return self._kernel(X, Y) def __str__(self): P, H = self.params, self.hyperparams p_str = ", ".join(["{}={}".format(k, v) for k, v in P.items()]) return "{}({})".format(H["op"], p_str) def summary(self): return { "op": self.hyperparams["op"], "params": self.params, "hyperparams": self.hyperparams, } class RBFKernel(KernelBase): def __init__(self, sigma=None): """ RBF 核。 """ super().__init__() self.hyperparams = {"op": "RBFKernel"} self.params = {"sigma": sigma} # 如果 sigma 未赋值则默认为 np.sqrt(n_features/2)，n_features 为特征数。 def _kernel(self, X, Y=None): """ 对 X 和 Y 的行的每一对计算 RBF 核。如果 Y 为空，则 Y=X。参数说明： X：输入数组，为 (n_samples, n_features) Y：输入数组，为 (m_samples, n_features) """ X = X.reshape(-1, 1) if X.ndim == 1 else X Y = X if Y is None else Y Y = Y.reshape(-1, 1) if Y.ndim == 1 else Y assert X.ndim == 2 and Y.ndim == 2, "X and Y must have 2 dimensions" sigma = np.sqrt(X.shape[1] / 2) if self.params["sigma"] is None else self.params["sigma"] X, Y = X / sigma, Y / sigma D = -2 * X @ Y.T + np.sum(Y**2, axis=1) + np.sum(X**2, axis=1)[:, np.newaxis] D[D < 0] = 0 return np.exp(-0.5 * D) class KernelInitializer(object): def __init__(self, param=None): self.param = param def __call__(self): r = r"([a-zA-Z0-9]*)=([^,)]*)" kr_str = self.param.lower() kwargs = dict([(i, eval(j)) for (i, j) in re.findall(r, self.param)]) if "rbf" in kr_str: kernel = RBFKernel(**kwargs) else: raise NotImplementedError("{}".format(kr_str)) return kernel class GPRegression: """ 高斯过程回归 """ def __init__(self, kernel="RBFKernel", sigma=1e-10): self.kernel = KernelInitializer(kernel)() self.params = {"GP_mean": None, "GP_cov": None, "X": None} self.hyperparams = {"kernel": str(self.kernel), "sigma": sigma} def fit(self, X, y): """ 用已有的样本集合得到 GP 先验。参数说明： X：输入数组，为 (n_samples, n_features) y：输入数组 X 的目标值，为 (n_samples) """ mu = np.zeros(X.shape[0]) Cov = self.kernel(X, X) self.params["X"] = X self.params["y"] = y self.params["GP_cov"] = Cov self.params["GP_mean"] = mu def predict(self, X_star, conf_interval=0.95): """ 对新的样本 X 进行预测。参数说明： X_star：输入数组，为 (n_samples, n_features) conf_interval：置信区间，浮点型 (0, 1)，default=0.95 """ X = self.params["X"] y = self.params["y"] K = self.params["GP_cov"] sigma = self.hyperparams["sigma"] K_star = self.kernel(X_star, X) K_star_star = self.kernel(X_star, X_star) sig = np.eye(K.shape[0]) * sigma K_y_inv = np.linalg.pinv(K + sig) mean = K_star @ K_y_inv @ y cov = K_star_star - K_star @ K_y_inv @ K_star.T percentile = norm.ppf(conf_interval) conf = percentile * np.sqrt(np.diag(cov)) return mean, conf, cov class BayesianOptimization: def __init__(self): self.model = GPRegression() def acquisition_function(self, Xsamples): mu, _, cov = self.model.predict(Xsamples) mu = mu if mu.ndim==1 else (mu.T)[0] ysample = np.random.multivariate_normal(mu, cov) return ysample def opt_acquisition(self, X, n_samples=20): # 样本搜索策略，一般方法有随机搜索、基于网格的搜索，或局部搜索 # 我们这里就用简单的随机搜索，这里也可以定义样本的范围 Xsamples = np.random.randint(low=1,high=50,size=n_samples*X.shape[1]) Xsamples = Xsamples.reshape(n_samples, X.shape[1]) # 计算采集函数的值并取最大的值 scores = self.acquisition_function(Xsamples) ix = np.argmax(scores) return Xsamples[ix, 0] def fit(self, f, X, y): # 拟合 GPR 模型 self.model.fit(X, y) # 优化过程 for i in range(15): x_star = self.opt_acquisition(X) # 下一个采样点 y_star = f(x_star) mean, conf, cov = self.model.predict(np.array([[x_star]])) # 添加当前数据到数据集合 X = np.vstack((X, [[x_star]])) y = np.vstack((y, [[y_star]])) # 更新 GPR 模型 self.model.fit(X, y) ix = np.argmax(y) print('Best Result: x=%.3f, y=%.3f' % (X[ix], y[ix])) return X[ix], y[ix] ================================================ FILE: code/chapter5.py ================================================ import numpy as np import cvxopt import math ########-----NaiveBayes------######### class NaiveBayes(): def __init__(self): self.parameters = [] # 保存每个特征针对每个类的均值和方差 self.y = None self.classes = None def fit(self, X, y): self.y = y self.classes = np.unique(y) # 类别 # 计算每个特征针对每个类的均值和方差 for i, c in enumerate(self.classes): # 选择类别为c的X X_where_c = X[np.where(self.y == c)] self.parameters.append([]) # 添加均值与方差 for col in X_where_c.T: parameters = {"mean": col.mean(), "var": col.var()} self.parameters[i].append(parameters) def _calculate_prior(self, c): """ 先验函数。 """ frequency = np.mean(self.y == c) return frequency def _calculate_likelihood(self, mean, var, X): """ 似然函数。 """ # 高斯概率 eps = 1e-4 # 防止除数为0 coeff = 1.0 / math.sqrt(2.0 * math.pi * var + eps) exponent = math.exp(-(math.pow(X - mean, 2) / (2 * var + eps))) return coeff * exponent def _calculate_probabilities(self, X): posteriors = [] for i, c in enumerate(self.classes): posterior = self._calculate_prior(c) for feature_value, params in zip(X, self.parameters[i]): # 独立性假设 # P(x1,x2|Y) = P(x1|Y)*P(x2|Y) likelihood = self._calculate_likelihood(params["mean"], params["var"], feature_value) posterior *= likelihood posteriors.append(posterior) # 返回具有最大后验概率的类别 return self.classes[np.argmax(posteriors)] def predict(self, X): y_pred = [self._calculate_probabilities(sample) for sample in X] return y_pred def score(self, X, y): y_pred = self.predict(X) accuracy = np.sum(y == y_pred, axis=0) / len(y) return accuracy ########-----LogisticRegression------######### def Sigmoid(x): return 1/(1 + np.exp(-x)) class LogisticRegression(): def __init__(self, learning_rate=.1): self.param = None self.learning_rate = learning_rate self.sigmoid = Sigmoid def _initialize_parameters(self, X): n_features = np.shape(X)[1] # 初始化参数theta， [-1/sqrt(N), 1/sqrt(N)] limit = 1 / math.sqrt(n_features) self.param = np.random.uniform(-limit, limit, (n_features,)) def fit(self, X, y, n_iterations=4000): self._initialize_parameters(X) # 参数theta的迭代更新 for i in range(n_iterations): # 求预测 y_pred = self.sigmoid(X.dot(self.param)) # 最小化损失函数，参数更新公式 self.param -= self.learning_rate * -(y - y_pred).dot(X) def predict(self, X): y_pred = self.sigmoid(X.dot(self.param)) return y_pred def score(self, X, y): y_pred = self.predict(X) accuracy = np.sum(y == y_pred, axis=0) / len(y) return accuracy ########-----SupportVectorMachine------######### # 隐藏cvxopt输出 cvxopt.solvers.options['show_progress'] = False def linear_kernel(**kwargs): """ 线性核 """ def f(x1, x2): return np.inner(x1, x2) return f def polynomial_kernel(power, coef, **kwargs): """ 多项式核 """ def f(x1, x2): return (np.inner(x1, x2) + coef)**power return f def rbf_kernel(gamma, **kwargs): """ 高斯核 """ def f(x1, x2): distance = np.linalg.norm(x1 - x2) ** 2 return np.exp(-gamma * distance) return f class SupportVectorMachine(): def __init__(self, kernel=linear_kernel, power=4, gamma=None, coef=4): self.kernel = kernel self.power = power self.gamma = gamma self.coef = coef self.lagr_multipliers = None self.support_vectors = None self.support_vector_labels = None self.intercept = None def fit(self, X, y): n_samples, n_features = np.shape(X) # gamma默认设置为1 / n_features if not self.gamma: self.gamma = 1 / n_features # 定义核函数 self.kernel = self.kernel( power=self.power, gamma=self.gamma, coef=self.coef) # 计算Gram矩阵 kernel_matrix = np.zeros((n_samples, n_samples)) for i in range(n_samples): for j in range(n_samples): kernel_matrix[i, j] = self.kernel(X[i], X[j]) # 构造二次规划问题 # 形式为 min (1/2)x.T*P*x+q.T*x, s.t. G*x<=h, A*x=b P = cvxopt.matrix(np.outer(y, y) * kernel_matrix, tc='d') q = cvxopt.matrix(np.ones(n_samples) * -1) A = cvxopt.matrix(y, (1, n_samples), tc='d') b = cvxopt.matrix(0, tc='d') G = cvxopt.matrix(np.identity(n_samples) * -1) h = cvxopt.matrix(np.zeros(n_samples)) # 用cvxopt求解二次规划问题 minimization = cvxopt.solvers.qp(P, q, G, h, A, b) lagr_mult = np.ravel(minimization['x']) # 非0的alpha值 idx = lagr_mult > 1e-7 # alpha值 self.lagr_multipliers = lagr_mult[idx] # 支持向量 self.support_vectors = X[idx] # 支持向量的标签 self.support_vector_labels = y[idx] # 通过第一个支持向量计算b self.intercept = self.support_vector_labels[0] for i in range(len(self.lagr_multipliers)): self.intercept -= self.lagr_multipliers[i] * self.support_vector_labels[ i] * self.kernel(self.support_vectors[i], self.support_vectors[0]) def predict(self, X): y_pred = [] for sample in X: # 对于输入的x, 计算f(x) prediction = 0 for i in range(len(self.lagr_multipliers)): prediction += self.lagr_multipliers[i] * self.support_vector_labels[ i] * self.kernel(self.support_vectors[i], sample) prediction += self.intercept y_pred.append(np.sign(prediction)) return np.array(y_pred) def score(self, X, y): y_pred = self.predict(X) accuracy = np.sum(y == y_pred, axis=0) / len(y) return accuracy ########-----KNN------######### class KNN(): def __init__(self, k=10): self._k = k def fit(self, X, y): self._unique_labels = np.unique(y) self._class_num = len(self._unique_labels) self._datas = X self._labels = y.astype(np.int32) def predict(self, X): # 欧式距离计算 dist = np.sum(np.square(X), axis=1, keepdims=True) - 2 * np.dot(X, self._datas.T) dist = dist + np.sum(np.square(self._datas), axis=1, keepdims=True).T dist = np.argsort(dist)[:,:self._k] return np.array([np.argmax(np.bincount(self._labels[dist][i])) for i in range(len(X))]) idx = lagr_mult > 1e-7 # alpha值 self.lagr_multipliers = lagr_mult[idx] # 支持向量 self.support_vectors = X[idx] # 支持向量的标签 self.support_vector_labels = y[idx] # 通过第一个支持向量计算b self.intercept = self.support_vector_labels[0] for i in range(len(self.lagr_multipliers)): self.intercept -= self.lagr_multipliers[i] * self.support_vector_labels[ i] * self.kernel(self.support_vectors[i], self.support_vectors[0]) def predict(self, X): y_pred = [] for sample in X: # 对于输入的x, 计算f(x) prediction = 0 for i in range(len(self.lagr_multipliers)): prediction += self.lagr_multipliers[i] * self.support_vector_labels[ i] * self.kernel(self.support_vectors[i], sample) prediction += self.intercept y_pred.append(np.sign(prediction)) return np.array(y_pred) def score(self, X, y): y_pred = self.predict(X) accuracy = np.sum(y == y_pred, axis=0) / len(y) return accuracy ########-----DecisionTree------######### class DecisionNode(): def __init__(self, feature_i=None, threshold=None, value=None, true_branch=None, false_branch=None): self.feature_i = feature_i # 当前结点测试的特征的索引 self.threshold = threshold # 当前结点测试的特征的阈值 self.value = value # 结点值（如果结点为叶子结点） self.true_branch = true_branch # 左子树（满足阈值，将特征值大于等于切分点值的数据划分为左子树） self.false_branch = false_branch # 右子树（未满足阈值，将特征值小于切分点值的数据划分为右子树） def divide_on_feature(X, feature_i, threshold): """ 依据切分变量和切分点，将数据集分为两个子区域 """ split_func = None if isinstance(threshold, int) or isinstance(threshold, float): split_func = lambda sample: sample[feature_i] >= threshold else: split_func = lambda sample: sample[feature_i] == threshold X_1 = np.array([sample for sample in X if split_func(sample)]) X_2 = np.array([sample for sample in X if not split_func(sample)]) return np.array([X_1, X_2]) class DecisionTree(object): def __init__(self, min_samples_split=2, min_impurity=1e-7, max_depth=float("inf"), loss=None): self.root = None # 根结点 self.min_samples_split = min_samples_split # 满足切分的最少样本数 self.min_impurity = min_impurity # 满足切分的最小纯度 self.max_depth = max_depth # 树的最大深度 self._impurity_calculation = None # 计算纯度的函数，如对于分类树采用信息增益 self._leaf_value_calculation = None # 计算y在叶子结点值的函数 self.one_dim = None # y是否为one-hot编码 def fit(self, X, y): self.one_dim = len(np.shape(y)) == 1 self.root = self._build_tree(X, y) def _build_tree(self, X, y, current_depth=0): """ 递归方法建立决策树 """ largest_impurity = 0 best_criteria = None # 当前最优分类的特征索引和阈值 best_sets = None # 数据子集 if len(np.shape(y)) == 1: y = np.expand_dims(y, axis=1) Xy = np.concatenate((X, y), axis=1) n_samples, n_features = np.shape(X) if n_samples >= self.min_samples_split and current_depth <= self.max_depth: # 对每个特征计算纯度 for feature_i in range(n_features): feature_values = np.expand_dims(X[:, feature_i], axis=1) unique_values = np.unique(feature_values) # 遍历特征i所有的可能值找到最优纯度 for threshold in unique_values: # 基于X在特征i处是否满足阈值来划分X和y， Xy1为满足阈值的子集 Xy1, Xy2 = divide_on_feature(Xy, feature_i, threshold) if len(Xy1) > 0 and len(Xy2) > 0: # 取出Xy中y的集合 y1 = Xy1[:, n_features:] y2 = Xy2[:, n_features:] # 计算纯度 impurity = self._impurity_calculation(y, y1, y2) # 如果纯度更高，则更新 if impurity > largest_impurity: largest_impurity = impurity best_criteria = {"feature_i": feature_i, "threshold": threshold} best_sets = { "leftX": Xy1[:, :n_features], # X的左子树 "lefty": Xy1[:, n_features:], # y的左子树 "rightX": Xy2[:, :n_features], # X的右子树 "righty": Xy2[:, n_features:] # y的右子树 } if largest_impurity > self.min_impurity: # 建立左子树和右子树 true_branch = self._build_tree(best_sets["leftX"], best_sets["lefty"], current_depth + 1) false_branch = self._build_tree(best_sets["rightX"], best_sets["righty"], current_depth + 1) return DecisionNode(feature_i=best_criteria["feature_i"], threshold=best_criteria[ "threshold"], true_branch=true_branch, false_branch=false_branch) # 如果是叶结点则计算值 leaf_value = self._leaf_value_calculation(y) return DecisionNode(value=leaf_value) def predict_value(self, x, tree=None): """ 预测样本，沿着树递归搜索 """ # 根结点 if tree is None: tree = self.root # 递归出口 if tree.value is not None: return tree.value # 选择当前结点的特征 feature_value = x[tree.feature_i] branch = tree.false_branch if isinstance(feature_value, int) or isinstance(feature_value, float): if feature_value >= tree.threshold: branch = tree.true_branch elif feature_value == tree.threshold: branch = tree.true_branch return self.predict_value(x, branch) def predict(self, X): y_pred = [self.predict_value(sample) for sample in X] return y_pred def score(self, X, y): y_pred = self.predict(X) accuracy = np.sum(y == y_pred, axis=0) / len(y) return accuracy def print_tree(self, tree=None, indent=" "): """ 输出树 """ if not tree: tree = self.root if tree.value is not None: print(tree.value) else: print("feature|threshold -> %s | %s" % (tree.feature_i, tree.threshold)) print("%sT->" % (indent), end="") self.print_tree(tree.true_branch, indent + indent) print("%sF->" % (indent), end="") self.print_tree(tree.false_branch, indent + indent) def calculate_entropy(y): log2 = lambda x: math.log(x) / math.log(2) unique_labels = np.unique(y) entropy = 0 for label in unique_labels: count = len(y[y == label]) p = count / len(y) entropy += -p * log2(p) return entropy def calculate_gini(y): unique_labels = np.unique(y) var = 0 for label in unique_labels: count = len(y[y == label]) p = count / len(y) var += p ** 2 return 1 - var class ClassificationTree(DecisionTree): """ 分类树，在决策书节点选择计算信息增益/基尼指数，在叶子节点选择多数表决。 """ def _calculate_gini_index(self, y, y1, y2): """ 计算基尼指数 """ p = len(y1) / len(y) gini = calculate_gini(y) gini_index = gini - p * \ calculate_gini(y1) - (1 - p) * \ calculate_gini(y2) return gini_index def _calculate_information_gain(self, y, y1, y2): """ 计算信息增益 """ p = len(y1) / len(y) entropy = calculate_entropy(y) info_gain = entropy - p * \ calculate_entropy(y1) - (1 - p) * \ calculate_entropy(y2) return info_gain def _majority_vote(self, y): """ 多数表决 """ most_common = None max_count = 0 for label in np.unique(y): count = len(y[y == label]) if count > max_count: most_common = label max_count = count return most_common def fit(self, X, y): self._impurity_calculation = self._calculate_gini_index self._leaf_value_calculation = self._majority_vote super(ClassificationTree, self).fit(X, y) def calculate_mse(y): return np.mean((y - np.mean(y)) ** 2) def calculate_variance(y): n_samples = np.shape(y)[0] variance = (1 / n_samples) * np.diag((y - np.mean(y)).T.dot(y - np.mean(y))) return variance class RegressionTree(DecisionTree): """ 回归树，在决策书节点选择计算MSE/方差降低，在叶子节点选择均值。 """ def _calculate_mse(self, y, y1, y2): """ 计算MSE降低 """ mse_tot = calculate_mse(y) mse_1 = calculate_mse(y1) mse_2 = calculate_mse(y2) frac_1 = len(y1) / len(y) frac_2 = len(y2) / len(y) mse_reduction = mse_tot - (frac_1 * mse_1 + frac_2 * mse_2) return mse_reduction def _calculate_variance_reduction(self, y, y1, y2): """ 计算方差降低 """ var_tot = calculate_variance(y) var_1 = calculate_variance(y1) var_2 = calculate_variance(y2) frac_1 = len(y1) / len(y) frac_2 = len(y2) / len(y) variance_reduction = var_tot - (frac_1 * var_1 + frac_2 * var_2) return sum(variance_reduction) def _mean_of_y(self, y): """ 计算均值 """ value = np.mean(y, axis=0) return value if len(value) > 1 else value[0] def fit(self, X, y): self._impurity_calculation = self._calculate_mse self._leaf_value_calculation = self._mean_of_y super(RegressionTree, self).fit(X, y) ########-----PCA------######### class PCA(): def __init__(self): pass def fit(self, X, n_components): n_samples = np.shape(X)[0] covariance_matrix = (1 / (n_samples-1)) * (X - X.mean(axis=0)).T.dot(X - X.mean(axis=0)) # 对协方差矩阵进行特征值分解 eigenvalues, eigenvectors = np.linalg.eig(covariance_matrix) # 对特征值（特征向量）从大到小排序 idx = eigenvalues.argsort()[::-1] eigenvalues = eigenvalues[idx][:n_components] eigenvectors = np.atleast_1d(eigenvectors[:, idx])[:, :n_components] # 得到低维表示 X_transformed = X.dot(eigenvectors) ########-----KMeans------######### def distEclud(x,y): """ 计算欧氏距离 """ return np.sqrt(np.sum((x-y)**2)) def randomCent(dataSet,k): """ 为数据集构建一个包含 K 个随机质心的集合 """ m,n = dataSet.shape centroids = np.zeros((k,n)) for i in range(k): index = int(np.random.uniform(0,m)) centroids[i,:] = dataSet[index,:] return centroids class KMeans(): def __init__(self): self.dataSet = None self.k = None def fit(self, dataSet, k): self.dataSet = dataSet self.k = k m = np.shape(dataSet)[0] # 第一列存样本属于哪一簇 # 第二列存样本的到簇的中心点的误差 clusterAssment = np.mat(np.zeros((m,2))) clusterChange = True centroids = randomCent(self.dataSet,k) while clusterChange: clusterChange = False for i in range(m): minDist = 1e6 minIndex = -1 # 遍历所有的质心, 找出最近的质心 for j in range(k): distance = distEclud(centroids[j,:], self.dataSet[i,:]) if distance < minDist: minDist = distance minIndex = j # 更新每一行样本所属的簇 if clusterAssment[i,0] != minIndex: clusterChange = True clusterAssment[i,:] = minIndex, minDist**2 # 更新质心 for j in range(k): pointsInCluster = dataSet[np.nonzero(clusterAssment[:,0].A == j)[0]] # 获取簇类所有的点 centroids[j,:] = np.mean(pointsInCluster,axis=0) # 对矩阵的行求均值 return centroids,clusterAssment return X_transformed ================================================ FILE: code/chapter6.py ================================================ from abc import ABC, abstractmethod import numpy as np import time import re import inspect from collections import OrderedDict import sys sys.path.append('../') from method.optimizer import OptimizerInitializer from method.weight import WeightInitializer from method.activation import ActivationInitializer def sigmoid(x): return 1 / (1 + np.exp(-x)) def softmax(x): e_x = np.exp(x - np.max(x, axis=-1, keepdims=True)) return e_x / e_x.sum(axis=-1, keepdims=True) class LayerBase(ABC): def __init__(self, optimizer="sgd"): self.X = [] # 网络层输入 self.gradients = {} # 网络层待梯度更新变量 self.params = {} # 网络层参数变量 self.acti_fn = None # 网络层激活函数 self.optimizer = OptimizerInitializer(optimizer)() # 网络层优化方法 @abstractmethod def _init_params(self, **kwargs): """ 函数作用：初始化参数 """ raise NotImplementedError @abstractmethod def forward(self, X, **kwargs): """ 函数作用：前向传播 """ raise NotImplementedError @abstractmethod def backward(self, out, **kwargs): """ 函数作用：反向传播 """ raise NotImplementedError def flush_gradients(self): """ 函数作用：重置更新参数列表 """ self.X = [] for k, v in self.gradients.items(): self.gradients[k] = np.zeros_like(v) for k, v in self.derived_variables.items(): self.derived_variables[k] = [] def update(self): """ 函数作用：更新参数 """ for k, v in self.gradients.items(): if k in self.params: self.params[k] = self.optimizer(self.params[k], v, k) class FullyConnected(LayerBase): """ 定义全连接层，实现 a=g(x*W+b)，前向传播输入x，返回a；反向传播输入 """ def __init__(self, n_out, acti_fn, init_w, optimizer=None): """ 参数说明： acti_fn：激活函数， str型 init_w：权重初始化方法， str型 n_out：隐藏层输出维数 optimizer：优化方法 """ super().__init__(optimizer) self.n_in = None # 隐藏层输入维数， int型 self.n_out = n_out # 隐藏层输出维数， int型 self.acti_fn = ActivationInitializer(acti_fn)() self.init_w = init_w self.init_weights = WeightInitializer(mode=init_w) self.is_initialized = False # 是否初始化， bool型变量 def _init_params(self): b = np.zeros((1, self.n_out)) W = self.init_weights((self.n_in, self.n_out)) self.params = {"W": W, "b": b} self.gradients = {"W": np.zeros_like(W), "b": np.zeros_like(b)} self.derived_variables = {"Z": []} self.is_initialized = True def forward(self, X, retain_derived=True): """ 全连接网络的前向传播，原理见上文反向传播算法部分。参数说明： X：输入数组，为（n_samples, n_in），float型 retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型 """ if not self.is_initialized: # 如果参数未初始化，先初始化参数 self.n_in = X.shape[1] self._init_params() W = self.params["W"] b = self.params["b"] z = X @ W + b a = self.acti_fn.forward(z) if retain_derived: self.X.append(X) return a def backward(self, dLda, retain_grads=True): """ 全连接网络的反向传播，原理见上文反向传播算法部分。参数说明： dLda：关于损失的梯度，为（n_samples, n_out），float型 retain_grads：是否计算中间变量的参数梯度，bool型 """ if not isinstance(dLda, list): dLda = [dLda] dX = [] X = self.X for da, x in zip(dLda, X): dx, dw, db = self._bwd(da, x) dX.append(dx) if retain_grads: self.gradients["W"] += dw self.gradients["b"] += db return dX[0] if len(X) == 1 else dX def _bwd(self, dLda, X): W = self.params["W"] b = self.params["b"] Z = X @ W + b dZ = dLda * self.acti_fn.grad(Z) dX = dZ @ W.T dW = X.T @ dZ db = dZ.sum(axis=0, keepdims=True) return dX, dW, db @property def hyperparams(self): return { "layer": "FullyConnected", "init_w": self.init_w, "n_in": self.n_in, "n_out": self.n_out, "acti_fn": str(self.acti_fn), "optimizer": { "hyperparams": self.optimizer.hyperparams, }, "components": { k: v for k, v in self.params.items() } } class ObjectiveBase(ABC): def __init__(self): super().__init__() @abstractmethod def loss(self, y_true, y_pred): """ 函数作用：计算损失 """ raise NotImplementedError @abstractmethod def grad(self, y_true, y_pred, **kwargs): """ 函数作用：计算代价函数的梯度 """ raise NotImplementedError class SquaredError(ObjectiveBase): """ 二次代价函数。 """ def __init__(self): super().__init__() def __call__(self, y_true, y_pred): return self.loss(y_true, y_pred) def __str__(self): return "SquaredError" @staticmethod def loss(y_true, y_pred): """ 参数说明： y_true：训练的 n 个样本的真实值，形状为(n,m)数组； y_pred：训练的 n 个样本的预测值，形状为(n,m)数组； """ (n, _) = y_true.shape return 0.5 * np.linalg.norm(y_pred - y_true) ** 2 / n @staticmethod def grad(y_true, y_pred, z, acti_fn): (n, _) = y_true.shape return (y_pred - y_true) * acti_fn.grad(z) / n class CrossEntropy(ObjectiveBase): """ 交叉熵代价函数。 """ def __init__(self): super().__init__() def __call__(self, y_true, y_pred): return self.loss(y_true, y_pred) def __str__(self): return "CrossEntropy" @staticmethod def loss(y_true, y_pred): """ 参数说明： y_true：训练的 n 个样本的真实值，要求形状为(n,m)二进制（每个样本均为 one-hot 编码）； y_pred：训练的 n 个样本的预测值，形状为(n,m)； """ (n, _) = y_true.shape eps = np.finfo(float).eps # 防止 np.log(0) cross_entropy = -np.sum(y_true * np.log(y_pred + eps)) / n return cross_entropy @staticmethod def grad(y_true, y_pred): (n, _) = y_true.shape grad = (y_pred - y_true) / n return grad def minibatch(X, batchsize=256, shuffle=True): """ 函数作用：将数据集分割成 batch，基于 mini batch 训练。 """ N = X.shape[0] idx = np.arange(N) n_batches = int(np.ceil(N / batchsize)) if shuffle: np.random.shuffle(idx) def mb_generator(): for i in range(n_batches): yield idx[i * batchsize : (i + 1) * batchsize] return mb_generator(), n_batches class DFN(object): def __init__( self, hidden_dims_1=None, hidden_dims_2=None, optimizer="sgd(lr=0.01)", init_w="std_normal", loss=CrossEntropy() ): self.optimizer = optimizer self.init_w = init_w self.loss = loss self.hidden_dims_1 = hidden_dims_1 self.hidden_dims_2 = hidden_dims_2 self.is_initialized = False def _set_params(self): """ 函数作用：模型初始化 FC1 -> Sigmoid -> FC2 -> Softmax """ self.layers = OrderedDict() self.layers["FC1"] = FullyConnected( n_out=self.hidden_dims_1, acti_fn="sigmoid", init_w=self.init_w, optimizer=self.optimizer ) self.layers["FC2"] = FullyConnected( n_out=self.hidden_dims_2, acti_fn="affine(slope=1, intercept=0)", init_w=self.init_w, optimizer=self.optimizer ) self.is_initialized = True def forward(self, X_train): Xs = {} out = X_train for k, v in self.layers.items(): Xs[k] = out out = v.forward(out) return out, Xs def backward(self, grad): dXs = {} out = grad for k, v in reversed(list(self.layers.items())): dXs[k] = out out = v.backward(out) return out, dXs def update(self): """ 函数作用：梯度更新 """ for k, v in reversed(list(self.layers.items())): v.update() self.flush_gradients() def flush_gradients(self, curr_loss=None): """ 函数作用：更新后重置梯度 """ for k, v in self.layers.items(): v.flush_gradients() def fit(self, X_train, y_train, n_epochs=20, batch_size=64, verbose=False, epo_verbose=True): """ 参数说明： X_train：训练数据 y_train：训练数据标签 n_epochs：epoch 次数 batch_size：每次 epoch 的 batch size verbose：是否每个 batch 输出损失 epo_verbose：是否每个 epoch 输出损失 """ self.verbose = verbose self.n_epochs = n_epochs self.batch_size = batch_size if not self.is_initialized: self.n_features = X_train.shape[1] self._set_params() prev_loss = np.inf for i in range(n_epochs): loss, epoch_start = 0.0, time.time() batch_generator, n_batch = minibatch(X_train, self.batch_size, shuffle=True) for j, batch_idx in enumerate(batch_generator): batch_len, batch_start = len(batch_idx), time.time() X_batch, y_batch = X_train[batch_idx], y_train[batch_idx] out, _ = self.forward(X_batch) y_pred_batch = softmax(out) batch_loss = self.loss(y_batch, y_pred_batch) grad = self.loss.grad(y_batch, y_pred_batch) _, _ = self.backward(grad) self.update() loss += batch_loss if self.verbose: fstr = "\t[Batch {}/{}] Train loss: {:.3f} ({:.1f}s/batch)" print(fstr.format(j + 1, n_batch, batch_loss, time.time() - batch_start)) loss /= n_batch if epo_verbose: fstr = "[Epoch {}] Avg. loss: {:.3f} Delta: {:.3f} ({:.2f}m/epoch)" print(fstr.format(i + 1, loss, prev_loss - loss, (time.time() - epoch_start) / 60.0)) prev_loss = loss def evaluate(self, X_test, y_test, batch_size=128): acc = 0.0 batch_generator, n_batch = minibatch(X_test, batch_size, shuffle=True) for j, batch_idx in enumerate(batch_generator): batch_len, batch_start = len(batch_idx), time.time() X_batch, y_batch = X_test[batch_idx], y_test[batch_idx] y_pred_batch, _ = self.forward(X_batch) y_pred_batch = np.argmax(y_pred_batch, axis=1) y_batch = np.argmax(y_batch, axis=1) acc += np.sum(y_pred_batch == y_batch) return acc / X_test.shape[0] @property def hyperparams(self): return { "init_w": self.init_w, "loss": str(self.loss), "optimizer": self.optimizer, "hidden_dims_1": self.hidden_dims_1, "hidden_dims_2": self.hidden_dims_2, "components": {k: v.params for k, v in self.layers.items()} } ================================================ FILE: code/chapter7.py ================================================ from abc import ABC, abstractmethod import numpy as np import math import re import progressbar from chapter5 import RegressionTree, DecisionTree, ClassificationTree #########---Regularizer---###### class RegularizerBase(ABC): def __init__(self, **kwargs): super().__init__() @abstractmethod def loss(self, **kwargs): raise NotImplementedError @abstractmethod def grad(self, **kwargs): raise NotImplementedError class L1Regularizer(RegularizerBase): def __init__(self, lambd=0.001): super().__init__() self.lambd = lambd def loss(self, params): loss = 0 pattern = re.compile(r'^W\d+') for key, val in params.items(): if pattern.match(key): loss += 0.5 * np.sum(np.abs(val)) * self.lambd return loss def grad(self, params): for key, val in params.items(): grad = self.lambd * np.sign(val) return grad class L2Regularizer(RegularizerBase): def __init__(self, lambd=0.001): super().__init__() self.lambd = lambd def loss(self, params): loss = 0 for key, val in params.items(): loss += 0.5 * np.sum(np.square(val)) * self.lambd return loss def grad(self, params): for key, val in params.items(): grad = self.lambd * val return grad class RegularizerInitializer(object): def __init__(self, regular_name="l2"): self.regular_name = regular_name def __call__(self): r = r"([a-zA-Z]*)=([^,)]*)" regular_str = self.regular_name.lower() kwargs = dict([(i, eval(j)) for (i, j) in re.findall(r, regular_str)]) if "l1" in regular_str.lower(): regular = L1Regularizer(**kwargs) elif "l2" in regular_str.lower(): regular = L2Regularizer(**kwargs) else: raise ValueError("Unrecognized regular: {}".format(regular_str)) return regular #######----Dataset Augmentation----#### class Image(object): def __init__(self, image): self._set_params(image) def _set_params(self, image): self.img = image self.row = image.shape[0] # 图像高度 self.col = image.shape[1] # 图像宽度 self.transform = None def Translation(self, delta_x, delta_y): """ 平移。参数说明： delta_x：控制左右平移，若大于0左移，小于0右移 delta_y：控制上下平移，若大于0上移，小于0下移 """ self.transform = np.array([[1, 0, delta_x], [0, 1, delta_y], [0, 0, 1]]) def Resize(self, alpha): """ 缩放。参数说明： alpha：缩放因子，不进行缩放设置为1 """ self.transform = np.array([[alpha, 0, 0], [0, alpha, 0], [0, 0, 1]]) def HorMirror(self): """ 水平镜像。 """ self.transform = np.array([[1, 0, 0], [0, -1, self.col-1], [0, 0, 1]]) def VerMirror(self): """ 垂直镜像。 """ self.transform = np.array([[-1, 0, self.row-1], [0, 1, 0], [0, 0, 1]]) def Rotate(self, angle): """ 旋转。参数说明： angle：旋转角度 """ self.transform = np.array([[math.cos(angle),-math.sin(angle),0], [math.sin(angle), math.cos(angle),0], [ 0, 0, 1]]) def operate(self): temp = np.zeros(self.img.shape, dtype=self.img.dtype) for i in range(self.row): for j in range(self.col): temp_pos = np.array([i, j, 1]) [x,y,z] = np.dot(self.transform, temp_pos) x = int(x) y = int(y) if x>=self.row or y>=self.col or x<0 or y<0: temp[i,j,:] = 0 else: temp[i,j,:] = self.img[x,y] return temp def __call__(self, act): r = r"([a-zA-Z]*)=([^,)]*)" act_str = act.lower() kwargs = dict([(i, eval(j)) for (i, j) in re.findall(r, act_str)]) if "translation" in act_str: self.Translation(**kwargs) elif "resize" in act_str: self.Resize(**kwargs) elif "hormirror" in act_str: self.HorMirror(**kwargs) elif "vermirror" in act_str: self.VerMirror(**kwargs) elif "rotate" in act_str: self.Rotate(**kwargs) return self.operate() #######----Early Stopping----#### def early_stopping(valid): """ 参数说明： valid：验证集正确率列表 """ if len(valid) > 5: if valid[-1] < valid[-5] and valid[-2] < valid[-5] and valid[-3] < valid[-5] and valid[-4] < valid[-5]: return True return False #####---Bagging--##### def bootstrap_sample(X, Y): N, M = X.shape idxs = np.random.choice(N, N, replace=True) return X[idxs], Y[idxs] class BaggingModel(object): def __init__(self, n_models): """ 参数说明： n_models：网络模型数目 """ self.models = [] self.n_models = n_models def fit(self, X, Y): self.models = [] for i in range(self.n_models): print("training {} base model:".format(i)) X_samp, Y_samp = bootstrap_sample(X, Y) model = DFN(hidden_dims_1=200, hidden_dims_2=10) model.fit(X_samp, Y_samp) self.models.append(model) def predict(self, X): model_preds = np.array([[np.argmax(t.forward(x)[0]) for x in X] for t in self.models]) return self._vote(model_preds) def _vote(self, predictions): out = [np.bincount(x).argmax() for x in predictions.T] return np.array(out) def evaluate(self, X_test, y_test): acc = 0.0 y_pred = self.predict(X_test) y_true = np.argmax(y_test, axis=1) acc += np.sum(y_pred == y_true) return acc / X_test.shape[0] #####----Dropout----####### class Dropout(ABC): def __init__(self, wrapped_layer, p): """ 参数说明： wrapped_layer：被 dropout 的层 p：神经元保留率 """ super().__init__() self._base_layer = wrapped_layer self.p = p self._init_wrapper_params() def _init_wrapper_params(self): self._wrapper_derived_variables = {"dropout_mask": None} self._wrapper_hyperparams = {"wrapper": "Dropout", "p": self.p} def flush_gradients(self): """ 函数作用：调用 base layer 重置更新参数列表 """ self._base_layer.flush_gradients() def update(self): """ 函数作用：调用 base layer 更新参数 """ self._base_layer.update() def forward(self, X, is_train=True): """ 参数说明： X：输入数组； is_train：是否为训练阶段，bool型； """ mask = np.ones(X.shape).astype(bool) if is_train: mask = (np.random.rand(*X.shape) < self.p) / self.p X = mask * X self._wrapper_derived_variables["dropout_mask"] = mask return self._base_layer.forward(X) def backward(self, dLda): return self._base_layer.backward(dLda) @property def hyperparams(self): hp = self._base_layer.hyperparams hpw = self._wrapper_hyperparams if "wrappers" in hp: hp["wrappers"].append(hpw) else: hp["wrappers"] = [hpw] return hp #####----Bagging----####### # 进度条 bar_widgets = [ 'Training: ', progressbar.Percentage(), ' ', progressbar.Bar(marker="-", left="[", right="]"), ' ', progressbar.ETA() ] def get_random_subsets(X, y, n_subsets, replacements=True): """从训练数据中抽取数据子集 (默认可重复抽样)""" n_samples = np.shape(X)[0] # 将 X 和 y 拼接，并将元素随机排序 Xy = np.concatenate((X, y.reshape((1, len(y))).T), axis=1) np.random.shuffle(Xy) subsets = [] # 如果抽样时不重复抽样，可以只使用 50% 的训练数据；如果抽样时可重复抽样，使用全部的训练数据，默认可重复抽样 subsample_size = int(n_samples // 2) if replacements: subsample_size = n_samples for _ in range(n_subsets): idx = np.random.choice( range(n_samples), size=np.shape(range(subsample_size)), replace=replacements) X = Xy[idx][:, :-1] y = Xy[idx][:, -1] subsets.append([X, y]) return subsets class Bagging(): """ Bagging分类器。使用一组分类树，这些分类树使用特征训练数据的随机子集。 """ def __init__(self, n_estimators=100, max_features=None, min_samples_split=2, min_gain=0, max_depth=float("inf")): self.n_estimators = n_estimators # 树的数目 self.min_samples_split = min_samples_split # 分割所需的最小样本数 self.min_gain = min_gain # 分割所需的最小纯度 (最小信息增益) self.max_depth = max_depth # 树的最大深度 self.progressbar = progressbar.ProgressBar(widgets=bar_widgets) # 初始化决策树 self.trees = [] for _ in range(n_estimators): self.trees.append( ClassificationTree( min_samples_split=self.min_samples_split, min_impurity=min_gain, max_depth=self.max_depth)) def fit(self, X, y): # 对每棵树选择数据集的随机子集 subsets = get_random_subsets(X, y, self.n_estimators) for i in self.progressbar(range(self.n_estimators)): X_subset, y_subset = subsets[i] # 用特征子集和真实值训练一棵子模型 (这里的数据也是训练数据集的随机子集) self.trees[i].fit(X_subset, y_subset) def predict(self, X): y_preds = np.empty((X.shape[0], len(self.trees))) # 每棵决策树都在数据上预测 for i, tree in enumerate(self.trees): # 基于特征做出预测 prediction = tree.predict(X) y_preds[:, i] = prediction y_pred = [] # 对每个样本，选择最常见的类别作为预测 for sample_predictions in y_preds: y_pred.append(np.bincount(sample_predictions.astype('int')).argmax()) return y_pred def score(self, X, y): y_pred = self.predict(X) accuracy = np.sum(y == y_pred, axis=0) / len(y) return accuracy #####----RandomForest----####### class RandomForest(): """ 随机森林分类器。使用一组分类树，这些分类树使用特征的随机子集训练数据的随机子集。 """ def __init__(self, n_estimators=100, max_features=None, min_samples_split=2, min_gain=0, max_depth=float("inf")): self.n_estimators = n_estimators # 树的数目 self.max_features = max_features # 每棵树的最大使用特征数 self.min_samples_split = min_samples_split # 分割所需的最小样本数 self.min_gain = min_gain # 分割所需的最小纯度 (最小信息增益) self.max_depth = max_depth # 树的最大深度 self.progressbar = progressbar.ProgressBar(widgets=bar_widgets) # 初始化决策树 self.trees = [] for _ in range(n_estimators): self.trees.append( ClassificationTree( min_samples_split=self.min_samples_split, min_impurity=min_gain, max_depth=self.max_depth)) def fit(self, X, y): n_features = np.shape(X)[1] # 如果 max_features 没有定义，取默认值 sqrt(n_features) if not self.max_features: self.max_features = int(math.sqrt(n_features)) # 对每棵树选择数据集的随机子集 subsets = get_random_subsets(X, y, self.n_estimators) for i in self.progressbar(range(self.n_estimators)): X_subset, y_subset = subsets[i] # 选择特征的随机子集 idx = np.random.choice(range(n_features), size=self.max_features, replace=True) # 保存特征的索引用于预测 self.trees[i].feature_indices = idx # 选择索引对应的特征 X_subset = X_subset[:, idx] # 用特征子集和真实值训练一棵子模型 (这里的数据也是训练数据集的随机子集) self.trees[i].fit(X_subset, y_subset) def predict(self, X): y_preds = np.empty((X.shape[0], len(self.trees))) # 每棵决策树都在数据上预测 for i, tree in enumerate(self.trees): # 使用该决策树训练使用的特征 idx = tree.feature_indices # 基于特征做出预测 prediction = tree.predict(X[:, idx]) y_preds[:, i] = prediction y_pred = [] # 对每个样本，选择最常见的类别作为预测 for sample_predictions in y_preds: y_pred.append(np.bincount(sample_predictions.astype('int')).argmax()) return y_pred def score(self, X, y): y_pred = self.predict(X) accuracy = np.sum(y == y_pred, axis=0) / len(y) return accuracy #####----Adaboost----####### # 决策树桩，作为 Adaboost 算法的弱分类器 (基分类器) class DecisionStump(): def __init__(self): self.polarity = 1 # 表示决策树桩默认输出的类别为 1 或是 -1 self.feature_index = None # 用于分类的特征索引 self.threshold = None # 特征的阈值 self.alpha = None # 表示分类器准确性的值 class Adaboost(): """ Adaboost 算法。 """ def __init__(self, n_estimators=5): self.n_estimators = n_estimators # 将使用的弱分类器的数量 self.progressbar = progressbar.ProgressBar(widgets=bar_widgets) def fit(self, X, y): n_samples, n_features = np.shape(X) # 初始化权重 (上文中的 D)，均为 1/N w = np.full(n_samples, (1 / n_samples)) self.trees = [] # 迭代过程 for _ in self.progressbar(range(self.n_estimators)): tree = DecisionStump() min_error = float('inf') # 使用某一特征值的阈值预测样本的最小误差 # 迭代遍历每个 (不重复的) 特征值，查找预测 y 的最佳阈值 for feature_i in range(n_features): feature_values = np.expand_dims(X[:, feature_i], axis=1) unique_values = np.unique(feature_values) # 将该特征的每个特征值作为阈值 for threshold in unique_values: p = 1 # 将所有样本预测默认值可以设置为 1 prediction = np.ones(np.shape(y)) # 低于特征值阈值的预测改为 -1 prediction[X[:, feature_i] < threshold] = -1 # 计算错误率 error = sum(w[y != prediction]) # 如果错误率超过 50%，我们反转决策树桩默认输出的类别 # 比如 error = 0.8 => (1 - error) = 0.2， # 原来计算的是输出到类别 1 的概率，类别 1 作为默认类别。反转后类别 0 作为默认类别 if error > 0.5: error = 1 - error p = -1 # 如果这个阈值导致最小的错误率，则保存 if error < min_error: tree.polarity = p tree.threshold = threshold tree.feature_index = feature_i min_error = error # 计算用于更新样本权值的 alpha 值，也是作为基分类器的系数。 tree.alpha = 0.5 * math.log((1.0 - min_error) / (min_error + 1e-10)) # 将所有样本预测默认值设置为 1 predictions = np.ones(np.shape(y)) # 如果特征值低于阈值，则修改预测结果，这里还需要考虑弱分类器的默认输出类别 negative_idx = (tree.polarity * X[:, tree.feature_index] < tree.polarity * tree.threshold) predictions[negative_idx] = -1 # 计算新权值，未正确分类样本的权值增大，正确分类样本的权值减小 w *= np.exp(-tree.alpha * y * predictions) w /= np.sum(w) # 保存分类器 self.trees.append(tree) def predict(self, X): n_samples = np.shape(X)[0] y_pred = np.zeros((n_samples, 1)) # 用每一个基分类器预测样本 for tree in self.trees: # 将所有样本预测默认值设置为 1 predictions = np.ones(np.shape(y_pred)) negative_idx = (tree.polarity * X[:, tree.feature_index] < tree.polarity * tree.threshold) predictions[negative_idx] = -1 # 对基分类器加权求和，权重 alpha y_pred += tree.alpha * predictions # 返回预测结果 1 或 -1 y_pred = np.sign(y_pred).flatten() return y_pred def score(self, X, y): y_pred = self.predict(X) accuracy = np.sum(y == y_pred, axis=0) / len(y) return accuracy #####----GBDT----####### class Loss(ABC): def __init__(self): super().__init__() @abstractmethod def loss(self, y_true, y_pred): return NotImplementedError() @abstractmethod def grad(self, y, y_pred): raise NotImplementedError() class SquareLoss(Loss): def __init__(self): pass def loss(self, y, y_pred): pass def grad(self, y, y_pred): return -(y - y_pred) def hess(self, y, y_pred): return 1 class CrossEntropyLoss(Loss): def __init__(self): pass def loss(self, y, y_pred): pass def grad(self, y, y_pred): return - (y - y_pred) def hess(self, y, y_pred): return y_pred * (1-y_pred) def softmax(x): e_x = np.exp(x - np.max(x, axis=-1, keepdims=True)) return e_x / e_x.sum(axis=-1, keepdims=True) def line_search(self, y, y_pred, h_pred): Lp = 2 * np.sum((y - y_pred) * h_pred) Lpp = np.sum(h_pred * h_pred) return 1 if np.sum(Lpp) == 0 else Lp / Lpp def to_categorical(x, n_classes=None): """ One-hot编码 """ if not n_classes: n_classes = np.amax(x) + 1 one_hot = np.zeros((x.shape[0], n_classes)) one_hot[np.arange(x.shape[0]), x] = 1 return one_hot class GradientBoostingDecisionTree(object): """ GBDT 算法。用一组基学习器 (回归树) 学习损失函数的梯度。 """ def __init__(self, n_estimators, learning_rate=1, min_samples_split=2, min_impurity=1e-7, max_depth=float("inf"), is_regression=False, line_search=False): self.n_estimators = n_estimators # 迭代的次数 self.learning_rate = learning_rate # 训练过程中沿着负梯度走的步长，也就是学习率 self.min_samples_split = min_samples_split # 分割所需的最小样本数 self.min_impurity = min_impurity # 分割所需的最小纯度 self.max_depth = max_depth # 树的最大深度 self.is_regression = is_regression # 分类问题或回归问题 self.line_search = line_search # 是否使用 line search self.progressbar = progressbar.ProgressBar(widgets=bar_widgets) # 回归问题采用基础的平方损失，分类问题采用交叉熵损失 self.loss = SquareLoss() if not self.is_regression: self.loss = CrossEntropyLoss() def fit(self, X, Y): # 分类问题将 Y 转化为 one-hot 编码 if not self.is_regression: Y = to_categorical(Y.flatten()) else: Y = Y.reshape(-1, 1) if len(Y.shape) == 1 else Y self.out_dims = Y.shape[1] self.trees = np.empty((self.n_estimators, self.out_dims), dtype=object) Y_pred = np.full(np.shape(Y), np.mean(Y, axis=0)) self.weights = np.ones((self.n_estimators, self.out_dims)) self.weights[1:, :] *= self.learning_rate # 迭代过程 for i in self.progressbar(range(self.n_estimators)): for c in range(self.out_dims): tree = RegressionTree( min_samples_split=self.min_samples_split, min_impurity=self.min_impurity, max_depth=self.max_depth) # 计算损失的梯度，并用梯度进行训练 if not self.is_regression: Y_hat = softmax(Y_pred) y, y_pred = Y[:, c], Y_hat[:, c] else: y, y_pred = Y[:, c], Y_pred[:, c] neg_grad = -1 * self.loss.grad(y, y_pred) tree.fit(X, neg_grad) # 用新的基学习器进行预测 h_pred = tree.predict(X) # line search if self.line_search == True: self.weights[i, c] *= line_search(y, y_pred, h_pred) # 加法模型中添加基学习器的预测，得到最新迭代下的加法模型预测 Y_pred[:, c] += np.multiply(self.weights[i, c], h_pred) self.trees[i, c] = tree def predict(self, X): Y_pred = np.zeros((X.shape[0], self.out_dims)) # 生成预测 for c in range(self.out_dims): y_pred = np.array([]) for i in range(self.n_estimators): update = np.multiply(self.weights[i, c], self.trees[i, c].predict(X)) y_pred = update if not y_pred.any() else y_pred + update Y_pred[:, c] = y_pred if not self.is_regression: # 分类问题输出最可能类别 Y_pred = Y_pred.argmax(axis=1) return Y_pred def score(self, X, y): y_pred = self.predict(X) accuracy = np.sum(y == y_pred, axis=0) / len(y) return accuracy class GradientBoostingRegressor(GradientBoostingDecisionTree): def __init__(self, n_estimators=200, learning_rate=1, min_samples_split=2, min_impurity=1e-7, max_depth=float("inf"), is_regression=True, line_search=False): super(GradientBoostingRegressor, self).__init__(n_estimators=n_estimators, learning_rate=learning_rate, min_samples_split=min_samples_split, min_impurity=min_impurity, max_depth=max_depth, is_regression=is_regression, line_search=line_search) class GradientBoostingClassifier(GradientBoostingDecisionTree): def __init__(self, n_estimators=200, learning_rate=1, min_samples_split=2, min_impurity=1e-7, max_depth=float("inf"), is_regression=False, line_search=False): super(GradientBoostingClassifier, self).__init__(n_estimators=n_estimators, learning_rate=learning_rate, min_samples_split=min_samples_split, min_impurity=min_impurity, max_depth=max_depth, is_regression=is_regression, line_search=line_search) #####----XGBoost----####### class XGBoostRegressionTree(DecisionTree): """ XGBoost 回归树。此处基于第五章介绍的决策树，故采用贪心算法找到特征上分裂点 (枚举特征上所有可能的分裂点)。 """ def __init__(self, min_samples_split=2, min_impurity=1e-7, max_depth=float("inf"), loss=None, gamma=0., lambd=0.): super(XGBoostRegressionTree, self).__init__(min_impurity=min_impurity, min_samples_split=min_samples_split, max_depth=max_depth) self.gamma = gamma # 叶子节点的数目的惩罚系数 self.lambd = lambd # 叶子节点的权重的惩罚系数 self.loss = loss # 损失函数 def _split(self, y): # y 包含 y_true 在左半列，y_pred 在右半列 col = int(np.shape(y)[1]/2) y, y_pred = y[:, :col], y[:, col:] return y, y_pred def _gain(self, y, y_pred): # 计算信息 nominator = np.power((y * self.loss.grad(y, y_pred)).sum(), 2) denominator = self.loss.hess(y, y_pred).sum() return nominator / (denominator + self.lambd) def _gain_by_taylor(self, y, y1, y2): # 分割为左子树和右子树 y, y_pred = self._split(y) y1, y1_pred = self._split(y1) y2, y2_pred = self._split(y2) true_gain = self._gain(y1, y1_pred) false_gain = self._gain(y2, y2_pred) gain = self._gain(y, y_pred) # 计算信息增益 return 0.5 * (true_gain + false_gain - gain) - self.gamma def _approximate_update(self, y): y, y_pred = self._split(y) # 计算叶节点权重 gradient = self.loss.grad(y, y_pred).sum() hessian = self.loss.hess(y, y_pred).sum() leaf_approximation = -gradient / (hessian + self.lambd) return leaf_approximation def fit(self, X, y): self._impurity_calculation = self._gain_by_taylor self._leaf_value_calculation = self._approximate_update super(XGBoostRegressionTree, self).fit(X, y) class XGBoost(object): """ XGBoost学习器。 """ def __init__(self, n_estimators=200, learning_rate=0.001, min_samples_split=2, min_impurity=1e-7, max_depth=2, is_regression=False, gamma=0., lambd=0.): self.n_estimators = n_estimators # 树的数目 self.learning_rate = learning_rate # 训练过程中沿着负梯度走的步长，也就是学习率 self.min_samples_split = min_samples_split # 分割所需的最小样本数 self.min_impurity = min_impurity # 分割所需的最小纯度 self.max_depth = max_depth # 树的最大深度 self.gamma = gamma # 叶子节点的数目的惩罚系数 self.lambd = lambd # 叶子节点的权重的惩罚系数 self.is_regression = is_regression # 分类或回归问题 self.progressbar = progressbar.ProgressBar(widgets=bar_widgets) # 回归问题采用基础的平方损失，分类问题采用交叉熵损失 self.loss = SquareLoss() if not self.is_regression: self.loss = CrossEntropyLoss() def fit(self, X, Y): # 分类问题将 Y 转化为 one-hot 编码 if not self.is_regression: Y = to_categorical(Y.flatten()) else: Y = Y.reshape(-1, 1) if len(Y.shape) == 1 else Y self.out_dims = Y.shape[1] self.trees = np.empty((self.n_estimators, self.out_dims), dtype=object) Y_pred = np.zeros(np.shape(Y)) self.weights = np.ones((self.n_estimators, self.out_dims)) self.weights[1:, :] *= self.learning_rate # 迭代过程 for i in self.progressbar(range(self.n_estimators)): for c in range(self.out_dims): tree = XGBoostRegressionTree( min_samples_split=self.min_samples_split, min_impurity=self.min_impurity, max_depth=self.max_depth, loss=self.loss, gamma=self.gamma, lambd=self.lambd) # 计算损失的梯度，并用梯度进行训练 if not self.is_regression: Y_hat = softmax(Y_pred) y, y_pred = Y[:, c], Y_hat[:, c] else: y, y_pred = Y[:, c], Y_pred[:, c] y, y_pred = y.reshape(-1, 1), y_pred.reshape(-1, 1) y_and_ypred = np.concatenate((y, y_pred), axis=1) tree.fit(X, y_and_ypred) # 用新的基学习器进行预测 h_pred = tree.predict(X) # 加法模型中添加基学习器的预测，得到最新迭代下的加法模型预测 Y_pred[:, c] += np.multiply(self.weights[i, c], h_pred) self.trees[i, c] = tree def predict(self, X): Y_pred = np.zeros((X.shape[0], self.out_dims)) # 生成预测 for c in range(self.out_dims): y_pred = np.array([]) for i in range(self.n_estimators): update = np.multiply(self.weights[i, c], self.trees[i, c].predict(X)) y_pred = update if not y_pred.any() else y_pred + update Y_pred[:, c] = y_pred if not self.is_regression: # 分类问题输出最可能类别 Y_pred = Y_pred.argmax(axis=1) return Y_pred def score(self, X, y): y_pred = self.predict(X) accuracy = np.sum(y == y_pred, axis=0) / len(y) return accuracy class XGBRegressor(XGBoost): def __init__(self, n_estimators=200, learning_rate=1, min_samples_split=2, min_impurity=1e-7, max_depth=float("inf"), is_regression=True, gamma=0., lambd=0.): super(XGBRegressor, self).__init__(n_estimators=n_estimators, learning_rate=learning_rate, min_samples_split=min_samples_split, min_impurity=min_impurity, max_depth=max_depth, is_regression=is_regression, gamma=gamma, lambd=lambd) class XGBClassifier(XGBoost): def __init__(self, n_estimators=200, learning_rate=1, min_samples_split=2, min_impurity=1e-7, max_depth=float("inf"), is_regression=False, gamma=0., lambd=0.): super(XGBClassifier, self).__init__(n_estimators=n_estimators, learning_rate=learning_rate, min_samples_split=min_samples_split, min_impurity=min_impurity, max_depth=max_depth, is_regression=is_regression, gamma=gamma, lambd=lambd) ================================================ FILE: code/chapter8.py ================================================ from chapter import LayerBase import numpy as np ######### 优化方法(Optimizer)见 method/optimizer ####### ######## 参数初始化(Parameter Initialization) 见method/weight ##### ######## BatchNorm1D ##### class BatchNorm1D(LayerBase): def __init__(self, momentum=0.9, epsilon=1e-5, optimizer=None): """ 参数说明： momentum：动量项，越趋于 1 表示对当前 Batch 的依赖程度越小，running_mean和running_var的计算越平滑 float型 (default: 0.9) epsilon：避免除数为0，float型 (default : 1e-5) optimizer：优化器 """ super().__init__(optimizer) self.n_in = None self.n_out = None self.epsilon = epsilon self.momentum = momentum self.params = { "scaler": None, "intercept": None, "running_var": None, "running_mean": None, } self.is_initialized = False def _init_params(self): scaler = np.random.rand(self.n_in) intercept = np.zeros(self.n_in) running_mean = np.zeros(self.n_in) running_var = np.ones(self.n_in) self.params = { "scaler": scaler, "intercept": intercept, "running_mean": running_mean, "running_var": running_var, } self.gradients = { "scaler": np.zeros_like(scaler), "intercept": np.zeros_like(intercept), } self.is_initialized = True def reset_running_stats(self): self.params["running_mean"] = np.zeros(self.n_in) self.params["running_var"] = np.ones(self.n_in) def forward(self, X, is_train=True, retain_derived=True): """ Batch 训练时 BN 的前向传播，原理见上文。 [train]: Y = scaler * norm(X) + intercept，其中 norm(X) = (X - mean(X)) / sqrt(var(X) + epsilon) [test]: Y = scaler * running_norm(X) + intercept，其中 running_norm(X) = (X - running_mean) / sqrt(running_var + epsilon) 参数说明： X：输入数组，为（n_samples, n_in），float型 is_train：是否为训练阶段，bool型 retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型 """ if not self.is_initialized: self.n_in = self.n_out = X.shape[1] self._init_params() epsi, momentum = self.hyperparams["epsilon"], self.hyperparams["momentum"] rm, rv = self.params["running_mean"], self.params["running_var"] scaler, intercept = self.params["scaler"], self.params["intercept"] X_mean, X_var = self.params["running_mean"], self.params["running_var"] if is_train and retain_derived: X_mean, X_var = X.mean(axis=0), X.var(axis=0) self.params["running_mean"] = momentum * rm + (1.0 - momentum) * X_mean self.params["running_var"] = momentum * rv + (1.0 - momentum) * X_var if retain_derived: self.X.append(X) X_hat = (X - X_mean) / np.sqrt(X_var + epsi) y = scaler * X_hat + intercept return y def backward(self, dLda, retain_grads=True): """ BN 的反向传播，原理见上文。参数说明： dLda：关于损失的梯度，为（n_samples, n_out），float型 retain_grads：是否计算中间变量的参数梯度，bool型 """ if not isinstance(dLda, list): dLda = [dLda] dX = [] X = self.X for da, x in zip(dLda, X): dx, dScaler, dIntercept = self._bwd(da, x) dX.append(dx) if retain_grads: self.gradients["scaler"] += dScaler self.gradients["intercept"] += dIntercept return dX[0] if len(X) == 1 else dX def _bwd(self, dLda, X): scaler = self.params["scaler"] epsi = self.hyperparams["epsilon"] n_ex, n_in = X.shape X_mean, X_var = X.mean(axis=0), X.var(axis=0) X_hat = (X - X_mean) / np.sqrt(X_var + epsi) dIntercept = dLda.sum(axis=0) dScaler = np.sum(dLda * X_hat, axis=0) dX_hat = dLda * scaler dX = (n_ex * dX_hat - dX_hat.sum(axis=0) - X_hat * (dX_hat * X_hat).sum(axis=0)) / ( n_ex * np.sqrt(X_var + epsi) ) return dX, dScaler, dIntercept @property def hyperparams(self): return { "layer": "BatchNorm1D", "acti_fn": None, "n_in": self.n_in, "n_out": self.n_out, "epsilon": self.epsilon, "momentum": self.momentum, "optimizer": { "cache": self.optimizer.cache, "hyperparams": self.optimizer.hyperparams, }, } ================================================ FILE: code/chapter9.py ================================================ from abc import ABC, abstractmethod import numpy as np from chapter6 import LayerBase, CrossEntropy, FullyConnected, minibatch, softmax from collections import OrderedDict ########## Padding ################ def calc_pad_dims_sameconv_2D(X_shape, out_dim, kernel_shape, stride, dilation=1): """ 当填充方式为相同卷积时，计算 padding 的数目，保证输入输出的大小相同。这里在卷积过程中考虑填充(Padding)，卷积步幅(Stride)，扩张率(Dilation rate)。根据扩张卷积的输出公式可以得到 padding 的数目。参数说明： X_shape：输入数组，为 (n_samples, in_rows, in_cols, in_ch) out_dim：输出数组维数，为 (out_rows, out_cols) kernel_shape：卷积核形状，为 (fr, fc) stride：卷积步幅，int 型 dilation：扩张率，int 型，default=1 """ d = dilation fr, fc = kernel_shape out_rows, out_cols = out_dim n_ex, in_rows, in_cols, in_ch = X_shape # 考虑扩张率 _fr, _fc = fr + (fr-1) * (d-1), fc + (fc-1) * (d-1) # 计算 padding 维数 pr = int((stride * (out_rows-1) + _fr - in_rows) / 2) pc = int((stride * (out_cols-1) + _fc - in_cols) / 2) # 校验，如不等 (right/bottom处) 添加不对称0填充 out_rows1 = int(1 + (in_rows + 2 * pr - _fr) / stride) out_cols1 = int(1 + (in_cols + 2 * pc - _fc) / stride) pr1, pr2 = pr, pr if out_rows1 == out_rows - 1: pr1, pr2 = pr, pr + 1 elif out_rows1 != out_rows: raise AssertionError pc1, pc2 = pc, pc if out_cols1 == out_cols - 1: pc1, pc2 = pc, pc + 1 elif out_cols1 != out_cols: raise AssertionError # 返回对 X 的 Padding 维数 (left, right, up, down) return (pr1, pr2, pc1, pc2) def pad2D(X, pad, kernel_shape=None, stride=None, dilation=1): """ 二维填充参数说明： X：输入数组，为 (n_samples, in_rows, in_cols, in_ch)，其中 padding 操作是应用到 in_rows 和 in_cols pad：padding 数目，4-tuple, int, 或 'same'，'valid' 在图片的左、右、上、下 (left, right, up, down) 0填充若为int，表示在左、右、上、下均填充数目为 pad 的 0，若为same，表示填充后为相同 (same) 卷积，若为valid，表示填充后为有效 (valid) 卷积 kernel_shape：卷积核形状，为 (fr, fc) stride：卷积步幅，int 型 dilation：扩张率，int 型，default=1 """ p = pad if isinstance(p, int): p = (p, p, p, p) if isinstance(p, tuple): X_pad = np.pad( X, pad_width=((0, 0), (p[0], p[1]), (p[2], p[3]), (0, 0)), mode="constant", constant_values=0, ) # 'same'卷积，首先计算 padding 维数 if p == "same" and kernel_shape and stride is not None: p = calc_pad_dims_sameconv_2D( X.shape, X.shape[1:3], kernel_shape, stride, dilation=dilation ) X_pad, p = pad2D(X, p) if p == "valid": p = (0, 0, 0, 0) X_pad, p = pad2D(X, p) return X_pad, p ####### conv2D ################## def conv2D(X, W, stride, pad, dilation=1): """ 二维卷积实现过程。参数说明： X：输入数组，为 (n_samples, in_rows, in_cols, in_ch) W：卷积层的卷积核参数，为 (kernel_rows, kernel_cols, in_ch, out_ch) stride：卷积核的卷积步幅，int型 pad：padding 数目，4-tuple, int, 或 'same'，'valid'型在图片的左、右、上、下 (left, right, up, down) 0填充若为int，表示在左、右、上、下均填充数目为 pad 的 0，若为same，表示填充后为相同 (same) 卷积，若为valid，表示填充后为有效 (valid) 卷积 dilation：扩张率，int 型，default=1 输出说明： Z：卷积结果，为 (n_samples, out_rows, out_cols, out_ch) """ s, d = stride, dilation X_pad, p = pad2D(X, pad, W.shape[:2], stride=s, dilation=d) pr1, pr2, pc1, pc2 = p fr, fc, in_ch, out_ch = W.shape n_samp, in_rows, in_cols, in_ch = X.shape # 考虑扩张率 _fr, _fc = fr + (fr-1) * (d-1), fc + (fc-1) * (d-1) out_rows = int((in_rows + pr1 + pr2 - _fr) / s + 1) out_cols = int((in_cols + pc1 + pc2 - _fc) / s + 1) Z = np.zeros((n_samp, out_rows, out_cols, out_ch)) for m in range(n_samp): for c in range(out_ch): for i in range(out_rows): for j in range(out_cols): i0, i1 = i * s, (i * s) + fr + (fr-1) * (d-1) j0, j1 = j * s, (j * s) + fc + (fc-1) * (d-1) window = X_pad[m, i0 : i1 : d, j0 : j1 : d, :] Z[m, i, j, c] = np.sum(window * W[:, :, :, c]) return Z ####### conv2D GEMM ############ """ conv2D 的 GEMM 实现过程，将 X 和 W 转化为 2D 矩阵，这里我们将 X 转化为 (kernel_rows*kernel_cols*in_ch, n_samples*out_rows*out_cols) W 转化为 (out_ch, kernel_rows*kernel_cols*in_ch) """ def _im2col_indices(X_shape, fr, fc, p, s, d=1): """ 生成输入矩阵的 (c,h_in,w_in) 三个维度的索引输出说明： i：输入矩阵的i值，(kernel_rows*kernel_cols*in_ch, out_rows*out_cols)，图示中第二维坐标 j：输入矩阵的j值，(kernel_rows*kernel_cols*in_ch, out_rows*out_cols)，图示中第三维坐标 k：输入矩阵的c值，(kernel_rows*kernel_cols*in_ch, 1)，图示中第一维坐标 """ pr1, pr2, pc1, pc2 = p n_ex, n_in, in_rows, in_cols = X_shape # 考虑扩张率 _fr, _fc = fr + (fr-1) * (d-1), fc + (fc-1) * (d-1) out_rows = int((in_rows + pr1 + pr2 - _fr) / s + 1) out_cols = int((in_cols + pc1 + pc2 - _fc) / s + 1) # i0/i1/j0/j1：用于得到i，j，k。i0/j0过程见图示，i1/j1由滑动过程得出 i0 = np.repeat(np.arange(fr), fc) i0 = np.tile(i0, n_in) * d i1 = s * np.repeat(np.arange(out_rows), out_cols) j0 = np.tile(np.arange(fc), fr * n_in) * d j1 = s * np.tile(np.arange(out_cols), out_rows) i = i0.reshape(-1, 1) + i1.reshape(1, -1) j = j0.reshape(-1, 1) + j1.reshape(1, -1) k = np.repeat(np.arange(n_in), fr * fc).reshape(-1, 1) return k, i, j def im2col(X, W_shape, pad, stride, dilation=1): """ im2col 实现参数说明： X：输入数组，为 (n_samples, in_rows, in_cols, in_ch)，此时还未 0 填充(padding) W_shape：卷积层的卷积核的形状，为 (kernel_rows, kernel_cols, in_ch, out_ch) pad：padding 数目，4-tuple, int, 或 'same'，'valid'型在图片的左、右、上、下 (left, right, up, down) 0填充若为int，表示在左、右、上、下均填充数目为 pad 的 0，若为same，表示填充后为相同 (same) 卷积，若为valid，表示填充后为有效 (valid) 卷积 stride：卷积核的卷积步幅，int型 dilation：扩张率，int 型，default=1 输出说明： X_col：输出结果，形状为 (kernel_rows*kernel_cols*n_in, n_samples*out_rows*out_cols) p：填充数，4-tuple """ fr, fc, n_in, n_out = W_shape s, p, d = stride, pad, dilation n_samp, in_rows, in_cols, n_in = X.shape X_pad, p = pad2D(X, p, W_shape[:2], stride=s, dilation=d) pr1, pr2, pc1, pc2 = p # 将输入的通道维数移至第二位 X_pad = X_pad.transpose(0, 3, 1, 2) k, i, j = _im2col_indices((n_samp, n_in, in_rows, in_cols), fr, fc, p, s, d) # X_col.shape = (n_samples, kernel_rows*kernel_cols*n_in, out_rows*out_cols) X_col = X_pad[:, k, i, j] X_col = X_col.transpose(1, 2, 0).reshape(fr * fc * n_in, -1) return X_col, p def conv2D_gemm(X, W, stride=0, pad='same', dilation=1): """ 二维卷积实现过程，依靠“im2col”函数将卷积作为单个矩阵乘法执行。参数说明： X：输入数组，为 (n_samples, in_rows, in_cols, in_ch) W：卷积层的卷积核参数，为 (kernel_rows, kernel_cols, in_ch, out_ch) stride：卷积核的卷积步幅，int型 pad：padding 数目，4-tuple, int, 或 'same'，'valid'型在图片的左、右、上、下 (left, right, up, down) 0填充若为int，表示在左、右、上、下均填充数目为 pad 的 0，若为same，表示填充后为相同 (same) 卷积，若为valid，表示填充后为有效 (valid) 卷积 dilation：扩张率，int 型，default=1 输出说明： Z：卷积结果，为 (n_samples, out_rows, out_cols, out_ch) """ s, d = stride, dilation _, p = pad2D(X, pad, W.shape[:2], s, dilation=dilation) pr1, pr2, pc1, pc2 = p fr, fc, in_ch, out_ch = W.shape n_samp, in_rows, in_cols, in_ch = X.shape # 考虑扩张率 _fr, _fc = fr + (fr-1) * (d-1), fc + (fc-1) * (d-1) # 输出维数，根据上面公式可得 out_rows = int((in_rows + pr1 + pr2 - _fr) / s + 1) out_cols = int((in_cols + pc1 + pc2 - _fc) / s + 1) # 将 X 和 W 转化为 2D 矩阵并乘积 X_col, _ = im2col(X, W.shape, p, s, d) W_col = W.transpose(3, 2, 0, 1).reshape(out_ch, -1) Z = (W_col @ X_col).reshape(out_ch, out_rows, out_cols, n_samp).transpose(3, 1, 2, 0) return Z ########### Conv2D ################## class Conv2D(LayerBase): def __init__( self, out_ch, kernel_shape, pad=0, stride=1, dilation=1, acti_fn=None, optimizer=None, init_w="glorot_uniform", ): """ 二维卷积参数说明： out_ch：卷积核组的数目，int 型 kernel_shape：单个卷积核形状，2-tuple acti_fn：激活函数，str 型 pad：padding 数目，4-tuple, int, 或 'same'，'valid'型在图片的左、右、上、下 (left, right, up, down) 0填充若为int，表示在左、右、上、下均填充数目为 pad 的 0，若为same，表示填充后为相同 (same) 卷积，若为valid，表示填充后为有效 (valid) 卷积 stride：卷积核的卷积步幅，int型 dilation：扩张率，int 型，default=1 init_w：权重初始化方法，str型 optimizer：优化方法，str型 """ super().__init__(optimizer) self.pad = pad self.in_ch = None self.out_ch = out_ch self.stride = stride self.dilation = dilation self.kernel_shape = kernel_shape self.init_w = init_w self.init_weights = WeightInitializer(mode=init_w) self.acti_fn = ActivationInitializer(acti_fn)() self.parameters = {"W": None, "b": None} self.is_initialized = False def _init_params(self): fr, fc = self.kernel_shape W = self.init_weights((fr, fc, self.in_ch, self.out_ch)) b = np.zeros((1, 1, 1, self.out_ch)) self.params = {"W": W, "b": b} self.gradients = {"W": np.zeros_like(W), "b": np.zeros_like(b)} self.derived_variables = {"Y": []} self.is_initialized = True def forward(self, X, retain_derived=True): """ 卷积层的前向传播，原理见上文。参数说明： X：输入数组，形状为 (n_samples, in_rows, in_cols, in_ch) retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型输出说明： a：卷积层输出，形状为 (n_samples, out_rows, out_cols, out_ch) """ if not self.is_initialized: self.in_ch = X.shape[3] self._init_params() W = self.params["W"] b = self.params["b"] n_samp, in_rows, in_cols, in_ch = X.shape s, p, d = self.stride, self.pad, self.dilation # 卷积操作 Y = conv2D(X, W, s, p, d) + b a = self.acti_fn(Y) if retain_derived: self.X.append(X) self.derived_variables["Y"].append(Y) return a def backward(self, dLda, retain_grads=True): """ 卷积层的反向传播，原理见上文。参数说明： dLda：关于损失的梯度，为 (n_samples, out_rows, out_cols, out_ch) retain_grads：是否计算中间变量的参数梯度，bool型输出说明： dXs：即dX，当前卷积层对输入关于损失的梯度，为 (n_samples, in_rows, in_cols, in_ch) """ if not isinstance(dLda, list): dLda = [dLda] W = self.params["W"] b = self.params["b"] Ys = self.derived_variables["Y"] Xs, d = self.X, self.dilation (fr, fc), s, p = self.kernel_shape, self.stride, self.pad dXs = [] for X, Y, da in zip(Xs, Ys, dLda): n_samp, out_rows, out_cols, out_ch = da.shape X_pad, (pr1, pr2, pc1, pc2) = pad2D(X, p, self.kernel_shape, s, d) dY = da * self.acti_fn.grad(Y) dX = np.zeros_like(X_pad) dW, db = np.zeros_like(W), np.zeros_like(b) for m in range(n_samp): for i in range(out_rows): for j in range(out_cols): for c in range(out_ch): i0, i1 = i * s, (i * s) + fr + (fr-1) * (d-1) j0, j1 = j * s, (j * s) + fc + (fc-1) * (d-1) wc = W[:, :, :, c] kernel = dY[m, i, j, c] window = X_pad[m, i0:i1:d, j0:j1:d, :] db[:, :, :, c] += kernel dW[:, :, :, c] += window * kernel dX[m, i0:i1:d, j0:j1:d, :] += ( wc * kernel ) if retain_grads: self.gradients["W"] += dW self.gradients["b"] += db pr2 = None if pr2 == 0 else -pr2 pc2 = None if pc2 == 0 else -pc2 dXs.append(dX[:, pr1:pr2, pc1:pc2, :]) return dXs[0] if len(Xs) == 1 else dXs @property def hyperparams(self): return { "layer": "Conv2D", "pad": self.pad, "init_w": self.init_w, "in_ch": self.in_ch, "out_ch": self.out_ch, "stride": self.stride, "dilation": self.dilation, "acti_fn": str(self.acti_fn), "kernel_shape": self.kernel_shape, "optimizer": { "cache": self.optimizer.cache, "hyperparams": self.optimizer.hyperparams, }, } ######### Conv2D GEMM ############# def col2im(X_col, X_shape, W_shape, pad, stride, dilation=0): """ col2im 实现，“col2im”函数将 2D 矩阵变为 4D 图像参数说明： X_col：X 经过 im2col 后 (列) 的矩阵，形状为 (Q, Z)，具体形状见上文 X_shape：原始的输入数组形状，为 (n_samples, in_rows, in_cols, in_ch)，此时还未 0 填充(padding) W_shape：卷积核组形状，4-tuple 为 (kernel_rows, kernel_cols, in_ch, out_ch) pad：padding 数目，4-tuple 在图片的左、右、上、下 (left, right, up, down) 0填充 stride：卷积核的卷积步幅，int型 dilation：扩张率，int 型，default=1 输出说明： img：输出结果，形状为 (n_samples, in_rows, in_cols, in_ch) """ s, d = stride, dilation pr1, pr2, pc1, pc2 = pad fr, fc, n_in, n_out = W_shape n_samp, in_rows, in_cols, n_in = X_shape X_pad = np.zeros((n_samp, n_in, in_rows + pr1 + pr2, in_cols + pc1 + pc2)) k, i, j = _im2col_indices((n_samp, n_in, in_rows, in_cols), fr, fc, pad, s, d) X_col_reshaped = X_col.reshape(n_in * fr * fc, -1, n_samp) X_col_reshaped = X_col_reshaped.transpose(2, 0, 1) np.add.at(X_pad, (slice(None), k, i, j), X_col_reshaped) pr2 = None if pr2 == 0 else -pr2 pc2 = None if pc2 == 0 else -pc2 return X_pad[:, :, pr1:pr2, pc1:pc2] class Conv2D_gemm(LayerBase): def __init__( self, out_ch, kernel_shape, pad=0, stride=1, dilation=1, acti_fn=None, optimizer=None, init_w="glorot_uniform", ): """ 二维卷积参数说明： out_ch：卷积核组的数目，int 型 kernel_shape：单个卷积核形状，2-tuple acti_fn：激活函数，str 型 pad：padding 数目，4-tuple, int, 或 'same'，'valid'型在图片的左、右、上、下 (left, right, up, down) 0填充若为int，表示在左、右、上、下均填充数目为 pad 的 0，若为same，表示填充后为相同 (same) 卷积，若为valid，表示填充后为有效 (valid) 卷积 stride：卷积核的卷积步幅，int型 dilation：扩张率，int 型，default=1 init_w：权重初始化方法，str型 optimizer：优化方法，str型 """ super().__init__(optimizer) self.pad = pad self.in_ch = None self.out_ch = out_ch self.stride = stride self.dilation = dilation self.kernel_shape = kernel_shape self.init_w = init_w self.init_weights = WeightInitializer(mode=init_w) self.acti_fn = ActivationInitializer(acti_fn)() self.parameters = {"W": None, "b": None} self.is_initialized = False def _init_params(self): fr, fc = self.kernel_shape W = self.init_weights((fr, fc, self.in_ch, self.out_ch)) b = np.zeros((1, 1, 1, self.out_ch)) self.params = {"W": W, "b": b} self.gradients = {"W": np.zeros_like(W), "b": np.zeros_like(b)} self.derived_variables = {"Y": []} self.is_initialized = True def forward(self, X, retain_derived=True): """ 卷积层的前向传播，原理见上文。参数说明： X：输入数组，形状为 (n_samples, in_rows, in_cols, in_ch) retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型输出说明： a：卷积层输出，形状为 (n_samples, out_rows, out_cols, out_ch) """ if not self.is_initialized: self.in_ch = X.shape[3] self._init_params() W = self.params["W"] b = self.params["b"] n_samp, in_rows, in_cols, in_ch = X.shape s, p, d = self.stride, self.pad, self.dilation # 卷积操作 Y = conv2D_gemm(X, W, s, p, d) + b a = self.acti_fn(Y) if retain_derived: self.X.append(X) self.derived_variables["Y"].append(Y) return a def backward(self, dLda, retain_grads=True): """ 卷积层的反向传播，原理见上文。参数说明： dLda：关于损失的梯度，为 (n_samples, out_rows, out_cols, out_ch) retain_grads：是否计算中间变量的参数梯度，bool型输出说明： dX：当前卷积层对输入关于损失的梯度，为 (n_samples, in_rows, in_cols, in_ch) """ if not isinstance(dLda, list): dLda = [dLda] dX = [] X = self.X Y = self.derived_variables["Y"] for da, x, y in zip(dLda, X, Y): dx, dw, db = self._bwd(da, x, y) dX.append(dx) if retain_grads: self.gradients["W"] += dw self.gradients["b"] += db return dX[0] if len(X) == 1 else dX def _bwd(self, dLda, X, Y): W = self.params["W"] d = self.dilation fr, fc, in_ch, out_ch = W.shape n_samp, out_rows, out_cols, out_ch = dLda.shape (fr, fc), s, p = self.kernel_shape, self.stride, self.pad dLdy = dLda * self.acti_fn.grad(Y) dLdy_col = dLdy.transpose(3, 1, 2, 0).reshape(out_ch, -1) W_col = W.transpose(3, 2, 0, 1).reshape(out_ch, -1).T X_col, p = im2col(X, W.shape, p, s, d) dW = (dLdy_col @ X_col.T).reshape(out_ch, in_ch, fr, fc).transpose(2, 3, 1, 0) db = dLdy_col.sum(axis=1).reshape(1, 1, 1, -1) dX_col = W_col @ dLdy_col dX = col2im(dX_col, X.shape, W.shape, p, s, d).transpose(0, 2, 3, 1) return dX, dW, db @property def hyperparams(self): return { "layer": "Conv2D", "pad": self.pad, "init_w": self.init_w, "in_ch": self.in_ch, "out_ch": self.out_ch, "stride": self.stride, "dilation": self.dilation, "acti_fn": str(self.acti_fn), "kernel_shape": self.kernel_shape, "optimizer": { "cache": self.optimizer.cache, "hyperparams": self.optimizer.hyperparams, }, } ######## Pool2D ################ class Pool2D(LayerBase): def __init__(self, kernel_shape, stride=1, pad=0, mode="max", optimizer=None): """ 二维池化参数说明： kernel_shape：池化窗口的大小，2-tuple stride：和卷积类似，窗口在每一个维度上滑动的步长，int型 pad：padding 数目，4-tuple, int, 或 str('same','valid')型 (default: 0) 和卷积类似 mode：池化函数，str型 (default: 'max')，可选{"max","average"} optimizer：优化方法，str型 """ super().__init__(optimizer) self.pad = pad self.mode = mode self.in_ch = None self.out_ch = None self.stride = stride self.kernel_shape = kernel_shape self.is_initialized = False def _init_params(self): self.derived_variables = {"out_rows": [], "out_cols": []} self.is_initialized = True def forward(self, X, retain_derived=True): """ 池化层前向传播参数说明： X：输入数组，形状为 (n_samp, in_rows, in_cols, in_ch) retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型输出说明： Y：输出结果，形状为 (n_samp, out_rows, out_cols, out_ch) """ if not self.is_initialized: self.in_ch = self.out_ch = X.shape[3] self._init_params() n_samp, in_rows, in_cols, nc_in = X.shape (fr, fc), s, p = self.kernel_shape, self.stride, self.pad X_pad, (pr1, pr2, pc1, pc2) = pad2D(X, p, self.kernel_shape, s) out_rows = int((in_rows + pr1 + pr2 - fr) / s + 1) out_cols = int((in_cols + pc1 + pc2 - fc) / s + 1) if self.mode == "max": pool_fn = np.max elif self.mode == "average": pool_fn = np.mean Y = np.zeros((n_samp, out_rows, out_cols, self.out_ch)) for m in range(n_samp): for i in range(out_rows): for j in range(out_cols): for c in range(self.out_ch): i0, i1 = i * s, (i * s) + fr j0, j1 = j * s, (j * s) + fc xi = X_pad[m, i0:i1, j0:j1, c] Y[m, i, j, c] = pool_fn(xi) if retain_derived: self.X.append(X) self.derived_variables["out_rows"].append(out_rows) self.derived_variables["out_cols"].append(out_cols) return Y def backward(self, dLdy, retain_grads=True): """ 池化层的反向传播，原理见上文。参数说明： dLdy：关于损失的梯度，为 (n_samples, out_rows, out_cols, out_ch) retain_grads：是否计算中间变量的参数梯度，bool型输出说明： dXs：即dX，当前卷积层对输入关于损失的梯度，为 (n_samples, in_rows, in_cols, in_ch) """ if not isinstance(dLdy, list): dLdy = [dLdy] Xs = self.X out_rows = self.derived_variables["out_rows"] out_cols = self.derived_variables["out_cols"] (fr, fc), s, p = self.kernel_shape, self.stride, self.pad dXs = [] for X, dy, out_row, out_col in zip(Xs, dLdy, out_rows, out_cols): n_samp, in_rows, in_cols, nc_in = X.shape X_pad, (pr1, pr2, pc1, pc2) = pad2D(X, p, self.kernel_shape, s) dX = np.zeros_like(X_pad) for m in range(n_samp): for i in range(out_row): for j in range(out_col): for c in range(self.out_ch): i0, i1 = i * s, (i * s) + fr j0, j1 = j * s, (j * s) + fc if self.mode == "max": xi = X[m, i0:i1, j0:j1, c] mask = np.zeros_like(xi).astype(bool) x, y = np.argwhere(xi == np.max(xi))[0] mask[x, y] = True dX[m, i0:i1, j0:j1, c] += mask * dy[m, i, j, c] elif self.mode == "average": frame = np.ones((fr, fc)) * dy[m, i, j, c] dX[m, i0:i1, j0:j1, c] += frame / np.prod((fr, fc)) pr2 = None if pr2 == 0 else -pr2 pc2 = None if pc2 == 0 else -pc2 dXs.append(dX[:, pr1:pr2, pc1:pc2, :]) return dXs[0] if len(Xs) == 1 else dXs @property def hyperparams(self): return { "layer": "Pool2D", "acti_fn": None, "pad": self.pad, "mode": self.mode, "in_ch": self.in_ch, "out_ch": self.out_ch, "stride": self.stride, "kernel_shape": self.kernel_shape, "optimizer": { "cache": self.optimizer.cache, "hyperparams": self.optimizer.hyperparams, }, } ############### Flatten ################## class Flatten(LayerBase): def __init__(self, keep_dim="first", optimizer=None): """ 将多维输入展开参数说明： keep_dim：展开形状，str (default : 'first') 对于输入 X，keep_dim可选 'first'->将 X 重构为(X.shape[0], -1)， 'last'->将 X 重构为(-1, X.shape[0])，'none'->将 X 重构为(1,-1) optimizer：优化方法 """ super().__init__(optimizer) self.keep_dim = keep_dim self._init_params() def _init_params(self): self.X = [] self.gradients = {} self.params = {} self.derived_variables = {"in_dims": []} def forward(self, X, retain_derived=True): """ 前向传播参数说明： X：输入数组 retain_derived：是否保留中间变量，以便反向传播时再次使用，bool型 """ if retain_derived: self.derived_variables["in_dims"].append(X.shape) if self.keep_dim == "none": return X.flatten().reshape(1, -1) rs = (X.shape[0], -1) if self.keep_dim == "first" else (-1, X.shape[-1]) return X.reshape(*rs) def backward(self, dLdy, retain_grads=True): """ 反向传播参数说明： dLdy：关于损失的梯度 retain_grads：是否计算中间变量的参数梯度，bool型输出说明： dX：将对输入的梯度进行重构为原始输入的形状 """ if not isinstance(dLdy, list): dLdy = [dLdy] in_dims = self.derived_variables["in_dims"] dX = [dy.reshape(*dims) for dy, dims in zip(dLdy, in_dims)] return dX[0] if len(dLdy) == 1 else dX @property def hyperparams(self): return { "layer": "Flatten", "keep_dim": self.keep_dim, "optimizer": { "cache": self.optimizer.cache, "hyperparams": self.optimizer.hyperparams, }, } ########### LeNet ################ class LeNet(object): def __init__( self, fc3_out=128, fc4_out=84, fc5_out=10, conv1_pad=0, conv2_pad=0, conv1_out_ch=6, conv2_out_ch=16, conv1_stride=1, pool1_stride=2, conv2_stride=1, pool2_stride=2, conv1_kernel_shape=(5, 5), pool1_kernel_shape=(2, 2), conv2_kernel_shape=(5, 5), pool2_kernel_shape=(2, 2), optimizer="adam", init_w="glorot_normal", loss=CrossEntropy() ): self.optimizer = optimizer self.init_w = init_w self.loss = loss self.fc3_out = fc3_out self.fc4_out = fc4_out self.fc5_out = fc5_out self.conv1_pad = conv1_pad self.conv2_pad = conv2_pad self.conv1_stride = conv1_stride self.conv1_out_ch = conv1_out_ch self.pool1_stride = pool1_stride self.conv2_out_ch = conv2_out_ch self.conv2_stride = conv2_stride self.pool2_stride = pool2_stride self.conv2_kernel_shape = conv2_kernel_shape self.pool2_kernel_shape = pool2_kernel_shape self.conv1_kernel_shape = conv1_kernel_shape self.pool1_kernel_shape = pool1_kernel_shape self.is_initialized = False def _set_params(self): """ 函数作用：模型初始化 Conv1 -> Pool1 -> Conv2 -> Pool2 -> Flatten -> FC3 -> FC4 -> FC5 -> Softmax """ self.layers = OrderedDict() self.layers["Conv1"] = Conv2D( out_ch=self.conv1_out_ch, kernel_shape=self.conv1_kernel_shape, pad=self.conv1_pad, stride=self.conv1_stride, acti_fn="sigmoid", optimizer=self.optimizer, init_w=self.init_w, ) self.layers["Pool1"] = Pool2D( mode="max", optimizer=self.optimizer, stride=self.pool1_stride, kernel_shape=self.pool1_kernel_shape, ) self.layers["Conv2"] = Conv2D( out_ch=self.conv1_out_ch, kernel_shape=self.conv1_kernel_shape, pad=self.conv1_pad, stride=self.conv1_stride, acti_fn="sigmoid", optimizer=self.optimizer, init_w=self.init_w, ) self.layers["Pool2"] = Pool2D( mode="max", optimizer=self.optimizer, stride=self.pool2_stride, kernel_shape=self.pool2_kernel_shape, ) self.layers["Flatten"] = Flatten(optimizer=self.optimizer) self.layers["FC3"] = FullyConnected( n_out=self.fc3_out, acti_fn="sigmoid", init_w=self.init_w, optimizer=self.optimizer ) self.layers["FC4"] = FullyConnected( n_out=self.fc4_out, acti_fn="sigmoid", init_w=self.init_w, optimizer=self.optimizer ) self.layers["FC5"] = FullyConnected( n_out=self.fc5_out, acti_fn="affine(slope=1, intercept=0)", init_w=self.init_w, optimizer=self.optimizer ) self.is_initialized = True def forward(self, X_train): Xs = {} out = X_train for k, v in self.layers.items(): Xs[k] = out out = v.forward(out) return out, Xs def backward(self, grad): dXs = {} out = grad for k, v in reversed(list(self.layers.items())): dXs[k] = out out = v.backward(out) return out, dXs def update(self): """ 函数作用：梯度更新 """ for k, v in reversed(list(self.layers.items())): v.update() self.flush_gradients() def flush_gradients(self, curr_loss=None): """ 函数作用：更新后重置梯度 """ for k, v in self.layers.items(): v.flush_gradients() def fit(self, X_train, y_train, n_epochs=20, batch_size=64, verbose=False, epo_verbose=True): """ 参数说明： X_train：训练数据 y_train：训练数据标签 n_epochs：epoch 次数 batch_size：每次 epoch 的 batch size verbose：是否每个 batch 输出损失 epo_verbose：是否每个 epoch 输出损失 """ self.verbose = verbose self.n_epochs = n_epochs self.batch_size = batch_size if not self.is_initialized: self.n_features = X_train.shape[1] self._set_params() prev_loss = np.inf for i in range(n_epochs): loss, epoch_start = 0.0, time.time() batch_generator, n_batch = minibatch(X_train, self.batch_size, shuffle=True) for j, batch_idx in enumerate(batch_generator): batch_len, batch_start = len(batch_idx), time.time() X_batch, y_batch = X_train[batch_idx], y_train[batch_idx] out, _ = self.forward(X_batch) y_pred_batch = softmax(out) batch_loss = self.loss(y_batch, y_pred_batch) grad = self.loss.grad(y_batch, y_pred_batch) _, _ = self.backward(grad) self.update() loss += batch_loss if self.verbose: fstr = "\t[Batch {}/{}] Train loss: {:.3f} ({:.1f}s/batch)" print(fstr.format(j + 1, n_batch, batch_loss, time.time() - batch_start)) loss /= n_batch if epo_verbose: fstr = "[Epoch {}] Avg. loss: {:.3f} Delta: {:.3f} ({:.2f}m/epoch)" print(fstr.format(i + 1, loss, prev_loss - loss, (time.time() - epoch_start) / 60.0)) prev_loss = loss def evaluate(self, X_test, y_test, batch_size=128): acc = 0.0 batch_generator, n_batch = minibatch(X_test, batch_size, shuffle=True) for j, batch_idx in enumerate(batch_generator): batch_len, batch_start = len(batch_idx), time.time() X_batch, y_batch = X_test[batch_idx], y_test[batch_idx] y_pred_batch, _ = self.forward(X_batch) y_pred_batch = np.argmax(y_pred_batch, axis=1) y_batch = np.argmax(y_batch, axis=1) acc += np.sum(y_pred_batch == y_batch) return acc / X_test.shape[0] @property def hyperparams(self): return { "init_w": self.init_w, "loss": str(self.loss), "optimizer": self.optimizer, "fc3_out": self.fc3_out, "fc4_out": self.fc4_out, "fc5_out": self.fc5_out, "conv1_pad": self.conv1_pad, "conv2_pad": self.conv2_pad, "conv1_stride": self.conv1_stride, "conv1_out_ch": self.conv1_out_ch, "pool1_stride": self.pool1_stride, "conv2_out_ch": self.conv2_out_ch, "conv2_stride": self.conv2_stride, "pool2_stride": self.pool2_stride, "conv2_kernel_shape": self.conv2_kernel_shape, "pool2_kernel_shape": self.pool2_kernel_shape, "conv1_kernel_shape": self.conv1_kernel_shape, "pool1_kernel_shape": self.pool1_kernel_shape, "components": {k: v.params for k, v in self.layers.items()} } ############# LeNet GEMM ################ class LeNet_gemm(object): def __init__( self, fc3_out=128, fc4_out=84, fc5_out=10, conv1_pad=0, conv2_pad=0, conv1_out_ch=6, conv2_out_ch=16, conv1_stride=1, pool1_stride=2, conv2_stride=1, pool2_stride=2, conv1_kernel_shape=(5, 5), pool1_kernel_shape=(2, 2), conv2_kernel_shape=(5, 5), pool2_kernel_shape=(2, 2), optimizer="adam", init_w="glorot_normal", loss=CrossEntropy() ): self.optimizer = optimizer self.init_w = init_w self.loss = loss self.fc3_out = fc3_out self.fc4_out = fc4_out self.fc5_out = fc5_out self.conv1_pad = conv1_pad self.conv2_pad = conv2_pad self.conv1_stride = conv1_stride self.conv1_out_ch = conv1_out_ch self.pool1_stride = pool1_stride self.conv2_out_ch = conv2_out_ch self.conv2_stride = conv2_stride self.pool2_stride = pool2_stride self.conv2_kernel_shape = conv2_kernel_shape self.pool2_kernel_shape = pool2_kernel_shape self.conv1_kernel_shape = conv1_kernel_shape self.pool1_kernel_shape = pool1_kernel_shape self.is_initialized = False def _set_params(self): """ 函数作用：模型初始化 Conv1 -> Pool1 -> Conv2 -> Pool2 -> Flatten -> FC3 -> FC4 -> FC5 -> Softmax """ self.layers = OrderedDict() self.layers["Conv1"] = Conv2D_gemm( out_ch=self.conv1_out_ch, kernel_shape=self.conv1_kernel_shape, pad=self.conv1_pad, stride=self.conv1_stride, acti_fn="sigmoid", optimizer=self.optimizer, init_w=self.init_w, ) self.layers["Pool1"] = Pool2D( mode="max", optimizer=self.optimizer, stride=self.pool1_stride, kernel_shape=self.pool1_kernel_shape, ) self.layers["Conv2"] = Conv2D_gemm( out_ch=self.conv1_out_ch, kernel_shape=self.conv1_kernel_shape, pad=self.conv1_pad, stride=self.conv1_stride, acti_fn="sigmoid", optimizer=self.optimizer, init_w=self.init_w, ) self.layers["Pool2"] = Pool2D( mode="max", optimizer=self.optimizer, stride=self.pool2_stride, kernel_shape=self.pool2_kernel_shape, ) self.layers["Flatten"] = Flatten(optimizer=self.optimizer) self.layers["FC3"] = FullyConnected( n_out=self.fc3_out, acti_fn="sigmoid", init_w=self.init_w, optimizer=self.optimizer ) self.layers["FC4"] = FullyConnected( n_out=self.fc4_out, acti_fn="sigmoid", init_w=self.init_w, optimizer=self.optimizer ) self.layers["FC5"] = FullyConnected( n_out=self.fc5_out, acti_fn="affine(slope=1, intercept=0)", init_w=self.init_w, optimizer=self.optimizer ) self.is_initialized = True def forward(self, X_train): Xs = {} out = X_train for k, v in self.layers.items(): Xs[k] = out out = v.forward(out) return out, Xs def backward(self, grad): dXs = {} out = grad for k, v in reversed(list(self.layers.items())): dXs[k] = out out = v.backward(out) return out, dXs def update(self): """ 函数作用：梯度更新 """ for k, v in reversed(list(self.layers.items())): v.update() self.flush_gradients() def flush_gradients(self, curr_loss=None): """ 函数作用：更新后重置梯度 """ for k, v in self.layers.items(): v.flush_gradients() def fit(self, X_train, y_train, n_epochs=20, batch_size=64, verbose=False, epo_verbose=True): """ 参数说明： X_train：训练数据 y_train：训练数据标签 n_epochs：epoch 次数 batch_size：每次 epoch 的 batch size verbose：是否每个 batch 输出损失 epo_verbose：是否每个 epoch 输出损失 """ self.verbose = verbose self.n_epochs = n_epochs self.batch_size = batch_size if not self.is_initialized: self.n_features = X_train.shape[1] self._set_params() prev_loss = np.inf for i in range(n_epochs): loss, epoch_start = 0.0, time.time() batch_generator, n_batch = minibatch(X_train, self.batch_size, shuffle=True) for j, batch_idx in enumerate(batch_generator): batch_len, batch_start = len(batch_idx), time.time() X_batch, y_batch = X_train[batch_idx], y_train[batch_idx] out, _ = self.forward(X_batch) y_pred_batch = softmax(out) batch_loss = self.loss(y_batch, y_pred_batch) grad = self.loss.grad(y_batch, y_pred_batch) _, _ = self.backward(grad) self.update() loss += batch_loss if self.verbose: fstr = "\t[Batch {}/{}] Train loss: {:.3f} ({:.1f}s/batch)" print(fstr.format(j + 1, n_batch, batch_loss, time.time() - batch_start)) loss /= n_batch if epo_verbose: fstr = "[Epoch {}] Avg. loss: {:.3f} Delta: {:.3f} ({:.2f}m/epoch)" print(fstr.format(i + 1, loss, prev_loss - loss, (time.time() - epoch_start) / 60.0)) prev_loss = loss def evaluate(self, X_test, y_test, batch_size=128): acc = 0.0 batch_generator, n_batch = minibatch(X_test, batch_size, shuffle=True) for j, batch_idx in enumerate(batch_generator): batch_len, batch_start = len(batch_idx), time.time() X_batch, y_batch = X_test[batch_idx], y_test[batch_idx] y_pred_batch, _ = self.forward(X_batch) y_pred_batch = np.argmax(y_pred_batch, axis=1) y_batch = np.argmax(y_batch, axis=1) acc += np.sum(y_pred_batch == y_batch) return acc / X_test.shape[0] @property def hyperparams(self): return { "init_w": self.init_w, "loss": str(self.loss), "optimizer": self.optimizer, "fc3_out": self.fc3_out, "fc4_out": self.fc4_out, "fc5_out": self.fc5_out, "conv1_pad": self.conv1_pad, "conv2_pad": self.conv2_pad, "conv1_stride": self.conv1_stride, "conv1_out_ch": self.conv1_out_ch, "pool1_stride": self.pool1_stride, "conv2_out_ch": self.conv2_out_ch, "conv2_stride": self.conv2_stride, "pool2_stride": self.pool2_stride, "conv2_kernel_shape": self.conv2_kernel_shape, "pool2_kernel_shape": self.pool2_kernel_shape, "conv1_kernel_shape": self.conv1_kernel_shape, "pool1_kernel_shape": self.pool1_kernel_shape, "components": {k: v.params for k, v in self.layers.items()} } ================================================ FILE: code/method/__init__.py ================================================ from . import optimizer from . import activation ================================================ FILE: code/method/activation/activation.py ================================================ from abc import ABC, abstractmethod import numpy as np import re class ActivationBase(ABC): def __init__(self, **kwargs): super().__init__() def __call__(self, z): if z.ndim == 1: z = z.reshape(1, -1) return self.forward(z) @abstractmethod def forward(self, z): raise NotImplementedError @abstractmethod def grad(self, x, **kwargs): raise NotImplementedError class Sigmoid(ActivationBase): """ Sigmoid(x) = 1 / (1 + e^(-x)) """ def __init__(self): super().__init__() def __str__(self): return "Sigmoid" def forward(self, z): return 1 / (1 + np.exp(-z)) def grad(self, x): return self.forward(x) * (1 - self.forward(x)) class Tanh(ActivationBase): """ Tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x)) """ def __init__(self): super().__init__() def __str__(self): return "Tanh" def forward(self, z): return np.tanh(z) def grad(self, x): return 1 - np.tanh(x) ** 2 class ReLU(ActivationBase): """ ReLU(x) = x if x > 0 0 otherwise """ def __init__(self): super().__init__() def __str__(self): return "ReLU" def forward(self, z): return np.clip(z, 0, np.inf) def grad(self, x): return (x > 0).astype(int) class LeakyReLU(ActivationBase): """ LeakyReLU(x) = alpha * x if x < 0 x otherwise """ def __init__(self, alpha=0.3): self.alpha = alpha super().__init__() def __str__(self): return "Leaky ReLU(alpha={})".format(self.alpha) def forward(self, z): _z = z.copy() _z[z < 0] = _z[z < 0] * self.alpha return _z def grad(self, x): out = np.ones_like(x) out[x < 0] *= self.alpha return out class Affine(ActivationBase): """ Affine(x) = slope * x + intercept """ def __init__(self, slope=1, intercept=0): self.slope = slope self.intercept = intercept super().__init__() def __str__(self): return "Affine(slope={}, intercept={})".format(self.slope, self.intercept) def forward(self, z): return self.slope * z + self.intercept def grad(self, x): return self.slope * np.ones_like(x) class SoftPlus(ActivationBase): """ SoftPlus(x) = log(1 + e^x) """ def __init__(self): super().__init__() def __str__(self): return "SoftPlus" def forward(self, z): return np.log(np.exp(z) + 1) def grad(self, x): return np.exp(x) / (np.exp(x) + 1) class ELU(ActivationBase): """ ELU(x) = x if x >= 0 alpha * (e^x - 1) otherwise """ def __init__(self, alpha=1.0): self.alpha = alpha super().__init__() def __str__(self): return "ELU(alpha={})".format(self.alpha) def forward(self, z): return np.where(z > 0, z, self.alpha * (np.exp(z) - 1)) def grad(self, x): return np.where(x >= 0, np.ones_like(x), self.alpha * np.exp(x)) class Exponential(ActivationBase): """ Exponential(x) = e^x """ def __init__(self): super().__init__() def __str__(self): return "Exponential" def forward(self, z): return np.exp(z) def grad(self, x): return np.exp(x) class SELU(ActivationBase): """ SELU(x) = scale * ELU(x, alpha) = scale * x if x >= 0 scale * [alpha * (e^x - 1)] otherwise """ def __init__(self): self.alpha = 1.6732632423543772848170429916717 self.scale = 1.0507009873554804934193349852946 self.elu = ELU(alpha=self.alpha) super().__init__() def __str__(self): return "SELU" def forward(self, z): return self.scale * self.elu.forward(z) def grad(self, x): return np.where( x >= 0, np.ones_like(x) * self.scale, np.exp(x) * self.alpha * self.scale ) class HardSigmoid(ActivationBase): """ HardSigmoid(x) = 0 if x < -2.5 0.2 * x + 0.5 if -2.5 <= x <= 2.5. 1 if x > 2.5 """ def __init__(self): super().__init__() def __str__(self): return "Hard Sigmoid" def forward(self, z): return np.clip((0.2 * z) + 0.5, 0.0, 1.0) def grad(self, x): return np.where((x >= -2.5) & (x <= 2.5), 0.2, 0) class ActivationInitializer(object): def __init__(self, acti_name="affine(slope=1, intercept=0)"): self.acti_name = acti_name def __call__(self): acti_str = self.acti_name.lower() if acti_str == "relu": acti_fn = ReLU() elif acti_str == "tanh": acti_fn = Tanh() elif acti_str == "sigmoid": acti_fn = Sigmoid() elif "affine" in acti_str: r = r"affine\(slope=(.*), intercept=(.*)\)" slope, intercept = re.match(r, acti_str).groups() acti_fn = Affine(float(slope), float(intercept)) elif "leaky relu" in acti_str: r = r"leaky relu\(alpha=(.*)\)" alpha = re.match(r, acti_str).groups()[0] acti_fn = LeakyReLU(float(alpha)) else: raise ValueError("Unknown activation: {}".format(acti_str)) return acti_fn ================================================ FILE: code/method/optimizer/optimizer.py ================================================ from abc import ABC, abstractmethod import numpy as np import re class OptimizerBase(ABC): def __init__(self): pass def __call__(self, params, params_grad, params_name): """ 参数说明： params：待更新参数，如权重矩阵 W； params_grad：待更新参数的梯度； params_name：待更新参数名； """ return self.update(params, params_grad, params_name) @abstractmethod def update(self, params, params_grad, params_name): raise NotImplementedError class SGD(OptimizerBase): """ sgd 优化方法 """ def __init__(self, lr=0.01): super().__init__() self.lr = lr self.cache = {} def __str__(self): return "SGD(lr={})".format(self.hyperparams["lr"]) def update(self, params, params_grad, params_name): update_value = self.lr * params_grad return params - update_value @property def hyperparams(self): return { "op": "SGD", "lr": self.lr } class Momentum(OptimizerBase): def __init__( self, lr=0.001, momentum=0.0, **kwargs ): """ 参数说明： lr：学习率，float (default: 0.001) momentum：考虑 Momentum 时的 alpha，决定了之前的梯度贡献衰减得有多快，取值范围[0, 1]，默认0 """ super().__init__() self.lr = lr self.momentum = momentum self.cache = {} def __str__(self): return "Momentum(lr={}, momentum={})".format(self.lr, self.momentum) def update(self, param, param_grad, param_name): C = self.cache lr, momentum = self.lr, self.momentum if param_name not in C: # save v C[param_name] = np.zeros_like(param_grad) update = momentum * C[param_name] - lr * param_grad self.cache[param_name] = update return param + update @property def hyperparams(self): return { "op": "Momentum", "lr": self.lr, "momentum": self.momentum } class AdaGrad(OptimizerBase): def __init__(self, lr=0.001, eps=1e-7, **kwargs): """ 参数说明： lr：学习率，float (default: 0.001) eps：delta 项，防止分母为0 """ super().__init__() self.lr = lr self.eps = eps self.cache = {} def __str__(self): return "AdaGrad(lr={}, eps={})".format(self.lr, self.eps) def update(self, param, param_grad, param_name): C = self.cache lr, eps = self.hyperparams["lr"], self.hyperparams["eps"] if param_name not in C: # save r C[param_name] = np.zeros_like(param_grad) C[param_name] += param_grad ** 2 update = lr * param_grad / (np.sqrt(C[param_name]) + eps) self.cache = C return param - update @property def hyperparams(self): return { "op": "AdaGrad", "lr": self.lr, "eps": self.eps } class RMSProp(OptimizerBase): def __init__( self, lr=0.001, decay=0.9, eps=1e-7, **kwargs ): """ 参数说明： lr：学习率，float (default: 0.001) eps：delta 项，防止分母为0 decay：衰减速率 """ super().__init__() self.lr = lr self.eps = eps self.decay = decay self.cache = {} def __str__(self): return "RMSProp(lr={}, eps={}, decay={})".format( self.lr, self.eps, self.decay ) def update(self, param, param_grad, param_name): C = self.cache lr, eps = self.hyperparams["lr"], self.hyperparams["eps"] decay = self.hyperparams["decay"] if param_name not in C: # save r C[param_name] = np.zeros_like(param_grad) C[param_name] = decay * C[param_name] + (1 - decay) * param_grad ** 2 update = lr * param_grad / (np.sqrt(C[param_name]) + eps) self.cache = C return param - update @property def hyperparams(self): return { "op": "RMSProp", "lr": self.lr, "eps": self.eps, "decay": self.decay } class AdaDelta(OptimizerBase): def __init__( self, lr=0.001, decay=0.95, eps=1e-7, **kwargs ): """ 参数说明： lr：学习率，float (default: 0.001) eps：delta 项，防止分母为0 decay：衰减速率 """ super().__init__() self.lr = lr self.eps = eps self.decay = decay self.cache = {} def __str__(self): return "AdaDelta(eps={}, decay={})".format(self.eps, self.decay) def update(self, param, param_grad, param_name): C = self.cache eps = self.hyperparams["eps"] decay = self.hyperparams["decay"] if param_name not in C: # save r, delta_theta C[param_name] = { "r": np.zeros_like(param_grad), "d": np.zeros_like(param_grad) } C[param_name]["r"] = decay * C[param_name]["r"] + (1 - decay) * param_grad ** 2 update = (np.sqrt(C[param_name]["d"] + eps)) * param_grad / (np.sqrt(C[param_name]["r"]) + eps) C[param_name]["d"] = decay * C[param_name]["d"] + (1 - decay) * update ** 2 self.cache = C return param - update @property def hyperparams(self): return { "op": "AdaDelta", "eps": self.eps, "decay": self.decay } class Adam(OptimizerBase): def __init__( self, lr=0.001, decay1=0.9, decay2=0.999, eps=1e-7, **kwargs ): """ 参数说明： lr：学习率，float (default: 0.01) eps：delta 项，防止分母为0 decay1：历史梯度的指数衰减速率，可以理解为考虑梯度均值 (default: 0.9) decay2：历史梯度平方的指数衰减速率，可以理解为考虑梯度方差 (default: 0.999) """ super().__init__() self.lr = lr self.decay1 = decay1 self.decay2 = decay2 self.eps = eps self.cache = {} def __str__(self): return "Adam(lr={}, decay1={}, decay2={}, eps={})".format( self.lr, self.decay1, self.decay2, self.eps ) def update(self, param, param_grad, param_name, cur_loss=None): C = self.cache d1, d2 = self.hyperparams["decay1"], self.hyperparams["decay2"] lr, eps= self.hyperparams["lr"], self.hyperparams["eps"] if param_name not in C: C[param_name] = { "t": 0, "mean": np.zeros_like(param_grad), "var": np.zeros_like(param_grad), } t = C[param_name]["t"] + 1 mean = C[param_name]["mean"] var = C[param_name]["var"] C[param_name]["t"] = t C[param_name]["mean"] = d1 * mean + (1 - d1) * param_grad C[param_name]["var"] = d2 * var + (1 - d2) * param_grad ** 2 self.cache = C m_hat = C[param_name]["mean"] / (1 - d1 ** t) v_hat = C[param_name]["var"] / (1 - d2 ** t) update = lr * m_hat / (np.sqrt(v_hat) + eps) return param - update @property def hyperparams(self): return { "op": "Adam", "lr": self.lr, "eps": self.eps, "decay1": self.decay1, "decay2": self.decay2 } class OptimizerInitializer(ABC): def __init__(self, opti_name="sgd"): self.opti_name = opti_name def __call__(self): r = r"([a-zA-Z]*)=([^,)]*)" opti_str = self.opti_name.lower() kwargs = dict([(i, eval(j)) for (i, j) in re.findall(r, opti_str)]) if "sgd" in opti_str: optimizer = SGD(**kwargs) elif "momentum" in opti_str: optimizer = Momentum(**kwargs) elif "adagrad" in opti_str: optimizer = AdaGrad(**kwargs) elif "rmsprop" in opti_str: optimizer = RMSProp(**kwargs) elif "adadelta" in opti_str: optimizer = AdaDelta(**kwargs) elif "adam" in opti_str: optimizer = Adam(**kwargs) else: raise NotImplementedError("{}".format(opt_str)) return optimizer ================================================ FILE: code/method/weight/weight.py ================================================ from abc import ABC, abstractmethod import numpy as np import re def calc_fan(weight_shape): """ 对权重矩阵计算 fan-in 和 fan-out 参数说明： weight_shape：权重形状 """ if len(weight_shape) == 2: fan_in, fan_out = weight_shape elif len(weight_shape) in [3, 4]: in_ch, out_ch = weight_shape[-2:] kernel_size = np.prod(weight_shape[:-2]) fan_in, fan_out = in_ch * kernel_size, out_ch * kernel_size else: raise ValueError("Unrecognized weight dimension: {}".format(weight_shape)) return fan_in, fan_out class random_uniform: """ 初始化网络权重 W--- 基于 Uniform(-b, b) 参数说明： weight_shape：权重形状 """ def __init__(self, b=1.0): self.b = b def __call__(self, weight_shape): return np.random.uniform(-b, b, size=weight_shape) class random_normal: """ 初始化网络权重 W--- 基于 TruncatedNormal(0, std) 参数说明： weight_shape：权重形状 std：权重标准差 """ def __init__(self, std=0.01): self.std = std def __call__(self, weight_shape): return truncated_normal(0, std, weight_shape) # def random_uniform(weight_shape, b=1.0): # """ # 初始化网络权重 W--- 基于 Uniform(-b, b) # 参数说明： # weight_shape：权重形状 # """ # return np.random.uniform(-b, b, size=weight_shape) # def random_normal(weight_shape, std=1.0): # """ # 初始化网络权重 W--- 基于 TruncatedNormal(0, std) # 参数说明： # weight_shape：权重形状 # std：权重标准差 # """ # return truncated_normal(0, std, weight_shape) class he_uniform: """ 初始化网络权重 W--- 基于 Uniform(-b, b)，其中 b=sqrt(6/fan_in)，常用于 ReLU 激活层参数说明： weight_shape：权重形状 """ def __init__(self): pass def __call__(self, weight_shape): fan_in, fan_out = calc_fan(weight_shape) b = np.sqrt(6 / fan_in) return np.random.uniform(-b, b, size=weight_shape) class he_normal: """ 初始化网络权重 W--- 基于 TruncatedNormal(0, std)，其中 std=2/fan_in，常用于 ReLU 激活层参数说明： weight_shape：权重形状 """ def __init__(self): pass def __call__(self, weight_shape): fan_in, fan_out = calc_fan(weight_shape) std = np.sqrt(2 / fan_in) return truncated_normal(0, std, weight_shape) # def he_uniform(weight_shape): # """ # 初始化网络权重 W--- 基于 Uniform(-b, b)，其中 b=sqrt(6/fan_in)，常用于 ReLU 激活层 # 参数说明： # weight_shape：权重形状 # """ # fan_in, fan_out = calc_fan(weight_shape) # b = np.sqrt(6 / fan_in) # return np.random.uniform(-b, b, size=weight_shape) # def he_normal(weight_shape): # """ # 初始化网络权重 W--- 基于 TruncatedNormal(0, std)，其中 std=2/fan_in，常用于 ReLU 激活层 # 参数说明： # weight_shape：权重形状 # """ # fan_in, fan_out = calc_fan(weight_shape) # std = np.sqrt(2 / fan_in) # return truncated_normal(0, std, weight_shape) class glorot_uniform: """ 初始化网络权重 W--- 基于 Uniform(-b, b)，其中 b=gain*sqrt(6/(fan_in+fan_out))，常用于 tanh 和 sigmoid 激活层参数说明： weight_shape：权重形状 """ def __init__(self, gain=1.0): self.gain = gain def __call__(self, weight_shape): fan_in, fan_out = calc_fan(weight_shape) b = self.gain * np.sqrt(6 / (fan_in + fan_out)) return np.random.uniform(-b, b, size=weight_shape) class glorot_normal: """ 初始化网络权重 W--- 基于 TruncatedNormal(0, std)，其中 std=gain^2*2/(fan_in+fan_out)，常用于 tanh 和 sigmoid 激活层参数说明： weight_shape：权重形状 """ def __init__(self, gain=1.0): self.gain = gain def __call__(self, weight_shape): fan_in, fan_out = calc_fan(weight_shape) std = self.gain * np.sqrt(2 / (fan_in + fan_out)) return truncated_normal(0, std, weight_shape) # def glorot_uniform(weight_shape, gain=1.0): # """ # 初始化网络权重 W--- 基于 Uniform(-b, b)，其中 b=gain*sqrt(6/(fan_in+fan_out))， # 常用于 tanh 和 sigmoid 激活层 # 参数说明： # weight_shape：权重形状 # """ # fan_in, fan_out = calc_fan(weight_shape) # b = gain * np.sqrt(6 / (fan_in + fan_out)) # return np.random.uniform(-b, b, size=weight_shape) # def glorot_normal(weight_shape, gain=1.0): # """ # 初始化网络权重 W--- 基于 TruncatedNormal(0, std)，其中 std=gain^2*2/(fan_in+fan_out)， # 常用于 tanh 和 sigmoid 激活层 # 参数说明： # weight_shape：权重形状 # """ # fan_in, fan_out = calc_fan(weight_shape) # std = gain * np.sqrt(2 / (fan_in + fan_out)) # return truncated_normal(0, std, weight_shape) def truncated_normal(mean, std, out_shape): """ 通过拒绝采样生成截断正态分布参数说明： mean：正态分布均值 std：正态分布标准差 out_shape：矩阵形状 """ samples = np.random.normal(loc=mean, scale=std, size=out_shape) reject = np.logical_or(samples >= mean + 2 * std, samples <= mean - 2 * std) while any(reject.flatten()): resamples = np.random.normal(loc=mean, scale=std, size=reject.sum()) samples[reject] = resamples reject = np.logical_or(samples >= mean + 2 * std, samples <= mean - 2 * std) return samples class WeightInitializer(object): def __init__(self, mode="he_normal"): """ mode：权重初始化策略 str型 (default: 'he_normal') """ self.mode = mode r = r"([a-zA-Z]*)=([^,)]*)" mode_str = self.mode.lower() kwargs = dict([(i, eval(j)) for (i, j) in re.findall(r, mode_str)]) if "random_uniform" in mode_str: self.init_fn = random_uniform(**kwargs) elif "random_normal" in mode_str: self.init_fn = random_normal(**kwargs) elif "he_uniform" in mode_str: self.init_fn = he_uniform(**kwargs) elif "he_normal" in mode_str: self.init_fn = he_normal(**kwargs) elif "glorot_uniform" in mode_str: self.init_fn = glorot_uniform(**kwargs) elif "glorot_normal" in mode_str: self.init_fn = glorot_normal(**kwargs) else: raise ValueError("Unrecognize initialization mode: {}".format(mode_str)) def __call__(self, weight_shape): W = self.init_fn(weight_shape) return W ================================================ FILE: contents.txt ================================================ 注：目录是基于《深度学习》的目录起的。基于本项目的内容，目录其实可以分的更细致，这里就分到目录的第三级为止。 **目录**: - 第二章线性代数 - 1 标量, 向量, 矩阵, 张量 - 2 矩阵转置 - 3 矩阵加法 - 4 矩阵乘法 - 5 单位矩阵 - 6 矩阵的逆 - 7 范数 - 8 特征值分解 - 9 奇异值分解 - 10 PCA (主成分分析) - 第三章概率与信息论 - 1 概率 - 1.1 概率与随机变量 - 1.2 概率分布 - 1.2.1 概率质量函数 - 1.2.2 概率密度函数 - 1.2.3 累积分布函数 - 1.3 条件概率与条件独立 - 1.4 随机变量的度量 - 1.5 常用概率分布 - 1.5.1 伯努利分布 (两点分布) - 1.5.2 范畴分布 (分类分布) - 1.5.3 高斯分布 (正态分布) - 1.5.4 多元高斯分布 (多元正态分布) - 1.5.5 指数分布 - 1.5.6 拉普拉斯分布 - 1.5.7 Dirac 分布 - 1.6 常用函数的有用性质 - 1.6.1 logistic sigmoid 函数 - 1.6.2 softplus 函数 - 2 信息论 - 3 图模型 - 3.1 有向图模型 - 3.1.1 贝叶斯网的独立性 - 3.2 无向图模型 - 3.1.2 马尔可夫网的条件独立性 - 第四章数值计算 - 1 上溢和下溢 - 2 优化方法 - 2.1 梯度下降法 - 2.2 牛顿法 - 2.3 约束优化 - 第五章机器学习基础 - 1 学习算法 - 1.1 举例:线性回归 - 2 容量、过拟合、欠拟合 - 2.1 泛化问题 - 2.2 容量 - 3 超参数与验证集 - 4 偏差和方差 - 4.1 偏差 - 4.2 方差 - 4.3 误差与偏差和方差的关系 - 5 最大似然估计 - 6 贝叶斯统计 - 7 最大后验估计 - 7.1 举例:线性回归 - 8 监督学习方法 - 8.1 概率监督学习 - 8.2 支持向量机 - 8.2.1 核技巧 - 8.3 k-近邻 - 8.4 决策树 - 8.4.1 特征选择 - 8.4.2 决策树生成 - 8.4.3 决策树正则化 - 9 无监督学习方法 - 9.1 主成分分析法 - 9.2 k-均值聚类 - 第六章深度前馈网络 - 1 深度前馈网络 - 2 DFN 相关设计 - 2.1 隐藏单元 - 2.2 输出单元 - 2.3 代价函数 - 2.4 架构设计 - 3 反向传播算法 - 3.1 单个神经元的训练 - 3.2 多层神经网络的训练 - 3.2.1 定义权重初始化方法 - 3.2.2 定义激活函数 - 3.2.3 定义优化方法 - 3.2.4 定义网络层的框架 - 3.2.5 定义代价函数 - 3.2.6 定义深度前馈网络 - 4 神经网络的万能近似定理 - 5 实例:学习 XOR - 第七章深度学习中的正则化 - 1 参数范数惩罚 - 1.1 L2 正则化 - 1.2 L1 正则化 - 1.3 总结 (L2 正则化与L1 正则化的解) - 1.4 作为约束的范数惩罚 - 1.5 欠约束问题 - 2 数据增强 - 2.1 数据集增强 - 2.2 噪声鲁棒性 - 3 训练方案 - 3.1 半监督学习 - 3.2 多任务学习 - 3.3 提前终止 - 4 模型表示 - 4.1 参数绑定与共享 - 4.2 稀疏表示 - 4.3 Bagging 及其他集成方法 - 4.3.1 Bagging 方法 - 4.3.2 随机森林 - 4.3.3 方法解决过拟合 - 4.4 Dropout - 5 样本测试 - 6 补充材料 - 6.1 Boosting - 6.1.1 前向分步加法模型 - 6.1.2 AdaBoost 算法 - 6.1.3 Boosting Tree 算法与 GBDT 算法 - 6.1.4 XGBoost 算法 - 第八章深度模型中的优化 - 1 基本优化算法 - 1.1 梯度 - 1.1.1 梯度下降 - 1.1.2 随机梯度下降 - 1.2 动量 - 1.2.1 Momentum 算法 - 1.2.2 NAG 算法 - 1.3 自适应学习率 - 1.3.1 AdaGrad 算法 - 1.3.2 RMSProp 算法 - 1.3.3 AdaDelta 算法 - 1.3.4 Adam 算法 - 1.4 二阶近似方法 - 1.4.1 牛顿法 - 1.4.2 拟牛顿法 - 2 优化策略 - 2.1 参数初始化 - 3 批标准化 - 4 坐标下降 - 5 Polyak 平均 - 6 监督预训练 - 7 设计有助于优化的模型 - 第九章卷积网络 - 1 卷积运算 - 2 池化 - 3 深度学习框架下的卷积 - 3.1 多个并行卷积 - 3.2 输入值与核 - 3.3 填充 (Padding) - 3.4 卷积步幅 (Stride) - 4 更多的卷积策略 - 4.1 深度可分离卷积 (Depthwise Separable Convolution) - 4.2 分组卷积 (Group Convolution) - 4.3 扩张卷积 (Dilated Convolution) - 5 GEMM 转换 - 6 卷积网络的训练 - 6.1 卷积网络示意图 - 6.2 单层卷积层/池化层 - 6.2.1 卷积函数的导数及反向传播 - 6.2.2 池化函数的导数及后向传播 - 6.3 多层卷积层/池化层 - 6.4 Flatten 层 & 全连接层 - 7 平移等变 - 8 代表性的卷积神经网络 - 8.1 卷积神经网络 (LeNet) - 第十一章实践方法论 - 1 实践方法论 - 2 性能度量指标 - 2.1 错误率与准确性 - 2.2 查准率、查全率与 F1 值 - 2.2.1 混淆矩阵 - 2.2.2 查准率和查全率的定义与关联 - 2.2.3 F1 值 - 2.3 PR 曲线 - 2.4 ROC 曲线与 AUC 值 - 2.4.1 ROC 曲线 - 2.4.2 AUC 值的计算方法 - 2.5 覆盖 - 2.6 指标性能的瓶颈 - 3 默认基准模型 - 4 确定是否收集更多数据 - 5 选择超参数 - 5.1 手动超参数调整 - 5.2 自动超参数优化算法 - 5.2.1 网格搜索 (Grid Search) - 5.2.2 随机搜索 (Random Search) - 5.2.3 基于模型的超参数优化 (Model-based Hyperparameter Optimization) ================================================ FILE: reference.txt ================================================ **参考文献**: - 全局参考 - https://github.com/exacity/deeplearningbook-chinese/ - 线性代数 - https://github.com/npetrakis/YearOfML - 概率与信息论 - https://github.com/akhilvasvani/Probability-and-Information-Theory - https://docs.scipy.org/doc/scipy/reference/stats.html - https://www.joinquant.com/view/community/detail/be3d8bc42275ea491897ac13fbf5838f - https://my.oschina.net/dillan/blog/134011 - 数值计算 - 机器学习基础 - 《统计学习方法》 - 《机器学习》 - https://github.com/paruby/ml-basics - https://github.com/akhilvasvani/machine-learning-basics - https://blog.csdn.net/ajianyingxiaoqinghan/article/details/72897399 - 深度前馈网络 - https://peterroelants.github.io/posts/cross-entropy-logistic/ - https://blog.csdn.net/weixin_36586536/article/details/80468426 - https://github.com/yg19930918/deep-learning-from-scratch-master - https://www.cnblogs.com/34fj/p/9036369.html - https://github.com/peterroelants/peterroelants.github.io - 深度学习中的正则化 - https://zhuanlan.zhihu.com/p/35893078 - https://www.jiqizhixin.com/articles/2017-06-23-5 - https://www.zybuluo.com/songying/note/1400484 - https://zhuanlan.zhihu.com/p/37120298 - https://kevinzakka.github.io/2016/09/14/batch_normalization/ - http://gitlinux.net/2018-10-29-xgboost/ - https://medium.com/swlh/boosting-and-bagging-explained-with-examples-5353a36eb78d - http://www.ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/slides/gradient_boosting.pdf - https://blog.csdn.net/liangjun_feng/article/details/79603705 - https://blog.csdn.net/sinat_22594309/article/details/60957594 - https://www.zybuluo.com/yxd/note/611571 - http://freemind.pluskid.org/machine-learning/sparsity-and-some-basics-of-l1-regularization/#ed61992b37932e208ae114be75e42a3e6dc34cb3 - 深度模型中的优化 - https://zhuanlan.zhihu.com/p/32626442 - https://github.com/exacity/deeplearningbook-chinese - http://cthorey.github.io./backpropagation/ - http://www.ludoart.cn/2019/02/22/Optimization-Methods/ - https://blog.csdn.net/itplus/article/details/21897715 - 卷积神经网络 - https://www.slideshare.net/kuwajima/cnnbp - https://github.com/exacity/simplified-deeplearning - https://github.com/yg19930918/deep-learning-from-scratch-master - https://adventuresinmachinelearning.com/keras-tutorial-cnn-11-lines/ - https://zhangting2020.github.io/2018/05/30/Transform-Invariance/ - https://zh.gluon.ai/chapter_convolutional-neural-networks/lenet.html - https://zhuanlan.zhihu.com/p/32702031 - https://blog.csdn.net/marsjhao/article/details/73088850 - 实践方法论 - https://github.com/masakazu-ishihata/BayesianOptimization - https://github.com/bjzhao143/MLwithPython - https://medium.com/inveterate-learner/deep-learning-book-chapter-11-c6ad1d3c3c08 - https://www.alexejgossmann.com/auc/ - https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-imbalanced-classification/ - https://www.yuque.com/books/share/f4031f65-70c1-4909-ba01-c47c31398466/kqbfug - http://bridg.land/posts/gaussian-processes-1 - https://zhuanlan.zhihu.com/p/76269142 ================================================ FILE: update.txt ================================================ **更新记录**: 2020/3/： 1. 修改第五章决策树部分，补充 ID3 和 CART 的原理，代码实现以 CART 为主。 2. 第七章添加 L1 和 L2 正则化最优解的推导 (即 L1稀疏解的原理)。 3. 第七章添加集成学习方法的推导与代码实现，包括 Bagging (随机森林)、Boosting (Adaboost、GBDT、XGBoost) 4. 第八章添加牛顿法与拟牛顿法 (DFP、BFGS、L-BFGS) 的推导。 5. 第十一章节添加高斯过程回归 (GPR) 与贝叶斯优化的推导与代码实现。