[
  {
    "path": ".github/ISSUE_TEMPLATE/issue---.md",
    "content": "---\nname: issue 模板\nabout: 请按照 issue 模板提问\ntitle: ''\nlabels: ''\nassignees: ''\n\n---\n\n**文档版本**\n说明你查看的是哪个 branch 的文档，即 K8S 版本，如 v1.8、v1.12。\n\n**现象描述**\n描述问题的现象。\n"
  },
  {
    "path": ".gitignore",
    "content": ".DS_Store\n"
  },
  {
    "path": "00.组件版本和配置策略.md",
    "content": "# 00. 组件版本和配置策略\n\n<!-- TOC -->\n\n- [00. 组件版本和配置策略](#00-组件版本和配置策略)\n    - [主要组件版本](#主要组件版本)\n    - [主要配置策略](#主要配置策略)\n\n<!-- /TOC -->\n\n## 主要组件版本\n\n| 组件 | 版本 | 发布时间 |\n| --- | --- | --- |\n| kubernetes | 1.16.6 | 2020-01-22 |\n| etcd | 3.4.3 | 2019-10-24 |\n| containerd | 1.3.3 | 2020-02-07 |\n| runc | 1.0.0-rc10 | 2019-12-23 |\n| calico | 3.12.0 | 2020-01-27 |\n| coredns | 1.6.6 | 2019-12-20 |\n| dashboard | v2.0.0-rc4 | 2020-02-06 |\n| k8s-prometheus-adapter | 0.5.0 | 2019-04-03|\n| prometheus-operator | 0.35.0 | 2020-01-13 |\n| prometheus | 2.15.2 | 2020-01-06 |\n| elasticsearch、kibana | 7.2.0 | 2019-06-25 |\n| cni-plugins | 0.8.5 | 2019-12-20 |\n| metrics-server | 0.3.6 | 2019-10-15 |\n\n\n## 主要配置策略\n\nkube-apiserver：\n\n+ 使用节点本地 nginx 4 层透明代理实现高可用；\n+ 关闭非安全端口 8080 和匿名访问；\n+ 在安全端口 6443 接收 https 请求；\n+ 严格的认证和授权策略 (x509、token、RBAC)；\n+ 开启 bootstrap token 认证，支持 kubelet TLS bootstrapping；\n+ 使用 https 访问 kubelet、etcd，加密通信；\n\nkube-controller-manager：\n\n+ 3 节点高可用；\n+ 关闭非安全端口，在安全端口 10252 接收 https 请求；\n+ 使用 kubeconfig 访问 apiserver 的安全端口；\n+ 自动 approve kubelet 证书签名请求 (CSR)，证书过期后自动轮转；\n+ 各 controller 使用自己的 ServiceAccount 访问 apiserver；\n\nkube-scheduler：\n\n+ 3 节点高可用；\n+ 使用 kubeconfig 访问 apiserver 的安全端口；\n\nkubelet：\n\n+ 使用 kubeadm 动态创建 bootstrap token，而不是在 apiserver 中静态配置；\n+ 使用 TLS bootstrap 机制自动生成 client 和 server 证书，过期后自动轮转；\n+ 在 KubeletConfiguration 类型的 JSON 文件配置主要参数；\n+ 关闭只读端口，在安全端口 10250 接收 https 请求，对请求进行认证和授权，拒绝匿名访问和非授权访问；\n+ 使用 kubeconfig 访问 apiserver 的安全端口；\n\nkube-proxy：\n\n+ 使用 kubeconfig 访问 apiserver 的安全端口；\n+ 在 KubeProxyConfiguration  类型的 JSON 文件配置主要参数；\n+ 使用 ipvs 代理模式；\n\n集群插件：\n\n+ DNS：使用功能、性能更好的 coredns；\n+ Dashboard：支持登录认证；\n+ Metric：metrics-server，使用 https 访问 kubelet 安全端口；\n+ Log：Elasticsearch、Fluend、Kibana；\n+ Registry 镜像库：docker-registry、harbor；"
  },
  {
    "path": "01.初始化系统和全局变量.md",
    "content": "tags: environment\n\n# 01. 初始化系统和全局变量\n\n<!-- TOC -->\n\n- [01. 初始化系统和全局变量](#01-初始化系统和全局变量)\n    - [集群规划](#集群规划)\n    - [设置主机名](#设置主机名)\n    - [添加节点信任关系](#添加节点信任关系)\n    - [更新 PATH 变量](#更新-path-变量)\n    - [安装依赖包](#安装依赖包)\n    - [关闭防火墙](#关闭防火墙)\n    - [关闭 swap 分区](#关闭-swap-分区)\n    - [关闭 SELinux](#关闭-selinux)\n    - [优化内核参数](#优化内核参数)\n    - [设置系统时区](#设置系统时区)\n    - [设置系统时钟同步](#设置系统时钟同步)\n    - [关闭无关的服务](#关闭无关的服务)\n    - [创建相关目录](#创建相关目录)\n    - [分发集群配置参数脚本](#分发集群配置参数脚本)\n    - [升级内核](#升级内核)\n    - [参考](#参考)\n\n<!-- /TOC -->\n\n## 集群规划\n\n+ zhangjun-k8s-01：172.27.138.251\n+ zhangjun-k8s-02：172.27.137.229\n+ zhangjun-k8s-03：172.27.138.239\n\n三台机器混合部署本文档的 etcd、master 集群和 woker 集群。\n\n如果没有特殊说明，需要在**所有节点**上执行本文档的初始化操作。\n\n## 设置主机名\n\n``` bash\nhostnamectl set-hostname zhangjun-k8s-01 # 将 zhangjun-k8s-01 替换为当前主机名\n```\n\n如果 DNS 不支持主机名称解析，还需要在每台机器的 `/etc/hosts` 文件中添加主机名和 IP 的对应关系：\n\n``` bash\ncat >> /etc/hosts <<EOF\n172.27.138.251 zhangjun-k8s-01\n172.27.137.229 zhangjun-k8s-02\n172.27.138.239 zhangjun-k8s-03\nEOF\n```\n\n退出，重新登录 root 账号，可以看到主机名生效。\n\n## 添加节点信任关系\n\n本操作只需要在 zhangjun-k8s-01 节点上进行，设置 root 账户可以无密码登录**所有节点**：\n\n``` bash\nssh-keygen -t rsa \nssh-copy-id root@zhangjun-k8s-01\nssh-copy-id root@zhangjun-k8s-02\nssh-copy-id root@zhangjun-k8s-03\n```\n\n## 更新 PATH 变量\n\n``` bash\necho 'PATH=/opt/k8s/bin:$PATH' >>/root/.bashrc\nsource /root/.bashrc\n```\n+ `/opt/k8s/bin` 目录保存本文档下载安装的程序；\n\n## 安装依赖包\n\n``` bash\nyum install -y epel-release\nyum install -y chrony conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget socat git\n```\n+ 本文档的 kube-proxy 使用 ipvs 模式，ipvsadm 为 ipvs 的管理工具；\n+ etcd 集群各机器需要时间同步，chrony 用于系统时间同步；\n\n## 关闭防火墙\n\n关闭防火墙，清理防火墙规则，设置默认转发策略：\n\n``` bash\nsystemctl stop firewalld\nsystemctl disable firewalld\niptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat\niptables -P FORWARD ACCEPT\n```\n\n## 关闭 swap 分区\n\n关闭 swap 分区，否则kubelet 会启动失败(可以设置 kubelet 启动参数 --fail-swap-on 为 false 关闭 swap 检查)：\n\n``` bash\nswapoff -a\nsed -i '/ swap / s/^\\(.*\\)$/#\\1/g' /etc/fstab \n```\n\n## 关闭 SELinux\n\n关闭 SELinux，否则 kubelet 挂载目录时可能报错 `Permission denied`：\n\n``` bash\nsetenforce 0\nsed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config\n```\n\n## 优化内核参数\n\n``` bash\ncat > kubernetes.conf <<EOF\nnet.bridge.bridge-nf-call-iptables=1\nnet.bridge.bridge-nf-call-ip6tables=1\nnet.ipv4.ip_forward=1\nnet.ipv4.tcp_tw_recycle=0\nnet.ipv4.neigh.default.gc_thresh1=1024\nnet.ipv4.neigh.default.gc_thresh2=2048\nnet.ipv4.neigh.default.gc_thresh3=4096\nvm.swappiness=0\nvm.overcommit_memory=1\nvm.panic_on_oom=0\nfs.inotify.max_user_instances=8192\nfs.inotify.max_user_watches=1048576\nfs.file-max=52706963\nfs.nr_open=52706963\nnet.ipv6.conf.all.disable_ipv6=1\nnet.netfilter.nf_conntrack_max=2310720\nEOF\ncp kubernetes.conf  /etc/sysctl.d/kubernetes.conf\nsysctl -p /etc/sysctl.d/kubernetes.conf\n```\n+ 关闭 tcp_tw_recycle，否则与 NAT 冲突，可能导致服务不通；\n\n## 设置系统时区\n\n``` bash\ntimedatectl set-timezone Asia/Shanghai\n```\n\n## 设置系统时钟同步\n\n``` bash\nsystemctl enable chronyd\nsystemctl start chronyd\n```\n\n查看同步状态：\n``` bash\ntimedatectl status\n```\n\n输出：\n``` text\nSystem clock synchronized: yes\n              NTP service: active\n          RTC in local TZ: no\n```\n+ `System clock synchronized: yes`，表示时钟已同步；\n+ `NTP service: active`，表示开启了时钟同步服务；\n\n\n``` bash\n# 将当前的 UTC 时间写入硬件时钟\ntimedatectl set-local-rtc 0\n\n# 重启依赖于系统时间的服务\nsystemctl restart rsyslog \nsystemctl restart crond\n```\n\n## 关闭无关的服务\n\n``` bash\nsystemctl stop postfix && systemctl disable postfix\n```\n\n## 创建相关目录\n\n创建目录：\n\n``` bash\nmkdir -p /opt/k8s/{bin,work} /etc/{kubernetes,etcd}/cert\n```\n\n## 分发集群配置参数脚本\n\n后续使用的环境变量都定义在文件 [environment.sh](manifests/environment.sh) 中，请根据**自己的机器、网络情况**修改。然后拷贝到**所有**节点：\n\n``` bash\nsource environment.sh # 先修改\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp environment.sh root@${node_ip}:/opt/k8s/bin/\n    ssh root@${node_ip} \"chmod +x /opt/k8s/bin/*\"\n  done\n```\n\n## 升级内核\n\nCentOS 7.x 系统自带的 3.10.x 内核存在一些 Bugs，导致运行的 Docker、Kubernetes 不稳定，例如：\n1. 高版本的 docker(1.13 以后) 启用了 3.10 kernel 实验支持的 kernel memory account 功能(无法关闭)，当节点压力大如频繁启动和停止容器时会导致 cgroup memory leak；\n2. 网络设备引用计数泄漏，会导致类似于报错：\"kernel:unregister_netdevice: waiting for eth0 to become free. Usage count = 1\";\n\n解决方案如下：\n1. 升级内核到 4.4.X 以上；\n2. 或者，手动编译内核，disable CONFIG_MEMCG_KMEM 特性；\n3. 或者，安装修复了该问题的 Docker 18.09.1 及以上的版本。但由于 kubelet 也会设置 kmem（它 vendor 了 runc），所以需要重新编译 kubelet 并指定 GOFLAGS=\"-tags=nokmem\"；\n  ``` bash\n  git clone --branch v1.14.1 --single-branch --depth 1 https://github.com/kubernetes/kubernetes\n  cd kubernetes\n  KUBE_GIT_VERSION=v1.14.1 ./build/run.sh make kubelet GOFLAGS=\"-tags=nokmem\"\n  ```\n\n这里采用升级内核的解决办法：\n\n``` bash\nrpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm\n# 安装完成后检查 /boot/grub2/grub.cfg 中对应内核 menuentry 中是否包含 initrd16 配置，如果没有，再安装一次！\nyum --enablerepo=elrepo-kernel install -y kernel-lt\n# 设置开机从新内核启动\ngrub2-set-default 0\n```\n\n重启机器：\n\n``` bash\nsync\nreboot\n```\n\n## 参考\n1. 系统内核相关参数参考：https://docs.openshift.com/enterprise/3.2/admin_guide/overcommit.html\n2. 3.10.x 内核 kmem bugs 相关的讨论和解决办法：\n    1. https://github.com/kubernetes/kubernetes/issues/61937\n    2. https://support.mesosphere.com/s/article/Critical-Issue-KMEM-MSPH-2018-0006\n    3. https://pingcap.com/blog/try-to-fix-two-linux-kernel-bugs-while-testing-tidb-operator-in-k8s/\n"
  },
  {
    "path": "02.创建CA根证书和秘钥.md",
    "content": "tags: TLS, CA, x509\n\n# 02. 创建 CA 根证书和秘钥\n\n<!-- TOC -->\n\n- [02. 创建 CA 根证书和秘钥](#02-创建-ca-根证书和秘钥)\n    - [安装 cfssl 工具集](#安装-cfssl-工具集)\n    - [创建配置文件](#创建配置文件)\n    - [创建证书签名请求文件](#创建证书签名请求文件)\n    - [生成 CA 证书和私钥](#生成-ca-证书和私钥)\n    - [分发证书文件](#分发证书文件)\n    - [参考](#参考)\n\n<!-- /TOC -->\n\n为确保安全，`kubernetes` 系统各组件需要使用 `x509` 证书对通信进行加密和认证。\n\nCA (Certificate Authority) 是自签名的根证书，用来签名后续创建的其它证书。\n\nCA 证书是集群所有节点共享的，**只需要创建一次**，后续用它签名其它所有证书。\n\n本文档使用 `CloudFlare` 的 PKI 工具集 [cfssl](https://github.com/cloudflare/cfssl) 创建所有证书。\n\n注意：如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**。\n\n## 安装 cfssl 工具集\n\n``` bash\n\nsudo mkdir -p /opt/k8s/cert && cd /opt/k8s/work\n\nwget https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl_1.4.1_linux_amd64\nmv cfssl_1.4.1_linux_amd64 /opt/k8s/bin/cfssl\n\nwget https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssljson_1.4.1_linux_amd64\nmv cfssljson_1.4.1_linux_amd64 /opt/k8s/bin/cfssljson\n\nwget https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl-certinfo_1.4.1_linux_amd64\nmv cfssl-certinfo_1.4.1_linux_amd64 /opt/k8s/bin/cfssl-certinfo\n\nchmod +x /opt/k8s/bin/*\nexport PATH=/opt/k8s/bin:$PATH\n```\n\n## 创建配置文件\n\nCA 配置文件用于配置根证书的使用场景 (profile) 和具体参数 (usage，过期时间、服务端认证、客户端认证、加密等)：\n\n``` bash\ncd /opt/k8s/work\ncat > ca-config.json <<EOF\n{\n  \"signing\": {\n    \"default\": {\n      \"expiry\": \"87600h\"\n    },\n    \"profiles\": {\n      \"kubernetes\": {\n        \"usages\": [\n            \"signing\",\n            \"key encipherment\",\n            \"server auth\",\n            \"client auth\"\n        ],\n        \"expiry\": \"876000h\"\n      }\n    }\n  }\n}\nEOF\n```\n+ `signing`：表示该证书可用于签名其它证书（生成的 `ca.pem` 证书中 `CA=TRUE`）；\n+ `server auth`：表示 client 可以用该该证书对 server 提供的证书进行验证；\n+ `client auth`：表示 server 可以用该该证书对 client 提供的证书进行验证；\n+ `\"expiry\": \"876000h\"`：证书有效期设置为 100 年；\n\n## 创建证书签名请求文件\n\n``` bash\ncd /opt/k8s/work\ncat > ca-csr.json <<EOF\n{\n  \"CN\": \"kubernetes-ca\",\n  \"key\": {\n    \"algo\": \"rsa\",\n    \"size\": 2048\n  },\n  \"names\": [\n    {\n      \"C\": \"CN\",\n      \"ST\": \"BeiJing\",\n      \"L\": \"BeiJing\",\n      \"O\": \"k8s\",\n      \"OU\": \"opsnull\"\n    }\n  ],\n  \"ca\": {\n    \"expiry\": \"876000h\"\n }\n}\nEOF\n```\n+ `CN：Common Name`：kube-apiserver 从证书中提取该字段作为请求的**用户名 (User Name)**，浏览器使用该字段验证网站是否合法；\n+ `O：Organization`：kube-apiserver 从证书中提取该字段作为请求用户所属的**组 (Group)**；\n+ kube-apiserver 将提取的 `User、Group` 作为 `RBAC` 授权的用户标识；\n\n注意：\n1. 不同证书 csr 文件的 CN、C、ST、L、O、OU 组合必须不同，否则可能出现 `PEER'S CERTIFICATE HAS AN INVALID SIGNATURE` 错误；\n2. 后续创建证书的 csr 文件时，CN 都不相同（C、ST、L、O、OU 相同），以达到区分的目的；\n\n## 生成 CA 证书和私钥\n\n``` bash\ncd /opt/k8s/work\ncfssl gencert -initca ca-csr.json | cfssljson -bare ca\nls ca*\n```\n\n## 分发证书文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p /etc/kubernetes/cert\"\n    scp ca*.pem ca-config.json root@${node_ip}:/etc/kubernetes/cert\n  done\n```\n\n## 参考\n\n1. [各种 CA 证书类型](https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md)"
  },
  {
    "path": "03.kubectl.md",
    "content": "tags: kubectl\n\n# 03. 安装和配置 kubectl\n\n<!-- TOC -->\n\n- [03. 安装和配置 kubectl](#03-安装和配置-kubectl)\n    - [下载和分发 kubectl 二进制文件](#下载和分发-kubectl-二进制文件)\n    - [创建 admin 证书和私钥](#创建-admin-证书和私钥)\n    - [创建 kubeconfig 文件](#创建-kubeconfig-文件)\n    - [分发 kubeconfig 文件](#分发-kubeconfig-文件)\n\n<!-- /TOC -->\n\n本文档介绍安装和配置 kubernetes 命令行管理工具 kubectl 的步骤。\n\n注意：\n1. 如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**；\n2. 本文档只需要**部署一次**，生成的 kubeconfig 文件是**通用的**，可以拷贝到需要执行 kubectl 命令的机器的 `~/.kube/config` 位置；\n\n## 下载和分发 kubectl 二进制文件\n\n``` bash\ncd /opt/k8s/work\nwget https://dl.k8s.io/v1.16.6/kubernetes-client-linux-amd64.tar.gz # 自行解决翻墙下载问题\ntar -xzvf kubernetes-client-linux-amd64.tar.gz\n```\n\n分发到所有使用 kubectl 工具的节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp kubernetes/client/bin/kubectl root@${node_ip}:/opt/k8s/bin/\n    ssh root@${node_ip} \"chmod +x /opt/k8s/bin/*\"\n  done\n```\n\n## 创建 admin 证书和私钥\n\nkubectl 使用 https 协议与 kube-apiserver 进行安全通信，kube-apiserver 对 kubectl 请求包含的证书进行认证和授权。\n\nkubectl 后续用于集群管理，所以这里创建具有**最高权限**的 admin 证书。\n\n创建证书签名请求：\n\n``` bash\ncd /opt/k8s/work\ncat > admin-csr.json <<EOF\n{\n  \"CN\": \"admin\",\n  \"hosts\": [],\n  \"key\": {\n    \"algo\": \"rsa\",\n    \"size\": 2048\n  },\n  \"names\": [\n    {\n      \"C\": \"CN\",\n      \"ST\": \"BeiJing\",\n      \"L\": \"BeiJing\",\n      \"O\": \"system:masters\",\n      \"OU\": \"opsnull\"\n    }\n  ]\n}\nEOF\n```\n+ `O: system:masters`：kube-apiserver 收到使用该证书的客户端请求后，为请求添加组（Group）认证标识 `system:masters`；\n+ 预定义的 ClusterRoleBinding `cluster-admin` 将 Group `system:masters` 与 Role `cluster-admin` 绑定，该 Role 授予操作集群所需的**最高**权限；\n+ 该证书只会被 kubectl 当做 client 证书使用，所以 `hosts` 字段为空；\n\n生成证书和私钥：\n\n``` bash\ncd /opt/k8s/work\ncfssl gencert -ca=/opt/k8s/work/ca.pem \\\n  -ca-key=/opt/k8s/work/ca-key.pem \\\n  -config=/opt/k8s/work/ca-config.json \\\n  -profile=kubernetes admin-csr.json | cfssljson -bare admin\nls admin*\n```\n+ 忽略警告消息 `[WARNING] This certificate lacks a \"hosts\" field.`；\n\n## 创建 kubeconfig 文件\n\nkubectl 使用 kubeconfig 文件访问 apiserver，该文件包含 kube-apiserver 的地址和认证信息（CA 证书和客户端证书）：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\n\n# 设置集群参数\nkubectl config set-cluster kubernetes \\\n  --certificate-authority=/opt/k8s/work/ca.pem \\\n  --embed-certs=true \\\n  --server=https://${NODE_IPS[0]}:6443 \\\n  --kubeconfig=kubectl.kubeconfig\n\n# 设置客户端认证参数\nkubectl config set-credentials admin \\\n  --client-certificate=/opt/k8s/work/admin.pem \\\n  --client-key=/opt/k8s/work/admin-key.pem \\\n  --embed-certs=true \\\n  --kubeconfig=kubectl.kubeconfig\n\n# 设置上下文参数\nkubectl config set-context kubernetes \\\n  --cluster=kubernetes \\\n  --user=admin \\\n  --kubeconfig=kubectl.kubeconfig\n\n# 设置默认上下文\nkubectl config use-context kubernetes --kubeconfig=kubectl.kubeconfig\n```\n+ `--certificate-authority`：验证 kube-apiserver 证书的根证书；\n+ `--client-certificate`、`--client-key`：刚生成的 `admin` 证书和私钥，与 kube-apiserver https 通信时使用；\n+ `--embed-certs=true`：将 ca.pem 和 admin.pem 证书内容嵌入到生成的 kubectl.kubeconfig 文件中(否则，写入的是证书文件路径，后续拷贝 kubeconfig 到其它机器时，还需要单独拷贝证书文件，不方便。)；\n+ `--server`：指定 kube-apiserver 的地址，这里指向第一个节点上的服务；\n\n## 分发 kubeconfig 文件\n\n分发到所有使用 `kubectl` 命令的节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p ~/.kube\"\n    scp kubectl.kubeconfig root@${node_ip}:~/.kube/config\n  done\n```"
  },
  {
    "path": "04.etcd集群.md",
    "content": "tags: etcd\n\n# 04. 部署 etcd 集群\n\n<!-- TOC -->\n\n- [04. 部署 etcd 集群](#04-部署-etcd-集群)\n    - [下载和分发 etcd 二进制文件](#下载和分发-etcd-二进制文件)\n    - [创建 etcd 证书和私钥](#创建-etcd-证书和私钥)\n    - [创建 etcd 的 systemd unit 模板文件](#创建-etcd-的-systemd-unit-模板文件)\n    - [为各节点创建和分发 etcd systemd unit 文件](#为各节点创建和分发-etcd-systemd-unit-文件)\n    - [启动 etcd 服务](#启动-etcd-服务)\n    - [检查启动结果](#检查启动结果)\n    - [验证服务状态](#验证服务状态)\n    - [查看当前的 leader](#查看当前的-leader)\n\n<!-- /TOC -->\n\netcd 是基于 Raft 的分布式 KV 存储系统，由 CoreOS 开发，常用于服务发现、共享配置以及并发控制（如 leader 选举、分布式锁等）。\n\nkubernetes 使用 etcd 集群持久化存储所有 API 对象、运行数据。\n\n本文档介绍部署一个三节点高可用 etcd 集群的步骤：\n\n+ 下载和分发 etcd 二进制文件；\n+ 创建 etcd 集群各节点的 x509 证书，用于加密客户端(如 etcdctl) 与 etcd 集群、etcd 集群之间的通信；\n+ 创建 etcd 的 systemd unit 文件，配置服务参数；\n+ 检查集群工作状态；\n\netcd 集群节点名称和 IP 如下：\n\n+ zhangjun-k8s-01：172.27.138.251\n+ zhangjun-k8s-02：172.27.137.229\n+ zhangjun-k8s-03：172.27.138.239\n\n注意：\n1. 如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**；\n2. flanneld 与本文档安装的 etcd v3.4.x 不兼容，如果要安装 flanneld（本文档使用 calio），则需要将 etcd **降级到 v3.3.x 版本**；\n\n## 下载和分发 etcd 二进制文件\n\n到 etcd 的 [release 页面](https://github.com/coreos/etcd/releases) 下载最新版本的发布包：\n\n``` bash\ncd /opt/k8s/work\nwget https://github.com/coreos/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz\ntar -xvf etcd-v3.4.3-linux-amd64.tar.gz\n```\n\n分发二进制文件到集群所有节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp etcd-v3.4.3-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin\n    ssh root@${node_ip} \"chmod +x /opt/k8s/bin/*\"\n  done\n```\n\n## 创建 etcd 证书和私钥\n\n创建证书签名请求：\n\n``` bash\ncd /opt/k8s/work\ncat > etcd-csr.json <<EOF\n{\n  \"CN\": \"etcd\",\n  \"hosts\": [\n    \"127.0.0.1\",\n    \"172.27.138.251\",\n    \"172.27.137.229\",\n    \"172.27.138.239\"\n  ],\n  \"key\": {\n    \"algo\": \"rsa\",\n    \"size\": 2048\n  },\n  \"names\": [\n    {\n      \"C\": \"CN\",\n      \"ST\": \"BeiJing\",\n      \"L\": \"BeiJing\",\n      \"O\": \"k8s\",\n      \"OU\": \"opsnull\"\n    }\n  ]\n}\nEOF\n``` \n+ `hosts`：指定授权使用该证书的 etcd 节点 IP 列表，**需要将 etcd 集群所有节点 IP 都列在其中**；\n\n生成证书和私钥：\n\n``` bash\ncd /opt/k8s/work\ncfssl gencert -ca=/opt/k8s/work/ca.pem \\\n    -ca-key=/opt/k8s/work/ca-key.pem \\\n    -config=/opt/k8s/work/ca-config.json \\\n    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd\nls etcd*pem\n```\n\n分发生成的证书和私钥到各 etcd 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p /etc/etcd/cert\"\n    scp etcd*.pem root@${node_ip}:/etc/etcd/cert/\n  done\n```\n\n## 创建 etcd 的 systemd unit 模板文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > etcd.service.template <<EOF\n[Unit]\nDescription=Etcd Server\nAfter=network.target\nAfter=network-online.target\nWants=network-online.target\nDocumentation=https://github.com/coreos\n\n[Service]\nType=notify\nWorkingDirectory=${ETCD_DATA_DIR}\nExecStart=/opt/k8s/bin/etcd \\\\\n  --data-dir=${ETCD_DATA_DIR} \\\\\n  --wal-dir=${ETCD_WAL_DIR} \\\\\n  --name=##NODE_NAME## \\\\\n  --cert-file=/etc/etcd/cert/etcd.pem \\\\\n  --key-file=/etc/etcd/cert/etcd-key.pem \\\\\n  --trusted-ca-file=/etc/kubernetes/cert/ca.pem \\\\\n  --peer-cert-file=/etc/etcd/cert/etcd.pem \\\\\n  --peer-key-file=/etc/etcd/cert/etcd-key.pem \\\\\n  --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\\\\n  --peer-client-cert-auth \\\\\n  --client-cert-auth \\\\\n  --listen-peer-urls=https://##NODE_IP##:2380 \\\\\n  --initial-advertise-peer-urls=https://##NODE_IP##:2380 \\\\\n  --listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \\\\\n  --advertise-client-urls=https://##NODE_IP##:2379 \\\\\n  --initial-cluster-token=etcd-cluster-0 \\\\\n  --initial-cluster=${ETCD_NODES} \\\\\n  --initial-cluster-state=new \\\\\n  --auto-compaction-mode=periodic \\\\\n  --auto-compaction-retention=1 \\\\\n  --max-request-bytes=33554432 \\\\\n  --quota-backend-bytes=6442450944 \\\\\n  --heartbeat-interval=250 \\\\\n  --election-timeout=2000\nRestart=on-failure\nRestartSec=5\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n+ `WorkingDirectory`、`--data-dir`：指定工作目录和数据目录为 `${ETCD_DATA_DIR}`，需在启动服务前创建这个目录；\n+ `--wal-dir`：指定 wal 目录，为了提高性能，一般使用 SSD 或者和 `--data-dir` 不同的磁盘；\n+ `--name`：指定节点名称，当 `--initial-cluster-state` 值为 `new` 时，`--name` 的参数值必须位于 `--initial-cluster` 列表中；\n+ `--cert-file`、`--key-file`：etcd server 与 client 通信时使用的证书和私钥；\n+ `--trusted-ca-file`：签名 client 证书的 CA 证书，用于验证 client 证书；\n+ `--peer-cert-file`、`--peer-key-file`：etcd 与 peer 通信使用的证书和私钥；\n+ `--peer-trusted-ca-file`：签名 peer 证书的 CA 证书，用于验证 peer 证书；\n\n## 为各节点创建和分发 etcd systemd unit 文件\n\n替换模板文件中的变量，为各节点创建 systemd unit 文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor (( i=0; i < 3; i++ ))\n  do\n    sed -e \"s/##NODE_NAME##/${NODE_NAMES[i]}/\" -e \"s/##NODE_IP##/${NODE_IPS[i]}/\" etcd.service.template > etcd-${NODE_IPS[i]}.service \n  done\nls *.service\n```\n+ NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP；\n\n分发生成的 systemd unit 文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.service\n  done\n```\n\n## 启动 etcd 服务\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}\"\n    ssh root@${node_ip} \"systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd \" &\n  done\n```\n+ 必须先创建 etcd 数据目录和工作目录;\n+ etcd 进程首次启动时会等待其它节点的 etcd 加入集群，命令 `systemctl start etcd` 会卡住一段时间，为正常现象；\n\n## 检查启动结果\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl status etcd|grep Active\"\n  done\n```\n\n确保状态为 `active (running)`，否则查看日志，确认原因：\n\n``` bash\njournalctl -u etcd\n```\n\n## 验证服务状态\n\n部署完 etcd 集群后，在任一 etcd 节点上执行如下命令：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    /opt/k8s/bin/etcdctl \\\n    --endpoints=https://${node_ip}:2379 \\\n    --cacert=/etc/kubernetes/cert/ca.pem \\\n    --cert=/etc/etcd/cert/etcd.pem \\\n    --key=/etc/etcd/cert/etcd-key.pem endpoint health\n  done\n```\n+ 3.4.3 版本的 etcd/etcdctl 默认启用了 V3 API，所以执行 etcdctl 命令时不需要再指定环境变量 `ETCDCTL_API=3`； \n+ 从 K8S 1.13 开始，不再支持 v2 版本的 etcd；\n\n预期输出：\n\n``` bash\n>>> 172.27.138.251\nhttps://172.27.138.251:2379 is healthy: successfully committed proposal: took = 2.756451ms\n>>> 172.27.137.229\nhttps://172.27.137.229:2379 is healthy: successfully committed proposal: took = 2.025018ms\n>>> 172.27.138.239\nhttps://172.27.138.239:2379 is healthy: successfully committed proposal: took = 2.335097ms\n```\n\n输出均为 `healthy` 时表示集群服务正常。\n\n## 查看当前的 leader\n\n``` bash\nsource /opt/k8s/bin/environment.sh\n/opt/k8s/bin/etcdctl \\\n  -w table --cacert=/etc/kubernetes/cert/ca.pem \\\n  --cert=/etc/etcd/cert/etcd.pem \\\n  --key=/etc/etcd/cert/etcd-key.pem \\\n  --endpoints=${ETCD_ENDPOINTS} endpoint status \n```\n\n输出：\n\n``` bash\n+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+\n|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |\n+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+\n| https://172.27.138.251:2379 | 4250b255e93e0076 |   3.4.3 |   20 kB |     false |      false |         2 |          8 |                  8 |        |\n| https://172.27.137.229:2379 | b3d912e6166f1213 |   3.4.3 |   20 kB |      true |      false |         2 |          8 |                  8 |        |\n| https://172.27.138.239:2379 | 8a4d4a2904de8446 |   3.4.3 |   20 kB |     false |      false |         2 |          8 |                  8 |        |\n+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+\n```\n+ 可见，当前的 leader 为 172.27.138.229。"
  },
  {
    "path": "05-1.master节点.md",
    "content": "tags: master, kube-apiserver, kube-scheduler, kube-controller-manager\n\n# 05-1. 部署 master 节点\n\n<!-- TOC -->\n\n- [05-1. 部署 master 节点](#05-1-部署-master-节点)\n    - [下载最新版本二进制文件](#下载最新版本二进制文件)\n\n<!-- /TOC -->\n\nkubernetes master 节点运行如下组件：\n+ kube-apiserver\n+ kube-scheduler\n+ kube-controller-manager\n\nkube-apiserver、kube-scheduler 和 kube-controller-manager 均以多实例模式运行：\n1. kube-scheduler 和 kube-controller-manager 会自动选举产生一个 leader 实例，其它实例处于阻塞模式，当 leader 挂了后，重新选举产生新的 leader，从而保证服务可用性；\n2. kube-apiserver 是无状态的，可以通过 kube-nginx 进行代理访问（见[06-2.apiserver高可用](06-2.apiserver高可用.md)），从而保证服务可用性；\n\n注意：如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s01 节点上执行**。\n\n## 下载最新版本二进制文件\n\n从 [CHANGELOG 页面](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md) 下载二进制 tar 文件并解压：\n\n``` bash\ncd /opt/k8s/work\nwget https://dl.k8s.io/v1.16.6/kubernetes-server-linux-amd64.tar.gz  # 自行解决翻墙问题\ntar -xzvf kubernetes-server-linux-amd64.tar.gz\ncd kubernetes\ntar -xzvf  kubernetes-src.tar.gz\n```\n\n将二进制文件拷贝到所有 master 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp kubernetes/server/bin/{apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@${node_ip}:/opt/k8s/bin/\n    ssh root@${node_ip} \"chmod +x /opt/k8s/bin/*\"\n  done\n```"
  },
  {
    "path": "05-2.apiserver集群.md",
    "content": "tags: master, kube-apiserver\n\n# 05-2. 部署 kube-apiserver 集群\n\n<!-- TOC -->\n\n- [05-2. 部署 kube-apiserver 集群](#05-2-部署-kube-apiserver-集群)\n    - [创建 kubernetes-master 证书和私钥](#创建-kubernetes-master-证书和私钥)\n    - [创建加密配置文件](#创建加密配置文件)\n    - [创建审计策略文件](#创建审计策略文件)\n    - [创建后续访问 metrics-server 或 kube-prometheus 使用的证书](#创建后续访问-metrics-server-或-kube-prometheus-使用的证书)\n    - [创建 kube-apiserver systemd unit 模板文件](#创建-kube-apiserver-systemd-unit-模板文件)\n    - [为各节点创建和分发 kube-apiserver systemd unit 文件](#为各节点创建和分发-kube-apiserver-systemd-unit-文件)\n    - [启动 kube-apiserver 服务](#启动-kube-apiserver-服务)\n    - [检查 kube-apiserver 运行状态](#检查-kube-apiserver-运行状态)\n    - [检查集群信息](#检查集群信息)\n    - [检查 kube-apiserver 监听的端口](#检查-kube-apiserver-监听的端口)\n\n<!-- /TOC -->\n\n本文档讲解部署一个三实例 kube-apiserver 集群的步骤.\n\n注意：如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**。\n\n## 创建 kubernetes-master 证书和私钥\n\n创建证书签名请求：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > kubernetes-csr.json <<EOF\n{\n  \"CN\": \"kubernetes-master\",\n  \"hosts\": [\n    \"127.0.0.1\",\n    \"172.27.138.251\",\n    \"172.27.137.229\",\n    \"172.27.138.239\",\n    \"${CLUSTER_KUBERNETES_SVC_IP}\",\n    \"kubernetes\",\n    \"kubernetes.default\",\n    \"kubernetes.default.svc\",\n    \"kubernetes.default.svc.cluster\",\n    \"kubernetes.default.svc.cluster.local.\",\n    \"kubernetes.default.svc.${CLUSTER_DNS_DOMAIN}.\"\n  ],\n  \"key\": {\n    \"algo\": \"rsa\",\n    \"size\": 2048\n  },\n  \"names\": [\n    {\n      \"C\": \"CN\",\n      \"ST\": \"BeiJing\",\n      \"L\": \"BeiJing\",\n      \"O\": \"k8s\",\n      \"OU\": \"opsnull\"\n    }\n  ]\n}\nEOF\n```\n+ hosts 字段指定授权使用该证书的 **IP 和域名列表**，这里列出了 master 节点 IP、kubernetes 服务的 IP 和域名；\n\n生成证书和私钥：\n\n``` bash\ncfssl gencert -ca=/opt/k8s/work/ca.pem \\\n  -ca-key=/opt/k8s/work/ca-key.pem \\\n  -config=/opt/k8s/work/ca-config.json \\\n  -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes\nls kubernetes*pem\n```\n\n将生成的证书和私钥文件拷贝到所有 master 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p /etc/kubernetes/cert\"\n    scp kubernetes*.pem root@${node_ip}:/etc/kubernetes/cert/\n  done\n```\n\n## 创建加密配置文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > encryption-config.yaml <<EOF\nkind: EncryptionConfig\napiVersion: v1\nresources:\n  - resources:\n      - secrets\n    providers:\n      - aescbc:\n          keys:\n            - name: key1\n              secret: ${ENCRYPTION_KEY}\n      - identity: {}\nEOF\n```\n\n将加密配置文件拷贝到 master 节点的 `/etc/kubernetes` 目录下：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp encryption-config.yaml root@${node_ip}:/etc/kubernetes/\n  done\n```\n\n## 创建审计策略文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > audit-policy.yaml <<EOF\napiVersion: audit.k8s.io/v1beta1\nkind: Policy\nrules:\n  # The following requests were manually identified as high-volume and low-risk, so drop them.\n  - level: None\n    resources:\n      - group: \"\"\n        resources:\n          - endpoints\n          - services\n          - services/status\n    users:\n      - 'system:kube-proxy'\n    verbs:\n      - watch\n\n  - level: None\n    resources:\n      - group: \"\"\n        resources:\n          - nodes\n          - nodes/status\n    userGroups:\n      - 'system:nodes'\n    verbs:\n      - get\n\n  - level: None\n    namespaces:\n      - kube-system\n    resources:\n      - group: \"\"\n        resources:\n          - endpoints\n    users:\n      - 'system:kube-controller-manager'\n      - 'system:kube-scheduler'\n      - 'system:serviceaccount:kube-system:endpoint-controller'\n    verbs:\n      - get\n      - update\n\n  - level: None\n    resources:\n      - group: \"\"\n        resources:\n          - namespaces\n          - namespaces/status\n          - namespaces/finalize\n    users:\n      - 'system:apiserver'\n    verbs:\n      - get\n\n  # Don't log HPA fetching metrics.\n  - level: None\n    resources:\n      - group: metrics.k8s.io\n    users:\n      - 'system:kube-controller-manager'\n    verbs:\n      - get\n      - list\n\n  # Don't log these read-only URLs.\n  - level: None\n    nonResourceURLs:\n      - '/healthz*'\n      - /version\n      - '/swagger*'\n\n  # Don't log events requests.\n  - level: None\n    resources:\n      - group: \"\"\n        resources:\n          - events\n\n  # node and pod status calls from nodes are high-volume and can be large, don't log responses\n  # for expected updates from nodes\n  - level: Request\n    omitStages:\n      - RequestReceived\n    resources:\n      - group: \"\"\n        resources:\n          - nodes/status\n          - pods/status\n    users:\n      - kubelet\n      - 'system:node-problem-detector'\n      - 'system:serviceaccount:kube-system:node-problem-detector'\n    verbs:\n      - update\n      - patch\n\n  - level: Request\n    omitStages:\n      - RequestReceived\n    resources:\n      - group: \"\"\n        resources:\n          - nodes/status\n          - pods/status\n    userGroups:\n      - 'system:nodes'\n    verbs:\n      - update\n      - patch\n\n  # deletecollection calls can be large, don't log responses for expected namespace deletions\n  - level: Request\n    omitStages:\n      - RequestReceived\n    users:\n      - 'system:serviceaccount:kube-system:namespace-controller'\n    verbs:\n      - deletecollection\n\n  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,\n  # so only log at the Metadata level.\n  - level: Metadata\n    omitStages:\n      - RequestReceived\n    resources:\n      - group: \"\"\n        resources:\n          - secrets\n          - configmaps\n      - group: authentication.k8s.io\n        resources:\n          - tokenreviews\n  # Get repsonses can be large; skip them.\n  - level: Request\n    omitStages:\n      - RequestReceived\n    resources:\n      - group: \"\"\n      - group: admissionregistration.k8s.io\n      - group: apiextensions.k8s.io\n      - group: apiregistration.k8s.io\n      - group: apps\n      - group: authentication.k8s.io\n      - group: authorization.k8s.io\n      - group: autoscaling\n      - group: batch\n      - group: certificates.k8s.io\n      - group: extensions\n      - group: metrics.k8s.io\n      - group: networking.k8s.io\n      - group: policy\n      - group: rbac.authorization.k8s.io\n      - group: scheduling.k8s.io\n      - group: settings.k8s.io\n      - group: storage.k8s.io\n    verbs:\n      - get\n      - list\n      - watch\n\n  # Default level for known APIs\n  - level: RequestResponse\n    omitStages:\n      - RequestReceived\n    resources:\n      - group: \"\"\n      - group: admissionregistration.k8s.io\n      - group: apiextensions.k8s.io\n      - group: apiregistration.k8s.io\n      - group: apps\n      - group: authentication.k8s.io\n      - group: authorization.k8s.io\n      - group: autoscaling\n      - group: batch\n      - group: certificates.k8s.io\n      - group: extensions\n      - group: metrics.k8s.io\n      - group: networking.k8s.io\n      - group: policy\n      - group: rbac.authorization.k8s.io\n      - group: scheduling.k8s.io\n      - group: settings.k8s.io\n      - group: storage.k8s.io\n      \n  # Default level for all other requests.\n  - level: Metadata\n    omitStages:\n      - RequestReceived\nEOF\n```\n\n分发审计策略文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp audit-policy.yaml root@${node_ip}:/etc/kubernetes/audit-policy.yaml\n  done\n```\n\n## 创建后续访问 metrics-server 或 kube-prometheus 使用的证书\n\n创建证书签名请求:\n\n``` bash\ncd /opt/k8s/work\ncat > proxy-client-csr.json <<EOF\n{\n  \"CN\": \"aggregator\",\n  \"hosts\": [],\n  \"key\": {\n    \"algo\": \"rsa\",\n    \"size\": 2048\n  },\n  \"names\": [\n    {\n      \"C\": \"CN\",\n      \"ST\": \"BeiJing\",\n      \"L\": \"BeiJing\",\n      \"O\": \"k8s\",\n      \"OU\": \"opsnull\"\n    }\n  ]\n}\nEOF\n```\n+ CN 名称需要位于 kube-apiserver 的 `--requestheader-allowed-names` 参数中，否则后续访问 metrics 时会提示权限不足。\n\n生成证书和私钥：\n\n``` bash\ncfssl gencert -ca=/etc/kubernetes/cert/ca.pem \\\n  -ca-key=/etc/kubernetes/cert/ca-key.pem  \\\n  -config=/etc/kubernetes/cert/ca-config.json  \\\n  -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client\nls proxy-client*.pem\n```\n\n将生成的证书和私钥文件拷贝到所有 master 节点：\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp proxy-client*.pem root@${node_ip}:/etc/kubernetes/cert/\n  done\n```\n\n## 创建 kube-apiserver systemd unit 模板文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > kube-apiserver.service.template <<EOF\n[Unit]\nDescription=Kubernetes API Server\nDocumentation=https://github.com/GoogleCloudPlatform/kubernetes\nAfter=network.target\n\n[Service]\nWorkingDirectory=${K8S_DIR}/kube-apiserver\nExecStart=/opt/k8s/bin/kube-apiserver \\\\\n  --advertise-address=##NODE_IP## \\\\\n  --default-not-ready-toleration-seconds=360 \\\\\n  --default-unreachable-toleration-seconds=360 \\\\\n  --feature-gates=DynamicAuditing=true \\\\\n  --max-mutating-requests-inflight=2000 \\\\\n  --max-requests-inflight=4000 \\\\\n  --default-watch-cache-size=200 \\\\\n  --delete-collection-workers=2 \\\\\n  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\\\\n  --etcd-cafile=/etc/kubernetes/cert/ca.pem \\\\\n  --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\\\\n  --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\\\\n  --etcd-servers=${ETCD_ENDPOINTS} \\\\\n  --bind-address=##NODE_IP## \\\\\n  --secure-port=6443 \\\\\n  --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\\\\n  --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\\\\n  --insecure-port=0 \\\\\n  --audit-dynamic-configuration \\\\\n  --audit-log-maxage=15 \\\\\n  --audit-log-maxbackup=3 \\\\\n  --audit-log-maxsize=100 \\\\\n  --audit-log-truncate-enabled \\\\\n  --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\\\\n  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \\\\\n  --profiling \\\\\n  --anonymous-auth=false \\\\\n  --client-ca-file=/etc/kubernetes/cert/ca.pem \\\\\n  --enable-bootstrap-token-auth \\\\\n  --requestheader-allowed-names=\"aggregator\" \\\\\n  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\\\\n  --requestheader-extra-headers-prefix=\"X-Remote-Extra-\" \\\\\n  --requestheader-group-headers=X-Remote-Group \\\\\n  --requestheader-username-headers=X-Remote-User \\\\\n  --service-account-key-file=/etc/kubernetes/cert/ca.pem \\\\\n  --authorization-mode=Node,RBAC \\\\\n  --runtime-config=api/all=true \\\\\n  --enable-admission-plugins=NodeRestriction \\\\\n  --allow-privileged=true \\\\\n  --apiserver-count=3 \\\\\n  --event-ttl=168h \\\\\n  --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\\\\n  --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\\\\n  --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\\\\n  --kubelet-https=true \\\\\n  --kubelet-timeout=10s \\\\\n  --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\\\\n  --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\\\\n  --service-cluster-ip-range=${SERVICE_CIDR} \\\\\n  --service-node-port-range=${NODE_PORT_RANGE} \\\\\n  --logtostderr=true \\\\\n  --v=2\nRestart=on-failure\nRestartSec=10\nType=notify\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\n+ `--advertise-address`：apiserver 对外通告的 IP（kubernetes 服务后端节点 IP）；\n+ `--default-*-toleration-seconds`：设置节点异常相关的阈值；\n+ `--max-*-requests-inflight`：请求相关的最大阈值；\n+ `--etcd-*`：访问 etcd 的证书和 etcd 服务器地址；\n+ `--bind-address`： https 监听的 IP，不能为 `127.0.0.1`，否则外界不能访问它的安全端口 6443；\n+ `--secret-port`：https 监听端口；\n+ `--insecure-port=0`：关闭监听 http 非安全端口(8080)；\n+ `--tls-*-file`：指定 apiserver 使用的证书、私钥和 CA 文件；\n+ `--audit-*`：配置审计策略和审计日志文件相关的参数；\n+ `--client-ca-file`：验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书；\n+ `--enable-bootstrap-token-auth`：启用 kubelet bootstrap 的 token 认证；\n+ `--requestheader-*`：kube-apiserver 的 aggregator layer 相关的配置参数，proxy-client & HPA 需要使用；\n+ `--requestheader-client-ca-file`：用于签名 `--proxy-client-cert-file` 和 `--proxy-client-key-file` 指定的证书；在启用了 metric aggregator 时使用；\n+ `--requestheader-allowed-names`：不能为空，值为逗号分割的 `--proxy-client-cert-file` 证书的 CN 名称，这里设置为 \"aggregator\"；\n+ `--service-account-key-file`：签名 ServiceAccount Token 的公钥文件，kube-controller-manager 的 `--service-account-private-key-file` 指定私钥文件，两者配对使用；\n+ `--runtime-config=api/all=true`： 启用所有版本的 APIs，如 autoscaling/v2alpha1；\n+ `--authorization-mode=Node,RBAC`、`--anonymous-auth=false`： 开启 Node 和 RBAC 授权模式，拒绝未授权的请求；\n+ `--enable-admission-plugins`：启用一些默认关闭的 plugins；\n+ `--allow-privileged`：运行执行 privileged 权限的容器；\n+ `--apiserver-count=3`：指定 apiserver 实例的数量；\n+ `--event-ttl`：指定 events 的保存时间；\n+ `--kubelet-*`：如果指定，则使用 https 访问 kubelet APIs；需要为证书对应的用户(上面 kubernetes*.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则，否则访问 kubelet API 时提示未授权；\n+ `--proxy-client-*`：apiserver 访问 metrics-server 使用的证书；\n+ `--service-cluster-ip-range`： 指定 Service Cluster IP 地址段；\n+ `--service-node-port-range`： 指定 NodePort 的端口范围；\n\n如果 kube-apiserver 机器**没有**运行 kube-proxy，则还需要添加 `--enable-aggregator-routing=true` 参数；\n\n关于 `--requestheader-XXX` 相关参数，参考：\n\n+ https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md\n+ https://docs.bitnami.com/kubernetes/how-to/configure-autoscaling-custom-metrics/\n\n注意：\n1. `--requestheader-client-ca-file` 指定的 CA 证书，必须具有 `client auth and server auth`；\n2. 如果 `--requestheader-allowed-names` 不为空,且 `--proxy-client-cert-file` 证书的 CN 名称不在 allowed-names 中，则后续查看 node 或 pods 的 metrics 失败，提示：\n  ``` bash\n  $ kubectl top nodes\n  Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User \"aggregator\" cannot list resource \"nodes\" in API group \"metrics.k8s.io\" at the cluster scope\n  ```\n\n## 为各节点创建和分发 kube-apiserver systemd unit 文件\n\n替换模板文件中的变量，为各节点生成 systemd unit 文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor (( i=0; i < 3; i++ ))\n  do\n    sed -e \"s/##NODE_NAME##/${NODE_NAMES[i]}/\" -e \"s/##NODE_IP##/${NODE_IPS[i]}/\" kube-apiserver.service.template > kube-apiserver-${NODE_IPS[i]}.service \n  done\nls kube-apiserver*.service\n```\n+ NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP；\n\n分发生成的 systemd unit 文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp kube-apiserver-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-apiserver.service\n  done\n```\n\n## 启动 kube-apiserver 服务\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p ${K8S_DIR}/kube-apiserver\"\n    ssh root@${node_ip} \"systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver\"\n  done\n```\n\n## 检查 kube-apiserver 运行状态\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl status kube-apiserver |grep 'Active:'\"\n  done\n```\n\n确保状态为 `active (running)`，否则查看日志，确认原因：\n\n``` bash\njournalctl -u kube-apiserver\n```\n\n## 检查集群状态\n\n``` bash\n$ kubectl cluster-info\nKubernetes master is running at https://172.27.138.251:6443\n\nTo further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\n\n$ kubectl get all --all-namespaces\nNAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE\ndefault     service/kubernetes   ClusterIP   10.254.0.1   <none>        443/TCP   3m53s\n\n$ kubectl get componentstatuses\nNAME                 AGE\ncontroller-manager   <unknown>\nscheduler            <unknown>\netcd-0               <unknown>\netcd-2               <unknown>\netcd-1               <unknown>\n```\n+ Kubernetes 1.16.6 存在 Bugs 导致返回结果一直为 `<unknown>`，但 `kubectl get cs -o yaml` 可以返回正确结果；\n\n## 检查 kube-apiserver 监听的端口\n\n``` bash\n$ sudo netstat -lnpt|grep kube\ntcp        0      0 172.27.138.251:6443     0.0.0.0:*               LISTEN      101442/kube-apiserv\n```\n+ 6443: 接收 https 请求的安全端口，对所有请求做认证和授权；\n+ 由于关闭了非安全端口，故没有监听 8080；"
  },
  {
    "path": "05-3.controller-manager集群.md",
    "content": "tags: master, kube-controller-manager\n\n# 05-3. 部署高可用 kube-controller-manager 集群\n\n<!-- TOC -->\n\n- [05-3. 部署高可用 kube-controller-manager 集群](#05-3-部署高可用-kube-controller-manager-集群)\n    - [创建 kube-controller-manager 证书和私钥](#创建-kube-controller-manager-证书和私钥)\n    - [创建和分发 kubeconfig 文件](#创建和分发-kubeconfig-文件)\n    - [创建 kube-controller-manager systemd unit 模板文件](#创建-kube-controller-manager-systemd-unit-模板文件)\n    - [为各节点创建和分发 kube-controller-mananger systemd unit 文件](#为各节点创建和分发-kube-controller-mananger-systemd-unit-文件)\n    - [启动 kube-controller-manager 服务](#启动-kube-controller-manager-服务)\n    - [检查服务运行状态](#检查服务运行状态)\n    - [查看输出的 metrics](#查看输出的-metrics)\n    - [查看当前的 leader](#查看当前的-leader)\n    - [测试 kube-controller-manager 集群的高可用](#测试-kube-controller-manager-集群的高可用)\n    - [参考](#参考)\n\n<!-- /TOC -->\n\n本文档介绍部署高可用 kube-controller-manager 集群的步骤。\n\n该集群包含 3 个节点，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用时，阻塞的节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。\n\n为保证通信安全，本文档先生成 x509 证书和私钥，kube-controller-manager 在如下两种情况下使用该证书：\n\n1. 与 kube-apiserver 的安全端口通信;\n2. 在**安全端口**(https，10252) 输出 prometheus 格式的 metrics；\n\n注意：如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**。\n\n## 创建 kube-controller-manager 证书和私钥\n\n创建证书签名请求：\n\n``` bash\ncd /opt/k8s/work\ncat > kube-controller-manager-csr.json <<EOF\n{\n    \"CN\": \"system:kube-controller-manager\",\n    \"key\": {\n        \"algo\": \"rsa\",\n        \"size\": 2048\n    },\n    \"hosts\": [\n      \"127.0.0.1\",\n      \"172.27.138.251\",\n      \"172.27.137.229\",\n      \"172.27.138.239\"\n    ],\n    \"names\": [\n      {\n        \"C\": \"CN\",\n        \"ST\": \"BeiJing\",\n        \"L\": \"BeiJing\",\n        \"O\": \"system:kube-controller-manager\",\n        \"OU\": \"opsnull\"\n      }\n    ]\n}\nEOF\n```\n+ hosts 列表包含**所有** kube-controller-manager 节点 IP；\n+ CN 和 O 均为 `system:kube-controller-manager`，kubernetes 内置的 ClusterRoleBindings `system:kube-controller-manager` 赋予 kube-controller-manager 工作所需的权限。\n\n生成证书和私钥：\n\n``` bash\ncd /opt/k8s/work\ncfssl gencert -ca=/opt/k8s/work/ca.pem \\\n  -ca-key=/opt/k8s/work/ca-key.pem \\\n  -config=/opt/k8s/work/ca-config.json \\\n  -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager\nls kube-controller-manager*pem\n```\n\n将生成的证书和私钥分发到所有 master 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp kube-controller-manager*.pem root@${node_ip}:/etc/kubernetes/cert/\n  done\n```\n\n## 创建和分发 kubeconfig 文件\n\nkube-controller-manager 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-controller-manager 证书等信息：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nkubectl config set-cluster kubernetes \\\n  --certificate-authority=/opt/k8s/work/ca.pem \\\n  --embed-certs=true \\\n  --server=\"https://##NODE_IP##:6443\" \\\n  --kubeconfig=kube-controller-manager.kubeconfig\n\nkubectl config set-credentials system:kube-controller-manager \\\n  --client-certificate=kube-controller-manager.pem \\\n  --client-key=kube-controller-manager-key.pem \\\n  --embed-certs=true \\\n  --kubeconfig=kube-controller-manager.kubeconfig\n\nkubectl config set-context system:kube-controller-manager \\\n  --cluster=kubernetes \\\n  --user=system:kube-controller-manager \\\n  --kubeconfig=kube-controller-manager.kubeconfig\n\nkubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig\n```\n+ kube-controller-manager 与 kube-apiserver 混布，故直接通过**节点 IP** 访问 kube-apiserver；\n\n分发 kubeconfig 到所有 master 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    sed -e \"s/##NODE_IP##/${node_ip}/\" kube-controller-manager.kubeconfig > kube-controller-manager-${node_ip}.kubeconfig\n    scp kube-controller-manager-${node_ip}.kubeconfig root@${node_ip}:/etc/kubernetes/kube-controller-manager.kubeconfig\n  done\n```\n\n## 创建 kube-controller-manager systemd unit 模板文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > kube-controller-manager.service.template <<EOF\n[Unit]\nDescription=Kubernetes Controller Manager\nDocumentation=https://github.com/GoogleCloudPlatform/kubernetes\n\n[Service]\nWorkingDirectory=${K8S_DIR}/kube-controller-manager\nExecStart=/opt/k8s/bin/kube-controller-manager \\\\\n  --profiling \\\\\n  --cluster-name=kubernetes \\\\\n  --controllers=*,bootstrapsigner,tokencleaner \\\\\n  --kube-api-qps=1000 \\\\\n  --kube-api-burst=2000 \\\\\n  --leader-elect \\\\\n  --use-service-account-credentials\\\\\n  --concurrent-service-syncs=2 \\\\\n  --bind-address=##NODE_IP## \\\\\n  --secure-port=10252 \\\\\n  --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\\\\n  --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\\\\n  --port=0 \\\\\n  --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\\\\n  --client-ca-file=/etc/kubernetes/cert/ca.pem \\\\\n  --requestheader-allowed-names=\"aggregator\" \\\\\n  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\\\\n  --requestheader-extra-headers-prefix=\"X-Remote-Extra-\" \\\\\n  --requestheader-group-headers=X-Remote-Group \\\\\n  --requestheader-username-headers=X-Remote-User \\\\\n  --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\\\\n  --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\\\\n  --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\\\\n  --experimental-cluster-signing-duration=876000h \\\\\n  --horizontal-pod-autoscaler-sync-period=10s \\\\\n  --concurrent-deployment-syncs=10 \\\\\n  --concurrent-gc-syncs=30 \\\\\n  --node-cidr-mask-size=24 \\\\\n  --service-cluster-ip-range=${SERVICE_CIDR} \\\\\n  --pod-eviction-timeout=6m \\\\\n  --terminated-pod-gc-threshold=10000 \\\\\n  --root-ca-file=/etc/kubernetes/cert/ca.pem \\\\\n  --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\\\\n  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\\\\n  --logtostderr=true \\\\\n  --v=2\nRestart=on-failure\nRestartSec=5\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n+ `--port=0`：关闭监听非安全端口（http），同时 `--address` 参数无效，`--bind-address` 参数有效；\n+ `--secure-port=10252`、`--bind-address=0.0.0.0`: 在所有网络接口监听 10252 端口的 https /metrics 请求；\n+ `--kubeconfig`：指定 kubeconfig 文件路径，kube-controller-manager 使用它连接和验证 kube-apiserver；\n+ `--authentication-kubeconfig` 和 `--authorization-kubeconfig`：kube-controller-manager 使用它连接 apiserver，对 client 的请求进行认证和授权。`kube-controller-manager` 不再使用 `--tls-ca-file` 对请求 https metrics 的 Client 证书进行校验。如果没有配置这两个 kubeconfig 参数，则 client 连接 kube-controller-manager https 端口的请求会被拒绝(提示权限不足)。\n+ `--cluster-signing-*-file`：签名 TLS Bootstrap 创建的证书；\n+ `--experimental-cluster-signing-duration`：指定 TLS Bootstrap 证书的有效期；\n+ `--root-ca-file`：放置到容器 ServiceAccount 中的 CA 证书，用来对 kube-apiserver 的证书进行校验；\n+ `--service-account-private-key-file`：签名 ServiceAccount 中 Token 的私钥文件，必须和 kube-apiserver 的 `--service-account-key-file` 指定的公钥文件配对使用；\n+ `--service-cluster-ip-range` ：指定 Service Cluster IP 网段，必须和 kube-apiserver 中的同名参数一致；\n+ `--leader-elect=true`：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态；\n+ `--controllers=*,bootstrapsigner,tokencleaner`：启用的控制器列表，tokencleaner 用于自动清理过期的 Bootstrap token；\n+ `--horizontal-pod-autoscaler-*`：custom metrics 相关参数，支持 autoscaling/v2alpha1；\n+ `--tls-cert-file`、`--tls-private-key-file`：使用 https 输出 metrics 时使用的 Server 证书和秘钥；\n+ `--use-service-account-credentials=true`: kube-controller-manager 中各 controller 使用 serviceaccount 访问 kube-apiserver；\n\n## 为各节点创建和分发 kube-controller-mananger systemd unit 文件\n\n替换模板文件中的变量，为各节点创建 systemd unit 文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor (( i=0; i < 3; i++ ))\n  do\n    sed -e \"s/##NODE_NAME##/${NODE_NAMES[i]}/\" -e \"s/##NODE_IP##/${NODE_IPS[i]}/\" kube-controller-manager.service.template > kube-controller-manager-${NODE_IPS[i]}.service \n  done\nls kube-controller-manager*.service\n```\n\n分发到所有 master 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp kube-controller-manager-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-controller-manager.service\n  done\n```\n\n## 启动 kube-controller-manager 服务\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p ${K8S_DIR}/kube-controller-manager\"\n    ssh root@${node_ip} \"systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager\"\n  done\n```\n\n## 检查服务运行状态\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl status kube-controller-manager|grep Active\"\n  done\n```\n\n确保状态为 `active (running)`，否则查看日志，确认原因：\n\n``` bash\njournalctl -u kube-controller-manager\n```\n\nkube-controller-manager 监听 10252 端口，接收 https 请求：\n\n``` bash\n$ sudo netstat -lnpt | grep kube-cont\ntcp        0      0 172.27.138.251:10252    0.0.0.0:*               LISTEN      108977/kube-control\n```\n\n## 查看输出的 metrics\n\n注意：以下命令在 kube-controller-manager 节点上执行。\n\n``` bash\n$ curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.27.138.251:10252/metrics |head\n# HELP ClusterRoleAggregator_adds (Deprecated) Total number of adds handled by workqueue: ClusterRoleAggregator\n# TYPE ClusterRoleAggregator_adds counter\nClusterRoleAggregator_adds 3\n# HELP ClusterRoleAggregator_depth (Deprecated) Current depth of workqueue: ClusterRoleAggregator\n# TYPE ClusterRoleAggregator_depth gauge\nClusterRoleAggregator_depth 0\n# HELP ClusterRoleAggregator_longest_running_processor_microseconds (Deprecated) How many microseconds has the longest running processor for ClusterRoleAggregator been running.\n# TYPE ClusterRoleAggregator_longest_running_processor_microseconds gauge\nClusterRoleAggregator_longest_running_processor_microseconds 0\n# HELP ClusterRoleAggregator_queue_latency (Deprecated) How long an item stays in workqueueClusterRoleAggregator before being requested.\n```\n\n## 查看当前的 leader\n\n``` bash\n$ kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml\napiVersion: v1\nkind: Endpoints\nmetadata:\n  annotations:\n    control-plane.alpha.kubernetes.io/leader: '{\"holderIdentity\":\"zhangjun-k8s-03_e334e88d-6b52-40e0-b2a1-a6f7e47593e1\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-02-07T07:01:32Z\",\"renewTime\":\"2020-02-07T07:01:44Z\",\"leaderTransitions\":1}'\n  creationTimestamp: \"2020-02-07T06:59:38Z\"\n  name: kube-controller-manager\n  namespace: kube-system\n  resourceVersion: \"561\"\n  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\n  uid: e5d52a8c-fe69-4910-a125-d7ec97cead16\n```\n\n可见，当前的 leader 为 zhangjun-k8s-03 节点。\n\n## 测试 kube-controller-manager 集群的高可用\n\n停掉一个或两个节点的 kube-controller-manager 服务，观察其它节点的日志，看是否获取了 leader 权限。\n\n## 参考\n\n1. 关于 controller 权限和 use-service-account-credentials 参数：https://github.com/kubernetes/kubernetes/issues/48208\n2. kubelet 认证和授权：https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization\n"
  },
  {
    "path": "05-4.scheduler集群.md",
    "content": "tags: master, kube-scheduler\n\n# 05-4. 部署高可用 kube-scheduler 集群\n\n<!-- TOC -->\n\n- [05-4. 部署高可用 kube-scheduler 集群](#05-4-部署高可用-kube-scheduler-集群)\n    - [创建 kube-scheduler 证书和私钥](#创建-kube-scheduler-证书和私钥)\n    - [创建和分发 kubeconfig 文件](#创建和分发-kubeconfig-文件)\n    - [创建 kube-scheduler 配置文件](#创建-kube-scheduler-配置文件)\n    - [创建 kube-scheduler systemd unit 模板文件](#创建-kube-scheduler-systemd-unit-模板文件)\n    - [为各节点创建和分发 kube-scheduler systemd unit 文件](#为各节点创建和分发-kube-scheduler-systemd-unit-文件)\n    - [启动 kube-scheduler 服务](#启动-kube-scheduler-服务)\n    - [检查服务运行状态](#检查服务运行状态)\n    - [查看输出的 metrics](#查看输出的-metrics)\n    - [查看当前的 leader](#查看当前的-leader)\n    - [测试 kube-scheduler 集群的高可用](#测试-kube-scheduler-集群的高可用)\n\n<!-- /TOC -->\n\n本文档介绍部署高可用 kube-scheduler 集群的步骤。\n\n该集群包含 3 个节点，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用后，剩余节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。\n\n为保证通信安全，本文档先生成 x509 证书和私钥，kube-scheduler 在如下两种情况下使用该证书：\n\n1. 与 kube-apiserver 的安全端口通信;\n2. 在**安全端口**(https，10251) 输出 prometheus 格式的 metrics；\n\n注意：如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**。\n\n## 创建 kube-scheduler 证书和私钥\n\n创建证书签名请求：\n\n``` bash\ncd /opt/k8s/work\ncat > kube-scheduler-csr.json <<EOF\n{\n    \"CN\": \"system:kube-scheduler\",\n    \"hosts\": [\n      \"127.0.0.1\",\n      \"172.27.138.239\",\n      \"172.27.137.229\",\n      \"172.27.138.251\"\n    ],\n    \"key\": {\n        \"algo\": \"rsa\",\n        \"size\": 2048\n    },\n    \"names\": [\n      {\n        \"C\": \"CN\",\n        \"ST\": \"BeiJing\",\n        \"L\": \"BeiJing\",\n        \"O\": \"system:kube-scheduler\",\n        \"OU\": \"opsnull\"\n      }\n    ]\n}\nEOF\n```\n+ hosts 列表包含**所有** kube-scheduler 节点 IP；\n+ CN 和 O 均为 `system:kube-scheduler`，kubernetes 内置的 ClusterRoleBindings `system:kube-scheduler` 将赋予 kube-scheduler 工作所需的权限；\n\n生成证书和私钥：\n\n``` bash\ncd /opt/k8s/work\ncfssl gencert -ca=/opt/k8s/work/ca.pem \\\n  -ca-key=/opt/k8s/work/ca-key.pem \\\n  -config=/opt/k8s/work/ca-config.json \\\n  -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler\nls kube-scheduler*pem\n```\n\n将生成的证书和私钥分发到所有 master 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp kube-scheduler*.pem root@${node_ip}:/etc/kubernetes/cert/\n  done\n```\n\n## 创建和分发 kubeconfig 文件\n\nkube-scheduler 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-scheduler 证书：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nkubectl config set-cluster kubernetes \\\n  --certificate-authority=/opt/k8s/work/ca.pem \\\n  --embed-certs=true \\\n  --server=\"https://##NODE_IP##:6443\" \\\n  --kubeconfig=kube-scheduler.kubeconfig\n\nkubectl config set-credentials system:kube-scheduler \\\n  --client-certificate=kube-scheduler.pem \\\n  --client-key=kube-scheduler-key.pem \\\n  --embed-certs=true \\\n  --kubeconfig=kube-scheduler.kubeconfig\n\nkubectl config set-context system:kube-scheduler \\\n  --cluster=kubernetes \\\n  --user=system:kube-scheduler \\\n  --kubeconfig=kube-scheduler.kubeconfig\n\nkubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig\n```\n\n分发 kubeconfig 到所有 master 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    sed -e \"s/##NODE_IP##/${node_ip}/\" kube-scheduler.kubeconfig > kube-scheduler-${node_ip}.kubeconfig\n    scp kube-scheduler-${node_ip}.kubeconfig root@${node_ip}:/etc/kubernetes/kube-scheduler.kubeconfig\n  done\n```\n\n## 创建 kube-scheduler 配置文件\n\n``` bash\ncd /opt/k8s/work\ncat >kube-scheduler.yaml.template <<EOF\napiVersion: kubescheduler.config.k8s.io/v1alpha1\nkind: KubeSchedulerConfiguration\nbindTimeoutSeconds: 600\nclientConnection:\n  burst: 200\n  kubeconfig: \"/etc/kubernetes/kube-scheduler.kubeconfig\"\n  qps: 100\nenableContentionProfiling: false\nenableProfiling: true\nhardPodAffinitySymmetricWeight: 1\nhealthzBindAddress: ##NODE_IP##:10251\nleaderElection:\n  leaderElect: true\nmetricsBindAddress: ##NODE_IP##:10251\nEOF\n```\n+ `--kubeconfig`：指定 kubeconfig 文件路径，kube-scheduler 使用它连接和验证 kube-apiserver；\n+ `--leader-elect=true`：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态；\n\n替换模板文件中的变量：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor (( i=0; i < 3; i++ ))\n  do\n    sed -e \"s/##NODE_NAME##/${NODE_NAMES[i]}/\" -e \"s/##NODE_IP##/${NODE_IPS[i]}/\" kube-scheduler.yaml.template > kube-scheduler-${NODE_IPS[i]}.yaml\n  done\nls kube-scheduler*.yaml\n```\n+ NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP；\n\n分发 kube-scheduler 配置文件到所有 master 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp kube-scheduler-${node_ip}.yaml root@${node_ip}:/etc/kubernetes/kube-scheduler.yaml\n  done\n```\n+ 重命名为 kube-scheduler.yaml;\n\n## 创建 kube-scheduler systemd unit 模板文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > kube-scheduler.service.template <<EOF\n[Unit]\nDescription=Kubernetes Scheduler\nDocumentation=https://github.com/GoogleCloudPlatform/kubernetes\n\n[Service]\nWorkingDirectory=${K8S_DIR}/kube-scheduler\nExecStart=/opt/k8s/bin/kube-scheduler \\\\\n  --config=/etc/kubernetes/kube-scheduler.yaml \\\\\n  --bind-address=##NODE_IP## \\\\\n  --secure-port=10259 \\\\\n  --port=0 \\\\\n  --tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem \\\\\n  --tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem \\\\\n  --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\\\\n  --client-ca-file=/etc/kubernetes/cert/ca.pem \\\\\n  --requestheader-allowed-names=\"\" \\\\\n  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\\\\n  --requestheader-extra-headers-prefix=\"X-Remote-Extra-\" \\\\\n  --requestheader-group-headers=X-Remote-Group \\\\\n  --requestheader-username-headers=X-Remote-User \\\\\n  --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\\\\n  --logtostderr=true \\\\\n  --v=2\nRestart=always\nRestartSec=5\nStartLimitInterval=0\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\n## 为各节点创建和分发 kube-scheduler systemd unit 文件\n\n替换模板文件中的变量，为各节点创建 systemd unit 文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor (( i=0; i < 3; i++ ))\n  do\n    sed -e \"s/##NODE_NAME##/${NODE_NAMES[i]}/\" -e \"s/##NODE_IP##/${NODE_IPS[i]}/\" kube-scheduler.service.template > kube-scheduler-${NODE_IPS[i]}.service \n  done\nls kube-scheduler*.service\n```\n\n分发 systemd unit 文件到所有 master 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp kube-scheduler-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-scheduler.service\n  done\n```\n\n## 启动 kube-scheduler 服务\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p ${K8S_DIR}/kube-scheduler\"\n    ssh root@${node_ip} \"systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler\"\n  done\n```\n\n## 检查服务运行状态\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl status kube-scheduler|grep Active\"\n  done\n```\n\n确保状态为 `active (running)`，否则查看日志，确认原因：\n\n``` bash\njournalctl -u kube-scheduler\n```\n\n## 查看输出的 metrics\n\n注意：以下命令在 kube-scheduler 节点上执行。\n\nkube-scheduler 监听 10251 和 10259 端口：\n+ 10251：接收 http 请求，非安全端口，不需要认证授权；\n+ 10259：接收 https 请求，安全端口，需要认证授权；\n\n两个接口都对外提供 `/metrics` 和 `/healthz` 的访问。\n\n```\n$ sudo netstat -lnpt |grep kube-sch\ntcp        0      0 172.27.138.251:10251    0.0.0.0:*               LISTEN      114702/kube-schedul\ntcp        0      0 172.27.138.251:10259    0.0.0.0:*               LISTEN      114702/kube-schedul\n```\n\n``` bash\n$ curl -s http://172.27.138.251:10251/metrics |head\n# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.\n# TYPE apiserver_audit_event_total counter\napiserver_audit_event_total 0\n# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.\n# TYPE apiserver_audit_requests_rejected_total counter\napiserver_audit_requests_rejected_total 0\n# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.\n# TYPE apiserver_client_certificate_expiration_seconds histogram\napiserver_client_certificate_expiration_seconds_bucket{le=\"0\"} 0\napiserver_client_certificate_expiration_seconds_bucket{le=\"1800\"} 0\n```\n\n``` bash\n$ curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.27.138.251:10259/metrics |head\n# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.\n# TYPE apiserver_audit_event_total counter\napiserver_audit_event_total 0\n# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.\n# TYPE apiserver_audit_requests_rejected_total counter\napiserver_audit_requests_rejected_total 0\n# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.\n# TYPE apiserver_client_certificate_expiration_seconds histogram\napiserver_client_certificate_expiration_seconds_bucket{le=\"0\"} 0\napiserver_client_certificate_expiration_seconds_bucket{le=\"1800\"} 0\n```\n\n## 查看当前的 leader\n\n``` bash\n$ kubectl get endpoints kube-scheduler --namespace=kube-system  -o yaml\napiVersion: v1\nkind: Endpoints\nmetadata:\n  annotations:\n    control-plane.alpha.kubernetes.io/leader: '{\"holderIdentity\":\"zhangjun-k8s-01_ce04632e-64e4-477e-b8f0-4e69020cd996\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-02-07T07:05:00Z\",\"renewTime\":\"2020-02-07T07:05:28Z\",\"leaderTransitions\":0}'\n  creationTimestamp: \"2020-02-07T07:05:00Z\"\n  name: kube-scheduler\n  namespace: kube-system\n  resourceVersion: \"756\"\n  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler\n  uid: 1b687724-a6e2-4404-9efb-a1f0e201fecc\n```\n\n可见，当前的 leader 为 zhangjun-k8s-01 节点。\n\n## 测试 kube-scheduler 集群的高可用\n\n随便找一个或两个 master 节点，停掉 kube-scheduler 服务，看其它节点是否获取了 leader 权限。\n"
  },
  {
    "path": "06-1.worker节点.md",
    "content": "tags: worker, containerd, calico, kubeconfig, kubelet, kube-proxy\n\n# 06-1. 部署 worker 节点\n\n<!-- TOC -->\n\n- [06-1. 部署 worker 节点](#06-1-部署-worker-节点)\n    - [安装依赖包](#安装依赖包)\n\n<!-- /TOC -->\n\nkubernetes worker 节点运行如下组件：\n\n+ containerd\n+ kubelet\n+ kube-proxy\n+ calico\n+ kube-nginx\n\n注意：如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**。\n\n## 安装依赖包\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"yum install -y epel-release\" &\n    ssh root@${node_ip} \"yum install -y chrony conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget socat git\" &\n  done\n```"
  },
  {
    "path": "06-2.apiserver高可用.md",
    "content": "tags: worker, kube-nginx\n\n# 06-2. apiserver 高可用\n\n<!-- TOC -->\n\n- [06-2. apiserver 高可用](#06-2-apiserver-高可用)\n    - [基于 nginx 代理的 kube-apiserver 高可用方案](#基于-nginx-代理的-kube-apiserver-高可用方案)\n    - [下载和编译 nginx](#下载和编译-nginx)\n    - [验证编译的 nginx](#验证编译的-nginx)\n    - [安装和部署 nginx](#安装和部署-nginx)\n    - [配置 systemd unit 文件，启动服务](#配置-systemd-unit-文件启动服务)\n    - [检查 kube-nginx 服务运行状态](#检查-kube-nginx-服务运行状态)\n\n<!-- /TOC -->\n\n本文档讲解使用 nginx 4 层透明代理功能实现 Kubernetes worker 节点组件高可用访问 kube-apiserver 集群的步骤。\n\n注意：如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**。\n\n## 基于 nginx 代理的 kube-apiserver 高可用方案\n\n+ 控制节点的 kube-controller-manager、kube-scheduler 是多实例部署且连接本机的 kube-apiserver，所以只要有一个实例正常，就可以保证高可用；\n+ 集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver， kube-dns 会自动解析出多个 kube-apiserver 节点的 IP，所以也是高可用的；\n+ 在每个节点起一个 nginx 进程，后端对接多个 apiserver 实例，nginx 对它们做健康检查和负载均衡；\n+ kubelet、kube-proxy 通过本地的 nginx（监听 127.0.0.1）访问 kube-apiserver，从而实现 kube-apiserver 的高可用；\n\n## 下载和编译 nginx\n\n下载源码：\n\n``` bash\ncd /opt/k8s/work\nwget http://nginx.org/download/nginx-1.15.3.tar.gz\ntar -xzvf nginx-1.15.3.tar.gz\n```\n\n配置编译参数：\n\n``` bash\ncd /opt/k8s/work/nginx-1.15.3\nmkdir nginx-prefix\nyum install -y gcc make\n./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module\n```\n+ `--with-stream`：开启 4 层透明转发(TCP Proxy)功能；\n+ `--without-xxx`：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小；\n\n输出：\n\n``` bash\nConfiguration summary\n  + PCRE library is not used\n  + OpenSSL library is not used\n  + zlib library is not used\n\n  nginx path prefix: \"/root/tmp/nginx-1.15.3/nginx-prefix\"\n  nginx binary file: \"/root/tmp/nginx-1.15.3/nginx-prefix/sbin/nginx\"\n  nginx modules path: \"/root/tmp/nginx-1.15.3/nginx-prefix/modules\"\n  nginx configuration prefix: \"/root/tmp/nginx-1.15.3/nginx-prefix/conf\"\n  nginx configuration file: \"/root/tmp/nginx-1.15.3/nginx-prefix/conf/nginx.conf\"\n  nginx pid file: \"/root/tmp/nginx-1.15.3/nginx-prefix/logs/nginx.pid\"\n  nginx error log file: \"/root/tmp/nginx-1.15.3/nginx-prefix/logs/error.log\"\n  nginx http access log file: \"/root/tmp/nginx-1.15.3/nginx-prefix/logs/access.log\"\n  nginx http client request body temporary files: \"client_body_temp\"\n  nginx http proxy temporary files: \"proxy_temp\"\n```\n\n编译和安装：\n\n``` bash\ncd /opt/k8s/work/nginx-1.15.3\nmake && make install\n```\n\n## 验证编译的 nginx \n\n``` bash\ncd /opt/k8s/work/nginx-1.15.3\n./nginx-prefix/sbin/nginx -v\n```\n\n输出：\n\n``` bash\nnginx version: nginx/1.15.3\n```\n\n## 安装和部署 nginx\n\n创建目录结构：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}\"\n  done\n```\n\n拷贝二进制程序：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}\"\n    scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx  root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx\n    ssh root@${node_ip} \"chmod a+x /opt/k8s/kube-nginx/sbin/*\"\n  done\n```\n+ 重命名二进制文件为 kube-nginx；\n\n配置 nginx，开启 4 层透明转发功能：\n\n``` bash\ncd /opt/k8s/work\ncat > kube-nginx.conf << \\EOF\nworker_processes 1;\n\nevents {\n    worker_connections  1024;\n}\n\nstream {\n    upstream backend {\n        hash $remote_addr consistent;\n        server 172.27.138.251:6443        max_fails=3 fail_timeout=30s;\n        server 172.27.137.229:6443        max_fails=3 fail_timeout=30s;\n        server 172.27.138.239:6443        max_fails=3 fail_timeout=30s;\n    }\n\n    server {\n        listen 127.0.0.1:8443;\n        proxy_connect_timeout 1s;\n        proxy_pass backend;\n    }\n}\nEOF\n```\n+ `upstream backend` 中的 server 列表为集群中各 kube-apiserver 的节点 IP，**需要根据实际情况修改**；\n\n分发配置文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp kube-nginx.conf  root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf\n  done\n```\n\n## 配置 systemd unit 文件，启动服务\n\n配置 kube-nginx systemd unit 文件：\n\n``` bash\ncd /opt/k8s/work\ncat > kube-nginx.service <<EOF\n[Unit]\nDescription=kube-apiserver nginx proxy\nAfter=network.target\nAfter=network-online.target\nWants=network-online.target\n\n[Service]\nType=forking\nExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t\nExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx\nExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload\nPrivateTmp=true\nRestart=always\nRestartSec=5\nStartLimitInterval=0\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\n分发 systemd unit 文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp kube-nginx.service  root@${node_ip}:/etc/systemd/system/\n  done\n```\n\n启动 kube-nginx 服务：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx\"\n  done\n```\n\n## 检查 kube-nginx 服务运行状态\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl status kube-nginx |grep 'Active:'\"\n  done\n```\n\n确保状态为 `active (running)`，否则查看日志，确认原因：\n\n``` bash\njournalctl -u kube-nginx\n```"
  },
  {
    "path": "06-3.containerd.md",
    "content": "tags: worker, containerd\n\n# 06-3. 部署 containerd 组件\n<!-- TOC -->\n\n- [06-3. 部署 containerd 组件](#06-3-部署-containerd-组件)\n    - [下载和分发二进制文件](#下载和分发二进制文件)\n    - [创建和分发 containerd 配置文件](#创建和分发-containerd-配置文件)\n    - [创建 containerd systemd unit 文件](#创建-containerd-systemd-unit-文件)\n    - [分发 systemd unit 文件，启动 containerd 服务](#分发-systemd-unit-文件启动-containerd-服务)\n    - [创建和分发 crictl 配置文件](#创建和分发-crictl-配置文件)\n\n<!-- /TOC -->\n\ncontainerd 实现了 kubernetes 的 Container Runtime Interface (CRI) 接口，提供容器运行时核心功能，如镜像管理、容器管理等，相比 dockerd 更加简单、健壮和可移植。\n\n注意：\n1. 如果没有特殊指明，本文档的所有操作均在 zhangjun-k8s01 节点上执行。\n2. 如果想使用 docker，请参考附件 [F.部署docker.md](F.部署docker.md)；\n3. docker 需要与 flannel 配合使用，且先安装 flannel；\n\n## 下载和分发二进制文件\n\n下载二进制文件：\n\n``` bash\ncd /opt/k8s/work\nwget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.17.0/crictl-v1.17.0-linux-amd64.tar.gz \\\n  https://github.com/opencontainers/runc/releases/download/v1.0.0-rc10/runc.amd64 \\\n  https://github.com/containernetworking/plugins/releases/download/v0.8.5/cni-plugins-linux-amd64-v0.8.5.tgz \\\n  https://github.com/containerd/containerd/releases/download/v1.3.3/containerd-1.3.3.linux-amd64.tar.gz \n```\n\n解压：\n\n``` bash\ncd /opt/k8s/work\nmkdir containerd\ntar -xvf containerd-1.3.3.linux-amd64.tar.gz -C containerd\ntar -xvf crictl-v1.17.0-linux-amd64.tar.gz\n\nmkdir cni-plugins\nsudo tar -xvf cni-plugins-linux-amd64-v0.8.5.tgz -C cni-plugins\n\nsudo mv runc.amd64 runc\n```\n\n分发二进制文件到所有 worker 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp containerd/bin/*  crictl  cni-plugins/*  runc  root@${node_ip}:/opt/k8s/bin\n    ssh root@${node_ip} \"chmod a+x /opt/k8s/bin/* && mkdir -p /etc/cni/net.d\"\n  done\n```\n\n## 创建和分发 containerd 配置文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat << EOF | sudo tee containerd-config.toml\nversion = 2\nroot = \"${CONTAINERD_DIR}/root\"\nstate = \"${CONTAINERD_DIR}/state\"\n\n[plugins]\n  [plugins.\"io.containerd.grpc.v1.cri\"]\n    sandbox_image = \"registry.cn-beijing.aliyuncs.com/zhoujun/pause-amd64:3.1\"\n    [plugins.\"io.containerd.grpc.v1.cri\".cni]\n      bin_dir = \"/opt/k8s/bin\"\n      conf_dir = \"/etc/cni/net.d\"\n  [plugins.\"io.containerd.runtime.v1.linux\"]\n    shim = \"containerd-shim\"\n    runtime = \"runc\"\n    runtime_root = \"\"\n    no_shim = false\n    shim_debug = false\nEOF\n```\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p /etc/containerd/ ${CONTAINERD_DIR}/{root,state}\"\n    scp containerd-config.toml root@${node_ip}:/etc/containerd/config.toml\n  done\n```\n\n## 创建 containerd systemd unit 文件\n\n``` bash\ncd /opt/k8s/work\ncat <<EOF | sudo tee containerd.service\n[Unit]\nDescription=containerd container runtime\nDocumentation=https://containerd.io\nAfter=network.target\n\n[Service]\nEnvironment=\"PATH=/opt/k8s/bin:/bin:/sbin:/usr/bin:/usr/sbin\"\nExecStartPre=/sbin/modprobe overlay\nExecStart=/opt/k8s/bin/containerd\nRestart=always\nRestartSec=5\nDelegate=yes\nKillMode=process\nOOMScoreAdjust=-999\nLimitNOFILE=1048576\nLimitNPROC=infinity\nLimitCORE=infinity\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\n## 分发 systemd unit 文件，启动 containerd 服务\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp containerd.service root@${node_ip}:/etc/systemd/system\n    ssh root@${node_ip} \"systemctl enable containerd && systemctl restart containerd\"\n  done\n```\n\n## 创建和分发 crictl 配置文件\n\ncrictl 是兼容 CRI 容器运行时的命令行工具，提供类似于 docker 命令的功能。具体参考[官方文档](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md)。\n\n``` bash\ncd /opt/k8s/work\ncat << EOF | sudo tee crictl.yaml\nruntime-endpoint: unix:///run/containerd/containerd.sock\nimage-endpoint: unix:///run/containerd/containerd.sock\ntimeout: 10\ndebug: false\nEOF\n```\n\n分发到所有 worker 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp crictl.yaml root@${node_ip}:/etc/crictl.yaml\n  done\n```\n"
  },
  {
    "path": "06-4.kubelet.md",
    "content": "tags: worker, kubelet\n\n# 06-4. 部署 kubelet 组件\n\n<!-- TOC -->\n\n- [06-4. 部署 kubelet 组件](#06-4-部署-kubelet-组件)\n    - [下载和分发 kubelet 二进制文件](#下载和分发-kubelet-二进制文件)\n    - [创建 kubelet bootstrap kubeconfig 文件](#创建-kubelet-bootstrap-kubeconfig-文件)\n    - [分发 bootstrap kubeconfig 文件到所有 worker 节点](#分发-bootstrap-kubeconfig-文件到所有-worker-节点)\n    - [创建和分发 kubelet 参数配置文件](#创建和分发-kubelet-参数配置文件)\n    - [创建和分发 kubelet systemd unit 文件](#创建和分发-kubelet-systemd-unit-文件)\n    - [授予 kube-apiserver 访问 kubelet API 的权限](#授予-kube-apiserver-访问-kubelet-api-的权限)\n    - [Bootstrap Token Auth 和授予权限](#bootstrap-token-auth-和授予权限)\n    - [自动 approve CSR 请求，生成 kubelet client 证书](#自动-approve-csr-请求生成-kubelet-client-证书)\n    - [启动 kubelet 服务](#启动-kubelet-服务)\n    - [查看 kubelet 情况](#查看-kubelet-情况)\n    - [手动 approve server cert csr](#手动-approve-server-cert-csr)\n    - [kubelet api 认证和授权](#kubelet-api-认证和授权)\n        - [证书认证和授权](#证书认证和授权)\n        - [bear token 认证和授权](#bear-token-认证和授权)\n        - [cadvisor 和 metrics](#cadvisor-和-metrics)\n    - [参考](#参考)\n\n<!-- /TOC -->\n\nkubelet 运行在每个 worker 节点上，接收 kube-apiserver 发送的请求，管理 Pod 容器，执行交互式命令，如 exec、run、logs 等。\n\nkubelet 启动时自动向 kube-apiserver 注册节点信息，内置的 cadvisor 统计和监控节点的资源使用情况。\n\n为确保安全，部署时关闭了 kubelet 的非安全 http 端口，对请求进行认证和授权，拒绝未授权的访问(如 apiserver、heapster 的请求)。\n\n注意：如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**。\n\n## 下载和分发 kubelet 二进制文件\n\n参考 [05-1.部署master节点.md](05-1.部署master节点.md)。\n\n## 创建 kubelet bootstrap kubeconfig 文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_name in ${NODE_NAMES[@]}\n  do\n    echo \">>> ${node_name}\"\n\n    # 创建 token\n    export BOOTSTRAP_TOKEN=$(kubeadm token create \\\n      --description kubelet-bootstrap-token \\\n      --groups system:bootstrappers:${node_name} \\\n      --kubeconfig ~/.kube/config)\n\n    # 设置集群参数\n    kubectl config set-cluster kubernetes \\\n      --certificate-authority=/etc/kubernetes/cert/ca.pem \\\n      --embed-certs=true \\\n      --server=${KUBE_APISERVER} \\\n      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig\n\n    # 设置客户端认证参数\n    kubectl config set-credentials kubelet-bootstrap \\\n      --token=${BOOTSTRAP_TOKEN} \\\n      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig\n\n    # 设置上下文参数\n    kubectl config set-context default \\\n      --cluster=kubernetes \\\n      --user=kubelet-bootstrap \\\n      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig\n\n    # 设置默认上下文\n    kubectl config use-context default --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig\n  done\n```\n+ 向 kubeconfig 写入的是 token，bootstrap 结束后 kube-controller-manager 为 kubelet 创建 client 和 server 证书；\n\n查看 kubeadm 为各节点创建的 token：\n\n``` bash\n$ kubeadm token list --kubeconfig ~/.kube/config\nTOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION               EXTRA GROUPS\n2sb8wy.euialqfpxfbcljby   23h       2020-02-08T15:36:30+08:00   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:zhangjun-k8s-02\nta7onm.fcen74h0mczyfbz2   23h       2020-02-08T15:36:30+08:00   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:zhangjun-k8s-01\nxk27zp.tylnvywx9kc8sq87   23h       2020-02-08T15:36:30+08:00   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:zhangjun-k8s-03\n```\n+ token 有效期为 1 天，**超期后将不能**再被用来 boostrap kubelet，且会被 kube-controller-manager 的 tokencleaner 清理；\n+ kube-apiserver 接收 kubelet 的 bootstrap token 后，将请求的 user 设置为 `system:bootstrap:<Token ID>`，group 设置为 `system:bootstrappers`，后续将为这个 group 设置 ClusterRoleBinding；\n\n## 分发 bootstrap kubeconfig 文件到所有 worker 节点\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_name in ${NODE_NAMES[@]}\n  do\n    echo \">>> ${node_name}\"\n    scp kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig\n  done\n```\n\n## 创建和分发 kubelet 参数配置文件\n\n从 v1.10 开始，部分 kubelet 参数需在**配置文件**中配置，`kubelet --help` 会提示：\n\n    DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag\n\n创建 kubelet 参数配置文件模板（可配置项参考[代码中注释](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/config/types.go)）：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > kubelet-config.yaml.template <<EOF\nkind: KubeletConfiguration\napiVersion: kubelet.config.k8s.io/v1beta1\naddress: \"##NODE_IP##\"\nstaticPodPath: \"\"\nsyncFrequency: 1m\nfileCheckFrequency: 20s\nhttpCheckFrequency: 20s\nstaticPodURL: \"\"\nport: 10250\nreadOnlyPort: 0\nrotateCertificates: true\nserverTLSBootstrap: true\nauthentication:\n  anonymous:\n    enabled: false\n  webhook:\n    enabled: true\n  x509:\n    clientCAFile: \"/etc/kubernetes/cert/ca.pem\"\nauthorization:\n  mode: Webhook\nregistryPullQPS: 0\nregistryBurst: 20\neventRecordQPS: 0\neventBurst: 20\nenableDebuggingHandlers: true\nenableContentionProfiling: true\nhealthzPort: 10248\nhealthzBindAddress: \"##NODE_IP##\"\nclusterDomain: \"${CLUSTER_DNS_DOMAIN}\"\nclusterDNS:\n  - \"${CLUSTER_DNS_SVC_IP}\"\nnodeStatusUpdateFrequency: 10s\nnodeStatusReportFrequency: 1m\nimageMinimumGCAge: 2m\nimageGCHighThresholdPercent: 85\nimageGCLowThresholdPercent: 80\nvolumeStatsAggPeriod: 1m\nkubeletCgroups: \"\"\nsystemCgroups: \"\"\ncgroupRoot: \"\"\ncgroupsPerQOS: true\ncgroupDriver: cgroupfs\nruntimeRequestTimeout: 10m\nhairpinMode: promiscuous-bridge\nmaxPods: 220\npodCIDR: \"${CLUSTER_CIDR}\"\npodPidsLimit: -1\nresolvConf: /etc/resolv.conf\nmaxOpenFiles: 1000000\nkubeAPIQPS: 1000\nkubeAPIBurst: 2000\nserializeImagePulls: false\nevictionHard:\n  memory.available:  \"100Mi\"\n  nodefs.available:  \"10%\"\n  nodefs.inodesFree: \"5%\"\n  imagefs.available: \"15%\"\nevictionSoft: {}\nenableControllerAttachDetach: true\nfailSwapOn: true\ncontainerLogMaxSize: 20Mi\ncontainerLogMaxFiles: 10\nsystemReserved: {}\nkubeReserved: {}\nsystemReservedCgroup: \"\"\nkubeReservedCgroup: \"\"\nenforceNodeAllocatable: [\"pods\"]\nEOF\n```\n+ address：kubelet 安全端口（https，10250）监听的地址，不能为 127.0.0.1，否则 kube-apiserver、heapster 等不能调用 kubelet 的 API；\n+ readOnlyPort=0：关闭只读端口(默认 10255)，等效为未指定；\n+ authentication.anonymous.enabled：设置为 false，不允许匿名\b访问 10250 端口；\n+ authentication.x509.clientCAFile：指定签名客户端证书的 CA 证书，开启 HTTP 证书认证；\n+ authentication.webhook.enabled=true：开启 HTTPs bearer token 认证；\n+ 对于未通过 x509 证书和 webhook 认证的请求(kube-apiserver 或其他客户端)，将被拒绝，提示 Unauthorized；\n+ authroization.mode=Webhook：kubelet 使用 SubjectAccessReview API 查询 kube-apiserver 某 user、group 是否具有操作资源的权限(RBAC)；\n+ featureGates.RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate：自动 rotate 证书，证书的有效期取决于 kube-controller-manager 的 --experimental-cluster-signing-duration 参数；\n+ 需要 root 账户运行；\n\n为各节点创建和分发 kubelet 配置文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do \n    echo \">>> ${node_ip}\"\n    sed -e \"s/##NODE_IP##/${node_ip}/\" kubelet-config.yaml.template > kubelet-config-${node_ip}.yaml.template\n    scp kubelet-config-${node_ip}.yaml.template root@${node_ip}:/etc/kubernetes/kubelet-config.yaml\n  done\n```\n\n## 创建和分发 kubelet systemd unit 文件\n\n创建 kubelet systemd unit 文件模板：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > kubelet.service.template <<EOF\n[Unit]\nDescription=Kubernetes Kubelet\nDocumentation=https://github.com/GoogleCloudPlatform/kubernetes\nAfter=containerd.service\nRequires=containerd.service\n\n[Service]\nWorkingDirectory=${K8S_DIR}/kubelet\nExecStart=/opt/k8s/bin/kubelet \\\\\n  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\\\\n  --cert-dir=/etc/kubernetes/cert \\\\\n  --network-plugin=cni \\\\\n  --cni-conf-dir=/etc/cni/net.d \\\\\n  --container-runtime=remote \\\\\n  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \\\\\n  --root-dir=${K8S_DIR}/kubelet \\\\\n  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\\\\n  --config=/etc/kubernetes/kubelet-config.yaml \\\\\n  --hostname-override=##NODE_NAME## \\\\\n  --image-pull-progress-deadline=15m \\\\\n  --volume-plugin-dir=${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/ \\\\\n  --logtostderr=true \\\\\n  --v=2\nRestart=always\nRestartSec=5\nStartLimitInterval=0\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n+ 如果设置了 `--hostname-override` 选项，则 `kube-proxy` 也需要设置该选项，否则会出现找不到 Node 的情况；\n+ `--bootstrap-kubeconfig`：指向 bootstrap kubeconfig 文件，kubelet 使用该文件中的用户名和 token 向 kube-apiserver 发送 TLS Bootstrapping 请求；\n+ K8S approve kubelet 的 csr 请求后，在 `--cert-dir` 目录创建证书和私钥文件，然后写入 `--kubeconfig` 文件；\n+ `--pod-infra-container-image` 不使用 redhat 的 `pod-infrastructure:latest` 镜像，它不能回收容器的僵尸；\n\n为各节点创建和分发 kubelet systemd unit 文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_name in ${NODE_NAMES[@]}\n  do \n    echo \">>> ${node_name}\"\n    sed -e \"s/##NODE_NAME##/${node_name}/\" kubelet.service.template > kubelet-${node_name}.service\n    scp kubelet-${node_name}.service root@${node_name}:/etc/systemd/system/kubelet.service\n  done\n```\n\n## 授予 kube-apiserver 访问 kubelet API 的权限\n\n在执行 kubectl exec、run、logs 等命令时，apiserver 会将请求转发到 kubelet 的 https 端口。这里定义 RBAC 规则，授权 apiserver 使用的证书（kubernetes.pem）用户名（CN：kuberntes-master）访问 kubelet API 的权限：\n\n``` bash\nkubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes-master\n```\n\n## Bootstrap Token Auth 和授予权限\n\nkubelet 启动时查找 `--kubeletconfig` 参数对应的文件是否存在，如果不存在则使用 `--bootstrap-kubeconfig` 指定的 kubeconfig 文件向 kube-apiserver 发送证书签名请求 (CSR)。\n\nkube-apiserver 收到 CSR 请求后，对其中的 Token 进行认证，认证通过后将请求的 user 设置为 `system:bootstrap:<Token ID>`，group 设置为 `system:bootstrappers`，这一过程称为 `Bootstrap Token Auth`。\n\n默认情况下，这个 user 和 group 没有创建 CSR 的权限，kubelet 启动失败，错误日志如下：\n\n``` bash\n$ sudo journalctl -u kubelet -a |grep -A 2 'certificatesigningrequests'\nMay 26 12:13:41 zhangjun-k8s-01 kubelet[128468]: I0526 12:13:41.798230  128468 certificate_manager.go:366] Rotating certificates\nMay 26 12:13:41 zhangjun-k8s-01 kubelet[128468]: E0526 12:13:41.801997  128468 certificate_manager.go:385] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:bootstrap:82jfrm\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope\n```\n\n解决办法是：创建一个 clusterrolebinding，将 group system:bootstrappers 和 clusterrole system:node-bootstrapper 绑定：\n\n``` bash\nkubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers\n```\n\n## 自动 approve CSR 请求，生成 kubelet client 证书\n\nkubelet 创建 CSR 请求后，下一步需要创建被 approve，有两种方式：\n1. kube-controller-manager 自动 aprrove；\n2. 手动使用命令 `kubectl certificate approve`；\n\nCSR 被 approve 后，kubelet 向 kube-controller-manager 请求创建 client 证书，kube-controller-manager 中的 `csrapproving` controller 使用 `SubjectAccessReview` API 来检查 kubelet 请求（对应的 group 是 system:bootstrappers）是否具有相应的权限。\n\n创建三个 ClusterRoleBinding，分别授予 group system:bootstrappers 和 group system:nodes 进行 approve client、renew client、renew server 证书的权限（server csr 是手动 approve 的，见后文）：\n\n``` bash\ncd /opt/k8s/work\ncat > csr-crb.yaml <<EOF\n # Approve all CSRs for the group \"system:bootstrappers\"\n kind: ClusterRoleBinding\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n   name: auto-approve-csrs-for-group\n subjects:\n - kind: Group\n   name: system:bootstrappers\n   apiGroup: rbac.authorization.k8s.io\n roleRef:\n   kind: ClusterRole\n   name: system:certificates.k8s.io:certificatesigningrequests:nodeclient\n   apiGroup: rbac.authorization.k8s.io\n---\n # To let a node of the group \"system:nodes\" renew its own credentials\n kind: ClusterRoleBinding\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n   name: node-client-cert-renewal\n subjects:\n - kind: Group\n   name: system:nodes\n   apiGroup: rbac.authorization.k8s.io\n roleRef:\n   kind: ClusterRole\n   name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient\n   apiGroup: rbac.authorization.k8s.io\n---\n# A ClusterRole which instructs the CSR approver to approve a node requesting a\n# serving cert matching its client cert.\nkind: ClusterRole\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n  name: approve-node-server-renewal-csr\nrules:\n- apiGroups: [\"certificates.k8s.io\"]\n  resources: [\"certificatesigningrequests/selfnodeserver\"]\n  verbs: [\"create\"]\n---\n # To let a node of the group \"system:nodes\" renew its own server credentials\n kind: ClusterRoleBinding\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n   name: node-server-cert-renewal\n subjects:\n - kind: Group\n   name: system:nodes\n   apiGroup: rbac.authorization.k8s.io\n roleRef:\n   kind: ClusterRole\n   name: approve-node-server-renewal-csr\n   apiGroup: rbac.authorization.k8s.io\nEOF\nkubectl apply -f csr-crb.yaml\n```\n+ auto-approve-csrs-for-group：自动 approve node 的第一次 CSR； 注意第一次 CSR 时，请求的 Group 为 system:bootstrappers；\n+ node-client-cert-renewal：自动 approve node 后续过期的 client 证书，自动生成的证书 Group 为 system:nodes;\n+ node-server-cert-renewal：自动 approve node 后续过期的 server 证书，自动生成的证书 Group 为 system:nodes;\n\n## 启动 kubelet 服务\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p ${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/\"\n    ssh root@${node_ip} \"/usr/sbin/swapoff -a\"\n    ssh root@${node_ip} \"systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet\"\n  done\n```\n+ 启动服务前必须先创建工作目录；\n+ 关闭 swap 分区，否则 kubelet 会启动失败；\n\nkubelet 启动后使用 --bootstrap-kubeconfig 向 kube-apiserver 发送 CSR 请求，当这个 CSR 被 approve 后，kube-controller-manager 为 kubelet 创建 TLS 客户端证书、私钥和 --kubeletconfig 文件。\n\n注意：kube-controller-manager 需要配置 `--cluster-signing-cert-file` 和 `--cluster-signing-key-file` 参数，才会为 TLS Bootstrap 创建证书和私钥。\n\n## 查看 kubelet 情况\n\n稍等一会，三个节点的 CSR 都被自动 approved：\n\n``` bash\n$ kubectl get csr\nNAME        AGE   REQUESTOR                     CONDITION\ncsr-5rwzm   43s   system:node:zhangjun-k8s-01   Pending\ncsr-65nms   55s   system:bootstrap:2sb8wy       Approved,Issued\ncsr-8t5hj   42s   system:node:zhangjun-k8s-02   Pending\ncsr-jkhhs   41s   system:node:zhangjun-k8s-03   Pending\ncsr-jv7dn   56s   system:bootstrap:ta7onm       Approved,Issued\ncsr-vb6p5   54s   system:bootstrap:xk27zp       Approved,Issued\n```\n+ Pending 的 CSR 用于创建 kubelet server 证书，需要手动 approve，参考后文。\n\n所有节点均注册（NotReady 状态是预期的，后续安装了网络插件后就好）：\n\n``` bash\n$ kubectl get node\nNAME              STATUS     ROLES    AGE   VERSION\nzhangjun-k8s-01   NotReady   <none>   10h   v1.16.6\nzhangjun-k8s-02   NotReady   <none>   10h   v1.16.6\nzhangjun-k8s-03   NotReady   <none>   10h   v1.16.6\n```\n\nkube-controller-manager 为各 node 生成了 kubeconfig 文件和公私钥：\n\n``` bash\n$ ls -l /etc/kubernetes/kubelet.kubeconfig\n-rw------- 1 root root 2246 Feb  7 15:38 /etc/kubernetes/kubelet.kubeconfig\n\n$ ls -l /etc/kubernetes/cert/kubelet-client-*\n-rw------- 1 root root 1281 Feb  7 15:38 /etc/kubernetes/cert/kubelet-client-2020-02-07-15-38-21.pem\nlrwxrwxrwx 1 root root   59 Feb  7 15:38 /etc/kubernetes/cert/kubelet-client-current.pem -> /etc/kubernetes/cert/kubelet-client-2020-02-07-15-38-21.pem\n```\n+ 没有自动生成 kubelet server 证书；\n\n## 手动 approve server cert csr\n\n基于[安全性考虑](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#kubelet-configuration)，CSR approving controllers 不会自动 approve kubelet server 证书签名请求，需要手动 approve：\n\n``` bash\n$ kubectl get csr\nNAME        AGE     REQUESTOR                     CONDITION\ncsr-5rwzm   3m22s   system:node:zhangjun-k8s-01   Pending\ncsr-65nms   3m34s   system:bootstrap:2sb8wy       Approved,Issued\ncsr-8t5hj   3m21s   system:node:zhangjun-k8s-02   Pending\ncsr-jkhhs   3m20s   system:node:zhangjun-k8s-03   Pending\ncsr-jv7dn   3m35s   system:bootstrap:ta7onm       Approved,Issued\ncsr-vb6p5   3m33s   system:bootstrap:xk27zp       Approved,Issued\n\n$ # 手动 approve\n$ kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve\n\n$ # 自动生成了 server 证书\n$  ls -l /etc/kubernetes/cert/kubelet-*\n-rw------- 1 root root 1281 Feb  7 15:38 /etc/kubernetes/cert/kubelet-client-2020-02-07-15-38-21.pem\nlrwxrwxrwx 1 root root   59 Feb  7 15:38 /etc/kubernetes/cert/kubelet-client-current.pem -> /etc/kubernetes/cert/kubelet-client-2020-02-07-15-38-21.pem\n-rw------- 1 root root 1330 Feb  7 15:42 /etc/kubernetes/cert/kubelet-server-2020-02-07-15-42-12.pem\nlrwxrwxrwx 1 root root   59 Feb  7 15:42 /etc/kubernetes/cert/kubelet-server-current.pem -> /etc/kubernetes/cert/kubelet-server-2020-02-07-15-42-12.pem\n```\n\n## kubelet api 认证和授权\n\nkubelet 配置了如下认证参数：\n\n+ authentication.anonymous.enabled：设置为 false，不允许匿名\b访问 10250 端口；\n+ authentication.x509.clientCAFile：指定签名客户端证书的 CA 证书，开启 HTTPs 证书认证；\n+ authentication.webhook.enabled=true：开启 HTTPs bearer token 认证；\n\n同时配置了如下授权参数：\n\n+ authroization.mode=Webhook：开启 RBAC 授权；\n\nkubelet 收到请求后，使用 clientCAFile 对证书签名进行认证，或者查询 bearer token 是否有效。如果两者都没通过，则拒绝请求，提示 `Unauthorized`：\n\n``` bash\n$ curl -s --cacert /etc/kubernetes/cert/ca.pem https://172.27.138.251:10250/metrics\nUnauthorized\n\n$ curl -s --cacert /etc/kubernetes/cert/ca.pem -H \"Authorization: Bearer 123456\" https://172.27.138.251:10250/metrics\nUnauthorized\n```\n\n通过认证后，kubelet 使用 `SubjectAccessReview` API 向 kube-apiserver 发送请求，查询证书或 token 对应的 user、group 是否有操作资源的权限(RBAC)；\n\n### 证书认证和授权\n\n``` bash\n$ # 权限不足的证书；\n$ curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /etc/kubernetes/cert/kube-controller-manager.pem --key /etc/kubernetes/cert/kube-controller-manager-key.pem https://172.27.138.251:10250/metrics\nForbidden (user=system:kube-controller-manager, verb=get, resource=nodes, subresource=metrics)\n\n$ # 使用部署 kubectl 命令行工具时创建的、具有最高权限的 admin 证书；\n$ curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.27.138.251:10250/metrics|head\n# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.\n# TYPE apiserver_audit_event_total counter\napiserver_audit_event_total 0\n# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.\n# TYPE apiserver_audit_requests_rejected_total counter\napiserver_audit_requests_rejected_total 0\n# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.\n# TYPE apiserver_client_certificate_expiration_seconds histogram\napiserver_client_certificate_expiration_seconds_bucket{le=\"0\"} 0\napiserver_client_certificate_expiration_seconds_bucket{le=\"1800\"} 0\n```\n+ `--cacert`、`--cert`、`--key` 的参数值必须是文件路径，如上面的 `./admin.pem` 不能省略 `./`，否则返回 `401 Unauthorized`；\n\n### bear token 认证和授权\n\n创建一个 ServiceAccount，将它和 ClusterRole system:kubelet-api-admin 绑定，从而具有调用 kubelet API 的权限：\n\n``` bash\nkubectl create sa kubelet-api-test\nkubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test\nSECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')\nTOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')\necho ${TOKEN}\n```\n\n``` bash\n$ curl -s --cacert /etc/kubernetes/cert/ca.pem -H \"Authorization: Bearer ${TOKEN}\" https://172.27.138.251:10250/metrics | head\n# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.\n# TYPE apiserver_audit_event_total counter\napiserver_audit_event_total 0\n# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.\n# TYPE apiserver_audit_requests_rejected_total counter\napiserver_audit_requests_rejected_total 0\n# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.\n# TYPE apiserver_client_certificate_expiration_seconds histogram\napiserver_client_certificate_expiration_seconds_bucket{le=\"0\"} 0\napiserver_client_certificate_expiration_seconds_bucket{le=\"1800\"} 0\n```\n\n### cadvisor 和 metrics\n\ncadvisor 是内嵌在 kubelet 二进制中的，统计所在节点各容器的资源(CPU、内存、磁盘、网卡)使用情况的服务。\n\n浏览器访问 https://172.27.138.251:10250/metrics 和 https://172.27.138.251:10250/metrics/cadvisor 分别返回 kubelet 和 cadvisor 的 metrics。\n\n注意：\n+ kubelet.config.json 设置 authentication.anonymous.enabled 为 false，不允许匿名证书访问 10250 的 https 服务；\n+ 参考[A.浏览器访问kube-apiserver安全端口.md](A.浏览器访问kube-apiserver安全端口.md)，创建和导入相关证书，然后访问上面的 10250 端口；\n\n## 参考\n\n1. kubelet 认证和授权：https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/\n"
  },
  {
    "path": "06-5.kube-proxy.md",
    "content": "tags: worker, kube-proxy\n\n# 06-5. 部署 kube-proxy 组件\n\n<!-- TOC -->\n\n- [06-5. 部署 kube-proxy 组件](#06-5-部署-kube-proxy-组件)\n    - [下载和分发 kube-proxy 二进制文件](#下载和分发-kube-proxy-二进制文件)\n    - [创建 kube-proxy 证书](#创建-kube-proxy-证书)\n    - [创建和分发 kubeconfig 文件](#创建和分发-kubeconfig-文件)\n    - [创建 kube-proxy 配置文件](#创建-kube-proxy-配置文件)\n    - [创建和分发 kube-proxy systemd unit 文件](#创建和分发-kube-proxy-systemd-unit-文件)\n    - [启动 kube-proxy 服务](#启动-kube-proxy-服务)\n    - [检查启动结果](#检查启动结果)\n    - [查看监听端口](#查看监听端口)\n    - [查看 ipvs 路由规则](#查看-ipvs-路由规则)\n\n<!-- /TOC -->\n\nkube-proxy 运行在所有 worker 节点上，它监听 apiserver 中 service 和 endpoint 的变化情况，创建路由规则以提供服务 IP 和负载均衡功能。\n\n本文档讲解部署 ipvs 模式的 kube-proxy 过程。\n\n注意：如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**，然后远程分发文件和执行命令。\n\n## 下载和分发 kube-proxy 二进制文件\n\n参考 [05-1.部署master节点.md](05-1.部署master节点.md)。\n\n## 创建 kube-proxy 证书\n\n创建证书签名请求：\n\n``` bash\ncd /opt/k8s/work\ncat > kube-proxy-csr.json <<EOF\n{\n  \"CN\": \"system:kube-proxy\",\n  \"key\": {\n    \"algo\": \"rsa\",\n    \"size\": 2048\n  },\n  \"names\": [\n    {\n      \"C\": \"CN\",\n      \"ST\": \"BeiJing\",\n      \"L\": \"BeiJing\",\n      \"O\": \"k8s\",\n      \"OU\": \"opsnull\"\n    }\n  ]\n}\nEOF\n```\n+ CN：指定该证书的 User 为 `system:kube-proxy`；\n+ 预定义的 RoleBinding `system:node-proxier` 将User `system:kube-proxy` 与 Role `system:node-proxier` 绑定，该 Role 授予了调用 `kube-apiserver` Proxy 相关 API 的权限；\n+ 该证书只会被 kube-proxy 当做 client 证书使用，所以 hosts 字段为空；\n\n生成证书和私钥：\n\n``` bash\ncd /opt/k8s/work\ncfssl gencert -ca=/opt/k8s/work/ca.pem \\\n  -ca-key=/opt/k8s/work/ca-key.pem \\\n  -config=/opt/k8s/work/ca-config.json \\\n  -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy\nls kube-proxy*\n```\n\n## 创建和分发 kubeconfig 文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nkubectl config set-cluster kubernetes \\\n  --certificate-authority=/opt/k8s/work/ca.pem \\\n  --embed-certs=true \\\n  --server=${KUBE_APISERVER} \\\n  --kubeconfig=kube-proxy.kubeconfig\n\nkubectl config set-credentials kube-proxy \\\n  --client-certificate=kube-proxy.pem \\\n  --client-key=kube-proxy-key.pem \\\n  --embed-certs=true \\\n  --kubeconfig=kube-proxy.kubeconfig\n\nkubectl config set-context default \\\n  --cluster=kubernetes \\\n  --user=kube-proxy \\\n  --kubeconfig=kube-proxy.kubeconfig\n\nkubectl config use-context default --kubeconfig=kube-proxy.kubeconfig\n```\n\n分发 kubeconfig 文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_name in ${NODE_NAMES[@]}\n  do\n    echo \">>> ${node_name}\"\n    scp kube-proxy.kubeconfig root@${node_name}:/etc/kubernetes/\n  done\n```\n\n## 创建 kube-proxy 配置文件\n\n从 v1.10 开始，kube-proxy **部分参数**可以配置文件中配置。可以使用 `--write-config-to` 选项生成该配置文件，或者参考 [源代码的注释](https://github.com/kubernetes/kubernetes/blob/release-1.14/pkg/proxy/apis/config/types.go)。\n\n创建 kube-proxy config 文件模板：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > kube-proxy-config.yaml.template <<EOF\nkind: KubeProxyConfiguration\napiVersion: kubeproxy.config.k8s.io/v1alpha1\nclientConnection:\n  burst: 200\n  kubeconfig: \"/etc/kubernetes/kube-proxy.kubeconfig\"\n  qps: 100\nbindAddress: ##NODE_IP##\nhealthzBindAddress: ##NODE_IP##:10256\nmetricsBindAddress: ##NODE_IP##:10249\nenableProfiling: true\nclusterCIDR: ${CLUSTER_CIDR}\nhostnameOverride: ##NODE_NAME##\nmode: \"ipvs\"\nportRange: \"\"\niptables:\n  masqueradeAll: false\nipvs:\n  scheduler: rr\n  excludeCIDRs: []\nEOF\n```\n+ `bindAddress`: 监听地址；\n+ `clientConnection.kubeconfig`: 连接 apiserver 的 kubeconfig 文件；\n+ `clusterCIDR`: kube-proxy 根据 `--cluster-cidr` 判断集群内部和外部流量，指定 `--cluster-cidr` 或 `--masquerade-all` 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT；\n+ `hostnameOverride`: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；\n+ `mode`: 使用 ipvs 模式；\n\n为各节点创建和分发 kube-proxy 配置文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor (( i=0; i < 3; i++ ))\n  do \n    echo \">>> ${NODE_NAMES[i]}\"\n    sed -e \"s/##NODE_NAME##/${NODE_NAMES[i]}/\" -e \"s/##NODE_IP##/${NODE_IPS[i]}/\" kube-proxy-config.yaml.template > kube-proxy-config-${NODE_NAMES[i]}.yaml.template\n    scp kube-proxy-config-${NODE_NAMES[i]}.yaml.template root@${NODE_NAMES[i]}:/etc/kubernetes/kube-proxy-config.yaml\n  done\n```\n\n## 创建和分发 kube-proxy systemd unit 文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > kube-proxy.service <<EOF\n[Unit]\nDescription=Kubernetes Kube-Proxy Server\nDocumentation=https://github.com/GoogleCloudPlatform/kubernetes\nAfter=network.target\n\n[Service]\nWorkingDirectory=${K8S_DIR}/kube-proxy\nExecStart=/opt/k8s/bin/kube-proxy \\\\\n  --config=/etc/kubernetes/kube-proxy-config.yaml \\\\\n  --logtostderr=true \\\\\n  --v=2\nRestart=on-failure\nRestartSec=5\nLimitNOFILE=65536\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\n分发 kube-proxy systemd unit 文件：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_name in ${NODE_NAMES[@]}\n  do \n    echo \">>> ${node_name}\"\n    scp kube-proxy.service root@${node_name}:/etc/systemd/system/\n  done\n```\n\n## 启动 kube-proxy 服务\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p ${K8S_DIR}/kube-proxy\"\n    ssh root@${node_ip} \"modprobe ip_vs_rr\"\n    ssh root@${node_ip} \"systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy\"\n  done\n```\n\n## 检查启动结果\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl status kube-proxy|grep Active\"\n  done\n```\n\n确保状态为 `active (running)`，否则查看日志，确认原因：\n\n``` bash\njournalctl -u kube-proxy\n```\n\n## 查看监听端口\n\n``` bash\n$ sudo netstat -lnpt|grep kube-prox\ntcp        0      0 172.27.138.251:10256    0.0.0.0:*               LISTEN      30590/kube-proxy\ntcp        0      0 172.27.138.251:10249    0.0.0.0:*               LISTEN      30590/kube-proxy\n```\n+ 10249：http prometheus metrics port;\n+ 10256：http healthz port;\n\n## 查看 ipvs 路由规则\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"/usr/sbin/ipvsadm -ln\"\n  done\n```\n\n预期输出：\n\n``` bash\n>>> 172.27.138.251\nIP Virtual Server version 1.2.1 (size=4096)\nProt LocalAddress:Port Scheduler Flags\n  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn\nTCP  10.254.0.1:443 rr\n  -> 172.27.137.229:6443          Masq    1      0          0         \n  -> 172.27.138.239:6443          Masq    1      0          0         \n  -> 172.27.138.251:6443          Masq    1      0          0         \n>>> 172.27.137.229\nIP Virtual Server version 1.2.1 (size=4096)\nProt LocalAddress:Port Scheduler Flags\n  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn\nTCP  10.254.0.1:443 rr\n  -> 172.27.137.229:6443          Masq    1      0          0         \n  -> 172.27.138.239:6443          Masq    1      0          0         \n  -> 172.27.138.251:6443          Masq    1      0          0         \n>>> 172.27.138.239\nIP Virtual Server version 1.2.1 (size=4096)\nProt LocalAddress:Port Scheduler Flags\n  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn\nTCP  10.254.0.1:443 rr\n  -> 172.27.137.229:6443          Masq    1      0          0         \n  -> 172.27.138.239:6443          Masq    1      0          0         \n  -> 172.27.138.251:6443          Masq    1      0          0 \n```\n\n可见所有通过 https 访问 K8S SVC kubernetes 的请求都转发到 kube-apiserver 节点的 6443 端口；\n"
  },
  {
    "path": "06-6.calico.md",
    "content": "tags: worker, calico\n\n# 06-6. 部署 calico 网络\n\nkubernetes 要求集群内各节点(包括 master 节点)能通过 Pod 网段互联互通。\n\ncalico 使用 IPIP 或 BGP 技术（默认为 IPIP）为各节点创建一个可以互通的 Pod 网络。\n\n如果使用 flannel，请参考附件 [E.部署flannel网络.md](E.部署flannel网络.md)（flannel 与 docker 结合使用）\n\n注意：如果没有特殊指明，本文档的所有操作均在 zhangjun-k8s01 节点上执行。\n\n## 安装 calico 网络插件\n\n``` bash\ncd /opt/k8s/work\ncurl https://docs.projectcalico.org/manifests/calico.yaml -O\n```\n\n修改配置：\n\n``` bash\n$ cp calico.yaml calico.yaml.orig\nzhangjun@zj-pc16:~$ diff -U 5 calico.yaml.orig calico.yaml\n--- calico.yaml.orig    2021-06-02 21:04:31.000000000 +0800\n+++ calico.yaml 2021-06-02 21:08:14.000000000 +0800\n@@ -3678,12 +3678,14 @@\n                   name: calico-config\n                   key: veth_mtu\n             # The default IPv4 pool to create on startup if none exists. Pod IPs will be\n             # chosen from this range. Changing this value after installation will have\n             # no effect. This should fall within `--cluster-cidr`.\n-            # - name: CALICO_IPV4POOL_CIDR\n-            #   value: \"192.168.0.0/16\"\n+            - name: CALICO_IPV4POOL_CIDR\n+              value: \"172.30.0.0/16\"\n+            - name: IP_AUTODETECTION_METHOD\n+              value: \"interface=eth.*\"\n             # Disable file logging so `kubectl logs` works.\n             - name: CALICO_DISABLE_FILE_LOGGING\n               value: \"true\"\n             # Set Felix endpoint to host default action to ACCEPT.\n             - name: FELIX_DEFAULTENDPOINTTOHOSTACTION\n@@ -3759,11 +3761,11 @@\n             path: /sys/fs/\n             type: DirectoryOrCreate\n         # Used to install CNI.\n         - name: cni-bin-dir\n           hostPath:\n-            path: /opt/cni/bin\n+            path: /opt/k8s/bin\n         - name: cni-net-dir\n           hostPath:\n             path: /etc/cni/net.d\n         # Used to access CNI logs.\n         - name: cni-log-dir\n```\n+ 将 Pod 网段地址修改为 `172.30.0.0/16`;\n+ calico 自动探查互联网卡，如果有多快网卡，则可以配置用于互联的网络接口命名正则表达式，如上面的 `eth.*`(根据自己服务器的网络接口名修改)；\n\n运行 calico 插件：\n\n``` bash\n$ kubectl apply -f  calico.yaml\n```\n+ calico 插架以 daemonset 方式运行在所有的 K8S 节点上。\n\n## 查看 calico 运行状态\n\n``` bash\n$ kubectl get pods -n kube-system -o wide\nNAME                                      READY   STATUS    RESTARTS   AGE     IP               NODE              NOMINATED NODE   READINESS GATES\ncalico-kube-controllers-77c4b7448-99lfq   1/1     Running   0          2m11s   172.30.184.128   zhangjun-k8s-03   <none>           <none>\ncalico-node-dxnjs                         1/1     Running   0          2m11s   172.27.137.229   zhangjun-k8s-02   <none>           <none>\ncalico-node-rknzz                         1/1     Running   0          2m11s   172.27.138.239   zhangjun-k8s-03   <none>           <none>\ncalico-node-rw84c                         1/1     Running   0          2m11s   172.27.138.251   zhangjun-k8s-01   <none>           <none>\n```\n\n使用 crictl 命令查看 calico 使用的镜像：\n\n``` bash\n$ crictl  images\nIMAGE                                                     TAG                 IMAGE ID            SIZE\ndocker.io/calico/cni                                      v3.12.0             cb6799752c46c       66.5MB\ndocker.io/calico/node                                     v3.12.0             fc05bc4225f39       89.7MB\ndocker.io/calico/pod2daemon-flexvol                       v3.12.0             98793d0a88c82       37.5MB\nregistry.cn-beijing.aliyuncs.com/images_k8s/pause-amd64   3.1                 21a595adc69ca       326kB\n```\n+ 如果 crictl 输出为空或执行失败，则有可能是缺少配置文件 `/etc/crictl.yaml` 导致的，该文件的配置如下：\n\n    ``` yaml\n    $ cat /etc/crictl.yaml\n    runtime-endpoint: unix:///run/containerd/containerd.sock\n    image-endpoint: unix:///run/containerd/containerd.sock\n    timeout: 10\n    debug: false\n    ```\n"
  },
  {
    "path": "07.验证集群功能.md",
    "content": "tags: verify\n\n# 07. 验证集群功能\n\n<!-- TOC -->\n\n- [07. 验证集群功能](#07-验证集群功能)\n    - [检查节点状态](#检查节点状态)\n    - [创建测试文件](#创建测试文件)\n    - [执行测试](#执行测试)\n    - [检查各节点的 Pod IP 连通性](#检查各节点的-pod-ip-连通性)\n    - [检查服务 IP 和端口可达性](#检查服务-ip-和端口可达性)\n    - [检查服务的 NodePort 可达性](#检查服务的-nodeport-可达性)\n\n<!-- /TOC -->\n\n本文档验证 K8S 集群是否工作正常。\n\n注意：如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**，然后远程分发文件和执行命令。\n\n## 检查节点状态\n\n``` bash\n$ kubectl get nodes\nNAME              STATUS   ROLES    AGE   VERSION\nzhangjun-k8s-01   Ready    <none>   15m   v1.16.6\nzhangjun-k8s-02   Ready    <none>   15m   v1.16.6\nzhangjun-k8s-03   Ready    <none>   15m   v1.16.6\n```\n\n都为 Ready 且版本为 v1.16.6 时正常。\n\n## 创建测试文件\n\n``` bash\ncd /opt/k8s/work\ncat > nginx-ds.yml <<EOF\napiVersion: v1\nkind: Service\nmetadata:\n  name: nginx-ds\n  labels:\n    app: nginx-ds\nspec:\n  type: NodePort\n  selector:\n    app: nginx-ds\n  ports:\n  - name: http\n    port: 80\n    targetPort: 80\n---\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n  name: nginx-ds\n  labels:\n    addonmanager.kubernetes.io/mode: Reconcile\nspec:\n  selector:\n    matchLabels:\n      app: nginx-ds\n  template:\n    metadata:\n      labels:\n        app: nginx-ds\n    spec:\n      containers:\n      - name: my-nginx\n        image: nginx:1.7.9\n        ports:\n        - containerPort: 80\nEOF\n```\n\n## 执行测试\n\n``` bash\nkubectl create -f nginx-ds.yml\n```\n\n## 检查各节点的 Pod IP 连通性\n\n``` bash\n$ kubectl get pods  -o wide -l app=nginx-ds\nNAME             READY   STATUS    RESTARTS   AGE   IP               NODE              NOMINATED NODE   READINESS GATES\nnginx-ds-j7v5g   1/1     Running   0          61s   172.30.244.1     zhangjun-k8s-01   <none>           <none>\nnginx-ds-js8g8   1/1     Running   0          61s   172.30.82.129    zhangjun-k8s-02   <none>           <none>\nnginx-ds-n2p4x   1/1     Running   0          61s   172.30.184.130   zhangjun-k8s-03   <none>           <none>\n```\n\n在所有 Node 上分别 ping 上面三个 Pod IP，看是否连通：\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh ${node_ip} \"ping -c 1 172.30.244.1\"\n    ssh ${node_ip} \"ping -c 1 172.30.82.129\"\n    ssh ${node_ip} \"ping -c 1 172.30.184.130\"\n  done\n```\n\n## 检查服务 IP 和端口可达性\n\n``` bash\n$ kubectl get svc -l app=nginx-ds                                                                                                                    \nNAME       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE\nnginx-ds   NodePort   10.254.116.22   <none>        80:30562/TCP   2m7s\n```\n\n可见：\n\n+ Service Cluster IP：10.254.116.22\n+ 服务端口：80\n+ NodePort 端口：30562\n\n在所有 Node 上 curl Service IP：\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh ${node_ip} \"curl -s 10.254.116.22\"\n  done\n```\n\n预期输出 nginx 欢迎页面内容。\n\n## 检查服务的 NodePort 可达性\n\n在所有 Node 上执行：\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh ${node_ip} \"curl -s ${node_ip}:30562\"\n  done\n```\n\n预期输出 nginx 欢迎页面内容。"
  },
  {
    "path": "08-1.部署集群插件.md",
    "content": "tags: addons\n\n# 08-1. 部署集群插件\n\n插件是集群的附件组件，丰富和完善了集群的功能。\n\n+ [8-2.coredns 插件](./08-2.coredns插件.md)\n+ [8-3.Dashboard](./08-3.dashboard插件.md)\n+ [8-4.Kube-Prometheus](./08-4.kube-prometheus插件.md)\n+ [8-5.EFK (elasticsearch、fluentd、kibana)](./08-5.EFK插件.md)"
  },
  {
    "path": "08-2.coredns插件.md",
    "content": "tags: addons, dns, coredns\n\n# 08-2. 部署 coredns 插件\n\n<!-- TOC -->\n\n- [08-2. 部署 coredns 插件](#08-2-部署-coredns-插件)\n    - [下载和配置 coredns](#下载和配置-coredns)\n    - [创建 coredns](#创建-coredns)\n    - [检查 coredns 功能](#检查-coredns-功能)\n    - [参考](#参考)\n\n<!-- /TOC -->\n\n如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**；\n\n## 下载和配置 coredns\n\n``` bash\ncd /opt/k8s/work\ngit clone https://github.com/coredns/deployment.git\nmv deployment coredns-deployment\n```\n\n\n## 创建 coredns\n\n``` bash\ncd /opt/k8s/work/coredns-deployment/kubernetes\nsource /opt/k8s/bin/environment.sh\n./deploy.sh -i ${CLUSTER_DNS_SVC_IP} -d ${CLUSTER_DNS_DOMAIN} | kubectl apply -f -\n```\n\n## 检查 coredns 功能\n\n``` bash\n$ kubectl get all -n kube-system -l k8s-app=kube-dns\nNAME                          READY   STATUS    RESTARTS   AGE\npod/coredns-76b74f549-cwm8d   1/1     Running   0          62s\n\nNAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE\nservice/kube-dns   ClusterIP   10.254.0.2   <none>        53/UDP,53/TCP,9153/TCP   62s\n\nNAME                      READY   UP-TO-DATE   AVAILABLE   AGE\ndeployment.apps/coredns   1/1     1            1           62s\n\nNAME                                DESIRED   CURRENT   READY   AGE\nreplicaset.apps/coredns-76b74f549   1         1         1       62s\n```\n\n新建一个 Deployment：\n\n``` bash\ncd /opt/k8s/work\ncat > my-nginx.yaml <<EOF\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: my-nginx\nspec:\n  replicas: 2\n  selector:\n    matchLabels:\n      run: my-nginx\n  template:\n    metadata:\n      labels:\n        run: my-nginx\n    spec:\n      containers:\n      - name: my-nginx\n        image: nginx:1.7.9\n        ports:\n        - containerPort: 80\nEOF\nkubectl create -f my-nginx.yaml\n```\n\nexpose 该 Deployment, 生成 `my-nginx` 服务：\n\n``` bash\n$ kubectl expose deploy my-nginx\nservice \"my-nginx\" exposed\n\n$ kubectl get services my-nginx -o wide\nNAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR\nmy-nginx   ClusterIP   10.254.67.218   <none>        80/TCP    5s    run=my-nginx\n```\n\n创建另一个 Pod，查看 `/etc/resolv.conf` 是否包含 `kubelet` 配置的 `--cluster-dns` 和 `--cluster-domain`，是否能够将服务 `my-nginx` 解析到上面显示的 Cluster IP `10.254.40.167`\n\n``` bash\ncd /opt/k8s/work\ncat > dnsutils-ds.yml <<EOF\napiVersion: v1\nkind: Service\nmetadata:\n  name: dnsutils-ds\n  labels:\n    app: dnsutils-ds\nspec:\n  type: NodePort\n  selector:\n    app: dnsutils-ds\n  ports:\n  - name: http\n    port: 80\n    targetPort: 80\n---\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n  name: dnsutils-ds\n  labels:\n    addonmanager.kubernetes.io/mode: Reconcile\nspec:\n  selector:\n    matchLabels:\n      app: dnsutils-ds\n  template:\n    metadata:\n      labels:\n        app: dnsutils-ds\n    spec:\n      containers:\n      - name: my-dnsutils\n        image: tutum/dnsutils:latest\n        command:\n          - sleep\n          - \"3600\"\n        ports:\n        - containerPort: 80\nEOF\nkubectl create -f dnsutils-ds.yml\n```\n\n``` bash\n$ kubectl get pods -lapp=dnsutils-ds -o wide \nNAME                READY   STATUS    RESTARTS   AGE   IP               NODE              NOMINATED NODE   READINESS GATES\ndnsutils-ds-7h9np   1/1     Running   0          69s   172.30.244.3     zhangjun-k8s-01   <none>           <none>\ndnsutils-ds-fthdl   1/1     Running   0          69s   172.30.82.131    zhangjun-k8s-02   <none>           <none>\ndnsutils-ds-w69zp   1/1     Running   0          69s   172.30.184.132   zhangjun-k8s-03   <none>           <none>\n```\n\n``` bash\n$ kubectl -it exec dnsutils-ds-7h9np  cat /etc/resolv.conf\nsearch default.svc.cluster.local svc.cluster.local cluster.local 4pd.io\nnameserver 10.254.0.2\noptions ndots:5\n```\n\n``` bash\n$ kubectl -it exec dnsutils-ds-7h9np nslookup kubernetes\nServer:         10.254.0.2\nAddress:        10.254.0.2#53\n\nName:   kubernetes.default.svc.cluster.local\nAddress: 10.254.0.1\n```\n\n``` bash\n$ kubectl -it exec dnsutils-ds-7h9np nslookup www.baidu.com\nServer:         10.254.0.2\nAddress:        10.254.0.2#53\n\nNon-authoritative answer:\n*** Can't find www.baidu.com: No answer\n```\n\n``` bash\n$ kubectl -it exec dnsutils-ds-7h9np nslookup www.baidu.com.\nServer:         10.254.0.2\nAddress:        10.254.0.2#53\n\nNon-authoritative answer:\nwww.baidu.com   canonical name = www.a.shifen.com.\nName:   www.a.shifen.com\nAddress: 220.181.38.150\nName:   www.a.shifen.com\nAddress: 220.181.38.149\n```\n\n``` bash\n$ kubectl -it exec dnsutils-ds-7h9np nslookup my-nginx\nServer:         10.254.0.2\nAddress:        10.254.0.2#53\n\nName:   my-nginx.default.svc.cluster.local\nAddress: 10.254.67.218\n```\n\n## 参考\n\n1. https://community.infoblox.com/t5/Community-Blog/CoreDNS-for-Kubernetes-Service-Discovery/ba-p/8187\n2. https://coredns.io/2017/03/01/coredns-for-kubernetes-service-discovery-take-2/\n3. https://www.cnblogs.com/boshen-hzb/p/7511432.html\n4. https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns"
  },
  {
    "path": "08-3.dashboard插件.md",
    "content": "tags: addons, dashboard\n\n# 08-3. 部署 dashboard 插件\n\n<!-- TOC -->\n\n- [08-3. 部署 dashboard 插件](#08-3-部署-dashboard-插件)\n    - [下载和修改配置文件](#下载和修改配置文件)\n    - [执行所有定义文件](#执行所有定义文件)\n    - [查看运行状态](#查看运行状态)\n    - [访问 dashboard](#访问-dashboard)\n        - [通过 port forward 访问 dashboard](#通过-port-forward-访问-dashboard)\n    - [创建登录 Dashboard 的 token 和 kubeconfig 配置文件](#创建登录-dashboard-的-token-和-kubeconfig-配置文件)\n        - [创建登录 token](#创建登录-token)\n        - [创建使用 token 的 KubeConfig 文件](#创建使用-token-的-kubeconfig-文件)\n    - [参考](#参考)\n\n<!-- /TOC -->\n\n如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**；\n\n## 下载和修改配置文件\n\n``` bash\ncd /opt/k8s/work\nwget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc4/aio/deploy/recommended.yaml\nmv  recommended.yaml dashboard-recommended.yaml\n```\n\n## 执行所有定义文件\n\n``` bash\ncd /opt/k8s/work\nkubectl apply -f  dashboard-recommended.yaml\n```\n\n## 查看运行状态\n\n``` bash\n$ kubectl get pods -n kubernetes-dashboard \nNAME                                         READY   STATUS    RESTARTS   AGE\ndashboard-metrics-scraper-7b8b58dc8b-dlk5t   1/1     Running   0          70s\nkubernetes-dashboard-6cfc8c4c9-j8vcm         1/1     Running   0          70s\n```\n\n## 访问 dashboard\n\n从 1.7 开始，dashboard 只允许通过 https 访问，如果使用 kube proxy 则必须监听 localhost 或 127.0.0.1。对于 NodePort 没有这个限制，但是仅建议在开发环境中使用。对于不满足这些条件的登录访问，在登录成功后**浏览器不跳转，始终停在登录界面**。\n\n### 通过 port forward 访问 dashboard\n\n启动端口转发：\n\n``` bash\n[root@zhangjun-k8s-01 work] kubectl port-forward -n kubernetes-dashboard  svc/kubernetes-dashboard 4443:443 --address 0.0.0.0\n```\n\n浏览器访问 URL：`https://172.27.138.251:4443`\n\n![dashboard-login](./images/dashboard-login.png)\n\n## 创建登录 Dashboard 的 token 和 kubeconfig 配置文件\n\ndashboard 默认只支持 token 认证（不支持 client 证书认证），所以如果使用 Kubeconfig 文件，需要将 token 写入到该文件。\n\n### 创建登录 token\n\n``` bash\nkubectl create sa dashboard-admin -n kube-system\nkubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin\nADMIN_SECRET=$(kubectl get secrets -n kube-system | grep dashboard-admin | awk '{print $1}')\nDASHBOARD_LOGIN_TOKEN=$(kubectl describe secret -n kube-system ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}')\necho ${DASHBOARD_LOGIN_TOKEN}\n```\n\n使用输出的 token 登录 Dashboard。\n\n### 创建使用 token 的 KubeConfig 文件\n\n``` bash\nsource /opt/k8s/bin/environment.sh\n# 设置集群参数\nkubectl config set-cluster kubernetes \\\n  --certificate-authority=/etc/kubernetes/cert/ca.pem \\\n  --embed-certs=true \\\n  --server=${KUBE_APISERVER} \\\n  --kubeconfig=dashboard.kubeconfig\n\n# 设置客户端认证参数，使用上面创建的 Token\nkubectl config set-credentials dashboard_user \\\n  --token=${DASHBOARD_LOGIN_TOKEN} \\\n  --kubeconfig=dashboard.kubeconfig\n\n# 设置上下文参数\nkubectl config set-context default \\\n  --cluster=kubernetes \\\n  --user=dashboard_user \\\n  --kubeconfig=dashboard.kubeconfig\n\n# 设置默认上下文\nkubectl config use-context default --kubeconfig=dashboard.kubeconfig\n```\n\n用生成的 dashboard.kubeconfig 登录 Dashboard。\n\n![images/dashboard.png](images/dashboard.png)\n\n## 参考\n\n1. https://github.com/kubernetes/dashboard/wiki/Access-control\n2. https://github.com/kubernetes/dashboard/issues/2558\n3. https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/\n4. https://github.com/kubernetes/dashboard/wiki/Accessing-Dashboard---1.7.X-and-above\n5. https://github.com/kubernetes/dashboard/issues/2540\n"
  },
  {
    "path": "08-4.kube-prometheus插件.md",
    "content": "tags: addons, kube-prometheus, prometheus, grafana\n\n# 08-4. 部署 kube-prometheus 插架\n<!-- TOC -->\n\n- [08-4. 部署 kube-prometheus 插架](#08-4-部署-kube-prometheus-插架)\n    - [下载和安装](#下载和安装)\n    - [查看运行状态](#查看运行状态)\n    - [访问 Prometheus UI](#访问-prometheus-ui)\n    - [访问 Grafana UI](#访问-grafana-ui)\n\n<!-- /TOC -->\n\nkube-prometheus 是一整套监控解决方案，它使用 Prometheus 采集集群指标，Grafana 做展示，包含如下组件：\n+ The Prometheus Operator\n+ Highly available Prometheus\n+ Highly available Alertmanager\n+ Prometheus node-exporter\n+ Prometheus Adapter for Kubernetes Metrics APIs （k8s-prometheus-adapter）\n+ kube-state-metrics\n+ Grafana\n\n其中 k8s-prometheus-adapter 使用 Prometheus 实现了 metrics.k8s.io 和 custom.metrics.k8s.io API，所以**不需要再部署** `metrics-server`。\n如果要单独部署 `metrics-server`，请参考：[C.metrics-server插件.md](C.metrics-server插件.md)\n\n如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**；\n\n## 下载和安装\n\n``` bash\ncd /opt/k8s/work\ngit clone https://github.com/coreos/kube-prometheus.git\ncd kube-prometheus/\nsed -i -e 's_quay.io_quay.mirrors.ustc.edu.cn_' manifests/*.yaml manifests/setup/*.yaml # 使用中科大的 Registry\nkubectl apply -f manifests/setup # 安装 prometheus-operator\nkubectl apply -f manifests/ # 安装 promethes metric adapter\n```\n\n## 查看运行状态\n\n``` bash\n$ kubectl get pods -n monitoring\nNAME                                   READY   STATUS    RESTARTS   AGE\nalertmanager-main-0                    2/2     Running       0          63s\nalertmanager-main-1                    2/2     Running       0          63s\nalertmanager-main-2                    2/2     Running       0          63s\ngrafana-76b8d59b9b-nd6gk               1/1     Running       0          11m\nkube-state-metrics-67b7c5dc78-sktzg    3/3     Running       0          73s\nnode-exporter-prsvf                    2/2     Running       0          34s\nnode-exporter-qdh6n                    2/2     Running       0          71s\nnode-exporter-z6h4z                    2/2     Running       0          69s\nprometheus-adapter-5f46ccd66d-bbsns    1/1     Running       0          73s\nprometheus-k8s-0                       3/3     Running       1          53s\nprometheus-k8s-1                       3/3     Running       1          53s\nprometheus-operator-6d8b95b467-htx56   1/1     Running       0          74s\n```\n\n``` bash\n$ kubectl top pods -n monitoring\nNAME                                  CPU(cores)   MEMORY(bytes)   \nalertmanager-main-0                    0m           18Mi            \nalertmanager-main-1                    2m           20Mi            \nalertmanager-main-2                    0m           19Mi            \ngrafana-76b8d59b9b-nd6gk               4m           49Mi            \nkube-state-metrics-67b7c5dc78-sktzg    11m          29Mi            \nkube-state-metrics-959876458-cjtr5     9m           37Mi            \nnode-exporter-prsvf                    4m           11Mi            \nnode-exporter-qdh6n                    1m           20Mi            \nnode-exporter-z6h4z                    5m           11Mi            \nprometheus-adapter-5f46ccd66d-bbsns    0m           17Mi            \nprometheus-k8s-0                       15m          190Mi           \nprometheus-k8s-1                       6m           199Mi           \nprometheus-operator-6d8b95b467-htx56   0m           20Mi   \n```\n\n## 访问 Prometheus UI\n\n启动服务代理：\n\n``` bash\n$ \n$ kubectl port-forward --address 0.0.0.0 pod/prometheus-k8s-0 -n monitoring 9090:9090\nForwarding from 0.0.0.0:9090 -> 9090\n```\n+ port-forward 依赖 socat。\n\n浏览器访问：http://172.27.138.251:9090/new/graph?g0.expr=&g0.tab=1&g0.stacked=0&g0.range_input=1h\n\n![prometheus](images/prometheus.png)\n\n\n## 访问 Grafana UI\n\n启动代理：\n\n``` bash\n$ kubectl port-forward --address 0.0.0.0 svc/grafana -n monitoring 3000:3000 \nForwarding from 0.0.0.0:3000 -> 3000\n```\n\n浏览器访问：http://172.27.138.251:3000/\n\n用 admin/admin 登录：\n![grafana_login](images/grafana_login.png)\n\n然后，就可以看到各种预定义的 dashboard 了：\n![grafana_dashboard](images/grafana_dashboard.png)\n\n"
  },
  {
    "path": "08-5.EFK插件.md",
    "content": "tags: addons, EFK, fluentd, elasticsearch, kibana\n\n# 08-5. 部署 EFK 插件\n\n<!-- TOC -->\n\n- [08-5. 部署 EFK 插件](#08-5-部署-efk-插件)\n    - [修改配置文件](#修改配置文件)\n    - [执行定义文件](#执行定义文件)\n    - [检查执行结果](#检查执行结果)\n    - [通过 kubectl proxy 访问 kibana](#通过-kubectl-proxy-访问-kibana)\n\n<!-- /TOC -->\n\n注意：\n1. 如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s-01 节点上执行**。\n2. kuberntes 自带插件的 manifests yaml 文件使用 gcr.io 的 docker registry，国内被墙，需要**手动替换**为其它 registry 地址；\n3. 可以从微软中国提供的 [gcr.io 免费代理](http://mirror.azure.cn/help/gcr-proxy-cache.html)下载被墙的镜像；\n\n## 修改配置文件\n\n将下载的 kubernetes-server-linux-amd64.tar.gz 解压后，再解压其中的 kubernetes-src.tar.gz 文件。\n\n``` bash\ncd /opt/k8s/work/kubernetes/\ntar -xzvf kubernetes-src.tar.gz\n```\n\nEFK 目录是 `kubernetes/cluster/addons/fluentd-elasticsearch`。\n\n``` bash\ncd /opt/k8s/work/kubernetes/cluster/addons/fluentd-elasticsearch\nsed -i -e 's_quay.io_quay.mirrors.ustc.edu.cn_' es-statefulset.yaml # 使用中科大的 Registry\nsed -i -e 's_quay.io_quay.mirrors.ustc.edu.cn_' fluentd-es-ds.yaml # 使用中科大的 Registry\n```\n\n## 执行定义文件\n\n``` bash\ncd /opt/k8s/work/kubernetes/cluster/addons/fluentd-elasticsearch\nkubectl apply -f .\n```\n\n## 检查执行结果\n\n``` bash\n$ kubectl get all -n kube-system |grep -E 'elasticsearch|fluentd|kibana'\npod/elasticsearch-logging-0                   1/1     Running   0          15m\npod/elasticsearch-logging-1                   1/1     Running   0          14m\npod/fluentd-es-v2.7.0-98slb                   1/1     Running   0          15m\npod/fluentd-es-v2.7.0-v25tz                   1/1     Running   0          15m\npod/fluentd-es-v2.7.0-zngpm                   1/1     Running   0          15m\npod/kibana-logging-75888755d6-nw6bc           1/1     Running   0          5m40s\nservice/elasticsearch-logging   ClusterIP   10.254.11.19     <none>        9200/TCP                 15m\nservice/kibana-logging          ClusterIP   10.254.207.146   <none>        5601/TCP                 15m\ndaemonset.apps/fluentd-es-v2.7.0   3         3         3       3            3           <none>                   15m\ndeployment.apps/kibana-logging            1/1     1            1           15m\nreplicaset.apps/kibana-logging-75888755d6           1         1         1       15m\nstatefulset.apps/elasticsearch-logging   2/2     15m\n```\n\nkibana Pod 第一次启动时会用**较长时间(0-20分钟)**来优化和 Cache 状态页面，可以 tailf 该 Pod 的日志观察进度：\n\n``` bash\n$ kubectl logs kibana-logging-75888755d6-nw6bc -n kube-system -f\n```\n\n注意：只有当 Kibana pod 启动完成后，浏览器才能查看 kibana dashboard，否则会被拒绝。\n\n## 通过 kubectl proxy 访问 kibana\n\n创建代理：\n\n``` bash\n$ kubectl proxy --address='172.27.138.251' --port=8086 --accept-hosts='^*$'\nStarting to serve on 172.27.138.251:8086\n```\n\n浏览器访问 URL：`http://172.27.138.251:8086/api/v1/namespaces/kube-system/services/kibana-logging/proxy`\n    \n在 Management -> Indices 页面创建一个 index（相当于 mysql 中的一个 database），选中 `Index contains time-based events`，使用默认的 `logstash-*` pattern，点击 `Create` ;\n\n![es-setting](./images/es-setting.png)\n\n创建 Index 后，稍等几分钟就可以在 `Discover` 菜单下看到 ElasticSearch logging 中汇聚的日志；\n\n![es-home](./images/es-home.png)"
  },
  {
    "path": "09.Registry.md",
    "content": "tags: registry, ceph\n\n# 09. 部署 docker registry\n\n<!-- TOC -->\n\n- [09. 部署 docker registry](#09-部署-docker-registry)\n    - [部署 ceph RGW 节点](#部署-ceph-rgw-节点)\n    - [创建测试账号 demo](#创建测试账号-demo)\n    - [创建 demo 账号的子账号 swift](#创建-demo-账号的子账号-swift)\n    - [创建 demo:swift 子账号的 sercret key](#创建-demoswift-子账号的-sercret-key)\n    - [创建 docker registry](#创建-docker-registry)\n    - [向 registry push image](#向-registry-push-image)\n    - [私有 registry 的运维操作](#私有-registry-的运维操作)\n        - [查询私有镜像中的 images](#查询私有镜像中的-images)\n        - [查询某个镜像的 tags 列表](#查询某个镜像的-tags-列表)\n        - [获取 image 或 layer 的 digest](#获取-image-或-layer-的-digest)\n        - [删除 image](#删除-image)\n        - [删除 layer](#删除-layer)\n    - [常见问题](#常见问题)\n        - [login 失败 416](#login-失败-416)\n        - [login 失败 503](#login-失败-503)\n\n<!-- /TOC -->\n\n本文档介绍使用 docker 官方的 registry v2 镜像部署私有仓库的步骤，你也可以参考附件文档部署 Harbor 私有仓库（[D.部署 Harbor 私有仓库](D.部署Harbor-Registry.md)）。 \n\n本文档讲解部署一个 TLS 加密、HTTP Basic 认证、用 ceph rgw 做后端存储的私有 docker registry 步骤，如果使用其它类型的后端存储，则可以从 “创建 docker registry” 节开始；\n\n示例两台机器 IP 如下：\n\n+ ceph rgw: 172.27.132.66\n+ docker registry: 172.27.132.67\n\n## 部署 ceph RGW 节点\n\n``` bash\n$ ceph-deploy rgw create 172.27.132.66 # rgw 默认监听7480端口\n$\n```\n\n## 创建测试账号 demo\n\n``` bash\n$ radosgw-admin user create --uid=demo --display-name=\"ceph rgw demo user\"\n$\n```\n\n## 创建 demo 账号的子账号 swift\n\n当前 registry 只支持使用 swift 协议访问 ceph rgw 存储，暂时不支持 s3 协议；\n\n``` bash\n$ radosgw-admin subuser create --uid demo --subuser=demo:swift --access=full --secret=secretkey --key-type=swift\n$\n```\n\n## 创建 demo:swift 子账号的 sercret key\n\n``` bash\n$ radosgw-admin key create --subuser=demo:swift --key-type=swift --gen-secret\n{\n    \"user_id\": \"demo\",\n    \"display_name\": \"ceph rgw demo user\",\n    \"email\": \"\",\n    \"suspended\": 0,\n    \"max_buckets\": 1000,\n    \"auid\": 0,\n    \"subusers\": [\n        {\n            \"id\": \"demo:swift\",\n            \"permissions\": \"full-control\"\n        }\n    ],\n    \"keys\": [\n        {\n            \"user\": \"demo\",\n            \"access_key\": \"5Y1B1SIJ2YHKEHO5U36B\",\n            \"secret_key\": \"nrIvtPqUj7pUlccLYPuR3ntVzIa50DToIpe7xFjT\"\n        }\n    ],\n    \"swift_keys\": [\n        {\n            \"user\": \"demo:swift\",\n            \"secret_key\": \"ttQcU1O17DFQ4I9xzKqwgUe7WIYYX99zhcIfU9vb\"\n        }\n    ],\n    \"caps\": [],\n    \"op_mask\": \"read, write, delete\",\n    \"default_placement\": \"\",\n    \"placement_tags\": [],\n    \"bucket_quota\": {\n        \"enabled\": false,\n        \"max_size_kb\": -1,\n        \"max_objects\": -1\n    },\n    \"user_quota\": {\n        \"enabled\": false,\n        \"max_size_kb\": -1,\n        \"max_objects\": -1\n    },\n        \"temp_url_keys\": []\n}\n```\n\n+ `ttQcU1O17DFQ4I9xzKqwgUe7WIYYX99zhcIfU9vb` 为子账号 demo:swift 的 secret key；\n\n## 创建 docker registry\n\n创建 registry 使用的 x509 证书\n\n``` bash\n$ mkdir -p registry/{auth,certs}\n$ cat > registry-csr.json <<EOF\n{\n  \"CN\": \"registry\",\n  \"hosts\": [\n      \"127.0.0.1\",\n      \"172.27.132.67\"\n  ],\n  \"key\": {\n    \"algo\": \"rsa\",\n    \"size\": 2048\n  },\n  \"names\": [\n    {\n      \"C\": \"CN\",\n      \"ST\": \"BeiJing\",\n      \"L\": \"BeiJing\",\n      \"O\": \"k8s\",\n      \"OU\": \"opsnull\"\n    }\n  ]\n}\nEOF\n$ cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \\\n    -ca-key=/etc/kubernetes/cert/ca-key.pem \\\n    -config=/etc/kubernetes/cert/ca-config.json \\\n    -profile=kubernetes registry-csr.json | cfssljson -bare registry\n$ cp registry.pem registry-key.pem registry/certs\n$\n```\n+ 这里复用以前创建的 CA 证书和秘钥文件；\n+ hosts 字段指定 registry 的 NodeIP；\n\n创建 HTTP Baisc 认证文件\n\n``` bash\n$ docker run --entrypoint htpasswd registry:2 -Bbn foo foo123  > registry/auth/htpasswd\n$ cat  registry/auth/htpasswd\nfoo:$2y$05$iZaM45Jxlcg0DJKXZMggLOibAsHLGybyU.CgU9AHqWcVDyBjiScN.\n```\n\n配置 registry 参数\n\n``` bash\nexport RGW_AUTH_URL=\"http://172.27.132.66:7480/auth/v1\"\nexport RGW_USER=\"demo:swift\"\nexport RGW_SECRET_KEY=\"ttQcU1O17DFQ4I9xzKqwgUe7WIYYX99zhcIfU9vb\"\ncat > config.yml << EOF\n# https://docs.docker.com/registry/configuration/#list-of-configuration-options\nversion: 0.1\nlog:\n  level: info\n  fromatter: text\n  fields:\n    service: registry\n\nstorage:\n  cache:\n    blobdescriptor: inmemory\n  delete:\n    enabled: true\n  swift:\n    authurl: ${RGW_AUTH_URL}\n    username: ${RGW_USER}\n    password: ${RGW_SECRET_KEY}\n    container: registry\n\nauth:\n  htpasswd:\n    realm: basic-realm\n    path: /auth/htpasswd\n\nhttp:\n  addr: 0.0.0.0:8000\n  headers:\n    X-Content-Type-Options: [nosniff]\n  tls:\n    certificate: /certs/registry.pem\n    key: /certs/registry-key.pem\n\nhealth:\n  storagedriver:\n    enabled: true\n    interval: 10s\n    threshold: 3\nEOF\n[k8s@zhangjun-k8s-01 cert]$ cp config.yml registry\n[k8s@zhangjun-k8s-01 cert]$ scp -r registry 172.27.132.67:/opt/k8s\n```\n\n+ storage.swift 指定后端使用 swfit 接口协议的存储，这里配置的是 ceph rgw 存储参数；\n+ auth.htpasswd 指定了 HTTP Basic 认证的 token 文件路径；\n+ http.tls 指定了 registry http 服务器的证书和秘钥文件路径；\n\n创建 docker registry：\n\n``` bash\nssh k8s@172.27.132.67\n$ docker run -d -p 8000:8000 --privileged \\\n    -v /opt/k8s/registry/auth/:/auth \\\n    -v /opt/k8s/registry/certs:/certs \\\n    -v /opt/k8s/registry/config.yml:/etc/docker/registry/config.yml \\\n    --name registry registry:2\n```\n\n+ 执行该 docker run 命令的机器 IP 为 172.27.132.67；\n\n## 向 registry push image\n\n将签署 registry 证书的 CA 证书拷贝到 `/etc/docker/certs.d/172.27.132.67:8000` 目录下\n\n``` bash\n[k8s@zhangjun-k8s-01 cert]$ sudo mkdir -p /etc/docker/certs.d/172.27.132.67:8000\n[k8s@zhangjun-k8s-01 cert]$ sudo cp /etc/kubernetes/cert/ca.pem /etc/docker/certs.d/172.27.132.67:8000/ca.crt\n```\n\n登陆私有 registry：\n\n``` bash\n$ docker login 172.27.132.67:8000\nUsername: foo\nPassword:\nLogin Succeeded\n```\n\n登陆信息被写入 `~/.docker/config.json` 文件：\n\n``` bash\n$ cat ~/.docker/config.json\n{\n        \"auths\": {\n                \"172.27.132.67:8000\": {\n                        \"auth\": \"Zm9vOmZvbzEyMw==\"\n                }\n        }\n}\n```\n\n将本地的 image 打上私有 registry 的 tag：\n\n``` bash\n$ docker tag prom/node-exporter:v0.16.0 172.27.132.67:8000/prom/node-exporter:v0.16.0\n$ docker images |grep pause\nprom/node-exporter:v0.16.0                            latest              f9d5de079539        2 years ago         239.8 kB\n172.27.132.67:8000/prom/node-exporter:v0.16.0                        latest              f9d5de079539        2 years ago         239.8 kB\n```\n\n将 image push 到私有 registry：\n\n``` bash\n$ docker push 172.27.132.67:8000/prom/node-exporter:v0.16.0\nThe push refers to a repository [172.27.132.67:8000/prom/node-exporter:v0.16.0]\n5f70bf18a086: Pushed\ne16a89738269: Pushed\nlatest: digest: sha256:9a6b437e896acad3f5a2a8084625fdd4177b2e7124ee943af642259f2f283359 size: 916\n```\n\n查看 ceph 上是否已经有 push 的 pause 容器文件：\n\n``` bash\n$ rados lspools\nrbd\ncephfs_data\ncephfs_metadata\n.rgw.root\nk8s\ndefault.rgw.control\ndefault.rgw.meta\ndefault.rgw.log\ndefault.rgw.buckets.index\ndefault.rgw.buckets.data\n\n$  rados --pool default.rgw.buckets.data ls|grep node-exporter\n1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/cdb7590af5f064887f3d6008d46be65e929c74250d747813d85199e04fc70463/link\n1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_manifests/revisions/sha256/55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086/link\n1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_manifests/tags/v0.16.0/current/link\n1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_manifests/tags/v0.16.0/index/sha256/55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086/link\n1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c/link\n1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/528dda9cf23d0fad80347749d6d06229b9a19903e49b7177d5f4f58736538d4e/link\n1f3f02c4-fe58-4626-992b-c6c0fe4c8acf.34107.1_files/docker/registry/v2/repositories/prom/node-exporter/_layers/sha256/188af75e2de0203eac7c6e982feff45f9c340eaac4c7a0f59129712524fa2984/link\n```\n\n## 私有 registry 的运维操作\n\n### 查询私有镜像中的 images\n\n``` bash\n$ curl  --user foo:foo123 --cacert /etc/docker/certs.d/172.27.132.67\\:8000/ca.crt https://172.27.132.67:8000/v2/_catalog\n{\"repositories\":[\"prom/node-exporter\"]}\n```\n\n### 查询某个镜像的 tags 列表\n\n``` bash\n$  curl  --user foo:foo123 --cacert /etc/docker/certs.d/172.27.132.67\\:8000/ca.crt https://172.27.132.67:8000/v2/prom/node-exporter/tags/list\n{\"name\":\"prom/node-exporter\",\"tags\":[\"v0.16.0\"]}\n```\n\n### 获取 image 或 layer 的 digest\n\n向 `v2/<repoName>/manifests/<tagName>` 发 GET 请求，从响应的头部 `Docker-Content-Digest` 获取 image digest，从响应的 body 的 `fsLayers.blobSum` 中获取 layDigests;\n\n注意，必须包含请求头：`Accept: application/vnd.docker.distribution.manifest.v2+json`：\n\n``` bash\n$ curl -v -H \"Accept: application/vnd.docker.distribution.manifest.v2+json\" --user foo:foo123 --cacert /etc/docker/certs.d/172.27.132.67\\:8000/ca.crt https://172.27.132.67:8000/v2/prom/node-exporter/manifests/v0.16.0\n* About to connect() to 172.27.132.67 port 8000 (#0)\n*   Trying 172.27.132.67...\n* Connected to 172.27.132.67 (172.27.132.67) port 8000 (#0)\n* Initializing NSS with certpath: sql:/etc/pki/nssdb\n*   CAfile: /etc/docker/certs.d/172.27.132.67:8000/ca.crt\n  CApath: none\n* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\n* Server certificate:\n*       subject: CN=registry,OU=4Paradigm,O=k8s,L=BeiJing,ST=BeiJing,C=CN\n*       start date: Jul 05 12:52:00 2018 GMT\n*       expire date: Jul 02 12:52:00 2028 GMT\n*       common name: registry\n*       issuer: CN=kubernetes,OU=4Paradigm,O=k8s,L=BeiJing,ST=BeiJing,C=CN\n* Server auth using Basic with user 'foo'\n> GET /v2/prom/node-exporter/manifests/v0.16.0 HTTP/1.1\n> Authorization: Basic Zm9vOmZvbzEyMw==\n> User-Agent: curl/7.29.0\n> Host: 172.27.132.67:8000\n> Accept: application/vnd.docker.distribution.manifest.v2+json\n>\n< HTTP/1.1 200 OK\n< Content-Length: 949\n< Content-Type: application/vnd.docker.distribution.manifest.v2+json\n< Docker-Content-Digest: sha256:55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086\n< Docker-Distribution-Api-Version: registry/2.0\n< Etag: \"sha256:55302581333c43d540db0e144cf9e7735423117a733cdec27716d87254221086\"\n< X-Content-Type-Options: nosniff\n< Date: Fri, 06 Jul 2018 06:18:41 GMT\n<\n{\n   \"schemaVersion\": 2,\n   \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n   \"config\": {\n      \"mediaType\": \"application/vnd.docker.container.image.v1+json\",\n      \"size\": 3511,\n      \"digest\": \"sha256:188af75e2de0203eac7c6e982feff45f9c340eaac4c7a0f59129712524fa2984\"\n   },\n   \"layers\": [\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 2392417,\n         \"digest\": \"sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 560703,\n         \"digest\": \"sha256:cdb7590af5f064887f3d6008d46be65e929c74250d747813d85199e04fc70463\"\n      },\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 5332460,\n         \"digest\": \"sha256:528dda9cf23d0fad80347749d6d06229b9a19903e49b7177d5f4f58736538d4e\"\n      }\n   ]\n```\n\n### 删除 image\n\n向 `/v2/<name>/manifests/<reference>` 发送 DELETE 请求，reference 为上一步返回的 Docker-Content-Digest 字段内容：\n\n``` bash\n$ curl -X DELETE  --user foo:foo123 --cacert /etc/docker/certs.d/172.27.132.67\\:8000/ca.crt https://172.27.132.67:8000/v2/prom/node-exporter/manifests/sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5\n$\n```\n\n### 删除 layer\n\n向 `/v2/<name>/blobs/<digest>`发送 DELETE 请求，其中 digest 是上一步返回的 `fsLayers.blobSum` 字段内容：\n\n``` bash\n$ curl -X DELETE  --user foo:foo123 --cacert /etc/docker/certs.d/172.27.132.67\\:8000/ca.crt https://172.27.132.67:8000/v2/prom/node-exporter/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\n$ curl -X DELETE  --cacert /etc/docker/certs.d/172.27.132.67\\:8000/ca.crt https://172.27.132.67:8000/v2/prom/node-exporter/blobs/sha256:04176c8b224aa0eb9942af765f66dae866f436e75acef028fe44b8a98e045515\n$\n```\n\n## 常见问题\n\n### login 失败 416\n\n执行 http://docs.ceph.com/docs/master/install/install-ceph-gateway/ 里面的 s3 test.py 程序失败：\n\n[k8s@zhangjun-k8s-01 cert]$ python s3test.py\nTraceback (most recent call last):\n  File \"s3test.py\", line 12, in <module>\n    bucket = conn.create_bucket('my-new-bucket')\n  File \"/usr/lib/python2.7/site-packages/boto/s3/connection.py\", line 625, in create_bucket\n    response.status, response.reason, body)\nboto.exception.S3ResponseError: S3ResponseError: 416 Requested Range Not Satisfiable\n\n解决版办法：\n1. 在管理节点上修改 ceph.conf\n1. ceph-deploy config push zhangjun-k8s-01 zhangjun-k8s-02 zhangjun-k8s-03\n1.  systemctl restart 'ceph-mds@zhangjun-k8s-03.service'\n    systemctl restart ceph-osd@0\n    systemctl restart 'ceph-mon@zhangjun-k8s-01.service'\n    systemctl restart 'ceph-mgr@zhangjun-k8s-01.service'\n\nFor anyone who is hitting this issue\nset default pg_num and pgp_num to lower value(8 for example), or set mon_max_pg_per_osd to a high value in ceph.conf\nradosgw-admin doesn' throw proper error when internal pool creation fails, hence the upper level error which is very confusing.\n\nhttps://tracker.ceph.com/issues/21497\n\n### login 失败 503\n\n[root@zhangjun-k8s-01 ~]# docker login  172.27.132.67:8000\nUsername: foo\nPassword:\nError response from daemon: login attempt to https://172.27.132.67:8000/v2/ failed with status: 503 Service Unavailable\n\n原因： docker run 缺少 --privileged 参数；"
  },
  {
    "path": "10.清理集群.md",
    "content": "tags: cleanup\n\n# 10. 清理集群\n\n<!-- TOC -->\n\n- [10. 清理集群](#10-清理集群)\n    - [清理 Node 节点](#清理-node-节点)\n    - [清理 Master 节点](#清理-master-节点)\n    - [清理 etcd 集群](#清理-etcd-集群)\n\n<!-- /TOC -->\n\n## 清理 Node 节点\n\n停相关进程：\n\n``` bash\n$ sudo systemctl stop kubelet kube-proxy kube-nginx\n$ sudo systemctl disable kubelet kube-proxy kube-nginx\n```\n\n停容器进程：\n\n``` bash\n$ crictl ps -q | xargs crictl stop\n$ killall -9 containerd-shim-runc-v1 pause\n```\n\n停 containerd 服务：\n\n``` bash\n$ systemctl stop containerd && systemctl disable containerd\n```\n\n清理文件：\n\n``` bash\n$ source /opt/k8s/bin/environment.sh\n$ # umount k8s 挂载的目录\n$ mount |grep -E 'kubelet|cni|containerd' | awk '{print $3}'|xargs umount\n$ # 删除 kubelet 目录\n$ sudo rm -rf ${K8S_DIR}/kubelet\n$ # 删除 docker 目录\n$ sudo rm -rf ${DOCKER_DIR}\n$ # 删除 containerd 目录\n$ sudo rm -rf ${CONTAINERD_DIR}\n$ # 删除 systemd unit 文件\n$ sudo rm -rf /etc/systemd/system/{kubelet,kube-proxy,containerd,kube-nginx}.service\n$ # 删除程序文件\n$ sudo rm -rf /opt/k8s/bin/*\n$ # 删除证书文件\n$ sudo rm -rf /etc/flanneld/cert /etc/kubernetes/cert\n$\n```\n\n清理 kube-proxy 和 calico 创建的 iptables：\n\n``` bash\n$ sudo iptables -F && sudo iptables -X && sudo iptables -F -t nat && sudo iptables -X -t nat\n$\n```\n\n## 清理 Master 节点\n\n停相关进程：\n\n``` bash\n$ sudo systemctl stop kube-apiserver kube-controller-manager kube-scheduler\n$\n```\n\n清理文件：\n\n``` bash\n$ # 删除 systemd unit 文件\n$ sudo rm -rf /etc/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service\n$ # 删除程序文件\n$ sudo rm -rf /opt/k8s/bin/{kube-apiserver,kube-controller-manager,kube-scheduler}\n$ # 删除证书文件\n$ sudo rm -rf /etc/flanneld/cert /etc/kubernetes/cert\n$\n```\n\n## 清理 etcd 集群\n\n停相关进程：\n\n``` bash\n$ sudo systemctl stop etcd\n$\n```\n\n清理文件：\n\n``` bash\n$ source /opt/k8s/bin/environment.sh\n$ # 删除 etcd 的工作目录和数据目录\n$ sudo rm -rf ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}\n$ # 删除 systemd unit 文件\n$ sudo rm -rf /etc/systemd/system/etcd.service\n$ # 删除程序文件\n$ sudo rm -rf /opt/k8s/bin/etcd\n$ # 删除 x509 证书文件\n$ sudo rm -rf /etc/etcd/cert/*\n$\n```"
  },
  {
    "path": "A.浏览器访问kube-apiserver安全端口.md",
    "content": "# A. 浏览器访问 kube-apiserver 安全端口\n\n浏览器访问 kube-apiserver 的安全端口 6443 时，提示证书不被信任：\n\n![ssl-failed](images/ssl-failed.png)\n\n这是因为 kube-apiserver 的 server 证书是我们创建的根证书 ca.pem 签名的，需要将根证书 ca.pem 导入操作系统，并设置永久信任。\n\n对于 Mac，操作如下：\n\n![keychain](images/keychain.png)\n\n对于 windows 系统使用以下命令导入 ca.perm：\n\n``` bash\nkeytool -import -v -trustcacerts -alias appmanagement -file \"PATH...\\\\ca.pem\" -storepass password -keystore cacerts\n```\n\n再次访问 [apiserver 地址](https://172.27.138.251:6443/)，已信任，但提示 401，未授权的访问：\n\n![ssl-success](images/ssl-success.png)\n\n我们需要给浏览器生成一个 client 证书，访问 apiserver 的 6443 https 端口时使用。\n\n这里使用部署 kubectl 命令行工具时创建的 admin 证书、私钥和上面的 ca 证书，创建一个浏览器可以使用 PKCS#12/PFX 格式的证书：\n\n``` bash\n$ openssl pkcs12 -export -out admin.pfx -inkey admin-key.pem -in admin.pem -certfile ca.pem\n```\n\n将创建的 admin.pfx 导入到系统的证书中。对于 Mac，操作如下：\n\n![admin-cert](images/admin-cert.png)\n\n**重启浏览器**，再次访问 [apiserver 地址](https://172.27.138.251:6443/)，提示选择一个浏览器证书，这里选中上面导入的 admin.pfx：\n\n![select-cert](images/select-cert.png)\n\n这一次，被授权访问 kube-apiserver 的安全端口：\n\n![chrome-authored](images/chrome-authored.png)\n\n## 客户端选择证书的原理\n\n1. 证书选择是在客户端和服务端 SSL/TLS 握手协商阶段商定的；\n1. 服务端如果要求客户端提供证书，则在握手时会向客户端发送一个它接受的 CA 列表；\n1. 客户端查找它的证书列表(一般是操作系统的证书，对于 Mac 为 keychain)，看有没有被 CA 签名的证书，如果有，则将它们提供给用户选择（证书的私钥）；\n1. 用户选择一个证书私钥，然后客户端将使用它和服务端通信；\n\n\n## 参考\n+ https://github.com/kubernetes/kubernetes/issues/31665\n+ https://www.sslshopper.com/ssl-converter.html\n+ https://stackoverflow.com/questions/40847638/how-chrome-browser-know-which-client-certificate-to-prompt-for-a-site\n"
  },
  {
    "path": "B.校验TLS证书.md",
    "content": "# B. 校验 TLS 证书\n\n以校验 kubernetes 证书(后续部署 master 节点时生成的)为例：\n\n## 使用 openssl 命令\n\n``` bash\n$ openssl x509  -noout -text -in  kubernetes.pem\n...\n    Signature Algorithm: sha256WithRSAEncryption\n        Issuer: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=Kubernetes\n        Validity\n            Not Before: Apr  5 05:36:00 2017 GMT\n            Not After : Apr  5 05:36:00 2018 GMT\n        Subject: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=kubernetes\n...\n        X509v3 extensions:\n            X509v3 Key Usage: critical\n                Digital Signature, Key Encipherment\n            X509v3 Extended Key Usage:\n                TLS Web Server Authentication, TLS Web Client Authentication\n            X509v3 Basic Constraints: critical\n                CA:FALSE\n            X509v3 Subject Key Identifier:\n                DD:52:04:43:10:13:A9:29:24:17:3A:0E:D7:14:DB:36:F8:6C:E0:E0\n            X509v3 Authority Key Identifier:\n                keyid:44:04:3B:60:BD:69:78:14:68:AF:A0:41:13:F6:17:07:13:63:58:CD\n\n            X509v3 Subject Alternative Name:\n                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:10.64.3.7, IP Address:10.254.0.1\n...\n```\n\n+ 确认 Issuer 字段的内容和 ca-csr.json 一致；\n+ 确认 Subject 字段的内容和 kubernetes-csr.json 一致；\n+ 确认 X509v3 Subject Alternative Name 字段的内容和 kubernetes-csr.json 一致；\n+ 确认 X509v3 Key Usage、Extended Key Usage 字段的内容和 ca-config.json 中 kubernetes profile 一致；\n\n## 使用 cfssl-certinfo 命令\n\n``` bash\n$ cfssl-certinfo -cert kubernetes.pem\n...\n{\n  \"subject\": {\n    \"common_name\": \"kubernetes\",\n    \"country\": \"CN\",\n    \"organization\": \"k8s\",\n    \"organizational_unit\": \"System\",\n    \"locality\": \"BeiJing\",\n    \"province\": \"BeiJing\",\n    \"names\": [\n      \"CN\",\n      \"BeiJing\",\n      \"BeiJing\",\n      \"k8s\",\n      \"System\",\n      \"kubernetes\"\n    ]\n  },\n  \"issuer\": {\n    \"common_name\": \"Kubernetes\",\n    \"country\": \"CN\",\n    \"organization\": \"k8s\",\n    \"organizational_unit\": \"System\",\n    \"locality\": \"BeiJing\",\n    \"province\": \"BeiJing\",\n    \"names\": [\n      \"CN\",\n      \"BeiJing\",\n      \"BeiJing\",\n      \"k8s\",\n      \"System\",\n      \"Kubernetes\"\n    ]\n  },\n  \"serial_number\": \"174360492872423263473151971632292895707129022309\",\n  \"sans\": [\n    \"kubernetes\",\n    \"kubernetes.default\",\n    \"kubernetes.default.svc\",\n    \"kubernetes.default.svc.cluster\",\n    \"kubernetes.default.svc.cluster.local\",\n    \"127.0.0.1\",\n    \"10.64.3.7\",\n    \"10.64.3.8\",\n    \"10.66.3.86\",\n    \"10.254.0.1\"\n  ],\n  \"not_before\": \"2017-04-05T05:36:00Z\",\n  \"not_after\": \"2018-04-05T05:36:00Z\",\n  \"sigalg\": \"SHA256WithRSA\",\n...\n```\n\n## 参考\n\n+ [Generate self-signed certificates](https://coreos.com/os/docs/latest/generate-self-signed-certificates.html)\n+ [Setting up a Certificate Authority and Creating TLS Certificates](https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/02-certificate-authority.md)\n+ [Client Certificates V/s Server Certificates](https://blogs.msdn.microsoft.com/kaushal/2012/02/17/client-certificates-vs-server-certificates/)"
  },
  {
    "path": "C.metrics-server插件.md",
    "content": "tags: addons, metrics, metrics-server\n\n# C. 部署 metrics-server 插件\n<!-- TOC -->\n\n- [C. 部署 metrics-server 插件](#c-部署-metrics-server-插件)\n    - [监控架构](#监控架构)\n    - [安装 metrics-server](#安装-metrics-server)\n    - [查看运行情况](#查看运行情况)\n    - [查看 metrics-server 输出的 metrics](#查看-metrics-server-输出的-metrics)\n    - [使用 kubectl top 命令查看集群节点资源使用情况](#使用-kubectl-top-命令查看集群节点资源使用情况)\n    - [参考](#参考)\n\n<!-- /TOC -->\n\nmetrics-server 通过 kube-apiserver 发现所有节点，然后调用 kubelet APIs（通过 https 接口）获得各节点（Node）和 Pod 的 CPU、Memory 等资源使用情况。\n\n从 Kubernetes 1.12 开始，kubernetes 的安装脚本移除了 Heapster，从 1.13 开始完全移除了对 Heapster 的支持，Heapster 不再被维护。\n\n替代方案如下：\n\n1. 用于支持自动扩缩容的 CPU/memory HPA metrics：metrics-server；\n2. 通用的监控方案：使用第三方可以获取 Prometheus 格式监控指标的监控系统，如 Prometheus Operator；\n3. 事件传输：使用第三方工具来传输、归档 kubernetes events；\n\n\n## 监控架构\n\n![monitoring_architecture.png](images/monitoring_architecture.png)\n\n没有安装 metrics-server 或 heapster 时，kubeclt top 命令将不能使用：\n\n``` bash\n$ kubectl top  node\nError from server (NotFound): the server could not find the requested resource (get services http:heapster:)\n```\n\n## 安装 metrics-server\n\n从 github clone 源码：\n\n``` \n$ cd /opt/k8s/work/\n$ git clone https://github.com/kubernetes-incubator/metrics-server.git\n$ cd metrics-server/deploy/1.8+/\n```\n\n修改 `metrics-server-deployment.yaml` 文件，为 metrics-server 添加三个命令行参数：\n\n``` bash\n$ cp metrics-server-deployment.yaml metrics-server-deployment.yaml.orig\n$ diff metrics-server-deployment.yaml.orig metrics-server-deployment.yaml\n32c32\n<         image: k8s.gcr.io/metrics-server-amd64:v0.3.6\n---\n>         image: gcr.azk8s.cn/google_containers/metrics-server-amd64:v0.3.6\n35a36,37\n>           - --metric-resolution=30s\n>           - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP\n```\n+ 使用微软的 grc 镜像；\n+ --metric-resolution=30s：从 kubelet 采集数据的周期；\n+ --kubelet-preferred-address-types：优先使用 InternalIP 来访问 kubelet，这样可以避免节点名称**没有 DNS 解析**记录时，通过节点名称调用节点 kubelet API 失败的情况（未配置时默认的情况）；\n\n部署 metrics-server：\n\n``` bash\n$ cd /opt/k8s/work/metrics-server/deploy/1.8+/\n$ kubectl create -f .\n```\n\n## 查看运行情况\n\n``` bash\n$ kubectl -n kube-system get all -l k8s-app=metrics-server\nNAME                                  READY   STATUS    RESTARTS   AGE\npod/metrics-server-77df59848f-sjjbd   1/1     Running   0          18s\n\nNAME                             READY   UP-TO-DATE   AVAILABLE   AGE\ndeployment.apps/metrics-server   1/1     1            1           19s\n\nNAME                                        DESIRED   CURRENT   READY   AGE\nreplicaset.apps/metrics-server-77df59848f   1         1         1       19s\n```\n\n## 查看 metrics-server 输出的 metrics\n\n```\nkubectl get --raw https://172.27.138.251:6443/apis/metrics.k8s.io/v1beta1/nodes | jq .\nkubectl get --raw https://172.27.138.251:6443/apis/metrics.k8s.io/v1beta1/pods | jq .\nkubectl get --raw https://172.27.138.251:6443/apis/metrics.k8s.io/v1beta1/nodes/<node-name> | jq .\nkubectl get --raw https://172.27.138.251:6443/apis/metrics.k8s.io/v1beta1/namespace/<namespace-name>/pods/<pod-name> | jq .\n```\n+ 替换 <xxx> 为实际内容；\n+ /apis/metrics.k8s.io/v1beta1/nodes 和 /apis/metrics.k8s.io/v1beta1/pods 返回的 usage 包含 CPU 和 Memory；\n\n## 使用 kubectl top 命令查看集群节点资源使用情况\n\nkubectl top 命令从 metrics-server 获取集群节点基本的指标信息：\n\n``` bash\nNAME              CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   \nzhangjun-k8s-01   177m         2%     9267Mi          58%       \nzhangjun-k8s-02   364m         4%     10338Mi         65%       \nzhangjun-k8s-03   185m         2%     5950Mi          37%   \n```\n\n## 参考\n\n1. https://kubernetes.feisky.xyz/zh/addons/metrics.html\n2. metrics-server RBAC：https://github.com/kubernetes-incubator/metrics-server/issues/40\n3. metrics-server 参数：https://github.com/kubernetes-incubator/metrics-server/issues/25\n4. https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/\n5. metrics-server 的 [APIs 文档](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/resource-metrics-api.md)。"
  },
  {
    "path": "D.部署Harbor-Registry.md",
    "content": "tags: registry, harbor\n\n# D. 部署 harbor 私有仓库\n\n<!-- TOC -->\n\n- [D. 部署 harbor 私有仓库](#d-部署-harbor-私有仓库)\n    - [使用的变量](#使用的变量)\n    - [下载文件](#下载文件)\n    - [导入 docker images](#导入-docker-images)\n    - [创建 harbor nginx 服务器使用的 x509 证书](#创建-harbor-nginx-服务器使用的-x509-证书)\n    - [修改 harbor.cfg 文件](#修改-harborcfg-文件)\n    - [加载和启动 harbor 镜像](#加载和启动-harbor-镜像)\n    - [访问管理界面](#访问管理界面)\n    - [harbor 运行时产生的文件、目录](#harbor-运行时产生的文件目录)\n    - [docker 客户端登陆](#docker-客户端登陆)\n    - [其它操作](#其它操作)\n\n<!-- /TOC -->\n\n本文档介绍使用 docker-compose 部署 harbor 私有仓库的步骤，你也可以使用 docker 官方的 registry 镜像部署私有仓库([部署 Docker Registry](11-部署Docker-Registry.md))。\n\n## 使用的变量\n\n本文档用到的变量定义如下：\n\n``` bash\n$ export NODE_IP=10.64.3.7 # 当前部署 harbor 的节点 IP\n$\n```\n\n## 下载文件\n\n从 docker compose [发布页面](https://github.com/docker/compose/releases)下载最新的 `docker-compose` 二进制文件\n\n``` bash\n$ wget https://github.com/docker/compose/releases/download/1.21.2/docker-compose-Linux-x86_64\n$ mv ~/docker-compose-Linux-x86_64 /opt/k8s/bin/docker-compose\n$ chmod a+x  /opt/k8s/bin/docker-compose\n$ export PATH=/opt/k8s/bin:$PATH\n$\n```\n\n从 harbor [发布页面](https://github.com/vmware/harbor/releases)下载最新的 harbor 离线安装包\n\n``` bash\n$ wget  --continue https://storage.googleapis.com/harbor-releases/release-1.5.0/harbor-offline-installer-v1.5.1.tgz\n$ tar -xzvf harbor-offline-installer-v1.5.1.tgz\n$\n```\n\n## 导入 docker images\n\n导入离线安装包中 harbor 相关的 docker images：\n\n``` bash\n$ cd harbor\n$ docker load -i harbor.v1.5.1.tar.gz\n$\n```\n\n## 创建 harbor nginx 服务器使用的 x509 证书\n\n创建 harbor 证书签名请求：\n\n``` bash\n$ cat > harbor-csr.json <<EOF\n{\n  \"CN\": \"harbor\",\n  \"hosts\": [\n    \"127.0.0.1\",\n    \"${NODE_IP}\"\n  ],\n  \"key\": {\n    \"algo\": \"rsa\",\n    \"size\": 2048\n  },\n  \"names\": [\n    {\n      \"C\": \"CN\",\n      \"ST\": \"BeiJing\",\n      \"L\": \"BeiJing\",\n      \"O\": \"k8s\",\n      \"OU\": \"opsnull\"\n    }\n  ]\n}\nEOF\n```\n\n+ hosts 字段指定授权使用该证书的当前部署节点 IP，如果后续使用域名访问 harbor 则还需要添加域名；\n\n生成 harbor 证书和私钥：\n\n``` bash\n$ cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \\\n  -ca-key=/etc/kubernetes/cert/ca-key.pem \\\n  -config=/etc/kubernetes/cert/ca-config.json \\\n  -profile=kubernetes harbor-csr.json | cfssljson -bare harbor\n\n$ ls harbor*\nharbor.csr  harbor-csr.json  harbor-key.pem harbor.pem\n\n$ sudo mkdir -p /etc/harbor/ssl\n$ sudo mv harbor*.pem /etc/harbor/ssl\n$ rm harbor.csr  harbor-csr.json\n```\n\n## 修改 harbor.cfg 文件\n\n``` bash\n$ cp harbor.cfg{,.bak}\n$ vim harbor.cfg\n$ diff harbor.cfg{,.bak}\n7c7\n< hostname = 172.27.129.81\n---\n> hostname = reg.mydomain.com\n11c11\n< ui_url_protocol = https\n---\n> ui_url_protocol = http\n23,24c23,24\n< ssl_cert =  /etc/harbor/ssl/harbor.pem\n< ssl_cert_key = /etc/harbor/ssl/harbor-key.pem\n---\n> ssl_cert = /data/cert/server.crt\n> ssl_cert_key = /data/cert/server.key\n\n$ cp prepare{,.bak}\n$ vim prepare\n$ diff prepare{,.bak}\n453a454\n>         print(\"%s %w\", args, kw)\n490c491\n<     empty_subj = \"/\"\n---\n>     empty_subj = \"/C=/ST=/L=/O=/CN=/\"\n\n```\n+ 需要修改 prepare 脚本的 empyt_subj 参数，否则后续 install 时出错退出：\n  \n    Fail to generate key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt\n\n参考：https://github.com/vmware/harbor/issues/2920\n\n## 加载和启动 harbor 镜像\n\n``` bash\n$ sudo mkdir /data\n$ sudo chmod 777 /var/run/docker.sock /data\n$ sudo apt-get install python\n$ ./install.sh\n\n[Step 0]: checking installation environment ...\n\nNote: docker version: 18.03.0\n\nNote: docker-compose version: 1.21.2\n\n[Step 1]: loading Harbor images ...\nLoaded image: vmware/clair-photon:v2.0.1-v1.5.1\nLoaded image: vmware/postgresql-photon:v1.5.1\nLoaded image: vmware/harbor-adminserver:v1.5.1\nLoaded image: vmware/registry-photon:v2.6.2-v1.5.1\nLoaded image: vmware/photon:1.0\nLoaded image: vmware/harbor-migrator:v1.5.1\nLoaded image: vmware/harbor-ui:v1.5.1\nLoaded image: vmware/redis-photon:v1.5.1\nLoaded image: vmware/nginx-photon:v1.5.1\nLoaded image: vmware/mariadb-photon:v1.5.1\nLoaded image: vmware/notary-signer-photon:v0.5.1-v1.5.1\nLoaded image: vmware/harbor-log:v1.5.1\nLoaded image: vmware/harbor-db:v1.5.1\nLoaded image: vmware/harbor-jobservice:v1.5.1\nLoaded image: vmware/notary-server-photon:v0.5.1-v1.5.1\n\n\n[Step 2]: preparing environment ...\nloaded secret from file: /data/secretkey\nGenerated configuration file: ./common/config/nginx/nginx.conf\nGenerated configuration file: ./common/config/adminserver/env\nGenerated configuration file: ./common/config/ui/env\nGenerated configuration file: ./common/config/registry/config.yml\nGenerated configuration file: ./common/config/db/env\nGenerated configuration file: ./common/config/jobservice/env\nGenerated configuration file: ./common/config/jobservice/config.yml\nGenerated configuration file: ./common/config/log/logrotate.conf\nGenerated configuration file: ./common/config/jobservice/config.yml\nGenerated configuration file: ./common/config/ui/app.conf\nGenerated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt\nThe configuration files are ready, please use docker-compose to start the service.\n\n\n[Step 3]: checking existing instance of Harbor ...\n\n\n[Step 4]: starting Harbor ...\nCreating network \"harbor_harbor\" with the default driver\nCreating harbor-log ... done\nCreating redis              ... done\nCreating harbor-adminserver ... done\nCreating harbor-db          ... done\nCreating registry           ... done\nCreating harbor-ui          ... done\nCreating harbor-jobservice  ... done\nCreating nginx              ... done\n\n✔ ----Harbor has been installed and started successfully.----\n\nNow you should be able to visit the admin portal at https://172.27.129.81.\nFor more details, please visit https://github.com/vmware/harbor .\n```\n\n## 访问管理界面\n\n确认所有组件都工作正常：\n\n``` bash\n$ docker-compose  ps\n       Name                     Command                  State                                    Ports\n-------------------------------------------------------------------------------------------------------------------------------------\nharbor-adminserver   /harbor/start.sh                 Up (healthy)\nharbor-db            /usr/local/bin/docker-entr ...   Up (healthy)   3306/tcp\nharbor-jobservice    /harbor/start.sh                 Up\nharbor-log           /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp\nharbor-ui            /harbor/start.sh                 Up (healthy)\nnginx                nginx -g daemon off;             Up (healthy)   0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp, 0.0.0.0:80->80/tcp\nredis                docker-entrypoint.sh redis ...   Up             6379/tcp\nregistry             /entrypoint.sh serve /etc/ ...   Up (healthy)   5000/tcp\n```\n\n浏览器访问 `https://${NODE_IP}`，示例的是 `https://172.27.129.81`；\n\n由于是在 virtualbox 虚机 zhangjun-k8s-02 中运行，所以需要做下端口转发，Vagrant 文件中已经指定 host 端口为 4443，也可以在 virtualbox 的 GUI 中直接添加端口转发：\n\n![virtualbox-harbor](./images/virtualbox-harbor.png)\n\n浏览器访问 `https://127.0.0.1:443`，用账号 `admin` 和 harbor.cfg 配置文件中的默认密码 `Harbor12345` 登陆系统。\n\n![harbor](./images/harbo.png)\n\n## harbor 运行时产生的文件、目录\n\nharbor 将日志打印到 /var/log/harbor 的相关目录下，使用 docker logs XXX 或 docker-compose logs XXX 将看不到容器的日志。\n\n``` bash\n$ # 日志目录\n$ ls /var/log/harbor\nadminserver.log  jobservice.log  mysql.log  proxy.log  registry.log  ui.log\n$ # 数据目录，包括数据库、镜像仓库\n$ ls /data/\nca_download  config  database  job_logs registry  secretkey\n```\n\n## docker 客户端登陆\n\n将签署 harbor 证书的 CA 证书拷贝到 `/etc/docker/certs.d/172.27.129.81` 目录下\n\n``` bash\n$ sudo mkdir -p /etc/docker/certs.d/172.27.129.81\n$ sudo cp /etc/kubernetes/cert/ca.pem /etc/docker/certs.d/172.27.129.81/ca.crt\n$\n```\n\n登陆 harbor\n\n``` bash\n$ docker login 172.27.129.81\nUsername: admin\nPassword:\n```\n\n认证信息自动保存到 `~/.docker/config.json` 文件。\n\n## 其它操作\n\n下列操作的工作目录均为 解压离线安装文件后 生成的 harbor 目录。\n\n``` bash\n$ # 停止 harbor\n$ docker-compose down -v\n$ # 修改配置\n$ vim harbor.cfg\n$ # 更修改的配置更新到 docker-compose.yml 文件\n$ ./prepare\nClearing the configuration file: ./common/config/ui/app.conf\nClearing the configuration file: ./common/config/ui/env\nClearing the configuration file: ./common/config/ui/private_key.pem\nClearing the configuration file: ./common/config/db/env\nClearing the configuration file: ./common/config/registry/root.crt\nClearing the configuration file: ./common/config/registry/config.yml\nClearing the configuration file: ./common/config/jobservice/app.conf\nClearing the configuration file: ./common/config/jobservice/env\nClearing the configuration file: ./common/config/nginx/cert/admin.pem\nClearing the configuration file: ./common/config/nginx/cert/admin-key.pem\nClearing the configuration file: ./common/config/nginx/nginx.conf\nClearing the configuration file: ./common/config/adminserver/env\nloaded secret from file: /data/secretkey\nGenerated configuration file: ./common/config/nginx/nginx.conf\nGenerated configuration file: ./common/config/adminserver/env\nGenerated configuration file: ./common/config/ui/env\nGenerated configuration file: ./common/config/registry/config.yml\nGenerated configuration file: ./common/config/db/env\nGenerated configuration file: ./common/config/jobservice/env\nGenerated configuration file: ./common/config/jobservice/app.conf\nGenerated configuration file: ./common/config/ui/app.conf\nGenerated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt\nThe configuration files are ready, please use docker-compose to start the service.\n$ sudo chmod -R 666 common ## 防止容器进程没有权限读取生成的配置\n$ # 启动 harbor\n$ docker-compose up -d\n```"
  },
  {
    "path": "E.部署flannel网络.md",
    "content": "tags: flanneld\n\n# E. 部署 flannel 网络\n\n<!-- TOC -->\n\n- [E. 部署 flannel 网络](#e-部署-flannel-网络)\n    - [下载和分发 flanneld 二进制文件](#下载和分发-flanneld-二进制文件)\n    - [创建 flannel 证书和私钥](#创建-flannel-证书和私钥)\n    - [向 etcd 写入集群 Pod 网段信息](#向-etcd-写入集群-pod-网段信息)\n    - [创建 flanneld 的 systemd unit 文件](#创建-flanneld-的-systemd-unit-文件)\n    - [分发 flanneld systemd unit 文件到**所有节点**](#分发-flanneld-systemd-unit-文件到所有节点)\n    - [启动 flanneld 服务](#启动-flanneld-服务)\n    - [检查启动结果](#检查启动结果)\n    - [检查分配给各 flanneld 的 Pod 网段信息](#检查分配给各-flanneld-的-pod-网段信息)\n    - [检查节点 flannel 网络信息](#检查节点-flannel-网络信息)\n    - [验证各节点能通过 Pod 网段互通](#验证各节点能通过-pod-网段互通)\n\n<!-- /TOC -->\n\nkubernetes 要求集群内各节点(包括 master 节点)能通过 Pod 网段互联互通。flannel 使用 vxlan 技术为各节点创建一个可以互通的 Pod 网络，使用的端口为 UDP 8472（**需要开放该端口**，如公有云 AWS 等）。\n\nflanneld 第一次启动时，从 etcd 获取配置的 Pod 网段信息，为本节点分配一个未使用的地址段，然后创建 `flannedl.1` 网络接口（也可能是其它名称，如 flannel1 等）。\n\nflannel 将分配给自己的 Pod 网段信息写入 `/run/flannel/docker` 文件，docker 后续使用这个文件中的环境变量设置 `docker0` 网桥，从而从这个地址段为本节点的所有 Pod 容器分配 IP。\n\n注意：\n1. 如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s01 节点上执行**，然后远程分发文件和执行命令；\n2. flanneld 与本文档部署的 etcd v3.4.x 不兼容，需要将 etcd 降级到 v3.3.x；\n3. flanneld 与 docker 结合使用；\n    \n\n## 下载和分发 flanneld 二进制文件\n\n从 flannel 的 [release 页面](https://github.com/coreos/flannel/releases) 下载最新版本的安装包：\n\n``` bash\ncd /opt/k8s/work\nmkdir flannel\nwget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz\ntar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel\n```\n\n分发二进制文件到集群所有节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp flannel/{flanneld,mk-docker-opts.sh} root@${node_ip}:/opt/k8s/bin/\n    ssh root@${node_ip} \"chmod +x /opt/k8s/bin/*\"\n  done\n```\n\n## 创建 flannel 证书和私钥\n\nflanneld 从 etcd 集群存取网段分配信息，而 etcd 集群启用了双向 x509 证书认证，所以需要为 flanneld 生成证书和私钥。\n\n创建证书签名请求：\n\n``` bash\ncd /opt/k8s/work\ncat > flanneld-csr.json <<EOF\n{\n  \"CN\": \"flanneld\",\n  \"hosts\": [],\n  \"key\": {\n    \"algo\": \"rsa\",\n    \"size\": 2048\n  },\n  \"names\": [\n    {\n      \"C\": \"CN\",\n      \"ST\": \"BeiJing\",\n      \"L\": \"BeiJing\",\n      \"O\": \"k8s\",\n      \"OU\": \"4Paradigm\"\n    }\n  ]\n}\nEOF\n```\n+ 该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空；\n\n生成证书和私钥：\n\n``` bash\ncfssl gencert -ca=/opt/k8s/work/ca.pem \\\n  -ca-key=/opt/k8s/work/ca-key.pem \\\n  -config=/opt/k8s/work/ca-config.json \\\n  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld\nls flanneld*pem\n```\n\n将生成的证书和私钥分发到**所有节点**（master 和 worker）：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p /etc/flanneld/cert\"\n    scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert\n  done\n```\n\n## 向 etcd 写入集群 Pod 网段信息\n\n注意：本步骤**只需执行一次**。\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\netcdctl \\\n  --endpoints=${ETCD_ENDPOINTS} \\\n  --ca-file=/opt/k8s/work/ca.pem \\\n  --cert-file=/opt/k8s/work/flanneld.pem \\\n  --key-file=/opt/k8s/work/flanneld-key.pem \\\n  mk ${FLANNEL_ETCD_PREFIX}/config '{\"Network\":\"'${CLUSTER_CIDR}'\", \"SubnetLen\": 21, \"Backend\": {\"Type\": \"vxlan\"}}'\n```\n+ flanneld **当前版本 (v0.11.0) 不支持 etcd v3**，故使用 etcd v2 API 写入配置 key 和网段数据；\n+ 写入的 Pod 网段 `${CLUSTER_CIDR}` 地址段（如 /16）必须小于 `SubnetLen`，必须与 `kube-controller-manager` 的 `--cluster-cidr` 参数值一致；\n\n## 创建 flanneld 的 systemd unit 文件\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > flanneld.service << EOF\n[Unit]\nDescription=Flanneld overlay address etcd agent\nAfter=network.target\nAfter=network-online.target\nWants=network-online.target\nAfter=etcd.service\nBefore=docker.service\n\n[Service]\nType=notify\nExecStart=/opt/k8s/bin/flanneld \\\\\n  -etcd-cafile=/etc/kubernetes/cert/ca.pem \\\\\n  -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\\\\n  -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\\\\n  -etcd-endpoints=${ETCD_ENDPOINTS} \\\\\n  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\\\\n  -iface=${IFACE} \\\\\n  -ip-masq\nExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker\nRestart=always\nRestartSec=5\nStartLimitInterval=0\n\n[Install]\nWantedBy=multi-user.target\nRequiredBy=docker.service\nEOF\n```\n+ `mk-docker-opts.sh` 脚本将分配给 flanneld 的 Pod 子网段信息写入 `/run/flannel/docker` 文件，后续 docker 启动时使用这个文件中的环境变量配置 docker0 网桥；\n+ flanneld 使用系统缺省路由所在的接口与其它节点通信，对于有多个网络接口（如内网和公网）的节点，可以用 `-iface` 参数指定通信接口;\n+ flanneld 运行时需要 root 权限；\n+ `-ip-masq`: flanneld 为访问 Pod 网络外的流量设置 SNAT 规则，同时将传递给 Docker 的变量 `--ip-masq`（`/run/flannel/docker` 文件中）设置为 false，这样 Docker 将不再创建 SNAT 规则；\n  Docker 的 `--ip-masq` 为 true 时，创建的 SNAT 规则比较“暴力”：将所有本节点 Pod 发起的、访问非 docker0 接口的请求做 SNAT，这样访问其他节点 Pod 的请求来源 IP 会被设置为 flannel.1 接口的 IP，导致目的 Pod 看不到真实的来源 Pod IP。\n  flanneld 创建的 SNAT 规则比较温和，只对访问非 Pod 网段的请求做 SNAT。\n\n## 分发 flanneld systemd unit 文件到**所有节点**\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp flanneld.service root@${node_ip}:/etc/systemd/system/\n  done\n```\n\n## 启动 flanneld 服务\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld\"\n  done\n```\n\n## 检查启动结果\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl status flanneld|grep Active\"\n  done\n```\n\n确保状态为 `active (running)`，否则查看日志，确认原因：\n\n``` bash\njournalctl -u flanneld\n```\n\n## 检查分配给各 flanneld 的 Pod 网段信息\n\n查看集群 Pod 网段(/16)：\n\n``` bash\nsource /opt/k8s/bin/environment.sh\netcdctl \\\n  --endpoints=${ETCD_ENDPOINTS} \\\n  --ca-file=/etc/kubernetes/cert/ca.pem \\\n  --cert-file=/etc/flanneld/cert/flanneld.pem \\\n  --key-file=/etc/flanneld/cert/flanneld-key.pem \\\n  get ${FLANNEL_ETCD_PREFIX}/config\n```\n\n输出：\n\n`{\"Network\":\"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": {\"Type\": \"vxlan\"}}`\n\n查看已分配的 Pod 子网段列表(/24):\n\n``` bash\nsource /opt/k8s/bin/environment.sh\netcdctl \\\n  --endpoints=${ETCD_ENDPOINTS} \\\n  --ca-file=/etc/kubernetes/cert/ca.pem \\\n  --cert-file=/etc/flanneld/cert/flanneld.pem \\\n  --key-file=/etc/flanneld/cert/flanneld-key.pem \\\n  ls ${FLANNEL_ETCD_PREFIX}/subnets\n```\n\n输出（结果视部署情况而定）：\n\n``` bash\n/kubernetes/network/subnets/172.30.80.0-24\n/kubernetes/network/subnets/172.30.32.0-24\n/kubernetes/network/subnets/172.30.184.0-24\n```\n\n查看某一 Pod 网段对应的节点 IP 和 flannel 接口地址:\n\n``` bash\nsource /opt/k8s/bin/environment.sh\netcdctl \\\n  --endpoints=${ETCD_ENDPOINTS} \\\n  --ca-file=/etc/kubernetes/cert/ca.pem \\\n  --cert-file=/etc/flanneld/cert/flanneld.pem \\\n  --key-file=/etc/flanneld/cert/flanneld-key.pem \\\n  get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.80.0-24\n```\n\n输出（结果视部署情况而定）：\n\n`{\"PublicIP\":\"172.27.137.240\",\"BackendType\":\"vxlan\",\"BackendData\":{\"VtepMAC\":\"ce:9c:a9:08:50:03\"}}`\n\n+ 172.30.80.0/21 被分配给节点 zhangjun-k8s01（172.27.137.240）；\n+ VtepMAC 为 zhangjun-k8s01 节点的 flannel.1 网卡 MAC 地址；\n\n## 检查节点 flannel 网络信息\n\n``` bash\n[root@zhangjun-k8s01 work]# ip addr show\n1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000\n    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\n    inet 127.0.0.1/8 scope host lo\n       valid_lft forever preferred_lft forever\n2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000\n    link/ether 00:22:0d:33:89:75 brd ff:ff:ff:ff:ff:ff\n    inet 172.27.137.240/20 brd 172.27.143.255 scope global dynamic eth0\n       valid_lft 100647283sec preferred_lft 100647283sec\n3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default\n    link/ether ce:9c:a9:08:50:03 brd ff:ff:ff:ff:ff:ff\n    inet 172.30.80.0/32 scope global flannel.1\n       valid_lft forever preferred_lft forever\n```\n+ flannel.1 网卡的地址为分配的 Pod 子网段的第一个 IP（.0），且是 /32 的地址；\n\n``` bash\n[root@zhangjun-k8s01 work]# ip route show |grep flannel.1\n172.30.32.0/24 via 172.30.32.0 dev flannel.1 onlink\n172.30.184.0/24 via 172.30.184.0 dev flannel.1 onlink\n```\n+ 到其它节点 Pod 网段请求都被转发到 flannel.1 网卡；\n+ flanneld 根据 etcd 中子网段的信息，如 `${FLANNEL_ETCD_PREFIX}/subnets/172.30.80.0-24` ，来决定进请求发送给哪个节点的互联 IP；\n\n## 验证各节点能通过 Pod 网段互通\n\n在**各节点上部署** flannel 后，检查是否创建了 flannel 接口(名称可能为 flannel0、flannel.0、flannel.1 等)：\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh ${node_ip} \"/usr/sbin/ip addr show flannel.1|grep -w inet\"\n  done\n```\n\n输出：\n\n``` bash\n>>> 172.27.137.240\n    inet 172.30.80.0/32 scope global flannel.1\n>>> 172.27.137.239\n    inet 172.30.32.0/32 scope global flannel.1\n>>> 172.27.137.238\n    inet 172.30.184.0/32 scope global flannel.1\n```\n\n在各节点上 ping 所有 flannel 接口 IP，确保能通：\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh ${node_ip} \"ping -c 1 172.30.80.0\"\n    ssh ${node_ip} \"ping -c 1 172.30.32.0\"\n    ssh ${node_ip} \"ping -c 1 172.30.184.0\"\n  done\n```"
  },
  {
    "path": "F.部署docker.md",
    "content": "tags: worker, docker\n\n# F. 部署 docker 组件\n\n<!-- TOC -->\n\n- [F. 部署 docker 组件](#f-部署-docker-组件)\n    - [安装依赖包](#安装依赖包)\n    - [下载和分发 docker 二进制文件](#下载和分发-docker-二进制文件)\n    - [创建和分发 systemd unit 文件](#创建和分发-systemd-unit-文件)\n    - [配置和分发 docker 配置文件](#配置和分发-docker-配置文件)\n    - [启动 docker 服务](#启动-docker-服务)\n    - [检查服务运行状态](#检查服务运行状态)\n    - [检查 docker0 网桥](#检查-docker0-网桥)\n    - [查看 docker 的状态信息](#查看-docker-的状态信息)\n    - [更新 kubelet 配置并重启服务（每个节点上都操作）](#更新-kubelet-配置并重启服务每个节点上都操作)\n\n<!-- /TOC -->\n\ndocker 运行和管理容器，kubelet 通过 Container Runtime Interface (CRI) 与它进行交互。\n\n注意：\n1. 如果没有特殊指明，本文档的所有操作**均在 zhangjun-k8s01 节点上执行**，然后远程分发文件和执行命令；\n2. 需要先安装 flannel，请参考附件 [E.部署flannel网络.md](E.部署flannel网络.md)；\n\n## 安装依赖包\n\n参考 [07-0.部署worker节点.md](07-0.部署worker节点.md)。\n\n## 下载和分发 docker 二进制文件\n\n到 [docker 下载页面](https://download.docker.com/linux/static/stable/x86_64/) 下载最新发布包：\n\n``` bash\ncd /opt/k8s/work\nwget https://download.docker.com/linux/static/stable/x86_64/docker-18.09.6.tgz\ntar -xvf docker-18.09.6.tgz\n```\n\n分发二进制文件到所有 worker 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp docker/*  root@${node_ip}:/opt/k8s/bin/\n    ssh root@${node_ip} \"chmod +x /opt/k8s/bin/*\"\n  done\n```\n\n## 创建和分发 systemd unit 文件\n\n``` bash\ncd /opt/k8s/work\ncat > docker.service <<\"EOF\"\n[Unit]\nDescription=Docker Application Container Engine\nDocumentation=http://docs.docker.io\n\n[Service]\nWorkingDirectory=##DOCKER_DIR##\nEnvironment=\"PATH=/opt/k8s/bin:/bin:/sbin:/usr/bin:/usr/sbin\"\nEnvironmentFile=-/run/flannel/docker\nExecStart=/opt/k8s/bin/dockerd $DOCKER_NETWORK_OPTIONS\nExecReload=/bin/kill -s HUP $MAINPID\nRestart=on-failure\nRestartSec=5\nLimitNOFILE=infinity\nLimitNPROC=infinity\nLimitCORE=infinity\nDelegate=yes\nKillMode=process\n\n[Install]\nWantedBy=multi-user.target\nEOF\n```\n\n+ EOF 前后有双引号，这样 bash 不会替换文档中的变量，如 `$DOCKER_NETWORK_OPTIONS` (这些环境变量是 systemd 负责替换的。)；\n+ dockerd 运行时会调用其它 docker 命令，如 docker-proxy，所以需要将 docker 命令所在的目录加到 PATH 环境变量中；\n+ flanneld 启动时将网络配置写入 `/run/flannel/docker` 文件中，dockerd 启动前读取该文件中的环境变量 `DOCKER_NETWORK_OPTIONS` ，然后设置 docker0 网桥网段；\n+ 如果指定了多个 `EnvironmentFile` 选项，则必须将 `/run/flannel/docker` 放在最后(确保 docker0 使用 flanneld 生成的 bip 参数)；\n+ docker 需要以 root 用于运行；\n+ docker 从 1.13 版本开始，可能将 **iptables FORWARD chain的默认策略设置为DROP**，从而导致 ping 其它 Node 上的 Pod IP 失败，遇到这种情况时，需要手动设置策略为 `ACCEPT`：\n\n  ``` bash\n  $ sudo iptables -P FORWARD ACCEPT\n  ```\n\n  并且把以下命令写入 `/etc/rc.local` 文件中，防止节点重启**iptables FORWARD chain的默认策略又还原为DROP**\n\n  ``` bash\n  /sbin/iptables -P FORWARD ACCEPT\n  ```\n\n分发 systemd unit 文件到所有 worker 机器:\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nsed -i -e \"s|##DOCKER_DIR##|${DOCKER_DIR}|\" docker.service\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    scp docker.service root@${node_ip}:/etc/systemd/system/\n  done\n```\n\n## 配置和分发 docker 配置文件\n\n使用国内的仓库镜像服务器以加快 pull image 的速度，同时增加下载的并发数 (需要重启 dockerd 生效)：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\ncat > docker-daemon.json <<EOF\n{\n    \"registry-mirrors\": [\"https://docker.mirrors.ustc.edu.cn\",\"https://hub-mirror.c.163.com\"],\n    \"insecure-registries\": [\"docker02:35000\"],\n    \"max-concurrent-downloads\": 20,\n    \"live-restore\": true,\n    \"max-concurrent-uploads\": 10,\n    \"debug\": true,\n    \"data-root\": \"${DOCKER_DIR}/data\",\n    \"exec-root\": \"${DOCKER_DIR}/exec\",\n    \"log-opts\": {\n      \"max-size\": \"100m\",\n      \"max-file\": \"5\"\n    }\n}\nEOF\n```\n\n分发 docker 配置文件到所有 worker 节点：\n\n``` bash\ncd /opt/k8s/work\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"mkdir -p  /etc/docker/ ${DOCKER_DIR}/{data,exec}\"\n    scp docker-daemon.json root@${node_ip}:/etc/docker/daemon.json\n  done\n```\n\n## 启动 docker 服务\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl daemon-reload && systemctl enable docker && systemctl restart docker\"\n  done\n```\n\n## 检查服务运行状态\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"systemctl status docker|grep Active\"\n  done\n```\n\n确保状态为 `active (running)`，否则查看日志，确认原因：\n\n``` bash\njournalctl -u docker\n```\n\n## 检查 docker0 网桥\n\n``` bash\nsource /opt/k8s/bin/environment.sh\nfor node_ip in ${NODE_IPS[@]}\n  do\n    echo \">>> ${node_ip}\"\n    ssh root@${node_ip} \"/usr/sbin/ip addr show flannel.1 && /usr/sbin/ip addr show docker0\"\n  done\n```\n\n确认各 worker 节点的 docker0 网桥和 flannel.1 接口的 IP 处于同一个网段中(如下 172.30.80.0/32 位于 172.30.80.1/21 中)：\n\n``` bash\n>>> 172.27.137.240\n3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default\n    link/ether ce:9c:a9:08:50:03 brd ff:ff:ff:ff:ff:ff\n    inet 172.30.80.0/32 scope global flannel.1\n       valid_lft forever preferred_lft forever\n4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default\n    link/ether 02:42:5c:c1:77:03 brd ff:ff:ff:ff:ff:ff\n    inet 172.30.80.1/21 brd 172.30.87.255 scope global docker0\n       valid_lft forever preferred_lft forever\n```\n\n注意: 如果您的服务安装顺序不对或者机器环境比较复杂, docker服务早于flanneld服务安装，此时 worker 节点的 docker0 网桥和 flannel.1 接口的 IP可能不会同处同一个网段下，这个时候请先停止docker服务, 手工删除docker0网卡，重新启动docker服务后即可修复:\n\n```\nsystemctl stop docker\nip link delete docker0\nsystemctl start docker\n```\n\n## 查看 docker 的状态信息\n\n``` bash\n$ ps -elfH|grep docker\n4 S root     116590      1  0  80   0 - 131420 futex_ 11:22 ?       00:00:01   /opt/k8s/bin/dockerd --bip=172.30.80.1/21 --ip-masq=false --mtu=1450\n4 S root     116668 116590  1  80   0 - 161643 futex_ 11:22 ?       00:00:03     containerd --config /data/k8s/docker/exec/containerd/containerd.toml --log-level debug\n```\n\n``` bash\n$ docker info\nContainers: 0\n Running: 0\n Paused: 0\n Stopped: 0\nImages: 0\nServer Version: 18.09.6\nStorage Driver: overlay2\n Backing Filesystem: extfs\n Supports d_type: true\n Native Overlay Diff: true\nLogging Driver: json-file\nCgroup Driver: cgroupfs\nPlugins:\n Volume: local\n Network: bridge host macvlan null overlay\n Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog\nSwarm: inactive\nRuntimes: runc\nDefault Runtime: runc\nInit Binary: docker-init\ncontainerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84\nrunc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30\ninit version: fec3683\nSecurity Options:\n apparmor\n seccomp\n  Profile: default\nKernel Version: 4.14.110-0.el7.4pd.x86_64\nOperating System: CentOS Linux 7 (Core)\nOSType: linux\nArchitecture: x86_64\nCPUs: 8\nTotal Memory: 15.64GiB\nName: zhangjun-k8s01\nID: VJYK:3T6T:EPHU:65SM:3OZD:DMNE:MT5J:O22I:TCG2:F3JR:MZ76:B3EF\nDocker Root Dir: /data/k8s/docker/data\nDebug Mode (client): false\nDebug Mode (server): true\n File Descriptors: 22\n Goroutines: 43\n System Time: 2019-05-26T11:26:21.2494815+08:00\n EventsListeners: 0\nRegistry: https://index.docker.io/v1/\nLabels:\nExperimental: false\nInsecure Registries:\n docker02:35000\n 127.0.0.0/8\nRegistry Mirrors:\n https://docker.mirrors.ustc.edu.cn/\n https://hub-mirror.c.163.com/\nLive Restore Enabled: true\nProduct License: Community Engine\n\nWARNING: No swap limit support\n```\n\n## 更新 kubelet 配置并重启服务（每个节点上都操作）\n\n需要删除 kubelet 的 systemd unit 文件(/etc/systemd/system/kubelet.service)，删除下面 4 行：\n\n``` bash\n  --network-plugin=cni \\\\\n  --cni-conf-dir=/etc/cni/net.d \\\\\n  --container-runtime=remote \\\\\n  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \\\\\n```\n\n然后重启 kubelet 服务：\n\n``` bash\nsystemctl restart kubelet\n```"
  },
  {
    "path": "LICENSE",
    "content": "follow-me-install-kubernetes-cluster (c) by Jun Zhang.\n\nfollow-me-install-kubernetes-cluster is licensed under a\nCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.\n\nYou should have received a copy of the license along with this\nwork. If not, see <http://creativecommons.org/licenses/by-nc-sa/4.0/>.\n\nCC BY-NC-SA 4.0 协议条款\n\n知识共享 署名-非商业性使用-相同方式共享 4.0 国际许可协议\n\n以下内容只是一份易于理解的条款内容摘要，并不是[完整的协议条款](https://creativecommons.org/licenses/by-nc-sa/4.0/)。\n\n除非特别说明，否则您可以自由地：\n\n+ 分享：以任何媒介或格式复制及分发本站内容；\n+ 修改：重混、转换本站内容，以及依照本站内容创建演绎作品。\n\n唯独需要遵守以下条件：\n\n+ 署名：您必须按照作者或者许可人指定的方式署名，提供指向授权条款的链接，以及指出本站的内容是否已被变更。您不得以任何方式暗示您是许可人。\n+ 非商业性使用：您不得将本站内容用于任何商业目的。\n+ 相同方式共享：如果您重混、转换本站内容，以及依照本站内容创建演绎作品，您必须采用相关内容的授权条款或者与之兼容的其它授权条款来发布您的演绎作品。\n+ 不得增加额外限制：您不能增设法律条款或科技措施来限制别人行使授权条款已经给予的权利。\n\n如何署名？\n\n+ 在署名时，您只要在显著的位置注明您所使用的内容对应的网址链接（URL）即可。"
  },
  {
    "path": "README.md",
    "content": "# 和我一步步部署 kubernetes 集群\n\n![dashboard-home](./images/dashboard-home.png)\n\n本系列文档介绍使用二进制部署 `kubernetes v1.16.6` 集群的所有步骤（Hard-Way 模式）。\n\n在部署的过程中，将详细列出各组件的启动参数，它们的含义和可能遇到的问题。\n\n部署完成后，你将理解系统各组件的交互原理，进而能快速解决实际问题。\n\n所以本文档主要适合于那些有一定 kubernetes 基础，想通过一步步部署的方式来学习和了解系统配置、运行原理的人。\n\n本系列系文档适用于 `CentOS 7` 及以上版本系统，**随着各组件的更新而更新**，有任何问题欢迎提 issue！\n\n由于启用了 `x509` 证书双向认证、`RBAC` 授权等严格的安全机制，建议**从头开始部署**，否则可能会认证、授权等失败！\n\n从 v1.16.x 版本开始，本文档做了如下调整：\n1. 容器运行时：用 containerd 替换 docker，更加简单、健壮；相应的命令行工具为 crictl；\n2. Pod 网络：用 calico 替换 flannel 实现 Pod 互通，支持更大规模的集群；\n\n新增指标监控系统：使用主流的 Prometheus、Grafana 技术栈实现集群指标采集和监控；\n\n如果想继续使用 docker 和 flannel，请参考附件文档。\n\n## 历史版本\n\n+ [v1.6.2](https://github.com/opsnull/follow-me-install-kubernetes-cluster/tree/v1.6.2)：已停止更新；\n+ [v1.8.x](https://github.com/opsnull/follow-me-install-kubernetes-cluster/tree/v1.8.x)：已停止更新；\n+ [v1.10.x](https://github.com/opsnull/follow-me-install-kubernetes-cluster/tree/v1.10.x)：已停止更新；\n+ [v1.12.x](https://github.com/opsnull/follow-me-install-kubernetes-cluster/tree/v1.12.x)：已停止更新；\n+ [v1.14.x](https://github.com/opsnull/follow-me-install-kubernetes-cluster/tree/v1.14.x)：继续更新；\n\n## 步骤列表\n\n1. [00.组件版本和配置策略](00.组件版本和配置策略.md)\n1. [01.初始化系统和全局变量](01.初始化系统和全局变量.md)\n1. [02.创建CA根证书和秘钥](02.创建CA根证书和秘钥.md)\t\t\t\n1. [03.部署kubectl命令行工具](03.kubectl.md)\t\t\t\n1. [04.部署etcd集群](04.etcd集群.md)\t\t\t\t\n1. [05-1.部署master节点.md](05-1.master节点.md)\n    1. [05-2.apiserver集群](05-2.apiserver集群.md)\n    1. [05-3.controller-manager集群](05-3.controller-manager集群.md)\t\n    1. [05-4.scheduler集群](05-4.scheduler集群.md)\n1. [06-1.部署woker节点](06-1.worker节点.md)\t\t\t\n    1. [06-2.apiserver高可用之nginx代理](06-2.apiserver高可用.md)\n    1. [06-3.containerd](06-3.containerd.md)\t\t\t\t\t\n    1. [06-4.kubelet](06-4.kubelet.md)\t\t\t\t\n    1. [06-5.kube-proxy](06-5.kube-proxy.md)\n    1. [06-6.部署calico网络](06-6.calico.md)\t\n1. [07.验证集群功能](07.验证集群功能.md)\t\t\t\n1. [08-1.部署集群插件](08-1.部署集群插件.md)\n    1. [08-2.coredns插件](08-2.coredns插件.md)\n    1. [08-3.dashboard插件](08-3.dashboard插件.md)\n    1. [08-4.kube-prometheus插件](08-4.kube-prometheus插件.md)\n\t1. [08-5.EFK插件](08-5.EFK插件.md)\t\t\t\n1. [09.部署Docker-Registry](09.Registry.md)\t\n1. [10.清理集群](10.清理集群.md)\t\n1. [A.浏览器访问apiserver安全端口](A.浏览器访问kube-apiserver安全端口.md)\n1. [B.校验TLS证书](B.校验TLS证书.md)\n1. [C.部署metrics-server插件](C.metrics-server插件.md)\n1. [D.部署Harbor-Registry](D.部署Harbor-Registry.md)\t\n\n## 在线阅读\n\n+ 建议：[GitBook](https://k8s-install.opsnull.com/)\n+ [Github](https://www.gitbook.com/book/opsnull/follow-me-install-kubernetes-cluster)\n\n## 电子书\n\n+ pdf 格式 [下载](https://www.gitbook.com/download/pdf/book/opsnull/follow-me-install-kubernetes-cluster)\n+ epub 格式 [下载](https://www.gitbook.com/download/epub/book/opsnull/follow-me-install-kubernetes-cluster)\n\n## 打赏\n\n如果你觉得这份文档对你有帮助，请微信扫描下方的二维码进行捐赠，加油后的 opsnull 将会和你分享更多的原创教程，谢谢！\n\n<p align=\"center\">\n  <img src=\"https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/images/weixin_qr.jpg?raw=true\" alt=\"weixin_qr.jpg\"/>\n</p>\n\n## 广告位\n\n## 版权\n\nCopyright 2017-2020 zhangjun (geekard@qq.com)\n\n知识共享 署名-非商业性使用-相同方式共享 4.0（CC BY-NC-SA 4.0），详情见 [LICENSE](LICENSE) 文件。\n"
  },
  {
    "path": "SUMMARY.md",
    "content": "# Summary\n\n## 和我一步步部署 kubernetes 集群\n\n* [00.组件版本和配置策略](00.组件版本和配置策略.md)\n* [01.初始化系统和全局变量](01.初始化系统和全局变量.md)\n* [02.创建CA根证书和秘钥](02.创建CA根证书和秘钥.md)\t\t\t\n* [03.部署kubectl命令行工具](03.kubectl.md)\t\t\t\n* [04.部署etcd集群](04.etcd集群.md)\t\t\t\t\n* [05-1.部署master节点.md](05-1.master节点.md)\n    * [05-2.apiserver集群](05-2.apiserver集群.md)\n    * [05-3.controller-manager集群](05-3.controller-manager集群.md)\t\n    * [05-4.scheduler集群](05-4.scheduler集群.md)\n* [06-1.部署woker节点](06-1.worker节点.md)\t\t\t\n    * [06-2.apiserver高可用之nginx代理](06-2.apiserver高可用.md)\n    * [06-3.containerd](06-3.containerd.md)\t\t\t\t\t\n    * [06-4.kubelet](06-4.kubelet.md)\t\t\t\t\n    * [06-5.kube-proxy](06-5.kube-proxy.md)\n    * [06-6.部署calico网络](06-6.calico.md)\t\n* [07.验证集群功能](07.验证集群功能.md)\t\t\t\n* [08-1.部署集群插件](08-1.部署集群插件.md)\n    * [08-2.coredns插件](08-2.coredns插件.md)\n    * [08-3.dashboard插件](08-3.dashboard插件.md)\n    * [08-4.kube-prometheus插件](08-4.kube-prometheus插件.md)\n\t* [08-5.EFK插件](08-5.EFK插件.md)\t\t\t\n* [09.部署Docker-Registry](09.Registry.md)\t\n* [10.清理集群](10.清理集群.md)\t\n* [A.浏览器访问apiserver安全端口](A.浏览器访问kube-apiserver安全端口.md)\n* [B.校验TLS证书](B.校验TLS证书.md)\n* [C.部署metrics-server插件](C.metrics-server插件.md)\n* [D.部署Harbor-Registry](D.部署Harbor-Registry.md)\t\n* [E.部署flannel网络](E.部署flannel网络.md)\t\n* [F.部署docker](F.部署docker.md)\t\n\n## 标签集合\n\n* [标签](tags.md)\n"
  },
  {
    "path": "book.json",
    "content": "{\n  \"title\": \"和我一步步部署kubernetes集群\",\n  \"description\": \"和我一步步部署kubernetes集群\",\n  \"language\": \"zh-cn\",\n  \"plugins\": [\n    \"expandable-chapters\",\n    \"anchors\",\n    \"disqus\",\n    \"github\",\n    \"editlink\",\n    \"prism\", \"-highlight\",\n    \"-lunr\", \"-search\", \"search-plus\",\n    \"baidu\",\n    \"codesnippet\",\n    \"splitter\",\n    \"sitemap\",\n    \"page-toc-button\",\n    \"image-captions\",\n    \"page-footer-ex\",\n    \"tags\",\n    \"multipart\"\n  ],\n  \"pdf\": {\n    \"toc\": true,\n    \"pageNumbers\": true,\n    \"fontSize\": 11\n  },\n  \"pluginsConfig\": {\n    \"image-captions\": {\n          \"caption\": \"图片 - _CAPTION_\"\n    },\n    \"page-toc-button\": {\n      \"maxTocDepth\": 2,\n      \"minTocSize\": 2\n   \t},\n    \"expandable-chapters\":{\n    },\n    \"disqus\": {\n      \"shortName\": \"knowl\"\n    },\n    \"github\": {\n      \"url\": \"https://github.com/opsnull/follow-me-install-kubernetes-cluster\"\n    },\n    \"editlink\": {\n      \"base\": \"https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/\",\n      \"label\": \"编辑本页\"\n    },\n    \"baidu\": {\n        \"token\": \"18c1967ba7abeaf5be1330bb512f0617\"\n    },\n    \"sitemap\": {\n        \"hostname\": \"http://k8s.opsnull.com\"\n    },\n    \"page-footer-ex\":{\n        \"copyright\":\"zhangjun\",\n        \"update_label\":\"最后更新：\",\n        \"update_format\":\"YYYY-MM-DD HH:mm:ss\"\n    },\n    \"tags\": {\n        \"placement\": \"bottom\"\n    },\n    \"sharing\": {\n        \"facebook\": true,\n        \"twitter\": true,\n        \"google\": true,\n        \"weibo\": true,\n        \"instapaper\": false,\n        \"vk\": false,\n        \"all\": [\n            \"facebook\", \"google\", \"twitter\",\n            \"weibo\", \"instapaper\"\n        ]\n    }\n  }\n}"
  },
  {
    "path": "manifests/environment.sh",
    "content": "#!/usr/bin/bash\n\n# 生成 EncryptionConfig 所需的加密 key\nexport ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)\n\n# 集群各机器 IP 数组\nexport NODE_IPS=(172.27.138.251 172.27.137.229 172.27.138.239)\n\n# 集群各 IP 对应的主机名数组\nexport NODE_NAMES=(zhangjun-k8s-01 zhangjun-k8s-02 zhangjun-k8s-03)\n\n# etcd 集群服务地址列表\nexport ETCD_ENDPOINTS=\"https://172.27.138.251:2379,https://172.27.137.229:2379,https://172.27.138.239:2379\"\n\n# etcd 集群间通信的 IP 和端口\nexport ETCD_NODES=\"zhangjun-k8s-01=https://172.27.138.251:2380,zhangjun-k8s-02=https://172.27.137.229:2380,zhangjun-k8s-03=https://172.27.138.239:2380\"\n\n# kube-apiserver 的反向代理(kube-nginx)地址端口\nexport KUBE_APISERVER=\"https://127.0.0.1:8443\"\n\n# 节点间互联网络接口名称\nexport IFACE=\"eth0\"\n\n# etcd 数据目录\nexport ETCD_DATA_DIR=\"/data/k8s/etcd/data\"\n\n# etcd WAL 目录，建议是 SSD 磁盘分区，或者和 ETCD_DATA_DIR 不同的磁盘分区\nexport ETCD_WAL_DIR=\"/data/k8s/etcd/wal\"\n\n# k8s 各组件数据目录\nexport K8S_DIR=\"/data/k8s/k8s\"\n\n## DOCKER_DIR 和 CONTAINERD_DIR 二选一\n# docker 数据目录\nexport DOCKER_DIR=\"/data/k8s/docker\"\n\n# containerd 数据目录\nexport CONTAINERD_DIR=\"/data/k8s/containerd\"\n\n## 以下参数一般不需要修改\n\n# TLS Bootstrapping 使用的 Token，可以使用命令 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 生成\nBOOTSTRAP_TOKEN=\"41f7e4ba8b7be874fcff18bf5cf41a7c\"\n\n# 最好使用 当前未用的网段 来定义服务网段和 Pod 网段\n\n# 服务网段，部署前路由不可达，部署后集群内路由可达(kube-proxy 保证)\nSERVICE_CIDR=\"10.254.0.0/16\"\n\n# Pod 网段，建议 /16 段地址，部署前路由不可达，部署后集群内路由可达(flanneld 保证)\nCLUSTER_CIDR=\"172.30.0.0/16\"\n\n# 服务端口范围 (NodePort Range)\nexport NODE_PORT_RANGE=\"30000-32767\"\n\n# kubernetes 服务 IP (一般是 SERVICE_CIDR 中第一个IP)\nexport CLUSTER_KUBERNETES_SVC_IP=\"10.254.0.1\"\n\n# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配)\nexport CLUSTER_DNS_SVC_IP=\"10.254.0.2\"\n\n# 集群 DNS 域名（末尾不带点号）\nexport CLUSTER_DNS_DOMAIN=\"cluster.local\"\n\n# 将二进制目录 /opt/k8s/bin 加到 PATH 中\nexport PATH=/opt/k8s/bin:$PATH"
  },
  {
    "path": "tags.md",
    "content": "# Tags"
  }
]